Feat: to_json Infinity/-Infinity Nan values support#3875
Feat: to_json Infinity/-Infinity Nan values support#3875kazantsev-maksim wants to merge 51 commits intoapache:mainfrom
Conversation
This reverts commit 768b3e9.
|
I think we need add new tests for +/- inf and NaN |
|
Test failed because since Spark 4.0.0 to_json use runtime expression replacement: https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala#L418 |
|
I marked sql-tests as ignored until to_json will not support Spark 4.0.0. CC @andygrove |
| SELECT to_json(named_struct('a', 1, 'b', 'hello')) | ||
|
|
||
| -- query expect_fallback(StructsToJson with options is not supported) | ||
| query ignore("Need support Spark 4.0.0") |
There was a problem hiding this comment.
Hmm does this include +/- Inf and Nan cases?
Also do you mind filing an issue to followup for Spark 4
There was a problem hiding this comment.
Thanks, added.
|
@kazuyukitanimura @andygrove Could you take another see when you have time. |
| -- query expect_fallback(Struct type: StructType(StructField(a,ArrayType(IntegerType,false),false)) contains unsupported types) | ||
| query ignore("Need support Spark 4.0.0") | ||
| SELECT to_json(named_struct(a, array(b))) |
There was a problem hiding this comment.
could you create a separate structs_to_json_spark4.sql for these tests and set min spark version 4 for that file, and change these to expect_fallback
There was a problem hiding this comment.
I updated the branch from master and updated the tests.
…support # Conflicts: # spark/src/main/scala/org/apache/comet/serde/structs.scala
3 participants
Which issue does this PR close?
Closes #3016
Rationale for this change
What changes are included in this PR?
How are these changes tested?
Tested with existing unit tests