Skip to content

test: add sql-file test confirming fallback on parquet variant reads#4021

Open
andygrove wants to merge 1 commit intoapache:mainfrom
andygrove:variant-fallback
Open

test: add sql-file test confirming fallback on parquet variant reads#4021
andygrove wants to merge 1 commit intoapache:mainfrom
andygrove:variant-fallback

Conversation

@andygrove
Copy link
Copy Markdown
Member

@andygrove andygrove commented Apr 21, 2026

Which issue does this PR close?

Part of #1637

Rationale for this change

Comet does not support Spark 4.0's VARIANT data type, so scans exposing a VariantType column must fall back to Spark. There is no regression test pinning this behavior today. If a future change silently lets a VariantType column through the scan, we would only notice through runtime errors or wrong results. A targeted sql-file test locks in the fallback contract.

What changes are included in this PR?

  • New file spark/src/test/resources/sql-tests/expressions/misc/variant.sql
  • Gated with -- MinSparkVersion: 4.0 so it is skipped on Spark 3.4 / 3.5 via CometSqlFileTestSuite
  • Creates parquet-backed tables with a VARIANT column (directly and nested inside STRUCT<v: VARIANT>), populated via parse_json
  • Uses query expect_fallback(Unsupported v of type VariantType) on five queries covering projection, variant_get extraction, variant_get in a predicate, a COUNT(*) with a not-null filter, and a struct-containing-variant projection

How are these changes tested?

Ran the new test under the Spark 4.0 profile:

./mvnw test -Pspark-4.0 -Dsuites="org.apache.comet.CometSqlFileTestSuite variant" -Dtest=none

All five queries in the file fall back with the expected reason and the suite reports Tests: succeeded 1, failed 0.

Adds a Spark 4.0+ sql-file test that creates parquet-backed tables with a VARIANT column (both directly and nested inside a struct) and asserts Comet falls back to Spark with reason "Unsupported v of type VariantType" for projection, filter, and aggregation queries. Gated by MinSparkVersion: 4.0 so older Spark profiles skip it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant