Skip to content

docs: Add documentation on Spark 4 limitations #4018

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
andygrove wants to merge 1 commit into
base: main
Choose a base branch
from
+21 −4
Open

docs: Add documentation on Spark 4 limitations #4018

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ benefits of Comet's acceleration capabilities without disrupting your Spark appl

## Getting Started

Comet supports Apache Spark 3.4 and 3.5, and provides experimental support for Spark 4.0. See the
Comet supports Apache Spark 3.4, 3.5, and 4.0. See the
[installation guide](https://datafusion.apache.org/comet/user-guide/installation.html) for the detailed
version, Java, and Scala compatibility matrix.

Expand Down
4 changes: 2 additions & 2 deletions docs/source/contributor-guide/roadmap.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,8 +34,8 @@ encryption) to the reader.

### Spark 4.0 Support

Comet has experimental support for Spark 4.0, but there is more work to do ([#1637]), such as enabling
more Spark SQL tests and fully implementing ANSI support ([#313]) for all supported expressions.
Comet fully supports Spark 4.0. There is ongoing work to fully implement ANSI support ([#313]) for all
supported expressions and to address remaining [Spark 4.0-specific limitations](../../user-guide/latest/compatibility.md#spark-40-limitations).

[#313]: https://github.com/apache/datafusion-comet/issues/313
[#1637]: https://github.com/apache/datafusion-comet/issues/1637
Expand Down
17 changes: 17 additions & 0 deletions docs/source/user-guide/latest/compatibility.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,23 @@ achieves the same semantic goals:
The only difference is that Comet's partition assignments will differ from Spark's. When results are sorted,
they will be identical to Spark. Unsorted results may have different row ordering.

## Spark 4.0 Limitations

The following limitations apply specifically when running Comet with Spark 4.0. These represent new Spark
4.0 features that Comet does not yet support. Queries that use these features will fall back to Spark.

### Variant Type

Comet does not yet support the Variant data type introduced in Spark 4.0. All variant-related operations
will fall back to Spark. Tracking issue: [#2209](https://github.com/apache/datafusion-comet/issues/2209)
Comment on lines +134 to +136
Copy link
Copy Markdown
Member Author

@andygrove andygrove Apr 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I need to confirm the actual behavior for variant

Copy link
Copy Markdown
Member Author

@andygrove andygrove Apr 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed in #4021


### Parquet Type Widening

Comet's native Parquet reader does not support Spark 4.0's type widening feature, which allows reading
Parquet columns whose types have been widened (e.g., `int` to `long`, `float` to `double`,
`date` to `timestamp_ntz`). These reads will fall back to Spark.
Tracking issue: [#3321](https://github.com/apache/datafusion-comet/issues/3321)

## Cast

Cast operations in Comet fall into three levels of support:
Expand Down
2 changes: 1 addition & 1 deletion docs/source/user-guide/latest/installation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ through the release vote. Release candidate jars can be found [here](https://rep
- [Comet plugin for Spark 3.4 / Scala 2.13](https://repo1.maven.org/maven2/org/apache/datafusion/comet-spark-spark3.4_2.13/$COMET_VERSION/comet-spark-spark3.4_2.13-$COMET_VERSION.jar)
- [Comet plugin for Spark 3.5 / Scala 2.12](https://repo1.maven.org/maven2/org/apache/datafusion/comet-spark-spark3.5_2.12/$COMET_VERSION/comet-spark-spark3.5_2.12-$COMET_VERSION.jar)
- [Comet plugin for Spark 3.5 / Scala 2.13](https://repo1.maven.org/maven2/org/apache/datafusion/comet-spark-spark3.5_2.13/$COMET_VERSION/comet-spark-spark3.5_2.13-$COMET_VERSION.jar)
- [Comet plugin for Spark 4.0 / Scala 2.13 (Experimental)](https://repo1.maven.org/maven2/org/apache/datafusion/comet-spark-spark4.0_2.13/$COMET_VERSION/comet-spark-spark4.0_2.13-$COMET_VERSION.jar)
- [Comet plugin for Spark 4.0 / Scala 2.13](https://repo1.maven.org/maven2/org/apache/datafusion/comet-spark-spark4.0_2.13/$COMET_VERSION/comet-spark-spark4.0_2.13-$COMET_VERSION.jar)

## Building from source

Expand Down
Loading