diff --git a/README.md b/README.md index aff95df508..8c3156362a 100644 --- a/README.md +++ b/README.md @@ -81,7 +81,7 @@ benefits of Comet's acceleration capabilities without disrupting your Spark appl ## Getting Started -Comet supports Apache Spark 3.4 and 3.5, and provides experimental support for Spark 4.0. See the +Comet supports Apache Spark 3.4, 3.5, and 4.0. See the [installation guide](https://datafusion.apache.org/comet/user-guide/installation.html) for the detailed version, Java, and Scala compatibility matrix. diff --git a/docs/source/contributor-guide/roadmap.md b/docs/source/contributor-guide/roadmap.md index c685e0fb80..46f6671a74 100644 --- a/docs/source/contributor-guide/roadmap.md +++ b/docs/source/contributor-guide/roadmap.md @@ -34,8 +34,8 @@ encryption) to the reader. ### Spark 4.0 Support -Comet has experimental support for Spark 4.0, but there is more work to do ([#1637]), such as enabling -more Spark SQL tests and fully implementing ANSI support ([#313]) for all supported expressions. +Comet fully supports Spark 4.0. There is ongoing work to fully implement ANSI support ([#313]) for all +supported expressions and to address remaining [Spark 4.0-specific limitations](../../user-guide/latest/compatibility.md#spark-40-limitations). [#313]: https://github.com/apache/datafusion-comet/issues/313 [#1637]: https://github.com/apache/datafusion-comet/issues/1637 diff --git a/docs/source/user-guide/latest/compatibility.md b/docs/source/user-guide/latest/compatibility.md index 1b66345210..e8f209bc1b 100644 --- a/docs/source/user-guide/latest/compatibility.md +++ b/docs/source/user-guide/latest/compatibility.md @@ -125,6 +125,23 @@ achieves the same semantic goals: The only difference is that Comet's partition assignments will differ from Spark's. When results are sorted, they will be identical to Spark. Unsorted results may have different row ordering. +## Spark 4.0 Limitations + +The following limitations apply specifically when running Comet with Spark 4.0. These represent new Spark +4.0 features that Comet does not yet support. Queries that use these features will fall back to Spark. + +### Variant Type + +Comet does not yet support the Variant data type introduced in Spark 4.0. All variant-related operations +will fall back to Spark. Tracking issue: [#2209](https://github.com/apache/datafusion-comet/issues/2209) + +### Parquet Type Widening + +Comet's native Parquet reader does not support Spark 4.0's type widening feature, which allows reading +Parquet columns whose types have been widened (e.g., `int` to `long`, `float` to `double`, +`date` to `timestamp_ntz`). These reads will fall back to Spark. +Tracking issue: [#3321](https://github.com/apache/datafusion-comet/issues/3321) + ## Cast Cast operations in Comet fall into three levels of support: diff --git a/docs/source/user-guide/latest/installation.md b/docs/source/user-guide/latest/installation.md index f04e6359ba..ed856513d1 100644 --- a/docs/source/user-guide/latest/installation.md +++ b/docs/source/user-guide/latest/installation.md @@ -74,7 +74,7 @@ through the release vote. Release candidate jars can be found [here](https://rep - [Comet plugin for Spark 3.4 / Scala 2.13](https://repo1.maven.org/maven2/org/apache/datafusion/comet-spark-spark3.4_2.13/$COMET_VERSION/comet-spark-spark3.4_2.13-$COMET_VERSION.jar) - [Comet plugin for Spark 3.5 / Scala 2.12](https://repo1.maven.org/maven2/org/apache/datafusion/comet-spark-spark3.5_2.12/$COMET_VERSION/comet-spark-spark3.5_2.12-$COMET_VERSION.jar) - [Comet plugin for Spark 3.5 / Scala 2.13](https://repo1.maven.org/maven2/org/apache/datafusion/comet-spark-spark3.5_2.13/$COMET_VERSION/comet-spark-spark3.5_2.13-$COMET_VERSION.jar) -- [Comet plugin for Spark 4.0 / Scala 2.13 (Experimental)](https://repo1.maven.org/maven2/org/apache/datafusion/comet-spark-spark4.0_2.13/$COMET_VERSION/comet-spark-spark4.0_2.13-$COMET_VERSION.jar) +- [Comet plugin for Spark 4.0 / Scala 2.13](https://repo1.maven.org/maven2/org/apache/datafusion/comet-spark-spark4.0_2.13/$COMET_VERSION/comet-spark-spark4.0_2.13-$COMET_VERSION.jar) ## Building from source