Merge branch 'main' into fix_comet_sum_compatbility_level

Author: coderfender

.claude/skills/audit-comet-expression/SKILL.md

@@ -13,7 +13,7 @@ This audit covers:
 1. Spark implementation across versions 3.4.3, 3.5.8, and 4.0.1
 2. Comet Scala serde implementation
 3. Comet Rust / DataFusion implementation
-4. Existing test coverage (SQL file tests and Scala tests)
+4. Existing test coverage (Comet SQL Tests and Comet Scala Tests)
 5. Gap analysis and test recommendations
 
 ---
@@ -161,7 +161,7 @@ Read the Rust implementation and check:
 
 ## Step 4: Locate Existing Comet Tests
 
-### SQL file tests
+### Comet SQL Tests
 
 ```bash
 # Find SQL test files for this expression
@@ -179,13 +179,13 @@ Read every SQL test file found and list:
 - Query modes used (`query`, `spark_answer_only`, `tolerance`, `ignore`, `expect_error`)
 - Any ConfigMatrix directives
 
-### Scala tests
+### Comet Scala Tests
 
 ```bash
 grep -r "$ARGUMENTS" spark/src/test/scala/ --include="*.scala" -l
 ```
 
-Read the relevant Scala test files and list:
+Read the relevant Comet Scala Tests and list:
 
 - Input types covered
 - Edge cases exercised
@@ -201,7 +201,7 @@ Compare the Spark test coverage (Step 2) against the Comet test coverage (Step 4
 
 For each of the following dimensions, note whether it is covered in Comet tests or missing:
 
-| Dimension                                                                                              | Spark tests it | Comet SQL test | Comet Scala test | Gap? |
+| Dimension                                                                                              | Spark tests it | Comet SQL Test | Comet Scala Test | Gap? |
 | ------------------------------------------------------------------------------------------------------ | -------------- | -------------- | ---------------- | ---- |
 | Column reference argument(s)                                                                           |                |                |                  |      |
 | Literal argument(s)                                                                                    |                |                |                  |      |
@@ -256,13 +256,13 @@ After presenting the gap analysis, ask the user:
 >
 > - [list each missing test case]
 >
-> I can add them as SQL file tests in `spark/src/test/resources/sql-tests/expressions/<category>/$ARGUMENTS.sql`
-> (or as Scala tests in `CometExpressionSuite` for cases that require programmatic setup).
+> I can add them as Comet SQL Tests in `spark/src/test/resources/sql-tests/expressions/<category>/$ARGUMENTS.sql`
+> (or as Comet Scala Tests in `CometExpressionSuite` for cases that require programmatic setup).
 
-If the user says yes, implement the missing tests following the SQL file test format described in
-`docs/source/contributor-guide/sql-file-tests.md`. Prefer SQL file tests over Scala tests.
+If the user says yes, implement the missing tests following the Comet SQL Tests format described in
+`docs/source/contributor-guide/sql-file-tests.md`. Prefer Comet SQL Tests over Comet Scala Tests.
 
-### SQL file test template
+### Comet SQL Tests template
 
 ```sql
 -- Licensed to the Apache Software Foundation (ASF) under one

.claude/skills/review-comet-pr/SKILL.md

@@ -138,15 +138,15 @@ Location: `native/spark-expr/src/`
 - [ ] No panics. Use `Result` types.
 - [ ] Efficient array operations (avoid row-by-row)
 
-#### Tests - Prefer SQL File-Based Framework
+#### Tests - Prefer Comet SQL Tests
 
-**Expression tests should use the SQL file-based framework (`CometSqlFileTestSuite`) where possible.** This framework automatically runs each query through both Spark and Comet and compares results. No Scala code is needed. Only fall back to Scala tests in `CometExpressionSuite` when the SQL framework cannot express the test. Examples include complex `DataFrame` setup, programmatic data generation, or non-expression tests.
+**Expression tests should use Comet SQL Tests (`CometSqlFileTestSuite`) where possible.** This framework automatically runs each query through both Spark and Comet and compares results. No Scala code is needed. Only fall back to Comet Scala Tests in `CometExpressionSuite` when Comet SQL Tests cannot express the test. Examples include complex `DataFrame` setup, programmatic data generation, or non-expression tests.
 
-**SQL file test location:** `spark/src/test/resources/sql-tests/expressions/<category>/`
+**Comet SQL Test location:** `spark/src/test/resources/sql-tests/expressions/<category>/`
 
 Categories include: `aggregate/`, `array/`, `string/`, `math/`, `struct/`, `map/`, `datetime/`, `hash/`, etc.
 
-**SQL file structure:**
+**Comet SQL Test structure:**
 
 ```sql
 -- Create test data
@@ -181,10 +181,10 @@ query ignore(https://github.com/apache/datafusion-comet/issues/NNNN)
 SELECT known_buggy_expr(v) FROM test_table
 ```
 
-**Running SQL file tests:**
+**Running Comet SQL Tests:**
 
 ```bash
-# All SQL file tests
+# All Comet SQL Tests
 ./mvnw test -Dsuites="org.apache.comet.CometSqlFileTestSuite" -Dtest=none
 
 # Specific test file (substring match)
@@ -199,7 +199,7 @@ SELECT known_buggy_expr(v) FROM test_table
 - [ ] Both literal values and column references tested (they use different code paths)
 - [ ] For timestamp/datetime expressions, timezone handling is tested (e.g., UTC, non-UTC session timezone, timestamps with and without timezone)
 - [ ] One expression per SQL file for easier debugging
-- [ ] If using Scala tests instead, literal tests MUST disable constant folding:
+- [ ] If using Comet Scala Tests instead, literal tests MUST disable constant folding:
   ```scala
   withSQLConf(SQLConf.OPTIMIZER_EXCLUDED_RULES.key ->
       "org.apache.spark.sql.catalyst.optimizer.ConstantFolding") {
@@ -256,7 +256,7 @@ If the PR adds a new expression or operator but does not update the relevant doc
 1. **Incomplete type support**: Spark expression supports types not handled in PR
 2. **Missing edge cases**: Null, overflow, empty string, negative values
 3. **Wrong return type**: Return type must match Spark exactly
-4. **Tests in wrong framework**: Expression tests should use the SQL file-based framework (`CometSqlFileTestSuite`) rather than adding to Scala test suites like `CometExpressionSuite`. Suggest migration if the PR adds Scala tests for expressions that could use SQL files instead.
+4. **Tests in wrong framework**: Expression tests should use Comet SQL Tests (`CometSqlFileTestSuite`) rather than adding to Comet Scala Tests like `CometExpressionSuite`. Suggest migration if the PR adds Comet Scala Tests for expressions that could use Comet SQL Tests instead.
 5. **Stale native code**: PR might need `./mvnw install -pl common -DskipTests`
 6. **Missing `getSupportLevel`**: Edge cases should be marked as `Incompatible`
 

.github/workflows/pr_build_linux.yml

@@ -140,6 +140,38 @@ jobs:
         run: |
           ./dev/ci/check-working-tree-clean.sh
 
+  # Compile-only verification for Spark 4.1. Tests are intentionally skipped: the spark-4.1
+  # profile is currently a build target only, and several runtime/test failures are tracked
+  # in follow-up PRs. Excluded from lint-java because semanticdb-scalac_2.13.17 is not yet
+  # published and the lint job activates -Psemanticdb.
+  build-spark-4-1:
+    needs: lint
+    name: Build Spark 4.1, JDK 17
+    runs-on: ubuntu-latest
+    container:
+      image: amd64/rust
+    steps:
+      - uses: actions/checkout@v6
+
+      - name: Setup Rust & Java toolchain
+        uses: ./.github/actions/setup-builder
+        with:
+          rust-version: ${{ env.RUST_VERSION }}
+          jdk-version: 17
+
+      - name: Cache Maven dependencies
+        uses: actions/cache@v5
+        with:
+          path: |
+            ~/.m2/repository
+            /root/.m2/repository
+          key: ${{ runner.os }}-java-maven-${{ hashFiles('**/pom.xml') }}-spark-4.1-build
+          restore-keys: |
+            ${{ runner.os }}-java-maven-
+
+      - name: Compile (skip tests)
+        run: ./mvnw -B install -DskipTests -Dmaven.test.skip=true -Pspark-4.1
+
   # Build native library once and share with all test jobs
   build-native:
     needs: lint
@@ -248,19 +280,9 @@ jobs:
             maven_opts: "-Pspark-3.4 -Pscala-2.12"
             scan_impl: "auto"
 
-          - name: "Spark 3.5.5, JDK 17, Scala 2.13"
+          - name: "Spark 3.5, JDK 17, Scala 2.13"
             java_version: "17"
-            maven_opts: "-Pspark-3.5 -Dspark.version=3.5.5 -Pscala-2.13"
-            scan_impl: "auto"
-
-          - name: "Spark 3.5.6, JDK 17, Scala 2.13"
-            java_version: "17"
-            maven_opts: "-Pspark-3.5 -Dspark.version=3.5.6 -Pscala-2.13"
-            scan_impl: "auto"
-
-          - name: "Spark 3.5, JDK 17, Scala 2.12"
-            java_version: "17"
-            maven_opts: "-Pspark-3.5 -Pscala-2.12"
+            maven_opts: "-Pspark-3.5 -Pscala-2.13"
             scan_impl: "native_iceberg_compat"
 
           - name: "Spark 4.0, JDK 17"

.github/workflows/spark_sql_test.yml

@@ -140,7 +140,7 @@ jobs:
         config:
           - {spark-short: '3.4', spark-full: '3.4.3', java: 11, scan-impl: 'auto'}
           - {spark-short: '3.5', spark-full: '3.5.8', java: 11, scan-impl: 'auto'}
-          - {spark-short: '4.0', spark-full: '4.0.1', java: 17, scan-impl: 'auto'}
+          - {spark-short: '4.0', spark-full: '4.0.2', java: 17, scan-impl: 'auto'}
       fail-fast: false
     name: spark-sql-${{ matrix.config.scan-impl }}-${{ matrix.module.name }}/spark-${{ matrix.config.spark-full }}
     runs-on: ${{ matrix.os }}

…g/apache/comet/shims/CometTypeShim.scala …g/apache/comet/shims/CometTypeShim.scalacommon/src/main/spark-4.0/org/apache/comet/shims/CometTypeShim.scala renamed to common/src/main/spark-4.x/org/apache/comet/shims/CometTypeShim.scala common/src/main/spark-4.0/org/apache/comet/shims/CometTypeShim.scala renamed to common/src/main/spark-4.x/org/apache/comet/shims/CometTypeShim.scala

@@ -3774,7 +3774,7 @@ index 4b27082e188..057b2430872 100644
 -class HiveUDFDynamicLoadSuite extends QueryTest with SQLTestUtils with TestHiveSingleton {
 +// Comet: mix in IgnoreCometSuite so these tests are reported as ignored when Comet is enabled
 +// (ENABLE_COMET=true). The jar these tests depend on (`hive-test-udfs.jar`) is stripped from the
-+// Spark 4.0.1 release source tag per the ASF binary-artifact policy, so the tests cannot run in
++// Spark 4.0.2 release source tag per the ASF binary-artifact policy, so the tests cannot run in
 +// Comet's CI. Ignoring keeps the suite passing without masking real regressions; the upstream
 +// tests still run in non-Comet Spark builds that ship the jar on branch-4.0.
 +class HiveUDFDynamicLoadSuite extends QueryTest with SQLTestUtils with TestHiveSingleton
@@ -3788,7 +3788,7 @@ index 4b27082e188..057b2430872 100644
      val jarPath = "src/test/noclasspath/hive-test-udfs.jar"
 -    assume(new java.io.File(jarPath).exists)
 +    // Comet: the upstream `assume(...)` runs here in the suite constructor (inside this foreach,
-+    // before `test(...)` registers a case). When the jar is missing - as it is on the v4.0.1
++    // before `test(...)` registers a case). When the jar is missing - as it is on the v4.0.2
 +    // release tag - `assume` throws TestCanceledException out of `<init>`, which ScalaTest
 +    // reports as a suite abort (not a per-test cancel) and fails the whole job. The
 +    // IgnoreCometSuite mixin above already reroutes these tests to `ignore` under Comet, so