test: re-enable sql_hive-1 for Spark 4.0 and fix two small failures#4047
Open
andygrove wants to merge 4 commits intoapache:mainfrom
Open
test: re-enable sql_hive-1 for Spark 4.0 and fix two small failures#4047andygrove wants to merge 4 commits intoapache:mainfrom
andygrove wants to merge 4 commits intoapache:mainfrom
Conversation
The Spark 4.0 `ShimSparkErrorConverter` was converting native FileNotFound
errors into a `SparkFileNotFoundException` with error class
`_LEGACY_ERROR_TEMP_2055`, but that error class was removed in Spark 4.0.
Throwing it triggers an internal error ("Cannot find main error class")
and fails tests such as `HiveMetadataCacheSuite` that assert on
`FAILED_READ_FILE.FILE_NOT_EXIST`.
Delegate to `QueryExecutionErrors.fileNotExistError`, which is the 4.0
replacement for `readCurrentFileNotFoundError` and produces the expected
error class and `path` parameter.
The 4.0.1 Spark diff commented out the `assume(new java.io.File(jarPath).exists)` guard in HiveUDFDynamicLoadSuite. The jar (hive-test-udfs.jar) is not shipped with Spark 4.0, so without the guard the five UDF/UDAF/UDTF tests run without the required class and fail with CANNOT_LOAD_FUNCTION_CLASS. Drop the hunk so upstream's assume() is preserved. The tests are cancelled on Spark 4.0 (jar missing) and continue to run on Spark 3.4/3.5 (jar present), matching upstream behavior. Closes apache#4049.
andygrove
changed the title
andygrove
changed the title
The upstream assume(jarPath.exists) runs during class construction (inside udfTestInfos.foreach, before test() registers a case), so when hive-test-udfs.jar is absent - as it is on the v4.0.1 release tag - the TestCanceledException propagates out of <init> and ScalaTest marks the whole suite as aborted, failing the job. Mix in IgnoreCometSuite so the five tests are reported as ignored under Comet, and comment out the constructor-time assume so it no longer aborts the suite.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #2946.
Closes #4049.
Rationale for this change
The
sql_hive-1job for Spark 4.0 has been excluded from CI since #2946 was filed (hang/timeout on the Hive suite with JDK 17). A re-run on currentmainshows the hang no longer reproduces: the job completes in about 63 minutes. It does surface two small issues, which this PR also fixes, so the job can be re-enabled and kept green.What changes are included in this PR?
.github/workflows/spark_sql_test.yml: remove theexcludematrix entry that skippedsql_hive-1forspark-4.0.1 / java 17 / auto.spark/src/main/spark-4.0/org/apache/spark/sql/comet/shims/ShimSparkErrorConverter.scala: theFileNotFoundcase was producing aSparkFileNotFoundExceptionwith error class_LEGACY_ERROR_TEMP_2055, which was removed in Spark 4.0. Delegate toQueryExecutionErrors.fileNotExistError(path, cause)so the error carriesFAILED_READ_FILE.FILE_NOT_EXIST, which is whatHiveMetadataCacheSuite(and similarcheckErrorassertions) expect.dev/diffs/4.0.1.diff: drop the hunk that commented outassume(new java.io.File(jarPath).exists)inHiveUDFDynamicLoadSuite. Spark 4.0 no longer shipshive-test-udfs.jar, so restoring the upstreamassumecancels the five UDF tests cleanly instead of running them without the required classes and failing withCANNOT_LOAD_FUNCTION_CLASS.How are these changes tested?
The CI run on this PR exercises the full
sql_hive-1job for Spark 4.0. The previously failing suites behave as follows:HiveMetadataCacheSuite(4 failures): now pass against the corrected error class.HiveUDFDynamicLoadSuite(5 failures): now cancelled on Spark 4.0 (jar absent); still run on Spark 3.4/3.5 (jar present).