Add Apache Spark 4.0 support (#787)#941
Open
imarios wants to merge 7 commits into
Open
Conversation
Adds an in-tree `frameless-*-spark40` module set targeting Spark 4.0.2, cross-built for Scala 2.13 only (Spark 4 dropped 2.12) and requiring JDK 17. No external shim dependency: version-divergent Catalyst access is isolated behind FramelessInternals in a `src/main/spark-4` source overlay, mirroring the existing spark-3 / spark-3.4+ pattern. Key adaptations for Spark 4: - Column no longer wraps a Catalyst Expression; bridge through classic.ExpressionUtils.column and an eager ColumnNodeToExpressionConverter (the lazy ColumnNodeExpression is Unevaluable and hides children, which broke self-join disambiguation and codegen). - Dataset/SparkSession split into abstract API + classic impl; internal helpers downcast to classic for logicalPlan/sessionState/sqlContext. - ExpressionEncoder now takes a leading AgnosticEncoder (SPARK-49025); supply a metadata-only JavaBeanEncoder stand-in carrying the right ClassTag. - AnalysisException is errorClass-based; MapGroups gets a spark-4 variant. - joinCross re-encodes its result via TypedExpressionEncoder, consistent with the other joins. - Hide the new catalyst expressions.With from TypedColumn's wildcard import. Test harness: disable ANSI mode (Spark 4 default) so the property generators keep their wrap-around/null semantics, and strip field metadata in SchemaTests. All changes are no-ops on Spark 3.x. CI: add a JDK 17 leg and pin root-spark40 to Scala 2.13 / JDK 17. dataset-spark40 passes 414/414 tests; verified end-to-end on a 2-worker standalone Spark 4.0.2 cluster (groupBy/agg, self-join, joinWith, executor closures) to confirm cross-node serialization. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adding a JDK 17 CI leg for Spark 4 made sbt-typelevel run the Generate Site job on JDK 17 (it picks the last configured Java). mdoc executes Spark code, which needs the module --add-opens flags on JDK 17. Fork the docs run, pass the flags through (extracted into sparkJava17Options, shared with the test config), and pin the forked run's working directory to the repo root so docs keep finding their relative data files (docs/iris.data). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Spark 4 port reworked FramelessInternals (internal version-compat plumbing, not intended public API): `column` is now the Expression->Column bridge and `mkDataset` derives the session from the source Dataset instead of taking a SQLContext. Both are binary-incompatible signature changes flagged by MiMa against the 3.x baselines (0.14.0/0.14.1), so exclude them. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Scala 2.12's scaladoc fails (fatally) on [[Expression]] / [[Column]] / [[ExpressionEncoder]] doc links in FramelessInternals because those Spark types aren't resolvable in the doc scope. Use backticks (code spans) instead. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The existing self-join tests only compare row counts. This collects and verifies the decoded (T, U) tuples through the colLeft/colRight disambiguation path - a regression guard for the Spark 4 ColumnNode rework, which broke that path (only count-level coverage would have missed a subtly wrong decode). Passes unchanged on Spark 3.x. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
imarios
commented
May 20, 2026
imarios
commented
May 20, 2026
imarios
commented
May 20, 2026
imarios
commented
May 20, 2026
Revert the opinionated merge of the standalone `import ...Encoder` into a braced group; add FramelessInternals as a separate plain import instead. scalafmt does not merge imports, so this stays linter-clean while staying closer to the original source. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
Author
|
Hey @pomadchin, thank you for taking the time to review. I have tried to keep this super focused on the most basic Spark 4.0 version with minimal scope creep. I followed the same shim approach we have historically taken to unblock Framelss adoption. I believe most folks today use Spark 4.x and this was preventing frameless from being an option for them. I obviously used Claude Code here but I did review the decisions closely. I am here to answer any questions. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an in-tree
frameless-*-spark40module set targeting Spark 4.0.2, addressing #787. Spark 4 is built for Scala 2.13 only (2.12 was dropped) and requires JDK 17+.The approach keeps the existing per-Spark-version source-overlay pattern (
spark-3/spark-3.4+) and adds aspark-4overlay — no external shim dependency. All version-divergent Catalyst access is isolated behindFramelessInternals.Key adaptations for Spark 4:
Expression. Bridge throughclassic.ExpressionUtils.columnand an eagerColumnNodeToExpressionConverter. (Spark'sExpressionUtils.expressionreturns a lazyColumnNodeExpressionthat isUnevaluableand exposes no children, which broke self-join disambiguation and codegen.)Dataset/SparkSessionsplit into abstract API +classicimpl. Internal helpers downcast toclassicforlogicalPlan/sessionState/sqlContext.ExpressionEncodernow takes a leadingAgnosticEncoder(SPARK-49025). We supply a metadata-onlyJavaBeanEncoderstand-in carrying the correctClassTag— the encoder field is only read forclsTagand the Option-wrapping check; serializer/deserializer/schema still come from frameless's own expressions.AnalysisExceptionis now errorClass-based;MapGroupsgets aspark-4variant.joinCrossre-encodes its result viaTypedExpressionEncoder, consistent with the other joins.catalyst.expressions.WithfromTypedColumn's wildcard import.Test harness (no-ops on Spark 3.x): disable ANSI mode (Spark 4 default) so property generators keep wrap-around/null semantics, and strip field metadata in
SchemaTests.CI: adds a JDK 17 leg and pins
root-spark40to Scala 2.13 / JDK 17; the 3.x roots stay on JDK 8.Test plan
dataset-spark40compiles against Spark 4.0.2 (Scala 2.13, JDK 17)dataset-spark40test suite: 414/414 passingcats-spark40/refined-spark40/ml-spark40test suites passinggroupBy/agg, self-join (colLeft/colRight),joinWithtuple decode, and executor-side closures all produce correct results across separate executor JVMs — confirming cross-node serializationCo-authored with Claude Code.