fix(databricks): use UC-compatible access mode when job paths resolve to Volumes - #227
Open
jtdossett wants to merge 1 commit into
Open
fix(databricks): use UC-compatible access mode when job paths resolve to Volumes#227jtdossett wants to merge 1 commit into
jtdossett wants to merge 1 commit into
Conversation
… to Volumes _build_job_spec only set data_security_mode=SINGLE_USER when the opt-in cluster_logs_volume was provided. Any other new-cluster submission whose artifacts_storage_path or python_file resolved to /Volumes/... still got Databricks' default "No Isolation Shared" cluster, which cannot read Unity Catalog Volumes regardless of grants. Detect the need for UC-compatible access mode from the shape of the resolved paths instead: add _is_uc_volume_path (mirrors the existing check in artifact_store.py and bootstrap.py), resolve python_file before the cluster_spec is built so it can be inspected, and gate data_security_mode / the spark.databricks.cluster.profile strip on needs_uc_mode rather than only the cluster_logs_volume branch. cluster_log_conf (log delivery) stays a separate, still-opt-in decision. Fixes gh#217
There was a problem hiding this comment.
Pull request overview
This PR fixes Databricks job submission when the job’s resolved paths land in Unity Catalog Volumes (/Volumes/...) by ensuring new job clusters are created with a UC-compatible access mode (data_security_mode=SINGLE_USER) whenever the resolved artifacts_storage_path or python_file points into Volumes.
Changes:
- Add a
/Volumespath predicate and use it to driveSINGLE_USERselection (not just thecluster_logs_volumeopt-in). - Resolve
python_filebefore building the cluster spec so UC-mode detection can use the final resolved path; broaden popping ofspark.databricks.cluster.profilewhenever UC mode is required. - Add unit tests covering the new UC Volumes detection cases and document the fix in
CHANGELOG.md.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| packages/kindling_sdk/kindling_sdk/platform_databricks.py | Detects UC Volumes paths and sets SINGLE_USER for new clusters when resolved job paths point into /Volumes/...; resolves python_file earlier to support that detection. |
| tests/unit/test_platform_databricks_sdk_job_config.py | Adds unit tests for the Volumes-path predicate and for UC-mode selection driven by resolved artifacts_storage_path/python_file. |
| CHANGELOG.md | Adds an Unreleased entry documenting the Databricks UC Volumes access-mode fix (gh#217). |
| return "/".join([root.rstrip("/"), *cleaned_parts]) | ||
|
|
||
| @staticmethod | ||
| def _is_uc_volume_path(path: str) -> bool: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
SINGLE_USERdata security mode whenever a job'sartifacts_storage_pathor resolvedpython_filepoints into a Unity Catalog Volumes path (/Volumes/...), matching what UC Volumes access actually requires.python_fileresolution now happens beforecluster_specconstruction so the Volumes check can see the final path.spark.databricks.cluster.profileis now popped whenever UC mode is needed (previously only when a log volume was set), since the incompatibility applies wheneverSINGLE_USERis set.Test plan
poe test-unit— 2341 passed, 2 skippedCloses #217
🤖 Generated with Claude Code