feat: expose dbt_task job_id, job_run_id, task_run_id in adapter_response - #1451
Merged
Conversation
…onse Adds three optional fields to the dbt-databricks adapter response so that dbt runs executed by a Databricks Jobs `dbt_task` can be correlated with their originating workflow execution via run_results.json. Source: parse the DBT_DATABRICKS_HTTP_SESSION_HEADERS env var set by the Jobs runtime on the dbt CLI subprocess. The header JSON carries the IDs directly (X-Databricks-Dbsql-Job-Id, X-Databricks-Dbsql-Run-Id) plus a base64-encoded query source containing job_run_id. Fields join cleanly to system tables: - job_id -> system.lakeflow.jobs.job_id - job_run_id -> system.lakeflow.job_run_timeline.run_id - task_run_id -> system.lakeflow.job_task_run_timeline.run_id All three are None when dbt runs outside a dbt_task (local CLI, notebook task, other task types); the new response type stays a backward-compatible subclass of AdapterResponse. Resolves #722 Refs #1376
3 tasks
Coverage report
Click to see where and how coverage changed
This report was generated by python-coverage-comment-action |
||||||||||||||||||||||||||||||
sd-db
changed the title
dbt_task job_id, job_run_id, task_run_id in adapter_response
Collaborator
Author
|
/integration-test |
|
Integration tests dispatched for PR #1451 by @sd-db. Track progress in the Actions tab. |
- handle.py: trim _DBT_TASK_HEADERS_ENV comment to one line; tighten DatabricksAdapterResponse docstring (drop per-field system-table mapping). - test_handle.py: drop assertion on internal _message field; remove three low-value tests (no_query_id duplicate, is_adapter_response_subclass, str_uses_base_message).
json.loads on the DBT_DATABRICKS_HTTP_SESSION_HEADERS env var (or on the base64-decoded X-Databricks-Sql-Query-Source value) accepts any valid JSON — including null, [], 123, "foo", true — and the previous code would raise AttributeError when calling .get() on those non-dict results. Add isinstance(..., dict) guards at both levels and parametrized regression tests covering each non-object shape.
|
Integration results for PR #1451 — UC cluster ✅ success · SQL warehouse ✅ success · All-purpose cluster ✅ success · Shard coverage ✅ success |
jprakash-db
approved these changes
May 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Exposes
job_id,job_run_id, andtask_run_idonadapter_responsefor dbt runs executed by a Databricks Jobsdbt_task, so users can joinrun_results.jsontosystem.lakeflow.{jobs, job_run_timeline, job_task_run_timeline}.Source: parses
DBT_DATABRICKS_HTTP_SESSION_HEADERS, the env var the Jobs runtime sets on the dbt CLI subprocess. Fields areNoneoutside adbt_task(local CLI, notebook task, etc.) — fully backward-compatible.Resolves #722
Refs #1376
Verification
dbt run-operation dump_ids(a macro that just log()s the raw DBT_DATABRICKS_HTTP_SESSION_HEADERS); the third randbt run. All three tasks reported the same job_run_id and three different task_run_ids, each matching that task's own run_id in the jobs API response as well as the values in system tables.