feat: include job_id, run_id, and task_key in adapter_response - #1376
feat: include job_id, run_id, and task_key in adapter_response#13764444J99 wants to merge 5 commits into
Conversation
When dbt runs inside a Databricks workflow task, the runtime exposes DATABRICKS_JOB_ID, DATABRICKS_RUN_ID, and DATABRICKS_TASK_KEY as environment variables. This change reads those variables and surfaces them in the adapter_response via a new DatabricksAdapterResponse subclass, making them available in run_results.json for downstream tooling to correlate dbt runs with their originating Databricks Job executions. Outside of a Databricks Job context the fields are None and the response remains fully backward-compatible. Resolves databricks#722
|
Hi team — checking in on this PR. It adds |
Coverage report
Click to see where and how coverage changed
This report was generated by python-coverage-comment-action |
||||||||||||||||||||||||||||||
Removes verbose explanation that duplicates content already documented in _get_job_run_context's docstring.
Per review feedback from the Databricks Jobs team (Gabor Ratky, Anton Forsberg): rename the run_id field to task_run_id for consistency with the canonical TaskRunContext proto, system.lakeflow.job_task_run_timeline column semantics, and the original feature request in databricks#722. The source env var DATABRICKS_RUN_ID is set by the runtime and remains unchanged.
|
Tested this end-to-end by submitting a real The dbt task type sets I think the PR requires changes in its current state, also I had done a minor renaming of |
|
Closing for #1451, where I have made the fixes. Thx for the contribution. I will followup and look to have this merged in the next few releases. |
…r_response (#1451) ### Summary Exposes `job_id`, `job_run_id`, and `task_run_id` on `adapter_response` for dbt runs executed by a Databricks Jobs `dbt_task`, so users can join `run_results.json` to `system.lakeflow.{jobs, job_run_timeline, job_task_run_timeline}`. Source: parses `DBT_DATABRICKS_HTTP_SESSION_HEADERS`, the env var the Jobs runtime sets on the dbt CLI subprocess. Fields are `None` outside a `dbt_task` (local CLI, notebook task, etc.) — fully backward-compatible. Resolves #722 Refs #1376 ### Verification - **Unit tests** cover env-var absent, empty, valid, missing query-source, malformed JSON/base64. Full unit suite: 771 passed. - Ran **dbt_task submissions on a live workspace** to verify. - **Single-task run**: extracted the three IDs from the resulting run_results.json, then queried system.lakeflow.job_task_run_timeline and system.lakeflow.job_run_timeline for that job_id. The task table's run_id matched our task_run_id, its job_run_id matched ours, its job_id matched. The parent table's run_id matched our job_run_id. Names line up with the join columns. - **Multi-task run**: three sibling dbt_tasks in one submission, no depends_on. Two ran `dbt run-operation dump_ids` (a macro that just log()s the raw DBT_DATABRICKS_HTTP_SESSION_HEADERS); the third ran `dbt run`. All three tasks reported the same job_run_id and three different task_run_ids, each matching that task's own run_id in the jobs API response as well as the values in system tables. - **Local dbt run**: outside a dbt_task writes a run_results.json with the three fields stripped (dbt-core drops None dataclass fields) — no log pollution.
2 participants
Resolves #722
Description
When dbt runs inside a Databricks workflow task, the Databricks runtime sets
DATABRICKS_JOB_ID,DATABRICKS_RUN_ID, andDATABRICKS_TASK_KEYas environment variables. This PR reads those variables and includes them in theadapter_response, making them available inrun_results.jsonso that downstream tooling can correlate a dbt run with its originating Databricks Job execution.Implementation:
DatabricksAdapterResponse, a subclass ofAdapterResponsethat adds three optional fields:job_id,run_id, andtask_key_get_job_run_context()reads the environment variables once and returns a dict with the values (orNonewhen running outside a Databricks Job)CursorWrapper.get_response()and the fallback path inDatabricksConnectionManager.get_response()both return the new response typeNone— fully backward-compatibleExample output in
run_results.json(when running inside a Databricks Job):{ "adapter_response": { "_message": "OK", "query_id": "01f0-...", "job_id": "123456", "run_id": "789012", "task_key": "dbt_run" } }Checklist
CHANGELOG.mdand added information about my change to the "dbt-databricks next" section.