Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@

### Under the Hood

- Document serverless environment configuration for Python models (thanks @TangoEnSkai!) ([#1649](https://github.com/databricks/dbt-databricks/pull/1649) resolves [#1055](https://github.com/databricks/dbt-databricks/issues/1055))

- Raise the `pytest-rerunfailures` lower bound to `>=16.2` and remove the `SchemaNameVarMixin` workaround so min-deps CI no longer pins 14.0, which leaked class-scoped dbt test fixtures across reruns (test-only, no runtime impact) ([#1618](https://github.com/databricks/dbt-databricks/pull/1618))
- Bump `databricks-sql-connector` ceiling to `<4.4.1` and pin to 4.4.0, which requires `thrift>=0.24.0`; resolves CVE-2026-48586 (data amplification DoS), CVE-2026-41603 (TLS cert hostname bypass), and CVE-2026-43868 (memory allocation) ([#1623](https://github.com/databricks/dbt-databricks/pull/1623) resolves [#1622](https://github.com/databricks/dbt-databricks/issues/1622))
- Reorganize `docs/` into `docs/flow/` and `docs/guides/`, and sync materialization flow diagrams with the current macros ([#1627](https://github.com/databricks/dbt-databricks/pull/1627))
Expand Down
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,7 @@ These following quick starts will get you up and running with the `dbt-databrick
- [Set up your dbt project with Databricks](https://docs.getdbt.com/guides/set-up-your-databricks-dbt-project)
- Using dbt Cloud with Databricks ([Azure](https://docs.microsoft.com/en-us/azure/databricks/integrations/prep/dbt-cloud) | [AWS](https://docs.databricks.com/integrations/prep/dbt-cloud.html))
- [Submitting Python models as Databricks Workflows](https://github.com/databricks/dbt-databricks/blob/main/docs/guides/workflow-job-submission.md)
- [Configuring serverless environments for Python models](https://github.com/databricks/dbt-databricks/blob/main/docs/guides/workflow-job-submission.md#serverless-environments)
- [Running a dbt project as a Databricks job](https://github.com/databricks/dbt-databricks/blob/main/docs/guides/databricks-jobs.md)
- [Using Unity Catalog with dbt-databricks](https://github.com/databricks/dbt-databricks/blob/main/docs/guides/uc.md)
- [Continuous integration in dbt](https://docs.getdbt.com/docs/deploy/continuous-integration)
Expand Down
2 changes: 1 addition & 1 deletion docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,6 @@ drift; treat them as starting points, not authoritative reference.
- **[guides/databricks-jobs.md](guides/databricks-jobs.md)** — running a dbt project as a Databricks
job.
- **[guides/workflow-job-submission.md](guides/workflow-job-submission.md)** — submitting Python
models as long-lived Databricks Workflows.
models as long-lived Databricks Workflows, including serverless environment configuration.
- **[guides/databricks-copy-into-macro-aws.md](guides/databricks-copy-into-macro-aws.md)** — loading
S3 data into Delta with the `databricks_copy_into` macro.
49 changes: 49 additions & 0 deletions docs/guides/workflow-job-submission.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,9 @@ method, but allow for additional configuration.

Some of that configuration can also be used for `job_cluster` models.

For one-time serverless runs instead of a long-lived workflow, see
[Serverless environments](#serverless-environments).

```python
# my_model.py
import pyspark.sql.types as T
Expand Down Expand Up @@ -110,6 +113,52 @@ dbt will generate a name based on the catalog, schema, and model identifier.
- Similarly, you can define a reusable job cluster for the workflow and tell the task to use that
- If none of those are in the configuration, the task cluster will be serverless

#### Serverless environments

Python models submitted with `serverless_cluster` can install dependencies in a
[Databricks serverless environment](https://docs.databricks.com/aws/en/compute/serverless/dependencies).
Set both `environment_key` and `environment_dependencies` in the model's YAML configuration:

```yaml
version: 2

models:
- name: my_python_model
config:
submission_method: serverless_cluster
environment_key: dbt_env
environment_dependencies:
- pandas==2.2.3
- /Workspace/Shared/libraries/my_package-1.0.0-py3-none-any.whl
```

`environment_key` assigns the environment to the Python model task. When
`environment_dependencies` is also set, dbt-databricks creates a serverless environment using
environment version `4` and installs each listed PyPI package or workspace file.

To control other environment settings, define the full Jobs API environment in
`python_job_config.environments`. The key used by the task must match the key in that list:

```yaml
version: 2

models:
- name: my_python_model
config:
submission_method: serverless_cluster
environment_key: dbt_env
python_job_config:
environments:
- environment_key: dbt_env
spec:
environment_version: "4"
dependencies:
- pandas==2.2.3
```

When `python_job_config.environments` contains an environment, it takes precedence over the
environment that dbt-databricks would generate from `environment_dependencies`.

```yaml
# Reusable job cluster config example

Expand Down