feat: add `skip_optimize` model config to opt out of post-materialization OPTIMIZE by moomindani · Pull Request #1485 · databricks/dbt-databricks

moomindani · 2026-05-27T08:18:06Z

Resolves #703

Description

Adds a skip_optimize model config that lets users opt out of the post-materialization OPTIMIZE call without removing zorder / liquid_clustered_by / auto_liquid_cluster from the table definition.

Motivation: the existing opt-out today is the run-wide DATABRICKS_SKIP_OPTIMIZE var, which forces an all-or-nothing decision for the entire invocation. Several users in #703 asked for a config-level opt-out so they can:

delegate OPTIMIZE to Predictive Optimization while keeping auto_liquid_cluster=true on the table
skip OPTIMIZE only for specific high-churn models and let it run for the rest
schedule OPTIMIZE out of band (workflow / job) instead of on the dbt critical path

Behavior:

New model config skip_optimize (bool, default false)
When truthy, databricks__optimize short-circuits to a no-op even if zorder / liquid_clustered_by / auto_liquid_cluster is set on the model — the clustering declaration remains in the table DDL, only the OPTIMIZE SQL emission is suppressed

Inherits via standard dbt config resolution: project → folder → model (more specific wins). Example:

# dbt_project.yml
models:
  my_project:
    +skip_optimize: true
    high_read_models:
      +skip_optimize: false

DATABRICKS_SKIP_OPTIMIZE var is unchanged (still skips run-wide)

Docs follow-up

User-facing config reference lives in dbt-labs/docs.getdbt.com (databricks-configs.md). A companion docs PR will be opened there to document skip_optimize alongside the existing zorder / liquid_clustered_by / auto_liquid_cluster entries.

Checklist

I have run this code in development and it appears to resolve the stated issue
This PR includes tests, or tests are not required/relevant for this PR
I have updated the CHANGELOG.md and added information about my change to the "dbt-databricks next" section.

…tion OPTIMIZE Adds a `skip_optimize` model config that lets users opt out of the post-materialization OPTIMIZE call without removing zorder/liquid_clustered_by/auto_liquid_cluster from the table definition. Complements the existing run-wide DATABRICKS_SKIP_OPTIMIZE var by allowing project-, folder-, or model-level opt-out via standard dbt config inheritance. Resolves databricks#703 Co-authored-by: Isaac

sd-db

Overall change looks good, added a minor comment on hardening.

We would also need to make a docs PR for the new config. Also worth to note in the same on how this config interacts with DATABRICKS_SKIP_OPTIMIZE (DATABRICKS_SKIP_OPTIMIZE cannot be overridden per-model with skip_optimize: false)

github-actions · 2026-06-15T10:22:41Z

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
dbt/adapters/databricks
impl.py
Project Total

_{This report was generated by python-coverage-comment-action}

…hiness Addresses review feedback from sd-db: using `| as_bool` makes the check robust to string values like "true"/"false" that Jinja2 would treat as truthy (non-empty strings). Also registers the as_bool filter in the macro test harness so tests remain runnable without a full dbt env. Co-authored-by: Isaac

sd-db · 2026-06-18T06:22:52Z

/integration-test

github-actions · 2026-06-18T06:23:57Z

Integration tests dispatched for PR #1485 by @sd-db. Track progress in the Actions tab.

github-actions · 2026-06-18T07:15:42Z

Integration results for PR #1485 — UC cluster ✅ success · SQL warehouse ✅ success · All-purpose cluster ✅ success · Shard coverage ✅ success

Run details.

moomindani · 2026-06-25T00:43:13Z

Thanks for the review! Opened the docs PR for the new config: dbt-labs/docs.getdbt.com#9496

It documents skip_optimize on the Databricks configurations page and notes the interaction with DATABRICKS_SKIP_OPTIMIZE — i.e. the run-wide variable takes precedence and OPTIMIZE cannot be re-enabled per-model with skip_optimize: false once it's set.

…ring tests (#1573) ## Description Adds functional-test coverage for the `skip_optimize` model config (#1485) and strengthens two existing liquid-clustering tests to assert server-observable state instead of log substrings. Test-only — no runtime change. ### `skip_optimize` functional coverage (#1485) `skip_optimize=true` short-circuits `databricks__optimize` to a no-op while the clustering declaration stays on the table. This was previously only unit-tested (`tests/unit/macros/relations/test_optimize_macros.py`). New functional tests assert the end-to-end effect on a live warehouse: - **`TestTableSkipOptimizeEffect`** — on a `table` model, `liquid_clustered_by` columns are present AND no `OPTIMIZE` appears in `DESCRIBE HISTORY`. - **`TestIncrementalSkipOptimizeEffect`** — same on an `incremental` model (the high-churn use case from #703), exercising the distinct incremental `optimize()` call site. The `skip_optimize=false`/unset control (OPTIMIZE still runs) is already covered by the existing `TestTableLiquidClusteringEffect`. ### Test strengthening (log substrings → server state) - **`TestLiquidClustering`**: replaced `assert "optimize" in logs` with a `DESCRIBE HISTORY` check that an `OPTIMIZE` operation actually ran. - **`TestAutoLiquidClustering`**: it previously used the non-auto fixture, making it a silent duplicate of `TestLiquidClustering` that never exercised auto clustering. Repointed to the auto fixture; it now asserts `clusterByAuto=true` via `SHOW TBLPROPERTIES`, adding auto-clustering-on-incremental coverage. ### Testing All pass on `databricks_uc_sql_endpoint`: - `tests/functional/adapter/liquid_clustering/` (both files) + `tests/functional/adapter/zorder/` — **11 passed**. - `ruff` / `ruff format` / `mypy` clean on the changed files. ### Checklist - [x] I have run this code in development and it appears to resolve the stated issue - [x] This PR includes tests - [x] No `CHANGELOG.md` entry — test-only change with no runtime/user-facing impact

moomindani requested review from jprakash-db and sd-db as code owners May 27, 2026 08:18

moomindani force-pushed the feat/skip-optimize-config branch from e720276 to 55a5713 Compare June 10, 2026 23:31

moomindani force-pushed the feat/skip-optimize-config branch from 55a5713 to 1f3e235 Compare June 13, 2026 01:56

sd-db reviewed Jun 15, 2026

View reviewed changes

Comment thread dbt/include/databricks/macros/relations/optimize.sql Outdated

Merge branch 'main' into feat/skip-optimize-config

98943e6

sd-db approved these changes Jun 15, 2026

View reviewed changes

moomindani and others added 2 commits June 17, 2026 09:18

Merge branch 'main' into feat/skip-optimize-config

004e9cb

sd-db merged commit 9212bba into databricks:main Jun 19, 2026
9 checks passed

moomindani mentioned this pull request Jun 25, 2026

Document skip_optimize model config for dbt-databricks dbt-labs/docs.getdbt.com#9496

Merged

2 tasks

sd-db mentioned this pull request Jul 1, 2026

test: functional coverage for skip_optimize; strengthen liquid clustering tests #1573

Merged

3 tasks

sd-db mentioned this pull request Aug 23, 2026

[Databricks] Support skip_optimize model configuration in Fusion dbt-labs/dbt-core#16055

Open

Navigation Menu

Uh oh!

feat: add `skip_optimize` model config to opt out of post-materialization OPTIMIZE - #1485

feat: add `skip_optimize` model config to opt out of post-materialization OPTIMIZE#1485
sd-db merged 4 commits into
databricks:mainfrom
moomindani:feat/skip-optimize-config

moomindani commented May 27, 2026 •

edited

Loading

Uh oh!

sd-db left a comment

Uh oh!

Uh oh!

github-actions Bot commented Jun 15, 2026 •

edited

Loading

Uh oh!

sd-db commented Jun 18, 2026

Uh oh!

github-actions Bot commented Jun 18, 2026

Uh oh!

github-actions Bot commented Jun 18, 2026

Uh oh!

Uh oh!

moomindani commented Jun 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

moomindani commented May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Docs follow-up

Checklist

Uh oh!

sd-db left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions Bot commented Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Coverage report

Uh oh!

sd-db commented Jun 18, 2026

Uh oh!

github-actions Bot commented Jun 18, 2026

Uh oh!

github-actions Bot commented Jun 18, 2026

Uh oh!

Uh oh!

moomindani commented Jun 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

moomindani commented May 27, 2026 •

edited

Loading

github-actions Bot commented Jun 15, 2026 •

edited

Loading