perf: skip unnecessary metadata fetch calls for tags when not configured by tejassp-db · Pull Request #1387 · databricks/dbt-databricks

tejassp-db · 2026-04-10T12:45:36Z

Summary

Skips fetch_tags and fetch_column_tags queries to information_schema during incremental and view materializations when the model has no tags configured
Falls back to fetching metadata when model config is unavailable (safe default)
No behavior change for models with tags configured — diffs still computed correctly

What changed

Added requires_server_metadata_for_diff() to TagsConfig and ColumnTagsConfig to signal when server metadata is needed
IncrementalTableAPI._describe_relation and ViewAPI._describe_relation now check model config before making tag fetch calls
get_relation_config accepts the model config to pass through to _describe_relation
Fixed duplicate TagsProcessor in StreamingTableConfig.config_components

Test plan

Unit tests: empty tags (skip), non-empty tags (fetch), null config fallback (fetch), both tags + column tags present, hive_metastore (skip)
Functional tests: override fetch_tags/fetch_column_tags macros to raise errors, confirming calls are actually skipped when expected

PECOBLR-2497

Skip fetch_tags and fetch_column_tags information_schema queries during incremental and view materializations when the model has no tags configured. This avoids unnecessary server roundtrips on every run for models that don't use tags, while preserving full fetch behavior when tags are present or when the model config is unavailable. PECOBLR-2497

github-actions · 2026-04-10T13:00:03Z

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
dbt/adapters/databricks
impl.py					970, 978-989, 1070, 1081, 1089, 1101, 1130, 1169, 1196, 1256, 1294, 1303-1315
dbt/adapters/databricks/relation_configs
base.py					41
column_tags.py
tags.py
Project Total

_{This report was generated by python-coverage-comment-action}

Add unit and functional coverage for skipping tag metadata queries when model config does not require them. This protects the new fetch-planning logic across incremental, view, streaming table, and materialized view test paths without changing unrelated unstaged work.

partial_parse.msgpack does not persist compiled_code, causing get_config_from_model to raise "Cannot compile model ... with no SQL query" for materialized view and streaming table change tests. Co-authored-by: Isaac

Co-authored-by: Isaac

The previous `dict and len(...) > 0` short-circuited to a dict literal when empty, breaking the declared `-> bool` return type. Co-authored-by: Isaac

sd-db

Added some more comments, overall much better. I still feel we should add a comment in all the materialisation types where we are doing this optimisation that this only works because we don't support deleting/removing tags through dbt-databricks, as that will break the optimisations done and better to encode as a comment

sd-db · 2026-05-07T10:36:38Z

-        results["information_schema.tags"] = adapter.execute_macro("fetch_tags", kwargs=kwargs)
        results["show_tblproperties"] = adapter.execute_macro("fetch_tbl_properties", kwargs=kwargs)
+
+        table_tag_config = model_config.config.get(TagsProcessor.name) if model_config else None


rename to model_table_tag_config we should differentiate between desired(model) configs and existing (relation) configs better

Config only comes from user and its obvious here.

sd-db

Reviewing tests 1/2

sd-db

Reviewing Tests 2/2

sd-db

Changes look good, thanks !

tejassp-db · 2026-05-12T07:03:23Z

/integration-test

github-actions · 2026-05-12T07:03:37Z

Integration tests dispatched for PR #1387 by @tejassp-db. Track progress in the Actions tab.

github-actions · 2026-05-12T07:55:19Z

Integration results for PR #1387 — UC cluster ❌ failure · SQL warehouse ❌ failure · All-purpose cluster ✅ success · Shard coverage ✅ success

Run details.

tejassp-db · 2026-05-13T05:51:43Z

/integration-test

github-actions · 2026-05-13T05:53:06Z

Integration tests dispatched for PR #1387 by @tejassp-db. Track progress in the Actions tab.

github-actions · 2026-05-13T06:45:37Z

Integration results for PR #1387 — UC cluster ✅ success · SQL warehouse ✅ success · All-purpose cluster ✅ success · Shard coverage ✅ success

Run details.

@sd-db

…ath (#1467) Resolves #1402 Follow-up to closed #1403 with a much smaller surface area, in response to @sd-db's earlier review. ### Description The V1 incremental merge path calls `adapter.get_relation_config()` unconditionally during every incremental run, forcing metadata diff queries (tags, column_tags, constraints, column_masks, tblproperties, describe_extended) on every model. The V2 path already honors `incremental_apply_config_changes: false` via the `process_config_changes()` macro; V1 ignored the flag. #1387 partially addressed this by auto-skipping `fetch_tags` / `fetch_column_tags` when those configs are absent. This PR is complementary, not duplicate: | | #1387 (merged) | This PR | |---|---|---| | Mechanism | Auto-detect via `requires_server_metadata_for_diff()` | Explicit user-facing flag | | Coverage | tags, column_tags (2 queries) | All metadata fetches behind `get_relation_config` | | Use case | Default behavior optimization | Deterministic opt-out: cost control, edge cases where auto-detect can't decide | **Change:** wrap the V1 `get_relation_config` call in an `if config.get('incremental_apply_config_changes', True)` block. When `false`, `_configuration_changes` stays `none` and the existing apply-block guard short-circuits the apply. Default behavior unchanged. **Security:** the V1 inline apply block never pulled `column_masks` from the changeset to begin with — so `column_masks` are not applied in V1 today, and remain not applied with this change. The flag set to `false` strictly reduces work; no new application path is introduced. The `apply_column_masks` V2-only guard from the final commit of #1403 is therefore not needed in this scoped change. **Diff size:** +6 / -3 in `incremental.sql`, plus CHANGELOG and one functional test. ### Test plan - [x] Unit tests pass locally (1081 passed) - [x] New functional test \`TestV1IncrementalApplyConfigChangesFalseSkipsTagFetch\` passes against a UC SQL warehouse on \`e2-demo-tokyo\` - [x] Verified the test FAILS on unfixed \`1.12.latest\` code (raises \"tags should not be called\") ### Checklist - [x] I have run this code in development and it appears to resolve the stated issue - [x] This PR includes tests, or tests are not required/relevant for this PR - [x] I have updated the \`CHANGELOG.md\` and added information about my change to the new \"dbt-databricks 1.12.1 (TBD)\" section. Co-authored-by: Shubham Dhal <shubham.dhal@databricks.com>

tejassp-db self-assigned this Apr 10, 2026

tejassp-db requested review from benc-db, jprakash-db and sd-db as code owners April 10, 2026 12:45

tejassp-db had a problem deploying to azure-prod April 10, 2026 12:45 — with GitHub Actions Failure

tejassp-db temporarily deployed to azure-prod April 10, 2026 12:45 — with GitHub Actions Inactive

tejassp-db temporarily deployed to azure-prod April 13, 2026 10:22 — with GitHub Actions Inactive

tejassp-db had a problem deploying to azure-prod April 13, 2026 10:22 — with GitHub Actions Failure

Merge branch 'main' into PECOBLR-2497/skip-unused-metadata-calls

815ff55

tejassp-db temporarily deployed to azure-prod April 15, 2026 05:25 — with GitHub Actions Inactive

tejassp-db had a problem deploying to azure-prod April 15, 2026 05:25 — with GitHub Actions Failure

tejassp-db temporarily deployed to azure-prod April 15, 2026 05:25 — with GitHub Actions Inactive

Merge branch 'main' into PECOBLR-2497/skip-unused-metadata-calls

7839417

tejassp-db temporarily deployed to azure-prod April 17, 2026 04:10 — with GitHub Actions Inactive

tejassp-db had a problem deploying to azure-prod April 17, 2026 04:10 — with GitHub Actions Failure

tejassp-db temporarily deployed to azure-prod April 17, 2026 04:10 — with GitHub Actions Inactive

tejassp-db and others added 9 commits April 27, 2026 10:28

Merge branch 'main' into PECOBLR-2497/skip-unused-metadata-calls

91e0dea

fix: backfill compiled_code from raw_code in MV/ST test helper

a84e0f0

partial_parse.msgpack does not persist compiled_code, causing get_config_from_model to raise "Cannot compile model ... with no SQL query" for materialized view and streaming table change tests. Co-authored-by: Isaac

Merge branch 'main' into PECOBLR-2497/skip-unused-metadata-calls

51a495e

chore: ruff format pass

0ed403b

Co-authored-by: Isaac

fix: return bool from requires_server_metadata_for_diff

f4399cd

The previous `dict and len(...) > 0` short-circuited to a dict literal when empty, breaking the declared `-> bool` return type. Co-authored-by: Isaac

Merge branch 'main' into PECOBLR-2497/skip-unused-metadata-calls

e6adab7

Merge branch 'main' into PECOBLR-2497/skip-unused-metadata-calls

ba26eb1

Fix failing tests.

f985dc8

Fix name shadowing

616239f

sd-db reviewed May 7, 2026

View reviewed changes

sd-db self-requested a review May 8, 2026 08:31

tejassp-db added 2 commits May 8, 2026 15:49

Add test for cascading table tags config

c727e76

Clarify backward compatible behavior with comments

8b57e75

sd-db reviewed May 8, 2026

View reviewed changes

Comment thread tests/functional/adapter/views/test_view_metadata_fetch_skips.py Outdated

Comment thread tests/functional/adapter/helpers.py Outdated

Comment thread tests/unit/relation_configs/test_materialized_view_config.py Outdated

sd-db reviewed May 8, 2026

View reviewed changes

Comment thread tests/unit/test_adapter.py

Reformat code

637d361

tejassp-db changed the base branch from main to 1.12.latest May 11, 2026 06:48

tejassp-db added 6 commits May 11, 2026 12:37

Merge branch '1.12.latest' into PECOBLR-2497/skip-unused-metadata-calls

f29ddeb

Add conditional tag fetch in MetricView

202249b

Refactor tests

4399e92

Refactor tests

c352241

MV fetch tags tests

637e84a

Refactor tests

aef43ea

sd-db approved these changes May 12, 2026

View reviewed changes

Update CHANGELOG

63542cc

Fix arguments to macro calls

c2d4999

tejassp-db merged commit 578375a into 1.12.latest May 13, 2026
7 checks passed

tejassp-db deleted the PECOBLR-2497/skip-unused-metadata-calls branch May 13, 2026 07:01

moomindani mentioned this pull request May 19, 2026

perf: honor incremental_apply_config_changes flag in V1 incremental path #1467

Merged

6 tasks

sd-db mentioned this pull request May 22, 2026

dbt queries the information_schema too much leading to throttling and failures #1466

Closed

This was referenced Aug 21, 2026

fix: align Databricks materialized view tag configuration dbt-labs/dbt-core#15712

Draft

[Databricks] Port get_relation_config metadata-fetch optimization to Fusion dbt-labs/dbt-core#16057

Open

Uh oh!

Conversation

tejassp-db commented Apr 10, 2026

Uh oh!

github-actions Bot commented Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Coverage report

Uh oh!

sd-db left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sd-db May 7, 2026

Choose a reason for hiding this comment

Uh oh!

tejassp-db May 12, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sd-db left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sd-db left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sd-db left a comment

Choose a reason for hiding this comment

Uh oh!

tejassp-db commented May 12, 2026

Uh oh!

github-actions Bot commented May 12, 2026

Uh oh!

github-actions Bot commented May 12, 2026

Uh oh!

tejassp-db commented May 13, 2026

Uh oh!

github-actions Bot commented May 13, 2026

Uh oh!

github-actions Bot commented May 13, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented Apr 10, 2026 •

edited

Loading