perf: allow V1 incremental path to skip config change metadata queries

[moomindani]

Resolves #1402

Description

The V1 incremental materialization path calls adapter.get_relation_config() unconditionally during every incremental merge run. This triggers 8 sequential metadata queries against information_schema and system tables — even when none of the related features (tags, constraints, column masks) are in use.

The V2 path already uses process_config_changes(), which respects the incremental_apply_config_changes config flag. This PR replaces the V1 inline code with the same macro, bringing V1 in line with V2.

Change:

• Remove inline config change detection (3 lines) and application (19 lines) from V1 incremental path
• Replace with single {{ process_config_changes(target_relation) }} call

Queries eliminated when incremental_apply_config_changes: false:

1. SELECT ... FROM system.information_schema.table_tags
2. SELECT ... FROM system.information_schema.column_tags
3. SELECT ... FROM information_schema.columns (NOT NULL constraints)
4. SELECT ... FROM information_schema.key_column_usage (PRIMARY KEY)
5. SELECT ... FROM information_schema.key_column_usage (FOREIGN KEY)
6. SELECT ... FROM system.information_schema.column_masks
7. SHOW TBLPROPERTIES
8. DESCRIBE TABLE EXTENDED

Measured results (Serverless SQL Warehouse, incremental merge model with auto_liquid_cluster: true):

	Default	incremental_apply_config_changes: false
Queries (model execution)	13	5
Model execution time	13.89s	6.55s
get_relation_config overhead	3.70s	0s
Speedup	—	2.12x

Test assets used for benchmarking

dbt_project.yml:

name: 'perf_test'
version: '1.0.0'
profile: 'perf_test'

model-paths: ["models"]
macro-paths: ["macros"]
test-paths: ["tests"]
seed-paths: ["seeds"]

target-path: "target"
clean-targets:
  - "target"
  - "dbt_packages"

flags:
  use_materialization_v2: false

models/incremental_merge_test.sql (default run):

{{
  config(
    materialized='incremental',
    incremental_strategy='merge',
    unique_key='id',
    auto_liquid_cluster=true
  )
}}

SELECT
  id, name, category, amount, updated_at
FROM (
  VALUES
    (1, 'Alice', 'A', 100.0, current_timestamp()),
    (2, 'Bob', 'B', 200.0, current_timestamp()),
    (3, 'Charlie', 'A', 150.0, current_timestamp()),
    (4, 'Diana', 'C', 300.0, current_timestamp()),
    (5, 'Eve', 'B', 250.0, current_timestamp())
) AS t(id, name, category, amount, updated_at)

models/incremental_merge_test.sql (skip run — only config diff):

{{
  config(
    materialized='incremental',
    incremental_strategy='merge',
    unique_key='id',
    auto_liquid_cluster=true,
    incremental_apply_config_changes=false
  )
}}

Steps: dbt run (initial table creation) → dbt run (incremental merge, default) → add incremental_apply_config_changes=false → dbt run (incremental merge, skip). Compare dbt.log from the last two runs.

Query-level breakdown (default run)

  1. 0.470s  CREATE TEMPORARY VIEW (temp relation)
  2. 0.290s  DESCRIBE TABLE EXTENDED (target)
  3. 0.180s  DESCRIBE TABLE EXTENDED (temp)
  4. 6.200s  MERGE INTO (model SQL)
  5. 0.360s  fetch_tags ← get_relation_config
  6. 0.350s  fetch_column_tags ← get_relation_config
  7. 0.480s  fetch_non_null_constraints ← get_relation_config
  8. 0.790s  fetch_primary_key_constraints ← get_relation_config
  9. 1.070s  fetch_foreign_key_constraints ← get_relation_config
 10. 0.320s  fetch_column_masks ← get_relation_config
 11. 0.330s  SHOW TBLPROPERTIES ← get_relation_config
 12. 0.350s  DESCRIBE TABLE EXTENDED ← get_relation_config
 13. 2.700s  OPTIMIZE

Query-level breakdown (skip run)

  1. 0.420s  CREATE TEMPORARY VIEW (temp relation)
  2. 0.260s  DESCRIBE TABLE EXTENDED (target)
  3. 0.160s  DESCRIBE TABLE EXTENDED (temp)
  4. 3.540s  MERGE INTO (model SQL)
  5. 2.170s  OPTIMIZE

Functional parity: apply_config_changeset (used by process_config_changes) handles all config types that the V1 inline code handled (tags, tblproperties, liquid_clustering, constraints) plus additional types (column_comments, column_tags, column_masks).

When incremental_apply_config_changes is true (default), behavior is unchanged — all 8 queries run and config changes are applied.

Checklist

• I have run this code in development and it appears to resolve the stated issue
• This PR includes tests, or tests are not required/relevant for this PR
• I have updated the CHANGELOG.md and added information about my change to the "dbt-databricks next" section.

[tejassp-db]

Hi @moomindani

Thanks for the PR.

Now that this code path uses process_config_changes, it adds column_tags, column_masks, and comment to the V1 incremental path as well. Can you please add V1 functional tests to validate these changes?

Also please add a functional test for behaviour when incremental_apply_config_changes is set to false and that the metadata are not fetched.

[tejassp-db]

Doesn't process_config_changes take care of persist_docs as well?

[moomindani]

Thanks for the review! Addressed both points in 2882e9c.

Re: persist_docs redundancy (line 174)

Yes — apply_config_changeset handles both comment and column_comments changes, making the separate persist_docs call redundant. Removed to align with V2 behavior.

Re: functional tests

Added three tests:

• TestV1IncrementalColumnTags: Validates column tag changes are applied via the V1 path
• TestV1IncrementalColumnMasks: Validates column mask changes are applied via the V1 path
• TestV1IncrementalSkipConfigChanges: Overrides all 6 metadata fetch macros (fetch_tags, fetch_column_tags, fetch_non_null_constraint_columns, fetch_primary_key_constraints, fetch_foreign_key_constraints, fetch_column_masks) to raise errors — verifies the second incremental run succeeds without calling any of them when incremental_apply_config_changes: false

[tejassp-db]

Thanks @moomindani. We will look to merge this.

[tejassp-db]

Hi @moomindani
We got more context around this from @benc-db "The primary reason why column masks were not adopted for V1 was that they couldn't be applied to the final table prior to the data arriving.  In other words, there would be a window in which the data would be in the table unmasked.  This could be a security/privacy vulnerability."

Keeping this in mind, we will not merge this PR. Sorry for the delayed response.

[moomindani]

Thanks for the context from @benc-db about the column masks security concern.

Addressed in c66209d:

• Guarded apply_column_masks to V2 only in apply_config_changeset (alter.sql): added adapter.behavior.use_materialization_v2 check so column masks are never applied in V1, where CTAS writes data before masks can be set.
• Added TestV1IncrementalColumnMasksNotApplied: overrides apply_column_masks to raise an error, verifying it is never called during V1 incremental runs even when column masks are configured.
• Kept TestV1IncrementalSkipConfigChanges: validates incremental_apply_config_changes: false skips all metadata fetch queries.

[sd-db]

Hi @moomindani can you look to rebase this on 1.12.latest ? The change is big enough (large surface area of impact) that I feel targeting a feature release would be best

[moomindani]

Thanks @sd-db, makes sense. I'll rebase onto 1.12.latest and update the base branch.

[moomindani]

@sd-db Rebased onto 1.12.latest and updated the base branch. Verified:

• Unit tests: 835 passed
• Functional tests added in this PR (V1 incremental): 2 passed against a UC SQL warehouse

Ready for re-review when you have time.