Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
- Apply column-level `databricks_tags` for incremental models on the V1 materialization path (`use_materialization_v2: false`, the default). They were silently dropped at create and on subsequent tag changes; the V1 incremental materialization now applies them, matching the `table` materialization and the V2 path. ([#1520](https://github.com/databricks/dbt-databricks/pull/1520) closes [#1307](https://github.com/databricks/dbt-databricks/issues/1307))
- Raise a `DbtRuntimeError` when a Python model job run terminates with a non-success `result_state` (e.g. `FAILED`/`TIMEDOUT`) instead of returning silently ([#1477](https://github.com/databricks/dbt-databricks/pull/1477))
- Fix PK/FK constraints declaring an `expression` (e.g. `RELY`) being dropped and re-added on every incremental run. The `expression` isn't readable from `information_schema`, so reconciliation never converged and issued `DROP CONSTRAINT ... CASCADE` each run — silently dropping dependent FKs, and erroring with `INTERNAL_ERROR` on newer Unity Catalog. PK/FK are now compared on `(name, columns)`. **Regression:** changing the `expression` on an existing PK/FK (`RELY`↔`NORELY`, or an expression-form FK's target) is no longer applied on incremental runs — use `--full-refresh`. ([#1552](https://github.com/databricks/dbt-databricks/pull/1552) closes [#1513](https://github.com/databricks/dbt-databricks/issues/1513))
- Honor `incremental_apply_config_changes` in the V1 incremental merge path, allowing users to skip metadata diff queries (tags, column_tags, constraints, column_masks, tblproperties, describe_extended) when set to `false`. Matches the existing V2 behavior. ([1467](https://github.com/databricks/dbt-databricks/pull/1467) partially solves [#1402](https://github.com/databricks/dbt-databricks/issues/1402))

### Under the Hood

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -141,9 +141,12 @@
{{ set_overwrite_mode('DYNAMIC') }}
{%- endif -%}
{#-- Relation must be merged --#}
{%- set model_config = adapter.get_config_from_model(config.model) -%}
{%- set _existing_config = adapter.get_relation_config(existing_relation, model_config) -%}
{%- set _configuration_changes = model_config.get_changeset(_existing_config) -%}
{%- set _configuration_changes = none -%}
{%- if config.get('incremental_apply_config_changes', True) | as_bool -%}
{%- set model_config = adapter.get_config_from_model(config.model) -%}
{%- set _existing_config = adapter.get_relation_config(existing_relation, model_config) -%}
{%- set _configuration_changes = model_config.get_changeset(_existing_config) -%}
{%- endif -%}
{%- call statement('create_temp_relation', language=language) -%}
{{ create_table_as(True, temp_relation, compiled_code, language) }}
{%- endcall -%}
Expand Down
10 changes: 10 additions & 0 deletions tests/functional/adapter/incremental/fixtures.py
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,16 @@
select cast(1 as bigint) as id
"""

metadata_fetch_incremental_skip_config_changes_sql = """
{{ config(
materialized = 'incremental',
unique_key = 'id',
incremental_apply_config_changes = false,
) }}

select cast(1 as bigint) as id
"""

metadata_fetch_no_tags_schema = """
version: 2

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
import pytest
from dbt.tests import util

from tests.functional.adapter.fixtures import (
MaterializationV1Mixin,
fail_if_tag_and_column_tag_fetch_called_macros,
)
from tests.functional.adapter.incremental import fixtures


@pytest.mark.skip_profile("databricks_cluster")
class TestV1IncrementalApplyConfigChangesFalseSkipsTagFetch(MaterializationV1Mixin):
"""When `incremental_apply_config_changes` is false, the V1 incremental merge path
must skip the get_relation_config metadata fetches even if the model declares tags.
"""

@pytest.fixture(scope="class")
def models(self):
return {
"metadata_fetch_incremental.sql": (
fixtures.metadata_fetch_incremental_skip_config_changes_sql
),
"schema.yml": fixtures.metadata_fetch_table_tags_schema,
}

@pytest.fixture(scope="class")
def macros(self):
return {"fail_if_tag_fetch_called.sql": fail_if_tag_and_column_tag_fetch_called_macros}

def test_v1_incremental_skips_metadata_fetch_when_flag_false(self, project):
# First run creates the table; second run exercises the existing-relation merge path
# where get_relation_config would normally fire metadata queries.
# Tags are declared on the model, so without the flag the run would call fetch_tags
# and fail. The flag must bypass the entire get_relation_config call.
util.run_dbt(["run"])
util.run_dbt(["run"])
Loading