AGENTS.md - AI Agent Guide for dbt-databricks

This guide helps AI agents quickly understand and work productively with the dbt-databricks adapter codebase.

🚀 Quick Start for Agents

Project Overview

What: dbt adapter for Databricks Lakehouse platform
Based on: dbt-spark adapter with Databricks-specific enhancements
Key Features: Unity Catalog support, Delta Lake, Python models, streaming tables
Language: Python 3.10+ with Jinja2 SQL macros
Architecture: Inherits from Spark adapter, extends with Databricks-specific functionality

Essential Files to Understand

dbt/adapters/databricks/
├── impl.py                    # Main adapter implementation (DatabricksAdapter class)
├── connections.py             # Connection management and SQL execution
├── credentials.py             # Authentication (token, OAuth, Azure AD)
├── relation.py               # Databricks-specific relation handling
├── dbr_capabilities.py       # DBR version capability system
├── python_models/            # Python model execution on clusters
├── relation_configs/         # Table/view configuration management
└── catalogs/                 # Unity Catalog vs Hive Metastore logic

dbt/include/databricks/macros/ # Jinja2 SQL templates
├── adapters/                 # Core adapter macros
├── materializations/         # Model materialization strategies
├── relations/                # Table/view creation and management
└── utils/                    # Utility macros

Materialization Execution Flow

Before changing a materialization, read its flow doc — docs/flow/ maps how each materialization type executes (table, view, incremental, seed, snapshot, streaming table, materialized view) as diagrams, plus the shared relation replace flow. Start at docs/flow/README.md, which also explains the use_materialization_v2 behavior flag that selects between the V1 (default) and V2 paths. The macros under dbt/include/databricks/macros/materializations/ are the source of truth; the flow docs mirror them.

🛠 Development Environment

Prerequisites: Python 3.10+ installed on your system

Install Hatch (recommended):

For Linux:

# Download and install standalone binary
curl -Lo hatch.tar.gz https://github.com/pypa/hatch/releases/latest/download/hatch-x86_64-unknown-linux-gnu.tar.gz
tar -xzf hatch.tar.gz
mkdir -p $HOME/bin
mv hatch $HOME/bin/hatch
chmod +x $HOME/bin/hatch
echo 'export PATH="$HOME/bin:$PATH"' >> ~/.zshrc
export PATH="$HOME/bin:$PATH"

# Create default environment (Hatch installs needed Python versions)
hatch env create

For other platforms: see https://hatch.pypa.io/latest/install/

Essential commands:

hatch run code-quality           # Format, lint, type-check
hatch run unit                   # Run unit tests
hatch run cluster-e2e-dev        # Run functional tests

# For specific tests, use pytest directly:
hatch run pytest path/to/test_file.py::TestClass::test_method -v

📖 See Development Guide for the full setup documentation 📖 See Testing Guide for the full testing documentation

🧪 Testing Strategy

Test Types & When to Use

Unit Tests (tests/unit/): Fast, isolated, no external dependencies
- Test individual functions, utility methods, SQL generation
- Mock external dependencies (database calls, API calls)
- Run with: hatch run unit
Functional Tests (tests/functional/): End-to-end with real Databricks
- Test complete dbt workflows (run, seed, test, snapshot)
- Require live Databricks workspace
- Run with: hatch run cluster-e2e-dev (or uc-cluster-e2e-dev, sqlw-e2e-dev).
Lowest-direct dependency tests (hatch run min-deps:X): Same unit + functional suites against the committed lower-bound lock
- Catches drift where wide dep ranges in pyproject.toml admit versions the code no longer supports
- Lock lives in requirements.lowest-direct.txt
- Run with: hatch run min-deps:unit, hatch run min-deps:parse, hatch run min-deps:e2e
- Details: docs/testing.md → Testing against lowest-direct dependencies

What to Assert in Each Test Type

Functional tests assert user-visible outcomes: the model materializes and the resulting rows are correct. Unit and macro tests assert implementation details like the exact SQL a macro generates or the exact text of a warning.

When there is nothing meaningful to assert in either category, the change does not need a test.

Three rules keep functional tests server-truthful and cheap (full detail in docs/testing.md → Functional Tests):

Assert server-observable state only — query data, SHOW TBLPROPERTIES, DESCRIBE DETAIL/EXTENDED/HISTORY, or information_schema. Never assert a log substring or generated-SQL text in a functional test (that's unit/macro-test domain), even where existing tests do.
Rerun-safe — a second run of the test must pass (RerunSafeMixin or unique names).
Cheapest home first — strengthen an existing test > add a case to a class > new class > new file/section; fixtures go in fixtures.py, never inlined.

Test Environments

HMS Cluster (databricks_cluster): Legacy Hive Metastore
Unity Catalog Cluster (databricks_uc_cluster): Modern UC features
SQL Warehouse (databricks_uc_sql_endpoint): Serverless compute

Writing Tests

Unit Test Example

from dbt.adapters.databricks.utils import redact_credentials

def test_redact_credentials():
    sql = "WITH (credential ('KEY' = 'SECRET_VALUE'))"
    expected = "WITH (credential ('KEY' = '[REDACTED]'))"
    assert redact_credentials(sql) == expected

Macro Test Example

from tests.unit.macros.base import MacroTestBase

class TestCreateTable(MacroTestBase):
    @pytest.fixture(scope="class")
    def template_name(self) -> str:
        return "create.sql"  # File in macros/relations/table/

    @pytest.fixture(scope="class")
    def macro_folders_to_load(self) -> list:
        return ["macros", "macros/relations/table"]

    def test_create_table_sql(self, template_bundle):
        result = self.run_macro(template_bundle.template, "create_table",
                               template_bundle.relation, "select 1")
        expected = "create table `database`.`schema`.`table` as (select 1)"
        self.assert_sql_equal(result, expected)

Functional Test Example

Important: SQL models and YAML schemas should be defined in a fixtures.py file in the same directory as the test, not inline in the test class. This keeps tests clean and fixtures reusable.

fixtures.py:

my_model_sql = """
{{ config(materialized='incremental', unique_key='id') }}
select 1 as id, 'test' as name
"""

my_schema_yml = """
version: 2
models:
  - name: my_model
    columns:
      - name: id
        description: 'ID column'
"""

test_my_feature.py:

from dbt.tests import util
from tests.functional.adapter.my_feature import fixtures

class TestIncrementalModel:
    @pytest.fixture(scope="class")
    def models(self):
        return {
            "my_model.sql": fixtures.my_model_sql,
            "schema.yml": fixtures.my_schema_yml,
        }

    def test_incremental_run(self, project):
        results = util.run_dbt(["run"])
        assert len(results) == 1
        # Verify table exists and has expected data
        results = project.run_sql("select count(*) from my_model", fetch="all")
        assert results[0][0] == 1

Code Comment Discipline

Default to no comment. Keep one only when it records a non-obvious constraint, workaround, or rationale the code cannot express. State why, not what, in one concise sentence. Remove comments that narrate code, repeat tests, preserve implementation history, or duplicate PR/changelog rationale. Apply the existing changelog rules without adding explanatory prose around entries.

📝 CHANGELOG Entries

Every PR that changes runtime behavior updates CHANGELOG.md (enforced by the PR template). Add the entry under the topmost version heading (the one marked (TBD)), in the correct section: ### Features, ### Fixes, or ### Under the Hood.

Write one line, present tense, describing the user-visible effect (not the implementation):

- <summary> ([#<PR>](https://github.com/databricks/dbt-databricks/pull/<PR>))

When the PR addresses a tracked issue, link it in the same parentheses after the PR:

- <summary> ([#<PR>](.../pull/<PR>) resolves [#<ISSUE>](.../issues/<ISSUE>))

Use resolves when merging the PR should close the issue (matches the PR template's Resolves #).
Use partially resolves when the PR only partly addresses the issue; the issue stays open, so don't put an auto-close keyword like Resolves # in the PR body.
Omit the issue link when there is no tracked issue.
Older entries use closes; leave them as-is and use resolves for new ones.

Community (external) contributions credit the author with (thanks @<author>!) before the links:

- <summary> (thanks @<author>!) ([#<PR>](.../pull/<PR>) resolves [#<ISSUE>](.../issues/<ISSUE>))

Test-only PRs still get an ### Under the Hood entry noted (test-only, no runtime impact).

🏗 Architecture Deep Dive

Adapter Inheritance Chain

DatabricksAdapter (impl.py)
  ↳ SparkAdapter (from dbt-spark)
    ↳ SQLAdapter (from dbt-core)
      ↳ BaseAdapter (from dbt-core)

Key Components

DBR Capability System (`dbr_capabilities.py`)

Purpose: Centralized management of DBR version-dependent features
Key Features:
- Per-compute caching (different clusters can have different capabilities)
- Named capabilities instead of magic version numbers
- Automatic detection of DBR version and SQL warehouse environments
Supported Capabilities: see docs/dbr-capability-system.md for the canonical list (each capability's minimum DBR version and SQL-warehouse support). The authoritative source is the DBRCapability enum and CAPABILITY_SPECS in dbr_capabilities.py; the doc mirrors them in human-readable form.

Usage in Code:

# In Python code
if adapter.has_capability(DBRCapability.ICEBERG):
    # Use Iceberg features

# In Jinja macros
{% if adapter.has_dbr_capability('comment_on_column') %}
    COMMENT ON COLUMN ...
{% else %}
    ALTER TABLE ... ALTER COLUMN ...
{% endif %}

{% if adapter.has_dbr_capability('insert_by_name') %}
    INSERT INTO table BY NAME SELECT ...
{% else %}
    INSERT INTO table SELECT ... -- positional
{% endif %}

Adding New Capabilities:
1. Add to DBRCapability enum
2. Add CapabilitySpec with version requirements
3. Use has_capability() or require_capability() in code
4. Update docs/dbr-capability-system.md (the human-readable mirror of the enum/specs) whenever you add, remove, or change a capability or its version/SQL-warehouse support
Important: Each compute resource (identified by http_path) maintains its own capability cache

Connection Management (`connections.py`)

Extends Spark connection manager for Databricks
Manages connection lifecycle and query execution
Handles query comments and context tracking
Integrates with credentials.py for authentication and handle.py for cursor operations

Authentication & Credentials (`credentials.py`)

Defines credential dataclass with all auth methods (token, OAuth, Azure AD)
Handles credential validation and session properties
Manages compute resource configuration

SQL Execution (`handle.py`)

Provides cursor wrapper for Databricks SQL connector
Implements retry logic and connection pooling
Handles SQL execution details and error handling

Relation Handling (`relation.py`)

Extends Spark relations with Databricks features
Handles Unity Catalog 3-level namespace (catalog.schema.table)
Manages relation metadata and configuration

Python Models (`python_models/`)

Executes Python models on Databricks clusters
Supports multiple submission methods (jobs, workflows, serverless)
Handles dependency management and result collection

Macros (`dbt/include/databricks/macros/`)

Jinja2 templates that generate SQL
Override Spark macros with Databricks-specific logic
Handle materializations (table, view, incremental, snapshot)
Implement Databricks features (liquid clustering, column masks, tags)
Important: To override a spark__macro_name macro, create databricks__macro_name (NOT spark__macro_name)

Jinja2 Whitespace Control:

Prefer using - in Jinja tags ({%-, -%}) to strip whitespace and avoid blank lines in generated SQL
Good: {%- if condition -%} - strips whitespace before and after
Without -: {% if condition %} - may leave blank lines in output
This keeps generated SQL clean and readable, especially for conditional column additions
Note: Sometimes whitespace stripping can break formatting, so use judgment

Example:

select
    column1,
    column2
    {%- if config.get('extra_column') -%}
    , extra_column
    {%- endif %}
from table

Multi-Statement SQL Execution

When a macro needs to execute multiple SQL statements (e.g., DELETE followed by INSERT), use the execute_multiple_statements helper:

Pattern for Multi-Statement Strategies:

{% macro my_multi_statement_strategy(args) %}
  {%- set statements = [] -%}
  
  {#-- Build first statement --#}
  {%- set statement1 -%}
    DELETE FROM {{ target_relation }}
    WHERE some_condition
  {%- endset -%}
  {%- do statements.append(statement1) -%}
  
  {#-- Build second statement --#}
  {%- set statement2 -%}
    INSERT INTO {{ target_relation }}
    SELECT * FROM {{ source_relation }}
  {%- endset -%}
  {%- do statements.append(statement2) -%}
  
  {{- return(statements) -}}
{% endmacro %}

How It Works:

Return a list of SQL strings from your strategy macro
The incremental materialization automatically detects lists and calls execute_multiple_statements()
Each statement executes separately via {% call statement('main') %}
Used by: delete+insert incremental strategy (DBR < 17.1 fallback), materialized views, streaming tables

Note: Databricks SQL connector does NOT support semicolon-separated statements in a single execute call. Always return a list.

Configuration System

Models can be configured with Databricks-specific options:

{{ config(
    materialized='table',
    file_format='delta',
    liquid_clustering=['column1', 'column2'],
    tblproperties={'key': 'value'},
    column_tags={'pii_col': ['sensitive']},
    location_root='/mnt/external/'
) }}

🔧 Common Development Tasks

Adding New Materialization

Create macro in macros/materializations/
Implement SQL generation logic
Add configuration options to relation configs
Write unit tests for macro
Write functional tests for end-to-end behavior
Update documentation

Adding New Adapter Method

Every @available method is permanent public API. Jinja is late-bound, so nothing warns you a method is unused, and each one must be independently re-implemented in the Fusion engine's Rust adapter — both its dispatch and its semantic classification — before Databricks behavior matches across engines. A macro built on the shared adapter contract inherits that machinery for free.

Before adding one, work down this list and stop at the first that works:

An existing shared-contract macro. get_columns_in_query, get_empty_subquery_sql, get_columns_in_relation, run_query, and the statement() family cover most metadata and schema-inference needs. get_columns_in_query in particular infers a query's schema with no DDL and no data scan.
Jinja expressions on data you already have. Filters (map, select, list, comparison) handle list and string shaping. A list comparison in a macro does not need a Python helper.
A relation config component under relation_configs/, if the logic is really about describing or diffing relation state.
A new @available method — only when the work genuinely cannot happen in Jinja: real Python control flow, SDK/API calls, credential handling, or non-trivial parsing.

When you do reach step 4, justify it in the PR description, keep the signature in portable types (strings, lists, dicts — not adapter-internal objects), and note that Fusion parity work is now owed.

Then: implement the logic, add the macro if SQL generation is needed, write unit tests with mocked database calls, and write functional tests against a real database.

Modifying SQL Generation

Locate relevant macro in macros/ directory
Test current behavior with unit tests
Modify macro logic
Update unit tests to verify new behavior
Run affected functional tests to ensure no regressions

Adding Configuration Option

Add field to appropriate config class in relation_configs/
Update macro to use new configuration
Add validation logic if needed
Write tests for both valid and invalid configurations

Updating Dependencies (`pyproject.toml` / `uv.lock`)

uv.lock pins the exact version CI tests against. Version bounds in pyproject.toml are independent of the pinned version — a loosened upper bound does NOT auto-bump the pinned version.

When changing a version bound in pyproject.toml:

Run uv lock --upgrade-package <name> to pick up the newest allowed version of that package (targeted, not a full resolve).
Inspect the uv.lock diff — confirm the pinned version now matches the version you actually want to test against.
Commit both pyproject.toml and uv.lock in the same commit.

Why this matters: a pre-commit hook runs uv lock --check and catches pyproject↔lock inconsistency, but it does NOT force pinned versions forward. If the previously pinned version still satisfies the new bound (e.g. raised <4.1.4 to <4.1.6 while 4.1.3 is already pinned), the lock stays consistent and CI keeps testing the old version — the whole point of the bound change is lost. Always run the targeted upgrade explicitly.

Adding a new dependency: uv add <name> updates both files; no separate lock step needed.

🐛 Debugging Guide

Common Issues

SQL Generation: Use macro unit tests with assert_sql_equal()
Connection Problems: Check credentials and environment variables
Python Model Failures: Check cluster configuration and dependencies
Test Failures: Review logs in logs/ directory, look for red text

Debugging Tools

IDE Test Runner: Set breakpoints and step through code
Log Analysis: dbt generates detailed debug logs by default
SQL Inspection: Print generated SQL in macros for debugging
Mock Inspection: Verify mocked calls in unit tests

📚 Key Resources

Documentation

Internal docs (this repo): — see docs/README.md for the full index

docs/flow/ - Materialization execution flow diagrams (table, view, incremental, seed, snapshot, streaming table, materialized view, replace)
docs/dbt-databricks-dev.md - Development setup and workflow
docs/testing.md - Testing guide (unit, macro, functional)
docs/dbr-capability-system.md - Version-dependent features
CONTRIBUTING.MD - Code standards and PR process

User-facing guides (docs/guides/, lower priority — may drift):

docs/guides/uc.md - Using Unity Catalog with dbt-databricks
docs/guides/databricks-jobs.md - Running a dbt project as a Databricks job
docs/guides/workflow-job-submission.md - Python models as Databricks Workflows
docs/guides/databricks-copy-into-macro-aws.md - Loading S3 data via databricks_copy_into

Keeping docs in sync: When a change alters materialization or execution behavior, update the corresponding docs/flow/ doc and bump its _Last updated:_ date. Code is always the source of truth over docs — if a doc and the code disagree, trust the code and fix the doc. Don't let a stale doc block a correct code change.

dbt documentation (docs.getdbt.com):

Databricks Configs - Model/resource configuration options
Materializations - Materialization types and behavior
Python Models - Python model support
Adapter Development - How dbt adapters work

Databricks documentation (docs.databricks.com):

SQL Language Reference - SQL syntax for CREATE, ALTER, etc.
Unity Catalog - UC architecture and governance
Delta Lake - Delta table features (liquid clustering, etc.)
User-Defined Functions - UDF types (SQL, Python, etc.)

Important Files for Agents

pyproject.toml - Project configuration, dependencies, tool settings
test.env.example - Template for test environment variables
tests/conftest.py - Global test configuration
tests/profiles.py - Test database profiles

Code Patterns to Follow

Error Handling: Use dbt's exception classes, provide helpful messages
Logging: Use logger from dbt.adapters.databricks.logging
SQL Generation: Prefer macros over Python string manipulation
Testing: Write both unit and functional tests for new features
Configuration: Use dataclasses with validation for new config options
Imports: Always import at the top of the file, never use local imports within functions or methods
Version Checks: Use capability system instead of direct version comparisons:
- ❌ if adapter.compare_dbr_version(16, 1) >= 0:
- ✅ if adapter.has_capability(DBRCapability.COMMENT_ON_COLUMN):
- ✅ {% if adapter.has_dbr_capability('comment_on_column') %}
Jinja2 Whitespace: Prefer using - in Jinja tags ({%-, -%}) to strip whitespace and prevent blank lines in generated SQL:
- Preferred: {%- if condition -%}
- Without: {% if condition %} (may create blank lines)
Jinja ↔ Python boundary: Keep it as narrow as possible. Prefer a shared-contract macro over a new @available method; prefer Jinja filters over a Python helper that only reshapes a list or string. Removing an @available method in favor of portable macros is a win, not churn.

🚨 Common Pitfalls for Agents

Don't modify dbt-spark behavior without understanding inheritance
Always run code-quality before committing changes
Test on multiple environments (HMS, UC cluster, SQL warehouse)
Mock external dependencies in unit tests properly
Use appropriate test fixtures from dbt-tests-adapter
Follow SQL normalization in test assertions with assert_sql_equal()
Handle Unity Catalog vs HMS differences in feature implementations
Consider backward compatibility when modifying existing behavior
Use capability system for version checks - Never add new compare_dbr_version() calls
Remember per-compute caching - Different clusters may have different capabilities in the same run
Multi-statement SQL: Don't use semicolons to separate statements - return a list instead and let execute_multiple_statements() handle it
Don't add an @available method for logic Jinja can already do - each one is permanent API and owes a matching Fusion implementation. Check the shared adapter contract first.

🎯 Success Metrics

When working on this codebase, ensure:

All tests pass (hatch run code-quality && hatch run unit)
CRITICAL: Run affected functional tests before declaring success
- If you modified connection/capability logic: Run tests that use multiple computes or check capabilities
- If you modified incremental materializations: Run tests/functional/adapter/incremental/
- If you modified Python models: Run tests/functional/adapter/python_model/
- If you modified macros: Run tests that use those macros
- NEVER declare "mission accomplished" without running functional tests for affected features
New features have both unit and functional tests
SQL generation follows Databricks best practices
Changes maintain backward compatibility
Code follows project style guidelines

This guide is maintained by the dbt-databricks team. When making significant architectural changes, update this guide to help future agents understand the codebase.

Uh oh!

FilesExpand file tree

AGENTS.md

Latest commit

History