feat: make the hash input cast datatype configurable per adapter by tkirschke · Pull Request #497 · ScalefreeCOM/datavault4dbt

tkirschke · 2026-08-05T14:12:13Z

Context

A user profiling a slow full refresh on MS SQL Server reported that changing the VARCHAR(MAX) casts in hash_standardization.sql to VARCHAR(8000) took a single satellite load from over a minute to a few seconds.

The report holds up. VARCHAR(MAX) is a large-value type: stored off-row, not handled in memory like a regular VARCHAR(n), and it blocks several optimizations for the REPLACE() / UPPER() / HASHBYTES() chain wrapped around it. There was no design reason for MAX specifically — Synapse and Fabric, which share the T-SQL codepath, already used VARCHAR(4000). SQL Server was the outlier.

Changes

Two casts happen before hashing, and they are now configurable independently:

Variable	Applies to
`datavault4dbt.hash_input_attribute_dtype`	the cast of one single column in `attribute_standardise`
`datavault4dbt.hash_input_concat_dtype`	the cast of the fully concatenated payload in `concattenated_standardise`

Both are adapter-keyed mapping dictionaries following the datavault4dbt.beginning_of_all_times convention, resolved by the new macros/supporting/hash_input_dtype.sql. That macro follows the lighter first_day_of_week.sql pattern (single default__ implementation using target.type) rather than the 11-macro-per-adapter string_default_dtype.sql pattern.

The current hardcoded values are kept as defaults:

	attribute	concat
bigquery / snowflake / databricks	`STRING`	`STRING`
exasol	`VARCHAR(20000) UTF8`	`VARCHAR(2000000) UTF8`
postgres / trino	`VARCHAR`	`VARCHAR`
redshift	(no cast, see below)	`VARCHAR`
synapse / fabric	`VARCHAR(4000)`	`VARCHAR(4000)`
oracle	`VARCHAR2(2000)`	`VARCHAR2(2000)`
sqlserver	`VARCHAR(MAX)`	`VARCHAR(MAX)`

Multi Active Satellites are deliberately excluded

multi_active_concattenated_standardise keeps its hardcoded datatype for all adapters and is untouched by this PR.

The reason is STRING_AGG(), which aggregates the payload of all active records of one group before hashing. On SQL Server it only returns VARCHAR(MAX) if its input expression is VARCHAR(MAX); with a shorter input it returns VARCHAR(8000) and raises an error once a group's aggregate exceeds 8000 bytes. Since that limit applies to a whole group rather than a single record, it is far easier to hit than the per-record limit of a regular satellite.

Excluding MA sats gives the property we want: a user can shorten the cast for hubs, links and regular satellites and get the speedup, with no way to accidentally break their MA sats through a global variable.

Truncation caveat

A datatype shorter than the actual hash input truncates it silently on most adapters. Truncated input changes the resulting hashes, and two rows differing only past the truncation point collapse into the same hashkey or hashdiff. Any later change to these variables is a full reload of the affected entities. This is called out on the global variables page and in the SQL Server adapter notes.

Verification

No dbt profile was available, so this was verified statically:

Every standardise_prefix / standardise_suffix / expr expression was extracted from both main and this branch, evaluated with the shipped defaults, and compared — all 33 macros render byte-identical to main. The generated SQL does not change for anyone who does not override the variables.
The 11 multi_active blocks are textually identical to main, not just render-equivalent.
All 223 format expressions evaluate without arity errors.
dbt_project.yml parses as YAML; both macro files pass a Jinja parse.
hash_input_dtype was render-tested across all branches: dict hit, dict missing the own adapter (warning + fallback), scalar override, and variable absent entirely.

Note

redshift__attribute_standardise has no cast at all — it trims the raw expression. Rather than introduce one and change Redshift behavior, it was left as is, so redshift has no key in hash_input_attribute_dtype. The macro is never called with type='attribute' on Redshift, so no warning fires in practice.

🤖 Generated with Claude Code

Before hashing, both a single input column and the fully concatenated payload are casted to a string datatype. These datatypes were hardcoded per adapter. On SQL Server the hardcoded value was VARCHAR(MAX), which is a large-value type: it is stored off-row and blocks several optimizations for the REPLACE(), UPPER() and HASHBYTES() calls wrapped around it. A user report measured a satellite load dropping from over a minute to a few seconds after switching to VARCHAR(8000). Both casts are now configurable through the new global variables `datavault4dbt.hash_input_attribute_dtype` and `datavault4dbt.hash_input_concat_dtype`. Both are adapter-keyed mapping dictionaries, following `datavault4dbt.beginning_of_all_times`. The current hardcoded values are kept as defaults, so the generated SQL is unchanged unless a user overrides them. Multi Active Satellites are deliberately excluded and keep their hardcoded datatype. Their payload is aggregated across all active records of one group before it is hashed, and on SQL Server STRING_AGG() only returns VARCHAR(MAX) if its input is VARCHAR(MAX) - with a shorter input it raises an error once a group exceeds 8000 bytes. Excluding them lets users shorten the cast for all other entities without that risk. Note that a datatype shorter than the actual hash input truncates it silently on most adapters, which changes the resulting hash values. This is documented on the global variables page and in the SQL Server notes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

remoteworkflow · 2026-08-10T06:49:44Z

dbt test combined result: ❌

Details

RESULTS for Synapse:
❌ dbt-core-tests
⚠️ dbt-fusion-tests
⚠️ dbt-macro-tests
⚠️ tech-tests

RESULTS for Postgres:
✅ dbt-core-tests
⚠️ dbt-fusion-tests
⚠️ dbt-macro-tests
⚠️ tech-tests

RESULTS for BigQuery:
✅ dbt-core-tests
✅ dbt-fusion-tests
⚠️ dbt-macro-tests
⚠️ tech-tests

RESULTS for Redshift:
✅ dbt-core-tests
✅ dbt-fusion-tests
⚠️ dbt-macro-tests
⚠️ tech-tests

RESULTS for Snowflake:
✅ dbt-core-tests
✅ dbt-fusion-tests
⚠️ dbt-macro-tests
⚠️ tech-tests

RESULTS for Exasol:
❌ dbt-core-tests
⚠️ dbt-fusion-tests
⚠️ dbt-macro-tests
⚠️ tech-tests

RESULTS for Fabric:
❌ dbt-core-tests
⚠️ dbt-fusion-tests
⚠️ dbt-macro-tests
⚠️ tech-tests

RESULTS for Oracle:
✅ dbt-core-tests
⚠️ dbt-fusion-tests
⚠️ dbt-macro-tests
⚠️ tech-tests

RESULTS for Databricks:
❌ dbt-core-tests
❌ dbt-fusion-tests
⚠️ dbt-macro-tests
⚠️ tech-tests

RESULTS for SQL Server:
✅ dbt-tests
⚠️ dbt-fusion-tests
⚠️ dbt-macro-tests
⚠️ tech-tests

RESULTS for Trino:
✅ dbt-tests
⚠️ dbt-fusion-tests
⚠️ dbt-macro-tests
⚠️ tech-tests

Link to workflow summary: https://github.com/ScalefreeCOM/datavault4dbt-ci-cd/actions/runs/31360291375

Copilot

Pull request overview

This PR introduces adapter-configurable datatypes for casting hash inputs prior to standardization and hashing, enabling performance tuning (notably for SQL Server) while keeping existing rendered SQL unchanged unless users override the new variables.

Changes:

Added hash_input_dtype macro to resolve per-adapter (or scalar) cast datatypes for hash inputs, split by attribute vs concat.
Updated hash_standardization.sql to use the new macro instead of hardcoded cast types across adapters (excluding multi-active satellites as intended).
Documented the new variables and SQL Server-specific guidance, and added default variable mappings in dbt_project.yml.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
macros/supporting/hash_standardization.sql	Replaces hardcoded hash-input casts with adapter-resolved datatypes via `hash_input_dtype` for attribute and concatenated payload paths.
macros/supporting/hash_input_dtype.sql	Adds new dispatchable macro that resolves cast datatypes from variables with adapter fallbacks.
docs/26_general-usage-notes/33_adapter-specific-notes/40_sqlserver/40_sqlserver.md	Documents SQL Server performance context, configuration example, and MA-satellite exclusion rationale.
docs/26_general-usage-notes/29_global-variables/29_global-variables.md	Adds the two new global variables plus truncation warning and MA-satellite exclusion note.
dbt_project.yml	Provides default values for the new variables as adapter-keyed mappings.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

remoteworkflow · 2026-08-11T14:38:02Z