feat: make the hash input cast datatype configurable per adapter

dbt_project.yml

@@ -31,6 +31,12 @@ vars:
   datavault4dbt.hashdiff_input_case_sensitive: TRUE
   datavault4dbt.hashdiff_use_trim: TRUE
 
+  # Datatypes the hash inputs are casted to before hashing. 'attribute' is one column, 'concat' the joined payload.
+  # Shortening either truncates hash input silently. On sqlserver, synapse and fabric a shortened 'attribute'
+  # also bounds the whole payload. Read the Global Variables docs page before changing them.
+  datavault4dbt.hash_input_attribute_dtype: {"bigquery":"STRING","snowflake":"STRING", "exasol": "VARCHAR(20000) UTF8", "postgres": "VARCHAR", "synapse": "VARCHAR(4000)", "fabric": "VARCHAR(4000)", "oracle":"VARCHAR2(2000)", databricks: "STRING", trino: "VARCHAR", "sqlserver": "VARCHAR(MAX)"}
+  datavault4dbt.hash_input_concat_dtype: {"bigquery":"STRING","snowflake":"STRING", "exasol": "VARCHAR(2000000) UTF8", "postgres": "VARCHAR", "redshift": "VARCHAR", "synapse": "VARCHAR(4000)", "fabric": "VARCHAR(4000)", "oracle":"VARCHAR2(2000)", databricks: "STRING", trino: "VARCHAR", "sqlserver": "VARCHAR(MAX)"}
+
   # Delimiters used when concatenating columns before hashing
   datavault4dbt.concat_string: '||'
   datavault4dbt.quote_character: '"'

docs/26_general-usage-notes/29_global-variables/29_global-variables.md

@@ -59,6 +59,21 @@ All the following variables are **prefixed with `datavault4dbt`**.
 | concat_string_replacement      | Stage | Token substituted for any occurrence of `concat_string` found *inside* the input data, so real values can never collide with the structural delimiter. Defaults to `dv4dbt-concat-replacement`. |
 | quote_character_replacement    | Stage | Token substituted for any occurrence of `quote_character` found *inside* the input data. Defaults to `dv4dbt-quote-replacement`. |
 | null_placeholder_string_replacement | Stage | Token substituted for any occurrence of `null_placeholder_string` found *inside* the input data. Defaults to `dv4dbt-null-replacement`. |
+| hash_input_attribute_dtype     | Stage | A mapping dictionary that defines, per database adapter, the datatype that a **single input column** is casted to inside `attribute_standardise`, before concatenation. Advanced. On T-SQL adapters a bounded value here also bounds the total payload, see the warning below. Leave at the default unless you have measured. |
+| hash_input_concat_dtype        | Stage | A mapping dictionary that defines, per database adapter, the datatype that the **fully concatenated payload** is casted to inside `concattenated_standardise`, before it is hashed. This is the variable to shorten if you want the performance benefit. |
+
+
+Multi Active Satellites are excluded from `hash_input_concat_dtype`: `multi_active_concattenated_standardise` keeps its hardcoded datatype, so their aggregated payload can never overflow `STRING_AGG`. They are **not** excluded from `hash_input_attribute_dtype`, which is shared with all other entities and applies per column, so a Multi Active Satellite needs the same width check as a regular Satellite.
+
+:::warning
+Shortening `hash_input_attribute_dtype` or `hash_input_concat_dtype` below the actual length of your hash input truncates that input silently on most adapters. Truncated input produces different hash values, and two rows that only differ behind the truncation point collapse into the same hashkey or hashdiff. Only lower these values if you are certain that your concatenated input stays below the chosen limit, and treat any later change as a full reload of the affected entities.
+:::
+
+:::danger
+On **sqlserver**, **synapse** and **fabric** the columns are joined with `CONCAT()` / `CONCAT_WS()`, and those functions derive their own maximum length from their arguments: the result is capped at 8000 characters unless at least one argument is an unbounded type. A bounded `hash_input_attribute_dtype` therefore bounds your **total** hash input, not the single column it is named after, and the later cast of the concatenated payload cannot recover what was already dropped.
+
+The safe per-column bound on those adapters is roughly `8000 / number_of_columns_in_the_hash`, not 8000. Twenty columns of 500 characters already exceed the limit, even though every column is far inside its own. Leave `hash_input_attribute_dtype` at its default unless you have measured the concatenated length of your widest entity, and re-measure whenever you add a column.
+:::
 
 ### STAGE CONFIGURATION
 

docs/26_general-usage-notes/33_adapter-specific-notes/40_sqlserver/40_sqlserver.md

@@ -18,3 +18,53 @@ The SQL Server macros fall back to `VARBINARY(16)` internally, but if you copied
 
 - **Switch to a binary hash type:** Set `datavault4dbt.hash_datatype` in your `dbt_project.yml` to `VARBINARY(16)` (for MD5). This ensures hash keys and hashdiffs are stored efficiently and compared correctly on SQL Server.
 
+## HASH INPUT DATA TYPE
+
+Before hashing, both the single input columns and the concatenated payload are casted to a string datatype. On SQL Server this defaults to `VARCHAR(MAX)`, which is a large-value type: it is stored off-row, cannot be held in memory the same way as a regular `VARCHAR(n)`, and blocks several optimizations for the `REPLACE()`, `UPPER()` and `HASHBYTES()` calls wrapped around it. On wide satellites this can dominate the runtime of a load.
+
+Both casts are configurable per adapter. **Shorten only the concatenated payload cast:**
+
+```yaml
+vars:
+  datavault4dbt.hash_input_concat_dtype: {"sqlserver": "VARCHAR(8000)"}
+```
+
+`VARCHAR(8000)` is the largest non-large-value `VARCHAR` on SQL Server, so this takes the value that `HASHBYTES()` consumes out of large-value handling while leaving the individual column casts unbounded.
+
+:::warning
+Any hash input longer than the configured length is truncated **silently**, which changes the resulting hashkeys and hashdiffs. Two rows that only differ behind the truncation point produce the same hash, and a Satellite then stops recording changes confined to that region. Before lowering this value, check that the concatenated input of your widest entity stays below the limit:
+
+- sum the lengths of all columns in the hash
+- add 2 characters per column for the quoting the package applies
+- add the length of `concat_string` between each pair of columns
+
+Treat a later change to this value as a full reload of all affected entities.
+:::
+
+### THE PER-COLUMN CAST
+
+`datavault4dbt.hash_input_attribute_dtype` sets the datatype each column is cast to **before** the columns are joined. Leave it at its default of `VARCHAR(MAX)` on SQL Server.
+
+The reason is the join itself. The package joins the columns with `CONCAT()` / `CONCAT_WS()`, and those functions derive their own maximum length from their arguments: the result is capped at 8000 characters unless at least one argument is an unbounded type. Bounding every column therefore bounds the **whole** concatenated payload, and it does so before the concatenated-payload cast runs, so that cast cannot widen it back:
+
+```sql
+DECLARE @a VARCHAR(8000) = REPLICATE('a', 8000);
+DECLARE @b VARCHAR(8000) = REPLICATE('b', 8000);
+
+SELECT LEN(CONCAT_WS('||', @a, @b))                        AS both_bounded,      -- 8000
+       LEN(CAST(CONCAT_WS('||', @a, @b) AS VARCHAR(MAX)))  AS bounded_then_cast, -- 8000
+       LEN(CONCAT_WS('||', CAST(@a AS VARCHAR(MAX)), @b))  AS one_unbounded;     -- 16002
+```
+
+Consequence: with `VARCHAR(8000)` per column, an entity of 20 columns averaging 500 characters concatenates to about 10 078 characters, the last ~2 000 are dropped, and roughly the final four columns never reach the hash function. The hashdiff stops depending on them, so a change confined to those columns produces no new hashdiff and the Satellite records nothing. No error is raised.
+
+If you still need the per-column cast, the safe bound is approximately `8000 / number_of_columns_in_the_hash`, and it must be re-checked whenever a column is added to the entity.
+
+### MULTI ACTIVE SATELLITES
+
+`hash_input_concat_dtype` does not reach Multi Active Satellites: `multi_active_concattenated_standardise` keeps its hardcoded datatype, so their aggregated payload can never overflow `STRING_AGG()`.
+
+`STRING_AGG()` is the reason. It only returns `VARCHAR(MAX)` if its input expression is `VARCHAR(MAX)`; with a shorter input it returns `VARCHAR(8000)` and **raises an error** as soon as the aggregated result of a single group exceeds 8000 bytes. Since that limit applies to a whole group instead of a single record, it is far easier to hit than the per-record limit of a regular Satellite.
+
+`hash_input_attribute_dtype` **does** reach them, because the per-column cast is shared with all other entities. A Multi Active Satellite therefore needs the same width check as a regular Satellite.
+

macros/supporting/hash_input_dtype.sql

@@ -0,0 +1,59 @@
+{#
+    Returns the string datatype that hash inputs are casted to, before they are handed over to the hash function.
+
+    Two different casts exist, which can be configured independently:
+      - 'attribute': The cast of one single column inside `attribute_standardise`.
+      - 'concat':    The cast of the fully concatenated payload inside `concattenated_standardise`.
+
+    Multi Active Satellites are deliberately NOT covered here. Their payload is aggregated across all
+    active records of one group before it is hashed, so a shortened datatype is far more likely to be
+    exceeded there. `multi_active_concattenated_standardise` therefore keeps its hardcoded datatype.
+
+    CAUTION: Choosing a datatype that is shorter than the actual hash input leads to a silent
+    truncation on most adapters, which changes the resulting hash values.
+#}
+
+{%- macro hash_input_dtype(type='concat') %}
+
+    {{ return(adapter.dispatch('hash_input_dtype', 'datavault4dbt')(type=type)) }}
+
+{%- endmacro -%}
+
+
+{%- macro default__hash_input_dtype(type) %}
+
+    {%- if type not in ['attribute', 'concat'] -%}
+        {%- do exceptions.raise_compiler_error("hash_input_dtype: type must be 'attribute' or 'concat', got: " ~ type) -%}
+    {%- endif -%}
+
+    {%- if type == 'attribute' -%}
+        {%- set var_name = 'datavault4dbt.hash_input_attribute_dtype' -%}
+        {%- set fallbacks = {"bigquery": "STRING", "snowflake": "STRING", "exasol": "VARCHAR(20000) UTF8", "postgres": "VARCHAR", "synapse": "VARCHAR(4000)", "fabric": "VARCHAR(4000)", "oracle": "VARCHAR2(2000)", "databricks": "STRING", "trino": "VARCHAR", "sqlserver": "VARCHAR(MAX)"} -%}
+    {%- else -%}
+        {%- set var_name = 'datavault4dbt.hash_input_concat_dtype' -%}
+        {%- set fallbacks = {"bigquery": "STRING", "snowflake": "STRING", "exasol": "VARCHAR(2000000) UTF8", "postgres": "VARCHAR", "redshift": "VARCHAR", "synapse": "VARCHAR(4000)", "fabric": "VARCHAR(4000)", "oracle": "VARCHAR2(2000)", "databricks": "STRING", "trino": "VARCHAR", "sqlserver": "VARCHAR(MAX)"} -%}
+    {%- endif -%}
+
+    {%- set global_var = var(var_name, none) -%}
+    {%- set adapter_name = (target.type | lower) -%}
+    {%- set hash_input_dtype = none -%}
+
+    {%- if global_var is mapping -%}
+        {%- set hash_input_dtype = global_var.get(adapter_name) -%}
+        {%- if hash_input_dtype is none -%}
+            {%- set hash_input_dtype = global_var.get(target.type) -%}
+        {%- endif -%}
+    {%- elif datavault4dbt.is_something(global_var) -%}
+        {%- set hash_input_dtype = global_var -%}
+    {%- endif -%}
+
+    {%- if not datavault4dbt.is_something(hash_input_dtype) -%}
+        {%- set hash_input_dtype = fallbacks.get(adapter_name, 'STRING') -%}
+        {%- if execute and global_var is mapping -%}
+            {%- do exceptions.warn("Warning: Adapter '" ~ target.type ~ "' not found in '" ~ var_name ~ "' variable. Defaulting to '" ~ hash_input_dtype ~ "'.") -%}
+        {%- endif -%}
+    {%- endif -%}
+
+    {{ return(hash_input_dtype) }}
+
+{%- endmacro -%}