test(hipdnn): reland SDPA fwd causal/hd192/FP8/GROUP golden data + DVC remote

.dvc/config

@@ -4,3 +4,6 @@
 ['remote "storage"']
     url = s3://therock-dvc/rocm-libraries
     allow_anonymous_login = true
+['remote "golden-data"']
+    url = s3://therock-dvc/rocm-libraries/hipdnn/golden-data
+    allow_anonymous_login = true

dnn-providers/integration-tests/integration_test_bundles/README.md

@@ -15,10 +15,72 @@ Binary data is stored in S3 via [DVC](https://dvc.org) — git only tracks small
 
 | Key            | Value                                |
 |----------------|--------------------------------------|
-| Remote         | `s3://therock-dvc/rocm-libraries`    |
+| Default remote | `storage` → `s3://therock-dvc/rocm-libraries` (legacy shared store; existing ops) |
+| hipDNN remote  | `golden-data` → `s3://therock-dvc/rocm-libraries/hipdnn/golden-data` (new ops, per-op subdirs) |
 | Tracking       | Per-bundle or per-sweep-case `.dvc` tracking (`.json` stays in git for PR review) |
 | Naming spec    | [RFC 0011 Section 4.1](../../../projects/hipdnn/docs/rfcs/0011_GoldenReferenceValidation.md) |
 
+### DVC Remote Layout (`golden-data`)
+
+New hipDNN golden data uses a single `golden-data` remote rooted at
+`s3://therock-dvc/rocm-libraries/hipdnn/golden-data`. DVC stores objects
+content-addressed under `<remote-url>/files/md5/`, so all ops on this remote share
+one store:
+
+```
+s3://therock-dvc/rocm-libraries/hipdnn/golden-data/files/md5/<ab>/<rest...>
+```
+
+**Per-op organization is in the git tree, not in S3.** Each op is a top-level
+folder under `integration_test_bundles/{Tier}/` (e.g. `SdpaFwd`, `BatchnormFwdInference`).
+Push/pull is **scoped by repo path** so you operate on one op at a time, even
+though the bytes all land in the same content-addressed S3 store:
+
+```bash
+# push only SDPA-forward bundles, to the golden-data remote:
+dvc push -r golden-data --recursive \
+  dnn-providers/integration-tests/integration_test_bundles/quick/SdpaFwd \
+  dnn-providers/integration-tests/integration_test_bundles/standard/SdpaFwd
+```
+
+**Each pointer declares its own remote (so the default `dvc pull` just works).**
+A bundle on the `golden-data` remote sets a per-output `remote:` key in its
+`.tensors.dvc`:
+
+```yaml
+outs:
+- path: Small.tensor0.bin
+  remote: golden-data        # <- DVC fetches/pushes this output from golden-data
+  md5: 437ad9f6e2b1db28ca900503e09f7a63
+  size: 524288
+  hash: md5
+```
+
+With this key, a **bare `dvc pull` (no `-r`)** pulls golden-data outputs from
+golden-data and everything else from the default `storage` remote — in one pass.
+This is why **CI needs no change**: the existing `dvc pull` step already fetches
+golden-data bundles. The remote travels *with the data pointer*, not in a separate
+workflow file every consumer must remember to flag.
+
+**Why a single remote (not one remote per op):**
+- DVC remotes are global names in `.dvc/config`; a `.tensors.dvc` pointer records
+  md5/size plus an optional per-output `remote:`. The pull/push location is that
+  per-output remote (falling back to the repo default).
+- One remote + the per-output `remote:` key keeps **CI to a single bare `dvc pull`**,
+  and every new op is just a new folder — no `.dvc/config` edit, no CI change.
+- Per-op isolation is **logical** (git folder taxonomy + path-scoped push), not a
+  separate S3 keyspace. Cross-op dedup is not expected (different ops produce
+  different tensors), so a shared store under the root is acceptable.
+- Trade-off: because all ops share `…/golden-data/files/md5/`, you cannot prune one
+  op's S3 objects in isolation with `dvc gc --cloud` (gc keys off what the repo
+  references across the whole remote). If a future op needs independent S3
+  lifecycle / gc, promote it to its own remote (`…/golden-data/<op>`) at that time.
+
+> The legacy `storage` remote (bare `rocm-libraries` root) remains the default.
+> Existing ops (e.g. Batchnorm) keep working unchanged; their pointers have no
+> `remote:` key so they resolve to `storage`. New hipDNN ops add `remote: golden-data`
+> to their pointers (and push with `-r golden-data`).
+
 ## Folder Convention
 
 Single-graph bundle:
@@ -70,14 +132,18 @@ verification.
 ```bash
 # From the repo root:
 
-# Pull all bundles
+# Pull everything. A bare pull resolves each output to its own remote:
+# golden-data outputs come from golden-data (via the per-output 'remote:' key),
+# all other outputs from the default 'storage' remote.
 dvc pull
 
-# Pull only quick-tier bundles (sufficient for smoke tests)
-dvc pull dnn-providers/integration-tests/integration_test_bundles/quick/
+# Pull only one op's quick-tier bundles (sufficient for smoke tests)
+dvc pull dnn-providers/integration-tests/integration_test_bundles/quick/SdpaFwd
 ```
 
-The Linux CI workflow (`therock-ci-linux.yml`) runs `dvc pull` automatically. Other pipelines (Windows, superbuild) may need it wired in separately — see backlog.
+CI needs no special flag: the existing `dvc pull` step in `therock-ci-linux.yml`
+already fetches golden-data bundles, because each golden-data pointer carries its
+own `remote: golden-data` key (see the DVC Remote Layout section above).
 
 > **Note:** DVC commands must be run from the repo root (`rocm-libraries/`), not from a subdirectory.
 
@@ -93,19 +159,59 @@ cp resnet50_layer3.tensor*.bin dnn-providers/integration-tests/integration_test_
 
 # 3. Author a single per-bundle DVC pointer listing every .bin, then let DVC fill in the hashes.
 #    DVC cannot generate a multi-file pointer itself, so we write the `outs:` list and `dvc commit`.
+#    For NEW hipDNN ops, tag each output with `remote: golden-data` so a bare
+#    `dvc pull` (and CI) fetches it from the golden-data remote.
 BUNDLE=dnn-providers/integration-tests/integration_test_bundles/quick/ConvFwd/nhwc/fp16/resnet50_layer3
-{ echo "outs:"; for f in "$BUNDLE"/*.tensor*.bin; do echo "- path: $(basename "$f")"; done; } > "$BUNDLE/resnet50_layer3.tensors.dvc"
+{ echo "outs:"; for f in "$BUNDLE"/*.tensor*.bin; do
+    echo "- path: $(basename "$f")"; echo "  remote: golden-data"; done; } \
+    > "$BUNDLE/resnet50_layer3.tensors.dvc"
 dvc commit -f "$BUNDLE/resnet50_layer3.tensors.dvc"
 
 # 4. Git-add the .json (for PR review) and the .tensors.dvc pointer
 git add "$BUNDLE/resnet50_layer3.json" "$BUNDLE/resnet50_layer3.tensors.dvc"
 
 # 5. Commit and push
+#    NEW hipDNN ops push to the golden-data remote; legacy 'storage' ops omit -r.
 git commit -m "Add ConvFwd resnet50_layer3 bundle"
-dvc push
+dvc push -r golden-data    # or: dvc push   (legacy 'storage' remote)
+git push
+```
+
+### SDPA forward bundles (regen-driven)
+
+SDPA-forward bundles are produced by a generator, not copied in. Regenerate,
+record pointers, and push to `golden-data`:
+
+```bash
+# 1. Regenerate all SDPA-fwd bundles (quick + standard tiers)
+cd dnn-providers/integration-tests/integration_test_bundles/quick/SdpaFwd
+bash generate_golden_data.sh                 # Tier A (bf16/fp16, nomask/causal, hd128/hd192)
+GENERATE_TIER_B=1 bash generate_golden_data.sh   # + FP8/GROUP (not yet CI-validated)
+cd -
+
+# 2. Record .bin hashes into the per-bundle pointers + cache (from repo root).
+#    `dvc commit` preserves the existing per-output `remote: golden-data` key, so
+#    pointers stay routed to golden-data across regenerations. (For a brand-new
+#    bundle dir, add the key once — see step 3 of "Add a New Bundle".)
+dvc commit -f --recursive \
+  dnn-providers/integration-tests/integration_test_bundles/quick/SdpaFwd \
+  dnn-providers/integration-tests/integration_test_bundles/standard/SdpaFwd
+
+# 3. Push tensor data to the golden-data remote
+dvc push -r golden-data --recursive \
+  dnn-providers/integration-tests/integration_test_bundles/quick/SdpaFwd \
+  dnn-providers/integration-tests/integration_test_bundles/standard/SdpaFwd
+
+# 4. Stage .json + .meta.json + .tensors.dvc (NOT .bin — gitignored), commit, git push
+git add dnn-providers/integration-tests/integration_test_bundles/{quick,standard}/SdpaFwd
+git commit -m "Update SDPA forward golden bundles"
 git push
 ```
 
+> `dvc push` uploads from the **local cache**, not the working tree — always
+> `dvc commit` (or `dvc add`) first so regenerated `.bin` are cached, or push will
+> silently skip them.
+
 ## Update an Existing Bundle
 
 ```bash

dnn-providers/integration-tests/integration_test_bundles/quick/SdpaFwd/bhsd/bf16/hd128_causal_batch/Small/Small.json

@@ -0,0 +1,134 @@
+{
+    "nodes": [
+        {
+            "type": "SdpaAttributes",
+            "compute_data_type": "float",
+            "name": "",
+            "inputs": {
+                "q_tensor_uid": 0,
+                "k_tensor_uid": 1,
+                "v_tensor_uid": 2,
+                "attn_mask_tensor_uid": null,
+                "scale_tensor_uid": null,
+                "seq_len_q_tensor_uid": null,
+                "seq_len_kv_tensor_uid": null,
+                "seed_tensor_uid": null,
+                "offset_tensor_uid": null,
+                "dropout_mask_tensor_uid": null,
+                "dropout_scale_tensor_uid": null,
+                "page_table_k_tensor_uid": null,
+                "page_table_v_tensor_uid": null,
+                "block_mask_tensor_uid": null,
+                "sink_token_tensor_uid": null,
+                "descale_q_tensor_uid": null,
+                "descale_k_tensor_uid": null,
+                "descale_v_tensor_uid": null,
+                "descale_s_tensor_uid": null,
+                "scale_s_tensor_uid": null,
+                "scale_o_tensor_uid": null
+            },
+            "outputs": {
+                "o_tensor_uid": 3,
+                "stats_tensor_uid": null,
+                "max_tensor_uid": null,
+                "sum_exp_tensor_uid": null,
+                "rng_dump_tensor_uid": null,
+                "amax_s_tensor_uid": null,
+                "amax_o_tensor_uid": null
+            },
+            "attributes": {
+                "generate_stats": null,
+                "alibi_mask": false,
+                "padding_mask": false,
+                "causal_mask": false,
+                "causal_mask_bottom_right": false,
+                "dropout_probability": null,
+                "attn_scale_value": 0.08838834764831843,
+                "left_bound": -1,
+                "right_bound": 0,
+                "max_seq_len_kv": null,
+                "diagonal_alignment": "BOTTOM_RIGHT",
+                "mma_core_mode": "float",
+                "implementation": "AUTO"
+            }
+        }
+    ],
+    "tensors": [
+        {
+            "uid": 0,
+            "name": "Q",
+            "dims": [
+                2,
+                4,
+                256,
+                128
+            ],
+            "strides": [
+                131072,
+                32768,
+                128,
+                1
+            ],
+            "data_type": "bfloat16",
+            "virtual": false
+        },
+        {
+            "uid": 1,
+            "name": "K",
+            "dims": [
+                2,
+                4,
+                256,
+                128
+            ],
+            "strides": [
+                131072,
+                32768,
+                128,
+                1
+            ],
+            "data_type": "bfloat16",
+            "virtual": false
+        },
+        {
+            "uid": 2,
+            "name": "V",
+            "dims": [
+                2,
+                4,
+                256,
+                128
+            ],
+            "strides": [
+                131072,
+                32768,
+                128,
+                1
+            ],
+            "data_type": "bfloat16",
+            "virtual": false
+        },
+        {
+            "uid": 3,
+            "name": "O",
+            "dims": [
+                2,
+                4,
+                256,
+                128
+            ],
+            "strides": [
+                131072,
+                32768,
+                128,
+                1
+            ],
+            "data_type": "bfloat16",
+            "virtual": false
+        }
+    ],
+    "io_data_type": "bfloat16",
+    "compute_data_type": "float",
+    "intermediate_data_type": "float",
+    "name": ""
+}

dnn-providers/integration-tests/integration_test_bundles/quick/SdpaFwd/bhsd/bf16/hd128_causal_batch/Small/Small.meta.json

@@ -0,0 +1,36 @@
+{
+    "generator": "generate_sdpa_fwd_golden.py",
+    "generator_sha256": "eaffaeaf20fd4204529d8b1d130271eb91888775e5d52ccc66d865d568f7f5b9",
+    "generated_at": "2026-06-26T20:55:25Z",
+    "reference_source": "PyTorch 2.12.0+rocm7.2",
+    "reference_backend": "pytorch_math_backend",
+    "rocm_version": "7.2",
+    "generator_version": "1.1.0",
+    "generation_precision": "like-for-like: BF16 inputs, FP32 intermediates",
+    "direction": "forward",
+    "seed": 42,
+    "input_range": [
+        -1.0,
+        1.0
+    ],
+    "deterministic": true,
+    "config": {
+        "batch": 2,
+        "num_heads_q": 4,
+        "num_heads_kv": 4,
+        "seq_q": 256,
+        "seq_kv": 256,
+        "head_dim_qk": 128,
+        "head_dim_v": 128,
+        "dtype": "bf16",
+        "causal": "bottom_right",
+        "window_left": -1,
+        "window_right": -1,
+        "group_mode": false,
+        "seq_lens_q": null,
+        "seq_lens_kv": null,
+        "stats": false,
+        "scale": 0.08838834764831843,
+        "gqa_ratio": 1
+    }
+}

dnn-providers/integration-tests/integration_test_bundles/quick/SdpaFwd/bhsd/bf16/hd128_causal_batch/Small/Small.tensors.dvc

@@ -0,0 +1,21 @@
+outs:
+- path: Small.tensor0.bin
+  remote: golden-data
+  md5: 437ad9f6e2b1db28ca900503e09f7a63
+  size: 524288
+  hash: md5
+- path: Small.tensor1.bin
+  remote: golden-data
+  md5: d7ebfe7c68541eb2b0583dec8d626700
+  size: 524288
+  hash: md5
+- path: Small.tensor2.bin
+  remote: golden-data
+  md5: 3c6f4153c81aeb2ec5fc0f16f6f5905b
+  size: 524288
+  hash: md5
+- path: Small.tensor3.bin
+  remote: golden-data
+  md5: 8092b6d5e8f521aa0e063f6e567a0b07
+  size: 524288
+  hash: md5