test(hipdnn): reland SDPA fwd causal/hd192/FP8/GROUP golden data + DVC remote#9247
Draft
AnaghaRaoAMD wants to merge 3 commits into
Draft
test(hipdnn): reland SDPA fwd causal/hd192/FP8/GROUP golden data + DVC remote#9247AnaghaRaoAMD wants to merge 3 commits into
AnaghaRaoAMD wants to merge 3 commits into
Conversation
…8522) Builds on the merged LSE/stats work (ALMIOPEN-2082, #8522) to extend forward golden data coverage toward the full AITER forward operation space. Generator (generate_sdpa_fwd_golden.py, v1.1.0): - Implement --causal {none,top_left,bottom_right} mask construction (matches the CPU reference's left/right-bound + diagonal-alignment semantics exactly). - Apply --window-left/right sliding-window bounds. - Add --dtype fp8 (FP8_E4M3 inputs, BF16 output, per-tensor descale). - Add --variable-seq-lens + --seq-lens-q/kv (GROUP mode, cumulative seq pointers). - All flags compose; mask construction is shared by the dense and per-sequence GROUP paths. Preserves develop's rank-4 LSE [B,H,Sq,1] and produces byte-identical output for all pre-existing nomask/stats bundles. Bundles (generate_golden_data.sh): - Add bottom-right causal and hd192 (D_qk=192, D_v=128) bundles for bf16+fp16 across quick (Small) and standard (Medium, Gqa) tiers. - Add a GENERATE_TIER_B section for FP8 and GROUP-mode bundles (generator-complete but not yet runnable by the CPU reference executor). Tests: 8 new quick + standard golden-reference suites for causal and hd192. Validated: 42/42 CPU golden-reference forward tests pass; 35/35 SDPA fwd plan unit tests pass. FP8/GROUP end-to-end CPU validation remains a follow-up (the executor does not yet accept descale / variable-seq-len attributes).
Store new hipDNN golden data on a dedicated remote rooted at s3://therock-dvc/rocm-libraries/hipdnn/golden-data, keeping the legacy 'storage' remote as the repo default so existing ops are unaffected. - .dvc/config: add the 'golden-data' remote (anonymous read for CI; writes use the AWS creds in the git-ignored config.local). - Tag each NEW SDPA .tensors.dvc output (causal, hd192, fp8, group — 23 pointers) with a per-output `remote: golden-data` key. A bare `dvc pull` then fetches these from golden-data and everything else from 'storage' in one pass, so CI needs NO workflow change. Pre-existing #8522 nomask/stats pointers are left untouched (they resolve to the default remote, as before). - README: document the single-remote / per-op-folder model, the per-output `remote:` key (why it removes the need for a CI flag), the SDPA regen->commit->push workflow, and the gc trade-off of a shared store. Bundles pushed to golden-data; `dvc status --cloud -r golden-data` reports in sync, and a bare `dvc pull` round-trip (simulating CI) restores them with md5 integrity.
…fwd bundles Make golden-data routing automatic and consistent for every SDPA-forward bundle: - generate_golden_data.sh: generate_bundle() now writes each bundle's .tensors.dvc pointer with a per-output `remote: golden-data` key. Regenerated and brand-new bundles are routed correctly with no manual step, so a bare `dvc pull` (and CI) always fetches them from the golden-data remote. The key survives `dvc commit`. - Add the key to the pre-existing nomask/stats pointers too (12 pointers), so ALL SDPA-forward bundles live on one remote. Only the `remote:` line is added — no tensor-data drift (md5/size unchanged); their data is already on golden-data. Verified: all 35 SDPA-fwd pointers carry the key, and a bare `dvc pull` round-trip on a nomask bundle fetches from golden-data.
❌ PR Check — Action Required
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
|
🚫 Please fix the failed policies before requesting reviews. The following policy checks failed:
The |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Warning
⛔ BLOCKED — do not merge yet. This reland will fail CI until develop's pinned TheRock ref includes the fetcher fix (TheRock #6424, merge commit
02a8930).Develop currently pins TheRock
d34f879(2026-07-06), which predates the fix (merged 2026-07-09). Merge only after a TheRock hash bump lands that contains02a8930.Gate check (safe to merge when this prints
identicalorbehind):Reland context
Relands PR #9157, which was reverted by #9182 (issue #9183).
Root cause of the revert was not in this PR's data or code. TheRock's in-tree
dvc pullreplacement (fetch_dvc_artifacts.py) read only the[core]default remote and ignored each pointer's per-outputremote:key, so blobs routed to the newgolden-dataremote were fetched from the default remote's prefix and 404'd (HeadObject 404 Not Found). Fixed upstream in TheRock #6424 (merged), whose testing verified the exactgolden-datapointers from this PR (including objecte1aab7b90300368b26ea8442792606e0) now fetch cleanly.This branch is unchanged from #9157 — same commits, rebased onto current develop. No content edits.
Summary
Extends SDPA forward golden-data coverage on top of the merged LSE/stats work (PR #8522): adds bottom-right causal, hd192, FP8, and GROUP-mode support to the generator and new golden bundles, and routes all SDPA-forward bundle data to a dedicated DVC remote so its storage and lifecycle are independent of the shared pool. Implements the bundle layout in RFC 0011.
JIRA ID : ALMIOPEN-2081
Risk Assessment
Low risk (2/5). Changes are scoped to test infrastructure — the golden-data generator, regen script, the CPU golden-reference test wiring, bundle pointers, and DVC remote config. No kernel selection, provider dispatch, public API, schema, or default-behavior changes. Pre-existing nomask/stats bundles regenerate byte-identically (only the DVC
remote:key is added). The CPU reference executor is not modified by this PR (that support already landed in #8522).ASIC Coverage
Passing PR CI is sufficient. The diff changes golden-data generation and CPU-reference test wiring only; it does not alter kernel selection, support surface, or default behavior, and adds no GPU-executed op. No dedicated multi-arch sweep is required.
Testing Summary
dvc status --cloudin sync on the golden-data remote, and a baredvc pull(simulating CI) restores bundles with md5 integrity from the per-outputremote:key.Testing Checklist
./build/bin/hipdnn_integration_tests --gtest_filter="*CpuSdpaFwdGoldenRef*"- Status: Passed (42/42)./build/bin/hipdnn_test_sdk_tests --gtest_filter="*SdpaFwd*"- Status: Passed (35/35).tensors.dvc- Status: Passeddvc pull/dvc status --cloud -r golden-data- Status: PassedTechnical Changes
generate_sdpa_fwd_golden.py, v1.0.1 → v1.1.0): add composable--causal {none,top_left,bottom_right},--window-left/right,--dtype fp8(FP8_E4M3 inputs, BF16 output, per-tensor descale), and--variable-seq-lens+--seq-lens-q/kv(GROUP mode). Mask construction is shared by the dense and per-sequence GROUP paths and matches the CPU reference's left/right-bound + diagonal-alignment semantics. Preserves the existing rank-4 LSE[B, H, Sq, 1].GENERATE_TIER_B=1(generator-complete; not yet runnable by the CPU reference executor, which does not accept descale / variable-seq-len attributes — tracked as follow-up).golden-dataremote (s3://therock-dvc/rocm-libraries/hipdnn/golden-data), keeping the defaultstorageremote unchanged. Every SDPA-forward.tensors.dvcoutput carries a per-outputremote: golden-datakey so a baredvc pull(and existing CI) fetches it with no-rflag and no workflow change. The regen script auto-stamps the key; the bundles README documents the single-remote / per-op-folder model and the gc trade-off.