[CI] Parity: count a disagreement only when CUDA PASSED#3371
Open
ethanwee1 wants to merge 1 commit into
Open
Conversation
ethanwee1
added a commit
to ethanwee1/pytorch
that referenced
this pull request
Jun 24, 2026
Count a ROCm SKIPPED/MISSED test as a disagreement only when CUDA actually PASSED it (s2 == PASSED), excluding parametrization variants CUDA never runs. AGREE% becomes ~99%, matching the triage spreadsheet. Mirrors ROCm#3371.
|
Jenkins build for 7f5f023dd5f5dc4cfc6480a335e9392b33d762ec commit finished as FAILURE |
jithunnair-amd
pushed a commit
that referenced
this pull request
Jun 26, 2026
#3375) Two user-reported bugs in the parity report. ## 1. LOG-BASED FAILURES duplication A test could be listed **twice** in the LOG-BASED FAILURES table — once as `FAILED` and once as `CONSISTENT_FAILURE` — because `detect_log_failures.py` emits both a per-test `FAILED` record and a `FAILED CONSISTENTLY` record for the same test, and `generate_summary.py` never deduped within `log_failures` (it only filtered against the XML FAILED TESTS table). **Fix:** add a shared `_select_rocm_log_failures()` helper (used by both the CSV and markdown renderers) that filters out XML-failed tests and **dedupes per test, preferring `CONSISTENT_FAILURE`** — so a test is never shown as both. Example that regressed: `mi350 · default · test_sparse · TestSparseMaskedReductionsCUDA · test_future_empty_dim_masked_prod_cuda_complex128` showed as both; it now shows only `CONSISTENT_FAILURE`. ## 2. Empty Job ID columns in FAILED TESTS `download_testlogs._shorten_unzipped_dirs` extracted the upstream CI job id with `re.search(r'_(\d{6,})\.zip$', ...)`, but the **unzipped** shard dirs end in `_<jobid>` with no `.zip` (e.g. `unzipped-...gfx950.2_83371682957`). So the `_<jobid>` suffix was dropped, dirs became `test-default-1-8`, and `summarize_xml`'s `parse_xml_reports_as_dict` (`_(\d+)$`) couldn't recover the id -> `job_url` was empty for every row -> the FAILED TESTS "Job ID" columns were always blank. **Fix:** make `.zip` optional: `r'_(\d{6,})(?:\.zip)?$'` (still matches the older `.zip` form). ## Validation - Both files `py_compile` / parse clean. - Dedup: ran `_select_rocm_log_failures` on a real `log_failures` CSV — the test that had both `FAILED` and `CONSISTENT_FAILURE` collapses to a single `CONSISTENT_FAILURE` row (0 keys left with multiple categories). - Job id: the new regex extracts `83371682957` from the real `unzipped-...` dir name, and the resulting `test-distributed-1-3_83371682957` parses correctly, so `job_url` populates. End-to-end `Job ID` population will be confirmed by the next parity run after merge. Independent of #3370 / #3371 (those touch a different section of `generate_summary.py`). Made with [Cursor](https://cursor.com) ## Validation Validated on a run of **this PR branch** (develop + both fixes) for the exact SHA that exhibited both bugs: - `206283a284889f3a4f62510f8b5a8068d41121c4 · mi350`: https://github.com/ROCm/pytorch/actions/runs/28190615761 - **Bug 1:** `test_sparse::TestSparseMaskedReductionsCUDA::test_future_empty_dim_masked_prod_cuda_complex128` is now listed **once** as `CONSISTENT_FAILURE` (was both FAILED + CONSISTENT_FAILURE); 0 keys with multiple categories across the LOG-BASED table. - **Bug 2:** FAILED TESTS table has **24 populated Job ID links**; in the status CSV `job_url_rocm` went 0 -> 308,793 and `job_url_cuda` 0 -> 354,344 non-empty. Also applied to `ethanwee1/pytorch@main` (`fe21d4e`); the fork run (https://github.com/ethanwee1/pytorch/actions/runs/28189128743) confirms Bug 1. Note: the fork's `summarize_xml_testreports.py` predates develop's `_wf_run_id`->`job_url` feature, so Job IDs there populate once the fork syncs that from `develop` (Bug 2's fix is the `download_testlogs` regex, which is on the fork).
The DISAGREE/AGREE metric counted a ROCm SKIPPED or MISSED test as a
disagreement whenever CUDA merely did not SKIP it - which includes the
large set of attention-backend parametrization variants CUDA never even
enumerates (CUDA MISSED). Those are not real ROCm-vs-CUDA coverage gaps and
inflated DISAGREE to ~3% (AGREE ~97%).
Count a disagreement only when CUDA actually PASSED the test (s2 == PASSED)
in both compute_test_config_stats and compute_overall_stats, and relabel the
two line items accordingly ('PASSED on <set2>'). This matches the triage
spreadsheet's adjusted definition; AGREE% becomes ~99%.
7f5f023 to
9d939b6
Compare
|
Jenkins build for 9d939b6f08e9c9f18f3896fa404afc7974d945d2 commit finished as ABORTED Detected error during Pytorch building: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Changes the parity DISAGREE/AGREE metric to count a ROCm
SKIPPED/MISSEDtest as a disagreement only when CUDA actuallyPASSEDit (status_cuda == PASSED), instead of "CUDA didn't SKIP it" (!= SKIPPED).Why
The old definition counted as disagreements the large set of attention-backend parametrization variants (e.g.
test_transformersFlash/CK/CUTLASS cases) that CUDA never even enumerates (CUDA =MISSED). Those aren't real ROCm-vs-CUDA coverage gaps — they just inflated DISAGREE to ~3% (AGREE ~97%). Counting only "CUDA passed, ROCm didn't" reflects the actionable gap and matches the triage spreadsheet's adjusted definition.Effect (mi350, real data)
SKIPPED+MISSED9,002 → 2,629, DISAGREE 3.0% → 0.88%SKIPPED (on rocm, PASSED on cuda)/MISSED (on rocm, PASSED on cuda)for accuracy.Scope
compute_test_config_stats+compute_overall_statsingenerate_summary.py. The dashboard's parity collector (ROCm/AI-Frameworks-Dashboard#49) is being updated with the same definition so the dashboard matches this report.Test plan
py_compileclean.Made with Cursor
Rebased onto current develop + validated
This PR was rebased onto current
develop(it was previously opened against an old lineage and showed unrelated files). Validation parity run on the clean SHAba9910c6(mi300): https://github.com/ethanwee1/pytorch/actions/runs/28884929807