[CI] Parity: add regex-filtered HUD link to the summary#3370
Open
ethanwee1 wants to merge 1 commit into
Open
Conversation
|
Jenkins build for f2ee7d4f251ef71477ee2e035e2dc6cfbd9a596a commit finished as FAILURE |
jithunnair-amd
pushed a commit
that referenced
this pull request
Jun 26, 2026
#3375) Two user-reported bugs in the parity report. ## 1. LOG-BASED FAILURES duplication A test could be listed **twice** in the LOG-BASED FAILURES table — once as `FAILED` and once as `CONSISTENT_FAILURE` — because `detect_log_failures.py` emits both a per-test `FAILED` record and a `FAILED CONSISTENTLY` record for the same test, and `generate_summary.py` never deduped within `log_failures` (it only filtered against the XML FAILED TESTS table). **Fix:** add a shared `_select_rocm_log_failures()` helper (used by both the CSV and markdown renderers) that filters out XML-failed tests and **dedupes per test, preferring `CONSISTENT_FAILURE`** — so a test is never shown as both. Example that regressed: `mi350 · default · test_sparse · TestSparseMaskedReductionsCUDA · test_future_empty_dim_masked_prod_cuda_complex128` showed as both; it now shows only `CONSISTENT_FAILURE`. ## 2. Empty Job ID columns in FAILED TESTS `download_testlogs._shorten_unzipped_dirs` extracted the upstream CI job id with `re.search(r'_(\d{6,})\.zip$', ...)`, but the **unzipped** shard dirs end in `_<jobid>` with no `.zip` (e.g. `unzipped-...gfx950.2_83371682957`). So the `_<jobid>` suffix was dropped, dirs became `test-default-1-8`, and `summarize_xml`'s `parse_xml_reports_as_dict` (`_(\d+)$`) couldn't recover the id -> `job_url` was empty for every row -> the FAILED TESTS "Job ID" columns were always blank. **Fix:** make `.zip` optional: `r'_(\d{6,})(?:\.zip)?$'` (still matches the older `.zip` form). ## Validation - Both files `py_compile` / parse clean. - Dedup: ran `_select_rocm_log_failures` on a real `log_failures` CSV — the test that had both `FAILED` and `CONSISTENT_FAILURE` collapses to a single `CONSISTENT_FAILURE` row (0 keys left with multiple categories). - Job id: the new regex extracts `83371682957` from the real `unzipped-...` dir name, and the resulting `test-distributed-1-3_83371682957` parses correctly, so `job_url` populates. End-to-end `Job ID` population will be confirmed by the next parity run after merge. Independent of #3370 / #3371 (those touch a different section of `generate_summary.py`). Made with [Cursor](https://cursor.com) ## Validation Validated on a run of **this PR branch** (develop + both fixes) for the exact SHA that exhibited both bugs: - `206283a284889f3a4f62510f8b5a8068d41121c4 · mi350`: https://github.com/ROCm/pytorch/actions/runs/28190615761 - **Bug 1:** `test_sparse::TestSparseMaskedReductionsCUDA::test_future_empty_dim_masked_prod_cuda_complex128` is now listed **once** as `CONSISTENT_FAILURE` (was both FAILED + CONSISTENT_FAILURE); 0 keys with multiple categories across the LOG-BASED table. - **Bug 2:** FAILED TESTS table has **24 populated Job ID links**; in the status CSV `job_url_rocm` went 0 -> 308,793 and `job_url_cuda` 0 -> 354,344 non-empty. Also applied to `ethanwee1/pytorch@main` (`fe21d4e`); the fork run (https://github.com/ethanwee1/pytorch/actions/runs/28189128743) confirms Bug 1. Note: the fork's `summarize_xml_testreports.py` predates develop's `_wf_run_id`->`job_url` feature, so Job IDs there populate once the fork syncs that from `develop` (Bug 2's fix is the `download_testlogs` regex, which is on the fork).
jithunnair-amd
requested review from
jataylo,
jeffdaily,
jithunnair-amd and
pruthvistony
as code owners
The HUD link added in #3258 was lost when parity.yml was rewritten. Re-add it in generate_summary.py (next to the Commit SHA header) so it lives in the report itself (step summary + CSV artifact) and survives workflow rewrites. The link targets the upstream HUD page for the resolved commit, regex-filtered to the trunk CUDA/inductor/rocm test jobs the report is built from. Parens/pipes are percent-encoded to keep the markdown link valid; normal runs already pass set1_name=rocm/set2_name=cuda. Verified end-to-end against a real mi350 status CSV.
ethanwee1
force-pushed
the
ethanwee/parity-hud-link
branch
from
f2ee7d4 to
617c2ad
Compare
|
Jenkins build for 617c2ad85d7f449d0ed48f7e219bc617009ab993 commit finished as ABORTED Detected error during Pytorch building: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Re-adds a clickable HUD link to the parity report summary, pointing at the upstream HUD page for the report's resolved commit, regex-filtered to the trunk CUDA / inductor / rocm test jobs the report is built from:
https://hud.pytorch.org/hud/pytorch/pytorch/<sha>/1?per_page=50&name_filter=(trunk.*cuda|inductor|rocm).*test.*(default|distributed|inductor),&useRegexFilter=trueWhy
The HUD link originally added in #3258 was dropped when
parity.ymlwas rewritten (during the #3278/#3231 work). This re-adds it ingenerate_summary.py(next to theCommit SHAheader) instead of the workflow, so it lives in the report itself — visible in both the GitHub Actions step summary and the CSV artifact's_summary.md— and survives future workflow rewrites.Notes
%28/%29) and pipes (%7C) are percent-encoded so the markdown link stays valid; the URL decodes to exactly(trunk.*cuda|inductor|rocm).*test.*(default|distributed|inductor),.Test plan
py_compileclean.generate_summary.pyon a real mi350 status CSV → summary renders**HUD: [parity jobs for this commit](…)**; decodedname_filtermatches the intended regex,useRegexFilter=true,per_page=50.Made with Cursor
Rebased onto current develop + validated
This PR was rebased onto current
develop(it was previously opened against an old lineage and showed unrelated files). Validation parity run on the clean SHAba9910c6(mi300): https://github.com/ethanwee1/pytorch/actions/runs/28884934635