[CI] Parity report: dedupe LOG-BASED FAILURES + fix empty Job ID links

[ethanwee1]

Two user-reported bugs in the parity report.

1. LOG-BASED FAILURES duplication

A test could be listed twice in the LOG-BASED FAILURES table — once as FAILED and once as CONSISTENT_FAILURE — because detect_log_failures.py emits both a per-test FAILED record and a FAILED CONSISTENTLY record for the same test, and generate_summary.py never deduped within log_failures (it only filtered against the XML FAILED TESTS table).

Fix: add a shared _select_rocm_log_failures() helper (used by both the CSV and markdown renderers) that filters out XML-failed tests and dedupes per test, preferring CONSISTENT_FAILURE — so a test is never shown as both.

Example that regressed: mi350 · default · test_sparse · TestSparseMaskedReductionsCUDA · test_future_empty_dim_masked_prod_cuda_complex128 showed as both; it now shows only CONSISTENT_FAILURE.

2. Empty Job ID columns in FAILED TESTS

download_testlogs._shorten_unzipped_dirs extracted the upstream CI job id with re.search(r'_(\d{6,})\.zip$', ...), but the unzipped shard dirs end in _<jobid> with no .zip (e.g. unzipped-...gfx950.2_83371682957). So the _<jobid> suffix was dropped, dirs became test-default-1-8, and summarize_xml's parse_xml_reports_as_dict (_(\d+)$) couldn't recover the id -> job_url was empty for every row -> the FAILED TESTS "Job ID" columns were always blank.

Fix: make .zip optional: r'_(\d{6,})(?:\.zip)?$' (still matches the older .zip form).

Validation

• Both files py_compile / parse clean.
• Dedup: ran _select_rocm_log_failures on a real log_failures CSV — the test that had both FAILED and CONSISTENT_FAILURE collapses to a single CONSISTENT_FAILURE row (0 keys left with multiple categories).
• Job id: the new regex extracts 83371682957 from the real unzipped-... dir name, and the resulting test-distributed-1-3_83371682957 parses correctly, so job_url populates. End-to-end Job ID population will be confirmed by the next parity run after merge.

Independent of #3370 / #3371 (those touch a different section of generate_summary.py).

Made with Cursor

Validation

Validated on a run of this PR branch (develop + both fixes) for the exact SHA that exhibited both bugs:

• 206283a284889f3a4f62510f8b5a8068d41121c4 · mi350: https://github.com/ROCm/pytorch/actions/runs/28190615761
  • Bug 1: test_sparse::TestSparseMaskedReductionsCUDA::test_future_empty_dim_masked_prod_cuda_complex128 is now listed once as CONSISTENT_FAILURE (was both FAILED + CONSISTENT_FAILURE); 0 keys with multiple categories across the LOG-BASED table.
  • Bug 2: FAILED TESTS table has 24 populated Job ID links; in the status CSV job_url_rocm went 0 -> 308,793 and job_url_cuda 0 -> 354,344 non-empty.

Also applied to ethanwee1/pytorch@main (fe21d4e); the fork run (https://github.com/ethanwee1/pytorch/actions/runs/28189128743) confirms Bug 1. Note: the fork's summarize_xml_testreports.py predates develop's _wf_run_id->job_url feature, so Job IDs there populate once the fork syncs that from develop (Bug 2's fix is the download_testlogs regex, which is on the fork).

[rocm-repo-management-api]

Jenkins build for b19f834b6cbbe67ff5c9de3d1c878448b3f51f80 commit finished as FAILURE
Links: Pipeline Overview / Build artifacts / Test Results