Consolidate reports across retry attempts

[Amaury Levé (Evangelink)]

--retry-failed-tests currently publishes only the final attempt's narrowed report at the top level, so report consumers can miss tests that passed earlier in the run. This change consolidates retry-aware formats while preserving every physical attempt for diagnostics.

Changes

• Relay typed report artifacts from each child attempt to the retry orchestrator through the existing retry pipe.
• Add an explicit retry-attempt post-processing mode so processors must opt in and receive inputs in attempt order.
• Collapse CTRF attempts into one logical result per test, including retryAttempts, retries, flaky, and corrected summary counters.
• Collapse JUnit attempts to each logical test's final outcome, which keeps CI totals and gating accurate without inventing non-portable retry elements.
• Keep all per-attempt artifacts under Retries/; TRX and HTML intentionally retain final-attempt top-level semantics.

Testing

• Full repository build and package generation
• Microsoft.Testing.Extensions unit tests
• Artifact post-processing unit tests
• CTRF, JUnit, and TRX retry acceptance scenarios

Fixes #10293

[Copilot]

Pull request overview

Consolidates retry-aware CTRF and JUnit reports while preserving every physical attempt under Retries/.

Changes:

• Relays typed artifacts and adds retry-specific post-processing modes.
• Collapses CTRF and JUnit attempts into logical test results.
• Preserves final-attempt semantics for TRX and HTML.

Show a summary per file

File	Description
test/UnitTests/Microsoft.Testing.Platform.UnitTests/Extensions/ArtifactPostProcessing/ArtifactPostProcessingTests.cs	Updates processor stub capabilities.
test/UnitTests/Microsoft.Testing.Extensions.UnitTests/TrxArtifactPostProcessorTests.cs	Verifies TRX modes.
test/UnitTests/Microsoft.Testing.Extensions.UnitTests/JUnitReportMergerTests.cs	Tests retry collapsing.
test/UnitTests/Microsoft.Testing.Extensions.UnitTests/JUnitArtifactPostProcessorTests.cs	Verifies JUnit modes.
test/UnitTests/Microsoft.Testing.Extensions.UnitTests/HtmlArtifactPostProcessorTests.cs	Verifies HTML modes.
test/UnitTests/Microsoft.Testing.Extensions.UnitTests/CtrfArtifactPostProcessorTests.cs	Tests CTRF retry processing.
test/IntegrationTests/MSTest.Acceptance.IntegrationTests/JUnitReportTests.cs	Validates consolidated JUnit output.
test/IntegrationTests/Microsoft.Testing.Platform.Acceptance.IntegrationTests/RetryFailedTestsTests.cs	Validates retry artifact layouts and CTRF.
test/IntegrationTests/Microsoft.Testing.Platform.Acceptance.IntegrationTests/DotnetTestPipe/DotnetTestPipeArtifactPostProcessingTests.cs	Updates sample processor modes.
src/Platform/Microsoft.Testing.Platform/PublicAPI/PublicAPI.Unshipped.txt	Tracks the new public API.
src/Platform/Microsoft.Testing.Platform/Extensions/ArtifactPostProcessing/IArtifactPostProcessor.cs	Defines post-processing modes.
src/Platform/Microsoft.Testing.Platform/Extensions/ArtifactPostProcessing/ArtifactPostProcessingManager.cs	Validates supported modes.
src/Platform/Microsoft.Testing.Platform/Extensions/ArtifactPostProcessing/ArtifactPostProcessingHandshakeProperties.cs	Restricts handshakes to module merging.
src/Platform/Microsoft.Testing.Platform/Extensions/ArtifactPostProcessing/ArtifactPostProcessingDispatcherTool.cs	Filters processors by mode.
src/Platform/Microsoft.Testing.Extensions.TrxReport/TrxArtifactPostProcessor.cs	Keeps TRX module-only.
src/Platform/Microsoft.Testing.Extensions.TrxReport/InternalAPI/InternalAPI.Unshipped.txt	Tracks TRX API changes.
src/Platform/Microsoft.Testing.Extensions.Retry/Serializers/ArtifactRequest.cs	Adds artifact IPC serialization.
src/Platform/Microsoft.Testing.Extensions.Retry/RetrySummaryReporter.cs	Preserves and publishes final artifacts.
src/Platform/Microsoft.Testing.Extensions.Retry/RetryOrchestrator.cs	Coordinates retry report processing.
src/Platform/Microsoft.Testing.Extensions.Retry/RetryLifecycleCallbacks.cs	Registers artifact serialization.
src/Platform/Microsoft.Testing.Extensions.Retry/RetryFailedTestsPipeServer.cs	Collects child artifacts.
src/Platform/Microsoft.Testing.Extensions.Retry/RetryDataConsumer.cs	Relays session artifacts.
src/Platform/Microsoft.Testing.Extensions.Retry/RetryArtifactProcessor.cs	Processes ordered attempt artifacts.
src/Platform/Microsoft.Testing.Extensions.Retry/Resources/xlf/ExtensionResources.zh-Hant.xlf	Updates localized resources.
src/Platform/Microsoft.Testing.Extensions.Retry/Resources/xlf/ExtensionResources.zh-Hans.xlf	Updates localized resources.
src/Platform/Microsoft.Testing.Extensions.Retry/Resources/xlf/ExtensionResources.tr.xlf	Updates localized resources.
src/Platform/Microsoft.Testing.Extensions.Retry/Resources/xlf/ExtensionResources.ru.xlf	Updates localized resources.
src/Platform/Microsoft.Testing.Extensions.Retry/Resources/xlf/ExtensionResources.pt-BR.xlf	Updates localized resources.
src/Platform/Microsoft.Testing.Extensions.Retry/Resources/xlf/ExtensionResources.pl.xlf	Updates localized resources.
src/Platform/Microsoft.Testing.Extensions.Retry/Resources/xlf/ExtensionResources.ko.xlf	Updates localized resources.
src/Platform/Microsoft.Testing.Extensions.Retry/Resources/xlf/ExtensionResources.ja.xlf	Updates localized resources.
src/Platform/Microsoft.Testing.Extensions.Retry/Resources/xlf/ExtensionResources.it.xlf	Updates localized resources.
src/Platform/Microsoft.Testing.Extensions.Retry/Resources/xlf/ExtensionResources.fr.xlf	Updates localized resources.
src/Platform/Microsoft.Testing.Extensions.Retry/Resources/xlf/ExtensionResources.es.xlf	Updates localized resources.
src/Platform/Microsoft.Testing.Extensions.Retry/Resources/xlf/ExtensionResources.de.xlf	Updates localized resources.
src/Platform/Microsoft.Testing.Extensions.Retry/Resources/xlf/ExtensionResources.cs.xlf	Updates localized resources.
src/Platform/Microsoft.Testing.Extensions.Retry/Resources/ExtensionResources.resx	Adds publication and failure messages.
src/Platform/Microsoft.Testing.Extensions.Retry/InternalAPI/InternalAPI.Unshipped.txt	Tracks retry internals.
src/Platform/Microsoft.Testing.Extensions.JUnitReport/JUnitReportMerger.cs	Collapses JUnit retry outcomes.
src/Platform/Microsoft.Testing.Extensions.JUnitReport/JUnitArtifactPostProcessor.cs	Enables retry-aware JUnit merging.
src/Platform/Microsoft.Testing.Extensions.JUnitReport/InternalAPI/InternalAPI.Unshipped.txt	Tracks JUnit internals.
src/Platform/Microsoft.Testing.Extensions.HtmlReport/InternalAPI/InternalAPI.Unshipped.txt	Tracks HTML mode capability.
src/Platform/Microsoft.Testing.Extensions.HtmlReport/HtmlArtifactPostProcessor.cs	Keeps HTML module-only.
src/Platform/Microsoft.Testing.Extensions.CtrfReport/InternalAPI/InternalAPI.Unshipped.txt	Tracks CTRF internals.
src/Platform/Microsoft.Testing.Extensions.CtrfReport/CtrfReportMerger.FileMerge.cs	Adds mode-aware file merging.
src/Platform/Microsoft.Testing.Extensions.CtrfReport/CtrfArtifactPostProcessor.cs	Enables retry-aware CTRF collapsing.
docs/RFCs/018-Artifact-Post-Processing.md	Documents processing modes.
docs/RFCs/016-JUnit-Report.md	Documents consolidated retry behavior.

Review details

Tip

Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

• Files reviewed: 48/48 changed files
• Comments generated: 2
• Review effort level: Balanced

[Copilot]

Review details

Suppressed comments (2)

src/Platform/Microsoft.Testing.Platform/Extensions/ArtifactPostProcessing/ArtifactPostProcessingManager.cs:40

• The new capability validation has no tests for any of its rejection cases (empty modes, duplicate modes, or an undefined enum value), while the existing kind/extension validation is covered case-by-case in ArtifactPostProcessingTests. Please add manager tests for these three invalid declarations so this opt-in boundary cannot regress silently.
  src/Platform/Microsoft.Testing.Extensions.Retry/RetryArtifactProcessor.cs:63
• This fail-closed guard is central to avoiding a valid-looking but incomplete retry report, but the added tests cover only complete happy paths. Add focused tests for a missing attempt and multiple artifacts from one attempt, asserting that no replacement is published (and ideally that the processor is not invoked).

            if (artifactsByAttempt.Length != attemptCount
                || artifactsByAttempt.Any(group => group.Count() != 1))
            {
                continue;

• Files reviewed: 49/49 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Copilot]

Review details

Suppressed comments (1)

src/Platform/Microsoft.Testing.Extensions.JUnitReport/JUnitReportMerger.cs:216

• The retry suite keeps the first attempt's full <testsuite> template. JUnitXmlWriter puts the child process's exit-code in that template, so a fail-then-pass run can publish failures="0" while still carrying the first attempt's failing exit-code="2" (the acceptance expectations confirm 2 for failed runs and 0 for successful runs). Recompute attempt-scoped suite metadata from the logical/final run instead of retaining it from the first attempt; preserve the earliest timestamp separately if needed.

                string suiteIdentity = BuildSuiteIdentity(suite);
                if (!suiteIndices.TryGetValue(suiteIdentity, out int suiteIndex))
                {
                    suiteIndex = suites.Count;
                    suiteIndices.Add(suiteIdentity, suiteIndex);
                    suites.Add(new RetrySuite(new XElement(suite)));

• Files reviewed: 49/49 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Copilot]

Review details

• Files reviewed: 49/49 changed files
• Comments generated: 1
• Review effort level: Balanced

[Copilot]

Review details

Suppressed comments (2)

test/IntegrationTests/MSTest.Acceptance.IntegrationTests/JUnitReportTests.cs:206

• This has the same ancestor-path ambiguity as the attempt-1 lookup: any workspace component named 2 makes unrelated report paths match and causes Single to fail. Restrict the comparison to the report's parent directory.
  test/IntegrationTests/MSTest.Acceptance.IntegrationTests/JUnitReportTests.cs:202
• This matches any .../1/... component in the full path, not specifically the retry-attempt directory. On agents whose workspace already contains a 1 component (for example Azure Pipelines' _work/1/...), both report paths match and Single throws. Compare the report's immediate parent directory instead.

This issue also appears on line 206 of the same file.

• Files reviewed: 49/49 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Copilot]

Review details

• Files reviewed: 49/49 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Copilot]

Review details

Suppressed comments (1)

src/Platform/Microsoft.Testing.Platform/Extensions/ArtifactPostProcessing/ArtifactPostProcessingHandshakeProperties.cs:29

• The new TestModules filter is not applied to requiredKinds below. If a RetryAttempts-only IArtifactPostProcessorRequiresPostProcessing is registered alongside any module processor, its kind is advertised as required in the normal SDK handshake; the producer then suppresses standalone output even though the dispatcher excludes that processor for TestModules. Filter required processors through SupportsTestModuleMerging as well.

• Files reviewed: 51/51 changed files
• Comments generated: 1
• Review effort level: Balanced

[Copilot]

Review details

• Files reviewed: 51/51 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Copilot]

Review details

• Files reviewed: 52/52 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[github-actions]

🧵 Parallel-safety audit — PR #10542

Parallelization — one row per test assembly audited:

Test assembly	Scope	Workers	Analyzer coverage
Microsoft.Testing.Extensions.UnitTests	MethodLevel	CPU count	coverable once the parallel-safety analyzers ship (attribute)
Microsoft.Testing.Platform.UnitTests	MethodLevel	CPU count	coverable once the parallel-safety analyzers ship (attribute)
MSTest.Acceptance.IntegrationTests	MethodLevel	CPU count	coverable once the parallel-safety analyzers ship (attribute)
Microsoft.Testing.Platform.Acceptance.IntegrationTests	MethodLevel	CPU count	coverable once the parallel-safety analyzers ship (attribute)

All four assemblies opt in via [assembly: Parallelize(Scope = ExecutionScope.MethodLevel, Workers = 0)] in their Program.cs, with no .runsettings or testconfig.json override found — the opt-in is fully analyzer-visible once MSTEST0074–0077 ship (today only MSTEST0073 is on main).

Findings: A (global-state) 0 · B (paths) 0 · C (declaration) 0 · D (over-serialization) 0 — by severity: Critical 0 · High 0 · Warning 0 · Info 0.

Analysis performed:

I reviewed all 11 changed test files (unit tests for CTRF/JUnit/HTML/TRX artifact post-processors, RetryTests.cs, and acceptance tests JUnitReportTests.cs, RetryFailedTestsTests.cs, DotnetTestPipeArtifactPostProcessingTests.cs). No changed test method, lifecycle member, or [ResourceLock]/[DoNotParallelize]/[Parallelize] declaration in this PR touches process-global state:

• New/changed tests in CtrfArtifactPostProcessorTests.cs and JUnitReportMergerTests.cs create temp directories via Path.Combine(Path.GetTempPath(), $"...-{Guid.NewGuid():N}") — per-test unique paths, no collision.
• RetryTests.cs new tests operate entirely on mocked IFileSystem/IOutputDevice/ILogger (Moq) — no real I/O, no shared state.
• AppInsightsProviderTests.cs change adds a ManualResetEventSlim used only within the single test method — purely local, actually improves determinism versus a fixed Task.Delay.
• The acceptance test changes (JUnitReportTests.cs, RetryFailedTestsTests.cs, DotnetTestPipeArtifactPostProcessingTests.cs) only change assertions on already-isolated per-test resultDirectory/reportPath values (each built from Guid.NewGuid()), and the relevant classes are already [DoNotParallelize] (JUnitReportMTPRetryExtensionTests, JUnitReportRetryAttributeTests) or use per-test unique directories (RetryFailedTestsTests, DotnetTestPipeArtifactPostProcessingTests).
• No Environment.SetEnvironmentVariable, CWD mutation, culture mutation, console redirection, or mutable-static writes were added or removed.
• No .runsettings/testconfig.json/Directory.Build.props/.csproj parallelization settings changed, and no .cs file's [assembly: Parallelize]/[assembly: DoNotParallelize] attribute changed.

Nothing to flag for parallel-safety in this PR.

_{Advisory only — heuristic, non-blocking. Re-run with /parallel-audit. This audit answers "is it parallel-safe?"; for testability, smells, or flakiness see the detect-static-dependencies / test-smell-detection / test-anti-patterns analyses.}

🤖 Automated content by GitHub Copilot. Generated by the Parallel-safety audit on PR (on open / sync) workflow. · auto · 119.5 AIC · ⌖ 3.05 AIC · ⊞ 24.8K · [◷]( · ◷)

[github-actions]

🧪 Expert test review — PR #10542

Grade	Test	Mutation	Notes	How to improve
A (90–100)	new CtrfArtifactPostProcessorTests. ProcessAsync_ ForRetryAttempts_ CollapsesToFinalOutcome	4/4 killed	Asserts summary counts, final status, retry count, and prior-attempt status on the collapsed report.	—
A (90–100)	new JUnitReportMergerTests. MergeRetryAttempts_ KeepsOneFinalOutcomePerLogicalTest	4/4 killed	Verifies collapsed counters, time aggregation, ordering, and that the final passing outcome clears the failure element.	—
A (90–100)	new JUnitReportMergerTests. MergeRetryAttempts_ UsesStableIdentityWhenDuplicateSuffixDisappears	3/3 killed	Directly targets the stable-identity regression called out in review: a renamed row across attempts still replaces its predecessor.	—
A (90–100)	new JUnitReportMergerTests. MergeRetryAttempts_ UsesFinalAttemptSuiteProperties	2/2 killed	Confirms suite-level properties (exit-code) come from the final attempt, not the first.	—
A (90–100)	new JUnitReportMergerTests. MergeRetryAttempts_ PreservesSameIdentityOccurrencesWithinAnAttempt	3/3 killed	Covers the follow-up finding: duplicate same-identity rows within one physical attempt must not collapse into each other.	—
A (90–100)	new RetryTests. SnapshotAttemptArtifacts_ CopiesExternalArtifactAndPreservesDestination	4/4 killed	Verifies snapshot path, destination preservation, attempt number, and the exact mocked file-system calls.	—
A (90–100)	new RetryTests. PublishExternalArtifacts_ UsesMergedReplacement	2/2 killed	Confirms the replacement path (not the snapshot) is copied to the original external destination.	—
A (90–100)	new RetryTests. MoveArtifactsAsync_ NormalizesRelativePathsBeforeApplyingReplacement	2/2 killed	Targets the exact reviewer-flagged bug: a relative path returned by GetFiles must still match an absolute replacement key.	—
A (90–100)	new ArtifactPostProcessingTests. HandshakeProperties_ ExcludesRetryOnlyRequiredKinds	2/2 killed	Checks both a positive (module kind present) and negative (no required-kinds key) handshake outcome for a retry-only processor.	—
A (90–100)	mod AppInsightsProviderTests. FlushFailure_ IsLogged	1/1 killed	Adds a dedicated ManualResetEventSlim gate so the test deterministically waits for the flush-failure log instead of racing dispose.	—
A (90–100)	mod RetryFailedTestsTests. RetryFailedTests_ CtrfReports_ ShareRunIdButNotReportId	3/3 killed	Extended to assert the new consolidated top-level CTRF report's summary, flaky status, and retry-attempt history.	—
A (90–100)	new RetryFailedTestsTests. RetryFailedTests_ CtrfAbsoluteFileName_ PublishesConsolidatedReport	3/3 killed	Regression for the reviewer-flagged bug where an absolute --report-ctrf-filename bypassed the consolidated-report replacement.	—
A (90–100)	mod JUnitReportTests. JUnitReport_ WithMTPRetryFailedTestsExtension_ PublishesConsolidatedReport	3/3 killed	Renamed and rewritten to assert per-attempt XML under Retries/ plus a genuinely consolidated top-level report with both test names.	—

Summary: This PR's test changes are consistently strong. Every new or
materially modified test targets a specific defect raised in prior review
threads (stable retry identity across renamed rows, same-identity duplicates
within one attempt, relative-vs-absolute path normalization for artifact
replacement, and absolute --report-ctrf-filename handling), and each
assertion set would fail if the corresponding fix were reverted. No
correctness, reliability, or convention issues were found; no inline
suggestions were warranted.

_{This advisory comment was generated automatically. Grades are heuristic
and informational — they do not block merging. Suggestions on the Files
changed tab can be applied with one click. Re-run with
/review-tests.}

🤖 Automated content by GitHub Copilot. Generated by the Test Reviewer on PR (on open / sync) workflow. · auto · 157.8 AIC · ⌖ 2.74 AIC · ⊞ 16.9K · [◷]( · ◷)