Consolidate reports across retry attempts

docs/RFCs/016-JUnit-Report.md

@@ -216,7 +216,7 @@ When the same `(classname, name)` pair is emitted more than once (parameterized
 Different retry mechanisms publish attempts to MTP differently, and the engine reflects that faithfully:
 
 - **MSTest `[Retry]` attribute** — The MSTest adapter retries in-process and publishes **every** attempt as a `TestNodeUpdateMessage` under the same test node uid, tagged with `RetryAttemptProperty`. The JUnit report generator drops the attempts marked `IsSuperseded` (JUnit has no notion of attempts, so keeping them would inflate the suite totals), so the report still contains a single `<testcase>` row per logical test with its eventual outcome. No per-attempt disambiguation is applied.
-- **`Microsoft.Testing.Extensions.Retry` (MTP-level orchestrator, `--retry-failed-tests`)** — The orchestrator re-runs the entire test-host child process on failure, and each re-run is filtered down to the tests that failed in the previous attempt. Every attempt is a separate process that writes **its own JUnit XML file** into that attempt's own result directory, `<results>/Retries/<id>/<n>/`; when the run ends, the orchestrator *moves* the **last** attempt's files up into `<results>/`, leaving that attempt's own directory under `Retries/` empty. Nothing is merged, so the JUnit file at the top of the results directory describes only the last attempt. When a retry was needed that is just the subset which had failed in the attempt before it — whatever those tests then did, so a run rescued by retry ends with a top-level file that records **no failures** (a previously-failing test may equally have been skipped on its final attempt: skipped outcomes are not failures, and under the default `allow-skipped` zero-test policy such an attempt still exits successfully). When no retry was needed, the first attempt is also the last one, so the top-level file describes the whole suite as usual. This follows from the orchestrator's "move the last attempt's artifacts" rule rather than from report-engine behavior: no special handling exists inside the report engine. Reconstructing the whole run therefore means reading the per-attempt files left under `Retries/` (attempts 1..N-1) *together with* the top-level file (attempt N). Whether the top-level file should instead describe the whole logical run is tracked by [#10293](https://github.com/microsoft/testfx/issues/10293); note that JUnit XML has no portable way to express "this test failed then passed" (see the dialect table above and the `<rerunFailure>` note below), so any such consolidation has to pick between one row per logical test (final outcome, history lost) and one row per attempt (history kept, failure counts inflated).
+- **`Microsoft.Testing.Extensions.Retry` (MTP-level orchestrator, `--retry-failed-tests`)** — The orchestrator re-runs the entire test-host child process on failure, and each re-run is filtered down to the tests that failed in the previous attempt. Every attempt keeps its own immutable JUnit XML file under `<results>/Retries/<id>/<n>/`. After the final attempt, the JUnit post-processor writes the top-level report as one row per logical test using each test's final outcome. JUnit XML has no portable retry vocabulary, so earlier failure history remains available only in the per-attempt files; the consolidated top-level report deliberately favors correct CI gating and suite totals over inflating failures with superseded attempts.
 - **Per-attempt `TestNodeUpdateMessage`s that are *not* tagged with `RetryAttemptProperty`** (e.g. some 3rd-party test frameworks) — A producer that tags its attempts has its superseded ones filtered out above, so this case covers only *unattributed* duplicates. The Jenkins/Surefire flavor has no native rerun element, but consumers require `(classname, name)` pairs to be unique within a suite (see [Duplicate test identities](#duplicate-test-identities) above). When the engine sees two or more such nodes with the same `(classname, name)` pair within one report it:
   - Preserves every attempt as its own `<testcase>` row (never drops history) so flaky-test dashboards can compute pass-rate over the run.
   - Disambiguates each row by appending `[attempt 1]`, `[attempt 2]`, … to `name` (preceded by a single space), so portable consumers see distinct entries.

docs/RFCs/018-Artifact-Post-Processing.md

@@ -144,6 +144,8 @@ public interface IArtifactPostProcessor : IExtension
 {
     // Inherited from IExtension: Uid, Version, DisplayName, Description, Task<bool> IsEnabledAsync().
 
+    IReadOnlyList<ArtifactPostProcessingMode> SupportedModes { get; }
+
     /// <summary>
     /// Whether this processor can consume the incomplete set of complete artifacts observed
     /// before --maximum-failed-tests or --timeout truncated a run.
@@ -186,8 +188,16 @@ public sealed class ArtifactPostProcessingContext
     public ArtifactPostProcessingContext(ArtifactPostProcessingTruncationReason truncationReason);
     public ArtifactPostProcessingContext(
         ArtifactPostProcessingTruncationReason truncationReason,
+        ArtifactPostProcessingMode mode);
+    public ArtifactPostProcessingContext(
+        ArtifactPostProcessingTruncationReason truncationReason,
+        ArtifactPostProcessingRunSummary? runSummary);
+    public ArtifactPostProcessingContext(
+        ArtifactPostProcessingTruncationReason truncationReason,
+        ArtifactPostProcessingMode mode,
         ArtifactPostProcessingRunSummary? runSummary);
     public bool IsTruncated { get; }
+    public ArtifactPostProcessingMode Mode { get; }
     public ArtifactPostProcessingTruncationReason TruncationReason { get; }
     public ArtifactPostProcessingRunSummary? RunSummary { get; }
 }
@@ -211,6 +221,12 @@ public sealed class ArtifactPostProcessingRunSummary
     public int TestModuleCount { get; }
 }
 
+public enum ArtifactPostProcessingMode
+{
+    TestModules,
+    RetryAttempts,
+}
+
 public enum ArtifactPostProcessingTruncationReason
 {
     None,
@@ -249,6 +265,7 @@ public sealed record ProcessedArtifact(
 - **Returning `null`** means "I looked but there's nothing to do" (e.g. < 2 inputs). The orchestrator then leaves the originals visible.
 - **Idempotent / deterministic.** The contract is intentionally pure-functional from `inputs` -> `output`. Implementations must not stash state across calls.
 - **Truncated runs are fail-closed.** A processor must explicitly return `true` from `SupportsTruncatedRuns` before the SDK or dispatcher offers it artifacts from a run truncated by `--maximum-failed-tests` or `--timeout`. Opting in means the processor accepts an incomplete *set* of artifacts and will clearly represent that state in its output; it does not mean partially written or malformed files are valid inputs.
+- **Retry merging is opt-in.** `RetryAttempts` inputs are ordered from the initial execution to the final attempt and overlap by definition. A processor must advertise that mode and deliberately collapse repeated logical tests; processors that only concatenate disjoint module results advertise `TestModules` alone.
 - **Required post-processing is capability-gated.** Internal coordination formats such as CI summary fragments implement `IArtifactPostProcessorRequiresPostProcessing`. Their kinds are advertised separately so a supporting orchestrator invokes the processor even for one input. Producers defer standalone output only after the SDK handshake response confirms support; older SDKs continue the existing direct behavior.
 - **`IExtension` base** gives us `Uid`, `Version`, `DisplayName`, `Description`, and `IsEnabledAsync()` for free, plus integration with the existing extension manifest tooling. (`IsEnabledAsync()` is therefore *not* a bespoke member of this contract; the sketch in Appendix A.1 implements the inherited member.)
 
@@ -663,6 +680,9 @@ One might avoid a relaunch by keeping an elected host process alive at end-of-ru
 ```csharp
 internal sealed class TrxArtifactPostProcessor : IArtifactPostProcessor
 {
+    public IReadOnlyList<ArtifactPostProcessingMode> SupportedModes
+        => [ArtifactPostProcessingMode.TestModules];
+
     public bool SupportsTruncatedRuns => false;
 
     public string Uid => "Microsoft.Testing.Extensions.TrxReport.PostProcessor";

src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/AzureDevOpsSummaryArtifactPostProcessor.cs

@@ -26,6 +26,7 @@ internal sealed class AzureDevOpsSummaryArtifactPostProcessor(
     internal const string ProviderSlug = "azure-devops";
 
     private static readonly string[] SupportedArtifactKinds = [FragmentArtifactKind];
+    private static readonly ArtifactPostProcessingMode[] SupportedPostProcessingModes = [ArtifactPostProcessingMode.TestModules];
     private readonly bool _isEnabled =
         (commandLineOptions.IsOptionSet(AzureDevOpsCommandLineOptions.AzureDevOpsSummary)
             && AzureDevOpsConstants.IsRunningInAzureDevOps(environment))
@@ -39,6 +40,8 @@ internal sealed class AzureDevOpsSummaryArtifactPostProcessor(
 
     public string Description => AzureDevOpsResources.Description;
 
+    public IReadOnlyList<ArtifactPostProcessingMode> SupportedModes => SupportedPostProcessingModes;
+
     public bool SupportsTruncatedRuns => true;
 
     public IReadOnlyList<string> SupportedKinds => SupportedArtifactKinds;

src/Platform/Microsoft.Testing.Extensions.CtrfReport/CtrfArtifactPostProcessor.cs

@@ -12,6 +12,8 @@ internal sealed class CtrfArtifactPostProcessor : IArtifactPostProcessor
 
     private static readonly string[] SupportedArtifactKinds = [CtrfReportGenerator.CtrfArtifactKind];
     private static readonly string[] NoSupportedExtensions = [];
+    private static readonly ArtifactPostProcessingMode[] SupportedPostProcessingModes =
+        [ArtifactPostProcessingMode.TestModules, ArtifactPostProcessingMode.RetryAttempts];
 
     public string Uid => "Microsoft.Testing.Extensions.CtrfReport.PostProcessor";
 
@@ -21,6 +23,8 @@ internal sealed class CtrfArtifactPostProcessor : IArtifactPostProcessor
 
     public string Description => ExtensionResources.CtrfArtifactPostProcessorDescription;
 
+    public IReadOnlyList<ArtifactPostProcessingMode> SupportedModes => SupportedPostProcessingModes;
+
     // CTRF has no marker that lets this merger label an incomplete input set as a partial run.
     public bool SupportsTruncatedRuns => false;
 
@@ -42,15 +46,18 @@ internal sealed class CtrfArtifactPostProcessor : IArtifactPostProcessor
             return null;
         }
 
-        InputArtifact[] orderedInputs =
-        [
-            .. inputs
-                .OrderBy(input => Path.GetFullPath(input.Path), StringComparer.Ordinal)
-                .ThenBy(input => input.ExecutionId, StringComparer.Ordinal),
-        ];
+        InputArtifact[] orderedInputs = context.Mode == ArtifactPostProcessingMode.RetryAttempts
+            ? [.. inputs]
+            :
+            [
+                .. inputs
+                    .OrderBy(input => Path.GetFullPath(input.Path), StringComparer.Ordinal)
+                    .ThenBy(input => input.ExecutionId, StringComparer.Ordinal),
+            ];
         string[] inputPaths = [.. orderedInputs.Select(input => input.Path)];
         string[] identityInputs =
         [
+            context.Mode.ToString(),
             .. orderedInputs.Select(input => $"{Path.GetFullPath(input.Path)}\0{input.ExecutionId}"),
         ];
         Guid artifactId = CtrfReportMerger.CreateDeterministicId(identityInputs);
@@ -62,7 +69,10 @@ .. orderedInputs.Select(input => $"{Path.GetFullPath(input.Path)}\0{input.Execut
         }
 
         string outputPath = Path.Combine(mergedDirectory, $"merged-{artifactId:N}.ctrf.json");
-        await CtrfReportMerger.MergeAllToFileAsync(inputPaths, outputPath, cancellationToken).ConfigureAwait(false);
+        CtrfMergeMode mergeMode = context.Mode == ArtifactPostProcessingMode.RetryAttempts
+            ? CtrfMergeMode.CollapseRetryAttempts
+            : CtrfMergeMode.Concatenate;
+        await CtrfReportMerger.MergeAllToFileAsync(inputPaths, outputPath, mergeMode, cancellationToken).ConfigureAwait(false);
 
         return new ProcessedArtifact(
             outputPath,

src/Platform/Microsoft.Testing.Extensions.CtrfReport/CtrfReportMerger.FileMerge.cs

@@ -23,6 +23,13 @@ internal static Task MergeAllToFileAsync(
         CancellationToken cancellationToken)
         => MergeToFileAsync(inputPaths, outputPath, CtrfMergeMode.Concatenate, requireAllReports: true, cancellationToken);
 
+    internal static Task MergeAllToFileAsync(
+        IReadOnlyList<string> inputPaths,
+        string outputPath,
+        CtrfMergeMode mode,
+        CancellationToken cancellationToken)
+        => MergeToFileAsync(inputPaths, outputPath, mode, requireAllReports: true, cancellationToken);
+
     internal static Task MergeToFileAsync(
         IReadOnlyList<string> inputPaths,
         string outputPath,

src/Platform/Microsoft.Testing.Extensions.CtrfReport/InternalAPI/InternalAPI.Unshipped.txt

@@ -7,6 +7,7 @@ Microsoft.Testing.Extensions.CtrfReport.CtrfArtifactPostProcessor.Description.ge
 Microsoft.Testing.Extensions.CtrfReport.CtrfArtifactPostProcessor.DisplayName.get -> string!
 Microsoft.Testing.Extensions.CtrfReport.CtrfArtifactPostProcessor.IsEnabledAsync() -> System.Threading.Tasks.Task<bool>!
 Microsoft.Testing.Extensions.CtrfReport.CtrfArtifactPostProcessor.ProcessAsync(System.Collections.Generic.IReadOnlyList<Microsoft.Testing.Platform.Extensions.ArtifactPostProcessing.InputArtifact!>! inputs, string! outputDirectory, Microsoft.Testing.Platform.Extensions.ArtifactPostProcessing.ArtifactPostProcessingContext! context, System.Threading.CancellationToken cancellationToken) -> System.Threading.Tasks.Task<Microsoft.Testing.Platform.Extensions.ArtifactPostProcessing.ProcessedArtifact?>!
+Microsoft.Testing.Extensions.CtrfReport.CtrfArtifactPostProcessor.SupportedModes.get -> System.Collections.Generic.IReadOnlyList<Microsoft.Testing.Platform.Extensions.ArtifactPostProcessing.ArtifactPostProcessingMode>!
 Microsoft.Testing.Extensions.CtrfReport.CtrfArtifactPostProcessor.SupportsTruncatedRuns.get -> bool
 Microsoft.Testing.Extensions.CtrfReport.CtrfArtifactPostProcessor.SupportedFileExtensionsFallback.get -> System.Collections.Generic.IReadOnlyList<string!>!
 Microsoft.Testing.Extensions.CtrfReport.CtrfArtifactPostProcessor.SupportedKinds.get -> System.Collections.Generic.IReadOnlyList<string!>!
@@ -18,6 +19,7 @@ Microsoft.Testing.Extensions.CtrfReport.CtrfMergeMode.Concatenate = 0 -> Microso
 Microsoft.Testing.Extensions.CtrfReport.CtrfReportMerger
 static Microsoft.Testing.Extensions.CtrfReport.CtrfReportMerger.CreateDeterministicId(System.Collections.Generic.IReadOnlyList<string!>! values) -> System.Guid
 static Microsoft.Testing.Extensions.CtrfReport.CtrfReportMerger.MergeAllToFileAsync(System.Collections.Generic.IReadOnlyList<string!>! inputPaths, string! outputPath, System.Threading.CancellationToken cancellationToken) -> System.Threading.Tasks.Task!
+static Microsoft.Testing.Extensions.CtrfReport.CtrfReportMerger.MergeAllToFileAsync(System.Collections.Generic.IReadOnlyList<string!>! inputPaths, string! outputPath, Microsoft.Testing.Extensions.CtrfReport.CtrfMergeMode mode, System.Threading.CancellationToken cancellationToken) -> System.Threading.Tasks.Task!
 static Microsoft.Testing.Extensions.CtrfReport.CtrfReportMerger.Merge(System.Collections.Generic.IReadOnlyList<string!>! inputReports) -> string!
 static Microsoft.Testing.Extensions.CtrfReport.CtrfReportMerger.Merge(System.Collections.Generic.IReadOnlyList<string!>! inputReports, Microsoft.Testing.Extensions.CtrfReport.CtrfMergeMode mode) -> string!
 static Microsoft.Testing.Extensions.CtrfReport.CtrfReportMerger.MergeToFileAsync(System.Collections.Generic.IReadOnlyList<string!>! inputPaths, string! outputPath, Microsoft.Testing.Extensions.CtrfReport.CtrfMergeMode mode, System.Threading.CancellationToken cancellationToken) -> System.Threading.Tasks.Task!

src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/GitHubActionsSummaryArtifactPostProcessor.cs

@@ -25,6 +25,7 @@ internal sealed class GitHubActionsSummaryArtifactPostProcessor(
     private const string StepSummaryEnvironmentVariable = "GITHUB_STEP_SUMMARY";
     private const int StepSummaryMaxWriteAttempts = 20;
     private static readonly string[] SupportedArtifactKinds = [FragmentArtifactKind];
+    private static readonly ArtifactPostProcessingMode[] SupportedPostProcessingModes = [ArtifactPostProcessingMode.TestModules];
     private static readonly TimeSpan StepSummaryRetryDelay = TimeSpan.FromMilliseconds(50);
     private readonly bool _isEnabled =
         GitHubActionsFeature.IsEnabled(commandLineOptions, environment, GitHubActionsCommandLineOptions.GitHubActionsStepSummary)
@@ -38,6 +39,8 @@ internal sealed class GitHubActionsSummaryArtifactPostProcessor(
 
     public string Description => GitHubActionsResources.Description;
 
+    public IReadOnlyList<ArtifactPostProcessingMode> SupportedModes => SupportedPostProcessingModes;
+
     public bool SupportsTruncatedRuns => true;
 
     public IReadOnlyList<string> SupportedKinds => SupportedArtifactKinds;

src/Platform/Microsoft.Testing.Extensions.HtmlReport/HtmlArtifactPostProcessor.cs

@@ -13,6 +13,7 @@ internal sealed class HtmlArtifactPostProcessor : IArtifactPostProcessor
     private const string MergedReportDirectoryName = "merged";
 
     private static readonly string[] SupportedArtifactKinds = [HtmlReportGenerator.HtmlArtifactKind];
+    private static readonly ArtifactPostProcessingMode[] SupportedPostProcessingModes = [ArtifactPostProcessingMode.TestModules];
 
     public string Uid => "Microsoft.Testing.Extensions.HtmlReport.PostProcessor";
 
@@ -22,6 +23,8 @@ internal sealed class HtmlArtifactPostProcessor : IArtifactPostProcessor
 
     public string Description => ExtensionResources.HtmlArtifactPostProcessorDescription;
 
+    public IReadOnlyList<ArtifactPostProcessingMode> SupportedModes => SupportedPostProcessingModes;
+
     public bool SupportsTruncatedRuns => false;
 
     public IReadOnlyList<string> SupportedKinds => SupportedArtifactKinds;

src/Platform/Microsoft.Testing.Extensions.HtmlReport/InternalAPI/InternalAPI.Unshipped.txt

@@ -6,6 +6,7 @@ Microsoft.Testing.Extensions.HtmlReport.HtmlArtifactPostProcessor.DisplayName.ge
 Microsoft.Testing.Extensions.HtmlReport.HtmlArtifactPostProcessor.HtmlArtifactPostProcessor() -> void
 Microsoft.Testing.Extensions.HtmlReport.HtmlArtifactPostProcessor.IsEnabledAsync() -> System.Threading.Tasks.Task<bool>!
 Microsoft.Testing.Extensions.HtmlReport.HtmlArtifactPostProcessor.ProcessAsync(System.Collections.Generic.IReadOnlyList<Microsoft.Testing.Platform.Extensions.ArtifactPostProcessing.InputArtifact!>! inputs, string! outputDirectory, Microsoft.Testing.Platform.Extensions.ArtifactPostProcessing.ArtifactPostProcessingContext! context, System.Threading.CancellationToken cancellationToken) -> System.Threading.Tasks.Task<Microsoft.Testing.Platform.Extensions.ArtifactPostProcessing.ProcessedArtifact?>!
+Microsoft.Testing.Extensions.HtmlReport.HtmlArtifactPostProcessor.SupportedModes.get -> System.Collections.Generic.IReadOnlyList<Microsoft.Testing.Platform.Extensions.ArtifactPostProcessing.ArtifactPostProcessingMode>!
 Microsoft.Testing.Extensions.HtmlReport.HtmlArtifactPostProcessor.SupportedFileExtensionsFallback.get -> System.Collections.Generic.IReadOnlyList<string!>!
 Microsoft.Testing.Extensions.HtmlReport.HtmlArtifactPostProcessor.SupportedKinds.get -> System.Collections.Generic.IReadOnlyList<string!>!
 Microsoft.Testing.Extensions.HtmlReport.HtmlArtifactPostProcessor.SupportsTruncatedRuns.get -> bool