Add telemetry collection for MSTest usage analytics#7570
Draft
Evangelink wants to merge 12 commits intomainfrom
Draft
Add telemetry collection for MSTest usage analytics#7570Evangelink wants to merge 12 commits intomainfrom
Evangelink wants to merge 12 commits intomainfrom
Conversation
Add infrastructure to collect aggregated telemetry about MSTest usage: - Track assertion API usage (Assert, CollectionAssert, StringAssert) - Track attribute usage and custom types during discovery - Track MSTest configuration settings and source - Send telemetry via MTP telemetry collector on session exit Key implementation details: - TelemetryCollector: thread-safe static counter in TestFramework using ConcurrentDictionary with atomic swap-and-drain pattern - MSTestTelemetryDataCollector: aggregates discovery and assertion data, builds metrics, anonymizes custom type names via SHA256 - Telemetry is opt-in via MTP telemetry infrastructure (respects TESTINGPLATFORM_TELEMETRY_OPTOUT) - VSTest mode collects but discards data (no telemetry sender available) Fixes applied during review: - Fix race condition in DrainAssertionCallCounts using Interlocked.Exchange - Fix thread safety of Current property using Volatile.Read/Write - Add synchronous SendTelemetryAndReset to avoid deadlock in sync callers - Use IDictionary<string, object> delegate type for consistency with MTP - Replace bare catch with catch (Exception) - Remove duplicate _serviceProvider field (use inherited ServiceProvider) - Add missing ContainsSingle telemetry tracking - Replace string interpolation with string.Concat in hot paths - Standardize blank lines after TrackAssertionCall calls
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds telemetry collection infrastructure for MSTest usage analytics, tracking assertion API usage, attribute usage, custom types (anonymized via SHA256), and configuration settings. Data is collected during test discovery and execution and sent via MTP's telemetry system on session exit. In VSTest mode, data is collected but silently discarded.
Changes:
- Adds
TelemetryCollector(static, thread-safe assertion counter) in TestFramework andMSTestTelemetryDataCollector(aggregates discovery + assertion data) in the adapter - Instruments all assertion methods across
Assert,CollectionAssert, andStringAssertwithTrackAssertionCallcalls - Integrates telemetry initialization and sending into
MSTestDiscoverer,MSTestExecutor, andMSTestBridgedTestFramework
Reviewed changes
Copilot reviewed 27 out of 27 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
TelemetryCollector.cs |
New static class with thread-safe assertion call counting using ConcurrentDictionary and atomic drain |
MSTestTelemetryDataCollector.cs |
New class aggregating settings, attribute usage, custom types, and assertion counts into telemetry metrics |
Assert.*.cs (multiple files) |
Added TrackAssertionCall at entry points of assertion methods |
CollectionAssert.cs |
Added TrackAssertionCall for collection assertion methods |
StringAssert.cs |
Added TrackAssertionCall for string assertion methods |
MSTestExecutor.cs |
Initializes telemetry collector, sends telemetry in finally block after test execution |
MSTestDiscoverer.cs |
Initializes telemetry collector, sends telemetry synchronously after discovery |
MSTestBridgedTestFramework.cs |
Creates MTP telemetry sender delegate for MTP mode |
MSTestSettings.cs |
Tracks configuration source (testconfig.json/runsettings/none) for telemetry |
TypeEnumerator.cs / AssemblyEnumerator.cs |
Passes telemetry collector through discovery pipeline |
Microsoft.Testing.Platform.csproj |
Adds InternalsVisibleTo for MSTest.TestAdapter |
TelemetryTests.cs |
Integration tests for MTP and VSTest telemetry scenarios |
You can also share your feedback on Copilot code review. Take the survey.
Convert.ToHexString is .NET 5+ only. Use BitConverter.ToString with Replace for the .NET Framework code path, matching the existing #if NET split for SHA256.
Remove the synchronous SendTelemetryAndReset overload from MSTestTelemetryDataCollector. The blocking call now lives in MSTestDiscoverer (the only sync caller), wrapped in Task.Run to avoid SynchronizationContext deadlocks.
Contributor
There was a problem hiding this comment.
Pull request overview
Adds MSTest session-level usage telemetry collection (assertion API usage, attribute usage, custom attribute subclasses, and configuration/settings source) and wires it into both MTP-hosted and VSTest-hosted execution/discovery flows, with new acceptance coverage for MTP scenarios.
Changes:
- Introduces
TelemetryCollectorin MSTest.TestFramework to track assertion call counts. - Adds
MSTestTelemetryDataCollectorin the adapter to aggregate settings/config source, discovery attributes, anonymized custom types, and drained assertion counts into telemetry metrics. - Integrates telemetry initialization/sending into discovery/execution paths (MTP bridge + VSTest adapter) and adds MTP acceptance tests.
Reviewed changes
Copilot reviewed 27 out of 27 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| test/IntegrationTests/MSTest.Acceptance.IntegrationTests/TelemetryTests.cs | Adds acceptance coverage to validate telemetry emission/absence via diagnostic logs. |
| src/TestFramework/TestFramework/Internal/TelemetryCollector.cs | New internal assertion-call counter with drain/reset semantics. |
| src/TestFramework/TestFramework/Assertions/StringAssert.cs | Adds telemetry tracking for StringAssert APIs. |
| src/TestFramework/TestFramework/Assertions/CollectionAssert.cs | Adds telemetry tracking for CollectionAssert APIs. |
| src/TestFramework/TestFramework/Assertions/Assert.ThrowsException.cs | Adds telemetry tracking for Throws* APIs using CallerMemberName. |
| src/TestFramework/TestFramework/Assertions/Assert.StartsWith.cs | Adds telemetry tracking for StartsWith/DoesNotStartWith. |
| src/TestFramework/TestFramework/Assertions/Assert.Matches.cs | Adds telemetry tracking for MatchesRegex/DoesNotMatchRegex. |
| src/TestFramework/TestFramework/Assertions/Assert.IsTrue.cs | Adds telemetry tracking for IsTrue/IsFalse. |
| src/TestFramework/TestFramework/Assertions/Assert.IsNull.cs | Adds telemetry tracking for IsNull/IsNotNull. |
| src/TestFramework/TestFramework/Assertions/Assert.IsInstanceOfType.cs | Adds telemetry tracking for IsInstanceOfType/IsNotInstanceOfType. |
| src/TestFramework/TestFramework/Assertions/Assert.IsExactInstanceOfType.cs | Adds telemetry tracking for IsExactInstanceOfType/IsNotExactInstanceOfType. |
| src/TestFramework/TestFramework/Assertions/Assert.Inconclusive.cs | Adds telemetry tracking for Inconclusive. |
| src/TestFramework/TestFramework/Assertions/Assert.IComparable.cs | Adds telemetry tracking for IComparable-based asserts. |
| src/TestFramework/TestFramework/Assertions/Assert.Fail.cs | Adds telemetry tracking for Fail. |
| src/TestFramework/TestFramework/Assertions/Assert.EndsWith.cs | Adds telemetry tracking for EndsWith/DoesNotEndWith. |
| src/TestFramework/TestFramework/Assertions/Assert.Count.cs | Adds telemetry tracking for count/empty asserts. |
| src/TestFramework/TestFramework/Assertions/Assert.Contains.cs | Adds telemetry tracking for Contains/DoesNotContain/IsInRange APIs. |
| src/TestFramework/TestFramework/Assertions/Assert.AreSame.cs | Adds telemetry tracking for AreSame/AreNotSame. |
| src/TestFramework/TestFramework/Assertions/Assert.AreEqual.cs | Adds telemetry tracking for AreEqual/AreNotEqual overloads. |
| src/Platform/Microsoft.Testing.Platform/Microsoft.Testing.Platform.csproj | Grants InternalsVisibleTo MSTest.TestAdapter for internal MTP telemetry extension access. |
| src/Adapter/MSTestAdapter.PlatformServices/Telemetry/MSTestTelemetryDataCollector.cs | New adapter-side aggregator that builds metrics and sends/discards telemetry. |
| src/Adapter/MSTestAdapter.PlatformServices/MSTestSettings.cs | Records telemetry configuration source (runsettings/testconfig.json/none). |
| src/Adapter/MSTestAdapter.PlatformServices/Discovery/TypeEnumerator.cs | Hooks discovery to record method/class attribute usage into telemetry collector. |
| src/Adapter/MSTestAdapter.PlatformServices/Discovery/AssemblyEnumerator.cs | Passes the current telemetry collector into TypeEnumerator. |
| src/Adapter/MSTest.TestAdapter/VSTestAdapter/MSTestExecutor.cs | Initializes collector for runs and triggers telemetry send/reset at run end. |
| src/Adapter/MSTest.TestAdapter/VSTestAdapter/MSTestDiscoverer.cs | Initializes collector for discovery and triggers telemetry send/reset at discovery end. |
| src/Adapter/MSTest.TestAdapter/TestingPlatformAdapter/MSTestBridgedTestFramework.cs | Creates an MTP telemetry sender delegate and injects it into discoverer/executor. |
Comments suppressed due to low confidence (1)
src/Adapter/MSTestAdapter.PlatformServices/Telemetry/MSTestTelemetryDataCollector.cs:267
- Same issue in the synchronous
SendTelemetryAndReset: the early return ontelemetrySender is null/HasDatameans assertion counters are never drained/reset when telemetry is unavailable, so counts can leak across sessions and the dictionary can grow unbounded in long-lived processes. Consider ensuring counters are reset even when telemetry is discarded.
You can also share your feedback on Copilot code review. Take the survey.
Add IsTelemetryOptedOut() to MSTestTelemetryDataCollector that checks the same environment variables as MTP's TelemetryManager: - TESTINGPLATFORM_TELEMETRY_OPTOUT - DOTNET_CLI_TELEMETRY_OPTOUT When either is '1' or 'true', skip initializing the collector entirely in VSTest mode (MSTestDiscoverer + MSTestExecutor). This avoids unnecessary data collection overhead when telemetry is disabled. In MTP mode, the opt-out is already handled by TelemetryManager which returns null from CreateTelemetrySender() when disabled.
Address the most important correctness issues in the telemetry branch: - drain assertion counters even when telemetry is not sent, avoiding stale usage leaking into later sessions - make collector initialization atomic with Interlocked.CompareExchange - track Assert.That and interpolated-string-handler assertion paths so modern call sites are no longer systematically undercounted
Contributor
There was a problem hiding this comment.
Pull request overview
Adds opt-in telemetry collection to MSTest to capture aggregated usage analytics (assertion APIs, attributes, custom types, and settings/config source) and wires it through the adapter for MTP sessions, with integration tests validating end-to-end behavior.
Changes:
- Added a framework-level
TelemetryCollectorand instrumented many assertion APIs to increment aggregated counters. - Added an adapter-side
MSTestTelemetryDataCollectorthat aggregates discovery/settings + drained assertion counts and sends a session-exit telemetry event (MTP only). - Added acceptance integration tests for MTP telemetry-enabled/disabled scenarios and VSTest regression coverage; added
InternalsVisibleToto enable adapter ↔ platform integration.
Reviewed changes
Copilot reviewed 28 out of 28 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| test/IntegrationTests/MSTest.Acceptance.IntegrationTests/TelemetryTests.cs | New acceptance tests validating telemetry emission (MTP) and ensuring no regressions in VSTest mode. |
| src/TestFramework/TestFramework/Internal/TelemetryCollector.cs | New internal aggregated counter store with swap-and-drain for assertion usage. |
| src/TestFramework/TestFramework/Assertions/StringAssert.cs | Adds telemetry tracking to StringAssert APIs. |
| src/TestFramework/TestFramework/Assertions/CollectionAssert.cs | Adds telemetry tracking to CollectionAssert APIs. |
| src/TestFramework/TestFramework/Assertions/Assert.ThrowsException.cs | Adds telemetry tracking to throws-related Assert APIs using caller member name. |
| src/TestFramework/TestFramework/Assertions/Assert.That.cs | Adds telemetry tracking to Assert.That. |
| src/TestFramework/TestFramework/Assertions/Assert.StartsWith.cs | Adds telemetry tracking to starts-with asserts. |
| src/TestFramework/TestFramework/Assertions/Assert.Matches.cs | Adds telemetry tracking to regex asserts. |
| src/TestFramework/TestFramework/Assertions/Assert.IsTrue.cs | Adds telemetry tracking to IsTrue/IsFalse (including interpolated handler compute paths). |
| src/TestFramework/TestFramework/Assertions/Assert.IsNull.cs | Adds telemetry tracking to IsNull/IsNotNull (including interpolated handler compute paths). |
| src/TestFramework/TestFramework/Assertions/Assert.IsInstanceOfType.cs | Adds telemetry tracking to instance-of asserts (including interpolated handler compute paths). |
| src/TestFramework/TestFramework/Assertions/Assert.IsExactInstanceOfType.cs | Adds telemetry tracking to exact-instance-of asserts (including interpolated handler compute paths). |
| src/TestFramework/TestFramework/Assertions/Assert.Inconclusive.cs | Adds telemetry tracking to Assert.Inconclusive. |
| src/TestFramework/TestFramework/Assertions/Assert.IComparable.cs | Adds telemetry tracking to comparable-based asserts. |
| src/TestFramework/TestFramework/Assertions/Assert.Fail.cs | Adds telemetry tracking to Assert.Fail. |
| src/TestFramework/TestFramework/Assertions/Assert.EndsWith.cs | Adds telemetry tracking to ends-with asserts. |
| src/TestFramework/TestFramework/Assertions/Assert.Count.cs | Adds telemetry tracking to count/empty asserts. |
| src/TestFramework/TestFramework/Assertions/Assert.Contains.cs | Adds telemetry tracking to contains/range asserts. |
| src/TestFramework/TestFramework/Assertions/Assert.AreSame.cs | Adds telemetry tracking to same/not-same asserts (including interpolated handler compute paths). |
| src/TestFramework/TestFramework/Assertions/Assert.AreEqual.cs | Adds telemetry tracking to equal/not-equal asserts (including interpolated handler compute paths). |
| src/Platform/Microsoft.Testing.Platform/Microsoft.Testing.Platform.csproj | Adds InternalsVisibleTo for MSTest.TestAdapter to support MTP telemetry integration. |
| src/Adapter/MSTestAdapter.PlatformServices/Telemetry/MSTestTelemetryDataCollector.cs | New adapter-side collector that aggregates settings/discovery + drained assertion counts and sends metrics. |
| src/Adapter/MSTestAdapter.PlatformServices/MSTestSettings.cs | Captures config source into telemetry collector during settings population. |
| src/Adapter/MSTestAdapter.PlatformServices/Discovery/TypeEnumerator.cs | Plumbs telemetry collector into discovery to track class/method attributes. |
| src/Adapter/MSTestAdapter.PlatformServices/Discovery/AssemblyEnumerator.cs | Passes current telemetry collector into type enumerator creation. |
| src/Adapter/MSTest.TestAdapter/VSTestAdapter/MSTestExecutor.cs | Initializes and flushes telemetry on test-run completion (sender injected for MTP). |
| src/Adapter/MSTest.TestAdapter/VSTestAdapter/MSTestDiscoverer.cs | Initializes and flushes telemetry on discovery completion (sender injected for MTP). |
| src/Adapter/MSTest.TestAdapter/TestingPlatformAdapter/MSTestBridgedTestFramework.cs | Provides MTP telemetry sender and injects it into discoverer/executor. |
You can also share your feedback on Copilot code review. Take the survey.
…ectory - Fix MTP_DiscoverTests_SendsTelemetryEvent regex to match across multiple lines between sessionexit event and attribute_usage - Add global.json with VSTest runner to VSTest test asset to opt out of MTP runner enforcement from root global.json - Add workingDirectory to VSTest test methods so dotnet test resolves the local global.json correctly
Youssef1313
reviewed
Mar 18, 2026
Youssef1313
reviewed
Mar 18, 2026
Youssef1313
reviewed
Mar 18, 2026
Youssef1313
reviewed
Mar 18, 2026
- Remove unused classType parameter from TrackDiscoveredClass - Fix null! to string? with null in TrackDiscoveredClass switch - Update DrainAssertionCallCounts doc to note best-effort semantics - Add assertion_usage verification in MTP run telemetry test - Add WIN_UI guards alongside WINDOWS_UWP for telemetry code - Replace System.Text.Json with manual JSON serialization - Fix Encoding.UTF8 to System.Text.Encoding.UTF8 for netstandard2.0
- Remove misleading 'for testing purposes' comment on MSTestDiscoverer internal constructor (it's also used by MSTestBridgedTestFramework) - Remove unnecessary Task.Run wrapper around SendTelemetryAndResetAsync in MSTestDiscoverer (no SyncContext deadlock risk in VSTest/MTP hosts, and SendTelemetryAndResetAsync catches all exceptions internally)
This was referenced Apr 1, 2026
- Remove reference to deleted MSTestSettings.TestSettingsFile property - Adapt TelemetryTests to single-asset TestAssetFixtureBase API - Remove unused VSTestAssetName constant - Update DotnetCli.RunAsync calls to match new signature
The MTP project's default SDK globbing was picking up vstest/UnitTest1.cs recursively, causing duplicate type definitions at build time.
Contributor
There was a problem hiding this comment.
Copilot's findings
Comments suppressed due to low confidence (1)
src/Adapter/MSTest.TestAdapter/VSTestAdapter/MSTestExecutor.cs:181
- Same issue as above for the sources-based run: if
MSTestDiscovererHelpers.InitializeDiscovery(...)returns false, the method returns without callingSendTelemetryAsync(), leaving the static collector/counters uncleared for subsequent sessions. Ensure telemetry reset/drain runs in a finally for all exit paths.
// Initialize telemetry collection if not already set
#if !WINDOWS_UWP && !WIN_UI
if (!MSTestTelemetryDataCollector.IsTelemetryOptedOut())
{
_ = MSTestTelemetryDataCollector.EnsureInitialized();
}
#endif
TestSourceHandler testSourceHandler = new();
if (!MSTestDiscovererHelpers.InitializeDiscovery(sources, runContext, frameworkHandle, configuration, testSourceHandler))
{
return;
}
- Files reviewed: 29/29 changed files
- Comments generated: 3
Comment on lines
+102
to
+124
| // Track attribute usage counts by base type name | ||
| string trackingName = attribute switch | ||
| { | ||
| TestMethodAttribute => nameof(TestMethodAttribute), | ||
| TestClassAttribute => nameof(TestClassAttribute), | ||
| DataRowAttribute => nameof(DataRowAttribute), | ||
| DynamicDataAttribute => nameof(DynamicDataAttribute), | ||
| TimeoutAttribute => nameof(TimeoutAttribute), | ||
| IgnoreAttribute => nameof(IgnoreAttribute), | ||
| DoNotParallelizeAttribute => nameof(DoNotParallelizeAttribute), | ||
| RetryBaseAttribute => nameof(RetryBaseAttribute), | ||
| ConditionBaseAttribute => nameof(ConditionBaseAttribute), | ||
| TestCategoryAttribute => nameof(TestCategoryAttribute), | ||
| #if !WIN_UI | ||
| DeploymentItemAttribute => nameof(DeploymentItemAttribute), | ||
| #endif | ||
| _ => attributeName, | ||
| }; | ||
|
|
||
| _attributeCounts[trackingName] = _attributeCounts.TryGetValue(trackingName, out long count) | ||
| ? count + 1 | ||
| : 1; | ||
| } |
There was a problem hiding this comment.
TrackDiscoveredMethod currently records all method-level attributes by default (_ => attributeName). This will include arbitrary user-defined attributes and send their type names in telemetry, which is both a privacy risk and can create high-cardinality payloads. Consider only counting a fixed allowlist of MSTest attributes (and ignoring the rest), or anonymizing/aggregating unknown attributes (e.g., bucket as "Other").
Comment on lines
+206
to
+246
| private static string SerializeCollection(IEnumerable<string> values) | ||
| { | ||
| System.Text.StringBuilder builder = new("["); | ||
| bool isFirst = true; | ||
|
|
||
| foreach (string value in values) | ||
| { | ||
| if (!isFirst) | ||
| { | ||
| builder.Append(','); | ||
| } | ||
|
|
||
| AppendJsonString(builder, value); | ||
| isFirst = false; | ||
| } | ||
|
|
||
| builder.Append(']'); | ||
| return builder.ToString(); | ||
| } | ||
|
|
||
| private static string SerializeDictionary(Dictionary<string, long> values) | ||
| { | ||
| System.Text.StringBuilder builder = new("{"); | ||
| bool isFirst = true; | ||
|
|
||
| foreach (KeyValuePair<string, long> value in values) | ||
| { | ||
| if (!isFirst) | ||
| { | ||
| builder.Append(','); | ||
| } | ||
|
|
||
| AppendJsonString(builder, value.Key); | ||
| builder.Append(':'); | ||
| builder.Append(value.Value.ToString(System.Globalization.CultureInfo.InvariantCulture)); | ||
| isFirst = false; | ||
| } | ||
|
|
||
| builder.Append('}'); | ||
| return builder.ToString(); | ||
| } |
There was a problem hiding this comment.
The JSON serialization helpers iterate over Dictionary/HashSet directly. HashSet enumeration order is nondeterministic (and can vary per-process due to randomized hashing), and reflection discovery order can also vary, so the resulting JSON strings can change order between runs even when the underlying data is identical. If the telemetry backend treats these as opaque strings, this will significantly increase metric cardinality. Consider producing deterministic output (e.g., sort keys/values before serialization) or emitting structured metrics instead of JSON strings.
Comment on lines
+126
to
137
| // Initialize telemetry collection if not already set | ||
| #if !WINDOWS_UWP && !WIN_UI | ||
| if (!MSTestTelemetryDataCollector.IsTelemetryOptedOut()) | ||
| { | ||
| _ = MSTestTelemetryDataCollector.EnsureInitialized(); | ||
| } | ||
| #endif | ||
|
|
||
| if (!MSTestDiscovererHelpers.InitializeDiscovery(from test in tests select test.Source, runContext, frameworkHandle, configuration, new TestSourceHandler())) | ||
| { | ||
| return; | ||
| } |
There was a problem hiding this comment.
MSTestTelemetryDataCollector is initialized before InitializeDiscovery, but if MSTestDiscovererHelpers.InitializeDiscovery(...) returns false the method returns early and SendTelemetryAsync() is never called. This leaves the static Current collector (and assertion counters) alive across sessions and can cause cross-run contamination/memory growth. Wrap the InitializeDiscovery + execution block in a try/finally (or move initialization after a successful InitializeDiscovery) so telemetry is always reset/drained on every exit path.
This issue also appears on line 169 of the same file.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add infrastructure to collect aggregated telemetry about MSTest usage within test sessions. This data helps understand which APIs are heavily used or unused to guide future investment.
What's collected
Architecture
TelemetryCollectorConcurrentDictionarywith atomic swap-and-drainMSTestTelemetryDataCollectorKey design decisions
TESTINGPLATFORM_TELEMETRY_OPTOUT)DrainAssertionCallCountsusesInterlocked.Exchangefor atomic swap;Currentproperty usesVolatile.Read/WriteSendTelemetryAndResetwithTask.Runto avoidSynchronizationContextcaptureTrackAssertionCallisAggressiveInlining; hot-path string building usesstring.Concatinstead of interpolationRemaining items for discussion
InterpolatedStringHandlerassertion overloads bypass telemetry tracking (they have independent code paths viaComputeAssertion()). Tracking those would require modifying handler structs.InternalsVisibleToforMSTest.TestAdapter→Microsoft.Testing.Platformtightens coupling — worth discussing if a proper API surface would be better.Test coverage