Skip to content

Auto-derive typedef-tag remaps from SDK headers - #2237

Closed
Jevan Saks (jevansaks) wants to merge 23 commits into
mainfrom
user/jevansaks/shift-left-metadata
Closed

Auto-derive typedef-tag remaps from SDK headers#2237
Jevan Saks (jevansaks) wants to merge 23 commits into
mainfrom
user/jevansaks/shift-left-metadata

Conversation

@jevansaks

@jevansaks Jevan Saks (jevansaks) commented Mar 27, 2026

Copy link
Copy Markdown
Member

Summary

Adds an \AutoDeriveRemaps\ MSBuild task that scans SDK header files at build time to automatically generate --remap\ entries for typedef-tag patterns, replacing ~11,800 manually-maintained entries in \scraper.settings.rsp.

What it does

The task parses C/C++ headers to find \ ypedef struct/enum/union TAG { ... } TYPEDEF;\ patterns and generates \TAG=TYPEDEF\ remap entries that ClangSharp uses to prefer the public typedef name over the internal tag name.

Before: 12,891 lines of manual remaps in \scraper.settings.rsp
After: 1,107 lines (930 residual entries that can't be auto-derived)

Residual entries (kept manual)

  • 53 semantic overrides — where the correct metadata name differs from both the tag and the first typedef (e.g., _GUID=Guid, _RTL_BARRIER=SYNCHRONIZATION_BARRIER)
  • 13 identity suppressions — types where auto-derivation picks the wrong typedef alias from headers with multiple aliases
  • ~870 other entries — type aliases (\DWORD_PTR=UIntPtr), anonymous members, MIDL-generated names not in traversed headers, etc.

Verification

  • Produces identical winmd output — zero differences vs last release (\CompareBinToLastRelease.ps1\ passes clean)
  • Same 23.2MB winmd as baseline
  • Exclusion mechanism prevents duplicate key errors with partition settings, autoTypes, and function pointer fixup RSPs

Files changed

File Change
\AutoDeriveRemaps.cs\ New MSBuild task (~450 lines)
\sdk.targets\ Wire up new task, add to scraper RSP chain
\scraper.settings.rsp\ Reduced from 12,891 to 1,107 lines
.vsconfig\ VS workload requirements for contributors

Future work

  • Function pointer fixup auto-derivation (currently still uses \ unctionPointerFixups.json)
  • Adding \Inputs\ attribute to the target for incremental build skip

Jevan Saks (jevansaks) and others added 23 commits March 23, 2026 23:13
@jevansaks
Add shift-left metadata plan document
89eaee2 
Comprehensive plan for moving sidecar metadata (.rsp, .json files) into
header annotations using SAL-like __attribute__((annotate(...))) macros.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jevansaks
Add annotation validation results and update plan
57f8ad6 
Rigorous testing of __attribute__((annotate(...))) through ClangSharp v17:
- Parameter-level annotations: ALL WORK (8 types validated)
- Function-level annotations: DROPPED by ClangSharp
- Type/typedef annotations: DROPPED by ClangSharp
- Struct field annotations: DROPPED by ClangSharp

Documented carrier-parameter workaround and extraction tool alternative.
Updated shift-left plan with revised strategy based on findings.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jevansaks
Update plans with ClangSharp v18 NativeAnnotation findings
421913a 
ClangSharp v18+ (PR #552) adds [NativeAnnotation] support that preserves
__attribute__((annotate(...))) on ALL declaration types - functions,
parameters, struct fields, and return values. This eliminates the need
for extraction tools or carrier-parameter workarounds. Upgrading from
v17 to v18+ is the path forward for shift-left metadata.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jevansaks
Auto-derive typedef-tag remaps from SDK headers
c68358c 
Add AutoDeriveRemaps MSBuild task that scans SDK header files to
automatically generate --remap entries for typedef-tag patterns
(e.g., typedef struct _FOO { ... } FOO -> _FOO=FOO).

This removes 11,805 manually-maintained entries from scraper.settings.rsp
(reduced from 12,891 lines to 1,086 lines). The remaining 913 entries are:
- 51 semantic overrides (e.g., _RTL_BARRIER=SYNCHRONIZATION_BARRIER)
- 862 entries not derivable from headers (type aliases, nested members, etc.)

The auto-derived remaps are generated at build time and produce the exact
same remap set as the original manual entries (verified: 0 missing, 0 changed).

New typedef-tag patterns in future SDK versions will be auto-discovered
without manual maintenance.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jevansaks
Pin ClangSharp as local tool to prevent version mismatch
0595ec4 
- Add .config/dotnet-tools.json pinning ClangSharpPInvokeGenerator to
  17.0.1 so builds use the correct version regardless of what is
  installed globally (a newer global version like v18 generates
  NativeAnnotation attributes that the emitter does not handle,
  causing GetDeclarationDiagnostics to take ~68 minutes instead of
  ~28 seconds)
- Generate a local tool manifest at build time in the scratch directory
  so this works both in-repo and when the GeneratorSdk is consumed as
  a NuGet MSBuild SDK package by external projects
- Invoke ClangSharp directly via its DLL path instead of dotnet tool run
  to avoid per-invocation overhead across 963 partition items
- Fix thread-safety bug: replace non-atomic ret &= ProcessPartition()
  in Parallel.ForEach with Interlocked.Increment on a failure counter
- Increase default scraping parallelism from ProcessorCount/2 to
  ProcessorCount
- Remove GenerateNuGetConfig and global tool install path

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jevansaks
Auto-derive typedef-tag remaps from SDK headers
9e68328 
Add AutoDeriveRemaps MSBuild task that scans SDK header files to
automatically generate --remap entries for typedef-tag patterns
(e.g., typedef struct _FOO { ... } FOO -> _FOO=FOO).

This removes ~11,800 manually-maintained entries from scraper.settings.rsp
(reduced from 12,891 lines to 1,107 lines). The remaining ~930 entries are:
- 53 semantic overrides (e.g., _GUID=Guid, _RTL_BARRIER=SYNCHRONIZATION_BARRIER)
- 13 identity suppression remaps for types where auto-derivation picks the wrong
  typedef from headers with multiple aliases
- ~870 entries not derivable from headers (type aliases, nested members, etc.)

Verified: produces identical winmd output with zero differences vs last release.

Also adds .vsconfig for VS workload requirements.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jevansaks
Bump version after publishing 70.0.11-preview (#2234)
fa07123 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jevansaks
Use ClangSharp PInvokeGenerator as library for AST-based remap discovery
b80d019 
Replace CLI invocation of ClangSharpPInvokeGenerator with direct library usage
via reflection. This lets us access _allValidNameRemappings after generation,
which contains all typedef-tag relationships discovered by libclang (13,896
remaps auto-discovered from the preprocessed AST).

Key changes to ScrapeHeaders.ProcessPartition:
- Load ClangSharp assemblies from tool store via Assembly.LoadFrom
- Pre-load native libclang.dll/libClangSharp.dll via NativeLibrary.Load
- Create PInvokeGeneratorConfiguration from parsed RSP files
- Call PInvokeGenerator.GenerateBindings via reflection
- Use IL-emitted delegate for CXTranslationUnit.TryParse (ReadOnlySpan params)
- Read _allValidNameRemappings via reflection after generation
- Custom outputStreamFactory (MemoryStream) avoids file locking issues
- Fall back to CLI for partitions with x86 inline assembly diagnostics

Handles macro-expanded names (e.g., VK_TO_WCHARS1 from ## pasting) that
the previous text-based scanner could not discover.

End-to-end verified: full build succeeds, 393/393 partitions scraped,
winmd generated successfully (24.3 MB, matching baseline).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jevansaks
Add auto type remappings plan document
90891eb 
Documents the architecture, implementation details, and next steps for
using ClangSharp PInvokeGenerator as a library to auto-derive typedef-tag
remaps from libclang's preprocessed AST.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jevansaks
Replace in-process library with CLI for process-isolated remap discovery
a645532 
Remove the in-process ClangSharp library integration (reflection, IL emission,
assembly loading, native library pre-loading) that ran all partitions in the
same host process. This shared a single libclang instance across 16 concurrent
threads, creating process isolation concerns: shared state, memory pressure,
thread safety, and file contention.

Instead, rely on the existing CLI invocation which naturally provides process
isolation (each partition runs in its own dotnet process). The
log-potential-typedef-remappings config option (already enabled in
baseSettings.rsp) causes ClangSharp to output typedef-tag relationships as
'Info: Potential remap: X=Y' lines, which ProcessPartition already parses.

New auto-rsp writing:
- WriteAutoRemapsRsp() writes discovered remaps to
  scraper.autoRemaps.generated.rsp after all partitions complete
- Merges with existing auto-rsp from previous builds (accumulates over time)
- sdk.targets includes auto-rsp in @(ScraperRsp) before manual entries so
  manual overrides take priority
- Condition=Exists handles first-build bootstrap (file doesn't exist yet)

Also fixes a pre-existing bug in remap value parsing: Split('=') was used
which incorrectly handles values containing '=' characters. Now uses
IndexOf('=') to split only on the first occurrence.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jevansaks
Add ClangSharpWorker subprocess for process-isolated remap discovery
41382c9 
Create ClangSharpWorker, a standalone console app that wraps
PInvokeGenerator as a library in its own process. It:
- Takes RSP files as CLI args (same format as ClangSharpPInvokeGenerator)
- Loads ClangSharp assemblies via Assembly.LoadFrom (reflection-based)
- Runs GenerateBindings() to produce C# output (same as CLI)
- Reads _allValidNameRemappings via reflection after generation
- Writes discovered typedef-tag remaps to a .remaps sidecar file
- Exits cleanly (full process isolation)

ScrapeHeaders spawns ClangSharpWorker instead of the stock CLI when
available. Each partition runs in its own dotnet process, eliminating
thread safety and shared-state concerns. After all partitions complete,
ScrapeHeaders reads the .remaps files, merges the discoveries, and
writes scraper.autoRemaps.generated.rsp for consumption on subsequent
builds.

Falls back to stock ClangSharpPInvokeGenerator CLI if the worker is
not available or fails.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jevansaks
Remove fallback logic, require ClangSharpWorker
cec2c90 
ClangSharpWorker.dll must be present — remove conditional fallback to
stock ClangSharpPInvokeGenerator CLI. ScrapeHeaders now errors if the
worker is not found at ToolsBinDir.

Also reads .remaps sidecar files from the worker to collect discovered
typedef-tag remaps after each successful partition.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jevansaks
Update plan doc with subprocess approach and research findings
df6b8dd 
Rewrite auto-type-remappings-plan.md to document:
- ClangSharpWorker subprocess architecture
- PInvokeGenerator extensibility research (sealed class, no public API
  for remapping data, no AST hooks)
- Useful private fields for future extraction (_usedRemappings)
- How VisitTypedefDecl discovers remaps
- Verified build results: 16,098 remaps from 393 partitions, 23.2 MB winmd

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jevansaks
Add verification results: winmd identical, 32 conflicts identified
bc63828 
Full IL comparison of 1,651,355 lines confirms the ClangSharpWorker
build produces an identical winmd to the origin/main CLI baseline.

Analysis of 16,098 auto-discovered remaps vs 12,705 manual entries:
- 12,196 (96%) are safely auto-derivable
- 32 conflicts where auto picks wrong name (e.g., _LARGE_INTEGER→
  LARGE_INTEGER vs manual long, _RTL_CRITICAL_SECTION→RTL_CRITICAL_SECTION
  vs manual CRITICAL_SECTION)
- 477 manual-only entries (semantic overrides, not discoverable from AST)
- 3,869 new discoveries not in manual file

The auto-rsp works correctly with manual overrides taking priority.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jevansaks
Emit typedef-tag remaps as .cs, apply in WinmdGenerator syntax cleaning
2604f15 
Change remap output from .rsp (consumed by scraper) to .cs (consumed by
WinmdGenerator). ScrapeHeaders writes AutoRemaps.generated.cs containing
tag=typedef entries as a static string array.

ClangSharpSourceToWinmd reads this .cs file and applies the remaps during
MetadataSyntaxTreeCleaner via:
- VisitIdentifierName: renames all type references globally
- VisitStructDeclaration: renames struct declarations
- VisitEnumDeclaration: renames enum declarations
- VisitCompilationUnit: strips the AutoRemaps namespace from output

This allows the full pipeline to work in a single build pass: the worker
discovers remaps during scraping, writes them as .cs, and the winmd
generator applies them during syntax tree cleaning.

Remove auto-rsp from sdk.targets since remaps now flow through .cs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jevansaks
Update plan doc: .cs pipeline verified, winmd identical to baseline
e4976bc 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jevansaks
Fix auto-remaps parser: Contains check was too strict
018d4e1 
The LoadAutoTypedefRemaps parser used Contains(=") which looked for
equals-then-quote. Changed to Contains(=) to match actual format.

Also restores scraper.settings.rsp -- manual remap removal revealed
that _allValidNameRemappings contains incorrect entries (IUnknown to
IXmlReaderInput) making blind global renaming unsafe.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jevansaks
Update plan: document blocking issue with auto-remap removal
2351239 
_allValidNameRemappings contains incorrect entries like IUnknown to
IXmlReaderInput that make blind global renaming unsafe. VisitIdentifierName
approach is too aggressive. Document three path-forward options.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jevansaks
Add heuristic filter for auto-remaps, disable global rename
ef8a4e4 
Add filtering in WriteAutoRemapsCsFile: only include entries where the
tag name looks internal (_, tag, __MIDL, ABI::, lowercase, trailing _).
Skip built-in defaults, identity remaps, and entries where the typedef
name looks worse than the tag name.

Disable VisitIdentifierName global rename in MetadataSyntaxTreeCleaner.
Testing revealed that applying new remap discoveries (not in the manual
--remap file) changes the API surface in ways that need review. Examples:
- smiOCTETS -> smiBITS (valid typedef but not previously remapped)
- IUnknown -> IXmlReaderInput (correctly filtered by heuristic)

The auto-remaps .cs file is still generated for analysis and reporting.
Applying the remaps at build time requires a more targeted approach
that only renames types matching the manual --remap entries.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jevansaks
Restore original scraper.settings.rsp
4921a66 
Revert the earlier attempted rsp cleanup. The auto-remap application
in MetadataSyntaxTreeCleaner is disabled, so the manual --remap
entries must remain for PInvokeGenerator to produce correct output.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jevansaks
Update plan doc with current blocking issues and path forward
8552c4e 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jevansaks
Implement two-phase worker: discover remaps then generate
85ac8d1 
ClangSharpWorker now runs PInvokeGenerator twice per partition:

Phase 1 (Discovery): Runs GenerateBindings with existing --remap
entries. Extracts _allValidNameRemappings. Filters using heuristic
(single typedef, internal-looking tag name). Output is discarded.

Phase 2 (Generation): Runs GenerateBindings again with merged remaps
(discovered + configured). Produces final C# output with correct
type names, including auto-discovered typedef-tag relationships.

Both phases run in the same process, using the same loaded assemblies.
Build time increases ~53% (15:51 vs 10:20) due to double parsing.

The heuristic filter:
- Single typedef per tag only (skip ambiguous)
- Tag must look internal (_, tag, __MIDL, ABI::, lowercase, trailing _)
- Skip if typedef looks worse than tag
- Configured --remap overrides take priority

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jevansaks
Final plan update: two-phase worker verified, manual rsp kept
ed193bf 
Two-phase ClangSharpWorker produces identical winmd to baseline
(0 IL differences across 1,651,338 lines). Build time ~53% overhead.

Removing manual --remap entries is not feasible: entries affect
PInvokeGenerator partition scoping beyond just naming (VDMCONTEXT
cross-namespace conflict). Manual rsp kept unchanged.

Going-forward benefit: new SDK types auto-remapped without manual entries.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vineeththomasalex Vineeth Thomas Alex (vineeththomasalex) Awaiting requested review from vineeththomasalex vineeththomasalex will be requested when the pull request is marked ready for review vineeththomasalex is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@jevansaks