Mutation baseline and CI ledger (Phase 0)

[AhtishamShahid]

Related issues

• Implements mitodl/hq#12637 — OVS React 18: Phase 0 — Stryker mutation baseline and CI ledger
• Part of epic mitodl/hq#7958 — React 18 upgrade (sub-issue 2 of 8)

⚠️ Stacked on #1564. Base is chore/rtl-harness-and-react-prep, not master — GitHub will
retarget automatically when #1564 merges. Review #1564 first.

Closing keywords don't work across repositories, so mitodl/hq#12637 needs closing manually.

---

What

The verification harness that makes the bulk Enzyme→RTL conversion safe. No product code changes —
one new test file is the only addition under static/js/.

Three independent safety nets:

	Answers
Stryker mutation testing	Do these tests actually detect bugs? — the question coverage cannot answer
CI ledger	Did anything quietly shrink?
Bundle smoke test	Does the app still start?

Why this before #12638

The dominant risk when an agent rewrites 31 Enzyme files is a test that renders a component and
asserts nothing. It passes, and it reports 100% coverage on every line it touches. Test count stays
at 493, coverage stays flat, and the suite quietly stops detecting anything.

Mutation score is the only number that moves. This PR records it before any bulk conversion, so
a regression during #12638 is attributable.

The baseline

48.81% total / 56.80% covered — 1,349 of 2,764 valid mutants detected across 81 files and 2,785
mutants, in 26m25s.

That is a reference line, not a grade to improve. Its only job is to reveal a conversion that
keeps the test count while weakening the assertions. .stryker-baseline.json is committed rather
than generated so it can only change under review, and carries a 2-point tolerance — converting
.instance() assertions to behavioural ones can legitimately kill fewer mutants while being the
better test, because RTL cannot reach private methods Enzyme could.

Changes

Commit	Change
f2efc5b	Stryker 9.6.1 + mocha-runner, stryker.conf.json, npm run mutation
e24fb31	.stryker-baseline.json — the recorded score
fc2a925	scripts/test/ledger.sh + npm run ledger + CI step
271e1cf	Production bundle smoke test

Stryker config — six load-bearing settings

Each was verified against a real failure during a timeboxed spike. Do not "simplify" any of them.

Setting	Without it
mutator.plugins incl. flow	BABEL_PARSE_ERROR on the first import type line — nothing runs
inPlace: true	Every test errors: @babel/register doesn't hook inside Stryker's copied sandbox, so mocha falls back to import() and the ESM loader rejects extensionless imports
ignorePatterns	Stryker walks .claude/worktrees/, a stale worktree holding a second copy of the project
disableTypeChecks: false	Preprocessor tries to parse ui/templates/base.html as markup
mochaOptions.file	Loads global_init.js before specs, reproducing js_test.sh rather than relying on argument order
no timeout key	mochaOptions is additionalProperties: false and rejects it — use top-level timeoutMS

⚠️ inPlace mutates the working tree and restores from a backup dir. Never run npm run mutation
on uncommitted work.

Ledger

Six metrics, each a build failure rather than a discovery weeks later:

PASS  passing tests                    493  (need ge 492)
PASS  enzyme test files                 31  (need le 31)
PASS  data-testid in components          0  (need le 15)
PASS  FlowFixMe occurrences             40  (need le 40)
PASS  js_test.sh allowlist lines         7  (need le 7)
PASS  mutation score                   48.8 (floor 46.8)

Counts mocha's reported total, not grep -c "it(" — the latter undercounts loop-generated tests
by 27% and an agent can satisfy it while deleting tests. The mutation check reads a report and SKIPs
when absent; it never runs Stryker itself, since 26 minutes makes that a nightly job.

Verification

Every gate was proven to fail, not just to pass:

• Ledger — broke the testid cap, added an 8th allowlist line, and raised the recorded baseline.
  All three exit 1; restoring returns to ledger OK.
• Smoke test — injected throw new Error(...) into the built bundle; test fails with the
  message. Also verified it skips with no build and skips on a dev build.

Suite 492 → 493 passing. npm run lint and npm run fmt:check clean. No product code modified.

Reviewer notes

Four bugs were caught by insisting each gate demonstrate failure:

1. Ledger score arithmetic, twice. First draft counted only Killed, omitting Timeout
   (Stryker treats a hung mutant as detected). Second excluded only Ignored/CompileError, not
   RuntimeError — computing 48.4% against a recorded 48.81%. Either would have failed CI on
   nothing. Now matches Stryker's arithmetic exactly.
2. data-testid counted test fixtures, reading 1 from an inline fixture. Now production files
   only — the thing worth capping is testids added to real components.
3. Smoke test hit a harness limitation, not an app bug. Automatic publicPath is not supported in this browser: webpack resolves its own URL from document.currentScript.src, which
   vm.runInContext doesn't provide. Fixed by appending a matching <script src>.
4. Dev builds would hang the smoke test — webpack-hot-middleware's client opens a permanently
   pending EventSource. It now detects and skips them, so a running docker-compose doesn't produce
   a mystery timeout.

Deviation from the issue

hq#12637 specifies a /terms/ smoke test. That framing assumes a running server, which in CI means
docker-compose orchestration or adding Playwright — disproportionate to the value. Booting the same
bundle directly in jsdom reaches the same goal (real built app, no auth) with zero new
dependencies. It is narrower: it proves the bundle initialises and React mounts, not that the page
renders end-to-end. A genuine browser-level test should be its own issue, sequenced with R3 where
createRoot timing makes it most valuable.

Next

mitodl/hq#12638 (convert Tier 1 + Tier 2, 17 files). Every conversion commit should run
npm run ledger; re-check the mutation score at the end of each tier rather than per file.