research: re-frame re-feed drift as inference-verification error (GPU-free)

[matteso1]

Re-analysis of the existing 780-pivot re-feed-drift data (arXiv 2606.15621, Qwen2.5-7B) through the inference-verification lens — the seed for the ".thawkv as inference receipt" thesis. No GPU.

Result: replay-based verifiers (DiFR, TOPLOC, SVIP, VeriLLM) all reconstruct state by re-feeding. From the existing data:

• re-feed recomputes a different argmax token than was generated at 23.1% vs a 7.8% floor (+15.3pp), robust across all 4 runs (+14–18pp);
• tight-tolerance false-reject: +23.6pp over floor at τ=0.01 nats.

So the whole replay-verification class silently mis-verifies ~15pp of low-margin tokens, and thaw's true-decode-KV replay is the fix.

Honest scope: a reframe of thaw's own data, not a head-to-head against a deployed verifier — that's the next (GPU) experiment. No README claim; evidence lives in benchmarks/verification_drift.py + site/receipts/2026-06-24_verification_drift.json.

Summary by CodeRabbit

• New Features
  • Added a command-line tool to summarize verification drift results from experiment data.
  • The report now includes disagreement rates, false-reject rates across thresholds, and sign-flip comparisons for both overall and low-margin cases.
  • Added a new receipt file with a complete example of verification drift results and provenance information.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
site	Ready	Preview, Comment	Jun 25, 2026 2:59am

[coderabbitai]

Caution

Review failed

Pull request was closed or merged during review

📝 Walkthrough

Walkthrough

Adds a verification drift CLI that reads a drift-ablation run JSON, computes argmax disagreement, false-reject sweep, and credit sign flip metrics, then prints summaries and can write a receipt JSON. It also adds a stored receipt payload for one run.

Changes

Verification drift CLI

Layer / File(s)	Summary
Script setup and helpers benchmarks/verification_drift.py	Module docstring, _rate, and sign establish the drift verification script and shared calculations.
Metric computations benchmarks/verification_drift.py	argmax_disagreement, false_reject_sweep, and credit_sign_flip filter pivots and compute the reported divergence metrics.
CLI analysis and receipt output benchmarks/verification_drift.py, site/receipts/2026-06-24_verification_drift.json	analyze, main, the entry point, and the receipt payload serialize the run metrics and source metadata.

Sequence Diagram(s)

sequenceDiagram
  participant main
  participant analyze
  participant run_json
  participant json_out
  main->>run_json: load experiment JSON
  main->>analyze: compute metrics with margin_split
  analyze-->>main: metric summary
  main->>json_out: write receipt JSON

Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Poem

🐇 I hopped through pivots, quick and spry,
Counted drift beneath the sky.
Argmax, flips, and taus all gleam,
In a tidy JSON stream.
Thump! The receipt is neat and bright,
A little ledger for the night.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 42.86% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title is concise and accurately summarizes the main change: reframing re-feed drift analysis as inference-verification error without GPU use.
Description check	✅ Passed	The description covers the change, motivation, results, and scope, but it omits the template's explicit verification details and checklist items.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch research/verification-drift

---

_{Comment @coderabbitai help to get the list of available commands.}