v1.60.2.0 fix: free-suite drift on dev machines (eval-list cwd, gemini regex, observability floor) - #2470
Merged
Merged
Conversation
…n't dodge the fixture store getProjectEvalDir() probes cwd-relative .claude/skills/gstack/bin/gstack-slug; with cwd=ROOT on a dev machine the self-symlink makes it succeed, routing reads to an empty project-scoped dir instead of the seeded legacy ~/.gstack-dev/evals. Neutral cwd + absolute script path fails both probes deterministically — same behavior as CI. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…i NOT-READY message The message now leads with 'Export GEMINI_API_KEY...' (free-tier OAuth deprecation); the old pattern only knew lowercase 'export'. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… removed promptFile unlink aa3bd6f deleted the prompt temp file (and its /* non-fatal */ marker) when it dropped shell interpolation. The invariant — every runner I/O path wrapped non-fatally — still holds at the 5 remaining sites, now named in the comment. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Five browse test files setTimeout(() => process.exit(0), 500) inside the shared bun process; the suite can exit 0 before the summary with real failures masked. Receipts + fix path filed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
😎 This pull request was merged. |
E2E Evals: ✅ PASS0/0 tests passed | $0 total cost | 13 parallel runners
13x ubicloud-standard-8 (Docker: pre-baked toolchain + deps) | wall clock ≈ slowest suite |
garrytan
added a commit
that referenced
this pull request
Aug 7, 2026
Main's #2470 fixes the three free-suite dev-machine drift failures this branch had independently diagnosed and receipted (eval-list dev-symlink cwd escape, benchmark-cli case-brittle regex, observability floor). Union merge: VERSION/package.json stay at this branch's 1.61.0.0 (> main's 1.60.2.0, allocator-consistent); CHANGELOG keeps 1.61.0.0 on top of main's 1.60.2.0; TODOS unions both waves' filings (incl. main's new P1 on force-exit-0 masking suite failures). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
garrytan
added a commit
that referenced
this pull request
Aug 7, 2026
Main's #2470 fixes the three free-suite dev-machine drift failures this branch had independently diagnosed and receipted (eval-list dev-symlink cwd escape, benchmark-cli case-brittle regex, observability floor). Union merge: VERSION/package.json stay at this branch's 1.61.0.0 (> main's 1.60.2.0, allocator-consistent); CHANGELOG keeps 1.61.0.0 on top of main's 1.60.2.0; TODOS unions both waves' filings (incl. main's new P1 on force-exit-0 masking suite failures). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Full-suite health check found three free-suite tests failing on dev machines while CI stayed green. All three are test-side drift, fixed here with per-file green receipts; one deeper infrastructure finding is filed as a P1 TODO rather than rushed.
Fixes
getProjectEvalDir()probes the cwd-relative.claude/skills/gstack/bin/gstack-slug; withcwd: ROOTon any machine with the dev self-symlink the probe succeeds and the CLI reads an empty project-scoped dir instead of the legacy store the test seeds. Now spawns from a neutral cwd with an absolute script path, so both slug probes fail deterministically (same behavior as CI). 2/2 pass.export. Pattern is now case-insensitive. 15/15 pass./* non-fatal */marker floor of 6 went stale when the shell-free spawn (aa3bd6f0) removed the promptFile unlink and its marker. Floor is now 5 with the surviving sites named. The invariant itself holds at every remaining I/O site. 15/15 pass.Filed, not rushed
P1 in TODOS.md: at least five browse test files force-exit the shared bun process (
setTimeout(() => process.exit(0), 500)), which can exit 0 before bun prints its summary — masking real failures. That is exactly how these three failures hid behind green exit codes. Removing the exits without fixing the handle leaks they paper over would trade silent failure for suite hangs, so it gets its own focused pass (receipts and fix path in the TODO).Verification
(fail)line being a load flake (gstack-artifacts-init under a concurrent 15-way eval run; 16/16 standalone).gen-skill-docsversion-match test green after the 1.60.2.0 sync (VERSION + package.json + CHANGELOG, slot confirmed clean viagstack-next-version).🤖 Generated with Claude Code