fix(judge): retry a transient empty/unparseable judge reply instead of banking a silent 0 - #190
Merged
Merged
Conversation
…f banking a silent 0 _create_with_retry only retries API-level errors. A 200 response with EMPTY content — a transient judge-model failure — parses to a loud UNPARSEABLE score-0 in _parse_judgement WITHOUT raising, so it slipped past that retry AND the benches' exception-only _judge_with_retry, banking a blank reply as a real 0. This is what surfaced as continuity_with=0.0 (raw_head:"") on the #189 descent run. Fix at the single chokepoint every score_* routes through: _ask_judge now re-issues the call while the reply can't be parsed (rationale startswith "UNPARSEABLE"), bounded by VLM_JUDGE_EMPTY_RETRIES (default 3), then falls back to the loud UNPARSEABLE 0. Both the view-loop and the paid benches benefit; a legit reply (incl. a real 0) never burns an extra call. 986 backend tests pass (3 new), coverage 86.81%.
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
eren23
added a commit
that referenced
this pull request
Aug 3, 2026
…tioning's contribution (#191) #189 showed place_lift is structurally ~0 on real-photo chains: a famous name (e.g. "Grand Canyon of the Yellowstone") lets the BASELINE render the place from text alone, so both arms ceiling and the metric can't see conditioning working — the fantasy/generated-map regime the product actually serves (a made-up place the model can only render from its DRAWN form) is unrepresentable with real names. DESCENT_BENCH_BLIND_LABEL strips the real name from BOTH arms' prompts (a generic "the place marked at this spot on the map"), so place_lift isolates what the region crop contributes. The judge never sees the label — it compares images — so this changes only what the arms are TOLD, not scoring. Live validation (n=2, ~$0.61), vs the real-name run's +0.0: mean place_lift: +6.000 (was +0.000 named) Grand Canyon place with=10.0 without=0.0 lift +10.0 continuity=10.0 Church place with=2.0 without=0.0 lift +2.0 continuity= 9.0 Blinded, the baseline generates a random place (verified: a generic "YOU ARE HERE" cottage-on-a-map → place_match 0); the crop-conditioned arm reproduces the canyon (10). Region-conditioning contributes the ENTIRE identity — a +10 lift — proving the #188 edit-seam carries a real place when the name can't. Even the Chester glyph transfers a weak +2. (continuity_with=10.0, not the old spurious 0 — the #190 empty-judge-retry fix confirmed working live.) 988 backend tests pass (2 new).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the judge-retry gap flagged in #189.
The bug
_ask_judgewraps its API call in_create_with_retry, which only retries API-level errors. But a200response with empty content — a transient judge-model failure — flows into_parse_judgement, which (correctly, by design) returns a loudUNPARSEABLEscore-0 without raising. So the empty reply slipped past_create_with_retryand the benches' exception-only_judge_with_retry, banking a blank response as a real0.This is exactly what surfaced as
continuity_with=0.0(raw_head:"") on the #189 descent run — the Kontext with-arm visibly continued the crop, but a transient empty judge reply scored it 0.The fix
At the single chokepoint every
score_*routes through,_ask_judgenow re-issues the call while the reply can't be parsed (rationalestartswith("UNPARSEABLE")), bounded byVLM_JUDGE_EMPTY_RETRIES(default 3, clamped 1–5), then falls back to the loudUNPARSEABLE0. One place → both the production view-loop judges and the paid benches benefit. A legit reply (including a real0) never burns an extra call.Tests (TDD — added failing first)
test_ask_judge_retries_empty_then_succeeds— empty body → retries → banks the valid score.test_ask_judge_gives_up_loudly_after_retries— persistent empty → bounded retries → loud UNPARSEABLE 0.test_ask_judge_does_not_retry_a_legit_score— a valid reply (incl. legit 0) costs exactly one call.The existing
_parse_judgementcontract tests (legit-0 vs UNPARSEABLE) still pass unchanged. 986 backend tests pass, coverage 86.81%.🤖 Generated with Claude Code