test(descent-bench): add a real non-degenerate golden chain (Grand Canyon of the Yellowstone) + verify Yellowstone desc - #189
Merged
Conversation
…y Yellowstone desc
Adds an exterior-closeup descent chain the bench can actually place-match:
forest-yellowstone-1904 :: 'Grand Canyon of the Yellowstone' -> a sourced
public-domain (CC BY-SA 4.0) Artist-Point photo of the same canyon, view=exterior.
The 1904 Northern Pacific panorama draws the canyon with a distinctive yellow-orange
gorge, so the region crop carries real place identity (unlike the seeded manor::church
glyph). Verifies forest-yellowstone-1904.json (was vlm_draft): corrects the
yellowstone-lake pos, which pointed at the bottom title edge instead of its own
border centroid; all other entity positions image-inspected via the overlay.
Live re-bench finding (n=2, ~$0.61): place_lift is +0.0 on BOTH chains, but for
OPPOSITE reasons, which characterizes the metric:
- Chester (glyph -> grand cathedral): both arms FLOOR at place_match 2.0 — the
map has no identity to transfer and the child isn't the mapped place.
- Grand Canyon (distinctive gorge -> real canyon): both arms CEILING at 10.0 —
the model renders "Grand Canyon of the Yellowstone" perfectly from the NAME
alone (verified: the text-only baseline is a photoreal dead-ringer), so
region-conditioning has no headroom to add.
place_lift only has room to move when the place NAME is uninformative but the CROP
is distinctive — i.e. the product's fantasy/generated-map regime, which a real-photo
corpus can't represent. The healthy signal here is the ABSOLUTE place_match_with:
2.0 (Chester) -> 10.0 (Grand Canyon) proves the edit-seam pipeline carries a real
place to ceiling when correspondence exists. The lift metric is really a guardrail
(with >= without), not a positive-on-famous-places number.
Note: the Grand Canyon run logged continuity_with=0.0 from a transient
judge.unparseable (empty judge response scored 0, not retried); visually the
Kontext with-arm clearly continues the crop. Bench judge-retry only fires on
exceptions, not on empty-parse — a separate robustness gap.
Corpus images are sha-pinned + re-fetched (make corpus-fetch), not git-tracked;
the manifest row is the artifact.
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
eren23
added a commit
that referenced
this pull request
Aug 2, 2026
…f banking a silent 0 (#190) _create_with_retry only retries API-level errors. A 200 response with EMPTY content — a transient judge-model failure — parses to a loud UNPARSEABLE score-0 in _parse_judgement WITHOUT raising, so it slipped past that retry AND the benches' exception-only _judge_with_retry, banking a blank reply as a real 0. This is what surfaced as continuity_with=0.0 (raw_head:"") on the #189 descent run. Fix at the single chokepoint every score_* routes through: _ask_judge now re-issues the call while the reply can't be parsed (rationale startswith "UNPARSEABLE"), bounded by VLM_JUDGE_EMPTY_RETRIES (default 3), then falls back to the loud UNPARSEABLE 0. Both the view-loop and the paid benches benefit; a legit reply (incl. a real 0) never burns an extra call. 986 backend tests pass (3 new), coverage 86.81%.
eren23
added a commit
that referenced
this pull request
Aug 3, 2026
…tioning's contribution (#191) #189 showed place_lift is structurally ~0 on real-photo chains: a famous name (e.g. "Grand Canyon of the Yellowstone") lets the BASELINE render the place from text alone, so both arms ceiling and the metric can't see conditioning working — the fantasy/generated-map regime the product actually serves (a made-up place the model can only render from its DRAWN form) is unrepresentable with real names. DESCENT_BENCH_BLIND_LABEL strips the real name from BOTH arms' prompts (a generic "the place marked at this spot on the map"), so place_lift isolates what the region crop contributes. The judge never sees the label — it compares images — so this changes only what the arms are TOLD, not scoring. Live validation (n=2, ~$0.61), vs the real-name run's +0.0: mean place_lift: +6.000 (was +0.000 named) Grand Canyon place with=10.0 without=0.0 lift +10.0 continuity=10.0 Church place with=2.0 without=0.0 lift +2.0 continuity= 9.0 Blinded, the baseline generates a random place (verified: a generic "YOU ARE HERE" cottage-on-a-map → place_match 0); the crop-conditioned arm reproduces the canyon (10). Region-conditioning contributes the ENTIRE identity — a +10 lift — proving the #188 edit-seam carries a real place when the name can't. Even the Chester glyph transfers a weak +2. (continuity_with=10.0, not the old spurious 0 — the #190 empty-judge-retry fix confirmed working live.) 988 backend tests pass (2 new).
eren23
added a commit
that referenced
this pull request
Aug 5, 2026
…ster (#195) AUDIT_BOX §4's remaining R&D asks whether metric pose recovery from generated images is possible. The bench's similarity fit (pos_aligned) fits and scores on the SAME label-matched pairs, so it forgives drift but cannot say whether the register would place an entity it was NOT fitted on — the actual recovery question for a read-side inverse register in the estimator seam. The probe answers it: Alignment.invert (observed -> expected/metric frame) + pose_probe_loo — for each matched pair, fit on the other N-1 and compare the held-out observation's raw error to its invert-registered error. Reported per cell as zero-weight diagnostics (pose_n / pose_err_raw / pose_err_recovered / pose_gain), the align_* pattern. Live phase-1 result (make eval-recon, 4 fresh yellowstone cells, $1.02 — the make target hardcodes RECON_BENCH_RUN=1; cached cells keep stored scores so they carry no probe fields): clamped fits (align_scale=0.5): mean gain -35.0 (inverting an untrusted fit DOUBLES error — dividing residuals by the clamped 0.5) healthy fits (scale ~1.1): mean gain +0.5 (real but marginal) Phase-1 conclusion: a naive product-side inverse register is UNSAFE on exactly the drift-y cells it would target — any POSE_REGISTER_FIX must gate on fit health (unclamped scale + residual + matches), and the recoverable regime's gain is marginal. True §4 recovery needs a richer model than 2-DOF similarity (rotation/perspective) — the written next step. Side observation from the same run (NOT this change): the sweep gate flagged height_order 0.650 < 0.67 band floor — first live eval-recon since #189 verified the yellowstone desc into the corpus; likely needs a corpus-change re-baseline, reported separately. 3 new golden tests (invert round-trip incl. flip; drift recovers ~exactly; scatter must not fake recovery) + the <3-pairs guard. 16 recon tests green.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #188. That PR fixed the descent bench's with-arm seam but its only golden chain (
manor::churchglyph → Chester Cathedral photo) was degenerate — the map never depicted the cathedral, soplace_liftfloored at 0. This adds a chain with genuine parent↔child place correspondence and re-benches live.What's added
forest-yellowstone-1904 :: 'Grand Canyon of the Yellowstone' → wm-grand-canyon-of-yellowstone-and-lower-falls-wyom(view:exterior). Child = a sourced CC-BY-SA 4.0 Artist-Point photo of the same canyon (Erik Whalen, via Wikimedia Commons; sha-pinned in the manifest, fetched bymake corpus-fetch). The 1904 Northern Pacific panorama draws the canyon as a distinctive yellow gorge, so the region crop carries real identity.forest-yellowstone-1904.json(wasvlm_draft): correctedyellowstone-lakepos, which pointed at the bottom title edge instead of its own border centroid; other entity positions image-inspected viaoverlay.py.Live re-bench (n=2, ~$0.61) — an instructive null
Both chains yield
place_lift ≈ 0, but for opposite reasons — which characterizes the metric:place_liftonly moves when the place NAME is uninformative but the CROP is distinctive — the product's fantasy/generated-map regime, which a real-photo corpus can't represent. Famous real landmarks always ceiling the baseline. The signal that did move is the absoluteplace_match_with: 2.0 → 10.0, proving the edit-seam pipeline (from #188) carries a real place to ceiling when correspondence exists. Soplace_liftis best read as a guardrail (with ≥ without), not a positive-on-famous-places headline.Notes
continuity_with=0.0from a transientjudge.unparseable(empty judge response scored 0, not retried — bench judge-retry only fires on exceptions, not empty-parse). Visually the Kontext with-arm clearly continues the crop; a separate robustness gap, not fixed here.🤖 Generated with Claude Code