Skip to content

fix(descent-bench): route the region-conditioned arm through the real edit seam (+5 hardening riders) - #188

Merged
eren23 merged 1 commit into
mainfrom
fix/descent-bench-edit-arm
Aug 2, 2026
Merged

fix(descent-bench): route the region-conditioned arm through the real edit seam (+5 hardening riders)#188
eren23 merged 1 commit into
mainfrom
fix/descent-bench-edit-arm

Conversation

@eren23

@eren23 eren23 commented Aug 2, 2026

Copy link
Copy Markdown
Owner

The headline: the descent bench was measuring a fiction

The descent bench's region-conditioned "with" arm called generate_image with the region crop as a reference_url. But fal's text-to-image endpoints silently ignore reference_urls (accept-but-drop, same as the fresh-gen nano endpoints per PR #109). So the arm that was supposed to prove region-conditioning helps was actually unconditioned — a fresh generation wearing a reference URL — and place_lift sat structurally ~0.

Fix: route the with-arm through the seam the product actually ships, with the crop as the edit SOURCE so reference pixels bite:

  • interior chains → build_enter_instruction + edit_image on the enter_scene model
  • exterior closeups → build_zoom_instruction + continue_image (Kontext), the closeup rung

Two regression guards (test_with_arm_rides_edit_seam_not_generate, test_exterior_with_arm_rides_zoom_continue) spy on which seam fires, so the with-arm can never silently slip back to generate_image.

Five hardening themes riding along

  1. Coordinate-scale ladder → shared module. Extract the fraction/percent/per-mille coercion out of llm/click.py into providers/coordinate_scale.py and wire it into detector.py + llm/extraction.py (was click-only). Closes the long-standing "world extraction/detector have the same scale flip" coverage loss — pixel-looking coords now drop instead of edge-clamping.
  2. Enter-retry model swap (off by default). ENTER_RETRY_MODEL_SWAP keeps attempt 0 on the router's production pick and swaps only judged retry attempts to a cheaper nano-banana-2/edit, threaded via a new render_for_attempt hook through render_looptap + both benches. An explicit request-level image_model override is respected on every attempt.
  3. Judge transport hardening. Pillow-downscale the VLM judge payload (VLM_JUDGE_MAX_SIDE_PX/VLM_JUDGE_JPEG_QUALITY) and route the judge call through _create_with_retry. Multi-MB bench JPEGs were large enough inline to reset OpenRouter/Gemini reads; falls back to the original bytes if Pillow can't decode.
  4. Web: disable prefetch/precompute warming in World Mode. World taps route through geometry / cold resolver context, but the precompute effect was seeding candidate-only warm entries carrying stale enter_as/place_form classifications. worldEnabled now early-returns from both the precompute effect and the hover-prefetch handler.
  5. Bench ergonomics. Artifact reuse (*_REUSE_ARTIFACTS), arm/case filters (VIEW_BENCH_ARMS/VIEW_BENCH_CASES), style-ref compaction before fal upload, judge retry+backoff, positioning-probe toggle.

Env docs updated in both .env.example files; conftest.py scrubs the two new env vars.

Verification

  • Backend: 983 passed, 2 skipped, coverage 86.71% (floor 85; new coordinate_scale.py + model_router.py at 100%).
  • Web: tsc --noEmit clean; pre-commit eslint --fix + ruff --fix clean.

Not in this PR

The live descent-bench run that empirically proves place_lift now lifts post-fix is not included here — it costs real fal image + VLM judge spend. Happy to run it as a follow-up (with DESCENT_BENCH_REUSE_ARTIFACTS off for a true measurement) once someone confirms the fal balance.

🤖 Generated with Claude Code

… edit seam

The descent bench's "with" arm called generate_image with a region
reference_url, but fal's text-to-image silently ignores reference URLs — so
the region-conditioned arm was actually unconditioned and place_lift sat
structurally ~0. Route it through the product seam instead: edit_image for
interior chains, continue_image (Kontext) for exterior closeups, with the crop
as the edit SOURCE so reference pixels actually bite. Regression guards spy on
which seam fires so the arm can't silently revert to generate_image.

Five hardening themes ride along on the same branch:

- coordinate scale: extract the fraction/percent/per-mille ladder into a shared
  providers/coordinate_scale.py and wire it into detector + extraction (was
  click-only), closing the world extraction/detector scale-flip coverage loss.
- enter-retry model swap (off by default): ENTER_RETRY_MODEL_SWAP keeps attempt
  0 on the router pick and swaps only judged retries to a cheaper nano edit
  slug, threaded via render_for_attempt through render_loop -> tap + benches.
- judge transport: Pillow-downscale the VLM judge payload
  (VLM_JUDGE_MAX_SIDE_PX/QUALITY) and route the judge call through
  _create_with_retry, so multi-MB bench JPEGs stop resetting OpenRouter reads.
- web: disable prefetch/precompute warming in World Mode — world taps route
  through geometry/cold resolver and were reading stale candidate-only warm
  entries with wrong enter_as/place_form.
- bench ergonomics: artifact reuse, arm/case filters, style-ref compaction,
  judge retry+backoff, positioning-probe toggle.

Backend: 983 passed, coverage 86.71% (floor 85). Web tsc clean.
@cursor

cursor Bot commented Aug 2, 2026

Copy link
Copy Markdown

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

@eren23

eren23 commented Aug 2, 2026

Copy link
Copy Markdown
Owner Author

Live descent-bench run (n=1, ~$0.30)

Ran make eval-descent (DESCENT_BENCH_RUN=1). One golden chain resolves: village-medieval-manor :: 'Church' -> wm-chester-cathedral-nave.

Result: mean place_lift: +0.000 (style_lift: −7.500), continuity_with: 9.0, both arms place_match = 2.0.

The seam fix is verified working. Logs confirm the with-arm rode image.edit (fal-ai/nano-banana-pro/edit, refs conditioning) and the without-arm rode image.generate (refs: 0) — no more generate_image-with-ignored-refs. continuity_with = 9.0 proves the region conditioning actually bit: the generated interior is coherent with the cropped map region.

Why place_lift is still 0 here — the chain is degenerate, not the fix. Verified by eyeballing all four artifacts:

  • region input = a vintage village map where "Church" is a tiny cross glyph (a small country parish).
  • with-arm = a faithful illustrated small-church interior (stays in the map's medium — correct descent behavior).
  • without-arm = a photoreal small parish church.
  • real child = a photo of Chester Cathedral — a colossal Gothic cathedral.

The map never depicted Chester Cathedral, so neither arm is the same place → place_match = 2.0 for both is the honest, correct score. style_lift = −7.5 is by-design (the with-arm preserves the illustrated medium, which is exactly why place_match is the medium-agnostic headline).

Takeaway: the branch's real achievement — the bench now measures the true product seam instead of an unconditioned fiction — is confirmed. Demonstrating place_lift > 0 needs a golden chain whose real child photo actually corresponds to a distinctive place depicted in the parent map (corpus work, out of scope for this PR).

Artifacts (gitignored) in tests/map_corpus/overlays/descent-wm-chester-cathedral-nave-{region,with,without}.jpg.

@eren23
eren23 merged commit 716668f into main Aug 2, 2026
9 checks passed
eren23 added a commit that referenced this pull request Aug 3, 2026
…tioning's contribution (#191)

#189 showed place_lift is structurally ~0 on real-photo chains: a famous name
(e.g. "Grand Canyon of the Yellowstone") lets the BASELINE render the place from
text alone, so both arms ceiling and the metric can't see conditioning working —
the fantasy/generated-map regime the product actually serves (a made-up place
the model can only render from its DRAWN form) is unrepresentable with real names.

DESCENT_BENCH_BLIND_LABEL strips the real name from BOTH arms' prompts (a generic
"the place marked at this spot on the map"), so place_lift isolates what the
region crop contributes. The judge never sees the label — it compares images —
so this changes only what the arms are TOLD, not scoring.

Live validation (n=2, ~$0.61), vs the real-name run's +0.0:
  mean place_lift: +6.000  (was +0.000 named)
    Grand Canyon  place with=10.0 without=0.0  lift +10.0   continuity=10.0
    Church        place with=2.0  without=0.0  lift  +2.0   continuity= 9.0
Blinded, the baseline generates a random place (verified: a generic "YOU ARE
HERE" cottage-on-a-map → place_match 0); the crop-conditioned arm reproduces the
canyon (10). Region-conditioning contributes the ENTIRE identity — a +10 lift —
proving the #188 edit-seam carries a real place when the name can't. Even the
Chester glyph transfers a weak +2. (continuity_with=10.0, not the old spurious
0 — the #190 empty-judge-retry fix confirmed working live.)

988 backend tests pass (2 new).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant