fix(descent-bench): route the region-conditioned arm through the real edit seam (+5 hardening riders) - #188
Conversation
… edit seam The descent bench's "with" arm called generate_image with a region reference_url, but fal's text-to-image silently ignores reference URLs — so the region-conditioned arm was actually unconditioned and place_lift sat structurally ~0. Route it through the product seam instead: edit_image for interior chains, continue_image (Kontext) for exterior closeups, with the crop as the edit SOURCE so reference pixels actually bite. Regression guards spy on which seam fires so the arm can't silently revert to generate_image. Five hardening themes ride along on the same branch: - coordinate scale: extract the fraction/percent/per-mille ladder into a shared providers/coordinate_scale.py and wire it into detector + extraction (was click-only), closing the world extraction/detector scale-flip coverage loss. - enter-retry model swap (off by default): ENTER_RETRY_MODEL_SWAP keeps attempt 0 on the router pick and swaps only judged retries to a cheaper nano edit slug, threaded via render_for_attempt through render_loop -> tap + benches. - judge transport: Pillow-downscale the VLM judge payload (VLM_JUDGE_MAX_SIDE_PX/QUALITY) and route the judge call through _create_with_retry, so multi-MB bench JPEGs stop resetting OpenRouter reads. - web: disable prefetch/precompute warming in World Mode — world taps route through geometry/cold resolver and were reading stale candidate-only warm entries with wrong enter_as/place_form. - bench ergonomics: artifact reuse, arm/case filters, style-ref compaction, judge retry+backoff, positioning-probe toggle. Backend: 983 passed, coverage 86.71% (floor 85). Web tsc clean.
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
Live descent-bench run (n=1, ~$0.30)Ran Result: The seam fix is verified working. Logs confirm the with-arm rode Why place_lift is still 0 here — the chain is degenerate, not the fix. Verified by eyeballing all four artifacts:
The map never depicted Chester Cathedral, so neither arm is the same place → Takeaway: the branch's real achievement — the bench now measures the true product seam instead of an unconditioned fiction — is confirmed. Demonstrating Artifacts (gitignored) in |
…tioning's contribution (#191) #189 showed place_lift is structurally ~0 on real-photo chains: a famous name (e.g. "Grand Canyon of the Yellowstone") lets the BASELINE render the place from text alone, so both arms ceiling and the metric can't see conditioning working — the fantasy/generated-map regime the product actually serves (a made-up place the model can only render from its DRAWN form) is unrepresentable with real names. DESCENT_BENCH_BLIND_LABEL strips the real name from BOTH arms' prompts (a generic "the place marked at this spot on the map"), so place_lift isolates what the region crop contributes. The judge never sees the label — it compares images — so this changes only what the arms are TOLD, not scoring. Live validation (n=2, ~$0.61), vs the real-name run's +0.0: mean place_lift: +6.000 (was +0.000 named) Grand Canyon place with=10.0 without=0.0 lift +10.0 continuity=10.0 Church place with=2.0 without=0.0 lift +2.0 continuity= 9.0 Blinded, the baseline generates a random place (verified: a generic "YOU ARE HERE" cottage-on-a-map → place_match 0); the crop-conditioned arm reproduces the canyon (10). Region-conditioning contributes the ENTIRE identity — a +10 lift — proving the #188 edit-seam carries a real place when the name can't. Even the Chester glyph transfers a weak +2. (continuity_with=10.0, not the old spurious 0 — the #190 empty-judge-retry fix confirmed working live.) 988 backend tests pass (2 new).
The headline: the descent bench was measuring a fiction
The descent bench's region-conditioned "with" arm called
generate_imagewith the region crop as areference_url. But fal's text-to-image endpoints silently ignorereference_urls(accept-but-drop, same as the fresh-gen nano endpoints per PR #109). So the arm that was supposed to prove region-conditioning helps was actually unconditioned — a fresh generation wearing a reference URL — andplace_liftsat structurally ~0.Fix: route the with-arm through the seam the product actually ships, with the crop as the edit SOURCE so reference pixels bite:
build_enter_instruction+edit_imageon theenter_scenemodelbuild_zoom_instruction+continue_image(Kontext), the closeup rungTwo regression guards (
test_with_arm_rides_edit_seam_not_generate,test_exterior_with_arm_rides_zoom_continue) spy on which seam fires, so the with-arm can never silently slip back togenerate_image.Five hardening themes riding along
llm/click.pyintoproviders/coordinate_scale.pyand wire it intodetector.py+llm/extraction.py(was click-only). Closes the long-standing "world extraction/detector have the same scale flip" coverage loss — pixel-looking coords now drop instead of edge-clamping.ENTER_RETRY_MODEL_SWAPkeeps attempt 0 on the router's production pick and swaps only judged retry attempts to a cheapernano-banana-2/edit, threaded via a newrender_for_attempthook throughrender_loop→tap+ both benches. An explicit request-levelimage_modeloverride is respected on every attempt.VLM_JUDGE_MAX_SIDE_PX/VLM_JUDGE_JPEG_QUALITY) and route the judge call through_create_with_retry. Multi-MB bench JPEGs were large enough inline to reset OpenRouter/Gemini reads; falls back to the original bytes if Pillow can't decode.enter_as/place_formclassifications.worldEnablednow early-returns from both the precompute effect and the hover-prefetch handler.*_REUSE_ARTIFACTS), arm/case filters (VIEW_BENCH_ARMS/VIEW_BENCH_CASES), style-ref compaction before fal upload, judge retry+backoff, positioning-probe toggle.Env docs updated in both
.env.examplefiles;conftest.pyscrubs the two new env vars.Verification
coordinate_scale.py+model_router.pyat 100%).tsc --noEmitclean; pre-commiteslint --fix+ruff --fixclean.Not in this PR
The live descent-bench run that empirically proves
place_liftnow lifts post-fix is not included here — it costs real fal image + VLM judge spend. Happy to run it as a follow-up (withDESCENT_BENCH_REUSE_ARTIFACTSoff for a true measurement) once someone confirms the fal balance.🤖 Generated with Claude Code