feat(descent-bench): DESCENT_BENCH_BLIND_LABEL — make place_lift measure region-conditioning's contribution - #191
Merged
Conversation
…tioning's contribution #189 showed place_lift is structurally ~0 on real-photo chains: a famous name (e.g. "Grand Canyon of the Yellowstone") lets the BASELINE render the place from text alone, so both arms ceiling and the metric can't see conditioning working — the fantasy/generated-map regime the product actually serves (a made-up place the model can only render from its DRAWN form) is unrepresentable with real names. DESCENT_BENCH_BLIND_LABEL strips the real name from BOTH arms' prompts (a generic "the place marked at this spot on the map"), so place_lift isolates what the region crop contributes. The judge never sees the label — it compares images — so this changes only what the arms are TOLD, not scoring. Live validation (n=2, ~$0.61), vs the real-name run's +0.0: mean place_lift: +6.000 (was +0.000 named) Grand Canyon place with=10.0 without=0.0 lift +10.0 continuity=10.0 Church place with=2.0 without=0.0 lift +2.0 continuity= 9.0 Blinded, the baseline generates a random place (verified: a generic "YOU ARE HERE" cottage-on-a-map → place_match 0); the crop-conditioned arm reproduces the canyon (10). Region-conditioning contributes the ENTIRE identity — a +10 lift — proving the #188 edit-seam carries a real place when the name can't. Even the Chester glyph transfers a weak +2. (continuity_with=10.0, not the old spurious 0 — the #190 empty-judge-retry fix confirmed working live.) 988 backend tests pass (2 new).
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #189, which showed
place_liftis structurally ~0 on real-photo chains: a famous name lets the baseline render the place from text alone, so both arms ceiling and the metric can't see conditioning working. The regime where region-conditioning actually matters — a made-up place the model can only render from its drawn form (the product's fantasy/generated-map use case) — was unrepresentable with real place-names.What
DESCENT_BENCH_BLIND_LABEL=1strips the real name from both arms' generation prompts, substituting a generic"the place marked at this spot on the map". The judge never sees the label (it compares images), so this changes only what the arms are told, not scoring — isolating what the region crop contributes.Live validation (n=2, ~$0.61) — vs #189's real-name +0.0
Blinded, the baseline generates a random place — verified by pixels: a generic "YOU ARE HERE" cottage-on-a-map →
place_match 0. The crop-conditioned arm reproduces the canyon (10). Region-conditioning contributes the entire identity — a +10 lift — proving the #188 edit-seam carries a real place when the name can't. Even the Chester glyph transfers a weak +2.Cross-confirmation:
continuity_with=10.0, not the old spurious0— the #190 empty-judge-retry fix is working live.Tests
test_prompt_label_blinds_only_when_enabled— the pure helper (off → real name, on → generic marker).test_blind_label_strips_the_name_from_both_arms— integration: neither the with-arm edit instruction nor the without-arm base prompt contains the real name under blind mode.988 backend tests pass. Bench flags stay code-documented (not in
.env.example, consistent with the otherDESCENT_BENCH_*knobs).🤖 Generated with Claude Code