fix(metrics): gpqa CoT fallback answer regex uses escaped brackets (never matches)#4323
Open
WatchTree-19 wants to merge 1 commit into
Open
fix(metrics): gpqa CoT fallback answer regex uses escaped brackets (never matches)#4323WatchTree-19 wants to merge 1 commit into
WatchTree-19 wants to merge 1 commit into
Conversation
extract_answer's second regex was written \[aA\]nswer: (literal brackets) instead of the [aA]nswer: character class, so it never matched real 'Answer: X' outputs and correct GPQA/MMLU-Pro CoT answers were scored wrong. +test. Signed-off-by: WatchTree-19 <119982314+WatchTree-19@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
problem
in
src/helm/benchmark/metrics/gpqa_chain_of_thought_metric.py, the fallback answer-extraction regex is written with escaped brackets:\[aA\]matches the literal characters[aA], not the intended character class[aA](matchaorA). so this branch only ever matches the impossible literal string"[aA]nswer:"and never a real output like"Answer: B". it's effectively dead code: when a model answers in theAnswer: Xformat (a common deviation from the requestedThe correct answer is (X)), the letter is not extracted and the response is scored as wrong (chain_of_thought_correctness = 0).this metric is wired into
capabilities_run_specs.pyfor bothgpqaandmmlu_prochain-of-thought run specs.fix
still satisfies the original
[answer: F]example (\.*matches the[,[aA]nswer:matchesanswer:).test
added
test_gpqa_chain_of_thought_metric.py. against the current code,Answer: C/The answer: B/answer: E/...Answer: D/[answer: F]all extractNone(fail); with the fix they extract the correct letter. the primaryThe correct answer is (X)path (handled by the first regex) and the no-match case are unchanged. flake8 + black clean (line-length 120).found by reading the metric. this only fixes a fallback branch that currently never fires, so it doesn't change any already-working extraction.