fix(metrics): f1_score silently returns 0 when gold/pred is a single article word (#2298)#4315
Open
nac7 wants to merge 1 commit into
Open
fix(metrics): f1_score silently returns 0 when gold/pred is a single article word (#2298)#4315nac7 wants to merge 1 commit into
nac7 wants to merge 1 commit into
Conversation
normalize_text removes English articles ("a", "an", "the") before
computing F1. When the gold or pred string consists entirely of such
a word -- for example, the letter "A" in MMLU-style multiple-choice
benchmarks -- article removal empties the token set. f_measure on an
empty set returns None, which f1_score silently converts to 0.0, so a
correct prediction scores 0 instead of 1.
Fix: after normalization, if either token set is empty, fall back to
normalization with should_remove_articles=False before calling
f_measure. Multi-word strings that contain articles alongside other
tokens are unaffected.
Adds test_evaluate_reference_metrics.py coverage for f1_score:
- test_f1_score_normal (baseline behavior unchanged)
- test_f1_score_article_gold (regression: "A"/"An"/"The" as gold)
- test_f1_score_article_in_longer_string (article removal still fires
for multi-word strings)
- test_f1_score_empty_pred (empty prediction still yields 0)
Fixes stanford-crfm#2298
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
f1_scorecallsnormalize_text()on both gold and pred before computingtoken overlap.
normalize_textstrips English articles (a,an,the)via
re.sub(r"\b(a|an|the)\b", ...).When the gold (or pred) string is entirely one of those words — for
example, the letter
"A"in a multiple-choice benchmark like MMLU —article removal produces an empty string.
set("".split()) == set(), andnltk.metrics.scores.f_measure(set(), ...)returnsNone, whichf1_scoresilently maps to0.0.Concrete failure (pre-fix):
Any benchmark that reports
f1_scorealongside single-letter gold answers(MMLU, ARC, HellaSwag …) will show 0.0 for correct predictions. The same
bug applies to the pred side, and to non-English benchmarks where
"a"isa meaningful token rather than an English article.
Fix
src/helm/benchmark/metrics/evaluate_reference_metrics.pyAfter computing the normalised token sets, if either set is empty, retry
with
should_remove_articles=Falsebefore callingf_measure. Stringsthat contain articles alongside other tokens are unaffected — article
removal fires as before.
Tests
src/helm/benchmark/metrics/test_evaluate_reference_metrics.py— four newtest functions (no prior
f1_scorecoverage existed):test_f1_score_normaltest_f1_score_article_gold"A"/"An"/"The"as gold — now scores 1.0 for a correct predtest_f1_score_article_in_longer_stringtest_f1_score_empty_predFixes #2298