Skip to content

fix(metrics): f1_score silently returns 0 when gold/pred is a single article word (#2298)#4315

Open
nac7 wants to merge 1 commit into
stanford-crfm:mainfrom
nac7:fix/f1-score-article-elision
Open

fix(metrics): f1_score silently returns 0 when gold/pred is a single article word (#2298)#4315
nac7 wants to merge 1 commit into
stanford-crfm:mainfrom
nac7:fix/f1-score-article-elision

Conversation

@nac7

@nac7 nac7 commented Jun 9, 2026

Copy link
Copy Markdown

What

f1_score calls normalize_text() on both gold and pred before computing
token overlap. normalize_text strips English articles (a, an, the)
via re.sub(r"\b(a|an|the)\b", ...).

When the gold (or pred) string is entirely one of those words — for
example, the letter "A" in a multiple-choice benchmark like MMLU —
article removal produces an empty string. set("".split()) == set(), and
nltk.metrics.scores.f_measure(set(), ...) returns None, which
f1_score silently maps to 0.0.

Concrete failure (pre-fix):

>>> f1_score("A", "A")
0.0   # should be 1.0
>>> f1_score("An", "An")
0.0   # should be 1.0

Any benchmark that reports f1_score alongside single-letter gold answers
(MMLU, ARC, HellaSwag …) will show 0.0 for correct predictions. The same
bug applies to the pred side, and to non-English benchmarks where "a" is
a meaningful token rather than an English article.

Fix

src/helm/benchmark/metrics/evaluate_reference_metrics.py

After computing the normalised token sets, if either set is empty, retry
with should_remove_articles=False before calling f_measure. Strings
that contain articles alongside other tokens are unaffected — article
removal fires as before.

# Before (one line):
ret = f_measure(set(normalize_text(gold).split()), set(normalize_text(pred).split()))

# After:
gold_tokens = set(normalize_text(gold).split())
pred_tokens = set(normalize_text(pred).split())
if not gold_tokens:
    gold_tokens = set(normalize_text(gold, should_remove_articles=False).split())
if not pred_tokens:
    pred_tokens = set(normalize_text(pred, should_remove_articles=False).split())
ret = f_measure(gold_tokens, pred_tokens)

Tests

src/helm/benchmark/metrics/test_evaluate_reference_metrics.py — four new
test functions (no prior f1_score coverage existed):

Test Checks
test_f1_score_normal Baseline partial-overlap and no-overlap cases
test_f1_score_article_gold Regression: "A"/"An"/"The" as gold — now scores 1.0 for a correct pred
test_f1_score_article_in_longer_string Multi-word strings: article removal still fires
test_f1_score_empty_pred Empty prediction still yields 0

Fixes #2298

normalize_text removes English articles ("a", "an", "the") before
computing F1.  When the gold or pred string consists entirely of such
a word -- for example, the letter "A" in MMLU-style multiple-choice
benchmarks -- article removal empties the token set.  f_measure on an
empty set returns None, which f1_score silently converts to 0.0, so a
correct prediction scores 0 instead of 1.

Fix: after normalization, if either token set is empty, fall back to
normalization with should_remove_articles=False before calling
f_measure.  Multi-word strings that contain articles alongside other
tokens are unaffected.

Adds test_evaluate_reference_metrics.py coverage for f1_score:
  - test_f1_score_normal         (baseline behavior unchanged)
  - test_f1_score_article_gold   (regression: "A"/"An"/"The" as gold)
  - test_f1_score_article_in_longer_string (article removal still fires
                                            for multi-word strings)
  - test_f1_score_empty_pred     (empty prediction still yields 0)

Fixes stanford-crfm#2298
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

English-specific quirks of F1 overlap score

1 participant