Skip to content

fix(medhelm): score PubMedQA with punctuation-tolerant metric#4277

Open
MukundaKatta wants to merge 1 commit into
stanford-crfm:mainfrom
MukundaKatta:codex/medhelm-pubmedqa-punctuation
Open

fix(medhelm): score PubMedQA with punctuation-tolerant metric#4277
MukundaKatta wants to merge 1 commit into
stanford-crfm:mainfrom
MukundaKatta:codex/medhelm-pubmedqa-punctuation

Conversation

@MukundaKatta

Copy link
Copy Markdown

Summary

  • compute quasi_leave_articles_exact_match for the MedHELM PubMedQA run spec
  • add the metric to the MedHELM schema
  • use it as the PubMedQA main metric so answers like "A." are scored consistently

Fixes #4187.

Tests

  • python3 -m compileall -q src/helm/benchmark/run_specs/medhelm_run_specs.py src/helm/benchmark/metrics/evaluate_reference_metrics.py
  • python3 schema assertion for quasi_leave_articles_exact_match and PubMedQA main metric
  • git diff --check

@goktugozkanmd

Copy link
Copy Markdown

One small consistency check before merge: the PR changes the MedHELM schema so PubMedQA reports quasi_leave_articles_exact_match as the main metric, but PubMedQAScenario.get_metadata() still declares main_metric="exact_match".

If that metadata is only legacy display data, no issue. If any docs, registry output, or downstream summary path reads it, users could still see PubMedQA described as exact match while the MedHELM schema is scoring the punctuation tolerant metric.

I would update the scenario metadata too, or leave a short note that schema_medhelm is the only source used for this leaderboard row.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

quasi_exact_match does not seem to work in MedHELM - scenario "PubMedQA - EM" - Punctuation issue?

2 participants