Skip to content

fix(medhelm): use quasi match for PubMedQA#4214

Open
MukundaKatta wants to merge 1 commit into
stanford-crfm:mainfrom
MukundaKatta:codex/helm-pubmedqa-quasi-match
Open

fix(medhelm): use quasi match for PubMedQA#4214
MukundaKatta wants to merge 1 commit into
stanford-crfm:mainfrom
MukundaKatta:codex/helm-pubmedqa-quasi-match

Conversation

@MukundaKatta

Copy link
Copy Markdown

Summary

  • switch PubMedQA to quasi_exact_match so punctuation-only differences do not count as wrong answers
  • update the checked-in MedHELM schema metadata to keep the leaderboard config in sync
  • add a regression test for the scenario main metric

Testing

  • python3 -m compileall src/helm/benchmark/scenarios/pubmed_qa_scenario.py src/helm/benchmark/scenarios/test_pubmed_qa_scenario.py
  • PYTHONPATH=src python3 -m pytest /tmp/helm-user/src/helm/benchmark/scenarios/test_pubmed_qa_scenario.py (fails locally: ModuleNotFoundError: No module named cattrs from repo test setup)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant