Create a multilingual eval suite

[geoalgo]

Multilingual Eval Suite

This is going to be based on Maria's report. Here is a list based on what's implemented so far in oellm-eval

• Belebele
• XWinograd
• Global MMLU
• INCLUDE
• XStoryCloze
• MGSM
• GPQA
• ARC challenge_mt
• SIB-200
• Global PIQA
• Open Subtitles
  • Added opensubtitles-eu #70
• FLORES-200
  • Fix flores200 metric wiring (chrf++) and cover full EU language set #101
• GLOBAL MGSM
  • Add Global-MGSM #103
• XCopa
  • Added Xcopa #66
• Hellaswag
  • Add Okapi HellaSwag #102
• PolyMath
  • Add PolyMath #94
• MMMLU
  • add MMMLU #98
• MMLU-ProX
  • Add MMLU-ProX #99
• MultiBlimp
  • Add MultiBLIMP #65
• xcsqa
  • Add XCSQA #100
• MathNet
  • Note: Contains Images as part of the prompt
• PolyMath-TUM
• XNLI
  • Add XNLI EU task group #105
• PAWS-x
  • Add PAWS-X EU task group #106
• Swesat
• WikiAnn NER / PAN-X
• Tatoeba challenge
• Taxi1500
• MENLO
• Doclevel-mt
• BFCL
• tau2-bench
• BLeND
• Toxigen

[geoalgo]

Thanks for adding the list @haideraltahan !
We can remove m-ArenaHard from the list as it requires judge-arena / LLM judges.

[Vinay152003]

I'd like to pick up XNLI from the list. I have it working as a custom task group (xnli-eu, the six EU languages XNLI ships: bg, de, el, en, es, fr) following the xcsqa pattern — with one difference worth flagging: the harness's built-in xnli tasks still point at the legacy xnli dataset id, which recent huggingface_hub versions reject and which can't be resolved from a pre-downloaded cache offline, so the custom tasks use the canonical facebook/xnli instead. Opening a PR shortly.

[Vinay152003]

Picking up PAWS-X next — same approach as XNLI (#105): custom pawsx-eu task group with the four EU languages the dataset ships (de, en, es, fr), using the canonical google-research-datasets/paws-x id since the built-in tasks point at the legacy bare paws-x id (same offline-cache problem). PR incoming.