Add ArxivRollBench scenario

[liangzid]

Summary

This PR adds ArxivRollBench as a HELM scenario and run spec.

ArxivRollBench is a rolling arXiv benchmark for recent scientific text reasoning. It is built from newly released arXiv papers and evaluates three multiple-choice task types:

• sequencing
• cloze
• next-sequence prediction

The benchmark spans multiple arXiv domains and has rolling releases (2024b, 2025a, 2026a) to reduce overfitting to static benchmark snapshots. The paper has been accepted to AAAI 2026: https://ojs.aaai.org/index.php/AAAI/article/view/41098. Project website: https://arxivrollbench.github.io/.

Why this benchmark fits HELM

ArxivRollBench is intended to fill a gap in temporally refreshed scientific reasoning evaluation. Many widely used academic reasoning benchmarks are static, so model scores can become harder to interpret as benchmark data ages or becomes exposed during model development. ArxivRollBench focuses on recent scientific text and releases new benchmark snapshots over time, which makes it useful for measuring whether LLMs can reason over contemporary research writing rather than only memorizing older public examples.

This matches HELM's goal of broad, reproducible, and transparent model evaluation across meaningful capabilities.

Changes

• Adds ArxivRollBenchScenario.
• Adds an arxivrollbench run spec.
• Supports:
  • release: 2024b, 2025a, 2026a, or all
  • domain: cs, q_fin / q-fin, math, physics, stat, q_bio / q-bio, econ, eess, or all
  • task_type: s, c, p, or all
  • split: compact or full
• Uses HELM's existing multiple-choice joint adapter.
• Uses exact match as the main metric.
• Adds offline tests for dataset paths, label normalization, domain aliases, and HELM Instance conversion.

The default run spec uses split=compact, corresponding to the lightweight -50 datasets. Full benchmark runs can pass split=full.

Example

helm-run \
  --run-entries arxivrollbench:release=2026a,domain=cs,task_type=s,split=compact,model=text \
  --suite arxivrollbench \
  --max-eval-instances 50

Validation

• python -m compileall -q src/helm/benchmark/scenarios/arxivrollbench_scenario.py src/helm/benchmark/run_specs/arxivrollbench_run_specs.py src/helm/benchmark/scenarios/test_arxivrollbench_scenario.py
• python -m black --check src/helm/benchmark/scenarios/arxivrollbench_scenario.py src/helm/benchmark/run_specs/arxivrollbench_run_specs.py src/helm/benchmark/scenarios/test_arxivrollbench_scenario.py
• PYTHONPATH=src python -m pytest -o addopts='' src/helm/benchmark/scenarios/test_arxivrollbench_scenario.py
• PYTHONPATH=src python -m helm.benchmark.run --run-entries arxivrollbench:release=2026a,domain=cs,task_type=s,split=compact,model=simple/model1 --suite arxivrollbench_smoke --max-eval-instances 1 --dry-run --output-path /tmp/helm-arxivrollbench-smoke

The smoke run loads the compact Hugging Face dataset, builds one HELM scenario state, and verifies the prompt path without external model calls.