Skip to content

VENV.md

Latest commit

 

History

History
107 lines (83 loc) · 4.32 KB

VENV.md

File metadata and controls

107 lines (83 loc) · 4.32 KB
Raw

Using Your Own Virtual Environment

Overview

Instead of using pre-built containers, you can run evaluations with your own Python virtual environment by passing --venv_path.

Setup

  1. Create a venv with Python 3.12:

    uv venv --python 3.12 /path/to/.venv

  2. Install lm-eval dependencies:

    uv pip install --python /path/to/.venv/bin/python -r requirements-venv.txt

  3. Install lighteval as isolated tool (avoids datasets version conflict):

    UV_TOOL_DIR=/path/to/.uv-tools UV_TOOL_BIN_DIR=/path/to/.venv/bin \
  uv tool install --python 3.12 \
    --with "langcodes[data]" --with "pillow" \
    "lighteval[multilingual] @ git+https://github.com/huggingface/lighteval.git"

Usage

oellm-eval schedule \
    --models HuggingFaceTB/SmolLM2-135M-Instruct \
    --task_groups multilingual \
    --venv_path /path/to/.venv

Why Two Install Steps?

lm-eval requires datasets<4.0.0 while lighteval requires datasets>=4.0.0. Installing lighteval as an isolated uv tool (like the containers do) avoids this conflict.

DCLM-core-22

dclm-core-22 needs lm-eval==0.4.9.2 (v0.4.10+ breaks agieval_lsat_ar in few-shot). Use requirements-venv-dclm.txt instead of the default requirements:

uv venv --python 3.12 dclm-core-venv
uv pip install --python dclm-core-venv/bin/python -r requirements-venv-dclm.txt

The jeopardy task is a custom task loaded via --include_path. lm-eval 0.4.9.2 has a bug where pretty_print_task assumes every task YAML lives under lm_eval/tasks/, so it crashes on any --include_path task (fixed upstream in lm-evaluation-harness#3436, but we can't upgrade since 0.4.10+ breaks agieval_lsat_ar). Apply the one-line fix to the venv after installing:

dclm-core-venv/bin/python - <<'PY'
import pathlib, lm_eval.tasks as t
f = pathlib.Path(t.__file__); s = f.read_text()
old = "        relative_yaml_path = yaml_path.relative_to(lm_eval_tasks_path)\n"
new = ("        try:\n"
       "            relative_yaml_path = yaml_path.relative_to(lm_eval_tasks_path)\n"
       "        except ValueError:\n"
       "            relative_yaml_path = yaml_path\n")
if "except ValueError" in s:
    print("already patched:", f)
elif old in s:
    f.write_text(s.replace(old, new)); print("patched:", f)
else:
    raise SystemExit(f"target line not found in {f} (unexpected lm-eval version?)")
PY
oellm-eval schedule \
    --models Qwen/Qwen3-0.6B-Base \
    --task_groups dclm-core-22 \
    --venv_path dclm-core-venv \
    --skip_checks true

Evalchemy (reasoning)

The reasoning task group includes 10 benchmarks: GSM8k, IFEval, and MBPP run via lm-eval-harness, while GPQADiamond, MATH500, LiveCodeBench, HumanEval, AIME24, AIME25, and AMC23 run via evalchemy.

Note: The evalchemy versions of GPQA and MATH500 differ from lm-eval-harness. Evalchemy uses free-form generation with CoT reasoning instead of log-likelihood scoring.

We use Ali's fork which includes a fix to randomize GPQA answer ordering to eliminate positional bias, along with context window safety fixes. The PR is yet to be merged upstream.

  1. Clone the repo at the pinned commit:

    git clone https://github.com/Ali-Elganzory/evalchemy.git evalchemy
cd evalchemy && git checkout 54ac97648230c4c3a22c3a2b93068b5a4e573f8d && cd ..

  2. Create a venv and install dependencies:

    uv venv --python 3.12 evalchemy-venv
uv pip install --python evalchemy-venv/bin/python -r requirements-venv-evalchemy.txt

  3. Run with EVALCHEMY_DIR pointing to the cloned repo:

    export HF_ALLOW_CODE_EVAL=1  # required by MBPP 
EVALCHEMY_DIR=$(pwd)/evalchemy oellm-eval schedule \
    --models HuggingFaceTB/SmolLM2-135M \
    --task_groups reasoning \
    --venv_path evalchemy-venv \
    --skip_checks true

Note: HF_ALLOW_CODE_EVAL=1 is required because MBPP (run via lm-eval-harness) uses HuggingFace's code_eval metric which executes model-generated code. The evalchemy benchmarks (GPQADiamond, MATH500, LiveCodeBench) do not require this variable as they handle code execution safely through internal guards.