Skip to content

fix(evals): make llm_run.py fault-tolerant (save partial snapshot on failure)#699

Open
c990china wants to merge 2 commits into
JuliusBrussee:mainfrom
c990china:fix/eval-llm-run-fault-tolerance
Open

fix(evals): make llm_run.py fault-tolerant (save partial snapshot on failure)#699
c990china wants to merge 2 commits into
JuliusBrussee:mainfrom
c990china:fix/eval-llm-run-fault-tolerance

Conversation

@c990china

Copy link
Copy Markdown

Problem

evals/llm_run.py ran every prompt x arm with no error handling and subprocess.run(check=True). A single failed claude -p call (auth expiry, rate limit, transient network, or a bad CAVEMAN_EVAL_MODEL) aborted the entire multi-dollar run and wrote nothing - SNAPSHOT.write_text(...) is the last statement in main().

Reproduced locally: mocking subprocess.run to fail on the 2nd call makes the real main() crash with CalledProcessError after 2 calls, with no results.json produced.

Fix

  • safe_run() wraps each claude call, catching CalledProcessError / TimeoutExpired / any exception and recording an ERROR: marker instead of crashing.
  • run_claude now takes a timeout (via CAVEMAN_EVAL_TIMEOUT, default 300s) so a hung call cannot block the whole run forever.
  • On any failure the run still writes results.partial.json (progress saved) and exits non-zero; the canonical results.json is left untouched, so CI measure.py stays correct.

Test

  • Add evals/test_llm_run.py (stdlib unittest, no claude CLI needed): the success path writes the canonical snapshot; a mid-run failure no longer crashes, writes the partial snapshot, and leaves the canonical untouched.

Verified locally: both cases pass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant