Skip to content

fix(ci): give e2e_multiple_blobs avm_check_circuit more CPUs and time#24085

Draft
AztecBot wants to merge 1 commit into
nextfrom
cb/avm-cc-multiple-blobs-timeout-27536567840
Draft

fix(ci): give e2e_multiple_blobs avm_check_circuit more CPUs and time#24085
AztecBot wants to merge 1 commit into
nextfrom
cb/avm-cc-multiple-blobs-timeout-27536567840

Conversation

@AztecBot

Copy link
Copy Markdown
Collaborator

What

Bump the per-test resource budget for avm_check_circuit on e2e_multiple_blobs transactions from the default TIMEOUT=30s / CPUS=2 to CPUS=8:TIMEOUT=180s in yarn-project/end-to-end/bootstrap.sh.

Why

The avm-check-circuit job has been failing on every next commit (runs 1786 Jun 14, 1787, 1788, and 1789 — the run that triggered this fix). All of them fail on the same test: bb-avm avm_check_circuit for an e2e_multiple_blobs tx, exiting with code 124 (the timeout -v 30s wrapper killing it).

Per-test log for the failing run (ci.aztec-labs.com/ca2b62a6682db814):

Resources: CPUS=2 MEM=8g TIMEOUT=30s
09:50:17 Simulating...
09:50:18 Generating trace... (mem: 825.80 MiB)
09:50:36 Checking circuit... (mem: 3911.55 MiB)
09:50:36 Running check (with skippable) circuit over 700560 rows.
09:50:47 timeout: sending signal TERM to command 'bash'

The e2e_multiple_blobs tx produces a ~700k-row AVM circuit. With only 2 CPUs, trace generation alone takes ~18s and the circuit check then runs out of the remaining ~11s before the 30s wall-clock timeout fires. Every other dumped tx checks in 4–7s, so the tight default budget is correct for them — only the multi-blob circuits are large enough to time out.

This is exactly the scenario the WARNING comment in avm_check_circuit_cmds anticipated ("transactions could need more CPU and MEM than we allocate by default … they might start timing out").

Change

  • Add a per-test resource override (same case pattern already used in test_cmds): e2e_multiple_blobs* gets CPUS=8:TIMEOUT=180s; all other txs keep the tight 30s/2-CPU default to preserve parallelism. More CPUs cut both trace generation and the row check (both parallel), and the higher timeout leaves generous headroom as the circuit grows.
  • Reword the WARNING comment to point future maintainers at the new case statement when another test dir starts timing out.

Verified the command generator emits ...:CPUS=8:TIMEOUT=180s:NAME=avm_cc_e2e_multiple_blobs_... only for the multi-blob inputs and leaves the others untouched (the later TIMEOUT overrides the default per ci3/source_test_params). MEM defaults to CPUS*4 = 32g, well above the observed 3.9 GiB peak.

This is a CI resource/timeout fix only — no AVM circuit or simulation logic changes.


Created by claudebox · group: slackbot

@AztecBot AztecBot added ci-draft Run CI on draft PRs. ci-no-fail-fast Sets NO_FAIL_FAST in the CI so the run is not aborted on the first failure claudebox Owned by claudebox. it can push to this PR. labels Jun 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-draft Run CI on draft PRs. ci-no-fail-fast Sets NO_FAIL_FAST in the CI so the run is not aborted on the first failure claudebox Owned by claudebox. it can push to this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant