fix(ci): give e2e_multiple_blobs avm_check_circuit more CPUs and time#24085
Draft
AztecBot wants to merge 1 commit into
Draft
fix(ci): give e2e_multiple_blobs avm_check_circuit more CPUs and time#24085AztecBot wants to merge 1 commit into
AztecBot wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Bump the per-test resource budget for
avm_check_circuitone2e_multiple_blobstransactions from the defaultTIMEOUT=30s/CPUS=2toCPUS=8:TIMEOUT=180sinyarn-project/end-to-end/bootstrap.sh.Why
The
avm-check-circuitjob has been failing on everynextcommit (runs 1786 Jun 14, 1787, 1788, and 1789 — the run that triggered this fix). All of them fail on the same test:bb-avm avm_check_circuitfor ane2e_multiple_blobstx, exiting with code124(thetimeout -v 30swrapper killing it).Per-test log for the failing run (ci.aztec-labs.com/ca2b62a6682db814):
The
e2e_multiple_blobstx produces a ~700k-row AVM circuit. With only 2 CPUs, trace generation alone takes ~18s and the circuit check then runs out of the remaining ~11s before the 30s wall-clock timeout fires. Every other dumped tx checks in 4–7s, so the tight default budget is correct for them — only the multi-blob circuits are large enough to time out.This is exactly the scenario the
WARNINGcomment inavm_check_circuit_cmdsanticipated ("transactions could need more CPU and MEM than we allocate by default … they might start timing out").Change
casepattern already used intest_cmds):e2e_multiple_blobs*getsCPUS=8:TIMEOUT=180s; all other txs keep the tight30s/2-CPUdefault to preserve parallelism. More CPUs cut both trace generation and the row check (both parallel), and the higher timeout leaves generous headroom as the circuit grows.WARNINGcomment to point future maintainers at the newcasestatement when another test dir starts timing out.Verified the command generator emits
...:CPUS=8:TIMEOUT=180s:NAME=avm_cc_e2e_multiple_blobs_...only for the multi-blob inputs and leaves the others untouched (the laterTIMEOUToverrides the default perci3/source_test_params).MEMdefaults toCPUS*4 = 32g, well above the observed 3.9 GiB peak.This is a CI resource/timeout fix only — no AVM circuit or simulation logic changes.
Created by claudebox · group:
slackbot