fix(ci): avoid full prover proof-window reorg on spartan train#24089
Draft
AztecBot wants to merge 2 commits into
Draft
fix(ci): avoid full prover proof-window reorg on spartan train#24089AztecBot wants to merge 2 commits into
AztecBot wants to merge 2 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Investigation
The merge-queue failure on PR #23971 was not just a 20m timeout. The idle tail starts after
e2e_prover_full_reallogsAwaiting proof for the previous epoch.The prover did begin epoch 1 proving and gathered the txs, but about one proof-submission window later it logged:
After that, the test kept polling
waitForProven(...)while the sequencer repeatedly attempted to build empty checkpoints withminTxs=1, so it looked idle until the outer wrapper sent TERM.With
PIPELINING_SETUP_OPTS, this test uses 12s slots and 32-slot epochs. The defaultaztecProofSubmissionEpochs: 1gives roughly a 6.4 minute proof-submission window, which is shorter than real epoch proving in this test. Other long-running pipelined e2e tests use a largeaztecProofSubmissionEpochsvalue to avoid pruning/reorgs while test-controlled proving is still running.Verification
cipassed on run https://github.com/AztecProtocol/aztec-packages/actions/runs/27550976312/job/81436926308./bootstrap.sh ciattempted locally, but this ref has no plaincibootstrap command and exits withUnknown command: ci../bootstrap.sh ci-full-no-test-cacheattempted as the relevant merge-queue path; it reached build/test orchestration but could not complete in this container (/var/run/docker.sockunavailable, Rust 1.89.0 warning, early test-engine failure).git diff --checkbash -O extglob -n yarn-project/end-to-end/bootstrap.shCI_FULL=1 ./yarn-project/end-to-end/bootstrap.sh test_cmds | rg "e2e_prover_full_real"emits the 45m wrapper timeout.yarn prettier --check ...attempted fromyarn-project, but this checkout has no Yarn install state (Couldn't find the node_modules state file).