Skip to content

fix(ci): avoid full prover proof-window reorg on spartan train#24089

Draft
AztecBot wants to merge 2 commits into
merge-train/spartanfrom
cb/fix-spartan-prover-timeout
Draft

fix(ci): avoid full prover proof-window reorg on spartan train#24089
AztecBot wants to merge 2 commits into
merge-train/spartanfrom
cb/fix-spartan-prover-timeout

Conversation

@AztecBot

@AztecBot AztecBot commented Jun 15, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Keep the full real-prover e2e proof-submission window open long enough for real epoch proving.
  • Keep the CI wrapper timeout aligned with the test's existing 45m Jest budget.

Investigation

The merge-queue failure on PR #23971 was not just a 20m timeout. The idle tail starts after e2e_prover_full_real logs Awaiting proof for the previous epoch.

The prover did begin epoch 1 proving and gathered the txs, but about one proof-submission window later it logged:

Epoch blocks changed underfoot
Error running epoch 1 prover job: Epoch proving failed: Proving cancelled
Error proving epoch 1: EmptyEpochError: No blocks found for epoch 1

After that, the test kept polling waitForProven(...) while the sequencer repeatedly attempted to build empty checkpoints with minTxs=1, so it looked idle until the outer wrapper sent TERM.

With PIPELINING_SETUP_OPTS, this test uses 12s slots and 32-slot epochs. The default aztecProofSubmissionEpochs: 1 gives roughly a 6.4 minute proof-submission window, which is shorter than real epoch proving in this test. Other long-running pipelined e2e tests use a large aztecProofSubmissionEpochs value to avoid pruning/reorgs while test-controlled proving is still running.

Verification

  • PR CI: ci passed on run https://github.com/AztecProtocol/aztec-packages/actions/runs/27550976312/job/81436926308
  • ./bootstrap.sh ci attempted locally, but this ref has no plain ci bootstrap command and exits with Unknown command: ci.
  • ./bootstrap.sh ci-full-no-test-cache attempted as the relevant merge-queue path; it reached build/test orchestration but could not complete in this container (/var/run/docker.sock unavailable, Rust 1.89.0 warning, early test-engine failure).
  • git diff --check
  • bash -O extglob -n yarn-project/end-to-end/bootstrap.sh
  • CI_FULL=1 ./yarn-project/end-to-end/bootstrap.sh test_cmds | rg "e2e_prover_full_real" emits the 45m wrapper timeout.
  • yarn prettier --check ... attempted from yarn-project, but this checkout has no Yarn install state (Couldn't find the node_modules state file).

@AztecBot AztecBot added ci-draft Run CI on draft PRs. ci-no-fail-fast Sets NO_FAIL_FAST in the CI so the run is not aborted on the first failure claudebox Owned by claudebox. it can push to this PR. labels Jun 15, 2026
@AztecBot AztecBot changed the title fix(ci): extend full prover e2e timeout on spartan train fix(ci): avoid full prover proof-window reorg on spartan train Jun 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-draft Run CI on draft PRs. ci-no-fail-fast Sets NO_FAIL_FAST in the CI so the run is not aborted on the first failure claudebox Owned by claudebox. it can push to this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant