fix(test): reliably find target proposer in sentinel_status_slash (A-1217)#24143
Merged
PhilWindle merged 2 commits intoJun 17, 2026
Merged
Conversation
The e2e helper `warpToSlotBeforeTargetProposer` intermittently threw "Target proposer ... not found with sufficient buffer within 20 epochs", masked as an e2e_p2p flake. It is a deterministic helper bug. The search window was `[currentSlot + minBufferSlots, (currentEpoch + 2) * AZTEC_EPOCH_DURATION - 1]`. With AZTEC_EPOCH_DURATION = 2 and minBufferSlots = 2 the buffer offset consumes a full epoch, so whenever the loop sits on an odd slot the window collapses to a single, always-odd slot. The proposer for each slot is a different RANDAO-shuffled committee member, so probing only odd slots never examines the even slot of any epoch — where the 1-of-6 target can be the proposer — and the loop exhausts all 20 attempts. Verified against CI run cc8c935bb167ed37: all 20 attempts probed a 1-slot window (13, 15, 17, ... 51 — all odd), the target 0x90f79b... was on the committee every epoch, and there were zero reverts. The window arithmetic, not network flakiness, is the cause. Fix: scan the full next epoch `[(currentEpoch + 1) * AZTEC_EPOCH_DURATION, (currentEpoch + 2) * AZTEC_EPOCH_DURATION - 1]`, covering both slot parities, and drop the search-time buffer entirely. The settle buffer is already provided by landing at `targetSlot - 2` (clamped so we never warp backwards), so no separate pre-search buffer is needed. The next epoch remains the only window scanned (the current epoch is partly elapsed and the second-next can revert with EpochNotStable).
Move the forward proposer-slot search into a shared `findUpcomingProposerSlot` in e2e_p2p/shared.ts, parameterized by `minLeadSlots`, and have the sentinel test use it. The helper scans forward one slot at a time from `currentSlot + minLeadSlots`, so it examines both epoch parities (the per-slot proposer is a different RANDAO-shuffled committee member, so a fixed per-epoch offset can leave the 1-of-N target unexamined when the epoch is short) and guarantees the returned slot is at least `minLeadSlots` ahead — the sentinel warps to `targetSlot - minLeadSlots` for its settle buffer with no risk of a backwards warp. EpochNotStable reverts are handled by warping one epoch forward and keeping the lead. Kept separate from `advanceToEpochBeforeProposer`, which serves a different pattern: those callers stay one epoch before the target to start sequencers, then warp to the epoch boundary, and rely on `warmupSlots` for their margin. That helper is unchanged, so its four callers are unaffected.
Collaborator
Flakey Tests🤖 says: This CI run detected 2 tests that failed, but were tolerated due to a .test_patterns.yml entry. |
fcarreiro
approved these changes
Jun 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The e2e helper
warpToSlotBeforeTargetProposer(insentinel_status_slash.parallel.test.ts) intermittently threwTarget proposer ... not found with sufficient buffer within 20 epochs, surfacing as ane2e_p2pflake. It's a deterministic helper bug, not network flakiness.The search window was derived as
searchStart = currentSlot + minBufferSlots,searchEnd = (currentEpoch + 2) * AZTEC_EPOCH_DURATION - 1. WithAZTEC_EPOCH_DURATION = 2andminBufferSlots = 2, the buffer offset consumes a full epoch, so whenevercurrentSlotis odd the window collapses to a single, always-odd slot. Because the proposer for each slot is a different RANDAO-shuffled committee member, probing only odd slots never examines the even slot of any epoch — where the 1-of-6 target can be the proposer — so the loop exhausts all attempts and throws.Log verification
Confirmed against CI run
cc8c935bb167ed37:13, 15, 17, … 51. Zero hit lines.0x90f79b…was on the committee every epoch (RANDAO-shuffled into different positions) and attested at slots 11–12 — reachable, just never at a probed (odd) slot.EpochNotStablereverts during the scan — the search ran clean, just structurally blind to even slots.Fix
Extract the proposer search into a shared
findUpcomingProposerSlotine2e_p2p/shared.ts, parameterized byminLeadSlots, and have the sentinel test use it:currentSlot + minLeadSlots, so it examines both epoch parities (fixing the odd-only blindness) and finds the RANDAO-shuffled target wherever it proposes.minLeadSlotsahead, so the sentinel can warp totargetSlot - minLeadSlotsfor its settle buffer with no risk of a backwards warp — the lead comes from the scan start, not from a per-epoch position constraint.EpochNotStableby warping one epoch forward and continuing, keeping the lead.The sentinel helper becomes a thin wrapper:
findUpcomingProposerSlot({ minLeadSlots: 2 })thenadvanceToSlot(targetSlot - 2).Kept separate from the existing
advanceToEpochBeforeProposer, which serves a genuinely different pattern: its four callers stay one epoch before the target to start sequencers, then warp to the epoch boundary, and rely onwarmupSlotsfor their warm-up margin (load-bearing — without it their proposals serialize past the slot boundary and are rejected as late). That helper is unchanged, so its callers are unaffected.Testing
shared.tsand the sentinel test changed.sentinel_status_slash.parallel.test.ts, ~20 min, real-time-dependent). The real validation is running it repeatedly to confirm the proposer is found and the suite's other two tests (which share the helper) still pass.Closes A-1217.