Skip to content

fix(l1): final mempool re-drain in build_payload_loop#6596

Open
edg-l wants to merge 3 commits into
mainfrom
fix/getpayload-mempool-redrain
Open

fix(l1): final mempool re-drain in build_payload_loop#6596
edg-l wants to merge 3 commits into
mainfrom
fix/getpayload-mempool-redrain

Conversation

@edg-l
Copy link
Copy Markdown
Contributor

@edg-l edg-l commented May 11, 2026

Summary

Closes a race in the payload builder that causes engine_getPayload to occasionally return a stale, often empty, payload when a transaction arrives in the narrow window between the loop's most recent rebuild and the cancellation triggered by engine_getPayload.

Fix: one additional synchronous build_payload call at the end of build_payload_loop, just before returning. It re-reads the mempool one last time so any transaction that landed during the race window is included.

Aim: kill the Hive Paris reorg-flake family

There is a family of four Hive Paris Engine tests that have been flaking on this repo for weeks. Three of them are already on a known-flaky ignore list in .github/scripts/check-hive-results.sh:62-66:

Invalid Missing Ancestor Syncing ReOrg, Timestamp,           EmptyTxs=False, CanonicalReOrg=False, Invalid P8
Invalid Missing Ancestor Syncing ReOrg, Timestamp,           EmptyTxs=False, CanonicalReOrg=True,  Invalid P8
Invalid Missing Ancestor Syncing ReOrg, Transaction Value,   EmptyTxs=False, CanonicalReOrg=False, Invalid P9
Invalid Missing Ancestor Syncing ReOrg, Transaction Nonce,   EmptyTxs=False, CanonicalReOrg=False, Invalid P9   ← new this week

The argument that this fix addresses all four follows.

Shared precondition

The four test names share parameter EmptyTxs=False. That parameter asserts "this test expects a payload containing at least one transaction." It is set at test-instance generation time, not a behavioral choice the runtime makes.

Shared script structure

The Invalid Missing Ancestor Syncing ReOrg family runs this exact sequence:

  1. eth_sendRawTransaction — seed the mempool
  2. engine_forkchoiceUpdatedV1 with attrs — start building
  3. engine_getPayloadV1 — pull the built payload
  4. Mutate one field of the payload (needs at least one tx in step 3)
  5. engine_newPayloadV1 with mutated payload — submit, expect INVALID
  6. Assert client behaviour on the reorg

The mutated field changes per variant (Timestamp, Transaction Value, Transaction Nonce), but step 4 only succeeds when step 3 produced a non-empty payload. So all four share the precondition "step 3 returns a payload with the seeded tx in it." The mutated field is irrelevant to the precondition.

The race produces exactly this precondition failure

Captured Hive log on the new failing variant shows:

FAIL: Unable to customize payload: no transactions available for modification

i.e. step 3 returned an empty payload. The race is in build_payload_loop:

T0   FCU-with-attrs           initial build pass, res = empty
T1   eth_sendRawTransaction   mempool.insert + tx_added.notify_waiters()
T2   loop wakes on notified   spawns pass 1 (would include the tx)
T3   engine_getPayload        cancel_token.cancel()
T4   run_until_cancelled      returns None, pass 1 result dropped
T5   loop exits               returns res = empty (from T0)

The mempool has the tx since T1 (we verified add_transaction holds the write lock until insert is complete, so the tx is durably visible to readers). The bug is solely on the build side: pass 1's result is dropped on cancellation.

Why this family is the canary

Other engine-api tests either:

  • Don't seed transactions (EmptyTxs=True variants) — empty payload is fine.
  • Use the CL-mock's own payload producer instead of ethrex (Modified Geth Module in the log) — bypasses ethrex's builder.
  • Pad the FCU-to-getPayload window with deliberate sleeps, hiding the race.

The reorg family doesn't pad — reorg semantics require tight timing between FCU and getPayload, so this family exposes the race more than any other.

Why the fix targets all four

After the loop exits — regardless of how it exited — the fix runs one more synchronous build_payload(payload). That call:

  • Reads the mempool fresh via fetch_mempool_transactions
  • Re-executes block building with whatever's in the mempool now
  • Assigns the result to res before returning

The fix doesn't care which field the test will mutate. It only guarantees that the returned payload reflects the current mempool state. If the mempool had the tx at any point before getPayload returned, the payload will include it.

Honest limits

  • Inference, not yet verified for all four. The argument is structural: same precondition, same race, same fix. The Hive run on this branch should confirm.
  • Possible alternative cause per variant. It is conceivable that one or more variants fail for a different reason (e.g. the timestamp variants might also have a sub-race on block timing). If so, the fix removes the empty-payload symptom but a different test failure could surface. The Hive log on this branch will reveal that.
  • The existing comment in check-hive-results.sh:61 is wrong. It says "These are hive framework issues, not ethrex bugs." Framework behavior is correct; the empty payload is the bug. Worth re-examining as a follow-up.

Fix

crates/blockchain/payload.rs — add one final build_payload call after the rebuild loop exits. Errors are downgraded to a warning that keeps the previous res; we never make the payload worse than the loop already produced.

Cost

One extra full payload build per engine_getPayload. For typical mempool depths this is a few milliseconds; geth and reth pay the same cost. The latency lands on the block-proposal critical path but is bounded by the same code path that runs in the rebuild loop today.

Affected paths

engine_getPayloadV1/V2/V3/V4 only. Non-builder full nodes never call get_payload and do not hit this code.

Test plan

  • cargo fmt --all
  • cargo check --workspace --all-targets
  • make lint-l1
  • Local Hive run: make run-hive SIMULATION=ethereum/engine TEST_PATTERN='engine-api/Invalid Missing Ancestor Syncing ReOrg, (Timestamp|Transaction (Value|Nonce))'
  • CI: Hive - Paris Engine tests passes without the entries in KNOWN_FLAKY_TESTS
  • No regression in Hive - Cancun Engine tests / Engine withdrawal tests / Reorg Tests

Follow-up (not in this PR)

After this PR lands and Hive shows clean on the four variants for a few runs:

  1. Remove the three entries from KNOWN_FLAKY_TESTS in .github/scripts/check-hive-results.sh.
  2. Revisit the architectural alternative: drop the periodic-rebuild loop entirely and lazy-build at getPayload time (snapshot-then-build model). That removes the timing dependency by construction.

@github-actions github-actions Bot added the L1 Ethereum client label May 11, 2026
@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 11, 2026

Lines of code report

Total lines added: 20
Total lines removed: 0
Total lines changed: 20

Detailed view
+-------------------------------------+-------+------+
| File                                | Lines | Diff |
+-------------------------------------+-------+------+
| ethrex/crates/blockchain/mempool.rs | 471   | +7   |
+-------------------------------------+-------+------+
| ethrex/crates/blockchain/payload.rs | 854   | +13  |
+-------------------------------------+-------+------+

@edg-l edg-l marked this pull request as ready for review May 11, 2026 10:00
@edg-l edg-l requested a review from a team as a code owner May 11, 2026 10:00
@ethrex-project-sync ethrex-project-sync Bot moved this to In Review in ethrex_l1 May 11, 2026
@github-actions
Copy link
Copy Markdown

🤖 Kimi Code Review

Review of crates/blockchain/payload.rs:

The fix for the race condition between mempool insertion and engine_getPayload is architecturally sound. However, there are several implementation concerns:

1. Performance: Unconditional clone of self (line 447)
Cloning the entire Blockchain struct can be expensive even if it uses Arc internally. Consider checking if there are new transactions since the last rebuild before spawning the task:

// Only rebuild if mempool has new content since last iteration
if mempool.has_new_transactions(since_timestamp) {
    // spawn rebuild
}

2. Missing timeout on blocking operation (lines 448-452)
The final rebuild has no timeout. If build_payload hangs (e.g., due to state trie computation on a huge block), this will delay the response to the Consensus Layer indefinitely:

match tokio::time::timeout(Duration::from_millis(500), 
    tokio::task::spawn_blocking(...)
).await { ... }

3. spawn_blocking usage correctness
Ensure build_payload is strictly CPU-bound (no async/await, no I/O). If it performs any async operations, spawn_blocking is the wrong abstraction and will panic or deadlock. Given this is EVM execution, it likely is CPU-bound, but verify it doesn't use block_on internally.

4. Error handling consistency
The warning logs on lines 449-450 don't include context about which payload ID this failure relates to. Add the payload ID to help debugging:

warn!(%err, payload_id = %payload.id, "Final payload rebuild failed");

5. Cancellation safety
If the outer async task is dropped while spawn_blocking is executing, the blocking thread continues consuming resources. Ensure build_payload doesn't hold locks that could stall other components.

Nit: The variable name final_self (line 447) is slightly unidiomatic; blockchain or builder would be clearer.

Overall, the logic correctly addresses the race condition, but the lack of timeout is a reliability concern for production validator operations.


Automated review by Kimi (Moonshot AI) · kimi-k2.5 · custom prompt

@github-actions
Copy link
Copy Markdown

🤖 Claude Code Review

Now I have a full picture. Here is my review:


PR Review: fix(l1): final mempool re-drain in build_payload_loop

Overview

This PR adds 13 lines to build_payload_loop in crates/blockchain/payload.rs. After the rebuild loop exits (whether from cancellation or slot timeout), a final unconditional build_payload call is made to ensure the returned payload reflects the latest mempool state. The stated goal is to eliminate a narrow race condition that causes engine_getPayload to return an empty payload when a transaction lands between the last successful rebuild and the cancellation signal.

The race analysis in the PR description is accurate and well-documented.


Correctness

The fix is correct for the race it targets. The sequence described (T0–T5 in the PR description) is real: run_until_cancelled returns None when the token fires mid-build, silently discarding the in-progress result. The final build bypasses this by running unconditionally outside the cancellable loop, reading a fresh mempool state.

Variable scope is handled correctly. Inside the while loop, let payload = payload.clone() introduces a shadowing binding that is moved into spawn_blocking. The original payload parameter remains intact and is correctly moved into the final closure on line 450. No double-free or use-after-move risk.

Error handling is appropriate. Errors in the final build are downgraded to warnings with the previous res preserved. This is strictly non-regressive: the returned payload is never worse than what the loop produced.


Issues and Suggestions

1. Unconditional execution adds latency to the slot-timeout path

The final build runs regardless of whether the loop exited via cancellation or the 12-second slot deadline. In the timeout path, no race occurred — the mempool was fully drained by the last loop iteration — so the extra build is redundant. Since build_payload executes a full mempool drain + EVM block assembly, this adds non-trivial latency to the normal (non-race) case on the critical path.

A simple guard eliminates the cost for the non-race path:

// Only rebuild if we were cancelled; slot-timeout path already drained the mempool.
if cancel_token.is_cancelled() {
    let final_self = self.clone();
    match tokio::task::spawn_blocking(move || final_self.build_payload(payload)).await {
        Ok(Ok(final_res)) => res = final_res,
        Ok(Err(err)) => warn!(%err, "Final payload rebuild failed; returning previous result"),
        Err(err) => warn!(%err, "Final payload rebuild task panicked"),
    }
}

The PR description acknowledges the cost ("a few milliseconds") but doesn't justify running it on the happy path. Given this lands on the block-proposal critical path, the guard seems worth adding.

2. Comment says "synchronous" but the call is async

// Final synchronous rebuild before returning ...
let final_self = self.clone();
match tokio::task::spawn_blocking(move || final_self.build_payload(payload)).await {

spawn_blocking + .await is not synchronous — it runs on the blocking thread pool and yields the async task while it waits. "Synchronous" here likely means "sequential, not racing against the cancellation token," which is accurate, but the wording will confuse future readers. Suggest: "Final sequential rebuild" or "unconditional final rebuild."

3. The existing spawn_blocking TODO still applies here

// TODO(#5011): this doesn't stop the building task, but only keeps it running in the background

The final spawn_blocking call has the same property: if the outer async task is dropped (e.g., the JoinHandle is dropped by the caller), the blocking thread continues to completion. This is acceptable here — the fix correctly awaits the result — but worth noting that the CPU-waste concern from #5011 applies to this path too if the caller ever drops the task before awaiting.

4. Interaction with the pre-existing background task leak (TODO #5011)

When run_until_cancelled drops the in-progress build task, that task continues to run in the background (per TODO #5011). The final build then runs a second concurrent full build on the blocking pool. Under high mempool load this could briefly double block-building CPU consumption. This is bounded (both tasks complete quickly) and pre-existing behavior, but worth noting in the commit message or a follow-up issue.


Positive Observations

  • The fix is minimal (13 lines, no new abstractions, no churn).
  • The PR description is exceptionally thorough: it correctly identifies the race, traces the exact failure sequence, names the affected Hive tests, and honestly states what has not yet been verified.
  • The warn! instead of error! for the final rebuild failure is the right severity choice — it avoids masking a real error while not hiding the fallback.
  • self.clone() for Arc<Blockchain> is cheap and consistent with the surrounding code.

Summary

The core fix is correct and well-reasoned. The main actionable feedback is:

  • Gate the final build on cancel_token.is_cancelled() to avoid a redundant full build on the slot-timeout path (Point 1 above — performance, not correctness).
  • Fix the "synchronous" terminology in the comment (Point 2 — minor).

Point 1 is the only change worth blocking on; it eliminates the always-on latency cost at zero complexity cost.


Automated review by Claude (Anthropic) · sonnet · custom prompt

@greptile-apps
Copy link
Copy Markdown

greptile-apps Bot commented May 11, 2026

Greptile Summary

This PR fixes a race condition in build_payload_loop where a transaction arriving in the mempool between the loop's last successful rebuild and the engine_getPayload cancellation signal could be dropped, causing engine_getPayload to return a stale or empty payload.

  • Adds one unconditional spawn_blocking(build_payload) call after the rebuild loop exits, re-reading the mempool and updating res before returning; errors are downgraded to warnings so the previous result is never discarded on failure.
  • Targets a family of Hive Paris Engine "Invalid Missing Ancestor Syncing ReOrg" tests that share the precondition of a non-empty payload; all four flake because the race produces exactly an empty payload at step 3 of their shared script structure.

Confidence Score: 4/5

The change is safe to merge; the extra rebuild after a natural slot-timeout is wasteful but not incorrect.

The fix is small and well-contained: one extra spawn_blocking call with correct error handling that preserves the previous result on failure. The only concern is that the final rebuild fires unconditionally on both the cancellation path (intended) and the slot-timeout path (unnecessary extra work on the proposal critical path). The logic is otherwise consistent with the existing loop and does not introduce new failure modes.

crates/blockchain/payload.rs — specifically the guard condition for the final rebuild in build_payload_loop.

Important Files Changed

Filename Overview
crates/blockchain/payload.rs Adds a final synchronous build_payload call after build_payload_loop exits to close the race window between cancellation and the last rebuild; error handling correctly downgrades failures to warnings and preserves the previous result.

Sequence Diagram

sequenceDiagram
    participant CL as Consensus Layer
    participant BPL as build_payload_loop
    participant MP as Mempool
    participant BP as build_payload

    CL->>BPL: engine_forkchoiceUpdated (start loop)
    BPL->>BP: initial build → res (empty, no txs yet)
    MP-->>BPL: tx_added.notify (eth_sendRawTransaction)
    BPL->>BP: spawn_blocking (pass 1, would include tx)
    CL->>BPL: engine_getPayload → cancel_token.cancel()
    Note over BPL: run_until_cancelled returns None, pass 1 result dropped
    Note over BPL,BP: FIX: final synchronous rebuild
    BPL->>BP: spawn_blocking (final pass, reads fresh mempool)
    BP-->>BPL: "res = payload with tx"
    BPL-->>CL: return res (non-empty)
Loading
Prompt To Fix All With AI
Fix the following 1 code review issue. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 1
crates/blockchain/payload.rs:450-454
**Final rebuild runs on every loop exit, including slot-timeout**

The final `build_payload` runs unconditionally regardless of whether the loop exited because of cancellation (the intended race fix) or because `SECONDS_PER_SLOT` elapsed naturally. In the timeout path the loop has already been rebuilding continuously for 12 seconds and the last successful rebuild is always fresh, so this extra pass adds latency without any benefit. Since `engine_getPayload` awaits the task to completion before returning to the CL, the surplus rebuild is on the block-proposal critical path every time. A simple guard like `if cancel_token.is_cancelled()` before the final `spawn_blocking` would limit the extra work to the cases where the race can actually occur.

Reviews (1): Last reviewed commit: "fix(l1): final mempool re-drain in build..." | Re-trigger Greptile

Comment thread crates/blockchain/payload.rs Outdated
Comment on lines +450 to +454
match tokio::task::spawn_blocking(move || final_self.build_payload(payload)).await {
Ok(Ok(final_res)) => res = final_res,
Ok(Err(err)) => warn!(%err, "Final payload rebuild failed; returning previous result"),
Err(err) => warn!(%err, "Final payload rebuild task panicked"),
}
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Final rebuild runs on every loop exit, including slot-timeout

The final build_payload runs unconditionally regardless of whether the loop exited because of cancellation (the intended race fix) or because SECONDS_PER_SLOT elapsed naturally. In the timeout path the loop has already been rebuilding continuously for 12 seconds and the last successful rebuild is always fresh, so this extra pass adds latency without any benefit. Since engine_getPayload awaits the task to completion before returning to the CL, the surplus rebuild is on the block-proposal critical path every time. A simple guard like if cancel_token.is_cancelled() before the final spawn_blocking would limit the extra work to the cases where the race can actually occur.

Prompt To Fix With AI
This is a comment left during a code review.
Path: crates/blockchain/payload.rs
Line: 450-454

Comment:
**Final rebuild runs on every loop exit, including slot-timeout**

The final `build_payload` runs unconditionally regardless of whether the loop exited because of cancellation (the intended race fix) or because `SECONDS_PER_SLOT` elapsed naturally. In the timeout path the loop has already been rebuilding continuously for 12 seconds and the last successful rebuild is always fresh, so this extra pass adds latency without any benefit. Since `engine_getPayload` awaits the task to completion before returning to the CL, the surplus rebuild is on the block-proposal critical path every time. A simple guard like `if cancel_token.is_cancelled()` before the final `spawn_blocking` would limit the extra work to the cases where the race can actually occur.

How can I resolve this? If you propose a fix, please make it concise.

@github-actions
Copy link
Copy Markdown

🤖 Codex Code Review

  1. The new final rebuild is unconditional, so every get_payload() now pays for an extra full build_payload() even when there was no interrupted rebuild to recover from. That is especially problematic because get_payload() holds self.payloads across to_payload().await, so the added work becomes head-of-line blocking for other payload operations too (crates/blockchain/payload.rs:365, crates/blockchain/payload.rs:444). On the proposer critical path, that latency regression can be enough to miss deadlines. I’d gate the extra pass behind a dirty_since_last_successful_build / cancelled_during_build condition instead of always rebuilding.

  2. If cancellation happens while the background spawn_blocking rebuild is running, the old builder already keeps running in the background (TODO(#5011)), and this patch immediately starts a second full rebuild afterwards (crates/blockchain/payload.rs:428, crates/blockchain/payload.rs:449). That can double CPU pressure exactly when engine_getPayload is being served. I’d avoid spawning a second builder when one is already in flight, or restructure the loop so the cancellation path can reuse the outstanding result instead of discarding it and rebuilding again.

I don’t see a direct EVM/gas/RLP/consensus correctness bug in the diff itself, but I would not merge this as-is because of the proposer-path latency regression above.

I couldn’t run cargo test in this sandbox because Cargo/rustup need write access under /home/runner/.cargo and /home/runner/.rustup, which is blocked here. I also didn’t find existing coverage around build_payload_loop, so a regression test for this cancellation race would be worth adding.


Automated review by OpenAI Codex · gpt-5.4 · custom prompt

Comment thread crates/blockchain/payload.rs Outdated
Comment thread crates/blockchain/payload.rs
@edg-l edg-l force-pushed the fix/getpayload-mempool-redrain branch from 9bfb9d4 to b028dd2 Compare May 12, 2026 07:49
edg-l added 3 commits May 13, 2026 17:50
`build_payload_loop` rebuilds the payload on every `tx_added`
notification, but when `engine_getPayload` cancels the loop mid-rebuild
the in-progress build is dropped (`run_until_cancelled` returns `None`)
and the returned `res` falls back to the previous, possibly empty,
payload.

Race timeline:
  T0  FCU-with-attrs   -> build pass 0, res = empty
  T1  tx arrives       -> notify_waiters fires
  T2  loop wakes       -> spawns pass 1 (would include the tx)
  T3  getPayload calls cancel.cancel()
  T4  run_until_cancelled returns None, pass 1 result dropped
  T5  loop exits, returns res = empty

Add one final synchronous `build_payload` call after the loop exits.
The mempool is read fresh, so any tx that landed during the race window
is included.

Cost: one extra block-build pass per `engine_getPayload` (few ms for
typical mempool sizes; geth/reth pay the same). The latency is on the
critical path of block proposal but is bounded by the existing payload
build time.
Bot review consensus (Claude/Codex/Greptile/Kimi) flagged that the
final rebuild ran on every loop exit, including the slot-timeout
path where the loop already drained the mempool continuously for
12s. Gate it on `cancel_token.is_cancelled()` so only the race
path pays the extra build.

Also re-word the comment ("synchronous" was misleading — the call
is async via `spawn_blocking + await`) and rename `final_self` to
`blockchain`.

Remove the three Hive Paris "Invalid Missing Ancestor Syncing ReOrg"
entries from KNOWN_FLAKY_TESTS in check-hive-results.sh. Local Hive
runs on this branch passed the full reorg-family pattern twice; with
the race closed there is no reason to keep bypassing them. CI will
verify on the real ethpandaops hive image.
Replace the is_cancelled() guard on the final build_payload pass with a
monotonic mempool tx_seq comparison. Catches the slot-timeout select!
race in addition to the cancellation race, and skips the rebuild when
res is already fresh.
@edg-l edg-l force-pushed the fix/getpayload-mempool-redrain branch from b028dd2 to ce610ad Compare May 13, 2026 16:32
@github-actions
Copy link
Copy Markdown

⚠️ Known Issues — intentionally skipped tests

Source: docs/known_issues.md

Known Issues

Tests intentionally excluded from CI. Source of truth for the Known
Issues
section the L1 workflow appends to each ef-tests job summary
and posts as a sticky PR comment.

Hive — bal-devnet-6 Amsterdam consume-engine tests — 32 functions / 54 cases

Same root cause as the blockchain-runner skip list (see EF Tests —
Blockchain
below): snobal-devnet-6 fixtures expect bal-devnet-6 spec
semantics, but our impl runs ahead due to the bal-devnet-7-prep
set_delegation SELFDESTRUCT-style refund subtraction. These fixtures
are routed through hive's eels/consume-engine simulator and produce
the same failures. Excluded via KNOWN_EXCLUDED_TESTS (substring
match on test_<name>[fork_Amsterdam, anchoring to the Amsterdam fork
so legacy Prague/Osaka variants still run).

Affected EELS test functions (32)
  • test_auth_refund_block_gas_accounting
  • test_auth_refund_bypasses_one_fifth_cap
  • test_auth_state_gas_scales_with_cpsb
  • test_auth_with_calldata_and_access_list
  • test_auth_with_multiple_sstores
  • test_authorization_exact_state_gas_boundary
  • test_authorization_to_precompile_address
  • test_authorization_with_sstore
  • test_bal_7702_delegation_clear
  • test_bal_7702_delegation_create
  • test_bal_7702_delegation_update
  • test_bal_7702_double_auth_reset
  • test_bal_7702_double_auth_swap
  • test_bal_7702_null_address_delegation_no_code_change
  • test_bal_all_transaction_types
  • test_bal_selfdestruct_to_7702_delegation
  • test_bal_withdrawal_to_7702_delegation
  • test_duplicate_signer_authorizations
  • test_existing_account_auth_header_gas_used_uses_worst_case
  • test_existing_account_refund
  • test_existing_account_refund_enables_sstore
  • test_existing_auth_with_reverted_execution_preserves_intrinsic
  • test_many_authorizations_state_gas
  • test_mixed_auths_header_gas_used_uses_worst_case
  • test_mixed_new_and_existing_auths
  • test_mixed_valid_and_invalid_auths
  • test_multi_tx_block_auth_refund_and_sstore
  • test_multiple_refund_types_in_one_tx
  • test_simple_gas_accounting
  • test_sstore_state_gas_all_tx_types
  • test_transfer_with_all_tx_types
  • test_varying_calldata_costs

EF Tests — Stateless coverage narrowed to EIP-8025 optional-proofs

make -C tooling/ef_tests/blockchain test calls test-stateless-zkevm
instead of test-stateless. The zkevm@v0.3.3 fixtures are filled against
bal@v5.6.1, out of sync with current bal spec; the broad target trips ~549
fixtures. Re-broaden once the zkevm bundle is regenerated.

Why and resolution path

PR #6527 broadened
test-stateless to extract the entire for_amsterdam/ tree from the
zkevm bundle and run all of it under --features stateless; combined with
this branch's bal-devnet-6+ semantics (and bal-devnet-7-prep
set_delegation re-application) that scope produces ~549
GasUsedMismatch / ReceiptsRootMismatch /
BlockAccessListHashMismatch failures.

test-stateless-zkevm filters cargo to the eip8025_optional_proofs
suite, which still validates the stateless harness without the bal-version
mismatch.

Re-broaden by switching test: back to test-stateless in
tooling/ef_tests/blockchain/Makefile once the zkevm bundle is regenerated
against the current bal spec.

EF Tests — Blockchain bal-devnet-6 (Amsterdam fork) — 74 tests

snobal-devnet-6 fixtures expect bal-devnet-6 spec semantics, but our impl
runs ahead due to the bal-devnet-7-prep set_delegation SELFDESTRUCT-style
refund subtraction. Skipped in
tooling/ef_tests/blockchain/tests/all.rs::SKIPPED_BASE, anchored on
[fork_Amsterdam so legacy Prague / Osaka variants still run.

Bucket breakdown (74 total) and resolution path
EIP Bucket Count
EIP-7702 set_code_txs 24
EIP-7702 set_code_txs_2 15
EIP-7702 gas 1
EIP-8037 state_gas_set_code 17
EIP-8037 state_gas_pricing 1
EIP-8037 state_gas_sstore 1
EIP-7928 block_access_lists_eip7702 8
EIP-7928 block_access_lists 1
EIP-7778 gas_accounting 3
EIP-7708 transfer_logs 1
EIP-7976 refunds 1
EIP-1344 chainid (Amsterdam fork-transition fixture) 1
Total 74

Re-enable once we either:

  • (a) bump fixtures to a snobal-devnet-7 release that locks in the new
    accounting; or
  • (b) revert the bal-devnet-7-prep subtraction for bal-devnet-6
    compatibility.
Full test list (74)

EIP-7702 — for_amsterdam/prague/eip7702_set_code_tx/set_code_txs/

  • delegation_clearing
  • delegation_clearing_and_set
  • delegation_clearing_failing_tx
  • delegation_clearing_tx_to
  • eoa_tx_after_set_code
  • ext_code_on_chain_delegating_set_code
  • ext_code_on_self_delegating_set_code
  • ext_code_on_self_set_code
  • ext_code_on_set_code
  • many_delegations
  • nonce_overflow_after_first_authorization
  • nonce_validity
  • reset_code
  • self_code_on_set_code
  • self_sponsored_set_code
  • set_code_multiple_valid_authorization_tuples_same_signer_increasing_nonce
  • set_code_multiple_valid_authorization_tuples_same_signer_increasing_nonce_self_sponsored
  • set_code_to_log
  • set_code_to_non_empty_storage_non_zero_nonce
  • set_code_to_self_destruct
  • set_code_to_self_destructing_account_deployed_in_same_tx
  • set_code_to_sstore
  • set_code_to_sstore_then_sload
  • set_code_to_system_contract

EIP-7702 — for_amsterdam/prague/eip7702_set_code_tx/set_code_txs_2/

  • call_pointer_to_created_from_create_after_oog_call_again
  • call_to_precompile_in_pointer_context
  • contract_storage_to_pointer_with_storage
  • delegation_replacement_call_previous_contract
  • double_auth
  • pointer_measurements
  • pointer_normal
  • pointer_reentry
  • pointer_resets_an_empty_code_account_with_storage
  • pointer_reverts
  • pointer_to_pointer
  • pointer_to_precompile
  • pointer_to_static
  • pointer_to_static_reentry
  • static_to_pointer

EIP-7702 — for_amsterdam/prague/eip7702_set_code_tx/gas/

  • account_warming

EIP-8037 — for_amsterdam/amsterdam/eip8037_state_creation_gas_cost_increase/state_gas_set_code/

  • auth_refund_block_gas_accounting
  • auth_refund_bypasses_one_fifth_cap
  • auth_with_calldata_and_access_list
  • auth_with_multiple_sstores
  • authorization_exact_state_gas_boundary
  • authorization_to_precompile_address
  • authorization_with_sstore
  • duplicate_signer_authorizations
  • existing_account_auth_header_gas_used_uses_worst_case
  • existing_account_refund
  • existing_account_refund_enables_sstore
  • existing_auth_with_reverted_execution_preserves_intrinsic
  • many_authorizations_state_gas
  • mixed_auths_header_gas_used_uses_worst_case
  • mixed_new_and_existing_auths
  • mixed_valid_and_invalid_auths
  • multi_tx_block_auth_refund_and_sstore

EIP-8037 — state_gas_pricing/

  • auth_state_gas_scales_with_cpsb

EIP-8037 — state_gas_sstore/

  • sstore_state_gas_all_tx_types

EIP-7928 — for_amsterdam/amsterdam/eip7928_block_level_access_lists/block_access_lists_eip7702/

  • bal_7702_delegation_clear
  • bal_7702_delegation_create
  • bal_7702_delegation_update
  • bal_7702_double_auth_reset
  • bal_7702_double_auth_swap
  • bal_7702_null_address_delegation_no_code_change
  • bal_selfdestruct_to_7702_delegation
  • bal_withdrawal_to_7702_delegation

EIP-7928 — block_access_lists/

  • bal_all_transaction_types

EIP-7778 — for_amsterdam/amsterdam/eip7778_block_gas_accounting_without_refunds/gas_accounting/

  • multiple_refund_types_in_one_tx
  • simple_gas_accounting
  • varying_calldata_costs

EIP-7708 — for_amsterdam/amsterdam/eip7708_eth_transfer_logs/transfer_logs/

  • transfer_with_all_tx_types

EIP-7976 — for_amsterdam/amsterdam/eip7976_increase_calldata_floor_cost/refunds/

  • gas_refunds_from_data_floor

EIP-1344 — for_amsterdam/istanbul/eip1344_chainid/chainid/

  • chainid (Amsterdam fork-transition fixture)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

L1 Ethereum client

Projects

Status: In Review

Development

Successfully merging this pull request may close these issues.

3 participants