Skip to content

feat: complete journal integration into node#153

Draft
Fraser999 wants to merge 1 commit into
fraser/eng-2017/persist-journal-hashesfrom
fraser/eng-2016/complete-journal-integration
Draft

feat: complete journal integration into node#153
Fraser999 wants to merge 1 commit into
fraser/eng-2017/persist-journal-hashesfrom
fraser/eng-2016/complete-journal-integration

Conversation

@Fraser999
Copy link
Copy Markdown
Contributor

@Fraser999 Fraser999 commented May 30, 2026

Summary

Completes journal-based sync by adding a journals strategy to the node: instead of executing host blocks, a syncing node subscribes to upstream journal WebSocket sources, applies the pre-computed state diffs to storage, and hands off to live block execution once it catches up to the host tip. Builds on the in-process journal chain and JournalHashes persistence from the downstack commits.

Changes

  • Add SyncStrategy (blocks default, journals) and journal-sync settings to JournalConfig: upstream source URLs and client stall-timeout/backoff, with SIGNET_JOURNAL_* env vars, cross-field validate(), and startup misconfiguration warnings.
  • New journal_sync module. JournalIngestor applies journal-chain events to hot/cold storage - journal application and reorg unwind - and broadcasts newHeads/reorg notifications to RPC subscribers. runtime owns client/ingestor setup and lifecycle, the run-loop, journal-client construction, and checkpoint seeding from the JournalHashes table so a restarted node resumes from its storage tip rather than re-bootstrapping from genesis.
  • Wire the strategy into the node. run_journal_sync spawns the journal client and ingestor alongside the chain, applies journals until the rollup tip is within a margin of the host tip, then hands off to run_block_sync for live execution. Block sync is extracted into its own method and gains a cancellation path; a new cancellation token drives graceful shutdown.
  • Extend HostNotifier with host_tip() (used to decide when the node has caught up) and backpressures_host() (default false; the reth ExEx returns true); implement both for the reth and RPC backends.
  • Drain and discard host notifications during journal sync for backends that backpressure the host (the reth ExEx). Because the node runs in-process as a reth ExEx, leaving notifications unconsumed would fill reth's buffer and stall its pipeline; draining keeps the host moving while state is derived from the journal feed. Finished-height is never signalled, so the host retains the blocks the post-catch-up handoff backfills.
  • Require with_cancellation_token on SignetNodeBuilder.
  • End-to-end tests: journal application, transition at startup and after catch-up, source exhaustion, reorg replay, restart-resume from persisted checkpoints, and host-notification draining. The test harness gains a shared host-tip counter and a backpressure-emulating notifier.

Notes

  • The journals strategy assumes the host (reth) is already at tip when the ExEx is enabled. During reth's own pipeline backfill an un-draining ExEx stalls the host; drain-and-discard mitigates this, and operationally the ExEx is enabled only after reth syncs (SIGNET_DISABLE_EXEX).
  • Journal-synced blocks carry state but no transaction/receipt/event history, so RPC queries over the journal-synced range return empty tx data.
  • Downstream signet-exex must add .with_cancellation_token(...) when bumping to this version.
  • ENG-2016.

🤖 Generated with Claude Code

Copy link
Copy Markdown
Contributor Author

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

This stack of pull requests is managed by Graphite. Learn more about stacking.

@Fraser999 Fraser999 changed the title complete journal integration into node feat: complete journal integration into node May 30, 2026
@Fraser999 Fraser999 force-pushed the fraser/eng-2016/complete-journal-integration branch from c79164a to 8cfafdd Compare May 30, 2026 16:43
@Fraser999 Fraser999 force-pushed the fraser/eng-2017/persist-journal-hashes branch from e485653 to 3adfca6 Compare May 30, 2026 16:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant