release: gloas#795
Open
mattevans wants to merge 67 commits into
Open
Conversation
…928 support fix(proto): implement proper handling of Gloas beacon blocks and their fields refactor(block_access_list.go): add functionality to decode and process block access list data from Gloas blocks fix(execution_transaction.go): adjust processing logic to handle transactions in Gloas blocks fix(beacon_blob.go): modify kzg commitments extraction for Gloas blocks to ensure proper retrieval fix(proposed_validator_block.go): update logic to correctly compute Gloas block size and handle transactions
feat(canonical): add ExecutionPayloadSlotNumber to canonical block batch fix(beacon): update GloasBlock message retrieval to use GetMessage method fix(event-ingester): fix casting of GloasBlock message to use GetMessage method test: add tests for BlockAccessList related functionalities style: reorder imports for clarity in conversion test file chore: refactor existing append functions to include ExecutionPayloadSlotNumber handling
Implement BeaconBlockAccessList type for handling access list events in the ETH V2 protocol. This adds support for new event type, enhances event routing, and allows for better structuring of events in the system.
…r improved stability chore(go.mod): update github.com/ethpandaops/beacon dependency to version v0.67.1 chore(go.sum): update checksum for the new github.com/ethpandaops/beacon version style(block_access_list.go): remove unnecessary empty line for cleaner code readability
… to ensure compatibility and access to bug fixes chore(go.sum): update checksum for ethpandaops/beacon to match new version
…or improved functionality fix(go.mod): change ethcore version to v0.0.0-20260323002709-50f492685a52 for compatibility fix(go.sum): update checksums for go-ethereum, c-kzg-4844, and ethcore dependencies fix(conversion.go): update access.Code to access.CodeChanges for clarity in block access list creation fix(conversion_test.go): adjust test to set CodeChange correctly in the test setup for block access list
… in Kafka and ClickHouse This change enhances the data pipeline by adding the beacon block access list event transformation and corresponding sinks, enabling better handling of this specific event data for analytics and processing downstream.
refactor: bals structure rlp decoded
… properties to use camelCase style(vector-kafka-clickhouse.yaml): standardize field naming conventions to match expected API format
…better customization
* master: chore(buf.yaml): update linting rules to exclude specific checks for better customization
…d storage read tracking fix(conversion): adjust handling of storage reads to use new BlockAccessListStorageRead structure
feat(gloas): add BAL storage reads and bump deps to bal-devnet-3
… with no state changes to enhance parallel execution analysis feat(proto): update BlockAccessListChange to include "touched" in change_type options for better clarity on account status
feat(gloas): add BAL storage reads and touched accounts
feat: prepare epbs types
…ClickHouse tables to improve data structure for valid bid tracking and payload attestation flow feat(migrations): create various new tables for tracking execution payloads, bids, and proposer preferences to enhance transaction handling and performance fix(migrations): reverse EIP-7732 changes in down scripts to ensure consistent rollback functionality and data integrity
feat(migrations): add epbs support
…BS conditional block routing
…): remove execution payload logic for Gloas blocks to simplify the code base and adapt to new event handling requirements
… execution payload and proposer preferences. feat(libp2p): add event names for libp2p Gossipsub events to manage payload attestation and proposer preferences. fix(routes_test): remove deprecated events from test coverage related to pending routes for ePBS. refactor(canonical): implement flattening methods for execution payload bid, payload attestation, and beacon block related events.
…ata methods to reduce code repetition and improve readability feat(beacon): add Slot field to beaconSlotEpochPropagationAdditional struct for better data handling
…con_block_payload_attestation.go): remove unnecessary blank lines and add lint ignore comments for clarity and improved readability
…riable for clarity and consistency refactor(canonical_beacon_block_payload_attestation.go): use explicit iteration for bit counting to address gosec G115 warning
…derivers feat: wire ePBS SSE handlers + cannon derivers
… up migrations for DataColumnSidecar tables to simplify schema chore(beacon): remove SidecarSlot and SidecarBeaconBlockRoot fields from beaconApiEthV1EventsDataColumnSidecar to reflect schema change chore(libp2p): remove SidecarSlot and SidecarBeaconBlockRoot fields from libp2pGossipsubDataColumnSidecar to reflect schema change
feat: add payload_status to ForkChoiceNodeV2
* master: (35 commits) feat(discovery): support bootstrapRpcUrl for RPC-backed Status (#815) cannon: add direct-to-clickhouse output sink (#808) [codex] improve execution mimicry peering (#811) fix(kafka): exporter size check should use MaxMessageBytes, not FlushBytes (#810) fix(kafka): make Producer.MaxMessageBytes configurable (#809) ci(smoke): probe vector-http-kafka via host port (busybox has no /dev/tcp) ci(govulncheck): whitelist unfixable docker/docker server-side vulns ci: pick latest Go for govulncheck, wait for vector-http-kafka in smoke test consumoor: remove kzg_commitments array from data column sidecar table consumoor: add clientId config for shared Kafka client-id fix attestation order by fix: bump grpc to v1.79.3 for GO-2026-4762, fix buf action hash consumoor: fix lint issues from CI consumoor: fix review findings — config strictness, failure isolation, watcher bug yeah (#804) consumoor: drop intentionally unsupported events; NAK unknown routes consumoor: fix error handling gaps — DLQ invalid events, permanent flatten errors, group retry consumoor: log bad events in JSON form for invalid events and flatten errors consumoor: add rate-limited logging, unknown event warnings, and invalid event diagnostics fix(consumoor): align data transformations with Vector pipeline ...
… and columns feat(migrations): add new columns to beacon block tables for enhanced bid tracking feat(migrations): establish new libp2p gossipsub tables for improved data handling fix(migrations): drop obsolete columns associated with deprecated features fix(migrations): correct data schema for canonical beacon block withdrawals
…y and readability of withdrawal_type classification
refactor: widen block_access_index from UInt16 to UInt32 per EIP-7928
* master: chore(workflows): update Golang version to 1.26.2 and golangci-lint version to 2.12.2 for consistency and compatibility chore(Dockerfile, go.mod): upgrade base Go image to 1.26.2 and update Go module version to match chore(go.mod): update dependencies to ensure compatibility with Go 1.26.2 and fix potential vulnerabilities
…ields for consistency and maintainability chore(log_fields): create log_fields.go to centralize log field keys for v1 and v2 beacon event handlers style(test): replace hardcoded snapshot IDs and client names in tests with constants for better readability and maintainability
feat: wire ePBS gossipsub producers for the 4 Gloas topics
…internals Adds the producer + consumer side for `BEACON_SYNTHETIC_PAYLOAD_STATUS_RESOLVED` and `BEACON_SYNTHETIC_BUILDER_PENDING_PAYMENT_SETTLEMENT` — TYSM-instrumented events that observe ePBS internal state transitions on every beacon node (multi-witness, not gossip). Naming follows the LIBP2P_TRACE_SYNTHETIC_HEARTBEAT precedent — these are "synthesized" because they don't correspond to any beacon API SSE event; they're emitted from inside prysm via TYSM's hook system. Captured by every TYSM-patched beacon node, every slot / every epoch. What you get from these: - PayloadStatusResolved: empty-slot rate, envelope timing distributions, PTC participation rate, fork-choice latency, INVALID payload transitions. - BuilderPendingPaymentSettlement: builder payment success rate (SETTLED vs DROPPED), lost-revenue totals per builder, quorum-margin distributions, builder market concentration. Changes: - pkg/proto/eth/v1/beacon_synthetic.proto — new PayloadStatusResolved + BuilderPendingPaymentSettlement message types with PayloadStatus (PENDING/FULL/EMPTY/INVALID) and BuilderPendingPaymentOutcome (SETTLED/DROPPED) enums. - pkg/proto/xatu/event_ingester.proto — 2 new Event_Name enum values (102, 103), 2 new AdditionalData messages, 2 new oneof entries in DecoratedEvent (Additional* IDs 108/109, data IDs 223/224). - pkg/clmimicry/ — 2 TraceEvent payload structs, 2 New*Payload builders, 2 TraceEvent_* constants, dispatch map entries, case arms in handleHermesRPCEvent, 2 EventConfig toggles, 2 Group D categorizer registrations, 2 TraceEventTo* converters, 2 handle* methods + 2 deriveAdditionalDataFor* helpers in a new event_beacon_synthetic.go. - pkg/server/service/event-ingester/event/beacon/synthetic/ — 2 new server handler files in a new `synthetic` package (parallels `eth/v1/`); registered in the event router. - deploy/migrations/clickhouse/003_gloas_epbs_support.up.sql — 2 new tables appended (no new migration since 003 is greenfields, not yet deployed). Both use ReplicatedReplacingMergeTree with meta_client_name in the ORDER BY composite to preserve multi-witness records. - pkg/clickhouse/route/beacon/beacon_synthetic_*.go — 2 hand-written FlattenTo routes calling the existing extractBeaconSlotEpochPropagation helper. The matching .gen.go files come from `make clickhouse-routes`. - pkg/clickhouse/route/cmd/generate/manifest.go — 1-line addition registering the `beacon_synthetic_` prefix in the codegen prefix-route table so the generator emits the new files into pkg/clickhouse/route/beacon/. - Updated existing .gen.go files from re-running `make clickhouse-routes` (rowgen tool produces a tighter output than the last commit on a few fields — the regen is consistent with the schema, no functional change). Tests: - pkg/clmimicry/event_categorizer_test.go — counts updated 33→35 events, GroupD ≥10→≥12 (the 2 new synthetic events are Group D — no Topic, no MsgID, same shape as engine API events / custody probes). - pkg/clickhouse/route/beacon/beacon_synthetic_*_test.go — placeholder snapshot tests following the existing `TODO(epbs)` pattern; they currently fail with the standard `"nil … payload: invalid event"` message awaiting real Gloas devnet captures. Verification: - buf lint / buf build → clean - go build ./... → clean - go test -race ./pkg/clmimicry/... → ok - make proto + make clickhouse-routes → clean output Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
feat: beacon_synthetic_* events
feat: PTC vote three-state breakdown + EIP-7843 slot_number coverage
feat: add OnPayloadAttestationProcessed observe hook
* master: chore: swap go-eth2-client to ethpandaops fork v0.1.2 (#828) feat(output): add s3blobstore sink for blob sidecar archival
Bumps: - go-ethereum replace pin: bal-devnet-3 (8f361e342cb9, 2026-03-23) → bal-devnet-6 HEAD (73944e329925, 2026-05-07). Matches the EL branch deployed on glamsterdam-devnet-3 per ethpandaops/glamsterdam-devnets ansible inventory. - ethpandaops/beacon: 0260912 → ddfa770 (carrying matching go-ethereum pin). - ethpandaops/ethcore: 4e1b5fb → f67d7a6 (carrying matching go-ethereum pin + new eth.NewPeer 5-arg call). bal-devnet-6 keeps AccountAccess.StorageChanges + EncodedStorage.ToHash required by pkg/proto/eth/v1/conversion.go and brings the latest BAL (EIP-7928) refinements plus Amsterdam slot-number support (EIP-7843).
Resolves enum + migration collision by shifting BAL+ePBS up by 1:
- BEACON_API_ETH_V1_EVENTS_FAST_CONFIRMATION keeps event enum 89, oneof
field 95, oneof data field 210, and migration 002.
- BEACON_API_ETH_V2_BEACON_BLOCK_ACCESS_LIST moves 89 → 90 (oneof 96/211,
migration 003).
- All 15 ePBS enums shift up by 1 (90 → 91 ... 104 → 105) with matching
oneof field renumbering.
Accepts master's consumoor-only local stack (deletes vector-http-kafka.yaml
and vector-kafka-clickhouse{,-libp2p}.yaml; adds xatu-consumoor service
config + xatu-server direct kafka output with topicPattern
"xatu-protobuf-${event-name}").
Bumps ethpandaops/beacon to release/gloas tip 81b9ef3 to pick up both ePBS
SSE hooks (release/gloas) and OnFastConfirmation (merged in from master).
# By Sam Calder-Mason # Via GitHub * master: feat(observability): full OTel traces with gRPC propagation (#835) # Conflicts: # pkg/proto/eth/v2/events_vtproto.pb.go # pkg/sentry/event/beacon/eth/v1/beacon_blob.go # pkg/sentry/event/beacon/eth/v1/beacon_committee.go # pkg/sentry/event/beacon/eth/v1/events_attestation.go # pkg/sentry/event/beacon/eth/v1/events_blob_sidecar.go # pkg/sentry/event/beacon/eth/v1/events_block.go # pkg/sentry/event/beacon/eth/v1/events_block_gossip.go # pkg/sentry/event/beacon/eth/v1/events_chain_reorg.go # pkg/sentry/event/beacon/eth/v1/events_contribution_and_proof.go # pkg/sentry/event/beacon/eth/v1/events_data_column_sidecar.go # pkg/sentry/event/beacon/eth/v1/events_finalized_checkpoint.go # pkg/sentry/event/beacon/eth/v1/events_head.go # pkg/sentry/event/beacon/eth/v1/events_single_attestation.go # pkg/sentry/event/beacon/eth/v1/events_voluntary_exit.go # pkg/sentry/event/beacon/eth/v1/proposer_duty.go
… consistency and to avoid unnecessary conversion
feat: port otel work to glammy work
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds Glamsterdam fork support to Xatu.
What this covers
EIP-7928 — Block-Level Access Lists (BALs)
Spec-compliant against
bal-devnet-3.BlockAccessList,BlockAccessListEntry,BlockAccessListChange,BlockAccessListStorageRead(incl."touched"change type for accounts with no state changes)BeaconBlockGloas/BeaconBlockBodyGloasproto messagesExecutionPayloadGloaswithblock_access_list+slot_numberBEACON_API_ETH_V2_BEACON_BLOCK_ACCESS_LIST = 89NewBlockAccessListFromGloas()(with test coverage)NewBlockAccessListDeriver(CannonType 16) — wired002_gloas_bals_support(tablecanonical_beacon_block_access_list+execution_payload_block_access_list_rootcolumn oncanonical_beacon_block)NewEventBlockFromGloas()EIP-7732 — Enshrined Proposer-Builder Separation (ePBS)
Internal plumbing wired end-to-end across server, sentry, cannon, fork-choice. Awaiting Gloas devnet for integration testing.
eth/v1files (incl.execution_payload_available.proto), 4 libp2p gossipsub summary types, 12 new event IDs (90–101), 12Additional*Datamessages, 3 newCannonTypeentries (16/17/18)003_gloas_epbs_support— 12 new tables + column additions (builder_index,bid_value,execution_payment,payload_present)event.go(incl. new GOSSIP / AVAILABLE handlers)OnExecutionPayload,OnExecutionPayloadGossip,OnExecutionPayloadAvailable,OnExecutionPayloadBid,OnPayloadAttestationMessage,OnProposerPreferencesNewPayloadAttestationDeriver,NewExecutionPayloadBidDeriver; backfilling checkpoint iterator handles CannonType 16/17/18 in bothGetMarkerandcreateLocationFromEpochNumberForkChoiceNodeV2.payload_statusround-trips viaextra_data["payload_status"]pkg/clickhouse/route/{beacon,canonical,libp2p}/What this does NOT yet cover
Devnet-blocked or out-of-scope:
pkg/clmimicry/) — server handlers and routes already exist; producer side is the gapExecutionTransaction/Withdrawal/BlockAccessList— depends onBeaconBlockBodyGloasno longer carryingExecutionPayload; wired onceenvelope events arrive on devnet
nil … payload: invalid eventawaiting Gloas devnet dataOnBlock/OnHeadGloas paths) — wire once devnet stabiliseseth_getBlockAccessListJSON-RPC, Engine API V5/V6 tracking (newPayloadV5,getPayloadV6,forkchoiceUpdatedV4)Intentionally not included (don't reintroduce without spec confirmation)
payload_voteprojection on attestation tables —committee_indexalready stores rawdata.index; consumers can derive 0/1 for Gloas+ rowssidecar_slot/sidecar_beacon_block_rootprojections on DataColumnSidecar tables — preemptive; revisit when real Gloas SSE payloads arriveMigrations
002_gloas_bals_support003_gloas_epbs_support