ingest: size each element once on the extract path, and return element views (+ stress-density benchmark)

[tamirms]

Four self-contained follow-ups to #5966, all inside ingest/. Three are allocation/size-pass removals on the extract path plus the benchmark that makes them measurable; the fourth is a deliberate, narrow breaking change to the events product types.

The motivation is stellar-rpc's hot ingest, which runs ExtractLedgerTxParts + the events product on every closed ledger under a hard real-time budget. Its stress fixture is ~6,000 transactions per ledger, where the per-element size passes — not the parsing — are essentially the whole cost.

The commits

1. presize ExtractLedgerTxParts output from the TxProcessing count

The walk's element count is knowable before the walk starts: the array's length prefix is an O(1) read. The parts slice was growing through append reallocations anyway. Now it's presized.

On a malformed prefix the count stays 0 and the walk itself reports the error, exactly as before — the presize is not a new validation point.

2. size each event exactly once when collecting raws

Two places were sizing the same bytes twice.

collectRaws consumed the generated Iter(), which yields untrimmed suffix views and sizes each element to advance — then MustRaw() sized that same element again to trim it. Since element views are plain byte views, the trimmed element is its raw bytes, so collecting by trimming and stepping past the trimmed extent gets both from one pass.

The V4 per-operation spine (opEventRaws) had the same shape one level up: the ops Iter() sized every operation's whole interior to advance, then MustEvents() re-sized its Changes to locate the events. Since events is the last OperationMetaV2 field, the events offset plus the events extent is exactly what positions the next operation, so each operation's interior is sized once.

Commit 4 supersedes the collectRaws half of this: once the products return element views, the generated MustAll() already has the single-pass property and the hand-rolled walk is deleted. The opEventRaws half survives — see below.

3. stress-density benchmark — 6,000 txs, ~10 events each

The existing pubnet fixture (249 txs) is representative but sparse — too sparse for the per-element size passes to show up. This adds a crafted ledger at stress density, with V3-soroban / V4-soroban / V4-classic metas cycling so every extractor arm runs, LCM V2 (the modern TransactionResultMetaV1 element), plus per-op ledger-entry changes and per-tx diagnostics for a realistic spine. It measures parts, the parts+events composition, and all three products.

4. return element views, not [][]byte, from the events products — breaking

TxEvents and LedgerTransactionView returned raw byte slices. That meant a consumer wanting to read a field re-wrapped every element back into a view (stellar-rpc's stage filter did literally xdr.TransactionEventView(raw)), while the producer could not use the generated MustAll() at all — it returns []E, and Go cannot retype the outer slice to [][]byte even though E is ~[]byte.

Element views ARE their wire bytes, so []byte(elem) and View(bytes) are both free; only the outer slice ever allocates. Typing the fields as element views costs a bytes-wanting consumer nothing, saves a field-reading consumer the re-wrap, and lets every collection site be MustAll().

TxEvents.TransactionEvents            [][]byte   -> []xdr.TransactionEventView
TxEvents.OperationEvents              [][][]byte -> [][]xdr.ContractEventView
LedgerTransactionView.DiagnosticEvents           -> []xdr.DiagnosticEventView
LedgerTransactionView.TransactionEvents          -> []xdr.TransactionEventView
LedgerTransactionView.ContractEvents             -> [][]xdr.ContractEventView

collectRaws/collectRawsExtent and their two constraint interfaces are deleted — net −26 lines. Callers wanting bytes write []byte(elem); callers wanting fields drop their wrapping step.

opEventRaws still steps the operations array by hand, and that is the one place it's warranted: Iter()/All() there would size each operation's whole interior to advance, then MustEvents() would re-walk Ext+Changes and collection would re-walk Events — about double. Its inner events array uses MustAll() like everything else, recovering the array's wire extent by summing the trimmed element lengths MustAll already produced.

Measurements

BenchmarkLedgerTxPartsStressDensity, this branch vs main. Both arms run the identical benchmark code — the base arm is main with commit 3's test file cherry-picked onto it, so the comparison isn't confounded by a differently-shaped benchmark on each side. -count=10, c6id.8xlarge (Xeon 8375C).

	main	this branch	delta
parts	3.319 ms	2.981 ms	−10.2%
parts_events	12.459 ms	7.923 ms	−36.4%
all_products	14.141 ms	9.621 ms	−32.0%
parts B/op	3456.1 Ki	704.2 Ki	−79.6%
parts_events B/op	7.914 Mi	2.602 Mi	−67.1%
parts allocs/op	26	9	−65.4%
parts_events allocs/op	36,030	16,010	−55.6%

All p=0.000 at n=10, run-to-run variance ±0–1%.

Within the branch, commit 4 contributes −3.39% on parts_events over commit 2's form (−1.48% from the type change, a further −1.94% from dropping the generic helper for the generated method); allocations are unchanged by it at 16,010/op. Its value is the API and the deleted code, not the allocation count.

Downstream check: pinned into stellar-rpc's hot ingest at 6,000 tx/ledger, this branch holds per-ledger ingest_total at p50 29.7 / p99 37.6 ms against a 100 ms budget.

Why these four and not more

This branch is deliberately the low-risk subset. A larger redesign of the views walk was prototyped and measured separately and is not proposed here: it required either a generated API surface that grows quadratically in type count, or runtime ordering knowledge from every caller, and it carried a hand-mirrored locate layer duplicating generated code. These four need none of that — they're local to ingest/, they don't touch the generated views, and each is independently revertible.

🤖 Generated with Claude Code