Bench cuda apc sweep 2026 06#3765
Draft
georgwiese wants to merge 54 commits into
Draft
Conversation
`PgoAdapter` gains `generate_apcs` (build + rank); the trim becomes a free `select_apcs(ranked, n, skip)`. `filter_blocks_and_create_apcs_with_pgo` stays as a default-impl one-shot for SP1's call site (auto-fills `apc_candidates = autoprecompiles + skip` for instruction/none; Cell overrides to skip the auto-fill). CLI gains a `generate-apcs` subcommand that caches under `generate/<h>`. `--pgo` and `--max-columns` move to `GenerateApcsArgs` (they shape the ranking, not the trim); `--apc-candidates <N>` caps the build size. `select-apcs` chains through `generate-apcs` and auto-fills `--apc-candidates = --autoprecompiles + --skip` for instruction/none so the common case stays ergonomic. Headline win: sweeping `--autoprecompiles` under `--pgo cell` against `--artifacts-dir` reuses a single generate-cache entry across iterations. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cell PGO now always builds every eligible candidate. `--apc-candidates` is ignored with a warning (in both standalone and fused contexts) — the dynamic density ranking needs the full post-opt cost of every candidate. The pre-build heuristic that previously kicked in when the user set `--apc-candidates` is gone. For instruction/none in a fused pipeline (`select-apcs` and beyond), unset `--apc-candidates` now defaults to `--autoprecompiles` (was `--autoprecompiles + --skip`). Validation: error if `--autoprecompiles + --skip > --apc-candidates`. Standalone `generate-apcs <guest>` keeps "unset = build all" for instruction/none — there's no fused selection to derive a default from. The CLI auto-fill is applied before computing each stage's cache hash, so the resolved value lives in the cache key. Sweeping `--autoprecompiles` under `--pgo cell` still shares a single `generate/<h>` entry regardless of whether the user also passes `--apc-candidates`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Too noisy for the main README — the basic examples cover the common cases. Sweep workflows can be a doc/follow-up if needed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Its only caller was openvm-riscv::compile_exe; inline generate_apcs + select_apcs there directly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When `--autoprecompiles 0` is selected downstream, the CLI sets `apc_candidates = Some(0)` for Cell so the library short-circuits without building anything. Positive caps are still ignored with a warning (Cell's density ranking needs every candidate). Restores the early-exit that the previous fused `create_apcs_with_pgo` had for `autoprecompiles == 0`, expressed through `apc_candidates` since the build stage no longer sees the selection count directly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The trim to (skip, autoprecompiles) now lives in `select_apcs`, so the greedy loop always runs to the column budget (or until candidates are exhausted). `max_selected` was always `usize::MAX` from the only caller. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The trait one-shot is no longer used inside powdr — openvm composes detect_blocks + generate_apcs + select_apcs directly. The internal helper run_filter_generate_select goes away with it. The SP1 fork at `powdr-labs/v6.0.2-apc-base` still calls this method. That fork needs an equivalent update (compose the three steps directly, or wrap them in a fork-local helper) before `cargo clippy --workspace` can pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Each guest section runs `run_bench` with multiple `apcs` values (apc000, apc003, apc010, apc030, …) against the same guest source and profile input. With cell PGO (the default), the generate stage doesn't depend on `--autoprecompiles`, so adding a shared `--artifacts-dir` per (guest, profile-input) lets the second-and-later sweeps cache-hit on the expensive build step. Only the cheap select+setup re-run each time. `--apc-candidates-dir` has to be shared too because it's part of the generate-stage cache key (it's a per-stage arg, not a global). Set it per (guest, profile-input) under the same cache root. `apc_candidates.json` is now in the shared candidates dir; per-run effectiveness plots read from there. Per-block `apc_candidate_*` files are still cleaned up after each run, but they only get written on the first cache-miss anyway — subsequent cache-hits don't re-create them. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
succinctlabs/sp1#2805 follows #3752's PgoAdapter split. Tracking that branch unblocks `cargo clippy --workspace`. Once SP1 merges #2805 into `powdr-labs/v6.0.2-apc-base`, this pin can revert. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Document the rule in code, restore the `--autoprecompiles + --skip` default in the help text and README (was stale after the function simplification), and drop the validation-error mention since the new function silently truncates instead. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Move the generic on-disk cache helpers into `powdr-autoprecompiles::staged_cache` and the per-stage runner into `powdr-openvm-riscv::pipeline` as `StagedPipeline`. cli-openvm-riscv's `Pipeline` collapses to a thin clap→library shim. `StagedPipeline` accepts caller-provided cache keys (anything `Hash`) and runs each stage through a `cached` helper, so downstream callers (openvm-eth, sp1) can adopt the same artifacts-dir caching without depending on the CLI.
PR #3763 changes the powdr autoprecompile API. The pre-existing pins target sp1 and openvm-eth commits that use the old API, so CI fails even though my refactor itself is correct. - sp1-benchmarks/Cargo.toml: bump sp1 dep branch from powdr-labs/generate-apcs-api → powdr-labs/split-powdr-config (succinctlabs/sp1#2830 — adopts the new powdr API). - .github/actions/patch-openvm-eth: bump ref from 05ed9cd3 (main) → e716b51a (powdr-labs/openvm-eth#8 tip, adopts the new powdr API). The "Verify openvm-eth ref is on main" check will (intentionally) fail until openvm-eth#8 merges — the comment in pr-tests-with-secrets.yml already accounts for this case, and the actual reth test runs against the compatible code.
The current `PgoConfig` enum bundles the ranking strategy (`PgoType`) with a heavy `ExecutionProfile` payload. The payload is *output* of a guest execution, not configuration — keeping it under the `*Config` name is confusing and forces the type to be unhashable. Pure rename in this commit. The next commit reclaims the `PgoConfig` name for a new type that's actually a config (Hash-able, holds only inputs), and threads it through `StagedPipeline`'s cache keys.
Reclaim the `PgoConfig` name for what's actually configuration:
```rust
pub struct PgoConfig {
pub pgo_type: PgoType,
pub max_columns: Option<usize>,
pub inputs: Vec<u8>, // opaque, hashed verbatim
}
```
`inputs` is the serialized form of whatever the `make_*` closures need
(typically the guest stdin). The closures take `(&Guest, &[u8])` and are
expected to round-trip the bytes back to their typed form — they're
pure functions of their arguments instead of relying on captured state.
`StagedPipeline` now hashes `(guest_hash, generate, pgo_config)` for the
generate stage and `(guest_hash, generate, pgo_config, select)` for
select/setup. Both stages include `pgo_config` — fixes a real bug where
switching `--pgo cell` ↔ `--pgo instruction` with the same select args
returned a stale select-stage blob from the cache (the upstream ranking
would have changed but the select hash didn't see it).
The `input_fp: &impl Hash` parameter is dropped (everything it carried
is now in `pgo_config.inputs`). CLI shim builds `PgoConfig` from clap
args + `serde_cbor::to_vec(&profile_input)`.
`compile_exe` is unaffected — it takes `PgoData` (the renamed enum
carrying the computed `ExecutionProfile`) directly and doesn't go
through `StagedPipeline`.
`Pipeline` was a thin shim around `StagedPipeline` whose only real job was running `compile_openvm(...)` once at construction and stashing `profile_args`. Its three `run_*` methods were pure plumbing from clap args to `StagedPipeline` calls — no policy, no ordering, no error handling. The `profile_args.profile_input` it stored was also reachable on every clap-arg struct the methods already took, so it was redundant. Replace with three free functions (`build_pipeline`, `run_generate_stage`, `run_select_stage`, `run_setup_stage`) and call them directly from `run_command`. Net −22 lines; the call graph is now flat (`run_command → run_X_stage → StagedPipeline::X`) with no intermediate struct.
Three threads, all on top of the upstream API change that made
`StagedPipeline::{select_apcs, setup}` take the closures directly
(instead of recursive `compute_*` callbacks):
1. Trait aliases for the closures (`pipeline.rs`).
The method signatures were dominated by their closure types — five
lines per parameter. Add two trait aliases (`MakePgoProfile`,
`MakeEmpiricalConstraints`) so each closure parameter becomes one
line. Each trait has a blanket impl over the matching `FnOnce`
so call sites are unchanged.
2. Restore cache-hierarchy laziness in `select_apcs` / `setup`.
The flat API was inadvertently eager: `let ranked = self.generate_apcs(...)`
ran *before* `cached(...)` checked the select-stage hit, so a
select-cache hit still paid for a generate-stage blob load + cbor
deserialize (and setup hits paid for both). Move the recursive
`self.generate_apcs(...)` / `self.select_apcs(...)` call *inside*
the `cached(...)` closure with `move`; the closures it captures
are forwarded down only on a miss.
3. CLI `SelectArgs::pipeline_inputs()` helper (`main.rs`).
The four chained call sites (SelectApcs / Setup / Execute / Prove)
all derived `(GenerateConfig, SelectConfig, PgoConfig)` the same
way — three lines of `with_select_defaults` / `pgo_config_from_args`.
Move into a method on `SelectArgs` so each call site is one line.
Without this, external callers of compile_exe (openvm-eth, sp1, ...) get config.apc_candidates = None and the Instruction/None PGO build loop fans out to every eligible block, then select_apcs throws all but `autoprecompiles + skip` away. Factor the existing CLI policy into a shared default_apc_candidates() helper and apply it inside compile_exe so the convenience wrapper preserves pre-split semantics for all callers.
Three call sites of `default_apc_candidates` collapse to one: - `default_apc_candidates`: short-circuit on `autoprecompiles == 0` so it is safe to apply unconditionally inside the library (standalone `generate-apcs` previously triggered `Some(0)` for Instruction/None). - `compile_exe`: drop the in-line default and route through `StagedPipeline` (no caching) — the convenience helper now shares the exact composition CLI / openvm-eth use. - CLI: drop `SelectArgs::set_apc_candidates_default` and pass the real `autoprecompiles` / `skip` to the generate-stage config (the cache key intentionally omits them so Cell-PGO sweeps still reuse the blob). - `InstructionPgo`: drop the stale `autoprecompiles == 0` short-circuit — `apc_candidates` is the load-bearing cap now, not the selection count.
The kitchen-sink `PowdrConfig` bundled fields from every pipeline stage into one struct. Generate-stage code therefore couldn't safely hash the config as a cache key (select-stage flags would invalidate the hash and break `--apc N` sweep reuse), forcing `StagedPipeline` to expose a `cache_key: &impl Hash` argument and pushing fingerprinting policy onto each caller — three callers, three conventions, easy to miss a field. This commit splits `PowdrConfig` along the natural pipeline boundary: * `GenerateConfig` (`autoprecompiles/src/lib.rs`) — fields the generate stage reads. Derives `Hash`; the library hashes it directly. * `SelectConfig` — `autoprecompiles` + `skip`. Derives `Hash`. * `degree_bound` keeps being passed plain into `setup`. * `GenerateConfig::with_select_defaults(pgo, select)` is the single policy definition for the `apc_candidates` cap default; callers chain it in one line when they know both halves. * `InstructionPgo` / `NonePgo` short-circuit on `apc_candidates == Some(0)` (mirroring Cell), set by `with_select_defaults` when `select.autoprecompiles == 0`. The stale `autoprecompiles == 0` field check in InstructionPgo is gone. `StagedPipeline`'s API drops `cache_key`; callers supply only an `input_fp: &impl Hash` covering anything hidden behind the closures (PGO stdin contents, RPC chain id, ...). The library hashes `(guest_hash, gen, pgo, max_columns, input_fp)` for generate and `(guest_hash, gen, select, input_fp)` for select/setup. Generate's hash omits `select` by construction, so a Cell-PGO `--apc N` sweep reuses the candidate ranking automatically (Cell's `with_select_defaults` returns `apc_candidates = None` regardless of selection size). `compile_exe` becomes a thin wrapper over `StagedPipeline` (no caching). `PowdrConfig` and `default_powdr_openvm_config` are deleted; a new `default_powdr_openvm_configs` returns the split pair. Downstream consumers (CLI shim, openvm-eth, sp1) migrate in follow-up commits.
The hand-inlined load/save/move dance was just `cached(...)` written out by hand because `setup` consumes `self` (it has to move `self.guest` into `customize_exe::setup`). That's fine inside a `move` closure as long as we pull `artifacts_dir` and `degree_bound` out first. Behavior is unchanged; the body is now uniform with `generate_apcs` / `select_apcs`.
`compile_exe` is now only referenced by tests in `openvm-riscv/src/lib.rs` itself (production callers — CLI, openvm-eth, sp1 — all use `StagedPipeline` directly or, in sp1's case, the lower-level `powdr_autoprecompiles` API). The function no longer needs to be `pub`, so move it into the `mod tests` block. Two simplifications fall out: - The `empirical_constraints: EmpiricalConstraints` parameter was always passed as `EmpiricalConstraints::default()` by every call site (the only non-default usage lives in the CLI's `maybe_compute_empirical_constraints`, which goes through `StagedPipeline`, not `compile_exe`). Drop the parameter; hardcode the default inside. - Route through `StagedPipeline::new(.., None)` so the test path exercises the same code as production. Saves having two near-copies of the generate → select → setup chain. Also: skip the `make_pgo_profile` closure call for `PgoType::None` inside `StagedPipeline::generate_apcs`. The profile is discarded for None anyway, and skipping the closure means `compile_exe` can pass an "unreachable for None" closure without needing `ExecutionProfile: Default`.
- run_guest_benches.sh: sweep every guest over {0,3,10,30,100,300} APCs,
BENCH_FEATURES override for cuda builds, BENCH_KEEP_GOING failure
tolerance (psrecord does not propagate exit codes; judge by metrics
file), ARG_MAX-safe candidate snapshot cleanup.
- run_reth_gpu_bench.sh: openvm-eth (pinned ref) reth sweep on GPU over
{0,3,10,30,100,300,500,1000}, mirroring the nightly test_apc_gpu job
with prefetch + compile-cache phases and per-count failure tolerance.
- collect_bench_results.sh: turn results/ into a committable tree
(drop .bench-cache, surface the guests' apc_candidates.json).
- generate_bench_results_readme.py: overridable blob/tree URL bases and
path prefix so readmes can target results committed outside the
bench-results repo (default output unchanged).
- openvm-eth-bench.patch: optional openvm-eth patch — RPC fetch
robustness (retry-all transport policy, RPC_PROOF_CHUNK_SIZE) and
wiring POWDR_APC_CANDIDATES_DIR into GenerateConfig for the
apc_candidates.json dump.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
GPU rerun of the autoprecompiles blog benchmarks on this branch:
all guests swept over {0,3,10,30,100,300} APCs, reth (openvm-eth
44564802, block 24171377) over {0,3,10,30,100,300,500,750,1000}.
Headlines (details + reproduction in bench-results/readme.md):
- reth proves up to 300 APCs with default aggregation params, 500 with
raised leaf/internal stacked-height caps (peak 20.2 GiB GPU memory);
750/1000 need a 2^23-row stacked leaf trace whose proving OOMs the
24 GiB card, although the app proof completes even at 1000 APCs.
- On this GPU the reth wall-clock does not improve with APCs (33.7 s at
0 -> 42.1 s at 300) despite fewer segments/cells, unlike the CPU
nightly (1383 s -> 1230 s at 100): per-AIR overheads outweigh cell
savings at 4090 throughput.
- Guest sweeps reproduce the blog's qualitative findings on pure
proving time; totals are dominated by interpreted preflight on GPU.
ecc projective/affine-hint fail at 300 APCs in GPU leaf aggregation
(LayoutHeightExceeded, recursion ceiling; CLI has no leaf-cap knob).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…esults Size the tunable guest workloads to ~50 segments in the 0-APC version, matching the autoprecompiles blog post: keccak 10000 -> 25000 hashes, sha256 30000 -> 80000 hashes, ecc 50 -> 100 scalar mults, ecrecover 20 -> 125 recoveries (pairing/u256/matmul run fixed workloads baked into the guest and are unchanged, as are the reth results). Measured software segment counts: keccak 48, sha256 44, ecc-projective 52, ecrecover 45. At this scale APCs cut totals ~2x (e.g. keccak 42.8 s -> 21.8 s) and pure proving time 2-7x (sha256 20.4 s -> 2.9 s, beating the manual precompile). New at the larger input: ecrecover now also hits the LayoutHeightExceeded recursion ceiling at 300 APCs (it passed at input=20), alongside both ecc variants. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
stark-backend branch autoopt-2026-04-12-clean (multi-stream Round 0 / GKR input eval, batched MLE-round + stacking-scatter kernels, batched stacked-reduction sync, pre-allocated GPU buffers). Already based on the pinned v2.0.0-beta.2-powdr, so no rebase — only its "Remove ColumnsAir" commit is reverted on top so the pinned openvm fork compiles (pushed as stark-backend branch autoopt-beta2-compat, 4f3755ce). Wired in via [patch] overlays kept as scripts/*-autoopt.patch; the branch itself still builds the baseline prover. Headline: the autoopt prover removes the per-AIR overhead that made reth proving *slower* with more APCs on GPU. Proving time 0->500 APCs is now flat (21.4 s -> 22.0 s, +3%) vs baseline +36% (20.7 s -> 28.1 s): -14% at 300 APCs, -22% at 500. At <=100 APCs it is ~3-4% slower (stream/batching overhead with few powdr AIRs). Guests improve 4-23% on proving time at their best counts, growing with APC count (pairing@300 -37%). APC-count limits and failures are unchanged (<=300 default caps, 500 with raised caps, 750/1000 leaf-agg OOM). Also: run_reth_gpu_bench.sh now only writes the powdr [patch] into openvm-eth/.cargo/config.toml when missing, so user-added patch sections (like the autoopt overlay) survive reruns. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Grouped bar charts (baseline vs autoopt, total_proof_time_excluding_trace_ms),
x-axis = experiment, two bars each: one panel per shared APC count
{0,3,10,30,100}, plus a dedicated reth sweep panel covering 300/500 where
autoopt's win is largest. Generator committed as
openvm-riscv/scripts/plot_autoopt_comparison.py.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Regroup the comparison charts by experiment instead of by APC count: one chart per metrics-viewer link, x-axis = number of autoprecompiles, two bars per count (baseline vs autoopt) — same format as the reth chart. ecc keeps both variants (projective, affine-hint) as two subplots under its single metrics link. Replaces the prior by-APC-count multi-panel. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make explicit that 300 APCs was swept for every guest but crashed for
ecc (both variants) and ecrecover in GPU leaf aggregation
(LayoutHeightExceeded { 22 > 21 }) — the same recursion ceiling as
reth-at-500, which the powdr CLI can't raise because it doesn't expose
the leaf/internal stacked-height knobs openvm-eth's run.sh has. Limit is
in the prover's recursion layout, not the APCs, identical across provers.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…026-06-06) CPU counterpart of bench-results/ (RTX 4090): same branch (powdr main after OpenVM 2.0 + #3763), same block (24171377) and guest inputs, on the blog's Intel Xeon Gold 5412U / 251 GiB hardware. reth swept {0,3,10,30,100,300} plus high-count probes {500,1000,2000}; guests swept {0,3,10,30,100,300} + manual baselines. Key CPU findings (see bench-results-cpu/readme.md): - reth: APCs reduce proving time, min at ~300 APCs (1969->1575 s, -20%); the CPU reaches 2000 APCs (raised aggregation caps) where the 24 GiB GPU OOM'd at 1000. Past ~300 per-AIR overhead makes it net-negative by 2000. - guests: large APC speedups (keccak 6.8x, sha256 6.5x); on excl-trace proving APCs beat the manual precompiles for sha256/u256. - the wide k256 guests (ecc affine-hint, ecrecover) OOM at low APC counts on this shared box; APCs shrink the footprint enough to prove from 10/30 APCs. New helper scripts: run_reth_cpu_bench.sh, run_reth_cpu_highcount.sh. run_guest_benches.sh: default JEMALLOC_SYS_WITH_MALLOC_CONF (prevents OOM on the big software baselines) + opt-in BENCH_SKIP_EXISTING resume guard. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Re-run reth on the blog post's exact block (133 txns, 9.6M gas) for direct
comparability, alongside the existing nightly-block (24171377) reth. Full
{0,3,10,30,100,300,500,1000,2000} sweep, no failures, in reth_blogblock/.
The APC speedup reproduces the blog: 1.36x at 100 APCs (1076->792 s) vs the
blog's 1.37x (341->248 s) on the same block. Absolute times are ~3x higher and
segmentation ~4x finer under OpenVM 2.0 (vs the blog's pre-2.0 prover). The
blog's 100-APC recursion ceiling is gone: the full prove-stark (with recursion)
completes to 2000 APCs here (the blog could only reach 500-2000 on the app
proof alone, without recursion).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.