Skip to content

feat: target the prod auth gateway from the capacity benchmark - #327

Open
tvinagre wants to merge 16 commits into
mainfrom
tl/benchmark-gateway-auth
Open

feat: target the prod auth gateway from the capacity benchmark#327
tvinagre wants to merge 16 commits into
mainfrom
tl/benchmark-gateway-auth

Conversation

@tvinagre

@tvinagre tvinagre commented Jul 18, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Adds the client and benchmark support needed to run the capacity/load
benchmark against the prod auth gateway (https://fynd-api.propellerheads.xyz/v1/<chain>).

Client (fynd-client)

  • FyndClientBuilder::with_api_key sends the key verbatim as the Authorization header (no Bearer prefix), matching the gateway's auth.lua, which compares the raw header value against its key store.
  • FyndClientBuilder::with_quote_path / with_health_path override the request path appended to the base URL (defaults /v1/quote, /v1/health). A chain-scoped base such as https://host/v1/base with /quote joins to /v1/base/quote, which the gateway rewrites to upstream /v1/quote.
  • HTTP 429 maps to a new non-retryable FyndError::RateLimited { retry_after }. Previously a 429 body failed to parse as a Fynd ErrorResponse and surfaced as a retryable Http error, so the client retried a rate-limited request three times, amplifying load and distorting latency stats.

Benchmark (fynd-benchmark)

  • The capacity and load subcommands accept FYND_API_KEY, QUOTE_PATH, and HEALTH_PATH (also --api-key, --quote-path, --health-path) and pass them to the client. Gateway 429s count toward the report's http_error_rate rather than being retried.

Tests

  • Client: auth header present with the raw value when a key is set and absent otherwise; overridden quote/health paths join correctly; a 429 returns RateLimited and is not retried (expect(1) against a 3-attempt retry policy); RateLimited is not retryable.
  • Benchmark: capacity parses the three gateway flags and defaults /v1/quote and /v1/health.

Note: overlaps PR #325

This branch is based on the release branch (tl/release-0.90.0-rc) rather than main, because the capacity/generate-requests subcommands it wires up are not yet on main. Its client-side auth (FyndClientBuilder::with_api_key + with_quote_path/with_health_path overrides, and the non-retryable FyndError::RateLimited 429 variant) overlaps with PR #325, which adds hosted-gateway auth (with_api_key/with_chain) to the same client.rs. Both PRs touch client.rs and will conflict on merge. Once #325 lands with the raw-Authorization header fix (the deployed gateway matches the raw token; a Bearer prefix returns 401), this client change should be reconciled with — or superseded by — #325's implementation. The benchmark wiring and manifests here run against the built rc image and are independent of which client API wins.

🤖 Generated with Claude Code

tvinagre and others added 16 commits July 13, 2026 16:21
@tvinagre @claude
feat: record per-pool queue wait and depth metrics at task pickup
dd76d9f 
Queue wait growing while solve time stays flat is the leading indicator of
worker saturation. Records worker_pool_queue_wait_seconds (histogram) and
worker_pool_queue_depth (gauge), both labeled by pool, at the point each
worker picks a task off the queue.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@tvinagre @claude
chore: drop stale worker_id TODO
fcf862a 
The TODO to make worker_id a string with the pool name is obsoleted by
the pool_name field added directly below it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@tvinagre @claude
feat: add ladder spec and SLO evaluation for capacity benchmark
3fd96f5 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@tvinagre @claude
feat: add --encoding toggle to benchmark request path
2bfa237 
Threads an encoding: bool through SwapRequest::to_quote_params_with_encoding,
ParallelizationMode::run, and its three strategy functions so load tests can
attach EncodingOptions (0.5% slippage, TransferFrom) and exercise the full
quote+encode server path. Exposed as --encoding on the load subcommand;
scale keeps the flag off.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@tvinagre @claude
feat: add capacity subcommand measuring max RPS within latency SLO
068c845 
Wires the ladder+SLO evaluation types from capacity_report.rs into a new
`capacity` CLI subcommand: measures an unloaded sequential baseline, steps
an RPS ladder with rate-based traffic until a step breaches the SLO, and
prints the resulting report as JSON after a marker line so in-cluster Jobs
can pull it from pod logs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@tvinagre @claude
feat: ship fynd-benchmark binary in the Docker image
f676ca3 
Task 6's in-cluster capacity Job runs fynd-benchmark from the
published image, so both build stages now compile the crate and the
runtime stage copies the binary to /usr/local/bin. The second build
stage copies tools/benchmark/src/ instead of stubbing it, since
pairs.json and trades_sample.json are embedded via include_str! and
must be present when the real binary is compiled.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@tvinagre @claude
fix: print capacity summary before the JSON report marker
023c625 
The human-readable "Capacity: N rps" trailer was printed after the
marker + JSON block, so the documented log-retrieval capture
(sed marker-to-EOF, drop the marker line) appended prose after the
closing brace and produced invalid JSON. Print the summary first so
marker-to-EOF minus the marker line is exactly the JSON.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@tvinagre @claude
fix: deflake capacity ladder test against fast wiremock
8391d09 
The orchestration test used SloPolicy::default(), whose 1.2x p95
multiplier sits inside OS scheduling jitter with a ~1ms mock baseline.
Worse, round-trips are measured in whole milliseconds, so an undelayed
local mock truncates the baseline p95 to 0ms — making the SLO limit
(baseline * multiplier) zero and failing every step regardless of the
multiplier. Give the mock a 20ms response delay so the baseline is
nonzero, and use a 100x multiplier since SLO pass/fail math is already
unit-tested in capacity_report; this test covers orchestration only.

Verified 10 consecutive passing runs locally.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@tvinagre @claude
fix: floor SLO baseline and pin break-on-first-failure test
2e40a10 
capacity_report.rs: floor the baseline p95 at 1ms before scaling by the
SLO multiplier. Sub-millisecond baselines against fast mocks/solvers
truncate p95 to 0ms, making the SLO limit 0 and failing every step
regardless of actual latency.

capacity_report.rs: replace the manual hex-encoding loop in sha256_hex
with alloy::hex::encode, matching the pattern already used elsewhere in
the tools workspace.

capacity.rs: add ladder_stops_at_first_failing_step, pinning that the
ladder orchestration breaks on the first failing step instead of
continuing.

README.md: note that request intervals are whole milliseconds, so
offered rates quantize above ~30 rps, and recommend reading a step's
achieved_rps over its target_rps.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@tvinagre @claude
fix: touch benchmark main.rs in Docker dep-freshness layer
be8cf04 
The second RUN block's touch list covered every real-source main.rs
except tools/benchmark/src/main.rs. On a fresh clone, cargo's mtime
freshness check could still treat the dep-layer's stub fynd-benchmark
binary as up to date, shipping a no-op binary in the image.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@tvinagre @claude
ci: make pre-releases safe to cut from any branch
9953d9b 
A GitHub pre-release currently behaves like a full release: it moves the
floating latest image tag, auto-promotes the tag to the staging
environment, and attempts crates.io/npm publishes. Guard all three on
github.event.release.prerelease so a pre-release builds and pushes only
its own version-tagged Docker image, deployed deliberately.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@tvinagre @claude
fix: skip same-token request templates at load
9ba4ff2 
The 10k aggregator trade dataset (and embedded sample, as a defensive
measure) can contain templates where token_in == token_out. The server
rejects these with 400 BadRequest, which the capacity subcommand counts
as an HTTP error, tripping the 0.1% SLO error-rate threshold on every
ladder step regardless of latency.

Filter same-token templates (case-insensitive) in load_request_templates
and load_embedded_trades, logging the skipped count via tracing::warn!
so a filtered dataset is never silently smaller than it looks. Error out
if filtering leaves zero templates. load_all_templates_from_file already
delegates to load_request_templates; load_all_embedded_templates gets
the same filtering and logging.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@tvinagre @claude
feat: expose error-rate and unsolved-rate SLO thresholds as flags
348bfb3 
Staging measurement showed a heterogeneous 10k-trade dataset's unsolved
rate fluctuates ±2pp between step samples (binomial noise), so the
hardcoded 0.001 excess-unsolved tolerance fails healthy capacity steps.
Add --max-http-error-rate and --max-excess-unsolved-rate flags so
operators can tune both SloPolicy thresholds per dataset, matching the
existing --slo-multiplier flag.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@tvinagre @claude
feat: add fynd-benchmark generate-requests subcommand
3c1c00a 
Generate a per-chain synthetic swap-request dataset for capacity load
testing, covering chains the Ethereum-only download-trades set does not.

Adds fynd_rpc::protocols::fetch_token_pool_stats, which reuses the Tycho
pool-count query behind derive-connector-tokens to return per-token pool
counts and decimals. The new subcommand samples token pairs weighted by
pool count (seeded StdRng, no self-pairs), log-spaces amounts around one
whole token, and writes a --requests-file-compatible JSON array.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@tvinagre @claude
feat(client): support gateway auth header and path overrides
752d462 
FyndClientBuilder gains with_api_key (sent verbatim as the Authorization
header, no Bearer prefix, matching the auth gateway) and with_quote_path /
with_health_path so a chain-scoped base like https://host/v1/base joins to
/v1/base/quote. HTTP 429 responses map to a new non-retryable
FyndError::RateLimited carrying the Retry-After hint, so a rate-limited
request is not retried into extra load.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@tvinagre @claude
feat(benchmark): target auth gateway from capacity and load
baddaf1 
Wire FYND_API_KEY, QUOTE_PATH, and HEALTH_PATH into the capacity and load
subcommands so they can drive a solver behind the auth gateway at
https://fynd-api.propellerheads.xyz/v1/<chain>.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@tvinagre tvinagre mentioned this pull request Jul 19, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@tvinagre