Skip to content

SCALING.md

Latest commit

 

History

History
221 lines (180 loc) · 11.3 KB

SCALING.md

File metadata and controls

221 lines (180 loc) · 11.3 KB
Raw

English · Português

Horizontal scaling in quark

quark scales horizontally by sharing storage across replicas. There are three deployment shapes, with different limits: pick the one that matches what you need. The subsystem-by-subsystem audit behind this page, with file:line evidence, is docs/research/2026-07-14-scale-audit.md.

The three shapes

flowchart TB
    subgraph F1["1. Pure binary (default)"]
        A1[single quark] --> L1[(local LMDB)]
    end
    subgraph F2["2. Replicas + shared Postgres + Valkey (recommended for multi-node)"]
        LB[Load Balancer] --> R1[quark 1]
        LB --> R2[quark 2]
        LB --> R3[quark N]
        R1 --> PG[(shared Postgres)]
        R2 --> PG
        R3 --> PG
        R1 -.pub/sub + L2.-> VK[(Valkey)]
        R2 -.pub/sub + L2.-> VK
        R3 -.pub/sub + L2.-> VK
    end
    subgraph F3["3. Multiple LMDB binaries (not a real multi-node)"]
        N1[quark node 0] --> D1[(local LMDB 0)]
        N2[quark node 1] --> D2[(local LMDB 1)]
    end
Loading
Shape Storage Multi-node Note
1. Pure binary Embedded LMDB No (1 node) Minimal footprint; up to 2^40 links
2. Replicas + Postgres + Valkey Shared Postgres + Valkey Yes Recommended path; any replica serves any link
3. Multiple LMDB Local LMDB per node Not for reads Each node only has the data it created (see limits below)

The honest scale matrix

Not every subsystem scales the same way. A "multi-node" deployment that shares the store but not Valkey is still degraded: rate limits become N-times the configured value, and cache coordination lags. Here is what each subsystem actually does per deployment shape.

Subsystem Single-node (LMDB) Multi-node (Postgres + Valkey + ClickHouse)
Redirect (hot path) fine, one node computed code + cache tier, any replica serves any link
ID allocation per-node counter + node_id prefix (needs a unique node_id) shared quark_id_seq, coordinated across replicas
Rate limit in-memory, per node (correct on one node) atomic global counter in Valkey
Cache per-node L1 (correct: store is not shared) per-node L1 + shared L2, near-instant invalidation via pub/sub
Analytics aggregation per-node blob read-modify-write (correct on one node) atomic Postgres counters (INSERT ... ON CONFLICT), or ClickHouse append-only + aggregate-on-read
Click ingestion bounded channel, try_send drops when full (at-most-once by design) same on every backend
Webhook delivery: lifecycle in-memory best-effort channel (no outbox on LMDB) durable Postgres outbox + leased relay, at-least-once
Webhook delivery: clicked/expired in-memory best-effort (hot path) in-memory best-effort by design (hot path)

Single-node (default): LMDB with the in-memory cache, rate-limit, and blob analytics is correct and needs no external dependency. This is the pure-binary shape.

Multi-node: requires Postgres (shared store) plus Valkey (shared rate-limit and cross-node cache invalidation). ClickHouse is recommended for analytics at high volume; the Postgres analytics path is also correct at scale because it uses atomic per-counter increments, not a per-link read-modify-write.

How to actually scale (shape 2)

Bring up N copies of the binary behind a load balancer, all with the same QUARK_KEY, the same QUARK_DATABASE_URL pointing at the shared Postgres, and the same QUARK_VALKEY_URL:

  • Unique ids: Postgres's quark_id_seq sequence is atomic and cluster-wide, so concurrent replicas never generate the same id. The permute width is 40 bits, so the global ceiling is 2^40 links (about 1.1 trillion) across the whole cluster.
  • Shared data: every replica reads/writes the same tables; no session affinity is needed (the load balancer can be plain round-robin).
  • Shared rate-limit and invalidation: point every replica at the same Valkey. Without it, each replica keeps its own in-memory counter and the effective rate limit becomes N-times the configured value, and cache changes propagate only on the per-node TTL.
  • Fail fast if you meant to cluster: set QUARK_STRICT_CLUSTER=1 on every replica and quark refuses to start unless both QUARK_DATABASE_URL and QUARK_VALKEY_URL are present. Any non-empty value turns it on. This turns a silent misconfiguration (N-times rate limits, stale caches, per-node LMDB files) into a startup error. Single-node deployments leave it unset and are unaffected.

Cross-node consistency windows

The cache is eventually consistent between replicas, bounded and closed by the Valkey pub/sub invalidation channel (quark:invalidate, in src/invalidate.rs):

  • Cache (patch/delete): without pub/sub, another replica's L1 can serve a stale link until its per-node TTL expires (60s). Each admin mutation publishes link:<id> and every replica drops that L1 entry on receipt, so the window drops from up to 60s to near-instant. The TTL stays as the backstop if a replica misses a message. The publish is bounded by a 100ms timeout and is fail-open, so a slow Valkey never blocks the admin write.

The subscriber applies each message to the local L1 only and never re-publishes, so there is no cross-node loop.

Multi-region reads: the read/write split

A single Postgres primary works across regions, but every read then pays the round trip to wherever the primary lives. To keep redirects near the user in a multi-region deployment, quark can read from a local Postgres read replica while all writes still go to the single primary.

  • Set QUARK_DATABASE_URL to the primary (all writes go here) and QUARK_REPLICA_DATABASE_URL to a local read replica. Reads then use the replica; writes stay on the primary.
  • Leave QUARK_REPLICA_DATABASE_URL unset and behavior is identical to today: both the read and the write path use the primary. The split is opt-in, and single-region or LMDB deployments are unchanged.

The replica gets its schema through Postgres streaming replication, so quark runs the schema init and every migration on the primary only, never on the replica.

Which reads stay on the primary. Three low-volume, correctness-sensitive reads are forced to the primary so a lagging replica can never serve stale auth state:

  • session lookup by token hash (a user who just logged in must not be 401'd by the panel right after),
  • API token lookup by hash (a freshly minted token must authenticate right away),
  • the Sheets connection (read immediately after the OAuth callback writes it).

Everything else reads the replica: the redirect hot path (link/alias lookup), the admin listings, and the analytics dashboards.

The consistency window. Replication is asynchronous, usually sub-second, so a read on the replica can trail the primary by the replication lag:

  • A brand-new link may 404 in a distant region for the lag window until the row replicates. The create response returns the computed code directly (it never reads it back), so the API and the panel are not affected. Only an immediate cross-region redirect races the lag, and it is bounded and closes on the same order as the cache TTL.
  • Analytics counts can trail the primary by the lag. This already matches quark's at-most-once, eventually-aggregated analytics model.

A true streaming replica is exercised on Fly.io; CI has no replica, so the gated tests point both URLs at the same database and only prove the routing wiring.

Analytics ingestion is at-most-once

A click is handed to the analytics worker through a bounded in-process channel (try_send, 10,000 capacity). Under a burst that fills the channel the redirect drops the event rather than block the 302. This is deliberate: the redirect hot path must never wait on analytics. The aggregation that follows is correct (atomic counters on Postgres, append on ClickHouse), but the ingestion itself is at-most-once, so treat click counts under extreme peaks as sampled, not exact.

Durable webhook delivery

On the Postgres backend the lifecycle events (link.created, link.updated, link.deleted) are durable. Each fires one row per matching active subscription into the webhook_deliveries outbox, and a leased relay worker delivers them at-least-once:

  • The relay claims a batch of due rows with SELECT ... FOR UPDATE SKIP LOCKED, so N replicas each take a disjoint set and never double-send, and a slow endpoint holds up only its own rows.
  • Retry is persisted: on failure the attempt count is bumped and the next-attempt time is pushed out with exponential backoff plus jitter, surviving a restart. After 8 attempts the row is flagged dead (dead-letter) and stops being claimed.
  • Idempotency: the webhook-id header is the row's stable delivery key (<event_id>.<subscription_id>), identical across every attempt and node.

link.clicked and link.expired stay best-effort on every backend, because they fire on the redirect hot path and a synchronous outbox write there would defeat its purpose. On the LMDB backend there is no outbox at all; every event, lifecycle included, rides the in-memory best-effort channel. See WEBHOOKS for the full delivery model.

Enqueue is atomic with the mutation. The outbox rows are inserted in the same transaction as the link mutation that produced the event. The handler reads the matching active subscriptions (outside the transaction) and hands the delivery rows to the storage layer, which commits the link change and the outbox inserts together. Either both land or neither does, so a crash can no longer lose an event between the link write and the outbox insert.

QUARK_NODE_ID: defensive LMDB partitioning

quark's code space is 40 bits. When QUARK_NODE_ID is set (0-255), the top 8 bits identify the node and the low 32 bits become that node's local counter:

Node bits Local bits Max nodes Links per node
8 32 256 ~4.3 billion
  • Unset (default): normal behavior, the counter uses the full 40 bits (~1.1 trillion links). This is single-node mode.
  • All-or-nothing rule: either every node runs without QUARK_NODE_ID (= 1 node), or every node runs with a distinct QUARK_NODE_ID. Never mix an un-partitioned node (full range) with partitioned ones: the spaces overlap.
  • Uniqueness is on you: the id MUST be unique per replica (a StatefulSet ordinal is a natural source). quark cannot detect a duplicate; two nodes with the same id silently reuse the same code space and collide.
  • An invalid QUARK_NODE_ID (outside 0-255) crashes the process at startup.
  • QUARK_NODE_ID is LMDB-only. On the Postgres backend it is ignored (the shared sequence handles allocation) and quark logs that it was ignored. The Postgres path has a single global ceiling of 2^40 links, not a per-node one.

The honest limit of shape 3

QUARK_NODE_ID guarantees that two LMDB nodes will not generate the same code, but it does not make one node serve another node's links. Each LMDB is local: a redirect that lands on the wrong node returns 404, because that node does not have the data. In other words, node-id is a collision guard-rail, not a real multi-node mode.

By design, a pure binary (LMDB, no database) is single-node: this is a deliberate constraint of the system, not a limitation to be removed. For multi-node, use shape 2 (shared Postgres + Valkey).