Skip to content

Postgres network - #452

Open
Tomperez98 wants to merge 7 commits into
mainfrom
postgres
Open

Postgres network#452
Tomperez98 wants to merge 7 commits into
mainfrom
postgres

Conversation

@Tomperez98

@Tomperez98 Tomperez98 commented Jul 12, 2026

Copy link
Copy Markdown
Member

What

A new PostgresNetwork that speaks the Resonate wire protocol directly to Postgres via resonate-pg — a complete Resonate server implemented as stored procedures in a single SQL file. With this network there is no Resonate server process at all: the database is the server.

r = Resonate(url="postgresql://user:pw@host:5432/db")   # that's it

Network selection is now scheme-aware and shared across all entry points: resonate.network.network_for_url routes a postgres:// / postgresql:// URL (passed explicitly or via RESONATE_URL) to PostgresNetwork and everything else to HttpNetwork, and both the full client (resonate.resonate.Resonate) and the serverless shim (resonate.faas.aws.Resonate) dispatch through it — a URL means the same thing everywhere. Every existing example runs unmodified with RESONATE_URL=postgresql://... just examples.

Why talk to Postgres directly

  • One less moving part. No Resonate server to deploy, monitor, or keep alive next to the database. psql -f resonate.sql on any Postgres 16+ and every SDK worker connects straight to it. State, queue, and timers all live in the database you already operate (and back up).
  • Transactional by construction. Every protocol action (promise create/settle, task acquire/fulfill, callbacks) is a stored procedure, so each step commits atomically with the state it mutates — there is no window where the queue and the promise store disagree.
  • Anycast without a broker. Message delivery is an outbox table drained with FOR UPDATE SKIP LOCKED: N workers sharing a group address each dequeue disjoint messages, giving exactly-once dispatch and load balancing for free — no NATS/SSE infrastructure.
  • Serverless-native. This is the same wire target the TS SupabaseNetwork uses. The database HTTP-pushes execute messages to any https:// target via pg_net — a Supabase edge function or an AWS Lambda Function URL — and the function only needs the outbound RPC path back. The resonate.faas.aws shim now routes its server URL through the same shared network_for_url selection, so setting RESONATE_URL=postgresql://... on a Lambda gives it a send-only PostgresNetwork: no poll connection, one task driven per invocation, and the database re-delivers on lease expiry (lapsed leases flip back to pending via process_timeouts, with task-version fencing so a zombie invocation can never commit). Long-running Python workers and serverless functions can share one database.

How it works

  • send — one SELECT resonate.resonate_rpc($1::jsonb) per request over a shared asyncpg pool. Identical JSON envelope to the HTTP transport. Connection-level failures retry with the same 1s→60s backoff as HttpNetwork (unblocked immediately by stop()); database errors (schema not installed, EXECUTE revoked) surface once as the new resonate.error.PostgresError without retrying.
  • recv — a dedicated connection LISTENs on per-address NOTIFY channels (resonate_q_<md5(address)>, matching the server's outbox_channel). Notifications are wake-only signals; the messages stay in the outbox until destructively dequeued (dequeue_execute / dequeue_unblock) and re-wrapped into the same execute/unblock envelopes Transport already parses. A drain also runs on (re)connect and on a 5s fallback tick, so a missed notification is latency, never data loss.
  • Addressespg://uni@group/pid (unicast) and pg://any@group (anycast). Note for reviewers: unlike poll://any@group/pid, the anycast address deliberately has no pid suffix — resonate-pg treats addresses as opaque strings, so group members must advertise the identical string for SKIP LOCKED to load-balance across them.
  • statement_cache_size=0 keeps the pool compatible with transaction-mode poolers (pgbouncer, Supabase's Supavisor), mirroring the TS client's prepare: false.

Requirements

  • Postgres 16+ with resonate-pg's resonate.sql applied.
  • pg_cron — required for durable timers (ctx.sleep, promise timeouts, schedules, lease expiry); it polls resonate.process_timeouts() every 5s. The RPC/outbox/NOTIFY paths work without it.
  • pg_net or pgsql_http — optional, only for HTTP push delivery (the serverless path).
  • asyncpg ≥ 0.31 — added as a main dependency. Open question for reviewers: should this be an optional extra (resonate-sdk[postgres]) like [nats]? That would require making the import in resonate/network/__init__.py lazy.
  • Production connections should use a role granted resonate_worker (the SQL file revokes EXECUTE from PUBLIC).

Testing

  • 746 tests pass; ruff (select = ALL) and ty clean.
  • New unit tests mirror the HttpNetwork suite: identity/addressing, retry-through-outage, stop-unblocks-backoff, no-retry-on-database-errors, drain dispatch — plus a golden test pinning the channel md5 derivation to the SQL outbox_channel definition (verified against a live server; drift here means LISTEN silently subscribes to a channel nobody notifies).
  • End-to-end against a real postgres:17 container with resonate.sql applied:
    • raw protocol loop: promise.create → execute delivered via NOTIFY+dequeue → task.acquire → listener/unblock delivery → task.fulfill;
    • a recursive durable workflow (fib via ctx.rpc + ctx.run) through the full Resonate client;
    • the serverless loop through the real resonate.faas.aws handler: SQL resonate.invoke() → execute message → Lambda handler with a postgresql:// RESONATE_URL → send-only PostgresNetwork → task acquired, children fenced, promise fulfilled — including a three-invocation suspend/resume sequence (parent suspends on a ctx.rpc child, child completes, resumed parent replays and finishes, with the task version fenced up on resume);
    • all 25 example invocations from just examples (fibonacci ×3 modes, error-handling ×6, saga ×3 incl. compensation paths, human-in-the-loop, recovery, detached, polling, structured-concurrency, retries, pydantic, pydantic-ai incl. ctx.sleep durable timers), outputs verified — not just exit codes.

Base automatically changed from ext-pydantic-ai to main July 14, 2026 17:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant