Postgres network - #452
Open
Tomperez98 wants to merge 7 commits into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A new
PostgresNetworkthat speaks the Resonate wire protocol directly to Postgres via resonate-pg — a complete Resonate server implemented as stored procedures in a single SQL file. With this network there is no Resonate server process at all: the database is the server.Network selection is now scheme-aware and shared across all entry points:
resonate.network.network_for_urlroutes apostgres:///postgresql://URL (passed explicitly or viaRESONATE_URL) toPostgresNetworkand everything else toHttpNetwork, and both the full client (resonate.resonate.Resonate) and the serverless shim (resonate.faas.aws.Resonate) dispatch through it — a URL means the same thing everywhere. Every existing example runs unmodified withRESONATE_URL=postgresql://... just examples.Why talk to Postgres directly
psql -f resonate.sqlon any Postgres 16+ and every SDK worker connects straight to it. State, queue, and timers all live in the database you already operate (and back up).FOR UPDATE SKIP LOCKED: N workers sharing a group address each dequeue disjoint messages, giving exactly-once dispatch and load balancing for free — no NATS/SSE infrastructure.SupabaseNetworkuses. The database HTTP-pushesexecutemessages to anyhttps://target viapg_net— a Supabase edge function or an AWS Lambda Function URL — and the function only needs the outbound RPC path back. Theresonate.faas.awsshim now routes its server URL through the same sharednetwork_for_urlselection, so settingRESONATE_URL=postgresql://...on a Lambda gives it a send-onlyPostgresNetwork: no poll connection, one task driven per invocation, and the database re-delivers on lease expiry (lapsed leases flip back topendingviaprocess_timeouts, with task-version fencing so a zombie invocation can never commit). Long-running Python workers and serverless functions can share one database.How it works
SELECT resonate.resonate_rpc($1::jsonb)per request over a shared asyncpg pool. Identical JSON envelope to the HTTP transport. Connection-level failures retry with the same 1s→60s backoff asHttpNetwork(unblocked immediately bystop()); database errors (schema not installed, EXECUTE revoked) surface once as the newresonate.error.PostgresErrorwithout retrying.NOTIFYchannels (resonate_q_<md5(address)>, matching the server'soutbox_channel). Notifications are wake-only signals; the messages stay in the outbox until destructively dequeued (dequeue_execute/dequeue_unblock) and re-wrapped into the same execute/unblock envelopesTransportalready parses. A drain also runs on (re)connect and on a 5s fallback tick, so a missed notification is latency, never data loss.pg://uni@group/pid(unicast) andpg://any@group(anycast). Note for reviewers: unlikepoll://any@group/pid, the anycast address deliberately has no pid suffix — resonate-pg treats addresses as opaque strings, so group members must advertise the identical string forSKIP LOCKEDto load-balance across them.statement_cache_size=0keeps the pool compatible with transaction-mode poolers (pgbouncer, Supabase's Supavisor), mirroring the TS client'sprepare: false.Requirements
resonate.sqlapplied.ctx.sleep, promise timeouts, schedules, lease expiry); it pollsresonate.process_timeouts()every 5s. The RPC/outbox/NOTIFY paths work without it.resonate-sdk[postgres]) like[nats]? That would require making the import inresonate/network/__init__.pylazy.resonate_worker(the SQL file revokes EXECUTE from PUBLIC).Testing
select = ALL) and ty clean.HttpNetworksuite: identity/addressing, retry-through-outage, stop-unblocks-backoff, no-retry-on-database-errors, drain dispatch — plus a golden test pinning the channel md5 derivation to the SQLoutbox_channeldefinition (verified against a live server; drift here means LISTEN silently subscribes to a channel nobody notifies).postgres:17container withresonate.sqlapplied:promise.create→ execute delivered via NOTIFY+dequeue →task.acquire→ listener/unblock delivery →task.fulfill;fibviactx.rpc+ctx.run) through the fullResonateclient;resonate.faas.awshandler: SQLresonate.invoke()→ execute message → Lambda handler with apostgresql://RESONATE_URL→ send-onlyPostgresNetwork→ task acquired, children fenced, promise fulfilled — including a three-invocation suspend/resume sequence (parent suspends on actx.rpcchild, child completes, resumed parent replays and finishes, with the task version fenced up on resume);just examples(fibonacci ×3 modes, error-handling ×6, saga ×3 incl. compensation paths, human-in-the-loop, recovery, detached, polling, structured-concurrency, retries, pydantic, pydantic-ai incl.ctx.sleepdurable timers), outputs verified — not just exit codes.