Deterministic durable ids under concurrency via task lineage

[Tomperez98]

The bug

A workflow that runs durable chains concurrently returns the wrong values after replay — deterministically wrong, every run:

async def wf(ctx):
    timer = ctx.sleep(timedelta(seconds=0.2))       # pending -> forces suspend + replay

    async def chain_a():
        a = await ctx.run(slow)                      # loses the live race (sleeps 0.1s)
        return await ctx.run(mark, "s2", a)

    async def chain_b():
        b = await ctx.run(fast)                      # wins the live race
        return await ctx.run(mark, "f2", b)

    task_a = asyncio.create_task(chain_a())
    task_b = asyncio.create_task(chain_b())
    r1, r2 = await task_a, await task_b
    await timer                                      # suspend, resume, replay from top
    return [r1, r2]

# expected ['s2:slow-result', 'f2:fast-result']
# got      ['f2:fast-result', 's2:slow-result']

Child ids were minted from one shared per-execution counter (Context._next_id) advanced at ctx.run call time. That maps a logical op to the same id on every replay only if the global call order is identical on every replay — true for sequential code, false under concurrency: which chain reaches its second ctx.run first is gated on which first-stage future settled first, and that is decided by wall-clock on the live pass but by instant promise recovery on the replay pass. The orders disagree, the ids swap, and each chain recovers the other chain's durably stored value.

Not a gather bug (create_task, TaskGroup, ensure_future all reproduce it), and not fixable by the creation Chain — ids already follow call order; the defect is that call order itself is not replay-stable. Full diagnosis in docs/concurrent-id-determinism.md.

The fix

Key ids on task structure instead of call order. Two facts are replay-stable, inductively, under the same determinism assumption sequential code already relies on:

• asyncio.create_task is called synchronously from the parent task, so within one task, spawn order is program order;
• within one task, code runs sequentially, so its durable-op order is program order.

New module resonate/lineage.py: a loop task factory (installed idempotently from Core.execute_until_blocked_inner, wrapping any pre-installed factory) stamps every task with a lineage path — its parent's path plus a spawn index — assigned synchronously inside create_task, before the event loop has any say in scheduling. Carried in contextvars, so it survives eager task factories and inline-awaited coroutines share their caller's path.

Context._next_id then keys its seq counter per lineage path relative to the state's anchor (the task its body runs in):

wf.1              ctx.sleep         path ()   -- body's own sequential slot, flat id as before
wf.t2.1  slow     path (2,) seq 1   -- chain_a's task branch
wf.t2.2  mark     path (2,) seq 2
wf.t3.1  fast     path (3,) seq 1   -- chain_b's task branch
wf.t3.2  mark     path (3,) seq 2

The id is a pure function of (task path, per-path call index) — both replay-stable — so the swap is structurally impossible. The spawn counter is re-armed at the user-code boundary (invoke_with_retry, per attempt) so tasks the SDK or the network stack (aiohttp/nats internals) spawns from the same task beforehand cannot skew user task indices.

Bare create_task / gather / TaskGroup / wait_for now replay deterministically — no new API, no combinator, no user-supplied ids. Sequential code has an empty path and keeps the exact flat {id}.{seq} ids it always had.

Trade-offs

• A loop-global hook. loop.set_task_factory is installed on whatever loop runs workflows — usually the application's. It wraps any existing factory and is a no-op for untracked tasks, but it is a process-visible mutation, and if something later replaces the factory, tracking silently degrades (see next point).
• Silent fallback, no misuse guard. A durable op reached from a task outside the tracked lineage tree (factory replaced, task smuggled across trees) degrades to the empty path — i.e. the old shared-counter behavior, which under concurrency is the old nondeterminism. Graceful for sequential code and existing tests; silent for the pathological cases. A loud detect-and-raise guard could be layered on later.
• Ids are structural, so structure changes re-map them. Adding/removing a create_task call site — including SDK-internal spawns like ctx.sleep's background task — shifts spawn indices and therefore ids. This is the same class of caveat the flat seq already had for statement order (in-flight workflows across a redeploy that changes workflow structure can mis-recover), but the sensitive surface now includes task-spawn sites, and an SDK upgrade that changes its own internal spawn pattern inside user tasks would shift indices too.
• User nondeterminism still breaks replay. Spawning tasks conditionally on a race outcome (e.g. create_task only for whichever future finished first) makes spawn order itself nondeterministic — out of scope, same contract as sequential code's determinism assumption.
• Id shape. Deep task nesting grows ids by one t{n} segment per level. The t marker keeps path segments from ever colliding with child-context promise ids (a promise id always ends in a numeric seq segment).
• Rust SDK parity. This port tracks the Rust SDK 1:1; this change diverges until the same lineage scheme is mirrored there so both SDKs share one concurrency contract.

Compared to the alternatives from the design doc: a ctx.gather combinator (option B) forces an API and leaves raw asyncio silently unsafe; deterministic replay via recorded settlement order (option C) needs server support and a scheduler-gate in the runtime core. This sits in between: option C's ergonomics (bare asyncio works) at roughly option B's implementation cost, paid for with the structural-id caveats above.

Testing

• tests/test_lineage.py — unit tests for the lineage module (creation-order indices, nesting, gather argument order, inline awaits, boundary reset, untracked/foreign fallback, factory wrap + idempotence).
• test_run_concurrent_task_chains_replay_deterministically in tests/test_resonate.py — end-to-end suspend/replay race against the in-process LocalNetwork; fails with the old id scheme, passes now.
• Full suite: 685 passed; ruff, format, ty clean.
• Repro script (the wf above, printing minted ids per pass) run 10/10 consecutive passes against a real resonate dev server; ids identical across live and replay passes. Full just examples suite passes against the same server.

🤖 Generated with Claude Code