EventLog pending remote writes are not retried on the existing backoff schedule after transient failures

[gjermundgaraba]

What version of Effect is running?

4.0.0-rc.111 and main@436f10d1e

What steps can reproduce the bug?

1. Create a local EventLog entry.
2. Register a fake remote whose first two write calls fail and subsequent writes succeed; on the affected base, those calls are the initial flush and the existing one-shot scope finalizer.
3. Wait until both failed attempts have occurred, then advance TestClock by 200 ms without creating another local entry.
4. Observe that write is not called a third time.

A minimal fake remote follows this shape:

const writes = yield* Queue.unbounded<void>()
let attempts = 0
const remote = EventLogRemote.EventLogRemote.of({
  id: EventJournal.makeRemoteIdUnsafe(),
  changes: () => Queue.unbounded<EventJournal.RemoteEntry, EventLogRemote.EventLogRemoteError>(),
  write: () => {
    const attempt = ++attempts
    return Queue.offer(writes, undefined).pipe(
      Effect.andThen(
        attempt <= 2
          ? Effect.fail(new EventLogRemote.EventLogRemoteError({ method: "write", cause: "offline" }))
          : Effect.void
      )
    )
  },
  whenAuthenticated: (effect) => effect
})

yield* log.write({ schema, event: "UserCreated", payload: { id: "user-1" } })
yield* registry.registerRemote(remote)
yield* Queue.take(writes)
yield* Queue.take(writes)
yield* TestClock.adjust("200 millis")
yield* Effect.yieldNow

// attempts is still 2. The base ran the initial flush and its one-shot
// scope finalizer, then terminated the worker without scheduling a third write.

What is the expected behavior?

Pending local entries should be retried after the remote becomes writable again, without requiring another local journal change. They should remain pending in the configured local journal while writes fail. Shutdown should interrupt scheduled retries and run only the existing one-shot final write.

What do you see instead?

Pending entries remain stored in the configured local journal, but the remote becoming writable again does not itself flush them.

• An initial outgoing flush failure terminates the per-remote worker after its existing scope finalizer makes one immediate best-effort write. If that final write also fails, pending entries receive no further attempts.
• After a successful initial flush, a change-triggered failure is logged and the loop returns to waiting for another journal notification.
• Only the latter case is attempted again after a subsequent successful local journal write; a local write cannot restart a worker terminated by the initial failure.

Additional information

The incoming remote stream already uses an exponential retry schedule capped at 10 seconds. The outgoing EventLog worker has no corresponding general retry policy for transient write failures; EventLogRemote.makeWith only retries Forbidden responses while re-authenticating.

A patch is ready that:

• applies the existing retry schedule to initial and change-triggered outgoing flushes;
• continuously drains journal notifications into a one-slot wake-up queue so a prolonged outage does not retain entry payloads in the unbounded PubSub subscription;
• keeps the shutdown finalizer as a one-shot write; and
• adds deterministic regression coverage for initial retries, later change-triggered retries, preserving a wake-up that arrives during an in-flight retry, and draining the journal PubSub subscription while a retry is blocked.