From b8937e1ca9488f7c6e44cbb5ce203db046c86b4b Mon Sep 17 00:00:00 2001 From: Matt Kane Date: Mon, 13 Jul 2026 11:25:14 +0100 Subject: [PATCH] docs(release-service): select historical ingest source --- .../implementation-plan.md | 27 +++++---- .../plans/delegated-release-service/spec.md | 57 ++++++++++++++++--- 2 files changed, 65 insertions(+), 19 deletions(-) diff --git a/.opencode/plans/delegated-release-service/implementation-plan.md b/.opencode/plans/delegated-release-service/implementation-plan.md index 1286b1f82..80f67c486 100644 --- a/.opencode/plans/delegated-release-service/implementation-plan.md +++ b/.opencode/plans/delegated-release-service/implementation-plan.md @@ -2,7 +2,7 @@ Companion: [Implementation spec](./spec.md) -Status: implementation in progress; deployed-PDS validation is deferred to `W12.7`, and history feasibility validation remains open +Status: implementation in progress; deployed-PDS validation is deferred to `W12.7`; Gate 0B complete This plan turns the delegated release service spec into independently deliverable workstreams. It defines ownership boundaries, dependencies, integration gates, and completion criteria. It intentionally contains no time estimates. @@ -11,7 +11,7 @@ This plan turns the delegated release service spec into independently deliverabl | Stage | Deliverable | Repository change allowed | | ------- | ---------------------------------------------------------------------------- | ----------------------------------------------------------------- | | Gate 0A | Complete: confidential OAuth custody feasibility | Spec and plan updates only | -| Gate 0B | History feasibility: event-specific, verifiable aggregator input | Spec and plan updates only | +| Gate 0B | Complete: `subscribeRepos` `#commit` events selected; W10.1 constraints documented | Spec and plan updates only | | Gate 1 | Experimental lexicons, generated types, and one shared verification contract | Production contract code and tests | | Gate 2 | Secure delegated release service vertical slice | Service, client, and installer code and tests | | Gate 3 | Independent install and minimum discovery enforcement | Installer and aggregator code and tests | @@ -37,7 +37,7 @@ The integration branch includes these completed merge units: | `W2.5` | #1951 | Production public Sigstore provenance verification landed. | | `W0.4` | Direct | Confidential OAuth custody and refresh are compatible with workerd. | -`W0.6` remains external-validation work. Deployed-PDS compatibility from `W0.3` moves to conformance and production smoke. The next shared-verification merge unit is combined `W2.6` and `W2.7`; `W1.6` and `W1.7` land together to establish the exact supported scope contract. +`W0.6` is complete: `subscribeRepos` `#commit` events selected; see Gate 0B and the Aggregator Changes section of the spec for W10.1 constraints. Deployed-PDS compatibility from `W0.3` moves to conformance and production smoke. The next shared-verification merge unit is combined `W2.6` and `W2.7`; `W1.6` and `W1.7` land together to establish the exact supported scope contract. ## Outcomes @@ -157,14 +157,15 @@ Deployed-PDS create-only compatibility is validated by the conformance suite and ### Gate 0B: Historical Ingest Feasibility -Required before `W10.1` and production historical-policy claims: +Complete. -- An aggregator event source can recover intermediate profile values with verifiable ordering. -- The source supplies event-specific record values, CIDs, revisions, ordering keys, and proof material sufficient for the selected trust model. +- Source selected: `subscribeRepos` firehose `#commit` events. +- The source provides event-specific record values, CIDs, revisions (`rev`), ordering keys (`seq`), and verifiable commit proof material (signed commit block in CAR, MST inversion via `prevData`). +- Trust model, retention constraints, backfill limits, fork/rebase/tombstone handling, and explicit W10.1 constraints are documented in the Aggregator Changes section of the spec. Gate owner: `W0`. -Gate 0B evidence is a source-selection decision and explicit `W10.1` constraints. Failure changes the RFC's historical-policy and cooldown guarantees before `W10.1` through `W10.3` and `W10.5` through `W10.7` proceed; it does not block minimum current-policy filtering in `W10.4` or invalidate the service or installer architecture. +`W10.1` through `W10.3` and `W10.5` through `W10.7` may proceed. Minimum current-policy filtering in `W10.4` was already unblocked. ### Gate 1: Protocol and Verification Foundation @@ -302,9 +303,13 @@ Dependencies: `W0.1` provenance draft. ### `W0.6` Prove historical aggregator input -Determine whether an event source can recover profile states `strict -> relaxed -> strict`, with a release between transitions, after queue delay. The selected source must provide event-specific record values, ordering keys, CIDs, revisions, and verifiable commit proof material. +Result: complete. `subscribeRepos` firehose `#commit` events are the selected source. -Output: a source-selection decision and explicit W10.1 constraints. If no source can provide this, return to the RFC before implementing cooldown semantics. Do not add an aggregator prototype to this repository. +Each `#commit` event provides: `seq` (relay-scoped ordering key, not comparable across relay and direct-PDS sources), `rev` (TID repo revision, per-repo logical clock), `since` (the `rev` of the preceding commit for this repo, the per-repo chain link), `commit` CID, `blocks` (CAR slice with signed commit and MST diff), `ops` (per-record operations with new CID and, for updates/deletes, `prev` CID), and `prevData` (previous MST root, required for inductive MST inversion). The signed commit is verifiable against the DID's signing key that was valid at the commit's `rev`. MST inversion against `prevData` requires inductive firehose state from the prior processed commit; it is not independently verifiable from a single event in isolation. + +Jetstream (current production and the experimental archival rewrite under development), `getRepo`, `getRecord`, PDS repo history, and the `did:plc` audit log were evaluated and rejected: none provide event-specific record values with verifiable commit proof material for intermediate profile states. `#identity` events signal identity changes and require DID resolution; they do not carry key material. + +Trust model, retention constraints, backfill limits, fork/rebase/tombstone handling, and explicit W10.1 constraints are documented in the Aggregator Changes section of the spec. Key constraints: `seq` is relay-scoped and must not be used for per-repo continuity — use `#commit.since` and `rev` instead; the relay backfill window is hours to days (not permanent); publishers active before the aggregator's subscription start can only be bootstrapped from current state via `getRepo`, with prior policy events marked unrecoverable; commit signature verification requires the key valid at the commit's `rev` and is not possible retroactively if that key is no longer in the DID document; persisting only the signed commit block is insufficient for later independent re-verification — the full MST proof slice (signed commit, record block, MST diff nodes) must be retained or the verification scope explicitly limited to ingest time; `#sync` and `tooBig` events break the inductive chain and require `getRepo` re-sync with the gap marked unrecoverable. Dependencies: none. @@ -318,7 +323,7 @@ Dependencies: none. ### W0 Completion -The RFC-derived work (`W0.1`, `W0.2`, `W0.5`, and `W0.7`) and OAuth custody validation (`W0.4`) are complete, so Gate 0A is complete. `W0.3` is deferred to `W12.7`. Gate 0B passes independently when `W0.6` selects a viable historical event source. An incompatible deployed-PDS result changes the support matrix or affected RFC guarantee before support is claimed or production launches. +The RFC-derived work (`W0.1`, `W0.2`, `W0.5`, and `W0.7`) and OAuth custody validation (`W0.4`) are complete, so Gate 0A is complete. `W0.3` is deferred to `W12.7`. `W0.6` is complete: `subscribeRepos` firehose `#commit` events are the selected source, and Gate 0B is complete. An incompatible deployed-PDS result changes the support matrix or affected RFC guarantee before support is claimed or production launches. ## Workstream W1: Protocol and Lexicons @@ -1126,7 +1131,7 @@ Start these independently, with at most three implementation branches active at 1. `W2.6` + `W2.7`: structured record/policy verification over authoritative direct-PDS reads. 2. `W1.6` + `W1.7`: implement the typed exact scope and narrow create-only publishing helper. -3. `W0.6`: select historical aggregator input and record Gate 0B constraints independently. +3. `W0.6`: complete. Gate 0B is closed. 4. `W3.6`: required-UV passkey primitives. 5. `W5.1` + `W5.8`: unreachable service and API foundations. 6. After `W5.1`, `W4.1` + `W4.2` and `W5.1a`: workload verification and the isolated verifier Worker. diff --git a/.opencode/plans/delegated-release-service/spec.md b/.opencode/plans/delegated-release-service/spec.md index f1b2c5330..6385607cc 100644 --- a/.opencode/plans/delegated-release-service/spec.md +++ b/.opencode/plans/delegated-release-service/spec.md @@ -1,6 +1,6 @@ # Delegated Release Service Implementation Spec -Status: implementation in progress; deployed-PDS validation is deferred to production conformance, and aggregator-history validation remains open +Status: implementation in progress; deployed-PDS validation is deferred to production conformance; Gate 0B complete Source: [RFC PR #1870](https://github.com/emdash-cms/emdash/pull/1870), Attested Automated Publishing @@ -925,10 +925,51 @@ The aggregator needs event-history work before it can implement the RFC's "polic ### Ingest ordering -Current Jetstream jobs discard `time_us`, event CID, and repo revision, then fetch the current record by rkey. That loses intermediate profile versions. Merely adding those fields to the job is insufficient because a later PDS read still returns only the current value. Phase 0 must choose and prove an event source that provides the event-specific record value plus a verifiable commit/CAR path, or explicitly document a weaker trust model. Carry its ordering key, event CID, repo revision, and record blocks through the queue and persist every profile-policy version. - -Do not claim downgrade cooldown support until intermediate versions are provably retained. - +**Gate 0B decision (W0.6): `subscribeRepos` firehose `#commit` events are the selected source. Gate 0B is complete.** + +The `com.atproto.sync.subscribeRepos` firehose `#commit` event is the selected viable standardized ATProto source that provides event-specific record values with verifiable ordering and cryptographic proof material. Each `#commit` event carries: `seq` (relay-scoped monotonic ordering key, not comparable across relay and direct-PDS sources), `rev` (TID repo revision, per-repo logical clock), `since` (the `rev` of the preceding commit for this repo, enabling per-repo chain continuity checks), `commit` (CID of the signed commit object), `blocks` (CAR slice containing the signed commit and MST diff nodes), `ops` (per-record operations with new CID and, for updates and deletes, `prev` CID of the prior record version), and `prevData` (previous MST root CID, required for MST inversion). The signed commit in `blocks` is verifiable against the DID's signing key that was valid at the time of the commit. MST inversion against `prevData` confirms the ops list is complete and unmanipulated, but requires the aggregator to have maintained inductive firehose state from the prior commit; it is not independently verifiable from a single event in isolation. + +**Sources evaluated and rejected:** + +- **Jetstream (current production deployment and the experimental archival rewrite under development):** Both strip CAR blocks and MST proof material, providing JSON-decoded record values without commit signatures or MST proofs. The current production Jetstream does not provide `prev` CID for updates and has no historical backfill API for intermediate record states. The experimental archival rewrite stores full-network segments on disk and is not yet deployed to production; its on-disk format is still changing and it does not expose a stable API for retrieving intermediate record states with proof material. Neither is sufficient for a verifiable trust model. +- **`getRepo` (CAR export):** Returns current repo state only. The `since` parameter returns a diff from a given rev, but only for what the PDS still holds; intermediate values between two revisions are not recoverable if the PDS has since advanced. Cannot reconstruct intermediate profile states after the fact. +- **`getRecord`:** Returns a CAR proof for the current record only. No historical versions. +- **PDS repo history:** The `prev` field in v3 commit objects is virtually always `null`. No standard API enumerates historical commits from a PDS. +- **`did:plc` audit log:** Provides a complete, hash-linked, verifiable chain of DID document operations (PDS endpoint, signing keys, handles). Does not contain profile record content. Useful only for identity resolution, not profile policy history. `#identity` firehose events signal that identity may have changed and prompt re-resolution; they do not carry key material themselves. + +**Trust model and constraints for W10.1:** + +The aggregator must subscribe to `subscribeRepos` from the relay (or directly from each publisher's PDS) and process `#commit` events in real time. For each profile-collection commit, extract the record value from the CAR blocks at the CID indicated by the op, attempt commit signature verification against the DID's signing key that was valid at the commit's `rev`, and verify MST inversion against `prevData` using inductive state from the prior processed commit. Persist the event-specific record value, CID, `rev`, `seq`, the signed commit block, and the MST proof slice needed to bind the record transition (the signed commit block, the referenced record block, and the MST diff nodes required for inversion). Retaining only the signed commit block is insufficient for later independent verification; the full proof slice must be retained if post-hoc re-verification is required, or the verification scope must be explicitly limited to ingest time. + +Trust assumptions: +- `seq` is relay-scoped and monotonically increasing within a single relay connection. It must not be compared across relay and direct-PDS sources, or across different relay instances. Per-repo continuity is tracked via `#commit.since` (which must equal the last processed `rev` for that DID) and `rev` (which must increase monotonically per DID). Gaps in `since`/`rev` continuity, not gaps in `seq`, are the signal for per-repo desynchronization. +- Commit signature verification requires the DID's signing key that was valid at the time of the commit. `#identity` events signal that the DID document may have changed and require the aggregator to re-resolve the DID document; they do not carry key material. The aggregator must maintain a per-DID key history sufficient to verify commits against the key active at each `rev`. If historical key material is unavailable (the old key has been removed from the DID document and was not retained by the aggregator), the commit signature cannot be verified retroactively. This is an explicit limitation of the atproto signing model; such events are marked "signature unverifiable at ingest" but their record value is retained if the MST inversion check passed at ingest time. +- `prevData` and `op.prev` validation requires inductive firehose state: the aggregator must have successfully processed the prior commit for the same DID and retained its MST root. A `tooBig` event (deprecated but may appear from older producers) or a `#sync` event breaks the inductive chain; both must be treated as a gap requiring `getRepo` re-sync, not as proof of intermediate history. +- The relay backfill window is hours to days, not permanent. Profile events that occurred before the aggregator began consuming are not recoverable from the relay. For publishers who registered before the aggregator's subscription start, the aggregator can only recover the current profile state via `getRepo` and must mark all prior policy events as unrecoverable. + +**Retention and backfill constraints:** +- The aggregator must start consuming the firehose before any publisher registers. For publishers already registered at aggregator launch, bootstrap via `getRepo` to capture current state; mark historical policy events before the bootstrap rev as unrecoverable. +- The relay backfill window (hours to days) is the only replay mechanism. The aggregator must maintain a persistent cursor and reconnect within the window after any outage. Outages exceeding the backfill window require `getRepo` re-sync and mark the gap period as unrecoverable. +- `listReposByCollection` on the relay enumerates all DIDs with records in a given collection, enabling targeted backfill of known publishers. + +**Fork, rebase, and tombstone handling:** +- The `rebase` field in `#commit` is deprecated and unused in v3 repos. Treat it as always false. +- A `#sync` event resets the repo to a new state without providing intermediate history. On receipt of a `#sync`, mark the repo as desynchronized, fetch the full repo via `getRepo`, and mark any gap in policy events as unrecoverable. +- A `tooBig` event (deprecated; producers should always set it to `false`) breaks the inductive chain in the same way as `#sync`. Treat it as a gap requiring `getRepo` re-sync. +- Account deletion (`#account` with `active=false, status=deleted`) makes the repo unavailable. Mark all policy events after the last known rev as unrecoverable. Historical events already persisted remain valid. +- Account deactivation (`status=deactivated`) is temporary; resume on reactivation. + +**Explicit W10.1 constraints:** +- W10.1 must subscribe to `subscribeRepos` from the relay and process `#commit` events in real time. +- For each profile-collection op, extract the record value from the CAR blocks, attempt commit signature verification against the DID's signing key valid at the commit's `rev`, and verify MST inversion using inductive state from the prior processed commit for that DID. +- Persist: `seq`, `rev`, `since`, `commit` CID, record CID, record value (CBOR bytes), the signed commit block, and the MST proof slice (signed commit block, referenced record block, and MST diff nodes needed for inversion). Retaining only the signed commit block is insufficient for later independent re-verification; the full proof slice is required for that guarantee, or the verification scope must be explicitly documented as ingest-time only. +- Track per-DID last-seen `rev`, `prevData`, and inductive MST state. Use `#commit.since` to detect per-repo chain breaks; do not rely on `seq` gaps for per-repo continuity. +- On chain break, `#sync`, `tooBig`, or `getRepo` re-sync, mark the gap as unrecoverable. +- On `#identity`, re-resolve the DID document and update the cached key history; do not assume the event carries key material. +- For publishers bootstrapped from `getRepo` at aggregator launch, mark all policy events before the bootstrap rev as unrecoverable. +- Unrecoverable gaps must be surfaced in the policy history view and must not be silently treated as "no policy change occurred." +- Commit signature verification against a key no longer in the DID document is not possible retroactively; such events are marked "signature unverifiable" but their record value is retained if MST inversion passed at ingest time. +- Do not claim downgrade cooldown accuracy for any period marked unrecoverable. ### Schema Add: @@ -1075,10 +1116,10 @@ Run a second suite against real GitHub OIDC, a Bluesky-hosted PDS, and at least - Deferred to conformance and production smoke: validate create-only permission support on every supported deployed PDS without broad fallback. - Complete: confirmed `@atcute/oauth-node-client@2.0.1` confidential-client persistence, DPoP nonce retry, D1 lock requirements, and client-key rotation behavior in workerd. - Complete: inspect a real GitHub provenance bundle and land the Workers-compatible Sigstore verifier plus exact field mapping. -- Pending Gate 0B: select an aggregator history source that can retain event-specific signed profile values and document `W10.1` constraints. +- Complete (Gate 0B): `subscribeRepos` firehose `#commit` events selected as the aggregator history source. Trust model, retention constraints, fork/rebase/tombstone handling, and explicit `W10.1` constraints documented in the Aggregator Changes section. - Complete: use `emdash-plugin` as the v1 public command. -OAuth custody feasibility is complete. Gate 0B passes when historical ingest has a viable source and trust model. External history validation adds no repository harnesses, production code, test scripts, package dependencies, or CI wiring; commit conclusions directly to the integration branch. +OAuth custody feasibility is complete. Gate 0B is complete: `subscribeRepos` firehose `#commit` events are the selected source; trust model, constraints, and W10.1 requirements are documented in the Aggregator Changes section. ### Phase 1: Protocol and verification foundation @@ -1157,4 +1198,4 @@ Phase 4 is a production launch gate for the default public service and registry, ## Implementation Blockers to Resolve First -1. Decide the authoritative historical event source and trust model for aggregator policy ordering. +All Gate 0 blockers are resolved. Gate 0A (OAuth custody) and Gate 0B (historical ingest source) are complete. Deployed-PDS compatibility (`W0.3`) is deferred to `W12.7` conformance and production smoke.