Record & replay OpenAI/OpenRouter stream usage (incl. OpenRouter usage.cost) #368

Summary

When recording a streaming OpenAI-compatible chat completion (including OpenRouter), aimock collapses the SSE body into a fixture that keeps content / reasoning / tool calls / timings — but drops the final usage chunk. On replay, token counts are either estimated (ceil(len/4)) or taken from a hand-authored response.usage override. OpenRouter's usage.cost is never recorded or replayed.

That makes it impossible to e2e-test consumer billing that depends on provider-reported usage from a recorded fixture tape (the same class of gap as #269 for fal x-fal-billable-units, which was fixed).

Observed with @copilotkit/aimock@1.37.1.

Where (record)

collapseOpenAISSE only accumulates deltas with non-empty choices. The OpenAI / OpenRouter final usage frame is exactly:

{
  "id": "…",
  "object": "chat.completion.chunk",
  "choices": [],
  "usage": {
    "prompt_tokens": 1234,
    "completion_tokens": 567,
    "total_tokens": 1801,
    "cost": 0.0042
  }
}

(stream_options.include_usage: true — OpenRouter also includes cost in USD.)

In the collapser, empty choices → continue, so usage never reaches CollapseResult. CollapseResult itself has no usage field, so the recorder cannot persist it even if it wanted to. Recorded fixtures end up:

{
  "match": { "userMessage": "…", "model": "…" },
  "response": { "content": "…" },
  "recordedTimings": { "ttftMs": …, "interChunkDelaysMs": […] }
}

Live proxy still forwards the real stream to the client (billing works during a record run). Only the saved fixture loses usage, so replay cannot exercise the cost path.

Where (replay)

• resolveUsage returns only { prompt_tokens, completion_tokens, total_tokens } from fixture overrides or length estimates — no cost.
• buildUsageChunk can emit a final empty-choices usage chunk when stream_options.include_usage is set, but only with those token fields.
• Fixture ResponseOverrides.usage is typed without cost / provider extensions.

Impact

Apps that bill from provider-reported cost (OpenRouter usage.cost, not a local rate table) cannot:

1. Re-record fixtures and automatically retain usage.
2. Replay and assert wallet / ledger deductions against real recorded amounts.
3. Distinguish “capture path broken” vs “mock never emits cost” in full-pipeline e2e.

Hand-editing response.usage with token counts is incomplete for OpenRouter billing; inventing rates in the app papers over missing provider cost.

Proposed fix

Mirror the fal billableUnits approach (#269):

1. Record: In collapseOpenAISSE (and non-stream JSON collapse if applicable), capture the last non-null usage object from the upstream body into CollapseResult.usage (pass through extra fields such as OpenRouter cost / native_tokens_* / prompt_tokens_details).
2. Persist: Write response.usage on the fixture when present (back-compat: omit when absent).
3. Replay:
   • Prefer fixture usage over estimateTokens.
   • Emit it on the final stream usage chunk and on non-stream completion envelopes verbatim (or at least preserve cost and the standard token fields).
   • Extend ResponseOverrides.usage types to allow cost?: number and unknown extra keys.
4. Optional: document that OpenRouter cost requires recording with stream_options.include_usage: true (or non-stream responses that include usage).

Happy to help test a PR from the OpenStory full-pipeline suite (record against OpenRouter, assert usage.cost round-trips on replay).

Related

• fal queue responses omit x-fal-request-id / x-fal-billable-units, so @tanstack/ai-fal can't surface unitsBilled on replay #269 — fal x-fal-billable-units / request-id (same “billing metadata lost on mock” theme; fixed for fal)
• Consumer path: TanStack AI / OpenRouter adapters surface stream RUN_FINISHED.usage including OpenRouter cost when the mock actually emits it