feat: structured NDJSON progress events on stderr with MCP progressToken mapping

[rshade]

Overview

Let long-running commands emit structured NDJSON progress events on stderr,
and have the MCP adapter map them to notifications/progress (progressToken)
so hosts render live progress. Today an agent gets silence from command start
until the final payload lands, with no way to distinguish "still working" from
"hung."

User Story

As an LLM agent (or its host) invoking a long-running ax-go command, I want
structured progress events during execution, So that I can show progress,
detect stalls, and decide whether to keep waiting instead of guessing from
silence.

Problem Statement

stderr is already the diagnostics channel (stream-separation mandate), but the
library emits nothing there during a long operation except opportunistic
zerolog lines — which are unstructured for progress purposes and not mapped to
any MCP notification. Under the MCP server runtime (mcp.Serve), the dispatcher
buffers the command's entire stdout and returns it as one text block only after
the command exits (internal/mcpserver/dispatch.go successResult), so a host
sees no incremental signal at all. The design questions are (1) the progress
event schema and (2) how it maps to the MCP progressToken mechanism when the
host advertises the capability.

Proposed Solution

Define a small, additive progress-event contract and a runtime emitter, plus an
MCP bridge that fires notifications/progress only when the host cooperates.

Technical Approach

• Define a ProgressEvent struct in contract/ (stable JSON shape): fields such
  as progress (completed units), total (optional), message, and correlation
  IDs (trace_id/span_id reusing contract.Metadata conventions). Keep it a
  struct, not a map, for determinism.
• Add a runtime emitter (root ax package) that writes one NDJSON event per line
  to stderr via the existing locked stderr writer
  (internal/telemetry.NewLockedWriter, already wired in execute.go:119) so
  progress lines never interleave with logs or the error envelope. Use zerolog
  field methods / struct marshaling — never fmt.Sprintf of user input
  (log-forgery guardrail).
• MCP mapping ("Declared Claims, Gated Projections"): the dispatcher reads the
  client's progressToken from the tool-call params and translates each stderr
  progress event into an SDK notifications/progress message
  (github.com/modelcontextprotocol/go-sdk/mcp) — but ONLY when the host
  advertised the capability AND the session is interactive; otherwise the events
  stay on stderr and behavior is unchanged (fall back, don't fail).
• Because the current dispatcher buffers stdout to completion, wiring live
  progress requires reading stderr incrementally during dispatch rather than only
  forwarding it after ExecuteContext returns (forwardStderr).

Files Likely Affected

• contract/ (new file, e.g. progress.go) - ProgressEvent shape + JSON contract
• logger.go / new root emitter - stderr NDJSON progress writer using the locked writer
• execute.go - expose the progress sink via context (like mode/dry-run/idempotency)
• internal/mcpserver/dispatch.go - capture progressToken, translate events to notifications/progress, incremental stderr handling
• mcp/server.go / mcp/options.go - surface any host-capability gating knobs
• examples/integration/main.go - a demo long-running command emitting progress

Acceptance Criteria

• A command can emit structured ProgressEvent NDJSON lines to stderr with stable field shapes
• Progress events never appear on stdout (stream-separation invariant holds)
• Under MCP with a host-advertised progressToken, events surface as notifications/progress; without it, events remain on stderr and dispatch behavior is unchanged
• Progress emission is race-clean under concurrent MCP dispatch (go test -race)
• Tests pass with adequate coverage (per AGENTS.md testing-first discipline)
• Documentation updated if needed

Out of Scope

• Progress bars / human-mode rendering (agents consume structured events; human formatting is separate)
• Cancellation semantics beyond the existing context cancellation
• Persisting or replaying progress history

Technical Notes

The ProgressEvent shape lives in the CONTRACT packages so thin consumers pin
it without the telemetry stack. Additive and additive-tolerant (Constitution
XI) — new event type + optional MCP notifications, no change to existing stdout
payloads or the error envelope. Stream separation is non-negotiable: progress
goes to stderr only. MCP behavior is host-cooperative — fires only when the
host advertises the capability and the session is interactive, else falls back
silently. Routes through the Spec Kit workflow; the event schema and MCP mapping
are the core research questions.

Testing Strategy

• Unit tests for the ProgressEvent NDJSON emitter (field shapes, stderr-only, locked-writer non-interleaving)
• Golden-file test for the ProgressEvent JSON shape
• MCP integration test: with a progressToken present, events map to notifications; without one, they don't (fall-back path)
• Race test under concurrent dispatch

Related

• Related issues: feat(mcp-server): enrich MCP tool descriptions with pkg.go.dev module metadata #67 (MCP tool metadata — same adapter surface), Envelope runtime trust signals (side-effects/idempotency-replay/handoff) #30 (envelope trust signals — distinct, but both enrich agent-facing signaling)
• Builds on: spec 011 MCP server runtime (internal/mcpserver)
• Provenance: kimi-features.md feature F13, 2026-07-17 eight-agent evaluation