feat(swift): VoiceProcessedAudioIO — capture+playback on one VP engine

Playback renders through the same voice-processed engine as capture, so
the assistant audio is guaranteed to be in the AEC reference path
(review feedback: the split-pair reference is device-level and
route-dependent). Recommended wiring: one instance as both input and
output. Internal lock serializes enqueue/flush/stop against runtime
actor reentrancy.

Refs #559

Co-authored-by: Claude <claude@anthropic.com>

Author: cornelcroi

docs/astro.config.mjs

@@ -288,6 +288,7 @@ export default defineConfig({
 										label: 'Audio',
 										items: [
 											{ label: 'Overview', slug: 'swift/audio/overview' },
+											{ label: 'VoiceProcessedAudioIO', slug: 'swift/audio/built-in/voice-processed-audio-io' },
 											{ label: 'MicCapture', slug: 'swift/audio/built-in/mic-capture' },
 											{ label: 'AudioPlayback', slug: 'swift/audio/built-in/audio-playback' },
 											{ label: 'Custom audio', slug: 'swift/audio/custom' },

docs/src/content/docs/swift/audio/built-in/audio-playback.md

@@ -5,6 +5,10 @@ description: AVAudioEngine-backed AudioOutput that converts PCM16 frames to floa
 
 `AudioPlayback` is part of the `AgentSquadAudio` product. It accepts PCM16 @ 24 kHz frames, converts them to float32, and schedules them on an `AVAudioPlayerNode` for continuous playback. `flush()` provides an instant barge-in cut by discarding all buffered audio.
 
+:::tip
+For voice sessions, prefer [`VoiceProcessedAudioIO`](/agent-squad/swift/audio/built-in/voice-processed-audio-io/) — playback renders through the same voice-processed engine as capture, guaranteeing echo cancellation of the assistant's audio.
+:::
+
 ```swift
 import AgentSquadAudio
 ```

docs/src/content/docs/swift/audio/built-in/mic-capture.md

@@ -7,6 +7,10 @@ description: AVAudioEngine-backed AudioInput that taps the microphone, converts
 
 By default it captures through Apple's **Voice-Processing I/O** unit: echo cancellation that uses the speaker signal as a hardware reference to subtract the assistant's own voice from the mic, plus noise suppression and automatic gain control.
 
+:::tip
+For voice sessions, prefer [`VoiceProcessedAudioIO`](/agent-squad/swift/audio/built-in/voice-processed-audio-io/) — capture and playback on **one** engine, which guarantees the assistant's audio is in the AEC reference path. With the split `MicCapture`/`AudioPlayback` pair the reference is device-level and route-dependent.
+:::
+
 ```swift
 import AgentSquadAudio
 ```

docs/src/content/docs/swift/audio/built-in/voice-processed-audio-io.md

@@ -0,0 +1,70 @@
+---
+title: VoiceProcessedAudioIO
+description: Capture and playback on one voice-processed AVAudioEngine — echo cancellation with the assistant's audio guaranteed in the reference path. The recommended wiring for voice sessions.
+---
+
+`VoiceProcessedAudioIO` is part of the `AgentSquadAudio` product. It runs microphone capture **and** assistant playback on a **single** `AVAudioEngine` with Apple's Voice-Processing I/O unit enabled. Because the assistant's audio renders through the same engine's voice-processed output, it is by construction in the echo canceller's reference path — the configuration the VP unit is designed around.
+
+Conforms to **both** `AudioInput` and `AudioOutput`: pass **one instance** as both `input:` and `output:`.
+
+```swift
+import AgentSquadAudio
+
+let io = VoiceProcessedAudioIO()
+let runtime = RealtimeRuntime(session: assistant, input: io, output: io)
+try await runtime.start()
+```
+
+Prefer this over the separate [`MicCapture`](/agent-squad/swift/audio/built-in/mic-capture/) + [`AudioPlayback`](/agent-squad/swift/audio/built-in/audio-playback/) pair for voice sessions — with two engines the echo reference is taken at the device level, which is route-dependent.
+
+---
+
+## Init
+
+```swift
+public init(
+    sampleRate: Double = 24_000,
+    maxBufferedFrames: Int = 16,
+    voiceProcessing: VoiceProcessing = .default,
+    sessionPolicy: AudioSessionPolicy = .managed,
+    configureEngine: (@Sendable (AVAudioEngine) throws -> Void)? = nil
+)
+```
+
+| Parameter | Default | Notes |
+|---|---|---|
+| `sampleRate` | `24_000` | Both capture and playback rate. Must match what the realtime session expects (OpenAI Realtime: PCM is always 24 kHz). |
+| `maxBufferedFrames` | `16` | Capacity of the capture `AsyncStream`; oldest frames dropped under back-pressure. |
+| `voiceProcessing` | `.default` | AEC + noise suppression + AGC tuning. **Non-optional** — raw capture defeats this class's purpose; use `MicCapture(voiceProcessing: nil)` for that. |
+| `sessionPolicy` | `.managed` | Who configures the `AVAudioSession` — see [AudioSessionPolicy](/agent-squad/swift/audio/built-in/mic-capture/#audiosessionpolicy-ios-only). |
+| `configureEngine` | `nil` | Escape hatch: runs with the raw `AVAudioEngine` after voice processing is enabled and the player is wired, before the tap is installed. |
+
+---
+
+## Public surface
+
+```swift
+public let frames: AsyncStream<Data>    // AudioInput — captured PCM16 LE mono frames
+
+public func start() async throws        // both roles; idempotent
+public func enqueue(_ pcm16: Data) async // AudioOutput — schedule one frame
+public func flush() async                // AudioOutput — instant barge-in cut
+public func stop()  async                // both roles; idempotent
+```
+
+`start()` and `stop()` are **idempotent** — `RealtimeRuntime` calls each twice on the same instance (once through the `AudioOutput` role, once through `AudioInput`), and the second call is a no-op. `enqueue`/`flush` before `start()` or after `stop()` are safe no-ops. One instance serves **one session**: `stop()` finishes the `frames` stream for good — create a new instance to start again.
+
+Failure modes are the shared [`MicCaptureError`](/agent-squad/swift/audio/built-in/mic-capture/#miccaptureerror) cases: `permissionDenied`, `converterUnavailable`, `voiceProcessingUnavailable`.
+
+:::caution
+The **simulator performs no echo cancellation** — validate AEC on a real device. Voice-processed output sounds "call-like" and slightly quieter; counter with `duckingLevel: .min`.
+:::
+
+---
+
+## Related pages
+
+- [Audio overview](/agent-squad/swift/audio/overview/) — the `AudioInput`/`AudioOutput` protocols
+- [MicCapture](/agent-squad/swift/audio/built-in/mic-capture/) — capture-only built-in (split-pair wiring, `VoiceProcessing`, `AudioSessionPolicy` docs)
+- [AudioPlayback](/agent-squad/swift/audio/built-in/audio-playback/) — playback-only built-in
+- [Voice overview](/agent-squad/swift/voice/overview/) — the `RealtimeRuntime` that consumes this class

docs/src/content/docs/swift/audio/overview.md

@@ -3,11 +3,11 @@ title: Audio overview
 description: The AgentSquadAudio product and the AudioInput/AudioOutput protocols that connect microphone capture and speaker playback to the voice runtime.
 ---
 
-`AgentSquadAudio` is a separate Swift package product that ships two AVFoundation-backed implementations — `MicCapture` and `AudioPlayback` — built on top of two protocols declared in the core `AgentSquad` module.
+`AgentSquadAudio` is a separate Swift package product that ships three AVFoundation-backed implementations — `VoiceProcessedAudioIO` (the recommended one), `MicCapture`, and `AudioPlayback` — built on top of two protocols declared in the core `AgentSquad` module.
 
 ```swift
 import AgentSquad        // AudioInput, AudioOutput protocols
-import AgentSquadAudio   // MicCapture, AudioPlayback
+import AgentSquadAudio   // VoiceProcessedAudioIO, MicCapture, AudioPlayback
 ```
 
 ---
@@ -48,14 +48,17 @@ public protocol AudioOutput: Sendable {
 
 ## How they feed the voice runtime
 
-[`RealtimeRuntime`](/agent-squad/swift/voice/overview/) accepts an `AudioInput` and an `AudioOutput` at construction time:
+[`RealtimeRuntime`](/agent-squad/swift/voice/overview/) accepts an `AudioInput` and an `AudioOutput` at construction time. The recommended wiring is one `VoiceProcessedAudioIO` instance in both roles — capture and playback share a single voice-processed engine, so the assistant's audio is guaranteed to be in the echo canceller's reference path:
 
 ```swift
-let runtime = RealtimeRuntime(
-    input:  MicCapture(),
-    output: AudioPlayback(),
-    // ... other config
-)
+let io = VoiceProcessedAudioIO()
+let runtime = RealtimeRuntime(session: assistant, input: io, output: io)
+```
+
+The split pair works too (the AEC reference is then device-level, which is route-dependent):
+
+```swift
+let runtime = RealtimeRuntime(session: assistant, input: MicCapture(), output: AudioPlayback())
 ```
 
 The runtime drives `start`, `stop`, `enqueue`, and `flush` from its single event pump, so implementations are never called concurrently by the runtime itself.
@@ -70,7 +73,8 @@ The wire format for both protocols is **PCM16 little-endian mono at 24 kHz**. Th
 
 | Type | Protocol | Description |
 |---|---|---|
-| [`MicCapture`](/agent-squad/swift/audio/built-in/mic-capture/) | `AudioInput` | AVAudioEngine tap → PCM16 @ 24 kHz, with iOS permission gating |
+| [`VoiceProcessedAudioIO`](/agent-squad/swift/audio/built-in/voice-processed-audio-io/) | `AudioInput` + `AudioOutput` | Capture and playback on **one** voice-processed engine — guaranteed AEC reference path; pass one instance as both input and output |
+| [`MicCapture`](/agent-squad/swift/audio/built-in/mic-capture/) | `AudioInput` | AVAudioEngine tap → PCM16 @ 24 kHz, voice-processed by default, with iOS permission gating |
 | [`AudioPlayback`](/agent-squad/swift/audio/built-in/audio-playback/) | `AudioOutput` | AVAudioEngine + AVAudioPlayerNode, with barge-in flush |
 
 ---

swift/README.md

@@ -214,7 +214,7 @@ the integrations you need (each isolates its own dependencies):
 |---|---|---|---|
 | **`AgentSquad`** (core) | `import AgentSquad` | **nothing external** | protocols, agents, orchestrator, native tools (`ToolKit`, `Tool.local`/`.http`, `HTTPToolGroup`, `AggregateToolProvider`), `FileChatStorage`, `DeviceChatStorage` _(iOS 17+)_, `InMemoryChatStorage`, `OSLogTracer` |
 | **`AgentSquadMCP`** | `import AgentSquadMCP` | the official [MCP Swift SDK](https://github.com/modelcontextprotocol/swift-sdk) | `MCPServer` (alias of `MCPToolProvider`) — connect any MCP server with `MCPServer(url:)`, with MCP Apps UI support |
-| **`AgentSquadAudio`** | `import AgentSquadAudio` | AVFoundation (Apple platforms only) | `MicCapture` (echo-cancelled by default) + `AudioPlayback` for the realtime voice runtime — requires `NSMicrophoneUsageDescription` in your Info.plist |
+| **`AgentSquadAudio`** | `import AgentSquadAudio` | AVFoundation (Apple platforms only) | `VoiceProcessedAudioIO` (echo-cancelled capture + playback on one engine — the recommended wiring), `MicCapture`, `AudioPlayback` for the realtime voice runtime — requires `NSMicrophoneUsageDescription` in your Info.plist |
 
 So an app that doesn't use MCP never downloads the MCP SDK. Future optional integrations
 (e.g. `AgentSquadLangfuse` for trace export) follow the same pattern — the core never grows a
@@ -229,7 +229,7 @@ dependency. Add a product to your target's `dependencies` to use it:
 
 ## Voice audio — echo cancellation and full control
 
-By default `MicCapture` captures through Apple's **Voice-Processing I/O** unit — the native
+The audio layer captures through Apple's **Voice-Processing I/O** unit by default — the native
 equivalent of what ChatGPT's voice mode gets via WebRTC. The signal sent to the speaker is used
 as a hardware reference to subtract the assistant's own voice from the mic, plus noise
 suppression and automatic gain control. Without it, the assistant hears itself through the
@@ -238,53 +238,58 @@ speaker and interrupts its own answers.
 The audio layer is configurable in four independent levels — each level keeps everything the
 previous levels give you:
 
-**Level 0 — defaults.** Echo-cancelled capture, nothing to configure:
+**Level 0 — defaults.** `VoiceProcessedAudioIO` runs capture **and** playback on one
+`AVAudioEngine`, so the assistant's audio renders through the voice-processed output and is by
+construction in the echo canceller's reference path. Pass one instance as both `input:` and
+`output:`:
 
 ```swift
 import AgentSquadAudio
 
-let runtime = RealtimeRuntime(session: assistant, input: MicCapture(), output: AudioPlayback())
+let io = VoiceProcessedAudioIO()
+let runtime = RealtimeRuntime(session: assistant, input: io, output: io)
 try await runtime.start()
 ```
 
+The separate `MicCapture` + `AudioPlayback` pair still works (both echo-cancelled on the capture
+side by default) — but with two engines the echo reference is taken at the device level, which is
+route-dependent. Prefer `VoiceProcessedAudioIO` for voice sessions.
+
 **Level 1 — tune voice processing** (or turn it off):
 
 ```swift
 // Keep AEC but disable gain control and minimize how much the system ducks playback volume:
-let mic = MicCapture(voiceProcessing: .init(automaticGainControl: false, duckingLevel: .min))
+let io = VoiceProcessedAudioIO(voiceProcessing: .init(automaticGainControl: false, duckingLevel: .min))
 
-// Raw capture — no AEC at all (the previous default):
+// Raw capture — no AEC at all (the previous default; split pair only):
 let rawMic = MicCapture(voiceProcessing: nil)
 ```
 
 If the Voice-Processing unit can't be enabled, `start()` throws
 `MicCaptureError.voiceProcessingUnavailable(_:)` rather than silently degrading — catch it and
 retry with `voiceProcessing: nil` if raw capture is an acceptable fallback for your app.
 
-**Level 2 — own the `AVAudioSession`, or reach the raw engine.** Both `MicCapture` and
-`AudioPlayback` take an `AudioSessionPolicy`; give them the same one so they can't fight:
+**Level 2 — own the `AVAudioSession`, or reach the raw engine.** All three audio classes take
+an `AudioSessionPolicy` (if you use the split pair, give both the same one so they can't fight):
 
 ```swift
 // Your app already manages its AVAudioSession (music, video, CallKit…) — AgentSquad won't touch it.
 // You must configure AND activate the session yourself before runtime.start().
-let mic = MicCapture(sessionPolicy: .external)
-let out = AudioPlayback(sessionPolicy: .external)
+let io = VoiceProcessedAudioIO(sessionPolicy: .external)
 
 // Or let AgentSquad drive the timing but with YOUR configuration (iOS):
-let policy = AudioSessionPolicy.custom { session in
+let io2 = VoiceProcessedAudioIO(sessionPolicy: .custom { session in
     try session.setCategory(.playAndRecord, mode: .voiceChat, options: [.allowBluetoothHFP])   // no speaker override
     try session.setActive(true)
-}
-let mic2 = MicCapture(sessionPolicy: policy)
-let out2 = AudioPlayback(sessionPolicy: policy)
+})
 ```
 
 The `configureEngine` hook hands you the underlying `AVAudioEngine` at the right lifecycle
 moment (after voice processing is enabled, before the tap is installed), so any AVFoundation
 API stays reachable without forking the class:
 
 ```swift
-let mic = MicCapture(configureEngine: { engine in
+let io = VoiceProcessedAudioIO(configureEngine: { engine in
     // e.g. inspect engine.inputNode, insert effect nodes, adopt future AVFoundation APIs…
 })
 ```

swift/SKILL.md

@@ -33,7 +33,7 @@ Every component is a `Sendable` protocol with one built-in implementation — sw
 |---|---|---|
 | `AgentSquad` | nothing external | protocols, `Agent`, `GroundedAgent`, `Orchestrator`, `LLMClassifier`, `ChatCompletionsClient`, `FileChatStorage`, `DeviceChatStorage`, `InMemoryChatStorage`, `OSLogTracer`, OTLP export |
 | `AgentSquadMCP` | MCP Swift SDK | `MCPServer` (= `MCPToolProvider`), `SDKMCPClient` |
-| `AgentSquadAudio` | AVFoundation | `MicCapture` (voice-processed/AEC by default), `AudioPlayback`, `VoiceProcessing`, `AudioSessionPolicy` (needs `NSMicrophoneUsageDescription`) |
+| `AgentSquadAudio` | AVFoundation | `VoiceProcessedAudioIO` (capture+playback, one engine, AEC — the recommended wiring), `MicCapture` (voice-processed/AEC by default), `AudioPlayback`, `VoiceProcessing`, `AudioSessionPolicy` (needs `NSMicrophoneUsageDescription`) |
 
 SwiftPM: `.package(url: "https://github.com/2FastLabs/agent-squad", branch: "main")`.
 
@@ -76,10 +76,13 @@ stream. `.final` is what the orchestrator persists. Inputs/messages are value ty
 - **Voice**: two `VoiceAssistant`s over a WebSocket — `OpenAIVoiceAssistant` (single LLM, speaks
   directly) and `OpenAIGroundedVoiceAssistant` (grounded Brain → Presenter). Both are self-sufficient
   (own `tracer`/`store`/`userId`/`sessionId`; with a `store`, completed turns persist and prior
-  history seeds on `start()`), wired to the mic/speaker by `RealtimeRuntime` with `MicCapture`/`AudioPlayback`.
-  `MicCapture` captures through Apple's Voice-Processing I/O unit by default (echo cancellation,
-  noise suppression, AGC — tune via `voiceProcessing:`, or pass `nil` for raw capture); both audio
-  classes take an `AudioSessionPolicy` (`.managed` / `.custom` / `.external` for apps that own the
+  history seeds on `start()`), wired to the mic/speaker by `RealtimeRuntime`. Preferred audio
+  wiring: ONE `VoiceProcessedAudioIO` instance passed as both `input:` and `output:` — capture and
+  playback share one voice-processed `AVAudioEngine`, so the assistant's audio is guaranteed to be
+  in the echo canceller's reference path. The split `MicCapture`/`AudioPlayback` pair also works
+  (capture is voice-processed by default; the AEC reference is then device-level/route-dependent;
+  `MicCapture(voiceProcessing: nil)` = raw capture). All three audio classes take an
+  `AudioSessionPolicy` (`.managed` / `.custom` / `.external` for apps that own the
   `AVAudioSession`) and a `configureEngine` hook exposing the raw `AVAudioEngine`.
   Session tuning on both: `transcriptionModel` (the user's STT only), `turnDetection`
   (`.semanticVAD(eagerness:)` / `.serverVAD(threshold:…)` / `.disabled`), and `sessionOverrides`
@@ -121,12 +124,14 @@ signatures live in `Sources/AgentSquad/`.
   pattern-scrub PII — supply a custom `Redactor` for that.
 - **Realtime** is a peer runtime, not an agent; its `events` stream is non-throwing; needs
   `NSMicrophoneUsageDescription`; always `stop()`.
-- **Voice processing (AEC)**: on by default in `MicCapture`; if it can't be enabled `start()`
-  throws `.voiceProcessingUnavailable` (degrade deliberately with `voiceProcessing: nil`). The
+- **Voice processing (AEC)**: on by default; if it can't be enabled `start()` throws
+  `.voiceProcessingUnavailable` (degrade deliberately with `MicCapture(voiceProcessing: nil)`).
+  For guaranteed echo cancellation use `VoiceProcessedAudioIO` and pass the **same instance** as
+  input and output (its `start()`/`stop()` are idempotent — the runtime calls each twice). The
   simulator does **no** AEC — validate on a device. VP quiets the speaker (counter with
-  `duckingLevel: .min`); never enable VP on the playback engine. With `sessionPolicy: .external`
-  the app must configure **and activate** its `AVAudioSession` before `start()`, and should pass
-  the same policy to both `MicCapture` and `AudioPlayback`.
+  `duckingLevel: .min`); never enable VP on a playback-only engine. With
+  `sessionPolicy: .external` the app must configure **and activate** its `AVAudioSession` before
+  `start()`, and should use the same policy everywhere.
 - **`ContentPart` Codable** keys off case + label names — renaming breaks stored history.
 
 ## Go deeper

swift/Sources/AgentSquadAudio/AudioConfiguration.swift

@@ -23,8 +23,21 @@ public struct VoiceProcessing: Sendable, Equatable {
     public static let `default` = VoiceProcessing()
 }
 
-/// Who configures the `AVAudioSession`. `MicCapture` and `AudioPlayback` take the same policy so
-/// the two can't fight over the session. No effect on macOS (no `AVAudioSession` there).
+extension VoiceProcessing.DuckingLevel {
+    @available(iOS 17.0, macOS 14.0, *)
+    var avLevel: AVAudioVoiceProcessingOtherAudioDuckingConfiguration.Level {
+        switch self {
+        case .default: .default
+        case .min: .min
+        case .mid: .mid
+        case .max: .max
+        }
+    }
+}
+
+/// Who configures the `AVAudioSession`. All audio classes take a policy; if you use the split
+/// `MicCapture`/`AudioPlayback` pair, give both the same one so they can't fight over the
+/// session. No effect on macOS (no `AVAudioSession` there).
 public enum AudioSessionPolicy: Sendable {
     /// AgentSquad configures it: `.playAndRecord`, `.voiceChat`, speaker output, Bluetooth HFP.
     case managed