Skip to content

Talk: add gateway TTS fallback when ElevenLabs is not configured #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
boydyngo wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Talk: add gateway TTS fallback when ElevenLabs is not configured #1

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 44 additions & 3 deletions apps/macos/Sources/OpenClaw/TalkModeRuntime.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -442,7 +442,7 @@ actor TalkModeRuntime {
if let apiKey = input.apiKey, !apiKey.isEmpty, let voiceId = input.voiceId {
try await self.playElevenLabs(input: input, apiKey: apiKey, voiceId: voiceId)
} else {
try await self.playSystemVoice(input: input)
try await self.playGatewayTTS(input: input)
}
Comment on lines 442 to 446

Copilot AI Apr 15, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This fallback now routes through gateway TTS, but some of the surrounding logging still says it is “falling back to system voice” when the ElevenLabs API key/voiceId is missing. Updating those warnings to mention gateway TTS (with system voice as the final fallback) will make diagnostics match runtime behavior.

Copilot uses AI. Check for mistakes.
} catch {
self.ttsLogger
Expand Down Expand Up @@ -643,6 +643,47 @@ actor TalkModeRuntime {
return await self.playMP3(stream: stream)
}

private func playGatewayTTS(input: TalkPlaybackInput) async throws {
self.ttsLogger.info("talk gateway TTS start chars=\(input.cleanedText.count, privacy: .public)")

let params: [String: AnyCodable] = ["text": AnyCodable(input.cleanedText)]
let data = try await GatewayConnection.shared.request(
method: "talk.speak",
params: params,
timeoutMs: Double(input.synthTimeoutSeconds * 1000))

struct TalkSpeakResponse: Decodable {
let audioBase64: String?
let provider: String?
let mimeType: String?
}
let response = try JSONDecoder().decode(TalkSpeakResponse.self, from: data)
guard let base64 = response.audioBase64, !base64.isEmpty,
let audioData = Data(base64Encoded: base64)
else {
throw NSError(
domain: "TalkGatewayTTS", code: 1,
userInfo: [NSLocalizedDescriptionKey: "gateway returned no audio"])
}

let provider = response.provider ?? "edge"
self.ttsLogger.info(
"talk gateway TTS received \(audioData.count, privacy: .public) bytes " +
"provider=\(provider, privacy: .public)")

guard self.isCurrent(input.generation) else { return }

if self.interruptOnSpeech {
guard await self.prepareForPlayback(generation: input.generation) else { return }
}
await MainActor.run { TalkModeController.shared.updatePhase(.speaking) }
self.phase = .speaking

let result = await TalkAudioPlayer.shared.play(data: audioData)
self.ttsLogger.info(
"talk gateway TTS done finished=\(result.finished, privacy: .public)")

Copilot AI Apr 15, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

playGatewayTTS ignores the playback outcome: if TalkAudioPlayer fails to decode/play the returned audio, it returns finished=false but this method still returns successfully, so the caller will not fall back to system voice. Consider treating finished == false as an error (throw) so the existing catch path can trigger the last-resort fallback, and also ensure this playback path is interruptible (currently stopSpeaking only stops the streaming players and the system synthesizer, not TalkAudioPlayer).

Suggested change
"talk gateway TTS done finished=\(result.finished, privacy: .public)")
"talk gateway TTS done finished=\(result.finished, privacy: .public)")
guard result.finished else {
guard self.isCurrent(input.generation) else { return }
throw NSError(
domain: "TalkGatewayTTS", code: 2,
userInfo: [NSLocalizedDescriptionKey: "gateway audio playback did not finish"])
}

Copilot uses AI. Check for mistakes.
}

private func playSystemVoice(input: TalkPlaybackInput) async throws {
self.ttsLogger.info("talk system voice start chars=\(input.cleanedText.count, privacy: .public)")
if self.interruptOnSpeech {
Expand Down Expand Up @@ -800,8 +841,8 @@ extension TalkModeRuntime {

do {
let snap: ConfigSnapshot = try await GatewayConnection.shared.requestDecoded(
method: .talkConfig,
params: ["includeSecrets": AnyCodable(true)],
method: .configGet,
params: nil,
timeoutMs: 8000)
Comment on lines 843 to 846

Copilot AI Apr 15, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Switching from talk.config (with includeSecrets) to config.get may change authorization/shape/secret-redaction semantics. In particular, talk.config is already used elsewhere to retrieve talk.apiKey when includeSecrets is true (e.g. apps/ios/Sources/Voice/TalkModeManager.swift:1674), but config.get has no includeSecrets parameter in the protocol model. If config.get returns redacted secrets or is restricted in some modes, this can break ElevenLabs configuration loading. Consider keeping talk.config for Talk mode config (and only fetching additional UI keys via config.get if needed).

Copilot uses AI. Check for mistakes.
let talk = snap.config?["talk"]?.dictionaryValue
let ui = snap.config?["ui"]?.dictionaryValue
Expand Down