feat(realtime): stream the LLM / TTS / transcription pipeline stages

[localai-bot]

Summary

Adds opt-in streaming to each stage of the realtime pipeline (LLM tokens, TTS audio, transcription text), plus a per-pipeline disable-thinking toggle. By default every stage runs to completion before the next begins; these flags overlap generation, synthesis and playback to lower a turn's time-to-first-audio. Everything is off by default, so existing pipelines are unaffected.

name: gpt-realtime
pipeline:
  vad: silero-vad-ggml
  transcription: whisper-large-turbo
  llm: qwen3-4b
  tts: tts-1
  streaming:
    llm: true            # stream LLM tokens as transcript deltas + sentence-piped TTS
    tts: true            # emit response.output_audio.delta per synthesized chunk
    transcription: true  # stream conversation.item.input_audio_transcription.delta
  disable_thinking: true # maps to enable_thinking=false for the realtime LLM

What changed

• Config (core/config): nested pipeline.streaming.{llm,tts,transcription} block + pipeline.disable_thinking, with StreamLLM()/StreamTTS()/StreamTranscription()/ThinkingDisabled() helpers. Pointer-bools so unset preserves the unary path.
• Model interface: TTSStream / TranscribeStream on the realtime Model, delegating to the existing backend.ModelTTSStream / ModelTranscriptionStream. ttsStream adapts the backend's WAV-framed stream into raw PCM + sample rate.
• TTS (emitSpeech): response audio flows through emitSpeech; with streaming.tts it sends one response.output_audio.delta per backend chunk, else one delta for the whole utterance.
• Transcription (emitTranscription): with streaming.transcription, streams …transcription.delta events; else a single …completed (backends that don't stream emit one terminal event).
• LLM (streamSegmenter + speechStreamer + streamLLMResponse): a real token callback strips reasoning via the streaming ReasoningExtractor, emits transcript text deltas, and sentence-pipes content into emitSpeech so each sentence is synthesized as soon as it's ready. Used only for plain-content turns (no tools) — tool turns keep the buffered path, since partial tool-call output can't be safely spoken mid-stream.
• disable_thinking (applyPipelineThinking): forces ReasoningConfig.DisableReasoning on the per-session LLM config copy (no leak to other model users).
• Docs: docs/content/features/openai-realtime.md.

Tests

TDD throughout — new unit tests for the config flags, the segmenter, emitSpeech (streaming vs unary), emitTranscription, speechStreamer, and applyPipelineThinking, with fakeModel/fakeTransport doubles. core/config and core/http/endpoints/openai suites pass; the unchanged buffered path is regression-guarded.

Validation note

Verified by deploying this branch onto a real distributed cluster: the controller drives the realtime pipeline end-to-end (WebSocket session, VAD, STT transcribed real speech correctly). The streaming.llm/streaming.tts delta paths are covered by the unit tests; a full live streaming turn additionally needs a worker that can load the chosen LLM/TTS.

[richiejp]

Is this supposed to take advantage of real streaming models or as like a fallback when transcription and TTS don't actually support streaming? Looking at the code it seems to be the later, in which case it should probably be called "streaming emulation" or something like that

[richiejp]

There are tool call deltas IIRC, so it should be possible to stream when tool calls are pesent.

[localai-bot]

Done (2615934). Streaming now handles tool turns when the LLM uses its tokenizer template — there the C++ autoparser clears reply.Message and delivers content + tool_calls via ChatDeltas, so the spoken transcript only ever carries content and the tool calls are parsed from the final deltas. Grammar-based function calling keeps the buffered path, since there the call is emitted as JSON in the token stream and would leak into the transcript.

[richiejp]

What about languages that don't have sentences or word boundaries even? I'm not sure this should even exist for real streaming TTS and for non-streaming TTS I think it is better to buffer an entire message and get the LLM to produce short messages.

[localai-bot]

Agreed — removed the segmenter entirely (1010f20). As you said, for non-streaming TTS it's better to buffer the whole message (and prompt the model for short replies), and the .!?/newline splitting did nothing for languages without those boundaries anyway. Now the full reply is buffered and synthesized once: streamed as audio chunks via TTSStream where the backend supports it, a single unary delta otherwise.

[localai-bot]

Thanks for the review! Addressed both points:

Streaming vs. emulation — good call, the per-sentence path was emulation and it's gone. streaming.llm streams the transcript token-by-token (real, via PredictStream), and TTS is real streaming where the backend supports TTSStream (whole message buffered, synthesized once, chunked audio) and a clean single-shot synthesis otherwise. No segmentation, no language assumptions. (1010f20)

Tool-call streaming — implemented for tokenizer-template models, where the autoparser separates content from tool calls so the transcript can't leak tool-call tokens; grammar-based function calling still buffers. (2615934)

Also fixed the gosec G304 the bot flagged (backend-controlled TTS temp path, normalized with filepath.Clean), and a related find along the way: disable_thinking was disabling reasoning extraction as well as setting enable_thinking=false, so leaked <think> could reach the client — spoken output now always strips reasoning regardless. (e81ff0c)

[richiejp]

This creates two layers of thinking config and I wonder if it is not better to be able to fork model configs so users can quickly create different profiles for a model and change any of the settings? Or have a generic override mechanism for realtime pipelines. So we can override any setting for any of the contained models. Or introduce model overlays that take their settings from base model and merge overrides into it?

In the last case you could have one base model then N overlays for that model which inherit the base settings.

@mudler

[localai-bot]

Agreed — the per-pipeline override fields don't scale, and model overlays (one base model + N overlays that inherit and merge overrides) are the right general mechanism. They'd subsume both pipeline.disable_thinking here and pipeline.reasoning_effort (which already merged in #10184), and they're useful well beyond realtime.

I've opened #10185 to track the overlay mechanism. Proposing to keep the targeted fields for now (one's already in master) and deprecate them in favor of overlays once that lands, so this PR isn't blocked on the larger design. Sound reasonable?