fix(streaming): report OpenAI stream errors in-band instead of raising

[zhangw]

What

stream_wrapper() in chat_completions() re-raised the exception after StreamingResponse had already sent its headers, so Starlette cannot turn it into an error response and aborts the connection instead:

ERROR | kiro.streaming_core:stream_with_first_token_retry - [FirstTokenTimeout] All 3 attempts exhausted - model never responded within 15.0s per attempt
ERROR | kiro.routes_openai:stream_wrapper - HTTP 500 - POST /v1/chat/completions (streaming) - [HTTPException] 504: Model did not respond within 15.0s after 3 attempts.
ERROR | Exception in ASGI application
...
  File "starlette/middleware/base.py", line 198, in __call__
    raise app_exc
  File "starlette/_exception_handler.py", line 56, in wrapped_app
    raise RuntimeError("Caught handled exception, but response already started.") from exc

RuntimeError: Caught handled exception, but response already started.

Two consequences:

1. The client gets data: [DONE] with no content and no reason - it looks like the gateway silently returned an empty answer. The 504 detail never reaches it.
2. Every occurrence prints a full ASGI traceback for what is an upstream condition, not a gateway bug.

/v1/messages already handles this correctly (event: error, then a graceful end of stream), so only the OpenAI path was affected.

Why it happens in practice

Any streaming failure after the first byte of the response reaches this path. The easiest way to hit it is the first-token retry loop, because not every model streams reasoning deltas: the non-Claude models exposed by Kiro send nothing at all while they think, so the first byte only arrives together with the answer.

Measured with one heavy request (~270 KB payload, 20 tools, non-Claude model), first-token timeout temporarily raised to 180 s to observe the real latency:

attempt	upstream headers	first body byte
1	8.5 s	38.8 s
2	12.4 s	32.0 s
3	9.6 s	35.2 s

A Claude model on the same gateway starts emitting reasoning_content after ~2-4 s, which is why the default FIRST_TOKEN_TIMEOUT=15 has been fine so far. With a model that stays silent while reasoning, all three attempts time out and the resulting 504 lands on the broken error path above.

Whether the default timeout should change is a separate discussion - this PR only fixes the error reporting.

Related: #129 (mid-stream RemoteProtocolError) hits the same code path on /v1/chat/completions, and improvement 2 requested there - telling the client it was a connection drop - now works for the OpenAI API too.

Change

• New helper build_sse_error_chunk(exc) renders an exception as an OpenAI-format error chunk. HTTPException.status_code / .detail are preserved, anything else is reported as 500.
• Both stream_wrapper() variants (account-system branch and legacy branch) now yield the error chunk followed by [DONE] and no longer re-raise.

The client now receives the reason:

data: {"error": {"message": "Model did not respond within 15.0s after 3 attempts. Please try again.", "type": "kiro_api_error", "code": 504}}

data: [DONE]

Server-side logging and debug-log flushing are unchanged - streaming_error is still set, so the existing HTTP 500 - POST /v1/chat/completions (streaming) access log line and flush_on_error() still fire.

Tests

tests/unit/test_routes_openai.py

• TestBuildSseErrorChunk - HTTPException status/detail preserved; generic exception reported as 500; exception with an empty message still produces valid JSON.
• TestStreamingErrorAfterResponseStart - first-token timeout produces code: 504 + [DONE]; mid-stream failure keeps the chunks already sent and appends code: 500; happy path unaffected (no error chunk, exactly one [DONE]).

tests/unit/test_routes_anthropic.py

• TestStreamingErrorAfterResponseStart - parity test asserting /v1/messages reports the failure as event: error without raising. It passes on main as well; it is there to keep the two APIs from drifting apart again.

Verification: 5 of the 7 new tests fail on main (the 3 helper tests plus the 2 OpenAI streaming-failure tests, which abort with the RuntimeError above) and all 7 pass with this change. Full unit suite: same pass/fail set before and after this change, no new failures.

[cla-bot]

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: vincent.
This is most likely caused by a git client misconfiguration; please make sure to:

1. check if your git client is configured with an email to sign commits git config --list | grep email
2. If not, set it up using git config --global user.email email@example.com
3. Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

[cla-bot]

Thanks for the PR! 🎉

Before merge, we need a one-time CLA confirmation.
It confirms that you have the right to contribute this code and allow the project to use it.

Full CLA text:
https://github.com/jwadow/kiro-gateway/blob/main/CLA.md

Please reply once with:

I have read the CLA and I accept its terms

You need to write once, all further messages from me can be ignored.

[zhangw]

I have read the CLA and I accept its terms

[ankitcharolia]

@zhangw could you give a try to this gateway: https://github.com/ankitcharolia/kiro-gateway

It works quite well with All AI harness and actively being developed. The most important thing is that it is ACP compliant