Skip to content

Gateway cannot cancel or time-bound an in-flight agent turn — a runaway turn blocks the whole session queue #3467

Description

@MervinPraison

Summary

The gateway has no way to stop an agent turn that is already running, and no per-turn timeout. Once a turn begins it runs to completion on the session's serial queue; every later message from that user queues behind it. A single slow tool call, a doom-looping agent, or a stuck provider request therefore wedges the conversation indefinitely, and neither the end user (over a channel) nor a WS/operator client can interrupt it.

The protocol contract for this already exists in core — EventType.MESSAGE_ABORT is defined and InterruptController is implemented — but the gateway server never wires either of them. The gateway even advertises this absence to clients in its capability handshake.

Current behaviour

The gateway explicitly does not advertise an abort method, with a comment stating so:

# src/praisonai-bot/praisonai_bot/gateway/server.py:2741
features = {
    "methods": ["message", "leave"],  # abort not implemented
    ...
}

The core protocol already reserves the event, but nothing handles it:

# src/praisonai-agents/praisonaiagents/gateway/protocols.py:184
MESSAGE_ABORT = "message_abort"

A core interrupt primitive already exists and is unused by the gateway:

# src/praisonai-agents/praisonaiagents/agent/interrupt.py:41
class InterruptController: ...
    def request(self, reason: str = "user") -> None: ...
    def is_set(self) -> bool: ...
    def check(self) -> None: ...

The turn is awaited with no timeout and no cancellation handle:

# src/praisonai-bot/praisonai_bot/gateway/server.py:3033
@staticmethod
async def _dispatch_agent_turn(agent, content):
    for _name in ("arun", "achat"):
        _fn = getattr(agent, _name, None)
        if _fn is not None and asyncio.iscoroutinefunction(_fn):
            return await _fn(content)          # <- no timeout, no cancel scope
    loop = asyncio.get_running_loop()
    return await loop.run_in_executor(None, agent.chat, content)

And it runs inside a strictly serial per-session loop, so a stuck turn starves every queued message:

# src/praisonai-bot/praisonai_bot/gateway/server.py:3050
async def _run_session_queue(self, session, agent, client_id):
    while True:
        content = session.get_next_message()
        ...
        response = await self._dispatch_agent_turn(agent, content)   # blocks the queue

There is no /stop-style channel command and no message_abort handler in _handle_client_message.

Desired behaviour

  • A running turn can be cancelled: (a) by a WS/operator client sending a message_abort frame, and (b) by a chat user via a portable stop command (e.g. /stop).
  • Each turn runs under a configurable per-turn timeout; on expiry it is cancelled and a typed terminal outcome is returned, not left hanging.
  • Cancellation cooperatively signals the running agent (via the existing InterruptController) and cancels the asyncio task driving _dispatch_agent_turn.
  • Any partial output produced before interruption is preserved and delivered, and the terminal state is normalised through the existing run_outcome / AgentRunOutcome machinery rather than surfacing as a coarse error string.

Layer placement

  • Primary layer: wrapper (praisonaipraisonai_bot)
  • Why not core: the run loop, session queue, and WS frame router that must be interrupted live in the wrapper gateway server; core already supplies the contract (MESSAGE_ABORT) and the primitive (InterruptController).
  • Why not wrapper CLI/YAML only: this is a runtime protocol capability, not a config surface.
  • Why not tools: cancellation is a runtime lifecycle concern, not an agent-callable tool.
  • Why not plugins: it is core gateway behaviour every channel needs, not an optional lifecycle hook.
  • Secondary touch (optional): core — expose an InterruptController on the run scope the gateway drives, and ensure agent.arun/achat accept a cancellation/timeout signal.
  • 3-way surface (CLI + YAML + Python): partial — a gateway.per_turn_timeout config knob is YAML/CLI-worthy; the abort itself is a runtime protocol method + chat command.

Proposed approach

  • Extension point: gateway WS method (message_abort) + portable stop command + per-turn asyncio.wait_for/cancel scope.
  • Minimal API sketch:
# advertise it
features["methods"] = ["message", "leave", "abort"]

# handle it
elif msg_type == "abort":
    self._abort_active_turn(client_id, reason="user")

# run each turn cancellably + time-bounded
task = asyncio.ensure_future(self._dispatch_agent_turn(agent, content, interrupt=controller))
self._active_turns[session.session_id] = (task, controller)
try:
    response = await asyncio.wait_for(task, timeout=self._per_turn_timeout)
except asyncio.CancelledError:
    response = self._finalise_aborted_turn(session, controller.reason)
except asyncio.TimeoutError:
    controller.request("timeout"); task.cancel()
    response = self._finalise_aborted_turn(session, "timeout")

Resolution sketch

# Before (today): a Telegram user sends a heavy request; the agent gets stuck in a tool
# call. The user types "stop" — nothing happens. Every further message queues behind the
# wedged turn. Only killing the gateway process clears it.

# After (proposed): the user types /stop (or a WS client sends {"type":"abort"}); the
# running turn is cancelled, partial output is delivered, and the queue resumes. A
# per-turn timeout also auto-cancels a runaway turn without operator intervention.

Severity

Critical — an un-cancellable, un-time-bounded turn on a serial per-session queue is a production liveness hazard for any always-on gateway/bot; there is currently no recourse short of restarting the process.

Validation

  • src/praisonai-bot/praisonai_bot/gateway/server.py:2741 — abort explicitly not advertised.
  • src/praisonai-bot/praisonai_bot/gateway/server.py:3033 and :3050 — turn awaited with no timeout inside a serial while True queue.
  • _handle_client_message handles only hello/join/message/leave; no message_abort branch.
  • src/praisonai-agents/praisonaiagents/gateway/protocols.py:184MESSAGE_ABORT defined but unhandled.
  • src/praisonai-agents/praisonaiagents/agent/interrupt.py:41InterruptController exists, unused by the gateway.
  • Distinct from Interactive CLI: interrupt an in-flight generation or tool call (cooperative cancellation + partial output) #3461, which targets interrupting the interactive CLI; this is the gateway/channel/WS surface.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingclaudeAuto-trigger Claude analysisperformance

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions