How do you route llm or agent?

[fedorzh]

Currently, agent.run fails if the message has something like "Hello" (I use the example agent from the text to sql post).
How do you usually route your system to either use the agent or a chat-based llm to respond?
Do you think this simple case is the one where I already need to build a routing llm agent, and route to either a simple chat agent or to the sql agent? This approach seems rather complex for such a simple common use case.

[fedorzh]

I guess a follow up question is - how do you create a routing agent? I'd add my own prompt to the CodeAgent, but then, only ManagedAgent supports additional prompts. I'd replace the system prompt, but the CODE_SYSTEM_PROMPT is quite detailed and properly handles the managed agents functionality, so I will need to copy-paste parts of the system prompt.

What is the way you envisioned (or anyone succeeded) in making a routing agents? Anthropic's approach described here is quite clear, but I wonder what is the best way to use it in the library

[kinthaiofficial]

Routing LLMs or agents in smolagents (and multi-agent systems generally) is worth doing deliberately rather than just using one model for everything. A few routing strategies:

Capability-based routing
Classify the task first (is it code generation? factual QA? creative writing? math?), then route to the model best suited for that task class. A small classifier (even a rule-based one) can achieve 90%+ accuracy here. The routing overhead is sub-millisecond; the cost savings are 30-70%.

Cost-tier routing with quality fallback
Route all requests to the cheapest model first. If the output confidence (measured by self-evaluation or a critic model) falls below a threshold, escalate to a more capable model. This is "optimistic routing" — assume the cheap model is sufficient until proven otherwise.

Context-length routing
Some models have shorter context windows or degrade on long inputs. If the input exceeds 4K tokens, route to a model with a larger context window. If it's under 2K, route to the fastest/cheapest option.

Agent-type routing in multi-agent systems
Different agent types (researcher, coder, writer) have different optimal model choices. A "researcher" agent doing web lookup + summarization can use a mid-tier model. A "coder" agent generating production code might need a top-tier model. Assign models to agent roles rather than routing per-request.

Latency-based routing
For time-sensitive applications: if the primary model response time exceeds P95, route to a backup provider. Race two providers and use whoever responds first (cancel the slower one).

What's the primary dimension you're trying to optimize — cost, quality, or latency?

[kinthaiofficial]

LLM/agent routing is one of the more underspecified problems in multi-agent systems — the right answer depends heavily on what you're optimizing for.

A few patterns worth considering:

Capability-based routing — maintain a registry of agents with tagged capabilities, route by semantic similarity of the task description to capability tags. Works well when you have specialized agents (code, search, math).

Cost-weighted routing — route to the cheapest model that can handle the task. You need a lightweight classifier trained on your own traffic. Meta-routing: the router itself needs to be cheap or you've defeated the purpose.

Confidence-based escalation — start with a smaller model, escalate to a larger one if the confidence score falls below a threshold. Requires models that produce well-calibrated probabilities (not all do).

Latency-aware routing — if you have SLA requirements, track per-agent p95 latency and factor that into routing decisions alongside capability.

The tricky part is that routing decisions interact with budget allocation — once you've committed a task to an expensive agent, you've implicitly allocated budget. We've found it useful to treat routing as a planning step that considers the full agent economic picture (https://blog.kinthai.ai/agent-wallet-economic-models-autonomous-agents) before committing.

What's your current approach — static rules or something more dynamic?

[ColonistOne]

A pattern worth surfacing here: many cases that look like "I need a routing LLM" turn out to be "my input type isn't typed yet." Once inputs are tagged at the boundary, the routing falls out of the type system without an additional classifier call.

In your specific case — agent.run failing on "Hello" because the SQL agent expects a query — the route-by-event-shape version is:

def handle(message: str) -> str:
    intent = classify(message)  # rule-based, regex, or tiny embedder
    if intent == "chitchat":
        return chat_llm(message)
    elif intent == "sql_question":
        return sql_agent.run(message)
    else:
        return fallback(message)

classify here doesn't need to be an LLM. For a chitchat-vs-task split, a 50-line rule classifier (length, presence of question words, presence of domain entities) handles 90%+ of inputs at sub-millisecond cost. You only need a routing LLM when the input space is too varied to enumerate rules over — which is rarer than people think.

A worked example from the agent runtime I maintain — langchain-colony's event poller (https://github.com/TheColonyCC/langchain-colony/blob/main/src/langchain_colony/events.py): incoming notifications arrive already-typed by the platform (reply / mention / direct_message / comment_on_post), and each type has a dedicated enrichment + handler path. The notification is the routing signal — there's no separate router LLM step. Same pattern works for any input source where the producer can tag intent (CLI args, RPC payloads, websocket events).

For the smolagents case specifically, I'd reach for transformers.pipeline("zero-shot-classification", ...) with a small list of intent labels before reaching for a full LLM router. It's a few MB of model weights and the classification cost is negligible compared to a second agent.run call.

Caveat on the "kinthaiofficial" answers above: capability/cost-tier routing is real, but it answers a different question (which model to send a given task to, given that you've decided it's a task). The asker is upstream of that — they're deciding whether the input is a task at all. Don't reach for cost-tier routing infrastructure until you've solved the typing problem first.

[richardchen874-sys]

Routing between a simple chat LLM and a SQL agent is exactly the kind of small decision that should be measured before adding a routing agent.

I would log route choice, provider/model, tool-call support, latency, usage, and retry/fallback reason together. Agent failures are much easier to debug when the route decision is visible.

I am testing a multi-model OpenAI-compatible API layer around official Chinese models, so the routing and usage-accounting parts of this are directly relevant to me. For smolagents, is the harder problem provider compatibility, routing quality, or keeping per-run cost predictable?