Developers Are Dismantling the LLM Router

Another Team Has Dismantled Its Own LLM Router

Recently, the Manifest team published a retrospective on a highly representative decision: while more and more AI companies are building LLM routers, they have instead retired their own routing layer.

This is not because routing has no value at all, nor because multi-model strategies are obsolete. Quite the opposite: as of July 31, 2026, there are more models to choose from than ever before. Closed-source models continue to diversify by capability, latency, and price; open-source models are evolving even faster; and specialized strengths have emerged in reasoning, coding, multimodality, and long-context processing. On the surface, this should be exactly the moment for intelligent routing to shine.

But the reality is: the more models there are, the less likely automatic model selection is to pay off—and the harder it may become to maintain.

Over the past two years, LLM routers have been presented as a natural engineering solution: send simple questions to inexpensive models and escalate complex ones to stronger models; route coding tasks to models with better coding capabilities and long documents to models with larger context windows; if one provider times out, automatically fail over to another. Ideally, users notice no difference while the bill drops substantially.

The problem is that a real production environment is not an exam in which every question has already been assigned a difficulty level. A router must determine which model can answer well before the answer is generated, while the quality signals it truly needs often become available only after generation.

This creates the fundamental paradox of LLM routing: if determining task difficulty itself requires a strong model, the cost advantage of routing is diminished; if an inexpensive classifier is used instead, it often fails to see what actually makes the task difficult.

The Router Is Not Unintelligent—It Simply Lacks Reliable Answers

A typical intelligent router reads a user request and outputs a model name or capability tier. It may use keyword rules, a lightweight classifier, embedding similarity, or a purpose-trained routing model. Research projects such as RouteLLM attempt to learn which requests are worth sending to stronger, more expensive models by using user preference data.

These methods can produce impressive cost-quality curves on fixed benchmarks, but online systems face an entirely different set of problems.

First, Prompts Cannot Fully Represent Task Difficulty

A request such as “Help me fix this login issue” may look simple, but it could involve dozens of files, multiple tools, an authorization system, and database state. Meanwhile, a several-thousand-word summarization request may be long but require nothing more than mechanical compression.

Character counts, keywords, and domain labels are all weak signals. What truly determines difficulty is often a set of implicit constraints:

• Whether cross-file reasoning is required;
• Whether external tools must be invoked;
• Whether the output must be parsed strictly by a program;
• How costly an incorrect answer would be to the business;
• Whether the current conversation has already accumulated critical state;
• Whether the model must understand images, audio, or extremely long attachments.

The router sees the surface of the request, while what the business actually cares about is the cost of failure.

Second, Model Capability Is Not a Straight Line from Weak to Strong

Many routing designs assume that models have a stable ranking: A is stronger than B, and B is stronger than C. Under this assumption, predicting task difficulty is enough to select the appropriate tier.

The model market in 2026 no longer fits this abstraction. The same model may excel at code completion but struggle with strict JSON; achieve high reasoning scores but exhibit unstable latency under heavy concurrency; support extremely long contexts but lack another provider’s prompt caching; or perform only moderately on text tasks while offering superior visual understanding.

As a result, routing is no longer a binary choice between a small model and a large model. It is closer to a continuously changing multi-objective optimization problem in which quality, price, time to first token, total latency, context window, cache hits, tool protocols, data compliance, and availability zones must all be considered simultaneously.

Every model upgrade requires this capability map to be redrawn.

Third, Production Systems Simply Lack Sufficiently Good Training Labels

To train a reliable quality router, a team must know at minimum: for the same real-world request, what result each model would produce, which result is better, and whether the difference is worth the additional cost.

That means calling multiple models in parallel and then comparing the outputs through human review, rules, or judge models. This is expensive in itself.

If the team records only the output of the model that was actually selected, it introduces classic selection bias: the router never learns whether the alternative path would have been better. User likes and retries can provide signals, but this feedback is sparse and delayed, and it is often confounded by factors such as the interface, network conditions, and prompt design.

Many teams ultimately find themselves maintaining not a routing model, but an entire system for data collection, offline evaluation, human annotation, version regression testing, and online experimentation. The router itself is only a small part of that system.

Engineering Coupling Is What Truly Overwhelms a Homegrown Routing Layer

If an LLM accepted only a piece of text and returned another piece of text, routing would be much simpler. But modern AI applications stopped working that way long ago.

Tool calling, structured output, reasoning budgets, multimodal input, prompt caching, batch processing, and persistent connections make model APIs look similar while their behavior becomes increasingly different. Even if every provider offers an OpenAI-style interface, that does not mean they are semantically compatible.

| Capability | Common Routing Problems | |---|---| | Tool calling | Parameter formats, parallel invocation, and tool-selection strategies differ | | Structured output | JSON Schema coverage and failure behavior differ | | Prompt caching | Cache keys, expiration periods, and billing models differ | | Long context | Identical advertised windows do not guarantee identical long-text quality | | Multimodality | Support for image counts, dimensions, video, and files differs | | Reasoning controls | Reasoning budgets, visible reasoning content, and latency differ | | Content safety | Refusal boundaries and moderation policies differ |

Switching models is not simply a matter of changing the model name in a request. It may require reorganizing messages, trimming context, rewriting tool descriptions, converting image formats, and reinterpreting error codes.

This is also why many homegrown routers gradually become bloated. They begin with a few dozen lines of conditional logic, then accumulate provider adapters, rate limiters, retry policies, circuit breakers, caches, log redaction, cost accounting, and canary releases. Eventually, they are no longer routers, but AI control planes maintained entirely by individual teams.

Worse still, this control plane sits on the critical path of every request. Any incorrect decision has an amplified blast radius.

Agents Make Per-Request Routing Even More Dangerous

In the chatbot era, choosing the wrong model for a request generally meant only that the answer was slightly worse. In an agent workflow, however, a single task may involve planning, retrieval, tool calls, code execution, result validation, and multiple rounds of correction. Different steps can theoretically use different models, but models cannot be switched from one turn to the next without constraints.

The reason is straightforward: models do not interpret tool state and conversation history in exactly the same way.

One model may generate a tool call, only for the next model to misunderstand why those parameters were chosen. One model may habitually preserve a detailed plan in the context, while another may ignore it. Switching can also invalidate provider-side prefix caches, causing a model combination that looks cheaper on paper to produce higher total costs and longer latency.

For long-running tasks, the reliable approach is usually not to guess the model again for every request, but to use “session pinning” or “workflow pinning”: select a model at the beginning of the task based on capability constraints, keep it stable during execution, and fall back only in cases of explicit failure, rate limiting, or service unavailability.

This is less glamorous than dynamic routing, but much easier to debug.

Retry Costs May Wipe Out Token Savings

Routers are often promoted using the average reduction in cost per call. But what production systems should actually calculate is the total cost per successful task.

Suppose an inexpensive model costs only one-third as much as a strong model for a single request, but a failure requires the stronger model to rerun the task. The team then pays for the inexpensive model call, the strong model call, the repeated input context, and the additional latency. For tool-using agents, searches, sandbox executions, or database queries may also be repeated.

A more subtle loss comes from tail latency. Routing decisions take time, the first model’s failure takes time, and regeneration by the fallback model takes more time. Average cost may fall while P95 latency and user churn rise.

For this reason, routing evaluations that show only per-call token prices have limited value. At minimum, the following should also be monitored:

1. End-to-end task success rate;
2. Total cost per successful task;
3. P50, P95, and P99 latency;
4. Escalation rate after the initial request fails;
5. Whether cache hit rates decline because of model switching;
6. The magnitude of regressions after different model versions are updated.

If these metrics cannot be collected continuously, intelligent routing can easily become infrastructure that only appears to save money.

Intelligent Guessing Is Receding, Not Model Gateways

Manifest’s decision should not be interpreted as evidence that model routing is dead. More precisely, what is receding is an overly optimistic design: having a general-purpose classifier read any request and automatically guess which model offers the best price-performance ratio.

Stable multi-model systems still need gateways, but their responsibilities should be more restrained.

The first layer is hard constraints. Filter out models that cannot be used based on context length, input modality, data residency, tool capabilities, and compliance requirements. This is not prediction; it is deterministic filtering.

The second layer is scenario configuration. The business explicitly specifies a default model. Code review, customer-service summarization, and contract extraction, for example, each use a validated model combination rather than letting the router improvise at runtime.

The third layer is reliability fallback. Switch models only in response to timeouts, rate limits, provider outages, or format-validation failures, and prepare compatible prompts and tool definitions for every fallback path.

Only the fourth layer is experimental routing. Test less expensive models on a small percentage of low-risk, automatically evaluable traffic, and use continuous evaluation to determine whether to expand deployment rather than immediately handing over all production requests.

In short: the routing layer is better suited to determining “which models are allowed” than guessing “which model will be smartest this time.”

Which Teams Can Still Benefit from Building Their Own Router

Building a custom router is not always uneconomical. Teams that meet the following conditions may still derive significant value from it:

• They handle enough calls each day that a cost change of a few percentage points can cover the platform investment;
• Their request types are highly repetitive, such as summarization, classification, extraction, and fixed-format generation;
• They have reliable automated evaluators that can quickly determine whether an output is acceptable;
• They have already accumulated comparative data from running the same requests through different models;
• They have a dedicated team maintaining evaluations, provider integrations, and online experiments;
• Their business can tolerate retries after failures without triggering irreversible external actions.

Conversely, if a team has only a few developers, its business prompts are still changing frequently, and its agents perform actions such as making payments, sending messages, or modifying databases, then building a learning-based router is likely just paying an infrastructure tax prematurely.

Such teams should first do three things: unify observability, define scenarios clearly, and prepare fallback paths. They can discuss training a router once their call volume and evaluation data are sufficient.

How Developers Should Redesign the Architecture

A more realistic multi-model architecture for 2026 does not need to cram all intelligence into the request entry point.

First, consolidate scattered model-selection logic from application code into versioned configuration. For each business scenario, record the default model, permitted alternatives, capability requirements, and cost ceiling. This allows model upgrades to be rolled out independently through canary deployments without modifying the entire application.

Second, retain complete end-to-end observability. In addition to token counts, record tool-call success rates, structured-output validation, reasons for retries, cache state, and final business outcomes. Without this data, any routing optimization is merely a local optimization.

Third, move evaluation into offline testing and shadow traffic. Let new models process a small portion of real requests in parallel without directly affecting users. Adjust the default model only after confirming quality, cost, and latency. Rather than asking the router to solve a problem with no definitive answer in real time, have the team solve it before release.

Finally, preserve explicit adaptation layers for provider differences. A unified interface can reduce integration costs, but it should not erase every model-specific feature. A genuinely valuable gateway does not pretend that all models are identical; it provides a consistent invocation method while allowing the business to declare which native capabilities it depends on.

This Retreat Is Really an Infrastructure Course Correction

Over the past year, LLM routers have attracted attention because they promised to solve the two most visible problems at once: too many models and calls that are too expensive. Engineering experience is now showing that these two problems cannot be solved once and for all by a classifier.

Model selection is fundamentally the intersection of product decisions, evaluation decisions, and reliability decisions. Handing all of it to a real-time routing layer means postponing uncertainty that should be resolved during the release process until every individual user request arrives.

This is also the significance of Manifest retiring its homegrown router. It does not signal the end of the multi-model era. Instead, it reminds developers that smarter abstraction layers are not always better. For most teams, a thin, observable model gateway centered on rules and fallbacks is more useful than a black-box router that judges task difficulty on its own.

Intelligent routing will not disappear, but it will retreat from the default architecture into specific scenarios where traffic is large enough, tasks are stable enough, and feedback is clear enough. Other teams do not need to turn their model invocation pipeline into the hardest system in the company to troubleshoot merely to save a portion of their token costs.

References

• RouteLLM: An Open-Source Model Routing Framework Driven by Preference Data — Demonstrates how to use preference data to train routing strategies that balance model quality and inference cost.
• A Chinese-Language Overview of the RouteLLM Paper and Methodology — Introduces the basic approach and cost trade-offs involved in training lightweight routing models from user preferences.
• LiteLLM — A popular multi-provider model gateway project covering unified interfaces, fallbacks, budgets, and observability.
• Portkey AI Gateway — An open-source AI gateway implementation useful for understanding the boundaries of routing, caching, and reliability controls in model invocation.