Can routers select the right model before generation happens?

A key distinction exists between reward modeling and LLM routing that shapes the entire design space. Reward modeling assesses response quality after an LLM generates it. Routing selects the appropriate LLM beforehand. This requires a fundamentally different capability: estimating query complexity and model-query fit, not evaluating output quality.

Two systems converge on the same architectural insight from different angles. RouteLLM trains routers on human preference data from Chatbot Arena with data augmentation, learning to predict when a weaker model's response will be comparable to a stronger model's. Hybrid-LLM trains a difficulty-conditional router with a tunable quality threshold that can be adjusted dynamically at test time — seamlessly trading quality for cost per scenario. Both achieve 40-50% cost reduction with no meaningful quality drop.

The critical architectural constraint both share: route to a single LLM per query. This contrasts with ensemble approaches (LLM-Blender queries multiple models and selects the best response) and cascade approaches (Frugal-GPT queries LLMs sequentially until a reliable response is obtained). Single-model routing minimizes latency — the router decision is cheap, and only one generation happens. The ensemble and cascade alternatives multiply latency by the number of models queried.

Since Can we allocate inference compute based on prompt difficulty?, routing adds a complementary optimization axis: not just how much compute per query, but which model per query. The two axes are independent — you could route to a smaller model AND give it less compute on easy queries, or route to a larger model AND give it more compute on hard ones. Because Can inference compute replace scaling up model size?, routing and TTS form a two-dimensional Pareto surface where the optimal point depends on the specific query.

The practical implication: routing is deployable today with existing model APIs. Unlike training a better model (which requires pretraining investment), routing optimizes across existing models — a post-hoc efficiency gain that compounds as the model ecosystem grows.

Source: Routers