RouteLLM: Routing Queries Between Strong and Cheaper Models

RESUMEN

En 30 segundos

Matthew Berman reviews RouteLLM, a framework from researchers including LMSYS and UC Berkeley that routes queries between a more capable model and a cheaper one. He discusses four learned routers, preference-data augmentation, benchmark results and a transfer test with a different model pair. The results depend on the benchmark, quality target and pricing assumptions. The 2024 blog’s headline summary does not match every comparison in the original paper’s cost table: its 50% gap-recovery point corresponds to about 95%, 92% and 87% of GPT-4’s score on MT Bench, MMLU and GSM8K respectively.

1The routing ideaA router assigns each request to one of two models, seeking a quality and cost tradeoff.
2An illustrative idealThe opening plot depicts an ideal router, not a measured RouteLLM result.
3Interpreting the headlineThe blog reports large savings at 95% GPT-4 performance; the original paper’s cost table uses benchmark-specific quality levels at its 50% gap-recovery point.
4Preference-based trainingThe researchers train four router types using Arena comparisons and augmented data, then compare with random routing.
5Transfer to other modelsThe same routers are evaluated on MT Bench with Claude 3 Opus and Llama 3 8B without retraining.

Conclusiones

RouteLLM chooses between a stronger and a cheaper model for each query using learned routing decisions.
The main experiments compare GPT-4 and Mixtral 8x7B, not GPT-4o as the video title implies.
On MT Bench, the augmented matrix-factorization router reached the paper’s approximately 95%-of-GPT-4 score using 13.4% GPT-4 calls.
The blog’s 85%, 45% and 35% cost-reduction summary should be read alongside the paper’s different benchmark-specific quality targets and cost assumptions.
A transfer test on Claude 3 Opus and Llama 3 8B did not require router retraining, but does not guarantee transfer to every pair.
Local inference still has hardware, electricity, maintenance and routing costs, and routing alone does not guarantee privacy or security.

TemasModel routing & gateways AI cost & performance Evaluaciones Local inference AI research

TEXTO COMPLETO

Lee el texto editado por capítulos

Ver transcripción

Solo la transcripción

Why route model queries?

Berman presents the tradeoff between expensive strong models and cheaper alternatives. The opening chart illustrates an idealized router.

RouteLLM is a 2024 open-source framework from LMSYS and collaborators that aims to reduce the cost of using language models. I discuss its approach and report the authors’ findings. The video title mentions GPT-4o, but the central experiments discussed here use GPT-4 and Mixtral 8x7B, not GPT-4o.

DIAPOSITIVA 1 / 4Illustrative cost and performance diagramThe “ideal router” marker illustrates a desired tradeoff; it is not a measured RouteLLM result.

The opening diagram puts model performance against cost and marks an “ideal router.” I describe the point as RouteLLM, but it is an illustration of the desired tradeoff, not a measured result for a particular implementation. The figure uses Chatbot Arena Elo and a 1:1 input-to-output token assumption for the plotted model costs.

I imagine a future stack combining routing, local models, agents and frontier models. The goal would be to use a less expensive model when it suffices and reserve a stronger model for harder requests. That architecture is a proposal; routing by itself does not automatically optimize privacy, security, energy or quality.

Training and evaluating RouteLLM

He explains preference comparisons and the two-model setup, then discusses reported savings. The study’s benchmark targets and cost assumptions qualify those figures.

A router must infer whether the cheaper model is likely to handle a query well. I say my local models handle most of my own questions, but that is an anecdote rather than a general measured rate. Local inference has hardware, power and maintenance costs even when it avoids a per-query API bill.

The researchers train routers on preference data: a prompt and a comparison of two responses labeled as a win, loss or tie. They use public Chatbot Arena data and augment it with additional labels. The setup studied in the video chooses between a strong model and a weaker, cheaper one for each request.

The LMSYS blog reports cost reductions of over 85% on MT Bench, 45% on MMLU and 35% on GSM8K against always using GPT-4 while retaining 95% of its performance. The original paper’s cost table needs a more precise reading: at the point recovering half the gap between the weak and strong models, its benchmark scores are approximately 95%, 92% and 87% of GPT-4’s score respectively. These figures should not be treated as one universal 95% guarantee.

Router methods and generalization

The researchers compare four routing methods with random routing and test transfer to another model pair. Berman closes with potential applications and future possibilities.

The main evaluation routes between GPT-4 and Mixtral 8x7B and compares learned routers with a random baseline. On MT Bench, the original paper reports that the matrix-factorization router augmented with judge-labeled data reaches a score of 8.8, approximately 95% of GPT-4’s 9.3, with 13.4% of requests sent to GPT-4. The random baseline needs about 49% at that same gap-recovery point.

DIAPOSITIVA 2 / 4MT Bench routing curvesThe chart compares random routing with learned methods before and after data augmentation.

That reduction is in the fraction of strong-model calls, used as a cost proxy because the study assumes the cheaper model’s cost is small relative to GPT-4. Actual total savings depend on model prices, token lengths, router overhead and deployment. The paper’s cost analysis uses historical price and token-ratio assumptions, and its reported savings table should not be applied directly to other workloads.

The four router approaches are similarity-weighted ranking, matrix factorization, a BERT classifier and a causal language-model classifier. The authors find that augmenting Arena preference data with a model judge improves the MT Bench results. They also use benchmark validation labels for MMLU augmentation; performance and the best router vary by dataset.

DIAPOSITIVA 3 / 4Preference data and router methodsThe page lists the comparison data and four router approaches used in the framework.

DIAPOSITIVA 4 / 4Transfer to a new model pairThe MT Bench chart evaluates the same routers with Claude 3 Opus and Llama 3 8B.

To test generalization, the researchers evaluate the same routers on MT Bench with Claude 3 Opus as the stronger model and Llama 3 8B as the weaker one, without retraining on that pair. The results are promising on that test, but do not demonstrate that every model pair or production distribution will transfer equally well.

I am excited by the possibility of lowering inference costs and making more applications feasible. Techniques such as multi-agent systems and chain-of-thought can consume additional tokens, so routing may be one part of a broader design. Whether it reduces total cost or energy in a given system needs measurement, including routing overhead and quality failures.

The researchers also released the paper and code. I welcome the ability to inspect and experiment with the framework; choosing thresholds, models and evaluation cases remains a practical task for each application.