Vals AI maps the case for model routers, with its own router results still to come

Vals AI says no single model leads its eight agentic benchmarks and is building an independent router evaluation, with results not yet published.

Vals AI published a post on October 14 arguing that routers, which pick a model per request or per agent step, are now part of the AI stack. It also announced it is building an independent router evaluation. If you're deciding between routing and pinning one model, the data so far is Vals's single-model scores, not router results. Vals says it "will publish results on a public leaderboard soon."

The case rests on what Vals calls a jagged frontier. Five models hold the top spots across the Vals Index's eight benchmarks, and none ranks in the top three on all of them. Claude Sonnet 5.5 and Claude Opus 5.5 rank second and third overall but 35th and 33rd on Harvey's Legal Agent Benchmark, which Muse Spark 1.2 tops. Vals also calculated a hindsight "oracle" router. It would beat the best single model by 14 points at about 40% less cost. Vals stresses that this is a ceiling, not something any real router achieves.

The planned method runs each router as shipped, consulted on every model call in a live agent run. It tracks cost per task, cache usage and completion time against the best and cheapest single models in that router's pool.

The trade-offs are already documented. OpenRouter, a router vendor, lists the failure modes on its own benchmarks page:

Creuto notes that OpenRouter's Auto Router ranks models by community share of spend, not quality. It advises pinning models in test environments.

The self-hosted auto-model-router reports an August 29 run of 10 coding tasks: 30/30 solved at $0.63, against $16.61 for Claude Opus 5. Its median time to first token was slower. These are self-reported numbers on coding tasks only.

Sources