Skip to main navigation Skip to search Skip to main content

LLMRouterBench: A Massive Benchmark and Unified Framework for LLM Routing

  • Hao Li
  • , Yiqun Zhang
  • , Zhaoyan Guo
  • , Chenxu Wang
  • , Shengji Tang
  • , Qiaosheng Zhang
  • , Yang Chen
  • , Biqing Qi
  • , Peng Ye
  • , Lei Bai
  • , Zhen Wang
  • , Shuyue Hu
  • Northwestern Polytechnical University Xian
  • Shanghai Artificial Intelligence Laboratory

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

2 Scopus citations

Abstract

Large language model (LLM) routing assigns each query to the most suitable model from an ensemble. We introduce LLMRouterBench, a large-scale benchmark and unified framework for LLM routing. It comprises over 400K instances from 21 datasets and 33 models. Moreover, it provides comprehensive metrics for both performance-oriented and performance-cost trade-off routing, and integrates 10 representative routing baselines. Using LLMRouterBench, we systematically re-evaluate the field. While confirming strong model complementarity—the central premise of LLM routing—we find that many routing methods exhibit similar performance under unified evaluation, and several recent approaches, including commercial routers, fail to reliably outperform a simple baseline. Meanwhile, a substantial gap remains to the Oracle, driven primarily by persistent model-recall failures. We further show that backbone embedding models have limited impact, that larger ensembles exhibit diminishing returns compared to careful model curation, and that the benchmark also enables latency-aware analysis. All code and data are available at https://github.com/ynulihao/ LLMRouterBench.

Original languageEnglish
Title of host publication64th Annual Meeting of the Association for Computational Linguistics, Findings of the Association for Computational Linguistics
Subtitle of host publicationACL 2026
EditorsMaria Liakata, Viviane P. Moreira, Jiajun Zhang, David Jurgens
PublisherAssociation for Computational Linguistics (ACL)
Pages37733-37754
Number of pages22
ISBN (Electronic)9798891763951
StatePublished - 2026
Event64th Annual Meeting of the Association for Computational Linguistics, ACL 2026 - San Diego, United States
Duration: 2 Jul 2026 → 7 Jul 2026

Publication series

NameProceedings of the Annual Meeting of the Association for Computational Linguistics
ISSN (Print)0736-587X

Conference

Conference64th Annual Meeting of the Association for Computational Linguistics, ACL 2026
Country/TerritoryUnited States
CitySan Diego
Period2/07/26 → 7/07/26

Fingerprint

Dive into the research topics of 'LLMRouterBench: A Massive Benchmark and Unified Framework for LLM Routing'. Together they form a unique fingerprint.

Cite this