Paper deep dive
Scaling Enterprise Agent Routing: Degradation, Diagnosis, and Recovery
Kellen Gillespie, Robyn Perry
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 6/20/2026, 10:01:45 AM
Summary
This paper investigates the degradation of routing accuracy in LLM-based enterprise assistants as the number of specialized tools and agents in a catalog scales. Through a study of a 110-agent, 584-tool catalog, the authors identify that routing F1 drops significantly (16-23 percentage points) due to a 'retrieval gap' (the model fails to surface the correct tool) and a 'confusion gap' (the model confuses semantically similar tools). The research demonstrates that embedding-based shortlisting (specifically using text-embedding-3-large with k=20) effectively recovers 10-11pp of F1 by addressing the retrieval gap, outperforming hierarchical and platform-level tool search methods. The findings were validated using both synthetic queries and a production study of 1,435 human-annotated utterances.
Entities (7)
Relation Signals (4)
GPT-5.4 → exhibits → recall-driven degradation
confidence 100% · Flat tool-level routing (GPT-5.4) drops from 58.2% to 42.1%.
embedding shortlisting → recovers → F1 Score
confidence 100% · Embedding shortlisting (text-embedding-3-large, k=20 tools) outperforms flat routing at every scale point (+6–11pp...)
retrieval gap → causedby → catalog scaling
confidence 90% · The degradation is recall-driven: precision drops moderately... while recall drops more than twice as fast... as the catalog grows.
confusion gap → characterizedby → semantic overlap
confidence 90% · The confusion gap is amplified by the enterprise productivity domain, where functionally similar tools... grow naturally with the catalog.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Production LLM assistants route user requests to growing libraries of specialized tools, but how does routing accuracy degrade as the catalog scales? We study single-step routing on a 110-agent, 584-tool catalog from a deployed enterprise productivity assistant, evaluating three frontier models from 10 to 110 agents. Routing F1 on under-specified requests drops 16--23 percentage points across models. An oracle analysis decomposes the degradation into a \emph{retrieval} gap (the model cannot surface the right tool) and a \emph{confusion} gap (even with perfect retrieval, the oracle ceiling drops 10pp). Embedding-based shortlisting recovers +10--11pp F1 at full scale across all three models and two providers. A production annotation study (1,435 human-labeled utterances, three annotators) confirms the recovery on real traffic at +10--17pp despite 10--15pp lower absolute performance.
Tags
Links
- Source: https://arxiv.org/abs/2606.17519v1
- Canonical: https://arxiv.org/abs/2606.17519v1
Trouble viewing inline? Open PDF directly →
Full Text
32,420 characters extracted from source content.
Expand or collapse full text
Scaling Enterprise Agent Routing: Degradation, Diagnosis, and Recovery Kellen Gillespie and Robyn Perry Superhuman, Inc. kellen.gillespie, robyn.perry@grammarly.com Abstract Production LLM assistants route user re- quests to growing libraries of specialized tools, but how does routing accuracy de- grade as the catalog scales?We study single-step routing on a 110-agent, 584-tool catalog from a deployed enterprise produc- tivity assistant, evaluating three frontier models from 10 to 110 agents. Routing F1 on under-specified requests drops 16–23 percentage points across models. An oracle analysis decomposes the degradation into a retrieval gap (the model cannot surface the right tool) and a confusion gap (even with perfect retrieval, the oracle ceiling drops 10p). Embedding-based shortlisting re- covers +10–11p F1 at full scale across all three models and two providers. A pro- duction annotation study (1,435 human- labeled utterances, three annotators) con- firms the recovery on real traffic at +10– 17p despite 10–15p lower absolute per- formance. 1 Introduction LLM-based assistants increasingly serve as or- chestration layers that route user requests to specialized agents for email, project tracking, scheduling, and more. As organizations add agents to these systems, the routing decision becomes harder and the model must select from a growing catalog of semantically over- lapping options. This scaling challenge is already driving platform-level responses. OpenAI introduced namespace-based tool search, Anthropic pro- vides BM25 retrieval over tool descriptions, and MCP server registries are growing beyond what flat tool lists can support. Prior work shows that tool-calling performance degrades with catalog size (Kate et al., 2025) and that retrieval errors dominate agent failures (Mo et al., 2025), but the mechanism (what breaks, at what scale, and what levers exist) remains undercharacterized. We present a controlled study of single-step routing accuracy across 10–110 agents on a production-sourced catalog from a deployed enterprise productivity assistant. Our analysis has two parts: 1. Scaling diagnosis (§4.1). F1 drops 16– 23p, driven primarily by recall. An or- acle analysis decomposes this into a re- trieval gap (the model cannot surface the right tool) and a confusion gap (the ora- cle ceiling drops from 79% to 69%). The confusion gap is amplified by the enter- prise productivity domain, where func- tionally similar tools (Gmail/Outlook for email, Improve/Paraphraser/Proofreader for writing, Jira/Asana for project man- agement) grow naturally with the catalog (Qin et al., 2024; Shi et al., 2025; Patil et al., 2025). 2. Shortlisting as intervention (§5,§5.3). Embedding shortlisting recovers +10– 11p F1 at full scale across three models from two providers. The recovery holds on 1,435 human-annotated production ut- terances (+10–17p). Tool-level retrieval outperforms all pack-level approaches (hi- erarchical LLM routing, pack-level em- bedding, platform tool search) by 2– 4p.An error composition analysis (§5.5) shows that shortlisting cuts rout- ing misses from 31% to 10% at the cost of a stable 9% shortlister miss rate. 2 Related Work Tool-count scaling. Kate et al. (2025) stress-test tool calling at 49–741 tools, re- porting 7–85% performance drops.LiveM- arXiv:2606.17519v1 [cs.CL] 16 Jun 2026 CPBench (Mo et al., 2025) finds retrieval er- rors account for ∼50% of agent failures across 527 tools. Toolshed (Lumer et al., 2025b), ScaleMCP (Lumer et al., 2025a), MonoScale (Shao et al., 2026), and RAG-MCP (Gan and Sun, 2025) document performance collapse with growing tool and agent pools. We add a precision/recall decomposition and controlled mitigations to this line of work. Tool retrieval and selection. Toolformer (Schick et al., 2023) teaches LMs to insert tool calls during generation. Retrieve-then- route approaches range from document re- trieval over API catalogs (Patil et al., 2024; Qin et al., 2024), fine-tuned retrievers (Shi et al., 2025; Zheng et al., 2026), reranking and query rewriting (Zheng et al., 2024; Chen et al., 2024), and token-level tool encoding (Hao et al., 2023; Wang et al., 2025). Bench- marks and data generation pipelines (Liu et al., 2024; Wu et al., 2024; Lu et al., 2025) complement these with evaluation methodol- ogy. ToolScope (Liu et al., 2026) tackles se- mantic overlap by merging similar tools at the catalog level. We show that dense embed- ding retrieval outperforms both platform ap- proaches and a fine-tuned retriever (Shi et al., 2025) without an LLM call. Agent system scaling. Kim et al. (2026) study when multi-agent coordination outper- forms single agents, while AgentArch (Bo- gavelli et al., 2025) varies agent architecture on fixed tool sets. HuggingGPT (Shen et al., 2023) and AnyTool (Du et al., 2024) route via hierarchical dispatch over models and API tiers respectively. ScaleCall (Osuagwu et al., 2025) evaluates hybrid retrieval for enterprise tool selection. We hold architecture constant and vary catalog size, comparing hierarchical dispatch against flat embedding retrieval. 3 Experimental Setup 3.1 Agent Catalog Our catalog comprises 110 agents and 584 individual tools from a deployed enterprise productivity assistant, ranging from single- purpose agents (Weather, Google Translate) to multi-tool suites (Gmail with 15+ actions, Jira with 20+). The catalog has natural se- mantic overlap (multiple email clients, writing tools, project trackers, and document editors), creating routing ambiguity uncommon in API- centric benchmarks (Qin et al., 2024; Shi et al., 2025; Patil et al., 2025). We evaluate mini- mal (name and description) and rich (name, description, examples, semantic tags, enriched descriptions) metadata variants. 3.2 Evaluation Data Synthetic queries. 4,105 queries generated by GPT-4o across difficulty levels: explicit (names the tool: “send a Gmail”) and im- plicit (describes the need without naming it: “email the team about Monday’s deadline”). Each query has a target tool and also-valid la- bels enabling dynamic ground truth that ad- justs to each sampled tool set. The routing models (GPT-5.x, Sonnet) differ from the gen- eration model, though GPT-4o and GPT-5.x share a provider, creating potential distribu- tional affinity. Cross-provider replication with Sonnet and the production validation (§5.3) on human-written utterances mitigate this con- cern. Productionqueries. 1,435utterances sampled from production traffic of the de- ployed system, stratified across agents with sufficient traffic (capped at 100 utterances per agent) and filtered for quality (language, safety).Three trained linguists indepen- dently annotated each utterance with top-5 candidates from an LLM-based shortlister (GPT-5.4), independent of the embedding retriever evaluated in§5.Each annotator rated every candidate as best option, also valid, or not applicable, and could nominate tools outside the shortlisted set; fewer than 1% of gold labels (13 cases) required out-of- pool nominations. Gold labels use majority vote (≥2/3 annotators). Per-candidate Krip- pendorff’s α=0.68 (ordinal), reflecting the catalog’s semantic overlap: annotators agree on which tools are relevant but often disagree on which is best among near-equivalents. At the item level, 94% of utterances have at least one shared valid tool across annotators. All production queries are implicit. Implicit queries as primary metric. Ex- plicit queries are near-ceiling (>90% F1) at all scales. We report on implicit queries through- out, as they represent realistic production traf- fic where the user does not name the target tool. 3.3 Models and Routing We evaluate three frontier models from two providers: GPT-5.1 and GPT-5.4 (Ope- nAI, function calling via Responses API) and Claude Sonnet 4.5 (Anthropic, native tool use). All use function-calling interfaces where the tool catalog is provided as callable func- tion definitions. 3.4 Scale Points and Sampling We evaluate at scale points of 10, 20, 30, 40, 60, 80, 100, and 110 agents (51–584 tools). At each non-endpoint scale, we sample k=3 agent subsets (folds) and report fold-averaged metrics with bootstrap 95% confidence inter- vals. The 110-agent endpoint is the full cat- alog (single subset), so CIs there are query- level only. Infrastructure agents (general as- sistant, knowledge search, web search) are al- ways present. 3.5 Metrics Multi-label precision, recall, and F1, com- puted per-query against the dynamic valid set (target tool plus any also-valid tools present in the current fold). 4 How Routing Degrades at Scale 4.1 Scaling Curves Figure 1 shows routing F1 on implicit queries as catalog size grows from 51 to 584 tools. Flat tool-level routing (GPT-5.4) drops from 58.2% to 42.1%. The degrada- tion is recall-driven: precision drops mod- erately (68%→60%) while recall drops more than twice as fast (55%→37%). As the cata- log grows, the model increasingly misses valid tools more often than it selects invalid ones. A fixed cohort of 731 queries present at ev- ery scale point shows the same degradation (14.9p), confirming it is driven by catalog growth rather than query composition (Ap- pendix D). Two-component decomposition. An or- acle shortlister (all dynamically-valid tools plus random distractors to fill 20 slots) es- tablishes an upper bound on what routing can achieve with perfect retrieval. The ora- cle drops from 79.0% to 68.8%, a 10p decline even with the correct tool always present. This reveals two independent degradation sources: (a) a retrieval gap, the 16p difference be- tween oracle and practical shortlisting at full scale, recoverable by better retrievers; and (b) a confusion gap, the oracle’s own 10p decline.This decline reflects both incom- plete coverage of growing equivalence classes (valid-set size increases from 1.6 to 3.2 tools per query at scale) and genuine inter-tool confusion.Valid-set growth affects recall but not precision, so the 8p precision drop (68%→60%) confirms confusion independent of the coverage effect. The effective confusion gap in practice is likely larger than 10p, since the oracle’s random distractors are less confus- able than a real retriever’s semantically similar candidates. Cross-model reproducibility. All three models exhibit the same recall-driven degra- dation pattern with an elbow at 40–60 agents (Appendix A). GPT-5.4 outperforms GPT-5.1 by 4–8p across scale. Sonnet starts higher (66.3% at 51 tools) but degrades faster (−20p vs. −16p for GPT-5.4).Stronger models provide a constant offset but follow the same curve. Metadata and tool search. Rich meta- data (examples, tags, enriched descriptions) provides a scale-increasing benefit at the agent level (+1.2p at 20, +4.2p at 110) but near- zero effect at the tool level (<1p). Metadata quality complements architectural changes but does not substitute for them.OpenAI’s namespace-based tool search provides partial relief at moderate scale but plateaus at larger catalogs (§5.4). 5 Shortlisting Across Scale We ask whether pre-filtering the catalog to a small shortlist can close the retrieval gap re- sponsible for most of the scaling loss. 5.1 Shortlisting Comparison Figure 1 compares four approaches across 51–584 tools. Embedding shortlisting (text- embedding-3-large, k=20 tools) outperforms flat routing at every scale point (+6–11p, paired bootstrap p<0.01 at all points) and 100200300400500600 Tools in catalog 40 50 60 70 80 F1 on implicit queries (%) Flat routing Platform tool search Embedding shortlist (k=20) Oracle (k=20) Figure 1: Routing F1 (implicit queries) across catalog scale (GPT-5.4). Shaded bands: fold standard deviation. Flat routing degrades from 58% to 42%. Embedding shortlisting (k=20) recovers +10p at full scale. Oracle ceiling (dashed) drops 10p, indicating confusion independent of retrieval quality. Tool search helps above ∼180 tools but plateaus. matches or exceeds platform tool search throughout. We fix k=20 based on a sen- sitivity analysis (Appendix C) showing that F1 plateaus at k≥10 and is statistically in- distinguishable from k=20 to k=50. At 584 tools embedding reaches 52.5%, while tool search reaches 50.3% and flat (no shortlisting) reaches 42.1%. The 16p gap between embedding and ora- cle at full scale reflects retrieval quality, since embedding returns semantically similar dis- tractors that are harder for the router than random ones. Across approaches, even oracle recall drops 15p (Figure 3, Appendix A). 5.2 Cross-Model Reproducibility Table 1 shows the shortlisting recovery across models and providers. All three converge to ∼+10p at full scale despite different baselines (GPT-5.4: 42.1%, GPT-5.1: 40.6%, Sonnet: 45.9%). Sonnet’s smaller delta at 120 tools (+2.7p) reflects its stronger baseline, leaving less recall to recover. As Sonnet’s baseline de- grades at larger catalogs, the shortlisting ben- efit grows, matching the GPT models at full scale. 5.3 Production Validation Table 2 validates the synthetic findings on 1,435 human-annotated production utter- ances. Despite flat baselines ranging from 28% to 36% F1 at full scale, all three models land 120325584 Modeltoolstoolstools GPT-5.4 flat56.148.742.1 + embk=2065.458.952.5 ∆+9.3+10.3+10.4 GPT-5.1 flat57.450.440.6 + embk=2066.360.351.9 ∆+8.9+9.9+11.3 Sonnet 4.5 flat66.857.045.9 + embk=2069.564.455.9 ∆+2.7+7.4+10.0 Table 1: Cross-model shortlisting recovery (F1 on implicit queries). GPT-5.4 at 325 tools is fold- averaged (k=3, σ=2.0); all other entries are fold 0. Sonnet on fold 0 only (API cost). Paired bootstrap 95% CIs on full-scale deltas: GPT-5.4 [9.2, 11.5], GPT-5.1 [10.1, 12.5], Sonnet [8.9, 11.1]; all exclude zero. within a 2p band (44–46%) once shortlisting is applied. Absolute F1 is 10–15p lower than syn- thetic, consistent with production traffic be- ing entirely implicit and containing mis- spellings, fragments, and ambiguous intent ab- sent from synthetic generation. The synthetic- production gap is comparable for GPT-5.4 (14p) and Sonnet (14p), suggesting it re- flects query difficulty rather than distribu- tional affinity between GPT-4o query gener- ation and GPT routing models. At the query level, with shortlisting at full scale 80% of syn- thetic queries and 60% of production queries 120325584 Modeltoolstoolstools GPT-5.4 flat42.236.827.9 + embk=2052.949.444.4 ∆+10.7+12.7+16.5 GPT-5.1 flat50.345.036.2 + embk=2052.950.946.0 ∆+2.6+6.0+9.8 Sonnet 4.5 flat53.641.932.2 + embk=2054.951.345.5 ∆+1.3+9.4+13.3 Table 2: Production validation: F1 on 1,435 human-annotated production utterances (implicit only). Scale 325 is fold-averaged (k=3); endpoints are single-fold. Shortlisting recovery replicates the synthetic pattern (Table 1): +10–17p at full scale, growing with catalog size. Paired bootstrap 95% CIs on full-scale deltas exclude zero: [12.8, 20.2], [5.7, 14.0], [9.9, 16.8]. receive at least one correct tool. 5.4 Retrieval Method Comparison Tool-level beats pack-level. For both em- bedding models, tool-level retrieval (k=20) outperforms pack-level (k=5 packs/agents, ex- panded to ∼30 tools) by 2–4p consistently, since pack expansion loads irrelevant sibling tools that dilute the candidate set. Part of this edge (∼2p) comes from ranked retrieval exploiting the router’s positional bias (Ap- pendix F). All pack-level approaches converge at full scale, with platform tool search (50.3%), pack-level embedding (49.1%), and hierarchi- cal LLM routing (47.9%) falling within 2.4p at 584 tools. The hierarchical baseline selects only 1.2 packs on average (83% hit rate), mak- ing the pack intermediate step a source of un- recoverable error. Largedenseretrieveroutperforms lexical and fine-tuned alternatives. Text-embedding-3-large (52.5%) outperforms ToolRet-e5 (Shi et al., 2025), a 335M re- triever fine-tuned on 200k tool-retrieval pairs (48.7%), by 4p. The fine-tuned model was trained on API-centric data and may face an out-of-domain penalty on enterprise produc- tivity tools. A domain-matched fine-tune at comparable scale could close or reverse this gap.The fine-tuned retriever runs locally (∼2ms/query) vs. an API call (∼50ms). BM25 shortlisting (k=20) falls below flat routing at every scale point (32.8% vs. 42.1% at 584 tools), as enterprise productivity tools share vocabulary across agents, making lexical matching ineffective. 5.5 Error Analysis Query outcomes. Figure 2a decomposes query outcomes into five categories. As the catalog scales, fully correct predictions drop from 39% to 17% in flat routing and from 46% to 22% with shortlisting, while partial matches grow to fill the gap. Shortlisting cuts routing misses from 31% to 10% at full scale at the cost of 9% shortlister misses. The trade is fa- vorable: correct-or-partial coverage rises from 69% to 80%, and the shortlister miss rate is stable across scale. Prediction-level accuracy. At the predic- tion level (Figure 2b), shortlisting improves ac- curacy from 61% to 75% correct at full scale. The largest error in flat routing is catch-all absorption (general-purpose agents absorbing specific queries, 19% of predictions), which shortlisting reduces to 4%. Cross-cluster con- fusion (routing to a wrong semantic cluster, 14%→17%) and same-cluster overlap (routing to a similar tool in the correct cluster, ∼3%) are largely unaffected, and intra-pack errors are negligible (<3%). Shortlisting helps by narrowing the candidate set, reducing the op- portunity for general-purpose agents to absorb specific queries. Positional bias in routing. The router ex- hibits primacy bias: ranked shortlisters pro- vide ∼2p of free accuracy from retrieval or- dering, and oracle upper bounds with fixed ground-truth position overstate the routing ceiling by ∼4p (Appendix F). 6 Discussion The retrieval gap (16p) is addressable with better retrievers. Embedding shortlisting at k=20 adds∼50ms (API) or∼2ms (local) while shrinking the routing prompt from 584 to 20 tool definitions.For catalogs beyond ∼30 agents (∼180 tools), the accuracy gain out- weighs this cost. The confusion gap (at least 10p) is not re- coverable by retrieval alone. Shortlisting nar- rows the candidate set but the router still con- fuses semantically similar tools. Promising di- (a) Query-level outcomes (b) Prediction-level accuracy Figure 2: Error analysis across catalog scale (GPT-5.4). (a) Query-level outcomes: shortlisting converts routing misses (red) into correct predictions (green) at the cost of bounded shortlister misses (purple, ∼9%). At full scale, 80% of queries receive at least one correct tool with shortlisting vs. 69% flat. (b) Prediction-level accuracy: catch-all absorption (red) drops from 19% to 4% with shortlisting while cross-cluster confusion is largely unchanged. rections include description deduplication (Liu et al., 2026), dynamic tool exclusion at routing time, and multi-turn clarification for ambigu- ous intent. These findings are measured on a single en- terprise productivity catalog with dense se- mantic overlap. Domains with more function- ally distinct tools may degrade slower, and the ∼30-agent threshold is catalog-specific. The qualitative pattern (recall-driven degra- dation, recoverable by retrieval) is more likely to transfer than the specific numbers. 7 Conclusion On a production catalog of 110 agents (584 tools), single-step routing degrades 16–23p as the catalog scales. The degradation is recall- driven and decomposes into a retrieval gap recoverable by better candidate selection and a confusion gap (at least 10p on this cata- log) that better retrieval alone cannot close. Embedding shortlisting recovers +10–11p F1 across three models and two providers, with tool-level retrieval consistently outperforming all pack-level approaches including platform tool search and hierarchical LLM routing. A production annotation study (1,435 human- labeled utterances, three-way annotation) val- idates the synthetic findings, with shortlist- ing recovery replicating at +10–17p on real traffic despite 10–15p lower absolute perfor- mance. Limitations Our primary evaluation uses synthetic queries; the production validation (§5.3) confirms the pattern with fold-averaging at intermediate scale and moderate inter-annotator agreement (α=0.68).Production queries are entirely implicit and 10–15p harder than synthetic, likely reflecting both distributional differences and annotation strictness. All experiments use a single enterprise pro- ductivity catalog. Domains with less semantic overlap (e.g., distinct API services) may de- grade slower, while domains with more overlap (e.g., multiple coding assistants) may degrade faster. GPT models use OpenAI function calling (Responses API); Sonnet uses Anthropic na- tive tool use. The cross-model comparison re- flects both model and interface differences. We verified that Sonnet’s native tool use and text- in-prompt routing produce similar F1 (<1p difference at scale 20), suggesting the interface effect is small for this model. References Tara Bogavelli, Roshnee Sharma, and Hari Sub- ramani. 2025.Benchmark of agentic config- urations for enterprise tasks. arXiv preprint arXiv:2509.10769. Yanfei Chen, Jinsung Yoon, Devendra Singh Sachan, Qingze Wang, Vincent Cohen-Addad, Mohammadhossein Bateni, Chen-Yu Lee, and Tomas Pfister. 2024. Re-invoke: Tool invocation rewriting for zero-shot tool retrieval. In Find- ings of EMNLP 2024. Yu Du, Fangyun Wei, and Hongyang Zhang. 2024. Anytool: Self-reflective, hierarchical agents for large-scale API calls.In Proceedings of the International Conference on Machine Learning (ICML). Tiantian Gan and Qiyao Sun. 2025. RAG-MCP: Mitigating prompt bloat in LLM tool selec- tion via retrieval-augmented generation. arXiv preprint arXiv:2505.03275. Shibo Hao, Tianyang Liu, Zhen Wang, and Zhit- ing Hu. 2023. ToolkenGPT: Augmenting frozen language models with massive tools via tool em- beddings. In Advances in Neural Information Processing Systems (NeurIPS). Kiran Kate, Tejaswini Pedapati, Kinjal Basu, Yara Rizk, Vijil Chenthamarakshan, Subhajit Chaud- hury, Mayank Agarwal, and Ibrahim Abdelaziz. 2025. Longfunceval: Measuring the effective- ness of long context models for function calling. arXiv preprint arXiv:2505.10570. Yubin Kim, Ken Gu, Chanwoo Park, Chunjong Park, Samuel Schmidgall, A. Ali Heydari, Yao Yan, Zhihan Zhang, Yuchen Zhuang, Yun Liu, Mark Malhotra, Paul Pu Liang, Hae Won Park, Yuzhe Yang, Xuhai Xu, Yilun Du, Shwetak Pa- tel, Tim Althoff, Daniel McDuff, and Xin Liu. 2026. Towards a science of scaling agent sys- tems. arXiv preprint arXiv:2512.08296. Marianne Menglin Liu, Daniel Garcia, Fjona Parl- laku, Vikas Upadhyay, Syed Fahad Allam Shah, and Dan Roth. 2026. Toolscope: Enhancing LLM agent tool use through tool merging and context-aware filtering. In Proceedings of ACL 2026. Zuxin Liu, Thai Hoang, Jianguo Zhang, Ming Zhu, Tian Lan, Shirley Kokane, Juntao Tan, Weiran Yao, Zhiwei Liu, Yihao Feng, Rithesh Murthy, Liangwei Yang, Silvio Savarese, Juan Car- los Niebles, Huan Wang, Shelby Heinecke, and Caiming Xiong. 2024.APIGen: Auto- mated pipeline for generating verifiable and di- verse function-calling datasets. arXiv preprint arXiv:2406.18518. Xuan Lu, Haohang Huang, Rui Meng, Yaohui Jin, Wenjun Zeng, and Xiaoyu Shen. 2025. Tools are under-documented: Simple document ex- pansion boosts tool retrieval. arXiv preprint arXiv:2510.22670. Elias Lumer, Anmol Gulati, Vamse Kumar Sub- biah, Pradeep Honaganahalli Basavaraju, and James A. Burke. 2025a.Auto-synchronizing MCP tool indexing with agentic-RAG retrieval. arXiv preprint arXiv:2505.06416. Elias Lumer, Vamse Kumar Subbiah, James A. Burke, Pradeep Honaganahalli Basavaraju, and Austin Huber. 2025b.Toolshed: Advanced RAG-tool fusion. In Proceedings of ICAART 2025. Guozhao Mo, Wenliang Zhong, Jiawei Chen, Qian- hao Yuan, Xuanang Chen, Yaojie Lu, Hongyu Lin, Ben He, Xianpei Han, and Le Sun. 2025. Livemcpbench:Evaluating tool-using agents with real-world MCP servers. arXiv preprint arXiv:2508.01780. Richard Osuagwu, Thomas Cook, Maraim Ma- soud, Koustav Ghosal, and Riccardo Mattivi. 2025. Scalecall: Agentic tool calling at scale for fintech. arXiv preprint arXiv:2511.00074. Shishir G. Patil, Huanzhi Mao, Fanjia Yan, Char- lie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. 2025. Berkeley func- tion calling leaderboard. In Proceedings of the International Conference on Machine Learning (ICML). Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. 2024. Gorilla: Large lan- guage model connected with massive APIs. In Advances in Neural Information Processing Sys- tems (NeurIPS). Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xian- gru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2024. ToolLLM: Facilitating large language models to master 16000+ real-world APIs. In Proceedings of the International Conference on Learning Representations (ICLR). Spotlight. Timo Schick, Jane Dwivedi-Yu, Roberto Dess`ı, Roberta Raileanu, Maria Lomeli, Eric Ham- bro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. In Advances in Neural Information Processing Sys- tems (NeurIPS). Shuai Shao, Yixiang Liu, Bingwei Lu, and Weinan Zhang. 2026.Scaling multi-agent system with monotonic improvement. arXiv preprint arXiv:2601.23219. Yongliang Shen, Kaitao Song, Xu Tan, Dong- sheng Li, Weiming Lu, and Yueting Zhuang. 2023.HuggingGPT: Solving AI tasks with ChatGPT and its friends in Hugging Face. In Advances in Neural Information Processing Sys- tems (NeurIPS). Zhengliang Shi, Yuhan Wang, Lingyong Yan, Pengjie Ren, Shuaiqiang Wang, Dawei Yin, and Zhaochun Ren. 2025.Retrieval models aren’t tool-savvy: Benchmarking tool retrieval for large language models. In Findings of ACL 2025. Renxi Wang, Xudong Han, Lei Ji, Shu Wang, Tim- othy Baldwin, and Haonan Li. 2025. Toolgen: Unified tool retrieval and calling via generation. In Proceedings of the International Conference on Learning Representations (ICLR). Mengsong Wu, Tong Zhu, Han Han, Chuanyuan Tan, Xiang Zhang, and Wenliang Chen. 2024. Seal-tools: Self-instruct tool learning dataset for agent tuning and detailed benchmark. arXiv preprint arXiv:2405.08355. YanZhao Zheng, ZhenTao Zhang, Chao Ma, Yuan- Qiang Yu, JiHuai Zhu, Yong Wu, Tianze Xu, Baohua Dong, Hangcheng Zhu, Ruohui Huang, and Gang Yu. 2026. Skillrouter: Skill rout- ing for LLM agents at scale. arXiv preprint arXiv:2603.22455. Yuanhang Zheng, Peng Li, Wei Liu, Yang Liu, Jian Luan, and Bin Wang. 2024. Adaptive and hierarchy-aware reranking for tool retrieval. In Proceedings of LREC-COLING 2024. A Full Scaling Results by Model Figure 3 shows precision and recall curves across scale for all four approaches. Tables 3– 5 report precision, recall, and F1 on implicit queries for flat routing and embedding short- listing (k=20 tools, text-embedding-3-large) at all scale points. GPT-5.4 and GPT-5.1 flat results are fold- averaged at intermediate scales (k=3 folds); all other entries are fold 0. Scale 110 (584 tools) is the full catalog and has a single fold. FlatEmbk=20 ToolsPRF1PRF1 5168.155.258.273.762.064.7 12064.254.556.175.062.865.4 18061.853.154.169.557.960.1 23059.150.051.369.658.760.6 32557.847.048.770.856.058.9 43059.540.944.370.149.453.2 53058.938.442.471.547.952.5 58460.237.342.172.547.352.5 Table 3: GPT-5.4 scaling results (implicit queries). Intermediate scales are fold-averaged (k=3). FlatEmbk=20 ToolsPRF1PRF1 5176.558.863.780.863.868.6 12069.652.857.479.261.666.3 18069.153.257.376.759.764.2 23066.551.255.174.957.962.3 32564.345.750.476.955.560.3 43067.039.645.873.947.653.1 53065.736.843.074.844.951.1 58462.534.740.677.445.151.9 Table 4: GPT-5.1 scaling results (implicit queries). Flat results are fold-averaged at intermediate scales. Embedding shortlisting on fold 0. FlatEmbk=20 ToolsPRF1PRF1 5179.161.866.380.464.268.6 12079.862.066.881.165.269.5 18076.760.864.679.765.869.3 23076.060.664.477.962.166.0 32570.851.356.279.060.364.4 43071.145.451.174.852.356.9 53069.841.948.175.850.255.3 58468.639.445.978.349.855.9 Table 5: Claude Sonnet 4.5 scaling results (implicit queries). Flat intermediate scales are fold-averaged (k=3); all other entries are fold 0. 100200300400500600 Tools in catalog 30 40 50 60 70 80 90 100 Recall on implicit queries (%) Precision Flat routing Platform tool search Embedding shortlist (k=20) Oracle (k=20) 100200300400500600 Tools in catalog Recall Figure 3: Precision (left) and recall (right) on implicit queries across scale. Precision is relatively stable across scale. Recall drives the degradation, dropping 15p even for the oracle. B Retriever Comparison Figure 4 isolates retrieval method and gran- ularity effects. Panel (a) compares four tool- level retrievers. BM25 and base e5-large-v2 both fall below flat routing, demonstrating that low-quality retrieval is counterproductive. Panel (b) shows that tool-level retrieval out- performs pack-level by 2–4p for both embed- ding models. C K-Sensitivity F1 plateaus at k≥10 tools and is statistically indistinguishable from k=20 to k=50 (boot- strap p=0.78 for k=20 vs. k=35).We fix k=20 for all scaling experiments as the cheap- est point on the plateau (93% embedding re- call at 3.4% of catalog). D Fixed-Cohort Validation To confirm degradation is not driven by harder queries entering the pool at larger scales, we track a fixed cohort of 731 implicit queries whose target tools are present at every scale point (fold 0). On this cohort, flat-routing F1 drops from 58.2% at 51 tools to 43.3% at 584 tools, a 14.9p degradation on iden- tical queries as the catalog grows. Embedding shortlisting partially recovers this (46.4% at 584 tools, +3.1p), confirming both the degra- dation and the recovery are genuine. E Pack-Level Approaches Platformtoolsearch. OpenAI’s namespace-based tool search exhibits a crossover effect: −6.3p at 10 agents, +7–9p from 30 agents onward.At small scale, namespace selection adds unnecessary indi- rection. At larger scale, namespace filtering reduces the option space. The ceiling near 60 agents reflects within-namespace tool selection degrading 14p, nearly double the namespace retrieval degradation. Hierarchical LLM routing. Two-stage routing (LLM selects pack, then routes within pack) achieves 47.9% F1 at full scale, below all other shortlisting approaches. The LLM se- lects only 1.2 packs on average (83% hit rate) despite a recall-oriented prompt, making the pack decision a source of unrecoverable error. F Positional Bias in Routing Embedding shortlisting returns candidates ranked by similarity, placing the correct tool near the top. To isolate the filtering bene- fit from any ranking advantage, we re-run all scale points with shuffled candidate order (per- query deterministic permutation). ConditionAvg ∆ vs flatPosition effect Ranked (default)+8.8p— Shuffled+6.8p2.0p Oracle ranked+25.8p— Oracle shuffled+22.0p3.8p Table 6: Positional bias decomposition (implicit F1, averaged across all scale points).Embed- ding ranking contributes ∼2p; oracle ranking contributes ∼4p because ground truth is always at rank 1. All deltas are statistically significant (paired bootstrap, p < 0.01). The router favors candidates presented ear- 100200300400500600 Tools in catalog 30 40 50 60 70 F1 on implicit queries (%) (a) Retriever comparison (tool-level) text-emb-3-large (k=20) ToolRet-e5 (k=20) e5-large-v2 base (k=20) BM25 (k=20) 100200300400500600 Tools in catalog (b) Tool-level vs. pack-level text-emb-3-large tool (k=20) text-emb-3-large pack (k=5) ToolRet-e5 tool (k=20) ToolRet-e5 pack (k=5) Figure 4: Retriever comparison across scale (GPT-5.4). (a) Tool-level retrievers ranked by end-to-end F1. General-purpose text-embedding-3-large outperforms fine-tuned ToolRet-e5 and base e5-large-v2. BM25 falls below flat routing (Figure 1) at all scales. (b) Same retriever, tool-level (solid) vs. pack-level (dotted). Tool-level consistently wins by 2–4p. lier in the tool list. For embedding shortlist- ing, this accounts for ∼2p of the total gain, stable across scale points. For oracle shortlist- ing, the effect is larger (∼4p) because ground truth is always at rank 1, meaning oracle up- per bounds overstate the routing ceiling.