Paper deep dive
FlexRouter: Learning Complementary Model Sets for Flexible LLM Routing
Wang Wei, Harry Yang, Tiankai Yang, Samyadeep Basu, Hongjie Chen, Andy Zhao, Franck Dernoncourt, Ryan A. Rossi, Hoda Eldardiry
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 10/1/2026, 8:38:32 AM
Summary
The paper introduces FlexRouter, a framework for routing queries to Large Language Models (LLMs) that explicitly models model complementarity using Determinantal Point Processes (DPPs). Unlike traditional methods that score models independently, FlexRouter optimizes for answer coverage by selecting diverse model subsets that minimize redundancy and maximize the probability of at least one correct response. It employs a training objective based on marginalizing over failure sets and uses an adaptive greedy inference strategy to determine subset sizes dynamically without a fixed computational budget.
Entities (8)
Relation Signals (6)
FlexRouter → evaluatedon → RouterEval
confidence 95% · Extensive experiments on the large-scale RouterEval benchmark demonstrate that our proposed FlexRouter achieves higher coverage
FlexRouter → optimizesfor → Answer Coverage
confidence 95% · FlexRouter optimizes for answer coverage, maximizing the probability that at least one selected model yields a correct response.
FlexRouter → uses → Determinantal Point Processes
confidence 95% · We formulate routing as a coverage-oriented subset selection problem and model the routing policy using Determinantal Point Processes (DPPs)
FlexRouter → outperforms → EmbedLLM
confidence 90% · FlexRouter achieves higher coverage with lower redundancy across both in-domain and out-of-domain tasks than strong baselines
FlexRouter → outperforms → BaRP
confidence 90% · FlexRouter achieves higher coverage with lower redundancy across both in-domain and out-of-domain tasks than strong baselines
FlexRouter → reduces → redundancy
confidence 90% · FlexRouter reduces redundancy in the selected subsets.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Existing Large Language Model (LLM) routing methods score LLMs independently to select top-$k$ models. However, this ignores model correlations and enforces a rigid computational budget. Consequently, routers often select redundant models that share failure modes, limiting the overall probability of success. To address this, we propose FlexRouter, a routing framework that explicitly models model complementarity. FlexRouter optimizes for \textit{answer coverage}, maximizing the probability that at least one selected model yields a correct response. This objective aligns with practical inference pipelines where multiple candidate outputs are generated and a downstream verifier or user selects the final one. We formulate routing as a coverage-oriented subset selection problem and model the routing policy using Determinantal Point Processes (DPPs), which naturally capture both model competence and redundancy. To directly optimize coverage without requiring a ground-truth target subset, we introduce a training objective based on marginalizing over failure sets. During inference, we employ a greedy strategy based on marginal log-determinant gains, enabling the router to adaptively determine subset sizes without a predefined budget. Extensive experiments on the large-scale RouterEval benchmark demonstrate that our proposed FlexRouter achieves higher coverage with lower redundancy across both in-domain and out-of-domain tasks than strong baselines while maintaining flexible inference cost.
Tags
Links
- Source: https://arxiv.org/abs/2609.38585v1
- Canonical: https://arxiv.org/abs/2609.38585v1
Trouble viewing inline? Open PDF directly →
Full Text
67,391 characters extracted from source content.
Expand or collapse full text
FlexRouter: Learning Complementary Model Sets for Flexible LLM Routing Wang Wei Affiliation: Virginia Tech Email: wangwei718@vt.edu Harry Yang Affiliation: Independent Researcher Email: hdardiry@vt.edu Tiankai Yang Affiliation: University of Southern California Samyadeep Basu Affiliation: Adobe Research Hongjie Chen Affiliation: Dolby Labs Andy Zhao Affiliation: University of Southern California Franck Dernoncourt Affiliation: Adobe Research Ryan A. Rossi Affiliation: Adobe Research Hoda Eldardiry †thanks: Corresponding author. Affiliation: Virginia Tech Abstract Existing Large Language Model (LLM) routing methods score LLMs independently to select top-k models. However, this ignores model correlations and enforces a rigid computational budget. Consequently, routers often select redundant models that share failure modes, limiting the overall probability of success. To address this, we propose FlexRouter, a routing framework that explicitly models model complementarity. FlexRouter optimizes for answer coverage, maximizing the probability that at least one selected model yields a correct response. This objective aligns with practical inference pipelines where multiple candidate outputs are generated and a downstream verifier or user selects the final one. We formulate routing as a coverage-oriented subset selection problem and model the routing policy using Determinantal Point Processes (DPPs), which naturally capture both model competence and redundancy. To directly optimize coverage without requiring a ground-truth target subset, we introduce a training objective based on marginalizing over failure sets. During inference, we employ a greedy strategy based on marginal log-determinant gains, enabling the router to adaptively determine subset sizes without a predefined budget. Extensive experiments on the large-scale RouterEval benchmark demonstrate that our proposed FlexRouter achieves higher coverage with lower redundancy across both in-domain and out-of-domain tasks than strong baselines while maintaining flexible inference cost. 1 Introduction Figure 1: Testing Success@10 score of baselines and FlexRouter on in-domain and out-of-domain tasks. Large Language Models (LLMs) are increasingly deployed as pools of heterogeneous models that differ in capability, specialization, and inference cost (Chen et al., 2023). A core problem in using LLMs systematically is routing: given an input query, which LLM(s) should be invoked to maximize answer quality under a compute budget. Routing is closely related to earlier methods such as classic conditional computation and mixture-of-experts, where a gating mechanism activates a small fraction of experts per input (Jacobs et al., 1991; Shazeer et al., 2017). In modern LLM serving, routing has become especially important since no single model dominates across all tasks (Srivatsa et al., 2024), and the candidate set can be large and rapidly evolving (Hu et al., 2024; Huang et al., 2025a). Question Answering Query: The car wash is only 100m away from my house, should I walk or drive? Top-k • Walk… • Walk… • Walk… Ours • Walk… • Drive… • Drive… Math reasoning Query: If you flip a fair coin twice, what is the expected number of heads Top-k • 11 • 11 • 11 Ours • 1 by enumerating outcomes • 1 by linearity of expectation • 1 by symmetry Coding Query: Write a Python palindrome checker (ignore space and case). Top-k • s == s[::-1] • s == s[::-1] • s == s[::-1] Ours • normalize + reverse • remove spaces + lowercase • s == s[::-1] Figure 2: Independent top-k routing often selects highly similar models, leading to redundant outputs and shared failure modes. FlexRouter instead selects complementary models that cover different interpretations, reasoning paths, and implementations, improving answer coverage across tasks. A common strategy in LLM routing is to assign each candidate model a predicted score and select the top-k models independently (Zhuang et al., 2025; Chen et al., 2024). Although simple and effective, this design overlooks an important characteristic of modern LLM systems: model correlation. Models trained with similar data or architectures often exhibit similar strengths and failure modes. As a result, independent selection of the highest-scoring models can lead to redundant selections that produce nearly identical outputs. When these outputs are incorrect, the entire subset fails, limiting the probability of obtaining at least one correct answer. As illustrated in Figure 2, SOTA routing models that do not take diversity into account tend to produce multiple similar responses that fail in the same way, while our approach that selects a diverse set of models yields varied reasoning paths and increases the likelihood that at least one response is correct. Such diversity is particularly important in real-world systems, where multiple candidate responses are generated and a downstream verifier, reranker, or human user selects the final answer. In these settings, the key objective is not that all selected models are similarly correct but that they could be diverse and at least one of them is correct. This does not by itself solve final answer selection, but it provides a necessary candidate pool for downstream selection. If all selected models fail, no downstream selector can recover the correct answer. Motivated by this observation, we formulate LLM routing as a coverage-oriented subset selection problem. Given an input query, our objective is to select a subset of models that maximizes the likelihood of obtaining at least one correct response while minimizing redundant selections. To achieve this, we propose FlexRouter, a routing framework that explicitly models both model competence and inter-model correlations. We parameterize the routing policy using Determinantal Point Processes (DPPs) (Han et al., 2017). DPPs naturally define a distribution over subsets, inherently favoring selections that are high-quality and diverse. Unlike independent scoring methods, DPPs penalize the joint selection of highly similar models, encouraging complementary model sets. A central challenge in this framework is learning under set-valued objectives (Dietterich et al., 1997; Cour et al., 2011). For any given query, there is generally no unique ground-truth subset, as any combination containing at least one successful model is acceptable. Because standard supervised learning requires fixed targets, we must instead optimize directly for answer coverage. To bridge this gap, we introduce a coverage-aligned training objective designed to maximize the probability of sampling a subset with at least one correct model. This objective is derived by marginalizing over failure sets, yielding a tractable objective in terms of DPP determinants. Figure 3: FlexRouter overview. For a query, the router encoder produces embeddings that define query-dependent competence for each LLM, while learned model embeddings capture pairwise similarity between models. These signals define a DPP kernel whose diagonal entries favor strong models and whose off-diagonals penalize redundant selections, thus encouraging individually strong yet mutually diverse model sets. Greedy selection with early stopping returns an adaptive-size subset. At inference time, FlexRouter performs greedy subset selection using marginal log-determinant gains (Nemhauser et al., 1978) and employs an adaptive stopping rule. This allows the router to dynamically determine the number of models to invoke for each query, allocating more resources to difficult queries while avoiding unnecessary computation on easier ones (Han et al., 2021). We evaluate FlexRouter on the large-scale RouterEval benchmark (Huang et al., 2025a), which provides extensive model–query performance records across a broad range of tasks. FlexRouter consistently achieves higher answer coverage than strong routing baselines with lower inference cost. Moreover, FlexRouter reduces redundancy in the selected subsets. These results highlight the value of explicitly modeling model correlations and performing adaptive subset selection for LLM routing at scale. Contributions. Our main contributions are: (i) we formulate LLM routing as a coverage-oriented subset selection problem that emphasizes the probability of obtaining at least one correct answer; (i) we propose FlexRouter, a DPP-based subset routing framework that jointly models model quality and redundancy to select complementary and diverse model sets; (i) we derive a tractable coverage training objective via failure-set marginalization that directly optimizes the probability of selecting at least one correct model; (iv) we develop an adaptive greedy inference strategy that selects variable-size subsets without requiring a fixed budget; (v) we demonstrate consistent improvements on RouterEval (Huang et al., 2025a), including higher coverage with lower redundancy under flexible inference cost. 2 Methodology 2.1 Problem Formulation Let ℳ=1,…,MM=\1,…,M\ be a set of available LLMs. For a given input query x, the objective of the router is to select a subset of models S⊆ℳS that maximizes the probability of generating a correct response while minimizing redundancy. We assume access to a dataset of query-response pairs for each model, where each query x is associated with a binary label vector y∈0,1My∈\0,1\^M. Here, yi=1y_i=1 if model i answers x correctly, and yi=0y_i=0 otherwise. For a specific query x, we partition the model space into a Correct Set CxC_x and a Failure Set FxF_x: Cx=i∈ℳ∣yi=1,Fx=i∈ℳ∣yi=0.C_x=\i y_i=1\, F_x=\i y_i=0\. (1) The routing task is successful if the selected subset S contains at least one model from CxC_x. Any subset that intersects CxC_x is considered successful under this objective, as it contains at least one correct model. We define the reward function as the indicator of coverage: R(x,S)=[S∩Cx≠∅]=1−∏i∈S(1−yi).R(x,S)=I[S∩ C_x≠ ]=1- _i∈ S(1-y_i). (2) 2.2 Complementary Model Sets Parameterization To capture both the individual capabilities of models and their pairwise correlations, we model the subset selection probability using a Determinantal Point Process (DPP) which defines a distribution over subsets that naturally favors high-quality yet non-redundant selections, aligning with the coverage-based routing objective. A DPP is fully parameterized by a positive semi-definite (PSD) L-ensemble matrix Lx∈ℝM×ML_x ^M× M. The probability of selecting a subset S is given by: Lx(S)=det((Lx)S)det(I+Lx),P_L_x(S)= ((L_x)_S) (I+L_x), (3) where (Lx)S(L_x)_S denotes the submatrix of LxL_x indexed by the elements of S, and I is the identity matrix. To enable end-to-end learning, we propose a complementary parameterization for the kernel LxL_x. This construction ensures the matrix is inherently PSD and explicitly models the trade-off between model performance and redundancy. Query and Model Embeddings. Let E(x)=vx∈ℝdE(x)=v_x ^d be the query embedding generated by a router encoder where E(⋅)E(·) is a shared text encoder that maps the input query into a dense representation. Each model i is assigned a learnable embedding ui∈ℝdu_i ^d representing its functional capabilities. We compute a scalar quality score qi(x)∈ℝ+q_i(x) ^+ for each model, representing the confidence that model i can answer query x. This is parameterized as: qi(x)=σ(vx⊤ui),q_i(x)=σ (v_x u_i ), (4) where qi(x)q_i(x) captures query-dependent model competence. We define a similarity matrix K∈ℝM×MK ^M× M based on the cosine similarity between model embeddings, which captures the static correlation between models (i.e., models with similar architectures or training data will have high similarity): Kij=ui⊤uj‖ui‖‖uj‖.K_ij= u_i u_j\|u_i\|\|u_j\|. (5) Kernel Construction. The final query-dependent kernel LxL_x is constructed as: Lx=diag(q(x))⋅K⋅diag(q(x))L_x=diag(q(x))·K·diag(q(x)) (6) Element-wise, this corresponds to (Lx)ij=qi(x)qj(x)Kij(L_x)_ij=q_i(x)q_j(x)K_ij. The diagonal entries (Lx)ii=qi(x)2(L_x)_i=q_i(x)^2 capture the individual quality of each model, while the off-diagonal entries penalize the joint selection of correlated models. Consequently, the determinant of a subset reflects both the quality and diversity of the selected models, as highly correlated models reduce the determinant. 2.3 Learn to Maximize Coverage The standard maximum likelihood estimate for DPPs assumes access to an observed target subset. In our setting, the supervision does not specify a unique optimal routing decision. Any subset that intersects the correct set CxC_x is considered successful. As a result, there is no single ground-truth subset to maximize likelihood against. Instead, we propose a Coverage Loss that directly maximizes the probability of obtaining at least one correct answer. Furthermore, in traditional DPPs, the observed supervision is a subset valued random variable Y⊆ℳY , which enables likelihood based learning via logPr(Y=S∗) (Y=S^*). In contrast, our setting observes only a binary correctness vector ∈0,1My∈\0,1\^M, which is not a realization of Y and therefore admits no valid DPP maximum likelihood objective. To efficiently compute the probability of success, we consider its complement, i.e., the probability that a sampled subset Y fails to cover the query (i.e., Y is entirely contained within the failure set FxF_x). This is given by the marginal probability of the failure set (see Appendix B, Proposition 3 for detailed analysis): P(Y⊆Fx)=∑S⊆FxLx(S)=det(I+(Lx)Fx)det(I+Lx).P(Y F_x)= _S F_xP_L_x(S)= (I+(L_x)_F_x) (I+L_x). (7) The probability of success (Hit) is therefore Psucc(x)=1−P(Y⊆Fx)P_succ(x)=1-P(Y F_x). We minimize the negative log-probability of success as: ℒhit(x)=−log(1−det(I+(Lx)Fx)det(I+Lx)).L_hit(x)=- (1- (I+(L_x)_F_x) (I+L_x) ). (8) Auxiliary Supervision. In addition to the coverage objective, we include a binary cross-entropy (BCE) loss on the predicted model-wise correctness scores qi(x)q_i(x) to provide direct supervision for individual model predictions. This auxiliary loss stabilizes training by guiding the encoder E(⋅)E(·) and the model embeddings ui\u_i\ to produce accurate estimates of per-model correctness. The overall training objective is: ℒ=ℒhit+λ∑i=1MBCE(qi(x),yi),L=L_hit+λ _i=1^MBCE(q_i(x),y_i), (9) where λ controls the strength of the auxiliary supervision. 2.4 Flexible Router Inference At inference time, we seek the subset S that maximizes the determinant that corresponds to selecting the most probable subset under the learned DPP. This is also known as MAP inference (Gillenwater et al., 2012). However, exact MAP inference is NP-hard (Civril and Magdon-Ismail, 2009). We employ a greedy algorithm with an adaptive stopping condition. We initialize S=∅S= . At each step, we select the model i∉Si∉ S that provides the maximal multiplicative gain to the determinant volume. The marginal gain gi(S)g_i(S) is defined as: gi(S)=det((Lx)S∪i)det((Lx)S).g_i(S)= ((L_x)_S∪\i\) ((L_x)_S). (10) Using the Schur complement, this can be computed efficiently as: gi(S)=(Lx)ii−(Lx)i,S[(Lx)S]−1(Lx)S,i.g_i(S)=(L_x)_i-(L_x)_i,S[(L_x)_S]^-1(L_x)_S,i. (11) Stopping Condition: Unlike top-k routing which enforces a fixed budget, FlexRouter dynamically determines the subset size. We stop adding models when the marginal gain gi(S)g_i(S) falls below a threshold τ≥0τ≥ 0 relative to the initial gain. Crucially, because the DPP log-determinant objective is non-monotone, adding highly correlated models can actually decrease the overall subset score. This adaptive stopping rule naturally stops selection when candidates offer no unique contributions (see Appendix B, Proposition 1-2 for formal proofs and intuition), allowing the router to adaptively balance coverage and computational cost. 3 Experiments We evaluate FlexRouter on large-scale LLM routing benchmarks to answer the following questions: (i) Does FlexRouter improve coverage over independent-scoring baselines? (i) Does modeling complementarity reduce redundancy in the selected model sets? (i) Can flexible routing reduce inference cost while maintaining high coverage? (iv) Does the learned routing policy generalize to unseen tasks? 3.1 Benchmarks and Baselines We evaluate FlexRouter on RouterEval (Huang et al., 2025b), a large-scale LLM routing benchmark that provides binary correctness labels for each query–model pair. We consider two settings with shared candidate pools: a medium-pool setting with 3811 candidate LLMs and a large-pool setting with 5000 candidate LLMs. Datasets. For the medium-pool setting, we train and evaluate in-domain on BBH (Suzgun et al., 2022), MATH (Hendrycks et al., 2021b), and GPQA (Rein et al., 2023), and test out-of-domain generalization on IFEval (Zhou et al., 2023) and MuSR (Sprague et al., 2024). For the large-pool setting, the in-domain tasks are MMLU (Hendrycks et al., 2021a), HellaSwag (Zellers et al., 2019), GSM8K (Cobbe et al., 2021), and ARC (Clark et al., 2018), while TruthfulQA (Lin et al., 2022) and WinoGrande (Sakaguchi et al., 2021) are used for out-of-domain evaluation. For in-domain evaluation, we report results on held-out 20% test splits of the training tasks. For out-of-domain evaluation, the router is trained only on the in-domain tasks and evaluated on unseen tasks. Details are in Table 7 and Appendix C.1. Baselines. We compare FlexRouter with strong routing baselines. Independent scoring ranks models by predicted correctness and selects the top-k candidates independently, without modeling inter-model correlations. EmbedLLM (Zhuang et al., 2025) learns query-dependent model scores in a shared embedding space, but still performs independent top-k selection. BaRP (Wei et al., 2025) formulates routing as a multi-objective contextual bandit and learns an adaptive policy under bandit feedback. We also report the Ref. score, which is the performance of a representative strong single model on each benchmark and serves as a single-model reference. Additional baseline details are deferred to Appendix C.2. For fixed-budget comparisons, all routing methods are evaluated with the same maximum selection budget. We also evaluate additional coverage- and diversity-oriented baselines, including Random-k, MaxDiversity, and MMR. We defer their definitions and results to Appendix D.2. 3.2 Evaluation Metrics Let Q denote the set of queries. We evaluate routing quality using Success@k, the fraction of queries for which at least one selected model is correct: Success@k=1||∑q∈[∃m∈q:yq,m=1].Success@k\;=\; 1|Q| _q 1\! [\,∃\,m _q:y_q,m=1 ]. (12) To measure redundancy, we report ILD@k (Intra-List Diversity), defined as the average pairwise cosine distance among the selected models in a fixed pretrained embedding space: ILD@k=1||∑q∈1(|q|2)∑i,j∈qi<j(1−i⊤j‖i‖‖j‖).ILD@k\;=\; 1|Q| _q 1 |S_q|2 _ subarrayci,j\,∈\,S_q\\ i<j subarray (1- u_i u_j\|u_i\|\,\|u_j\| ). (13) Higher ILD indicates less redundant and more complementary selections. Further metric details are provided in Appendix C.3. 3.3 Implementation Details FlexRouter uses a shared query encoder to map each query into a representation, and learnable model embeddings to represent candidate models. Queries are encoded using RoBERTa-base and projected into a shared 128-dimensional space, which is used to construct the query-dependent DPP kernel. The model is trained with Adam for up to 100 epochs. The training objective combines the coverage loss with an auxiliary binary cross-entropy loss with weight λ=1.0λ=1.0. At inference time, model selection is performed using a greedy MAP procedure based on marginal log-determinant gains, with adaptive stopping and a maximum subset size of k=10k=10. All experiments are conducted on NVIDIA A100 80GB GPUs. Details and computational complexity are provided in Appendix C.4. Downstream Answer Selection. FlexRouter focuses on candidate-pool construction rather than final answer selection. Its objective is to select a complementary subset of models such that at least one selected model is likely to produce a correct response for the given task. Therefore, Success@k should be interpreted as a routing-stage coverage metric, not as final deployed accuracy. In practice, the routed candidates can be passed to a task-specific verifier, reward model, LLM-as-judge reranker, self-consistency or aggregation method, or human user. However, these downstream mechanisms are not guaranteed to recover a lone correct response in every setting. A full end-to-end evaluation with a concrete selector is therefore complementary to our work and remains an important future direction. 3.4 Routing Performance Table 1: Success@10 on the medium-pool setting (3811 models). Ref. score is the reference average accuracy for each task of a representative LLM, such as GPT-4. Best results are bolded; second-best are underlined. In-Domain Out-of-Domain Method BBH MATH GPQA IFEval MuSR Avg Ref. score 0.830 0.400 0.397 0.769 0.699 0.619 EmbedLLM 0.9419 0.8264 0.7731 0.9298 0.6931 0.8329 BaRP 0.9124 0.7132 0.8445 0.9187 0.7712 0.8320 Ours 0.9471 0.8075 0.8487 0.9390 0.7738 0.8632 Table 2: Success@10 on the large-pool setting (5000 models). Best results are bolded; second-best are underlined. In-Domain Out-of-Domain Method MMLU HellaSwag GSM8K ARC TruthfulQA WinoGrande Avg Ref. score 0.864 0.953 0.920 0.852 0.669 0.875 0.855 EmbedLLM 0.9865 0.9776 0.9848 0.9957 0.9682 0.9684 0.9802 BaRP 0.9783 0.7979 0.9432 0.8889 0.8972 0.9795 0.9142 Ours 0.9907 0.9851 0.9886 0.9915 0.9951 0.9976 0.9914 Tables 1 and 2 report routing performance measured by Success@10 on the two RouterEval settings. Across both settings, FlexRouter achieves the highest average success rate, demonstrating the benefit of selecting complementary model subsets. Medium-pool setting results. The medium-pool setting (Table 1) contains fewer candidate models and several challenging reasoning benchmarks. FlexRouter achieves the best overall performance with an average Success@10 of 0.8632. The improvement is most pronounced on GPQA, where FlexRouter substantially outperforms both baselines. The method also achieves the highest performance on both out-of-domain tasks, suggesting that the learned routing policy generalizes well to unseen tasks. Large-pool setting results. On the large-pool setting (Table 2), FlexRouter achieves the best overall performance with an average Success@10 of 0.9914. The method obtains the highest score on five of the six tasks and performs particularly well on the out-of-domain benchmarks TruthfulQA and WinoGrande. These results indicate that explicitly modeling correlations between models helps identify complementary candidates and increases the likelihood that at least one selected model produces a correct answer. EmbedLLM achieves the highest score on MATH. This behavior likely reflects that MATH contains a small number of specialist models with strong performance, where selecting the highest-scoring models independently can already perform well. Nevertheless, FlexRouter consistently achieves the best average performance across tasks, indicating that modeling model complementarity provides a robust advantage across diverse benchmarks. Additional coverage analysis. To further analyze the behavior across smaller budgets and the composition of selected pools, Appendix D.1 reports Success@k curves, Avg-Correct@10, and Zero-Correct Rate. These analyses show that FlexRouter’s advantage appears in the multi-model setting and that it reduces all-wrong selected pools. 3.5 Subset Diversity Table 3: ILD@10 in the fixed pre-trained embedding space on the medium-pool setting. Higher is better. Best results are bolded. In-Domain Out-of-Domain Method BBH MATH GPQA IFEval MuSR Avg EmbedLLM 0.2114 0.1432 0.2196 0.2524 0.2295 0.2112 BaRP 0.1983 0.1993 0.1990 0.1966 0.1980 0.1982 Ours 0.7249 0.7515 0.7413 0.7895 0.8027 0.7620 Table 4: ILD@10 in the fixed pre-trained embedding space on the large-pool setting. Higher is better. Best results are bolded. In-Domain Out-of-Domain Method MMLU HellaSwag GSM8K ARC TruthfulQA WinoGrande Avg EmbedLLM 0.1302 0.2765 0.1775 0.4536 0.3747 0.2732 0.2810 BaRP 0.5520 0.5521 0.5521 0.5520 0.5521 0.5521 0.5521 Ours 0.4709 0.7052 0.6483 0.7653 0.7396 0.7516 0.6802 We next examine whether the selected model subsets exhibit higher diversity. Tables 3 and 4 report ILD@10, which measures the average pairwise cosine distance between embeddings of the selected models. Higher values indicate that the router selects models that are less redundant and more complementary. Across both settings, FlexRouter consistently produces the most diverse model subsets. On the large-pool setting (Table 4), FlexRouter achieves the highest average ILD@10 and obtains the best result on five of the six tasks. In particular, the diversity gains are most pronounced on ARC, TruthfulQA, and WinoGrande, where the selected models are substantially more dissimilar than those chosen by the baselines. Although BaRP produces relatively high diversity score on MMLU, its selections are nearly constant across tasks, suggesting that it relies on a fixed set of models rather than adapting to query-specific complementarity. The difference is even more pronounced on the medium-pool setting (Table 3), where FlexRouter achieves substantially higher ILD@10 on every task. In contrast, EmbedLLM and BaRP produce relatively low diversity scores, indicating that their routing strategies tend to select similar models. These results provide empirical evidence that FlexRouter effectively captures model complementarity during routing. By explicitly modeling correlations between models, the router selects subsets that are both high-quality and diverse, which directly supports the coverage improvements observed in Section 3.4. 3.6 Generalization to Unseen Tasks We next examine whether the learned routing policy generalizes to tasks that are not observed during training. Across both RouterEval settings, FlexRouter consistently achieves the best Success@10 on the out-of-domain benchmarks according to Tables 1 and 2, indicating that the routing strategy learned on the in-domain tasks transfers effectively to unseen evaluation settings. The diversity analysis further supports this observation. As shown in Tables 3 and 4, FlexRouter selects substantially more diverse model subsets on the out-of-domain tasks compared with both baselines. This behavior suggests that the router captures general patterns of model complementarity rather than memorizing task-specific preferences. Together, these results indicate that modeling correlations between models enables FlexRouter to construct complementary model subsets that remain effective even when routing queries from previously unseen tasks. 3.7 Ablation on Kernel Design The DPP kernel requires a similarity measure between model embeddings. Our default design uses cosine similarity, while an alternative is a Gaussian RBF kernel (as detailed in Appendix A) Sij=exp(−γ‖i−j‖2)S_ij= (-γ\|u_i-u_j\|^2) with a trainable bandwidth γ. Table 5 reports results on both RouterEval settings at k=10k=10. On the large-pool setting, the RBF kernel achieves slightly higher in-domain Success@10, while cosine performs better on the out-of-domain tasks and consistently produces more diverse model subsets. On the medium-pool setting, cosine achieves higher in-domain success and again selects more diverse subsets, while RBF attains slightly higher out-of-domain success. Overall, cosine similarity consistently yields higher diversity while maintaining competitive success rates across both settings. We therefore adopt cosine similarity as the default kernel for FlexRouter. Table 5: Kernel design ablation at k=10k=10. ILD is measured in the fixed embedding space. Setting Kernel ID Succ ID ILD OOD Succ OOD ILD Large-pool RBF 0.9920 0.6019 0.9954 0.6923 Cosine 0.9854 0.6474 0.9962 0.7456 Medium-pool RBF 0.8804 0.7109 0.8757 0.7883 Cosine 0.9058 0.7392 0.8486 0.7961 3.8 Adaptive Subset Size via Stopping Threshold τ FlexRouter allows adaptive control of inference cost through a stopping threshold τ in the greedy MAP procedure. A model is added only if its marginal log-determinant gain exceeds τ times the first gain, allowing the router to stop early when additional models provide limited benefit. Figure 4 illustrates the resulting coverage–cost trade-off on the combined in-domain test set of the large-pool setting (kmax=10k_ =10). When τ is small, the router selects nearly all candidate models and achieves the highest coverage. As τ increases, the average subset size decreases while Success@10 gradually declines. A favorable operating point appears around τ≈0.2τ≈ 0.2, where the router reduces the average subset size substantially while maintaining high coverage. Beyond this point, further reductions in subset size lead to a sharper drop in coverage. Exact numerical results are reported in Table 6. Figure 4: Success@10 and Average Subset Size over Stopping Threshold τ. When τ≈0.2τ≈ 0.2, the router reduces the average subset size substantially while maintaining high coverage. These results demonstrate that the stopping threshold provides a simple mechanism for controlling the coverage–cost trade-off. Importantly, the subset size varies across queries, indicating that FlexRouter adapts its inference budget to query difficulty rather than using a fixed number of models. 4 Limitations and Scope FlexRouter focuses on routing-stage candidate-pool construction rather than final answer selection. Success@k measures whether the selected subset contains at least one correct model, so it should not be interpreted as final deployed accuracy. The selected candidates may be passed to a verifier, reranker, aggregation method, or human user, but these downstream mechanisms are not guaranteed to recover a correct answer in every case. Jointly evaluating FlexRouter with concrete downstream selectors is an important future direction. Our diversity analysis measures model-level diversity using fixed model embeddings, which does not necessarily imply output-level semantic diversity for every query. Since RouterEval provides correctness labels but not full generated responses for all query–model pairs, response-level diversity and answer-agreement analysis are left for future work. Our cost analysis uses the number of invoked models, or average subset size, as a proxy for inference cost. Real deployment cost also depends on model-specific latency, token price, output length, batching, and hardware constraints. Extending FlexRouter to optimize such model-specific costs is a useful future direction. 5 Conclusion We presented FlexRouter, a coverage-oriented routing framework that selects complementary subsets of LLMs under flexible inference budgets. By modeling routing as a subset selection problem with determinantal point processes, FlexRouter explicitly accounts for model correlations and reduces redundant model selections. A coverage-aligned training objective enables learning without requiring a ground-truth target subset while an adaptive greedy inference procedure determines subset sizes on a per-query basis without enforcing a fixed budget. Experiments and ablation studies on the RouterEval benchmark demonstrate that FlexRouter achieves higher coverage than strong baselines while maintaining flexible inference cost and lower redundancy, highlighting the benefits of correlation-aware and adaptive subset selection for LLM routing at scale. Ethics Statement This work studies routing among existing LLMs and does not introduce new models or training data. As such, it inherits potential biases and risks present in the underlying models. By promoting diverse model selection, our method may surface a broader range of outputs, including both useful alternatives and undesirable content. In practice, it should be combined with downstream safeguards such as verification, filtering, or human oversight. Our approach may improve efficiency by reducing redundant model usage, but adaptive routing can also increase cost for difficult queries if not properly controlled. References Affandi et al. (2014) R. H. Affandi, E. Fox, R. Adams, and B. Taskar Learning the parameters of determinantal point process kernels. In International Conference on Machine Learning, p. 1224–1232. Cited by: Appendix E. Chen et al. (2023) L. Chen, M. Zaharia, and J. Zou Frugalgpt: how to use large language models while reducing cost and improving performance. arXiv preprint arXiv:2305.05176. Cited by: §1. Chen et al. (2024) S. Chen, W. Jiang, B. Lin, J. T. Kwok, and Y. Zhang RouterDC: query-based router by dual contrastive learning for assembling large language models. In Advances in Neural Information Processing Systems, Cited by: Appendix E, §1. Cho et al. (2019) S. Cho, C. Li, D. Yu, H. Foroosh, and F. Liu Multi-document summarization with determinantal point processes and contextualized representations. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, p. 98–103. Cited by: Appendix E. Civril and Magdon-Ismail (2009) A. Civril and M. Magdon-Ismail On selecting a maximum volume sub-matrix of a matrix and related problems. Theoretical Computer Science 410 (47-49), p. 4801–4811. Cited by: §2.4. Clark et al. (2018) P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457. Cited by: §C.1, §3.1. Cobbe et al. (2021) K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Cited by: §C.1, §3.1. Cour et al. (2011) T. Cour, B. Sapp, and B. Taskar Learning from partial labels. The Journal of Machine Learning Research 12, p. 1501–1536. Cited by: §1. Deng et al. (2020) Q. Deng, K. Wang, M. Zhao, Z. Zou, R. Wu, J. Tao, C. Fan, and L. Chen Personalized bundle recommendation in online games. In Proceedings of the 29th ACM international conference on information & knowledge management, p. 2381–2388. Cited by: Appendix E. Dietterich et al. (1997) T. G. Dietterich, R. H. Lathrop, and T. Lozano-Pérez Solving the multiple instance problem with axis-parallel rectangles. Artificial intelligence 89 (1-2), p. 31–71. Cited by: §1. Gillenwater et al. (2012) J. Gillenwater, A. Kulesza, and B. Taskar Near-optimal map inference for determinantal point processes. Advances in Neural Information Processing Systems 25. Cited by: §2.4. Han et al. (2017) I. Han, P. Kambadur, K. Park, and J. Shin Faster greedy map inference for determinantal point processes. In International Conference on Machine Learning, p. 1384–1393. Cited by: §1. Han et al. (2021) Y. Han, G. Huang, S. Song, L. Yang, H. Wang, and Y. Wang Dynamic neural networks: a survey. IEEE transactions on pattern analysis and machine intelligence 44 (11), p. 7436–7456. Cited by: §1. Hendrycks et al. (2021a) D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300. Cited by: §C.1, §3.1. Hendrycks et al. (2021b) D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt Measuring mathematical problem solving with the math dataset. External Links: 2103.03874, Link Cited by: §C.1, §3.1. Hu et al. (2024) Q. J. Hu, J. Bieker, X. Li, N. Jiang, B. Keigwin, G. Ranganath, K. Keutzer, and S. K. Upadhyay RouterBench: a benchmark for multi-LLM routing system. arXiv preprint arXiv:2403.12031. Cited by: §1. Huang et al. (2025a) Z. Huang, G. Ling, V. S. Liang, Y. Lin, Y. Chen, S. Zhong, H. Wu, and L. Lin RouterEval: a comprehensive benchmark for routing LLMs to explore model-level scaling up in LLMs. In Findings of the Association for Computational Linguistics: EMNLP 2025, Cited by: §1, §1, §1. Huang et al. (2025b) Z. Huang, G. Ling, Y. Lin, Y. Chen, S. Zhong, H. Wu, and L. Lin RouterEval: a comprehensive benchmark for routing LLMs to explore model-level scaling up in LLMs. In Findings of the Association for Computational Linguistics: EMNLP 2025, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng (Eds.), Suzhou, China, p. 3860–3887. External Links: Link, Document, ISBN 979-8-89176-335-7 Cited by: §C.1, §3.1. Jacobs et al. (1991) R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton Adaptive mixtures of local experts. Neural Computation 3 (1), p. 79–87. External Links: Document Cited by: §1. Jiang et al. (2023) D. Jiang, X. Ren, and B. Y. Lin Llm-blender: ensembling large language models with pairwise ranking and generative fusion. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 14165–14178. Cited by: Appendix E. Kulesza and Taskar (2012) A. Kulesza and B. Taskar Determinantal point processes for machine learning. Foundations and Trends® in Machine Learning. Cited by: Appendix E. Lin et al. (2022) S. Lin, J. Hilton, and O. Evans TruthfulQA: measuring how models mimic human falsehoods. External Links: 2109.07958, Link Cited by: §C.1, §3.1. Nemhauser et al. (1978) G. L. Nemhauser, L. A. Wolsey, and M. L. Fisher An analysis of approximations for maximizing submodular set functions—i. Mathematical programming 14 (1), p. 265–294. Cited by: §1. OpenAI et al. (2024) OpenAI, J. Achiam, S. Adler, S. Agarwal, et al. GPT-4 technical report. External Links: 2303.08774, Link Cited by: §C.2. Rein et al. (2023) D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman GPQA: a graduate-level google-proof q&a benchmark. External Links: 2311.12022, Link Cited by: §C.1, §3.1. Sakaguchi et al. (2021) K. Sakaguchi, R. Le Bras, C. Bhagavatula, and Y. Choi WinoGrande: an adversarial winograd schema challenge at scale. Communications of the ACM. Cited by: §C.1, §3.1. Shazeer et al. (2017) N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean Outrageously large neural networks: the sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: §1. Sprague et al. (2024) Z. Sprague, X. Ye, K. Bostrom, S. Chaudhuri, and G. Durrett MuSR: testing the limits of chain-of-thought with multistep soft reasoning. External Links: 2310.16049, Link Cited by: §C.1, §3.1. Srivatsa et al. (2024) K. A. Srivatsa, K. K. Maurya, and E. Kochmar Harnessing the power of multiple minds: lessons learned from LLM routing. In Proceedings of the Fifth Workshop on Insights from Negative Results in NLP, External Links: Document Cited by: §1. Suzgun et al. (2022) M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, and J. Wei Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261. Cited by: §C.1, §3.1. Wang et al. (2024) J. Wang, J. Wang, B. Athiwaratkun, C. Zhang, and J. Zou Mixture-of-agents enhances large language model capabilities. arXiv preprint arXiv:2406.04692. Cited by: Appendix E. Wang et al. (2022) X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171. Cited by: Appendix E. Wei et al. (2025) W. Wei, T. Yang, H. Chen, Y. Zhao, F. Dernoncourt, R. A. Rossi, and H. Eldardiry Learning to route llms from bandit feedback: one policy, many trade-offs. External Links: 2510.07429, Link Cited by: §C.2, Appendix E, §3.1. Wilhelm et al. (2018) M. Wilhelm, A. Ramanathan, A. Bonomo, S. Jain, E. H. Chi, and J. Gillenwater Practical diversified recommendations on youtube with determinantal point processes. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, p. 2165–2173. Cited by: Appendix E. Zellers et al. (2019) R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi Hellaswag: can a machine really finish your sentence?. arXiv preprint arXiv:1905.07830. Cited by: §C.1, §3.1. Zhou et al. (2023) J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, Y. Luan, D. Zhou, and L. Hou Instruction-following evaluation for large language models. External Links: 2311.07911, Link Cited by: §C.1, §3.1. Zhuang et al. (2025) R. Zhuang, T. Wu, Z. Wen, A. Li, J. Jiao, and K. Ramchandran EmbedLLM: learning compact representations of large language models. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §C.2, Appendix E, §1, §3.1. Appendix Appendix A Alternative Similarity Modeling Gaussian. We model the correlation between models using a Gaussian Radial Basis Function (RBF) kernel. This captures the intuition that models with similar embeddings (close in the functional space) are redundant. We define the similarity matrix K∈ℝM×MK ^M× M as: Kij=exp(−γ‖ui−uj‖22),K_ij= (-γ\|u_i-u_j\|_2^2 ), (1) where ui,uju_i,u_j are the learnable model embeddings and γ>0γ>0 is a trainable scaling parameter (inverse bandwidth). This formulation guarantees that KK is positive semi-definite and bounded between (0,1](0,1]. Appendix B Theoretical Results Proposition 1 (Submodularity of Log-Determinant). Let Lx≻0L_x 0 be a positive definite kernel matrix over ℳM. Define f(S)=logdet((Lx)S)f(S)= ((L_x)_S) for S⊆ℳS , where (Lx)S(L_x)_S denotes the principal submatrix of LxL_x indexed by S, and f(∅)=0f( )=0 by convention. Then f is submodular: for all S⊆T⊆ℳS T and i∉Ti∉ T, f(S∪i)−f(S)≥f(T∪i)−f(T).f(S∪\i\)-f(S)≥ f(T∪\i\)-f(T). (2) However, f is not monotone in general. The marginal gain of adding model i to set S is f(S∪i)−f(S)=log(Lx)ii⋅S,f(S∪\i\)-f(S)= (L_x)_i· S, (3) where (Lx)ii⋅S=(Lx)ii−(Lx)i,S[(Lx)S]−1(Lx)S,i(L_x)_i· S=(L_x)_i-(L_x)_i,S[(L_x)_S]^-1(L_x)_S,i is the Schur complement. This quantity is positive (since Lx≻0L_x 0) but may be less than one, in which case log(Lx)ii⋅S<0 (L_x)_i· S<0 and f decreases. Proof. For any S⊆ℳS and i∉Si∉ S, the determinant formula for bordered matrices gives det((Lx)S∪i)det((Lx)S)=(Lx)ii−(Lx)i,S[(Lx)S]−1(Lx)S,i=(Lx)ii⋅S. ((L_x)_S∪\i\) ((L_x)_S)=(L_x)_i-(L_x)_i,S[(L_x)_S]^-1(L_x)_S,i=(L_x)_i· S. (4) For S⊆TS T, the Schur complement satisfies (Lx)ii⋅T≤(Lx)ii⋅S(L_x)_i· T≤(L_x)_i· S, since conditioning on a larger set can only reduce the residual variance in a positive semi-definite matrix. Taking logarithms preserves the inequality, establishing submodularity. Since Lx≻0L_x 0, all Schur complements are strictly positive, so f(S∪i)−f(S)=log(Lx)ii⋅S>−∞f(S∪\i\)-f(S)= (L_x)_i· S>-∞. However, nothing prevents (Lx)ii⋅S<1(L_x)_i· S<1. ∎ The non-monotonicity of f(S)=logdet((Lx)S)f(S)= ((L_x)_S) has a natural interpretation in the routing context: adding a model that is highly correlated with the existing selection can reduce the overall determinant, reflecting the fact that redundant coverage does not improve the DPP objective. Proposition 2 (Greedy Determinant Maximality with Adaptive Stopping). Let Lx≻0L_x 0 and let f(S)=logdet((Lx)S)f(S)= ((L_x)_S) as above. Consider the greedy algorithm that initializes S=∅S= and iteratively selects i∗=argmaxi∉Sgi(S)=argmaxi∉S(Lx)ii⋅S,i^*= _i∉ S\;g_i(S)= _i∉ S\;(L_x)_i· S, (5) terminating when gi∗(S)≤1g_i^*(S)≤ 1 for the best remaining candidate. Let SgreedyS_greedy denote the output. Then for all T⊇SgreedyT S_greedy with T≠SgreedyT≠ S_greedy, det((Lx)T)≤det((Lx)Sgreedy). ((L_x)_T)≤ ((L_x)_S_greedy). (6) That is, SgreedyS_greedy globally maximizes det((Lx)S) ((L_x)_S) over all supersets of itself. Proof. When the algorithm terminates, gi(Sgreedy)≤1g_i(S_greedy)≤ 1 for all i∉Sgreedyi∉ S_greedy, meaning (Lx)ii⋅Sgreedy≤1(L_x)_i· S_greedy≤ 1. For any superset T=Sgreedy∪i1,…,irT=S_greedy∪\i_1,…,i_r\, the determinant telescopes as det((Lx)T)=det((Lx)Sgreedy)∏j=1r(Lx)ijij⋅Sgreedy∪i1,…,ij−1. ((L_x)_T)= ((L_x)_S_greedy) _j=1^r(L_x)_i_ji_j· S_greedy∪\i_1,…,i_j-1\. (7) By submodularity of f (Theorem 1), (Lx)ijij⋅Sgreedy∪i1,…,ij−1≤(Lx)ijij⋅Sgreedy≤1(L_x)_i_ji_j· S_greedy∪\i_1,…,i_j-1\≤(L_x)_i_ji_j· S_greedy≤ 1, so each factor in the product is at most 11, giving det((Lx)T)≤det((Lx)Sgreedy) ((L_x)_T)≤ ((L_x)_S_greedy). ∎ This shows that the greedy algorithm with adaptive stopping finds a set that no superset can improve upon. In particular, SgreedyS_greedy is a global maximizer of det((Lx)S) ((L_x)_S) over all sets S⊇SgreedyS S_greedy, including ℳM itself. Proposition 3 (Coverage-Diversity Equivalence). Let Cx⊆ℳC_x denote the correct set and Fx=ℳ∖CxF_x=M C_x the failure set. The hit probability decomposes as Psucc(x)=1−P(Y⊆Fx)=∑S⊆ℳS∩Cx≠∅Lx(S).P_succ(x)=1-P(Y F_x)= _ subarraycS \\ S∩ C_x≠ subarrayP_L_x(S). (8) Consequently, minimizing the coverage loss ℒhit(x)L_hit(x) is equivalent to maximizing the total DPP probability mass assigned to subsets that contain at least one correct model. Proof. By the law of total probability, 1=∑S⊆ℳLx(S)=∑S⊆ℳS∩Cx≠∅Lx(S)+∑S⊆ℳS∩Cx=∅Lx(S).1= _S P_L_x(S)= _ subarraycS \\ S∩ C_x≠ subarrayP_L_x(S)+ _ subarraycS \\ S∩ C_x= subarrayP_L_x(S). (9) The second sum equals P(Y⊆Fx)P(Y F_x) since S∩Cx=∅S∩ C_x= if and only if S⊆FxS F_x. Therefore, Psucc(x)=1−P(Y⊆Fx)=∑S⊆ℳS∩Cx≠∅Lx(S).P_succ(x)=1-P(Y F_x)= _ subarraycS \\ S∩ C_x≠ subarrayP_L_x(S). (10) Since ℒhit(x)=−logPsucc(x)L_hit(x)=- P_succ(x) and −log- is monotone decreasing, minimizing ℒhit(x)L_hit(x) is equivalent to maximizing Psucc(x)P_succ(x). ∎ Appendix C Experiment Details Table 6: Effect of the relative stopping threshold τ on coverage and inference cost (combined ID test, large-pool setting, kmax=10k_ =10). τ is expressed as a fraction of the first marginal log-determinant gain. Size Reduction and Coverage Drop are relative to τ=0τ=0. τ Avg Subset Size Success@10 Size Reduction Coverage Drop 0.000 10.00 0.9885 0.0 % 0.00 % 0.001 10.00 0.9885 0.0 % 0.00 % 0.010 10.00 0.9885 0.0 % 0.00 % 0.050 10.00 0.9885 0.0 % 0.00 % 0.100 9.95 0.9883 0.5 % 0.02 % 0.200 6.41 0.9772 35.9 % 1.13 % 0.300 4.25 0.9562 57.5 % 3.24 % 0.500 1.41 0.8828 85.9 % 10.57 % C.1 Benchmarks and Datasets We conduct experiments on RouterEval (Huang et al., 2025b), a large-scale benchmark designed for evaluating LLM routing methods. RouterEval aggregates multiple widely used reasoning and knowledge benchmarks and records the performance of thousands of candidate LLMs on each prompt. For every (query,model)(query,model) pair, the benchmark provides a binary correctness label indicating whether the candidate model answers the query correctly. This design enables systematic evaluation of routing algorithms under large candidate model pools. Our experiments cover two RouterEval settings: the medium-pool setting and the large-pool setting. The medium-pool setting contains 3811 candidate LLMs, while the large-pool setting contains 5000 candidate LLMs. Within each setting, all tasks share the same candidate model pool, enabling consistent comparison across routing methods. For the medium-pool setting, we use three tasks for in-domain training and evaluation: BBH (Suzgun et al., 2022), MATH (Hendrycks et al., 2021b), and GPQA (Rein et al., 2023). We evaluate out-of-domain generalization on two additional tasks that are not observed during training: IFEval (Zhou et al., 2023) and MuSR (Sprague et al., 2024). These tasks cover diverse reasoning settings including multi-step reasoning, mathematical reasoning, graduate-level question answering, instruction-following evaluation, and multi-step reasoning over structured contexts. For the large-pool setting, the in-domain tasks include MMLU (Hendrycks et al., 2021a), HellaSwag (Zellers et al., 2019), GSM8K (Cobbe et al., 2021), and ARC (Clark et al., 2018). Out-of-domain evaluation is conducted on TruthfulQA (Lin et al., 2022) and WinoGrande (Sakaguchi et al., 2021). These datasets span knowledge-intensive reasoning, commonsense inference, mathematical problem solving, and truthfulness evaluation. Table 7 summarizes the datasets used in our experiments. Following prior work, we evaluate FlexRouter both in-domain and out-of-domain. For the in-domain tasks, the router is evaluated on the held-out 20% test splits of the tasks used for training, measuring its ability to handle unseen instances from familiar tasks. For the out-of-domain tasks, the router is trained on the in-domain tasks but evaluated on additional tasks that are entirely excluded from training. This setup assesses the router’s ability to generalize to previously unseen tasks. Dataset Category #Prompts #LLMs BBH Complex Reasoning 5761 3811 MATH Mathematical Reasoning 1324 3811 GPQA Graduate-level QA 1192 3811 IFEval Instruction Following 541 3811 MuSR Multi-step Reasoning 756 3811 MMLU Knowledge 14042 5000 HellaSwag Commonsense Reasoning 10042 5000 GSM8K Mathematical Reasoning 1319 5000 ARC Science QA 1172 5000 TruthfulQA Truthfulness Evaluation 817 5000 WinoGrande Commonsense Reasoning 1267 5000 Table 7: Statistics of datasets used in our work. Each dataset consists of prompts evaluated across a shared pool of candidate LLMs within each setting. C.2 Baselines We compare FlexRouter with several strong routing baselines that represent commonly used model selection strategies using Independent scoring, which ranks candidate models according to predicted correctness scores and selects the top-k models with the highest scores. Each model is evaluated independently, and correlations between models are not considered during selection. This setting represents a common strategy used in many routing and ensemble systems where models are chosen solely based on their estimated accuracy. EmbedLLM (Zhuang et al., 2025) learns query-dependent scores by jointly embedding queries and candidate models into a shared representation space. The router predicts the likelihood that each model will answer a query correctly based on these embeddings. However, model selection is still performed independently for each candidate, without explicitly modeling correlations between models. BaRP (Wei et al., 2025) is a learned routing method that models routing as a multi-objective contextual bandit problem. It uses REINFORCE on bandit feedback to learn a routing policy that balances performance and cost when selecting models. BaRP serves as a strong learned baseline for adaptive model selection under resource constraints. Reference model (Ref. score). The Ref. score corresponds to the performance of a representative high-capacity LLM evaluated directly on the benchmark without routing. This reference model is not a routing method and does not perform model selection. Instead, it reflects the performance of a single strong model such as GPT-4 (OpenAI et al., 2024). The reference score provides a useful point of comparison because it represents the outcome of always relying on a single capable model. For all baselines, we evaluate performance under identical inference budgets. Each routing method selects the same number of candidate models per query, ensuring that performance differences reflect the quality of the routing strategy rather than differences in computational cost. C.3 Evaluation Metrics We evaluate routing performance using metrics that capture both success probability and model diversity. Success@k measures the fraction of queries for which the selected subset contains at least one model that produces a correct answer. This metric directly reflects the routing objective of maximizing the probability that at least one selected model succeeds. Success@k=1||∑q∈[∃m∈q:yq,m=1].Success@k\;=\; 1|Q| _q 1\! [\,∃\,m _q:y_q,m=1 ]. (11) Here Q denotes the set of queries, qS_q is the subset of models selected for query q, and yq,m∈0,1y_q,m∈\0,1\ indicates whether model m answers query q correctly. ILD@k (Intra-List Diversity) measures the average pairwise cosine distance among the embeddings of the selected models. Larger values indicate lower redundancy and greater complementarity among the selected models. ILD@k=1||∑q∈1(|q|2)∑i,j∈qi<j(1−i⊤j‖i‖‖j‖),ILD@k\;=\; 1|Q| _q 1 |S_q|2 _ subarrayci,j\,∈\,S_q\\ i<j subarray (1- u_i u_j\|u_i\|\,\|u_j\| ), (12) where mu_m denotes the embedding of model m. The metric ranges from 00 to 22, where 00 indicates that all selected models are identical in the embedding space and 22 indicates that model embeddings are maximally dissimilar. We compute ILD only for queries with at least two selected models. To ensure fair comparison across routing methods, the embeddings mu_m are taken from a fixed pre-trained embedding space derived from model metadata. As a result, ILD reflects model diversity independently of each method’s internal scoring mechanism. C.4 Implementation Details FlexRouter uses a shared query encoder together with learnable model embeddings to parameterize the routing policy. Queries are encoded using RoBERTa-base, and both query and model representations are projected into a shared latent space with dimension 128. The resulting representations are used to construct the query-dependent DPP kernel that defines the subset selection distribution. The model is trained using the Adam optimizer with a learning rate of 10−310^-3. Training proceeds for up to 100 epochs with early stopping based on validation Success@k, using a patience of 15 epochs. The training objective combines the coverage-oriented objective and an auxiliary binary cross-entropy loss. The weight for auxiliary supervision is set to λ=1.0λ=1.0 for all experiments. During inference, FlexRouter selects models using a greedy maximum a posteriori procedure based on marginal log-determinant gains of the DPP objective. Model selection proceeds iteratively and stops when the marginal gain becomes non-positive or when the maximum subset size k=10k=10 is reached. This adaptive stopping rule allows the router to adjust the number of selected models according to query difficulty while maintaining a bounded inference budget. All experiments were conducted on NVIDIA A100 80GB GPUs. While the kernel is of size M×M× M, greedy inference only requires computing marginal gains with respect to the current subset and maintaining a Cholesky factorization, resulting in O(k2M)O(k^2M) complexity. In practice, we further restrict evaluation to a top-K0K_0 candidate pool for efficiency. Appendix D Additional Results The results in this section are from additional runs conducted for further analysis. They follow the same evaluation protocol as the main experiments, but may differ slightly from Tables 1-2 due to stochastic training and rerunning the models. The qualitative trends remain consistent. D.1 Additional Coverage Analysis The main experiments report Success@10 because all routing methods are evaluated under the same maximum budget kmax=10k_ =10. Since RouterEval contains thousands of candidate LLMs in both the medium-pool and large-pool settings, selecting 10 models is still a compact subset of the full candidate pool. To further examine whether the advantage of FlexRouter only appears at k=10k=10, we rerun the experiments and report average Success@k across different budgets in Table 8. FlexRouter trails EmbedLLM at k=1k=1, which is expected because the method is not optimized for selecting a single strongest model. However, FlexRouter overtakes independent selection once routing becomes genuinely multi-model, e.g., from k=3k=3 in the medium-pool setting and from k=2k=2 in the large-pool setting. This supports the intended use case of FlexRouter: constructing a compact candidate pool whose selected models cover complementary strengths rather than redundantly selecting similar high-scoring models. Pool composition. Success@k measures whether the selected pool contains at least one correct model, but it does not show how many correct models are selected or whether improvements come from reducing all-wrong pools. We therefore report two additional pool-composition metrics. Avg-Correct@10 measures the average number of correct models in the selected pool, while Zero-Correct Rate measures the fraction of queries for which all selected models are incorrect. The results are shown in Table 9. The results show a trade-off between correct-candidate density and candidate-pool coverage. EmbedLLM obtains slightly higher Avg-Correct@10, but it also leaves more queries with zero correct candidates. In contrast, FlexRouter reduces the Zero-Correct Rate while maintaining a high number of correct models in the selected pool. Compared with EmbedLLM, FlexRouter reduces the Zero-Correct Rate from 0.160 to 0.128 in the medium-pool setting and from 0.022 to 0.008 in the large-pool setting. This suggests that independent top-k selection may concentrate correct models on some queries while leaving more queries completely uncovered, whereas FlexRouter reduces the number of all-wrong selected pools. Table 8: Average Success@k across different model-call budgets. FlexRouter is designed for complementary subset construction rather than single-model routing. It becomes stronger once the setting allows multiple models to be selected. Setting k FlexRouter EmbedLLM BaRP Medium-pool 1 0.417 0.542 0.527 Medium-pool 3 0.709 0.707 0.639 Medium-pool 5 0.789 0.765 0.732 Medium-pool 7 0.832 0.805 0.784 Medium-pool 10 0.872 0.840 0.844 Large-pool 1 0.700 0.750 0.665 Large-pool 2 0.928 0.925 0.822 Large-pool 5 0.975 0.956 0.878 Large-pool 10 0.992 0.978 0.916 Table 9: Pool-composition analysis at k=10k=10. Avg-Correct@10 measures the average number of correct models in the selected subset, while Zero-Correct Rate measures the fraction of queries for which the selected subset contains no correct model. Setting Method Avg-Correct@10 ↑ Zero-Correct Rate ↓ Success@10 ↑ Medium-pool EmbedLLM 5.17 0.160 0.840 Medium-pool BaRP 4.51 0.156 0.844 Medium-pool FlexRouter 4.90 0.128 0.872 Large-pool EmbedLLM 7.94 0.022 0.978 Large-pool BaRP 6.64 0.084 0.916 Large-pool FlexRouter 7.49 0.008 0.992 D.2 Coverage-Oriented Baselines The main experiments compare FlexRouter with strong routing baselines, including EmbedLLM and BaRP. However, these methods mainly score candidate models independently and are not explicitly designed to optimize Success@k. To further test whether FlexRouter’s gains come from the DPP coverage objective rather than from adding any diversity heuristic, we compare with three additional coverage- and diversity-oriented baselines. Random-k. Random-k uniformly selects k models from the candidate pool. MaxDiversity. MaxDiversity greedily selects models to maximize pairwise embedding diversity. We initialize the selected set with the highest predicted-quality model and then greedily add the model that maximizes the minimum pairwise distance to the already selected models. MMR. MMR greedily balances predicted model quality and redundancy. At each step, it selects i∗=argmaxi∉Sαqi(x)−(1−α)maxj∈Ssim(i,j),i^*= _i∉ Sα q_i(x)-(1-α) _j∈ Ssim(i,j), (13) where qi(x)q_i(x) is the predicted quality score and sim(i,j)sim(i,j) is cosine similarity in the fixed model-embedding space. This baseline is closest to a diversity-aware top-k heuristic, since it applies a post-hoc pairwise penalty to independently predicted quality scores. Table 10: Coverage-oriented baselines across medium- and large-pool settings. FlexRouter outperforms Random-k, MaxDiversity, and MMR across different budgets. Setting k FlexRouter MMR MaxDiversity Random-k Medium-pool 3 0.709 0.706 0.689 0.592 Medium-pool 5 0.789 0.757 0.754 0.697 Medium-pool 10 0.872 0.842 0.837 0.806 Large-pool 3 0.954 0.921 0.915 0.745 Large-pool 5 0.975 0.942 0.935 0.812 Large-pool 10 0.992 0.961 0.950 0.874 FlexRouter consistently outperforms the coverage-oriented baselines in both RouterEval settings. The comparison with MMR is especially informative: MMR applies diversity as a post-hoc penalty to independently predicted quality scores, while FlexRouter jointly learns query-dependent model quality and redundancy through the DPP coverage objective. These results suggest that the improvement is not simply due to adding a generic diversity heuristic, but comes from learning complementarity under the coverage-oriented subset objective. Table 11: Supervision ablation in the medium-pool setting. FlexRouter remains effective when trained with fewer labeled queries. Training Queries MATH BBH GPQA IFEval MuSR Avg Success@10 25% 0.785 0.943 0.836 0.963 0.772 0.860 50% 0.793 0.941 0.845 0.954 0.792 0.865 100% 0.793 0.946 0.878 0.963 0.838 0.884 D.3 Supervision Ablation RouterEval provides dense binary correctness labels for each query–model pair. To study how much query-level supervision FlexRouter needs, we train on 25%, 50%, and 100% of the available training queries in the medium-pool setting, while evaluating all checkpoints on the same held-out test set. For each retained query, we keep all model-level correctness labels. Therefore, this ablation measures query-level supervision efficiency rather than model-level label sparsity. FlexRouter degrades gracefully as the number of labeled training queries decreases. With only 25% of the training queries, it achieves 0.860 average Success@10, within 2.4 percentage points of the full-supervision setting. With 50% of the training queries, it achieves 0.865 average Success@10, within 1.9 percentage points of the full-supervision setting. This suggests that FlexRouter can learn useful quality-diversity structure without using the full query pool. This experiment studies query-level label sparsity. It does not address model-level label sparsity, where only a subset of candidate models is labeled for each query. That setting is closer to matrix completion or bandit-style exploration and remains an important direction for scalable routing. Appendix E Related Work LLM Routing. Routing across LLMs has emerged as a practical approach for improving performance under heterogeneous model capabilities and costs. A common strategy is to predict model-wise correctness scores and select the top-k models independently. Methods such as EmbedLLM (Zhuang et al., 2025) learn joint representations of queries and models to estimate query-dependent performance. Other recent approaches, such as RouterDC (Chen et al., 2024) and BaRP (Wei et al., 2025), formulate routing as representation learning or contextual bandit problems. However, these methods typically score models independently and do not explicitly account for correlations or shared failure modes, which can lead to redundant selections. Multi-Model Ensembles and Verification. Our coverage-oriented objective aligns with multi-generation LLM pipelines where a downstream verifier or ranker selects the final answer. This paradigm is widely used in ensemble methods like LLM-Blender (Jiang et al., 2023), which ranks outputs from multiple diverse models, and self-consistency frameworks (Wang et al., 2022) that aggregate multiple reasoning paths. More recently, frameworks like Mixture-of-Agents (Wang et al., 2024) have demonstrated that layering multiple LLMs can significantly enhance generation quality through collaborative refinement. While these methods benefit from diverse candidate pools, they typically rely on fixed, manually curated model sets. FlexRouter automates and optimizes the construction of these complementary candidate sets on a per-query basis. Determinantal Point Processes. Determinantal Point Processes (Kulesza and Taskar, 2012) have been widely used for subset selection tasks that require balancing quality and diversity, including document summarization (Cho et al., 2019), recommendation (Wilhelm et al., 2018), and information retrieval (Affandi et al., 2014; Deng et al., 2020). A DPP defines a distribution over subsets where the determinant of a kernel matrix captures both item quality and pairwise diversity. Prior work has explored learning DPP kernels from data and applying DPPs to subset selection problems with fixed ground-truth subsets. In contrast, our setting involves weak supervision where no single optimal subset is observed. We therefore design a coverage-oriented objective that directly optimizes the probability of selecting at least one correct model, while leveraging the diversity-inducing properties of DPPs to model complementarity among LLMs.