Paper deep dive
ConsRoute:Consistency-Aware Adaptive Query Routing for Cloud-Edge-Device Large Language Models
Haoyu Qiao, Hao Zhang, Shanwen Mao, Siyao Cheng, Jie Liu
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/26/2026, 2:25:28 AM
Summary
ConsRoute is a lightweight, semantic-aware, and adaptive query routing framework for cloud-edge-device collaborative LLM inference. It improves inference efficiency by using a reranker to assess semantic consistency between model tiers, reusing hidden states from the LLM prefilling stage to minimize device-side overhead, and employing Bayesian optimization for dynamic, cluster-specific routing thresholds.
Entities (4)
Relation Signals (3)
ConsRoute â optimizes â Inference Efficiency
confidence 95% · ConsRoute, a lightweight, semantic-aware, and adaptive routing framework that significantly improves inference efficiency
ConsRoute â uses â Bayesian Optimization
confidence 95% · Bayesian optimization is employed to learn cluster-specific routing thresholds
ConsRoute â deployedin â Cloud-Edge-Device Collaborative Inference
confidence 90% · ConsRoute, a consistency-aware query routing mechanism for cloudâedgeâdevice collaborative LLM inference
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large language models (LLMs) deliver impressive capabilities but incur substantial inference latency and cost, which hinders their deployment in latency-sensitive and resource-constrained scenarios. Cloud-edge-device collaborative inference has emerged as a promising paradigm by dynamically routing queries to models of different capacities across tiers. In this paper, we propose ConsRoute, a lightweight, semantic-aware, and adaptive routing framework that significantly improves inference efficiency while minimizing impact on response quality. Unlike prior routing methods that rely on predicting coarse-grained output quality gaps, ConsRoute leverages a reranker to directly assess the semantic consistency between responses generated by models at different tiers, yielding fine-grained soft supervision signals for routing. To minimize device-side overhead, ConsRoute reuses hidden states from the LLM prefilling stage as compact query representations, avoiding additional encoders or inference passes. Furthermore, these representations are clustered, and Bayesian optimization is employed to learn cluster-specific routing thresholds that dynamically balance quality, latency, and cost under heterogeneous query distributions. Extensive experiments demonstrate that ConsRoute achieves near-cloud performance (>=95%) while reducing end-to-end latency and inference cost by nearly 40%, consistently outperforming existing routing baselines in both response quality and system efficiency.
Tags
Links
- Source: https://arxiv.org/abs/2603.21237v1
- Canonical: https://arxiv.org/abs/2603.21237v1
Trouble viewing inline? Open PDF directly â
Full Text
81,954 characters extracted from source content.
Expand or collapse full text
ConsRoute: Consistency-Aware Adaptive Query Routing for CloudâEdgeâDevice Large Language Models Haoyu Qiao, Hao Zhang, Shanwen Mao, Siyao Cheng, and Jie Liu This work is partly supported by the Project of Laboratory of Advanced Agricultural Sciences of Heilongjiang Province under Grant No. ZY04JD05-010, the Key Research and Development Program of Heilongjiang Province under Grant No. 2022ZX01A22, and the National Natural Science Foundation of Heilongjiang Province under Grant No. YQ2019F007. (Corresponding author: Hao Zhang.)Haoyu Qiao, Zhang Hao, Shanwen Mao, and Siyao Cheng are with the Department of Computer Science and Technology, Harbin Institute of Technology, Harbin 150001, China and also with the National Key Laboratory of Smart Farm Technologies and Systems, Harbin 150001, China (e-mail:24s103393@stu.hit.edu.cn; zhh1000@hit.edu.cn; 24s103313@stu.hit.edu.cn;csy@hit.edu.cn).Jie Liu is with the Department of Computer Science and Technology, Harbin Institute of Technology (ShenZhen), ShenZhen 518055, China and also with the National Key Laboratory of Smart Farm Technologies and Systems, Harbin 150001, China (e-mail: jieliu@hit.edu.cn). Abstract Large language models (LLMs) deliver impressive capabilities but incur substantial inference latency and cost, which hinders their deployment in latency-sensitive and resource-constrained scenarios. Cloudâedgeâdevice collaborative inference has emerged as a promising paradigm by dynamically routing queries to models of different capacities across tiers. In this paper, we propose ConsRoute, a lightweight, semantic-aware, and adaptive routing framework that significantly improves inference efficiency while minimizing impact on response quality. Unlike prior routing methods that rely on predicting coarse-grained output quality gaps, ConsRoute leverages a reranker to directly assess the semantic consistency between responses generated by models at different tiers, yielding fine-grained soft supervision signals for routing. To minimize device-side overhead, ConsRoute reuses hidden states from the LLM prefilling stage as compact query representations, avoiding additional encoders or inference passes. Furthermore, these representations are clustered, and Bayesian optimization is employed to learn cluster-specific routing thresholds that dynamically balance quality, latency, and cost under heterogeneous query distributions. Extensive experiments demonstrate that ConsRoute achieves near-cloud performance (â„ 95%) while reducing end-to-end latency and inference cost by nearly 40%, consistently outperforming existing routing baselines in both response quality and system efficiency. â publicationid: pubid: 0000â0000/00$00.00 © 2021 IEEE I Introduction Large language models (LLMs) have demonstrated remarkable capabilities across a wide spectrum of tasks, including natural language understanding, reasoning, and generation [3, 2, 27]. These advances have spurred growing interest in deploying LLM-powered services in mobile and ubiquitous computing environments, such as personal assistants, intelligent sensing, and edge intelligence applications. However, bringing LLMs into such resource-constrained settings exposes a fundamental and increasingly critical tension: while larger models generally offer superior response quality and robustness, they incur prohibitive inference latency, computation, and energy costs; conversely, smaller models are significantly more efficient but often suffer from degraded reasoning and generation quality. This trade-off is particularly pronounced in mobile and IoT systems, where device-side computation, memory capacity and battery lifetime are all limited and dynamically varying. As a result, no single LLM instance can simultaneously satisfy the diverse and often conflicting requirements of real-world mobile applications, including high response quality, low end-to-end latency, and low device-side resource consumption. Addressing this challenge is therefore of both practical importance and fundamental relevance to mobile computing systems, motivating the design of collaborative inference frameworks that coordinate multiple LLMs across the device, edge, and cloud to jointly optimize quality, latency, and cost. Figure 1: Cloud-Edge-Device LLM Collaboration System A widely adopted paradigm to realize such collaboration is hierarchical deployment across the cloudâedgeâdevice continuum [36, 24]. As illustrated in Figure 1, a large-scale LLM is deployed in the cloud (CLM), e.g., 30B + parameter models such as DeepSeek-V3 [17] or Qwen3-32B [33], providing high-quality responses for complex or high-risk queries at the expense of higher latency and communication overhead. At the edge, a medium-scale LLM (ELM), typically in the 7â30B range (e.g., Qwen3-14B [33]), serves as a middle ground, offering a more favorable balance between capability and cost. On the device, a lightweight LLM (DLM), often with fewer than 4B parameters (e.g., Qwen3-1.7B [33]), is co-located with a query router to provide low-latency and energy-efficient responses for simple or low-risk requests. By selectively invoking different model tiers, this architecture enables adaptive trade-offs between performance and efficiency in dynamic mobile environments. At the core of this hierarchical architecture lies query routing, which determines whether a given request should be processed locally on the device, offloaded to the edge, or escalated to the cloud [28][14]. An effective routing decision hinges on whether the response generated by a lower-tier model is sufficiently consistent with that of a more capable model. If high consistency can be anticipated, the system can safely rely on the device-side model, thereby minimizing latency and resource usage; otherwise, the query should be forwarded to a higher-tier model to ensure response quality. Consequently, accurately predicting cross-model response consistency before generation is a key enabler for efficient and reliable collaborative inference. Early work, such as FrugalGPT [4] and AutoMix [1], adopts a cascading strategy that sequentially invokes models from smallest to largest, escalating only when the confidence of the smaller modelâs output falls below a predefined threshold. While conceptually simple, this approach incurs additional latency due to the need to generate full responses at multiple levels. More recent methods, including Zooter [20], RouteLLM [23], and OptLLM [19], attempt to mitigate this overhead by making routing decisions prior to generation, typically via learned predictors that estimate the expected quality or utility of candidate models. These predictive routing approaches enable faster model selection and have demonstrated improved efficiency. Despite these advances, existing routing methods still suffer from several fundamental limitations. First, many approaches rely on offline-obtained scalar quality or reward scores and use the difference between small and large models as supervision for router training. Mapping rich, structured text outputs to a single scalar inevitably discards fine-grained semantic information and fails to capture nuanced inconsistencies between model responses. Second, query representations are often extracted using fine-tuned BERT-style encoders or external embedding APIs, which either impose substantial on-device computation overhead or introduce additional communication latency, making them ill-suited for real-time mobile scenarios. Third, most existing methods employ a single, global threshold to determine routing decisions. Such static thresholds cannot adapt to heterogeneous query types, task sensitivities, or evolving data distributions, often resulting in suboptimal trade-offs between quality and efficiency. To address these challenges, we propose ConsRoute, a lightweight, consistency-aware query routing framework for cloudâedgeâdevice collaborative LLM inference. ConsRoute advances the state of the art in three key aspects. First, inspired by re-ranking techniques in information retrieval, we directly measure the semantic gap between model outputs in a high-dimensional space, constructing routing supervision signals based on textual semantic similarity rather than indirect scalar rewards. This provides a more faithful characterization of cross-model consistency. Second, we design a lightweight consistency predictor that reuses the DLMâs prefill hidden states, avoiding additional text encoding. By appending a short prompt after the query and extracting the final-layer hidden state of the last token, we obtain a compact semantic representation that is fed into a small MLP for routing, significantly reducing on-device latency and resource consumption. Third, we introduce a dynamic thresholding mechanism based on query clustering and Bayesian optimization, enabling routing policies to adapt to different query categories and system conditions in an online and data-driven manner. Our contributions are summarized as follows: âą We propose ConsRoute, a consistency-aware query routing mechanism for cloudâedgeâdevice collaborative LLM inference, which leverages semantic similarity between model outputs and efficiently reuses DLM prefill hidden states to enable fast and lightweight on-device routing. âą We design a dynamic, online-updatable thresholding strategy based on query clustering and Bayesian optimization, allowing routing decisions to adapt to heterogeneous query types and evolving workloads. âą We integrate ConsRoute into a hierarchical collaborative inference system and demonstrate through extensive experiments that it achieves response quality comparable to large cloud models, while reducing end-to-end latency and inference cost by nearly 40%. The remainder of this paper is organized as follows. Section I reviews related work on cloudâedgeâdevice collaborative LLM inference and multi-LLM query routing. Section I formalizes the cloudâedgeâdevice collaborative inference setting, including the hierarchical architecture, routing policy, and optimization objective. Section IV presents the proposed ConsRoute framework in detail, covering prompt-guided semantic representation extraction, semantic-consistency supervision, and cluster-based adaptive thresholding with online Bayesian optimization. Section V reports extensive experimental results on RouterBench and additional benchmarks, together with ablation studies, online adaptation analysis, and network condition sensitivity. Section VI concludes the paper and discusses future research directions. I Related Work TABLE I: Comparison of representative multi-LLM methods. Columns indicate whether a method targets cloudâedgeâdevice deployment (For Cloud-Edge-Device Deployment), performs pre-generation model choice (Predictive Routing), avoids additional query encoders beyond the LLMs (No-Extra Encoder), reuses device-side LLM internal states for routing (Reuse DLM info), employs non-global or task-aware thresholds (Adaptive Threshold), and supports online updating of the routing policy (Online Router updating). â denotes support, Ă denotes lack of support, and âââ denotes not applicable. Method For Cloud-Edge-Device Predictive No-Extra Reuse Adaptive Online Deployment Routing Encoder DLM info Threshold Router updating VELO [34] â â â Ă â â Mixture of Thought [35] Ă Ă â â â Ă FrugalGPT [4] Ă Ă Ă Ă Ă Ă AutoMix [1] Ă Ă â â â Ă HybridLLM [7] Ă â Ă Ă Ă Ă Zooter [20] Ă â â Ă â Ă RouteLLM [23] Ă â Ă Ă Ă Ă ME-Switch [18] Ă â Ă Ă â Ă MixLLM [29] Ă â Ă Ă â â ConsRoute (ours) â â â â â â I-A CloudâEdgeâDevice Collaborative LLM Inference Collaborative inference across cloud, edge and device has become a key paradigm for deploying large models in mobile and ubiquitous computing scenarios [8][16]. These systems aim to jointly optimize end-to-end latency, resource utilization, and quality of service (QoS) by carefully deciding where to execute each request under constrained computation, bandwidth, and energy budgets. Instead of running a single monolithic model in the cloud, they exploit the heterogeneous capabilities of cloud servers, edge nodes, and end devices to deliver more responsive and cost-effective services. He et al. [10] propose an active-inference-based offloading framework for LLM tasks in cloudâedge environments, addressing the data inefficiency, latency insensitivity, and poor adaptability to workload shifts observed in prior deep reinforcement learning solutions. VELO [34] introduces a vector-database-assisted framework that leverages multi-agent reinforcement learning (MARL) to optimize QoS for edge LLM users, reducing both latency and resource consumption while significantly improving user satisfaction. Hao et al. [9] proposes a dynamic token-level Edge-Cloud collaboration for LLM Inference. These works exemplify how intelligent scheduling and resource allocation can substantially improve the performance of LLM services in distributed infrastructures. CE-LSLM [39] introduced a key-value (KV) cache reuse mechanism to enhance the semantic understanding of edge models through contextual guidance from the cloud, while significantly reducing edge-side computational and storage overhead I-B Query Routing for Multi-LLM Systems Beyond macro-level task offloading and resource management, an essential research direction in multi-LLM systems is query routing: deciding, for each incoming request, which model tier to invoke under accuracy, latency, and cost constraints. In hierarchical cloudâedgeâdevice architectures, the router is the core component that bridges user queries and heterogeneous models, and its behavior directly determines the overall efficiency and reliability of the system. Existing work explores a variety of routing paradigms, including cascaded inference strategies that escalate from small to large models, learned routers trained under different supervision signals derived from model outputs, and diverse architectural choices and online adaptation mechanisms. In the following, we review these lines of work. I-B1 Cascaded Inference Strategies Early approaches to multi-LLM collaboration often adopt cascaded inference strategies that sequentially invoke models of increasing capacity and cost. The central idea is to start from a cheaper model, estimate its confidence in the generated response, and only escalate to a larger model when confidence is deemed insufficient. This design aims to trade off quality and cost without requiring complex predictors or upfront routing models. Mixture of Thought [35] treats the self-consistency of multiple samples from a weaker LLM as a signal of problem difficulty, proposing several sampling and consistency-checking schemes. While this improves over naive single-pass decoding, it still relies on the modelâs own confidence and does not address overconfidence in small models; moreover, sampling multiple full-length responses significantly increases latency. FrugalGPT [4] arranges multiple LLMs into a cost-ordered chain and uses a DistilBERT classifier [25] to predict the correctness of each intermediate answer; if the predicted correctness is low, the router escalates to the next, more expensive model. AutoMix [1] models the escalation process as a partially observable Markov decision process (POMDP), allowing a small model to evaluate the confidence of its own outputs and decide whether to invoke a higher-tier model. Although cascaded strategies can effectively reduce average cost under certain workloads, they are inherently non-predictive and sequential: the system must wait for the small model to generate (possibly multiple) responses before making a routing decision. This design introduces additional routing latency and is particularly problematic in mobile and edge settings, where end-to-end responsiveness and energy efficiency are critical. I-B2 Supervision Signals for Query Routing The choice of supervision signal is fundamental to the performance of a learned query router, as it determines which property of model behavior the router is trained to approximate. Most prior methods construct labels by evaluating and comparing the quality of outputs from different models, effectively using some notion of quality difference as the target for training. HybridLLM [7] trains a DeBERTa-based router to predict the quality gap between a small language model (SLM) and a large language model (LLM), where the gap is defined as the difference in BartScore between their outputs. Zooter [20] uses a reward model (QwenRM [2]) to assign utility scores to model responses and distills it into a lightweight classifier that makes routing decisions based on expected utility; it further performs label augmentation by combining per-query rewards with cluster-level average rewards. RouteLLM [23] leverages human preference data from Chatbot Arena, training routers to estimate model preferences across queries using several learning strategies. In all these methods, the supervision is derived from scalar quality, preference, or reward scores, and routing labels are defined by comparing such scores across models. The core idea behind these methods is to evaluate the quality of the outputs of different models and then use the difference in quality as a supervision signal. While effective in many scenarios, this approach of mapping rich textual information to a one-dimensional quality or reward score and then calculating the difference between them inevitably loses information about fine-grained semantic relationships between outputs and may fail to capture fine-grained semantic consistency between responses. I-B3 Router Architectures and Online Adaptation Beyond supervision signals, existing work also differs in how the query router itself is architected and how it adapts to changing workloads or constraints. A router must both understand query semantics sufficiently to estimate model performance and remain lightweight enough for deployment in resource-constrained mobile or edge environments. Achieving this balance between expressiveness and efficiency is a central design challenge in cloudâedgeâdevice LLM systems. OptLLM [19] formulates routing as a multi-objective optimization problem that jointly considers accuracy and cost, employing random forests and ensemble voting to estimate whether each LLM can produce a correct answer and to output confidence scores for different models. ME-Switch [18] concatenates the query with a routing prompt and feeds it into a fine-tuned Qwen1.5-1.8B model, using the prompted routerâs explicit output to select a model. MetaLLM [22] casts routing as a multi-armed bandit problem that adaptively balances performance and cost through online exploration and exploitation. MixLLM [29] introduces label-augmented embeddings and online optimization to dynamically control routing under latency constraints, improving robustness to workload variations. InferenceDynamics [26] proposes a flexible and scalable multi-dimensional routing framework that explicitly models the capability and knowledge profiles of a large pool of specialized LLMs. These methods highlight rich design spaces for router architectures and online adaptation, but they typically treat the router as a separate, often heavyweight module that does not reuse internal states of the underlying LLMs. This separation leads to significant additional computational and memory overhead, making it less suitable for on-device deployment and difficult to achieve a balance between accurate query understanding and lightweight implementation. Table I summarizes the characteristics of the most representative methods. We compare them with our proposed method ConsRoute, with detailed explanations provided in Appendix B. The comparison highlights that ConsRoute is, to the best of our knowledge, the only approach that simultaneously targets cloudâedgeâdevice deployment, avoids additional encoders by reusing DLM states, and supports adaptive, online-updated routing thresholds, making it particularly suitable for resource-constrained mobile settings. I Problem Setting In this section, we formalize the cloudâedgeâdevice collaborative inference problem considered in this work. We first describe the hierarchical deployment of LLMs across different tiers. We then define queries, responses, and the routing policy that determines where each query is served. Finally, we introduce the performance metrics of interest and formulate the routing objective. I-A Hierarchical CloudâEdgeâDevice Architecture We consider a hierarchical architecture composed of three tiers: the device tier, the edge tier, and the cloud tier. Each tier hosts a LLM with different capacity, latency, and resource requirements, referred to as the DLM, ELM, and CLM, respectively. The set of available serving options is denoted by âł=DLM,ELM,CLM.M=\DLM,ELM,CLM\. User queries are generated at the device tier. Upon the arrival of each query, the system can either process it locally using the DLM or offload it to the ELM or CLM for inference. We focus on the design of a routing policy that dynamically selects an appropriate serving tier from âłM for each incoming query, aiming to balance response quality, latency, and computational cost. I-B Query, Response, and Routing Policy Let X denote the space of user queries, and let xâx represent a query sampled from an underlying workload distribution. For each model mââłm , we denote by ymâ(x)y_m(x) the textual response generated by model m when serving query x. A routing policy determines which tier is responsible for answering each query. Formally, we model the router as a policy Ï:ââł,Ï:X , which maps a query x to a selected serving option Ïâ(x)ââłÏ(x) . Given a fixed routing policy Ï, the system behavior is fully determined: each query x is always served by the model m=Ïâ(x)m=Ï(x), incurring the corresponding quality, latency, and cost characteristics of that tier. I-C Performance Metrics Under a routing policy Ï, the system induces a distribution over how queries are assigned to different model tiers. This distribution, in turn, determines the overall answer quality, end-to-end latency, and computational cost of the system. We denote by Accâ(Ï)Acc(Ï) the expected answer quality under policy Ï, measured as the average correctness of responses over all served queries. We denote by Latencyâ(Ï)Latency(Ï) the expected end-to-end latency, which includes both computation time and communication delays associated with offloading. Finally, we denote by Costâ(Ï)Cost(Ï) the expected inference cost under policy Ï. Following [30], we estimate Costâ(Ï)Cost(Ï) based on the total number of activated model parameters multiplied by the number of generated tokens. Together, these metrics provide a compact yet expressive characterization of the trade-offs induced by different routing strategies. Figure 2: Overview of the ConsRoute framework. The bottom right shows the Semantic Representation Extractor, which leverages the DLM to extract input semantics (Section IV-B). The bottom left shows the Training Data Construction process for the predictor and the top right shows the Lightweight Consistency Predictor, which decides which model tier a query should be routed to (Section IV-C). The top left presents the Adaptive Routing Policy, where appropriate routing thresholds are determined via Bayesian optimization (Section IV-D). I-D Optimization Objective The objective of the routing policy is to preserve, as much as possible, the answer quality achieved by always invoking the strongest model (i.e., the CLM), while significantly reducing inference latency and computational cost by routing suitable queries to lower-tier, more efficient models. This naturally leads to a multi-objective optimization problem over the space of routing policies. A convenient way to express this goal is through a scalar utility function Uâ(Ï)=fâ(Accâ(Ï),Latencyâ(Ï),Costâ(Ï)),U(Ï)=f (Acc(Ï),Latency(Ï),Cost(Ï) ), where fâ(â )f(·) is a monotone function that increases with answer quality and decreases with latency and cost (e.g., a weighted combination of these three terms). The optimal routing policy is then defined as ÏââargâĄmaxÏâĄUâ(Ï).Ï â _ÏU(Ï). In the remainder of the paper, we instantiate this abstract formulation with a concrete, lightweight routing framework that leverages device-side signals to approximate favorable trade-offs between response quality, latency, and cost in practical cloudâedgeâdevice inference settings. IV ConsRoute: Consistency-Aware Query Routing for CloudâEdgeâDevice LLMs In this section, we first outline the overall workflow of our proposed consistency-guided query routing framework and summarize the key questions that drive its design. We then provide a detailed description of each component and answer these questions. Table I lists the main mathematical notations used in the subsequent sections. IV-A Overview of ConsRoute TABLE I: Main Notations Used in Our Method. Item Description DLM Large Language Model on the Device tier. ELM Large Language Model on the Edge tier. CLM Large Language Model on the Cloud tier. x User query. p Fixed natural-language instruction prompt. xâČx Concatenated input sequence xâČ:=xââpââ[EOS]x :=x\,\|\,p\,\|\,[EOS]. hTh_T Consistency-aware representation extracted from the DLM for xâČx . frankf_rank Pre-trained reranker used to compute semantic similarity between responses. SIMâ(x)SIM(x) Reranker-based semantic similarity between DLM and stronger model outputs. AUGâ(x)AUG(x) Augmented supervision signal from references or an LLM judge. Scloudâ(x)S_cloud(x) Consistency label for the DLMâCLM response pair. Sedgeâ(x)S_edge(x) Consistency label for the DLMâELM response pair. Sfusedâ(x)S_fused(x) Fused soft label combining Scloudâ(x)S_cloud(x) and Sedgeâ(x)S_edge(x). fΞf_Ξ Lightweight MLP head that predicts consistency from hTh_T. S^â(x) S(x) Predicted consistency score S^â(x):=fΞâ(hT)â[0,1] S(x):=f_Ξ(h_T)â[0,1]. kC_k k-th semantic cluster of queries in representation space. ckc_k Centroid of cluster kC_k. (Ï1(k),Ï2(k))( _1^(k), _2^(k)) Cluster-specific routing thresholds for DLM/ELM/CLM. kU_k Cluster-level utility that trades off accuracy, latency, and cost. uâ(x)u(x) Per-query utility sample used for online threshold adaptation. λ1,λ2,λ3 _1, _2, _3 Weights balancing accuracy, latency, and cost in the utility. We propose ConsRoute, a lightweight consistency-aware query routing framework designed for hierarchical LLM deployment across the device, edge, and cloud tiers. The core idea is to route each query to the cheapest model whose response is expected to remain semantically consistent with that of a stronger model, thereby preserving answer quality while substantially reducing latency and inference cost. As summarized in Figure 2, ConsRoute operates in two phases: an offline supervision construction phase (left) and an online routing phase (right). In the offline phase, we construct semantically grounded consistency labels that directly measure the agreement between responses generated by models at different tiers, which are used to train a lightweight on-device predictor and to initialize cluster-specific routing thresholds. In the online phase, as detailed in Algorithm 1, ConsRoute reuses the device-side LLMâs prefilling computation to extract a consistency-aware semantic representation of each query, predicts its expected consistency with stronger models, and applies adaptive, cluster-specific thresholds to determine whether the query should be served locally, escalated to the edge, or forwarded to the cloud. This design allows ConsRoute to jointly address three key challenges in multi-tier LLM deployment: (i) enabling accurate routing decisions without incurring additional on-device inference overhead, (i) supervising routing using semantic consistency rather than coarse scalar quality gaps, and (i) adapting routing policies to heterogeneous query types and evolving system conditions. Concretely, ConsRoute consists of four key components: 1. Consistency Label Construction: Offline, we build soft consistency labels by comparing the device-side answer with the edge-side and cloud-side answers using a reranker model, and then lightly augment these similarity scores with correctness and alignment signals from references or an LLM judge. These labels are used as supervision for the router. 2. Prompt-Guided Representation Extraction: On the device, the DLM processes the user query concatenated with a fixed instruction during the prefilling stage and produces a task-aware hidden representation at the last token. This representation is encouraged to encode how consistent the DLMâs answer would be with stronger models, while avoiding any extra encoder or additional forward pass. 3. Lightweight Consistency Prediction: A compact MLP head is then trained to map the DLM representation to a predicted consistency score, using the offline consistency labels as regression targets. This yields an efficient on-device predictor that can estimate deviceâedgeâcloud agreement without running reward models, rerankers, or external APIs at inference time. 4. Cluster-Based Adaptive Thresholding: Finally, we cluster queries in the representation space and assign each cluster its own routing thresholds, which are tuned by Bayesian optimization over a utility function that jointly considers accuracy, latency, and cost. During deployment, these cluster-specific thresholds are further refined online as traffic patterns and network conditions evolve, enabling adaptive trade-offs between quality and efficiency. Algorithm 1 Consistency-Guided Hierarchical Routing with Adaptive Thresholds 0: Query x, prompt p, frozen DLM, trained predictor fΞf_Ξ, cluster centroids ckk=1K\c_k\_k=1^K, thresholds (Ï1(k),Ï2(k))k=1K\( _1^(k), _2^(k))\_k=1^K (learned by Algorithm 2) 0: Selected model tier âDLM,ELM,CLMâ\DLM,ELM,CLM\ 1: Construct input xâČâxââpââ[EOS]x â x\,\|\,p\,\|\,[EOS] 2: Extract semantics hTâEncodeâ(xâČ;DLM)h_T (x ;DLM) 3: Predict routing score S^â(x)âfΞâ(hT) S(x)â f_Ξ(h_T) 4: Assign cluster index kââargâĄminkâĄâhTâckâ2k^*â _k\|h_T-c_k\|_2 5: Retrieve thresholds (Ï1,Ï2)â(Ï1(kâ),Ï2(kâ))( _1, _2)â( _1^(k^*), _2^(k^*)) 6: if S^â(x)>Ï1 S(x)> _1 then 7: return DLM 8: else if S^â(x)>Ï2 S(x)> _2 then 9: return ELM 10: else 11: return CLM 12: end if In the following, we detail each component of ConsRoute by answering the following research questions: âą RQ1: How can we extract a consistency-aware semantic representation for routing by reusing the device-side LLM, without introducing additional encoders or incurring noticeable on-device overhead? âą RQ2: How can we construct reliable and semantically grounded supervision signals that accurately capture response consistency across device-, edge-, and cloud-level LLMs? âą RQ3: How can we design routing thresholds that adapt to heterogeneous queries and evolving workloads, while achieving favorable trade-offs between answer quality, latency, and inference cost? IV-B Semantic Representation (RQ1) Motivation During the prefilling stage of generation, the DLM has already processed the full input query and computed deep hidden representations for all tokens. This provides a valuable opportunity to directly reuse these hidden states for semantic representation extraction. To make the representation task-aware, we lightly guide the DLM using a fixed natural language prompt appended to the query. Prompt-Guided Representation To obtain a consistency-aware representation without deploying an additional encoder, we let the DLM itself produce a task-specific embedding through a prompt, as illustrated in Figure 3. Figure 3: Prompt-guided representation learning. The user query is concatenated with a fixed instruction and an EOS token.The DLM processes the input and the final-layer hidden state of the EOS token is used as a consistency-aware representation for routing. Given a user query x, we first append a fixed natural-language instruction p and then terminate the sequence with an end-of-sequence token [EOS]. The resulting input token sequence is xâČ:=xââpââ[EOS],x :=x\,\|\,p\,\|\,[EOS], (1) where the instruction prompt is âThe consistency between the small language model and large language model responses for the above query is:â Figure 4: Comparison of consistency prediction signals.The left and middle plots show the relationship between score differences (LLM vs. DLM) from a reward model (Qwen2.5-PRM-7B) and BartScore, respectively, and human-annotated consistency labels. The right plot shows the same analysis using a reranker model (Qwen3-reranker-4B). The reranker score exhibits a stronger linear correlation with human labels, suggesting it better reflects true semantic consistency between responses. This instruction explicitly asks the model to consider the agreement between its own answer and that of a stronger LLM, nudging the internal representation to encode consistency-relevant information rather than only generic semantics. We feed xâČx into the DLM and take the final-layer hidden state of the EOS token as the semantic representation (Line 2 in Algorithm 1): hT:=DLMâ(xâČ)â[EOS],h_T:=DLM(x )[EOS], (2) where DLMâ(xâČ)DLM(x ) denotes the sequence of hidden states at the last transformer layer and [EOS] indexes the hidden state corresponding to the EOS token. This EOS-pooling scheme follows the design of recent instruction-based LLM embedding models, in which a short natural-language instruction and a special pooling token are used to obtain task-aligned sentence representations. The resulting vector hTh_T is then consumed by the router to predict the DLMâLLM consistency score and decide whether the query should be answered by the DLM or forwarded to the LLM. Importantly, the computation of hTh_T is fully integrated into the DLMâs prefilling stage: if the router decides to keep the query on the DLM, the keyâvalue cache produced when encoding xâČx can be directly reused for decoding. This eliminates redundant forward passes, avoids additional encoders (e.g., BERT-base) or external embedding APIs, and thus reduces both memory footprint and on-device latency. IV-C Lightweight Consistency Predictor (RQ2) IV-C1 Motivation Existing routing methods usually rely on reward models or scalar scores that measure the gap between generated content and ground-truth answers, which may fail to capture semantic differences between responses. Conceptually, these approaches first map rich, high-dimensional textual semantics into a one-dimensional quality or reward score for each model, and then compute the difference of these scalar scores as the supervision signal. This two-step compression discards much of the fine-grained relational information between model outputs: both the DLM and CLM may receive similarly high scores while still conveying different factual content, so their score difference remains small even when their responses are semantically inconsistent. As a result, quality-gapâbased supervision can systematically overlook semantic inconsistencies and lead to suboptimal routing decisions. As shown in Figure 5, we use a representative example to illustrate this problem. Figure 5: Example illustrating the quality gap fails to reveal the semantic inconsistency. The reward model gives similar scores to both responses, while the reranker identifies their semantic mismatch. The example in Figure 5 shows that even if two responses differ significantly in terms of compliance with instructions and content consistency, the reward model may still assign similar high scores. This motivates us to shift from quality-based signals to semantic consistency. We evaluate how closely the DLM output aligns with that of a stronger model, thus directly supervising the routing signal based on meaning preservation. To support multi-tier routing, we construct consistency supervision signals for both DLMâCLM and DLMâELM response pairs. Since the construction procedures are identical for both cases, we describe the details using the DLMâCLM setting as a representative. IV-C2 Semantic Similarity as Consistency Supervision To quantify semantic consistency, we adopt a pre-trained reranker model frankf_rank, specifically, the Qwen3-Reranker-4B [37]. While rerankers are typically used to score and reorder a list of retrieved candidates based on their relevance to a given query [31], we repurpose it here to measure the semantic agreement between responses generated by different models. Given a query x and two responses ysy_s and yly_l from the DLM and CLM respectively, the reranker outputs a similarity score: SIMâ(x):=frankâ(ys,yl).SIM(x):=f_rank(y_s,y_l). (3) This score captures whether the DLM response preserves the core semantics of the CLM response and serves as the primary supervision signal for training a consistency predictor. Empirical analysis validates the effectiveness of semantic similarity as a training signal. As shown in Figure 4, the reranker-based score exhibits significantly higher agreement with human-labeled consistency judgments than traditional metrics like BartScore [7] or reward models [20]. This indicates that semantic similarity is a more reliable target for consistency-aware routing. IV-C3 Data Augmentation with Additional Signals While reranker-based similarity scores provide strong signals for semantic alignment, they may not fully capture all aspects of response consistency, particularly in ambiguous or under-specified cases. Moreover, relying solely on a single scoring model may introduce bias or reduce label diversity. To improve robustness and coverage, we incorporate complementary supervision signals via data augmentation: âą With references: When the ground-truth yây^* is available for a query x, we apply rule-based supervision. If the DLM response ysy_s is incorrect while the CLM response yly_l is correct (based on answer matching), we assign a hard consistency label AUGâ(x)=0AUG(x)=0; otherwise, AUGâ(x)=1AUG(x)=1 (including cases where both responses are correct, or both are incorrect, or DLM is correct while CLM is not). âą Without references: For queries without ground-truth, we introduce an advanced LLM as a judge to estimate the semantic agreement between the two responses. Specifically, we use a powerful open-ended LLM (DeepSeek V3) fjudgef_judge, providing it with the query x, the two responses ysy_s and yly_l, and a prompt asking whether the two responses convey the same meaning. The model returns a soft agreement score: AUGâ(x):=fjudgeâ(x,ys,yl)â[0,1].AUG(x):=f_judge(x,y_s,y_l)â[0,1]. (4) These are combined into a unified consistency labels: Sâ(x):=αâ SIMâ(x)+(1âα)â AUGâ(x),S(x):=α·SIM(x)+(1-α)·AUG(x), (5) where a tunable weight αâ[0,1]αâ[0,1] controls the balance between general semantic similarity and augmented judgment. IV-C4 Predictor Training To support three-tier routing, we construct fused labels combining DLMâELM and DLMâCLM consistency: Sfusedâ(x):=ÎČâ Scloudâ(x)+(1âÎČ)â Sedgeâ(x),S_fused(x):=ÎČ· S_cloud(x)+(1-ÎČ)· S_edge(x), (6) where Scloudâ(x)S_cloud(x) and Sedgeâ(x)S_edge(x) denote the consistency labels for DLMâCLM and DLMâELM pairs respectively, and ÎČâ[0,1]ÎČâ[0,1] controls their relative importance. We use hTh_T extracted from DLM as input to a lightweight MLP head fΞf_Ξ, producing the predicted consistency score: S^â(x):=fΞâ(hT)â[0,1]. S(x):=f_Ξ(h_T)â[0,1]. (7) The model is trained to minimize the mean squared error (MSE) loss against the fused soft label: âmse=âS^â(xi)âSfusedâ(xi)â2.L_mse=\| S(x_i)-S_fused(x_i)\|^2. (8) IV-D Adaptive Threshold via Bayesian Optimization (RQ3) Motivation Due to the heterogeneity of queries, fixed routing thresholds Ï1 _1 and Ï2 _2 may not generalize well. In practice, query complexity, sensitivity, or task type vary greatly. Some queries, such as simple open-ended questions, can be handled relatively tolerantly by DLM or ELM even with low consistency scores; while others, such as mathematical problems, are more risky and require the highest response quality even if the prediction consistency appears high. This difference suggests that a one-size-fits-all threshold strategy may lead to suboptimal routing decisions in heterogeneous environments. To address this challenge, we adopt an adaptive threshold selection mechanism based on Bayesian optimization [21]. Instead of using a single global pair of thresholds, we assign distinct routing thresholds (Ï1(k),Ï2(k))( _1^(k), _2^(k)) for each category of queries, allowing the system to better reflect query-specific preferences and risk profiles. Algorithm 2 summarizes this cluster-based threshold learning procedure and its online adaptation. Algorithm 2 Cluster-Based Threshold Optimization with Online Bayesian Adaptation 0: Historical queries xi\x_i\ with representations hi\h_i\, utility kâ(Ï1,Ï2)U_k( _1, _2), offline BO budget ToffT_off, online update interval M 0: Centroids ckk=1K\c_k\_k=1^K, thresholds (Ï1(k),Ï2(k))k=1K\( _1^(k), _2^(k))\_k=1^K 1: // Offline clustering and threshold learning 2: Determine K via the elbow method and run K-means on hi\h_i\ to obtain clusters kk=1K\C_k\_k=1^K and centroids ckk=1K\c_k\_k=1^K 3: for k=1,âŠ,Kk=1,âŠ,K do 4: Initialize kââ D_kâ 5: for t=1,âŠ,Tofft=1,âŠ,T_off do 6: Fit GP surrogate fkf_k on kD_k 7: Select (Ï1,Ï2)( _1, _2) by maximizing EI under Ï1>Ï2 _1> _2, Ï1,Ï2â[0,1] _1, _2â[0,1] 8: Evaluate uâkâ(Ï1,Ï2)u _k( _1, _2) on kC_k and update kâkâȘ((Ï1,Ï2),u)D_k _kâȘ\(( _1, _2),u)\ 9: end for 10: (Ï1(k),Ï2(k))âargâĄmax((Ï1,Ï2),u)âkâĄu( _1^(k), _2^(k))â _(( _1, _2),u) _ku 11: end for 12: // Online adaptation (streaming phase) 13: Initialize global counter tâ0tâ 0 14: for each incoming query x do 15: tât+1tâ t+1 16: Obtain h and predicted score S^â(x) S(x) using Algorithm 1, Lines 1â4 17: Assign cluster kââargâĄminkâĄâhâckâ2k^*â _k\|h-c_k\|_2 and route x with (Ï1(kâ),Ï2(kâ))( _1^(k^*), _2^(k^*)) 18: After observing correctness, latency, and cost, compute per-query utility uâ(x)u(x) 19: Update kââkââȘ((Ï1(kâ),Ï2(kâ)),uâ(x))D_k^* _k^*âȘ\(( _1^(k^*), _2^(k^*)),u(x))\ 20: if tmodM=0t M=0 then 21: for k=1,âŠ,Kk=1,âŠ,K do 22: Fit GP surrogate fkf_k on kD_k 23: Run a small number of BO steps on cluster kC_k using fkf_k and kD_k to refresh (Ï1(k),Ï2(k))( _1^(k), _2^(k)) 24: end for 25: end if 26: end for Cluster-Based Bayesian Optimization To identify categories of queries with different routing preferences, we treat the queries used for training the router as historical queries, and cluster them based on their semantic representations hTh_T using classical K-means. The number of clusters K is automatically determined using the elbow method [15]. For each cluster kC_k with centroid ckc_k, we learn optimal thresholds (Ï1(k),Ï2(k))( _1^(k), _2^(k)) (Ï1(k)>Ï2(k) _1^(k)> _2^(k) and Ï1(k),Ï2(k)â[0,1] _1^(k), _2^(k)â[0,1]) that maximizes system utility: kâ(Ï1(k),Ï2(k)):=λ1â Accâλ2â Latencyâλ3â Cost,U_k( _1^(k), _2^(k)):= _1·Acc- _2·Latency- _3·Cost, (9) where Acc denotes the average response correctness under the current threshold-based routing, Latency represents the average inference delay, and Cost reflects the estimated computation cost. The weights λ1 _1, λ2 _2, and λ3 _3 control the trade-off among these objectives. For each query x in cluster kC_k, the predicted consistency score S^â(x) S(x) determines the selected model (DLM, ELM, or CLM) based on the corresponding thresholds (Ï1(k),Ï2(k))( _1^(k), _2^(k)). The accuracy term Acc is computed as the average correctness of selected model responses within the cluster. If a ground-truth reference answer is available, a response is considered correct if it exactly matches the reference; otherwise, we use an advanced LLM judge to assess the semantic appropriateness of the response. The Latency term includes end-to-end inference time, covering both model execution and communication delays. The Cost term is estimated based on the number of activated parameters and the length of generated tokens, following the compute cost model in [30]. Notably, the ratios λ1/λ2 _1/ _2 and λ1/λ3 _1/ _3 (λ1,λ2,λ3>0 _1, _2, _3>0) reflect the systemâs preference between accuracy and efficiency: a larger ratio favors quality, while a smaller ratio emphasizes speed and cost reduction. We perform Gaussian Process-based Bayesian optimization [21] to automatically determine the optimal thresholds (Ï1(k),Ï2(k))( _1^(k), _2^(k)) for each cluster kC_k. At each iteration, we fit a surrogate model to the observed utility values kU_k evaluated at previous threshold pairs, and use an acquisition function (Expected Improvement [13]) to propose the next threshold configuration to evaluate. This iterative process continues until the evaluation budget is exhausted, and is summarized in the offline phase of Algorithm 2, Lines 1â11. Online Adaptation of Cluster Thresholds The above procedure learns cluster-specific thresholds (Ï1(k),Ï2(k))( _1^(k), _2^(k)) from historical data. In practice, however, traffic patterns, query difficulty, and user preferences may drift over time. To keep the thresholds aligned with the current environment, we further endow our framework with an online adaptation mechanism based on incremental Bayesian optimization. For each cluster kC_k, we maintain a Gaussian Process surrogate fkf_k over the threshold space and a set of observed utility samples k=((Ï1,Ï2),u)D_k=\(( _1, _2),u)\. The offline optimization phase initializes kD_k with utility values kâ(Ï1(k),Ï2(k))U_k( _1^(k), _2^(k)) evaluated on historical queries. At deployment time, when a new query x arrives, we first assign it to a semantic cluster kâk^* and route it using the current thresholds (Ï1(kâ),Ï2(kâ))( _1^(k^*), _2^(k^*)). After the response is generated, we observe its correctness, latency, and cost, and compute a per-query utility uâ(x)=λ1â â[correct]âλ2â Latencyâ(x)âλ3â Costâ(x),u(x)= _1·I[correct]- _2·Latency(x)- _3·Cost(x), (10) where â[correct]I[correct] is the indicator function that equals 11 if the response is judged correct and 0 otherwise. In our experiments, correctness is determined using the same criterion as in the offline phase: exact match against the ground-truth answer when available, and an advanced LLM judge otherwise.In a real deployment, this term can be instantiated with any task-specific binary success signal (e.g., automatic verifiers or delayed user feedback), without changing the optimization procedure. This quantity can be viewed as a stochastic sample of the cluster-level utility kâU_k^*. We then append the pair ((Ï1(kâ),Ï2(kâ)),uâ(x))(( _1^(k^*), _2^(k^*)),u(x)) to kâD_k^* and incrementally update the surrogate fkâf_k^*. Periodically (after accumulating a fixed number of new queries in kD_k), we run Gaussian Process-based Bayesian optimization on each cluster in the background: using fkf_k and the acquisition function Expected Improvement, (EI) [13], we propose new candidate threshold pairs and update (Ï1(k),Ï2(k))( _1^(k), _2^(k)) if they yield higher estimated utility. Here EI measures the expected amount by which a candidate (Ï1,Ï2)( _1, _2) can improve over the best utility observed so far. This incremental procedure interleaves exploitation of the current best thresholds with exploration of promising alternatives, and allows the routing thresholds to continuously adapt to evolving query distributions without adding latency to the per-query routing path. The overall online adaptation loop corresponds to the streaming phase of Algorithm 2, Lines 12â26. Inference-Time Threshold Assignment At runtime, once we obtain the semantic representation hTh_T of the input query x, we determine its routing thresholds by assigning it to the nearest semantic cluster. We measure the Euclidean distance between hTh_T and each cluster centroid ckc_k, and select the closest cluster (Line 4 in Algorithm 1): kâ:=argâĄminkâĄâhTâckâ2.k^*:= _k\|h_T-c_k\|_2. (11) We then apply the optimized thresholds (Ï1(kâ),Ï2(kâ))( _1^(k^*), _2^(k^*)) associated with cluster kâC_k^* for routing decision (Line 5 in Algorithm 1 and the routing step in Algorithm 2, Lines 12â26). This enables context-sensitive thresholds to adjust to the semantic characteristics of the query, resulting in more precise trade-offs between performance, latency, and cost. V Experiments V-A Experimental Settings Data Construction We conduct experiments on the RouterBench dataset [12], which consists of 36.5K queries from 8 NLP datasets in both Chinese and English. For each query, we take it as input and collect the outputs of small and large language models. To construct training labels for the consistency predictor, we combine two sources: (1) Qwen3-Reranker-4B [37] for semantic similarity between DLM and higher-tier outputs, and (2) data augmentation via correctness-based rules (with references) or DeepSeek V3 [17] as a judge (without references). For queries with known correct answers, we label cases where the DLM is incorrect but the CLM is correct as inconsistent. For queries without known correct answers, we apply an advanced model to estimate response consistency. The final label is obtained by averaging the similarity score and the augmentation score. Datasets Our main results focus on MMLU [11] (general knowledge), GSM8K [6] (math reasoning), HumanEval [5] (code generation) and MT-Bench [38] (conversation, judged by GPT-4o). Baseline Algorithms We compare our approach with several representative baselines. The LLM-only baseline routes all queries to the large language model regardless of difficulty or cost. The DLM-only baseline handles all queries with the small language model, maximizing efficiency but potentially sacrificing accuracy. The Edge-only baseline processes all queries using an edge-optimized model with a moderate resource footprint. We further compare against learning-based routing methods. RouteLLM (BERT) [23] uses a fine-tuned BERT encoder to obtain a representation of each query, and feeds the [CLS] token into a logistic regression classifier to predict the probability that the DLM performs better than the CLM. RouteLLM (SW ranking) embeds the query with a text embedding model, retrieves its nearest historical query in the embedding space, and follows the routing decision associated with that most similar past query. MixLLM [29] encodes the query using a tag-enhanced BERT model and feeds the representation into multiple predictors (e.g., for quality, latency, and cost); a policy then decides whether to route to the DLM or CLM based on the estimated latencyâutility trade-off. Deployment Environment and Model Configuration To simulate a realistic hierarchical deployment scenario, we use different hardware platforms and language model configurations to represent device-side, edge-side, and cloud-side environments. On the device, we deploy Qwen3-1.7B [33] as the device language model (DLM), running on an Intel Core i5-12500H CPU paired with an NVIDIA RTX 3050 GPU, which mimics the capabilities of a typical consumer-grade endpoint. On the edge, we use a server equipped with a single NVIDIA RTX A6000 GPU to host the edge language model (ELM), specifically Qwen3-14B, representing a high-capability on-premise or near-edge setup. In the cloud, we deploy Qwen3-32B across two RTX A6000 GPUs to simulate access to large-scale language models with near-unlimited compute resources. This three-tier configuration reflects a practical deployment scenario with ascending model capacity and latency from device to cloud, and is used consistently throughout our experiments. Network Configuration for Edge and Cloud Links Based on our previous measurements of wide-area mobile networks and edge networks [32], we simulated end-to-edge and end-to-cloud connections by configuring the device-side downlink/uplink bandwidth, packet loss rate, one-way latency, and DNS latency. We simulated two network environments: âgoodâ and âbadâ. In the âgoodâ network condition, the device-to-edge link was configured with: 10,000 Kbps downlink bandwidth, 5,000 Kbps uplink bandwidth, 0.1% bidirectional packet loss rate, one-way latency of 40 ms (downlink) and 20 ms (uplink), and a DNS latency of 50 ms; the device-to-cloud link was configured with: 8,000 Kbps downlink bandwidth, 4,000 Kbps uplink bandwidth, 0.1% packet loss rate, one-way latency of 80 ms and 40 ms, and a DNS latency of 70 ms. This represents a relatively stable and well-configured connection. In the âbadâ network condition, we simulated connections with reduced bandwidth, higher packet loss rates, and higher latency. The device-to-edge link was configured with: 2,000 Kbps downlink bandwidth, 500 Kbps uplink bandwidth, 1% packet loss rate, one-way latency of 120 ms and 80 ms, and a DNS latency of 200 ms. The device-to-cloud link had even stricter limitations: 800 Kbps downlink bandwidth, 200 Kbps uplink bandwidth, 3% packet loss rate, one-way latency of 250 ms and 200 ms, and a DNS latency of 400 ms. In the âNetwork Condition Sensitivityâ experiment, we used three settings: âgoodâ, âbadâ and âbadâââgoodâ, which allowed us to observe how ConsRoute adjusts its routing thresholds as the underlying network improves. All other experiments were conducted under âgoodâ network conditions. (a) GSM8K (b) MMLU (c) HumanEval (d) MT-Bench (e) GSM8K (f) MMLU (g) HumanEval (h) MT-Bench Figure 6: Comparison of accuracy, latency, and cost across benchmarks. ConsRoute achieves near-cloud accuracy with significantly reduced latency and cost. V-B Overall Performance and Cross-Model Adaptability V-B1 Overall Results As shown in Figure 6a to Figure 6c, and Figure 6e to Figure 6g, ConsRoute consistently outperforms baseline methods in the latency-performance trade-off across all three benchmarksâGSM8K, MMLU, HumanEval and MT-Benchâand achieves competitive or superior results in the cost-performance trade-off in most cases. Under the current experimental setup, ConsRoute achieves near-cloud performance (â„ 95%) using only 60%-65% of the latency required by a cloud-only strategy. In comparison, baseline methods such as MixLLM and RouteLLM typically require 70%-85% of the cloud latency to reach similar accuracy levels. In terms of cost, ConsRoute achieves comparable performance using only about 70% of the cloud-only cost, and outperforms most baselines across a wide range of routing thresholds. To enable fair and interpretable comparison, latency, cost, and score are all normalized relative to three fixed reference points: (0, 0) for device-only (DLM), (100, 100) for cloud-only (CLM), and a mid-point representing edge-only (ELM) based on actual measured latency and cost. Each method generates a set of routing decisions under varying latency or cost constraints, which are then plotted as a curve reflecting its trade-off behavior. These results demonstrate that ConsRoute is able to make better use of hierarchical model capacities in cloud-edge-device collaborative inference, reaching high-quality predictions more efficiently than existing approaches. V-B2 Cross-Family CloudâEdgeâDevice Deployments To verify that our framework does not rely on homogeneity, we further consider a heterogeneous deployment in which each tier is served by a different model family: LLaMA-3.2-3B on the device, Qwen3-14B on the edge, and DeepSeek-V3 [17] in the cloud (accessed via API). For DeepSeek-V3, a mixture-of-experts model, we estimate inference cost using the number of activated parameters (37B) rather than the full model size (671B). This setting mimics realistic multi-vendor deployments where models differ in architecture, tokenizer, and training pipeline. Results on GSM8K (Figure 7) show that ConsRoute still achieves favorable accuracyâlatency and accuracyâcost trade-offs under this cross-family configuration, comparable to those observed with the homogeneous stack. This demonstrates that our routing framework can generalize across heterogeneous model combinations and remains effective even when the cloud, edge, and device tiers are served by different LLM families. (a) End-to-end latency vs. accuracy. (b) Inference cost vs. accuracy. Figure 7: Adaptability of ConsRoute under heterogeneous model deployments on GSM8K. V-C Analysis of Routing Signals and DLM Representations (RQ1 & RQ2) V-C1 Routing Latency and Device-Side Extra Resource Overhead To assess deployment feasibility, we compare our routing latency and device-side overhead with RouteLLM and MixLLM. For RouteLLM-SW Ranking, latency includes OpenAI embedding API calls and local vector retrieval. RouteLLM-BERT and MixLLM incur latency from BERT encoding and MLP classification, with RouteLLM-BERT further requiring on-device execution of a 110M-parameter BERT-base modelâintroducing notable memory and compute costs on resource-limited devices. Figure 8: Routing Latency and additional device-side resource overhead of different routers. âConsRoute-x%â denotes the latency and overhead when the system achieves x% of the CLM-only accuracy. RouteLLM and MixLLM have the same latency and additional overhead with different accuracy rates. In contrast, our method significantly reduces routing latency and resource overhead. When routing to a higher-tier model, latency includes only the DLM prefill (for hidden state extraction) and the lightweight MLP prediction. If the DLM is selected for generation, its prefill is part of inference, and routing delay reduces to just the MLPâs inference time. The MLP itself has only 5M parameters, adding negligible computation and memory overhead. This compact design ensures minimal impact on edge devices. Figure 8 illustrates the routing latency and additional device-side resource overhead (measured by extra parameter count) for different methods. Our approach consistently achieves faster routing speed compared to baselines, while introducing negligible additional overhead on the device, making it well-suited for real-time applications. V-C2 Effect of Prompting Strategies on Routing We further conduct a study to examine how different prompting strategies applied to the DLM affect routing performance, contrasting explicit routing decisions made by the DLM with implicit representations fed to a separate router. As shown in Table I, on the GSM8K dataset we compare four variants: (1) explicit selection without CoT, where the DLM is directly asked to choose the execution tier (device, edge, or cloud) and outputs a discrete decision without any reasoning; (2) explicit selection with CoT, where the DLM first explains its routing decision via chain-of-thought and then outputs the final choice; (3) implicit selection without prompt, where we simply feed the last-token hidden state of the DLM without any additional prompt into the router; and (4) implicit selection with prompt (our method), where a fixed consistency-oriented instruction is appended and the last token hidden state is used as the router input. The exact prompts used for the explicit routing baselines (with and without CoT) are provided in Appendix A. The results demonstrate that prompt-guided implicit representation is the most effective and efficient design. The naive implicit variant without prompt achieves the lowest accuracy , indicating that generic next-token hidden states are not sufficiently aligned with the consistency prediction task. Adding an explicit prompt and using last token hidden state substantially improves accuracy, while keeping router latency almost unchanged, confirming that our prompt primarily reshapes the internal representation without introducing extra routing overhead. The non-CoT variant attains moderate accuracywith relatively high latency , whereas enabling chain-of-thought slightly improves accuracy but incurs prohibitive routing latency. These findings validate our design choice of using prompt-guided implicit representations: it captures consistency-aware semantics better than unprompted hidden states and achieves a more favorable accuracyâlatency trade-off than explicit DLM-based routing. TABLE I: Effect of prompting strategies on routing performance Method Accuracy Router Latency (ms) Explicit selection (w/o CoT) 85.79 36.2 Explicit selection (w/ CoT) 86.44 1868.2 Implicit selection (no prompt) 82.34 20.4 Implicit selection (prompted) 87.92 20.9 V-C3 Comparison of Supervision Sources To investigate the impact of different training signals for consistency-aware routing, we compare three types of supervision labels: reward model scores (from Qwen2.5-PRM-7B), BartScore [7], and reranker-based (from Qwen3-Reranker-4B) semantic similarity.For each label type, we train the same routing architecture using identical data and evaluate their performance under consistent conditions. As shown in Figure 9, reranker-based supervision consistently achieves lower latency and cost at the same level of performance. These results confirm that semantic similarity is a more faithful and effective training signal for consistency-aware routing. (a) Latency vs Accuracy (b) Cost vs Accuracy Figure 9: Impact of supervision source on routing efficiency. Each curve shows the latency or cost required to reach a target CLM-only accuracy. V-D Threshold Adaptation and System Trade-offs (RQ3) V-D1 Online Threshold Adaptation To evaluate the effectiveness of the proposed online adaptation mechanism, we simulate a streaming deployment scenario on a held-out subset of RouterBench that is not used for training or offline threshold tuning. Queries in this subset are ordered to form an input stream and divided into non-overlapping windows of 200 consecutive requests. Each window is treated as one time step on the horizontal axis in Figure 10, and we record the routing accuracy within each window as the performance at that time step. After processing a window, the online variant is allowed to update cluster-specific thresholds using the new feedback, whereas the static variant keeps thresholds fixed. We compare two variants of our framework under this setting. ConsRoute-static uses the cluster-specific thresholds learned in the offline phase and keeps them unchanged throughout the entire stream. ConsRoute-online starts from the same offline thresholds but, after each window of 200 requests, performs an incremental Bayesian optimization step to refine thresholds for clusters that received new data. Both variants share the same consistency predictor, clustering assignments, and deployment configuration. Figure 10 plots the routing accuracy over time, where the horizontal axis denotes the index of the 200-request window and the vertical axis shows the average accuracy within that window. As the query stream progressed, the accuracy of ConsRoute-static decreased slightly, while ConsRoute-online, after several threshold updates, was able to recover and maintain higher accuracy. These results demonstrate that online threshold adaptation can effectively track distributional changes and improve long-term routing performance. Figure 10: Online threshold adaptation of ConsRoute. ConsRoute-online maintains consistently higher accuracy as the query distribution drifts. V-D2 Sensitivity to Network Conditions To study how ConsRoute reacts to varying network conditions in a cloudâedgeâdevice setting, we simulate three types of network environments (Good, Bad, and Badâ ) on an online evaluation subset of RouterBench. Guided by prior measurement studies on mobile and edge networks [32], we emulate device-to-edge and device-to-cloud connections by configuring their downlink/uplink bandwidth, packet loss rate, one-way delay, and DNS delay. The Badâ condition starts with the Bad configuration in the first half of the query stream and switches to the Good configuration in the second half, mimicking a recovery from a congested to a normal network. The detailed parameter settings for the edge and cloud links under the Good and Bad profiles are provided in Experimental Settings. All other settings are identical to those in the Online Threshold Adaptation experiment. Figure 11 shows six curves: three accuracy trajectories (Good-Acc, Bad-Acc, Badâ -Acc) and three latency trajectories (Good-Latency, Bad-Latency, Badâ -Latency) over time. Under the Good condition, ConsRoute-online balances quality and communication cost: it is willing to route a non-trivial fraction of queries to edge and cloud models, achieving higher accuracy. Under the Bad condition, the increased communication delay is heavily penalized in the utility function, and the incremental Bayesian optimization gradually shifts cluster-specific thresholds toward more conservative escalation. As a result, the fraction of queries served by the on-device DLM increases, leading to a noticeable reduction in average latency, accompanied by a modest drop in accuracy that remains within an acceptable range. In the Badâ condition, we observe a two-phase adaptive behavior. During the initial Bad phase, the trends resemble those of the pure Bad condition: thresholds are adjusted to favor on-device routing, decreasing latency while slightly degrading accuracy. Once the network switches back to the Good configuration, the new latency feedback is incorporated into the utility estimates, and Bayesian optimization begins to relax the thresholds, making the system more willing to route difficult queries to edge or cloud models. These results demonstrate that ConsRoute-online can automatically adapt its routing thresholds to changing network conditions, achieving intuitive accuracyâlatency trade-offs without manual retuning or retraining of the router. Figure 11: Network condition sensitivity of ConsRoute. Accuracy (left axis) and end-to-end latency (right axis) over time under three network profiles: Good, Bad, and Badâ . In the Badâ setting, the network switches from the Bad profile to the Good profile at time step 7. V-D3 Sensitivity to Utility Hyperparameters To understand how the choice of utility weights on correctness, latency, and cost in the Bayesian optimization utility function shapes the routing behavior, we conduct a sensitivity study on the HumanEval dataset. We normalize the accuracy weight λ1 _1 and vary the relative penalty on latency and cost by adjusting the ratios Îș1=λ1/λ2 _1= _1/ _2 and Îș2=λ1/λ3 _2= _1/ _3. For each hyperparameter setting, we record the fraction of queries routed to device, edge, and cloud, as well as the resulting end-to-end accuracy. Figure 12 summarizes the results. The bar plots show the proportions of queries assigned to the DLM, ELM, and CLM under different utility configurations, while the overlaid line reports the corresponding accuracy. When the utility heavily penalizes latency and cost (small Îș1 _1 and Îș2 _2), ConsRoute routes the majority of queries to the DLM and ELM, achieving the lowest accuracy but the highest efficiency. As the weight on correctness increases, the routing gradually shifts from device and edge to the cloud tier, and the overall accuracy improves monotonically. In the most accuracy-oriented setting, a substantial fraction of queries are escalated to the CLM, yielding the highest accuracy at the expense of higher latency and cost. These results confirm that the proposed utility-based formulation provides a smooth and interpretable knob for trading off quality versus efficiency in the cloud-edge-device collaboration. Figure 12: Effect of utility weights on routing behavior and accuracy on HumanEval. Bars show the proportions of queries routed to device, edge, and cloud under different weight settings, and the line reports the corresponding accuracy. Îș1 _1 and Îș2 _2 denote the normalized weights Îș1=λ1/λ2 _1= _1/ _2 and Îș2=λ1/λ3 _2= _1/ _3. V-E Other Ablation Studies To assess the contributions of key components in our routing framework, we conduct additional ablation experiments on the HumanEval dataset. We study the effect of two design choices: label augmentation during training, and dynamic thresholding via Bayesian optimization. The comparison includes three configurations: (1) a minimal version without label augmentation and with a fixed global threshold; (2) a variant with label augmentation but a static threshold; and (3) our full model, which incorporates both. (a) Performance vs Latency (b) Performance vs Cost Figure 13: Impact of supervision source on routing efficiency. Each curve shows the latency or cost required to reach a target CLM-only accuracy. V-E1 Effect of Label Augmentation Figure 13 shows that incorporating label augmentation improves routing accuracy. Compared to using reranker-only supervision, the addition of correctness- and consistency-based labels leads to a notable performance gain. This confirms that enriched supervision provides more reliable guidance for the consistency predictor. V-E2 Effect of Dynamic Thresholding Figure 13 demonstrates that using dynamically optimized thresholds yields clear performance improvements over static thresholding. This validates that query-specific thresholds, learned via Bayesian optimization, enable the router to better adapt to diverse query complexities and risk levels. In contrast, static heuristics often fail to generalize across tasks, leading to suboptimal routing decisions. The full model, which combines both dynamic thresholding and label augmentation, consistently achieves the best results across all metrics, demonstrating the complementarity of these two components. VI Conclusion This paper presents a lightweight and semantics-aware query routing framework for collaborative LLM inference across cloud, edge, and device tiers. Our approach leverages semantic consistency between outputs for more reliable routing. By combining reranker-based supervision with consistency-oriented data augmentation, and reusing deep hidden states for query representation, the framework enables adaptive, low-overhead routing. A class-specific thresholding strategy via Bayesian optimization further improves query sensitivity. Experiments show strong performanceâefficiency trade-offs.In future work, we plan to extend ConsRoute to broader application domains and richer adaptivity signals, and to study its behavior under larger-scale, real-world mobile deployments. References [1] P. Aggarwal, A. Madaan, A. Anand, S. P. Potharaju, S. Mishra, P. Zhou, A. Gupta, D. Rajagopal, K. Kappaganthu, Y. Yang, et al. (2024) Automix: automatically mixing language models. Advances in Neural Information Processing Systems 37, p. 131000â131034. Cited by: §I, §I-B1, TABLE I. [2] J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, et al. (2023) Qwen technical report. arXiv preprint arXiv:2309.16609. Cited by: §I, §I-B2. [3] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. (2020) Language models are few-shot learners. Advances in neural information processing systems 33, p. 1877â1901. Cited by: §I. [4] L. Chen, M. Zaharia, and J. Zou (2023) Frugalgpt: how to use large language models while reducing cost and improving performance. arXiv preprint arXiv:2305.05176. Cited by: §I, §I-B1, TABLE I. [5] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. (2021) Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. Cited by: §V-A. [6] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. (2021) Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Cited by: §V-A. [7] D. Ding, A. Mallick, C. Wang, R. Sim, S. Mukherjee, V. Ruhle, L. V. Lakshmanan, and A. H. Awadallah (2024) Hybrid llm: cost-efficient and quality-aware query routing. arXiv preprint arXiv:2404.14618. Cited by: §I-B2, TABLE I, §IV-C2, §V-C3. [8] W. Fan, L. Zhao, X. Liu, Y. Su, S. Li, F. Wu, and Y. Liu (2024) Collaborative service placement, task scheduling, and resource allocation for task offloading with edge-cloud cooperation. IEEE Transactions on Mobile Computing 23 (1), p. 238â256. External Links: Document Cited by: §I-A. [9] Z. Hao, H. Jiang, S. Jiang, J. Ren, and T. Cao (2024) Hybrid slm and llm for edge-cloud collaborative inference. In Proceedings of the Workshop on Edge and Mobile Foundation Models, p. 36â41. Cited by: §I-A. [10] Y. He, J. Fang, F. R. Yu, and V. C. Leung (2024) Large language models (llms) inference offloading and resource allocation in cloud-edge computing: an active inference approach. IEEE Transactions on Mobile Computing 23 (12), p. 11253â11264. External Links: Document Cited by: §I-A. [11] D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt (2020) Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300. Cited by: §V-A. [12] Q. J. Hu, J. Bieker, X. Li, N. Jiang, B. Keigwin, G. Ranganath, K. Keutzer, and S. K. Upadhyay (2024) Routerbench: a benchmark for multi-llm routing system. arXiv preprint arXiv:2403.12031. Cited by: §V-A. [13] D. R. Jones, M. Schonlau, and W. J. Welch (1998) Efficient global optimization of expensive black-box functions. Journal of Global optimization 13 (4), p. 455â492. Cited by: §IV-D, §IV-D. [14] A. M. Kassem, B. Schölkopf, and Z. Jin (2025) How robust are router-llms? analysis of the fragility of llm routing capabilities. arXiv preprint arXiv:2504.07113. Cited by: §I. [15] L. Kaufman and P. J. Rousseeuw (2009) Finding groups in data: an introduction to cluster analysis. John Wiley & Sons. Cited by: §IV-D. [16] S. Li, H. Wang, W. Xu, R. Zhang, S. Guo, J. Yuan, X. Zhong, T. Zhang, and R. Li (2025) Collaborative inference and learning between edge slms and cloud llms: a survey of algorithms, execution, and open challenges. arXiv preprint arXiv:2507.16731. Cited by: §I-A. [17] A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, et al. (2024) Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. Cited by: §I, §V-A, §V-B2. [18] J. Liu, R. Gong, M. Zhang, Y. He, J. Cai, and B. Zhuang (2024) ME-switch: a memory-efficient expert switching framework for large language models. arXiv preprint arXiv:2406.09041. Cited by: §I-B3, TABLE I. [19] Y. Liu, H. Zhang, Y. Miao, V. Le, and Z. Li (2024) OptLLM: optimal assignment of queries to large language models. In 2024 IEEE International Conference on Web Services (ICWS), p. 788â798. Cited by: §I, §I-B3. [20] K. Lu, H. Yuan, R. Lin, J. Lin, Z. Yuan, C. Zhou, and J. Zhou (2023) Routing to the expert: efficient reward-guided ensemble of large language models. arXiv preprint arXiv:2311.08692. Cited by: §I, §I-B2, TABLE I, §IV-C2. [21] J. Mockus (2005) The bayesian approach to global optimization. In System Modeling and Optimization: Proceedings of the 10th IFIP Conference New York City, USA, August 31âSeptember 4, 1981, p. 473â481. Cited by: §IV-D, §IV-D. [22] Q. H. Nguyen, T. Dao, D. C. Hoang, J. Decugis, S. Manchanda, N. V. Chawla, and K. D. Doan (2024) Metallm: a high-performant and cost-efficient dynamic framework for wrapping llms. arXiv preprint arXiv:2407.10834. Cited by: §I-B3. [23] I. Ong, A. Almahairi, V. Wu, W. Chiang, T. Wu, J. E. Gonzalez, M. W. Kadous, and I. Stoica (2024) RouteLLM: learning to route llms from preference data. In The Thirteenth International Conference on Learning Representations, Cited by: §I, §I-B2, TABLE I, §V-A. [24] G. Qu, Q. Chen, W. Wei, Z. Lin, X. Chen, and K. Huang (2025) Mobile edge intelligence for large language models: a contemporary survey. IEEE Communications Surveys & Tutorials. Cited by: §I. [25] V. Sanh, L. Debut, J. Chaumond, and T. Wolf (2019) DistilBERT, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108. Cited by: §I-B1. [26] H. Shi, T. Zheng, W. Wang, B. Xu, C. Li, C. Chan, T. Fan, Y. Song, and Q. Yang (2025) INFERENCEDYNAMICS: efficient routing across llms through structured capability and knowledge profiling. arXiv preprint arXiv:2505.16303. Cited by: §I-B3. [27] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. RoziĂšre, N. Goyal, E. Hambro, F. Azhar, et al. (2023) Llama: open and efficient foundation language models. arXiv preprint arXiv:2302.13971. Cited by: §I. [28] C. Varangot-Reille, C. Bouvard, A. Gourru, M. Ciancone, M. Schaeffer, and F. Jacquenet (2025) Doing more with lessâimplementing routing strategies in large language model-based systems: an extended survey. arXiv preprint arXiv:2502.00409. Cited by: §I. [29] X. Wang, Y. Liu, W. Cheng, X. Zhao, Z. Chen, W. Yu, Y. Fu, and H. Chen (2025) Mixllm: dynamic routing in mixed large language models. arXiv preprint arXiv:2502.18482. Cited by: §I-B3, TABLE I, §V-A. [30] G. Wilkins, S. Keshav, and R. Mortier (2024) Hybrid heterogeneous clusters can lower the energy consumption of llm inference workloads. In Proceedings of the 15th ACM International Conference on Future and Sustainable Energy Systems, p. 506â513. Cited by: §I-C, §IV-D. [31] S. Xiao, Z. Liu, P. Zhang, N. Muennighoff, D. Lian, and J. Nie (2024) C-pack: packed resources for general chinese embeddings. In Proceedings of the 47th international ACM SIGIR conference on research and development in information retrieval, p. 641â649. Cited by: §IV-C2. [32] M. Xu, Z. Fu, X. Ma, L. Zhang, Y. Li, F. Qian, S. Wang, K. Li, J. Yang, and X. Liu (2021) From cloud to edge: a first look at public edge platforms. In Proceedings of the 21st ACM internet measurement conference, p. 37â53. Cited by: §V-A, §V-D2. [33] A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. (2025) Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: §I, §V-A. [34] Z. Yao, Z. Tang, W. Yang, and W. Jia (2025) Enhancing llm qos through cloud-edge collaboration: a diffusion-based multi-agent reinforcement learning approach. IEEE Transactions on Services Computing. Cited by: §I-A, TABLE I. [35] M. Yue, J. Zhao, M. Zhang, L. Du, and Z. Yao (2023) Large language model cascades with mixture of thoughts representations for cost-efficient reasoning. arXiv preprint arXiv:2310.03094. Cited by: §I-B1, TABLE I. [36] M. Zhang, X. Shen, J. Cao, Z. Cui, and S. Jiang (2024) Edgeshard: efficient llm inference via collaborative edge computing. IEEE Internet of Things Journal. Cited by: §I. [37] Y. Zhang, M. Li, D. Long, X. Zhang, H. Lin, B. Yang, P. Xie, A. Yang, D. Liu, J. Lin, F. Huang, and J. Zhou (2025) Qwen3 embedding: advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05176. Cited by: §IV-C2, §V-A. [38] L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al. (2023) Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems 36, p. 46595â46623. Cited by: §V-A. [39] P. Zhu and T. Yang (2025) CE-lslm: efficient large-small language model inference and communication via cloud-edge collaboration. arXiv preprint arXiv:2505.14085. Cited by: §I-A.