Paper deep dive
OpWeave: Flexible Operator Disaggregation for Heterogeneous LLM Serving
Zikun Li, Yixuan Mei, Shiqi Pan, Zixuan Chen, Xiaowen Zhang, Mengdi Wu, Shuhuai Lin, Yutong Yang, Zhihao Zhang, Xupeng Miao, Rashmi Vinayak, Zhihao Jia
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 9/21/2026, 5:03:28 AM
Summary
OpWeave is an end-to-end framework for heterogeneous operator-level disaggregated serving (ODS) of Large Language Models (LLMs). It addresses inefficiencies in colocated serving by separating operators like attention and FFN/MoE across heterogeneous GPU clusters. OpWeave features an analytical cost model to bound performance gains, a regularity-aware planner for joint optimization of partitioning and deployment, and a vLLM-based runtime. Evaluations show cost reductions of up to 1.78x on homogeneous and 1.89x on heterogeneous GPU clusters compared to baselines.
Entities (10)
Relation Signals (9)
Zikun Li → affiliatedwith → Carnegie Mellon University
confidence 95% · Zikun Li Affiliation: Carnegie Mellon University
OpWeave → implements → Operator-level disaggregated serving
confidence 95% · We present OpWeave, an end-to-end framework for heterogeneous ODS.
OpWeave → evaluatedon → Gemma-3-27B
confidence 90% · OpWeave plans require fewer pipeline stages... for Gemma-3-27B
OpWeave → evaluatedon → Qwen3-235B
confidence 90% · Solid curves show gains for Qwen3-235B... calculated from our theoretical model
OpWeave → outperforms → Attention-FFN Disaggregation
confidence 90% · Compared with attention–FFN disaggregation (AFD), OpWeave plans require fewer pipeline stages
OpWeave → reducescoston → H100
confidence 90% · On homogeneous H100 clusters, OpWeave reduces serving cost by up to 1.78x
OpWeave → reducescoston → A100
confidence 90% · On heterogeneous clusters combining H100 with L40S or A100 GPUs... OpWeave reduces serving cost
OpWeave → reducescoston → L40S
confidence 90% · On heterogeneous clusters combining H100 with L40S... OpWeave reduces serving cost
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:LLM serving systems increasingly disaggregate inference into finer-grained stages, with recent approaches separating attention from FFN or MoE execution during decode. This operator-level disaggregated serving (ODS) can improve hardware matching and enable independent scaling, particularly across heterogeneous devices. However, existing systems fix operator boundaries and lack a unified characterization of when disaggregation reduces serving cost. We present OpWeave, an end-to-end framework for heterogeneous ODS. OpWeave provides an analytical cost model that bounds the gains of homogeneous and heterogeneous ODS over colocated serving. It jointly optimizes operator partitioning and deployment configuration through a regularity-aware planner that keeps the search tractable even for hybrid-attention models. A vLLM-based runtime executes the synthesized plans with flexible operator stages across heterogeneous device groups. In our evaluation, OpWeave reduces serving cost by up to $1.78\times$ on homogeneous and $1.89\times$ on heterogeneous GPU clusters relative to the best feasible baseline, while meeting latency SLOs.
Tags
Links
- Source: https://arxiv.org/abs/2609.14237v1
- Canonical: https://arxiv.org/abs/2609.14237v1
Trouble viewing inline? Open PDF directly →
Full Text
133,104 characters extracted from source content.
Expand or collapse full text
OpWeave: Flexible Operator Disaggregation for Heterogeneous LLM Serving Zikun Li Note: Zikun Li and Yixuan Mei contributed equally to this work. Affiliation: Carnegie Mellon University email: zikunl@andrew.cmu.edu , Yixuan Mei Affiliation: Carnegie Mellon University email: yixuanm@andrew.cmu.edu , Shiqi Pan Affiliation: Carnegie Mellon University email: shiqip@andrew.cmu.edu , Zixuan Chen Affiliation: Carnegie Mellon University email: zixuanc3@andrew.cmu.edu , Xiaowen Zhang Affiliation: Carnegie Mellon University email: xiaowen5@andrew.cmu.edu , Mengdi Wu Affiliation: Carnegie Mellon University email: mengdiwu@andrew.cmu.edu , Shuhuai Lin Affiliation: Carnegie Mellon University email: shuhuailinux@gmail.com , Yutong Yang Affiliation: Carnegie Mellon University email: aprilytyang@gmail.com , Zhihao Zhang Affiliation: Carnegie Mellon University email: zhihaoz3@andrew.cmu.edu , Xupeng Miao Note: Work done at Purdue. Affiliation: Peking University email: xupeng.miao@pku.edu.cn , Rashmi Vinayak Affiliation: Carnegie Mellon University and Google email: rvinayak@andrew.cmu.edu and Zhihao Jia Affiliation: Carnegie Mellon University email: zhihao@cmu.edu Abstract. LLM serving systems increasingly disaggregate inference into finer-grained stages, with recent approaches separating attention from FFN or MoE execution during decode. This operator-level disaggregated serving (ODS) can improve hardware matching and enable independent scaling, particularly across heterogeneous devices. However, existing systems fix operator boundaries and lack a unified characterization of when disaggregation reduces serving cost. We present OpWeave, an end-to-end framework for heterogeneous ODS. OpWeave provides an analytical cost model that bounds the gains of homogeneous and heterogeneous ODS over colocated serving. It jointly optimizes operator partitioning and deployment configuration through a regularity-aware planner that keeps the search tractable even for hybrid-attention models. A vLLM-based runtime executes the synthesized plans with flexible operator stages across heterogeneous device groups. In our evaluation, OpWeave reduces serving cost by up to 1.78×1.78× on homogeneous and 1.89×1.89× on heterogeneous GPU clusters relative to the best feasible baseline, while meeting latency SLOs. 1. Introduction Figure 1. Colocated serving versus heterogeneous operator-level disaggregated serving (ODS). AFD fixes stage boundaries at attention–FFN interfaces, whereas OpWeave supports general operator partitions across heterogeneous device groups. Large language model (LLM) inference accounts for a substantial share of AI infrastructure spending and requires significant investment in compute (Tully et al., 2025; Merizzi et al., 2025; Noffsinger et al., 2025). At this scale, even modest improvements in throughput and hardware utilization, or reductions in per-token cost, can yield substantial savings. Improving LLM serving efficiency has therefore become a central systems challenge (Yu et al., 2022; Kwon et al., 2023; Agrawal et al., 2024; Patel et al., 2024; Zhong et al., 2024). Modern serving systems increasingly separate inference phases to optimize and scale them independently. Most existing approaches disaggregate coarse-grained phases, particularly prefill and decode (Patel et al., 2024; Zhong et al., 2024; Singh et al., 2025; Dong et al., 2025; Lohmeyer and Bradstock, 2025; Gangasani et al., 2026). However, these approaches generally retain decode as a single execution unit, despite substantial differences in the resource demands of its constituent operators. Recent systems extend disaggregation into the decode phase by separating attention from feed-forward network (FFN) or mixture-of-experts (MoE) execution (Aubrey and Ghodsian, 2026; Zhu et al., 2025; StepFun et al., 2025; Song et al., 2026; Liu et al., 2026). This development motivates our focus on operator-level disaggregated serving. Operator-level disaggregated serving (ODS) partitions decode computation into separately executed operator stages, such as attention and FFN or MoE stages. Heterogeneous ODS further assigns these stages to different device types, as illustrated in Figure 1. This design addresses two sources of inefficiency in conventional serving engines (Yu et al., 2022; Kwon et al., 2023; Zheng et al., 2023; Agrawal et al., 2024). First, colocating operators can lead to poor hardware matching: decode attention is often limited by memory bandwidth and underutilizes tensor cores, while FFN and MoE operators rely heavily on matrix multiplications and benefit from high arithmetic throughput (Zhu et al., 2024; Zhu et al., 2025; StepFun et al., 2025). Second, colocation introduces memory-capacity coupling: as sequence length increases, the growing key–value (KV) cache limits the feasible batch size for the entire workload, potentially constraining FFN and MoE throughput. Heterogeneous ODS mitigates these inefficiencies by pipelining operator stages across groups of heterogeneous GPUs (He et al., 2024; Zhu et al., 2025; StepFun et al., 2025; Song et al., 2026; Liu et al., 2026). First, it improves hardware matching by assigning each stage to devices suited to its resource requirements. Second, it enables independent scaling, allowing FFN and MoE stages to use batch sizes that are not directly constrained by the KV-cache capacity of an individual attention stage. Together, these capabilities can improve decode efficiency and reduce serving cost. Despite this potential, realizing these benefits requires addressing three challenges. (a) (b) Figure 2. Decode-phase bottlenecks of Gemma-3-27B motivating ODS. (a) Tensor-core utilization of attention and FFN operators across batch sizes (B200): attention remains underutilized while batching improves FFN utilization. (b) Maximum feasible batch size across sequence lengths (H100): KV-cache growth constrains batching. Theory gap. Existing work provides empirical evidence and analytical models for specific ODS designs (Zhu et al., 2025; StepFun et al., 2025; Song et al., 2026; Liu et al., 2026). A unified analysis is needed to determine when ODS reduces cost relative to colocated serving, establish bounds on the benefits of homogeneous and heterogeneous ODS, and characterize how these benefits depend on sequence length, model architecture, and hardware characteristics. Planning gap. Existing systems optimize deployments around fixed operator boundaries, such as attention–FFN or attention–MoE splits (Zhu et al., 2025; StepFun et al., 2025). These fixed boundaries constrain joint optimization of operator partitioning, hardware assignment, parallelism, and batching. A general formulation is needed to synthesize efficient deployment plans across this broader design space, particularly for hybrid-attention models whose operator structures vary across layers. Runtime gap. Existing ODS runtimes organize execution and communication around specific attention–FFN or attention–MoE splits (Zhu et al., 2025; StepFun et al., 2025). Executing synthesized plans with different operator boundaries and pipeline schedules requires flexible operator staging, efficient pipeline orchestration, and low-overhead communication across heterogeneous device groups. To address these challenges, we present OpWeave, an end-to-end framework for heterogeneous ODS that integrates theoretical analysis, deployment planning, and runtime execution. First, OpWeave uses an analytical cost model for colocated serving, homogeneous ODS, and heterogeneous ODS. The analysis characterizes how cost reductions depend on sequence length, model architecture, and hardware, and upper-bounds the gains of disaggregation and heterogeneous hardware assignment. Under this model, core-attention disaggregation (CAD) realizes the idealized ODS cost when its stages are balanced and communication is fully overlapped. The analysis also identifies two limitations of CAD: hardware and workload constraints can prevent stage balance, and layerwise variation in hybrid-attention models can introduce pipeline imbalance under a fixed two-stage partition. Second, OpWeave formulates heterogeneous ODS planning as a joint optimization problem over operator partitioning, hardware assignment, parallelism, batching, and pipeline orchestration. To make this combinatorial search tractable, we introduce the partition block: the smallest repeating, layer-aligned operator pattern in a model, used as the unit of plan construction and reuse. Building on this abstraction, OpWeave develops a regularity-aware planner that reduces the search space by reusing partitioning decisions across layers with the same high-level operator structure, even when attention mechanisms differ. Together they support operator partitions beyond fixed attention–FFN or attention–MoE splits. Figure 3. Pipelined attention–FFN disaggregation (AFD). Third, OpWeave executes synthesized serving plans through a distributed runtime built on vLLM. The runtime’s staged engines support flexible operator partitions and pipeline schedules across heterogeneous device groups. To reduce host orchestration overhead, each worker submits its computation and communication sequence in one native call per decoding iteration. Low-overhead communication further reduces transfer costs. Finally, we evaluate OpWeave across representative models, hardware configurations, and long-context workloads. On homogeneous H100 clusters, OpWeave reduces serving cost by up to 1.78×1.78× compared to the best existing approaches while attaining time-per-output-token (TPOT) service-level objectives (SLOs). Compared with attention–FFN disaggregation (AFD), OpWeave plans require fewer pipeline stages (20.7 versus 124 on average for Gemma-3-27B) and reduce inter-node data transfer per output token by up to 22.8×22.8×; the resulting latency reduction lets OpWeave meet stringent TPOT SLOs that AFD cannot satisfy. OpWeave also keeps hardware executing the model over 80% of the time, whereas AFD leaves it idle roughly half the time. On heterogeneous clusters combining H100 with L40S or A100 GPUs, heterogeneous OpWeave reduces serving cost by up to 1.89×1.89× compared to the best non-OpWeave baseline and by up to 21.8%21.8\% compared to homogeneous OpWeave. 2. Background and Motivation Transformer Inference Transformer-based LLMs stack repeated layers of attention and feed-forward computation, implemented as FFN or MoE blocks. Inference consists of two phases with distinct computational profiles: prefill processes the input context in parallel and is typically compute-bound, whereas decode generates one token per autoregressive iteration. This contrast motivates prefill-decode disaggregation, which places the two phases on separate resources for independent optimization (Patel et al., 2024; Zhong et al., 2024). For generation-heavy workloads, such as reasoning tasks that produce long output sequences, decode often dominates latency and serving cost. Figure 4. Theoretical cost gain of homogeneous ODS over colocated serving. Solid curves show gains for Qwen3-235B and GPT-OSS-120B on H100 under different attention-TP/MoE-EP configurations; dashed lines show upper bounds. Decode-Phase Tensions Serving systems batch concurrent requests to amortize model-weight accesses and improve hardware utilization, particularly for GEMM-heavy FFN and MoE operators. However, attention and FFN/MoE benefit differently from batching. Attention remains memory-bandwidth-bound and exhibits low tensor-core utilization even as batch size increases, as shown in Figure 2a. Moreover, as context length grows, the attention key-value (KV) cache reduces the maximum feasible batch size, as shown in Figure 2b, preventing FFN and MoE operators from reaching the batch sizes needed for efficient execution. Attention–FFN Disaggregation Prior heterogeneous ODS systems, including Step-3 and MegaScale-Infer, primarily adopt a fixed attention–FFN disaggregation (AFD) design rather than searching over general operator partitions (StepFun et al., 2025; Zhu et al., 2025). The attention stage executes the full attention module, including the QKV and output projections, while the FFN/MoE stage executes feed-forward or expert computation. Because attention and FFN/MoE alternate within each Transformer layer, hidden states must be transferred between the device groups at each stage boundary, as shown in Figure 3. These systems divide batches into microbatches and pipeline their execution to keep both stages busy while overlapping inter-stage communication with computation. Consequently, performance depends strongly on stage balance: latency mismatches create pipeline bubbles and reduce utilization. Although prior systems demonstrate the feasibility of AFD, their fixed partitions leave unresolved when heterogeneous ODS reduces serving cost and what determines its benefit, the question we address next through theoretical analysis. 3. Theoretical Analysis 3.1. Analytical Setup Scope and assumptions. We analyze steady-state decode serving with memory-bound attention and GEMM-dominated projections, FFN, and MoE computation. Each GPU type has unit-time monetary cost c, memory bandwidth β, memory capacity M, and peak compute throughput F; we minimize GPU cost per decoded token. We model attention execution time as memory traffic divided by memory bandwidth, and GEMM execution time as FLOP count divided by peak GPU compute throughput only in the compute-bound regime. We assume disaggregation enables GEMM batches large enough for compute-bound execution, and that microbatching and pipelining hide inter-device communication. Appendices A.1 and A.2 give the operator-level roofline model and formal regime assumptions. Figure 5. Heterogeneous vs. homogeneous ODS for Qwen3-235B on H100 and H20 GPUs. The heterogeneous setup runs core attention on H20 and GEMM-dominated operators on H100. Left: normalized cost per token. Right: cost gain over the best homogeneous deployment. Cost model. Let mkv(s)m_kv(s) be the KV-cache memory per request at average sequence length s. Let PactP_act be the effective number of GEMM-side parameters activated per request after model-parallel sharding, and let MweightsM_weights be the resident model-weight footprint on the modeled device group, including inactive MoE experts. For GPU type j, define αj≜cj/βj _j c_j/ _j and γj≜cj/Fj _j c_j/F_j as monetary cost per byte of memory traffic and per FLOP, respectively. Under the regime assumptions above, homogeneous ODS on type j has cost (1) CPTjhom(s)=αjmkv(s)+2γjPact,CPT^hom_j(s)= _jm_kv(s)+2 _jP_act, while heterogeneous ODS with attention on GPU type j and GEMM-dominated operators on type k has cost (2) CPTj,khet(s)=αjmkv(s)+2γkPact.CPT^het_j,k(s)= _jm_kv(s)+2 _kP_act. For colocated serving on type j with memory capacity M, define the continuous relaxation b¯(s) b(s) and the exact integer batch limit bmax(s)b_ (s) as (3) b¯(s)≜M−Mweightsmkv(s),bmax(s)=⌊b¯(s)⌋. b(s) M-M_weightsm_kv(s), b_ (s)= b(s) . For feasible sequence lengths, the minimum colocated cost under the continuous batch-size relaxation is (4) CPTjcoloc(s)=αjmkv(s)+max(αjMweightsb¯(s), 2γjPact).CPT^coloc_j(s)= _jm_kv(s)+ \! ( _jM_weights b(s),\,2 _jP_act ). The largest relaxed batch b¯(s) b(s) attains this minimum because cost is non-increasing in batch size. The maximum captures whether GEMM is limited by weight loading or compute. GEMM threshold. For local batch size b, Eqs. 1–4 use the approximations that attention traffic scales as bmkv(s)b\,m_kv(s), active GEMM work as 2bPact2bP_act, and GEMM weight traffic as MweightsM_weights. Equating GEMM weight-loading time and compute time gives the threshold b∗=MweightsF2Pactβb^*= M_weightsF2P_actβ, which is the batch size at which GEMM becomes compute-bound. Appendix A.3 derives the cost expressions, and Appendix A.4 states the scaling approximations and threshold derivation in full. 3.2. Homogeneous Disaggregation vs. Colocation We first analyze when homogeneous operator-level disaggregation reduces cost relative to colocation, and how that gain depends on sequence length. We fix the GPU type and omit its index. We restrict the comparison to sequence lengths for which colocation is feasible, i.e., bmax(s)≥1b_ (s)≥ 1, equivalently b¯(s)≥1 b(s)≥ 1, and assume that mkv(s)m_kv(s) is strictly increasing on this domain. Define the cost gain of homogeneous disaggregation over colocation as Ghom(s)≜CPTcoloc(s)CPThom(s)G_hom(s) CPT^coloc(s)CPT^hom(s). Theorem 3.1 (Cost gain of homogeneous disaggregation over colocation). Consider a homogeneous GPU setting with parameters (c,β,M,F)(c,β,M,F). Suppose attention is memory-bound and adopt the scaling approximations from Section 3.1. Let b∗b^* be the GEMM compute-bound threshold (Section 3.1) and let b¯(s) b(s) be the relaxed feasible batch size defined in Eq. 3. Then, for every feasible sequence length s with bmax(s)≥1b_ (s)≥ 1: (1) (No gain at short sequences.) If b¯(s)≥b∗ b(s)≥ b^*, then Ghom(s)=1G_hom(s)=1. (2) (Strictly increasing gain.) On the feasible region where b¯(s)<b∗ b(s)<b^*, Ghom(s)>1G_hom(s)>1 and GhomG_hom is strictly increasing in s. (3) (Bounded gain.) The gain satisfies (5) 1≤Ghom(s)<M−Mweights.1\;≤\;G_hom(s)\;<\; MM-M_weights. Homogeneous disaggregation helps only after KV-cache growth pushes the colocated feasible batch size below the GEMM compute-bound threshold. Beyond that point, the gain increases monotonically with sequence length but remains bounded by M/(M−Mweights)M/(M-M_weights). Disaggregation improves efficiency by decoupling GEMM batching from KV-cache memory pressure, so its benefit is fundamentally limited by how much device memory is already consumed by model weights. Figure 4 shows cost gains and upper bounds calculated from our theoretical model for Qwen3-235B and GPT-OSS-120B on H100 under different parallelism configurations. See Appendix A.5.1 for the proof. 3.3. Heterogeneous vs. Homogeneous Disaggregation Optimizing Eqs. 1–2 over all available GPU types j and k gives (6) CPT⋆hom(s) ^hom_ (s) =minjCPTjhom(s), = _jCPT^hom_j(s), (7) CPT⋆het(s) ^het_ (s) =minj(αj)mkv(s)+mink(γk) 2Pact. = _j( _j)\,m_kv(s)+ _k( _k)\,2P_act. The optimized heterogeneous deployment assigns attention to the GPU type with the lowest cost per byte and GEMM-dominated operators to the type with the lowest cost per FLOP; the homogeneous deployment uses one type for both. Define Ghet(s)≜CPT⋆hom(s)CPT⋆het(s)G_het(s) CPT^hom_ (s)CPT^het_ (s) as the gain over homogeneous disaggregation. For the theorem below, restrict these minima to j,k∈1,2j,k∈\1,2\ and define η≜(β1F2)/(β2F1)η ( _1F_2)/( _2F_1) as the hardware ratio. Assume η≥1η≥ 1 without loss of generality. Let ℐI be a sequence-length interval where all compared ODS deployments are feasible and mkv(s)m_kv(s) is continuous and strictly increasing. Theorem 3.2 (Cost gain of heterogeneous over homogeneous disaggregation). Consider two GPU types with parameters (β1,F1)( _1,F_1) and (β2,F2)( _2,F_2), and respective unit-time costs c1c_1 and c2c_2. Adopt the scaling approximations of Section 3.1. Then, over the feasible interval ℐI: (1) (Non-negative gain.) For every s∈ℐs , Ghet(s)≥1G_het(s)≥ 1. (2) (No gain under dominance.) If one GPU type dominates the other on both cost-efficiency metrics, i.e., αj≤αk _j≤ _k and γj≤γk _j≤ _k for some j≠kj≠ k, then Ghet(s)=1G_het(s)=1 for every s∈ℐs . (3) (Unimodal and bounded gain.) Otherwise, under the labeling η≥1η≥ 1, we have α1<α2 _1< _2 and γ2<γ1 _2< _1. Define the homogeneous-cost crossing x∗≜2Pact(γ1−γ2)α2−α1.x^* 2P_act( _1- _2) _2- _1. As a function of x=mkv(s)x=m_kv(s), the gain strictly increases for x<x∗x<x^* and strictly decreases for x>x∗x>x^*. If x∗∈mkv(ℐ)x^*∈ m_kv(I), the gain therefore has a unique peak at the corresponding s∗∈ℐs^* ; otherwise, it is monotone over ℐI. Moreover, for every s∈ℐs , (8) Ghet(s)≤1+η2.G_het(s)\;≤\; 1+ η2. The theorem shows that heterogeneous assignment helps only when the two GPU types trade off bandwidth efficiency against compute efficiency, rather than one dominating on both dimensions. When the homogeneous-cost crossing lies in the feasible interval, heterogeneity helps most there: at shorter sequences the best homogeneous GPU already matches the compute side well, while at longer sequences the best homogeneous GPU already matches the bandwidth side well. If the crossing lies outside the feasible interval, only the increasing or decreasing branch appears in that interval. Figure 5 shows normalized costs per token and the gain of heterogeneous disaggregation over the best homogeneous deployment, calculated from our theoretical model for Qwen3-235B using H100 and H20 GPUs. The full proof is given in Appendix A.5.2. 3.4. From Bounds to Plans: CAD and Its Limits The bounds above assume that attention and the remaining operators run on hardware suited to their bottlenecks and that communication is fully hidden by computation. Core-Attention Disaggregation (CAD) implements this separation by placing core attention on one device group and the remaining projection, FFN, and MoE GEMMs on another. Under the assumptions in Section 3.1, attaining the corresponding idealized ODS cost requires balanced stage latencies and sufficient microbatching to hide communication (Appendix A.3). The first limitation, however, is whether stage balance is feasible. GPU memory must accommodate the KV caches of all resident microbatches and attention layers, constraining each attention operation’s KV-cache size and hence its maximum latency under our memory-bound model. GEMM-based operators, meanwhile, must load their weights from HBM, imposing a latency floor that our compute-bound cost model does not capture. If this floor exceeds the maximum feasible attention latency, the two stages’ feasible latency ranges do not overlap, and the attention stage remains idle for part of every pipeline cycle. Higher memory bandwidth on the GEMM devices lowers this floor and can therefore make stage balance feasible. Figure 6 illustrates both cases, and Appendix A.6 derives the overlap condition for uniform-layer models. The NVIDIA Vera Rubin and Groq 3 LPX design illustrates this principle: it retains the KV cache and decode attention on GPUs while placing FFN or MoE weights in LPX’s SRAM (Aubrey and Ghodsian, 2026). SRAM residency reduces weight-loading latency, lowering the FFN/MoE stage’s latency floor and potentially enabling its feasible latency range to overlap with the attention stage’s. Although this design uses AFD rather than CAD, the same two-stage balance principle applies. Figure 6. Feasibility of balanced CAD. Case 1: the attention and rest-of-model latency ranges overlap, admitting a balanced operating point. Case 2: the ranges are disjoint, so pipeline bubbles are unavoidable. The second limitation is layerwise variation. Hybrid architectures commonly mix full attention with sliding-window, linear, or recurrent sequence operators (Gemma Team et al., 2025; OpenAI et al., 2025; Qwen Team, 2025; Kimi Team et al., 2025; Ai2, 2026). Full attention loads KV cache for the entire context, whereas efficient operators access a bounded window or compact recurrent state. CAD places both on the same attention device group, but the rest-of-model stage has one fixed latency for a given microbatch size and cannot match both layer types. This mismatch leaves one stage waiting for the other at some layers, creating the pipeline bubbles shown in Figure 7. Figure 7. CAD pipeline schedule for a hybrid-attention model. A fixed rest-of-model stage cannot balance both efficient- and full-attention layers, leaving idle pipeline slots. 4. Serving-Plan Formulation CAD’s limitations motivate searching over general operator partitions rather than fixed module splits. Our formulation jointly determines how operators are partitioned into stages, which device-group types execute them with what hardware and parallelism, and how resident batches and microbatches are sized. It evaluates each plan using a steady-state latency model that accounts for both stage execution and inter-stage activation transfers, and minimizes serving cost subject to memory and TPOT constraints. Partition-block abstraction. The key challenge is to capture cross-layer heterogeneity without searching an unconstrained partition of the entire model. We address it with a partition block, the smallest layer-aligned operator pattern that repeats across the model. A partition block may contain one layer in a uniform transformer or span a full period of full-attention, sliding-window, and other layer types in a hybrid-attention model. Within a block of NblockN_block ordered operators, the planner chooses a contiguous stage template =(q0,…,qS)q=(q_0,…,q_S), where 0=q0<⋯<qS=Nblock0=q_0<·s<q_S=N_block and stage j contains operators qj−1+1,…,qjq_j-1+1,…,q_j. Stages may span layer boundaries, allowing the planner to balance heterogeneous operators jointly across layers. Reusing the resulting template across all partition blocks preserves the model’s regularity and reduces the partition search to one repeating unit. Each stage then serves as the basic unit of placement and pipeline execution; Appendix B.1 provides the formal DAG, ordering, and block definitions. Structured placement and batching. Given the stage template, the planner determines where each stage executes and with what resources. We represent the deployment using K ordered device-group types. Type m has configuration (hm,πm)(h_m, _m), consisting of a GPU type and a supported parallelism strategy such as tensor or expert parallelism, and may be instantiated by multiple replicas. All replicas of a type execute the same assigned stage positions while processing disjoint subsets of the global batch. To obtain a reusable pipeline structure, we require S=kKS=kK for some positive integer k and assign stage j using the structured round-robin map σ(j)=((j−1)modK)+1σ(j)=((j-1) K)+1. Each type therefore owns the same k stage positions in every partition block, so one template and pipeline schedule serve the entire model even though the stages’ operator contents and execution costs may differ. A replica of type m has resident batch size bmb_m, divided into μ microbatches of size b^m=bm/μ b_m=b_m/μ. Communication-aware latency model. Plan latency depends on both stage execution and communication across stage boundaries. For each stage j, let tj=flat(⋅)t_j=f_lat(·) denote its profiled execution latency under the assigned device-group configuration and microbatch size. Let uj=fcomm(⋅)u_j=f_comm(·) denote the profiled transfer latency after stage j, including any redistribution required by the source and destination parallel layouts. The boundary after stage S connects to stage 1 of the next partition block and is omitted after the final block. With BtotalB_total repeated partition blocks, the steady-state pipeline round-trip time is Trtt=frtt(μ,Btotal,,)T_rtt=f_rtt(μ,B_total,t,u), where =(t1,…,tS)t=(t_1,…,t_S) and =(u1,…,uS)u=(u_1,…,u_S). We evaluate frttf_rtt by schedule simulation, which enforces execution and transfer dependencies, models resource contention, and permits computation–communication overlap only when the schedule and resources allow. Appendices B.2 and B.3 provide the complete placement, batching, execution, and communication definitions. Design-space expressiveness. The planner can isolate expensive full-attention operators, combine efficient-attention operators with neighboring projections, or map repeated stage positions to heterogeneous device-group types while reusing one block-level template. This flexibility directly addresses the layerwise imbalance of Section 3.4. AFD and CAD are recovered as restricted two-way templates: AFD separates complete attention modules from FFN or MoE operators, whereas CAD isolates only the core attention kernels; colocated serving is the single-stage template. Optimization problem. A serving plan is specified by =(S,K,,((,,,,,))m=1K,μ),P= (S,K,q, ((h_m, _m,b_m) )_m=1^K,μ ), where q is the stage template defined above. The variables S, K, μ, and bmb_m are positive integers with K|SK S and μ|bmμ b_m for every type m. Each hmh_m is drawn from the available hardware types, and πm _m is a parallelism strategy supported by hmh_m. The indexed sequence of device-group configurations is ordered because stage j is assigned to type σ(j)σ(j). (9a) minCPTplan _P _plan =Trtt∑m=1Kcmbm =T_rtt _m=1^K c_mb_m (9b) s.t.Trtt .t. T_rtt ≤TSLO, ≤ T_SLO, (9c) Mweights,m+μKVm(b^m,s) M_weights,m+μ\,KV_m( b_m,s) ≤Mm,m=1,…,K. ≤ M_m, m=1,…,K. The objective in equation 9a minimizes serving cost per generated token. For device-group type m, cmc_m is the unit-time cost of one replica. Thus, ∑mcm/bm _mc_m/b_m is the aggregate cost rate per global resident request; since each resident request produces one token per pipeline round trip, multiplying by TrttT_rtt gives the steady-state cost per token. Appendix B.5 derives this objective from the global batch size and replica counts. The constraints require the pipeline round-trip time to meet the TPOT SLO and each replica to fit its assigned model weights and the resident KV cache for all μ microbatches at sequence length s; Appendix B.4 provides the complete memory notation. Coupled search challenge. The planning decisions cannot be optimized independently: stage boundaries, device-group configurations, parallelism strategies, resident batch sizes, and microbatch count jointly determine execution latency, transfer overhead, memory feasibility, and replica count. A lower-cost device may increase total cost if its smaller feasible batch size requires more replicas, while a compute-balanced partition may miss the TPOT SLO after boundary transfers are scheduled. Section 5 exploits partition-block regularity and Pareto structure to search this coupled space efficiently. 5. Algorithms The serving-plan search in Section 4 is difficult for three reasons. First, independently placing every legal stage boundary creates a combinatorial partition space. Second, each device-group type, which we call an owner because it owns fixed stage positions in every partition block, admits many hardware, parallelism, and batching configurations. Third, the cross-owner product of these configurations is too large to enumerate and simulate exhaustively. Our planner addresses them in order: it parameterizes partitions with regularity-aware sub-block templates, constructs a network-safe Pareto frontier for each owner, and combines the surviving frontiers using exact branch-and-bound. Only combinations that survive all three reductions reach the schedule simulator. 5.1. Regularity-Aware Sub-Block Templates An unrestricted partition that chooses every stage boundary independently produces (Nblock−1S−1) N_block-1S-1 templates for a block with NblockN_block operators and S stages, even though the same operator order often repeats across many layers. This redundancy is especially large in hybrid-attention models: layers may use different attention variants while retaining the same sequence of attention projections, attention core, and FFN or MoE operators. We remove it by searching a smaller sub-partition block. Let LblockL_block be the number of layers in a partition block, and let LsubL_sub be a sub-block length in layers that divides LblockL_block and preserves the repeated layer pattern; each block then contains n=Lblock/Lsubn=L_block/L_sub sub-block repetitions. For a chosen number of owners K, the planner selects a legal contiguous template sub=(q0,…,qK)q_sub=(q_0,…,q_K) inside one sub-block and tiles it over the n repetitions. Stage position m in every repetition is assigned to owner m, giving S=nKS=nK stages per partition block (Section 4’s S=kKS=kK with k=nk=n). This parameterization reduces the number of templates to (Nblock/n−1K−1) N_block/n-1K-1 while retaining layer-dependent execution costs after the template is expanded; it is a structural restriction of the searched plan family, not a cost-based pruning heuristic. Figure 8. Distributed execution plane. Black dashed arrows carry activations between stages; the orange dashed path returns next-token IDs to the first-stage workers. Algorithm 1 Regularity-Aware Exact Serving-Plan Search 1: Model operator pattern, finite search grid, profiles, TPOT SLO 2: Cheapest feasible plan in the structured space 3: best←∞best←∞ 4: for each regular sub-block and legal template subq_sub do 5: Tile subq_sub over the model 6: for each feasible (B,μ)(B,μ) do 7: Build network-safe frontiers ℱ1,…,ℱKF_1,…,F_K from memory-feasible owner points 8: Explore(1,∅1, ) 9: return best plan 10: procedure Explore(i,Pi,P) 11: if optimistic resource and cost bounds for P rule out a feasible improvement over bestbest then return 12: if i>Ki>K then 13: Tighten bounds; if P survives, simulate and update bestbest if feasible and improved 14: return 15: for each p∈ℱip _i do 16: Explore(i+1,P∪pi+1,P∪\p\) 5.2. Network-Safe Per-Owner Pareto Frontiers The structural reduction still leaves many configurations for every owner. Given a template, global resident batch B, and microbatch count μ, a feasible point p for owner m chooses a hardware and parallelism configuration and a per-replica microbatch size b^m(p) b_m(p), which fix an integral replica count. Profiling gives the execution latencies m(p)=(tj(p))j∈mT_m(p)= (t_j(p) )_j _m of the stage positions m=j:σ(j)=mS_m=\j:σ(j)=m\ owned by m. The point also has normalized cost and GPU coordinates rm(p)=cm(p)/bm(p)r_m(p)=c_m(p)/b_m(p) and ρm(p)=gm(p)/bm(p) _m(p)=g_m(p)/b_m(p), where bm(p)=μb^m(p)b_m(p)=μ b_m(p) and gm(p)g_m(p) is the GPU count per replica. Latency and cost alone are insufficient for safe local pruning. Two points with similar execution latency can induce different replica counts, total GPU use, and per-endpoint transfer sizes when combined with neighboring owners. We therefore compare points using (10) m(p)=(m(p),rm(p),ρm(p),b^m(p)).z_m(p)= (T_m(p),r_m(p), _m(p), b_m(p) ). Point p dominates point p′p only if both use the same tensor-parallel degree, p is no worse in every coordinate of equation 10, and it is strictly better in at least one; the degree restriction is needed because transfer volume varies non-monotonically with tensor parallelism. The GPU coordinate preserves the planner’s preference for fewer GPUs among otherwise equal plans; the microbatch coordinate preserves the inputs of the batch-dependent transfer model. This dominance relation yields the network-safe frontier ℱmF_m. Retaining mT_m as a per-occurrence vector prevents a point that accelerates one layer type but slows another from being removed, so the frontier eliminates only configurations that cannot improve any feasible complete plan. 5.3. Exact Branch-and-Bound over Frontier Products Explicitly evaluating ℱ1×⋯×ℱKF_1×·s×F_K can still require exponentially many schedule simulations, so the planner explores the product with depth-first branch-and-bound. For each prefix of assigned owners, it forms an optimistic completion from the best remaining coordinates of the unassigned frontiers: mandatory serialized compute work lower-bounds each owner’s latency contribution, and mandatory transfer work at each network endpoint lower-bounds communication, with unassigned endpoints minimized independently over feasible replica counts, keeping the estimate optimistic. The larger of the two is a valid lower bound on TrttT_rtt, and multiplying it by the suffix-minimal normalized cost lower-bounds the objective. A subtree is discarded if its optimistic completion violates the TPOT SLO or cannot improve the incumbent. Once every owner is assigned, the bounds are recomputed with the complete configuration, and only survivors invoke the schedule simulator. Appendix C.1 states these bounds formally. The schedule simulator remains authoritative for every surviving candidate, so the planner is exact over the configured finite hardware, batching, and template grid; its worst-case complexity remains ∏m|ℱm| _m|F_m| when no bound prunes. 6. System Design and Implementation OpWeave is implemented in 32K lines of Python and C++ on the vLLM GPU model runner (Kwon et al., 2023); each worker loads only its assigned model components and state. Inter-node activation transfers use point-to-point NCCL, and intra-node fanout uses CUDA interprocess communication (IPC) peer copies with GPU doorbells. The planner-selected batch size fixes the plan’s scheduling capacity: logical slots, microbatch shapes, stage placement, and per-rank action order are static, so CUDA graphs and buffers are bound once, while token, position, and KV-cache metadata change between iterations. 6.1. System Overview and Execution Flow Runtime organization. Figure 8 shows the distributed execution plane after plan installation. A OpWeave deployment consists of a global coordinator (outside the figure) and replicas of the device-group types selected by the plan; each replica runs a staged engine with one GPU worker process per GPU rank. The coordinator acts only at setup, instantiating the replicas and installing the plan; during decoding, workers execute the compiled stage schedule without coordinator or RPC intervention between stages and microbatches. Table 1. GPU specifications (NVIDIA, 2026b; NVIDIA, 2026c; NVIDIA, 2026a). Prices are averaged across selected public clouds. GPU type Memory BW (GB/s) Memory (GB) BF16 (TFLOPS) Price ($/GPU-h) H100 SXM 3,350 80 989 3.49 L40S 864 48 362.05 1.09 A100 SXM 2,039 80 312 1.74 Request and stage execution. First-stage workers consume the current token IDs of batched requests. Each microbatch traverses the ordered stages, transferring activations whenever consecutive stages reside on different replicas; terminal workers produce next-token IDs, which return to the first-stage workers on the GPU (Section 6.2). A stage runs once its input activation has arrived, enforced by receive-completion events, and the preceding compute action in its worker’s fixed action order has completed; independent computation and communication still overlap. 6.2. Optimizations Fine-grained disaggregation creates many short compute and transfer actions per token. If each traverses Python or becomes a separate network operation, its overhead can erase the benefit of finer placement. We apply three optimizations. Single native submission. A naive executor returns to Python for every stage, send, and receive, placing host dispatch between otherwise short GPU actions. Instead, OpWeave compiles the complete per-rank sequence; for each token, one native submission enqueues all CUDA-graph launches, staging copies, communication rounds, and event dependencies, eliminating Python dispatch, allocation, and RPCs between stages and microbatches. GPU-resident decoding. A centralized implementation would copy generated token IDs to the host, assemble and redistribute them, and rebuild execution metadata at every step, serializing a CPU–GPU round trip into each token. In OpWeave, terminal workers compute token IDs on the GPU, one NCCL all-reduce distributes the token vector to embedding owners, and tokens stay device-resident between iterations; only global rank zero retains the token history for eventual host-side output. Each stage reuses preconstructed metadata and preallocated buffers, updating only the tensors that change between iterations. Low-overhead communication. Our runtime packs compatible tensors into shared buffers and batches transfers into NCCL rounds, reducing per-operation overhead without increasing activation payload. When an unsharded activation crosses a stage boundary into a TP group, the runtime sends one inter-node copy to a destination leader rather than one copy per rank; the leader fans it out to the remaining TP ranks locally through CUDA IPC, and leader assignment rotates across routes to balance cross-node traffic. Table 2. Evaluated models and TPOT SLOs. Eff. Attn. is the efficient attention mechanism interleaved with full attention; Pattern is the ratio of efficient- to full-attention layers. Model MoE Eff. Attn. Pattern Strict SLO (ms) Relaxed SLO (ms) Gemma-3-27B ✗ SWA 5:1 25 60 Qwen3-Next-80B-A3B ✓ Linear 3:1 40 80 Figure 9. Measured serving cost across context lengths and TPOT SLOs. Crosses mark infeasible configurations. Labels above each group show g×g× cheaper, where g is the best feasible baseline cost divided by OpWeave cost.Four framed grouped bar charts in one horizontal row show Gemma-3-27B under 25- and 60-millisecond TPOT SLO labels, followed by Qwen3-Next-80B-A3B under 40- and 80-millisecond labels. Each panel has 8K, 32K, and 128K context groups with four slots ordered as Colocation, AFD, CAD, and . Bar heights retain measured USD per million output tokens, with an independent zero-based linear scale for each panel. Crosses mark the eight infeasible configurations. Small labels above each context group show the cheapest feasible Colocation, AFD, or CAD baseline cost divided by cost; a ratio below one indicates higher cost. Panel captions are below the charts and a shared legend is above. Figure 10. Homogeneous-machine evaluation results. OpW denotes OpWeave. (a) and (d): Mean number of pipeline stages. (b) and (e): Mean inter-node stage-to-stage payload per output token (MiB). (c) and (f): Mean price-weighted compute occupancy (Section 7.2).Six panels in one horizontal row, with Gemma-3-27B in the first three and Qwen3-Next-80B-A3B in the last three. Panels a and d show average full-model pipeline stage counts for CAD, AFD, and : 124, 124, and 20.67 for Gemma; 96, 96, and 24 for Qwen3-Next. Panels b and e show mean inter-node stage-to-stage payload in MiB per output token: 6.54, 8.84, and 0.39 for Gemma; 2.29, 2.16, and 0.35 for Qwen3-Next. Panels c and f show price-weighted compute occupancy: 49.71, 47.01, and 88.08 percent for Gemma; 51.45, 54.37, and 81.74 percent for Qwen3-Next. (a) Gemma-3-27B. (b) Qwen3-Next-80B-A3B. Figure 11. Mean per-GPU batch size at the relaxed TPOT SLO, normalized to OpWeave within each context and averaged across contexts. Higher is better.Two bar charts in one horizontal row show mean normalized per-GPU batch size at the relaxed TPOT SLO for Colocation, AFD, CAD, and . The Gemma-3-27B values are 0.68, 0.38, 0.39, and 1.00. The Qwen3-Next-80B-A3B values are 0.36, 0.98, 0.98, and 1.00. Figure 12. Simulated and measured TPOT for AFD, CAD, and OpWeave, normalized by measured TPOT. Labels show signed simulation errors.Two grouped bar charts in one horizontal row compare simulation and real-machine TPOT on the same zero-based normalized scale, with Gemma-3-27B on the left and Qwen3-Next-80B-A3B on the right, and one shared legend centered above the panels. Each pair is normalized by its real-machine TPOT, so all real-machine bars equal 1. For AFD, CAD, and respectively, Gemma simulation bars are 0.935, 0.939, and 0.952; Qwen3-Next simulation bars are 0.990, 0.914, and 1.033. Labels above the groups show signed percentage differences: -6.5\%, -6.1\%, and -4.8\% for Gemma, and -1.0\%, -8.6\%, and +3.3\% for Qwen3-Next. 7. Evaluation 7.1. Evaluation Setup Devices and Models. Table 1 lists the GPUs used in our evaluation. We conduct the homogeneous evaluation on a cluster of H100 nodes connected by RDMA over Converged Ethernet (RoCE). For the simulator fidelity tests, we use H100 and L40S GPUs on AWS, where RDMA transfers stage GPU data through host memory. We evaluate two representative hybrid-attention models (Gemma Team et al., 2025; Qwen Team, 2025), with their architectures and TPOT SLOs summarized in Table 2. Metrics. We use serving cost per million output tokens as our primary efficiency metric and time per output token (TPOT) as our latency metric. A configuration satisfies a target TPOT service-level objective (SLO) if its p95 step-level TPOT across the measured iterations does not exceed the target; for each serving policy, we report the lowest-cost configuration that satisfies the SLO. Baselines. We compare OpWeave against three baselines: colocated serving (COL), attention–FFN disaggregation (AFD) (Zhu et al., 2025; StepFun et al., 2025), and core-attention disaggregation (CAD). COL is implemented with vLLM 0.26.0 (Kwon et al., 2023) and executes all operators on GPUs within the same node. AFD and CAD are implemented in our OpWeave runtime as fixed-partition special cases. AFD separates complete attention modules from FFN/MoE operators, whereas CAD isolates core-attention kernels from all remaining operators; their two device groups may be placed on different nodes. This setup gives COL a well-optimized production implementation and keeps the runtime identical across disaggregated policies. Search Configuration. For each model, workload, and TPOT SLO, our planner searches the serving-plan space of Section 5 for the lowest-cost plan meeting the SLO. We restrict each plan to at most 32 GPUs, three device-group types, four replicas per type, and four microbatches. OpWeave and all baselines use the same search configuration and resource constraints; their admissible plans differ only in the imposed disaggregation policy. The planner returns the optimal feasible plan within this configured finite search space. Workloads. We evaluate steady-state decoding using long-context workloads derived from OpenThoughts3-1.2M (Guha et al., 2025). We tokenize the dataset to obtain an empirical distribution of generation-time context lengths and rescale it to construct synthetic request cohorts with mean context lengths of 8K, 32K, and 128K tokens. Requests are constructed deterministically, and all serving policies use the same distribution and construction procedure, ensuring comparable workloads. Figure 13. Simulated serving cost across context lengths and TPOT SLOs, normalized to heterogeneous OpWeave within each context. Labels show g×g× cheaper, where g is the best feasible non-OpWeave baseline cost divided by heterogeneous OpWeave cost. 7.2. Homogeneous End-to-End Evaluation We evaluate OpWeave and all baselines on H100 GPUs across both models, both SLOs (Table 2), and all three context lengths. As shown in Figure 9, OpWeave achieves the lowest serving cost in 11 of the 12 evaluated model–context–SLO settings. OpWeave is up to 1.78×1.78× cheaper than the best feasible baseline for Gemma-3-27B and up to 1.76×1.76× cheaper for Qwen3-Next-80B-A3B. OpWeave finds feasible plans in all evaluated scenarios and achieves lower serving costs than AFD and CAD wherever they are feasible. Its first advantage is lower latency through fewer pipeline stages. Whereas AFD and CAD impose two stages per layer, OpWeave selects stage boundaries across an entire partition block, enabling a larger, more flexible partition space. As shown in Figure 10(a) and (d), the selected OpWeave plans average only 20.7 stages for Gemma-3-27B and 24 for Qwen3-Next-80B-A3B, compared with 124 and 96, respectively, for both baselines. Fewer stage boundaries reduce network transfers: Figure 10(b) and (e) show that OpWeave reduces mean inter-node data transferred per output token by factors of 16.9 and 22.8 relative to CAD and AFD, respectively, for Gemma, and by factors of 6.5 and 6.1 for Qwen3-Next. Because inter-stage communication lies on the execution critical path of each microbatch, this reduction lowers decoding latency, helping OpWeave satisfy strict TPOT SLOs in settings where neither baseline finds a feasible plan. OpWeave’s second advantage is higher hardware occupancy by model execution, reducing idle time during decoding. We report system-level occupancy as the average of per-replica occupancy, weighting each replica type by its price times its replica count, i.e., the fraction of hardware spending that performs model execution. As shown in Figure 10(c) and (f), OpWeave achieves mean compute occupancy above 80% for both models, while CAD and AFD leave roughly half of the hardware time unused. One reason for the baselines’ lower occupancy is heterogeneity across layers: the selected hybrid models interleave attention mechanisms with substantially different execution latencies, making their fixed partitions difficult to balance. For example, at 128K context and batch size 1 on one H100, our profiles estimate 700 μ for a Qwen3-Next full-attention module versus 58 μ for linear attention, a 12×12× gap. OpWeave’s flexible partition boundaries allow it to balance work across these layers. Moreover, AFD and CAD’s high network transfer volume leaves limited latency headroom for increasing the number of microbatches to overlap work across device groups. Even under Gemma’s relaxed 60-ms SLO, their selected plans still use only one microbatch. The two device groups therefore execute dependent stages sequentially, leaving one group idle while the other works. OpWeave also alleviates the memory contention that limits batching in colocated serving. The attention KV cache competes with model weights for GPU memory, so longer contexts reduce the maximum feasible batch size. OpWeave mitigates this constraint by decoupling GEMM batching from KV-cache memory pressure. Consistent with our theoretical analysis, Figure 11 shows that OpWeave achieves larger mean per-GPU batch sizes than colocation for both models under the relaxed SLOs. AFD and CAD similarly alleviate this constraint for Qwen3-Next, achieving batch sizes close to OpWeave’s. For Gemma, however, their high communication latency forces them to use smaller batches to satisfy the TPOT SLO, resulting in lower per-GPU batch sizes even than colocation. 7.3. Simulator Fidelity We use the simulator for large-scale heterogeneous evaluation because real-machine experiments at this scale are prohibitively expensive and heterogeneous clusters with high-speed inter-node connections are difficult to allocate on public clouds. We validate it using two H100 and two L40S GPUs for Gemma-3-27B and four of each for Qwen3-Next-80B-A3B. As shown in Figure 12, the average absolute simulation errors across AFD, CAD, and OpWeave are 5.8% for Gemma-3-27B and 4.3% for Qwen3-Next-80B-A3B. 7.4. Heterogeneous End-to-End Evaluation We use our simulator to evaluate OpWeave and all baselines on two heterogeneous setups, H100 with L40S and H100 with A100 (Table 1), using the relaxed TPOT SLOs and all three context lengths. As shown in Figure 13, heterogeneous OpWeave achieves lower serving cost than all non-OpWeave baselines in all 12 evaluated model–hardware–context settings. On H100 with L40S, heterogeneous OpWeave is up to 1.54×1.54× cheaper for Gemma-3-27B and 1.59×1.59× cheaper for Qwen3-Next-80B-A3B than the best feasible non-OpWeave baseline. On H100 with A100, heterogeneous OpWeave is up to 1.89×1.89× and 1.66×1.66× cheaper, respectively. OpWeave thus retains its cost advantage over non-OpWeave baselines in heterogeneous settings. Compared with homogeneous OpWeave, heterogeneous OpWeave is 1.04×1.04× cheaper on average (arithmetic mean of per-setting cost ratios across both models and contexts) and up to 1.11×1.11× cheaper on H100 with L40S. On H100 with A100, heterogeneous OpWeave is 1.14×1.14× cheaper on average and up to 1.28×1.28× cheaper. The larger benefit on H100 with A100 is consistent with the analysis in Section 3.3: a greater disparity in the GPUs’ compute-to-memory-bandwidth ratios permits a higher upper bound on the idealized gain from heterogeneous assignment. Using the specifications in Table 1, the hardware ratio η is 1.42 for H100 with L40S and 1.93 for H100 with A100, yielding idealized gain bounds of 1.10×1.10× and 1.19×1.19×, respectively, under Eq. 8. The measured gains can exceed these idealized bounds because the bounds assume both deployments fully disaggregate memory-bound attention from compute-bound GEMM-based operators, which neither deployment achieves in practice. Figure 14. Network sensitivity of OpWeave, AFD and CAD.Two line plots compare AFD, CAD, and . Latency is swept over 5, 20, 50, 200, and 500 microseconds at 32 GB/s; bandwidth is swept over 1, 2, 4, 8, 16, and 32 GB/s at 20 microseconds. has the lowest reported cost at every comparable setting. AFD and CAD have no reported winner at 500 microseconds. HW Method Candidates Simulations Time (s) H100 Full 4.78M (1.00×) 4.78M (1.00×) 228.3 (1.00×) No-PF 8.83M (1.85×) 8.83M (1.85×) 264.4 (1.16×) No-B&B 219.79M (45.96×) 4.78M (1.00×) 290.3 (1.27×) Neither 293.24M (61.31×) 8.83M (1.85×) 330.5 (1.45×) A100 +H100 Full 13.89M (1.00×) 13.89M (1.00×) 446.7 (1.00×) No-PF 52.14M (3.75×) 52.14M (3.75×) 776.4 (1.74×) No-B&B 630.97M (45.42×) 13.89M (1.00×) 636.6 (1.42×) Neither 2.33B (167.79×) 52.14M (3.75×) 1,495.8 (3.35×) Table 3. Planner search ablation. Full enables Pareto-frontier (PF) pruning and branch-and-bound (B&B); No-PF, No-B&B, and Neither disable the corresponding optimizations. Parentheses show degradation relative to Full on the same hardware. Lower is better; all methods find the same best plan. 7.5. Ablation Study and Sensitivity Study Planner Search Ablation. This ablation quantifies the benefits of Pareto-frontier pruning and branch-and-bound for planner search. We search for Gemma-3-27B plans at 32K context under a 60-ms TPOT SLO, using 64 CPU cores for every run. Table 3 reports candidates considered, simulator invocations, and end-to-end search time. Because inexpensive memory and SLO checks discard many candidates before the costly simulator is invoked, reducing candidate count does not translate proportionally into wall-clock time. Pareto-frontier pruning reduces simulator invocations, whereas branch-and-bound removes candidates before simulation. Disabling PF increases simulator invocations by 1.85×1.85× on H100 and 3.75×3.75× on A100+H100, increasing wall-clock time by 1.16×1.16× and 1.74×1.74×, respectively. Disabling B&B instead increases considered candidates by 45.96×45.96× and 45.42×45.42× without increasing simulator invocations; these are rejected by the cheaper checks, so wall-clock time increases by only 1.27×1.27× and 1.42×1.42×. Together, they make search 1.45×1.45× faster on H100 and 3.35×3.35× on A100+H100 than using neither. Network Sensitivity. We use simulation to evaluate the sensitivity of OpWeave, AFD, and CAD to network conditions, varying inter-node transfer latency and bandwidth for Gemma-3-27B at 8K context under a 60-ms TPOT SLO. Each method is reoptimized at every setting over H100, L40S, and mixed deployments. As shown in Figure 14, OpWeave is less sensitive than AFD and CAD to both network latency and bandwidth. This robustness follows from OpWeave’s much lower network transfer volume (Figure 10(b) and (e)): the remaining transfers can be overlapped with model execution, leaving serving performance less affected by network conditions. 8. Related Work LLM Serving Optimization. General-purpose LLM serving systems improve efficiency through continuous batching (Yu et al., 2022), paged KV-cache management (Kwon et al., 2023), KV reuse and prefix caching (Zheng et al., 2023; Qin et al., 2024), prefill–decode disaggregation and scheduling (Patel et al., 2024; Zhong et al., 2024; Agrawal et al., 2024), and speculative decoding (Chen et al., 2023; Cai et al., 2024; Chen et al., 2025; Chen et al., 2024; Huang et al., 2025; Leviathan et al., 2022; Li et al., 2024b; Li et al., 2024c; Li et al., 2025; Liu et al., 2024; Miao et al., 2024; Oliaro et al., 2024; Li et al., 2026). These techniques substantially improve LLM serving, but they do not directly address the different bottlenecks of attention and GEMMs. Operator-Level Disaggregated Serving. MegaScale-Infer and Step-3 AFD are closest to our setting: both use a fixed two-way split that disaggregates attention from FFN, MLP, or MoE during decoding (Zhu et al., 2025; StepFun et al., 2025). FastDecode similarly separates attention and its KV cache from the dense layers, but offloads attention to distributed CPU workers rather than another GPU group (He et al., 2024). Infinite-LLM distributes attention computation and KV-cache capacity across instances, rather than general operator-class placement (Lin et al., 2024). NanoFlow also operates at operation granularity, but overlaps compute, memory, and network work within a device instead of separating attention and FFN onto distinct device groups (Zhu et al., 2024). Unlike these systems, we provide a theoretical analysis of the achievable ODS cost gains, formalize the broader serving-plan space, and show why fixed two-way splits can fail for hybrid-attention architectures such as Gemma 3, GPT-OSS, and Qwen3-Next (Gemma Team et al., 2025; OpenAI et al., 2025; Qwen Team, 2025). Heterogeneous LLM Serving. A growing line of work serves LLMs across heterogeneous GPU resources. Helix formulates heterogeneous serving as a max-flow problem, jointly optimizing layer placement and request routing across GPU types (Mei et al., 2024). Other systems apply asymmetric pipeline parallelism to decentralized heterogeneous or volunteer GPUs (Jiang et al., 2024; Borzunov et al., 2023a; Borzunov et al., 2023b; Tong et al., 2025; Wu et al., 2025; Li et al., 2024a; Kim et al., 2025; Jiang et al., 2025b; Jiang et al., 2025a; Peng et al., 2025), or exploit price-performance gaps across GPU types by routing requests to the best-suited hardware (Griggs et al., 2024; Jiang et al., 2026; Mei et al., 2026). All of these systems partition at the transformer-layer boundary or coarser; in contrast, this paper disaggregates at the operator level, matching heterogeneous GPU types to operator classes with different resource bottlenecks inside a single layer. 9. Conclusion This paper presented OpWeave, an end-to-end framework for heterogeneous operator-level disaggregated serving that connects theory, planning, and execution. OpWeave develops an analytical cost model that characterizes and bounds the gains of homogeneous and heterogeneous ODS over colocated serving, a regularity-aware planner that jointly optimizes operator partitioning, hardware assignment, parallelism, and batching over the partition block abstraction, and a vLLM-based distributed runtime that executes the synthesized plans across heterogeneous device groups. In our evaluation, OpWeave is up to 1.78×1.78× cheaper than the best feasible baseline on homogeneous GPUs and up to 1.89×1.89× cheaper on heterogeneous clusters, while attaining latency SLOs. We hope the analysis and abstractions in OpWeave provide a foundation for future work on operator-level disaggregation across increasingly heterogeneous models and hardware. Acknowledgments We used ChatGPT, Claude, and Gemini to improve the clarity and readability of the manuscript. We also used Claude and GPT to assist with code development for this work. This work was partially supported by NSF awards CNS-2211882 and CNS-2239351, a Sloan Research Fellowship, and research awards from Amazon, Cisco, Google, Jane Street, Meta, NVIDIA, Oracle, Qualcomm, and Samsung. References Agrawal et al. (2024) Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, Alexey Tumanov, and Ramachandran Ramjee. 2024. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association. Ai2 (2026) Ai2. 2026. Introducing Olmo Hybrid: Combining Transformers and Linear RNNs for Superior Scaling. Allen Institute for Artificial Intelligence. https://allenai.org/blog/olmohybrid Aubrey and Ghodsian (2026) Kyle Aubrey and Farshad Ghodsian. 2026. Inside NVIDIA Groq 3 LPX: The Low-Latency Inference Accelerator for the NVIDIA Vera Rubin Platform. https://developer.nvidia.com/blog/inside-nvidia-groq-3-lpx-the-low-latency-inference-accelerator-for-the-nvidia-vera-rubin-platform/. NVIDIA Technical Blog. Accessed: 2026-04-16. Borzunov et al. (2023a) Alexander Borzunov, Dmitry Baranchuk, Tim Dettmers, Max Ryabinin, Younes Belkada, Artem Chumachenko, Pavel Samygin, and Colin Raffel. 2023a. Petals: Collaborative Inference and Fine-tuning of Large Models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations). Association for Computational Linguistics, 558–568. doi:10.18653/v1/2023.acl-demo.54 Borzunov et al. (2023b) Alexander Borzunov, Max Ryabinin, Artem Chumachenko, Dmitry Baranchuk, Tim Dettmers, Younes Belkada, Pavel Samygin, and Colin Raffel. 2023b. Distributed Inference and Fine-tuning of Large Language Models Over the Internet. In Advances in Neural Information Processing Systems (NeurIPS). Cai et al. (2024) Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. 2024. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads. arXiv preprint arXiv:2401.10774 (2024). Chen et al. (2023) Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. 2023. Accelerating Large Language Model Decoding with Speculative Sampling. arXiv preprint arXiv:2302.01318 (2023). Chen et al. (2025) Siyuan Chen, Zhipeng Jia, Samira Khan, Arvind Krishnamurthy, and Phillip B. Gibbons. 2025. SLOs-Serve: Optimized Serving of Multi-SLO LLMs. arXiv preprint arXiv:2504.08784 (2025). Chen et al. (2024) Zhuoming Chen, Avner May, Ruslan Svirschevski, Yuhsun Huang, Max Ryabinin, Zhihao Jia, and Beidi Chen. 2024. Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding. arXiv preprint arXiv:2402.12374 (2024). Dong et al. (2025) Xianzhe Dong, Tongxuan Liu, Yuting Zeng, Liangyu Liu, Yang Liu, Siyu Wu, Yu Wu, Hailong Yang, Ke Zhang, and Jing Li. 2025. HydraInfer: Hybrid Disaggregated Scheduling for Multimodal Large Language Model Serving. arXiv preprint arXiv:2505.12658 (2025). https://arxiv.org/abs/2505.12658 Gangasani et al. (2026) Vivek Gangasani, Andrew Smith, and Goutham Annem. 2026. Introducing Disaggregated Inference on AWS powered by llm-d. https://aws.amazon.com/blogs/machine-learning/introducing-disaggregated-inference-on-aws-powered-by-llm-d/. AWS Blog. Accessed: 2026-04-16. Gemma Team et al. (2025) Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, Gaël Liu, Francesco Visin, Kathleen Kenealy, Lucas Beyer, Xiaohai Zhai, Anton Tsitsulin, Robert Busa-Fekete, Alex Feng, Noveen Sachdeva, Benjamin Coleman, Yi Gao, Basil Mustafa, Iain Barr, Emilio Parisotto, David Tian, Matan Eyal, Colin Cherry, Jan-Thorsten Peter, Danila Sinopalnikov, Surya Bhupatiraju, Rishabh Agarwal, Mehran Kazemi, Dan Malkin, Ravin Kumar, David Vilar, Idan Brusilovsky, Jiaming Luo, Andreas Steiner, Abe Friesen, Abhanshu Sharma, Abheesht Sharma, Adi Mayrav Gilady, Adrian Goedeckemeyer, Alaa Saade, Alex Feng, Alexander Kolesnikov, Alexei Bendebury, Alvin Abdagic, Amit Vadi, András György, André Susano Pinto, Anil Das, Ankur Bapna, Antoine Miech, Antoine Yang, Antonia Paterson, Ashish Shenoy, Ayan Chakrabarti, Bilal Piot, Bo Wu, Bobak Shahriari, Bryce Petrini, Charlie Chen, Charline Le Lan, Christopher A. Choquette-Choo, CJ Carey, Cormac Brick, Daniel Deutsch, Danielle Eisenbud, Dee Cattle, Derek Cheng, Dimitris Paparas, Divyashree Shivakumar Sreepathihalli, Doug Reid, Dustin Tran, Dustin Zelle, Eric Noland, Erwin Huizenga, Eugene Kharitonov, Frederick Liu, Gagik Amirkhanyan, Glenn Cameron, Hadi Hashemi, Hanna Klimczak-Plucińska, Harman Singh, Harsh Mehta, Harshal Tushar Lehri, Hussein Hazimeh, Ian Ballantyne, Idan Szpektor, Ivan Nardini, Jean Pouget-Abadie, Jetha Chan, Joe Stanton, John Wieting, Jonathan Lai, Jordi Orbay, Joseph Fernandez, Josh Newlan, Ju yeong Ji, Jyotinder Singh, Kat Black, Kathy Yu, Kevin Hui, Kiran Vodrahalli, Klaus Greff, Linhai Qiu, Marcella Valentine, Marina Coelho, Marvin Ritter, Matt Hoffman, Matthew Watson, Mayank Chaturvedi, Michael Moynihan, Min Ma, Nabila Babar, Natasha Noy, Nathan Byrd, Nick Roy, Nikola Momchev, Nilay Chauhan, Noveen Sachdeva, Oskar Bunyan, Pankil Botarda, Paul Caron, Paul Kishan Rubenstein, Phil Culliton, Philipp Schmid, Pier Giuseppe Sessa, Pingmei Xu, Piotr Stanczyk, Pouya Tafti, Rakesh Shivanna, Renjie Wu, Renke Pan, Reza Rokni, Rob Willoughby, Rohith Vallu, Ryan Mullins, Sammy Jerome, Sara Smoot, Sertan Girgin, Shariq Iqbal, Shashir Reddy, Shruti Sheth, Siim Põder, Sijal Bhatnagar, Sindhu Raghuram Panyam, Sivan Eiger, Susan Zhang, Tianqi Liu, Trevor Yacovone, Tyler Liechty, Uday Kalra, Utku Evci, Vedant Misra, Vincent Roseberry, Vlad Feinberg, Vlad Kolesnikov, Woohyun Han, Woosuk Kwon, Xi Chen, Yinlam Chow, Yuvein Zhu, Zichuan Wei, Zoltan Egyed, Victor Cotruta, Minh Giang, Phoebe Kirk, Anand Rao, Kat Black, Nabila Babar, Jessica Lo, Erica Moreira, Luiz Gustavo Martins, Omar Sanseviero, Lucas Gonzalez, Zach Gleicher, Tris Warkentin, Vahab Mirrokni, Evan Senter, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, Yossi Matias, D. Sculley, Slav Petrov, Noah Fiedel, Noam Shazeer, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Jean-Baptiste Alayrac, Rohan Anil, Dmitry (Dima) Lepikhin, Sebastian Borgeaud, Olivier Bachem, Armand Joulin, Alek Andreev, Cassidy Hardin, Robert Dadashi, and Léonard Hussenot. 2025. Gemma 3 Technical Report. arXiv:2503.19786 [cs.CL] https://arxiv.org/abs/2503.19786 Griggs et al. (2024) Tyler Griggs, Xiaoxuan Liu, Jiaxiang Yu, Doyoung Kim, Wei-Lin Chiang, Alvin Cheung, and Ion Stoica. 2024. Mélange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneity. arXiv preprint arXiv:2404.14527 (2024). Guha et al. (2025) Etash Guha, Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smyrnis, Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, Ashima Suvarna, Benjamin Feuer, Liangyu Chen, Zaid Khan, Eric Frankel, Sachin Grover, Caroline Choi, Niklas Muennighoff, Shiye Su, Wanjia Zhao, John Yang, Shreyas Pimpalgaonkar, Kartik Sharma, Charlie Cheng-Jie Ji, Yichuan Deng, Sarah Pratt, Vivek Ramanujan, Jon Saad-Falcon, Jeffrey Li, Achal Dave, Alon Albalak, Kushal Arora, Blake Wulfe, Chinmay Hegde, Greg Durrett, Sewoong Oh, Mohit Bansal, Saadia Gabriel, Aditya Grover, Kai-Wei Chang, Vaishaal Shankar, Aaron Gokaslan, Mike A. Merrill, Tatsunori Hashimoto, Yejin Choi, Jenia Jitsev, Reinhard Heckel, Maheswaran Sathiamoorthy, Alexandros G. Dimakis, and Ludwig Schmidt. 2025. OpenThoughts: Data Recipes for Reasoning Models. arXiv:2506.04178 [cs.LG] https://arxiv.org/abs/2506.04178 He et al. (2024) Jiaao He, Kezhao Huang, and Jidong Zhai. 2024. FASTDECODE: High-Throughput LLM Serving through Disaggregating Attention Computation. In Proceedings of the 1st Workshop on Long-Context Foundation Models. https://openreview.net/forum?id=GahfuPsGw2 Huang et al. (2025) Kaiyu Huang, Hao Wu, Zhubo Shi, Han Zou, Minchen Yu, and Qingjiang Shi. 2025. AdaSpec: Adaptive Speculative Decoding for Fast, SLO-Aware Large Language Model Serving. arXiv preprint arXiv:2503.05096 (2025). Jiang et al. (2025a) Youhe Jiang, Fangcheng Fu, Xiaozhe Yao, Taiyi Wang, Bin Cui, Ana Klimovic, and Eiko Yoneki. 2025a. ThunderServe: High-performance and Cost-efficient LLM Serving in Cloud Environments. In Proceedings of Machine Learning and Systems (MLSys). Jiang et al. (2026) Youhe Jiang, Fangcheng Fu, and Eiko Yoneki. 2026. BOute: Cost-Efficient LLM Serving with Heterogeneous LLMs and GPUs via Multi-Objective Bayesian Optimization. arXiv preprint arXiv:2602.10729 (2026). Jiang et al. (2024) Youhe Jiang, Ran Yan, Xiaozhe Yao, Yang Zhou, Beidi Chen, and Binhang Yuan. 2024. HexGen: Generative Inference of Large Language Model over Heterogeneous Environment. In Proceedings of the 41st International Conference on Machine Learning (ICML). Jiang et al. (2025b) Youhe Jiang, Ran Yan, and Binhang Yuan. 2025b. HexGen-2: Disaggregated Generative Inference of LLMs in Heterogeneous Environment. In The Thirteenth International Conference on Learning Representations (ICLR). Kim et al. (2025) Kihyun Kim, Jinwoo Kim, James J. Kim, Dong Li, and Youngjae Kim. 2025. FlexLLM: Flexible and Cost-Efficient LLM Serving with Heterogeneous GPUs. In Proceedings of the IEEE International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS). Kimi Team et al. (2025) Kimi Team, Yu Zhang, Zongyu Lin, Xingcheng Yao, Jiaxi Hu, Fanqing Meng, Chengyin Liu, Xin Men, Songlin Yang, Zhiyuan Li, Wentao Li, Enzhe Lu, Weizhou Liu, Yanru Chen, Weixin Xu, Longhui Yu, Yejie Wang, Yu Fan, Longguang Zhong, Enming Yuan, Dehao Zhang, Yizhi Zhang, T. Y. Liu, Haiming Wang, Shengjun Fang, Weiran He, Shaowei Liu, Yiwei Li, Jianlin Su, Jiezhong Qiu, Bo Pang, Junjie Yan, Zhejun Jiang, Weixiao Huang, Bohong Yin, Jiacheng You, Chu Wei, Zhengtao Wang, Chao Hong, Yutian Chen, Guanduo Chen, Yucheng Wang, Huabin Zheng, Feng Wang, Yibo Liu, Mengnan Dong, Zheng Zhang, Siyuan Pan, Wenhao Wu, Yuhao Wu, Longyu Guan, Jiawen Tao, Guohong Fu, Xinran Xu, Yuzhi Wang, Guokun Lai, Yuxin Wu, Xinyu Zhou, Zhilin Yang, and Yulun Du. 2025. Kimi linear: An expressive, efficient attention architecture. arXiv preprint arXiv:2510.26692 (2025). Kwon et al. (2023) Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. arXiv preprint arXiv:2309.06180 (2023). Leviathan et al. (2022) Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2022. Fast Inference from Transformers via Speculative Decoding. arXiv preprint arXiv:2211.17192 (2022). Li et al. (2024b) Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2024b. EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees. arXiv preprint arXiv:2406.16858 (2024). Li et al. (2024c) Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2024c. EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty. arXiv preprint arXiv:2401.15077 (2024). Li et al. (2025) Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2025. EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test. arXiv preprint arXiv:2503.01840 (2025). Li et al. (2026) Zikun Li, Zhuofu Chen, Remi Delacourt, Gabriele Oliaro, Zeyu Wang, Qinghan Chen, Shuhuai Lin, April Yang, Zhihao Zhang, Zhuoming Chen, Yi-Hsiang Lai, Xinhao Cheng, Xupeng Miao, and Zhihao Jia. 2026. AdaServe: Accelerating Multi-SLO LLM Serving with SLO-Customized Speculative Decoding. In Proceedings of the 21st European Conference on Computer Systems (EuroSys ’26). ACM, New York, NY, USA, 36–54. doi:10.1145/3767295.3769315 Li et al. (2024a) Zonghang Li, Wenjiao Feng, Mohsen Guizani, and Hongfang Yu. 2024a. TPI-LLM: Serving 70B-scale LLMs Efficiently on Low-resource Edge Devices. arXiv preprint arXiv:2410.00531 (2024). Lin et al. (2024) Bin Lin, Chen Zhang, Tao Peng, Hanyu Zhao, Wencong Xiao, Minmin Sun, Anmin Liu, Zhipeng Zhang, Lanbo Li, Xiafei Qiu, Shen Li, Zhigang Ji, Tao Xie, Yong Li, and Wei Lin. 2024. Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache. arXiv preprint arXiv:2401.02669 (2024). Liu et al. (2026) Guowei Liu, Hongming Li, Yaning Guo, Yongxi Lyu, Mo Zhou, Yi Liu, Zhaogeng Li, and Yanpeng Wang. 2026. Revealing the Challenges of Attention-FFN Disaggregation for Modern MoE Models and Hardware Systems. arXiv preprint arXiv:2602.09721 (2026). https://arxiv.org/abs/2602.09721 Liu et al. (2024) Xiaoxuan Liu, Jongseok Park, Langxiang Hu, Woosuk Kwon, Zhuohan Li, Chen Zhang, Kuntai Du, Xiangxi Mo, Kaichao You, Alvin Cheung, Zhijie Deng, Ion Stoica, and Hao Zhang. 2024. TurboSpec: Closed-loop Speculation Control System for Optimizing LLM Serving Goodput. arXiv preprint arXiv:2406.14066 (2024). Lohmeyer and Bradstock (2025) Mark Lohmeyer and Drew Bradstock. 2025. GKE Inference Gateway and Quickstart are GA. https://cloud.google.com/blog/products/ai-machine-learning/gke-inference-gateway-and-quickstart-are-ga. Google Cloud Blog. Accessed: 2026-04-16. Mei et al. (2026) Yixuan Mei, Zikun Li, Zixuan Chen, Shiqi Pan, Mengdi Wu, Xupeng Miao, Zhihao Jia, and K. V. Rashmi. 2026. Coral: Cost-Efficient Multi-LLM Serving over Heterogeneous Cloud GPUs. arXiv preprint arXiv:2605.04357 (2026). Mei et al. (2024) Yixuan Mei, Yonghao Zhuang, Xupeng Miao, Juncheng Yang, Zhihao Jia, and Rashmi Vinayak. 2024. Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flow. arXiv preprint arXiv:2406.01566 (2024). Merizzi et al. (2025) Nicholas Merizzi, Chris Thomas, and Ed Burns. 2025. The AI Infrastructure Reckoning: Optimizing Compute Strategy in the Age of Inference Economics. https://w.deloitte.com/us/en/insights/topics/technology-management/tech-trends/2026/ai-infrastructure-compute-strategy.html. Deloitte Insights. Accessed: 2026-04-15. Miao et al. (2024) Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, Chunan Shi, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia. 2024. SpecInfer: Accelerating Large Language Model Serving with Tree-based Speculative Inference and Verification. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3. ACM, New York, NY, USA, 932–949. Noffsinger et al. (2025) Jesse Noffsinger, Mark Patel, Pankaj Sachdeva, Arjita Bhan, Haley Chang, and Maria Goodpaster. 2025. The Cost of Compute: A $7 Trillion Race to Scale Data Centers. https://w.mckinsey.com/industries/technology-media-and-telecommunications/our-insights/the-cost-of-compute-a-7-trillion-dollar-race-to-scale-data-centers. McKinsey & Company. Accessed: 2026-04-18. NVIDIA (2026a) NVIDIA. 2026a. A100 Tensor Core GPU: Specifications. https://w.nvidia.com/en-us/data-center/a100/. Accessed: 2026-09-07. NVIDIA (2026b) NVIDIA. 2026b. H100 Tensor Core GPU: Product Specifications. https://w.nvidia.com/en-sg/data-center/h100/. Accessed: 2026-09-07. NVIDIA (2026c) NVIDIA. 2026c. L40S GPU Specifications. https://w.nvidia.com/en-us/data-center/l40s/. Accessed: 2026-09-07. Oliaro et al. (2024) Gabriele Oliaro, Zhihao Jia, Daniel Campos, and Aurick Qiao. 2024. SuffixDecoding: Extreme Speculative Decoding for Emerging AI Applications. arXiv preprint arXiv:2411.04975 (2024). OpenAI et al. (2025) OpenAI, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, Che Chang, Kai Chen, Mark Chen, Enoch Cheung, Aidan Clark, Dan Cook, Marat Dukhan, Casey Dvorak, Kevin Fives, Vlad Fomenko, Timur Garipov, Kristian Georgiev, Mia Glaese, Tarun Gogineni, Adam Goucher, Lukas Gross, Katia Gil Guzman, John Hallman, Jackie Hehir, Johannes Heidecke, Alec Helyar, Haitang Hu, Romain Huet, Jacob Huh, Saachi Jain, Zach Johnson, Chris Koch, Irina Kofman, Dominik Kundel, Jason Kwon, Volodymyr Kyrylov, Elaine Ya Le, Guillaume Leclerc, James Park Lennon, Scott Lessans, Mario Lezcano-Casado, Yuanzhi Li, Zhuohan Li, Ji Lin, Jordan Liss, Lily (Xiaoxuan) Liu, Jiancheng Liu, Kevin Lu, Chris Lu, Zoran Martinovic, Lindsay McCallum, Josh McGrath, Scott McKinney, Aidan McLaughlin, Song Mei, Steve Mostovoy, Tong Mu, Gideon Myles, Alexander Neitz, Alex Nichol, Jakub Pachocki, Alex Paino, Dana Palmie, Ashley Pantuliano, Giambattista Parascandolo, Jongsoo Park, Leher Pathak, Carolina Paz, Ludovic Peran, Dmitry Pimenov, Michelle Pokrass, Elizabeth Proehl, Huida Qiu, Gaby Raila, Filippo Raso, Hongyu Ren, Kimmy Richardson, David Robinson, Bob Rotsted, Hadi Salman, Suvansh Sanjeev, Max Schwarzer, D. Sculley, Harshit Sikchi, Kendal Simon, Karan Singhal, Yang Song, Dane Stuckey, Zhiqing Sun, Philippe Tillet, Sam Toizer, Foivos Tsimpourlas, Nikhil Vyas, Eric Wallace, Xin Wang, Miles Wang, Olivia Watkins, Kevin Weil, Amy Wendling, Kevin Whinnery, Cedric Whitney, Hannah Wong, Lin Yang, Yu Yang, Michihiro Yasunaga, Kristen Ying, Wojciech Zaremba, Wenting Zhan, Cyril Zhang, Brian Zhang, Eddie Zhang, and Shengjia Zhao. 2025. gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925 (2025). Patel et al. (2024) Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. 2024. Splitwise: Efficient Generative LLM Inference Using Phase Splitting. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). IEEE, 118–132. Peng et al. (2025) You Peng, Youhe Jiang, Wenqi Jiang, Chen Wang, and Binhang Yuan. 2025. HEXGEN-FLOW: Optimizing LLM Inference Request Scheduling for Agentic Text-to-SQL. arXiv preprint arXiv:2505.05286 (2025). Qin et al. (2024) Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. 2024. Mooncake: A KVCache-Centric Disaggregated Architecture for LLM Serving. arXiv preprint arXiv:2407.00079 (2024). Qwen Team (2025) Qwen Team. 2025. Qwen3-Next: Towards Ultimate Training & Inference Efficiency. https://qwen.ai/blog?id=4074cca80393150c248e508a62983f9cb7d27cd. Qwen blog, September 10, 2025. Singh et al. (2025) Gursimran Singh, Xinglu Wang, Yifan Hu, Timothy Tin Long Yu, Linzi Xing, Wei Jiang, Zhefeng Wang, Xiaolong Bai, Yi Li, Ying Xiong, Yong Zhang, and Zhenan Fan. 2025. Efficiently Serving Large Multimodal Models Using EPD Disaggregation. In Proceedings of the 42nd International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 267). 55740–55756. https://openreview.net/forum?id=n7VLFYNoCb Song et al. (2026) Chendong Song, Meixuan Wang, Hang Zhou, Hong Liang, Yuan Lyu, Zixi Chen, Yuwei Fan, and Zijie Zhou. 2026. Analytical Provisioning for Attention-FFN Disaggregated LLM Serving under Stochastic Workloads. arXiv preprint arXiv:2601.21351 (2026). https://arxiv.org/abs/2601.21351 StepFun et al. (2025) StepFun, Bin Wang, Bojun Wang, Changyi Wan, Guanzhe Huang, Hanpeng Hu, Haonan Jia, Hao Nie, Mingliang Li, Nuo Chen, Siyu Chen, Song Yuan, Wuxun Xie, Xiaoniu Song, Xing Chen, Xingping Yang, Xuelin Zhang, Yanbo Yu, Yaoyu Wang, Yibo Zhu, Yimin Jiang, Yu Zhou, Yuanwei Lu, Houyi Li, Jingcheng Hu, Ka Man Lo, Ailin Huang, Binxing Jiao, Bo Li, Boyu Chen, Changxin Miao, Chang Lou, Chen Hu, Chen Xu, Chenfeng Yu, Chengyuan Yao, Daokuan Lv, Dapeng Shi, Deshan Sun, Ding Huang, Dingyuan Hu, Dongqing Pang, Enle Liu, Fajie Zhang, Fanqi Wan, Gulin Yan, Han Zhang, Han Zhou, Hanghao Wu, Hangyu Guo, Hanqi Chen, Hanshan Zhang, Hao Wu, Haocheng Zhang, Haolong Yan, Haoran Lv, Haoran Wei, Hebin Zhou, Heng Wang, Heng Wang, Hongxin Li, Hongyu Zhou, Hongyuan Wang, Huiyong Guo, Jia Wang, Jiahao Gong, Jialing Xie, Jian Zhou, Jianjian Sun, Jiaoren Wu, Jiaran Zhang, Jiayu Liu, Jie Cheng, Jie Luo, Jie Yan, Jie Yang, Jieyi Hou, Jinguang Zhang, Jinlan Cao, Jisheng Yin, Junfeng Liu, Junhao Huang, Junzhe Lin, Kaijun Tan, Kaixiang Li, Kang An, Kangheng Lin, Kenkun Liu, Lei Yang, Liang Zhao, Liangyu Chen, Lieyu Shi, Liguo Tan, Lin Lin, Lin Zhang, Lina Chen, Liwen Huang, Liying Shi, Longlong Gu, Mei Chen, Mengqiang Ren, Ming Li, Mingzhe Chen, Na Wang, Nan Wu, Qi Han, Qian Zhao, Qiang Zhang, Qianni Liu, Qiaohui Chen, Qiling Wu, Qinglin He, Qinyuan Tan, Qiufeng Wang, Qiuping Wu, Qiuyan Liang, Quan Sun, Rui Li, Ruihang Miao, Ruosi Wan, Ruyan Guo, Shangwu Zhong, Shaoliang Pang, Shengjie Fan, Shijie Shang, Shilei Jiang, Shiliang Yang, Shiming Hao, Shuli Gao, Siming Huang, Siqi Liu, Tiancheng Cao, Tianhao Cheng, Tianhao Peng, Wang You, Wei Ji, Wen Sun, Wenjin Deng, Wenqing He, Wenzhen Zheng, Xi Chen, Xiangwen Kong, Xianzhen Luo, Xiaobo Yang, Xiaojia Liu, Xiaoxiao Ren, Xin Han, Xin Li, Xin Wu, Xu Zhao, Yanan Wei, Yang Li, Yangguang Li, Yangshijie Xu, Yanming Xu, Yaqiang Shi, Yeqing Shen, Yi Yang, Yifei Yang, Yifeng Gong, Yihan Chen, Yijing Yang, Yinmin Zhang, Yizhuang Zhou, Yuanhao Ding, Yuantao Fan, Yuanzhen Yang, Yuchu Luo, Yue Peng, Yufan Lu, Yuhang Deng, Yuhe Yin, Yujie Liu, Yukun Chen, Yuling Zhao, Yun Mou, Yunlong Li, Yunzhou Ju, Yusheng Li, Yuxiang Yang, Yuxiang Zhang, Yuyang Chen, Zejia Weng, Zhe Xie, Zheng Ge, Zheng Gong, Zhenyi Lu, Zhewei Huang, Zhichao Chang, Zhiguo Huang, Zhirui Wang, Zidong Yang, Zili Wang, Ziqi Wang, Zixin Zhang, Binxing Jiao, Daxin Jiang, Heung-Yeung Shum, and Xiangyu Zhang. 2025. Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decoding. arXiv preprint arXiv:2507.19427 (2025). Tong et al. (2025) Chris Tong, Youhe Jiang, Gufeng Chen, Tianyi Zhao, Sibian Lu, Wenjie Qu, Eric Yang, Lynn Ai, and Binhang Yuan. 2025. Parallax: Efficient LLM Inference Service over Decentralized Environment. arXiv preprint arXiv:2509.26182 (2025). Tully et al. (2025) Tim Tully, Joff Redfern, Deedy Das, and Derek Xiao. 2025. 2025 Mid-Year LLM Market Update: Foundation Model Landscape & Economics. https://menlovc.com/perspective/2025-mid-year-llm-market-update/. Menlo Ventures. Accessed: 2026-04-15. Wu et al. (2025) Linyu Wu, Xiaoyuan Liu, Tianneng Shi, Zhe Ye, and Dawn Song. 2025. DeServe: Towards Affordable Offline LLM Inference via Decentralization. arXiv preprint arXiv:2501.14784 (2025). Yu et al. (2022) Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A Distributed Serving System for Transformer-Based Generative Models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). USENIX Association, 521–538. Zheng et al. (2023) Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. 2023. SGLang: Efficient Execution of Structured Language Model Programs. arXiv preprint arXiv:2312.07104 (2023). Zhong et al. (2024) Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. 2024. DistServe: Disaggregating Prefill and Decoding for Goodput-Optimized Large Language Model Serving. arXiv preprint arXiv:2401.09670 (2024). Zhu et al. (2024) Kan Zhu, Yufei Gao, Yilong Zhao, Liangyu Zhao, Gefei Zuo, Yile Gu, Dedong Xie, Tian Tang, Qinyu Xu, Zihao Ye, Keisuke Kamahori, Chien-Yu Lin, Ziren Wang, Stephanie Wang, Arvind Krishnamurthy, and Baris Kasikci. 2024. NanoFlow: Towards Optimal Large Language Model Serving Throughput. arXiv preprint arXiv:2408.12757 (2024). Zhu et al. (2025) Ruidong Zhu, Ziheng Jiang, Chao Jin, Peng Wu, Cesar A. Stuardo, Dongyang Wang, Xinlei Zhang, Huaping Zhou, Haoran Wei, Yang Cheng, Jianzhe Xiao, Xinyi Zhang, Lingjun Liu, Haibin Lin, Li-Wen Chang, Jianxi Ye, Xiao Yu, Xuanzhe Liu, Xin Jin, and Xin Liu. 2025. MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelism. arXiv preprint arXiv:2504.02263 (2025). Appendix A Analytical Model and Proofs for Section 3 This appendix section collects the analytical model, proofs, and CAD feasibility details used by Section 3. A.1. Detailed analytical model We now formalize the analytical abstraction used in Section 3.1. We model one decode step of an LLM as the sequential execution of an ordered operator sequence =(o1,o2,…,oNmodel),o=(o_1,o_2,…,o_N_model), where each oio_i denotes one computational kernel invoked during the decode step. This abstraction follows the execution pattern of current serving engines, which invoke the model’s kernels in a fixed topological order during each decoding iteration. For an operator oio_i executed at batch size b, let Di(b)D_i(b) denote the total number of bytes loaded from memory and let Wi(b)W_i(b) denote the total number of floating-point operations. We characterize each GPU type by four quantities: unit-time monetary cost c, memory bandwidth β, memory capacity M, and peak compute throughput F. Under the exclusive-occupancy assumption, the monetary cost of operator oio_i is Cost(oi)=T(oi)⋅c,Cost(o_i)=T(o_i)· c, where T(oi)T(o_i) is the execution latency of the operator. We model this latency using a roofline-style expression: T(oi)=max(Di(b)β,Wi(b)F).T(o_i)= \! ( D_i(b)β, W_i(b)F ). The first term is the memory-load time and the second term is the compute time, so the maximum captures whether the operator is memory-bound or compute-bound on the target GPU. The cost of a full decode step is then the sum of the costs of all operators in the sequence: Coststep=∑i=1NmodelCost(oi).Cost_step= _i=1^N_modelCost(o_i). Since one decode step produces one token per request, the GPU cost per token is obtained by dividing CoststepCost_step by the batch size b. Throughout the theoretical analysis, we intentionally focus on GPU-side cost. Although a disaggregated pipeline introduces activation transfers between device groups, the analysis studies the idealized steady-state regime in which these transfers are overlapped with computation through microbatching and pipelining. Under this assumption, communication does not lie on the throughput-critical path and is therefore omitted from the analytical objective. A.2. Execution-regime assumptions This subsection formalizes the execution-regime assumptions used in Section 3.1. Under the roofline model above, an operator is memory-bound when Di(b)β≥Wi(b)F, D_i(b)β≥ W_i(b)F, and compute-bound when the reverse inequality holds. We assume that decode-phase attention lies in the memory-bound regime. During decoding, each request contributes only one query token while the attention operator must load the full KV cache associated with the prior context, which keeps arithmetic intensity low and makes latency primarily determined by memory traffic. Formally, for attention operators oiattno_i^attn, we assume Diattn(b)β≥Wiattn(b)F. D_i^attn(b)β≥ W_i^attn(b)F. For GEMM-dominated operators, we assume that there exists a batch-size threshold b∗b^* such that these operators become compute-bound once the batch size is sufficiently large. Formally, for GEMM operators oigemmo_i^gemm, we assume that for all b≥b∗b≥ b^*, Wigemm(b)F≥Digemm(b)β. W_i^gemm(b)F≥ D_i^gemm(b)β. The threshold b∗b^* marks the transition at which arithmetic time overtakes memory-load time, so beyond this point further execution is limited by compute throughput rather than bandwidth. The role of disaggregation is to make this compute-bound GEMM regime attainable. In colocated serving, GEMM operators share device memory with the KV cache, so long contexts reduce the maximum feasible batch size and can keep GEMM in a memory-bound regime. Under operator-level disaggregation, the compute side no longer needs to reserve memory for the KV cache, and requests forwarded from multiple attention-side replicas can be aggregated on the GEMM side. This decoupling allows the GEMM side to sustain a larger effective batch size and thus operate in the compute-bound regime assumed in the main-text analysis. A.3. Derivation and interpretation of the final cost expressions This subsection derives the cost expressions used in Section 3.1 from the operator-level roofline model. Operator-class aggregates. Let attnO_attn denote the set of attention operators executed in one decode step and let gemmO_gemm denote the set of GEMM-dominated operators, where the latter includes the linear projections, FFN, and MoE computation. We define the aggregate quantities Dattn(b) D^attn(b) =∑oi∈attnDi(b), \;=\; _o_i _attnD_i(b), Dgemm(b) D^gemm(b) =∑oi∈gemmDi(b), \;=\; _o_i _gemmD_i(b), Wgemm(b) W^gemm(b) =∑oi∈gemmWi(b). \;=\; _o_i _gemmW_i(b). These aggregate terms collect, respectively, the total attention-side memory traffic, the total GEMM-side memory traffic, and the total GEMM-side floating-point work for a batch of size b in one decode step. Substituting these class aggregates into the roofline cost model of Appendix A.1 and dividing by the batch size yields the per-token costs as follows. Homogeneous disaggregated serving. In homogeneous ODS, attention and GEMM-dominated operators are placed on separate device groups of the same GPU type (c,β,M,F)(c,β,M,F). Under the execution-regime assumptions of Section 3.1, decode-phase attention is memory-bound while GEMM-dominated operators are compute-bound. Their class-level costs therefore reduce to Coststepattn=Dattn(b)βc,Coststepgemm=Wgemm(b)Fc.Cost^attn_step= D^attn(b)β\,c, ^gemm_step= W^gemm(b)F\,c. Dividing the sum by b yields CPThom=cb(Dattn(b)β+Wgemm(b)F).CPT^hom= cb ( D^attn(b)β+ W^gemm(b)F ). Heterogeneous disaggregated serving. In heterogeneous ODS, the attention side and the GEMM side may run on different GPU types. If attention runs on GPU type 1 with parameters (c1,β1,M1,F1)(c_1, _1,M_1,F_1) and GEMM-dominated operators run on GPU type 2 with parameters (c2,β2,M2,F2)(c_2, _2,M_2,F_2), the same regime assumptions give Coststepattn=Dattn(b)β1c1,Coststepgemm=Wgemm(b)F2c2,Cost^attn_step= D^attn(b) _1\,c_1, ^gemm_step= W^gemm(b)F_2\,c_2, and thus CPT1,2het=1b(Dattn(b)β1c1+Wgemm(b)F2c2).CPT^het_1,2= 1b ( D^attn(b) _1c_1+ W^gemm(b)F_2c_2 ). The cost-efficiency metrics αj _j and γj _j defined in Section 3.1 measure monetary cost per byte of memory traffic and per FLOP, respectively. Heterogeneous specialization allows attention to use the GPU type with the lower αj _j and GEMM-dominated operators to use the type with the lower γj _j. Colocated serving. In colocated serving, all operators share the same device memory, so the feasible batch size is constrained by both model weights and the KV cache. Let MweightsM_weights denote the memory occupied by model weights and let mkv(s)m_kv(s) denote the KV-cache memory required per request at average sequence length s. Neglecting activation memory for simplicity, the maximum feasible batch size is bmax(s)=⌊M−Mweightsmkv(s)⌋.b_ (s)= M-M_weightsm_kv(s) . As s grows, mkv(s)m_kv(s) increases and the feasible batch size shrinks. Attention remains memory-bound under colocation, so its contribution is still Dattn(b)βc. D^attn(b)β\,c. For GEMM-dominated operators, however, the smaller feasible batch size may prevent the compute side from reaching the compute-bound regime. We therefore retain the full roofline expression, Coststepgemm=max(Dgemm(b)β,Wgemm(b)F)c,Cost^gemm_step= \! ( D^gemm(b)β, W^gemm(b)F )c, which gives, for 1≤b≤bmax(s)1≤ b≤ b_ (s), CPTcoloc(s,b) ^coloc(s,b) =cb(Dattn(b)βCLOSE = cb ( D^attn(b)β OPEN+max(Dgemm(b)β,Wgemm(b)F)). + \! ( D^gemm(b)β, W^gemm(b)F ) ). Intuition for the small-batch regime. The max term in the colocated expression is important because GEMM efficiency depends on batch size. When the feasible batch size remains large enough, GEMM can stay compute-bound and its contribution is governed by Wgemm(b)/FW^gemm(b)/F. But once KV-cache growth drives the feasible batch size down, GEMM may enter a small-batch regime in which loading weights dominates the arithmetic work. In that case, Dgemm(b)β>Wgemm(b)F, D^gemm(b)β\;>\; W^gemm(b)F, so GEMM becomes memory-bound and the GPU’s compute capability is underutilized. This is precisely the inefficiency that operator-level disaggregation removes: by decoupling the compute side from KV-cache memory pressure, disaggregation allows GEMM-dominated operators to recover the larger effective batch sizes needed to operate in the compute-bound regime. Conditional achievability via CAD The idealized disaggregated cost expressions above are attainable by the Core-Attention Disaggregation (CAD) partition under pipeline balance. CAD assigns the core attention kernels to the attention device group and assigns the remaining GEMM-dominated operators, including projections, FFN, and MoE computation, to the rest-of-model device group. Under the regime assumptions in Appendix A.2, the attention side therefore incurs bandwidth cost while the rest-of-model side incurs compute cost. If the two stages are provisioned so that their steady-state latencies match and inter-stage communication is overlapped by microbatching, the realized per-token GPU cost is exactly the homogeneous or heterogeneous ODS cost expression derived above. A.4. Scaling approximations and GEMM threshold This subsection formalizes the approximations used in Section 3.1 to obtain closed-form analytical results. Attention-side memory traffic. During decoding, each request contributes one newly generated token, but attention must load the KV cache associated with that request’s prior context. As a result, the total attention-side memory traffic grows linearly with the batch size and with the average sequence length. We therefore approximate Dattn(b)≈bmkv(s),D^attn(b)≈ b\,m_kv(s), where mkv(s)m_kv(s) denotes the KV-cache memory required per request at average sequence length s. GEMM-side floating-point work. We approximate the total GEMM work in one decode step as Wgemm(b)≈2bPact,W^gemm(b)≈ 2bP_act, where PactP_act is the effective number of GEMM-side parameters activated by one request on the modeled device group after any model-parallel sharding. For dense models, this includes all GEMM-side parameters assigned to the device group. For MoE models, it includes shared dense parameters and only the routed expert parameters used by the request, rather than all resident expert parameters. The factor of 22 reflects one multiply-add per active parameter per request. This approximation captures the fact that GEMM-side arithmetic scales linearly with the number of concurrently processed requests. GEMM-side memory traffic. For GEMM-dominated operators, we approximate the total memory traffic by the model-weight footprint: Dgemm(b)≈Mweights,D^gemm(b)≈ M_weights, where MweightsM_weights is the resident footprint of all model weights assigned to the modeled device group after sharding. In an MoE model, inactive experts do not contribute to PactP_act for an individual request, but their resident weights still contribute to MweightsM_weights. The approximation Dgemm(b)≈MweightsD^gemm(b)≈ M_weights corresponds to a sufficiently large batch whose routed tokens collectively access and amortize the resident expert-weight shards; the runtime planner uses measured operator costs rather than relying on this approximation. Continuous form of the colocated batch-size limit. In the main text, the exact colocated feasible batch size is defined as bmax(s)=⌊M−Mweightsmkv(s)⌋.b_ (s)= M-M_weightsm_kv(s) . For the closed-form analysis, we use the continuous relaxation b¯(s)=M−Mweightsmkv(s). b(s)= M-M_weightsm_kv(s). Dropping the floor changes the feasible batch size by at most one request. Derivation of the GEMM threshold. The threshold b∗b^* is defined as the batch size at which GEMM transitions from memory-bound to compute-bound execution. Under the approximations above, the GEMM memory-load time is Dgemm(b)β≈Mweightsβ, D^gemm(b)β≈ M_weightsβ, while the GEMM compute time is Wgemm(b)F≈2bPactF. W^gemm(b)F≈ 2bP_actF. Equating these two terms gives the transition point: Mweightsβ=2bPactF, M_weightsβ= 2bP_actF, and therefore b∗=MweightsF2Pactβ.b^*= M_weightsF2P_actβ. Interpretation. When b≥b∗b≥ b^*, GEMM compute time exceeds weight-loading time, so GEMM operates in the compute-bound regime. When b<b∗b<b^*, the opposite holds and GEMM is memory-bound. This threshold is central to the later gain analysis: homogeneous ODS begins to improve over colocation precisely when KV-cache pressure pushes the colocated feasible batch size below this compute-bound threshold. Scope of the approximation. These approximations are introduced only to obtain interpretable closed-form characterizations of when ODS helps and what bounds its gain. They are not intended as a full performance model of the runtime system; the later planner and end-to-end evaluation use richer model- and hardware-specific information. A.5. Proofs for Section 3 This appendix provides complete proofs for the theorems stated in Section 3. We use the same notation and scaling approximations introduced in Section 3.1: Dattn(b)=b⋅mkv(s)D^attn(b)=b· m_kv(s), Wgemm(b)=2bPactW^gemm(b)=2bP_act, Dgemm(b)=MweightsD^gemm(b)=M_weights, and the continuous relaxation b¯(s)=(M−Mweights)/mkv(s) b(s)=(M-M_weights)/m_kv(s). For the homogeneous comparison, we restrict attention to sequence lengths with bmax(s)≥1b_ (s)≥ 1 and assume mkv(s)m_kv(s) is strictly increasing on this feasible domain, as stated in Section 3.2. A.5.1. Proof of Theorem 3.1: Homogeneous Disaggregation vs. Colocation Step 1: Simplify the disaggregated CPT Suppressing the fixed GPU-type index and expanding α=c/βα=c/β and γ=c/Fγ=c/F in Eq. 1 gives: (11) CPThom=cb(b⋅mkv(s)β+2bPactF)=c(mkv(s)β+2PactF).CPT^hom= cb ( b· m_kv(s)β+ 2bP_actF )=c ( m_kv(s)β+ 2P_actF ). The batch size b cancels because both the attention memory traffic and the GEMM compute scale linearly with b. The disaggregated CPT depends on s only through mkv(s)m_kv(s). Step 2: Simplify the colocated CPT For an arbitrary local batch size b, the colocated cost derived in Appendix A.3 becomes: CPTcoloc(s,b) ^coloc(s,b) =cb(b⋅mkv(s)β+max(Mweightsβ,2bPactF)) = cb\! ( b· m_kv(s)β+ \! ( M_weightsβ,\; 2bP_actF ) ) (12) =c(mkv(s)β+1bmax(Mweightsβ,2bPactF)). =c\! ( m_kv(s)β+ 1b \! ( M_weightsβ,\; 2bP_actF ) ). Here b is a local batch-size variable; Section 4 distinguishes the global, per-replica, and microbatch quantities used by the planner. The cost is non-increasing in b, so its minimum over 1≤b≤b¯(s)1≤ b≤ b(s) is attained at b=b¯(s)b= b(s), yielding CPTcoloc(s)CPT^coloc(s) in Eq. 4. The two GEMM regimes yield: Compute-bound GEMM (b¯(s)≥b∗ b(s)≥ b^*): (13) CPTcoloc=c(mkv(s)β+2PactF).CPT^coloc=c ( m_kv(s)β+ 2P_actF ). Memory-bound GEMM (b¯(s)<b∗ b(s)<b^*): (14) CPTcoloc=c(mkv(s)β+Mweightsb¯(s)⋅β).CPT^coloc=c ( m_kv(s)β+ M_weights b(s)·β ). Step 3: Case 1: Compute-bound GEMM (b¯(s)≥b∗ b(s)≥ b^*) When the colocated batch size is large enough for GEMM to saturate compute, comparing Eq. 11 and Eq. 13 gives CPTcoloc=c(mkv(s)β+2PactF)=CPThom.CPT^coloc=c ( m_kv(s)β+ 2P_actF )=CPT^hom. Therefore Ghom(s)=1G_hom(s)=1. This establishes claim 1: when the batch size is large enough for GEMM to fully utilize compute under colocation, there is no efficiency gap for disaggregation to exploit. Step 4: Case 2: Memory-bound GEMM (b¯(s)<b∗ b(s)<b^*) When KV-cache pressure pushes b¯(s) b(s) below b∗b^*, GEMM under colocation becomes memory-bound while disaggregation maintains compute-bound GEMM. Step 4a: Rewrite the colocated CPT. Substituting b¯(s)=(M−Mweights)/mkv(s) b(s)=(M-M_weights)/m_kv(s) into Eq. 14: CPTcoloc ^coloc =c(mkv(s)β+Mweights⋅mkv(s)(M−Mweights)⋅β) =c ( m_kv(s)β+ M_weights· m_kv(s)(M-M_weights)·β ) =c⋅mkv(s)β(1+MweightsM−Mweights) = c· m_kv(s)β (1+ M_weightsM-M_weights ) (15) =c⋅mkv(s)β⋅M−Mweights. = c· m_kv(s)β· MM-M_weights. Step 4b: Cost gain expression. Dividing Eq. 15 by Eq. 11: (16) Ghom(s)=mkv(s)/βmkv(s)/β+2Pact/F⋅M−Mweights.G_hom(s)= m_kv(s)/βm_kv(s)/β+2P_act/F· MM-M_weights. Step 4c: Monotonicity. Let x=mkv(s)/βx=m_kv(s)/β, which increases with s since mkv(s)m_kv(s) is increasing in s. Define A=M/(M−Mweights)>1A=M/(M-M_weights)>1 (since Mweights>0M_weights>0) and κ=2Pact/F>0κ=2P_act/F>0. Then (17) Ghom=Ax+κ,dGhomdx=Aκ(x+κ)2>0.G_hom= Axx+κ, dG_homdx= Aκ(x+κ)^2>0. Since dGhom/dx>0dG_hom/dx>0 for all x>0x>0, the cost gain GhomG_hom is strictly increasing in x and therefore in s. This establishes claim 2. Step 4d: Upper bound. For any feasible s in the memory-bound regime, Ghom(s)<AG_hom(s)<A because Ax/(x+κ)<AAx/(x+κ)<A for every finite x>0x>0. Combining this fact with the compute-bound case, where Ghom(s)=1≤AG_hom(s)=1≤ A, gives the following bound for every feasible s with bmax(s)≥1b_ (s)≥ 1: 1≤Ghom(s)<M−Mweights.1\;≤\;G_hom(s)\;<\; MM-M_weights. This establishes claim 3 and completes the proof. □ Summary The cost gain is Ghom(s)=1G_hom(s)=1 when b¯(s)≥b∗ b(s)≥ b^*. When b¯(s)<b∗ b(s)<b^*, it is (18) Ghom(s)=mkv(s)/βmkv(s)/β+2Pact/F⋅M−Mweights.G_hom(s)= m_kv(s)/βm_kv(s)/β+2P_act/F· MM-M_weights. Here, equality holds in the compute-bound case, while the displayed formula applies in the memory-bound case. Over the feasible memory-bound domain, GhomG_hom is strictly increasing in s and bounded above by M/(M−Mweights)M/(M-M_weights). A.5.2. Proof of Theorem 3.2: Heterogeneous vs. Homogeneous Disaggregation Let ℐI be the feasible interval from Theorem 3.2, and let x=mkv(s)x=m_kv(s). We first establish the shape of the analytically extended gain as a function of x≥0x≥ 0, then restrict it to x∈mkv(ℐ)x∈ m_kv(I). Step 1: Ghet(s)≥1G_het(s)≥ 1 (Claims 1 and 2) For each GPU type j, the heterogeneous CPT satisfies CPT⋆het ^het_ =min(α1,α2)⋅mkv(s)+min(γ1,γ2)⋅2Pact = ( _1, _2)· m_kv(s)+ ( _1, _2)· 2P_act (19) ≤αj⋅mkv(s)+γj⋅2Pact=CPTjhom, ≤\; _j· m_kv(s)+ _j· 2P_act\;=\;CPT^hom_j, because min(α1,α2)≤αj ( _1, _2)≤ _j and min(γ1,γ2)≤γj ( _1, _2)≤ _j for all j. Since this holds for every j, it holds for the minimum: CPT⋆het≤minjCPTjhom=CPT⋆hom.CPT^het_ ≤ _j\,CPT^hom_j=CPT^hom_ . Therefore Ghet(s)≥1G_het(s)≥ 1 for every s∈ℐs , establishing claim 1. For claim 2: if GPU type j dominates type k on both metrics (αj≤αk _j≤ _k and γj≤γk _j≤ _k), then min(α1,α2)=αj ( _1, _2)= _j and min(γ1,γ2)=γj ( _1, _2)= _j. Hence CPT⋆het=CPTjhom=CPT⋆homCPT^het_ =CPT^hom_j=CPT^hom_ and Ghet(s)=1G_het(s)=1 for every s∈ℐs . Step 2: Unimodal structure (Claim 3) When neither type dominates, the metric orderings must be crossed. Under the theorem’s labeling η≥1η≥ 1, the only possible crossed ordering is α1<α2 _1< _2 and γ2<γ1 _2< _1; the reverse ordering would imply η<1η<1. Define r=α2/α1>1r= _2/ _1>1 and q=γ1/γ2>1q= _1/ _2>1. The cost expressions become: (20) CPT1hom ^hom_1 =α1⋅mkv(s)+qγ2⋅2Pact, = _1· m_kv(s)+q _2· 2P_act, (21) CPT2hom ^hom_2 =rα1⋅mkv(s)+γ2⋅2Pact, =r _1· m_kv(s)+ _2· 2P_act, (22) CPT⋆het ^het_ =α1⋅mkv(s)+γ2⋅2Pact. = _1· m_kv(s)+ _2· 2P_act. Step 2a: Crossing point. The two homogeneous CPTs are equal when α1⋅mkv(s)+qγ2⋅2Pact _1· m_kv(s)+q _2· 2P_act =rα1⋅mkv(s)+γ2⋅2Pact, =r _1· m_kv(s)+ _2· 2P_act, which gives the crossing traffic (23) x∗=(q−1)γ2⋅2Pact(r−1)α1=2Pact(γ1−γ2)α2−α1.x^*= (q-1)\, _2· 2P_act(r-1)\, _1= 2P_act( _1- _2) _2- _1. For x<x∗x<x^*, GPU type 2 is the better homogeneous choice (CPT2hom<CPT1homCPT^hom_2<CPT^hom_1). For x>x∗x>x^*, GPU type 1 is the better homogeneous choice. Step 2b: Monotonicity in each sub-region. Let x=mkv(s)x=m_kv(s) for brevity. Sub-region 1 (x≤x∗x≤ x^*, short sequences, homogeneous uses GPU type 2): (24) Ghet(x)=rα1x+γ2⋅2Pactα1x+γ2⋅2Pact.G_het(x)= r _1x+ _2· 2P_act _1x+ _2· 2P_act\,. Differentiating, (25) dGhetdx=(r−1)α1⋅γ2⋅2Pact(α1x+γ2⋅2Pact)2>0. dG_hetdx= (r-1)\, _1· _2· 2P_act( _1x+ _2· 2P_act)^2>0. So GhetG_het is strictly increasing in this sub-region. At x=0x=0: Ghet(0)=(γ2⋅2Pact)/(γ2⋅2Pact)=1G_het(0)=( _2· 2P_act)/( _2· 2P_act)=1. Sub-region 2 (x≥x∗x≥ x^*, long sequences, homogeneous uses GPU type 1): (26) Ghet(x)=α1x+qγ2⋅2Pactα1x+γ2⋅2Pact.G_het(x)= _1x+q _2· 2P_act _1x+ _2· 2P_act\,. Differentiating, (27) dGhetdx=−(q−1)α1⋅γ2⋅2Pact(α1x+γ2⋅2Pact)2<0. dG_hetdx= -(q-1)\, _1· _2· 2P_act( _1x+ _2· 2P_act)^2<0. So GhetG_het is strictly decreasing in this sub-region. As x→∞x→∞: Ghet→α1/α1=1G_het→ _1/ _1=1. Combining both sub-regions, the analytic gain Ghet(x)G_het(x) increases from 11 at x=0x=0, peaks at x=x∗x=x^*, and decreases toward 11 as x→∞x→∞. Because mkv(s)m_kv(s) is continuous and strictly increasing on ℐI, if x∗∈mkv(ℐ)x^*∈ m_kv(I), there is a unique s∗∈ℐs^* satisfying mkv(s∗)=x∗m_kv(s^*)=x^* and the restricted gain peaks there. If x∗x^* lies outside mkv(ℐ)m_kv(I), the feasible interval contains only one side of the analytic curve, so the restricted gain is monotone. This establishes claim 3. Step 3: Upper-bound part of Claim 3 By the unimodality established in Step 2, the supremum of the analytic gain over x≥0x≥ 0 is attained at the crossing point x=x∗x=x^*. The same upper bound therefore applies to any restricted feasible interval. Step 3a: Peak gain at x∗x^*. Using the sub-region 1 expression at x=x∗x=x^* and noting from Eq. 23 that α1x∗=(q−1)γ2⋅2Pact/(r−1) _1x^*=(q-1) _2· 2P_act/(r-1): Ghet(x∗) G_het(x^*) =rα1x∗+γ2⋅2Pactα1x∗+γ2⋅2Pact=r⋅(q−1)(r−1)+1(q−1)(r−1)+1 = r _1x^*+ _2· 2P_act _1x^*+ _2· 2P_act= r· (q-1)(r-1)+1 (q-1)(r-1)+1 (28) =r(q−1)+(r−1)(q−1)+(r−1)=rq−1r+q−2. = r(q-1)+(r-1)(q-1)+(r-1)= rq-1r+q-2. Step 3b: Factor out cost ratio. Let w=c1/c2w=c_1/c_2. Then r=α2/α1=β1/(wβ2)r= _2/ _1= _1/(w\, _2) and q=γ1/γ2=wF2/F1q= _1/ _2=w\,F_2/F_1. The product is cost-independent: (29) rq=β1F2β2F1=η.rq= _1\,F_2 _2\,F_1=η. The peak gain Ghet(x∗)=(η−1)/(r+q−2)G_het(x^*)=(η-1)/(r+q-2) is decreasing in r+qr+q for fixed rq=η>1rq=η>1. To obtain the tightest bound, we minimize r+qr+q subject to rq=ηrq=η: r+q=β1wβ2+wF2F1.r+q= _1w\, _2+ w\,F_2F_1. By the AM-GM inequality, r+q≥2rq=2ηr+q≥ 2 rq=2 η, with equality when r=q=ηr=q= η, which occurs at the cost ratio w∗=β1F1β2F2.w^*= _1F_1 _2F_2. Step 3c: Evaluate the bound. Substituting r+q=2ηr+q=2 η and rq=ηrq=η: supx≥0,wGhet(x) _x≥ 0,\,w\;G_het(x) =η−12η−2=(η−1)(η+1)2(η−1) = η-12 η-2= ( η-1)( η+1)2( η-1) (30) =1+η2. = 1+ η2. This establishes the upper-bound part of claim 3 and completes the proof. □ Summary The cost gain in the non-trivial case (α1<α2 _1< _2, γ2<γ1 _2< _1) can be expressed as (31) Ghet(s)=rα1mkv(s)+γ2⋅2Pactα1mkv(s)+γ2⋅2Pactifmkv(s)≤x∗,α1mkv(s)+qγ2⋅2Pactα1mkv(s)+γ2⋅2Pactifmkv(s)≥x∗.G_het(s)= cases r _1m_kv(s)+ _2· 2P_act _1m_kv(s)+ _2· 2P_act&if\;m_kv(s)≤ x^*,\\[10.0pt] _1m_kv(s)+q _2· 2P_act _1m_kv(s)+ _2· 2P_act&if\;m_kv(s)≥ x^*. cases The analytic gain is unimodal in x=mkv(s)x=m_kv(s), with Ghet(0)=1G_het(0)=1, a peak of (rq−1)/(r+q−2)(rq-1)/(r+q-2) at x=x∗x=x^*, and Ghet(x)→1G_het(x)→ 1 as x→∞x→∞. On the feasible interval ℐI, this peak is attained only when x∗∈mkv(ℐ)x^*∈ m_kv(I); otherwise, the restricted gain is monotone. The tightest upper bound over all non-negative traffic values and all cost ratios is supx≥0,wGhet(x)=1+β1F2/(β2F1)2. _x≥ 0,\,w\;G_het(x)= 1+ _1F_2/( _2F_1)2. A.6. CAD Feasibility Condition for Uniform-Layer Models This section gives a simple condition for the feasibility mismatch discussed in Section 3.4. The condition applies to the uniform-layer CAD setting, where every layer has the same attention pattern and therefore the attention side can be described by a single feasible latency range. Hybrid-attention architectures, where full-attention, sliding-window-attention, or linear-attention layers may have different latency and memory behavior, require the separate discussion in Section 3.4. Following the heterogeneous notation in Section 3.1, let GPU type 1 serve the attention side and GPU type 2 serve the rest-of-model side. Let M1M_1 be the memory capacity of one GPU of type 1, β1 _1 be its memory bandwidth, and μ be the number of micro-batches sharing the attention-side memory. Then a simple upper bound on the feasible attention-stage latency is (32) Tattnmax=M1μβ1.T_attn = M_1μ _1. This expression reflects that each micro-batch can use at most M1/μM_1/μ bytes of attention-side memory, which can be streamed at bandwidth β1 _1. Let π2 _2 denote the parallelism strategy on the rest-of-model side. Let Mweights,2(π2)M_weights,2( _2) be the per-GPU model-weight footprint on GPU type 2 under this strategy, and let β2 _2 be the memory bandwidth of GPU type 2. The rest-of-model stage must at least load its shard of the model weights, so its latency is lower-bounded by (33) Trestmin(π2)=Mweights,2(π2)β2.T_rest ( _2)= M_weights,2( _2) _2. For example, under an even sharding strategy over g2(π2)g_2( _2) GPUs, one can approximate (34) Mweights,2(π2)≈Mweightsg2(π2).M_weights,2( _2)≈ M_weightsg_2( _2). For CAD to be feasible under this simplified uniform-layer model, the feasible latency ranges of the two stages must overlap. A necessary condition is therefore (35) Tattnmax≥Trestmin(π2),i.e.,M1μβ1≥Mweights,2(π2)β2.T_attn ≥ T_rest ( _2), .e., M_1μ _1≥ M_weights,2( _2) _2. When this inequality fails, even the largest feasible attention-stage latency is smaller than the unavoidable weight-loading latency on the rest-of-model side. In that case, the two CAD stages cannot be balanced, and pipeline bubbles are unavoidable. Appendix B Detailed Serving-Plan Formulation This appendix section expands the compact formulation in Section 4. The main text keeps only the abstractions needed to motivate the planner; here we spell out the structural definitions, batching notation, execution and communication latency, and memory constraint. B.1. Model Representation and Partition We model one decode step as a computation DAG =(,ℰ)G=(V,E), where each node v∈v is an operator and each edge (u,v)∈ℰ(u,v) is a data dependency. During serving, operators are executed according to a fixed topological order. For transformer inference this order is effectively a linear operator sequence =(o1,o2,…,oNmodel),o=(o_1,o_2,…,o_N_model), and all partitioning decisions below refer to positions in this sequence. Let LmodelL_model be the number of transformer layers. A layer boundary is the first operator of a layer; we denote these positions by ℓ1<ℓ2<⋯<ℓLmodel _1< _2<·s< _L_model, with ℓ1=1 _1=1. A partition block is the minimal layer-aligned repeating unit of the operator sequence. Formally, it spans LblockL_block consecutive layers, or equivalently NblockN_block consecutive operators, where LblockL_block is the smallest positive period such that the operator pattern of layers ℓk,…,ℓk+Lblock−1 _k,…, _k+L_block-1 repeats for every block-aligned starting layer k. We take the first partition block to start at ℓ1 _1 and assume LblockL_block divides LmodelL_model, giving Btotal=Lmodel/LblockB_total=L_model/L_block partition blocks. Within one partition block, the optimizer chooses a stage template =(q0,q1,…,qS),0=q0<q1<⋯<qS=Nblock.q=(q_0,q_1,…,q_S), 0=q_0<q_1<·s<q_S=N_block. Stage j contains operators at relative positions qj−1+1,…,qjq_j-1+1,…,q_j inside the block. The same template is applied to every partition block in the model, so stage j denotes the same relative operator slice in each repeated block. B.2. Device-Group Types, Replicas, and Placement A device-group type m is a logical configuration (hm,πm)(h_m, _m), where hmh_m is the GPU type and πm _m is the intra-group parallelism strategy, such as tensor parallelism, expert parallelism, or their composition. A device-group replica is one physical instance of that type. Replicas of the same type have identical configuration and execute the same stage set, but serve disjoint subsets of the global resident batch. We restrict the number of stages to S=kKS=kK for an integer k≥1k≥ 1. Stages are assigned to the ordered device-group types by the round-robin map (36) σ(j)=((j−1)modK)+1,j=1,…,S.σ(j)=((j-1) K)+1, j=1,…,S. This map is applied identically within every partition block. Consequently, each type owns exactly k stage positions per block, and the stage-to-type assignment repeats with the same partition template. The stage set assigned to type m is therefore m=j:1≤j≤S,σ(j)=m.S_m=\j:1≤ j≤ S,\ σ(j)=m\. Every replica of type m executes all stages in mS_m. Let B be the global resident batch size across the full deployment. If a replica of type m has resident batch size bmb_m, then the number of replicas of that type is (37) nm=Bbm,m=1,…,K,n_m= Bb_m, m=1,…,K, which requires bmb_m to divide B. This coupling ensures that all device-group types collectively process the same global batch. B.3. Scheduling Parameters The resident batch on each replica is split into μ microbatches. We write B^=Bμ,b^m=bmμ, B= Bμ, b_m= b_mμ, for the global microbatch size and per-replica microbatch size, respectively. The divisibility requirements are μ|Bμ B and μ|bmμ b_m for all device-group types. The resident batch bmb_m determines the number of tokens completed by a replica per round trip, while b^m b_m determines the work executed in one pipeline scheduling step. The execution latency of stage j is (38) tj=flat(qj−1+1:qj,(hσ(j),πσ(j)),b^σ(j)),j=1,…,S.t_j=f_lat\! (o_q_j-1+1:q_j,(h_σ(j), _σ(j)), b_σ(j) ), j=1,…,S. Let AjA_j denote the activation interface at the boundary after stage j, including its tensor shape, byte size per request, and parallel layout. Define j+=j+1j^+=j+1 for j<Sj<S and j+=1j^+=1 for j=Sj=S; the latter is the boundary between consecutive partition blocks. The corresponding transfer latency is (39) uj=fcomm(CLOSEAj,hσ(j),πσ(j),b^σ(j),OPENhσ(j+),πσ(j+),b^σ(j+)), splitu_j=f_comm\! (&A_j,h_σ(j), _σ(j), b_σ(j),\\ &h_σ(j^+), _σ(j^+), b_σ(j^+) ), split where fcommf_comm is obtained from communication profiles and accounts for any resharding or redistribution between the source and destination replicas. It returns the local handoff cost when the adjacent stages execute on the same replica. The j=Sj=S transfer occurs between consecutive blocks and is omitted after the final block. The steady-state round-trip time is then (40) Trtt=frtt(μ,Btotal,,),T_rtt=f_rtt(μ,B_total,t,u), where =(t1,…,tS)t=(t_1,…,t_S) and =(u1,…,uS)u=(u_1,…,u_S). The scheduling model enforces execution and transfer dependencies, models contention for compute and communication resources, and permits overlap only when the selected schedule and resources allow it. The planner evaluates this function by lightweight simulation. B.4. Memory Constraint For each device-group type m, Mweights,mM_weights,m denotes the weight memory for the stages assigned to that type, including all repeated partition blocks hosted by a replica. Let KVm(b^m,s)KV_m( b_m,s) denote the KV-cache memory required by one per-replica microbatch at sequence length s; this term is zero for stages that do not own KV-resident attention state. Since all μ microbatches are resident in the pipeline, a replica of type m must satisfy Mweights,m+μKVm(b^m,s)≤Mm,M_weights,m+μ\,KV_m( b_m,s)≤ M_m, where MmM_m is the effective memory capacity of the device group after accounting for the parallelism strategy πm _m. Together with the SLO constraint and the objective in equation 9a, this yields the compact optimization problem in equation 9. The cost-per-token objective is derived from global batch size and replica counts in Appendix B.5. B.5. Derivation of the Cost-per-Token Objective We derive the general serving-plan objective CPTplanCPT_plan in equation 9a from the perspective of the global batch size and device-group replicas introduced in Section B.2. Setup. Consider a global resident batch of B requests processed in parallel across all replicas. Each replica of device-group type m holds a per-replica resident batch of bmb_m requests. When the schedule uses μ micro-batches, the global micro-batch size is B^=B/μ B=B/μ and the per-replica micro-batch size is b^m=bm/μ b_m=b_m/μ. The system therefore requires (41) nm=Bbm=B^b^mn_m\;=\; Bb_m\;=\; B b_m replicas of type m. Throughput. In steady-state pipeline execution, each replica completes one decoding step (i.e., generates one output token for each of its bm=μb^mb_m=μ b_m resident requests) every TrttT_rtt seconds. Since there are nm=B/bmn_m=B/b_m replicas of type m, and every request passes through every device-group type, the system-wide throughput is determined by the global resident batch: (42) Throughput=BTrtttokens/s.Throughput\;=\; BT_rtt /s. Total cost rate. The instantaneous monetary cost rate of operating all replicas is (43) CostRatetotal=∑m=1Knm⋅cm=∑m=1KBbmcm=B∑m=1Kcmbm.CostRate_total\;=\; _m=1^Kn_m· c_m\;=\; _m=1^K Bb_m\,c_m\;=\;B\, _m=1^K c_mb_m. Cost per token. Dividing the total cost rate by the throughput yields (44) CPTplan _plan =CostRatetotalThroughput=B∑m=1KcmbmBTrtt =\; CostRate_totalThroughput\;=\; B\, _m=1^K c_mb_m BT_rtt =Trtt⋅∑m=1Kcmbm, =\;T_rtt\;·\; _m=1^K c_mb_m, which recovers equation 9a. Note that the global batch size B cancels, confirming that the cost-per-token objective depends only on the device-group configurations and per-replica resident batch sizes, equivalently on μb^mμ b_m, not on the absolute scale of the deployment. This cancellation assumes scale-invariant replication: adding replicas repeats the same per-replica execution and transfer pattern. Appendix C Planner Details for Section 5 This appendix details the latency and cost lower bounds used by the branch-and-bound search in Section 5.3. These bounds allow the planner to prune partial assignments before invoking the schedule simulator. C.1. Branch-and-Bound Lower Bounds After selecting a prefix Pi=(p1,…,pi)P_i=(p_1,…,p_i) of owner points, the planner computes an optimistic completion using the best remaining coordinate from each unassigned frontier. For a point p of owner m, the mandatory serialized compute work is Cm(p)=μ∑j∈mtj(p)C_m(p)=μ _j _mt_j(p). The compute lower bound for a prefix is therefore (45) Lcomp(Pi)=maxmaxm≤iCm(pm),maxm>iminp∈ℱmCm(p).L_comp(P_i)= \ _m≤ iC_m(p_m), _m>i _p _mC_m(p) \. The planner similarly lower-bounds communication by accumulating mandatory serialized transfer work at each network endpoint. A boundary whose endpoint has not yet been assigned is minimized over that owner’s feasible replica counts. Boundaries are minimized independently, which can only make the estimate optimistic. Denoting this endpoint bound by Lnet(Pi)L_net(P_i) gives the valid latency lower bound (46) LT(Pi)=maxLcomp(Pi),Lnet(Pi)≤Trtt.L_T(P_i)= \L_comp(P_i),L_net(P_i) \≤ T_rtt. Suffix minima also lower-bound the normalized cost of any completion; let RLB(Pi)R_LB(P_i) denote this bound. The best possible objective below the prefix satisfies CPTLB(Pi)=LT(Pi)RLB(Pi)≤CPTplanCPT_LB(P_i)=L_T(P_i)\,R_LB(P_i) _plan. Appendix D Notation Symbol Description Model and workload o Operator sequence (o1,…,oNmodel)(o_1,…,o_N_model) of one decode step NmodelN_model Total number of operators LmodelL_model Total number of transformer layers PactP_act GEMM-side parameters activated per request after model-parallel sharding (routed experts only for MoE) MweightsM_weights Resident model-weight footprint after sharding, including inactive experts Dattn(b),Dgemm(b)D^attn(b),\,D^gemm(b) Aggregated attention and GEMM memory traffic at batch size b Wgemm(b)W^gemm(b) Aggregated GEMM FLOPs at batch size b s Average sequence length (workload parameter) mkv(s)m_kv(s) Per-request KV-cache memory at sequence length s b∗b^* GEMM compute-bound threshold batch size bmax(s)b_ (s) Max feasible batch size under memory at sequence length s b¯(s) b(s) Continuous relaxation of bmax(s)b_ (s) used in closed-form analysis GPU hardware (per GPU type j) cjc_j Unit-time monetary cost of GPU type j ($/s); shorthand c in the single-type case, c1,c2c_1,c_2 for two types βj _j Memory bandwidth (bytes/s); shorthand β, or β1,β2 _1, _2 FjF_j Peak compute throughput (FLOPS); shorthand F, or F1,F2F_1,F_2 MjM_j Memory capacity (bytes); shorthand M, or M1,M2M_1,M_2 αj,γj _j,\, _j Monetary cost per byte of memory traffic and per FLOP, respectively, for GPU type j CPTCPT Cost per token; CPTplanCPT_plan is the serving-plan objective; superscripts coloccoloc/homhom/hethet denote serving modes Theoretical analysis (Section 3) Ghom(s)G_hom(s) Cost gain of homogeneous ODS over colocated serving Ghet(s)G_het(s) Cost gain of heterogeneous over homogeneous ODS η Hardware ratio (β1F2)/(β2F1)( _1F_2)/( _2F_1) of two GPU types Model structure (partition block; Section 4) LblockL_block Number of transformer layers per partition block NblockN_block Number of operators per partition block BtotalB_total Number of partition blocks (=Lmodel/Lblock=L_model/L_block) Decision variables (Section 4) S Number of stages per partition block q Stage-boundary offsets (q0,…,qS)(q_0,…,q_S) within a block K Number of device-group types (hm,πm)(h_m, _m) Device-group configuration (GPU type, parallelism) of type m bmb_m Per-replica resident batch size of device-group type m b^m b_m Per-replica micro-batch size of device-group type m (=bm/μ=b_m/μ) μ Number of micro-batches Derived / scheduling quantities (Section 4) mS_m Stages assigned to device-group type m (placement group) σ(j)σ(j) Placement map from stage j to its device-group type B Global resident batch size across all replicas nmn_m Replicas of device-group type m (=B/bm=B/b_m) cmc_m Unit-time monetary cost of one replica of device-group type m MmM_m Effective memory capacity of a replica of type m under πm _m Mweights,mM_weights,m Total weight memory of stages assigned to type m KVm(b^m,s)KV_m( b_m,s) KV-cache memory for one per-replica micro-batch on a replica of type m tjt_j Execution latency of stage j uju_j Activation-transfer latency at the boundary after stage j TrttT_rtt Pipeline round-trip time (per-token latency in steady state) TSLOT_SLO TPOT service-level objective Planner (Section 5) LsubL_sub, n Sub-block length in layers and repetitions per block (n=Lblock/Lsubn=L_block/L_sub) subq_sub Sub-block stage template, tiled over the n repetitions m(p)T_m(p) Stage-latency vector of owner m’s point p over mS_m rm(p),ρm(p)r_m(p),\, _m(p) Normalized cost and GPU count of point p: cm(p)/bm(p)c_m(p)/b_m(p), gm(p)/bm(p)g_m(p)/b_m(p) ℱmF_m Network-safe Pareto frontier of owner m Table 4. Unified notation used throughout the paper.