Paper deep dive
X-Stage: An Overlooked Pipeline Stage for Communication-Computation Overlap in DiT Inference
Jianwen Xian, Zhiyuan Xu, Yuchen Li, Ziliang Lai, Kang He, Zhen Huang, Aichen Feng, Jinyan Chen, Yilin Zhang, Qinqin Chen, Chengru Song
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Fine-grained, device-initiated communication lets persistent GPU kernels in distributed diffusion transformer (DiT) inference issue remote stores and overlap data movement with Tensor Core computation. Existing systems schedule when communication is issued and when received data becomes consumable, but omit post-issue progress before remote-visible completion, making sender backpressure hard to predict. We identify X-Stage, a software-visible post-issue pipeline stage. Measurements on an eight-GPU node with a recent NVIDIA architecture show that short remote-store bursts drain as the issuer resumes work, whereas sustained injection exhausts finite outstanding capacity and delays later issues. A lightweight Burst-Gap model parameterized by backpressure-free issue time, effective drain rate, and outstanding capacity predicts issue overhead, recovery between bursts, and the onset of backpressure. Guided by the model, we redesign two communication-computation fused kernels. For DeepGEMM MegaMoE, interleaving Linear-1 and Linear-2 work across expert waves places computation between concentrated remote-store bursts, yielding a 1.18x geometric-mean and 1.62x maximum kernel speedup over the Expert-Wave baseline across 84 configurations. For Ulysses sequence-parallel attention, tile-granular fusion of the post-attention All-to-All with FlashAttention lets an output-tile owner issue remote stores and resume computation without a dedicated communication warp or streaming multiprocessor. FlashAttention-3 and FlashAttention-4 reach maximum sender-visible speedups of 1.43x and 1.42x over serial execution, and at long sequences their steady-state times approach those of FlashAttention alone. These results establish post-issue progress as a measurable scheduling lever for shaping bursts, avoiding backpressure, and hiding sender-side overhead.
Tags
Links
- Source: https://arxiv.org/abs/2607.23264v1
- Canonical: https://arxiv.org/abs/2607.23264v1
Trouble viewing inline? Open PDF directly →
Full Text
96,612 characters extracted from source content.
Expand or collapse full text
X-Stage: An Overlooked Pipeline Stage for Communication–Computation Overlap in DiT Inference Jianwen Xian KlingAI ResearchChina , Zhiyuan Xu Tsinghua UniversityChina , Yuchen Li Tsinghua UniversityChina , Ziliang Lai KlingAI ResearchChina , Kang He KlingAI ResearchChina , Zhen Huang NVIDIAUnited States , Aichen Feng NVIDIAUnited States , Jinyan Chen NVIDIAUnited States , Yilin Zhang NVIDIAUnited States , Qinqin Chen NVIDIAUnited States and Chengru Song KlingAI ResearchChina Abstract. Fine-grained, device-initiated communication lets persistent GPU kernels in distributed diffusion transformer (DiT) inference issue remote stores and overlap data movement with Tensor Core computation. Existing systems schedule when communication is issued and when received data becomes consumable, but omit post-issue progress before remote-visible completion, making sender backpressure hard to predict. We identify X-Stage, a software-visible post-issue pipeline stage. Measurements on an eight-GPU node with a recent NVIDIA architecture show that short remote-store bursts drain as the issuer resumes work, whereas sustained injection exhausts finite outstanding capacity and delays later issues. A lightweight Burst–Gap model parameterized by backpressure-free issue time, effective drain rate, and outstanding capacity predicts issue overhead, recovery between bursts, and the onset of backpressure. Guided by the model, we redesign two communication–computation fused kernels. For DeepGEMM MegaMoE, interleaving Linear-1 and Linear-2 work across expert waves places computation between concentrated remote-store bursts, yielding a 1.18×1.18× geometric-mean and 1.62×1.62× maximum kernel speedup over the Expert-Wave baseline across 84 configurations. For Ulysses sequence-parallel attention, tile-granular fusion of the post-attention All-to-All with FlashAttention lets an output-tile owner issue remote stores and resume computation without a dedicated communication warp or streaming multiprocessor. FlashAttention-3 and FlashAttention-4 reach maximum sender-visible speedups of 1.43×1.43× and 1.42×1.42× over serial execution, and at long sequences their steady-state times approach those of FlashAttention alone. These results establish post-issue progress as a measurable scheduling lever for shaping bursts, avoiding backpressure, and hiding sender-side overhead. GPU communication, NVLink, remote store, communication–computation fusion, mixture of experts, sequence parallelism, performance modeling 1. Introduction The widening gap between GPU compute throughput and intra-node interconnect bandwidth increasingly places inter-GPU data movement on the critical path of distributed-model execution (Sul et al., 2026; Jin et al., 2026). Tensor, expert, and sequence parallelism all exchange activations or intermediate results across GPUs, making communication a central scalability bottleneck for large-model training and inference (Chang et al., 2024; Zhang et al., 2025; Jacobs et al., 2023; Jin et al., 2026). Conventional GPU collectives are launched by the host or by separate communication kernels and overlap with computation mainly at operator or stream granularity. Peer memory mapping, symmetric memory, and device-initiated communication instead allow a GPU kernel to construct a remote address and move data directly to peer memory (NVIDIA, 2026d; Ma et al., 2026). We refer to writes from a GPU kernel to peer-mapped or symmetric memory as device-initiated remote stores. They may be implemented as global-store instructions to a peer address or through higher-level one-sided primitives. Recent systems, including FLUX, Comet, TileLink, MegaScale-MoE, and ParallelKittens, decompose communication and computation at cooperative thread array (CTA), tile, or warp granularity. Kernel fusion, task reordering, device-side signaling, resource partitioning, and wave scheduling expose substantially more overlap (Chang et al., 2024; Zhang et al., 2025; Zheng et al., 2025; Jin et al., 2026; Sul et al., 2026). These techniques describe which communication and compute tiles may execute concurrently, which role issues each transfer, and when a consumer may use the remote data. This information is sufficient to construct a correct tile pipeline, but not to predict the sender’s execution after a remote store is issued. In particular, existing abstractions do not expose how far the issuer may advance ahead of remote-visible completion or when sustained injection will exhaust downstream resources and backpressure the compute pipeline. We encountered this limitation while analyzing MegaMoE, DeepGEMM’s persistent mixture-of-experts (MoE) kernel. MegaMoE fuses token Dispatch, Linear-1, activation, Linear-2, and Combine into a persistent kernel organized as expert waves (DeepSeek-AI, 2026b, a). Its task-level timeline represents Combine as one communication phase. Under a conservative completion-coupled interpretation—in which the remote-store issuer is assumed unable to resume subsequent local work until the issued stores become remotely visible—the entire Combine interval lies on the local critical path. Using the published stage times, this interpretation predicts at most approximately 1.5×1.5× speedup over the serial stage sum, below the 1.56×1.56× reported by the implementation (DeepSeek-AI, 2026b, a). This mismatch suggests that the issuer can resume useful execution after issuing the stores while the requests continue to progress toward remote-visible completion. DiT-MoE workloads combine long sequences and fine-grained experts with input-dependent routing (Fei et al., 2024; Sun et al., 2025; Yuan et al., 2025). Although EPLB mitigates placement imbalance, it does not eliminate per-input routing skew (DeepSeek-AI, 2025). Longer sequences amplify Combine traffic to persistent hot experts, while fine-grained experts shorten the gaps between Linear-2 bursts, making MegaMoE more susceptible to sender-side backpressure. Parameterized microbenchmarks reveal two distinct regimes. Short remote-store bursts incur near-baseline sender-visible issue time, while the issued requests continue to progress toward remote-visible completion after the issuer resumes execution. Under sustained and concentrated injection, outstanding requests accumulate and increase the issue time of subsequent bursts. A producer-side gap that introduces no new remote stores allows these requests to drain and restores issue time toward its backpressure-free baseline. We call this software-visible post-issue pipeline stage—the interval between remote-store issue and remote-visible completion—X-Stage. As Figure 1 illustrates, an epilogue may issue a burst and return to a subsequent mainloop while the burst drains through X-Stage. Figure 1. Remote stores issued by an epilogue continue to make progress in X-Stage while the next mainloop performs useful computation. The remote-store burst and the intervening mainloop form the recurring burst–gap pattern used throughout this paper. Timeline showing an epilogue remote-store burst followed by a compute mainloop; the burst drains through X-Stage while computation proceeds. X-Stage creates both an overlap opportunity and a backpressure hazard. Useful computation can cover post-issue draining, but repeated bursts can fill the finite effective outstanding capacity and stall later stores, the epilogue, and eventually the Tensor Core producer. Consequently, aggregate compute and communication times alone cannot determine whether communication remains hidden. We capture this behavior with a lightweight Burst–Gap model. Each burst injects a volume of remote stores over a short interval; the following producer-side gap issues no new remote stores and allows outstanding requests to drain concurrently with useful work. Three measurable quantities, the backpressure-free issue time, effective drain rate, and effective outstanding capacity, predict the sender-visible issue time, the gap needed for recovery, and the onset of backpressure. This abstraction matches GEMM-like kernels, in which an epilogue emits a burst and the next matrix multiply–accumulate (MMA) mainloop supplies the gap. The model leads to two complementary scheduling actions. In MegaMoE, an expert wave executes many Linear-2 epilogues consecutively, concentrating Combine stores into long bursts. We interleave ready Linear-1 work from later waves with Linear-2 work, redistributing computation between bursts without changing the dependencies or communication volume. Across 84 configurations, the resulting interleaved scheduler achieves a 1.18×1.18× geometric-mean speedup, a 1.17×1.17× median speedup, and a 1.62×1.62× maximum speedup over the Expert-Wave baseline. For Ulysses sequence-parallel attention (Jacobs et al., 2023), we fuse each FlashAttention output tile with its post-attention All-to-All. The role that owns the output issues the corresponding remote stores and immediately resumes the next tile; the next FlashAttention Q-loop supplies a gap in which the stores drain. This piggybacked design reserves neither a dedicated communication warp nor a dedicated streaming multiprocessor (SM). FlashAttention-3 and FlashAttention-4 achieve maximum sender-visible speedups of 1.43×1.43× and 1.42×1.42×, respectively, over serial execution. As the sequence length and Q-loop compute gap grow, the fused sender-visible steady-state time approaches FlashAttention-only time. This paper makes the following contributions: (1) A software-visible post-issue abstraction. We identify X-Stage, the finite interval in which an accepted remote store can make progress concurrently with subsequent computation. This abstraction explains why neither fully blocking nor unbounded-asynchrony models capture sender-visible behavior. (2) Characterization and prediction. Parameterized microbenchmarks of remote stores measure issue time without backpressure, effective drain rate, burst–gap recovery, and effective outstanding capacity. The resulting Burst–Gap model predicts steady-state period, sender-visible issue overhead, and capacity-induced backpressure without application-specific refitting. (3) X-Stage-aware kernel designs. For MegaMoE, cross-wave Linear-1/Linear-2 interleaving reshapes concentrated bursts of remote stores. For fused FlashAttention and A2A, a piggybacked tile pipeline uses the existing Q-loop as a post-issue drain window and avoids long-lived communication roles. (4) Performance and mechanism-level validation. We evaluate 84 MegaMoE configurations and two FlashAttention generations. Kernel timelines and per-tile instrumentation connect the measured speedups to the issue, drain, capacity, and staging effects predicted by the model. 2. Background and Motivation 2.1. GPU Communication and Fine-Grained Fusion Modern multi-GPU nodes allow a GPU to access peer memory directly over high-speed fabrics such as NVLink (NVIDIA, 2026c). Peer memory mapping, a unified virtual address space, and symmetric memory let a kernel construct addresses in remote memory and initiate data movement from the device (NVIDIA, 2026a, d). NVSHMEM, for example, provides a partitioned-global-address-space (PGAS) model and symmetric-memory interface for fine-grained one-sided operations without returning to the host or launching a separate collective kernel for every transfer (NVIDIA, 2026d). Fine-grained communication is increasingly important in large-model parallelism. Tensor parallelism invokes All-Gather, Reduce-Scatter, or All-Reduce (Shoeybi et al., 2019); expert parallelism dispatches tokens before expert computation and combines outputs afterward; Ulysses sequence parallelism uses two All-to-All operations to convert between sequence and head partitions around attention (Jacobs et al., 2023). As low-precision Tensor Core throughput increases, each compute phase shortens relative to inter-GPU data movement. Operator- or stream-level concurrency may therefore leave substantial communication exposed, motivating decomposition at tile, CTA, or warp granularity. Recent systems construct fine-grained pipelines through task decomposition, kernel fusion, specialized execution roles, and device-side synchronization (Chang et al., 2024; Zhang et al., 2025; Zheng et al., 2025; Jin et al., 2026; Sul et al., 2026). Although their designs differ in task granularity, resource organization, and scheduling mechanism, they primarily expose the software schedule: which communication and compute tiles may execute concurrently, which role issues each transfer, and when remote data becomes ready for consumption. A tile-level signal can report remote readiness, but it does not reveal how far the sender may continue before that event or when finite downstream resources will backpressure the issuer. X-Stage complements, rather than replaces, tile-level scheduling by modeling this post-issue progress. At a higher level, an ideal fused latency is often estimated as Tideal=max(Tcompute,Tcomm).T_ideal= \! (T_compute,T_comm ). This aggregate model asks whether total computation can cover total communication, but treats communication as a single interval. Remote-store issue and remote-visible completion are distinct events. Issue means that the sender has injected a write request, not that the data is already visible at the destination. The request may continue to progress while later computation runs; under sustained injection, however, unfinished requests may accumulate, lengthen later issue operations, and backpressure the epilogue and upstream computation. Aggregate compute and communication times alone cannot capture this transition. 2.2. MegaMoE Execution MegaMoE is a representative fine-grained, communication–computation fused MoE kernel. In an expert-parallel MoE layer, a router selects one or more experts for each token. Because experts reside on different GPUs, Dispatch first sends each token to its destination rank. Each expert then evaluates a two-layer feed-forward network: Linear-1 projects the hidden dimension to an intermediate dimension, an activation such as SwiGLU (Shazeer, 2020) is applied, and Linear-2 projects back to the hidden dimension. Combine returns expert outputs to the source rank and aggregates the top-k results using the router weights. A conventional implementation launches separate kernels for Dispatch, grouped GEMM, activation, the second grouped GEMM, and Combine. MegaMoE fuses these stages into a persistent mega-kernel and uses warp specialization for Dispatch, data movement, Tensor Core computation, and epilogue work (DeepSeek-AI, 2026a). In the public implementation, the Linear-2 epilogue reads accumulator results and writes them to a symmetric Combine buffer with remote stores. Although separate roles execute the Tensor Core mainloop and the epilogue, finite on-chip accumulator and staging resources couple them as producer and consumer. MegaMoE groups local experts into expert waves to organize locality and execution. The original schedule generally executes a wave’s Linear-1 work, then its activation and Linear-2 work, before advancing to the next wave. The Linear-1 work of different waves has no direct neural-network dependency. Once Dispatch data and a destination buffer are ready, a later wave’s Linear-1 can, in principle, begin before the previous wave’s remote Combine stores complete. 2.3. A Task-Level Modeling Mismatch MegaMoE’s published timeline represents Dispatch, Linear-1, activation, Linear-2, and Combine as task-level stages (DeepSeek-AI, 2026b); Figure 2(a) shows this view. A natural conservative interpretation is that local epilogue and staging resources remain coupled to a Linear-2 result until the corresponding Combine operation becomes remotely visible. We call this the completion-coupled interpretation. Let TLin1T_Lin1, TActT_Act, and TLin2T_Lin2 denote the steady-state stage times of Linear-1, activation, and Linear-2 within an expert wave. Let TCombineccT_Combine^c span the local epilogue through remote-visible completion. Ignoring pipeline fill and drain, the per-wave steady-state time is approximated by Twavecc≈max(TLin1,TAct)+max(TLin2,TCombinecc).T_wave^c≈ \! (T_Lin1,T_Act )+ \! (T_Lin2,T_Combine^c ). We decompose the completion-coupled Combine interval as TCombinecc=Tlocalepilogue+TRSissue+Tpost-issuecompletion,T_Combine^c=T_local\ epilogue+T_RS\ issue+T_post -issue\ completion, where the final term is the remaining time from accepted issue to remote visibility. This model permits temporary overlap between the Linear-2 producer and result-sending role through finite on-chip staging, but treats remote completion as the steady-state progress constraint. Applying the model to the reported stage times yields an estimated speedup ceiling of about 1.5×1.5× over the serial baseline, whereas the implementation reaches 1.56×1.56× (DeepSeek-AI, 2026b, a). Although this small discrepancy is not by itself a mechanism proof, the completion-coupled estimate cannot explain the observed speedup and motivates a direct measurement of post-issue progress. 2.4. Decomposing Combine: From Issue to Remote Visibility Figure 2. Task-level and X-Stage-aware views of the MegaMoE expert-wave timeline. The task-level view treats Combine as a single communication stage, whereas the X-Stage-aware view separates sender-visible remote-store issue from subsequent progress toward remote-visible completion. Side-by-side timelines of the original expert-wave schedule and the X-Stage-aware interleaved schedule. Figure 2(a) treats Combine as one communication stage. This representation captures application dependencies and the task-level timeline but conflates two sender-side events. A Linear-2 epilogue first reads accumulators, performs data conversion and address calculation, and writes the results to the symmetric Combine buffer. The remote-store issue portion ends when the sender accepts those write instructions; it does not imply that the corresponding data is already visible remotely. Figure 2(b) separates these events. Once the remote stores are accepted, the issuing role may continue with later computation while the requests progress toward the destination. This decoupling explains why a completion-coupled estimate can understate MegaMoE speedup. Application latency alone, however, cannot reveal the progress rate, the amount by which the issuer can lead completion, or how sustained injection changes sender-visible execution. We therefore use controlled remote-store microbenchmarks to measure effective draining, outstanding capacity, and backpressure directly. 2.5. Research Questions The analysis above motivates three research questions: • RQ1: Post-issue progress and resource bounds. How do device-initiated remote stores progress after issue, how far can the issuing role advance ahead of remote-visible completion, and which effective resource bounds are visible to software? • RQ2: Characterization and prediction. Can a lightweight model based on backpressure-free issue time, effective drain rate, and effective outstanding capacity predict execution period, recovery, and backpressure across burst volumes, producer configurations, and compute gaps? • RQ3: X-Stage-aware kernel design. How can these measurements guide fused kernels? Specifically, can a scheduler reshape MegaMoE bursts to reduce request accumulation, and can a FlashAttention–A2A pipeline cover post-issue progress without reserving long-lived communication resources? 3. Characterizing and Modeling X-Stage Figure 3. Remote-store microbenchmark characterization of X-Stage. (a) Effect of the producer-side gap on sender-visible issue time; (b) measured burst period compared with the Burst–Gap model; (c) the effective-capacity knee for an isolated burst; and (d) the shared effective drain rate across producer configurations. Four plots showing issue-time recovery with gap, the Burst–Gap period model, the isolated-burst capacity knee, and drain-rate calibration. The application-level mismatch in Section 2.3 suggests that remote-store issue and remote-visible completion are not fully coupled. This section uses controlled microbenchmarks to characterize that post-issue behavior and develops a Burst–Gap model for recovery, steady-state burst period, and capacity-induced backpressure. The experiments isolate three quantities: how quickly the sender can inject a burst in the absence of backpressure, how quickly accepted requests drain, and how much data can remain effectively outstanding before issue stalls. We use producer in this section for a concurrent role that issues remote stores; it is distinct from a Tensor Core compute producer unless stated otherwise. Table 1 summarizes the notation. Table 1. Notation for the X-Stage Burst–Gap model. Symbol Definition K Concurrent remote-store producer count. ℬB Bytes per producer per burst. V Aggregate burst volume, =ℬV=KB; tV_t denotes per-tile volume. bm,bnb_m,b_n Compute-tile M/N dimensions. G Useful producer-side time between bursts with no remote stores. Tiss(;,)T_iss(G;K,V) Sender-visible burst issue time, including backpressure. Tiss0(,)T_iss^0(K,V) Backpressure-free burst issue time. ΔTiss T_iss Backpressure overhead, ΔTiss=Tiss−Tiss0 T_iss=T_iss-T_iss^0. TperiodT_period Burst start-to-start period, Tperiod=Tiss+T_period=T_iss+G. ℛR Effective aggregate drain rate (717 GB/s measured). ()Q(K) Effective outstanding capacity for K producers. ∗(,)G^*(K,V) Minimum recovery gap, ∗=[/ℛ−Tiss0]+G^*=[V/R-T_iss^0]_+. qpeak0(,)q_peak^0(K,V) Peak outstanding volume without a capacity limit, [−ℛTiss0]+[V-RT_iss^0]_+. ℳXM_X Platform parameters (Tiss0,ℛ,)(T_iss^0,R,Q). 3.1. Remote-Store Microbenchmarks Our parameterized benchmark generates a burst from K concurrent producers. Each producer writes ℬB bytes, for an aggregate volume =ℬV=KB. We use two modes. Periodic bursts. Producers repeatedly issue a burst of volume V, modeling the remote stores in a GEMM epilogue, and then execute a producer-side gap of duration G, modeling a subsequent MMA mainloop. We record both TissT_iss and the start-to-start period TperiodT_period. A zero gap drives the path to a drain-limited steady state and exposes the effective drain rate. Sweeping G reveals how requests left by one burst affect the next burst. Isolated bursts. Each measurement starts after sufficient recovery, issues one burst of volume V, and records its sender-visible issue time. This mode detects whether a single burst can exhaust the effective outstanding capacity. A local-memory control uses the same number of producers, store width, address generation, and loop structure. It separates ordinary instruction overhead from stalls caused by outstanding requests on the remote path. 3.2. The X-Stage Burst–Gap Model Periodic bursts form a producer–consumer process: software injects requests, and the downstream path drains them. We describe its software-visible behavior with three quantities: • the backpressure-free issue time Tiss0T_iss^0 for a recovered burst with (,)(K,V); • ℛR, the effective aggregate drain rate; and • ()Q(K), the effective outstanding capacity for the producer configuration. The model assumes a work-conserving effective drain at rate ℛR over the measured regime. It is a fluid, steady-state abstraction rather than a claim about a particular physical queue. If the gap is long enough to drain the previous burst, its unfinished requests do not measurably delay the next issue, and Tiss=Tiss0T_iss=T_iss^0. With a shorter gap, requests carry across periods. Flow conservation in a drain-dominated steady state requires each period to drain the volume injected during that period: (1) Tperiod(;,)=ℛ.T_period(G;K,V)= VR. By definition, a period comprises issue followed by the producer-side gap: (2) Tperiod(;,)=Tiss(;,)+.T_period(G;K,V)=T_iss(G;K,V)+G. Thus, in the drain-dominated regime, Tiss(;,)=ℛ−.T_iss(G;K,V)= VR-G. Issue cannot become faster than the backpressure-free baseline, giving the complete Burst–Gap relation (3) Tiss(;,)=max(Tiss0(,),ℛ−).T_iss(G;K,V)= \! (T_iss^0(K,V), VR-G ). The resulting issue-time overhead is (4) ΔTiss(;,)=[ℛ−Tiss0(,)]+, T_iss(G;K,V)= [ VR-G-T_iss^0(K,V) ]_+, where [x]+=max(x,0)[x]_+= (x,0). A scheduler can reduce this overhead by decreasing burst volume or increasing useful work between bursts. The recovered plateau is reached when (5) ≥∗(,)=[ℛ−Tiss0(,)]+.G ^*(K,V)= [ VR-T_iss^0(K,V) ]_+. 3.3. Calibrating the Effective Drain Rate We first calibrate ℛR independently. With zero-gap periodic bursts, producers continuously inject requests and reach a drain-dominated steady state, for which Tperiodss()≈ℛ.T_period^s(V)≈ VR. Figure 3(d) reports the steady-state period across producer counts K and aggregate volumes V. Different /ℬK/B decompositions of the same volume produce similar periods, and larger volumes converge to one linear trend. The downstream drain, not a single producer’s local issue throughput, therefore controls this regime. Fitting the drain-dominated points to Tperiodss=/ℛT_period^s=V/R gives ℛ≈717R≈ 717 GB/s on the measured system. All subsequent gap and capacity experiments use this value without per-curve refitting. 3.4. Validating Burst–Gap Behavior After fixing ℛR from Figure 3(d), we sweep G and obtain Tiss0T_iss^0 from the large-gap plateau. Equations 1 and 3 then predict the full sweep with no additional fitted parameter. Figure 3(a) shows the sender-visible view. At short gaps, requests left by the preceding burst lengthen the next issue. Increasing the gap reduces this extra stall approximately as /ℛ−V/R-G until the issue time reaches Tiss0T_iss^0. The local-memory control is nearly gap-insensitive, indicating that recovery comes from post-issue progress on the remote path rather than from the gap instructions or store loop itself. Figure 3(b) shows the corresponding steady-state period. At short gaps, measurements stay on the drain-limited plateau /ℛV/R. Near the predicted ∗G^*, execution switches to the Tiss0+T_iss^0+G branch and increases linearly with the gap. The measurements follow Tperiod=max(Tiss0+,/ℛ),T_period= \! (T_iss^0+G,V/R ), rather than the completion-coupled sum Tperiodcc=+ℛ.T_period^c=G+ VR. The producer-side gap therefore overlaps with draining; it is not simply added to communication time. 3.5. Measuring Effective Outstanding Capacity Periodic bursts expose steady state across cycles. Isolated bursts ask whether one sufficiently large burst backpressures itself even when X-Stage is initially empty. During a backpressure-free issue interval, the producers inject V bytes while approximately ℛTiss0(,)RT_iss^0(K,V) bytes drain. The unconstrained outstanding volume at the end of issue is therefore (6) qpeak0(,)=[−ℛTiss0(,)]+.q_peak^0(K,V)= [V-RT_iss^0(K,V) ]_+. If qpeak0≤()q_peak^0 (K), the isolated issue remains at Tiss0T_iss^0. Otherwise, injection beyond the capacity must wait for downstream draining, yielding (7) Tissiso(,)=max(Tiss0(,),−()ℛ).T_iss^iso(K,V)= \! (T_iss^0(K,V), V-Q(K)R ). Figure 3(c) shows the isolated issue time versus aggregate volume. Small bursts remain on the backpressure-free baseline. Beyond a producer-configuration-dependent knee, issue time rises toward the drain-limited branch, demonstrating that the lead over downstream completion is finite. For =148K=148, the isolated issue-time knee occurs at about 33 KB per producer, corresponding to an aggregate burst volume of approximately 4.77 MiB. At ℬ=32B=32 KB, the measured backpressure-free issue floor is Tiss0≈0.76μT_iss^0≈ 0.76~ . Applying Equation 6 at the knee and accounting for the data drained during issue gives ≈4.25Q≈ 4.25 MiB. 3.6. Model Summary The three independently measurable quantities form the platform model ℳX=(Tiss0(,),ℛ,()).M_X= (T_iss^0(K,V),R,Q(K) ). Tiss0T_iss^0 captures backpressure-free injection for a producer configuration and burst volume, ℛR captures aggregate post-issue draining, and Q captures the finite lead over downstream completion. We extract them from the recovered issue-time plateau, zero-gap steady-state period, and isolated-burst knee, respectively. Together they explain three observations: (1) sender-visible issue can end before remote-visible completion, allowing accepted requests to progress concurrently with subsequent work; (2) a producer-side gap overlaps with post-issue draining, so steady-state period follows a max law rather than a completion-coupled sum; and (3) when outstanding volume exceeds effective capacity, backpressure first lengthens remote-store issue and can then propagate through local staging to the compute producer. The next section applies the same model in two regimes. MegaMoE interleaves work across expert waves to enlarge gaps between concentrated Linear-2 bursts. FlashAttention–A2A uses the following Q-loop to cover the post-issue progress of the previous output tile. 4. X-Stage-Aware Kernel Design Once the sender accepts a remote store, the request can continue toward remote completion through X-Stage. This post-issue progress changes the scheduling boundary of a fused kernel: software must control not only when communication is issued, but also how quickly accepted requests accumulate and how much useful work is available while they drain. A conventional fused GEMM is often described as a Load–Compute–Epilogue pipeline. When the epilogue contains remote stores, ending this description at issue hides an additional stage, as Figure 1 shows. There are two distinct levels of decoupling. Local staging provides compute–issue decoupling: a Tensor Core producer deposits a result in an accumulator or staging buffer, and an epilogue role issues the stores. X-Stage provides issue–completion decoupling: after issue is accepted, requests can progress without continuously occupying the issuing warp. For an aggregate burst volume V followed by a producer-side gap G, the Burst–Gap model gives (8) Tiss(;,)=max(Tiss0(,),ℛ−),T_iss(G;K,V)= \! (T_iss^0(K,V), VR-G ), and (9) ΔTiss(;,)=[ℛ−Tiss0(,)]+. T_iss(G;K,V)= [ VR-G-T_iss^0(K,V) ]_+. The immediate scheduling objective is to reduce ΔTiss T_iss without adding work or violating dependencies. 4.1. A Design Test with Two Actions The model yields a compact test for recurring bursts. Under the fluid-model and steady-state assumptions of Section 3.2, X-Stage does not add sender-visible backpressure when both (10) Tiss0+≤ℛ⏟long-term injection-rate bound,[−ℛTiss0]+≤⏟single-burst capacity bound VT_iss^0+G _long-term injection-rate bound, [V-RT_iss^0 ]_+ _single-burst capacity bound hold. The first bound prevents accumulation across periods; the second prevents one initially isolated burst from exhausting effective capacity. Satisfying these bounds removes the modeled backpressure overhead. It does not, by itself, prove that all issue instructions, synchronization, or completion requirements are absent from the application critical path. An X-Stage-aware design follows three steps. First, calibrate ℳX=(Tiss0,ℛ,)M_X=(T_iss^0,R,Q) with the microbenchmarks in Section 3. Second, audit the target kernel to obtain its aggregate burst volume V and natural compute gap G. Third, choose one of two actions: • Criterion violated: reshape injection. If the natural gap is too short or the burst too large, redistribute useful work between bursts or reduce burst aggregation. MegaMoE follows this path (Section 4.2): consecutive Linear-2 epilogues inject faster than the path drains, so ready Linear-1 work from later waves is moved between them. • Criterion satisfied: piggyback on existing computation. If the existing compute pipeline already supplies a sufficient gap, a role that owns the output can issue a short burst and return to useful work while X-Stage drains it. A dedicated communication role cannot increase the measured downstream drain rate merely by waiting after issue, although it may still be useful for address generation, issue throughput, or synchronization in a different design. FlashAttention–A2A follows this path (Section 4.3): a full Q-loop separates output bursts, and the gap grows with sequence length. Both actions use the same calibrated parameters. MegaMoE changes the ordering of independent Linear-1 and Linear-2 tiles to smooth injection and control X-Stage occupancy. FlashAttention–A2A lets the existing output-owning role issue each burst and resume computation, avoiding a long-lived communication warp or SM whose sole purpose would be post-issue progress. 4.2. MegaMoE: Capacity-Aware Scheduling across Expert Waves Fine-grained experts shorten the Linear-2 MMA mainloop, which is the natural gap between successive epilogue bursts. The original expert-wave schedule therefore frequently operates in the drain-dominated regime and violates the rate bound in Equation 10. Figure 4. X-Stage-aware MegaMoE scheduling with a separate epilogue role. The expert-wave scheduler places Linear-2 tiles consecutively, causing their epilogues to issue concentrated remote-store bursts. The interleaved scheduler inserts ready Linear-1 work between these bursts, allowing earlier requests to drain during useful computation. Comparison of concentrated Linear-2 remote-store bursts with a schedule that interleaves ready Linear-1 computation. Figure 4(a) shows the original schedule. Let tV_t be the Combine volume emitted by one Linear-2 tile on one producer, and let actK_act be the number of producers whose tile boundaries align in a burst. The aggregate volume used by the X-Stage model is =acttV=K_actV_t. Let TL1mmaT_L1^mma and TL2mmaT_L2^mma denote the Linear-1 and Linear-2 tile mainloop times. Consecutive Linear-2 tiles provide only wave≈TL2mmaG_wave≈ T_L2^mma. For fine-grained experts, Tiss0+TL2mma>ℛ, VT_iss^0+T_L2^mma>R, so outstanding requests rise to the effective capacity. They cannot grow further in steady state; instead, backpressure lengthens issue and propagates through local staging to the Tensor Core producer. 3procedure Scheduler(g,r)(g,\ r) 4 k1←gk_1← g; k2←gk_2← g 5 while k1<P⋅P1k_1<P\!·\!P_1 or k2<P⋅P2k_2<P\!·\!P_2 do 6 p1←⌊k1/P1⌋p_1← k_1/P_1 (+∞+∞ if L1L_1 done) 7 p2←⌊k2/P2⌋p_2← k_2/P_2 8 if k2<P⋅P2k_2<P\!·\!P_2 and p2+D≤p1p_2+D≤ p_1 then 9 emit L2⟨e(p2),L_2 e(p_2), m=p2−Oe(p2),m=p_2-O_e(p_2), n=2(k2modP2)+r⟩n=2(k_2 P_2)+r 10 k2←k2+Gk_2← k_2+G 11 12 else 13 emit L1⟨e(p1),L_1 e(p_1), m=p1−Oe(p1),m=p_1-O_e(p_1), n=2(k1modP1)+r⟩n=2(k_1 P_1)+r 14 k1←k1+Gk_1← k_1+G 15 16 17 [2pt]g: instance id, stride G; r∈0,1r∈\0,1\: role in pair; P: rows/stream; PiP_i: pairs/row; kik_i: pair cursor; pi=⌊ki/Pi⌋p_i= k_i/P_i : row; e(p)e(p): expert of row p; OeO_e: its first row; D: minimum row lead of L1L_1 over L2L_2. Algorithm 1 Interleaved Scheduler [2pt]g: instance id, stride G; r∈0,1r∈\0,1\: role in pair; P: rows/stream; PiP_i: pairs/row; kik_i: pair cursor; pi=⌊ki/Pi⌋p_i= k_i/P_i : row; e(p)e(p): expert of row p; OeO_e: its first row; D: minimum row lead of L1L_1 over L2L_2. Trading issue stall for a useful gap. In the drain-dominated regime, Equations 1 and 2 give (11) Tiss+=Tperiod=ℛ.T_iss+G=T_period= VR. For fixed communication volume and drain rate, scheduling cannot shorten this period. It can, however, determine how much of the period is useful computation and how much is sender-visible issue stall. Under the expert-wave schedule, Tiss=ℛ−TL2mma>Tiss0.T_iss= VR-T_L2^mma>T_iss^0. Increasing the useful gap by one unit reduces issue stall by one unit until TissT_iss reaches Tiss0T_iss^0. This exchange relies on post-issue progress. Under a completion-coupled model, Tperiodcc=+/ℛT_period^c=G+V/R, so moving computation between bursts would only delay the next burst rather than reduce stall. Algorithm 1 merges the ready Linear-1 and Linear-2 tile streams. It greedily emits Linear-2 only when the Linear-1 stream leads by at least D scheduler rows; otherwise, it emits a ready Linear-1 tile, which adds useful work without generating Combine stores. In steady state, the pattern contains n2n_2 Linear-2 tiles and n1n_1 Linear-1 tiles per cycle. Their ratio is fixed by tile shapes. The lead D changes the phase and readiness of the streams, not their long-run ratio, which separates two conditions. Condition 1: Is interleaving sufficient? Averaged over a steady-state cycle, the natural gaps are (12) wave≈TL2mma,int≈TL2mma+n1n2TL1mma.G_wave≈ T_L2^mma, _int≈ T_L2^mma+ n_1n_2T_L1^mma. Interleaving adds neither computation nor communication. It redistributes Linear-1 work that would otherwise be grouped at a wave boundary. Substituting Equation 12 into Equation 9 gives (13) ΔTissint=[ΔTisswave−n1n2TL1mma]+. T_iss^int= [ T_iss^wave- n_1n_2T_L1^mma ]_+. Thus, useful computation offsets issue overhead one-for-one until the overhead reaches zero. Complete recovery requires int≥∗G_int ^*, equivalently (14) n2n2(Tiss0+TL2mma)+n1TL1mma≤ℛ. n_2Vn_2\! (T_iss^0+T_L2^mma )+n_1T_L1^mma . This test depends on the aggregate burst volume, tile mix and mainloop times, and calibrated (Tiss0,ℛ)(T_iss^0,R), but not on D. It can therefore predict before implementation whether reordering existing computation is sufficient. If Equation 14 fails, the scheduler must also reduce aggregate burst volume. For the evaluated configurations, the equation predicts that interleaving should move each tile’s sender-visible remote-store span from the drain-limited regime to the backpressure-free floor. We test this prediction in Section 5.2.3. From epilogue stall to the compute critical path. The Burst–Gap model predicts the epilogue’s sender-visible span from the available post-issue drain gap: TRS≈Tiss0+ΔTissT_RS≈ T_iss^0+ T_iss plus fixed staging overhead. Epilogue stall is not necessarily compute stall. Double-buffered staging allows the MMA warpgroup to advance through the next mainloop before it must reuse the occupied slot. Let TcovermmaT_cover^mma denote this configuration-specific cover window provided by local staging. It is the following Linear-2 mainloop in the wave schedule and, when an inserted Linear-1 immediately follows, the Linear-1 mainloop in the interleaved schedule. Mixed cycles use the actual successor for each sampled tile. The critical-path stall of a tile is (15) Δt=[TRS−Tcovermma]+. t= [T_RS-T_cover^mma ]_+. Interleaving can therefore help at two independent levels. It enlarges the drain gap and reduces TRST_RS; it can also replace the immediate mainloop cover window with a longer Linear-1 mainloop. Conversely, even if Equation 14 does not drive ΔTissint T_iss^int to zero, residual epilogue stall remains off the Tensor Core critical path whenever TRS≤TcovermmaT_RS≤ T_cover^mma. Section 5.2.3 tests Equation 15 using instrumentation on both sides of the staging boundary. Condition 2: What lead prevents starvation? The steady-state gap is realized only if Linear-2 does not spin while waiting for its Linear-1 dependencies. A Linear-2 tile consumes Linear-1 outputs produced across multiple SMs and synchronized through the l2_full counter. If D is too small, the Linear-2 stream catches up, spins on an unready dependency, and allows bursts to reconcentrate. The implementation maps N-tiles in paired CTAs and advances strided streams across SMK_SM participating SMs. The resulting cross-SM phase skew gives the approximate anti-starvation knee (16) Dknee=⌈bnSM2hinter⌉,D_knee= b_n\,K_SM2h_inter , where bnb_n is the N-tile width and hinterh_inter is the expert intermediate dimension. Here D is a lead in scheduler rows; the row layout determines its corresponding token range. This lower bound comes from cross-SM data readiness, not X-Stage draining, and therefore does not contain ℛR, Tiss0T_iss^0, or Q. Choosing D<DkneeD<D_knee lets Linear-2 catch up and spin, allowing remote-store bursts to reconcentrate; increasing D toward the knee removes this anti-starvation bottleneck. Increasing D far beyond the knee does not change the long-run Linear-1/Linear-2 ratio or the X-Stage rate condition, but it grows the Linear-2-only tail and concurrent expert-weight working set, potentially eroding cache locality. Thus DkneeD_knee is sufficient for the model; any additional margin is implementation-specific tuning. Algorithm 1 preserves all within-expert dependencies and reorders only independent tiles. A future-wave Linear-1 tile is eligible only after its Dispatch input and destination ring slot are ready. The finite software pipeline depth separately limits in-flight epilogues and must remain compatible with the calibrated Q. 4.3. FlashAttention–A2A: Piggybacking without a Dedicated Communication Role Figure 5. Ulysses sequence-parallel attention. QKV projection and the pre-attention All-to-All transform sequence-partitioned tokens into head partitions (a–c). FlashAttention and the post-attention All-to-All produce the output and restore the original sequence partition (d–e); our FA+A2A design fuses these latter two operations. Data-layout transformations in Ulysses sequence parallelism: QKV projection and an All-to-All precede FlashAttention, followed by a fused FlashAttention and All-to-All that restores the sequence partition. Figure 6. FlashAttention Q-loop and output tiling. Each CTA owns a block of query rows and traverses the K/V tiles in the inner loop. Completing the traversal produces an output tile; successive output tiles form the outer loop. FlashAttention loop structure showing query rows partitioned across CTAs, traversal over key and value tiles, and the resulting output tiles. We next consider Ulysses sequence parallelism, illustrated in Figure 5, a common strategy for diffusion transformer (DiT) workloads. The evaluated DiT setting uses full, non-causal attention, rather than the causal attention used by typical autoregressive LLMs (Peebles and Xie, 2023). Ulysses executes an All-to-All before attention to convert a sequence partition into a head partition, and another All-to-All (A2A) afterward to restore the original layout (Jacobs et al., 2023). We fuse FlashAttention with the post-attention All-to-All at tile granularity: when an output tile becomes ready, the owning role issues the tile’s remote stores at the boundary and then proceeds to the next tile. Figure 6 illustrates the Q-loop that separates consecutive output-tile boundaries. We first audit the design using Equation 10. A FlashAttention output tile has shape bm×db_m× d, where d is the head dimension. Each persistent CTA writes ℬt=bmdsoutB_t=b_mds_out bytes at a tile boundary, where souts_out is the output element size. In the conservative case where K producers align their boundaries, the aggregate burst is =ℬtV=KB_t; actual persistent CTAs are generally phase-shifted and inject more smoothly. Unlike MegaMoE, consecutive output bursts are separated by the next output tile’s complete Q-loop, which visits all ⌈M/bn⌉ M/b_n KV tiles for sequence length M (Shah et al., 2024). The resulting gap Qloop(M)G_Qloop(M) grows approximately linearly with M. For the evaluated FlashAttention-4 configuration, bm=128b_m=128, d=128d=128, the output is bf16, and ≤148K≤ 148. Each producer writes 32 KB, below the approximately 33 KB per-producer effective-capacity knee measured in Section 3.5. In the worst aligned case, ≈4.6V≈ 4.6 MiB and [−ℛTiss0]+≈4.1MiB<≈4.25MiB. [V-RT_iss^0 ]_+≈ 4.1~MiB<Q≈ 4.25~MiB. The capacity bound therefore holds. The rate bound holds when Qloop(M)≥∗G_Qloop(M) ^*. Section 4.3 derives a conservative sequence-length upper bound Mub∗M_ub^*; all M≥Mub∗M≥ M_ub^* satisfy both bounds of the piggybacking design test. Figure 7. Piggybacked FlashAttention–A2A pipeline for FlashAttention-4. A single warpgroup per CTA issues QKTQK^T and PVPV while specialized roles execute operations such as softmax. At an output-tile boundary, the epilogue issues remote stores (RS) and immediately starts the next tile. The previous tile drains through X-Stage during the next complete Q-loop, without a dedicated communication warp or SM. FlashAttention-4 pipeline in which each output tile issues All-to-All remote stores and the following Q-loop overlaps their X-Stage drain. Post-issue progress extends the drain window. A completion-coupled counterfactual makes the role of X-Stage explicit. If the issuing role had to remain blocked until remote visibility, the exposed time of each aligned output burst would be approximated by (17) Δtcc=[ℛ+Tlat−Tcover]+,Tcover≲TQKmma, t^c= [ VR+T_lat-T_cover ]_+, T_cover T_QK^mma, where TlatT_lat is a fixed completion-latency component and TcoverT_cover is the progress available from finite local staging while the issuer is blocked. In FlashAttention-4, one warpgroup drives the CTA’s MMA pipeline, so this local cover window is at most on the order of one QKTQK^T tile mainloop. Its duration and compute volume 2bmbnd2b_mb_nd do not grow with M. If Δtcc>0 t^c>0, the total completion-coupled exposure over Nt∝MN_t M output-burst waves grows as Ecc=NtΔtcc∝M.E^c=N_t t^c M. With X-Stage, the issuing role returns after the sender accepts the stores. The next full Q-loop, rather than one local MMA step, becomes the post-issue drain window. If that window is insufficient, Equation 4 gives the issue-side backpressure (18) Δt(M)=[ℛ−Qloop(M)−Tiss0]+. t(M)= [ VR-G_Qloop(M)-T_iss^0 ]_+. As in MegaMoE, this epilogue-side stall reaches the compute critical path only after it exceeds the additional double-buffer cover window QKG_QK: (19) Δt′(M)=[Δt(M)−QK]+. t (M)= [ t(M)-G_QK ]_+. Because Qloop(M)G_Qloop(M) grows with M while the aligned burst volume is fixed, Δt t decreases and eventually reaches zero. The weaker critical-path condition Δt≤QK t _QK may be reached even earlier. The contrast between Equations 17 and 18 follows from one property: whether accepted stores can leave the issuing role and progress through X-Stage. This distinction also guides role assignment. Reserving communication warps or SMs can decouple an MMA role from address generation and issue (Jin et al., 2026), but consumes registers, shared memory, occupancy, or Tensor Core capacity. In the audited region, waiting after issue does not improve the calibrated downstream drain rate. We therefore attach the short issue operation to the role that already owns the output tile. As Figure 7 shows, tile i drains while tile i+1i+1 executes its full Q-loop. Communication tiles retain the compute-tile shape, so the design minimally changes the FlashAttention data path. Unlike the MegaMoE reshape action, piggybacking does not reorder computation; it removes a long-lived communication role and uses the workload’s natural drain window. A conservative convergence bound Mub∗M_ub^*. For a GEMM tile of shape bm×bnb_m× b_n and reduction dimension k, the compute work is CGEMM=2bmbnk.C_GEMM=2b_mb_nk. A full-attention output tile performs both QKTQK^T and PVPV over all KV tiles, so CFA=4bmbnd⌈Mbn⌉≈4bmdM.C_FA=4b_mb_nd Mb_n ≈ 4b_mdM. Equating this work with the GEMM expression gives the equivalent reduction dimension kequiv=2d⌈Mbn⌉∝M.k_equiv=2d Mb_n M. The factor of two accounts for the two matrix multiplications in attention. The measured Q-loop gap has a positive fixed component and an approximately linear steady-state slope. For a conservative threshold, we drop the positive fixed component and use the lower bound Qloop(M)≥σ⌈Mbn⌉,σ≈0.79μs per KV tile.G_Qloop(M)≥σ Mb_n , σ≈ 0.79~ per KV tile. The model-predicted drain-induced residual over all burst waves is Edrain(M)=Nt[Δt(M)−QK]+.E_drain(M)=N_t [ t(M)-G_QK ]_+. This quantity excludes the finite cost of executing backpressure-free issue instructions, which is included in the measured residual Eres=Tfused−TFAE_res=T_fused-T_FA in Section 5.3. Dropping the positive fixed gap yields the tile-aligned sufficient upper bound (20) Mub∗=bn⌈∗σ⌉,∗=[ℛ−Tiss0]+.M_ub^*=b_n G^*σ , ^*= [ VR-T_iss^0 ]_+. Using the once-calibrated platform parameters and the worst aligned case =148K=148 gives ∗≈6.0μG^*≈ 6.0~ and Mub∗≈1.0M_ub^*≈ 1.0K tokens. For M≥Mub∗M≥ M_ub^*, the drain-induced component EdrainE_drain vanishes; the measured EresE_res may retain a small issue, launch, or measurement floor. The true threshold is no larger than this bound because the omitted fixed gap is positive. It can be smaller still because the compute critical path needs only Δt≤QK t _QK and shorter sequences may use fewer than 148 concurrent producers. At the smallest evaluated sequence, M=8,192M=8,192, the measured Q-loop gap is about 80μ80~ , roughly 13×13× the worst-case ∗G^*. The evaluated range therefore separates two prior predictions. X-Stage predicts no drain-induced growth with sequence length, EresXS(M)≈Eissue(M),E_res^XS(M)≈ E_issue(M), where EissueE_issue is a small backpressure-free issue cost. The completion-coupled counterfactual predicts an additional component Ecc(M)=NtΔtcc∝M.E^c(M)=N_t t^c M. Section 5.3 tests the difference in slope rather than relying on a single short-sequence point. 4.4. Summary MegaMoE and FlashAttention–A2A instantiate the same design test in different regions. MegaMoE violates the rate bound because concentrated Linear-2 bursts are separated by a short mainloop. Cross-wave Linear-1/Linear-2 interleaving reshapes injection; the dependency-derived DkneeD_knee supplies an anti-starvation threshold, while unnecessarily larger leads can erode cache locality. FlashAttention–A2A satisfies the capacity bound, and its Q-loop gap exceeds the recovery gap for M≥Mub∗M≥ M_ub^*. The output-owning role can therefore piggyback issue on a tile boundary and resume computation while X-Stage drains the request. Both designs preserve communication volume; they differ in whether the workload requires injection reshaping or already supplies a sufficient drain window. 5. Evaluation The evaluation asks two questions. First, do the two model-selected actions improve end-to-end performance: reshaping in MegaMoE (Section 5.2) and piggybacking in FlashAttention–A2A (Section 5.3)? Second, do mechanism-level measurements follow the Burst–Gap model quantitatively (Sections 5.2.2 and 5.2.3)? We use three principles to support attribution. Each comparison holds the compute kernel, input, process mapping, and all paths except the target schedule or communication mechanism constant. Platform parameters (Tiss0,ℛ,)(T_iss^0,R,Q) are calibrated once by the microbenchmarks in Section 3 and are not refitted to applications. Finally, end-to-end latency, execution timelines, and per-tile instrumentation cross-check the same mechanism at different scales. 5.1. Experimental Setup Experiments run on one eight-GPU system with a recent NVIDIA architecture. Each GPU has 148 SMs; NVLink/NVSwitch fully connects the GPUs. Experiments use eight-way expert parallelism (EP=8). The software stack is CUDA 13.1, PyTorch 2.9, DeepGEMM at commit 7f2a703, and FlashAttention at commit 77aacb6. The characterization in Section 3 uses unidirectional peer-to-peer remote stores. MegaMoE and FlashAttention–A2A use symmetric memory and device-initiated remote stores. Unless noted otherwise, we warm up each configuration, repeat the measurement, and report the median. All paired comparisons use identical inputs, process mappings, and kernel configurations. CUDA events measure kernel latency. Nsight Systems GPU metrics provide Tensor Core timelines, and in-kernel clock64() instrumentation measures sender-visible remote-store spans. The application experiments hold the independently calibrated Tiss0T_iss^0, ℛR, and Q fixed. For each MegaMoE cell, one untimed invocation performs JIT compilation and warm-up. We then collect 20 barrier-aligned CUDA-event trials per rank. For each trial, we take the maximum latency across ranks and report the median of these per-trial maxima. We rotate scheduler order across cells; two anomalous cells are additionally repeated in three order-rotated paired runs. FlashAttention–A2A timings use a sender-side steady-state boundary. A CUDA event completes after the kernel has consumed its source staging and issued the remote stores; a barrier synchronizes remote visibility once after the back-to-back replay rather than per iteration, matching deployments where the next dependent operator synchronizes before consuming the fused operator’s output. Back-to-back replay exposes sustained sender-side backpressure. 5.2. MegaMoE Evaluation We first report kernel-level performance across the full configuration matrix, then use Tensor Core timelines and per-tile instrumentation to test the backpressure mechanism. 5.2.1. Kernel Performance We compare the Expert-Wave scheduler with the interleaved scheduler proposed in Section 4.2 on seven model shapes, W4A8 and W8A8 precision, balanced and skewed routing, and multiple sequence-length and expert configurations, for 84 configurations in total. Both implementations share the same compute kernels, warp specialization, symmetric buffers, epilogues, and synchronization paths, and change only the scheduling order of ready Linear-1 and Linear-2 tiles. We report Speedup=TwaveTinterleaved.Speedup= T_waveT_interleaved. Across all configurations, the interleaved scheduler achieves a 1.18×1.18× geometric-mean speedup, a 1.17×1.17× median speedup, and a 1.62×1.62× maximum speedup. Figure 8 breaks down the results by model, precision, and routing mode. Figure 8. MegaMoE kernel speedup. Speedup of the X-Stage-aware interleaved scheduler over the expert-wave scheduler, across models, precisions, and routing configurations. Grouped bars show interleaved-scheduler speedups over the expert-wave baseline for each model, weight precision, and balanced or skewed routing. Skewed routing benefits more than balanced routing. Concentrating tokens on hot experts creates longer runs of Linear-2 epilogues under Expert Wave, so ready Linear-1 work supplies a more valuable drain window. Balanced routing produces shorter bursts and more natural separation, leaving less sender-visible stall to remove. A few configurations regress. For example, some balanced configurations with large expert-weight working sets drop to about 0.94×0.94×. Profiling shows that cross-expert interleaving can reduce the L2 locality of expert weights, offsetting the communication-side gain. X-Stage-aware scheduling therefore must also balance communication draining against data locality. 5.2.2. Tensor Core Timeline Kernel speedup alone does not attribute the gain to X-Stage. The mechanism experiments therefore examine the interleaved scheduler across all evaluated shapes to isolate the transformation. We proceed from a global symptom to a quantitative mechanism. Tensor Core timelines show whether concentrated remote-store injection creates low-activity intervals before the per-tile analysis tests the critical-path stall equation derived before implementation (Section 5.2.3). If X-Stage backpressure causes the gain, concentrated bursts under the wave schedule should periodically suppress Tensor Core activity, while interleaving should shorten those intervals without changing communication volume. Figure 9 confirms this pattern for representative balanced and skewed configurations. The expert-wave scheduler shows phased execution: after consecutive Linear-2 epilogues inject remote stores, Tensor Core activity drops or develops a long tail. Interleaving distributes shorter bursts among Linear-1 and Linear-2 work, raising average Tensor-pipeline activity and shortening low-utilization intervals. The timeline establishes correlation with injection reshaping but does not locate the stall or test its magnitude; per-tile measurements do so next. Figure 9. MegaMoE Tensor Core timeline (Tensor-pipe active, percentage of peak). The interleaved scheduler distributes communication bursts among Linear-1 and Linear-2 work, reducing backpressure-associated intervals of low Tensor Core activity. Tensor-pipe activity timelines comparing expert-wave and interleaved MegaMoE schedules. 5.2.3. Per-Tile Remote-Store Span and Critical-Path Stall We instrument two quantities for every Linear-2 tile. The epilogue records the sender-visible remote-store span TRST_RS, including instruction issue and any waiting induced by X-Stage backpressure. Independently, the MMA warpgroup records time spent at tile entry waiting for a staging slot (tmem_empty). The first observes the source of backpressure; the second observes what reaches the compute critical path. Span distribution. If the wave schedule supplies an insufficient gap, the Burst–Gap model predicts a drain-limited aggregate span near actt/ℛK_actV_t/R, where actK_act is the number of producers participating in the aligned burst. Interleaving should move the span toward a backpressure-free floor consisting of Tiss0T_iss^0 plus fixed staging overhead. Figure 10 exhibits both regimes. For skewed W8A8 routing, the seven expert-wave medians range from 7.97.9 to 9.9μ9.9~ and have pronounced tails. After interleaving, all seven narrow to 3.63.6–3.9μ3.9~ . For three representative shapes, we additionally redirect the stores to local HBM: the interleaved remote-store medians are within 0.5μ0.5~ of these local-store controls. The other four shapes fall in the same narrow range but do not have an independent local-store control. The shift is therefore directly tied to the backpressure-free floor for three shapes and is consistent with that floor for all seven. The robust application-level result is that changing only the schedule moves all seven span distributions from a broad, model-sensitive regime to a narrow floor-like regime. Figure 10. Sender-visible remote-store span for one MegaMoE Linear-2 tile. Consecutive Linear-2 work under the expert-wave scheduler produces longer, heavy-tailed spans. Interleaving shifts the distribution toward the backpressure-free range, where Linear-1 computation can cover the stores. Distributions of sender-visible Linear-2 remote-store spans before and after interleaving. Mapping span to compute stall. The reduction in span is not itself overall kernel speedup. By Equation 15, only the portion beyond the immediately available mainloop cover window reaches the MMA critical path. For each sampled tile, we substitute its measured epilogue span and the full-tile mainloop time of its successor into Equation 15. This yields a prediction for Δt t, which we compare against an independent MMA-side stall measurement. Table 2 reports the comparison. For six skewed configurations with visible wave-schedule backpressure, the equation predicts 1.51.5–4.9μ4.9~ per tile and the measured medians are 0.650.65–4.3μ4.3~ . The approximation consistently overpredicts these medians by 0.450.45–0.82μ0.82~ , but preserves their ordering and scale. For the interleaved schedule, it predicts zero and measurements are at most 0.05μ0.05~ . DSv4-Pro provides a negative control: its 9.54μ9.54~ Linear-2 mainloop already covers the 7.88μ7.88~ wave span, so both the equation and instrumentation indicate negligible stall, and its kernel-level gain is correspondingly small. These observations support the proposed path from X-Stage backpressure, through double-buffered staging, to the Tensor Core critical path while also exposing the approximation error of the one-window model. Table 2. Test of the per-tile stall model (W8A8, skewed routing, 55,808 tokens, 8 GPUs; times in µs s). Span entries are wave/interleaved medians; TmmaT^mma entries are full-tile L2/L1 mainloop medians. Each Δt t entry is prediction/measurement. Span TmmaT^mma Δt t (pred. / meas.) Model Wave / int. L2 / L1 Wave Int. DiT-MoE 9.93 / 3.77 5.30 / 9.77 4.63 / 4.18 0 / 0.04 Qwen3.5 9.17 / 3.82 4.24 / 14.40 4.94 / 4.31 0 / 0.04 Hy3 9.68 / 3.59 6.09 / 10.79 3.60 / 3.08 0 / 0.05 MiMo-V2.5 9.22 / 3.94 7.77 / 7.57 1.45 / 0.71 0 / 0.05 GLM-5.2 9.53 / 3.63 7.99 / 18.77 1.54 / 0.92 0 / 0.04 DSv4-Flash 9.48 / 3.83 8.01 / 7.40 1.47 / 0.65 0 / 0.05 DSv4-Pro 7.88 / 3.55 9.54 / 11.57 0 / 0.06 0 / 0.04 5.3. FlashAttention–A2A End-to-End Performance We fuse the Ulysses All-to-All path with FlashAttention-3 (FA3) and FlashAttention-4 (FA4) and sweep sequence length M. The compared implementations are: • FA-only: the same FlashAttention compute path used by the corresponding fused implementation; • A2A-only: the same post-attention All-to-All communication volume executed independently; • Serial: FlashAttention followed by the same post-attention All-to-All, without tile-level fusion; and • X-Stage-Fused: existing compute or epilogue roles issue remote stores at tile boundaries, with no dedicated communication warp or SM. In addition to sender-visible speedup over Serial, we report the measured sender-side residual beyond FlashAttention alone, Eres=Tfused−TFA.E_res=T_fused-T_FA. A residual near zero means that issue and backpressure overhead are nearly absent from the measured sender-side path; it does not assert remote-visible completion. We also report the sender-visible hiding ratio (21) H=1−Tfused−TFATserial−TFA.H=1- T_fused-T_FAT_serial-T_FA. The denominator uses the measured incremental cost in the serial composition, rather than an independently timed A2A kernel, because cache state, synchronization, and launch overhead can differ across compositions. Table 3. Sender-visible X-Stage-aware FlashAttention–A2A performance. Times other than hiding ratio and speedup are in µs s. Method / M 8,192 16,384 32,768 49,152 65,536 FA3 + A2A FA3 only 195.2 785.5 3,236.7 7,275.6 13,249.4 Serial 295.6 968.3 3,579.6 7,767.9 13,936.6 Fused 207.0 795.5 3,241.5 7,285.7 13,218.3 Hiding ratio 88% 95% 99% 98% ∼ 100%∗ Speedup 1.428× 1.217× 1.104× 1.066× 1.054× FA4 + A2A† FA4 only 85.4 346.5 1,439.4 3,281.4 5,340.7 Serial 140.9 462.4 1,660.9 3,661.7 5,813.0 Fused 99.4 356.6 1,429.2 3,282.1 5,348.9 EresE_res +14.0+14.0 +10.1+10.1 −10.2-10.2 +0.7+0.7 +8.2+8.2 Hiding ratio 75% 91% ∼ 100%∗ 100% 98% Speedup 1.417× 1.297× 1.162× 1.116× 1.087× ∗Values at or slightly above 100% reflect measurement noise once A2A issue overhead is hidden within the sender-side timing boundary. †FA4 uses steady-state CUDA Graph replay; the M=65,536M=65,536 column is the clock-stabilized rerun described in the text. FA3 uses per-iteration steady-state timing. Metrics are computed from unrounded medians. Measurement methodology. Three effects complicate the FlashAttention measurements. First, the CuteDSL Python launch overhead is comparable to short-sequence FA4 kernel time and would impose a host-side timing floor. We remove it from the timing window by replaying each FA4 variant from a CUDA Graph and measuring device execution with events; profiler-reported device time provides a cross-check. FA3 launch overhead is small relative to its kernel time, so steady-state event timing is sufficient. Second, the hiding ratio has a small denominator at short sequences, while EresE_res at long sequences subtracts similar millisecond-scale measurements; both amplify noise and can produce small negative residuals. We reduce this noise through matched-path comparisons, medians, rotated execution order, and repeated runs. Third, thermal dynamic voltage and frequency scaling (DVFS) (Mei et al., 2017) can shift SM clocks across experiment blocks. We monitor clocks and repeat affected points in short, clock-stabilized runs. The resulting residuals are consistent with a small nonnegative issue overhead; we treat slightly negative values as noise rather than evidence that communication accelerates attention. End-to-end speedup. Table 3 shows the expected trend. At shorter evaluated sequences, All-to-All occupies a larger fraction of serial time, and FA4 and FA3 reach maximum speedups of 1.42×1.42× and 1.43×1.43×, respectively, at M=8,192M=8,192. At longer sequences, speedup falls to approximately 1.051.05–1.09×1.09×. This does not indicate worse sender-side hiding. If the measured A2A issue path is fully covered, Tfused≈TFA,T_fused≈ T_FA, and the ideal speedup is Smax≈TFA+TA2ATFA=1+TA2ATFA.S_ ≈ T_FA+T_A2AT_FA=1+ T_A2AT_FA. As full-attention compute grows faster than the measured A2A contribution, this upper bound naturally approaches one. Speedup measures the sender-visible A2A overhead available to cover; EresE_res and H measure how much of that overhead remains within the stated boundary. Residual-exposure test. The sequence-length trend tests the opposing predictions from Section 4.3. The worst-case threshold Mub∗(=148)≈1.0M_ub^*(K=148)≈ 1.0K is below all measured points. The X-Stage model therefore predicts no drain-induced growth over the measured range: the residual should remain near a small issue and measurement floor. The completion-coupled counterfactual predicts an additional component that grows from approximately 16μ16~ at M=8,192M=8,192 to approximately 120μ120~ at M=65,536M=65,536. Figure 11 overlays these prior predictions with FA4 measurements. At M≤16,384M≤ 16,384, EresE_res is a small positive 1010–14μ14~ and does not grow. The three longer-sequence estimates (−10.2-10.2, +0.7+0.7, and +8.2μ+8.2~ ) are within 0.7% of the millisecond-scale kernel times and are consistent with zero after accounting for the DVFS noise band. Direct profiling attributes the remaining 33–12μ12~ primarily to communication issue inside the kernel; it grows only slowly with the number of burst waves and remains on the scale expected for backpressure-free issue. The completion-coupled prediction at M=65,536M=65,536 is nearly an order of magnitude above the largest positive measured residual and, more importantly, has the opposite slope. The counterfactual refers to the same persistent fused kernel with post-issue progress disabled, not to the Serial baseline; it retains tile fusion and avoids separate-kernel launch and synchronization. The comparison tests whether the issuing role decouples from post-issue progress, not when a downstream consumer can safely read the final output. Its 16μ16~ short-sequence prediction overlaps the fixed issue floor, so the first point alone cannot distinguish the models. The sequence-length slope does. FA3 follows the same pattern: its residual estimates range from −31.1-31.1 to +10.1μ+10.1~ , and their absolute magnitude is below 0.25% of FA-only time at M≥32,768M≥ 32,768. Figure 11. Relative sender-visible residual Eres(M)/TFAE_res(M)/T_FA for FA4+A2A. Prior predictions use the once-calibrated platform parameters and measured compute-side timing, without application refitting. The dotted reference represents an approximately 16μ16~ issue floor, and the blue band is a ±2%± 2\% DVFS envelope. The worst-case convergence threshold Mub∗(=148)≈1.0M_ub^*(K=148)≈ 1.0K lies below all measured points. Measured relative residual exposure across sequence lengths compared with X-Stage and completion-coupled predictions. 6. Discussion and Limitations 6.1. X-Stage as a Calibrated Execution Abstraction X-Stage abstracts the aggregate post-issue behavior observed between remote-store issue and remote-visible completion. This behavior may reflect the combined effects of the store pipeline, cache hierarchy, fabric injection, flow-control credits, and receiver-side resources. Rather than requiring software to identify or model each component separately, X-Stage summarizes their externally visible effects as ℳX=(Tiss0,ℛ,).M_X= (T_iss^0,R,Q ). Different systems may produce different parameter values. The same calibration and scheduling method applies only while their external behavior is adequately described by the issue–drain–capacity model. A change in topology, routing, store width, producer count, memory placement, or GPU generation may require recalibration and may expose behavior outside the current fluid approximation. The drain horizon /ℛV/R predicts when the sender can return to low-cost issue. It does not replace remote completion or memory-ordering semantics. X-Stage-aware kernels retain the required fences, signals, buffer lifetime rules, and consumer-side readiness checks. 6.2. Two Levels of Pipeline Decoupling A fused GEMM epilogue often writes computed results to remote memory. Local buffering and warp specialization decouple the Tensor Core warpgroup from the epilogue or issuing role. This software layer can absorb a short mismatch, but finite slots eventually fill if issue remains backpressured. X-Stage exposes a second level after accepted issue. While effective outstanding capacity remains available, requests can advance toward remote visibility and the issuing role can resume computation or epilogue work. This decoupling is finite, not unbounded. Sustained injection above the effective drain rate accumulates requests, lengthens later issue operations, and propagates backpressure through local staging. Adding software buffers can delay this propagation but cannot remove the rate or capacity constraint. 6.3. Adaptive Burst Shaping and Scope MegaMoE interleaving is a form of burst shaping: it preserves total work and communication volume but redistributes remote stores in time. More generally, a scheduler can use Tiss0+≤ℛ,[−ℛTiss0]+≤ VT_iss^0+G , [V-RT_iss^0 ]_+ to jointly choose tile size, useful gap, software staging depth, and the number of issuing resources. Reordering across tiles or experts can also change cache locality when too much cross-wave work is kept in flight. A practical generator or runtime should compare predicted X-Stage stall reduction with locality cost and select the original or interleaved schedule for the current routing and working set. The evidence presented here has two limitations. First, the microbenchmarks study one-sided remote stores; loads, atomics, collectives with different progress engines, and cross-node networks may exhibit different constraints. Second, the model predicts sender-side backpressure but does not by itself account for synchronization, receiver congestion, or all launch and scheduling overheads. These limitations are why we treat X-Stage as a calibrated execution abstraction rather than a universal hardware description. 7. Related Work 7.1. Device-Initiated One-Sided Communication Multi-GPU systems differ in how communication is initiated and which resources advance it. Conventional collectives are submitted by the host as separate communication kernels on CUDA streams and overlap with computation at kernel or operator granularity (NVIDIA, 2026b; Unat et al., 2026). Peer memory mapping, symmetric memory, and device-initiated communication let a GPU kernel issue one-sided remote operations directly, embedding fine-grained data movement in a long-running compute kernel (NVIDIA, 2026d; Ma et al., 2026). Communication may execute in independent kernels or on dedicated SMs, or it may share CTAs, warps, and SMs with computation. Hybrid systems select between these organizations according to the operation and workload (Sul et al., 2026; Unat et al., 2026). Dedicated execution can provide more predictable progress but consumes compute resources; shared execution avoids a separate launch and long-lived reservation but requires finer coordination. These choices determine who issues remote operations. X-Stage is orthogonal: it models when finite post-issue resources backpressure that issuer and, through local staging, upstream computation. 7.2. Fine-Grained Communication–Computation Fusion Prior work constructs fine-grained overlap through task decomposition, kernel generation, and resource scheduling. CoCoNet represents communication and computation as first-class program structures and applies fusion and overlap transformations. FLUX over-decomposes both kinds of work into one fused kernel. Comet uses MoE dependencies and task reordering to coordinate communication and expert computation (Jangda et al., 2022; Chang et al., 2024; Zhang et al., 2025). GC3 supplies a programmable collective DSL and optimizing compiler. TileLink uses tile-centric primitives to connect communication with computation and generate fused implementations (Cowan et al., 2023; Zheng et al., 2025). MegaScale-MoE combines device-side signals, communication-resource tuning, and swizzling for tile-level overlap. ParallelKittens supports intra- and inter-SM scheduling and selects shared or dedicated resources according to communication structure (Jin et al., 2026; Sul et al., 2026). These systems expose task decomposition, issue roles, dependencies, and remote tile readiness. Their public abstractions generally do not quantify the finite sender-visible state after remote-store issue or predict when it will backpressure computation. X-Stage adds this issue–drain–capacity boundary and uses it to shape burst volume and compute gaps. 7.3. Analytical Performance Models Roofline relates attainable throughput to arithmetic intensity and compute and memory-bandwidth ceilings (Williams et al., 2009). LogP and LogGP describe distributed communication using latency, overhead, per-message gap, and a long-message extension (Culler et al., 1993; Alexandrov et al., 1995). Queueing identities such as Little’s law relate average in-flight occupancy, throughput, and residence time (Little, 1961). These models capture aggregate bottlenecks or steady-state costs. The Burst–Gap model instead targets sender-visible remote-store issue, finite post-issue capacity, and recovery between repeated bursts inside a fused GPU kernel. 7.4. MoE Communication and Persistent Mega-Kernels Expert-parallel MoE layers use Dispatch before expert computation and Combine afterward. Communication varies with expert count, routing skew, and parallel scale. Existing systems use grouped GEMM, expert batching, resource partitioning, and communication–computation overlap to reduce this cost (Zhang et al., 2025; Jin et al., 2026). Sparse models such as Mixtral and DeepSeek-V3 activate only a subset of experts for each token (Jiang et al., 2024; DeepSeek-AI, 2024). MegaBlocks maps sparse expert computation to block-sparse operations, while FasterMoE models and schedules dynamic expert workloads (Gale et al., 2023; He et al., 2022). Routing skew motivates complementary balancing mechanisms: auxiliary-loss-free balancing shapes router decisions, EPLB adjusts expert placement, and Metro and ReaLB target serving-time expert imbalance (Wang et al., 2024; DeepSeek-AI, 2025; Yu et al., 2025; Wang et al., 2026). These methods change routing, placement, or sparse computation; X-Stage instead reshapes when an unchanged volume of Combine traffic enters the communication path. Lancet and FSMoE search the wider training graph for computation that can overlap All-to-All and construct pipelines from profiles and dependencies. FLUX, Comet, and MegaScale-MoE use decomposition, fine-grained scheduling, and resource partitioning for tile-level overlap. Cui et al. use tile-level signals to overlap expert computation and the second All-to-All on separate SM partitions. DeepGEMM MegaMoE and UniEP further fuse Dispatch, expert computation, and Combine into persistent mega-kernels (Hong et al., 2026; Pan et al., 2025; Chang et al., 2024; Zhang et al., 2025; Jin et al., 2026; DeepSeek-AI, 2026a; Zheng et al., 2026; Cui et al., 2026). DiT-MoE, EC-DIT, and Race-DiT study sparse expert architectures, expert-choice routing, and joint token–expert competition, respectively, showing that dynamic token-to-expert routing is an important scaling dimension for DiTs (Fei et al., 2024; Sun et al., 2025; Yuan et al., 2025). X-Stage quantifies the finite decoupling between remote-store issue and post-issue draining. We use that model to interleave Linear-1 and Linear-2 work across expert waves, changing the temporal distribution of Combine stores while preserving dependencies and communication volume. 7.5. Sequence Parallelism and FlashAttention Distributed long-sequence attention commonly uses All-to-All or ring-based communication. DeepSpeed-Ulysses uses All-to-All to convert between sequence and head partitions. Ring Attention circulates K/V blocks and overlaps transfer with blockwise attention. USP combines these approaches for different model shapes and network topologies (Jacobs et al., 2023; Liu et al., 2024; Fang and Zhao, 2024). The FlashAttention family uses IO-aware tiling and asynchronous pipelines to improve single-GPU attention efficiency (Dao et al., 2022; Dao, 2024; Shah et al., 2024; Zadouri et al., 2026). Other systems place tile-level communication around projection, attention, or output stages (Jin et al., 2026; Sul et al., 2026). Our distinction is the explicit use of post-issue remote-store progress: the output-owning role returns to computation after issue, while X-Stage drains the accepted request. In the evaluated region, this avoids reserving a warp or SM solely to wait for communication progress. 8. Conclusion This paper identifies X-Stage, a software-visible execution phase between remote-store issue and remote-visible completion. Accepted stores can progress while the issuer resumes useful work, but sustained injection eventually fills finite effective capacity and backpressures later issue, the epilogue, and the Tensor Core producer. A lightweight Burst–Gap model captures this behavior with backpressure-free issue time, effective drain rate, and effective outstanding capacity. The model guides two complementary kernel transformations. Cross-wave Linear-1/Linear-2 interleaving reshapes MegaMoE Combine bursts and provides a 1.18×1.18× geometric-mean and 1.62×1.62× maximum kernel speedup across 84 configurations. A piggybacked FlashAttention–A2A pipeline lets the output-owning role issue remote stores and resume computation without a dedicated communication warp or SM. FA3 and FA4 reach maximum speedups of 1.43×1.43× and 1.42×1.42×, respectively, and their sender-visible steady-state times approach FlashAttention-only time as the Q-loop covers post-issue draining. X-Stage complements tile dependencies and remote-readiness protocols by exposing a missing scheduling dimension: the rate and capacity of accepted but not yet completed remote stores. The current characterization covers one-sided remote stores, so other communication operations require independent validation. Within that scope, the model provides a measurable connection between remote-store behavior and fused-kernel scheduling. References A. Alexandrov, M. F. Ionescu, K. E. Schauser, and C. Scheiman (1995) LogGP: incorporating long messages into the LogP model. In Proceedings of the Seventh Annual ACM Symposium on Parallel Algorithms and Architectures (SPAA), p. 95–105. Cited by: §7.3. L. Chang, W. Bao, Q. Hou, C. Jiang, N. Zheng, Y. Zhong, X. Zhang, Z. Song, C. Yao, Z. Jiang, H. Lin, X. Jin, and X. Liu (2024) FLUX: Fast Software-based Communication Overlap on GPUs Through Kernel Fusion. arXiv preprint arXiv:2406.06858. Cited by: §1, §1, §2.1, §7.2, §7.4. M. Cowan, S. Maleki, M. Musuvathi, O. Saarikivi, and Y. Xiong (2023) MSCCLang: microsoft collective communication language. In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (ASPLOS), Note: Originally released as GC3, arXiv:2201.11840 External Links: Document Cited by: §7.2. M. Cui, A. Wingkvist, and M. Ericsson (2026) Fine-grained computation-communication overlap via tile-level signaling and scheduling for mixture-of-experts. arXiv preprint arXiv:2607.19539. Cited by: §7.4. D. Culler, R. Karp, D. Patterson, A. Sahay, K. E. Schauser, E. Santos, R. Subramonian, and T. von Eicken (1993) LogP: towards a realistic model of parallel computation. In Proceedings of the Fourth ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (PPoPP), p. 1–12. Cited by: §7.3. T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. Ré (2022) FlashAttention: fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems 35 (NeurIPS), Cited by: §7.5. T. Dao (2024) FlashAttention-2: faster attention with better parallelism and work partitioning. In International Conference on Learning Representations (ICLR), p. 35549–35562. Cited by: §7.5. DeepSeek-AI (2024) DeepSeek-V3 Technical Report. arXiv preprint arXiv:2412.19437. Cited by: §7.4. DeepSeek-AI (2025) EPLB: Expert Parallelism Load Balancer. Note: GitHub repositoryhttps://github.com/deepseek-ai/EPLB Cited by: §1, §7.4. DeepSeek-AI (2026a) DeepGEMM: High-performance Tensor Core Kernels and Mega MoE. Note: GitHub repositoryWorking reference; add commit hash and access date before submission Cited by: §1, §2.2, §2.3, §7.4. DeepSeek-AI (2026b) DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv preprint arXiv:2606.19348. Cited by: §1, §2.3, §2.3. J. Fang and S. Zhao (2024) USP: a unified sequence parallelism approach for long context generative AI. arXiv preprint arXiv:2405.07719. Cited by: §7.5. Z. Fei, M. Fan, C. Yu, D. Li, and J. Huang (2024) Scaling diffusion transformers to 16 billion parameters. arXiv preprint arXiv:2407.11633. Cited by: §1, §7.4. T. Gale, D. Narayanan, C. Young, and M. Zaharia (2023) MegaBlocks: efficient sparse training with mixture-of-experts. In Proceedings of Machine Learning and Systems (MLSys), Vol. 5, p. 288–304. Cited by: §7.4. J. He, J. Zhai, T. Antunes, H. Wang, F. Luo, S. Shi, and Q. Li (2022) FasterMoE: modeling and optimizing training of large-scale dynamic pre-trained models. In Proceedings of the 27th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (PPoPP), p. 120–134. External Links: Document Cited by: §7.4. K. Hong, X. Li, M. Liu, Q. Mao, T. Wu, Z. Huang, L. Chen, Z. Wang, Y. Zhang, Z. Zhu, G. Dai, and Y. Wang (2026) Efficient and adaptable overlapping for computation and communication via signaling and reordering. In Proceedings of the 21st European Conference on Computer Systems (EuroSys), p. 1894–1911. External Links: Document Cited by: §7.4. S. A. Jacobs, M. Tanaka, C. Zhang, M. Zhang, S. L. Song, S. Rajbhandari, and Y. He (2023) DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models. arXiv preprint arXiv:2309.14509. Cited by: §1, §1, §2.1, §4.3, §7.5. A. Jangda, J. Huang, G. Liu, A. H. N. Sabet, S. Maleki, Y. Miao, M. Musuvathi, T. Mytkowicz, and O. Saarikivi (2022) Breaking the computation and communication abstraction barrier in distributed machine learning workloads. In Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), p. 402–416. External Links: Document Cited by: §7.2. A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. Singh Chaplot, D. de las Casas, E. Bou Hanna, F. Bressand, G. Lengyel, G. Bour, G. Lample, L. Renard Lavaud, L. Saulnier, M. Lachaux, P. Stock, S. Subramanian, S. Yang, S. Antoniak, T. Le Scao, T. Gervet, T. Lavril, T. Wang, T. Lacroix, and W. El Sayed (2024) Mixtral of experts. arXiv preprint arXiv:2401.04088. Cited by: §7.4. C. Jin, Z. Jiang, Z. Bai, Z. Zhong, J. Liu, X. Li, N. Zheng, X. Wang, C. Xie, Q. Huang, W. Heng, Y. Ma, W. Bao, S. Zheng, X. Zheng, Y. Peng, H. Lin, X. Liu, X. Jin, and X. Liu (2026) MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production. In Proceedings of the Twenty-First European Conference on Computer Systems (EuroSys), p. 366–382. External Links: Document Cited by: §1, §1, §2.1, §4.3, §7.2, §7.4, §7.4, §7.5. J. D. C. Little (1961) A proof for the queuing formula: L=λWL=λ W. Operations Research 9 (3), p. 383–387. Cited by: §7.3. H. Liu, M. Zaharia, and P. Abbeel (2024) RingAttention with blockwise transformers for near-infinite context. In International Conference on Learning Representations (ICLR), p. 3992–4008. Cited by: §7.5. Y. Ma, S. Shen, T. Chen, A. Langer, J. Kraus, B. Glick, C. Belusar, J. Hammond, and T. Hoefler (2026) Demystifying NVSHMEM: A System-Level Analysis on Symmetric Memory and Device-Initiated Operations in GPU Communication. arXiv preprint arXiv:2606.05951. Cited by: §1, §7.1. X. Mei, Q. Wang, and X. Chu (2017) A survey and measurement study of GPU DVFS on energy conservation. Digital Communications and Networks 3 (2), p. 89–100. Cited by: §5.3. NVIDIA (2026a) CUDA C++ Programming Guide: Peer Device Memory Access. Note: NVIDIA Developer DocumentationAccessed 2026; add exact URL and access date before submission Cited by: §2.1. NVIDIA (2026b) NVIDIA Collective Communications Library (NCCL) Documentation. Note: https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/Version 2.30.7, accessed July 2026 Cited by: §7.1. NVIDIA (2026c) NVLink and NVSwitch System Overview. Note: NVIDIA Data Center DocumentationAccessed 2026; add exact URL and access date before submission Cited by: §2.1. NVIDIA (2026d) NVSHMEM: GPU-Initiated Communication Library. Note: NVIDIA Developer DocumentationAccessed 2026; add exact URL and access date before submission Cited by: §1, §2.1, §7.1. X. Pan, W. Lin, L. Zhang, S. Shi, Z. Tang, R. Wang, B. Li, and X. Chu (2025) FSMoE: a flexible and scalable training system for sparse mixture-of-experts models. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1 (ASPLOS), p. 524–539. External Links: Document Cited by: §7.4. W. Peebles and S. Xie (2023) Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 4195–4205. Cited by: §4.3. J. Shah, G. Bikshandi, Y. Zhang, V. Thakkar, P. Ramani, and T. Dao (2024) FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision. In Advances in Neural Information Processing Systems 37 (NeurIPS), p. 68658–68685. Cited by: §4.3, §7.5. N. Shazeer (2020) GLU variants improve transformer. arXiv preprint arXiv:2002.05202. Cited by: §2.2. M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro (2019) Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. arXiv preprint arXiv:1909.08053. Cited by: §2.1. S. H. Sul, S. Arora, B. F. Spector, and C. Ré (2026) ParallelKittens: Systematic and Practical Simplification of Multi-GPU AI Kernels. In Proceedings of Machine Learning and Systems (MLSys), Cited by: §1, §1, §2.1, §7.1, §7.2, §7.5. H. Sun, T. Lei, B. Zhang, Y. Li, H. Huang, R. Pang, B. Dai, and N. Du (2025) EC-DIT: scaling diffusion transformers with adaptive expert-choice routing. In International Conference on Learning Representations (ICLR), p. 72383–72401. Cited by: §1, §7.4. D. Unat, I. Turimbetov, M. Issa, D. Sagbili, F. Vella, D. D. Sensi, and I. Ismayilov (2026) The landscape of GPU-centric communication. ACM Computing Surveys 58 (12). External Links: Document Cited by: §7.1, §7.1. L. Wang, H. Gao, C. Zhao, X. Sun, and D. Dai (2024) Auxiliary-loss-free load balancing strategy for mixture-of-experts. arXiv preprint arXiv:2408.15664. Cited by: §7.4. Y. Wang, Y. Wu, X. Wu, J. Cui, W. Cai, Z. Guo, and J. Huang (2026) ReaLB: real-time load balancing for multimodal MoE inference. arXiv preprint arXiv:2604.19503. Cited by: §7.4. S. Williams, A. Waterman, and D. Patterson (2009) Roofline: an insightful visual performance model for multicore architectures. Communications of the ACM 52 (4), p. 65–76. Cited by: §7.3. Y. Yu, H. Ma, K. Agarwal, N. Oswald, Q. Huang, H. Linsenmaier, C. Mei, R. Zhao, R. Borkar, B. Darvish Rouhani, D. Nellans, R. Krashinsky, and A. Khandelwal (2025) Efficient MoE serving in the memory-bound regime: balance activated experts, not tokens. arXiv preprint arXiv:2512.09277. Cited by: §7.4. Y. Yuan, Z. Wang, Z. Huang, D. Zhu, X. Zhou, J. Yu, and Q. Min (2025) Expert race: a flexible routing strategy for scaling diffusion transformer with mixture of experts. In Proceedings of the 42nd International Conference on Machine Learning (ICML), p. 73671–73682. Cited by: §1, §7.4. T. Zadouri, M. Hoehnerbach, J. Shah, T. Liu, V. Thakkar, and T. Dao (2026) FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling. arXiv preprint arXiv:2603.05451. Cited by: §7.5. S. Zhang, N. Zheng, H. Lin, Z. Jiang, W. Bao, C. Jiang, Q. Hou, W. Cui, S. Zheng, L. Chang, Q. Chen, and X. Liu (2025) Comet: Fine-grained Computation-Communication Overlapping for Mixture-of-Experts. In Proceedings of Machine Learning and Systems (MLSys), Cited by: §1, §1, §2.1, §7.2, §7.4, §7.4. S. Zheng, J. Fang, X. Zheng, Q. Hou, W. Bao, N. Zheng, Z. Jiang, D. Wang, J. Ye, H. Lin, L. Chang, and X. Liu (2025) TileLink: Generating Efficient Compute-Communication Overlapping Kernels Using Tile-Centric Primitives. In Proceedings of Machine Learning and Systems (MLSys), Cited by: §1, §2.1, §7.2. S. Zheng, X. Zheng, L. Chang, and J. Zhai (2026) UniEP: unified expert-parallel megakernel MoE for LLM training. In Proceedings of the 35th International Symposium on High-Performance Parallel and Distributed Computing (HPDC), p. 387–401. External Links: Document Cited by: §7.4.