Paper deep dive
Performance Foundations of Parallel & Distributed Reasoning Language Models
Maciej Besta, Leonard Schmidt, Lara Nonino, Robert Gerstenberger, Pierre Pang, Patrik Okanovic, Ales Kubicek, Tiancheng Chen, Baraq Lipshitz, Torsten Hoefler
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 8/28/2026, 4:36:17 AM
Summary
This paper analyzes the performance foundations of parallel and distributed systems for training Reasoning Language Models (RLMs) using Reinforcement Learning (RL). It systematizes the RL-for-LLM paradigm, analyzing algorithms like PPO and GRPO, and develops a taxonomy of intra- and inter-model parallelism strategies. The work highlights that RLM training is a significant distributed systems challenge due to massive computational requirements and complex multi-model pipelines, offering guidelines for scalable and cost-effective development.
Entities (17)
Relation Signals (13)
PPO → uses → Critic Model
confidence 95% · A standard Proximal Policy Optimization (PPO)-style setup maintains four distinct LLMs (actor (policy), reward, critic (value), and a reference model)
PPO → uses → Reference Model
confidence 95% · A standard Proximal Policy Optimization (PPO)-style setup maintains four distinct LLMs (actor (policy), reward, critic (value), and a reference model)
PPO → uses → Reward Model
confidence 95% · A standard Proximal Policy Optimization (PPO)-style setup maintains four distinct LLMs (actor (policy), reward, critic (value), and a reference model)
PPO → uses → Actor Model
confidence 95% · A standard Proximal Policy Optimization (PPO)-style setup maintains four distinct LLMs (actor (policy), reward, critic (value), and a reference model)
Paper → analyzes → Work-Depth Model
confidence 90% · We harness the work-depth model of parallel computing to make our taxonomy and its insights rigorous and portable.
RLM → uses → GRPO
confidence 90% · provide a compute-centric analysis of prominent post-training algorithmic frameworks: ... Group Relative Policy Optimization (GRPO)
RLM → uses → PPO
confidence 90% · provide a compute-centric analysis of prominent post-training algorithmic frameworks: Proximal Policy Optimization (PPO)... as well as their variants.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Reinforcement Learning with Verifiable Rewards (RLVR) and other RL-style post-training paradigms have been used for aligning large language models (LLMs) with reasoning standards. The resulting recent Reasoning Language Models (RLMs) such as DeepSeek-R1, o3, and Kimi k1.5 show that such RL-style post-training ("RL-for-LLMs") can substantially improve chain-of-thought reasoning, long-horizon planning, and self-correction. However, the computational footprint of these systems is massive: state-of-the-art RLM training requires millions of GPU-hours and tightly coupled multi-model pipelines that stress modern hardware far beyond classical supervised LLM training. This makes RLM training as much a parallel and distributed systems problem as an algorithmic one. In this work, to facilitate developing RLMs that are simultaneously high-performance, scalable, and cost-effective, we first systematize the RL-for-LLM paradigm and provide a compute-centric analysis of prominent post-training algorithmic frameworks: Proximal Policy Optimization (PPO), Group Relative Policy Optimization (GRPO), as well as their variants. Second, we develop a taxonomy of intra- and inter-model parallelism strategies for RL-for-LLMs, covering both traditional techniques (data, tensor, pipeline, sequence, context, and expert parallelism) as well as novel forms of parallelism and optimization techniques for multi-model RLM training, for example disaggregated placement, stage fusion, hybrid parallelism, and asynchronous execution. We harness the work-depth model of parallel computing to make our taxonomy and its insights rigorous and portable. Finally, we analyze existing RLM frameworks and we distill practical guidelines and outline open research directions for building scalable, fast, and cost-effective RLMs.
Tags
Links
- Source: https://arxiv.org/abs/2608.27046v1
- Canonical: https://arxiv.org/abs/2608.27046v1
Trouble viewing inline? Open PDF directly →
Full Text
312,432 characters extracted from source content.
Expand or collapse full text
Performance Foundations of Parallel & Distributed Reasoning Language Models Maciej Besta†* Leonard Schmidt* Lara Nonino Robert Gerstenberger Pierre Pang Affiliation: Patrik Okanovic, Ales Kubicek, Tiancheng Chen, Baraq Lipshitz, Torsten Hoefler Affiliation: ETH Zurich †Corresponding author *Core contributions Abstract Reinforcement Learning with Verifiable Rewards (RLVR) and other RL-style post-training paradigms have been used for aligning large language models (LLMs) with reasoning standards. The resulting recent Reasoning Language Models (RLMs) such as DeepSeek-R1, o3, and Kimi k1.5 show that such RL-style post-training (“RL-for-LLMs”) can substantially improve chain-of-thought reasoning, long-horizon planning, and self-correction. However, the computational footprint of these systems is massive: state-of-the-art RLM training requires millions of GPU-hours and tightly coupled multi-model pipelines that stress modern hardware far beyond classical supervised LLM training. This makes RLM training as much a parallel and distributed systems problem as an algorithmic one. In this work, to facilitate developing RLMs that are simultaneously high-performance, scalable, and cost-effective, we first systematize the RL-for-LLM paradigm and provide a compute-centric analysis of prominent post-training algorithmic frameworks: Proximal Policy Optimization (PPO), Group Relative Policy Optimization (GRPO), as well as their variants. Second, we develop a taxonomy of intra- and inter-model parallelism strategies for RL-for-LLMs, covering both traditional techniques (data, tensor, pipeline, sequence, context, and expert parallelism) as well as novel forms of parallelism and optimization techniques for multi-model RLM training, for example disaggregated placement, stage fusion, hybrid parallelism, and asynchronous execution. We harness the work–depth model of parallel computing to make our taxonomy and its insights rigorous and portable. Finally, we analyze existing RLM frameworks and we distill practical guidelines and outline open research directions for building scalable, fast, and cost-effective RLMs. Index Terms: Parallel Reasoning Language Models, Distributed Reasoning Language Models, Parallel RLVR, Distributed RLVR, Asynchronous Reasoning Language Models, Asynchronous RLVR. †aftertitle: I Introduction Fig. 1: Overview of the contributions and analyses in this work. Reinforcement learning (RL) is now the computationally dominant post-training paradigm for aligning large language models (LLMs) with reasoning standards. After pretraining and supervised fine-tuning (SFT), RL optimizes a policy against learned reward models or other feedback signals to improve task-specific performance beyond next-token prediction [192, 20]. The resulting frontier Reasoning Language Models (RLMs), such as DeepSeek-R1 [107], OpenAI’s o3 [188], and Kimi k1.5 [86], demonstrate that carefully designed preference learning and policy optimization over Chain-of-Thought trajectories can produce substantially stronger reasoning, long-horizon planning, and self-correction than SFT alone. A key paradigm behind this trend is RL with Verifiable Rewards (RLVR), in which reward signals are obtained from automatically checkable outcomes, such as exact-answer verification in mathematics, unit tests in code, or other task-specific verifiers [243, 118, 173]. The computational footprint of RLMs is enormous [156]. For example, OpenAI reports that in developing o3 it scaled RL by an additional order of magnitude in both train-time RL compute and inference-time reasoning compute, while still observing clear gains from further scaling [187]. More broadly, cost analyses of frontier model training indicate that end-to-end training runs for leading systems already cost tens to hundreds of millions of US dollars in compute alone, and they may reach more than $1B by 2027 [73]. In contrast to classical supervised LLM training, RLM pipelines [26] couple several large models and stages. A standard Proximal Policy Optimization (PPO)-style setup maintains four distinct LLMs (actor (policy), reward, critic (value), and a reference model) and repeatedly executes them across three stages (Generation, Assessment11 1 While the Assessment stage is often called “Inference” in the literature, we use the term “Assessment” because it more precisely describes its functional role. Indeed, inference is not specific to this stage: Generation also performs inference, albeit autoregressively., Training) with complex data dependencies. Recent reasoning-oriented systems often use critic-free Group Relative Policy Optimization (GRPO)-style training in practice, largely because removing the critic reduces memory pressure and simplifies large-scale rollout training [213, 107]. Nevertheless, PPO-style actor–critic training remains a key and relevant design point as its learned value baseline reduces gradient variance and stabilizes policy updates, providing more faithful token-level credit assignment than critic-free alternatives such as GRPO [79, 239]. PPO has been used or revived in reasoning-focused RL systems such as VAPO [268]. Overall, both PPO and GRPO as well as their variants are crucial RLM training frameworks. Systems used for RL post-training, for each ReaL [169], RLHFuse [282], and OpenRLHF [117], have shown that naively reusing supervised-training parallelization strategies for RLMs yields suboptimal GPU utilization. Instead, the RLM workflow demands joint reasoning about model placement, inter-model scheduling, and heterogeneous workload characteristics. This combination of massive scale and multi-model RL pipelines makes RLMs fundamentally a parallel and distributed systems problem. To facilitate the development of next-generation scalable and cost-effective RLMs, we conduct an in-depth investigation into the foundations of their parallel and distributed computational characteristics; a roadmap of the paper can be found in Figure 1. First, we systematize the RL-for-LLM paradigm and provide a compute-centric analysis of several prominent post-training frameworks (contribution 1), including PPO-style [279] and GRPO-like methods [213]. We also consider preference-gradient approaches such as Direct Preference Optimization (DPO) [250] – while these methods do not use RL, they have been shown to also enhance reasoning capabilities [242, 134, 238, 252]. Next, we develop a taxonomy of parallelism for RLMs (contribution 2). The first part focuses on intra-model parallelism, instantiated by LLM-based policies, critics, rewards, and references. For this, we analyze traditional forms of parallelism and how they interact with the RL pipeline, for instance which RL stage benefits most from which form of intra-model parallelism. Here, we consider data parallelism (DP) [146], tensor parallelism (TP, also referred to as operator parallelism [219, 25], pipeline parallelism (P) [120, 176], sequence parallelism (SP) [132], context parallelism (CP) [160], and expert parallelism (EP) [141]). The second part of our taxonomy targets inter-model parallelism: opportunities to run different models and RL stages concurrently, and system-level optimizations that reduce end-to-end iteration time by reshaping how these components interact. We show that the available forms of inter-model parallelism are largely determined by a small set of concrete design choices: model structure configurations, which can eliminate redundant model executions (e.g., critic–reward merging), model placement strategies, which enable component-level concurrency by co-locating models on the same devices or disaggregating them onto separate device groups/services; stage and sub-task fusion, which introduces stage-level overlap (e.g., overlapping long-tailed generation with reward/value evaluation, or pipelining actor and critic updates) to remove serialization between stages; hybrid intra-model configurations across components, where different models and stages use different mixtures of intra-model parallelism; and asynchronous execution, which decouples rollout generation from learning via bounded staleness. Our taxonomy provides a common performance-oriented language for formulating, comparing, and implementing new optimizations for large-scale RLM training. To ensure that all the insights are portable across different architectures, we use work/depth/memory complexities for all the parts of the taxonomy (contribution 4). We then use the taxonomy to conduct a systematic analysis of existing RLM models and frameworks and analyze them through the lens of the parallelism and optimization strategies they implement (contribution 5). Concretely, we examine general-purpose libraries such as TRL [231], ColossalChat [267], DeepSpeed-Chat [261], OpenRLHF [117], HybridFlow [217], and NeMo RL [214, 184], as well as more specialized systems including ReaL [169], RLHFuse [282], FlexRLHF [251], StreamRL [281], Asynchronous RLHF [180], AReaL [92], and Pipe-RLHF [262]. For each framework, we catalogue which intra- and inter-model techniques are supported. This enables practitioners to make informed choices about which framework best matches their needs, and it reveals systematic gaps where our taxonomy suggests additional optimizations (for example, underexplored stage-fusion and placement policies for multi-model PPO-style pipelines). In this way, our study not only describes the current ecosystem but also points to concrete directions for enhancing existing RL frameworks and designing new ones. I-A Complementary Analyses & Related Work There exist general studies on the blueprint for RLMs [26] and broad surveys on LLM-enhanced RL [237, 61]. Other works provide comprehensive analyses of specific components of the RL pipeline [279, 232, 250]. Some surveys detail techniques for resource-efficient LLMs, covering algorithms, model compression, and systems [83, 17, 253]. More broadly, [25] offers a foundational concurrency analysis of parallel and distributed deep learning. However, these works do not focus on the unique demands of RLM pipelines. Closer to our work, Liu et al. explore acceleration techniques for deep RL [162] and Chen et al. survey the effects of scaling LLM’s reasoning capabilities [68]. However, these works do not explain the underlying performance foundations of the parallel and distributed systems that enable RLMs to scale. Specifically, the latter focuses on how reasoning capabilities emerge with scale, not on the concurrency and system bottlenecks of the RLM training loop. Similarly, the former analyzes acceleration for general deep RL, not the specific parallel architectures required by modern RLM pipelines. We complement these studies by providing the first in-depth analysis to demystify the performance foundations of parallel and distributed RLMs. I-B Analysis of Parallel Algorithms We use formal models for reasoning about parallelism. Specifically, we use the work-depth (WD) analysis, an established approach for bounding runtimes of parallel algorithms. The work (W) of an algorithm is the total number of operations and the depth (D) is defined as the longest sequential chain of execution in the algorithm, and it forms the lower bound on the algorithm execution time [56, 53]. One usually wants to minimize depth while preventing work from increasing too much. Finally, Memory accounts for model parameters, gradients, optimizer state (Adam first and second order moments), and stored activation tensors during a single training iteration. Models & training signals used in the RL pipeline πθ _θ Policy (Actor) model: the trainable LLM being optimized. πref _ref Reference model: the frozen baseline policy (usually a checkpoint right after SFT). RφR_ Reward model: an assessment tool that maps prompt–response pairs to scalar rewards. VψV_ψ Critic model (Value function): an assessment tool that estimates expected future returns from the reward model. rb(i),Ab,t(i),G^b,t(i)r_b^(i),A_b,t^(i), G_b,t^(i) Reward for the i-th candidate of prompt b, advantage at token step t, and discounted return estimated at token step t. Data & Dimensions ,X,Y Space of input prompts and token sequences, respectively X=x1,…,xBX=\x_1,…,x_B\ Batch of B input prompts; X⊂X . y=(y1,…,yT)y=(y_1,…,y_T) A response sequence consisting of T tokens (y∈y ). K Number of candidate responses generated per prompt (in the Generation stage). S,TS,T The average prompt length and the average response length. LLM & system design N,L,V,BN,L,V,B Number of parallel devices, number of layers, vocabulary size, batch size, respectively. d,dff,h,dk=d/hd,d_f,h,d_k=d/h Hidden dimension, the intermediate FFN dimension, number of parallel heads, and per-head dimension, respectively. E,Ea,deE,E_a,d_e The number of experts per layer, the number of active experts per token, and the expert hidden dimension, respectively. TABLE I: Overview of basic mathematical notation and concepts used in the paper. Notation specific to the complexity analyses is detailed separately in Table I. Fig. 2: Overview of policy optimization algorithms and used models. No AI was used to conceive or to draw the figure. I Overview & Foundations of RLMs We first overview basic RLM post-training concepts. I-A RL Post-Training Pipeline: Overview The essence of RL for LLM can be captured as a simple three-stage loop that iterates continuously: Generation → Assessment → Training → Generation →…→…. These stages operate over a batch of input prompts, denoted by X=x1,x2,…,xBX=\x_1,x_2,…,x_B\, where B is the batch size. Starting from a language model that has typically been pre-trained and then fine-tuned via supervised learning (SFT), this loop further improves this model (referred to as the policy model πθ _θ) through RL-based optimization. In each iteration, the Generation stage produces candidate responses to input prompts using the current policy πθ _θ. Then, Assessment evaluates those responses and produces feedback (which can be both scalar and vector). Finally, Training uses the resulting feedback to update the policy πθ _θ. Both Assessment and Training stages may invoke one or more auxiliary models that provide the feedback signals used to improve the policy πθ _θ, i.e., the reward model RφR_ , the critic model VψV_ψ, and the reference model πref _ref. This iterative process continues until convergence criteria are met or a predetermined number of training loops are completed, with the updated πθ _θ model from each training stage serving as the starting point for the next cycle. We overview this process in Figure 2; additional mathematical details are in Appendix A and in Table I. I-B Terminology: RLVR, RLHF, RLAIF & Others We use RL-style reasoning post-training as the umbrella term for reasoning-oriented post-training pipelines that use feedback beyond next-token prediction. This term is broader than strict online RL: it includes (1) RLVR, where online RL algorithms such as PPO and GRPO-style methods use verifiers as rewards; (2) online RL-based post-training with model-based assessment, including RL from Human Feedback (RLHF), RL from AI Feedback (RLAIF), direct RLAIF (d-RLAIF), and learned reward methods; and (3) offline preference-optimization methods such as DPO, which are not RL in their basic form but have also been used to enhance reasoning. This broader umbrella is also reflected in systems practice: frameworks originally named after RLHF are now used for broader reasoning post-training. For example, OpenRLHF [117] supports PPO- and GRPO-style training, while RLHFuse [282] optimizes the same staged workflow used by both RLHF and RLVR systems. Strict RLVR denotes online RL with rewards produced by an automatic verifier rather than a learned reward model. Examples include exact-answer matching in mathematics, unit-test execution or compilation in code, theorem proving, and other programmatic correctness checks [213, 107]. A second class uses model-based assessment. In classical RLHF, rewards are derived from human preference data, typically through a learned reward model [192]. In canonical RLAIF, human preference labels are replaced by AI-generated preferences or critiques, which are then used to train a reward model; Constitutional AI is a canonical example [21, 140]. A closely related variant is direct RLAIF (d-RLAIF), where an LLM directly evaluates policy outputs and provides rewards during RL, without training a separate reward model [140]. Thus, LLM-as-a-judge reward schemes can naturally be viewed as instances of d-RLAIF rather than a separate feedback class. Neither RLAIF nor d-RLAIF is strict RLVR because their Assessment signal is an AI-generated judgment rather than an externally verifiable correctness check. Learned assessment models are nevertheless important for reasoning. Process-supervised reward models improve mathematical reasoning [155]; Math-Shepherd trains a process reward model and uses step-level PPO to improve GSM8K and MATH performance [235]; there are further similar examples [138, 76, 103]. These examples motivate retaining reward-model terms in our complexity analysis. When Assessment is implemented by a learned outcome or process reward model, it requires LLM forward passes; when process-level feedback is used, the evaluator may be invoked at many reasoning steps. Such learned-assessment pipelines can therefore be more compute-heavy than lightweight exact-match RLVR, even though both instantiate the same Assessment role. A third class consists of offline preference-optimization methods. DPO is not RL in its basic form: it inherits the preference-learning objective of RLHF but optimizes static preference pairs without online rollout generation or explicit reward-model inference [200]. Yet, DPO-style methods have also been adapted to reasoning, for example through iterative reasoning preference optimization and step-wise DPO [242, 134, 238, 252]. We include them because they share parts of the same post-training staged dataflow with policy/reference forward passes and policy backpropagation. Overall, all these methods build upon all or part of the same systems abstraction: Generation → Assessment → Training. Online RL methods such as RLHF, RLAIF/d-RLAIF, and RLVR execute the full loop: the policy samples trajectories, Assessment maps them to rewards, model judgments, verifier outcomes, advantages, or other learning signals, and Training updates the policy and potentially critic. Offline preference methods such as DPO remove online Generation and explicit reward Assessment from the inner loop, but retain the policy/reference comparison and Training components. Our analysis therefore targets the broad class of RL-style reasoning post-training systems, where the Assessment stage may be instantiated by a learned reward model, an LLM evaluator, an automatic verifier, or an offline preference signal. I-C Auxiliary Models in the RL-LLM Pipeline We now investigate the functional and execution characteristics of models used in the RL pipeline, see Table I and Figure 2 (top). While conceptually distinct, in practical settings, they share similar LLM-based architectures. Model State Used in System design remarks Policy (Actor) πθ _θ Trainable Generation, Training Bottleneck: Sequential decoding dominates wall-clock time. Reward RφR_ (Outcome) Frozen Assessment Efficient parallel evaluation; no backward pass. Easy to offload. Reward RφprR_ ^pr (Process) Frozen Assessment High overhead; sequential decoding if using sequential verification (e.g., MCTS). Critic (Value) VψV_ψ Trainable Assessment, Training Doubles training compute (fwd+bwd). High memory pressure (optimizer states). Reference πref _ref Frozen Assessment Required for KL/DPO. Forward-only; often quantized to save memory. TABLE I: Overview of models used in the RLM pipeline. I-C1 Reward Model The reward model is a function Rφ:×→ℝR_ :X×Y that maps a prompt–response pair (x,y)(x,y) to a scalar reward r, i.e., Rφ(x,y)=rR_ (x,y)=r, representing overall response quality, helpfulness, or alignment with desired behavior. These rewards are typically learned from human preference data and serve as the principal alignment signal for the policy. During the Assessment stage, the reward model performs a single forward pass per full sequence to produce rb(i)r_b^(i). It is a frozen model (no gradients computed). Moreover, as scalar rewards are usually single-token, its execution is usually non‑autoregressive, which makes its evaluation considerably cheaper than autoregressive policy generation. There are two types of reward models: outcome-based and process-based reward models (RφprocR^proc_ ). The former is a standard mechanism that assigns a single scalar score r to a completed sequence. Process-based reward models provide feedback at intermediate steps of reasoning. Formally, it maps a sequence to a vector of rewards. While this provides a richer supervision signal, it significantly increases computational cost (as it may require multiple forward passes or Monte Carlo Tree Search (MCTS) integration) and is difficult to train [107, 26]. I-C2 Critic Model The critic model is a function defined as Vψ:×≤t→ℝV_ψ:X×Y_≤ t . It estimates the expected future reward from a partial sequence, i.e., Vψ(x,y<t)=vtV_ψ(x,y_<t)=v_t. Given a prompt x and prefix y<ty_<t, the critic predicts vtv_t as an estimate of the return that the policy can expect from that point onward, i.e., Vψ(x,y<t)≈πθ[∑τ=tTγτ−trτ∣x,y<t]V_ψ(x,y_<t) _ _θ\! [ _τ=t^Tγ^τ-tr_τ x,y_<t ]. When r comes from a learned reward model, we have rT=Rφ(x,y1:T)r_T=R_ (x,y_1:T) and rt=0r_t=0 for t<T,γ=1t<T,γ=1. Unlike the reward model, the critic provides token-level information, producing value estimates for all positions t in a single forward pass over the full sequence. During the Assessment stage, these predictions are combined with observed rewards to compute advantage estimates AtA_t. These advantage estimates are then used in Training to guide gradient updates of the policy model. The critic runs a full forward pass per sampled sequence and a backward pass for its parameter update, often in parallel with the policy gradient computation. Unlike reward models, the critic is trainable. The reward model RφR_ acts as a fixed proxy for task quality; keeping it fixed during policy optimization provides a stationary reward objective. The critic VψV_ψ, however, estimates the expected return under the current policy. As πθ _θ changes, the distribution of rollouts and their expected returns changes as well. Therefore, PPO-style methods update VψV_ψ after each rollout batch by minimizing a value-regression loss 1|ℬ|∑(x,y)∈ℬ∑t(Vψ(x,y<t)−G^t)2 1|B| _(x,y) _t (V_ψ(x,y_<t)- G_t )^2, where G^t G_t is a Monte-Carlo or GAE-style return target computed from the rewards collected on that batch. In this way, the critic tracks the return function induced by the evolving policy. I-C3 Reference Model The reference model πref _ref is a frozen policy πref:×→[0,1] _ref:X×Y→[0,1] that serves as a fixed baseline to stabilize optimization. It is typically a snapshot of the policy after SFT and before any RL. During the Assessment stage, it provides per-token log-probabilities logπref(yt∣x,y<t) _ref(y_t x,y_<t) used to measure deviation from the current policy. During the Training stage, these outputs are reused for regularization, e.g., through a KL-divergence penalty DKL(πθ∥πref)D_KL( _θ\| _ref) or preference deltas in DPO. Since it remains frozen, πref _ref requires only forward passes, but it can still be expensive to evaluate at scale because it performs a full-sequence teacher-forced forward pass over every rollout. I-D Auxiliary Models vs. Algorithmic RL Frameworks Depending on what auxiliary models are used, there are PPO-like methods, GRPO-like methods, and DPO-like methods, with distinct computational trade-offs. PPO [208] uses all four models, requiring both forward and backward passes for πθ _θ and VψV_ψ, and additional forward-only evaluation for RφR_ and πref _ref. GRPO [107, 156, 213] omits the critic, saving backward passes but still requiring reward inference. DPO [200] relies solely on two models (πθ _θ and πref _ref), resulting in an offline teacher-forced forward/backward pipeline (without online Generation or explicit reward-model inference) that is computationally lightweight and easier to scale; however, it also takes as input the pre-computed preference dataset. Thus, the combination of these models not only defines the algorithmic behavior of the RL method but also determines its computational profile, parallelism potential, and system-level design. I-D1 Variants and Emerging Frameworks There are numerous variants of PPO, GRPO, and DPO-like methods, we now summarize most important ones, focusing on their computational characteristics. Within the PPO family, VAPO [268] retains the value model but improves value-based training efficiency and stability through value pretraining and decoupled GAE, so its compute remains recognizably actor–critic rather than critic-free. RTO [280] shifts supervision toward token-level rewards and advantages, increasing per-token bookkeeping during training. Safe-RLHF [75] extends PPO-like RLHF with a separate cost model and a Lagrangian safety constraint, thereby adding another assessment model while making helpfulness–harmlessness trade-offs explicit. Other PPO-style trust-region variants include divergence-based DPPO [198], which replaces ratio clipping with direct divergence estimates while preserving the basic online actor–critic profile, and mirror-descent formulations such as MDPO [228], which largely preserve the actor–critic compute structure while changing the policy update from PPO’s clipped-ratio surrogate to a Bregman- or KL-regularized mirror-descent step around the previous policy. Within the GRPO family, DAPO [265] improves sample and FLOP efficiency through asymmetric clipping, dynamic sampling, token-level losses, and better handling of overlong trajectories, while pruning-based extensions such as DPPO [286] reduce wasted decoding and training work by pruning unpromising trajectories with unbiased correction. Other GRPO-like extensions such as Graph-GRPO [60], RiskPO [203], GBMPO [266], MicroCoder-GRPO [154], and PIPO [233] mostly preserve the same critic-free online compute profile while changing the regularizer, feedback signal, or application setting. Related critic-free online methods such as RLOO [6] and ReMax [153] also avoid value-model training, but replace learned critics with leave-one-out, sample-based, or greedy-response baselines rather than GRPO’s group-relative normalization. DPO-style variants typically preserve the offline, SFT-like compute profile in which the policy is updated by batched teacher-forced passes over preference data. IPO [96] changes the pairwise objective without materially changing this profile; ORPO [114] combines the SFT term with an odds-ratio preference penalty and removes the explicit reference model; SimPO [170] also removes the reference model and thus reduces memory and forward cost; KTO [87] uses binary desirability signals instead of paired preferences; and AlphaDPO [246] introduces adaptive margins while keeping the basic offline scaling behavior. Reward-aware preference objectives [223] partially reintroduce reward information into this family: they add quality-aware signals to the preference loss without returning to the full online actor–critic loop. Finally, emerging likelihood-oriented methods such as MaxRL [224] target reasoning-heavy settings with compute-indexed sampling objectives that interpolate between standard RL and maximum-likelihood optimization as additional sampling compute is allocated. Overall, these variants do not change the main systems distinction: online methods are dominated by rollout generation, critic-free methods remove VψV_ψ-dependent costs, and offline preference methods trade exploration for cheaper batched training. I-E Computational Analysis of RL-LLM We now analyze computational complexities of RL-LLMs. Mathematical derivations are detailed in Appendix B. Cat. Symbol Meaning and remarks Mathematical expression Work Cftok(M)C_f^tok(M) Forward-pass FLOPs per token for model M on a context of length S+TS+T (input prompt length + response length). The reference policy πref _ref is architecturally identical to the actor πθ _θ. Unembedding FLOPs per token are 2Vdπ2Vd_π (policy only). The reward model RφR_ adds a projection head applied to the final token only, contributing 2dRS+T 2d_RS+T FLOPs per token, whereas the value model VψV_ψ applies a projection head to every token, contributing 2dV2d_V FLOPs per token. Cftok(πθ)= 2Lπ[4dπ2+2dπdπf+(S+T)dπ]+2VdπCftok(πref)=Cftok(πθ)Cftok(Rφ)= 2LR[4dR2+2dRdRf+(S+T)dR]+2dRS+TCftok(Vψ)= 2LV[4dV2+2dVdVf+(S+T)dV]+2dV aligned &C_f^tok( _θ)&=&\ 2L_π [4d_π^2+2d_πd_ +(S+T)d_π ]+2Vd_π\\ &C_f^tok( _ref)&=&\ C_f^tok( _θ)\\ &C_f^tok(R_ )&=&\ 2L_R [4d_R^2+2d_Rd_Rf+(S+T)d_R ]+ 2d_RS+T\\ &C_f^tok(V_ψ)&=&\ 2L_V [4d_V^2+2d_Vd_Vf+(S+T)d_V ]+2d_V\\ aligned Cbtok(M)C_b^tok(M) Backward-pass FLOPs per token for model M on a context of length S+TS+T (input prompt length + response length). Cbtok(M)≈2⋅Cftok(M) aligned C_b^tok(M)\ \ ≈ 2· C_f^tok(M) aligned Cgenroll(πθ)C_gen^roll( _θ) Autoregressive generation FLOPs per rollout for generating T tokens (response length) from a context of length S (input prompt length) with KV caching (policy only). It is decomposed into prefill and decode. Prefill FLOPs are for processing the prompt of length S once and initializing the KV cache before autoregressive decoding begins. Prefill attention is assumed to be dense; as it is causal only half of the full M product are needed, giving ≈S2dπ≈ S^2d_π FLOPs for both QKTQK^T and for PVPV; O(S2)O(S^2) and O(Sd)O(Sd) account for lower-order contributions (respectively – softmax and scaling by 1/dh1/ d_h as well as RoPE and layer norms). Decode FLOPs are for generating the remaining T−1T-1 tokens autoregressively with KV caching. At decode step t, the model attends to context length S+tS+t. Cgenroll(πθ)=Cpfroll(πθ)+Cdecroll(πθ)Cpfroll(πθ)=2LπS(4dπ2+2dπdπf+Sdπ)+Lπ(O(S2)+O(Sdπ))+2VdπCdecroll(πθ)=∑t=1T−1[2Lπ(4dπ2+2dπdπf+2(S+t)dπ)+2Vdπ]+Lπ(O(S+t)+O(dπ)) aligned C_gen^roll( _θ)&=C_pf^roll( _θ)+C_dec^roll( _θ)\\ C_pf^roll( _θ)&=2L_πS\! (4d_π^2+2d_πd_ +Sd_π )\\ &+L_π (O(S^2)+O(Sd_π) )+2Vd_π\\ C_dec^roll( _θ)&= _t=1^T-1\! [2L_π\! (4d_π^2+2d_πd_ +2(S+t)d_π )+2Vd_π\! ]\\ &+L_π (O(S+t)+O(d_π) ) aligned Depth Df(M)D_f(M) Forward-pass depth of model M on a context of length S+TS+T, defined as the length of the critical path assuming unbounded parallelism. The policy πθ _θ includes an additional unembedding (output projection) step, which contributes a logdπ d_π term to the depth. For the reward model RφR_ and the critic VψV_ψ, the additional projection head contributes an extra logd d term to the depth. Df(πθ)=O(Lπ[logdπ+log(S+T)]+logdπ)Df(πref)=Df(πθ)Df(Rφ)=O(LR[logdR+log(S+T)]+logdR)Df(Vψ)=O(LV[logdV+log(S+T)]+logdV) aligned &D_f( _θ)&=&\ O (L_π [ d_π+ (S+T) ]+ d_π )\\ &D_f( _ref)&=&\ D_f( _θ)\\ &D_f(R_ )&=&\ O (L_R [ d_R+ (S+T) ]+ d_R )\\ &D_f(V_ψ)&=&\ O (L_V [ d_V+ (S+T) ]+ d_V ) aligned DfTP(M)D_f^TP\!(M) Forward-pass depth of model M with tensor parallelism degree PtP_t on a context of length S+TS+T. DfTP(M)=O(LM[logdMPt+log(S+T)]+logdMPt) aligned D_f^TP\!(M)\!&=O (L_M [ \! d_MP_t+ (S+T) ]+ d_MP_t ) aligned Db(M)D_b(M) Backward-pass depth of model M on a context of length S+TS+T. It equals the depth of the forward pass for the corresponding model M (which entails computing the gradients with respect to activations), plus the additional term O(log(B(S+T)))O( (B (S+T))) coming from computing the gradients with respect to model weights, where one has to accumulate gradients across batch (as these two branches can overlap, the expression given is a conservative bound). Db(M)=Df(M)+O(log(B(S+T))) aligned D_b(M)\ \ &=D_f(M)+O( (B (S+T))) aligned Dgen(πθ)D_gen( _θ) Autoregressive generation depth for generating T tokens given a prompt of length S. Prefill contributes one forward-pass depth (over context of length S); decode contributes one sequential forward-pass depth per generated token (i.e., over context of length S+tS+t for the t-th token). Dgen(πθ)=Df(πθ,S)+∑t=1T−1Df(πθ,S+t)=O(T⋅Df(πθ,S+T)) aligned D_gen( _θ)&=D_f( _θ;S)+ _t=1^T-1D_f( _θ;S+t)\\ &=O\! (T· D_f( _θ;S+T) ) aligned Memory |M| M Model parameter count of M (weights and embeddings). It always includes the parameters from query, key, value and output projections (4d24d^2), two FFN projections (2ddff2d_f), and the final logit computation (VdVd). For reward and critic models, there are also d parameters in the final head that outputs the score(s). |πθ|=Lπ(4dπ2+2dπdπf)+Vdπ,|πref|=|πθ||πac|=Ls(4dac2+2dacdac,f)+Vdac+dac|Rφ|=LR(4dR2+2dRdRf)+VdR+dR|Vψ|=LV(4dV2+2dVdVf)+VdV+dV aligned & _θ \ \ \ &=&\ L_π(4d_π^2+2d_πd_ )+Vd_π,\, _ref = _θ \\[-2.0pt] & _ac &=&\ L_s(4d_ac^2+2d_acd_ac,f)+Vd_ac+d_ac\\[-2.0pt] & R_ &=&\ L_R(4d_R^2+2d_Rd_Rf)+Vd_R+d_R\\[-2.0pt] & V_ψ &=&\ L_V(4d_V^2+2d_Vd_Vf)+Vd_V+d_V\\ aligned MActM_Act Training activation memory per processed token for model M. Under the leading-order activation model used throughout the paper, activations required for backpropagation scale with the number of layers and hidden width. Constant factors from Q/K/V tensors, FFN intermediates, normalization, residual, and other temporary tensors are suppressed. This assumes no activation checkpointing. MAct=Θ(LMdM)M_Act= (L_Md_M) MInfM_Inf Forward-only inference-buffer memory per processed token for model M. Because intermediate layer buffers can be reused across layers and the attention matrix is assumed not to be fully materialized, the leading-order buffer scales with hidden width rather than layer count. MInf(M)=Θ(dM)M_Inf(M)= (d_M) MKVM_KV KV-cache memory required for one rollout of prompt length S and response length T during generation. If all BKBK rollouts are generated concurrently, the peak KV-cache memory is BK⋅MKVBK· M_KV; if rollouts are generated sequentially across the K samples per prompt, the peak reduces accordingly but the generation depth increases. MKV=2(S+T)Lπdπ aligned M_KV&=2(S+T)L_πd_π aligned TABLE I: Computational building blocks used across the paper. We assume canonical multi-head attention and two-projection FFN. For computing FLOP costs, an addition and a multiplication count as two separate FLOPs (i.e., a dot product of vectors of dimensionalities d results in d+(d−1)≈2d+(d-1)≈ 2d FLOPs under this model). CftokC_f^tok, CbtokC_b^tok, CgenrollC_gen^roll are the costs of (respectively) forward pass, backward pass, and autoregressive generation; defining them per token and per rollout effectively clarifies the notation for the subsequent analyses. The explicit depth expressions count the selected GEMM reduction chains and parameter-gradient accumulation; they suppress the additional logarithmic reductions from normalization, attention/vocabulary softmax, sampling, and distributed collectives. For generation, we explicitly distinguish prefill and decode. Prefill processes the prompt once, while decode generates tokens sequentially with KV caching. This decomposition is important because rollout generation is the dominant online bottleneck and is not equivalent to a single teacher-forced forward pass over a sequence of length S+TS+T. We first derive the computational costs of building blocks, see Table I. Namely, each harnessed model comes with forward and backward pass costs (work, depth, and memory consumption). These costs are similar or identical (in the asymptotic sense) across different models, because all models are based on the transformer architecture. Framework PPO (Online) GRPO (Online) DPO (Offline) Work (Total FLOPs) Generation O(BK⋅Cgenroll(πθ))O(BK· C_gen^roll( _θ)) O(BK⋅Cgenroll(πθ))O(BK· C_gen^roll( _θ)) −- † Assessment O(BK(S+T)⋅[Cftok(πref)+Cftok(Rφ)+Cftok(Vψ)])O(BK(S+T)·[C_f^tok( _ref)+C_f^tok(R_ )+C_f^tok(V_ψ)]) O(BK(S+T)⋅[Cftok(πref)+Cftok(Rφ)])O(BK(S+T)·[C_f^tok( _ref)+C_f^tok(R_ )]) O(Bpairs(S+T)⋅[Cftok(πref)+Cftok(πθ)])O(B_pairs(S+T)·[C_f^tok( _ref)+C_f^tok( _θ)]) Training O(BK(S+T)⋅[Ctrtok(πθ)+Ctrtok(Vψ)])O(BK(S+T)·[C_tr^tok( _θ)+C_tr^tok(V_ψ)]) O(BK(S+T)⋅Ctrtok(πθ))O(BK(S+T)· C_tr^tok( _θ)) O(Bpairs(S+T)⋅Ctrtok(πθ))O(B_pairs(S+T)· C_tr^tok( _θ)) Depth (Critical Path Latency) Generation O(Dgen(πθ))O(D_gen( _θ)) O(Dgen(πθ))O(D_gen( _θ)) −- Assessment O(maxDf(πref),Df(Rφ),Df(Vψ))O( \D_f( _ref),\,D_f(R_ ),\,D_f(V_ψ)\) ‡ O(maxDf(πref),Df(Rφ))O( \D_f( _ref),\,D_f(R_ )\) ‡ O(maxDf(πθ),Df(πref))O( \D_f( _θ),\,D_f( _ref)\) Training O(maxDtr(πθ),Dtr(Vψ))O( \D_tr( _θ),\,D_tr(V_ψ)\) O(Dtr(πθ))O(D_tr( _θ)) O(Dtr(πθ))O(D_tr( _θ)) Memory Cost Generation O(|πθ|+BK⋅MKV)O(| _θ|+BK· M_KV) O(|πθ|+BK⋅MKV)O(| _θ|+BK· M_KV) −- Assessment O(|πref|+|Rφ|+|Vψ|+BK(S+T)MInf)O(| _ref|+|R_ |+|V_ψ|+BK(S+T)M_Inf) O(|πref|+|Rφ|+BK(S+T)MInf)O(| _ref|+|R_ |+BK(S+T)M_Inf) O(|πθ|+|πref|+Bpairs(S+T)MAct)O(| _θ|+| _ref|+B_pairs(S+T)M_Act) Training O(|πθ|+|Vψ|+BK(S+T)⋅MAct)O(| _θ|+|V_ψ|+BK(S+T)· M_Act) O(|πθ|+BK(S+T)⋅MAct)O(| _θ|+BK(S+T)· M_Act) O(|πθ|+Bpairs(S+T)⋅MAct)O(| _θ|+B_pairs(S+T)· M_Act) TABLE IV: Asymptotic Work, Depth, and Memory analysis of PPO, GRPO, and DPO. S is prompt length, T is response length. We denote parameter counts by |⋅||·|. “tr” subscript: For conciseness, for training-stage model invocations, we define Ctrtok(M):=Cftok(M)+Cbtok(M)≈3Cftok(M)C_tr^tok(M):=C_f^tok(M)+C_b^tok(M)≈ 3C_f^tok(M) and Dtr(M):=Df(M)+Db(M)D_tr(M):=D_f(M)+D_b(M), because a parameter update requires a forward pass followed by backpropagation. Note on Work: CftokC_f^tok and CbtokC_b^tok are the cost of forward and backward passes per token and CgenrollC_gen^roll is the autoregressive generation cost per rollout; these are derived in Table I. Work scales with total tokens (S+T)(S+T). Note on Depth: Generation is depth-bound by T (sequential), while in Assessment/Training there is no outer linear T-step autoregressive chain. † DPO is offline; generation occurs prior to the training loop. ‡ Max depth assumes parallel (disaggregated) execution; sequential co-located execution is additive. MKVM_KV is Key-Value cache memory; MActM_Act is activation memory (training); MInfM_Inf is inference buffer (assessment). Note that πref _ref never requires MActM_Act. Next, we obtain the complexities for PPO, GRPO, and DPO, see Table IV. Overall, the work–depth expressions make the ordering precise. Among the online methods, PPO is the most expensive because its per-iteration work contains all major terms, and its training memory also includes both trainable models. GRPO removes the critic VψV_ψ, so it eliminates the additional Θ(BK(S+T)[Cftok(Vψ)+Cbtok(Vψ)]) (BK(S+T)[C_f^tok(V_ψ)+C_b^tok(V_ψ)]) work and the corresponding Θ(|Vψ|) (|V_ψ|) model-state memory, but it retains the same dominant online bottleneck as PPO, namely the autoregressive generation depth Dgen=Θ(T⋅Df(πθ))D_gen= (T· D_f( _θ)). By contrast, DPO removes online rollout generation and explicit reward/critic evaluation from the inner loop, leaving only batched forward/backward passes over preference pairs. Thus, unlike PPO and GRPO, DPO has no T-step autoregressive term in its inner-loop depth, which explains why its systems profile is much closer to standard SFT. The trade-off is algorithmic: because DPO optimizes over a fixed preference dataset rather than fresh online rollouts, it cannot directly explore behavior outside that dataset. Later in Section V, we also present detailed algorithmic parallelism-focused specifications (Algorithms 1, 2, 3, and 4) that explicitly annotate most relevant parallel execution. I-F Key Insights & Takeaways We summarize key takeaways. Autoregressive generation imposes an irreducible per-rollout dependency. For PPO and GRPO, each rollout is generated token by token, yielding a per-trajectory depth of O(TDf(πθ,S+T))O\! (TD_f( _θ;S+T) ). Parallelizing Assessment or Training cannot remove this sequential dependency within an individual trajectory. Consequently, efficient decoding kernels remain central to reducing rollout latency and increasing generation throughput. Generation is not necessarily the end-to-end system bottleneck. The sequential depth of an individual rollout does not imply that the entire Generation stage must dominate iteration time. Through inter-stage fusion, completed samples or rollout subbatches can be streamed into Assessment while longer trajectories or later rollout waves are still being generated. This overlap is useful when completion lengths are skewed or when finite generation capacity forces the rollout batch to be processed in multiple waves. It trades additional memory (e.g., through live KV-cache and communication buffers) for reduced pipeline idle time. Auxiliary models define the systems profile. PPO uses πref,Rφ,Vψ\ _ref,R_ ,V_ψ\, GRPO removes VψV_ψ, and DPO removes both RφR_ and VψV_ψ from the inner loop. Hence the PPO–GRPO gap is precisely the critic cost Θ(BK(S+T)[Cftok(Vψ)+Cbtok(Vψ)]) \! (BK(S+T)[C_f^tok(V_ψ)+C_b^tok(V_ψ)] ) plus critic model-state memory. The GRPO–DPO gap is more structural: DPO also removes online rollout generation from the optimization loop, eliminating the T-step generation depth term. Frozen models are placement opportunities. The reward and reference models are forward-only. They require no gradients or optimizer state, so they can be replicated, quantized, offloaded, or served as independent inference services. By contrast, trainable models πθ _θ and VψV_ψ dominate memory through activations, gradients, and optimizer state. Reward granularity is a compute–credit-assignment trade-off. Outcome rewards are cheap: one scalar per completion. Process rewards can improve reasoning supervision but may require step-level annotations, verifier calls, or search, turning Assessment from one batched forward pass into a much heavier verification workload. DPO is SFT-like, but not free. DPO still performs WDPO=Θ(Bpairs(S+T)[Cftok(πθ)+Cftok(πref)+Cbtok(πθ)])W_DPO= \! (B_pairs(S+T)[C_f^tok( _θ)+C_f^tok( _ref)+C_b^tok( _θ)] ), but these are teacher-forced passes over static preference pairs. Its systems advantage is therefore the removal of online generation and reward/critic inference; its algorithmic limitation is that it cannot directly explore beyond the support of the preference dataset. Fig. 3: Mathematical notation and equations for the forward and backward passes of each decoder block. Further details on computational aspects are in Figures 4-5 while parallelization details are in Figures 6-7. No AI was used to conceive or to draw the figure. Fig. 4: Intra-model execution details (forward pass) for each RLM model execution. The design details are based on Llama-3. Further details on parallelization and the corresponding taxonomy are provided in Figures 6-7. No AI was used to conceive or to draw the figure. Fig. 5: Intra-model execution details (backward pass) for each RLM model execution. Legend is provided in Figure 4. The design details are based on Llama-3. Further details on parallelization and the corresponding taxonomy are provided in Figures 6-7. No AI was used to conceive or to draw the figure. Fig. 6: Intra-model parallelism (data, pipeline, tensor, and expert parallelism) details for each RLM model execution; legend is provided in Figure 4. No AI was used to conceive or to draw the figure. Fig. 7: Intra-model parallelism (sequence & context parallelism, as well as prefill vs. decode and flash attention) details for each RLM model execution; legend is provided in Figure 4. No AI was used to conceive or to draw the figure. I Intra-Model Parallelism Efficient training and inference for each individual model in the RL-LLM pipeline is essential. Intra-model parallelism denotes established techniques that allow a single model to be trained or served across multiple devices by partitioning its parameters, activations, or computation graph. The main families are: (i) data parallelism, (i) tensor/operator parallelism, (i) sequence and context parallelism, (iv) pipeline parallelism, (v) expert parallelism, and (vi) memory-centric optimizations such as optimizer sharding and activation checkpointing. These techniques are typically combined into hybrid schemes (e.g., ZeRO-style sharded data parallelism plus tensor parallelism) to balance memory footprint, communication cost, and compute utilization. As these techniques are well-known [146, 219, 120, 176, 132, 160, 141], we summarize them and instead focus on implications for RL–LLM pipelines. We detail the computational aspects of the intra-model execution Figure 3 (mathematical details and flow diagrams), Figure 4 (forward pass), Figure 5 (backward pass), and in Figures 6-7 (details on intra-model parallelism and taxonomy). I-A Data Parallelism Data parallelism (DP) replicates the model across N devices and shards the input batch across replicas. Each replica performs a local forward and backward pass on its shard, followed by a gradient synchronization step (typically an all-reduce [63, 227]) to keep parameters identical across devices. This strategy is conceptually simple and scales throughput almost linearly when the model fits into a single device and communication is not a bottleneck. Naive DP quickly becomes memory-limited for very large models because each device must store a full copy of parameters, gradients, and optimizer state. Hence, DP is most effective when combined with memory-centric optimizations (e.g., ZeRO/FSDP [201]) so that parameters, gradients, and optimizer state are partially sharded rather than fully replicated. Moreover, the all-reduce step can become a major source of overhead as the number of devices grows or when interconnect bandwidth is limited. Here, modern systems apply several communication optimizations, such as overlap of computation and communication, gradient bucketing, and gradient accumulation [146]. I-A1 Implications for RL-LLM Pipelines In RL-LLM pipelines, a crucial point for DP is whether a given model or stage requires gradients or is forward-only. Forward-only models. If a model is not updated in a given stage, DP reduces to independent batched inference with replicated weights and scales almost linearly in the number of devices until limited by input or network I/O. This is the typical regime for the actor during Generation (sampling rollouts), the actor during Assessment when only forward log-probabilities and KL terms are needed, the reward model during Assessment, and the reference model during Assessment. OpenRLHF, for example, uses Ray and vLLM to run actor, reward, and reference as large-scale batched inference services, separate from training loops [117]. Here, DP is highly effective and typically the first choice. Trainable models. Here, DP must all-reduce gradients across workers and thus benefits from sharded data-parallel variants such as ZeRO-2/3 or FSDP. This applies to the actor in the Training stage (PPO/GRPO/DPO updates) and to the critic in actor–critic methods. Frameworks such as OpenRLHF explicitly apply ZeRO-3/FSDP-style sharded data parallelism to actor and critic training while keeping forward-only models replicated [117, 201]. In RLM pipelines, a common pattern is therefore: DP + ZeRO for trainable components; pure DP for forward-only components. I-B Tensor Parallelism Tensor (or operator) parallelism (TP) partitions individual layers across devices by sharding their parameters along hidden dimensions, so that each device computes only a slice of the layer. Synchronization via all-reduce or all-gather then reconciles partial results. In Transformer blocks, TP is commonly applied to both feed-forward networks (FFNs) and to the multi-head attention (MHA). For FFN, the first linear projection (e.g., XAXA) is split column-wise across devices, and the second projection (e.g., YBYB) is split row-wise. Each device holds a shard of A or B and computes a partial result. For MHA, query/key/value projections are sharded column-wise, assigning subsets of heads to devices. The output projection is sharded row-wise, mirroring the FFN pattern. TP is typically combined with DP (and sometimes pipeline parallelism) to form 2D, 3D, or 5D parallelism configurations capable of training trillion parameter models [219, 176]. Megatron-LM and related systems implement TP using communication operators that behave differently in forward and backward passes, e.g., an operator f that is identity in the forward pass and all-reduce in the backward, and an operator g that is all-reduce in the forward and identity in the backward [219]. This enables communication-computation overlap and reduces idle time. I-B1 Implications for RL–LLM pipelines Tensor parallelism is essential when a single model does not fit on one device even after applying memory optimizations (e.g., ZeRO, mixed precision, quantization). However, for many RLHF workloads, inference dominates runtime. For example, OpenRLHF reports that rollout generation and forward Assessment account for more than 90% of wall-clock time in RLHF loops [117]. Here, TP can hurt throughput when not strictly necessary. For instance, Megatron-style TP introduces two all-reduces per Transformer block during inference and four during training [219, 176]. When a model (actor, reward, or reference) fits on a single device, empirical studies from vLLM and DeepSpeed show that enabling TP reduces total tokens-per-second because collective communication becomes the bottleneck [230, 201]. AMD’s Qwen2-7B RLHF experiments on MI300X (192 GB) similarly report that disabling TP gives 2.3–4.3× higher rollout throughput than when sharding with TP by 2 or 4 for the same model, as communication is negligible in the single-device setting [263]. The resulting rule-of-thumb for using TP in RLM pipelines is: (1) use TP only when a model does not fit on a single device even with ZeRO/FSDP and mixed precision; (2) prefer no TP for inference-heavy stages (Generation, forward-only Assessment) whenever memory allows; and (3) tolerate TP overhead more readily in training stages, where gradient storage already inflates memory by ≈3×≈ 3× and TP may be necessary to fit the model [219]. In practice, large RLM deployments often use TP inside high-bandwidth device groups and combine it with pipeline and data parallelism across nodes [176]. I-C Sequence Parallelism TP shards heavyweight projections but leaves token-wise operations such as layer normalization and dropout replicated on each TP shard, leading to duplicated activations and higher memory use, especially for long sequences. Sequence parallelism (SP) [132] addresses this by partitioning activations along the sequence dimension. Because layer norm and dropout operate independently per token, their activations can be split across devices without changing the model’s semantics. This reduces per-device activation memory at essentially no additional computation cost. Sequence parallelism has been adopted in large-scale stacks such as Megatron-LM, DeepSpeed, and ColossalAI for long-context or high-batch training [132, 219, 125, 147], where activation memory is a primary limiter. I-C1 Implications for RL-LLM Pipelines Sequence parallelism is primarily beneficial in training stages where activations must be stored or recomputed for every token, such as policy/critic updates over long responses and reasoning traces. Sharding activations by sequence length reduces per-GPU memory and allows larger T or batch sizes under fixed memory budgets. Because SP reuses the same collective bandwidth as TP (by switching all-reduce to reduce-scatter/all-gather pairs), its net communication volume stays similar, with modest changes in latency due to additional synchronization points [132, 219]. I-D Context Parallelism Context parallelism extends sequence-based partitioning further by maintaining the sequence split throughout the entire Transformer layer, including attention and FFN blocks [256]. Most operations remain token-wise and are naturally compatible; the main challenge is attention, which requires access to keys and values from all sequence partitions. Recent implementations address this using Ring Attention [160], which pipelines key/value communication in a ring topology to reduce latency and memory pressure. CP enables training and inference with very long contexts (hundreds of thousands or millions of tokens) by combining tensor, sequence, and context partitioning without replicating activations or weights unnecessarily [256]. I-D1 Implications for RL-LLM Pipelines Context parallelism is most useful when RLM training is dominated by long prompt–trajectory contexts (e.g., long CoT traces, tool histories, verifier outputs, etc.). In this regime, the bottleneck is often not only model size but also attention and activation memory over the full context. CP directly targets this axis by partitioning the sequence dimension across devices while preserving exact attention through cross-device KV exchange [160, 125, 183]. This yields concrete implications for RLMs. First, CP can make long-CoT rollouts feasible when the actor’s context length exceeds single-device memory. Second, in verifier- or tool-augmented RL, CP allows prompts to include larger execution traces, retrieved evidence, and interaction histories without truncating the reasoning state. However, CP does not remove the autoregressive dependence of online Generation: it reduces per-device context memory and attention work, but tokens are still sampled sequentially. Thus, CP is primarily a feasibility and long-context scaling mechanism; online rollout latency still requires complementary methods such as TP, efficient KV-cache exchange, continuous batching, or stage-level overlap. Recent long-context systems validate this direction: Ring Attention overlaps KV-block communication with blockwise attention, DeepSpeed-Ulysses uses sequence partitioning and all-to-all communication for long-sequence training, and CP-style inference systems report near-linear scaling for million-token prefill workloads [160, 125, 256]. I-E Pipeline Parallelism Pipeline parallelism (P) partitions the layers of a model into stages, each placed on a different device. Mini-batches are further split into micro-batches that flow through the pipeline stages. This allows models that are too deep to fit on a single device to be trained, at the cost of pipeline bubbles (periods where some stages are idle). P is often combined with DP and TP, yielding 3D/5D parallelism that supports extremely large models [176]. Numerous pipeline scheduling strategies have been proposed, examples include AFAB [120] (all-forward-all-backward, all micro-batches complete forward passes before any backward pass starts – simple but bubble-heavy), 1F1B [175] (forward and backward passes interleave, improving memory efficiency and reducing bubbles), Interleaved 1F1B [176] (devices host multiple non-contiguous pipeline stages to further overlap computation and communication), Zero Bubble [197] (decomposes the backward pass into B-steps (gradients w.r.t. activations) and W-steps (gradients w.r.t. weights) scheduling them separately to eliminate bubbles without increasing peak memory), and DualPipe [156] (runs two pipelines in opposite directions across the same devices, overlapping forward and backward streams to improve utilization). I-E1 Implications for RL-LLM Pipelines The relevance of P to RL-LLM heavily depends on the considered stage. In Generation, the actor performs autoregressive decoding, so tokens must traverse all pipeline stages sequentially; thus, P helps fit large actors but does not remove the O(T)O(T) critical path, and naive schedules can suffer from large pipeline bubbles and poor utilization [251, 262]. By contrast, Assessment and Training operate on complete sequences and therefore benefit more directly from standard micro-batched P schedules such as GPipe- or 1F1B-style execution. This heterogeneity makes mode-aware P especially important in RLHF systems. The best sharding and scheduling strategy for low-latency Generation need not match the best one for high-throughput Training. Recent systems therefore decouple configurations across stages and overlap different RL iterations, e.g., by letting one batch generate while another is being assessed or trained [262, 117, 261]. Such inter-batch pipelining can substantially reduce idle time. In large reasoning-oriented systems, this design is often combined with explicit communication–computation overlap to reduce pipeline bubbles further [156]. P also interacts with optimization stability. Interleaved or asynchronous schedules may introduce weight staleness, where forward and backward passes for a micro-batch observe different parameter versions. In RLHF, where gradients are already noisy, this can destabilize training unless explicit versioning or consistency mechanisms are used [251, 262]. Finally, P strengthens the case for disaggregated placement: placing actor, critic, reward, and reference on separate device groups allows each component to use a P configuration matched to its role, instead of forcing a single compromise configuration for the whole pipeline [251, 117]. I-F Expert Parallelism Expert parallelism (EP) is the standard way to scale Mixture-of-Experts (MoE) models by distributing whole experts across devices. Unlike DP, which replicates the full model, or TP, which shards individual operators, EP assigns different experts to different GPUs or nodes. Since only a small subset of experts is activated per token, EP enables very large model capacity with much smaller per-token compute than an equally sized dense model [74, 156]. Modern MoE designs also separate shared and routed experts. Shared experts are activated for all tokens and capture common linguistic features, while routed experts specialize in narrower domains such as mathematics or code. In practice, EP is rarely used alone; it is typically combined with TP, P, and DP in hybrid multi-dimensional layouts [74, 156]. Formally, an MoE layer contains E experts and a router G(x)G(x) that selects the top-k experts for each token representation x, with k≪Ek E. The output can be written as y=∑i=1kgi(x)fi(x)y= _i=1^kg_i(x)f_i(x), where fif_i is the transformation of the i-th selected expert and gi(x)g_i(x) is its routing weight. EP exploits this sparsity by storing experts on different devices and executing only the selected ones [74]. EP’s main systems cost is communication. Each MoE layer typically requires two all-to-all phases: dispatch, which sends token activations to the devices hosting the selected experts, and combine, which returns expert outputs to their originating ranks. As expert count and cluster size increase, this communication can dominate runtime, so practical EP deployments rely on hierarchical collectives, locality-aware routing, and communication-computation overlap [264]. I-F1 Implications for RL-LLM Pipelines EP can adapt a very large model while activating and updating only a small subset of parameters per token. This is particularly valuable for long reasoning trajectories, where dense models would make both rollout generation and policy updates prohibitively expensive [156, 74]. However, RL post-training makes routing harder. As the policy evolves, the token distribution shifts, so expert loads can become highly imbalanced. Load balancing schemes can help hardware efficiency, but they may also blur expert specialization by forcing artificially uniform routing [108]. For this reason, recent reasoning-oriented MoE systems increasingly prefer auxiliary-loss-free or bias-based balancing mechanisms that preserve specialization while correcting large load skew at the batch level [156, 108]. More broadly, EP in RLHF requires joint optimization of routing quality, communication cost, and systems balance: if routing is good but overloaded experts are poorly placed, all-to-all communication and straggler effects can erase the theoretical gains from sparse activation [264, 178]. EP also interacts strongly with RL framework design. Grouped sampling methods such as GRPO may create many structurally similar trajectories for the same prompt, which can stress the same experts simultaneously. Thus, efficient RL-MoE training often requires combining EP with dynamic load-balancing policies [156, 117, 178]. I-G Complexity Analysis We now analyze how intra-model parallelism changes the work, depth, and memory of the Transformer invocations inside RL-LLM pipelines; detailed derivations are in Appendix B-C. The same local formulas apply to the actor in Generation, the reward/reference/critic models in Assessment, and the policy or critic in Training; the RL-specific consequences come from whether the invocation is autoregressive, forward-only, or trainable. Generation uses the actor in an autoregressive loop, so reducing the depth of one model invocation helps but cannot remove the outer T-step dependence. Assessment is teacher-forced and forward-only for frozen models, so it is mainly a batched-inference and memory-placement problem. Training requires backward passes, activations, gradients, and optimizer state, so memory sharding becomes central. Tables V–VIII therefore should be read as architecture-independent scaling laws for the individual model calls that compose the RL-LLM loop, not as hardware-calibrated throughput predictions. A logical model invocation may be distributed across N participating ranks according to one or more parallelism dimensions. Depending on the parallelization strategy, each rank may operate on a partition of the input, parameters, layers, activations, or experts while other components remain replicated. We report both per-rank costs, which characterize local device pressure, and global costs, which aggregate costs across all N ranks participating in the complete logical model invocation. Global work is the sum of arithmetic FLOPs executed across these ranks, including replicated computation on every rank where it occurs. Global memory analogously sums resident state across all participating ranks. Depth denotes the critical path of the complete distributed execution. Throughout Tables V–VIII, we use an idealized arithmetic work–depth–memory model intended to expose scaling laws rather than predict hardware runtime. Unless stated otherwise, we exclude communication, synchronization, kernel-launch and scheduling overheads, finite-device utilization, load imbalance, and temporary communication workspaces. Pipeline parallelism assumes an approximately uniform partition of the L layers and does not model the number of microbatches, pipeline schedules, fill/drain bubbles, or inter-stage activation transfers. Training activation memory is represented by the leading-order term B(S+T)LdB(S+T)Ld, suppressing constant-factor storage for Q/K/V tensors, FFN intermediates, normalization temporaries, and other implementation-specific buffers; activation checkpointing is treated separately. The Adam model-state expressions count parameters, gradients, and first and second moments using a common element-size abstraction, omitting precision-specific byte factors and transient buffers. The TP activation terms use an idealized shardable-activation model; replicated token-wise components are suppressed. Attention is assumed not to materialize the full attention matrix, as in FlashAttention-style execution. For MoE models, the tables retain the dominant shared-Transformer and expert-FFN terms; router projection, softmax/top-k, auxiliary load-balancing operations, and their relatively small parameter state are omitted. Finally, the isolated EP row assumes that only expert FFNs are partitioned across PeP_e ranks while shared dense computation is replicated; the combined 5D row instead assumes a coupled execution in which EP ranks also process disjoint token/batch shards for the shared path, so dense arithmetic is not redundantly executed across the EP dimension. Each parallelism strategy partitions a different dimension. DP splits the batch, reducing per-group work and activation memory but leaving the critical path unchanged. P splits layers, reducing per-stage work, memory, and architectural depth by replacing L with L/PpL/P_p, though realized runtime also depends on microbatch bubbles. TP splits hidden-dimensional operators, reducing per-device GEMM work and replacing hidden-dimension depth terms by log(d/Pt) (d/P_t). CP splits the sequence dimension, reducing context-dependent attention and activation memory, which is crucial for long reasoning traces. EP splits MoE experts, reducing per-device expert memory and compute while leaving dense attention largely unchanged. Finally, 3D/5D parallelism combines data, pipeline, and tensor sharding, giving the strongest per-device memory reduction but also the most communication and scheduling constraints. For RL-LLMs, this means DP is usually best for throughput-oriented batched Generation or Assessment when models fit per device; TP/P are needed for large actors, critics, or low-latency single invocations; CP is most useful for long-context reasoning; and 3D/5D/ZeRO-style sharding is most important during Training, where optimizer state and activations dominate memory. Note that pipeline parallelism does not reduce the global depth layer-wise dependency from L to L/PpL/P_p; the L/PpL/P_p factor appears only in the per-rank depth. This is because, for global end-to-end critical path, the microbatch still passes through all PpP_p pipeline stages, so it traverses all L layers. Context parallelism reduces the number of query positions and stored activations per device, but each query still depends on the global key/value context. Therefore, under the work–depth model, CP does not replace the global attention reduction length S by S/PcS/P_c; its principal benefits are reduced per-rank work and memory. Distributed attention additionally incurs communication-round dependencies, which are outside the present FLOP-based depth abstraction. I-G1 ZeRO & FSDP The memory expressions in Tables V–VIII use a simplified Adam model-state factor consisting of parameters, gradients, and first and second moments. ZeRO [201] and FSDP [277] refine this term without changing the layerwise Transformer computation. If P denotes the parameter count of the trainable model and PzP_z the sharding degree, then standard data parallelism stores approximately 4P4P model-state elements per replica. ZeRO-1 shards only optimizer states, giving P+P+2P/PzP+P+2P/P_z; ZeRO-2 shards optimizer states and gradients, giving P+3P/PzP+3P/P_z; and ZeRO-3/FSDP shards parameters, gradients, and optimizer states, giving approximately 4P/Pz4P/P_z, up to transient all-gather buffers. I-G2 Activation Checkpointing The activation terms in the tables correspond to stored training activations without checkpointing. Activation checkpointing reduces this term by storing only a subset of intermediate activations and recomputing the missing ones during backpropagation. In our notation, this replaces MActM_Act by κckptMAct _ckptM_Act for some 0<κckpt<10< _ckpt<1, while increasing training work and depth by the extra forward recomputation required during the backward pass. This trade-off is most relevant for actor and critic training, especially for long reasoning trajectories. Parallelism Work Depth Memory None† O(BmL(d2+md))O\! (BmL(d^2+md) ) O(L[logd+logm]+log(Bm))O\! (L[ d+ m]+ (Bm) ) O(Ld2+Vd+BmLd)O\! (Ld^2+Vd+BmLd ) Data O(BmL(d2+md)Pd)O\! ( BmL(d^2+md)P_d ) O(L[logd+logm]+log(BPdm))O\! (L[ d+ m]+ \! ( BP_dm ) ) O(Ld2+Vd+BmLdPd)O\! (Ld^2+Vd+ BmLdP_d ) Pipeline∗ O(BmL(d2+md)Pp)O\! ( BmL(d^2+md)P_p ) O(LPp[logd+logm]+log(Bm))O\! ( LP_p[ d+ m]+ (Bm) ) O(Ld2Pp+Vd+BmLdPp)O\! ( Ld^2P_p+Vd+ BmLdP_p ) Tensor O(BmL(d2+md)Pt)O\! ( BmL(d^2+md)P_t ) O(L[logdPt+logm]+log(Bm))O\! (L\! [ dP_t+ m ]+ (Bm) ) O(Ld2+VdPt+BmLdPt)O\! ( Ld^2+VdP_t+ BmLdP_t ) Context O(BmL(d2+md)Pc)O\! ( BmL(d^2+md)P_c ) O(L[logd+logm]+log(BmPc))O\! (L[ d+ m]+ \! ( BmP_c ) ) O(Ld2+Vd+BmLdPc)O\! (Ld^2+Vd+ BmLdP_c ) Expert O(BmL[(d2+md)+EaddePe])O\! (BmL [(d^2+md)+ E_ad_eP_e ] ) O(L[logd+logm]+log(Bm))O\! (L[ d+ m]+ (Bm) ) O(L[d2+EddePe]+Vd+BmLd)O\! (L [d^2+ Edd_eP_e ]+Vd+BmLd ) 3D‡ O(BmL(d2+md)PdPpPt)O\! ( BmL(d^2+md)P_dP_pP_t ) O(LPp[logdPt+logm]+log(BPdm))O\! ( LP_p [ dP_t+ m ]+ \! ( BP_dm ) ) O(Ld2+VdPpPt+BmLdPdPpPt)O\! ( Ld^2+VdP_pP_t+ BmLdP_dP_pP_t ) 5D‡ O(BmLPdPePcPpPt[d2+md+Eadde])O\! ( BmLP_dP_eP_cP_pP_t [d^2+md+E_ad_e ] ) O(LPp[logdPt+logm]+log(BmPdPePc))O\! ( LP_p [ dP_t+ m ]+ \! ( BmP_dP_eP_c ) ) O(Ld2PpPt+LEddePpPtPe+VdPt+BmLdPdPePcPpPt)O\! ( Ld^2P_pP_t+ LEdd_eP_pP_tP_e+ VdP_t+ BmLdP_dP_eP_cP_pP_t ) TABLE V: Complexity Analysis (Per rank). Asymptotic work, depth, and memory for a single Transformer training iteration (forward and backward pass), expressed in Big-O notation. For clarity, we use m:=S+Tm:=S+T. The work expressions use the regime where dff=O(d)d_f=O(d). †: Baseline configuration without parallelism. ∗For P, the VdVd memory term denotes the embedding/output state resident on boundary pipeline ranks; interior ranks need not store this state. Thus, this term represents the boundary/peak per-rank footprint rather than replication across all PpP_p ranks. ‡: 3D and 5D parallelism combine data, pipeline, tensor (3D), context, and expert (5D) parallelism, with PdPpPt=NP_dP_pP_t=N (3D) and PdPpPtPcPe=NP_dP_pP_tP_cP_e=N (5D) total devices. For the combined 5D configuration, we assume the EP dimension also partitions source-token ownership for shared-layer computation. Training memory assumes Adam without activation checkpointing. Parallelism Work Depth Memory None O(BmL(d2+md))O\! (BmL(d^2+md) ) O(L[logd+logm]+log(Bm))O\! (L[ d+ m]+ (Bm) ) O(Ld2+Vd+BmLd)O\! (Ld^2+Vd+BmLd ) Data O(BmL(d2+md))O\! (BmL(d^2+md) ) O(L[logd+logm]+log(Bm))O\! (L[ d+ m]+ (Bm) ) O(Pd[Ld2+Vd]+BmLd)O\! (P_d[Ld^2+Vd]+BmLd ) Pipeline O(BmL(d2+md))O\! (BmL(d^2+md) ) O(L[logd+logm]+log(Bm))O\! (L[ d+ m]+ (Bm) ) O(Ld2+Vd+BmLd)O\! (Ld^2+Vd+BmLd ) Tensor O(BmL(d2+md))O\! (BmL(d^2+md) ) O(L[log(d)+logm]+log(Bm))O\! (L\! [ d+ m ]+ (Bm) ) O(Ld2+Vd+BmLd)O\! (Ld^2+Vd+BmLd ) Context O(BmL(d2+md))O\! (BmL(d^2+md) ) O(L[logd+logm]+log(Bm))O\! (L[ d+ m]+ (Bm) ) O(Pc[Ld2+Vd]+BmLd)O\! (P_c[Ld^2+Vd]+BmLd ) Expert O(BmL[Pe(d2+md)+Eadde])O\! (BmL [P_e(d^2+md)+E_ad_e ] ) O(L[logd+logm]+log(Bm))O\! (L[ d+ m]+ (Bm) ) O(Pe[Ld2+Vd+BmLd]+LEdde)O\! (P_e[Ld^2+Vd+BmLd]+LEdd_e ) 3D O(BmL(d2+md))O\! (BmL(d^2+md) ) O(L[log(d)+logm]+log(Bm))O\! (L [ d+ m ]+ (Bm) ) O(Pd[Ld2+Vd]+BmLd)O\! (P_d[Ld^2+Vd]+BmLd ) 5D O(BmL[d2+md+Eadde])O\! (BmL [d^2+md+E_ad_e ] ) O(L[log(d)+logm]+log(Bm))O\! (L [ d+ m ]+ (Bm) ) O(PdPcPe[Ld2+Vd]+PdPcLEdde+BmLd)O\! (P_dP_cP_e[Ld^2+Vd]+P_dP_cLEdd_e+BmLd ) TABLE VI: Complexity Analysis (Global). Asymptotic work, depth, and memory for a single Transformer training iteration (forward & backward pass). For clarity, we use m:=S+Tm:=S+T. The work expressions use the regime where dff=O(d)d_f=O(d). Parallelism Work Depth Memory None 6BmL(4d2+2ddff+md)6BmL (4d^2+2d_f+md ) 2Llog(d4dffmh)+log(Bm)2L \! ( d^4d_fmh )+ (Bm) L(16d2+8ddff)+4Vd+BmLdL(16d^2+8d_f)+4Vd+BmLd Data 6BPdmL(4d2+2ddff+md)6 BP_dmL (4d^2+2d_f+md ) 2Llog(d4dffmh)+log(BPdm)2L \! ( d^4d_fmh )+ \! ( BP_dm ) L(16d2+8ddff)+4Vd+BPdmLdL(16d^2+8d_f)+4Vd+ BP_dmLd Pipeline 6BmLPp(4d2+2ddff+md)6Bm LP_p (4d^2+2d_f+md ) 2LPplog(d4dffmh)+log(Bm)2 LP_p \! ( d^4d_fmh )+ (Bm) LPp(16d2+8ddff)+4Vd+BmLPpd LP_p(16d^2+8d_f)+4Vd+Bm LP_pd Tensor 6BmL(4d2+2ddff+mdPt)6BmL ( 4d^2+2d_f+mdP_t ) 2Llog(d4dffmPt2h)+log(Bm)2L \! ( d^4d_fmP_t^2h )+ (Bm) L(16d2+8ddff)Pt+4VdPt+BmLdPt L(16d^2+8d_f)P_t+ 4VdP_t+ BmLdP_t Context 6BS+TPcL(4d2+2ddff+md)6B S+TP_cL (4d^2+2d_f+md ) 2Llog(d4dffmh)+log(BmPc)2L \! ( d^4d_fmh )+ \! ( BmP_c ) L(16d2+8ddff)+4Vd+BS+TPcLdL(16d^2+8d_f)+4Vd+B S+TP_cLd Expert 6BmL(4d2+2EaddePe+md)6BmL (4d^2+ 2E_ad_eP_e+md ) 2Llog(d4demh)+log(Bm)2L \! ( d^4d_emh )+ (Bm) L(16d2+8EddePe)+4Vd+BmLdL (16d^2+ 8Edd_eP_e )+4Vd+BmLd 3D 6BPdmLPp4d2+2ddff+mdPt6 BP_dm LP_p 4d^2+2d_f+mdP_t 2LPplog(d4dffmPt2h)+log(BPdm)2 LP_p \! ( d^4d_fmP_t^2h )+ \! ( BP_dm ) L(16d2+8ddff)PpPt+4VdPt+BmLdPdPpPt L(16d^2+8d_f)P_pP_t+ 4VdP_t+ BmLdP_dP_pP_t 5D 6BPdPemPcLPp(4d2+2Eadde+mdPt)6 BP_dP_e mP_c LP_p ( 4d^2+2E_ad_e+mdP_t ) 2LPplog(d4demPt2h)+log(BmPdPePc)2 LP_p \! ( d^4d_emP_t^2h )+ \! ( BmP_dP_eP_c ) 16Ld2PpPt+8LEddePpPtPe+4VdPt+BmLdPdPePcPpPt 16Ld^2P_pP_t+ 8LEdd_eP_pP_tP_e+ 4VdP_t+ BmLdP_dP_eP_cP_pP_t TABLE VII: Complexity Analysis (Per rank). Explicit leading-order costs under our simplified Transformer cost model, for a single Transformer training iteration (forward and backward pass), including explicit constant factors. For clarity, we use m:=S+Tm:=S+T. Parallelism Work Depth Memory None 6BmL(4d2+2ddff+md)6BmL (4d^2+2d_f+md ) 2Llog(d4dffm/h)+log(Bm)2L \! (d^4d_fm/h )+ (Bm) L(16d2+8ddff)+4Vd+BmLdL(16d^2+8d_f)+4Vd+BmLd Data 6BmL(4d2+2ddff+md)6BmL (4d^2+2d_f+md ) 2Llog(d4dffm/h)+log(Bm)2L \! (d^4d_fm/h )+ (Bm) Pd[L(16d2+8ddff)+4Vd]+BmLdP_d [L(16d^2+8d_f)+4Vd ]+BmLd Pipeline 6BmL(4d2+2ddff+md)6BmL (4d^2+2d_f+md ) 2Llog(d4dffm/h)+log(Bm)2L \! (d^4d_fm/h )+ (Bm) L(16d2+8ddff)+4Vd+BmLdL(16d^2+8d_f)+4Vd+BmLd Tensor 6BmL(4d2+2ddff+md)6BmL (4d^2+2d_f+md ) 2Llog(d4dffm/h)+log(Bm)2L \! (d^4d_fm/h )+ (Bm) L(16d2+8ddff)+4Vd+BmLdL(16d^2+8d_f)+4Vd+BmLd Context 6BmL(4d2+2ddff+md)6BmL (4d^2+2d_f+md ) 2Llog(d4dffm/h)+log(Bm)2L \! (d^4d_fm/h )+ (Bm) Pc[L(16d2+8ddff)+4Vd]+BmLdP_c [L(16d^2+8d_f)+4Vd ]+BmLd Expert 6BmL(Pe(4d2+md)+2Eadde)6BmL (P_e\! (4d^2+md )+2E_ad_e ) 2Llog(d4dem/h)+log(Bm)2L \! (d^4d_em/h )+ (Bm) Pe(16Ld2+4Vd+BmLd)+8LEddeP_e (16Ld^2+4Vd+BmLd )+8LEdd_e 3D 6BmL(4d2+2ddff+md)6BmL (4d^2+2d_f+md ) 2Llog(d4dffm/h)+log(Bm)2L \! (d^4d_fm/h )+ (Bm) Pd[L(16d2+8ddff)+4Vd]+BmLdP_d [L(16d^2+8d_f)+4Vd ]+BmLd 5D 6BmL(4d2+2Eadde+md)6BmL (4d^2+2E_ad_e+md ) 2Llog(d4dem/h)+log(Bm)2L \! (d^4d_em/h )+ (Bm) PdPcPe(16Ld2+4Vd)+8PdPcLEdde+BmLdP_dP_cP_e (16Ld^2+4Vd )+8P_dP_cLEdd_e+BmLd TABLE VIII: Complexity Analysis (Global). Explicit leading-order costs under our simplified Transformer cost model, for a single Transformer training iteration (forward and backward pass), including explicit constant factors. For clarity, we use m:=S+Tm:=S+T. I-H Key Insights & Takeaways The main lesson is stage-aware hybridization: no single intra-model strategy is best for the whole RL-LLM loop. Generation, Assessment, and Training invoke similar Transformer building blocks, but they stress different dimensions of the system. Generation is autoregressive and latency-sensitive; Assessment is mostly teacher-forced, forward-only, and throughput-oriented; Training is memory-intensive as it uses activations, gradients, and optimizer state. DP scales throughput, not single-sample latency. Data parallelism is ideal for increasing the number of prompts, completions, or preference pairs processed per unit time. In online Generation, it can process more prompts or candidates in parallel, but each replica still executes the full autoregressive chain Dgen(πθ,S,T)=O(TDf(πθ,S+T))D_gen( _θ;S,T)=O(TD_f( _θ;S+T)). Thus, DP does not shorten the critical path of one rollout trajectory or one model invocation. TP and P are capacity/latency tools with communication costs. Tensor and pipeline parallelism are useful when a model does not fit on one device or when the latency of a single invocation must be reduced. In Generation, they reduce the cost of each actor invocation but do not remove the outer token-by-token dependence. In Assessment and Training, they help execute larger reward, reference, actor, or critic models. Their benefit must be balanced against TP collectives, P activation transfers, pipeline bubbles, and resharding overheads. SP/CP matter increasingly for reasoning. Long CoT traces, tool histories, retrieved documents, verifier outputs, and multi-turn interaction histories increase S+TS+T, making activation and attention memory central bottlenecks. Sequence and context parallelism directly target this regime by partitioning the sequence/context dimension. They are especially useful for long-context Assessment and Training, and for actor Generation when the rollout context exceeds single-device memory. Frozen and trainable models prefer different layouts. Reward and reference models are usually frozen and forward-only, so they often scale best as replicated batched-inference services when they fit per device. The actor and critic are trainable, and therefore dominate memory through activations, gradients, and optimizer state. Consequently, actor/critic Training benefits most from ZeRO/FSDP-style sharding, TP/P, and 3D/5D combinations. PPO is the most demanding case because it may train both πθ _θ and VψV_ψ, whereas GRPO and DPO train only πθ _θ. EP scales capability but can create systems skew. MoE policies can increase reasoning capacity at limited per-token compute by activating only a subset of experts. However, dense attention remains on the critical path, and expert routing introduces all-to-all communication. RL post-training further complicates EP because the evolving policy changes the token distribution, which can create expert-load imbalance. Efficient EP for RLMs therefore requires routing-aware load balancing and communication-aware expert placement. Practical RLM systems should specialize by model and stage. Actor Generation may prioritize KV-cache memory, rollout latency, and batching; reward/reference Assessment may use cheap replicated inference; actor/critic Training may require 3D parallelism and optimizer-state sharding. This intra-model specialization is precisely what motivates the inter-model placement, stage fusion, hybrid execution, and asynchronous strategies discussed next. Fig. 8: Inter-model parallelism details (asynchronous execution, hybrid execution, stage fusion). No AI was used to conceive or to draw the figure. Fig. 9: Inter-model parallelism details (placement strategies, model structure configurations). No AI was used to conceive or to draw the figure. IV Inter-Model Parallelism We organize inter-model parallelism into five categories. Model structure configurations decide whether trainable components such as the actor and critic share parameters. Model placement strategies decide whether model roles are co-located, partially co-located, or disaggregated across device groups. Stage fusion removes coarse barriers between stages or sub-stages by streaming partial outputs downstream. Hybrid execution allows different invocations of the same model to use different intra-model layouts, often requiring resharding or automated planning. Finally, asynchronous execution overlaps different RL iterations by allowing bounded-stale parameter reads. These categories are often combined in high-performance systems such as ReaL, RLHFuse, HybridFlow/verl, OpenRLHF, StreamRL, and AReaL [169, 282, 217, 117, 281, 92]. Figures 8 and 9 illustrate this taxonomy. IV-A Model Structure Configurations Model structure configurations determine the architectural relation between the trainable model components in the RL-LLM loop. The central question is whether the actor πθ _θ and critic VψV_ψ are implemented as independent models, as a shared actor–critic network, or whether the critic is removed by the learning algorithm. PPO-style RLHF uses an actor and a value model, so this choice directly affects work, memory, and optimization stability [208, 279]. Critic-free methods such as GRPO remove VψV_ψ and replace value-based advantage estimation by group-relative normalization or related baselines, changing both the algorithm and the systems profile [213, 107]. In a shared actor–critic configuration, a single Transformer backbone fωf_ω produces hidden states that feed two task-specific heads: a policy head hπh_π for token logits and a value head hVh_V for scalar value estimates (i.e., πθ(⋅∣x,y<t)=hπ(fω(x,y<t)) _θ(· x,y_<t)=h_π(f_ω(x,y_<t)) and Vψ(x,y<t)=hV(fω(x,y<t))V_ψ(x,y_<t)=h_V(f_ω(x,y_<t))). The main systems benefit is that the actor and critic no longer require two separate backbones. This reduces parameter, gradient, optimizer-state, and activation memory, and can replace separate actor/critic forward passes by one shared backbone pass followed by two small heads. In a PPO-style pipeline, this is an inter-model decision: it changes the number of trainable model invocations in Assessment and Training, not merely the internal architecture of one model. The trade-off is optimization coupling. The policy-gradient loss and value loss both update fωf_ω, so their gradients may pull the shared representation in different directions. A single backbone must serve both action selection and return prediction, and a shared optimizer schedule must accommodate losses with different scales and curvature. In practice, this often requires careful value loss weighting, separate learning rates for heads, or partial sharing in which only lower layers are shared and upper layers remain task-specific. Shared actor–critic is therefore most attractive when memory is scarce or when the critic is close in scale to the actor; it is less attractive when value learning requires substantially different representations or optimization dynamics. With independent actor and critic models, πθ _θ and VψV_ψ have separate backbones, optimizer states, and hyperparameters. This avoids gradient interference and allows the critic to be sized differently from the actor, for example by using a smaller critic to reduce memory and compute. The cost is a larger model-state footprint and a separate critic forward/backward path. In the work–depth analysis of Section I-E, this is exactly the additional PPO cost Θ(BK(S+T)[Cftok(Vψ)+Cbtok(Vψ)]) \! (BK(S+T)[C_f^tok(V_ψ)+C_b^tok(V_ψ)] ) plus the corresponding critic model-state memory. Finally, critic-free methods move this trade-off from architecture to algorithm design. GRPO removes the learned value model and computes relative advantages from groups of completions sampled for the same prompt, while DPO removes the online actor–critic loop altogether and optimizes over preference pairs. IV-B Model Placement Strategies Model placement specifies the mapping from model execution to device groups. A device group is a set of GPUs that jointly stores and executes one or more model invocations, possibly using an intra-model strategy internally. The placement decision controls aspects such as memory co-residency, inter-stage communication, and the amount of concurrency available across model roles. In fully co-located placement, all model roles share one device group and execute through temporal multiplexing. The same pool of GPUs first performs actor generation, then reward/reference/critic inference, and then actor/critic training. This maximizes locality: intermediate tensors can remain within the same device group, and no inter-group communication is needed to pass rollouts, log-probabilities, values, or rewards. It also simplifies orchestration because the runtime schedules one device pool rather than multiple distributed services. The cost is co-resident memory pressure and poor role specialization. The device group must be provisioned for the largest memory requirement, often Training, yet it is also used for Generation, which has different arithmetic intensity and is often memory-bandwidth or latency dominated. Moreover, “redundant memory” in this context means that the same devices must hold the resident state for multiple model roles even when only one role is active. If the system uses data parallelism, this co-residency may be replicated across data-parallel groups: each replica may need actor, critic, reward, and reference state or the ability to materialize them. To mitigate this, one can employ techniques such as offloading, loading and unloading idle models, or weight gathering from sharded states. These techniques reduce memory pressure but introduce context-switching and data-movement overheads. As a result, fully co-located execution is simple and locality-friendly, but it often leaves hardware idle because the stage service times and resource needs are highly imbalanced. Partially co-located, or interleaved, placement sits between full colocation and full disaggregation. Selected models or stages share a device group; others are placed separately. For example, actor generation and training may share a group to avoid repeated actor-weight transfers, while reward and reference inference run on smaller inference-oriented groups. Alternatively, reward and reference models may be co-located because both are frozen and forward-only, while the trainable actor and critic are separated. This design is useful when some model pairs benefit from locality while others benefit from concurrency. It reduces the worst co-residency pressure of full colocation without requiring every stage boundary to become a network boundary. Yet, it also complicates scheduling: groups that share some but not all models must coordinate execution order, memory residency, and data handoff. The best interleaving depends on model sizes, stage service times, cluster topology, and whether a given edge carries small metadata, token IDs, activations, or large parameter shards. In disaggregated placement, different model roles or stages run on dedicated, disjoint device groups. The actor group stores and executes the actor, the reward group stores the reward model, the critic group stores the value model, and the reference group stores the frozen reference model. This eliminates cross-model co-residency: a reward-model GPU no longer needs actor or critic weights, and a training GPU no longer needs to keep frozen reward/reference models resident. It also enables heterogeneous hardware specialization. Large trainable actors can be assigned to high-memory GPUs, while smaller frozen reward or reference models can run on cheaper or inference-optimized devices. StreamRL explicitly argues for such disaggregation because colocated generation and training couple resources with very different compute and memory profiles, while disaggregated stream generation enables flexible resource allocation, heterogeneous training setups, and even cross-datacenter deployment [281]. Disaggregation is a prerequisite for many forms of inter-model concurrency, but it is not by itself a speedup. If the stages still execute strictly Generation → Assessment → Training, then the iteration latency remains the sum of stage times and some groups will wait idle. The benefit appears when disaggregation is combined with stage fusion, streaming, or asynchronous execution. The price is explicit communication: rollouts, token IDs, log-probabilities, rewards, values, and sometimes updated parameters must cross device-group boundaries. Systems such as OpenRLHF, HybridFlow/verl, ReaL, FlexRLHF, and StreamRL explore different points in this placement space, using Ray-style orchestration, hierarchical APIs, parameter reallocation, or streaming data paths to manage the additional complexity [117, 217, 169, 251, 281]. IV-C Stage Fusion Stage fusion removes coarse barriers between consecutive stages or between subtasks within a stage. In the baseline RLHF loop, the Assessment stage waits until the entire Generation batch has completed, and Training waits until all assessment outputs are ready. This barrier structure is inefficient for RLMs because generation lengths are long-tailed: short trajectories may finish much earlier than the longest ones, yet their reward/reference/value inference is delayed by stragglers. Stage fusion decomposes stages into finer-grained subtasks and schedules downstream work as soon as its input is available. There are two common forms. In inter-stage fusion, completed samples are streamed from Generation into Assessment without waiting for the whole rollout batch. A short completion can be scored by the reward model and processed by the reference or critic while the actor is still decoding longer completions. This turns generation-length skewness into useful overlap. In intra-stage fusion, operators within the same stage interleave microbatches or pipeline schedules. For example, actor and critic training pipelines may be interleaved so that one model’s bubble time is filled by another model’s forward or backward microbatch. RLHFuse is a canonical example: it splits generation and assessment into sample-level subtasks for inter-stage fusion and splits training into microbatch-level subtasks for intra-stage fusion, enhancing throughput by up to 3.7×3.7× by mitigating long-tail generation skew and pipeline bubbles [282]. Related overlap-oriented work such as OPPO also targets PPO-style serialization and long-tail response lengths by overlapping pipeline stages [255]. The key point is that fusion does not reduce total work; it reduces depth and idle time by changing when ready subtasks may execute. Fusion is throughput-oriented and can increase single-sample latency. A sample that finishes early may wait in a queue until enough samples form an efficient microbatch, or it may be delayed by backpressure from a downstream stage. Thus, the steady-state time per batch can improve even if the time from one sample’s generation to its training update increases. Fusion also increases peak memory because multiple stages are live simultaneously: model weights, KV caches, activations, queues, and communication buffers may coexist. The design problem is therefore to choose a granularity that is fine enough to hide skew and bubbles but not so fine that scheduling overhead, fragmentation, or memory pressure dominates. IV-D Hybrid Parallelism and Adaptive Scheduling Hybrid parallelism means that different model invocations in the same RL-LLM pipeline may use different intra-model strategies, device allocations, or execution modes. This is distinct from ordinary 3D parallelism inside one model: the same actor πθ _θ may be invoked once for autoregressive generation and later for training, and these two invocations have different optimal layouts. Generation favors low-latency inference and efficient KV-cache use while Training favors higher memory capacity, efficient gradient synchronization, activation checkpointing, or optimizer-state sharding. A single static layout forces a compromise; hybrid execution allows each role to use a matched layout. For example, if the actor is stored in a ZeRO/FSDP-style sharded training layout, generation may require gathering or resharding the weights into an inference-centered layout. Conversely, after generation, the system may need to redistribute weights or optimizer state back into the training layout. DeepSpeed-Chat’s Hybrid Engine is an early example of this idea, combining training-mode memory optimizations with inference-mode kernel and parallelism optimizations [261]. ReaL generalizes the idea as parameter reallocation: an execution plan chooses role-specific allocations and parallelization strategies, and the runtime redistributes parameters between them [169]. HybridFlow/verl’s 3D-HybridEngine similarly targets efficient actor resharding between training and generation with low redundancy, while NeMo RL exposes a mode that can train with pipeline parallelism but run TensorRT-LLM inference in a tensor-parallel layout [217, 214, 184]. The cost is resharding. When two invocations of the same model use incompatible layouts or different device groups, parameter shards must be communicated and repartitioned. During the transfer, the system may need extra memory for source and destination layouts, and the transfer may sit on the critical path unless it is overlapped with unrelated work. Hybrid execution is profitable only when the per-stage gains from using specialized layouts exceed the resharding cost. Pipe-RLHF is an example of computation-mode-aware parallelism: it uses stage-specific parallelization to improve resource utilization [262]. ReaL and HybridFlow also move in this direction by representing the RLHF pipeline as a dataflow or execution plan rather than a fixed sequence of scripts [169, 217]. The trade-off is engineering complexity: an adaptive scheduler requires profiling, cost modeling, memory feasibility checks, and robust orchestration. It is most valuable at scales where a static hand-designed layout leaves substantial hardware idle. IV-E Asynchronous Execution Stage fusion overlaps work within an iteration. Asynchronous execution overlaps different RL iterations by relaxing cross-iteration parameter freshness. In a synchronous loop, iteration k+1k+1 cannot generate data until iteration k has completed Training and published updated parameters. In an asynchronous loop, Generation or Assessment may read a bounded-stale snapshot while Training from the previous iteration has still not completed. Bounded asynchrony introduces a systems–algorithm trade-off. Systems benefit because rollout workers and learners no longer wait for one another. This can substantially improve utilization when generation is long-tailed or much slower than training. Asynchronous RLHF explicitly studies this off-policy setting and shows that generation and learning can be decoupled for better efficiency, while AReaL further develops a fully asynchronous RL system with a staleness-aware PPO [180, 92]. Laminar pushes the systems side further through trajectory-level asynchrony and a distributed parameter relay tier, while StaleFlow explicitly coordinates rollouts under staleness constraints to balance convergence and throughput [216, 145]. The algorithmic risk is policy drift. A trajectory may have been sampled from a behavior policy πbeh _beh, while the learner updates a newer policy πθ _θ. To alleviate this, systems can track statistics such as staleness, token-level KL, ratio variance, clip fraction, effective sample size, and reward/advantage shifts. Standard PPO tolerates small drift because clipped importance ratios limit the update, but large staleness can move the trust-region center toward an outdated low-quality policy. AReaL addresses this by separating the behavior policy used for off-policy correction from the proximal policy used as the trust-region center. Asynchrony also changes memory accounting. If generation and training run on disjoint device groups and must proceed concurrently, generation needs a stable read-only snapshot while training updates another copy. This creates a shadow pair: a training copy with optimizer state and a generation copy used for inference. In the worst case, this roughly doubles the parameter memory for each asynchronously consumed trainable model. The factor is not necessarily a full 2×2× increase in total training memory, because optimizer states usually remain only on the training side and the shadow copy may be BF16, quantized, or offloaded; nevertheless, any table or memory model for asynchronous execution should include an additional persistent inference-side parameter copy whenever trainable models are disaggregated across stale readers and writers. IV-F Complexity Analysis Config Stage Global Work Global Depth Peak Memory per-device group Baseline Gen. BK⋅Cgen(πθ)BK\!·\!C_gen( _θ) Dgen(πθ)D_gen( _θ) |πθ|+BK⋅MKV| _θ|+BK\!·\!M_KV Assess. BK(S+T)BK(S+T) ⋅[Cftok(πref)+Cftok(Rφ)+Cftok(Vψ)]·[C_f^tok( _ref)+C_f^tok(R_ )+C_f^tok(V_ψ)] Df(πref)+Df(Rφ)+Df(Vψ)D_f( _ref)+D_f(R_ )+D_f(V_ψ) |πref|+|Rφ|+|Vψ|| _ref|+|R_ |+|V_ψ| +BK(S+T)+BK(S+T) ⋅maxMInf(πref),MInf(Rφ),MInf(Vψ)· \M_Inf( _ref),M_Inf(R_ ),M_Inf(V_ψ)\ Train. BK(S+T)⋅[Ctrtok(πθ)+Ctrtok(Vψ)]BK(S+T)\!·\![C_tr^tok( _θ)+C_tr^tok(V_ψ)] Dtr(πθ)+Dtr(Vψ)D_tr( _θ)+D_tr(V_ψ) ⋅(|πθ|+|Vψ|)4\!·\!(| _θ|+|V_ψ|) +BK(S+T)(Lπdπ+Lψdψ)+BK(S+T)(L_πd_π+L_ψd_ψ) AC shared Gen. BK⋅Cgen(πSH)BK\!·\!C_gen( _SH) Dgen(πSH)D_gen( _SH) |πSH|+BK⋅MKV| _SH|+BK\!·\!M_KV Assess. BK(S+T)BK(S+T) ⋅[Cftok(πref)+Cftok(Rφ)+Cftok(πSH)]·[C_f^tok( _ref)+C_f^tok(R_ )+C_f^tok( _SH)] Df(πref)+Df(Rφ)+Df(πSH)D_f( _ref)+D_f(R_ )+D_f( _SH) |πref|+|Rφ|+|πSH|| _ref|+|R_ |+| _SH| +BK(S+T)+BK(S+T) ⋅maxMInf(πref),MInf(Rφ),MInf(πSH)· \M_Inf( _ref),M_Inf(R_ ),M_Inf( _SH)\ Train. BK(S+T)⋅Ctrtok(πSH)BK(S+T)\!·\!C_tr^tok( _SH) Dtr(πSH)D_tr( _SH) ⋅|πSH|+BK(S+T)LSHdSH4\!·\!| _SH|+BK(S+T)L_SHd_SH Disagg. Gen. BK⋅Cgen(πθ)BK\!·\!C_gen( _θ) Dgen(πθ)D_gen( _θ) |πθ|+BK⋅MKV| _θ|+BK\!·\!M_KV Assess. BK(S+T)BK(S+T) ⋅[Cftok(πref)+Cftok(Rφ)+Cftok(Vψ)]·[C_f^tok( _ref)+C_f^tok(R_ )+C_f^tok(V_ψ)] maxDf(πref),Df(Rφ),Df(Vψ) \D_f( _ref),D_f(R_ ),D_f(V_ψ)\ max|πref|+BK(S+T)MInf(πref), \| _ref|+BK(S+T)M_Inf( _ref), |Rφ|+BK(S+T)MInf(Rφ),|R_ |+BK(S+T)M_Inf(R_ ), |Vψ|+BK(S+T)MInf(Vψ)|V_ψ|+BK(S+T)M_Inf(V_ψ)\ Train. BK(S+T)⋅[Ctrtok(πθ)+Ctrtok(Vψ)]BK(S+T)\!·\![C_tr^tok( _θ)+C_tr^tok(V_ψ)] maxDtr(πθ),Dtr(Vψ) \D_tr( _θ),D_tr(V_ψ)\ max4|πθ|+BK(S+T)Lπdπ, \4| _θ|+BK(S+T)L_πd_π, 4|Vψ|+BK(S+T)Lψdψ4|V_ψ|+BK(S+T)L_ψd_ψ\ Hybrid Gen. BK⋅Cgen(πθ)BK\!·\!C_gen( _θ) DgenTP(πθ)D_gen^TP( _θ) |πθ|+BK⋅MKV| _θ|+BK\!·\!M_KV Assess. BK(S+T)BK(S+T) ⋅[Cftok(πref)+Cftok(Rφ)+Cftok(Vψ)]·[C_f^tok( _ref)+C_f^tok(R_ )+C_f^tok(V_ψ)] DfTP(πref)+DfTP(Rφ)+DfTP(Vψ)D_f^TP( _ref)+D_f^TP(R_ )+D_f^TP(V_ψ) |πref|+|Rφ|+|Vψ|| _ref|+|R_ |+|V_ψ| +BK(S+T)+BK(S+T) ⋅maxMInf(πref),MInf(Rφ),MInf(Vψ)· \M_Inf( _ref),M_Inf(R_ ),M_Inf(V_ψ)\ Train. BK(S+T)⋅[Ctrtok(πθ)+Ctrtok(Vψ)]BK(S+T)\!·\![C_tr^tok( _θ)+C_tr^tok(V_ψ)] Dtr(πθ)+Dtr(Vψ)D_tr( _θ)+D_tr(V_ψ) ⋅(|πθ|+|Vψ|)4\!·\!(| _θ|+|V_ψ|) +BK(S+T)(Lπdπ+Lψdψ)+BK(S+T)(L_πd_π+L_ψd_ψ) Stage F. Inter BKCgen(πθ)+BK(S+T)BK\,C_gen( _θ)+BK(S+T) ⋅[Cftok(πref)+Cftok(Rφ)+Cftok(Vψ)]·[C_f^tok( _ref)+C_f^tok(R_ )+C_f^tok(V_ψ)] maxDgen(πθ),Df(πref), \D_gen( _θ),D_f( _ref), Df(Rφ),Df(Vψ)D_f(R_ ),D_f(V_ψ)\ max|πθ|+BKMKV,|πref|+|Rφ|+|Vψ| \| _θ|+BKM_KV,\,| _ref|+|R_ |+|V_ψ|\ Intra BK(S+T)⋅[Ctrtok(πθ)+Ctrtok(Vψ)]BK(S+T)\!·\![C_tr^tok( _θ)+C_tr^tok(V_ψ)] maxDtr(πθ),Dtr(Vψ) \D_tr( _θ),D_tr(V_ψ)\ ⋅(|πθ|+|Vψ|)4\!·\!(| _θ|+|V_ψ|) +BK(S+T)(Lπdπ+Lψdψ)+BK(S+T)(L_πd_π+L_ψd_ψ) Async Aggr. BKCgen(πθ)+BK(S+T)BK\,C_gen( _θ)+BK(S+T) ⋅[Cftok(πref)+Cftok(Rφ)+Cftok(Vψ)·[C_f^tok( _ref)+C_f^tok(R_ )+C_f^tok(V_ψ) +Ctrtok(πθ)+Ctrtok(Vψ)] 8.19447pt+C_tr^tok( _θ)+C_tr^tok(V_ψ)] maxDgen(πθ)+maxDf(πref),Df(Rφ),Df(Vψ), \D_gen( _θ)+ \D_f( _ref),D_f(R_ ),D_f(V_ψ)\, maxDtr(πθ),Dtr(Vψ) \D_tr( _θ),D_tr(V_ψ)\ \ max|πθ|+BKMKV+Msh(πθ), \| _θ|+BKM_KV+M_sh( _θ), |πref|,|Rφ|,|Vψ|,| _ref|,|R_ |,|V_ψ|, 4|πθ|+BK(S+T)Lπdπ,4| _θ|+BK(S+T)L_πd_π, 4|Vψ|+BK(S+T)Lψdψ4|V_ψ|+BK(S+T)L_ψd_ψ\ Combined Aggr. BKCgen(πθ)+BK(S+T)BK\,C_gen( _θ)+BK(S+T) ⋅[Cftok(πref)+Cftok(Rφ)+Cftok(Vψ)·[C_f^tok( _ref)+C_f^tok(R_ )+C_f^tok(V_ψ) +Ctrtok(πθ)+Ctrtok(Vψ)] 8.19447pt+C_tr^tok( _θ)+C_tr^tok(V_ψ)] maxmaxDgenTP(πθ),Df(πref), \ \D_gen^TP( _θ),D_f( _ref), Df(Rφ),Df(Vψ), 16.38895pt 8.19447ptD_f(R_ ),D_f(V_ψ)\, maxDtr(πθ),Dtr(Vψ) \D_tr( _θ),D_tr(V_ψ)\ \ max|πθ|+BKMKV+Msh(πθ), \| _θ|+BKM_KV+M_sh( _θ), |πref|,|Rφ|,|Vψ|,| _ref|,|R_ |,|V_ψ|, 4|πθ|+BK(S+T)Lπdπ,4| _θ|+BK(S+T)L_πd_π, 4|Vψ|+BK(S+T)Lψdψ4|V_ψ|+BK(S+T)L_ψd_ψ\ TABLE IX: Inter-model parallelism. Global work, global depth, and peak memory per device group reported for the Generation, Assessment, and Training stages of a single RL iteration within one epoch under different inter-model parallelism techniques. Global work and depth report total FLOPs and the critical-path length per stage. Peak memory is measured per device group and taken as the maximum over all groups active within a stage; in configurations without disaggregation, all models share the same device group and execute through temporal multiplexing, where different models become active at different stages. “TP” superscript & “SH” subscript: A TPTP superscript denotes tensor-parallel execution; πSH _SH denotes the shared actor–critic configuration. “tr” subscript: For conciseness, for training-stage model invocations, we define Ctrtok(M):=Cftok(M)+Cbtok(M)≈3Cftok(M)C_tr^tok(M):=C_f^tok(M)+C_b^tok(M)≈ 3C_f^tok(M) and Dtr(M):=Df(M)+Db(M)D_tr(M):=D_f(M)+D_b(M), because a parameter update requires a forward pass followed by backpropagation. Msh(πθ)M_sh( _θ) denotes the persistent inference-side shadow copy of the actor required when asynchronous consumers use a bounded-stale snapshot. Also, sample-time actor log-probabilities are assumed to be computed on the fly during Generation from the same logits used for sampling; therefore they do not add a separate actor forward pass. Baselines: The Baseline configuration sequentially (i.e., one device group) executes all RL components, resulting in additive depth terms. AC shared uses a shared Transformer backbone for actor and critic. All models run sequentially in a single device group. Disaggregated execution assigns models to distinct device groups, allowing concurrent execution. The Hybrid configuration applies TP to Generation, TP/P to Assessment, and ZeRO-3 to Training. All models share the same device group. Stage F. applies inter-stage fusion to Generation and Assessment and intra-stage fusion to Training. Asynchronous execution overlaps Generation and Assessment with Training by allowing bounded weight staleness. Combined execution combines asynchronous overlap with disaggregated model placement and hybrid intra-model parallelism, following the RLHFuse execution model. Training memory assumes the Adam optimizer without activation checkpointing, accounting for parameters, gradients, optimizer states, and activations. We now rigorously quantify the inter-model strategies. The analysis is conducted for a PPO-style online RL-LLM pipeline, which is the most demanding common setting; GRPO and DPO can be recovered by deleting the critic and/or online assessment components. Detailed derivations are in Appendix B-D. We first define stage-level terms. Generation comes with WG W_G =BKCgen(πθ) =BK\,C_gen( _θ) (work), (work), DG D_G =Dgen(πθ) =D_gen( _θ) (depth), (depth), MG M_G =|πθ|+BKMKV =| _θ|+BK\,M_KV (memory). (memory). Here CgenC_gen and DgenD_gen include the prefill–decode decomposition from Table I. Next, the work of the assessment stage is WA=BK(S+T)[Cftok(πref)+Cftok(Rφ)+Cftok(Vψ)].W_A=BK(S+T)\,[C_f^tok( _ref)+C_f^tok(R_ )+C_f^tok(V_ψ)]. For depth, we distinguish co-located sequential execution DAΣ=Df(πref)+Df(Rφ)+Df(Vψ),D_A =D_f( _ref)+D_f(R_ )+D_f(V_ψ), from disaggregated execution DAmax=maxDf(πref),Df(Rφ),Df(Vψ).D_A = \D_f( _ref),D_f(R_ ),D_f(V_ψ)\. Similarly, training work is WT=BK(S+T)[Ctrtok(πθ)+Ctrtok(Vψ)],W_T=BK(S+T)\,[C_tr^tok( _θ)+C_tr^tok(V_ψ)], with sequential and disaggregated depths DTΣ=Dtr(πθ)+Dtr(Vψ),DTmax=maxDtr(πθ),Dtr(Vψ).D_T =D_tr( _θ)+D_tr(V_ψ), D_T = \D_tr( _θ),D_tr(V_ψ)\. For memory, we use MAΣM_A for the co-resident assessment footprint, MTΣM_T for the co-resident actor–critic training footprint, and MTmaxM_T for the maximum over disaggregated actor and critic training groups. When asynchronous execution requires a persistent inference-side copy of a trainable model, we write this shadow-copy cost as MshM_sh. Table IX summarizes the resulting work, depth, and peak per-device-group memory. The most important point is that most inter-model techniques do not reduce total work: they reduce depth by replacing sequential sums with maxima, or reduce peak memory by eliminating co-residency. The main exception is shared actor–critic, which also reduces work and model state by replacing two trainable backbones with one shared backbone. The baseline has critical path DG+DAΣ+DTΣD_G+D_A +D_T . Disaggregation preserves work but changes within-stage composition from sums to maxima, giving DG+DAmax+DTmaxD_G+D_A +D_T . Stage fusion further overlaps Generation and Assessment, resulting in maxDG,DAmax+DTmax≈DG+DTmax \D_G,D_A \+D_T ≈ D_G+D_T whenever generation dominates assessment. Bounded asynchrony overlaps generation/assessment of one iteration with training of another, yielding a steady-state recurrence depth maxDG+DAmax,DTmax \D_G+D_A ,D_T \. The strongest configuration combines intra-model sharding, disaggregated placement, stage fusion, and bounded asynchrony. Its idealized depth is maxDGTP,DTshard \D_G^TP,D_T^shard\, up to any unhidden assessment tail, resharding, and communication overheads. This expression makes the central limitation explicit: inter-model parallelism can hide or overlap non-generation work, but it cannot remove the autoregressive decode chain inside DGD_G. Reducing that term requires intra-model inference parallelism, faster decoding kernels, batching, or algorithmic changes that shorten generated trajectories. IV-G Key Insights & Takeaways The inter-model analysis yields several lessons. Most inter-model schemes reduce depth, not work. Placement, fusion, hybrid execution, and asynchrony mainly change the critical path by replacing sequential sums with maxima or by overlapping stages. The total FLOP work remains essentially unchanged. Shared actor–critic is the main exception because it removes a separate critic backbone and therefore reduces both work and trainable model-state memory. Disaggregation is a memory and concurrency enabler, not a standalone speedup. Moving actor, critic, reward, and reference models to separate device groups reduces co-resident memory and turns independent subcomputations into max-depth terms. However, if the pipeline still executes strict Generation → Assessment → Training barriers, disaggregation only moves idle time to different device groups. It must be combined with fusion, streaming, or asynchrony to reduce iteration depth. Stage fusion and asynchrony attack different barriers. Stage fusion removes within-iteration barriers, especially the barrier between long-tailed Generation and Assessment. Asynchrony removes cross-iteration barriers by allowing bounded-stale parameter reads. Fusion gives maxDG,DAmax+DTmax \D_G,D_A \+D_T , while asynchrony gives maxDG+DAmax,DTmax \D_G+D_A ,D_T \. They can be combined. The actor remains the dominant bottleneck. Even the combined configuration cannot remove the autoregressive decode chain in DG=Dgen(πθ)=O(TDf(πθ))D_G=D_gen( _θ)=O(TD_f( _θ)). Inter-model scheduling can hide reward, reference, critic, and training work around actor generation, but reducing DGD_G itself requires better inference parallelism (most notably tensor parallelism), batching, KV-cache management, shorter trajectories, or algorithmic changes. Asynchrony trades freshness for hardware utilization and memory. Bounded staleness can reduce steady-state depth from a sum to a max, but it introduces policy drift and may require shadow copies. Thus, the relevant control variables are not only throughput and memory, but also staleness, KL drift between behavior and current policies, PPO clip fraction, and others. The best systems are stage- and role-aware. A scalable RLM system should not assign one global execution mode to all models. Actor Generation, reward/reference Assessment, critic inference, actor Training, and critic Training have different bottlenecks. The strongest designs therefore combine shared or critic-free structure when appropriate, disaggregated placement for memory and concurrency, stage fusion for long-tail overlap, hybrid layouts for stage-specific efficiency, and bounded asynchrony when the learning rule tolerates stale data. Shared actor–critic is effective under resource constraints. Shared actor–critic architectures are particularly effective when device groups and memory are scarce and inter-model concurrency is limited thanks to sharing a single backbone between the policy and value functions. V Parallelism-Focused Specifications We present a unified algorithmic and mathematical formulation of PPO (Algorithm 2) and GRPO (Algorithm 3), which share a common rollout-generation routine (Algorithm 1), together with the offline counterpart DPO (Algorithm 4). The goal is to expose where the RL-LLM pipeline admits intra- and inter-model parallelism, and this to facilitate the development of more efficient RLM architectures. We use a notation in which boldface denotes a per-token vector and a non-bold letter with a t subscript denotes one of its per-token components. For example, n←Vψ(n,n) V_n← V_ψ( x_n, y_n) abbreviates (Vn,t)t=1|n|←(Vψ(n,n,<t))t=1|n|(V_n,t)_t=1^| y_n|←(V_ψ( x_n, y_n,<t))_t=1^| y_n|, with all |n|| y_n| entries produced by a single teacher-forced forward pass. A plain letter without a t subscript is a scalar (e.g., rn←Rφ(n,n)r_n← R_ ( x_n, y_n)). The unit basis vector t e_t has 11 at position t and 00 elsewhere. The all-ones vector 1 has 11 at every position, with length inferred from context. If the actor and critic share a backbone, the PPO actor/critic two invocations should instead be read as a single shared-model invocation with two heads. For DPO, the reference path may be omitted from the online training loop if ℓref,w(n) _ref,w^(n) and ℓref,l(n) _ref,l^(n) are precomputed and stored with the preference dataset. Background colors indicate model roles. The label [inter-model concurrency] marks following fork–join (colored) regions in which distinct model invocations have no data dependency and may run concurrently when placed on separate device groups. The label [DP] (data parallelism) marks independent sample-level work, such as prompts, completions, rollouts, or preference pairs. The tags [TP], [P], [SP], [CP], and [EP] indicate intra-model parallel strategies that can be used for the corresponding invocation. Boldface (e.g., TP) indicates a usual use, non-bold font (e.g., SP) is a possible but not a necessarily common use case. Each tag describes the next following line or code block. TP/P are mainly capacity or single-invocation latency tools, SP is mainly useful for trainable teacher-forced passes with large activation memory, CP is useful for long-context invocations, and EP applies only to MoE models. Algorithm 1 A common part of Generation, used by PPO and GRPO Input: Batch of prompts X=bb=1BX=\ x_b\_b=1^B, candidates/prompt K. Output: Set ℬrollB_roll of generated samples paired with per-token rollout log-probabilities. 1 Initialize ℬroll←∅B_roll← ; 2 [DP] foreach prompt b∈X x_b∈ X do 3 [DP] for candidate i=1i=1 to K do 4 Initialize b(i)←[] y_b^(i)←[] and ℓold,b(i)←[] _old,b^(i)←[]; 5 6 PAR 7 8 laneActorlaneActoractor path 9 for t=1,2,…t=1,2,… until EOS or max length do 10 [TP, P, SP, CP, EP] 11 Sample next token yb,t(i)∼πθ(⋅∣b,b,<t(i))y_b,t^(i) _θ (· x_b, y_b,<t^(i) ) via stochastic decoding; 12 Actor log-prob ℓold,b,t(i)←logπθ(yb,t(i)∣b,b,<t(i)) _old,b,t^(i)← _θ (y_b,t^(i) x_b, y_b,<t^(i) ); 13 Append yb,t(i)y_b,t^(i) to b(i) y_b^(i) and ℓold,b,t(i) _old,b,t^(i) to ℓold,b(i) _old,b^(i); 14 end for 15 Append (b,b(i),ℓold,b(i)) ( x_b, y_b^(i), _old,b^(i) ) to ℬrollB_roll; 16 end for 17 end foreach Algorithm 2 Algorithmic and Mathematical Specification of PPO Input: GAE discount γ and smoothing λ, KL coefficient β, PPO clip range ε , entropy coefficient cHc_H; set of prompts X, prompt batch size B, candidates/prompt K, training epochs EPPOE_PPO. Output: Trained parameters (θ,ψ)(θ,ψ). 1 [DP] foreach prompt batch bb=1B⊂X\ x_b\_b=1^B⊂ X do 2 — Generation (Online) — 3 Generate completions ℬroll←(b,b(i),ℓold,b(i))b=1,i=1B,KB_roll← \ ( x_b, y_b^(i), _old,b^(i) ) \_b=1,i=1^B,K via Algorithm 1; 4 Flatten ℬroll←(n,n,ℓold(n))n=1BKB_roll← \ ( x_n, y_n, _old^(n) ) \_n=1^BK; 5 — Assessment — 6 [DP] foreach (n,n,ℓold(n))∈ℬroll ( x_n, y_n, _old^(n) ) _roll do 7 [inter-model concurrency] 8 9 PAR 10 11 laneRewardlaneRewardreward path 12 [TP, P, SP, CP, EP] 13 Compute scalar reward sn←Rφ(n,n)s_n← R_ ( x_n, y_n ); 14 15 PAR 16 17 laneCriticlaneCriticcritic path 18 [TP, P, SP, CP, EP] 19 Compute values n←Vψ(n,n) V_n← V_ψ( x_n, y_n); 20 EOS terminal value Vn,|n|+1←0V_n,| y_n|+1← 0; 21 22 PAR 23 24 laneReferencelaneReferencereference path 25 [TP, P, SP, CP, EP] 26 Compute reference log-probs ℓref(n)←logπref(n∣n) _ref^(n)← _ref( y_n x_n); 27 Per-token KL-shaped rewards n←−β(ℓold(n)−ℓref(n))+sn|n| r_n←-β\, ( _old^(n)- _ref^(n) )+s_n\, e_| y_n| (KL penalty enters via reward and propagates through GAE; no KL term in PPO loss) 28 Initialize n←[] A_n←[], ^n←[] G_n←[], and EOS terminal advantage An,|n|+1←0A_n,| y_n|+1← 0; 29 for t=|n|t=| y_n| to 11 do 30 Temporal difference residual δn,t←rn,t+γVn,t+1−Vn,t _n,t← r_n,t+γ V_n,t+1-V_n,t; 31 Compute advantage (GAE) An,t←δn,t+γλAn,t+1A_n,t← _n,t+γλ A_n,t+1; 32 Compute discounted return G^n,t←Vn,t+An,t G_n,t← V_n,t+A_n,t; 33 Prepend An,tA_n,t to n A_n and G^n,t G_n,t to ^n G_n; 34 end for 35 Augment rollout n in ℬrollB_roll with (n,^n) ( A_n, G_n ); 36 end foreach 37 — Training via PPO — 38 for e=1e=1 to EPPOE_PPO do 39 Shuffle ℬrollB_roll and partition into minibatches ℬ\B\; 40 [DP] foreach minibatch ℬB do 41 [DP] foreach rollout n∈ℬn do 42 [inter-model concurrency] 43 44 PAR 45 46 laneActorlaneActoractor path 47 [TP, P, SP, CP, EP] 48 Current log-probs ℓθ(n)←logπθ(n∣n) _θ^(n)← _θ( y_n x_n); 49 Importance ratios n←exp(ℓθ(n)−ℓold(n)) ρ_n← \! ( _θ^(n)- _old^(n) ); 50 51 PAR 52 53 laneCriticlaneCriticcritic path 54 [TP, P, SP, CP, EP] 55 Current values n′←Vψ(n,n) V _n← V_ψ( x_n, y_n); 56 end foreach 57 Number of tokens in batch Z←∑n∈ℬ|n|Z← _n | y_n|; 58 [inter-model concurrency] 59 60 PAR 61 62 laneActorlaneActoractor path 63 Compute PPO loss: 64 ℒPPO=−1Z∑n∈ℬ∑t=1|n|min(ρn,tAn,t,L_PPO=- 1Z _n _t=1^| y_n| ( _n,tA_n,t, 65 OPEN clip(ρn,t,1±ε)An,t) \ \ \ \ \ \ \ \ \ clip( _n,t,1± )\,A_n,t); 66 (Optional) Entropy bonus; H(⋅)H(·) is the actor’s per-token Shannon entropy (cH=0c_H=0 disables) 67 ℒH=−1Z∑n∈ℬ∑t=1|n|H(πθ(⋅∣n,n,<t))L_H=- 1Z _n _t=1^| y_n|H ( _θ (· x_n, y_n,<t ) ); 68 Define actor objective: ℒPPO←ℒPPO+cHℒHL_PPO _PPO+c_H\,L_H; 69 [TP, P, SP, CP, EP] 70 Backpropagate & update on θ to minimize ℒPPOL_PPO; 71 72 PAR 73 74 laneCriticlaneCriticcritic path 75 Compute critic loss: 76 ℒcritic=1Z∑n∈ℬ∑t=1|n|(Vn,t′−G^n,t)2L_critic= 1Z _n _t=1^| y_n|(V _n,t- G_n,t)^2; 77 [TP, P, SP, CP, EP] 78 Backpropagate & update on ψ to minimize ℒcriticL_critic; 79 end foreach 80 end for 81 end foreach Algorithm 3 Algorithmic and Mathematical Specification of GRPO Input: KL coefficient β, GRPO clip range ε ; set of prompts X, prompt batch size B, candidates/prompt K, training epochs EGRPOE_GRPO. Output: Trained parameters θ. 1 foreach prompt batch bb=1B⊂X\ x_b\_b=1^B⊂ X do 2 — Generation (Online) — 3 Generate completions ℬroll←(b,b(i),ℓold,b(i))b=1,i=1B,KB_roll← \ ( x_b, y_b^(i), _old,b^(i) ) \_b=1,i=1^B,K via Algorithm 1; 4 — Assessment — 5 [DP] for prompt b=1b=1 to B do 6 [DP] for candidate i=1i=1 to K do 7 [inter-model concurrency] 8 9 PAR 10 11 laneRewardlaneRewardreward path 12 [TP, P, SP, CP, EP] 13 Compute scalar reward rb(i)←Rφ(b,b(i))r_b^(i)← R_ ( x_b, y_b^(i) ); 14 15 PAR 16 17 laneReferencelaneReferencereference path 18 [TP, P, SP, CP, EP] 19 Compute reference log-probs ℓref,b(i)←logπref(b(i)∣b) _ref,b^(i)← _ref ( y_b^(i) x_b ); 20 end for 21 Group mean μb←1K∑i=1Krb(i) _b← 1K _i=1^Kr_b^(i); 22 Group std σb←(1K∑i=1K(rb(i)−μb)2)1/2 _b← ( 1K _i=1^K (r_b^(i)- _b )^2 )^1/2; 23 [DP] for candidate i=1i=1 to K do 24 Per-token advantage (broadcast across tokens) b(i)←rb(i)−μbσb+10−8 1 A_b^(i)← r_b^(i)- _b _b+10^-8\, 1; 25 Augment rollout (b,i)(b,i) in ℬrollB_roll with (b(i),ℓref,b(i)) ( A_b^(i),\, _ref,b^(i) ); 26 end for 27 end for 28 Flatten ℬroll←(n,n,ℓold(n),n,ℓref(n))n=1BKB_roll← \ ( x_n, y_n, _old^(n), A_n, _ref^(n) ) \_n=1^BK; 29 — Training via GRPO — 30 for e=1e=1 to EGRPOE_GRPO do 31 Shuffle ℬrollB_roll and partition into minibatches ℬ\B\; 32 [DP] foreach minibatch ℬB do 33 [DP] foreach rollout n∈ℬn do 34 35 PAR 36 37 laneActorlaneActoractor path 38 [TP, P, SP, CP, EP] 39 Current log-probs ℓθ(n)←logπθ(n∣n) _θ^(n)← _θ( y_n x_n); 40 Importance ratios n←exp(ℓθ(n)−ℓold(n)) ρ_n← \! ( _θ^(n)- _old^(n) ); 41 Per-token KL approximation n←exp(ℓref(n)−ℓθ(n))−(ℓref(n)−ℓθ(n))−1 d_n← \! ( _ref^(n)- _θ^(n) )- ( _ref^(n)- _θ^(n) )-1; 42 end foreach 43 Number of tokens in batch Z←∑n∈ℬ|n|Z← _n | y_n|; 44 45 PAR 46 47 laneActorlaneActoractor path 48 Compute GRPO loss: 49 ℒGRPO=−1Z∑n∈ℬ∑t=1|n|[min(ρn,tAn,t,L_GRPO=- 1Z _n _t=1^| y_n| [ ( _n,tA_n,t, 50 clip(ρn,t,1±ε)An,t)−βdn,t] \ \ \ \ \ \ \ \ \ clip( _n,t,1± )\,A_n,t)-β\,d_n,t ]; 51 [TP, P, SP, CP, EP] 52 Backpropagate & update on θ to minimize ℒGRPOL_GRPO; 53 end foreach 54 end for 55 end foreach Algorithm 4 Algorithmic and Mathematical Specification of DPO Input: KL coefficient β, training epochs EDPOE_DPO; preference dataset =(n,w(n),l(n))n=1ND=\! \ ( x_n, y_w^(n), y_l^(n) ) \_n=1^N. Output: Trained parameters θ. 1 — Training via DPO — 2 for e=1e=1 to EDPOE_DPO do 3 Shuffle D and partition into minibatches ℬ\B\; 4 [DP] foreach minibatch ℬB do 5 [DP] foreach (n,w(n),l(n))∈ℬ ( x_n, y_w^(n), y_l^(n) ) do 6 [inter-model concurrency] 7 8 PAR 9 10 laneActorlaneActoractor path 11 [TP, P, SP, CP, EP] Actor log-probs: 12 ℓθ,w(n)←logπθ(w(n)∣n) _θ,w^(n)← _θ ( y_w^(n) x_n ) (preferred completion); 13 ℓθ,l(n)←logπθ(l(n)∣n) _θ,l^(n)← _θ ( y_l^(n) x_n ) (dispreferred completion); 14 15 PAR 16 17 laneReferencelaneReferencereference path 18 [TP, P, SP, CP, EP] Reference log-probs: 19 ℓref,w(n)←logπref(w(n)∣n) _ref,w^(n)← _ref ( y_w^(n) x_n ) (preferred completion); 20 ℓref,l(n)←logπref(l(n)∣n) _ref,l^(n)← _ref ( y_l^(n) x_n ) (dispreferred completion); 21 Trajectory-level log-ratio difference: hn←⊤(ℓθ,w(n)−ℓref,w(n))−⊤(ℓθ,l(n)−ℓref,l(n))h_n← 1 \! ( _θ,w^(n)- _ref,w^(n) )- 1 \! ( _θ,l^(n)- _ref,l^(n) ); 22 end foreach 23 24 PAR 25 26 laneActorlaneActoractor path 27 Compute DPO loss, where σ(⋅)σ(·) is the sigmoid: 28 ℒDPO=−1|ℬ|∑n∈ℬlogσ(βhn)L_DPO=- 1|B| _n σ (β\,h_n ); 29 [TP, P, SP, CP, EP] 30 Backpropagate & update on θ to minimize ℒDPOL_DPO; 31 end foreach 32 end for The intra-model annotations are attached only to operations that invoke a Transformer model or backpropagate through one. For independent samples, prompts, candidates, rollouts, or preference pairs, DP is the natural default because these units have no semantic dependency and can be assigned to different replicas with only later reductions for losses, statistics, or gradients. For model invocations, TP and P are marked whenever the actor, critic, reward, or reference model may need to be sharded to fit memory or reduce single-invocation latency; however, they are conditional because their collectives, activation transfers, and pipeline bubbles can hurt throughput when the model already fits on one device. CP is marked on long-context forward or training invocations, including autoregressive decode, because KV/attention memory of RLM rollouts may exceed single-device capacity. SP is marked only on trainable teacher-forced forward/backward paths, not on ordinary forward-only Assessment or decode-time Generation, because its main benefit is reducing stored activation memory during Training. EP is marked only as an MoE-dependent option: it applies if the corresponding actor, critic, reward, or reference model contains routed experts, where it reduces per-device expert storage and compute but introduces routing and all-to-all communication. VI Analysis of Existing Models & Designs We also analyze existing models and frameworks. VI-A Reasoning Language Models Model / family Access Scale / arch. Post-training signal Selected system details General aligned large language models (LLMs) InstructGPT [192] Closed 1.3B, 6B, 175B dense PPO-like (SFT + RM + PPO) Canonical actor–reward–reference PPO loop. GPT-4 [4] Closed Undisclosed PPO-like (rule-based rewards) Frontier-scale alignment; limited public systems detail. Gemini 1.x [12] Closed Undisclosed PPO-like (iterative RM refinement) Repeated RM/RL cycles increase assessment cost. Claude 3 [13, 21] Closed Undisclosed PPO-like (RLAIF / Constitutional AI) AI-generated preference labels from written principles. Reka [191] Mixed 7B, 21B dense PPO-like (multi-round RLHF) Standard PPO-style alignment at moderate scale. InternLM2 [59] Open 1.8B, 7B, 20B dense PPO-like (conditional RM) Multiple domain-specific reward heads. Zephyr [229, 114] Open MoE variant reported DPO-like (ORPO) No explicit reward model or PPO loop. Phi-3 [2] Open 3.8B, 7B, 14B dense DPO-like SFT-like alignment cost. Phi-4 [3] Open 7B, 14B dense DPO-like (RLAIF data) Preference tuning over AI/human feedback data. ChatGLM [284] Open 6B, 9B dense Mixed (PPO-like vs. DPO-like) Controlled comparison of online vs. DPO training. Gemma 2 [204] Open 2B, 9B, 27B dense PPO-like (Bradley–Terry RM) Reward-model alignment. Starling-7B [285] Open 7B dense PPO-like (RLAIF; Plackett–Luce RM) Ranking-based RM; partial model updates reduce memory. Hermes 3 [226] Open 8B, 70B, 405B dense DPO-like (LoRA-DPO) Only LoRA/adapters are trained; optimizer memory is small. Athene-70B [91] Open 70B dense PPO-like (details limited) A model with preference/safety tuning. Llama 3 & 3.1 [104] Open 8B, 70B, 405B dense DPO-like (RM + rejection sampling) — Qwen2 & Qwen2.5 [258] Open 0.5B–72B dense; 57B MoE (14B active) DPO-like (offline + online refresh) New preference pairs are created from online model samples. Nemotron-4 340B [5] Open 340B-class model DPO-like (RPO) Quality-aware preference optimization without PPO. DeepSeek-V3 [156] Open 671B MoE (∼ 37B active) Hybrid alignment MoE, MLA, FP8, MTP, and DualPipe. Llama 4 Scout [171] Open ∼ 109B MoE (17B active) Hybrid alignment Long context. Llama 4 Maverick [171] Open ∼ 400B MoE (17B active) Hybrid alignment MoE and multimodality. Reasoning language models (RLMs) OpenAI o1 [186] Closed Undisclosed Hybrid RL (CoT) “Reasoning tokens” become inference-time cost. OpenAI o3 [188] Closed Undisclosed Hybrid RL (CoT + tools) Training-time RL and tool-using inference both dominate. OpenAI o4-mini [188] Closed Undisclosed Hybrid RL (CoT + tools) Lower-cost reasoning model; latency/quality trade-off. GPT-5 [220] Closed Undisclosed Hybrid (fast/thinking router) Runtime routes between fast and reasoning modes. GPT-5.4 Thinking [190] Closed Undisclosed Hybrid (coding + agents) Agentic workflow. GPT-5.5 Thinking [189] Closed Undisclosed Hybrid (agentic reasoning) Higher autonomy increases orchestration cost. Claude 3.7 Sonnet [14] Closed Undisclosed PPO-like/RLAIF + thinking mode Same model supports quick and extended reasoning. Claude 4 family [15] Closed Undisclosed PPO-like/RLAIF + thinking mode Reasoning budget is a user/system cost knob. Gemini 2.5 Pro [71] Closed Sparse MoE; multimodal Hybrid RL (thinking mode) Long context. Gemini 2.5 Flash [71] Closed Sparse MoE with a lower-latency variant Hybrid RL (low-cost thinking) Cheaper reasoning. Gemini 2.5 Deep Think [102] Closed Gemini 2.5 family Search-like, parallel reasoning Multiple reasoning paths increase per-query compute. DeepSeek-R1 [107] Open 671B MoE / ∼ 37B active GRPO-like (RLVR; no critic) Removes VψV_ψ; generation remains bottleneck. DAPO [265] Open 32B dense GRPO-like (DAPO; no critic) Critic-free long-CoT RL. Qwen3 [257] Open Dense/MoE, 0.6B–235B Hybrid (thinking/non-thinking) Variable reasoning length. Kimi k1.5 [86] Closed Undisclosed; long-context multimodal DPO-like/custom RL (KL-DPO) Long trajectories dominate training and serving cost. Kimi K2 [19] Open 1T MoE (32B active) Agentic RL-like (tool environments) — Kimi K2.5 [18] Open MoE; multimodal agentic extension Agentic RL-like (visual tools) Adds multimodal environment overheads. Phi-4-reasoning [1] Open 14B dense DPO-like (RLVR) Small strong STEM model at low serving cost. rStar-Math [106] Open 7B-scale policy/reward components Search/RLVR-like (MCTS + PRM) Search improves math but increases inference compute. Grok 4 [249] Closed Undisclosed Hybrid RL — Grok 4 Heavy [249] Closed Undisclosed; multi-agent variant Agentic / parallel reasoning Parallel agents increase inference compute. TABLE X: A comparison of representative general aligned LLMs and RLMs. The upper block lists general aligned LLMs, which primarily target instruction following, safety, preference alignment, and broad assistant quality. The lower block lists RLMs, which explicitly target reasoning trajectories, long-CoT, RLVR/verifiable rewards, search, tools, multi-agent inference, or controllable test-time computation. The “Post-training signal” column follows the taxonomy used in this paper: PPO-like methods use reward-model- or RLAIF-based policy optimization; GRPO-like methods use critic-free grouped RL/RLVR; DPO-like methods use direct preference optimization; Hybrid denotes mixed or undisclosed pipelines; Search/Agentic denotes explicit search, tools, environments, or multi-agent reasoning. Proprietary details are based on public reports and should be treated as approximate. We compare representative post-trained LLMs and RLMs in Table X. We classify a model as a general aligned LLM if post-training primarily improves instruction-following, helpfulness, safety, or general assistant behavior, even when the model can solve reasoning tasks. We classify a model as an RLM if its training or inference pipeline explicitly targets reasoning trajectories, verifiable rewards, long CoT, search, tool use, or controllable test-time computation. This distinction is often blurred in practice: recent systems increasingly unify fast response modes and slow reasoning modes inside a single routed or hybrid model family. Overall, the comparison shows a shift from reference-driven alignment as preference optimization to alignment as reasoning-time computation. Early RL-enhanced LLMs primarily used PPO-like RLHF or more lightweight DPO-style preference optimization to improve instruction following, safety, and general assistant quality, whereas RLMs increasingly optimize or allocate compute to long-CoT trajectories, RLVR, search, tools, agentic environments, or hybrid fast/thinking modes. The training signal is often more predictive of reasoning gains than parameter count alone: exact-answer rewards, unit tests, process/preference models, MCTS, or environment feedback can make smaller specialized models competitive on narrow math/code tasks, while frontier systems retain broader coverage through scale, data diversity, and tool integration. Architecturally, sparse MoE has become central for open frontier models because it increases total capacity at moderate active-parameter cost, but it shifts systems pressure toward expert parallelism, routing balance, all-to-all communication, and memory placement; dense models remain simpler and more predictable for deployment. VI-B Training & Inference Frameworks for RLMs Intra-Model Parallelism Inter-Model Parallelism Framework Remarks DP Z1 Z2 Z3 TP CP EP P AC Share Dis- agg. Stage Fusion Hybrid Async TRL [231] Accelerate; TP/vLLM mostly rollout; CP via FSDP2/Ulysses ColossalChat [267] Colossal-AI HybridParallel + Gemini; TP/P/SP, Z1/2 and Z3 DeepSpeed-Chat [261] Train ZeRO-2/3; Hybrid Engine uses TP mainly for rollout OpenRLHF [117] Train: ZeRO-3/AutoTP; rollout: vLLM TP/P; RingAttention HybridFlow / verl [217] Train: FSDP or Megatron; rollout: vLLM/SGLang/TensorRT-LLM NeMo RL [214, 184] Train: DTensor or Megatron; rollout: vLLM/Megatron/SGLang ReaL [169] Per-stage DP/TP/P/SP plans; ZeRO backend not stage-enumerated RLHFuse [282] Scheduling layer; inter-stage sample fusion + intra-stage fused P FlexRLHF [251] DeepSpeed-based; AC-share/nonshare; interleaving/disaggregated layouts AReaL [92] Train: Megatron/FSDP/Archon; infer: SGLang default, vLLM optional MindSpeed RL [89] Ascend stack; co-card/decoupled deploy, repartition, partial rollout slime [287] Megatron train + SGLang rollout ROLL [240] train: DeepSpeed/FSDP2/Megatron; infer: vLLM/SGLang/Megatron StreamRL [281] outside system focus Disaggregation-first; stream generation; full overlap in async mode AsyncFlow [111] outside system focus Distributed data/param streaming; producer–consumer async Laminar [216] outside system focus Relay-worker parameter service; trajectory-level async repack StaleFlow [145] outside system focus Data+parameter servers; explicit global staleness control TABLE XI: Comparison of public training / inference frameworks for LLM and RLM post-training under the systems taxonomy. Intra-Model Parallelism: DP=data parallel; Z1/Z2/Z3=ZeRO/FSDP-like sharding levels (optimizer / optimizer+gradients / parameters+gradients+optimizer; FSDP2 is counted under Z3); TP=tensor parallel; CP=context or long-sequence parallelism; EP=expert parallelism; P=pipeline parallelism. Inter-Model Parallelism: AC Share=shared actor/critic parameters or value-head mode; Disagg.=separate services or device groups for rollout, reward, or training; Stage Fusion=explicit overlap or fusion across normally sequential RL stages or subtasks; Hybrid=stage-specific backends/layouts or runtime resharding/refit; Async=asynchronous or bounded-stale execution. Symbols: documented support; indirect, partial, backend-specific, or train-vs-rollout-specific support; no documented public support; undocumented or unclear in public materials. Table XI shows that modern RL-LLM frameworks differ primarily by how aggressively they separate, specialize, and overlap the rollout and training workloads. In practice, wall-clock time is often dominated by rollout/generation under long-output reasoning settings, while gradient updates are comparatively more parallelizable; consequently, the highest-throughput frameworks either (a) maximize rollout throughput via inference engines and stage-specific parallelism, or (b) overlap generation with evaluation and training via stage fusion or asynchronous execution. Library-centric stacks such as TRL [231], DeepSpeed-Chat [261], and ColossalChat [267] are easiest to use and rely mainly on DP/ZeRO/FSDP-style sharding [201, 277] plus optional fast rollout engines such as vLLM [133], but they expose limited inter-model scheduling. Orchestrator-centric stacks such as OpenRLHF [117], HybridFlow/verl [217], SkyRL [181], NeMo RL [214, 184], slime [287], ROLL [240] and ReaL [169] treat RLHF as a multi-role dataflow. They provide explicit placement and resource management for actor, rollout, reward, reference, critic, and data-buffer components, and commonly combine training backends such as DeepSpeed ZeRO, PyTorch FSDP/FSDP2, or Megatron-Core that support TP/P/CP/DP/EP [219, 176, 160, 125], with rollout engines such as vLLM, SGLang, or TensorRT-LLM [133, 278, 185]. These systems often support heterogeneous parallelism across roles, weight synchronization, and resharding between training and rollout with explicit weight synchronization or resharding [169, 217]. Scheduling-centric systems such as RLHFuse [282], Pipe-RLHF [262], StreamRL [281], AReaL [92], AsyncFlow [111], Laminar [216], StaleFlow [145], and MindSpeed RL [89] are best viewed as emphasizing scheduling policies rather than defining a disjoint framework class. In practice, many orchestrator-centric frameworks can also manually schedule workloads or incorporate similar optimizations. These systems target the dominant bottlenecks of long-tailed autoregressive rollout by improving utilization by stage fusion [282], streaming and disaggregation [281], dynamic load balancing [111], and bounded-stale asynchronous execution [180, 92, 145], at the cost of more complex queues, weight-version management, and convergence validation. Practically, small experiments should favor library-centric stacks; memory-bounded large-model runs should favor mature state-sharding backends such as FSDP/ZeRO or Megatron [201, 277, 219]; RL post-training workloads whose wall-clock time is dominated by rollout/generation should favor disaggregated vLLM/SGLang-style rollout [133, 278, 281]; and frontier-scale reasoning RL increasingly requires async or streaming systems that explicitly control staleness and generation-length skew [92, 216, 145]. Across frameworks, data parallelism remains a common outer abstraction, but modern RL post-training stacks increasingly compose multiple forms of parallelism across roles and stages. FSDP/ZeRO-style state sharding and Megatron-FSDP occupy the same broad design space for sharding optimizer states, gradients, and parameters, while Megatron-style backends can further combine DP with TP, P, CP, and EP when model size, sequence length, or MoE require it. Thus, the main systems issue is not a fixed choice between FSDP/ZeRO and Megatron, but how the framework places heterogeneous roles, synchronizes or reshards weights between the trainer and rollout workers, and balances optimizer-state memory, communication, KV-cache pressure, and variable rollout lengths. In this sense, orchestrator-centric and scheduling-centric systems are better viewed as different emphases within the same design space: the former provides the multi-role execution substrate, while the latter emphasizes utilization policies for long-tailed generation workloads. VII Research Opportunities We briefly outline potential opportunities for future research in the performance aspects of RLMs. Resource-aware test-time compute. RLMs increasingly trade inference-time compute for accuracy through long-CoT generation, self-consistency, search, tool use, and other strategies, exploring different scaling regimes and tradeoffs [247, 221, 115, 270, 225, 130, 115, 188, 71]. Future systems could treat reasoning as a constrained optimization problem: maximize expected utility subject to token, latency, memory, tool-call, and energy budgets. This opens the door to interesting research into making reasoning budget a first-class scheduling variable, and into online predictors of task difficulty, verifier value, branch utility, and stopping time. Efficient RLM execution structures beyond chains. RLM reasoning is usually represented as a linear token sequence. Prior works such as Tree of Thoughts, Graph of Thoughts, and Hypergraph-of-Thoughts work shows that nonlinear reasoning structures can improve search and aggregation in-context [260, 179, 33, 259, 26, 45]. Some efforts to distill this behavior into weights have been made into this direction by harnessing aggregation during fine-tuning [8, 150, 275, 151]. Efficient and effective integration of such structures and ideas in the RL execution pipeline is an interesting novel direction in the science of RLM performance and parallel design. Automatic selection of parallelization schemes. One concrete idea of how to enhance the parallel design of RLM pipelines, is to provide an effective way of automating the intra-parallelization of each model invocation based on the available hardware and cluster conditions. There have been works into this direction, for example AutoDDL [65] and PyTorch/XLA SPMD [127], but they focus on an individual Transformer invocations and not whole pipelines [65, 127]. Automatic derivation of reasoning topologies and schedules. In the RLM designs that harness explicit structures such as MCTS, most methods still use hand-designed chains, trees, beams, and others [26]. A potential direction is to automatically derive both the reasoning topology and the execution schedule from aspects such as task difficulty, model uncertainty, verifier availability, hardware availability and performance properties, and others. Dynamic graph representations and algorithms could offer useful templates for such planners in terms of performance-focused aspects such as algorithm design [32], effective task graph decompositions and partitioning [101, 58], scheduling [77, 27], and others. Example questions to pursuit would be how many branches to explore, when to prune a branch, when to call a verifier, when to merge thoughts, how to map the resulting DAG to device groups, and others. RLVR beyond final-answer rewards. RLVR scales reasoning post-training through exact-answer checks, unit tests, and task-specific verifiers, but its mechanisms remain incompletely understood; for example, weak or spurious rewards can sometimes still improve reasoning, and outcomes are model-family dependent [212]. Future work could focus on developing reliable and efficient process-level rewards, building upon existing work [165, 271, 69, 241, 149, 273, 129], and potentially even extend them to consider the structural aspects of the reasoning process beyond chains. Since process rewards create many assessment nodes, their usefulness will depend on performance-oriented policies that involve caching, batching, placement, and overlap with generation. Harnessing data analytics for enhanced reasoning. A rich landscape of research opportunities exist at the overlap of reasoning and data analytics. One could delegate certain parts of reasoning tasks that are hard to instill into model weights to existing algorithms and frameworks; examples are graph mining and analytics [16, 193, 36, 99, 46]. One could also harness graph learning for reasoning supervision, i.e., graph neural networks (GNNs) [131, 248, 274, 283, 50] and broader graph representation learning methods [57, 62, 28, 110, 100] may help score branches, help constructing process rewards, or predict useful tool calls, based on learning useful reasoning patterns. The systems challenge is to integrate such designs efficiently [40, 49] without adding bottlenecks to the Generation–Assessment–Training loop or broader MCTS. Enhancing data analytics pipelines. On the other hand, data analytics frameworks [236, 95, 54, 137, 109, 32, 52, 31] as well as databases [10, 11, 33, 78, 35, 41] could also integrate reasoning LLMs and agents for more effective data processing, especially at the user–framework interface. Different designs have already been proposed to integrate LLMs into analytics architectures [244, 42, 139, 64, 269, 55, 67, 199, 85]. This includes document analysis (e.g., Aryn [9], DocETL [211], Palimpzest [158], PalimpChat [159]), tabular data (e.g., InsightPilot [164], CoddLLM [272], Pneuma [23], Chat2data [276], LOTUS [194], DB-GPT [254]), video processing [167], and others [80]. Here, harnessing RLMs and their efficient integration into such frameworks is an interesting novel research direction. Asynchronous and stale-data RL. Online RLM training is limited by long-tailed autoregressive generation. Asynchronous systems such as AReaL, AsyncFlow, Laminar, StreamRL, and StaleFlow show that bounded staleness and streaming rollouts can improve utilization [92, 111, 216, 281, 145]. The open problem is to characterize in a more rigorous way when and to what degree involve staleness – for example, when stale rollouts remain useful, how to correct policy drift, and how to trade staleness against throughput without degrading reasoning quality. Retrieval, tools, and external state as operators. Effective and efficient integration of tools into the reasoning process is another interesting research opportunity [215, 161]. Various tool calls (e.g., retrieval [206, 93, 126, 142, 44], simulators, verifiers [105, 47], agent calls [245, 119, 70, 48, 29], etc.) could be modeled as operators in the same execution graph as actor generation, reward evaluation, and training. Retrieval-augmented and graph-based LLM methods such as Topologies of Reasoning provide starting points for such integration [45]. Harnessing HPC architectures as well as emerging hardware. RLMs stress hardware through aspects such as – among others – its long outputs [177], large KV caches [144, 218, 157], repeated verifier calls, tool use, and many samples per prompt. Beyond established forms of parallelism, promising directions include harnessing emerging hardware such as processing-in-memory [7, 97, 174, 209, 43], RDMA and SmartNICs [94, 38, 82, 81, 39, 207, 222], serverless architectures [72, 128, 152, 168, 210, 112, 152], next-generation interconnects [135, 51, 30, 136], chiplets [121, 90, 172, 163, 166, 148, 124], and others [37, 34, 122, 123, 98]. Reasoning for discovering novel optimizations. Inspired by the schemes such as AlphaTensor [88] and enabled by the emergence of RLMs with their innate coding capabilities [202, 234, 66, 22, 205, 84], such models could also help discover novel kernels and other performance-centric schemes, including collectives, placement heuristics, and routing policies, following recent LLM-driven algorithm-discovery systems such as AlphaEvolve [182]. Evaluation and reproducibility. The evaluation of RLMs involves a plethora of aspects such as reasoning budget, sampling strategy, rollout length, staleness, and metrics associated with whole RLM components and subsystems such as tools, retrieval, and verification. It poses an opportunity for designing effective evaluation pipelines, reusing and extending evaluation methodologies for other domains such as parallel programming [113, 24]. Shared-weight RLHF and snapshot-consistent rollout workers. Another example concrete systems opportunity is to reduce the cost of synchronizing actor weights in disaggregated or asynchronous RLHF pipelines. Current systems often push full policy snapshots from the learner to rollout workers, which creates latency, bandwidth pressure, and synchronization stalls at large model scale. A promising alternative is a shared-weight design in which rollout actors access versioned trainer-resident parameter slots, switching only at rollout boundaries to preserve snapshot consistency. Such a system could use read-copy-update-style double or triple buffering, version counters, and publication fences so that actors never observe partially written parameters. On a single node, this suggests CUDA IPC [195] or peer-to-peer refresh over NVLink or PCIe [143]; across nodes, one can harness GPUDirect RDMA [196] or NVSHMEM-style one-sided communication [116]. VIII Conclusion Reasoning Language Models (RLMs) and their underlying paradigms such as Reinforcement Learning with Verifiable Rewards (RLVR) are not only an algorithmic advance, but also an enormous parallel and distributed systems challenge due to their compute requirements. Our work systematizes the RL-for-LLM pipeline, analyzes PPO-, GRPO-, and DPO-like frameworks through work–depth–memory complexity, and develops a taxonomy of intra- and inter-model parallelism strategies for scalable RLM training and inference. The result is a unified performance vocabulary for understanding where computation, memory, and critical-path bottlenecks arise, and for offering opportunities for performance optimizations while considering performance-critical aspects such as autoregressive rollout generation, auxiliary-model assessment, trainable actor/critic updates, long-context memory, model-state sharding, placement, fusion, and bounded-stale execution. By making these trade-offs explicit, our analysis provides a foundation for designing the next generation of RLM systems. Acknowledgements We thank Hussein Harake, Colin McMurtrie, Mark Klein, Angelo Mangili, and the whole CSCS team granting access to the Ault and Alps machines, and for their excellent technical support. We gratefully acknowledge Polish high-performance computing infrastructure PLGrid (HPC Center: ACK Cyfronet AGH) for providing compute facilities within computational grant no. PLG/2026/019437; we also thank Łukasz Flis and the whole Cyfronet team for their excellent technical support. We thank Timo Schneider for help with infrastructure at SPCL. This project received funding from the European Research Council (Project PSAP, No. 101002047), and the European High-Performance Computing Joint Undertaking (JU) under grant agreement No. 955513 (MAELSTROM). This project was supported by the ETH Future Computing Laboratory (EFCL), financed by a donation from Huawei Technologies. We acknowledge the Swiss AI Initiative for the computational grant. A language model served as an editorial tool for this manuscript, while ideas and content are original work of the authors. References [1] M. Abdin, S. Agarwal, A. Awadallah, V. Balachandran, H. Behl, L. Chen, G. de Rosa, S. Gunasekar, M. Javaheripi, N. Joshi et al., “Phi-4-reasoning Technical Report,” Apr. 2025, arXiv:2504.21318. [Online]. Available: https://arxiv.org/abs/2504.21318 [2] M. Abdin, J. Aneja, H. Awadalla, A. Awadallah, A. A. Awan, N. Bach, A. Bahree, A. Bakhtiari, J. Bao, H. Behl et al., “Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone,” Aug. 2024, arXiv:2404.14219. [Online]. Available: https://arxiv.org/abs/2404.14219 [3] M. Abdin, J. Aneja, H. Behl, S. Bubeck, R. Eldan, S. Gunasekar, M. Harrison, R. J. Hewett, M. Javaheripi, P. Kauffmann et al., “Phi-4 Technical Report,” Dec. 2024, arXiv:2412.08905. [Online]. Available: https://arxiv.org/abs/2412.08905 [4] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman et al., “GPT-4 Technical Report,” Mar. 2024, arXiv:2303.08774. [Online]. Available: https://arxiv.org/abs/2303.08774 [5] B. Adler, N. Agarwal, A. Aithal, D. H. Anh, P. Bhattacharya, A. Brundyn, J. Casper, B. Catanzaro, S. Clay, J. Cohen et al., “Nemotron-4 340B Technical Report,” Aug. 2024, arXiv:2406.11704. [Online]. Available: https://arxiv.org/abs/2406.11704 [6] A. Ahmadian, C. Cremer, M. Gallé, M. Fadaee, J. Kreutzer, O. Pietquin, A. Üstün, and S. Hooker, “Back to Basics: Revisiting REINFORCE-Style Optimization for Learning from Human Feedback in LLMs,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ser. ACL ’24, L.-W. Ku, A. Martins, and V. Srikumar, Eds. Bangkok, Thailand: Association for Computational Linguistics, Aug. 2024, p. 12 248–12 267. [Online]. Available: https://aclanthology.org/2024.acl-long.662/ [7] J. Ahn, S. Yoo, O. Mutlu, and K. Choi, “PIM-Enabled Instructions: A Low-Overhead, Locality-Aware Processing-In-Memory Architecture,” SIGARCH Comput. Archit. News, vol. 43, no. 3S, p. 336–348, Jun. 2015. [Online]. Available: https://doi.org/10.1145/2872887.2750385 [8] R. Ai, Y. Pan, D. Simchi-Levi, M. Tambe, and H. Xu, “Beyond Majority Voting: LLM Aggregation by Leveraging Higher-Order Information,” May 2026, arXiv:2510.01499. [Online]. Available: https://arxiv.org/abs/2510.01499 [9] E. Anderson, J. Fritz, A. Lee, B. Li, M. Lindblad, H. Lindeman, A. Meyer, P. Parmar, T. Ranade, M. A. Shah, B. Sowell, D. Tecuci, V. Thapliyal, and M. Welsh, “The Design of an LLM-Powered Unstructured Analytics System,” in Proceedings of the 15th Annual Conference on Innovative Data Systems Research, ser. CIDR ’25. Amsterdam, Netherlands: VLDB Endowment, Jan. 2025, p. 1–10. [Online]. Available: https://vldb.org/cidrdb/2025/the-design-of-an-llm-powered-unstructured-analytics-system.html [10] R. Angles and C. Gutierrez, “Survey of Graph Database Models,” ACM Comput. Surv., vol. 40, no. 1, p. 1:1–1:39, Feb. 2008. [Online]. Available: https://doi.org/10.1145/1322432.1322433 [11] —, “An Introduction to Graph Data Management,” in Graph Data Management: Fundamental Issues and Recent Developments, ser. Data-Centric Systems and Applications (DCSA), G. Fletcher, J. Hidders, and J. L. Larriba-Pey, Eds. Berlin, Germany: Springer, Nov. 2018, p. 1–32. [Online]. Available: https://link.springer.com/chapter/10.1007/978-3-319-96193-4_1 [12] R. Anil, S. Borgeaud, J.-B. Alayrac, R. S. Jiahui Yu, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican, D. Silver et al., “Gemini: A Family of Highly Capable Multimodal Models,” May 2025, arXiv:2312.11805. [Online]. Available: https://arxiv.org/abs/2312.11805 [13] Anthropic, “Claude, Large Language Model Conversational AI,” https://claude.ai/new, 2024, [Accessed June 11, 2026]. [14] —, “Claude 3.7 Sonnet and Claude Code,” https://w.anthropic.com/news/claude-3-7-sonnet, Feb. 2025, [Accessed June 12, 2026]. [15] —, “Building with Extended Thinking,” https://platform.claude.com/docs/en/build-with-claude/extended-thinking, 2026, [Accessed June 12, 2026]. [16] G. Atluri, A. Karpatne, and V. Kumar, “Spatio-Temporal Data Mining: A Survey of Problems and Methods,” ACM Comput. Surv., vol. 51, no. 4, p. 83:1–83:41, Aug. 2018. [Online]. Available: https://doi.org/10.1145/3161602 [17] G. Bai, Z. Chai, C. Ling, S. Wang, J. Lu, N. Zhang, T. Shi, Z. Yu, M. Zhu, Y. Zhang, X. Song, C. Yang, Y. Cheng, and L. Zhao, “Beyond Efficiency: A Systematic Survey of Resource-Efficient Large Language Models,” Dec. 2024, arXiv:2401.00625. [Online]. Available: https://arxiv.org/abs/2401.00625 [18] T. Bai, Y. Bai, Y. Bao, S. Cai, Y. Cao, Y. Charles, H. Che, C. Chen, G. Chen, H. Chen et al., “Kimi K2.5: Visual Agentic Intelligence,” Feb. 2026, arXiv:2602.02276. [Online]. Available: https://arxiv.org/abs/2602.02276 [19] Y. Bai, Y. Bao, Y. Charles, C. Chen, G. Chen, H. Chen, H. Chen, J. Chen, N. Chen, R. Chen et al., “Kimi K2: Open Agentic Intelligence,” Feb. 2026, arXiv:2507.20534. [Online]. Available: https://arxiv.org/abs/2507.20534 [20] Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, N. Joseph et al., “Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback,” Apr. 2022, arXiv:2204.05862. [Online]. Available: https://arxiv.org/abs/2204.05862 [21] Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon et al., “Constitutional AI: Harmlessness from AI Feedback,” Dec. 2022, arXiv:2212.08073. [Online]. Available: https://arxiv.org/abs/2212.08073 [22] R. Bairi, A. Sonwane, A. Kanade, V. D. C., A. Iyer, S. Parthasarathy, S. Rajamani, B. Ashok, and S. Shet, “CodePlan: Repository-Level Coding using LLMs and Planning,” Proc. ACM Softw. Eng., vol. 1, no. FSE, p. 31:1–31:24, Jul. 2024. [Online]. Available: https://doi.org/10.1145/3643757 [23] M. I. L. Balaka, D. Alexander, Q. Wang, Y. Gong, A. Krisnadhi, and R. Castro Fernandez, “Pneuma: Leveraging LLMs for Tabular Data Representation and Retrieval in an End-to-End System,” Proc. ACM Manag. Data, vol. 3, no. 3, p. 200:1–200:28, Jun. 2025. [Online]. Available: https://doi.org/10.1145/3725337 [24] T. Ben-Nun, M. Besta, S. Huber, A. N. Ziogas, D. Peter, and T. Hoefler, “A Modular Benchmarking Infrastructure for High-Performance and Reproducible Deep Learning,” in Proceedings of the IEEE 33rd International Parallel and Distributed Processing Symposium, ser. IPDPS ’19. Rio de Janeiro, Brazil: IEEE Press, May 2019, p. 66–77. [Online]. Available: https://ieeexplore.ieee.org/document/8821020 [25] T. Ben-Nun and T. Hoefler, “Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis,” ACM Comput. Surv., vol. 52, no. 4, p. 65:1–65:43, Aug. 2019. [Online]. Available: https://doi.org/10.1145/3320060 [26] M. Besta, J. Barth, E. Schreiber, A. Kubicek, A. Catarino, R. Gerstenberger, P. Nyczyk, P. Iff, Y. Li, S. Houliston, T. Sternal, M. Copik, G. Kwaśniewski, J. Müller, Łukasz Flis, H. Eberhard, Z. Chen, H. Niewiadomski, and T. Hoefler, “Reasoning Language Models: A Blueprint,” Jun. 2025, arXiv:2501.11223. [Online]. Available: https://arxiv.org/abs/2501.11223 [27] M. Besta, A. Carigiet, K. Janda, Z. Vonarburg-Shmaria, L. Gianinazzi, and T. Hoefler, “High-Performance Parallel Graph Coloring with Strong Guarantees on Work, Depth, and Quality,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, ser. SC ’20. Atlanta, GA, USA: IEEE Press, Nov. 2020, p. 99:1–99:17. [Online]. Available: https://ieeexplore.ieee.org/document/9355236 [28] M. Besta, A. C. Catarino, L. Gianinazzi, N. Blach, P. Nyczyk, H. Niewiadomski, and T. Hoefler, “HOT: Higher-Order Dynamic Graph Representation Learning with Efficient Transformers,” in Proceedings of the Second Learning on Graphs Conference (LOG ’23), ser. Proceedings of Machine Learning Research, S. Villar and B. Chamberlain, Eds., vol. 231. Virtual Event: PMLR, Nov. 2023, p. 15:1–15:20. [Online]. Available: https://proceedings.mlr.press/v231/besta24a.html [29] M. Besta, S. Chandran, R. Gerstenberger, M. Lindner, M. Chrapek, S. H. Martschat, T. Ghandi, P. Iff, H. Niewiadomski, P. Nyczyk, J. Müller, and T. Hoefler, “Psychologically Enhanced AI Agents,” Sep. 2025, arXiv:2509.04343. [Online]. Available: https://arxiv.org/abs/2509.04343 [30] M. Besta, J. Domke, M. Schneider, M. Konieczny, S. Di Girolamo, T. Schneider, A. Singla, and T. Hoefler, “High-Performance Routing with Multipathing and Path Diversity in Ethernet and HPC Networks,” IEEE Transactions on Parallel and Distributed Systems, vol. 32, no. 4, p. 943–959, Apr. 2021. [Online]. Available: https://ieeexplore.ieee.org/document/9248644 [31] M. Besta, M. Fischer, T. Ben-Nun, D. Stanojevic, J. D. F. Licht, and T. Hoefler, “Substream-Centric Maximum Matchings on FPGA,” ACM Trans. Reconfigurable Technol. Syst., vol. 13, no. 2, p. 8:1–8:33, Apr. 2020. [Online]. Available: https://doi.org/10.1145/3377871 [32] M. Besta, M. Fischer, V. Kalavri, M. Kapralov, and T. Hoefler, “Practice of Streaming Processing of Dynamic Graphs: Concepts, Models, and Systems,” IEEE Transactions on Parallel and Distributed Systems, vol. 34, no. 6, p. 1860–1876, Jun. 2023. [Online]. Available: https://ieeexplore.ieee.org/document/9629281 [33] M. Besta, R. Gerstenberger, M. Fischer, M. Podstawski, N. Blach, B. Egeli, G. Mitenkov, W. Chlapek, M. Michalewicz, H. Niewiadomski, J. Mueller, and T. Hoefler, “The Graph Database Interface: Scaling Online Transactional and Analytical Graph Workloads to Hundreds of Thousands of Cores,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, ser. SC ’23. Denver, CO, USA: Association for Computing Machinery, Nov. 2023, p. 22:1–22:18. [Online]. Available: https://doi.org/10.1145/3581784.3607068 [34] M. Besta, R. Gerstenberger, P. Iff, P. Sonawane, J. G. Luna, R. Kanakagiri, R. Min, O. Mutlu, T. Hoefler, R. Appuswamy, and A. O Mahony, “Hardware Acceleration for Knowledge Graph Processing: Challenges & Recent Developments,” Nov. 2024, arXiv:2408.12173. [Online]. Available: https://arxiv.org/abs/2408.12173 [35] M. Besta, R. Gerstenberger, E. Peter, M. Fischer, M. Podstawski, C. Barthels, G. Alonso, and T. Hoefler, “Demystifying Graph Databases: Analysis and Taxonomy of Data Organization, System Designs, and Graph Queries,” ACM Comput. Surv., vol. 56, no. 2, p. 31:1–31:40, Sep. 2023. [Online]. Available: https://doi.org/10.1145/3604932 [36] M. Besta, R. Grob, C. Miglioli, N. Bernold, G. Kwasniewski, G. Gjini, R. Kanakagiri, S. Ashkboos, L. Gianinazzi, N. Dryden, and T. Hoefler, “Motif Prediction with Graph Neural Networks,” in Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, ser. KDD ’22. Washington, DC, USA: Association for Computing Machinery, Aug. 2022, p. 35–45. [Online]. Available: https://doi.org/10.1145/3534678.3539343 [37] M. Besta, S. M. Hassan, S. Yalamanchili, R. Ausavarungnirun, O. Mutlu, and T. Hoefler, “Slim NoC: A Low-Diameter On-Chip Network Topology for High Energy Efficiency and Scalability,” SIGPLAN Not., vol. 53, no. 2, p. 43–55, Mar. 2018. [Online]. Available: https://doi.org/10.1145/3296957.3177158 [38] M. Besta and T. Hoefler, “Fault Tolerance for Remote Memory Access Programming Models,” in Proceedings of the 23rd International Symposium on High-Performance Parallel and Distributed Computing, ser. HPDC ’14. Vancouver, BC, Canada: Association for Computing Machinery, Jun. 2014, p. 37–48. [Online]. Available: https://doi.org/10.1145/2600212.2600224 [39] —, “Active Access: A Mechanism for High-Performance Distributed Data-Centric Computations,” in Proceedings of the 29th ACM on International Conference on Supercomputing, ser. ICS ’15. Newport Beach, CA, USA: Association for Computing Machinery, Jun. 2015, p. 155–164. [Online]. Available: https://doi.org/10.1145/2751205.2751219 [40] —, “Parallel and Distributed Graph Neural Networks: An In-Depth Concurrency Analysis,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 5, p. 2584–2606, May 2024. [Online]. Available: https://ieeexplore.ieee.org/document/10443519 [41] M. Besta, P. Iff, F. Scheidl, K. Osawa, N. Dryden, M. Podstawski, T. Chen, and T. Hoefler, “Neural Graph Databases,” in Proceedings of the First Learning on Graphs Conference, ser. Proceedings of Machine Learning Research, B. Rieck and R. Pascanu, Eds., vol. 198. Virtual Event: PMLR, Dec. 2022. [Online]. Available: https://proceedings.mlr.press/v198/besta22a.html [42] M. Besta, Ł. Jarmocik, O. Hrycyna, S. Klaiman, K. Maczka, R. Gerstenberger, J. Müller, P. Nyczyk, H. Niewiadomski, and T. Hoefler, “GraphSeek: Next-Generation Graph Analytics with LLMs,” Mar. 2026, arXiv:2602.11052. [Online]. Available: https://arxiv.org/abs/2602.11052 [43] M. Besta, R. Kanakagiri, G. Kwaśniewski, R. Ausavarungnirun, J. Beránek, K. Kanellopoulos, K. Janda, Z. Vonarburg-Shmaria, L. Gianinazzi, I. Stefan, J. G. Luna, J. Golinowski, M. Copik, L. Kapp-Schwoerer, S. Di Girolamo, N. Blach, M. Konieczny, O. Mutlu, and T. Hoefler, “SISA: Set-Centric Instruction Set Architecture for Graph Mining on Processing-in-Memory Systems,” in Proceedings of the 54th Annual IEEE/ACM International Symposium on Microarchitecture, ser. MICRO ’21. Virtual Event: Association for Computing Machinery, Oct. 2021, p. 282–297. [Online]. Available: https://doi.org/10.1145/3466752.3480133 [44] M. Besta, A. Kubicek, R. Gerstenberger, M. Chrapek, R. Niggli, P. Okanovic, Y. Zhu, P. Iff, M. Podstawski, L. Weitzendorf, M. Chi, J. Gajda, P. Nyczyk, J. Müller, H. Niewiadomski, and T. Hoefler, “Multi-Head RAG: Solving Multi-Aspect Problems with LLMs,” Sep. 2025, arXiv:2406.05085. [Online]. Available: https://arxiv.org/abs/2406.05085 [45] M. Besta, F. Memedi, Z. Zhang, R. Gerstenberger, G. Piao, N. Blach, P. Nyczyk, M. Copik, G. Kwaśniewski, J. Müller, L. Gianinazzi, A. Kubicek, H. Niewiadomski, A. O’Mahony, O. Mutlu, and T. Hoefler, “Demystifying Chains, Trees, and Graphs of Thoughts,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 47, no. 12, p. 10 967–10 989, Dec. 2025. [Online]. Available: https://ieeexplore.ieee.org/document/11123142 [46] M. Besta, C. Miglioli, P. S. Labini, J. Tětek, P. Iff, R. Kanakagiri, S. Ashkboos, K. Janda, M. Podstawski, G. Kwaśniewski, N. Gleinig, F. Vella, O. Mutlu, and T. Hoefler, “ProbGraph: High-Performance and High-Accuracy Graph Mining with Probabilistic Set Representations,” in Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, ser. SC ’22. Dallas, TX, USA: IEEE Press, Nov. 2022, p. 43:1–43:17. [Online]. Available: https://ieeexplore.ieee.org/document/10046121 [47] M. Besta, L. Paleari, M. Copik, R. Gerstenberger, A. Kubicek, P. Nyczyk, P. Iff, E. Schreiber, T. Srindran, T. Lehmann, H. Niewiadomski, and T. Hoefler, “CheckEmbed: Effective Verification of LLM Solutions to Open-Ended Tasks,” Jul. 2025, arXiv:2406.02524. [Online]. Available: https://arxiv.org/abs/2406.02524 [48] M. Besta, L. Paleari, J. H. A. Jiang, R. Gerstenberger, Y. Wu, J. G. Hannesson, P. Iff, A. Kubicek, P. Nyczyk, D. Khimey et al., “Affordable AI Assistants with Knowledge Graph of Thoughts,” Oct. 2025, arXiv:2504.02670. [Online]. Available: https://arxiv.org/abs/2504.02670 [49] M. Besta, P. Renc, R. Gerstenberger, P. Sylos Labini, A. Ziogas, T. Chen, L. Gianinazzi, F. Scheidl, K. Szenes, A. Carigiet, P. Iff, G. Kwaśniewski, R. Kanakagiri, C. Ge, S. Jaeger, J. Was, F. Vella, and T. Hoefler, “High-Performance and Programmable Attentional Graph Neural Networks with Global Tensor Formulations,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, ser. SC ’23. Denver, CO, USA: Association for Computing Machinery, Nov. 2023, p. 66:1–66:16. [Online]. Available: https://doi.org/10.1145/3581784.3607067 [50] M. Besta, F. Scheidl, L. Gianinazzi, G. Kwaśniewski, S. Klaiman, J. Müller, and T. Hoefler, “Demystifying Higher-Order Graph Neural Networks,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 48, no. 3, p. 2544–2565, Mar. 2026. [Online]. Available: https://ieeexplore.ieee.org/document/11267259 [51] M. Besta, M. Schneider, M. Konieczny, K. Cynk, E. Henriksson, S. Di Girolamo, A. Singla, and T. Hoefler, “FatPaths: Routing in Supercomputers and Data Centers When Shortest Paths Fall Short,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, ser. SC ’20. Atlanta, GA, USA: IEEE Press, Nov. 2020, p. 27:1–27:18. [Online]. Available: https://ieeexplore.ieee.org/document/9355307 [52] M. Besta, Z. Vonarburg-Shmaria, Y. Schaffner, L. Schwarz, G. Kwaśniewski, L. Gianinazzi, J. Beranek, K. Janda, T. Holenstein, S. Leisinger, P. Tatkowski, E. Ozdemir, A. Balla, M. Copik, P. Lindenberger, M. Konieczny, O. Mutlu, and T. Hoefler, “GraphMineSuite: Enabling High-Performance and Programmable Graph Mining Algorithms with Set Algebra,” Proc. VLDB Endow., vol. 14, no. 11, p. 1922–1935, Jul. 2021. [Online]. Available: https://doi.org/10.14778/3476249.3476252 [53] G. Bilardi and A. Pietracaprina, “Models of Computation, Theoretical,” in Encyclopedia of Parallel Computing, D. Padua, Ed. Boston, MA, USA: Springer, Sep. 2011, p. 1150–1158. [Online]. Available: https://link.springer.com/rwe/10.1007/978-0-387-09766-4_218 [54] S. M. Birjandi and S. H. Khasteh, “A Survey on Data Mining Techniques Used in Medicine,” Journal of Diabetes & Metabolic Disorders, vol. 20, no. 2, p. 2055–2071, Dec. 2021. [Online]. Available: https://link.springer.com/article/10.1007/s40200-021-00884-2 [55] A. Biswal, L. Patel, S. Jha, A. Kamsetty, S. Liu, J. E. Gonzalez, C. Guestrin, and M. Zaharia, “Text2SQL is Not Enough: Unifying AI and Databases with TAG,” in Proceedings of the 15th Annual Conference on Innovative Data Systems Research, ser. CIDR ’25. Amsterdam, Netherlands: VLDB Endowment, Jan. 2025, p. 1–10. [Online]. Available: https://vldb.org/cidrdb/2025/text2sql-is-not-enough-unifying-ai-and-databases-with-tag.html [56] G. E. Blelloch and B. M. Maggs, “Parallel Algorithms,” ACM Comput. Surv., vol. 28, no. 1, p. 51–54, Mar. 1996. [Online]. Available: https://doi.org/10.1145/234313.234339 [57] M. M. Bronstein, J. Bruna, Y. LeCun, A. Szlam, and P. Vandergheynst, “Geometric Deep Learning: Going Beyond Euclidean Data,” IEEE Signal Processing Magazine, vol. 34, no. 4, p. 18–42, Jul. 2017. [Online]. Available: https://ieeexplore.ieee.org/document/7974879 [58] A. Buluç, H. Meyerhenke, I. Safro, P. Sanders, and C. Schulz, “Recent Advances in Graph Partitioning,” in Algorithm Engineering: Selected Results and Surveys, ser. Lecture Notes in Computer Science, L. Kliemann and P. Sanders, Eds. Springer Nature, Nov. 2016, vol. 9220, p. 117–158. [Online]. Available: https://link.springer.com/chapter/10.1007/978-3-319-49487-6_4 [59] Z. Cai, M. Cao, H. Chen, K. Chen, K. Chen, X. Chen, X. Chena, Z. Chen, Z. Chen, P. Chu et al., “InternLM2 Technical Report,” Mar. 2024, arXiv:2403.17297. [Online]. Available: https://arxiv.org/abs/2403.17297 [60] Y. Cang, X. Zhang, E. Zhao, Z. Ji, Y. Liu, Y. He, Z. Ning, Y. Chen, W. Que, and L. Shi, “Graph-GRPO: Stabilizing Multi-Agent Topology Learning via Group Relative Policy Optimization,” Mar. 2026, arXiv:2603.02701. [Online]. Available: https://arxiv.org/abs/2603.02701 [61] Y. Cao, H. Zhao, Y. Cheng, T. Shu, Y. Chen, G. Liu, G. Liang, J. Zhao, J. Yan, and Y. Li, “Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methods,” IEEE Transactions on Neural Networks and Learning Systems, vol. 36, no. 6, p. 9737–9757, Jun. 2025. [Online]. Available: https://ieeexplore.ieee.org/document/10766898 [62] I. Chami, S. Abu-El-Haija, B. Perozzi, C. Ré, and K. Murphy, “Machine Learning on Graphs: A Model and Comprehensive Taxonomy,” Journal of Machine Learning Research, vol. 23, p. 89:1–89:64, May 2022. [Online]. Available: http://jmlr.org/papers/v23/20-852.html [63] E. Chan, M. Heimlich, A. Purkayastha, and R. van de Geijn, “Collective Communication: Theory, Practice, and Experience,” Concurrency and Computation: Practice and Experience, vol. 19, no. 13, p. 1749–1783, Jul. 2007. [Online]. Available: https://onlinelibrary.wiley.com/doi/10.1002/cpe.1206 [64] J. Chang and F. Nargesian, “Approximating Opaque Top-k Queries,” Proc. ACM Manag. Data, vol. 3, no. 3, p. 129:1–129:25, Jun. 2025. [Online]. Available: https://doi.org/10.1145/3725266 [65] J. Chen, S. Li, R. Guo, J. Yuan, and T. Hoefler, “AutoDDL: Automatic Distributed Deep Learning With Near-Optimal Bandwidth Cost,” IEEE Transactions on Parallel and Distributed Systems, vol. 35, no. 8, p. 1331–1344, Aug. 2024. [Online]. Available: https://ieeexplore.ieee.org/document/10521778 [66] J. Chen, Z. Li, X. Hu, and X. Xia, “NLPerturbator: Studying the Robustness of Code LLMs to Natural Language Variations,” ACM Trans. Softw. Eng. Methodol., vol. 35, no. 4, p. 89:1–89:20, Mar. 2026. [Online]. Available: https://doi.org/10.1145/3745764 [67] L. Chen, B. Acun, N. Ardalani, Y. Sun, F. Kang, H. Lyu, Y. Kwon, R. Jia, C.-J. Wu, M. Zaharia, and J. Zou, “Data Acquisition: A New Frontier in Data-Centric AI,” Journal of Data-Centric Machine Learning Research, vol. 2, Jan. 2025. [Online]. Available: https://data.mlr.press/assets/pdf/v02-11.pdf [68] Z. Chen, S. Wang, Z. Tan, X. Fu, Z. Lei, P. Wang, H. Liu, C. Shen, and J. Li, “A Survey of Scaling in Large Language Model Reasoning,” Apr. 2026, arXiv:2504.02181. [Online]. Available: https://arxiv.org/abs/2504.02181 [69] S. Choudhury, “Process Reward Models for LLM Agents: Practical Framework and Directions,” Feb. 2025, arXiv:2502.10325. [Online]. Available: https://arxiv.org/abs/2502.10325 [70] Z. Chu, S. Wang, J. Xie, T. Zhu, Y. Yan, J. Ye, A. Zhong, X. Hu, J. Liang, P. S. Yu, and Q. Wen, “LLM Agents for Education: Advances and Applications,” in Findings of the Association for Computational Linguistics: EMNLP 2025, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng, Eds. Suzhou, China: Association for Computational Linguistics, Nov. 2025, p. 13 782–13 810. [Online]. Available: https://aclanthology.org/2025.findings-emnlp.743/ [71] G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosen et al., “Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities,” Dec. 2025, arXiv:2507.06261. [Online]. Available: https://arxiv.org/abs/2507.06261 [72] M. Copik, G. Kwaśniewski, M. Besta, M. Podstawski, and T. Hoefler, “SeBS: A Serverless Benchmark Suite for Function-as-a-Service Computing,” in Proceedings of the 22nd International Middleware Conference, ser. Middleware ’21. Québec City, Canada: Association for Computing Machinery, Dec. 2021, p. 64–78. [Online]. Available: https://doi.org/10.1145/3464298.3476133 [73] B. Cottier, R. Rahman, L. Fattorini, N. Maslej, T. Besiroglu, and D. Owen, “The Rising Costs of Training Frontier AI Models,” Feb. 2025, arXiv:2404.05952. [Online]. Available: https://arxiv.org/abs/2405.21015 [74] D. Dai, C. Deng, C. Zhao, R. Xu, H. Gao, D. Chen, J. Li, W. Zeng, X. Yu, Y. Wu, Z. Xie, Y. Li, P. Huang, F. Luo, C. Ruan, Z. Sui, and W. Liang, “DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ser. ACL ’24, L.-W. Ku, A. Martins, and V. Srikumar, Eds. Bangkok, Thailand: Association for Computational Linguistics, Aug. 2024, p. 1280–1297. [Online]. Available: https://aclanthology.org/2024.acl-long.70/ [75] J. Dai, X. Pan, R. Sun, J. Ji, X. Xu, M. Liu, Y. Wang, and Y. Yang, “Safe RLHF: Safe Reinforcement Learning from Human Feedback,” in Proceedings of the Twelfth International Conference on Learning Representations, ser. ICLR ’24, B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun, Eds., Vienna, Austria, May 2024, p. 50 750–50 777. [Online]. Available: https://proceedings.iclr.c/paper_files/paper/2024/hash/d1577afd396928ed64216f3f1fd5556-Abstract-Conference.html [76] N. Dai, Z. Wu, R. Zheng, Z. Wei, W. Shi, X. Jin, G. Liu, C. Dun, L. Huang, and L. Yan, “Process Supervision-Guided Policy Optimization for Code Generation,” Feb. 2025, arXiv:2410.17621. [Online]. Available: https://arxiv.org/abs/2410.17621 [77] A. Dandashi and M. Al-Mouhamed, “Graph Coloring for Class Scheduling,” in Proceedings of the ACS/IEEE International Conference on Computer Systems and Applications, ser. AICCSA ’10. Hammamet, Tunisia: IEEE Press, May 2010, p. 1–4. [Online]. Available: https://ieeexplore.ieee.org/document/5586963 [78] A. Davoudian, L. Chen, and M. Liu, “A Survey on NoSQL Stores,” ACM Comput. Surv., vol. 51, no. 2, p. 40:1–40:43, Apr. 2018. [Online]. Available: https://doi.org/10.1145/3158661 [79] B. L. M. de Oliveira, F. V. Frujeri, M. P. C. M. Queiroz, L. G. B. Martins, T. W. de L. Soares, and L. C. Melo, “Learning Without Critics? Revisiting GRPO in Classical Reinforcement Learning Environments,” Nov. 2025, arXiv:2511.03527. [Online]. Available: https://arxiv.org/abs/2511.03527 [80] S. Devunuri and L. Lehe, “TransitGPT: A Generative AI-Based Framework for Interacting with GTFS Data Using Large Language Models,” Public Transport, vol. 17, no. 2, p. 319–345, Jun. 2025. [Online]. Available: https://link.springer.com/article/10.1007/s12469-025-00395-w [81] S. Di Girolamo, D. De Sensi, K. Taranov, M. Malesevic, M. Besta, T. Schneider, S. Kistler, and T. Hoefler, “Building Blocks for Network-Accelerated Distributed File Systems,” in Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, ser. SC ’22. Dallas, TX, USA: IEEE Press, Nov. 2022, p. 10:1–10:14. [Online]. Available: https://doi.org/10.1109/SC41404.2022.00015 [82] S. Di Girolamo, K. Taranov, A. Kurth, M. Schaffner, T. Schneider, J. Beránek, M. Besta, L. Benini, D. Roweth, and T. Hoefler, “Network-Accelerated Non-Contiguous Memory Transfers,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, ser. SC ’19. Denver, CO, USA: Association for Computing Machinery, Nov. 2019, p. 56:1–56:14. [Online]. Available: https://doi.org/10.1145/3295500.3356189 [83] T. Ding, T. Chen, H. Zhu, J. Jiang, Y. Zhong, J. Zhou, G. Wang, Z. Zhu, I. Zharkov, and L. Liang, “The Efficiency Spectrum of Large Language Models: An Algorithmic Survey,” Apr. 2024, arXiv:2312.00678. [Online]. Available: https://arxiv.org/abs/2312.00678 [84] Y. Dong, X. Jiang, J. Qian, T. Wang, K. Zhang, Z. Jin, and G. Li, “A Survey on Code Generation with LLM-Based Agents,” Sep. 2025, arXiv:2508.00083. [Online]. Available: https://arxiv.org/abs/2508.00083 [85] A. Dorbani, S. Yasser, J. Lin, and A. Mhedhbi, “Beyond Quacking: Deep Integration of Language Models and RAG into DuckDB,” Proc. VLDB Endow., vol. 18, no. 12, p. 5415–5418, Sep. 2025. [Online]. Available: https://doi.org/10.14778/3750601.3750685 [86] A. Du, B. Gao, B. Xing, C. Jiang, C. Chen, C. Li, C. Xiao, C. Du, C. Liao, C. Tang et al., “Kimi k1.5: Scaling Reinforcement Learning with LLMs,” Jun. 2025, arXiv:2501.12599. [Online]. Available: https://arxiv.org/abs/2501.12599 [87] K. Ethayarajh, W. Xu, N. Muennighoff, D. Jurafsky, and D. Kiela, “Model Alignment as Prospect Theoretic Optimization,” in Proceedings of the 41st International Conference on Machine Learning (ICML ’24), ser. Proceedings of Machine Learning Research, R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp, Eds., vol. 235. Vienna, Austria: PMLR, Jul. 2024, p. 29 634–29 651. [Online]. Available: https://proceedings.mlr.press/v235/ethayarajh24a.html [88] A. Fawzi, M. Balog, A. Huang, T. Hubert, B. Romera-Paredes, M. Barekatain, A. Novikov, F. J. R. Ruiz, J. Schrittwieser, G. Swirszcz, D. Silver, D. Hassabis, and P. Kohli, “Discovering Faster Matrix Multiplication Algorithms with Reinforcement Learning,” Nature, vol. 610, no. 7930, p. 47–53, Oct. 2022. [Online]. Available: https://w.nature.com/articles/s41586-022-05172-4 [89] L. Feng, C. Pan, X. Guo, F. Mei, B. Ning, J. Zhang, X. Liu, B. Zhou, Z. Shu, C. Liu, G. Yang, Z. Han, J. Wang, and B. Wang, “MindSpeed RL: Distributed Dataflow for Scalable and Efficient RL Training on Ascend NPU Cluster,” Jul. 2025, arXiv:2507.19017. [Online]. Available: https://arxiv.org/abs/2507.19017 [90] Y. Feng and K. Ma, “Chiplet Actuary: A Quantitative Cost Model and Multi-Chiplet Architecture Exploration,” in Proceedings of the 59th ACM/IEEE Design Automation Conference, ser. DAC ’22. San Francisco, CA, USA: Association for Computing Machinery, Jul. 2022, p. 121–126. [Online]. Available: https://doi.org/10.1145/3489517.3530428 [91] E. Frick, P. Jin, T. Li, K. Ganesan, J. Zhang, J. Jiao, and B. Zhu, “Athene-70B,” https://huggingface.co/Nexusflow/Athene-70B, Jul. 2024, [Accessed June 11, 2026]. [92] W. Fu, J. Gao, S. Xu, Z. Mei, C. Zhu, X. Shen, C. He, G. Wei, J. Mei, W. Jiashu, T. Yang, B. Yuan, and Y. Wu, “AREAL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning,” in Proceedings of the 3rd Workshop on Efficient Systems for Foundation Models, ser. ES-FoMo ’25. Vancouver, Canada: OpenReview, Jul. 2025, p. 1–11. [Online]. Available: https://openreview.net/forum?id=qJ0okaW9Z9 [93] Y. Gao, Y. Xiong, X. Gao, K. Jia, J. Pan, Y. Bi, Y. Dai, J. Sun, M. Wang, and H. Wang, “Retrieval-Augmented Generation for Large Language Models: A Survey,” Mar. 2024, arXiv:2312.10997. [Online]. Available: https://arxiv.org/abs/2312.10997 [94] R. Gerstenberger, M. Besta, and T. Hoefler, “Enabling Highly-Scalable Remote Memory Access Programming with MPI-3 One Sided,” in Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, ser. SC ’13. Denver, CO, USA: Association for Computing Machinery, Nov. 2013, p. 53:1–53:12. [Online]. Available: https://doi.org/10.1145/2503210.2503286 [95] M. Gheisari, H. Hamidpour, Y. Liu, P. Saedi, A. Raza, A. Jalili, H. Rokhsati, and R. Amin, “Data Mining Techniques for Web Mining: A Survey,” Artificial Intelligence and Applications, vol. 1, no. 1, p. 3–10, Jan. 2023. [Online]. Available: https://ojs.bonviewpress.com/index.php/AIA/article/view/290 [96] M. Gheshlaghi Azar, Z. Daniel Guo, B. Piot, R. Munos, M. Rowland, M. Valko, and D. Calandriello, “A General Theoretical Paradigm to Understand Learning from Human Preferences,” in Proceedings of the 27th International Conference on Artificial Intelligence and Statistics (AISTATS ’24), ser. Proceedings of Machine Learning Research, S. Dasgupta, S. Mandt, and Y. Li, Eds., vol. 238. Valencia, Spain: PMLR, May 2024, p. 4447–4455. [Online]. Available: https://proceedings.mlr.press/v238/gheshlaghi-azar24a.html [97] S. Ghose, A. Boroumand, J. S. Kim, J. Gómez-Luna, and O. Mutlu, “Processing-in-Memory: A Workload-Driven Perspective,” IBM Journal of Research and Development, vol. 63, no. 6, p. 3:1–3:19, Nov. 2019. [Online]. Available: https://ieeexplore.ieee.org/document/8792187 [98] L. Gianinazzi, T. Ben-Nun, M. Besta, S. Ashkboos, Y. Baumann, P. Luczynski, and T. Hoefler, “The Spatial Computer: A Model for Energy-Efficient Parallel Computation,” Jan. 2023, arXiv:2205.04934. [Online]. Available: https://arxiv.org/abs/2205.04934 [99] L. Gianinazzi, M. Besta, Y. Schaffner, and T. Hoefler, “Parallel Algorithms for Finding Large Cliques in Sparse Graphs,” in Proceedings of the 33rd Symposium on Parallelism in Algorithms and Architectures, ser. SPAA ’21. Virtual Event: Association for Computing Machinery, Jul. 2021, p. 243–253. [Online]. Available: https://doi.org/10.1145/3409964.3461800 [100] L. Gianinazzi, M. Fries, N. Dryden, T. Ben-Nun, M. Besta, and T. Hoefler, “Learning Combinatorial Node Labeling Algorithms,” May 2022, arXiv:2106.03594. [Online]. Available: https://arxiv.org/abs/2106.03594 [101] L. Gianinazzi, P. Kalvoda, A. De Palma, M. Besta, and T. Hoefler, “Communication-Avoiding Parallel Minimum Cuts and Connected Components,” SIGPLAN Not., vol. 53, no. 1, p. 219–232, Feb. 2018. [Online]. Available: https://doi.org/10.1145/3200691.3178504 [102] Google DeepMind, “Gemini 2.5 Deep Think,” https://blog.google/products-and-platforms/products/gemini/gemini-2-5-deep-think/, Aug. 2025, [Accessed June 15, 2026]. [103] P. Gorinski, M. Zimmer, G. Lampouras, D. G. X. Deik, and I. Iacobacci, “Automatic Unit Test Data Generation and Actor-Critic Reinforcement Learning for Code Synthesis,” in Findings of the Association for Computational Linguistics: EMNLP 2023, H. Bouamor, J. Pino, and K. Bali, Eds. Singapore: Association for Computational Linguistics, Dec. 2023, p. 370–384. [Online]. Available: https://aclanthology.org/2023.findings-emnlp.28/ [104] A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan et al., “The Llama 3 Herd of Models,” Nov. 2024, arXiv:2407.21783. [Online]. Available: https://arxiv.org/abs/2407.21783 [105] J. Gu, X. Jiang, Z. Shi, H. Tan, X. Zhai, C. Xu, W. Li, Y. Shen, S. Ma, H. Liu, S. Wang, K. Zhang, Z. Lin, B. Zhang, L. Ni, W. Gao, Y. Wang, and J. Guo, “A Survey on LLM-as-a-Judge,” The Innovation, vol. 7, no. 6, p. 101 253:1–101 253:30, Jun. 2026. [Online]. Available: https://w.sciencedirect.com/science/article/pii/S2666675825004564 [106] X. Guan, L. L. Zhang, Y. Liu, N. Shang, Y. Sun, Y. Zhu, F. Yang, and M. Yang, “rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking,” in Proceedings of the 42nd International Conference on Machine Learning (ICML ’25), ser. Proceedings of Machine Learning Research, A. Singh, M. Fazel, D. Hsu, S. Lacoste-Julien, F. Berkenkamp, T. Maharaj, K. Wagstaff, and J. Zhu, Eds., vol. 267. Vancouver, Canada: PMLR, Jul. 2025, p. 20 640–20 661. [Online]. Available: https://proceedings.mlr.press/v267/guan25f.html [107] D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Bi et al., “DeepSeek-R1 Incentivizes Reasoning in LLMs Through Reinforcement Learning,” Nature, vol. 645, no. 8081, p. 633–638, Sep. 2025. [Online]. Available: https://w.nature.com/articles/s41586-025-09422-z [108] H. Guo, H. Lu, G. Nan, B. Chu, J. Zhuang, Y. Yang, W. Che, X. Cao, S. Leng, Q. Cui, and X. Jiang, “Advancing Expert Specialization for Better MoE,” in Proceedings of the Thirty-Ninth Annual Conference on Neural Information Processing Systems (NeurIPS ’25), ser. Advances in Neural Information Processing Systems, D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen, Eds., vol. 38. San Diego, CA, USA: Curran Associates, Dec. 2025, p. 48 767–48 809. [Online]. Available: https://proceedings.neurips.c/paper_files/paper/2025/hash/4598de7d243d528e38eb0c5d8155fb52-Abstract-Conference.html [109] M. K. Gupta and P. Chandra, “A Comprehensive Survey of Data Mining,” International Journal of Information Technology, vol. 12, no. 4, p. 1243–1257, Dec. 2020. [Online]. Available: https://link.springer.com/article/10.1007/s41870-020-00427-7 [110] W. L. Hamilton, R. Ying, and J. Leskovec, “Representation Learning on Graphs: Methods and Applications,” Bulletin of the Technical Committee on Data Engineering, vol. 40, no. 3, p. 52–74, Sep. 2017. [Online]. Available: http://sites.computer.org/debull/A17sept/p52.pdf [111] Z. Han, A. You, H. Wang, K. Luo, G. Yang, W. Shi, M. Chen, S. Zhang, Z. Lan, C. Deng et al., “AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Training,” Jul. 2025, arXiv:2507.01663. [Online]. Available: https://arxiv.org/abs/2507.01663 [112] H. B. Hassan, S. A. Barakat, and Q. I. Sarhan, “Survey on Serverless Computing,” Journal of Cloud Computing, vol. 10, no. 1, p. 39:1–39:29, Jul. 2021. [Online]. Available: https://link.springer.com/article/10.1186/s13677-021-00253-7 [113] T. Hoefler and R. Belli, “Scientific Benchmarking of Parallel Computing Systems: Twelve Ways to Tell the Masses When Reporting Performance Results,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, ser. SC ’15. Austin, TX, USA: Association for Computing Machinery, Nov. 2015, p. 73:1–73:12. [Online]. Available: https://doi.org/10.1145/2807591.2807644 [114] J. Hong, N. Lee, and J. Thorne, “ORPO: Monolithic Preference Optimization Without Reference Model,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, ser. EMNLP ’24, Y. Al-Onaizan, M. Bansal, and Y.-N. Chen, Eds. Miami, FL, USA: Association for Computational Linguistics, Nov. 2024, p. 11 170–11 189. [Online]. Available: https://aclanthology.org/2024.emnlp-main.626/ [115] Z. Hou, P. Du, Y. Niu, Z. Du, A. Zeng, X. Liu, M. Huang, H. Wang, J. Tang, and Y. Dong, “Does RLHF Scale? Exploring the Impacts From Data, Model, and Method,” Dec. 2024, arXiv:2412.06000. [Online]. Available: https://arxiv.org/abs/2412.06000 [116] C.-H. Hsu, N. Imam, A. Langer, S. Potluri, and C. J. Newburn, “An Initial Assessment of NVSHMEM for High Performance Computing,” in Proceedings of the IEEE International Parallel and Distributed Processing Symposium Workshops, ser. IPDPSW ’20. New Orleans, LA, USA: IEEE Press, May 2020, p. 1–10. [Online]. Available: https://ieeexplore.ieee.org/document/9150438 [117] J. Hu, X. Wu, W. Shen, J. K. Liu, W. Wang, S. Jiang, H. Wang, H. Chen, B. Chen, W. Fang, Xianyu, Y. Cao, H. Xu, and Y. Liu, “OpenRLHF: A Ray-Based Easy-to-Use, Scalable and High-Performance RLHF Framework,” in Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, ser. EMNLP ’25, I. Habernal, P. Schulam, and J. Tiedemann, Eds. Suzhou, China: Association for Computational Linguistics, Nov. 2025, p. 656–666. [Online]. Available: https://aclanthology.org/2025.emnlp-demos.48/ [118] J. Hu, Y. Zhang, Q. Han, D. Jiang, X. Zhang, and H.-Y. Shum, “Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model,” in Proceedings of the Thirty-Ninth Annual Conference on Neural Information Processing Systems (NeurIPS ’25), ser. Advances in Neural Information Processing Systems, D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen, Eds., vol. 38. San Diego, CA, USA: Curran Associates, Dec. 2025, p. 162 239–162 262. [Online]. Available: https://proceedings.neurips.c/paper_files/paper/2025/hash/ed873d79e7c268c020c4b4db13a2812a-Abstract-Conference.html [119] X. Huang, W. Liu, X. Chen, X. Wang, H. Wang, D. Lian, Y. Wang, R. Tang, and E. Chen, “Understanding the Planning of LLM Agents: A Survey,” Feb. 2024, arXiv:2402.02716. [Online]. Available: https://arxiv.org/abs/2402.02716 [120] Y. Huang, Y. Cheng, A. Bapna, O. Firat, D. Chen, M. Chen, H. Lee, J. Ngiam, Q. V. Le, Y. Wu, and Z. Chen, “GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism,” in Proceedings of the Thirty-Third Annual Conference on Neural Information Processing Systems (NeurIPS ’19), ser. Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, Eds., vol. 32. Vancouver, Canada: Curran Associates, Dec. 2019, p. 103–112. [Online]. Available: https://proceedings.neurips.c/paper_files/paper/2019/hash/093f65e080a295f8076b1c5722a46a2-Abstract.html [121] P. Iff, M. Besta, M. Cavalcante, T. Fischer, L. Benini, and T. Hoefler, “HexaMesh: Scaling to Hundreds of Chiplets with an Optimized Chiplet Arrangement,” in Proceedings of the 60th ACM/IEEE Design Automation Conference, ser. DAC ’23, San Francisco, CA, USA, Jul. 2023, p. 1–6. [Online]. Available: https://ieeexplore.ieee.org/document/10248006 [122] —, “Sparse Hamming Graph: A Customizable Network-on-Chip Topology,” in Proceedings of the 2023 60th ACM/IEEE Design Automation Conference, ser. DAC ’23. San Francisco, CA, USA: IEEE Press, Jul. 2023, p. 1–6. [Online]. Available: https://doi.org/10.1109/DAC56929.2023.10247754 [123] P. Iff, T. Bonato, M. Besta, L. Benini, and T. Hoefler, “Network Design for Wafer-Scale Systems with Wafer-on-Wafer Hybrid Bonding,” Mar. 2026, arXiv:2603.05266. [Online]. Available: https://arxiv.org/abs/2603.05266 [124] P. Iff, B. Bruggmann, M. Besta, L. Benini, and T. Hoefler, “PlaceIT: Placement-Based Inter-Chiplet Interconnect Topologies,” Feb. 2025. [Online]. Available: https://arxiv.org/abs/2502.01449 [125] S. A. Jacobs, M. Tanaka, C. Zhang, M. Zhang, R. Y. Aminadabi, S. L. Song, S. Rajbhandari, and Y. He, “System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models,” in Proceedings of the 43rd ACM Symposium on Principles of Distributed Computing, ser. PODC ’24. Nantes, France: Association for Computing Machinery, Jun. 2024, p. 121–130. [Online]. Available: https://doi.org/10.1145/3662158.3662806 [126] Z. Jiang, F. Xu, L. Gao, Z. Sun, Q. Liu, J. Dwivedi-Yu, Y. Yang, J. Callan, and G. Neubig, “Active Retrieval Augmented Generation,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, ser. EMNLP ’23, H. Bouamor, J. Pino, and K. Bali, Eds. Singapore: Association for Computational Linguistics, Dec. 2023, p. 7969–7992. [Online]. Available: https://aclanthology.org/2023.emnlp-main.495/ [127] N. M. Johnson and J. Cao, “Announcing PyTorch/XLA 2.3: Distributed Training, Dev Improvements, and GPUs,” https://cloud.google.com/blog/products/ai-machine-learning/introducing-pytorch-xla-2-3, Apr. 2024, [Accessed June 16, 2026]. [128] E. Jonas, J. Schleier-Smith, V. Sreekanti, C.-C. Tsai, A. Khandelwal, Q. Pu, V. Shankar, J. Carreira, K. Krauth, N. Yadwadkar, J. E. Gonzalez, R. A. Popa, I. Stoica, and D. A. Patterson, “Cloud Programming Simplified: A Berkeley View on Serverless Computing,” Feb. 2019, arXiv:1902.03383. [Online]. Available: https://arxiv.org/abs/1902.03383 [129] M. Khalifa, R. Agarwal, L. Logeswaran, J. Kim, H. Peng, M. Lee, H. Lee, and L. Wang, “Process Reward Models That Think,” Transactions on Machine Learning Research, p. 1–37, Mar. 2026, j2C Certification. [Online]. Available: https://openreview.net/forum?id=FPVCb0WMuN [130] J. Kim, E. Ewer, T. Moon, J. Park, and D. Papailiopoulos, “Not All Bits Are Equal: Scale-Dependent Memory Optimization Strategies for Reasoning Models,” in Proceedings of the Fourteenth International Conference on Learning Representations, ser. ICLR ’26. Rio de Janeiro, Brazil: OpenReview, Apr. 2026, p. 1–24. [Online]. Available: https://openreview.net/forum?id=b6qQmQ2F13 [131] T. N. Kipf and M. Welling, “Semi-Supervised Classification with Graph Convolutional Networks,” in Proceedings of the International Conference on Learning Representations, ser. ICLR ’17. Toulon, France: OpenReview, Apr. 2017, p. 1–14. [Online]. Available: https://openreview.net/forum?id=SJU4ayYgl [132] V. A. Korthikanti, J. Casper, S. Lym, L. McAfee, M. Andersch, M. Shoeybi, and B. Catanzaro, “Reducing Activation Recomputation in Large Transformer Models,” in Proceedings of the Sixth Conference on Machine Learning and Systems (MLSys ’23), ser. Proceedings of Machine Learning and Systems, D. Song, M. Carbin, and T. Chen, Eds., vol. 5. Miami Beach, FL, USA: Curran Associates, Jun. 2023, p. 341–353. [Online]. Available: https://proceedings.mlsys.org/paper_files/paper/2023/hash/80083951326cf5b35e5100260d64ed81-Abstract-mlsys2023.html [133] W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, “Efficient Memory Management for Large Language Model Serving with PagedAttention,” in Proceedings of the 29th Symposium on Operating Systems Principles, ser. SOSP ’23. Koblenz, Germany: Association for Computing Machinery, Oct. 2023, p. 611–626. [Online]. Available: https://doi.org/10.1145/3600006.3613165 [134] X. Lai, Z. Tian, Y. Chen, S. Yang, X. Peng, and J. Jia, “Step-DPO: Step-Wise Preference Optimization for Long-Chain Reasoning of LLMs,” Jun. 2024, arXiv:2406.18629. [Online]. Available: https://arxiv.org/abs/2406.18629 [135] K. Lakhotia, M. Besta, L. Monroe, K. Isham, P. Iff, T. Hoefler, and F. Petrini, “PolarFly: A Cost-Effective and Flexible Low-Diameter Topology,” in Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, ser. SC ’22. Dallas, TX, USA: IEEE Press, Nov. 2022, p. 12:1–12:15. [Online]. Available: https://ieeexplore.ieee.org/document/10046084 [136] K. Lakhotia, L. Monroe, K. Isham, M. Besta, N. Blach, T. Hoefler, and F. Petrini, “PolarStar: Expanding the Horizon of Diameter-3 Networks,” in Proceedings of the 36th ACM Symposium on Parallelism in Algorithms and Architectures, ser. SPAA ’24. Nantes, France: Association for Computing Machinery, Jun. 2024, p. 345–357. [Online]. Available: https://doi.org/10.1145/3626183.3659975 [137] K. Lan, D.-t. Wang, S. Fong, L.-s. Liu, K. K. Wong, and N. Dey, “A Survey of Data Mining and Deep Learning in Bioinformatics,” Journal of Medical Systems, vol. 42, no. 8, p. 139, Jun. 2018. [Online]. Available: https://link.springer.com/article/10.1007/s10916-018-1003-9 [138] H. Le, Y. Wang, A. D. Gotmare, S. Savarese, and S. C. H. Hoi, “CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning,” in Proceedings of the Thirty-Sixth Annual Conference on Neural Information Processing Systems (NeurIPS ’22), ser. Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., vol. 35. New Orleans, LA, USA: Curran Associates, Dec. 2022, p. 21 314–21 328. [Online]. Available: https://proceedings.neurips.c/paper_files/paper/2022/hash/8636419dea1a9fbd25fc4248e702da4-Abstract-Conference.html [139] A. W. Lee, J. Chan, M. Fu, N. Kim, A. Mehta, D. Raghavan, and U. Çetintemel, “Semantic Integrity Constraints: Declarative Guardrails for AI-Augmented Data Processing Systems,” Proc. VLDB Endow., vol. 18, no. 11, p. 4073–4080, Sep. 2025. [Online]. Available: https://doi.org/10.14778/3749646.3749677 [140] H. Lee, S. Phatale, H. Mansoor, T. Mesnard, J. Ferret, K. R. Lu, C. Bishop, E. Hall, V. Carbune, A. Rastogi, and S. Prakash, “RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback,” in Proceedings of the 41st International Conference on Machine Learning (ICML ’24), ser. Proceedings of Machine Learning Research, R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp, Eds., vol. 235. Vienna, Austria: PMLR, Jul. 2024, p. 26 874–26 901. [Online]. Available: https://proceedings.mlr.press/v235/lee24t.html [141] D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen, “GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding ,” in Proceedings of the Ninth International Conference on Learning Representations, ser. ICLR ’21. Virtual Event: OpenReview, May 2021, p. 1–23. [Online]. Available: https://openreview.net/forum?id=qrwe7XHTmYb [142] P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel, S. Riedel, and D. Kiela, “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,” in Proceedings of the Thirty-Fourth Annual Conference on Neural Information Processing Systems (NeurIPS ’20), ser. Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., vol. 33. Virtual Event: Curran Associates, Dec. 2020, p. 9459–9474. [Online]. Available: https://proceedings.neurips.c/paper_files/paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html [143] A. Li, S. L. Song, J. Chen, J. Li, X. Liu, N. R. Tallent, and K. J. Barker, “Evaluating Modern GPU Interconnect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirect,” IEEE Transactions on Parallel and Distributed Systems, vol. 31, no. 1, p. 94–110, Jan. 2020. [Online]. Available: https://ieeexplore.ieee.org/document/8763922 [144] H. Li, Y. Li, A. Tian, T. Tang, Z. Xu, X. Chen, N. HU, W. Dong, L. Qing, and L. Chen, “A Survey on Large Language Model Acceleration Based on KV Cache Management,” Transactions on Machine Learning Research, p. 1–61, May 2025. [Online]. Available: https://openreview.net/forum?id=z3JZzu9EA3 [145] H. Li, S. Lin, F. Fu, Y. Zhou, X. Ji, Y. Zhao, L. Wang, J. Jiang, and B. Cui, “Unleashing Efficient Asynchronous RL Post-Training via Staleness-Constrained Rollout Coordination,” Jan. 2026, arXiv:2601.12784. [Online]. Available: https://arxiv.org/abs/2601.12784 [146] S. Li, Y. Zhao, R. Varma, O. Salpekar, P. Noordhuis, T. Li, A. Paszke, J. Smith, B. Vaughan, P. Damania, and S. Chintala, “PyTorch Distributed: Experiences on Accelerating Data Parallel Training,” Proc. VLDB Endow., vol. 13, no. 12, p. 3005–3018, Aug. 2020. [Online]. Available: https://doi.org/10.14778/3415478.3415530 [147] S. Li, H. Liu, Z. Bian, J. Fang, H. Huang, Y. Liu, B. Wang, and Y. You, “Colossal-AI: A Unified Deep Learning System for Large-Scale Parallel Training,” in Proceedings of the 52nd International Conference on Parallel Processing, ser. ICPP ’23. Salt Lake City, UT, USA: Association for Computing Machinery, Aug. 2023, p. 766–775. [Online]. Available: https://doi.org/10.1145/3605573.3605613 [148] T. Li, J. Hou, J. Yan, R. Liu, H. Yang, and Z. Sun, “Chiplet Heterogeneous Integration Technology–Status and Challenges,” Electronics, vol. 9, no. 4, p. 670:1–670:12, Apr. 2020. [Online]. Available: https://w.mdpi.com/2079-9292/9/4/670 [149] W. Li and Y. Li, “Process Reward Model with Q-Value Rankings,” in Proceedings of the Thirteenth International Conference on Learning Representations, ser. ICLR ’25, Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu, Eds., Singapore, Apr. 2025, p. 14 708–14 726. [Online]. Available: https://proceedings.iclr.c/paper_files/paper/2025/hash/26494b66ae1114d314673e25b4967288-Abstract-Conference.html [150] Y. Li, Z. Wang, T. Fu, G. Cui, S. Yang, and Y. Cheng, “From Drafts to Answers: Unlocking LLM Potential via Aggregation Fine-Tuning,” Jan. 2025, arXiv:2501.11877. [Online]. Available: https://arxiv.org/abs/2501.11877 [151] Z. Li, X. Feng, Y. Cai, Z. Zhang, T. Liu, C. Liang, W. Chen, H. Wang, and T. Zhao, “LLMs Can Generate a Better Answer by Aggregating Their Own Responses,” Apr. 2025, arXiv:2503.04104. [Online]. Available: https://arxiv.org/abs/2503.04104 [152] Z. Li, L. Guo, J. Cheng, Q. Chen, B. He, and M. Guo, “The Serverless Computing Survey: A Technical Primer for Design Architecture,” ACM Comput. Surv., vol. 54, no. 10s, p. 220:1–220:34, Sep. 2022. [Online]. Available: https://doi.org/10.1145/3508360 [153] Z. Li, T. Xu, Y. Zhang, Z. Lin, Y. Yu, R. Sun, and Z.-Q. Luo, “ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models,” in Proceedings of the 41st International Conference on Machine Learning (ICML ’24), ser. Proceedings of Machine Learning Research, R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp, Eds., vol. 235. Vienna, Austria: PMLR, Jul. 2024, p. 29 128–29 163. [Online]. Available: https://proceedings.mlr.press/v235/li24cd.html [154] Z. Li, S. Huang, Z. Chi, Y. Su, L. Zhou, L. Dong, N. Collier, and F. Wei, “Breaking Training Bottlenecks: Effective and Stable Reinforcement Learning for Coding Models,” Mar. 2026, arXiv:2603.07777. [Online]. Available: https://arxiv.org/abs/2603.07777 [155] H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe, “Let’s Verify Step by Step,” in Proceedings of the Twelfth International Conference on Learning Representations, ser. ICLR ’24, B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun, Eds., Vienna, Austria, May 2024, p. 39 578–39 601. [Online]. Available: https://proceedings.iclr.c/paper_files/paper/2024/hash/aca97732e30bcf1303bc22ac3924fd16-Abstract-Conference.html [156] A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan et al., “DeepSeek-V3 Technical Report,” Feb. 2025, arXiv:2412.19437. [Online]. Available: https://arxiv.org/abs/2412.19437 [157] A. Liu, J. Liu, Z. Pan, Y. He, G. Haffari, and B. Zhuang, “MiniCache: KV Cache Compression in Depth Dimension for Large Language Models,” in Proceedings of the Thirty-Eighth Annual Conference on Neural Information Processing Systems (NeurIPS ’24), ser. Advances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, Eds., vol. 37. Vancouver, Canada: Curran Associates, Dec. 2024, p. 139 997–140 031. [Online]. Available: https://proceedings.neurips.c/paper_files/paper/2024/hash/fd0705710bf01b88a60a3d479ea341d9-Abstract-Conference.html [158] C. Liu, M. Russo, M. Cafarella, L. Cao, P. B. Chen, Z. Chen, M. Franklin, T. Kraska, S. Madden, R. Shahout, and G. Vitagliano, “Palimpzest: Optimizing AI-Powered Analytics with Declarative Query Processing,” in Proceedings of the 15th Annual Conference on Innovative Data Systems Research, ser. CIDR ’25. Amsterdam, Netherlands: VLDB Endowment, Jan. 2025, p. 1–7. [Online]. Available: https://w.vldb.org/cidrdb/2025/palimpzest-optimizing-ai-powered-analytics-with-declarative-query-processing.html [159] C. Liu, G. Vitagliano, B. Rose, M. Printz, D. A. Samson, and M. Cafarella, “PalimpChat: Declarative and Interactive AI Analytics,” in Companion of the 2025 International Conference on Management of Data, ser. SIGMOD/PODS ’25. Berlin, Germany: Association for Computing Machinery, Jun. 2025, p. 183–186. [Online]. Available: https://doi.org/10.1145/3722212.3725122 [160] H. Liu, M. Zaharia, and P. Abbeel, “RingAttention with Blockwise Transformers for Near-Infinite Context,” in Proceedings of the Twelfth International Conference on Learning Representations, ser. ICLR ’24, B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun, Eds., Vienna, Austria, May 2024, p. 3992–4008. [Online]. Available: https://proceedings.iclr.c/paper_files/paper/2024/hash/1119587863e78451f080da2a768c4935-Abstract-Conference.html [161] W. Liu, X. Huang, X. Zeng, X. Hao, S. Yu, D. Li, S. Wang, W. Gan, Z. Liu, Y. Yu, Z. Wang, Y. Wang, W. Ning, Y. Hou, B. Wang, C. Wu, W. Xinzhi, Y. Liu, Y. Wang, D. Tang, D. Tu, L. Shang, X. Jiang, R. Tang, D. Lian, Q. Liu, and E. Chen, “ToolACE: Winning the Points of LLM Function Calling,” in Proceedings of the Thirteenth International Conference on Learning Representations, ser. ICLR ’25, Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu, Eds., Singapore, Apr. 2025, p. 41 359–41 381. [Online]. Available: https://proceedings.iclr.c/paper_files/paper/2025/hash/663865ea167425c6c562cb0b6bcf76c7-Abstract-Conference.html [162] Z. Liu, X. Xu, P. Qiao, and D. Li, “Acceleration for Deep Reinforcement Learning using Parallel and Distributed Computing: A Survey,” ACM Comput. Surv., vol. 57, no. 4, p. 91:1–91:35, Dec. 2024. [Online]. Available: https://doi.org/10.1145/3703453 [163] G. H. Loh, S. Naffziger, and K. Lepak, “Understanding Chiplets Today to Anticipate Future Integration Opportunities and Limits,” in Proceedings of the Design, Automation & Test in Europe Conference & Exhibition, ser. DATE ’21. Grenoble, France: IEEE Press, Feb. 2021, p. 142–145. [Online]. Available: https://ieeexplore.ieee.org/document/9474021 [164] P. Ma, R. Ding, S. Wang, S. Han, and D. Zhang, “InsightPilot: An LLM-Empowered Automated Data Exploration System,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, ser. EMNLP ’23, Y. Feng and E. Lefever, Eds. Singapore: Association for Computational Linguistics, Dec. 2023, p. 346–352. [Online]. Available: https://aclanthology.org/2023.emnlp-demo.31/ [165] Q. Ma, H. Zhou, T. Liu, J. Yuan, P. Liu, Y. You, and H. Yang, “Let’s Reward Step by Step: Step-Level Reward Model as the Navigators for Reasoning,” Oct. 2023, arXiv:2310.10080. [Online]. Available: https://arxiv.org/abs/2310.10080 [166] X. Ma, Y. Wang, Y. Wang, X. Cai, and Y. Han, “Survey on Chiplets: Interface, Interconnect and Integration Methodology,” CCF Transactions on High Performance Computing, vol. 4, no. 1, p. 43–52, Mar. 2022. [Online]. Available: https://link.springer.com/article/10.1007/s42514-022-00093-0 [167] S. Madden, M. Cafarella, M. Franklin, and T. Kraska, “Databases Unbound: Querying All of the World’s Bytes with AI,” Proc. VLDB Endow., vol. 17, no. 12, p. 4546–4554, Aug. 2024. [Online]. Available: https://doi.org/10.14778/3685800.3685916 [168] G. McGrath and P. R. Brenner, “Serverless Computing: Design, Implementation, and Performance,” in Proceedings of the IEEE 37th International Conference on Distributed Computing Systems Workshops, ser. ICDCSW ’17. Atlanta, GA, USA: IEEE Press, Jun. 2017, p. 405–410. [Online]. Available: https://ieeexplore.ieee.org/document/7979855 [169] Z. Mei, W. Fu, K. Li, G. Wang, H. Zhang, and Y. Wu, “ReaL: Efficient RLHF Training for Large Language Models through Parameter Reallocation,” in Proceedings of the Eighth Annual Conference on Machine Learning and Systems (MLSys ’25), ser. Proceedings of Machine Learning and Systems, M. Zaharia, G. Joshi, and Y. Lin, Eds., vol. 7, Santa Clara, CA, USA, May 2025, p. 1–20. [Online]. Available: https://proceedings.mlsys.org/paper_files/paper/2025/hash/3b3889d313ba9476c12c2d77ea66b24f-Abstract-Conference.html [170] Y. Meng, M. Xia, and D. Chen, “SimPO: Simple Preference Optimization with a Reference-Free Reward,” in Proceedings of the Thirty-Eighth Annual Conference on Neural Information Processing Systems (NeurIPS ’24), ser. Advances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, Eds., vol. 37. Vancouver, Canada: Curran Associates, Dec. 2024, p. 124 198–124 235. [Online]. Available: https://proceedings.neurips.c/paper_files/paper/2024/hash/e099c1c9699814af0be873a175361713-Abstract-Conference.html [171] Meta AI, “The Llama 4 Herd: The Beginning of a New Era of Natively Multimodal AI Innovation,” https://ai.meta.com/blog/llama-4-multimodal-intelligence/, Apr. 2025, [Accessed June 15, 2026]. [172] G. Mounce, J. Lyke, S. Horan, W. Powell, R. Doyle, and R. Some, “Chiplet Based Approach for Heterogeneous Processing and Packaging Architectures,” in Proceedings of the IEEE Aerospace Conference, ser. AERO ’16. Big Sky, MT, USA: IEEE Press, Mar. 2016, p. 1–12. [Online]. Available: https://ieeexplore.ieee.org/document/7500830 [173] Y. Mroueh, “Reinforcement Learning with Verifiable Rewards: GRPO’s Effective Loss, Dynamics, and Success Amplification,” Oct. 2025, arXiv:2503.06639. [Online]. Available: https://arxiv.org/abs/2503.06639 [174] O. Mutlu, S. Ghose, J. Gómez-Luna, and R. Ausavarungnirun, “A Modern Primer on Processing in Memory,” in Emerging Computing: From Devices to Systems - Looking Beyond Moore and Von Neumann, ser. Computer Architecture and Design Methodologies (CADM), M. M. S. Aly and A. Chattopadhyay, Eds. Springer Nature, Jul. 2022, p. 171–243. [Online]. Available: https://link.springer.com/chapter/10.1007/978-981-16-7487-7_7 [175] D. Narayanan, A. Phanishayee, K. Shi, X. Chen, and M. Zaharia, “Memory-Efficient Pipeline-Parallel DNN Training,” in Proceedings of the 38th International Conference on Machine Learning (ICML ’21), ser. Proceedings of Machine Learning Research, M. Meila and T. Zhang, Eds., vol. 139. Virtual Event: PMLR, Jul. 2021, p. 7937–7947. [Online]. Available: https://proceedings.mlr.press/v139/narayanan21a.html [176] D. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, A. Phanishayee, and M. Zaharia, “Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, ser. SC ’21. St. Louis, MO, USA: Association for Computing Machinery, Nov. 2021, p. 58:1–58:15. [Online]. Available: https://doi.org/10.1145/3458817.3476209 [177] S. Nayab, G. Rossolini, M. Simoni, A. Saracino, G. Buttazzo, N. Manes, and F. Giacomelli, “Concise Thoughts: Impact of Output Length on LLM Reasoning and Cost,” Jan. 2025, arXiv:2407.19825. [Online]. Available: https://arxiv.org/abs/2407.19825 [178] X.-P. Nguyen, S. Pandit, A. Xu, C. Xiong, and S. Joty, “Least-Loaded Expert Parallelism: Load Balancing an Imbalanced Mixture-of-Experts,” Jan. 2026, arXiv:2601.17111. [Online]. Available: https://arxiv.org/abs/2601.17111 [179] X. Ning, Z. Lin, Z. Zhou, Z. Wang, H. Yang, and Y. Wang, “Skeleton-of-Thought: Prompting LLMs for Efficient Parallel Generation,” in Proceedings of the Twelfth International Conference on Learning Representations, ser. ICLR ’24, B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun, Eds., Vienna, Austria, May 2024, p. 917–967. [Online]. Available: https://proceedings.iclr.c/paper_files/paper/2024/hash/03d7e13f0092405804f3a381ade8f3f0-Abstract-Conference.html [180] M. Noukhovitch, S. Huang, S. Xhonneux, A. Hosseini, R. Agarwal, and A. Courville, “Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models,” Singapore, p. 4003–4029, Apr. 2025. [Online]. Available: https://proceedings.iclr.c/paper_files/paper/2025/hash/0b99315234c95e6ef281f9155b68832-Abstract-Conference.html [181] NovaSky-AI, “SkyRL: A Modular Full-Stack RL Library for LLMs,” https://github.com/novasky-ai/skyrl, Apr. 2026, [Accessed June 12, 2026]. [182] A. Novikov, N. Vũ, M. Eisenberger, E. Dupont, P.-S. Huang, A. Z. Wagner, S. Shirobokov, B. Kozlovskii, F. J. Ruiz, A. Mehrabian, M. P. Kumar, A. See, S. Chaudhuri, G. Holland, A. Davies, S. Nowozin, P. Kohli, and M. Balog, “AlphaEvolve: A Coding Agent for Scientific and Algorithmic Discovery,” Jun. 2025, arXiv:2506.13131. [Online]. Available: https://arxiv.org/abs/2506.13131 [183] NVIDIA, “Context Parallelism Overview,” https://docs.nvidia.com/megatron-core/developer-guide/latest/user-guide/features/context_parallel.html, 2026, [Accessed June 17, 2026]. [184] —, “NeMo RL: Scalable Toolkit for Efficient Model Reinforcement,” https://github.com/nvidia-nemo/rl, Apr. 2026, [Accessed June 12, 2026]. [185] —, “TensorRT-LLM,” https://github.com/NVIDIA/TensorRT-LLM, Apr. 2026, [Accessed June 12, 2026]. [186] OpenAI, “OpenAI o1,” https://openai.com/o1/, Dec. 2024, [Accessed June 11, 2026]. [187] —, “OpenAI o3,” https://openai.com/index/introducing-o3-and-o4-mini/, Apr. 2025, [Accessed June 11, 2026]. [188] —, “OpenAI o3 and o4-mini System Card,” https://openai.com/index/o3-o4-mini-system-card/, Apr. 2025, [Accessed June 15, 2026]. [189] —, “GPT-5.5 in ChatGPT,” https://help.openai.com/en/articles/11909943-gpt-55-in-chatgpt, Jun. 2026, [Accessed June 12, 2026]. [190] —, “Introducing GPT-5.4,” https://openai.com/index/introducing-gpt-5-4/, Mar. 2026, [Accessed June 12, 2026]. [191] A. Ormazabal, C. Zheng, C. de Masson d’Autume, D. Yogatama, D. Fu, D. Ong, E. Chen, E. Lamprecht, H. Pham, I. Ong et al., “Reka Core, Flash, and Edge: A Series of Powerful Multimodal Language Models,” Apr. 2024, arXiv:2404.12387. [Online]. Available: https://arxiv.org/abs/2404.12387 [192] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al., “Training Language Models to Follow Instructions with Human Feedback,” in Proceedings of the Thirty-Sixth Annual Conference on Neural Information Processing Systems (NeurIPS ’22), ser. Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., vol. 35. New Orleans, LA, USA: Curran Associates, Dec. 2022, p. 27 730–27 744. [Online]. Available: https://proceedings.neurips.c/paper_files/paper/2022/hash/b1efde53be364a73914f58805a001731-Abstract-Conference.html [193] D. Papakyriakou and I. S. Barbounakis, “Data Mining Methods: A Review,” International Journal of Computer Applications, vol. 183, no. 48, p. 5–19, Jan. 2022. [Online]. Available: https://ijcaonline.org/archives/volume183/number48/32253-2022921884/ [194] L. Patel, S. Jha, M. Pan, H. Gupta, P. Asawa, C. Guestrin, and M. Zaharia, “Semantic Operators and Their Optimization: Enabling LLM-Based Data Processing with Accuracy Guarantees in LOTUS,” Proc. VLDB Endow., vol. 18, no. 11, p. 4171–4184, Sep. 2025. [Online]. Available: https://doi.org/10.14778/3749646.3749685 [195] S. Potluri, H. Wang, D. Bureddy, A. Singh, C. Rosales, and D. K. Panda, “Optimizing MPI Communication on Multi-GPU Systems Using CUDA Inter-Process Communication,” in Proceedings of the IEEE 26th International Parallel and Distributed Processing Symposium Workshops & PhD Forum, ser. IPDPSW ’12. Shanghai, China: IEEE Press, May 2012, p. 1848–1857. [Online]. Available: https://ieeexplore.ieee.org/document/6270863 [196] S. Potluri, K. Hamidouche, A. Venkatesh, D. Bureddy, and D. K. Panda, “Efficient Inter-Node MPI Communication Using GPUDirect RDMA for InfiniBand Clusters with NVIDIA GPUs,” in Proceedings of the 42nd International Conference on Parallel Processing, ser. ICPP ’13. Lyon, France: IEEE Press, Oct. 2013, p. 80–89. [Online]. Available: https://ieeexplore.ieee.org/document/6687341 [197] P. Qi, X. Wan, G. Huang, and M. Lin, “Zero Bubble (Almost) Pipeline Parallelism,” in Proceedings of the Twelfth International Conference on Learning Representations, ser. ICLR ’24, B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun, Eds., Vienna, Austria, May 2024, p. 48 869–48 884. [Online]. Available: https://proceedings.iclr.c/paper_files/paper/2024/hash/d5a8e37f38a08c68162452dcba89ae9c-Abstract-Conference.html [198] P. Qi, X. Zhou, Z. Liu, T. Pang, C. Du, M. Lin, and W. S. Lee, “Rethinking the Trust Region in LLM Reinforcement Learning,” Jun. 2026, arXiv:2602.04879. [Online]. Available: https://arxiv.org/abs/2602.04879 [199] Z. Qiu, C. Li, Y. Peng, G. He, B. Yuan, and C. Wang, “TQA-Bench: Evaluating LLMs for Multi-Table Question Answering,” Jun. 2026, arXiv:2411.19504. [Online]. Available: https://arxiv.org/abs/2411.19504 [200] R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn, “Direct Preference Optimization: Your Language Model is Secretly a Reward Model,” in Proceedings of the Thirty-Seventh Annual Conference on Neural Information Processing Systems (NeurIPS ’23), ser. Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds., vol. 36. New Orleans, LA, USA: Curran Associates, Dec. 2023, p. 53 728–53 741. [Online]. Available: https://proceedings.neurips.c/paper_files/paper/2023/hash/a85b405ed65c6477a4fe8302b5e06ce7-Abstract-Conference.html [201] S. Rajbhandari, J. Rasley, O. Ruwase, and Y. He, “ZeRO: Memory Optimizations Toward Training Trillion Parameter Models,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, ser. SC ’20. Atlanta, GA, USA: IEEE Press, Nov. 2020, p. 20:1–20:16. [Online]. Available: https://ieeexplore.ieee.org/document/9355301 [202] S. Rando, L. Romani, A. Sampieri, L. Franco, J. Yang, Y. Kyuragi, F. Galasso, and T. Hashimoto, “LongCodeBench: Evaluating Coding LLMs at 1M Context Windows,” in Proceedings of the Second Conference on Language Modeling, ser. COLM ’25. Montreal, Canada: OpenReview, Oct. 2025, p. 1–17. [Online]. Available: https://openreview.net/forum?id=GFPoM8Ylp8 [203] T. Ren, J. Jiang, H. Yang, W. Tian, M. Zou, G. Li, Z. Zhang, Q. Wang, S. Qin, Y. Zhao, R. Tao, H. Shao, and Y. Peng, “RiskPO: Risk-Based Policy Optimization with Verifiable Reward for LLM Post-Training,” in Proceedings of the Fourteenth International Conference on Learning Representations, ser. ICLR ’26. Rio de Janeiro, Brazil: OpenReview, Apr. 2026, p. 1–20. [Online]. Available: https://openreview.net/forum?id=KjHB7rebQO [204] M. Riviere, S. Pathak, P. G. Sessa, C. Hardin, S. Bhupatiraju, L. Hussenot, T. Mesnard, B. Shahriari, A. Ramé, J. Ferret et al., “Gemma 2: Improving Open Language Models at a Practical Size,” Oct. 2024, arXiv:2408.00118. [Online]. Available: https://arxiv.org/abs/2408.00118 [205] M. Robeyns, M. Szummer, and L. Aitchison, “A Self-Improving Coding Agent,” in Proceedings of the Workshop on Scaling Self-Improving Foundation Models, ser. SSI-FM ’25. Singapore: OpenReview, Apr. 2025, p. 1–18. [Online]. Available: https://openreview.net/forum?id=rShJCyLsOr [206] A. Salemi and H. Zamani, “Evaluating Retrieval Quality in Retrieval-Augmented Generation,” in Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, ser. SIGIR ’24. Washington, DC, USA: Association for Computing Machinery, Jul. 2024, p. 2395–2400. [Online]. Available: https://doi.org/10.1145/3626772.3657957 [207] P. Schmid, M. Besta, and T. Hoefler, “High-Performance Distributed RMA Locks,” in Proceedings of the 25th ACM International Symposium on High-Performance Parallel and Distributed Computing, ser. HPDC ’16. Kyoto, Japan: Association for Computing Machinery, Jun. 2016, p. 19–30. [Online]. Available: https://doi.org/10.1145/2907294.2907323 [208] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal Policy Optimization Algorithms,” Aug. 2017, arXiv:1707.06347. [Online]. Available: https://arxiv.org/abs/1707.06347 [209] V. Seshadri, D. Lee, T. Mullins, H. Hassan, A. Boroumand, J. Kim, M. A. Kozuch, O. Mutlu, P. B. Gibbons, and T. C. Mowry, “Ambit: In-Memory Accelerator for Bulk Bitwise Operations Using Commodity DRAM Technology,” in Proceedings of the 50th Annual IEEE/ACM International Symposium on Microarchitecture, ser. MICRO ’17. Cambridge, MA, USA: Association for Computing Machinery, Oct. 2017, p. 273–287. [Online]. Available: https://doi.org/10.1145/3123939.3124544 [210] H. Shafiei, A. Khonsari, and P. Mousavi, “Serverless Computing: A Survey of Opportunities, Challenges, and Applications,” ACM Comput. Surv., vol. 54, no. 11s, p. 239:1–239:32, Nov. 2022. [Online]. Available: https://doi.org/10.1145/3510611 [211] S. Shankar, T. Chambers, T. Shah, A. G. Parameswaran, and E. Wu, “DocETL: Agentic Query Rewriting and Evaluation for Complex Document Processing,” Proc. VLDB Endow., vol. 18, no. 9, p. 3035–3048, Sep. 2025. [Online]. Available: https://doi.org/10.14778/3746405.3746426 [212] R. Shao, S. S. Li, R. Xin, S. Geng, Y. Wang, S. Oh, S. S. Du, N. Lambert, S. Min, R. Krishna et al., “Spurious Rewards: Rethinking Training Signals in RLVR,” Feb. 2026, arXiv:2506.10947. [Online]. Available: https://arxiv.org/abs/2506.10947 [213] Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, D. Guo et al., “DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models,” Apr. 2024, arXiv:2402.03300. [Online]. Available: https://arxiv.org/abs/2402.03300 [214] G. Shen, Z. Wang, O. Delalleau, J. Zeng, Y. Dong, D. Egert, S. Sun, J. J. Zhang, S. Jain, A. Taghibakhshi, M. S. Ausin, A. Aithal, and O. Kuchaiev, “NeMo-Aligner: Scalable Toolkit for Efficient Model Alignment,” in Proceedings of the First Conference on Language Modeling, ser. COLM ’24. Philadelphia, PA, USA: OpenReview, Oct. 2024, p. 1–16. [Online]. Available: https://openreview.net/forum?id=yK2eGE8QVW [215] Z. Shen, “LLM With Tools: A Survey,” Sep. 2024, arXiv:2409.18807. [Online]. Available: https://arxiv.org/abs/2409.18807 [216] G. Sheng, Y. Tong, B. Wan, W. Zhang, C. Jia, X. Wu, Y. Wu, X. Li, C. Zhang, Y. Peng, H. Lin, X. Liu, and C. Wu, “Laminar: A Scalable Asynchronous RL Post-Training Framework,” Oct. 2025, arXiv:2510.12633. [Online]. Available: https://arxiv.org/abs/2510.12633 [217] G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng, H. Lin, and C. Wu, “HybridFlow: A Flexible and Efficient RLHF Framework,” in Proceedings of the Twentieth European Conference on Computer Systems, ser. EuroSys ’25. Rotterdam, The Netherlands: Association for Computing Machinery, Apr. 2025, p. 1279–1297. [Online]. Available: https://doi.org/10.1145/3689031.3696075 [218] L. Shi, H. Zhang, Y. Yao, Z. Li, and H. Zhao, “Keep the Cost Down: A Review on Methods to Optimize LLM’s KV-Cache Consumption,” in Proceedings of the First Conference on Language Modeling, ser. COLM ’24. Philadelphia, PA, USA: OpenReview, Oct. 2024, p. 1–19. [Online]. Available: https://openreview.net/forum?id=8tKjqqMM5z [219] M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro, “Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism,” Mar. 2020, arXiv:1909.08053. [Online]. Available: https://arxiv.org/abs/1909.08053 [220] A. Singh, A. Fry, A. Perelman, A. Tart, A. Ganesh, A. El-Kishky, A. McLaughlin, A. Low, A. Ostrow, A. Ananthram et al., “OpenAI GPT-5 System Card,” May 2026, arXiv:2601.03267. [Online]. Available: https://arxiv.org/abs/2601.03267 [221] C. Snell, J. Lee, K. Xu, and A. Kumar, “Scaling LLM Test-Time Compute Optimally Can be More Effective than Scaling Parameters for Reasoning,” in Proceedings of the Thirteenth International Conference on Learning Representations, ser. ICLR ’25, Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu, Eds., Singapore, Apr. 2025, p. 10 131–10 165. [Online]. Available: https://proceedings.iclr.c/paper_files/paper/2025/hash/1b623663fd9b874366f3ce019fdfdd44-Abstract-Conference.html [222] A. Strausz, F. Vella, S. Di Girolamo, M. Besta, and T. Hoefler, “Asynchronous Distributed-Memory Triangle Counting and LCC with RMA Caching,” in Proceedings of the IEEE International Parallel and Distributed Processing Symposium, ser. IPDPS ’22. Lyon, France: IEEE Press, May 2022, p. 291–301. [Online]. Available: https://ieeexplore.ieee.org/document/9820724 [223] S. Sun, Y. Zhang, A. Bukharin, D. Mosallanezhad, J. Zeng, S. Singhal, G. Shen, A. Renduchintala, T. Konuk, Y. Dong et al., “Reward-Aware Preference Optimization: A Unified Mathematical Framework for Model Alignment,” Feb. 2025, arXiv:2502.00203. [Online]. Available: https://arxiv.org/abs/2502.00203 [224] F. Tajwar, G. Zeng, Y. Zhou, Y. Song, D. Arora, Y. Jiang, J. Schneider, R. Salakhutdinov, H. Feng, and A. Zanette, “Maximum Likelihood Reinforcement Learning,” Feb. 2026, arXiv:2602.02710. [Online]. Available: https://arxiv.org/abs/2602.02710 [225] Z. Tan, H. Geng, X. Yu, M. Zhang, G. Wan, Y. Zhou, Q. He, X. Xue, H. Zhou, Y. Fan et al., “Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoning,” Apr. 2026, arXiv:2509.25300. [Online]. Available: https://arxiv.org/abs/2509.25300 [226] R. Teknium, J. Quesnelle, and C. Guang, “Hermes 3 Technical Report,” Aug. 2024, arXiv:2408.11857. [Online]. Available: https://arxiv.org/abs/2408.11857 [227] R. Thakur, R. Rabenseifner, and W. Gropp, “Optimization of Collective Communication Operations in MPICH,” The International Journal of High Performance Computing Applications, vol. 19, no. 1, p. 49–66, Feb. 2005. [Online]. Available: https://journals.sagepub.com/doi/10.1177/1094342005051521 [228] M. Tomar, L. Shani, Y. Efroni, and M. Ghavamzadeh, “Mirror Descent Policy Optimization,” in Proceedings of the Tenth International Conference on Learning Representations, ser. ICLR ’22. Virtual Event: OpenReview, Apr. 2022, p. 1–24. [Online]. Available: https://openreview.net/forum?id=aBO5SvgSt1 [229] L. Tunstall, E. E. Beeching, N. Lambert, N. Rajani, K. Rasul, Y. Belkada, S. Huang, L. V. Werra, C. Fourrier, N. Habib et al., “Zephyr: Direct Distillation of LM Alignment,” in Proceedings of the First Conference on Language Modeling, ser. COLM ’24. Philadelphia, PA, USA: OpenReview, Oct. 2024, p. 1–15. [Online]. Available: https://openreview.net/forum?id=aKkAwZB6JV [230] vLLM Contributors, “Parallelism and Scaling,” https://docs.vllm.ai/en/stable/serving/parallelism_scaling/, May 2026, [Accessed June 11, 2026]. [231] L. von Werra, Y. Belkada, L. Tunstall, E. Beeching, T. Thrush, N. Lambert, S. Huang, K. Rasul, and Q. Gallouédec, “TRL – Transformer Reinforcement Learning,” https://github.com/huggingface/trl, May 2026, [Accessed June 11, 2026]. [232] B. Wang, R. Zheng, L. Chen, Y. Liu, S. Dou, C. Huang, W. Shen, S. Jin, E. Zhou, C. Shi et al., “Secrets of RLHF in Large Language Models Part I: Reward Modeling,” Jan. 2024, arXiv:2401.06080. [Online]. Available: https://arxiv.org/abs/2401.06080 [233] H. Wang, X. Li, D. Wang, H. Zhou, Z. Huang, Y. Yang, J. Li, and Y. Ban, “Policy Improvement Reinforcement Learning,” Jun. 2026, arXiv:2604.00860. [Online]. Available: https://arxiv.org/abs/2604.00860 [234] J. Wang and Y. Chen, “A Review on Code Generation with LLMs: Application and Evaluation,” in Proceedings of the IEEE International Conference on Medical Artificial Intelligence, ser. MedAI ’23. Beijing, China: IEEE Press, Nov. 2023, p. 284–289. [Online]. Available: https://ieeexplore.ieee.org/document/10403378 [235] P. Wang, L. Li, Z. Shao, R. Xu, D. Dai, Y. Li, D. Chen, Y. Wu, and Z. Sui, “Math-Shepherd: Verify and Reinforce LLMs Step-by-Step without Human Annotations,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ser. ACL ’24, L.-W. Ku, A. Martins, and V. Srikumar, Eds. Bangkok, Thailand: Association for Computational Linguistics, Aug. 2024, p. 9426–9439. [Online]. Available: https://aclanthology.org/2024.acl-long.510/ [236] S. Wang, J. Cao, and P. S. Yu, “Deep Learning for Spatio-Temporal Data Mining: A Survey,” IEEE Transactions on Knowledge and Data Engineering, vol. 34, no. 8, p. 3681–3700, Aug. 2022. [Online]. Available: https://ieeexplore.ieee.org/document/9204396 [237] S. Wang, S. Zhang, J. Zhang, R. Hu, X. Li, T. Zhang, J. Li, F. Wu, G. Wang, and E. Hovy, “Reinforcement Learning Enhanced LLMs: A Survey,” Feb. 2025, arXiv:2412.10400. [Online]. Available: https://arxiv.org/abs/2412.10400 [238] T. Wang, S. Li, and W. Lu, “Self-Training with Direct Preference Optimization Improves Chain-of-Thought Reasoning,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ser. ACL ’26, L.-W. Ku, A. Martins, and V. Srikumar, Eds. Bangkok, Thailand: Association for Computational Linguistics, Aug. 2024, p. 11 917–11 928. [Online]. Available: https://aclanthology.org/2024.acl-long.643/ [239] T. Wang, Y. Li, L. Li, Y. Chen, S. Huang, Y. Chen, P. Li, Y. Liu, and G. Chen, “SPPO: Sequence-Level PPO for Long-Horizon Reasoning Tasks,” in Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ser. ACL ’26, M. Liakata, V. P. Moreira, J. Zhang, and D. Jurgens, Eds. San Diego, CA, USA: Association for Computational Linguistics, Jul. 2026, p. 7831–7853. [Online]. Available: https://aclanthology.org/2026.acl-long.356/ [240] W. Wang, S. Xiong, G. Chen, W. Gao, S. Guo, Y. He, J. Huang, J. Liu, Z. Li, X. Li et al., “Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Library,” Jun. 2025, arXiv:2506.06122. [Online]. Available: https://arxiv.org/abs/2506.06122 [241] W. Wang, Z. Gao, L. Chen, Z. Chen, J. Zhu, X. Zhao, Y. Liu, Y. Cao, S. Ye, X. Zhu et al., “VisualPRM: An Effective Process Reward Model for Multimodal Reasoning,” Mar. 2025, arXiv:2503.10291. [Online]. Available: https://arxiv.org/abs/2503.10291 [242] Z. Wang, C. Li, Y. Zhang, H. Liu, B. Wang, D. Chu, and D. Sui, “VPO: Reasoning Preferences Optimization Based on V-Usable Information,” in Proceedings of the Thirty-Ninth Annual Conference on Neural Information Processing Systems (NeurIPS ’25), ser. Advances in Neural Information Processing Systems, D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen, Eds., vol. 38. Mexico City, Mexico: Curran Associates, Dec. 2025, p. 171 903–171 928. [Online]. Available: https://proceedings.neurips.c/paper_files/paper/2025/hash/fad622800f1d8ed14ef2e917469f443a-Abstract-Conference.html [243] X. Wen, Z. Liu, S. Zheng, S. Ye, Z. Wu, Y. Wang, Z. Xu, X. Liang, J. Li, Z. Miao, J. Bian, and M. Yang, “Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs,” in Proceedings of the Fourteenth International Conference on Learning Representations, ser. ICLR ’26. Rio de Janeiro, Brazil: OpenReview, Apr. 2026, p. 1–34. [Online]. Available: https://openreview.net/forum?id=jGbRWwIidy [244] L. Weng, X. Wang, J. Lu, Y. Feng, Y. Liu, H. Feng, D. Huang, and W. Chen, “InsightLens: Augmenting LLM-Powered Data Analysis with Interactive Insight Management and Navigation,” IEEE Transactions on Visualization and Computer Graphics, vol. 31, no. 6, p. 3719–3732, Jun. 2025. [Online]. Available: https://ieeexplore.ieee.org/document/10989518 [245] G. Wölflein, D. Ferber, D. Truhn, O. Arandjelovic, and J. N. Kather, “LLM Agents Making Agent Tools,” in Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ser. ACL ’25, W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar, Eds. Vienna, Austria: Association for Computational Linguistics, Jul. 2025, p. 26 092–26 130. [Online]. Available: https://aclanthology.org/2025.acl-long.1266/ [246] J. Wu, X. Wang, Z. Yang, J. Wu, J. Gao, B. Ding, X. Wang, and X. He, “AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization,” in Proceedings of the 42nd International Conference on Machine Learning (ICML ’25), ser. Proceedings of Machine Learning Research, A. Singh, M. Fazel, D. Hsu, S. Lacoste-Julien, F. Berkenkamp, T. Maharaj, K. Wagstaff, and J. Zhu, Eds., vol. 267. Vancouver, Canada: PMLR, Jul. 2025, p. 67 793–67 809. [Online]. Available: https://proceedings.mlr.press/v267/wu25af.html [247] Y. Wu, Z. Sun, S. Li, S. Welleck, and Y. Yang, “Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for LLM Problem-Solving,” in Proceedings of the Thirteenth International Conference on Learning Representations, ser. ICLR ’25, Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu, Eds., Singapore, Apr. 2025, p. 55 824–55 845. [Online]. Available: https://proceedings.iclr.c/paper_files/paper/2025/hash/8c3caae2f725c8e2a55ecd600563d172-Abstract-Conference.html [248] Z. Wu, S. Pan, F. Chen, G. Long, C. Zhang, and P. S. Yu, “A Comprehensive Survey on Graph Neural Networks,” IEEE Transactions on Neural Networks and Learning Systems, vol. 32, no. 1, p. 4–24, Jan. 2021. [Online]. Available: https://ieeexplore.ieee.org/document/9046288 [249] xAI, “Grok 4,” https://x.ai/news/grok-4, Jul. 2025, [Accessed June 12, 2026]. [250] W. Xiao, Z. Wang, L. Gan, S. Zhao, Z. Li, R. Lei, W. He, L. A. Tuan, L. Chen, H. Jiang, Z. Zhao, and F. Wu, “A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications,” Jun. 2026, arXiv:2410.15595. [Online]. Available: https://arxiv.org/abs/2410.15595 [251] Y. Xiao, Z. Zhou, F. Mao, W. Wu, S. Zhao, L. Ju, L. Liang, X. Zhang, and J. Zhou, “FlexRLHF: A Flexible Placement and Parallelism Framework for Efficient RLHF Training,” in Proceedings of the IEEE International Parallel and Distributed Processing Symposium, ser. IPDPS ’25. Milano, Italy: IEEE Press, Jun. 2025, p. 358–369. [Online]. Available: https://ieeexplore.ieee.org/document/11078517 [252] H. Xu, X. Mao, F.-L. Li, X. Wu, W. Chen, W. Zhang, and A. T. Luu, “Full-Step-DPO: Self-Supervised Preference Optimization with Step-Wise Rewards for Mathematical Reasoning,” in Findings of the Association for Computational Linguistics: ACL 2025, W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar, Eds. Vienna, Austria: Association for Computational Linguistics, Jul. 2025, p. 24 343–24 356. [Online]. Available: https://aclanthology.org/2025.findings-acl.1249/ [253] M. Xu, D. Cai, W. Yin, S. Wang, X. Jin, and X. Liu, “Resource-Efficient Algorithms and Systems of Foundation Models: A Survey,” ACM Comput. Surv., vol. 57, no. 5, p. 110:1–110:39, Jan. 2025. [Online]. Available: https://doi.org/10.1145/3706418 [254] S. Xue, D. Qi, C. Jiang, F. Cheng, K. Chen, Z. Zhang, H. Zhang, G. Wei, W. Zhao, F. Zhou, H. Yi, S. Liu, H. Yang, and F. Chen, “Demonstration of DB-GPT: Next Generation Data Interaction System Empowered by Large Language Models,” Proc. VLDB Endow., vol. 17, no. 12, p. 4365–4368, Aug. 2024. [Online]. Available: https://doi.org/10.14778/3685800.3685876 [255] K. Yan, Y. Yu, Y. Yu, H. Zheng, and F. Lai, “OPPO: Accelerating PPO-Based RLHF via Pipeline Overlap,” Mar. 2026, arXiv:2509.25762. [Online]. Available: https://arxiv.org/abs/2509.25762 [256] A. Yang, J. Yang, A. Ibrahim, X. Xie, B. Tang, G. Sizov, J. Park, and J. Huang, “Context Parallelism for Scalable Million-Token Inference,” in Proceedings of the Eighth Annual Conference on Machine Learning and Systems (MLSys ’25), ser. Proceedings of Machine Learning and Systems, M. Zaharia, G. Joshi, and Y. Lin, Eds., vol. 7, Santa Clara, CA, USA, May 2025, p. 1–16. [Online]. Available: https://proceedings.mlsys.org/paper_files/paper/2025/hash/78834433edc3291f4c6cbbd2759324db-Abstract-Conference.html [257] A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv et al., “Qwen3 Technical Report,” May 2025, arXiv:2505.09388. [Online]. Available: https://arxiv.org/abs/2505.09388 [258] A. Yang, B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou, C. Li, C. Li, D. Liu, F. Huang et al., “Qwen2 Technical Report,” Sep. 2024, arXiv:2407.10671. [Online]. Available: https://arxiv.org/abs/2407.10671 [259] F. Yao, C. Tian, J. Liu, Z. Zhang, Q. Liu, L. Jin, S. Li, X. Li, and X. Sun, “Thinking Like an Expert: Multimodal Hypergraph-of-Thought (HoT) Reasoning to Boost Foundation Modals,” Aug. 2023, arXiv:2308.06207. [Online]. Available: https://arxiv.org/abs/2308.06207 [260] S. Yao, D. Yu, J. Zhao, I. Shafran, T. Griffiths, Y. Cao, and K. Narasimhan, “Tree of Thoughts: Deliberate Problem Solving with Large Language Models,” in Proceedings of the Thirty-Seventh Annual Conference on Neural Information Processing Systems (NeurIPS ’23), ser. Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds., vol. 36. New Orleans, LA, USA: Curran Associates, Dec. 2023, p. 11 809–11 822. [Online]. Available: https://proceedings.neurips.c/paper_files/paper/2023/hash/271db9922b8d1f4d7aaef84ed5ac703-Abstract-Conference.html [261] Z. Yao, R. Y. Aminabadi, O. Ruwase, S. Rajbhandari, X. Wu, A. A. Awan, J. Rasley, M. Zhang, C. Li, C. Holmes, Z. Zhou, M. Wyatt, M. Smith, L. Kurilenko, H. Qin, M. Tanaka, S. Che, S. L. Song, and Y. He, “DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-Like Models at All Scales,” Aug. 2023, arXiv:2308.01320. [Online]. Available: https://arxiv.org/abs/2308.01320 [262] X. Ying, W. Mengdi, C. Long, L. Lian, Z. Shixin, Z. Lei, and W. Ying, “Pipe-RLHF: A Computation Mode-Aware Parallel Framework for RLHF,” Journal of Computer Research and Development, vol. 62, no. 6, p. 1513–1529, Jun. 2025. [Online]. Available: https://crad.ict.ac.cn/en/article/doi/10.7544/issn1000-1239.202550127 [263] Yotta Labs, “Performance Optimization for Reinforcement Learning on AMD GPUs,” https://w.yottalabs.ai/post/performance-optimization-for-reinforcement-learning-on-amd-gpus, Oct. 2025, [Accessed June 11, 2026]. [264] F. Yu, T. Liu, and K. Sun, “Optimizing Communication for Mixture-of-Experts Training with Hybrid Expert Parallel,” https://developer.nvidia.com/blog/optimizing-communication-for-mixture-of-experts-training-with-hybrid-expert-parallel/, Feb. 2026, [Accessed June 17, 2026]. [265] Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, W. Dai, T. Fan, G. Liu, J. Liu et al., “DAPO: An Open-Source LLM Reinforcement Learning System at Scale,” in Proceedings of the Thirty-Ninth Annual Conference on Neural Information Processing Systems (NeurIPS ’25), ser. Advances in Neural Information Processing Systems, D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen, Eds., vol. 38. San Diego, CA, USA: Curran Associates, Dec. 2025, p. 113 222–113 244. [Online]. Available: https://proceedings.neurips.c/paper_files/paper/2025/hash/a4277440d50f1f15d2cb4c14f7e0c0d2-Abstract-Conference.html [266] R. Yuan, M. Khandoga, and V. K. Sankarapu, “Beyond KL Divergence: Policy Optimization with Flexible Bregman Divergences for LLM Reasoning,” Feb. 2026, arXiv:2602.04380. [Online]. Available: https://arxiv.org/abs/2602.04380 [267] Y. Yue, “ColossalChat: An Open-Source Solution for Cloning ChatGPT with a Complete RLHF Pipeline,” https://medium.com/pytorch/5edf08fb538b, Mar. 2023, [Accessed June 11, 2026]. [268] Y. Yue, Y. Yuan, Q. Yu, X. Zuo, R. Zhu, W. Xu, J. Chen, C. Wang, T. Fan, Z. Du et al., “VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks,” Apr. 2025, arXiv:2504.05118. [Online]. Available: https://arxiv.org/abs/2504.05118 [269] S. Zeighami, Y. Lin, S. Shankar, and A. Parameswaran, “LLM-Powered Proactive Data Systems,” Bulletin of the Technical Committee on Data Engineering, vol. 49, no. 1, p. 90–103, Mar. 2025. [Online]. Available: http://sites.computer.org/debull/A25mar/p90.pdf [270] L. Zeng, L. Zhong, L. Zhao, T. Wei, L. Yang, J. He, C. Cheng, R. Hu, Y. Liu, S. Yan, H. Fang, and Y. Zhou, “Skywork-Math: Data Scaling Laws for Mathematical Reasoning in Large Language Models–The Story Goes On,” Jul. 2024. [Online]. Available: https://arxiv.org/abs/2407.08348 [271] H. Zhang, P. Wang, S. Diao, Y. Lin, R. Pan, H. Dong, D. Zhang, P. Molchanov, and T. Zhang, “Entropy-Regularized Process Reward Model,” Transactions on Machine Learning Research, p. 1–22, Jun. 2025. [Online]. Available: https://openreview.net/forum?id=cSxDH7N3x9 [272] J. Zhang, H. Zhang, R. Chakravarti, Y. Hu, P. Ng, A. Katsifodimos, H. Rangwala, G. Karypis, and A. Halevy, “CoddLLM: Empowering Large Language Models for Data Analytics,” Feb. 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2502.00329 [273] Z. Zhang, C. Zheng, Y. Wu, B. Zhang, R. Lin, B. Yu, D. Liu, J. Zhou, and J. Lin, “The Lessons of Developing Process Reward Models in Mathematical Reasoning,” in Findings of the Association for Computational Linguistics: ACL 2025, W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar, Eds. Vienna, Austria: Association for Computational Linguistics, Jul. 2025, p. 10 495–10 516. [Online]. Available: https://aclanthology.org/2025.findings-acl.547/ [274] Z. Zhang, P. Cui, and W. Zhu, “Deep Learning on Graphs: A Survey,” IEEE Transactions on Knowledge and Data Engineering, vol. 34, no. 1, p. 249–270, Jan. 2022. [Online]. Available: https://ieeexplore.ieee.org/document/9039675 [275] W. Zhao, P. Aggarwal, S. Saha, A. Celikyilmaz, J. Weston, and I. Kulikov, “The Majority Is Not Always Right: RL Training for Solution Aggregation,” Sep. 2025, arXiv:2509.06870. [Online]. Available: https://arxiv.org/abs/2509.06870 [276] X. Zhao, X. Zhou, and G. Li, “Chat2Data: An Interactive Data Analysis System with RAG, Vector Databases and LLMs,” Proc. VLDB Endow., vol. 17, no. 12, p. 4481–4484, Aug. 2024. [Online]. Available: https://doi.org/10.14778/3685800.3685905 [277] Y. Zhao, A. Gu, R. Varma, L. Luo, C.-C. Huang, M. Xu, L. Wright, H. Shojanazeri, M. Ott, S. Shleifer, A. Desmaison, C. Balioglu, P. Damania, B. Nguyen, G. Chauhan, Y. Hao, A. Mathews, and S. Li, “PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel,” Proc. VLDB Endow., vol. 16, no. 12, p. 3848–3860, Aug. 2023. [Online]. Available: https://doi.org/10.14778/3611540.3611569 [278] L. Zheng, L. Yin, Z. Xie, C. Sun, J. Huang, C. H. Yu, S. Cao, C. Kozyrakis, I. Stoica, J. E. Gonzalez, C. Barrett, and Y. Sheng, “SGLang: Efficient Execution of Structured Language Model Programs,” in Proceedings of the Thirty-Eighth Annual Conference on Neural Information Processing Systems (NeurIPS ’24), ser. Advances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, Eds., vol. 37. Vancouver, Canada: Curran Associates, Dec. 2024, p. 62 557–62 583. [Online]. Available: https://proceedings.neurips.c/paper_files/paper/2024/hash/fd0705710bf01b88a60a3d479ea341d9-Abstract-Conference.html [279] R. Zheng, S. Dou, S. Gao, Y. Hua, W. Shen, B. Wang, Y. Liu, S. Jin, Q. Liu, Y. Zhou, L. Xiong, L. Chen, Z. Xi, N. Xu, W. Lai, M. Zhu, C. Chang, Z. Yin, R. Weng, W. Cheng, H. Huang, T. Sun, H. Yan, T. Gui, Q. Zhang, X. Qiu, and X. Huang, “Secrets of RLHF in Large Language Models Part I: PPO,” Jul. 2023, arXiv:2307.04964. [Online]. Available: https://arxiv.org/abs/2307.04964 [280] H. Zhong, Z. Shan, G. Feng, W. Xiong, X. Cheng, L. Zhao, D. He, J. Bian, and L. Wang, “DPO Meets PPO: Reinforced Token Optimization for RLHF,” in Proceedings of the 42nd International Conference on Machine Learning (ICML ’25), ser. Proceedings of Machine Learning Research, A. Singh, M. Fazel, D. Hsu, S. Lacoste-Julien, F. Berkenkamp, T. Maharaj, K. Wagstaff, and J. Zhu, Eds., vol. 267. Vancouver, Canada: PMLR, Jul. 2025, p. 78 498–78 521. [Online]. Available: https://proceedings.mlr.press/v267/zhong25b.html [281] Y. Zhong, Z. Zhang, X. Song, H. Hu, C. Jin, B. Wu, N. Chen, Y. Chen, Y. Zhou, C. Wan, H. Zhou, Y. Jiang, Y. Zhu, and D. Jiang, “StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generation,” Apr. 2025, arXiv:2504.15930. [Online]. Available: https://arxiv.org/abs/2504.15930 [282] Y. Zhong, Z. Zhang, B. Wu, S. Liu, Y. Chen, C. Wan, H. Hu, L. Xia, R. Ming, Y. Zhu, and X. Jin, “Optimizing RLHF Training for Large Language Models with Stage Fusion,” in Proceedings of the 22nd USENIX Symposium on Networked Systems Design and Implementation, ser. NSDI ’25. Philadelphia, PA, USA: USENIX Association, Apr. 2025, p. 489–503. [Online]. Available: https://w.usenix.org/conference/nsdi25/presentation/zhong [283] J. Zhou, G. Cui, S. Hu, Z. Zhang, C. Yang, Z. Liu, L. Wang, C. Li, and M. Sun, “Graph Neural Networks: A Review of Methods and Applications,” AI Open, vol. 1, p. 57–81, 2020. [Online]. Available: https://w.sciencedirect.com/science/article/pii/S2666651021000012 [284] P. Zhou, C. Liu, J. Ren, X. Zhou, Y. Xie, M. Cao, Z. Rao, Y.-L. Huang, D. Chong, J. Liu, J. B. Kim, S. Wang, R. C.-W. Wong, and S. Kim, “When Large Vision Language Models Meet Multimodal Sequential Recommendation: An Empirical Study,” in Proceedings of the ACM on Web Conference, ser. W ’25. Sydney, Australia: Association for Computing Machinery, Apr. 2025, p. 275–292. [Online]. Available: https://doi.org/10.1145/3696410.3714764 [285] B. Zhu, E. Frick, T. Wu, H. Zhu, K. Ganesan, W.-L. Chiang, J. Zhang, and J. Jiao, “Starling-7B: Improving Helpfulness and Harmlessness with RLAIF,” in Proceedings of the First Conference on Language Modeling, ser. COLM ’24. Philadelphia, PA, USA: OpenReview, Oct. 2024, p. 1–26. [Online]. Available: https://openreview.net/forum?id=GqDntYTTbk [286] H. Zhu, Y. Ren, Y. Li, M. Lin, L. Yang, X. Liu, X. Zhen, H. Liu, and B. Zhang, “Unbiased Dynamic Pruning for Efficient Group-Based Policy Optimization,” Mar. 2026, arXiv:2603.04135. [Online]. Available: https://arxiv.org/abs/2603.04135 [287] Z. Zhu, C. Xie, X. Lv, and slime Contributors, “slime: An LLM Post-Training Framework for RL Scaling,” https://github.com/THUDM/slime, May 2026, [Accessed June 15, 2026]. Appendix A RLM Pipeline: Functional Description In the RLM pipeline, each stage can be understood as a function acting on batches, with clearly defined inputs and outputs. We now describe these stages in more detail. A-A Generation Stage The Generation stage takes as input a batch of prompts X=x1,…,xB⊂X=\x_1,…,x_B\ and the behavior policy πθold _ _old, which either comes from the previous RL iteration, or – for the 1st iteration – is the base model. Here, X is the space of input prompts. For each input prompt xb∈x_b , the policy model πθ(y∣xb) _θ(y x_b) generates a set of K candidate completions Genθ(xb)=yb(i)i=1K.Gen_θ(x_b)= \y_b^(i) \_i=1^K. The i-th candidate response to a prompt xbx_b is denoted as yb(i)∈y_b^(i) and it is a sequence of T tokens (yb,1(i),…,yb,T(i)) (y_b,1^(i),…,y_b,T^(i) ), where yb,t(i)y_b,t^(i) represents a token at time-step t; Y is the space of token sequences. The output is thus the rollout batch ℬroll=(xb,yb(i))b,iB_roll= \ (x_b,y_b^(i) ) \_b,i. Along with the sampled outputs, PPO-like methods often also compute per-token actor log-probabilities logπθ(yb,t(i)∣xb,yb,<t(i)) _θ (y_b,t^(i) x_b,y_b,<t^(i) ) for each step t∈1,…,Tt∈\1,…,T\, for use in the training stage. A-B Assessment Stage The Assessment stage takes the rollout batch ℬrollB_roll and evaluates it with the auxiliary models required by the algorithm. In PPO-like methods, this usually includes the reward model RφR_ , reference policy πref _ref, and critic VψV_ψ that are used to transform Generation’s output into numerical learning signals. The most important such signals are sequence-level scalar rewards rb(i)r_b^(i): a single quality score assigned to each complete sequence yb(i)y_b^(i), typically computed by the frozen reward model Rφ(xb,yb(i))R_ (x_b,y_b^(i) ). These scores quantify alignment with human preferences, such as helpfulness or safety. Moreover, the stage also produces token-level training targets Ab(i)=(Ab,1(i)Ab,2(i)…Ab,T(i))A_b^(i)= (A_b,1^(i)A_b,2^(i)...A_b,T^(i) ); each such target is a dense vector of values for each token position t∈1,…,Tt∈\1,…,T\. These values, commonly referred to as advantages, enable granular credit assignment by comparing the empirical return G^b,t(i) G_b,t^(i) against the estimate return by the critic model Vψ(xb,yb,<t(i))V_ψ (x_b,y_b,<t^(i) ). They are constructed from returns/rewards and, for PPO-like actor–critic methods, and critic value estimates, e.g., Ab,t(i)=G^b,t(i)−vb,t,iA_b,t^(i)= G_b,t^(i)-v_b,t,i. In critic-free methods such as GRPO, no VψV_ψ pass is performed; advantages are computed from group-normalized rewards instead. Finally, the Assessment stage also involves computing reference log-probabilities logπref(yb,t(i)∣xb,yb,<t(i)) _ref (y_b,t^(i) x_b,y_b,<t^(i) ). A-C Training Stage The Training stage takes the rollout batch and the learning signals from Assessment: rewards, reference log-probabilities, stored sample-time actor log-probabilities, and, when applicable, advantages and value targets. It updates the trainable model components required by the algorithm. In PPO, the actor uses policy-ratio terms such as πθ(yb,t(i)∣xb,yb,<t(i))/πθold(yb,t(i)∣xb,yb,<t(i)) _θ (y_b,t^(i) x_b,y_b,<t^(i) )/ _ _old (y_b,t^(i) x_b,y_b,<t^(i) ), while the critic is updated with a value-regression loss. In GRPO, only the actor is updated because no learned critic is used. In DPO, Training operates on preference pairs rather than online rollouts and updates only the policy using teacher-forced likelihoods of preferred and dispreferred responses. Thus, Training outputs the next policy πθ′ _θ , and updated critic parameters ψ′ψ only for algorithms that train a critic. A-D Iterative Loop This process is repeated in an iterative loop: after the policy update, the new parameters θ′←θ ←θ are used in the next Generation stage. The full loop therefore evolves as θ→Genθ(X)→Assess(X,Y)→Train(⋅)→θ′→…θ _θ(X) (X,Y) (·)→θ →… and continues until convergence or until a predetermined number of iterations is reached. Appendix B Details of Mathematical Derivations We offer details of mathematical derivations. To simplify equations, we use Cftok≡CfC_f^tok≡ C_f, Cbtok≡CbC_b^tok≡ C_b, and Cgenroll≡CgenC_gen^roll≡ C_gen. B-A Results for Building Blocks We now derive the model-specific building blocks used in Table I for forward and backward passes in terms of architectural parameters such as the number of layers L, hidden size d, feed-forward width dffd_f, vocabulary size V, and sequence lengths S,TS,T. We parameterize these values with a given model M in question, i.e., hidden size dMd_M, feed-forward width dM,ffd_M,f, and LML_M layers. B-A1 Forward-Pass FLOPs We begin with a standard dense Transformer layer. For a model M, one layer contains four dense attention projections: WQ,WK,WV,WO∈ℝdM×dMW_Q,W_K,W_V,W_O ^d_M× d_M, contributing 4dM24d_M^2 parameters, and two FFN projections, Wup∈ℝdM×dM,ffW_up ^d_M× d_M,f and Wdown∈ℝdM,f×dMW_down ^d_M,f× d_M, contributing 2dMdM,f2d_Md_M,f parameters. Thus, one layer contains 4dM2+2dMdM,f4d_M^2+2d_Md_M,f trainable projection parameters. Under the standard FLOP accounting for matrix multiplication, each parameter contributes approximately two FLOPs per processed token in the forward pass (one multiply and one add). Hence, the parameter-dependent forward FLOPs per token per layer are 2(4dM2+2dMdM,f).2 (4d_M^2+2d_Md_M,f ). Across LML_M layers, this becomes 2LM(4dM2+2dMdM,f).2L_M (4d_M^2+2d_Md_M,f ). In addition, self-attention performs parameter-free token-mixing operations, namely score computation and value aggregation. Under the same coarse cost model used throughout the paper, these contribute 2LM(S+T)dM2L_M(S+T)d_M FLOPs per token on a sequence of total length S+TS+T. Therefore, the base forward-pass cost per token for model M is Cf(M)=2LM(4dM2+2dMdM,f+(S+T)dM),C_f(M)=2L_M (4d_M^2+2d_Md_M,f+(S+T)d_M ), up to model-specific output heads. Policy and reference model We parameterize M∈π,refM∈\π,ref\. For the policy model πθ _θ, one additionally computes logits over the vocabulary, i.e. an unembedding/output projection of size dπ→Vd_π→ V. Under the same multiply–add accounting, this contributes 2Vdπ2Vd_π FLOPs per token. Hence Cf(πθ)=2Lπ(4dπ2+2dπdffπ+(S+T)dπ)+2Vdπ.C_f( _θ)=2L_π (4d_π^2+2d_πd^π_f+(S+T)d_π )+2Vd_π. Because the reference model πref _ref is architecturally identical to the actor, Cf(πref)=Cf(πθ).C_f( _ref)=C_f( _θ). Reward model Now, M=RM=R. The reward model RφR_ processes the full sequence but produces a single scalar score from the final hidden state. Its transformer backbone therefore costs 2LR(4dR2+2dRdffR+(S+T)dR).2L_R (4d_R^2+2d_Rd^R_f+(S+T)d_R ). The scalar reward head is a projection dR→1d_R→ 1 applied once per sequence, i.e. 2dR2d_R FLOPs per sequence, or equivalently 2dRS+T 2d_RS+T FLOPs per token when normalized by sequence length. Thus, Cf(Rφ)=2LR(4dR2+2dRdffR+(S+T)dR)+2dRS+T.C_f(R_ )=2L_R (4d_R^2+2d_Rd^R_f+(S+T)d_R )+ 2d_RS+T. Value model / critic Finally, M=VM=V. The critic VψV_ψ outputs one scalar value per token. Its transformer backbone contributes 2LV(4dV2+2dVdffV+(S+T)dV),2L_V (4d_V^2+2d_Vd^V_f+(S+T)d_V ), and its value head dV→1d_V→ 1 is applied at every token position, contributing 2dV2d_V FLOPs per token. Hence Cf(Vψ)=2LV(4dV2+2dVdffV+(S+T)dV)+2dV.C_f(V_ψ)=2L_V (4d_V^2+2d_Vd^V_f+(S+T)d_V )+2d_V. B-A2 Backward-pass FLOPs For dense Transformer training, a standard approximation is that the backward pass costs about twice the forward pass, because gradients must be propagated both with respect to activations and with respect to weights. Accordingly, for any trainable model M we use Cb(M)≈2Cf(M).C_b(M)≈ 2\,C_f(M). This is the approximation used throughout the paper when deriving training-stage costs. B-A3 Autoregressive generation: prefill and decode For policy generation, it is important to distinguish prefill from decode. Prefill Given a prompt of length S, the prefill pass processes the prompt once, initializes the KV cache, and produces the logits needed to begin generation. Therefore, its cost is simply the forward cost evaluated at sequence length S: Cprefill(πθ,S)=2LπS(4dπ2+2dπdffπ+Sdπ)+2Vdπ C_prefill( _θ;S)=2L_πS (4d_π^2+2d_πd^π_f+Sd_π )+2Vd_π Lπ[O(S2)+O(Sdπ)]. L_π [O(S^2)+O(Sd_π) ]. Decode After prefill, the model generates tokens autoregressively. At decode step t, the current token attends to a context of length S+tS+t, consisting of the prompt plus the t previously generated tokens. Hence the cost of the full decode phase is Cdec(πθ,S,T)=∑t=1T−1[2Lπ(4dπ2+2dπdffπ+(S+t)dπ)+2Vdπ].C_dec( _θ;S,T)= _t=1^T-1 [2L_π (4d_π^2+2d_πd^π_f+(S+t)d_π )+2Vd_π ]. Using ∑t=1T−1(S+t)=(T−1)S+(T−1)T2, _t=1^T-1(S+t)=(T-1)S+ (T-1)T2, this can be written as Cdec(πθ,S,T)= C_dec( _θ;S,T)= 2(T−1)Lπ(4dπ2+2dπdffπ) \ 2(T-1)L_π (4d_π^2+2d_πd^π_f ) + + 4Lπdπ((T−1)S+(T−1)T2) \ 4L_πd_π\! ((T-1)S+ (T-1)T2 ) + + 2(T−1)Vdπ. \ 2(T-1)Vd_π. Total generation cost The total generation cost is therefore Cgen(πθ,S,T)=Cprefill(πθ,S)+Cdec(πθ,S,T).C_gen( _θ;S,T)=C_prefill( _θ;S)+C_dec( _θ;S,T). This is the quantity that should be used in the most detailed generation accounting. For more compact asymptotic comparisons, one may further summarize it as Cgen(πθ,S,T)=O((S+T)Cf(πθ,S+T)),C_gen( _θ;S,T)=O\! ((S+T)\,C_f( _θ;S+T) ), but this coarser form should be understood as an upper-level abstraction of the explicit prefill+decode decomposition above. B-A4 Depth We now derive the depth terms used in Table I. Here, depth means the critical-path length under unbounded parallelism. Single forward pass Within one Transformer layer, the dense projections in the FFN contribute reduction depth logdM+logdM,f. d_M+ d_M,f. For the multi-head attention block, the depth comes from the sequential dependencies among the projection, attention, aggregation, and output-projection subcomputations. The query, key, and value projections are independent and can be computed in parallel; therefore, they contribute only logdM d_M to the critical path. Once Q and K are available, the attention-score computation within each head reduces over the per-head hidden dimension dM/hMd_M/h_M, contributing log(dM/hM)=logdM−loghM. (d_M/h_M)= d_M- h_M. The subsequent weighted aggregation of values reduces over the sequence length, contributing log(S+T) (S+T). Finally, the attention output projection maps the concatenated heads back to the hidden dimension and contributes another logdM d_M. Thus, suppressing constant factors, the multi-head attention depth is logdM+log(dM/hM)+log(S+T)+logdM= d_M+ (d_M/h_M)+ (S+T)+ d_M= 3logdM+log(S+T)−loghM. 3 d_M+ (S+T)- h_M. Since layers are sequential, the total forward depth scales linearly with LML_M. In the simplified cost model used in Table I, this is summarized as Df(M)=O(LM[logdM+log(S+T)]),D_f(M)=O\! (L_M[ d_M+ (S+T)] ), plus model-specific output-head terms. For the policy/reference model, the additional vocabulary projection contributes one more logarithmic reduction, yielding Df(πθ) D_f( _θ) =O(Lπ[logdπ+log(S+T)]+logdπ), =O\! (L_π[ d_π+ (S+T)]+ d_π ), Df(πref) D_f( _ref) =Df(πθ). =D_f( _θ). For the reward and value models, the scalar head contributes an additional logd d term: Df(Rφ)=O(LR[logdR+log(S+T)]+logdR),D_f(R_ )=O\! (L_R[ d_R+ (S+T)]+ d_R ), Df(Vψ)=O(LV[logdV+log(S+T)]+logdV).D_f(V_ψ)=O\! (L_V[ d_V+ (S+T)]+ d_V ). Tensor-parallel forward depth If tensor parallelism of degree PtP_t shards the hidden dimension, the corresponding reduction depth decreases accordingly, giving DfTP(M)=O(LM[log(dM/Pt)+log(S+T)]+log(dM/Pt)).D_f^TP(M)=O\! (L_M[ (d_M/P_t)+ (S+T)]+ (d_M/P_t) ). Backward depth Under the same coarse-grained model, the backward pass has a different dependency structure to that of the forward pass. Namely, for any product of activations with model weights, one must consider computing gradients with respect to both the activations and the weights. Computing the gradients with respect to activations has the same asymptotic dependency structure as in the forward pass. Computing the gradients with respect to model weights, however, has to accumulate gradients across the whole batch. Thus Db(M)=Df(M)+O(log(B(S+T))) D_b(M)=D_f(M)+O( (B (S+T))) =O(L(logdM+log(S+T))+log(B(S+T))). =O(L( d_M+ (S+T))+ (B (S+T))). As gradients with respect to weights do not need to be backpropagated to the previous layer, log(B(S+T)) (B(S+T)) does not have to be multiplied with L. Generation depth Autoregressive generation has two components: one prefill pass plus a sequence of sequential decode steps. Therefore, Dgen(πθ,S,T)=Df(πθ,S)+∑t=1T−1Df(πθ,S+t).D_gen( _θ;S,T)=D_f( _θ;S)+ _t=1^T-1D_f( _θ;S+t). In compact asymptotic form, this is Dgen(πθ,S,T)=O(T⋅Df(πθ,S+T)),D_gen( _θ;S,T)=O\! (T· D_f( _θ;S+T) ), which makes explicit that the dominant sequential dependence comes from the T decode steps. B-A5 Parameter counts We finally derive the parameter-count and memory expressions used in Table I. Policy/reference model Ignoring biases and layer-norm parameters, the actor contains |πθ|=Lπ(4dπ2+2dπdffπ)+Vdπ| _θ|=L_π(4d_π^2+2d_πd^π_f)+Vd_π parameters: the first term comes from the stacked Transformer blocks, and the second from token embeddings / output vocabulary projection. Since the reference model shares the same architecture, |πref|=|πθ|.| _ref|=| _θ|. Shared actor–critic backbone For a shared actor–critic model with backbone width dacd_ac and LacL_ac layers, plus a policy head of size VdacVd_ac and a scalar value head of size dacd_ac, we obtain |πac|=Lac(4dac2+2dacdffac)+Vdac+dac.| _ac|=L_ac(4d_ac^2+2d_acd^ac_f)+Vd_ac+d_ac. Reward model The reward model consists of a Transformer backbone plus a scalar reward head: |Rφ|=LR(4dR2+2dRdffR)+VdR+dR.|R_ |=L_R(4d_R^2+2d_Rd^R_f)+Vd_R+d_R. Value model / critic Similarly, the value model consists of a Transformer backbone plus a scalar head: |Vψ|=LV(4dV2+2dVdffV)+VdV+dV.|V_ψ|=L_V(4d_V^2+2d_Vd^V_f)+Vd_V+d_V. KV-cache memory During generation, each layer stores both keys and values for every processed token. For one rollout of total length S+TS+T, this yields MKV=2(S+T)Lπdπ,M_KV=2(S+T)L_πd_π, where the factor 22 accounts for keys and values. If all BKBK rollouts are generated concurrently, the corresponding peak KV-cache memory is BK⋅MKV.BK· M_KV. If fewer rollouts are generated concurrently, the peak memory decreases proportionally, at the cost of reduced concurrency. Optimizer state For trainable models, optimizer-state memory is approximated by a factor of four times the parameter count, corresponding to parameters, gradients, and Adam first and second moments. Training activations and inference buffers For a trainable model M, backpropagation requires intermediate activations from the forward pass. Under the leading-order activation model used throughout the paper, we retain one hidden-state-sized contribution per layer and processed token: MAct(M)=Θ(LMdM)M_Act(M)= (L_Md_M) per token. Hence, for a batch of Q sequences of length S+TS+T, MAct,total(M)=Θ(Q(S+T)LMdM).M_Act,total(M)= \! (Q(S+T)L_Md_M ). This expression suppresses constant-factor storage for Q/K/V tensors, FFN intermediates, normalization and residual temporaries, and other implementation-specific buffers, and assumes no activation checkpointing. For a frozen forward-only model, intermediate layer buffers need not be retained for backpropagation and can be reused across layers. Assuming the attention matrix is not fully materialized, the leading-order inference buffer is MInf(M)=Θ(dM)M_Inf(M)= (d_M) per processed token, or MInf,total(M)=Θ(Q(S+T)dM)M_Inf,total(M)= \! (Q(S+T)d_M ) for Q sequences. Model parameters and autoregressive KV-cache storage are accounted for separately. B-B Results for RL-LLM Frameworks We start with derivations for results in Section I-E. B-B1 Online Frameworks Online methods such as PPO and GRPO repeatedly execute the full Generation → Assessment → Training loop, so their per-iteration cost is the sum of these three stages. Generation Generation is dominated by autoregressive sampling with the current policy πθ _θ. For each of the B prompts, the policy generates K candidate responses, yielding BKBK rollouts in total. The cost of one rollout should be decomposed into a prefill pass over the prompt of length S and a sequence of T decode steps: Cgen(πθ)=Cprefill(πθ,S)+∑t=1T−1Cdecode(πθ,S+t).C_gen( _θ)=C_prefill( _θ;S)+ _t=1^T-1C_decode( _θ;S+t). Here, the prefill initializes the KV cache from the prompt, while decode step t produces the next token while attending to a context of length S+tS+t. Thus, the total generation work over the whole batch is Wgen=Θ(BK⋅Cgen(πθ)).W_gen= \! (BK· C_gen( _θ) ). The critical path is determined by the autoregressive dependence across generated tokens: token t+1t+1 cannot be produced before token t has been generated. Therefore, the generation depth scales linearly in T: Dgen=Θ(T⋅Df(πθ)).D_gen= \! (T· D_f( _θ) ). This sequential dependence makes generation the primary wall-clock bottleneck in online RL-LLM pipelines. Assessment After generation, the completed prompt–response pairs are evaluated by auxiliary models such as the reward model RφR_ , the critic VψV_ψ, and the reference policy πref _ref. For PPO, all three are used; for GRPO, the critic is omitted. Each model processes the full sampled sequences in teacher-forced, non-autoregressive mode, so the work per model scales as Wassess(M)=Θ(BK(S+T)Cf(M)).W_assess(M)= \! (BK(S+T)\,C_f(M) ). Unlike generation, this stage does not incur a factor-T sequential dependence across output tokens. Its critical path is therefore the depth of a single forward pass through the corresponding model: Dassess(M)=Θ(Df(M)).D_assess(M)= \! (D_f(M) ). If the assessment models are placed on disjoint device groups and executed concurrently, the stage depth is the maximum of their forward-pass depths; if they are co-located and executed sequentially, these depths add. Training In the training stage, the policy and, for actor–critic methods, the critic are updated by backpropagation over the sampled sequences. Since training is performed in teacher-forced mode on the complete prompt–response sequences, the work per updated model and per optimization epoch scales as Wtrain(M)=Θ(BK(S+T)Ctr(M)).W_train(M)= \! (BK(S+T)\,C_tr(M) ). Again, there is no autoregressive factor-T dependence across tokens in the critical path; instead, the depth is that of a single backward pass: Dtrain(M)=Θ(Dtr(M)).D_train(M)= \! (D_tr(M) ). Thus, PPO incurs updates for both πθ _θ and VψV_ψ, whereas GRPO updates only πθ _θ. B-B2 Offline Frameworks Offline frameworks such as DPO decouple generation from optimization. Instead of sampling fresh rollouts inside the training loop, they optimize the policy πθ _θ on a static dataset of preference pairs (x,yw,yl)(x,y_w,y_l). As a result, the inner loop contains only batched forward and backward passes over fixed data. For a batch of BpairsB_pairs preference pairs, the forward work is Wforward=Θ(Bpairs(S+T)[Cf(πθ)+Cf(πref)]),W_forward= \! (B_pairs(S+T)\,[C_f( _θ)+C_f( _ref)] ), while the backward work is Wbackward=Θ(Bpairs(S+T)Cb(πθ)).W_backward= \! (B_pairs(S+T)\,C_b( _θ) ). The critical path is therefore the depth of a single batched forward/backward evaluation rather than an autoregressive decode chain: DDPO=Θ(maxDf(πθ),Df(πref),Db(πθ)).D_DPO= \! ( \D_f( _θ),D_f( _ref),D_b( _θ)\ ). B-C Results for Intra-Model Parallelism We now derive the results for Section I-G (Tables V, VI, VII, and VIII). The goal is to expose how standard intra-model parallelism strategies transform the work, depth, and memory terms derived in Appendix B-A. B-C1 Baseline Cost Model Appendix B-A derives the forward and backward costs of Transformer-based models in terms of layer count, hidden dimension, FFN width, vocabulary size, and sequence length. For the present intra-model analysis, we consider a dense Transformer training iteration with L layers, hidden dimension d, FFN dimension dffd_f, vocabulary size V, prompt length S, response length T, and batch size B. The complete teacher-forced training sequence therefore has length S+TS+T. Since Tables V–VIII analyze a generic Transformer training step rather than a role-specific actor, critic, reward, or reference model, we ignore small model-specific head differences and use the common dense Transformer body. From Appendix B-A, the forward-pass cost per token of the dense Transformer body is Cf=2L(4d2+2ddff+(S+T)d),C_f=2L (4d^2+2d_f+(S+T)d ), where 4d24d^2 comes from the query, key, value, and output projections, 2ddff2d_f from the two FFN projections, and (S+T)d(S+T)d from the sequence-dependent attention score/value operations. Using the same approximation as in Table I, Cb≈2Cf.C_b≈ 2C_f. Thus, the forward-plus-backward work per token is Cf+Cb≈3Cf=6L(4d2+2ddff+(S+T)d).C_f+C_b≈ 3C_f=6L (4d^2+2d_f+(S+T)d ). For B(S+T)B(S+T) processed tokens, the baseline work is Wbase=6B(S+T)L(4d2+2ddff+(S+T)d),W_base=6B(S+T)L (4d^2+2d_f+(S+T)d ), which gives the asymptotic summary Wbase=O(B(S+T)L(d2+(S+T)d))W_base=O\! (B(S+T)L(d^2+(S+T)d) ) in the parameter-dominated regime. We next refine the depth expression. A single forward Transformer layer has reduction depth 4logd+logdff+log(S+T)−logh,4 d+ d_f+ (S+T)- h, where h is the number of attention heads. Hence, Df=L(4logd+logdff+log(S+T)−logh).D_f=L (4 d+ d_f+ (S+T)- h ). The activation-gradient path in the backward pass has a comparable layerwise dependency structure. However, trainable projections also require parameter gradients. For a projection Y=XWY=XW, with the batch and sequence dimensions flattened into B(S+T)B(S+T) rows, ∇Xℒ=∇YℒW⊤,∇Wℒ=X⊤∇Yℒ. _XL= _YL\,W , _WL=X _YL. The second product accumulates contributions to each shared parameter over the B(S+T)B(S+T) batch-token positions and therefore introduces a reduction of depth log(B(S+T)). (B(S+T)). These parameter-gradient reductions branch from the activation-gradient backward path and can overlap with propagation through preceding layers; therefore they are not multiplied by L. We conservatively account for this additional critical-path contribution as Db=O(L[logd+log(S+T)]+log(B(S+T))).D_b=O\! (L[ d+ (S+T)]+ (B(S+T)) ). Under the explicit reduction-depth model used in Tables VII–VIII, we therefore use Dbase D_base =2L(4logd+logdff+log(S+T)−logh) =2L (4 d+ d_f+ (S+T)- h ) +log(B(S+T)) + (B(S+T)) =2Llog(d4dff(S+T)h)+log(B(S+T)). =2L \! ( d^4d_f(S+T)h )+ (B(S+T)). The corresponding asymptotic form is Dbase=O(L[logd+log(S+T)]+log(B(S+T))).D_base=O\! (L[ d+ (S+T)]+ (B(S+T)) ). Finally, the memory model follows the same parameter-counting convention as Appendix B-A. The dominant model-state terms are parameters, gradients, and Adam first and second moments, giving a factor of four over the parameter count: 4L(4d2+2ddff)+4Vd=L(16d2+8ddff)+4Vd.4L(4d^2+2d_f)+4Vd=L(16d^2+8d_f)+4Vd. Stored activations contribute B(S+T)Ld.B(S+T)Ld. Therefore, Mbase=L(16d2+8ddff)+4Vd+B(S+T)Ld,M_base=L(16d^2+8d_f)+4Vd+B(S+T)Ld, or asymptotically Mbase=O(Ld2+Vd+B(S+T)Ld).M_base=O\! (Ld^2+Vd+B(S+T)Ld ). The remaining derivations apply each intra-model parallelism strategy as a partitioning transformation of these baseline work, depth, and memory terms. B-C2 Data Parallelism Data parallelism of degree PdP_d partitions the batch dimension. Each rank processes B/PdB/P_d sequences but stores a full model replica. Therefore, the per-rank work is WDP,rank W_DP,rank =1PdWbase = 1P_dW_base =6BPd(S+T)L(4d2+2ddff+(S+T)d), =6 BP_d(S+T)L (4d^2+2d_f+(S+T)d ), while the global work remains WDP,global=Wbase.W_DP,global=W_base. The layerwise forward/backward dependency is unchanged. Locally, however, each rank forms parameter gradients from only BPd(S+T) BP_d(S+T) batch-token positions. Hence, DDP,rank=2Llog(d4dff(S+T)h)+log(BPd(S+T)).D_DP,rank=2L \! ( d^4d_f(S+T)h )+ \! ( BP_d(S+T) ). The globally aggregated parameter gradient still depends on all B(S+T)B(S+T) positions, so, ignoring communication latency while retaining the logical gradient-reduction dependency, DDP,global=Dbase.D_DP,global=D_base. The per-rank activation memory is reduced by PdP_d, but model-state memory is replicated: MDP,rank=L(16d2+8ddff)+4Vd+BPd(S+T)Ld.M_DP,rank=L(16d^2+8d_f)+4Vd+ BP_d(S+T)Ld. At the global level, the activation term sums back to B(S+T)LdB(S+T)Ld, while the model-state term is replicated PdP_d times: MDP,global=Pd[L(16d2+8ddff)+4Vd]+B(S+T)Ld.M_DP,global=P_d\! [L(16d^2+8d_f)+4Vd ]+B(S+T)Ld. Thus, DP reduces per-rank work and activation memory, while the full logical model update retains the global batch-gradient dependency. B-C3 Pipeline Parallelism Pipeline parallelism of degree PpP_p partitions the layer dimension. Each pipeline stage stores and computes approximately L/PpL/P_p layers. Applying L↦LPpL LP_p to the baseline per-rank formulas gives WPP,rank=6B(S+T)LPp(4d2+2ddff+(S+T)d),W_P,rank=6B(S+T) LP_p (4d^2+2d_f+(S+T)d ), and DPP,rank=2LPplog(d4dff(S+T)h)+log(B(S+T)).D_P,rank=2 LP_p \! ( d^4d_f(S+T)h )+ (B(S+T)). The layer-dependent memory terms are similarly divided across pipeline stages: MPP,rank=LPp(16d2+8ddff)+4Vd+B(S+T)LPpd.M_P,rank= LP_p(16d^2+8d_f)+4Vd+B(S+T) LP_pd. The vocabulary term 4Vd4Vd is written separately because embeddings and output heads are commonly pinned to boundary stages rather than evenly partitioned across all stages. Globally, pipeline parallelism preserves total work and total model-state memory up to boundary-stage effects: WPP,global=Wbase,MPP,global≈Mbase.W_P,global=W_base, M_P,global≈ M_base. For global depth, a microbatch still traverses all PpP_p stages and hence all L layers. Therefore, DPP,global=Dbase.D_P,global=D_base. Realized runtime additionally depends on the microbatch schedule, pipeline fill/drain bubbles, and whether one counts the latency of a single microbatch or steady-state throughput. Those hardware-schedule effects are outside the present work–depth abstraction. B-C4 Tensor Parallelism Tensor parallelism of degree PtP_t partitions hidden-dimensional operators within each layer. In the simplified work–depth model, this shards the dominant GEMM work, model-state memory, and activation memory by PtP_t. Thus, WTP,rank=6B(S+T)L(4d2+2ddff+(S+T)dPt),W_TP,rank=6B(S+T)L ( 4d^2+2d_f+(S+T)dP_t ), and MTP,rank=L(16d2+8ddff)Pt+4VdPt+B(S+T)LdPt.M_TP,rank= L(16d^2+8d_f)P_t+ 4VdP_t+ B(S+T)LdP_t. The hidden-dimensional reductions are correspondingly shortened on each tensor shard, while the parameter-gradient accumulation still spans all B(S+T)B(S+T) batch-token positions. Under the explicit model used in the tables, DTP=2Llog(d4dff(S+T)Pt2h)+log(B(S+T)).D_TP=2L \! ( d^4d_f(S+T)P_t^2h )+ (B(S+T)). Asymptotically, DTP=O(L[log(d/Pt)+log(S+T)]+log(B(S+T))).D_TP=O\! (L[ (d/P_t)+ (S+T)]+ (B(S+T)) ). Globally, TP preserves total work and total memory order: WTP,global=Wbase,MTP,global≈Mbase,W_TP,global=W_base, M_TP,global≈ M_base, ignoring communication buffers and collective overheads. Thus, TP primarily reduces per-device pressure and hidden-dimensional critical-path reductions, at the cost of layer-wise communication not modeled in these tables. B-C5 Context Parallelism Context parallelism of degree PcP_c partitions the sequence dimension. Each rank owns a local query/context shard of length (S+T)/Pc(S+T)/P_c. Each local query, however, still depends on keys and values from the global context of length S+TS+T. Thus, CP partitions sequence-local work and activations without replacing the global attention-reduction length S+TS+T by (S+T)/Pc(S+T)/P_c. The per-rank work is WCP,rank=6BS+TPcL(4d2+2ddff+(S+T)d).W_CP,rank=6B S+TP_cL (4d^2+2d_f+(S+T)d ). The per-rank memory is MCP,rank=L(16d2+8ddff)+4Vd+BS+TPcLd.M_CP,rank=L(16d^2+8d_f)+4Vd+B S+TP_cLd. Under the computation-only depth abstraction, the attention part retains the global S+TS+T reduction dependency. The local parameter-gradient reduction is over B(S+T)/PcB(S+T)/P_c token instances, giving DCP,rank=2Llog(d4dff(S+T)h)+log(B(S+T)Pc).D_CP,rank=2L \! ( d^4d_f(S+T)h )+ \! ( B(S+T)P_c ). Globally, the partial parameter gradients span all sequence partitions, so DCP,global=Dbase.D_CP,global=D_base. Distributed attention additionally introduces communication-round dependencies, which are outside the present FLOP-based depth abstraction. Globally, model parameters are replicated across context partitions, while the activation term sums back to the original order: WCP,global=Wbase,W_CP,global=W_base, MCP,global=Pc[L(16d2+8ddff)+4Vd]+B(S+T)Ld.M_CP,global=P_c\! [L(16d^2+8d_f)+4Vd ]+B(S+T)Ld. Thus, CP is primarily a sequence-work, memory, and long-context feasibility technique; it does not reduce the global exact-attention reduction from S+TS+T to (S+T)/Pc(S+T)/P_c. B-C6 Expert Parallelism Expert parallelism applies to MoE layers. Suppose the model has E experts, EaE_a active experts per token, expert hidden dimension ded_e, and expert-parallel degree PeP_e. For the isolated EP results in Tables V–VIII, we assume that EP is the only partitioning mechanism: the non-expert Transformer computation is replicated across the PeP_e ranks, while expert FFN parameters and routed expert computation are partitioned across them. Thus, no additional DP, TP, P, or CP partitioning is assumed outside the expert FFN. The per-rank work is therefore WEP,rank=6B(S+T)L(4d2+2EaddePe+(S+T)d).W_EP,rank=6B(S+T)L (4d^2+ 2E_ad_eP_e+(S+T)d ). Because the dense/shared computation is replicated, while expert computation is partitioned, global executed work is WEP,global=6B(S+T)L(Pe[4d2+(S+T)d]+2Eadde).W_EP,global=6B(S+T)L (P_e[4d^2+(S+T)d]+2E_ad_e ). Since experts are selected inside the same layer position, EP does not remove the dense Transformer critical path. The dense parameter-gradient reduction also spans all B(S+T)B(S+T) token instances. Thus, DEP=2Llog(d4de(S+T)h)+log(B(S+T)),D_EP=2L \! ( d^4d_e(S+T)h )+ (B(S+T)), or asymptotically, DEP=O(L[logd+log(S+T)]+log(B(S+T))).D_EP=O\! (L[ d+ (S+T)]+ (B(S+T)) ). The per-rank memory is MEP,rank=L(16d2+8EddePe)+4Vd+B(S+T)Ld.M_EP,rank=L (16d^2+ 8Edd_eP_e )+4Vd+B(S+T)Ld. Globally, the dense model state and dense activations are replicated across the PeP_e ranks, while expert parameters sum across expert partitions: MEP,global=Pe(16Ld2+4Vd+B(S+T)Ld)+8LEdde,M_EP,global=P_e (16Ld^2+4Vd+B(S+T)Ld )+8LEdd_e, up to the same optimizer-state convention used in the baseline memory model. B-C7 3D Parallelism 3D parallelism combines data, pipeline, and tensor parallelism with degrees (Pd,Pp,Pt)(P_d,P_p,P_t). It applies the substitutions B↦BPd,L↦LPp,d-sharded GEMM terms↦1PtB BP_d, L LP_p, d-sharded GEMM terms 1P_t to the corresponding baseline terms. Therefore, the per-rank work is W3D,rank=6BPd(S+T)LPp(4d2+2ddff+(S+T)dPt).W_3D,rank=6 BP_d(S+T) LP_p ( 4d^2+2d_f+(S+T)dP_t ). The per-rank depth combines pipeline and tensor reductions, while the local parameter-gradient reduction spans the BPd(S+T) BP_d(S+T) batch-token positions assigned to the data-parallel replica: D3D,rank=2LPplog(d4dff(S+T)Pt2h)+log(BPd(S+T)),D_3D,rank=2 LP_p \! ( d^4d_f(S+T)P_t^2h )+ \! ( BP_d(S+T) ), or asymptotically, D3D,rank D_3D,rank =O(LPp[log(d/Pt)+log(S+T)]CLOSE =O\! ( LP_p[ (d/P_t)+ (S+T)] OPEN+log(BPd(S+T))). + \! ( BP_d(S+T) ) ). The memory expression combines layer partitioning, tensor partitioning, and batch partitioning: M3D,rank=L(16d2+8ddff)PpPt+4VdPt+B(S+T)LdPdPpPt.M_3D,rank= L(16d^2+8d_f)P_pP_t+ 4VdP_t+ B(S+T)LdP_dP_pP_t. Globally, the dominant work remains the baseline work, while model-state memory is replicated across data-parallel groups: W3D,global=Wbase,W_3D,global=W_base, M3D,global=Pd[L(16d2+8ddff)+4Vd]+B(S+T)Ld.M_3D,global=P_d [L(16d^2+8d_f)+4Vd ]+B(S+T)Ld. The global depth restores the full L-layer pipeline path and the full batch-gradient dependency, while retaining the tensor-parallel hidden reduction: D3D,global=2Llog(d4dff(S+T)Pt2h)+log(B(S+T)),D_3D,global=2L \! ( d^4d_f(S+T)P_t^2h )+ (B(S+T)), or D3D,global D_3D,global =O(L[log(d/Pt)+log(S+T)]CLOSE =O\! (L[ (d/P_t)+ (S+T)] OPEN+log(B(S+T))). + (B(S+T)) ). This explains the main role of 3D parallelism in the tables: it gives strong per-device memory reduction because it simultaneously partitions batch, layers, and hidden-dimensional computation. Its realized runtime, however, depends on communication, pipeline scheduling, and microbatching, which are deliberately outside the work–depth–memory abstraction used here. B-C8 5D Parallelism We finally consider a combined configuration using data, pipeline, tensor, context, and expert parallelism with degrees (Pd,Pp,Pt,Pc,Pe)(P_d,P_p,P_t,P_c,P_e), where PdPpPtPcPe=N.P_dP_pP_tP_cP_e=N. An important distinction from the isolated EP analysis above is required. The isolated EP row is intentionally a conceptual abstraction designed to expose the effect of expert partitioning alone: only expert FFNs are partitioned across the PeP_e ranks, while the same shared dense Transformer computation is executed on the same token set by every EP rank. Consequently, its global work contains PeP_e copies of the shared dense arithmetic. This should not be interpreted as the execution strategy necessarily used in a practical multidimensional MoE system. For the 5D configuration, we instead model a practically motivated coupled execution in which the ranks participating in the EP dimension also own disjoint source-token shards for the shared Transformer path. Expert parallelism itself determines where expert parameters reside; the additional source-token partition specifies where tokens reside before expert dispatch. These are distinct notions. Formally, if U denotes the B(S+T)B(S+T) batch-token positions of the global training invocation, then we assume =⨆rd=1Pd⨆rc=1Pc⨆re=1Perd,rc,re,U= _r_d=1^P_d _r_c=1^P_c _r_e=1^P_eU_r_d,r_c,r_e, with approximately balanced partitions |rd,rc,re|≈B(S+T)PdPcPe.|U_r_d,r_c,r_e|≈ B(S+T)P_dP_cP_e. Thus, a shared dense parameter may be replicated across the EP dimension, but each source token is processed by the shared path on only one EP rank. After routing, token representations are redistributed by the expert-parallel all-to-all to the ranks hosting the selected experts, and expert outputs are subsequently returned to the corresponding source-token ranks. We assume balanced routing, so the resulting token–expert assignments are approximately uniformly distributed across the PeP_e expert ranks. Tensor parallelism is applied to both shared dense and expert GEMMs. Under these assumptions, the per-rank work is W5D,rank=6BPdPeS+TPcLPp(4d2+2Eadde+(S+T)dPt).W_5D,rank=6 BP_dP_e S+TP_c LP_p ( 4d^2+2E_ad_e+(S+T)dP_t ). The factors have distinct origins: PdP_d partitions the batch, PcP_c partitions source sequence positions, PeP_e further partitions source-token ownership across the expert-parallel ranks, PpP_p partitions layers, and PtP_t partitions intra-layer matrix operations. The 1/Pe1/P_e factor is therefore applied to the source-token workload rather than separately to the expert term; under balanced routing, each EP rank already receives approximately 1/Pe1/P_e of the global token–expert assignments. Summing executed arithmetic across all N=PdPpPtPcPeN=P_dP_pP_tP_cP_e ranks gives W5D,global=6B(S+T)L(4d2+2Eadde+(S+T)d).W_5D,global=6B(S+T)L (4d^2+2E_ad_e+(S+T)d ). In particular, the shared dense term is not multiplied by PeP_e, unlike in the isolated EP abstraction. The reason is that the execution mapping assigns disjoint source-token subsets to the EP ranks, so the same dense token computation is not redundantly executed on every EP rank. This illustrates the distinction between parameter replication and arithmetic replication: shared dense parameters may remain replicated across the EP dimension even though the corresponding FLOPs are evaluated on disjoint token subsets. The per-rank depth is D5D,rank=2LPplog(d4de(S+T)Pt2h)+log(B(S+T)PdPePc).D_5D,rank=2 LP_p \! ( d^4d_e(S+T)P_t^2h )+ \! ( B(S+T)P_dP_eP_c ). The first term combines pipeline and tensor partitioning of the layerwise critical path while retaining the global attention span S+TS+T. The second term is the local parameter-gradient reduction over the source batch-token positions assigned to the rank. As in the other tables, communication, including the expert dispatch/combine all-to-all and distributed-attention collectives, is outside the arithmetic-depth abstraction. At the global level, the complete invocation traverses all L pipeline layers and its parameter gradients depend on all B(S+T)B(S+T) batch-token positions. Hence, D5D,global=2Llog(d4de(S+T)Pt2h)+log(B(S+T)).D_5D,global=2L \! ( d^4d_e(S+T)P_t^2h )+ (B(S+T)). The corresponding per-rank memory is M5D,rank=16Ld2PpPt+8LEddePpPtPe+4VdPt+B(S+T)LdPdPePcPpPt.M_5D,rank= 16Ld^2P_pP_t+ 8LEdd_eP_pP_tP_e+ 4VdP_t+ B(S+T)LdP_dP_eP_cP_pP_t. Shared dense model state is partitioned by P and TP but remains replicated across the DP, CP, and EP dimensions. Expert model state is additionally partitioned across PeP_e, while the leading-order activation term follows the combined source-token, layer, and tensor partitioning assumed above. Consequently, aggregate global memory is M5D,global= M_5D,global= PdPcPe(16Ld2+4Vd) P_dP_cP_e (16Ld^2+4Vd ) +8PdPcLEdde+B(S+T)Ld, +8P_dP_cLEdd_e+B(S+T)Ld, up to the same boundary-stage and embedding-placement conventions used for P and TP. Isolated EP provides a clean limiting case in which only expert FFNs are partitioned, making the cost of replicated shared computation explicit. The 5D configuration instead captures a more practically relevant multidimensional MoE execution in which expert placement is combined with source-token distribution and other intra-model parallelism dimensions. As a result, shared dense arithmetic is partitioned rather than replicated across the EP dimension, whereas replicated shared state can still contribute a factor PeP_e to global memory. B-D Results for Inter-Model Parallelism We provide derivations for results in Section IV-F and in Table IX. The analysis uses the building blocks from Table I: forward costs Cf(⋅)C_f(·), backward costs Cb(⋅)C_b(·), autoregressive generation cost Cgen(⋅)C_gen(·), forward/backward depths Df(⋅)D_f(·), Db(⋅)D_b(·), and generation depth Dgen(⋅)D_gen(·). Communication, kernel efficiency, and scheduling overheads are outside the work–depth abstraction; they are represented only indirectly through memory buffers such as MreshardM_reshard and shadow copies MshM_sh. B-D1 Stage Primitives For a PPO-style iteration with B prompts, K rollouts per prompt, prompt length S, and response length T, Generation invokes the actor autoregressively. Using the prefill–decode decomposition from Table I, WG W_G =BKCgen(πθ), =BK\,C_gen( _θ), DG D_G =Dgen(πθ), =D_gen( _θ), MG M_G =|πθ|+BKMKV. =| _θ|+BK\,M_KV. This assumes the rollout-time actor log-probabilities are stored during generation. If an implementation recomputes old actor log-probabilities in a teacher-forced pass, an additional BK(S+T)Cf(πθ)BK(S+T)C_f( _θ) work term and Df(πθ)D_f( _θ) depth term should be added to Assessment or Training, depending on where the recomputation is performed. Assessment consists of teacher-forced forward passes through πref _ref, RφR_ , and VψV_ψ: WA=BK(S+T)[Cf(πref)+Cf(Rφ)+Cf(Vψ)].W_A=BK(S+T)[C_f( _ref)+C_f(R_ )+C_f(V_ψ)]. If these models are co-located and executed sequentially, the depth is DAΣ=Df(πref)+Df(Rφ)+Df(Vψ).D_A =D_f( _ref)+D_f(R_ )+D_f(V_ψ). If they are placed on disjoint device groups and executed concurrently, the depth becomes DAmax=maxDf(πref),Df(Rφ),Df(Vψ).D_A = \D_f( _ref),D_f(R_ ),D_f(V_ψ)\. The corresponding co-located memory footprint is MAΣ=|πref|+|Rφ|+|Vψ|+BK(S+T)MInf.M_A =| _ref|+|R_ |+|V_ψ|+BK(S+T)M_Inf. Training updates the actor and critic. For a trainable model M, we define Ctr(M) C_tr(M) :=Cf(M)+Cb(M)≈3Cf(M), :=C_f(M)+C_b(M)≈ 3C_f(M), Dtr(M) D_tr(M) :=Df(M)+Db(M), :=D_f(M)+D_b(M), since a parameter update requires a forward pass followed by backpropagation. Thus, WT=BK(S+T)[Ctr(πθ)+Ctr(Vψ)].W_T=BK(S+T)[C_tr( _θ)+C_tr(V_ψ)]. Sequential co-located training has depth DTΣ=Dtr(πθ)+Dtr(Vψ),D_T =D_tr( _θ)+D_tr(V_ψ), whereas disaggregated actor/critic training has depth DTmax=maxDtr(πθ),Dtr(Vψ).D_T = \D_tr( _θ),D_tr(V_ψ)\. Under the Adam memory model without activation checkpointing, the co-located training footprint is MTΣ=4(|πθ|+|Vψ|)+BK(S+T)(Lπdπ+LVdV).M_T =4(| _θ|+|V_ψ|)+BK(S+T)(L_πd_π+L_Vd_V). For disaggregated training, the peak per device group is MTmax M_T =max4|πθ|+BK(S+T)Lπdπ, = \4| _θ|+BK(S+T)L_πd_π,\, 4|Vψ|+BK(S+T)LVdV. 4|V_ψ|+BK(S+T)L_Vd_V\. B-D2 Baseline Co-Located Execution The baseline uses separate actor and critic models and a fully co-located device group. All stages execute sequentially. Therefore, Wbase=WG+WA+WT,W_base=W_G+W_A+W_T, Dbase=DG+DAΣ+DTΣ,D_base=D_G+D_A +D_T , and Mbase=maxMG,MAΣ,MTΣ.M_base= \M_G,M_A ,M_T \. The maximum appears because the stages are temporally multiplexed on the same device group: the peak is the largest stage footprint, not the sum over all stages. If an implementation keeps all models resident simultaneously, then the persistent part of the memory expression should instead sum the resident model states. B-D3 Shared Actor–Critic With a shared actor–critic model πac _ac, the actor and value heads share a Transformer backbone. Generation uses πac _ac as the policy: WGac=BKCgen(πac),DGac=Dgen(πac).W_G^ac=BK\,C_gen( _ac), D_G^ac=D_gen( _ac). Assessment no longer requires an independent critic model; instead, value estimates are produced by the shared model: WAac=BK(S+T)[Cf(πref)+Cf(Rφ)+Cf(πac)].W_A^ac=BK(S+T)[C_f( _ref)+C_f(R_ )+C_f( _ac)]. Training performs one forward pass followed by backpropagation through the shared backbone: WTac=BK(S+T)Ctr(πac).W_T^ac=BK(S+T)C_tr( _ac). Thus, shared actor–critic is the only inter-model configuration in Table IX that reduces global work relative to the separate-backbone PPO baseline. Its training memory becomes MTac=4|πac|+BK(S+T)Lacdac,M_T^ac=4| _ac|+BK(S+T)L_acd_ac, up to the small policy/value head activations. The benefit is largest when |Vψ||V_ψ| is comparable to |πθ|| _θ|; it is smaller when the critic is already much smaller than the actor. B-D4 Disaggregated Placement Disaggregated placement assigns actor, reference, reward, and critic operators to separate device groups. It does not change total work: Wdisagg=WG+WA+WT.W_disagg=W_G+W_A+W_T. It changes depth by replacing independent sequential subcomputations with parallel fork–join regions: Ddisagg=DG+DAmax+DTmax.D_disagg=D_G+D_A +D_T . It also changes memory from co-resident memory to per-group memory: Mdisagg=maxMG,Mπref,MRφ,MVψ,MT,π,MT,V.M_disagg= \M_G,M_ _ref,M_R_ ,M_V_ψ,M_T,π,M_T,V\. This captures the main memory advantage of disaggregation: reward-model devices need not hold actor or critic state, and actor-training devices need not hold frozen reward/reference models. Disaggregation alone does not remove the Generation → Assessment → Training dependency; it only enables concurrency where the dataflow graph has independent branches. B-D5 Hybrid Execution and Resharding Hybrid execution allows the same model to use different intra-model layouts in different stages. Let DGinfD_G^inf and MGinfM_G^inf denote Generation depth and memory under an inference-oriented layout, and let DTtrainD_T^train and MTtrainM_T^train denote Training depth and memory under a training-oriented layout such as ZeRO/FSDP or 3D parallelism. The FLOP work is unchanged: Whyb=WG+WA+WT,W_hyb=W_G+W_A+W_T, but the stage depths become layout-specific: Dhyb=DGinf+DAinf,Σ+DTtrain,D_hyb=D_G^inf+D_A^inf, +D_T^train, assuming the same device group is temporally multiplexed and therefore provides no inter-model concurrency by itself. The cost of hybrid execution is resharding. If two invocations of the same model use incompatible layouts, weights or optimizer state must be redistributed. This does not change FLOP work in the work–depth table, but it affects realized runtime and may require an additional transient buffer: Mhyb=maxMGinf,MAinf,MTtrain+Mreshard.M_hyb= \M_G^inf,M_A^inf,M_T^train\+M_reshard. Hybrid execution is useful only when the per-stage gains from specialized layouts exceed the resharding and orchestration costs. B-D6 Stage Fusion Stage fusion preserves total work: Wfusion=WG+WA+WT.W_fusion=W_G+W_A+W_T. Its effect is on depth. Inter-stage fusion streams completed generations into Assessment, so the fused Generation–Assessment depth is DG/Afused=maxDG,DAmaxD_G/A^fused= \D_G,D_A \ under ideal overlap. Training remains downstream, so Dfusion=maxDG,DAmax+DTmax.D_fusion= \D_G,D_A \+D_T . When generation dominates assessment, this simplifies to Dfusion≈DG+DTmax.D_fusion≈ D_G+D_T . Intra-stage fusion similarly replaces sequential actor/critic training pipelines with a max term when the two pipelines can be overlapped: DTΣ→DTmax.D_T → D_T . The memory footprint may increase because multiple stage fragments are live simultaneously: Mfusion=maxMG/Afused,MTmax.M_fusion= \M_G/A^fused,M_T \. Here MG/AfusedM_G/A^fused includes the live generation KV cache, the assessment model states, and any queues or communication buffers needed to stream completed samples. B-D7 Asynchronous Execution Bounded asynchrony overlaps generation/assessment of one iteration with training of another. Total work is unchanged: Wasync=WG+WA+WT.W_async=W_G+W_A+W_T. The steady-state recurrence depth becomes Dasync=maxDG+DAmax,DTmax,D_async= \D_G+D_A ,D_T \, assuming disaggregated generation/assessment and training groups. If assessment is also fused with generation, then the first term can be replaced by maxDG,DAmax \D_G,D_A \. Asynchrony requires parameter snapshots. If a trainable model is read by a stale consumer on a disjoint device group, the consumer needs a stable inference-side copy while the training side updates another copy. We denote this additional persistent copy by Msh(m).M_sh(m). For actor asynchrony, Msh(πθ)M_sh( _θ) is typically a BF16, possibly quantized, inference copy of the actor. Thus, Masync=maxMG/Aasync+Msh(πθ),MTmax.M_async= \M_G/A^async+M_sh( _θ),M_T \. This exposes the memory trade-off: asynchrony reduces depth by overlapping iterations, but it may require roughly one extra parameter copy for each asynchronously consumed trainable model. This is not necessarily a full 2×2× increase in total memory, because optimizer states remain on the training side and the shadow copy may be lower precision or sharded, but it must be accounted for. B-D8 Combined Configuration The combined configuration composes the strongest ingredients: stage-specific intra-model sharding, disaggregated model placement, stage fusion, and bounded asynchrony. The total FLOP work is still Wcomb=WG+WA+WT,W_comb=W_G+W_A+W_T, unless the model structure is also changed, for example by using a shared actor–critic backbone. With ideal fusion and bounded asynchrony, the recurrence depth is Dcomb=maxDGTP,DTshard,D_comb= \D_G^TP,D_T^shard\, where DGTPD_G^TP denotes actor generation depth under an inference-oriented sharded layout, and DTshardD_T^shard denotes the depth of the sharded training layout. A more conservative expression keeps the unhidden assessment tail: Dcomb=maxmax(DGTP,DAmax),DTshard.D_comb= \ (D_G^TP,D_A ),D_T^shard\. The memory expression is Mcomb=maxMGTP+Msh(πθ),Mπref,MRφ,MVψ,MTshard.M_comb= \M_G^TP+M_sh( _θ),M_ _ref,M_R_ ,M_V_ψ,M_T^shard\. Thus, the combined strategy gives the smallest idealized depth and peak per-device-group memory in Table IX, but only under sufficient hardware, careful scheduling, and acceptable staleness.