Paper deep dive
Buffer Matters: Unleashing the Power of Off-Policy Reinforcement Learning in Large Language Model Reasoning
Xu Wan, Yansheng Wang, Wenqi Huang, Mingyang Sun
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 7/20/2026, 2:08:34 PM
Summary
The paper introduces Batch Adaptation Policy Optimization (BAPO), an off-policy Reinforcement Learning with Verifiable Rewards (RLVR) framework designed to improve data efficiency in Large Language Model (LLM) post-training. BAPO addresses the experience waste and reward homogeneity issues found in on-policy methods like GRPO by dynamically selecting training batches. It utilizes a difficulty-aware experience replay mechanism that re-evaluates historically difficult samples and reuses high-quality historical trajectories, ensuring a lower bound guarantee for policy improvement. Experiments show BAPO achieves a 12.5% average improvement over GRPO across mathematics, planning, and visual reasoning tasks, resolving 40.7% of problems that base models consistently fail to solve.
Entities (20)
Relation Signals (16)
BAPO → improves → GRPO
confidence 95% · BAPO achieves an average 12.5% improvement over GRPO across mathematics, planning, and visual reasoning tasks.
BAPO → isa → RLVR
confidence 95% · we introduce Batch Adaptation Policy Optimization (BAPO), an off-policy RLVR framework
BAPO → uses → experience replay
confidence 92% · We propose a difficulty-aware experience replay mechanism as a practical solution for efficient off-policy data utilization.
Mingyang Sun → affiliatedwith → Peking University
confidence 90% · Mingyang Sun ... ♡ Peking University
Xu Wan → affiliatedwith → Zhejiang University
confidence 90% · Xu Wan ... ♠ Zhejiang University
Yansheng Wang → affiliatedwith → Bytedance Seed Robotics
confidence 90% · Yansheng Wang ... ♣ Bytedance Seed Robotics
Wenqi Huang → affiliatedwith → China Southern Power Grid
confidence 90% · Wenqi Huang ... ♢ China Southern Power Grid
BAPO → resolves → reward homogeneity
confidence 90% · Traditional on-policy ... RLVR frameworks suffer from experience waste and reward homogeneity ... BAPO ... improve the data efficiency
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Traditional on-policy Reinforcement Learning with Verifiable Rewards (RLVR) frameworks suffer from experience waste and reward homogeneity, which directly hinders learning efficiency on difficult samples during large language models post-training. In this paper, we introduce Batch Adaptation Policy Optimization (BAPO), an off-policy RLVR framework to improve the data efficiency in large language models post-training. It dynamically selects training batches by re-evaluating historically difficult samples and reusing high-quality ones, while holding a lower bound guarantee for policy improvement. Extensive experiments further demonstrate that BAPO achieves an average 12.5% improvement over GRPO across mathematics, planning, and visual reasoning tasks. Crucially, BAPO successfully resolves 40.7% of problems that base models consistently fail to solve.
Tags
Links
- Source: https://arxiv.org/abs/2602.20722v2
- Canonical: https://arxiv.org/abs/2602.20722v2
Trouble viewing inline? Open PDF directly →
Full Text
74,872 characters extracted from source content.
Expand or collapse full text
Buffer Matters: Unleashing the Power of Off-Policy Reinforcement Learning in Large Language Model Reasoning Xu Wan♠♣♡ \ \ , Yansheng Wang♣, Wenqi Huang♢, Mingyang Sun∗♡ ♠ Zhejiang University ♣ Bytedance Seed Robotics ♢ China Southern Power Grid ♡ Peking University Coresponding Author Abstract Traditional on-policy Reinforcement Learning with Verifiable Rewards (RLVR) frameworks suffer from experience waste and reward homogeneity, which directly hinders learning efficiency on difficult samples during large language models post-training. In this paper, we introduce Batch Adaptation Policy Optimization (BAPO), an off-policy RLVR framework to improve the data efficiency in large language models post-training. It dynamically selects training batches by re-evaluating historically difficult samples and reusing high-quality ones, while holding a lower bound guarantee for policy improvement. Extensive experiments further demonstrate that BAPO achieves an average 12.5% improvement over GRPO across mathematics, planning, and visual reasoning tasks. Crucially, BAPO successfully resolves 40.7% of problems that base models consistently fail to solve. The code is available in Here. 1 Introduction Reinforcement Learning from Human Feedback (RLHF) has emerged as a transformative paradigm for aligning Large Language Models (LLMs) with human preferences and improving their performance on complex reasoning tasks (Ouyang et al., 2022; Bai et al., 2022). A significant recent evolution is Reinforcement Learning with Verifiable Rewards (RLVR) (Lambert et al., 2024), which replaces costly neural reward models with deterministic verification functions for more efficient and reliable training (Guo et al., 2025). Numerous on-policy RL optimization methods, particularly Group Relative Policy Optimization (GRPO) (Shao et al., 2024), and its variants like Dynamic Sampling Policy Optimization (DAPO) (Yu et al., 2025), Group Sequence Policy Optimization (GSPO) (Zheng et al., 2025), have demonstrated remarkable success in LLM post-training scenarios, achieving exceptional performance on mathematical reasoning, code generation, and various downstream applications (Yang et al., 2025; Chen et al., 2025a; Shen et al., 2025). Figure 1: Tracking the sample counts across accuracy groups of the mathematical dataset before and after GRPO post-training. Although with lower bound guarantees of policy improvement theoretically (Mroueh, 2025), existing RL post-training frameworks still face significant efficiency challenges in practice. As shown in Figure 1, models after GRPO post-training struggle to handle difficult samples, especially those with zero accuracy in the initial rollout group. The reasons are twofold: (1) Homogeneous rewards: Recent investigations (Hong et al., 2025; Simoni et al., 2025) reveal that samples at both extremes of difficulty offer minimal benefit for post-training policy improvement. This arises because advantage estimation in most GRPO-based methods relies heavily on relative reward diversity within each group. Consequently, when intra-group rewards are identical, the lower bound guarantee for policy improvement collapses (Zhang et al., 2025; Mroueh et al., 2025), resulting in negligible effective gradient contributions (Liu et al., 2025; Yu et al., 2025). (2) Waste of experience: Given the sensitivity of policy improvement to intra-group reward variance, uneven difficulty distributions yield significantly fewer high-quality samples than the configured batch size implies. Crucially, since these methods are primarily on-policy and lack experience replay, each rollout group is consumed only once, leading to a substantial waste of valuable training data (Sun et al., 2025; Li et al., 2025). A straightforward solution is to adopt off-policy rather than on-policy training paradigms, which has been established in traditional RL tasks as a viable solution to increase sample efficiency and diversity in the training batch (Queeney et al., 2021; Hilton et al., 2022; Meng et al., 2023). However, naively applying sample-reusing schemes to RL frameworks may exacerbate instability during LLM post-training, leading to entropy collapse, and ultimately performance degradation (Yu et al., 2025; He et al., 2025; Chen et al., 2025c). Thus, to systematically exploring the utility of stale off-policy experience in RLVR post-training, we incorporates multiple off-policy strategies into on-policy RLVR framework to dissect effective pathways for historical data utilization. The main contributions of this paper are as follows: (1) We propose a difficulty-aware experience replay mechanism as a practical solution for efficient off-policy data utilization. Unlike the simple mixing of the buffer’s data and online data, we actively re-evaluate historical hard prompts to drive exploration while directly reusing high-quality trajectories with a dynamic quality threshold. (2) Theoretically, we prove that under certain assumptions, the proposed adaptive construction mechanism mitigates the homogeneous reward issue via adaptive batch construction and KL-constrained updates. (3) By integrating it into multiple reasoning tasks with different LLM backbones, we validate the proposed Batch Adaptation Policy Optimization (BAPO) method achieves better convergence and yields greater improvements on solving difficult samples compared to existing on-policy and off-policy RLVR frameworks. 2 Related Work 2.1 On-policy RL Post-training Framework We first review the concept of on-policy RLVR, where the core objective is to optimize an LLM policy to maximize the outcome response reward. Let x∈x represent the input prompts, and y∈y denote responses generated by the LLM policy πθ _θ. The terminal reward r(x,y)∈0,1r(x,y)∈\0,1\ is determined by a deterministic verification function (Lambert et al., 2024; Guo et al., 2025). Following the setting of GRPO (Shao et al., 2024), the objective is formulated as: 1G∑i=1G1|yi|∑t=1|yi|min(ρi,t(θ)A^i,t,clip(ρi,t(θ),1−ε,1+ε)A^i,t)−β⋅KL(πθ||πref) 1G _i=1^G 1|y_i| _t=1^|y_i| ( _i,t(θ) A_i,t,clip( _i,t(θ),1- ,1+ ) A_i,t )-β·D_KL( _θ|| _ref) (1) where =y1,y2,…,yGG=\y_1,y_2,…,y_G\ represents a G-size group of responses sampled from πθt(⋅|x) _ _t(·|x) for each input x; ρi,t(θ) _i,t(θ) is the probability ratio πθ(yit∣yi<t,x)πθold(yit∣yi<t,x) _θ (y_i^t y_i^<t,x ) _ _old (y_i^t y_i^<t,x ) between current policy and old policy πθold _ _old for the i-th responses’ t-th token, ε limits the magnitude of policy updates; and KLD_KL constrains the policy πθ _θ from deviating too far from a reference policy πref _ref. Crucially, A^i,t A_i,t denotes the estimated advantage of response yiy_i for input x, which is derived from the standardization of rewards using the statistical properties of group G. For the i-th response yi∈y_i with reward ri=r(x,yi)r_i=r(x,y_i), the estimated advantage is: A^i,t=ri−mean(rℓ)std2(rℓ)+ε A_i,t= r_i-mean(\r_ \) std^2(\r_ \)+ (2) where mean(rℓ)mean(\r_ \) and std2(rℓ)std^2(\r_ \) are the empirical mean and variance of rewards in group G, respectively. To enhance the practical efficiency of GRPO, a series of improved on-policy frameworks has been proposed. For instance, DAPO (Yu et al., 2025) sets distinct clipping ranges εlow _low and εhigh _high, and employs a dynamic sampling strategy to ensure A^i,t≠0 A_i,t≠ 0. However, it consumes approximately four times the number of rollouts (Qu et al., 2025) compared to GRPO. Meanwhile, GSPO (Zheng et al., 2025) abandons the token-level ratio ρi,t(θ) _i,t(θ) and shifts to the sequence level si(θ)s_i(θ), which has been validated to maintain more stable training, particularly in Mixture-of-Experts (MoE) architectures. While the details of these methods vary, they all adhere to the on-policy framework for sampling and updates: the inference server is updated in synchronization with the trainer parameters, and the sampling strategy follows the “ use-once-and-discard” principle throughout the training process. 2.2 Off-policy RL Post-training Framework In contrast, as shown in Figure 2, off-policy RL post-training frameworks operate under a distinct paradigm, characterized by two core components: off-policy rollout for generating responses and off-policy training for constructing the training batch, as detailed below. Figure 2: The overview of the (a) on-policy and (b) off-policy RL Post-training framework Off-policy Rollout avoids exclusive reliance on the current training policy for generation, instead leveraging past policies or external guidance. For example, AReaL (Fu et al., 2025) employs a fully asynchronous architecture that decouples generation from training, allowing rollout workers to use past policy. Mroueh et al. (Mroueh et al., 2025) fix the rollout policy on the vLLM inference server for multiple iterations to ensure stable sample generation. LUFFY (Yan et al., 2025) incorporates traces from stronger external policies to enhance reasoning capabilities beyond the model’s initial limits. Off-policy Training uses replay buffers to manage samples from historical policies with varying activation strategies. ARPO (Lu et al., 2025) dynamically samples non-zero reward samples from the buffer only when current batches contain all-zero rewards. DOTS (Sun et al., 2025) maintains a FIFO buffer that consistently reuses recent valid rollouts. RePO (Li et al., 2025) mixes buffer samples with on-policy samples using diverse retrieval strategies. ReMix (Liang et al., 2025) blends samples at fixed ratios while increasing the update-to-data ratio for efficiency. ReLIFT (Ma et al., 2025) stores high-quality solutions to challenging problems in its buffer and refines them through interleaved supervised fine-tuning. Kimi K1.5 (Team et al., 2025) stores both complete and partial trajectories to reduce temporal correlations while maintaining computational efficiency. However, most off-policy RLVR methods ignore the policy stability of experiences. Samples entering the buffer at different training steps may exhibit varying policy distributions. These discrepancies introduce excessive noise into policy learning, which in turn exacerbates training instability. More importantly, simply reusing historical samples may even hinder the policy’s improvement. The high-accuracy historical samples may cause the model to overly focus on existing reasoning paths with high advantages, suppressing the model’s exploration capability and resulting in premature convergence to suboptimal solutions (Cui et al., 2025). 3 Method In this section, we detail the core components of BAPO, particularly the adaptive construction strategy for the training batch, and provide a theoretical guarantee for the training stability of BAPO’s policy update. Figure 3 provides an overview of the off-policy rollout and training workflow. Figure 3: The workflow of (a) off-policy rollout and (b) off-policy training in our RLVR framework 3.1 Formal Definitions We first formalize our training objective ℒα(πθ)L_α( _θ) as a combination of online rollout-derived and historical buffer-derived contributions: ℒα(πθ)=(x,y)∼α[ρα(θ)⋅A^(x,y)]⏟Contribution from fresh samples+(x,y)∼ℬ[ραℬ(θ)⋅A^(x,y)]⏟Contribution from historical samples−β⋅KL(πθ∥α)L_α( _θ)= E_(x,y) [rgb]0,0,0 [named]pgfstrokecolorrgb0,0,0 @color@gray@stroke0 @color@gray@fill0α [ _ [rgb]0,0,0 [named]pgfstrokecolorrgb0,0,0 @color@gray@stroke0 @color@gray@fill0α(θ)· A(x,y) ]_Contribution from fresh samples+ E_(x,y) [rgb]0,0,0 [named]pgfstrokecolorrgb0,0,0 @color@gray@stroke0 @color@gray@fill0B [ _ [rgb]0,0,0 [named]pgfstrokecolorrgb0,0,0 @color@gray@stroke0 @color@gray@fill0 _B(θ)· A(x,y) ]_Contribution from historical samples-β·D_KL( _θ\| [rgb]0,0,0 [named]pgfstrokecolorrgb0,0,0 @color@gray@stroke0 @color@gray@fill0α) (3) where (x,y)∼α(x,y) α refers to filtered online samples from the rollout policy α=πθt−vα= _ _t-v with v>0v>0 representing the delay timesteps. (x,y)∼ℬ(x,y) denotes historical samples from the replay buffer ℬB. The importance sampling ratios are defined as ρα=πθ(y|x)α(y|x) _α= _θ(y|x)α(y|x) for the online rollout samples and ραℬ=πθ(y|x)αℬ(y|x) _ _B= _θ(y|x) _B(y|x) for buffer samples, αℬ _B is the historical rollout policies that generated the buffer. Each entity in the buffer ℬB is formally defined as: ℬ=(ui,xi,jj=1G,yi,jj=1G,ri,jj=1G,αℬ(yi,j|xi)j=1G)i=1|ℬ|B=\(u_i,\x_i,j\_j=1^G,\y_i,j\_j=1^G,\r_i,j\_j=1^G,\ \ _B(y_i,j|x_i)\_j=1^G)\_i=1^|B| (4) where uiu_i is the unique identifier of each prompt, xi,j\x_i,j\, yi,j\y_i,j\, ri,j\r_i,j\ represent the set of prompts, generated responses, and corresponding rewards, respectively. αℬ(yi,j|xi)j=1G\ _B(y_i,j|x_i)\_j=1^G is the rollout policy’s probability, which is stored for calculating ραℬ(θ) _ _B(θ) when reusing, and |ℬ||B| is the buffer size. 3.2 Adaptive Training Batch Construction The core of off-policy RLVR lies in how to integrate historical experiences with online samples, to maintain non-homogeneous rewards and an appropriate difficulty distribution in each training step. For BAPO, we introduce a filter function I(x)I(x) in Definition 3.1 that decomposes the data selection criteria for each training step’s batch into three parts. Definition 3.1 (Training Batch Filtering Function). Define μπ,r(x)=y∼π(⋅|x)[r(x,y)] _π,r(x)=E_y π(·|x)[r(x,y)] as the expected reward under policy π for input x. The training batch indicator function I:→0,1I:X→\0,1\ is formulated as: I(x)=11G≤μα,r(x)≤G−1G⏟Filtered Fresh+1μαℬ,r(x)≤c1∧μπθt,r(x)>c1⏟Improved Historical Difficult+1c2≤μαℬ,r(x)≤c3⏟Historical High-qualityI(x)= 1_\ 1G≤ _α,r(x)≤ G-1G\_ [rgb]0,0,0 [named]pgfstrokecolorrgb0,0,0 @color@gray@stroke0 @color@gray@fill0Filtered Fresh+ 1_\ _ _B,r(x)≤ c_1 _ _ _t,r(x)>c_1\_Improved Historical Difficult+ 1_\c_2≤ _ _B,r(x)≤ c_3\_Historical High-quality (5) where α denotes the delayed rollout policy and αℬ _B denotes the policy associated with buffer samples. The function selects samples based on three criteria, yielding subsets 1X_1, 2X_2 and 3X_3 respectively. Next, we explain the selection principles for I(x)I(x) and derive three categories of samples, namely 1X_1, 2X_2, and 3X_3, which are obtained from these three conditions, respectively. (1) Filtered Fresh Samples (1X_1). To prevent gradient vanishing and maintain training stability, we filter the online rollout batch to exclude samples with zero variance. Specifically, we retain fresh samples where the group mean reward satisfies μα,r(x)∈[1G,G−1G] _α,r(x)∈[ 1G, G-1G]. While other filtering strategies (e.g., Gaussian sampling or uniform sampling) can be applied, we find that simple truncation sufficient for effective learning. A detailed discussion and comparison of different online filtering functions are provided in Appendix A.3. (2) Improved Historical Difficult Samples (2X_2). Samples exhibiting extremely low group mean rewards, where μα,r(x)∈[0,c1] _α,r(x)∈[0,c_1], present significant challenges to the current policy and typically yield negligible policy improvement. However, as the model evolves, these historically difficult queries may eventually become tractable for a successor policy. To harness this, we periodically re-generate responses using the current policy πθt _ _t every m training steps and construct the subset 2X_2 based on the observable improvement. Let ℬbad⊆ℬB_bad denote the buffer for difficult samples. To manage the computational overhead associated with the re-evaluation process, we limit the buffer capacity |ℬbad||B_bad| to be equal to the training batch size. A First-In-First-Out (FIFO) mechanism is employed to automatically discard outdated samples when the buffer reaches capacity. 2X_2 is formulated as: 2=(x,y′)∣(x,y)∈ℬbad,y′∼πθt(⋅∣x),c1<μπθt,r(x)<1X_2= \(x,y ) (x,y) _bad,y _ _t(· x),c_1< _ _ _t,r(x)<1 \ (6) where y′y represents the new response generated by πθt _ _t, and we specifically select samples that show improvement such that c1<μπθt,r(x)<1c_1< _ _ _t,r(x)<1. (3) Reused Historical High-quality Samples (3X_3). To prevent underfilled batches caused by the scarcity of 1X_1 and 2X_2, we maintain a FIFO auxiliary buffer ℬhigh⊆ℬB_high . To mitigate training instability from stale data, ℬhighB_high is restricted to high-quality trajectories from the three most recent steps. The subset 3X_3 is randomly sampled to fill the remaining capacity: 3=(ℬhigh,min(|ℬhigh|,B−|1|−|2|))X3=S (B_high, (|B_high|,B-|X_1|-|X_2| ) ) (7) where B is the configured training batch size and (⋅,k)S(·,k) denotes the random sampling of k elements. Furthermore, to progressively master increasingly difficult tasks, we employ a linear mapping to shift the historical “high-quality” from easier to harder instances, scaling in accordance with the global average performance rtotr_tot: ci=rtot⋅(cihigh−cilow)+cilow,i∈2,3c_i=r_tot·(c_i^high-c_i^low)+c_i^low, i∈2,3 (8) 3.3 Theoretical Analysis In this section, we further provide theoretical analysis in Theorem 3.2 to establish BAPO’s training stability based on (Mroueh et al., 2025)’s theorem. We show that, under certain assumptions, our constructed adaptive batches can consistently maintain guaranteed bounded policy improvement. Theorem 3.2 (Policy Improvement Lower Bound with Adaptive Training Batch). Assume rewards are bounded: 0≤r≤10≤ r≤ 1. Let πθt _ _t be the current policy, α1=πθt−v _1= _ _t-v be the delayed rollout policy, α2=πθt _2= _ _t be the current policy for re-evaluation, α3=αℬ _3= _B be the buffer policy distribution, and I(x)I(x) be the filtering function partitioning samples into 1X_1, 2X_2, and 3X_3. Suppose c1,c2,c3∈(0,1)c_1,c_2,c_3∈(0,1) with c2<c3c_2<c_3, and the following TV distance constraints hold: TV(πθt(⋅|x),πθt−v(⋅|x)) ( _ _t(·|x), _ _t-v(·|x)) ≤δ1∀x∈1 ≤ _1 ∀ x _1 (9) TV(πθt(⋅|x),αℬ(⋅|x)) ( _ _t(·|x), _B(·|x)) ≤δ3∀x∈3 ≤ _3 ∀ x _3 (10) where δ1,δ3>0 _1, _3>0 are sufficiently small such that the variance lower bounds remain positive. Then, for the policy update objective in Equation 3, the expected policy improvement over filtered samples satisfies: x∼ρ[I(x)(J(πθ(⋅|x))−J(πθt(⋅|x)))]≥∑i=13ℒi(πθ,αi)E_x _X[I(x)(J( _θ(·|x))-J( _ _t(·|x)))]≥ _i=1^3L_i( _θ, _i) where: J(πθ(⋅∣x))=y∼πθ(⋅∣x)r(x,y)J ( _θ(· x) )\;=\;E_y _θ(· x)\,r(x,y) ℒi(πθ,αi)=x∈i[Lαi(πθ(⋅|x))−2Ki⋅TV(πθ(⋅|x),αi(⋅|x))−2TV(πθt(⋅|x),αi(⋅|x))]L_i( _θ, _i)=E_x _i [L_ _i( _θ(·|x))-2 [rgb]0,0,0 [named]pgfstrokecolorrgb0,0,0 @color@gray@stroke0 @color@gray@fill0K_i·TV( _θ(·|x), _i(·|x))-2TV( _ _t(·|x), _i(·|x)) ] with Lαi(πθ(⋅|x))=1σαi,r,ε(x)(J(πθ(⋅|x))−J(αi(⋅|x)))L_ _i( _θ(·|x))= 1 _ _i,r, (x)(J( _θ(·|x))-J( _i(·|x))). The constants are: K1 K_1 =1−G−1G2+εG−1G2+ε = 1- G-1G^2+ G-1G^2+ (11) K2 K_2 =1−c1(1−c1)+εc1(1−c1)+ε = 1- c_1(1-c_1)+ c_1(1-c_1)+ (12) K3 K_3 =1−min(c2(1−c2),c3(1−c3))+εmin(c2(1−c2),c3(1−c3))+ε = 1- (c_2(1-c_2),c_3(1-c_3))+ (c_2(1-c_2),c_3(1-c_3))+ (13) More importantly, we highlight several properties from this theorem: Bounded Stability. All constants K1K_1, K2K_2, and K3K_3 are finite positive values, which guarantee that the training process remains numerically stable and theoretically bounded. Off-policy Tolerance. The stability of trust-region methods inherently constrain the magnitude of single-step policy updates. Consequently, the divergence between the current policy πθt _ _t and the delayed rollout policy α remains bounded over short intervals. Furthermore, the strict FIFO mechanism with limited buffer capacity ensures that only samples from recent policies are retained, thereby maintaining policy consistency within the training batch. 4 Experimental Setup To comprehensively evaluate the effectiveness of our off-policy RLVR framework, we conduct extensive experiments across different tasks and backbones, following the experimental setup described in (Qu et al., 2025). First, we select three representative reasoning tasks, as detailed below: Mathematics. Following prior work (Luo et al., 2025), we use the DeepSeek R1 Distilled 1.5B (Guo et al., 2025) and Qwen3 8B (Yang et al., 2025) as the base model, and conducted post-training on the DeepScaleR-Preview-Dataset (Aggarwal and Welleck, 2025), which contains 40 thousand question-answer pairs sourced from several mathematics competitions. Evaluation is performed on multiple mathematics benchmarks, including AIME24, AMC23, MATH500 (Hendrycks et al., 2021), Minerva Math (Minerva) (Lewkowycz et al., 2022), and OlympiadBench (Olympiad)He et al. (2024). Planning. We choose Qwen2.5 Math 1.5B and 7B (Yang et al., 2024) as the backbone, and adopted the Countdown Number Game as the specific task. For training, we used a 10,000-problem subset of the Countdown-34 dataset, where each problem provides 3-4 source numbers. Evaluation was conducted on two variants: Countdown-3to4 (CD-34) test set using a 200-problem held-out split, and the more challenging Countdown-4 (CD-4) test set with 200 problems that consistently provide four source numbers (Chen et al., 2025b). Visual Geometry. We train Qwen2.5 VL 3B and 7B (Bai et al., 2025) on the 2,101-problem training split of the Geometry3K dataset (Lu et al., 2021), where each problem consists of a geometric diagram paired with a natural language question requiring spatial and logical reasoning. Evaluation was performed on the official 300-problem validation split (Geo-3K val) and 601-problem test split of Geometry3K (Geo-3K test). Besides, we select several on-policy and off-policy RLVR frameworks as baselines: On-policy. We select GRPO (Shao et al., 2024), DAPO (Yu et al., 2025), and MoPPS (Qu et al., 2025) as representative on-policy RLVR methods. GRPO is the first to integrate group-relative advantage estimation into the RLVR framework, while DAPO further improves training stability and efficiency. MoPPS incorporates difficulty-aware prediction into prompt selection. Off-policy. We compare our approach with three representative off-policy methods: GRPO (v=5v=5) (Mroueh et al., 2025), RePO (Li et al., 2025), and Remix-GRPO (Liang et al., 2025). Specifically, GRPO (v=5v=5) delays the rollout policy with a frequency of 5, whereas RePO and Remix-GRPO adopt diverse replay strategies to retrieve off-policy samples from a replay buffer. Implementation Details. All comparative experiments were run on 8 A100 GPUs with 80GB memory based on the Verl framework (Sheng et al., 2025). Identical parameters were used to ensure fair comparison, with specific details in Appendix A.7. 5 Results Analysis 5.1 Main Results We evaluate BAPO across three reasoning tasks to demonstrate its broad applicability. Experimental results show that BAPO consistently outperforms existing baselines throughout training (Figure 4) and testing (Figure 12). Notably, in mathematical tasks, the GRPO baseline exhibits severe training instability, as evidenced by significant oscillations in its early-stage training curve. This is attributed to the high variance in problem difficulty within the DeepScalerR dataset. Under the same settings, BAPO achieves smoother convergence and higher reward bounds. In Tables 1, BAPO achieves an average 12.5% accuracy improvement over baselines. Crucially, while DAPO approaches BAPO’s performance in some metrics, it requires approximately 2.5× more rollouts (as visualized in Figure 9), imposing a substantial computational burden. Figure 4: Training Curves of Reward Changes for mathematics, planning, and geometry tasks using DeepSeek Distilled Qwen 1.5B, Qwen2.5 Math 1.5B, and Qwen2.5 VL 3B, respectively. Table 1: Comprehensive Evaluation Results. ’+’ indicates fine-tuning via the corresponding method. Accuracy is averaged over 32 runs. The bold value denotes the top result, and the underlined value denotes the second-top result. (a) Mathematics Benchmarks Method AIME24 AMC MATH500 Minerva. Olympiad. Avg. ↑ Rollouts ↓ Type DeepSeek R1 Distill Qwen 1.5B 28.80 62.90 82.80 26.50 44.42 48.90 - - +GRPO (Guo et al., 2025) 30.73 67.47 85.40 28.95 45.33 51.58 677k on +DAPO (Yu et al., 2025) 35.73 70.08 86.05 30.70 48.48 54.20 1921k on +MoPPS∗ (Qu et al., 2025) 33.33 65.29 84.94 28.88 45.93 51.67 737k on +GRPO (v=5v=5) (Mroueh et al., 2025) 30.49 65.09 86.72 28.16 46.18 51.57 677k off +RePO (Li et al., 2025) 30.42 64.76 83.75 28.33 45.44 50.54 677k off +Remix-GRPO∗ (Liang et al., 2025) 33.33 65.06 84.60 26.10 43.55 50.53 - off +BAPO (Ours) 38.54 72.74 89.18 29.55 50.06 56.01 733k off (b) Planning and Visual Geometry Benchmarks Method CD-34 CD-4 Avg Method Geo-3K(val) Geo-3K(test) Avg Qwen2.5 Math 1.5B 1.12 0.37 0.75 Qwen2.5 VL 3B 14.77 19.18 16.98 +GRPO (Guo et al., 2025) 62.94 35.88 49.41 +GRPO (Guo et al., 2025) 36.44 43.12 39.78 +DAPO (Yu et al., 2025) 70.56 45.87 58.22 +DAPO (Yu et al., 2025) 40.11 45.18 42.65 +BAPO w/o 2X_2 (Ours) 60.31 35.31 47.81 +BAPO w/o 2X_2 (Ours) 30.57 36.92 33.75 +BAPO w/o 3X_3 (Ours) 64.43 38.75 51.59 +BAPO w/o 3X_3 (Ours) 32.22 39.79 36.01 +BAPO (Ours) 73.00 47.50 60.25 +BAPO (Ours) 40.11 46.33 43.22 Qwen2.5 Math 7B 2.68 0.94 1.81 Qwen2.5 VL 7B 30.40 36.10 33.25 +GRPO (Guo et al., 2025) 70.75 50.25 60.50 +GRPO (Guo et al., 2025) 40.79 47.15 43.97 +DAPO (Yu et al., 2025) 78.75 57.43 68.09 +DAPO (Yu et al., 2025) 40.87 47.02 43.95 +BAPO (Ours) 79.13 57.13 68.13 +BAPO (Ours) 41.89 48.77 45.33 *This method’s performance is taken from the corresponding paper. 5.2 Mechanism Analysis To deeply investigate whether BAPO’s success stems from sensitive hyperparameter tuning or its core batch reconstruction mechanism, we conducted both Minimalist Verification and Hyperparameter Robustness experiments. Off-policy Components >> Off-policy Hyperparameters The performance gains of BAPO primarily stem from the structural logic of its off-policy components rather than specific hyperparameter settings. The framework remains effective even under rigid, parameter-free conditions. Figure 5: Test Curves of Group Accuracy Changes on AIME for different RLVR methods based on Qwen3 8B. Left: Standard BAPO vs. GRPO. Medium: BAPO (mini test) vs. GRPO. Right: Standard BAPO vs. DAPO. Minimalist Verification. To validate the theoretical implications of Theorem 3.2 without relying on hyperparameter engineering, specifically avoiding the tuning of thresholds c1,c2,c3c_1,c_2,c_3 and update frequencies, we devised a “Mini-test” experiment. We trained Qwen3 8B on the mathematics task under 4K length constraints using a stripped-down, parameter-free BAPO logic for constructing training batch: 1X_1: We apply strictly standard zero-advantage filtering, removing only the prompts where all G responses are entirely correct or entirely wrong. 2X_2: We replay historical all-wrong samples (μα,r(x)=0 _α,r(x)=0). These correspond exactly to the difficult cases discarded by 1X_1, creating a closed-loop system that recovers waste data without requiring a difficulty threshold c1c_1. 3X_3: Instead of a dynamic accuracy range, we reuse historical samples with exactly 50% accuracy. As formally proven in Proposition A.3, samples with accuracy μα,r(x)=12 _α,r(x)= 12 maximize the reward variance, thereby providing the theoretical maximum potential for single-step policy improvement J(πθ)−J(πθt)J( _θ)-J( _ _t). The results in Figure 5 demonstrate that even in the hyperparameter-free “Mini-test”, BAPO maintains a clear advantage over GRPO. This confirms that the structural introduction of 2X_2 and 3X_3 drives the performance, not the specific tuning of c values. Component Efficacy. To evaluate the contribution of re-evaluated difficult samples 2X_2 and reused high-quality samples 3X_3, we conduct ablation studies shown in Table 1 and Figure 6 (Column 2). Both components are essential: removing 2X_2 causes a ∼ 21% performance drop, underscoring the importance of explicitly targeting difficult samples. Hyperparameter Robustness. We further evaluate the sensitivity of BAPO to its key hyperparameters: rollout delay v, re-rollout frequency m, and difficulty thresholds. Frequency (v,mv,m): As shown in Figure 6 (Column 1), performance remains stable within reasonable ranges (e.g., v=5,m=5v=5,m=5). Extreme delays only degrade performance when policy divergence becomes excessive, aligning with our theoretical analysis regarding the trust region. Difficulty Thresholds (c2,c3c_2,c_3): While our adaptive boundary mechanism yields the best convergence, Figure 6 (Column 3) shows that using fixed ranges still significantly outperforms baselines. This indicates that the presence of diverse historical data is more critical than the precise values of the thresholds. Figure 6: Ablation Studies for BAPO. The first column presents ablations on frequency-related hyperparameters (m,vm,v). The second column shows ablations on buffer subsets (2,3X_2,X_3). The third column compares fixed vs. adaptive difficulty thresholds. 5.3 Detailed Analysis We analyze BAPO’s internal mechanisms below. For extended analysis on training dynamics, computation, and visualization, please refer to Appendices A.4, A.5 and A.6. Tracking Difficult Samples. We visualize the training dynamics in Figure 7. BAPO exhibits a superior capability to ”unlock” difficult problems: after 3 epochs, BAPO successfully improves 31% of the samples that were initially unsolvable (0/80/8 accuracy), compared to only 19% for GRPO. Figure 7: Tracking changes in the Number of Different Accuracy Bins on the DeepScalerR training subset. Special attention is paid to the reduction of bad samples (red bars). Sample Distribution & Efficiency. To uncover the source of BAPO’s efficiency, we analyze the dynamic batch construction in Figure 8 alongside the rollout costs in Figure 9. As observed in Figure 8, the assembled training batch size frequently fluctuates below the maximum configured capacity. This reduction in backward propagation load effectively offsets the computational overhead caused by off-policy re-evaluation and log-probability re-computation. Consequently, as detailed in Table. 2, BAPO maintains a training speed comparable to GRPO while requiring significantly fewer rollouts than DAPO, achieving a superior trade-off between convergence performance and computational cost. Figure 8: Dynamic Sample Distribution. The composition of BAPO’s 1,2,3X_1,X_2,X_3 and the total samples compared to the fixed GRPO batch size (Red line). Figure 9: Cumulative Rollout Batches Comparison between BAPO and DAPO. The maximum rollout time for DAPO is set to 4. Efficient Batch Adaptation BAPO maintains training efficiency comparable to GRPO. While the periodic re-evaluation of 2X_2 introduces additional generation overhead, this cost is effectively offset by reduced training samples, particularly during the initial stages of training. 6 Conclusion In this paper, we propose BAPO, an off-policy RLVR framework for LLM post-training. It aims to utilize historical training data better and thereby improve training efficiency. Specifically, we appropriately delay the rollout policy to stabilize the policy discrepancies of buffer samples. More importantly, we construct training batches by re-evaluating difficult samples and reusing historical high-quality ones, thereby enhancing the efficiency of post-training. We validate the strong adaptability of the BAPO framework through experiments on three distinct reasoning tasks using different LLM backbones, and the results demonstrate that BAPO significantly outperforms baselines in both convergence performance and training efficiency. Nevertheless, exploring how to adapt BAPO to large models with MoE architectures, as well as to agentic RL frameworks, remains a significant challenge. Acknowledgements This work was supported by the National Natural Science Foundation of China under Grant 72571007 and Grant 72595830/72595831, and by Beijing Nova Program (No. 20250484850). Ethics Statement All authors of this study strictly adhere to the ICLR code of ethics. Our research does not involve any potential conflicts of interest or sponsorship issues. We have carefully considered and addressed concerns related to discrimination, bias, and fairness in our methodology. The study raises no privacy or security concerns, maintains full legal compliance, and upholds the highest standards of research integrity. All experimental procedures and data handling practices follow established ethical guidelines for machine learning research. Reproducibility statement To ensure full reproducibility of our results, we provide comprehensive implementation details of the proposed BAPO training algorithm in the supplementary materials. All experimental settings, hyperparameters, and dataset specifications are clearly documented. For our theoretical contributions, complete proofs and clear explanations of all assumptions are included in the appendix. Code and data will be made available upon acceptance to facilitate replication of our findings. The Use of Large Language Models In this research, we employed LLMs solely as language editing tools to improve the clarity and readability of our manuscript. LLMs were used for grammar checking, style refinement, and language polishing purposes only. All core research ideas, experimental design, analysis, and conclusions are entirely the original work of the authors. The use of LLMs did not contribute to the conceptual or technical content of this study. References P. Aggarwal and S. Welleck (2025) L1: controlling how long a reasoning model thinks with reinforcement learning. arXiv preprint arXiv:2503.04697. Cited by: §4. S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, et al. (2025) Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923. Cited by: §4. Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. (2022) Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. Cited by: §1. Q. Chen, L. Qin, J. Liu, D. Peng, J. Guan, P. Wang, M. Hu, Y. Zhou, T. Gao, and W. Che (2025a) Towards reasoning era: a survey of long chain-of-thought for reasoning large language models. arXiv preprint arXiv:2503.09567. Cited by: §1. X. Chen, J. Lu, M. Kim, D. Zhang, J. Tang, A. Piché, N. Gontier, Y. Bengio, and E. Kamalloo (2025b) Self-evolving curriculum for llm reasoning. arXiv preprint arXiv:2505.14970. Cited by: §4. Y. Chen, Z. Yang, Z. Liu, C. Lee, P. Xu, M. Shoeybi, B. Catanzaro, and W. Ping (2025c) Acereason-nemotron: advancing math and code reasoning through reinforcement learning. arXiv preprint arXiv:2505.16400. Cited by: §1. G. Cui, Y. Zhang, J. Chen, L. Yuan, Z. Wang, Y. Zuo, H. Li, Y. Fan, H. Chen, W. Chen, et al. (2025) The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505.22617. Cited by: §2.2. W. Fu, J. Gao, X. Shen, C. Zhu, Z. Mei, C. He, S. Xu, G. Wei, J. Mei, J. Wang, et al. (2025) AReaL: a large-scale asynchronous reinforcement learning system for language reasoning. arXiv preprint arXiv:2505.24298. Cited by: §2.2. D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. (2025) Deepseek-r1: incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. Cited by: §1, §2.1, §4, Table 1, Table 1, Table 1, Table 1, Table 1. C. He, R. Luo, Y. Bai, S. Hu, Z. L. Thai, J. Shen, J. Hu, X. Han, Y. Huang, Y. Zhang, et al. (2024) Olympiadbench: a challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402.14008. Cited by: §4. J. He, J. Liu, C. Y. Liu, R. Yan, C. Wang, P. Cheng, X. Zhang, F. Zhang, J. Xu, W. Shen, et al. (2025) Skywork open reasoner 1 technical report. arXiv preprint arXiv:2505.22312. Cited by: §1. D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt (2021) Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874. Cited by: §4. J. Hilton, K. Cobbe, and J. Schulman (2022) Batch size-invariance for policy optimization. Advances in Neural Information Processing Systems 35, p. 17086–17098. Cited by: §1. W. Hong, W. Yu, X. Gu, G. Wang, G. Gan, H. Tang, J. Cheng, J. Qi, J. Ji, L. Pan, et al. (2025) GLM-4.1 v-thinking: towards versatile multimodal reasoning with scalable reinforcement learning. arXiv preprint arXiv:2507.01006. Cited by: §1. N. Lambert, J. Morrison, V. Pyatkin, S. Huang, H. Ivison, F. Brahman, L. J. V. Miranda, A. Liu, N. Dziri, S. Lyu, et al. (2024) Tulu 3: pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124. Cited by: §1, §2.1. A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ramasesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, et al. (2022) Solving quantitative reasoning problems with language models. Advances in neural information processing systems 35, p. 3843–3857. Cited by: §4. S. Li, Z. Zhou, W. Lam, C. Yang, and C. Lu (2025) RePO: replay-enhanced policy optimization. arXiv preprint arXiv:2506.09340. Cited by: §1, §2.2, §4, Table 1. J. Liang, H. Tang, Y. Ma, J. Liu, Y. Zheng, S. Hu, L. Bai, and J. Hao (2025) Squeeze the soaked sponge: efficient off-policy reinforcement finetuning for large language model. arXiv preprint arXiv:2507.06892. Cited by: §2.2, §4, Table 1. Z. Liu, C. Gong, X. Fu, Y. Liu, R. Chen, S. Hu, S. Zhang, R. Liu, Q. Zhang, and D. Tu (2025) GHPO: adaptive guidance for stable and efficient llm reinforcement learning. arXiv preprint arXiv:2507.10628. Cited by: §1. F. Lu, Z. Zhong, S. Liu, C. Fu, and J. Jia (2025) ARPO: end-to-end policy optimization for gui agents with experience replay. arXiv preprint arXiv:2505.16282. Cited by: §2.2. P. Lu, R. Gong, S. Jiang, L. Qiu, S. Huang, X. Liang, and S. Zhu (2021) Inter-gps: interpretable geometry problem solving with formal language and symbolic reasoning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), p. 6774–6786. Cited by: §4. M. Luo, S. Tan, J. Wong, X. Shi, W. Y. Tang, M. Roongta, C. Cai, J. Luo, T. Zhang, L. E. Li, et al. (2025) Deepscaler: surpassing o1-preview with a 1.5 b model by scaling rl. Notion Blog. Cited by: §4. L. Ma, H. Liang, M. Qiang, L. Tang, X. Ma, Z. H. Wong, J. Niu, C. Shen, R. He, B. Cui, et al. (2025) Learning what reinforcement learning can’t: interleaved online fine-tuning for hardest questions. arXiv preprint arXiv:2506.07527. Cited by: §2.2. W. Meng, Q. Zheng, G. Pan, and Y. Yin (2023) Off-policy proximal policy optimization. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37, p. 9162–9170. Cited by: §1. Y. Mroueh, N. Dupuis, B. Belgodere, A. Nitsure, M. Rigotti, K. Greenewald, J. Navratil, J. Ross, and J. Rios (2025) Revisiting group relative policy optimization: insights into on-policy and off-policy training. arXiv preprint arXiv:2505.22257. Cited by: §1, §2.2, §3.3, §4, Table 1. Y. Mroueh (2025) Reinforcement learning with verifiable rewards: grpo’s effective loss, dynamics, and success amplification. arXiv preprint arXiv:2503.06639. Cited by: §1. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. (2022) Training language models to follow instructions with human feedback. Advances in neural information processing systems 35, p. 27730–27744. Cited by: §1. Y. Qu, Q. C. Wang, Y. Mao, V. T. Hu, and X. Ji (2025) Can prompt difficulty be online predicted for accelerating rl finetuning of reasoning models?. arXiv preprint arXiv:2507.04632. Cited by: §2.1, §4, §4, Table 1. J. Queeney, Y. Paschalidis, and C. G. Cassandras (2021) Generalized proximal policy optimization with sample reuse. Advances in Neural Information Processing Systems 34, p. 11909–11919. Cited by: §1. Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. (2024) Deepseekmath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: §1, §2.1, §4. H. Shen, P. Liu, J. Li, C. Fang, Y. Ma, J. Liao, Q. Shen, Z. Zhang, K. Zhao, Q. Zhang, et al. (2025) Vlm-r1: a stable and generalizable r1-style large vision-language model. arXiv preprint arXiv:2504.07615. Cited by: §1. G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng, H. Lin, and C. Wu (2025) Hybridflow: a flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, p. 1279–1297. Cited by: §4. M. Simoni, A. Fontana, G. Rossolini, and A. Saracino (2025) GTPO: trajectory-based policy optimization in large language models. External Links: 2508.03772, Link Cited by: §1. Y. Sun, J. Shen, Y. Wang, T. Chen, Z. Wang, M. Zhou, and H. Zhang (2025) Improving data efficiency for llm reinforcement fine-tuning through difficulty-targeted online data selection and rollout replay. arXiv preprint arXiv:2506.05316. Cited by: §1, §2.2. K. Team, A. Du, B. Gao, B. Xing, C. Jiang, C. Chen, C. Li, C. Xiao, C. Du, C. Liao, et al. (2025) Kimi k1. 5: scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599. Cited by: §2.2. J. Yan, Y. Li, Z. Hu, Z. Wang, G. Cui, X. Qu, Y. Cheng, and Y. Zhang (2025) Learning to reason under off-policy guidance. arXiv preprint arXiv:2504.14945. Cited by: §2.2. A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. (2025) Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: §1, §4. A. Yang, B. Zhang, B. Hui, B. Gao, B. Yu, C. Li, D. Liu, J. Tu, J. Zhou, J. Lin, et al. (2024) Qwen2. 5-math technical report: toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122. Cited by: §4. Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, W. Dai, T. Fan, G. Liu, L. Liu, et al. (2025) Dapo: an open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476. Cited by: §1, §1, §1, §2.1, §4, Table 1, Table 1, Table 1, Table 1, Table 1. X. Zhang, J. Wang, Z. Cheng, W. Zhuang, Z. Lin, M. Zhang, S. Wang, Y. Cui, C. Wang, J. Peng, et al. (2025) Srpo: a cross-domain implementation of large-scale reinforcement learning on llm. arXiv preprint arXiv:2504.14286. Cited by: §1. C. Zheng, S. Liu, M. Li, X. Chen, B. Yu, C. Gao, K. Dang, Y. Liu, R. Men, A. Yang, et al. (2025) Group sequence policy optimization. arXiv preprint arXiv:2507.18071. Cited by: §1, §2.1. Appendix A Appendix A.1 Glossary of Terms and Notations Term Definition c1,c2,c3c_1,c_2,c_3 Thresholds for classifying historical samples by difficulty (group mean reward). 1,2,3X_1,X_2,X_3 Subsets of training batch: fresh, re-evaluated difficult, and historical high-quality samples. m Re-evaluation frequency for historically difficult samples. v Delay steps for updating the rollout policy. G Group size, number of responses generated per prompt during rollout. ℬB Replay buffer storing historical samples. A^i,t A_i,t Estimated advantage for token t in response i. ε Clipping parameter in PPO-style objectives. β Coefficient for KL penalty in the objective function. I(x)I(x) Filter function for constructing BAPO’s training batch. KLD_KL Kullback–Leibler divergence, used to constrain policy deviation. α Rollout policy for BAPO, which synchronizes to πθ _θ every v steps. πθ _θ LLM policy parameterized by θ. πref _ref Reference policy (e.g., initial pre-trained model). ρ(θ)ρ(θ) Importance sampling ratio: πθ(y|x)πold(y|x) _θ(y|x) _old(y|x). r(x,y)r(x,y) Reward function, we set to binary (0/1) based on correctness. μα,r(x) _α,r(x) Expected reward under policy π for input x. We approximate this value using the mean of r(x,y)r(x,y) corresponding to G responses y generated by the rollout policy α for each prompt x. σα,r,ε(x) _α,r, (x) Standard deviation of rewards under policy α for input x, with smoothing ε . J(π(⋅|x))J(π(·|x)) Expected reward of policy π for input x: y∼π(⋅|x)[r(x,y)]E_y π(·|x)[r(x,y)]. (μα,r(x)∣μ,σ2)N( _α,r(x) μ,σ^2) A sampling method that assigns weights to online rollouts based on a normal distribution centered at μ with standard deviation σ, used to filter samples by their group mean reward μα,r(x) _α,r(x). A.2 Theoretical Analysis Lemma A.1 (Kantorovich-Rubenstein duality of total variation distance). The Kantorovich-Rubinstein duality (variational representation) of the total variation distance is as follows: TV(m1,m2)=12Lsupg∈LZ∼m1[g(Z)]−Z∼m2[g(Z)],TV(m_1,m_2)= 12L _g _L \E_Z m_1[g(Z)]-E_Z m_2[g(Z)] \, (14) where L=g:→ℝ,‖g‖∞≤LG_L=\g:Z ,\|g\|_∞≤ L\. Theorem A.2 (Policy Improvement Lower Bound with Adaptive Training Batch). Assume rewards are bounded: 0≤r≤10≤ r≤ 1. Let πθt _ _t be the current policy, α1=πθt−v _1= _ _t-v be the delayed rollout policy, α2=πθt _2= _ _t be the current policy for re-evaluation, α3=αℬ _3= _B be the buffer policy distribution, and I(x)I(x) be the filtering function partitioning samples into 1X_1, 2X_2, and 3X_3. Suppose c1,c2,c3∈(0,1)c_1,c_2,c_3∈(0,1) with c2<c3c_2<c_3. and the following TV distance constraints hold: TV(πθt(⋅|x),πθt−v(⋅|x)) ( _ _t(·|x), _ _t-v(·|x)) ≤δ1∀x∈1 ≤ _1 ∀ x _1 (15) TV(πθt(⋅|x),αℬ(⋅|x)) ( _ _t(·|x), _B(·|x)) ≤δ3∀x∈3 ≤ _3 ∀ x _3 (16) where δ1,δ3>0 _1, _3>0 are sufficiently small such that the variance lower bounds remain positive. Then, for the policy update objective in Equation 3, the expected policy improvement over filtered samples satisfies: x∼ρ[I(x)(J(πθ(⋅|x))−J(πθt(⋅|x)))]≥∑i=13ℒi(πθ,αi)E_x _X[I(x)(J( _θ(·|x))-J( _ _t(·|x)))]≥ _i=1^3L_i( _θ, _i) where: ℒi(πθ,αi)=x∈i[Lαi(πθ(⋅|x))−2Ki⋅TV(πθ(⋅|x),αi(⋅|x))−2TV(πθt(⋅|x),αi(⋅|x))]L_i( _θ, _i)=E_x _i [L_ _i( _θ(·|x))-2 [rgb]0,0,0 [named]pgfstrokecolorrgb0,0,0 @color@gray@stroke0 @color@gray@fill0K_i·TV( _θ(·|x), _i(·|x))-2TV( _ _t(·|x), _i(·|x)) ] with Lαi(πθ(⋅|x))=1σαi,r,ε(x)(J(πθ(⋅|x))−J(αi(⋅|x)))L_ _i( _θ(·|x))= 1 _ _i,r, (x)(J( _θ(·|x))-J( _i(·|x))). The constants are: K1 K_1 =1−G−1G2+εG−1G2+ε = 1- G-1G^2+ G-1G^2+ (17) K2 K_2 =1−c1(1−c1)+εc1(1−c1)+ε = 1- c_1(1-c_1)+ c_1(1-c_1)+ (18) K3 K_3 =1−min(c2(1−c2),c3(1−c3))+εmin(c2(1−c2),c3(1−c3))+ε = 1- (c_2(1-c_2),c_3(1-c_3))+ (c_2(1-c_2),c_3(1-c_3))+ (19) Proof. We prove the bound by analyzing each filtered sample set separately, applying off-policy policy improvement bounds tailored to the reference distribution used in each region. Step 1: Core inequality for off-policy samples. For any x such that I(x)=1I(x)=1, we establish the fundamental inequality: J(πθ(⋅|x))−J(πθt(⋅|x)) J( _θ(·|x))-J( _ _t(·|x)) ≥Lαi(πθ(⋅|x))−2Ki⋅TV(πθ(⋅|x),αi(⋅|x)) ≥ L_ _i( _θ(·|x))-2K_i·TV( _θ(·|x), _i(·|x)) (20) −2TV(πθt(⋅|x),αi(⋅|x)) -2TV( _ _t(·|x), _i(·|x)) (21) where Ki=1−σαi,r,ε(x)σαi,r,ε(x)K_i= 1- _ _i,r, (x) _ _i,r, (x) is a constant that depends on the variance of rewards in each filtered subset. First, we expand the advantage objective. By definition: Lαi(πθ(⋅|x)) L_ _i( _θ(·|x)) =y∼αi(⋅|x)[πθ(y|x)αi(y|x)Aαi(x,y)] =E_y _i(·|x) [ _θ(y|x) _i(y|x)A_ _i(x,y) ] (22) =y∼αi(⋅|x)[πθ(y|x)αi(y|x)⋅r(x,y)−μαi,r(x)σαi,r,ε(x)] =E_y _i(·|x) [ _θ(y|x) _i(y|x)· r(x,y)- _ _i,r(x) _ _i,r, (x) ] (23) =1σαi,r,ε(x)(J(πθ(⋅|x))−J(αi(⋅|x))) = 1 _ _i,r, (x)(J( _θ(·|x))-J( _i(·|x))) (24) Next, we establish the key algebraic identity relating Lαi(πθ(⋅|x))L_ _i( _θ(·|x)) to J(πθ(⋅|x))−J(πθt(⋅|x))J( _θ(·|x))-J( _ _t(·|x)): Lαi(πθ(⋅|x))−(J(πθ(⋅|x))−J(πθt(⋅|x))) L_ _i( _θ(·|x))-(J( _θ(·|x))-J( _ _t(·|x))) (25) =1−σαi,r,ε(x)σαi,r,ε(x)(J(πθ(⋅|x))−J(αi(⋅|x)))+(J(πθt(⋅|x))−J(αi(⋅|x))) = 1- _ _i,r, (x) _ _i,r, (x)(J( _θ(·|x))-J( _i(·|x)))+(J( _ _t(·|x))-J( _i(·|x))) (26) Application of Kantorovich-Rubenstein duality: For bounded rewards with ‖r‖∞=1\|r\|_∞=1, the Kantorovich-Rubenstein duality Lemma A.1 provides: |J(πθ(⋅|x))−J(αi(⋅|x))| |J( _θ(·|x))-J( _i(·|x))| ≤2⋅TV(πθ(⋅|x),αi(⋅|x)) ≤ 2·TV( _θ(·|x), _i(·|x)) (27) |J(πθt(⋅|x))−J(αi(⋅|x))| |J( _ _t(·|x))-J( _i(·|x))| ≤2⋅TV(πθt(⋅|x),αi(⋅|x)) ≤ 2·TV( _ _t(·|x), _i(·|x)) (28) Since 0≤r≤10≤ r≤ 1, we have σαi,r,ε(x)<1 _ _i,r, (x)<1, ensuring Ki=1−σαi,r,ε(x)σαi,r,ε(x)≥0K_i= 1- _ _i,r, (x) _ _i,r, (x)≥ 0. Combining these bounds yields the desired inequality. Step 2: Analysis for 1X_1 (Filtered fresh samples). For x∈1x _1, samples are generated by the delayed rollout policy α1=πθt−v _1= _ _t-v and selected via Gaussian sampling with group-level accuracy μα1,r(x)∈1G,2G,…,G−1G _ _1,r(x)∈\ 1G, 2G,…, G-1G\, excluding extremes 0,1\0,1\. Variance analysis on discrete set: For the variance function f(p)=p(1−p)f(p)=p(1-p) over the discrete set 1G,2G,…,G−1G\ 1G, 2G,…, G-1G\, the minimum value occurs at the boundary points p=1Gp= 1G or p=G−1Gp= G-1G, both yielding f(p)=G−1G2f(p)= G-1G^2. Therefore: σα1,r2(x)=μα1,r(x)(1−μα1,r(x))≥G−1G2 σ^2_ _1,r(x)= _ _1,r(x)(1- _ _1,r(x))≥ G-1G^2 (29) Thus: σα1,r,ε(x)≥G−1G2+ε _ _1,r, (x)≥ G-1G^2+ , yielding: K1=1−G−1G2+εG−1G2+εK_1= 1- G-1G^2+ G-1G^2+ Step 3: Analysis for 2X_2 (Re-evaluated difficult samples). For x∈2x _2, samples are generated by the current policy α2=πθt _2= _ _t through re-evaluation of historically difficult samples. The selection criterion ensures that historically difficult samples (μαB,r(x)≤c1 _ _B,r(x)≤ c_1) now achieve improved performance (c1<μπθt,r(x)<1c_1< _ _ _t,r(x)<1) under the current policy. Since these samples are directly generated by πθt _ _t, we have α2=πθt _2= _ _t, and the constraint c1<μπθt,r(x)<1c_1< _ _ _t,r(x)<1 provides a natural lower bound, yielding: σα2,r2(x)=μα2,r(x)(1−μα2,r(x))>c1(1−c1) σ^2_ _2,r(x)= _ _2,r(x)(1- _ _2,r(x))>c_1(1-c_1) (30) Therefore: σα2,r,ε(x)>c1(1−c1)+ε _ _2,r, (x)> c_1(1-c_1)+ , giving us: K2=1−c1(1−c1)+εc1(1−c1)+εK_2= 1- c_1(1-c_1)+ c_1(1-c_1)+ Step 4: Analysis for 3X_3 (Historical high-quality samples). For x∈3x _3, samples are generated by historical buffer policies α3=αB _3= _B with μαB,r(x)∈[c2,c3] _ _B,r(x)∈[c_2,c_3]. Since μα3,r(x)(1−μα3,r(x)) _ _3,r(x)(1- _ _3,r(x)) achieves its minimum at the endpoints of the interval [c2,c3][c_2,c_3]: σα3,r2(x)≥min(c2(1−c2),c3(1−c3)) σ^2_ _3,r(x)≥ (c_2(1-c_2),c_3(1-c_3)) (31) Therefore: σα3,r,ε(x)≥min(c2(1−c2),c3(1−c3))+ε _ _3,r, (x)≥ (c_2(1-c_2),c_3(1-c_3))+ , yielding: K3=1−min(c2(1−c2),c3(1−c3))+εmin(c2(1−c2),c3(1−c3))+εK_3= 1- (c_2(1-c_2),c_3(1-c_3))+ (c_2(1-c_2),c_3(1-c_3))+ Step 5: Combining the results. Taking expectations over x∼ρx _X and applying the indicator function decomposition: x∼ρ[I(x)(J(πθ(⋅|x))−J(πθt(⋅|x)))] _x _X[I(x)(J( _θ(·|x))-J( _ _t(·|x)))] (32) =∑i=13x∼ρ[x∈i(J(πθ(⋅|x))−J(πθt(⋅|x)))] = _i=1^3E_x _X[1_\x _i\(J( _θ(·|x))-J( _ _t(·|x)))] (33) ≥∑i=13x∈i[Lαi(πθ(⋅|x))−2Ki⋅TV(πθ(⋅|x),αi(⋅|x))−2TV(πθt(⋅|x),αi(⋅|x))] ≥ _i=1^3E_x _i [L_ _i( _θ(·|x))-2K_i·TV( _θ(·|x), _i(·|x))-2TV( _ _t(·|x), _i(·|x)) ] (34) =∑i=13ℒi(πθ,αi) = _i=1^3L_i( _θ, _i) (35) All constants K1K_1, K2K_2, K3K_3 are finite, since denominators are strictly positive by construction and numerators are bounded by 1 under c1,c2,c3∈(0,1)c_1,c_2,c_3∈(0,1), completing the proof. ∎ Proposition A.3. For binary reward tasks where r(x,y)∈0,1r(x,y)∈\0,1\, the contribution to the policy improvement lower bound is maximized when the expected group reward of the sample is μ=0.5μ=0.5. Proof. Recalling Theorem 3.2, the lower bound for policy improvement on a specific data distribution involves the constant K, which scales the penalty for policy divergence. The tightness of this bound is governed by the standard deviation of the rewards σα,r(x) _α,r(x). Due to advantage standardization A^∝1σ A 1σ, the effective step size in the advantage estimation and consequently the gradient magnitude is proportional to the inverse of the standard deviation. However, in the context of the lower bound analysis in Theorem 3.2, the stability constant K is defined as: K(μ)=1−σ(μ)σ(μ) K(μ)= 1-σ(μ)σ(μ) (36) where a smaller K indicates a tighter bound and thus a larger guaranteed improvement step. For a binary reward function r∈0,1r∈\0,1\, the reward distribution follows a Bernoulli distribution with parameter μ(x)=[r|x]μ(x)=E[r|x]. The variance is given by: σ2(μ)=μ(1−μ) σ^2(μ)=μ(1-μ) (37) To find the μ that maximizes variance, we take the derivative with respect to μ: dμ(μ−μ2)=1−2μ ddμ(μ-μ^2)=1-2μ (38) Setting the derivative to zero: 1−2μ=0⟹μ=0.5 1-2μ=0 μ=0.5 (39) Since the second derivative d2dμ2=−2<0 d^2dμ^2=-2<0, this is a global maximum. At μ=0.5μ=0.5, the variance is maximized (σ2=0.25,σ=0.5σ^2=0.25,σ=0.5). This corresponds to the state of maximum entropy, where the model is most ”uncertain” about the outcome. Training on these samples provides the strongest gradient signal for distinguishing between correct and incorrect reasoning paths, effectively maximizing the information gain per step. Conversely, as μ→0μ→ 0 or μ→1μ→ 1, σ→0σ→ 0, causing the advantage estimates to numerical instability or the gradient signal to vanish. Therefore, selecting samples with μ=0.5μ=0.5 theoretically offers the most efficient learning signal and the most favorable stability bound. ∎ A.3 Online Filter Mechanism Analysis To investigate the impact of fresh sample selection on training stability and convergence, we conduct an ablation study using Qwen3 8B with a 4K response length limit. We compare three distinct filtering strategies for the online component (1X_1): Mode 1 (Range Filter): It retains samples with group mean rewards μ∈[1G,G−1G]μ∈[ 1G, G-1G]. This effectively removes only the zero-advantage samples (all-correct or all-incorrect) that contribute minimal gradients. Mode 2 (Gaussian Filter): A difficulty-weighted strategy that prioritizes samples with high variance (accuracy near 0.5) using a Gaussian distribution, thereby reducing the proportion of extremely easy or hard samples. Mode 3 (Uniform Filter): A baseline that randomly selects 60% of the fresh samples regardless of their quality. This ratio was chosen to match the approximate data retention rates of Mode 1 and Mode 2 (approximately 40%–60%) for a fair comparison of data volume. Figure 10: Ablation on Online Filtering Strategies. Comparison of Range Filter, Gaussian Filter, and Uniform Filter on training stability (Grad Norm) and performance (Mean@8). The star symbol indicates the best checkpoint for BAPO. The Value of Quality over Randomness. As illustrated in Figure 10, the uniform filter mechanism exhibits severe instability, characterized by exploding gradient norms and a complete collapse in performance after 150 steps. Since this strategy blindly includes all-wrong samples (where μ=0μ=0), the model is forced to update based on low-quality, zero-advantage signals. Suppressing the token probabilities of incorrect responses without a corresponding positive signal introduces significant noise and uncertainty, ultimately destabilizing the policy. This failure highlights that the quality of the training batch, particularly the exclusion of zero-advantage noise, is crucial. Convergence Speed and Final Performance. The Gaussian filter demonstrates faster convergence in the early stages. By focusing heavily on samples with the highest variance (accuracy ≈ 0.5), it provides the steepest learning signal initially. However, its final convergence performance is lower than that of the range filter. We hypothesize that the Gaussian filter restricts sample diversity by aggressively filtering out samples that are slightly easier or harder but still informative. In contrast, the range filter retains a broader spectrum of valid samples. While it learns slightly slower initially, it maintains a rich distribution of training data, preventing premature plateauing and ultimately achieving the highest asymptotic performance. A.4 Training Dynamics and Test Curves As illustrated in Figure 11 and Figure 12, we present more detailed training dynamics and test curves for the Planning and Vision Geometry tasks. The results indicate that both BAPO and DAPO consistently outperform GRPO in terms of training rewards. Interestingly, BAPO exhibits higher entropy, reflecting better exploration capability compared to other algorithms, which also results in longer response lengths. Figure 11: Training Dynamics during BAPO, GRPO, and DAPO post-training, including training rewards, training entropy, and response lengths. Figure 12: Test Curves of Group Accuracy Changes for mathematics, planning, and geometry tasks among AMC, CD-4 test set, and Geo-3K test set, respectively. A.5 Computation Analysis From Table 2, we observe that BAPO’s computational overhead correlates with the number of samples requiring re-evaluation and the actual training batch size. For the Planning task, BAPO (w/o 2X_2) achieves the fastest training time by eliminating bad case re-evaluation, but this comes at the cost of reduced performance. For the Mathematics task, the high number of bad cases (as shown by the 0/8 accuracy samples in Figure 7) means that under our re-evaluation frequency setting of m=5m=5, inference time exceeds that of GRPO. However, this additional time investment proves valuable, yielding better bad-case handling rates and overall test performance, as shown in Figure 4 and Table LABEL:tab:math_exp. We plan to explore lower re-evaluation frequencies to assess the performance trade-offs. BAPO (c2=0.375,c3=0.5c_2=0.375,c_3=0.5) runs significantly faster than BAPO (c2=0,c3=0.25c_2=0,c_3=0.25) due to the larger historical data volume in the latter configuration. This causes BAPO (c2=0,c3=0.25c_2=0,c_3=0.25) to maintain a larger effective batch size than BAPO (c2=0.375,c3=0.5c_2=0.375,c_3=0.5). Training logs also confirm this observation: BAPO (c2=0,c3=0.25c_2=0,c_3=0.25) consistently utilizes 100% of the configured batch size (equivalent to on-policy methods’ batch size), while BAPO (c2=0.375,c3=0.5c_2=0.375,c_3=0.5) operates at approximately 70% capacity. Table 2: Computational Overhead Analysis. “Batch size” (a,b)(a,b) represents the sample batch size a and train mini batch size b. “Time” is measured in total training time (d=days, h=hours, m=minutes) on 8 A100 GPUs. Tasks Methods Batch Size Num Epoch Time Mathematics GRPO (256, 64) 3 1d 16h 58m DAPO (256, 64) 3 2d 15h 30m BAPO (256, 64) 3 1d 22h 37m Planning GRPO (256, 64) 3 3h 47m DAPO (256, 64) 3 6h 35m BAPO (256, 64) 3 3h 23m BAPO (w/o 2X_2) (256, 64) 3 2h 38m BAPO (w/o 3X_3) (256, 64) 3 3h 4m BAPO (c2=0,c3=0.25c_2=0,c_3=0.25) (256, 64) 3 3h 54m BAPO (c2=0.375,c3=0.5c_2=0.375,c_3=0.5) (256, 64) 3 3h 4m Visual Geometry GRPO (256, 64) 30 7h 55m DAPO (256, 64) 30 12h 19m BAPO (256, 64) 30 5h 50m BAPO (w/o 2X_2) (256, 64) 30 3h 42m BAPO (w/o 3X_3) (256, 64) 30 4h 31m Figure 13: Tracking changes in the Number of Different Accuracy Bins on the Countdown (upper) and Geometry3K training sets (lower) for the baseline model, GRPO, and our BAPO method. Special attention is paid to the change in the number of bad samples (red bars) that the base model fails to handle. Figure 14: Batch Distribution Visualization of 1X_1, 2X_2, 3X_3 for Mathematics, Planning, and Visual Geometry Tasks (left to right) during BAPO’s training. A.6 Visualization We present additional visualization details, including the sample accuracy tracking for the Countdown and Geometry3K datasets, as shown in Figure 13. Meanwhile, we visualize the source of samples in each training batch and their respective proportions during the training process, as illustrated in Figure 12. It can be observed that approximately 40-60% of the actual training samples for BAPO come from online samples 1X_1, while the remaining samples are derived from 2X_2 or 3X_3. Figure 15: Accuracy Migration Matrix Analysis. We track a fixed subset of 1,000 randomly selected prompts from the training set and visualize their movement between accuracy bins (0/8 to 8/8) at Steps 0, 150, 300, and 471 (the last step). The y-axis represents the initial accuracy bin at Step 0, while the x-axis represents the current accuracy bin. The scarcity of samples in the lower triangle demonstrates that performance degradation is rare. Stability of Historical High-Quality Samples. A potential concern regarding the reuse of historical high-quality samples (3X_3 in Eq. 5) is the assumption of policy consistency—specifically, whether samples that were high-quality under a past policy remain valid for the current policy. To address this, we visualize the evolution of sample difficulty in Figure 15 by tracking the accuracy migration of a training subset. The heatmaps in Figure 15 reveal a distinct pattern: the mass is concentrated along the diagonal (performance maintenance) and the upper triangle (performance improvement). Crucially, the proportion of samples exhibiting significant performance degradation (migrating to the lower triangle) is negligible. For example, samples that initially achieved 8/88/8 accuracy predominantly remain in the high-accuracy bins throughout the training process, with minimal regression to lower bins. This empirical evidence demonstrates that high-quality reasoning paths learned by RL are robust and resistant to forgetting. Consequently, historical high-quality samples stored in the buffer likely remain high-quality under the current policy, validating the consistency of the 3X_3 data source. A.7 Hyperparameter Setting Hyperparmeters The major hyperparameter choices are shown in Table 3. Table 3: Hyperparameter Configuration for BAPO Framework on Mathematics Task. For planning and visual geometry tasks, some parameters differ slightly; specific configuration scripts are provided in our code repository. Parameter Value Parameter Value Parameter Value Rollout Configuration Top-p 1 Top-k -1 Temperature 1 Group size (G) 8 Max prompt length 2048 Max response length 8192 Rollout workers 8 Sample batch size 256 Seed 42 Training Configuration Learning rate 1e-6 Train mini batch size 64 GAE lambda 1.0 Training epochs 3 KL coefficient (β) 0.001 Entropy coefficient 0.001 Off-policy Configuration c1c_1 threshold 1/81/8 c2c_2 range [1/8,4/8][1/8,4/8] c3c_3 range [2/8,5/8][2/8,5/8] Buffer size (|B||B|) 256 Rollout delay (v) 5 Re-evaluation freq (m) 5 Gaussian std (σ) 0.2 Gaussian mean (μ) 0.5 Max re-evaluate prompts 128 Reward Function To evaluate the impact of our method, we adopt a simple reward function as below. All training experiments employ the same reward function. r(x,y)=1,if y is correct0,otherwiser(x,y)= cases1,&if y is correct\\ 0,&otherwise cases Datasets and Benchmarks To evaluate the models above, we use three training datasets and eight benchmarks categorized into mathematical, planning and vision geometry reasoning benchmarks as described in Table 4. Table 4: Datasets and Benchmarks used in this study. Dataset #Train #Test Task Type Domain License Source Training Datasets DeepScaleR-1.5B-Preview 40,000 – Math reasoning Mathematics Apache 2.0 Link Countdown-Tasks-3to4 49,000 – Logic reasoning Planning Apache 2.0 Link Geometry3k 2,100 – Visual reasoning Visual Geometry Apache 2.0 Link Test Benchmarks AIME24 – 30 Math competition Mathematics MIT Link AMC – 83 Math competition Mathematics Apache 2.0 Link MATH500 – 500 Math reasoning Mathematics - Link Minerva – 272 Math reasoning Mathematics Apache 2.0 Link Olympiad – 674 Math competition Mathematics Apache 2.0 Link Countdown-Tasks-3to4 – 200∗ Logic reasoning Planning Apache 2.0 Link Countdown-Tasks-4 – 200∗ Logic reasoning Planning Apache 2.0 Link Geometry3k – 901 Visual reasoning Visual Geometry Apache 2.0 Link *We only use a random subset of this benchmark for faster ablation studies. A.8 Algorithm Algorithm 1 presents the proposed BAPO, which can be seamlessly integrated with any GRPO-like RLVR algorithm. Algorithm 1 Batch Adaptation Policy Optimization (BAPO) 0: Policy πθ0 _ _0, buffer ℬ=∅B= , thresholds c1,c2,c3c_1,c_2,c_3, delay steps v, re-evaluate frequency m 1: for t=1t=1 to T do 2: // Off-policy Rollout Phase 3: if tmodv=0t v=0 then 4: Synchronize rollout policy’s parameter with trainer: α=πθtα= _ _t 5: end if 6: Using rollout policy α to generate G responses yjj=1G\y_j\_j=1^G for each question x 7: Compute log probabilities α(y|x)α(y|x) and rewards r for constructing the online batch onX_on 8: Store samples into buffer ℬbad←(x,y,α(y|x),r)∈on:μα,r(x)≤c1B_bad←\(x,y,α(y|x),r) _on: _α,r(x)≤ c_1\ 9: Store samples into buffer ℬhigh←(x,y,α(y|x),r)∈on:c2≤μα,r(x)≤c3B_high←\(x,y,α(y|x),r) _ on:c_2≤ _α,r(x)≤ c_3\ 10: // Off-policy Training Phase 11: 1←X_1← online filter on onX_on with μα,r(x)∈1G,…,G−1G _α,r(x)∈\ 1G,…, G-1G\ (Filtered Fresh Samples) 12: 2←∅X_2← 13: if tmodm=0t m=0 then 14: Re-evaluate ℬbadB_bad with πθt _ _t to get 2X_2 using Equation 6 (Re-evaluated Difficult Samples) 15: end if 16: 3←X_3← Sample from (x,y)∈ℬhigh:μαℬ,r(x)∈[c2,c3]\(x,y) _high: _ _B,r(x)∈[c_2,c_3]\ (Historical High-quality Samples) 17: Final batch ←1∪2∪3 _1 _2 _3 18: Compute advantages and update critic/actor with final_batch 19: Add tD_t to buffer ℬB 20: end for A.9 Generalization Analysis To demonstrate the algorithmic generalizability of our framework, we extended the Batch Adaptation paradigm to Proximal Policy Optimization (PPO), denoted as BA-PPO. In this experiment, both the Actor and Critic networks were initialized with the Qwen3-4B backbone and trained on the DeepScaleR dataset with a maximum response length of 4K tokens. We maintained consistency with the foundational BAPO configuration by applying standard zero-advantage filtering for 1X_1 (removing only all-correct and all-wrong groups), utilizing the initial BAPO values for thresholds c1,c2,c3c_1,c_2,c_3, and setting the buffer size to 64. Figure 16: Generalization to Actor-Critic Algorithms (BA-PPO). Performance comparison between standard PPO (orange triangles) and BA-PPO (purple circles) on the AIME 2024 benchmark using Qwen3-4B. The star (⋆ ) marks the peak performance of BA-PPO (0.3250.325). As illustrated in Figure 16, BA-PPO achieved a remarkable performance gain of +5.5 on the AIME 2024 benchmark compared to the standard PPO baseline. This result further confirms that the core principle of dynamic batch construction is effective not only for GRPO but also functions as a robust, algorithm-agnostic enhancement for actor-critic methods.