Paper deep dive
Discounted Beta--Bernoulli Reward Estimation for Sample-Efficient Reinforcement Learning with Verifiable Rewards
Haechan Kim, Soohyun Ryu, Gyouk Chu, Doohyuk Jang, Eunho Yang
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/22/2026, 6:04:21 AM
Summary
The paper introduces Discounted Beta–Bernoulli (DBB) reward estimation to address sample inefficiency and variance collapse in Reinforcement Learning with Verifiable Rewards (RLVR). By modeling rewards as a non-stationary distribution and using a Bayesian framework to incorporate historical statistics, DBB reduces estimator variance and mean squared error compared to standard point estimation. Experiments show that integrating DBB into Group Relative Policy Optimization (GRPO) consistently improves performance on mathematical reasoning benchmarks across different model scales.
Entities (5)
Relation Signals (3)
GRPO-DBB → evaluatedon → MATH500
confidence 95% · We evaluate GRPO with the DBB reward estimator (GRPO-DBB) on... MATH500
Discounted Beta–Bernoulli (DBB) → improves → Group Relative Policy Optimization (GRPO)
confidence 95% · GRPO with DBB consistently outperforms naive GRPO
RLVR → uses → Group Relative Policy Optimization (GRPO)
confidence 90% · Since most RLVR algorithms, including Group Relative Policy Optimization (GRPO; Shao et al., 2024), rely on group-based advantage estimation
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Reinforcement learning with verifiable rewards (RLVR) has emerged as an effective post-training paradigm for improving the reasoning capabilities of large language models. However, existing group-based RLVR methods often suffer from severe sample inefficiency. This inefficiency stems from reliance on point estimation of rewards from a small number of rollouts, leading to high estimation variance, variance collapse, and ineffective utilization of generated responses. In this work, we reformulate RLVR from a statistical estimation perspective by modeling rewards as samples drawn from a policy-induced distribution and casting advantage computation as the problem of estimating the reward distribution from finite data. Building on this view, we propose Discounted Beta--Bernoulli (DBB) reward estimation, which leverages historical reward statistics for the non-stationary distribution. Although biased, the resulting estimator exhibits reduced and stable variance, theoretically avoids estimated variance collapse, and achieves lower mean squared error than standard point estimation. Extensive experiments across six in-distribution and three out-of-distribution reasoning benchmarks demonstrate that GRPO with DBB consistently outperforms naive GRPO, achieving average Acc@8 improvements of 3.22/2.42 points in-distribution and 12.49/6.92 points out-of-distribution on the 1.7B and 8B models, respectively, without additional computational cost or memory usage.
Tags
Links
- Source: https://arxiv.org/abs/2603.18444v1
- Canonical: https://arxiv.org/abs/2603.18444v1
Trouble viewing inline? Open PDF directly →
Full Text
54,009 characters extracted from source content.
Expand or collapse full text
Discounted Beta–Bernoulli Reward Estimation for Sample-Efficient Reinforcement Learning with Verifiable Rewards Haechan Kim Soohyun Ryu Gyouk Chu Doohyuk Jang Eunho Yang Abstract Reinforcement learning with verifiable rewards (RLVR) has emerged as an effective post-training paradigm for improving the reasoning capabilities of large language models. However, existing group-based RLVR methods often suffer from severe sample inefficiency. This inefficiency stems from reliance on point estimation of rewards from a small number of rollouts, leading to high estimation variance, variance collapse, and ineffective utilization of generated responses. In this work, we reformulate RLVR from a statistical estimation perspective by modeling rewards as samples drawn from a policy-induced distribution and casting advantage computation as the problem of estimating the reward distribution from finite data. Building on this view, we propose Discounted Beta–Bernoulli (DBB) reward estimation, which leverages historical reward statistics for the non-stationary distribution. Although biased, the resulting estimator exhibits reduced and stable variance, theoretically avoids estimated variance collapse, and achieves lower mean squared error than standard point estimation. Extensive experiments across six in-distribution and three out-of-distribution reasoning benchmarks demonstrate that GRPO with DBB consistently outperforms naive GRPO, achieving average Acc@8 improvements of 3.22/2.42 points in-distribution and 12.49/6.92 points out-of-distribution on the 1.7B and 8B models, respectively, without additional computational cost or memory usage. 1 Introduction Figure 1: Comparison between point estimation and DBB estimation. By trading a small bias for substantial variance reduction via shrinkage, DBB estimation achieves lower mean squared error. Compared to naive GRPO using point estimation, GRPO with DBB estimation consistently demonstrates superior performance across all benchmarks and both model scales. RLVR (Lambert et al., 2025) has recently emerged as a key post-training paradigm for enhancing the complex reasoning capabilities of large language models (Plaat et al., 2025) and improving their performance on downstream tasks (Guo et al., 2025; Yang et al., 2025; Yu et al., 2025b). Since most RLVR algorithms, including Group Relative Policy Optimization (GRPO; Shao et al., 2024), rely on group-based advantage estimation (Yu et al., 2025a; Liu et al., 2025; Zheng et al., 2025a; Zhao et al., 2025), they require generating multiple responses per prompt, which can account for a large fraction of the overall training time, often approaching 50% (Le et al., 2025). Despite this substantial computational cost, many existing group-based RLVR algorithms fail to utilize information from the generated responses efficiently. This sample inefficiency can be attributed to two fundamental characteristics of RLVR. The first characteristic is that the variance of rewards can collapse to zero when all generated responses receive identical rewards under group-relative estimation. This variance collapse issue not only wastes the computational cost of response generation but also eliminates meaningful training signals within the batch (Zhang et al., 2025). To address this issue, approaches such as dynamic sampling (Yu et al., 2025a) and GRPO with Efficient Selective Rollout (GRESO; Zheng et al., 2025b) have been proposed. However, dynamic sampling requires several times more rollout cost, and GRESO cannot fully resolve the problem since it probabilistically filters prompts that are likely to result in variance collapse. The second characteristic is that, due to the on-policy nature of RLVR algorithms, information from all generated responses is discarded after a single gradient update. Recent replay-based methods (Li et al., 2025; Zhan et al., 2025) attempt to reuse rollouts generated from the training history. However, unbiased reuse of off-policy data requires importance sampling, which in turn necessitates storing all token-level probabilities under historical policies and performing additional forward passes on the current policy. As a result, replay-based approaches introduce substantial GPU memory overhead and additional computational cost, limiting their practical scalability. These two sources of sample inefficiency primarily arise because most group-based RLVR algorithms rely on point estimators that consider only the rewards from the current rollout group when estimating the underlying reward distribution for advantage estimation. While such estimators can be reliable when the number of sampled responses per group is sufficiently large (Casella and Berger, 2024), practical RLVR settings often operate in low-sample regimes due to computational constraints. In these settings, point estimators remain unbiased but suffer from high variance, which can lead to variance collapse and unstable training dynamics. To address this limitation, we adopt a statistical perspective that models rewards as stochastic outcomes drawn from a distribution induced by the policy. Instead of relying on point estimation, we employ a Bayesian framework (Gelman et al., 1995) to model uncertainty explicitly and incorporate temporal reward dynamics. Within this framework, we propose the Discounted Beta–Bernoulli (DBB) reward estimation, which tracks the evolving reward distribution by discounting historical observations. While DBB introduces a small bias, it significantly reduces variance and avoids variance collapse, thereby providing more stable and informative training signals (Figure 1). Empirically, DBB yields a lower mean-squared error with respect to the true reward distribution than point estimation in low-sample scenarios. We evaluate GRPO with the DBB reward estimator (GRPO-DBB) on two model scales, Qwen3-1.7B-Base and Qwen3-8B-Base (Yang et al., 2025), across six in-distribution mathematical reasoning benchmarks, including MATH500, Minerva, AIME24/25, AMC24, and OlympiadBench. GRPO-DBB consistently outperforms naive GRPO and other baselines across all benchmarks and model sizes. Specifically, compared to GRPO, GRPO-DBB achieves average Acc@8 improvements of 3.22 and 2.42 points on Qwen3-1.7B-Base and Qwen3-8B-Base, respectively. Furthermore, on three out-of-distribution reasoning benchmarks such as MMLU-Pro, GPQA-Diamond, and Big-Bench Hard, GRPO-DBB yields average Acc@8 gains of 12.49 and 6.92 points over GRPO on 1.7B and 8B models, respectively. 2 Preliminaries We briefly introduce RLVR and its commonly used baseline, GRPO, to establish the foundation for our method. 2.1 Reinforcement Learning with Verifiable Rewards Given a prompt q∼q from the training dataset and a policy πθ _θ, a generated response is o∼πθ(⋅∣q)o _θ(· q). In RLVR, the reward function r(⋅)r(·) is typically defined as a binary signal indicating whether the response contains a correct answer. r(o,q)=1,if o contains the answer of q0,otherwise.r(o,q)= cases1,&if $o$ contains the answer of $q$\\ 0,&otherwise. cases (1) 2.2 Group Relative Policy Optimization (GRPO) GRPO (Shao et al., 2024) is a foundational baseline for RLVR that eliminates the need for an explicit value model and Generalized Advantage Estimation (Schulman et al., 2017). Instead, it relies on group-relative normalization to estimate advantages. For a given prompt q, GRPO samples N independent responses oii=1N\o_i\_i=1^N from the old policy πθold _ _old and computes advantages by normalizing the corresponding rewards r(oi,q)i=1N\r(o_i,q)\_i=1^N within the group: A^i=r(oi,q)−μσ, A_i= r(o_i,q)-μσ, (2) where μ=∑i=1Nr(oi,q)Nμ= _i=1^Nr(o_i,q)N and σ2=∑i=1N(r(oi,q)−μ)2N−1σ^2= _i=1^N (r(o_i,q)-μ )^2N-1. The resulting advantage A^i A_i is broadcast to all tokens in the response, i.e., A^i,t=A^i A_i,t= A_i for all token positions t. The policy is then updated by maximizing a clipped surrogate objective based on estimated advantages: GRPO(θ)=q∼,oii=1N∼πθold(⋅∣q)[1N∑i=1N1|oi|∑t=1|oi| _GRPO(θ)=E_q ,\,\o_i\_i=1^N _ _old(· q) [ 1N _i=1^N 1|o_i| _t=1^|o_i| (3) min(wi,t(θ)A^i,t,clip(wi,t(θ),1−ϵ,1+ϵ)A^i,t)], (w_i,t(θ)\, A_i,t,clip\! (w_i,t(θ),1-ε,1+ε )\, A_i,t ) ], where ϵε is the clipping hyperparameter and wi,t(θ)=πθ(oi,t∣q,oi,<t)πθold(oi,t∣q,oi,<t)w_i,t(θ)= _θ(o_i,t q,o_i,<t) _ _old(o_i,t q,o_i,<t) denotes the per-token importance weight. Following DAPO (Yu et al., 2025a), we omit the KL divergence term between the online and reference policies, as it may restrict exploration. 3 Method Many group-based RLVR methods (Shao et al., 2024; Liu et al., 2025; Xie et al., 2025; Le et al., 2025) primarily focus on the design of advantage estimators, while paying comparatively little attention to the more fundamental problem of reward estimation that underlies advantage computation. In practice, computational and memory constraints limit the number of sampled responses per prompt, making accurate reward estimation inherently difficult. In contrast, we introduce a new perspective that reformulates RLVR through the lens of statistical estimation (Section 3.1). Specifically, we view the reward not as a deterministic signal, but as a distribution induced by the policy, and frame advantage computation as a problem of estimating this distribution from finite samples. Building on this reformulation, we propose Discounted Beta–Bernoulli (DBB) reward estimation for RLVR, which estimates the non-stationary reward distribution by leveraging historical rewards (Section 3.2). Despite introducing bias, DBB reduces estimator variance and achieves lower MSE. Moreover, it fundamentally prevents variance collapse and preserves informative training signals (Section 3.3). 3.1 Reward Estimation as Distributional Inference For a given prompt q∼q and training step η with corresponding epoch τ, each response generated by the policy πθη,old _ _η,old produces a binary reward indicating whether the response is correct. Under our formulation, this reward is naturally modeled as a Bernoulli random variable Xτ,i∼Bernoulli(pτ),X_τ,i \! (p_τ ), (4) where pτ=ℙ(Xτ,i=1∣q,πθη,old)p_τ=P\! (X_τ,i=1 q, _ _η,old ) denotes the probability of obtaining a correct response. Within this framework, group-based RLVR algorithms such as GRPO and Dr.GRPO can be interpreted as implicitly estimating the reward distribution from a finite set of rollout samples. Given N observed rewards Xτ,ii=1N\X_τ,i\_i=1^N, existing approaches rely on point estimation of pτp_τ via the empirical mean p^τpt=1N∑i=1NXτ,i. p_τ^pt= 1N _i=1^NX_τ,i. (5) The corresponding reward variance is then estimated using the sample variance, Var^pt(Xτ)=∑i=1N(Xτ,i−p^τpt)2N−1=N⋅p^τpt(1−p^τpt)N−1. Var^pt(X_τ)=\; _i=1^N (X_τ,i- p_τ^pt )^2N-1= N· p_τ^pt (1- p_τ^pt )N-1\,. (6) Since the estimated variance Var^pt Var^pt is determined by the point estimator (mean) p^τpt p_τ^pt, it suffices to consider only the point estimator in our analysis. Concretely, the point estimator p^τpt p_τ^pt has expectation and variance as follows: [p^τpt∣pτ]=pτ,E[ p_τ^pt p_τ]=p_τ, (7) Var(p^τpt∣pτ)=pτ(1−pτ)N.Var( p_τ^pt p_τ)= p_τ(1-p_τ)N. (8) While the point estimator is unbiased, its variance grows as the number of rollouts N decreases. As a result, individual rollout outcomes can exert disproportionate influence on the estimate, causing the estimator to fluctuate widely. Moreover, the estimated variance can collapse to zero when all sampled rewards are identical. Since group-relative methods compute advantages directly from these reward estimates, such estimation noise naturally propagates to the advantage signal, leading to unstable policy updates. 3.2 Discounted Beta–Bernoulli Reward Estimation To estimate the Bernoulli reward distribution beyond empirical averaging, we adopt a well-established Bayesian perspective that leverages the conjugacy between the Beta prior and the Bernoulli likelihood. In this setting, the reward probability is modeled using a Beta prior and updated via a Beta posterior after observing rollout outcomes, leading to the naive Beta–Bernoulli model defined below. Definition 1 (Beta–Bernoulli Reward Model). To model uncertainty in the Bernoulli reward distribution, we place a Beta prior over the reward probability: pτ∼Beta(ατ,βτ),p_τ ( _τ, _τ), (9) where (ατ,βτ)( _τ, _τ) denote the posterior parameters for a prompt at epoch τ, with initialization α0=β0=1 _0= _0=1. Given N rollouts with SτS_τ successes, the posterior distribution is given by: ατ=ατ−1+Sτ,βτ=βτ−1+N−Sτ. _τ= _τ-1+S_τ, _τ= _τ-1+N-S_τ. (10) However, the above model assumes a stationary reward distribution. In RLVR, this assumption does not hold: due to the on-policy nature of training, the policy πθ _θ evolves over time, inducing a non-stationary reward distribution. As a result, historical observations may become outdated and should not be weighted equally with recent rollouts. Algorithm 1 Discounted Beta–Bernoulli Reward Estimation 0: LLM policy πθ _θ, training dataset D 1: Initialize (α0q,β0q)←(1,1)∀q∈(α^q_0,β^q_0)←(1,1) ∀ q 2: for epochτ=1,…,Iepoch\ τ=1,…,I do 3: for iteration=1,…,Miteration=1,…,M do 4: Sample a minibatch b⊂D_b 5: Set old policy πθold←πθ _ _old← _θ 6: for each prompt q∈bq _b do 7: Sample outputs oii=1N∼i.i.d.πθold(⋅∣q)\o_i\_i=1^N i.i.d. _ _old(· q) 8: Compute rewards Xi=r(oi,q)i=1N\X_i=r(o_i,q)\_i=1^N 9: Update posterior parameters: 10: ατq←λατ−1q+∑i=1NXi,βτq←λβτ−1q+N−∑i=1NXi aligned α^q_τ&←λα^q_τ-1+ _i=1^NX_i,\\ β^q_τ&←λβ^q_τ-1+N- _i=1^NX_i aligned 11: Compute advantage A^i A_i with (ατq,βτq)(α^q_τ,β^q_τ) and XiX_i 12: end for 13: for update=1,…,Uupdate=1,…,U do 14: Update πθ _θ by maximizing the objective (Eq. 3) 15: end for 16: end for 17: end for To address this challenge, we introduce the Discounted Beta–Bernoulli (DBB) reward model, which gradually discounts past information to estimate the reward distribution accurately. Definition 2 (Discounted Beta–Bernoulli Reward Model). Given posterior parameters (ατ−1,βτ−1)( _τ-1, _τ-1) and N rollouts with SτS_τ successes at epoch τ, the update is defined as ατ=λατ−1+Sτ,βτ=λβτ−1+N−Sτ, _τ=λ _τ-1+S_τ, _τ=λ _τ-1+N-S_τ, (11) where the discount factor λ∈(0,1]λ∈(0,1] controls the influence of historical observations. Under the DBB reward model, we estimate the mean and variance of the Bernoulli reward distribution as follows: p^τdbb=ατατ+βτ, p_τ^dbb= _τ _τ+ _τ, (12) Var^dbb(Xτ∣ατ,βτ)=ατβτ(ατ+βτ)2. Var^dbb(X_τ _τ, _τ)= _τ _τ( _τ+ _τ)^2. (13) To understand the statistical behavior of the DBB estimator, we analyze its expectation and variance: [p^τdbb∣pτ,ατ−1,βτ−1]=wμτ−1+(1−w)pτ,E\! [ p_τ^dbb p_τ, _τ-1, _τ-1 ]=w\, _τ-1+(1-w)\,p_τ, (14) Var(p^τdbb∣pτ,ατ−1,βτ−1)=(1−w)2pτ(1−pτ)N,Var\! ( p_τ^dbb p_τ, _τ-1, _τ-1 )=(1-w)^2\, p_τ(1-p_τ)N, (15) where μτ−1=ατ−1ατ−1+βτ−1 _τ-1= _τ-1 _τ-1+ _τ-1 denotes the historical posterior mean and w=λ(ατ−1+βτ−1)λ(ατ−1+βτ−1)+Nw= λ( _τ-1+ _τ-1)λ( _τ-1+ _τ-1)+N controls the contribution of past observations. These show that the DBB estimator introduces bias through shrinkage toward the historical mean, but substantially reduces variance relative to the point estimator (Equation 8). Importantly, unlike estimated variance based on small-sample rollouts, the posterior variance in DBB cannot collapse to zero, ensuring stable and informative reward signals throughout training. Algorithm 1 summarizes the overall training procedure of the group-based RLVR algorithm with DBB. 3.3 Mean Squared Error of the DBB estimator To assess the effectiveness of our DBB estimator in predicting the reward distribution, we compute its mean squared error (MSE). To make the estimator’s dependence on the underlying reward probabilities p1,…,pτ\p_1,…,p_τ\, we re-express its expectation and variance, originally formulated in terms of the posterior parameters (ατ−1,βτ−1)( _τ-1, _τ-1). Based on the update defined in Equation 11, posterior parameters can be expressed as ατ=λτα0+∑k=1τλτ−kSk,βτ=λτβ0+∑k=1τλτ−k(N−Sk). _τ=λ^τ _0+ _k=1^τλ^τ-kS_k,\; _τ=λ^τ _0+ _k=1^τλ^τ-k(N-S_k). For convenience, define the sum of the posterior parameters as HτH_τ: Hτ=ατ+βτ=λτ(α0+β0)+N∑k=1τλτ−k.H_τ= _τ+ _τ=λ^τ( _0+ _0)+N _k=1^τλ^τ-k. (16) Under our formulation (Equation 4), the number of successful rollouts at epoch k satisfies Sk∼Binomial(N,pk)S_k (N,p_k). Given this, the expectation and variance of the DBB estimator can be written as follows: [p^τdbb∣p1:τ]=∑k=0τckpk,E\! [ p_τ^dbb p_1:τ ]= _k=0^τc_kp_k, (17) Var(p^τdbb∣p1:τ)=∑k=1τλ2(τ−k)Npk(1−pk)Hτ2,Var\! ( p_τ^dbb p_1:τ )= _k=1^τλ^2(τ-k)Np_k(1-p_k)H_τ^2, (18) where p0=α0α0+β0p_0= _0 _0+ _0 is the reward probability of the initial prior, and the weights c0=λτ(α0+β0)Hτc_0= λ^τ( _0+ _0)H_τ and ck=Nλτ−kHτc_k= Nλ^τ-kH_τ are constants. A detailed derivation is provided in Appendix B. Combining Equations (17) and (18), the MSE of the DBB estimator under non-stationary rewards is MSE(p^τdbb∣p1:τ)=([p^τdbb∣p1:τ]−pτ)2+Var(p^τdbb∣p1:τ).MSE\! ( p_τ^dbb p_1:τ )= (E\! [ p_τ^dbb p_1:τ ]-p_τ )^2+Var\! ( p_τ^dbb p_1:τ ). (19) In contrast, the MSE of the point estimator depends only on the current reward distribution MSE(p^τpt∣pτ)=pτ(1−pτ)N,MSE\! ( p_τ^pt p_τ )= p_τ(1-p_τ)N, (20) and is therefore more prone to high variance, especially in low-sample settings. The bias of the DBB estimator becomes smaller when the underlying reward probabilities evolve gradually, as past values pkk<τ\p_k\_k<τ remain close to the current pτp_τ. In contrast, the variance depends on both the discount factor λ and the number of rollouts N. On the other hand, the point estimator relies solely on pτp_τ and is therefore more susceptible to high variance, particularly in low-sample settings. Considering these factors, we empirically evaluate the resulting MSE and present the findings in Section 5.2. 4 Experiments We empirically demonstrate that HBB estimation is effective in the RLVR setting. Section 4.1 describes the models, datasets, baselines, and detailed setup used in our experiments, while Section 4.2 presents the experimental results. 4.1 Experimental Settings Models & Datasets. We conduct experiments on two model sizes, Qwen3-1.7B-Base and Qwen3-8B-Base (Yang et al., 2025), to investigate the effect of model scales. For both model scales, we use DAPO-Math-17k (Yu et al., 2025a) as the training dataset. As in-distribution evaluation benchmarks for mathematical reasoning, we consider six widely used datasets: MATH500 (Hendrycks et al., 2021), Minerva (Lewkowycz et al., 2022), AIME24/25, AMC24 (Li et al., 2024), and OlympiadBench (Olympiad; He et al., 2024). To further assess out-of-distribution generalization, we evaluate performance on MMLU-Pro (Wang et al., 2024), GPQA-Diamond (GPQA-D; Rein et al., 2023), and Big-Bench Hard (BBH; Suzgun et al., 2022). Baselines. The DBB reward estimation aims to improve reward distribution estimation by partially leveraging historical information. To isolate the effect of DBB, we first compare GRPO with the DBB estimator (GRPO-DBB) and naive GRPO, which relies on point estimation. In addition, to compare against methods that more fully exploit historical information, we include RePO as a representative method in this category. Training & Evaluation Setups. All training experiments are conducted using the verl111https://github.com/verl-project/verl framework, and Math-Verify222https://github.com/huggingface/Math-Verify is employed to extract and verify final answers. Experiments with Qwen3-1.7B-Base and Qwen3-8B-Base are conducted on 4×H200 and 8×H200 GPUs, respectively, except for RePO. Due to the requirement for additional GPU memory, RePO is trained on 8×H200 GPUs for both model scales. The rollout batch size is set to 128, and the gradient update batch size is set to 64. We sample 8 responses per query from the on-policy model and train the model for a total of four epochs. For evaluation, we sample 8 responses per query for all benchmarks and report Avg@8 as the evaluation metric. The training setup details for each baseline and the sampling parameters used for validation and evaluation are also provided in Appendix C. 4.2 Main Results Table 1: In-distribution (ID) and out-of-distribution (OOD) evaluation results trained on DAPO-Math-17k. GRPO with the DBB process (GRPO-DBB) consistently outperforms naive GRPO across all benchmarks and model scales. Δ denotes the absolute Acc@8 improvement of GRPO-DBB over GRPO. In-Distribution Out-of-Distribution Method MATH500 Minerva AIME24 AIME25 AMC24 Olympiad Avg. MMLU-Pro GPQA-D BBH Avg. Qwen3-8B-Base trained with DAPO-Math-17k GRPO 88.05 39.25 30.00 26.67 56.39 54.82 49.20 50.02 42.93 59.18 50.71 RePO 86.20 35.71 27.92 22.92 56.11 52.61 46.91 53.15 43.93 42.96 46.68 GRPO-DBB 88.92 39.48 34.17 30.83 60.00 56.34 51.62 63.12 46.46 63.32 57.63 Δ w.r.t. GRPO +0.87 +0.23 +4.17 +4.16 +3.61 +1.52 +2.42 +13.10 +3.53 +4.14 +6.92 Qwen3-1.7B-Base trained with DAPO-Math-17k GRPO 68.37 25.74 8.75 4.58 23.61 29.41 26.74 33.72 20.70 12.71 22.38 RePO 67.87 23.76 6.67 6.67 21.67 29.17 25.97 35.49 26.26 30.95 30.90 GRPO-DBB 71.95 26.31 14.17 7.08 26.94 33.29 29.96 40.83 29.80 33.97 34.87 Δ w.r.t. GRPO +3.58 +0.57 +5.42 +2.50 +3.33 +3.88 +3.22 +7.11 +9.10 +21.26 +12.49 Table 1 reports Acc@8 results across six mathematical reasoning in-distribution benchmarks and three general reasoning out-of-distribution (OOD) benchmarks for three different RLVR algorithms. For GRPO-DBB, the discounting factor λ is set to 0.50.5 for Qwen3-1.7B-Base and 0.750.75 for Qwen3-8B-Base; these values are empirically selected based on performance trends under varying λ. Further details are provided in Section 5.2. For in-distribution evaluation, GRPO-DBB consistently outperforms both GRPO and RePO across all benchmarks and model scales. Compared to GRPO, Acc@8 improves by an average of 3.22 points for Qwen3-1.7B-Base and 2.42 points for Qwen3-8B-Base. This indicates that, in the RLVR setting, replacing GRPO’s point estimator with the DBB estimator is more effective for in-distribution downstream tasks. When compared to RePO, GRPO-DBB achieves improvements of 3.99 and 4.71 points for Qwen3-1.7B-Base and Qwen3-8B-Base, respectively. Although RePO leverages diverse forms of historical information, including tokens, token probabilities, and rewards, GRPO-DBB relies solely on rewards. Despite this, its strong downstream performance suggests that the DBB estimator utilizes historical information both efficiently and effectively. For out-of-distribution evaluation, GRPO-DBB consistently outperforms both baselines across all OOD benchmarks and model scales. In particular, it achieves substantial gains over GRPO, with improvements of 12.49 and 6.92 points for Qwen3-1.7B-Base and Qwen3-8B-Base, respectively. These results demonstrate that the DBB estimation provides a strong advantage in improving generalization, which can be attributed to the characteristic of the Bayesian approach to account for uncertainty. Table 2: Effect of the discount factor λ of GRPO-DBB on in-distribution Acc@8 performance. Qwen3-8B-Base achieves the best performance at λ=0.75λ=0.75, while Qwen3-1.7B-Base performs best at λ=0.5λ=0.5. Method MATH500 Minerva AIME24 AIME25 AMC24 Olympiad Avg. Qwen3-8B-Base trained with DAPO 17k λ=1.0λ=1.0 89.08 37.91 33.33 26.25 59.44 55.19 50.20 λ=0.75λ=0.75 88.92 39.48 34.17 30.83 60.00 56.34 51.62 λ=0.5λ=0.5 89.35 38.56 33.33 27.08 59.17 56.19 50.61 λ=0.25λ=0.25 88.82 39.98 30.83 25.83 60.83 56.34 50.44 GRPO 88.05 39.25 30.00 26.67 56.39 54.82 49.20 Qwen3-1.7B-Base trained with DAPO 17k λ=1.0λ=1.0 67.30 25.55 9.58 3.33 23.33 28.65 26.29 λ=0.75λ=0.75 72.28 25.60 11.25 7.50 26.39 32.68 29.28 λ=0.5λ=0.5 71.95 26.31 14.17 7.08 26.94 33.29 29.96 λ=0.25λ=0.25 72.22 25.97 14.17 6.25 22.50 32.88 29.00 GRPO 68.37 25.74 8.75 4.58 23.61 29.41 26.74 5 Analysis & Discussion In this section, we provide an in-depth analysis of the training behavior and estimation properties of GRPO with the DBB reward estimator. We first examine training dynamics to understand how DBB influences optimization stability, exploration behavior, and reward progression over time (Section 5.1). We then conduct ablation studies on the discount factor λ to clarify its role in adapting to different learning regimes (Section 5.2). Next, we empirically analyze the mean squared error (MSE) of DBB relative to point estimation, connecting estimation accuracy to downstream performance (Section 5.3). Finally, we demonstrate that DBB can be readily integrated into alternative advantage formulations, highlighting its generality beyond GRPO (Section 5.4). 5.1 Training Dynamics Figure 2: Training dynamics of naive GRPO and GRPO with the DBB estimator (GRPO-DBB) on Qwen3-1.7B-Base (top) and Qwen3-8B-Base (bottom). GRPO-DBB achieves higher validation Acc@8 and training rewards, while maintaining longer responses with controlled entropy compared to GRPO, indicating more stable exploration during training. Figure 2 illustrates the validation accuracy and training dynamics of GRPO-DBB and GRPO during training. For GRPO-DBB, the discounting factor λ is set to 0.50.5 for Qwen3-1.7B-Base and 0.750.75 for Qwen3-8B-Base. For both model scales, GRPO-DBB achieves higher validation accuracy and training rewards than GRPO throughout most of the training process. Examining the response length over the course of training, we observe that GRPO-DBB consistently produces longer responses than GRPO throughout training. In contrast, the entropy under GRPO grows rapidly, indicating increasingly unstable exploration behavior. Taken together, these observations suggest that GRPO-DBB attains higher rewards by pruning the breadth of response trajectories while allowing exploration to proceed along longer trajectories, corresponding to depth-wise exploration. 5.2 Ablation Study on λ To examine the effect of the discount factor λ in the DBB estimation, we conduct training experiments of GRPO-DBB with λ=1.0,0.75,0.5,λ=1.0,0.75,0.5, and 0.250.25 for both model scales. The hyperparameter λ controls the extent to which the estimation of the Bernoulli parameter pτp_τ for a given query at epoch τ depends on historical rollout statistics, namely ατ−1 _τ-1 and βτ−1 _τ-1. When the learning dynamics are fast and the discrepancy between pτp_τ and pτ−1p_τ-1 is large, a smaller λ is generally more effective, as it places greater emphasis on recent observations and reduces estimation bias. Conversely, when the learning dynamics are slower and more stationary, a larger λ tends to yield better performance by incorporating more historical information and thereby reducing estimation variance. This behavior can be interpreted in conjunction with the training reward dynamics shown in Figure 2. For Qwen3-1.7B-Base, the training reward exhibits sustained changes throughout the training process, indicating continuously evolving estimates of pτp_τ. In contrast, for Qwen3-8B-Base, the training reward increases rapidly during the very early stage of training (within approximately the first 0.3 epoch) and remains relatively stationary for the remainder of training. As a result, a smaller discounting factor (λ=0.5λ=0.5) is more effective for Qwen3-1.7B-Base, while a larger value (λ=0.75λ=0.75) better matches the slower and more gradual learning dynamics of Qwen3-8B-Base. Figure 3: MSE as a function of the discount factor λ and the number of rollouts N. The DBB estimator yields lower MSE than the point estimator across a wide range of λ, and it achieves lower MSE for rollout budgets up to N=16N=16 when λ=0.4λ=0.4. 5.3 Analysis of MSE To evaluate how accurately the point estimator p^τpt p_τ^pt and the DBB estimator p^τdb p_τ^db approximate the Bernoulli parameter pτp_τ during training, we conduct an empirical estimation error analysis. Since obtaining the exact ground-truth value of pτp_τ would require infinitely many samples, we instead approximate it using the empirical mean of rewards from a finite number of rollouts. During GRPO training of Qwen3-1.7B-Base, when a randomly selected 10% subset of training prompts τQ_τ appears in a rollout batch, we additionally sample 128 responses for each such prompt using the policy at that training step. These additional samples are used exclusively for analysis and are not incorporated into training. We then define the empirical mean of the resulting rewards as the reference value p~τ(q) p_τ(q). p~τ(q)≜1128∑i=1128Xτ,i, p_τ(q)\; \; 1128 _i=1^128X_τ,i, (21) where oτ,i∼πθτ(⋅∣q)o_τ,i _ _τ(· q) and Xτ,i=r(oτ,i,q).X_τ,i=r(o_τ,i,q). Since training is conducted for a total of four epochs, this procedure yields a sequence of reference values p~τ(q)τ=14\ p_τ(q)\_τ=1^4 for each query. Using p~τ(q) p_τ(q) as a surrogate for the true Bernoulli parameter, we measure the average MSE of the DBB estimator and the point estimator following Equation (19) and (20), respectively. Effect of λ on MSE Since the value of the DBB estimator depends on the discount factor λ, we empirically analyze how the MSE varies as a function of λ. As shown in Figure 3(a), the MSE attains its minimum at λ=0.4λ=0.4, and for most values of λ—except for λ=0.95λ=0.95 and λ=1.0λ=1.0—the DBB estimator yields a lower average MSE than the point estimator. This trend is closely aligned with downstream performance. As reported in Table 2, GRPO-DBB achieves the best performance at λ=0.5λ=0.5, while at λ=1.0λ=1.0 it performs slightly worse than GRPO with the point estimator. These observations mirror the behavior of the MSE across different values of λ. Taken together, the results indicate that more accurate reward distribution estimation is strongly correlated with improved RLVR performance. Effect of N on MSE We further examine how the MSE varies as the number of rollouts N increases. Figure 2(b) reports the average MSE as a function of N when λ=0.4λ=0.4. The results show that the DBB estimator consistently outperforms the point estimator for N up to 16. This suggests that, in regimes where computational resource constraints necessitate small rollout budgets, the DBB estimator can serve as an effective alternative to point estimation. 5.4 Evaluation on Alternative Advantage Formulations Table 3: In-distribution evaluation results on Qwen3-1.7B-Base trained with DAPO-MATH-17k. We apply the proposed discounted Beta–Bernoulli reward distribution estimation to the advantage term of Dr.GRPO and observe consistent performance improvements. Δ denotes the absolute Acc@8 improvement over naive Dr.GRPO. Method MATH500 Minerva AIME24 AIME25 AMC24 Olympiad Avg. Qwen3-1.7B-Base trained with DAPO-Math-17k Dr.GRPO 69.00 25.14 9.17 7.92 23.89 32.78 27.98 Dr.GRPO-DBB 70.60 26.15 12.08 9.17 28.06 33.59 29.94 Δ w.r.t. Dr.GRPO +1.60 +1.01 +2.92 +1.25 +4.17 +0.81 +1.96 To examine whether the DBB estimation can be applied to other advantage formulations, we conduct experiments using the advantage term of Dr.GRPO, defined as Ai=Xi−[p^τ]A_i=X_i-E[ p_τ]. All experiments are conducted on Qwen3-1.7B-Base with the discounting factor set to λ=0.5λ=0.5 for the Dr.GRPO with the DBB estimation (Dr.GRPO-DBB). As shown in Table 3, Dr.GRPO-DBB consistently improves performance over the naive Dr.GRPO across all in-distribution benchmarks, yielding an average Acc@8 gain of 1.96 points. These results suggest that the DBB estimation is not limited to the advantage term of GRPO, but can be broadly applied to other RLVR algorithms that rely on point estimation, yielding effective and consistent performance gains. 6 Related Work Reinforcement Learning with Verifiable Rewards. Reinforcement learning with verifiable rewards (RLVR) has emerged as an effective post-training paradigm for improving the reasoning capabilities of large language models by leveraging automatically verifiable reward signals (Lambert et al., 2025; Plaat et al., 2025). In particular, group-based RLVR algorithms, including GRPO and its variants (Shao et al., 2024; Yu et al., 2025a; Liu et al., 2025; Zheng et al., 2025a; Zhao et al., 2025), have demonstrated strong performance on reasoning benchmarks without relying on explicit value models (Yu et al., 2025b). Variance Collapse in Group-Based RLVR. A well-known limitation of group-based RLVR methods is variance collapse, where the estimated reward variance becomes zero when all sampled responses receive identical rewards, eliminating the training signal and leading to ineffective policy updates (Yu et al., 2025a; Zheng et al., 2025b; Le et al., 2025; Zhang et al., 2025). Prior approaches such as Dynamic sAmpling Policy Optimization (DAPO; Yu et al., 2025a) and GRESO (Zheng et al., 2025b) mitigate this issue by modifying the rollout or sampling strategy. However, these methods either incur substantial additional computational cost or fail to fundamentally eliminate variance collapse due to their probabilistic or heuristic nature. In contrast, our work addresses variance collapse at the reward estimator level by explicitly modeling uncertainty in the reward distribution. By maintaining a Beta posterior with strictly positive parameters, the DBB process theoretically prevents variance collapse without additional rollouts or changes to the sampling strategy. Replay- and History-Based RLVR Methods. Replay-based methods aim to improve sample efficiency in RLVR by reusing historical rollouts, as exemplified by RePO (Li et al., 2025) and ExGRPO (Zhan et al., 2025). While potentially effective, these approaches require storing tokens and their probabilities under historical policies and performing additional forward passes, introducing non-trivial memory and computational overhead. Our approach differs in that it leverages only historical reward statistics rather than full trajectory information, eliminating the need for additional GPU memory or extra forward passes. Empirical results demonstrate that reward signals alone can effectively capture useful historical information for improving RLVR performance. 7 Conclusion We revisited RLVR from the perspective of reward distribution estimation and identified point estimation under limited rollouts as a key source of sample inefficiency and variance collapse. To this end, we proposed Discounted Beta–Bernoulli (DBB) reward estimation for RLVR, which leverages historical reward information for accurate estimation. Although biased, the DBB estimator achieves substantially lower variance and mean squared error, theoretically avoids variance collapse, and preserves informative training signals without additional rollouts or replay. Experiments across two model scales demonstrate consistent improvements over baselines on all in-distribution and out-of-distribution benchmarks. Our findings highlight reward distribution estimation as a critical yet underexplored component of effective RLVR and suggest that principled estimator design offers a promising direction for improving large-scale reinforcement learning of language models. As future work, we plan to design a dynamic discount factor λ that adapts across different stages of training. Impact Statements This paper presents work whose goal is to advance the field of machine learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here. References G. Casella and R. Berger (2024) Statistical inference. Chapman and Hall/CRC. Cited by: §1. A. Gelman, J. B. Carlin, H. S. Stern, and D. B. Rubin (1995) Bayesian data analysis. Chapman and Hall/CRC. Cited by: §1. D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. (2025) Deepseek-r1: incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. Cited by: §1. C. He, R. Luo, Y. Bai, S. Hu, Z. L. Thai, J. Shen, J. Hu, X. Han, Y. Huang, Y. Zhang, J. Liu, L. Qi, Z. Liu, and M. Sun (2024) OlympiadBench: a challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. External Links: 2402.14008, Link Cited by: §4.1. D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt (2021) Measuring mathematical problem solving with the math dataset. External Links: 2103.03874, Link Cited by: §4.1. N. Lambert, J. Morrison, V. Pyatkin, S. Huang, H. Ivison, F. Brahman, L. J. V. Miranda, A. Liu, N. Dziri, S. Lyu, Y. Gu, S. Malik, V. Graf, J. D. Hwang, J. Yang, R. L. Bras, O. Tafjord, C. Wilhelm, L. Soldaini, N. A. Smith, Y. Wang, P. Dasigi, and H. Hajishirzi (2025) Tulu 3: pushing frontiers in open language model post-training. External Links: 2411.15124, Link Cited by: §1, §6. T. V. Le, M. Jeon, K. Vu, V. Lai, and E. Yang (2025) No prompt left behind: exploiting zero-variance prompts in llm reinforcement learning via entropy-guided advantage shaping. External Links: 2509.21880, Link Cited by: §1, §3, §6. A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ramasesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, Y. Wu, B. Neyshabur, G. Gur-Ari, and V. Misra (2022) Solving quantitative reasoning problems with language models. External Links: 2206.14858, Link Cited by: §4.1. J. Li, E. Beeching, L. Tunstall, B. Lipkin, R. Soletskyi, S. Huang, K. Rasul, L. Yu, A. Q. Jiang, Z. Shen, et al. (2024) Numinamath: the largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository 13 (9), p. 9. Cited by: §4.1. S. Li, Z. Zhou, W. Lam, C. Yang, and C. Lu (2025) RePO: replay-enhanced policy optimization. External Links: 2506.09340, Link Cited by: §1, §6. Z. Liu, C. Chen, W. Li, P. Qi, T. Pang, C. Du, W. S. Lee, and M. Lin (2025) Understanding r1-zero-like training: a critical perspective. External Links: 2503.20783, Link Cited by: §1, §3, §6. A. Plaat, A. Wong, S. Verberne, J. Broekens, N. van Stein, and T. Back (2025) Multi-step reasoning with large language models, a survey. External Links: 2407.11511, Link Cited by: §1, §6. D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman (2023) GPQA: a graduate-level google-proof q&a benchmark. External Links: 2311.12022, Link Cited by: §4.1. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal policy optimization algorithms. External Links: 1707.06347, Link Cited by: §2.2. Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. (2024) Deepseekmath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: §1, §2.2, §3, §6. M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, and J. Wei (2022) Challenging big-bench tasks and whether chain-of-thought can solve them. External Links: 2210.09261, Link Cited by: §4.1. Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang, T. Li, M. Ku, K. Wang, A. Zhuang, R. Fan, X. Yue, and W. Chen (2024) MMLU-pro: a more robust and challenging multi-task language understanding benchmark. External Links: 2406.01574, Link Cited by: §4.1. C. Xie, R. Pan, X. Wu, Y. Zhang, J. Fu, T. Gao, and G. Zhou (2025) Unlocking exploration in rlvr: uncertainty-aware advantage shaping for deeper reasoning. arXiv preprint arXiv:2510.10649. Cited by: §3. A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. Li, T. Tang, W. Yin, X. Ren, X. Wang, X. Zhang, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Zhang, Y. Wan, Y. Liu, Z. Wang, Z. Cui, Z. Zhang, Z. Zhou, and Z. Qiu (2025) Qwen3 technical report. External Links: 2505.09388, Link Cited by: §1, §1, §4.1. Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, W. Dai, T. Fan, G. Liu, L. Liu, X. Liu, H. Lin, Z. Lin, B. Ma, G. Sheng, Y. Tong, C. Zhang, M. Zhang, W. Zhang, H. Zhu, J. Zhu, J. Chen, J. Chen, C. Wang, H. Yu, Y. Song, X. Wei, H. Zhou, J. Liu, W. Ma, Y. Zhang, L. Yan, M. Qiao, Y. Wu, and M. Wang (2025a) DAPO: an open-source llm reinforcement learning system at scale. External Links: 2503.14476, Link Cited by: §1, §1, §2.2, §4.1, §6, §6. T. Yu, B. Ji, S. Wang, S. Yao, Z. Wang, G. Cui, L. Yuan, N. Ding, Y. Yao, Z. Liu, M. Sun, and T. Chua (2025b) RLPR: extrapolating rlvr to general domains without verifiers. External Links: 2506.18254, Link Cited by: §1, §6. R. Zhan, Y. Li, Z. Wang, X. Qu, D. Liu, J. Shao, D. F. Wong, and Y. Cheng (2025) ExGRPO: learning to reason from experience. External Links: 2510.02245, Link Cited by: §1, §6. Y. Zhang, W. Yao, C. Yu, Y. Liu, Q. Yin, B. Yin, H. Yun, and L. Li (2025) Improving sampling efficiency in rlvr through adaptive rollout and response reuse. arXiv preprint arXiv:2509.25808. Cited by: §1, §6. Y. Zhao, Y. Liu, J. Liu, J. Chen, X. Wu, Y. Hao, T. Lv, S. Huang, L. Cui, Q. Ye, F. Wan, and F. Wei (2025) Geometric-mean policy optimization. External Links: 2507.20673, Link Cited by: §1, §6. C. Zheng, S. Liu, M. Li, X. Chen, B. Yu, C. Gao, K. Dang, Y. Liu, R. Men, A. Yang, J. Zhou, and J. Lin (2025a) Group sequence policy optimization. External Links: 2507.18071, Link Cited by: §1, §6. H. Zheng, Y. Zhou, B. R. Bartoldson, B. Kailkhura, F. Lai, J. Zhao, and B. Chen (2025b) Act only when it pays: efficient reinforcement learning for llm reasoning via selective rollouts. External Links: 2506.02177, Link Cited by: §1, §6. Appendix A Comparison of Statistics Between Point Estimation and Discounted Beta–Bernoulli Estimation We present Table 4 that summarizes the statistics of point estimation and Discounted Beta–Bernoulli estimation, as discussed in Sections 3.1 and 3.2, in a concise and accessible manner. Table 4: Comparison between point estimation and DBB estimation for reward distribution modeling. We denote the Bernoulli distribution by Bern(⋅)Bern(·). Point Estimation DBB Estimation Xτ,iX_τ,i Xτ,i∼Bern(pτ)X_τ,i (p_τ) Xτ,i∼Bern(pτ)X_τ,i (p_τ) pτp_τ fixed but unknown pτ∼Beta(ατ,βτ)p_τ ( _τ, _τ) p^τ p_τ p^τpt=1N∑i=1NXτ,i p_τ^pt= 1N _i=1^NX_τ,i p^τdb=ατατ+βτ p_τ^db= _τ _τ+ _τ Var^(Xτ) Var(X_τ) N−1p^τpt(1−p^τpt) NN-1\, p_τ^pt\! (1- p_τ^pt ) p^τdb(1−p^τdb) p_τ^db\! (1- p_τ^db ) [p^τ]E[ p_τ] pτp_τ wμτ−1+(1−w)pτw _τ-1+(1-w)p_τ Bias(p^τ)Bias( p_τ) 0 w(μτ−1−pτ)w( _τ-1-p_τ) Var(p^τ)Var( p_τ) pτ(1−pτ)N p_τ(1-p_τ)N (1−w)2pτ(1−pτ)N (1-w)^2 p_τ(1-p_τ)N Appendix B Derivation of the Mean, Variance, and MSE of the DBB Estimator In this section, we provide a detailed derivation of Equations (17) and (18) in the main text. To avoid ambiguity caused by implicit conditioning, we re-express the analysis in terms of the conditional distribution given the sequence of true reward probabilities p1,p2,…,pτ\p_1,p_2,…,p_τ\. B.1 Expansion of Posterior Parameters under Historical Rewards At each training step k, rollout rewards satisfy Sk∣pk∼Binomial(N,pk),Xk,i∣pk∼Bernoulli(pk),S_k p_k (N,p_k), X_k,i p_k (p_k), (22) and the DBB updates are given by αk=λαk−1+Sk,βk=λβk−1+(N−Sk). _k=λ _k-1+S_k, _k=λ _k-1+(N-S_k). (23) Unrolling the recursion yields ατ=λτα0+∑k=1τλτ−kSk, _τ=λ^τ _0+ _k=1^τλ^τ-kS_k, (24) and the total mass Hτ≜ατ+βτ=λτ(α0+β0)+N∑k=1τλτ−k.H_τ\; \; _τ+ _τ=λ^τ( _0+ _0)+N _k=1^τλ^τ-k. (25) Crucially, when N and λ are fixed, HτH_τ is deterministic and does not depend on the random variables Skk=1τ\S_k\_k=1^τ. All stochasticity in the estimator arises from ατ _τ alone. B.2 Mean of the DBB Estimator The DBB estimator is defined as p^τdbb=ατHτ. p_τ^dbb= _τH_τ. (26) Since HτH_τ is deterministic, [p^τdbb∣p1:τ]=[ατ∣p1:τ]Hτ.E\! [ p_τ^dbb p_1:τ ]= E[ _τ p_1:τ]H_τ. (27) Using [Sk∣pk]=NpkE[S_k p_k]=Np_k, we obtain [ατ∣p1:τ]=λτα0+N∑k=1τλτ−kpk.E[ _τ p_1:τ]=λ^τ _0+N _k=1^τλ^τ-kp_k. (28) Therefore, [p^τdbb∣p1:τ]=λτα0+N∑k=1τλτ−kpkλτ(α0+β0)+N∑k=1τλτ−k E\! [ p_τ^dbb p_1:τ ]= λ^τ _0+N _k=1^τλ^τ-kp_kλ^τ( _0+ _0)+N _k=1^τλ^τ-k (29) For interpretability, define weights c0≜λτ(α0+β0)Hτ,ck≜Nλτ−kHτ,k=1,…,τ,p0≜α0α0+β0.c_0 λ^τ( _0+ _0)H_τ, c_k Nλ^τ-kH_τ, k=1,…,τ, p_0 _0 _0+ _0. (30) Then the estimator mean can be written as [p^τdbb∣p1:τ]=∑k=0τckpk,∑k=0τck=1, E\! [ p_τ^dbb p_1:τ ]= _k=0^τc_kp_k, _k=0^τc_k=1, (31) revealing an exponentially weighted average of historical reward probabilities. B.3 Variance of the DBB Estimator Again using the determinism of HτH_τ, Var(p^τdbb∣p1:τ)=Var(ατ∣p1:τ)Hτ2.Var\! ( p_τ^dbb p_1:τ )= Var( _τ p_1:τ)H_τ^2. (32) Since Sk\S_k\ are independent conditioned on pk\p_k\ and Var(Sk∣pk)=Npk(1−pk)Var(S_k p_k)=Np_k(1-p_k), Var(ατ∣p1:τ)=∑k=1τλ2(τ−k)Npk(1−pk).Var( _τ p_1:τ)= _k=1^τλ^2(τ-k)Np_k(1-p_k). (33) Hence, Var(p^τdbb∣p1:τ)=∑k=1τλ2(τ−k)Npk(1−pk)(λτ(α0+β0)+N∑k=1τλτ−k)2 Var\! ( p_τ^dbb p_1:τ )= _k=1^τλ^2(τ-k)Np_k(1-p_k) (λ^τ( _0+ _0)+N _k=1^τλ^τ-k )^2 (34) B.4 Bias and Mean Squared Error of the DBB Estimator By definition, MSE(p^τdbb∣p1:τ)=Bias2+Var.MSE\! ( p_τ^dbb p_1:τ )=Bias^2+Var. (35) Using Eq. (31), the conditional bias is Bias(p^τdbb∣p1:τ)=∑k=0τ−1ck(pk−pτ), Bias\! ( p_τ^dbb p_1:τ )= _k=0^τ-1c_k(p_k-p_τ), (36) where the k=τk=τ term vanishes since pτ−pτ=0p_τ-p_τ=0. Combining Eqs. (34) and (36), we obtain MSE(p^τdbb∣p1:τ)=(∑k=0τ−1ck(pk−pτ))2+∑k=1τλ2(τ−k)Npk(1−pk)Hτ2, MSE\! ( p_τ^dbb p_1:τ )= ( _k=0^τ-1c_k(p_k-p_τ) )^2+ _k=1^τλ^2(τ-k)Np_k(1-p_k)H_τ^2, (37) where Hτ=λτ(α0+β0)+N∑k=1τλτ−kH_τ=λ^τ( _0+ _0)+N _k=1^τλ^τ-k. Appendix C Implementation Details Table 5: Common Training hyperparameter settings for all experiments. Hyperparameter Qwen3-1.7B-Base Qwen3-8B-Base Training Configuration Training batch size 128 128 Mini-batch size 64 64 Number of epochs 4 4 Total gradient steps 1080 1080 Samples per prompt 8 8 Max response length 4096 8192 Sampling Configuration Training temperature 1.0 1.0 Training top-p 1.0 1.0 Validation temperature 0.6 0.6 Validation top-p 0.95 0.95 Optimization Optimizer AdamW AdamW Learning rate 1×10−61× 10^-6 1×10−61× 10^-6 LR warmup steps 0 0 LR scheduler constant constant In this section, we describe the experimental setup details for the RLVR algorithm with the DBB process (RLVR-DBB) and all baseline methods. All training experiments are conducted using the verl framework, and Math-Verify is employed to extract and normalize final answers from model responses. Training Qwen3-1.7B-Base is performed using 4×H200 GPUs, while Qwen3-8B-Base is trained using 8×H200 GPUs. Due to its substantially higher GPU memory requirements, RePO is trained using 8×H200 GPUs for both model scales. All training experiments are conducted using the DAPO-Math-17k dataset. For training stability, we filter prompts to a maximum length of 1024 tokens. In addition, to facilitate analysis, we further restrict the dataset so that its size is an integer multiple of the rollout batch size, which is 128. As a result, out of the original 17,391 prompts, 17,280 are used for training. The hyperparameters shared across all training experiments are summarized in Table 5. Our experimental setup follows configurations that are most commonly adopted in recent RLVR studies. Following DAPO, we remove the KL regularization term during training. In addition, under the clip-higher strategy, we use a clipping range of (cliplow,cliphigh)=(0.2,0.28)(clip_low,clip_high)=(0.2,0.28) for all methods except RLVR-DBB. For RLVR-DBB, the importance ratios tend to be larger than those of naive methods. As a result, using the same clipping range would cause a substantially larger fraction of meaningful update signals to be clipped. To avoid overly aggressive clipping, we therefore adopt a wider clipping range of (0.98,0.98)(0.98,0.98) specifically for RLVR-DBB. For RePO, which introduces additional hyperparameters and configuration options, we follow the optimal settings recommended in the original paper. Specifically, we set the number of replay samples to 8, use a replay cache size of 16, and adopt the reward-oriented replay strategy. For both evaluation and validation, we use the sampling parameters specified in Table 5. To reduce training overhead, the validation set is restricted to AIME24/25, AMC24, Minerva, and MATH500. Evaluation on OlympiadBench and the out-of-distribution benchmarks (MMLU-Pro, GPQA-Diamond, and Big-Bench Hard) is performed only on the best checkpoint selected for each experiment.