Paper deep dive
Generalized Proximal Policy Optimization: Unifying LLM RL Algorithms from the Perspective of Divergence
Zhicheng Cai, Xinyuan Guo, Hanlin Wu, Weinan Dai, Yuyan Zhou, Wei-Ying Ma, Ya-Qin Zhang, Hao Zhou
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 10/12/2026, 6:46:19 AM
Summary
The paper introduces Generalized Proximal Policy Optimization (GPPO), a unified framework for LLM reinforcement learning that addresses the limitations of Proximal Policy Optimization (PPO). GPPO establishes a two-dimensional design space based on Divergence geometry and Estimator topology. It demonstrates that PPO's reliance on Total Variation (TV) divergence with single-sample Monte Carlo estimation leads to premature clipping and poor exploration. By generalizing to other f-divergences (like KL, Pearson chi-squared) and novel estimators (Atomic, Binary), GPPO provides adaptive resistance to policy shifts. Experiments show that GPPO variants, particularly KL with Atomic estimation, significantly outperform standard PPO and GRPO on math and coding benchmarks.
Entities (12)
Relation Signals (11)
KL Divergence → combinedwith → Atomic Estimation
confidence 95% · Notably, KL with Atomic estimation achieves the strongest performance
Atomic Estimation → outperforms → Single-Sample Monte Carlo
confidence 95% · Atomic estimation consistently outperforms single-sample MC across all evaluated divergences.
GPPO → proposes → Atomic Estimation
confidence 95% · we propose two alternative topologies, Atomic and Binary, that more faithfully preserve the distribution-level divergence structure.
GPPO → proposes → Binary Estimation
confidence 95% · we propose two alternative topologies, Atomic and Binary
GPPO → surpasses → GRPO
confidence 95% · The best-performing GPPO instantiation surpasses standard GRPO by up to 50% and 14% on math and coding
GPPO → unifies → PPO
confidence 95% · Under this framework, PPO and recent variants such as DAPO, DPPO, and RIPO emerge as special cases
PPO → uses → Single-Sample Monte Carlo
confidence 95% · PPO implicitly constrains a single-sample Monte Carlo (MC) estimate
PPO → uses → Total Variation divergence
confidence 95% · PPO implicitly constrains a single-sample Monte Carlo (MC) estimate of the Total Variation (TV) divergence
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Reinforcement learning has become the foundational paradigm for enhancing the reasoning capabilities of LLMs. Yet the dominant approach, Proximal Policy Optimization (PPO), relies on a geometrically degenerate clipping proxy that cannot adapt its resistance to policy shifts, leading to premature clipping and degraded exploration. Subsequent variants alleviate these issues through fragmented and heuristic modifications. We propose Generalized Proximal Policy Optimization (GPPO), a principled and unified framework for understanding and designing LLM RL algorithms through the lens of divergence. We establish a generalized $f$-divergence trust-region theory with a monotonic performance improvement guarantee, and show that different $f$-divergences induce distinct geometries and policy update dynamics. We further introduce novel estimator topologies that provide more faithful divergence estimates than the conventional single-sample Monte Carlo estimation. Under this framework, PPO and recent variants such as DAPO, DPPO, and RIPO emerge as special cases, establishing a principled two-dimensional design space of {Divergence $\times$ Estimator}. Extensive experiments on eight competition-level math and coding benchmarks show that both dimensions are critical. The best-performing GPPO instantiation surpasses standard GRPO by up to 50% and 14% on math and coding, delivering SOTA performance.
Tags
Links
- Source: https://arxiv.org/abs/2610.13015v1
- Canonical: https://arxiv.org/abs/2610.13015v1
Trouble viewing inline? Open PDF directly →
Full Text
58,358 characters extracted from source content.
Expand or collapse full text
Generalized Proximal Policy Optimization: Unifying LLM RL Algorithms from the Perspective of DivergenceThanks: Corresponding author: zhouhao@air.tsinghua.edu.cn Zhicheng Cai Xinyuan Guo Hanlin Wu Weinan Dai Yuyan Zhou Affiliation: Wei-Ying Ma, Ya-Qin Zhang, Hao Zhou Affiliation: Institute for AI Industry Research (AIR), Tsinghua University Affiliation: SIA-Lab of Tsinghua AIR and ByteDance Seed Abstract Reinforcement learning has become the foundational paradigm for enhancing the reasoning capabilities of LLMs. Yet the dominant approach, Proximal Policy Optimization (PPO), relies on a geometrically degenerate clipping proxy that cannot adapt its resistance to policy shifts, leading to premature clipping and degraded exploration. Subsequent variants alleviate these issues through fragmented and heuristic modifications. We propose Generalized Proximal Policy Optimization (GPPO), a principled and unified framework for understanding and designing LLM RL algorithms through the lens of divergence. We establish a generalized f-divergence trust-region theory with a monotonic performance improvement guarantee, and show that different f-divergences induce distinct geometries and policy update dynamics. We further introduce novel estimator topologies that provide more faithful divergence estimates than the conventional single-sample Monte Carlo estimation. Under this framework, PPO and recent variants such as DAPO, DPPO, and RIPO emerge as special cases, establishing a principled two-dimensional design space of Divergence × Estimator. Extensive experiments on eight competition-level math and coding benchmarks show that both dimensions are critical. The best-performing GPPO instantiation surpasses standard GRPO by up to 50% and 14% on math and coding, delivering SOTA performance. 1 Introduction Reinforcement learning (RL) has established itself as the primary paradigm for advancing complex reasoning in LLMs (Guo et al., 2025; Comanici et al., 2025; OpenAI, 2024), powering breakthroughs across long-horizon domains such as mathematical reasoning (Shao et al., 2024; Wang et al., 2024; Luo et al., 2025), code generation (Dou et al., 2024; Anthropic, 2024), and deep search (OpenAI, 2025; Jin et al., 2025). Proximal Policy Optimization (PPO) (Schulman et al., 2017) has become a dominant approach for stable and effective RL training, with its “proximal” behavior enforced through clipping. Yet beneath this heuristic mechanism lies a geometrically degenerate proxy: PPO implicitly constrains a single-sample Monte Carlo (MC) estimate of the Total Variation (TV) divergence, whose generating function has zero curvature and strictly symmetric bounds. This linear geometry cannot adapt its resistance to the magnitude of policy shifts, treating small and large deviations alike. This rigidity can lead to premature clipping, distorted trust-region scaling, and exploration collapse (Yu et al., 2025; Cai et al., 2026; Qi et al., 2026). To alleviate these pathologies, recent variants (e.g., DAPO (Yu et al., 2025), DPPO (Qi et al., 2026), RIPO (Cai et al., 2026)) refine trust-region constraints through relaxing clipping bounds or adjusting Riemannian metrics. Despite their empirical gains, these methods are largely motivated by specific empirical observations, leading to fragmented modifications of the trust-region constraint. Consequently, existing approaches offer no unified, first-principles account of how these seemingly different mechanisms are related, why they improve optimization, or how to systematically design trust-region constraints beyond these isolated heuristics. We propose Generalized Proximal Policy Optimization (GPPO), a principled framework that unifies the fragmented landscape of LLM RL through the lens of divergence. Rather than introducing another isolated algorithm, GPPO establishes a principled two-dimensional design space of Divergence × Estimator, enabling systematic design of proximal policy optimization. At its core, GPPO establishes a generalized f-divergence trust-region theory, showing that policy optimization under f-divergence level-set constraints admits a monotonic performance improvement guarantee. Crucially, we show that different f-divergences induce distinct geometries, providing a principled basis for understanding their different policy update dynamics. Beyond the choice of divergence, GPPO further generalizes how divergences are estimated, introducing novel estimator topologies that reduce single-sample Monte Carlo variance and provide more faithful divergence estimates. Under this unified lens, PPO and recent variants such as DAPO, DPPO, and RIPO emerge naturally as special cases of GPPO. Empirically, our experiments reveal that proximal policy optimization is highly sensitive to both divergence geometry and estimator topology. Compared with the linear and symmetric TV geometry, divergences with non-zero curvature generally yield stronger performance. Estimator topology is equally important for that Atomic estimation consistently outperforms single-sample MC across all evaluated divergences. Notably, KL with Atomic estimation achieves the strongest performance, improving the average score by 50% on math and 14% on coding relative to the standard TV-MC formulation. These results validate the proposed two-dimensional design space that divergence geometry and estimator topology are both critical to effective RL optimization. In summary, our contributions are threefold: • We propose Generalized Proximal Policy Optimization (GPPO), establishing a two-dimensional design space of Divergence × Estimator, which unifies PPO and recent proximal variants, and inspires principled exploration of new proximal policy optimization algorithms. • We establish a generalized f-divergence trust-region theory with a monotonic performance improvement guarantee, and illustrate how divergence geometry shapes policy update dynamics. • We introduce novel estimator topologies that provide more faithful divergence estimates than single-sample Monte Carlo estimation, achieving SOTA performance on math and coding. 2 Preliminary 2.1 Monotonic Improvement under Trust Region Constraint Schulman et al. (2015) establishes a trust-region lower bound on the performance improvement between a behavior policy μ and a target policy π. Specifically, Theorem 1 (Trust-Region Performance Bound). For behavior policy μ and target policy π, the performance improvement satisfies η(π)−η(μ)≥1−γs∼ρμ,a∼μ(⋅|s)[r(s,a)Aμ(s,a)]−4ξγ(−γ)2DTVmax(π∥μ)2,η(π)-η(μ)≥ 11\!-\!γE_s ρ^μ,a μ(·|s) [r(s,a)A^μ(s,a) ]- 4ξγ(1\!-\!γ)^2D_TV (π μ)^2, (1) where η(π)η(π) denotes the expected discounted return, r(s,a)=π(a|s)μ(a|s)r(s,a)= π(a|s)μ(a|s) is the importance ratio, AμA^μ is the advantage function under μ, ξ=maxs,a|Aμ(s,a)|ξ= _s,a|A^μ(s,a)|, and DTVmax(π∥μ)=maxsDTV(π(⋅|s)∥μ(⋅|s))D_TV (π μ)= _sD_TV(π(·|s) μ(·|s)) is the maximum state-wise Total Variation (TV) divergence. This bound motivates maximizing the surrogate objective Lμ(π)=s∼ρμ,a∼μ(⋅|s)[r(s,a)Aμ(s,a)]L_μ(π)\!=\!E_s ρ^μ,\,a μ(·|s) [r(s,a)A^μ(s,a) ], while controlling the policy discrepancy measured by DTVmax(π∥μ)D_TV (π μ), providing a formal monotonic improvement guarantee. This principle underlies Trust Region Policy Optimization (TRPO) (Schulman et al., 2015), which maximizes the surrogate objective ℒμ(π)L_μ(π) subject to a constraint that keeps the new policy within a trust region around the current policy. Define δ>0δ>0 as the trust region radius, formally, a TV-constrained trust-region formulation can be written as : maxπ _π ℒμ(π) _μ(π) (2) s.t. .t. DTVmax(π∥μ)≤δ. D_TV (π μ)≤δ. Alternatively, this trust-region constraint can be formulated using the KL divergence, as theoretically justified by Pinsker’s inequality: DTV(π∥μ)2≤12DKL(π∥μ)D_TV(π μ)^2≤ 12D_KL(π μ). 2.2 Proximal Policy Optimization as a Divergence Proxy Despite the theoretical guarantees of TRPO, its constrained second-order optimization incurs substantial computational overhead, limiting its scalability in modern LLM RL. Proximal Policy Optimization (PPO) (Schulman et al., 2017) addresses this limitation by replacing the second-order optimization with a first-order clipped surrogate objective: PPO(π)=s∼ρμ,a∼μ(⋅|s)[min(r(s,a)A^(s,a),clip(r(s,a),1−ϵ,1+ϵ)A^(s,a))],J_PPO(π)=E_s ρ^μ,\,a μ(·|s) [ (r(s,a) A(s,a),clip (r(s,a),1-ε,1+ε ) A(s,a) ) ], (3) where A A denotes the estimated advantage, typically obtained using GAE (Schulman et al., 2016) in PPO and group-relative advantages in modern LLM RL methods such as GRPO (Guo et al., 2025) and its variants (Yu et al., 2025; Cai et al., 2026). The connection between PPO-Clip and the trust-region constraint can be understood through the TV divergence, for each state s, DTV(π(⋅|s)∥μ(⋅|s))=12a∼μ(⋅|s)[|r(s,a)−1|].D_TV(π(·|s) μ(·|s))= 12E_a μ(·|s) [ |r(s,a)-1 | ]. (4) Thus, |r(s,a)−1||r(s,a)-1| is a single-sample Monte Carlo estimator of DTV(π(⋅|s)∥μ(⋅|s))D_TV(π(·|s) μ(·|s)) (Qi et al., 2026) (ignore the constant coefficiency). From this perspective, PPO-Clip imposes a per-sample bound (that is proximal) on a noisy single-sample proxy of the divergence. While effective in practice, this proxy has two fundamental limitations. First, from a geometric perspective, the underlying TV geometry has zero curvature and symmetric penalties, providing no adaptive resistance to different magnitudes of policy shifts. Second, from an estimation perspective, PPO constrains only a single-sample Monte Carlo estimate rather than the full expectation defining the TV divergence, making the effective trust region sensitive to sampling noise. These limitations can manifest as premature clipping, distorted trust-region scaling, and exploration collapse (Yu et al., 2025; Cai et al., 2026; Qi et al., 2026). 3 Generalized Proximal Policy Optimization The preceding analysis reveals that PPO is not merely a clipping heuristic, but a particular instantiation of trust-region policy optimization characterized by a specific divergence and its estimator. We introduce Generalized Proximal Policy Optimization (GPPO), a principled framework that systematically generalizes proximal policy optimization along two complementary dimensions: the divergence being constrained and the estimator used to compute it. 3.1 Generalized f-Divergence Trust Region We first generalize the trust-region constraint from Total Variation (TV) to arbitrary f-divergences. The key theoretical step is to establish a tight quantitative relationship between an arbitrary f-divergence and the quadratic TV divergence DTV2D_TV^2 appearing in the monotonic improvement bound. Specifically, Vajda’s inequality provides the tightest possible lower bound of an f-divergence in terms of TV divergence: Theorem 2 (Vajda’s Tight Lower Bound). Let Df(P∥Q)=∫q(x)f(p(x)q(x))dxD_f(P Q)= q(x)f\! ( p(x)q(x) )\!dx be an f-divergence generated by a convex function f satisfying f(1)=0f(1)=0. For V=DTV(P∥Q)∈[0,1]V=D_TV(P Q)∈[0,1], we have Df(P∥Q)≥Lf(V):=infβ∈[0, 1−V][βf(1+Vβ)+(1−β)f(1−V1−β)].D_f(P Q)≥ L_f(V):= _β∈[0,\,1-V] [β f\! (1+ Vβ )+(1-β)f\! (1- V1-β ) ]. (5) where β∈[0,1−V]β∈[0,1-V] is the auxiliary probability-mass parameter in the tight binary reduction. Consequently, inverting the tight lower bound yields DTV(P∥Q)≤Lf−1(Df(P∥Q))D_TV(P Q)≤ L_f^-1\! (D_f(P Q) ). To establish a tractable quadratic trust-region bound, we seek the largest constant Cf>0C_f>0 such that Df(P∥Q)≥CfDTV(P∥Q)2D_f(P Q)≥ C_fD_TV(P Q)^2 holds uniformly for all P,QP,Q. Using Vajda’s tight lower bound, this optimal constant is: CfVajda:=inf0<V≤1Lf(V)V2,C_f^Vajda:= _0<V≤ 1 L_f(V)V^2, (6) which establishes the universal quadratic lower bound 11 1 The detailed derivation is provided in Appendix.: DTV(P∥Q)2≤1CfVajdaDf(P∥Q).D_TV(P Q)^2≤ 1C_f^VajdaD_f(P Q). (7) We can now generalize the TV trust-region constraint in Theorem 1 to an arbitrary f-divergence: Theorem 3 (Generalized f-Divergence Trust-Region Performance Bound). Define Dfmax(μ∥π):=maxsDf(π(⋅|s)∥μ(⋅|s))D_f (μ\! \!π)\!:=\! _sD_f\! (π(·|s) μ(·|s) ). Then, for any f-divergence with CfVajda>0C_f^Vajda>0, η(π)−η(μ) η(π)-η(μ) ≥11−γℒμ(π)−4ξγ(1−γ)2DTVmax(π∥μ)2 ≥ 11-γL_μ(π)- 4ξγ(1-γ)^2D_TV (π μ)^2 (8) ≥11−γℒμ(π)−4ξγCfVajda(1−γ)2Dfmax(π∥μ). ≥ 11-γL_μ(π)- 4ξγC_f^Vajda(1-γ)^2D_f (π μ). Thus, constraining any f-divergence to a sufficiently small level set guarantees monotonic performance improvement, establishing a unified trust-region framework for TV, Pearson χ2χ^2, KL, Hellinger, and other f-divergences. This naturally motivates the generalized trust-region optimization problem: maxπ _π ℒμ(π) _μ(π) (9) s.t. .t. Dfmax(π∥μ)≤δ. D_f (π μ)≤δ. From f-Divergence Trust Regions to Generalized Proximal Policy Optimization. Directly solving this generalized constrained optimization problem is computationally prohibitive for large-scale language model policy optimization. We therefore retain the first-order proximal paradigm of PPO, while replacing its fixed clipping rule with a clipped proximal region induced by the chosen f-divergence. Let D^f D_f denote an estimator of the underlying f-divergence. The corresponding proximal region is defined as the δ-sublevel set of D^f D_f: ℛ^f(δ)=r>0:D^f(r)≤δ. R_f(δ)= \r>0: D_f(r)≤δ \. (10) This induces the objective of generalized proximal policy optimization: GPPO(π)=s∼ρμ,a∼μ(⋅|s)[min(r(s,a)A^(s,a),clipℛ^f(δ)(r(s,a))A^(s,a))].J_GPPO(π)=E_s ρ^μ,\,a μ(·|s) [ (r(s,a) A(s,a),clip_ R_f(δ)(r(s,a)) A(s,a) ) ]. 3.00003pt (11) Thus the policy ratio is clipped whenever it moves outside the divergence-induced proximal region ℛ^f(δ) R_f(δ), thereby suppressing the gradient associated with the corresponding samples. 3.2 Divergence Geometry and Policy Update Dynamics GPPO in Eq. 11 reduces generalized f-divergence trust regions to practical updates. We now characterize how the geometry of f affects the proximal region and policy update dynamics. Under our convention in Eq. 9, we constrain Df(π(⋅|s)∥μ(⋅|s))=a∼μ(⋅|s)[f(r(s,a))]≤δD_f(π(·|s) μ(·|s))=E_a μ(·|s) [f(r(s,a)) ]≤δ. Thus the generating function f(r)f(r) determines the shape of the divergence level set and hence the admissible proximal region of policy-ratio, for which its first derivative f′(r)f (r) characterizes the marginal divergence cost while its curvature f′(r)f (r) governs how that cost grows with the magnitude of a finite ratio shift. This distinction becomes particularly critical beyond the local regime r≈1r≈ 1. Local Fisher–Rao Equivalence and Finite-Shift Geometry. For smooth f-divergences, a Taylor expansion around the reference point r=1r=1 gives f(r)=f(1)+f′(1)(r−1)+12f′(1)(r−1)2+o((r−1)2).f(r)=f(1)+f (1)(r-1)+ 12f (1)(r-1)^2+o\! ((r-1)^2 ). (12) Since a∼μ[r−1]=0E_a μ[r-1]=0 and f(1)=0f(1)=0, the linear term vanishes after taking expectation, yielding Df(π∥μ)=f′(1)2a∼μ[(r−1)2]+o(∥r−1∥2).D_f(π μ)= f (1)2E_a μ\! [(r-1)^2 ]+o\! (\|r-1\|^2 ). (13) Hence, all twice-differentiable f-divergences with f′(1)>0f (1)>0 share the same local Fisher–Rao geometry up to a scaling factor. Their distinction emerges in the finite-shift regime (r≉1r ≈ 1), where higher-order and asymmetric terms determine how resistance varies with the magnitude and direction of the policy-ratio deviation. Figure 1: Schematic of four f-divergence geometries and the corresponding proximal regions they induce, shown for three sampled policy ratios r=(r1,r2,r3)r=(r_1,r_2,r_3) centred at r=1r=1. Zero Curvature of TV and Rigid Clipping. For TV divergence, the generating function gives fTV(r)=12|r−1|,fTV′(r)=12sign(r−1),fTV′(r)=0(r≠1).f_TV(r)= 12|r-1|, f_TV (r)= 12sign(r-1), f_TV (r)=0 (r≠ 1). -3.00003pt (14) Thus, TV’s polyhedral geometry is piecewise linear with zero curvature almost everywhere, yielding a marginal penalty independent of the magnitude of the ratio deviation |r−1||r-1|. PPO-Clip reflects this linear penalty structure through a fixed pointwise clipping threshold, beyond which the corresponding surrogate gradient is suppressed. Hence, the distinctive limitation of the TV-induced geometry is not the existence of a fixed clipping boundary itself, but the absence of curvature that would otherwise provide magnitude-dependent resistance. This rigidity provides a geometric explanation for the premature clipping and exploration-related issues discussed in Section 2.2 and explored by subsequent variants (Yu et al., 2025; Cai et al., 2026; Qi et al., 2026). Adaptive Resistance of Smooth Divergences. In contrast, smooth f-divergences provide adaptive resistance through non-zero curvature. For Pearson χ2χ^2 divergence, fχ2(r)=(r−1)2,fχ2′(r)=2(r−1),fχ2′(r)=2,f_χ^2(r)=(r-1)^2, f_χ^2 (r)=2(r-1), f_χ^2 (r)=2, -3.00003pt (15) induces an ellipsoidal (Mahalanobis) geometry where the penalty grows quadratically with the ratio deviation, yielding progressively stronger resistance as |r−1||r-1| increases. For KL divergence, fKL(r)=−logr,fKL′(r)=−1r,fKL′(r)=1r2,f_KL(r)=- r, f_KL (r)=- 1r, f_KL (r)= 1r^2, -3.00003pt (16) introduces an asymmetric and logarithmically warped geometry where contraction r→0r→ 0 is increasingly penalized, while expansion toward large r faces diminishing marginal resistance, thus facilitating probability expansion of low-probability actions and thereby favor exploration. For Hellinger, fH(r)=(r−1)2,fH′(r)=1−1r,fH′(r)=12r3/2.f_H(r)=( r-1)^2, f_H (r)=1- 1 r, f_H (r)= 12r^3/2. -3.00003pt (17) It induces a spherical geometry under the square-root embedding whose curvature likewise increases toward r→0r→ 0 and decreases as r grows, resulting in asymmetric resistance. Fig. 1 visualizes the geometry of above four f-divergence. To conclude, Divergence Geometry determines the shape of the admissible policy-ratio region and its finite-shift resistance profile, thereby shaping where and how policy updates are constrained. 3.3 Divergence Estimator Topologies The proximal region in Eq. 10 depends on how the underlying f-divergence is estimated. Beyond the classical Single-Sample Monte Carlo estimator, we propose two alternative topologies, Atomic and Binary, that more faithfully preserve the distribution-level divergence structure. Single-Sample Monte Carlo Estimation. Single-Sample Monte Carlo estimation directly evaluates the divergence integrand at the sampled action: D^fMC(r(s,a))=f(r(s,a)),a∼μ(⋅|s). D_f^MC(r(s,a))=f(r(s,a)), a μ(·|s). (18) This induces the proximal region ℛ^fMC(δ)=r>0:f(r)≤δ R_f^MC(δ)=\r>0:f(r)≤δ\, suppressing sample gradients whenever f(r(s,a))>δf(r(s,a))>δ. Although unbiased, D^fMC D_f^MC relies entirely on single-sample realizations, making its proximal clipping decisions highly susceptible to MC sampling variance. Atomic Estimation. To mitigate the sensitivity of MC-estimated constraints to individual sampled outliers, we propose Atomic Estimation, which provides a probability-aware alternative. Recall that the exact divergence expands over discrete actions as Df(π∥μ)=∑aμ(a|s)f(r(s,a))D_f(π μ)= _aμ(a|s)f(r(s,a)). Atomic estimation imposes the proximal constraint directly on each atomic mass contribution: D^fAtom(r(s,a))=μ(a|s)f(r(s,a)),a∼μ(⋅|s). D_f^Atom(r(s,a))=μ(a|s)f(r(s,a)), a μ(·|s). (19) This induces the probability-aware proximal region ℛ^fAtom(δ)=r>0:μ(a|s)f(r)≤δ R_f^Atom(δ)=\r>0:μ(a|s)f(r)≤δ\. Crucially 22 2 Detailed illustration and complete theoretical analysis are provided in Appendix., Atomic estimation distinguishes the magnitude of a ratio deviation from its probability-weighted contribution: a large shift on a low-probability action does not necessarily constitute a large divergence contribution, thereby avoiding premature clipping of rare exploratory actions. Furthermore, we prove in Appendix that atomic estimation offers an improved bias-variance trade-off than MC, enabling stable and efficient training. Binary Estimation. To capture distribution-level discrepancies beyond the randomly-sampled action, we consider Binary Estimation (Qi et al., 2026), which collapses the action space into the sampled action and its complement, yielding the binary f-divergence estimation: D^fBin(r(s,a))=μ(a|s)f(r(s,a))+(1−μ(a|s))f(1−μ(a|s)r(s,a)1−μ(a|s)),a∼μ(⋅|s). D_f^Bin(r(s,a))=μ(a|s)f (r(s,a) )+ (1-μ(a|s) )\,f ( 1-μ(a|s)r(s,a)1-μ(a|s) ), a μ(·|s). (20) This induces the proximal region ℛ^fBin(δ)=r(s,a):D^fBin(r(s,a))≤δ R_f^Bin(δ)=\r(s,a): D_f^Bin(r(s,a))≤δ\. Unlike MC and Atomic estimation, which evaluate only the sampled action contribution, Binary estimation accounts for the complementary probability mass and therefore captures the resulting distribution-level shift. 33 3 We also explored Top-K estimation since both distributions of μ,πμ,π are known. However, it incurs prohibitively high memory overhead ((K)O(K)) and exhibits elevated variance across candidate actions, making the proximal threshold δ excessively difficult to tune. These empirical findings suggest that overly fine-grained estimation does not necessarily benefit, and can even compromise overall system robustness.. To conclude, these estimators provide distinct ways of translating the underlying f-divergence geometry into a practical proximal constraint. 3.4 Unified View of Existing Proximal Policy Optimization Methods GPPO provides a principled framework for interpreting existing proximal policy optimization methods within a two-dimensional design space of Divergence f × Estimator T. Specifically, PPO (Schulman et al., 2017), as previously stated, essentially constrains the TV divergence under single-sample MC estimation, follower by GRPO (Shao et al., 2024; Guo et al., 2025). GSPO (Zheng et al., 2025) and GMPO (Zhao et al., 2026) retain the TV divergence and MC estimator as PPO does, while changing the sampling unit from individual tokens to complete rollouts. DAPO (Yu et al., 2025) adopts an asymmetric clipping mechanism: DAPO(π)=s,a[min(r(s,a)A^(s,a),clip(r(s,a),1−ϵlow,1+ϵhigh)A^(s,a))],J_DAPO(π)=E_s,a [ (r(s,a) A(s,a),clip (r(s,a),1- _low,1+ _high ) A(s,a) ) ], which corresponds to constraining the KL divergence under single-sample MC estimation. According to Eq. 18, D^KLMC(r(s,a))=−logr D_KL^MC(r(s,a))\!=\!-\! r, constraining −δ≤logr≤δ-δ\!≤\! r\!≤\!δ gives ℛ^KLMC(δ)=e−δ≤r≤eδ R_KL^MC(δ)\!=\!\e^-δ\!≤\!r\!≤\!e^δ\. Let δ=0.23δ=0.23, thus ≤r≤1.260.8\!≤\!r\!≤\!1.26, matching DAPO’s asymmetric clipping bounds of ϵlow _low and ϵhigh _high. DPPO (Qi et al., 2026) constrains |π(a|s)−μ(a|s)|≤δ|π(a|s)-μ(a|s)|≤δ, which corresponds exactly to constraining TV divergence under binary estimation. According to Eq. 20 we obtain, D^TVBin(r(s,a))=μ(a|s)2|r(s,a)−1|+(−μ(a|s))2|−μ(a|s)r(s,a)−μ(a|s)−1|=|π(a|s)−μ(a|s)|≤δ, D_TV^Bin(r(s,a))= μ(a|s)2|r(s,a)\!-\!1|+ (1\!-\!μ(a|s) )2\, | 1\!-\!μ(a|s)r(s,a)1\!-\!μ(a|s)\!-\!1 |=|π(a|s)\!-\!μ(a|s)|≤δ, which yields the actual proximal region ℛ^TVBin(δ)=1−δμ(a|s)≤r(s,a)≤1+δμ(a|s) R_TV^Bin(δ)=\1- δμ(a|s)≤ r(s,a)≤ 1+ δμ(a|s)\. Thus, DPPO can be equivalently written as: DPPO(π)=s,a[min(r(s,a)A^(s,a),clip(r(s,a),1−δμ(a|s),1+δμ(a|s))A^(s,a))].J_DPPO(π)=E_s,a [ (r(s,a) A(s,a),clip (r(s,a),1- δμ(a|s),1+ δμ(a|s) ) A(s,a) ) ]. RIPO (Cai et al., 2026) guarantees isometric policy updates through the clipped objective: RIPO(π)=s,a[min(r(s,a)A^(s,a),clip(r(s,a),−δμ(a|s),+δμ(a|s))A^(s,a))].J_RIPO(π)=E_s,a [ (r(s,a) A(s,a),clip (r(s,a),1\!-\! δμ(a|s),1\!+\! δμ(a|s) ) A(s,a) ) ]. which essentially constrains χ2χ^2 divergence under atomic estimation, since D^χ2Atom(r(s,a))=μ(a|s)(r(s,a)−1)2≤δ D_χ^2^Atom(r(s,a))=μ(a|s)(r(s,a)-1)^2≤δ induces exactly the proximal region ℛ^χ2Atom(δ)=−δμ(a|s)≤r(s,a)≤+δμ(a|s) R_χ^2^Atom(δ)=\1\!-\! δμ(a|s)≤ r(s,a)≤ 1\!+\! δμ(a|s)\ used by RIPO. In summary, existing proximal methods correspond to different (f,)(f,T) choices, inducing distinct proximal regions for policy updates. More broadly, GPPO provides a systematic principle for designing new methods: f specifies the desired resistance profile, while T determines how the resulting divergence constraint is estimated and enforced from sampled data. Table 1: Results of the different divergences and estimators across various competition-level mathematical reasoning and code generation benchmarks (evaluated by Avg@8). We also compare the performance of other baselines. We highlight each cell as best, second best, and third best. Mathematical Reasoning Divergence Estimator AIME24 AIME25 BRUMO25 SMT25 Avg. TV MC (GRPO) 34.6 20.8 33.3 21.9 27.7 ( +0.00%) Binary (DPPO) 46.7 32.5 42.5 34.2 39.0 (+40.79%) Atomic 45.0 28.3 40.8 30.9 36.3 (+31.05%) Hellinger MC 37.9 30.8 35.4 23.1 31.8 (+14.80%) Binary 42.1 32.9 37.5 29.0 35.4 (+27.80%) Atomic 42.9 32.5 43.3 29.0 36.9 (+33.21%) χ2χ^2 MC 34.6 20.8 33.3 21.9 27.7 ( +0.00%) Binary 45.0 29.2 36.7 28.1 34.8 (+25.63%) Atomic (RIPO) 47.9 32.5 47.5 34.2 40.5 (+46.21%) KL MC (DAPO) 45.0 27.5 37.5 28.5 34.6 (+24.91%) Binary 47.1 29.6 42.9 29.0 37.2 (+34.30%) Atomic 48.9 33.9 48.7 34.4 41.5 (+49.82%) Other Baselines GSPO 42.9 28.7 36.3 32.5 35.1 (+26.71%) GMPO 41.7 30.0 35.8 33.0 35.1 (+26.71%) SAPO 37.3 28.7 41.7 32.3 35.0 (+26.35%) CISPO 31.7 23.8 38.3 28.0 30.5 (+10.11%) Clip-Cov 36.3 22.9 36.6 30.2 31.5 (+13.72%) Base Model 3.8 3.3 15.0 10.8 8.2 (-70.40%) Code Generation Divergence Estimator Codeforces CodeContests TACO APPS Avg. TV MC (GRPO) 61.9 57.4 36.2 54.2 52.4 ( +0.00%) Binary (DPPO) 63.5 58.6 37.6 55.6 53.8 ( +2.67%) Atomic 64.2 60.5 38.1 56.7 54.9 ( +4.77%) Hellinger MC 66.4 58.2 36.3 56.8 54.4 ( +3.82%) Binary 66.3 60.2 39.2 57.7 55.9 ( +6.68%) Atomic 68.2 63.7 40.3 60.3 58.1 (+10.88%) χ2χ^2 MC 61.9 57.4 36.2 54.2 52.4 ( +0.00%) Binary 64.8 59.3 38.6 56.3 54.8 ( +4.58%) Atomic (RIPO) 68.2 62.3 39.3 58.9 57.2 ( +9.16%) KL MC (DAPO) 64.0 59.1 37.4 56.8 54.3 ( +3.63%) Binary 66.4 61.4 38.3 57.3 55.9 ( +6.68%) Atomic 70.7 65.1 42.3 60.8 59.7 (+13.93%) Other Baselines GSPO 63.8 57.9 36.6 56.3 53.7 ( +2.48%) GMPO 64.0 59.2 37.4 56.8 54.4 ( +3.82%) SAPO 63.7 57.0 37.5 55.9 53.5 ( +2.10%) CISPO 61.6 54.6 35.1 53.9 51.3 ( -2.10%) Clip-Cov 62.9 57.4 36.0 55.5 53.0 ( +1.15%) Base Model 32.3 32.1 16.5 32.1 28.3 (-45.99%) 4 Experiments To validate GPPO as a principled framework for proximal policy optimization and to characterize how the constrained divergence and its estimator shape policy update, we conduct extensive experiments across eight competition-level benchmarks spanning both mathematical reasoning and code generation, and benchmark instantiations drawn from the GPPO design space against nine representative RL algorithms. 4.1 Experimental Setup Baselines. Since GPPO unifies proximal policy optimization along two complementary axes, namely, the constrained divergence f and its estimator T, we instantiate it across four divergences (TV, Pearson χ2χ^2, Hellinger, and KL) and three estimators (MC, Atomic, and Binary), forming a 4×34× 3 design space. Within this space, several existing methods are recovered as special cases occupying specific (f,)(f,T) cells, including GRPO (Shao et al., 2024) (fTV,MC)(f_TV,T_MC), DAPO (Yu et al., 2025) (fKL,MC)(f_KL,T_MC), DPPO (Qi et al., 2026) (fTV,Bin)(f_TV,T_Bin), and RIPO (Cai et al., 2026) (fχ2,Atom)(f_χ^2,T_Atom). Rather than tuning the trust-region radius δ directly, we calibrate it per divergence by matching the contraction-side clip of its MC proximal region ℛ^fMC(δ)=r>0:f(r)≤δ R_f^MC(δ)=\r>0:f(r)≤δ\ to standard PPO-Clip, isolating divergence geometry from trust-region scale: fixing the lower bound to rmin=0.8r_ =0.8 directly gives δ=f(0.8)δ=f(0.8). Accordingly, δTV=0.2,δχ2=0.04,δH≈0.01 _TV=0.2,\ _χ^2=0.04,\ _H≈ 0.01, and δKL≈0.22 _KL≈ 0.22. While the upper bound rmaxr_ follows from f(rmax)=δf(r_ )=δ. The Atomic and Binary estimators directly reuse this δ, so that all estimators of a divergence share a common radius while their realized bounds adapt to the reference probability μ(a∣s)μ(a s). We further compare against other RL algorithms, including GSPO (Zheng et al., 2025) and GMPO (Zhao et al., 2026), which apply the TV–MC constraint at the full-rollout rather than per-token level; SAPO (Gao et al., 2025) and CISPO (Chen et al., 2025), which preserve otherwise-clipped gradients via soft gating and importance-weight clipping; and Clip-Cov (Cui et al., 2025b), which clips high-covariance tokens. All following their default hyper-parameter settings. Following common practice (Yang et al., 2025b; Ye et al., 2020), we apply dual clipping to the importance ratio with lower and upper bounds of 0.50.5 and 1010, and remove the KL penalty term for all methods (Yu et al., 2025; Yang et al., 2025b). All algorithms adopt the GRPO-style group-relative advantage estimator, and trained on Qwen3-8B-Base (Yang et al., 2025a). Training. We evaluate the improvement induced by each RL algorithm on two complementary domains, mathematical reasoning and code generation, both of which admit verifiable rewards and are widely regarded as strong indicators of general reasoning ability. For mathematical reasoning we train on DAPO-Math-17k (Yu et al., 2025), and for code generation on Eurus-Code-25k (Cui et al., 2025a). For each query we generate 8 rollouts (namely, the group size is 8) with a maximum response length of 20k tokens for thorough reasoning. In each RL iteration, the behavior policy πθold _ _old totally generates 1024 rollouts with a train batch size of 128, and the current policy πθ _θ is updated 8 times with a mini-batch size of 16. Rewards are set to 11 for correct responses and 00 otherwise. We uniformly adopt token-mean loss aggregation and optimize all models to convergence using AdamW (Loshchilov and Hutter, 2017) with a constant learning rate of 1×10−61× 10^-6, training for 400 steps on mathematical reasoning and 600 steps on code generation. All experiments are conducted on the VeRL (Sheng et al., 2025) framework using 8×H20 GPUs. Evaluation. We comprehensively evaluate the effectiveness of these algorithms on eight competition-level benchmarks without contamination, spanning the two domains. For mathematical reasoning we use AIME24, AIME25, BRUMO25, and SMT25 (Balunović et al., 2025); for code generation we use Codeforces, CodeContests, TACO, and APPS (Cui et al., 2025a). We repeat the evaluation set for 8 times and report avg@8avg@8 metric for results stability. Figure 2: Training dynamics of different GPPO instantiations on Math and Code. Solid lines represent the training score, while dash-dotted lines represent the policy entropy. 4.2 Main Results Table 1 summarizes the main experimental results across eight competition-level benchmarks. Fig 2 visualized the training dynamics of different GPPO instantiations on math and coding. GPPO establishes a new state-of-the-art across reasoning tasks. As highlighted in Table 1, our top-performing GPPO instantiation (KL divergence under Atomic estimation) achieves an average of 41.541.5 on mathematical reasoning and 59.759.7 on code generation, drastically improving upon the standard TV–MC baseline (GRPO) by +49.8%+49.8\% and +13.9%+13.9\%, respectively, outperforming all existing RL algorithms. Beyond this optimal configuration, GPPO instantiations consistently dominate the design space, every divergence equipped with an Atomic or Binary estimator outperforms the strongest baseline (GMPO). These gains are uniform rather than dataset-specific, confirming that systematic exploration of the Divergence × Estimator interplay provides a fundamentally superior foundation for policy optimization than ad-hoc clipping heuristics. Divergence geometry shapes policy optimization dynamics. Under single-sample MC, KL improves over TV by 24.9%24.9\% on math and 3.6%3.6\% on code, while Hellinger yields gains of 14.8%14.8\% and 3.8%3.8\%, both enhancing the entropy level of TV. TV–MC and χ2χ^2–MC share the same results for that our matched calibration of single-sample MC coincidentally assigns them the same symmetric interval r∈[0.8,1.2]r∈[0.8,1.2], their geometric differences become contributing when estimated more faithfully with Atomic or Binary. Under Atomic estimation, KL surpasses TV by +14.3%+14.3\% and +8.7%+8.7\% on math and code. Binary reverses this on math but yields only modest gains for non-TV divergences on code, showing that divergence geometry is filtered through estimator topology rather than universally ranked. One possible explanation is that TV–Binary directly constrains the absolute mass change |π(a∣s)−μ(a∣s)|≤δ|π(a s)-μ(a s)|≤δ, whereas curved divergences convert the same radius into a μ(a∣s)μ(a s)-dependent ratio penalty that can be tighter for rare actions. This could favor TV when useful mathematical updates require reallocating mass among a few low-probability reasoning branches, while coding’s larger set of valid implementations may attenuate the difference. Estimator topology determines how this geometry is enforced under sampling. Across four divergences, both proposed probability-aware estimators improve substantially over MC, while Atomic ranks first in every column of the code table and achieves the best math average. Averaged over the four divergences, Binary improves over MC by 21.3%21.3\% on math and 3.2%3.2\% on code, whereas Atomic yields larger gains of 28.3%28.3\% and 7.7%7.7\%, respectively. As Figure 2 shows, the entropy of MC collapse rapidly toward zero with the score curves plateau early at comparatively low levels. In contrast, Atomic and Binary maintain moderate range of entropy and continue improving over a longer horizon. This phenomenon follows directly from the estimator: Atomic avoids clipping large but low-probability exploratory updates by weighting with reference probability, and Binary instead accounts for the complementary probability mass. 5 Conclusion This paper introduces GPPO, a principled framework that unifies proximal policy optimization through a two-dimensional design space of divergence geometry and estimator topology, supported by a generalized f-divergence trust-region theorem. Across eight mathematical reasoning and code generation benchmarks, GPPO substantially outperforms standard PPO, demonstrating that both dimensions are critical for effective LLM reinforcement learning. Ultimately, GPPO provides a systematic foundation for understanding and designing future LLM RL algorithms. References Anthropic (2024) Anthropic Claude for coding. Note: https://w.anthropic.com/claude Cited by: §1. Balunović et al. (2025) M. Balunović, J. Dekoninck, I. Petrov, N. Jovanović, and M. Vechev Matharena: evaluating llms on uncontaminated math competitions. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmark. Cited by: §4.1. Cai et al. (2026) Z. Cai, X. Guo, H. Wu, M. Wang, W. Ma, Y. Zhang, and H. Zhou Beyond euclidean clipping: overcoming exploration collapse in llm rl via riemannian isometric policy optimization. International conference on machine learning. Cited by: §1, §1, §2.2, §2.2, §3.2, §3.4, §4.1. Chen et al. (2025) A. Chen, A. Li, B. Gong, B. Jiang, B. Fei, B. Yang, B. Shan, C. Yu, C. Wang, C. Zhu, et al. Minimax-m1: scaling test-time compute efficiently with lightning attention. arXiv preprint arXiv:2506.13585. Cited by: §4.1. Comanici et al. (2025) G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosen, et al. Gemini 2.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261. Cited by: §1. Cui et al. (2025a) G. Cui, L. Yuan, Z. Wang, H. Wang, W. Li, B. He, Y. Fan, T. Yu, Q. Xu, W. Chen, et al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456. Cited by: §4.1, §4.1. Cui et al. (2025b) G. Cui, Y. Zhang, J. Chen, L. Yuan, Z. Wang, Y. Zuo, H. Li, Y. Fan, H. Chen, W. Chen, et al. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505.22617. Cited by: §4.1. Dou et al. (2024) S. Dou, Y. Liu, H. Jia, L. Xiong, E. Zhou, W. Shen, J. Shan, C. Huang, X. Wang, X. Fan, et al. Stepcoder: improve code generation with reinforcement learning from compiler feedback. arXiv preprint arXiv:2402.01391. Cited by: §1. Gao et al. (2025) C. Gao, C. Zheng, X. Chen, K. Dang, S. Liu, B. Yu, A. Yang, S. Bai, J. Zhou, and J. Lin Soft adaptive policy optimization. arXiv preprint arXiv:2511.20347. Cited by: §4.1. Guo et al. (2025) D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. Deepseek-r1: incentivizing reasoning capability in llms via reinforcement learning. Nature. Cited by: §1, §2.2, §3.4. Jin et al. (2025) B. Jin, H. Zeng, Z. Yue, J. Yoon, S. Arik, D. Wang, H. Zamani, and J. Han Search-r1: training llms to reason and leverage search engines with reinforcement learning. Proceedings of the Conference on Language Modeling (COLM). Cited by: §1. Loshchilov and Hutter (2017) I. Loshchilov and F. Hutter Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. Cited by: §4.1. Luo et al. (2025) H. Luo, Q. Sun, C. Xu, P. Zhao, J. Lou, C. Tao, X. Geng, Q. Lin, S. Chen, and D. Zhang Wizardmath: empowering mathematical reasoning for large language models via reinforced evol-instruct. International Conference on Learning Representations. Cited by: §1. OpenAI (2024) OpenAI Learning to reason with large language models. Note: https://openai.com/research/learning-to-reason-with-llms Cited by: §1. OpenAI (2025) OpenAI Introducing deep research. Note: https://openai.com/index/introducing-deep-research/ Cited by: §1. Qi et al. (2026) P. Qi, X. Zhou, Z. Liu, T. Pang, C. Du, M. Lin, and W. S. Lee Rethinking the trust region in llm reinforcement learning. International conference on machine learning. Cited by: §1, §1, §2.2, §2.2, §3.2, §3.3, §3.4, §4.1. Schulman et al. (2015) J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz Trust region policy optimization. In International conference on machine learning, p. 1889–1897. Cited by: §2.1, §2.1. Schulman et al. (2016) J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel High-dimensional continuous control using generalized advantage estimation. International Conference on Learning Representations. Cited by: §2.2. Schulman et al. (2017) J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Cited by: §1, §2.2, §3.4. Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. Deepseekmath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: §1, §3.4, §4.1. Sheng et al. (2025) G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng, H. Lin, and C. Wu Hybridflow: a flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, p. 1279–1297. Cited by: §4.1. Wang et al. (2024) P. Wang, L. Li, Z. Shao, R. Xu, D. Dai, Y. Li, D. Chen, Y. Wu, and Z. Sui Math-shepherd: verify and reinforce llms step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 9426–9439. Cited by: §1. Yang et al. (2025a) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: §4.1. Yang et al. (2025b) S. Yang, C. Dou, P. Guo, K. Lu, Q. Ju, F. Deng, and R. Xin Dcpo: dynamic clipping policy optimization. arXiv preprint arXiv:2509.02333. Cited by: §4.1. Ye et al. (2020) D. Ye, Z. Liu, M. Sun, B. Shi, P. Zhao, H. Wu, H. Yu, S. Yang, X. Wu, Q. Guo, et al. Mastering complex control in moba games with deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, Vol. 34, p. 6672–6679. Cited by: §4.1. Yu et al. (2025) Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, W. Dai, T. Fan, G. Liu, L. Liu, et al. Dapo: an open-source llm reinforcement learning system at scale. Proceedings of the Neural Information Processing Systems. Cited by: §1, §1, §2.2, §2.2, §3.2, §3.4, §4.1, §4.1, §4.1. Zhao et al. (2026) Y. Zhao, Y. Liu, J. Liu, J. Chen, Y. Hao, T. Lv, S. Huang, L. Cui, Q. Ye, F. Wan, et al. Geometric-mean policy optimization. In International Conference on Learning Representations, Vol. 2026, p. 41354–41374. Cited by: §3.4, §4.1. Zheng et al. (2025) C. Zheng, S. Liu, M. Li, X. Chen, B. Yu, C. Gao, K. Dang, Y. Liu, R. Men, A. Yang, et al. Group sequence policy optimization. arXiv preprint arXiv:2507.18071. Cited by: §3.4, §4.1. Appendix A Derivation for Theorem 2 Derivation of Vajda’s Tight Lower Bound. We derive the tight lower bound by reducing the original probability space to a binary partition induced by the total variation distance. Let V=DTV(P∥Q)=12∫|p(x)−q(x)|dx.V=D_TV(P Q)= 12 _X|p(x)-q(x)|\,dx. Consider the Hahn decomposition induced by the signed measure P−QP-Q, with A:=x∈:p(x)≥q(x).A:=\x :p(x)≥ q(x)\. and let Ac=∖A^c=X A denote its complement. By construction, p(x)−q(x)≥0p(x)-q(x)≥ 0 on A and p(x)−q(x)<0p(x)-q(x)<0 on AcA^c. Therefore, 2V 2V =∫A(p(x)−q(x))x+∫Ac(q(x)−p(x))x = _A(p(x)-q(x))\,dx+ _A^c(q(x)-p(x))\,dx =P(A)−Q(A)+Q(Ac)−P(Ac). =P(A)-Q(A)+Q(A^c)-P(A^c). (21) Since A and AcA^c form a partition of X, P(Ac)=1−P(A),Q(Ac)=1−Q(A).P(A^c)=1-P(A), Q(A^c)=1-Q(A). It follows that Q(Ac)−P(Ac) Q(A^c)-P(A^c) =(1−Q(A))−(1−P(A)) =(1-Q(A))-(1-P(A)) =P(A)−Q(A). =P(A)-Q(A). (22) Substituting this equality into equation 21 yields V=P(A)−Q(A)=Q(Ac)−P(Ac).V=P(A)-Q(A)=Q(A^c)-P(A^c). (23) Introduce the auxiliary probability-mass parameter β:=Q(A).β:=Q(A). By equation 23, the probability masses of the two distributions on the binary partition A,Ac\A,A^c\ are P(A) P(A) =β+V, =β+V, Q(A) Q(A) =β, =β, (24) P(Ac) P(A^c) =1−β−V, =1-β-V, Q(Ac) Q(A^c) =1−β. =1-β. (25) Non-negativity of these probabilities requires β∈[0,1−V].β∈[0,1-V]. We next partition the f-divergence according to A and AcA^c. Recall that Df(P∥Q)=∫q(x)f(p(x)q(x))dx.D_f(P Q)= _Xq(x)f ( p(x)q(x) )\,dx. Thus, Df(P∥Q) D_f(P Q) =∫Aq(x)f(p(x)q(x))x+∫Acq(x)f(p(x)q(x))x. = _Aq(x)f ( p(x)q(x) )\,dx+ _A^cq(x)f ( p(x)q(x) )\,dx. (26) Consider first the integral over A. For β=Q(A)>0β=Q(A)>0, we can rewrite it as ∫Aq(x)f(p(x)q(x))x _Aq(x)f ( p(x)q(x) )\,dx =Q(A)∫Aq(x)Q(A)f(p(x)q(x))x. =Q(A) _A q(x)Q(A)f ( p(x)q(x) )\,dx. (27) The normalized density q(x)/Q(A)q(x)/Q(A) defines the conditional distribution Q(⋅∣A)Q(· A). Since f is convex, Jensen’s inequality gives ∫Aq(x)f(p(x)q(x))x _Aq(x)f ( p(x)q(x) )\,dx ≥Q(A)f(∫Aq(x)Q(A)p(x)q(x)x) ≥ Q(A)f ( _A q(x)Q(A) p(x)q(x)\,dx ) =Q(A)f(P(A)Q(A)). =Q(A)f ( P(A)Q(A) ). (28) Similarly, applying Jensen’s inequality to the complementary set AcA^c yields ∫Acq(x)f(p(x)q(x))x _A^cq(x)f ( p(x)q(x) )\,dx ≥Q(Ac)f(P(Ac)Q(Ac)). ≥ Q(A^c)f ( P(A^c)Q(A^c) ). (29) Combining equation 26, equation 28, and equation 29, we obtain Df(P∥Q)≥Q(A)f(P(A)Q(A))+Q(Ac)f(P(Ac)Q(Ac)).D_f(P Q)≥ Q(A)f ( P(A)Q(A) )+Q(A^c)f ( P(A^c)Q(A^c) ). (30) This inequality can be interpreted as the data processing inequality for the binary mapping T(x)=A(x)T(x)=1_A(x): aggregating the original sample space into the two events A and AcA^c cannot increase the f-divergence. Substituting the probability masses in equation 24 and equation 25 into equation 30, we obtain Df(P∥Q)≥βf(β+Vβ)+(1−β)f(1−β−V1−β). D_f(P Q)≥β f ( β+Vβ )+(1-β)f ( 1-β-V1-β ). Rearranging the arguments of f gives Df(P∥Q)≥βf(1+Vβ)+(1−β)f(1−V1−β). D_f(P Q)≥β f (1+ Vβ )+(1-β)f (1- V1-β ). (31) Since β is an auxiliary parameter determined by the particular pair (P,Q)(P,Q), while constrained only by β∈[0,1−V]β∈[0,1-V], taking the infimum over all feasible values of β yields Df(P∥Q)≥infβ∈[0,1−V][βf(1+Vβ)+(1−β)f(1−V1−β)].D_f(P Q)≥ _β∈[0,1-V] [β f (1+ Vβ )+(1-β)f (1- V1-β ) ]. (32) Therefore, Df(P∥Q)≥Lf(V).D_f(P Q)≥ L_f(V). The boundary cases β=0β=0 and β=1−Vβ=1-V are understood by continuity through the corresponding limiting binary distributions. In particular, when a denominator vanishes, the expression is interpreted using the extended-valued convention for f-divergences. Finally, the lower bound is tight. For any V∈[0,1]V∈[0,1] and β∈[0,1−V]β∈[0,1-V], consider the binary distributions Pβ=(β+V,1−β−V),Qβ=(β,1−β).P_β=(β+V,1-β-V), Q_β=(β,1-β). They satisfy DTV(Pβ∥Qβ)=12(|β+V−β|+|1−β−V−(1−β)|)=V.D_TV(P_β Q_β)= 12 (|β+V-β|+|1-β-V-(1-β)| )=V. Moreover, Df(Pβ∥Qβ) D_f(P_β Q_β) =βf(β+Vβ)+(1−β)f(1−β−V1−β) =β f ( β+Vβ )+(1-β)f ( 1-β-V1-β ) =βf(1+Vβ)+(1−β)f(1−V1−β). =β f (1+ Vβ )+(1-β)f (1- V1-β ). (33) Thus, taking the infimum over β∈[0,1−V]β∈[0,1-V] shows that Lf(V)L_f(V) is achieved by binary distributions whenever the infimum is attained. More generally, a sequence of binary distributions can approach Lf(V)L_f(V) arbitrarily closely. Hence, no larger universal lower bound depending solely on V=DTV(P∥Q)V=D_TV(P Q) can hold for all probability distributions. Derivation for the Optimal Constant. We next derive the optimal constant for a quadratic lower bound between the total variation distance and an arbitrary f-divergence. Specifically, we seek the largest constant Cf>0C_f>0 such that Df(P∥Q)≥CfDTV(P∥Q)2D_f(P Q)≥ C_fD_TV(P Q)^2 (34) holds uniformly for all probability distributions P and Q. Let V=DTV(P∥Q).V=D_TV(P Q). From Vajda’s tight lower bound established above, we have Df(P∥Q)≥Lf(V).D_f(P Q)≥ L_f(V). (35) For any V>0V>0, dividing both sides of equation 35 by V2V^2 gives Df(P∥Q)DTV(P∥Q)2≥Lf(V)V2. D_f(P Q)D_TV(P Q)^2≥ L_f(V)V^2. (36) Since the right-hand side depends only on the total variation distance, a universal quadratic lower bound must hold for all possible values V∈(0,1]V∈(0,1]. Therefore, taking the infimum over V yields CfVajda:=inf0<V≤1Lf(V)V2.C_f^Vajda:= _0<V≤ 1 L_f(V)V^2. (37) By definition of the infimum, Lf(V)V2≥CfVajdafor all V∈(0,1]. L_f(V)V^2≥ C_f^Vajda all V∈(0,1]. Combining this inequality with equation 35, we obtain Df(P∥Q) D_f(P Q) ≥Lf(V) ≥ L_f(V) ≥CfVajdaV2 ≥ C_f^VajdaV^2 =CfVajdaDTV(P∥Q)2. =C_f^VajdaD_TV(P Q)^2. (38) Equivalently, DTV(P∥Q)2≤1CfVajdaDf(P∥Q).D_TV(P Q)^2≤ 1C_f^VajdaD_f(P Q). (39) It remains to show that CfVajdaC_f^Vajda is the largest possible universal constant. Suppose there exists a constant C>CfVajdaC>C_f^Vajda such that Df(P∥Q)≥CDTV(P∥Q)2D_f(P Q)≥ CD_TV(P Q)^2 holds for all probability distributions P and Q. By the tightness of Vajda’s bound, for every V∈(0,1]V∈(0,1] there exist binary distributions (Pβ,Qβ)(P_β,Q_β) satisfying DTV(Pβ,Qβ)=VD_TV(P_β,Q_β)=V and Df(Pβ∥Qβ)=Lf(V),D_f(P_β Q_β)=L_f(V), where β minimizes the binary lower-bound expression (or approaches its infimum arbitrarily closely). Applying the assumed quadratic bound to these distributions gives Lf(V)≥CV2,L_f(V)≥ CV^2, and hence Lf(V)V2≥Cfor all V∈(0,1]. L_f(V)V^2≥ C all V∈(0,1]. Taking the infimum over V contradicts the definition of CfVajdaC_f^Vajda, since it would imply CfVajda=inf0<V≤1Lf(V)V2≥C>CfVajda.C_f^Vajda= _0<V≤ 1 L_f(V)V^2≥ C>C_f^Vajda. Therefore, no constant larger than CfVajdaC_f^Vajda can satisfy the quadratic lower bound uniformly over all distributions. Consequently, CfVajdaC_f^Vajda is the optimal universal quadratic constant. Local Quadratic Behavior of the Vajda Bound. We further characterize the local quadratic behavior of the binary lower bound around V=0V=0. For a fixed β∈(0,1)β∈(0,1), define gβ(V)=βf(1+Vβ)+(1−β)f(1−V1−β).g_β(V)=β f (1+ Vβ )+(1-β)f (1- V1-β ). (40) The Vajda bound gives Df(P∥Q)≥gβ(V),V=DTV(P,Q).D_f(P Q)≥ g_β(V), V=D_TV(P,Q). (41) Since f(1)=0f(1)=0, we have gβ(0)=0.g_β(0)=0. (42) Differentiating with respect to V gives gβ′(V) g_β (V) =f′(1+Vβ)−f′(1−V1−β), =f (1+ Vβ )-f (1- V1-β ), (43) gβ′(V) g_β (V) =1βf′(1+Vβ)+11−βf′(1−V1−β). = 1βf (1+ Vβ )+ 11-βf (1- V1-β ). (44) Consequently, gβ′(0)=0,gβ′(0)=f′(1)β(1−β).g_β (0)=0, g_β (0)= f (1)β(1-β). (45) Applying Taylor’s theorem around V=0V=0, we obtain gβ(V) g_β(V) =f′(1)2β(1−β)V2+o(V2),V→0. = f (1)2β(1-β)V^2+o(V^2), V→ 0. (46) Therefore, for every fixed β∈(0,1)β∈(0,1), Df(P∥Q)≥f′(1)2β(1−β)DTV(P,Q)2+o(DTV(P,Q)2).D_f(P Q)≥ f (1)2β(1-β)D_TV(P,Q)^2+o (D_TV(P,Q)^2 ). (47) Since β(1−β)≤14,β(1-β)≤ 14, the smallest possible local quadratic coefficient over β∈(0,1)β∈(0,1) is Cflocal=minβ∈(0,1)f′(1)2β(1−β)=2f′(1),C_f^local= _β∈(0,1) f (1)2β(1-β)=2f (1), (48) attained at β=1/2β=1/2. This local coefficient should be distinguished from the global Vajda constant CfVajda=inf0<V≤1Lf(V)V2.C_f^Vajda= _0<V≤ 1 L_f(V)V^2. (49) The former characterizes the local curvature of the binary lower bound around V=0V=0, whereas the latter controls the entire range V∈(0,1]V∈(0,1]. Consequently, CfVajda≤Cflocal=2f′(1),C_f^Vajda≤ C_f^local=2f (1), (50) with equality when the global minimum of Lf(V)/V2L_f(V)/V^2 is attained asymptotically at V=0V=0. Appendix B Variance Reduction of Atomic Estimation We next analyze the variance reduction induced by Atomic Estimation. Let h(a)=f(r(s,a)),a∼μ(⋅∣s),h(a)=f(r(s,a)), a μ(· s), (51) so that Df(π∥μ)=∑aμ(a∣s)h(a).D_f(π μ)= _aμ(a s)h(a). (52) Since the variance of a random contribution decomposes into its second moment minus its squared mean, we focus on the second-moment term, which captures the contribution of large-magnitude samples and is the dominant source of variance in long-tailed importance-weighted estimation. Under MC estimation, the sampled divergence contribution is D^fMC=h(a). D_f^MC=h(a). (53) Its second moment is a∼μ[(D^fMC)2]=∑aμ(a∣s)h(a)2.E_a μ [ ( D_f^MC )^2 ]= _aμ(a s)h(a)^2. (54) Thus, the contribution of action a to the second moment is vMC(a)=μ(a∣s)h(a)2.v_MC(a)=μ(a s)h(a)^2. (55) Atomic Estimation instead evaluates the probability-weighted atomic contribution D^fAtom=μ(a∣s)h(a). D_f^Atom=μ(a s)h(a). (56) Its corresponding second moment is a∼μ[(D^fAtom)2]=∑aμ(a∣s)3h(a)2,E_a μ [ ( D_f^Atom )^2 ]= _aμ(a s)^3h(a)^2, (57) so the contribution of action a becomes vAtom(a)=μ(a∣s)3h(a)2.v_Atom(a)=μ(a s)^3h(a)^2. (58) Consequently, vAtom(a)=μ(a∣s)2vMC(a)≤vMC(a),v_Atom(a)=μ(a s)^2v_MC(a)≤ v_MC(a), (59) where the inequality follows from μ(a∣s)≤1μ(a s)≤ 1. The suppression is particularly strong for rare actions. For μ(a∣s)≪1μ(a s) 1, the second-moment contribution of action a is reduced by the additional factor μ(a∣s)2μ(a s)^2 under Atomic Estimation. Hence, even when a rare action exhibits a large ratio deviation and consequently a large raw value of h(a)h(a), its contribution to the probability-weighted proximal constraint remains controlled. This analysis shows that Atomic Estimation mitigates the long-tail sensitivity of MC-based constraints by suppressing the second-moment contribution of low-probability actions. It therefore provides a probability-aware mechanism for improving the stability of proximal estimation, while distinguishing the magnitude of a ratio deviation from its actual contribution to the overall divergence.