Paper deep dive
Group Orthogonalized Policy Optimization:Group Policy Optimization as Orthogonal Projection in Hilbert Space
Wang Zixian
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 7/20/2026, 1:47:45 PM
Summary
The paper introduces Group Orthogonalized Policy Optimization (GOPO), a novel alignment algorithm for Large Language Models that reformulates policy optimization within the geometry of Hilbert function spaces (L2). By lifting alignment from the probability simplex to a Hilbert space, GOPO transforms the probability conservation constraint into a linear orthogonality condition. This allows the optimal policy update to be derived via the Hilbert Projection Theorem, resulting in a work-dissipation functional with constant Hessian curvature and non-saturating gradients. The method introduces a Bounded Hilbert Projection to enforce exact sparsity for poor actions and utilizes group sampling to empirically realize the theory, where group-normalized advantages cause the chemical potential (Lagrange multiplier) to vanish, simplifying the loss function. Experiments demonstrate competitive performance on mathematical reasoning benchmarks compared to methods like GRPO and PPO.
Entities (9)
Relation Signals (8)
GOPO → derivedfrom → Hilbert Space
confidence 95% · GOPO is derived from the geometry of Hilbert function spaces L2(pi_k).
GOPO → uses → Hilbert Projection Theorem
confidence 95% · The global maximum of this functional is then given elegantly by the Hilbert Projection Theorem.
GOPO → employs → Bounded Hilbert Projection
confidence 90% · enforcing the non-negativity boundary v>=-1 upgrades the solution to a Bounded Hilbert Projection (BHP)
Bounded Hilbert Projection → produces → Exact Sparsity
confidence 90% · yielding exact sparsity: catastrophically poor actions receive zero target probability via a hard analytical threshold.
GOPO → replaces → KL Divergence
confidence 90% · Unlike existing methods that formulate alignment as constrained optimization on the probability simplex... inheriting the exponential curvature of Kullback–Leibler divergence, we propose a fundamentally different approach
GOPO → achieves → Vanishing Chemical Potential
confidence 85% · because group-normalized advantages inherently sum to zero, the Lagrange multiplier (chemical potential) enforcing probability conservation exactly vanishes
GOPO → outperformsorcompeteswith → GRPO
confidence 85% · Experiments on mathematical reasoning benchmarks demonstrate that GOPO achieves competitive generalization (47% on MATH Level 4 vs. 44% for GRPO/DAPO)
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We present Group Orthogonalized Policy Optimization (GOPO), a new alignment algorithm for large language models derived from the geometry of Hilbert function spaces. Instead of optimizing on the probability simplex and inheriting the exponential curvature of Kullback-Leibler divergence, GOPO lifts alignment into the Hilbert space L2(pi_k) of square-integrable functions with respect to the reference policy. Within this space, the simplex constraint reduces to a linear orthogonality condition <v, 1> = 0, defining a codimension-one subspace H0. Minimizing distance to an unconstrained target u_star yields the work-dissipation functional J(v) = <g, v> - (mu / 2) ||v||^2, whose maximizer follows directly from the Hilbert projection theorem. Enforcing the boundary v >= -1 produces a bounded Hilbert projection that induces exact sparsity, assigning zero probability to catastrophically poor actions through a closed-form threshold. To connect this functional theory with practice, GOPO projects from infinite-dimensional L2(pi_k) to a finite empirical subspace induced by group sampling. Because group-normalized advantages sum to zero, the Lagrange multiplier enforcing probability conservation vanishes exactly, reducing the constrained projection to an unconstrained empirical loss. The resulting objective has constant Hessian curvature mu I, non-saturating linear gradients, and an intrinsic dead-zone mechanism without heuristic clipping. Experiments on mathematical reasoning benchmarks show that GOPO achieves competitive generalization while maintaining stable gradient dynamics and entropy preservation in regimes where clipping-based methods plateau.
Tags
Links
- Source: https://arxiv.org/abs/2602.21269v1
- Canonical: https://arxiv.org/abs/2602.21269v1
Trouble viewing inline? Open PDF directly →
Full Text
50,553 characters extracted from source content.
Expand or collapse full text
Group Orthogonalized Policy Optimization Group Policy Optimization as Orthogonal Projection in Hilbert Space Wang Zixian China Mobile Communications Group Shandong Co., Ltd. Tai’an Branch wangzixian@sd.chinamobile.com Abstract We present Group Orthogonalized Policy Optimization (GOPO), a new alignment algorithm for large language models derived entirely from the geometry of Hilbert function spaces. Unlike existing methods that formulate alignment as constrained optimization on the probability simplex—inevitably inheriting the exponential curvature of Kullback–Leibler divergence—we propose a fundamentally different approach: lifting the problem into the Hilbert space L2(πk)L^2( _k) of square-integrable functions with respect to the current reference policy. In this function space, we make three observations that jointly yield a principled algorithm. First, the probability simplex constraint reduces to a single linear orthogonality condition ⟨v,⟩=0 v,1 =0, defining a codimension-one closed subspace ℋ0H_0. Second, the alignment objective emerges naturally from the Geometric Principle of Minimum Distance: minimizing the geometric distance between v and the unconstrained target u∗u^* algebraically yields the work-dissipation functional (v)=⟨g,v⟩−μ2‖v‖2J(v)= g,v - μ2\|v\|^2. The global maximum of this functional is then given elegantly by the Hilbert Projection Theorem. Third, enforcing the non-negativity boundary v≥−1v≥-1 upgrades the solution to a Bounded Hilbert Projection (BHP), yielding exact sparsity: catastrophically poor actions receive zero target probability via a hard analytical threshold. To bridge the gap between this global functional theory and practical training, GOPO transitions the projection from the infinite-dimensional L2(πk)L^2( _k) to a finite-dimensional empirical subspace induced by group sampling. A key structural result emerges: because group-normalized advantages inherently sum to zero, the Lagrange multiplier (chemical potential) enforcing probability conservation exactly vanishes, collapsing the constrained projection into an unconstrained empirical loss. The resulting objective features constant Hessian curvature μIμ I, non-saturating linear gradients, and an intrinsic dead-zone mechanism—all without heuristic clipping. Experiments on mathematical reasoning benchmarks demonstrate that GOPO achieves competitive generalization while sustaining healthy gradient dynamics and entropy preservation in regimes where clipping-based methods plateau. 1 Introduction Aligning Large Language Models (LLMs) with human preferences via Reinforcement Learning from Human Feedback (RLHF) [5, 6] has become a standard practice. Dominant algorithms—Proximal Policy Optimization (PPO) [2], Direct Preference Optimization (DPO) [1], and Group Relative Policy Optimization (GRPO) [4]—all share a common structural element: they regularize policy updates via the Kullback–Leibler (KL) divergence, either explicitly as a penalty term or implicitly through ratio clipping in log-probability space. KL-regularized objectives carry an inherent geometric limitation. The KL divergence DKL(πθ∥πref)D_KL( _θ\| _ref) induces an exponential geometry: its Hessian in log-ratio coordinates scales as σ(m)(1−σ(m))σ(m)(1-σ(m)), where m is the logit margin. This data-dependent curvature means that as the policy becomes more confident—producing large log-ratios—the gradient signal decays exponentially. The resulting gradient saturation forces practitioners to either accept premature plateaus or resort to heuristic mechanisms such as aggressive ratio clipping [2], asymmetric clip bounds [17], or entropy bonuses. We argue that this limitation is not intrinsic to alignment but rather to the choice of geometric space in which optimization is performed. The probability simplex, equipped with KL geometry, conflates two independent design choices: what to learn (the sampling/advantage signal) and how to regularize (the optimization curvature). Adjusting one inevitably perturbs the other. A Hilbert Space Perspective. In this work, we abandon the probability simplex altogether and reformulate alignment as an optimization problem in the Hilbert function space L2(πk)L^2( _k). This shift yields three structural advantages: 1. The probability conservation constraint ∑yπ(y)=1 _yπ(y)=1 becomes a linear orthogonality condition ⟨v,⟩πk=0 v,1 _ _k=0, where v(y)=π(y)/πk(y)−1v(y)=π(y)/ _k(y)-1 is the density fluctuation field. This defines a closed linear subspace ℋ0⊂L2(πk)H_0⊂ L^2( _k). 2. The optimal policy update is obtained naturally by the Hilbert Projection Theorem: the physically valid probability distribution that is geometrically closest to the unconstrained target. No heuristic clipping or ad-hoc normalization is needed; the classic work-dissipation functional emerges strictly as an algebraic consequence of this minimum distance. 3. The Hessian of the resulting objective is the constant scalar μIμ I, completely independent of the data distribution or the current policy state. This constant curvature guarantee ensures that gradients remain linear and non-saturating throughout training. Building on the functional-analytic foundation established in Orthogonalized Policy Optimization (OPO) [18], we derive Group Orthogonalized Policy Optimization (GOPO), which seamlessly transitions the orthogonal projection from the infinite-dimensional global space L2(πk)L^2( _k) to a finite-dimensional empirical subspace induced by group sampling. A particularly elegant result emerges at the group level: because standardized advantages structurally sum to zero, they already reside in the zero-mean subspace, and the chemical potential enforcing probability conservation exactly vanishes. GOPO thus reduces to a simple, unconstrained quadratic loss over policy ratios. Contributions. • We formulate LLM alignment as a constrained optimization in L2(πk)L^2( _k) and show that the optimal policy is the orthogonal projection of the advantage-driven target onto the probability-conserving subspace. • We extend the unconstrained projection to the Bounded Hilbert Projection (BHP), which enforces non-negativity of the target policy and yields exact sparsity for catastrophically poor actions. • We derive GOPO as the empirical realization of this framework, proving that the chemical potential vanishes under group normalization, producing a practical loss with constant curvature and an intrinsic dead-zone mechanism. • Experiments on mathematical reasoning benchmarks demonstrate competitive generalization (47% on MATH Level 4 vs. 44% for GRPO/DAPO), monotonically improving validation accuracy, and the healthiest gradient dynamics among all tested methods. 2 Related Work Preference Optimization and RLHF. RLHF [5, 6] typically involves learning a reward model from preferences and optimizing a policy via PPO [2]. DPO [1] simplifies this by deriving a closed-form solution to the KL-constrained problem. IPO [7] adds a regularization term; SimPO [8] simplifies the reference-free objective. GRPO [4] eliminates the critic network by using group-relative advantages. All these methods inherit the exponential geometry of KL divergence, leading to gradient saturation in high-confidence regimes. f-Divergences and Alternative Geometries. The f-divergence family [9, 10] provides a unified framework for distributional discrepancy. The Pearson χ2χ^2 divergence, a member of this family, induces quadratic rather than exponential penalties. Prior work has explored f-divergences in variational inference [11], GANs [12], and imitation learning [13]. In RL, α [14] studied α-divergence as a trust-region constraint. APO [15] explored combining forward and reverse KL dynamics. OPO [18] first proposed lifting alignment into L2(πk)L^2( _k) and deriving the optimal update via the Hilbert Projection Theorem. GOPO extends this framework from the global Hilbert space to empirical group-level subspaces, proving that the chemical potential vanishes under group normalization and introducing the Bounded Hilbert Projection for exact sparsity. Trust-Region Methods. TRPO [3] enforces stability via explicit KL constraints. PPO [2] approximates this with ratio clipping. DAPO [17] introduces dynamic clip bounds and entropy management. ADPO [16] uses anchored coordinates for implicit trust regions. GOPO replaces all such mechanisms with a single quadratic penalty in ratio space, whose geometric justification is the Hilbert Projection Theorem rather than heuristic approximation. 3 Theoretical Framework: Policy Optimization in Hilbert Space We develop the mathematical apparatus underlying GOPO. The central idea is to reformulate policy optimization in the Hilbert space L2(πk)L^2( _k), where the probability simplex constraint becomes a linear subspace condition and the optimal policy update is obtained by orthogonal projection. 3.1 The Hilbert Space of Density Fluctuations Let πk _k denote the current reference policy (typically the policy from the previous iteration under on-policy anchoring). We define the ambient space as the Hilbert space of square-integrable functions with respect to πk _k: ℋ=L2(πk),⟨f,g⟩πk=πk[f(y)g(y)]H=L^2( _k), f,g _ _k=E_ _k[f(y)g(y)] (1) with induced norm ‖f‖πk=⟨f,f⟩πk\|f\|_ _k= f,f _ _k. Instead of directly optimizing the probability vector π(y|x)π(y|x) on the simplex, we shift to a centered coordinate system. Define the density fluctuation field: v(y)=π(y)πk(y)−1v(y)= π(y) _k(y)-1 (2) The target policy is exactly recovered as π(y)=πk(y)(1+v(y))π(y)= _k(y)(1+v(y)). The field v lives in ℋH and represents the relative deviation of the target policy from the reference. This coordinate change is the key step that transforms the non-linear simplex geometry into a linear Hilbert space geometry. 3.2 Probability Conservation as Orthogonality Any valid probability distribution must satisfy ∑yπ(y)=1 _yπ(y)=1. In the fluctuation coordinate, this fundamental physical law simplifies dramatically: ∑yπ(y)=∑yπk(y)(1+v(y))=1+∑yπk(y)v(y)=1+πk[v]=1 _yπ(y)= _y _k(y)(1+v(y))=1+ _y _k(y)v(y)=1+E_ _k[v]=1 (3) Hence, probability conservation is equivalent to the zero-mean condition: πk[v]=⟨v,⟩πk=0E_ _k[v]= v,1 _ _k=0 (4) This is a single linear constraint in ℋH, defining a closed subspace of codimension one: ℋ0=f∈ℋ:⟨f,⟩πk=0H_0=\f : f,1 _ _k=0\ (5) Remark 3.1 (Geometric Interpretation). ℋ0H_0 is precisely the orthogonal complement of the one-dimensional subspace spanned by the constant function 1 in L2(πk)L^2( _k). Every valid policy fluctuation must be orthogonal to 1—probability mass gained at some outputs must be exactly compensated by mass lost elsewhere. The term “orthogonalized” in our method name refers directly to this geometric fact. 3.3 The Driving Force: Metric Modulation In reinforcement learning, the optimization is driven by the advantage function A(y)A(y). To incorporate advanced sampling strategies, we define a Metric Modulation Operator ℳαM_α that reshapes the raw advantage into an effective driving field. Let Eα(y)E_α(y) be an α-parameterized escort multiplier that reweights contributions according to the density ratio: Eα(y)=(π(y)πk(y))αE_α(y)= ( π(y) _k(y) )^α (6) The multiplication operator ℳαM_α acts on functions in ℋH as (ℳαf)(y)=Eα(y)f(y)(M_αf)(y)=E_α(y)f(y). The effective driving field is: gα(y)=Eα(y)⋅A(y)g_α(y)=E_α(y)· A(y) (7) When α=0α=0, the raw advantage drives the update directly. Nonzero α focuses the driving force toward regions where the policy already assigns significant mass, controlling the exploit-explore tradeoff independently of the optimization geometry. 4 Methodology: Group Orthogonalized Policy Optimization We now derive GOPO in four stages: (i) identifying the unconstrained target and formulating the geometric minimum-distance principle, (i) showing that the classic work-dissipation functional emerges algebraically from this distance, (i) obtaining the closed-form solution via the Hilbert Projection Theorem and its bounded extension, and (iv) transitioning to the empirical group-level objective. 4.1 The Geometric Principle: Minimum Distance to the Unconstrained Target Suppose for a moment that there were no physical constraints on the probability distribution. Driven entirely by the external metric-modulated field gαg_α, the ideal, unconstrained policy fluctuation would simply align with the driving force, scaled by a compliance (or stiffness) parameter μ>0μ>0: u∗=gαμu^*= g_αμ (8) However, the reality is that any valid fluctuation field v must strictly obey probability conservation, locking it within the zero-mean hyperplane ℋ0H_0. The unconstrained target u∗u^* typically violates this constraint because πk[gα]≠0E_ _k[g_α]≠ 0. What is the most geometric and principled way to resolve this? We seek the valid fluctuation v∈ℋ0v _0 that is geometrically closest to the ideal target u∗u^* in the Hilbert space. Mathematically, this corresponds to minimizing the L2L^2 distance (scaled by μ/2μ/2 for dimension consistency): minv∈ℋ0μ2‖v−u∗‖πk2 _v _0\; μ2 \|v-u^* \|^2_ _k (9) 4.2 Emergence of the Work-Dissipation Functional By algebraically expanding this pure geometric distance, a remarkable physical structure naturally emerges. Substituting u∗=gα/μu^*=g_α/μ, we have: μ2‖v−gαμ‖πk2 μ2 \|v- g_αμ \|^2_ _k =μ2(‖v‖πk2−2μ⟨gα,v⟩πk+1μ2‖gα‖πk2) = μ2 (\|v\|^2_ _k- 2μ g_α,v _ _k+ 1μ^2\|g_α\|^2_ _k ) (10) =μ2‖v‖πk2⏟Dissipation−⟨gα,v⟩πk⏟Work+12μ‖gα‖πk2⏟Constant independent of v = μ2\|v\|^2_ _k_Dissipation- g_α,v _ _k_Work+ 12μ\|g_α\|^2_ _k_Constant independent of v (11) Minimizing the geometric distance is therefore strictly equivalent to maximizing the negative of the v-dependent terms. Thus, the alignment objective becomes: argminv∈ℋ0μ2‖v−u∗‖πk2≡argmaxv∈ℋ0(⟨gα,v⟩πk⏟External Work−μ2‖v‖πk2⏟Quadratic Dissipation) _v _0 μ2 \|v-u^* \|^2_ _k≡ _v _0 ( g_α,v _ _k_External Work- μ2\|v\|^2_ _k_Quadratic Dissipation ) (12) The classic “work-dissipation” functional (v)=⟨gα,v⟩πk−μ2‖v‖πk2J(v)= g_α,v _ _k- μ2\|v\|^2_ _k is not an arbitrarily constructed objective; it is the algebraic inevitability of measuring squared distance in a Hilbert space. 4.3 The Orthogonal Projection Solution Because the problem is fundamentally one of finding the closest point in a closed subspace ℋ0H_0 to a target vector u∗u^*, we do not need to rely on Lagrangian differentiation. The Hilbert Projection Theorem directly yields the unique global optimum as the orthogonal projection: Theorem 4.1 (Optimal Fluctuation via Orthogonal Projection). The optimal probability-conserving fluctuation v∗v^* is the orthogonal projection of the unconstrained target u∗u^* onto ℋ0H_0: v∗=Pℋ0(u∗)=u∗−⟨u∗,⟩πk=1μ(gα−πk[gα]⋅)v^*=P_H_0(u^*)=u^*- u^*,1 _ _k1= 1μ (g_α-E_ _k[g_α]·1 ) (13) Proof. Since ℋ0=⟂H_0=\1\ , the orthogonal projection Pℋ0P_H_0 onto ℋ0H_0 is: Pℋ0(f)=f−⟨f,⟩πk‖πk2=f−πk[f]⋅P_H_0(f)=f- f,1 _ _k\|1\|^2_ _k1=f-E_ _k[f]·1 (14) using ‖πk2=πk[1]=1\|1\|^2_ _k=E_ _k[1]=1. Applying this to f=u∗=gα/μf=u^*=g_α/μ yields the result. ∎ Remark 4.2 (The Chemical Potential). The subtracted term λ∗=πk[gα]/μλ^*=E_ _k[g_α]/μ is the Lagrange multiplier enforcing probability conservation. In our framework, it obtains the purest geometric interpretation: it is exactly the orthogonal component of the target vector u∗u^* along the illegal normal direction (the constant function 1) that is explicitly sliced off by the projection. 4.4 Bounded Hilbert Projection and Exact Sparsity The unconstrained projection (13) may produce v∗(y)<−1v^*(y)<-1 for actions with strongly negative driving signals, yielding invalid negative probabilities. To enforce the physical constraint π(y)≥0π(y)≥ 0, equivalently v(y)≥−1v(y)≥-1, we extend the projection to a Bounded Hilbert Projection (BHP). Define the positive orthant +=f∈ℋ:f(y)≥−1,∀yC_+=\f :f(y)≥-1,∀ y\ and the feasible convex set: =ℋ0∩+K=H_0 _+ (15) The BHP problem seeks the point in K closest to gα/μg_α/μ. We formulate the KKT conditions via the Lagrangian: ℒ(v,λ,η)=πk[gαv−μ2v2−λv+η(v+1)]L(v,λ,η)=E_ _k\! [g_αv- μ2v^2-λ v+η(v+1) ] (16) where λ enforces πk[v]=0E_ _k[v]=0 and the KKT multiplier η(y)≥0η(y)≥ 0 enforces v(y)≥−1v(y)≥-1. Theorem 4.3 (Bounded Projection Solution). The KKT complementarity condition η(y)(v(y)+1)=0η(y)(v(y)+1)=0 yields the closed-form bounded solution: v∗(y)=max(−1,gα(y)−λ∗μ)v^*(y)= \! (-1,\; g_α(y)-λ^*μ ) (17) where the dynamic chemical potential λ∗λ^* is the unique value ensuring πk[v∗]=0E_ _k[v^*]=0. This result has a profound theoretical consequence: Corollary 4.4 (Exact Sparsity). Actions satisfying gα(y)<λ∗−μg_α(y)<λ^*-μ are assigned v∗(y)=−1v^*(y)=-1, i.e., their target probability is exactly zero: π(y)=πk(y)(1+v∗(y))=0π(y)= _k(y)(1+v^*(y))=0. The BHP thus provides an analytical mechanism for eliminating catastrophically poor outputs (e.g., hallucinations) via a hard, data-adaptive threshold, without requiring any external filtering heuristic. 1 (Constant Function)Unconstrained Targetu∗=gα/μu^*=g_α/μ λ∗λ^* (Chemical Potential) Truncation (Dead Zone) GOPO Policyv∗=P(u∗)v^*\!=\!P_K(u^*) Reference πk _k (v=0v\!=\!0)Feasible Polytope=ℋ0∩+K=H_0 _+ ℋ0H_0: Probability Conserving Subspace⟨v,⟩πk=0 v,1 _ _k=0 Figure 1: Geometric interpretation of GOPO. The theory operates in L2(πk)L^2( _k). The reference policy πk _k sits at the origin (v=0v=0). Valid policies must reside in ℋ0H_0 (gray plane) and satisfy non-negativity, restricting them to the feasible polytope K (green). The unconstrained target u∗=gα/μu^*=g_α/μ (red) is first projected vertically onto ℋ0H_0 by subtracting the chemical potential λ∗λ^*, then truncated along the plane onto the K boundary (dead zone), yielding the final bounded GOPO update v∗v^* (blue). 4.5 From Global to Empirical: The GOPO Objective Optimizing the distance minimization (9) in the global space L2(πk)L^2( _k) requires access to the full distribution, which is computationally intractable for autoregressive language models. GOPO bridges this gap by transitioning the orthogonal projection to an empirical discrete subspace. For a given prompt x, sample a group of G responses =y1,…,yG∼πk(⋅|x)G=\y_1,…,y_G\ _k(·|x). Define the empirical inner product space L^2() L^2(G) with metric: ⟨f,g⟩=1G∑i=1Gf(yi)g(yi) f,g _G= 1G _i=1^Gf(y_i)g(y_i) (18) The probability conservation constraint is naturally relaxed to the empirical zero-mean condition: ℋ^0=v∈L^2():1G∑i=1Gv(yi)=0 H_0= \v∈ L^2(G): 1G _i=1^Gv(y_i)=0 \ (19) The Vanishing Chemical Potential. In standard RLHF pipelines, raw rewards are normalized into group-relative advantages Ai=ri−r¯A_i=r_i- r, satisfying ∑i=1GAi=0 _i=1^GA_i=0 by construction. Consequently, the empirical driving vector =[A1,…,AG]⊤ A=[A_1,…,A_G] intrinsically resides in ℋ^0 H_0. Because the driving force is already orthogonal to 1, the projection operator Pℋ^0P_ H_0 acts as the identity on A: Pℋ^0()=−1G∑i=1GAi⋅=P_ H_0( A)= A- 1G _i=1^GA_i·1= A (20) The chemical potential λ∗λ^* required to enforce probability conservation exactly vanishes at the empirical group level. Remark 4.5 (Structural Simplification). This vanishing is not a numerical approximation but a structural consequence of group normalization. It eliminates the need to solve for λ∗λ^*, collapsing the constrained projection into an unconstrained optimization problem. The GOPO Loss Function. Substituting the empirical advantage AiA_i for the driving field gαg_α, and replacing the analytical fluctuation v(yi)v(y_i) with the parameterized density ratio ρθ(yi)=πθ(yi|x)/πk(yi|x) _θ(y_i)= _θ(y_i|x)/ _k(y_i|x), the constrained functional maximization (12) reduces to: ℒGOPO(θ)=−x∼[1G∑i=1G[Aiρθ(yi)−μ2(ρθ(yi)−1)2]] L_GOPO(θ)=-E_x [ 1G _i=1^G [A_i _θ(y_i)- μ2 ( _θ(y_i)-1 )^2 ] ] (21) Structural Comparison with GRPO. It is instructive to contrast (21) with the GRPO loss [4]: ℒGRPO(θ)=−x∼[1G∑i=1G1|yi|∑t=1|yi|min(ρθ(yi,t)A^i,clip(ρθ(yi,t),1−ϵ,1+ϵ)A^i)−βDKL(πθ∥πk)]L_GRPO(θ)=-E_x [ 1G _i=1^G 1|y_i| _t=1^|y_i| \! ( _θ(y_i,t) A_i,\;clip( _θ(y_i,t),1-ε,1+ε) A_i )-β D_KL( _θ\| _k) ] (22) where A^i A_i denotes the standardized advantage, ρθ(yi,t) _θ(y_i,t) is the token-level density ratio, and ϵε is the clip radius. The two losses share the same fundamental motivation—optimizing a policy via group-sampled responses and normalized advantages—but differ in three fundamental ways: 1. Trust region mechanism. GRPO enforces the trust region via hard clipping of the token-level ratio ρθ(yi,t) _θ(y_i,t) alongside an explicit KL penalty, which introduces a non-smooth, piecewise-linear landscape with zero-gradient plateaus wherever |ρθ−1|>ϵ| _θ-1|>ε. GOPO replaces this with a smooth quadratic penalty μ2(ρθ−1)2 μ2( _θ-1)^2 on the sequence-level ratio that continuously penalizes deviation from the reference—the ratio can move freely while the restoring force grows linearly, never producing a gradient dead zone. 2. Optimization curvature. The Hessian of GRPO’s clipped surrogate is either 0 (inside the flat clipped region) or undefined (at the clip boundary); its effective curvature is data- and state-dependent. GOPO’s Hessian is the constant μ, yielding a uniformly convex landscape with predictable convergence (Theorem˜5.1). 3. Gradient behavior at convergence. As ρθ→1 _θ→ 1 and the policy approaches the reference, the GRPO gradient is dominated by the advantage-weighted policy gradient A^i∇θlogπθ A_i _θ _θ, which scales with the advantage magnitude but does not account for the distance already traveled. GOPO’s gradient −Ai+μ(ρθ−1)-A_i+μ( _θ-1) is proportional to the displacement from equilibrium, providing a natural deceleration as the policy approaches its target ρ∗=1+Ai/μρ^*=1+A_i/μ. In summary, GOPO trades GRPO’s hard combinatorial constraint (clip) for a soft geometric one (projection), gaining smoothness, constant curvature, and a principled equilibrium structure at no additional computational cost. Bounded GOPO Loss. The BHP truncation logic is absorbed into the loss bounds via: ℒGOPOBHP(θ)=1G∑i=1Gmax(0,−Aiρθ(yi)+μ2(ρθ(yi)−1)2)L_GOPO^BHP(θ)= 1G _i=1^G \! (0,\;-A_i _θ(y_i)+ μ2 ( _θ(y_i)-1 )^2 ) (23) where the max(0,⋅) (0,·) acts as a ReLU-like floor, halting gradients once ρθ(yi)→0 _θ(y_i)→ 0 for highly negative advantages. 5 Theoretical Analysis 5.1 Constant Curvature and Decoupled Geometry The primary vulnerability of PPO, DPO, and GRPO is their reliance on the exponential divergence DKLD_KL. The Hessian of KL-based objectives scales non-linearly with the policy output, causing gradient saturation that must be mitigated via heuristic clipping. Theorem 5.1 (Constant Curvature Optimization). Let ℓ(ρ)=−Aρ+μ2(ρ−1)2 (ρ)=-Aρ+ μ2(ρ-1)^2 be the pointwise un-truncated GOPO loss. The Hessian with respect to the density ratio ρ is the constant scalar μ, independent of the advantage signal A, the current policy state, or the data distribution. Proof. ∇ρℓ=−A+μ(ρ−1) _ρ =-A+μ(ρ-1), hence ∇ρ2ℓ=μ∇^2_ρ =μ. ∎ This constant curvature has immediate consequences for the decoupling of design axes: Corollary 5.2 (Structural Decoupling). In the GOPO objective, the first-order driving force depends on the advantage signal: ∇ρℓ=−A+μ(ρ−1) _ρ =-A+μ(ρ-1) (24) while the second-order curvature ∇ρ2ℓ=μ∇^2_ρ =μ is independent of A. Consequently, changing the advantage weighting (sampling geometry) does not alter the optimization landscape curvature (optimization geometry), and vice versa. Remark 5.3 (Contrast with KL-Based Methods). In DPO, the effective loss is ℓDPO≈−logσ(βm) _DPO≈- σ(β m), where m is the logit margin. The local curvature β2σ(m)(1−σ(m))β^2σ(m)(1-σ(m)) depends on both the temperature β and the current margin m. Changing β simultaneously alters the gradient magnitude and the stability profile. This data-dependent curvature is the root cause of gradient saturation. Corollary 5.4 (Global Linear Convergence). Gradient descent in ratio space on the GOPO objective follows the linear system: ρk+1−ρ∗=(1−ημ)(ρk−ρ∗) _k+1-ρ^*=(1-ημ)( _k-ρ^*) (25) where ρ∗=1+A/μρ^*=1+A/μ. For step size 0<η<2/μ0<η<2/μ, this exhibits global contraction to the unique equilibrium at rate |1−ημ||1-ημ|, independent of the advantage distribution. 5.2 Gradient Dynamics: Non-Saturation Guarantee We quantify the gradient behavior to contrast with KL-based methods. Gradient Saturation in DPO. For logistic losses, the gradient magnitude satisfies |∇mℓDPO|=|βσ(m)(1−σ(m))|≤β/4| _m _DPO|=|βσ(m)(1-σ(m))|≤β/4, and crucially, |∇mℓDPO|→0| _m _DPO|→ 0 exponentially as |m|→∞|m|→∞. Non-Saturating Gradient in GOPO. For the GOPO loss, the gradient magnitude is: |∇ρℓGOPO|=|−A+μ(ρ−1)|=μ|ρ−ρ∗|| _ρ _GOPO|=|-A+μ(ρ-1)|=μ|ρ-ρ^*| (26) This is exactly proportional to the distance from equilibrium. For any non-equilibrium state with |ρ−ρ∗|≥δ|ρ-ρ^*|≥δ: |∇GOPO|≥μδ| _GOPO|≥μδ (27) The gradient maintains a linear driving force that never vanishes away from equilibrium, preventing the saturation endemic to KL-based methods. 5.3 Dead Zone Dynamics and Gradient Hard Stops The BHP truncation establishes an implicit “dead zone” for gradient flow. Taking the derivative of the bounded GOPO loss (23): ∇θℒGOPOBHP∝1G∑i=1G(ρθ(yi)>0)⏟Dead Zone Gate⋅(−Ai+μ(ρθ(yi)−1))⏟Restoring Force∇θρθ(yi) _θL_GOPO^BHP 1G _i=1^G I\! ( _θ(y_i)>0 )_Dead Zone Gate· (-A_i+μ( _θ(y_i)-1) )_Restoring Force _θ _θ(y_i) (28) When an action receives an extremely negative advantage Ai≪0A_i 0, the ratio ρθ(yi) _θ(y_i) is rapidly suppressed. Once it approaches the physical boundary ρθ→0 _θ→ 0 (equivalently v(yi)→−1v(y_i)→-1), the indicator function triggers a hard stop: the gradient for this action is exactly zeroed out. This mechanism prevents the network from wasting representational capacity on already-suppressed outputs, redirecting gradient bandwidth exclusively toward distinguishing among higher-quality responses. 5.4 Connection to χ2χ^2 Divergence The Hilbert space framework provides a natural connection to statistical divergences. Proposition 5.5 (χ2χ^2 Interpretation). The dissipation term in the GOPO functional is proportional to the Pearson χ2χ^2 divergence: 12πk[(v(y))2]=12πk[(π(y)πk(y)−1)2]=Dχ2(π∥πk) 12E_ _k\! [(v(y))^2 ]= 12E_ _k\! [ ( π(y) _k(y)-1 )^2 ]=D_χ^2(π\| _k) (29) Thus, the GOPO objective can be equivalently read as maximizing advantage-weighted work subject to a χ2χ^2 trust region. Proposition 5.6 (Total Variation Bound). Bounding [v2]≤ϵE[v^2]≤ε via the χ2χ^2 penalty guarantees, by Jensen’s inequality: TV(π,πk)=12[|v|]≤12[v2]≤ϵ2TV(π, _k)= 12E[|v|]≤ 12 E[v^2]≤ ε2 (30) providing a distributional stability guarantee in Total Variation distance. 5.5 Comparison with Parametric L2 Regularization A natural question is whether GOPO’s quadratic penalty differs from standard weight decay λ2‖θ−θ0‖2 λ2\|θ- _0\|^2. The distinction is fundamental: 1. Functional vs. Parametric: GOPO’s penalty μ2[v2] μ2E[v^2] acts in the function space of probability ratios. It penalizes deviations where the policy assigns probability mass. Weight decay acts uniformly in parameter space, blind to output-level consequences. 2. Adaptive Restoring Force: The gradient μ(ρ−1)μ(ρ-1) pulls each specific output ratio back toward unity. Weight decay suppresses all weights uniformly, potentially hindering the formation of sharp, high-confidence reasoning chains. 3. Geometric Guarantee: The χ2χ^2 penalty provides distributional bounds (Proposition˜5.6). Weight decay offers no such output-space guarantee. 6 Algorithm and Implementation Algorithm 1: Group Orthogonalized Policy Optimization (GOPO) Input: Initial policy πθ _θ, stiffness μ, group size G, learning rate η for each iteration do 1. Set Reference: πk←πθ _k← _θ (on-policy anchoring) 2. Sample Group: For each prompt x, sample =y1,…,yG∼πk(⋅|x)G=\y_1,…,y_G\ _k(·|x) 3. Score and Normalize: Compute rewards rir_i; set Ai=ri−r¯A_i=r_i- r (∑Ai=0Σ A_i=0) 4. Compute Ratios: ρθ(yi)=πθ(yi|x)/πk(yi|x) _θ(y_i)= _θ(y_i|x)/ _k(y_i|x) 5. Compute Loss: ℒ=−1G∑i=1G[Aiρθ(yi)−μ2(ρθ(yi)−1)2]L=- 1G _i=1^G [A_i _θ(y_i)- μ2( _θ(y_i)-1)^2 ] 6. Update: θ←θ−η∇θℒθ←θ-η _θL end for Figure 2: GOPO algorithm. Step 3 guarantees that the advantage vector lies in the zero-mean subspace ℋ^0 H_0, eliminating the chemical potential. Steps 4–5 implement the empirical orthogonal projection with quadratic dissipation. Current Policy πθ _θ Group Sampling G Reward & Normalize AiA_i Hilbert Projection 1. Ratio ρ=πθ/πkρ= _θ/ _k 2. Work: AiρiA_i _i 3. Dissipation: μ2(ρi−1)2 μ2( _i-1)^2 Update θ Next Iteration Figure 3: GOPO flowchart. The core operation (green) implements the empirical orthogonal projection: the advantage signal provides the linear driving force, while the quadratic dissipation term provides the constant-curvature regularization. Implementation Notes. GOPO introduces no additional model forward/backward passes compared to GRPO. The only overhead is the computation of (ρθ−1)2( _θ-1)^2, which is negligible. The reference policy πk _k is set to the policy from the start of each iteration (on-policy anchoring), ensuring that the trust region ρ≈1ρ≈ 1 holds locally. No critic network is required. 7 Experiments We evaluate GOPO against four strong baselines on mathematical reasoning tasks, focusing on out-of-distribution generalization performance. 7.1 Setup Model and Data. We use Qwen3-1.7B as the base model, trained with the VERL framework [19] on 4× RTX 4090 GPUs. To stress-test sample efficiency and generalization, we deliberately use a small training set: approximately 10% of MATH Level 3 problems (sampled with seed 42). Validation is performed every 10 training steps on 100 held-out MATH Level 4 problems—a strictly harder difficulty tier not seen during training—to measure out-of-distribution generalization. All methods share identical hyperparameters: batch size 48, learning rate 2×10−62× 10^-6, 8 epochs over the training data, and G=6G=6 rollout generations per prompt. Baselines. • OPO [18]: The full Orthogonalized Policy Optimization with importance-sampling reweighting (ωα _α), from which GOPO is derived. • GRPO [4]: Token-level policy gradient with group-normalized advantages and PPO-style ratio clipping (ϵ=0.2ε=0.2). • GSPO: Sentence-level variant of GRPO with step-level advantage normalization. • DAPO [17]: Advanced baseline with asymmetric clip bounds, unnormalized advantages, and overlong reward shaping. GOPO uses stiffness μ=0.5μ=0.5, escort exponent α=0.5α=0.5, and on-policy anchoring. 7.2 Results Overall Performance. Table˜1 summarizes the final metrics for all methods. Table 1: Comparison of policy optimization algorithms on MATH benchmarks. All methods use Qwen3-1.7B trained on ∼ 10% of MATH Level 3, validated on 100 MATH Level 4 problems. Reward is averaged over the entire training process. Algorithm Mean Reward↑ Val Acc (L4)↑ Grad Norm Entropy GRPO 0.544 44% 0.674 0.115 DAPO 0.548 44% 0.213 0.126 GSPO 0.553 48% 0.623 0.128 OPO 0.558 48% 1.279 0.126 GOPO (Ours) 0.555 47% 1.029 0.134 Analysis. • Generalization vs. Training Reward. OPO and GOPO achieve the highest mean rewards (0.558 and 0.555) over the training process, which directly translates to their superior generalization performance (48% and 47% validation accuracy). In contrast, GRPO struggles with the lowest mean reward (0.544) and poorest generalization (44%), struggling to efficiently learn from the small training set. GOPO achieves competitive validation accuracy while maintaining an upward trajectory (Figure˜4b). • Gradient Dynamics. OPO and GOPO maintain substantially higher gradient norms (1.28 and 1.03) than ratio-clipped methods (GRPO: 0.67, GSPO: 0.62, DAPO: 0.21), empirically confirming the non-saturation prediction of Theorem˜5.1. DAPO’s severely diminished gradients (∼ 0.21) indicate the conservative clipping is overly restrictive. • Entropy Preservation. GOPO maintains the highest policy entropy (0.134) among all methods. We attribute this to the sequence-level nature of the GOPO update: unlike token-level methods (GRPO, DAPO) that impose dense per-token supervision forcing rapid mode collapse, GOPO aligns total trajectory probability without micromanaging individual tokens. Training Dynamics. Figure˜4 shows the full training trajectories across four diagnostic metrics. Figure 4: Training dynamics comparison. (a) Training reward: OPO and GOPO achieve the highest mean rewards over the training process. (b) Validation accuracy on MATH Level 4: OPO and GSPO lead at 48%, with GOPO at 47% and monotonically improving. (c) Gradient norm: OPO/GOPO maintain healthy norms throughout, while DAPO exhibits severe gradient saturation. (d) Policy entropy: GOPO preserves the most diversity, preventing premature mode collapse. Generalization Analysis. The validation accuracy trajectories (Figure˜4b) reveal important dynamics: • GOPO shows monotonic improvement in generalization (38% → 46% → 47%), suggesting the algorithm has not yet plateaued and would benefit from longer training. • OPO starts strong (49% at step 10) but exhibits non-monotonic behavior (50% → 44% → 48%), consistent with the importance-sampling variance inherent in the full OPO framework. • GRPO and DAPO plateau at 44%, despite GRPO’s higher training reward—a signature of the reward hacking phenomenon where clipping-based methods exploit the training distribution rather than learning generalizable reasoning patterns. 8 Discussion Why Hilbert Space? Our framework is not merely a notational change but a deliberate choice of the mathematical arena best suited to the problem’s intrinsic structure. Policy alignment fundamentally operates in an abstract, high-dimensional space where the action vocabulary can range from tens of thousands of tokens to unbounded sequences. In finite-dimensional Euclidean space, optimization over such objects requires explicit coordinates and encounters the curse of dimensionality. The Hilbert space L2(πk)L^2( _k), by contrast, is purpose-built for abstract infinite-dimensional analysis: it provides a complete inner product structure that makes distances, angles, and projections well-defined regardless of the underlying dimensionality. Consequently, mappings and constraints that appear intractably nonlinear on the probability simplex—such as the normalization condition ∑yπ(y)=1 _yπ(y)=1—collapse into elementary linear-algebraic operations (inner products, closed subspaces, orthogonal projections) once lifted into this space. The probability simplex, equipped with KL geometry, fundamentally entangles the sampling signal with the optimization curvature; the Hilbert space disentangles them entirely. The Hilbert Projection Theorem provides a principled replacement for heuristic clipping, and the chemical potential emerges as a natural geometric by-product of the projection rather than an ad-hoc normalization constant. Abstract Representations and the Geometry of Distance. The adoption of Hilbert space is not without precedent in sciences that confront fundamentally abstract, high-dimensional state spaces. In quantum mechanics, the state of a physical system is a vector in a Hilbert space, and all measurable predictions—probabilities, expectation values, transition amplitudes—reduce to inner products and projections within that space [20]. The power of this formalism lies precisely in its coordinate-free nature: the mathematical apparatus operates identically whether the state space is two-dimensional (a qubit) or infinite-dimensional (a quantum field), because the inner product provides a complete metric structure—distance, angle, and projection—independent of any particular basis. A structurally parallel situation arises in language model alignment. The output of an LLM is a probability distribution over an astronomically large discrete space of possible utterances. What we seek to optimize is not any individual coordinate of this distribution, but its global geometric relationship to a target: how “close” the aligned policy is to the ideal, constrained by the requirement of remaining a valid distribution. The L2(πk)L^2( _k) framework provides exactly the right notion of distance (‖v−u∗‖πk\|v-u^*\|_ _k) for this task—one that is complete, geometrically meaningful, and agnostic to the dimensionality of the action space. The alignment problem thus reduces to its purest geometric essence: find the nearest valid point to an ideal target, an operation that Hilbert space is uniquely equipped to perform via its Projection Theorem. This perspective also connects to a broader trend in computational cognitive science, where abstract representational spaces—from semantic embeddings to conceptual structures—are increasingly modeled as inner product spaces in which “similarity” corresponds to geometric proximity [21, 22]. The Hilbert space framework may thus offer a principled mathematical language not only for policy optimization but for any domain where the fundamental task is to navigate distances between abstract, high-dimensional representations under structural constraints. The Role of Group Normalization. The vanishing chemical potential at the group level is a structural consequence of the geometry. Group normalization ensures that the advantage vector already lies in the zero-mean subspace ℋ^0 H_0, so no projection is needed along the 1 direction. This eliminates an entire class of potential failure modes related to improper constraint enforcement. Relation to Existing Methods. • Setting μ→0μ→ 0 recovers an unconstrained policy gradient (no regularization). • GOPO’s quadratic penalty can be viewed as the Bregman divergence induced by the Euclidean mirror map Ψ(v)=12‖v‖2 (v)= 12\|v\|^2, connecting to the mirror descent literature. However, the Hilbert space derivation is more direct and reveals additional structure (subspaces, projections, chemical potential). • The dead-zone mechanism (Section˜5.3) provides a principled alternative to the clip-higher strategy of DAPO [17], with the advantage of arising from the geometry rather than being hand-designed. Limitations. GOPO introduces the stiffness parameter μ, which may require tuning. Our experiments focus on mathematical reasoning; validation on diverse domains (instruction following, code generation) remains future work. The BHP truncation is currently implemented via a soft ReLU approximation rather than the exact hard threshold derived in Theorem˜4.3; studying the impact of this approximation gap is an open direction. 9 Conclusion We have presented Group Orthogonalized Policy Optimization (GOPO), an alignment algorithm derived entirely from the geometry of Hilbert function spaces. By lifting policy optimization from the probability simplex into L2(πk)L^2( _k), we transform the nonlinear normalization constraint into a linear orthogonality condition. The alignment objective itself emerges algebraically from the geometric principle of minimum distance to the unconstrained target, and its closed-form solution is given by the Hilbert Projection Theorem. The Bounded Hilbert Projection extends this to enforce non-negativity, yielding exact sparsity for catastrophic outputs. At the empirical group level, the chemical potential vanishes under standard advantage normalization, producing a simple loss with constant curvature, non-saturating gradients, and an intrinsic dead-zone mechanism. Experiments on mathematical reasoning benchmarks confirm that GOPO sustains learning in high-confidence regimes where clipping-based methods plateau, achieving competitive generalization performance with the healthiest gradient dynamics and entropy preservation among all tested methods. References [1] R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS, 2023. [2] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal Policy Optimization Algorithms. arXiv:1707.06347, 2017. [3] J. Schulman, S. Levine, P. Moritz, M. I. Jordan, and P. Abbeel. Trust Region Policy Optimization. ICML, 2015. [4] Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300, 2024. [5] P. Christiano, J. Leike, T. B. Brown, M. Martic, S. Legg, and D. Amodei. Deep Reinforcement Learning from Human Preferences. NeurIPS, 2017. [6] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, et al. Training Language Models to Follow Instructions with Human Feedback. NeurIPS, 2022. [7] M. G. Azar, M. Rowland, B. Piot, D. Guo, D. Calandriello, M. Valko, and R. Munos. A General Theoretical Paradigm to Understand Learning from Human Preferences. AISTATS, 2024. [8] Y. Meng, M. Xia, and D. Chen. SimPO: Simple Preference Optimization with a Reference-Free Reward. NeurIPS, 2024. [9] I. Csiszár. Information-type measures of difference of probability distributions and indirect observations. Studia Sci. Math. Hungar., 2:299–318, 1967. [10] S. M. Ali and S. D. Silvey. A General Class of Coefficients of Divergence of One Distribution from Another. JRSS-B, 28(1):131–142, 1966. [11] Y. Li and R. E. Turner. Rényi Divergence Variational Inference. NeurIPS, 2016. [12] S. Nowozin, B. Cseke, and R. Tomioka. f-GAN: Training Generative Neural Samplers Using Variational Divergence Minimization. NeurIPS, 2016. [13] S. K. S. Ghasemipour, R. Zemel, and S. Gu. A Divergence Minimization Perspective on Imitation Learning Methods. CoRL, 2020. [14] H. Xu et al. Improving Proximal Policy Optimization with Alpha Divergence. Neurocomputing, 2023. [15] Z. Wang. APO: Alpha-Divergence Preference Optimization. arXiv:2512.22953, 2025. [16] Z. Wang. ADPO: Anchored Direct Preference Optimization. arXiv:2510.18913, 2025. [17] Q. Yu et al. DAPO: An Open-Source LLM Reinforcement Learning System. arXiv:2503.14476, 2025. [18] Z. Wang. Orthogonalized Policy Optimization: Policy Optimization as Orthogonal Projection in Hilbert Space. arXiv:2601.12415, 2026. [19] G. Sheng, C. Xu, S. Yuan, et al. HybridFlow: A Flexible and Efficient RLHF Framework. arXiv:2409.19256, 2024. [20] J. von Neumann. Mathematical Foundations of Quantum Mechanics. Springer, 1932. (English translation: Princeton University Press, 1955.) [21] P. Gärdenfors. Conceptual Spaces: The Geometry of Thought. MIT Press, 2000. [22] J. R. Busemeyer and P. D. Bruza. Quantum Models of Cognition and Decision. Cambridge University Press, 2012. Appendix A Proofs and Derivations A.1 Derivation of the Projection Operator Proof of Theorem˜4.1. The Hilbert space ℋ=L2(πk)H=L^2( _k) admits the orthogonal decomposition ℋ=ℋ0⊕spanH=H_0 \1\. For any f∈ℋf , the projection onto ℋ0H_0 is: Pℋ0(f)=f−⟨f,⟩πk⟨,⟩πk=f−πk[f]P_H_0(f)=f- f,1 _ _k 1,1 _ _k1=f-E_ _k[f] (31) As derived in Section˜4.2, minimizing μ2‖v−u∗‖2 μ2\|v-u^*\|^2 over v∈ℋ0v _0 is equivalent to maximizing (v)=⟨gα,v⟩−μ2‖v‖2J(v)= g_α,v - μ2\|v\|^2, since the two differ only by the constant 12μ‖gα‖2 12μ\|g_α\|^2. The minimum-distance problem has the unique solution v∗=Pℋ0(u∗)=Pℋ0(gα/μ)v^*=P_H_0(u^*)=P_H_0(g_α/μ) by the Hilbert Projection Theorem. Applying the projection formula: v∗=gαμ−πk[gαμ]=1μ(gα−πk[gα])v^*= g_αμ-E_ _k\! [ g_αμ ]= 1μ (g_α-E_ _k[g_α] ) (32) ∎ A.2 Derivation of the Bounded Hilbert Projection Proof of Theorem˜4.3. The KKT stationarity condition for the Lagrangian gives: gα(y)−μv(y)−λ+η(y)=0g_α(y)-μ v(y)-λ+η(y)=0 (33) Combined with the complementarity condition η(y)(v(y)+1)=0η(y)(v(y)+1)=0 and η(y)≥0η(y)≥ 0: Case 1: v(y)>−1v(y)>-1. Then η(y)=0η(y)=0, so v(y)=(gα(y)−λ)/μv(y)=(g_α(y)-λ)/μ. Case 2: v(y)=−1v(y)=-1. Then η(y)=λ−μ−gα(y)≥0η(y)=λ-μ-g_α(y)≥ 0, which requires gα(y)≤λ∗−μg_α(y)≤λ^*-μ. These cases unify as v∗(y)=max(−1,(gα(y)−λ∗)/μ)v^*(y)= (-1,(g_α(y)-λ^*)/μ), where λ∗λ^* solves πk[v∗]=0E_ _k[v^*]=0. ∎ A.3 Log-Ratio Approximation Lemma A.1 (Log-Ratio Approximation Error). Let Δθ(y)=logπθ(y)−logπk(y) _θ(y)= _θ(y)- _k(y) and vθ(y)=exp(Δθ)−1v_θ(y)= ( _θ)-1. For δ=‖Δθ‖∞<1δ=\| _θ\|_∞<1: |vθ−Δθ|≤12δ2eδ,|vθ2−Δθ2|≤δ3e2δ|v_θ- _θ|≤ 12δ^2e^δ, |v_θ^2- _θ^2|≤δ^3e^2δ (34) The GOPO loss using log-ratios Δθ _θ in place of vθ=ρθ−1v_θ= _θ-1 incurs an approximation error of O([Δθ3])O(E[ _θ^3]), which is controlled by the on-policy trust region. A.4 χ2χ^2-Constrained Maximization Duality Proposition A.2 (Lagrange Dual Equivalence). The constrained problem maxv∈L2(πk)πk[gαv] _v∈ L^2( _k)E_ _k[g_αv] subject to πk[v2]≤ϵE_ _k[v^2]≤ε has Lagrangian relaxation equal to the GOPO functional (v)=[gαv]−μ2[v2]J(v)=E[g_αv]- μ2E[v^2], where μ is the dual variable corresponding to the constraint radius ϵε. The optimal solution v∗=gα/μv^*=g_α/μ confirms that the stiffness μ purely controls the trust-region radius while gαg_α purely shapes the alignment direction.