Paper deep dive
Relative Value Learning
Marc Höftmann, Jan Robine, Stefan Harmeling
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 7/24/2026, 2:39:54 AM
Summary
The paper introduces Relative Value Learning (RV), a reinforcement learning framework that learns value differences between states directly using an antisymmetric function, rather than estimating absolute state values. The authors prove that a pairwise Bellman operator is a contraction with a unique fixed point equal to true value differences. They derive 1-step, n-step, and lambda-return targets for this operator and reconstruct Generalized Advantage Estimation (GAE) as R-GAE, providing an unbiased policy-gradient estimator. RV is integrated with Proximal Policy Optimization (PPO) and demonstrates competitive performance on the Atari benchmark compared to standard PPO and Direct Advantage Estimation (DAE).
Entities (8)
Relation Signals (7)
Relative Value Learning → integratedwith → PPO
confidence 98% · we integrate RV with PPO and achieve competitive performance
Relative Value Learning → evaluatedon → Atari Benchmark
confidence 97% · achieve competitive performance on the Atari benchmark
Relative Value Learning → produces → R-GAE
confidence 95% · reconstruct generalized advantage estimation from pairwise differences to obtain an unbiased policy-gradient estimator (R-GAE)
Relative Value Learning → uses → Antisymmetric Function
confidence 95% · RV learns value differences directly via an antisymmetric function
R-GAE → isvariantof → generalized advantage estimation
confidence 94% · reconstruct generalized advantage estimation from pairwise differences to obtain... R-GAE
Relative Value Learning → introduces → Pairwise Bellman Operator
confidence 92% · We introduce a pairwise Bellman operator and prove it is a γ-contraction
Direct Advantage Estimation → comparedwith → Relative Value Learning
confidence 90% · Table 1: PPO+RV (ours) is competitive with PPO and DAE
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:In reinforcement learning, critics typically estimate absolute state values $V(s)$, estimating how good a particular situation is in isolation. However, it turns out that only differences in value are relevant for control. Motivated by this, we propose Relative Value Learning (RV), a framework that learns value differences directly via an antisymmetric function $\Delta(s_i, s_j) = V(s_i) - V(s_j)$. We introduce a pairwise Bellman operator and prove it is a $\gamma$-contraction with a unique fixed point equal to the true value differences, derive well-posed $1$-step, $n$-step and $\lambda$-return targets and reconstruct generalized advantage estimation from pairwise differences to obtain an unbiased policy-gradient estimator (R-GAE). Beyond theoretical results, we integrate RV with PPO and achieve competitive performance on the Atari benchmark (49 ALE games) compared to standard PPO, indicating that relative value estimation is an effective alternative to absolute critics.
Tags
Links
- Source: https://arxiv.org/abs/2607.21120v1
- Canonical: https://arxiv.org/abs/2607.21120v1
Trouble viewing inline? Open PDF directly →
Full Text
48,073 characters extracted from source content.
Expand or collapse full text
Published as a conference paper at ICLR 2026 RELATIVE VALUE LEARNING Marc Höftmann, Jan Robine & Stefan Harmeling Department of Computer Science, Technical University of Dortmund, Germany Lamarr Institute for Machine Learning and Artificial Intelligence marc.hoeftmann,jan.robine,stefan.harmeling@tu-dortmund.de ABSTRACT In reinforcement learning, critics typically estimate absolute state valuesV (s), estimating how good a particular situation is in isolation. However, it turns out that only differences in value are relevant for control. Motivated by this, we propose Relative Value Learning (RV), a framework that learns value differences directly via an antisymmetric function∆(s i ,s j ) = V (s i )− V (s j ). We introduce a pairwise Bellman operator and prove it is aγ-contraction with a unique fixed point equal to the true value differences, derive well-posed1-step,n-step andλ-return targets and reconstruct generalized advantage estimation from pairwise differences to obtain an unbiased policy-gradient estimator (R-GAE). Beyond theoretical results, we integrate RV with PPO and achieve competitive performance on the Atari benchmark (49 ALE games) compared to standard PPO, indicating that relative value estimation is an effective alternative to absolute critics. Our code is available at https://github.com/Hauf3n/relative-value-learning. 1MOTIVATION Figure 1: Relative Value vs. Absolute Value Learning.. RV (left) learns value differences be- tween states for decision making while AV (right) learns the value for each state in isolation and then decides for the best decision (e.g. by taking maximum in Q-learning). In control, actions are chosen by comparisons, not by absolute magnitudes. What matters is how good one state (or action) is relative to another. Formally, forγ < 1,V π is uniquely determined by the Bellman equation, but shifting it by any constant leaves advantages and greedy choices unchanged. This gauge freedom makes the abso- lute scale behaviorally meaningless: only differ- ences matter. AdvantagesA π (s,a)or greedy action selectionmax a Q π (s,a)imply that ab- solute scales are not behaviorally meaningful. Optimal control depends not on absolute mag- nitudes but exclusively on relative differences. Despite this, standard value-based RL trains a critic to approximateV π (orQ π ) and treats dif- ferences as a derived quantity. This introduces unnecessary degrees of freedom (the unpinned offset), invites drift under reward shaping or base- line changes, and can be ill-posed in settings where only comparisons or implicit feedback are given (e.g., preference-based or human-in-the- loop RL), precisely where the absolute scale is ambiguous while pairwise relations remain well- defined. A formulation that places relative infor- mation at the center would match the invariances of the decision making problem itself. We therefore adopt the following viewpoint: make value differences the primary learning objective. Concretely, we learn an antisymmetric function∆ θ : S×S → Rwith∆ θ (s i ,s j ) =−∆ θ (s j ,s i )that approximates V π (s i )− V π (s j ). Working directly with ∆ θ eliminates the gauge degree of freedom by construction and aligns the critic with the invariants already exploited by policy-gradient methods 1 arXiv:2607.21120v1 [cs.LG] 23 Jul 2026 Published as a conference paper at ICLR 2026 through baselines. In addition, advantages can be reconstructed from pairwise differences without knowing the absolute value of any state, providing an unbiased policy-gradient estimator (R-GAE). This perspective is not just aesthetic. It enables a clean analytic foundation. Our Contributions: 1.Pairwise Value Operator. We formalize a Bellman operator on antisymmetric functions and prove γ-contraction with a fixed point equal to true value differences. 2.Value Targets. We derive 1-step /n-step /λ-return targets using only observable rewards and non terminal pairwise terms, ensuring well-posed bootstrapping targets. 3.Relative GAE (R-GAE). We show that GAE can be reconstructed from pairwise differ- ences that also results in an unbiased policy gradient estimator. In addition we derive the relationship between GAE and R-GAE and show that RV achieves competitive performance on Atari. 2RELATED WORK Classical value-based RL trains absolute state or action values using Bellman operators, e.g. TD(λ), DQN, Double DQN, Rainbow, and other actor critic variants (Sutton & Barto, 2018; Mnih et al., 2013; Van Hasselt et al., 2016; Wang et al., 2016a; Hessel et al., 2018) Distributional critics (Bellemare et al., 2017; Dabney et al., 2018) restructure the target but continue to operate in an absolute space. Although adding a constant toV π orQ π leaves action preferences unchanged (Sutton & Barto, 2018), absolute critics still predict a scalar on an arbitrary scale. By contrast, RV removes the offset degree of freedom at the model level by learning antisymmetric value differences over state pairs, aligning the function class with the invariances of decision making. This invariance is a special case of policy-invariant reward transformations (potential-based shaping) (Ng et al., 1999). Several methods estimate or emphasize advantages rather than values. Direct Advantage Estimation (DAE) Pan et al. (2022) directly learns advantagesA π (s,a)to bypass value learning. Dueling networks decompose learning Q π (s,a) = V π (s) + A π (s,a)to improve sample efficiency and robustness Wang et al. (2016b). Actor-critic variants such as A2C/A3C Mnih et al. (2016), TRPO Schulman et al. (2015a) and clipped PPO Schulman et al. (2017) then use baselines and advantage estimates within trust-region-style updates. In contrast, RV directly learns value differences∆(s i ,s j )with an explicit pairwise Bellman operator to provide the actor relative advantages. RV’s critic uses a siamese difference head that enforces antisymmetry and zero self-difference by design. Earlier work in stochastic control has also advocated learning value differences rather than absolute values. Bertsekas (1997) introduces differential training of rollout policies, approximating cost-to-go differences with TD-style methods. In RL, pairwise objectives have appeared mainly in preference-based and human-in-the-loop RL (Christiano et al., 2017; Leike et al., 2018) or in inverse RL, but not as a Bellman-consistent value critic. Our work fills this gap. 3RELATIVE VALUE LEARNING We now present Relative Value Learning (RV), which learns an antisymmetric function over all state pairs with a neural network ∆ θ :S×S → R,∆ θ (s i ,s j ) =−∆ θ (s j ,s i ),(1) that tries to approximate the value difference∆ π (s i ,s j ) := V π (s i )− V π (s j ), under a fixed policy π. As a result, RV avoids exact value estimates during training and integrates naturally with on-policy actor-critic methods (e.g., PPO) by supplying relative advantages (R-GAE). 3.1PRELIMINARIES We consider a discounted Markov decision process (MDP)(S,A,P ,r,γ)with state spaceS, action spaceA, transition functionP (s ′ | s,a), bounded reward functionr : S× A → [r min ,r max ], and discount factor γ ∈ [0, 1). Considering a stochastic policy π(a| s), the value function under π is V π (s) := E π,P h ∞ X t=0 γ t r(s t ,a t ) s 0 = s i ,(2) 2 Published as a conference paper at ICLR 2026 which satisfies the Bellman equation V π (s) = r π (s) + γ E s ′ ∼P π (·|s) V π (s ′ ) , r π (s) := E a∼π(·|s) [r(s,a)].(3) Gauge Freedom and Relative Values.Shifted value functions do also satisfy a Bellman equation like Equation 3 (Lemma B.1 explains how to shape the reward). These shifts are irrelevant for control, since the advantages or greedy action selection are invariant. Hence the absolute scale ofV π is not meaningful, and only differences of values are invariant. We therefore define the pairwise value difference ∆ π (s i ,s j ) = V π (s i )− V π (s j ),(s i ,s j )∈ S× S.(4) Note that it is antisymmetric ∆ π (s i ,s j ) =−∆ π (s j ,s i ), and satisfies ∆ π (s,s) = 0 for all s. Pairwise Bellman Identity. Fix(s i ,s j ) ∈ S × S. Draws ′ i ∼ P π (· | s i ) ands ′ j ∼ P π (· | s j ) independently (only conditional ons i ors j respectively). Subtracting the Bellman equations of (3) for s i and s j gives the recursive identity ∆ π (s i ,s j ) = r π (s i )− r π (s j ) + γ E s ′ i ∼P π (·|s i ) s ′ j ∼P π (·|s j ) ∆ π (s ′ i ,s ′ j ) .(5) Equation 5 depends only on observable one-step rewards viar π and on pairwise differences at successors. It is invariant to any additive shift of V π as we have shown in Lemma B.1. 3.2PAIRWISE BELLMAN OPERATOR Fix a policy π. We work on the Banach space of bounded antisymmetric pairwise functions F := n ∆ :S×S → R ∆(s i ,s j ) =−∆(s j ,s i ), ∥∆∥ ∞ <∞ o ,(6) ∥∆∥ ∞ = sup (s i ,s j ) |∆(s i ,s j )|. In addition, let ∆r π (s i ,s j ) := r π (s i )− r π (s j ) be the immediate reward difference. Operator form. Let b P π act onF by ( b P π ∆)(s i ,s j ) = E s ′ i ∼P π (·|s i ),s ′ j ∼P π (·|s j ) ∆(s ′ i ,s ′ j ) .(7) Define the pairwise Bellman operator T π :F →F by (T π ∆)(s i ,s j ) := ∆r π (s i ,s j ) + γ ( b P π ∆)(s i ,s j ).(8) This coincides with the definition from Equation (5) used in our method whens ′ i ands ′ j are drawn independently from P π (·|s i ) and P π (·|s j ), respectively. Theorem 3.1 (Contraction and uniqueness). For any ∆ 1 , ∆ 2 ∈F , ∥T π ∆ 1 − T π ∆ 2 ∥ ∞ ≤ γ∥∆ 1 − ∆ 2 ∥ ∞ . Consequently, by the Banach fixed-point theorem,T π has a unique fixed point∆ π ∈F. Moreover, this fixed point equals the true value differences, i.e.,∆ π (s i ,s j ) = V π (s i )−V π (s j )for all(s i ,s j )∈ S×S . Proof. The immediate difference ∆r π cancels in T π ∆ 1 − T π ∆ 2 , hence for each (s i ,s j ), (T π ∆ 1 − T π ∆ 2 )(s i ,s j ) = γ E (∆ 1 − ∆ 2 )(s ′ i ,s ′ j ) ≤ γ E |(∆ 1 − ∆ 2 )(s ′ i ,s ′ j )| ≤ γ∥∆ 1 − ∆ 2 ∥ ∞ . Since this bound holds for all pairs(s i ,s j )thus taking the supremum over(s i ,s j )concludes with ∥T ∆ 1 −T ∆ 2 ∥ ∞ ≤ γ∥∆ 1 − ∆ 2 ∥ ∞ . Related ‘relative’ value functions arise in average-reward MDPs, where values are defined only up to an additive constant and algorithms fix the gauge via relative value iteration (Abounadi et al., 2001; Puterman, 2014; Bertsekas, 2025). 3 Published as a conference paper at ICLR 2026 3.3RELATIVE GENERALIZED ADVANTAGE ESTIMATION (R-GAE) In this section, we define the R-GAE estimator which is in essence analogous to GAE (Schulman et al., 2015b). Furthermore, we derive the relationship between GAE and R-GAE and show that both estimators learn the same (optimal) policy. Then, R-GAE is combined with PPO (Schulman et al., 2017) to provide relative advantages. Hereby, one of our key theoretical contributions is the fact that GAE can be reconstructed without knowing the exact value of a state. Relative GAE. First, we construct relative values ̃ V θ for each state of an arbitrary environment rollout(s 0 ,s 1 ,... ,s T ), whereTdenotes the rollout length. Note in our notation thats 0 is not the environment’s start state, but it can be any state. This notation simplifies the following descriptions. We define the relative value sequence by telescoping the differences: ̃ V θ (s 0 ) := 0, ̃ V θ (s t ) := t−1 X k=0 ∆ θ (s k+1 ,s k ) (t≥ 1).(9) Note, that it is also possible to use a larger step size for calculating relative values, e.g. one can directly compute ̃ V θ (s t ) = ∆ θ (s t ,s 0 ) . If∆ θ = ∆ π , then the relationship becomes ̃ V θ (s t ) = V π (s t )− V π (s 0 ). Analogous to GAE, define relative TD residuals ̃ δ t := r t + γ ̃ V θ (s t+1 )− ̃ V θ (s t ),(10) and finally construct the relative GAE similarly ̃ A t := T−t X l=0 (γλ) l ̃ δ t+l .(11) Essentially, RV acts as the critic and replaces GAE with R-GAE in the PPO clipping objective. Furthermore, we give additional theoretical insight about R-GAE in the following. In short, these insights conclude that: 1. There exists a relationship between GAE and R-GAE that is given as ̃ A t = A t + B t . For more details, see Lemma 3.2. 2. The optimal policy for both estimators is identical. See Corollary 3.3. Lemma 3.2 (Relationship between GAE and R-GAE). We callC := V π (s 0 )the trajectory constant. If ∆ θ = ∆ π , then the following equality holds ̃ A t = A t + B t ,(12) where A t is the standard GAE computed from V π , and B t = (1− γ)C P T−t l=0 (γλ) l . Proof. By construction of the relative values, we have that ̃ V (s t ) = V π (s t )− Cfor allt. Hence inserting this form in Equation 10 gives ̃ δ t = r t + γ V π (s t+1 )− C − V π (s t )− C = r t + γV π (s t+1 )− V π (s t ) |z δ t + (1− γ)C, (13) where δ t is the TD residual. Therefore, for each timestep the relative advantage is ̃ A t = T−t X l=0 (γλ) l ̃ δ t+l = T−t X l=0 (γλ) l δ t+l + (1− γ)C T−t X l=0 (γλ) l = A t + B t , Gauge Invariance vs. Trajectory-constant Baseline.To get a better understanding of the trajectory constantB t , we discuss its role and properties in Appendix C. Lemma B.1 states that adding any scalarc ∈ RtoV π (together with the corresponding reward shaping) leaves action preferences, advantages and pairwise value differences invariant; hence the absolute gauge ofV π is behaviorally irrelevant. Lemma 3.2, however, shows when we reconstruct advantages from pairwise differences by 4 Published as a conference paper at ICLR 2026 anchoring to zero on each rollout, there exists indeed a difference quantified byB t . These statements are compatible and are not contradicting each other. Anchoring ̃ V (s 0 ) = 0is a gauge fixing analogous to the differential (relative) value normalization used in average-reward MDPs (Abounadi et al., 2001; Puterman, 2014; Bertsekas, 2025). The only difference is scope: Ours is per-trajectory and theirs is global, so unbiasedness (see Corollary 3.3) is preserved while a trajectory-constant B t may appear. As a remark, we can achieve pointwise equality with GAE, if we learned the non-antisymmetric two-argument function ∆ γ (s ′ ,s) := γV (s ′ )− V (s),(14) so that the temporal-difference residualδ t = r t + ∆ γ (s t+1 ,s t )is modeled precisely. The correspond- ing Bellman operator is indeed a contractive self-map on the pairwise space, which can be derived analogous to Theorem 3.1. However, we tried to apply∆ γ (s ′ ,s)in practice, but the results are worse compared to R-GAE. Therefore, we continue to work with R-GAE and show in Section 4 how to reduce the variance of B t . Corollary 3.3 (Unbiasedness for policy gradient). Let∇ φ J (φ) := E t [∇ φ logπ φ (a t | s t )A t ]denote the standard policy gradient (PG) underπ φ . If the advantages in the score-function estimator are replaced by the relative advantages from Lemma 3.2, i.e., ∇ φ ̃ J(φ) := E t h ∇ φ logπ φ (a t | s t ) ̃ A t i , then ∇ φ ̃ J(φ) = ∇ φ J (φ).(15) Proof. By Lemma 3.2, ̃ A t = A t + B t with B t = C ( 1−(γλ) T−t+1 ) 1−γλ as trajectory constant, hence E t h ∇ φ logπ φ (a t | s t ) ̃ A t i = E t [∇ φ logπ φ (a t | s t )A t ] + E t [∇ φ logπ φ (a t | s t )B t ] .(16) Condition ons t by the trajectory prefix up to timet(so thatB t does not depend ona t ) and then using the score-function identity gives E[∇ φ logπ φ (a t | s t )B t | s t ] = B t E a t ∼π φ (·|s t ) [∇ φ logπ φ (a t | s t )] = 0.(17) Taking expectations over t yields E t h ∇ φ logπ φ (a t | s t ) ̃ A t i = E t [∇ φ logπ φ (a t | s t )A t ] ,(18) which is precisely Equation 15. 3.4RELATIVE VALUE TARGETS When approximating the pairwise Bellman operator (Eq. 8) from two sampled 1-step transitions τ i = (s i ,a i ,r i ,d i ,s i+1 ), τ j = (s j ,a j ,r j ,d j ,s j+1 ),terminal successor states make the naive bootstrap∆(s i+1 ,s j+1 )ill-posed because absolute values are not available in our formulation. For example, if s i+1 is a terminal state (d i = 1), then we need to calculate ∆(s i+1 ,s j+1 ) = 0− V (s j+1 ) =−V (s j+1 ), which is simply not accessible in that way. Therefore, we need to rearrange all bootstrapping targets in terms of observable rewards and non terminal pairwise differences∆ θ (·,·). In this way the targets get well-posed and are compatible with the operatorT π on antisymmetric functions. Due to limited space, the complete formal derivation is given in Appendix A. 1-step Target. Given the prediction ∆ θ (s i ,s j ), the corresponding 1-step target takes the form y (1) ij := (r i − r j ) + γ δ ij ,(19) where the bootstrap term δ ij depends on the successor terminal flags d i ,d j ∈0, 1: δ ij = ∆ θ (s i+1 ,s j+1 ),if d i = 0, d j = 0, ∆ θ (s i+1 ,s j ) + r j ,if d i = 0, d j = 1, ∆ θ (s i ,s j+1 ) − r i ,if d i = 1, d j = 0, ∆ θ (s i ,s j ) + r j − r i , if d i = 1, d j = 1. (20) 5 Published as a conference paper at ICLR 2026 Whend i =d j =1, one may replace the last line of Equation 20 byδ ij =0(both successors are absorbing with zero value) to reduce variance at ends of the two episodes. We useδ ij =0by default, so the derived fourth case in Appendix A is optional. N-step Target.To extend beyond 1-step temporal difference targets, we define the pairwisen-step case by considering two trajectories τ i =(s i+k ,r i+k ,d i+k ,s i+k+1 ) k≥0 , τ j =(s j+k ,r j+k ,d j+k ,s j+k+1 ) k≥0 . Then, the target becomes y (n) ij := n−1 X k=0 γ k (r i+k − r j+k ) + γ n ∆ θ (s i+n ,s j+n ),(21) with the assumption that neither trajectory terminates within then-step window (i.e.,d i+k = d j+k = 0 fork < n). If the trajectories have different length, then we take the minimum length. Note, that the final estimated difference ∆ θ (s i+n ,s j+n ) needs to consider the case distinction from Eq. 20. λ -Return. Interpolating between high-bias/low-variance (n=1) and Monte-Carlo limits gives the pairwise λ-return y (λ) ij := (1− λ) ∞ X n=1 λ n−1 y (n) ij , λ∈ [0, 1],(22) with the convention that y (n) ij is truncated at the first encountered terminal using the case distinction from Equation 20. 4RELATIVE VALUE INITIALIZATION As demonstrated in Lemma 3.2, the zero-anchor ̃ V (s 0 ) = 0(see Equation 9) induces a trajectory- constant offsetB t for R-GAE. If the unknown|C|is large (recallC = V π (s 0 )), thenB t inflates the magnitude of ̃ A t . It thereby can increase the variance of our policy-gradient estimate (see Appendix C). The increased variance makes the credit assignment problem harder and we therefore seek a data-dependent initialization that drivesE t [B t ]≈ 0over a collected batch. Intuitively speaking, our solution is to rank trajectories relative to each other which can be understood as anchoring. For a visual example, see Figure 2. 4.1TRAJECTORY RANKING GivenMtraining rolloutsτ (m) M m=1 with statesτ (m) = (s (m) 0 ,... ,s (m) T )and done flags d (m) t T−1 t=0 indicating whether s (m) t+1 begins a new episode. Define K (m) :=0 ∪ t∈1,... ,T : d (m) t−1 = 1,(23) as the index set where either a sub-episode or new episode starts. In the following, we refer to these selected states as start states. Enumerate allN = P m |K (m) |start states ass (n) start N n=1 , where each s (n) start = s (m) k for somek ∈K (m) . Let∆ θ :S×S → Rbe the learned (noisy) difference model and define the N × N matrix of pairwise differences ∆ ij := ∆ θ s (i) start , s (j) start ,1≤ i,j ≤ N .(24) To obtain an offset estimation that is robust to prediction noise and is identifiable up to a batchwise constant (which is sufficient for ranking), we estimate all start state offsetsOvia row-wise averaging and then subtract the batch minimum to get non-negative values with ranking O(s (n) start ) := 1 N N X j=1 ∆ nj , b V θ (s (n) start ) := O(s (n) start )− min 1≤ℓ≤N O(s (ℓ) start ) .(25) Finally and to avoid overly complex notation, for any statesin a rollout, use the most recent start state in that same rollout, call it s start . Set the relative value for s as ̄ V θ (s) = b V θ (s start ) + ∆ θ (s,s start ),(26) and use these values for R-GAE. In that way, each state gets its episode-specific offset. 6 Published as a conference paper at ICLR 2026 0 τ 1 τ 2 τ 3 t RELATIVE VALUES 0 τ 1 τ 2 τ 3 t RELATIVE VALUES WITH OFFSET ̃ V θ (s t ) ̄ V θ (s t ) 0 ∆ 00 ∆ 00 ∆ 00 0 ∆ 00 ∆ 00 ∆ 00 0 s 0 s 0 s 0 s 0 s 0 s 0 RV DIFFERENCES 1 row mean O(s 0 ) O(s 0 ) O(s 0 ) 2 ESTIMATE OFFSET b V θ (s 0 ) b V θ (s 0 ) b V θ (s 0 ) Refinement: b V θ (s 0 ) := O(s 0 )− min O(s 0 ),O(s 0 ),O(s 0 ) 3 REFINE OFFSET use ̄ V θ (s t ) := b V θ (s 0 ) + ∆ θ (s t ,s 0 ) ̄ V θ (s t ) := b V θ (s 0 ) + ∆ θ (s t ,s 0 ) ̄ V θ (s t ) := b V θ (s 0 ) + ∆ θ (s t ,s 0 ) 4 ADD OFFSET Figure 2: Trajectory Ranking. When training batches contain samples from more than one episode, the initialization ̃ V (s 0 ) = 0for each trajectoryτ i is not correct. The trajectories need to be ranked relative to each other by adding an offset that is calculated with∆(s i ,s j ). Note thats 0 ,s 0 ,s 0 are start states ofτ 1 ,τ 2 ,τ 3 indicated by color. For simplicity, assume in this figure that start states are only present at t = 0 for each rollout, so we can think about each τ as one episode. 5TRAINING OBJECTIVE The PPO objective uses relative advantages ̃ A t and the usual clipped surrogate: L policy (θ) = E t min(r t (θ) ̃ A t , clip(r t (θ), 1− ε, 1 + ε) ̃ A t ) , r t (θ) = π θ (a t | s t ) π old (a t | s t ) .(27) The critic loss and entropy bonus are L critic (θ) = E (i,j)∼μ ∆ θ (s i ,s j )− y (n) ij 2 , L ent (θ) =−E t [H(π θ (·| s t ))],(28) where we use the n-step value target (see Eq. 21) and finally optimize the combined loss L(θ) =−L policy (θ) + c v L critic (θ) + c e L ent (θ).(29) 5.1NETWORK ARCHITECTURE For all Atari experiments we use the exact same architecture as PPO Schulman et al. (2017). The pol- icy and relative value function share the CNN encoderf enc (s)∈ R d and then split their computation by using a single linear layer for their respective outputs. Relative Critic. Relative values are obtained by applying the same encoderf enc (s)to both states and projecting the difference of their embeddings. We do not use an additional target encoder or stop-gradients. Formally, the value difference for a state pair (s i ,s j ) is given by ∆ θ (s i ,s j ) = Φ f enc (s i )− f enc (s j ) ,(30) whereΦis the projection head. For fair comparisons, our experiments use a single learned vector w ∈ R d without bias term to ensure antisymmetry∆ θ (s i ,s j ) =−∆ θ (s j ,s i )and∆ θ (s i ,s i ) = 0by design. It is also feasible to buildΦas an non-linear MLP that is antisymmetric in nature (e.g. by using tanh activations and no bias term in linear layers). But so far, we have not observed additional improvements by using such non-linear heads. 6EXPERIMENTS We evaluate Relative Value Learning (RV) as a drop-in critic for on-policy policy-gradient methods on the Arcade Learning Environment (ALE) for Atari. Observations follow the standard PPO 7 Published as a conference paper at ICLR 2026 preprocessing: random no-op resets, frame skip with max-over-two, grayscale resizing to84×84, stackingm=4frames, and input scaling to[0, 1]. Networks use orthogonal initialization with a small policy logit scale (0.01) and unit scale for the value head, matching widely used PPO implementations. We run with EnvPool Weng et al. (2022) for high-throughput environment simulation and train for 40M frames (10M environment steps). For each game we use10independent seeds and report performance as the average score over the last100training episodes. All hyperparameters stay identical over 49 games and are reported in Appendix D. Table 1: PPO+RV (ours) is competitive with PPO and DAE. Mean final scores (last 100 episodes) with standard deviation of PPO, DAE and our method (PPO+RV) after 40 M game frames. GamePPO (Schulman et al., 2017)DAE (Pan et al., 2022)PPO + RV (ours) Alien1850.3±376.81372.7±349.31748.8±408.7 Amidar674.6±108.5394.6±121.9504.7±108.8 Assault4971.9±642.42205.1±362.33901.7±219.2 Asterix4532.5±1570.73750.1±522.93862.2±678.2 Asteroids2097.5±88.81392.3±62.31489.5±91.5 Atlantis2 311 815.0±420555.52 888 011.1±225889.1 3 101 686.1±451117.9 BankHeist1280.6±2.7257.8±193.21209.0±77.5 BattleZone17 366.7±1045.016 302.0±2042.521 780.0±1516.1 BeamRider1590.0±276.31729.9±172.72044.0±396.3 Bowling40.1±13.836.1±9.146.3±7.9 Boxing94.6±0.925.9±9.594.8±0.8 Breakout274.8±20.2234.9±28.2263.0±27.7 Centipede4386.4±165.63915.8±694.41226.1±104.5 ChopperCommand3516.3±991.61587.3±423.64435.2±965.5 CrazyClimber110 202.0±3547.7112 319.5±6125.4111 622.2±5645.4 DemonAttack11 378.4±2800.52477.2±344.38944.7±534.3 DoubleDunk−14.9±1.3−12.5±3.0−23.3±5.3 Enduro758.3±31.20.0±0.01080.2±121.1 FishingDerby17.8±2.8−60.4±7.219.8±5.2 Freeway32.5±0.319.5±13.731.9±0.2 Frostbite314.2±4.9367.9±278.2522.6±399.0 Gopher2932.9±1870.61137.2±156.83719.1±249.5 Gravitar737.2±150.9443.5±50.31441.0±561.4 IceHockey−4.2±0.3−5.1±0.5−3.9±0.4 Jamesbond560.7±94.9507.8±21.7568.9±43.7 Kangaroo9928.7±7089.91331.0±1060.85649.0±2940.0 Krull7942.3±555.19034.0±774.68870.9±473.1 KungFuMaster23 310.3±2251.920 535.3±2810.724 483.6±6794.6 MontezumaRevenge42.0±57.40.1±0.31.4±3.4 MsPacman2096.5±157.12501.0±600.91721.2±230.1 NameThisGame6254.9±160.96016.3±283.36685.3±938.3 Pitfall−32.9±19.0−12.0±23.1−27.3±26.3 Pong20.7±0.220.7±0.316.8±3.1 PrivateEye69.5±55.286.0±14.993.3±17.6 Qbert14 293.3±293.111 119.6±3465.815 261.6±502.1 Riverraid8393.6±385.43150.8±549.99465.8±1062.5 RoadRunner25 076.0±10851.216 146.3±3074.043 346.3±6741.1 Robotank5.5±0.76.9±2.719.5±3.1 Seaquest1204.5±481.12049.8±430.31659.7±209.4 SpaceInvaders942.5±266.1914.9±219.4712.3±134.5 StarGunner32 689.0±949.85849.2±740.424 830.8±10990.8 Tennis−14.8±3.8−17.6±0.9−31.7±3.4 TimePilot4342.0±331.17252.7±1173.610 212.7±492.0 Tutankham254.4±42.5196.0±26.2224.2±23.1 UpNDown95 445.0±21584.185 438.4±19664.6113 991.7±21370.6 Venture0.0±0.00.0±0.05.5±15.6 VideoPinball37 389.0±15876.623 958.6±3936.0138 564.8±77425.7 WizardOfWor4185.3±1029.44161.3±696.05084.1±522.6 Zaxxon5008.7±1261.35612.2±1712.5845.8±1567.9 Compute Resources. Each40M-frame run completes in approximately65minutes on a single A100 GPU with12CPU cores. Across all49games and10seeds (490runs total), this corresponds to 530 GPU-hours or 22.10 A100-days. 0.60.81.0 DAE PPO PPO+RV Median 0.600.750.90 IQM 4.85.25.66.0 Mean 0.360.400.440.48 (ours) Optimality Gap Human Normalized Score Figure 3: Comparison between Methods. Aggregate metrics with 95% stratified bootstrap con- fidence intervals (Agarwal et al., 2021). Higher median, interquartile mean (IQM), and mean, but lower optimality gap indicate better performance. 8 Published as a conference paper at ICLR 2026 0M10M20M30M40M 500 1000 1500 2000 Score Alien 0M10M20M30M40M 0 200 400 Amidar 0M10M20M30M40M 1000 2000 3000 4000 Assault 0M10M20M30M40M 1000 2000 3000 4000 Asterix 0M10M20M30M40M 750 1000 1250 1500 Asteroids 0M10M20M30M40M 0 1 2 3 1e6 Atlantis 0M10M20M30M40M 0 500 1000 BankHeist 0M10M20M30M40M 10000 20000 Score BattleZone 0M10M20M30M40M 500 1000 1500 2000 2500 BeamRider 0M10M20M30M40M 20 30 40 50 60 Bowling 0M10M20M30M40M 0 25 50 75 100 Boxing 0M10M20M30M40M 0 100 200 Breakout 0M10M20M30M40M 1000 2000 3000 Centipede 0M10M20M30M40M 2000 4000 ChopperCommand 0M10M20M30M40M 25000 50000 75000 100000 Score CrazyClimber 0M10M20M30M40M 1000 2000 3000 4000 Gopher 0M10M20M30M40M 500 1000 1500 Seaquest 0M10M20M30M40M 500 1000 1500 2000 MsPacman 0M10M20M30M40M 0 2500 5000 7500 DemonAttack 0M10M20M30M40M 30 20 DoubleDunk 0M10M20M30M40M 0 500 1000 Enduro 0M10M20M30M40M 100 50 0 Score FishingDerby 0M10M20M30M40M 0 10 20 30 Freeway 0M10M20M30M40M 200 400 600 800 Frostbite 0M10M20M30M40M 500 1000 1500 Gravitar 0M10M20M30M40M 10 8 6 4 IceHockey 0M10M20M30M40M 0 200 400 600 Jamesbond 0M10M20M30M40M 0 2000 4000 6000 8000 Kangaroo 0M10M20M30M40M 2000 4000 6000 8000 Score Krull 0M10M20M30M40M 0 10000 20000 30000 KungFuMaster 0M10M20M30M40M 0 5 10 MontezumaRevenge 0M10M20M30M40M 2000 4000 6000 NameThisGame 0M10M20M30M40M 150 100 50 0 50 Pitfall 0M10M20M30M40M 20 0 20 Pong 0M10M20M30M40M 400 200 0 200 PrivateEye 0M10M20M30M40M 0 5000 10000 15000 Score Qbert 0M10M20M30M40M 2000 4000 6000 8000 Riverraid 0M10M20M30M40M 0 20000 40000 RoadRunner 0M10M20M30M40M 5 10 15 20 Robotank 0M10M20M30M40M 200 400 600 800 SpaceInvaders 0M10M20M30M40M 0 10000 20000 30000 StarGunner 0M10M20M30M40M 40 35 30 25 Tennis 0M10M20M30M40M 4000 6000 8000 10000 Score TimePilot 0M10M20M30M40M 0 100 200 Tutankham 0M10M20M30M40M 0 50000 100000 UpNDown 0M10M20M30M40M 0 100 200 Venture 0M10M20M30M40M 0 50000 100000 150000 200000 VideoPinball 0M10M20M30M40M 0 2000 4000 6000 WizardOfWor 0M10M20M30M40M 0 2000 4000 6000 Zaxxon ZeroOffset Figure 4: Ablation for Value Initialization. This figure compares the performance difference between zero (see Equation 9) and offset initialization (see Equation 26) for relative values. Usually the proposed offset initialization is needed to improve credit assignment, but for some games the algorithm can handle zero initialization as well. 6.1ARCADE LEARNING ENVIRONMENT (ALE) Table 1 presents the per-game scores of PPO (Schulman et al., 2017) and Direct Advantage Estimation (DAE) (Pan et al., 2022) and extends the table with an additional column, PPO+RV. This column represents our algorithm where the absolute value critic is replaced by the RV critic. Across the benchmark, PPO+RV attains competitive performance: it exceeds PPO on 30 out of 49 games (or 61%) and DAE on 37 out of 49 games (or 75%). In addition we report aggregated metrics in Figure 3 for further analysis. 6.2ABLATION STUDY Furthermore, we present the influence of relative value initialization in Figure 4. In general, we see that relative ranking is needed, but sometimes actors are not bothered by the influence ofB t which we speculate is due to the PPO clipping objective. 9 Published as a conference paper at ICLR 2026 7LIMITATIONS Gauge fixing induces a trajectory-constant baselineB t in R-GAE that inflates variance for long horizons or whenγλ→ 1, but it is only partially reduced by relative value initialization. The enforced antisymmetry improves stability but restricts critic expressivity (our near-linear difference head may underfit complex value differences). Pairwise training isO(B 2 ), whereBis the batch size, in the naive form and relies on subsampling that trades compute for estimator variance. Trajectory ranking used for initialization assumes post-baseline values are on a comparable scale across trajectories. Ranking signal may become uninformative. Empirically, experiments are limited to PPO on discrete- action Atari. Generality to continuous control, off-policy regimes, and preference-based settings remains to be demonstrated. 8CONCLUSION In this paper we established that reinforcement learning can operate on value differences rather than absolute values. We proposed Relative Value Learning (RV), a gauge-invariant alternative to absolute critics that learns antisymmetric value differences∆ π (s i ,s j ) = V π (s i )− V π (s j )as a primitive representation. On the theoretical side, we defined a pairwise Bellman operator that is aγ-contraction on bounded antisymmetric functions with a unique fixed point equal to the true value differences, and we derived well-posed bootstrapping targets (1-step/n-step/λ) that operate entirely on observable reward differences and non terminal pairwise terms. We further introduced trajectory ranking and reconstructed generalized advantage estimation from pairwise differences (R-GAE), showing that it ensures an unbiased policy-gradient estimator and clarifying its relationship to standard GAE. Empirically, replacing the PPO critic with our relative critic gives competitive performance across Atari while simplifying the value learning objective to the quantities that matter for control: differences rather than absolutes. We hope RV serves as a building block for algorithms that reason natively in the relative domain where decisions are actually made. 9ACKNOWLEDGMENTS This research has been funded and supported by the Federal Ministry of Research, Technology and Space of Germany and the state of North Rhine-Westphalia as part of the Lamarr Institute for Machine Learning and Artificial Intelligence. REFERENCES Jinane Abounadi, Dimitrib Bertsekas, and Vivek S Borkar. Learning algorithms for markov decision processes with average cost. SIAM Journal on Control and Optimization, 40(3):681–698, 2001. Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34:29304–29320, 2021. Marc G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. In International conference on machine learning, p. 449–458. PMLR, 2017. Dimitri P Bertsekas. Differential training of rollout policies. In Proceedings of the Annual Allerton Conference on Communication Control and Computing, volume 35, p. 913–922. UNIVERSITY OF ILLINOIS, 1997. Dimitri P Bertsekas. Neuro-dynamic programming. In Encyclopedia of optimization, p. 1–6. Springer, 2025. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017. Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. In International conference on machine learning, p. 1096– 1105. PMLR, 2018. 10 Published as a conference paper at ICLR 2026 Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018. Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: A research direction. arxiv 2018. arXiv preprint arXiv:1811.07871, 2018. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013. Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, p. 1928–1937. PmLR, 2016. Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Icml, volume 99, p. 278–287. Citeseer, 1999. Hsiao-Ru Pan, Nico Gürtler, Alexander Neitz, and Bernhard Schölkopf. Direct advantage estimation. Advances in Neural Information Processing Systems, 35:11869–11880, 2022. Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014. John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, p. 1889–1897. PMLR, 2015a. John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015b. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018. Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016. Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Remi Munos, Koray Kavukcuoglu, and Nando De Freitas. Sample efficient actor-critic with experience replay. arXiv preprint arXiv:1611.01224, 2016a. Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In International conference on machine learning, p. 1995–2003. PMLR, 2016b. Jiayi Weng, Min Lin, Shengyi Huang, Bo Liu, Denys Makoviichuk, Viktor Makoviychuk, Zichen Liu, Yufan Song, Ting Luo, Yukun Jiang, Zhongwen Xu, and Shuicheng Yan. En- vPool: A highly parallel reinforcement learning environment execution engine. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, p. 22409–22421. Curran Associates, Inc., 2022. URLhttps://proceedings.neurips.c/paper_files/paper/2022/file/ 8caaf08e49ddbad6694fae067442e21-Paper-Datasets_and_Benchmarks. pdf. 11 Published as a conference paper at ICLR 2026 ATARGET DERIVATION FOR 1-STEP BOOTSTRAPPING When approximating the pairwise Bellman operator from Equation 8 by using two randomly sampled transitions τ i = (s i ,a i ,r i ,d i ,s i+1 ), τ j = (s j ,a j ,r j ,d j ,s j+1 ), we can encounter terminal successor states that result in ill-defined targets for∆(s i+1 ,s j+1 ). We present four exhaustive cases that arise from the possible combinations of terminal and non-terminal successor states whered i ,d j ∈0, 1are terminal state indicators fors i+1 ands j+1 . In each case, we derive an equivalent expression for∆(s i+1 ,s j+1 )by only using (1) observable rewardsr i ,r j and (2) non-terminal, well-defined value differences∆ θ (·,·). The rearrangement is then used for the bootstrapping target in Equation (20) to work for sampling. Case 1: Both successor states are non-terminal (d i = 0,d j = 0). Then, the relative value ∆(s i+1 ,s j+1 ) = V (s i+1 )− V (s j+1 ),(31) is well-posed and directly expressible with ∆ θ (s i+1 ,s j+1 ). Case 2: Assume the successors i+1 is terminal, ands j+1 is non-terminal (d i = 1,d j = 0). To address this, we derive a modified target: ∆(s i+1 ,s j+1 ) = V (s i+1 )− V (s j+1 ) = V (s i+1 )− V (s j+1 ) + V (s i )− V (s i ) = ∆(s i ,s j+1 )− V (s i ) + V (s i+1 ) |z =0 = ∆(s i ,s j+1 )− E s i [R t + γR t+1 + ... ] using τ i ≈ ∆(s i ,s j+1 )− r i (32) Case 3: Assume the successors i+1 is non-terminal, and iss j+1 terminal (d i = 0,d j = 1). To address this, we derive a modified target: ∆(s i+1 ,s j+1 ) = V (s i+1 )− V (s j+1 ) = V (s i+1 )− V (s j+1 ) + V (s j )− V (s j ) = ∆(s i+1 ,s j ) + V (s j )− V (s j+1 ) = ∆(s i+1 ,s j ) + E s j [R t + γR t+1 + ... ] using τ j ≈ ∆(s i+1 ,s j ) + r j (33) Case 4: Assume that both successors are terminal (d i = 1,d j = 1). We derive a modified target: ∆(s i+1 ,s j+1 ) = V (s i+1 )− V (s j+1 ) = V (s i+1 )− V (s j+1 ) + V (s i )− V (s i ) + V (s j )− V (s j ) = ∆(s i ,s j )− E s i [R t + γR t+1 + ... ] + E s j [R t + γR t+1 + ... ] using τ i ,τ j ≈∆(s i ,s j )− r i + r j (34) For better training stability, we simply say that ∆(s i+1 ,s j+1 ) = V (s i+1 )− V (s j+1 ) = 0 (35) since the value of both terminal states, by using MDP definition, is equal to zero. 12 Published as a conference paper at ICLR 2026 BGAUGE FREEDOM Lemma B.1 (Constant-offset gauge invariance). SupposeV π satisfies the Bellman equation V π = r π + γP π V π . For any c∈ R, define the shaped reward and shifted value r ′ π (s) := r π (s) + (1− γ)c, V π′ (s) := V π (s) + c.(36) Then V π′ satisfies the transformed Bellman equation V π′ = r ′ π + γP π V π′ .(37) Moreover, if one equivalently defines the reward shapingr ′ (s,a) = r(s,a) + (1− γ)cso that r ′ π (s) = E a∼π(·|s) [r ′ (s,a)], then action preferences and pairwise differences are invariant: A π′ (s,a) = A π (s,a) and ∆ π′ (s i ,s j ) = ∆ π (s i ,s j ) for all (s i ,s j )∈ S× S.(38) Proof. By direct calculation, r ′ π + γP π V π′ = r π + (1− γ)c + γP π (V π + c) = r π + γP π V π + (1− γ)c + γc = V π + c = V π′ , soV π′ satisfies the transformed Bellman equation. For invariance: withr ′ (s,a) = r(s,a) + (1−γ)c, Q π′ (s,a) = r ′ (s,a) + γ E s ′ [V π′ (s ′ )] = r(s,a) + γ E s ′ [V π (s ′ )] + c = Q π (s,a) + c.(39) ThusA π′ (s,a) = Q π′ (s,a)−V π′ (s) = A π (s,a), and∆ π′ (s i ,s j ) = (V π (s i )+c)−(V π (s j )+c) = ∆ π (s i ,s j ). Constant offsets are a special case of potential-based shaping (Ng et al., 1999). 13 Published as a conference paper at ICLR 2026 CVARIANCE ANALYSIS OF THE RELATIVE POLICY GRADIENT In Section 4, we argue that the unknown trajectory constant|C|can increase the variance of the policy gradient estimator. Here, we provide the formal derivation supporting this claim and reference Figure 4 for empirical validation. First, recall from Lemma 3.2 that the relative advantage is given by ̃ A t = A t + B t , whereB t is a trajectory constant determined by the initialization offsetC. While Corollary 3.3 proves that the estimator remains unbiased (i.e., the first moment is unchanged), the second moment differs. Lemma C.1 (Variance Inflation). Letg std =∇ φ logπ φ (a t |s t )A t be the standard gradient estimator andg rel = ∇ φ logπ φ (a t |s t ) ̃ A t be the relative gradient estimator. The variance of the relative estimator is given by: Var(g rel ) = Var(g std ) + E ∥∇ φ logπ φ (a t |s t )∥ 2 B 2 t + 2E A t B t ∥∇ φ logπ φ (a t |s t )∥ 2 (40) Crucially, the strictly positive termE[∥∇ φ logπ φ ∥ 2 B 2 t ]scales quadratically with the trajectory offset C 2 . Proof.The variance of any estimatorgis defined asVar(g) = E[∥g∥ 2 ]−∥E[g]∥ 2 . From Corollary 3.3, we know thatE[g rel ] = E[g std ] =∇ φ J (φ). Since the expected values are identical, the difference in variance is determined entirely by the second moment E[∥g∥ 2 ]. Let u t =∇ φ logπ φ (a t |s t ) denote the score function. Expanding it for the second moment gives: ∥g rel ∥ 2 =∥u t ̃ A t ∥ 2 (41) =∥u t (A t + B t )∥ 2 (42) =∥u t A t ∥ 2 +∥u t B t ∥ 2 + 2(u t A t ) ⊤ (u t B t )(43) Note that A t and B t are scalars, so we can factor them out: ∥g rel ∥ 2 =∥u t ∥ 2 A 2 t | z ∥g std ∥ 2 +∥u t ∥ 2 B 2 t + 2∥u t ∥ 2 A t B t (44) Recall from Lemma 3.2 thatB t = (1− γ)C P (γλ) l . Thus,B t is directly proportional to the initialization offset C. 1. Noise Term: The termE[∥u t ∥ 2 B 2 t ] is strictly non-negative. SinceB t ∝ C, this noise term scales with C 2 . 2. Correlation Term: The term2E[∥u t ∥ 2 A t B t ]represents the correlation between the true advantage and the offset. While this term can be negative, it scales linearly with C. Consequently, for a sufficiently large uncorrected offset|C|, the quadratic noise term (C 2 ) will dominate the linear correlation term, resulting in an increase in estimator variance. This necessitates the Trajectory Ranking strategy (from Section 4) to minimize|C|and driveB t ≈ 0. Empirically, our ablation in Figure 4 validates the importance of decreasing estimator variance. In addition to validateB t empirically, Table 2 presents a small example for a seven step trajectory. Given are rewards and true values to compute the true GAE for comparison to relative GAE: [r 0 ,... ,r 5 ] := [1, 1, 1, 1, 1, 1] [V (s 0 ),... ,V (s 6 )] := [2, 3, 4, 5, 6, 7] In this exampleV (s 0 ) = 2, so one can compute all predicted differencesB t usingC = 2. Over all time steps, the observed advantage differences match exactly the predictionB t , confirming the algebraic derivation from Lemma 3.2. In this example, we rounded to two decimal digits for clean visualization. Figure 5 generalizes the intuition of Table 2 by visualizing the prediction and decay ofB t over a T = 128timestep rollout, which has the same length as in our experiments. In the figure, we also 14 Published as a conference paper at ICLR 2026 Table 2: Empirical Evidence for the Trajectory-constant. This tables shows a small environment trajectory and validates the trajectory-constant baseline empirically. To compute true and relative GAE, we set γ = 0.9,λ = 0.8 and use C = 2. t = 0 t = 1 t = 2 t = 3 t = 4 t = 5 t = 6 Values r t 111111– V (s t )2345678 ∆(s t ,s 0 )–123456 ̃ V (s t )0123456 True & Relative GAE (γ = 0.9,λ = 0.8) δ t 1.701.601.501.401.301.20– ̃ δ t 1.901.801.701.601.501.40– A t 4.734.213.632.962.161.20– ̃ A t 5.354.794.153.412.511.40– GAE Differences (C = 2) ̃ A t − A t 0.610.580.520.450.340.20– Predicted via B t 0.610.580.520.450.340.20– 020406080100120 Time step t 0.75 0.50 0.25 0.00 0.25 0.50 0.75 Baseline B t Expected Difference to True GAE C=-5 C=-2 C=-1 C=1 C=2 C=5 Figure 5: Variance Visualization. This figure shows the expected differenceB t to the true GAEA t using different baseline valuesC. Note, that the difference is higher for larger|C|, becauseγapplies a stronger absolute discount. consider the same hyperparameter setting (γ = 0.99,λ = 0.95) which influences the magnitude of B t . At the startt = 0of a rollout, one can see the absolute value ofB 0 is large but almost constant, and inflates the GAE value. Then after some time,B t falls off and finally (t→128) almost vanishes, so R-GAE≈ GAE for the last few timesteps. This can be demonstrated for arbitrary C. 15 Published as a conference paper at ICLR 2026 DHYPERPARAMETERS We present the hyperparameter configuration for the Atari benchmark (ALE) which is used for all 49 games. Table 3: Hyperparameters for PPO+RV used in our experiments. HyperparameterValue Discount factor γ0.99 GAE parameter λ0.95 N-step return target5 Clip parameter ε0.1 Number of epochs per update5 Minibatch size128 Number of parallel environments8 Rollout length T128 Learning rate2.5× 10 −4 OptimizerAdam Adam epsilon1× 10 −5 Entropy coefficient c e 0.01 RV loss coefficient c v 1.25 RV value clipping0.15 Gradient clipping (max-norm)0.5 Additional Hyperparameters Details.For training the relative critic we do not use purely random state pairing. With probability 33%, the second state is chosen from the same episode as the reference states i , encouraging temporally coherent comparisons. Otherwise, the partner is sampled randomly in the batch. See Appendix E for an ablation study. 16 Published as a conference paper at ICLR 2026 EABLATION ON PAIR SAMPLING In this section, we present Table 4 as an ablation study for different pair sampling strategiesμthat can be applied for Equation 28. In our case, “Biased” means that with probabilityp ∈ [0, 1]the second states j is randomly sampled from the same episode. In general, we can see that the sampling strategy has a rather small influence except whenpgets too large. Then there is a small decrease in performance. Table 4: Ablation of Pair Sampling. Mean final scores (last 100 episodes) with standard deviation of our method PPO+RV with different pair sampling strategies after 40 M game frames. The results average over five seeds. GameRandomBiased (p = 0.33)Biased (p = 0.66) Alien2123.4±346.01746.5±491.71802.0±600.5 Assault4214.3±338.34018.8±174.13615.2±290.2 Asterix4303.2±1024.84495.2±1111.03821.2±461.1 Atlantis2 683 819.6±515071.2 2 991 307.2±542845.7 3 711 740.4±431880.2 BattleZone22 512.0±1038.622 576.0±921.124 616.0±1228.4 BeamRider2199.2±573.91966.5±130.11814.8±247.2 Breakout237.0±18.5258.5±31.6241.7±7.6 DemonAttack9364.3±1069.88669.9±734.87181.1±858.8 Gopher3224.7±1097.33585.7±183.13626.2±338.1 Gravitar1414.2±759.31379.0±488.91232.0±406.9 IceHockey−3.7±0.6−3.7±0.6−3.9±0.4 Krull9048.9±729.28527.1±393.78540.5±504.5 MsPacman1640.0±147.41783.4±125.81680.9±447.8 NameThisGame7000.3±1446.86880.2±937.05890.7±296.8 Qbert16 630.4±938.015 472.7±417.315 631.9±789.1 Riverraid9941.6±1440.79271.0±1276.28717.8±478.7 Robotank22.5±1.021.3±2.019.7±1.5 Seaquest1592.6±307.41612.4±109.61236.1±321.7 TimePilot9701.6±435.39825.6±819.39692.8±584.0 UpNDown66 393.6±27590.7101 955.1±25061.7100 891.3±33878.0 VideoPinball144 597.1±89747.6110 187.1±57998.882 002.4±16130.5 WizardOfWor5501.2±787.04890.0±494.95128.0±374.3 17 Published as a conference paper at ICLR 2026 FTHE USE OF LARGE LANGUAGE MODELS (LLMS) Finally, we report on the use of LLMs in this section. The image in Figure 1 is generated with Gemini 2.5 Flash Image (Nano Banana). Furthermore, we used large language models (LLMs) as writing assistants in the preparation of this manuscript. LLMs were employed to draft and refine text as well as to generate preliminary versions of some proof derivations. All mathematical results, derivations, and theoretical claims were independently verified and validated by the authors. 18