Paper deep dive
Policy as Data: Replay-Based Policy Dual Averaging via Advantage Regression
Nianli Peng, Geoffrey J. Gordon, Kianté Brantley
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Actor-critic methods reuse past experience to improve sample efficiency. However, historical data are typically regarded as off-policy samples for the current policy-improvement update. This work introduces Regularized Dual Averaging Actor Critic (RDA2C), which assigns a distinct role to replay. In regularized dual averaging, the subsequent policy is determined by accumulated policy-improvement feedback, so historical advantage estimates contribute directly to the actor objective rather than solely to the most recent update. RDA2C stores state-action samples with critic-estimated advantage labels, fits a dual score model $Z_\theta$ to the aggregated dataset, and derives the current policy from the accumulated score model using the entropy mirror map. In this way, replay defines an empirical dual objective from which the policy is computed. To analyze RDA2C, we establish a finite-time value-gap decomposition, separating the regularized dual-averaging term from errors due to stale-replay supervised fitting, critic bias, finite-buffer variance, and replay coverage, and stating the assumptions under which each error is bounded. RDA2C accepts advantage labels from any critic. With GAE labels, RDA2C outperforms PPO on six of eight MuJoCo tasks and eight of twelve Atari games. RDA2C also outperforms AAPDA, the closest dual-averaging baseline, on six of eight MuJoCo tasks. With twin-$Q$ labels, RDA2C matches SAC at matched batch size and update frequency.
Tags
Links
- Source: https://arxiv.org/abs/2610.04638v1
- Canonical: https://arxiv.org/abs/2610.04638v1
Trouble viewing inline? Open PDF directly →
Full Text
153,898 characters extracted from source content.
Expand or collapse full text
Policy as Data: Replay-Based Policy Dual Averaging via Advantage Regression Nianli Peng Affiliation: Harvard University Geoffrey J. Gordon Affiliation: Carnegie Mellon University Kianté Brantley Affiliation: Harvard University Abstract Actor-critic methods reuse past experience to improve sample efficiency. However, historical data are typically regarded as off-policy samples for the current policy-improvement update. This work introduces Regularized Dual Averaging Actor Critic (RDA2C), which assigns a distinct role to replay. In regularized dual averaging, the subsequent policy is determined by accumulated policy-improvement feedback, so historical advantage estimates contribute directly to the actor objective rather than solely to the most recent update. RDA2C stores state-action samples with critic-estimated advantage labels, fits a dual score model ZθZ_θ to the aggregated dataset, and derives the current policy from the accumulated score model using the entropy mirror map. In this way, replay defines an empirical dual objective from which the policy is computed. To analyze RDA2C, we establish a finite-time value-gap decomposition, separating the regularized dual-averaging term from errors due to stale-replay supervised fitting, critic bias, finite-buffer variance, and replay coverage, and stating the assumptions under which each error is bounded. RDA2C accepts advantage labels from any critic. With GAE labels, RDA2C outperforms PPO on six of eight MuJoCo tasks and eight of twelve Atari games. RDA2C also outperforms AAPDA, the closest dual-averaging baseline, on six of eight MuJoCo tasks. With twin-Q labels, RDA2C matches SAC at matched batch size and update frequency. 1 Introduction Experience replay is an important mechanism for improving sample efficiency in deep reinforcement learning by allowing previously collected transitions to support multiple learning updates (Mnih et al., 2015; Schaul et al., 2015; Fedus et al., 2020). For value-based learning, the role of replay is relatively straightforward. Past transitions serve to calculate supplementary targets for temporal-difference updates or Bellman regression. In contrast, using historical data for policy learning introduces additional complexities. The policy improvement signal is defined with respect to the current policy, whereas replayed actions originate from previous behavior policies. Standard actor updates therefore treat replayed samples as off-policy data for the current policy-improvement step (Wang et al., 2017; Espeholt et al., 2018). Near-on-policy methods limit reuse and constrain the likelihood ratio between the behavior and current policies, while more explicitly off-policy methods use importance weighting or related corrections (Schulman et al., 2017; Espeholt et al., 2018; Wang et al., 2017). Alternatively, methods such as DDPG, TD3, and SAC learn an off-policy critic from replay and update the actor through that critic, so the actor update itself stays on-policy while the critic absorbs the off-policy data, using target networks and slowly changing value estimates to stabilize learning (Lillicrap et al., 2016; Fujimoto et al., 2018; Haarnoja et al., 2018). We propose an alternative perspective and objective for the use of replay data. The analysis begins with regularized dual averaging (RDA), which determines the subsequent decision rule using accumulated first-order feedback instead of relying solely on a local update around the latest iterate. For RDA2C, this feedback is provided by critic-estimated advantages. From this perspective, an older advantage estimate is one of the observations that defines the cumulative dual objective from which the next policy is computed, and it counts as much as the newest one. Replay thus serves both as a source of additional optimization steps on prior experience and as a statistical representation of the actor’s optimization problem. The distinction is most apparent when comparing dual averaging with local mirror descent. Methods such as natural policy gradient, TRPO, PPO, and policy mirror descent can be interpreted as local policy-space updates (Kakade, 2001; Schulman et al., 2015; Schulman et al., 2017; Lan, 2023). Their optimization history is summarized by the most recent policy or actor parameters. In contrast, regularized dual averaging and Follow the Regularized Leader optimize accumulated linearized feedback in conjunction with a regularizer (Nesterov, 2009; Xiao, 2010; McMahan, 2011). This cumulative formulation does not claim a universal optimization rate advantage over mirror descent. It highlights a concrete statistical object, a historical average of advantage feedback that can be estimated from replay. We instantiate this idea in Regularized Dual Averaging Actor Critic. At each iteration, RDA2C stores state-action samples along with critic-estimated advantage labels in a replay buffer. A dual score model Zθ(s,a)Z_θ(s,a) is trained via regression on this aggregated dataset, thereby estimating a replay-weighted average of historical policy-improvement feedback. The current policy is subsequently derived directly from the learned score using the entropy mirror map. Thus the learned dual model provides the policy’s natural-coordinate representation, with no separately optimized projection actor. This construction motivates the phrase policy as data. In conventional recursive actor updates, historical improvement information is implicitly compressed into the current network parameters. In RDA2C, the replay buffer explicitly stores the advantage evidence that defines a moving empirical dual objective, and ZθZ_θ summarizes this objective. Modifying the contents of the buffer alters the actor objective, and fitting ZθZ_θ and applying the mirror map computes the policy induced by the updated data. This perspective turns replay from a mechanism for reusing stale experience into an explicit representation of the policy-improvement problem (Figure 1), leading to the algorithmic, theoretical, and empirical contributions below.11 1 Code: https://github.com/InsultedByMaths/rda2c. • A replay-based dual-averaging actor update. We derive an entropy-regularized actor–critic method in which replayed advantage-labeled samples define an empirical cumulative dual objective, a learned score model ZθZ_θ estimates that objective, and the policy is obtained by an exact mirror-map readout for both finite and continuous action spaces. • A finite-time analysis of replayed dual feedback. We give a value-gap decomposition that separates the standard RDA optimization term from the errors introduced by a practical replay implementation. The analyzed update closely tracks the implemented one, with the bounds applying to the replayed labels, finite FIFO window, and fitted score model. The decomposition covers local stale-replay drift, supervised fitting error, critic-label bias, finite-buffer variance, and replay coverage. • Evaluation across critic constructions and action spaces. We instantiate RDA2C using both GAE and twin-Q advantage labels. Across eight MuJoCo-v5 tasks and twelve Atari games, the results show that the same replay-based dual-score actor can be used with on-policy-style value critics, off-policy twin-Q critics, continuous Gaussian policies, and discrete softmax policies. 2 Preliminaries Figure 1: Left Standard actor updates compress policy-improvement history into the current parameters. Right In RDA2C, the policy is determined by the replayed data. The buffer stores accumulated advantage evidence, the dual score summarizes that evidence, and the mirror map converts it into the next policy. We consider a discounted Markov decision process ℳ=(,,P,r,ρ0,γ)M=(S,A,P,r, _0,γ), where S is the state space, A is the action space, P(⋅∣s,a)P(· s,a) is the transition kernel, r(s,a)r(s,a) is the reward function, ρ0 _0 is the initial state distribution, and γ∈[0,1)γ∈[0,1) is the discount factor. A stationary policy π:→Δ()π:S→ (A) maps each state s to an action distribution. We write τ=(s0,a0,s1,a1,…)∼πτ=(s_0,a_0,s_1,a_1,…) π for a trajectory generated by s0∼ρ0s_0 _0, at∼π(⋅∣st)a_t π(· s_t), and st+1∼P(⋅∣st,at)s_t+1 P(· s_t,a_t). The discounted return objective is J(π)=τ∼π[∑t=0∞γtr(st,at)]J(π)=E_τ π [ _t=0^∞γ^tr(s_t,a_t) ]. For a policy π, the value function, action-value function, and advantage function are Vπ(s)=τ∼π[∑t=0∞γtr(st,at)|s0=s]V^π(s)=E_τ π [ _t=0^∞γ^tr(s_t,a_t)\, |\,s_0=s ], Qπ(s,a)=r(s,a)+γs′∼P(⋅∣s,a)[Vπ(s′)]Q^π(s,a)=r(s,a)+ _s P(· s,a) [V^π(s ) ], and Aπ(s,a)=Qπ(s,a)−Vπ(s)A^π(s,a)=Q^π(s,a)-V^π(s). Equivalently, Vπ(s)=a∼π(⋅∣s)[Qπ(s,a)]V^π(s)=E_a π(· s)[Q^π(s,a)], and hence a∼π(⋅∣s)[Aπ(s,a)]=0E_a π(· s)[A^π(s,a)]=0. A policy π induces the normalized discounted state visitation distribution dπ(s)=(1−γ)∑t=0∞γtℙπ(st=s)d^π(s)=(1-γ) _t=0^∞γ^tP_π(s_t=s), where the probability is over trajectories generated by π. We write d¯t=1t∑k=1tdπk d_t= 1t _k=1^td _k for the average historical mixture of visitation distributions, and use d without a policy superscript to denote an arbitrary state density over S. We also write μπμ^π for the induced discounted state-action occupancy, with μπ(s,a)=dπ(s)π(a∣s)μ^π(s,a)=d^π(s)π(a s) and dπ(s)=∑aμπ(s,a)d^π(s)= _aμ^π(s,a), replacing the sum by an integral for continuous actions. 3 Background Mirror descent is a first-order method for optimization over non-Euclidean geometries (Nemirovsky and Yudin, 1983; Beck and Teboulle, 2003). It measures movement with a Bregman divergence induced by a strictly convex mirror potential in place of the Euclidean norm. Let φ:Δ()→ℝ : (A) be a differentiable, strictly convex statewise mirror potential on the action simplex. In this paper, we use the statewise negative entropy φ(p)=∑a∈p(a)logp(a) (p)= _a p(a) p(a) as the mirror potential. The potential on the full policy is its state-weighted average, φd(π)=s∼d[φ(π(⋅∣s))] _d(π)=E_s d[ (π(· s))] for a state weighting d, and every statewise statement below extends to the policy space through this average. For a fixed state, the Bregman divergence induced by this potential is the KL divergence Dφ(p,q)=DKL(p∥q)D_ (p,q)=D_KL(p\|q). The entropy mirror potential also identifies the action probabilities π(⋅∣s)π(· s) as primal variables and policy logits as dual coordinates. We represent the latter by logπ(⋅∣s) π(· s), omitting action-independent offsets in ∇φ∇ throughout because they do not affect the map back to the simplex. Conversely, a dual score vector z∈ℝ||z ^|A| maps back to the simplex by the inverse entropy mirror map, equivalently ∇φ∗(z)∇ ^*(z) with φ∗(z)=log∑a∈exp(z(a)) ^*(z)= _a (z(a)), yielding πz(a∣s)=∇φ∗(z)(a)=exp(z(a))∑b∈exp(z(b)) _z(a s)=∇ ^*(z)(a)= (z(a)) _b (z(b)). We call this map from dual scores to a policy the readout. It is the gradient of the conjugate potential, so it belongs to the mirror geometry, and any method that maintains dual scores obtains its policy through it. 3.1 Direct functional representation In the direct functional representation, the policy itself is the state indexed collection of action distributions π(⋅∣s):s∈\π(· s):s \. At iteration t, actor–critic methods obtain a local functional gradient direction from the critic and use it to update the policy. In policy space, this can be viewed as freezing the current visitation density dπtd _t and forming a linearized improvement objective around πt _t. With respect to the dπtd _t-weighted inner product ⟨f,π⟩d=s∼d[∑af(s,a)π(a∣s)] f,π _d=E_s d[ _af(s,a)π(a s)] between score functions and policies, the current-policy advantage AπtA _t represents the functional gradient of J at πt _t, satisfying ⟨∇πJ(πt),π⟩dπt=⟨Aπt,π⟩dπt _πJ( _t),π _d _t= A _t,π _d _t, up to terms independent of π. We therefore use AπtA _t as the functional gradient representative and write the local linearized improvement term as ⟨Aπt,π⟩dπt A _t,π _d _t. Because dπtd _t changes across rounds, accumulated feedback needs one common coordinate system. 3.2 Regularized Dual Averaging in Policy Space Regularized dual averaging (RDA) uses the same policy-space mirror geometry as Policy Mirror Descent (PMD) (Lan, 2023), but applies it to a cumulative linearized objective rather than to a one-step local surrogate (Nesterov, 2009; Xiao, 2010). In its general form, RDA selects xt+1=argminx1t∑k=1t⟨gk,x⟩+Ψ(x)+βtφ(x)x_t+1= _x\ 1t _k=1^t g_k,x + (x)+ _tt (x)\ for loss gradients gkg_k, and the three parts play different roles. The averaged linear terms carry the feedback, and every round contributes one term at equal weight. The composite term Ψ is persistent, maintaining a constant weight relative to the feedback average and thereby shaping the solution to which the iterates converge. The stabilizer φ damps early iterates, and its relative weight βt/t _t/t must vanish for the feedback to take over, which holds whenever βt _t grows sublinearly. Our instantiation takes both Ψ and φ to be entropy, with persistent weight λ and, in our experiments, a constant βt=β _t=β. The equal-weight sum over past feedback is also the term our replay window later truncates with a finite window that keeps only recent rounds, trading the variance reduction of a long memory against the staleness of old advantage estimates (Section 5). In actor–critic methods, the feedback from round t is generated under the visitation distribution dπtd _t, so each linear term retains the state weighting of the policy that produced it. To regularize different policies on the same historical state support, we weight by the historical mixture d¯t d_t from Section 1. The resulting state-weighted policy-space RDA update is πt+1=argmaxπ∈Π1t∑k=1t⟨Aπk,π⟩dπk−Ψd¯t(π)−βtφd¯t(π), _t+1= _π∈ \ 1t _k=1^t A _k,π _d _k- _ d_t(π)- _tt _ d_t(π) \, (1) where Ψd¯t _ d_t is the persistent composite regularizer, φd¯t _ d_t is a strongly convex stabilizer, and βt≥0 _t≥ 0 is the stabilizer weight. With gk=−Aπkg_k=-A _k, minimizing the averaged loss is maximizing the averaged advantage, which gives the sign convention of (1). 4 Regularized Dual Averaging Actor Critic We now present RDA2C, a replay-based actor–critic algorithm that instantiates the policy space RDA update from the previous section. The central object in RDA2C is a dual score function Zθ:×→ℝZ_θ:S×A , which approximates the state-density-weighted average advantage ZtZ_t defined in Section 4.1. Throughout, a score is a function on state-action pairs read in dual (logit) coordinates, not the likelihood score ∇θlogπθ _θ _θ used in policy gradient methods. RDA2C estimates the cumulative policy improvement signal required by RDA and then converts this score directly into a policy using entropy geometry. Concretely, at each policy round the critic attaches an advantage label to each sampled state–action pair. These labeled examples are retained for several recent rounds, and ZθZ_θ is trained by supervised regression to predict their replay-weighted average. The policy is then obtained directly from the learned score through the entropy mirror map. RDA2C thus separates the actor update into advantage-label generation, dual-score regression, and mirror-map readout. 4.1 Deriving the Dual Function RDA is written in terms of an average dual object, so we first rewrite the accumulated, state weighted exact advantage feedback as a single inner product under the historical state mixture. For states with d¯t(s)>0 d_t(s)>0, define the density weighted average advantage score by 1t∑k=1t⟨Aπk,π⟩dπk=⟨Zt,π⟩d¯t,whereZt(s,a)≔∑k=1tdπk(s)Aπk(s,a)∑k=1tdπk(s). 1t _k=1^t A _k,π _d _k= Z_t,π _ d_t, where Z_t(s,a) _k=1^td _k(s)A _k(s,a) _k=1^td _k(s). (2) The cumulative advantage feedback is then represented by the single average dual score ZtZ_t. For RDA2C, we set the composite regularizer in (1) as Ψd¯t=λφd¯t _ d_t=λ _ d_t. Both terms use the negative-entropy geometry, with the roles described in Section 3.2, where λφd¯tλ _ d_t is the persistent entropy floor, and (βt/t)φd¯t( _t/t) _ d_t is the transient stabilizer. Together they give the effective entropy temperature αt=λ+βt/t _t=λ+ _t/t. Thus, after rewriting the normalized cumulative advantage feedback as ⟨Zt,π⟩d¯t Z_t,π _ d_t, the RDA update becomes πt+1=argmaxπ∈Π⟨Zt,π⟩d¯t+αtHd¯t(π) _t+1= _π∈ \ Z_t,π _ d_t+ _tH_ d_t(π) \, where Hd¯t=−φd¯tH_ d_t=- _ d_t. Although the density-weighted objective in (1) is written over the product policy space, it decomposes statewise. For any state with d¯t(s)>0 d_t(s)>0, the same positive scalar d¯t(s) d_t(s) multiplies both the linear score term and the entropy term, so it does not affect the optimizer at that state, provided the score vector Zt(s,⋅)Z_t(s,·) can be set independently at each state, as in the tabular case. The density weights therefore determine how the average score ZtZ_t is formed and which states are covered, and once Zt(s,⋅)Z_t(s,·) is fixed, the policy at state s is obtained by the entropy mirror-map readout. With function approximation the weights matter again, because the approximator trades larger error at low-weight states for smaller error at high-weight ones. For finite action spaces, the entropy conjugate readout ∇φ∗∇ ^* is the softmax map. For continuous action spaces, it is the normalized Boltzmann readout. We use the quadratic score class Zθ(s,a)=−12∑i(ai−μθ,i(s))2/σθ,i2(s)+bθ(s)Z_θ(s,a)=- 12 _i(a_i- _θ,i(s))^2/ _θ,i^2(s)+b_θ(s), for which this readout is the diagonal Gaussian (μθ(s),αtdiag(σθ2(s)))N ( _θ(s), _tdiag( _θ^2(s)) ), and the bias bθ(s)b_θ(s) cancels in the normalization (Appendix A.2). Solving the statewise objective in closed-form gives the readout: πt+1RDA(⋅∣s)=softmax(Zt(s,⋅)αt) _t+1^RDA(· s)=softmax ( Z_t(s,·) _t ) (3) Algorithm 1 RDA2C: Regularized Dual Averaging Actor Critic Input: replay capacity or window schedule, regularization λ, stabilizer schedule βtt≥1\ _t\_t≥ 1 Initialize dual score model ZθZ_θ, critic learner C, replay buffer ℬ0=∅B_0= , and initial policy π1 _1 for iteration t=1,…,Tt=1,…,T do Collect trajectories using current policy πt _t Compute advantage labels A^t A_t with critic learner C e.g., GAE or twin-Q labels Form labeled data t=(si,ai,A^i)D_t=\(s_i,a_i, A_i)\ and push tD_t into ℬtB_t evicting old data as needed Update critic learner C using the backbone’s return or TD targets Set temperature αt=λ+βt/t _t=λ+ _t/t Fit ZθZ_θ on minibatches from ℬtB_t using (4) Read out the next policy πt+1=∇φ∗(Zθt/αt) _t+1=∇ ^*(Z_ _t/ _t) end for 4.2 Policy as data: fitting the aggregated advantage The update in (3) is defined through the average score ZtZ_t from (2), and no algorithm can maintain that object exactly, because it averages full advantage functions over rounds. RDA2C realizes the average statistically instead, through dataset aggregation (Ross et al., 2011). Each round contributes advantage-labeled samples, a sliding window retains the recent rounds, and one squared-loss regression fits a single score model to everything retained. The regression does the averaging. The mirror map does the readout. The full loop is Algorithm 1. Three variants of the algorithm follow from what kind of advantage feedback is available. The first stores the full vector A(s,⋅)A(s,·) at each visited state. It shows most directly how regression performs dual averaging: when a state is visited in several rounds, the regression drives Zθ(s,⋅)Z_θ(s,·) toward the average of the stored vectors. It requires enumerating the action set and a critic that can be queried at every action, including actions the recent policies never took. The second stores only the action taken at each visited state, which is all a value critic can label. Regression then targets the behavior-weighted average at each pair instead of the unweighted one, an extra error that the analysis in Section 5 accounts for, in exchange for feasibility with large action spaces and on-policy labels. The third sits between the two. With a Q critic, we score two actions sampled from the current policy at each replayed state, one as the label and one as the baseline, which gives a one-sample advantage label without enumeration. The GAE settings on MuJoCo and Atari use the second variant, and the twin-Q MuJoCo setting uses the third. Appendix A gives the label construction. Formally, for the second variant, at iteration t the current policy πt _t generates labeled samples t=(st,i,at,i,A^t,i)i=1ntD_t=\(s_t,i,a_t,i, A_t,i)\_i=1^n_t, and a FIFO or sliding window retains the recent rounds, ℬt=Window(1,…,t)B_t=Window(D_1,…,D_t). At actor update time, minibatches are drawn from the realized buffer, so the sampled tuples follow the buffer’s empirical distribution. Let PtrepP_t^rep denote the population replay law over labeled triples (s,a,A^)(s,a, A) induced by the window at iteration t. The population dual regression problem is to minimize ℒdual(θ,t)=(s,a,A^)∼Ptrep[(Zθ(s,a)−A^)2].L_dual(θ,t)=E_(s,a, A) P_t^rep [(Z_θ(s,a)- A)^2 ]. (4) In implementation, this loss is approximated by minibatches from ℬtB_t. The reported GAE runs on MuJoCo and Atari replace the squared loss by the clipped variant of Appendix A.4, which the ablation in Appendix D.4 shows matters in practice. The twin-Q runs use the squared loss on standardized labels. For the squared-loss objective, the population regression target is the replay conditional mean label Z^t(s,a)=Ptrep[A^∣S=s,A=a] Z_t(s,a)=E_P_t^rep[ A S=s,A=a]. The regression therefore fits the replay Bayes target Z^t Z_t, and fitting on a finite buffer introduces additional sampling error. This is the sense in which the policy is represented by data, with the buffer defining an empirical dual objective, ZθZ_θ serving as its fitted representation, and the entropy readout of ZθZ_θ giving the current policy. The action weighting is the one gap the analysis closes by assumption: the bounds charge it to a coverage coefficient, which is finite when the replayed policies put positive probability on every action at every state the comparator visits (Appendix F.4). 4.3 Connections to Key Baselines We highlight the algorithmic differences of the baselines most relevant to our method; a broader discussion of related work appears in Appendix G. Connection to SAC. SAC and the twin-Q variant of RDA2C use the same critic-learning procedure but differ in their actor targets and update mechanisms. SAC’s actor tracks the softmax of the latest Q estimate, whereas RDA2C derives its policy from a fitted replay-weighted average of recent critic advantage labels. SAC optimizes a KL projection objective by stochastic gradient through reparameterized action samples a=fϕ(ϵ,s)a=f_φ(ε;s), which requires continuous actions (Haarnoja et al., 2018). RDA2C instead fits a quadratic dual score to stored labels by supervised regression and obtains its policy through the entropy mirror map. Despite these differences, we hypothesize comparable performance under matched critic learning. Tomar et al. (2022) show that SAC’s actor step is an entropy-regularized follow-the-regularized-leader update, the same form as the regularized dual averaging step behind RDA2C, so with the critic shared the two methods differ only in how the cumulative score is formed. They also report that off-policy MDPO-KL performs on par with SAC across all six MuJoCo tasks in their evaluation, providing an empirical precedent for comparable performance despite differences in the actor update. This parity is useful because RDA2C is not tied to a particular critic construction or action space. The same actor update can use either GAE labels from a value critic or advantage labels from twin-Q critics, and applies to both discrete and continuous action spaces. SAC, by contrast, relies on a soft Q-critic construction for its actor update and is therefore not directly applicable in settings where only trajectory-based GAE labels are available, and its reparameterized actor step does not extend to discrete actions. SAC is not applicable to discrete action settings without a separate derivation (Christodoulou, 2019), and the resulting discrete variant needs its own stabilization (Zhou et al., 2022). Our results show that RDA2C improves over PPO in the GAE setting while matching SAC when twin-Q labels are available. Parity at SAC’s own strength, together with gains over PPO where SAC does not apply, makes RDA2C the better global default across the two suites. Connection to AAPDA. AAPDA (Gao et al., 2026) and RDA2C both build on policy dual averaging, but differ in how they obtain the policy from the accumulated feedback. AAPDA learns a separate actor to approximate the policy dual-averaging target. RDA2C instead fits the dual score from replayed advantage labels and obtains the policy directly through the entropy mirror map, without a separate actor-fitting stage. 5 Theoretical Analysis The analysis is for finite action spaces and separates the entropy-RDA optimization term from the replay score errors introduced by the fitted actor. The fitted actor is the entropy readout of the learned replay score. As in (3), πθt _ _t is the policy that Algorithm 1 reads out at the end of round t−1t-1 from Zθt−1Z_ _t-1 with temperature αt−1 _t-1, πθt(⋅∣s)=softmax(Zθt−1(s,⋅)/αt−1),αt−1=λ+β/(t−1),λ>0,β≥0. _ _t(· s)=softmax\! (Z_ _t-1(s,·)/ _t-1 ), _t-1=λ+β/(t-1), λ>0,\ β≥ 0. (5) Here λ is the persistent entropy floor and β/(t−1)β/(t-1) is the decaying RDA stabilizer. The analysis specializes the stabilizer schedule βt\ _t\ of Algorithm 1 to a constant, βt≡β _t≡β, giving αt=λ+β/t _t=λ+β/t. The proof converts policy-space regret into value loss through the performance difference lemma in Appendix F.1. Replay window and regression bridge. The theory analyzes a replay window t=t−Wt,…,t−1W_t=\t-W_t,…,t-1\ of length Wt=max1,⌊tζ⌋W_t= \1, t^ζ \. Using the state-action occupancy notation from Section 1, set μk=μπθk _k=μ _ _k, i.e., μk(s,a)=dπθk(s)πθk(a∣s) _k(s,a)=d _ _k(s) _ _k(a s), and let μt(s,a)=Wt−1∑k∈tμk(s,a) _t(s,a)=W_t^-1 _k _t _k(s,a) be the population replay marginal. The squared-loss replay regression fits Zθt−1Z_ _t-1 to the replay Bayes target of (4) and, with true advantage labels Ak=AπθkA_k=A _ _k, this target is ZtA(s,a)=(∑k∈tμk(s,a)Ak(s,a))/(∑k∈tμk(s,a)),Z_t^A(s,a)= ( _k _t _k(s,a)A_k(s,a) ) / ( _k _t _k(s,a) ), on the support of μt _t. Thus ZtAZ_t^A is the replay-weighted average advantage that bridges the current advantage AtA_t and the learned score Zθt−1Z_ _t-1, where Ak=AπθkA_k=A _ _k are the advantages of the learned policies. The comparator coverage by replay is measured by Ccov(π⋆)=max1≤t≤Tesssups,adπ⋆(s)/μt(s,a)C_cov(π )= _1≤ t≤ T *ess\,sup_s,ad^π (s)/ _t(s,a), which is finite if μt(s,a)>0 _t(s,a)>0 for every action a at every state with dπ⋆(s)>0d^π (s)>0, a universal action coverage condition at comparator states. We first state the bound for the regularized value gap, the average regret against the comparator π⋆π with the persistent entropy correction kept on the left-hand side: Treg(π⋆)≔1T−1∑t=2T[(J(π⋆)−J(πθt))+λ1−γs∼dπ⋆(φ(πθt(⋅∣s))−φ(π⋆(⋅∣s)))].G_T^reg(π ) 1T-1 _t=2^T [ (J(π )-J( _ _t) )+ λ1-γE_s d^π ( ( _ _t(· s))- (π (· s)) ) ]. Theorem 5.1 (Fitted replay value bound). Assume A finite, |r(s,a)|≤Rmax|r(s,a)|≤ R_ , the policy path stability of Assumption F.2 (constant CπC_π) and the replay error bounds of Assumption F.8 (Appendix F.5), and Ccov(π⋆)<∞C_cov(π )<∞. Let ζ∈(0,1)ζ∈(0,1) and Cloc=(1+γ)RmaxCπ(1−γ)2(1−2ζ−1)C_loc= (1+γ)R_ C_π(1-γ)^2(1-2^ζ-1). Then for every stationary comparator π⋆π satisfying the above assumptions, Treg(π⋆)≤11−γ[ _T^reg(π )≤ 11-γ [ βln||(1+lnT)T−1+2ClocζTζT−1 β |A|(1+ T)T-1+ 2C_locζ T^ζT-1 +2Ccov(π⋆)(ϵ¯fit+ϵ0+Cvar1−ζ/2T1−ζ/2T−1)]=:ℬT(π⋆). +2 C_cov(π ) ( ε_fit+ _0+ C_var1-ζ/2 T^1-ζ/2T-1 ) ]=:B_T(π ). Converting the regularized gap to ordinary value costs an entropy bias term. The conversion turns the guarantee into an ordinary average regret bound against π⋆π , comparable with unregularized methods, with an explicit bias term that scales with λ and represents the price of the entropy floor retained on the left-hand side of the regularized bound. Corollary 5.2 (Unregularized value bound). Under the assumptions of Theorem 5.1, moving the entropy correction to the right gives 1T−1∑t=2T(J(π⋆)−J(πθt))≤λln||1−γ+ℬT(π⋆), 1T-1 _t=2^T (J(π )-J( _ _t) )≤ λ |A|1-γ+B_T(π ), with ℬT(π⋆)B_T(π ) the right hand side of Theorem 5.1. With ζ=2/3ζ=2/3 and fixed λ>0λ>0 the ordinary value gap converges at rate T−1/3T^-1/3 to the floor set by the entropy bias λln||1−γ λ |A|1-γ and the persistent fitting and critic error 2Ccov(π⋆)1−γ(ϵ¯fit+ϵ0) 2 C_cov(π )1-γ( ε_fit+ _0). 6 Experiments We evaluate whether the replay estimator proposed in Section 4 gives a useful actor update in both continuous and discrete control. The experiments ask five questions: whether RDA2C improves over PPO when both methods use GAE labels, whether the same actor holds parity with SAC when labels come from twin-Q critics, whether it improves on AAPDA, the prior dual-averaging actor, at that method’s released defaults, whether the same update carries over to image-based discrete control on Atari, and whether the replay window behaves like the finite replay and drift terms in Section 5. Additional ablations and control experiments in Appendix D examine the effects of individual algorithmic components and implementation choices. 6.1 Experimental setup Figure 2: RDA2C improves over matched PPO on six of eight MuJoCo tasks and over AAPDA on six of eight. Bars show per-environment relative improvement, 100⋅(R¯ours−R¯base)/|R¯base|100·( R_ours- R_base)/| R_base|. Left: PPO matched to RDA2C’s architecture and hyperparameters (GAE labels, online return). Right: AAPDA at its released defaults (deterministic evaluation). Exact values: Appendix B. Environments and evaluation. For the continuous control suite, we use eight Gymnasium MuJoCo v5 tasks (Todorov et al., 2012). Both the GAE and twin-Q critic MuJoCo comparisons run for 1M environment steps and use 10 random seeds. The SAC comparison reports deterministic mean action evaluation over 20 episodes every 50K steps. For the Atari suite, we evaluate on twelve Atari 2600 games (Bellemare et al., 2013) for 10M frames with 10 random seeds, using the EnvPool vectorized environment wrapper (Weng et al., 2022). Across both suites, we follow the evaluation and reporting conventions used in the corresponding baseline literature, as detailed in Appendix B. Baselines and model classes. The GAE-based MuJoCo comparison uses PPO (Schulman et al., 2017) as the baseline. We match the 2×1282× 128 trunks, learning rate, gradient clipping, GAE λ, batch sizes, update epochs, and horizon, so the actor update is the intended difference. Both methods use a value network separate from the actor. RDA2C derives its policy from a dual score that is quadratic in the actions (Appendix A.2), whereas PPO directly parameterizes a Gaussian policy with a global log-standard-deviation. The twin-Q MuJoCo comparison uses SAC (Haarnoja et al., 2018) with matched transition-replay settings, twin-Q critic architectures, target updates, optimizer, batch size (512512), and policy update frequency (11). RDA2C retains the critic-learning procedure but replaces SAC’s actor update by fitting a quadratic dual score to replayed advantage labels and obtaining the policy through the entropy mirror map. We also compare against actor-accelerated policy dual averaging (AAPDA) (Gao et al., 2026) on MuJoCo. AAPDA learns a separate actor to approximate the policy dual-averaging target, whereas RDA2C reads its policy directly from the fitted dual score. We use AAPDA’s released defaults without per-task tuning, with both methods trained for 1M environment steps over 10 paired training seeds. The evaluation protocol is detailed in Appendix B. For Atari, RDA2C uses a discrete softmax readout and separate convolutional feature extractors for the score model and value function, while PPO keeps the shared trunk of its standard implementation (Huang et al., 2022). 6.2 Experimental results GAE labels: RDA2C improves over PPO on six of eight MuJoCo tasks Figure 2 summarizes the relative improvement over matched PPO, and Appendix Table 4 reports the raw mean and standard deviation over 10 seeds. RDA2C improves over matched PPO on six of eight MuJoCo tasks and gives a fixed-suite average per-environment relative improvement of +49.5%+49.5\% (paired seed bootstrap 95% CI [+42.1%,+56.2%][+42.1\%,+56.2\%]).22 2 Percentile bootstrap intervals can undercover at seed counts like ours: Jordan et al. (2020) measure failure rates above twice the nominal level, and in a simulation matched to our data the suite-level (mujoco and atari suites) procedure used here covered a nominally 95% target 90.5% of the time. We therefore read these intervals as descriptive uncertainty summaries. This caveat applies to every bootstrap interval in the paper. The largest gains are on HalfCheetah-v5, where RDA2C reaches 70997099 compared with PPO’s 18641864, and Humanoid-v5, where RDA2C reaches 37683768 compared with PPO’s 23702370. These gains are obtained against a strengthened PPO baseline. The matched configuration used here has higher mean returns than PPO’s CleanRL defaults on five of eight tasks (Appendix B). Twin-Q critic labels: RDA2C matches SAC under matched settings RDA2C is higher on four tasks, ties on one, and is lower on three. Appendix Table 5 reports the underlying best deterministic evaluations within the 1M-step budget over 10 seeds. The suite-level summary is a fixed-suite average per-environment relative change of +0.64%+0.64\% (paired seed bootstrap 95% CI [−2.80%,+4.06%][-2.80\%,+4.06\%])2. The largest single-task difference favors RDA2C on Walker2d-v5 (45054505 vs 40804080), and the largest in SAC’s favor is on HalfCheetah-v5 (95259525 vs 1011910119). No taskwise difference is decisive at 10 seeds. The outcome matches the expectation in Section 4.3, and it answers the question the experiment asks: the same dual-score regression works with twin-Q labels in place of GAE, at SAC’s level of return. The comparison against SAC at its CleanRL-default batch size and update frequency appears in Appendix B. AAPDA: RDA2C improves on the closest dual-averaging baseline AAPDA is the closest runnable dual-averaging baseline (Gao et al., 2026), described in Appendix G.2. We ran the authors’ released defaults for 1M steps with no per-task tuning. RDA2C is higher on six of eight tasks. The right panel of Figure 2 summarizes the comparison, and Appendix Table 6 gives the per-task returns. The fixed-suite mean relative change is +41.8%+41.8\% (paired seed bootstrap 95% CI [+26.3%,+71.2%][+26.3\%,+71.2\%])2 and the median is +13.8%+13.8\% (CI [+4.9%,+27.0%][+4.9\%,+27.0\%])2. AAPDA is ahead on HumanoidStandup-v5 and Walker2d-v5. Atari: RDA2C improves over PPO on eight of twelve games Figure 3 reports relative improvement over PPO after 10M frames, and Appendix Table 9 gives the raw returns. RDA2C improves over PPO on 8 of the 12 Atari games, with a fixed-suite average per-environment relative improvement of +112.4%+112.4\% (paired seed bootstrap 95% CI [+88.5%,+141.2%][+88.5\%,+141.2\%])2 and a median relative improvement of +32.9%+32.9\%. The largest gains are on Seaquest-v5, Enduro-v5, and BeamRider-v5. PPO is higher on Breakout-v5, Freeway-v5, Pong-v5, and Qbert-v5. The Atari results show that the same average advantage parameterization can scale to image-based discrete control. The table reports mean running return and seed standard deviation over 10 random seeds. Figure 3: RDA2C improves over PPO on eight of twelve Atari games. Bars show relative improvement over PPO, 100⋅(R¯RDA2C−R¯PPO)/|R¯PPO|100·( R_RDA2C- R_PPO)/| R_PPO|, after 10M frames. The value axis is broken between +95%+95\% and +235%+235\%. The bars for BeamRider-v5, Enduro-v5, and Seaquest-v5 continue in the right segment. Exact mean ± std values are reported in Appendix B, Table 9. 7 Conclusion and Future Work We introduced RDA2C, a replay-based actor–critic implementation of regularized dual averaging via advantage regression. In RDA2C, replayed advantage-labeled samples define an empirical cumulative dual objective, which is summarized by a learned score model. The policy is then obtained directly through the entropy mirror map. Replay then serves both as a source of additional optimization steps and as the representation of policy-improvement feedback from which the current actor is derived. AI use statement In this work, we used generative AI tools to implement analysis and sweep code. Additionally, we used generative AI tools to edit parts of the paper for readability and structure and to create and modify figures. We have reviewed all AI-assisted work. All experiments were run by the authors, all AI-generated code was checked and tested by the authors, and all AI-drafted text was revised by the authors. We take responsibility for the final content of this work, including text, claims, or artifacts produced with the aid of generative AI. Reproducibility statement Appendix A gives network architectures, hyperparameters for every setting, and the label constructions. Appendix B reports the numerical results behind every figure, with seed counts and evaluation protocols. The assumptions and complete proofs of the theoretical results are in Appendix F. The code for all three entrypoints follows the CleanRL single-file convention and is available at https://github.com/InsultedByMaths/rda2c. Acknowledgments KB and NP acknowledge this work has been made possible in part by a gift from the Chan Zuckerberg Initiative Foundation to establish the Kempner Institute for the Study of Natural and Artificial Intelligence. References Abbasi-Yadkori et al. (2019) Y. Abbasi-Yadkori, P. Bartlett, K. Bhatia, N. Lazic, C. Szepesvári, and G. Weisz POLITEX: regret bounds for policy iteration using expert prediction. In Proceedings of the 36th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 97, p. 3692–3702. Cited by: §G.2. Abdolmaleki et al. (2018) A. Abdolmaleki, J. T. Springenberg, Y. Tassa, R. Munos, N. Heess, and M. Riedmiller Maximum a posteriori policy optimisation. In International Conference on Learning Representations, External Links: Link Cited by: §G.1. Agarwal et al. (2019) A. Agarwal, S. M. Kakade, J. D. Lee, and G. Mahajan On the theory of policy gradient methods: optimality, approximation, and distribution shift. arXiv preprint arXiv:1908.00261. External Links: 1908.00261, Document, Link Cited by: §G.1. Beck and Teboulle (2003) A. Beck and M. Teboulle Mirror descent and nonlinear projected subgradient methods for convex optimization. Operations Research Letters 31 (3), p. 167–175. Cited by: §3. Bellemare et al. (2013) M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling The arcade learning environment: an evaluation platform for general agents. Journal of Artificial Intelligence Research 47, p. 253–279. Cited by: §6.1. Brantley et al. (2025) K. Brantley, M. Chen, Z. Gao, J. D. Lee, W. Sun, W. Zhan, and X. Zhang Accelerating RL for LLM reasoning with optimal advantage regression. arXiv preprint arXiv:2505.20686. External Links: 2505.20686, Document, Link Cited by: §G.4. Christodoulou (2019) P. Christodoulou Soft actor-critic for discrete action settings. arXiv preprint arXiv:1910.07207. Cited by: §4.3. Espeholt et al. (2018) L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, I. Dunning, S. Legg, and K. Kavukcuoglu IMPALA: scalable distributed deep-RL with importance weighted actor-learner architectures. In Proceedings of the 35th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 80, p. 1407–1416. External Links: Link Cited by: §1. Fedus et al. (2020) W. Fedus, P. Ramachandran, R. Agarwal, Y. Bengio, H. Larochelle, M. Rowland, and W. Dabney Revisiting fundamentals of experience replay. In International conference on machine learning, p. 3061–3071. Cited by: §1. Fujimoto et al. (2018) S. Fujimoto, H. van Hoof, and D. Meger Addressing function approximation error in actor-critic methods. In Proceedings of the 35th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 80, p. 1587–1596. External Links: Link Cited by: Appendix B, §1. Gao et al. (2026) J. Gao, C. Ju, G. Lan, and Z. Tong Actor-accelerated policy dual averaging for reinforcement learning in continuous action spaces. arXiv preprint arXiv:2603.10199. External Links: 2603.10199, Link Cited by: Appendix B, §G.2, §4.3, §6.1, §6.2. Gao et al. (2024a) Z. Gao, J. D. Chang, W. Zhan, O. Oertell, G. Swamy, K. Brantley, T. Joachims, J. A. Bagnell, J. D. Lee, and W. Sun REBEL: reinforcement learning via regressing relative rewards. arXiv preprint arXiv:2404.16767. External Links: 2404.16767, Document, Link Cited by: §G.4. Gao et al. (2024b) Z. Gao, W. Zhan, J. D. Chang, G. Swamy, K. Brantley, J. D. Lee, and W. Sun Regressing the relative future: efficient policy optimization for multi-turn RLHF. arXiv preprint arXiv:2410.04612. External Links: 2410.04612, Document, Link Cited by: §G.4. Geist et al. (2019) M. Geist, B. Scherrer, and O. Pietquin A theory of regularized markov decision processes. In Proceedings of the 36th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 97, p. 2160–2169. External Links: Link Cited by: §G.1, §G.1. Haarnoja et al. (2018) T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference on Machine Learning, External Links: 1801.01290, Document, Link Cited by: Appendix B, §G.1, §G.3, §1, §4.3, §6.1. Huang et al. (2022) S. Huang, R. F. J. Dossa, A. Raffin, A. Kanervisto, and W. Wang The 37 implementation details of proximal policy optimization. Note: ICLR Blog Track External Links: Link Cited by: Appendix B, §6.1. Iwaki (2025) R. Iwaki Mirror descent actor critic via bounded advantage learning. arXiv preprint arXiv:2502.03854. External Links: 2502.03854, Document, Link Cited by: §G.1. Jordan et al. (2020) S. M. Jordan, Y. Chandak, D. Cohen, M. Zhang, and P. S. Thomas Evaluating the performance of reinforcement learning algorithms. In Proceedings of the 37th International Conference on Machine Learning, Cited by: footnote 2. Ju and Lan (2022) C. Ju and G. Lan Policy optimization over general state and action spaces. arXiv preprint arXiv:2211.16715. External Links: 2211.16715, Document, Link Cited by: §G.2. Kakade and Langford (2002) S. Kakade and J. Langford Approximately optimal approximate reinforcement learning. In Proceedings of the Nineteenth International Conference on Machine Learning, p. 267–274. Cited by: Lemma F.1. Kakade (2001) S. M. Kakade A natural policy gradient. In Advances in neural information processing systems, p. 1531–1538. Cited by: §G.1, §1. Lan (2023) G. Lan Policy mirror descent for reinforcement learning: linear convergence, new sampling complexity, and generalized problem classes. Mathematical Programming. External Links: Link Cited by: §G.1, §1, §3.2. Lillicrap et al. (2016) T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra Continuous control with deep reinforcement learning. In International Conference on Learning Representations, External Links: 1509.02971, Link Cited by: §1. McMahan (2011) B. McMahan Follow-the-regularized-leader and mirror descent: equivalence theorems and L1 regularization. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, Proceedings of Machine Learning Research, Vol. 15, p. 525–533. External Links: Link Cited by: §G.2, §1. Mei et al. (2019) J. Mei, C. Xiao, R. Huang, D. Schuurmans, and M. Müller On principled entropy exploration in policy optimization. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI-19), External Links: Link Cited by: §G.1. Mnih et al. (2015) V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep reinforcement learning. Nature 518 (7540), p. 529–533. Cited by: §1. Nair et al. (2020) A. Nair, A. Gupta, M. Dalal, and S. Levine AWAC: accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359. Cited by: §G.4. Nemirovsky and Yudin (1983) A. S. Nemirovsky and D. B. Yudin Problem complexity and method efficiency in optimization. Wiley-Interscience. Cited by: §3. Nesterov (2009) Y. Nesterov Primal-dual subgradient methods for convex problems. Mathematical programming 120 (1), p. 221–259. Cited by: §G.2, §1, §3.2. Neu et al. (2017) G. Neu, A. Jonsson, and V. Gómez A unified view of entropy-regularized markov decision processes. arXiv preprint arXiv:1705.07798. External Links: 1705.07798, Document Cited by: §G.1. Peng et al. (2019) X. B. Peng, A. Kumar, G. Zhang, and S. Levine Advantage-weighted regression: simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177. Cited by: §G.4. Ross and Bagnell (2014) S. Ross and J. A. Bagnell Reinforcement and imitation learning via interactive no-regret learning. arXiv preprint arXiv:1406.5979. Cited by: §G.4. Ross et al. (2011) S. Ross, G. Gordon, and D. Bagnell A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, p. 627–635. Cited by: §G.4, §4.2. Schaul et al. (2015) T. Schaul, J. Quan, I. Antonoglou, and D. Silver Prioritized experience replay. arXiv preprint arXiv:1511.05952. Cited by: §1. Schulman et al. (2015) J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz Trust region policy optimization. In International conference on machine learning, p. 1889–1897. Cited by: §G.1, §1. Schulman et al. (2016) J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel High-dimensional continuous control using generalized advantage estimation. In Proceedings of the International Conference on Learning Representations (ICLR), Cited by: §A.2. Schulman et al. (2017) J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Cited by: §A.4, §G.1, §G.3, §1, §1, §6.1. Shani et al. (2020) L. Shani, Y. Efroni, and S. Mannor Adaptive trust region policy optimization: global convergence and faster rates for regularized mdps. In Proceedings of the AAAI conference on artificial intelligence, Vol. 34, p. 5668–5675. Cited by: §G.1. Todorov et al. (2012) E. Todorov, T. Erez, and Y. Tassa MuJoCo: a physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, p. 5026–5033. Cited by: §6.1. Tomar et al. (2022) M. Tomar, L. Shani, Y. Efroni, and M. Ghavamzadeh Mirror descent policy optimization. In International Conference on Learning Representations, External Links: 2005.09814, Document, Link Cited by: §G.1, §4.3. Toneva et al. (2019) M. Toneva, A. Sordoni, R. Tachet des Combes, A. Trischler, Y. Bengio, and G. J. Gordon An empirical study of example forgetting during deep neural network learning. In International Conference on Learning Representations, Cited by: §G.4. Vaswani et al. (2022) S. Vaswani, O. Bachem, S. Totaro, R. Müller, S. Garg, M. Geist, M. C. Machado, P. S. Castro, and N. Le Roux A general class of surrogate functions for stable and efficient reinforcement learning. In Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, Proceedings of Machine Learning Research, Vol. 151, p. 8619–8649. External Links: Link Cited by: §G.1. Vieillard et al. (2020a) N. Vieillard, T. Kozuno, B. Scherrer, O. Pietquin, R. Munos, and M. Geist Leverage the average: an analysis of KL regularization in reinforcement learning. In Advances in Neural Information Processing Systems, Vol. 33. External Links: 2003.14089 Cited by: §G.2. Vieillard et al. (2020b) N. Vieillard, O. Pietquin, and M. Geist Deep conservative policy iteration. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34, p. 6070–6077. External Links: 1906.09784 Cited by: §G.2. Wang et al. (2017) Z. Wang, V. Bapst, N. Heess, V. Mnih, R. Munos, K. Kavukcuoglu, and N. de Freitas Sample efficient actor-critic with experience replay. In International Conference on Learning Representations, External Links: 1611.01224, Link Cited by: §1. Wang et al. (2020) Z. Wang, A. Novikov, K. Zolna, J. Merel, J. T. Springenberg, S. Reed, B. Shahriari, N. Siegel, C. Gulcehre, N. Heess, and N. de Freitas Critic regularized regression. In Advances in Neural Information Processing Systems, Vol. 33. External Links: 2006.15134 Cited by: §G.4. Weng et al. (2022) J. Weng, M. Lin, S. Huang, B. Liu, D. Makoviichuk, V. Makoviychuk, Z. Liu, Y. Song, T. Luo, Y. Jiang, Z. Xu, and S. Yan EnvPool: a highly parallel reinforcement learning environment execution engine. arXiv preprint arXiv:2206.10558. External Links: 2206.10558, Link Cited by: §6.1. Xiao (2010) L. Xiao Dual averaging methods for regularized stochastic learning and online optimization. Journal of Machine Learning Research 11 (Oct), p. 2543–2596. Cited by: §G.2, §1, §3.2. Xiong et al. (2024) Z. Xiong, M. Fazel, and L. Xiao Dual approximation policy optimization. arXiv preprint arXiv:2410.01249. External Links: 2410.01249, Document, Link Cited by: §G.2. Zhou et al. (2022) H. Zhou, T. Wei, Z. Lin, J. Li, J. Xing, Y. Shi, L. Shen, C. Yu, and D. Ye Revisiting discrete soft actor-critic. arXiv preprint arXiv:2209.10081. Note: Accepted at Transactions on Machine Learning Research Cited by: §4.3. Appendix Contents Appendix A Implementation Details A.1 Network Architectures We utilize standard architectures for continuous and discrete control domains and report the relevant capacity differences explicitly. For GAE MuJoCo, RDA2C uses the quadratic dual score actor described below with a separate critic, while PPO uses the standard CleanRL Gaussian policy/value architecture. Continuous control (MuJoCo, GAE labels). For the GAE MuJoCo experiments, the actor is the quadratic dual score network described in Section 4. It has a two-layer Tanh MLP trunk of width 128 and three heads: Gaussian mean, state dependent log standard deviation, and scalar bias bθ(s)b_θ(s). The value function is a separate two-layer Tanh MLP of width 128. During rollout collection we also store the old dual score prediction Zold(s,a)Z_old(s,a) for each sampled transition, which is used only by the optional clipped dual regression loss of Appendix A.4. • Dual Score / Policy Network: Input state → Linear(128) → Tanh → Linear(128) → Tanh → Mean / LogStd / Bias heads. • Value Network: Input state → Linear(128) → Tanh → Linear(128) → Tanh → Linear(1). Update schedule (MuJoCo, GAE labels). Each iteration collects one rollout of 20482048 steps from a single environment, computes GAE labels, and pushes the 20482048 tuples (s,a,A^,Zold)(s,a, A,Z_old) into the FIFO label buffer of capacity W=50,000W=50,000, which therefore holds the most recent 2424 rollouts once full. The stored action is the unsquashed Gaussian sample. The environment receives that sample clipped to the action bounds, so the policy is a clipped Gaussian and the regression is fit on the pre-clip action. The dual regression then runs 1010 epochs of 3232 minibatches of 6464 tuples, 320320 gradient steps per iteration. The 20,48020,480 tuples are drawn uniformly with replacement from the buffer once per iteration, standardized as a batch, shuffled, and consumed sequentially, so a stored label is used about 0.40.4 times per iteration on average and about 1010 times over its residence in the buffer. The value loss in the same gradient step uses the on-policy rollout minibatch of the same size. The temperature αt=λ+β/t _t=λ+β/t is computed once per rollout, so with λ=0.05λ=0.05 and β=1β=1 it decays from 1.051.05 to λ+β/488λ+β/488 over the 488488 rollouts that make up 1M steps. Continuous control (MuJoCo, twin-Q critic labels). For the SAC comparison, both SAC and RDA2C with twin-Q labels use the standard CleanRL SAC backbone: two-layer MLPs with width 256 and ReLU activations, together with twin-Q critics. The SAC baseline uses the standard squashed Gaussian actor and default SAC hyperparameters. Our RDA2C with twin-Q labels variant reuses the same replay / critic backbone and the same width actor trunk, but replaces the SAC actor loss with regression onto the learned quadratic dual score from Section 4. Concretely, the RDA2C with twin-Q labels actor has mean, log standard deviation, and scalar bias heads, while each Q-network maps (s,a)(s,a) through 256-256 ReLU layers to a scalar estimate. The reported twin-Q critic RDA2C runs use automatic temperature tuning in place of the hand-designed RDA schedule used in the GAE experiments. Specifically, for RDA2C twin-Q we optimize a scalar logαt _t against target entropy −|A|-|A|. This learned temperature determines the sampling scale of the squashed Gaussian policy through σ(s)αtσ(s) _t and enters both the soft critic bootstrap target and the actor regression labels. For K=2K=2 current-policy samples a1,…,aKa_1,…,a_K at a replayed state s, we define xi(s)=min(Q1,Q2)(s,ai)−αtlogπθ(ai∣s),gi(s)=xi(s)−1K−1∑j≠ixj(s).x_i(s)= (Q_1,Q_2)(s,a_i)- _t _θ(a_i s), g_i(s)=x_i(s)- 1K-1 _j≠ ix_j(s). The regression buffer stores (s,ai,gi)(s,a_i,g_i), where aia_i is the corresponding pre-Tanh Gaussian sample. We store raw labels in a FIFO label buffer of total size W=12,800W=12,800, standardize them within sampled regression minibatches, clip the standardized targets to [−10,10][-10,10], and regress the quadratic dual score Zθ(s,Ai)Z_θ(s,A_i) to these buffered targets. In the primary comparison both methods use batch size 512 and policy frequency 1. SAC at its CleanRL defaults, batch size 256 and policy frequency 2, is reported separately in Appendix B. Both methods use 5 parallel environments and deterministic evaluation over 20 episodes. Because autotuning is enabled throughout this comparison, the RDA schedule parameters (λ,β)(λ,β) are inactive. All MuJoCo runs used 2 CPU cores and no GPUs. All Atari runs used a single H100 GPU. A.2 Action classes and label settings Handling discrete and continuous actions. For discrete actions, Zθ(s)Z_θ(s) outputs one score per action, and the entropy readout is the corresponding softmax policy. For continuous control, we choose a quadratic dual score class, Zθ(s,a)=−12∑i(ai−μθ,i(s))2σθ,i2(s)+bθ(s)Z_θ(s,a)=- 12 _i (a_i- _θ,i(s))^2 _θ,i^2(s)+b_θ(s). Under the same entropy mirror map, softmax(Zθ(s,⋅)/αt)softmax(Z_θ(s,·)/ _t) is the diagonal Gaussian (μθ(s),αtdiag(σθ2(s)))N ( _θ(s), _t\,diag( _θ^2(s)) ). The additive term bθ(s)b_θ(s) cancels in the normalization and does not affect the policy. Labels, replay windows, and stabilization. The GAE label variants in MuJoCo and Atari compute advantage labels from a value critic using generalized advantage estimates (Schulman et al., 2016). The twin-Q label variant constructs labels from the two critics by sampling actions from the current policy at replayed states and taking the smaller of the resulting soft values. The full label construction is given above. All RDA2C variants use a fixed sliding replay window for the dual regression problem. This label window is separate from the transition replay buffer that the twin-Q backbone shares with SAC. Unless changed in the ablations, we use W=50,000W=50,000 label tuples for the GAE MuJoCo setting, W=12,800W=12,800 label tuples for the twin-Q critic MuJoCo setting, and W=500,000W=500,000 frames for Atari. The Atari GAE runs use clipped dual regression with clip radius 1.01.0, stabilizer β=0.6β=0.6, and entropy floor λ=0.005λ=0.005. The clipped dual regression is described in Appendix A.4. A.3 Hyperparameters We provide the hyperparameter configurations for both domains. To ensure a fair comparison, we match the Total Batch Size between RDA2C and the PPO baseline. Method-specific learning-rate and stabilization settings are listed explicitly below. Continuous control (MuJoCo, GAE labels). Table 1 lists the hyperparameters for the GAE MuJoCo benchmark. Table 1: MuJoCo Hyperparameters (GAE labels). Comparison of the GAE RDA2C configuration reported in Table 4 and the PPO baseline. RDA2C uses a replay buffer of size W=50,000W=50,000 and enables optional clipped prediction regression with ϵclip=0.2 _clip=0.2. The target-advantage clip is inactive for this configuration. Parameter RDA2C (Ours) PPO (Baseline) Shared Rollout / Optimization Settings Optimizer Adam Adam Discount Factor (γ) 0.99 0.99 Total Timesteps 1,000,0001,000,000 1,000,0001,000,000 Total Batch Size 2048 2048 Minibatch Size 64 64 VF Coefficient 0.5 0.5 Update Epochs 10 10 Num. Environments 1 1 Steps per Env 2048 2048 Method Specific Settings Actor Trunk 2×1282× 128 Tanh MLP 2×1282× 128 Tanh MLP Actor Heads Mean / state LogStd / bias Mean / global LogStd Value Network 2×1282× 128 Tanh MLP 2×1282× 128 Tanh MLP Learning Rate 2×10−42× 10^-4 2×10−42× 10^-4 Hidden Width 128 128 Max Gradient Norm 3.0 3.0 GAE Lambda (λGAE _GAE) 0.80 0.80 PPO Policy Ratio Clip – 0.2 Value Loss Clipping Yes, ϵ=0.2ε=0.2 Yes, ϵ=0.2ε=0.2 RDA2C Dual Regression Settings Replay Window Size (W) 50,00050,000 – Regularizer (λ) 0.05 – Stabilizer (β) 1.0 – Dual Prediction Clip (ϵclip _clip) 0.2 – Continuous control (MuJoCo, twin-Q critic labels). Table 2 lists the hyperparameters for the SAC comparison. Both methods use the standard SAC replay and twin-Q architecture, but RDA2C with twin-Q labels uses a tuned actor regression configuration in place of the default SAC update schedule. The table therefore separates common backbone/evaluation settings from the method specific batch, policy frequency, temperature, and label construction settings. Table 2: MuJoCo Hyperparameters (twin-Q critic labels). Comparison of the SAC baseline at matched batch size and policy update frequency and the RDA2C with twin-Q labels instantiation reported in Table 5. Parameter RDA2C with twin-Q labels (Ours) SAC (Baseline) Common Backbone / Evaluation Settings Actor Trunk 2×2562× 256 ReLU MLP 2×2562× 256 ReLU MLP Actor Heads Mean, LogStd, bias Mean, LogStd Twin-Q Networks 2×2562× 256 ReLU MLP 2×2562× 256 ReLU MLP Total Timesteps 1,000,0001,000,000 1,000,0001,000,000 Num. Environments 5 5 Replay Buffer Size 1,000,0001,000,000 1,000,0001,000,000 Discount Factor (γ) 0.99 0.99 Target Smoothing (τ) 0.005 0.005 Policy Learning Rate 3×10−43× 10^-4 3×10−43× 10^-4 Q-Learning Rate 1×10−31× 10^-3 1×10−31× 10^-3 Learning Starts 5,0005,000 5,0005,000 Target Network Frequency 1 1 Temperature Tuning Autotune Autotune Evaluation Policy Deterministic mean action Deterministic mean action Eval Frequency / Episodes 50,00050,000 / 20 50,00050,000 / 20 Method Specific SAC-Loop Settings Batch Size 512 512 Policy Frequency 1 1 Target Entropy −|A|-|A| −|A|-|A| RDA2C twin-Q Actor Regression Settings Label Samples (K) 2 – Replay States per Actor Step (B) 512 – Label Window Size (W) 12,80012,800 – Advantage Clip 10.0 – Dual Prediction Clip (ϵclip _clip) ∞ (off) – Discrete Control (Atari). For Atari, we compare against PPO using the same total batch size per update, the same CleanRL preprocessing and EnvPool execution stack, and the same 10M-frame budget. The remaining rollout and optimization settings follow each method’s own defaults and are tabled side by side in Appendix A. Table 3 lists the hyperparameters used for the 10M frame benchmark. Note that while the number of environments and steps per environment differ, the total batch size is fixed at 10241024 for both algorithms. Table 3: Atari Hyperparameters. Final clipped RDA2C setting compared with PPO. Parameter RDA2C (Ours) PPO (Baseline) Shared Settings CNN Backbone Nature CNN, 512-unit output Nature CNN, 512-unit output Actor–Critic Trunks Independent policy and value trunks Shared trunk Optimizer Adam Adam Learning Rate 2.5×10−42.5× 10^-4 2.5×10−42.5× 10^-4 LR Schedule Linear Decay Linear Decay Discount Factor (γ) 0.99 0.99 Total Batch Size 1024 1024 Minibatch Size 256 256 VF Coefficient 0.5 0.5 Max Gradient Norm 2.0 0.5 Sampling Strategy Num. Environments 4 8 Steps per Env 256 128 Algorithm Specific Update Epochs 2 4 GAE Lambda (λGAE _GAE) 0.85 0.95 Window Size (W) 500,000 – Entropy Coeff (λ) 0.005 0.01 Stabilizer (β) 0.6 – Dual Prediction Clip (ϵclip _clip) 1.0 – PPO Policy Ratio Clip – 0.1 A.4 Optional clipped dual regression The main algorithm uses the squared dual regression loss in (4). In the final GAE experiments, we also use a clipped prediction variant to prevent abrupt changes in the fitted dual score under repeated replay updates. When a transition enters the buffer, we store the old prediction Zold(s,a)Z_old(s,a). For clip radius ϵclip>0 _clip>0, define Zθclip(s,a) Z_θ^clip(s,a) =Zold(s,a)+clip(Zθ(s,a)−Zold(s,a),−ϵclip,ϵclip), =Z_old(s,a)+clip (Z_θ(s,a)-Z_old(s,a),- _clip, _clip ), ℒclip(θ,t) _clip(θ,t) =12(s,a,A^,Zold)∼ℬt[max(Zθ(s,a)−A^)2,(Zθclip(s,a)−A^)2]. = 12\,E_(s,a, A,Z_old) _t [ \(Z_θ(s,a)- A)^2,(Z_θ^clip(s,a)- A)^2\ ]. This is a regression stabilizer analogous in spirit to PPO value function clipping (Schulman et al., 2017). It is not a policy ratio clip and does not change the mirror map readout in (3). We use ϵclip=0.2 _clip=0.2 for GAE MuJoCo and ϵclip=1.0 _clip=1.0 for the final GAE Atari runs. We set ϵclip=∞ _clip=∞ when this stabilizer is disabled. Appendix B Main-Text Numerical Result Tables This appendix collects the exact numerical tables corresponding to the bar-plot summaries in Figures 2 and 3. Evaluation conventions. The three comparisons use three return conventions, each the one used to evaluate the corresponding baseline. The PPO comparisons on MuJoCo and Atari report the online episodic return during training, the convention of the PPO implementation we match (Huang et al., 2022). The SAC comparison reports the best deterministic evaluation return within the 1M-step budget. Evaluating with the mean action is the SAC protocol (Haarnoja et al., 2018), and reporting the maximum evaluation return within a 1M-step budget is the convention of Fujimoto et al. (2018). The AAPDA comparison reports the mean of the final five deterministic evaluations, following the final-five-epoch average of Gao et al. (2026), with both methods evaluated on the same ten fixed evaluation seeds at checkpoints every 20,00020,000 environment steps, RDA2C’s rounded up to its next rollout boundary. The best-within-budget convention is optimistic for both methods it is applied to, and the conventions are not comparable across comparisons, so no number is compared across them. Aggregate bootstrap intervals. The main-text figures plot per-environment relative improvement. For each environment, the point estimate is computed from the seed-averaged returns, Δ%=100⋅(R¯ours−R¯base)/|R¯base| \%=100·( R_ours- R_base)/| R_base|. Percentages are computed from unrounded seed means, so recomputing them from the rounded table entries can differ in the last digit. To summarize uncertainty at the panel level, we report fixed suite paired bootstrap intervals: each bootstrap replicate resamples matched baseline/ours seed pairs within each environment, computes the environment level percent improvements using that environment’s fixed baseline mean for normalization, averages these improvements over the fixed benchmark suite, and takes the 2.5 and 97.5 percentiles across bootstrap replicates. MuJoCo GAE label comparison. Table 4 reports the running average episodic returns underlying the MuJoCo/PPO panel in Figure 2. RDA2C uses GAE labels and clipped dual regression in this setting, and the table shows both the mean return and the environment-level relative improvement over PPO. Table 4: MuJoCo Continuous Control Results. Mean episodic return (± std) over 10 seeds. PPO is matched to RDA2C’s architecture and optimization settings (Appendix A); the CleanRL-default comparison appears in Appendix B. RDA2C uses clipped dual regression with ϵclip=0.2 _clip=0.2 and value-loss clipping enabled. Last column reports relative change Δ%≜100⋅(RDA2C−PPO)/(|PPO|) \% 100·(RDA2C-PPO)/(|PPO|) (higher is better). Environment PPO (Baseline) RDA2C (Ours) Δ% \% Ant-v5 1953±3721953± 372 ±2831± 583 +45.0%+45.0\% HalfCheetah-v5 1864±781864± 78 ±7099± 645 +280.9%+280.9\% Hopper-v5 2333±5902333± 590 ±2607± 272 +11.7%+11.7\% Humanoid-v5 2370±9762370± 976 ±3768± 1290 +59.0%+59.0\% HumanoidStandup-v5 ±170412± 8967 156068±9614156068± 9614 −8.4%-8.4\% InvertedPendulum-v5 989±36989± 36 999.2±2.6999.2± 2.6 +1.1%+1.1\% Pusher-v5 −33.5±2.2-33.5± 2.2 −30.6±3.0-30.6± 3.0 +8.9%+8.9\% Walker2d-v5 ±3626± 604 3546±6593546± 659 −2.2%-2.2\% MuJoCo twin-Q critic comparison. Table 5 reports deterministic evaluation returns for the SAC-style twin-Q critic backbone. RDA2C and SAC share the replay and critic components, so the table isolates the effect of replacing the SAC actor improvement step with dual-score regression and entropy readout. Table 5: MuJoCo twin-Q critic results. Best deterministic evaluation return (± seed std) within the 1M-step budget over 10 seeds. Both methods use batch size 512512, policy update frequency 11, 5 parallel environments, and 20 evaluation episodes; the CleanRL-default SAC comparison appears in Appendix B. Last column reports relative percent change against SAC (higher is better). Environment SAC (Baseline) RDA2C with twin-Q labels (Ours) Δ% \% Ant-v5 4389.60±1100.194389.60± 1100.19 4240.94±877.824240.94± 877.82 −3.4%-3.4\% HalfCheetah-v5 10118.79±894.3210118.79± 894.32 9524.52±1388.409524.52± 1388.40 −5.9%-5.9\% Hopper-v5 3345.29±204.843345.29± 204.84 3408.27±121.763408.27± 121.76 +1.9%+1.9\% Humanoid-v5 5376.44±175.725376.44± 175.72 5195.96±314.715195.96± 314.71 −3.4%-3.4\% HumanoidStandup-v5 158309.50±4088.25158309.50± 4088.25 158784.44±6868.90158784.44± 6868.90 +0.3%+0.3\% InvertedPendulum-v5 1000.00±0.001000.00± 0.00 1000.00±0.001000.00± 0.00 0.0%0.0\% Pusher-v5 −29.58±1.55-29.58± 1.55 −28.05±1.51-28.05± 1.51 +5.2%+5.2\% Walker2d-v5 4079.87±371.704079.87± 371.70 4504.91±788.334504.91± 788.33 +10.4%+10.4\% AAPDA comparison. Table 6 reports the comparison with AAPDA described in Section 6: released defaults, 1M steps, ten paired training seeds, the same ten fixed evaluation seeds at every checkpoint, and the mean of the final five aligned deterministic checkpoints. Table 6: MuJoCo results against AAPDA. Mean of the final five aligned deterministic checkpoints (± seed std) over 10 paired seeds. Last column reports relative change Δ%≜100⋅(RDA2C−AAPDA)/(|AAPDA|) \% 100·(RDA2C-AAPDA)/(|AAPDA|). Environment AAPDA RDA2C (Ours) Δ% \% Ant-v5 2562±3742562± 374 ±4043± 691 +57.8%+57.8\% HalfCheetah-v5 4948±7054948± 705 ±7808± 629 +57.8%+57.8\% Hopper-v5 2594±7942594± 794 ±3218± 196 +24.0%+24.0\% Humanoid-v5 1209±13351209± 1335 ±3653± 1459 +202.1%+202.1\% HumanoidStandup-v5 ±162208± 2025 156485±9003156485± 9003 −3.5%-3.5\% InvertedPendulum-v5 980.7±49.6980.7± 49.6 1000.0±0.01000.0± 0.0 +2.0%+2.0\% Pusher-v5 −28.2±1.1-28.2± 1.1 −27.1±1.7-27.1± 1.7 +3.6%+3.6\% Walker2d-v5 ±4424± 251 4001±5584001± 558 −9.6%-9.6\% Baseline default configurations. The main-text comparisons use baselines matched to RDA2C’s shared settings. For reference, this section reports both baselines at their CleanRL default configurations. For PPO the defaults are 2×642× 64 trunks, learning rate 3×10−43× 10^-4, gradient clip 0.50.5, and GAE λ=0.85λ=0.85. For SAC they are batch size 256256 and policy update frequency 22. Matching strengthens the baselines: the matched configuration improves PPO over its defaults on five of eight tasks, and the batch-512512, frequency-11 SAC matches or improves the default SAC on seven of eight tasks. Table 7: MuJoCo results against CleanRL-default PPO. Mean episodic return (± std) over 10 seeds. Same RDA2C runs as Table 4. Environment PPO (default) RDA2C (Ours) Δ% \% Ant-v5 1855±5191855± 519 ±2831± 583 +52.6%+52.6\% HalfCheetah-v5 1884±781884± 78 ±7099± 645 +276.8%+276.8\% Hopper-v5 2588±8312588± 831 ±2607± 272 +0.7%+0.7\% Humanoid-v5 1100±3881100± 388 ±3768± 1290 +242.5%+242.5\% HumanoidStandup-v5 ±158572± 12341 156068±9614156068± 9614 −1.6%-1.6\% InvertedPendulum-v5 1000.0±0.01000.0± 0.0 999.2±2.6999.2± 2.6 −0.1%-0.1\% Pusher-v5 −36.2±2.0-36.2± 2.0 −30.6±3.0-30.6± 3.0 +15.6%+15.6\% Walker2d-v5 3450±4753450± 475 ±3546± 659 +2.8%+2.8\% Table 8: MuJoCo twin-Q results against CleanRL-default SAC (batch 256256, policy frequency 22). Best deterministic evaluation return (± seed std) within the 1M-step budget over 10 seeds. Same RDA2C runs as Table 5. Environment SAC (default) RDA2C with twin-Q labels (Ours) Δ% \% Ant-v5 4048.10±859.124048.10± 859.12 4240.94±877.824240.94± 877.82 +4.8%+4.8\% HalfCheetah-v5 9844.07±914.739844.07± 914.73 9524.52±1388.409524.52± 1388.40 −3.2%-3.2\% Hopper-v5 3348.58±157.323348.58± 157.32 3408.27±121.763408.27± 121.76 +1.8%+1.8\% Humanoid-v5 5228.44±241.405228.44± 241.40 5195.96±314.715195.96± 314.71 −0.6%-0.6\% HumanoidStandup-v5 156360.89±7197.51156360.89± 7197.51 158784.44±6868.90158784.44± 6868.90 +1.5%+1.5\% InvertedPendulum-v5 1000.00±0.001000.00± 0.00 1000.00±0.001000.00± 0.00 0.0%0.0\% Pusher-v5 −29.80±1.55-29.80± 1.55 −28.05±1.51-28.05± 1.51 +5.9%+5.9\% Walker2d-v5 3871.58±736.893871.58± 736.89 4504.91±788.334504.91± 788.33 +16.4%+16.4\% Atari GAE label comparison. Table 9 reports the twelve Atari games underlying Figure 3. Table 9: Atari results. Mean running return (± seed std) over 10 seeds on twelve Atari games with clipped dual regression, ϵclip=1.0 _clip=1.0, β=0.6β=0.6, and λ=0.005λ=0.005. Last column reports relative change Δ%≜100⋅(RDA2C−PPO)/(|PPO|) \% 100·(RDA2C-PPO)/(|PPO|) when PPO ≠0≠ 0. Environment PPO (Baseline) RDA2C (Ours) Δ% \% Asterix 4061±10144061± 1014 ±6024± 1336 +48.3%+48.3\% BeamRider 2540±3162540± 316 ±9142± 786 +259.9%+259.9\% Breakout ±422± 24 408±29408± 29 −3.5%-3.5\% Enduro 980±135980± 135 ±5348± 1720 +445.6%+445.6\% Freeway 32.7±0.632.7± 0.6 23.6±16.323.6± 16.3 −27.8%-27.8\% Gravitar 686±187686± 187 ±835± 376 +21.8%+21.8\% MsPacman 2085±2682085± 268 ±3243± 936 +55.6%+55.6\% Pong 20.6±0.720.6± 0.7 20.5±1.420.5± 1.4 −0.6%-0.6\% PrivateEye 70.1±47.170.1± 47.1 89.8±31.689.8± 31.6 +28.2%+28.2\% Qbert ±15804± 1993 14562±128314562± 1283 −7.9%-7.9\% Seaquest 1541±4271541± 427 ±9114± 7642 +491.4%+491.4\% SpaceInvaders 998±187998± 187 ±1373± 220 +37.6%+37.6\% Appendix C Additional Experimental Results C.1 MuJoCo GAE label learning curves Figure 4 displays the 10-seed training dynamics for the PPO versus clipped-dual-regression RDA2C comparison across the 8 continuous control environments. The curves highlight large gains on HalfCheetah and Humanoid, a clear improvement on Ant, and PPO-level behavior on the remaining tasks, with HumanoidStandup remaining slightly below PPO in the final aggregate table. Figure 4: PPO versus RDA2C MuJoCo learning curves. Blue denotes RDA2C and red denotes PPO. Shaded regions show standard deviation over 10 seeds. The clipped-dual-regression configuration yields large gains on HalfCheetah, Humanoid, and Ant, while remaining close to PPO on the other environments. C.2 MuJoCo twin-Q critic label learning curves Figure 5 presents the 10-seed online training-return curves for the SAC versus RDA2C with twin-Q labels comparison. These curves correspond to the logged running reward summaries in Table 10, not to the deterministic evaluation returns in Table 5. Across environments, the off-policy RDA actor update with soft twin-Q RLOO labels is generally stronger during data collection: RDA2C with twin-Q labels has higher final running reward on seven of eight tasks and lower seed standard deviation on six of eight tasks. The largest online return gaps appear on Hopper-v5 and Walker2d-v5. HalfCheetah-v5 remains the main environment where SAC is clearly stronger. Table 10: MuJoCo online training return with twin-Q critic labels. Final logged running reward (± seed std) over 10 seeds at 1,000,0001,000,000 environment steps. This is an online data collection metric, distinct from the deterministic evaluation return in Table 5. Last column reports relative percent change against SAC (higher is better). Environment SAC (Baseline) RDA2C with twin-Q labels (Ours) Δ% \% Ant-v5 3414.91±873.593414.91± 873.59 3683.40±842.943683.40± 842.94 +7.9%+7.9\% HalfCheetah-v5 9182.51±888.079182.51± 888.07 8538.56±1167.478538.56± 1167.47 −7.0%-7.0\% Hopper-v5 2289.03±637.752289.03± 637.75 2948.40±357.632948.40± 357.63 +28.8%+28.8\% Humanoid-v5 4771.82±320.914771.82± 320.91 4979.12±215.644979.12± 215.64 +4.3%+4.3\% HumanoidStandup-v5 152428.66±7472.47152428.66± 7472.47 155100.32±5994.39155100.32± 5994.39 +1.8%+1.8\% InvertedPendulum-v5 935.64±72.85935.64± 72.85 969.61±33.70969.61± 33.70 +3.6%+3.6\% Pusher-v5 −30.01±1.24-30.01± 1.24 −28.87±1.50-28.87± 1.50 +3.8%+3.8\% Walker2d-v5 3124.79±1354.773124.79± 1354.77 4067.66±1117.914067.66± 1117.91 +30.2%+30.2\% Figure 5: MuJoCo online training curves with twin-Q critic labels. Blue denotes RDA2C twin-Q and red denotes SAC. Shaded regions show standard deviation over 10 seeds. The curves report the online running reward metric logged during data collection. RDA2C twin-Q achieves higher final running reward on Ant, Hopper, Humanoid, HumanoidStandup, InvertedPendulum, Pusher, and Walker2d, while SAC remains stronger on HalfCheetah. C.3 Atari Learning Curves Figure 6 presents the learning dynamics for the twelve Atari games used in the main comparison. Curves represent the average episodic return over 10 random seeds with standard deviation shading. Analysis. The curves give a more detailed view of the summary in the main text. RDA2C improves substantially over PPO on Seaquest, BeamRider, Enduro, and Asterix. In these runs, the replayed dual averaging update continues to improve after the PPO curve has largely flattened. The remaining games include threshold and saturated regimes. Pong is near saturated for both methods, while Freeway remains a high variance threshold game where PPO is higher in the final 10-seed aggregate. RDA2C improves PrivateEye, but it does not add a directed exploration mechanism. Figure 6: Full Training Dynamics (10M Frames). Comparison of RDA2C and PPO across the twelve Atari games used in the main comparison. Appendix D Ablation Studies This section reports the ablations: the replay window (Section D.1), the RDA stabilizer (Section D.2), the entropy floor (Section D.3), the advantage-label loss (Section D.4), the SAC replay buffer (Section D.5), and a PPO capacity control (Section D.6). D.1 Replay Window Size We evaluate sensitivity to the replay / label window size W in both the GAE Atari setting and the twin-Q critic label MuJoCo setting. The two sweeps differ because the Atari implementation stores replayed transitions, whereas the twin-Q critic implementation stores labeled (s,a,g)(s,a,g) tuples in a dedicated FIFO actor regression buffer. This sweep should be interpreted qualitatively. The theoretical discussion in Section 5 predicts a variance–staleness tradeoff and, in a simplified asymptotic bound, balances the replay error terms with Wt≍t2/3W_t t^2/3. Our finite budget sweeps do not estimate this exponent: they vary fixed window sizes and report achieved return, and they do not measure score estimation error directly. The empirical question is therefore whether small windows are too noisy and whether very long memories can become stale. The replay window controls how much past dual evidence defines the actor objective. Small windows use fresher labels but provide fewer samples for fitting the dual score, while large windows reduce label variance but mix in older policy-improvement signals. The Atari sweep in Appendix D.1 shows this finite-budget tradeoff, with intermediate windows often strongest. The best window varies across games and is not monotone in W. This behavior qualitatively matches the finite replay and local drift terms in Theorem 5.1. The twin-Q MuJoCo sweep (Appendix D.1.2) shows the same nonmonotone pattern, and the one-round current-label setting is never the best window. A current-label control for the GAE setting (Appendix D.1.3) isolates retention alone: the historical target is higher on seven of eight tasks. D.1.1 Atari We evaluate the sensitivity of RDA2C to the Atari replay window size W. Figure 7 shows the learning curves, and Table 11 provides the final asymptotic performance. The window sweep uses the plain squared dual regression loss, whereas the main Atari comparison in Table 9 uses the clipped loss of Appendix A.4, so the two tables are not comparable at W=500,000W=500,000. Only the comparison across windows within the sweep is meaningful. Table 11: Atari Window Size Sweep Results. Mean episodic return at 10M frames, plain squared loss. Bold indicates the best performing configuration. Environment W=300,000W=300,000 W=500,000W=500,000 W=700,000W=700,000 W=1,000,000W=1,000,000 Asterix ±5448± 1063 5305±2045305± 204 4758±9294758± 929 4122±5054122± 505 BeamRider 9768±4299768± 429 ±10509± 874 9823±3179823± 317 9831±8329831± 832 Breakout ±369± 17 353±6353± 6 319±22319± 22 314±31314± 31 Enduro ±3474± 372 3374±2733374± 273 3077±5203077± 520 3076±3403076± 340 Freeway 33.8±0.033.8± 0.0 33.8±0.033.8± 0.0 33.8±0.033.8± 0.0 33.9±0.033.9± 0.0 Gravitar 690±93690± 93 ±830± 169 675±40675± 40 670±65670± 65 MsPacman 2788±5762788± 576 ±3974± 669 3056±8023056± 802 3245±4283245± 428 Pong 20.9±0.020.9± 0.0 21.0±0.021.0± 0.0 21.0±0.021.0± 0.0 20.9±0.120.9± 0.1 PrivateEye 66.6±47.166.6± 47.1 66.6±47.166.6± 47.1 100.0±0.0100.0± 0.0 99.9±0.099.9± 0.0 Qbert 15341±39215341± 392 15690±30915690± 309 14310±115014310± 1150 ±15892± 929 Seaquest 5484±18195484± 1819 ±14158± 5832 13335±491113335± 4911 13843±422013843± 4220 SpaceInvaders 1359±441359± 44 ±1457± 237 1361±1881361± 188 1405±1721405± 172 Figure 7: Effect of Window Size (W) on Atari. Training curves for W∈300k,500k,700k,1MW∈\300k,500k,700k,1M\ on the twelve Atari games used in the main comparison. W=500kW=500k offers the most consistent performance across tasks. D.1.2 MuJoCo with twin-Q Critics We additionally sweep the RDA label buffer size on five representative MuJoCo tasks: Ant-v5, HalfCheetah-v5, Hopper-v5, Humanoid-v5, and Walker2d-v5. We summarize each setting by the best deterministic evaluation mean return achieved during training, averaged over 10 seeds. In this implementation, each actor trigger samples B=512B=512 replay states and generates K=2K=2 labels per state, so it inserts KB=1024KB=1024 fresh (s,a,g)(s,a,g) tuples into the FIFO label buffer. Thus W=1024W=1024 acts as a one round current-label control. Table 12 shows a nonmonotone mean return pattern: the current-label control is not the top mean setting on any of the five tasks, larger windows substantially improve HalfCheetah-v5 and moderately improve Walker2d-v5, and Ant-v5, Hopper-v5, and Humanoid-v5 are comparatively flat once W>1024W>1024. This is consistent with the qualitative bias–variance/staleness tradeoff from Section 5: averaging recent actor-improvement labels can reduce noise, but overly long memories can become stale. Table 12: MuJoCo window size sweep with twin-Q critic labels. Mean best deterministic evaluation return (± seed std) over 10 seeds. Here W is the FIFO label buffer size in stored (s,a,g)(s,a,g) tuples. With batch size 512512 and K=2K=2 label samples, W=1024W=1024 retains approximately one actor trigger’s fresh labels and therefore acts as a current label control. Bold indicates the best window per environment. Environment W=1024W=1024 W=2048W=2048 W=4096W=4096 W=8192W=8192 W=12,800W=12,800 W=25,600W=25,600 Ant-v5 4755±9854755± 985 4008±9534008± 953 4314±9094314± 909 ±4798± 747 4241±8784241± 878 4672±9464672± 946 HalfCheetah-v5 7068±36537068± 3653 8800±25658800± 2565 9387±15429387± 1542 8081±34108081± 3410 9525±13889525± 1388 ±9819± 1029 Hopper-v5 3404±1953404± 195 3330±1113330± 111 3365±2073365± 207 3351±1333351± 133 3408±1223408± 122 ±3430± 150 Humanoid-v5 5280±1105280± 110 5237±1595237± 159 5279±2005279± 200 ±5294± 127 5196±3155196± 315 5218±1075218± 107 Walker2d-v5 4444±4594444± 459 3810±12913810± 1291 ±4724± 522 3566±13023566± 1302 4505±7884505± 788 4179±9774179± 977 D.1.3 Current-label control (GAE MuJoCo) This control changes label retention and nothing else. The buffer size W=2048W=2048 retains exactly one rollout of labels, so every regression target comes from the current policy. The main-results setting W=50,000W=50,000 retains about 24 rollouts. Architecture, optimizer state, and training budget are fixed. Table 13 shows the result over ten paired seeds: the historical target is higher on seven of eight tasks. The gains are large where they appear: Humanoid-v5 goes from 500500 to 37683768 and Ant-v5 from 884884 to 28312831. This is the most direct evidence that the improvement comes from aggregating labels across several recent policies. Score regression on current labels alone does not produce it. Table 13: Current-label control (GAE labels, MuJoCo). Mean running episodic return (± seed std) over 10 paired seeds. Last column reports relative change Δ%≜100⋅(history−current)/(|current|) \% 100·(history-current)/(|current|) (higher is better). Environment current (W=2048W=2048) history (W=50,000W=50,000) Δ% \% Ant-v5 884±412884± 412 ±2831± 583 +220.1%+220.1\% HalfCheetah-v5 5162±12985162± 1298 ±7099± 645 +37.5%+37.5\% Hopper-v5 2112±8022112± 802 ±2607± 272 +23.4%+23.4\% Humanoid-v5 500±75500± 75 ±3768± 1290 +654.0%+654.0\% HumanoidStandup-v5 94062±2466994062± 24669 ±156068± 9614 +65.9%+65.9\% InvertedPendulum-v5 1000.0±0.01000.0± 0.0 999.2±2.6999.2± 2.6 −0.1%-0.1\% Pusher-v5 −31.9±1.3-31.9± 1.3 −30.6±3.0-30.6± 3.0 +4.3%+4.3\% Walker2d-v5 2973±7072973± 707 ±3546± 659 +19.3%+19.3\% D.2 RDA Stabilizer Figure 8 varies the RDA stabilizer β while holding the entropy floor and replay window fixed. Larger values damp early policy changes more strongly. The aggregate results show that some transient stabilization is useful, with moderate values performing best, while the per-task heatmaps show that the best β is environment dependent. Very large β can make the early readout too conservative, whereas β=0β=0 removes this transient damping entirely. Figure 8: Sensitivity to the RDA stabilizer β. Returns are normalized independently within each environment across the swept β values before aggregation. Error bars in the aggregate panels denote task SEM. For the reason in footnote 2, they do not denote confidence intervals. They illustrate the scale of variation across tasks. D.3 Entropy Floor Sensitivity We sweep the persistent entropy floor λ in the RDA temperature αt=λ+β/t _t=λ+β/t while holding the replay window and β fixed. Figure 9 shows a broad small λ plateau on Atari and a moderate λ plateau on MuJoCo. Very large entropy floors keep the readout too diffuse and hurt return. Figure 9: Sensitivity to the entropy floor λ. Returns are min-max normalized independently within each environment across the swept λ values before aggregation. Error bars in the aggregate panels denote task SEM. For the reason in footnote 2, they do not denote confidence intervals. They illustrate the scale of variation across tasks. D.4 Advantage-Label Loss We compare four dual regression losses for GAE advantage labels: plain MSE, Huber regression, normalized target clipping, and PPO style prediction clipping. Figure 10 reports Atari and MuJoCo separately. Prediction clipping is the strongest default, especially on MuJoCo. Target clipping is competitive on Atari but is less reliable in continuous control. Huber regression alone does not recover the performance of prediction clipping. Figure 10: Advantage label loss ablation. Atari results are shown above and MuJoCo results below. Returns are min-max normalized within each environment across loss types before aggregation. Error bars in the aggregate panels denote task SEM. For the reason in footnote 2, they do not denote confidence intervals. They illustrate the scale of variation across tasks. Prediction clipping gives the best aggregate, while Huber alone is weaker. D.5 SAC Replay Buffer Size Sensitivity to the shared transition replay buffer. We examine sensitivity to the capacity of the ordinary transition replay buffer used by the shared SAC/twin-Q backbone. In the main comparison, we retain the 1M capacity from the default SAC configuration used in our implementation and apply the same capacity to both SAC and RDA2C. This setting preserves comparability with the standard baseline. It was not selected using the present sweep. For this diagnostic ablation, we vary the shared transition-buffer capacity over 50k, 100k, 250k, 500k, and 1M, while holding RDA2C’s separate advantage-label buffer fixed at 12,800. We normalize jointly across both methods and all tested capacities within each environment, so the aggregate summarizes sensitivity to the shared replay capacity and the method gap on a common scale. Figure 11 shows that aggregate normalized performance peaks near 100k for both methods and decreases for larger capacities. The similar dependence across SAC and RDA2C suggests that this effect comes primarily from recency in the shared transition-replay backbone and not from the RDA-specific label buffer, which is held fixed. We retain 1M in the main comparison to report results under the default SAC configuration. Figure 11: Sensitivity to standard SAC replay buffer size. We vary the ordinary SAC replay buffer and keep the RDA label window fixed at 12,80012,800. Returns are min-max normalized independently within each environment across all method and buffer configurations before aggregation. Error bars denote task SEM. InvertedPendulum is omitted from the normalized aggregate because it is saturated for every setting. D.6 PPO Network Capacity Control (MuJoCo, GAE) The matched comparison in Table 4 sets PPO to RDA2C’s width and learning rate. This ablation asks the complementary question: does capacity alone close the gap when PPO keeps its own defaults? Starting from the CleanRL default 2×642× 64 trunk (Appendix B), we retrain PPO with a 2×1282× 128 trunk, keeping the global log-standard-deviation and the remaining default hyperparameters. We report both the default learning rate 3×10−43× 10^-4 and the RDA2C rate 2×10−42× 10^-4. Table 14 and Figure 12 show that enlarging PPO does not close the gap: at 3×10−43× 10^-4 the wider PPO is often worse than the 2×642× 64 baseline (especially on Ant-v5 and Hopper-v5), and lowering the learning rate recovers some of that loss but still leaves RDA2C ahead on the tasks with the largest gains. Table 14: PPO network capacity control (MuJoCo, GAE labels). Mean running episodic return (± seed std) over 10 seeds. All PPO variants use global log-standard-deviation. Bold marks the best mean in each row. Environment PPO 2×642× 64 PPO 2×1282× 128 PPO 2×1282× 128 RDA2C 2×1282× 128 lr 3×10−43× 10^-4 lr 3×10−43× 10^-4 lr 2×10−42× 10^-4 lr 2×10−42× 10^-4 Ant-v5 1855±5191855± 519 1260±4411260± 441 1537±7571537± 757 ±2831± 583 HalfCheetah-v5 1884±781884± 78 1940±691940± 69 2190±10592190± 1059 ±7099± 645 Hopper-v5 2588±8312588± 831 2154±7192154± 719 2363±4642363± 464 ±2607± 272 Humanoid-v5 1100±3881100± 388 1187±5291187± 529 1573±9011573± 901 ±3768± 1290 HumanoidStandup-v5 158572±12341158572± 12341 ±161394± 6265 159009±8458159009± 8458 156068±9614156068± 9614 InvertedPendulum-v5 1000.0±0.01000.0± 0.0 952.6±149.9952.6± 149.9 997.3±8.4997.3± 8.4 999.2±2.6999.2± 2.6 Pusher-v5 −36.2±2.0-36.2± 2.0 −36.2±2.1-36.2± 2.1 −36.5±2.4-36.5± 2.4 −30.6±3.0-30.6± 3.0 Walker2d-v5 3450±4753450± 475 3262±9513262± 951 ±3812± 675 3546±6593546± 659 Figure 12: PPO network capacity control on MuJoCo (GAE labels). Returns are min–max normalized within each environment across the four configurations before aggregation. Error bars in the aggregate panel denote task SEM. Matching PPO’s width to RDA2C does not reproduce the RDA2C gains. Appendix E RDA2C Gradient Boosting with XGBoost E.1 Overview To complement our neural network instantiation of the dual predictor ZθZ_θ, we performed a small study in which ZθZ_θ is instantiated as a gradient boosted decision tree regressor (XGBoost). This setting makes the connection between dual averaging and additive function approximation particularly explicit, and it allows us to compare two concrete approximations of the running dual average: xgb_boosting (incremental boosting of the cumulative dual) and xgb_dagger (DAgger style dataset aggregation with a FIFO buffer). Supervised representation. In both variants, we train a regressor on features ϕ(s,a)=[s,onehot(a)]φ(s,a)=[s,onehot(a)] and targets given by an advantage label derived from a learned value function. The resulting predictor Z^(s,a) Z(s,a) is converted into a policy by the closed-form mirror map: π(a∣s)∝exp(Z^(s,a)α),π(a s) \! ( Z(s,a)α ), using the same entropy regularized Boltzmann form as in the main algorithm. In all runs reported here, we apply clipped importance sampling (IPS) by reweighting advantage targets by 1/πbeh(a∣s)1/ _beh(a s) with a maximum weight (iw_max). See the configuration tables below. Incremental boosting (xgb_boosting). We maintain a single XGBoost booster whose prediction represents an unnormalized cumulative dual U(s,a)≈∑k=1Kgk(s,a)U(s,a)≈ _k=1^Kg_k(s,a), where gkg_k is the (normalized and optionally IPS weighted) advantage signal from update k. Operationally, after each minibatch we add one tree (warm start) using a dual target y=Uprev+gy=U_prev+g so that the new tree fits the residual advantage. At action selection time, logits are formed as U(s,a)/(Kλ+β)U(s,a)/(Kλ+β), where K is the current number of boosted rounds. This matches the scaling implied by the ratio Zt/αtZ_t/ _t in (3) when identifying K with the number of dual updates. Dataset aggregation (xgb_dagger). We maintain a FIFO buffer of advantage labeled tuples (s,a,A^)(s,a, A) collected from all past rollouts (up to a fixed capacity), and periodically continue boosting on the full buffer. This corresponds to fitting a single predictor to the pooled dataset, which approximates the average dual ZtZ_t under the mixture of visited state–action distributions. The policy uses the standard temperature schedule αt=λ+β/t _t=λ+β/t and applies the same softmax mapping. E.2 Hyperparameters Experimental protocol. We ran both variants for 10610^6 environment steps on CartPole-v1 and Acrobot-v1, using 8 parallel environments with rollout length 256 (batch size 2048). We report mean ± standard deviation of the final episodic return over 5 random seeds. Hyperparameters. Tables 15 and 16 list the configuration used for each environment. Table 15: XGBoost dataset aggregation (xgb_dagger) configurations. Parameter CartPole-v1 Acrobot-v1 Total timesteps 1,000,0001,000,000 1,000,0001,000,000 Num envs / steps 88 / 256256 88 / 256256 Batch size 20482048 20482048 γ / GAE λGAE _GAE 0.990.99 / 0.80.8 0.990.99 / 0.80.8 Value LR / epochs 2.5×10−32.5×10^-3 / 44 2.5×10−32.5×10^-3 / 44 Anneal LR True False Max grad norm 0.50.5 0.50.5 FIFO capacity 100,000100,000 200,000200,000 Fit every (iters) 44 22 Advantage clip 1010 2525 IPS / iw_max True / 1010 True / 55 αt _t schedule λ=0.15,β=2λ=0.15,\,β=2 λ=0.2,β=2λ=0.2,\,β=2 Trees per fit 11 22 Tree LR (eta) 0.050.05 0.050.05 Max depth 33 77 Min child weight 33 33 Subsample / colsample 1.01.0 / 1.01.0 1.01.0 / 1.01.0 Table 16: XGBoost incremental boosting (xgb_boosting) configurations. Parameter CartPole-v1 Acrobot-v1 Total timesteps 1,000,0001,000,000 1,000,0001,000,000 Num envs / steps 88 / 256256 88 / 256256 Batch size 20482048 20482048 γ / GAE λGAE _GAE 0.990.99 / 0.80.8 0.990.99 / 0.80.8 Value LR / epochs 3×10−33×10^-3 / 11 10−210^-2 / 11 Anneal LR True True Max grad norm 1010 1010 Normalize adv True True Advantage clip 55 55 IPS / iw_max True / 1010 True / 1010 αt _t schedule λ=10−4,β=1λ=10^-4,\,β=1 λ=10−4,β=1λ=10^-4,\,β=1 Tree LR (eta) 0.50.5 0.450.45 Max depth 33 33 Min child weight 55 55 Subsample / colsample 1.01.0 / 1.01.0 1.01.0 / 1.01.0 E.3 Results Table 17: RDA2C gradient boosting results. Final episodic return (mean ± std over 5 seeds). Environment Method Final return CartPole-v1 xgb_dagger 500.00±0.00500.00± 0.00 CartPole-v1 xgb_boosting 500.00±0.00500.00± 0.00 Acrobot-v1 xgb_dagger −84.15±4.57-84.15± 4.57 Acrobot-v1 xgb_boosting −88.98±2.43-88.98± 2.43 Figure 13: RDA2C gradient boosting learning curves. On CartPole-v1 (left), both xgb_dagger and xgb_boosting reach 500500 within 10610^6 steps. On Acrobot-v1 (right), xgb_dagger is slightly more sample efficient early, and both methods converge similarly by 10610^6 steps. Appendix F Proofs and implications for policy performance The proof below analyzes the learned replay actor through a direct regularized score to value argument. The three layers are: 1. Replay window Bayes target and local drift. The replay regression Bayes target ZtAZ_t^A is a convex combination of recent advantages at each state action pair, so it tracks the current advantage AtA_t up to a local window drift term. 2. Direct regularized learned readout bound. Since the learned policy πθt _ _t is the exact entropy regularized maximizer of the learned score Zθt−1Z_ _t-1, we prove regularized approximate optimality for πθt _ _t. The residual Zθt−1−ZtAZ_ _t-1-Z_t^A against the regression Bayes target ZtAZ_t^A enters through a direct inner product error under the comparator occupancy. 3. Replay decomposition. The learned score error ‖Zθt−1−ZtA‖2,μt\|Z_ _t-1-Z_t^A\|_2, _t is decomposed into supervised fitting error, finite replay concentration, and critic label bias. Estimation chain. The estimation chain summarizes the construction, and each arrow is one estimation step: Zt→replay weightingZ^t→regressionZθ→mirror mapπθt+1.Z_t\; replay weighting\; Z_t\; regression\;Z_θ\; mirror map\; _ _t+1. The exact score ZtZ_t is the state-weighted average advantage. Replay sampling replaces it with the Bayes target Z^t Z_t, which carries the behavior action weights of the stored tuples. Fitting gives ZθZ_θ, and the entropy mirror map turns ZθZ_θ into the next policy. Section 5 bounds each gap in this chain. For states outside the support of d¯t d_t, ZtZ_t can be assigned any fixed value, since those states have zero weight in the objective in (1) until later data cover them. F.1 Performance difference lemma With normalized discounted occupancies, the following lemma converts policy space advantage feedback into value differences. Lemma F.1 (Performance difference lemma (Kakade and Langford, 2002)). For any stationary comparator π⋆π and any policy πθt _ _t, with At=AπθtA_t=A _ _t, J(π⋆)−J(πθt)=11−γs∼dπ⋆[⟨π⋆(⋅∣s)−πθt(⋅∣s),At(s,⋅)⟩].J(π )-J( _ _t)= 11-γE_s d^π [ π (· s)- _ _t(· s),A_t(s,·) ]. (6) F.2 Window notation and replay Bayes target Fix an exponent ζ∈(0,1)ζ∈(0,1). For every t≥2t≥ 2, let Wt=max1,⌊tζ⌋W_t= \1, t^ζ \ be the number of policy iterations represented in the replay window, and define t=t−Wt,…,t−1.W_t=\t-W_t,…,t-1\. Thus |t|=Wt|W_t|=W_t and Wt≤tζW_t≤ t^ζ. Replacing ⌊tζ⌋ t^ζ by ⌈tζ⌉ t^ζ only changes constants and has no effect on the rate. If the implementation measures replay capacity in samples rather than iterations, a fixed batch size converts sample capacity into a constant multiple of WtW_t, and that constant is absorbed into CvarC_var below. Throughout this section, J(π)=Jρ(π):=Vπ(ρ)=s0∼ρ[Vπ(s0)]J(π)=J_ρ(π):=V^π(ρ)=E_s_0 ρ[V^π(s_0)] denotes the scalar start distribution value from the fixed initial distribution ρ. We also assume that rewards are uniformly bounded by a global constant Rmax<∞R_ <∞: |r(s,a)|≤Rmaxfor all (s,a).|r(s,a)|≤ R_ all (s,a). We write GAG_A for a uniform advantage bound. Under this two sided reward bound one may take GA=2Rmax/(1−γ)G_A=2R_ /(1-γ), although the analysis below only uses ‖At‖∞≤GA\|A_t\|_∞≤ G_A. Write Ak(s,a)=Aπθk(s,a)A_k(s,a)=A _ _k(s,a) for the advantage of the learned policy at round k. When we need the population replay distribution explicitly, we reuse the occupancy notation from Section 1: μk(s,a)=μπθk(s,a)=dπθk(s)πθk(a∣s) _k(s,a)=μ _ _k(s,a)=d _ _k(s) _ _k(a s) for the state action occupancy of the data generated at round k, and define the window replay marginal μt(s,a)=1Wt∑k∈tμk(s,a). _t(s,a)= 1W_t _k _t _k(s,a). Thus μt _t is the population state action distribution represented by the current replay window. Ratios involving μt(s,a) _t(s,a) are always understood on the support where μt(s,a)>0 _t(s,a)>0. The score learned by the squared loss replay regression has, with true advantage labels, the population conditional mean target ZtA(s,a)=∑k∈tμk(s,a)Ak(s,a)∑k∈tμk(s,a),∑k∈tμk(s,a)>0,1Wt∑k∈tAk(s,a),∑k∈tμk(s,a)=0.Z_t^A(s,a)= cases _k _t _k(s,a)A_k(s,a) _k _t _k(s,a),& _k _t _k(s,a)>0,\\[9.68747pt] 1W_t _k _tA_k(s,a),& _k _t _k(s,a)=0. cases On the support of μt _t this is the replay regression Bayes target. The second line is a harmless tie breaker on state action pairs with zero window replay mass. It is included only so that the pointwise supremum ‖At−ZtA‖∞\|A_t-Z_t^A\|_∞ used in Layer 1 is defined on every (s,a)(s,a), and it does not affect the L2(μt)L_2( _t) score error term, which never sees the off support branch. At every (s,a)(s,a), ZtA(s,a)Z_t^A(s,a) is therefore a convex combination of the recent advantages Ak(s,a):k∈t\A_k(s,a):k _t\, with coefficients proportional to the joint occupancy μk(s,a) _k(s,a) on the support and uniform off it. This convex combination property is all that the local drift bound uses. The algorithm uses a fixed stabilizer β≥0β≥ 0. The temperature of the readout that produces the round-t policy is that of (3), αt−1=λ+βt−1,λ>0. _t-1=λ+ βt-1, λ>0. For t≥2t≥ 2, define the learned readout by πθt(⋅∣s)=softmax(Zθt−1(s,⋅)αt−1). _ _t(· s)=softmax\! ( Z_ _t-1(s,·) _t-1 ). Here Zθt−1Z_ _t-1 denotes the score learned using data available before round t. The proof is stated for the average over t=2,…,Tt=2,…,T, so every policy in the average is read out from a nonempty past window. F.3 Layer 1: replay window Bayes target and local drift The role of Layer 1 is now only to connect the window target appearing in the score residual to the current advantage in the performance difference lemma. The main bridge is the regression Bayes target ZtAZ_t^A. Since ZtA(s,a)Z_t^A(s,a) is a convex combination of recent advantages at each fixed (s,a)(s,a), the local policy path stability assumption implies ‖At−ZtA‖∞=O(tζ−1)\|A_t-Z_t^A\|_∞=O(t^ζ-1). Assumption F.2 (Learned policy path stability). There exists Cπ>0C_π>0 such that the learned policies satisfy sups‖πθk+1(⋅∣s)−πθk(⋅∣s)‖1≤Cπk _s \| _ _k+1(· s)- _ _k(· s) \|_1≤ C_πk for every k≥1k≥ 1. This is the algorithm side stability condition needed because the replay buffer is generated by the learned policies πθk _ _k, not by an auxiliary exact windowed readout. For learned policies, this need not follow from average fitting error alone: consecutive learned policies can move by a constant amount while maintaining small average replay error. We therefore state this assumption explicitly. It is natural for warm started actor updates with 1/k1/k-scale step sizes, trust region constraints, clipping, or projection onto a path stable policy class. Lemma F.3 (Value and advantage Lipschitzness in policy). For any two stationary policies π and π′π , ‖Vπ−Vπ′‖∞≤Rmax(1−γ)2sups‖π(⋅∣s)−π′(⋅∣s)‖1. \|V^π-V^π \|_∞≤ R_ (1-γ)^2 _s \|π(· s)-π (· s) \|_1. Moreover, ‖Aπ−Aπ′‖∞≤(1+γ)Rmax(1−γ)2sups‖π(⋅∣s)−π′(⋅∣s)‖1. \|A^π-A^π \|_∞≤ (1+γ)R_ (1-γ)^2 _s \|π(· s)-π (· s) \|_1. Proof. For any state s, Vπ(s)−Vπ′(s) V^π(s)-V^π (s) =⟨π(⋅∣s)−π′(⋅∣s),Qπ′(s,⋅)⟩ = π(· s)-π (· s),Q^π (s,·) +a∼π(⋅∣s)[Qπ(s,a)−Qπ′(s,a)]. +E_a π(· s) [Q^π(s,a)-Q^π (s,a) ]. By the Bellman relation, Qπ(s,a)−Qπ′(s,a)=γs′∼P(⋅∣s,a)[Vπ(s′)−Vπ′(s′)].Q^π(s,a)-Q^π (s,a)= _s P(· s,a) [V^π(s )-V^π (s ) ]. Therefore, by Hölder’s inequality, |Vπ(s)−Vπ′(s)| |V^π(s)-V^π (s) | ≤‖π(⋅∣s)−π′(⋅∣s)‖1‖Qπ′‖∞+γ‖Vπ−Vπ′‖∞. ≤ \|π(· s)-π (· s) \|_1 \|Q^π \|_∞+γ \|V^π-V^π \|_∞. Taking the supremum over s and using ‖Qπ′‖∞≤Rmax/(1−γ) \|Q^π \|_∞≤ R_ /(1-γ) gives ‖Vπ−Vπ′‖∞≤Rmax1−γsups‖π(⋅∣s)−π′(⋅∣s)‖1+γ‖Vπ−Vπ′‖∞. \|V^π-V^π \|_∞≤ R_ 1-γ _s \|π(· s)-π (· s) \|_1+γ \|V^π-V^π \|_∞. Rearranging proves the value bound. For the advantage bound, ‖Qπ−Qπ′‖∞≤γ‖Vπ−Vπ′‖∞, \|Q^π-Q^π \|_∞≤γ \|V^π-V^π \|_∞, and since Aπ=Qπ−VπA^π=Q^π-V^π, ‖Aπ−Aπ′‖∞≤‖Qπ−Qπ′‖∞+‖Vπ−Vπ′‖∞≤(1+γ)‖Vπ−Vπ′‖∞. \|A^π-A^π \|_∞≤ \|Q^π-Q^π \|_∞+ \|V^π-V^π \|_∞≤(1+γ) \|V^π-V^π \|_∞. Substituting the value bound completes the proof. ∎ Corollary F.4 (Local advantage drift). Suppose Assumption F.2 holds. If Ak=AπθkA_k=A _ _k, then for every k≥1k≥ 1, ‖Ak+1−Ak‖∞≤LAk,LA=(1+γ)RmaxCπ(1−γ)2. \|A_k+1-A_k \|_∞≤ L_Ak, L_A= (1+γ)R_ C_π(1-γ)^2. Proof. By Lemma F.3, ∥Ak+1−Ak∥∞≤(1+γ)Rmax(1−γ)2sups∥πθk+1(⋅∣s)−πθk(⋅∣s)∥1.\|A_k+1-A_k\|_∞≤ (1+γ)R_ (1-γ)^2 _s\| _ _k+1(· s)- _ _k(· s)\|_1. Applying Assumption F.2 to the right hand side gives the claim with LA=(1+γ)RmaxCπ/(1−γ)2L_A=(1+γ)R_ C_π/(1-γ)^2. ∎ Lemma F.5 (The recent window tracks the current advantage). Under Corollary F.4, for every t≥2t≥ 2, ‖At−ZtA‖∞≤Cloctζ−1, \|A_t-Z_t^A \|_∞≤ C_loct^ζ-1, where one may take Cloc=LA1−2ζ−1.C_loc= L_A1-2^ζ-1. Consequently, for every T≥2T≥ 2, 2T−1∑t=2T‖At−ZtA‖∞≤2ClocζTζT−1. 2T-1 _t=2^T \|A_t-Z_t^A \|_∞≤ 2C_locζ T^ζT-1. Proof. For k∈tk _t, Corollary F.4 gives ‖At−Ak‖∞ \|A_t-A_k \|_∞ ≤∑j=kt−1‖Aj+1−Aj‖∞ ≤ _j=k^t-1 \|A_j+1-A_j \|_∞ ≤LA∑j=kt−11j. ≤ L_A _j=k^t-1 1j. Since k≥t−Wtk≥ t-W_t and Wt≤tζW_t≤ t^ζ, we have k≥t−tζk≥ t-t^ζ. Also, since tζ/t=tζ−1≤2ζ−1t^ζ/t=t^ζ-1≤ 2^ζ-1 for every t≥2t≥ 2, t−Wt≥t−tζ≥(1−2ζ−1)t.t-W_t≥ t-t^ζ≥ (1-2^ζ-1 )t. Therefore, ∑j=kt−11j≤t−kt−Wt≤Wt(1−2ζ−1)t≤tζ−11−2ζ−1. _j=k^t-1 1j≤ t-kt-W_t≤ W_t(1-2^ζ-1)t≤ t^ζ-11-2^ζ-1. Thus, uniformly over k∈tk _t, ‖At−Ak‖∞≤Cloctζ−1. \|A_t-A_k \|_∞≤ C_loct^ζ-1. For each fixed (s,a)(s,a), ZtA(s,a)Z_t^A(s,a) is a convex combination of the values Ak(s,a):k∈t\A_k(s,a):k _t\ (coefficients proportional to μk(s,a) _k(s,a) on the support of μt _t, and uniform off it). Hence |At(s,a)−ZtA(s,a)| |A_t(s,a)-Z_t^A(s,a) | ≤maxk∈t|At(s,a)−Ak(s,a)| ≤ _k _t |A_t(s,a)-A_k(s,a) | ≤Cloctζ−1. ≤ C_loct^ζ-1. Taking the supremum over (s,a)(s,a) proves the pointwise bound. For the average, multiply the pointwise bound by 22 and use ∑t=2Ttζ−1≤∫1Txζ−1x=Tζ−1ζ≤Tζ. _t=2^Tt^ζ-1≤ _1^Tx^ζ-1\,dx= T^ζ-1ζ≤ T^ζ. ∎ F.4 Layer 2: direct regularized learned readout value transfer Layer 2 proves regularized approximate optimality directly for the learned policy πθt _ _t. The only transfer needed is from an L2(μt)L_2( _t) score error bound to a comparator state expectation. This is where the quadratic coverage coefficient enters. Where the action sampling mismatch is paid. The replay Bayes target weights each action by how often the retained behavior policies took it, so the fitted score is accurate where replay puts probability mass and is unconstrained elsewhere. The analysis does not estimate this mismatch. It bounds it through the assumption that the coverage coefficient below is finite, which holds exactly when the replay occupancy μt _t reaches every action at every state the comparator visits. The burden of the mismatch therefore sits on this coefficient. It grows when the comparator prefers actions the replayed policies rarely take, and the guarantee weakens accordingly. Removing the assumption, for example through importance weighting on the stored labels or through a drift argument, is open. For a comparator π⋆π , define the quadratic replay to comparator coverage coefficient Ctcov(π⋆)=esssups,adπ⋆(s)μt(s,a),C_t^cov(π )= *ess\,sup_s,a d^π (s) _t(s,a), with the convention that the ratio is +∞+∞ whenever dπ⋆(s)>0d^π (s)>0 and μt(s,a)=0 _t(s,a)=0. Let μtS(s)=∑aμt(s,a) _t^S(s)= _a _t(s,a) be the replay state marginal. In finite action spaces this coefficient is easy to interpret. If the replay conditional action distribution satisfies ρt(a∣s)=μt(s,a)μtS(s)≥η _t(a s)= _t(s,a) _t^S(s)≥η whenever μtS(s)>0 _t^S(s)>0, then Ctcov(π⋆)≤1ηesssupsdπ⋆(s)μtS(s).C_t^cov(π )≤ 1η *ess\,sup_s d^π (s) _t^S(s). Thus action exploration still enters the theory through the coverage transfer, even though it no longer enters the replay reweighting lemma. Lemma F.6 (Direct regularized optimality of the learned readout). For every comparator policy π⋆π , every state s, and every t≥2t≥ 2, ⟨π⋆(⋅∣s)−πθt(⋅∣s),At(s,⋅)⟩+λ(φ(πθt(⋅∣s))−φ(π⋆(⋅∣s))) π (· s)- _ _t(· s),A_t(s,·) +λ ( ( _ _t(· s))- (π (· s)) ) ≤βt−1ln||+2‖At(s,⋅)−ZtA(s,⋅)‖∞+2‖Zθt−1(s,⋅)−ZtA(s,⋅)‖2. ≤ βt-1 |A|+2 \|A_t(s,·)-Z_t^A(s,·) \|_∞+2 \|Z_ _t-1(s,·)-Z_t^A(s,·) \|_2. Proof. Fix s and t≥2t≥ 2. By definition of the readout, with αt−1=λ+β/(t−1) _t-1=λ+β/(t-1), πθt(⋅∣s)∈argmaxp∈Δ()⟨p,Zθt−1(s,⋅)⟩−αt−1φ(p). _ _t(· s)∈ _p∈ (A) \ p,Z_ _t-1(s,·) - _t-1 (p) \. Since this objective is concave, evaluating it at πθt(⋅∣s) _ _t(· s) and at π⋆(⋅∣s)π (· s) and using maximality gives ⟨π⋆(⋅∣s)−πθt(⋅∣s),Zθt−1(s,⋅)⟩+αt−1(φ(πθt(⋅∣s))−φ(π⋆(⋅∣s)))≤0. π (· s)- _ _t(· s),Z_ _t-1(s,·) + _t-1 ( ( _ _t(· s))- (π (· s)) )≤ 0. Writing αt−1=λ+β/(t−1) _t-1=λ+β/(t-1) and carrying the score from Zθt−1Z_ _t-1 to AtA_t through the regression Bayes target ZtAZ_t^A (all quantities evaluated at s), ⟨π⋆−πθt,At⟩+λ(φ(πθt)−φ(π⋆)) π - _ _t,A_t +λ ( ( _ _t)- (π ) ) =⟨π⋆−πθt,Zθt−1⟩+αt−1(φ(πθt)−φ(π⋆))⏟≤0−βt−1(φ(πθt)−φ(π⋆)) = π - _ _t,Z_ _t-1 + _t-1 ( ( _ _t)- (π ) )_≤ 0- βt-1 ( ( _ _t)- (π ) ) +⟨π⋆−πθt,At−ZtA⟩+⟨π⋆−πθt,ZtA−Zθt−1⟩. + π - _ _t,A_t-Z_t^A + π - _ _t,Z_t^A-Z_ _t-1 . We bound the three remaining terms. Since φ(p)=∑ap(a)lnp(a)∈[−ln||,0] (p)= _ap(a) p(a)∈[- |A|,0], −βt−1(φ(πθt)−φ(π⋆))≤βt−1ln||.- βt-1 ( ( _ _t)- (π ) )≤ βt-1 |A|. By Hölder’s inequality with ‖π⋆−πθt‖1≤2\|π - _ _t\|_1≤ 2, ⟨π⋆−πθt,At−ZtA⟩≤2‖At(s,⋅)−ZtA(s,⋅)‖∞. π - _ _t,A_t-Z_t^A ≤ 2 \|A_t(s,·)-Z_t^A(s,·) \|_∞. By the Cauchy–Schwarz inequality with ‖π⋆−πθt‖2≤‖π⋆‖2+‖πθt‖2≤2\|π - _ _t\|_2≤\|π \|_2+\| _ _t\|_2≤ 2 (each distribution has ℓ2 _2 norm at most 11), ⟨π⋆−πθt,ZtA−Zθt−1⟩≤2‖Zθt−1(s,⋅)−ZtA(s,⋅)‖2. π - _ _t,Z_t^A-Z_ _t-1 ≤ 2 \|Z_ _t-1(s,·)-Z_t^A(s,·) \|_2. Summing the three bounds proves the claim. ∎ Proposition F.7 (Direct regularized windowed value transfer). For every comparator policy π⋆π and every T≥2T≥ 2, 1T−1∑t=2T[(J(π⋆)−J(πθt))+λ1−γs∼dπ⋆(φ(πθt(⋅∣s))−φ(π⋆(⋅∣s)))] 1T-1 _t=2^T [ (J(π )-J( _ _t) )+ λ1-γE_s d^π ( ( _ _t(· s))- (π (· s)) ) ] ≤11−γ[βln||T−1∑t=2T1t−1+2T−1∑t=2T‖At−ZtA‖∞ ≤ 11-γ [ β |A|T-1 _t=2^T 1t-1+ 2T-1 _t=2^T \|A_t-Z_t^A \|_∞ +2T−1∑t=2TCtcov(π⋆)‖Zθt−1−ZtA‖2,μt]. + 2T-1 _t=2^T C_t^cov(π ) \|Z_ _t-1-Z_t^A \|_2, _t ]. Consequently, if Ctcov(π⋆)≤Ccov(π⋆)C_t^cov(π )≤ C_cov(π ) for all t=2,…,Tt=2,…,T, the last sum is bounded by 2Ccov(π⋆)⋅1T−1∑t=2T∥Zθt−1−ZtA∥2,μt2 C_cov(π )· 1T-1 _t=2^T\|Z_ _t-1-Z_t^A\|_2, _t. Proof. By the performance difference lemma, J(π⋆)−J(πθt)=11−γs∼dπ⋆[⟨π⋆(⋅∣s)−πθt(⋅∣s),At(s,⋅)⟩],J(π )-J( _ _t)= 11-γE_s d^π [ π (· s)- _ _t(· s),A_t(s,·) ], where At=AπθtA_t=A _ _t. Apply Lemma F.6 pointwise in s and take s∼dπ⋆E_s d^π : J(π⋆)−J(πθt)+λ1−γs∼dπ⋆(φ(πθt)−φ(π⋆)) J(π )-J( _ _t)+ λ1-γE_s d^π ( ( _ _t)- (π ) ) ≤11−γ[βt−1ln||+2‖At−ZtA‖∞+2s∼dπ⋆‖Zθt−1(s,⋅)−ZtA(s,⋅)‖2]. ≤ 11-γ [ βt-1 |A|+2 \|A_t-Z_t^A \|_∞+2\,E_s d^π \|Z_ _t-1(s,·)-Z_t^A(s,·) \|_2 ]. For the last term, by Jensen’s inequality and the change of measure bound (as in the coverage discussion above), s∼dπ⋆‖Zθt−1(s,⋅)−ZtA(s,⋅)‖2≤(∑s,adπ⋆(s)(Zθt−1(s,a)−ZtA(s,a))2)1/2≤Ctcov(π⋆)‖Zθt−1−ZtA‖2,μt.E_s d^π \|Z_ _t-1(s,·)-Z_t^A(s,·) \|_2≤ ( _s,ad^π (s) (Z_ _t-1(s,a)-Z_t^A(s,a) )^2 )^1/2≤ C_t^cov(π ) \|Z_ _t-1-Z_t^A \|_2, _t. Averaging over t=2,…,Tt=2,…,T gives the display. The final assertion uses Ctcov(π⋆)≤Ccov(π⋆)C_t^cov(π )≤ C_cov(π ). ∎ F.5 Layer 3: replay regression error decomposition Layer 2 reduces the learned policy value bound to the score error ‖Zθt−1−ZtA‖2,μt, \|Z_ _t-1-Z_t^A \|_2, _t, the residual of the learned score against the regression Bayes target ZtAZ_t^A. The norm is taken under the window replay marginal μt _t defined above. We recall the population replay target with true advantage labels, ZtA(s,a)=∑k∈tμk(s,a)Ak(s,a)∑k∈tμk(s,a)Z_t^A(s,a)= _k _t _k(s,a)A_k(s,a) _k _t _k(s,a) on the support of μt _t. Its critic label analogue is Z^t(s,a)=∑k∈tμk(s,a)A^k(s,a)∑k∈tμk(s,a). Z_t(s,a)= _k _t _k(s,a) A_k(s,a) _k _t _k(s,a). Let Z~t Z_t be the finite buffer empirical target represented by the actual replay sample. In the infinite buffer limit, Z~t=Z^t. Z_t= Z_t. The score error chain is Zθt−1⟶Z~t⟶Z^t⟶ZtA.Z_ _t-1 Z_t Z_t Z_t^A. Assumption F.8 (Primitive windowed replay error bounds). For every T≥2T≥ 2, the following bounds hold. 1. Fitting error: 1T−1∑t=2T‖Zθt−1−Z~t‖2,μt≤ϵ¯fit. 1T-1 _t=2^T \|Z_ _t-1- Z_t \|_2, _t≤ ε_fit. That is, the fitting error is not growing on average. 2. Finite replay concentration: ‖Z~t−Z^t‖2,μt≤Cvart−ζ/2for every t≥2. \| Z_t- Z_t \|_2, _t≤ C_vart^-ζ/2 every t≥ 2. This rate is the standard N−1/2N^-1/2 concentration for the empirical mean of Nt≍tζN_t t^ζ bounded labels around its conditional expectation, with CvarC_var depending on the label range GAG_A, the regression function class capacity, and a logarithmic factor from a union bound over t. 3. Critic label bias: 1T−1∑t=2T‖Z^t−ZtA‖2,μt≤ϵ0. 1T-1 _t=2^T \| Z_t-Z_t^A \|_2, _t≤ _0. That is, the critic bias is not growing. Lemma F.9 (Averaged windowed score error). Under Assumption F.8, for every T≥2T≥ 2, 1T−1∑t=2T‖Zθt−1−ZtA‖2,μt 1T-1 _t=2^T \|Z_ _t-1-Z_t^A \|_2, _t ≤ϵ¯fit+ϵ0+Cvar1−ζ/2T1−ζ/2T−1. ≤ ε_fit+ _0+ C_var1-ζ/2 T^1-ζ/2T-1. Proof. For each t, apply the triangle inequality along the chain of replay targets ending at ZtAZ_t^A: ‖Zθt−1−ZtA‖2,μt \|Z_ _t-1-Z_t^A \|_2, _t ≤‖Zθt−1−Z~t‖2,μt+‖Z~t−Z^t‖2,μt+‖Z^t−ZtA‖2,μt. ≤ \|Z_ _t-1- Z_t \|_2, _t+ \| Z_t- Z_t \|_2, _t+ \| Z_t-Z_t^A \|_2, _t. Average over t=2,…,Tt=2,…,T. The fitting and critic terms are controlled by the averaged bounds in Assumption F.8. For finite replay concentration, 1T−1∑t=2TCvart−ζ/2 1T-1 _t=2^TC_vart^-ζ/2 ≤CvarT−1∑t=2Tt−ζ/2 ≤ C_varT-1 _t=2^Tt^-ζ/2 ≤Cvar1−ζ/2T1−ζ/2T−1. ≤ C_var1-ζ/2 T^1-ζ/2T-1. Combining these estimates proves the claim. ∎ F.6 Final windowed value bound This subsection contains the proofs of Theorem 5.1 and Corollaries 5.2 and F.11 from the main text. Define the learned policy regularized gap Treg(π⋆):=1T−1∑t=2T[(J(π⋆)−J(πθt))+λ1−γs∼dπ⋆(φ(πθt(⋅∣s))−φ(π⋆(⋅∣s)))].G_T^reg(π ):= 1T-1 _t=2^T [ (J(π )-J( _ _t) )+ λ1-γE_s d^π ( ( _ _t(· s))- (π (· s)) ) ]. Proof of Theorem 5.1. Start from Proposition F.7, whose left hand side is Treg(π⋆)G_T^reg(π ). Bound the local drift term ‖At−ZtA‖∞\|A_t-Z_t^A\|_∞ using Lemma F.5 and the score error term using Lemma F.9. Under the reward bound |r(s,a)|≤Rmax|r(s,a)|≤ R_ , LA=(1+γ)RmaxCπ(1−γ)2,Cloc=(1+γ)RmaxCπ(1−γ)2(1−2ζ−1).L_A= (1+γ)R_ C_π(1-γ)^2, C_loc= (1+γ)R_ C_π(1-γ)^2(1-2^ζ-1). The stabilizer contribution satisfies βln||T−1∑t=2T1t−1≤βln||(1+lnT)T−1. β |A|T-1 _t=2^T 1t-1≤ β |A|(1+ T)T-1. Substituting these estimates gives the bound in Theorem 5.1. Equivalently, after expanding ClocC_loc, Treg(π⋆) _T^reg(π ) ≤11−γ[βln||(1+lnT)T−1+2(1+γ)RmaxCπζ(1−γ)2(1−2ζ−1)TζT−1 ≤ 11-γ [ β |A|(1+ T)T-1+ 2(1+γ)R_ C_πζ(1-γ)^2(1-2^ζ-1) T^ζT-1 +2Ccov(π⋆)(ϵ¯fit+ϵ0+Cvar1−ζ/2T1−ζ/2T−1)]. +2 C_cov(π ) ( ε_fit+ _0+ C_var1-ζ/2 T^1-ζ/2T-1 ) ]. For the balanced choice ζ=2/3ζ=2/3, 2Clocζ=3(1+γ)RmaxCπ(1−γ)2(1−2−1/3),Cvar1−ζ/2=32Cvar, 2C_locζ= 3(1+γ)R_ C_π(1-γ)^2(1-2^-1/3), C_var1-ζ/2= 32C_var, so Treg(π⋆) _T^reg(π ) ≤11−γ[βln||(1+lnT)T−1+3(1+γ)RmaxCπ(1−γ)2(1−2−1/3)T2/3T−1 ≤ 11-γ [ β |A|(1+ T)T-1+ 3(1+γ)R_ C_π(1-γ)^2(1-2^-1/3) T^2/3T-1 +2Ccov(π⋆)(ϵ¯fit+ϵ0+32CvarT2/3T−1)]. +2 C_cov(π ) ( ε_fit+ _0+ 32C_var T^2/3T-1 ) ]. Thus, for fixed λ>0λ>0, bounded coverage, and vanishing fitting and critic bias floors, Treg(π⋆)=O(T−1/3)G_T^reg(π )=O(T^-1/3). ∎ Remark F.10 (Reading the bound). The regularized gap is the natural goal for an entropy-regularized method. It is the quantity the RDA iterates optimize, the entropy floor and the stabilizer control it directly, and it converts to the ordinary value gap at the cost of one explicit entropy bias term (Corollary 5.2). The bound has three groups of terms: the dual averaging stabilizer (optimization) term, the local replay window drift, and the replay regression error, namely supervised fit ϵ¯fit ε_fit, critic label bias ϵ0 _0, and finite buffer variance CvarC_var, transferred to the comparator through coverage. The proof is given in Appendix F. The replay terms expose the window length tradeoff. With Wt≍tζW_t t^ζ, finite buffer variance scales as T−ζ/2T^-ζ/2 and replay drift as Tζ−1T^ζ-1. Balancing the two gives ζ=2/3ζ=2/3 and a T−1/3T^-1/3 rate. Smaller ζ uses fresher but noisier data, while larger ζ averages more data but increases stale replay drift. Proof of Corollary 5.2. Starting from Theorem 5.1, add λ1−γs∼dπ⋆[φ(π⋆(⋅∣s))−φ(πθt(⋅∣s))] λ1-γE_s d^π [ (π (· s))- ( _ _t(· s)) ] to the regularized gap. Since φ(p)∈[−ln||,0] (p)∈[- |A|,0], φ(π⋆(⋅∣s))−φ(πθt(⋅∣s))≤ln||. (π (· s))- ( _ _t(· s))≤ |A|. This gives the exact bound stated in Corollary 5.2. For ζ=2/3ζ=2/3 and fixed λ>0λ>0, collecting the nonvanishing fitting and critic terms separately from the stabilizer, drift, and variance terms gives 1T−1∑t=2T(J(π⋆)−J(πθt))≤λln||1−γ+2Ccov(π⋆)1−γ(ϵ¯fit+ϵ0)+O(T−1/3). 1T-1 _t=2^T (J(π )-J( _ _t) )≤ λ |A|1-γ+ 2 C_cov(π )1-γ( ε_fit+ _0)+O\! (T^-1/3 ). The O(T−1/3)O(T^-1/3) coefficient is free of any explicit 1/λ1/λ factor. The primitive constants CπC_π, CvarC_var, Ccov(π⋆)C_cov(π ), and constants hidden in ϵ¯fit ε_fit and ϵ0 _0 may still depend on λ through the learned policy path, replay coverage, and regression difficulty. ∎ Annealed-λ remark. Because λ now appears only in the entropy bias term (the score error coefficient is λ-free), annealing λ no longer inflates the replay terms. This is the improvement over the old softmax stability route, where the GA/λG_A/λ factor forced λT≍T−1/6 _T T^-1/6 and an unregularized T−1/6T^-1/6 rate. Corollary F.11 (Annealed unregularized rate). In the setting of Corollary 5.2, suppose the fitting and critic floors vanish, ϵ¯fit=ϵ0=0 ε_fit= _0=0, and Cπ,Cvar,Ccov(π⋆)C_π,C_var,C_cov(π ) remain uniformly bounded as λ→0λ→ 0. Then ζ=2/3ζ=2/3 and λT≍T−1/3 _T T^-1/3 give 1T−1∑t=2T(J(π⋆)−J(πθt))=O(T−1/3) 1T-1 _t=2^T(J(π )-J( _ _t))=O(T^-1/3). Proof of Corollary F.11. Under the hypotheses of Corollary F.11, set ϵ¯fit=ϵ0=0 ε_fit= _0=0. In Corollary 5.2, the only term with explicit dependence on λ is then the entropy bias λln||1−γ. λ |A|1-γ. The remaining terms are O(T−1/3)O(T^-1/3) and contain no explicit 1/λ1/λ factor. Under the stated uniform boundedness assumptions on CπC_π, CvarC_var, Ccov(π⋆)C_cov(π ), and the constants in the replay error bounds, their coefficients remain bounded along the schedule λ=λT→0λ= _T→ 0. Taking λT≍T−1/3 _T T^-1/3 makes the entropy bias O(T−1/3)O(T^-1/3), which proves 1T−1∑t=2T(J(π⋆)−J(πθt))=O(T−1/3). 1T-1 _t=2^T (J(π )-J( _ _t) )=O\! (T^-1/3 ). ∎ Because the score error coefficient 2Ccov(π⋆)2 C_cov(π ) is free of any explicit 1/λ1/λ factor, annealing the entropy weight leaves the replay terms at T−1/3T^-1/3, so the optimization and replay contributions match. The condition λ>0λ>0 controls the optimization term. The unregularized ablation λ=0λ=0 (Appendix D) is an empirical stress test. The hypothesis of Corollary F.11 that CπC_π, CvarC_var, and Ccov(π⋆)C_cov(π ) stay bounded as λ→0λ→ 0 is a genuine restriction, since a vanishing entropy floor sharpens the readout and can inflate both the path constant and the coverage coefficient. Corollary 5.2 at fixed λ is therefore the primary statement, and the annealed rate is conditional on that hypothesis. F.7 Density weighted surrogate and the action sampling mismatch This subsection connects the direct proof to the main text product space RDA surrogate. The main proof bridges through the replay regression Bayes target ZtAZ_t^A, which is exactly the object fit by squared loss. The main text product space surrogate instead induces a state density weighted window score Z¯tW Z_t^W. Define the average state occupancy in the current window by d¯tW(s)=1Wt∑k∈tdπθk(s). d_t^W(s)= 1W_t _k _td _ _k(s). The density weighted windowed score is Z¯tW(s,a)=∑k∈tdπθk(s)Ak(s,a)∑k∈tdπθk(s),∑k∈tdπθk(s)>0,1Wt∑k∈tAk(s,a),∑k∈tdπθk(s)=0. Z_t^W(s,a)= cases _k _td _ _k(s)A_k(s,a) _k _td _ _k(s),& _k _td _ _k(s)>0,\\[9.68747pt] 1W_t _k _tA_k(s,a),& _k _td _ _k(s)=0. cases The second line is only a tie breaker on states with zero window state mass. It does not change the density weighted surrogate, but it makes Z¯tW Z_t^W pointwise defined everywhere. The exact windowed product space surrogate corresponding to the main text is argmaxπ∈Π1Wt∑k∈t⟨Ak,π⟩dπθk−αt−1φd¯tW(π), _π∈ \ 1W_t _k _t A_k,π _d _ _k- _t-1 _ d_t^W(π) \, where φ(p)=∑ap(a)logp(a)=−H(p) (p)= _ap(a) p(a)=-H(p). Expanding the state expectation gives 1Wt∑k∈t⟨Ak,π⟩dπθk−αt−1φd¯tW(π) 1W_t _k _t A_k,π _d _ _k- _t-1 _ d_t^W(π) =1Wt∑k∈t∑sdπθk(s)⟨π(⋅∣s),Ak(s,⋅)⟩−αt−1∑sd¯tW(s)φ(π(⋅∣s)) = 1W_t _k _t _sd _ _k(s) π(· s),A_k(s,·) - _t-1 _s d_t^W(s) (π(· s)) =∑sd¯tW(s)[⟨π(⋅∣s),Z¯tW(s,⋅)⟩−αt−1φ(π(⋅∣s))] = _s d_t^W(s) [ π(· s), Z_t^W(s,·) - _t-1 (π(· s)) ] =⟨Z¯tW,π⟩d¯tW−αt−1φd¯tW(π), = Z_t^W,π _ d_t^W- _t-1 _ d_t^W(π), where the equality uses the density weighted branch on states with d¯tW(s)>0 d_t^W(s)>0, and states with d¯tW(s)=0 d_t^W(s)=0 carry zero mass in the display. Thus the state density factor multiplies both the linear advantage term and the entropy term, and the statewise optimizer on the support of d¯tW d_t^W is softmax(Z¯tW(s,⋅)/αt−1)softmax( Z_t^W(s,·)/ _t-1). The gap between the replay Bayes target ZtAZ_t^A and the density weighted surrogate Z¯tW Z_t^W is the action sampling mismatch. The next corollary bounds it and shows that a value guarantee phrased against the density weighted surrogate holds at the same rate. Corollary F.12 (Action sampling mismatch and the density weighted surrogate). Under the assumptions of Theorem 5.1, for every t≥2t≥ 2, ‖ZtA−Z¯tW‖2,μt≤2Cloctζ−1. \|Z_t^A- Z_t^W \|_2, _t≤ 2C_loct^ζ-1. Consequently, replacing the replay Bayes target ZtAZ_t^A by the density weighted surrogate Z¯tW Z_t^W in the score residual adds at most 2Ccov(π⋆)1−γ⋅2ClocζTζT−1=O(Tζ−1) 2 C_cov(π )1-γ· 2C_locζ T^ζT-1=O(T^ζ-1) to the bound of Theorem 5.1, the same order as the local drift term, and does not change the T−1/3T^-1/3 rate. Proof. On the support of μt _t, both ZtA(s,a)Z_t^A(s,a) and Z¯tW(s,a) Z_t^W(s,a) are convex combinations of Ak(s,a):k∈t\A_k(s,a):k _t\, so ‖ZtA−Z¯tW‖2,μt \|Z_t^A- Z_t^W \|_2, _t ≤sups,a|ZtA(s,a)−Z¯tW(s,a)| ≤ _s,a |Z_t^A(s,a)- Z_t^W(s,a) | ≤2maxk∈t‖Ak−At‖∞≤2Cloctζ−1, ≤ 2 _k _t \|A_k-A_t \|_∞≤ 2C_loct^ζ-1, where the last inequality is the convex combination stability bound from the proof of Lemma F.5. For the second claim, the triangle inequality gives ‖Zθt−1−Z¯tW‖2,μt≤‖Zθt−1−ZtA‖2,μt+2Cloctζ−1, \|Z_ _t-1- Z_t^W \|_2, _t≤ \|Z_ _t-1-Z_t^A \|_2, _t+2C_loct^ζ-1, and averaging the extra term over t=2,…,Tt=2,…,T contributes at most 2ClocζTζT−1 2C_locζ T^ζT-1 to the averaged score residual, which enters the value bound through the prefactor 2Ccov(π⋆)1−γ 2 C_cov(π )1-γ of Theorem 5.1. The addition is of the same order as the local drift term already present in Theorem 5.1. ∎ The main statements are kept against ZtAZ_t^A because this removes the action sampling mismatch from the critical path while retaining the transparent connection to the main text surrogate written here. F.8 Scope of the guarantee The theorem provides the design rationale for the estimator and window schedule and analyzes the core algorithmic procedure implemented in code. Its objects are the stored labels, the finite window, and the fitted score. Two assumptions hold as stated in the implementation, with bounded rewards on every benchmark and an exact readout within the chosen score class, using a softmax on Atari and a diagonal Gaussian induced by the quadratic score in MuJoCo. The temperature schedule is the analyzed αt=λ+β/t _t=λ+β/t. The analyzed window grows as t2/3t^2/3 to balance replay concentration against drift as t→∞t→∞. At a fixed budget the same tradeoff selects a fixed window, and the window ablation (Appendix D.1) confirms that the optimum is interior. The remaining conditions concern the learned path, and each has a mechanism behind it. Path stability first. In full-history RDA the dual score is a running average of k rounds of feedback, so consecutive scores differ by O(1/k)O(1/k) and the readout inherits that rate. This is where Assumption F.2 originates. A sliding window gives up that mechanism in exchange for faster adaptation, and the implementation supplies stability by other means: the regression is warm-started from the previous round, each round replaces a small fraction of the window (one of 2424 rollouts in the GAE MuJoCo setting), the clipped dual regression bounds the movement of every stored prediction by ϵclip _clip per update, and the temperature decays. We do not verify the Cπ/kC_π/k rate, and the reported runs did not need a step-size schedule, but a schedule gives a direct handle on it. With gradient clipping, the parameter movement per round is bounded by the step size times a constant, so a 1/k1/k step size enforces 1/k1/k policy movement whenever the readout is Lipschitz in the parameters uniformly over states. Coverage next. The coverage coefficient is finite exactly when replay puts positive mass on every action at every state the comparator visits. The entropy floor supplies the action half: with λ>0λ>0 every readout policy has full support on a finite action set, so μt(s,a)>0 _t(s,a)>0 at every visited state. What remains is state coverage, that replay visits the states the comparator visits, the same exploration requirement that every comparator-based policy improvement guarantee carries. Fitting and concentration last. The fitting error enters as a per-round average of the error of the score the optimizer returns, which the theorem keeps as a floor. The finite replay concentration rate Cvart−ζ/2C_vart^-ζ/2 is the variance scaling of an average over Wt≍tζW_t t^ζ samples. Only two implementation heuristics lie outside the analyzed update: the clipped dual regression of Appendix A.4, and the per-batch normalization and clipping of advantage labels. Both act in the direction of the assumptions. The clip bounds per-update movement, which is what path stability asks for, and the normalization is an affine rescaling of the labels, which changes the effective temperature scale and nothing else in the readout. The twin-Q runs autotune αt _t as SAC does, which sits outside the analyzed schedule, and MuJoCo actions are continuous while the proof assumes finite actions. The Gaussian readout is the continuous instance of the same mirror map, and the experiments test whether the finite-action guarantee transfers. Appendix G Related Work This appendix reviews the work closest to RDA2C. The organizing distinction is between (i) last iterate policy space mirror updates, (i) cumulative FTRL/RDA updates, and (i) practical regression or replay mechanisms used to instantiate these ideas with function approximation. RDA2C sits at the intersection of the last two: it uses a dual averaging actor objective, but estimates the cumulative dual object through replayed advantage labels in a deep actor–critic loop. G.1 Policy mirror descent, natural gradients, and trust region methods A long line of work interprets modern policy optimization as approximate mirror descent in policy space, with KL divergence as the Bregman divergence induced by negative entropy. Natural policy gradient introduced the use of the Fisher geometry for policy search (Kakade, 2001), and TRPO later operationalized a trust region version of policy improvement with a KL constraint (Schulman et al., 2015). PPO replaces the constrained update by a clipped surrogate that approximately limits deviation from the behavior policy (Schulman et al., 2017), while MPO uses an EM-like KL regularized policy improvement step (Abdolmaleki et al., 2018). Maximum entropy methods such as SAC optimize entropy regularized objectives and can be interpreted within the broader regularized MDP and KL regularized policy optimization family (Haarnoja et al., 2018; Geist et al., 2019). Theoretical work has made this mirror descent view increasingly explicit. Convex analytic treatments of entropy regularized MDPs relate exact trust region and regularized policy improvement updates to mirror descent and duality (Neu et al., 2017; Geist et al., 2019). Adaptive trust region and policy mirror descent analyses give global convergence and sampling complexity guarantees for exact or near exact policy space updates (Shani et al., 2020; Lan, 2023). Functional mirror ascent and surrogate optimization analyses further clarify how policy improvement objectives can be derived independently of a particular parameterization (Vaswani et al., 2022). Empirically motivated mirror descent actor–critic methods, including mirror descent policy optimization (MDPO), exploratory conservative policy optimization (ECPO), and mirror descent actor–critic (MDAC), instantiate related KL/entropy regularized updates in deep RL systems (Tomar et al., 2022; Mei et al., 2019; Iwaki, 2025). Q–NPG is relevant to RDA2C because it connects value function regression, compatible function approximation, and natural gradient policy updates. In the policy gradient theory of Agarwal et al. (2019), NPG and Q–NPG updates are written through regression problems whose solutions define a natural policy gradient step. This is close in spirit to our use of advantage labels, but the update remains a last iterate PMD/NPG step: the fitted object determines a local update from the current policy. RDA2C instead fits a replay estimated state weighted average advantage score and reads the policy from that cumulative object through the entropy mirror map. G.2 FTRL, regularized dual averaging, and policy dual averaging Regularized dual averaging originates in online convex optimization and stochastic optimization, where the algorithm maintains an average of past subgradients in the dual space and maps back to the primal variable through a regularized mirror map. Foundational treatments include Nesterov’s primal-dual subgradient schemes (Nesterov, 2009) and Xiao’s regularized dual-averaging method (Xiao, 2010). RDA is closely related to follow-the-regularized-leader: McMahan (2011) establishes equivalence results connecting mirror descent and FTRL updates under suitable choices of regularization and linearization. This equivalence is important for positioning RDA2C. We do not claim that FTRL/RDA generically improves the worst case online learning rate over mirror descent. Our use is representational: FTRL/RDA exposes the cumulative feedback as a single object. PMD/OMD writes the next update as a local step around the last policy. RDA/FTRL writes the next update as a regularized optimization over the cumulative linearized feedback. In reinforcement learning, that cumulative feedback is naturally represented by accumulated advantage or Q labels. This is what makes replay useful as an actor representation: the buffer approximates the object the RDA update needs. The closest precedent for the update itself is POLITEX (Abbasi-Yadkori et al., 2019), which sets the next policy to the softmax of the cumulative sum of past action-value estimates and proves regret bounds through the reduction to expert prediction. RDA2C keeps that cumulative-softmax form but replaces the explicit sum of stored critics by one regression on replayed advantage labels, adds the persistent entropy floor and the decaying stabilizer of RDA, and analyzes a finite replay window in place of the full history. In the same direction, Vieillard et al. (2020a) show that KL-regularized policy iteration implicitly averages past Q estimates, and Deep CPI (Vieillard et al., 2020b) makes the related conservative mixture update practical with function approximation. Both hold the cumulative object recursively, in the current parameters. RDA2C holds it in the label buffer, which is what makes the window an explicit design variable. Within RL, a close theoretical precursor is policy dual averaging. Ju and Lan (2022) study policy optimization over general state and action spaces and introduce PDA methods designed to work with function approximation more directly than exact policy mirror descent. That line studies cleaner regimes with exact or bounded error evaluation, and the original continuous action update is defined by a statewise optimization subproblem. The actor accelerated PDA follow up (Gao et al., 2026) addresses this computational issue by learning an actor that approximates the PDA target, while DAPO studies dual-side approximation error through the Bregman divergence induced by the mirror potential (Xiong et al., 2024). RDA2C differs by eliminating the separate projection tracking actor: the learned dual score model ZθZ_θ is the policy’s natural coordinate representation, and the current policy is obtained directly through the entropy readout ∇φ∗(Zθ/αt)∇ ^*(Z_θ/ _t). Two further contrasts follow. PDA solves the readout statewise and numerically, while RDA2C replaces that solve with one regression and an exact mirror map. DAPO measures the error of a given dual approximation, while RDA2C specifies how to construct the dual object from replayed labels. The analyses differ the same way: prior guarantees assume exact or bounded-error policy evaluation, while our finite-time analysis charges the update for critic bias, finite-buffer variance, and replay-induced distribution shift. G.3 Replay, off policy actor–critic learning, and stale data Experience replay is a standard mechanism for improving sample efficiency in deep RL. In off-policy actor–critic methods, replay usually provides data for Bellman regression and actor-gradient estimates, as in SAC (Haarnoja et al., 2018). In on-policy or near on-policy methods such as PPO, data are typically reused only for a small number of epochs, with clipping or KL control limiting the effect of policy drift (Schulman et al., 2017). In both cases, stale data are primarily handled as an off-policy correction, stabilization, or variance-reduction issue. RDA2C changes the role of replay on the actor side. Because the ideal RDA update depends on a historical average of policy space feedback, past advantage labeled samples are not merely stale approximations to the newest gradient. They are samples from the empirical dual objective that defines the next actor. This distinction also explains why buffer size appears explicitly in our theory. A short window reduces distribution shift but raises estimator variance, while a long window averages more labels but can bias the dual estimate toward older policies. The main theorem formalizes this replay-specific decomposition through critic bias, finite buffer variance, and replay induced distribution shift. G.4 Reduction and regression views of policy improvement RDA2C is also related to reductions that cast sequential decision making as iterative supervised learning on aggregated data. DAgger aggregates expert-labeled states to reduce imitation learning to no-regret online learning (Ross et al., 2011). NRPI extends this perspective to reinforcement learning by aggregating cost-to-go examples under an exploration distribution and reducing approximate policy iteration to no regret learning (Ross and Bagnell, 2014). These methods are close in spirit to RDA2C because the dataset is not an incidental implementation detail: it is the object through which the next policy improvement problem is specified. Recent regression based policy optimization methods make a related move in modern RL and RLHF settings. REBEL reduces reinforcement learning to squared loss regression on relative rewards, REFUEL extends a regression and aggregation viewpoint to multi-turn RLHF under self-generated data, and A⋆-PO estimates an optimal-advantage-like object before applying simple regression policy updates (Gao et al., 2024a; Gao et al., 2024b; Brantley et al., 2025). RDA2C follows the same broad reduction to regression philosophy, but its regression target is derived from regularized dual averaging: the model ZθZ_θ approximates a windowed average of policy space improvement feedback, and the actor update is the exact entropy mirror map readout from that learned dual score. Advantage-weighted methods also turn policy improvement into supervised learning. AWR (Peng et al., 2019), AWAC (Nair et al., 2020), and CRR (Wang et al., 2020) fit the policy by weighted maximum likelihood on replayed actions, with weights that increase with the estimated advantage. That regression is in the primal: the target is a reweighted behavior distribution, and the policy is the fitted likelihood model. RDA2C regresses in the dual: the target is the advantage label itself, the fitted object is a score, and the policy is its mirror-map readout. The two differ in what the buffer represents. For advantage weighting, replay supplies actions to imitate. For RDA2C, replay supplies the cumulative feedback that defines the next policy. Empirical studies of example forgetting in neural networks provide an additional motivation for keeping policy improvement history in an explicit buffer (Toneva et al., 2019). Recursive updates keep that history only in the actor weights. This motivation is secondary. The primary distinction is algorithmic: RDA2C’s replay buffer estimates the dual object required by the RDA update, and recursive actor updates keep past feedback only in the current parameter vector.