Paper deep dive
DAMPER: Return-Prioritized Gradient Control for Smooth Policies
Seokmin Ko, Taewon Goo, Kihyuk Hong
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 10/4/2026, 3:30:22 AM
Summary
The paper introduces DAMPER, a gradient control method for actor-critic reinforcement learning algorithms (TD3, SAC) that reduces action oscillation while prioritizing return maximization. DAMPER combines the native actor gradient with a temporal-consistency gradient using conflict-conditioned projection and adaptive magnitude control, ensuring the native objective remains primary. Experiments on six continuous-control tasks demonstrate reduced action oscillation compared to baselines like CAPS, Grad-CAPS, ASAP, L2C2, and PAVE.
Entities (17)
Relation Signals (19)
Seokmin Ko → affiliatedwith → KAIST
confidence 95% · Seokmin Ko Taewon Goo Kihyuk Hong Affiliation: KAIST
Kihyuk Hong → affiliatedwith → KAIST
confidence 95% · Seokmin Ko Taewon Goo Kihyuk Hong Affiliation: KAIST
Taewon Goo → affiliatedwith → KAIST
confidence 95% · Seokmin Ko Taewon Goo Kihyuk Hong Affiliation: KAIST
DAMPER → improves → action oscillation
confidence 95% · Experiments with TD3 and SAC on six continuous-control tasks show reduced action oscillation relative to the native agents in all 12 task-backbone pairs
DAMPER → isappliedto → TD3
confidence 95% · Experiments with TD3 and SAC... show reduced action oscillation
DAMPER → isappliedto → SAC
confidence 95% · Experiments with TD3 and SAC... show reduced action oscillation
DAMPER → isevaluatedon → Hopper
confidence 95% · Experiments with TD3 and SAC on six continuous-control tasks... Hopper...
DAMPER → isevaluatedon → Ant
confidence 95% · Experiments with TD3 and SAC on six continuous-control tasks... Ant...
DAMPER → →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Actor-critic methods achieve strong performance in continuous control, but their policies can produce highly oscillatory actions. A common remedy is to add auxiliary smoothness losses. However, their contribution can be negligible when their gradients are small relative to the native actor gradient. Moreover, existing methods often combine multiple auxiliary losses, complicating loss balancing without necessarily improving the return-smoothness trade-off. We introduce DAMPER (Direction-Aware Magnitude-Controlled Projection with Explicit Return Priority), which combines the native actor gradient with a temporal-consistency gradient through conflict-conditioned projection and adaptive magnitude control. It removes the auxiliary component opposing the actor gradient and scales the retained temporal direction relative to the actor gradient norm, preserving positive alignment with the native actor gradient. Experiments with TD3 and SAC on six continuous-control tasks show reduced action oscillation relative to the native agents in all 12 task-backbone pairs and the best oscillation score among the compared methods in eight, with task-dependent return trade-offs.
Tags
Links
- Source: https://arxiv.org/abs/2609.38903v1
- Canonical: https://arxiv.org/abs/2609.38903v1
Trouble viewing inline? Open PDF directly →
Full Text
59,998 characters extracted from source content.
Expand or collapse full text
DAMPER: Return-Prioritized Gradient Control for Smooth Policies Seokmin Ko Taewon Goo Kihyuk Hong Affiliation: KAIST Affiliation: komin0407, tetrise9, kihyukh@kaist.ac.kr Abstract Actor–critic methods achieve strong performance in continuous control, but their policies can produce highly oscillatory actions. A common remedy is to add auxiliary smoothness losses. However, their contribution can be negligible when their gradients are small relative to the native actor gradient. Moreover, existing methods often combine multiple auxiliary losses, complicating loss balancing without necessarily improving the return–smoothness trade-off. We introduce Damper (Direction-Aware Magnitude-Controlled Projection with Explicit Return Priority), which combines the native actor gradient with a temporal-consistency gradient through conflict-conditioned projection and adaptive magnitude control. It removes the auxiliary component opposing the actor gradient and scales the retained temporal direction relative to the actor gradient norm, preserving positive alignment with the native actor gradient. Experiments with TD3 and SAC on six continuous-control tasks show reduced action oscillation relative to the native agents in all 12 task–backbone pairs and the best oscillation score among the compared methods in eight, with task-dependent return trade-offs. Our code is available at https://github.com/komin0407/DAMPER. 1 Introduction Actor–critic algorithms are effective tools for continuous control, but high episodic return does not imply a well-behaved control signal. A learned policy may vary its action sharply between adjacent time steps even when the system state changes only slightly. Simulation often tolerates these oscillations; physical platforms do not. Rapid control changes can waste energy, amplify unmodeled dynamics, and increase actuator wear [5]. Learning policies that are both effective and temporally smooth is therefore an important requirement for deploying reinforcement learning on real systems. Figure 1: Overview of Damper. Most existing approaches introduce smoothness through an auxiliary loss. They regularize policy outputs at consecutive or nearby states, constrain local policy sensitivity, or stabilize the critic gradients used by the actor [20, 14, 16, 15, 17]. These methods can reduce action variation, but the smoothing they achieve is often limited. Moreover, many existing methods introduce two or more auxiliary losses to control smoothness [20, 14, 15, 17], which complicates loss balancing without necessarily improving the return–smoothness trade-off, as illustrated by our comparison with a tuned single-loss baseline in Section 6.4. We approach this problem as primary–auxiliary gradient optimization. The primary objective is the native actor loss of the underlying algorithm. The auxiliary objective is the established consecutive-state consistency penalty (s,s′)[∥πθ(s′)−πθ(s)∥22]E_(s,s )[ _θ(s )- _θ(s) _2^2], where πθ _θ is the deterministic action used for evaluation. Our contribution is not this temporal loss itself, but a rule for combining its gradient with the native actor gradient. We introduce Damper, which combines the native actor and temporal gradients through two operations (Figure 1). First, when the temporal gradient conflicts with the native actor gradient, Damper removes only its return-opposing component. The native actor gradient is never projected, and an aligned temporal component is retained rather than discarded. Second, Damper controls the remaining direction relative to the native gradient norm. A bounded parameter η∈[0,1]η∈[0,1] interpolates between two endpoints: at η=0η=0, the temporal component is only capped at the native gradient norm, and at η=1η=1, it is rescaled to that norm. Unlike a scalar regularization coefficient, η controls how the observed gradient magnitudes are reconciled after the objectives have been differentiated. Figure 1 provides an overview of these two operations. Across six continuous-control tasks and two backbones, Damper improves the FFT-based oscillation score over the corresponding unregularized agent in all 12 task–backbone pairs and achieves the best score among the compared methods in 8. The improvement is more consistent for action stability than for return: several locomotion settings exhibit a clear smoothness–return trade-off. We report this limitation explicitly and isolate the role of conflict-conditioned projection in an ablation study. Our contributions are as follows: • We introduce DAMPER, a return-prioritized actor update that combines conflict-conditioned projection with norm-ratio interpolation. • We establish finite-step descent for the fixed actor surrogate and characterize temporal progress at matched first-order primary progress. • DAMPER reduces action oscillation relative to native TD3 and SAC on all six tasks, achieving the best score in eight of 12 task–backbone pairs. Projection ablations, interpolation sweeps, and fixed-weight scalarization comparisons assess the contributions of directional and magnitude control. 2 Related Work 2.1 Policy Smoothness in Reinforcement Learning Policy-output regularization. CAPS combines temporal consistency between consecutive states with spatial consistency under state perturbations [20]. Closely related work directly penalizes consecutive deterministic policy outputs [4]. We use the same form of temporal surrogate, but do not add it to the native actor loss with a fixed coefficient. Instead, we modify its gradient according to its alignment and scale relative to the native actor gradient. Sensitivity and critic regularization. Grad-CAPS regularizes policy sensitivity [16], ASAP combines action alignment, action prediction, and second-order temporal regularization [15], and L2C2 constrains local changes in both the policy and value function [14]. PAVE instead regularizes the critic to stabilize the action-gradient field that drives policy learning [17]. These approaches encode richer notions of regularity than our single temporal loss, but multi-component scalarized objectives introduce additional relative weights. Damper is complementary: it focuses on how a temporal auxiliary gradient should enter the actor update when the native actor objective remains primary. Structured smooth control. Smooth behavior can also arise from architectural or action-space structure. LipsNet controls policy sensitivity through an adaptive Lipschitz architecture [25]; generalized state-dependent exploration produces temporally correlated exploration noise [23]; and action-rate parameterizations smooth the executed control by construction [2]. In contrast, Damper changes only training-time actor gradients and leaves the policy architecture and execution interface unchanged. 2.2 Gradient Combination and Objective Priority Multi-objective optimization. MGDA searches for a common descent direction [6, 24], GradNorm balances task-gradient magnitudes [1], PCGrad removes conflicting gradient components [28], and CAGrad constructs a conflict-averse common update [18]. These methods typically treat objectives symmetrically. Our setting is hierarchical: the native actor objective is always primary, so its gradient is never altered to accommodate temporal consistency. Primary–auxiliary learning. Gradient-similarity weighting [7], Dynamic Barrier [9], and Bloop [12] prevent auxiliary objectives from degrading a primary task, while MetaBalance [11] adapts auxiliary-gradient magnitudes to a target gradient. Damper combines the two relevant ideas: it conditionally removes only the primary-opposing temporal component and separately controls its magnitude through a bounded norm-ratio interpolation. Priority-aware reinforcement learning. PEGrad prioritizes reward over energy minimization by orthogonalizing and capping the lower-priority gradient [21]; GCR-PPO conditionally projects lower-priority reward components [19]; and LPPG-RL considers lexicographically ordered policy-gradient objectives [22]. FCGrad addresses individual–collective return conflicts in mixed-motive multi-agent learning and changes priority according to current returns [13]. Our hierarchy is fixed, projection occurs only under conflict, and the retained temporal direction is scaled by an explicit interpolation between cap-only and norm-balanced updates. The contribution is this combination for temporal policy smoothing, rather than gradient projection in isolation. 3 Preliminaries Actor–critic learning. We consider a discounted Markov decision process ℳ=(,,p,r,γ)M=(S,A,p,r,γ), with transition kernel p(s′∣s,a)p(s s,a), reward r(s,a)r(s,a), and discount factor γ∈[0,1)γ∈[0,1). A policy πθ _θ seeks to maximize the expected return J(θ):=πθ[∑t=0∞γtr(st,at)].J(θ):=E_ _θ [ _t=0^∞γ^tr(s_t,a_t) ]. Actor–critic methods use a learned action-value function Qϕ(s,a)Q_φ(s,a) to guide policy optimization. We adopt the loss-minimization convention and denote the underlying algorithm’s native actor loss by LR(θ)L_R(θ). Algorithm-specific definitions are provided in Appendix E. Temporal policy consistency. Let πθ(s) _θ(s) denote the differentiable deterministic action map associated with the current policy. For consecutive states (s,s′)(s,s ) drawn from an observed transition distribution D, we define the temporal-consistency loss LT(θ):=(s,s′)∼[‖πθ(s′)−πθ(s)‖22].L_T(θ):=E_(s,s ) [ _θ(s )- _θ(s) _2^2 ]. (1) Both actions are computed using the same current policy. Minimizing this loss encourages similar actions at consecutive states, promoting temporal consistency along environment trajectories. Combining objectives. We denote the actor-parameter gradients by gR:=∇θLR(θ),gT:=∇θLT(θ).g_R:= _θL_R(θ), g_T:= _θL_T(θ). (2) The term “return gradient” refers to gRg_R, the critic-based actor-loss gradient, rather than a direct gradient of measured episodic return. A conventional auxiliary-loss formulation minimizes ℒλ(θ):=LR(θ)+λLT(θ),λ>0,L_λ(θ):=L_R(θ)+λ L_T(θ), λ>0, (3) which yields the combined gradient gR+λgTg_R+λ g_T. 4 Method We introduce Damper, an actor update rule designed to reduce action oscillation while prioritizing return maximization. It uses a temporal-consistency objective to encourage similar actions at consecutive states, with the native actor objective taking priority. To combine their gradients, Damper first removes any temporal component that opposes the native actor gradient, then controls the remaining contribution relative to the native gradient norm. These two operations address directional conflict and scale imbalance when incorporating temporal consistency into the actor update. Figure 1 summarizes the complete update. 4.1 Conflict-Conditioned Projection We use conflict-conditioned projection to prevent the temporal gradient from hindering first-order progress on the native actor objective. Let gR=∇θLRg_R= _θL_R and gT=∇θLTg_T= _θL_T denote the gradients of the objectives defined in Section 3. When gR⊤gT<0g_R g_T<0, the temporal gradient contains a component that opposes the native actor gradient. We remove this component by projecting gTg_T onto the subspace orthogonal to gRg_R. Otherwise, we leave gTg_T unchanged. Denoting the resulting temporal gradient by g~T g_T, we obtain g~T:=gT−gR⊤gT∥gR∥22gR,gR⊤gT<0,gT,gR⊤gT≥0. g_T:= casesg_T- g_R g_T g_R _2^2g_R,&g_R g_T<0,\\[5.69054pt] g_T,&g_R g_T≥ 0. cases (4) The native gradient gRg_R is never modified. The retained temporal gradient satisfies gR⊤g~T≥0g_R g_T≥ 0, so adding a nonnegative multiple of g~T g_T preserves the first-order decrease in the native actor loss provided by gRg_R. The Conflict and Aligned panels in Figure 1 illustrate the two cases. 4.2 Adaptive Magnitude Control Projection resolves directional conflict, but a temporal gradient that is small relative to the native actor gradient may have little influence on policy smoothness, whereas a much larger one may dominate the actor update. We therefore rescale the temporal gradient according to the relative norms of the original actor and temporal gradients. For nonzero gradients, let nR:=∥gR∥2n_R:= g_R _2 and nT:=∥gT∥2n_T:= g_T _2. Before accounting for directional conflict, consider the target combined gradient gtarget(s)≔gR+sgTnT,s>0.g_target(s) g_R+s g_Tn_T, s>0. This target adds a temporal contribution of magnitude s along gTg_T to the unchanged actor gradient gRg_R. The target may, however, reduce first-order progress on the native actor objective relative to using gRg_R alone. We therefore make the smallest adjustment, measured in Euclidean distance, to gtarget(s)g_target(s) that preserves this progress: g(s)≔argmind:gR⊤d≥∥gR∥22∥d−gtarget(s)∥22.g(s) _d:g_R d≥ g_R _2^2\|d-g_target(s)\|_2^2. The constraint requires at least the first-order decrease in the native actor loss achieved by gRg_R. It can be shown (Appendix A) that the unique solution to the optimization problem is g(s)=gR+snTg~T,g(s)=g_R+ sn_T g_T, where g~T g_T is exactly the conflict-conditioned projection from the previous subsection. We consider two reference target magnitudes. The cap-only choice s0≔min(nT,nR)s_0 (n_T,n_R) caps the target temporal magnitude at the native actor gradient norm nRn_R, without amplifying weak temporal gradients. The norm-matched choice s1≔nRs_1 n_R instead rescales the temporal contribution to nRn_R before projection, amplifying it when nT<nRn_T<n_R. To obtain intermediate behavior, we geometrically interpolate between these choices using a parameter η∈[0,1]η∈[0,1]: sη≔s01−ηs1η,gη≔g(sη)=gR+sηnTg~T.s_η s_0^1-ηs_1^η, g_η g(s_η)=g_R+ s_ηn_T g_T. (5) Equivalently, defining q≔min(nT/nR,1)q (n_T/n_R,1) and wη≔q1−ηw_η q^1-η gives sη=nRwηs_η=n_Rw_η. For nT<nRn_T<n_R, increasing η strengthens the temporal gradient. 4.3 DAMPER Actor Update Algorithm 1 summarizes the core actor update for nonzero gradients. Damper computes the two gradients separately, applies conflict-conditioned projection and adaptive magnitude control, and passes the merged gradient to the native actor optimizer. Algorithm 1 Damper actor update Input: Actor parameters θ, replay buffer D, native actor loss LRL_R, interpolation η∈[0,1]η∈[0,1]. 1 Sample a minibatch ℬB from D 2 Evaluate LR(θ)L_R(θ) and LT(θ)L_T(θ) in equation 1 3 gR←∇θLRg_R← _θL_R, gT←∇θLTg_T← _θL_T 4 nR←∥gR∥2n_R← g_R _2, nT←∥gT∥2n_T← g_T _2 5 g~T←gT g_T← g_T 6 if gR⊤gT<0g_R g_T<0 then 7 g~T←gT−gR⊤gTnR2gR g_T← g_T- g_R g_Tn_R^2g_R 8 q←min(nT/nR,1)q← (n_T/n_R,1), wη←q1−ηw_η← q^1-η 9 g←gR+(nR/nT)wηg~Tg← g_R+(n_R/n_T)w_η g_T 10 Assign g to the actor-gradient buffers and apply the native actor optimizer 5 Analysis of the Primary–Auxiliary Update We analyze a single Euclidean actor step using the direction constructed by Algorithm 1, with the critic and sampled transitions held fixed. 5.1 Descent on the Native Actor Loss Consistent with our primary–auxiliary formuation, in which the native actor objective tasks priority over temporal consistency, we first show that Damper preserves descent on the native actor loss even when the two gradients conflicts. Theorem 1 (Finite-step descent for the fixed actor surrogate). Assume that LRL_R is differentiable and globally βR _R-smooth for βR>0 _R>0, and that gR=∇θLR(θ)≠0g_R= _θL_R(θ)≠ 0. Let g be the direction returned by Algorithm 1. Then every step θ+=θ−αg,0<α<1βR,θ^+=θ-α g, 0<α< 1 _R, (6) strictly decreases the fixed native actor loss: LR(θ+)<LR(θ)L_R(θ^+)<L_R(θ). Thus, for sufficiently small steps, Damper incorporates the temporal gradient while preserving descent on the native actor loss. The proof is in Appendix B.1. 5.2 Temporal Progress at Matched Primary Progress We next compare the temporal progress of DAMPER and fixed-weight scalarization. For this comparison, we rescale each update to give the same predicted decrease in the native actor loss, using a first-order approximation. This holds primary improvement fixed, allowing us to compare the temporal benefit of the update directions independently of their overall scale. For any direction d with gR⊤d>0g_R d>0, define (d):=gT⊤dgR⊤d−gT⊤gR∥gR∥22.G(d):= g_T dg_R d- g_T g_R g_R _2^2. (7) The first term measures the predicted temporal decrease per unit of predicted native-actor-loss decrease; subtracting the corresponding quantity for the native-gradient update gives the additional temporal gain. Using this measure, the next result shows that, for noncollinear gradients and fixed η, DAMPER achieves greater additional temporal gain than scalarization with any fixed positive weight when the temporal gradient is sufficiently small relative to the native actor gradient. Theorem 2 (Temporal gain in the weak-gradient regime). Let gR=∇θLR(θ)g_R= _θL_R(θ) and gT=∇θLT(θ)g_T= _θL_T(θ) be nonzero, and define q:=min∥gT∥2∥gR∥2,1,c:=gR⊤gT∥gR∥2∥gT∥2.q:= \ g_T _2 g_R _2,1 \, c:= g_R g_T g_R _2 g_T _2. Fix η∈[0,1]η∈[0,1] and any λ>0λ>0. Assume 0<q<10<q<1 and λq<1−δλ q<1-δ for some fixed δ∈(0,1)δ∈(0,1). For the analytic Damper direction gηg_η in equation 5 and gλ=gR+λgTg_λ=g_R+λ g_T, the matched temporal gains satisfy (gη)=Θ((1−c2)q2−η),(gλ)=Θ((1−c2)q2).G(g_η)= \! ((1-c^2)q^2-η ), (g_λ)= \! ((1-c^2)q^2 ). (8) Here, Θ denotes two-sided bounds with positive constants independent of q and c, but possibly dependent on λ and δ. The result highlights the role of magnitude control in the weak-temporal regime, where the temporal gradient is small relative to the native actor gradient. For fixed η>0η>0 and λ>0λ>0, DAMPER provides greater additional temporal gain than scalarization and sufficiently small q, when the first-order primary progress is matched. Appendix B.2 gives the proof. 6 Experiments 6.1 Experimental Setup In this subsection, we describe the tasks, baselines, and evaluation protocol used to assess Damper. Tasks and backbones. We consider six continuous-control tasks from Gymnasium [27] and MuJoCo [26]: LunarLander, Pendulum, Reacher, Ant, Hopper, and Walker. Each method is evaluated with both TD3 and SAC. Our implementation and protocol follow the PAVE codebase, and all neural networks use SiLU activations. Baselines. We compare with the unregularized TD3 and SAC agents and five smooth-control methods: CAPS, Grad-CAPS, ASAP, L2C2, and PAVE. The comparison therefore spans policy-output regularization, policy-sensitivity regularization, joint policy–value constraints, and critic-gradient stabilization. Evaluation protocol. Each method is trained with five independent random seeds. For each seed, the learned policy is evaluated for ten episodes; tables report the mean and standard deviation across seeds. We measure cumulative episodic return (rere, higher is better) and action oscillation (smsm, lower is better). The latter is a spectral smoothness metric computed using the fast Fourier transform (FFT), following [20, 3]. 6.2 Overall Performance In this subsection, we show that Damper consistently reduces action oscillation across TD3 and SAC, with task-dependent effects on return. It lowers smsm relative to the unregularized agents in all 12 task–backbone pairs and achieves the lowest score among the compared methods in eight (Tables 1 and 2). Figure 2 also shows smaller consecutive action changes on Walker for both backbones. Figure 2: Damper reduces consecutive action variation on Walker under both actor–critic backbones. We plot ∥at−at−1∥2 a_t-a_t-1 _2 over a 1,000-step rollout. Table 1: TD3 results. Entries are mean (standard deviation) across five seeds. Higher return (rere) and lower oscillation score (smsm) are better. The best value in each column is highlighted in bold. LunarLander Pendulum Reacher Ant Hopper Walker Method re↑re\! sm↓sm\! re↑re\! sm↓sm\! re↑re\! sm↓sm\! re↑re\! sm↓sm\! re↑re\! sm↓sm\! re↑re\! sm↓sm\! TD3 205.5 [-0.15em](98.5) 1.815 [-0.15em](0.955) −-157.6 [-0.15em](84.0) 2.308 [-0.15em](1.422) −3.34-3.34 [-0.15em](1.45) 0.050 [-0.15em](0.015) 4749 [-0.15em](1226) 2.044 [-0.15em](0.307) 3604 [-0.15em](150) 3.061 [-0.15em](0.421) 4596 [-0.15em](328) 1.937 [-0.15em](0.139) CAPS 246.1 [-0.15em](81.6) 0.634 [-0.15em](0.233) −-159.4 [-0.15em](81.7) 0.454 [-0.15em](0.140) −3.34-3.34 [-0.15em](1.47) 0.046 [-0.15em](0.015) 5370 [-0.15em](275) 2.064 [-0.15em](0.101) 2928 [-0.15em](1025) 1.730 [-0.15em](0.341) 5032 [-0.15em](241) 1.865 [-0.15em](0.176) Grad-CAPS 225.5 [-0.15em](61.4) 0.905 [-0.15em](0.333) −153.4-153.4 [-0.15em](77.9) 1.011 [-0.15em](0.503) −3.34-3.34 [-0.15em](1.49) 0.040 [-0.15em](0.011) 4201 [-0.15em](1859) 1.713 [-0.15em](0.408) 3371 [-0.15em](619) 1.287 [-0.15em](0.128) 5046 [-0.15em](460) 1.412 [-0.15em](0.291) ASAP 245.5 [-0.15em](58.6) 1.345 [-0.15em](0.616) −-155.3 [-0.15em](78.3) 2.013 [-0.15em](1.047) −-3.36 [-0.15em](1.44) 0.048 [-0.15em](0.015) 4925 [-0.15em](1210) 2.041 [-0.15em](0.343) 2581 [-0.15em](1448) 1.866 [-0.15em](0.883) 4880 [-0.15em](745) 1.615 [-0.15em](0.288) L2C2 218.5 [-0.15em](83.3) 1.626 [-0.15em](0.727) −-156.0 [-0.15em](79.1) 3.293 [-0.15em](1.034) −-3.38 [-0.15em](1.52) 0.048 [-0.15em](0.014) 4078 [-0.15em](1658) 1.974 [-0.15em](0.402) 2807 [-0.15em](1371) 2.427 [-0.15em](1.087) 4960 [-0.15em](1048) 1.707 [-0.15em](0.320) PAVE 238.1 [-0.15em](71.5) 0.841 [-0.15em](0.601) −-154.5 [-0.15em](80.4) 0.410 [-0.15em](0.121) −-3.55 [-0.15em](1.51) 0.046 [-0.15em](0.016) 5064 [-0.15em](1245) 1.865 [-0.15em](0.237) 3328 [-0.15em](541) 1.022 [-0.15em](0.148) 4996 [-0.15em](700) 1.991 [-0.15em](0.410) DAMPER (ours) 266.4 [-0.15em](35.8) 0.313 [-0.15em](0.036) −-154.6 [-0.15em](78.7) 0.463 [-0.15em](0.131) −-3.63 [-0.15em](1.44) 0.048 [-0.15em](0.016) 5281 [-0.15em](887) 1.261 [-0.15em](0.147) 3078 [-0.15em](599) 0.240 [-0.15em](0.035) 4275 [-0.15em](322) 0.346 [-0.15em](0.088) TD3 results. Damper achieves the lowest oscillation score on LunarLander, Ant, Hopper, and Walker (Table 1). On LunarLander, it improves both metrics, increasing return from 205.5205.5 to 266.4266.4 and reducing smsm from 1.8151.815 to 0.3130.313. Hopper and Walker also show large reductions in smsm, from 3.0613.061 to 0.2400.240 and from 1.9371.937 to 0.3460.346, respectively, accompanied by lower returns than unregularized TD3. Table 2: SAC results. Entries are mean (standard deviation) across five seeds. Higher return (rere) and lower oscillation score (smsm) are better. The best value in each column is highlighted in bold. LunarLander Pendulum Reacher Ant Hopper Walker Method re↑re\! sm↓sm\! re↑re\! sm↓sm\! re↑re\! sm↓sm\! re↑re\! sm↓sm\! re↑re\! sm↓sm\! re↑re\! sm↓sm\! SAC 162.6 [-0.15em](139.9) 0.402 [-0.15em](0.143) −-149.9 [-0.15em](76.9) 0.479 [-0.15em](0.128) −-3.49 [-0.15em](1.43) 0.053 [-0.15em](0.016) 5001 [-0.15em](1145) 1.935 [-0.15em](0.234) 3059 [-0.15em](825) 0.708 [-0.15em](0.074) 4771 [-0.15em](292) 0.747 [-0.15em](0.081) CAPS 120.0 [-0.15em](171.6) 0.279 [-0.15em](0.065) −-152.7 [-0.15em](80.1) 0.325 [-0.15em](0.107) −-3.48 [-0.15em](1.39) 0.048 [-0.15em](0.014) 5439 [-0.15em](707) 1.931 [-0.15em](0.170) 3222 [-0.15em](548) 0.653 [-0.15em](0.058) 4955 [-0.15em](296) 0.745 [-0.15em](0.164) Grad-CAPS 269.0 [-0.15em](27.6) 0.277 [-0.15em](0.064) −148.9-148.9 [-0.15em](75.7) 0.379 [-0.15em](0.119) −-3.51 [-0.15em](1.41) 0.046 [-0.15em](0.013) 5675 [-0.15em](960) 1.797 [-0.15em](0.248) 3144 [-0.15em](653) 0.494 [-0.15em](0.061) 5060 [-0.15em](262) 0.585 [-0.15em](0.067) ASAP 270.1 [-0.15em](19.6) 0.303 [-0.15em](0.080) −-149.2 [-0.15em](76.0) 0.505 [-0.15em](0.138) −-3.46 [-0.15em](1.40) 0.053 [-0.15em](0.016) 4564 [-0.15em](2104) 1.727 [-0.15em](0.326) 2831 [-0.15em](908) 0.637 [-0.15em](0.104) 5035 [-0.15em](408) 0.760 [-0.15em](0.155) L2C2 239.4 [-0.15em](58.2) 0.407 [-0.15em](0.168) −-149.3 [-0.15em](76.2) 0.510 [-0.15em](0.145) −3.45-3.45 [-0.15em](1.38) 0.052 [-0.15em](0.016) 4947 [-0.15em](1565) 1.810 [-0.15em](0.415) 3101 [-0.15em](580) 0.732 [-0.15em](0.106) 4834 [-0.15em](277) 0.901 [-0.15em](0.221) PAVE 263.6 [-0.15em](24.2) 0.203 [-0.15em](0.034) −-151.6 [-0.15em](78.7) 0.334 [-0.15em](0.114) −-3.51 [-0.15em](1.49) 0.051 [-0.15em](0.015) 5543 [-0.15em](1007) 1.741 [-0.15em](0.218) 2403 [-0.15em](862) 0.584 [-0.15em](0.097) 4544 [-0.15em](646) 0.713 [-0.15em](0.090) DAMPER (ours) 281.5 [-0.15em](24.4) 0.102 [-0.15em](0.037) −-156.6 [-0.15em](78.5) 0.336 [-0.15em](0.164) −-3.57 [-0.15em](1.57) 0.038 [-0.15em](0.018) 4564 [-0.15em](1577) 1.207 [-0.15em](0.253) 3384 [-0.15em](273) 0.571 [-0.15em](0.022) 4402 [-0.15em](1086) 0.268 [-0.15em](0.044) SAC results. Damper achieves the lowest oscillation score on LunarLander, Reacher, Ant, and Walker (Table 2). It also obtains the highest return on LunarLander and Hopper; on Hopper, smsm decreases from 0.7080.708 to 0.5710.571 relative to unregularized SAC. On Ant and Walker, the lowest oscillation scores are accompanied by lower returns than the unregularized agent, again showing that the return–smoothness balance depends on the task. 6.3 Ablation Studies Projection Ablation. We show that conflict-conditioned projection achieves the highest mean return among the evaluated update rules on three TD3 tasks: LunarLander, Hopper, and Walker. We compare Damper with Unconditional Sum, which always adds the scaled temporal direction, and Unconditional Projection, which orthogonalizes it at every update. The former retains components that oppose the native actor gradient, while the latter removes aligned components as well. Table 3: Ablation of conflict-conditioned projection with TD3. Entries are mean (standard deviation); each variant uses five training seeds and ten evaluation episodes per seed. The numerically best mean for each metric is shown in bold. LunarLander Hopper Walker Method re↑re\! sm↓sm\! re↑re\! sm↓sm\! re↑re\! sm↓sm\! DAMPER 266.4 (35.8) 0.313 (0.036) 3078 (599) 0.240 (0.035) 4275 (322) 0.346 (0.088) Unconditional projection 206.9 (122.7) 0.370 (0.223) 3050 (587) 0.249 (0.040) 3525 (1576) 0.314 (0.072) Unconditional sum 263.5 (68.8) 0.279 (0.113) 317 (341) 0.053 (0.106) 3575 (514) 0.268 (0.088) On Hopper, unconditional summation reduces mean return from 30783078 to 317317, despite achieving a lower oscillation score (Table 3). This result highlights the need to assess smoothness together with task performance. Unconditional projection yields a similar Hopper return but lowers Walker return from 42754275 to 35253525, with greater variability. On LunarLander, unconditional summation remains close to Damper in mean return, whereas unconditional projection has a lower mean and greater variability. These results support resolving conflicting components while retaining aligned temporal information. Table 4: PEGrad-based baseline with TD3. Mean (standard deviation); five training seeds and ten evaluation episodes per seed. Task re↑re\! sm↓sm\! LunarLander 198.8 (123.5) 0.619 (0.353) Hopper 2760 (835) 0.636 (0.046) Walker 4361 (176) 0.712 (0.142) Comparison with a PEGrad-Based Baseline. We compare Damper with a PEGrad-based baseline [21]. This baseline doesn’t use conditional projection and limits the temporal gradient’s magnitude, but lacks a parameter for strengthening weak temporal gradients relative to the native actor gradient. Damper can amplify weak temporal gradients through wη=q1−ηw_η=q^1-η. Across the three tasks, Damper lowers the mean oscillation score by 4949–62%62\% (Tables 3 and 4), while achieving higher mean returns on LunarLander and Hopper and a slightly lower mean return on Walker. These results are consistent with the benefit of magnitude interpolation alongside conflict-conditioned projection. 6.4 Comparison with Fixed-Weight Scalarization We evaluate fixed-weight scalarization on all six TD3 tasks. For each of the 20 weights λ∈Λ:=0.05,0.10,…,1.0λ∈ :=\0.05,0.10,…,1.0\, we train agents using ℒλ(θ):=LR(θ)+λLT(θ),λ∈Λ,L_λ(θ):=L_R(θ)+λ L_T(θ), λ∈ , (9) with λ fixed throughout training. Figure 3 shows the Hopper comparison with Damper at η=1η=1. Figure 3: Fixed-weight scalarization on TD3 Hopper. Points show means for λ=0.05,0.10,…,1.0λ=0.05,0.10,…,1.0; dashed lines mark Damper. Across the six tasks, Damper achieves a lower mean oscillation score than every evaluated fixed weight on five tasks, with Reacher as the exception. On Hopper, even the smoothest scalarized policy, obtained at λ=0.8λ=0.8, has sm=0.714sm=0.714, compared with 0.2400.240 for Damper. These results are consistent with the temporal-progress analysis in Theorem 2. Tuned scalarization nevertheless achieves competitive returns and lower oscillation than several specialized smoothness baselines. On LunarLander, Ant, and Hopper, selected weights improve both mean metrics over multiple such baselines, suggesting limited additional benefit from their more elaborate mechanisms over a well-tuned single temporal loss in these settings (Appendix C). 6.5 Effect of the Interpolation Parameter Figure 4: Effect of η on task-normalized action oscillation with TD3. We sweep η∈ℋ=0,0.2,…,1.0η =\0,0.2,…,1.0\ on LunarLander, Hopper, and Walker using TD3. For each task e, we normalize the mean oscillation score over the sweep as sm~e(η)=sme(η)−meMe−me, sm_e(η)= sm_e(η)-m_eM_e-m_e, (10) where mem_e and MeM_e are the minimum and maximum mean scores over ℋH, respectively. Zero therefore denotes the smoothest tested setting for each task; absolute scores remain task-dependent. Figure 4 shows that oscillation decreases monotonically on LunarLander and Hopper, while Walker exhibits a temporary increase at η=0.6η=0.6. All three tasks attain their lowest mean score at η=1η=1. This trend is consistent with Theorem 2: increasing η makes (gη)G(g_η) nondecreasing at matched first-order primary progress. 7 Conclusion We introduced Damper, a primary–auxiliary actor update that combines conflict-conditioned temporal-gradient projection with norm-ratio magnitude control. The update leaves the native actor gradient unchanged, admits a local descent guarantee for a fixed critic and minibatch, and reduces the reported action-oscillation score across all 12 evaluated task–backbone pairs. Return is not uniformly preserved, however, and our theoretical analysis is limited to a single actor update rather than the full training process. Evaluation on physical systems is an important next step for understanding when smoother control can be obtained without sacrificing task performance. AI Use Statement In this work, we used generative AI tools to assist with writing and coding. For writing, AI tools were used to edit and polish author-written text for grammar, clarity, and readability, and to help format LaTeX (e.g., tables and references). For coding, AI tools were used to assist in creating and editing software code, including debugging, refactoring, and writing experiment and plotting scripts. Generative AI tools were also used to assist with checking and refining mathematical derivations and proofs, including identifying gaps and assessing consistency between theorem statements, assumptions, and supporting arguments. These tools served as supplementary aids, with the authors retaining responsibility for the validity of all theoretical claims and proofs. AI tools were not used to originate research hypotheses or design the research methodology or experiments. Synthetic dataset generation, translation, dataset cleaning, and qualitative data analysis are not applicable to this work. We have reviewed all AI-assisted work. All AI-edited text was checked by the authors to ensure that it accurately reflects our intended meaning and claims. All AI-assisted code was reviewed, tested, and verified for correctness by the authors before being used to produce the reported results. We take responsibility for the final content of this work, including text, claims, and artifacts produced with the aid of generative AI. References References [1] Z. Chen, V. Badrinarayanan, C. Lee, and A. Rabinovich (2018) GradNorm: gradient normalization for adaptive loss balancing in deep multitask networks. In Proceedings of the 35th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 80, p. 794–803. External Links: Link Cited by: §2.2. [2] E. Chisari, A. Liniger, A. Rupenyan, L. Van Gool, and J. Lygeros (2021) Learning from simulation, racing in reality. In 2021 IEEE International Conference on Robotics and Automation (ICRA), p. 8046–8052. External Links: Document Cited by: §2.1. [3] G. Christmann, Y. Luo, H. Mandala, and W. Chen (2024) Benchmarking smoothness and reducing high-frequency oscillations in continuous control policies. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 627–634. External Links: Document Cited by: §6.1. [4] B. De Cooman, J. A. K. Suykens, and A. Ortseifen (2021) Improving temporal smoothness of deterministic reinforcement learning policies with continuous actions. In Proceedings of BNAIC/BENELEARN 2021, p. 217–240. External Links: Link Cited by: §2.1. [5] R. de la Presilla, S. Wandel, M. Stammler, M. Grebe, G. Poll, and S. Glavatskih (2023) Oscillating rolling element bearings: a review of tribotesting and analysis approaches. Tribology International 188, p. 108805. External Links: ISSN 0301-679X, Document Cited by: §1. [6] J. Désidéri (2012) Multiple-gradient descent algorithm (MGDA) for multiobjective optimization. Comptes Rendus Mathématique 350 (5–6), p. 313–318. External Links: Document Cited by: §2.2. [7] Y. Du, W. M. Czarnecki, S. M. Jayakumar, M. Farajtabar, R. Pascanu, and B. Lakshminarayanan (2018) Adapting auxiliary losses using gradient similarity. arXiv preprint arXiv:1812.02224. Cited by: §2.2. [8] S. Fujimoto, H. van Hoof, and D. Meger (2018) Addressing function approximation error in actor-critic methods. In Proceedings of the 35th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 80, p. 1587–1596. External Links: Link Cited by: Appendix E. [9] C. Gong, X. Liu, and Q. Liu (2021) Automatic and harmless regularization with constrained and lexicographic optimization: a dynamic barrier approach. In Advances in Neural Information Processing Systems, Vol. 34, p. 29630–29642. Cited by: §2.2. [10] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine (2018) Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 80, p. 1861–1870. External Links: Link Cited by: Appendix E. [11] Y. He, X. Feng, C. Cheng, G. Ji, Y. Guo, and J. Caverlee (2022) MetaBalance: improving multi-task recommendations via adapting gradient magnitudes of auxiliary tasks. In Proceedings of the ACM Web Conference 2022, p. 2205–2215. External Links: Document Cited by: §2.2. [12] Y. Hsieh, J. Thornton, E. Ndiaye, M. Klein, M. Cuturi, and P. Ablin (2024) Careful with that scalpel: improving gradient surgery with an EMA. In Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 235, p. 19085–19100. External Links: Link Cited by: §2.2. [13] W. Kim and K. Sycara (2025) Fair cooperation in mixed-motive games via conflict-aware gradient adjustment. arXiv preprint arXiv:2508.17696. Cited by: §2.2. [14] T. Kobayashi (2022) L2C2: locally lipschitz continuous constraint towards stable and smooth reinforcement learning. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 4032–4039. External Links: Document Cited by: §1, §2.1. [15] K. Kwak and H. Hwang (2026) Enhancing control policy smoothness by aligning actions with predictions from preceding states. Proceedings of the AAAI Conference on Artificial Intelligence 40 (27), p. 22707–22715. External Links: Document Cited by: §1, §2.1. [16] I. Lee, H. Cao, C. Dao, Y. Chen, and I. Wu (2024) Gradient-based regularization for action smoothness in robotic control with reinforcement learning. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 603–610. External Links: Document Cited by: §1, §2.1. [17] J. W. Lee, K. Kwak, D. Kim, and H. Hwang (2026) Stabilizing the Q-gradient field for policy smoothness in actor-critic. arXiv preprint arXiv:2601.22970. Cited by: Appendix D, Table 7, §1, §2.1. [18] B. Liu, X. Liu, X. Jin, P. Stone, and Q. Liu (2021) Conflict-averse gradient descent for multi-task learning. In Advances in Neural Information Processing Systems, Vol. 34, p. 18878–18890. Cited by: §2.2. [19] H. Munn, B. Tidd, P. Böhm, M. Gallagher, and D. Howard (2026) Scalable multi-objective robot reinforcement learning through gradient conflict resolution. In 2026 IEEE International Conference on Robotics and Automation (ICRA), Cited by: §2.2. [20] S. Mysore, B. Mabsout, R. Mancuso, and K. Saenko (2021) Regularizing action policies for smooth control with reinforcement learning. In 2021 IEEE International Conference on Robotics and Automation (ICRA), p. 1810–1816. External Links: Document Cited by: §1, §2.1, §6.1. [21] S. Peri, A. Perincherry, B. Pandit, and S. Lee (2025) Non-conflicting energy minimization in reinforcement learning based robot control. In Proceedings of the 9th Conference on Robot Learning, Proceedings of Machine Learning Research, Vol. 305, p. 221–237. External Links: Link Cited by: §2.2, §6.3. [22] R. Qiu, R. Wang, G. Yang, X. Li, and Z. Shao (2026) LPPG-RL: lexicographically projected policy gradient reinforcement learning with subproblem exploration. Proceedings of the AAAI Conference on Artificial Intelligence 40 (30), p. 25009–25017. External Links: Document Cited by: §2.2. [23] A. Raffin, J. Kober, and F. Stulp (2022) Smooth exploration for robotic reinforcement learning. In Proceedings of the 5th Conference on Robot Learning, Proceedings of Machine Learning Research, Vol. 164, p. 1634–1644. External Links: Link Cited by: §2.1. [24] O. Sener and V. Koltun (2018) Multi-task learning as multi-objective optimization. In Advances in Neural Information Processing Systems, Vol. 31. Cited by: §2.2. [25] X. Song, J. Duan, W. Wang, S. E. Li, C. Chen, B. Cheng, B. Zhang, J. Wei, and X. S. Wang (2023) LipsNet: a smooth and robust neural network with adaptive lipschitz constant for high accuracy optimal control. In Proceedings of the 40th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 202, p. 32253–32272. External Links: Link Cited by: §2.1. [26] E. Todorov, T. Erez, and Y. Tassa (2012) MuJoCo: a physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, p. 5026–5033. External Links: Document Cited by: §6.1. [27] M. Towers, A. Kwiatkowski, J. Terry, J. U. Balis, G. De Cola, T. Deleu, M. Goulão, A. Kallinteris, M. Krimmel, A. KG, R. Perez-Vicente, A. Pierré, S. Schulhoff, J. J. Tai, H. Tan, and O. G. Younis (2024) Gymnasium: a standard interface for reinforcement learning environments. arXiv preprint arXiv:2407.17032. Cited by: §6.1. [28] T. Yu, S. Kumar, A. Gupta, S. Levine, K. Hausman, and C. Finn (2020) Gradient surgery for multi-task learning. In Advances in Neural Information Processing Systems, Vol. 33, p. 5824–5836. Cited by: §2.2. Appendix A Optimization Characterization of the Combined Gradient The following lemma establishes the closed-form solution to the optimization problem in Section 4.2. Lemma 1 (Closest primary-preserving combined gradient). Let gR,gT∈ℝpg_R,g_T ^p be nonzero gradients, with nR:=∥gR∥2n_R:= g_R _2 and nT:=∥gT∥2n_T:= g_T _2. For any s>0s>0, define the target combined gradient gtarget(s):=gR+sgTnT.g_target(s):=g_R+s g_Tn_T. Then the optimization problem g(s):=argmind∈ℝp:gR⊤d≥nR2‖d−gtarget(s)‖22g(s):= d ^p:\,g_R d≥ n_R^2arg\,min d-g_target(s) _2^2 (11) has the unique solution g(s)=gR+snTg~T,g~T:=gT−mingR⊤gT,0nR2gR.g(s)=g_R+ sn_T g_T, g_T:=g_T- \g_R g_T,0\n_R^2g_R. (12) Here, g~T g_T is the conflict-conditioned projection defined in Section 4.1. Proof. Write t:=gtarget(s)t:=g_target(s) and b:=gR⊤gTb:=g_R g_T. If b≥0b≥ 0, then gR⊤t=nR2+snTb≥nR2.g_R t=n_R^2+ sn_Tb≥ n_R^2. Thus, t is feasible and attains an objective value of zero, making it the unique minimizer. Since g~T=gT g_T=g_T in this case, the claimed solution follows. If b<0b<0, define d⋆:=t−sbnTnR2gR.d :=t- sbn_Tn_R^2g_R. Then gR⊤d⋆=nR2g_R d =n_R^2, so d⋆d is feasible. Moreover, d⋆−td -t is a positive multiple of gRg_R. For any feasible d, we have gR⊤(d−d⋆)≥0g_R (d-d )≥ 0, and hence ∥d−t∥22−∥d⋆−t∥22 d-t _2^2- d -t _2^2 =∥d−d⋆∥22+2(d−d⋆)⊤(d⋆−t) = d-d _2^2+2(d-d ) (d -t) ≥∥d−d⋆∥22. ≥ d-d _2^2. The right-hand side is strictly positive unless d=d⋆d=d . Therefore, d⋆d is the unique minimizer. Substituting the definition of t gives d⋆=gR+snT(gT−bnR2gR)=gR+snTg~T,d =g_R+ sn_T (g_T- bn_R^2g_R )=g_R+ sn_T g_T, as required. ∎ Connection to magnitude control. The cap-only and norm-matched target magnitudes are s0:=min(nT,nR)s_0:= (n_T,n_R) and s1:=nRs_1:=n_R, respectively. For η∈[0,1]η∈[0,1], define sη:=s0 1−ηs1η,q:=min(nTnR,1),wη:=q 1−η.s_η:=s_0^\,1-ηs_1^\,η, q:= \! ( n_Tn_R,1 ), w_η:=q^\,1-η. Since s0=nRqs_0=n_Rq and s1=nRs_1=n_R, we obtain sη=nRwηs_η=n_Rw_η. Lemma 1 therefore yields gη=g(sη)=gR+sηnTg~T=gR+nRnTwηg~T,g_η=g(s_η)=g_R+ s_ηn_T g_T=g_R+ n_Rn_Tw_η g_T, (13) recovering the norm-ratio combination rule. Because ∥g~T∥2≤nT g_T _2≤ n_T and sη≤nRs_η≤ n_R, the temporal contribution satisfies ∥gη−gR∥2=sη∥g~T∥2nT≤sη≤nR. g_η-g_R _2=s_η g_T _2n_T≤ s_η≤ n_R. (14) Thus, sηs_η specifies the temporal magnitude before projection, not necessarily the magnitude of the final temporal contribution. Under conflict, ∥g~T∥2<nT g_T _2<n_T, so this contribution is strictly smaller than sηs_η. Using the original norm nTn_T in the denominator preserves this attenuation rather than amplifying the projected gradient back to the target magnitude. Appendix B Proofs for the Primary–Auxiliary Analysis B.1 Proof of Theorem 1 Proof. Write r:=gRr:=g_R and nR:=∥r∥2>0n_R:= r _2>0. If Algorithm 1 activates its numerical guard, then g=rg=r. The descent lemma gives LR(θ−αr)≤LR(θ)−α∥r∥22+βRα22∥r∥22=LR(θ)−α(1−βRα2)nR2. splitL_R(θ-α r)&≤ L_R(θ)-α r _2^2+ _Rα^22 r _2^2\\ &=L_R(θ)-α (1- _Rα2 )n_R^2. split (15) This is strictly smaller than LR(θ)L_R(θ) whenever 0<α<2/βR0<α<2/ _R, and hence under the stated condition 0<α<1/βR0<α<1/ _R. Otherwise, g=gηg=g_η. Define uR:=gR/nRu_R:=g_R/n_R and u~T:=g~T/nT u_T:= g_T/n_T, where nT:=∥gT∥2n_T:= g_T _2. Let c+:=maxc,0,c−:=max−c,0.c_+:= \c,0\, c_-:= \-c,0\. (16) From equation 4, uR⊤u~T=c+,∥u~T∥22=1−c−2.u_R u_T=c_+, u_T _2^2=1-c_-^2. (17) Using r=nRuRr=n_Ru_R and equation 5, r⊤gη r g_η =nR2uR⊤(uR+wηu~T) =n_R^2u_R (u_R+w_η u_T ) =nR2(1+wηc+)>0. =n_R^2 (1+w_ηc_+ )>0. (18) Moreover, ∥gη∥22 g_η _2^2 =nR2‖uR+wηu~T‖22 =n_R^2 u_R+w_η u_T _2^2 =nR2[1+2wηc++wη2(1−c−2)]. =n_R^2 [1+2w_ηc_++w_η^2(1-c_-^2) ]. (19) The descent lemma therefore yields LR(θ−αgη)−LR(θ)≤ L_R(θ-α g_η)-L_R(θ)≤ −αnR2(1+wηc+) -α n_R^2(1+w_ηc_+) +βRα2nR22[1+2wηc++wη2(1−c−2)]. + _Rα^2n_R^22 [1+2w_ηc_++w_η^2(1-c_-^2) ]. (20) The right-hand side is strictly negative when 0<α<2(1+wηc+)βR[1+2wηc++wη2(1−c−2)].0<α< 2(1+w_ηc_+) _R[1+2w_ηc_++w_η^2(1-c_-^2)]. (21) Because the numerical guard is inactive, q∈(0,1]q∈(0,1], and hence wη=q1−η∈(0,1]w_η=q^1-η∈(0,1]. Consequently, 1+2wηc++wη2(1−c−2) 1+2w_ηc_++w_η^2(1-c_-^2) ≤2+2wηc+ ≤ 2+2w_ηc_+ =2(1+wηc+). =2(1+w_ηc_+). (22) The upper bound in equation 21 is therefore at least 1/βR1/ _R, proving the theorem. ∎ B.2 Proof of Theorem 2 Matched-progress interpretation. For gR⊤d>0g_R d>0, define d^:=∥gR∥22gR⊤dd. d:= g_R _2^2g_R d\,d. Then gR⊤d^=∥gR∥22g_R d= g_R _2^2, so d d and gRg_R yield the same first-order decrease in the native actor loss. As α→0α→ 0, a Taylor expansion gives LT(θ−αgR)−LT(θ−αd^) L_T(θ-α g_R)-L_T(θ-α d) (23) =αgT⊤(d^−gR)+o(α) =α g_T ( d-g_R)+o(α) =α∥gR∥22[gT⊤dgR⊤d−gT⊤gR∥gR∥22]+o(α) =α g_R _2^2 [ g_T dg_R d- g_T g_R g_R _2^2 ]+o(α) =α∥gR∥22(d)+o(α). =α g_R _2^2G(d)+o(α). Hence, (d)G(d) measures the additional first-order temporal decrease per unit primary decrease relative to the native-gradient update. Proof. Write w:=q1−ηw:=q^1-η and c+:=maxc,0c_+:= \c,0\. Since 0<q<10<q<1, the cap is inactive, q=∥gT∥2/∥gR∥2q= g_T _2/ g_R _2, and 0<w≤10<w≤ 1. The projection rule gives gη=(1−wminc,0)gR+q−ηgT.g_η= (1-w \c,0\ )g_R+q^-ηg_T. Consequently, gR⊤gη=∥gR∥22(1+wc+)>0.g_R g_η= g_R _2^2(1+wc_+)>0. Substitution into equation 7 yields (gη)=q2−η(1−c2)1+wc+.G(g_η)= q^2-η(1-c^2)1+wc_+. (24) For scalarization, gλ=gR+λgTg_λ=g_R+λ g_T, and gR⊤gλ=∥gR∥22(1+λqc).g_R g_λ= g_R _2^2(1+λ qc). Since λq<1−δλ q<1-δ and |c|≤1 c ≤ 1, δ<1+λqc<2−δ.δ<1+λ qc<2-δ. (25) Thus, the scalarized direction also has positive primary progress. Direct substitution gives (gλ) (g_λ) =qc+λq1+λqc−qc =q c+λ q1+λ qc-qc (26) =λq2(1−c2)1+λqc. = λ q^2(1-c^2)1+λ qc. Since 1≤1+wc+≤21≤ 1+wc_+≤ 2, equation 24 gives 12(1−c2)q2−η≤(gη)≤(1−c2)q2−η. 12(1-c^2)q^2-η (g_η)≤(1-c^2)q^2-η. (27) Similarly, combining equation 26 with equation 25 yields λ2−δ(1−c2)q2≤(gλ)≤λδ(1−c2)q2. λ2-δ(1-c^2)q^2 (g_λ)≤ λδ(1-c^2)q^2. (28) Because λ>0λ>0 and δ∈(0,1)δ∈(0,1) are fixed, these bounds establish equation 8, with positive constants independent of q and c. They also cover |c|=1 c =1, where both gains vanish. Finally, if |c|<1 c <1, both gains are positive, and their common factor 1−c21-c^2 cancels: (gη)(gλ)=q−ηλ1+λqc1+wc+. G(g_η)G(g_λ)= q^-ηλ 1+λ qc1+wc_+. (29) Combining 1≤1+wc+≤21≤ 1+wc_+≤ 2 with equation 25 gives δ2λq−η≤(gη)(gλ)≤2−δλq−η. δ2λq^-η≤ G(g_η)G(g_λ)≤ 2-δλq^-η. (30) Thus, (gη)(gλ)=Θ(q−η), G(g_η)G(g_λ)= (q^-η), with constants independent of q and c. For every fixed η>0η>0 and λ>0λ>0, the lower bound diverges as q→0q→ 0, establishing the claimed asymptotic advantage for noncollinear gradients. ∎ Appendix C Additional Fixed-Weight Scalarization Results We report the five TD3 task sweeps omitted from Section 6.4: LunarLander, Pendulum, Reacher, Ant, and Walker. Each sweep uses the 20 fixed weights Λ=0.05,0.10,…,1.0 =\0.05,0.10,…,1.0\, with five training seeds and ten evaluation episodes per seed. Points show mean return and FFT-based oscillation score; dashed lines show the reported TD3 means of Damper. The highlighted point has the lowest mean oscillation score within each sweep. Axis ranges vary by task. Figure 5: Fixed-weight scalarization on TD3 LunarLander. The lowest scalarization score is sm=0.681sm=0.681 at λ=0.80λ=0.80, compared with 0.3130.313 for Damper. Figure 6: Fixed-weight scalarization on TD3 Pendulum. The lowest scalarization score is sm=0.825sm=0.825 at λ=0.85λ=0.85, compared with 0.4630.463 for Damper. Figure 7: Fixed-weight scalarization on TD3 Reacher. Reacher is the exception: scalarization reaches sm=0.0325sm=0.0325 at λ=0.95λ=0.95, below the reported 0.0480.048 for Damper. Figure 8: Fixed-weight scalarization on TD3 Ant. The lowest scalarization score is sm=1.327sm=1.327 at λ=0.90λ=0.90, compared with 1.2611.261 for Damper. Figure 9: Fixed-weight scalarization on TD3 Walker. The lowest scalarization score is sm=0.674sm=0.674 at λ=0.95λ=0.95, compared with 0.3460.346 for Damper. Comparison with smoothing baselines. We compare tuned scalarization with the five smoothing baselines in Table 1, excluding Damper. On Ant, the weight λ=0.75λ=0.75 gives (re,sm)=(5486,1.574)(re,sm)=(5486,1.574). Its return exceeds the highest baseline return (53705370, CAPS), while its oscillation score is 8.1%8.1\% below the lowest baseline score (1.7131.713, Grad-CAPS). Thus, it improves both metrics over all five baselines. On LunarLander, λ=0.80λ=0.80 gives (259.8,0.681)(259.8,0.681), improving both metrics over Grad-CAPS, ASAP, L2C2, and PAVE; PAVE, for example, obtains (238.1,0.841)(238.1,0.841). On Hopper, λ=0.80λ=0.80 gives (3338,0.714)(3338,0.714), improving both metrics over CAPS, ASAP, L2C2, and PAVE. Its oscillation score is 30.1%30.1\% below the best baseline score (1.0221.022, PAVE), with a similar return (33383338 versus 33283328). On Pendulum, λ=0.85λ=0.85 gives (−154.6,0.825)(-154.6,0.825), improving on both ASAP (−155.3,2.013-155.3,2.013) and L2C2 (−156.0,3.293-156.0,3.293) in both metrics. These gains are task dependent. CAPS remains smoother on LunarLander, and CAPS and PAVE remain smoother on Pendulum. The lowest scalarization scores on Reacher (0.03250.0325) and Walker (0.6740.674) improve on all five baselines, but their returns are lower: −3.79-3.79 versus −3.55-3.55 to −3.34-3.34 on Reacher, and 42934293 versus 48804880 to 50465046 on Walker. These results make a tuned temporal-consistency loss a strong baseline: additional smoothing mechanisms do not consistently improve the return–smoothness trade-off in these experiments. Comparisons use reported means and weights selected from the sweep; they do not establish statistical significance. Appendix D Experimental Details Baseline configurations. We follow the baseline hyperparameter settings reported in the PAVE appendix [17]. Table 5 lists the settings for CAPS, Grad-CAPS, ASAP, and PAVE from Appendix E.1 (Tables 10 and 11) of that work. Grad-CAPS is denoted as GRAD in the original tables. CAPS, Grad-CAPS, and ASAP use the same settings under TD3 and SAC; PAVE uses backbone-specific loss weights. Table 5: Baseline hyperparameters adopted from PAVE. Settings apply to both TD3 and SAC unless a backbone is specified. Parameter notation follows each baseline. Method Parameter LunarLander Pendulum Reacher Ant Hopper Walker CAPS λT _T 0.1 1.0 0.1 0.1 0.1 0.1 λS _S 0.5 5.0 0.5 0.5 0.5 0.5 σ 0.2 0.2 0.2 0.2 0.2 0.2 Grad-CAPS λT _T 1.0 1.0 1.0 1.0 1.0 1.0 ASAP λT _T 0.005 0.005 0.1 0.05 0.07 0.05 λS _S 0.03 0.03 0.1 0.3 0.3 0.3 λP _P 2.0 2.0 2.0 2.0 2.0 2.0 PAVE (TD3) λ1 _1 0.1 2.0 0.1 0.1 0.1 0.1 λ2 _2 0.1 0.005 0.1 0.005 0.005 0.1 λ3 _3 0.01 2.0 0.01 0.5 0.5 0.01 PAVE (SAC) λ1 _1 0.1 0.1 0.1 0.1 2.0 2.0 λ2 _2 0.5 0.005 0.0005 0.0005 0.0005 0.005 λ3 _3 0.05 0.5 1.0 1.0 3.0 2.0 For PAVE, λ1 _1, λ2 _2, and λ3 _3 weight mixed-partial regularization, vector-field consistency, and curvature preservation, respectively. Its perturbation scale σ=0.01σ=0.01 and curvature floor δ=1.0δ=1.0 are fixed across all tasks and both backbones. L2C2. For both TD3 and SAC, we use σ=1.0σ=1.0, λ¯=0.01 λ=0.01, λ¯=1.0 λ=1.0, and β=0.1β=0.1 on all six tasks. Damper. Table 6 lists the interpolation parameter η used for each task–backbone pair in our main experiments. The parameter sweeps used for ablation are reported separately. Table 6: Interpolation parameter η for Damper. Backbone LunarLander Pendulum Reacher Ant Hopper Walker TD3 0.8 1.0 1.0 0.0 1.0 1.0 SAC 1.0 1.0 0.8 0.0 0.0 1.0 Hardware. Experiments were conducted using NVIDIA GeForce RTX 2080 and RTX 5060 Ti GPUs. Table 7: Training budgets and network sizes following the PAVE protocol [17]. Setting TD3 SAC Training iterations LunarLander 500,000 500,000 Pendulum 100,000 100,000 Reacher 500,000 500,000 Ant 1,000,000 1,000,000 Hopper 1,000,000 1,000,000 Walker2d 1,000,000 1,000,000 Hidden-layer widths (400,300)(400,300) (256,256)(256,256) Appendix E Native Actor Objectives We specify the native actor losses and deterministic action maps used in our TD3 and SAC implementations. Expectations over states use the state marginal of the replay transition distribution D. The critic parameters are held fixed during actor optimization, while gradients pass through the critic’s action input. TD3. For TD3 [8], the deterministic actor μθ _θ is optimized using LRTD3(θ)=−s∼[Qϕ1(s,μθ(s))].L_R^TD3(θ)=-E_s [Q_ _1(s, _θ(s)) ]. (31) The temporal-consistency loss uses μ¯θ(s)=μθ(s) μ_θ(s)= _θ(s). SAC. For SAC [10], let aθ(s,ϵ)a_θ(s,ε) denote a reparameterized action with ϵ∼(0,I)ε (0,I), and let αent _ent denote the entropy temperature. The native actor loss is LRSAC(θ)=s∼,ϵ[αentlogπθ(aθ(s,ϵ)∣s)−minj∈1,2Qϕj(s,aθ(s,ϵ))]. splitL_R^SAC(θ)=E_s ,ε [& _ent _θ(a_θ(s,ε) s)\\ &- _j∈\1,2\Q_ _j(s,a_θ(s,ε)) ]. split (32) The temporal-consistency loss uses the squashed mean action πθ(s)=aθ(s,0) _θ(s)=a_θ(s,0), including any action-bound rescaling used by the policy. Appendix F Computational Cost We measure the per-iteration update time of TD3 and SAC and their smoothness variants on an NVIDIA GeForce RTX 5060 Ti GPU using Hopper-v5 transitions collected under random actions. Each configuration starts from freshly initialized networks and uses a batch size of 256. We perform 50 warmup iterations followed by 1,000 timed iterations. The measured operations include forward computation, loss evaluation, gradient computation, and optimizer steps; logging, diagnostics, and target-network copy operations are excluded. During the timed interval, TD3 performs 500 actor updates and SAC performs 1,000. We therefore report time per update iteration and normalize it to the unregularized baseline within each backbone. Table 8: Average update time on Hopper-v5. Relative time is the ratio to the corresponding unregularized backbone. Lower is better. Each entry averages 1,000 timed update iterations. TD3 SAC Method Time (ms) Relative (×) Time (ms) Relative (×) Unregularized 4.54 1.00 10.87 1.00 CAPS 7.32 1.61 18.78 1.73 Grad-CAPS 6.35 1.40 18.60 1.71 ASAP 9.85 2.17 23.38 2.15 L2C2 12.71 2.80 48.36 4.45 PAVE 24.07 5.30 32.22 2.96 Damper 6.57 1.45 14.26 1.31 Table 8 shows that Damper with η=0η=0 requires 6.57 ms per iteration with TD3 and 14.26 ms with SAC, corresponding to overheads of 44.8% and 31.3%, respectively, over the unregularized backbones. On TD3, Damper is slightly slower than Grad-CAPS but faster than CAPS, ASAP, L2C2, and PAVE. On SAC, it has the lowest measured runtime among the compared smoothness methods. These measurements characterize update computation under the tested configuration and do not represent end-to-end training time.