Paper deep dive
Taming the Adversary: Stable Minimax Deep Deterministic Policy Gradient via Fractional Objectives
Taeho Lee, Donghwan Lee
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 3/22/2026, 6:31:36 AM
Summary
The paper introduces Minimax Deep Deterministic Policy Gradient (MMDDPG), a reinforcement learning framework designed to improve policy robustness in continuous control tasks. By formulating the training as a minimax game between a user policy and an adversarial disturbance policy, and utilizing a novel fractional objective function, the method stabilizes the interaction between agents and mitigates the issue of excessively aggressive disturbances, demonstrating superior performance in MuJoCo environments.
Entities (5)
Relation Signals (3)
MMDDPG → basedon → DDPG
confidence 98% · we incorporate the architecture and training techniques of deep deterministic policy gradient (DDPG)
MMDDPG → improvesrobustnessin → MuJoCo
confidence 95% · Experimental evaluations in MuJoCo environments demonstrate that the proposed MMDDPG achieves significantly improved robustness
MMDDPG → motivatedby → H-infinity Control
confidence 92% · The above problem has been strongly motivated by H∞ control in control theory.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Reinforcement learning (RL) has achieved remarkable success in a wide range of control and decision-making tasks. However, RL agents often exhibit unstable or degraded performance when deployed in environments subject to unexpected external disturbances and model uncertainties. Consequently, ensuring reliable performance under such conditions remains a critical challenge. In this paper, we propose minimax deep deterministic policy gradient (MMDDPG), a framework for learning disturbance-resilient policies in continuous control tasks. The training process is formulated as a minimax optimization problem between a user policy and an adversarial disturbance policy. In this problem, the user learns a robust policy that minimizes the objective function, while the adversary generates disturbances that maximize it. To stabilize this interaction, we introduce a fractional objective that balances task performance and disturbance magnitude. This objective prevents excessively aggressive disturbances and promotes robust learning. Experimental evaluations in MuJoCo environments demonstrate that the proposed MMDDPG achieves significantly improved robustness against both external force perturbations and model parameter variations.
Tags
Links
- Source: https://arxiv.org/abs/2603.12110v1
- Canonical: https://arxiv.org/abs/2603.12110v1
Trouble viewing inline? Open PDF directly →
Full Text
55,508 characters extracted from source content.
Expand or collapse full text
Taming the Adversary: Stable Minimax Deep Deterministic Policy Gradient via Fractional Objectives Taeho Lee and Donghwan Lee This work was supported by the Institute of Information Communications Technology Planning Evaluation (IITP) funded by the Korea government under Grant 2022-0-00469 and the BK21 FOUR from the Ministry of Education (Republic of Korea).Taeho Lee and Donghwan Lee are with the School of Electrical Engineering, Korea Advanced Institute of Science and Technology, , OH 45435, USA eho0228,@kaist.ac.kr,donghwan,@kaist.ac.kr Abstract Reinforcement learning (RL) has achieved remarkable success in a wide range of control and decision-making tasks. However, RL agents often exhibit unstable or degraded performance when deployed in environments subject to unexpected external disturbances and model uncertainties. Consequently, ensuring reliable performance under such conditions remains a critical challenge. In this paper, we propose minimax deep deterministic policy gradient (MMDDPG), a framework for learning disturbance-resilient policies in continuous control tasks. The training process is formulated as a minimax optimization problem between a user policy and an adversarial disturbance policy. In this problem, the user learns a robust policy that minimizes the objective function, while the adversary generates disturbances that maximize it. To stabilize this interaction, we introduce a fractional objective that balances task performance and disturbance magnitude. This objective prevents excessively aggressive disturbances and promotes robust learning. Experimental evaluations in MuJoCo environments demonstrate that the proposed MMDDPG achieves significantly improved robustness against both external force perturbations and model parameter variations. I Introduction Deep neural networks have driven major advances in reinforcement learning (RL) because they provide powerful function approximators. With these models, RL agents achieve strong performance in complex, high-dimensional environments such as competitive games [14, 8] and nonlinear control systems [6, 4]. Despite these successes, RL agents remain highly sensitive to external disturbances and model uncertainties [12, 11, 19, 20, 5]. Policies that perform well in nominal training conditions often fail when the environment changes, which can cause unstable behavior or severe performance degradation. In real-world applications, physical systems encounter unmodeled dynamics, parameter variations, sensor noise, and environmental disturbances. Such discrepancies between the training and deployment environments can lead to unstable behavior and severe performance degradation in safety-critical domains such as robotics, autonomous systems, and industrial control. For these reasons, robustness to uncertainty is a central requirement for practical RL-based control. Adversarial RL addresses this problem by introducing a second agent that generates disturbances [12, 11, 5, 16]. This approach models robust policy learning as a two-player zero-sum game between a controller (user) and an adversary. The adversary produces perturbations that challenge the controller, while the controller attempts to maintain performance under these conditions. However, direct minimax training often becomes unstable. The adversary can produce disturbances that are excessively large, and these disturbances can dominate the optimization process. As a result, policy improvement becomes difficult. To overcome these limitations, we propose minimax deep deterministic policy gradient (MMDDPG), a framework for learning disturbance-resilient policies in continuous control tasks. The training process is formulated as a minimax optimization problem between a user policy and an adversarial disturbance policy. To stabilize this interaction, we introduce a fractional objective that balances task performance and disturbance magnitude. This objective limits unrealistically large perturbations and still allows the adversary to challenge the controller effectively. We evaluate the proposed method on MuJoCo continuous control benchmarks [17]. Experimental results show that MMDDPG achieves substantially improved robustness to external force disturbances and resilience to parametric mismatches induced by variations in actuator-related parameters compared with conventional RL baselines. I Related Works Robust reinforcement learning (RRL) [9] explicitly incorporates robustness against model inaccuracies and external disturbances to improve the reliability of reinforcement learning algorithms. Early RRL methods formulate the control problem as a differential game inspired by the H∞H_∞ control theory [1]. In this formulation, the user (controller) minimizes a cost function under worst-case disturbances. While these approaches provide strong robustness properties for nonlinear control tasks, they are typically restricted to low-dimensional systems due to the computational intractability of solving the associated Hamilton-Jacobi-Isaacs equations in high dimensions. Building on these foundations, recent works extend RRL to deep reinforcement learning (DRL) frameworks to improve robustness in high-dimensional continuous control problems [11, 16, 5, 20, 7]. Notably, robust adversarial reinforcement learning (RARL) formulates policy learning as a two-player zero-sum game between the user and the adversary that applies disturbances [11]. Despite its success, training stability remains a significant hurdle. In many cases, the adversary converges faster than the protagonist, which leads to overly aggressive disturbances that destabilize the learning process. To mitigate training instability, several works incorporate stability constraints derived from robust control. For example, Zhai et al. [20] extend dissipativity and L2L_2-gain conditions from the H∞H_∞ control to Markov decision processes (MDP). This approach enforces stability through inequality constraints. Similarly, Long et al. [7] integrate the H∞H_∞-inspired constraints to regulate the interaction between the policy and the disturbance generator. In a related direction, Lee and Lee [5] incorporate the H∞H_∞ control principles into reward shaping to jointly train both the user policy and the adversarial disturbance policy. Although effective, these constraint-based methods introduce additional computational overhead and require delicate hyperparameter tuning. Another line of research focuses on action-robust formulations, such as action-robust Markov decision process (AR-MDP), where the adversary directly perturbs or replaces the agent’s actions [16, 3]. While effective against certain types of action uncertainty, these approaches are less suited for handling persistent external disturbances that affect system dynamics. In contrast to prior methods, our work introduces a novel objective function that incorporates robustness directly into the learning problem. This objective enables the agent to account for disturbance effects without relying on explicit stability constraints or action perturbations. Moreover, most existing adversarial robust RL approaches focus on on-policy stochastic algorithms. Our method instead addresses robustness within an off-policy deterministic policy gradient framework. This design improves training stability and sample efficiency in continuous control environments. I Preliminaries I-A Two-player zero-sum Markov game In a two-player zero-sum Markov game (TZMG) [10], with the state space S and the action space, A and W, of a user and an adversary, respectively, the user selects an action a∈a and the adversary selects its action (or disturbance) w∈w simultaneously at the current state s∈s , then the state transits to the next state s′∈s with probability P(s′|s,a,w)P(s |s,a,w), and the transition incurs a cost c(s,a,w,s′)c(s,a,w,s ), where P(s′|s,a,w)P(s |s,a,w) is the state transition probability from the current state s∈s to the next state s′∈s under action a∈a and disturbance w∈w , and c:×→ℝc:S×A×W×S is the cost function. For convenience, we condsider a deterministic cost function and simply write ck+1:=c(sk,ak,wk,sk+1)c_k+1:=c(s_k,a_k,w_k,s_k+1), where k∈0,1,…k∈\0,1,...\ is the time step. Let π:→π:S and μ:→μ:S denote the policies of the user and the adversary, respectively. The objectives of the user and the adversary are to minimize and maximize the cumulative discounted cost over an infinite time horizon respectively, Jπ,μ=[∑k=0∞γkck+1|π,μ]J^π,μ=E [ . _k=0^∞γ^kc_k+1 |π,μ ], where γ∈[0,1)γ∈[0,1) is the discount factor, (s0,a0,w0,s1,a1,w1,…)(s_0,a_0,w_0,s_1,a_1,w_1,...) is a state-action trajectory generated by the Markov chain under policies π and μ, and [⋅|π,μ]E[·|π,μ] is an expectation conditioned on the policies π and μ. Specially, we consider a TZMG with state transition dynamics given by sk+1∼P(sk,ak,wk), s_k+1 P(s_k,a_k,w_k), where sks_k and sk+1∈ℝns_k+1 ^n denote the current and next states, respectively, ak∈ℝma_k ^m is the action selected by the user, and wk∈ℝdw_k ^d represents the disturbance generated by the adversary. At each time step k, both players select their decisions based on the current state sks_k through deterministic policies. Accordingly, we define deterministic state-feedback policies for the user and the adversary as πθ:ℝn→ℝm _θ:R^n ^m and μϕ:ℝn→ℝd _φ:R^n ^d, which are parameterized by θ and ϕφ, respectively, ak=πθ(sk),wk=μϕ(sk). a_k= _θ(s_k), w_k= _φ(s_k). Under these state-feedback policies, the closed-loop system dynamics reduces to a Markov chain governed by sk+1∼P(sk,πθ(sk),μϕ(sk)). s_k+1 P (s_k, _θ(s_k), _φ(s_k) ). Figure 1: Overview of the minimax deep deterministic policy gradient (MMDDPG). Two players, the user and adversarial agents, interact in environment generating the action ata_t and the disturbance wtw_t according to the state sts_t. The action-value function Qψ1(s,a,w)Q_ _1(s,a,w) and Qψ2(s,w)Q_ _2(s,w) are updated by the cost ct+1c_t+1 and wtw_t. The policy of user πθ _θ is updated to minimize the fractional objective function Jπθ,μϕJ _θ, _φ while the policy of adversary μϕ _φ is updated to maximize it. IV Fractional robust objective In TZMG, the user seeks a control policy that minimizes the expected cumulative cost, whereas the adversary aims to maximize it by choosing disturbances. Under deterministic policies πθ _θ and μϕ _φ, we define the primary performance objective as the discounted expected return J1πθ,μϕ:=[∑k=0∞γkck+1|πθ,μϕ]. J_1 _θ, _φ:=E\! [ . _k=0^∞γ^kc_k+1\ | _θ, _φ ]. (1) Accordingly, the robust control problem in TZMG can be formulated as the following minimax optimization problem. Problem 1 Given deterministic policies πθ _θ (user) and μϕ _φ (adversary), find a saddle-point solution to the following minimax optimization problem: minπθmaxμϕJ1πθ,μϕ _ _θ _ _φ\ J_1 _θ, _φ Although the above minimax formulation captures the robust interaction between the user and the adversary, directly optimizing J1πθ,μϕJ_1 _θ, _φ often leads to unstable learning dynamics. Since the adversary seeks to maximize the cumulative cost, it can increase the objective arbitrarily by enlarging the disturbance magnitude in the absence of explicit regularization. This behavior destabilizes learning and prevents convergence to a meaningful saddle point. To mitigate this issue, we introduce an additional objective that quantifies the cumulative squared disturbance norm, J2μϕ:=[∑k=0∞γk‖wk‖22|μϕ]. J_2 _φ:=E\! [ . _k=0^∞γ^k\|w_k\|_2^2\ | _φ ]. (2) By penalizing the disturbance magnitude through J2μϕJ_2 _φ, the adversary is discouraged from generating extreme perturbations. As a result, the training process becomes more stable. To jointly account for task performance and disturbance magnitude, we reformulate the minimax problem using the following fractional objective: Jπθ,μϕ=J1πθ,μϕJ2μϕ=[∑k=0∞γkck+1|πθ,μϕ][∑k=0∞γk∥wk∥22|μϕ]. J _θ, _φ= J_1 _θ, _φJ_2 _φ= E\! [ _k=0^∞γ^kc_k+1\, |\, _θ, _φ ]E\! [ _k=0^∞γ^k\|w_k\|_2^2\, |\, _φ ]. (3) Problem 2 Given deterministic policies πθ _θ (user) and μϕ _φ (adversary), find a saddle-point solution to the following minimax optimization problem: minπθmaxμϕJπθ,μϕ=minπθmaxμϕJ1πθ,μϕJ2μϕ. _ _θ _ _φJ _θ, _φ= _ _θ _ _φ J_1 _θ, _φJ_2 _φ. Remark. The above problem has been strongly motivated by H∞H_∞ control in control theory. By defining the cost as the squared output norm, the objective can be interpreted in a form analogous to the performance criterion used in the H∞H_∞ control [1, 2, 15]. In particular, the H∞H_∞ norm, ‖Tπ‖∞ \|T_π \|_∞, characterizes the worst-case disturbance-to-output gain, ‖Tπ‖∞=supw≠0∑k=0∞‖yk‖22∑k=0∞‖wk‖22. \|T_π \|_∞= _w≠ 0 _k=0^∞\|y_k\|_2^2 _k=0^∞\|w_k\|_2^2. V Actor and critic update To update the actor policies using gradient-based optimization, we need to compute the gradients of the objective Jπθ,μϕJ _θ, _φ with respect to the policy parameters. A direct approach is to differentiate the fractional objective Jπθ,μϕJ _θ, _φ itself. However, the ratio structure complicates the gradient derivation and makes the optimization difficult to analyze and implement in a stable manner. To address this issue, we apply the following logarithmic transformation: minθmaxϕln(J1πθ,μϕJ2μϕ)=minθmaxϕ(lnJ1πθ,μϕ−lnJ2μϕ). _θ _φ ( J_1 _θ, _φJ_2 _φ )= _θ _φ ( J_1 _θ, _φ- J_2 _φ ). Since the logarithm is strictly increasing, maximizing Jπθ,μϕJ _θ, _φ is equivalent to maximizing lnJπθ,μϕ J _θ, _φ. Therefore, the saddle-point solution of the original fractional objective is preserved. Moreover, the logarithmic transformation converts the ratio into a difference of two terms, which simplifies the gradient-based optimization. The logarithmic transformation requires the objectives to remain strictly positive. Therefore, we introduce the following assumption. Assumption. For all admissible policies πθ _θ and μϕ _φ, the objectives J1πθ,μϕJ_1 _θ, _φ and J2μϕJ_2 _φ are strictly positive. This condition holds when the cost and squared disturbance norm are nonnegative. This assumption is satisfied in many practical control and RL environments where the stage cost and the squared disturbance norm are defined as nonnegative quantities, such as quadratic cost functions commonly used in continuous control tasks. Note that the objective J2μϕJ_2 _φ is always nonnegative since it is defined as the cumulative squared disturbance norm. Therefore, the positivity condition mainly concerns the objective J1πθ,μϕJ_1 _θ, _φ in practice. Even when the cost is not strictly positive, the assumption can be enforced through a simple modification by adding a sufficiently large positive constant or introducing a small positive offset. Such transformations preserve the saddle-point structure of the original problem while ensuring that the logarithmic transformation remains well-defined. Under this assumption, we define the following transformed objective: L(θ,ϕ):=lnJ1πθ,μϕ−lnJ2μϕ. L(θ,φ):= J_1 _θ, _φ- J_2 _φ. The gradients of the transformed objective with respect to the user parameter θ and the adversary parameter ϕφ are given by ∇θL(θ,ϕ) _θL(θ,φ) =∇θJ1πθ,μϕJ1πθ,μϕ, = _θJ_1 _θ, _φJ_1 _θ, _φ, (4) ∇ϕL(θ,ϕ) _φL(θ,φ) =∇ϕJ1πθ,μϕJ1πθ,μϕ−∇ϕJ2μϕJ2μϕ. = _φJ_1 _θ, _φJ_1 _θ, _φ- _φJ_2 _φJ_2 _φ. (5) Furthermore, the gradients of J1πθ,μϕJ_1 _θ, _φ and J2μϕJ_2 _φ can be expressed by using the deterministic policy gradient theorem [13] as follows: ∇θJ1πθ,μϕ=s∼ρ[∇θQπ,μϕ(s,πθ(s),μϕ(s))|π=πθ] _θJ_1 _θ, _φ=E_s ρ [ _θQ^π, _φ (s, _θ(s), _φ(s) ) |_π= _θ ] =s∼ρ[∇θπθ(s)∇aQπθ,μϕ(s,a,μϕ(s))|a=πθ(s)], =E_s ρ [ _θ _θ(s)\; _aQ _θ, _φ(s,a, _φ(s)) |_a= _θ(s) ], ∇ϕJ1πθ,μϕ=s∼ρ[∇ϕQπθ,μ(s,πθ(s),μϕ(s))|μ=μϕ] _φJ_1 _θ, _φ=E_s ρ [ _φQ _θ,μ (s, _θ(s), _φ(s) ) |_μ= _φ ] =s∼ρ[∇ϕμϕ(s)∇wQπθ,μϕ(s,πθ(s),w)|w=μϕ(s)], =E_s ρ [ _φ _φ(s)\; _wQ _θ, _φ(s, _θ(s),w) |_w= _φ(s) ], ∇ϕJ2μϕ=s∼ρ[∇ϕQμ(s,μϕ(s))|μ=μϕ] _φJ_2 _φ=E_s ρ [ _φQ^μ (s, _φ(s) ) |_μ= _φ ] =s∼ρ[∇ϕμϕ(s)∇wQμϕ(s,w)|w=μϕ(s)], =E_s ρ [ _φ _φ(s)\; _wQ _φ(s,w) |_w= _φ(s) ], where ρπθ,μϕρ _θ, _φ represents the on-policy distribution induced by the policies πθ _θ and μϕ _φ, and Qπθ,μϕQ _θ, _φ and QμϕQ _φ are the action-value functions. Qπθ,μϕQ _θ, _φ represents the expected cumulative cost starting from a given initial state, action, and disturbance, after which the policies are followed, whereas QμϕQ _φ represents the expected cumulative squared disturbance norm starting from a given initial state and disturbance defined as follows: Qπθ,μϕ(s,a,w)=[∑k=0∞γkck+1|s0=s,a0=a,w0=w,πθ,μϕ],(s,a,w)∈×,Q _θ, _φ(s,a,w)\\ =E [ . _k=0^∞γ^kc_k+1 |s_0=s,a_0=a,w_0=w, _θ, _φ ],\\ (s,a,w) ×A×W, Qμϕ(s,w)=[∑k=0∞γk‖wk‖22|s0=s,w0=w,μϕ],(s,w)∈×.Q _φ(s,w)=E [ . _k=0^∞γ^k||w_k||_2^2 |s_0=s,w_0=w, _φ ],\\ (s,w) ×W. Using these gradient expressions (4) and (5), the user updates its policy parameter θ via gradient descent, while the adversary updates its parameter ϕφ via gradient ascent: θk+1 _k+1 =θk−αuser∇θL(θ,ϕ)|θ=θk, = _k- _user . _θL(θ,φ) |_θ= _k, ϕk+1 _k+1 =ϕk+αadv∇ϕL(θ,ϕ)|ϕ=ϕk, = _k+ _adv . _φL(θ,φ) |_φ= _k, where αuser>0 _user>0 and αadv>0 _adv>0 denote the learning rates for the user and adversary actors, respectively. Under deterministic policies πθ _θ and μϕ _φ, the action-value functions satisfy Bellman equations associated with the cost and disturbance objectives defined as follows: Qπθ,μϕ(s,a,w) Q _θ, _φ(s,a,w) =[c(s,a,w,s′) =E\! [c(s,a,w,s ) . +γQπθ,μϕ(s′,πθ(s′),μϕ(s′))], .+γ\,Q _θ, _φ (s , _θ(s ), _φ(s ) ) ], Qμϕ(s,w) Q _φ(s,w) =[‖w‖22+γQμϕ(s′,μϕ(s′))]. =E\! [\|w\|_2^2+γ\,Q _φ (s , _φ(s ) ) ]. The Bellman equations lead to the following temporal-difference updates: Qk+1πθ,μϕ(sk,ak,wk)= Q _θ, _φ_k+1(s_k,a_k,w_k)= Qkπθ,μϕ(sk,ak,wk) \ Q _θ, _φ_k(s_k,a_k,w_k) +αk(y1−Qkπθ,μϕ(sk,ak,wk)), + _k (y_1-Q _θ, _φ_k(s_k,a_k,w_k) ), (6) Qk+1μϕ(sk,wk)= Q _φ_k+1(s_k,w_k)= Qkμϕ(sk,wk) \ Q _φ_k(s_k,w_k) +αk(y2−Qkμϕ(sk,wk)), + _k (y_2-Q _φ_k(s_k,w_k) ), (7) where αcritic>0 _ critic>0 is the critic learning rate, and y1y_1 and y2y_2 denote the target defined as follows: y1 y_1 =c(sk,ak,wk,sk+1) =c(s_k,a_k,w_k,s_k+1) +(sk+1)Qkπθ,μϕ(sk+1,ak+1,wk+1), + 1(s_k+1)Q _θ, _φ_k(s_k+1,a_k+1,w_k+1), y2 y_2 =‖wk‖22+(sk+1)Qkμϕ(sk+1,wk+1). =\|w_k\|_2^2+ 1(s_k+1)Q _φ_k(s_k+1,w_k+1). Here, (sk+1) 1(s_k+1) is an indicator function defined as (sk+1):=0ifsk+1=terminalstate1else 1(s_k+1):= \ array[]*20c0& if\,\,s_k+1= terminal\,\, state\\ 1& else array . These updates correspond to a SARSA-type temporal-difference learning rule under deterministic policies. VI Minimax deep deterministic policy gradient To implement these actor and critic update in high-dimensional continuous control tasks, we incorporate the architecture and training techniques of deep deterministic policy gradient (DDPG) [6]. In the following, we present the implementation of the above actor–critic updates within the DDPG framework, which leads to the proposed minimax deep deterministic policy gradient (MMDDPG) algorithm. VI-A Actor update Following the DDPG framework [6], we employ neural network critics Qψ1Q_ _1 and Qψ2Q_ _2, parameterized by ψ1 _1 and ψ2 _2, to approximate the action-value functions Qπθ,μϕQ _θ, _φ and QμϕQ _φ, respectively. Consequently, the objectives J1πθ,μϕJ_1 _θ, _φ and J2μϕJ_2 _φ can be expressed as expectations of the corresponding action-value functions: J1πθ,μϕ J_1 _θ, _φ =s0∼ρ0[Qπθ,μϕ(s0,πθ(s0),μϕ(s0))], =E_s_0 _0 [Q _θ, _φ(s_0, _θ(s_0), _φ(s_0)) ], J2μϕ J_2 _φ =s0∼ρ0[Qμϕ(s0,μϕ(s0))], =E_s_0 _0 [Q _φ(s_0, _φ(s_0)) ], where ρ0 _0 denotes the initial state distribution. In practice, these expectations are approximated using mini-batch samples drawn from the replay buffer D. Given a mini-batch B=(s,a,w,s′)B=\(s,a,w,s )\ sampled uniformly from D, we estimate the expectations of the action-value functions by the following batch means: M(Qψ1) M(Q_ _1) =1|B|∑(s,a,w,s′)∈BQψ1(s,a,w), = 1|B| _(s,a,w,s )∈ BQ_ _1(s,a,w), M(Qψ2) M(Q_ _2) =1|B|∑(s,a,w,s′)∈BQψ2(s,w), = 1|B| _(s,a,w,s )∈ BQ_ _2(s,w), where |B||B| denotes the mini-batch size. These batch means provide empirical estimates of the expectations in J1πθ,μϕJ_1 _θ, _φ and J2μϕJ_2 _φ. Substituting these estimates into the logarithmic gradients in (4) and (5) yields the following mini-batch approximations: ∇θL(θ,ϕ)≈1|B|∑(s,a,w,s′)∈B∇θQψ1(s,πθ(s),μϕ(s))M(Qψ1)+ϵ, _θL(θ,φ)≈ 1|B| _(s,a,w,s )∈ B _θQ_ _1 (s, _θ(s), _φ(s) )M(Q_ _1)+ε, (8) ∇ϕL(θ,ϕ)≈1|B|∑(s,a,w,s′)∈B(∇ϕQψ1(s,πθ(s),μϕ(s))M(Qψ1)+ϵ _φL(θ,φ)≈ 1|B| _(s,a,w,s )∈ B ( _φQ_ _1(s, _θ(s), _φ(s))M(Q_ _1)+ε . −∇ϕQψ2(s,μϕ(s))M(Qψ2)+ϵ), .- _φQ_ _2(s, _φ(s))M(Q_ _2)+ε ), (9) where ϵ>0ε>0 is a small constant introduced for numerical stability. This normalization prevents vanishing gradients and unintended sign flipping, and maintains a balanced competition between the user and the adversary when Q-values fluctuate during training. To implement these gradient approximations in practice, we define the following joint actor loss L(θ,ϕ;B)L(θ,φ;B): L(θ,ϕ;B):=1|B|∑s∈B[Qψ1(s,πθ(s),μϕ(s))M(Qψ1)+ϵ−Qψ2(s,μϕ(s))M(Qψ2)+ϵ].L(θ,φ;B):=\\ 1|B| _s∈ B [ Q_ _1(s, _θ(s), _φ(s))M(Q_ _1)+ε- Q_ _2(s, _φ(s))M(Q_ _2)+ε ]. The gradients of this loss correspond to the mini-batch approximations of the logarithmic policy gradients in (8) and (9). Thus, the resulting sampled deterministic policy gradients are ∇θL(θ,ϕ;B) _θL(θ,φ;B) =1|B|∑(s,a,w,s′)∈B∇θQψ1(s,πθ(s),μϕ(s))M(Qψ1)+ϵ, = 1|B| _(s,a,w,s )∈ B _θQ_ _1 (s, _θ(s), _φ(s) )M(Q_ _1)+ε, (10) ∇ϕL(θ,ϕ;B) _φL(θ,φ;B) =1|B|∑(s,a,w,s′)∈B(∇ϕQψ1(s,πθ(s),μϕ(s))M(Qψ1)+ϵ = 1|B| _(s,a,w,s )∈ B ( _φQ_ _1(s, _θ(s), _φ(s))M(Q_ _1)+ε . −∇ϕQψ2(s,μϕ(s))M(Qψ2)+ϵ). - . _φQ_ _2(s, _φ(s))M(Q_ _2)+ε ). (11) The online actor parameters θ and ϕφ are updated using sampled deterministic policy gradients [6, 13]: θ θ ←θ−αuser∇θL(θ,ϕ;B), ←θ- _user _θL(θ,φ;B), ϕ φ ←ϕ+αadv∇ϕL(θ,ϕ;B). ←φ+ _adv _φL(θ,φ;B). After updating the online actors, the target parameters θ′θ and ϕ′φ are softly updated as θ′ θ ←τθ+(1−τ)θ′, ←τθ+(1-τ)θ , ϕ′ φ ←τϕ+(1−τ)ϕ′, ←τφ+(1-τ)φ , where τ∈(0,1)τ∈(0,1) is the interpolation coefficient controlling the update rate. This soft update improves training stability by ensuring slow variation of the target policies. Algorithm 1 Minimax deep deterministic policy gradient (MMDDPG) 1:Initialize the online critic networks Qψ1,Qψ2Q_ _1,Q_ _2 2:Initialize the online actor networks πθ,μϕ _θ, _φ for the user and adversary, respectively. 3:Initialize the target parameters ψ1′←ψ1,ψ2′←ψ2 _1 ← _1, _2 ← _2, θ′←θ,ϕ′←ϕ θ ←θ,φ ←φ 4:Initialize the replay buffer D 5:for Episode i=1,2,…Nepisodei=1,2,...N_episode do 6: Sample the initial state s0∼ρ0s_0 _0 7: for Time step k=0,1,2,…T−1k=0,1,2,...T-1 do 8: The user and adversary select an action and a disturbance 9: ak=πθ(sk)+ξka,wk=μϕ(sk)+ξkwa_k= _θ(s_k)+ _k^a, w_k= _φ(s_k)+ _k^w 10: where ξka,ξkw _k^a, _k^w are OU noise for exploration. 11: Observe next state sk+1s_k+1 12: Compute the cost ck+1:=c(sk,ak,wk,sk+1)c_k+1:=c(s_k,a_k,w_k,s_k+1) 13: Store (sk,ak,wk,ck+1,sk+1)(s_k,a_k,w_k,c_k+1,s_k+1) in D 14: Uniformly sample a mini-batch B from D 15: Update the online critic parameters according to (VI-B) and (13): ψ1 _1 ←ψ1−αcritic∇ψ1L(ψ1;B) ← _1- _ critic _ _1L( _1;B) ψ2 _2 ←ψ2−αcritic∇ψ2L(ψ2;B) ← _2- _ critic _ _2L( _2;B) 16: Update the online actor parameters according to (10) and (11): θ θ ←θ−αuser∇θL(θ,ϕ;B), ←θ- _ user _θL(θ,φ;B), ϕ φ ←ϕ+αadv∇ϕL(θ,ϕ;B) ←φ+ _ adv _φL(θ,φ;B) 17: Soft update target networks: θ′←τθ+(1−τ)θ′,ϕ′←τϕ+(1−τ)ϕ′θ ←τθ+(1-τ)θ , φ ←τφ+(1-τ)φ ψ1′←τψ1+(1−τ)ψ1′,ψ2′←τψ2+(1−τ)ψ2′ _1 ←τ _1+(1-τ) _1 , _2 ←τ _2+(1-τ) _2 18: end for 19:end for Figure 2: Mean and standard deviation of cumulative discounted costs across ten random seeds under random Gaussian disturbances. Error bars indicate one standard deviation. Each row corresponds to a different algorithm: MMDDPG (minimax deep deterministic policy gradient), DDPG (deep deterministic policy gradient) [6], RARL (robust adversarial reinforcement learning) [11], PR-DDPG (probabilistic action-robust DDPG), and NR-DDPG (noisy action-robust DDPG) [16]. While other baseline methods exhibit increased cost and variance as task complexity grows, MMDDPG consistently achieves the lowest average cost with minimal variance across both environments. VI-B Critic update To implement the temporal-difference updates derived in the previous section within a deep RL framework, we follow the approach of DDPG [6]. As mentioned earlier, the action-value functions Qπθ,μϕQ _θ, _φ and QμϕQ _φ are approximated by neural network critics Qψ1Q_ _1 and Qψ2Q_ _2, respectively, and trained using mini-batch samples drawn from the replay buffer D. In this setting, the TD updates in (V) and (7) are implemented using sampled transitions. This results in a mini-batch SARSA-style temporal-difference learning rule. Given a mini-batch of transitions B=(s,a,w,s′)B=\(s,a,w,s )\ sampled from the replay buffer D, the critic losses are defined as the mean squared TD errors L(ψ1;B) L( _1;B) =121|B|∑(s,a,w,s′)∈B(y1−Qψ1(s,a,w))2, = 12 1|B| _(s,a,w,s )∈ B (y_1-Q_ _1(s,a,w) )^2, L(ψ2;B) L( _2;B) =121|B|∑(s,a,w,s′)∈B(y2−Qψ2(s,w))2, = 12 1|B| _(s,a,w,s )∈ B (y_2-Q_ _2(s,w) )^2, where the target are defined as follows: y1 y_1 =c(s,a,w,s′)+γQψ1′(s′,πθ′(s′),μϕ′(s′)), =c(s,a,w,s )+γ\,Q_ _1 \! (s , _θ (s ), _φ (s ) ), y2 y_2 =‖w‖22+γQψ2′(s′,μϕ′(s′)). =\|w\|_2^2+γ\,Q_ _2 \! (s , _φ (s ) ). The online critic parameters are updated via gradient descent: ψ1 _1 ←ψ1−αcritic∇ψ1L(ψ1;B), ← _1- _critic _ _1L( _1;B), ψ2 _2 ←ψ2−αcritic∇ψ2L(ψ2;B), ← _2- _critic _ _2L( _2;B), where αcritic>0 _critic>0 is the learning rate, and the gradients of the critic losses are given by ∇ψ1L(ψ1;B)= _ _1L( _1;B)= 1|B|∑(s,a,w,s′)∈B(y1−Qψ1(s,a,w))∇ψ1Qψ1(s,a,w), 1|B| _(s,a,w,s )∈ B (y_1-Q_ _1(s,a,w) ) _ _1Q_ _1 (s,a,w ), (12) ∇ψ2L(ψ2;B)= _ _2L( _2;B)= 1|B|∑(s,a,w,s′)∈B(y2−Qψ2(s,w))∇ψ2Qψ2(s,w). 1|B| _(s,a,w,s )∈ B (y_2-Q_ _2(s,w) ) _ _2Q_ _2 (s,w ). (13) After updating the online critics, the target critic parameters are softly updated as ψ1′ _1 ←τψ1+(1−τ)ψ1′, ←τ _1+(1-τ) _1 , ψ2′ _2 ←τψ2+(1−τ)ψ2′, ←τ _2+(1-τ) _2 , where τ∈(0,1)τ∈(0,1) is the interpolation coefficient. VI-C Exploration To enable exploration, we perturb both the user and the adversary policies using temporally correlated Ornstein–Uhlenbeck (OU) noise [18], following the standard DDPG approach [6]: ak a_k =πθ(sk)+ξka, = _θ(s_k)+ _k^a, wk w_k =μϕ(sk)+ξkw, = _φ(s_k)+ _k^w, where the noise terms ξka _k^a and ξkw _k^w evolve according to ξk+1=ξk+θξ(μξ−ξk)Δt+σξΔtϵk,ϵk∼(0,1). _k+1= _k+ _ξ( _ξ- _k) t+ _ξ t\, _k, _k (0,1). where θξ _ξ, μξ _ξ, and σξ _ξ control the rate of mean reversion, the long-term mean, and the noise scale, respectively. The overall algorithm is described at Algorithm 1. Figure 3: Performance heatmaps under model parameter uncertainties in Reacher (top) and Pusher (bottom) environments. The x-axis and y-axis represent the gear scale and joint damping scale, respectively. Darker colors indicate lower mean discounted costs. Each row corresponds to a different algorithm: MMDDPG (minimax deep deterministic policy gradient), DDPG (deep deterministic policy gradient) [6], RARL (robust adversarial reinforcement learning) [11], PR-DDPG (probabilistic action-robust DDPG), and NR-DDPG (noisy action-robust DDPG) [16]. MMDDPG maintains a consistently low-cost region across the entire parameter grid, demonstrating superior robustness to parametric mismatches compared to adversarial and action-robust baselines. VII Experiment and results VII-A Experiment setup We conduct two sets of experiments to evaluate the robustness of the proposed algorithm in two MuJoCo environments [17], Reacher and Pusher. These environments are selected because their cost functions can be reformulated as minimization objectives that are coercive with respect to disturbances. This coercive property ensures that the cost increases sharply as the state deviates under external perturbations. As a result, it provides a principled foundation for learning policies that effectively suppress disturbances. Detailed descriptions of the environments are presented in Appendix A-A. VII-A1 Robustness against external disturbances The first set of experiments assesses the agent’s resilience to external perturbations. The disturbance w is modeled as a stochastic process sampled from a Gaussian distribution, (μ,σ2)N(μ,σ^2), which enables evaluation under both biased and random perturbations. • In Reacher, disturbances are applied to the fingertip along the x- and y-axes to perturb the end-effector during target-reaching. • In Pusher, three-dimensional disturbances are applied to the robotic arm, affecting both the reaching trajectory and the interaction with the object. VII-A2 Robustness to model parameter variations The second set of experiments evaluates the resilience of the learned policies to parametric uncertainties. In both the Reacher and Pusher environments, we simulate model mismatches by perturbing the actuator-related parameters of the robotic arm. Specifically, we vary the scale of joint damping and gear coefficients, which directly govern the damping forces and the torque transmission of the joints. To assess robustness to actuator uncertainty, we evaluate the policies under diverse parameter scales ranging from under-damped to over-damped regimes. This setting introduces significant deviations from the nominal training dynamics and allows us to examine whether MMDDPG maintains stable control performance. Such analysis is important for real-world deployment, where friction and motor constants are often difficult to identify accurately. We compare MMDDPG with the baseline algorithm DDPG [6] and adversarial RL methods including RARL [11] and action-robust DDPG variants (PR-MDP and NR-MDP) [16, 3]. All algorithms are trained for 100100k steps across ten independent random seeds to ensure statistical significance. Detailed hyperparameters and neural network architectures are provided in Appendix A-B. VII-B Results VII-B1 Robustness to disturbances Figure 2 summarizes the cumulative discounted costs across ten independent seeds under episode-wise constant Gaussian disturbances. In the Reacher environment, both MMDDPG and RARL [11] exhibit competitive performance by achieving low average costs with minimal variance. This suggests that in relatively simple, short-horizon tasks, standard adversarial minimax formulations can effectively capture the necessary robustness. However, DDPG and action-robust variants (PR-DDPG and NR-DDPG) show higher sensitivity to disturbances, as evidenced by their larger error bars. A more distinct performance gap emerges in the more complex Pusher environment. As task complexity and the temporal horizon increase, RARL suffers from higher average costs and significantly larger variances. This performance degradation indicates that strict minimax optimization may induce overly aggressive adversarial interactions in high-dimensional spaces, which leads to unstable learning trajectories. In contrast, MMDDPG consistently achieves the lowest average cost and variance in Pusher. This improvement is primarily attributed to the proposed fractional objective function, which provides a principled balance between nominal performance and disturbance attenuation. Unlike strict worst-case optimization, the fractional formulation enables smoother trade-offs and results in more stable policy updates. Action-robust methods, such as PR-DDPG and NR-DDPG [16], perform notably worse, particularly in the Pusher task. These methods primarily address action-space stochasticity rather than the dynamic uncertainty caused by persistent external disturbances. Furthermore, while these baselines require sensitive hyperparameter tuning (e.g., the noise parameter α [16]), MMDDPG embeds robustness directly into the objective level. This design eliminates the need for delicate parameter adjustment and results in more consistent performance across different disturbance realizations and random seeds. Overall, these results indicate that robustness induced through objective-level design, as in MMDDPG, is more effective and scalable than adversarially aggressive or noise-based action-robust formulations, especially in complex environments subject to external disturbances. VII-B2 Robustness to model uncertainty Figure 3 presents the performance of each algorithm across a grid of variations in gear and damping scales for the Reacher and Pusher environments. The heatmaps illustrate the sensitivity of each policy to actuator-related model mismatches, where darker (blue) regions represent lower cumulative discounted costs. Across all parameter configurations, MMDDPG exhibits consistently low costs with a remarkably smooth performance profile. This consistency indicates that the proposed policy does not merely overfit to the nominal training environment but generalizes effectively across a wide range of actuator dynamics. In contrast, other algorithms—particularly in the Pusher environment—show pronounced performance fluctuations as the gear and damping scales vary; this trend reflects high sensitivity to variations in actuator parameters. While RARL can perform competitively in simpler settings, its strict min–max formulation leads to unstable learning in complex environments such as Pusher. Furthermore, action-robust baselines relying on noise injection fail to cope with persistent disturbances and structural parameter variations. These results highlight that objective-level robustness is more effective and scalable than adversarially aggressive or noise-based action perturbations. VIII Conclusion This paper proposes minimax deep deterministic policy gradient (MMDDPG), an adversarial reinforcement learning framework that integrates the fractional objective function with deep deterministic policy gradients (DDPG). The proposed framework directly addresses robustness challenges arising from external disturbances and model uncertainties by formulating the learning process as a principled min–max optimization problem. Comprehensive evaluations in MuJoCo environments demonstrate that MMDDPG achieves consistently low cumulative costs and superior learning stability under both stochastic disturbances and actuator-related parametric uncertainties. Compared to baseline algorithms, MMDDPG exhibits significantly reduced performance sensitivity and improved generalization across a wide range of operating conditions. These results suggest that incorporating robustness directly at the objective level through a fractional formulation provides an effective and scalable approach to robust policy learning. Future work will explore extensions of this framework to real-world robotic systems, as well as its applicability to broader classes of nonlinear uncertainties and multi-agent settings. References [1] T. Başar and P. Bernhard (2008) H-infinity optimal control and related minimax design problems: a dynamic game approach. Springer Science & Business Media, Boston. Cited by: §I, §IV. [2] T. Basar (1989) A dynamic games approach to controller design: disturbance rejection in discrete time. In Proceedings of the 28th IEEE Conference on Decision and Control,, p. 407–414. Cited by: §IV. [3] A. M. Deshpande, A. A. Minai, and M. Kumar (2021) Robust deep reinforcement learning for quadcopter control. IFAC-PapersOnLine 54 (20), p. 90–95. Cited by: §I, §VII-A2. [4] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, et al. (2018) Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on Robot Learning, p. 651–673. Cited by: §I. [5] T. Lee and D. Lee (2025) Robust deterministic policy gradient for disturbance attenuation and its application to quadrotor control. arXiv preprint arXiv:2502.21057. Cited by: §I, §I, §I, §I. [6] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra (2019) Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971. Cited by: §I, Figure 2, Figure 3, §VI-A, §VI-A, §VI-B, §VI-C, §VI, §VII-A2. [7] J. Long, W. Yu, Q. Li, Z. Wang, D. Lin, and J. Pang (2024) Learning H-infinity locomotion control. arXiv preprint arXiv:2404.14405. Cited by: §I, §I. [8] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. (2015) Human-level control through deep reinforcement learning. Nature 518 (7540), p. 529–533. Cited by: §I. [9] J. Morimoto and K. Doya (2005) Robust reinforcement learning. Neural computation 17 (2), p. 335–359. Cited by: §I. [10] J. Perolat, B. Scherrer, B. Piot, and O. Pietquin (2015) Approximate dynamic programming for two-player zero-sum markov games. In International Conference on Machine Learning, p. 1321–1329. Cited by: §I-A. [11] L. Pinto, J. Davidson, R. Sukthankar, and A. Gupta (2017) Robust adversarial reinforcement learning. In International Conference on Machine Learning, p. 2817–2826. Cited by: §I, §I, §I, Figure 2, Figure 3, §VII-A2, §VII-B1. [12] L. Schott, J. Delas, H. Hajri, E. Gherbi, R. Yaich, N. Boulahia-Cuppens, F. Cuppens, and S. Lamprier (2024) Robust deep reinforcement learning through adversarial attacks and training: a survey. arXiv preprint arXiv:2403.00420. Cited by: §I, §I. [13] D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. Riedmiller (2014) Deterministic policy gradient algorithms. In International Conference on Machine Learning, p. 387–395. Cited by: §V, §VI-A. [14] D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, et al. (2017) Mastering the game of go without human knowledge. Nature 550 (7676), p. 354–359. Cited by: §I. [15] A. A. Stoorvogel and A. J. Weeren (2002) The discrete-time riccati equation related to the h/sub/spl infin//control problem. IEEE Transactions on Automatic Control 39 (3), p. 686–691. Cited by: §IV. [16] C. Tessler, Y. Efroni, and S. Mannor (2019) Action robust reinforcement learning and applications in continuous control. In International Conference on Machine Learning, p. 6215–6224. Cited by: §I, §I, §I, Figure 2, Figure 3, §VII-A2, §VII-B1. [17] E. Todorov, T. Erez, and Y. Tassa (2012) Mujoco: a physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, p. 5026–5033. Cited by: §I, §VII-A. [18] G. E. Uhlenbeck and L. S. Ornstein (1930) On the theory of the brownian motion. Physical review 36 (5), p. 823. Cited by: §VI-C. [19] E. Vinitsky, Y. Du, K. Parvate, K. Jang, P. Abbeel, and A. Bayen (2020) Robust reinforcement learning using adversarial populations. arXiv preprint arXiv:2008.01825. Cited by: §I. [20] P. Zhai, J. Luo, Z. Dong, L. Zhang, S. Wang, and D. Yang (2022) Robust adversarial reinforcement learning with dissipation inequation constraint. In Proceedings of the AAAI Conference on Artificial Intelligence, p. 5431–5439. Cited by: §I, §I, §I. Appendix A Experiments details A-A Environments Figure 4: Experiment environments. Left: Reacher, Right Psuher (a) Reacher The Reacher is a planar robotic arm with two rotational joints that operates in a two-dimensional workspace. The task is to control the arm so that its end-effector reaches a randomly placed target position. The environment provides an 11-dimensional state space, which includes joint angles, joint angular velocities, and the relative position of the target. The action space is 2-dimensional and continuous, corresponding to the torques applied at the shoulder and elbow joints. (b) Pusher The Pusher environment consists of a planar robotic arm tasked with pushing a cylindrical object to a designated target location on a table. The environment provides a 23-dimensional state space, which includes the positions and velocities of the robot joints, the object, and the target. The action space is 7-dimensional and continuous, corresponding to the torques applied to the robot arm joints. TABLE I: Details about state, action, disturbance spaces Reacher (b) Pusher dim() (S) 1717 2323 dim() (A) 66 33 Action range [−1,1][-1,1] [−1,1][-1,1] dim() (W) 22 33 Disturbance range [−1,1][-1,1] [−0.5,0.5][-0.5,0.5] Disturbance body name tips_arm r_wrist_roll_link A-B Algorithm details The hyperparameters of algorithms used in experiments are described in Table I. The overall descriptions of DDPG and RARL are desribed in Algorithm 2 and 3 TABLE I: Details about the hyperparameters of each algorithms Value Number of steps for training 100100K Buffer size 11M Learning rate for critic αcritic _ critic 0.0010.001 Learning rate for actor of user αuser _ user 0.00010.0001 Learning rate for actor of adversary αadv _ adv 0.00010.0001 Hidden layer sizes [256,256][256,256] Soft update interpolation coefficient τ 0.0050.005 Batch size |B||B| 128128 Activation function in actor network tanh Optimizer Adam Discounted factor γ 0.990.99 Policy noise OU noise Mean of OU noise μξ _ξ 0 Standard deviation of OU noise σξ _ξ 0.20.2 Algorithm 2 Deterministic deep policy gradient (DDPG) 1:Initialize the online critic networks QψQ_ψ 2:Initialize the actor networks πθ _θ for the user. 3:Initialize the target parameters ψ′←ψ,θ′←θψ ←ψ,θ ←θ 4:Initialize the replay buffer D 5:for Episode i=1,2,…Niteri=1,2,...N_iter do 6: Observe the initial state s0s_0 7: for Time step k=0,1,2,…T−1k=0,1,2,...T-1 do 8: User selects actions ak=πθ(sk)+ξkaa_k= _θ(s_k)+ _k^a 9: where ξka _k^a is OU noise for exploration. 10: Observe the next state sk+1s_k+1 11: Compute the cost ck+1:=c(sk,ak,sk+1)c_k+1:=c(s_k,a_k,s_k+1) 12: Store the tuple (sk,ak,ck+1,sk+1)(s_k,a_k,c_k+1,s_k+1) in D 13: Uniformly sample a mini-batch B from D 14: Update critic network: ψ←ψ−αcritic∇ψL(ψ;B) ψ←ψ- _ critic _ψL(ψ;B) 15: Update actor networks by the DPG: θ←θ−αuser∇θL(θ;B) θ←θ- _ user _θL(θ;B) 16: Soft update target parameters ψ′ψ and θ′θ 17: end for 18:end for where L(ψ;B)L(ψ;B) and L(θ;B)L(θ;B) defined as follows L(ψ;B) L(ψ;B) :=1|B|∑(s,a,c,s′)∈B(c+γQψ′(s′,πθ′(s′))−Qψ(s,a))2 := 1|B| _(s,a,c,s )∈ B(c+γ Q_ψ (s , _θ (s ))-Q_ψ(s,a))^2 L(θ;B):=1|B|∑(s,a,c,s′)∈BQψ(s,πθ(s)) L(θ;B):= 1|B| _(s,a,c,s )∈ BQ_ψ(s, _θ(s)) Algorithm 3 Robust adversarial reinforcement learning (RARL) 1:Initialize the online critic networks Qψ1,Qψ2Q_ _1,Q_ _2 2:Initialize the actor networks πθ,μϕ _θ, _φ for the user and adversary. 3:Initialize the target parameters ψ1′←ψ1,ψ2′←ψ2 _1 ← _1, _2 ← _2, θ′←θ,ϕ′←ϕ θ ←θ,φ ←φ 4:Initialize the replay buffer D 5:for Episode i=1,2,…Niteri=1,2,...N_iter do 6: Observe the initial state s0s_0 7: for Time step k=0,1,2,…T−1k=0,1,2,...T-1 do 8: User selects an action 9: and adversary selects an disturbance 10: ak=πθ(sk)+ξka,wk=μϕ(sk)+ξkwa_k= _θ(s_k)+ _k^a, w_k= _φ(s_k)+ _k^w 11: where ξka,ξkw _k^a, _k^w are OU noise for exploration. 12: Observe the next state sk+1s_k+1 13: Compute the cost ck+1:=c(sk,ak,wk,sk+1)c_k+1:=c(s_k,a_k,w_k,s_k+1) 14: Store the tuple (sk,ak,wk,ck+1,sk+1)(s_k,a_k,w_k,c_k+1,s_k+1) in D 15: Uniformly sample a mini-batch B from D 16: Update critic network: ψi←ψi−αcritic∇ψiL(ψi;B),i∈1,2 _i← _i- _ critic _ _iL( _i;B), i∈\1,2\ 17: Update actor networks by the DPG: θ θ ←θ−αuser∇θL(θ;B) ←θ- _ user _θL(θ;B) ϕ φ ←ϕ+αadv∇ϕL(ϕ;B) ←φ+ _ adv _φL(φ;B) 18: Soft update target networks ψ1′,ψ2′,θ′ψ _1,ψ _2,θ and ϕ′φ : 19: end for 20:end for where two critic loss are defined as L(ψ1;B) L( _1;B) :=1|B|∑(s,a,w,c,s′)∈B(c+γQψ1′(s′,πθ′(s′))−Qψ1(s,a))2 := 1|B| _(s,a,w,c,s )∈ B(c+γ Q_ψ _1(s , _θ (s ))-Q_ _1(s,a))^2 L(ψ2;B) L( _2;B) :=1|B|∑(s,a,w,c,s′)∈B(c+γQψ2′(s′,μϕ′(s′))−Qψ2(s,w))2 := 1|B| _(s,a,w,c,s )∈ B(c+γ Q_ψ _2(s , _φ (s ))-Q_ _2(s,w))^2 and two actor loss are defined as L(θ;B):= L(θ;B):= 1|B|∑(s,a,w,r,s′)∈B[Qψ1(s,πθ(s))] 1|B| _(s,a,w,r,s )∈ B [Q_ _1(s, _θ(s)) ] L(ϕ;B):= L(φ;B):= 1|B|∑(s,a,w,r,s′)∈B[Qψ2(s,μϕ(s))] 1|B| _(s,a,w,r,s )∈ B [Q_ _2(s, _φ(s)) ] A-C Experiments A-C1 Robustness to external disturbances To evaluate robustness against external disturbances, we applied episode-wise constant disturbances to specific links of the robot in the Reacher and Pusher environments. In the Reacher environment, external disturbances were applied to the tips_arm link along the x- and y-axes. For each evaluation episode, a disturbance bias was sampled from a Gaussian distribution whose mean magnitude was selected from 0.0,1.0,2.0,3.0,5.0\0.0,1.0,2.0,3.0,5.0\, and whose standard deviation was chosen from 0.0,0.5,1.0,2.0\0.0,0.5,1.0,2.0\. The sampled disturbance remained constant throughout the episode, modeling persistent but uncertain external perturbations. In the Pusher environment, disturbances were applied to the r_wrist_roll_link along the x-, y-, and z-axes. The disturbance for each episode was sampled from a zero-mean Gaussian distribution with axis-wise standard deviations selected from [0.5,0.5,0.2],[1.0,1.0,0.5],[2.0,2.0,1.0]\[0.5,0.5,0.2],[1.0,1.0,0.5],[2.0,2.0,1.0]\. As in the Reacher experiments, the disturbance was held constant for the duration of each episode. For both environments, each trained policy was evaluated over 500500 independent test episodes for each disturbance configuration. This evaluation protocol assesses the robustness of the learned policies under persistent external disturbances acting throughout an entire episode. A-C2 Robustness to model parameter uncertainty To assess robustness with respect to model parameter uncertainty, we varied both the gear ratio and the joint damping of the actuators in the Reacher and Pusher environments. Specifically, the damping scale was selected from 0.2,0.5,1.0,2.0,5.0\0.2,0.5,1.0,2.0,5.0\,while the gear scale was chosen from 0.5,0.8,1.0,1.2,1.5\0.5,0.8,1.0,1.2,1.5\. This resulted in a total of 2525 distinct parameter configurations. For each configuration, the trained policies were evaluated over 100100 independent simulation episodes, and the average cumulative discounted cost was recorded to assess robustness under actuator-related parameter variations.