Paper deep dive
Turning Safety into Competence: Minimally Exploitable Robot Policies via Safety-Filtered Reinforcement Learning
Ruihan Wu, Rui Yang, Donggeon David Oh, Duy Nguyen, Haimin Hu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 9/24/2026, 5:07:57 AM
Summary
The paper introduces Safety to Competence (S2C), a two-stage reinforcement learning framework that separates safety synthesis from competitive task learning to create minimally exploitable robot policies. By formulating interactions as safety-critical Markov games and using a robust safety filter learned via adversarial RL, S2C ensures that policies remain safe and non-exploitable. The method outperforms existing safe RL baselines in simulated touchdown games and demonstrates robustness in hardware tests against human opponents.
Entities (10)
Relation Signals (6)
S2C → uses → Safety Filter
confidence 95% · S2C learns a robust safety filter via adversarial RL, embeds it in the environment during task policy training...
S2C → reduces → Exploitability
confidence 93% · S2C outperforms eight safe RL baselines, achieving the highest win rate and Elo rating, and the lowest exploitability.
S2C → outperforms → Safe Reinforcement Learning
confidence 92% · In simulated touchdown games, S2C outperforms eight safe RL baselines...
Adversarial RL → trains → Safety Filter
confidence 91% · S2C learns a robust safety filter via adversarial RL...
Safety Filter → enables → Non-Exploitability
confidence 90% · We prove that perfect filtering preserves policy non-exploitability when all players commit to safe maneuvers.
S2C → appliesto → Touchdown Game
confidence 89% · In simulated touchdown games, S2C outperforms eight safe RL baselines...
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Robots deployed for competitive tasks must outmaneuver their opponents without sacrificing safety. Existing approaches, including safe reinforcement learning (RL), train a single policy to achieve task success and avoid failures simultaneously. This coupling can complicate training and leave the learned policy exploitable by deliberate attacks. We propose Safety to Competence (S2C), a two-stage RL framework that separates safety synthesis from competitive task learning. We formulate competitive interactions as safety-critical Markov games and prove that perfect filtering preserves policy non-exploitability when all players commit to safe maneuvers. S2C learns a robust safety filter via adversarial RL, embeds it in the environment during task policy training, and retains the same filter at deployment. In simulated touchdown games, S2C outperforms eight safe RL baselines, achieving the highest win rate and Elo rating, and the lowest exploitability. Hardware stress tests against a human opponent confirm S2C's competence.
Tags
Links
- Source: https://arxiv.org/abs/2609.27312v1
- Canonical: https://arxiv.org/abs/2609.27312v1
Trouble viewing inline? Open PDF directly →
Full Text
48,169 characters extracted from source content.
Expand or collapse full text
Turning Safety into Competence: Minimally Exploitable Robot Policies via Safety-Filtered Reinforcement Learning Ruihan Wu Rui Yang Donggeon David Oh Duy P. Nguyen Haimin Hu Abstract Robots deployed for competitive tasks must outmaneuver their opponents without sacrificing safety. Existing approaches, including safe reinforcement learning (RL), train a single policy to achieve task success and avoid failures simultaneously. This coupling can complicate training and leave the learned policy exploitable by deliberate attacks. We propose Safety to Competence (S2C), a two-stage RL framework that separates safety synthesis from competitive task learning. We formulate competitive interactions as safety-critical Markov games and prove that perfect filtering preserves policy non-exploitability when all players commit to safe maneuvers. S2C learns a robust safety filter via adversarial RL, embeds it in the environment during task policy training, and retains the same filter at deployment. In simulated touchdown games, S2C outperforms eight safe RL baselines, achieving the highest win rate and Elo rating, and the lowest exploitability. Hardware stress tests against a human opponent confirm S2C’s competence. Fig. 1: Our proposed Safety to Competence (S2C) enforces robot safety without sacrificing competitive performance by training multi-agent reinforcement learning (RL) policies with a pretrained robust safety filter embedded in the environment. Here, a quadrupedal robot (orange trajectory) plays a touchdown game against a human opponent (grey trajectory), where the attacker must reach the green touchdown region before timeout; the robot loses immediately if it falls, crosses the red field boundary, or initiates a collision. Top: S2C learns effective attacking and defending tactics. As attacker, it feints past the human to score a touchdown; as defender, it tracks rapid human direction changes and holds the line. Bottom: safe RL baselines fail frequently under competitive opponent strategies: the ET attacker exits the field while turning, the CPO attacker collides with the human, and the ET defender falls while pursuing the human. Project webpage: https://alliance-ai.cs.jhu.edu/s2c/. †footnotetext: 1R. Wu, R. Yang, and H. Hu are with the Department of Computer Science, Johns Hopkins University, Baltimore, MD 21218, USA. rwu63,yrui5@jh.edu, haimin@cs.jhu.edu 2D. D. Oh and D. P. Nguyen are with the Department of Electrical and Computer Engineering, Princeton University, Princeton, NJ 08544, USA. do9948,duyn@princeton.edu ∗Equal contribution. †Equal advising. I Introduction A wide range of robotic applications involves competitive interactions with other agents in close proximity, ranging from drone racing [16] and motorsports [34, 22] to robot soccer [12] and pursuit–evasion games [4]. These tasks require strategic play under strict safety constraints, since a safety violation can cause catastrophic hardware damage or bodily harm and, in these games, result in an immediate loss. Existing approaches for safety-critical, competitive robot motion planning, including a broad class of safe reinforcement learning (RL) algorithms [11], focus on synthesizing a single policy that achieves safety and task competence simultaneously. Common techniques include penalizing safety violations in the reward [30], constraining the expected cumulative safety cost [2, 1], or terminating the episode upon a violation [29]. Coupling the two objectives in this way complicates training and leaves a trade-off between safety and competence that is difficult to tune [30]. Training therefore tends to converge to an overly conservative or overly aggressive strategy. Both are vulnerable to deliberate attack by an exploitative opponent (see, for example, Figs. 1 and 2). Fig. 2: Symmetric touchdown game results. Each cell corresponds to 3900 trials. S2C outperforms all baselines, achieving the highest Elo rating and lowest empirical exploitability. Left: round-robin win-rate matrix, where each cell reports the row method’s win rate against the column method, averaged over all pairs of training seeds. Top right: Elo ratings computed jointly based on all matches (higher is better). Bottom right: empirical exploitability, computed for each seed as the net win rate achieved by its strongest evaluated opponent and then averaged across seeds (lower is better). Safety filters [14] offer a principled approach to separate safety from the task policy. A safety filter supervises the task policy and modifies its proposed action to prevent an upcoming failure. Recent work demonstrates that (adversarial) RL can be used to synthesize such filters for robots that interact with antagonistic opponents and operate in high-dimensional observation and action spaces [33, 21, 27, 24]. Restricting the task policy’s actions with a filter is often seen as sacrificing the robot’s overall performance. Recent work [23] addresses this concern by proving that for a single agent, perfect (i.e., least restrictive) filtering preserves the asymptotic optimal return in safety-constrained RL. However, this guarantee does not carry over directly to competitive multi-agent settings, where each player’s task performance and safety depend on the opponent’s strategy. Contributions. This paper establishes a game-theoretic foundation for safety-critical, competitive robot interaction and develops a practical algorithmic framework for minimizing policy exploitability without sacrificing safety. Our key insight is that separating safety from task objectives in RL can substantially reduce policy exploitability, provided the same safety filter is applied during both training and deployment. Our key contributions are: • We formulate competitive robot interactions as safety-critical Markov games and show for the first time that perfect safety filters yield non-exploitable Nash equilibria within the class of safe policies. • We propose a two-stage S2C framework that first learns a robust safety filter and then embeds it in the RL environment to learn safe and competitive robot policies. • We extensively evaluate S2C in simulated quadruped touchdown games. It achieves the highest win rates and, in the symmetric case, the highest Elo rating and lowest empirical exploitability compared to eight safe RL baselines. Hardware tests further demonstrate S2C’s robustness to human strategies unseen during training. I Related Work Safety Filters. A safety filter supervises a task controller and modifies its proposed action only when necessary to avoid catastrophic failures. Existing filter designs include, for example, model predictive safety filters (MPSFs) [32], control barrier functions (CBFs) [3, 25], and Hamilton–Jacobi (HJ) reachability [5]. Applying these methods to high-dimensional systems with black-box dynamics presents modeling and computational challenges. MPSFs require a predictive dynamics model and online trajectory optimization, which can be computationally expensive. Conventional model-based CBF filters require known dynamics information; constructing such a certificate is challenging for nonlinear dynamics and calls for a case-by-case analysis. Classical grid-based reachability methods scale poorly with the state dimension. Neural approximations have recently enabled filter synthesis for higher-dimensional robotic systems [9, 6]. To account for disturbances that deliberately seek to cause failures, adversarial safety RL methods train a disturbance policy against the controller, jointly learning a safety critic and a fallback policy [15, 33]. Oh et al. [24] use this approach to synthesize a robust Q-control barrier function (Q-CBF), enabling model-free, smooth safety filtering. We adopt this technique in our S2C framework to approximate perfect filters and achieve task-efficient safety overrides. Safe Reinforcement Learning. Existing safe RL methods predominantly incorporate safety into policy learning through reward penalties or cost constraints [11]. A widely adopted safe RL formulation is the constrained Markov decision process (CMDP), which maximizes the expected task payoff subject to a budget on the expected cumulative safety cost. Constrained Policy Optimization (CPO) [1] approximately enforces this budget during policy updates, while Lagrangian methods [30] convert the constraint into an adaptive penalty. Early termination instead ends the episode at a failure event, but does not actively prevent the policy from reaching that condition. These approaches do not generally guarantee safety in competitive interactions. A positive expected-cost budget can still permit rare, yet catastrophic violations, and reward penalties may favor higher task return at the expense of safety on some trajectories. Moreover, the trajectory distribution depends on the opponent’s strategy; a policy that satisfies its cost budget against training opponents may exceed it against an unseen opponent at deployment. In this paper, we advocate for a safety–task separation principle that first trains a robust safety filter and then embeds it in task-policy training. We prove that, under perfect filtering, Nash equilibria of the filtered game induce non-exploitable policies. Empirically, we demonstrate that this approach achieves substantially stronger performance and lower exploitability than the above safe RL baselines. Game-Theoretic Multi-Robot Interaction. Dynamic games model robot interactions where each agent’s outcome depends on both its own policy and others’ responses [7]. A popular model-based game solver, the iterative linear-quadratic game [10], computes local feedback Nash equilibria by linearizing the dynamics and quadratizing the players’ objectives. Lidard et al. [18] extend this approach to incorporate a data-driven reference policy. These approaches generally require known dynamics and differentiable objectives, and their runtime computational cost can limit their applicability to high-dimensional robotic systems. For complex or unknown dynamics, multi-agent reinforcement learning (MARL) instead learns strategic policies through simulated interactions. Centralized-training methods use joint information during training to mitigate the nonstationarity caused by simultaneously learning agents [19], while self-play and population-based methods train against an evolving set of opponents [17]. Independent proximal policy optimization (IPPO) [26] provides a decentralized approach by optimizing policies for each agent using its own observations without requiring a centralized critic or an explicit opponent model. We use IPPO as the task-policy training method in the filtered environment. I Problem Formulation We consider competitive, safety-critical interactions between two robots (or two groups of robots), in which each player seeks to maximize its task performance over time against its opponent; a player loses immediately if it violates a safety constraint. Formally, we model such interactions with the following Markov game. Definition 1 (Safety-Critical Markov Game). A Safety-Critical Markov Game (SC-MG) is defined as the tuple SC:=(,1,2,P,γ,r,ρ,g1,g2,g1,c,g2,c,α), splitG_SC:= (S,A^1,A^2,P,γ;r,ρ;g^1,g^2;g^1,c,g^2,c;α ), split (1) where s=(s1,s2)∈s=(s^1,s^2) is the physical joint state, and iA^i is the action space for player i∈1,2i∈\1,2\, P(⋅∣s,a1,a2)P(· s,a^1,a^2) is the transition kernel, γ∈(0,1)γ∈(0,1) is a discount factor, task reward r:→ℝr:S is bounded, parameter ρ>0ρ>0 specifies the absorbing win–loss reward, and α is the adjudication map, which we explain next. Safety Decomposition. For each player ii, safety is specified by a solo margin function gi(si)g^i(s^i) and a coupled margin gi,c(s1,s2)g^i,c(s^1,s^2), where negative values indicate violations. The total safety margin is gtoti(s):=mingi(si),gi,c(s1,s2),g_tot^i(s):= \g^i(s^i),g^i,c(s^1,s^2) \, and the corresponding failure set is ℱi:=s∈|gtoti(s)<0.F^i:= \s \; |\;g_tot^i(s)<0 \. If s enters ℱ1∪ℱ2F^1 ^2, an adjudicated outcome is triggered and the game is terminated. Fault Attribution. In an SC-MG, the winning condition depends on fault attribution. Conceptually, a player ii wins if the opposing party violates safety. However, it is possible that both players’ safety specifications are violated simultaneously, i.e., s∈ℱ1∩ℱ2s ^1 ^2. Below, we specify how such states are adjudicated. Assumption 1 (Unique Fault Attribution). The adjudication map α:→1,2α:S→\1,2\ identifies the unique player held at fault at each state s∈ℱ1∩ℱ2s ^1 ^2, and α(s)=iα(s)=i whenever s∈ℱi∖ℱ−is ^i ^-i. Running Example: Throughout the paper, we use a symmetric two-player quadruped touchdown game as a running example. The game is played on a 5.2×3.05.2× 3.0 m field, with two Unitree Go2 quadrupedal robots initialized at opposite ends and competing to reach the opponent’s touchdown line first. A player receives a (sparse) reward r=+1r=+1 when it first reaches its specified touchdown line and −1-1 when the opponent does. The solo safety margin gig^i is defined as the minimum distance to the ground and the field boundary, and the coupled margin gi,cg^i,c measures the relative distance to the opponent. The adjudication map α is defined such that, in the event of an inter-robot collision, the robot with the higher velocity projected toward the opponent (i.e., one that closes faster on its opponent) is attributed fault. In order to formally show that SC-MG (1) is zero-sum, we consider an equivalent state space ¯:=(∖(ℱ1∪ℱ2))∪⊤,⊥ S:= (S (F^1 ^2) )∪\ , \, where ⊤ and ⊥ are absorbing states denoting an immediate win and loss for Player 1. Subsequently, we consider the following augmented reward: r¯(s):=+ρ,s=⊤,−ρ,s=⊥,r(s),s∈∖(ℱ1∪ℱ2). r(s):= cases+ρ,&s= ,\\ -ρ,&s= ,\\ r(s),&s (F^1 ^2). cases (2) At each time, player ii receives stage reward σir¯σ^i r, where σ1:=+1σ^1:=+1 and σ2:=−1σ^2:=-1. Consequently, along every realized trajectory, ∑t=0∞γt(σ1r¯(st)+σ2r¯(st))=0 _t=0^∞γ^t (σ^1 r(s_t)+σ^2 r(s_t) )=0. Proposition 1. Under Assumption 1, SC-MG (1) is zero-sum. Exploitability and Game Value. To quantify a policy’s robust competitiveness in this zero-sum game, we next establish the game’s value and introduce exploitability, which measures how vulnerable a robot is to a best-responding, exploitative opponent that deliberately targets its weaknesses. Recall that the SC-MG (1) is a discounted zero-sum stochastic game with bounded stage rewards. Under the standard minimax assumptions [28], the discounted stochastic game guarantees that its max–min and min–max payoffs coincide. We therefore obtain the following proposition. Proposition 2 (Existence of the Game Value). The SC-MG (1) admits a value at every s∈¯s∈ S: V∗(s) V^*(s) =supπ1∈Π1infπ2∈Π2J(π1,π2)(s) = _π^1∈ ^1 _π^2∈ ^2J(π^1,π^2)(s) (3) =infπ2∈Π2supπ1∈Π1J(π1,π2)(s), = _π^2∈ ^2 _π^1∈ ^1J(π^1,π^2)(s), where J(π1,π2)(s):=[∑t=0∞γtr¯(st)|s0=s]J(π^1,π^2)(s):= \! [ _t=0^∞γ^t r(s_t)\, |\,s_0=s ]. Proposition 2 implies that there exists a stationary Nash equilibrium (NE) policy pair (π1,∗,π2,∗)(π^1,*,π^2,*) that guarantees the value V∗V^* against every opponent policy. In this case, an NE policy is non-exploitable: neither player can improve its expected payoff by changing to another policy [7, Ch. 2]. Definition 2 (Exploitability). For a policy πiπ^i, its exploitability is defined as the gap between the game value and the worst-case payoff: expl(πi)(s):=σiV∗(s)−infπ−i∈Π−iσiJ(π1,π2)(s), (π^i)(s):=σ^iV^*(s)- _π^-i∈ ^-iσ^iJ(π^1,π^2)(s), (4) where σ1=+1σ^1=+1 and σ2=−1σ^2=-1. At an NE (π1,∗,π2,∗)(π^1,*,π^2,*) of SC-MG, neither player’s strategy can be exploited, i.e., expl(π1,∗)(s)=expl(π2,∗)(s)=0 (π^1,*)(s)= (π^2,*)(s)=0. Fig. 3: Overview of the proposed S2C training and deployment pipeline, illustrated with the asymmetric touchdown game. IV Analysis: Non-Exploitability under Perfect Safety Filtering In this paper, our goal is to improve robot competitiveness with safe yet minimally exploitable policies. However, direct equilibrium learning in an SC-MG is challenging since it couples task optimization with safety reasoning; each player needs to preempt safety risks induced by close-call interactions with the opponent while searching for winning strategies. To this end, we propose a two-stage policy synthesis framework that separates safety enforcement from competitive task objectives (Fig. 3). First, we synthesize a robust safety filter [14] for each player using HJ-based adversarial RL. Second, we train task policies in an RL environment that incorporates the filters into its transition dynamics, and deploy each policy with the same filter. Because the same filters are used during both training and deployment, the task policy adapts to the filtered game dynamics without requiring an explicit filter model or additional reward terms. In this section, we establish the theoretical foundation for safety–task separation in zero-sum, safety-critical games. Our main result shows that, under perfect filtering, an NE of the filtered game induces an NE of the original SC-MG in the space of safe policies. Consequently, equilibrium policies executed under the same filters remain safe and non-exploitable against safety-aware opponents. We then introduce a practical instantiation of this framework in Sec. V. Let Ωi∗⊆∖ℱi ^i* ^i denote the maximal robust controlled-invariant set [8] for player ii. Inside Ωi∗ ^i*, player ii is guaranteed a strategy to stay within Ωi∗ ^i* regardless of the opponent’s actions. Next, we define the safe action set as safei(s):=ai∣P(Ωi∗∣s,ai,a−i)=1,∀a−i∈−i.A^i_safe(s):= \a^i P( ^i* s,a^i,a^-i)=1,\ ∀ a^-i ^-i \. Definition 3 (Perfect Safety Filter [14]). A (robust) safety filter ϕi:×i→iφ^i:S×A^i ^i is perfect if it satisfies ϕi(s,a)∈safei(s),∀a∈i,∀s∈Ωi,φ^i(s,a) ^i_safe(s),~∀ a ^i,~∀ s∈ ^i, and leaves every safe action unchanged: ϕi(s,a)=a,∀a∈safei(s),∀s∈Ωiφ^i(s,a)=a,~∀ a ^i_safe(s),∀ s∈ ^i. Definition 4 (Filtered Game). Given perfect filters ϕ=(ϕ1,ϕ2)φ=(φ^1,φ^2), we define a filtered game as ϕ:=(¯,1,2,Pϕ,r¯,γ),G_φ:=( S,A^1,A^2,P_φ, r,γ), (5) where Pϕ(⋅∣s,a1,a2):=P¯(⋅∣s,ϕ1(s,a1),ϕ2(s,a2))P_φ(· s,a^1,a^2):= P(· s,φ^1(s,a^1),φ^2(s,a^2)). Since perfect filters leave safe actions unchanged, any payoff improvement a player can achieve by deviating to another safe policy is also achievable in the filtered game. This property allows us to compare the equilibria of a filtered game ϕG_φ with those of SCsafeG_SC^safe, the original SC-MG with each player committed to a safe strategy. Theorem 1 (Non-Exploitability under Perfect Filtering). Let ϕ=(ϕ1,ϕ2)φ=(φ^1,φ^2) be perfect filters, and let SCsafeG_SC^safe denote the original SC-MG with each player restricted to safe policy classes Πsafei:=πi∈Πi|πi(safei(s)∣s)=1,∀s∈Ωi∗ ^i_safe:= \π^i∈ ^i\; |\;π^i (A^i_safe(s) s )=1,\ ∀\,s∈ ^i* \. For any initial state s∈Ω∗s∈ ^*, an NE (π1,∗,π2,∗)(π^1,*,π^2,*) of ϕG_φ induces, through the same filters, an executed pair (πexec1,πexec2)(π^1_exec,π^2_exec) that is an NE of SCsafeG_SC^safe. Consequently, both players are non-exploitable in their respective games: explSCsafe(πexeci)(s)=explϕ(πi,∗)(s)=0. _G_SC^safe(π^i_exec)(s)= _G_φ(π^i,*)(s)=0. (6) Proof. Let (π1,∗,π2,∗)(π^1,*,π^2,*) be an NE of ϕG_φ. Suppose a player could improve its payoff by a safe unilateral deviation from the executed pair (πexec1,πexec2)(π^1_exec,π^2_exec) in SCsafeG_SC^safe. Since a perfect filter leaves safe actions unchanged, the deviating policy could be used as a nominal task policy in ϕG_φ, which achieves the same improvement in ϕG_φ, contradicting its NE property. Hence, the executed policy pair is an NE of SCsafeG_SC^safe. The equilibrium payoffs coincide, and Definition 2 ensures zero exploitability for both players in their respective games. ∎ Theorem 1 shows that in competitive games, the safety–task separation does not necessarily sacrifice exploitability when both players are safety-aware. Based on this insight, Sec. V presents an RL framework for synthesizing competitive robot policies using learned robust safety filters. V Method: Learning Minimally Exploitable Robot Policies with Filtered RL In this section, we present our main algorithmic contribution, Safety to Competence (S2C), a two-stage RL framework that trains safe and minimally exploitable robot policies. It sequentially trains a robust safety filter and a task policy with the filter embedded in the environment. Our overall system architecture and algorithmic approach are summarized in Fig. 3 and Algorithm 1. Our framework is modular and agnostic to the specific methods used for filter synthesis, task policy training, and filter deployment. In this paper, we instantiate filter synthesis, task-policy training, and filter deployment with MAGICS [33], IPPO [26], and Robust Q-CBF [24, 22], respectively. Algorithm 1 S2C: Safety-Filtered Competitive RL 1: margins gi,gi,cg^i,g^i,c, task reward r, initial task policy πinitiπ^i_ init, robust filter synthesis algorithm Filter_Synthesis, multi-agent reinforcement learning algorithm MARL 2: Training Stage 1: Robust Filter Synthesis 3: (Q ,π1 ,π2 )←(Q [*]shield-alt,π [*]shield-alt_1,π [*]shield-alt_2)← Filter_Synthesis(g1,g1,c,g2,g2,cg^1,g^1,c,g^2,g^2,c) 4: Robust_Filter ←(Q ,π1 ,π2 )←(Q [*]shield-alt,π [*]shield-alt_1,π [*]shield-alt_2) 5: Training Stage 2: Filtered Multi-Agent RL 6: Initialize policy pool: ←∅P← 7: Initialize task policies: (π1,π2)←(πinit1,πinit2)(π^1,π^2)←(π^1_ init,π^2_ init) 8: for k=1,2,…k=1,2,… do 9: Collect rollouts with Robust_Filter; each player faces its opponent using π−iπ^-i or, with probability p, a policy in P 10: Train π1,π2π^1,π^2 with MARL 11: Periodically update ←∪π1,π2P ∪\π^1,π^2\ 12: Deployment 13: πe←π1π^e←π^1 or π2π^2 14: ate←a^e_t← Robust_Filter(st,πes_t,π^e) V-A Synthesizing Robust Safety Filters For each player ii, we first synthesize its robust safety filter independently of the task objective. We formulate safety as a zero-sum game between the robot and a worst-case opponent: the ego robot seeks to maintain its own safety, while the adversary actively drives it toward failure. This yields a learned safety fallback policy πi π [*]shield-alt_i trained to protect player ii against adversarial opponent behavior. To solve this safety game for general, high-dimensional interactive tasks (e.g., the touchdown game in the Running Example with 62D observation and 12D action spaces), we use MAGICS [33], a game-theoretic adversarial RL algorithm for neural safety synthesis. MAGICS jointly trains a safety fallback πi π [*]shield-alt_i, a worst-case adversarial policy, and a safety critic Q Q [*]shield-alt. It is shown to converge to a local minimax solution, providing a scalable approximation to HJ-based robust safety analysis that would yield a perfect safety filter [23]. The fallback πi π [*]shield-alt_i and critic Q Q [*]shield-alt form essential parts of the robust safety filter ϕi(s,πi)φ^i(s,π^i), which takes a state s and a task policy πiπ^i as inputs, simulates the worst-case opponent action, and returns a filtered task action aia^i. We explain the deployment of this robust filter in Sec. V-C. Running Example: In the touchdown game, both the safety fallback and critic operate with a 62D observation comprising both ego and opponent’s positions of the body frame, body velocities and orientations, axial rotational rates, and angular velocity. The robot action ai∈i=[−1,1]12a^i ^i=[-1,1]^12 is a normalized joint-position increment command for the 12 actuated leg joints, tracked by a joint-level PD controller. Both the safety fallback and the critic are three-layer MLPs, with hidden layer widths of 512 and 256 units, respectively, both trained with 2048 parallel environments in MuJoCo [31] on a workstation with an RTX 5080 GPU. V-B Learning Task Policies in the Filtered Environment We train each agent’s task policy πiπ^i in an RL environment that embeds its safety filter ϕiφ^i, as in the filtered game of Definition 4. To train both agents’ task policies simultaneously, our framework calls for a MARL algorithm, e.g., IPPO [26] or MAPPO [35]. At each training step, each player’s task policy πiπ^i proposes a candidate action aia^i, and the simulator executes the action modified by the safety filter. Crucially, the task policies are learned based on the agents’ interaction data with the environment, requiring neither explicit information on the safety filter nor extra reward terms to account for safety overrides. To stabilize training and prevent overfitting to a specific opponent strategy, we maintain a policy pool P of historical policy checkpoints [13, 17] and sample opponents from this pool in a fraction of parallel environments. We prioritize opponents with a win rate close to 50%50\% to focus learning on balanced matches. Running Example: The task reward consists of a sparse ±1± 1 touchdown reward and a dense locomotion shaping term that is annealed after 1000 iterations. We train each player’s task policy πiπ^i with IPPO [26]. The task policy uses the same observation space as the safety filter and outputs 12D joint-position targets, which are converted into joint-position increments prior to filtering. Each player’s policy is parameterized by an MLP with three hidden layers of 512, 256, and 128 neurons. Training is conducted across 8192 parallel environments with an opponent-pool sampling probability of p=0.35p=0.35, running for 5000 iterations. V-C Deploying Task Policies with Robust Safety Filters At deployment, each learned task policy πiπ^i is paired with the same robust safety filter ϕiφ^i used during training. The filter is constructed from the safety critic Q Q [*]shield-alt and fallback policy π π [*]shield-alt learned in Sec. V-A. At every control step, the task policy proposes a candidate action, which is passed through unchanged when it satisfies the learned safety constraint and otherwise corrected by the filter. Using the same filter during training and deployment is critical to our framework. During training, the task policy experiences the consequences of filter interventions directly through the filtered environment and learns to cooperate with the filter to minimize exploitability (Theorem 1) by applying task strategies that avoid unnecessary overrides. Running Example: For the touchdown game, we approximate the perfect filter in Definition 3 using a Q-CBF, which has been shown to robustly maintain safety while preserving task performance in quadrupedal locomotion [24], with the learned critic Q Q [*]shield-alt and fallback πi π [*]shield-alt_i. In particular, the filter performs a fixed number of gradient-descent steps to search for an action closest to the proposed task action and certified safe by Q Q [*]shield-alt. If no such action is found, the filter uses the learned fallback policy πi π [*]shield-alt_i. VI Experiments We use our proposed S2C framework to synthesize the safety filter and train touchdown game policies in both symmetric and asymmetric scenarios. We evaluate their safety and competence at scale in simulation, and deploy the robot in both roles (attacker and defender) against a human opponent in a real-world experiment. Hypotheses. Our experiments test two hypotheses that reflect the key advantages of S2C: • H1 (Safe competence). S2C yields more competitive policies without sacrificing safety. • H2 (Robustness and generalization). S2C generalizes better to out-of-distribution opponents. Baselines. We compare S2C to four baseline safe RL methods that jointly optimize safety and task objectives. • Early termination (ET) ends the current training episode on a safety violation. • Reward penalty (Nom) adds a penalty to the reward for each safety violation. • Constrained policy optimization (CPO) enforces a violation budget bib^i during a trust-region update. • PPO-Lagrangian (Lag) enforces a violation budget through an adaptive multiplier. • Filtered baselines († ) equips each baseline above with the same filter used by S2C at deployment. All baselines are trained with the same IPPO algorithm, network architecture, and training schedule as S2C. We also swept additional parameters for the CPO and Lag baselines. Metrics. We select checkpoints that achieve the best self-play payoff for all nine methods and use them to form the evaluation pool ℬB. We evaluate robot performance with the following metrics: • Safe rate (SR). The fraction of episodes free of a safety violation indicated by gtoti(s)<0g_tot^i(s)<0. • Win rate (WR). The averaged share of won episodes against opponent methods. • Empirical exploitability. We measure empirical exploitability for the symmetric game (V∗=0V^*=0). We then evaluate (4) on the empirical games over ℬB [17] as maxπ′∈ℬJ^(π′,π) _π J(π ,π), where J^(π,π′)=(nwin(π)−nwin(π′))/n J(π,π )= (n_win(π)-n_win(π ) )/n is the net win rate over n episodes. • Elo. We calculate Elo ratings for symmetric games based on all pool matches using the standard scale factor of 400, with bootstrap confidence intervals. Simulation values are mean ± one standard deviation across the five random training seeds of each method. VI-A Simulated Touchdown Games Fig. 4: Safety versus win rate in symmetric touchdown games. Results. We report the results of the simulated symmetric game (Running Example) in this section. Following the setup in Sections V-A and V-B, we evaluate all nine methods in a round-robin tournament. As shown in Fig. 2 and Table I, S2C achieves the highest win rate across all pairwise matches with an average of 76.5%76.5\%. S2C also achieves the highest Elo rating (139139), significantly exceeding the second-place competitor, Nom† (3737) by more than 100100 points. In terms of empirical exploitability, the strongest pool opponent achieves a net win rate of 0.110.11 against S2C, compared to 0.540.54–0.750.75 against the baselines. Because the evaluation pool includes unfiltered baselines, this indicates that S2C remains hard to exploit even by reckless opponents not committed to using safe policies. Moreover, S2C achieves a high safe rate of 90.0%90.0\%, close to that of Lag†. These results validate H1. Filtered ablation. We conduct an ablation study by deploying the learned safety filter in S2C for the baselines at runtime (see Figs. 2 and 4). The filter enhances safety as expected: the safe rates of filtered baselines improve across the board by up to 22.422.4 percentage points (p), bringing both Lag† and Nom† to safety levels comparable to S2C. However, this added safety does not transfer to a notable gain in the robot’s competitive competence: baseline win rates change by at most 7.47.4 p, remaining substantially below S2C, and their exploitability stays high. This is because the baseline policies were trained in unfiltered environments, and therefore did not learn to cooperate with the safety filter. In some states, they repeatedly propose unsafe actions, so the filter intervenes frequently and thereby degrades the task performance. This ablation shows that a safety filter alone does not grant strategic competence; rather, competence must be co-learned within the filtered environment as in S2C. Qualitative observations. During training, we observe that competitive tactics such as feints and abrupt stop-and-turn maneuvers (see Fig. 1) emerge noticeably earlier in S2C than in the baseline methods. We attribute this to the safety filter intervening to brake or stabilize the robot whenever it approaches danger. It spares the policy from having to discover constraint-satisfying strategies through trial-and-error failures, allowing it to focus its exploratory effort more directly on task objectives. In contrast, because the baselines must map safety boundaries through catastrophic failures, individual runs often collapse into overly conservative play or stereotyped gaits that specific opponents can easily exploit during evaluation. TABLE I: Round-robin results in simulated touchdown games. Asymmetric game Symmetric game Method WR (%\%) ↑ SR (%\%) ↑ WR (%\%) ↑ SR (%\%) ↑ S2C 77.5 ± 2.8 87.9 ± 2.4 76.5 ± 2.0 90.0 ± 1.9 ET 68.1 ± 12.5 69.7 ± 18.5 56.0 ± 8.7 84.2 ± 4.4 Nom 51.4 ± 6.3 68.9 ± 8.6 52.8 ± 11.4 75.9 ± 10.8 CPO 22.3 ± 2.3 46.9 ± 5.2 37.6 ± 6.6 65.3 ± 5.7 Lag 63.0 ± 12.9 67.6 ± 18.2 37.2 ± 6.3 69.1 ± 5.4 ET† 46.0 ± 3.4 69.1 ± 7.6 51.3 ± 5.5 87.6 ± 3.6 Nom† 41.6 ± 2.4 76.6 ± 3.0 53.5 ± 7.8 89.2 ± 5.2 CPO† 30.6 ± 2.3 71.4 ± 4.4 40.5 ± 8.6 84.3 ± 7.6 Lag† 49.5 ± 5.0 75.7 ± 5.4 44.6 ± 7.9 91.5 ± 2.2 Each row corresponds to 31200 games. WR: win rate. SR: safe rate. VI-B Hardware Stress Tests Against Human Opponents For the real-world evaluation, we consider an asymmetric attack–defense game where the attacker needs to reach a target line within a time limit to win. This asymmetry induces more intense interactions, such as frequent close contact between the two players. The asymmetric game is trained and evaluated in simulation under the same protocol as the symmetric one, where S2C attains both the highest win rate and the highest safe rate (Table I). We further conduct this game in hardware experiments and evaluate the policies against a human opponent. Hardware System Setup. We use an HTC Vive Lighthouse system to localize both the Unitree Go2 quadrupedal robot and the human. This system consists of two base stations placed at the diagonal corners of the field and two SteamVR trackers mounted on the robot and its human opponent. Policy inference and the safety filter run on the Jetson Nano at 5050 Hz. All five methods are trained under the same domain-randomization strategy. We evaluate S2C and all four baselines for ten rounds in each attacker/defender role. Results. TABLE I: Hardware trials against a human opponent. Robot attacking Method TDR (%\%) ↑ SR (%\%) ↑ HPS (m//s) HPA (m//s2) S2C 70.0 80.0 2.14±0.362.14± 0.36 5.83±1.305.83± 1.30 ET 50.0 60.0 1.86±0.261.86± 0.26 3.17±1.033.17± 1.03 Nom 30.0 30.0 1.46±0.191.46± 0.19 2.35±1.232.35± 1.23 CPO 0.0 0.0 1.30±0.381.30± 0.38 2.37±0.732.37± 0.73 Lag 20.0 20.0 0.74±0.430.74± 0.43 1.04±0.581.04± 0.58 Robot defending Method TOR (%\%) ↑ SR (%\%) ↑ HPS (m//s) HPA (m//s2) S2C 70.0 90.0 2.40±0.382.40± 0.38 4.31±1.144.31± 1.14 ET 60.0 80.0 2.07±0.162.07± 0.16 4.34±1.214.34± 1.21 Nom 40.0 50.0 1.22±0.151.22± 0.15 2.46±0.792.46± 0.79 CPO 0.0 100.0 1.30±0.221.30± 0.22 2.15±0.702.15± 0.70 Lag 10.0 60.0 1.10±0.211.10± 0.21 1.85±0.661.85± 0.66 Each row corresponds to ten trials. TDR/TOR: touchdown/timeout rate. SR: safe rate. HPS: human’s peak speed. HPA: human’s peak acceleration. Table I summarizes task completion, safe rates, and opponent kinematics across the hardware trials. We record the following numbers as measures of human competence: • Human peak speed (HPS). The human opponent’s peak speed in m//s. • Human peak acceleration (HPA). The human opponent’s peak acceleration in m//s2. Across both roles, S2C achieves the highest task completion (70%70\%) while maintaining high safe rates (80%80\% as attacker and 90%90\% as defender). In comparison, ET and Nom compromise safety under competitive pressure, with safe rates dropping to 30%30\%–60%60\% when attacking. Meanwhile, CPO fails to reconcile the two objectives: it causes a safety violation on every attacking trial (zero touchdowns), and achieves zero defensive violations only by passively traversing the field, unable to contest the human attacker in close proximity. From the HPS and HPA values, S2C has to respond to the most demanding human movements with the highest mean peak speed in both roles, the highest attacking acceleration (5.83m/s25.83\,m/s^2), and defending acceleration comparable to that of ET. Those human strategies were not encountered during training. These trials provide evidence that S2C is more robust against out-of-distribution opponents and support H2. VII Conclusion Limitations. While our analysis assumes perfect safety filters, deriving formal guarantees for ϵε-Nash equilibria under approximation errors stemming from imperfect filters and task policies remains an open research question. We also see an exciting opportunity to integrate S2C with latent-space safety filters with visual feedback [20] for competitive robot interactions in unstructured, open-world environments. Summary. In this paper, we introduced S2C, a two-stage framework that decouples safety from competitive task-policy learning in multi-robot interactions. We proved that under perfect safety filtering, equilibrium policies of the filtered game remain non-exploitable within the safe policy class of the original safety-critical Markov game. We validated S2C on a quadruped touchdown game in both simulation and hardware tests. The results demonstrate that S2C achieves the highest task performance and the lowest exploitability across the evaluation pool while maintaining high safe rates. Our work establishes a principled framework for learning safe and minimally exploitable robot policies for competitive interactions. References [1] J. Achiam, D. Held, A. Tamar, and P. Abbeel (2017) Constrained policy optimization. In International Conference on Machine Learning, Proceedings of Machine Learning Research, p. 22–31. Cited by: §I, §I. [2] E. Altman (1995) Constrained Markov decision processes. Technical report Technical Report R-2574, INRIA. Cited by: §I. [3] A. D. Ames, S. Coogan, M. Egerstedt, G. Notomista, K. Sreenath, and P. Tabuada (2019) Control barrier functions: theory and applications. In European Control Conference, p. 3420–3431. Cited by: §I. [4] A. Bajcsy, A. Loquercio, A. Kumar, and J. Malik (2024) Learning vision-based pursuit-evasion robot policies. In IEEE International Conference on Robotics and Automation, p. 9197–9204. Cited by: §I. [5] S. Bansal, M. Chen, S. Herbert, and C. J. Tomlin (2017) Hamilton–Jacobi reachability: a brief overview and recent advances. In IEEE Conference on Decision and Control, p. 2242–2253. Cited by: §I. [6] S. Bansal and C. J. Tomlin (2021) DeepReach: a deep learning approach to high-dimensional reachability. In IEEE International Conference on Robotics and Automation, Vol. , p. 1817–1824. External Links: Document Cited by: §I. [7] T. Başar and G. J. Olsder (1998) Dynamic noncooperative game theory. SIAM, Philadelphia. Cited by: §I, §I. [8] F. Blanchini (1999) Set invariance in control. Automatica 35 (11), p. 1747–1767. External Links: Document Cited by: §IV. [9] J. F. Fisac, N. F. Lugovoy, V. Rubies-Royo, S. Ghosh, and C. J. Tomlin (2019) Bridging hamilton-jacobi safety analysis and reinforcement learning. In IEEE International Conference on Robotics and Automation, Vol. , p. 8550–8556. External Links: Document Cited by: §I. [10] D. Fridovich-Keil, E. Ratner, L. Peters, A. D. Dragan, and C. J. Tomlin (2020) Efficient iterative linear-quadratic approximations for nonlinear multi-player general-sum differential games. In IEEE International Conference on Robotics and Automation, Vol. , p. 1475–1481. External Links: Document Cited by: §I. [11] J. García and F. Fernández (2015) A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research 16 (42), p. 1437–1480. Cited by: §I, §I. [12] T. Haarnoja, B. Moran, G. Lever, S. H. Huang, D. Tirumala, J. Humplik, M. Wulfmeier, S. Tunyasuvunakool, N. Y. Siegel, R. Hafner, M. Bloesch, K. Hartikainen, A. Byravan, L. Hasenclever, Y. Tassa, F. Sadeghi, N. Batchelor, F. Casarini, S. Saliceti, C. Game, N. Sreendra, K. Patel, M. Gwira, A. Huber, N. Hurley, F. Nori, R. Hadsell, and N. Heess (2024) Learning agile soccer skills for a bipedal robot with deep reinforcement learning. Science Robotics 9 (89), p. eadi8022. Cited by: §I. [13] J. Heinrich, M. Lanctot, and D. Silver (2015) Fictitious self-play in extensive-form games. In International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 37, p. 805–813. Cited by: §V-B. [14] K. Hsu, H. Hu, and J. F. Fisac (2024) The safety filter: a unified view of safety-critical control in autonomous systems. Annual Review of Control, Robotics, and Autonomous Systems 7 (1). Cited by: §I, §IV, Definition 3. [15] K. Hsu, D. P. Nguyen, and J. F. Fisac (2023) ISAACS: iterative soft adversarial actor-critic for safety. In Learning for Dynamics and Control Conference, Cited by: §I. [16] E. Kaufmann, L. Bauersfeld, A. Loquercio, M. Müller, V. Koltun, and D. Scaramuzza (2023) Champion-level drone racing using deep reinforcement learning. Nature 620 (7976), p. 982–987. External Links: Document Cited by: §I. [17] M. Lanctot, V. F. Zambaldi, A. Gruslys, A. Lazaridou, K. Tuyls, J. Pérolat, D. Silver, and T. Graepel (2017) A unified game-theoretic approach to multiagent reinforcement learning. Advances in Neural Information Processing Systems 30. Cited by: §I, §V-B, 3rd item. [18] J. Lidard, H. Hu, A. Hancock, Z. Zhang, A. G. Contreras, V. Modi, J. DeCastro, D. Gopinath, G. Rosman, N. Leonard, M. Santos, and J. F. Fisac (2024) Blending data-driven priors in dynamic games. Robotics: Science and Systems. External Links: Document Cited by: §I. [19] R. Lowe, Y. Wu, A. Tamar, J. Harb, P. Abbeel, and I. Mordatch (2017) Multi-agent actor-critic for mixed cooperative-competitive environments. Advances in Neural Information Processing Systems 30. Cited by: §I. [20] K. Nakamura, L. Peters, and A. Bajcsy (2025) Generalizing safety beyond collision-avoidance via latent-space reachability analysis. In Robotics: Science and Systems, Cited by: §VII. [21] D. P. Nguyen, K. Hsu, W. Yu, J. Tan, and J. F. Fisac (2025) Gameplay filters: robust zero-shot safety through adversarial imagination. In Conference on Robot Learning, p. 387–407. External Links: Link Cited by: §I. [22] D. D. Oh, J. Lidard, H. Hu, H. Sinhmar, E. Lazarski, D. Gopinath, E. S. Sumner, J. A. DeCastro, G. Rosman, N. E. Leonard, et al. (2025) Safety with agency: human-centered safety filter with application to AI-assisted motorsports. In Robotics: Science and Systems, Cited by: §I, §V. [23] D. D. Oh, D. P. Nguyen, H. Hu, and J. F. Fisac (2026) Provably optimal reinforcement learning under safety filtering. In Conference of the International Association for Safe and Ethical Artificial Intelligence, Cited by: §I, §V-A. [24] D. D. Oh, D. P. Nguyen, H. Hu, and J. F. Fisac (2026) Synthesis and deployment of maximal robust control barrier functions through adversarial reinforcement learning. In IEEE Conference on Decision and Control, Cited by: §I, §I, §V-C, §V. [25] A. Robey, H. Hu, L. Lindemann, H. Zhang, D. V. Dimarogonas, S. Tu, and N. Matni (2020) Learning control barrier functions from expert demonstrations. In IEEE Conference on Decision and Control, Vol. , p. 3717–3724. External Links: Document Cited by: §I. [26] C. Schroeder de Witt, T. Gupta, D. Makoviichuk, V. Makoviychuk, P. H. S. Torr, M. Sun, and S. Whiteson (2020) Is independent learning all you need in the StarCraft multi-agent challenge?. External Links: 2011.09533 Cited by: §I, §V-B, §V-B, §V. [27] J. Seo, K. Nakamura, and A. Bajcsy (2025) Uncertainty-aware latent safety filters for avoiding out-of-distribution failures. In Conference on Robot Learning, p. 4442–4472. Cited by: §I. [28] L. S. Shapley (1953) Stochastic games. Proceedings of the National Academy of Sciences 39 (10), p. 1095–1100. Cited by: §I. [29] H. Sun, Z. Xu, M. Fang, Z. Peng, J. Guo, B. Dai, and B. Zhou (2021) Safe exploration by solving early terminated MDP. External Links: 2107.04200 Cited by: §I. [30] C. Tessler, D. J. Mankowitz, and S. Mannor (2019) Reward constrained policy optimization. In International Conference on Learning Representations, Cited by: §I, §I. [31] E. Todorov, T. Erez, and Y. Tassa (2012) MuJoCo: a physics engine for model-based control. In IEEE/RSJ International Conference on Intelligent Robots and Systems, p. 5026–5033. Cited by: §V-A. [32] K. P. Wabersich and M. N. Zeilinger (2022) Predictive control barrier functions: enhanced safety mechanisms for learning-based control. IEEE Transactions on Automatic Control 68 (5), p. 2638–2651. Cited by: §I. [33] J. Wang, H. Hu, D. P. Nguyen, and J. F. Fisac (2024) MAGICS: adversarial RL with minimax actors guided by implicit critic Stackelberg for convergent neural synthesis of robot safety. In Algorithmic Foundations of Robotics, Cited by: §I, §I, §V-A, §V. [34] P. R. Wurman, S. Barrett, K. Kawamoto, J. MacGlashan, K. Subramanian, T. J. Walsh, R. Capobianco, A. Devlic, F. Eckert, F. Fuchs, et al. (2022) Outracing champion gran turismo drivers with deep reinforcement learning. Nature 602 (7896), p. 223–228. External Links: Document Cited by: §I. [35] C. Yu, A. Velu, E. Vinitsky, J. Gao, Y. Wang, A. Bayen, and Y. Wu (2022) The surprising effectiveness of PPO in cooperative multi-agent games. Advances in Neural Information Processing Systems 35, p. 24611–24624. External Links: Link Cited by: §V-B.