Paper deep dive

Interactive Distributionally Robust Multi-Agent Learning with General Function Approximation

Debamita Ghosh, George K. Atia, Yue Wang

Year: 2026Venue: arXiv preprintArea: cs.LGType: PreprintEmbeddings: 317

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%

Last extracted: 9/29/2026, 12:37:41 PM

Summary

This paper introduces RoMEX-ϕ, a model-free framework for online distributionally robust multi-agent reinforcement learning (MARL) in general-sum Distributionally Robust Markov Games (DRMGs) with general function approximation. The method integrates equilibrium-based exploration with dual fitted learning, utilizing a functional dual representation of the robust multi-agent Bellman operator. The authors propose the robust Multi-Agent Decoupling Coefficient (robust MADC) as an intrinsic complexity measure that governs exploration and regret, replacing tabular dependence on state and joint action spaces. Theoretical results establish sublinear robust regret guarantees, and numerical experiments demonstrate the framework's resilience to transition shifts compared to non-robust counterparts.

Entities (7)

RoMEX-ϕ · algorithm · 98%Distributionally Robust Markov Games · problemsetting · 97%robust MADC · metric · 96%Multi-Agent Reinforcement Learning · field · 95%Bellman Operator · mathematicalconcept · 94%ϕ-divergence · mathematicalconcept · 93%Total Variation · uncertaintysettype · 92%

Relation Signals (5)

RoMEX-ϕ → solves → Distributionally Robust Markov Games

confidence 98% · We propose RoMEX-ϕ, a model-free framework that integrates equilibrium-based exploration with dual fitted learning... for online learning in general-sum DRMGs

RoMEX-ϕ → uses → robust MADC

confidence 96% · Using this complexity measure [robust MADC], we establish sublinear robust regret guarantees for RoMEX-ϕ

robust MADC → characterizes → intrinsic exploration complexity

confidence 95% · robust MADC... captures the intrinsic exploration complexity arising from strategic interactions and adversarial transition uncertainty.

RoMEX-ϕ → utilizes → ϕ-divergence

confidence 93% · We propose RoMEX-ϕ... under general function approximation and ϕ-divergence uncertainty sets.

RoMEX-ϕ → outperforms → non-robust counterpart

confidence 90% · RoMEX-ϕ is substantially more resilient to transition shifts than its non-robust counterpart

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Model misspecification poses a fundamental challenge in multi-agent reinforcement learning, where transition uncertainty can be amplified by strategic interactions among agents. Distributionally robust Markov games (DRMGs) provide a principled framework for addressing such uncertainty, yet existing methods often rely on restrictive assumptions or scale poorly to large state and joint action spaces. We study online learning in general-sum DRMGs with general function approximation and $\phi$-divergence uncertainty sets. We propose RoMEX-$\phi$, a model-free framework that integrates equilibrium-based exploration with dual fitted learning. Through a functional dual representation of the robust multi-agent Bellman operator, RoMEX-$\phi$ enables tractable worst-case value estimation from nominal interaction data using a centered empirical robust discrepancy. We introduce the robust Multi-Agent Decoupling Coefficient (robust MADC) to characterize the intrinsic exploration complexity arising from strategic interactions and adversarial transition uncertainty. We establish sublinear robust regret guarantees governed by the robust MADC rather than explicitly by the state and joint action space sizes, replacing tabular dependence with intrinsic function-class complexity. Numerical experiments on a scalable general-sum DRMG under total variation uncertainty show that RoMEX-$\phi$ is substantially more resilient to transition shifts than its non-robust counterpart while remaining competitive with an exact tabular robust baseline. Our results provide a scalable framework for distributionally robust multi-agent reinforcement learning with general function approximation.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

313,403 characters extracted from source content.

Expand or collapse full text

Interactive Distributionally Robust Multi-Agent Learning with General Function Approximation Debamita Ghosh Affiliation: Department of Electrical and Computer Engineering, University of Central Florida, Orlando, FL 32816, USA George K. Atia Affiliation: Department of Electrical and Computer Engineering, University of Central Florida, Orlando, FL 32816, USA Affiliation: Department of Computer Science, University of Central Florida, Orlando, FL 32816, USA Yue Wang Affiliation: Department of Electrical and Computer Engineering, University of Central Florida, Orlando, FL 32816, USA Affiliation: Department of Computer Science, University of Central Florida, Orlando, FL 32816, USA Abstract Model misspecification poses a fundamental challenge in multi-agent reinforcement learning, where transition uncertainty can be amplified by strategic interactions among agents. Distributionally robust Markov games (DRMGs) offer a principled approach to addressing such uncertainty, yet existing methods often rely on restrictive assumptions or scale poorly to large state and joint action spaces. In this work, we study online multi-agent reinforcement learning in general-sum DRMGs with general function approximation and ϕφ-divergence uncertainty sets. We propose RoMEX-ϕφ, a model-free framework that integrates equilibrium-based exploration with dual fitted learning. By developing a functional dual representation of the robust multi-agent Bellman operator, RoMEX-ϕφ enables tractable worst-case value estimation from nominal interaction data through a unified objective based on a centered empirical robust discrepancy. To characterize the learning complexity, we introduce the robust Multi-Agent Decoupling Coefficient (robust MADC), which captures the intrinsic exploration complexity arising from strategic interactions and adversarial transition uncertainty. Using this complexity measure, we establish sublinear robust regret guarantees for RoMEX-ϕφ that depend explicitly on the robust MADC rather than the state-space and joint action-space sizes, thereby replacing unfavorable tabular dependence with intrinsic function-class complexity and improving scalability to large-scale multi-agent systems. We complement our theoretical results with numerical experiments on a scalable general-sum DRMG under total variation uncertainty, where RoMEX-ϕφ exhibits substantially greater resilience to transition shifts than its non-robust counterpart while remaining competitive with an exact tabular robust baseline. Together, these results provide a scalable framework for distributionally robust multi-agent reinforcement learning with general function approximation. 1 Introduction Multi-agent reinforcement learning (MARL) (Albrecht et al., 2024) provides a fundamental framework for modeling sequential decision-making with multiple interacting agents. It has enabled significant advances in domains such as strategic games (Silver et al., 2016; Vinyals et al., 2019), autonomous systems (Shalev-Shwartz and others, 2016; Hua et al., 2024), and multi-robot coordination (Lowe et al., 2017; Matignon et al., 2012). Despite this progress, a key obstacle to real-world deployment is the sim-to-real gap (Zhao et al., 2020; Peng et al., 2018): policies trained under nominal or simulated dynamics often degrade significantly under even small model mismatches due to unmodeled effects, sensor noise, or environmental variations (Padakandla et al., 2020; Rajeswaran et al., 2016). This challenge is significantly exacerbated in multi-agent systems due to the intrinsic coupling of agent behaviors. A small perturbation affecting one agent can propagate through strategic interactions, inducing cascading adaptations across agents and resulting in highly non-stationary and unstable dynamics (Papoudakis et al., 2019; Canese et al., 2021; Wong et al., 2023). Consequently, even minor model inaccuracies can destabilize the entire system, making robustness a fundamental requirement for MARL. To address such uncertainty, Distributionally Robust Markov Games (DRMGs) provide a principled framework (Zhang et al., 2020; Kardeş et al., 2011; Blanchet et al., 2023). DRMGs model transition uncertainty via some divergence-based ambiguity set of transition probabilities, and adopt the Principle of Pessimism to consider the worst-case expected returns over the sets. This pessimistic formulation ensures baseline performance guarantees under model misspecification and acts as an implicit regularizer, promoting policies that generalize better across uncertain environments (Vinitsky et al., 2020; Abdullah et al., 2019; Liu et al., 2025). Although promising, existing learning algorithms for DRMGs are still falling short in practical deployments due to two limitations. Firstly, most methods are limited to idealized data generation settings, assuming either access to a generative model (Shi et al., 2024b; Jiao and Li, 2024) or comprehensive offline datasets (Blanchet et al., 2023; Li et al., 2025). In many real-world applications—such as autonomous systems (Demontis et al., 2022) and healthcare (Alaa Eldin, 2023; Lu et al., 2021)—assumptions of high-fidelity simulator or high-quality dataset can be unrealistic, and agents must instead learn online via directly interactions. However, the online DRMG setting introduces a fundamentally harder problem: agents face an off-dynamic (Holla, 2021; Eysenbach et al., 2020) setting: they interact and collect data from the nominal/training environment, whereas their goal is to find equilibria under the worst case. Therefore, they must balance exploration and exploitation under worst-case dynamics, which can be significantly more challenging that the previous settings. The second limitation of existing studies is their poor scalability. In DRMGs, it is recently shown that online learning algorithms inherently suffer from the curse of multi-agency (Farhat et al., 2026; Shi et al., 2024b), i.e., the sample complexity depends on the joint action space size, which becomes exponentially large in large-scale systems. To break this curse, either additional uncertainty set structures (Shi et al., 2024a; Jiao and Li, 2024) or a worst performance estimation oracle (Ma et al., 2023) are required. A more recent work (Zheng and Lin, 2026) attempts to improve the scalability of robust MARL by utilizing the linear function approximation technique. However, due to the restrictive linear function class, the DRMG is assumed to be linear, which can be restrictive and hard to verify in practice. On the other hand, general function approximation aims to approximate the value function through some general function classes (like neural networks), and is well understood in standard RL through intrinsic complexity measures such as Bellman rank and Bellman–Eluder dimension (Jin et al., 2021; Jin et al., 2022). However, these measures rely on a single Bellman operator, which breaks down in general-sum robust multi-agent systems. Therefore, it is still unclear how to design provable complexity measurements and sample-efficient algorithms for robust MARL. Taken together, these limitations expose a fundamental research gap: Can we design a robust MARL algorithm for DRMGs without restrictive assumptions and with a scalable sample complexity that breaks the multi-agency curse? In this paper, we provide an affirmative answer to the above question by developing a unified framework for online robust MARL with general function approximation, together with scalable sample-efficiency guarantees. Our main contributions are summarized as follows. Scalable dual robust fitted learning for online robust MARL with general function approximation. We propose RoMEX-ϕφ, the first algorithm for purely online learning in general-sum DRMGs under general function approximation (instead of restrictive linear functions) and ϕφ-divergence uncertainty sets. Our approach integrates robustness and exploration via a dual reformulation of the robust multi-agent Bellman operator, enabling policy evaluation directly from nominal data. Specifically, the algorithm computes a functional dual minimizer for each candidate joint policy and plugs it into a fitted robust Bellman recursion, yielding a dual robust fitted-learning procedure that converts intractable worst-case updates into tractable empirical objectives. This formulation both approximates the worst-case Bellman operator and enables a global, function-class-level quantification of uncertainty via the induced robust Bellman residual. Unlike tabular methods relying on per-state–action bonuses, RoMEX-ϕφ employs a global regularization mechanism—through centered empirical robust discrepancy and regularized payoffs—that penalizes poor Bellman consistency and guides coordinated exploration across agents, while remaining computationally efficient in large-scale settings. Robust MADC: intrinsic complexity for robust multi-agent learning. We introduce the robust Multi-Agent Decoupling Coefficient (robust MADC), dMADCrob,maxd_MADC^rob, , as the intrinsic complexity measure governing exploration and regret in online DRMGs with general function approximation. It quantifies how empirical robust Bellman errors—computed from nominal data—control value prediction under worst-case dynamics across interacting agents and policy-induced distributions. Unlike BE-dimension-based measures (Ghosh et al., 2025a), which are defined for a single Bellman operator, robust MADC captures the strategic coupling and equilibrium structure in general-sum DRMGs while accounting for the statistical mismatch between nominal learning and adversarial evaluation. This yields a unified characterization of function complexity, extending intrinsic-complexity theory to the robust multi-agent setting. Sharp regret guarantees via robust intrinsic complexity. We then establish the first sublinear regret guarantee for online general-sum DRMGs with general function approximation, showing that the robust regret of RoMEX-ϕφ scales as ⁡(dMADCrob,maxK+HdMADCrob,max+H(Bϕmax)2Klog⁡(|ℱ|||KHm/δ)+εdualapprox).O\! (d_MADC^rob, K+Hd_MADC^rob, +H(B_φ )^2 K (|F||G|KHm/δ)+ _dual^approx ). Our bound reveals two fundamental sources of difficulty: the intrinsic exploration complexity, characterized by the robust MADC dMADCrob,maxd_MADC^rob, , and the robust Bellman estimation cost incurred when learning the worst-case Bellman operator from nominal interaction data, captured by BϕmaxB_φ and the dual approximation error εdualapprox _dual^approx. Importantly, the regret has no explicit dependence on the state-space size |||S| or the joint action-space size |||A|, thereby replacing the tabular dependence with intrinsic function-class complexity while explicitly accounting for the additional statistical cost of distributional robustness. Numerical validation. We complement our theory with experiments on a scalable general-sum DRMG under total variation (TV) uncertainty (Section 7). RoMEX-ϕφ exhibits substantially greater resilience to transition shift than its non-robust counterpart and remains competitive with an exact tabular robust baseline, demonstrating the practical effectiveness of robust learning with function approximation. 2 Related Work We discuss the relevant literature on DRMGs and its connections to function approximation below. Tabular Distributionally Robust Markov Games. A substantial body of work studies distributionally robust Markov games (DRMGs) in the tabular setting under different data-access models. Early works (Kardeş et al., 2011; Zhang et al., 2020; Roch and Wang, 2025) address foundational questions such as equilibrium existence and convergence under uncertainty. In the offline setting, (Blanchet et al., 2023) proposes a unified framework (P2M2PO) with sample complexity ⁡(H5||2||2/ε)O(H^5|S|^2|A|^2/ ), while subsequent works (Shi et al., 2024b; Jiao and Li, 2024) extend these results to the generative model setting, achieving minimax-optimal guarantees under strong assumptions. In the more realistic online setting, where agents learn purely from interaction, only a limited number of works are available. In particular, (Ma et al., 2023) studies decentralized robust learning but requires the uncertainty level to satisfy σi≤max⁡ε/(||H2),pmin/H _i≤ \ /(|S|H^2),p_ /H\, which can be restrictive. More recently, (Farhat et al., 2026) establishes regret guarantees under divergence-based uncertainty sets, highlighting challenges such as support mismatch and dependence on the joint action space. Despite these advances, all existing approaches rely on explicit state–action enumeration and per-state exploration, and thus do not scale to large or continuous domains. In contrast, our work considers online DRMGs under general function approximation, where both robustness and exploration must be addressed at the level of function classes, and the robust Bellman operator cannot be directly estimated from data. Single-agent distributionally robust RL with function approximation. A large body of work studies sample-efficient single-agent RL with function approximation under nominal dynamics, where learnability is characterized via intrinsic complexity measures such as the Eluder dimension (Russo and Van Roy, 2013), Bellman rank (Jiang et al., 2017), and Bellman–Eluder (BE) dimension (Jin et al., 2021; Jin et al., 2022). These frameworks show that efficient exploration is possible without explicit dependence on |||S| and |||A|, provided Bellman prediction errors generalize across policies and state distributions. Subsequent works extend these ideas to structured settings, including linear and bilinear models (Sun et al., 2019; Foster et al., 2021; Du et al., 2021b). Extending these techniques to the robust setting is substantially more challenging. The robust Bellman operator involves worst-case expectations that are not directly observable from nominal data and may violate standard realizability assumptions, making Bellman-error-based generalization fundamentally more delicate due to the mismatch between nominal data and worst-case evaluation. As a result, existing work is largely restricted to structured settings, particularly linear function approximation (Ma et al., 2022; Liu and Xu, 2024a; Liu et al., 2024; Wang and Zou, 2021). In the offline (or off-dynamics) setting, recent works (Liu and Xu, 2024a; Liu et al., 2024) develop robust least-squares value iteration methods under coverage assumptions, achieving regret (or sample complexity) guarantees of order ~(dHmin⁡H,1/σK) O\! (dH \H,1/σ\ K ), together with nearly matching lower bounds up to a factor of H H. These results highlight how uncertainty-dependent contraction effects (e.g., min⁡H,1/σ \H,1/σ\ under TV divergence) can be exploited under strong coverage conditions. In contrast, (He et al., 2023) studies online learning in linear MDPs under the standard (non-robust) setting, and does not address distributional robustness. Taken together, these works illustrate a fundamental gap: existing robust methods either rely on offline coverage assumptions or are restricted to non-robust online settings, whereas our work addresses fully online learning in general-sum DRMGs under distributional robustness and general function approximation. Beyond linear models, existing approaches either assume strong coverage (offline or hybrid regimes) (Panaganti et al., 2022; Panaganti et al., 2024) or rely on additional structure. Recent work (Ghosh et al., 2025a) extends intrinsic-complexity ideas to the robust setting via a dual reformulation, enabling sample-efficient learning under general function approximation. However, these approaches rely either on structural assumptions or single-agent formulations, and do not extend to multi-agent settings with strategic coupling. Non-robust MARL with function approximation. A growing literature studies sample-efficient MARL with function approximation, broadly divided into zero-sum and general-sum settings. In zero-sum games, intrinsic-complexity frameworks such as BE dimension extend naturally, yielding efficient algorithms under low complexity (Huang et al., 2021; Jin et al., 2022; Jin et al., 2021; Du et al., 2021a; Zhong et al., 2022). These results rely on the minimax structure and a single Nash value function, and therefore do not generalize to general-sum games. Beyond zero-sum settings, a line of work studies online linear Markov games, including centralized approaches (Xie et al., 2020; Chen et al., 2022b; Cisneros-Velarde and Koyejo, 2023) and decentralized approaches (Cui et al., 2023; Wang et al., 2023; Dai et al., 2024). While decentralized methods mitigate the curse of multi-agency via independent learning, they rely on stronger structural assumptions and more complex algorithmic design. Moreover, these works are largely restricted to linear function classes and do not provide a unified intrinsic-complexity characterization that governs exploration under strategic interaction. In general-sum settings, each agent has its own value function and equilibrium behavior must be defined via NE, CCE, or CE, with no unified Bellman fixed point. Recent works address this via model-based DEC frameworks (Chen et al., 2022a; Foster et al., 2023), model-free CCE/CE methods (Zhan et al., 2022; Wang et al., 2023; Cui et al., 2023), and hybrid structured approaches (Ni et al., 2022). The most relevant development for our work is the MAMEX framework (Xiong et al., 2024), which introduces the Multi-Agent Decoupling Coefficient (MADC) as an intrinsic complexity measure. MADC quantifies how in-sample Bellman error translates into out-of-sample value prediction under strategic interaction, explicitly capturing multi-agent coupling. Its low-complexity regime includes extensions of BE-dimension models (Jin et al., 2021; Jin et al., 2022; Huang et al., 2021), bilinear classes (Du et al., 2021b), and witness-rank models (Sun et al., 2019; Huang et al., 2021), making it a natural foundation for scalable general-sum MARL. Online robust MARL with linear function approximation. A recent work by (Zheng and Lin, 2026) studies online distributionally robust general-sum Markov games under linear function approximation. They propose a least-squares value-iteration-type algorithm for learning approximate robust CCE and establish a sample-complexity bound of order ~(H3dlin2(min⁡H,1/σ)2ε−2) O\! (H^3d_lin^2 ( \H,1/σ\ )^2 ^-2 ) under TV-divergence uncertainty, together with a nearly matching lower bound of order Ω~(H2dlin2(min⁡H,1/σ)2ε−2). \! (H^2d_lin^2 ( \H,1/σ\ )^2 ^-2 ). Their result provides an important step beyond tabular robust MARL, achieving sharp dependence on the feature dimension by exploiting linear structure. However, the analysis is tied to linear realizability and TV-type uncertainty, and does not yield an intrinsic-complexity characterization for broader function classes. In contrast, our framework applies to general function approximation and multiple ϕφ-divergence uncertainty sets, with sample complexity governed by the robust MADC, which captures how Bellman errors under nominal data translate into value prediction under worst-case dynamics. Challenges in extending to robust multi-agent learning. Crucially, the guarantees and techniques developed for non-robust MARL with function approximation do not extend directly to the robust setting due to the intractability and non-observability of the robust Bellman operator. In DRMGs, the standard Bellman operator is replaced by a robust multi-agent Bellman operator involving worst-case expectations over uncertainty sets, which introduces several fundamental challenges. (i) Non-observability of the robust operator: the worst-case Bellman backup cannot be evaluated from nominal trajectories, as data is collected under the nominal kernel while performance is defined under adversarial dynamics, creating a fundamental statistical mismatch. (i) Breakdown of standard realizability and concentration: unlike the nominal setting, where Bellman errors can be controlled via standard generalization arguments, the robust operator is inherently non-linear and may violate realizability, making error propagation significantly more delicate. (i) Coupled equilibrium structure: in general-sum games, each agent’s value depends on others’ strategies, and learning requires controlling deviations across agents under worst-case dynamics, which is not captured by single-agent complexity measures or zero-sum analyses. These challenges are further amplified in the model-free setting considered in this work, where agents must learn solely from interaction without access to a simulator or an offline dataset. In this regime, exploration must simultaneously account for statistical uncertainty and adversarial model shift, while operating at the level of function classes rather than individual state–action pairs. As a result, existing intrinsic-complexity frameworks—whether BE dimension in single-agent RL or MADC in nominal MARL—cannot be directly applied. Taken together, these limitations expose a fundamental gap: existing works either address robustness in single-agent settings or multi-agent learning under nominal dynamics, but no prior framework simultaneously captures robustness, equilibrium learning, and general function approximation in a purely online, model-free setting. Our work addresses this gap by developing a unified framework that combines dual-based robust Bellman estimation, equilibrium-based policy learning, and a robust extension of intrinsic complexity (robust MADC), leading to the first sample-efficient, model-free algorithm for online general-sum DRMGs with general function approximation. 3 Preliminaries and Problem Formulation Robust MARL is generally formulated as a distributionally robust Markov game (DRMG). A finite-horizon n-player general-sum DRMG is specified as ℳrob=ℳ,,,H,P⋆,ii∈ℳ,,ρ0MG_rob= \M,S,A,H,P ,\P_i\_i , r, _0 \, where ℳ=1,…,mM=\1,...,m\ is the set of m agents, =1,2,…,SS=\1,2,…,S\ denotes the finite state space, A denotes the joint action space for all agents as =1×⋯×mA=A_1×·s×A_m, where i=1,2,…,AiA_i=\1,2,…,A_i\ is the action space of agent i, H denotes the horizon length, P⋆P is the nominal transition kernel, and =ri1≤i≤m r=\r_i\_1≤ i≤ m is the reward function where ri:×↦[0,1]r_i:S×A [0,1]. Agents in a DRMG maintain their own uncertainty sets of transition kernels iP_i, to capture the potential environment uncertainties from their perspective. At each step, the environment transitions follow an arbitrary kernel from the uncertainty set. We follow the standard agent-wise (s,)(s, a)-rectangular uncertainty structure (Shi et al., 2024b; Shi et al., 2024a; Zhang et al., 2020; Farhat et al., 2026) due to its computational tractability (Iyengar, 2005; Wiesemann et al., 2013; Zhou et al., 2021; Shi et al., 2023)11 1 Robust MDPs without the rectangularity assumption can be NP-hard to solve (Wiesemann et al., 2013).. Specifically, for each agent i, iP_i is defined independently across all horizons, states, and joint actions as i=⨂(h,s,)∈[H]×i,h,ϕσi(s,),P_i= _(h,s, a)∈[H]×S×AP_i,h,φ _i(s, a), where ⊗ denotes the Cartesian product. We consider the ϕφ-divergence uncertainty set standard in literature (Clavier et al., 2023; Shi et al., 2023; Panaganti et al., 2022; Yang et al., 2022a; Wang et al., 2024b; Zhang et al., 2025), that each local uncertainty set i,h,ϕσi(s,)P _i_i,h,φ(s,a) is centered around a nominal transition kernel P⋆P , through some ϕφ-divergence, ϕ(P,Ph⋆(⋅|s,))=∑s′∈ϕ(P(s′)/Ph⋆(s′|s,))Ph⋆(s′|s,)φ (P,\ P_h (·|s, a) )=Σ _s φ (P(s ) /P_h (s |s, a) )P_h (s |s, a): Definition 1 (ϕφ-Divergence Uncertainty Set). The ϕφ-divergence uncertainty set is defined as i,h,ϕσi(s,)≜P∈Δ():ϕ(P,Ph⋆(⋅|s,))≤σi,Supp(P)⊆Supp(Ph⋆(⋅|s,))P _i_i,h,φ(s, a) \P∈ (S):φ (P,P_h (·|s, a) )≤ _i,Supp(P) (P_h (·|s, a)) \. Support inclusion is essential for online robust RL because states outside the nominal support cannot be observed through online interaction, creating an information bottleneck that can lead to exponential regret (Lu et al., 2024; Ghosh et al., 2025a; Farhat et al., 2026). We consider TV, χ2χ^2, and KL uncertainty sets, for which support inclusion holds automatically under χ2χ^2 and KL, and under TV given Assumption 4. Robust Value Functions. Considering a DRMG, we aim for each agent to maximize its performance against the worst-case transition kernels within its respective uncertainty set. Each agent follows its own policy defined as πi=πi,h:→Δ⁡(i)h=1H. _i=\ _i,h:S→ (A_i)\_h=1^H. The robust Q function for each agent Qi,hπ,σiQ_i,h^π, _i for an initial state s and joint action a is defined as Qi,hπ,σi(s,)≜infP~∈iπ,~[∑t=hHri,t(st,t)|sh=s,h=]Q_i,h^π, _i(s, a) _ P _iE_π, P [ _t=h^Hr_i,t(s_t, a_t) |s_h=s, a_h= a ], where the expectation is taken over the randomness of the joint policy π=πh:→Δ⁡()h=1Hπ=\ _h:S→ (A) \_h=1^H, and the kernel P~ P over its own uncertainty set, and one can similarly define the corresponding robust value function Vi,hπ,σi(s)V_i,h^π, _i(s) for an initial state s. For any joint policy π, the robust value functions to satisfy the robust Bellman equations (Iyengar, 2005) Qi,hπ,σi(s,)=ri,h(s,)+i,h,ϕσi[Vi,h+1π,σi]andVi,hπ,σi(s)=∼π[Qi,hπ,σi(s,)], Q_i,h^π, _i(s, a)=r_i,h(s, a)+E_P_i,h,φ _i[V_i,h+1^π, _i] V_i,h^π, _i(s)=E_ a π[Q_i,h^π, _i(s, a)], (1) where i,h,ϕσi(s,)[Vi,h+1π,σi]≜infPh∈is′∼Ph[Vi,h+1π,σi(s′)]E_P_i,h,φ _i(s, a)[V_i,h+1^π, _i] _P_h _iE_s P_h [V_i,h+1^π, _i(s ) ] is the worst-case expectation. Robust best-response policy. For any given joint policy π, let π−i _-i denote the collection of marginal policies of all agents except agent i. The best-response policy of agent i to π−i _-i, denoted by πi†,σi(π−i)π , _i_i( _-i), is defined as the policy that maximizes its own robust value function at a given step h and state s, i.e., πi†,σi(π−i)≜argmaxπi′∈Δ⁡(i)Vi,hπ−i×πi′,σi(s),π , _i_i( _-i) _i ∈ (A_i) V_i,h _-i× _i , _i(s), and we denote the corresponding robust value function as Vi,h†,π−i,σi(s)≜maxπi′∈Δ⁡(i)Vi,hπi′×π−i,σi(s).V_i,h , _-i, _i(s) _i ∈ (A_i) V_i,h _i × _-i, _i(s). DRMG Equilibria. Due to potentially conflicting objectives among agents, the goal in a DRMG is to find an appropriate equilibrium policy under which no agent can improve its robust value through unilateral deviation (Fudenberg and Tirole, 1991). We consider three standard equilibrium notions adapted to the robust setting: robust Nash Equilibrium (NE), robust Coarse Correlated Equilibrium (CCE), and robust Correlated Equilibrium (CE), whose existence has been established in prior work (Blanchet et al., 2023). We present the definition of robust NE, robust CCE and CE below . Since computing a Nash equilibrium is generally computationally intractable (Daskalakis et al., 2009), we also consider the more computationally tractable CCE and CE notions. Definition 2 (ϵε-Robust NE). Let π∈Δ⁡(1)×⋯×Δ⁡(m)π∈ (A_1)×…× (A_m) be a product policy. It is an ϵε-robust-NE if GapNE(π,s)≜maxi∈ℳ⁡Vi,1†,π−i,σi(s1)−Vi,1π,σi(s1)≤ϵ.Gap^NE(π,s) _i \V_i,1 , _-i, _i(s_1)-V_i,1^π, _i(s_1) \≤ε. Following the formulation of Robust NE, one can relax the independent policy assumption as shown in (Liu et al., 2021) by considering correlated joint policies and define robust CCE as follows: Definition 3 (ϵε-Robust CCE). Let π∈Δ⁡()π∈ (A) be a correlated joint policy, and it is an ϵε-robust-CCE if GapCCE(π,s)≜maxi∈ℳ⁡Vi,1†,π−i,σi(s)−Vi,1π,σi(s)≤ϵ,∀s∈.Gap^CCE(π,s) _i \V_i,1 , _-i, _i(s)-V_i,1^π, _i \(s)≤ε,∀ s . We define a strategy modification ψh,s:i→i _h,s _i _i where ψ∈Ψiψ∈ _i, the set of all possible strategy modifications for agent i. Applying the strategy modification ψ to a joint policy π induces a new joint policy ψ⋄πψ π, which replaces agent i’s action aia_i with ψh,s(ai) _h,s(a_i). Definition 4 (ϵε-Robust CE). Let π∈Δ⁡()π∈ (A) be a correlated joint policy. We say π is an ϵε-robust-CE if for any s∈s , GapCE(π,s)≜maxi∈ℳ⁡maxψ∈Ψi⁡Vi,1ψ⋄π,σi(s)−Vi,1π,σi(s)≤ϵ.Gap^CE(π,s) _i \ _ψ∈ _iV_i,1^ψ π, _i(s)-V_i,1^π, _i \(s)≤ε. In the online setting, agents need to learn from direct interaction with the nominal environment P⋆P . The interaction will be deployed for K episodes, each starting from s1ks_1^k, execute the policy πkπ^k to collect new samples, and update the policy. To evaluate learning performance, we utilize the robust regret (Farhat et al., 2026), which measures the accumulated equilibrium gap. Definition 5 (Robust Regret). Let πkπ^k be a policy executed at episode k. The robust regret over K episodes is defined as RegNE,CE,CCE(K)≜∑k=1KGapNE,CE,CCE(πk,s1k).Reg^\NE,CE,CCE\(K) _k=1^KGap^\NE,CE,CCE\(π^k,s_1^k). 4 Robust MARL with Function Approximation In this subsection, we discuss the structural and computational challenges of online DRMG with general function approximation, and present our approach to overcoming them. Model-Free General Function Approximation. In DRMGs, large state–action spaces make tabular MARL methods computationally intractable (Farhat et al., 2026; Ma et al., 2023). To address this, we adopt a model-free function approximation framework, where each agent’s robust action-value function is represented using a structured hypothesis class ℱ=⨂i∈ℳℱi,whereℱi=⨂h=1Hℱi,h,F= _i F_i, _i= _h=1^HF_i,h, where for each agent i and stage h, ℱi,h=fi,h:×→ℝF_i,h=\f_i,h:S×A \ consists of candidate functions (e.g., low-dimensional parametric models like neural networks) used to approximate the robust value functions. For any fixed agent i, we denote a candidate function as fi=fi,hh=1Hf_i=\f_i,h\_h=1^H, and denote the one-step robust Bellman backup under policy π as [i,hπ,ϕ,σifi,h+1](s,)≜ri,h(s,)+i,h,ϕσi(s,)[Vi,h+1π,f], [T_i,h^π,φ, _if_i,h+1 ](s, a) r_i,h(s, a)+E_P_i,h,φ _i(s, a) [V^π,f_i,h+1 ], (2) where the policy-averaged value function is given by Vi,hπ,fi(s′)≜′∼πh(⋅|s′)[fi,h(s′,′)]V_i,h^π,f_i(s ) _ a _h(·|s )[f_i,h(s , a )]. To enable statistically and computationally efficient learning with function approximation, prior work imposes representation conditions that characterize how the function class F interacts with the underlying (robust) MDP dynamics (Russo and Van Roy, 2013; Jiang et al., 2017; Sun et al., 2019; Wang et al., 2020; Jin et al., 2021; Panaganti et al., 2022). n standard RL, these include realizability and completeness. However, in the robust setting, realizability of the optimal robust value function is often overly restrictive due to the presence of worst-case expectations. For instance, in the linear function case (Zheng and Lin, 2026), the underlying DRMG is restrictively assumed to be linear to ensure realizability. Instead, following recent advances in robust RL (Ma et al., 2022; Liu and Xu, 2024b; Liu et al., 2024; Wang et al., 2024a), realizability is effectively relaxed and can be implicitly ensured through appropriate assumptions on the dual function class used to represent the robust Bellman operator. Therefore, in our setting, we adopt the standard completeness condition to guarantee that the function class remains closed under the robust Bellman backup: Assumption 1 (Robust Completeness). Let π be any joint policy. For any i∈ℳi and fi,h+1∈ℱi,h+1f_i,h+1 _i,h+1, we have i,hπ,ϕ,σifi,h+1∈ℱi,h.T_i,h^π,φ, _if_i,h+1 _i,h. This assumption ensures closure under the robust Bellman operator and directly implies robust realizability, eliminating the need for a separate realizability assumption (Ghosh et al., 2025a). Dual Formulation of robust Bellman operator and functional optimization. Using standard duality (Yang et al., 2022b), the worst-case expectation in the robust Bellman operator is gven by i,h,ϕσi(s,)[Vi,h+1π,f]=infλ>0,ν∈ℝλσi−ν+λs′∼Ph⋆[ϕ⋆((ν−Vi,h+1π,f(s′))/λ)],E_P_i,h,φ _i(s, a)[V^π,f_i,h+1]= _λ>0,ν \λ _i-ν+λ\,E_s P_h [φ \! ((ν-V^π,f_i,h+1(s ))/λ ) ] \, (3) which yields the dual form of the robust Bellman operator [i,hπ,ϕ,σifi,h+1](s,)=ri,h(s,)−infλ,νs′∼Ph⋆[Lossi,h,ϕ(fi,s,,s′,λ,ν)], [T^π,φ, _i_i,hf_i,h+1 ](s, a)=r_i,h(s, a)- _λ,νE_s P_h [Loss_i,h,φ(f_i;s, a,s ;λ,ν) ], (4) where Lossi,h,ϕLoss_i,h,φ denotes the pointwise dual integrand for a single next-state s′s and joint action a for agent i∈ℳi : Lossi,h,ϕ(fi;s,,s′;λ,ν)≜λσi−ν+λϕ⋆(−(Vi,h+1π,f(s′)−ν)/λ).Loss_i,h,φ(f_i;s, a,s ;λ,ν) λ _i-ν+λφ (- (V^π,f_i,h+1(s )-ν ) /λ ). While the dual reformulation removes the need to optimize over transition kernels, evaluating the resulting operator still involves solving an inner optimization over (λ,ν)(λ,ν) for each state–action pair (s,)(s, a). Consequently, the computational cost scales with the size of the state space, rendering direct evaluation of the robust Bellman operator impractical in large-scale settings. To address this, we adopt a functional optimization perspective (Panaganti et al., 2022; Ghosh et al., 2025a), replacing pointwise optimization with a single global problem over the state–action space. Let μhπμ^π_h denote the state–joint action visitation distribution induced by policy π under the nominal kernel. We define the dual loss LossDuali,h,ϕ(g,fi)=(s,)∼μhπ[s′∼Ph⋆Lossi,h,ϕ(fi,s,,s′,g⁡(s,))],LossDual_i,h,φ(g;f_i)=E_(s, a) μ^π_h [E_s P_h Loss_i,h,φ(f_i;s, a,s ;g(s, a)) ], (5) where g⁡(s,)=(giλ,giν)g(s, a)=(g_i^λ,g_i^ν) denotes a dual function.22 2 Here, g ranges over an appropriate function space (e.g., ℒ1(μπ)L^1(μ^π)), and Lossi,h,ϕLoss_i,h,φ is the pointwise dual integrand as above. This formulation is equivalent to the original robust expectation under mild conditions; see (Panaganti et al., 2022; Ghosh et al., 2025a) for details. We can further show that minimizing the dual loss in (5) over ℒ1(μ,ℝ2)L^1(μ,R^2) exactly recovers the robust Bellman operator in expectation under μhπ _h^π (refer to Lemma 2 in Appendix B for details). Thus, instead of solving an inner optimization for each (s,)(s, a), we can optimize for a single dual function that captures worst-case transitions globally. This observation allows us to replace the intractable pointwise optimization with a single functional optimization of the empirical dual loss of (5), LossDuali,h,ϕ^(g,fi) LossDual_i,h,φ(g;f_i). We further introduce a structured function class =⨂i,hi,hG= _i,hG_i,h, where each i,h=gi=(giλ,giν):×→Θϕ⊆ℝ2G_i,h=\g_i=(g^λ_i,g^ν_i):S×A→ _φ ^2\, to replace ℒ1(μ,ℝ2)L^1(μ,R^2) and to approximate the dual solution. We adopt the following assumption to quantify the approximation error (Panaganti et al., 2022; Panaganti et al., 2024; Ghosh et al., 2025a). Assumption 2. There exists a constant εdualapprox ^approx_dual, such that for any agent i∈ℳi , horizon h∈[H]h∈[H], function fi∈ℱf_i , and policy π, infg∈iLossDuali,h,ϕ(g,fi)−infg∈ℒi1(μπ,ℝ2)LossDuali,h,ϕ(g,fi)≤εdualapprox. _g _iLossDual_i,h,φ(g;f_i)- _g ^1_i(μ^π;R^2)LossDual_i,h,φ(g;f_i)\;≤\; ^approx_dual. Empirical dual minimization and operator approximation. Based on the discussion above, we can approximate the robust Bellman operator by first minimizing the empirical dual loss over G: gi^f=arg⁡ming∈LossDual^i,h,ϕ(g,fi), g_i^f= _g LossDual_i,h,φ(g;f_i), (6) and defining the approximate Bellman update by plugging gi^if g_i_i^f into the dual formulation: [^i,hπ,ϕ,σifi](s,)=ri,h(s,)−s′∼Ph⋆[Lossi,h,ϕ(fi,s,,s′,gi^if(s,))]. [ T^π,φ, _i_i,hf_i ](s, a)=r_i,h(s, a)-E_s P_h [Loss_i,h,φ(f_i;s, a,s ; g_i_i^f(s, a)) ]. (7) To characterize the approximation of the true robust Bellman operator, we adopt the following assumption, which holds for all considered uncertainty sets (Ghosh et al., 2025a). Assumption 3. We assume the existence of a compact set Θϕ⊆ℝ+×ℝ _φ _+×R and a finite constant Bϕ(σi)B_φ( _i) such that the dual loss remains uniformly bounded. In particular, for all h∈[H]h∈[H], i∈ℳi , any fi,h+1∈ℱi,h+1f_i,h+1 _i,h+1, all (λ,ν)∈Θϕ(λ,ν)∈ _φ, and every (s,,s′)∈×(s, a,s ) ×A×S, we have |Lossi,h,ϕ(fi,s,,s′,λ,ν)|≤Bϕ(σi). |Loss_i,h,φ(f_i;s, a,s ;λ,ν) |≤ B_φ( _i). Additionally, we assume that the dual function class G is restricted to take values within the set Θϕ _φ. We then develop the following error quantification, whose proof is deferred to Appendix C. Lemma 1 (Uniform dual-operator approximation under adaptive data). Fix an agent i∈ℳi and a stage h∈[H]h∈[H]. Let πtt=1n\π^t\_t=1^n be a predictable sequence of behavior policies, and let hD_h be the stage-h dataset collected by executing πtt=1n\π^t\_t=1^n over n consecutive episodes. Define the random historical occupancy mixture μ¯hn:=∑t=1nμhπt/n. μ_h^\,n:= _t=1^n _h^π^t/n. Assume Assumptions 1–3 hold and let the function classes ℱiF_i and iG_i be finite. For each π∈Πpurπ∈ pur and fi,h+1∈ℱi,h+1f_i,h+1 _i,h+1, let g^if,π g_i^\,f,π be an empirical minimizer of the dual loss on hD_h as defined in (6). Then, for any δ∈(0,1)δ∈(0,1), with probability at least 1−δ1-δ: supfi,h+1∈ℱi,h+1,π∈Πpur‖i,hπ,ϕ,σifi,h+1−g^if,π,ϕ,σifi,h+1‖1,μ¯hn _ subarraycf_i,h+1 _i,h+1,π∈ pur subarray \|T_i,h^π,φ, _if_i,h+1-T_ g_i^\,f,π^π,φ, _if_i,h+1 \|_1, μ_h^\,n ≤CBϕ(σi)log⁡(|ℱi,h+1‖i,h‖Πpur|/δ)/n+εdualapprox. ≤ CB_φ( _i) \! (|F_i,h+1||G_i,h|| pur|/δ ) /n+ _ dual approx. Prior single-agent results (Panaganti et al., 2022; Panaganti et al., 2024) establish similar guarantees under a fixed offline distribution, whereas our result holds uniformly over all policies and their induced state–joint action distributions in the multi-agent setting. Lemma 1 shows that the empirical dual optimization uniformly approximates the robust Bellman operator under the ℒi1(μπ,ℝ2)L^1_i(μ^π;R^2) norm, with the approximation error controlled globally over the distribution μπμ^π rather than pointwise over individual (s,)(s, a). This global control enables the construction of robust confidence sets and provides a unified error term underlying both our algorithm design and theoretical analysis. 5 Interactive Robust Multi-Agent Reinforcement Learning We then present our Robust Multi-Agent Exploration with ϕφ-Divergence Uncertainty Set (RoMEX-ϕφ) (Algorithm 1). In each episode, we will learn value function approximations via fitted methods, where the fitting error is defined through our empirical robust operator. We describe our algorithm in this section, and defer our discussions on algorithmic novelties to Appendix 6.1. Algorithm 1 Robust Multi-Agent Exploration with ϕφ-Divergence Uncertainty Set (RoMEX-ϕφ) 1: Input: Number of episodes K, function classes ℱ=⊗i=1MℱiF= _i=1^MF_i and =⊗i=1MiG= _i=1^MG_i, regularization parameter β>0β>0, equilibrium oracle EqOracle⁡(⋅,NE/CCE/CE)EqOracle(·;NE/CCE/CE) 2: Initialize: h0←∅D_h^0← for all h∈[H]h∈[H] 3: for k=1,2,…,Kk=1,2,…,K do 4: for each pure joint policy π∈Πpurπ∈ ^pur do 5: Set fi,H+1k,π≡0f_i,H+1^k,π≡ 0 for each agent i∈ℳi . 6: for h=H,H−1,…,1h=H,H-1,…,1 do 7: Compute empirical dual minimizer g^i,hk,π g_i,h^k,π as in (8) for each agent i∈ℳi . 8: Compute plug-in robust Bellman fit fi,hk,πf_i,h^k,π (by Step 2) for each agent i∈ℳi . 9: Compute regularised robust payoff U^ik(π) U_i^k(π) as given in (10) for i∈ℳi . 10: Construct the normal-form game ^k:=(Πipuri∈ℳ,U^iki∈ℳ) ^k:= (\ _i^pur\_i ,\ U_i^k\_i ) and compute equilibrium policy πk←EqOracle⁡(^k,NE/CCE/CE)π^k ( ^k;NE/CCE/CE). 11: Execute πkπ^k in the nominal environment P⋆P for one episode and collect τk=(shk,hk,rhk,sh+1k)h=1Hτ^k=\(s_h^k, a_h^k,r_h^k,s_h+1^k)\_h=1^H and update dataset hk←hk−1∪(shk,hk,sh+1k)D_h^k _h^k-1∪\(s_h^k, a_h^k,s_h+1^k)\ for h∈[H]h∈[H]. 12: Return πout≔πkk=1Kπ^out \π^k\_k=1^K or 1K∑k=1Kπk 1K _k=1^Kπ^k. Fitted estimation. Let Πipur _i^pur denote the class of pure policies for agent i, and let Πpur:=Π1pur×⋯×Πmpur ^pur:= _1^pur×·s× _m^pur be the set of pure joint policies. Fix an episode k, a candidate joint policy π∈Πpurπ∈ ^pur, and an agent i∈[M]i∈[M]. The algorithm computes a fitted robust value sequence fi,hk,πh=1H+1\f_i,h^k,π\_h=1^H+1 via backward dynamic programming on the collected dataset, with terminal condition fi,H+1k,π≡0f_i,H+1^k,π≡ 0 (Line 5). For each stage h=H,…,1h=H,…,1, two coupled steps are performed. Step 1: Empirical dual minimization (Line 7). Using the stage-h dataset hk−1D_h^k-1 collected up to episode k−1k-1, we compute the empirical dual minimizer g^i,hk,π∈arg⁡ming∈i,hLossDual^i,h,ϕk−1(g,fi,h+1k,π,π), g_i,h^k,π∈ _g _i,h LossDual_i,h,φ^k-1(g;f_i,h+1^k,π,π), (8) where LossDual^i,h,ϕk−1 LossDual_i,h,φ^k-1 is the empirical version of the dual loss defined in (5) using hk−1D_h^k-1. Step 2: Robust Bellman fitting with plug-in residual (Line 8). Once the dual minimizer g^i,hk,π g_i,h^k,π has been obtained, the algorithm uses it to define a plug-in empirical approximation of the robust Bellman operator. It then computes the next-stage h fitted value function by minimizing the empirical plug-in robust Bellman residual: fi,hk,π∈arg⁡min⁡∑(s,,s′)∈hk−1f∈ℱi,h⁡δ^i,h(fi,g^i,hk,π,π,s,,s′)2,f_i,h^k,π∈ _f _i,h _(s, a,s ) _h^k-1 δ_i,h(f_i; g_i,h^k,π;π;s, a,s )^2, where δ^i,h δ_i,h is the plug-in empirical discrepancy defined as δ^i,h(fi,gi,π,s,,s′):=fi(s,)−ri,h(s,)+Lossi,h,ϕ(fi,s,,s′,gi,π). δ_i,h(f_i;g_i;π;s, a,s ):=f_i(s, a)-r_i,h(s, a)+Loss_i,h,φ(f_i;s, a,s ;g_i,π). (9) Note that δ^i,h δ_i,h penalizes hypotheses that do not match the robust Bellman equation well on the historical data. The dual minimizer enters precisely through this term: the Bellman discrepancy is no longer computed using the nominal transition expectation, but through the dual-induced robust integrand. Intuitively, the dual minimizer g^i,hk,π g_i,h^k,π converts the robust Bellman backup into a tractable empirical target, and the fitted value function fi,hk,πf_i,h^k,π best fits this target on the observed data. Regularized robust payoff (Line 10). For each episode k∈[K]k∈[K], agent i∈ℳi , and candidate pure joint policy π∈Πpurπ∈ ^pur, we define the revised empirical payoff by U^ik(π):=fi,1k,π(s1k)−β^ik−1(fik,π,g^ik,π,π), U_i^k(π):=f_i,1^k,π(s_1^k)-β K_i^k-1\! (f_i^k,π; g_i^k,π;π ), (10) where β>0β>0 is the regularization parameter, g^i,hk,π g_i,h^k,π is the empirical dual minimizer in (6) for fi,h+1k,πf_i,h+1^k,π and π, and ^ik−1(fi,gi,π) K_i^k-1(f_i;g_i;π) denotes the cumulative centered empirical robust discrepancy: ∑h=1H[ _h=1^H [ ∑j=1k−1δ^i,h(fi;gi,h;π;shj,hj,sh+1j)2−inffi′∈ℱi∑j=1k−1δ^i,h(fi′;gi,h;π;shj,hj,sh+1j)2], _j=1^k-1 δ_i,h(f_i;g_i,h;π;s_h^j, a_h^j,s_h+1^j)^2- _f _i _i _j=1^k-1 δ_i,h(f _i;g_i,h;π;s_h^j, a_h^j,s_h+1^j)^2 ], (11) with ^i0(fi,gi,π)=0 K_i^0(f_i;g_i;π)=0. The first term in (10) favors policies with large estimated robust value, while the second penalizes large empirical robust Bellman discrepancy. The resulting objective acts as a conservative surrogate value, trading off value estimation and Bellman consistency rather than introducing an explicit optimistic bonus. Similar to MAMEX (Xiong et al., 2024), this discrepancy term serves as a global regularizer, and in our setting, it controls the gap between empirical robust Bellman error and robust value prediction via the robust MADC. Policy selection and data collection (Line 12-15). We then define a normal-form game ^k:=(Πipuri∈[M],U^iki∈[M]), ^k:= (\ _i^pur\_i∈[M],\ U_i^k\_i∈[M] ), whose pure-strategy profile is π∈Πpurπ∈ ^pur and whose payoff to agent i at profile π is U^ik(π) U_i^k(π). Once the normal-form game ^k ^k is formed, the algorithm computes the desired equilibrium through some oracle: πk←EqOracle⁡(^k,NE/CCE/CE).π^k ( ^k;NE/CCE/CE). As we mainly focus on statistical complexity, we assume such oracles. Specifically, the NE oracle is generally NP-hard (Daskalakis, 2013), while CE and CCE can be computed with polynominal complexity (Zheng and Lin, 2026). The policy πkπ^k is then executed for one episode in the nominal environment P⋆P , producing the trajectory τk=(shk,hk,rhk,sh+1k)h=1H.τ^k=\(s_h^k, a_h^k,r_h^k,s_h+1^k)\_h=1^H. For each stage h, the transition tuple (shk,hk,sh+1k)(s_h^k, a_h^k,s_h+1^k) is appended to the dataset hk−1D_h^k-1 to form hkD_h^k, for the following algorithm executions. The final output is either the sequence πkk=1K\π^k\_k=1^K or the uniform mixture π¯=1K∑k=1Kπk π= 1K _k=1^Kπ^k, depending on the equilibrium notion under consideration. 6 Theoretical Guarantee We now develop our regret analysis. Inspired by the non-robust studies (Xiong et al., 2024), we first introduce a fundamental complexity measure, named robust multi-agent decoupling coefficient (Robust MADC), to quantify the inherent difficulty of exploration in DRMGs. This notion will enable us to develop our analysis from an optimization perspective. Definition 6 (Robust Multi-Agent Decoupling Coefficient). For each agent i∈ℳi , policy π∈Πpurπ∈ ^pur, and stage h∈[H]h∈[H], let μhπμ^π_h denote the stage-h state-joint action visitation distribution induced by policy π under the nominal kernel P⋆P . Then, for any i∈ℳi , μ,c>0μ,c>0, fik∈ℱif_i^k _i, and joint policy πkk=1K\π^k\_k=1^K, the robust MADC for agent i∈ℳi is the smallest constant 1≤di,MADCrob<∞1≤ d_i,MADC^rob<∞ such that: ∑k=1K(Vi,1πk,fik(s1k)−Vi,1πk,σi(s1k))≤1μ∑k=1K∑s=1k−1ei,1s(fik,πk)+μdi,MADCrob+cHdi,MADCrob, _k=1^K(V_i,1^π^k,f^k_i(s_1^k)-V_i,1^π^k, _i(s_1^k))≤ 1μ _k=1^K _s=1^k-1e_i,1^s(f_i^k,π^k)+μ\,d_i,MADC^rob+cHd_i,MADC^rob, (12) where ei,1s(f,π)e_i,1^s(f,π) is the discrepancy function defined as the mean-squared robust Bellman error: ei,1s(f,π)≜∑h=1H(sh,h)∼μhπs[(fi,h(sh,h)−[i,hπ,ϕ,σifi,h+1](sh,h))2],∀s∈[K],∀f∈ℱi.e_i,1^s(f,π) _h=1^HE_(s_h, a_h) _h^π^s [ (f_i,h(s_h, a_h)-[T_i,h^π,φ, _if_i,h+1](s_h, a_h) )^2 ],∀ s∈[K],∀ f _i. Interpretation of robust MADC. Our robust MADC extends the decoupling coefficient in (Dann et al., 2021; Xiong et al., 2024) to robust settings, capturing the joint difficulty of strategic interactions and transition uncertainty. At a high level, di,MADCrobd^rob_i,MADC quantifies the decoupling between (i) the training error, given by the cumulative empirical robust Bellman residual, and (i) the prediction error, defined as the gap between the learned value and the true robust value under worst-case dynamics. Definition 6 ensures that the cumulative prediction error is controlled by the cumulative training error up to a term governed by di,MADCrobd^rob_i,MADC. Unlike the non-robust setting (Xiong et al., 2024), the prediction error is evaluated under adversarial dynamics while the training error is computed from nominal interaction data. Thus, di,MADCrobd^rob_i,MADC characterizes how effectively nominal interaction data controls value prediction under worst-case dynamics, capturing the intrinsic exploration complexity of DRMGs. Structural perspective. Beyond the decoupling interpretation, robust MADC also provides a structural view: a smaller di,MADCrobd^rob_i,MADC indicates that small empirical robust Bellman error translates into small value prediction error, reflecting better alignment of the function class with the robust Bellman operator. This parallels Bellman-error-based complexity measures such as the BE dimension, while additionally capturing the mismatch between nominal data and adversarial evaluation. This perspective extends naturally to structured function classes, such as bilinear models (Du et al., 2021b), where low-dimensional representations enable accurate estimation of the robust Bellman operator from nominal data, thereby controlling the discrepancy terms, yielding a bounded robust MADC, and improving generalization under worst-case dynamics. In previous RL studies, the (multi-agent) Bellman-Eluder (MABE) dimension is another widely used complexity measure in RL (Jin et al., 2021; Ghosh et al., 2025a). In the non-robust setting, a low BE dimension implies a low MADC (Xiong et al., 2024), making MADC a more general complexity measure. In the robust setting, the robust BE dimension was introduced for single-agent RL in (Ghosh et al., 2025a) and can be extended to multi-agent settings. Our robust MADC instead directly quantifies how robust Bellman errors estimated from nominal interactions control value prediction under uncertain transition kernels. Moreover, Theorem 3 shows that a low robust MABE, extending the robust BE dimension of (Ghosh et al., 2025a), implies a low robust MADC, establishing robust MADC as a more general complexity measure. Extending results from the non-robust setting (Jin et al., 2021; Jin et al., 2022), the inherent robust MABE can be substantially smaller than the tabular complexity S∏i=1mAiS _i=1^mA_i for structured function classes. By Theorem 3, the robust MADC can therefore also be much smaller than S∏i=1mAiS _i=1^mA_i under function approximation. Since our sample-complexity guarantees are characterized by the robust MADC, this enables improved scalability over tabular approaches. We now present the theoretical guarantees of RoMEX-ϕφ; full details are deferred to Appendix D. Theorem 1. Assume Assumptions 1–3 and the function classes ℱiF_i and dual function class iG_i are finite33 3 This assumption is made for technical simplicity. While extensions to infinite function classes may be possible using covering arguments (e.g., ϵε-nets (Panaganti et al., 2022)), establishing such results rigorously in the robust multi-agent setting remains non-trivial and is left for future work.. Let πkk=1K\π^k\_k=1^K be policies produced by Algorithm 1 with different oracles, and denote dMADCrob,max=maxi∈ℳ⁡di,MADCrobd_MADC^rob, = _i d_i,MADC^rob, Bϕmax=maxi∈ℳ⁡Bϕ(σi)B _φ= _i B_φ( _i) and ιℱ,δ:=log⁡(|ℱ‖Πpur|KHmδ) _F,G^δ:= \! ( |F||G|| ^pur|KHmδ ). Then, with probability at least 1−δ1-δ, the corresponding robust regret satisfies Reg⁡(K)≤⁡(dMADCrob,maxK+HdMADCrob,max+H(Bϕmax)2Kιℱ,δ+εdualapprox). (K) \! (d_MADC^rob, K+Hd_MADC^rob, +H(B_φ )^2 K _F,G^δ+ _dual^approx ). Theorem 1 shows that RoMEX-ϕφ efficiently learns robust equilibria in general-sum DRMGs with general function approximation. The bound decomposes into four terms: (i) the exploration term dMADCrob,maxKd_MADC^rob, K, governed by the robust MADC, which—analogous to (Xiong et al., 2024)—links cumulative in-sample robust Bellman discrepancy to out-of-sample robust value prediction error and thus captures the intrinsic difficulty of exploration. Optimizing the decoupling parameter yields the K K rate. While small cumulative discrepancy ensures that the exploration-induced component of regret is sublinear, the overall regret additionally depends on the robust estimation term arising from approximating the worst-case Bellman operator from nominal data; (i) a burn-in horizon-dependent term HdMADCrob,maxH\,d_MADC^rob, , which reflects horizon-dependent initialization effects and does not affect the asymptotic rate; (i) a robust estimation term H(Bϕmax)2Kιℱ,δH\,(B_φ )^2 K\, _F,G^δ, which is unique to the robust setting. This term captures the statistical cost of estimating the robust Bellman operator from nominal data via the dual formulation. Here, BϕmaxB_φ controls the dual loss envelope, ιℱ,δ _F,G^δ reflects the complexity of the primal and dual function classes; and (iv) the dual approximation error εdualapprox _dual^approx, which captures the representational error of the dual function class and vanishes under realizability of G. Overall, the bound reveals two core challenges: exploration under strategic interaction, captured by dMADCrob,maxd_MADC^rob, and robust Bellman estimation under distributional mismatch, reflected in by (Bϕmax)2ιℱ,δ(B_φ )^2 _F,G^δ. Refer to subsection 6.1 for a detailed discussion on the technical novelties of RoMEX-ϕφ. As a consequence of our regret guarantees, applying a standard online-to-batch conversion (Cesa-Bianchi et al., 2001) gives the first sample-complexity guarantees for learning an ε -optimal policy with RoMEX-ϕφ. As corollary, we obtain regret bound under total variation, χ2χ^2- and KL-divergence uncertainty sets, presented in Appendix D.1 and Appendix E. To our knowledge, these results constitute the first sample-complexity characterization for distributionally robust MARL with general function approximation. Crucially, the bounds are governed by the robust MADC and contain no explicit dependence on the state-space size |||S| or the joint action-space size |||A|. Thus, compared with tabular robust MARL, our analysis replaces the unfavorable joint-action-space dependence with an intrinsic function-class complexity, improving scalability and mitigating the curse of multi-agency (Shi et al., 2024b; Farhat et al., 2026). To further interpret the sharpness and generality of our results, we specialize the general bound to representative settings. In particular, we consider the tabular case, robust MABE/BE-type characterizations, and linear function approximation. A more detailed and comprehensive comparison with prior work is provided in subsection 6.2. 6.1 Novelties of RoMEX-ϕφ In this section, we discuss both the algorithm and technical novelties of RoMEX-ϕφ. Algorithm novelties. Our algorithm is conceptually motivated by two complementary lines of work: the MAMEX framework (Xiong et al., 2024), which constructs payoffs using nominal Bellman errors but does not account for distributional uncertainty, and the RFL method (Ghosh et al., 2025a), which uses dual-based robust Bellman residuals to enable worst-case value estimation in single-agent robust RL. However, neither approach directly applies to online DRMGs with general function approximation: MAMEX lacks robustness, while RFL does not address strategic multi-agent interactions or equilibrium-based payoff construction. This creates a central algorithmic challenge: how to construct payoffs that are simultaneously robust, statistically estimable from nominal data, and suitable for equilibrium-based exploration. (1) Robust payoff design via dual fitted learning. The central novelty of RoMEX-ϕφ is the construction of a robust payoff function that combines worst-case value estimation with exploration-aware regularization. In nominal MARL, payoff construction can rely on Bellman consistency under the data-generating dynamics. In contrast, in DRMGs, policy values are defined through adversarial transition kernels that are not directly observed from nominal interaction data. To overcome this mismatch, RoMEX-ϕφ uses a dual-based robust fitted-learning procedure: for each candidate joint policy, a functional dual minimizer is estimated from nominal data and then plugged into a fitted robust Bellman recursion. This produces a tractable empirical surrogate for the robust value. The resulting payoff is then regularized by a centered empirical robust discrepancy, which penalizes the excess robust Bellman residual relative to the best in-class fit. Thus, the payoff does not merely subtract a raw residual; it subtracts a centered function-class-level measure of robust Bellman inconsistency. (2) Function-class-level uncertainty quantification under robustness. The centered empirical robust discrepancy provides a global uncertainty quantification mechanism tailored to robust multi-agent learning. Since it is defined as an excess empirical robust Bellman error relative to the best hypothesis in the class, it measures how much additional robust Bellman inconsistency is incurred by the fitted value function beyond the best available in-class approximation. This is crucial under general function approximation, where uncertainty cannot be captured by tabular state-action bonuses. Instead, RoMEX-ϕφ uses this centered discrepancy as a global regularizer in the payoff, and the regret analysis relates this empirical robust discrepancy to robust value prediction error through the robust MADC. (3) Coupling robust evaluation with equilibrium-based learning. RoMEX-ϕφ integrates dual-based robust policy evaluation with equilibrium-based learning by using the regularized robust payoffs to define a surrogate normal-form game over candidate pure joint policies. The equilibrium oracle is then applied to this surrogate game, so the selected policy is guided not only by estimated robust value but also by the centered robust Bellman discrepancy. This aligns equilibrium computation with the robust learning objective: agents are encouraged to choose policies whose robust values are large and whose fitted robust Bellman recursions are statistically reliable under the accumulated nominal data. Overall, the novelty of RoMEX-ϕφ lies in resolving the incompatibility between MAMEX-style payoff-based exploration and dual-based robust Bellman evaluation. The key algorithmic contribution is a regularized robust payoff that combines dual fitted robust value estimation with a centered empirical robust discrepancy, enabling purely online equilibrium learning in general-sum DRMGs under general function approximation. Technical novelties. The primary technical challenge in our setting arises from the fact that the robust Bellman operator is defined through a worst-case transition kernel and is therefore not directly observable from nominal interaction data. This creates a fundamental mismatch between the data-generating distribution and the target of optimization, a difficulty that does not arise in non-robust MARL. To address this, our analysis develops a unified framework that simultaneously controls three sources of error: (i) approximation error induced by plug-in dual-based robust Bellman operators, (i) statistical generalization error under function approximation, and (i) strategic deviation error due to multi-agent interactions. A key ingredient is the use of a centered empirical robust discrepancy, which enables a telescoping control of estimation errors across episodes. This centering is critical: it eliminates systematic bias introduced by the evolving plug-in operators and prevents linear accumulation of estimation errors over time. Building on this, we introduce a robust extension of the MADC, which quantifies how empirical robust Bellman error—computed from nominal data—controls value prediction under adversarial transition dynamics. This leads to a regret bound that explicitly captures the interplay between function approximation, equilibrium computation, and distributional robustness in a fully online setting. To the best of our knowledge, this provides the first intrinsic-complexity characterization of exploration in general-sum DRMGs under ϕφ-divergence uncertainty sets. 6.2 Comparison with Prior Works Table 1: Comparison of sample-complexity guarantees across general function approximation, tabular models, BE/MABE-type characterizations, and linear function approximation in multi-agent RL. We focus on online model-free methods whenever possible. For compactness, let ιℱ,δ:=log⁡(|ℱ‖Πpur|KHm/δ) _F,G^δ:= \! (|F||G|| ^pur|KHm/δ ), Ajoint:=∏i=1mAiA_joint:= _i=1^mA_i, and define the divergence-dependent factors ΨTV(σ):=(min⁡H,1/σmin)4,Ψχ2(σ):=(1+σmax)4,ΨKL(σ):=max⁡1,σmax4. _TV(σ):= ( \H,1/ _ \ )^4, _χ^2(σ):=(1+ _ )^4, _KL(σ):= \1, _ ^4\. For notational simplicity, we further denote dMABErob,max:=dMABErob,max(ℱ,Π,1/K)d_MABE^rob, :=d_MABE^rob, (F, ,1/K) (robust MABE-dimension) and dMABE:=dMABE(ℱ,Π,1/K)d_MABE:=d_MABE(F, ,1/K) (non-robust MABE-dimension). Our bounds keep the robust-to-nominal concentrability coefficient CcovC_cov explicit. Setting Method Robust Divergence Sample complexity General Online (Xiong et al., 2024) No – ~(m2H3dMADC2ιℱδε−2) O\! (m^2H^3d_MADC^2\, _F^δ\, ^-2 ) Online RoMEX-ϕφ (Ours) Yes TV ~(max⁡H2(dMADCrob,max)2ε2,Ccov2H7ΨTV(σ)ιℱ,δε2) O\! ( \ H^2(d_MADC^rob, )^2 ^2, C_cov^2H^7 _TV(σ)\, _F,G^δ ^2 \ ) χ2χ^2 ~(max⁡H2(dMADCrob,max)2ε2,H7Ψχ2(σ)ιℱ,δε2) O\! ( \ H^2(d_MADC^rob, )^2 ^2, H^7 _χ^2(σ)\, _F,G^δ ^2 \ ) KL ~(max⁡H2(dMADCrob,max)2ε2,H7ΨKL(σ)ιℱ,δε2) O\! ( \ H^2(d_MADC^rob, )^2 ^2, H^7 _KL(σ)\, _F,G^δ ^2 \ ) Lower bound – – N/A Tabular Online (Ma et al., 2023) Yes KL ~(ε−2H5S(maxi⁡Ai)2) O\! ( ^-2H^5S( _iA_i)^2 ) with an oracle Online (Farhat et al., 2026) Yes TV ~(ε−2H3SAjointmin⁡σmin−1,H) O\! ( ^-2H^3SA_joint \ _ ^-1,H\ ) KL ~(ε−2σmin−2(Pmin⋆)−1H5e2H2SAjoint) O\! ( ^-2 _ ^-2(P_ )^-1H^5e^2H^2SA_joint ) Online RoMEX-ϕφ (Ours) Yes TV ~(max⁡H2S2Ajoint2ε2,Ccov2H7ΨTV(σ)SAjointε2) O\! ( \ H^2S^2A_joint^2 ^2, C_cov^2H^7 _TV(σ)SA_joint ^2 \ ) χ2χ^2 ~(max⁡H2S2Ajoint2ε2,H7Ψχ2(σ)SAjointε2) O\! ( \ H^2S^2A_joint^2 ^2, H^7 _χ^2(σ)SA_joint ^2 \ ) KL ~(max⁡H2S2Ajoint2ε2,H7ΨKL(σ)SAjointε2) O\! ( \ H^2S^2A_joint^2 ^2, H^7 _KL(σ)SA_joint ^2 \ ) Lower bound (Generative) (Shi et al., 2024b) Yes TV Ω~(ε−2H3S(maxi⁡Ai)min⁡σmin−1,H) \! ( ^-2H^3S( _iA_i) \ _ ^-1,H\ ) MABE Dimension Online (Xiong et al., 2024) No – ~(m2H3dMABE2ιℱδε−2) O\! (m^2H^3d_MABE^2\, _F^δ\, ^-2 ) Online (Jin et al., 2022) No – ~(H3dMABE2log⁡(|ℱ|HdBE/δ)ε−2) O\! (H^3d_MABE^2 (|F|Hd_BE/δ) ^-2 ) Online RoMEX-ϕφ (Ours) Yes TV ~(max⁡H4(dMABErob,max)2ε2,Ccov2H7ΨTV(σ)ιℱ,δε2) O\! ( \ H^4(d_MABE^rob, )^2 ^2, C_cov^2H^7 _TV(σ)\, _F,G^δ ^2 \ ) χ2χ^2 ~(max⁡H4(dMABErob,max)2ε2,H7Ψχ2(σ)ιℱ,δε2) O\! ( \ H^4(d_MABE^rob, )^2 ^2, H^7 _χ^2(σ)\, _F,G^δ ^2 \ ) KL ~(max⁡H4(dMABErob,max)2ε2,H7ΨKL(σ)ιℱ,δε2) O\! ( \ H^4(d_MABE^rob, )^2 ^2, H^7 _KL(σ)\, _F,G^δ ^2 \ ) Linear Online (Xiong et al., 2024) No – ~(m2H3dlin2ε−2) O\! (m^2H^3d_lin^2 ^-2 ) Online (Wang et al., 2023) No – ~(m2H6dlin4(maxi⁡Ai5)ε−2) O\! (m^2H^6d_lin^4( _iA_i^5) ^-2 ) Online (Zheng and Lin, 2026) Yes TV ~(H3dlin2(min⁡H,1/σmin)2ε−2) O\! (H^3d_lin^2( \H,1/ _ \)^2 ^-2 ) Online RoMEX-ϕφ (Ours) Yes TV ~(max⁡γ2Ccov2H4dlin2ε2,Ccov2H7ΨTV(σ)ιlinδε2) O\! ( \ γ^2C_cov^2H^4d_lin^2 ^2, C_cov^2H^7 _TV(σ)\, _lin^δ ^2 \ ) χ2χ^2 ~(max⁡γ2H4dlin2ε2,H7Ψχ2(σ)ιlinδε2) O\! ( \ γ^2H^4d_lin^2 ^2, H^7 _χ^2(σ)\, _lin^δ ^2 \ ) KL ~(max⁡γ2H4dlin2ε2,H7ΨKL(σ)ιlinδε2) O\! ( \ γ^2H^4d_lin^2 ^2, H^7 _KL(σ)\, _lin^δ ^2 \ ) Lower bound (Zheng and Lin, 2026) Yes TV Ω~(H2dlin2(min⁡H,1/σmin)2ε−2) \! (H^2d_lin^2( \H,1/ _ \)^2 ^-2 ) Table 1 summarizes the sample-complexity guarantees of RoMEX-ϕφ and positions them relative to existing results across four regimes: general function approximation, tabular models, BE/MABE-type characterizations, and linear structure. The comparison focuses on online model-free methods in general-sum Markov games, while also including two-player zero-sum BE-dimension benchmarks as reference points. The main message is that our guarantees extend the intrinsic-complexity viewpoint of non-robust MARL to distributionally robust MARL: exploration is governed by a robust decoupling coefficient, while robustness contributes additional terms reflecting the geometry of the uncertainty set and the mismatch between nominal data and worst-case evaluation. General function approximation: robust extension of intrinsic complexity characterizations. In non-robust multi-agent RL with general function approximation, the sharpest known guarantees are naturally phrased in terms of intrinsic complexity rather than explicit dependence on (S,A)(S,A). In particular, the MAMEX framework shows that sample complexity scales as ~(m2H3dMADC2ιℱδε−2), O\! (m^2H^3d_MADC^2\, _F^δ\, ^-2 ), where dMADCd_MADC controls how empirical Bellman errors translate into value-prediction errors across agents (Xiong et al., 2024). This extends BE-style complexity ideas from single-agent and two-player zero-sum settings to general-sum games, where there is no single Bellman operator and agents’ value functions are coupled through equilibrium constraints. Our result extends this intrinsic characterization to the distributionally robust setting. The revised sample-complexity bound has the generic form ~(max⁡H2(dMADCrob,max)2ε2,H7Ψϕ(σ)ιℱ,δε2). O\! ( \ H^2(d_MADC^rob, )^2 ^2, H^7 _φ(σ)\, _F,G^δ ^2 \ ). The first term is the robust exploration term, governed by the robust MADC. It is the analogue of the non-robust decoupling term, but now the prediction error is measured against robust values induced by adversarial transition kernels. The second term is unique to the robust setting: it is the cost of estimating the plug-in robust Bellman operator from nominal interaction data. The factor Bϕ(σ)B_φ(σ), equivalently summarized through Ψϕ(σ) _φ(σ), comes from the dual-loss envelope associated with the ϕφ-divergence uncertainty set. Moreover, CcovC_cov captures the robust-to-nominal distribution mismatch for TV-case. Thus, robustness does not simply multiply the MADC term; instead, it introduces a separate robust-estimation term in addition to the exploration complexity. BE dimension, MABE, and robust MADC. A central conceptual distinction behind Table 1 is the difference between BE-dimension-based analyses and MADC-based analyses. In two-player zero-sum Markov games, the analysis of (Jin et al., 2022) relies on a single Nash value function induced by a minimax Bellman operator. This unified operator allows exploration to be controlled through one sequence of Bellman errors, yielding sample complexity of order ~(H3dMABE2log⁡(|ℱ|HdMABE/δ)ε−2). O\! (H^3d_MABE^2 (|F|Hd_MABE/δ) ^-2 ). In general-sum games, however, this structure is absent: each agent has its own value function, equilibrium behavior is defined through coupled unilateral deviations, and there is no single Bellman fixed point that governs all agents. This is precisely the role of MADC. It quantifies how multi-agent Bellman prediction errors propagate into equilibrium-relevant value errors across agents. Our robust extension preserves this structure but replaces the nominal Bellman residual by the robust Bellman residual. The robust MADC therefore captures the structural difficulty of using nominal Bellman training errors to control robust value prediction under worst-case dynamics. Theorem 3 further shows that low robust MABE implies low robust MADC: dMADCrob,max=~(HdMABErob,max).d_MADC^rob, = O\! (Hd_MABE^rob, ). Substituting this into the general theorem yields the BE/MABE rows in Table 1. Importantly, the robust MABE-to-MADC implication is a structural statement, where we introduce CcovC_cov for the TV case in the final sample complexity through the robust-estimation term in Theorem 2. Tabular setting: recovering uncertainty-dependent scaling. The tabular setting provides the clearest interpretation of how the uncertainty radius affects the sample complexity. Under full tabular function classes, the robust MADC satisfies dMADCrob,max=~(SAjoint),d_MADC^rob, = O(SA_joint), and the logarithmic complexity of the primal and dual classes scales as log⁡(|ℱ|||)=~(SAjoint). (|F||G|)= O(SA_joint). Substituting these into the generic bound gives ~(max⁡H2S2Ajoint2ε2,H7Ψϕ(σ)SAjointε2), O\! ( \ H^2S^2A_joint^2 ^2, H^7 _φ(σ)SA_joint ^2 \ ), which is reported in Table 1. For TV-case, we get the bound as ~(max⁡H2S2Ajoint2ε2,Ccov2H7Ψϕ(σ)SAjointε2). O\! ( \ H^2S^2A_joint^2 ^2, C^2_covH^7 _φ(σ)SA_joint ^2 \ ). This specialization should be interpreted carefully. For the TV-case, the coefficient CcovC_cov is a robust-to-nominal distribution-shift coefficient as mentioned in Definition 7, not a purely combinatorial quantity such as SAjointSA_joint. Therefore we keep it explicit rather than replacing it by a tabular dimension. The key point is that the tabular dependence emerges from the same robust MADC framework used for general function approximation, rather than from a separate tabular analysis. Moreover, the divergence factor becomes ΨTV(σ)=(min⁡H,1/σmin)4 _TV(σ)=( \H,1/ _ \)^4, recovering the same qualitative contraction dependence that appears in recent tabular robust MDP and DRMG analyses. For χ2χ^2 and KL uncertainty, the same framework yields the factors (1+σmax)4(1+ _ )^4 and max⁡1,σmax4 \1, _ ^4\, respectively. Our tabular bounds are not intended to sharpen every parameter relative to specialized tabular algorithms; rather, they demonstrate that the uncertainty-dependent behavior follows from a unified Bellman-error-based robust MADC analysis. The benefit of using robust MADC rather than the tabular quantity SAjointSA_joint is most visible under structured function approximation. In the full tabular class, Bellman-error-based complexity measures may scale as SAjointSA_joint, and our framework recovers this dependence. However, this is only the worst-case specialization. Prior work on BE dimension shows that, for structured function classes such as low Bellman-rank, linear, generalized-linear, and kernel models, the Bellman-error complexity is governed by an intrinsic dimension d, rather than by the number of state–joint-action pairs (Jin et al., 2021; Jin et al., 2022; Ghosh et al., 2025a). In particular, low Bellman rank d implies dimMABE≤~(d), _MABE≤ O(d), (Jin et al., 2021, Proposition 11), which can be much smaller than SAjointSA_joint. Our robust MABE-to-MADC result plays the same role in the robust multi-agent setting by Theorem 3, where dMADCrob,max=~(HdMABErob,max).d_MADC^rob, = O\! (Hd_MABE^rob, ). Thus, whenever the robust Bellman-error class has low robust MABE dimension dMABErob,max(ℱ,Π,1/K)≪SAjointd_MABE^rob, (F, ,1/K) SA_joint, the robust MADC is also much smaller than the tabular complexity up to horizon and logarithmic factors. This is the precise sense in which our bound avoids explicit state–joint-action dependence: it replaces the combinatorial tabular quantity by an intrinsic complexity measure that captures how the chosen function class generalizes robust Bellman errors across policies, agents, and adversarial transition dynamics. Linear function approximation: reduction from the bilinear theory. The linear setting serves as a useful benchmark for comparing our general intrinsic-complexity theory with specialized linear robust MARL analyses. In the non-robust setting, MAMEX achieves ~(m2H3dlin2ε−2), O\! (m^2H^3d_lin^2 ^-2 ), while decentralized linear Markov game methods such as (Wang et al., 2023) obtain bounds of order ~(m2H6dlin4(maxi⁡Ai5)ε−2). O\! (m^2H^6d_lin^4( _iA_i^5) ^-2 ). These methods exploit linear structure and, in decentralized settings, reduce the burden of joint exploration, but they remain tied to linear realizability and specialized algorithmic designs. In the robust linear setting, the closest comparison is (Zheng and Lin, 2026), which studies online distributionally robust general-sum Markov games under linear function approximation and TV uncertainty. Their sample complexity is ~(H3dlin2(min⁡H,1/σmin)2ε−2), O\! (H^3d_lin^2( \H,1/ _ \)^2 ^-2 ), with a nearly matching lower bound Ω~(H2dlin2(min⁡H,1/σmin)2ε−2). \! (H^2d_lin^2( \H,1/ _ \)^2 ^-2 ). These rates are sharper in H and in the TV-radius dependence because the analysis is tailored to linear structure and TV uncertainty. Our linear specialization follows instead from the general robust bilinear framework. Theorem 4 gives, under robust-to-nominal feature coverage, dMADCrob,max=~(γ2Hdlin).d_MADC^rob, = O\! (γ^2Hd_lin ). Substituting this into Theorem 1 yields the linear rows in Table 1. In particular, under TV uncertainty, ~(max⁡γ2H4dlin2ε2,H7(min⁡H,1/σmin)4ιlinδε2) O\! ( \ γ^2H^4d_lin^2 ^2, H^7( \H,1/ _ \)^4\, _lin^δ ^2 \ ) is obtained up to logarithmic factors. Compared with (Zheng and Lin, 2026), this bound has worse horizon and uncertainty dependence, but it is derived as a direct specialization of a much broader general-function-approximation theory and applies uniformly to TV, χ2χ^2, and KL uncertainty sets. Thus our result should be viewed as complementary to linear-specific analyses rather than as a minimax-optimal linear result. Overall interpretation. The table highlights a trade-off between specialization and generality. Specialized tabular or linear algorithms achieve sharper rates by exploiting problem-specific structure, often under a fixed divergence such as TV. In contrast, our framework gives a unified analysis for general-sum DRMGs with general function approximation and multiple ϕφ-divergence uncertainty sets. The resulting rates separate three effects: the intrinsic exploration complexity dMADCrob,maxd_MADC^rob, , and the divergence geometry Ψϕ(σ) _φ(σ). This separation clarifies what is structural, what is statistical, and what is specific to the chosen uncertainty model. 7 Numerical Validation We provide a preliminary numerical evaluation of RoMEX-ϕφ to examine three aspects of the proposed framework: (i) whether robust training improves performance under transition-kernel shift relative to its non-robust counterpart; (i) whether the use of general function approximation preserves the robustness achieved by an exact tabular robust method; and (i) whether increasing the expressiveness of the dual function class reduces the approximation error underlying Assumption 2. Our experiments therefore complement the theoretical analysis by evaluating the practical behavior of the two central components of RoMEX-ϕφ: robust fitted value learning and functional approximation of the dual robust Bellman operator. 7.1 Experimental Setup Risky-Coordination DRMG. We consider a scalable general-sum DRMG, termed Risky-Coordination, which extends the two-agent risky/safe coordination environment of Farhat et al. (2026) to multiple agents and a tunable state space. The environment consists of m=4m=4 agents and horizon H=2H=2. At h=0h=0, each agent i observes an independently sampled context ci∈0,…,nctx−1c_i∈\0,…,n_ ctx-1\ and chooses between a safe and a risky action. The contexts are resampled in every episode and affect neither the transition dynamics nor rewards; instead, nctxn_ ctx controls the size of the state space and hence the difficulty of explicit tabular estimation. Let ρ denote the fraction of agents choosing the risky action. The resulting joint action leads to a trap with probability ptrap(ρ)=(trap_base+σ)ρ,p_ trap(ρ)=( trap\_base+σ)ρ, where trap_base=0.6 trap\_base=0.6 and σ controls the transition-kernel shift. Conditional on avoiding the trap, the reward level is sampled from Binomial⁡(nreward−1,ρ)Binomial(n_ reward-1,ρ). A fully safe joint action has zero trap probability for every σ, providing a shift-invariant alternative to the potentially higher-return risky action. The outcome is realized through the transition at h=0h=0 and the corresponding reward is received at h=1h=1, after which the episode terminates. We consider a TV-divergence ambiguity set around the nominal transition kernel, so that σ directly controls the severity of the transition perturbation. Baselines. We compare RoMEX-ϕφ with two complementary baselines. • MAMEX (Xiong et al., 2024) provides the non-robust function-approximation comparison. We use the same fitted-learning architecture as RoMEX-ϕφ, but set the training robustness radius to σtrain=0 _ train=0. Since the robust Bellman operator reduces to the nominal Bellman operator when σ=0σ=0, this comparison isolates the effect of robust training while keeping the function-approximation machinery otherwise fixed. • MORNAVI (Farhat et al., 2026) provides the robust tabular comparison. It estimates the transition model explicitly and performs tabular robust planning using the closed-form TV worst-case expectation and an exploration bonus. Both MORNAVI and RoMEX-ϕφ are trained with σtrain=0.3 _ train=0.3. This comparison examines whether replacing explicit tabular robust planning with the dual function-approximation machinery of RoMEX-ϕφ incurs a substantial loss in robustness. Implementation and evaluation protocol. We instantiate RoMEX-ϕφ under TV uncertainty (RoMEX-TV). Each agent uses an independent double-Q network together with a dual network that estimates the local upper and lower value bounds required by the TV robust backup. The resulting Bellman target takes the form y⁡(s,a)=r+γ⁡(1−done)[vnext−σ⁡(gmax−gmin)],y(s,a)=r+γ(1-done) [v_ next-σ (g_ -g_ ) ], which agrees with the closed-form TV worst-case expectation under the calibration used in our experiments. For computational tractability, we use independent per-agent learners for all methods rather than explicitly constructing the equilibrium oracle over enumerated joint policies. Each method is trained once at a fixed robustness radius: σtrain=0.3 _ train=0.3 for RoMEX-ϕφ and MORNAVI, and σtrain=0 _ train=0 for MAMEX. After training, we freeze the learned policies and evaluate them under transition shifts σ∈0.0,0.1,0.2,0.3,0.4,0.5,σ∈\0.0,0.1,0.2,0.3,0.4,0.5\, without further adaptation. Results are averaged over three random seeds and 200200 evaluation episodes per seed; shaded regions indicate one standard deviation across seeds. Thus, the evaluation measures robustness to deployment-time transition shift rather than continued online adaptation. 7.2 Results (a) Robust vs. non-robust training. (b) Function approximation vs. tabular. (c) Dual approximation error. Figure 1: Performance of RoMEX-ϕφ. (a) Expected return under increasing transition shift for RoMEX-ϕφ (σtrain=0.3 _ train=0.3) and its non-robust counterpart MAMEX (σtrain=0 _ train=0). (b) Comparison with the robust tabular baseline MORNAVI, with both methods trained at σtrain=0.3 _ train=0.3. (c) Empirical dual approximation error as the capacity of the dual function class increases. Shaded regions denote one standard deviation across three random seeds. Does robust training improve resilience to transition shift? Figure 1(a) compares RoMEX-ϕφ with the non-robust MAMEX baseline. At the nominal kernel (σ=0σ=0), MAMEX achieves a higher return (0.3920.392 vs. 0.3380.338), illustrating the expected trade-off between nominal performance and robustness. As the transition shift increases, however, MAMEX deteriorates substantially faster. The curves cross around σ≈0.1σ≈ 0.1, after which RoMEX-ϕφ consistently achieves higher return. At σ=0.4σ=0.4, for example, MAMEX decreases from 0.3920.392 to 0.0630.063, whereas RoMEX-ϕφ decreases from 0.3380.338 to 0.2770.277. These results indicate that optimizing the robust objective sacrifices some nominal performance but yields substantially greater stability under transition mismatch. Does function approximation sacrifice robust performance? Figure 1(b) compares RoMEX-ϕφ against MORNAVI, with both methods trained using the same robustness radius σtrain=0.3 _ train=0.3. Despite replacing explicit tabular transition estimation and robust planning with learned value and dual functions, RoMEX-ϕφ remains competitive with MORNAVI around the training radius and achieves higher return for the larger shifts considered in our evaluation. Across the shift sweep, MORNAVI’s return decreases from approximately 0.3670.367 to 0.1560.156, whereas RoMEX-ϕφ exhibits substantially milder degradation. Thus, in this environment, the use of function approximation does not introduce an observable loss in robustness relative to the tabular robust baseline. This provides empirical support for the motivation behind our framework: robust value estimation can be performed through learned functions without explicitly enumerating the state space. How expressive must the dual function class be? Finally, Figure 1(c) examines the approximation component underlying Assumption 2. We increase the hidden-layer width of the dual network from 8×88× 8 to 256×256256× 256 and measure its empirical approximation error. The error decreases from 0.224±0.0320.224± 0.032 to 0.099±0.0020.099± 0.002, while the variability across seeds also decreases. This trend is consistent with Assumption 2: a richer dual function class reduces the empirical gap associated with approximating the functional dual solution. We emphasize that this experiment does not establish Assumption 2 in general; rather, it provides evidence that the dual approximation error can be made small with a moderately expressive function class in the environment considered here. 7.3 Ablation: Sensitivity to the Robustness Radius The preceding experiments use σtrain=0.3 _ train=0.3 for RoMEX-ϕφ. We next examine whether the observed robustness gain is sensitive to this particular choice and characterize how the training robustness radius affects nominal and shifted performance. We train RoMEX-ϕφ with σtrain∈0.1,0.2,0.3,0.4,0.5 _ train∈\0.1,0.2,0.3,0.4,0.5\, using three random seeds and nctx=10n_ ctx=10, while keeping the remaining experimental protocol unchanged from Section 7.1. We evaluate each trained policy without further adaptation in the nominal environment (σ=0σ=0) and under a representative large transition shift (σ=0.4σ=0.4). For reference, σtrain=0 _ train=0 corresponds to the non-robust MAMEX baseline. Table 2: Sensitivity of RoMEX-ϕφ to the training robustness radius σtrain _ train on the Risky-Coordination DRMG (nctx=10n_ ctx=10, three seeds, and 50 evaluation episodes per seed). The case σtrain=0 _ train=0 corresponds to MAMEX. σtrain _ train Return at σ=0σ=0 Return at σ=0.4σ=0.4 0.0 (MAMEX) 0.366±0.4440.366± 0.444 0.037±0.1050.037± 0.105 0.1 0.349±0.2360.349± 0.236 0.214±0.1600.214± 0.160 0.2 0.338±0.1390.338± 0.139 0.254±0.1420.254± 0.142 0.3 0.336±0.1120.336± 0.112 0.262±0.1370.262± 0.137 0.4 0.334±0.1190.334± 0.119 0.257±0.1400.257± 0.140 0.5 0.336±0.1220.336± 0.122 0.259±0.1380.259± 0.138 Table 2 reveals a clear trade-off between nominal performance and robustness to transition shift. At nominal deployment (σ=0σ=0), introducing robustness results in a modest decrease in average return, from 0.3660.366 at σtrain=0 _ train=0 to approximately 0.330.33–0.350.35 for positive robustness radii. In contrast, under the larger shift σ=0.4σ=0.4, even a small robustness radius substantially improves performance: the average return increases from 0.0370.037 at σtrain=0 _ train=0 to 0.2140.214 at σtrain=0.1 _ train=0.1 and to 0.2540.254 at σtrain=0.2 _ train=0.2. Beyond σtrain=0.2 _ train=0.2, performance remains relatively stable, with average returns between 0.2540.254 and 0.2620.262; given the observed variability, these differences should not be interpreted as a meaningful ordering among the larger radii. Overall, the ablation shows that the robustness gain observed in Figure 1(a) is not specific to the choice σtrain=0.3 _ train=0.3: training with a positive robustness radius consistently improves performance under substantial transition shift, while increasing the radius beyond a moderate level yields diminishing empirical gains in this environment. 7.4 Discussion and Limitations Taken together, the experiments support three observations. First, robust training provides substantially greater resilience to transition-kernel shift than an otherwise comparable non-robust learner. Second, the function-approximation implementation of RoMEX-ϕφ remains competitive with an explicit tabular robust method, suggesting that the scalability gained through functional approximation need not come at the expense of robust performance. Third, increasing dual-network capacity systematically reduces the empirical dual approximation error, supporting the practical feasibility of the functional dual formulation. These experiments are intended as a controlled validation of the proposed mechanism rather than a comprehensive benchmark study. In particular, we use independent per-agent learners instead of the explicit equilibrium oracle in Algorithm 1, and our implementation focuses on TV uncertainty because its closed-form worst-case expectation enables direct validation of the learned dual approximation. Extending the empirical study to χ2χ^2 and KL uncertainty requires the corresponding dual objectives and robust backups. Finally, our implementation of MORNAVI uses a Hoeffding-style exploration bonus rather than the tighter Bernstein-type bonus appearing in its theoretical analysis. 8 Conclusion We study online distributionally robust MARL in general-sum Markov games under general function approximation. We propose RoMEX-ϕφ, a model-free framework that uses a dual reformulation of the robust Bellman operator to enable tractable worst-case evaluation from nominal data via an equilibrium oracle. We introduce the robust Multi-Agent Decoupling Coefficient (robust MADC) as an intrinsic complexity measure and establish the first sublinear regret guarantees for learning robust equilibria. Our results decompose sample complexity into intrinsic exploration, governed by robust MADC, and robust Bellman estimation from nominal data, while avoiding explicit dependence on the state and joint action spaces. Robust MADC further provides a unified view connecting intrinsic complexity with structured function classes. An important direction for future work is extending the analysis to richer function classes, such as infinite or neural parameterizations, as well as to decentralized or partially observed multi-agent settings. References [1] Y. Abbasi-Yadkori, D. Pál, and C. Szepesvári (2011) Improved Algorithms for Linear Stochastic Bandits. In NIPS, Vol. 11, p. 2312–2320. Cited by: Appendix J. [2] M. A. Abdullah, H. Ren, H. B. Ammar, V. Milenkovic, R. Luo, M. Zhang, and J. Wang (2019) Wasserstein Robust Reinforcement Learning. arXiv preprint arXiv:1907.13196. Cited by: §1. [3] A. Agarwal, D. Hsu, S. Kale, J. Langford, L. Li, and R. Schapire (2014) Taming the Monster: A Fast and Simple Algorithm for Contextual Bandits. In Proc. International Conference on Machine Learning (ICML), p. 1638–1646. Cited by: Lemma T.3. [4] B. Alaa Eldin (2023) Why applying Deep Reinforcement Learning in Healthcare is hard. Note: https://medium.com/@baraa.alaa.eldin/why-applying-deep-reinforcement-learning-in-healthcare-is-hard-ffc6e05ab7caAccessed: 2025-07-28 Cited by: §1. [5] S. V. Albrecht, F. Christianos, and L. Schäfer (2024) Multi-Agent Reinforcement Learning: Foundations and Modern Approaches. MIT Press. Cited by: §1. [6] J. Blanchet, M. Lu, T. Zhang, and H. Zhong (2023) Double Pessimism is Provably Efficient for Distributionally Robust Offline Reinforcement Learning: Generic Algorithm and Robust Partial Coverage. Advances in Neural Information Processing Systems 36, p. 66845–66859. Cited by: 2nd item, §1, §2, §3. [7] L. Canese, G. C. Cardarilli, L. Di Nunzio, R. Fazzolari, D. Giardino, M. Re, and S. Spanò (2021) Multi-Agent Reinforcement Learning: A Review of Challenges and Applications. Applied Sciences 11 (11), p. 4948. Cited by: §1. [8] N. Cesa-Bianchi, A. Conconi, and C. Gentile (2001) On the Generalization Ability of On-Line Learning Algorithms. Advances in neural information processing systems 14. Cited by: Appendix E, §6. [9] F. Chen, S. Mei, and Y. Bai (2022) Unified Algorithms for RL with Decision-Estimation Coefficients: No-Regret, PAC, and Reward-Free Learning . Cited by: §2. [10] Z. Chen, D. Zhou, and Q. Gu (2022) Almost Optimal Algorithms for Two-player Zero-Sum Linear Mixture Markov Games. In Proceedings of The 33rd International Conference on Algorithmic Learning Theory, S. Dasgupta and N. Haghtalab (Eds.), Proceedings of Machine Learning Research, Vol. 167, p. 227–261. External Links: Link Cited by: §2. [11] P. Cisneros-Velarde and S. Koyejo (2023) Finite-sample Guarantees for Nash Q-learning with Linear Function Approximation . In Uncertainty in Artificial Intelligence, p. 424–432. Cited by: §2. [12] P. Clavier, E. L. Pennec, and M. Geist (2023) Towards Minimax Optimality of Model-based Robust Reinforcement Learning. arXiv preprint arXiv:2302.05372. Cited by: §3. [13] Q. Cui, K. Zhang, and S. Du (2023) Breaking the Curse of Multiagents in a Large State Space: RL in Markov Games with Independent Linear Function Approximation. In The Thirty Sixth Annual Conference on Learning Theory, p. 2651–2652. Cited by: §2. [14] Y. Dai, Q. Cui, and S. S. Du (2024) Refined Sample Complexity for Markov Games with Independent Linear Function Approximation. In The Thirty Seventh Annual Conference on Learning Theory, p. 1260–1261. Cited by: §2. [15] C. Dann, M. Mohri, T. Zhang, and J. Zimmert (2021) A Provably Efficient Model-Free Posterior Sampling Method for Episodic Reinforcement Learning. Advances in neural information processing systems 34, p. 12040–12051. Cited by: §6. [16] C. Daskalakis, P. W. Goldberg, and C. H. Papadimitriou (2009) The Complexity of Computing a Nash Equilibrium. Commun. ACM 52 (2), p. 89–97. External Links: ISSN 0001-0782, Document Cited by: §3. [17] C. Daskalakis (2013) On the Complexity of Approximating a Nash Equilibrium. ACM Transactions on Algorithms (TALG) 9 (3), p. 1–35. Cited by: §5. [18] A. Demontis, M. Pintor, L. Demetrio, K. Grosse, H. Lin, C. Fang, B. Biggio, and F. Roli (2022) A survey on reinforcement learning security with application to autonomous driving. External Links: 2212.06123, Link Cited by: §1. [19] S. Du, S. Kakade, J. Lee, S. Lovett, G. Mahajan, W. Sun, and R. Wang (2021) Bilinear Classes: A Structural Framework for Provable Generalization in RL. In Proceedings of the 38th International Conference on Machine Learning, M. Meila and T. Zhang (Eds.), Proceedings of Machine Learning Research, Vol. 139, p. 2826–2836. Cited by: §2. [20] S. S. Du, S. M. Kakade, J. D. Lee, S. Lovett, G. Mahajan, W. Sun, and R. Wang (2021) Bilinear classes: a structural framework for provable generalization in rl. arXiv preprint arXiv:2103.10897. Cited by: §2, §2, §6. [21] B. Eysenbach, S. Asawa, S. Chaudhari, S. Levine, and R. Salakhutdinov (2020) Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifiers. arXiv preprint arXiv:2006.13916. Cited by: §1. [22] Z. U. Farhat, D. Ghosh, G. K. Atia, and Y. Wang (2026) Sample-efficient distributionally robust multi-agent reinforcement learning via online interaction. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: §1, §2, §3, §3, §3, §4, Table 1, §6, 2nd item, §7.1. [23] D. Foster, D. J. Foster, N. Golowich, and A. Rakhlin (2023) On the complexity of multi-agent decision making: from learning in games to partial monitoring. In The Thirty Sixth Annual Conference on Learning Theory, p. 2678–2792. Cited by: §2. [24] D. J. Foster, S. M. Kakade, J. Qian, and A. Rakhlin (2021) The statistical complexity of interactive decision making. arXiv preprint arXiv:2112.13487. Cited by: §2. [25] D. Fudenberg and J. Tirole (1991) Game Theory. MIT press. Cited by: §3. [26] D. Ghosh, G. K. Atia, and Y. Wang (2025) Scaling Online Distributionally Robust Reinforcement Learning: Sample-Efficient Guarantees with General Function Approximation. arXiv preprint arXiv:2512.18957. Cited by: Appendix F, Appendix G, §1, §2, §3, §4, §4, §4, §4, §6.1, §6.2, §6, Definition 8, Definition 9, footnote 2. [27] D. Ghosh, G. K. Atia, and Y. Wang (2025) ORVIT: Near-Optimal Online Distributionally Robust Reinforcement Learning. External Links: 2508.03768, Link Cited by: Appendix B, Appendix B, Appendix B, Appendix B. [28] J. He, H. Zhao, D. Zhou, and Q. Gu (2023) Nearly minimax optimal reinforcement learning for linear markov decision processes. In International Conference on Machine Learning, p. 12790–12822. Cited by: §2. [29] Y. He, Z. Liu, W. Wang, and P. Xu (2025) Sample Complexity of Distributionally Robust Off-Dynamics Reinforcement Learning with Online Interaction. In Forty-second International Conference on Machine Learning, Cited by: 2nd item, §D.1, Appendix E. [30] J. A. Holla (2021) On the off-dynamics approach to reinforcement learning. McGill University (Canada). Cited by: §1. [31] M. Hua, D. Chen, X. Qi, K. Jiang, Z. E. Liu, Q. Zhou, and H. Xu (2024) Multi-agent reinforcement learning for connected and automated vehicles control: recent advancements and future prospects. External Links: 2312.11084, Link Cited by: §1. [32] B. Huang, J. D. Lee, Z. Wang, and Z. Yang (2021) Towards general function approximation in zero-sum markov games. arXiv preprint arXiv:2107.14702. Cited by: §2, §2. [33] G. N. Iyengar (2005) Robust Dynamic Programming. Mathematics of Operations Research 30 (2), p. 257–280. Cited by: §3, §3. [34] N. Jiang, A. Krishnamurthy, A. Agarwal, J. Langford, and R. E. Schapire (2017) Contextual decision processes with low Bellman rank are PAC-learnable. In Proc. International Conference on Machine Learning (ICML), p. 1704–1713. Cited by: §2, §4. [35] Y. Jiao and G. Li (2024) Minimax-optimal multi-agent robust reinforcement learning. arXiv preprint arXiv:2412.19873. Cited by: §1, §1, §2. [36] C. Jin, Q. Liu, and S. Miryoosefi (2021) Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithms. Advances in neural information processing systems 34, p. 13406–13418. Cited by: Appendix F, §1, §2, §2, §2, §4, §6.2, §6, §6, Definition 10. [37] C. Jin, Q. Liu, and T. Yu (2022) The power of exploiter: provable multi-agent rl in large state spaces. In Proc. International Conference on Machine Learning (ICML), p. 10251–10279. Cited by: §1, §2, §2, §2, §6.2, §6.2, Table 1, §6. [38] E. Kardeş, F. Ordóñez, and R. W. Hall (2011) Discounted robust stochastic games and an application to queueing control. Operations research 59 (2), p. 365–382. Cited by: §1, §2. [39] N. Li, Z. Zheng, W. Ni, H. Shan, W. Zhang, and X. Li (2025) Sample efficient robust offline self-play for model-based reinforcement learning. Note: Manuscript, OpenReview preprint External Links: Link Cited by: §1. [40] G. Liu, S. Iloglu, M. Caldara, J. W. Durham, and M. M. Zavlanos (2025) Distributionally robust multi-agent reinforcement learning for dynamic chute mapping. In Proc. International Conference on Machine Learning (ICML), Cited by: §1. [41] Q. Liu, T. Yu, Y. Bai, and C. Jin (2021) A Sharp Analysis of Model-based Reinforcement Learning with Self-Play. In Proceedings of the 38th International Conference on Machine Learning (ICML), M. Meila and T. Zhang (Eds.), Proceedings of Machine Learning Research (PMLR), Vol. 139, p. 7001–7010. Cited by: §3. [42] Z. Liu, W. Wang, and P. Xu (2024) Upper and lower bounds for distributionally robust off-dynamics reinforcement learning. arXiv preprint arXiv:2409.20521. Cited by: Appendix E, §2, §4. [43] Z. Liu and P. Xu (2024) Distributionally robust off-dynamics reinforcement learning: provable efficiency with linear function approximation. In International Conference on Artificial Intelligence and Statistics, p. 2719–2727. Cited by: Appendix E, §2. [44] Z. Liu and P. Xu (2024) Minimax Optimal and Computationally Efficient Algorithms for Distributionally Robust Offline Reinforcement Learning. Advances in Neural Information Processing Systems 37, p. 86602–86654. Cited by: §4. [45] R. Lowe, Y. Wu, A. Tamar, J. Harb, P. Abbeel, and I. Mordatch (2017) Multi-agent actor-critic for mixed cooperative-competitive environments. In Proc. Advances in Neural Information Processing Systems (NIPS), p. 6379–6390. Cited by: §1. [46] M. Lu, H. Zhong, T. Zhang, and J. Blanchet (2024) Distributionally robust reinforcement learning with interactive data collection: fundamental hardness and near-optimal algorithm. The Thirty-eighth Annual Conference on Neural Information Processing Systemss. Cited by: Appendix E, §3. [47] M. Lu, Z. Shahn, D. Sow, F. Doshi-Velez, and L. H. Lehman (2021) Is deep reinforcement learning ready for practical applications in healthcare? A sensitivity analysis of duel-DDQN for hemodynamic management in sepsis patients. In AMIA annual symposium proceedings, Vol. 2020, p. 773. Cited by: §1. [48] S. Ma, Z. Chen, S. Zou, and Y. Zhou (2023) Decentralized robust v-learning for solving markov games with model uncertainty. Journal of Machine Learning Research 24 (371), p. 1–40. Cited by: §1, §2, §4, Table 1. [49] X. Ma, Z. Liang, J. Blanchet, M. Liu, L. Xia, J. Zhang, Q. Zhao, and Z. Zhou (2022) Distributionally Robust Offline Reinforcement Learning with Linear Function Approximation. arXiv preprint arXiv:2209.06620. Cited by: §2, §4. [50] L. Matignon, G. J. Laurent, and N. Le Fort-Piat (2012) Independent reinforcement learners in cooperative markov games: a survey regarding coordination problems. The Knowledge Engineering Review 27 (1), p. 1–31. Cited by: §1. [51] C. Ni, Y. Song, X. Zhang, C. Jin, and M. Wang (2022) Representation learning for general-sum low-rank markov games. arXiv preprint arXiv:2210.16976. Cited by: §2. [52] S. Padakandla, P. KJ, and S. Bhatnagar (2020) Reinforcement learning algorithm for non-stationary environments. Applied Intelligence 50 (11), p. 3590–3606. Cited by: §1. [53] K. Panaganti, A. Wierman, and E. Mazumdar (2024) Model-Free Robust ϕφ-Divergence Reinforcement Learning Using Both Offline and Online Data. arXiv preprint arXiv:2405.05468. Cited by: Appendix B, §2, §4, §4. [54] K. Panaganti, Z. Xu, D. Kalathil, and M. Ghavamzadeh (2022) Robust Reinforcement Learning using Offline Data. Advances in neural information processing systems 35, p. 32211–32224. Cited by: Appendix J, Appendix B, Appendix B, Appendix B, §2, §3, §4, §4, §4, §4, footnote 2, footnote 3. [55] G. Papoudakis, F. Christianos, A. Rahman, and S. V. Albrecht (2019) Dealing with non-stationarity in multi-agent deep reinforcement learning. arXiv preprint arXiv:1906.04737. Cited by: §1. [56] X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel (2018) Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference on robotics and automation (ICRA), p. 3803–3810. Cited by: §1. [57] A. Rajeswaran, S. Ghotra, B. Ravindran, and S. Levine (2016) Epopt: learning robust neural network policies using model ensembles. arXiv preprint arXiv:1610.01283. Cited by: §1. [58] Z. Roch and Y. Wang (2025) Distributionally robust markov games with average reward. arXiv preprint arXiv:2508.03136. Cited by: §2. [59] R. T. Rockafellar and R. J. Wets (1998) Variational analysis. Springer. Cited by: Appendix B, Lemma T.1. [60] D. Russo and B. Van Roy (2013) Eluder dimension and the sample complexity of optimistic exploration. Advances in Neural Information Processing Systems 26. Cited by: §2, §4. [61] S. Shalev-Shwartz and S. Ben-David (2014) Understanding machine learning: from theory to algorithms. Cambridge university press. Cited by: Appendix J. [62] S. Shalev-Shwartz et al. (2016) Safe, multi-agent, reinforcement learning for autonomous driving. arXiv preprint arXiv:1610.03295. Cited by: §1. [63] L. Shi, J. Gai, E. Mazumdar, Y. Chi, and A. Wierman (2024) Breaking the curse of multiagency in robust multi-agent reinforcement learning. arXiv preprint arXiv:2409.20067. Cited by: §1, §3. [64] L. Shi, G. Li, Y. Wei, Y. Chen, M. Geist, and Y. Chi (2023) The Curious Price of Distributional Robustness in Reinforcement Learning with a Generative Model. Advances in Neural Information Processing Systems 36, p. 79903–79917. Cited by: §3. [65] L. Shi, E. Mazumdar, Y. Chi, and A. Wierman (2024) Sample-Efficient Robust Multi-Agent Reinforcement Learning in the Face of Environmental Uncertainty. arXiv preprint arXiv:2404.18909. Cited by: §1, §1, §2, §3, Table 1, §6. [66] D. Silver, A. Huang, C. Maddison, A. Guez, L. Sifre, G. Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever, T. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis (2016) Mastering the game of go with deep neural networks and tree search. Nature 529, p. 484–489. External Links: Document Cited by: §1. [67] W. Sun, N. Jiang, A. Krishnamurthy, A. Agarwal, and J. Langford (2019) Model-based rl in contextual decision processes: pac bounds and exponential improvements over model-free approaches. In Conference on learning theory, p. 2898–2933. Cited by: §2, §2, §4. [68] E. Vinitsky, Y. Du, K. Parvate, K. Jang, P. Abbeel, and A. Bayen (2020) Robust reinforcement learning using adversarial populations. arXiv preprint arXiv:2008.01825. Cited by: §1. [69] O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. Choi, R. Powell, T. Ewalds, P. Georgiev, J. Oh, D. Horgan, M. Kroiss, I. Danihelka, A. Huang, L. Sifre, T. Cai, J. P. Agapiou, M. Jaderberg, A. S. Vezhnevets, R. Leblond, T. Pohlen, V. Dalibard, D. Budden, Y. Sulsky, J. Molloy, T. L. Paine, C. Gulcehre, Z. Wang, T. Pfaff, Y. Wu, R. Ring, D. Yogatama, D. Wünsch, K. McKinney, O. Smith, T. Schaul, T. P. Lillicrap, K. Kavukcuoglu, D. Hassabis, C. Apps, and D. Silver (2019) Grandmaster level in starcraft i using multi-agent reinforcement learning. Nature 575, p. 350 – 354. External Links: Link Cited by: §1. [70] H. Wang, L. Shi, and Y. Chi (2024) Sample Complexity of Offline Distributionally Robust Linear Markov Decision Processes. arXiv preprint arXiv:2403.12946. Cited by: §4. [71] R. Wang, R. Salakhutdinov, and L. F. Yang (2020) Reinforcement Learning with General Value Function Approximation: Provably Efficient Approach via Bounded Eluder Dimension. arXiv:2005.10804 [cs, math, stat] (en). Note: arXiv: 2005.10804 Cited by: §4. [72] Y. Wang, Q. Liu, Y. Bai, and C. Jin (2023) Breaking the curse of multiagency: provably efficient decentralized multi-agent rl with function approximation. In The Thirty Sixth Annual Conference on Learning Theory, p. 2793–2848. Cited by: §2, §6.2, Table 1. [73] Y. Wang, Z. Sun, and S. Zou (2024) A Unified Principle of Pessimism for Offline Reinforcement Learning under Model Mismatch. Advances in Neural Information Processing Systems 37, p. 9281–9328. Cited by: §3. [74] Y. Wang and S. Zou (2021) Online Robust Reinforcement Learning with Model Uncertainty. Advances in Neural Information Processing Systems 34, p. 7193–7206. Cited by: §2. [75] W. Wiesemann, D. Kuhn, and B. Rustem (2013) Robust Markov Decision Processes. Mathematics of Operations Research 38 (1), p. 153–183. Cited by: §3, footnote 1. [76] A. Wong, T. Bäck, A. V. Kononova, and A. Plaat (2023) Deep multiagent reinforcement learning: challenges and directions. Artificial Intelligence Review 56 (6), p. 5023–5056. Cited by: §1. [77] Q. Xie, Y. Chen, Z. Wang, and Z. Yang (2020) Learning zero-sum simultaneous-move markov games using function approximation and correlated equilibrium. In Proc. Annual Conference on Learning Theory (CoLT), p. 3674–3682. Cited by: §2. [78] N. Xiong, Z. Liu, Z. Wang, and Z. Yang (2024) Sample-Efficient Multi-Agent RL: An Optimization Perspective. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, Cited by: Appendix F, Appendix G, Appendix G, Appendix G, Appendix G, Appendix G, Appendix H, Appendix H, Appendix H, §2, §5, §6.1, §6.2, Table 1, Table 1, Table 1, §6, §6, §6, §6, 1st item, Lemma T.4, Lemma T.5. [79] W. Yang, L. Zhang, and Z. Zhang (2022) Toward Theoretical Understandings of Robust Markov Decision Processes: Sample Complexity and Asymptotics. The Annals of Statistics 50 (6), p. 3223–3248. Cited by: §D.1, §3. [80] W. Yang, L. Zhang, and Z. Zhang (2022) Toward Theoretical Understandings of Robust Markov Decision Processes: Sample Complexity and Asymptotics. The Annals of Statistics 50, p. . External Links: Document Cited by: §4. [81] W. Zhan, J. D. Lee, and Z. Yang (2022) Decentralized optimistic hyperpolicy mirror descent: provably no-regret learning in markov games. arXiv preprint arXiv:2206.01588. Cited by: §2. [82] C. Zhang, Z. U. Farhat, G. K. Atia, and Y. Wang (2025) Model-free offline reinforcement learning with enhanced robustness. In Proc. International Conference on Learning Representations (ICLR), Cited by: §3. [83] K. Zhang, T. Sun, Y. Tao, S. Genc, S. Mallya, and T. Basar (2020) Robust multi-agent reinforcement learning with model uncertainty. In Proc. Advances in Neural Information Processing Systems (NeurIPS), Vol. 33. Cited by: §1, §2, §3. [84] W. Zhao, J. P. Queralta, and T. Westerlund (2020) Sim-to-real transfer in deep reinforcement learning for robotics: a survey. In 2020 IEEE symposium series on computational intelligence (SSCI), p. 737–744. Cited by: §1. [85] Z. Zheng and Y. Lin (2026) Distributionally robust online markov game with linear function approximation. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40, p. 28892–28900. Cited by: §1, §2, §4, §5, §6.2, §6.2, Table 1, Table 1. [86] H. Zhong, W. Xiong, S. Zheng, L. Wang, Z. Wang, Z. Yang, and T. Zhang (2022) A posterior sampling framework for interactive decision making. arXiv preprint arXiv:2211.01962 2 (3). Cited by: §2. [87] Z. Zhou, Z. Zhou, Q. Bai, L. Qiu, J. Blanchet, and P. Glynn (2021) Finite-Sample Regret Bound for Distributionally Robust Offline Tabular Reinforcement Learning. In International Conference on Artificial Intelligence and Statistics, p. 3331–3339. Cited by: §3. Appendix A Additional Preliminaries of DRMGs Appendix B Dual Formulation and Functional Optimization Functional formulation. To avoid pointwise optimization in the dual Bellman operator, we consider a functional optimization problem over dual functions. Let μhπ _h^π denote the state–joint action visitation distribution induced by policy π under the nominal kernel P⋆P . We define the dual function space ℒi1(μhπ;ℝ2):=gi=(giλ,giν):×→ℝ2|giλ,giν∈ℒ1(μhπ),L^1_i( _h^π;R^2):= \g_i=(g_i^λ,g_i^ν):S×A ^2\; |\;g_i^λ,g_i^ν ^1( _h^π) \, and the dual loss (restating Equation 5) LossDuali,h,ϕ(g,fi)=(s,)∼μhπ[s′∼Ph⋆Lossi,h,ϕ(fi,s,,s′,g⁡(s,))].LossDual_i,h,φ(g;f_i)=E_(s, a) _h^π [E_s P_h Loss_i,h,φ(f_i;s, a,s ;g(s, a)) ]. Dual equivalence. The following lemma extend the results of single-agent in [54] (which considers the TV-divergence set in an offline setting), [27] (which consider ϕφ-divergence set in an online setting), and [53] (which considers regularized MDPs in a hybrid setting), and show that for general-sum ϕφ-divergence DRMG, minimization of the dual loss is equivalent to the support function under some distribution for the multi-agent setup. Lemma 2 (Multi-agent Dual loss minimization). Let LossDuali,ϕLossDual_i,φ be defined as the dual loss function as in eq. 5 for agent i∈ℳi . Then, for any function fi:×→[0,H]f_i:S×A→[0,H] for any i∈ℳi , we have infg∈ℒi1(μ,ℝ2)LossDuali,hϕ(g,fi)=(s,)∼μ[i,h,ϕσi(s,)[Vi,h+1π,f]],∀i∈ℳ. _g ^1_i(μ;R^2)LossDual_i,hφ(g;f_i)=E_(s, a) μ [E_P_i,h,φ _i(s, a) [V^π,f_i,h+1 ] ],∀ i . Proof. We prove the result for an arbitrary but fixed agent i∈ℳi and stage h∈[H]h∈[H]. However, once the agent i and policy π are fixed, the continuation value Vi,h+1π,f(s′)V_i,h+1^π,f(s ) is simply a scalar measurable function of s′s , and the rest of the argument is identical to the single-agent functional dual-minimization proof [27, 54]. The detailed proof-lines are given below. Step 1: Define the pointwise continuation value and the pointwise dual objective. For each next state s′∈s , recall the definition of the continuation value as Vi,h+1π,f(s′):=′∼πh+1(⋅∣s′)[fi,h+1(s′,′)].V_i,h+1^π,f(s ):=E_ a _h+1(· s ) [f_i,h+1(s , a ) ]. Since πh+1(⋅∣s′) _h+1(· s ) is a probability distribution on the joint action space A and fi,h+1f_i,h+1 is measurable on ×S×A, the mapping s′↦Vi,h+1π,f(s′)s V_i,h+1^π,f(s ) is measurable. Now, for every joint state-action pair (s,)∈×(s, a) ×A, we define the pointwise scalar optimization objective as Ψi,hf(s,;λ,ν):=s′∼Ph⋆(⋅∣s,)[λσi−ν+λϕ⋆(−Vi,h+1π,f(s′)−νλ)],λ>0,ν∈ℝ. _i,h^f(s, a;λ,ν):=E_s P_h (· s, a)\! [λ _i-ν+λφ \! (- V_i,h+1^π,f(s )-νλ ) ], λ>0,\ ν . (13) Equivalently, if we define the pointwise integrand as Lossi,h,ϕ(fi,s,,s′,λ,ν)Loss_i,h,φ(f_i;s, a,s ;λ,ν) as before, then Ψi,h(fi;s,;λ,ν)=s′∼Ph⋆(⋅∣s,)[Lossi,h,ϕ(fi;s,,s′;λ,ν)]. _i,h(f_i;s, a;λ,ν)=E_s P_h (· s, a) [Loss_i,h,φ(f_i;s, a,s ;λ,ν) ]. Step 2: Pointwise dual representation of the robust support function. By the dual reformulation of the ϕφ-divergence robust Bellman operator used in the draft, for every fixed (s,)(s, a) we have i,h,ϕσ(s,)[Vi,h+1π,f]=infλ>0,ν∈ℝΨi,h(fi,s,,λ,ν).E_P^σ_i,h,φ(s, a) [V_i,h+1^π,f ]= _λ>0,ν _i,h(f_i;s, a;λ,ν). That is, infλ>0,ν∈ℝs′∼Ph⋆(⋅∣s,)[λσi−ν+λϕ⋆(−Vi,h+1π,f(s′)−νλ)]=i,h,ϕσ(s,)[Vi,h+1π,f]. _λ>0,ν E_s P_h (· s, a)\! [λ _i-ν+λφ \! (- V_i,h+1^π,f(s )-νλ ) ]=E_P^σ_i,h,φ(s, a) [V_i,h+1^π,f ]. (14) This is exactly the same pointwise duality principle as in the single-agent robust MDP proof, except that here the action variable is the joint action a, and the ambiguity set is the agent-specific set i,h,ϕσ(s,)P^σ_i,h,φ(s, a). The multi-agent structure does not alter the duality argument at this step; it only changes the domain from (s,ai)(s,a_i) or (s,a)(s,a) in the single-agent [27, 54] to the present joint-action pair (s,)∈×(s, a) ×A. Step 3: Functional dual loss in terms of the pointwise objective. Recall that the dual function is gi=(giλ,giν)∈ℒi1(μ,ℝ2)g_i=(g_i^λ,g_i^ν) ^1_i(μ;R^2), so at each (s,)(s, a) the function gig_i selects a pair gi(s,)=(giλ(s,),giν(s,)).g_i(s, a)= (g_i^λ(s, a),g_i^ν(s, a) ). By the definition of the dual loss as given in (5), we have Lossi,h,ϕDual(gi;fi)=(s,)∼μ[s′∼Ph⋆(⋅∣s,)[Lossi,h,ϕ(fi;s,,s′;gi)]],Loss^Dual_i,h,φ(g_i;f_i)=E_(s, a) μ [E_s P_h (· s, a) [Loss_i,h,φ(f_i;s, a,s ;g_i) ] ], where Lossi,h,ϕ(fi,s,,s′,gi)=giλ(s,)σi−giν(s,)+giλ(s,)ϕ⋆(−Vi,h+1π,f(s′)−giν(s,)giλ(s,)).Loss_i,h,φ(f_i;s, a,s ;g_i)=g_i^λ(s, a) _i-g_i^ν(s, a)+g_i^λ(s, a)φ \! (- V_i,h+1^π,f(s )-g_i^ν(s, a)g_i^λ(s, a) ). Therefore, Lossi,h,ϕDual(gi,fi) ^Dual_i,h,φ(g_i;f_i) =(s,)∼μ[Ψi,h(fi,s,a,giλ(s,a),giν(s,a))]. =E_(s, a) μ [ _i,h (f_i;s,a;g_i^λ(s,a),g_i^ν(s,a) ) ]. (15) Hence, infgi∈ℒi1(μ,ℝ2)Lossi,h,ϕDual(gi,fi)=infgi∈ℒi1(μ,ℝ2)(s,a)∼μ[Ψi,h(fi,s,a,giλ(s,a),giν(s,a))]. _g_i ^1_i(μ;R^2)Loss^Dual_i,h,φ(g_i;f_i)= _g_i ^1_i(μ;R^2)E_(s,a) μ [ _i,h (f_i;s,a;g_i^λ(s,a),g_i^ν(s,a) ) ]. (16) Step 4: Verify that the minimization–integration interchange theorem applies. We now want to interchange the functional minimization over gig_i and the expectation over (s,)∼μ(s, a) μ. For this, define the integrand ℓ⁡((s,),x) ((s, a),x) :=Ψi,h(fi;s,;x),λ>0+∞λ≤0,wherex=(λ,ν)∈ℝ2. := cases _i,h(f_i;s, a;x), &λ>0\\ +∞ &λ≤ 0 cases, x=(λ,ν) ^2. We will apply the integral-interchange of [59][Theorem 14.60] as stated in Lemma T.1, exactly as done in the single-agent robust offline RL proof [27]. To justify this, we check the needed properties. • The decision space is decomposable. The space ℒ1(μ,ℝ2)L^1(μ;R^2) is decomposable. Indeed, if g0,g1∈ℒ1(μ,ℝ2)g_0,g_1 ^1(μ;R^2) and B∈Σ⁡(×)B∈ (S×A) is measurable with finite μ-measure, define g⁡(s,):=B(s,)g1(s,)+Bc(s,)g0(s,).g(s, a):=1_B(s, a)\,g_1(s, a)+1_B^c(s, a)\,g_0(s, a). Then g is measurable and integrable componentwise, and thus g∈ℒ1(μ,ℝ2)g ^1(μ;R^2). Hence ℒ1(μ,ℝ2)L^1(μ;R^2) is decomposable. • Measurability in (s,)(s, a). For every fixed x=(λ,ν)x=(λ,ν) with λ>0λ>0, the map (s,)↦ℓ⁡((s,),x)(s, a) ((s, a),x) is measurable, because (s,)↦Ph⋆(⋅∣s,)(s, a) P_h (· s, a) is a measurable kernel, and the inner expression s′↦λσi−ν+λϕ⋆(−Vi,h+1π,f(s′)−νλ)s λ _i-ν+λφ \! (- V_i,h+1^π,f(s )-νλ ) is measurable in s′s . • Lower semicontinuity in x=(λ,ν)x=(λ,ν). For every fixed (s,)(s, a), the map (λ,ν)↦ℓ⁡((s,),(λ,ν))(λ,ν) ((s, a),(λ,ν)) is lower semicontinuous on the domain (λ,ν):λ>0,ν∈ℝ\(λ,ν):λ>0,ν \. Indeed, the terms λσi−νλ _i-ν are continuous, the convex conjugate ϕ⋆φ is convex and lower semicontinuous, the map (λ,ν,z)↦λϕ⋆(−z−νλ)(λ,ν,z) λφ \! (- z-νλ ) is lower semicontinuous for λ>0λ>0, and expectation with respect to Ph⋆(⋅∣s,)P_h (· s, a) preserves lower semicontinuity under standard measurability and integrability conditions. Thus ℓ is a normal integrand. Therefore, the assumptions of Rockafellar & Wets integral-interchange lemma T.1 hold for the present problem. Step 5: Interchange the infimum and the expectation. Applying Rockafellar & Wets Theorem (Lemma T.1) yields infg∈ℒi1(μ,ℝ2)(s,)∼μ[Ψi,h(fi,s,,giλ(s,),giν(s,))] _g ^1_i(μ;R^2)E_(s, a) μ [ _i,h (f_i;s, a;g_i^λ(s, a),g_i^ν(s, a) ) ] =(s,)∼μ[inf(λ,ν)∈(0,∞)×ℝΨi,h(fi,s,a,λ,ν)]. =E_(s, a) μ [ _(λ,ν)∈(0,∞)×R _i,h(f_i;s,a;λ,ν) ]. (17) Combining (16) and (17), we obtain infgi∈Li1(μ,ℝ2)LossDuali,h,ϕ(gi,fi)=(s,a)∼μ[infλ>0,ν∈ℝΨi,hfi(s,a,λ,ν)]. _g_i∈ L^1_i(μ;R^2)LossDual_i,h,φ(g_i;f_i)=E_(s,a) μ [ _λ>0,ν _i,h^f_i(s,a;λ,ν) ]. (18) Step 6: Substitute the pointwise robust support-function identity. Now substitute the pointwise identity (14) into (18). This gives infg∈ℒi1(μ,ℝ2)Lossi,h,ϕDual(g,fi) _g ^1_i(μ;R^2)Loss^Dual_i,h,φ(g;f_i) =(s,)∼μ[i,h,ϕσ(s,)[Vi,h+1π,f]]. =E_(s, a) μ [E_P^σ_i,h,φ(s, a) [V_i,h+1^π,f ] ]. Hence, the result follows.∎ Appendix C Proof of Lemma 1 Proof. Fix any agent i∈ℳi and stage h∈[H]h∈[H]. Steps 1–4 and the first part of Step 5 fix an arbitrary policy π∈Πpurπ∈ pur; the dependence of ζf _f, ζ^f ζ_f and g^if,π g_i^\,f,π on π is suppressed wherever there is no ambiguity, and is restored in the concentration argument of Step 5, which is established uniformly over π∈Πpurπ∈ pur. For notational simplicity we likewise suppress the dependence on (i,h)(i,h) throughout. Step 1: Define the population dual objective. For any fi,h+1∈ℱi,h+1f_i,h+1 _i,h+1 and any dual function g∈i,hg _i,h, define ζf(g):=(s,)∼μ¯hn[s′∼Ph⋆(⋅∣s,)[Lossi,h,ϕ(fi;s,,s′;g)]]. _f(g):=E_(s, a) μ_h^\,n [E_s P_h (· s, a) [Loss_i,h,φ(f_i;s, a,s ;g) ] ]. Since μ¯hn=1n∑t=1nμhπt μ_h^\,n= 1n _t=1^n _h^π^t is an average of probability measures on ×S×A, linearity of the integral in the measure gives the equivalent form ζf(g)=1n∑t=1n(s,)∼μhπt[s′∼Ph⋆(⋅∣s,)[Lossi,h,ϕ(fi;s,,s′;g)]], _f(g)= 1n _t=1^nE_(s, a) _h^π^t [E_s P_h (· s, a) [Loss_i,h,φ(f_i;s, a,s ;g) ] ], (19) which will be used in Step 5. By the definition of the dual Bellman operator, we have [gπ,ϕ,σifi](s,)=ri,h(s,)−s′∼Ph⋆(⋅∣s,)[Lossi,h,ϕ(fi;s,,s′;g)]. [T^π,φ, _i_gf_i ](s, a)=r_i,h(s, a)-E_s P_h (· s, a) [Loss_i,h,φ(f_i;s, a,s ;g) ]. (20) On the other hand, by the dual reformulation of the robust Bellman operator as given in (4), we have [i,hπ,ϕ,σifi](s,)=ri,h(s,)−infλ>0,ν∈ℝs′∼Ph⋆(⋅∣s,)[Lossi,h,ϕ(fi;s,,s′;λ,ν)]. [T^π,φ, _i_i,hf_i ](s, a)=r_i,h(s, a)- _λ>0,ν E_s P_h (· s, a) [Loss_i,h,φ(f_i;s, a,s ;λ,ν) ]. Therefore, for every (s,)(s, a), [i,hπ,ϕ,σifi](s,)−[gπ,ϕ,σifi](s,) [T^π,φ, _i_i,hf_i ](s, a)- [T^π,φ, _i_gf_i ](s, a) =s′∼Ph⋆(⋅∣s,)[Lossi,h,ϕ(fi;s,,s′;g)] =E_s P_h (· s, a) [Loss_i,h,φ(f_i;s, a,s ;g) ] −infλ>0,ν∈ℝs′∼Ph⋆(⋅∣s,)[Lossi,h,ϕ(fi;s,,s′;λ,ν)]. - _λ>0,ν E_s P_h (· s, a) [Loss_i,h,φ(f_i;s, a,s ;λ,ν) ]. Since the second term is the infimum over all admissible dual parameters, the right-hand side is nonnegative pointwise in (s,)(s, a). Hence the absolute value may be removed under any probability measure on ×S×A, and in particular under μ¯hn μ_h^\,n: ‖i,hπ,ϕ,σifi−gπ,ϕ,σifi‖1,μ¯hn \|T^π,φ, _i_i,hf_i-T^π,φ, _i_gf_i \|_1, μ_h^\,n =(s,)∼μ¯hn[[i,hπ,ϕ,σifi](s,)−[gπ,ϕ,σifi](s,)] =E_(s, a) μ_h^\,n [ [T^π,φ, _i_i,hf_i ](s, a)- [T^π,φ, _i_gf_i ](s, a) ] =ζf(g)−(s,)∼μ¯hn[infλ>0,ν∈ℝs′∼Ph⋆(⋅∣s,)[Lossi,h,ϕ(fi;s,,s′;λ,ν)]]. = _f(g)-E_(s, a) μ_h^\,n [ _λ>0,ν E_s P_h (· s, a) [Loss_i,h,φ(f_i;s, a,s ;λ,ν) ] ]. (21) Step 2: Use Lemma 2 to identify the infimum term. Lemma 2 holds for any probability measure on ×S×A; applying it with μ¯hn μ_h^\,n, the interchange rule for the dual functional objective implies (s,)∼μ¯hn[infλ>0,ν∈ℝs′∼Ph⋆(⋅∣s,)[Lossi,h,ϕ(fi;s,,s′;λ,ν)]]=infg∈ℒi1(μ¯hn,ℝ2)ζf(g).E_(s, a) μ_h^\,n [ _λ>0,ν E_s P_h (· s, a) [Loss_i,h,φ(f_i;s, a,s ;λ,ν) ] ]= _g ^1_i( μ_h^\,n;R^2) _f(g). Substituting into (21) yields ‖i,hπ,ϕ,σifi−gπ,ϕ,σifi‖1,μ¯hn=ζf(g)−infg′∈ℒi1(μ¯hn,ℝ2)ζf(g′). \|T^π,φ, _i_i,hf_i-T^π,φ, _i_gf_i \|_1, μ_h^\,n= _f(g)- _g ^1_i( μ_h^\,n;R^2) _f(g ). (22) Step 3: Specialize to the empirical ERM minimizer. Recall n=|h|n=|D_h|, and write (st,t,st′)(s_t, a_t,s _t) for the stage-h transition tuple contributed to hD_h by episode t, t=1,…,nt=1,…,n. Let g^if,π∈arg⁡ming∈i,hζ^f(g) g_i^\,f,π∈ _g _i,h ζ_f(g) be an empirical minimizer, where ζ^f(g):=1n∑t=1nLossi,h,ϕ(fi,st,t,st′,g). ζ_f(g):= 1n _t=1^nLoss_i,h,φ(f_i;s_t, a_t,s _t;g). Applying (22) with g=g^if,πg= g_i^\,f,π gives ‖i,hπ,ϕ,σifi−g^if,π,ϕ,σifi‖1,μ¯hn \|T^π,φ, _i_i,hf_i-T^π,φ, _i_ g_i^\,f,πf_i \|_1, μ_h^\,n =ζf(g^if,π)−infg∈ℒi1(μ¯hn,ℝ2)ζf(g) = _f( g_i^\,f,π)- _g ^1_i( μ_h^\,n;R^2) _f(g) =(ζf(g^if,π)−infg∈i,hζf(g))⏟(I)+(infg∈i,hζf(g)−infg∈ℒi1(μ¯hn,ℝ2)ζf(g))⏟(I). = ( _f( g_i^\,f,π)- _g _i,h _f(g) )_(I)+ ( _g _i,h _f(g)- _g ^1_i( μ_h^\,n;R^2) _f(g) )_(I). (23) Step 4: Bound the approximation term (I)(I). Since μ¯hn μ_h^\,n is an admissible stage-wise distribution, Assumption 2 applies with μ¯hn μ_h^\,n in place of μhπ _h^π and gives infg∈i,hζf(g)−infg∈ℒi1(μ¯hn,ℝ2)ζf(g)≤εdualapprox. _g _i,h _f(g)- _g ^1_i( μ_h^\,n;R^2) _f(g)≤ _ dual approx. Hence, (I)≤εdualapprox.(I)≤ _ dual approx. (24) Step 5: Bound the ERM excess-risk term (I)(I). We now bound the excess-risk term (I)=ζf(g^if,π)−infg∈i,hζf(g).(I)= _f( g_i^\,f,π)- _g _i,h _f(g). For any fixed fi,h+1∈ℱi,h+1f_i,h+1 _i,h+1, let gf⋆∈arg⁡ming∈i,hζf(g)g_f ∈ _g _i,h _f(g) be a population minimizer over i,hG_i,h. Since g^if,π g_i^\,f,π minimizes the empirical objective, ζ^f(g^if,π)≤ζ^f(gf⋆). ζ_f( g_i^\,f,π)≤ ζ_f(g_f ). Hence, (I)=ζf(g^if,π)−ζf(gf⋆) (I)= _f( g_i^\,f,π)- _f(g_f ) =(ζf(g^if,π)−ζ^f(g^if,π))+(ζ^f(g^if,π)−ζ^f(gf⋆))+(ζ^f(gf⋆)−ζf(gf⋆)) = ( _f( g_i^\,f,π)- ζ_f( g_i^\,f,π) )+ ( ζ_f( g_i^\,f,π)- ζ_f(g_f ) )+ ( ζ_f(g_f )- _f(g_f ) ) ≤2supg∈i,h|ζf(g)−ζ^f(g)|, ≤ 2 _g _i,h | _f(g)- ζ_f(g) |, (25) where the middle term is nonpositive by optimality of g^if,π g_i^\,f,π. Taking suprema over fi,h+1∈ℱi,h+1f_i,h+1 _i,h+1 and π∈Πpurπ∈ pur yields supπ∈Πpursupfi,h+1∈ℱi,h+1(I)≤2supπ∈Πpursupfi,h+1∈ℱi,h+1supg∈i,h|ζf(g)−ζ^f(g)|. _π∈ pur _f_i,h+1 _i,h+1(I)≤ 2 _π∈ pur _f_i,h+1 _i,h+1 _g _i,h| _f(g)- ζ_f(g)|. It therefore suffices to control |ζf(g)−ζ^f(g)|| _f(g)- ζ_f(g)| uniformly over (fi,h+1,g,π)(f_i,h+1,g,π); note that the uniformity is what permits the subsequent substitution of the data-dependent minimizer g^if,π g_i^\,f,π. By Assumption 3, for every admissible (f,g,π)(f,g,π) and every sample (s,,s′)(s, a,s ) we have |Lossi,h,ϕ(fi,s,,s′,g)|≤Bϕ(σi) |Loss_i,h,φ(f_i;s, a,s ;g) |≤ B_φ( _i). Restoring the dependence on π, define the finite class ℋi:=(s,,s′)↦Lossi,h,ϕ(fi;s,,s′;g,π):fi,h+1∈ℱi,h+1,g∈i,h,π∈Πpur,H_i:= \(s, a,s ) _i,h,φ(f_i;s, a,s ;g,π):\;f_i,h+1 _i,h+1,\ g _i,h,\ π∈ pur \, so that all functions in ℋiH_i are uniformly bounded by Bϕ(σi)B_φ( _i) and |ℋi|≤|ℱi,h+1‖i,h‖Πpur||H_i|≤|F_i,h+1||G_i,h|| pur|. Martingale structure. Fix (fi,h+1,g,π)∈ℱi,h+1×i,h×Πpur(f_i,h+1,g,π) _i,h+1×G_i,h× pur and define, for t=1,…,nt=1,…,n, Zt(fi,g,π):=Lossi,h,ϕ(fi,st,t,st′,g)−⁡[Lossi,h,ϕ(fi,st,t,st′,g)|ℋt−1].Z_t(f_i,g,π):=Loss_i,h,φ(f_i;s_t, a_t,s _t;g)-E [Loss_i,h,φ(f_i;s_t, a_t,s _t;g)\, |\,H_t-1 ]. Because the behavior policy πtπ^t is ℋt−1H_t-1-measurable, the conditional law of (st,t)(s_t, a_t) given ℋt−1H_t-1 is μhπt _h^π^t, and st′∼Ph⋆(⋅∣st,t)s _t P_h (· s_t, a_t); hence [Lossi,h,ϕ(fi;st,t,st′;g)|ℋt−1]=(s,)∼μhπt[s′∼Ph⋆(⋅∣s,)[Lossi,h,ϕ(fi;s,,s′;g)]].E [Loss_i,h,φ(f_i;s_t, a_t,s _t;g)\, |\,H_t-1 ]=E_(s, a) _h^π^t [E_s P_h (· s, a) [Loss_i,h,φ(f_i;s, a,s ;g) ] ]. Averaging over t=1,…,nt=1,…,n and comparing with (19) gives 1n∑t=1n[⋅∣ℋt−1]=ζf(g) 1n _t=1^nE[\,· _t-1]= _f(g), and therefore ζ^f(g)−ζf(g)=1n∑t=1nZt(fi,g,π). ζ_f(g)- _f(g)= 1n _t=1^nZ_t(f_i,g,π). (26) By construction ⁡[Zt∣ℋt−1]=0E[Z_t _t-1]=0, and |Zt|≤2Bϕ(σi)|Z_t|≤ 2B_φ( _i) by Assumption 3; thus Zt(fi,g,π)t=1n\Z_t(f_i,g,π)\_t=1^n is a bounded martingale difference sequence with respect to the filtration ℋtt≥0\H_t\_t≥ 0. Concentration. The conditional variances satisfy ∑t=1n⁡[Zt2∣ℋt−1]≤4nBϕ(σi)2 _t=1^nE[Z_t^2 _t-1]≤ 4nB_φ( _i)^2. By Freedman’s inequality (Lemma T.3), for a fixed triple (fi,h+1,g,π)(f_i,h+1,g,π) and any δ′∈(0,1)δ ∈(0,1), with probability at least 1−δ′1-δ , |∑t=1nZt(fi,g,π)|≤C1Bϕ(σi)nlog⁡(1/δ′)+C2Bϕ(σi)log⁡(1/δ′). | _t=1^nZ_t(f_i,g,π) |≤ C_1B_φ( _i) n (1/δ )+C_2B_φ( _i) (1/δ ). If log⁡(1/δ′)>n (1/δ )>n the claimed bound below is vacuous, since |ζ^f(g)−ζf(g)|≤2Bϕ(σi)| ζ_f(g)- _f(g)|≤ 2B_φ( _i) always; we may therefore assume log⁡(1/δ′)≤n (1/δ )≤ n, in which case the second term is dominated by the first. Combining with (26), |ζf(g)−ζ^f(g)|≤C3Bϕ(σi)log⁡(1/δ′)n. | _f(g)- ζ_f(g) |≤ C_3B_φ( _i) (1/δ )n. Taking a union bound over the finite class ℋiH_i, i.e. over (fi,h+1,g,π)∈ℱi,h+1×i,h×Πpur(f_i,h+1,g,π) _i,h+1×G_i,h× pur with δ′=δ/(|ℱi,h+1‖i,h‖Πpur|)δ =δ/(|F_i,h+1||G_i,h|| pur|), we obtain that with probability at least 1−δ1-δ, supπ∈Πpursupfi,h+1∈ℱi,h+1supg∈i,h|ζf(g)−ζ^f(g)|≤CBϕ(σi)log⁡(|ℱi,h+1‖i,h‖Πpur|/δ)n. _π∈ pur _f_i,h+1 _i,h+1 _g _i,h | _f(g)- ζ_f(g) |≤ C\,B_φ( _i) (|F_i,h+1||G_i,h|| pur|/δ )n. We emphasise that the union bound is taken over the hypothesis and policy classes only, and not over the realizations of the behavior sequence πtt=1n\π^t\_t=1^n; the adaptivity of that sequence is absorbed by the martingale structure above. Substituting into (25) gives supπ∈Πpursupfi,h+1∈ℱi,h+1(I)≤2CBϕ(σi)log⁡(|ℱi,h+1‖i,h‖Πpur|/δ)n. _π∈ pur _f_i,h+1 _i,h+1(I)≤ 2C\,B_φ( _i) (|F_i,h+1||G_i,h|| pur|/δ )n. (27) Step 6: Combine the bounds. Combining (23), (24), and (27), we conclude that with probability at least 1−δ1-δ, supπ∈Πpursupfi,h+1∈ℱi,h+1‖i,hπ,ϕ,σifi,h+1−g^if,π,ϕ,σifi,h+1‖1,μ¯hn _π∈ pur _f_i,h+1 _i,h+1 \|T^π,φ, _i_i,hf_i,h+1-T^π,φ, _i_ g_i^\,f,πf_i,h+1 \|_1, μ_h^\,n ≤C′Bϕ(σi)log⁡(|ℱi,h+1‖i,h‖Πpur|/δ)n ≤ C B_φ( _i) (|F_i,h+1||G_i,h|| pur|/δ )n +εdualapprox, + _ dual approx, for another absolute constant C′>0C >0. This proves the result.∎ Appendix D Proof of Theorem 1 Notation. For each agent i∈ℳi , we consider πkπ^k to be the output of the equilibrium oracle for a target equilibrium notion ∈NE,CCE,CE Eq∈\NE,CCE,CE\ and we define the corresponding unilateral deviation policy: ~i†,k,=πi,NE†,k(π−ik)×π−ik,=NE,πi,CCE†,k(π−ik)×π−ik,=CCE,ψi†,k⋄πk,=CE. π ,k, Eq_i= cases _i,NE ,k( _-i^k)× _-i^k,& Eq=NE,\\[6.0pt] _i,CCE ,k( _-i^k)× _-i^k,& Eq=CCE,\\[6.0pt] _i ,k π^k,& Eq=CE. cases (28) Proof. We prove the result for robust NE regret. The robust CCE and robust CE cases follow by replacing the surrogate Nash-deviation inequality with the corresponding surrogate CCE/CE deviation inequality from Lemma 3. Let ιℱ,δ:=log⁡(|ℱ‖Πpur|KHmδ),Bi:=Bϕ(σi). _F,G^δ:= \! ( |F||G|| ^pur|KHmδ ), B_i:=B_φ( _i). Step 1: Reduce robust regret to per-agent gaps. Let πi†,k(π−ik) _i ,k( _-i^k) be a robust best response of agent i against π−ik _-i^k, and define ~i†,k:=πi†,k(π−ik)×π−ik. π_i ,k:= _i ,k( _-i^k)× _-i^k. For each episode k and agent i, define Gapi,kNE:=Vi,1~i†,k,σi(s1k)−Vi,1πk,σi(s1k).Gap_i,k^NE:=V_i,1 π_i ,k, _i(s_1^k)-V_i,1^π^k, _i(s_1^k). By Definition 5, RegNE(K)≤max⁡∑k=1Ki∈ℳ⁡Gapi,kNE.Reg^NE(K)≤ _i _k=1^KGap_i,k^NE. (29) Thus, it suffices to bound ∑k=1KGapi,kNE _k=1^KGap_i,k^NE for each fixed agent i. Step 2: Surrogate decomposition. Fix an agent i∈ℳi . For each episode k, add and subtract the surrogate payoff of the deviation policy and the played policy: Gapi,kNE _i,k^NE =Vi,1~i†,k,σi(s1k)−~i†,k[U^ik]⏟=:Ai,k+~i†,k[U^ik]−πk[U^ik]⏟=:Bi,k = V_i,1 π_i ,k, _i(s_1^k)-E_ π_i ,k[ U_i^k]_=:A_i,k+ E_ π_i ,k[ U_i^k]-E_π^k[ U_i^k]_=:B_i,k +πk[U^ik]−Vi,1πk,σi(s1k)⏟=:Ci,k. + E_π^k[ U_i^k]-V_i,1^π^k, _i(s_1^k)_=:C_i,k. (30) Since πkπ^k is a Nash equilibrium of the surrogate normal-form game ^k ^k, Lemma 3 gives πk[U^ik]≥~i†,k[U^ik].E_π^k[ U_i^k] _ π_i ,k[ U_i^k]. Hence Bi,k≤0B_i,k≤ 0, and therefore Gapi,kNE≤Ai,k+Ci,k.Gap_i,k^NE≤ A_i,k+C_i,k. (31) Step 3: Bound the deviation-side term Ai,kA_i,k. Apply Lemma 7 with π=~i†,kπ= π_i ,k. Let fi†,k,⋆:=fi~i†,k,σif_i ,k, :=f_i π_i ,k, _i be the robust Bellman-consistent benchmark sequence for the deviation policy. Then, for every k, Ai,k≤ A_i,k≤ β^ik−1(fi†,k,⋆,g^ifi†,k,⋆,~i†,k,~i†,k)+ℛi,kA, \;β K_i^k-1\! (f_i ,k, ; g_i^f_i ,k, , π_i ,k; π_i ,k )+R^A_i,k, (32) where the residuals satisfy the cumulative bound ∑k=1Kℛi,kA≤⁡(βHBi2Kιℱ,δ+HBiKιℱ,δ+εdualapprox). _k=1^KR^A_i,k \! (β HB_i^2K _F,G^δ+HB_i K _F,G^δ+ _dual^approx ). (33) Here, (33) is the cumulative form of the benchmark-to-surrogate comparison: the CcovC_cov-term is summed using ∑k=1Kιℱ,δ|hk−1|∨1≤⁡(Kιℱ,δ), _k=1^K _F,G^δ|D_h^k-1| 1 ( K _F,G^δ), and the dual-switch and plug-in approximation terms are controlled on the uniform high-probability event. Moreover, by Lemma 6, applied uniformly over all (i,k,π)(i,k,π), and using the revised cumulative centered empirical discrepancy, we have ∑k=1K^ik−1(fi†,k,⋆,g^ifi†,k,⋆,~i†,k,~i†,k)≤⁡(HBi2Kιℱ,δ+εdualapprox). _k=1^K K_i^k-1\! (f_i ,k, ; g_i^f_i ,k, , π_i ,k; π_i ,k ) \! (HB_i^2K _F,G^δ+ _dual^approx ). (34) Summing (32) over k=1,…,Kk=1,…,K, and using (33) and (34), gives ∑k=1KAi,k≤⁡(βHBi2Kιℱ,δ+HBiKιℱ,δ+εdualapprox). _k=1^KA_i,k \! (β HB_i^2K _F,G^δ+HB_i K _F,G^δ+ _dual^approx ). (35) Step 4: Bound the played-policy term Ci,kC_i,k. By the definition of the regularized robust payoff, U^ik(πk)=fi,1k,πk(s1k)−β^ik−1(fik,πk,g^ik,πk,πk). U_i^k(π^k)=f_i,1^k,π^k(s_1^k)-β K_i^k-1(f_i^k,π^k; g_i^k,π^k;π^k). Therefore, Ci,k C_i,k =Vi,1πk,fik,πk(s1k)−Vi,1πk,σi(s1k)−β^ik−1(fik,πk,g^ik,πk,πk), =V_i,1^π^k,f_i^k,π^k(s_1^k)-V_i,1^π^k, _i(s_1^k)-β K_i^k-1(f_i^k,π^k; g_i^k,π^k;π^k), (36) where Vi,hπ,fi(s):=∼πh(⋅|s)[fi,h(s,)].V_i,h^π,f_i(s):=E_ a _h(·|s)[f_i,h(s, a)]. Now apply Definition 6 to the played sequence fik:=fik,πkf_i^k:=f_i^k,π^k and the played policies πkk=1K\π^k\_k=1^K. For any μ>0μ>0, ∑k=1K(Vi,1πk,fik(s1k)−Vi,1πk,σi(s1k)) _k=1^K (V_i,1^π^k,f_i^k(s_1^k)-V_i,1^π^k, _i(s_1^k) ) ≤1μ∑k=1K∑s=1k−1eis(fik,πk)+μdi,MADCrob+cHdi,MADCrob. ≤ 1μ _k=1^K _s=1^k-1e_i^s(f_i^k,π^k)+μ d_i,MADC^rob+cHd_i,MADC^rob. (37) Combining (36) and (37), we get ∑k=1KCi,k≤ _k=1^KC_i,k≤ 1μ∑k=1K∑s=1k−1eis(fik,πk)−β∑k=1K^ik−1(fik,g^ik,πk)+μdi,MADCrob+cHdi,MADCrob. 1μ _k=1^K _s=1^k-1e_i^s(f_i^k,π^k)-β _k=1^K K_i^k-1(f_i^k; g_i^k;π^k)+μ d_i,MADC^rob+cHd_i,MADC^rob. (38) Using Lemma 5, in its cumulative form under the revised aggregate definition of K, we have ∑k=1K∑s=1k−1eis(fik,πk)≤2∑k=1K^ik−1(fik,g^ik,πk)+⁡(HBi2Kιℱ,δ+εdualapprox). _k=1^K _s=1^k-1e_i^s(f_i^k,π^k)≤ 2 _k=1^K K_i^k-1(f_i^k; g_i^k;π^k)+O\! (HB_i^2K _F,G^δ+ _dual^approx ). (39) Substituting (39) into (38), we obtain ∑k=1KCi,k≤ _k=1^KC_i,k≤ (2μ−β)∑k=1K^ik−1(fik,g^ik,πk)+⁡(1μHBi2Kιℱ,δ+1μεdualapprox) ( 2μ-β ) _k=1^K K_i^k-1(f_i^k; g_i^k;π^k)+O\! ( 1μHB_i^2K _F,G^δ+ 1μ _dual^approx ) +μdi,MADCrob+cHdi,MADCrob. +μ d_i,MADC^rob+cHd_i,MADC^rob. (40) Choose β≥2/μβ≥ 2/μ. Since ^ik−1(⋅)≥0 K_i^k-1(·)≥ 0, the first term on the right-hand side of (40) is nonpositive and can be discarded. Thus, ∑k=1KCi,k≤⁡(1μHBi2Kιℱ,δ+μdi,MADCrob+cHdi,MADCrob+εdualapprox). _k=1^KC_i,k \! ( 1μHB_i^2K _F,G^δ+μ d_i,MADC^rob+cHd_i,MADC^rob+ _dual^approx ). (41) Step 5: Combine the deviation and played-policy bounds. By (31), ∑k=1KGapi,kNE≤∑k=1KAi,k+∑k=1KCi,k. _k=1^KGap_i,k^NE≤ _k=1^KA_i,k+ _k=1^KC_i,k. Combining (35) and (41), we obtain ∑k=1KGapi,kNE≤ _k=1^KGap_i,k^NE≤ ⁡(βHBi2Kιℱ,δ+1μHBi2Kιℱ,δ+HBiKιℱ,δCLOSE \! (β HB_i^2K _F,G^δ+ 1μHB_i^2K _F,G^δ+HB_i K _F,G^δ OPEN+μdi,MADCrob+cHdi,MADCrob+εdualapprox). +μ d_i,MADC^rob+cHd_i,MADC^rob+ _dual^approx ). (42) Step 6: Choose β and μ. Set μ=K,β=2μ=2K.μ= K, β= 2μ= 2 K. Then β≥2/μβ≥ 2/μ, and substituting these choices into (42) yields ∑k=1KGapi,kNE≤⁡(HBi2Kιℱ,δ+HBiKιℱ,δ+di,MADCrobK+cHdi,MADCrob+εdualapprox). _k=1^KGap_i,k^NE \! (HB_i^2 K _F,G^δ+HB_i K _F,G^δ+d_i,MADC^rob K+cHd_i,MADC^rob+ _dual^approx ). (43) Step 7: Maximize over agents. Taking the maximum over i∈ℳi , using dMADCrob,max:=maxi∈ℳ⁡di,MADCrob,Bϕmax:=maxi∈ℳ⁡Bϕ(σi),d_MADC^rob, := _i d_i,MADC^rob, B_φ := _i B_φ( _i), and applying (29), we obtain RegNE(K)≤⁡(dMADCrob,maxK+HdMADCrob,max+H(Bϕmax)2Kιℱ,δ+HBϕmaxKιℱ,δ+εdualapprox). ^NE(K) \! (d_MADC^rob, K+Hd_MADC^rob, +H(B_φ )^2 K _F,G^δ+HB_φ K _F,G^δ+ _dual^approx ). (44) Since Bϕmax≥1B_φ ≥ 1, the two robust estimation terms can be combined as ⁡(H(Bϕmax)2Kιℱ,δ+HBϕmaxKιℱ,δ)≤⁡(H(Bϕmax)2Kιℱ,δ).O (H(B_φ )^2 K _F,G^δ+HB_φ K _F,G^δ ) \! (H(B_φ )^2 K _F,G^δ ). Therefore, RegNE(K)≤⁡(dMADCrob,maxK+HdMADCrob,max+H(Bϕmax)2Kιℱ,δ+εdualapprox). ^NE(K) \! (d_MADC^rob, K+Hd_MADC^rob, +H(B_φ )^2 K _F,G^δ+ _dual^approx ). (45) This completes the proof of the robust NE regret bound. Robust CCE-regret bound. For each episode k∈[K]k∈[K] and agent i∈ℳi , define the unilateral coarse deviation ~i,CCE†,k:=πi†,k,CCE(π−ik)×π−ik, π_i,CCE ,k:= _i ,k,CCE( _-i^k)× _-i^k, where πi,CCE†,k(π−ik)∈arg⁡maxπi′Vi,1πi′×π−ik,σi(s1k). _i,CCE ,k( _-i^k)∈ _ _i V_i,1 _i × _-i^k, _i(s_1^k). Define the per-episode robust CCE gap Gapi,kCCE:=Vi,1~i,CCE†,k,σi(s1k)−Vi,1πk,σi(s1k).Gap_i,k^CCE:=V_i,1 π_i,CCE ,k, _i(s_1^k)-V_i,1^π^k, _i(s_1^k). By Definition 5, RegCCE(K)≤max⁡∑k=1Ki∈ℳ⁡Gapi,kCCE.Reg^CCE(K)≤ _i _k=1^KGap_i,k^CCE. If the regret definition instead sums over agents, the same argument gives the same bound with an additional factor m. We decompose each gap as Gapi,kCCE=Ai,kCCE+Bi,kCCE+Ci,kCCE,Gap_i,k^CCE=A_i,k^CCE+B_i,k^CCE+C_i,k^CCE, where Ai,kCCE A_i,k^CCE =Vi,1~i,CCE†,k,σi(s1k)−~i,CCE†,k[U^ik], =V_i,1 π_i,CCE ,k, _i(s_1^k)-E_ π_i,CCE ,k[ U_i^k], Bi,kCCE B_i,k^CCE =~i,CCE†,k[U^ik]−πk[U^ik], =E_ π_i,CCE ,k[ U_i^k]-E_π^k[ U_i^k], Ci,kCCE C_i,k^CCE =πk[U^ik]−Vi,1πk,σi(s1k). =E_π^k[ U_i^k]-V_i,1^π^k, _i(s_1^k). Key step: surrogate CCE inequality. Since πkπ^k is a CCE of the surrogate game ^k ^k, Lemma 3 gives πk[U^ik]≥~i,CCE†,k[U^ik].E_π^k[ U_i^k] _ π_i,CCE ,k[ U_i^k]. Thus Bi,kCCE≤0.B_i,k^CCE≤ 0. Bounding Ai,kCCEA_i,k^CCE and Ci,kCCEC_i,k^CCE. The terms Ai,kCCEA_i,k^CCE and Ci,kCCEC_i,k^CCE are bounded identically to the Nash case. Specifically, Ai,kCCEA_i,k^CCE is bounded by applying Lemma 7 to the coarse deviation policy ~i,CCE†,k π_i,CCE ,k, and then using Lemma 6 to control the cumulative benchmark discrepancy. The term Ci,kCCEC_i,k^CCE is the same played-policy term as in the Nash proof and is bounded by applying the robust MADC inequality together with Lemma 5. Therefore, using Bi≥1B_i≥ 1 for each fixed agent i, we have ∑k=1KGapi,kCCE≤⁡(di,MADCrobK+Hdi,MADCrob+HBi2Kιℱ,δ+εdualapprox). _k=1^KGap_i,k^CCE \! (d_i,MADC^rob K+Hd_i,MADC^rob+HB_i^2 K _F,G^δ+ _dual^approx ). Maximizing over i and using the definitions of dMADCrob,maxd_MADC^rob, and BϕmaxB_φ , we obtain RegCCE(K)≤⁡(dMADCrob,maxK+HdMADCrob,max+H(Bϕmax)2Kιℱ,δ+εdualapprox),Reg^CCE(K) \! (d_MADC^rob, K+Hd_MADC^rob, +H(B_φ )^2 K _F,G^δ+ _dual^approx ), up to a factor m if the CCE regret is defined as a sum over agents rather than a maximum over agents. Robust CE-regret bound. For each episode k∈[K]k∈[K] and agent i∈ℳi , define the strategy-modification deviation ~i†,k,CE:=ψi†,k⋄πk, π_i ,k,CE:= _i ,k π^k, where ψi†,k∈arg⁡maxψi∈ΨiVi,1ψi⋄πk,σi(s1k). _i ,k∈ _ _i∈ _iV_i,1 _i π^k, _i(s_1^k). Define the per-episode robust CE gap Gapi,kCE:=Vi,1~i†,k,CE,σi(s1k)−Vi,1πk,σi(s1k).Gap_i,k^CE:=V_i,1 π_i ,k,CE, _i(s_1^k)-V_i,1^π^k, _i(s_1^k). By Definition 5, RegCE(K)≤max⁡∑k=1Ki∈ℳ⁡Gapi,kCE.Reg^CE(K)≤ _i _k=1^KGap_i,k^CE. Again, if the CE regret is defined as a sum over agents, the same argument gives an additional factor m. We decompose Gapi,kCE=Ai,kCE+Bi,kCE+Ci,kCE,Gap_i,k^CE=A_i,k^CE+B_i,k^CE+C_i,k^CE, where Ai,kCE A_i,k^CE =Vi,1~i†,k,CE,σi(s1k)−~i†,k,CE[U^ik], =V_i,1 π_i ,k,CE, _i(s_1^k)-E_ π_i ,k,CE[ U_i^k], Bi,kCE B_i,k^CE =~i†,k,CE[U^ik]−πk[U^ik], =E_ π_i ,k,CE[ U_i^k]-E_π^k[ U_i^k], Ci,kCE C_i,k^CE =πk[U^ik]−Vi,1πk,σi(s1k). =E_π^k[ U_i^k]-V_i,1^π^k, _i(s_1^k). Key step: surrogate CE inequality. Since πkπ^k is a CE of the surrogate game ^k ^k, Lemma 3 gives πk[U^ik]≥~i†,k,CE[U^ik].E_π^k[ U_i^k] _ π_i ,k,CE[ U_i^k]. Therefore, Bi,kCE≤0.B_i,k^CE≤ 0. Bounding Ai,kCEA_i,k^CE and Ci,kCEC_i,k^CE. The bounds on Ai,kCEA_i,k^CE and Ci,kCEC_i,k^CE are identical to the Nash case. The term Ai,kCEA_i,k^CE is handled by Lemma 7 applied to the strategy-modification deviation ~i†,k,CE π_i ,k,CE, followed by Lemma 6. The term Ci,kCEC_i,k^CE is the same played-policy term and is controlled by the robust MADC inequality and Lemma 5. Hence, using Bi≥1B_i≥ 1 for each fixed agent i, we have ∑k=1KGapi,kCE≤⁡(di,MADCrobK+Hdi,MADCrob+HBi2Kιℱ,δ+εdualapprox). _k=1^KGap_i,k^CE \! (d_i,MADC^rob K+Hd_i,MADC^rob+HB_i^2 K _F,G^δ+ _dual^approx ). Maximizing over i gives RegCE(K)≤⁡(dMADCrob,maxK+HdMADCrob,max+H(Bϕmax)2Kιℱ,δ+εdualapprox),Reg^CE(K) \! (d_MADC^rob, K+Hd_MADC^rob, +H(B_φ )^2 K _F,G^δ+ _dual^approx ), again up to a factor m under a sum-over-agents regret definition. Combining the NE, CCE, and CE cases proves the theorem for ∈NE,CCE,CE Eq∈\NE,CCE,CE\. ∎ D.1 Regret bound under χ2χ^2 and KL Special cases: χ2χ^2, and KL For concreteness, we recall the resulting one-dimensional variational forms for three choices frequently used in robust RL– TV, χ2χ^2 and KL; detailed derivations can be found in [79, 29]. Here, we focus on χ and KL divergence case, and fro TV-divergence refer to Appendix E. Under the ×S×A-rectangularity assumption and eq. 4, the robust expectation for any V:→[0,H]V:S→[0,H] and Ph⋆P _h admits the following equivalent forms: • χ2χ^2-divergence (ϕ⁡(t)=(t−1)2φ(t)=(t-1)^2). One obtains a variance-sensitive form: i,h,χ2σi(s,)[f]= _P_i,h,χ^2 _i(s, a)[f]= −infν∈[0,H]σVarPh⋆(⋅|s,a)(ν−maxa′f(s′,a′))+ - _ν∈[0,H] \ σ\,Var_P_h (·|s,a) (ν- _a f(s ,a ) )_+ +s′∼ℙ⋆h(⋅|s,a)[maxa′f(s′,a′)−ν]+. +E_s _h(·|s,a) [ _a f(s ,a )-ν ]_+ \. (46) • KL-divergence (ϕ⁡(t)=tlog⁡tφ(t)=t t). The robust expectation can be written as i,h,KLσi(s,)[f]=−infν∈[ν¯,H]νlog(s′∼ℙ⋆h(⋅|s,a)[exp−maxa′f(s′,a′)/ν])+νσ, _P_i,h,KL _i(s, a)[f]=- _ν∈[ ν,\,H] \ν \! (E_s _h(·|s,a) [ \- _a f(s ,a )/ν\ ] )+νσ \, (47) where ν¯>0 ν>0 is a regularity bound on the optimal dual variable, as commonly assumed in [6, 29]. Corollary 1. Consider the same setup as Theorem 1. Define σmax:=maxi∈ℳ⁡σi. _ := _i _i. Then, with probability at least 1−δ1-δ, RoMEX-ϕ an ε -robust- Eq equilibrium with sample complexity of T=KH≍max⁡H2(dMADCrob,max)2/ε2,H7(1+σmax)4ιℱ,δ/ε2,ϕ=χ2,max⁡H2(dMADCrob,max)2/ε2,H7max⁡1,σmax4ιℱ,δ/ε2,ϕ=KL.T=KH cases \H^2 (d_MADC^rob, )^2 / ^2,\;H^7(1+ _ )^4 _F,G^δ / ^2 \,&φ=χ^2,\\[12.91663pt] \H^2 (d_MADC^rob, )^2 / ^2,\;H^7 \1, _ ^4\ _F,G^δ / ^2 \,&φ=KL. cases Proof. We provide the proof for the robust NE case. The CCE and CE cases follow analogously by replacing the NE gaps with the corresponding equilibrium gaps and using the surrogate CCE/CE deviation inequalities. For each agent i∈ℳi and episode k∈[K]k∈[K], define the per-episode robust NE gap Gapi,kNE:=Vi,1†,πk−i,σi(s1k)−Vi,1πk,σi(s1k).Gap^NE_i,k:=V , _k^-i, _i_i,1(s_1^k)-V^π^k, _i_i,1(s_1^k). Recall that RegNE(K):=max⁡∑k=1Ki∈ℳ⁡Gapi,kNE.Reg^NE(K):= _i _k=1^KGap^NE_i,k. Let the output policy be the uniform mixture πout:=1K∑k=1Kπk.π^out:= 1K _k=1^Kπ^k. Step 1: Relating output gap to cumulative regret. Fix any agent i∈ℳi . By the standard online-to-batch averaging argument, the robust gap of the uniform output policy is bounded by the average of the per-episode robust fogaps: GapiNE(πout)≤1K∑k=1KGapi,kNE.Gap^NE_i(π^out)≤ 1K _k=1^KGap^NE_i,k. Taking the maximum over agents gives GapNE(πout)=maxi∈ℳ⁡GapiNE(πout)≤1KRegNE(K).Gap^NE(π^out)= _i Gap^NE_i(π^out)≤ 1KReg^NE(K). Step 2: Applying the revised regret bound. By Theorem 1, with εdualapprox=0 _dual^approx=0, with probability at least 1−δ1-δ, RegNE(K)≤C⁡(dMADCrob,maxK+HdMADCrob,max+H(Bϕmax)2Kιℱ,δ),Reg^NE(K)≤ C (d_MADC^rob, K+Hd_MADC^rob, +H(B_φ )^2 K _F,G^δ ), where C>0C>0 is a universal constant, dMADCrob,max:=maxi∈ℳ⁡di,MADCrob,Bϕmax:=maxi∈ℳ⁡Bϕ(σi).d_MADC^rob, := _i d_i,MADC^rob, B_φ := _i B_φ( _i). Therefore, GapNE(πout)≤C⁡(dMADCrob,maxK+HdMADCrob,maxK+H(Bϕmax)2ιℱ,δK).Gap^NE(π^out)≤ C ( d_MADC^rob, K+ Hd_MADC^rob, K+H(B_φ )^2 _F,G^δK ). Step 3: Choosing K. To ensure GapNE(πout)≤εGap^NE(π^out)≤ , it is sufficient that dMADCrob,maxK≲ε,HdMADCrob,maxK≲ε,H(Bϕmax)2ιℱ,δK≲ε. d_MADC^rob, K , Hd_MADC^rob, K , H(B_φ )^2 _F,G^δK . Equivalently, K≳max⁡(dMADCrob,max)2ε2,HdMADCrob,maxε,H2(Bϕmax)4ιℱ,δε2.K \ (d_MADC^rob, )^2 ^2, Hd_MADC^rob, , H^2(B_φ )^4 _F,G^δ ^2 \. Multiplying by H, the sample complexity T=KHT=KH satisfies T=⁡(max⁡H(dMADCrob,max)2ε2,H2dMADCrob,maxε,H3(Bϕmax)4ιℱ,δε2).T=O\! ( \ H(d_MADC^rob, )^2 ^2, H^2d_MADC^rob, , H^3(B_φ )^4 _F,G^δ ^2 \ ). Since H≥1H≥ 1, dMADCrob,max≥1d_MADC^rob, ≥ 1, and ε∈(0,1] ∈(0,1], the first two terms are jointly dominated by H2(dMADCrob,max)2ε2. H^2(d_MADC^rob, )^2 ^2. Thus, it is enough to take T=⁡(max⁡H2(dMADCrob,max)2ε2,H3(Bϕmax)4ιℱ,δε2).T=O\! ( \ H^2(d_MADC^rob, )^2 ^2, H^3(B_φ )^4 _F,G^δ ^2 \ ). Under this choice, GapNE(πout)≤εGap^NE(π^out)≤ , so πoutπ^out is an ε -robust NE policy. Step 4: Divergence-specific sample-complexity bounds. Finally, substitute the divergence-specific bounds from Assumption 3 and by using (46) (for χ2χ^2) and (47) (for KL), we get Bχ2(σi)=⁡(H⁡(1+σi)),andBKL(σi)=⁡(Hmax⁡1,σi).B_χ^2( _i)=O\! (H(1+ _i) ), B_KL( _i)=O\! (H \1, _i\ ). For heterogeneous radii, define σmin:=mini∈ℳ⁡σi,σmax:=maxi∈ℳ⁡σi. _ := _i _i, _ := _i _i. Then Bχ2max=⁡(H⁡(1+σmax)),andBKLmax=⁡(Hmax⁡1,σmax).B_χ^2 =O\! (H(1+ _ ) ), B_KL =O\! (H \1, _ \ ). Substituting these bounds into T=⁡(max⁡H2(dMADCrob,max)2ε2,H3(Bϕmax)4ιℱ,δε2)T=O\! ( \ H^2(d_MADC^rob, )^2 ^2, H^3(B_φ )^4 _F,G^δ ^2 \ ) gives ϕ=χ2:T=⁡(max⁡H2(dMADCrob,max)2ε2,H7(1+σmax)4ιℱ,δε2),φ=χ^2: T=O\! ( \ H^2(d_MADC^rob, )^2 ^2, H^7(1+ _ )^4 _F,G^δ ^2 \ ), and ϕ=KL:T=⁡(max⁡H2(dMADCrob,max)2ε2,H7max⁡1,σmax4ιℱ,δε2).φ=KL: T=O\! ( \ H^2(d_MADC^rob, )^2 ^2, H^7 \1, _ ^4\ _F,G^δ ^2 \ ). The CCE and CE cases follow identically from the same online-to-batch argument using the robust CCE and robust CE regret bounds from Theorem 1. Hence, under the stated choice of T, the output policy is an ε -robust- Eq policy for ∈NE,CCE,CE Eq∈\NE,CCE,CE\. ∎ Appendix E Theoretical Guarantee of TV-divergence Uncertainty Set Focusing on DRMGs with TV-distance uncertainty sets, [46, 42]. showed that efficient online learning is achievable under suitable assumptions; in particular, [43] introduced the following fail-states assumption. Assumption 4 (Failure States). For any agent i∈ℳi , there exists an (agent-specified) set of failure states fi⊆S_f^i , such that ri(s,)=0r_i(s, a)=0, and Ph⋆(s′|s,)=0P _h(s |s, a)=0, ∀∈,∀s∈fi,∀s′∉fi∀ a ,∀ s ^i_f,∀ s ^i_f. The paper [29] highlights that existing efficient learning results are tailored to TV-distance uncertainty sets and do not extend to more general ϕφ-divergence settings (e.g., KL or χ2χ^2). In these broader settings, key structural properties break down, making online learning in robust MDPs fundamentally challenging without additional assumptions. To address this, the authors introduce an intrinsic difficulty measure based on the ratio between nominal and worst-case visitation distributions, and assume this ratio is bounded (polynomially), enabling tractable analysis. Motivated by this, we apply the following definition to tackle the issue of the TV case, as follows: Definition 7 (Robust-to-nominal concentrability). For every agent i∈ℳi , stage h∈[H]h∈[H], episode k∈[K]k∈[K], and policy π∈Πpurπ∈ ^pur, let νi,hπ _i,h^π denote the stage-h state–joint-action occupancy induced by π under the worst-case transition kernels attaining the robust Bellman recursion for agent i. Let μ¯hk−1:=1k−1∑j=1k−1μhπj μ_h^k-1:= 1k-1 _j=1^k-1 _h^π^j denote the historical nominal data-generating mixture distribution at stage h, with the convention that the first episode is handled separately. We denote the finite constant Ccov≥1C_cov≥ 1 as the smallest value such that ‖dνi,hπdμ¯hk−1‖∞≤Ccov. \| d _i,h^πd μ_h^k-1 \|_∞≤ C_cov. This value exists due to the support inclusion Definition 1. Theorem 2 (Unified robust equilibrium-regret bound for RoMEX-ϕφ). Consider the same setup as in Theorem 1. Then, under Definition 7 with probability at least 1−δ1-δ, the corresponding robust equilibrium regret for TV-divergence set satisfies RegTV(K)≤⁡(dMADCrob,maxK+HdMADCrob,max+CcovH(Bϕmax)2Kιℱ,δ+εdualapprox). Eq_TV(K) \! (d_MADC^rob, K+Hd_MADC^rob, +C_covH(B_φ )^2 K _F,G^δ+ _dual^approx ). Consequently, for εdualapprox=0 ^approx_dual=0, and assuming Assumption 4 for TV-divergence, with probability at least 1−δ1-δ, the sample complexity of RoMEX-TV to obtain an ε -robust- Eq policy is T=KHT=KH, where T=⁡(max⁡H2(dMADCrob,max)2/ε2,Ccov2H7(min⁡H,1/σmin)4ιℱ,δ/ε2).T=O\! ( \H^2 (d_MADC^rob, )^2 / ^2,\;C_cov^2H^7( \H,1/ _ \)^4 _F,G^δ / ^2 \ ). Proof. For the TV case, we first revise the proof of Lemma 7, where we apply Definition 7 in (207) and (208), we have ∑h=1H‖i,hπ,ϕ,σifi,h+1k,π−g^i,hk,π,i,hπ,ϕ,σifi,h+1k,π‖1,νi,hπ _h=1^H \|T_i,h^π,φ, _if_i,h+1^k,π-T_ g_i,h^k,π,i,h^π,φ, _if_i,h+1^k,π \|_1, _i,h^π ≤CcovHBϕ(σi)log⁡(|ℱ‖Πpur|KHm/δ)|hk−1|∨1 ≤ C_covHB_φ( _i) \! (|F||G|| ^pur|KHm/δ )|D_h^k-1| 1 +CcovHϵ¯dual2. +C_covH ε_dual^2. (48) ∑h=1Hνi,hπ[(g^i,hk,π,i,hπ,ϕ,σifi,h+1k,π−fi,hk,π)+] _h=1^HE_ _i,h^π [ (T_ g_i,h^k,π,i,h^π,φ, _if_i,h+1^k,π-f_i,h^k,π )_+ ] ≤CcovHBϕ(σi)log⁡(|ℱ‖Πpur|KHm/δ)|hk−1|∨1 ≤ C_covHB_φ( _i) \! (|F||G|| ^pur|KHm/δ )|D_h^k-1| 1 +CcovHϵ¯dual2. +C_covH ε_dual^2. (49) Substituting (202), (203), (48) and (49) into (198), and choosing ϵ¯dual1≤εdualapproxβHKBϕ(σi),ϵ¯dual2≤εdualapproxCcovH, ε_dual^1≤ _dual^approxβ HKB_φ( _i), ε_dual^2≤ _dual^approxC_covH, we obtain the revised bound of Lemma 7 as Vi,1π,σi(s1k)−U^ik(π)≤ V_i,1^π, _i(s_1^k)- U_i^k(π)≤ β^ik−1(fi⋆,g^ifi⋆,π,π)+⁡(βHBϕ(σi)2Klog⁡|ℱ‖Πpur|KHmδCLOSE \;β K_i^k-1 (f_i ; g_i^f_i ,π;π )+O\! (β HB_φ( _i)^2 K \! |F||G|| ^pur|KHmδ OPEN+CcovHBϕ(σi)log⁡(|ℱ‖Πpur|KHm/δ)|hk−1|∨1+εdualapprox). +C_covHB_φ( _i) \! (|F||G|| ^pur|KHm/δ )|D_h^k-1| 1+ _dual^approx ). (50) We now apply (50) in the proof of Theorem 1, particularly (32)–(35), we get ∑k=1KAi,k≤⁡(βHBi2Kιℱ,δ+CcovHBiKιℱ,δ+εdualapprox). _k=1^KA_i,k \! (β HB_i^2K _F,G^δ+C_covHB_i K _F,G^δ+ _dual^approx ). (51) We use (51) and follow the proof lines (36)–(44). Therefore, we get the regret bound of TV-divergence as RegNE(K)≤⁡(dMADCrob,maxK+HdMADCrob,max+H(Bϕmax)2Kιℱ,δ+CcovHBϕmaxKιℱ,δ+εdualapprox). ^NE(K) \! (d_MADC^rob, K+Hd_MADC^rob, +H(B_φ )^2 K _F,G^δ+C_covHB_φ K _F,G^δ+ _dual^approx ). (52) Using the fact Since Bϕmax≥1B_φ ≥ 1 and Ccov≥1C_cov≥ 1, we have RegNE(K)≤⁡(dMADCrob,maxK+HdMADCrob,max+CcovH(Bϕmax)2Kιℱ,δ+εdualapprox). ^NE(K) \! (d_MADC^rob, K+Hd_MADC^rob, +C_covH(B_φ )^2 K _F,G^δ+ _dual^approx ). (53) The robust CCE and CE regret bounds follow by the same argument, replacing the surrogate Nash deviation step with the CCE/CE deviation inequalities in the revised bound of Lemma 3 (i.e. (50)). Therefore, for ∈NE,CCE,CE Eq∈\NE,CCE,CE\, Reg(K)≤⁡(dMADCrob,maxK+HdMADCrob,max+CcovH(Bϕmax)2Kιℱ,δ+εdualapprox).Reg Eq(K) \! (d_MADC^rob, K+Hd_MADC^rob, +C_covH(B_φ )^2 K _F,G^δ+ _dual^approx ). We now follow the proof lines for Corollary 1 for the TV case. Under the ×S×A-rectangularity assumption, Assumption 4, and eq. 4, the robust expectation for any V:→[0,H]V:S→[0,H] and Ph⋆P _h admits the following equivalent form of TV-divergence uncertainty set (ϕ⁡(t)=|t−1|φ(t)=|t-1|) as i,h,TVσi(s,)[f]=−infν∈[0,2H/σ]s′∼ℙ⋆h(⋅|s,a)[ν−maxa′f(s′,a′)]++(1−σ)ν. _P_i,h,TV _i(s, a)[f]=- _ν∈[0,2H/σ] \E_s _h(·|s,a) [ν- _a f(s ,a ) ]_++(1-σ)ν \. (54) We follow the standard online-to-batch averaging argument [8]. We use (54) for the proof of the Corollary 1 for TV case. We apply εdualapprox=0 _dual^approx=0 with the fact that dMADCrob,max:=maxi∈ℳ⁡di,MADCrobd_MADC^rob, := _i d_i,MADC^rob, and Bϕmax:=maxi∈ℳ⁡Bϕ(σi).B_φ := _i B_φ( _i). and get the revised bound of GapEq(πout)Gap^Eq(π^out) as GapNE(πout)≤C⁡(dMADCrob,maxK+HdMADCrob,maxK+CcovH(Bϕmax)2ιℱ,δK).Gap^NE(π^out)≤ C ( d_MADC^rob, K+ Hd_MADC^rob, K+C_covH(B_φ )^2 _F,G^δK ). Using the facts H≥1H≥ 1, dMADCrob,max≥1d_MADC^rob, ≥ 1, and ε∈(0,1] ∈(0,1], we get T=⁡(max⁡H2(dMADCrob,max)2ε2,Ccov2H3(Bϕmax)4ιℱ,δε2).T=O\! ( \ H^2(d_MADC^rob, )^2 ^2, C^2_covH^3(B_φ )^4 _F,G^δ ^2 \ ). Finally, substitute the divergence-specific bounds from Assumption 3 and by using (54) (for TV), BTV(σi)=⁡(Hmin⁡H,1/σi).B_TV( _i)=O\! (H \H,1/ _i\ ). For heterogeneous radii, define σmin:=mini∈ℳ⁡σi,σmax:=maxi∈ℳ⁡σi. _ := _i _i, _ := _i _i. Then BTVmax=maxi∈ℳ⁡BTV(σi)=⁡(Hmin⁡H,1/σmin),B_TV = _i B_TV( _i)=O\! (H \H,1/ _ \ ), because min⁡H,1/σi \H,1/ _i\ is decreasing in σi _i. Substituting these bounds into T=⁡(max⁡H2(dMADCrob,max)2ε2,Ccov2H3(Bϕmax)4ιℱ,δε2)T=O\! ( \ H^2(d_MADC^rob, )^2 ^2, C_cov^2H^3(B_φ )^4 _F,G^δ ^2 \ ) gives T=⁡(max⁡H2(dMADCrob,max)2ε2,Ccov2H7(min⁡H,1/σmin)4ιℱ,δε2).T=O\! ( \ H^2(d_MADC^rob, )^2 ^2, C_cov^2H^7( \H,1/ _ \)^4 _F,G^δ ^2 \ ). This completes the proof. ∎ Appendix F Multi-Agent Robust Bellman Eluder Dimension The work [36] defines the Bellman-Eluder dimension in the standard non-robust setup. We combine with the work of [26] and extend to the DRMG setup. Let X be a domain and Φ⊆(→ℝ) (X ) be a function class. Let Π be a family of probability distributions over X. Definition 8 (Distributional ϵε-Dependence [26]). A distribution μ∈Πμ∈ is said to be ϵε-dependent on a set μ1,…,μn⊆Π\ _1,..., _n\ with respect to Ξ if for all ξ∈Ξξ∈ , |μ[ξ]|≤ϵ+∑i=1nμi[ξ]2. _μ[ξ] ≤ε+ _i=1^nE_ _i[ξ]^2. (55) Otherwise, μ is said to be ϵε-independent of μ1,…,μn\ _1,..., _n\. Definition 9 (Distributional Eluder (DE) Dimension [26]). The distributional Eluder dimension of Ξ with respect to Π at scale ϵ>0ε>0, denoted as dimDE(Ξ,Π,ϵ) _DE( , ,ε), is the length of the longest sequence (x1,…,xn)∈Π(x_1,...,x_n)∈ such that for each j∈[n]j∈[n] there exists ϵ′>ϵε >ε such that xjx_j is ϵ′ε -independent of x1,…,xj−1\x_1,...,x_j-1\. To formally define the robust Bellman-Eluder (BE) dimension, we must first define the robust Bellman residual class per agent at stage h as, Resiπ(ℱi)≔fh−i,hπ,ϕ,σifh+1:f∈ℱi.Res^π_i(F_i) \f_h-T_i,h^π,φ, _if_h+1:f _i\. (56) Now, we define the robust BE as follows: Definition 10 (Robust Multi-Agent Bellman Eluder Dimension). Let Π=Πhh=1H =\ _h\_h=1^H be the set of H classes where Πh _h is the distributions over ×S×A at horizon h. We then define the robust Bellman-Eluder dimension of ℱiF_i by dMABErob,max(ℱ,Π,ϵ) d_MABE^rob, (F, ,ε) ≔maxi∈ℳ⁡di,MABErob(ℱi,Π,ϵ), _i d_i,MABE^rob(F_i, ,ε), wheredi,MABErob(ℱi,Π,ϵ) d_i,MABE^rob(F_i, ,ε) ≔maxh∈H⁡dimDE(⋃πh∈ΠPurResiπ(ℱi),Πh,ϵ), _h∈ H \dim_DE ( _ _h∈ ^PurRes^π_i(F_i), _h,ε ) \, (57) where Resi(ℱi)Res_i(F_i) is the robust Bellman residual class and dimDE(Ξ,Π,ϵ)dim_DE( , ,ε) denote the DE dimension of a function class Ξ with respect to a distribution family Π at scale ϵε [36]. Following [78], we define a robust version of the Multi-Agent Decoupling Coefficient in Definition 6. Theorem 3 (Low Robust MABE ⇒ Low Robust MADC). Fix an agent i∈ℳi . Let K be any integer and ℱF be a hypothesis class under the model-free setting as given in Section 3 as ℱ=⨂i∈ℳℱi,whereℱi=⨂h=1Hℱi,h, = _i F_i, _i= _h=1^HF_i,h, (58) where (58) is a class of Q-functions that satisfies Assumption 1. For ε>0 >0, we denote dMABErob,max(ℱ,Π,ϵ)≔maxi∈ℳ⁡di,MABErob(ℱi,Π,ϵ)d_MABE^rob, (F, ,ε) _i d_i,MABE^rob(F_i, ,ε) as the robust MABE dimension at scale ε , as defined in Definition 10 for the function class ℱF. Let K≥2K≥ 2. Let di,MADCrobd^rob_i,MADC denote the robust MADC as defined in Definition 6 by setting ε=1/K =1/K and we denote dMADCrob,max=maxi∈ℳ⁡di,MADCrobd_MADC^rob, = _i d_i,MADC^rob. Then, with the discrepancy function, the robust MADC satisfies dMADCrob,max:=max⁡cHdMABErob,max(ℱ,Π,1/K)log⁡K, 1=⁡(HdMABErob,max(ℱ,Π,1/K)log⁡K). d_MADC^rob, := \! \c\,H\,d_MABE^rob, (F, ,1/K) K,\ 1 \=O\! (Hd_MABE^rob, (F, ,1/K) K ). (59) Proof. We prove the theorem for a fixed agent i∈ℳi , since the robust MADC in Definition 6 is defined agent-wise. For notational simplicity, throughout the proof we suppress the agent index i whenever there is no ambiguity. The proof follows the bucketing argument used in the non-robust MAMEX analysis, but with one important modification. The training-error terms are computed under the nominal historical visitation distributions μhπss<k\ _h^π^s\_s<k, whereas the robust value prediction error is controlled through the worst-case occupancy measures νhπkk∈[K]\ _h^π^k\_k∈[K]. This distinction is necessary because robust MADC relates nominally observed Bellman errors to value prediction under adversarial transition kernels. Define robust Bellman residuals. Fix an arbitrary sequence (fk,πk)k=1K\(f^k,π^k)\_k=1^K, where fk=fhkh=1H∈ℱif^k=\f_h^k\_h=1^H _i. For each episode k and stage h, define the robust Bellman residual ψhk(s,):=fhk(s,)−[hπk,ϕ,σifh+1k](s,). _h^k(s, a):=f_h^k(s, a)- [T_h^π^k,φ, _if_h+1^k ](s, a). (60) By Assumption 1, hπk,ϕ,σifh+1k∈ℱi,hT_h^π^k,φ, _if_h+1^k _i,h, and hence ψhk _h^k belongs to the corresponding robust Bellman residual class. Define nominal cross-episode residuals and robust diagonal residuals. For each behavior episode s<ks<k, define the nominal cross-episode residual mean ahs,k:=|(sh,h)∼μhπs[ψhk(sh,h)]|. a_h^s,k:= |E_(s_h, a_h) _h^π^s [ _h^k(s_h, a_h) ] |. (61) Here, μhπs _h^π^s is the stage-h state–joint-action visitation distribution induced by executing πsπ^s under the nominal kernel P⋆P . For the diagonal prediction term, let νhπk _h^π^k denote the stage-h state–joint-action occupancy induced by πkπ^k under the worst-case transition kernels attaining the robust Bellman recursion for agent i. Define bhk:=|(sh,h)∼νhπk[ψhk(sh,h)]|. b_h^k:= |E_(s_h, a_h) _h^π^k [ _h^k(s_h, a_h) ] |. (62) The distinction between ahs,ka_h^s,k and bhkb_h^k is central: the former is measured under the nominal historical data distributions, while the latter controls robust value prediction error under worst-case dynamics. For a scale ε>0 >0, define the truncated quantities uhs,k:=ahs,kahs,k>ε,s<k, u_h^s,k:=a_h^s,k1\a_h^s,k> \, s<k, (63) and vhk:=bhkbhk>ε. v_h^k:=b_h^k1\b_h^k> \. (64) Bucket construction. Fix a stage h∈[H]h∈[H]. Initialize buckets Bh0,Bh1,…,BhK−1B_h^0,B_h^1,…,B_h^K-1. Process episodes k=1,…,Kk=1,…,K sequentially. If vhk=0v_h^k=0, then episode k is not inserted into any bucket. Otherwise, insert k into the first bucket BhjB_h^j satisfying ∑s<ks∈Bhj(uhs,k)2<(vhk)2. _ subarraycs<k\\ s∈ B_h^j subarray (u_h^s,k )^2< (v_h^k )^2. (65) Let bhk∈0,…,K−1b_h^k∈\0,…,K-1\ denote the bucket index assigned to episode k. Lower bound from the bucket rule. If episode k is inserted into bucket BhbhkB_h^b_h^k, then for every j<bhkj<b_h^k, the insertion rule must have failed. Therefore, ∑s<ks∈Bhj(uhs,k)2≥(vhk)2,j=0,1,…,bhk−1. _ subarraycs<k\\ s∈ B_h^j subarray (u_h^s,k )^2≥ (v_h^k )^2, j=0,1,…,b_h^k-1. (66) Summing over j=0,…,bhk−1j=0,…,b_h^k-1, we get ∑j=0bhk−1∑s<ks∈Bhj(uhs,k)2≥bhk(vhk)2. _j=0^b_h^k-1 _ subarraycs<k\\ s∈ B_h^j subarray (u_h^s,k )^2≥ b_h^k (v_h^k )^2. (67) Now summing over k=1,…,Kk=1,…,K, the left-hand side is a sub-sum of ∑k=1K∑s=1k−1(uhs,k)2 _k=1^K _s=1^k-1(u_h^s,k)^2. Hence, ∑k=1K∑s=1k−1(uhs,k)2≥∑k=1Kbhk(vhk)2. _k=1^K _s=1^k-1 (u_h^s,k )^2≥ _k=1^Kb_h^k (v_h^k )^2. (68) Bucket size bound via robust MABE. Fix a bucket Bhj=k1<k2<⋯<km.B_h^j=\k_1<k_2<·s<k_m\. Suppose episode kℓk_ is inserted into BhjB_h^j. By the insertion rule, ∑q<ℓ(uhkq,kℓ)2<(vhkℓ)2. _q< (u_h^k_q,k_ )^2< (v_h^k_ )^2. (69) Since vhkℓ>0v_h^k_ >0, we have bhkℓ>εb_h^k_ > . Choose ε′∈(ε,bhkℓ) ∈( ,b_h^k_ ) such that ∑q<ℓ(uhkq,kℓ)2≤ε′. _q< (u_h^k_q,k_ )^2≤ . (70) Then, |νhπkℓψhkℓ|=bhkℓ>ε′, |E_ _h^π^k_ _h^k_ |=b_h^k_ > , (71) while ∑q<ℓ(μhπkqψhkℓ)2≤∑q<ℓ(uhkq,kℓ)2≤ε′. _q< (E_ _h^π^k_q _h^k_ )^2≤ _q< (u_h^k_q,k_ )^2≤ . (72) Thus, the worst-case occupancy νhπkℓ _h^π^k_ is ε′ -independent of the preceding nominal occupancies μhπkqq<ℓ\ _h^π^k_q\_q< with respect to the robust Bellman residual class. By Definition 10, the number of such insertions in one bucket is at most Dεi:=di,MABErob(ℱi,Π,ε).D_ ^i:=d_i,MABE^rob(F_i, , ). Therefore, |Bhj|≤Dεi,∀j. |B_h^j|≤ D_ ^i, ∀ j. (73) Lower bound by regrouping and Cauchy–Schwarz. Regroup the right-hand side of (68) by buckets: ∑k=1Kbhk(vhk)2=∑j=1K−1j∑k∈Bhj(vhk)2. _k=1^Kb_h^k (v_h^k )^2= _j=1^K-1j _k∈ B_h^j (v_h^k )^2. (74) For each bucket BhjB_h^j, Cauchy–Schwarz gives ∑k∈Bhj(vhk)2≥(∑k∈Bhjvhk)2|Bhj|≥(∑k∈Bhjvhk)2Dεi, _k∈ B_h^j (v_h^k )^2≥ ( _k∈ B_h^jv_h^k )^2|B_h^j|≥ ( _k∈ B_h^jv_h^k )^2D_ ^i, (75) where the last step uses (73). Defining xj:=∑k∈Bhjvhk,x_j:= _k∈ B_h^jv_h^k, we obtain ∑k=1K∑s=1k−1(uhs,k)2≥1Dεi∑j=1K−1jxj2. _k=1^K _s=1^k-1 (u_h^s,k )^2≥ 1D_ ^i _j=1^K-1jx_j^2. (76) Lower bound via the numerical lemma. Applying Lemma T.4 to the nonnegative sequence xjj=1K−1\x_j\_j=1^K-1 yields ∑j=1K−1jxj2≥11+log⁡K(∑j=1K−1xj)2. _j=1^K-1jx_j^2≥ 11+ K ( _j=1^K-1x_j )^2. (77) Since ∑j=1K−1xj=∑k∈[K]∖Bh0vhk, _j=1^K-1x_j= _k∈[K] B_h^0v_h^k, combining (76) and (77) gives ∑k∈[K]∖Bh0vhk≤Dεi(1+log⁡K)∑k=1K∑s=1k−1(uhs,k)2. _k∈[K] B_h^0v_h^k≤ D_ ^i(1+ K) _k=1^K _s=1^k-1 (u_h^s,k )^2. (78) Control the diagonal truncated terms. By construction, bhk=bhkbhk≤ε+bhkbhk>ε≤ε+vhk.b_h^k=b_h^k1\b_h^k≤ \+b_h^k1\b_h^k> \≤ +v_h^k. Therefore, ∑k=1Kbhk≤Kε+∑k=1Kvhk. _k=1^Kb_h^k≤ K + _k=1^Kv_h^k. (79) Next, decompose ∑k=1Kvhk=∑k∈Bh0vhk+∑k∈[K]∖Bh0vhk. _k=1^Kv_h^k= _k∈ B_h^0v_h^k+ _k∈[K] B_h^0v_h^k. Since the value functions are bounded and the robust Bellman backups are also bounded, there exists an absolute constant CH=⁡(H)C_H=O(H) such that |ψhk(s,)|≤CH.| _h^k(s, a)|≤ C_H. For concreteness, one may take CH=2HC_H=2H. Thus, ∑k∈Bh0vhk≤2H|Bh0|≤2HDεi. _k∈ B_h^0v_h^k≤ 2H|B_h^0|≤ 2HD_ ^i. (80) Combining (79), (80), and (78), we get ∑k=1Kbhk≤Kε+2HDεi+Dεi(1+log⁡K)∑k=1K∑s=1k−1(uhs,k)2. _k=1^Kb_h^k≤ K +2HD_ ^i+ D_ ^i(1+ K) _k=1^K _s=1^k-1 (u_h^s,k )^2. (81) Sum over stages. Summing (81) over h=1,…,Hh=1,…,H and using ∑h=1Hzh≤H∑h=1Hzh _h=1^H z_h≤ H _h=1^Hz_h, we obtain ∑h=1H∑k=1Kbhk _h=1^H _k=1^Kb_h^k ≤HKε+2H2Dεi+DεiH(1+log⁡K)∑h=1H∑k=1K∑s=1k−1(uhs,k)2. ≤ HK +2H^2D_ ^i+ D_ ^iH(1+ K) _h=1^H _k=1^K _s=1^k-1 (u_h^s,k )^2. (82) Relate cross terms to the robust Bellman training error. By Jensen’s inequality and the definition of uhs,ku_h^s,k, (uhs,k)2 (u_h^s,k )^2 ≤(μhπsψhk)2≤μhπs[(ψhk)2]. ≤ (E_ _h^π^s _h^k )^2 _ _h^π^s [ ( _h^k )^2 ]. (83) We have defined the robust discrepancy as eis(fik,πk):=∑h=1H(sh,h)∼μhπs[((I−i,hπk,ϕ,σi)fi,hk)2]. e_i^s(f_i^k,π^k):= _h=1^HE_(s_h, a_h) _h^π^s [ ((I-T_i,h^π^k,φ, _i)f_i,h^k )^2 ]. (84) Therefore, (83) over (h,k,s)(h,k,s) yields ∑h=1H∑k=1K∑s=1k−1(uhs,k)2 _h=1^H _k=1^K _s=1^k-1 (u_h^s,k )^2 ≤∑k=1K∑s=1k−1∑h=1Hμhπs[(fhk−hπk,ϕ,σifh+1k)2] ≤ _k=1^K _s=1^k-1 _h=1^HE_ _h^π^s [ (f_h^k-T_h^π^k,φ, _if_h+1^k )^2 ] =∑k=1K∑s=1k−1ei,1s(fik,πk). = _k=1^K _s=1^k-1e_i,1^s(f_i^k,π^k). (85) Substituting (85) into (82) yields ∑h=1H∑k=1Kbhk≤HKε+2H2Dεi+DεiH(1+log⁡K)∑k=1K∑s=1k−1ei,1s(fik,πk). _h=1^H _k=1^Kb_h^k≤ HK +2H^2D_ ^i+ D_ ^iH(1+ K) _k=1^K _s=1^k-1e_i,1^s(f_i^k,π^k). (86) Robust Bellman telescoping under worst-case occupancy. We now relate the diagonal residual terms to the robust value prediction error. For each episode k, by the robust Bellman telescoping identity along the worst-case trajectory distribution induced by πkπ^k, we have Vi,1πk,fik(s1k)−Vi,1πk,σi(s1k)≤∑h=1H|(sh,h)∼νhπk[ψhk(sh,h)]|=∑h=1Hbhk. V_i,1^π^k,f_i^k(s_1^k)-V_i,1^π^k, _i(s_1^k)≤ _h=1^H |E_(s_h, a_h) _h^π^k [ _h^k(s_h, a_h) ] |= _h=1^Hb_h^k. (87) Summing over k=1,…,Kk=1,…,K, we obtain ∑k=1K(Vi,1πk,fik(s1k)−Vi,1πk,σi(s1k))≤∑h=1H∑k=1Kbhk. _k=1^K (V_i,1^π^k,f_i^k(s_1^k)-V_i,1^π^k, _i(s_1^k) )≤ _h=1^H _k=1^Kb_h^k. (88) Combining (88) and (86) gives ∑k=1K(Vi,1πk,fik(s1k)−Vi,1πk,σi(s1k))≤HKε+2H2Dεi+DεiH(1+log⁡K)∑k=1K∑s=1k−1ei,1s(fik,πk). _k=1^K (V_i,1^π^k,f_i^k(s_1^k)-V_i,1^π^k, _i(s_1^k) )≤ HK +2H^2D_ ^i+ D_ ^iH(1+ K) _k=1^K _s=1^k-1e_i,1^s(f_i^k,π^k). (89) Apply Young’s inequality. Let Z:=∑k=1K∑s=1k−1ei,1s(fik,πk),Aε:=DεiH(1+log⁡K).Z:= _k=1^K _s=1^k-1e_i,1^s(f_i^k,π^k), A_ :=D_ ^iH(1+ K). Then the square-root term in (89) is AεZ A_ Z. For any α>0α>0, Young’s inequality gives AεZ≤1αZ+αAε, A_ Z≤ 1αZ+α A_ , up to an absolute constant absorbed into the final ⁡(⋅)O(·) notation. Hence, ∑k=1K(Vi,1πk,fik(s1k)−Vi,1πk,σi(s1k))≤1α∑k=1K∑s=1k−1ei,1s(fik,πk)+αDεiH(1+log⁡K)+HKε+2H2Dεi. _k=1^K (V_i,1^π^k,f_i^k(s_1^k)-V_i,1^π^k, _i(s_1^k) )≤ 1α _k=1^K _s=1^k-1e_i,1^s(f_i^k,π^k)+α D_ ^iH(1+ K)+HK +2H^2D_ ^i. (90) Choose the scale and identify robust MADC. Set ε=1/K =1/K. Then HKε=HHK =H. Define D1/Ki:=di,MABErob(ℱi,Π,1/K).D_1/K^i:=d_i,MABE^rob(F_i, ,1/K). For K≥2K≥ 2, let di,MADCrob:=CHD1/Kilog⁡Kd_i,MADC^rob:=C\,HD_1/K^i K for a sufficiently large absolute constant C>0C>0. Then αD1/KiH(1+log⁡K)≤αdi,MADCrob,α D_1/K^iH(1+ K)≤α d_i,MADC^rob, and H+2H2D1/Ki≤cHdi,MADCrob,H+2H^2D_1/K^i≤ cHd_i,MADC^rob, after enlarging constants if necessary. Therefore, ∑k=1K(Vi,1πk,fik(s1k)−Vi,1πk,σi(s1k))≤1α∑k=1K∑s=1k−1ei,1s(fik,πk)+αdi,MADCrob+cHdi,MADCrob. _k=1^K (V_i,1^π^k,f_i^k(s_1^k)-V_i,1^π^k, _i(s_1^k) )≤ 1α _k=1^K _s=1^k-1e_i,1^s(f_i^k,π^k)+α d_i,MADC^rob+cHd_i,MADC^rob. (91) Since α>0α>0 was arbitrary, this is precisely the robust MADC inequality in Definition 6. Consequently, di,MADCrob=⁡(HD1/Kilog⁡K).d_i,MADC^rob=O\! (HD_1/K^i K ). Finally, taking the maximum over agents gives dMADCrob,max=maxi∈ℳ⁡di,MADCrob(ℱ,Π,1/K)=⁡(HdMABErob,max(ℱ,Π,1/K)log⁡K),d_MADC^rob, = _i d_i,MADC^rob(F, ,1/K)=O\! (Hd_MABE^rob, (F, ,1/K) K ), where dMABErob,max(ℱ,Π,ε)≔maxi∈ℳ⁡di,MABErob(ℱi,Π,ε)d_MABE^rob, (F, , ) _i d_i,MABE^rob(F_i, , ) for ε=1/K =1/K. This completes the proof. ∎ Combining Theorem 1 with Theorem 3, we obtain that RoMEX-ϕφ achieves sublinear robust equilibrium regret whenever the robust MABE dimension is finite. In particular, substituting dMADCrob,max=~(HdMABErob,max(ℱ,Π,1/K))d^rob, _MADC= O\! (Hd_MABE^rob, (F, ,1/K) ) into the regret bound of Theorem 1 yields RegEq(K)=~(CLOSE ^Eq(K)= O\! ( HdMABErob,max(ℱ,Π,1/K)K+H2dMABErob,max(ℱ,Π,1/K) Hd_MABE^rob, (F, ,1/K) K+H^2d_MABE^rob, (F, ,1/K) OPEN+H(Bϕmax)2K+εdualapprox), +H\,(B _φ)^2 K+ ^approx_dual ), where ~(⋅) O(·) hides logarithmic factors in K, H, |ℱ||F|, |||G|, and |Πpur|| ^pur|. For TV-case, the regret bound of Theorem 2 yields ~(HdMABErob,max(ℱ,Π,1/K)K+H2dMABErob,max(ℱ,Π,1/K)+CcovH(Bϕmax)2K+εdualapprox) O\! (Hd_MABE^rob, (F, ,1/K) K+H^2d_MABE^rob, (F, ,1/K)+C_covH\,(B _φ)^2 K+ ^approx_dual ) This decomposition reveals two fundamental sources of complexity. The first is an exploration term HdMABErob,max(ℱ,Π,1/K)KHd_MABE^rob, (F, ,1/K) K, governed by the robust MABE dimension through dMADCrob,maxd^rob, _MADC, which captures how well empirical robust Bellman errors—computed from nominal data—control value prediction under adversarial transition dynamics. The second is a robust estimation term H(Bϕmax)2KH\,(B _φ)^2 K, which arises from approximating the worst-case Bellman operator via the dual formulation using nominal interaction data, and reflects the statistical cost of learning under distributional mismatch. The remaining term H2dMABErob,max(ℱ,Π,1/K)H^2d_MABE^rob, (F, ,1/K) is a lower-order horizon-dependent contribution inherited from the decoupling inequality, while εdualapprox ^approx_dual captures the approximation error of the dual function class. Overall, this result shows that, analogous to the nominal MAMEX framework, structural conditions implying low decoupling complexity translate directly into sample-efficient online learning. However, in the robust setting, the complexity is additionally influenced by the difficulty of estimating the robust Bellman operator from nominal data, and the divergence-dependent envelope BϕmaxB_φ . Appendix G Robust Multi-Agent Bilinear Classes We next extend the multi-agent bilinear class construction of [78] to the distributionally robust setting. In the nominal case, bilinear classes constitute a tractable family of MDP models with general function approximation, where the Bellman residual admits a bilinear factorization under arbitrary stage-wise visitation distributions. This structure enables provably sample-efficient learning in the non-robust setting. In the robust setting, we generalize this idea by requiring that the robust Bellman residual admits an analogous bilinear factorization in an appropriate Hilbert space, where expectations are taken with respect to arbitrary stage-wise visitation distributions consistent with the admissible class in Definition 10 (robust MABE). This extension preserves the key structural property that enables tractable learning, while accommodating worst-case transition dynamics. As we show, the resulting multi-agent robust bilinear class retains a small robust MADC, which is finite, ensuring that our algorithm achieves sample-efficient learning in these settings. Recall that for any agent i∈ℳi , policy π∈Πpurπ∈ ^pur, stage h∈[H]h∈[H], and fi=fi,hh=1H+1∈ℱif_i=\f_i,h\_h=1^H+1 _i, we define the robust Bellman residual class per agent at stage h as defined in (56) as Resiπ(ℱi)≔fh−i,hπ,ϕ,σifh+1:f∈ℱi,Res^π_i(F_i) \f_h-T_i,h^π,φ, _if_h+1:f _i\, (92) Resiπ(ℱi)Res_i^π(F_i) is the policy-specific residual class, and ⋃π∈ΠpurResiπ(ℱi) _π∈ ^purRes_i^π(F_i) is the class appearing in Definition 10. As in the nominal bilinear-class construction, we compare an arbitrary candidate fif_i to the robust Bellman-consistent benchmark associated with a unilateral robust best response. Fix an agent i∈ℳi and a pure joint policy π∈Πpurπ∈ ^pur. Let ~i†,σi:=(πi†,σi(π−i),π−i), π_i , _i:= ( _i , _i( _-i),\, _-i ), where πi†,σi(π−i) _i , _i( _-i) is a robust best response of agent i against π−i _-i. Since the action spaces are finite, a best response can be taken to be pure. Under Assumption 1, the associated robust Bellman-consistent benchmark sequence fi~i†,σi,σi∈ℱif_i π_i , _i, _i _i is well-defined by fi,h~i†,σi,σi=i,h~i†,σi,ϕ,σifi,h+1~i†,σi,σi,h∈[H],fi,H+1~i†,σi,σi≡0.f_i,h π_i , _i, _i=T_i,h π_i , _i,φ, _if_i,h+1 π_i , _i, _i, h∈[H], f_i,H+1 π_i , _i, _i≡ 0. (93) We can now state the robust analogue of multi-agent bilinear classes. Definition 11 (Robust Multi-Agent Bilinear Class). Fix an agent i∈ℳi . Let Πhh=1H\ _h\_h=1^H be the family of stage-wise distributions over ×S×A as in Definition 10, and let (,⟨⋅,⋅⟩)(W, ·,· _W) be a Hilbert space with norm ∥⋅∥\|·\|_W. We say that the DRMG belongs to the robust multi-agent bilinear class for agent i if, for every stage h∈[H]h∈[H], there exist mappings Xi,hrob:ℱi×Πpur→,Yi,hrob:Πh→,X_i,h^rob:F_i× ^pur , Y_i,h^rob: _h , and a constant Crob>0C_rob>0, such that the following hold. (i) Bilinear factorization of robust residual means. For every fi∈ℱif_i _i, every pure joint policy π∈Πpurπ∈ ^pur, every stage h∈[H]h∈[H], and every distribution μh∈Πh _h∈ _h, (sh,h)∼μh[Resi,hπ,σi(fi)(sh,h)]=⟨Xi,hrob(fi,π)−Xi,hrob(fi~i†,σi,σi,~i†,σi),Yi,hrob(μh)⟩.E_(s_h, a_h) _h [Res_i,h^π, _i(f_i)(s_h, a_h) ]= X_i,h^rob(f_i,π)-X_i,h^rob\! (f_i π_i , _i, _i, π_i , _i ),\,Y_i,h^rob( _h) _W. (94) (i) Regularity. For every stage h∈[H]h∈[H], supμh∈Πh‖Yi,hrob(μh)‖≤1,supfi∈ℱi,π∈Πpur‖Xi,hrob(fi,π)‖≤Crob. _ _h∈ _h\|Y_i,h^rob( _h)\|_W≤ 1, _f_i _i,\;π∈ ^pur\|X_i,h^rob(f_i,π)\|_W≤ C_rob. (95) Remark 1 (Worst-case vs. nominal visitation density). For KL and χ2χ^2-divergence uncertainty sets, every admissible worst-case kernel P∈hϕ,σ(s,a)P _h^φ,σ(s,a) is absolutely continuous with respect to the nominal kernel Ph⋆(⋅∣s,a)P_h (· s,a). Consequently, for any fixed policy π∈Πpurπ∈ ^pur, the worst-case stage-h occupancy measure νi,hπ _i,h^π is absolutely continuous with respect to the nominal occupancy measure μhπ _h^π, and there exists an absolute constant C0≥1C_0≥ 1 such that dνi,hπ(s,)≤C0dμhπ(s,),∀(s,),h∈[H].d _i,h^π(s, a)≤ C_0\,d _h^π(s, a), ∀(s, a),\ h∈[H]. (96) Discussion. Definition 11 is the exact robust counterpart of the multi-agent bilinear class in the non-robust MAMEX framework [78], with one essential modification: the nominal Bellman residual is replaced by the robust Bellman residual Resi,hπ,σi(fi)=fi,h−i,hπ,ϕ,σifi,h+1Res_i,h^π, _i(f_i)=f_i,h-T_i,h^π,φ, _if_i,h+1, and the reference fixed point is replaced by the robust Bellman-consistent benchmark (93). Expectations are taken over admissible stage-wise distributions in Πh _h, which include nominal visitation distributions used for training-error terms and worst-case occupancies used for robust value-prediction terms. A useful subtlety is worth emphasising. In the nominal setting, linearity of the one-step model often directly implies linearity of the Bellman image. In the robust setting, this is not automatic, because the support operator V↦infP∈i,h,ϕσi(s,a)s′∼P[V⁡(s′)]V _P _i,h,φ _i(s,a)E_s P[V(s )] is generally nonlinear in V. Therefore, robust linear examples must be stated either under an explicit robust Bellman completeness assumption or under an explicit robust support linearity assumption. This is precisely why the examples below are formulated in that way. To quantify the complexity of the family Yi,hrob(μh):μh∈Πh\Y_i,h^rob( _h): _h∈ _h\, we introduce the associated information gain. Definition 12 (Information gain for robust bilinear classes). For each stage h∈[H]h∈[H], define i,hrob:=Yi,hrob(μh):μh∈Πh,hrob:=⋃i∈ℳi,hrob,rob:=⋃h=1Hhrob.Y_i,h^rob:= \Y_i,h^rob( _h): _h∈ _h \, _h^rob:= _i Y_i,h^rob, ^rob:= _h=1^HY_h^rob. Suppose W is a Hilbert space such that rob⊆Y^rob . Then, for any ε>0 >0 and integer K≥1K≥ 1, define ∇hK(ε,hrob):=maxy1,…,yK∈hroblogdet(I+1ε∑k=1Kykyk⊤),∇^K_h( ,Y_h^rob):= _y_1,…,y_K _h^rob \! (I+ 1 _k=1^Ky_ky_k ), and ∇K(ε,rob):=∑h=1H∇hK(ε,hrob).∇^K( ,Y^rob):= _h=1^H∇^K_h( ,Y_h^rob). (97) The next theorem is the robust counterpart of Theorem 5.8 in [78]. Theorem 4 (Robust Multi-Agent Bilinear Classes ⊆ Low Robust MADC). Fix an agent i∈ℳi , and suppose the DRMG belongs to the robust multi-agent bilinear class of Definition 11. Assume that the robust-to-nominal feature coverage condition holds with constant CcovC_cov. Let di,MADCrobd_i,MADC^rob denote the agent-wise robust MADC from Definition 6. Let γ∈(0,H]γ∈(0,H] be a uniform bound on the robust value functions. Then, for every K≥2K≥ 2, dMADCrob,max=maxi∈ℳ⁡di,MADCrob=⁡(max⁡1,(1+γ2)Ccov∇K(1KCrob2,rob)).d_MADC^rob, = _i d_i,MADC^rob=O\! ( \1,\;(1+γ^2)\,C_cov\,∇^K\! ( 1KC_rob^2,Y^rob ) \ ). Proof. We prove the result for a fixed agent i∈ℳi . Throughout the proof, we suppress the agent index i whenever there is no ambiguity. The proof follows the same general structure as [78, Theorem 5.8], but with two robust modifications. First, the nominal Bellman residual is replaced by the robust Bellman residual induced by the robust Bellman operator hπ,ϕ,σiT_h^π,φ, _i. Second, the diagonal prediction terms are evaluated under worst-case occupancies, whereas the historical training terms are evaluated under nominal visitation distributions. This distinction is essential because robust MADC relates robust value prediction under adversarial dynamics to Bellman-error information gathered from nominal data. Define robust residual means. For each episode k∈[K]k∈[K] and stage h∈[H]h∈[H], define the stage-h robust Bellman residual ψhk(s,):=Reshπk,σi(fk)(s,)=fhk(s,)−[hπk,ϕ,σifh+1k](s,). _h^k(s, a):=Res_h^π^k, _i(f^k)(s, a)=f_h^k(s, a)- [T_h^π^k,φ, _if_h+1^k ](s, a). (98) By Assumption 1, hπk,ϕ,σifh+1k∈ℱh,T_h^π^k,φ, _if_h+1^k _h, and hence ψhk _h^k belongs to the robust Bellman residual class at stage h. Let νhk:=νi,hπk _h^k:= _i,h^π^k denote the stage-h worst-case state–joint-action occupancy induced by policy πkπ^k and the robust Bellman recursion for agent i. Let μhk:=μhπk _h^k:= _h^π^k denote the stage-h nominal visitation distribution induced by executing πkπ^k in the nominal environment P⋆P . Define the episode-k robust value-prediction error Δk:=V1πk,fk(s1k)−V1πk,σi(s1k). _k:=V_1^π^k,f^k(s_1^k)-V_1^π^k, _i(s_1^k). (99) By robust Bellman telescoping along the worst-case occupancy sequence νhkh=1H\ _h^k\_h=1^H, we have Δk≤∑h=1H|(sh,h)∼νhk[ψhk(sh,h)]|. _k≤ _h=1^H |E_(s_h, a_h) _h^k [ _h^k(s_h, a_h) ] |. (100) Now apply the robust bilinear factorization with f=fkf=f^k, π=πkπ=π^k, and μh=νhk _h= _h^k. Since the robust bilinear class is defined for the admissible stage-wise distribution family Πh _h, and νhk∈Πh _h^k∈ _h, this yields νhk[ψhk]=⟨θhk,zhk⟩,E_ _h^k[ _h^k]= _h^k,\,z_h^k _W, (101) where θhk _h^k :=Xhrob(fk,πk)−Xhrob(f~†,σi(πk),σi,~†,σi(πk)), :=X_h^rob(f^k,π^k)-X_h^rob (f π , _i(π^k), _i, π , _i(π^k) ), (102) zhk z_h^k :=Yhrob(νhk). :=Y_h^rob( _h^k). (103) For historical nominal data distributions, define similarly yhs:=Yhrob(μhs),s<k.y_h^s:=Y_h^rob( _h^s), s<k. (104) By the regularity condition of the robust bilinear class, ‖θhk‖≤2Crob,‖zhk‖≤1,‖yhs‖≤1.\| _h^k\|_W≤ 2C_rob, \|z_h^k\|_W≤ 1, \|y_h^s\|_W≤ 1. (105) Combining (100) and (101), we obtain Δk≤∑h=1H|⟨θhk,zhk⟩|. _k≤ _h=1^H | _h^k,z_h^k _W |. (106) Since both V1πk,fk(s1k)V_1^π^k,f^k(s_1^k) and V1πk,σi(s1k)V_1^π^k, _i(s_1^k) are bounded by γ, we also have Δk≤2γ. _k≤ 2γ. Therefore, Δk≤∑h=1Hmin⁡|⟨θhk,zhk⟩|,2γ. _k≤ _h=1^H \ | _h^k,z_h^k _W |,2γ \. (107) Introduce Gram matrices and decompose the sum. Fix ε>0 >0, to be chosen later. For each stage h and episode k, define the regularized Gram matrix using the historical nominal feature directions: Σhk:=εI+∑s=1k−1yhs(yhs)⊤. _h^k:= I+ _s=1^k-1y_h^s(y_h^s) . (108) Then Σhk⪰εI _h^k I, and the self-normalized Cauchy–Schwarz inequality gives |⟨θhk,zhk⟩|≤‖θhk‖Σhk‖zhk‖(Σhk)−1. | _h^k,z_h^k _W |≤\| _h^k\|_ _h^k\,\|z_h^k\|_( _h^k)^-1. (109) Using (107), decompose 1=∥zhk∥(Σhk)−1≤1+∥zhk∥(Σhk)−1>1.1=1\\|z_h^k\|_( _h^k)^-1≤ 1\+1\\|z_h^k\|_( _h^k)^-1>1\. Then ∑k=1KΔk≤A+B, _k=1^K _k≤ A+B, (110) where A A :=∑k=1K∑h=1Hmin|⟨θhk,zhk⟩|,2γ∥zhk∥(Σhk)−1≤1, := _k=1^K _h=1^H \ | _h^k,z_h^k _W |,2γ \1\\|z_h^k\|_( _h^k)^-1≤ 1\, (111) B B :=∑k=1K∑h=1Hmin|⟨θhk,zhk⟩|,2γ∥zhk∥(Σhk)−1>1. := _k=1^K _h=1^H \ | _h^k,z_h^k _W |,2γ \1\\|z_h^k\|_( _h^k)^-1>1\. (112) Step 3: Bound the term A. On the event ∥zhk∥(Σhk)−1≤1\\|z_h^k\|_( _h^k)^-1≤ 1\, combining (109) with min⁡|x|,2γ≤2γ|x| \|x|,2γ\≤ 2γ|x| for |x|≤1|x|≤ 1, we obtain A A ≤2γ∑k=1K∑h=1H‖θhk‖Σhkmin⁡‖zhk‖(Σhk)−1,1. ≤ 2γ _k=1^K _h=1^H\| _h^k\|_ _h^k \\|z_h^k\|_( _h^k)^-1,1\. (113) Expanding the weighted norm gives ‖θhk‖Σhk \| _h^k\|_ _h^k =[ε‖θhk‖2+∑s=1k−1⟨θhk,yhs⟩2]1/2≤2εCrob+[∑s=1k−1⟨θhk,yhs⟩2]1/2, = [ \| _h^k\|_W^2+ _s=1^k-1 _h^k,y_h^s _W^2 ]^1/2≤ 2 \,C_rob+ [ _s=1^k-1 _h^k,y_h^s _W^2 ]^1/2, (114) where we used (105) and a+b≤a+b a+b≤ a+ b. Substituting (114) into (113), we get A≤A1+A2,A≤ A_1+A_2, where A1 A_1 :=4γεCrob∑k=1K∑h=1Hmin⁡‖zhk‖(Σhk)−1,1, :=4γ \,C_rob _k=1^K _h=1^H \\|z_h^k\|_( _h^k)^-1,1\, (115) A2 A_2 :=2γ∑k=1K∑h=1H[∑s=1k−1⟨θhk,yhs⟩2]1/2min⁡‖zhk‖(Σhk)−1,1. :=2γ _k=1^K _h=1^H [ _s=1^k-1 _h^k,y_h^s _W^2 ]^1/2 \\|z_h^k\|_( _h^k)^-1,1\. (116) • Bound A1A_1. By Cauchy–Schwarz, A1 A_1 ≤4γεCrobKH(∑k=1K∑h=1Hmin⁡‖zhk‖(Σhk)−12,1)1/2. ≤ 4γ \,C_rob KH ( _k=1^K _h=1^H \\|z_h^k\|_( _h^k)^-1^2,1\ )^1/2. (117) At this point, because zhk=Yhrob(νhk)z_h^k=Y_h^rob( _h^k) is a worst-case feature direction while Σhk _h^k is built from nominal directions yhs=Yhrob(μhs)y_h^s=Y_h^rob( _h^s), we require the robust-to-nominal feature coverage condition: ∑k=1K∑h=1Hmin⁡‖zhk‖(Σhk)−12,1≤Ccov∇K(ε,rob), _k=1^K _h=1^H \\|z_h^k\|_( _h^k)^-1^2,1\≤ C_cov\,∇^K( ,Y^rob), (118) where Ccov≥1C_cov≥ 1 is the robust-to-nominal feature coverage constant assumed in the statement of the theorem. A sufficient condition under which this constant is finite is given in Remark 1 (for KL and χ2χ^2-divergence uncertainty sets, where one may take Ccov=C0C_cov=C_0 with C0C_0 the absolute constant of (96)); the corresponding sufficient condition for TV-divergence uncertainty sets, via Definition 7, is given in Remark 2 below. This is the bilinear-feature analogue of the robust-to-nominal concentrability condition: it controls the worst-case directions by the nominal data directions accumulated in the Gram matrix. Using (118) in (117), we obtain A1 A_1 ≤4γεKHCrob2Ccov∇K(ε,rob). ≤ 4γ KHC_rob^2C_cov\,∇^K( ,Y^rob). (119) Applying Young’s inequality gives A1≤2εKHCrob2+8γ2Ccov∇K(ε,rob).A_1≤ 2 KHC_rob^2+8γ^2C_cov\,∇^K( ,Y^rob). (120) • Bound A2A_2. Applying Cauchy–Schwarz, A2 A_2 ≤2γ(∑k=1K∑h=1H∑s=1k−1⟨θhk,yhs⟩2)1/2(∑k=1K∑h=1Hmin⁡‖zhk‖(Σhk)−12,1)1/2. ≤ 2γ ( _k=1^K _h=1^H _s=1^k-1 _h^k,y_h^s _W^2 )^1/2 ( _k=1^K _h=1^H \\|z_h^k\|_( _h^k)^-1^2,1\ )^1/2. (121) By the robust bilinear factorization applied to the nominal distribution μhs _h^s, ⟨θhk,yhs⟩=(sh,h)∼μhs[ψhk(sh,h)]. _h^k,y_h^s _W=E_(s_h, a_h) _h^s[ _h^k(s_h, a_h)]. Therefore, by Jensen’s inequality, ⟨θhk,yhs⟩2 _h^k,y_h^s _W^2 ≤(sh,h)∼μhs[(ψhk(sh,h))2]. _(s_h, a_h) _h^s[( _h^k(s_h, a_h))^2]. (122) Summing over h and s<ks<k gives ∑h=1H∑s=1k−1⟨θhk,yhs⟩2≤∑s=1k−1eis(fik,πk), _h=1^H _s=1^k-1 _h^k,y_h^s _W^2≤ _s=1^k-1e_i^s(f_i^k,π^k), (123) where eis(fik,πk)e_i^s(f_i^k,π^k) is exactly the robust Bellman training error appearing in Definition 6. Combining (121), (118), and (123), we obtain A2 A_2 ≤2γCcov∇K(ε,rob)∑k=1K∑s=1k−1eis(fik,πk). ≤ 2γ C_cov\,∇^K( ,Y^rob) _k=1^K _s=1^k-1e_i^s(f_i^k,π^k). (124) Applying Young’s inequality in the form 2ab≤1μa+μb,2 ab≤ 1μa+μ b, with a=∑k=1K∑s=1k−1eis(fik,πk),b=γ2Ccov∇K(ε,rob),a= _k=1^K _s=1^k-1e_i^s(f_i^k,π^k), b=γ^2C_cov\,∇^K( ,Y^rob), yields A2≤1μ∑k=1K∑s=1k−1eis(fik,πk)+μγ2Ccov∇K(ε,rob).A_2≤ 1μ _k=1^K _s=1^k-1e_i^s(f_i^k,π^k)+μγ^2C_cov\,∇^K( ,Y^rob). (125) Step 4: Bound the term B. By the definition of B, B≤2γ∑k=1K∑h=1H∥zhk∥(Σhk)−1>1.B≤ 2γ _k=1^K _h=1^H1\\|z_h^k\|_( _h^k)^-1>1\. Since x>1≤min1,x21\x>1\≤ \1,x^2\ for x≥0x≥ 0, B B ≤2γ∑k=1K∑h=1Hmin⁡1,‖zhk‖(Σhk)−12 ≤ 2γ _k=1^K _h=1^H \1,\|z_h^k\|_( _h^k)^-1^2\ ≤2γCcov∇K(ε,rob), ≤ 2γ C_cov\,∇^K( ,Y^rob), (126) where the last step again uses (118). Step 5: Combine the bounds and choose ε . Combining (110), (120), (125), and (126), we obtain ∑k=1KΔk _k=1^K _k ≤1μ∑k=1K∑s=1k−1eis(fik,πk)+2εKHCrob2 ≤ 1μ _k=1^K _s=1^k-1e_i^s(f_i^k,π^k)+2 KHC_rob^2 +(8+μ)γ2Ccov∇K(ε,rob)+2γCcov∇K(ε,rob). +(8+μ)γ^2C_cov\,∇^K( ,Y^rob)+2γ C_cov\,∇^K( ,Y^rob). (127) Choose ε=1KCrob2. = 1KC_rob^2. Then 2εKHCrob2=2H.2 KHC_rob^2=2H. Thus, ∑k=1KΔk _k=1^K _k ≤1μ∑k=1K∑s=1k−1eis(fik,πk)+2H+(8+μ)γ2Ccov∇K(1KCrob2,rob) ≤ 1μ _k=1^K _s=1^k-1e_i^s(f_i^k,π^k)+2H+(8+μ)γ^2C_cov\,∇^K\! ( 1KC_rob^2,Y^rob ) +2γCcov∇K(1KCrob2,rob). +2γ C_cov\,∇^K\! ( 1KC_rob^2,Y^rob ). (128) Define di,MADCrob:=max⁡1,c0(1+γ2)Ccov∇K(1KCrob2,rob),d_i,MADC^rob:= \1,\;c_0(1+γ^2)C_cov\,∇^K\! ( 1KC_rob^2,Y^rob ) \, for a sufficiently large universal constant c0>0c_0>0. Then the terms (8+μ)γ2Ccov∇K(⋅),2γCcov∇K(⋅),2H(8+μ)γ^2C_cov∇^K(·), 2γ C_cov∇^K(·), 2H are absorbed into μdi,MADCrob+cHdi,MADCrob,μ d_i,MADC^rob+cHd_i,MADC^rob, for a sufficiently large universal constant c>0c>0. Consequently, ∑k=1KΔk≤1μ∑k=1K∑s=1k−1eis(fik,πk)+μdi,MADCrob+cHdi,MADCrob. _k=1^K _k≤ 1μ _k=1^K _s=1^k-1e_i^s(f_i^k,π^k)+μ d_i,MADC^rob+cHd_i,MADC^rob. This is exactly the decoupling inequality in Definition 6. Hence, di,MADCrob≤⁡(max⁡1,(1+γ2)Ccov∇K(1KCrob2,rob)).d_i,MADC^rob \! ( \1,\;(1+γ^2)C_cov\,∇^K\! ( 1KC_rob^2,Y^rob ) \ ). Taking the maximum over agents gives dMADCrob,max=maxi∈ℳ⁡di,MADCrob≤⁡(max⁡1,(1+γ2)Ccov∇K(1KCrob2,rob)).d_MADC^rob, = _i d_i,MADC^rob \! ( \1,\;(1+γ^2)C_cov\,∇^K\! ( 1KC_rob^2,Y^rob ) \ ). This proves the theorem. ∎ We now introduce concrete instances of the multi-agent robust bilinear classes, focusing on general-sum DRMGs with linear function approximation. In the single-agent robust RL setting [26], linear Bellman-complete models assume that the robust Bellman operator preserves linear structure with respect to a given feature representation. Extending this idea to the multi-agent setting, we consider DRMGs in which the robust Bellman operator satisfies a stagewise linear completeness condition, so that the class of linear Q-functions for each agent remains closed under robust Bellman updates. This yields a natural and tractable subclass of robust bilinear models, and serves as a canonical example illustrating how Definition 11 can be verified in structured multi-agent robust settings. Example: Distributionally robust linear Bellman-complete DRMGs. We first introduce a general linear robust class and then show that a robust zero-sum linear Markov game is a concrete special case of this class under an additional structural condition on the worst-case support operator. Part I: Distributionally robust linear Bellman-complete DRMGs. We say that a DRMG is a distributionally robust linear Bellman-complete DRMG of dimension dlind_lin if, for every stage h∈[H]h∈[H], there exists a known feature map ϕh:×→ℝdlin,‖ϕh(s,)‖2≤1,∀(s,)∈×, _h:S×A ^d_lin, \| _h(s, a)\|_2≤ 1, ∀(s, a) ×A, such that, for every agent i∈ℳi , the stagewise function class satisfies ℱi,h⊆(s,)↦⟨θ,ϕh(s,)⟩:θ∈ℝdlin,∥θ∥2≤Cθ,F_i,h \(s, a) θ, _h(s, a) :θ ^d_lin,\ \|θ\|_2≤ C_θ \, for some constant Cθ>0C_θ>0, and Assumption 1 holds, i.e., i,hπ,ϕ,σifi,h+1∈ℱi,h,∀fi,h+1∈ℱi,h+1,∀π∈Πpur.T_i,h^π,φ, _if_i,h+1 _i,h, ∀ f_i,h+1 _i,h+1, ∀π∈ ^pur. This is the robust analogue of the linear Bellman-complete class in the nominal setting: the nominal Bellman image is replaced by the robust Bellman image under i,hπ,ϕ,σiT_i,h^π,φ, _i. • Step 1: Verify the bilinear factorization in Definition 11. Fix an agent i∈ℳi , a stage h∈[H]h∈[H], a pure joint policy π∈Πpurπ∈ ^pur, and a hypothesis fi∈ℱif_i _i. Since fi,h∈ℱi,hf_i,h _i,h, there exists a coefficient vector θi,hf∈ℝdlin _i,h^f ^d_lin such that fi,h(s,)=⟨θi,hf,ϕh(s,)⟩,‖θi,hf‖2≤Cθ.f_i,h(s, a)= _i,h^f, _h(s, a) , \| _i,h^f\|_2≤ C_θ. Moreover, by Assumption 1, the robust Bellman image of fi,h+1f_i,h+1 also belongs to ℱi,hF_i,h. Hence there exists another vector wi,hf,π,σi∈ℝdlinw_i,h^f,π, _i ^d_lin such that [i,hπ,ϕ,σifi,h+1](s,)=⟨wi,hf,π,σi,ϕh(s,)⟩. [T_i,h^π,φ, _if_i,h+1 ](s, a)= w_i,h^f,π, _i, _h(s, a) . Therefore, the robust Bellman residual remains linear in the same feature map: Resi,hπ,σi(fi)(s,) _i,h^π, _i(f_i)(s, a) =fi,h(s,)−[i,hπ,ϕ,σifi,h+1](s,) =f_i,h(s, a)- [T_i,h^π,φ, _if_i,h+1 ](s, a) =⟨θi,hf−wi,hf,π,σi,ϕh(s,)⟩. = _i,h^f-w_i,h^f,π, _i, _h(s, a) . (129) Now let μh∈Πh _h∈ _h be any admissible stagewise visitation distribution in the sense of Definition 11. In particular, μh _h may correspond either to a nominal visitation distribution or to a worst-case occupancy induced by the robust Bellman recursion. Taking expectation under μh _h in (129), we obtain (sh,h)∼μh[Resi,hπ,σi(fi)(sh,h)] _(s_h, a_h) _h [Res_i,h^π, _i(f_i)(s_h, a_h) ] =⟨θi,hf−wi,hf,π,σi,μh[ϕh(sh,h)]⟩. = _i,h^f-w_i,h^f,π, _i,\;E_ _h[ _h(s_h, a_h)] . (130) Thus, the bilinear factorization in Definition 11 holds by taking =ℝdlin,Yi,hrob(μh)=μh[ϕh(sh,h)],W=R^d_lin, Y_i,h^rob( _h)=E_ _h[ _h(s_h, a_h)], and using the centered feature coefficient Xi,hrob(fi,π)−Xi,hrob(fi⋆,π,σi,π)=θi,hf−wi,hf,π,σi.X_i,h^rob(f_i,π)-X_i,h^rob (f_i ,π, _i,π )= _i,h^f-w_i,h^f,π, _i. Equivalently, the Bellman-image coefficient wi,hf,π,σiw_i,h^f,π, _i can be absorbed into the centered X-map in Definition 11. • Step 2: Verify the regularity condition. First, since ‖ϕh(s,)‖2≤1\| _h(s, a)\|_2≤ 1 for all (s,)(s, a), Jensen’s inequality gives ‖Yi,hrob(μh)‖2=‖μh[ϕh(sh,h)]‖2≤μh‖ϕh(sh,h)‖2≤1.\|Y_i,h^rob( _h)\|_2= \|E_ _h[ _h(s_h, a_h)] \|_2 _ _h\| _h(s_h, a_h)\|_2≤ 1. This bound holds uniformly over both nominal visitation distributions and worst-case occupancies. Second, because both fi,hf_i,h and i,hπ,ϕ,σifi,h+1T_i,h^π,φ, _if_i,h+1 belong to the same linear class, ‖θi,hf−wi,hf,π,σi‖2≤‖θi,hf‖2+‖wi,hf,π,σi‖2≤2Cθ.\| _i,h^f-w_i,h^f,π, _i\|_2≤\| _i,h^f\|_2+\|w_i,h^f,π, _i\|_2≤ 2 C_θ. Hence one may take Crob=2Cθ.C_rob=2 C_θ. • Step 3: Consequence for robust information gain and robust MADC. Therefore, distributionally robust linear Bellman-complete DRMGs belong to the robust multi-agent bilinear class of Definition 11. Since the feature dimension is dlind_lin and the features are uniformly bounded, the associated robust bilinear information gain satisfies ∇K(1KCrob2,rob)=~(Hdlin),∇^K\! ( 1KC_rob^2,Y^rob )= O(Hd_lin), where ~ O hides only absolute constants and logarithmic factors. To translate this information-gain bound into a robust MADC bound, we also use the robust-to-nominal feature coverage condition from Theorem 4. This condition accounts for the mismatch between worst-case feature directions Yi,hrob(νi,hπ)Y_i,h^rob( _i,h^π) and the nominal historical feature directions Yi,hrob(μhπ)Y_i,h^rob( _h^π) accumulated from interaction with P⋆P . Consequently, by Theorem 4, dMADCrob,max=~((1+γ2)C0Hdlin).d_MADC^rob, = O\! ((1+γ^2)\,C_0\,Hd_lin ). When γ≥1γ≥ 1, this simplifies to dMADCrob,max=~(γ2C0Hdlin).d_MADC^rob, = O\! (γ^2C_0Hd_lin ). • Step 4: Regret implication. Combining the above with the regret bound in Theorem 1, it follows that RoMEX-ϕφ achieves ~(γ2C0HdlinK+γ2C0H2dlin+C0H(Bϕmax)2K+εdualapprox) O\! (γ^2C_0Hd_lin K+γ^2C_0H^2d_lin+C_0H(B _φ)^2 K+ ^approx_dual ) robust equilibrium regret in this class, up to the logarithmic factors already present in Theorem 1. Thus, robust linear Bellman-complete DRMGs provide a concrete structured family in which the robust bilinear information gain is finite and the robust MADC is finite under robust-to-nominal feature coverage. The resulting regret scales polynomially in the horizon and linearly in the feature dimension, up to logarithmic factors, while explicitly reflecting the distribution mismatch through C0C_0. Part I: Robust zero-sum linear Markov games as a special case. We now specialize the above construction to the two-player zero-sum setting motivated by [78, Example 5.10]. Consider a two-player zero-sum DRMG with joint action (a,b)∈×ℬ(a,b) ×B. Suppose that, for each stage h∈[H]h∈[H], there exists a known feature map φh:×ℬ→ℝdlin,‖φh(s,a,b)‖2≤1, _h:S×A×B ^d_lin, \| _h(s,a,b)\|_2≤ 1, and a reward parameter θh∈ℝdlin _h ^d_lin such that rh(s,a,b)=⟨θh,φh(s,a,b)⟩,‖θh‖2≤Cθ.r_h(s,a,b)= _h, _h(s,a,b) , \| _h\|_2≤ C_θ. Assume also that the nominal transition kernel is linear in the same feature map: Ph⋆(⋅∣s,a,b)=⟨φh(s,a,b),μh(⋅)⟩,P_h (· s,a,b)= _h(s,a,b), _h(·) , for a vector of signed measures μh=μh,jj=1dlin _h=\ _h,j\_j=1^d_lin. In the nominal setting, such linearity is sufficient to imply that the Bellman image remains linear. In the robust setting, however, this implication is no longer automatic, because the robust support operator V↦infP∈i,h,ϕσi(s,a,b)s′∼P[V⁡(s′)]V _P _i,h,φ _i(s,a,b)E_s P[V(s )] is generally nonlinear in V. Consequently, nominal linearity of Ph⋆P_h alone is insufficient for verifying Definition 11. To recover a bilinear structure, we impose an additional robust support linearity assumption: for each player i∈1,2i∈\1,2\, each stage h∈[H]h∈[H], and every bounded continuation value V:→ℝV:S , there exists a vector βi,hσi(V)∈ℝdlin _i,h _i(V) ^d_lin such that infP∈i,h,ϕσi(s,a,b)s′∼P[V⁡(s′)]=⟨βi,hσi(V),φh(s,a,b)⟩,∀(s,a,b)∈×ℬ. _P _i,h,φ _i(s,a,b)E_s P[V(s )]= _i,h _i(V), _h(s,a,b) , ∀(s,a,b) ×A×B. (131) • Step 5: Verify that robust zero-sum linear MGs fit the bilinear framework. Fix an agent i∈1,2i∈\1,2\, a stage h∈[H]h∈[H], a pure joint policy π∈Πpurπ∈ ^pur, and a hypothesis fi∈ℱif_i _i. Suppose fi,h(s,a,b)=⟨ϑi,hf,φh(s,a,b)⟩,‖ϑi,hf‖2≤Cθ.f_i,h(s,a,b)= _i,h^f, _h(s,a,b) , \| _i,h^f\|_2≤ C_θ. Using the continuation value Vi,h+1π,fi(s′)=(a′,b′)∼πh(⋅∣s′)[fi,h+1(s′,a′,b′)],V_i,h+1^π,f_i(s )=E_(a ,b ) _h(· s )[f_i,h+1(s ,a ,b )], and applying (131), we obtain [i,hπ,ϕ,σifi,h+1](s,a,b) [T_i,h^π,φ, _if_i,h+1 ](s,a,b) =rh(s,a,b)+infP∈i,h,ϕσi(s,a,b)s′∼P[Vi,h+1π,fi(s′)] =r_h(s,a,b)+ _P _i,h,φ _i(s,a,b)E_s P[V_i,h+1^π,f_i(s )] =⟨θh,φh(s,a,b)⟩+⟨βi,hσi(Vi,h+1π,fi),φh(s,a,b)⟩ = _h, _h(s,a,b) + _i,h _i(V_i,h+1^π,f_i), _h(s,a,b) =⟨θh+βi,hσi(Vi,h+1π,fi),φh(s,a,b)⟩. = _h+ _i,h _i(V_i,h+1^π,f_i), _h(s,a,b) . (132) Hence the robust Bellman image is again linear in the same feature map. Therefore, the robust Bellman residual becomes Resi,hπ,σi(fi)(s,a,b) _i,h^π, _i(f_i)(s,a,b) =fi,h(s,a,b)−[i,hπ,ϕ,σifi,h+1](s,a,b) =f_i,h(s,a,b)- [T_i,h^π,φ, _if_i,h+1 ](s,a,b) =⟨ϑi,hf−θh−βi,hσi(Vi,h+1π,fi),φh(s,a,b)⟩. = _i,h^f- _h- _i,h _i(V_i,h+1^π,f_i), _h(s,a,b) . (133) Now let μh∈Πh _h∈ _h be any admissible stagewise visitation distribution (including both nominal visitation distributions and worst-case occupancies). Taking expectation under μh _h in (133), we obtain (sh,ah,bh)∼μh[Resi,hπ,σi(fi)(sh,ah,bh)] _(s_h,a_h,b_h) _h [Res_i,h^π, _i(f_i)(s_h,a_h,b_h) ] =⟨ϑi,hf−θh−βi,hσi(Vi,h+1π,fi),μh[φh(sh,ah,bh)]⟩. = _i,h^f- _h- _i,h _i(V_i,h+1^π,f_i),\;E_ _h[ _h(s_h,a_h,b_h)] . (134) Thus, robust zero-sum linear MGs belong to the robust multi-agent bilinear class with =ℝdlin,Yi,hrob(μh)=μh[φh(sh,ah,bh)],W=R^d_lin, Y_i,h^rob( _h)=E_ _h[ _h(s_h,a_h,b_h)], and the centered X-map defined through the coefficient difference, consistent with Definition 11. • Step 6: Verify regularity and derive robust MADC. The bound ‖Yi,hrob(μh)‖2≤1\|Y_i,h^rob( _h)\|_2≤ 1 follows exactly as in Part I by Jensen’s inequality and holds uniformly over both nominal and worst-case distributions. To control the coefficient vector in (133), assume that the support-linearity coefficient is uniformly bounded over bounded continuation values, namely suph∈[H]sup‖V‖∞≤γ‖βi,hσi(V)‖2≤Bβ. _h∈[H] _\|V\|_∞≤γ\| _i,h _i(V)\|_2≤ B_β. Then ‖ϑi,hf−θh−βi,hσi(Vi,h+1π,fi)‖2≤2Cθ+Bβ, \| _i,h^f- _h- _i,h _i(V_i,h+1^π,f_i) \|_2≤ 2 C_θ+B_β, so that one may take Crob=2Cθ+Bβ.C_rob=2 C_θ+B_β. Since the feature dimension remains dlind_lin, the associated robust information gain satisfies ∇K(1KCrob2,rob)=~(Hdlin).∇^K\! ( 1KC_rob^2,Y^rob )= O(Hd_lin). However, translating this into a robust MADC bound requires controlling the mismatch between worst-case feature directions and nominal historical feature directions. Under the robust-to-nominal feature coverage condition of Theorem 4, we obtain dMADCrob,max=~(γ2C0Hdlin).d_MADC^rob, = O\! (γ^2\,C_0\,Hd_lin ). • Step 7: Regret implication for the zero-sum robust linear subclass. Combining the above with Theorem 1, we conclude that RoMEX-ϕφ achieves ~(γ2C0HdlinK+γ2C0H2dlin+C0H(Bϕmax)2K+εdualapprox) O\! (γ^2C_0Hd_lin K+γ^2C_0H^2d_lin+C_0H(B _φ)^2 K+ ^approx_dual ) (135) robust equilibrium regret in this subclass, up to logarithmic factors. In particular, this provides a concrete structured two-player zero-sum robust class in which the robust MADC is finite and the regret remains sublinear in K, with an explicit dependence on the distribution mismatch through C0C_0. Remark 2 (Bilinear Regret for TV). For TV-case, we apply Definition 7 to handle the support-shift issue and we apply Ccov≥1C_cov≥ 1 in (135) instead of C0C_0 to get the revised regret bound as ~(γ2CcovHdlinK+γ2CcovH2dlin+CcovH(Bϕmax)2K+εdualapprox) O\! (γ^2C_covHd_lin K+γ^2C_covH^2d_lin+C_covH(B _φ)^2 K+ ^approx_dual ). This merged example highlights the key structural distinction between the nominal and robust settings. In the nominal case, linearity of the transition model directly implies linearity of the Bellman operator and hence immediately yields a bilinear factorization. In the robust case, this implication fails because the worst-case support operator is nonlinear in the continuation value. The general robust linear Bellman-complete DRMG class addresses this by assuming closure of the robust Bellman image, while the robust zero-sum linear MG specialization makes explicit the additional support-linearity condition under which this closure can be verified constructively. In both cases, the resulting information gain scales as ~(Hdlin) O(Hd_lin). However, in the robust setting this alone is not sufficient to control value prediction, due to the mismatch between nominal training distributions and worst-case occupancies. When combined with the robust-to-nominal feature coverage condition, this yields a bounded robust MADC of order ~(γ2Hdlin) O(γ^2Hd_lin). Consequently, by Theorem 1, this structure ensures sublinear robust equilibrium regret in bilinear classes. Appendix H Auxiliary Lemmas We first prove several auxiliary lemmas. Lemma 3 (Unified surrogate deviation inequality). Fix an episode k∈[K]k∈[K], and let ^k ^k be the surrogate normal-form game induced by the regularized robust payoffs U^iki∈ℳ\ U_i^k\_i defined in (26). Let πkπ^k be the output of the equilibrium oracle for a target equilibrium notion ∈NASH,CCE,CE Eq∈\NASH,CCE,CE\. For each agent i∈ℳi , define the corresponding surrogate unilateral deviation policy ~i†,k, π ,k, Eq_i as defined in (28). Then, for every agent i∈ℳi , the surrogate payoff satisfies πk[U^ik]≥~i†,k,[U^ik].E_π^k\! [ U_i^k ] _ π ,k, Eq_i\! [ U_i^k ]. Proof. Fix k∈[K]k∈[K] and i∈ℳi . By construction, ^k ^k is a normal-form game whose pure strategies are elements of Πpur ^pur, and whose payoff to agent i at pure profile π is U^ik(π) U_i^k(π) as defined in (10). Depending on the target equilibrium notion, the oracle returns either (i) a product mixed strategy over Πipuri∈ℳ\ _i^pur\_i in the NASH case, or (i) a correlated distribution over Πpur ^pur in the CCE/CE cases. Since πkπ^k is returned by the Equilibrium oracle applied to the surrogate normal-form game ^k ^k under the target equilibrium notion =NASH/CCE/CE Eq=\NASH/CCE/CE\, it satisfies the unilateral deviation property: for any agent i∈ℳi and any mixed strategy πi′∈Δ⁡(Πipur)π _i∈ ( _i^pur), πk[U^ik]≥πi′×π−ik[U^ik].E_π^k\! [ U_i^k ]\;≥\;E_π _i× _-i^k\! [ U_i^k ]. (136) • Case 1: = Eq=NASH. Since πkπ^k is a Nash equilibrium of ^k ^k, no agent can improve its surrogate payoff by unilateral deviation. Hence, πk[U^ik]≥πi′×π−ik[U^ik]∀πi′.E_π^k\! [ U_i^k ] _ _i × _-i^k\! [ U_i^k ] ∀ _i . Taking πi′=πi,NASH†,k(π−ik) _i = _i,NASH ,k( _-i^k) gives the result. • Case 2: = Eq=CCE. Since πkπ^k is a coarse correlated equilibrium of ^k ^k, no agent can improve its surrogate payoff by deviating to any fixed policy chosen independently of the recommendation. Therefore, πk[U^ik]≥πi′×π−ik[U^ik]∀πi′.E_π^k\! [ U_i^k ] _ _i × _-i^k\! [ U_i^k ] ∀ _i . Taking πi′=πi,CCE†,k(π−ik) _i = _i,CCE ,k( _-i^k) yields the result. Here π−ik _-i^k denotes the marginal distribution over the opponents’ pure strategies induced by the correlated distribution πkπ^k in the surrogate game. • Case 3: = Eq=CE. Since πkπ^k is a correlated equilibrium of Γ^k ^k, no agent can improve its surrogate payoff by applying any strategy modification ψi∈Ψ^i _i∈ _i to its recommendation. Hence, πk[U^ik]≥ψi⋄πk[U^ik],∀ψi∈Ψ^i,E_π^k[ U_i^k] _ _i π^k[ U_i^k], ∀ _i∈ _i, where we define Ψ^i:=ψi:Πipur→Πipur _i:=\ _i: ^pur_i→ ^pur_i\. Taking ψi=ψi†,k _i= _i ,k gives the result. Combining the three cases proves the lemma. This argument follows the same outer normal-form equilibrium step used in [78, Appendix A.2]; see, e.g., Eqs. (A.13)–(A.14) for an analogous deviation inequality. ∎ Lemma 4 (Robust payoff domination by plug-in surrogate value). Fix an episode k, a pure joint policy π∈Πpurπ∈ ^pur, and an agent i∈ℳi . Let fi,hk,πh=1H+1\f_i,h^k,π\_h=1^H+1 be the backward fitted sequence constructed by the algorithm, with fi,H+1k,π≡0f_i,H+1^k,π≡ 0, and let g^i,hk,πh=1H\ g_i,h^k,π\_h=1^H be the corresponding empirical dual minimizers. Let νi,hπ _i,h^π denote the stage-h state–joint-action occupancy induced by policy π and the worst-case transition kernels attaining the robust Bellman recursion for agent i. Then, for every episode k, pure joint policy π, and agent i, Vi,1π,σi(s1k)≤ V_i,1^π, _i(s_1^k)≤\; U^ik(π)+β^ik−1(fik,π,g^ik,π,π)+Δi,kop(π)+∑h=1Hνi,hπ[(g^i,hk,π,i,hπ,ϕ,σifi,h+1k,π−fi,hk,π)+], U_i^k(π)+β K_i^k-1\! (f_i^k,π; g_i^k,π;π )+ _i,k^op(π)+ _h=1^HE_ _i,h^π [ (T_ g_i,h^k,π,i,h^π,φ, _if_i,h+1^k,π-f_i,h^k,π )_+ ], where the robust-occupancy operator residual is defined as Δi,kop(π):=∑h=1H‖i,hπ,ϕ,σifi,h+1k,π−g^i,hk,π,i,hπ,ϕ,σifi,h+1k,π‖1,νi,hπ. _i,k^op(π):= _h=1^H \|T_i,h^π,φ, _if_i,h+1^k,π-T_ g_i,h^k,π,i,h^π,φ, _if_i,h+1^k,π \|_1, _i,h^π. Proof. Fix k, π, and i. For notational simplicity, write Vh:=Vi,hπ,σi,Qh:=Qi,hπ,σi,fh:=fi,hk,π,g^h:=g^i,hk,π,V_h:=V_i,h^π, _i, Q_h:=Q_i,h^π, _i, f_h:=f_i,h^k,π, g_h:= g_i,h^k,π, and h:=i,hπ,ϕ,σi,^h:=g^i,hk,π,i,hπ,ϕ,σi.T_h:=T_i,h^π,φ, _i, T_h:=T_ g_i,h^k,π,i,h^π,φ, _i. Let fhπ(s):=∼πh(⋅∣s)[fh(s,)].f_h^π(s):=E_ a _h(· s)[f_h(s, a)]. Since π∈Πpurπ∈ ^pur, this reduces to fhπ(s)=fh(s,πh(s)).f_h^π(s)=f_h(s, _h(s)). Step 1: Robust Bellman recursion. By the robust Bellman equation, Qh(s,)=ri,h(s,)+infP∈i,hσi,ϕ(s,)s′∼P[Vh+1(s′)],Q_h(s, a)=r_i,h(s, a)+ _P _i,h _i,φ(s, a)E_s P[V_h+1(s )], and Vh(s)=∼πh(⋅∣s)[Qh(s,)].V_h(s)=E_ a _h(· s)[Q_h(s, a)]. Equivalently, Qh=hVh+1,Vh(s)=∼πh(⋅∣s)[(hVh+1)(s,)].Q_h=T_hV_h+1, V_h(s)=E_ a _h(· s)[(T_hV_h+1)(s, a)]. Also, VH+1≡0,fH+1≡0.V_H+1≡ 0, f_H+1≡ 0. Step 2: Define the robust occupancy distribution. For the fixed policy π and agent i, let νi,hπ _i,h^π denote the stage-h state-action occupancy induced by π and the worst-case transition kernels attaining the robust Bellman recursion for Vi,hπ,σiV_i,h^π, _i. That is, (sh,h)∼νi,hπ(s_h, a_h) _i,h^π when the trajectory starts from s1ks_1^k, follows policy π, and evolves under the worst-case kernels associated with the robust value of agent i. Step 3: Expected robust performance-difference identity. For any function sequence f=fhh=1H+1f=\f_h\_h=1^H+1 with fH+1≡0f_H+1≡ 0, the robust Bellman telescoping identity gives V1(s1k)−f1π(s1k)=∑h=1Hνi,hπ[(hfh+1)(sh,h)−fh(sh,h)]. V_1(s_1^k)-f_1^π(s_1^k)= _h=1^HE_ _i,h^π [(T_hf_h+1)(s_h, a_h)-f_h(s_h, a_h) ]. (137) This follows by adding and subtracting fhf_h along the robust trajectory: Vh(sh)−fh(sh,h)=(hfh+1)(sh,h)−fh(sh,h)+[Vh+1(sh+1)−fh+1π(sh+1)∣sh,h],V_h(s_h)-f_h(s_h, a_h)=(T_hf_h+1)(s_h, a_h)-f_h(s_h, a_h)+E[V_h+1(s_h+1)-f_h+1^π(s_h+1) s_h, a_h], and then summing over h=1,…,Hh=1,…,H. The terminal term vanishes because VH+1=fH+1=0V_H+1=f_H+1=0. Step 4: Add and subtract the dual plug-in operator. For each stage h, hfh+1−fh=(hfh+1−^hfh+1)+(^hfh+1−fh).T_hf_h+1-f_h= (T_hf_h+1- T_hf_h+1 )+ ( T_hf_h+1-f_h ). Using x≤|x|x≤|x| and x≤x+x≤ x_+, we obtain (hfh+1−fh)(s,)≤ (T_hf_h+1-f_h)(s, a)≤ |(hfh+1)(s,)−(^hfh+1)(s,)| |(T_hf_h+1)(s, a)-( T_hf_h+1)(s, a) | +((^hfh+1)(s,)−fh(s,))+. + (( T_hf_h+1)(s, a)-f_h(s, a) )_+. (138) Substituting (138) into (137) gives V1(s1k)−f1π(s1k)≤ V_1(s_1^k)-f_1^π(s_1^k)≤ ∑h=1H‖hfh+1−^hfh+1‖1,νi,hπ+∑h=1Hνi,hπ[(^hfh+1−fh)+]. _h=1^H \|T_hf_h+1- T_hf_h+1 \|_1, _i,h^π+ _h=1^HE_ _i,h^π [ ( T_hf_h+1-f_h )_+ ]. (139) Step 5: Substitute the fitted functions and payoff definition. Taking fh=fi,hk,πf_h=f_i,h^k,π, we have f1π(s1k)=fi,1k,π(s1k),f_1^π(s_1^k)=f_i,1^k,π(s_1^k), because π is pure. Therefore, Vi,1π,σi(s1k)≤ V_i,1^π, _i(s_1^k)≤ fi,1k,π(s1k)+∑h=1H‖i,hπ,ϕ,σifi,h+1k,π−g^i,hk,π,i,hπ,ϕ,σifi,h+1k,π‖1,νi,hπ f_i,1^k,π(s_1^k)+ _h=1^H \|T_i,h^π,φ, _if_i,h+1^k,π-T_ g_i,h^k,π,i,h^π,φ, _if_i,h+1^k,π \|_1, _i,h^π +∑h=1Hνi,hπ[(g^i,hk,π,i,hπ,ϕ,σifi,h+1k,π−fi,hk,π)+]. + _h=1^HE_ _i,h^π [ (T_ g_i,h^k,π,i,h^π,φ, _if_i,h+1^k,π-f_i,h^k,π )_+ ]. (140) Here ^ik−1 K_i^k-1 is the cumulative centered empirical robust discrepancy defined in (11). By the definition of the regularized robust payoff, U^ik(π)=fi,1k,π(s1k)−β^ik−1(fik,π,g^ik,π,π). U_i^k(π)=f_i,1^k,π(s_1^k)-β K_i^k-1(f_i^k,π; g_i^k,π;π). Equivalently, fi,1k,π(s1k)=U^ik(π)+β^ik−1(fik,π,g^ik,π,π).f_i,1^k,π(s_1^k)= U_i^k(π)+β K_i^k-1(f_i^k,π; g_i^k,π;π). Substituting this into (140), we obtain Vi,1π,σi(s1k)≤ V_i,1^π, _i(s_1^k)≤ U^ik(π)+β^ik−1(fik,π,g^ik,π,π)+∑h=1H‖i,hπ,ϕ,σifi,h+1k,π−g^i,hk,π,i,hπ,ϕ,σifi,h+1k,π‖1,νi,hπ U_i^k(π)+β K_i^k-1(f_i^k,π; g_i^k,π;π)+ _h=1^H \|T_i,h^π,φ, _if_i,h+1^k,π-T_ g_i,h^k,π,i,h^π,φ, _if_i,h+1^k,π \|_1, _i,h^π +∑h=1Hνi,hπ[(g^i,hk,π,i,hπ,ϕ,σifi,h+1k,π−fi,hk,π)+]. + _h=1^HE_ _i,h^π [ (T_ g_i,h^k,π,i,h^π,φ, _if_i,h+1^k,π-f_i,h^k,π )_+ ]. (141) This proves the claim. ∎ Lemma 5 (Robust Empirical Discrepancy Concentration). Assume Assumptions 1–3 hold and define Λi:=log⁡(|ℱi‖i‖Πpur|KHmδ). _i:= \! ( |F_i||G_i|| ^pur|KHmδ ). Fix any agent i∈ℳi , any pure joint policy π∈Πpurπ∈ ^pur, any function fi∈ℱif_i _i, and any k∈[K]k∈[K]. Then, for K≥ΛiK≥ _i, with probability at least 1−δ1-δ, we have ∑s=1k−1ei,1s(fi,π)≤2^ik−1(fi,g^ifi,π,π)+⁡(HBϕ(σi)2Klog⁡(|ℱi‖i‖Πpur|KHmδ)+εdualapprox). _s=1^k-1e_i,1^s(f_i,π)≤ 2\, K_i^k-1(f_i; g_i^f_i,π;π)+O (HB_φ( _i)^2 K \! ( |F_i||G_i|| ^pur|KHmδ )+ ^approx_dual ). Proof. We follow the same overall structure as the proof of Lemma A.1 in [78], but with one additional step to control the error induced by approximating the robust Bellman operator through the dual plug-in operator. For each stage h∈[H]h∈[H] and episode k≥2k≥ 2, recall the historical occupancy mixture from Lemma 1, μ¯hk−1:=1k−1∑s=1k−1μhs,π. μ_h^k-1:= 1k-1 _s=1^k-1 _h^s,π. (142) Equivalently, for every integrable function φ , ∑s=1k−1μhs,π[φ]=(k−1)μ¯hk−1[φ]. _s=1^k-1E_ _h^s,π[ ]=(k-1)\,E_ μ_h^k-1[ ]. (143) Step 1: Expand the cumulative population training error. By definition (84), ∑s=1k−1ei,1s(fi,π)=∑h=1H∑s=1k−1μhs,π[(fi,h−i,hπ,ϕ,σifi,h+1)2]. _s=1^k-1e_i,1^s(f_i,π)= _h=1^H _s=1^k-1E_ _h^s,π [ (f_i,h-T_i,h^π,φ, _if_i,h+1 )^2 ]. (144) Step 2: Add and subtract the plug-in robust Bellman operator. For each stage h, let g^i,hfi,π g_i,h^f_i,π denote the empirical dual minimizer computed from the historical dataset hk−1D_h^k-1, and define the corresponding plug-in robust dual-Bellman operator [g^i,hfi,π,i,hπ,ϕ,σifi,h+1](s,) [T_ g_i,h^f_i,π,i,h^π,φ, _if_i,h+1 ](s, a) as in (20). Then, fi,h−i,hπ,ϕ,σifi,h+1 f_i,h-T_i,h^π,φ, _if_i,h+1 =fi,h−g^i,hfi,π,i,hπ,ϕ,σifi,h+1⏟plug-in residual = f_i,h-T_ g_i,h^f_i,π,i,h^π,φ, _if_i,h+1_plug-in residual +g^i,hfi,π,i,hπ,ϕ,σifi,h+1−i,hπ,ϕ,σifi,h+1⏟operator approximation error. + T_ g_i,h^f_i,π,i,h^π,φ, _if_i,h+1-T_i,h^π,φ, _if_i,h+1_operator approximation error. (145) Using (x+y)2≤2x2+2y2(x+y)^2≤ 2x^2+2y^2, we obtain ∑s=1k−1ei,1s(fi,π)≤2Ai,k(fi,π)+2Bi,k(fi,π), _s=1^k-1e_i,1^s(f_i,π)≤ 2A_i,k(f_i,π)+2B_i,k(f_i,π), (146) where Ai,k(fi,π) A_i,k(f_i,π) :=∑h=1H∑s=1k−1μhs,π[(fi,h−g^i,hfi,π,i,hπ,ϕ,σifi,h+1)2], := _h=1^H _s=1^k-1E_ _h^s,π [ (f_i,h-T_ g_i,h^f_i,π,i,h^π,φ, _if_i,h+1 )^2 ], (147) Bi,k(fi,π) B_i,k(f_i,π) :=∑h=1H∑s=1k−1μhs,π[(g^i,hfi,π,i,hπ,ϕ,σifi,h+1−i,hπ,ϕ,σifi,h+1)2]. := _h=1^H _s=1^k-1E_ _h^s,π [ (T_ g_i,h^f_i,π,i,h^π,φ, _if_i,h+1-T_i,h^π,φ, _if_i,h+1 )^2 ]. (148) Step 3: Control of the operator approximation term Bi,kB_i,k. For notational convenience, define Δi,hfi,π(s,):=(g^i,hfi,π,i,hπ,ϕ,σifi,h+1−i,hπ,ϕ,σifi,h+1)(s,). _i,h^f_i,π(s, a):= (T_ g_i,h^f_i,π,i,h^π,φ, _if_i,h+1-T_i,h^π,φ, _if_i,h+1 )(s, a). (149) Then, by (143) applied with φ=(Δi,hfi,π)2 =( _i,h^f_i,π)^2, Bi,k(fi,π) B_i,k(f_i,π) =∑h=1H∑s=1k−1μhs,π[(Δi,hfi,π)2] = _h=1^H _s=1^k-1E_ _h^s,π [ ( _i,h^f_i,π )^2 ] =(k−1)∑h=1Hμ¯hk−1[(Δi,hfi,π)2]. =(k-1) _h=1^HE_ μ_h^k-1 [ ( _i,h^f_i,π )^2 ]. (150) By Assumption 3, the dual integrand is uniformly bounded: |Lossi,h,ϕ(fi,h+1,s,,s′,λ,ν)|≤Bϕ(σi)∀(s,,s′),∀(λ,ν)∈Θϕ. |Loss_i,h,φ(f_i,h+1;s, a,s ;λ,ν) |≤ B_φ( _i) ∀(s, a,s ), ∀(λ,ν)∈ _φ. Since both the exact robust Bellman operator and the plug-in robust dual-Bellman operator are obtained from expectations of such bounded quantities, and their reward terms cancel, it follows that |Δi,hfi,π(s,)|≤2Bϕ(σi)∀(s,). | _i,h^f_i,π(s, a) |≤ 2B_φ( _i) ∀(s, a). (151) Consequently, (Δi,hfi,π(s,))2≤2Bϕ(σi)|Δi,hfi,π(s,)|. ( _i,h^f_i,π(s, a) )^2≤ 2B_φ( _i) | _i,h^f_i,π(s, a) |. (152) Taking expectation under μ¯hk−1 μ_h^k-1 gives μ¯hk−1[(Δi,hfi,π)2]≤2Bϕ(σi)‖Δi,hfi,π‖1,μ¯hk−1.E_ μ_h^k-1 [ ( _i,h^f_i,π )^2 ]≤ 2B_φ( _i) \| _i,h^f_i,π \|_1, μ_h^k-1. (153) Lemma 1, applied with h=hk−1D_h=D_h^k-1 (so n=k−1n=k-1), already guarantees a bound on ‖Δi,hfi,π‖1,μ¯hk−1\| _i,h^f_i,π\|_1, μ_h^k-1 holding uniformly over π∈Πpurπ∈ ^pur and fi,h+1∈ℱi,h+1f_i,h+1 _i,h+1, for a fixed agent i and stage h. Taking a further union bound over the remaining indices i∈[m]i∈[m], h∈[H]h∈[H], and k∈2,…,Kk∈\2,…,K\ (i.e., only over the indices not already ranged over inside Lemma 1), we obtain that, with probability at least 1−δ1-δ, simultaneously for all such choices and all fi∈ℱif_i _i, ‖Δi,hfi,π‖1,μ¯hk−1≤c⁡(Bϕ(σi)log⁡(|ℱi‖i‖Πpur|KHmδ)k−1+ε¯dual). \| _i,h^f_i,π \|_1, μ_h^k-1≤ c (B_φ( _i) \! ( |F_i||G_i|| ^pur|KHmδ )k-1+ _dual ). (154) Substituting (154) into (153) yields μ¯hk−1[(Δi,hfi,π)2] _ μ_h^k-1 [ ( _i,h^f_i,π )^2 ] ≤c1Bϕ(σi)2log⁡(|ℱi‖i‖Πpur|KHmδ)k−1+c2Bϕ(σi)ε¯dual. ≤ c_1B_φ( _i)^2 \! ( |F_i||G_i|| ^pur|KHmδ )k-1+c_2B_φ( _i) _dual. (155) Combining (150) and (155), and summing over h∈[H]h∈[H], gives Bi,k(fi,π) B_i,k(f_i,π) ≤c1HBϕ(σi)2(k−1)log⁡(|ℱi‖i‖Πpur|KHmδ) ≤ c_1HB_φ( _i)^2 (k-1) \! ( |F_i||G_i|| ^pur|KHmδ ) +c2H(k−1)Bϕ(σi)ε¯dual. +c_2H(k-1)B_φ( _i) _dual. (156) Since k−1≤Kk-1≤ K and ε¯dual≤εdualapproxHKBϕ(σi), _dual≤ _dual^approxHKB_φ( _i), we conclude that Bi,k(fi,π)≤⁡(HBϕ(σi)2Klog⁡(|ℱi‖i‖Πpur|KHmδ)+εdualapprox). B_i,k(f_i,π) (HB_φ( _i)^2 K \! ( |F_i||G_i|| ^pur|KHmδ )+ _dual^approx ). (157) Step 4: Control the plug-in residual term Ai,kA_i,k using the aggregate empirical discrepancy. For each stage h, function fi∈ℱif_i _i, and fixed dual function gi,h∈i,hg_i,h _i,h, define the aggregate empirical squared loss L^i,hk−1(fi,gi,h,π):=∑j=1k−1δ^i,h(fi,gi,h,π,shj,hj,sh+1j)2, L_i,h^k-1(f_i;g_i,h,π):= _j=1^k-1 δ_i,h\! (f_i;g_i,h;π;s_h^j, a_h^j,s_h+1^j )^2, (158) and its corresponding conditional population loss Li,hk−1(fi,gi,h,π):=∑j=1k−1⁡[δ^i,h(fi,gi,h,π,shj,hj,sh+1j)2|ℋj−1]. L_i,h^k-1(f_i;g_i,h,π):= _j=1^k-1E\! [ δ_i,h\! (f_i;g_i,h;π;s_h^j, a_h^j,s_h+1^j )^2 |H_j-1 ]. (159) Here gi,hg_i,h is held fixed when taking the conditional expectation. With the revised definition of the centered empirical robust discrepancy, ^ik−1(fi,g^ifi,π,π)=∑h=1H[L^i,hk−1(fi,g^i,hfi,π,π)−inffi′∈ℱiL^i,hk−1(fi′,g^i,hfi,π,π)]. K_i^k-1 (f_i; g_i^f_i,π;π )= _h=1^H [ L_i,h^k-1 (f_i; g_i,h^f_i,π,π )- _f _i _i L_i,h^k-1 (f _i; g_i,h^f_i,π,π ) ]. (160) By Assumption 3, the plug-in residual is uniformly bounded: |δ^i,h(fi,gi,h,π,s,,s′)|≤CBϕ(σi) | δ_i,h\! (f_i;g_i,h;π;s, a,s ) |≤ CB_φ( _i) (161) for every admissible fi,gi,h,πf_i,g_i,h,π and transition (s,,s′)(s, a,s ). For every fixed (fi,gi,h,π)∈ℱi×i,h×Πpur(f_i,g_i,h,π) _i×G_i,h× ^pur, define Zj(fi,gi,h,π):= Z_j(f_i,g_i,h,π):= δ^i,h(fi,gi,h,π,shj,hj,sh+1j)2 δ_i,h\! (f_i;g_i,h;π;s_h^j, a_h^j,s_h+1^j )^2 −⁡[δ^i,h(fi,gi,h,π,shj,hj,sh+1j)2|ℋj−1]. -E\! [ δ_i,h\! (f_i;g_i,h;π;s_h^j, a_h^j,s_h+1^j )^2 |H_j-1 ]. (162) Since the behavior policy used in episode j is ℋj−1H_j-1-measurable, Zj(fi,gi,h,π)j≥1\Z_j(f_i,g_i,h,π)\_j≥ 1 is a bounded martingale-difference sequence. Moreover, |Zj(fi,gi,h,π)|≤CBϕ(σi)2.|Z_j(f_i,g_i,h,π)|≤ CB_φ( _i)^2. Applying Freedman’s inequality (Lemma T.3) and taking a union bound over i∈[m],h∈[H],k∈[K],π∈Πpur,fi∈ℱi,gi,h∈i,h,i∈[m], h∈[H], k∈[K], π∈ ^pur, f_i _i, g_i,h _i,h, we obtain that, with probability at least 1−δ1-δ, simultaneously for all such choices, |L^i,hk−1(fi,gi,h,π)−Li,hk−1(fi,gi,h,π)|≤ϵi,k, | L_i,h^k-1(f_i;g_i,h,π)-L_i,h^k-1(f_i;g_i,h,π) |≤ _i,k, (163) where, writing Λ:=log⁡(|ℱi‖i,h‖Πpur|KHmδ), := \! ( |F_i||G_i,h|| ^pur|KHmδ ), we may take ϵi,k=⁡(Bϕ(σi)2[(k−1)Λ+Λ]). _i,k=O\! (B_φ( _i)^2 [ (k-1) + ] ). (164) In particular, since k−1≤Kk-1≤ K, ϵi,k=⁡(Bϕ(σi)2[KΛ+Λ]). _i,k=O\! (B_φ( _i)^2 [ K + ] ). (165) Because (163) holds uniformly over gi,h∈i,hg_i,h _i,h, we may now substitute the data-dependent empirical minimizer gi,h=g^i,hfi,π.g_i,h= g_i,h^f_i,π. We next pass from uniform loss concentration to excess-loss concentration. For brevity, fix g^i,h:=g^i,hfi,π, g_i,h:= g_i,h^f_i,π, and let fi,hpop∈arg⁡minfi′∈ℱiLi,hk−1(fi′,g^i,h,π).f_i,h^pop∈ _f _i _iL_i,h^k-1(f _i; g_i,h,π). Then, Li,hk−1(fi,g^i,h,π)−inffi′∈ℱiLi,hk−1(fi′,g^i,h,π) L_i,h^k-1(f_i; g_i,h,π)- _f _i _iL_i,h^k-1(f _i; g_i,h,π) ≤L^i,hk−1(fi,g^i,h,π)−inffi′∈ℱiL^i,hk−1(fi′,g^i,h,π)+2ϵi,k. ≤ L_i,h^k-1(f_i; g_i,h,π)- _f _i _i L_i,h^k-1(f _i; g_i,h,π)+2 _i,k. (166) We now relate this population excess loss to the plug-in residual appearing in Ai,kA_i,k. For a fixed gi,hg_i,h, the conditional bias–variance decomposition gives Li,hk−1(fi,gi,h,π)= L_i,h^k-1(f_i;g_i,h,π)= ∑j=1k−1μhj,π[(fi,h−gi,h,i,hπ,ϕ,σifi,h+1)2]+Vari,hk−1(fi,h+1,gi,h,π), _j=1^k-1E_ _h^j,π [ (f_i,h-T_g_i,h,i,h^π,φ, _if_i,h+1 )^2 ]+Var_i,h^k-1(f_i,h+1;g_i,h,π), (167) where the variance term depends on the next-stage function fi,h+1f_i,h+1, the fixed dual function gi,hg_i,h, and the evaluation policy π, but not on the current-stage coordinate fi,hf_i,h. For each stage h, define a comparator f¯ih,fi,π∈ℱi f_i^h,f_i,π _i whose relevant coordinates satisfy f¯i,h,fi,π f_i,h^h,f_i,π =g^i,hfi,π,i,hπ,ϕ,σifi,h+1, =T_ g_i,h^f_i,π,i,h^π,φ, _if_i,h+1, (168) f¯i,h+1h,fi,π f_i,h+1^h,f_i,π =fi,h+1, =f_i,h+1, (169) with the remaining coordinates chosen arbitrarily from their corresponding classes. By the product structure of ℱi=∏h=1Hℱi,hF_i= _h=1^HF_i,h and Assumption 1, such a comparator belongs to ℱiF_i. Because f¯ih,fi,π f_i^h,f_i,π uses the same next-stage function and the same plug-in dual function as fif_i, the two losses have the same conditional variance term, while the comparator’s conditional-mean residual is zero. Therefore, Li,hk−1(fi,g^i,hfi,π,π)−Li,hk−1(f¯ih,fi,π,g^i,hfi,π,π)=∑j=1k−1μhj,π[(fi,h−g^i,hfi,π,i,hπ,ϕ,σifi,h+1)2]. L_i,h^k-1 (f_i; g_i,h^f_i,π,π )-L_i,h^k-1 ( f_i^h,f_i,π; g_i,h^f_i,π,π )= _j=1^k-1E_ _h^j,π [ (f_i,h-T_ g_i,h^f_i,π,i,h^π,φ, _if_i,h+1 )^2 ]. (170) Since inffi′∈ℱiLi,hk−1(fi′,g^i,hfi,π,π)≤Li,hk−1(f¯ih,fi,π,g^i,hfi,π,π), _f _i _iL_i,h^k-1 (f _i; g_i,h^f_i,π,π )≤ L_i,h^k-1 ( f_i^h,f_i,π; g_i,h^f_i,π,π ), (171) it follows that ∑j=1k−1μhj,π[(fi,h−g^i,hfi,π,i,hπ,ϕ,σifi,h+1)2] _j=1^k-1E_ _h^j,π [ (f_i,h-T_ g_i,h^f_i,π,i,h^π,φ, _if_i,h+1 )^2 ] ≤Li,hk−1(fi,g^i,hfi,π,π)−inffi′∈ℱiLi,hk−1(fi′,g^i,hfi,π,π). ≤ L_i,h^k-1 (f_i; g_i,h^f_i,π,π )- _f _i _iL_i,h^k-1 (f _i; g_i,h^f_i,π,π ). (172) Combining (166) and (172), and then summing over h∈[H]h∈[H], yields Ai,k(fi,π)≤ A_i,k(f_i,π)≤ ^ik−1(fi,g^ifi,π,π) K_i^k-1 (f_i; g_i^f_i,π;π ) +⁡(HBϕ(σi)2[Klog⁡(|ℱi‖i‖Πpur|KHmδ)+log⁡(|ℱi‖i‖Πpur|KHmδ)]). +O\! (HB_φ( _i)^2 [ K \! ( |F_i||G_i|| ^pur|KHmδ )+ \! ( |F_i||G_i|| ^pur|KHmδ ) ] ). (173) Step 5: Combine the two bounds. Substituting (173) and (157) into (146), we obtain ∑s=1k−1ei,1s(fi,π)≤ _s=1^k-1e_i,1^s(f_i,π)≤ 2^ik−1(fi,g^ifi,π,π)+⁡(HBϕ(σi)2[KΛ+Λ]+εdualapprox), 2\, K_i^k-1 (f_i; g_i^f_i,π;π )+O\! (HB_φ( _i)^2 [ K + ]+ _dual^approx ), (174) where Λ:=log⁡(|ℱi‖i‖Πpur|KHmδ). := \! ( |F_i||G_i|| ^pur|KHmδ ). Under the standard regime K≥ΛK≥ , we have Λ≤KΛ, ≤ K , and hence the additive logarithmic term can be absorbed into the leading square-root term. Therefore, ∑s=1k−1ei,1s(fi,π)≤ _s=1^k-1e_i,1^s(f_i,π)≤ 2^ik−1(fi,g^ifi,π,π)+⁡(HBϕ(σi)2Klog⁡(|ℱi‖i‖Πpur|KHmδ)+εdualapprox). 2\, K_i^k-1 (f_i; g_i^f_i,π;π )+O\! (HB_φ( _i)^2 K \! ( |F_i||G_i|| ^pur|KHmδ )+ _dual^approx ). (175) This proves the claim. ∎ Lemma 6 (Optimal concentration for Bellman-consistent robust benchmark). Assume Assumptions 1–3 hold, and define Λi:=log⁡(|ℱi‖i‖Πpur|KHmδ) _i:= \! ( |F_i||G_i|| ^pur|KHmδ ). Fix any agent i∈ℳi , any pure joint policy π∈Πpurπ∈ ^pur, and let fiπ,σi=fi,hπ,σih=1H+1∈ℱif_i^π, _i=\f_i,h^π, _i\_h=1^H+1 _i be a robust Bellman-consistent sequence satisfying fi,hπ,σi=i,hπ,ϕ,σifi,h+1π,σi,h∈[H],withfi,H+1π,σi≡0. f_i,h^π, _i=T_i,h^π,φ, _if_i,h+1^π, _i, h∈[H], f_i,H+1^π, _i≡ 0. (176) Then, for K≥ΛiK≥ _i, with probability at least 1−δ1-δ, for every k∈[K]k∈[K], ^ik−1(fiπ,σi,g^ifiπ,σi,π,π)≤⁡(HBϕ(σi)2Klog⁡(|ℱ‖Πpur|KHmδ)+εdualapprox), K_i^k-1(f_i^π, _i; g_i^\,f_i^π, _i,π;π) \! (HB_φ( _i)^2 K \! ( |F||G|\,| ^pur|KHmδ )+ ^approx_dual ), where ^ik−1 K_i^k-1 denotes the cumulative centered empirical robust discrepancy as defined in (11). Proof. We fix an agent i∈ℳi , a pure joint policy π∈Πpurπ∈ ^pur, and an index k∈[K]k∈[K]. Throughout the proof, we abbreviate fi⋆:=fiπ,σi,Λ:=log⁡(|ℱ‖Πpur|KHmδ).f_i :=f_i^π, _i, := \! ( |F||G|| ^pur|KHmδ ). By assumption, fi⋆f_i is Bellman-consistent with respect to the true robust Bellman operator, namely (176) holds, i.e., fi,h⋆=i,hπ,ϕ,σifi,h+1⋆,h∈[H].f_i,h =T_i,h^π,φ, _if_i,h+1 , h∈[H]. Step 1: Define empirical and population aggregate losses. For each stage h∈[H]h∈[H], and for any fixed (fi,gi,π)∈ℱi×i×Πpur(f_i,g_i,π) _i×G_i× ^pur, define L^i,hk−1(fi,gi):=∑j=1k−1δ^i,h(fi,gi,h,π,shj,hj,sh+1j)2, L_i,h^k-1(f_i;g_i):= _j=1^k-1 δ_i,h\! (f_i;g_i,h;π;s_h^j, a_h^j,s_h+1^j )^2, (177) and Li,hk−1(fi,gi):=∑j=1k−1⁡[δ^i,h(fi,gi,h,π,shj,hj,sh+1j)2|ℋj−1]. L_i,h^k-1(f_i;g_i):= _j=1^k-1E\! [ δ_i,h\! (f_i;g_i,h;π;s_h^j, a_h^j,s_h+1^j )^2 |H_j-1 ]. (178) The concentration argument below is first proved uniformly over fixed (fi,gi,π)(f_i,g_i,π). Since the resulting event holds simultaneously for all gi∈ig_i _i, we may then substitute the data-dependent empirical dual minimizer gi=g^ifi⋆,πg_i= g_i^\,f_i ,π. On this uniform event, taking gi=g^ifi⋆,πg_i= g_i^\,f_i ,π, the centered empirical robust discrepancy satisfies (11), as ^ik−1(fi⋆,g^ifi⋆,π,π)=∑h=1H[L^i,hk−1(fi⋆,g^ifi⋆,π)−inffi′∈ℱiL^i,hk−1(fi′,g^ifi⋆,π)]. K_i^k-1(f_i ; g_i^\,f_i ,π;π)= _h=1^H [ L_i,h^k-1(f_i ; g_i^\,f_i ,π)- _f _i _i L_i,h^k-1(f _i; g_i^\,f_i ,π) ]. (179) The proof follows the similar general strategy as in [78, Lemma A.2]: we compare the empirical discrepancy of the benchmark function to the smallest achievable empirical discrepancy, introduce a martingale difference sequence, and apply Freedman’s inequality. The new ingredient in the robust case is that the benchmark fi⋆f_i is a fixed point of the true robust Bellman operator, whereas the empirical discrepancy is built from the plug-in robust Bellman operator induced by the empirical dual minimizer. Therefore, unlike the non-robust proof, an additional operator-approximation term appears and must be controlled by Lemma 1. Step 2: Uniform concentration for the aggregate excess risk. We show that the empirical excess risk is controlled by its population counterpart. • Boundedness of the squared loss. By Assumption 3, the plug-in residual is uniformly bounded: there exists a constant CB>0C_B>0 such that, for all fi∈ℱif_i _i, |δ^i,h(fi,g^i,hfi⋆,π,π,s,a,s′)|≤CBBϕ(σi). | δ_i,h\! (f_i; g_i,h^\,f_i ,π;π;s,a,s ) |≤ C_BB_φ( _i). Hence, the squared loss satisfies 0≤δ^i,h(⋅)2≤CB2Bϕ(σi)2.0≤ δ_i,h(·)^2≤ C_B^2B_φ( _i)^2. This boundedness enables martingale concentration. • Martingale difference decomposition. Fix i,h,π,ki,h,π,k, fi∈ℱif_i _i, and gi∈ig_i _i. Define Zj(fi,gi):=δ^i,h(fi,gi,h,π,shj,hj,sh+1j)2−⁡[δ^i,h(fi,gi,h,π,shj,hj,sh+1j)2|ℋj−1],Z_j(f_i,g_i):= δ_i,h\! (f_i;g_i,h;π;s_h^j, a_h^j,s_h+1^j )^2-E\! [ δ_i,h\! (f_i;g_i,h;π;s_h^j, a_h^j,s_h+1^j )^2 |H_j-1 ], where we suppress arguments for brevity. Then Zj(fi,gi)j=1k−1\Z_j(f_i,g_i)\_j=1^k-1 is a martingale difference sequence with |Zj(fi,gi)|≤CBϕ(σi)2.|Z_j(f_i,g_i)|≤ CB_φ( _i)^2. • Concentration for a fixed function. By Freedman’s inequality (Lemma T.3), for any fixed fif_i, with probability at least 1−δ′1-δ , |∑j=1k−1Zj(fi,gi)|≤CBϕ(σi)2[(k−1)log⁡1δ′+log⁡1δ′]. | _j=1^k-1Z_j(f_i,g_i) |≤ CB_φ( _i)^2 [ (k-1) 1δ + 1δ ]. This is the correct form of Freedman’s inequality: unlike Lemma 1’s Step 5, where the analogous martingale sum is divided by the sample size n (so the linear-in-log⁡(1/δ′) (1/δ ) term becomes negligible relative to the square-root term automatically), here we are bounding a raw sum, and the additive term must be tracked explicitly. Since ∑j=1k−1Zj(fi,gi)=L^i,hk−1(fi,gi)−Li,hk−1(fi,gi), _j=1^k-1Z_j(f_i,g_i)= L_i,h^k-1(f_i,g_i)-L_i,h^k-1(f_i,g_i), we obtain |L^i,hk−1(fi,gi)−Li,hk−1(fi,gi)|≤CBϕ(σi)2[(k−1)log⁡1δ′+log⁡1δ′]. | L_i,h^k-1(f_i,g_i)-L_i,h^k-1(f_i,g_i) |≤ CB_φ( _i)^2 [ (k-1) 1δ + 1δ ]. • Uniform control over the function class. We now extend the above bound uniformly over all (fi,gi)∈ℱi×i(f_i,g_i) _i×G_i, and over all i,h,π,ki,h,π,k, using a union bound. Setting δ′=δ|ℱ‖Πpur|KHm,δ = δ|F||G|| ^pur|KHm, we obtain that, with probability at least 1−δ1-δ, simultaneously for all i,h,π,ki,h,π,k and all fi∈ℱif_i _i, |L^i,hk−1(fi,gi)−Li,hk−1(fi,gi)|≤ϵk,h, | L_i,h^k-1(f_i,g_i)-L_i,h^k-1(f_i,g_i) |≤ _k,h, (180) where, using k−1≤Kk-1≤ K, ϵk,h:=⁡(Bϕ(σi)2[KΛ+Λ]). _k,h:=O\! (B_φ( _i)^2 [ K + ] ). • Passing from uniform loss concentration to excess-risk concentration. Fix h∈[H]h∈[H] and fix a dual sequence gi∈ig_i _i. Let f^i∈arg⁡minfi′∈ℱiL^i,hk−1(fi′,gi). f_i∈ _f _i _i L_i,h^k-1(f _i;g_i). Then, by definition of f^i f_i, L^i,hk−1(fi⋆,gi)−inffi′∈ℱiL^i,hk−1(fi′,gi)=L^i,hk−1(fi⋆,gi)−L^i,hk−1(f^i,gi). L_i,h^k-1(f_i ;g_i)- _f _i _i L_i,h^k-1(f _i;g_i)= L_i,h^k-1(f_i ;g_i)- L_i,h^k-1( f_i;g_i). Now add and subtract the corresponding population losses: L^i,hk−1(fi⋆,gi)−L^i,hk−1(f^i,gi) L_i,h^k-1(f_i ;g_i)- L_i,h^k-1( f_i;g_i) =[Li,hk−1(fi⋆,gi)−Li,hk−1(f^i,gi)] = [L_i,h^k-1(f_i ;g_i)-L_i,h^k-1( f_i;g_i) ] +[L^i,hk−1(fi⋆,gi)−Li,hk−1(fi⋆,gi)] + [ L_i,h^k-1(f_i ;g_i)-L_i,h^k-1(f_i ;g_i) ] −[L^i,hk−1(f^i,gi)−Li,hk−1(f^i,gi)]. - [ L_i,h^k-1( f_i;g_i)-L_i,h^k-1( f_i;g_i) ]. On the uniform concentration event (180), applied to both fi⋆f_i and f^i f_i, we have L^i,hk−1(fi⋆,gi)−Li,hk−1(fi⋆,gi)≤ϵk,h, L_i,h^k-1(f_i ;g_i)-L_i,h^k-1(f_i ;g_i)≤ _k,h, and −[L^i,hk−1(f^i,gi)−Li,hk−1(f^i,gi)]≤ϵk,h.- [ L_i,h^k-1( f_i;g_i)-L_i,h^k-1( f_i;g_i) ]≤ _k,h. Therefore, L^i,hk−1(fi⋆,gi)−inffi′∈ℱiL^i,hk−1(fi′,gi) L_i,h^k-1(f_i ;g_i)- _f _i _i L_i,h^k-1(f _i;g_i) ≤Li,hk−1(fi⋆,gi)−Li,hk−1(f^i,gi)+2ϵk,h. ≤ L_i,h^k-1(f_i ;g_i)-L_i,h^k-1( f_i;g_i)+2 _k,h. Since f^i∈ℱi f_i _i, we have Li,hk−1(f^i,gi)≥inffi′∈ℱiLi,hk−1(fi′,gi).L_i,h^k-1( f_i;g_i)≥ _f _i _iL_i,h^k-1(f _i;g_i). Hence, L^i,hk−1(fi⋆,gi)−inffi′∈ℱiL^i,hk−1(fi′,gi)≤Li,hk−1(fi⋆,gi)−inffi′∈ℱiLi,hk−1(fi′,gi)+2ϵk,h. L_i,h^k-1(f_i ;g_i)- _f _i _i L_i,h^k-1(f _i;g_i)≤ L_i,h^k-1(f_i ;g_i)- _f _i _iL_i,h^k-1(f _i;g_i)+2 _k,h. (181) • Specializing to the empirical dual minimizer and summing over stages. The event (180) holds uniformly over gi∈ig_i _i. Therefore, we may substitute the data-dependent empirical dual minimizer gi=g^ifi⋆,πg_i= g_i^\,f_i ,π into (181). Summing over h∈[H]h∈[H] and using (179), we obtain ^ik−1(fi⋆,g^ifi⋆,π,π)≤∑h=1H[Li,hk−1(fi⋆,g^ifi⋆,π)−inffi′∈ℱiLi,hk−1(fi′,g^ifi⋆,π)]+2∑h=1Hϵk,h. K_i^k-1(f_i ; g_i^\,f_i ,π;π)≤ _h=1^H [L_i,h^k-1(f_i ; g_i^\,f_i ,π)- _f _i _iL_i,h^k-1(f _i; g_i^\,f_i ,π) ]+2 _h=1^H _k,h. (182) Since ϵk,h=⁡(Bϕ(σi)2[KΛ+Λ]) _k,h=O\! (B_φ( _i)^2[ K + ] ), we get ^ik−1(fi⋆,g^ifi⋆,π,π)≤ K_i^k-1(f_i ; g_i^\,f_i ,π;π)≤ ∑h=1H[Li,hk−1(fi⋆,g^ifi⋆,π)−inffi′∈ℱiLi,hk−1(fi′,g^ifi⋆,π)] _h=1^H [L_i,h^k-1(f_i ; g_i^\,f_i ,π)- _f _i _iL_i,h^k-1(f _i; g_i^\,f_i ,π) ] +⁡(HBϕ(σi)2[KΛ+Λ]). +O\! (HB_φ( _i)^2 [ K + ] ). (183) Step 3: Conditional bias–variance decomposition. Fix h∈[H]h∈[H], j≤k−1j≤ k-1, and a fixed pair (fi,gi)∈ℱi×i(f_i,g_i) _i×G_i. Conditional on ℋj−1H_j-1, the only remaining randomness is sh+1j∼Ph⋆(⋅|shj,hj)s_h+1^j P_h (·|s_h^j, a_h^j). Therefore, ⁡[δ^i,h(fi,g^i,hfi⋆,π,π,shj,hj,sh+1j)2|ℋj−1]=μhj,π[(fi,h−Tg^i,hfi⋆,π,i,hπ,ϕ,σifi,h+1)2] \! [ δ_i,h\! (f_i; g_i,h^\,f_i ,π;π;s_h^j, a_h^j,s_h+1^j )^2 |H_j-1 ]=E_ _h^j,π [ (f_i,h-T_ g_i,h^\,f_i ,π,i,h^π,φ, _if_i,h+1 )^2 ] +μhj,π[Vars′∼Ph⋆(⋅|s,a)(δ^i,h(fi;g^i,hfi⋆,π;π;s,a,s′))]. +E_ _h^j,π [Var_s P_h (·|s,a) ( δ_i,h(f_i; g_i,h^\,f_i ,π;π;s,a,s ) ) ]. (184) This is the conditional bias–variance identity. The second term is the irreducible next-state variance; the proof below cancels it using the aggregate excess-risk comparison. Step 4: Choose a plug-in Bellman comparator and cancel the variance. We first carry out this step for an arbitrary fixed dual sequence gi∈ig_i _i. This avoids any measurability issue with the data-dependent empirical minimizer. At the end of the argument, we specialize to gi=g^ifi⋆,πg_i= g_i^\,f_i ,π, which is valid because the preceding concentration event holds uniformly over gi∈ig_i _i. For fixed gig_i, define Δi,hk,π,gi:=i,hπ,ϕ,σifi,h+1⋆−gi,h,i,hπ,ϕ,σifi,h+1⋆. _i,h^k,π,g_i:=T_i,h^π,φ, _if_i,h+1 -T_g_i,h,i,h^π,φ, _if_i,h+1 . (185) Since fi⋆f_i is Bellman-consistent with respect to the true robust Bellman operator, fi,h⋆=i,hπ,ϕ,σifi,h+1⋆,f_i,h =T_i,h^π,φ, _if_i,h+1 , we have fi,h⋆−gi,h,i,hπ,ϕ,σifi,h+1⋆=Δi,hk,π,gi.f_i,h -T_g_i,h,i,h^π,φ, _if_i,h+1 = _i,h^k,π,g_i. For each h∈[H]h∈[H], choose a comparator f¯ik,h,π,gi∈ℱi f_i^k,h,π,g_i _i whose h-th coordinate is f¯i,hk,h,π,gi=gi,h,i,hπ,ϕ,σifi,h+1⋆, f_i,h^k,h,π,g_i=T_g_i,h,i,h^π,φ, _if_i,h+1 , whose (h+1)(h+1)-th coordinate is fi,h+1⋆f_i,h+1 , and whose remaining coordinates are chosen so that f¯ik,h,π,gi∈ℱi f_i^k,h,π,g_i _i. This is valid by the product structure of ℱiF_i together with robust Bellman completeness for the plug-in operator. For this comparator, f¯i,hk,h,π,gi−gi,h,i,hπ,ϕ,σif¯i,h+1k,h,π,gi=0, f_i,h^k,h,π,g_i-T_g_i,h,i,h^π,φ, _i f_i,h+1^k,h,π,g_i=0, because f¯i,h+1k,h,π,gi=fi,h+1⋆ f_i,h+1^k,h,π,g_i=f_i,h+1 . Now apply the conditional bias–variance decomposition from (184) twice: once with fi=fi⋆f_i=f_i and once with fi=f¯ik,h,π,gif_i= f_i^k,h,π,g_i. Since both residuals use the same next-stage function fi,h+1⋆f_i,h+1 , the same fixed dual function gi,hg_i,h, and the same next-state sample, the conditional variance terms are identical and cancel in the difference. Therefore, for every j≤k−1j≤ k-1, ⁡[δ^i,h(fi⋆,gi,h,π,shj,hj,sh+1j)2−δ^i,h(f¯ik,h,π,gi,gi,h,π,shj,hj,sh+1j)2|ℋj−1] \! [ δ_i,h(f_i ;g_i,h;π;s_h^j, a_h^j,s_h+1^j)^2- δ_i,h( f_i^k,h,π,g_i;g_i,h;π;s_h^j, a_h^j,s_h+1^j)^2 |H_j-1 ] =μhj,π[(Δi,hk,π,gi)2]. =E_ _h^j,π [( _i,h^k,π,g_i)^2 ]. (186) Since f¯ik,h,π,gi∈ℱi f_i^k,h,π,g_i _i, we have inffi′∈ℱiLi,hk−1(fi′,gi)≤Li,hk−1(f¯ik,h,π,gi,gi). _f _i _iL_i,h^k-1(f _i;g_i)≤ L_i,h^k-1( f_i^k,h,π,g_i;g_i). Consequently, Li,hk−1(fi⋆,gi)−inffi′∈ℱiLi,hk−1(fi′,gi) L_i,h^k-1(f_i ;g_i)- _f _i _iL_i,h^k-1(f _i;g_i) ≤Li,hk−1(fi⋆,gi)−Li,hk−1(f¯ik,h,π,gi,gi) ≤ L_i,h^k-1(f_i ;g_i)-L_i,h^k-1( f_i^k,h,π,g_i;g_i) =∑j=1k−1μhj,π[(Δi,hk,π,gi)2]. = _j=1^k-1E_ _h^j,π [( _i,h^k,π,g_i)^2 ]. (187) We now specialize to the empirical dual minimizer gi=g^ifi⋆,πg_i= g_i^\,f_i ,π. For notational simplicity, write Δi,hk,π:=Δi,hk,π,g^ifi⋆,π=i,hπ,ϕ,σifi,h+1⋆−g^i,hfi⋆,π,i,hπ,ϕ,σifi,h+1⋆. _i,h^k,π:= _i,h^k,π, g_i^\,f_i ,π=T_i,h^π,φ, _if_i,h+1 -T_ g_i,h^\,f_i ,π,i,h^π,φ, _if_i,h+1 . Combining (183) with (187) gives ^ik−1(fi⋆,g^ifi⋆,π,π)≤∑h=1H∑j=1k−1μhj,π[(Δi,hk,π)2]+⁡(HBϕ(σi)2[KΛ+Λ]). K_i^k-1(f_i ; g_i^\,f_i ,π;π)≤ _h=1^H _j=1^k-1E_ _h^j,π [( _i,h^k,π)^2 ]+O\! (HB_φ( _i)^2 [ K + ] ). (188) Step 5: Bound the operator-approximation term. Recall the historical occupancy mixture μ¯hk−1:=1k−1∑j=1k−1μhj,π μ_h^k-1:= 1k-1 _j=1^k-1 _h^j,π from Lemma 5, satisfying the exact identity ∑j=1k−1μhj,π[φ]=(k−1)μ¯hk−1[φ] _j=1^k-1E_ _h^j,π[ ]=(k-1)E_ μ_h^k-1[ ] for any integrable φ . Applying this identity with φ=(Δi,hk,π)2 =( _i,h^k,π)^2, ∑j=1k−1μhj,π[(Δi,hk,π)2]=(k−1)μ¯hk−1[(Δi,hk,π)2]. _j=1^k-1E_ _h^j,π [( _i,h^k,π)^2 ]=(k-1)\,E_ μ_h^k-1 [( _i,h^k,π)^2 ]. (189) By the boundedness of the true robust Bellman operator and the plug-in robust Bellman operator, ‖Δi,hk,π‖∞≤C′Bϕ(σi).\| _i,h^k,π\|_∞≤ C B_φ( _i). Therefore, using x2≤‖x‖∞|x|x^2≤\|x\|_∞|x|, μ¯hk−1[(Δi,hk,π)2] _ μ_h^k-1 [( _i,h^k,π)^2 ] ≤‖Δi,hk,π‖∞‖Δi,hk,π‖1,μ¯hk−1. ≤\| _i,h^k,π\|_∞\| _i,h^k,π\|_1, μ_h^k-1. (190) The dataset hk−1D_h^k-1 is generated by executing π1,…,πk−1π^1,…,π^k-1 over stage h, i.e., it is exactly the data that μ¯hk−1 μ_h^k-1 is drawn from. Applying Lemma 1 with h=hk−1D_h=D_h^k-1 (so n=k−1n=k-1) – which already holds uniformly over π∈Πpurπ∈ ^pur and the benchmark continuation fi,h+1⋆∈ℱi,h+1f_i,h+1 _i,h+1 – and taking a further union bound over the remaining indices i∈[m]i∈[m], h∈[H]h∈[H], k∈2,…,Kk∈\2,…,K\, we obtain, with probability at least 1−δ1-δ, simultaneously for all such choices, ‖Δi,hk,π‖1,μ¯hk−1≤C⁡(Bϕ(σi)Λk−1+ϵ¯dual). \| _i,h^k,π\|_1, μ_h^k-1≤ C (B_φ( _i) k-1+ ε_dual ). (191) Substituting (191) into (190) and then into (189) gives ∑j=1k−1μhj,π[(Δi,hk,π)2] _j=1^k-1E_ _h^j,π [( _i,h^k,π)^2 ] ≤C′Bϕ(σi)(k−1)C(Bϕ(σi)Λk−1+ϵ¯dual) ≤ C B_φ( _i)\,(k-1)\,C (B_φ( _i) k-1+ ε_dual ) =CC′Bϕ(σi)2(k−1)Λ+CC′(k−1)Bϕ(σi)ϵ¯dual. =C B_φ( _i)^2 (k-1) +C (k-1)B_φ( _i) ε_dual. (192) Summing (192) over h∈[H]h∈[H], and using k−1≤Kk-1≤ K, we obtain ∑h=1H∑j=1k−1μhj,π[(Δi,hk,π)2]≤⁡(HBϕ(σi)2KΛ+HKBϕ(σi)ϵ¯dual). _h=1^H _j=1^k-1E_ _h^j,π [( _i,h^k,π)^2 ] \! (HB_φ( _i)^2 K +HKB_φ( _i) ε_dual ). (193) Choosing ϵ¯dual≤εdualapproxHKBϕ(σi) ε_dual≤ _dual^approxHKB_φ( _i) and substituting (193) into (188), we get ^ik−1(fi⋆,g^ifi⋆,π,π)≤⁡(HBϕ(σi)2[KΛ+Λ]+εdualapprox). K_i^k-1(f_i ; g_i^\,f_i ,π;π) \! (HB_φ( _i)^2 [ K + ]+ _dual^approx ). Under the standing hypothesis K≥ΛiK≥ _i (with Λ=Λi = _i), we have Λ≤KΛ ≤ K , so the additive logarithmic term is absorbed into the leading square-root term, giving ^ik−1(fi⋆,g^ifi⋆,π,π)≤⁡(HBϕ(σi)2KΛ+εdualapprox). K_i^k-1(f_i ; g_i^\,f_i ,π;π) \! (HB_φ( _i)^2 K + _dual^approx ). This proves the lemma. ∎ Lemma 7 (Benchmark-to-surrogate comparison). Assume Assumptions 1–3, the robust-to-nominal concentrability condition (Definition 7, constant CcovC_cov), and the uniform dual-switch stability event used in the proof hold. Let Λi:=log⁡(|ℱi‖i‖Πpur|KHmδ) _i:= \! ( |F_i||G_i|| ^pur|KHmδ ), and assume K≥ΛiK≥ _i. Fix any episode k∈[K]k∈[K], agent i∈ℳi , and pure joint policy π∈Πpurπ∈ ^pur. Let fi⋆:=fiπ,σi:=fi,hπ,σih=1H+1∈ℱif_i :=f_i^π, _i:=\f_i,h^π, _i\_h=1^H+1 _i be the robust Bellman-consistent benchmark sequence from Lemma 6 satisfying fi,h⋆=i,hπ,ϕ,σifi,h+1⋆,h∈[H],fi,H+1⋆≡0.f_i,h =T_i,h^π,φ, _if_i,h+1 , h∈[H], f_i,H+1 ≡ 0. Then, with probability at least 1−δ1-δ, uniformly over all (i,k,π)(i,k,π), Vi,1π,σi(s1k)−U^ik(π)≤ V_i,1^π, _i(s_1^k)- U_i^k(π)≤ β^ik−1(fi⋆,g^ifi⋆,π,π)+⁡(βHBϕ(σi)2KΛiCLOSE \;β K_i^k-1 (f_i ; g_i^f_i ,π;π )+O\! (β HB_φ( _i)^2 K _i OPEN+CcovHBϕ(σi)Λi|hk−1|∨1+εdualapprox). +C_covHB_φ( _i) _i|D_h^k-1| 1+ _dual^approx ). (194) Proof. Fix k∈[K]k∈[K], i∈ℳi , and π∈Πpurπ∈ ^pur, and write Λ:=Λi := _i. Let fi⋆:=fiπ,σi=fi,h⋆h=1H+1f_i :=f_i^π, _i=\f_i,h \_h=1^H+1 be the robust Bellman-consistent benchmark sequence: fi,h⋆=i,hπ,ϕ,σifi,h+1⋆,h∈[H],fi,H+1⋆≡0.f_i,h =T_i,h^π,φ, _if_i,h+1 , h∈[H], f_i,H+1 ≡ 0. (195) Since π is pure, π[U^ik]=U^ik(π).E_π[ U_i^k]= U_i^k(π). Step 1: Apply robust payoff domination. By Lemma 4, applied to the fixed policy π, we have Vi,1π,σi(s1k) V_i,1^π, _i(s_1^k) ≤U^ik(π)+β^ik−1(fik,π,g^ik,π,π)+ℛi,kν(π), ≤ U_i^k(π)+β K_i^k-1(f_i^k,π; g_i^k,π;π)+R_i,k^ν(π), (196) where ℛi,kν(π):= _i,k^ν(π):= ∑h=1H‖i,hπ,ϕ,σifi,h+1k,π−g^i,hk,π,i,hπ,ϕ,σifi,h+1k,π‖1,νi,hπ+∑h=1Hνi,hπ[(g^i,hk,π,i,hπ,ϕ,σifi,h+1k,π−fi,hk,π)+]. _h=1^H \|T_i,h^π,φ, _if_i,h+1^k,π-T_ g_i,h^k,π,i,h^π,φ, _if_i,h+1^k,π \|_1, _i,h^π+ _h=1^HE_ _i,h^π [ (T_ g_i,h^k,π,i,h^π,φ, _if_i,h+1^k,π-f_i,h^k,π )_+ ]. (197) Subtracting U^ik(π) U_i^k(π) gives Vi,1π,σi(s1k)−U^ik(π)≤β^ik−1(fik,π,g^ik,π,π)+ℛi,kν(π). V_i,1^π, _i(s_1^k)- U_i^k(π)≤β K_i^k-1(f_i^k,π; g_i^k,π;π)+R_i,k^ν(π). (198) Step 2: Compare the fitted discrepancy to the benchmark discrepancy. For each h∈[H]h∈[H], define the aggregate empirical loss L^i,hk−1(fi,gi,π):=∑j=1k−1δ^i,h(fi,gi,h,π,shj,hj,sh+1j)2. L_i,h^k-1(f_i;g_i,π):= _j=1^k-1 δ_i,h (f_i;g_i,h;π;s_h^j, a_h^j,s_h+1^j )^2. By the revised cumulative definition, ^ik−1(fi,gi,π)=∑h=1H[L^i,hk−1(fi,gi,π)−inffi′∈ℱiL^i,hk−1(fi′,gi,π)]. K_i^k-1(f_i;g_i;π)= _h=1^H [ L_i,h^k-1(f_i;g_i,π)- _f _i _i L_i,h^k-1(f _i;g_i,π) ]. For fixed g^ik,π g_i^k,π, the fitted sequence fik,πf_i^k,π is the empirical Bellman fit. Hence, by empirical optimality, ^ik−1(fik,π,g^ik,π,π)≤^ik−1(fi⋆,g^ik,π,π). K_i^k-1(f_i^k,π; g_i^k,π;π)≤ K_i^k-1(f_i ; g_i^k,π;π). (199) The right-hand side still uses the dual minimizer associated with the fitted sequence. We therefore compare it with the benchmark dual g^ifi⋆,π g_i^f_i ,π. Define the dual-switch residual i,kdual(π):=[^ik−1(fi⋆,g^ik,π,π)−^ik−1(fi⋆,g^ifi⋆,π,π)]+. S_i,k^dual(π):= [ K_i^k-1(f_i ; g_i^k,π;π)- K_i^k-1(f_i ; g_i^f_i ,π;π) ]_+. (200) Then ^ik−1(fi⋆,g^ik,π,π)≤^ik−1(fi⋆,g^ifi⋆,π,π)+i,kdual(π). K_i^k-1(f_i ; g_i^k,π;π)≤ K_i^k-1(f_i ; g_i^f_i ,π;π)+ S_i,k^dual(π). (201) Combining (199) and (201), we obtain ^ik−1(fik,π,g^ik,π,π)≤^ik−1(fi⋆,g^ifi⋆,π,π)+i,kdual(π). K_i^k-1(f_i^k,π; g_i^k,π;π)≤ K_i^k-1(f_i ; g_i^f_i ,π;π)+ S_i,k^dual(π). (202) On the uniform dual-switch stability event, under K≥ΛK≥ the residual satisfies i,kdual(π)≤⁡(HBϕ(σi)2KΛ+HKBϕ(σi)ϵ¯dual1). S_i,k^dual(π) \! (HB_φ( _i)^2 K +HKB_φ( _i) ε_dual^1 ). (203) This is the step that explicitly accounts for the fact that g^ik,π g_i^k,π is optimized for fik,πf_i^k,π, whereas the benchmark comparison uses fi⋆f_i ; the K≥ΛK≥ requirement matches the one needed in Lemma 6 to absorb the additive Freedman term into the leading KΛ K rate. Step 3: Control the robust-occupancy residual. Unlike Lemma 5 and Lemma 6, the residual ℛi,kν(π)R_i,k^ν(π) is evaluated under the worst-case occupancy νi,hπ _i,h^π of the current evaluation policy π, not under a sum or mixture over the historical policies π1,…,πk−1π^1,…,π^k-1. Since π need not coincide with any of the past behavior policies, there is no historical data directly sampled from νi,hπ _i,h^π (or even from its nominal counterpart μhπ _h^π); the bridge to the historical dataset hk−1D_h^k-1 is instead supplied by the robust-to-nominal concentrability condition (Definition 7): there exists Ccov≥1C_cov≥ 1 such that, for every measurable u, ‖u‖1,νi,hπ≤Ccov‖u‖1,μ¯hk−1,μ¯hk−1:=1k−1∑j=1k−1μhπj. \|u\|_1, _i,h^π≤ C_cov\|u\|_1, μ_h^k-1, μ_h^k-1:= 1k-1 _j=1^k-1 _h^π^j. (204) Applying (204) with u=i,hπ,ϕ,σifi,h+1k,π−g^i,hk,π,i,hπ,ϕ,σifi,h+1k,πu=T_i,h^π,φ, _if_i,h+1^k,π-T_ g_i,h^k,π,i,h^π,φ, _if_i,h+1^k,π, ‖i,hπ,ϕ,σifi,h+1k,π−g^i,hk,π,i,hπ,ϕ,σifi,h+1k,π‖1,νi,hπ≤Ccov‖i,hπ,ϕ,σifi,h+1k,π−g^i,hk,π,i,hπ,ϕ,σifi,h+1k,π‖1,μ¯hk−1. \|T_i,h^π,φ, _if_i,h+1^k,π-T_ g_i,h^k,π,i,h^π,φ, _if_i,h+1^k,π \|_1, _i,h^π≤ C_cov \|T_i,h^π,φ, _if_i,h+1^k,π-T_ g_i,h^k,π,i,h^π,φ, _if_i,h+1^k,π \|_1, μ_h^k-1. (205) Since μ¯hk−1 μ_h^k-1 is exactly the historical mixture measure that hk−1D_h^k-1 (of size k−1k-1) is drawn from, applying Lemma 1 with n=k−1n=k-1 gives, with probability at least 1−δ1-δ, simultaneously over i,h,π,ki,h,π,k, ‖i,hπ,ϕ,σifi,h+1k,π−g^i,hk,π,i,hπ,ϕ,σifi,h+1k,π‖1,μ¯hk−1≤C⁡(Bϕ(σi)Λ|hk−1|∨1+ϵ¯dual2). \|T_i,h^π,φ, _if_i,h+1^k,π-T_ g_i,h^k,π,i,h^π,φ, _if_i,h+1^k,π \|_1, μ_h^k-1≤ C (B_φ( _i) |D_h^k-1| 1+ ε_dual^2 ). (206) Combining (205) and (206) and summing over h∈[H]h∈[H], ∑h=1H‖i,hπ,ϕ,σifi,h+1k,π−g^i,hk,π,i,hπ,ϕ,σifi,h+1k,π‖1,νi,hπ≤CCcovHBϕ(σi)Λ|hk−1|∨1+CCcovHϵ¯dual2. _h=1^H \|T_i,h^π,φ, _if_i,h+1^k,π-T_ g_i,h^k,π,i,h^π,φ, _if_i,h+1^k,π \|_1, _i,h^π≤ C_covHB_φ( _i) |D_h^k-1| 1+C_covH ε_dual^2. (207) Similarly, since (x)+≤|x|(x)_+≤|x|, the same coverage transfer (204) applied to u=g^i,hk,π,i,hπ,ϕ,σifi,h+1k,π−fi,hk,πu=T_ g_i,h^k,π,i,h^π,φ, _if_i,h+1^k,π-f_i,h^k,π, together with the empirical Bellman-fit residual bound entering Lemma 1’s proof, yields ∑h=1Hνi,hπ[(g^i,hk,π,i,hπ,ϕ,σifi,h+1k,π−fi,hk,π)+]≤CCcovHBϕ(σi)Λ|hk−1|∨1+CCcovHϵ¯dual2. _h=1^HE_ _i,h^π [ (T_ g_i,h^k,π,i,h^π,φ, _if_i,h+1^k,π-f_i,h^k,π )_+ ]≤ C_covHB_φ( _i) |D_h^k-1| 1+C_covH ε_dual^2. (208) Combining (207) and (208), we get ℛi,kν(π)≤CCcovHBϕ(σi)Λ|hk−1|∨1+CCcovHϵ¯dual2. _i,k^ν(π)≤ C_covHB_φ( _i) |D_h^k-1| 1+C_covH ε_dual^2. (209) Step 4: Combine the bounds. Substituting (202), (203), and (209) into (198), we obtain Vi,1π,σi(s1k)−U^ik(π)≤β^ik−1(fi⋆,g^ifi⋆,π,π)+⁡(βHBϕ(σi)2KΛCLOSE V_i,1^π, _i(s_1^k)- U_i^k(π)≤\;β K_i^k-1(f_i ; g_i^f_i ,π;π)+O\! (β HB_φ( _i)^2 K OPEN+βHKBϕ(σi)ϵ¯dual1+CcovHBϕ(σi)Λ|hk−1|∨1+CcovHϵ¯dual2). +β HKB_φ( _i) ε_dual^1+C_covHB_φ( _i) |D_h^k-1| 1+C_covH ε_dual^2 ). (210) Finally, choose ϵ¯dual1≤εdualapproxβHKBϕ(σi),ϵ¯dual2≤εdualapproxCcovH. ε_dual^1≤ _dual^approxβ HKB_φ( _i), ε_dual^2≤ _dual^approxC_covH. Substituting these choices into (210) gives (194). Since fi⋆=fiπ,σif_i =f_i^π, _i, the proof is complete. ∎ Appendix I Covering and Bracketing Numbers. When the function class ℱF is not finite, its cardinality can no longer be used directly in complexity bounds. In such cases, two standard substitutes are the δ-covering number and the δ-bracketing number. These notions are widely used in statistical learning theory to quantify the size of infinite function classes. Definition 13 (δ-Covering Number). For a function class ℱF equipped with a metric d, the δ-covering number of ℱF, denoted by ℱ(δ,d)N_F(δ,d), is the smallest integer m such that there exists a subset ℱ′=f(1),…,f(m)⊆ℱF =\f^(1),…,f^(m)\ with the property that every f∈ℱf is within distance δ of at least one element of ℱ′F , i.e., ∀f∈ℱ,∃f′∈ℱ′ such that d(f,f′)≤δ.∀ f , ∃\,f such that d(f,f )≤δ. When the metric is the supremum norm, we abbreviate ℱ(δ,∥⋅∥∞)asℱ(δ).N_F(δ,\|·\|_∞) _F(δ). Definition 14 (δ-Bracketing Number). A δ-bracket for the class ℱF is a finite collection of pairs (g1i,g2i)i=1N,\(g_1^i,g_2^i)\_i=1^N, where each g1ig_1^i and g2ig_2^i is a real-valued function of a policy π and a trajectory τ, such that the following two conditions hold: 1. For every i∈[N]i∈[N] and every π∈Ππ∈ , ‖g1i(π,⋅)−g2i(π,⋅)‖≤δ.\|g_1^i(π,·)-g_2^i(π,·)\|≤δ. 2. For every f∈ℱf , there exists some index i∈[N]i∈[N] such that g1i(π,τH)≤ℙfπ(τH)≤g2i(π,τH)g_1^i(π, _H) _f^π( _H)≤ g_2^i(π, _H) for all policies π and all admissible trajectories τH _H. The δ-bracketing number of ℱF, written as ℬℱ(δ)B_F(δ), is defined as the minimum number N for which such a δ-bracket exists. Appendix J Technical Lemmas Lemma T.1 (Integral interchange principle; cf. [59], Theorem 14.60). Let (Ω,,μ)( ,A,μ) be a σ-finite measure space, and let X be a collection of measurable functions x:Ω→ℝx: that is decomposable. Suppose f:Ω×ℝ→ℝf: ×R is a finite-valued normal integrand. Then infx∈∫Ωf⁡(ω,x⁡(ω))μ(ω)=∫Ωinfu∈ℝf⁡(ω,u)μ(ω). _x _ f(ω,x(ω))\,μ(dω)= _ _u f(ω,u)\,μ(dω). In addition, provided the common infimum above is greater than −∞-∞, a function x⋆∈x is an optimizer of the integral problem, that is, x⋆∈arg⁡min⁡∫Ωx∈⁡f⁡(ω,x⁡(ω))μ(ω),x ∈ _x _ f(ω,x(ω))\,μ(dω), if and only if it solves the pointwise minimization problem μ-almost everywhere: x⋆(ω)∈arg⁡minu∈ℝ⁡f⁡(ω,u)for μ-a.e. ω∈Ω.x (ω)∈ _u f(ω,u) $μ$-a.e.\ ω∈ . We recall a standard performance guarantee for empirical risk minimization (ERM), adapted from classical statistical learning results (see, e.g., [61] and [54]). Lemma T.2 (Generalization bound for ERM). Let (,P)(X,P) be a probability space, and let ℱF be a class of real-valued functions defined on X. Consider a loss function ℓ:ℱ×→ℝ :F×X satisfying the uniform boundedness condition |ℓ⁡(f,x)|≤B,∀f∈ℱ,x∈,| (f,x)|≤ B, ∀\,f ,\;x , for some constant B>0B>0. Given an i.i.d. sample =x1,…,xND=\x_1,…,x_N\ drawn from P, define the empirical minimizer f^∈arg⁡minf∈ℱ1N∑i=1Nℓ⁡(f,xi), f∈ _f 1N _i=1^N (f,x_i), and let f⋆∈arg⁡minf∈ℱx∼P[ℓ⁡(f,x)]f ∈ _f E_x P[ (f,x)] denote a minimizer of the expected loss. Then, for any confidence level δ∈(0,1)δ∈(0,1), with probability at least 1−δ1-δ, the excess risk of f f satisfies x∼P[ℓ⁡(f^,x)]−x∼P[ℓ⁡(f⋆,x)]≤2ℜN(ℓ∘ℱ)+5B2log⁡(8/δ)N, _x P[ ( f,x)]-E_x P[ (f ,x)]≤ 2\, R_N( )+5B 2 (8/δ)N, (211) where ℜN(ℓ∘ℱ) R_N( ) denotes the empirical Rademacher complexity of the function class ℓ∘ℱ , given by ℜN(ℓ∘ℱ)=1Nεii=1N[supg∈ℓ∘ℱ∑i=1Nεig(xi)], R_N( )= 1NE_\ _i\_i=1^N [ _g∈ _i=1^N _ig(x_i) ], with εii=1N\ _i\_i=1^N being independent Rademacher variables taking values in ±1\± 1\ with equal probability. In addition, if ℱF is a finite class and the following conditions hold: |f⁡(x)|≤M∀f∈ℱ,x∈,andℓ⁡(f,x) is L-Lipschitz in f, |f(x)|≤ M ∀\,f ,\;x , (f,x) is $L$-Lipschitz in $f$, (212) then, with probability at least 1−δ1-δ, the bound simplifies to x∼P[ℓ⁡(f^,x)]−x∼P[ℓ⁡(f⋆,x)]≤2LM2log⁡|ℱ|N+5B2log⁡(8/δ)N. _x P[ ( f,x)]-E_x P[ (f ,x)]≤ 2LM 2 |F|N+5B 2 (8/δ)N. (213) Lemma T.3 (Freedman’s inequality (e.g., [3])). Let Mtt≤T\M_t\_t≤ T be a real-valued martingale difference sequence w.r.t. filtration t\G_t\ with |Mt|≤b|M_t|≤ b a.s. and let ST=∑t=1T⁡[Mt2∣t−1]S_T= _t=1^TE[M_t^2 _t-1]. Then for any δ∈(0,1)δ∈(0,1), Pr⁡(∑t=1TMt≥2STln⁡(1/δ)+b3ln⁡(1/δ))≤δ. ( _t=1^TM_t≥ 2S_T (1/δ)+ b3 (1/δ) )≤δ. Lemma T.4 ([78][Lemma B.1). ] For any positive sequence (x1,…,xm)(x_1,…,x_m), we have ∑j=1mxj∑j=1mjxj2≤1+log⁡m. _j=1^mx_j _j=1^mjx_j^2~≤~ 1+ m. Lemma T.5 (Elliptical Potential Inequality [78]). Consider a sequence of vectors xkk=1K⊂ℝd\x_k\_k=1^K ^d. Let ϵ>0ε>0 be a regularization parameter, and define the matrix sequence Σk:=ϵI+∑s=1k−1xsxs⊤,k=1,…,K+1. _k\;:=\;ε I+ _s=1^k-1x_sx_s , k=1,…,K+1. Then, the cumulative self-normalized squared norms satisfy ∑k=1Kmin⁡1,‖xk‖Σk−12≤ 2log⁡(det(ΣK+1)det(Σ1))≤ 2logdet(I+1ϵ∑k=1Kxkxk⊤). _k=1^K \! \1,\;\|x_k\|_ _k^-1^2 \\;≤\;2 \! ( ( _K+1) ( _1) )\;≤\;2 \! (I+ 1ε _k=1^Kx_kx_k ). Proof. This result follows from standard properties of self-normalized processes and the monotonicity of the log-determinant function applied to positive definite matrices. A complete argument can be found in classical analyses of regularized least-squares estimators and linear bandit methods. A detailed proof is given in [1, Lemma 11].∎

Instant research discovery