Paper deep dive
Quality-Aware Exploration Budget Allocation for Cooperative Multi-Agent Reinforcement Learning
Dahyun Oh, Minhyuk Yoon, H. Jin Kim
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 7/8/2026, 8:42:19 AM
Summary
The paper introduces a quality-aware exploration framework for cooperative multi-agent reinforcement learning that dynamically allocates exploration budgets based on intrinsic reward signal reliability. It integrates a global return-conditioned sigmoid schedule (RCB) to adapt exploration intensity, a per-agent reward signal quality (RSQ) metric to suppress noisy agents, and Successor Distance (SD) as an intrinsic reward mechanism that naturally differentiates signal quality. This approach prevents coordination collapse and achieves top-tier performance across multiple cooperative benchmarks.
Entities (10)
Relation Signals (9)
Return-Conditioned Beta (RCB) → adapts → Exploration Intensity
confidence 96% · RCB adapts β to team learning progress, substantially reducing the need for per-task manual tuning
Reward Signal Quality (RSQ) → measures → Signal Quality
confidence 95% · RSQ measures per-agent intrinsic reward reliability and maps it to modulation weights
Cooperative Multi-Agent Reinforcement Learning → uses → Intrinsic Motivation
confidence 95% · Intrinsic motivation, which augments task rewards with novelty bonuses, is a popular approach for driving exploration
Successor Distance (SD) → enables → Signal Quality Differentiation
confidence 94% · SD provides the geometrically structured intrinsic signal that makes per-agent quality differentiation possible
Reward Signal Quality (RSQ) → prevents → Coordination Collapse
confidence 93% · RSQ enables exploration intensities that would otherwise cause training collapse, by selectively attenuating agents with unreliable signals
Signal-to-Noise Ratio → determines → Exploration Allocation
confidence 92% · The signal-to-noise ratio (SNR) emerges as the natural quality metric, and the affine RSQ modulation preserves the allocation ordering
Proposed Framework → evaluatedon → MPE
confidence 90% · On seven cooperative benchmarks (MPE, SMAX, MABrax), our method achieves top-tier returns across all environments.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Cooperative multi-agent reinforcement learning (MARL) requires agents to discover joint strategies in a combinatorially large state-action space, yet effective coordination configurations are exceedingly rare. Intrinsic motivation, which augments task rewards with novelty bonuses, is a popular approach for driving exploration, but its effectiveness hinges on the exploration intensity $\beta$, where too large a value overwhelms the task signal and causes coordination collapse, while too small a value prevents discovery of rare strategies. We address two complementary challenges: adapting $\beta$ globally over training, and allocating the exploration budget across agents whose intrinsic reward signals vary in reliability. Our framework combines a return-conditioned sigmoid schedule (RCB) for global intensity control with a per-agent Reward Signal Quality (RSQ) metric that concentrates the exploration budget on agents with reliable signals. The core insight is that agents receiving noisy intrinsic rewards should explore less aggressively, and this allocation can be determined automatically from signal-to-noise statistics. Successor Distance (SD), a quasimetric intrinsic reward, naturally produces distinguishable per-agent signal quality, completing the framework with convergence and ordering preservation guarantees. On seven cooperative benchmarks (MPE, SMAX, MABrax), our method achieves top-tier returns across all environments.
Tags
Links
- Source: https://arxiv.org/abs/2605.01865v1
- Canonical: https://arxiv.org/abs/2605.01865v1
PDF not stored locally. Use the link above to view on the source site.
Full Text
81,911 characters extracted from source content.
Expand or collapse full text
Quality-Aware Exploration Budget Allocation for Cooperative Multi-Agent Reinforcement Learning Dahyun Oh qlass33@snu.ac.kr Minhyuk Yoon likenuclear@snu.ac.kr H. Jin Kim hjinkim@snu.ac.kr Abstract Cooperative multi-agent reinforcement learning (MARL) requires agents to discover joint strategies in a combinatorially large state-action space, yet effective coordination configurations are exceedingly rare. Intrinsic motivation, which augments task rewards with novelty bonuses, is a popular approach for driving exploration, but its effectiveness hinges on the exploration intensity β, where too large a value overwhelms the task signal and causes coordination collapse, while too small a value prevents discovery of rare strategies. We address two complementary challenges: adapting β globally over training, and allocating the exploration budget across agents whose intrinsic reward signals vary in reliability. Our framework combines a return-conditioned sigmoid schedule (RCB) for global intensity control with a per-agent Reward Signal Quality (RSQ) metric that concentrates the exploration budget on agents with reliable signals. The core insight is that agents receiving noisy intrinsic rewards should explore less aggressively, and this allocation can be determined automatically from signal-to-noise statistics. Successor Distance (SD), a quasimetric intrinsic reward, naturally produces distinguishable per-agent signal quality, completing the framework with convergence and ordering preservation guarantees. On seven cooperative benchmarks (MPE, SMAX, MABrax), our method achieves top-tier returns across all environments. keywords: multi-agent reinforcement learning , exploration scheduling , signal-to-noise ratio , intrinsic motivation , reward signal quality †journal: Neurocomputing [1]organization=Department of Aerospace Engineering, Seoul National University, city=Seoul, country=Republic of Korea 1 Introduction Cooperative multi-agent reinforcement learning (MARL) trains teams of autonomous agents to jointly solve shared tasks through trial and error [1]. A fundamental challenge is exploration: the joint state-action space grows combinatorially with the number of agents, yet configurations yielding effective coordination are exceedingly rare [2, 3]. Without sufficient exploration, agents converge to suboptimal behaviors because they never discover the coordinated strategies that produce high team returns. Figure 1: (a) Allocating exploration budget without considering signal quality lets noisy agents destabilize coordination. (b) Our framework adapts globally via RCB and per-agent via RSQ, concentrating the exploration budget on agents with reliable signals, which yields stronger and more stable returns than the tested baselines. A wide variety of exploration-enhancing techniques have been proposed for cooperative MARL [4], including shared latent variables for coordinated behavior [5], constrained optimization of exploration budgets [6], and dual policy heads [7], among many others [8, 9, 10, 11]. Among these diverse approaches, augmenting the task reward with intrinsic motivation bonuses has become the most widely adopted paradigm due to its generality [12, 13, 14, 15]. In the intrinsic motivation framework, the extrinsic reward rextr^ext (the task-specific signal from the environment, e.g., reaching a goal or capturing a target) is supplemented with an intrinsic reward riintr_i^int that each agent generates internally to quantify the novelty of its experience, yielding the augmented reward ri=rext+β⋅riintr_i=r^ext+β· r_i^int for each agent i. The exploration intensity β>0β>0 governs how strongly exploration influences learning, and setting it correctly is the central difficulty. Existing methods either fix β to a single value that must be tuned per task [5, 8, 9, 7] or adapt per-agent budgets based on task performance constraints [6] without considering whether each agent’s intrinsic reward is actually reliable. Even when the exploration budget is well-allocated for task feasibility, a subtler problem remains: different agents produce intrinsic rewards of different quality, and an agent whose novelty signal is dominated by noise can destabilize coordination regardless of how much budget it receives. Moreover, detecting these quality differences requires that the intrinsic reward itself produce signals of varying reliability across agents, which depends on how the intrinsic reward is computed (Sec. 4.3). We identify two fundamental challenges that current methods leave unresolved. First, the optimal β is task-dependent, and tasks that demand extensive exploration (e.g., navigating physical bottlenecks) require large β, while tasks where coordination emerges easily need small β to avoid overwhelming the task signal. Manually tuning β is impractical, especially since small changes can cause catastrophic training failures. Second, within a single task, different agents experience different intrinsic reward dynamics. An agent whose intrinsic reward signal is reliable can explore productively, while an agent with noisy or inconsistent intrinsic motivation risks destabilizing coordination through uncontrolled exploration. As illustrated in Fig. 1(a), allocating exploration budget without accounting for intrinsic reward quality allows agents with noisy exploration signals to disrupt team coordination. To address these challenges, we propose an integrated exploration framework whose three components each address a distinct requirement (Fig. 1(b)). For global scheduling, we derive a Return-Conditioned β (RCB) schedule, sigmoidal in team return, with provable convergence that adapts the exploration intensity to team learning progress. For per-agent allocation, we introduce the Reward Signal Quality (RSQ) metric, a computationally efficient affine modulation based on signal-to-noise ratio that preserves the allocation ordering of the information-optimal solution. For RSQ to differentiate agents, the intrinsic reward must produce signals whose reliability naturally varies across agents. Successor Distance (SD) [16], which measures how far apart two states are in terms of the time it takes to travel between them, satisfies this condition. An agent that can explore on its own produces a consistent novelty signal across episodes, while an agent whose experience depends on what teammates happen to do produces a noisy, fluctuating signal. SD naturally reflects this difference, and RSQ detects it automatically. The three components address complementary situations. RCB actively modulates global exploration on tasks with wide return ranges, RSQ prevents training collapse by attenuating noisy agents on large-scale tasks where return variation is narrow, and SD provides the geometrically structured intrinsic signal that makes per-agent quality differentiation possible. Together, they achieve the highest or statistically tied return on all seven benchmarks. Our contributions are as follows: 1. A Return-Conditioned β (RCB) for global exploration intensity control with provable convergence under a contraction condition. RCB adapts β to team learning progress, substantially reducing the need for per-task manual tuning (Sec. 4.1). 2. A Reward Signal Quality (RSQ) metric for per-agent exploration budget allocation, motivated by information-theoretic analysis. The signal-to-noise ratio (SNR) emerges as the natural quality metric, and the affine RSQ modulation preserves the allocation ordering of the information-optimal solution. The key design guarantee is that RSQ enables exploration intensities that would otherwise cause training collapse, by selectively attenuating agents with unreliable signals (Sec. 4.2). 3. A complete framework that integrates RCB, RSQ, and Successor Distance (SD) into a unified system. SD’s quasimetric structure naturally produces distinguishable per-agent signal quality, a necessary condition for RSQ to differentiate agents. Replacing SD with entropy, RND, or count-based rewards eliminates this differentiation and causes training collapse (Sec. 4.3). 4. Experiments on seven cooperative benchmarks (two discrete MPE tasks, two SMAX combat tasks with 8 and 27 agents, three continuous MABrax tasks) with 10 seeds per configuration. Our method achieves top-tier returns across all environments. The source code is available at https://github.com/DH-O/RRS. 2 Related Work 2.1 Exploration in Single-Agent Reinforcement Learning Exploration in reinforcement learning has evolved from simple randomization to structured novelty-seeking. Classical approaches such as ϵε-greedy [17] and optimistic initialization [18] add undirected randomness, while UCB methods [19] and posterior sampling [20] provide principled uncertainty-driven exploration. In function-approximation settings, count-based methods [21] generalize visitation counts via density models, ICM [12] uses dynamics prediction error as a novelty proxy, RND [13] measures novelty through distillation error from a random target network, and random curiosity modules [22] stabilize intrinsic rewards against catastrophic forgetting. Maximum entropy RL [23] encourages broad action distributions by incorporating policy entropy into the objective. These methods are designed for a single learner and do not account for multi-agent coordination, where one agent’s exploration can disrupt another’s learning [24]. 2.2 Exploration Paradigms in Cooperative MARL Exploration in cooperative MARL has been approached through diverse paradigms. Latent-variable methods [5] condition agents on shared variables to produce diverse team behaviors, episodic memory [15] provides joint-state novelty signals, diversity maximization [9] encourages behavioral variety via mutual information objectives, learned communication protocols [8, 25, 26] reduce partial observability through information sharing, and mutual-information-based teammate modeling [27] optimizes informative inter-agent communication. Maximum-entropy methods such as HASAC [28] encourage exploration via policy entropy regularization rather than reward augmentation, while COIN [7] maintains separate exploration and exploitation policy heads to decouple the two objectives. Among these paradigms, augmenting the task reward with per-agent intrinsic motivation bonuses (ri=rext+β⋅riintr_i=r^ext+β· r_i^int) has emerged as the dominant framework due to its generality and compatibility with standard policy gradient architectures. Within this additive framework, various novelty signals have been proposed. Social influence [14] and influence-based exploration [24] reward agents for affecting teammates, while coordinated prediction errors [29] and individualized intrinsic rewards [11] align bonuses with the team objective. Recent extensions include trajectory entropy maximization [30], curiosity calibration via information bottleneck [31], and sequential entropy accumulation [32]. Independently, Nimonkar et al. [33] show that contrastive distance functions naturally produce cooperative exploration, supporting the use of quasimetric distances as a basis for multi-agent intrinsic rewards. Despite this progress, the form of exploration (what novelty signal to use) has received far more attention than the intensity of exploration (how much to explore). Nearly all additive methods above apply a fixed β to every agent throughout training without considering whether each agent’s intrinsic reward is reliable, leaving two questions unresolved: how should β adapt over the course of learning, and how should the exploration budget be distributed across agents with different intrinsic reward quality? Our work addresses both questions, providing a global schedule that adapts to team learning progress and a per-agent allocation based on intrinsic reward signal quality. 2.3 Adaptive Exploration and Constrained Approaches Adaptive methods tune exploration parameters online. In single-agent RL, meta-gradient methods [34] and self-tuning actor-critics [35] adjust entropy coefficients automatically, SASR [36] adapts reward shaping via success-rate-driven Beta distributions, and maximum entropy exploration methods [37] separate target and explorer policies to handle deceptive rewards. In multi-agent settings, ADER [10] learns per-agent entropy coefficients via value factorization. ADER is the closest prior work to per-agent exploration budget allocation and shares our motivation of agent-specific adaptation. However, it differs in three key aspects: (i) it modulates policy entropy rather than intrinsic reward intensity, which does not distinguish agents by intrinsic signal quality, (i) it lacks an explicit optimality criterion for the resulting per-agent allocation, and (i) it requires QMIX-style value decomposition (factoring the joint Q-value into per-agent components), making it incompatible with the MAPPO/IPPO architecture used by our baselines and most recent cooperative MARL work [38, 39]. EIPO [6] treats intrinsic rewards as constraints rather than additive bonuses, preventing the exploration signal from overwhelming the task objective, and CIM [40] extends this line with constrained policy optimization for intrinsic reward coefficient tuning. These constrained approaches address whether to explore (safe vs. unsafe) but not how to allocate the exploration budget across agents whose signal quality varies. Our framework fills this gap. RCB provides global scheduling with provable convergence, while RSQ allocates exploration budgets across agents in proportion to their signal quality. 2.4 Signal Quality and Information-Theoretic Perspectives The signal-to-noise ratio (SNR) is a fundamental measure of information reliability, quantifying how much useful signal a measurement contains relative to background noise [41]. In reinforcement learning, policy gradient estimates are inherently noisy, and their SNR determines how reliably the gradient points toward improvement. Roberts and Tedrake [42] and Kuba et al. [43] analyze the SNR and variance of policy gradients as diagnostic tools, and noisy value estimates have been shown to degrade cooperative policies through overestimation [44], but these works do not use signal quality to modulate exploration. Han et al. [45] recently show that the noise-to-signal ratio of policy gradients increases sharply near optima, providing independent theoretical support for SNR-based modulation. From an information-theoretic perspective, information-directed sampling [46] connects exploration to mutual information by allocating effort proportional to information gain, and Hsu et al. [47] extend provably efficient randomized exploration to cooperative MARL with regret bounds, though without per-agent signal quality differentiation. Our work applies this reasoning at a different level: rather than designing the exploration mechanism, we optimize the allocation of exploration budget across agents by modeling each agent’s intrinsic reward as a noisy channel and applying SNR analysis to determine which agents can productively use their exploration budget [48]. 3 Background 3.1 Decentralized Partially Observable MDP We model cooperative MARL as a Dec-POMDP [49], defined by the tuple (,,i,T,R,Ωi,O,γ)(N,\,S,\,\A^i\,\,T,\,R,\,\ ^i\,\,O,\,γ). Here =1,…,nN=\1,…,n\ is the agent set, S is the state space, iA^i is agent i’s action space, T:×→Δ()T ×A→ (S) is the transition function, R:×→ℝR ×A is the shared team reward, Ωi ^i is agent i’s observation space, O is the observation function, and γ∈[0,1)γ∈[0,1) is the discount factor. We denote by xti∈ℝdxx_t^i ^d_x the local state features extracted from agent i’s observation otio_t^i (e.g., spatial coordinates or joint velocities), which serve as input to per-agent Successor Distance encoders. 3.2 Intrinsic Motivation for Exploration Intrinsic motivation supplements the task reward with exploration bonuses [21, 12, 13], yielding the augmented per-agent reward ri=rext+β⋅riint,r_i\;=\;r^ext+β· r_i^int, (1) where riintr_i^int is agent i’s intrinsic reward. The exploration intensity β governs the trade-off between task performance and state-space coverage, and any β>0β>0 injects exploration pressure into the policy gradient. The central challenge is setting β, since too large a value overwhelms the task signal, too small a value yields insufficient coverage, and the optimal setting is both task-dependent and agent-dependent [6]. 3.3 Signal-to-Noise Ratio In communication theory [50, 48], the signal-to-noise ratio SNR=μ2/σ2SNR=μ^2/σ^2 quantifies signal reliability. Adaptive modulation adjusts transmission parameters based on channel SNR: high SNR permits aggressive transmission, low SNR requires conservative modulation [51]. We apply this principle to exploration management: high intrinsic reward signal quality permits aggressive exploration, low quality requires attenuated exploration. 3.4 Successor Distance and Quasimetric Structure Successor Distance (SD) [16] measures temporal reachability between states. Let pγπ(sf=y∣s0=x)p_γ^π(s_f=y s_0=x) denote the discounted successor measure, defined as (1−γ)∑t=0∞γtPrπ(st=y∣s0=x)(1-γ) _t=0^∞γ^t ^π(s_t=y s_0=x). Definition 1 (Successor Distance). dSDπ(x,y)=log(pγπ(sf=y∣s0=y)pγπ(sf=y∣s0=x)).d_SD^π(x,y)= ( p_γ^π(s_f=y s_0=y)p_γ^π(s_f=y s_0=x) ). (2) SD satisfies the axioms of a quasimetric [16], 1. identity, d(x,y)=0⇔x=yd(x,y)=0 x=y, 2. triangle inequality, d(x,z)≤d(x,y)+d(y,z)d(x,z)≤ d(x,y)+d(y,z), 3. non-negativity, d(x,y)≥0d(x,y)≥ 0, while allowing asymmetry d(x,y)≠d(y,x)d(x,y)≠ d(y,x) that captures directed reachability. This structure implies path-consistent exploration incentives that neither count-based nor RND methods provide. In multi-agent settings, each agent i maintains an encoder ϕi _i operating on local state features xti∈ℝdxx_t^i ^d_x extracted from otio_t^i, rather than the full observation. The intrinsic reward ri,tint=mink<tdϕi(xki,xti)r_i,t^int= _k<td_ _i(x_k^i,x_t^i) [52] encourages visits to states far from the agent’s trajectory history. 4 Methodology Figure 2: The two scheduling mechanisms of our framework. (a) RCB adjusts global β(k)β^(k) via a sigmoid of the team’s EMA return: high β early (explore), decaying as performance improves (exploit). (b) RSQ measures per-agent intrinsic reward reliability and maps it to modulation weights hih_i. Agents above RSQrefRSQ_ref receive amplified exploration (hi>1h_i>1), while noisy agents are suppressed (hi<1h_i<1). 4.1 Return-Conditioned Beta (RCB) Schedule A fixed β in Eq. (1) struggles in two distinct ways. Across tasks, dense-reward environments need only a small bonus to nudge exploration, whereas sparse-reward environments require a much larger one to drive coverage at all, so no single value works for both. Within a single task, agents benefit from aggressive exploration early in training when most of the state space is unfamiliar, but the same intensity becomes counterproductive later when the bonus starts to distract from refining a near-optimal policy. An ideal adaptive schedule must therefore satisfy three requirements, (1) the intensity must remain bounded, since unbounded β overwhelms the task signal while zero β stops exploration entirely, (2) it must decrease as team performance improves, so that agents shift from exploration to exploitation as they learn, and (3) the schedule must converge to a stable equilibrium rather than oscillating. We show that a sigmoid schedule driven by team return satisfies all three with provable convergence under a mild contraction condition. 4.1.1 Team Return Tracking and Sigmoid Schedule Throughout this section we use two exploration parameters that index the current training iteration k. The first, β(k)>0β^(k)>0, is simply the value of β used at iteration k, a global exploration intensity shared by all agents and updated once per iteration (formally introduced in Definition 2 below). The second, hi(k)>0h_i^(k)>0, is a per-agent multiplicative weight that scales β(k)β^(k) for agent i based on the quality of its intrinsic reward signal at iteration k (formally introduced in Definition 4 of Sec. 4.2). The augmented reward in this section can simply be read as ri=rext+β(k)hi(k)riintr_i=r^ext+β^(k)h_i^(k)r_i^int. Each training iteration k then consists of collecting a batch of rollouts across NenvsN_envs parallel environments (NstepsN_steps timesteps each), computing β(k)β^(k) and hi(k)h_i^(k) from the collected data, assembling the augmented reward, and performing a policy update. The exploration parameters are thus determined after rollout collection and applied retroactively to the same batch for advantage computation, while action selection during rollout uses only the current policy, not β. Let R(k)R^(k) denote the mean team return over the completed episodes at iteration k. Because per-iteration returns fluctuate noisily, especially in sparse-reward tasks, the schedule should react to a slowly varying estimate of training progress rather than the raw R(k)R^(k). We therefore track return progress via an exponential moving average (EMA), Rema(k)R_ema^(k), which collapses the noisy sequence R(k)\R^(k)\ into a smooth signal that the sigmoid in Definition 2 can act on, Rema(k+1)=αR⋅R(k)+(1−αR)⋅Rema(k),R_ema^(k+1)= _R· R^(k)+(1- _R)· R_ema^(k), (3) where αR∈(0,1) _R∈(0,1) is the smoothing constant and we initialize Rema(0)=0R_ema^(0)=0. The reference return level RtargetR_target centers the sigmoid transition, and its robustness properties are discussed in Remark 1. Definition 2 (Return-Conditioned Beta). The return-conditioned exploration intensity at iteration k is: β(k)=βmin+(βmax−βmin)⋅σ(κ(Rtarget−Rema(k))),β^(k)= _ +( _ - _ )·σ\! (κ(R_target-R_ema^(k)) ), (4) where σ(x)=1/(1+exp(−x))σ(x)=1/(1+ (-x)) is the logistic sigmoid, κ>0κ>0 controls transition sharpness, and βmin,βmax>0 _ , _ >0 are the intensity floor and ceiling. When Rema≪RtargetR_ema R_target, β≈βmaxβ≈ _ (explore aggressively), and when Rema≫RtargetR_ema R_target, β≈βminβ≈ _ (exploit). The sigmoid’s bounded range, monotonicity in RemaR_ema, and centering at RtargetR_target satisfy all three design requirements by construction. Remark 1 (Robustness of RtargetR_target and κ). RtargetR_target and κ are not precision-sensitive hyperparameters. Because the logistic sigmoid saturates smoothly, RtargetR_target only needs to lie within the range of returns achievable during training, and κ only needs to be small enough that the explore-to-exploit transition occurs gradually over that range. In practice, both can be set from coarse task-level information (e.g., approximate return scale), analogous to target entropy −log||- |A| in SAC [23]. We provide a formal derivation of the sigmoid’s transition bandwidth and an empirical sensitivity sweep in Sec. 5.3.3. 4.1.2 RCB Convergence Assumption 1 (Return Response). For a fixed exploration intensity β, suppose the policy is trained long enough to reach approximate equilibrium. Let R¯(β) R(β) denote the resulting steady-state expected team return. We assume R¯(β) R(β) is continuous, bounded in [Rmin,Rmax][R_ ,R_ ], and that the observed return at iteration k satisfies R(k)=R¯(β(k))+ξ(k)R^(k)= R(β^(k))+ξ^(k), where ξ(k)ξ^(k) is zero-mean noise with bounded variance σξ2 _ξ^2. This is a standard assumption in adaptive learning rate analysis [53]: the exploration parameter (β) changes slowly enough relative to the policy that the policy approximately converges before β updates again. The key idea behind the following theorem is that the RCB update forms a feedback loop (β→return→Rema→β → R_ema→β), and if each step of this loop shrinks differences (a contraction), then the loop converges to a unique equilibrium. Theorem 1 (RCB Schedule Convergence). Under Assumption 1, if the contraction condition κ(βmax−βmin)⋅supβ|R¯′(β)|⋅14<1κ( _ - _ )· _β| R (β)|· 14<1 (5) holds (the 1/41/4 comes from the maximum slope of the logistic sigmoid), then: (1) Uniqueness: the equilibrium (R∗,β∗)(R^*,β^*) is unique. (2) Deterministic convergence: the tracking error contracts geometrically, |Rema(k)−R∗|≤ρk|Rema(0)−R∗|,ρ=1−αR(1−LΦ)∈(0,1),|R_ema^(k)-R^*|≤ρ^k\,|R_ema^(0)-R^*|, ρ=1- _R(1-L_ )∈(0,1), (6) where LΦ≜supβ|Φ′(β)|L_ _β| (β)| is the Lipschitz constant of the RCB map Φ (derived in A). (3) Stochastic convergence: when observed returns are noisy (σξ2>0 _ξ^2>0), RemaR_ema satisfies [|Rema(k)−R∗|2]→O(αRσξ21−LΦ).E [|R_ema^(k)-R^*|^2 ]\;→\;O\! ( _R\, _ξ^21-L_ ). (7) Proof. The proof applies Banach’s contraction mapping theorem [54] to the composed map Φ(β)=βmin+(βmax−βmin)σ(κ(Rtarget−R¯(β))), (β)= _ +( _ - _ )\,σ\! (κ(R_target- R(β)) ), then bounds the stochastic residual via a linear recursion. See A for the proof sketch. ∎ Remark 2 (What Theorem 1 delivers). Theorem 1 differs from a generic contraction-mapping argument in two specific ways. First, the Lipschitz constant is obtained in closed form, LΦ=κ(βmax−βmin)supβ|R¯′(β)|/4L_ =κ( _ - _ ) _β| R (β)|/4, by combining the chain rule with the analytic slope bound σ′(⋅)≤1/4σ (·)≤ 1/4, rather than left as an abstract symbol. Second, part (3) is not a fixed-point result but an L2L^2 recursion, in the stochastic-approximation style of [53], that handles the noisy iteration R(k)=R¯(β(k))+ξ(k)R^(k)= R(β^(k))+ξ^(k). These two features are what allow Theorem 1 to deliver, beyond a generic Banach citation, a verifiable hyperparameter inequality on (κ,βmin,βmax)(κ, _ , _ ) that the practitioner can check before training and an explicit noise-floor bound O(αRσξ2/(1−LΦ))O( _R _ξ^2/(1-L_ )) tying the EMA smoothing constant αR _R to the asymptotic tracking error. In practice, the contraction condition is easily satisfied. With κ=0.01κ=0.01 for all experiments, the left-hand product of Eq. (5) stays well below 1, and we confirm this empirically in Sec. 5.3. 4.2 Reward Signal Quality (RSQ) RCB provides a global exploration intensity β(k)β^(k), and the remaining question is how to distribute this budget across n different agents. Each agent’s intrinsic reward contains both useful signal and noise, much like a communication channel. In communication theory, the classical water-filling solution [41, 48] distributes transmission power across parallel channels in proportion to their signal quality. We apply the same principle to exploration: agents with reliable intrinsic rewards (high SNR) receive more exploration budget, while noisy agents receive less. This analysis motivates the computationally efficient RSQ metric, a simple affine approximation that preserves the SNR-based ordering of the optimal allocation. 4.2.1 Multi-Agent Exploration as Information Maximization Under uniform allocation, all agents explore with the same intensity β(k)β^(k), regardless of whether their intrinsic reward reliably indicates productive exploration or is dominated by noise. A better strategy concentrates the budget on agents with high-quality signals (Fig. 2b). We formalize this intuition by modeling each agent i’s intrinsic reward as a noisy observation: a useful signal fif_i (e.g., how much novel territory the agent covers) plus zero-mean noise ϵi _i from stochastic transitions and other agents’ actions. The ratio of signal power to noise power defines a signal-to-noise ratio SNRiSNR_i for each agent (we make this precise in Sec. 4.2.2). To quantify the useful information in each agent’s intrinsic reward, we model each agent as a noisy channel. Among all distributions with a given mean and variance, the Gaussian carries the least mutual information [41]. Any allocation that is optimal under the Gaussian model therefore remains valid (though possibly suboptimal) for the true distribution, giving a conservative lower bound. Under this model, the mutual information between agent i’s exploration and its intrinsic reward is: Ii=12log(1+βi2⋅SNRi).I_i= 12 (1+ _i^2·SNR_i). (8) High IiI_i means the intrinsic reward reliably distinguishes informative actions from uninformative ones. The goal is to choose per-agent intensities βi _i that maximize the total information gain ∑iIi _iI_i subject to a budget constraint. Since the augmented reward is ri=rext+βiriintr_i=r^ext+ _ir_i^int, the variance of the intrinsic component in the policy gradient scales as βi2Var(riint) _i^2Var(r_i^int), making βi2 _i^2 the natural measure of each agent’s exploration impact. The total impact is ∑iβi2=n(β(k))2≜B _i _i^2=n(β^(k))^2 B. Maximizing ∑iIi _iI_i subject to ∑iβi2≤B _i _i^2≤ B is the classical power allocation problem, whose solution is the standard water-filling strategy [41, Ch. 10]. Identifying pi=βi2p_i= _i^2 as the allocated power for agent i, pi∗,WF=(ν−1SNRi)+,βi∗,WF=pi∗,WF,p_i^*,WF= (ν- 1SNR_i )^+, _i^*,WF= p_i^*,WF, (9) where (x)+=max(x,0)(x)^+= (x,0) and the water level ν is chosen so ∑ipi∗,WF=B _ip_i^*,WF=B. Agents with 1/SNRi≥ν1/SNR_i≥ν receive zero allocation. We subsequently express the per-agent intensity directly as βi=β⋅hi _i=β· h_i, so that pi=(βhi)2p_i=(β h_i)^2. 4.2.2 Reward Signal Quality: Computing SNR in Practice The water-filling allocation (9) requires a per-agent SNRiSNR_i to implement. We now make the abstract SNRiSNR_i concrete. At each training iteration, agent i produces Nsteps×NenvsN_steps× N_envs intrinsic reward samples across all timesteps and parallel environments. The batch mean reflects the consistent signal strength, while the batch variance captures unpredictable fluctuations. We track these via exponential moving averages (EMAs), which smooth out iteration-to-iteration noise: μi(k+1) _i^(k+1) =α⋅r¯iint,(k)+(1−α)⋅μi(k), =α· r_i^int,(k)+(1-α)· _i^(k), (10) (σi(k+1))2 ( _i^(k+1))^2 =α⋅Var^(riint,(k))+(1−α)⋅(σi(k))2, =α· Var(r_i^int,(k))+(1-α)·( _i^(k))^2, (11) where r¯iint,(k) r_i^int,(k) and Var^(riint,(k)) Var(r_i^int,(k)) are the batch mean and variance of agent i’s intrinsic reward at iteration k. The Gaussian assumption is justified by the central limit theorem. The signal-to-noise ratio is then SNRi=μi2/σi2SNR_i= _i^2/ _i^2, following the standard power decomposition [X2]=μX2+σX2E[X^2]= _X^2+ _X^2 [41]. Definition 3 (Reward Signal Quality). For agent i at iteration k: RSQi(k)=(μi(k))2(μi(k))2+(σi(k))2+ϵ,RSQ_i^(k)= ( _i^(k))^2( _i^(k))^2+( _i^(k))^2+ε, (12) where we apply the transform x↦x/(1+x)x x/(1+x) to SNRiSNR_i, mapping it to [0,1][0,1] so that RSQ=1RSQ=1 means a perfect signal. We set ϵ=10−8ε=10^-8 for numerical stability. By construction, RSQ is invariant to positive rescaling of intrinsic rewards, eliminating per-task calibration. 4.2.3 Per-Agent Exploration Intensity via Affine Modulation Definition 4 (Affine RSQ Modulation). The per-agent exploration intensity combines global RCB with individual RSQ: βi(k)=β(k)⋅h(RSQi(k)), _i^(k)=β^(k)· h(RSQ_i^(k)), (13) where the modulation function h:[0,1]→ℝ>0h:[0,1] _>0 is defined as: h(RSQi)=clip(1+λ(RSQi−RSQref),hmin,hmax),h(RSQ_i)=clip\! (1+λ(RSQ_i-RSQ_ref),\;h_ ,\;h_ ), (14) with λ>0λ>0 controlling sensitivity, RSQrefRSQ_ref the reference RSQ level, and [hmin,hmax][h_ ,h_ ] bounding the modulation range. The modulation function has three regimes: agents with RSQ above RSQrefRSQ_ref receive amplified exploration (h>1h>1), those at the reference level are unchanged (h=1h=1), and those with low RSQ are suppressed (h<1h<1). The clipping bounds [hmin,hmax][h_ ,h_ ] prevent extreme amplification or complete suppression. When all agents have similar intrinsic reward quality, hi≈1h_i≈ 1 for all, recovering pure RCB. The specific parameter values are listed in Table 1. Remark 3 (Default Values for RSQrefRSQ_ref, hminh_ , and hmaxh_ ). These three parameters have natural default values that require less per-task tuning than the remaining hyperparameters. The reference level RSQref=0.5RSQ_ref=0.5 corresponds to SNR=1SNR=1 (equal signal and noise power), the natural boundary between “reliable” and “noisy” agents. The upper bound hmax=2.0h_ =2.0 caps amplification at 2×2× the global intensity, preventing any single agent from dominating the exploration budget. The lower bound hmin=0.1h_ =0.1 ensures that even the noisiest agents retain at least 10%10\% of the global intensity, avoiding the irreversible suppression problem that afflicts exact water-filling (C). These values are shared across all seven environments (Table 1). 4.2.4 Why Affine Modulation Instead of Exact Water-Filling The water-filling solution (9) is information-theoretically optimal but has a practical weakness: it assigns βi=0 _i=0 to agents whose estimated SNR falls below the water level, permanently suppressing their exploration based on potentially noisy EMA estimates. The affine RSQ modulation (Definition 4) replaces exact water-filling with a single-line formula that clips allocations to [hmin,hmax][h_ ,h_ ], so that every agent retains a minimum exploration intensity. The following proposition confirms that this simple approximation allocates in exactly the same order as the optimal solution: agents with higher SNR always receive more exploration budget. C validates this design choice empirically: exact water-filling causes training collapse on corridor because noisy SNR estimates permanently suppress agents, while the affine modulation achieves stable coordination. Proposition 1 (Allocation Ordering Preservation). Let n agents have signal-to-noise ratios SNR1≥SNR2≥⋯≥SNRn>0SNR_1 _2≥·s _n>0. Then: (i) The water-filling allocation satisfies p1WF≥p2WF≥⋯≥pnWFp_1^WF≥ p_2^WF≥·s≥ p_n^WF. (i) The affine RSQ modulation satisfies h1≥h2≥⋯≥hnh_1≥ h_2≥·s≥ h_n, yielding per-agent intensities β1≥β2≥⋯≥βn _1≥ _2≥·s≥ _n. (i) For any pair (i,j)(i,j) with SNRi>SNRjSNR_i>SNR_j, if both modulation weights are unsaturated (hmin<hj≤hi<hmaxh_ <h_j≤ h_i<h_ ), then the ordering is strict: hi>hjh_i>h_j. Proof sketch. The composition of the strictly increasing RSQ(SNR)=SNR/(1+SNR)RSQ(SNR)=SNR/(1+SNR) with the non-decreasing clipped affine h is non-decreasing in SNRSNR, with strict increase whenever hi,hj∈(hmin,hmax)h_i,h_j∈(h_ ,h_ ). Part (i) inherits the same monotonicity from (9), since 1/SNRi1/SNR_i is decreasing in SNRiSNR_i. ∎ Empirically, the affine modulation outperforms exact water-filling on corridor with substantially lower variance and higher mean return (Table 5). See C for the full comparison. Algorithm 1 Our framework: RCB schedule and RSQ modulation over per-agent Successor Distance 0: Policies πi\π^i\, critic V, SD encoders ϕi\ _i\ 0: βmin,βmax,κ,αR,λ,RSQref,hmin,hmax _ , _ ,κ, _R,λ,RSQ_ref,h_ ,h_ 1: Initialize: Rema←0R_ema← 0, μi←0 _i← 0, σi2←1 _i^2← 1 2: while not converged do 3: Sample rollouts across E parallel environments 4: R(k)←R^(k)← mean team return from completed episodes 5: // Phase 1: Global RCB Schedule 6: Rema←αRR(k)+(1−αR)RemaR_ema← _RR^(k)+(1- _R)R_ema 7: β←βmin+(βmax−βmin)σ(κ(Rtarget−Rema))β← _ +( _ - _ )σ(κ(R_target-R_ema)) 8: // Phase 2: Per-Agent RSQ Modulation 9: Compute ri,tintr_i,t^int via SD for each agent i 10: for each agent i do 11: Update μi,σi2 _i, _i^2 from riintr_i^int via EMA (10)–(11) 12: RSQi←μi2/(μi2+σi2+ϵ)RSQ_i← _i^2/( _i^2+ _i^2+ε) 13: hi←clip(1+λ(RSQi−RSQref),hmin,hmax)h_i (1+λ(RSQ_i-RSQ_ref),h_ ,h_ ) 14: end for 15: // Phase 3: Policy Update 16: ri,t←rtext+β⋅hi⋅ri,tintr_i,t← r_t^ext+β· h_i· r_i,t^int 17: Compute advantages A^i,t A_i,t via GAE, update via PPO 18: // Phase 4: Update SD Encoders 19: Update SD encoders ϕi\ _i\ via contrastive loss 20: end while 4.3 Successor Distance under RSQ Modulation Following [16], we use per-agent Successor Distance (SD) for intrinsic motivation. Each agent i maintains an encoder ϕi _i trained via symmetric InfoNCE [55] on local state features xti∈ℝdxx_t^i ^d_x (e.g., spatial coordinates for grid worlds, joint velocities for continuous control). The intrinsic reward ri,tint=mink<tdϕi(xki,xti)r_i,t^int= _k<t\,d_ _i(x_k^i,x_t^i) [52] encourages visits to states far from the agent’s trajectory history. We use per-agent SD rather than joint SD because it avoids the exponential sample complexity of learning distances in the joint state space [56], and because SD produces intrinsic rewards that differ substantially across agents depending on their spatial roles. An agent exploring open space consistently receives large SD values (high μi _i, low σi _i, high RSQ), while an agent stuck near obstacles receives small, erratic values (low μi _i, high σi _i, low RSQ). This per-agent variation is what enables RSQ to assign meaningfully different hih_i values (Assumption 2). Assumption 2 (Per-Agent Signal Quality Variation). There exists a minimum pairwise RSQ gap Δmin>0 _ >0 such that, for at least one pair of agents (i,j)(i,j), |RSQi−RSQj|≥Δmin|RSQ_i-RSQ_j|≥ _ holds during the active exploration phase. That is, the intrinsic reward method produces distinguishable per-agent signal quality. When Δmin=0 _ =0, all modulation weights collapse to hi≈1h_i≈ 1 and the method reduces to RCB-only. Insight 1 (SD Satisfies Assumption 2 Empirically). On corridor, SD produces inter-agent RSQ gaps exceeding 0.30.3 (Fig. 8). Replacing SD with entropy, RND, or count-based rewards yields near-uniform RSQ (hi≈1h_i≈ 1), violating Assumption 2 and yielding strongly negative mean return (D, Table 6). 4.4 Complete Training Algorithm Algorithm 1 presents the unified framework. The EMA initialization (μi=0 _i=0, σi2=1 _i^2=1) provides a natural warmup. Initially RSQi≈0RSQ_i≈ 0 for all agents, yielding hi=hminh_i=h_ , so the effective per-agent intensity is βmax⋅hmin=0.05 _ · h_ =0.05 regardless of the global β. As intrinsic reward statistics accumulate over the first few hundred updates, RSQ values rise and differentiation emerges, gradually increasing the exploration budget toward its intended level. Six of ten framework hyperparameters are shared across all environments. The remaining four are adapted per domain from 2–3 candidates on a single seed. Table 1 lists all values. Table 1: Hyperparameters of our framework per environment. Four parameters (αR _R, α, RSQrefRSQ_ref, hmaxh_ ) are shared across all environments. The domain-adapted parameters were selected as follows: RtargetR_target is set to a reasonable task return (robust over 20×20× range, Table 4), and the remaining parameters were chosen from 2–3 candidates on a single seed per environment. Parameter Corridor Tag 3s5z 27m Ant Ant-ball HC Shared across all environments αR _R (return EMA) 0.030.03 α (RSQ EMA) 0.10.1 RSQrefRSQ_ref 0.50.5 hmaxh_ 2.02.0 Adapted per domain βmax _ 0.50.5 0.50.5 0.20.2 0.50.5 0.50.5 0.50.5 0.50.5 κ 0.010.01 0.0150.015 0.010.01 0.010.01 0.010.01 0.010.01 0.010.01 βmin _ 0.10.1 0.150.15 0.050.05 0.30.3 0.050.05 0.050.05 0.050.05 RtargetR_target 400400 400400 1.51.5 3.03.0 500500 500500 500500 λ 3.03.0 2.02.0 2.02.0 5.05.0 2.02.0 2.02.0 2.02.0 hminh_ 0.10.1 0.10.1 0.80.8 0.10.1 0.10.1 0.10.1 0.10.1 5 Experiments We evaluate our method on seven cooperative benchmarks: two MPE tasks [57], two SMAX combat tasks, and three MABrax locomotion tasks, all implemented in JaxMARL [39]. 5.1 Setup 5.1.1 Environments Figure 3: MPE-corridor environment. (a) Environment layout: 8 agents (4 top, 4 bottom) must navigate through a narrow bottleneck (width 0.8) to reach goals on the opposite side. (b) Baseline methods that do not account for per-agent signal quality fail to coordinate, with edge agents getting stuck while center agents cause congestion at the bottleneck. (c) Our method achieves coordinated passage by attenuating noisy agents’ exploration and adapting the global exploration intensity to team progress. Figs. 3 and 4 together illustrate the seven evaluation environments. We use two MPE tasks [57], namely corridor (8 agents, narrow bottleneck, Fig. 3) and tag (6 predators vs. 2 scripted prey, Fig. 4a). Two SMAX tasks [39], a JAX reimplementation of SMAC [58] with simplified combat mechanics: 3s5z (8 agents vs. 9 opponents, heterogeneous unit types) and 27m (27 marines vs. 30 opponents). SMAX returns are not directly comparable to SMAC. Following recent evaluations [59, 60], we report mean episode return rather than binary win rate. Three MABrax [39, 61] continuous locomotion tasks: ant_4x2 (4-segment ant, each segment is an agent with 2 actuators), ant_ball (same ant carrying a ball on a tray with partial observability adapted from [62]), and halfcheetah_6x1 (6-segment cheetah, 1 actuator each). On ant_ball, the forward reward is amplified (3×3× weight) and each agent observes only a neighbor’s joint velocities in a directed ring (24 of 46 dimensions), making coordinated gaits essential. Environment details (observation spaces, reward structure, episode lengths) are in E. Figure 4: Evaluation environments (excluding corridor, shown in Fig. 3). (a) MPE-Tag: 6 predators (blue) chase 2 scripted prey (red). (b) SMAX-27m: 27 allied marines (blue) vs. 30 enemies (red). (c) MABrax Ant (ant_4x2): 4 segment ant with 2 actuators per segment, where each segment is an independent agent. (d) MABrax Ant-Ball: same ant carrying a ball (red) on a tray while walking forward. (e) MABrax HalfCheetah (halfcheetah_6x1): 6 segment locomotion with 1 actuator per segment. These environments span a range of coordination challenges. Corridor stresses spatial coordination under physical constraints, tag requires strategic role assignment, the two SMAX tasks test combat coordination under numerical disadvantage at different scales (8 and 27 agents), and the MABrax tasks test continuous multi-agent locomotion with varying degrees of agent heterogeneity. 5.1.2 Baselines We compare against eight methods: IPPO [63] (independent PPO without parameter sharing, MABrax only), MAPPO [38] (MAPPO without intrinsic reward, MPE and SMAX), Linear (fixed β=0.1β=0.1, tuned), COIN [7] (counterfactual intrinsic motivation), Lagrangian (Lagrange multiplier λ balances intrinsic and extrinsic advantages as Aint+λAextA^int+λ A^ext, with λ updated via dual gradient descent on mean extrinsic advantage), MAVEN [5] (multi-agent variational exploration with latent space coordination), RCB-only (our RCB schedule without RSQ, i.e., λ=0λ=0), and RSQ-only (per-agent RSQ modulation without return-conditioned adaptation). 5.1.3 Implementation For MPE and SMAX, all methods use MAPPO [38] with GRU-based actors and a shared critic. For MABrax, all methods use IPPO [63] with per-agent feed-forward networks and no shared critic. These choices follow standard practice for each domain [38, 39]. Intrinsic rewards are computed via per-agent Successor Distance [16] using a Metric Residual Network encoder. The local state features xtix_t^i used for SD differ by domain: spatial coordinates (x,y)(x,y) for MPE (dx=2d_x=2), (health, x-position, y-position, cooldown) for SMAX (dx=4d_x=4), and agent-specific features for MABrax (dx=2d_x=2 for ant, dx=10d_x=10 for ant-ball, dx=3d_x=3 for halfcheetah), following the per-agent factorization in Sec. 4.3. Full training hyperparameters, SD encoder configuration, and per-domain details are provided in E. All experiments use 10 random seeds per method on NVIDIA RTX A5000 GPUs. The framework is applied identically across all settings. 5.2 Main Results (a) MPE-corridor (8 agents) (b) MABrax ant_4x2 (4 agents, 2 actuators) Figure 5: Learning curves on representative discrete (left) and continuous (right) multi-agent tasks. Shaded regions indicate ±1± 1 standard deviation across seeds. On corridor, our method is the only tested method to achieve positive final returns. On ant_4x2, our method achieves the highest final return among all baselines. Table 2: Mean episode return (± std) across seven environments. Bold marks the highest mean and † marks a statistical tie with the best (p>0.05p>0.05, Welch’s t-test). All methods share the same backbone per domain, namely MAPPO for MPE/SMAX and IPPO for MABrax (Sec. 5.1.3). Linear uses a fixed exploration weight β=0.1β=0.1. Dashes indicate methods not applicable to that domain. 10 seeds per method. MPE (discrete) SMAX MABrax (continuous) Method Corridor Tag 3s5z 27m ant_4x2 ant_ball hc_6x1 IPPO — — — — 1659(±41)1659^(± 41) 830(±158)830^(± 158) 2720(±114)2720^(± 114) MAPPO −1477(±288)-1477^(± 288) +3622(±565)+3622^(± 565) 0.542(±0.009)0.542^(± 0.009) 0.430(±0.015)0.430^(± 0.015) — — — RSQ-only −20(±554)†-20^(± 554) +3823(±714)+3823^(± 714) 0.476(±0.033)0.476^(± 0.033) 0.448(±0.007)†0.448^(± 0.007) 2577(±368)2577^(± 368) 1342(±159)1342^(± 159) 2589(±170)2589^(± 170) COIN −2729(±388)-2729^(± 388) +216(±663)+216^(± 663) 0.193(±0.017)0.193^(± 0.017) 0.120(±0.034)0.120^(± 0.034) 2724(±427)†2724^(± 427) 1702(±173)†1702^(± 173) 1678(±476)1678^(± 476) MAVEN −1584(±432)-1584^(± 432) +2329(±374)+2329^(± 374) 0.450(±0.094)0.450^(± 0.094) 0.238(±0.039)0.238^(± 0.039) 2282(±270)2282^(± 270) 1498(±128)1498^(± 128) 2398(±103)2398^(± 103) Lagrangian −1238(±1019)-1238^(± 1019) +2400(±180)+2400^(± 180) 0.200(±0.056)0.200^(± 0.056) 0.258(±0.065)0.258^(± 0.065) 2623(±427)†2623^(± 427) 1697(±292)†1697^(± 292) (±141)2874^(± 141) RCB-only −552(±556)-552^(± 556) +3392(±584)+3392^(± 584) 0.582(±0.019)†0.582^(± 0.019) 0.141(±0.021)0.141^(± 0.021) 2583(±351)2583^(± 351) 1690(±231)†1690^(± 231) 2762(±168)†2762^(± 168) Linear −259(±708)†-259^(± 708) +3542(±410)+3542^(± 410) 0.585(±0.013)†0.585^(± 0.013) 0.449(±0.003)0.449^(± 0.003) 2509(±341)2509^(± 341) 1540(±182)1540^(± 182) 2746(±114)2746^(± 114) Ours +(±224)+190^(± 224) +(±404)+4930^(± 404) 0.591(±0.014)0.591^(± 0.014) 0.447(±0.005)†0.447^(± 0.005) (±312)2940^(± 312) (±261)1846^(± 261) 2870(±113)†2870^(± 113) We assess pairwise significance using Welch’s t-test [64] with p<0.05p<0.05 as the threshold, marking statistical ties (p>0.05p>0.05) with † in Table 2. 5.2.1 Discrete Control (MPE) On MPE-corridor, which demands tight spatial coordination, our method achieves the highest mean return with the lowest variance (Table 2). The primary distinction over Linear (second-best) is reliability, with our method achieving 3.2×3.2× lower standard deviation. Every other exploration method ends with strongly negative mean return on this task. On MPE-tag (Fig. 6(a)), our method achieves the highest mean return (Table 2). RSQ-only and Linear are the closest competitors but trail our method in mean return while exhibiting larger run-to-run variance. 5.2.2 Large-Scale Coordination (SMAX) (a) MPE-tag (6 predators, 2 prey) (b) SMAX-3s5z (8 vs. 9) Figure 6: Learning curves on MPE-tag (left) and SMAX-3s5z (right). Shaded regions indicate ±1± 1 standard deviation. On tag, our method converges to the highest return. On SMAX-3s5z, three baselines collapse below 0.20.2 while our method maintains the top position. Figure 7: βmin _ sensitivity without RSQ on SMAX-27m. Without RSQ, increasing fixed βmin _ beyond 0.150.15 drives the mean return below 0.20.2 at βmin=0.3 _ =0.3 and to near zero at βmin=0.5 _ =0.5. With RSQ, Ours operates safely at the same βmin=0.3 _ =0.3. On SMAX-3s5z (Table 2, Fig. 6(b)), our method achieves the highest mean return (0.5910.591), with Linear and RCB-only statistically tied (p>0.10p>0.10). The key contrast lies in stability, with COIN and Lagrangian both falling below 0.20.2 and MAVEN reaching intermediate performance with 5.5×5.5× higher variance than Ours. This task is also the only benchmark with genuinely heterogeneous agent types (stalkers vs. zealots), and Our per-agent RSQ modulation naturally adapts to the resulting differences in intrinsic reward dynamics across unit types. On SMAX-27m (Table 2), our method and Linear are statistically indistinguishable at convergence (p=0.29p=0.29), while all other exploration baselines fall below 0.260.26. The key distinction is exploration safety (Fig. 7), where fixed β=0.3β=0.3 without RSQ drives the mean return below 0.20.2, whereas our method operates safely at the same βmin=0.3 _ =0.3 by selectively attenuating noisy agents.111The β-sensitivity data also confirms the contraction condition (5) empirically: on SMAX (κ=0.01κ=0.01, βmax−βmin=0.2 _ - _ =0.2), the product is 5×10−5≪15× 10^-5 1. 5.2.3 Continuous Control (MABrax) On ant_4x2, our method achieves the highest return with COIN and Lagrangian statistically tied (Table 2). On ant_ball, our method leads with a 9.2%9.2\% improvement over RCB-only, confirming that per-agent RSQ modulation adds value beyond global scheduling. IPPO collapses on ant_ball, indicating that independent learning cannot solve this tightly coupled locomotion-balance task. On halfcheetah, our method and Lagrangian share the top position (p=0.95p=0.95), consistent with the homogeneous agent roles that limit RSQ differentiation. Across all three continuous tasks, our method delivers consistently strong performance, indicating that per-agent signal quality awareness combined with return-conditioned scheduling is well suited to multi-agent locomotion. 5.3 Ablation Studies 5.3.1 Component Synergy Neither RCB-only nor RSQ-only matches the combined framework on corridor. RCB-only provides a global schedule but lacks per-agent signal awareness, so agents with unreliable intrinsic rewards destabilize the policy. RSQ-only improves substantially over unmodulated baselines (corridor mean −20±554-20± 554 vs. MAPPO’s −1477-1477) but applies a fixed high exploration intensity throughout training, leaving residual instability on corridor and a 22%22\% lower return on tag (+3823+3823 vs. +4930+4930). On SMAX-27m, RSQ-only achieves 0.448±0.0070.448± 0.007 (p=0.72p=0.72 vs. Ours), suggesting that per-agent modulation is the dominant component. The narrow return range (0–0.450.45 vs. Rtarget=3.0R_target=3.0) keeps β nearly constant, limiting RCB’s contribution. On wide-range tasks (corridor, tag, MABrax), both components contribute actively. The combined framework achieves the highest or statistically tied return on all seven environments. 5.3.2 RSQ Sensitivity (λ) Table 3: λ sensitivity on MPE-corridor (λ=0λ=0 is RCB-only). Best in bold. λ Mean Std 0.00.0 −551.5-551.5 555.7555.7 0.50.5 −143.0-143.0 536.2536.2 1.01.0 +27.2+27.2 430.0430.0 2.02.0 −82.7-82.7 576.7576.7 3.03.0 +190.1+190.1 224.0224.0 Table 4: RtargetR_target sensitivity on MPE-corridor. Best in bold. RtargetR_target Mean Return Std 5050 −213.8-213.8 502.7502.7 100100 −229.3-229.3 599.2599.2 200200 +151.2+151.2 314.3314.3 400400 +190.1+190.1 224.0224.0 600600 +234.5+234.5 299.6299.6 800800 +105.2+105.2 144.5144.5 10001000 +198.0+198.0 290.5290.5 Table 3 shows that any RSQ modulation (λ>0λ>0) improves both mean return and stability over the unmodulated baseline (λ=0λ=0). The best result occurs at λ=3.0λ=3.0 with 2.5×2.5× lower standard deviation. 5.3.3 RtargetR_target Sensitivity As noted in Remark 1, the sigmoid’s 5%–95% transition spans ΔR≈5.89/κ R≈ 5.89/κ return units (≈590≈ 590 for κ=0.01κ=0.01), so shifting RtargetR_target by hundreds of units barely changes the β trajectory. Table 4 confirms this, with all values in [200,1000][200,1000] yielding positive mean return and only extreme values (Rtarget≤100R_target≤ 100) destabilizing training. 5.4 RSQ Dynamics Analysis Figure 8: Per-agent RSQ dynamics during training on MPE-corridor (left) and MPE-tag (right). (a,b) Per-agent RSQ values differentiate over training, with corridor exhibiting wider inter-agent spread than tag. (c,d) Modulation weights hih_i reflect RSQ differences: agents with low RSQ are attenuated toward hminh_ , concentrating exploration budget on reliable agents. (e,f) The global exploration intensity β (RCB) adapts to team performance, decreasing as returns improve. The interaction between per-agent RSQ modulation and global RCB adaptation produces the corridor reliability property observed in Table 2. Fig. 8 visualizes the per-agent RSQ dynamics during a representative training run. On corridor, agents exhibit substantial RSQ differentiation early in training (steps 5M–15M). Open-space agents produce stable SD values across the batch (high RSQ), while bottleneck agents experience high variance due to stochastic multi-agent interactions (low RSQ, lower modulation weights). As agents learn coordinated passage, RSQ values converge as SD intrinsic rewards decline with exploration saturation, but the RCB schedule has already reduced β to near βmin _ at this stage. On tag, all agents achieve similar RSQ, consistent with symmetric predator roles. The global β responds to team progress: on corridor it remains high until step 15M, while on tag it drops early as returns improve quickly. On halfcheetah, homogeneous roles produce near-uniform RSQ, consistent with the tied performance against Lagrangian in Table 2. 6 Conclusions and Future Work We presented a framework that resolves the safety-performance dilemma in multi-agent exploration through two complementary mechanisms. Return-conditioned scheduling (RCB) adapts global intensity with provable convergence (Theorem 1), and per-agent RSQ modulation concentrates the exploration budget on agents with reliable intrinsic reward signals while preserving the ordering of the information-optimal allocation (Proposition 1). Across seven cooperative benchmarks spanning discrete, large-scale, and continuous domains, our method achieves top-tier returns. Limitations. RSQ modulation requires that the intrinsic reward method produces per-agent signal quality variation (Assumption 2). In our experiments, only Successor Distance satisfies this requirement, while entropy, RND, and count-based rewards yield near-uniform RSQ across agents (D). Additionally, the current framework assumes cooperative settings with a shared team reward, limiting direct applicability to competitive or mixed-motive scenarios. Future Work. Extending RSQ to work with a broader family of intrinsic reward methods is an important open problem. Mixed-motive and competitive settings where the team reward assumption no longer holds, as well as dynamic team compositions where agents join or leave mid-episode, would further test the adaptability of per-agent RSQ allocation. Combining RSQ with topology-aware intrinsic reward designs that exploit the communication graph structure may enhance signal quality differentiation in large-scale teams. Acknowledgment Funding information will be provided upon acceptance. CRediT authorship contribution statement Dahyun Oh: Conceptualization, Methodology, Software, Validation, Formal analysis, Investigation, Data curation, Writing – original draft, Visualization. Minhyuk Yoon: Developing mathematical formulations including the proofs. H. Jin Kim: Supervision, Funding acquisition, Writing – review & editing. Data availability The source code is publicly available at https://github.com/DH-O/RRS. Appendix A Proof of Theorem 1 The proof proceeds in three steps: (1) establishing that the RCB map is a contraction, (2) bounding the deterministic tracking error, and (3) bounding the stochastic residual. Step 1: Contraction. Define the RCB map as a function of the EMA return: g(Rema)=βmin+(βmax−βmin)σ(κ(Rtarget−Rema)),g(R_ema)\;=\; _ +( _ - _ )\,σ\! (κ(R_target-R_ema) ), so that β(k)=g(Rema(k))β^(k)=g(R_ema^(k)). The composed one-step map on β is Φ(β)=g(R¯(β))=βmin+(βmax−βmin)σ(κ(Rtarget−R¯(β))) (β)=g ( R(β) )= _ +( _ - _ )\,σ(κ(R_target- R(β))). Since σ∈(0,1)σ∈(0,1), Φ maps [βmin,βmax][ _ , _ ] to itself. By the chain rule, Φ′(β)=−(βmax−βmin)⋅κ⋅σ′(⋅)⋅R¯′(β). (β)=-( _ - _ )·κ·σ (·)· R (β). Since σ′(x)=σ(x)(1−σ(x))≤1/4σ (x)=σ(x)(1-σ(x))≤ 1/4, we have LΦ≜supβ|Φ′(β)|≤κ(βmax−βmin)⋅supβ|R¯′(β)|⋅14.L_ \; \; _β| (β)|\;≤\;κ( _ - _ )· _β| R (β)|· 14. The contraction condition (5) states that the right-hand side is strictly less than 1, which gives LΦ<1L_ <1. By Banach’s fixed-point theorem [54], Φ has a unique fixed point β∗=Φ(β∗)β^*= (β^*), with corresponding equilibrium return R∗=R¯(β∗)R^*= R(β^*). Step 2: Deterministic convergence. Let e(k)=Rema(k)−R∗e^(k)=R_ema^(k)-R^*. From the EMA update (3) with ξ(k)=0ξ^(k)=0: e(k+1) e^(k+1) =αR(R¯(β(k))−R¯(β∗))+(1−αR)e(k). = _R ( R(β^(k))- R(β^*) )+(1- _R)\,e^(k). We bound each term. For the first term, by the mean value theorem, |R¯(β(k))−R¯(β∗)|≤LR|β(k)−β∗|| R(β^(k))- R(β^*)|≤ L_R|β^(k)-β^*| where LR=supβ|R¯′(β)|L_R= _β| R (β)|. Since β(k)=g(Rema(k))β^(k)=g(R_ema^(k)) and β∗=g(R∗)β^*=g(R^*), the Lipschitz constant of g is |g′(Rema)|=(βmax−βmin)κσ′(⋅)≤(βmax−βmin)κ/4|g (R_ema)|=( _ - _ )κ\,σ (·)≤( _ - _ )κ/4, giving |β(k)−β∗|=|g(Rema(k))−g(R∗)|≤(βmax−βmin)κ4|e(k)|.|β^(k)-β^*|\;=\;|g(R_ema^(k))-g(R^*)|\;≤\; ( _ - _ )κ4\,|e^(k)|. Combining: αR|R¯(β(k))−R¯(β∗)|≤αRLR⋅(βmax−βmin)κ4|e(k)|=αRLΦ|e(k)| _R| R(β^(k))- R(β^*)|≤ _RL_R· ( _ - _ )κ4\,|e^(k)|= _RL_ \,|e^(k)|, where the last equality uses LΦ=LR⋅(βmax−βmin)κ/4L_ =L_R·( _ - _ )κ/4 from Step 1. Therefore, |e(k+1)|≤[αRLΦ+(1−αR)]|e(k)|=ρ|e(k)|,|e^(k+1)|\;≤\; [ _RL_ +(1- _R) ]\,|e^(k)|\;=\;ρ\,|e^(k)|, where ρ=1−αR(1−LΦ)ρ=1- _R(1-L_ ). Since αR∈(0,1) _R∈(0,1) and LΦ<1L_ <1 (from the contraction condition), we have ρ∈(0,1)ρ∈(0,1), so the error contracts geometrically. Step 3: Stochastic convergence. With noise ξ(k)ξ^(k) (Assumption 1), the EMA update becomes e(k+1)=αR(R¯(β(k))−R¯(β∗))+(1−αR)e(k)⏟D(k)+αRξ(k),e^(k+1)= _R ( R(β^(k))- R(β^*) )+(1- _R)\,e^(k)_D^(k)+ _Rξ^(k), where D(k)D^(k) is the deterministic part satisfying |D(k)|≤ρ|e(k)||D^(k)|≤ρ|e^(k)| from Step 2. Squaring: |e(k+1)|2=|D(k)|2+2αRD(k)ξ(k)+αR2|ξ(k)|2|e^(k+1)|^2=|D^(k)|^2+2 _RD^(k)ξ^(k)+ _R^2|ξ^(k)|^2. Taking expectations, the cross term [D(k)ξ(k)]=0E[D^(k)ξ^(k)]=0 because D(k)D^(k) depends only on Rema(k)R_ema^(k) (and hence on ξ(0),…,ξ(k−1)ξ^(0),…,ξ^(k-1)), while ξ(k)ξ^(k) is the noise at iteration k, which is zero-mean and independent of prior iterations by Assumption 1. This gives: [|e(k+1)|2]≤ρ2[|e(k)|2]+αR2σξ2.E [|e^(k+1)|^2 ]\;≤\;ρ^2\,E [|e^(k)|^2 ]+ _R^2\, _ξ^2. This is a linear recursion ak+1≤ρ2ak+αR2σξ2a_k+1≤ρ^2a_k+ _R^2 _ξ^2 with ρ2<1ρ^2<1. Unrolling gives ak≤ρ2ka0+αR2σξ2/(1−ρ2)a_k≤ρ^2ka_0+ _R^2 _ξ^2/(1-ρ^2). As k→∞k→∞, the first term vanishes and limk→∞[|e(k)|2]≤αR2σξ21−ρ2. _k→∞E [|e^(k)|^2 ]\;≤\; _R^2 _ξ^21-ρ^2. To connect with the O-notation in Theorem 1: since ρ=1−αR(1−LΦ)ρ=1- _R(1-L_ ), we have 1−ρ2=(1−ρ)(1+ρ)=αR(1−LΦ)(1+ρ)≥αR(1−LΦ)1-ρ^2=(1-ρ)(1+ρ)= _R(1-L_ )(1+ρ)≥ _R(1-L_ ) (because ρ≥0ρ≥ 0), so αR2σξ21−ρ2≤αRσξ21−LΦ=O(αRσξ21−LΦ). _R^2 _ξ^21-ρ^2\;≤\; _R\, _ξ^21-L_ \;=\;O\! ( _R\, _ξ^21-L_ ). □ Appendix B Proof of Proposition 1 (Ordering Preservation) The water-filling formula (9) is standard textbook material [41, Ch. 10], and we omit its derivation. Our claim of interest is that the affine RSQ modulation preserves the SNR ordering it induces. Write hi=(h∘RSQ)(SNRi)h_i=(h )(SNR_i), with RSQ(SNR)=SNR/(1+SNR)RSQ(SNR)=SNR/(1+SNR) and h(RSQ)=clip(1+λ(RSQ−RSQref),hmin,hmax)h(RSQ)=clip(1+λ(RSQ-RSQ_ref),h_ ,h_ ). The map RSQRSQ has derivative 1/(1+SNR)2>01/(1+SNR)^2>0 and is therefore strictly increasing on (0,∞)(0,∞). The clipped affine h has slope λ>0λ>0 in the unsaturated region (hmin,hmax)(h_ ,h_ ) and slope 0 when clipped, hence is non-decreasing. The composition h∘RSQh is therefore non-decreasing in SNRSNR, giving Part (i). Part (i) follows directly from (9), since 1/SNRi1/SNR_i is decreasing in SNRiSNR_i. For Part (i), if hi,hj∈(hmin,hmax)h_i,h_j∈(h_ ,h_ ) then h is the unclipped affine on both, so hi−hj=λ(RSQi−RSQj)h_i-h_j=λ(RSQ_i-RSQ_j), which is strictly positive whenever SNRi>SNRjSNR_i>SNR_j by the strict monotonicity of RSQRSQ. Appendix C Exact Water-Filling vs. Affine RSQ Modulation The methodology (Sec. 4.2) argues that the affine RSQ modulation is preferable to exact water-filling because the latter permanently suppresses agents whose estimated SNR falls below the water level. We validate this claim empirically on corridor (10 seeds each, identical hyperparameters except the allocation mechanism). Table 5: Exact water-filling vs. affine RSQ modulation on corridor (10 seeds). Exact water-filling exhibits 2.5× higher variance and a strongly degraded mean return because noisy SNR estimates trigger permanent agent suppression (βi=0 _i=0), destabilizing team coordination. Allocation Mean Return Std Exact water-filling −55-55 560560 Affine RSQ (Ours) ++190 224 Table 5 confirms that the affine modulation substantially outperforms exact water-filling, which suffers from irreversible agent suppression when noisy SNR estimates push agents below the water level. Appendix D Validating the SD Requirement (Assumption 2) To empirically validate Assumption 2 and Insight 1, we replace Successor Distance with three alternative intrinsic reward methods on MPE-corridor: policy entropy, Random Network Distillation (RND) [13], and visitation count. For each source, we apply the RSQ-modulated configuration over 5 seeds. Table 6: MPE-corridor performance by intrinsic reward source under RSQ modulation. SD uses 10-seed data from the main experiments (Table 2). Entropy, RND, and count use 5 seeds each. Only Successor Distance produces sufficient inter-agent RSQ variation for effective modulation. Entropy, RND, and count yield uniform hi≈1h_i≈ 1 across agents, rendering RSQ modulation ineffective regardless of λ. Intrinsic Modulation Mean Std SD (Ours) RSQ (λ=3.0λ=3.0) ++190 224224 Entropy RSQ (λ=1.5λ=1.5) −1297-1297 275275 RND RSQ (λ=1.5λ=1.5) −1626-1626 5151 Count RSQ (λ=1.5λ=1.5) −1097-1097 936936 Table 6 confirms that only SD produces sufficient inter-agent RSQ variation for effective modulation. Appendix E Implementation Details Table 7 summarizes all training and SD encoder hyperparameters. All baselines were tuned with comparable effort, selecting hyperparameters from 2–3 candidates on a single seed per environment, matching the tuning protocol used for our method (Table 1 caption). Table 7: Backbone training hyperparameters by domain. Parameter MPE MABrax SMAX backbone MAPPO IPPO MAPPO shared critic yes no yes parameter sharing no no no network type GRU F GRU hidden dim 64 64 / 128† 64 GRU dim 256 — 128 FC layers 2 2 2 actor learning rate 1×10−31× 10^-3§ 3×10−43× 10^-4 5×10−45× 10^-4 critic learning rate 5×10−45× 10^-4 3×10−43× 10^-4 5×10−45× 10^-4 LR annealing none none none discount γ 0.99 0.99 0.99 GAE λGAE _GAE 0.95 0.95 0.95 clip ratio ϵε 0.3 0.2 0.2 entropy coeff 0.015 0.01 0.01 max grad norm 0.6 10.0 10.0 update epochs 10 4 8 training steps 3×1073× 10^7 5×1075× 10^7 2×1072× 10^7 parallel envs 200 512 128 rollout length 256 64 128 intrinsic reward scale 2.0 1.0 0.01 warmup rollouts 0 5 0 / 20‡ seeds 10 10 10 †halfcheetah uses 128-dim; ant/ant-ball use 64-dim. §Tag uses 5×10−45× 10^-4 actor LR. ‡SMAX-27m uses 20 warmup rollouts; SMAX-3s5z uses 0. Table 8: Successor Distance (SD) encoder hyperparameters, shared across all domains. Parameter Value encoder architecture Metric Residual Network (MRN) residual blocks 10 latent dim 64 output dim 32 contrastive learning rate 1×10−31× 10^-3 SD discount γSD _SD 0.99 update epochs per rollout 25 batch size 512 Table 9: Per-domain SD input features xtix_t^i, using each agent’s own continuous state features and excluding constant indicators (e.g., unit-type bits). Domain Features dxd_x MPE (corridor, tag) spatial position (x,y)(x,y) 2 SMAX health, x-pos, y-pos, cooldown 4 MABrax (ant_4x2) joint velocities 2 MABrax (ant_ball) obs[0:10] (torso + joints) 10 MABrax (halfcheetah) joint velocities 3 Declaration of generative AI and AI-assisted technologies in the manuscript preparation process During the preparation of this work the authors used Claude (Anthropic) in order to assist with grammar checking and code debugging. The authors reviewed and edited all output and take full responsibility for the content of the published article. References [1] S. V. Albrecht, F. Christianos, L. Schäfer, Multi-agent reinforcement learning: Foundations and modern approaches, MIT Press, 2024. [2] L. Panait, S. Luke, Cooperative multi-agent learning: The state of the art, Autonomous agents and multi-agent systems 11 (3) (2005) 387–434. [3] K. Hu, M. Li, Z. Song, K. Xu, Q. Xia, N. Sun, P. Zhou, M. Xia, A review of research on reinforcement learning algorithms for multi-agents, Neurocomputing 599 (2024) 128068. [4] S. Gronauer, K. Diepold, Multi-agent deep reinforcement learning: a survey, Artificial Intelligence Review 55 (2) (2022) 895–943. [5] A. Mahajan, T. Rashid, M. Samvelyan, S. Whiteson, Maven: Multi-agent variational exploration, Advances in neural information processing systems 32 (2019). [6] E. Chen, Z.-W. Hong, J. Pajarinen, P. Agrawal, Redeeming intrinsic rewards via constrained optimization, Advances in neural information processing systems 35 (2022) 4996–5008. [7] J. Li, K. Kuang, B. Wang, X. Li, F. Wu, J. Xiao, L. Chen, Two heads are better than one: A simple exploration framework for efficient multi-agent reinforcement learning, Advances in neural information processing systems 36 (2023) 20038–20053. [8] J. Foerster, I. A. Assael, N. De Freitas, S. Whiteson, Learning to communicate with deep multi-agent reinforcement learning, Advances in neural information processing systems 29 (2016). [9] C. Li, T. Wang, C. Wu, Q. Zhao, J. Yang, C. Zhang, Celebrating diversity in shared multi-agent reinforcement learning, Advances in Neural Information Processing Systems 34 (2021) 3991–4002. [10] W. Kim, Y. Sung, An adaptive entropy-regularization framework for multi-agent reinforcement learning, in: International Conference on Machine Learning, PMLR, 2023, p. 16829–16852. [11] Y. Du, L. Han, M. Fang, J. Liu, T. Dai, D. Tao, Liir: Learning individual intrinsic reward in multi-agent reinforcement learning, Advances in neural information processing systems 32 (2019). [12] D. Pathak, P. Agrawal, A. A. Efros, T. Darrell, Curiosity-driven exploration by self-supervised prediction, in: International conference on machine learning, PMLR, 2017, p. 2778–2787. [13] Y. Burda, H. Edwards, A. Storkey, O. Klimov, Exploration by random network distillation, arXiv preprint arXiv:1810.12894 (2018). [14] N. Jaques, A. Lazaridou, E. Hughes, C. Gulcehre, P. Ortega, D. Strouse, J. Z. Leibo, N. De Freitas, Social influence as intrinsic motivation for multi-agent deep reinforcement learning, in: International conference on machine learning, PMLR, 2019, p. 3040–3049. [15] L. Zheng, J. Chen, J. Wang, J. He, Y. Hu, Y. Chen, C. Fan, Y. Gao, C. Zhang, Episodic multi-agent reinforcement learning with curiosity-driven exploration, Advances in Neural Information Processing Systems 34 (2021) 3757–3769. [16] V. Myers, C. Zheng, A. Dragan, S. Levine, B. Eysenbach, Learning temporal distances: Contrastive successor features can provide a metric structure for decision-making, arXiv preprint arXiv:2406.17098 (2024). [17] R. S. Sutton, A. G. Barto, Reinforcement learning: An introduction, 2nd Edition, MIT Press, 2018. [18] M. C. Machado, S. Srinivasan, M. Bowling, Domain-independent optimistic initialization for reinforcement learning, in: AAAI Workshop on Learning for General Competency in Video Games, 2015. [19] P. Auer, N. Cesa-Bianchi, P. Fischer, Finite-time analysis of the multiarmed bandit problem, Machine Learning 47 (2-3) (2002) 235–256. [20] I. Osband, C. Blundell, A. Pritzel, B. Van Roy, Deep exploration via bootstrapped dqn, in: Advances in Neural Information Processing Systems, Vol. 29, 2016. [21] M. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, R. Munos, Unifying count-based exploration and intrinsic motivation, in: Advances in Neural Information Processing Systems, Vol. 29, 2016. [22] J. Li, X. Shi, J. Li, X. Zhang, J. Wang, Random curiosity-driven exploration in deep reinforcement learning, Neurocomputing 418 (2020) 139–147. [23] T. Haarnoja, A. Zhou, P. Abbeel, S. Levine, Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor, in: International Conference on Machine Learning, 2018, p. 1861–1870. [24] T. Wang, J. Wang, Y. Wu, C. Zhang, Influence-based multi-agent exploration, in: International Conference on Learning Representations, 2020. [25] S. Sukhbaatar, R. Fergus, et al., Learning multiagent communication with backpropagation, in: Advances in Neural Information Processing Systems, Vol. 29, 2016. [26] A. Das, T. Gervet, J. Romoff, D. Batra, D. Parikh, M. Rabbat, J. Pineau, Tarmac: Targeted multi-agent communication, in: International Conference on Machine Learning, 2019, p. 1538–1546. [27] R. Jiang, X. Zhang, Y. Liu, Y. Xu, X. Zhang, Y. Zhuang, Multi-agent cooperative strategy with explicit teammate modeling and targeted informative communication, Neurocomputing 586 (2024) 127638. [28] J. Liu, Y. Zhong, S. Hu, H. Fu, Q. Chang, Y. Yang, Maximum entropy heterogeneous-agent reinforcement learning, in: International Conference on Learning Representations, 2024. [29] S. Iqbal, F. Sha, Coordinated exploration via intrinsic rewards for multi-agent reinforcement learning, arXiv preprint arXiv:1905.12127 (2019). [30] T. Li, K. Zhu, Toward efficient multi-agent exploration with trajectory entropy maximization, in: International Conference on Learning Representations, 2025. [31] Y. Pan, Z. Liu, H. Wang, Wonder wins ways: Curiosity-driven exploration through multi-agent contextual calibration, in: Advances in Neural Information Processing Systems, Vol. 38, 2025. [32] X. He, H. Ge, Y. Hou, J. Yu, SAEIR: Sequentially accumulated entropy intrinsic reward for cooperative multi-agent reinforcement learning with sparse reward, in: International Joint Conference on Artificial Intelligence, 2024. [33] C. Nimonkar, S. Shah, C. Ji, B. Eysenbach, Self-supervised goal-reaching results in multi-agent cooperation and exploration, arXiv preprint arXiv:2509.10656 (2025). [34] Z. Xu, H. P. van Hasselt, D. Silver, Meta-gradient reinforcement learning, in: Advances in Neural Information Processing Systems, Vol. 31, 2018. [35] T. Zahavy, Z. Xu, V. Veeriah, M. Hessel, J. Oh, H. van Hasselt, D. Silver, S. Singh, A self-tuning actor-critic algorithm, Advances in Neural Information Processing Systems 33 (2020). [36] H. Ma, Z. Luo, T. V. Vo, K. Sima, T.-Y. Leong, Highly efficient self-adaptive reward shaping for reinforcement learning, in: International Conference on Learning Representations, 2025. [37] C. Li, X. Wei, Y. Zhao, X. Geng, An effective maximum entropy exploration approach for deceptive game in reinforcement learning, Neurocomputing 403 (2020) 98–108. [38] C. Yu, A. Velu, E. Vinitsky, J. Gao, Y. Wang, A. Bayen, Y. Wu, The surprising effectiveness of ppo in cooperative multi-agent games, Advances in neural information processing systems 35 (2022) 24611–24624. [39] A. Rutherford, B. Ellis, M. Gallici, J. Cook, A. Lupu, G. Ingvarsson, T. Willi, R. Hammond, A. Khan, C. S. de Witt, et al., Jaxmarl: Multi-agent rl environments and algorithms in jax, Advances in Neural Information Processing Systems 37 (2024) 50925–50951. [40] X. Zheng, X. Ma, C. Shen, C. Wang, Constrained intrinsic motivation for reinforcement learning, in: Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, 2024, p. 5608–5616. [41] T. M. Cover, J. A. Thomas, Elements of Information Theory, 2nd Edition, Wiley-Interscience, 2006. [42] J. W. Roberts, R. Tedrake, Signal-to-noise ratio analysis of policy gradient algorithms, in: Advances in Neural Information Processing Systems, Vol. 21, 2008. [43] J. G. Kuba, M. Wen, L. Meng, H. Zhang, D. Mguni, J. Wang, Y. Yang, et al., Settling the variance of multi-agent policy gradients, Advances in Neural Information Processing Systems 34 (2021) 13458–13470. [44] H. Wu, J. Zhang, Z. Wang, Y. Lin, H. Li, Sub-AVG: Overestimation reduction for cooperative multi-agent reinforcement learning, Neurocomputing 474 (2022) 94–106. [45] H. Han, H. Yang, Non-uniform noise-to-signal ratio in the REINFORCE policy-gradient estimator, arXiv preprint arXiv:2602.01460 (2026). [46] D. Russo, B. Van Roy, Learning to optimize via information-directed sampling, Advances in neural information processing systems 27 (2014). [47] H.-L. Hsu, W. Wang, M. Pajic, P. Xu, Randomized exploration in cooperative multi-agent reinforcement learning, in: Advances in Neural Information Processing Systems, Vol. 37, 2024. [48] A. Goldsmith, Wireless Communications, Cambridge University Press, 2005. [49] F. A. Oliehoek, C. Amato, A concise introduction to decentralized POMDPs, Springer, 2016. [50] C. E. Shannon, A mathematical theory of communication, The Bell System Technical Journal 27 (3) (1948) 379–423. [51] A. J. Goldsmith, S.-G. Chua, Variable-rate variable-power mqam for fading channels, IEEE Transactions on Communications 45 (10) (1997) 1218–1230. [52] Y. Jiang, Q. Liu, Y. Yang, X. Ma, D. Zhong, H. Hu, J. Yang, B. Liang, B. Xu, C. Zhang, Q. Zhao, Episodic novelty through temporal distance, in: International Conference on Learning Representations, 2025. [53] V. S. Borkar, Stochastic approximation: a dynamical systems viewpoint, Cambridge University Press, 2008. [54] A. Granas, J. Dugundji, Fixed Point Theory, Springer, New York, 2003. [55] A. v. d. Oord, Y. Li, O. Vinyals, Representation learning with contrastive predictive coding, arXiv preprint arXiv:1807.03748 (2018). [56] B. Poole, S. Ozair, A. Van Den Oord, A. Alemi, G. Tucker, On variational bounds of mutual information, in: International Conference on Machine Learning, 2019, p. 5171–5180. [57] R. Lowe, Y. Wu, A. Tamar, J. Harb, P. Abbeel, I. Mordatch, Multi-agent actor-critic for mixed cooperative-competitive environments, in: Advances in Neural Information Processing Systems, Vol. 30, 2017. [58] M. Samvelyan, T. Rashid, C. S. De Witt, G. Farquhar, N. Nardelli, T. G. J. Rudner, C.-M. Hung, P. H. S. Torr, J. Foerster, S. Whiteson, The starcraft multi-agent challenge, in: International Conference on Autonomous Agents and Multiagent Systems, 2019. [59] O. Mahjoub, S. Abramowitz, R. de Kock, W. Khlifi, S. du Toit, J. Daniel, L. Ben Nessir, L. Beyers, C. Formanek, L. Clark, A. Pretorius, Sable: a performant, efficient and scalable sequence model for multi-agent reinforcement learning, in: International Conference on Machine Learning, 2025. [60] K.-a. A. Tessera, A. Rahman, A. Storkey, S. V. Albrecht, Hypermarl: Adaptive hypernetworks for multi-agent rl, in: Advances in Neural Information Processing Systems, 2025. [61] C. D. Freeman, E. Frey, A. Raichuk, S. Girber, I. Mordatch, O. Bachem, Brax – a differentiable physics engine for large scale rigid body simulation, arXiv preprint arXiv:2106.13281 (2021). [62] I. Jang, J. Park, C. E. Mballo, S. Cho, C. J. Tomlin, H. J. Kim, EigenSafe: A spectral framework for learning-based stochastic safety filtering, arXiv preprint arXiv:2509.17750 (2025). [63] C. S. de Witt, T. Gupta, D. Makoviichuk, V. Makoviychuk, P. H. Torr, M. Sun, S. Whiteson, Is independent learning all you need in the StarCraft multi-agent challenge?, arXiv preprint arXiv:2011.09533 (2020). [64] B. L. Welch, The generalization of “student’s’ problem when several different population variances are involved, Biometrika 34 (1-2) (1947) 28–35.