Paper deep dive
Aggregate in the Advantage, Not the Ratio: A Canonical-Form Analysis of Cooperative Multi-Agent Policy Optimization
Zijian Zhao, Sen Li
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 7/21/2026, 2:36:47 AM
Summary
This paper analyzes cooperative Multi-Agent Reinforcement Learning (MARL) policy optimization, specifically focusing on how to aggregate information from neighboring agents in Proximal Policy Optimization (PPO) methods. The authors formalize advantage and ratio supports as matrices and prove that the expected gradient depends only on their product, establishing redundancy between the two. Through variance analysis, they demonstrate that aggregating rewards in the advantage (additive variance) is statistically superior to aggregating likelihood ratios in the importance weight (multiplicative variance). The proposed design principle is to aggregate neighbors in the advantage while keeping the importance ratio per-agent to minimize estimator variance.
Entities (10)
Relation Signals (9)
Advantage Support → aggregates → Rewards
confidence 95% · the advantage aggregates rewards as a sum
Ratio Support → aggregates → Likelihood Ratios
confidence 95% · the ratio aggregates likelihood ratios as a product
Paper Authors → proposes → Aggregate in Advantage, Not Ratio
confidence 95% · The resulting design principle is unambiguous: aggregate neighbors in the advantage... and keep the ratio per-agent.
Advantage Support → hasvariancetype → Additive Variance
confidence 90% · the advantage aggregates rewards as a sum (additive variance
Ratio Support → hasvariancetype → Multiplicative Variance
confidence 90% · the ratio aggregates likelihood ratios as a product (multiplicative variance
PPO → isexampleof → Multi-Agent Reinforcement Learning
confidence 90% · Multi-agent policy optimization, exemplified by PPO-based methods, is a key branch of cooperative Multi-Agent Reinforcement Learning (MARL).
HAPPO → uses → Sequential Ratios
confidence 90% · HAPPO employs sequential ratios with per-agent advantages
MAPPO → uses → Team-level Advantage
confidence 90% · MAPPO pairs a team-level advantage with per-agent ratios
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Multi-agent policy optimization, exemplified by PPO-based methods, is a key branch of cooperative Multi-Agent Reinforcement Learning (MARL). A central design question is how many neighboring agents\footnote{In this paper, "neighbors" refer not only to physical proximity but also to agents whose actions influence one another.} to aggregate in order to effectively utilize global information for cooperation. This decision must be made along two dimensions: in the advantage (which agents' rewards contribute to the credit signal) and in the ratio (which agents' likelihood ratios form the clipped importance weight). Existing methods occupy scattered, underexplored points on these two axes: IPPO treats both separately; MAPPO pairs a team-level advantage with per-agent ratios; HAPPO employs sequential ratios with per-agent advantages; and single-agent reductions operating on factorized joint policies aggregate both into fully joint products. We formalize these two design choices as support matrices $\SA$ and $\SR$, and prove a canonical structure: the expected multi-agent policy optimization objective depends on the pair $(\SA,\SR)$ only through their matrix product $\tS=\SR\SA$. This yields two key consequences: (i) Redundancy: the two support matrices are interchangeable with respect to the signal, meaning neither aggregation pattern is inherently superior.(ii) Variance Ordering: the advantage aggregates rewards as a sum (additive variance with an interior bias-variance optimum at the coupling neighborhood), whereas the ratio aggregates likelihood ratios as a product (multiplicative variance that grows exponentially with support size, with no accompanying bias reduction). The resulting design principle is unambiguous: aggregate neighbors in the advantage, sized to the coupling neighborhood, and keep the ratio per-agent.
Tags
Links
- Source: https://arxiv.org/abs/2607.17924v1
- Canonical: https://arxiv.org/abs/2607.17924v1
Trouble viewing inline? Open PDF directly →
Full Text
90,652 characters extracted from source content.
Expand or collapse full text
Aggregate in the Advantage, Not the Ratio: A Canonical-Form Analysis of Cooperative Multi-Agent Policy Optimization Zijian Zhao1, Sen Li1,2 1The Hong Kong University of Science and Technology 2The Hong Kong University of Science and Technology (Guangzhou) Corresponding Author: Sen Li Abstract Multi-agent policy optimization, exemplified by PPO-based methods, is a key branch of cooperative Multi-Agent Reinforcement Learning (MARL). A central design question is how many neighboring agents111In this paper, ”neighbors” refer not only to physical proximity but also to agents whose actions influence one another. to aggregate in order to effectively utilize global information for cooperation. This decision must be made along two dimensions: in the advantage (which agents’ rewards contribute to the credit signal) and in the ratio (which agents’ likelihood ratios form the clipped importance weight). Existing methods occupy scattered, underexplored points on these two axes: IPPO treats both separately; MAPPO pairs a team-level advantage with per-agent ratios; HAPPO employs sequential ratios with per-agent advantages; and single-agent reductions operating on factorized joint policies aggregate both into fully joint products. We formalize these two design choices as support matrices SAS^A and SRS^R, and prove a canonical structure: the expected multi-agent policy optimization objective depends on the pair (SA,SR)(S^A,S^R) only through their matrix product S~=SRSA S=S^RS^A. This yields two key consequences: (i) Redundancy: the two support matrices are interchangeable with respect to the signal, meaning neither aggregation pattern is inherently superior. (i) Variance Ordering: the advantage aggregates rewards as a sum (additive variance with an interior bias-variance optimum at the coupling neighborhood), whereas the ratio aggregates likelihood ratios as a product (multiplicative variance that grows exponentially with support size, with no accompanying bias reduction). The resulting design principle is unambiguous: aggregate neighbors in the advantage, sized to the coupling neighborhood, and keep the ratio per-agent. This explains why neighbor-based advantages are more prevalent than neighbor-based ratios in prior heuristic and empirical designs. We prove these results under specified assumptions and validate them across four carefully designed synthetic cooperative games and a real-world large-scale traffic-signal control task. The code for our experiments is available at https://github.com/RS2002/MAPO. 1 Introduction Cooperative Multi-Agent Reinforcement Learning (MARL) built on policy optimization, represented by Proximal Policy Optimization (PPO) (Schulman et al., 2017) and Trust Region Policy Optimization (TRPO) (Schulman et al., 2015a), faces a recurring question when updating each agent: How many other agents should be aggregated into that agent’s update? This question is easy to overlook because it is answered twice, in two different places: • Advantage support SAS^A: which agents’ rewards are summed into agent i’s advantage—the credit signal that indicates how good the joint action was for i; and • Ratio support SRS^R: which agents’ likelihood ratios are multiplied into agent i’s clipped importance weight—the trust-region object that keeps the update near the behavior policy. Both range from per-agent (aggregate no one else) to joint (aggregate everyone), and existing methods sit at different points on each: IPPO (De Witt et al., 2020) keeps both per-agent, while MAPPO (Yu et al., 2022) pairs a team advantage (all rewards) with a per-agent ratio; Traffic and networked methods use neighborhood advantages (Chu et al., 2019) with per-agent ratios; Sequential learning methods employ joint sequential ratio products while keeping the advantage per-agent (Kuba et al., 2022; Zhong et al., 2024); Casting the whole team as a single multi-action agent and running vanilla PPO on the factorized joint policy (ratio) and joint advantage yields a route taken by centralized single-agent reductions of MARL (Zhao et al., 2026; Zhao and Li, 2026). Yet the two supports have never been analyzed together, and there is no principle for where to sit on either, which this paper aims to address through canonical-form analysis. Our starting point is that the two supports are not independent degrees of freedom. We prove (Theorem 1) that the expected multi-agent policy gradient depends on the pair (SA,SR)(S^A,S^R) only through their matrix product S~=SRSA S=S^RS^A. This is a gauge freedom: SAS^A and SRS^R are interchangeable up to their product, so neither support is intrinsically redundant; any cross-agent aggregation placed in the ratio can equally be placed in the advantage, and vice versa (Corollary 1). What breaks this symmetry is variance, not signal. Aggregating rewards through the advantage is a sum of bounded terms (additive variance), whereas aggregating ratios is a product of importance weights, whose variance compounds multiplicatively in the number of factors (Lemma 2). Since the two routes realize the same expected gradient but the ratio route strictly inflates variance, the variance-optimal choice is to place all cross-agent aggregation in the advantage and keep the ratio per-agent (Corollary 2). It is in this precise, variance-ordered sense that cross-agent importance ratios are redundant: they add no signal that the advantage cannot, and cost variance that the advantage does not. This also explains a standing asymmetry in previous empirical and heuristic designs: neighborhood advantages are widely used, while local ratios are comparatively rare. In conclusion, we contribute: (i) a canonical form showing that the two supports enter the expected gradient only through S~=SRSA S=S^RS^A, establishing their mutual redundancy; (i) a variance-ordering that breaks the tie—advantage sums are additive while ratio products are exponential—yielding the design rule: aggregate in the advantage and keep the ratio per-agent; and (i) an empirical validation across synthetic games and a real-world traffic signal control task that pin down when each effect appears. We discuss connections to prior work throughout, and provide an extended treatment in Appendix D. 2 Preliminaries Single-agent policy optimization. PPO (Schulman et al., 2017) optimizes a policy π _ θ by taking several gradient steps on a batch collected under a behavior policy πold _ θ_old, correcting the off-policy mismatch with a clipped importance ratio ϱ()=π(a∣s)/πold(a∣s) ( θ)= _ θ(a s)/ _ θ_old(a s) and the advantage A, estimated by Generalized Advantage Estimation (GAE) (Schulman et al., 2015b): LPPO()=[min(ϱA,clip(ϱ,1−ϵ,1+ϵ)A)].L^PPO( θ)=E [ ( A,\ clip( ,1-ε,1+ε)A ) ]. (1) Two objects drive the update: the advantage A (the credit signal) and the ratio ϱ (the trust-region weight). Cooperative multi-agent policy optimization. In a cooperative Markov game (Littman, 1994) (formalized in Appendix A) with n agents, each agent i has a policy πiπ^i_ θ; under Centralized-Training with Decentralized-Execution (CTDE) and Centralized-Training with Centralized-Execution (CTCE) schemes alike (Jin et al., 2025), the joint policy factorizes as π(∣s)=∏iπi(ai∣s) _ θ( a s)= _iπ^i_ θ(a^i s). Lifting (1) to n agents requires two design decisions that are usually made implicitly: 1. Which reward drives agent i’s advantage? MAPPO (Yu et al., 2022) uses the team advantage (all agents share one A built from the global reward); independent learners (IPPO) (De Witt et al., 2020) use each agent’s own reward; value-decomposition, difference-reward, and neighbor-advantage methods sit in between. 2. Whose ratios enter agent i’s clipped weight? Independent and decentralized actors (IPPO, MAPPO) clip a per-agent ratio ϱi ^i. A fully centralized controller that treats π=∏iπiπ= _iπ^i as one policy clips the joint ratio ∏jϱj _j ^j, in which every factor is differentiated jointly (Zhao et al., 2026). Sequential methods such as HAPPO (Kuba et al., 2022) form a compound ratio ∏j≤iϱj _j≤ i ^j. We study the jointly-differentiated case, whose supports range from per-agent (SR=IS^R=I) to fully joint (SR=⊤S^R=11 ). These are the two knobs this paper isolates. We call the first the advantage support and the second the ratio support, and formalize both as 0/10/1 matrices in Section 3.1. Why the choice matters: a bias–variance tension. Both supports can reduce the same bias, but they pay for it very differently—a tension we preview here with a direct measurement of the multi-agent policy gradient, deferring the full study to Sec. 4. On a single-step dense pairwise game (n=20n=20 agents, K=4K=4 actions each, ring coupling of radius ρ⋆=4ρ =4; full definition in Sec. 4.1), we fix a reference policy, estimate the true team-return policy gradient by Monte Carlo, and then form the clipped-surrogate gradient estimator induced by a given pair of supports. For that estimator we report three quantities (all of the gradient estimator, in log scale): its squared bias, variance, and their sum, i.e., the Mean Square Error (MSE). Here a support of radius ρ means each agent aggregates its ρ-hop neighborhood on the coupling graph: ρ=0ρ=0 is per-agent (aggregate no one else), ρ=ρ⋆ρ=ρ exactly covers the true coupling neighborhood, and larger ρ over-aggregates. Panels (a,b) vary one support at a time—the advantage radius ρA _A with the ratio held per-agent (a), and the ratio radius ρR _R with the advantage held per-agent (b). The results show that the two bias curves are identical: enlarging either support removes the same missing-coupling bias, so on the expected gradient the two supports are interchangeable (a redundancy we prove in Sec. 3.1). The two variance curves, in contrast, differ sharply: the advantage aggregates rewards additively, so its variance grows gently and its MSE bottoms out near the coupling radius ρ⋆ρ , whereas the ratio aggregates likelihood ratios multiplicatively, so its variance grows far faster and its MSE turns up at a smaller radius. Panels (c,d) isolate this: they fix a matched effective support (so both realize the same expected gradient) and compare the two ways of reaching it—aggregating through the advantage (path P) versus through the ratio (path Q)—plotting their bias (c) and variance (d) as the support grows; the bias tracks together while the variance separates by up to 9×9×. Same benefit, different cost—understanding why, and what it implies for where to sit on each axis, is the question we take up. Figure 1: Bias, variance, and MSE of the multi-agent policy-gradient estimator on a single-step dense-pairwise game, as a function of the aggregation support (log axes throughout). (a) Sweep of the advantage-support radius ρA _A (with a per-agent ratio) and (b) sweep of the ratio-support radius ρR _R (with a per-agent advantage): each panel plots the estimator’s squared bias, its variance, and their sum (MSE), with the MSE minimum marked (∘ ) and the coupling radius ρ⋆=4ρ =4 shown dashed. (c,d) At a matched effective support S~=ring S=ring of increasing size, the advantage path P (SA=ring,SR=IS^A=ring,S^R=I) and the ratio path Q (SA=I,SR=ringS^A=I,S^R=ring), which share the same expected gradient: (c) squared bias and (d) variance of each path, the annotations in (d) giving the ratio Var()/Var()Var(Q)/Var(P). 3 Analysis: Two Supports, One Product 3.1 Problem Setup We first analyze a single decision step; Remark 6 and the episodic experiment (Sec. 4) provide the finite-horizon extension. Agents, policies, actions. Let =1,…,nN=\1,…,n\ be the set of agents and =1,…,KA=\1,…,K\ a finite action set. Agent i has policy πi(⋅)∈Δ()π^i_ θ(·)∈ (A) with its own parameter block i θ^i; the full parameter is =(1,…,n) θ=( θ^1,…, θ^n) with disjoint blocks. The joint policy factorizes as π()=∏i=1nπi(ai),=(a1,…,an)∈n. _ θ( a)= _i=1^nπ^i_ θ(a^i), a=(a^1,…,a^n) ^n. (2) Rewards and coupling. Agent i receives reward ri()r_i( a); the team return is R()=∑iri()R( a)= _ir_i( a). The coupling graph has symmetric adjacency C∈0,1n×nC∈\0,1\^n× n with Cii=1C_i=1, where Cij=1C_ij=1 iff rir_i depends on aja^j. We write ∂i=j:Cij=1∂ i=\j:C_ij=1\. (For simplicity, we omit the dependence on the state in the reward notation here.) Two supports. Fix a behavior parameter old θ_old and, for a candidate θ, define the per-agent likelihood ratio ϱj()=πj(aj)/πoldj(aj) _j( a)=π^j_ θ(a^j)/π^j_ θ_old(a^j). Let SA,SR∈0,1n×nS^A,S^R∈\0,1\^n× n be symmetric support matrices with Sii=1S_i=1. The advantage and importance weight of agent i are given by Ai=∑j:SijA=1rj()−bi,wi()=∏j:SijR=1ϱj(),A_i= _j:\,S^A_ij=1\,r_j( a)-b_i, w_i( θ)=\!\! _j:\,S^R_ij=1\!\! _j( a), (3) with bib_i any control variate independent of a. Special cases include SA=IS^A=I (independent advantage), SA=⊤S^A=11 (team advantage), SR=IS^R=I (per-agent ratio), and SR=⊤S^R=11 (joint ratio ∏jϱj _j _j). Surrogate and gradient. The (unclipped) surrogate is L()=∑i∼πold[wi()Ai]L( θ)= _iE_ a _ θ_old[w_i( θ)A_i], with AiA_i detached (constant in θ). Since ∇wi=wi∑j:SijR=1∇logπj(aj) _ θw_i=w_i _j:S^R_ij=1 _ θ π^j(a^j) and ∇mlogπj=0 _ θ^m π^j=0 for j≠mj≠ m, we have m()=∇mL=∼πold[(∑i:SimR=1wi()Ai)∇mlogπm(am)]. g_m( θ)= _ θ^mL=E_ a _ θ_old [ ( _i:\,S^R_im=1w_i( θ)A_i ) _ θ^m π^m(a^m) ]. (4) 3.2 Assumptions and Their Justification Assumption 1 (Factorized policy / conditional action independence). Under both old θ_old and any candidate θ, the joint policy factorizes as in (2); equivalently, given the parameters, actions are sampled independently across agents. Justification. This is not an extra modeling restriction but rather the defining structure of the methods under study. In most MARL algorithms (e.g. IPPO, MAPPO, HAPPO) each agent samples ai∼πi(⋅∣oi)a^i π^i(· o^i) independently by construction, so (2) holds exactly. The assumption would fail only for policies with an explicitly autoregressive action head (e.g., a decoder that conditions aia^i on realized a<ia^<i); we exclude that case and note that it corresponds to a different (chain-rule) factorization (Wen et al., 2022). Importantly, Assumption 1 concerns conditional independence given parameters (and context); it does not claim that the agents’ actions are marginally uncorrelated—they are correlated through shared observations and, during learning, through the coupled reward. Assumption 2 (Full support / finite second moment). πi(a)>0π^i_ θ(a)>0 for all i,ai,a and all θ in a neighborhood of old θ_old, so each ϱj _j is finite with [ϱj2]<∞E[ _j^2]<∞. Justification. Softmax policies, the standard parameterization, satisfy full support automatically. Finite second moments are required for any importance-weighted estimator to have finite variance and are standard in PPO and TRPO analysis. Assumption 3 (Symmetric coupling). C is symmetric with Cii=1C_i=1, and rir_i depends on aja^j iff Cij=1C_ij=1. Justification. Symmetry holds whenever coupling arises from shared or pairwise terms (shared resources, pairwise congestion or collision, common team reward), which covers the cooperative tasks of interest. The directional case is a routine extension (Remark 5) obtained by replacing C with the directed influence matrix; the main theorem does not require symmetry, which we adopt only to state the coupling radius cleanly. Assumption 4 (Bounded rewards). |ri()|≤rmax<∞|r_i( a)|≤ r_ <∞ for all i,i, a. Justification. This is standard and holds for any bounded reward; by truncation, it also applies to sub-Gaussian rewards up to negligible tails. It ensures that the advantage-side aggregation has variance O(|SA|)O(|S^A|) rather than exponential. 3.3 Ratio Moments Lemma 1 (Unbiased weight). Under Assumptions 1–2, for any S⊆S , ∼πold[∏j∈Sϱj]=1E_ a _ θ_old [ _j∈ S _j ]=1; in particular [wi]=1E[w_i]=1 for every ratio support. This follows because the behavior-policy expectation factorizes over agents and each per-agent ratio has unit mean; a proof is provided in Appendix B.1. Lemma 2 (Multiplicative variance). Under Assumptions 1–2, with χj2:=χ2(πj∥πoldj)=[ϱj2]−1≥0χ^2_j:=χ^2(π^j_ θ\,\|\,π^j_old)=E[ _j^2]-1≥ 0, Var(∏j∈Sϱj)=∏j∈S(1+χj2)−1.Var ( _j∈ S _j )= _j∈ S(1+χ^2_j)-1. (5) Hence the weight variance is nondecreasing in S, and if χj2≥c>0χ^2_j≥ c>0 then it is ≥(1+c)|S|−1≥(1+c)^|S|-1, i.e., exponential in |S||S|. The multiplicative form follows from the independence of the per-agent ratios, which causes the second moments to factorize; the full derivation is given in Appendix B.2. Remark 1. At =old θ= θ_old, every χj2=0χ^2_j=0 and wi≡1w_i≡ 1, so the ratio only activates once θ departs from old θ_old, i.e., across PPO’s inner epochs. This is precisely why the two supports coincide on-policy (Thm. 1) yet diverge off-policy (Prop. 1). 3.4 Main results 3.4.1 The expected gradient factorizes through S~=SRSA S=S^RS^A Theorem 1 (Support factorization). Under Assumptions 1–3, at =old θ= θ_old the expected gradient (4) is m=∑j:Cjm=1S~mj[rj∇mlogπm(am)],S~=SRSA, g_m= _j:\,C_jm=1 S_mj\,E [r_j\, _ θ^m π^m(a^m) ], S=S^RS^A, (6) where S~ S is the matrix product of SRS^R and SAS^A. Thus m g_m depends on (SA,SR)(S^A,S^R) only through S~ S: any two support pairs with the same product SRSAS^RS^A induce the same expected gradient. The key step evaluates the surrogate gradient at old θ_old, where wi=1w_i=1, and collects terms by the score-function identity; the complete proof is in Appendix B.3. 3.4.2 Canonical form and redundancy of cross-agent ratios Corollary 1 (Per-agent ratio canonical form). Under Theorem 1, every estimator with supports (SA,SR)(S^A,S^R) has, at the on-policy point, the same expected gradient as the per-agent-ratio estimator (SR=I,SA=S~)(S^R=I,\ S^A= S) whose advantage is the reweighted quantity Am′=∑jS~mjrjA _m= _j S_mjr_j with S~=SRSA S=S^RS^A. In particular the joint ratio SR=⊤S^R=11 paired with advantage SAS^A is gradient-equivalent to a per-agent ratio with advantage reweighted by ⊤SA11 S^A. Hence cross-agent importance ratios add no expected-gradient signal beyond a linear reweighting of the advantage. This is immediate from Theorem 1 by reading off the coefficient of each score term (Appendix B.4). Remark 2 (The team-advantage scalar). With team advantage SA=⊤S^A=11 , per-agent ratio gives S~=⊤ S=11 while joint ratio gives S~=⊤⊤=n 11⊤ S=11 11 =n\,11 . The two expected gradients are therefore identical up to the global scalar n: the joint ratio merely rescales the effective step by the support size. Any fair comparison must control for this scalar (we do so in Sec. 4 by matching effective step size). 3.4.3 Variance Domination We now compare the two canonical realizations of a target product S~=C S=C: (P) advantage path: SA=C,SR=I;(Q) ratio path: SA=I,SR=C.(P) advantage path: S^A=C,\ S^R=I; (Q) ratio path: S^A=I,\ S^R=C. (7) Both realize the same expected gradient ∑j∈∂m[rj∇logπm] _j∈∂ mE[r_j∇ π^m] by Theorem 1. Proposition 1 (Off-policy variance domination). Under Assumptions 1–4: (i) at =old θ= θ_old, P and Q have identical mean and identical variance; (i) for ≠old θ≠ θ_old, the P-weight ∑j∈∂mrj _j∈∂ mr_j is bounded by |∂m|rmax|∂ m|\,r_ with variance independent of θ, whereas the Q-weight ∑i∈∂m(∏j∈∂iϱj)ri _i∈∂ m ( _j∈∂ i _j )r_i has second moment [(∏j∈∂iϱj)2]=∏j∈∂i(1+χj2)E[( _j∈∂ i _j)^2]= _j∈∂ i(1+χ^2_j), so once any χj2>0χ^2_j>0 its variance is bounded below by a positive multiple of ∏j∈∂m(1+χj2)−1 _j∈∂ m(1+χ^2_j)-1. Hence Var(mQ)/Var(mP)→∞Var( g^Q_m)/Var( g^P_m)→∞ as the policy step or |∂m||∂ m| grows. The bound follows by comparing the bounded P-weight to the Q-weight’s exploding second moment from Lemma 2; see Appendix B.5. Remark 3 (Clipping widens the gap). Reinstating the PPO clip applies a per-weight, 11-Lipschitz truncation. For P, the clipped object is a single low-variance ratio ϱm _m, so clipping rarely activates; for Q, it is the heavy-tailed product (Lemma 2), which triggers clipping more often, injecting truncation bias and leaving the retained variance elevated. Clipping thus cannot reverse Proposition 1 and typically amplifies it. Corollary 2 (Design rule). Among all (SA,SR)(S^A,S^R) that realize a target unbiased product S~ S, the choice SR=IS^R=I (per-agent ratio) with SA=S~S^A= S minimizes the gradient-estimator variance. In particular, the joint (compound) ratio is weakly dominated on-policy and strictly dominated off-policy: it should be replaced by a per-agent ratio with the corresponding advantage reweighting. 3.4.4 The Advantage Support: A Bias–Variance Tradeoff At a matched effective support, the ratio support does not provide any bias reduction advantage over the advantage support: any bias it can remove is already removed by the advantage support at strictly lower variance (Corollary 1, Prop. 1). Consequently, its variance-optimal setting is the per-agent boundary SR=IS^R=I. The advantage support is qualitatively different: with the ratio fixed at per-agent, it trades bias against variance solely through SAS^A, and this tradeoff has an interior optimum. This asymmetry between the two supports is fundamental, and we can characterize the advantage side as sharply as the ratio side. Proposition 2 (Advantage-support bias–variance tradeoff). Fix the per-agent ratio SR=IS^R=I and consider the on-policy estimator m g_m with advantage support SAS^A. Write the true (team-return) gradient block as m⋆=∑j∈∂m[rj∇mlogπm] g_m = _j∈∂ mE[r_j _ θ^m π^m]. Then, under Assumptions 1–4: (i) (Bias) [m]=∑j∈∂mSmjA[rj∇mlogπm]E[ g_m]= _j∈∂ mS^A_mj\,E[r_j _ θ^m π^m], so m g_m is unbiased iff SmjA=1S^A_mj=1 for every coupled agent j∈∂mj∈∂ m; a support that misses a coupled agent omits its term and is biased. (i) (Variance) Including an uncoupled agent j∉∂mj∉∂ m leaves the mean unchanged (its term has zero expectation by the score-function identity) but adds a nonnegative variance contribution, strictly positive whenever rjr_j is conditionally non-degenerate given ama^m. Consequently, the mean-squared-error-optimal advantage support is exactly the coupling neighborhood ∂m∂ m: smaller supports are biased, larger supports inflate variance. The bias and variance terms are computed separately and their sum minimized; the derivation is in Appendix B.6. Remark 4 (The asymmetry, precisely). Propositions 1 and 2 together establish the central asymmetry of the paper as a theoretical result, not merely an empirical observation. With the ratio fixed to per-agent, the advantage support has a bias term that larger supports remove, yielding an interior MSE optimum at the coupling neighborhood. In contrast, the ratio support offers no bias-reduction benefit: by the canonical form (Corollary 1), any bias reduction it could achieve is equally attainable through the advantage, and by Proposition 1, at strictly lower variance. Hence, on the variance-optimal frontier, the ratio is never used to reduce bias, and its optimum is the boundary SR=IS^R=I. Although the two knobs appear symmetric in the surrogate (3), they play categorically different roles. Remark 5 (Directional-coupling extension). If coupling is directional, replace C by the directed influence matrix D with Dij=1D_ij=1 iff rir_i depends on aja^j. Theorem 1 holds with the survival condition Djm=1D_jm=1 and S~=SRSA S=S^RS^A; the variance-optimal design uses SR=IS^R=I, SA=D⊤S^A=D . Remark 6 (Finite-horizon extension). In a finite-horizon Markov game, the argument applies per time step to the GAE advantage and the per-step ratio. The multiplicative variance of Lemma 2 then compounds across agents and time, so the joint ratio’s variance grows in n⋅Hn· H; the on-policy factorization of Theorem 1 is unchanged. The episodic experiment (Sec. 4) confirms the redundancy in this setting. 4 Experiments 4.1 Experiment Setup The theory yields three testable predictions. (R) Redundancy: estimators with the same product S~=SRSA S=S^RS^A have identical expected gradients. (D) Domination: realizing a fixed S~ S through the ratio product incurs strictly higher variance off-policy than through the advantage sum, with the gap growing multiplicatively in the support size. (Fig. 1) (A) Asymmetry: at the gradient-estimator level, the advantage support presents a bias–variance tradeoff with an interior MSE optimum at the coupling neighborhood (Prop. 2), whereas the ratio support has no bias effect and a boundary optimum at ρR=0 _R=0 (Prop. 1). During training, the advantage radius influences the learned policy only insofar as bias alters the equilibrium—decisive when the externality is not self-internalized, mild otherwise—while the ratio radius, being bias-free, never changes the learned policy and affects only stability. We test these predictions through a multi-family training study that traces how each support affects the learned policy across four coupling structures and a scaling study that identifies the regime in which the ratio’s variance penalty becomes manifest. All environments are cooperative games with a prescribed coupling graph C, so the ground-truth coupling neighborhood is known exactly. Each agent i observes a phase or context and selects ai∈1,…,Ka^i∈\1,…,K\; per-agent rewards rir_i are defined in Appendix C.1, and the team return is R=∑iriR= _ir_i; ring neighborhoods of radius ρ⋆ρ define ∂i∂ i. We use four game families that span the regimes distinguished by the theory: a dense pairwise game (symmetric payoff over a ring neighborhood, with internalized externality); a directed dilemma, where a high-value action imposes a nuisance on downstream successors that the actor does not bear (a non-internalized externality); local congestion, where neighbors choosing the same resource split its value (crowding partly self-borne); and a block community game with all-to-all coupling inside disjoint blocks (a non-geometric graph). A more detailed description is provided in Appendix C.1. In addition, we provide a real-world large-scale traffic signal control validation in Appendix C.3.2. 4.2 Experiment Results We first examine PPO training across the four coupling families, which differ in how agents interact. Each is a finite-horizon Markov game with phase-conditioned tabular actors trained by PPO (clip 0.20.2); we sweep the advantage radius ρA _A (with per-agent ratio fixed) and, separately, the ratio radius ρR _R (with team advantage fixed). Curves are converged over the final 4040 iterations and averaged over 55 seeds. The advantage radius selects the outcome only for non-internalized externalities (Fig. 2). Figure 2 plots, for each of the four coupling families, the final team return as a function of the advantage radius ρA _A (per-agent ratio fixed), with the true coupling radius ρ⋆ρ marked. The four families separate sharply. In the directed dilemma, the advantage radius is decisive: independent advantage (ρA=0 _A=0) converges to the tragedy equilibrium (team return −159-159), while any ρA≥1 _A≥ 1 internalizes enough of the successor externality to reach the social optimum (+158+158). In the other three families, the effect is mild and the outcome-optimal radius is small (ρA∈0,1 _A∈\0,1\): when an agent already bears (part of) the coupling it induces—symmetric pairwise payoffs, shared congestion, within-block rewards—the independent advantage is nearly unbiased, so enlarging ρA _A buys little signal and adds variance, and the return is flat or slightly decreasing in ρA _A. This is the training-level counterpart of the estimator tradeoff in Fig. 1a: the advantage radius should match the coupling neighborhood, but the outcome moves only when the un-internalized part of the externality is large enough to change the equilibrium. Full training curves for all four families are provided in Appendix C.2. Figure 2: Advantage-support sweep across the four coupling families: final team return versus the advantage radius ρA _A, with the ratio held per-agent. One panel per family; the dashed vertical line marks the coupling radius ρ⋆ρ . Bands indicate ±1± 1 standard deviation across seeds; in the directed dilemma, the standard deviation is below 0.40.4 and smaller than the marker, as the two equilibria are reached near-deterministically. Moreover, redundancy implies that the ratio radius carries no benefit; Proposition 1 states that it carries a variance cost that grows multiplicatively with support size and with the per-update policy shift χ2χ^2. Whether that cost is visible in training therefore depends on how far off-policy each update travels—which a trust-region method controls through its step size. We make this explicit by scaling the number of agents under two configurations that differ only in aggressiveness: a standard PPO setting (lr=3×10−3lr=3× 10^-3, batch 6464) and an off-policy stress setting (lr=0.15lr=0.15, batch 1616); both use the team advantage and match the effective step across ratio supports (Remark 2), so any difference is due to variance, not step size. Masked under conservative updates, revealed under stress (Fig. 3). Figure 3 varies the number of agents n (log axis) for each family and reports two quantities: the joint/per-agent final return ratio (solid, left axis)—indicating how much return the joint ratio sacrifices relative to a per-agent ratio—and the joint ratio’s clip fraction (dashed, right axis), under both the standard (blue) and off-policy stress (red) configurations. Under the standard configuration, the joint and per-agent ratios are statistically indistinguishable up to n=160n=160 (return ratio 1.001.00, joint clip fraction ≈0≈ 0), with a 44–8%8\% gap emerging only at n=320n=320: the trust region keeps χ2χ^2 so small (χj2≈6×10−4χ^2_j≈ 6× 10^-4 per update) that the variance factor (1+χ2)n−1(1+χ^2)^n-1 is negligible even for n in the hundreds. Under the stress configuration, the same experiment reproduces the folklore instability across all four families: the joint ratio’s clip fraction rises monotonically with n (reaching 0.20.2–0.450.45 at the largest n tested per family), while the per-agent ratio’s clip fraction remains at 0 throughout—the signature of Lemma 2. A product of n ratios leaves the trust region almost surely once n is large, so it is clipped increasingly often and its gradient is throttled, whereas a single ratio never is. Where the coupling is sharp enough that this throttling starves the update of signal—the directed dilemma and dense pairwise families—the return degrades with it. Where the reward is more forgiving (local congestion, block community), the same clipping leaves the final return closer to parity, making the clip fraction the more universal diagnostic. Figure 3: Agent-count scaling across the four coupling families. For each family and agent count n (log axis), solid curves report the joint/per-agent final return ratio (left axis), and dashed curves show the joint ratio’s clip fraction (right axis). Results are shown under a standard PPO configuration (blue; lr=3×10−3lr=3× 10^-3, batch 6464) and an off-policy stress configuration (red; lr=0.15lr=0.15, batch 1616); both use the team advantage and match the effective step across ratio supports. Error bars indicate ±1± 1 standard deviation over seeds (on the standard curves they are smaller than the markers). Remark 7 (Step size, not agent count, is the trigger). The variance factor (1+χ2)n−1(1+χ^2)^n-1 has two levers: the support size n and the per-update shift χ2χ^2. Fig. 3 shows they act together—the gap grows with n, but only once χ2χ^2 is non-negligible. A conservative learning rate makes χ2χ^2 vanishingly small, so even n in the hundreds is harmless; an aggressive rate makes it bite. This reconciles the theorem with the common observation that MAPPO’s per-agent ratio and various joint-ratio schemes often perform comparably: they do, precisely in the near-on-policy regime where the penalty is dormant. The per-agent ratio weakly dominates always and strictly dominates whenever updates are pushed off-policy—at no cost, since the benefit of a larger ratio support is exactly zero. Remark 8 (Why we recommend against the joint ratio, not merely note its risk). It might seem that, since the joint ratio is harmless in the near-on-policy regime, one could simply use it with a small step. We caution against this for a practical reason that our controlled environments understate. Here we know the coupling, the reward scale, and the effective off-policyness, so we can see that the penalty is dormant. In a real large-scale system, one does not know these in advance: the effective χ2χ^2 varies across states, agents, and training phases, and a minibatch that happens to be more off-policy—after a reward spike, an exploration burst, or a learning-rate warmup—can push the n-fold product out of the trust region and stall learning, with no diagnostic that distinguishes this from ordinary noise. Because the joint ratio offers zero upside over a per-agent ratio (Cor. 1) and an unbounded, hard-to-anticipate downside that grows with n, the asymmetry of the bet is decisive: keep the ratio per-agent. This is especially pertinent at the scales where cooperative MARL is deployed—traffic grids, sensor and robot fleets, power networks—where n is in the hundreds or thousands and the (1+χ2)n(1+χ^2)^n factor is one adverse batch away from biting. Our 196196-intersection traffic network bears this out directly: with n=196n=196, the joint ratio collapses even at standard settings, while per-agent and neighborhood ratios learn normally (Appendix C.3.2). 5 Conclusion In this paper, we decomposed the credit-assignment structure of multi-agent policy optimization into two independent design axes: an advantage support and a ratio support. Our analysis reveals that the expected policy gradient depends on these supports only through their matrix product S~=SRSA S=S^RS^A, establishing a gauge freedom that renders the two supports mutually redundant in terms of the gradient signal. This redundancy is broken by variance: aggregating through the advantage corresponds to a sum of rewards, yielding additive variance with an interior optimum at the coupling neighborhood, whereas aggregating through the ratio corresponds to a product of importance weights, incurring multiplicative, exponentially growing variance in the support size. These results lead to a clear and actionable design principle: aggregate neighbors in the advantage, sized according to the true reward coupling, and keep the ratio strictly per-agent. This rule applies universally to any factorized-policy PPO method and is supported by rigorous theoretical analysis. We validate our conclusions across four carefully designed cooperative MAMDP games and a real-world large-scale traffic signal control task, demonstrating both the correctness of the canonical form and the practical relevance of the variance asymmetry. More discussion is provided in Appendix E. References L. N. Alegre (2019) SUMO-RL. GitHub. Note: https://github.com/LucasAlegre/sumo-rl Cited by: §C.3. M. Behrisch, L. Bieker, J. Erdmann, and D. Krajzewicz (2011) SUMO–simulation of urban mobility: an overview. In Proceedings of SIMUL 2011, the third international conference on advances in system simulation, Cited by: §C.3. T. Chu, J. Wang, L. Codecà, and Z. Li (2019) Multi-agent deep reinforcement learning for large-scale traffic signal control. IEEE transactions on intelligent transportation systems 21 (3), p. 1086–1095. Cited by: Appendix D, §1. C. S. De Witt, T. Gupta, D. Makoviichuk, V. Makoviychuk, P. H. Torr, M. Sun, and S. Whiteson (2020) Is independent learning all you need in the starcraft multi-agent challenge?. arXiv preprint arXiv:2011.09533. Cited by: §1, item 1. J. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson (2018) Counterfactual multi-agent policy gradients. In Proceedings of the AAAI conference on artificial intelligence, Vol. 32. Cited by: Appendix D. W. Jin, H. Du, B. Zhao, X. Tian, B. Shi, and G. Yang (2025) A comprehensive survey on multi-agent cooperative decision-making: scenarios, approaches, challenges and perspectives. arXiv preprint arXiv:2503.13415. Cited by: §2. S. Kim, G. Park, W. Kim, J. Jeon, S. Han, and Y. Sung (2026) Generalized per-agent advantage estimation for multi-agent policy optimization. arXiv preprint arXiv:2603.02654. Cited by: Appendix D. J. Kuba, R. Chen, M. Wen, Y. Wen, F. Sun, J. Wang, and Y. Yang (2022) Trust region policy optimisation in multi-agent reinforcement learning. In ICLR 2022-10th International Conference on Learning Representations, p. 1046. Cited by: Appendix D, §1, item 2. K. Kurach, A. Raichuk, P. Stańczyk, M. Zając, O. Bachem, L. Espeholt, C. Riquelme, D. Vincent, M. Michalski, O. Bousquet, et al. (2020) Google research football: a novel reinforcement learning environment. In Proceedings of the AAAI conference on artificial intelligence, Vol. 34, p. 4501–4510. Cited by: §C.1.1. Y. Li, G. Xie, and Z. Lu (2022) Difference advantage estimation for multi-agent policy gradients. In International Conference on Machine Learning, p. 13066–13085. Cited by: Appendix D. M. L. Littman (1994) Markov games as a framework for multi-agent reinforcement learning. In Machine learning proceedings 1994, p. 157–163. Cited by: Appendix A, §2. G. Qu, Y. Lin, A. Wierman, and N. Li (2020a) Scalable multi-agent reinforcement learning for networked systems with average reward. Advances in Neural Information Processing Systems 33, p. 2074–2086. Cited by: Appendix D, Appendix E. G. Qu, A. Wierman, and N. Li (2020b) Scalable reinforcement learning of localized policies for multi-agent networked systems. In Learning for Dynamics and Control, p. 256–266. Cited by: Appendix D, Appendix E. J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz (2015a) Trust region policy optimization. In International conference on machine learning, p. 1889–1897. Cited by: §1. J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel (2015b) High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438. Cited by: §2. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Cited by: §1, §2. M. Wen, J. Kuba, R. Lin, W. Zhang, Y. Wen, J. Wang, and Y. Yang (2022) Multi-agent reinforcement learning is a sequence modeling problem. Advances in Neural Information Processing Systems 35, p. 16509–16521. Cited by: §3.2. S. Whiteson, M. Samvelyan, T. Rashid, C. De Witt, G. Farquhar, N. Nardelli, T. Rudner, C. Hung, P. Torr, and J. Foerster (2019) The starcraft multi-agent challenge. In Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS, p. 2186–2188. Cited by: §C.1.1. D. H. Wolpert and K. Tumer (2001) Optimal payoff functions for members of collectives. Advances in Complex Systems 4 (02n03), p. 265–279. Cited by: Appendix D. C. Yu, A. Velu, E. Vinitsky, J. Gao, Y. Wang, A. Bayen, and Y. Wu (2022) The surprising effectiveness of ppo in cooperative multi-agent games. Advances in neural information processing systems 35, p. 24611–24624. Cited by: Appendix D, §1, item 1. Z. Zhao, J. Gao, and S. Li (2026) Bridging marl to sarl: an order-independent multi-agent transformer via latent consensus. arXiv preprint arXiv:2604.13472. Cited by: Appendix E, Appendix E, §1, item 2. Z. Zhao and S. Li (2026) Triple-BERT: do we really need MARL for order dispatch on ride-sharing platforms?. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: §1. Y. Zhong, J. G. Kuba, X. Feng, S. Hu, J. Ji, and Y. Yang (2024) Heterogeneous-agent reinforcement learning. Journal of Machine Learning Research 25 (32), p. 1–67. Cited by: §1. Appendix Contents Appendix A MAMDP Formulation We formalize the cooperative setting as a multi-agent Markov decision process (MAMDP) (Littman, 1994), equivalently a cooperative Markov game. This appendix provides the full formulation summarized in the main text. Definition. A MAMDP is a tuple (,,ii∈,P,rii∈,γ,μ0) (N,S,\A_i\_i ,P,\r_i\_i ,γ, _0 ), where =1,…,nN=\1,…,n\ is the set of agents; S is the state space; iA_i is the finite action set of agent i, with joint action space =∏iA= _iA_i and joint action =(a1,…,an) a=(a^1,…,a^n); P(s′∣s,)P(s s, a) is the transition kernel; ri:×→ℝr_i:S×A is agent i’s reward, with team reward R(s,)=∑iri(s,)R(s, a)= _ir_i(s, a); γ∈[0,1)γ∈[0,1) is the discount factor; and μ0 _0 is the initial-state distribution. The setting is fully cooperative: all agents share the single team objective defined below. Factorized policy. Each agent acts through its own policy πi(ai∣s)∈Δ(i)π^i_ θ^i(a^i s)∈ (A_i) with a disjoint parameter block i θ^i, so =(1,…,n) θ=( θ^1,…, θ^n). Actions are conditionally independent across agents given the state, so the joint policy factorizes as π(∣s)=∏i=1nπi(ai∣s). _ θ( a s)= _i=1^nπ^i_ θ^i(a^i s). (8) This factorization—equivalently, independent action heads sharing a state encoder—is the single structural assumption used in our analysis; it also covers the centralized multi-action policy that treats the whole team as one agent whose action is a. Objective. The agents jointly maximize the expected discounted team return J()=s0∼μ0,t∼π(⋅∣st),st+1∼P(⋅∣st,t)[∑t≥0γtR(st,t)].J( θ)=E_s_0 _0,\; a_t _ θ(· s_t),\;s_t+1 P(· s_t, a_t)\! [ _t≥ 0γ^t\,R(s_t, a_t) ]. (9) Values and advantages. The state value is Vπ(s)=π[∑t≥0γtR(st,t)∣s0=s]V^π(s)=E_π\! [ _t≥ 0γ^tR(s_t, a_t) s_0=s ], and QπQ^π, the generalized advantage estimate (GAE), and the per-agent advantage are defined as usual from the per-agent rewards. The advantage support SA∈0,1n×nS^A∈\0,1\^n× n selects which agents’ rewards form agent i’s advantage: with A^j A_j the single-agent advantage built from rjr_j, agent i’s aggregated advantage is Ai=∑jSijAA^jA_i= _jS^A_ij A_j. Coupling graph. The coupling graph C∈0,1n×nC∈\0,1\^n× n has Cij=1C_ij=1 iff rir_i depends on aja^j (with Cii=1C_i=1). Its smallest neighborhood radius ρ⋆ρ is the coupling radius; ∂i=j:Cij=1∂ i=\j:C_ij=1\ is the coupling neighborhood of agent i. Clipped surrogate with two supports. Fixing a behavior policy old θ_old, the per-agent likelihood ratio is ϱj=πj(aj∣s)/πoldj(aj∣s) ^j=π^j_ θ(a^j s)/π^j_ θ_old(a^j s). The ratio support SR∈0,1n×nS^R∈\0,1\^n× n selects which agents’ ratios enter agent i’s importance weight, wi=∏j:SijR=1ϱjw_i= _j:S^R_ij=1 ^j, and the (clipped) multi-agent PPO surrogate is ∑i[min(wiAi,clip(wi,1±ϵ)Ai)] _iE [ (w_iA_i,\ clip(w_i,1±ε)A_i) ]. The pair (SA,SR)(S^A,S^R) is the object of study: SA=SR=IS^A=S^R=I corresponds to IPPO, SA=⊤,SR=IS^A=11 ,S^R=I to MAPPO, and SA=SR=⊤S^A=S^R=11 to the fully centralized multi-action reduction. Appendix B Proofs Throughout, all expectations are over ∼πold a _ θ_old unless noted, and we write ϱj=πj(aj)/πoldj(aj) _j=π^j_ θ(a^j)/π^j_ θ_old(a^j) for the per-agent likelihood ratio and sm:=∇mlogπm(am)s^m:= _ θ^m π^m(a^m) for the per-agent score. B.1 Proof of Lemma 1 (unbiased weight) Proof. Fix S⊆S . By Assumption 1, the actions ajj∈S\a^j\_j∈ S are independent under πold _ θ_old, so the expectation of the product factorizes: ∼πold[∏j∈Sϱj] _ a _ θ_old [ _j∈ S _j ] =∏j∈S∼πold[ϱj]=∏j∈S∑a∈πoldj(a)πj(a)πoldj(a) = _j∈ SE_ a _ θ_old [ _j ]= _j∈ S\ _a π^j_ θ_old(a)\, π^j_ θ(a)π^j_ θ_old(a) (10a) =∏j∈S∑a∈πj(a)=∏j∈S1=1, = _j∈ S\ _a π^j_ θ(a)= _j∈ S1=1, (10b) where (10a) uses independence (Assumption 1) and the definition of ϱj _j, and the cancellation in (10b) is valid because πoldj(a)>0π^j_ θ_old(a)>0 (Assumption 2); the last equality follows from the normalization of πjπ^j_ θ. Taking S=j:SijR=1S=\j:S^R_ij=1\ gives [wi]=1E[w_i]=1 for every ratio support. ∎ B.2 Proof of Lemma 2 (multiplicative variance) Proof. By Lemma 1, ∼πold[∏j∈Sϱj]=1E_ a _ θ_old[ _j∈ S _j]=1, so Var(∏j∈Sϱj)=∼πold[(∏j∈Sϱj)2]−1Var( _j∈ S _j)=E_ a _ θ_old[( _j∈ S _j)^2]-1. The squared product factorizes over agents by independence (Assumption 1): ∼πold[(∏j∈Sϱj)2]=∏j∈S∼πold[ϱj2], _ a _ θ_old [ ( _j∈ S _j )^2 ]= _j∈ SE_ a _ θ_old [ _j^2 ], (10) and ∼πold[ϱj2]=∑a∈πoldj(a)πj(a)2πoldj(a)2=∑aπj(a)2πoldj(a). _ a _ θ_old [ _j^2 ]= _a π^j_ θ_old(a) π^j_ θ(a)^2π^j_ θ_old(a)^2= _a π^j_ θ(a)^2π^j_ θ_old(a). (11) The rightmost sum is 1+χ2(πj∥πoldj)=1+χj21+χ^2\! (π^j_ θ\,\|\,π^j_ θ_old )=1+χ^2_j by the definition of the χ2χ^2-divergence. Substituting this into (11) and subtracting 11 gives (5): Var(∏j∈Sϱj)=∏j∈S(1+χj2)−1Var( _j∈ S _j)= _j∈ S(1+χ^2_j)-1. Since each factor 1+χj2≥11+χ^2_j≥ 1, the product is nondecreasing in S; if χj2≥c>0χ^2_j≥ c>0 for all j, then ∏j∈S(1+χj2)−1≥(1+c)|S|−1 _j∈ S(1+χ^2_j)-1≥(1+c)^|S|-1, which is exponential in |S||S|. ∎ B.3 Proof of Theorem 1 (support factorization) Proof. Evaluate the gradient (4) at =old θ= θ_old, where every ϱj=1 _j=1 and hence wi=1w_i=1. Writing Ai=∑jSijArj−biA_i= _jS^A_ijr_j-b_i, m g_m =∼πold[(∑i:SimR=1Ai)sm]=∑i:SimR=1∑jSijA∼πold[rjsm]−∑i:SimR=1bi∼πold[sm]. =E_ a _ θ_old [ ( _i:S^R_im=1A_i )\,s^m ]= _i:S^R_im=1\ _jS^A_ij\,E_ a _ θ_old [r_j\,s^m ]\;-\; _i:S^R_im=1b_i\,E_ a _ θ_old [s^m ]. (12) The baseline term vanishes: bib_i is constant in a and ∼πold[sm]=∑amπoldm(am)∇mlogπm(am)=∇m∑amπm(am)=∇m1=0E_ a _ θ_old[s^m]= _a^mπ^m_ θ_old(a^m) _ θ^m π^m(a^m)= _ θ^m _a^mπ^m(a^m)= _ θ^m1=0. Next, by Assumption 3 (symmetric coupling) and conditional independence, ∼πold[rjsm]=0E_ a _ θ_old[r_j\,s^m]=0 whenever Cjm=0C_jm=0: if rjr_j does not depend on ama^m, conditioning on a−ma^-m gives ∼πold[rjsm]=∼πold[rj∼πold[sm∣a−m]]=∼πold[rj]⋅∼πold[sm]=0, _ a _ θ_old [r_j\,s^m ]=E_ a _ θ_old [r_j\,E_ a _ θ_old [s^m a^-m ] ]=E_ a _ θ_old [r_j ]·E_ a _ θ_old [s^m ]=0, (13) using ∼πold[sm∣a−m]=∼πold[sm]=0E_ a _ θ_old[s^m a^-m]=E_ a _ θ_old[s^m]=0 (action independence). Substituting (13) into (12) and exchanging the order of the i and j sums, m=∑j:Cjm=1(∑iSimRSijA)∼πold[rjsm]=∑j:Cjm=1(SRSA)mj∼πold[rjsm], g_m= _j:\,C_jm=1 ( _iS^R_imS^A_ij )E_ a _ θ_old [r_j\,s^m ]= _j:\,C_jm=1 (S^RS^A )_mj\,E_ a _ θ_old [r_j\,s^m ], (14) where the last step uses SimR=SmiRS^R_im=S^R_mi (symmetry), so ∑iSmiRSijA=(SRSA)mj=S~mj _iS^R_miS^A_ij=(S^RS^A)_mj= S_mj. This is (6). Since S~=SRSA S=S^RS^A is the only way (SA,SR)(S^A,S^R) enter, any two pairs with the same product induce the same m g_m. ∎ B.4 Proof of Corollary 1 (per-agent-ratio canonical form) Proof. The estimator (SR=I,SA=S~)(S^R=I,S^A= S) has product S~′=I⋅S~=S~ S =I· S= S, identical to that of (SA,SR)(S^A,S^R). By Theorem 1, the expected gradient depends on the supports only through this product, so the two estimators share the same m g_m; concretely, reading from (6), both equal ∑j:Cjm=1S~mj∼πold[rjsm] _j:C_jm=1 S_mjE_ a _ θ_old[r_js^m], which is exactly the gradient of a per-agent-ratio estimator with advantage Am′=∑jS~mjrjA _m= _j S_mjr_j (a nonnegative integer-weighted reward sum, hence admissible). For the joint ratio SR=⊤S^R=11 , the reweighting is S~=⊤SA S=11 S^A. Thus, cross-agent ratios contribute nothing to m g_m beyond the linear advantage reweighting r↦S~r Sr. ∎ B.5 Proof of Proposition 1 (off-policy variance domination) Proof. Both paths in (7) realize the same product S~=C S=C, so by Theorem 1 they share the mean m=∑j∈∂m∼πold[rjsm] g_m= _j∈∂ mE_ a _ θ_old[r_js^m]. Write the per-step weights whose product with sms^m is averaged: for the advantage path (P), WmP=∑j∈∂mrjW^P_m= _j∈∂ mr_j; for the ratio path (Q), WmQ=∑i∈∂m(∏j∈∂iϱj)riW^Q_m= _i∈∂ m ( _j∈∂ i _j )r_i. (i) On-policy. At =old θ= θ_old, every ϱj=1 _j=1, so WmQ=∑i∈∂mri=WmPW^Q_m= _i∈∂ mr_i=W^P_m pathwise; the two estimators are identical random variables and share mean and variance. (i) Off-policy. WmPW^P_m has no θ dependence and is bounded: |WmP|≤|∂m|rmax|W^P_m|≤|∂ m|\,r_ (Assumption 4), so Var(WmP)Var(W^P_m) is a fixed constant. For Q, take a single summand i∈∂mi∈∂ m and condition on sms^m; by Assumption 4, there is δ>0δ>0 with ∼πold[ri2∣am]≥δE_ a _ θ_old[r_i^2 a^m]≥δ on a set of positive probability, and by independence of the ratios from rir_i and Lemma 2, ∼πold[(∏j∈∂iϱj)2ri2]≥δ∼πold[(∏j∈∂iϱj)2]=δ∏j∈∂i(1+χj2). _ a _ θ_old [ ( _j∈∂ i _j )^2\,r_i^2 ]\;≥\;δ\,E_ a _ θ_old [ ( _j∈∂ i _j )^2 ]\;=\;δ _j∈∂ i(1+χ^2_j). (15) Hence the second moment of the Q-weight, and therefore Var(mQ)Var( g^Q_m), is bounded below by a positive multiple of ∏j∈∂i(1+χj2) _j∈∂ i(1+χ^2_j), which grows without bound as the policy step increases (each χj2→∞χ^2_j→∞) or as |∂m||∂ m| grows (more factors). Since Var(mP)Var( g^P_m) stays bounded, Var(mQ)/Var(mP)→∞Var( g^Q_m)/Var( g^P_m)→∞. ∎ B.6 Proof of Proposition 2 (advantage bias–variance tradeoff) Proof. Take SR=IS^R=I, so S~=SA S=S^A and by Theorem 1 m=∑j:Cjm=1SmjA∼πold[rjsm] g_m= _j:C_jm=1S^A_mj\,E_ a _ θ_old[r_js^m]. The unbiased (full-coupling) gradient is m⋆=∑j∈∂m∼πold[rjsm] g _m= _j∈∂ mE_ a _ θ_old[r_js^m], i.e., the case SmjA=1S^A_mj=1 for all j∈∂mj∈∂ m. Bias. Subtracting, m−m⋆=∑j∈∂m(SmjA−1)∼πold[rjsm]=−∑j∈∂m:SmjA=0∼πold[rjsm], g_m- g _m= _j∈∂ m(S^A_mj-1)\,E_ a _ θ_old[r_js^m]=-\!\! _j∈∂ m:\ S^A_mj=0\!\!E_ a _ θ_old[r_js^m], (16) so the bias is exactly the sum of the score–reward couplings of the coupled agents omitted by SAS^A. It is zero iff SA⊇∂mS^A ∂ m and strictly nonzero when a genuinely coupled agent with ∼πold[rjsm]≠0E_ a _ θ_old[r_js^m]≠ 0 is dropped. Adding agents outside ∂m∂ m changes nothing, since ∼πold[rjsm]=0E_ a _ θ_old[r_js^m]=0 there by (13). Variance. The aggregated advantage Am=∑jSmjArjA_m= _jS^A_mjr_j is additive. Including an extra agent j adds the term rjsmr_js^m to the estimator. This term has mean ∼πold[rjsm]E_ a _ θ_old[r_js^m] and contributes ΔVar=Var(rjsm)+2Cov(rjsm,∑k∈SmA∖jrksm), =Var (r_js^m )+2\,Cov (r_js^m,\ _k∈ S^A_m jr_ks^m ), (17) whose dominant, always-nonnegative part is ∼πold[rj2(sm)2]≥0E_ a _ θ_old[r_j^2(s^m)^2]≥ 0 (strictly positive when rjr_j is conditionally nondegenerate). Thus each added agent raises the variance, while only agents in ∂m∂ m reduce the bias (16). The MSE ‖m−m⋆‖2+Var(m)\| g_m- g _m\|^2+Var( g_m) is therefore minimized by including exactly the coupled agents, SA=j:Cjm=1=∂mS^A=\\,j:C_jm=1\,\=∂ m: smaller supports are biased, larger supports inflate variance. ∎ Appendix C Experimental Details and Additional Results C.1 Experimental Setup C.1.1 Choice of Environments Our study requires environments with two properties that standard cooperative benchmarks lack: a per-agent reward (so the advantage support is meaningful) and a known coupling graph (so the coupling neighborhood is defined). Popular suites such as StarCraft (Whiteson et al., 2019) and Google Research Football (Kurach et al., 2020) expose a single shared team reward and no explicit coupling structure; verifying the advantage-support axis there would require synthesizing per-agent signals via difference rewards, introducing approximation and forfeiting the ground-truth coupling graph. We therefore study synthetic cooperative games in which the coupling graph is prescribed exactly, allowing us to vary the coupling family, its range, and the agent count while keeping the ground truth fixed. Applying the same analysis to real systems that natively provide per-agent rewards and a physical coupling graph—traffic-signal control being a canonical example—is a natural direction we leave to future work. Our goal here is a clean verification of the mechanism, not benchmark leaderboard performance, so the environments are simple and fully specified. The families, shown as Fig. 4, are chosen to separate the two axes the theory distinguishes, and to stress each in turn. They differ first in whether an agent internalizes its own externality: in the directed dilemma, an agent’s action harms only its successors, not itself, so an independent advantage is badly biased and the advantage support is decisive for the learned policy; in dense pairwise and local congestion, the coupling term enters the actor’s own reward (fully or partly), so an independent advantage is already near-unbiased and the advantage support mainly trades variance. This spread lets us show that the advantage radius matters for the outcome exactly when the externality is not self-internalized (Fig. 2). The families differ second in graph structure: the ring families (dilemma, pairwise, congestion) provide a geometric coupling with a well-defined radius, while block community gives a non-geometric, community-structured graph—verifying that matching the advantage support to the coupling neighborhood is about the coupling graph, not about spatial distance. The ratio-support conclusions (redundancy and variance ordering), by contrast, depend only on the number of factors and hold across all four, which is why we use every family for the ratio studies but highlight the dilemma for the advantage-outcome effect. C.1.2 Coupling Families: Reward Definitions All environments used in this paper are cooperative games with a prescribed coupling graph C, so the ground-truth coupling neighborhood is known exactly. For completeness, we briefly restate the notation. There are n agents; each agent i observes a phase or context and selects a discrete action ai∈1,…,Ka^i∈\1,…,K\, where K is the number of actions per agent. We write the joint action as =(a1,…,an) a=(a^1,…,a^n). Each agent receives a per-agent reward ri()r_i( a), defined per family below, and the team return is their sum R=∑iriR= _ir_i. The coupling graph C∈0,1n×nC∈\0,1\^n× n records reward dependencies, with Cij=1C_ij=1 iff rir_i depends on aja^j; its neighborhood ∂i=j:Cij=1∂ i=\j:C_ij=1\ is the set of agents whose actions enter rir_i. For the ring families, ∂i∂ i consists of the agents within graph distance ρ⋆ρ on a ring, and we refer to ρ⋆ρ as the coupling radius. Zero-mean Gaussian observation noise εi∼(0,σ2) _i (0,σ^2) with standard deviation σ is added where noted. Figure 4: Schematic of the four coupling families (illustrated at a reduced scale; nodes represent agents, edges represent reward dependencies). (a) Dense pairwise: undirected edges couple ring neighbors symmetrically, so that each agent’s action affects its neighbors as much as itself. (b) Directed dilemma: each agent transmits a one-way nuisance (red) to its downstream successors, which the actor does not bear. (c) Local congestion: neighbors selecting the same resource share the reward. (d) Block community: all-to-all coupling within disjoint blocks, with no coupling across blocks. Dense pairwise (symmetric, internalized). This family uses a symmetric pairwise payoff over the ring neighborhood, ri()=∑j∈∂i,j≠iM[ai,aj]+εi,M=M⊤∈ℝK×K,εi∼(0,σ2),r_i( a)= _j∈∂ i,\,j≠ iM[a^i,a^j]+ _i, M=M ^K× K,\ _i (0,σ^2), (18) where M is a fixed symmetric K×K× K payoff matrix whose entry M[ai,aj]M[a^i,a^j] is the payoff agent i obtains from the action pair (ai,aj)(a^i,a^j) with neighbor j, and εi _i is the observation noise defined above. Because M is symmetric, the same edge M[ai,aj]M[a^i,a^j] contributes to both rir_i and rjr_j; hence an agent’s action affects its neighbors’ reward as strongly as its own—the externality is internalized. This game supplies the single-step results in Fig. 1 (n=20n=20, K=4K=4, ρ⋆=4ρ =4, σ=0.6σ=0.6); the training study uses n=24n=24. Directed dilemma (non-internalized externality). On a directed ring, choosing a high-value action gives the actor a private benefit but imposes a nuisance on its ρ⋆ρ successors: ri()=v(ai)−c∑j:i∈∂+jg(aj),r_i( a)=v(a^i)-c\!\! _j:\,i∈∂^+j\!g(a^j), (19) where v(ai)v(a^i) is the private value agent i obtains from its own action, g(aj)g(a^j) is the nuisance that agent j’s action imposes on its downstream neighbors, c>0c>0 is the weight of that nuisance, and ∂+j∂^+j denotes the ρ⋆ρ successors of j on the directed ring; the sum thus runs over those agents j whose successor set contains i, i.e., the predecessors that emit onto i. A rotating phase makes the tempting action time-varying. Since the actor does not bear the harm it causes, an independent advantage converges to the tragedy-of-the-commons equilibrium, whereas a neighborhood advantage recovers the social optimum. We use n=20n=20, ρ⋆=2ρ =2, c=3c=3. Local congestion (self-internalized). Agents within ∂i∂ i that choose the same resource split its value: ri()=v(ai)|j∈∂i:aj=ai|+εi,r_i( a)= v(a^i) |\j∈∂ i:a^j=a^i\ |+ _i, (20) where v(ai)v(a^i) is the base value of the resource selected by action aia^i, and the denominator counts the number of agents in the neighborhood ∂i∂ i (including i itself) that chose the same resource, so the resource value is shared equally among them. An agent that over-subscribes a popular resource thus depresses its own reward as well as its neighbors’—the crowding cost is partly self-borne. We use n=20n=20, ρ⋆=2ρ =2. Block community (non-geometric coupling). Agents are partitioned into disjoint blocks; coupling is all-to-all within a block and absent across blocks, so C is block-diagonal, with an intra-block symmetric payoff matrix M as in the dense-pairwise case (18) (here ∂i∂ i is i’s entire block rather than a ring neighborhood). This setup tests a coupling graph that is not a geometric ring. We use n=20n=20 and block size 55. In every case, the reward parameters (M, v, g, block assignment) are fixed per environment; we report mean ± standard deviation over random seeds. Full hyperparameters are deferred to the appendix. C.2 Extended Training-Curve Analysis This subsection reports the full per-family training dynamics summarized in the main text: the advantage-support curves (complementing the sweep in Fig. 2) and the ratio-support redundancy curves. Together, they demonstrate, family by family, that enlarging the advantage support can change the learned policy, whereas enlarging the ratio support never does. Figure 5 plots the team return over PPO iterations for each family under three advantage supports—independent (ρA=0 _A=0), coupling-radius, and joint—with the ratio held per-agent. The directed dilemma shows a dramatic separation: independent advantage descends to the tragedy equilibrium, while coupling-radius and joint advantage climb to the social optimum. In the other three families, where the externality is self-internalized, independent advantage already learns effectively, and larger supports add variance rather than altering the outcome. This illustrates the advantage side of the asymmetry: the advantage support can shift the learned policy. To demonstrate that the ratio radius is redundant, we fix the team advantage and sweep the ratio support ρR∈per-agent,neighborhood,joint _R∈\per-agent,neighborhood,joint\ (Fig. 6). Within every family, the three learning curves are indistinguishable—overlapping to within seed noise, with clip fraction 0 throughout—so enlarging the ratio support changes neither the trajectory nor the final outcome, exactly as predicted by Corollary 1. This is the ratio-side counterpart to Fig. 5: enlarging the advantage support can change the outcome, whereas enlarging the ratio support never does. Figure 5: Advantage-support training curves across the four coupling families. Team return versus PPO iteration (55 seeds, ±1± 1 std bands) under three advantage supports: independent (ρA=0 _A=0), coupling-radius, and joint, with the ratio held per-agent. One panel per family. In the directed-dilemma and dense-pairwise panels, the seed variance is below 1.5%1.5\% of the return scale, so the bands are narrower than the line width. Figure 6: Ratio-support redundancy under standard PPO across the four coupling families. Team return versus PPO iteration (±1± 1 std bands over seeds) with team advantage fixed. Each panel shows three ratio supports: per-agent (ρR=0 _R=0), coupling-radius, and joint. C.3 Real-World Traffic-Signal Control Finally, we test the advantage-support prediction (Proposition 2) on a real cooperative traffic-signal control problem that natively provides both ingredients the theory requires: a per-agent reward (each intersection’s local queue, delay, and throughput) and a physical coupling graph (the road network). We use the SUMO (Behrisch et al., 2011) Manhattan 28×728× 7 grid (n=196n=196 traffic signals) via sumo-rl (Alegre, 2019), with the standard MLP-actor MAPPO from the toy suite; the only tunable parameter is the advantage support, built by k-hop expansion on the road-adjacency graph, with a per-agent ratio throughout. Each intersection i is an agent with a local observation oio_i (the standard sumo-rl encoding): a one-hot of the active green phase, a binary flag indicating whether the minimum green time has elapsed, and, for every incoming lane, the normalized vehicle density and queue length. The action aia_i is discrete—the choice of the next green phase from that intersection’s signal program—applied on a fixed 1010 s control cycle with a 22 s yellow transition and a 55 s/5050 s minimum/maximum green. The shared MLP actor maps oio_i to a categorical distribution over aia_i; agents share parameters but act independently, so the joint policy factorizes as ∏iπi(ai) _iπ^i(a_i), exactly as our analysis assumes. Each agent’s reward is a purely local combination of its own traffic state: ri=wvv¯i−wqqi−wωωi−wppi,(wv,wq,wω,wp)=(3.0, 1.0, 0.3, 1.5),r_i\;=\;w_v\, v_i\;-\;w_q\,q_i\;-\;w_ω\, _i\;-\;w_p\,p_i, (w_v,\,w_q,\,w_ω,\,w_p)=(3.0,\,1.0,\,0.3,\,1.5), (21) where qiq_i is the total queue length, ωi _i the total waiting time, pip_i the inflow–outflow pressure |in−out||in-out|, and v¯i v_i the mean speed, each normalized per lane (by 5050, 10001000, 2020 vehicles, and 1515 m/s, respectively). Here rir_i depends on other intersections only through physical spillover onto adjacent roads—so the reward coupling graph is exactly the road-network adjacency (each signal coupled to its up-to-four grid neighbors), and the range ρ⋆ρ is what our advantage support aims to match. C.3.1 Advantage-Support Study: Global Advantage Collapses; Local Advantages Work (Fig. 7) Figure 7 plots the three traffic metrics—team reward (the trained objective), average queue length, and average waiting time—over PPO iterations, comparing the global (team) advantage against three local advantage supports; Table 1 reports the converged metrics. A team advantage—the radius-∞ support that aggregates all 196196 intersections’ rewards, as in vanilla MAPPO—fails to learn: its team return remains flat near its initial value while local advantages climb steadily, and at convergence it is 36%36\% worse in return, with 2.4×2.4× the average queue and 3.7×3.7× the average waiting time (Table 1). This is Proposition 2(i) at scale: summing 196196 per-intersection rewards drowns each signal’s own learning contribution in the noise of 195195 others. In contrast, the three local supports—independent (ρA=0 _A=0) and one/two-hop neighborhoods—all learn well and closely track one another, with a mild monotone gain from including immediate neighbors; the insets confirm that these three curves differ only marginally and within the seed bands. The small spread among local radii indicates that traffic coupling is largely self-internalized (an intersection’s own queue strongly reflects its own action), placing this system in the regime where the advantage-support bias is small and the dominant effect is the variance cost of an over-large support—exactly the failure mode exhibited by the team advantage. The team–local gap far exceeds the seed spread. Figure 7: Traffic-signal control on a 196196-intersection network. Advantage-support training curves (per-agent ratio; mean over 33 seeds, ±1± 1 std shaded bands) on all three metrics—team reward, average queue, and average waiting time—versus PPO iteration. Each panel compares three local advantage supports (independent (ρA=0 _A=0) and one/two-hop neighborhoods) against the global/team advantage (ρA=∞ _A=∞). Insets zoom the last 6060 iterations for the three local supports alone (ρA=0,1,2 _A=0,1,2). Table 1: Traffic-signal control, 196196 intersections, final metrics (mean ± std over 33 seeds, last 20%20\% of training). Local advantages show strong performance; the global (team) advantage collapses. advantage support team return ↑ avg queue ↓ avg wait (s) ↓ clip frac independent (ρA=0 _A=0) 1652.8±4.51652.8±4.5 6.77±0.046.77±0.04 298.0±47.1298.0±47.1 0.0140.014 11-hop neighborhood 1663.7±7.51663.7±7.5 6.68±0.216.68±0.21 275.9±57.7275.9±57.7 0.0160.016 22-hop neighborhood 1663.8±6.01663.8±6.0 6.50±0.066.50±0.06 258.6±45.4258.6±45.4 0.0160.016 global / team (ρA=∞ _A=∞) 1068.1±7.51068.1±7.5 15.62±0.2315.62±0.23 968.9±48.7968.9±48.7 0.0000.000 C.3.2 Ratio-Support Study: Redundancy and Variance Ordering The advantage study above fixes a per-agent ratio and varies the advantage support. We now perform the converse: vary the ratio support—per-agent (SR=IS^R=I), 11-hop and 22-hop neighborhoods, and joint (SR=⊤S^R=11 , the 196196-way product)—and, to verify that the ratio-side conclusion does not depend on how the advantage is chosen, we repeat the entire sweep under two fixed advantage baselines: the 11-hop neighborhood advantage and the independent (SA=IS^A=I) advantage. All runs use a standard PPO configuration. This isolates the ratio support on a real system and tests the two ratio-side predictions—redundancy (Corollary 1) and variance ordering that worsens with n (Proposition 1, Fig. 3). Figure 8 plots, for each advantage baseline (rows) and each metric (columns), the training curve of every ratio support. The three small ratio supports—per-agent, 11-hop, and 22-hop—learn equally well and reach essentially the same reward, queue, and waiting time: enlarging the ratio support from 0 to 22 hops changes nothing about the learned policy, exactly as the redundancy corollary (Corollary 1) predicts. The joint 196196-way ratio, in contrast, collapses outright and it does so whether the advantage is the 11-hop neighborhood or fully independent. This is the sharpest real-system signature of the n-dependence in Fig. 3: with n=196n=196 agents, the joint ratio is a 196196-way product whose variance factor ∏j(1+χj2) _j(1+χ^2_j) is enormous, drowning the surrogate gradient in variance well before any benefit could accrue. Crucially, no off-policy stress is needed to expose this—unlike the toy games, where n≤24n≤ 24 keeps the standard-configuration penalty dormant, the traffic network is large enough that the joint ratio fails under an ordinary, conservative PPO configuration. That the collapse and the redundancy are identical across the two advantage baselines confirms that the ratio-side conclusion is a property of the ratio support alone, independent of the advantage support with which it is paired. Figure 8: Ratio-support study on the 196196-intersection network under a standard PPO configuration (mean over 33 seeds, ±1± 1 std bands). Top row: the 11-hop neighborhood advantage held fixed. Bottom row: the independent (SA=IS^A=I) advantage held fixed. Columns show team reward, average queue, and average waiting time versus PPO iteration. Within each panel, four curves vary the ratio support: per-agent (ρR=0 _R=0), 11-hop, 22-hop, and joint (196196-way). Table 2: Ratio-support study on the traffic network, final metrics (mean ± std over 33 seeds, last 20%20\%), under two fixed advantage baselines. Ratio supports of 0, 11, and 22 hops learn; the joint 196196-way ratio collapses under either baseline. advantage ratio support team return ↑ avg queue ↓ avg wait (s) ↓ clip frac 11-hop nbr per-agent (ρR=0 _R=0) 1659.5±4.61659.5±4.6 6.69±0.046.69±0.04 286.0±39.0286.0±39.0 0.0140.014 11-hop 1657.1±2.71657.1±2.7 6.71±0.066.71±0.06 280.3±36.6280.3±36.6 0.0160.016 22-hop 1659.4±6.21659.4±6.2 6.72±0.156.72±0.15 295.8±29.2295.8±29.2 0.0160.016 joint (196196-way) 1071.1±6.71071.1±6.7 15.63±0.1815.63±0.18 950.6±37.1950.6±37.1 0.0410.041 independent per-agent (ρR=0 _R=0) 1656.5±6.51656.5±6.5 6.77±0.096.77±0.09 301.4±42.9301.4±42.9 0.0150.015 11-hop 1658.6±1.51658.6±1.5 6.68±0.136.68±0.13 286.4±60.3286.4±60.3 0.0160.016 22-hop 1649.7±6.11649.7±6.1 6.79±0.126.79±0.12 291.0±8.1291.0±8.1 0.0150.015 joint (196196-way) 1070.6±7.01070.6±7.0 15.63±0.2115.63±0.21 945.8±46.1945.8±46.1 0.0410.041 Appendix D Related Work Trust-region and proximal MARL. Extending PPO and TRPO to cooperative MARL requires deciding how the importance ratio is formed across agents. MAPPO (Yu et al., 2022) pairs a shared team advantage with a per-agent ratio and serves as a strong benchmark baseline; IPPO trains each agent independently, also with a per-agent ratio. At the other extreme, HATRPO and HAPPO (Kuba et al., 2022) update agents sequentially and clip a compound ratio—a product of the ratios of previously updated agents—justified by the multi-agent advantage-decomposition lemma. These methods span the ratio-support axis we study, from the per-agent extreme (SR=IS^R=I) to the fully joint extreme (SR=⊤S^R=11 ), yet the choice is typically made by architectural convention rather than analysis. Our results speak directly to this decision: the compound or joint ratio is gradient-redundant relative to a per-agent ratio with a suitably reweighted advantage (Theorem 1), and incurs strictly higher variance (Proposition 1). Locality and scalability in networked MARL. A parallel line of work exploits the observation that, in networked systems, an agent’s influence decays with graph distance. Qu et al. (2020b; a) formalize this as an exponential decay property and use it to prove that truncating each agent’s value estimate to a κ-hop neighborhood incurs only O(ρκ)O(ρ^κ) error, yielding provably scalable actor–critic algorithms. This provides the theoretical grounding for our advantage-support axis: if reward coupling has finite range ρ⋆ρ , then aggregating rewards over the ρ⋆ρ -neighborhood is nearly unbiased, while a smaller support omits real externalities and a larger one only adds variance. We complement this literature by (i) separating the advantage support from the ratio support and (i) showing that the two are redundant on the expected gradient—questions orthogonal to value-function truncation. Counterfactual and difference credit. COMA (Foerster et al., 2018) and difference-reward methods (Wolpert and Tumer, 2001; Li et al., 2022) reduce variance by subtracting a counterfactual baseline whose expected gradient contribution is zero; recent per-agent advantage estimators (Kim et al., 2026) prove policy-gradient invariance of such baselines under the factorized policy. These works concern the advantage or baseline and its zero-offset property; none addresses the ratio support or the factorization of the expected gradient through S~=SRSA S=S^RS^A, which is the object of our study. Local advantages and local ratios. Several applied methods already restrict the advantage to a neighborhood without a general analysis. In traffic-signal control, MA2C (Chu et al., 2019) introduces a spatial discount factor that down-weights distant agents’ rewards inside each local return—an advantage support tapered by graph distance. We observe an asymmetry in this literature: a rich set of methods aggregate the advantage locally, but we are not aware of any cooperative method that aggregates the ratio locally (e.g., a local importance-sampling weight ∏j∈∂iϱj _j∈∂ i ^j). Where a joint ratio does appear, it is typically not chosen for its own sake but inherited from casting the whole team as a single multi-action agent: running PPO on the factorized joint policy ∏iπi _iπ^i treats a as one action and thus clips the product of all agents’ ratios. This reduction is attractive because it turns MARL into single-agent RL—with its simpler objective and convergence story—and is used, for example, to make combinatorial cooperative tasks tractable as a single-controller problem. Our results explain the resulting high variance: the two supports are interchangeable on the expected gradient (Theorem 1), so the joint ratio buys nothing over a per-agent ratio, yet it is strictly the higher-variance route (Proposition 1)—a product rather than a sum. This variance is therefore not a flaw of any particular single-agent-reduction algorithm but an inherent cost of aggregating on the ratio side, and it can be removed by moving the aggregation into the advantage. The field’s revealed preference for local advantages over local ratios is exactly what this variance-ordering predicts; we subsume the local-advantage heuristics as particular advantage-support choices and provide the design principle behind them. Appendix E Discussion Existing methods on the bias–variance plane. The two supports provide a common coordinate system for methods that otherwise appear unrelated. IPPO sits at the low-variance, high-bias corner (SA=SR=IS^A=S^R=I): both supports per-agent, so its estimator has minimal variance but ignores every externality. MAPPO moves the advantage to the team (SA=⊤,SR=IS^A=11 ,S^R=I): it removes the missing-coupling bias but, on a large team, pays the additive variance of summing all rewards—and, as our 196196-intersection experiment shows, this already fails when the team is large and only a local neighborhood is truly coupled. Neighborhood-advantage methods (spatial-discount and local-critic actor–critics) occupy the principled middle: SAS^A is a k-hop neighborhood, SR=IS^R=I, which our analysis identifies as the MSE-optimal choice when the coupling has range k. At the far corner lie single-agent reductions that treat ∏iπi _iπ^i as one multi-action policy—for example, CMAT (Zhao et al., 2026), which factorizes the joint policy through a latent consensus and then optimizes it with single-agent PPO on the joint action. Such methods effectively push both supports to fully joint (SA=SR=⊤S^A=S^R=11 ), so S~=n 11⊤ S=n\,11 —the maximal effective support. In our terms, this is the least biased and most variant point in the plane: it aggregates every neighbor’s reward and multiplies every neighbor’s ratio. Our results indicate that the first is unnecessary beyond the coupling neighborhood and the second is strictly harmful; the elegant equivalence to single-agent RL is real at the level of the expected objective, but it is bought with a variance that, in a large system, need not pay off. This is not a defect of any particular such algorithm but an inherent property of the reduction—the maximal-support corner is high-variance by construction—and our canonical form shows the variance is removable without changing what is learned, by moving all aggregation onto the advantage and sizing it to the coupling graph. This reframes per-agent versus centralized not as a binary but as a point in a two-dimensional support plane whose optimal location the theory pins down. Does the result apply to multi-action (centralized) PPO? A natural concern is that our analysis concerns multi-agent PPO, whereas a centralized controller may treat the whole team as a single agent whose action is the joint =(a1,…,an) a=(a^1,…,a^n) and run vanilla single-agent PPO on it—the multi-action view underlying CMAT (Zhao et al., 2026). Our results apply directly to this setting, because the only structural assumption we use is that the policy factorizes across action components, π()=∏iπi(ai)π( a)= _iπ^i(a^i) (Assumption 1)—i.e., independent categorical heads, which is exactly how such multi-action policies are built. Under that factorization, multi-action PPO is the corner (SA,SR)=(⊤,⊤)(S^A,S^R)=(11 ,11 ): a single scalar advantage (the team return) is broadcast to every action component, and the importance weight is the ratio of the joint action, ∏iϱi _i ^i. The canonical form (Theorem 1) therefore governs it unchanged—the joint ratio is gradient-redundant relative to a per-agent ratio with a reweighted advantage, and strictly higher-variance (Prop. 1)—so the design rule transfers verbatim: clip each action component’s ratio separately rather than as one n-way product, which turns multi-action PPO back into a per-agent-ratio scheme (a MAPPO-style update) at no cost to the expected gradient. Two boundaries are worth stating. First, the factorization is essential: an autoregressive joint head (component aja^j conditioned on a<ja^<j) uses a different chain rule and is out of scope. Second, when the environment provides only a single non-decomposable team reward, the advantage cannot be localized (there is no per-agent reward to aggregate over a neighborhood), so the bias-reduction half of our rule is inapplicable; however, the variance half still holds—the joint ratio should be replaced by a per-agent ratio regardless—which is precisely the sense in which a MAPPO-style per-agent ratio dominates a fully centralized multi-action update even in the single-team-reward regime. Two caveats the experiments make precise. The design rule is exact at the level of the expected gradient, but two qualifications govern how visibly it matters in training. First, the advantage support reshapes the learned policy only when the coupling externality is not already internalized by an agent’s own reward: in the directed dilemma, independent advantage collapses to the tragedy equilibrium and a neighborhood advantage recovers the social optimum, whereas in symmetric-payoff, congestion, and block games, independent advantage is already near-unbiased and larger supports only add variance (Fig. 2). Second, the ratio’s variance penalty is gated by the per-update policy shift χ2χ^2: a conservative trust region keeps χ2χ^2 so small that per-agent and joint ratios are indistinguishable even at hundreds of agents, and the penalty surfaces only as updates are driven off-policy (Fig. 3). Neither qualification weakens the design rule—the ratio’s benefit is exactly zero and its cost is non-negative throughout—but both explain why the folklore that per-agent ratios are more stable is only intermittently observed in practice. Implications for large-scale deployment. The design principle is most consequential precisely where cooperative MARL is scaling: systems with hundreds to thousands of agents and local physical coupling—traffic-signal grids, sensor and robot fleets, power and communication networks, warehouse logistics. Two concrete recommendations follow. (1) Never aggregate the ratio. At these scales, the joint ratio’s variance factor (1+χ2)n(1+χ^2)^n is enormous the moment any batch drifts off-policy, and our 196196-intersection result shows the analogous failure for an over-large advantage (a global team advantage never learns, while local advantages cut queues by 2.4×2.4× and waiting time by 3.7×3.7×). Keep the ratio per-agent; put every bit of cross-agent aggregation in the advantage, where the cost is additive. (2) Size the advantage to the physical coupling. Because these systems come with a known interaction graph (the road network, the communication topology, the spatial adjacency), the MSE-optimal advantage support is directly available—the k-hop neighborhood for a coupling of range k—rather than something to be tuned blindly. This turns a hyperparameter search into a modeling choice and connects to the scalable-MARL line (Qu et al., 2020b; a), whose exponential-decay property is exactly the condition under which a small advantage neighborhood is near-unbiased. Neither recommendation requires changing the network architecture or the training loop—only which rewards and which ratios enter each agent’s update—so they are immediately actionable for existing MAPPO and IPPO codebases. Future work: learning the support. Our design rule assumes the coupling neighborhood is known—true for physically networked systems, but not in general. When the coupling graph is unknown or state-dependent, the open problem is to learn the right advantage support: which neighbors to aggregate, and at what radius, so as to minimize gradient MSE. This is a bias–variance model-selection problem—estimate each agent’s influence graph (e.g., from reward sensitivities or attention over other agents’ states) and include an agent in the support when its estimated coupling outweighs the variance its reward adds. A dynamic, per-state support that grows in strongly-coupled regimes and shrinks otherwise would sit at the MSE optimum adaptively, generalizing the fixed k-hop rule; the canonical form guarantees that whatever support is learned, it should be realized on the advantage and never on the ratio.