Paper deep dive
Turnover-Orthogonal Credit Assignment for Open-Team Multi-Agent Reinforcement Learning
Amit Thakur, Mukesh Singhal
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 10/5/2026, 6:59:35 AM
Summary
The paper introduces Turnover-Orthogonal Credit Assignment (TOCA), a value decomposition method for open-team multi-agent reinforcement learning (MARL). TOCA separates team value into action effects, pure turnover effects, and action-turnover interactions to prevent surviving agents from being penalized or rewarded for exogenous team changes. It utilizes a permutation-invariant centralized critic and an event-conditioned baseline to derive counterfactual per-agent credit signals. Experiments on the Dynamic Spread benchmark demonstrate that TOCA, particularly the TOCA-β variant which softly weights interaction terms, improves robustness and return under high turnover rates compared to standard MAPPO-style critics.
Entities (5)
Relation Signals (5)
TOCA → achievesbestperformancein → Dynamic Spread
confidence 95% · In a replacement-only Dynamic Spread benchmark, TOCA-β achieves the best mean return at high turnover rates
TOCA → solves → Open-Team MARL
confidence 95% · We introduce turnover-orthogonal credit assignment (TOCA), a value decomposition for open teams...
TOCA → decomposes → Team Value
confidence 90% · TOCA decomposes the event-conditioned value into Q(h,a,e)=μ(h)+qA(h,a)+qE(h,e)+qAE(h,a,e)
TOCA → outperforms → MAPPO
confidence 90% · Controlled diagnostic experiments show that TOCA improves return over event-aware MAPPO-style critics
TOCA → uses → Permutation-Invariant Critic
confidence 90% · We instantiate this idea with a permutation-invariant centralized critic over variable-size agent sets and event tokens
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Open-team multi-agent reinforcement learning studies cooperative systems in which agents may join, leave, or be replaced during an episode. In such settings, the team return changes both because agents choose useful actions and because the active population itself changes. Standard centralized critics and shared advantages often mix these two effects into one scalar credit signal, allowing surviving agents to be rewarded or penalized for exogenous turnover events outside their control. We introduce turnover-orthogonal credit assignment (TOCA), a value decomposition for open teams that separates action effects, pure turnover effects, and action--turnover interactions. Under exogenous turnover, the event-conditioned value admits a centered decomposition whose event-conditioned baseline removes the pure turnover component while preserving credit for actions that make the team robust to future replacements. We instantiate this idea with a permutation-invariant centralized critic over variable-size agent sets and event tokens, and derive both a counterfactual per-agent credit signal and a softly weighted interaction variant, TOCA-$\beta$, for high-variance control environments. Controlled diagnostic experiments show that TOCA improves return over event-aware MAPPO-style critics and that removing interaction credit substantially hurts performance. In a replacement-only Dynamic Spread benchmark, TOCA-$\beta$ achieves the best mean return at high turnover rates and improves over its no-interaction ablation. These results suggest that explicitly separating turnover from action credit is a useful principle for robust learning in dynamic cooperative teams.
Tags
Links
- Source: https://arxiv.org/abs/2610.02847v1
- Canonical: https://arxiv.org/abs/2610.02847v1
Trouble viewing inline? Open PDF directly →
Full Text
40,919 characters extracted from source content.
Expand or collapse full text
Turnover-Orthogonal Credit Assignment for Open-Team Multi-Agent Reinforcement Learning Amit Thakur Mukesh Singhal Affiliation: University of California, Merced Abstract Open-team multi-agent reinforcement learning studies cooperative systems in which agents may join, leave, or be replaced during an episode. In such settings, the team return changes both because agents choose useful actions and because the active population itself changes. Standard centralized critics and shared advantages often mix these two effects into one scalar credit signal, allowing surviving agents to be rewarded or penalized for exogenous turnover events outside their control. We introduce turnover-orthogonal credit assignment (TOCA), a value decomposition for open teams that separates action effects, pure turnover effects, and action–turnover interactions. Under exogenous turnover, the event-conditioned value admits a centered decomposition whose event-conditioned baseline removes the pure turnover component while preserving credit for actions that make the team robust to future replacements. We instantiate this idea with a permutation-invariant centralized critic over variable-size agent sets and event tokens, and derive both a counterfactual per-agent credit signal and a softly weighted interaction variant, TOCA-β, for high-variance control environments. Controlled diagnostic experiments show that TOCA improves return over event-aware MAPPO-style critics and that removing interaction credit substantially hurts performance. In a replacement-only Dynamic Spread benchmark, TOCA-β achieves the best mean return at high turnover rates and improves over its no-interaction ablation. These results suggest that explicitly separating turnover from action credit is a useful principle for robust learning in dynamic cooperative teams. 1 Introduction Cooperative multi-agent reinforcement learning (MARL) has made substantial progress in fixed-population settings, where the same agents interact throughout training and evaluation. Many real systems are not fixed: Robot teams lose units and receive replacements; ride-sharing and delivery fleets change membership throughout a shift; distributed sensing systems operate with intermittent devices; and human–AI teams may change participants mid-task. In these settings, agents must act while the active population is itself a dynamic part of the environment. This paper studies credit assignment in such open-team MARL. The core difficulty is that the learning signal conflates two different causes of value change: what the agents did, and what happened to the team. After a joining, leaving, or replacement event, a standard temporal-difference error can penalize a survivor because a strong teammate left, or reward it because a capable teammate arrived, even when that survivor repeated exactly the same action. Conversely, if an agent anticipates turnover and moves into a configuration that is robust to teammate loss, that benefit should remain part of its credit. A useful signal must therefore remove the pure effect of turnover while preserving action-dependent robustness to turnover. This distinction is important both scientifically and practically: Scientifically, confounded credit makes it hard to tell whether a learned policy is genuinely robust or is merely benefiting from favorable team changes. Practically, misattribution can slow learning, destabilize coordination, and produce brittle behavior in domains where team membership changes are routine. In robotics, logistics, sensing, and human–AI collaboration, the desired behavior is not just high average return, but resilience under teammate loss, failure, or replacement. That makes source-of-credit separation a central problem rather than a cosmetic refinement. Existing centralized training methods partially address coordination by conditioning critics on joint observations and actions Lowe et al. (2017); Foerster et al. (2018); Yu et al. (2022). Value factorization methods decompose team value across agents Sunehag et al. (2018); Rashid et al. (2018); Son et al. (2019). However, these methods are usually formulated for a fixed set of agents. Event-aware critics can condition on turnover information, but conditioning alone does not determine how much of the resulting value should be attributed to actions versus the event itself. We therefore argue that open-team credit assignment requires a decomposition with explicit orthogonality to pure turnover effects. We propose turnover-orthogonal credit assignment (TOCA). Let h denote the centralized history, a the joint action of active agents, and e the turnover event. TOCA decomposes the event-conditioned value into Qπ(h,a,e)=μ(h)+qA(h,a)+qE(h,e)+qAE(h,a,e),Q^π(h,a,e)=μ(h)+q_A(h,a)+q_E(h,e)+q_AE(h,a,e), (1) where μ(h)μ(h) is the average state value, qAq_A is the action main effect, qEq_E is the pure turnover effect, and qAEq_AE is the action–turnover interaction. The baseline μ(h)+qE(h,e)μ(h)+q_E(h,e) is event-conditioned but action-independent. Subtracting it gives a turnover-orthogonal advantage that removes pure event credit while retaining qA+qAEq_A+q_AE. In high-variance control tasks, we also use TOCA-β, which softly weights the interaction term. Our contributions are as follows: First, we formulate open-team credit assignment as a decomposition problem over actions and turnover events. Second, we derive an event-conditioned baseline that is orthogonal to pure turnover effects under exogenous turnover. Third, we instantiate the decomposition with a permutation-invariant critic suitable for variable-size teams. Fourth, we evaluate the method in a controlled diagnostic benchmark and a replacement-only Dynamic Spread benchmark, showing that interaction credit is important and that TOCA-β improves high-turnover robustness. 2 Related Work Credit assignment in cooperative MARL: Difference rewards and counterfactual baselines aim to isolate an agent’s contribution to a team return. COMA marginalizes one agent’s action with a centralized critic Foerster et al. (2018); MADDPG uses centralized critics for decentralized policies Lowe et al. (2017); and MAPPO, HATRPO/HAPPO, and recent practice studies show the strength of PPO-style and trust-region training in cooperative MARL Schulman et al. (2017); Yu et al. (2022); Kuba et al. (2021); Fu et al. (2022). These methods improve action credit but do not explicitly separate action credit from population-turnover credit. Value factorization and sequence models: VDN, QMIX, QTRAN, and related methods decompose joint value functions into agent-level terms with structural constraints Sunehag et al. (2018); Rashid et al. (2018); Son et al. (2019). Recent transformer-based MARL models, including MAT and TransMix, provide stronger sequence or mixing architectures for cooperative decision making Wen et al. (2022); Khan et al. (2022). TOCA is complementary: it decomposes value by sources of variation–actions, turnover events, and their interaction–rather than only by agents or sequence position. Open, variable-size, and permutation-invariant teams: Open-team problems require representations that do not depend on a fixed ordering or fixed number of agents. Deep Sets and Set Transformers provide permutation-invariant or equivariant neural architectures for set-valued inputs Zaheer et al. (2017); Lee et al. (2019). Recent work on dynamic team composition and open-environment cooperative MARL highlights settings where agents with different capabilities may join or leave during operation Liu et al. (2021); Yuan et al. (2023). We build on this motivation by encoding the active agent set and turnover-event tokens with shared encoders. PettingZoo and multi-agent particle environments provide common tools for evaluation Mordatch and Abbeel (2018); Terry et al. (2021); our Dynamic Spread benchmark extends this style of task with fixed-size replacement events. 2.1 Why Event Conditioning Alone Is Not Enough A natural baseline for open teams is to append event information to a centralized critic. This can improve value prediction because the critic knows whether an unfavorable replacement occurred. However, better prediction alone does not guarantee better credit assignment. If the actor still updates from a target that mixes action-caused and event-caused variation, then the policy can be pushed by return changes that were independent of the sampled action. The issue is especially visible after disruptive replacement events: the critic predicts a lower return, but without an event-conditioned baseline the replacement shock can still be distributed across all survivors. This is the sense in which TOCA is “turnover-orthogonal.” The actor signal should be orthogonal to value variation caused only by exogenous population change, while still preserving action–turnover interactions. A method that removed all event dependence would also erase robustness credit; TOCA removes only the action-independent part. This is also why TOCA is different from standard value factorization: factorization composes team value from agents, whereas TOCA decomposes team value into sources of variation relevant for credit assignment. 3 Open-Team Problem Setting We consider a cooperative partially observable Markov game Littman (1994); Oliehoek and Amato (2016) with a dynamic active set tN_t. At time t, active agents observe local observations otio_t^i, choose actions atia_t^i, and receive a shared reward rtr_t. The active population evolves through turnover events. Let t+1=(t∖ℒt)∪t,N_t+1=(N_t _t) _t, (2) where ℒtL_t is the set of leaving agents and tJ_t is the set of joining agents. We denote the event by Et=et=(t,ℒt,κt),E_t=e_t=(J_t,L_t, _t), (3) where κt _t stores event attributes such as replacement type, capability, or inherited position. In the neural critic, this structured event is encoded as an event token: a compact vector containing the event type, joining/leaving masks, and attributes such as replacement capability or inherited-position flags. Let Ht=hH_t=h be the centralized history available during training and let At=aA_t=a be the joint action of the currently active agents. The event-conditioned value is Qπ(h,a,e)=π[Gt∣Ht=h,At=a,Et=e]Q^π(h,a,e)=E_π[G_t H_t=h,A_t=a,E_t=e] (4) where Gt=∑k≥0γkrt+kG_t= _k≥ 0γ^kr_t+k is the discounted return. We assume decentralized execution: each agent acts from its local observation and identity-free features, while the critic may use centralized information during training. 3.1 Illustrative Open-Team Credit Example Consider a navigation team with two landmarks and two active agents. If a fast agent is replaced by a slow agent, the team return may immediately drop even if the remaining agent repeats the same action. This drop is a pure turnover effect. Now suppose the remaining agent moves toward the landmark previously covered by the fast agent. The usefulness of this action depends on the replacement event: it may be unnecessary if no replacement happens, but valuable when the fast teammate disappears. This is an action–turnover interaction. A standard shared advantage observes only the final return. A purely event-conditioned critic predicts the drop more accurately but may still provide an unclear actor signal. TOCA assigns the unavoidable replacement penalty to qEq_E and the useful compensatory movement to qAEq_AE. The same logic applies beyond navigation: a sensing robot can reposition after a sensor failure, or a human–AI assistant can summarize context so that an incoming participant can contribute quickly. In each case, the membership change itself is not the survivor’s action, but preparation for or response to that change is. The central assumption for turnover-orthogonal credit is exogenous turnover: Et⟂At|HtE_t A_t H_t (5) This assumption does not mean that turnover is unimportant. It means that, conditioned on the current history, the event is not caused by the sampled joint action. This covers randomized replacement, device failure, externally scheduled team changes, and stress-test evaluations. If turnover is action-caused, then event effects may legitimately be part of action credit; as discussed in the Additional Analysis and Limitations section. The assumption also clarifies the role of the state. We do not require events to be independent of the history. Replacement may be more likely in difficult states or after a fixed schedule. What we require is that, after conditioning on the history used by the critic, the sampled action is not the cause of the event. Under this condition, subtracting an event-conditioned but action-independent baseline does not bias the policy gradient, while it can reduce misattribution and variance. 4 Turnover-Orthogonal Decomposition 4.1 Credit-Faithfulness Criterion We say that an advantage estimator is turnover-faithful if adding a pure event offset to the return does not change the expected actor update. Formally, let c(h,e)c(h,e) be any scalar that depends on the history and event but not on the sampled joint action. For an advantage estimator A A, turnover faithfulness requires that replacing the return target by Gt+c(h,e)G_t+c(h,e) changes only the baseline term and not the action-dependent policy-gradient direction. An ordinary shared return target does not satisfy this property unless its baseline also tracks c(h,e)c(h,e). The event-conditioned baseline in Equation 12 is the minimal baseline that removes such offsets while retaining the interaction residual. This criterion is weaker than demanding perfect causal identification in the full environment. It is intended for the common experimental and operational setting where turnover events are externally injected or logged. In that setting, the method does not decide whether a replacement was good or bad; it decides whether the resulting change should be assigned to the agents’ actions. 4.2 Centered Value Decomposition For a fixed history h, consider the random variables A∼π(⋅∣h)A π(· h) and E∼p(⋅∣h)E p(· h). Under exogenous turnover, the joint distribution factorizes as p(a,e∣h)=π(a∣h)p(e∣h)p(a,e h)=π(a h)p(e h). We decompose Qπ(h,a,e)Q^π(h,a,e) as in Equation 1 with centering constraints A[qA(h,A)] _A[q_A(h,A)] =0, =0, E[qE(h,E)] _E[q_E(h,E)] =0, =0, (6) A[qAE(h,A,e)] _A[q_AE(h,A,e)] =0, =0, E[qAE(h,a,E)] _E[q_AE(h,a,E)] =0 =0 (7) The decomposition mirrors an analysis-of-variance decomposition for an event-conditioned value function. The terms are identifiable up to function approximation error when expectations are taken under the policy and event distribution: μ(h) μ(h) =A,E[Qπ(h,A,E)] =E_A,E[Q^π(h,A,E)] (8) qA(h,a) q_A(h,a) =E[Qπ(h,a,E)]−μ(h) =E_E[Q^π(h,a,E)]-μ(h) (9) qE(h,e) q_E(h,e) =A[Qπ(h,A,e)]−μ(h) =E_A[Q^π(h,A,e)]-μ(h) (10) qAE(h,a,e) q_AE(h,a,e) =Qπ(h,a,e)−μ(h)−qA(h,a)−qE(h,e) =Q^π(h,a,e)-μ(h)-q_A(h,a)-q_E(h,e) (11) Thus qEq_E captures value changes due only to turnover, while qAEq_AE captures whether the action is more or less useful under a particular event. Term Credit interpretation μ(h)μ(h) expected value of the current history qA(h,a)q_A(h,a) action effect averaged over events qE(h,e)q_E(h,e) pure turnover effect averaged over actions qAE(h,a,e)q_AE(h,a,e) action-specific robustness or fragility to event Table 1: Interpretation of TOCA components. Only qAq_A and qAEq_AE should affect the actor update. Proposition 1: Under Equation 5, using Bπ(h,e)=μ(h)+qE(h,e)B^π(h,e)=μ(h)+q_E(h,e) as a policy-gradient baseline removes all additive pure-turnover offsets without changing the expected policy-gradient direction. Proof sketch: Let c(h,e)c(h,e) be any event-dependent, action-independent offset. Its policy-gradient contribution is [∇θlogπθ(a∣h)c(h,e)]E[ _θ _θ(a h)c(h,e)]. Here the score function is ∇θlogπθ(a∣h) _θ _θ(a h), whose expectation under a∼πθ(⋅∣h)a _θ(· h) is zero. By exogeneity, c(h,e)c(h,e) separates from the action score, so the pure event effect is a valid baseline term. The interaction term does not factor this way because it depends jointly on a and e, and removing it would discard valid action-dependent credit. 4.3 Turnover-Orthogonal Advantage The event-conditioned baseline is Bπ(h,e)=A∼π(⋅∣h)Qπ(h,A,e)=μ(h)+qE(h,e)B^π(h,e)=E_A π(· h)Q^π(h,A,e)=μ(h)+q_E(h,e) (12) It depends on the turnover event but not on the sampled action. The turnover-orthogonal advantage is ATO(h,a,e)=Qπ(h,a,e)−Bπ(h,e)=qA(h,a)+qAE(h,a,e)A^TO(h,a,e)=Q^π(h,a,e)-B^π(h,e)\\ =q_A(h,a)+q_AE(h,a,e) (13) Equation 13 removes pure event credit qEq_E while retaining the action–event interaction. This distinction is important. If an agent is unlucky because a capable teammate is replaced, the pure replacement penalty should not be assigned to that agent. If the same agent previously moved to cover a landmark that a replacement is unlikely to cover, the resulting interaction should remain part of its credit. For per-agent counterfactual credit, define BiTO(h,a−i,e)=ai′∼πi(⋅∣oi)Qπ(h,(ai′,a−i),e)B_i^TO(h,a_-i,e)=E_a_i _i(· o_i)Q^π(h,(a_i ,a_-i),e) (14) and AiTO=Qπ(h,a,e)−BiTO(h,a−i,e)A_i^TO=Q^π(h,a,e)-B_i^TO(h,a_-i,e) (15) This compares agent i’s action to alternatives under the same turnover event. In experiments, we use a shared actor loss with either the event-conditioned baseline in Equation 12 or its softly weighted variant below. 4.4 Soft Interaction Credit In high-variance continuous-control-like tasks, a learned interaction head may be noisy early in training. We therefore introduce TOCA-β: ATO-β(h,a,e)=qA(h,a)+βqAE(h,a,e),β∈[0,1]A^TO-β(h,a,e)=q_A(h,a)+β q_AE(h,a,e), β∈[0,1] (16) The special case β=0β=0 removes interaction credit; β=1β=1 recovers full TOCA. Equivalently, the actor subtracts μ+qE+(1−β)qAEμ+q_E+(1-β)q_AE from the target. We use β=0.5β=0.5 for Dynamic Spread after a small ablation over 0,0.1,0.3,0.5,0.7,1\0,0.1,0.3,0.5,0.7,1\. 5 Method 5.1 Permutation-Invariant Critic The critic receives a variable-size set of active-agent features and an event token. Each active agent contributes a feature vector containing its observation, action, active mask, and optional capability. We use shared encoders followed by pooling to obtain a state representation gsg_s and a state-action representation gsag_sa. A key implementation detail is that the pure baseline terms must be action-independent. Here ϕφ denotes the learnable parameters of the centralized critic, including the shared set encoders and the four output heads fμf_μ, fAf_A, fEf_E, and fAEf_AE: μϕ(h) _φ(h) =fμ(gs), =f_μ(g_s), qE,ϕ(h,e) q_E,φ(h,e) =fE(gs,e), =f_E(g_s,e), (17) qA,ϕ(h,a) q_A,φ(h,a) =fA(gsa), =f_A(g_sa), qAE,ϕ(h,a,e) q_AE,φ(h,a,e) =fAE(gsa,e) =f_AE(g_sa,e) (18) Using state-action features for μ or qEq_E would make the baseline action-dependent and would violate the orthogonality intended by Equation 12. The full critic is Qϕ(h,a,e)=μϕ(h)+qA,ϕ(h,a)+qE,ϕ(h,e)+qAE,ϕ(h,a,e)Q_φ(h,a,e)= _φ(h)+q_A,φ(h,a)+q_E,φ(h,e)+q_AE,φ(h,a,e) (19) The critic is permutation-invariant over active agents because each agent is encoded with shared weights and then pooled over the active set. 5.2 Training Objective The critic is trained by squared Bellman error together with centering regularizers. For a target yt=rt+γVϕ¯(ht+1,et+1)y_t=r_t+γ V_ φ(h_t+1,e_t+1), the value loss is ℒQ=[(Qϕ(ht,at,et)−yt)2]L_Q=E[(Q_φ(h_t,a_t,e_t)-y_t)^2] (20) The centering loss enforces approximate versions of Equations 6–7 using minibatch averages and counterfactual samples: ℒctr=‖^AqA‖2+‖^EqE‖2+‖^AqAE‖2+‖^EqAE‖2L_ctr=\| E_Aq_A\|^2+\| E_Eq_E\|^2+\| E_Aq_AE\|^2+\| E_Eq_AE\|^2 (21) The actor uses a PPO-style clipped objective Schulman et al. (2017) with ATOA^TO or ATO-βA^TO-β. The total loss is ℒ=ℒactor+cvℒQ+cℒctr−cHℋ(π)L=L_actor+c_vL_Q+c_cL_ctr-c_HH(π) (22) where ℋH is policy entropy. 5.3 Practical Estimation In implementation, the expectations in the centering constraints are approximated with minibatch averages over sampled actions and events. For discrete actions, counterfactual action samples can be enumerated for small action spaces or sampled from the current policy. For the replacement event, we sample event tokens from the empirical event distribution in the replayed batch. This produces a lightweight regularizer rather than a separate supervised objective for each component. The critic uses two information streams: The state-only stream encodes active positions, capabilities, masks, and event-independent features. The state-action stream additionally appends one-hot or embedded actions. The separation is crucial: μ and qEq_E are produced only from the state-only stream, while qAq_A and qAEq_AE use the state-action stream. Early experiments with a single state-action stream made the baseline action-dependent and degraded Dynamic Spread performance. This observation is consistent with the theory, because an action-dependent baseline can subtract away the very policy signal it is meant to preserve. TOCA adds little conceptual overhead to MAPPO-style training. The actor and rollout procedure are unchanged. The additional cost is in the critic heads and the centering loss. For a pooled set representation of dimension d, the decomposition heads are small multilayer perceptrons. Thus, the asymptotic cost is dominated by the same set encoder used by the centralized critic. In the experiments below, this overhead was small compared with environment simulation. 5.4 Metrics We use three metric families: average episode return for performance, high-event return for robustness at the largest replacement rate, and diagnostic faithfulness when ground-truth components are available. For j∈A,E,AEj∈\A,E,AE\, component error is Eqj=1||∑(h,a,e)∈|q^j(h,a,e)−qj⋆(h,a,e)|E_q_j= 1|D| _(h,a,e) | q_j(h,a,e)-q_j (h,a,e) | (23) with unused arguments suppressed for qAq_A and qEq_E. Let =t:et≠0S=\t:e_t≠ 0\ be diagnostic turnover-shock steps, Ct⋆=qA⋆(ht,at)+qAE⋆(ht,at,et)C_t =q_A (h_t,a_t)+q_AE (h_t,a_t,e_t), and C^t=q^A(ht,at)+q^AE(ht,at,et) C_t= q_A(h_t,a_t)+ q_AE(h_t,a_t,e_t). Here (⋅)⋆(·) denotes the ground-truth diagnostic component, while (⋅) (·) denotes the corresponding learned critic estimate. We report ShockMAE=1||∑t∈|C^t−Ct⋆|ShockMAE= 1|S| _t | C_t-C_t | (24) which measures action credit assigned around exogenous turnover after removing the pure event term. Lower values indicate more faithful credit. Algorithm 1 Turnover-Orthogonal Actor-Critic Update 1: Collect trajectories with active sets tN_t, actions ata_t, rewards rtr_t, and events ete_t. 2: Encode active-agent set to obtain gsg_s and state-action set to obtain gsag_sa. 3: Predict μϕ _φ, qA,ϕq_A,φ, qE,ϕq_E,φ, and qAE,ϕq_AE,φ. 4: Train Qϕ=μ+qA+qE+qAEQ_φ=μ+q_A+q_E+q_AE with Bellman loss and centering loss. 5: Form actor advantage A=qA+βqAEA=q_A+β q_AE, with β=1β=1 for full TOCA. 6: Update decentralized actor using PPO-style clipping and entropy regularization. 6 Experiments We evaluate two questions: First, does the decomposition recover useful credit signals when the ground-truth sources of return variation are controlled? Second, does turnover-orthogonal credit improve robustness in a MARL control benchmark with replacement events? 6.1 Diagnostic Environment The diagnostic environment is a controlled open-team task with known ground-truth components qA⋆q_A , qE⋆q_E , and qAE⋆q_AE . The return combines additive action and event effects with a multiplicative action–event interaction; event effects vary with replacement rate and capability shock, while the interaction rewards actions useful only under particular turnover events. This benchmark acts as a unit test for credit assignment rather than a full physics task. A method can obtain high return while placing event effects in the action head, so we report both policy return and decomposition faithfulness. We compare MAPPO, an event-aware set critic, TOCA, TOCA without interaction, and TOCA without centering over five seeds, team sizes N∈3,4,5N∈\3,4,5\, and event rates p∈0,0.05,0.1,0.2p∈\0,0.05,0.1,0.2\, yielding 360 completed runs. (a) Policy performance on the diagnostic benchmark Method Avg. Return ↑ MAPPO 319.375±1.267319.375± 1.267 Event-Set-MAPPO 319.419±1.271319.419± 1.271 TOCA 341.436±6.928341.436± 6.928 TOCA w/o interaction 299.437±8.817299.437± 8.817 TOCA w/o centering 374.070±3.422374.070± 3.422 (b) Decomposition quality for methods with explicit TOCA heads Method EqA↓E_q_A EqE↓E_q_E EqAE↓E_q_AE Shock MAE ↓ TOCA 3.486±0.0043.486± 0.004 2.798±0.0192.798± 0.019 1.619±0.0181.619± 0.018 2.135±0.0152.135± 0.015 TOCA w/o interaction 3.503±0.0023.503± 0.002 2.743±0.0252.743± 0.025 1.474±0.0251.474± 0.025 2.164±0.0152.164± 0.015 TOCA w/o centering 0.705±0.1140.705± 0.114 0.902±0.0560.902± 0.056 3.065±0.0663.065± 0.066 3.629±0.1143.629± 0.114 Table 2: Diagnostic benchmark split into policy performance and decomposition quality. Component errors and shock misattribution are reported only for methods that learn explicit TOCA decomposition heads; MAPPO-style baselines do not define qAq_A, qEq_E, or qAEq_AE. Table 2 separates policy performance from decomposition quality. Panel (a) shows that TOCA improves average return over MAPPO and Event-Set-MAPPO. Removing the interaction term reduces return substantially, from 341.436341.436 to 299.437299.437, indicating that action–turnover interaction credit is important in the controlled setting. Panel (b) reports component errors only for methods with explicit decomposition heads. The no-centering ablation achieves the highest raw return, but its interaction error and shock misattribution are much worse. We interpret this as a return–faithfulness tradeoff: without centering, the network can fit returns while mixing the semantic roles of the components. Since the goal of TOCA is credit-faithful decomposition, centering is essential. 6.2 Dynamic Spread with Replacement Events We next evaluate a replacement-only Dynamic Spread benchmark based on cooperative particle navigation. Four active agents must cover landmarks. At each event, a high-capability active agent is replaced by a weak agent while the active team size remains fixed. This isolates turnover from team-size changes and makes the event disruptive: a greedy sanity policy degrades from −7.24-7.24 at event rate 00 to −28.64-28.64 at event rate 0.30.3. The reward is the negative average distance from landmarks to their closest effective covering agent, plus collision penalties. Replacement affects the effective coverage distance by lowering the capability of the incoming agent. This makes replacement harmful even when the new agent inherits the old position. Thus, the event is not merely a random perturbation; it changes the team’s ability to cover landmarks. We use replacement rather than addition or removal because it keeps the active team size fixed and makes the comparison focused on credit assignment rather than variable action-space size. The high-event setting is particularly important. At event rate zero, turnover-aware structure is less useful because the task reduces to an ordinary fixed-team navigation problem. At high event rates, the policy repeatedly observes that teammates change capability, making it valuable to learn which actions are robust to replacement. This is the regime where TOCA should help. We train MAPPO, Event-Set-MAPPO, TOCA-β with β=0.5β=0.5, and TOCA without interaction. We use five seeds and two training event rates, 0.10.1 and 0.20.2, for 40 runs. Evaluation uses event rates 0,0.1,0.2,0.3\0,0.1,0.2,0.3\. Returns are negative distances and penalties, so higher is better. Method Avg. Return ↑ High-Event Return ↑ MAPPO −32.166±0.638-32.166± 0.638 −39.300-39.300 Event-Set-MAPPO −32.090±0.717-32.090± 0.717 −39.214-39.214 TOCA-β (β=0.5β=0.5) −32.063±0.724-32.063± 0.724 −38.329-38.329 TOCA w/o interaction −32.449±0.663-32.449± 0.663 −41.369-41.369 Table 3: Replacement-only Dynamic Spread. TOCA-β is competitive in average return and achieves the best high-turnover return. Table 3 shows that TOCA-β obtains the best mean return, although the margin over Event-Set-MAPPO is modest. The high-event return is more informative: TOCA-β improves over Event-Set-MAPPO and substantially improves over the no-interaction ablation. This suggests that softly weighted interaction credit helps agents remain robust when replacements are frequent. 000.10.10.20.20.30.3−40-40−35-35−30-30Evaluation event rate ppEval. return ↑ Spread by replacement rateMAPPOEvent-SetTOCA-β /o interaction001,0001,0002,0002,0003,0003,000−36-36−34-34−32-32Training episodeTraining return ↑ return over episodesMAPPOEvent-SetTOCA-β /o interaction Figure 1: Dynamic Spread return plots. Left: evaluation return versus replacement event rate, shown with mean lines and shaded standard-error bands for a cleaner comparison. Right: training return versus episode, averaged over five seeds and two training event rates. TOCA-β is most useful at high turnover and learns competitively with the event-aware baseline. Figure 1 shows both evaluation robustness and learning dynamics for Dynamic Spread. At low event rates, MAPPO is best or tied. At high event rates, TOCA-β becomes best. The learning curves show that the methods have similar early training behavior, while TOCA-β remains competitive through the end of training. This matches the motivation for turnover-orthogonal credit: when replacements are rare, ordinary action credit is often sufficient; when replacements are frequent, removing pure turnover effects and preserving action–turnover interactions becomes more useful. Matched-condition paired comparisons give the same picture. Across 40 matched evaluation conditions, TOCA-β improves over Event-Set-MAPPO by 0.825±0.6950.825± 0.695 return with a 67.5%67.5\% win rate, and improves over TOCA without interaction by 2.717±0.8692.717± 0.869. Restricting to high-turnover evaluations p≥0.2p≥ 0.2, the corresponding improvements are 0.986±1.0060.986± 1.006 and 2.934±1.0512.934± 1.051. These results suggest that TOCA-β provides a consistent robustness trend over Event-Set-MAPPO under frequent replacement, while the clearer effect is its improvement over the no-interaction ablation, which directly supports the importance of the action–turnover interaction term qAEq_AE. 6.3 Qualitative Interpretation of Results The diagnostic and Dynamic Spread experiments serve complementary roles. The diagnostic benchmark verifies the decomposition directly: full TOCA outperforms the no-interaction variant, showing that qAEq_AE can be learned and can improve optimization when interaction structure is clean. Dynamic Spread is harder because return depends on geometry, collisions, and replacement-induced capability changes. In that setting, full interaction credit can be noisy, but the softened TOCA-β signal is most useful at high event rates. This pattern supports the main design principle: Pure turnover should be removed as a baseline, while action–turnover interaction should not be discarded. Event-Set-MAPPO is a strong baseline because it observes replacement tokens and predicts event-conditioned returns, but it does not specify which event-conditioned variations are action credit. TOCA-β helps where this distinction matters most: high replacement rates and comparisons against the no-interaction ablation. 6.4 Beta Ablation The beta sweep compares β∈0,0.1,0.3,0.5,0.7,1β∈\0,0.1,0.3,0.5,0.7,1\. We select β=0.5β=0.5 because it gives the strongest high-event return among stable intermediate settings. The ablation suggests that interaction credit is useful, but injecting it at full strength can add variance in noisy control benchmarks. 7 Additional Analysis and Limitations TOCA changes the actor signal from “what return followed this transition?” to “what return followed after accounting for the turnover event?” This matters when actions can change the team’s sensitivity to turnover: an agent may duplicate a teammate’s coverage, hedge against replacement, or communicate state that helps an incoming teammate. If events only add noise and agents cannot respond to them, then qAEq_AE should be small and the no-interaction variant should match TOCA-β. The framework is designed for exogenous or externally specified turnover. If the event itself is caused by an agent’s action, removing qEq_E may remove legitimate action credit; endogenous turnover therefore requires explicit causal modeling. A second limitation is statistical strength: Dynamic Spread is best interpreted as a robustness trend rather than a decisive benchmark victory over Event-Set-MAPPO. The strongest evidence is the diagnostic decomposition and the paired improvement over the no-interaction ablation under high turnover. 8 Broader Applicability Although the experiments focus on cooperative navigation-style tasks, the decomposition is domain-agnostic. Any open-team task with logged joins, departures, failures, or replacements can form event tokens and train the same critic heads. Suitable application classes include robot fleets, warehouse systems, sensor networks, service teams, and human–AI collaboration, where evaluating only average return can hide whether a policy remains effective under unexpected team change. For deployments involving human participants, event descriptors should avoid sensitive or protected attributes; TOCA is a cooperative-control credit assignment method, not a personnel evaluation mechanism. 9 Conclusion We introduced TOCA, a turnover-orthogonal credit assignment framework for open-team MARL. TOCA decomposes event-conditioned value into action, turnover, and interaction terms, removing pure turnover credit while preserving credit for actions that help the team respond to replacement events. Diagnostic experiments validate the decomposition and show that interaction credit matters. Dynamic Spread experiments show that TOCA-β improves high-turnover robustness and outperforms the no-interaction ablation. These results suggest that separating action credit from population-turnover credit is a promising direction for robust learning in dynamic cooperative teams. References Foerster et al. (2018) Foerster, J.; Farquhar, G.; Afouras, T.; Nardelli, N.; and Whiteson, S. 2018. Counterfactual Multi-Agent Policy Gradients. In Proceedings of the AAAI Conference on Artificial Intelligence. Lee et al. (2019) Lee, J.; Lee, Y.; Kim, J.; Kosiorek, A.; Choi, S.; and Teh, Y. W. 2019. Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks. In Proceedings of the International Conference on Machine Learning. Littman (1994) Littman, M. L. 1994. Markov Games as a Framework for Multi-Agent Reinforcement Learning. In Proceedings of the International Conference on Machine Learning. Liu et al. (2021) Liu, B.; Liu, Q.; Stone, P.; Garg, A.; Zhu, Y.; and Anandkumar, A. 2021. Coach-Player Multi-Agent Reinforcement Learning for Dynamic Team Composition. In Proceedings of the International Conference on Machine Learning. Lowe et al. (2017) Lowe, R.; Wu, Y.; Tamar, A.; Harb, J.; Abbeel, P.; and Mordatch, I. 2017. Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments. In Advances in Neural Information Processing Systems. Fu et al. (2022) Fu, W.; Yu, C.; Xu, Z.; Yang, J.; and Wu, Y. 2022. Revisiting Some Common Practices in Cooperative Multi-Agent Reinforcement Learning. In Proceedings of the International Conference on Machine Learning. Khan et al. (2022) Khan, M. J.; Ahmed, S. H.; and Sukthankar, G. 2022. Transformer-Based Value Function Decomposition for Cooperative Multi-Agent Reinforcement Learning. arXiv preprint arXiv:2208.07298. Kuba et al. (2021) Kuba, J. G.; Chen, R.; Wen, M.; Wen, Y.; Sun, F.; Wang, J.; and Yang, Y. 2021. Trust Region Policy Optimisation in Multi-Agent Reinforcement Learning. arXiv preprint arXiv:2109.11251. Mahajan et al. (2019) Mahajan, A.; Rashid, T.; Samvelyan, M.; and Whiteson, S. 2019. MAVEN: Multi-Agent Variational Exploration. In Advances in Neural Information Processing Systems. Mordatch and Abbeel (2018) Mordatch, I.; and Abbeel, P. 2018. Emergence of Grounded Compositional Language in Multi-Agent Populations. In Proceedings of the AAAI Conference on Artificial Intelligence. Oliehoek and Amato (2016) Oliehoek, F. A.; and Amato, C. 2016. A Concise Introduction to Decentralized POMDPs. Springer. Rashid et al. (2018) Rashid, T.; Samvelyan, M.; Schroeder, C.; Farquhar, G.; Foerster, J.; and Whiteson, S. 2018. QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning. In Proceedings of the International Conference on Machine Learning. Schulman et al. (2017) Schulman, J.; Wolski, F.; Dhariwal, P.; Radford, A.; and Klimov, O. 2017. Proximal Policy Optimization Algorithms. arXiv preprint arXiv:1707.06347. Son et al. (2019) Son, K.; Kim, D.; Kang, W. J.; Hostallero, D. E.; and Yi, Y. 2019. QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning. In Proceedings of the International Conference on Machine Learning. Stone and Veloso (2000) Stone, P.; and Veloso, M. 2000. Multiagent Systems: A Survey from a Machine Learning Perspective. Autonomous Robots, 8: 345–383. Sunehag et al. (2018) Sunehag, P.; Lever, G.; Gruslys, A.; Czarnecki, W. M.; Zambaldi, V.; Jaderberg, M.; Lanctot, M.; Sonnerat, N.; Leibo, J. Z.; Tuyls, K.; and Graepel, T. 2018. Value-Decomposition Networks for Cooperative Multi-Agent Learning. In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems. Terry et al. (2021) Terry, J. K.; Black, B.; Grammel, N.; Jayakumar, M.; Hari, A.; Sullivan, R.; Santos, L. S.; Perez, R.; Horsch, C.; Dieffendahl, C.; Williams, N. L.; Lokesh, Y.; Ravi, P.; and Ravindran, P. 2021. PettingZoo: Gym for Multi-Agent Reinforcement Learning. In Advances in Neural Information Processing Systems Datasets and Benchmarks. Wen et al. (2022) Wen, M.; Kuba, J. G.; Lin, R.; Zhang, W.; Wen, Y.; Wang, J.; and Yang, Y. 2022. Multi-Agent Reinforcement Learning is a Sequence Modeling Problem. Advances in Neural Information Processing Systems. Yuan et al. (2023) Yuan, L.; Zhang, Z.; Li, L.; Guan, C.; and Yu, Y. 2023. A Survey of Progress on Cooperative Multi-Agent Reinforcement Learning in Open Environment. arXiv preprint arXiv:2312.01058. Yu et al. (2022) Yu, C.; Velu, A.; Vinitsky, E.; Gao, J.; Wang, Y.; Bayen, A.; and Wu, Y. 2022. The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games. In Advances in Neural Information Processing Systems. Zaheer et al. (2017) Zaheer, M.; Kottur, S.; Ravanbakhsh, S.; Poczos, B.; Salakhutdinov, R.; and Smola, A. 2017. Deep Sets. In Advances in Neural Information Processing Systems.