Paper deep dive
Value-Aware Prediction for Robust Multi-Agent Coordination Under Communication Loss
Kemal Devrim Kafadar, Eren Özaltun, Mahmud Efnan Şanlı, Feyza Orak, Emirhan Gazi, Kubilay Kağan Kömürcü, Nazım Kemal Üre
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 7/21/2026, 2:35:34 AM
Summary
The paper proposes Value-Aware MARO, a method for robust multi-agent coordination under communication loss. It extends the Multi-Agent Observation Sharing under Communication Dropout (MARO) framework by introducing a value-aware predictor objective. This objective dynamically weights the predictor's loss function using advantage estimates from the underlying actor-critic architecture, focusing learning on high-return dynamics rather than stochastic noise. The method is evaluated in the Multi-Agent Particle Environment, showing significant improvements in mean returns and reduced performance variance compared to standard baselines, particularly under low communication reliability.
Entities (8)
Relation Signals (7)
Value-Aware MARO → extends → MARO
confidence 95% · we propose a value-aware extension of Multi-Agent Observation Sharing under Communication Dropout (MARO)
Value-Aware MARO → addresses → communication loss
confidence 93% · Value-Aware Prediction for Robust Multi-Agent Coordination Under Communication Loss
Value-Aware MARO → uses → Advantage Estimate
confidence 92% · By dynamically weighting the predictor's loss function using advantage estimates derived from the underlying actor-critic architecture
Value-Aware MARO → evaluatedon → Multi-Agent Particle Environment
confidence 90% · We evaluate our framework on several tasks within the Multi-Agent Particle Environment
Value-Aware MARO → improves → mean returns
confidence 88% · our method achieves an average improvement in mean returns of more than 20%
Value-Aware MARO → reduces → Performance Variance
confidence 88% · reduces performance variance by a mean of 64.7% compared to the standard unweighted baseline
MARO → uses → LSTM
confidence 85% · a Long Short-Term Memory (LSTM)-based transition model is trained to predict next-step observation deltas
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Robust multi-agent coordination relies heavily on inter-agent communication, which is frequently disrupted by physical and environmental constraints in real-world deployments. To maintain operation during these intermittent communication failures, agents can employ internal prediction models to estimate missing shared state information. However, predictors trained with standard reconstruction objectives treat all transitions equally. In a Reinforcement Learning context, this forces the model to waste capacity learning stochastic exploration noise and the outdated dynamics of suboptimal policies. In this paper, we propose a value-aware extension of Multi-Agent Observation Sharing under Communication Dropout (MARO) to patch communication gaps; we refer to this method as Value-Aware MARO. By dynamically weighting the predictor's loss function using advantage estimates derived from the underlying actor-critic architecture, our objective explicitly couples the predictor's learning process to the policy's evolution. This formulation focuses the model's capacity on the intentional, high-return dynamics actively reinforced by the agents. We evaluate our framework on several tasks within the Multi-Agent Particle Environment under varying communication reliability levels. Experimental results demonstrate that our approach maintains performance under declining communication reliability, particularly below 40%. While our method performs comparably in tasks where the baseline already maintains high coordination, our value-aware weighting effectively prevents the performance collapse observed in the standard predictor during high-attrition scenarios. In these environments, our method achieves an average improvement in mean returns of more than 20% and reduces performance variance by a mean of 64.7% compared to the standard unweighted baseline.
Tags
Links
- Source: https://arxiv.org/abs/2607.17914v1
- Canonical: https://arxiv.org/abs/2607.17914v1
Trouble viewing inline? Open PDF directly →
Full Text
41,189 characters extracted from source content.
Expand or collapse full text
Value-Aware Prediction for Robust Multi-Agent Coordination Under Communication Loss Kemal Devrim Kafadar1∗, Eren Özaltun2∗, Mahmud Efnan Şanlı3, Feyza Orak3, Emirhan Gazi3, Kubilay Kağan Kömürcü3, Nazım Kemal Üre4 ∗ Equal Contribution1Department of Computer Engineering, Istanbul Technical University, Istanbul, Türkiye. kafadar19@itu.edu.tr2Department of Computer Science, University of Stuttgart, Stuttgart, Germany. st201002@stud.uni-stuttgart.de,3Istanbul Technical University Artificial Intelligence and Data Science Application and Research Center, Istanbul, Türkiye. sanli21, orakf20, gazi20, komurcu17@itu.edu.tr4Department of Aeronautics and Astronautics, Stanford University, Stanford, CA 94305, U.S.A. ure@stanford.edu Source code is available at: https://github.com/robust-comm-marl-IROS2026/Value-Aware-Prediction-Under-Communication-LossThis work was supported in part by the Scientific and Technological Research Council of Türkiye (TÜBİTAK) through the 2210-A Scholarship Program awarded to K. D. Kafadar. © 2026 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. Abstract Robust multi-agent coordination relies heavily on inter-agent communication, which is frequently disrupted by physical and environmental constraints in real-world deployments. To maintain operation during these intermittent communication failures, agents can employ internal prediction models to estimate missing shared state information. However, predictors trained with standard reconstruction objectives treat all transitions equally. In a Reinforcement Learning context, this forces the model to waste capacity learning stochastic exploration noise and the outdated dynamics of suboptimal policies. In this paper, we propose a value-aware extension of Multi-Agent Observation Sharing under Communication Dropout (MARO) to patch communication gaps; we refer to this method as Value-Aware MARO. By dynamically weighting the predictor’s loss function using advantage estimates derived from the underlying actor-critic architecture, our objective explicitly couples the predictor’s learning process to the policy’s evolution. This formulation focuses the model’s capacity on the intentional, high-return dynamics actively reinforced by the agents. We evaluate our framework on several tasks within the Multi-Agent Particle Environment under varying communication reliability levels. Experimental results demonstrate that our approach maintains performance under declining communication reliability, particularly below 40%. While our method performs comparably in tasks where the baseline already maintains high coordination, our value-aware weighting effectively prevents the performance collapse observed in the standard predictor during high-attrition scenarios. In these environments, our method achieves an average improvement in mean returns of more than 20% and reduces performance variance by a mean of 64.7% compared to the standard unweighted baseline. I Introduction The successful deployment of Multi-Agent Reinforcement Learning (MARL) in cooperative tasks is primarily dependent on robust inter-agent coordination. Since individual agents typically operate under partial observability, they historically rely on persistent communication to share local sensory data and approximate a cohesive global state [1, 2]. Although Reinforcement Learning (RL) provides a powerful, data-driven paradigm for sequential decision-making in uncertain environments [3], the assumption of perfect connectivity is unrealistic in practical systems such as autonomous swarms, where signal obstruction or bandwidth constraints lead to intermittent connectivity and risk mission failure [4, 5, 6]. Therefore, a resilient control strategy must handle communication losses gracefully to avoid catastrophic performance degradation [7, 8, 9], particularly in aerial robotics, where RL has already demonstrated effectiveness in managing complex kinematics and momentum [10, 11]. To address this challenge, researchers have explored internal state predictors that allow agents to estimate missing shared information during communication gaps. Typically, these predictors are trained using standard supervised learning objectives to model the system dynamics [6, 7, 5, 9, 8]. However, this methodology is fundamentally misaligned with the agents’ learning process. It treats all transitions during RL training with the same importance regardless of their relevance to the evolving policies of the agents. By failing to prioritize intentional dynamics, we argue that standard predictors often waste capacity modeling stochastic exploration noise or behaviors that the RL agents have already learned to discard. Building on Multi-Agent Observation Sharing under Communication Dropout (MARO) [8], we propose a value-weighted predictor objective that leverages the RL agent’s internal success signals to emphasize these critical transitions, ensuring reliable estimates when coordination is most vital. We refer to this method as Value-Aware MARO. Our primary contributions in this work are: • A Value-Aware Predictor with an RL-synchronized training flow that utilizes importance weights to prioritize high-consequence transitions over stochastic environment noise. • A coordination framework that patches communication gaps with value-prioritized estimates, allowing decentralized policies to function even under total signal loss. • Empirical validation against a strong baseline in five tasks, demonstrating significant improvements in robustness under severe communication loss. I Related Work MARL has achieved remarkable success in complex cooperative settings. To address foundational challenges like non-stationarity and partial observability, the Centralized Training with Decentralized Execution (CTDE) framework has become the standard. CTDE leverages global information during training via centralized critics or value factorization to enable scalable, decentralized execution [12, 13, 14, 15]. However, while these methods excel under ideal conditions, sustaining robust coordination under communication uncertainty remains a critical open challenge. Recent works propose differentiable communication learning mechanisms to enhance coordination. Methods such as gradient-based protocols [2], attention-based selective communication [16], and targeted message passing [1] demonstrate that structured information exchange improves cooperative behavior. However, most assume reliable communication during execution. In practical multi-agent systems such as UAV swarms, robotic teams, or distributed sensor networks, communication can be intermittent, delayed, or lost due to environmental or hardware constraints. A growing body of literature studies MARL under communication constraints. Approaches include stochastic message dropout during training [4], recurrent policy networks to maintain memory of past observations [17], and adaptive coordination strategies for heterogeneous systems [18]. Although these methods improve robustness against communication degradation, they generally do not explicitly reconstruct missing global information and often rely on structured or periodic assumptions about communication patterns. To directly address missing information, several works incorporate predictive modeling. Methods include employing Adaptive Predictive Variational Autoencoders [6] and intention-sharing frameworks that encode predicted future trajectories [19]. More directly related to communication loss, Centralized Training with Hybrid Execution (CTHE) via Predictive Observation Imputation [8] imputes missing observations during decentralized execution using a predictor trained with centralized information. Collectively, these studies establish the importance of predictive reconstruction mechanisms; however, they primarily rely on standard reconstruction objectives that treat all state transitions with equal importance during training. Taken together, these studies motivate a next step that not only reconstructs missing information, but also emphasizes robustness where coordination is most critical. Our approach follows this direction by incorporating value awareness into predictive imputation. I Background I-A Problem Formulation: H-POMDP We model the multi-agent coordination problem under intermittent communication using the Hybrid Partially Observable Markov Decision Process (H-POMDP) framework introduced by Santos et al. [8]. Operating over discrete time steps t∈ℕt , an H-POMDP extends the standard Decentralized POMDP (Dec-POMDP) via the tuple ([n],,,,r,γ,Ω,,)([n],S,A,P,r,γ, ,O,C). These components govern the n agents, state space S, joint actions A, transition probabilities P, rewards r, and discount factor γ∈[0,1)γ∈[0,1). The joint observation space is denoted by Ω , with the observation probability function formalized as (o|s,a)O(o|s,a). Crucially, at any step t, the joint observation ot∈Ωo_t∈ is the concatenation of all individual local observations, ot=ot1,…,otno_t=\o_t^1,...,o_t^n\. The defining feature of this formulation is the n×n× n communication matrix C. At any time step t, the matrix entry []i,j=pi,j[C]_i,j=p_i,j denotes the probability of agent i successfully receiving agent j’s local observation otjo_t^j. In a physical robotics context, this localized payload otjo_t^j typically contains agent j’s immediate sensor readouts, such as its spatial coordinates, velocities, or detected target landmarks, as in our experimental tasks. By varying C, this framework naturally interpolates between fully decentralized (=IC=I) and fully centralized execution. In our target-search scenarios, agents face an unknown communication matrix C sampled from a distribution μ during execution. We specifically focus on a probabilistic failure model where shared connections drop with probability 1−p1-p. Consequently, an agent i either receives the joint observation oto_t or falls back to a partial local view. I-B Actor-Critic Architecture in MARL CTDE bridges the gap between the necessity of decentralized execution, dictated by the H-POMDP’s communication constraints, and sample-efficient learning. Our framework builds upon the Proximal Policy Optimization (PPO) algorithm [3], specifically its multi-agent extensions: Independent PPO (IPPO) and Multi-Agent PPO (MAPPO) [20]. Both algorithms utilize an actor-critic architecture. The actor networks, πθi(ati|oti) _θ^i(a_t^i|o_t^i), represent decentralized policies mapping local observations to actions. The critic network, Vω(⋅)V_ω(·), evaluates the quality of these actions by computing an advantage estimate A^t A_t, typically via Generalized Advantage Estimation (GAE) [21]. The critic differs by algorithm: MAPPO conditions on joint information with a single global network, while IPPO uses each agent’s local observation with separate networks. GAE computes the exponentially weighted sum of temporal difference (TD) errors, as defined in (1): A^t=∑l=0∞(γλ)lδt+l A_t= _l=0^∞(γλ)^l _t+l (1) where δt=rt+γVω(ot+1)−Vω(ot) _t=r_t+γ V_ω(o_t+1)-V_ω(o_t) is the TD error, γ is the discount factor, and λ is a hyperparameter controlling the bias-variance tradeoff. This advantage quantifies how much better a taken action was compared to the policy’s average expectation. I-C Observation Imputation To overcome the intermittent communication failures defined by the H-POMDP, our work builds upon the MARO architecture [8]. MARO is designed for hybrid execution in MARL, enabling cooperative agents to operate effectively under arbitrary and dynamic communication levels by employing an autoregressive predictive model that estimates missing information from prior observations. During the centralized training phase, a Long Short-Term Memory (LSTM)-based transition model is trained to predict next-step observation deltas, denoted as Δot o_t, given the current joint observations oto_t and a recurrent history state hth_t. The parameters of this predictive model, ϕφ, are optimized via supervised learning by minimizing the negative log-likelihood (NLL) of the target next-step deltas across all agents, as formulated in (2): ℒMARO(ϕ)=−∑i=1nlogpϕ(Δoti|ot,ht)L_MARO(φ)=- _i=1^n p_φ( o_t^i|o_t,h_t) (2) During decentralized execution, each agent i maintains an independent instance of this predictive model to impute any dropped observations from teammates. This preserves a stable, estimated joint-observation state for the policy to act upon. While MARO provides a robust architectural foundation for patching missing data, its standard NLL objective treats all observed transitions equally. Section IV details our proposed framework to directly address this fundamental misalignment. IV Methodology Figure 1: MARLxPredictor Integration During Execution: The predictor intercepts local observations to patch missing shared data during communication failures, providing a reconstructed state to the policy where otio^i_t is local observation of agent i at time t and ht−1h_t-1 is the hidden state of the LSTM-based predictor network. Figure 2: Value-Aware Training Pipeline: The RL training phase generates advantage estimates A^t A_t, which are detached and used as importance weights for the predictor’s Advantage Weighted NLL Loss (formally defined as Jbatch′(ϕ)J _batch(φ)), ensuring focus on intentional dynamics. In this section, we present a control framework designed to sustain multi-agent coordination under varying degrees of communication failure. We adopt a CTDE architecture where agents rely on local sensor data and intermittent shared state information from allies. To bridge the informational gaps caused by signal loss, we integrate a standalone prediction model that imputes the missing parts of the shared information and operates in parallel with the policy network. Unlike standard approaches that treat all state dynamics equally, as discussed in previous sections, our framework introduces a value-aware training objective. This mechanism couples the predictor’s learning process directly to the policy’s evolution, ensuring that the generated state estimates during communication blackouts are optimized to maintain cooperative performance rather than minimizing generic reconstruction error. Operating within the H-POMDP framework detailed in Section I, our system actively mitigates the performance degradation caused by probabilistic signal drops. We achieve this by predicting the missing shared information, enabling the decentralized policy πθi(ati|o^t) _θ^i(a_t^i| o_t) to act on an approximated joint observation and preserve swarm coordination despite severe communication loss. We detail the prediction architecture and derive our proposed objective in the remainder of this section. IV-A Observation Prediction under Signal Loss The employed prediction model, pϕ(Δot|ot)p_φ( o_t|o_t), is tasked with estimating the evolution of the environment dynamics. Rather than predicting the next full observation directly, the model learns to predict the change in observations, denoted as Δot=ot+1−ot o_t=o_t+1-o_t. During execution, the predictor intercepts the raw sensory stream before it reaches the policy network and continuously models the environment in parallel, following the system architecture illustrated in Fig. 2. At each step, it utilizes the available joint observation from the previous timestep, ot−1o_t-1, to estimate the forward transition Δ^ot−1∼pϕ(Δot−1|ot−1) o_t-1 p_φ( o_t-1|o_t-1), computing an expected current full observation o^t≈ot−1+Δ^ot−1 o_t≈ o_t-1+ o_t-1. The predictor then evaluates the integrity of the communication link. If the signal is fully intact, the policy receives the ground-truth observation. However, upon detecting signal loss with an agent, the predictor estimates the missing communicated components in the agent’s current partial observation. This reconstructed observation is then passed to the policy πθ _θ, which processes it as an imitation of the real observation, allowing the agent to continue its cooperative task without explicit awareness of the underlying communication failure. IV-B Value-Aware Predictor Objective We formalize the Value-Aware objective illustrated in Fig. 2, which addresses the equal-weighting limitation of the existing methods discussed in Sec. I by scaling each transition’s contribution using the critic’s advantage estimate, focusing learning on intentional, high-return dynamics. In practice, this objective serves as the empirical loss that is minimized during the predictor’s optimization step. Formally, our proposed objective for the prediction model over a batch of transitions is defined in (3): Jbatch′(ϕ)=t[wt(−logpϕ(Δot|ot))]J _batch(φ)=E_t [w_t (- p_φ( o_t|o_t) ) ] (3) where the normalized advantage-based importance weight wt=w~t/t′[w~t′]w_t= w_t\,/\,E_t [ w_t ] with w~t=ReLU(1+λA^t) w_t=ReLU(1+λ A_t) is derived directly from the critic’s advantage estimate A^t A_t, and λ≥0λ≥ 0 is a scaling hyperparameter. To obtain our objective, we derive the predictor’s loss evolution with respect to the policy’s parameter updates. The objective function of the predictor model for a single transition (o,Δo)(o, o) is defined as the NLL of the observed changes in (4): J(ϕ)=−logpϕ(Δo|o)J(φ)=- p_φ( o|o) (4) where Δo=(a,o)−o o=T(a,o)-o represents the change in observations. Here, T is the environment’s transition function, and the joint action a is sampled from the joint policy a∼πθ(a|o)=∏i=1nπθi(ai|oi)a _θ(a|o)= _i=1^n _θ^i(a^i|o^i). For notational simplicity in the following derivation, we treat the decentralized multi-agent system as a single coordinated entity and derive the objective with respect to the joint policy parameters θ. Substituting these into (4), we obtain (5): J(ϕ)=−logpϕ((a,o)−o|o)J(φ)=- p_φ(T(a,o)-o|o) (5) To understand how the predictor’s loss changes as the policy parameters θ evolve, we consider the first-order Taylor expansion of (5) with respect to θ, as formulated in (6): J′(ϕ)≈J(ϕ)+ΔJ(ϕ)≈J(ϕ)+∇θJ(ϕ)TΔθJ (φ)≈ J(φ)+ J(φ)≈ J(φ)+ _θJ(φ)^T θ (6) The gradient ∇θJ(ϕ) _θJ(φ) describes how an infinitesimal change in the policy’s action selection affects the resulting predictor loss. We derive this relationship in (7) by using the log-derivative trick: ∇θJ(ϕ) _θJ(φ) =∇θ∫πθ(a|o)[−logpϕ(Δo|o)]a = _θ _θ(a|o)[- p_φ( o|o)]\,da =∫πθ(a|o)∇θlogπθ(a|o)[−logpϕ(Δo|o)]a = _θ(a|o) _θ _θ(a|o)[- p_φ( o|o)]\,da =a∼πθ[∇θlogπθ(a|o)(−logpϕ(Δo|o))] =E_a _θ [ _θ _θ(a|o) (- p_φ( o|o) ) ] (7) In PPO, the policy parameters are updated to maximize a clipped surrogate objective, which prevents excessively large policy updates. For a given state-action pair (o,a)(o,a), the objective is defined in (8): LCLIP(θ)=t[min(rt(θ)A^t,clip(rt(θ),1−ϵ,1+ϵ)A^t)]L^CLIP(θ)=E_t [ (r_t(θ) A_t,clip(r_t(θ),1-ε,1+ε) A_t ) ] (8) where rt(θ)=πθ(at|ot)πθold(at|ot)r_t(θ)= _θ(a_t|o_t) _ _old(a_t|o_t) is the probability ratio. To determine the update direction Δθ θ, we consider the gradient of this objective. Within the trust region, the gradient with respect to θ is evaluated in (9): ∇θLCLIP(θ) _θL^CLIP(θ) ≈t[∇θπθ(at|ot)πθold(at|ot)A^t] _t [ _θ _θ(a_t|o_t) _ _old(a_t|o_t) A_t ] =t[πθ(at|ot)πθold(at|ot)∇θlogπθ(at|ot)A^t] =E_t [ _θ(a_t|o_t) _ _old(a_t|o_t) _θ _θ(a_t|o_t) A_t ] (9) Evaluating this gradient at θ=θoldθ= _old simplifies the ratio rt(θ)r_t(θ) to unity. The resulting parameter update Δθ θ for a single transition is proportional to this stochastic gradient, as formulated in (10): Δθ∝αA^∇θlogπθ(a|o) θ α A _θ _θ(a|o) (10) where α is a positive constant. This term represents the direction in the parameter space that most efficiently increases the expected advantage for the current transition. To motivate our surrogate predictor objective J′(ϕ)J (φ), we substitute the expression for ∇θJ(ϕ) _θJ(φ) from (7) for a single transition, and the policy parameter update Δθ θ from (10) into the first-order Taylor expansion in (6), yielding (11): J′(ϕ) J (φ) ≈J(ϕ)+∇θJ(ϕ)TΔθ ≈ J(φ)+ _θJ(φ)^T θ ≈J(ϕ)+(∇θlogπθ(a|o)[−logpϕ(Δo|o)])T ≈ J(φ)+ ( _θ _θ(a|o)[- p_φ( o|o)] )^T ⋅(αA^∇θlogπθ(a|o)) · (α A _θ _θ(a|o) ) =J(ϕ)+αA^∥∇θlogπθ(a|o)∥2[−logpϕ(Δo|o)] =J(φ)+α A\| _θ _θ(a|o)\|^2[- p_φ( o|o)] (11) The term ∥∇θlogπθ(a|o)∥2\| _θ _θ(a|o)\|^2 in (11) represents the squared magnitude of the policy gradient for the given transition. While this magnitude varies per sample, it is a strictly non-negative scaling factor that does not change the direction of the advantage signal. To formulate a practical surrogate objective, we employ a heuristic approximation: we treat the product of the constant α and this varying magnitude as a single constant hyperparameter λ≥0λ≥ 0. This greatly simplifies optimization and reduces computational overhead during training. Substituting J(ϕ)=−logpϕ(Δo|o)J(φ)=- p_φ( o|o) back into the expression, we obtain the heuristically weighted loss in (12): J′(ϕ) J (φ) ≈[−logpϕ(Δo|o)]+λA^[−logpϕ(Δo|o)] ≈[- p_φ( o|o)]+λ A[- p_φ( o|o)] =(1+λA^)[−logpϕ(Δo|o)] =(1+λ A)[- p_φ( o|o)] (12) This formulation effectively weights the predictor’s loss based on the advantage A A calculated by the RL agent. It ensures that the model prioritizes learning from transitions that contribute most significantly to the policy update. While the previous theoretical motivation focuses on the point-wise impact of a single transition, our final surrogate objective must operate over mini-batches sampled from a replay buffer. To transition to a stable, implementable empirical loss, we first define the unnormalized importance weights as w~t=ReLU(1+λA^t) w_t=ReLU(1+λ A_t). The ReLU operator ensures non-negativity, preventing gradient inversion from highly negative advantages and the resulting destabilization of training. We then normalize these weights across the mini-batch to yield wtw_t, stabilizing the effective update scale as advantage magnitudes fluctuate during training. This results in our final, batch-level surrogate objective function for the prediction model, shown in (13): Jbatch′(ϕ)=t[wt(−logpϕ(Δot|ot))]J _batch(φ)=E_t [w_t (- p_φ( o_t|o_t) ) ] (13) Ultimately, the proposed objective Jbatch′(ϕ)J _batch(φ) in (13) acts as a value-aware surrogate loss. By dynamically weighting the predictor’s NLL, the model explicitly filters stochastic exploration noise and prioritizes transitions that align with the current cooperative strategy. This ensures the predictor’s capacity is dedicated to modeling the intentional dynamics that generate high returns, naturally tracking shifts in behavior as the policy evolves. This formulation serves as the core predictive objective utilized throughout our experiments. V Experiments V-A Experimental Setup The performance of Value-Aware MARO is evaluated across five multi-agent coordination tasks. These tasks, derived from the benchmark suite proposed by Santos et al. [8], are designed as H-POMDPs to emphasize the necessity of inter-agent information sharing. The information exchanged in these tasks, such as ally and target coordinates, mirrors what a physical swarm can share and lose under degraded communication. They utilize the Multi-Agent Particle Environment (MPE) physics engine [22], incorporating continuous-space kinematics, inertia, and friction to simulate realistic multi-agent interactions where agents try to reach targets and avoid collisions by communicating. Detailed physical dynamics for the environment are provided in Appendix A with illustrations in Fig. 5. The tasks used are: • HearSee (HS): A heterogeneous task where the “Hear” agent observes the target landmark but does not know its own coordinates, while the “See” agent observes agent coordinates but not the target. • SpeakerListener (SL): A stationary “Speaker” observes a target and must guide a mobile “Listener” who is blind to the goal. • SimpleSpreadXY-2 & SimpleSpreadXY-4 (SXY-2 & SXY-4): Navigation tasks with 2 or 4 agents where they observe only one axis (x or y) and must communicate to resolve the full joint observation. • SpreadBlindfold (SBF): A three-agent task where local observations are restricted to an agent’s own state, requiring communication to localize peers. As established in our H-POMDP formalization (Sec. I), we model communication failures probabilistically. We evaluate the system across connectivity levels p∈1.0,0.8,0.6,0.4,0.2,0.0p∈\1.0,0.8,0.6,0.4,0.2,0.0\, representing transitions from full centralization to total communication blackout. V-A1 Training and Evaluation Protocol Following the CTHE paradigm, we utilize MAPPO as the underlying RL algorithm. To isolate the impact of our proposed objective, we compare the baseline MARO against our Value-Aware MARO. Tables I and I in Appendix B show details about the configuration for RL and predictive models. Note that all models were trained with a p=1.0p=1.0 sampling scheme for communication to ensure the availability of ground-truth signals for the advantage-weighted objective. V-B Results and Analysis The training results for the HS task through training are shown in Fig. 3. In the extreme case of total communication loss (p=0.0p=0.0), the baseline MARO fails to converge to a coordinated policy, resulting in a significantly lower average return. In contrast, the Value-Aware objective allows the agents to reconstruct the specific observation deltas that correlate with high-advantage states. This focused prediction enables the swarm to maintain an effective estimation of the observations, leading to a higher and more stable reward trajectory throughout the training process. We also provide all training curves for the remaining tasks for MAPPO and also IPPO to show generalization across algorithms in Figs. 6 and 7 in Appendix C. Figure 3: MAPPOxMARO comparisons between performances of evaluations throughout training in the HS environment. We evaluate the average return across various levels of communication constraints (p∈1.0,0.8,0.6,0.4,0.2,0.0p∈\1.0,0.8,0.6,0.4,0.2,0.0\). The plots demonstrate that our proposed Value-Aware MARO method maintains higher performance compared to the baseline MARO, particularly as communication bandwidth becomes increasingly restricted. Environment Predictor No Comm (p=0.0p=0.0) Low Comm (p=0.2p=0.2) Full Comm (p=1.0p=1.0) HS Baseline −102.6±26.9-102.6± 26.9 −35.9±4.4-35.9± 4.4 −24.4±0.7-24.4± 0.7 Value-Aware −37.5±2.9-37.5± 2.9 −26.3±1.2-26.3± 1.2 −24.3±1.2-24.3± 1.2 SXY-4 Baseline −1264.4±19.9-1264.4± 19.9 −981.7±17.6-981.7± 17.6 −834.4±11.4-834.4± 11.4 Value-Aware −1133.6±9.3-1133.6± 9.3 −937.0±15.8-937.0± 15.8 −834.4±11.4-834.4± 11.4 SXY-2 Baseline −213.8±20.6-213.8± 20.6 −171.4±6.5-171.4± 6.5 −153.8±0.7-153.8± 0.7 Value-Aware −188.9±2.6-188.9± 2.6 −163.5±1.6-163.5± 1.6 −153.8±0.7-153.8± 0.7 SBF Baseline −422.0±3.9-422.0± 3.9 −421.5±5.1-421.5± 5.1 −423.4±2.5-423.4± 2.5 Value-Aware −424.6±2.9-424.6± 2.9 −423.5±4.5-423.5± 4.5 −423.4±4.3-423.4± 4.3 SL Baseline −27.0±3.5-27.0± 3.5 −26.6±1.1-26.6± 1.1 −27.2±2.2-27.2± 2.2 Value-Aware −27.4±0.9-27.4± 0.9 −29.0±0.2-29.0± 0.2 −27.2±1.0-27.2± 1.0 TABLE I: Average evaluation returns after training. Our Value-Aware objective prevents the performance collapse seen in the baseline as communication probability p decreases. The converged performance metrics across all five tasks are summarized in Table I. While the baseline MARO’s performance drops drastically as communication levels decrease, our advantage-weighted objective improves it significantly across every task where the baseline experiences performance loss, and also decreases the standard deviation. This indicates that the advantage-based weighting prevents the predictor from being distracted by observation noise during RL training. Figure 4: Robustness profile across three coordination tasks: SXY-4, SXY-2, and HS. The plots illustrate the average return as a function of communication probability p∈[0,1]p∈[0,1], highlighting the performance of Value-Aware MARO compared to the baseline under varying degrees of connectivity. Finally, Fig. 4 illustrates the performances of the converged policies under all communication levels for the tasks in which the baseline MARO has drastic performance drops. While the baseline MARO exhibits a sharp performance collapse as p drops below 0.40.4, Value-Aware MARO maintains a much higher performance. The overlap in performance at p=1.0p=1.0 confirms that both methods start with the same RL performance, ensuring that the comparisons are fair. This shows that any performance gaps at lower communication levels are caused solely by the predictors , not by differences in the underlying policy. This effect is further amplified as p decreases, where predicted observations increasingly dominate the agent’s input; consequently, the quality of those predictions has a compounding effect on policy performance, making the value-aware prioritization of high-consequence transitions progressively more critical. VI Conclusion In this study, we introduced an objective that prioritizes the policy’s learning signals for prediction models in hybrid execution settings for MARL systems operating under unreliable communication. By weighting the predictor’s loss with the RL advantage signal, our framework ensures that models track the intentional dynamics of the cooperative strategy rather than environment noise. Our results across five tasks show that our approach improves performance, particularly in high communication loss scenarios. The current study evaluates up to four agents on MPE tasks under a probabilistic communication dropout model. Extending the framework to larger teams, temporally correlated or bandwidth-limited communication, and more complex environments remains an open direction. Ultimately, deployment on physical drone swarms is a natural next step toward real-world applicability. References [1] A. Das, T. Gervet, J. Romoff, D. Batra, D. Parikh, M. Rabbat, and J. Pineau, “Tarmac: Targeted multi-agent communication,” in International Conference on machine learning. PMLR, 2019, p. 1538–1546. [2] J. Foerster, I. A. Assael, N. De Freitas, and S. Whiteson, “Learning to communicate with deep multi-agent reinforcement learning,” Advances in neural information processing systems, vol. 29, 2016. [3] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” arXiv preprint arXiv:1707.06347, 2017. [4] W. Kim, M. Cho, and Y. Sung, “Message-dropout: An efficient training method for multi-agent deep reinforcement learning,” in Proceedings of the AAAI conference on artificial intelligence, vol. 33, no. 01, 2019, p. 6079–6086. [5] R. Zhang, J. Wang, J. Ge, and Q. Huang, “Multiagent cooperative search learning with intermittent communication,” IEEE Intelligent Systems, vol. 39, no. 2, p. 11–20, 2024. [6] L. Fu, J. Wang, and H. Luo, “Multi-agent reinforcement learning for cooperative search under aperiodically intermittent communication,” Expert Systems with Applications, vol. 280, p. 127526, 2025. [7] J. Bloom, P. Paliwal, A. Mukherjee, and C. Pinciroli, “Decentralized multi-agent reinforcement learning with global state prediction,” in 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2023, p. 8854–8861. [8] P. P. Santos, D. S. Carvalho, M. Vasco, A. Sardinha, P. A. Santos, A. Paiva, and F. S. Melo, “Centralized training with hybrid execution in multi-agent reinforcement learning via predictive observation imputation,” Artificial Intelligence, p. 104404, 2025. [9] Z. Zhang, B. Cheng, Z. Wang, Y. Zhou, G. Li, P. Lu, B. He, and J. Chen, “Pagnet: Pluggable adaptive generative networks for information completion in multi-agent communication,” arXiv preprint arXiv:2502.03845, 2025. [10] J. Hwangbo, I. Sa, R. Siegwart, and M. Hutter, “Control of a quadrotor with reinforcement learning,” IEEE Robotics and Automation Letters, vol. 2, no. 4, p. 2096–2103, 2017. [11] Y. Song, M. Steinweg, E. Kaufmann, and D. Scaramuzza, “Autonomous drone racing with deep reinforcement learning,” in 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2021, p. 1205–1212. [12] R. Lowe, Y. I. Wu, A. Tamar, J. Harb, O. Pieter Abbeel, and I. Mordatch, “Multi-agent actor-critic for mixed cooperative-competitive environments,” Advances in neural information processing systems, vol. 30, 2017. [13] J. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson, “Counterfactual multi-agent policy gradients,” in Proceedings of the AAAI conference on artificial intelligence, vol. 32, no. 1, 2018. [14] T. Rashid, M. Samvelyan, C. S. De Witt, G. Farquhar, J. Foerster, and S. Whiteson, “Monotonic value function factorisation for deep multi-agent reinforcement learning,” Journal of Machine Learning Research, vol. 21, no. 178, p. 1–51, 2020. [15] P. Sunehag, G. Lever, A. Gruslys, W. M. Czarnecki, V. Zambaldi, M. Jaderberg, M. Lanctot, N. Sonnerat, J. Z. Leibo, K. Tuyls, et al., “Value-decomposition networks for cooperative multi-agent learning,” arXiv preprint arXiv:1706.05296, 2017. [16] J. Jiang and Z. Lu, “Learning attentional communication for multi-agent cooperation,” Advances in neural information processing systems, vol. 31, 2018. [17] R. E. Wang, M. Everett, and J. P. How, “R-maddpg for partially observable environments and limited communication,” arXiv preprint arXiv:2002.06684, 2020. [18] N. K. Ure, S. Omidshafiei, B. T. Lopez, A.-a. Agha-Mohammadi, J. P. How, and J. Vian, “Online heterogeneous multiagent learning under limited communication with applications to forest fire management,” in 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2015, p. 5181–5188. [19] W. Kim, J. Park, and Y. Sung, “Communication in multi-agent reinforcement learning: Intention sharing,” in International conference on learning representations, 2020. [20] C. Yu, A. Velu, E. Vinitsky, J. Gao, Y. Wang, A. Bayen, and Y. Wu, “The surprising effectiveness of ppo in cooperative multi-agent games,” Advances in neural information processing systems, vol. 35, p. 24 611–24 624, 2022. [21] J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel, “High-dimensional continuous control using generalized advantage estimation,” arXiv preprint arXiv:1506.02438, 2015. [22] J. Terry, B. Black, N. Grammel, M. Jayakumar, A. Hari, R. Sullivan, L. S. Santos, C. Dieffendahl, C. Horsch, R. Perez-Vicente, et al., “Pettingzoo: Gym for multi-agent reinforcement learning,” Advances in Neural Information Processing Systems, vol. 34, p. 15 032–15 043, 2021. Appendix A Physical Dynamics of MPE We utilize the MPE [22], a continuous-space physics engine with the following constraints: 1) Kinematics and Inertia: Agents are circular entities with mass m=1.0m=1.0 kg and radius r=0.05r=0.05. State transitions follow semi-implicit Euler integration (dt=0.1dt=0.1s), where actions apply forces (F=5.0F=5.0 N), which leads the predictor to model second-order temporal dynamics. 2) Friction and Damping: A linear damping factor of 0.250.25 simulates air resistance, where agents retain 75% of their velocity per step. The velocity update is: vt+1=vt×(1−0.25)+Fmdtv_t+1=v_t×(1-0.25)+ Fmdt (14) 3) Collision Physics: MPE uses a soft-repulsion model. When distance d<ri+rjd<r_i+r_j, a 100100 N contact force is applied via a log-sum-exp formulation for differentiable gradients, and these interactions cause high-frequency velocity changes. Fig. 5 illustrates the environments, highlighting their specific semantics and communication-based dependencies. (a) HS (b) SL (c) SXY (d) SBF Figure 5: Representative illustrations of the MPE tasks, demonstrating the specific task objectives and communication-based dependencies of the swarm. Appendix B Experimental Hyperparameters For fair comparisons, the RL agents were trained using two primary configurations as shown in Table I. Setting 1 was utilized for most environments, while Setting 2 was applied to SXY-4 and SBF since it increased the pure RL performance in those scenarios. Table I shows the hyperparameters of the predictor models. Notably, the baseline MARO retains the exact optimal settings prescribed in its original implementation. We preserved these original baseline parameters for a fair comparison. For Value-Aware MARO, the predictor batch size is set to match the RL batch size, as the advantage estimates A^t A_t are computed per RL minibatch and we keep them structurally aligned rather than using a separate buffer to store the transitions and sample later. A static advantage weight of λ=1.0λ=1.0 is maintained across all tasks to demonstrate robustness without environment-specific tuning. TABLE I: Reinforcement Learning Hyperparameters (MAPPO/IPPO) Hyperparameter Setting 1 Setting 2 Learning Rate 1×10−41× 10^-4 3×10−43× 10^-4 Entropy Coefficient 0.0010.001 0.010.01 Network Type MLP GRU Batch Size 10 10 Hidden Dimensions 256 256 Reward Standardization True True n-step 5 5 TABLE I: Predictor Hyperparameters (Value-Aware MARO) Hyperparameter Value Architecture LSTM Hidden Dimension 128 Learning Rate 1×10−31× 10^-3 Advantage Weight (λ) (VA) 1.0 Gradient Clipping 1.0 Batch Size (VA / Baseline) 10 / 32 Buffer Size 5,000 Training Comm. Prob (p) 1.0 Appendix C Extended Experimental Results Extended experimental results are given in Figs. 6 and 7. For further details about the environments, models and experiments; please check the github repository at: https://github.com/robust-comm-marl-IROS2026/Value-Aware-Prediction-Under-Communication-Loss (a) (b) (c) (d) Figure 6: Extended training performance with MAPPO across varying communication levels where p∈1.0,0.8,0.6,0.4,0.2,0.0p∈\1.0,0.8,0.6,0.4,0.2,0.0\. The tasks shown are: (a) SXY-4, (b) SXY-2, (c) SBF, and (d) SL. The Value-Aware objective (blue) consistently demonstrates superior robustness compared to the baseline MARO (orange). (a) SXY-4 (b) SXY-2 (c) HS (d) SBF (e) SL Figure 7: Evaluation curves for IPPO during training of each task are given as comparisons between MARO and Value-Aware MARO to show generalization. The Value-Aware objective remains algorithm-agnostic, providing superior coordination compared to the baseline across both centralized and independent optimization schemes.