Paper deep dive
CCKS: Consensus-based Communication and Knowledge Sharing
Jinyuan Zu, Xiaowei Lv, Yongcai Wang, Deying Li, Yunjun Han, Wenping Chen, Fengyi Zhang, Naiqi Wu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 7/9/2026, 4:49:34 AM
Summary
The paper introduces CCKS, a Consensus-based Communication and Knowledge Sharing framework for Decentralized Training and Decentralized Execution (DTDE) in Multi-Agent Reinforcement Learning (MARL). It addresses excessive advising and poor teacher-student compatibility by employing contrastive learning to build consensus models. Agents score and select actions based on shared knowledge and a 'think twice' mechanism, significantly improving cooperation efficiency, learning speed, and performance on SMAC and GRF benchmarks compared to baselines like IQL.
Entities (8)
Relation Signals (7)
CCKS → evaluatedon → SMAC
confidence 97% · Experiments conducted in the Google Research Football environment and the complex StarCraft II Multi-Agent Challenge demonstrate that the integration with CCKS significantly improves cooperation efficiency.
CCKS → evaluatedon → GRF
confidence 97% · Experiments conducted in the Google Research Football environment and the complex StarCraft II Multi-Agent Challenge demonstrate that the integration with CCKS significantly improves cooperation efficiency.
CCKS → employs → Consensus Model
confidence 96% · The key is the consensus model construction, for which we propose to employ contrastive learning to construct consensus models based on local observations.
CCKS → improves → DTDE
confidence 94% · CCKS integrates seamlessly with existing DTDE algorithms and addresses training instability and suboptimal performance inherent to DTDE.
Consensus Model → trainedvia → Contrastive Learning
confidence 93% · we propose to employ contrastive learning to construct consensus models based on local observations in the agents' training phase.
CCKS → outperforms → IQL
confidence 91% · Integrating CCKS with Independent Q-Learning (IQL) accelerates learning, improves decision accuracy, and enhances stability compared to current DTDE baselines.
CCKS → incorporates → Think Twice Mechanism
confidence 89% · a 'think twice before you leap' mechanism where agents resample when low-probability actions are suggested, avoiding premature suboptimal decisions.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:In Decentralized Training and Decentralized Execution (DTDE) for cooperative Multi-Agent Reinforcement Learning (MARL), action-advising-based knowledge sharing promotes interpretable and scalable cooperation among agents. However, current action advising approaches often adhere too much to the teacher's guidance without evaluating teacher-student compatibility, which causes excessive advising, suboptimal stability, and degraded performance. To overcome these challenges, this paper presents a Consensus-based Communication and Knowledge Sharing (CCKS) framework, which allows agents to adopt recommendations based on consensus-derived constraints and to follow the teacher's instructions more smartly. This mechanism enables agents to balance exploration and learning from experienced teachers, improving overall performance. The key is the consensus model construction, for which we propose to employ contrastive learning to construct consensus models based on local observations in the agents' training phase. In action selection, agents score and choose actions based on consensus and shared knowledge. Designed as a plug-and-play solution, CCKS integrates seamlessly with existing DTDE algorithms. Experiments conducted in the Google Research Football environment and the complex StarCraft II Multi-Agent Challenge demonstrate that the integration with CCKS significantly improves cooperation efficiency, learning speed, and overall performance compared with current DTDE baselines. The code is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2606.12281v1
- Canonical: https://arxiv.org/abs/2606.12281v1
Trouble viewing inline? Open PDF directly →
Full Text
32,727 characters extracted from source content.
Expand or collapse full text
CCKS: Consensus-based Communication and Knowledge Sharing Jinyuan Zu 1,+ , Xiaowei Lv 1 , Yongcai Wang 1,∗ , Deying Li 1 , Yunjun Han 2 Wenping Chen 1 , Fengyi Zhang 3 and Naiqi Wu 4 Abstract— In Decentralized Training and Decentralized Ex- ecution (DTDE) for cooperative Multi-Agent Reinforcement Learning (MARL), action-advising-based knowledge sharing promotes interpretable and scalable cooperation among agents. However, current action advising approaches often adhere too much to the teacher’s guidance without evaluating teacher- student compatibility, which causes excessive advising, subop- timal stability, and degraded performance. To overcome these challenges, this paper presents a Consensus-based Commu- nication and Knowledge Sharing (CCKS) framework, which allows agents to adopt recommendations based on consensus- derived constraints and to follow the teacher’s instructions more smartly. This mechanism enables agents to balance exploration and learning from experienced teachers, improving overall performance. The key is the consensus model construction, for which we propose to employ contrastive learning to construct consensus models based on local observations in the agents’ training phase. In action selection, agents score and choose actions based on consensus and shared knowledge. Designed as a plug-and-play solution, CCKS integrates seamlessly with existing DTDE algorithms. Experiments conducted in the Google Research Football environment and the complex StarCraft I Multi-Agent Challenge demonstrate that the integration with CCKS significantly improves cooperation efficiency, learning speed, and overall performance compared with current DTDE baselines. The code is available athttps://github.com/ yuanxpy/CCKS. I. INTRODUCTION Multi-agent reinforcement learning (MARL) enables mul- tiple agents to collaborate toward common goals by learning policies that maximize cumulative returns [1], [2], [3]. Most MARL tasks withnagents are modeled as Decentral- ized Partially Observable Markov Decision Processes (Dec- POMDPs) [4], emphasizing the agents only get the local observations.Based on Dec-POMDP, two paradigms have been recently proposed: “Centralized Training and Decentralized Execution” (CTDE) [5], [6], [7], [8] and “Decentralized Training and Decentralized Execution” (DTDE) [9], [10], *This work was supported by the the National Natural Science Foundation of China Grant No.12071478 and No.61972404, Public Computing Cloud, Renmin University of China, and the Blockchain Lab. School of Information, Renmin University of China. 1 School of Information, Renmin University of China, Beijing 100872, China 2 State Key Laboratory of Multimodal Artificial Intelligence Systems, Beijing Engineering Research Center of Intelligent Systems and Technology, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China 3 The Information Science Academy, China Electronics Technology Group Corporation, Beijing 100043, China 4 Department of Mechatronics Engineering, Guangdong University of Technology, Guangzhou 510090, China + Jinyuan Zu is the co-first author ∗ Yongcai Wang is the corresponding author: ycw@ruc.edu.cn [11], [12]. DTDE offers greater adaptability in complex real- world scenarios by eliminating centralized training but suffers from training instability [5] and suboptimal performance due to insufficient inter-agent coordination. To address the issue of insufficient cooperation in DTDE, communication-based knowledge sharing has been proposed, where agents exchange local observations or processed information to improve policy [13], [14], [15]. However, these schemes often exhibit limitations in interpretability and generalizability. Action advising, inspired by teacher-student paradigms, significantly reduces both training and decision- making costs by having teachers directly suggest actions rather than share raw information with better interpretability [16], [17], [18]. Yet, current implementations face challenges: students may blindly follow advice from poorly chosen teachers, slowing training or degrading performance, and over-imitation can limit cooperation and exploration [16], [18]. We claim that the key problem causing the above issues is the advice adoption scheme. Agents are distributed and observe the environment locally and partially. Even if a teacher agent is highly experienced in its field, its opinion may be useless for an agent facing a totally different scenario. Only agents sharing a common perceptual consensus can provide valuable action suggestions to others, since they are facing similar scenarios. Inspired by this insight and sociological principles of consensus [19], we propose a Consensus-based Communication and Knowledge Sharing (CCKS) framework. In CCKS, local observations are regarded as multiple views of the global state, with consensus representing the universal information extracted from these views. The key contributions include: (1) pioneering consensus learning to assist action advising, enabling faster convergence to shared environmental understanding; (2) adaptive teacher- student role generation for efficient multi-source knowledge transfer; (3) selective advice adoption based on consensus similarity, ensuring integration of relevant knowledge; and (4) a "think twice before you leap" mechanism where agents resample when low-probability actions are suggested, avoiding premature suboptimal decisions. The framework is plug-and-play, compatible with various DTDE algorithms. We validate the proposed CCKS framework using two widely recognized benchmark environments: the StarCraft I Multi-Agent Challenge environment [20] and the Google Research Football environment [21]. Integrating CCKS with Independent Q-Learning (IQL) [10] accelerates learning, improves decision accuracy, and enhances stability. Ablation studies confirm the benefits of consensus learning and the arXiv:2606.12281v1 [cs.MA] 10 Jun 2026 "think twice" mechanism, demonstrating CCKS’s effective- ness and potential for broader MARL applications. I. RELATED WORK A. Distributed execution training framework Distributed execution training frameworks mainly include centralized training distributed execution (CTDE) [5], [6], [7], [8] and distributed training distributed execution (DTDE) [9], [10], [11], [12]. CTDE leverages global information during training but executes with local observations. However, it is impractical when inter-agent observation sharing is difficult (e.g., human-computer tasks [22]) or communication is restricted [23]. DTDE, in contrast, relies solely on local observations throughout, offering scalability and robustness. While this may lead to suboptimal policies due to the absence of global information, its scalability makes it promising for real-world MARL applications. B. Knowledge Sharing framework Knowledge sharing in Multi-Agent Reinforcement Learn- ing (MARL) enhances coordination by enabling agents to exchange information via communication [14], [18], [24] or centralized training [25], [26]. Drawing on Theory of Mind (ToM) [27], agents leverage others’ insights to inform action selection, mirroring human collaboration. We categorize schemes into observation-level and policy-level sharing. Observation-level approaches (e.g., [13], [14]) share local observations or embeddings to enrich perception. Policy-level methods convey higher-level information, such as predicting future intentions [15] or assigning teacher-student roles [25], [18] for novices to follow experts without independent decisions. C. Contrastive Learning Contrastive Learning has become a prominent paradigm in MARL, particularly for enhancing decentralized policies. It helps agents distinguish between similar and dissimilar experiences, which is essential for learning effective coopera- tive strategies. In MARL, contrastive learning aligns agent representations toward a consensus, improving stability and convergence [28]. For instance, [29] introduces a method that uses contrastive learning to decompose the value function while accounting for individual and collective agent identi- ties. Another work [30] presents a framework for learning grounded communication via contrastive objectives. I. BACKGROUND A. Problem Formulation We formalize our cooperative multi-agent reinforce- ment learning as a partially observed Markov game fornagents [31], which can be defined by a tuple (N,S,O,A, Ω,P, R i i∈N ,γ) . Here,N = 1,...,nde- notes the set ofnagents;Sis the state space;O = O 1 × ...×O n denotes the joint observation space;A = A 1 ×...×A n denotes the joint action space. For each agenti∈Nat each time step, it can only obtain its local observationo i ∈ O i from the observation functionΩ(s,i)wheres ∈ S, and then choose the actiona i ∈ A i according to the policy π ω i (a i |o i ) : O i → A i , whereω i parameterizes the policy. The environment changes according to a state transition function P : S ×A → S ′ , which returns a probability distribution of a new stateS ′ based on the current stateSand the joint actionA. Similarly, the agent can obtain new observations o ′ i through the observation functiono ′ i = Ω(s ′ ,i) . After that, each agenti∈Nwill receive its own rewardr i t at the current time steptbased on the reward functionR i :S×A×S ′ →R. The purpose of each agent is to optimize the parameterω i of the policyπ ω i to maximize the expected discounted reward E [ P T t=0 γ t r i t | π ω i ], whereγis the discount factor andTis the total time step. To enhance coordination, we integrate a communication mechanism where each agentican send and receive messages m i ∈Mbased on its local observationo i and actiona i . The communication strategyφ ω c i (m i |o i ,a i ) : O i × A i → M determines the message content. Received messages are used to update each agent’s policyπ ω i , enabling more informed and coordinated actions. This mechanism improves collective performance by allowing agents to share critical information and adjust their strategies accordingly. IV. METHOD In this section, we introduce CCKS’s four stages: Con- sensus Model Training, Observation Request Broadcast, Teacher Strategy Response, and Consensus-based Knowledge Understanding. First, contrastive learning trains a consensus model for shared environmental understanding, reducing complexity. Second, agents broadcast observation requests for timely sharing and information symmetry. Third, agents dynamically assume teacher roles using Q-function and entropy, enhancing flexibility. Finally, agents filter advice via entropy and consensus, ensuring quality knowledge transfer. See Figures 1(a) and 1(b). A. Consensus Model Training In the conventional action advising scheme, the agent often learns only from the individual experiences of more capable agents, rather than learning to achieve better cooperative outcomes, which is the core problem we aim to solve. In CCKS, we introduce a consensus learning modelf θ into the DTDE paradigm. Each agent uses its local observationo i to derive a consensus representationc i of the global state, which supplementso i for decision-making. The consensus inferred by the agents serves as a shared signal, akin to a global state, guiding their actions toward a common goal. The specific formula of the consensus c i is as follows: c i = f θ (o i )(1) We use a three-layer multilayer perceptron (MLP) with ReLU activation to train the consensus model via contrastive learning. The ultimate goal of the consensus model is to ensure that the consensus outputs of all agents align as closely as possible. Environment ...... Agent 1 Agent n Agent i Agent j · · · · · · Judgment Policy Yes Request Broadcast Give up No 퐀 퐀 Knowledge Understanding · · · · · · · · · · · · . . . L2 loss Entropy 퐀 퐀 퐀 ...... 퐀 퐀 퐀 퐀 퐀 퐀 Score Calc Score Teacher Response 퐀 퐀 퐀 퐀 ...... 퐀 퐀 퐀 퐀 퐀 퐀 퐀 퐀 퐀 퐀 Environment 퐀퐀倀퐀 퐀 = (퐀 퐀 퐀 퐀 , 퐀 퐀 퐀 퐀 , ...... , 퐀 퐀 퐀 퐀 , 퐀 퐀 ) 퐀퐀᠀퐀 퐀 =(퐀 퐀 , 퐀 퐀 , max퐀 퐀 퐀 (퐀 퐀 ), 퐀蠀퐀 퐀 퐀 (퐀 퐀 ), 퐀(퐀 퐀 퐀 (퐀 퐀 ))) (a) Environment ...... · · · · · · Decision Environment Contrastive Learning Loss Agent 1 Agent n 퐀 퐀 퐀 퐀 퐀 퐀 퐀 퐀 퐀 퐀 퐀 퐀 퐀 퐀 Consensus Model Agent n’s Decision ...... 퐀 퐀 Policy 퐀 퐀 퐀 퐀 퐀 퐀 퐀 ...... embedding 퐀 퐀 Score . . · · · · · · 퐀 퐀 Judgment Yes No 퐀 퐀 퐀 ...... 퐀 퐀 퐀 퐀 퐀 퐀 퐀 퐀 퐀 Again Output 퐀 퐀 퐀 퐀 ...... Request Broadcast (b) Fig. 1.This figure provides an overview of CCKS. (a) The communication process among agents. Specifically, the blue sections indicate the agents’ Observation Request Broadcast, the red sections signify the Teacher Strategy Response, and the green sections represent the agents’ Shared Knowledge Learning. (b) depicts the interaction process between the agents and the environment. Specifically, the orange sections denote Consensus Model Training, and the black sections correspond to Actions Sampling. B. Observation Request Broadcast In the MARL framework, the Observation Request Broad- cast serves as a pivotal mechanism for facilitating inter- agent communication and knowledge dissemination. This phase is essential for enabling each agent to share its current state representation and associated Q-value estimations with its counterparts, thereby fostering a collaborative learning environment. Each agentiwithin the system computes a consensusc i for its observed stateo i using its consensus model. Once the consensusc i is computed, agentiproceeds to broadcast a request to all other agents. This request comprises the following components: Request i = (o i ,c i , maxQ ω i (o i ), minQ ω i (o i ),H(Q ω i (o i ))) (2) whereH(·)represents the entropy of the Q-values, while maxQ ω i (o i )andminQ ω i (o i )denote the maximum and minimum Q-value estimations for state o i , respectively. C. Teacher Strategy Response In the Teacher Strategy Response phase each agentjwill decide whether to share its knowledge with others to facilitate knowledge transfer and collaboration. Each agentjevaluates the received observations and Q- value estimations from its peers to decide on knowledge sharing. Agentjshares its knowledge with agentionly if both of the following conditions are met: 1)The observationo i is not new to agentj(i.e.,n j o i ≥ 1) and the entropy of agentj’s Q-values foro i is not greater than that of agent i: n j o i ≥ 1 and H(Q ω i (o i ))≥ H(Q ω j (o i ))(3) 2)The Q-value estimation of agentjfor stateo i lies within the interval defined by agenti’s min and max Q-values: minQ ω i (o i )≤ Q ω j (o i )≤ maxQ ω i (o i )(4) If either condition fails, agentjrefrains from sharing knowledge with agenti. Otherwise, it proceeds to share its action probabilitiesp j a l for each actiona l in the action space, computed using its policy network Q ω j : Response j = p j a 1 ,p j a 2 ,...,p j a m ,c j (5) D. Consensus-based Knowledge Understanding 1) Shared Knowledge learning: In the Consensus-based Knowledge Sharing phase, each agentireceives responses from its teachers who denoted asTand computes a guiding coefficientE t for each teachert∈ Tbased on the discrepancy between its own consensusc i and that of teachert, as well as the entropy of the teacher’s Q-values: E t =−∥c i − f θ (o t )∥· H(Q ω t (o i ))(6) Here, the consensus difference∥c i − f θ (o t )∥measures agent-teacher alignment, whileH(Q ω t (o i )) denotes Q-value entropy, reflecting teacher uncertainty. Guiding weightsμ t are then computed via softmax normalization: μ t = e E t P t e E t (7) These weights determine the influence of each teacher on the agenti, ensuring that more reliable and diverse knowledge sources have a greater impact. With these weights, each action’s score S a l is calculated as: S a l = X t μ t · (p t a l )− S mean (8) whereS mean = 1 m P m l=1 P t μ t ·p t a l is the average score across all actions. This scoring mechanism uses the probability of the teachers’ policies to score each action, guiding the agent to adjust its policy toward more optimal actions. 2) Actions sampling: In the conventional action advising scheme, the actions of the student agent are often directly overwritten or modified, which can conflict with its policy, impair learning, and misalign with its mental model. Hence, our scheme avoids directly influencing the agent’s decision- making process. In the decision-making process of the agent with a Q-function, we replace the regular observationo i with (a)(b) Fig. 2. Experiment environments. (a) The StarCraft I Multi-Agent Challenge (SMAC). (b) Google Research Football (GRF). the concatenation of observationo i and consensusc i ; the action of agent i is then selected as follows: a i = Sample Q i (embedding(o i ,c i ),a i ) (9) Meanwhile, the scoring mechanism participates in the process through two methods, namely “Think twice before you leap“ and reward shaping. Given the hyper-parameter K, if an agent selects an action from the set ofKactions with the lowest scores (deemed by teachers as likely to “fall into pitfalls”’), it will perform a secondary sampling, inspired by the human tendency to “think twice”. Additionally, considering that “Think twice before you leap” does not effectively utilize high-scoring actions, we use the scores for reward shaping. Specifically, the original rewardris updated using the hyper-parameter τ (τ ∈ [0, 1]) as follows: r new = r + τ · S a i (10) V. EXPERIMENTS In this section, we evaluate the performance of CCKS as a new DTDE framework by comparing it with baseline algorithms. In our experiments, Independent Q-Learning (IQL) [10] was used as the base algorithm to assess the effectiveness of CCKS. We implement CCKS and compare it with different baselines, including IQL [10], CONS [18], I2Q [12], and AdHocTD [24]. Finally, we study the conditions and components that affect CCKS’s performance through ablation experiments. Our testing environments include the StarCraft Multi-Agent Challenge (SMAC) [20] and Google Research Football (GRF) [21], depicted in Figure 2. SMAC provides real-time strategy micro-management tasks in StarCraft I. GRF simulates football matches requiring coordinated tactical execution. For each environment, we conduct experiments on multiple maps and scenarios. All results are reported as the mean and standard deviation over five random seeds. A. Performance Figure 3 presents the StarCraft I results, showing the mean test win rate over timesteps, with shaded areas in- dicating the error in performance across different seeds. The learning speed and overall performance of the CCKS algorithm have been significantly enhanced. The extent of improvement varies across different scenarios. Notably, during the initial stages of training, CCKS exhibits a substantial performance gap compared to other algorithms such as IQL and I2Q. We attribute this to the consensus mechanism of CCKS, which enables implicit cooperation among agents. In contrast, agents in the IQL algorithm must independently explore the environment, often selecting actions that are more advantageous for individual agents rather than the team. For AdHocTD, excessive imitation of the teacher agent frequently leads to suboptimal outcomes. The I2Q algorithm trains an ideal transition model using global information; however, the complexity of this information results in slower performance improvements. CONS provides richer information for agent decision-making through action suggestions, but the limited knowledge sharing for a single action leads to insufficient learning. Compared with these methods, CCKS establishes consensus under communication conditions and implicitly learns knowledge through a scoring mechanism. This approach enables agents to “think twice” before acting, thereby achieving better overall performance. Figure 4 presents the experimental outcomes from the Google Football environment, illustrating the performance of various algorithms over timesteps, focusing on mean test episode length and mean test score reward. The results indicate that CCKS is the most effective algorithm, achieving both shorter episode lengths and higher score rewards over time, suggesting that it is better at learning efficient and high-quality actions in a multi-agent setting compared to the other algorithms tested, such as AdHocTD, CONS, I2Q, and IQL, which exhibit more instability and less consistent performance improvements. B. Further Analysis 1) Ablations: To assess the efficacy of the CCKS algo- rithm, an ablation study was conducted on the 2s3z map within the SMAC environment, evaluating the contributions of its key components: the consensus model, knowledge sharing, scoring mechanism, and the “think twice” mech- anism. Configurations lacking these features were denoted asCCKS-NO-CB,CCKS-NO-KS,CCKS-NO-SCORE, and CCKS-NO-TWICE. Results, as shown in Figure 5(a) and Figure 5(b), indicate that CCKS achieves the highest mean test win rate, significantly outperforming other configurations as the number of timesteps increases. This suggests that the full CCKS algorithm is most effective in learning strategies lead- ing to higher win rates.CCKS-NO-CBandCCKS-NO-KS show improvements in win rate over time but perform less consistently and slightly lower compared to the full CCKS algorithm.CCKS-NO-SCOREandCCKS-NO-TWICEex- hibit more variability, with CCKS-NO-SCORE showing a particularly noticeable decline in win rate at certain points, indicating that the absence of scoring mechanisms or the “think twice” mechanism may negatively impact learning and overall performance. The ablation studies suggest that each component of the CCKS algorithm contributes to its overall performance, and removing any of these components leads to a decrease in effectiveness. Additionally, the impact of the hyperparameterKon CCKS performance was investigated within the GRF environment. With all other settings constant, performance under variousK values was compared. Figure 5(c) and Figure 5(d) illustrate the 0.000.250.500.751.001.251.501.752.00 Timesteps 1e6 0.0 0.2 0.4 0.6 0.8 1.0 Mean Test Win Rate 8m AdHocTD CCKS CONS I2Q IQL (a) 0.000.250.500.751.001.251.501.752.00 Timesteps 1e6 0.0 0.1 0.2 0.3 0.4 0.5 Mean Test Win Rate 3s_vs_5z AdHocTD CCKS CONS I2Q IQL (b) 0.00.51.01.52.0 Timesteps 1e6 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 Mean Test Win Rate 3s5z AdHocTD CCKS CONS I2Q IQL (c) 0.00.51.01.52.0 Timesteps 1e6 0.0 0.1 0.2 0.3 0.4 Mean Test Win Rate 1c3s5z AdHocTD CCKS CONS I2Q IQL (d) Fig. 3.Experimental results on the StarCraft Multi-Agent Challenge Environment: The mean test win rate of the five algorithms over 5 seeds under a partially observable SMAC task. 012345 Timesteps 1e6 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Mean Test Score Reward academy_3_vs_1_with_keeper AdHocTD CCKS CONS I2Q IQL (a) 012345 Timesteps 1e6 0.0 0.1 0.2 0.3 0.4 Mean Test Score Reward academy_corner AdHocTD CCKS CONS I2Q IQL (b) 012345 Timesteps 1e6 0.0 0.2 0.4 0.6 0.8 Mean Test Score Reward academy_pass_and_shoot_with_keeper AdHocTD CCKS CONS I2Q IQL (c) 012345 Timesteps 1e6 0.00 0.05 0.10 0.15 0.20 Mean Test Score Reward academy_counterattack_hard AdHocTD CCKS CONS I2Q IQL (d) Fig. 4.Experimental results on the Google Research Football Environment: The mean test score reward of the five algorithms over 5 seeds under a partially observable GRF task. 0.00.51.01.52.0 Timesteps 1e6 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Mean Test Score Reward 2s3z CCKS CCKS-NO-CB CCKS-NO-KS CCKS-NO-SCORE CCKS-NO-TWICE (a) CKS CKS-NO-CB CKS-NO-KS CKS-NO-SCORE CKS-NO-TWICE 0.0 0.2 0.4 0.6 0.8 Mean Test Win Rate 2s3z (b) 012345 Timesteps 1e6 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Mean Test Score Reward academy_run_pass_and_shoot_with_keeper K=1 K=4 K=8 K=13 K=15 (c) K=1K=4K=8K=13K=15 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 Mean Test Score Reward academy_run_pass_and_shoot_with_keeper (d) Fig. 5.Ablation experimental results on SMAC and GRF showing the influence of the components and K of CCKS. training curves and execution outcomes. In the GRF scenario, smallerKvalues result in slower learning speeds and significant variance. As K increases, performance improves significantly compared to smallerKvalues. However, asK continues to increase, despite faster initial training speeds, final performance declines. This suggests that an appropriate increase inKcan enhance learning speed, but an excessively largeKcan decrease exploratory capabilities, leading to local optima. Selecting an appropriateKvalue can significantly enhance CCKS’s experimental outcomes. 2) Limitations and future work: A limitation of CCKS is the added computational overhead from the consensus model during training—a trade-off for improved learning efficiency. This overhead must be considered in resource-limited or real-time settings. Moreover, since knowledge sharing relies on probabilities over all discrete actions, expanding the action space significantly increases communication overhead, slowing training. This is especially pertinent as real-world tasks often have larger action spaces than those in GRF or SMAC. Future work should also explore the further alignment of the relationships between teacher and student agents, as well as the design of reward mechanisms. Employing interpretable learning processes can foster policy models closer to human cognition, improving AI interactions. VI. CONCLUSIONS In this paper, we propose the CCKS framework as a plug- and-play component to enhance the capabilities of existing cooperative MARL algorithms in order to address the lack of cooperation in the DTDE paradigm. CCKS introduces consensus learning, which takes each agent’s local observation as input and outputs a consensus representation as auxiliary information. A knowledge sharing process enables agents to obtain guidance from others, where student agents compute weights for teachers and score actions based on information entropy and L2 loss. Using the resulting score table, the “Think twice before you leap” mechanism and reward shaping refine agents’ actions and rewards. Extensive experiments on SMAC and GRF demonstrate that CCKS significantly improves the performance of baseline cooperative MARL algorithms. REFERENCES [1]T. Fan, P. Long, W. Liu, and J. Pan, “Distributed multi-robot collision avoidance via deep reinforcement learning for navigation in complex scenarios,” The International Journal of Robotics Research, vol. 39, no. 7, p. 856–892, 2020. [2]L. M. Schmidt, J. Brosig, A. Plinge, B. M. Eskofier, and C. Mutschler, “An introduction to multi-agent reinforcement learning and review of its application to autonomous mobility,” in 2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC). IEEE, 2022, p. 1342–1349. [3]D. Ye, Z. Liu, M. Sun, B. Shi, P. Zhao, H. Wu, H. Yu, S. Yang, X. Wu, Q. Guo, et al., “Mastering complex control in moba games with deep reinforcement learning,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 04, 2020, p. 6672–6679. [4]F. A. Oliehoek, “Decentralized pomdps,” in Reinforcement learning: state-of-the-art. Springer, 2012, p. 471–503. [5]R. Lowe, Y. I. Wu, A. Tamar, J. Harb, O. Pieter Abbeel, and I. Mordatch, “Multi-agent actor-critic for mixed cooperative-competitive environments,” Advances in neural information processing systems, vol. 30, 2017. [6]T. Rashid, M. Samvelyan, C. S. De Witt, G. Farquhar, J. Foerster, and S. Whiteson, “Monotonic value function factorisation for deep multi- agent reinforcement learning,” Journal of Machine Learning Research, vol. 21, no. 178, p. 1–51, 2020. [7]S. Iqbal and F. Sha, “Actor-attention-critic for multi-agent reinforcement learning,” in International conference on machine learning. PMLR, 2019, p. 2961–2970. [8]I.-J. Liu, R. A. Yeh, and A. G. Schwing, “Pic: permutation invariant critic for multi-agent deep reinforcement learning,” in Conference on Robot Learning. PMLR, 2020, p. 590–602. [9]M. Tan, “Multi-agent reinforcement learning: Independent vs. coopera- tive agents,” in Proceedings of the tenth international conference on machine learning, 1993, p. 330–337. [10]A. Tampuu, T. Matiisen, D. Kodelja, I. Kuzovkin, K. Korjus, J. Aru, J. Aru, and R. Vicente, “Multiagent cooperation and competition with deep reinforcement learning,” PloS one, vol. 12, no. 4, p. e0172395, 2017. [11]B. Peng, T. Rashid, C. Schroeder de Witt, P.-A. Kamienny, P. Torr, W. Böhmer, and S. Whiteson, “Facmac: Factored multi-agent centralised policy gradients,” Advances in Neural Information Processing Systems, vol. 34, p. 12 208–12 221, 2021. [12]J. Jiang and Z. Lu, “I2q: A fully decentralized q-learning algorithm,” Advances in Neural Information Processing Systems, vol. 35, p. 20 469– 20 481, 2022. [13]S. Sukhbaatar, R. Fergus, et al., “Learning multiagent communication with backpropagation,” Advances in neural information processing systems, vol. 29, 2016. [14]Z. Ding, T. Huang, and Z. Lu, “Learning individually inferred communication for multi-agent cooperation,” Advances in neural information processing systems, vol. 33, p. 22 069–22 079, 2020. [15]W. Kim, J. Park, and Y. Sung, “Communication in multi-agent reinforcement learning: Intention sharing,” in International conference on learning representations, 2020. [16]C. Zhu, H.-F. Leung, S. Hu, and Y. Cai, “A q-values sharing framework for multi-agent reinforcement learning under budget constraint,” ACM Transactions on Autonomous and Adaptive Systems (TAAS), vol. 15, no. 2, p. 1–28, 2021. [17]Y. Guo, J. Campbell, S. Stepputtis, R. Li, D. Hughes, F. Fang, and K. Sycara, “Explainable action advising for multi-agent reinforcement learning,” in 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023, p. 5515–5521. [18]Y. Ba, X. Liu, X. Chen, H. Wang, Y. Xu, K. Li, and S. Zhang, “Cautiously-optimistic knowledge sharing for cooperative multi-agent reinforcement learning,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 16, 2024, p. 17 299–17 307. [19]S. Snyman and J. Rogers, “Consensus-based partnerships: the heart of effective interprofessional education and collaborative practice,” Sustainability and interprofessional collaboration: Ensuring leadership resilience in collaborative health care, p. 59–82, 2020. [20]M. Samvelyan, T. Rashid, C. S. De Witt, G. Farquhar, N. Nardelli, T. G. Rudner, C.-M. Hung, P. H. Torr, J. Foerster, and S. Whiteson, “The starcraft multi-agent challenge,” arXiv preprint arXiv:1902.04043, 2019. [21]K. Kurach, A. Raichuk, P. Sta ́ nczyk, M. Zaj ̨ac, O. Bachem, L. Espeholt, C. Riquelme, D. Vincent, M. Michalski, O. Bousquet, et al., “Google research football: A novel reinforcement learning environment,” in Proceedings of the AAAI conference on artificial intelligence, vol. 34, no. 04, 2020, p. 4501–4510. [22]C.-M. Huang, M. Cakmak, and B. Mutlu, “Adaptive coordination strategies for human-robot handovers.” in Robotics: science and systems, vol. 11. Rome, Italy, 2015, p. 1–10. [23]W. Ying and S. Dayong, “Multi-agent framework for third party logistics in e-commerce,” Expert Systems with Applications, vol. 29, no. 2, p. 431–436, 2005. [24]F. L. Da Silva, R. Glatt, and A. H. R. Costa, “Simultaneously learning and advising in multiagent reinforcement learning,” in Proceedings of the 16th conference on autonomous agents and multiagent systems, 2017, p. 1100–1108. [25]D.-K. Kim, M. Liu, S. Omidshafiei, S. Lopez-Cot, M. Riemer, G. Habibi, G. Tesauro, S. Mourad, M. Campbell, and J. P. How, “Learning hierarchical teaching policies for cooperative agents,” arXiv preprint arXiv:1903.03216, 2019. [26]N. Gupta, G. Srinivasaraghavan, S. Mohalik, N. Kumar, and M. E. Taylor, “Hammer: Multi-level coordination of reinforcement learning agents via learned messaging,” Neural Computing and Applications, p. 1–16, 2023. [27]M. Tomasello, M. Carpenter, J. Call, T. Behne, and H. Moll, “Un- derstanding and sharing intentions: The origins of cultural cognition,” Behavioral and brain sciences, vol. 28, no. 5, p. 675–691, 2005. [28]Z. Xu, B. Zhang, D. Li, Z. Zhang, G. Zhou, H. Chen, and G. Fan, “Consensus learning for cooperative multi-agent reinforcement learning,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 10, 2023, p. 11 726–11 734. [29]S. Liu, Y. Zhou, J. Song, T. Zheng, K. Chen, T. Zhu, Z. Feng, and M. Song, “Contrastive identity-aware learning for multi-agent value decomposition,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 10, 2023, p. 11 595–11 603. [30]Y. L. Lo and B. Sengupta, “Learning to ground decentralized multi-agent communication with contrastive learning,” arXiv preprint arXiv:2203.03344, 2022. [31]M. L. Littman, “Markov games as a framework for multi-agent rein- forcement learning,” in Machine learning proceedings 1994. Elsevier, 1994, p. 157–163.