Paper deep dive
Adaptive Punishment for Cooperation in Mixed-Motive Games
Min Tang, Fanqi Kong, Linyuan Lü, Xue Feng
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 7/8/2026, 7:45:21 PM
Summary
This paper introduces Adaptive Punishment for Cooperation (APC), a distributed multi-agent reinforcement learning method designed to promote cooperation in mixed-motive games. APC dynamically adjusts punishment probability and intensity based on defection severity and historical effectiveness, utilizing a defection awareness module to identify and assess defection. Theoretical and empirical analyses demonstrate that APC effectively balances punishment costs and efficacy, significantly outperforming existing baselines across iterated public goods games and sequential social dilemmas by strategically deterring defection and fostering long-term collective welfare.
Entities (10)
Relation Signals (11)
Adaptive Punishment for Cooperation (APC) → evaluatedon → Iterated Public Goods Game
confidence 95% · Theoretical analysis and empirical results show APC performs effectively in iterated public goods game.
Adaptive Punishment for Cooperation (APC) → evaluatedon → Sequential Social Dilemmas
confidence 95% · Empirically, APC also significantly outperforms existing baselines across sequential social dilemmas...
Adaptive Punishment for Cooperation (APC) → uses → Defection Awareness Module
confidence 95% · APC consists of two core modules: defection awareness and adaptive punishment.
Multi-Agent Reinforcement Learning → addresses → Mixed-Motive Games
confidence 90% · Multi-Agent Reinforcement Learning (MARL) allows agents to interact in shared environments, yet mixed-motive scenarios pose unique challenges...
Sequential Social Dilemmas → includes → Sequential Stag-Hunt
confidence 90% · Sequential Stag-Hunt (SSH) [20] (8×8 map, 4 agents) requires coordination...
Sequential Social Dilemmas → includes → Foraging
confidence 90% · Finally, Foraging [12] (10×10 map, 12 agents) involves common agents collecting berries...
Sequential Social Dilemmas → includes → Coin Game
confidence 90% · We evaluate APC on four spatiotemporal-extended SSDs mixed-motive games. The Coin Game...
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Mixed-motive scenarios are ubiquitous in real-world multi-agent interactions, where self-interested agents often defect for immediate rewards, overlooking the potential of altruistic cooperation to improve long-term gains and collective welfare. Peer punishment can deter defection, but as costly second-order altruism, its persistent imposition may undermine the punisher's interests. Existing approaches often struggle to effectively implement punishment to promote cooperation. To balance the efficacy and cost of punishment, we propose Adaptive Punishment for Cooperation (APC), a distributed method that determines punishment intensity based on both a dynamic punishment probability and the severity of defection. This dynamic probability substantially reduces costly and ineffective punishment while also promotes cooperation. To accurately assess defection and its severity, we use a defection awareness module, whose learning is guided by game reward. Theoretical analysis and empirical results show APC performs effectively in iterated public goods game. Empirically, APC also significantly outperforms existing baselines across sequential social dilemmas, learning rational and effective punishment policies that foster cooperation by strategically deterring defection.
Tags
Links
- Source: https://arxiv.org/abs/2605.24516v1
- Canonical: https://arxiv.org/abs/2605.24516v1
Trouble viewing inline? Open PDF directly →
Full Text
37,608 characters extracted from source content.
Expand or collapse full text
Adaptive Punishment for Cooperation in Mixed-Motive Games Min Tang1,2, Fanqi Kong2, Linyuan Lü1,†, Xue Feng2,†, 1University of Science and Technology of China, Hefei, China 2State Key Laboratory of General Artificial Intelligence, BIGAI, Beijing, China tangmin8554@mail.ustc.edu.cn, linyuan.lv@ustc.edu.cn Abstract Mixed-motive scenarios are ubiquitous in real-world multi-agent interactions, where self-interested agents often defect for immediate rewards, overlooking the potential of altruistic cooperation to improve long-term gains and collective welfare. Peer punishment can deter defection, but as costly second-order altruism, its persistent imposition may undermine the punisher’s interests. Existing approaches often struggle to effectively implement punishment to promote cooperation. To balance the efficacy and cost of punishment, we propose Adaptive Punishment for Cooperation (APC), a distributed method that determines punishment intensity based on both a dynamic punishment probability and the severity of defection. This dynamic probability substantially reduces costly and ineffective punishment while also promotes cooperation. To accurately assess defection and its severity, we use a defection awareness module, whose learning is guided by game reward. Theoretical analysis and empirical results show APC performs effectively in iterated public goods game. Empirically, APC also significantly outperforms existing baselines across sequential social dilemmas, learning rational and effective punishment policies that foster cooperation by strategically deterring defection. 22footnotetext: Corresponding author I Introduction Multi-Agent Reinforcement Learning (MARL) allows agents to interact in shared environments, yet mixed-motive scenarios [13, 8] pose unique challenges compared to purely cooperative or competitive tasks [16]. In these settings, dynamic agent relationships often lead to social dilemmas where short-term individual gains supersede long-term collective welfare. Traditional training paradigms often fail to resolve these issues: Centralized Training with Decentralized Execution (CTDE) [24] relies on unrealistic assumptions of joint optimization among self-interested agents, while Decentralized Training (DTDE) frequently converges to suboptimal local equilibria (see Fig. 2(a)). Consequently, there is a critical need for methods that guide decentralized agents out of these dilemmas toward socially beneficial outcomes. To promote cooperation in mixed-motive games, it is crucial to both deter defection and incentivize collective welfare. Inspired by human societies, punishment raises the cost of defection, discouraging short-term selfish behavior and guiding agents toward long-term group benefits [6]. However, peer punishment imposes a penalty on others while incurring a cost for the punisher (e.g., time and effort spent stopping someone from smoking in public). That is, punishment is second-order altruism. Excessive punishment may harm the punisher’s own interest, and agents may avoid punishing due to its cost, failing to achieve cooperation [12]. To address these challenges, we propose a distributed decision-making method named APC (Adaptive Punishment for Cooperation), which is context-sensitive and adaptively adjusts the probability and severity of punishment. Guided by reward signals, APC learns a defection awareness module to evaluate the defectiveness of others’ actions, which determines the probability and intensity of punishment. Notably, to prevent ineffective punishment and avoidable cost, punishment probability is dynamically adjusted according to the historical effectiveness of punishment, based on the reduction in defections. The main contributions of this work are: (1) We propose a novel opponent-adaptive punishment method, APC, that dynamically adjusts punishment intensity to avoid ineffective punishment and excessive costs, thereby promoting cooperation in mixed-motive games. (2) We develop a self-learning defection awareness algorithm for detecting different degrees of defection in multi-agent interactions. (3) We provide theoretical analysis and demonstrate empirically that APC outperforms existing baselines across Iterated Public Goods Game and Sequential Social Dilemmas (SSDs). I Related Work Multi-agent reinforcement learning has been widely applied to address problems related to social dilemmas. Various reward mechanisms, including centralized redistribution [5] and decentralized peer-to-peer rewarding [11], have been proposed to promote cooperation by aligning individual and collective interests in mixed-motive games. Punishment, which promote cooperation without requiring additional reward resources, have gained increasing attention. In evolutionary game theory, punishment is widely regarded as a key factor in achieving stable cooperation [23, 7]. Some approaches [15, 25] rely on centralized controllers or self-organized structures among individuals to implement punishment, further investigating how punishment influence the evolution and stability of cooperative behaviors. Other methods [3, 22, 18] adopt conditional punishment policy, where the decision to punish depends on conditions such as the difference between an individual’s own payoff and the average payoff of its neighbors, thereby improving the specificity and effectiveness of punishment. In MARL, although some studies have attempted to incorporate punishment method into multi-agent systems, some designs remain relatively simple—typically adding a punishment action into the action space [13, 9, 19]. However, experimental results, shown as in Figure 3, have shown that merely relying on such punishment action combined with standard Independent MARL methods often fails to promote cooperation in SSDs. Other approaches [26, 1] depend on auxiliary mechanisms or specific environmental assumptions to support the effectiveness of punishment, such as introducing reputation systems or assuming that punishment behaviors can generate positive incentives. Some methods further assume that punishment can directly yield positive rewards for the punisher [21, 27], which is clearly unrealistic. We propose APC, which does not rely on environmental assumptions but instead identifies defection behaviors through a defection predictor network and applies targeted punishment. This guides agents to learn reasonable and effective punishment policy, thereby promoting cooperation more effectively. I Preliminaries An N-player Partially Observable Markov Game (POMG) is defined as ℳ=⟨N,,i,i,T,Ri⟩M= N,S,\O^i\,\A^i\,T,\R^i\ . Here, S is the state space, while iO^i and iA^i denote the observation and action spaces for agent i, respectively. The state transition function T:×1×⋯×N×→[0,1]T:S×A^1×…×A^N×S→[0,1] defines the probability distribution over S, representing the probability of transitioning to state s′s given the current state s and joint action a→=(a1,…,aN) a=(a^1,…,a^N). Each agent i follows a policy πi(ai|oi)π^i(a^i|o^i) to maximize its expected discounted return: Viπi(s0)=a→t∼π,st+1∼T[∑t=0∞γtRi(st,a→t)],V_i^π^i(s_0)=E_ a_t π,s_t+1 T [ _t=0^∞γ^tR^i(s_t, a_t) ], (1) where γ∈[0,1)γ∈[0,1) is the discount factor. In our decentralized MARL framework, agents are trained using the Advantage Actor-Critic (A2C) method. The policy πiπ^i is parameterized by θiθ^i, and the critic evaluates the actor via the TD-error: δti=Ri(st,a→t)+γVπθi(o→t+1)−Vπθi(o→t) _t^i=R^i(s_t, a_t)+γ V _θ^i( o_t+1)-V _θ^i( o_t), where o→t o_t represents the joint observation at timestep t. IV Methodology Figure 1: Overview of the APC Framework. The framework consists of two core components: Defection Awareness and Adaptive Punishment. Any Agent i interacts with the environment to collect trajectories, which are stored in a replay buffer to train the defection predictor network μiμ^i. μiμ^i is trained using oi,a−jo^i,a^-j, and rir^i to predict the probability distribution σij=pμi(⋅|oi,a−j)σ^ij=p_μ^i(·|o^i,a^-j) over jA^j. A higher-probability action aja^j indicates a greater degree of defection. Adaptive punishment computes the probability pijp^ij (Eq.(3)) based on the defection frequency fijf^ij over the past L timesteps. The punishment intensity weight wijw^ij is then determined: if σij(aj)>1|j|σ^ij(a^j)> 1|A^j|, wijw^ij is set proportional to σij(aj)σ^ij(a^j) with probability pijp^ij; otherwise, wij=0w^ij=0. Finally, agent i’s total reward for policy update is ri,tot=ri−∑j=1,j≠iNwijc−∑j=1,j≠iNwjiδr^i,tot=r^i- _j=1,j≠ i^Nw^ijc- _j=1,j≠ i^Nw^jiδ, where c and δ are unit cost and penalty. To address the dilemma of cooperation and defection in multi-agent systems, we propose a distributed MARL method, named Adaptive Punishment for Cooperation (APC). As shown in Fig. 1, APC consists of two core modules: defection awareness and adaptive punishment. The defection awareness module takes agents’ local observation trajectories as input and produces a probability distribution over the opponent’s actions, where actions with higher probabilities are regarded as more severe defections. The adaptive punishment module then uses both the defection judgment and the variation in defection frequency as input to generate a dynamic punishment intensity—that is, the more frequent the defections, the harsher the punishment. Moreover, it adjusts the punishment probability based on the effectiveness of punishment in reducing defections. Next, we take agent i as the focal agent. IV-A Defection Awareness We propose a defection awareness method to train the Defection Predictor Network μiμ^i to identify defection, where the network learns to predict the behavior of a target agent j that would harm agent i’s reward rtir^i_t more. μiμ^i outputs probability distribution of defection σtij=pμi(⋅|oti,at−j)σ^ij_t=p_μ^i(·|o_t^i,a_t^-j) over the agent j’s action space jA^j, given agent i’s observation otio_t^i and the joint actions of all other non-target agents at−ja_t^-j. The dimensionality of at−ja_t^-j is fixed, and actions of agents outside of the observation range are represented using a default placeholder value of −1-1. If σtij(atj)>1|j|σ^ij_t(a_t^j)> 1|A^j|, atja_t^j is considered a defection by agent i. The training objective of μiμ^i is formalized as a maximization problem: J(μi)=atj∼pμi[−rti(st,a→t)+βH(σtij)].J(μ^i)=E_a^j_t p_μ^i [-r^i_t(s_t, a_t)+β H(σ^ij_t) ]. (2) Here, −rti(st,a→t)-r^i_t(s_t, a_t) drives the network to minimize the reward, and βH(σtij)β H(σ^ij_t) incorporates entropy regularization to encourage behavioral diversity and prevent premature convergence to a peaked distribution. H(σtij)=atj∼σtij[−logσtij(atj)]H(σ^ij_t)=E_a^j_t σ^ij_t [- σ^ij_t(a^j_t) ] denotes the entropy of the distribution σtijσ^ij_t, which measures its randomness: a larger entropy indicates a more uniform distribution. The regularization parameter β the entropy’s importance in the optimization objective. The objective function J(μi)J(μ^i) is designed to guide agent i toward learning to identify defection with sufficient exploration. Through this process, μiμ^i effectively identifies the behaviors of target agent j that would more harm agent i’s reward, given different otio^i_t and at−ja_t^-j. Once μiμ^i has converged, its parameters are fixed during the training of policy network θiθ^i, ensuring stable defection awareness. This learned policy serves as a criterion for identifying defection actions in an opponent-aware manner. The entire training is performed via gradient descent using multi-agent trajectory data. IV-B Adaptive Punishment We introduce an adaptive punishment mechanism to guide agents toward rational and graded punitive behavior. Adaptivity is reflected in two aspects: (1) punishment probability is dynamically adjusted according to whether past punishments have successfully reduced defections; and (2) the punishment intensity is modulated based on the degree of defection. The punishment probability ptijp_t^ij denotes the probability with which agent i punishes agent j upon defection. ptijp_t^ij is dynamically adjusted to prevent ineffective punishment, in which past punishments have failed to reduce defections. It is evaluated by the change of defection frequency of agent j from the perspective of agent i, denoted as ftijf_t^ij, where defection is determined by μi _i. Specifically, ftij,sf_t^ij,s denotes the value of ftijf_t^ij in window s, where each window spans L timesteps. If ftij,sf_t^ij,s does not decrease compared to ftij,s−1f_t^ij,s-1, the punishment in window s is deemed ineffective. Consequently, ptijp_t^ij should be reduced in window s+1s+1. We compute ptijp_t^ij in window m as follows: ptij=1−1m−1∑s=1m−1[s],p_t^ij=1- 1m-1 _s=1^m-11 [C_s ], (3) where sC_s represents the condition for the punishment being considered ineffective in window s, defined as: s=((ftij,s≥ftij,s−1∨|ftij,s−f¯tij,s|<ε)∧ftij,s≥ε),C_s= ( (f_t^ij,s≥ f_t^ij,s-1 |f_t^ij,s- f_t^ij,s |< ) f_t^ij,s≥ ), (4) where f¯tij,s=1s∑k=1sftij,s−k f_t^ij,s= 1s _k=1^sf_t^ij,s-k is the average defection frequency over the past s windows. [⋅]1[·] is the indicator function that outputs 1 if the condition inside is true, indicating that the punishment in window s is considered ineffective, and 0 otherwise and ε is a tolerance threshold. The condition ftij,s≥ftij,s−1f_t^ij,s≥ f_t^ij,s-1 indicates that the punishment in window s failed to reduce ftijf_t^ij. The term |ftij,s−1s∑k=1sftij,s−k|<ε |f_t^ij,s- 1s _k=1^sf_t^ij,s-k |< suggests that ftijf_t^ij in window s has not significantly decreased compared to the average defection frequency over the past s−1s-1 windows. The constraint ftij,s≥εf_t^ij,s≥ avoids misinterpreting a low defection frequency as signs of ineffectiveness. When ftij,sf_t^ij,s is very small, the punishment in window s is still effective because it helps maintain cooperation. These suggest that agent j is unlikely to change its behavior in response to punishment, and continuing to apply punishment in such cases would lead to unnecessary costs and a decrease in rti,totr_t^i,tot. Specifically, punishment is considered effective by default in window 0 and 11. Accordingly, ptijp_t^ij is initialized to 1 in window 0 and 11, and is subsequently updated after window 11 based on Eq.(3). Any agent i holds a Punishment Intensity Weight (PIW) vector at time step t: wti=[wtij]j=1Nw_t^i= [w_t^ij ]_j=1^N, where wtij∈[0,1]w_t^ij∈[0,1] is the fraction of agent i’s punishment to agent j. Let BtijB_t^ij be a Bernoulli random variable with success probability ptijp_t^ij, i.e., Btij∼Bernoulli(ptij)B_t^ij (p_t^ij). wtijw_t^ij is computed as follows: wtij=Btij⋅0,σtij(atj)≤1|j|,σtij(atj)maxa∈jσtij(a),σtij(atj)>1|j|.w_t^ij=B_t^ij\;·\; cases0,& _t^ij(a_t^j)≤ 1|A^j|,\\ _t^ij(a_t^j) _a ^j _t^ij(a),& _t^ij(a_t^j)> 1|A^j|. cases (5) If agent i punish agent j, agent i pays wtijcw_t^ijc and agent j loses wtijδw_t^ijδ. In partially observable environments, agent i can only punish j if j is within its observation range; otherwise, punishment is not applied. Agent i’s total reward is rti,tot=rti−∑j=1,j≠iNwtijc−∑j=1,j≠iNwtjiδr_t^i,tot=r_t^i- _j=1,j≠ i^Nw_t^ijc- _j=1,j≠ i^Nw_t^jiδ, where rtir_t^i is the original reward from environment without punishment. Policy πθi(ati|oti)π^i_θ^i(a^i_t|o^i_t) is trained to maximize πi[∑t=0Hγtrti,tot]E_π^i\! [ _t=0^Hγ^tr^i,tot_t ] using the TD-error, where H denotes the length of an episode. (a) Learning Curves (b) Parameter Analysis Figure 2: (a) Learning curves for IPGG of self-play training. (b) Cooperation rate of APC after convergence across different combinations of c and δ in IPGG. The X-axis and Y-axis represent the values of δ and c, respectively, where δ,c∈[0:0.1:1.4]δ,c∈[0:0.1:1.4]. (a) Coingame (b) SSG (c) SSH (d) Foraging Figure 3: Learning curves for four types of SSDs of self-play training. The curves represent the collective environment rewards, with all data collected from five training runs using different random seeds. The solid lines indicate the mean across runs, and the shaded areas represent the standard deviation. Note that all experimental results in this paper are evaluated over five random seeds, and this information will be omitted in the subsequent figure captions for brevity. (a) IPGG-APr (b) MIPGG-AIn (c) SSH-APr (d) MSSH-AIn Figure 4: Adaptive punishment capability of APC. (a,c) APC adjusts punishment frequency: high for defection, low for cooperation, and decreases when punishment proves ineffective. (b,d) Learning curves of the action probability distribution of DPN in MIPGG and MSSH. APC scales Adaptive Intensity (AIn): stronger for severer defections, achieving proportional response. IV-C Analysis in Iterated Public Goods Game In Iterated Public Goods Game (IPGG), n agents each decide whether to contribute an endowment e to a common pool, which is multiplied by r and split equally. Proposition 1. APC agents converge to mutual cooperation in Iterated Public Goods Game. Proof. Let b the expected punishment cost. Given the probability of others cooperating pcp_c, the expected rewards for contribution (cooperation, rcir^i_c) and no contribution (defection, rdir^i_d) are formulated as: rci r^i_c =pc(n−1)ern+ern−e−b =p_c (n-1)ern+ ern-e-b (6) rdi r^i_d =pc(n−1)ern−(n−1)δ−b =p_c (n-1)ern-(n-1)δ-b (7) The contribution probability θiθ^i is updated via gradient ascent: θi θ^i =θi+α∇θiVi(θ) =θ^i+α _θ^iV^i(θ) (8) ≈θi+α1−γ⋅∇θi[θi⋅rci+(1−θi)⋅rdi] ≈θ^i+ α1-γ· _θ^i [θ^i· r^i_c+(1-θ^i)· r^i_d ] =θi+α1−γ(ern−e+(n−1)δ) =θ^i+ α1-γ ( ern-e+(n-1)δ ) The parameters are typically set to [n,e,r]=[5,1,3][n,e,r]=[5,1,3], and the gradient becomes positive when δ>0.1δ>0.1. As shown in Fig. 2(a), experiments on the IPGG confirm that APC effectively promotes cooperation. Sensitivity analysis of punishment parameters (see Fig. 2(b)) reveals that full cooperation is achieved when δ≥0.3δ≥ 0.3 for c∈[0,1.4]c∈[0,1.4], and when δ=0.2δ=0.2 for c∈[0,0.4]c∈[0,0.4]. Experimental results are highly consistent with our theoretical derivations. Specifically, the failure of cooperation at δ=0.2δ=0.2 for c∈[0.5,1.4]c∈[0.5,1.4] can be attributed to the fact that a small δ exerts insufficient deterrence, while a high cost c simultaneously hinders the learning of cooperative policies. However, as δ increases, this negative impact of c is effectively diminished. V Experiment V-A Environments We evaluate APC on four spatiotemporal-extended SSDs mixed-motive games. The Coin Game [14] (5×55× 5 map, 2 agents) is a spatial extension of the Iterated Prisoner’s Dilemma: agents receive 1 point for any coin but penalize partners by 2 points when collecting the opponent’s color. In the Sequential Snowdrift Game (SSG) (8×88× 8 map, 4 agents), clearing a snow pile grants 6 points to all agents while the clearing agent incurs a cost of 4, incentivizing free-riding. Sequential Stag-Hunt (SSH) [20] (8×88× 8 map, 4 agents) requires coordination, as hunting a stag yields 10 shared points but requires at least two participants, whereas hunting a hare is a low-reward (1 point) individual task. Finally, Foraging [12] (10×1010× 10 map, 12 agents) involves common agents collecting berries (3 points) and special agents who can harvest forbidden berries (4 points); the latter triggers permanent resource degradation, reducing common berry values to 1 point. To deter defection, APC introduces punishment parameters [δ,c][δ,c], representing the fine imposed and the cost incurred by the punisher. We set c=δc=δ, choosing δ to offset potential gains from defection. Parameter configurations [δ,c][δ,c] are: [0.7,0.7][0.7,0.7] for IPGG, [1.1,1.1][1.1,1.1] for Coin Game, [2.1,2.1][2.1,2.1] for SSG, [0.4,0.4][0.4,0.4] for SSH, and [0.7,0.7][0.7,0.7] for Foraging. V-B Implementations We adopt a Decentralized Training Decentralized Execution (DTDE) architecture with independently parameterized networks. The defection predictor consists of two convolutional layers for encoding, an LSTM for temporal modeling, and ReLU-activated fully connected layers, processing multi-channel binary tensors. The policy network follows an actor-critic framework, where both components share an architecture similar to the predictor. The actor’s output dimension matches the predictor’s, while the critic produces a single scalar value. V-C Baselines Independent Advantage Actor-Critic (IA2C) is a classic gradient-based reinforcement learning algorithm suitable for agents to learn policies under completely independent conditions [17]. LOLA considers the learning process of other agents when updating its own policy parameters [2]. SI achieves coordination by rewarding agents for having causal influence over other agents’ actions [10]. D3C guides self-interested agents toward collectively efficient cooperative equilibria by having them mix rewards and follow the gradient of an efficiency bound during learning [4]. To enable punitive capability, all above methods are equipped with a punitive action within their action spaces. We additionally introduce three punishment-based methods for comparison. LPOLA simultaneously predicts environmental actions and determines which actions of other agents to penalize, applying penalties when predictions match reality [21]. CNM fosters cooperation by establishing social norms, whose punishments are socially enforced based on group consensus [27]. The core idea of both LPOLA and CNM methods is to encourage punishment by attaching pseudo-rewards. RL Punish utilizes two distinct networks: a policy network for making primary action decisions and a punishment network for deciding which agents to penalize. Both networks are trained via the Advantage Actor-Critic (A2C) algorithm with the objective of maximizing the collective environmental reward. TABLE I: Collective Environmental Reward of Ablation Study Collective Env Reward Coingame SSG SSH Foraging APC 24.628 119.618 18.96 439.283 APC w/o DPN 0.010 31.804 0.014 60.645 APC w/o APr 24.617 119.458 18.959 438.760 V-D Main Results In Coingame, APC most rapidly guides agents to avoid collecting coins of other agents’ colors, significantly improving collective rewards (Fig. 3(a)), whereas D3C achieves this at a relatively slower pace. Although LOLA and SI learn to some extent to prevent agents from picking other agents’ coins, both fail to completely avoid such behavior. For CNM, both agents eventually converge on punishing all actions, which leads to solely optimizing pseudo-rewards and results in very low total return. Consequently, none of the agents successfully learn to collect coins. In addition, agents of other baselines fail to learn an effective punishment strategy and fail to realize that picking coins of other agent’s color would lead to punishment. As a result, they continue to selfishly collect coins. In SSG, APC agents learn to clear all snow piles faster than D3C, SI, and LOLA, achieving near-optimal collective rewards (Fig. 3(b)). However, LPOLA, due to interference from pseudo-rewards associated with punitive actions, neglects the pursuit of actual environmental rewards, falling below IA2C. Other baselines remain trapped in the snowdrift dilemma by expecting others to clear snow while free-riding, leaving numerous piles uncleared. In SSH, agents in APC, concerned about being punished by other agents for hunting hares, ultimately choose to cooperate in hunting stags, thereby maximizing collective rewards (Fig. 3(c)). D3C gets it into the lazy problem [24], and early hunting leads to moving out of the environment and failing to obtain the group rewards of others’ later hunting. Thus, D3C may not hunt until the last few steps, and likely miss the the opportunity of cooperating to hunt stags. Due to pseudo-rewards, LPOLA also neglects the pursuit of actual environmental rewards, resulting in performance inferior to IA2C. In other baseline methods, agents, seeing only short-term personal rewards, choose to hunt hares, failing to engage in group cooperation. In Foraging, APC successfully prevents special agents from collecting forbidden berries, thereby enhancing the subsequent collective returns and achieving optimal performance (as shown in Fig. 3(d)). In contrast, although D3C performed well in IPGG and the first three SSDs, its performance in Foraging was even worse than that of IA2C. Meanwhile, LPOLA was still affected by pseudo-rewards, which led to its performance being inferior to IA2C. Other baseline methods resulted in common agents consistently collecting common berries while special agents persistently pursued forbidden berries, with none learning to prevent the special agents from collecting the forbidden ones. This failure led to a significant gap between the overall reward and the optimal outcome. V-E Ablation Study To further evaluate the performance of Defection Predictor Network (DPN) and Adaptive Probability (APr) (Eq.(3)), two ablation experiments are designed. The first ablation experiment is as follows: we removed the trained DPN and replaced it with a randomly initialized policy network to verify the critical role of a reasonable defection predictor in improving overall performance. This method is referred to as APC w/o DPN. The second ablation experiment is as follows: rather than using an adaptive probability, we simply set it to 1 throughout, which is denoted as APC w/o APr. The results of ablation experiments are shown in Table I. For APC w/o DPN, due to the failure to learn effective and reasonable punishment policy, it is unable to leverage the advantages of the punishment. The performance of APC w/o DPN was generally poor, there was no significant improvement in performance. For APC w/o APr, in the self-training setting (i.e., APC w/o APr agents interacting with each other), opponents learn to avoid punishment and reduce their defection behaviors, leading to low punishment frequency, despite the agent maintaining a high probability pijp^ij. As a result, its performance remains close to that of APC. However, when facing rule-based opponents (see Fig. 4(a), 4(c)), APC w/o APr agent sustains high punishment frequency, failing to adapt. This highlights the importance of adaptive probability in avoiding ineffective and excessive punishment when opponents do not respond to it. It is worth noting that in the results of APC w/o APr vs Defect in Fig. 4(c), the noise is caused by the stochastic nature of SSH, where the number of hare-hunting actions (defections) may vary across different time windows. V-F Adaptive Punishment Capability We evaluate APC’s adaptive capability from two perspectives: frequency and intensity. First, we assess adaptive punishment frequency by examining whether APC dynamically adjusts sanctions based on opponent behavior. The focal APC agent was tested against rule-based (Always-Defect, Always-Cooperate, Random) and shapeable (PPO, APC) opponents in IPGG and SSH. As shown in Fig. 4(a) and 4(c), APC initially targets defectors and random agents with high-frequency punishment while sparing cooperators. Over time, punishment self-regulates: the agent reduces ptijp_t^ij (Eq.(3)) if sanctions prove ineffective or if opponents (PPO/APC) shift toward cooperation, thereby avoiding excessive costs while maintaining behavioral control. Second, we evaluate adaptive punishment intensity using modified environments (MIPGG and MSSH) featuring varying defection severities. In MIPGG, we introduced partial contributions (C-0.2, C-0.1) alongside full contribution (C) and defection (D). In MSSH, actions included hunting hare with additional payoff reductions (H-0.2, H-0.3) compared to standard hare hunting (H) or other cooperative actions. Results in Fig. 4(b) and 4(d) show APC’s DPN assigns proportionate Punishment Intensity Weights (PIW). Specifically, PIW followed Eq.(5), yielding [1,0.79,0.70,0][1,0.79,0.70,0] for [D, C-0.1, C-0.2, C] in MIPGG, and [1,0.543,0.360,0][1,0.543,0.360,0] for [H-0.3, H-0.2, H, Other] in MSSH. These findings confirm APC’s ability to impose sanctions proportional to defection severity. VI Conclusions We propose Adaptive Punishment for Cooperation (APC), which combines dynamic punishment intensity with explicit defection awareness to align punishment intensity with defection severity, reducing exploitation and defection frequency. A limitation of this work is that APC has only been evaluated in relatively simplified environments. Its effectiveness and scalability in more complex and realistic multi-agent scenarios remain to be validated. Additionally, this work focuses solely on punishment; how APC complements reward-based method for cooperation remains an open question. References [1] N. Dasgupta and M. Musolesi (2025) Investigating the impact of direct punishment on the emergence of cooperation in multi-agent reinforcement learning systems. Auton. Agents Multi Agent Syst. 39 (1), p. 19. Cited by: §I. [2] J. N. Foerster, R. Y. Chen, M. Al-Shedivat, S. Whiteson, P. Abbeel, and I. Mordatch (2018) Learning with opponent-learning awareness. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, p. 122–130. Cited by: §V-C. [3] S. Gao, J. Du, and J. Liang (2020) Evolution of cooperation under punishment. Physical Review E 101 (6), p. 062419. Cited by: §I. [4] I. Gemp, K. R. McKee, R. Everett, E. A. Duéñez-Guzmán, Y. Bachrach, D. Balduzzi, and A. Tacchetti (2022) D3C: reducing the price of anarchy in multi-agent learning. In 21st International Conference on Autonomous Agents and Multiagent Systems, p. 498–506. Cited by: §V-C. [5] I. Gemp, K. R. McKee, R. Everett, E. A. Duéñez-Guzmán, Y. Bachrach, D. Balduzzi, and A. Tacchetti (2022) D3C: reducing the price of anarchy in multi-agent learning. In 21st International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2022, Auckland, New Zealand, May 9-13, 2022, p. 498–506. External Links: Document Cited by: §I. [6] J. Henrich, R. McElreath, A. Barr, J. Ensminger, C. Barrett, A. Bolyanatz, J. C. Cardenas, M. Gurven, E. Gwako, N. Henrich, et al. (2006) Costly punishment across human societies. Science 312 (5781), p. 1767–1770. Cited by: §I. [7] Y. Huang, A. Liu, F. Kong, Y. Yang, S. Zhu, and X. Feng (2024) Efficient adaptation in mixed-motive environments via hierarchical opponent modeling and planning. In Forty-first International Conference on Machine Learning, ICML 2024, Cited by: §I. [8] Y. Huang, X. Wang, H. Liu, F. Kong, A. Qin, M. Tang, X. Wang, S. Zhu, M. Bi, S. Qi, and X. Feng (2024) AdaSociety: an adaptive environment with social structures for multi-agent decision-making. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Cited by: §I. [9] E. Hughes, J. Z. Leibo, M. Phillips, K. Tuyls, E. A. Duéñez-Guzmán, A. G. Castañeda, I. Dunning, T. Zhu, K. R. McKee, R. Koster, H. Roff, and T. Graepel (2018) Inequity aversion improves cooperation in intertemporal social dilemmas. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems, p. 3330–3340. Cited by: §I. [10] N. Jaques, A. Lazaridou, E. Hughes, Ç. Gülçehre, P. A. Ortega, D. Strouse, J. Z. Leibo, and N. de Freitas (2019) Social influence as intrinsic motivation for multi-agent deep reinforcement learning. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, Vol. 97, p. 3040–3049. Cited by: §V-C. [11] F. Kong, Y. Huang, S. Zhu, S. Qi, and X. Feng (2024) Learning to balance altruism and self-interest based on empathy in mixed-motive games. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, Cited by: §I. [12] R. Köster, D. Hadfield-Menell, R. Everett, L. Weidinger, G. K. Hadfield, and J. Z. Leibo (2022) Spurious normativity enhances learning of compliance and enforcement behavior in artificial agents. Proceedings of the National Academy of Sciences 119 (3), p. e2106028118. Cited by: §I, §V-A. [13] J. Z. Leibo, E. A. Duéñez-Guzmán, A. Vezhnevets, J. P. Agapiou, P. Sunehag, R. Koster, J. Matyas, C. Beattie, I. Mordatch, and T. Graepel (2021) Scalable evaluation of multi-agent reinforcement learning with melting pot. In Proceedings of the 38th International Conference on Machine Learning, Vol. 139, p. 6187–6199. Cited by: §I, §I. [14] A. Lerer and A. Peysakhovich (2017) Maintaining cooperation in complex social dilemmas using deep reinforcement learning. arXiv preprint arXiv:1707.01068. Cited by: §V-A. [15] J. Li, Y. Liu, Z. Wang, and H. Xia (2021) Egoistic punishment outcompetes altruistic punishment in the spatial public goods game. Scientific reports 11 (1), p. 6584. Cited by: §I. [16] R. Lowe, Y. I. Wu, A. Tamar, J. Harb, O. Pieter Abbeel, and I. Mordatch (2017) Multi-agent actor-critic for mixed cooperative-competitive environments. Advances in neural information processing systems 30. Cited by: §I. [17] V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu (2016) Asynchronous methods for deep reinforcement learning. In International conference on machine learning, p. 1928–1937. Cited by: §V-C. [18] T. Ohdaira (2022) The probabilistic pool punishment proportional to the difference of payoff outperforms previous pool and peer punishment. Scientific Reports 12 (1), p. 6604. Cited by: §I. [19] J. Pérolat, J. Z. Leibo, V. F. Zambaldi, C. Beattie, K. Tuyls, and T. Graepel (2017) A multi-agent reinforcement learning model of common-pool resource appropriation. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems, p. 3643–3652. Cited by: §I. [20] A. Peysakhovich and A. Lerer (2018) Prosocial learning agents solve generalized stag hunts better than selfish ones. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, p. 2043–2044. Cited by: §V-A. [21] K. Schmid, L. Belzner, and C. Linnhoff-Popien (2021) Learning to penalize other learning agents. In 2021 Conference on Artificial Life, ALIFE 2021, online, July 19-23, 2021, J. Cejková, S. Holler, L. B. Soros, and O. Witkowski (Eds.), p. 59. Cited by: §I, §V-C. [22] Q. Song, Z. Cao, R. Tao, W. Jiang, C. Liu, and J. Liu (2020) Conditional neutral punishment promotes cooperation in the spatial prisoner’s dilemma game. Appl. Math. Comput. 368. External Links: Document Cited by: §I. [23] X. Sun, M. Li, H. Kang, Y. Shen, and Q. Chen (2023) Combined effect of pure punishment and reward in the public goods game. Appl. Math. Comput. 445, p. 127853. External Links: Document Cited by: §I. [24] P. Sunehag, G. Lever, A. Gruslys, W. M. Czarnecki, V. Zambaldi, M. Jaderberg, M. Lanctot, N. Sonnerat, J. Z. Leibo, K. Tuyls, et al. (2017) Value-decomposition networks for cooperative multi-agent learning. arXiv preprint arXiv:1706.05296. Cited by: §I, §V-D. [25] S. Wang, L. Liu, and X. Chen (2021) Tax-based pure punishment and reward in the public goods game. Physics Letters A 386, p. 126965. Cited by: §I. [26] Z. Wang, Z. Song, C. Shen, and S. Hu (2023) Emergence of punishment in social dilemma with environmental feedback. In Thirty-Seventh AAAI Conference on Artificial Intelligence, p. 11708–11716. Cited by: §I. [27] A. Yaman, J. Z. Leibo, G. Iacca, and S. W. Lee (2022) The emergence of division of labor through decentralized social sanctioning. CoRR abs/2208.05568. Cited by: §I, §V-C.