Paper deep dive
Equilibrium stability as a driver of cooperation among Q-learners
Janusz M. Meylahn, Maximilian Schäfer
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 7/18/2026, 9:59:39 AM
Summary
This paper investigates algorithmic collusion among Q-learning agents in a repeated Prisoner's Dilemma with constant exploration. It challenges the assumption that cooperation requires vanishing exploration, proposing instead that equilibrium stability drives time-averaged cooperative behavior. The authors derive an analytic boundary based on Q-value differences to predict when cooperative strategies dominate over defection, validated by simulations showing high predictive accuracy.
Entities (8)
Relation Signals (6)
Janusz M. Meylahn → authored → Equilibrium stability as a driver of cooperation among Q-learners
confidence 99% · Equilibrium stability as a driver of cooperation among Q-learners Janusz M. Meylahn
Maximilian Schäfer → authored → Equilibrium stability as a driver of cooperation among Q-learners
confidence 99% · Maximilian Schäfer... Equilibrium stability as a driver of cooperation among Q-learners
Q-learning → appliedto → Prisoner's Dilemma
confidence 95% · we study learning dynamics... in the repeated Prisoner’s Dilemma... based on the expected dynamics of the Q-learning process.
Equilibrium Stability → drives → Cooperation
confidence 92% · Equilibrium stability as a driver of cooperation among Q-learners... cooperative strategies can be dominant in this time-averaged sense
Win-Stay Lose-Shift → comparedwith → All-Defect
confidence 90% · comparing the stability of a forgiving reward-punishment strategy (win-stay, lose-shift) with that of all-defection
Constant Exploration → enables → Algorithmic Collusion
confidence 85% · Motivated by the observation that algorithms deployed in practice are likely to continue exploring... we study learning dynamics under constant exploration.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Algorithmic collusion among pricing algorithms has raised concerns about sustained supra-competitive prices and their implications for social welfare. Existing work has largely focused on the probability that reinforcement-learning algorithms converge to cooperative strategies, typically under the assumption that exploration vanishes over time. Motivated by the observation that algorithms deployed in practice are likely to continue exploring in order to remain adaptive to changing environments, we study learning dynamics under constant exploration. In this setting, the relevant question is no longer whether an algorithm converges to a particular strategy profile, but rather what fraction of time the algorithms spend playing cooperative strategies. Even in the benchmark case of the repeated Prisoner's Dilemma with one-period memory, this yields high-dimensional stochastic learning dynamics, for which a complete analytic treatment is intractable. We show that cooperative strategies can be dominant in this time-averaged sense and derive a boundary predicting when such dominance arises, based on the expected dynamics of the Q-learning process. Extensive simulations show that this boundary is a strong predictor for non-defection-dominated behaviour under epsilon-greedy Q-learning.
Tags
Links
- Source: https://arxiv.org/abs/2607.13607v1
- Canonical: https://arxiv.org/abs/2607.13607v1
Trouble viewing inline? Open PDF directly →
Full Text
87,898 characters extracted from source content.
Expand or collapse full text
Equilibrium stability as a driver of cooperation among Q-learners Janusz M. Meylahn 111University of Twente, Department of Applied Mathematics, Drienerloolaan 5, 7522 NB Enschede, The Netherlands. j.m.meylahn@utwente.nl Maximilian Schäfer 222Institut Mines-Télécom Business School, Department of Data Analytics, Economics, and Finance. 9 Rue Charles Fourier, 91000 Evry, France. maximilian.schafer@imt-bs.eu (July 15, 2026) Abstract Algorithmic collusion among pricing algorithms has raised concerns about sustained supra-competitive prices and their implications for social welfare. Existing work has largely focused on the probability that reinforcement-learning algorithms converge to cooperative strategies, typically under the assumption that exploration vanishes over time. Motivated by the observation that algorithms deployed in practice are likely to continue exploring in order to remain adaptive to changing environments, we study learning dynamics under constant exploration. In this setting, the relevant question is no longer whether an algorithm converges to a particular strategy profile, but rather what fraction of time the algorithms spend playing cooperative strategies. Even in the benchmark case of the repeated Prisoner’s Dilemma with one-period memory, this yields high-dimensional stochastic learning dynamics, for which a complete analytic treatment is intractable. We show that cooperative strategies can be dominant in this time-averaged sense and derive a boundary predicting when such dominance arises, based on the expected dynamics of the Q-learning process. Extensive simulations show that this boundary is a strong predictor for non-defection-dominated behaviour under epsilon-greedy Q-learning. Mathematics Subject Classification 2010: 68T05, 60J20, 91A05 JEL Classification: D21, D43, D83, L12, L13 Artificial Intelligence, Pricing-Algorithms, Collusion, Reinforcement Learning, Q-Learning 1 Introduction A large amount of recent work has highlighted the potential danger of algorithmic collusion among pricing algorithms (Assad et al., 2024; Brown and MacKay, 2023). Such collusion would lead to supra-competitive prices and is most likely to be legal under current antitrust law (MacKay and Weinstein, 2022; Harrington, 2018). Early work already showed that reinforcement-learning algorithms can coordinate on supra-competitive outcomes in oligopoly settings (Waltman and Kaymak, 2008), while more recent studies have shown that Q-learning algorithms may sustain elevated prices in combination with reward-punishment schemes consistent with collusive behaviour (Calvano et al., 2020; Klein, 2021); alongside proposals for auditing algorithms for collusion (Hartline et al., 2024). Whether algorithmic collusion poses a realistic threat to social welfare remains debated, with multiple studies reporting mixed evidence and questioning the robustness and interpretation of such cooperative outcomes (Asker et al., 2022; Lambin, 2024). Much of the reinforcement-learning literature studies algorithms with a constant learning rate and allows for persistent stochasticity in the environment. This modelling choice is standard and well motivated: a constant learning rate ensures that the algorithm remains responsive to changes or non-stationarities in payoffs, while environmental noise guarantees continued exploration of the state space (Kushner and Huang, 1981a, b; Beck and Srikant, 2012). In this setting, stochasticity is exogenous to the algorithm and serves to support adaptability. By contrast, in the algorithmic collusion literature, stochasticity typically originates from the agents’ own exploration. Many studies assume that exploration rates decline over time, so that learning eventually becomes effectively deterministic (Calvano et al., 2020, 2021; Asker et al., 2022). While this facilitates convergence to stable pricing policies, it also reduces the ability of algorithms to adapt to changes in the environment or to the behaviour of other learning agents. This limitation is particularly salient in multi-agent settings, which are inherently non-stationary (den Boer et al., 2024; den Boer and Meylahn, 2024; Lambin, 2024). Moreover, real-world demand conditions are unlikely to be stable over long horizons, suggesting that pricing algorithms deployed in practice may need to continue exploring in order to remain effective. In this paper, we study cooperation among Q-learners with a memory of one period in the prisoner’s dilemma when both the exploration and learning rates are kept constant. Our focus on memory-one strategies builds on recent work characterising equilibrium structure and learning dynamics in repeated prisoner’s dilemma games (Usui and Ueda, 2021; Meylahn and Janssen, 2022). With constant exploration and learning, the resulting learning dynamics no longer converge to a particular strategy profile and stay there indefinitely. Instead, the system transitions persistently between strategy profiles. This observation motivates a shift in perspective: rather than asking whether learning converges to a cooperative outcome, we measure cooperation by the fraction of time the learning dynamics spend in cooperative versus non-cooperative strategy profiles. We provide a theoretical characterisation of these learning dynamics and propose a heuristic boundary predicting when cooperation dominates in this time-averaged sense. The boundary is derived from the relative stability of equilibrium strategy profiles in the presence of persistent fluctuations. Related work has shown that stochastic fluctuations can play a central role in sustaining or disrupting cooperation in learning and evolutionary dynamics, including in repeated prisoner’s dilemma settings (Barfuss and Meylahn, 2023; Dolgopolov, 2024; Xu and Zhao, 2024). This stability-based mechanism uncovers a novel route to cooperation among reinforcement-learning algorithms–one that does not rely on vanishing exploration or absorption into a cooperative equilibrium. Extensive simulations show that the proposed boundary predicts the empirical transition between cooperative and non-cooperative regimes with high accuracy This perspective complements existing work in the algorithmic collusion literature, where collusion is typically assessed by the probability of converging to supra-competitive pricing once learning has effectively ceased. We introduce a new perspective by focusing on the long-run fraction of time that algorithms price supra-competitively as the measure of collusion. From a regulatory standpoint, this distinction is important because the extent of harm to social welfare is a function not only of the price but also of the time that the price is used. Our results show that equilibrium stability can be used to predict the fraction of time spent pricing supra-competitively. Interpreted economically, this stability reflects the robustness of algorithmic cartels to ongoing fluctuations induced by learning and exploration. Social welfare losses from algorithmic collusion thus depend not only on the existence of collusive strategies, but on their resilience to persistent noise in the learning process. We approach the problem by considering the joint dynamics of the state and Q-values as a Markov process. Here the state space is discrete while the Q-value space is continuous. Analysing the true dynamics of this process is intractable, but by projecting from the Q-value to the strategy profile space, we obtain a hidden Markov model with a finite state space. Our observable of interest is the distribution of occupation times over strategy profiles. We prove that there is a positive probability of observing a transition from any strategy profile to another strategy profile, where each of the strategies is a best response to some strategy. Strategy profiles with this property thus form part of a unique recurrent set of the dynamics. Based on this proof, we conjecture that the distribution of occupation times converges to a unique stationary distribution in the long time limit. By considering an idealised learning process where we only take into account the expected learning dynamics and linking this to properties of the unique stationarity distribution, we derive an analytic condition on the parameters for predicting when the stationary distribution places more mass on cooperative strategy profiles than on competitive strategy profiles. Concretely, this is based on comparing the optimal Q-value differences at different stationary points of the expected learning dynamics. This comparison captures the idea that the stability of a strategy profile is predominantly determined by the probability of observing a reversal in the order of the Q-values. The probability of such a reversal is higher when the optimal Q-value difference is smaller. Even though the stationary distribution of the occupation time is reached in the long time limit, the time it takes to be stationary depends on the choice of parameters. Most important in this regard are the learning and exploration rates. For our simulations, we thus fix a time horizon and identify the regions of the parameter space for which stationarity is reached within this time horizon. This gives us the region in which we can expect our theoretical framework to apply. We proceed by sweeping the parameter space and recording the average empirical occupation time in the stationary distribution for the strategy profiles that occur most commonly in the simulations. We compare the theoretical boundary to the simulation outcomes by overlaying it on heatmaps of average empirical occupation times. This visual comparison shows that the boundary closely aligns with the transition between cooperative and non-cooperative strategy profiles. To provide a more systematic evaluation, we interpret the boundary as a decision rule to predict the dominance of cooperative outcomes and assess its performance using standard statistical metrics for classification. Despite being based on a heuristic that abstracts from several intricacies of the learning dynamics, the boundary performs remarkably well, achieving macro F1 scores ranging from 0.85 to 0.95. 2 Related Literature Here, we provide a targeted review of the literature, highlighting its relevance to debates on algorithmic collusion in economics and its connection to, and extension of, prior work on algorithmic interactions in the prisoner’s dilemma and on stochastic effects in reinforcement-learning-based cooperation. Economic Literature on Algorithmic Collusion. A series of studies has nuanced the interpretation of simulation-based evidence of algorithmic collusion (Asker et al., 2024; Abada and Lambin, 2023; Epivent and Lambin, 2024; Abada et al., 2024b). More specifically, Abada and Lambin (2023) and Lambin (2024) identify vanishing exploration schedules as a key driver of cooperative patterns in algorithmic pricing environments, arguing that observed reward-punishment schemes may be artefacts of the learning process. This perspective calls into question the economic interpretation of reward-punishment schemes emphasised by Calvano et al. (2020) and Klein (2021). Motivated by this critique, we study a setting with persistent exploration, in which learning does not converge to a single policy and cooperative behaviour must instead be assessed through time-averaged dominance rather than convergence. Within this perspective, we show that cooperative strategy profiles can dominate long-run behaviour and derive a predictive boundary based on the relative stability of equilibrium strategy profiles under ongoing fluctuations. In particular, our analysis compares the stability of a forgiving reward-punishment strategy (win-stay, lose-shift) with that of all-defection, yielding a mechanism for sustained cooperation that does not rely on exploration vanishing to zero. Algorithmic interactions in memory-one prisoner’s dilemma games. Recent work has focused on analysing the dynamics of stateless Q-learners in the prisoner’s dilemma under a variety of self-play (in the sense of coupling between Q-tables) assumptions (Banchio and Mantegazza, 2022, 2023). The case where the players make use of a memory of one period as a state space has received significant attention recently. Conditions for the existence of equilibria have been derived (Usui and Ueda, 2021; Meylahn and Janssen, 2022). Convergence and a basin of attraction analysis of a batched version of Q-learning have been shown for the prisoner’s dilemma (Meylahn, 2025) and a three-action variant thereof (Meylahn, 2023). The metagame of hyperparameter selection has been analysed (Carissimo et al., 2025), and conditions for the transition from defection to cooperation have been derived under a self-play assumption (Bertrand et al., 2025). Previous work has also focused on the relationship between the speed of learning and exploration rate decrease (den Boer et al., 2024). Fluctuations. The papers most closely related to our work are those that examine the interaction between fluctuations and cooperation. Two recent studies employ the framework of stochastic stability to analyse mechanisms of algorithmic cooperation in the prisoner’s dilemma without memory (Dolgopolov, 2024; Xu and Zhao, 2024). Closely related is the finding that intrinsic fluctuations can facilitate cooperation in batched versions of Q-learning (Barfuss and Meylahn, 2023). An alternative modelling approach treats actions as choices among memory-one strategies played for fixed periods, with reinforcement learning used to adapt strategy selection. When the strategy set includes Tit-for-Tat, Always-Defect, and Always-Cooperate, stochasticity can induce cycles between cooperation and defection (Galla, 2011). Similar dynamics arise when the Win-Stay, Lose-Shift strategy is included (Bladon et al., 2010). Our heuristic boundary builds on these fluctuation-based insights by identifying the Q-value differences that are critical for assessing the relative stability of competing equilibria. 3 Model In this section, we introduce the environment and algorithm we will consider. Together they determine a Markovian process. To facilitate our analysis in Section 4, we also introduce a related idealised process. The environment we consider is the iterated prisoner’s dilemma, where two players, labelled i∈1,−1i∈\1,-1\, employ a state space which consists of all one-period histories and can thus condition their action-selection probabilities on the actions taken in the previous round. This means that we have =(D,D),(D,C),(C,D),(C,C)S=\(D,D),(D,C),(C,D),(C,C)\ and i==D,CA_i=A=\D,C\, where D denotes the “defect” action and C denotes the “cooperate” action. We denote the state at time t∈0,1,…t∈\0,1,…\ by st∈s_t and the actions taken at time t by t=(at1,at−1) a_t=(a^1_t,a^-1_t), with atia_t^i the action of player i. Note that st+1=ts_t+1= a_t, which thus defines our transition dynamics. The rewards ()=(r1(),r−1()) r( a)=(r^1( a),r^-1( a)) are independent of the state and are defined as r1()=Pwhen =(D,D)Twhen =(D,C)Swhen =(C,D)Rwhen =(C,C),andr−1()=Pwhen =(D,D)Swhen =(D,C)Twhen =(C,D)Rwhen =(C,C), r^1( a)= casesP a=(D,D)\\ T a=(D,C)\\ S a=(C,D)\\ R a=(C,C)\\ cases, r^-1( a)= casesP a=(D,D)\\ S a=(D,C)\\ T a=(C,D)\\ R a=(C,C)\\ cases, (3.1) with T>R>P>ST>R>P>S. The players want to maximise the expected sum of discounted rewards, i.e., [∑t=1∞δtrti],E [ _t=1^∞δ^tr_t^i ], (3.2) where δ∈(0,1)δ∈(0,1). We are interested in studying the dynamics of the system Qt+1i(s,a)=(1−α)Qti(s,a)+α[(1−δ)rt()for (s,a)=(st,ati) and =t+δmaxa′Qti(st+1,a′)]Qti(s,a)else,Q_t+1^i(s,a)= cases(1-α)Q_t^i(s,a)+α[(1-δ)r_t( a)&for (s,a)=(s_t,a_t^i) and a= a_t\\ 56.9055pt+\;δ _a Q_t^i(s_t+1,a )]\\ Q_t^i(s,a) &else cases, where we initialise the Q-values Q0iQ_0^i as U[S,T]U[S,T] (uniformly between S and T), α∈(0,1)α∈(0,1) is the learning rate, and we normalise the payoffs by 1−δ1-δ to ensure that the Q-values remain in =[S,T]Q=[S,T] for aesthetic purposes and without loss of generality. At each point in time, player i selects an action according to the ϵε-greedy mechanism, with ϵ∈(0,1)ε∈(0,1) ati=argmaxa∈Qti(st,a)with probability 1−ϵ/2argmina∈Qti(st,a)with probability ϵ/2.a_t^i= cases *arg\,max_a Q_t^i(s_t,a) &with probability 1-ε/2\\ *arg\,min_a Q_t^i(s_t,a) &with probability ε/2 cases. (3.3) This means that a player plays their greedy action with probability 1−ϵ1-ε and takes an action uniformly at random with probability ϵε. Our setting is thus parameterised by θ=(T,R,P,S,δ,ϵ,α)∈Θθ=(T,R,P,S,δ,ε,α)∈ , where Θ denotes the space of allowed values for the parameters. We can split θ into the environmental parameters θe=(T,R,P,S)θ^e=(T,R,P,S) and the algorithmic hyperparameters θa=(δ,ϵ,α)θ^a=(δ,ε,α), such that θ=(θe,θa)θ=(θ^e,θ^a). The underlying Q-value dynamics together with the action-selection mechanism lead to trajectories of state-action pairs s0,0,s1,1,…,\s_0, a_0,s_1, a_1,…\, (3.4) where s0s_0 is taken uniformly from S. Since we have st+1=ts_t+1= a_t, it is sufficient to consider s0,s1,s2,….\s_0,s_1,s_2,…\. (3.5) The state of the system as a whole consists of (st,t)∈×,(s_t, Q_t) ×Q×Q, (3.6) where t=(Qt1,Qt−1) Q_t=(Q_t^1,Q_t^-1). The Q-table of a player can be projected onto the space of ϵε-greedy strategies Πϵ _ε with a fixed ϵε using π(Qti)=( π(Q^i_t)= ( argmaxa∈Qti((D,D),a),argmaxa∈Qti((D,C),a), *arg\,max_a Q_t^i((D,D),a), *arg\,max_a Q_t^i((D,C),a), argmaxa∈Qti((C,D),a),argmaxa∈Qti((C,C),a)). *arg\,max_a Q_t^i((C,D),a), *arg\,max_a Q_t^i((C,C),a) ). For example, if player i has Qti(s,D)>Qti(s,C)Q_t^i(s,D)>Q_t^i(s,C) for all s∈s , then they are playing the ϵε-greedy All-Defect (AD) strategy with π(Qi)=(D,D,D,D)π(Q^i)=(D,D,D,D). In our setting, there are |Πϵ|=16| _ε|=16 such strategies and therefore 256 strategy profiles ∈Πϵ×Πϵ π∈ _ε× _ε. We will predominantly be studying the dynamics of (st,(t))∈×Πϵ×Πϵ.(s_t, π( Q_t)) × _ε× _ε. (3.7) Note that the dynamics of (st,t)(s_t, Q_t) are Markovian, but that the dynamics of (st,(t))(s_t, π( Q_t)) are not, since the probability of observing a particular transition between ϵε-greedy strategy profiles depends on the exact Q-values. We are thus studying a hidden Markov model. The advantage of studying (st,(t))(s_t, π( Q_t)) is that these dynamics occur in a discrete space. Idealised Process. To derive our boundary predicting cooperation, we will analyse points of the dynamics at which the expected dynamics are stationary. To identify these, we make use of an idealised process, which we describe next. At every point in time, we can consider a single player i to be facing a stationary environment defined by the combination of the prisoner’s dilemma and the opponent’s ϵε-greedy strategy πt−i _t^-i. This stationary environment defines an optimal best response BR(πt−i)BR( _t^-i) which is unique for almost all111The best response is not unique for a set of parameters with Lebesgue measure zero. T,R,P,S,δT,R,P,S,δ, and ϵε. In an idealised learning process, each agent would have perfect information of this stationary environment and could thus update their strategy to or towards the best response at each learning step. The best-response function BR(⋅)BR(·) induces a functional relation on Πϵ _ε, called the Individual Best-Response (IBR) graph, in which Nash equilibria (NE) appear as self-loops or two-cycles (Meylahn, 2025). In previous work, it was shown that in the prisoner’s dilemma with a memory of one period there are three possible NE (Usui and Ueda, 2021; Meylahn and Janssen, 2022), namely, both players playing πAD=(D,D,D,D),πGT=(D,D,D,C) or πWSLS=(C,D,D,C).π^AD=(D,D,D,D), π^GT=(D,D,D,C)\; or \;π^WSLS=(C,D,D,C). (3.8) Each NE is associated with optimal Q-values, which we name QAD,QGTQ^AD,Q^GT and QWSLSQ^WSLS, respectively. These are defined as the solution to Bellman’s optimality equation when both players are playing the strategy associated with the NE, i.e., the solution to QNE(s,a;θ)=NE[(1−δ)ri()+δmaxa∈QNE(s′,a;θ)],Q^NE(s,a;θ)=E_NE [(1-δ)r^i( a)+δ _a Q^NE(s ,a;θ) ], (3.9) where the expectation is with respect to the dynamics induced by both players playing NE and s′s is the next state. Here we make the dependence on the parameters θ explicit. For both the idealised and true learning process, the expected dynamics of Q_t are stationary at these values, that is, [t+1|t=(QNE,QNE)]=(QNE,QNE),E[ Q_t+1| Q_t=(Q^NE,Q^NE)]=(Q^NE,Q^NE), (3.10) for NE∈AD,GT,WSLSNE∈\AD,GT,WSLS\. 4 Theoretical Analysis In this section, we give our theoretical analysis of the dynamics of the model described in Section 3. The Markovian dynamics of (st,t)(s_t, Q_t) are not necessarily ergodic, since it could be that there are regions of the Q-value space Q that cannot be reached once they have been left222Adding, for example, Gaussian noise to the rewards would likely make the dynamics ergodic. (Meyn and Tweedie, 2012). We are interested in studying the fraction of time spent in each of the strategy profiles and the fraction of time spent in each of the four states. To this end, we introduce the empirical occupation time of strategy profiles for a trajectory up to time horizon ThT_h τTh=0,1,…Th _T_h=\ π_0, π_1,… π_T_h\ (4.1) as O(τTh,)=1Th∑t=1Th=.O( _T_h, π)= 1T_h _t=1^T_h 1_\ _t= π\. (4.2) O(τTh,)O( _T_h, π) captures the fraction of time that was spent in π during trajectory τTh _T_h. Similarly, and with a slight abuse of notation,333The second argument of the function makes clear which occupation time is meant. we define the empirical occupation time of the states to be O(σTh,s)=1Th∑t=1Thst=s,O( _T_h,s)= 1T_h _t=1^T_h 1_\s_t=s\, (4.3) with σTh=s0,s1,…,sTh _T_h=\s_0,s_1,…,s_T_h\. For Conjecture 4.2 we argue that the occupation time of strategy profiles converges to a unique stationary distribution when α is small enough. If this is the case, the distribution over states will also be stationary and unique. Theorem 4.1 (Unique Recurrent Set). Fix T,R,P,S,δT,R,P,S,δ, and ϵε, then there exists αcα^c such that for α<αcα<α^c the set of strategy profiles (π~1,π~−1):π~i=BR(π) for some π∈Πϵ\( π^1, π^-1): π^i=BR(π) for some π∈ _ε\ (4.4) form part of a unique recurrent set. The proof of Theorem 4.1 is given in Appendix A, but we provide a brief sketch of the proof here. Starting at any (st,t)(s_t, Q_t), we observe any finite sequence of play of length K with positive probability. In particular, we may observe a sequence that looks as if it were generated by the players playing strategy profile (π1,π−1)(π^1,π^-1) for any π1,π−1∈Ππ^1,π^-1∈ . From the perspective of player i, it thus seems as if they are facing an opponent playing π−iπ^-i for those K periods. Using Beck and Srikant (2012), we can establish that the Q-values that player i learns during this sequence of play are sufficiently close to the Q-values corresponding to BR(π−i)BR(π^-i) that the player actually learns the best response. This occurs with positive probability when α is less than some critical value, which we calculate explicitly. Since this occurs with positive probability in a finite number of steps for any initial condition, we can split the time horizon into non-overlapping blocks of length K to conclude that we will return to all strategy profiles consisting of strategies that are a best response to some other strategy infinitely often. Note that there may be more strategy profiles in the unique recurrent set than the ones identified here. Conjecture 4.2 (Unique Stationary Distribution). Fix T,R,P,S,δT,R,P,S,δ, and ϵε, then there exists αcα^c such that for α<αcα<α^c limTh→∞O(τTh,)=ρ(θ,) _T_h→∞O( _T_h, π)=ρ(θ, π) (4.5) for all π and 0 π_0 and some ρ(θ,)ρ(θ, π). From Theorem 4.1 we know that the Q-values will return to the neighbourhood of a set of Q-values corresponding to strategy profiles of a particular kind infinitely often when α<αcα< _c, but this does not immediately imply that a stationary distribution for the Markov process exists and is unique. A potential approach for showing that this is the case would be to use the Krylov-Bogolioubov Theorem Kryloff and Bogoliouboff (1937) to show that there exists at least one invariant probability measure of the process and then to use (Hairer, 2010, Corollary 2.7) to show that there exists at most one invariant measure. To apply these two results, the process must (1) take place in a Polish space, (2) there must be at least one accessible point, (3) there must exist an initial condition x0=(s0,0)x_0=(s_0, Q_0) so that the sequence n(x0,⋅)P^n(x_0,·) is tight, and (4) the operator P of the process must be a strong Feller Markov operator. Now (1) is satisfied, (2) also holds as shown in the proof of Theorem 4.1 and (3) holds as the space is compact. It remains to be shown that P is a strong Feller operator. Verifying this condition is more involved, and it is unclear whether it holds.444Note that the conjecture might be true even if the fourth condition doesn’t hold, as then other proof strategies are available. However, our simulations suggest that Conjecture 4.2 holds. Figure 1: Critical condition for the stability of WSLS being greater than that of AD. On the left, we plot the critical condition on δ as a function of ϵε for different values of R (with T=1T=1, S=0S=0 and P=0.1P=0.1), and on the right we plot the critical condition on R as a function of P for different values of ϵε (with T=1T=1, S=0S=0 and δ=0.8δ=0.8). The question we are interested in is when the fraction of time spent in cooperative strategy profiles dominates. Of the three NE, only WSLS leads to stable cooperation. So if ρ(θ,)ρ(θ, π) only places positive mass on NE555This premise is not satisfied in our case, but the argument serves as a basis for deriving our heuristic boundary., we would want to know when ρ(θ,πWSLS)>0.5.ρ(θ,π^WSLS)>0.5. (4.6) We will proceed heuristically to find a critical condition θcθ^c on the parameters that predicts when cooperative strategy profiles dominate the stationary distribution. To this end, note that the fraction of time spent in the three equilibria depends on their stability. The stability of an equilibrium depends on at least two components (Freidlin and Wentzell, 2012): 1. the size of the fluctuations in the equilibrium, 2. and the height of the barrier that the fluctuations must overcome to exit the equilibrium. For our boundary, we will focus only on the second component. To motivate this, note that the exploration rate, which is the source of fluctuations, is the same in all equilibria.666This does not mean that the fluctuations in Q-value estimates are the same as these also depend on the rewards and continuation payoffs. It is thus reasonable to assume that the difference in stability between equilibria is predominantly determined by the second component. To exit an equilibrium, the Q-value orderings must be reversed in one of the four states. Intuitively, this is most likely to occur in the most vulnerable state, by which we mean the state with the smallest optimal Q-value difference, and the height of the barrier is characterised by the optimal Q-value differences in that state. For each equilibrium, we thus define the Q-value differences in each state as ΔQNE(s;θ):=QNE(s,argmaxa∈QNE(s,a;θ);θ)−QNE(s,argmina∈QNE(s,a;θ);θ). Q^NE(s;θ):=Q^NE(s, *arg\,max_a Q^NE(s,a;θ);θ)-Q^NE(s, *arg\,min_a Q^NE(s,a;θ);θ). (4.7) To determine when cooperation will be dominant, we want to know when the WSLS equilibrium is more stable than the AD equilibrium777Note that AD is always a Nash equilibrium, while WSLS is only a Nash equilibrium in some parts of the parameter space (Meylahn, 2025).. Our critical condition is thus defined by solving mins∈ΔQWSLS(s;θ)=mins∈ΔQAD(s;θ) _s Q^WSLS(s;θ)= _s Q^AD(s;θ) (4.8) for θ. Solving this, we find the critical condition for the discount factor δc:=2(T+P−(R+S))(1−ϵ)[2(R−P)+ϵ(P+S−(R+T))],δ^c:= 2(T+P-(R+S))(1-ε)[2(R-P)+ε(P+S-(R+T))], (4.9) such that WSLS is more stable than AD when δ>δcδ>δ^c. The details of this derivation are given in Appendix B. In Figure 1, we plot the critical condition in two slices of the parameter space. The boundary behaves sensibly in the following sense. δcδ^c increases as ϵε increases and as R decreases, meaning that the required discount factor for cooperation dominating increases as the exploration rate increases and as the value of cooperation decreases. On the other hand, RcR^c increases as P increases and as ϵε increases. Note that the critical boundary derived here does not depend on the learning rate α. However, α does play a role in determining when our boundary applies on the timescale of our simulations (see Section 5.2). 5 Simulations 5.1 Simulation Details In this subsection, we describe the payoff normalisation used in the simulations, specify which strategy profiles are tracked explicitly, and explain how interaction dynamics outside this focal set of tracked strategies are incorporated. Payoff Normalisation Throughout our simulations, we normalise payoffs by setting T=1T=1 and S=0S=0. This normalisation imposes natural bounds on the remaining payoff parameters R and P, allowing us to simulate the full range of incentive structures consistent with the prisoner’s dilemma on a bounded parameter grid, which substantially improves computational tractability. Focal Strategy Profiles Rather than tracking occupation times for all 256256 possible memory-one strategy profiles, we focus on a small set of theoretically and empirically salient profiles. Specifically, we consider the three Nash equilibrium strategy profiles emerging from our theoretical analysis, together with two additional profiles that appear frequently in preliminary simulations: All-Cooperate (ACAC) and Anti-Grim Trigger (AGTAGT), in which both players cooperate following mutual defection and defect otherwise. This set defines the focal strategy class Π^ϵ _ε. The empirical relevance of ACAC and AGTAGT can be summarised by the following stylised facts: 1. Best-response prevalence. Both ACAC and AGTAGT arise as best responses to other strategies over nontrivial regions of the parameter space. In particular, ACAC can be a best response to GTGT, (D,C,D,D)(D,C,D,D), (D,C,D,C)(D,C,D,C), (D,C,C,C)(D,C,C,C), and (C,C,D,C)(C,C,D,C), while AGTAGT can be a best response to (D,C,C,C)(D,C,C,C) (see Table IV in Meylahn (2025)). By Theorem 4.1, this implies a positive probability of reaching mutual play of ACAC or AGTAGT. 2. Proximity to the stag-hunt region. Both strategies occur predominantly when R is close to T. In this region, the game approaches a stag hunt, in which both ACAC and AGTAGT are Nash equilibria Meylahn (2025). When R≈TR≈ T, learning dynamics may therefore blur distinctions between underlying game classes, since Q-values encode only approximations of expected continuation payoffs. 3. Game-class boundaries. The strategy AGTAGT rarely appears when P is close to zero, where the game approaches a snowdrift game in which AGTAGT is not an equilibrium. By contrast, when P is close to zero and R is close to one, the game approaches a harmony game, in which ACAC is always an equilibrium, explaining its prevalence in that region of the parameter space. Tracking Interaction Dynamics Outside the Focal Strategy Profiles Our focus on a restricted set of strategy profiles is guided by two considerations: alignment with the theoretical analysis, which compares stability properties of specific profiles, and the desire to capture a substantial share of empirically observed behaviour while maintaining computational and representational tractability. At the same time, the total fraction of time spent in the five focal profiles varies across the parameter space.888Table 2 shows part of this variation and confirms that the five focal strategies dominate in the stationary distribution throughout our simulations. This raises the question of whether untracked strategies might affect the interpretation of cooperative and non-cooperative regimes. To address this concern, we complement the profile-based analysis with a state-space perspective that examines occupation times of the four action states (C,C),(C,D),(D,C),(D,D)\(C,C),(C,D),(D,C),(D,D)\. Because state occupation aggregates behaviour across all possible strategy profiles, this analysis provides a comprehensive and tractable robustness check that incorporates interaction dynamics beyond Π^ϵ _ε. As such, the state-space analysis ensures that our conclusions are informed by the full range of interaction dynamics, including those generated by strategies outside the focal set. 5.2 Accessible Parameter Range under a Finite Time Horizon The theoretical analysis in Section 4 characterises stable cooperation under the assumption that the learning dynamics converge to a unique stationary distribution in the infinite time limit. In practice, however, computational constraints restrict simulations to a finite time horizon. Throughout our simulations, we therefore fix the horizon at Th=200MT_h=200M periods. Before evaluating the quality of the proposed decision boundary, it is necessary to assess under which conditions this finite horizon is sufficient for the learning dynamics to converge to a stationary distribution. Given Th=200MT_h=200M, we define the accessible parameter range as the set of algorithmic hyperparameters for which the induced dynamics converge to a stationary distribution within the allotted time horizon. Method to Scope the Accessible Parameter Range Our approach is to simulate multiple trajectories from diverse initial conditions that span a large portion of the initial Q-value space. If all trajectories converge to the same empirical occupation-time distribution–independently of initialisation–we treat the corresponding parameter configuration as having reached the unique stationary distribution within the chosen time horizon. Initialisations. We consider ten initialisations. These include an optimistic initialisation with Q0(s,a)=1Q_0(s,a)=1 for all (s,a)(s,a), a pessimistic initialisation with Q0(s,a)=0Q_0(s,a)=0 for all (s,a)(s,a), and three initialisations at the optimal Q-values corresponding to the Nash equilibrium strategy profiles ADAD, GTGT, and WSLSWSLS, i.e., Q0(s,a)=QNE(s,a;θ)Q_0(s,a)=Q^NE(s,a;θ) for all (s,a)(s,a) and NE∈AD,GT,WSLSNE∈\AD,GT,WSLS\. In addition, we include five trajectories with Q-values initialised independently and uniformly at random on [0,1][0,1]. Stationarity measure. To assess convergence across initialisations, we compare empirical occupation times over strategy profiles. Specifically, for a given parameter vector θ, we compute the maximum absolute difference in occupation times across all initialisations and across all strategy profiles in the focal set Π^ϵ _ε, which accounts for the bulk of the stationary distribution: ΔO(θ):=maxi,j∈ℐmaxπ∈Π^ϵ|O(τThi(θ),(π,π)))−O(τThj(θ),(π,π)))|, O(θ):= _i,j _π∈ _ε |O( _T_h^i(θ),(π,π)))-O( _T_h^j(θ),(π,π))) |, (5.1) where ℐI denotes the set of initialisations described above and τThi(θ) _T_h^i(θ) denotes the trajectory generated from initialisation i up to time ThT_h under parameters θ. We classify a parameter setting θ as stationary if ΔO(θ)<Oc O(θ)<O^c. In all simulations, we set the threshold to Oc=0.05O^c=0.05. Illustrative examples. Figure 2: Examples of occupation-time trajectories for ten initialisations under three parameter settings θ. Left: a well-mixed setting in which trajectories converge to the same occupation times within the simulation horizon. Middle: an intermediate case in which trajectories appear close to convergence but exhibit persistent switching across strategy profiles. Right: a non-mixing setting in which occupation times diverge across initialisations even at long horizons. We select strategy profiles that have meaningful shares in at least one of three cases considered. Figure 2 illustrates the classification procedure. In the left panel, occupation times converge to the same constant across all initialisations within the simulation horizon, indicating convergence to a stationary distribution. In contrast, the right panel shows persistent differences across trajectories, implying that stationarity has not been reached. The middle panel illustrates a more ambiguous case. While trajectories may appear to approach similar occupation levels, the pronounced switching behaviour suggests slow mixing, with extended periods spent in different strategy profiles. In such cases, empirical occupation times over the considered finite horizon may be misleading. Parameter grid. From preliminary simulations, we observe that convergence to stationarity depends primarily on the algorithmic parameters θa=(δ,ϵ,α)θ^a=(δ,ε,α). For this reason, we fix the environmental parameters θe=(R,P)θ^e=(R,P) for each θaθ^a rather than simulating the full five-dimensional parameter space when identifying the accessible parameter range999Note that we do simulate the full parameter space when assessing the quality of the boundary.. Previous work (Barfuss and Meylahn, 2023; Meylahn, 2025) shows that while ADAD always exists as an equilibrium, GTGT and WSLSWSLS exist only for subsets of the (R,P)(R,P) space. Given a choice of θaθ^a, we therefore select R and P as the centroid of the region in which all three equilibria coexist. For θaθ^a, we simulate over the grid =Θα(m)×Θϵ(n)×Θδ(p)G= _α^(m)× _ε^(n)× _δ^(p), where Θα(m)=0.01,0.02,…,0.20 _α^(m)=\0.01,0.02,…,0.20\, Θϵ(n)=0.01,0.02,…,0.20 _ε^(n)=\0.01,0.02,…,0.20\, and Θδ(p)=0.65,0.70,…,0.85 _δ^(p)=\0.65,0.70,…,0.85\. Results The heatmaps in Figure 3 display ΔO(θ) O(θ) for all combinations of α∈Θα(m)α∈ _α^(m) and ϵ∈Θϵ(n)ε∈ _ε^(n) across different values of δ∈Θδ(p)δ∈ _δ^(p). Each heatmap also includes the isoquant ΔO(θ)=0.05 O(θ)=0.05. We observe that the boundary separating stationary from non-stationary regions is sharp, relatively smooth, and approximately convex. Figure 3: Heatmaps of ΔO(θ) O(θ). Each panel shows values of ΔO(θ) O(θ) across ten initialisations for all combinations of α and ϵε at a fixed value of δ. Regions with ΔO(θ)<0.05 O(θ)<0.05 are classified as stationary at time T=2×108T=2× 10^8. The boundary appears to be governed by a nonlinear relationship among learning, exploration, and discounting, consistent with a scaling of the form ϵc∼δα.ε^c δα. (5.2) For larger values of δ, such as δ=0.85δ=0.85, the boundary becomes less well defined, with a broad region in which trajectories resembling the intermediate case in Figure 2 are common. This observation motivates our choice of δ=0.85δ=0.85 as the upper bound for the discount factor in subsequent simulations. Excluding higher values of δ is further justified by the fact that the variance of Q-value estimates scales as (1−δ)−1(1-δ)^-1 (Devraj and Meyn, 2017), implying increasingly large fluctuations as δ approaches one. Similar effects have been documented for Q-learning in the prisoner’s dilemma (den Boer et al., 2024). Based on these results, we restrict α and ϵε to values greater than or equal to 0.10.1 in the main simulations analysing the heuristic boundary, while retaining the same values of δ. This restriction confines attention to regions of the parameter space in which stationarity is reliably attained, particularly for lower values of δ. For larger δ, the parameter combinations (α,ϵ)∈(0.1,0.1),(0.1,0.15),(0.15,0.1)(α,ε)∈\(0.1,0.1),(0.1,0.15),(0.15,0.1)\ lie close to the boundary between stationary and non-stationary regimes and are therefore treated as boundary cases. 5.3 Assessing the Quality of the Heuristic Boundary Parameter grid. To assess the quality of the heuristic boundary, we simulate over the grid =Θα(m)×Θϵ(n)×Θδ(p)×ΘP(q)×ΘR(s),G= _α^(m)× _ε^(n)× _δ^(p)× _P^(q)× _R^(s), where Θα(m)=Θϵ(n)=0.1,0.15,0.2 _α^(m)= _ε^(n)=\0.1,0.15,0.2\ and Θδ(p)=0.55,0.65,0.75,0.85 _δ^(p)=\0.55,0.65,0.75,0.85\. For the payoff parameters, we set ΘP(q)=0.025,0.05,…,0.5 _P^(q)=\0.025,0.05,…,0.5\ and ΘR(s)=0.525,0.55,…,0.975 _R^(s)=\0.525,0.55,…,0.975\, ensuring that the prisoner’s dilemma ordering T=1>R>P>S=0T=1>R>P>S=0 holds throughout. This yields 13,68013,680 parameter combinations. For each combination, we simulate ten trajectories using the initialisations described in Section 5.2, resulting in 136,000136,000 trajectories in total. We plot heatmaps of the average empirical occupation time at time ThT_h defined as O^(θ,)=1|ℐ|∑i∈ℐO(τTh,), O(θ, π)= 1|I| _i O( _T_h, π), (5.3) for the five focal strategy profiles. We normalise by O^(θ,Π^ϵ):=∑∈Π^ϵO^(θ,), O(θ, _ε):= _ π∈ _ε O(θ, π), (5.4) to make the occupation times comparable across different choices of θ. Figure 4: Heatmaps of average empirical occupation times for ADAD, GTGT, WSLSWSLS, AGTAGT, and ACAC at α=ϵ=0.1α=ε=0.1. Each heatmap shows the normalised average occupation time across initialisations for all combinations of P∈ΘP(q)P∈ _P^(q) and R∈ΘR(s)R∈ _R^(s), with rows corresponding to different values of δ∈Θδ(p)δ∈ _δ^(p). Occupation times are normalised by the total time spent in the five focal strategies. The green line denotes the heuristic boundary, while the yellow lines indicate the 10%10\%, 50%50\%, and 90%90\% isoquants of the normalised occupation shares. Figure 4 provides a visual assessment of the heuristic boundary for the case α=ϵ=0.1α=ε=0.1. Each heatmap reports the average empirical occupation time of one of the five focal strategy profiles across initialisations, for all combinations of P∈ΘP(q)P∈ _P^(q) and R∈ΘR(s)R∈ _R^(s), and for different values of δ∈Θδ(p)δ∈ _δ^(p). Occupation times are normalised by the total time spent in the five focal strategies, which allows for consistent comparison of their relative shares within the focal set across parameter configurations Across the parameter space, the strategy GTGT is practically absent, a pattern that also holds for other values of θaθ^a. Since the normalised share of cooperative strategies equals one minus the normalised share of non-cooperative strategies–and since ADAD accounts for the vast majority of non-cooperative play–the quality of the heuristic boundary can be assessed primarily by inspecting the heatmaps for ADAD. In the (R,P)(R,P) plane, the heuristic boundary is linear, whereas the empirical boundary inferred from occupation data is nonlinear and deviates increasingly from linearity as δ increases. This divergence coincides with approaching the non-stationary region of the parameter space. Nevertheless, the heuristic boundary captures the main qualitative features of the empirical transition and, in particular, shifts in the same direction as the empirical boundary when ϵε varies. The results for α=ϵ=0.1α=ε=0.1 shown in Figure 4 are representative of the general pattern observed across parameter combinations. Heatmaps for all remaining cases are reported in Appendix C. Evaluation Metric. To assess the quality of the heuristic boundary in Section 4, we evaluate its predictive performance using standard classification metrics. Interpreting the boundary as a decision rule, Equation 4.9 yields a prediction of whether cooperation or defection dominates for a given parameter configuration. We measure predictive performance using the macro F1-score, which jointly captures precision and recall, is invariant to the choice of the positive class, and is robust to class imbalance across regions of the parameter space. The macro F1-score takes values in [0,1][0,1], where a value of 11 corresponds to perfect classification. As a natural benchmark, a trivial classifier that predicts the same outcome for all parameter combinations–such as always predicting defection–achieves a macro F1-score of at most 0.50.5. More generally, values close to 0.50.5 indicate that the decision rule fails to meaningfully discriminate between cooperative and non-cooperative regimes, while values substantially above 0.50.5 reflect increasing discriminatory power. In this sense, macro F1-scores above 0.80.8 indicate strong separation between regimes, and scores approaching 11 indicate near-perfect alignment between the boundary and the empirically observed transition. If the heuristic boundary is informative, it should therefore achieve macro F1-scores well above this trivial baseline and outperform nearby perturbations of the decision threshold. In particular, relative to small perturbations of the boundary, the heuristic boundary should locally maximise the macro F1-score. To determine whether cooperation or defection dominates for a given parameter combination, we employ two complementary outcome-labelling criteria. The first criterion is based on time-averaged occupation measures over the five focal strategy profiles described in Section 5.1. The second criterion relies on time-averaged occupation measures over the four joint-action states induced by the actions available to the Q-learning algorithms. Strategy-profile approach Under the strategy–profile approach, we label an outcome as cooperative if the sum of normalised average occupation times across cooperative strategy profiles exceeds 50%50\%. The normalisation divides each profile’s raw occupation time by the total time spent in the five focal strategy profiles. We classify ACAC, WSLSWSLS, and AGTAGT as cooperative, and ADAD and GTGT as uncooperative. We classify GTGT as uncooperative because, in the presence of noise, its stationary distribution concentrates on mutual defection. If our proposed boundary constitutes a meaningful classifier, then using it as a decision rule to predict the dominance of cooperative strategy profiles should yield an F1 score close to one. Table 2 reports the macro F1-scores obtained for different combinations of the learning and exploration rates, pooling observations across the remaining dimensions of the parameter grid G. The table shows excellent and stable F1-scores across the accessible parameter range. In Appendix E, we report recall and precision scores under an adversarial approach that computes the worst-case recall and precision over definitions of the positive class. Even under this adversarial evaluation, the performance of the boundary remains excellent. Appendix E also shows how the performance of the boundary as a decision rule degrades when considering parameter combinations outside the accessible parameter range discussed in Section 5.2. Table 1: Sum of focal profiles ϵε α 0.1 0.15 0.2 0.1 0.901 0.837 0.814 (0.144) (0.200) (0.225) 0.15 0.851 0.810 0.792 (0.199) (0.235) (0.246) 0.2 0.825 0.791 0.771 (0.232) (0.254) (0.260) • Note: The table reports the mean of the sum of occupation times across the five focal strategy profiles for each combination of the learning and exploration rates. The mean is computed across all hyperparameter settings and payoff specifications within each learning–exploration rate combination. Values in parentheses indicate standard deviations. Table 2: F1 Scores ϵε α 0.1 0.15 0.2 0.1 0.918 0.924 0.920 [0.902, 0.933] [0.907, 0.940] [0.904, 0.936] 0.15 0.921 0.910 0.912 [0.904, 0.938] [0.891, 0.929] [0.893, 0.929] 0.2 0.908 0.908 0.914 [0.888, 0.927] [0.888, 0.926] [0.894, 0.931] • Note: The table reports the macro F1 score obtained when using the heuristic boundary as a classifier to predict the dominance of cooperative strategy profiles. Among the five focal strategy profiles, we classify ACAC, WSLSWSLS, and AGTAGT as cooperative, and ADAD and GTGT as uncooperative. The macro F1 score is computed using all hyperparameter settings and payoff specifications within each learning–exploration rate combination. Values in brackets report 95%95\% confidence intervals obtained via a bootstrapping procedure with 1,0001,000 bootstrap samples. Each panel of Figure 5 reports the macro F1-score obtained by applying the decision boundary to subsamples of the parameter grid corresponding to different algorithmic hyperparameters. Across all panels, the F1-score is consistently near 0.90.9. The macro F1-score is largely insensitive to the learning rate α and the exploration rate ϵε. By contrast, we observe a decline in performance as the discount factor increases. We attribute this effect to the findings discussed in Section 5.2, which suggest that higher discount factors push the dynamics toward regimes where the simulation horizon is insufficient for the theoretical framework to apply. Figure 5: Macro F1 scores across algorithmic hyperparameters. For each value of a given hyperparameter, we compute macro F1 scores by aggregating over all remaining hyperparameter combinations and payoff specifications. Error bars indicate 95%95\% confidence intervals obtained via a bootstrapping procedure with 1,0001,000 samples. Figure 6: Macro F1 scores under perturbations of the boundary in Equation (4.9) along different dimensions. Scores are aggregated over all hyperparameter settings and payoff specifications. Error bars indicate 95%95\% confidence intervals from 1,0001,000 bootstrap samples. Perturbations of the discount factor are truncated to [−1,1][-1,1]. Figure 6 illustrates how the macro F1 score varies as the decision boundary is perturbed along the R, P, and δ dimensions. For instance, the left panel replaces the boundary in R implied by Equation (4.9), RcR^c, with an alternative threshold R′R , and reports the resulting macro F1 score as a function of the Euclidean distance ∥R′−Rc∥ R -R^c . Across all dimensions, the decision boundary characterised by Equation (4.9) closely aligns with the empirical maximisers of the macro F1 score. Moreover, the macro F1 score decreases monotonically as the boundary deviates from this benchmark, indicating that the proposed boundary is locally optimal and sharply identified. State-space approach The state-space approach labels outcomes solely based on the frequency with which the learning dynamics visit the four joint-action states. We thus switch from focusing on O^(θ,) O(θ, π) to O^(θ,s) O(θ,s), which is defined analogously (see Equations (5.3)). This allows us to assess whether the quality of the decision boundary depends on the specific choice of focal strategy profiles or whether it generalises more broadly. Because state occupation times aggregate behaviour across all possible strategy profiles, this approach provides a robustness check that is not tied to the restricted strategy profile set. The left panel of Figure 7 shows that occupation of asymmetric states is most pronounced near the decision boundary and in regions where the boundary predicts cooperation. This pattern suggests that the prevalence of asymmetric profiles–and how they are treated in the definition of cooperation–can materially affect the performance of the proposed boundary. The right panel of Figure 7 shows that players split symmetrically between the roles of traitor and sucker. In the parameter regimes considered in our simulations, alternating between the sucker and temptation payoffs yields an average payoff exceeding the Nash equilibrium payoff. This observation provides a rationale for classifying time spent in asymmetric states as cooperative. Figure 7: Asymmetric state occupations. The left panel plots the average total occupation time in (C,D)(C,D) and (D,C)(D,C) as a function of the distance to the decision boundary along the R dimension, separately for each exploration rate considered in the simulations. The right panel plots, for each simulation run, the share of time spent in state (D,C)(D,C) against the share of time spent in (C,D)(C,D), illustrating how players equally split between the roles of traitor and sucker. Figure 8: Macro F1 scores across algorithmic hyperparameters. For each value of a given hyperparameter, macro F1 scores are computed by aggregating over all remaining hyperparameter combinations and payoff specifications. Error bars indicate 95%95\% confidence intervals obtained via a bootstrapping procedure with 1,0001,000 samples. Scores are reported for three outcome–labelling approaches: dominance of the (C,C)(C,C) state occupation (blue), dominance of normalised (C,C)(C,C) occupation (orange), and dominance of the (D,D)(D,D) state occupation (green). To avoid relying exclusively on a single definition of cooperation within the state-space approach, we compare three alternative outcome-labelling criteria. The first classifies an outcome as cooperative if the occupation time of state (D,D)(D,D) is below 50%50\%. Although this criterion may be criticised for implicitly treating asymmetric states as cooperative, it is appropriate when the primary objective is to predict whether mutual defection dominates the interaction. The second approach defines an outcome as cooperative if the occupation time of the state (C,C)(C,C) is at least 50%50\%. While this may appear to be the most natural definition, we anticipate that the prevalence of asymmetric profiles in the region where our boundary predicts cooperation will adversely affect the boundary’s performance under this criterion. Moreover, there is a more fundamental limitation to using (C,C)(C,C) dominance as the benchmark for successful cooperation: the stationary distribution induced by WSLSWSLS, the strategy profile motivating our heuristic boundary, assigns a non-negligible share of time to (D,D)(D,D). The third approach therefore defines an outcome as cooperative if the occupation time of (C,C)(C,C) exceeds 50%50\% of the occupation time that the stationary distribution induced by WSLSWSLS assigns to (C,C)(C,C). This definition more closely mirrors the theoretical intuition underlying our boundary, which is motivated by a forgiving cooperative strategy that systematically restores cooperation following unilateral deviations from mutual cooperation. Figure 8 reports the macro F1-scores obtained under three alternative outcome-labelling approaches when applying the decision boundary to subsamples of the parameter grid corresponding to different algorithmic hyperparameters. The boundary exhibits strong predictive performance when the objective is to identify the dominance of mutual defection, achieving scores of approximately 0.950.95.101010To visualise the quality of the boundary, we report heatmaps of (D,D)(D,D) state occupation in Appendix D. By contrast, performance deteriorates when predicting the dominance of mutual cooperation, with scores in the range of 0.60.6 to 0.70.7. However, when cooperation rates are normalised relative to the occupation times implied by the stationary distribution induced by WSLSWSLS, predictive performance improves substantially, with macro F1-scores increasing to between 0.70.7 and 0.90.9. This improvement relative to the non-normalised definition indicates that the apparent weakness of the boundary under stricter cooperation criteria primarily reflects a mismatch between the outcome metric and the underlying strategic dynamics, rather than a lack of discriminatory power of the boundary itself. The remaining performance gap between predicting mutual defection and normalised mutual cooperation reflects a substantive feature of the interaction dynamics: near and above the boundary, asymmetric strategy profiles account for a non-negligible share of the stationary distribution. As a consequence, cooperation criteria that discount asymmetric outcomes tend to conflate regions with qualitatively distinct but closely related dynamics. Taken together, these results suggest that the boundary is best interpreted as identifying the transition away from defection-dominated behaviour, rather than as a sharp predictor of full mutual cooperation. This interpretation is consistent with the theoretical motivation of the boundary and highlights the importance of outcome definitions that align with the underlying strategic dynamics. 6 Conclusion We have shown and investigated a new form of cooperation between reinforcement learning algorithms. While previous work has looked at the likelihood with which algorithms learn a cooperative strategy, here we instead consider the fraction of time that is spent playing cooperative strategies. This shift in perspective is motivated by the need for algorithms to continue exploring and learning when implemented in practice, due to the possibility of non-stationarities in the environment (or opponent’s behaviour). Our simulations show that cooperation can also be dominant in this new perspective. Furthermore, we identify a mechanism leading to this domination based on the relative stability of equilibrium strategy profiles and derive a theoretical boundary predicting when cooperation will dominate. This boundary does not take into account all factors relevant for stability, but nevertheless predicts cooperation surprisingly well. The phenomenon of spending a majority of the time in cooperative strategy profiles does not yet constitute algorithm collusion, as it would be necessary to argue that the firms are incentivised and likely to use the algorithm (cf. (Abada et al., 2024a; den Boer and Meylahn, 2024; den Boer et al., 2024)). Nevertheless, our results are relevant for the algorithmic collusion literature, as they suggest that algorithmic collusion may not only be a matter of learning collusive strategies, but may require analysis of the robustness of such learnt strategies to fluctuations arising from the environment or from the algorithms’ exploration. 7 Declaration of generative AI use During the preparation of this work, the authors used M365 Copilot and Claude in order to improve the language and phrasing as well as to assist in the coding of the experiments. After using this tool/service, the authors reviewed and edited the content as needed and take full responsibility for the content of the published article. References I. Abada, J. E. Harrington Jr, X. Lambin, and J. M. Meylahn (2024a) Algorithmic collusion: where are we and where should we be going?. Available at SSRN 4891033. Cited by: §6. I. Abada, X. Lambin, and N. Tchakarov (2024b) Collusion by mistake: does algorithmic sophistication drive supra-competitive profits?. European Journal of Operational Research. External Links: Document, Link Cited by: §2. I. Abada and X. Lambin (2023) Artificial intelligence: can seemingly collusive outcomes be avoided?. Management Science 69 (9), p. 5042–5065. Cited by: §2. J. Asker, C. Fershtman, and A. Pakes (2022) Artificial intelligence, algorithm design, and pricing. In AEA Papers and Proceedings, Vol. 112, p. 452–456. Cited by: §1, §1. J. Asker, C. Fershtman, and A. Pakes (2024) The impact of artificial intelligence design on pricing. Journal of Economics & Management Strategy 33 (2), p. 276–304. Cited by: §2. S. Assad, R. Clark, D. Ershov, and L. Xu (2024) Algorithmic pricing and competition: empirical evidence from the german retail gasoline market. Journal of Political Economy 132 (3), p. 723–771. Cited by: §1. M. Banchio and G. Mantegazza (2022) Artificial intelligence and spontaneous collusion. arXiv preprint arXiv:2202.05946. Cited by: §2. M. Banchio and G. Mantegazza (2023) Adaptive algorithms and collusion via coupling. In Proceedings of the 24th ACM Conference on Economics and Computation, EC ’23, New York, NY, USA, p. 208. External Links: ISBN 9798400701047, Link, Document Cited by: §2. W. Barfuss and J. M. Meylahn (2023) Intrinsic fluctuations of reinforcement learning promote cooperation. Scientific Reports 13 (1), p. 1309. External Links: Link, Document Cited by: §1, §2, §5.2. C. L. Beck and R. Srikant (2012) Error bounds for constant step-size Q-learning. Systems & Control Letters 61 (12), p. 1203–1208. Cited by: Appendix A, §1, §4. Q. Bertrand, J. A. Duque, E. Calvano, and G. Gidel (2025) Self-play Q-learners can provably collude in the iterated prisoner’s dilemma. In International Conference on Machine Learning, Cited by: §2. A. J. Bladon, T. Galla, and A. J. McKane (2010) Evolutionary dynamics, intrinsic noise, and cycles of cooperation. Physical Review E 81 (6), p. 066122. Cited by: §2. Z. Y. Brown and A. MacKay (2023) Competition in pricing algorithms. American Economic Journal: Microeconomics 15 (2), p. 109–156. Cited by: §1. E. Calvano, G. Calzolari, V. Denicolo, and S. Pastorello (2020) Artificial intelligence, algorithmic pricing, and collusion. American Economic Review 110 (10), p. 3267–3297. Cited by: §1, §1, §2. E. Calvano, G. Calzolari, V. Denicoló, and S. Pastorello (2021) Algorithmic collusion with imperfect monitoring. International journal of industrial organization 79, p. 102712. Cited by: §1. C. Carissimo, F. Falniowski, S. Rahimi, and H. Nax (2025) Algorithmic collusion is algorithm orchestration. arXiv preprint arXiv:2508.14766. Cited by: §2. A. V. den Boer and J. M. Meylahn (2024) A (mathematical) definition of algorithmic collusion. Available at SSRN 5012923. Cited by: §1, §6. A. V. den Boer, J. M. Meylahn, and M. P. Schinkel (2024) Artificial collusion: examining supracompetitive pricing by Q-learning algorithms. Available at SSRN 4213600. External Links: Document Cited by: §1, §2, §5.2, §6. A. M. Devraj and S. Meyn (2017) Zap Q-learning. Advances in Neural Information Processing Systems 30. Cited by: §5.2. A. Dolgopolov (2024) Reinforcement learning in a prisoner’s dilemma. Games and Economic Behavior 144, p. 84–103. External Links: Document, Link Cited by: §1, §2. A. Epivent and X. Lambin (2024) On algorithmic collusion and reward–punishment schemes. Economics Letters 237, p. 111661. Cited by: §2. M. I. Freidlin and A. D. Wentzell (2012) Random perturbations of dynamical systems. Springer Berlin Heidelberg, Berlin, Heidelberg. External Links: ISBN 978-3-642-25847-3, Document, Link Cited by: §4. T. Galla (2011) Cycles of cooperation and defection in imperfect learning. Journal of Statistical Mechanics: Theory and Experiment 2011 (08), p. P08007. External Links: Document, Link Cited by: §2. M. Hairer (2010) Convergence of markov processes. Lecture notes 18 (26), p. 11. Cited by: §4. J. E. Harrington (2018) Developing competition law for collusion by autonomous artificial agents. Journal of Competition Law & Economics 14 (3), p. 331–363. Cited by: §1. J. D. Hartline, S. Long, and C. Zhang (2024) Regulation of algorithmic collusion. In Proceedings of the 2024 Symposium on Computer Science and Law, p. 98–108. Cited by: §1. T. Klein (2021) Autonomous algorithmic collusion: q-learning under sequential pricing. The RAND Journal of Economics 52 (3), p. 538–558. Cited by: §1, §2. N. Kryloff and N. Bogoliouboff (1937) La théorie générale de la mesure dans son application À l’Étude des systémes dynamiques de la mécanique non linéaire. Annals of Mathematics 38 (1), p. 65–113. External Links: ISSN 0003486X, 19398980, Link Cited by: §4. H. J. Kushner and H. Huang (1981a) Asymptotic properties of stochastic approximations with constant coefficients. SIAM Journal on Control and Optimization 19 (1), p. 87–105. Cited by: §1. H. J. Kushner and H. Huang (1981b) Averaging methods for the asymptotic analysis of learning and adaptive systems, with small adjustment rate. SIAM Journal on Control and Optimization 19 (5), p. 635–650. Cited by: §1. X. Lambin (2024) Less than meets the eye: simultaneous experiments as a source of algorithmic seeming collusion. Available at SSRN 4498926. Cited by: §1, §1, §2. A. MacKay and S. N. Weinstein (2022) Dynamic pricing algorithms, consumer harm, and regulatory response. Wash. UL Rev. 100, p. 111. Cited by: §1. J. M. Meylahn and L. Janssen (2022) Limiting dynamics for Q-learning with memory one in symmetric two-player, two-action games. Complexity 2022, p. 1–20. External Links: Link, Document Cited by: §1, §2, §3. J. M. Meylahn (2023) Does an intermediate price facilitate algorithmic collusion?. Available at SSRN: 4594415. External Links: Link Cited by: §2. J. M. Meylahn (2025) Quantifying the likelihood of learning collusive strategy equilibria. Chaos: An Interdisciplinary Journal of Nonlinear Science 35 (8). Cited by: §2, §3, item 1, item 2, §5.2, footnote 7. S. P. Meyn and R. L. Tweedie (2012) Markov chains and stochastic stability. Springer Science & Business Media. Cited by: §4. Y. Usui and M. Ueda (2021) Symmetric equilibrium of multi-agent reinforcement learning in repeated prisoner’s dilemma. Applied Mathematics and Computation 409, p. 126370. External Links: Document Cited by: §1, §2, §3. L. Waltman and U. Kaymak (2008) Q-learning agents in a cournot oligopoly model. Journal of Economic Dynamics and Control 32 (10), p. 3275–3293. Cited by: §1. Z. Xu and W. Zhao (2024) On mechanism underlying algorithmic collusion. arXiv preprint arXiv:2409.01147. Cited by: §1, §2. Appendix A Proof of Theorem 4.1 Note that given any t Q_t, there is a positive probability of both players behaving as if they were playing any strategy in Πϵ _ε for a finite number of rounds K, say πiπ^i for player i. This can be quantified by using, for example, the Kullback-Leibler divergence, but a simple lower bound is given by ϵ2Kε^2K since both players explore with probability ϵε. In particular, a strategy profile π induces a stationary distribution on state-action pairs η(s,a)η π(s,a). Let κK(t)=t,…t+K _K(t)=\ a_t,… a_t+K\ (A.1) denote a sequence of play of length K. Then κK _K gives rise to an empirical distribution on state-action pairs η^(κK) η π( _K) and there exists a set of sequences of play of length K that minimize the total variation distance between the the empirical and true distributions κK=argminκK(t)dTV(η,η^(κK(t))). _K π= *arg\,min_ _K(t)d_TV(η π, η π( _K(t))). (A.2) Now at any time t, we have ℙ(κK(t)∈κK|t)≥ϵ2K. ( _K(t)∈ _K π| Q_t)≥ε^2K. This is the case even if the strategies of the players, as encoded by their Q-value orderings, change during these K rounds, since there always remains a probability of ϵ2ε^2 of the players taking any combination of actions. From the perspective of a single player, they are facing a stationary environment during these K rounds. For any π−iπ^-i the opponent could be behaving as, it is the case that there is a set of optimal Q-values Qπ−iQ^π^-i encoding the BR(π−i)BR(π^-i). This means that we can use the results on the finite-sample complexity of Q-learning with a constant learning rate (Beck and Srikant, 2012). In particular, Theorem 2.1 of Beck and Srikant (2012) states that, if the QtiQ_t^i remain bounded for all i and t, ℙ[|Qti−Qπ−i|∞≥x]≤1x((1−α(1−δ))K+32δ1−δα2−α),P[|Q_t^i-Q^π^-i|_∞≥ x]≤ 1x ((1-α(1-δ))^K+ 32δ1-δ α2-α ), (A.3) for any x>0x>0. Here, we have used that for us the iterates are bounded by Qti≤Qmax=1Q_t^i≤ Q_max=1 and Wmax=4Qmax2|×|=32W_max=4Q_max^2|S×A|=32. Given the parameters θ, we can calculate the minimum separation between optimal Q-values over all possible π−iπ^-i, which we define as D(θ)D(θ), i.e., D(θ)=mins∈,π∈Πϵ|Qπ(s,C)−Qπ(s,D)|. D(θ)= _s ,π∈ _ε|Q^π(s,C)-Q^π(s,D)|. (A.4) From Equation (A.3) we obtain that there is a positive probability of all Q-values of player i being within D(θ)/2D(θ)/2 of Qπ−iQ^π^-i within K rounds for some K and α<αcα<α^c, for some αcα^c. In particular, ℙ[|Qti−Qπ−i|∞<D(θ)2]≥1−2D(θ)((1−α(1−δ))K+32δ1−δα2−α).P [|Q_t^i-Q^π^-i|_∞< D(θ)2 ]≥ 1- 2D(θ) ((1-α(1-δ))^K+ 32δ1-δ α2-α ). (A.5) Thus the event |Qti−Qπ−i|∞<D(θ)2|Q_t^i-Q^π^-i|_∞< D(θ)2 occurs with positive probability if 2D(θ)((1−α(1−δ))K+32δ1−δα2−α)<1. 2D(θ) ((1-α(1-δ))^K+ 32δ1-δ α2-α )<1. (A.6) The term (1−α(1−δ))K(1-α(1-δ))^K can be made arbitrarily small by increasing K since α,δ∈(0,1)α,δ∈(0,1). This means that a large enough K to satisfy the inequality exists if 2D(θ)32δ1−δα2−α<1. 2D(θ) 32δ1-δ α2-α<1. (A.7) Solving for α, gives that such a K exists if α<αc:=2D2(θ)(1−δ)2D2(θ)(1−δ)2+4096δ2.α< _c:= 2D^2(θ)(1-δ)^2D^2(θ)(1-δ)^2+4096\,δ^2. (A.8) This brings the Q-values of a player close enough to the optimal Q-values to guarantee that they will have transitioned to the best-response strategy. Since this holds for both players, there is a positive probability that after K rounds the players are playing (BR(π−1),BR(π1))(BR(π^-1),BR(π^1)). We conclude that there is a positive probability of transitioning from any strategy profile to a strategy profile where both players are playing the best response to some strategy within K rounds when K is large enough. The set of strategy profiles Πa:=(π~1,π~−1):π~i∈BR(π) for some π∈Πϵ _a:=\( π^1, π^-1): π^i (π) for some π∈ _ε\ (A.9) is thus accessible from all π and thus forms part of a communicating class Πc _c if α<αcα< _c. To obtain that they thus form part of a unique recurrent set, note that if α<αcα< _c, we can pick K such that the probability ζ of hitting Πc _c is bounded away from zero for any state the process could be in at the beginning of the K time periods. Splitting the time horizon into non-overlapping blocks of length K and applying the result in each block, we obtain that the strategy profiles in Πa _a will be visited infinitely often. This completes the proof. Appendix B Details of heuristic boundary derivation In this appendix, we provide the details of the derivation of the critical boundary. By solving the Bellman optimality equation, assuming that both players play the AD strategy, we find that the optimal Q-values are QAD(s,D;θ)=12(ϵ(T−P)+2P), Q^AD(s,D;θ)= 12 (ε(T-P)+2P ), (B.1) and QAD(s,C)=12(2δP−δϵP+ϵR−δϵR+2S−2δS−ϵS+δϵS+δϵT), Q^AD(s,C)= 12 (2δ P-δε P+ε R-δε R+2S-2δ S-ε S+δε S+δε T ), (B.2) for all s∈s . Similarly, QWSLS(s,D;θ)= Q^WSLS(s,D;θ)= 14[4T+(2δ(1−ϵ)+ϵ)2P+δ(2−ϵ)(R−P)+δϵ(S−T)−2T] 14[4T+(2δ(1-ε)+ε)\2P+δ(2-ε)(R-P)+δε(S-T)-2T\] (B.3) and QWSLS(s,C;θ)=14[2(2−ϵ)R+2ϵS+δϵ(2−ϵ)P−2R+ϵ(R−S+T)] Q^WSLS(s,C;θ)= 14[2(2-ε)R+2ε S+δε\(2-ε)P-2R+ε(R-S+T)\] (B.4) for s∈(D,D),(C,C)s∈\(D,D),(C,C)\, and QWSLS(s,D;θ)=14[(2−δ(2−ϵ))(2−ϵ)P+2ϵT−δ(2−ϵ)ϵ(T−S)−(2−ϵ)R] Q^WSLS(s,D;θ)= 14[(2-δ(2-ε))(2-ε)P+2ε T-δ(2-ε)\ε(T-S)-(2-ε)R\] (B.5) and QWSLS(s,C;θ)=14[2ϵ(R−S)+4S+2δ2(1−ϵ)2R−(2−ϵ)P−ϵ(R−S+T) Q^WSLS(s,C;θ)= 14[2ε(R-S)+4S+2δ^2(1-ε)\2R-(2-ε)P-ε(R-S+T)\ +δ(2−ϵ)2P−4S+ϵ(2(S+T)−ϵ(R−S+T))] +δ\(2-ε)^2P-4S+ε(2(S+T)-ε(R-S+T))\] (B.6) for s∈(D,C),(C,D)s∈\(D,C),(C,D)\. Since the Q-values for the two actions are the same in all states when considering AD, we find that mins∈ΔQAD(s;θ)=12[(2−ϵ)P−2S+ϵ(S+T−R)]. _s Q^AD(s;θ)= 12[(2-ε)P-2S+ε(S+T-R)]. (B.7) For WSLS, the states are split into two pairs in terms of their Q-value difference so ΔQWSLS(s;θ)=R−T+ϵ2(T+S−(P+R))+δ2(1−ϵ)2R−(2−ϵ)P−ϵ(T+R−S) Q^WSLS(s;θ)=R-T+ ε2(T+S-(P+R))+ δ2(1-ε)\2R-(2-ε)P-ε(T+R-S)\ for s∈(D,D),(C,C)s∈\(D,D),(C,C)\ and ΔQWSLS(s;θ)=12[(1−δ(1−ϵ))(2−ϵ)P−2S+ϵ(T+S−R)+δ(1−ϵ)(2−ϵ)R−ϵ(T−S)] Q^WSLS(s;θ)= 12[(1-δ(1-ε))(2-ε)P-2S+ε(T+S-R)+δ(1-ε)\(2-ε)R-ε(T-S)\] for s∈(D,C),(C,D)s∈\(D,C),(C,D)\. Now we have ΔQWSLS((D,C);θ)−ΔQWSLS((D,D);θ)=T−R+P−S>0 Q^WSLS((D,C);θ)- Q^WSLS((D,D);θ)=T-R+P-S>0 (B.8) so that ΔQWSLS((D,D);θ)=ΔQWSLS((C,C);θ)<ΔQWSLS((D,C);θ)=ΔQWSLS((C,D);θ). Q^WSLS((D,D);θ)= Q^WSLS((C,C);θ)< Q^WSLS((D,C);θ)= Q^WSLS((C,D);θ). (B.9) Therefore mins∈ΔQWSLS(s;θ)=R−T+ϵ2(T+S−(P+R))+δ2(1−ϵ)2R−(2−ϵ)P−ϵ(T+R−S). _s Q^WSLS(s;θ)=R-T+ ε2(T+S-(P+R))+ δ2(1-ε)\2R-(2-ε)P-ε(T+R-S)\. (B.10) By equating Equation (B.7) and Equation (B.10) and solving for δ, we obtain the boundary. Appendix C Normalized All-Defection Heat Maps In this appendix, we present heatmaps of the average empirical occupation time of ADAD across all simulated pairs of the learning rate α and the exploration rate ϵε. These heatmaps visualize how frequently the interaction converges to persistent defection as algorithmic parameters vary. We focus on ADAD because, among the five focal strategy profiles, it is the only non-cooperative strategy that attains substantial occupation time in our simulations; by contrast, GTGT is almost never observed. As a result, occupation of ADAD serves as a clear and informative proxy for the failure of cooperation and provides a succinct evaluation of the effectiveness of the proposed decision boundary. Figure 9: Heatmaps of the average empirical occupation time of ADAD at α=0.1α=0.1. Each heatmap reports the normalized average occupation time, averaged across initializations, for all combinations of P∈ΘP(q)P∈ _P^(q) and R∈ΘR(s)R∈ _R^(s). Rows correspond to different values of δ∈Θδ(p)δ∈ _δ^(p), and columns correspond to different values of ϵ∈Θϵ(n)ε∈ _ε^(n). Raw occupation times are normalized by dividing by the total occupation time across the five focal strategy profiles (ADAD, GTGT, WSLSWSLS, AGTAGT, and ACAC). The green line indicates the heuristic boundary, while the yellow lines denote the 10%10\%, 50%50\%, and 90%90\% isoquants of the normalized occupation share. Regions with high ADAD occupation coincide closely with the boundary’s prediction of non-cooperative outcomes. Figure 10: Heatmaps of the average empirical occupation time of ADAD at α=0.15α=0.15. Each heatmap reports the normalized average occupation time, averaged across initializations, for all combinations of P∈ΘP(q)P∈ _P^(q) and R∈ΘR(s)R∈ _R^(s). Rows correspond to different values of δ∈Θδ(p)δ∈ _δ^(p), and columns correspond to different values of ϵ∈Θϵ(n)ε∈ _ε^(n). Raw occupation times are normalized by dividing by the total occupation time across the five focal strategy profiles (ADAD, GTGT, WSLSWSLS, AGTAGT, and ACAC). The green line indicates the heuristic boundary, while the yellow lines denote the 10%10\%, 50%50\%, and 90%90\% isoquants of the normalized occupation share. Regions with high ADAD occupation coincide closely with the boundary’s prediction of non-cooperative outcomes. Figure 11: Heatmaps of the average empirical occupation time of ADAD at α=0.2α=0.2. Each heatmap reports the normalized average occupation time, averaged across initializations, for all combinations of P∈ΘP(q)P∈ _P^(q) and R∈ΘR(s)R∈ _R^(s). Rows correspond to different values of δ∈Θδ(p)δ∈ _δ^(p), and columns correspond to different values of ϵ∈Θϵ(n)ε∈ _ε^(n). Raw occupation times are normalized by dividing by the total occupation time across the five focal strategy profiles (ADAD, GTGT, WSLSWSLS, AGTAGT, and ACAC). The green line indicates the heuristic boundary, while the yellow lines denote the 10%10\%, 50%50\%, and 90%90\% isoquants of the normalized occupation share. Regions with high ADAD occupation coincide closely with the boundary’s prediction of non-cooperative outcomes. Appendix D State Occupation Heat Maps for (D,D)(D,D) In this appendix, we present heatmaps of the average empirical occupation time of the state (D,D)(D,D) across all simulated combinations of the learning rate α and the exploration rate ϵε. These heatmaps illustrate how frequently the state (D,D)(D,D) is visited as the algorithmic parameters vary. In contrast to the analysis of strategy profiles, for which we report normalized shares, we use raw (unnormalized) shares in this case. Figure 12: Heatmaps of the average empirical occupation time of (D,D)(D,D) at α=0.1α=0.1. Each heatmap reports the normalized average occupation time, averaged across initializations, for all combinations of P∈ΘP(q)P∈ _P^(q) and R∈ΘR(s)R∈ _R^(s). Rows correspond to different values of δ∈Θδ(p)δ∈ _δ^(p), and columns correspond to different values of ϵ∈Θϵ(n)ε∈ _ε^(n). The green line indicates the heuristic boundary, while the yellow lines denote the 10%10\%, 50%50\%, and 90%90\% isoquants. Figure 13: Heatmaps of the average empirical occupation time of (D,D)(D,D) at α=0.15α=0.15. Each heatmap reports the normalized average occupation time, averaged across initializations, for all combinations of P∈ΘP(q)P∈ _P^(q) and R∈ΘR(s)R∈ _R^(s). Rows correspond to different values of δ∈Θδ(p)δ∈ _δ^(p), and columns correspond to different values of ϵ∈Θϵ(n)ε∈ _ε^(n). The green line indicates the heuristic boundary, while the yellow lines denote the 10%10\%, 50%50\%, and 90%90\% isoquants. Figure 14: Heatmaps of the average empirical occupation time of ADAD at α=0.2α=0.2. Each heatmap reports the normalized average occupation time, averaged across initializations, for all combinations of P∈ΘP(q)P∈ _P^(q) and R∈ΘR(s)R∈ _R^(s). Rows correspond to different values of δ∈Θδ(p)δ∈ _δ^(p), and columns correspond to different values of ϵ∈Θϵ(n)ε∈ _ε^(n). The green line indicates the heuristic boundary, while the yellow lines denote the 10%10\%, 50%50\%, and 90%90\% isoquants. Appendix E F1 Scores, Precision and Recall for Dominance of Cooperative Strategies This appendix reports the macro F1 score, minimum precision, and minimum recall across combinations of the learning and exploration rates. For completeness, we also present results for two combinations of the learning and exploration rates that lie outside the accessible parameter range identified in Section 5.2, namely α=ϵ=0.025α=ε=0.025 and α=ϵ=0.05α=ε=0.05. For these parameter combinations, we evaluate the classifier over the same values of the remaining free parameters as in the original grid G. Table 3 reports the macro F1 scores. These results replicate those in Table 2 while extending the analysis to the non-accessible parameter range. The table reveals a monotonic decline in the macro F1 score as the learning and exploration rates move further into the non-accessible region. A potential limitation of the macro F1 score is that it may mask poor performance in either recall or precision, depending on the definition of the positive class (dominance of cooperative versus defection strategy profiles). To address this concern, Tables 4 and 5 adopt an adversarial, worst-case evaluation approach. Specifically, for each parameter combination, we compute recall and precision under both possible definitions of the positive class and report the minimum value. Both metrics range from 0 to 11, with lower values indicating weaker performance. The results show strong performance across all metrics within the accessible parameter range, but substantially degraded performance outside this range. Table 3: F1 Scores ϵε α 0.025 0.05 0.1 0.15 0.2 0.025 0.819 – – – – [0.799, 0.838] – – – – 0.05 – 0.849 – – – – [0.831, 0.867] – – – 0.1 – – 0.918 0.924 0.920 – – [0.902, 0.933] [0.907, 0.940] [0.904, 0.936] 0.15 – – 0.921 0.910 0.912 – – [0.904, 0.938] [0.891, 0.929] [0.893, 0.929] 0.2 – – 0.908 0.908 0.914 – – [0.888, 0.927] [0.888, 0.926] [0.894, 0.931] • Note: The table reports the macro F1 score obtained when using the heuristic boundary as a classifier to predict the dominance of cooperative strategy profiles. Values in brackets indicate 95%95\% confidence intervals obtained via a bootstrapping procedure with 1,0001,000 bootstrap samples. Performance degrades for parameter combinations outside the accessible range (α=ϵ=0.025α=ε=0.025 and α=ϵ=0.05α=ε=0.05). For these combinations of the learning and exploration rates, the classifier is evaluated over the remaining free parameters of the original grid, Θδ(p)×ΘP(q)×ΘR(s) _δ^(p)× _P^(q)× _R^(s). Table 4: Minimum Precision Scores ϵε α 0.025 0.05 0.1 0.15 0.2 0.025 0.773 – – – – [0.755, 0.791] – – – – 0.05 – 0.817 – – – – [0.801, 0.834] – – – 0.1 – – 0.933 0.918 0.879 – – [0.922, 0.945] [0.889, 0.945] [0.847, 0.912] 0.15 – – 0.907 0.855 0.845 – – [0.875, 0.940] [0.818, 0.896] [0.807, 0.884] 0.2 – – 0.844 0.836 0.844 – – [0.799, 0.887] [0.794, 0.879] [0.801, 0.884] • Note: The table reports the minimum precision obtained when using the heuristic boundary as a classifier to predict the dominance of cooperative strategy profiles. Values in brackets indicate 95%95\% confidence intervals obtained via a bootstrapping procedure with 1,0001,000 bootstrap samples. With two classes, precision is computed for each class and the minimum value is reported. Table 5: Minimum Recall Scores ϵε α 0.025 0.05 0.1 0.15 0.2 0.025 0.636 – – – – [0.598, 0.672] – – – – 0.05 – 0.668 – – – – [0.632, 0.705] – – – 0.1 – – 0.805 0.850 0.872 – – [0.768, 0.842] [0.815, 0.886] [0.837, 0.904] 0.15 – – 0.843 0.852 0.869 – – [0.804, 0.881] [0.811, 0.893] [0.830, 0.908] 0.2 – – 0.848 0.854 0.866 – – [0.802, 0.893] [0.808, 0.895] [0.819, 0.908] • Note: The table reports the minimum recall obtained when using the heuristic boundary as a classifier to predict the dominance of cooperative strategy profiles. Values in brackets indicate 95%95\% confidence intervals obtained via a bootstrapping procedure with 1,0001,000 bootstrap samples. With two classes, recall is computed for each class and the minimum value is reported.