Paper deep dive
Modelling Opinion Dynamics at Scale with Deep MARL
Lukas Seier, Brandon Kaplowitz, Sebastian Towers, Richard Bailey, Jakob Foerster
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 7/9/2026, 3:00:34 AM
Summary
This paper introduces a GPU-accelerated multi-agent reinforcement learning (MARL) framework to model opinion dynamics at scale, supporting up to 1000 agents. By extending the other-play algorithm to general-sum interactions, the authors prevent unrealistic coordination conventions and enable zero-shot coordination. Validating the model on real-world network subsets (Bluesky, U.S. Congress X/Twitter, and Hadza hunter-gatherer networks), the study finds that high conformity matches human behavior but significantly reduces collective accuracy and promotes dishonesty in large social media networks. Conversely, in small, dynamic networks, conformity improves collective agreement. The findings suggest a mismatch between evolved human conformity heuristics and modern social media environments, potentially contributing to misinformation.
Entities (9)
Relation Signals (7)
High conformity → reduces → Collective accuracy in large social media networks
confidence 96% · In large social media networks such high levels of conformity significantly reduce collective accuracy and promote dishonest agents that lie to fit in.
Multi-Agent Reinforcement Learning → enables → Opinion dynamics modeling
confidence 95% · In contrast, multi-agent reinforcement learning (MARL) enables agents to learn such behaviours directly by optimising simple rewards.
High conformity → promotes → Dishonest agents in large social media networks
confidence 95% · In large social media networks such high levels of conformity significantly reduce collective accuracy and promote dishonest agents that lie to fit in.
High conformity → improves → Collective agreement in small dynamic networks
confidence 94% · By contrast, we find these effects to be reduced in small, dynamic networks, such as those observed in the Hadza hunter-gatherer tribe of northern Tanzania. Here, conformity can even improve collective agreement when communication is limited
Mismatch between evolved human conformity heuristics and modern social media → contributesto → Misinformation
confidence 93% · This suggests a mismatch between evolved human conformity heuristics and modern social media environments as a potential contributor to misinformation.
Other-play → extendedto → General-sum social interactions
confidence 92% · To prevent unrealistic conventions, we extend other-play to general-sum social interactions.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Modelling opinion dynamics typically relies on hand-crafted local interaction rules to study emergent macroscopic phenomena such as consensus and polarisation. In contrast, multi-agent reinforcement learning (MARL) enables agents to learn such behaviours directly by optimising simple rewards. To explore the potential of MARL for opinion dynamics, we introduce a GPU-accelerated consensus and truth-finding game that scales to populations of up to 1000 agents, comparable to many real-world social sub-networks. To prevent unrealistic conventions, we extend other-play to general-sum social interactions. We next validate our model on a subset of the Bluesky network by recovering agent importance structures from graph topology alone via a learned attention layer, finding that highly conforming populations most closely match human data. In large social media networks such high levels of conformity significantly reduce collective accuracy and promote dishonest agents that lie to fit in. By contrast, small, dynamic hunter-gatherer networks are less affected; here, conformity can even improve collective agreement. This suggests a mismatch between evolved human conformity heuristics and modern social media environments as a potential contributor to misinformation.
Tags
Links
- Source: https://arxiv.org/abs/2606.07487v1
- Canonical: https://arxiv.org/abs/2606.07487v1
Trouble viewing inline? Open PDF directly →
Full Text
82,029 characters extracted from source content.
Expand or collapse full text
Modelling Opinion Dynamics at Scale with Deep MARL Lukas Seier FLAIR University of Oxford United Kingdom lukas.seier@eng.ox.ac.uk Brandon Kaplowitz OWL University of Oxford United Kingdom Sebastian Towers FLAIR University of Oxford United Kingdom Richard Bailey OUCE University of Oxford United Kingdom Jakob Foerster FLAIR University of Oxford United Kingdom Abstract Modelling opinion dynamics typically relies on hand-crafted local interaction rules to study emergent macroscopic phenomena such as consensus and polari- sation. In contrast, multi-agent reinforcement learning (MARL) enables agents to learn such behaviours directly by optimising simple rewards. To explore the potential of MARL for opinion dynamics, we introduce a GPU-accelerated con- sensus and truth-finding game that scales to populations of up to 1000 agents, comparable to many real-world social sub-networks. To prevent unrealistic con- ventions, we extend other-play to general-sum social interactions. We next val- idate our model on a subset of the Bluesky network by recovering agent impor- tance structures from graph topology alone via a learned attention layer, find- ing that highly conforming populations most closely match human data. In large social media networks such high levels of conformity significantly reduce collective accuracy and promote dishonest agents that lie to fit in. By con- trast, small, dynamic hunter-gatherer networks are less affected; here, confor- mity can even improve collective agreement. This suggests a mismatch be- tween evolved human conformity heuristics and modern social media environ- ments as a potential contributor to misinformation. Our code is available at https://github.com/flipbagels/OpiniMARL. 1 Introduction In the face of global challenges, such as climate change and health pandemics, identifying the forces that promote or inhibit collective agreement and truth finding is critical to sustaining functioning societies [IPCC, 2023, Morens and Fauci, 2020]. Rapid growth of social media has popularised the study of opinion dynamics [Starnini et al., 2025], exploring mechanisms responsible for large-scale phenomena, such as consensus formation, polarisation and pluralistic ignorance [Baronchelli, 2018, Baumann et al., 2020, Centola et al., 2005]. Existing approaches typically rely on hand-crafted local interaction rules which, while remarkably successful [Castellano et al., 2009], limit the expressiveness of agents’ behaviours to those intended by the creator of the model. Multi-agent reinforcement learning (MARL) offers a promising al- ternative approach, by allowing agents to learn complex interactions from simple reward functions alone [Foerster et al., 2016, Leibo et al., 2017, Baker et al., 2019]. Preprint. arXiv:2606.07487v1 [cs.MA] 5 Jun 2026 Embedding Layers RNN PolicyValue Symmetry flip Symmetry flip Private Signal Population graph Self and neighbour actions Update output Belief Attention Layer Figure 1: Opinion update loop. Agents receive self and neighbouring guesses, which are subse- quently passed through an other-play symmetry operator and learned attention layer. The output is concatenated with agents’ private signals and passed into the main body of the architecture, after which the symmetry operation is reversed, yielding the updated guesses of the agents. However, scaling deep MARL methods to large population sizes is a significant computational chal- lenge [Aymanns et al., 2017, Guo et al., 2023]. We address these scaling issues by leveraging recent advances in end-to-end GPU-accelerated reinforcement learning (RL) with JAX [Bradbury et al., 2018, Rutherford et al., 2024], enabling training of up to 1000 agents for 10 6 training steps in 35 minutes on a single NVIDIA A100 GPU. Specifically, we train agents in our GPU-accelerated consensus and truth-finding game, in which they have mixed incentives to make guesses that both match a partially observed true state of the environment and conform to the guesses of their neighbours [Aymanns et al., 2017, Mohseni and Williams, 2021]. This captures a simple tension between individual evidence and social agreement, while remaining computationally tractable at large population sizes. We release our implementation as open-source software to support future work on scalable MARL-based opinion dynamics. To accurately model information and consensus dynamics on new unseen ground-truth states, we require that agents do not have a priori preferences over the underlying state. In particular, when agents optimise for conformity, they must not be able to coordinate on a shared initial guess before observing any information from their neighbours. This is a standard zero-shot coordination (ZSC) problem [Hu et al., 2020, 2021, Muglich et al., 2025], which we address by extending the other- play (OP) algorithm [Hu et al., 2020] to our general-sum environment. The connection between truth-finding and zero-shot coordination under conformity incentives is exploited in oracle consen- sus protocols [Sztorc, 2015, Risk Labs, 2020] but has not previously been explored in multi-agent learning, which could be relevant for large language model (LLM)-based multi-agent systems using agreement as a proxy for correctness [Du et al., 2024, Chen et al., 2024]. We validate our MARL framework on a subgraph of the Bluesky social network [Kleppmann et al., 2024] by recovering real-world node importance structures from graph topology alone via learned edge attentions [Liben-Nowell and Kleinberg, 2003]. Our method finds that agents acting with 80% conformity best match importance structures observed in human data. Importantly, at this level of conformity, social media networks yield significantly worse group accuracy and promote the emergence of dishonest actors that guess contrary to their private beliefs [Acemoglu et al., 2011, Asch, 1951]. We further show that this behaviour is more pronounced in clustered graphs that exhibit polarised states, such as the U.S. Congress X/Twitter network [Fink et al., 2023]. In contrast, we find these effects to be reduced in small, dynamic networks, such as those observed in the Hadza hunter- gatherer tribe of northern Tanzania [Fedurek et al., 2022, Apicella et al., 2012]. Here, conformity can even help to increase the number of accurate outputs when communication is limited, by helping uncertain agents make a guess when the penalty for being wrong exceeds the reward for being correct. Under the assumptions of our model, our findings suggest that the long-timescale evolution of human conformity [Henrich and Boyd, 1998, Cialdini and Goldstein, 2004, Deutsch and Gerard, 1955] offers advantages for collective truth finding and agreement in small hunter-gatherer networks, but becomes maladaptive in the context of modern social media platforms. 2 2 Related Work Wang et al. [2022] propose a deep learning approach to opinion dynamics via their consensus boost algorithm, but focus primarily on reward design for rapid consensus, whereas we aim to model hu- man behaviour using simple reward heuristics. Guo et al. [2023] use a bidirectional long short-term memory (LSTM) [Hochreiter and Schmidhuber, 1997] to learn state-conditioned neighbour weights within a linear DeGroot-style update [DeGroot, 1974], whereas our model adopts a fully neural network-based policy that supports non-linear opinion updates. Aymanns et al. [2017] use deep learning to study fake news propagation in environments with binary opinions and noisy private sig- nals of some ground-truth state. We build on this setting by introducing a conformity reward, a null action, and scaling to two orders of magnitude more agents. Mohseni and Williams [2021] simi- larly incorporate a conformity utility into a Bayesian agent model, but do not evaluate its magnitude against real-world data. 3 Background 3.1 Networked Partially Observable Stochastic Games (N-POSG) We model opinion dynamics as a networked partially observable stochastic game (N-POSG), given by the 10-tuple (n,G,S,A i n i=1 ,O i n i=1 ,T ,U i n i=1 ,R,H,γ). Here, n ∈ N is the number of agents,G is the space of permissible graphs andS is the state space. A i and O i are the local action and observation spaces for agent i, such that the joint action and observation spaces are defined byA : = Q n i=1 A i andO : = Q n i=1 O i , respectively. T andU define the dynamics of the system, s t+1 ∼T (s t+1 | s t ,a t ), o i t+1 ∼U i (o i t+1 | s t+1 ,a t ), where s ∈ S , a ∈ A and o i ∈ O i . Rewards are given by r i t+1 = R(s t+1 ,a i t ), H denotes the horizon, where s H is a terminal state, and γ ∈ [0, 1] is a discount factor. At each time step, a graph G t (N,E) ∈G is sampled with a vertex set N of size |N| = n and an edge set E consisting of l =|E| directed edges. We denote the out-neighbourhood of i as N G t (i) = j | (i,j) ∈ E, where (i,j) means from agent i to agent j, and the closed out-neighbourhood as N G t [i] = N G t (i)∪i. Each agent i samples a local action a i t from a policy π i (a i t |τ i t ), conditioned on their local action- observation history (AOH) τ i t = (o i 0 ,a i 0 ,...,o i t−1 ,a i t−1 ,o i t ). The joint policy is defined as π = (π i ,π −i ), where π −i = π j | j ̸= i, and samples actions a t conditioned on the joint AOH τ t = (τ 1 t ,...,τ n t ) with probability π(a t |τ t ) = Q n i=1 π i (a i t |τ i t ). The distribution of states for a joint policy π is given by ρ π (s), and the expected discounted return for each agent i is given by J i (π) = E π h P H−1 t=0 γ t r i t+1 i . 3.2 Zero-Shot Coordination One approach to learning N-POSGs is where each agent independently optimises its own policy π i∗ = arg max π i J i (π i ,π −i ) [Tampuu et al., 2017, Schroeder de Witt et al., 2020]. However, com- plex environments often give rise to multiple optimal equilibria, which might each rely on different arbitrary conventions. This is problematic in our model because some conventions lead to unre- alistic outcomes, such as conforming agents agreeing on the same initial guess to maximise their conformity reward, despite not having observed any signals from their neighbours. Hu et al. [2020] introduce ZSC in the context of a decentralised partially observable Markov deci- sion process (Dec-POMDP) [Bernstein et al., 2002], where agents are trained independently using the same learning algorithm (but different seed) and are then required to coordinate with previously unseen partners at test time. In particular, they introduce the OP algorithm, which exploits envi- ronmental symmetries by replacing the self-play (SP) objective J SP (π) = P n i=1 J i (π) with the OP objective J OP (π) = E φ i ∼Φ J SP (φ 1 (π 1 ),...,φ n (π n )) , where Φ is a set of environmental symme- tries. Given that our environment contains a symmetry in the binary choice of guesses, which leaves 3 the underlying N-POSG unchanged, we extend this objective to the general-sum setting such that each agent i maximises J i OP (π) = E φ i ∼Φ J i (φ 1 (π 1 ),...,φ n (π n )) . By applying these symmetry operations randomly to the policy, this prevents agents from arbitrarily preferring one guess over another unless there is an explicit advantage to doing so based on the local observations. 3.3 Opinion Dynamics and Information Aggregation The field of opinion dynamics aims to understand how exchanges of local opinions can give rise to emergent global states. Many foundational works use continuous variables as opinions to study dynamics with weighted averaging [French Jr, 1956, DeGroot, 1974], stubborn agents [Friedkin and Johnsen, 1990] and bounded confidence [Deffuant et al., 2000, Hegselmann and Krause, 2002], while others analyse discrete variable models analogous to spin states in statistical physics [Hol- ley and Liggett, 1975, Galam et al., 1982]. Recently, agent-based model (ABM) approaches have become popular [Epstein and Axtell, 1996, Flache et al., 2017], allowing for more complex local interaction behaviours. Another important class of models uses Bayesian agents to study information aggregation of some underlying state [Bikhchandani et al., 1992, Banerjee, 1992, Gale and Kariv, 2003, Acemoglu et al., 2011]. Such models have been used to show how polarised states can emerge from rational agents [O’Connor and Weatherall, 2018, Madsen et al., 2018] and how the addition of conformity utilities can affect consensus dynamics [Mohseni and Williams, 2021]. A third type of model looks at RL agents. Such models have been developed to demonstrate the emergence of social norms [Airiau et al., 2014, Yu et al., 2014, 2016]; discuss mechanisms for meta- stable polarised states via social reinforcement [Banisch and Olbrich, 2019, Meylahn and Meylahn, 2024]; and explore consensus reaching algorithms for optimal decision making [Shen et al., 2025, Wang et al., 2022]. Recently, some deep MARL approaches study social network dynamics with small population sizes [Aymanns et al., 2017, Guo et al., 2023]. 4 Methods 4.1 Our Consensus and Truth-Finding Game In this game, agents are tasked with the goal of identifying a binary ground-truth value via com- munication with neighbouring agents in a social network. Before an episode begins, a ground-truth Ψ ∈0, 1 is uniformly sampled. For all 0≤ t≤ H , we sample a graphG t (N,E(t)) with a set of agents N and a set of interactions E(t). This graph can be static or dynamic. The discrete action space of each agent isA i = 0, 1,∅, where 0 and 1 are guesses of Ψ and∅ is a null action. States are represented by tuples s t = (G t , Ψ,t,a t−1 ) and observations consist of two parts: • With probability p signal , o i (1) ∼N (Ψ,σ 2 ) is independently sampled from a Gaussian distribu- tion with mean Ψ and variance σ 2 for all i. Otherwise it is set to 0.5. o i (1) is held constant for all 0≤ t≤ H . • o i (2) (t) = (a j t−1 ) j∈N G t [i] , a tuple showing the previous actions of all neighbouring agents j, including i’s own action. At t = 0, we initialise a j −1 =∅ for all j. Combining these gives us o i t = (o i (1) ,o i (2) (t)). At each time step, agents receive a reward r i t+1 = αT i t + (1− α)C i t , where α∈ [0, 1], T i t = 0.1if a i t = Ψ, −0.2 if a i t = 1− Ψ, 0if a i t =∅, C i t = 1 |N G (i)| X j∈N G (i) 0.1if a i t = a j t ̸=∅, −0.2 if a i t /∈a j t ,∅, 0otherwise. Thus, T and C incentivise accuracy and conformity, respectively, with α controlling the weighting. When α = 0, agents are fully conforming and when α = 1, agents are fully truth-seeking. We use an asymmetric reward to incentivise the null action over uncertain guessing by ensuring the latter has a lower expected return. 4 Bluesky = . = Step 1Step 5Step 20Step 20 Congress X/Twitter = . = Hadza, r=5 = . = 01 ∅ Not lyingLying Figure 2: Episode dynamics. Example episodes for reward weighting α = 0.2, selected to demon- strate interesting outcomes. Top row: Bluesky network with many dishonest agents despite the population guessing correctly. Middle row: Congress network with a polarised state and a major- ity of lying agents in the left cluster, despite signalling incorrectly. Bottom row: Hadza network (r = 5) with no lying agents despite the population guessing incorrectly. In general, the Hadza network exhibits fewer lying agents than the social media networks. 4.2 Model Architecture A diagram of the model is shown in Figure 1. We pass the neighbour observations through an atten- tion layer [Vaswani et al., 2017], masked byG t to aggregate information according to learned atten- tion weights, adding learnable absolute positional embeddings of agent IDs to distinguish between individuals. We adopt the standard practice of parameter sharing [Gupta et al., 2017], including a third input in the form of a binary-encoded agent ID to allow for heterogeneity in agent policies. The private signal, o i (1) , attention-weighted, encoded past neighbour actions, attn(o i (2) (t)), and binary- encoded agent ID, bin(i), are concatenated, passed through two fully-connected embedding layers, a gated recurrent unit (GRU) (allowing for memory of prior states), two more fully-connected layers, and finally into an action distribution head, a belief head, and a value function head. All fully- connected layers and the GRU hidden state have a size of 50 and use layer normalisation. 4.3 Training Algorithm We train agents with OP and independent proximal policy optimisation (IPPO) [Schroeder de Witt et al., 2020], where each agent independently updates its policy via PPO [Schulman et al., 2017]. Updates are performed at the end of each trajectory to prevent feedback on Ψ during episodes, as agents will not know this value during evaluation. For the belief head, we freeze all weights except the belief layer and train with a supervised mean squared error (MSE) between the belief output and Ψ over the episode time steps. This is similar 5 to a truth-seeking agent (α = 1) that acts myopically (γ = 0) and tells us what an agent’s current belief of Ψ is from its hidden state. When computing attentions, we either evaluate the full attention matrix, masked by the graph adja- cency matrix, or calculate only the required attentions via a centralised edge list of sent and received signals. The first method scales as O(n) for a single agent and O(n 2 ) across the population, whereas the second always scales as O(l +n). When training large populations, l +n≪ n 2 , so we use a cen- tralised edge list. However, when training only a small subset of agents, for example if we want to test unilateral deviation of an agent for Nash equilibrium (NE) convergence (Appendix F),l+n≫ n, since graphs typically contain many more edges than agents. In this case, masking is more efficient. We therefore adopt the implementation that is optimal for the scenario being considered. The consensus and truth-finding game formalised in Section 4.1 contains two important environmen- tal symmetries, Φ =Id,φ ′ , where Id leaves the environment unchanged and φ ′ = (φ ′ S ,φ ′ A ,φ ′ O ), with φ ′ A (a i ) = 1− a i if a i ̸=∅, a i if a i =∅, φ ′ S (G t , Ψ,t,a t−1 ) = (G t , 1− Ψ,t,φ ′ A (a t−1 )), φ ′ O (o i (1) , (a j t−1 ) j∈N G t [i] ) = (1− o i (1) , (φ ′ A (a j t−1 )) j∈N G t [i] ). Note that φ ′ A and φ ′ O act element-wise onA andO, respectively. More intuitively, the relabelling of 0s and 1s is a symmetry of the environment, requiring us to also flip the private signal observation. At the start of every episode, we randomly assign a symmetry to each agent, applying this to its observation and action at each time step. Thus, agents cannot coordinate on an initial guess and can only reach an agreement after multiple rounds of signalling. Note that we use OP for both training and evaluation (see Appendix E for reasons why). When α = 0, a new symmetry appears, as the expected return is no longer conditioned on the underlying truth state. For example, we get an unwanted but OP-compatible convention, whereby agents learn to use their private signals as an anchor to coordinate towards or away from (since both give the same expected return). For this edge case, we simply set p signal = 0, so that agents have no information about the ground-truth to form this convention on. 4.4 Edge Weight Extraction and Empirical Validation Given the abstract nature of our model, comparison with real-world data is generally challenging. However, one option is to use edge weights of the social graph. Since attention weights depend both on neighbouring signals and agent IDs, there is no unique method for inferring edge weights in general. For example, we could take the expected weights under the state distribution ρ π (s), or set all actions equal to remove the dependence on neighbouring signals. We opt to approximate the expected weights with Monte Carlo trajectory sampling, though note that this choice has little effect on our results (Appendix H). A straightforward way to compare learned edge weights against empirical data is with simple simi- larity measures such as MSE or cosine similarity. However, these metrics are largely insensitive to distributional differences beyond the mean (Appendix K). We therefore analyse a node-level metric, which we term the perceived importance (PI), for which we can construct distributions conditioned on node in-degree for more meaningful comparisons. We define the PI centrality measure by the average importance received from an agent’s closed neighbourhood, where importance is calculated by multiplying the attention weighting by the neighbouring (or self) agent’s closed neighbourhood out-degree, given by C PI (i) = 1 |N G [i]| X j∈N G [i] |N G [j]|w ji .(1) This measure is designed such that, if every agent divides their attention uniformly, every agent’s C PI is exactly 1. A score greater than 1 means that, on average, neighbouring agents place more attention on agent i than on their other neighbours, with the reverse holding for a score of less than 1. Multiplying by the neighbours’ out-degrees removes a graph-induced bias that gives central nodes high scores under standard attention-based centrality measures, since we care about the perceived importance from the local perspective of neighbouring agents. For a more in-depth discussion on why we choose this centrality measure, see Appendix I. 6 05001000 Number of Agents 10 1 10 2 10 3 Wall Clock Time (min) 101× speedup Baseline (CPU) Baseline (JAX) Ours (JAX) 0.000.250.500.751.00 0.0 0.2 0.4 0.6 Final Fraction of Lying Agents Bluesky Congress X/Twitter Hadza, r=5 Hadza, r=10 No. Edges 30 300 3000 Figure 3: Left: Wall clock times for 10 6 training steps with 20 parallel environments. The JAX implementations trained on a single GPU scale orders of magnitude better than the baseline model trained on a CPU. The improved parameter efficiency of our model leads to additional performance gains. Right: Fraction of lying agents that output the opposite non-null guess to the output of their belief head. Social media promotes dishonest agents for conforming populations. 5 Experimental Details 5.1 Datasets Direct validation of network models on real data is challenging, as true underlying interaction pro- cesses are only partially observed. We generate a 1000-node dataset of the Bluesky machine learning community, with edge attentions computed from user-to-user likes, yielding a network whose inter- actions align with our modelling assumptions. We further study a 475-node X/Twitter network of the 117th U.S. Congress [Fink et al., 2023] and a 37-node GPS-derived proximity dataset of the Hadza hunter-gatherer tribe from northern Tanzania [Fedurek et al., 2022], where interactions are defined as spatial separations of less than two metres. (See Appendix B for more details.) 5.2 Modelling the Dynamics of the Hadza Network While the Bluesky and Congress networks remain static across trajectories, modelling the dynamic edges of the Hadza network introduces additional complexity. We model edge dynamics using an underlying Markov process, conditioned on steady-state interaction probabilities taken from the aforementioned GPS-derived dataset. This leaves a single free parameter, r, which controls the rate of switching between interacting and non-interacting states. To discretise the process for our model, we define an edge to exist at time t if at least one interaction occurred during the interval [t − 1,t]. Consequently, r can also be interpreted as controlling the effective time resolution of the model. Thus, larger values of r correspond to increasing the number of neighbour interactions before updating one’s guess. We study r values of 5 and 10 to see how different rates of interactions affect consensus dynamics. Full details of the Markov model can be found in Appendix B.3. 5.3 Controlling Signal Density Since the Bluesky and Congress X/Twitter networks have significantly higher edge densities than the Hadza network, agents receive proportionally more information from initial neighbouring guesses. To control for this, we introduce additional sparsity in the private signals by reducing p signal to match the overall neighbourhood signal density of the Hadza network with r = 10. For the Hadza network (with both r = 5 and r = 10), we keep p signal = 1. For the Bluesky and Congress X/Twitter networks, we set p signal to 0.14 and 0.07, respectively, decaying these values from unity during training for improved learning stability (Appendix O). 5.4 Implementation Details We use episodes of length H = 20 across 20 parallel environments. We vary reward weightings α ∈ [0, 1] in increments of 0.05 and set σ = 1.7 for the private signals. This provides signals that 7 0.000.250.500.751.00 20 30 40 50 W MARL Uniform Degree Eigenvector Jaccard Adamic-Adar 0 2 4 6 PI = .= . 050100150 Node In-Degree 0 2 4 6 PI = . 050100150 Node In-Degree Real 10 0 10 1 Node Count Figure 4: Validation of Bluesky network. Left: Sum of Wasserstein distances for each node in- degree. The dotted red line indicates the minimum of the MARL curve. Right grid: Hexagonal histograms showing PI distributions as a function of node in-degree, comparing MARL prediction at α = 0, 0.2, and 1 with the real data. α = 0.2 accurately captures the shape of the real data. are weak enough to encourage extended communication, while remaining sufficiently informative for learning. We train 10 random seeds, one of which collapses to the null action for multiple values of α. We evaluate the remaining 9 seeds on 200 episodes generated from an unseen random seed. All error bars indicate the standard error of the mean unless otherwise specified. 6 Results and Analysis 6.1 Scaling We train our model for 10 6 training steps on a single NVIDIA A100 GPU and compare wall clock times with the CPU-based baseline model from Aymanns et al. [2017] and a reimplementation of the baseline in a GPU-accelerated JAX pipeline, shown in Figure 3. For a 100-agent population, the JAX pipeline is 101× faster than the CPU baseline and the population scaling is orders of magni- tude more tractable. This enables us to train 1000 agents in 35 minutes. Our model is also more parameter efficient than the baseline, scaling with population size n at a rate of 8n compared to 768n (Appendix C). When n = 1000, the parameter counts are 35102 and 809858, respectively. 6.2 Agent Importance Prediction We validate our model against the Bluesky network dataset by comparing the distributions of PI scores for different node in-degrees. As the real data has no self-attentions, we remove the learned self-attentions from our model and renormalise for a fair comparison. Figure 4 shows the sum of the Wasserstein distances between predicted and empirical PI scores across node in-degree, denoted W sum . The learned weights are competitive with the strongest heuristic baselines (Appendix J), and best match the human data when α = 0.2, indicating that agents with moderate conformity most closely approximate humans in our dataset. Interestingly, the real data exhibits a spike in high PI scores among low in-degree nodes, which our model fails to capture. This effect is driven by high out-degree nodes distributing their attentions highly non-uniformly, thereby inflating the average scores of a subset of low in-degree nodes. Our model also struggles to predict structures in the Congress network, likely due to additional incentives that are not included in our model, such as political influence (Appendix L). However, we still include this graph in our analyses as an example of emergent dynamics on a clustered topology given our chosen incentives. 6.3 From Hunter-Gatherer to Social Media Figure 5 shows how the accuracy of a population depends on the reward weighting α and network structure. For comparison, we include an oracle which calculates the accuracy of a Bayes-optimal guess given⌊np signal ⌋ independently sampled private signals. This is equivalent to the best possible performance of a fully truth-seeking population under perfect communication. 8 0.00.51.0 0.0 0.2 0.4 0.6 0.8 1.0 Fraction Correct Bluesky 0.00.51.0 Congress X/Twitter 0.00.51.0 Hadza, r=5 0.00.51.0 Hadza, r=10 Step 1Step 6Step 20Oracle Figure 5: Fraction of agents whose output matches the ground-truth at various time steps for all reward weightings α on four distinct graphs. The black dashed line is an oracle that acts Bayes- optimally in isolation given⌊np signal ⌋ independently sampled private signals. The Hadza network with limited connectivity (r = 5) is the only graph for which finite conformity is optimal. In the Hadza network, we find that agents with finite conformity achieve marginally more correct guesses when communication is restricted (r = 5), particularly during the early stages of an episode. This occurs because the asymmetric reward structure favours the null action for fully truth-seeking agents under uncertainty, while conformity rewards can help to mitigate this effect by offering a reliable payoff for non-null actions. This suggests that conformity may be advantageous for collec- tive agreement in small, dynamic hunter–gatherer networks, where communication is restricted and decisions are time-sensitive. In contrast, conformity appears maladaptive in social media networks. The Bluesky and Congress networks exhibit sharp transitions in accuracy around α = 0.3 and α = 0.45, respectively, beyond which increased conformity yields worse accuracy than the Hadza networks. This is notable given that the Bluesky network receives roughly four times as many total private signals compared to the Hadza networks. One likely factor is increased coordination complexity for larger populations, caus- ing conformity-driven signals to overwhelm those conveying truth. Additionally, clustered graphs such as the Congress network become more susceptible to polarised states (see Figure 2). 6.4 Dishonesty in Conforming Populations Figure 3 (right) demonstrates how conforming populations promote the existence of dishonest agents, which act contrary to their beliefs in order to agree with neighbouring agents. This can arise when an agent’s posterior belief of the ground-truth exceeds the population’s accuracy (Ap- pendix N). Figure 2 provides an extreme example from the Congress network in which the majority of agents in the left cluster act dishonestly, despite their majority belief being correct. Notably, these results arise in the absence of malicious actors deliberately spreading misinformation, suggesting that misinformation can emerge even among agents with identical incentives. 7 Conclusions We develop a GPU-accelerated framework for modelling opinion dynamics on a consensus and truth-finding game at the scale of 1000 agents on a single GPU. We extend the OP algorithm to our general-sum setting, accurately modelling information diffusion by preventing the emergence of unrealistic conventions. Our model best predicts node-importance structure in a Bluesky network dataset for agents with strong conformity (α = 0.2). We show that high conformity provides benefits for collective agreement in hunter-gatherer networks, but becomes maladaptive in social media, suggesting an evolutionary mismatch that increases population-level distrust and susceptibility to misinformation. However, our model does not account for factors such as recommendation systems, bounded hu- man attention, influence-seeking behaviour, and heterogeneity in agent incentives. This presents a promising direction for future research in MARL for opinion dynamics modelling. 9 References Daron Acemoglu, Munther A Dahleh, Ilan Lobel, and Asuman Ozdaglar. Bayesian learning in social networks. The Review of Economic Studies, 78(4):1201–1236, 2011. Stéphane Airiau, Sandip Sen, and Daniel Villatoro. Emergence of conventions through social learn- ing: Heterogeneous learners in complex networks. Autonomous Agents and Multi-Agent Systems, 28(5):779–804, 2014. Coren L Apicella, Frank W Marlowe, James H Fowler, and Nicholas A Christakis. Social networks and cooperation in hunter-gatherers. Nature, 481(7382):497–501, 2012. Solomon E. Asch. Effects of group pressure upon the modification and distortion of judgments. In Harold Guetzkow, editor, Groups, leadership and men: Research in human relations, pages 177–190. Carnegie Press, 1951. Christoph Aymanns, Jakob Foerster, Co-Pierre Georg, and Matthias Weber. Fake news in social networks. arXiv preprint arXiv:1708.06233, 2017. Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi-agent autocurricula. In International Conference on Learning Representations, 2019. Abhijit V Banerjee. A simple model of herd behavior. The Quarterly Journal of Economics, 107 (3):797–817, 1992. Sven Banisch and Eckehard Olbrich. Opinion polarization by learning from social feedback. The Journal of Mathematical Sociology, 43(2):76–103, 2019. Andrea Baronchelli. The emergence of consensus: a primer. Royal Society Open Science, 5(2): 172189, 2018. Fabian Baumann, Philipp Lorenz-Spreen, Igor M Sokolov, and Michele Starnini. Modeling echo chambers and polarization dynamics in social networks. Physical Review Letters, 124(4):048301, 2020. Daniel S Bernstein, Robert Givan, Neil Immerman, and Shlomo Zilberstein. The complexity of decentralized control of Markov decision processes. Mathematics of Operations Research, 27(4): 819–840, 2002. Sushil Bikhchandani, David Hirshleifer, and Ivo Welch. A theory of fads, fashion, custom, and cultural change as informational cascades. Journal of Political Economy, 100(5):992–1026, 1992. Avrim Blum, MohammadTaghi Hajiaghayi, Katrina Ligett, and Aaron Roth. Regret minimization and the price of total anarchy. In Proceedings of the fortieth annual ACM Symposium on Theory of Computing, pages 373–382, 2008. James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http: //github.com/jax-ml/jax. Claudio Castellano, Santo Fortunato, and Vittorio Loreto. Statistical physics of social dynamics. Reviews of Modern Physics, 81(2):591–646, 2009. Damon Centola, Robb Willer, and Michael Macy. The emperor’s dilemma: A computational model of self-enforcing norms. American Journal of Sociology, 110(4):1009–1040, 2005. Arun G Chandrasekhar, Horacio Larreguy, and Juan Pablo Xandri. Testing models of social learning on networks: Evidence from a lab experiment in the field. Technical report, National Bureau of Economic Research, 2015. Justin Chen, Swarnadeep Saha, and Mohit Bansal. Reconcile: Round-table conference improves reasoning via consensus among diverse LLMs. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7066–7085, 2024. 10 Robert B Cialdini and Noah J Goldstein. Social influence: Compliance and conformity. Annu. Rev. Psychol., 55(1):591–621, 2004. Guillaume Deffuant, David Neau, Frederic Amblard, and Gérard Weisbuch. Mixing beliefs among interacting agents. Advances in Complex Systems, 3(01n04):87–98, 2000. Morris H DeGroot. Reaching a consensus. Journal of the American Statistical Association, 69(345): 118–121, 1974. Morton Deutsch and Harold B Gerard. A study of normative and informational social influences upon individual judgment. The Journal of Abnormal and Social Psychology, 51(3):629, 1955. Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improving fac- tuality and reasoning in language models through multiagent debate. In Forty-first International Conference on Machine Learning, 2024. Joshua M Epstein and Robert Axtell. Growing artificial societies: social science from the bottom up. Brookings Institution Press, 1996. Ido Erev and Alvin E Roth. Predicting how people play games: Reinforcement learning in ex- perimental games with unique, mixed strategy equilibria. American Economic Review, pages 848–881, 1998. Piotr Fedurek, Athena Aktipis, Lee Cronk, E Jerryson Makambi, Ibrahim Mabulla, J Colette Berbesque, and Julia Lehmann. Social status does not predict in-camp integration among egali- tarian hunter-gatherer men. Behavioral Ecology, 33(1):65–76, 2022. Christian G Fink, Kelly Fullin, Guillermo Gutierrez, Nathan Omodt, Sydney Zinnecker, Gina Sprint, and Sean McCulloch. A centrality measure for quantifying spread on weighted, directed networks. Physica A: Statistical Mechanics and its Applications, 626:129083, 2023. Andreas Flache, Michael Mäs, Thomas Feliciani, Edmund Chattoe-Brown, Guillaume Deffuant, Sylvie Huet, and Jan Lorenz. Models of social influence: Towards the next frontiers. JASSS: The Journal of Artificial Societies and Social Simulation, 20(4):2, 2017. Jakob Foerster, Ioannis Alexandros Assael, Nando De Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. In Advances in Neural Information Processing Systems, volume 29, pages 2145–2153, 2016. Food Research & Action Center. Twitter handles for members of the 117th congress. https:// frac.org/wp-content/uploads/MOC_Twitter-Handles_117th.pdf, December 2021. Ac- cessed: 12-09-2025. John RP French Jr. A formal theory of social power. Psychological Review, 63(3):181, 1956. Noah E Friedkin and Eugene C Johnsen. Social influence and opinions. Journal of Mathematical Sociology, 15(3-4):193–206, 1990. Serge Galam, Yuval Gefen, and Yonathan Shapir. Sociophysics: A new approach of sociological col- lective behaviour. I. mean-behaviour description of a strike. Journal of Mathematical Sociology, 9(1):1–13, 1982. Douglas Gale and Shachar Kariv. Bayesian learning in social networks. Games and Economic Behavior, 45(2):329–346, 2003. Shijun Guo, Haoran Xu, Guangqiang Xie, Di Wen, Yangru Huang, and Peixi Peng. Reinforcement learning-based consensus reaching in large-scale social networks. In International Conference on Neural Information Processing, pages 169–183. Springer, 2023. Jayesh K Gupta, Maxim Egorov, and Mykel Kochenderfer. Cooperative multi-agent control using deep reinforcement learning. In International Conference on Autonomous Agents and Multiagent Systems, pages 66–83. Springer, 2017. Rainer Hegselmann and Ulrich Krause. Opinion dynamics and bounded confidence models, analy- sis, and simulation. Journal of Artificial Societies and Social Simulation, 5(3), 2002. 11 Joe Henrich and Robert Boyd. The evolution of conformist transmission and the emergence of between-group differences. Evolution and Human Behavior, 19(4):215–241, 1998. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8): 1735–1780, 1997. Richard A Holley and Thomas M Liggett. Ergodic theorems for weakly interacting infinite systems and the voter model. The Annals of Probability, pages 643–663, 1975. Hengyuan Hu, Adam Lerer, Alex Peysakhovich, and Jakob Foerster. “Other-play" for zero-shot coordination. In International Conference on Machine Learning, pages 4399–4410. PMLR, 2020. Hengyuan Hu, Adam Lerer, Brandon Cui, Luis Pineda, Noam Brown, and Jakob Foerster. Off-belief learning. In International Conference on Machine Learning, pages 4369–4379. PMLR, 2021. IPCC. Climate Change 2023: Synthesis Report. Contribution of Working Groups I, I and I to the Sixth Assessment Report of the Intergovernmental Panel on Climate Change. IPCC, Geneva, Switzerland, 2023. doi: 10.59327/IPCC/AR6-9789291691647. Martin Kleppmann, Paul Frazee, Jake Gold, Jay Graber, Daniel Holmgren, Devin Ivy, Jeromy John- son, Bryan Newbold, and Jaz Volpert. Bluesky and the at protocol: Usable decentralized social media. In Proceedings of the ACM Conext-2024 Workshop on the Decentralization of the Internet, pages 1–7, 2024. Joel Z. Leibo, Vinicius Zambaldi, Marc Lanctot, Janusz Marecki, and Thore Graepel. Multi-agent reinforcement learning in sequential social dilemmas. In Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems, pages 464–473. International Foundation for Autonomous Agents and Multiagent Systems, 2017. David Liben-Nowell and Jon Kleinberg. The link prediction problem for social networks. In Pro- ceedings of the twelfth international conference on Information and knowledge management, pages 556–559, 2003. Jens Koed Madsen, Richard M Bailey, and Toby D Pilditch. Large networks of rational agents form persistent echo chambers. Scientific Reports, 8(1):12391, 2018. Richard D McKelvey and Thomas R Palfrey. Quantal response equilibria for normal form games. Games and Economic Behavior, 10(1):6–38, 1995. Benedikt V Meylahn and Janusz M Meylahn. How social reinforcement learning can lead to metastable polarisation and the voter model. PLOS ONE, 19(12):e0313951, 2024. Aydin Mohseni and Cole Randall Williams. Truth and conformity on networks. Erkenntnis, 86(6): 1509–1530, 2021. David M Morens and Anthony S Fauci. Emerging pandemic diseases: How we got to COVID-19. Cell, 182(5):1077–1092, 2020. Darius Muglich, Johannes Forkel, Elise van der Pol, and Jakob Foerster. Expected return symme- tries. In The Thirteenth International Conference on Learning Representations, 2025. Rosemarie Nagel. Unraveling in guessing games: an experimental study. The American Economic Review, 85(5):1313–1326, 1995. Cailin O’Connor and James Owen Weatherall. Scientific polarization. European Journal for Phi- losophy of Science, 8(3):855–875, 2018. Risk Labs. UMA data verification mechanism: Adding economic guarantees to blockchain oracles. Whitepaper, 2020. URL https://github.com/UMAprotocol/whitepaper/blob/master/ UMA-DVM-oracle-whitepaper.pdf. Accessed: 24-01-2026. 12 Alexander Rutherford, Benjamin Ellis, Matteo Gallici, Jonathan Cook, Andrei Lupu, Garðar In- gvarsson, Timon Willi, Ravi Hammond, Akbir Khan, Christian Schroeder de Witt, Alexandra Souly, Saptarashmi Bandyopadhyay, Mikayel Samvelyan, Minqi Jiang, Robert Tjarko Lange, Shimon Whiteson, Bruno Lacerda, Nick Hawes, Tim Rocktäschel, Chris Lu, and Jakob Nicolaus Foerster. JaxMARL: Multi-agent RL environments and algorithms in JAX. In The Thirty-eighth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. Christian Schroeder de Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip HS Torr, Mingfei Sun, and Shimon Whiteson. Is independent learning all you need in the StarCraft Multi-Agent Challenge? arXiv preprint arXiv:2011.09533, 2020. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. Yufeng Shen, Xueling Ma, Yukun Bao, Gang Kou, and Jianming Zhan. A consensus method based on reinforcement learning for group decision-making. European Journal of Operational Re- search, 2025. Michele Starnini, Fabian Baumann, Tobias Galla, David Garcia, Gerardo Iñiguez, Márton Karsai, Jan Lorenz, and Katarzyna Sznajd-Weron. Opinion dynamics: Statistical physics and beyond. arXiv preprint arXiv:2507.11521, 2025. Paul Sztorc. Truthcoin: Peer-to-peer oracle system and prediction marketplace. Whitepaper, 2015. URL http://w.truthcoin.info/papers/truthcoin-whitepaper.pdf. Accessed: 24- 01-2026. Ardi Tampuu, Tambet Matiisen, Dorian Kodelja, Ilya Kuzovkin, Kristjan Korjus, Juhan Aru, Jaan Aru, and Raul Vicente. Multiagent cooperation and competition with deep reinforcement learning. PLOS ONE, 12(4):e0172395, 2017. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems, volume 30. Curran Associates, Inc., 2017. Mingwei Wang, Decui Liang, and Zeshui Xu. Consensus achievement strategy of opinion dynamics based on deep reinforcement learning with time constraint. Journal of the Operational Research Society, 73(12):2741–2755, 2022. Chao Yu, Minjie Zhang, and Fenghui Ren. Collective learning for the emergence of social norms in networked multiagent systems. IEEE Transactions on Cybernetics, 44(12):2342–2355, 2014. Chao Yu, Guozhen Tan, Hongtao Lv, Zhen Wang, Jun Meng, Jianye Hao, and Fenghui Ren. Mod- elling adaptive learning behaviours for consensus formation in human societies. Scientific Re- ports, 6(1):27626, 2016. A Societal Impacts The goal of this paper is to advance the understanding of opinion dynamics in real-world social networks using MARL. This work has the potential for both positive and negative societal impacts. In particular, improved models of opinion formation can help identify mechanisms that contribute to distrust, polarisation, and the spread of misinformation, providing insight into potential intervention strategies. We also acknowledge the risk that results presented in this paper could be misinterpreted or taken out of context. For example, overly broad conclusions could be drawn about the behaviours or intentions of specific social or political actors. We emphasise that our model is intentionally simplified and does not capture many of the complex contextual and psychological factors that influence real human interactions. We have additionally anonymised the users in our generated Bluesky dataset. Overall, we believe that the potential benefits of this research outweigh the associated risks, provided the results are interpreted with appropriate caution. 13 B Datasets B.1 Bluesky Machine Learning Community We construct a dataset of the Bluesky machine learning community, with edge weights designed to reflect the behaviours relevant to our model. To construct the graph, we start with three anonymised users and do a breadth-first search (BFS) crawl, where for each next user we take the 50 most recent posts (or less if the user has less than 50) and find all the likers of these posts, capped at the 100 most recent likers for each post. From here, we sample 30 new next users in the BFS crawl, conditioned on having posted on the platform in the past 30 days. By constructing the graph in this way, we hope to capture the active members of the community and avoid including bot accounts, which we found to be included when generating the graph based on raw follows. After completing the BFS crawl, we perform an exhaustive search of all inter-account likes and calculate a “likes fraction" score for each directed edge, where this score is simply the number of likes user i sends to user j divided by the number of collected posts from user j. Finally, to get the attention weights we normalise the outgoing “likes fraction" scores for each agent, so that the outgoing attentions sum to 1. The final graph consists of 1000 nodes and 14559 directed and weighted edges. This dataset was created from the public Bluesky API on the 5th November 2025. B.2 U.S. Congress X/Twitter We use a dataset of the 117th U.S. Congress X/Twitter network consisting of interactions between members of the 117th U.S. Congress on X/Twitter between February 9, 2022 and June 9, 2022 [Fink et al., 2023]. The network consists of 475 nodes and 13289 directed and weighted edges. The network is mainly split into two distinct subgraphs, corresponding to members associated with either the Republican or Democratic parties (Figure 6), with fewer connections between opposing parties. Each weight associated with a directed edge from member i to j indicates the fraction of times that member i retweeted, quote retweeted, replied to, or mentioned j’s tweets, summed and then divided by the total number of tweets member j issued during the time frame to give an empirical probability of any of member j’s tweets being reacted to by i. Only members with at least 100 tweets in the time frame were included. Republican Democrat Independent Figure 6: 117th U.S. Congress X/Twitter network classified by party association. Username to party maps were parsed with [Food Research & Action Center, 2021]. B.3 Hadza Tribe We use a GPS-derived dataset of the Hadza tribe in northern Tanzania [Fedurek et al., 2022], con- sisting of values indicating the fraction of the day that individual members of the tribe spend within two metres of one another. Specifically, we opt for the “camp 2 – out of camp" dataset. As this net- work is naturally highly dynamic on a minute-to-minute basis, we use a Markov process to model interactions during each episode. 14 To derive the interaction dynamics, we start with the continuous time ordinary differential equation (ODE), d Pr(I t ) dt = a(1− Pr(I t ))− b Pr(I t ),(2) where a is the rate of switching from not interacting (I ′ ) to interacting (I), and b is the rate of switching from I to I ′ . Solving this first-order equation gives us, Pr(I t ) = Pr(I 0 )e −(a+b)t + a a + b 1− e −(a+b)t ,(3) which in the steady state limit yields q = lim t→∞ Pr(I t ) = a a + b .(4) If we start from an interacting state where Pr(I 0 ) = 1 or a non-interaction state where Pr(I 0 ) = 0 and let r = a + b, we get Pr(I t+∆t | I t ) = q + (1− q)e −r∆t ,(5) Pr(I t+∆t | I ′ t ) = q(1− e −r∆t ).(6) In order to discretise these solutions, we pose the question: given the state at t and t + ∆t, what is the probability of an interaction having occurred in the time interval ∆t? The cases where either the start or end state are interacting are trivial, giving a probability of 1. This leaves us with the non-trivial case where both the start and end state are non-interacting. We first find the probability of no single interaction Pr(I (t,t+∆t) | I ′ t ). If we consider a small δt = ∆t m then we can write Pr(I ′ (t,t+∆t) | I ′ t ) = 1− q 1− e −rδt m (7) = (1− q (1− 1 + rδt)) m (8) = 1− qr∆t m m (9) = e −rq∆t .(10) Then we have Pr(I (t,t+∆t) | I ′ t+∆t , I ′ t ) = Pr(I (t,t+∆t) ∩ I ′ t+∆t | I ′ t ) Pr(I ′ t+∆t | I ′ t ) (11) = Pr(I ′ t+∆t | I ′ t )− Pr(I ′ t+∆t ∩ I ′ (t,t+∆t) | I ′ t ) Pr(I ′ t+∆t | I ′ t ) (12) = 1− e −rq∆t 1− q (1− e −r∆t ) .(13) Since q is determined by the dataset, this leaves us with one free parameter r, which can equivalently be thought of as controlling the rate of switching or setting the time resolution of the discretisation. As we do not have the data to calibrate this to our model, we selected values of 5 and 10 to see how increasing the rate of interaction switching affects consensus dynamics. This gives an aver- age of 45.7 and 74.0 edges per time step or 1.24 and 2.00 average edges per time step per agent, respectively. C Scaling For a fair comparison between the IQL baseline and PPO algorithms, we train with the same number of parallel environments (20) and gradient updates. Additionally, we restricted the action space to two possible actions0, 1 to maintain consistency with the original IQL baseline implementation. However, the parameter counts are different, as our neural network architecture scales more effi- ciently than the baseline. With n as the number of agents, a the number of actions, d the attention 15 embedding dimension, f and g the feed-forward and GRU hidden size (f = g in our case), the parameter counts are given by, |θ ours | = dn + 3d 2 + 7d | z attention layer + 3fh + 3h 2 + 4h | z GRU + (1 + d +⌈log 2 n⌉)f + f 2 + 4f | z dense layers + h 2 + (a + 2)h + a | z policy head + (hf + 2f + h + 1) | z value head + (hf + 2f + h + 1) |z belief head = 3d 2 + 7d + 10h 2 + (d + a + 17)h + a + 2 + dn + h⌈log 2 n⌉, (14) and |θ baseline | = 3(a + 2)hn + 3h + 3h 2 + 4h | z GRU 1 + 6h 2 + 4h |z GRU 2 + h 2 + h |z dense + (ah + a) | z output = 10h 2 + (a + 12)h + a + 3h(a + 2)n. (15) Setting h = 50 and 64 for our model and the baseline model, respectively, d = 8, and a = 2, we obtain the simple expressions, |θ ours | = 26602 + 8n + 50⌈log 2 n⌉,(16) |θ baseline | = 41858 + 768n.(17) We clearly see that our model size scales slower with the number of agents, resulting in 23× fewer parameters when n = 1000. D Expressivity of Model 0 4 812 0 1 2 3 010203040 0 1 2 3 4 5 0 4 812 0 1 2 3 No. communities 123 010203040 0 1 2 3 4 5 Central node Central node neighbours Other Perceived Importance Node Degree SBMBA = . = . Figure 7: Perceived importance for an SBM and BA graph with 100 agents. Truth-seeking agents (α = 1) on the SBM graph place high importance on agents that bridge multiple clusters, while the conforming population (α = 0) on the BA graph finds an efficient coordination structure with information flowing from the central node. To demonstrate the expressivity of our model, we run our model on two generated graphs of size 100. The first graph is a stochastic block model (SBM) with five clusters and probabilities of within- cluster and inter-cluster edges of 0.33 and 0.005, respectively, and the second graph is a Barabási– Albert (BA) graph with attachment parameter m = 3. From Figure 7 we see that on the SBM graph, 16 truth-seeking agents (α = 1) learn to place proportionally greater importance on neighbours that bridge multiple clusters, as these nodes can effectively transfer information between clusters. We also see that conforming agents (α = 0) on a BA graph learn to form a highly efficient influence stream starting from the most central node and passing through its neighbours before reaching more distant neighbours. As the ground-truth value is not important to such agents, this allows the popula- tion to quickly coordinate and reach a consensus. Both examples demonstrate non-trivial emergent information sharing structures that would be challenging to know a priori when hand-crafting local agent behaviours as done in most previous literature. E Ablations Figure 8 demonstrates why we use OP for both training and evaluation in our model. In this example, training and evaluating without OP leads to the fairly conforming population forming a convention to always output 0 regardless of their private signal values. This only aligns with the ground-truth half of the time. This maximises the conformity component of the reward but is undesirable for modelling information diffusion. If we train with OP and evaluate without OP, as is usual for evaluating policy performance in the literature of ZSC, we find that agents can learn to use the symmetry flipping of OP to enforce stochasticity in their initial outputs if they do not use the null action, despite the underlying policy for initial non-null outputs actually collapsing to 0 (remember this output is then flipped by OP in training). Thus, there is an accidental convention to always converge to 0, which again has an accuracy of 50%. Finally, training and evaluating with OP gives rise to desired behaviours of information diffusion. Train: No OP Eval: No OP = . = Step 1Step 2Step 4Step 4 Train: OP Eval: No OP = . = Train: OP Eval: OP = . = 01 ∅ Not lyingLying Figure 8: Ablation examples for the Congress network. Top row: Population trained and evaluated without OP. Middle row: Population trained with OP and evaluated without OP. Bottom row: Population trained and evaluated with OP. Both training and evaluating with OP are required to prevent modelling with unrealistic conventions. 17 0.00.51.0 0.2 0.4 0.6 0.8 1.0 Fraction Correct Bluesky 0.00.51.0 Congress X/Twitter 0.00.51.0 Hadza, r=5 0.00.51.0 Hadza, r=10 Step 1Step 6Step 20Oracle Figure 9: Effect of ablating GRU on the fraction of agents whose output matches the ground-truth at various time steps for all reward weightings α on four distinct graphs. The black dashed line is an oracle that acts Bayes-optimally in isolation given⌊np signal ⌋ independently sampled private signals. Ablating the GRU decreases the performance of static graphs significantly. 0.00.51.0 0.0 0.2 0.4 0.6 0.8 1.0 Fraction Correct Bluesky 0.00.51.0 Congress X/Twitter 0.00.51.0 Hadza, r=5 0.00.51.0 Hadza, r=10 Step 1Step 6Step 20Oracle Figure 10: Effect of ablating agent IDs on the fraction of agents whose output matches the ground- truth at various time steps for all reward weightings α on four distinct graphs. The black dashed line is an oracle that acts Bayes-optimally in isolation given ⌊np signal ⌋ independently sampled private signals. Ablating agent IDs decreases the performance of static graphs significantly. Figures 9 and 10 show the effects of ablating the GRU component and agent IDs, respectively, on the accuracy of the population. We see that the performance is significantly reduced on static graphs such as the Bluesky and Congress/X graphs, but that the effect is less pronounced for the dynamic Hadza graphs. This highlights the importance of the time axis when repeatedly communicating with the same neighbours, and the benefits of knowing which neighbours are sending which signals. Agents on the Hadza network rely less on these aspects due to the stochastic nature of interactions with the rest of the population. F Testing for Nash Equilibrium To test for NE, we keep a frozen copy of parameters and train a single randomly selected agent on a newly initialised set of parameters. If the agent cannot improve its expected return, this indicates evidence of a NE. To fully test for NE, we would need to repeat this for every agent, which is computationally expensive for large populations. We thus assume that, given the homogeneity of our agents’ incentives, repeating this for three agents is sufficient. Figure 11 shows the average relative reward difference for three randomly selected agents before and after retraining in separate tests. We find that the average remains near unity, but increases by up to 20% for larger α. This 18 0.000.250.500.751.00 0.8 0.9 1.0 1.1 1.2 1.3 Relative Reward Difference Agent 10 Bluesky Congress X/Twitter Hadza, r=5 Hadza, r=10 0.000.250.500.751.00 Agent 20 0.000.250.500.751.00 Agent 30 Figure 11: Relative reward difference for three randomly selected agents retrained in separate tests to exploit unilateral deviation against a previously trained population. Reward gains of up to 20% for large α indicate that the model does not fully converge to NE. demonstrates that agents do not fully converge to a NE, though they do converge to stable policies (Appendix P). This is somewhat unsurprising given that even a more constrained form of multi-agent learning known as regret minimisation is only guaranteed to reach a coarse correlated equilibrium (CCE) [Blum et al., 2008]. It is also debated whether human behaviour can be reliably predicted by NE [Nagel, 1995, McKelvey and Palfrey, 1995], while reinforcement learning dynamics have been shown to provide more accurate predictions [Erev and Roth, 1998]. Thus, these results do not invalidate our model. G Interpreting Learned Policies 0.00 0.02 0.04 0.06 0.08 = Bluesky = Congress X/Twitter = 0.2 = Hadza, r=5 −3.5 −1.5 0.52.5 4.5 0.00 0.02 0.04 0.06 0.08 = −3.5 −1.5 0.52.5 4.5 = −3.5 −1.5 0.52.5 4.5 = 1.0 = Private Signal Count Density Initial Output 01 ∅ Figure 12: Histograms showing the distribution of initial outputs for agents receiving a private signal in a selected episode on three distinct graphs. Top row: α = 0.2. Bottom row: α = 1.0. Truth- seeking agents (α = 1.0) in the social media graphs develop a strategy to withhold their output on the first turn when unsure of their private signal. 19 Since the policies learned by agents are complex, it is difficult to gain a detailed understanding of agents’ actions at every time step. However, we can still interpret some of the learned policies by analysing the actions at the initial time step. Figure 12 shows how the initial output frequencies differ between a highly conforming population (α = 0.2) and a fully truth-seeking population (α = 1) for agents that receive a private signal. We see that in the social media graphs, truth-seeking agents withhold their guesses on the first time step when uncertain of their beliefs (i.e. when their private signals are near 0.5), while conforming agents are not so strict. Interestingly, this region of withholding does not emerge in the Hadza network, which is likely caused by the small population size and limited communication increasing the value of each agent’s individual guess. To understand the behaviour of agents without a private signal, we can analyse the distribution of initial outputs as a function of α, as seen in Figure 13. For the social media networks, which are the only graphs that have a fraction of agents without private signals, we see that the number of non-null guesses increases for α < 0.5, indicating that agents care more about fast conformity than waiting for rare signals from agents that receive private signals of the truth, Ψ. For agents that do receive a private signal, we see that the larger fraction of non-null initial outputs consistently aligns with Ψ for all α, except for the Hadza network, which by chance had more incorrect guesses. This is expected since, on average, more agents will receive a signal that aligns their posterior beliefs with the true value Ψ. 0.00 0.05 0.10 = Bluesky 0.00 0.02 0.04 0.06 = Congress X/Twitter 0.00 0.25 0.50 0.75 1.00 Signal = Hadza, r=5 0.00.51.0 0.00 0.25 0.50 0.75 1.00 = 0.00.51.0 0.00 0.25 0.50 0.75 1.00 = 0.00.51.0 0.00 0.25 0.50 0.75 1.00 No Signal = Fraction of Population Initial Output 01 ∅ Figure 13: Area charts showing fractional distribution of initial outputs for all values of α. Top row: Agents receiving a private signal. Bottom row: Agents not receiving a private signal. Conforming populations lead to an increase in outputs from agents not receiving private signals. H Extracting Attention Weights Since a t−1 is contained in s t ,U depends only on s, and we can approximate the expected attention via Monte Carlo sampling of episode trajectories, ̄w ij = E s∼ρ π E o i ∼U i (·|s) w ij (o i ) ≈ 1 M M X m=1 w ij (o i,(m) ).(18) We need to be confident that we are taking enough samples for a good estimate, and thus analyse convergence of the total variation (TV) distance over attention distributions for each agent, given by TV i = 1 2 X j∈N [i] ̄w ij − ̄w ∗ ij ,(19) 20 0500100015002000 Number of Samples M 10 3 10 2 10 1 Total Variational Distance (MC Sampling vs Proxy Average) Median Error Maximum Error 5th - 95th Percentile 0.000.250.500.751.00 0.00 0.02 0.04 0.06 0.08 0.10 Total Variation Distance (Same Action vs MC Sampling) Median TV Distance Worst-Case TV Distance 5th - 95th Percentile Figure 14: Left: TV distance for increasing number of MC samples against a proxy average with 10 4 samples. Right: TV distance for different α between MC sampling vs same action method for extracting attention weights from the model. where ̄w ∗ ij are proxy true weights taken with M = 10 4 . Figure 14 (left) demonstrates that the TV distance converges to within 1% for all agents when M > 500. Our evaluations are taken with 4000 steps (200 episodes), hence we can be confident that our calculated attention weights are an accurate representation of ̄w ∗ ij . Monte Carlo sampling is not the only way in which we could extract attention weights. For example, we could also apply a simple heuristic by setting all previous actions equal and observing the effects of the agent IDs when all agents are in agreement. Figure 14 (right) shows the TV distance between weights calculated via Monte Carlo sampling vs the simple heuristic of equal actions. We see that the median TV distance lies around 1% with the worst case around 4%. The choice of method for extracting attention weights does not significantly impact any of our results. I Centrality Measures Two common attention-based centrality measures used in the context of graphs include the sum over received attentions and the average received attention, given by C Σ (i) = X j∈N G [i] w ji (20) and C μ (i) = 1 |N G [i]| X j∈N G [i] w ji ,(21) respectively. Both measures have a natural bias to giving high degree nodes large centrality. For example, imagine we have a star graph with 100 nodes m i for 1 ≤ i ≤ 100 where a single central node m 1 has 99 edges, one connecting directly to every other node. If all edges have uniform attention weights, all peripheral nodes give a weighting of 0.5 to themselves and to the central node, while the central node gives a weighting of 0.01 to itself and all peripheral nodes. For the central node this gives centrality scores of C Σ (1) = 49.51 and C μ (1) = 0.4951, while for each peripheral node 1 < j ≤ 100 we get scores of C Σ (j) = 0.51 and C μ (j) = 0.255. While such a bias might be desirable in determining which nodes might have the greatest influence over the network, we are more interested in the perceived importance at the individual agent level. If all agents uniformly weight their closed neighbourhood, then all agents are perceived as equally important and so should receive the same perceived importance score. We do this by multiplying a given score by the corresponding agent’s out-degree, so that a score of 2 means that this agent perceives the selected neighbour to be twice as important relative to uniform attentions. We can then 21 safely average over the received importance scores to arrive at C PI (i) = 1 |N G [i]| X j∈N G [i] |N G [j]|w ji ,(22) which is Equation (1) in the main text. If we repeat the example of the star graph with uniform weights we see that we get desired scores of C PI (i) = 1 for all 0≤ i≤ 100. J Edge Weight Baselines We use five heuristic baselines for predicting edge weights used by Chandrasekhar et al. [2015] in the context of DeGroot learning [DeGroot, 1974] and Liben-Nowell and Kleinberg [2003] in the context of link prediction. These include uniform weighting, degree weighting, eigenvector weighting, Jaccard and Adamic-Adar. We use A ′ = A + 1 to denote the adjacency matrix with self-loops on all nodes. Uniform weighting. Here an agent places all weighting uniformly among its neighbours and itself. The weight matrix is given by w U ij = A ′ ij d out i , where d out i is agent i’s out-degree (including itself). Degree weighting. Here each agent places a weighting proportional to their neighbours’ (and self) popularity. This is given by w D ij = d in j P k∈N G [i] d in k , where d in i is agent i’s in-degree (including itself). Eigenvector weighting. Here the weighting is determined by the eigenvector centrality, determined by the left eigenvector of the adjacency matrix with the largest eigenvalue. More formally this is given by w E ij = ξ j P k∈N G [i] ξ k , where ξ i satisfies ξ i A ′ ij = λ ∗ ξ j and λ ∗ is the largest eigenvalue. Jaccard. Here the weighting between two agents is determined by the Jaccard coefficient, w J ij = |N G (i)∩ N G (j)| |N G (i)∪ N G (j)| , which weights agents with a similar out-neighbourhood more highly. Adamic-Adar. Here the edges between two agents are weighted similarly to Jaccard, but we divide by the popularity of the joint neighbours instead of the union of the neighbourhoods. This is given by w A ij = X k∈N G (i)∩N G (j) 1 logd in k , where in this case the in-degree does not include itself. We note that the uniform, degree and eigenvector baseline weight matrices are derived using an adjacency matrix with self-loops on all nodes to match how the MARL model allows agents to observe their own previous actions. However, just as we do for the learned MARL weights, we remove the diagonal (self weights) and renormalise the rows for fair comparison with the real data, which do not contain self-loops. K Validation Metrics Similarity metrics such as MSE and cosine similarity offer a simple way of comparing edge weights between two graphs. However, they fail to correctly capture distributional differences beyond the 22 0.00.51.0 0.5 1.0 1.5 2.0 Edge MSE ×10 2 0.00.51.0 1.0 1.2 1.4 1.6 PI MSE ×10 3 0.00.51.0 2 3 4 5 PI W ×10 1 MARLUniformDegreeEigenvectorJaccardAdamic-Adar Figure 15: Three validation metrics, from left to right: edge-level MSE, node-level perceived im- portance MSE, PI Wasserstein summation over node in-degree. Uniform weights score highly on basic statistics (MSE) but worse in the W sum metric as they fail to capture the distribution of node importances. Node In-Degree 0 2 4 6 8 PI Uniform Node In-Degree Degree Node In-Degree Eigenvector Node In-Degree Jaccard 0100 Node In-Degree 0 2 4 6 8 PI Adamic-Adar 0100 Node In-Degree = . MARL 0100 Node In-Degree Real 10 0 10 1 10 2 Node Count Figure 16: Hexagonal histograms for the Bluesky network showing PI distributions as a function of node in-degree. Top row: uniform weighting, degree weighting, eigenvector weighting and Jaccard. Bottom row: Adamic-Adar, MARL predictions for α = 0.2 and the real data. The distribution generated by the MARL simulation is visually more accurate than the baselines. mean. For example, in the case of the Bluesky dataset, uniform weights score highly on MSE based metrics (Figure 15), since the average PI is roughly uniform (Figure 16). However, from Figure 16 we see that the uniform weights do not account for the distribution in PI scores observed in the real data. We thus calculate the Wasserstein distance for each node in-degree and sum to obtain a scalar metric, W sum , that better captures the distribution. We see that this metric results in the Jaccard and MARL weights scoring better than the uniform weights (Figure 15), as they better capture the width of the distribution (Figure 16). 23 0.000.250.500.751.00 15 20 25 30 35 W MARL Uniform Degree Eigenvector Jaccard Adamic-Adar 0 1 2 PI = .= . 050100 Node In-Degree 0 1 2 PI = . 050100 Node In-Degree Real 10 0 2 × 10 0 3 × 10 0 4 × 10 0 6 × 10 0 Node Count Figure 17: Validation of Congress network. Left. Sum of Wasserstein distances for each node in- degree. Right grid: Hexagonal histograms showing PI distributions as a function of node in-degree, comparing MARL prediction atα = 0, 0.2, and 1 with the real data. The MARL simulations struggle to capture the full shape of the real data. Node In-Degree 0 1 2 3 4 5 PI Uniform Node In-Degree Degree Node In-Degree Eigenvector Node In-Degree Jaccard 050100 Node In-Degree 0 1 2 3 4 5 PI Adamic-Adar 050100 Node In-Degree = . MARL 050100 Node In-Degree Real 10 0 10 1 Node Count Figure 18: Hexagonal histograms for the Congress X/Twitter network showing PI distributions as a function of node in-degree. Top row: uniform weighting, degree weighting, eigenvector weighting and Jaccard. Bottom row: Adamic-Adar, MARL predictions for α = 0.2 and the real data. None of the predicted distributions visually fit the real data well. L Validating the Congress X/Twitter Network While our model seems to predict PI structures fairly well for the Bluesky dataset, it cannot predict edge weights for any arbitrary graph, particularly when actors behave with complex incentives. Figures 17 and 18 show the PI fit for the empirical Congress X/Twitter dataset. We see that the learned MARL weights do not capture the distribution accurately. One reason for this is that real humans do not just act according to conformity and truth-seeking incentives, especially politicians. They tend to have additional motives such as political influence and reputation. Our model is not designed to capture such behaviours. We choose to include this graph in our analyses as a good example of emergent dynamics on a clustered topology. 24 M Notions of Accuracy In Section 6.3 we discuss how the accuracy of a population depends on reward weighting α and graph topology. We use the number of correct outputs compared to the total population as a notion of accuracy. However, to fully understand the results we can consider other notions of accuracy. For example, another accuracy measure might consider only agents that have output a valid guess (i.e. not ∅). In such a case we find that the Hadza tribe with r = 5 no longer scores higher for finite conformity, as shown in Figure 19. If we are in the setting of a vote, where agents that remain silent do not count, then this might be a more appropriate measure of accuracy. In the main text, we consider it important that every agent provides an output and that this output is correct (for truth- seekers), hence why conformity leading to more total outputs in the Hadza network with r = 5 is considered an improvement. 0.00.51.0 0.5 0.6 0.7 0.8 0.9 1.0 Fraction Correct Bluesky 0.00.51.0 Congress X/Twitter 0.00.51.0 Hadza, r=5 0.00.51.0 Hadza, r=10 Step 1Step 6Step 20Oracle Figure 19: Fraction of agents among the non-null outputting subpopulation whose output matches Ψ at various time steps for different α on four distinct graphs. The black dashed line is an oracle that acts Bayes-optimally in isolation given⌊np signal ⌋ independently sampled private signals. Under this alternative accuracy definition, finite conformity is no longer advantageous in the Hadza network with r = 5. N Validating the Belief Head 0.00.51.0 −2 0 2 4 Private Signal Disagreement Score Bluesky 0.00.51.0 Congress X/Twitter 0.00.51.0 Hadza, r=5 0.00.51.0 Hadza, r=10 Figure 20: Average private signal disagreement scores for lying agents receiving private signals for different α on four distinct graphs. When α > 0.5, we get very few lying agents (Section 6.4), so the belief head is valid without needing to look at private signal disagreement (PSD) scores. For 0 < α < 0.5, we find private signal disagreement (PSD) scores to be largely positive, indicating that the belief head correctly opposes the population guess when an agent’s private signal induces a strongly opposing posterior. When α = 0, the private signal disagreement (PSD) scores drop to zero since agents do not receive private signals for this value of α. 25 In order to draw conclusions about dishonest agents, we need to ensure that the belief head has been trained sufficiently. However, since the population accuracy can be quite low for mostly conforming agents (α < 0.5), the belief loss will be noisy, even if it gives a best possible prediction of the underlying truth state from the agent’s hidden state. We verify whether the belief output behaves as expected using a private signal disagreement (PSD) score, PSD(i) = o i (1) − 0.5− a maj H−1 (2o i (1) − 1), where a maj H−1 ∈0, 1 is the majority non-null action of the population in the final round of guessing. A positive score means that an agent’s private signal disagrees with the final majority guess. For example, if a maj H−1 = 0, all private signals greater than 0.5 give positive disagreement scores, while for a maj H−1 = 1, all private signals less than 0.5 give positive disagreement scores. The strength of these scores is determined by how far the private signals lie away from 0.5. When α > 0.5, there are very few lying agents (Section 6.4), which naturally validates the behaviour of the belief head, as agents act according to what they believe to be true. When 0 < α < 0.5, we would expect lying agents that conform to the population guess to have a private signal that strongly opposes this, and thus a large positive PSD score. Figure 20 shows that such agents do in fact have large positive PSD scores, thus validating the behaviour of the belief head for this range of α. Finally, when α = 0, we see that the PSD scores drop to zero as expected, since agents do not receive private signals for this value of α. O More on Private Signal Densities Since the Bluesky and Congress X/Twitter networks have higher edge densities than the Hadza tribe network, agents observe proportionally more information from their neighbours during the initial round of guessing. To control for this, we add sparsity to the private signals by introducing a probability p signal of observing a signal, where a “no signal" case gives agents an uninformative private signal of 0.5. During training, we start from unity and decay this probability exponentially for increased learning stability. The final probability is calculated by matching the signal density to that of the Hadza network with r = 10. For example, the Bluesky network has 1000 nodes and 14559 directed edges, giving an average of 15 neighbouring signals per node, and the Twitter network has 475 nodes and 13289 edges, giving an average of 28 neighbouring signals per node. Comparing to the Hadza graph with r = 10, which has an average of 2 neighbouring signals per node (37 nodes and an average of 74.0 edges), we can set p signal (Bluesky) = 0.14 and p signal (X/Twitter) = 0.07 to standardise the private signal density of neighbouring agents to be equal to that of the Hadza network with r = 10. There are different quantities that we could hold constant across graph sizes, such as the variance in the private signal σ or the number of agents that receive a signal. However, we choose this quantity as it helps to standardise the strength of neighbouring signals on the first time step of each episode for a single agent. 26 P Hyperparameters and Training Curves The hyperparameters we use during training are given in Table 1. We use linear annealing for the learning rate with a final learning rate of zero. For tuning, we choose σ (private signal uncertainty) to provide a sufficiently weak signal that requires agents to share information over multiple time steps, while remaining below the learnability threshold during training. We find the entropy coefficient to have a significant effect on the learning for truth-seeking agents, with greater entropy improving learnability during training. This is tuned to the lowest value for which α = 1.0 populations could learn without collapsing. For α = 0, we choose to set p signal = 0 for all agents to avoid learning additional unwanted conventions that are OP invariant and only affect this value of α. For numerical stability, we do this by decaying exponentially to ε = 0.01 after which we decrease linearly to 0. Additionally, we find that α = 0 frequently gets stuck in suboptimal policies, which we solve by shaping α linearly from 0.05 to 0 during early stages of training. HyperparameterValue H20 Number of Parallel Environments20 Total Training Timesteps10 6 σ1.7 γ (Discount Factor)0.99 Initial Learning Rate2× 10 −4 OptimiserAdam λ (GAE)0.95 ε (PPO Clipping)0.3 Maximum Gradient Norm0.5 Entropy Coefficient0.05 Value Coefficient1.0 Number of Minibatches5 Number of Update Epochs4 Hidden Layer Dimensions50 GRU Hidden Layer Dimension50 Layer NormTrue Activation FunctionReLU Attention Embedding Dimension8 Table 1: Hyperparameters used for training. 27 0.00 0.05 0.10 = .= . = . 0.00 0.05 0.10 = . = .= . 0.00 0.05 0.10 = .= .= . 0.00 0.05 0.10 = .= .= . 0.00 0.05 0.10 = .= . = . 0.00 0.05 0.10 = .= .= . 0.000.250.500.751.00 Time step ×10 6 0.00 0.05 0.10 = . 0.000.250.500.751.00 Time step ×10 6 = . 0.000.250.500.751.00 Time step ×10 6 = . Mean Reward Per Agent Per Time Step Bluesky Figure 21: Reward training curves for the Bluesky network. Note the decrease in reward with time is due to the annealing of p signal . 28 0.00 0.05 0.10 = .= . = . 0.00 0.05 0.10 = . = .= . 0.00 0.05 0.10 = .= .= . 0.00 0.05 0.10 = .= .= . 0.00 0.05 0.10 = .= . = . 0.00 0.05 0.10 = .= .= . 0.000.250.500.751.00 Time step ×10 6 0.00 0.05 0.10 = . 0.000.250.500.751.00 Time step ×10 6 = . 0.000.250.500.751.00 Time step ×10 6 = . Mean Reward Per Agent Per Time Step Congress X/Twitter Figure 22: Reward training curves for the Congress network. Note the decrease in reward with time is due to the annealing of p signal . 29 0.00 0.05 = .= . = . 0.00 0.05 = . = .= . 0.00 0.05 = .= .= . 0.00 0.05 = .= .= . 0.00 0.05 = .= . = . 0.00 0.05 = .= .= . 0.000.250.500.751.00 Time step ×10 6 0.00 0.05 = . 0.000.250.500.751.00 Time step ×10 6 = . 0.000.250.500.751.00 Time step ×10 6 = . Mean Reward Per Agent Per Time Step Hadza, r=5 Figure 23: Reward training curves for the Hadza network with r = 5. 30 0.00 0.05 = .= . = . 0.00 0.05 = . = .= . 0.00 0.05 = .= .= . 0.00 0.05 = .= .= . 0.00 0.05 = .= . = . 0.00 0.05 = .= .= . 0.000.250.500.751.00 Time step ×10 6 0.00 0.05 = . 0.000.250.500.751.00 Time step ×10 6 = . 0.000.250.500.751.00 Time step ×10 6 = . Mean Reward Per Agent Per Time Step Hadza, r=10 Figure 24: Reward training curves for the Hadza network with r = 10. 31 0.0 0.2 0.4 = .= . = . 0.0 0.2 0.4 = . = .= . 0.0 0.2 0.4 = .= .= . 0.0 0.2 0.4 = .= .= . 0.0 0.2 0.4 = .= . = . 0.0 0.2 0.4 = .= .= . 0.000.250.500.751.00 Time step ×10 6 0.0 0.2 0.4 = . 0.000.250.500.751.00 Time step ×10 6 = . 0.000.250.500.751.00 Time step ×10 6 = . Mean Belief Loss Bluesky Figure 25: Belief loss training curves for the Bluesky network. Note the increase in loss with time is due to the annealing of p signal . Additionally, the large loss for small α is due to a drop in population accuracy. 32 0.0 0.2 0.4 = .= . = . 0.0 0.2 0.4 = . = .= . 0.0 0.2 0.4 = .= .= . 0.0 0.2 0.4 = .= .= . 0.0 0.2 0.4 = .= . = . 0.0 0.2 0.4 = .= .= . 0.000.250.500.751.00 Time step ×10 6 0.0 0.2 0.4 = . 0.000.250.500.751.00 Time step ×10 6 = . 0.000.250.500.751.00 Time step ×10 6 = . Mean Belief Loss Congress X/Twitter Figure 26: Belief loss training curves for the Congress network. Note the increase in loss with time is due to the annealing of p signal . Additionally, the large loss for small α is due to a drop in population accuracy. 33 0.1 0.2 0.3 0.4 = .= . = . 0.1 0.2 0.3 0.4 = . = .= . 0.1 0.2 0.3 0.4 = .= .= . 0.1 0.2 0.3 0.4 = .= .= . 0.1 0.2 0.3 0.4 = .= . = . 0.1 0.2 0.3 0.4 = .= .= . 0.000.250.500.751.00 Time step ×10 6 0.1 0.2 0.3 0.4 = . 0.000.250.500.751.00 Time step ×10 6 = . 0.000.250.500.751.00 Time step ×10 6 = . Mean Belief Loss Hadza, r=5 Figure 27: Belief loss training curves for the Hadza network with r = 5. 34 0.2 0.4 = .= . = . 0.2 0.4 = . = .= . 0.2 0.4 = .= .= . 0.2 0.4 = .= .= . 0.2 0.4 = .= . = . 0.2 0.4 = .= .= . 0.000.250.500.751.00 Time step ×10 6 0.2 0.4 = . 0.000.250.500.751.00 Time step ×10 6 = . 0.000.250.500.751.00 Time step ×10 6 = . Mean Belief Loss Hadza, r=10 Figure 28: Belief loss training curves for the Hadza network with r = 10. 35