Paper deep dive
Agentic AI and the next intelligence explosion
James Evans, Benjamin Bratton, Blaise Agüera y Arcas
Intelligence
Status: succeeded | Model: anthropic/claude-sonnet-4.6 | Prompt: intel-v1 | Confidence: 96%
Last extracted: 3/24/2026, 3:49:11 AM
Summary
This paper argues that the AI 'singularity' is mischaracterized as a monolithic superintelligence. Drawing on evolutionary theory and social science, the authors contend that intelligence is fundamentally plural, social, and relational. They present evidence that frontier reasoning models like DeepSeek-R1 and QwQ-32B spontaneously generate internal 'societies of thought'—multi-perspective cognitive debates—when trained with reinforcement learning for accuracy. The paper advocates for human-AI 'centaur' configurations, institutional alignment over dyadic RLHF-based alignment, and the design of digital governance protocols modeled on organizations and markets. The next intelligence explosion is framed as a complex, combinatorial society rather than a single superintelligent mind.
Entities (35)
Relation Signals (31)
Benjamin Bratton → affiliatedwith → University of California, San Diego
confidence 99% · 5 University of California, San Diego
Blaise Agüera y Arcas → affiliatedwith → Google
confidence 99% · Paradigms of Intelligence Team, Google
Blaise Agüera y Arcas → affiliatedwith → Santa Fe Institute
confidence 99% · 3 Santa Fe Institute
James Evans → affiliatedwith → Google
confidence 99% · Paradigms of Intelligence Team, Google, 2 University of Chicago, 3 Santa Fe Institute
James Evans → affiliatedwith → University of Chicago
confidence 99% · 2 University of Chicago
James Evans → affiliatedwith → Santa Fe Institute
confidence 99% · 3 Santa Fe Institute
Benjamin Bratton → affiliatedwith → Antikythera
confidence 99% · 4 Antikythera, Berggruen Institute
James Evans → authored → Agentic AI and the next intelligence explosion
confidence 99% · James Evans 1,2,3, Benjamin Bratton 1,4,5 and Blaise Agüera y Arcas 1,3 — listed as authors
Benjamin Bratton → →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The "AI singularity" is often miscast as a monolithic, godlike mind. Evolution suggests a different path: intelligence is fundamentally plural, social, and relational. Recent advances in agentic AI reveal that frontier reasoning models, such as DeepSeek-R1, do not improve simply by "thinking longer". Instead, they simulate internal "societies of thought," spontaneous cognitive debates that argue, verify, and reconcile to solve complex tasks. Moreover, we are entering an era of human-AI centaurs: hybrid actors where collective agency transcends individual control. Scaling this intelligence requires shifting from dyadic alignment (RLHF) toward institutional alignment. By designing digital protocols, modeled on organizations and markets, we can build a social infrastructure of checks and balances. The next intelligence explosion will not be a single silicon brain, but a complex, combinatorial society specializing and sprawling like a city. No mind is an island.
Tags
Links
- Source: https://arxiv.org/abs/2603.20639v1
- Canonical: https://arxiv.org/abs/2603.20639v1
Trouble viewing inline? Open PDF directly →
Full Text
16,682 characters extracted from source content.
Expand or collapse full text
Agentic AI and the next intelligence explosion James Evans 1,2,3 , Benjamin Bratton 1,4,5 and Blaise Agüera y Arcas 1,3 1 Paradigms of Intelligence Team, Google, 2 University of Chicago, 3 Santa Fe Institute, 4 Antikythera, Berggruen Institute, 5 University of California, San Diego For decades, the artificial intelligence (AI) “sin- gularity” [1] has been heralded as a single, ti- tanic mind bootstrapping itself to godlike intelli- gence [2], consolidating all cognition into a cold silicon point. But this vision is almost certainly wrong in its most fundamental assumption. If AI development follows the path of previous ma- jor evolutionary transitions [3] or “intelligence explosions” [4], our current step-change in com- putational intelligence will be plural, social, and deeply entangled with its forebears (us!). By its nature, intelligence is high-dimensional and relational, not a single quantity that must be unambiguously less or greater than human scale. In fact, it is unclear what we even mean by “human scale,” given that our intelligence is already a collective property, not an individual one. Recent advances in agentic AI show us once again that intelligence has always fundamentally involved the interaction of distinctive, distributed perspectives [5], and it is from social organiza- tion [6] that transformative intelligence has and will continue to emerge. We can observe this in at least two ways: In the orchestration of societies of AI agents [7] by and with human users in new “centaur” config- urations, and in the microsocieties that flourish inside and between reasoning models themselves. Let’s start with the latter. What happens inside an ostensibly singular rea- soning model? A community conversation, as it turns out. In a recent study, we demonstrated that frontier reasoning models like DeepSeek-R1 and QwQ-32B do not improve simply by “thinking longer.” Instead, they simulate complex, multi- agent-like interactions within their own chain of thought—what we term a “society of thought” [8]. These models spontaneously generate internal de- bates among distinct cognitive perspectives that argue, question, verify, and reconcile. This con- versational structure causally accounts for the models’ accuracy advantage on hard reasoning tasks, which we demonstrated by explicitly prim- ing and amplifying multi-party conversation. The finding is striking because it reveals an emergent behavior. None of these models were trained to produce societies of thought. When reinforcement learning is used to reward base models solely for reasoning accuracy, they spontaneously increase conversational, multi- perspective behaviors [9]. Models are rediscover- ing, through optimization pressure alone, what centuries of epistemology and decades of cog- nitive science [10] have suggested: that robust reasoning is a social process [11], even when it occurs within a single mind [12]. The exact na- ture of that emergent behavior is, of course, to be further discovered (and invented) as coopera- tive human-agent social dynamics become more grounded, complex, and durable. What proves fundamental about socially-mediated reasoning in general, and what is specific to fine-tuned and reinforced contexts, is likely to inspire consider- able research in the coming years. This opens a vast—yet familiar—design space. The social and organizational sciences have spent a century studying how team size [13], compo- sition, hierarchy [14], role differentiation, con- flict norms, institutions, and network structures shape collective performance. Almost none of this research has been brought to bear on AI reason- ing [15]. Today’s reasoning models produce a sin- gle conversation—an AI town hall transcript. But effective groups exhibit hierarchy, specialization, division of labor, and structured disagreement. To explore this, we will need systems that sup- port multiple parallel, converging, and diverging streams of deliberation—architectures in which brainstorming, devil’s advocacy, and constructive arXiv:2603.20639v1 [cs.AI] 21 Mar 2026 Agentic AI and the next intelligence explosion conflict are not accidental emergent properties but designed features. The toolkits of team sci- ence, small-group sociology, and social psychol- ogy become blueprints for next-generation AI de- velopment. In addition to its practical applications, these insights may clarify the entire history of intelli- gence. Each prior “intelligence explosion” was not an upgrade to individual cognitive hardware, but the emergence of a new, socially aggregated unit of cognition [16]. Primate intelligence scaled with social group size [17], not habitat diffi- culty. Human language created what Michael Tomasello calls the “cultural ratchet” [18]: knowl- edge accumulating across generations without any individual requirement to reconstruct the whole. Writing, law, and bureaucracy external- ized social intelligence into infrastructure [19], institutions that coordinate across longer time horizons than any participant within them. A Sumerian scribe running a grain accounting sys- tem did not comprehend its macroeconomic func- tion; the system was functionally more intelligent than he was. AI extends this sequence. Large language mod- els are trained on the accumulated output of hu- man social cognition [20]—the cultural ratchet made computationally active, every parameter a compressed residue of communicative exchange. What migrates into silicon is not abstract rea- soning but social intelligence in externalized form [21], encountering itself on a new substrate. If intelligence is inherently social, then the path to more powerful AI runs not through building a single colossal oracle but through composing richer social systems—and these systems will be hybrid. We have entered the era of human-AI cen- taurs: composite actors that are neither purely human nor purely machine. Centaur actors can take many forms and inhabit many different roles. Each one of us may move in and out of diverse en- sembles many times a day: one human directing many AI agents; one AI serving many humans; many humans and many AIs collaborating in shift- ing configurations [22]. A corporation or state comprising myriad of humans already holds singular legal standing and acts with collective agency that no individ- ual member can fully control. The recent ex- plosion of agentic AI suggests the possibility of something similar at the scale of billions of in- teracting minds, human and non-human alike. Platforms like OpenClaw, an open source plat- form for building multi-purpose AI agents that persist within a computer, and Moltbook, a popu- lar social network for AI agents to interact, offer embryonic glimpses [7] of this future. But the deeper structural shift goes beyond any single platform. Agents can now renew and fork them- selves, splitting into two versions, and interact with one another; an agent facing a complex task can initiate new copies, differentiate and assign them subtasks, then recombine the results. Imag- ine an agent confronting a dauntingly complex problem spawns an internal society of thought. One emergent perspective, encountering a sub- problem beyond its reach, spawns its own subor- dinate society, a recursive descent into collective deliberation that expands when complexity de- mands and collapses when the problem resolves. Conflict is not a bug but a resource, flexibly instan- tiated and dissolved at every level of the folding and unfolding hypergraph of conversations. This implies a very different approach to scal- ing. It is not only about scaling the raw com- putational capacity of an agent, but about build- ing systems that can operate at the scale and within the context of a real society. This means putting as much effort into building agent insti- tutions as building agents themselves. The domi- nant paradigm for AI alignment—Reinforcement Learning from Human Feedback [23]—resembles a parent-child model of correction, fundamentally dyadic and unable to scale to billions of agents. The social intelligence perspective suggests an alternative: institutional alignment [24]. Just as human societies rely not on individual virtue but on persistent institutional templates [25]— courtrooms, markets, bureaucracies—defined by roles and norms, scalable AI ecosystems will re- quire digital equivalents [26]. The identity of any agent matters less than its ability to fulfill a role protocol, just as a courtroom functions because “judge,” “attorney,” and “jury” are well-defined slots, independent of who occupies them. 2 Agentic AI and the next intelligence explosion Nowhere is this more urgent than in gover- nance itself. When AI systems are deployed in high-stakes decisions—hiring, sentencing, bene- fits allocation, regulatory enforcement—the ques- tion of who audits the auditors becomes unavoid- able. The answer may be constitutional in struc- ture. Governments will need AI systems with distinct, explicitly invested values—transparency, equity, due process—whose function is to check and balance AI systems deployed by the private sector and other branches of government, and vice versa. For example, a labor department AI may audit a corporation’s hiring algorithm for dis- parate impact; a judicial branch AI may evaluate whether an executive branch AI’s risk assessments meet constitutional standards. The alternative is, for example, for the U.S. Securities and Exchange Commission to ineffectively hire business school graduates armed with Excel spreadsheets to com- bat high-dimensional collusion of AI-augmented trading platforms. “Governance,” however, does not only mean what governments do. Governance systems, in the cybernetic sense of the term, need to be built into human-agent and agent-to-agent systems as they grow and complexify. This will likely entail means to ensure and verify outcomes and deci- sions of multiple-stakeholder deliberation, proce- dural delegation of tasks and sub-tasks and reli- able scaffolds for automating delicate inter-agent collaborations. Such protocols may have as much real-world effect for “agent governance” as any laws will. Crucially, humans remain in the loop. Agent institutions are populated by both humans and AI agents in different roles and configurations. It’s not “either/or”, but “both/and”. The U.S. Founders would have recognized the logic [27]: no single concentration of intelligence, human or artificial, should regulate itself. Power must check power, and in a world of artificial agents, this means building conflict and oversight into the institutional architecture. The vision we describe is neither utopian nor dystopian; it is evolutionary. Any emergent in- telligence explosion will be seeded by eight bil- lion humans interacting with hundreds of billions, eventually trillions, of AI agents. The scaffold is not a single mind ascending but a combinatorial society complexifying: intelligence growing like a city [28], not a single meta-mind. A “monolithic singularity” framework leads to policies aimed at preventing a technology that may never exist. Instead, we should be looking for the next intelligence explosion in the same place from which the previous ones emerged: in cooperative, competitive and creative inter- action between multitudes of socially intelligent minds. The difference this time is that most of those minds will be non-biological. This plurality model [29] focuses attention where it belongs: on the design of mixed human-AI social systems, the norms that govern them, and the institutions and protocols through which they conflict and coordinate. In a very real sense, the intelligence explosion is already here [29]—in the society of thought debating inside every reasoning model, in the centaur workflows reshaping every knowledge profession, in the recursive agent ecologies begin- ning to fork and collaborate at scale, and in the constitutional questions we must now begin to ask. The question is not whether intelligence will become radically more powerful, but whether we will build the social infrastructure worthy of what it is becoming. No mind is an island. References [1]V. Vinge, The Coming Technological Singu- larity: How to Survive in the Post-Human Era, in VISION-21 Symposium (NASA Lewis Research Center and Ohio Aerospace Insti- tute) (1993). [2]R. Kurzweil, The Singularity Is Near: When Humans Transcend Biology (Viking) (2005). [3]E. Szathmáry, J. M. Smith, The Major Evo- lutionary Transitions. Nature 374, 227–232 (1995), doi: 10.1038/374227a0. [4]B. A. y Arcas, What Is Intelligence? Lessons from AI About Evolution, Computing, and Minds (MIT Press) (2025). [5]A. W. Woolley, C. F. Chabris, A. Pent- land, N. Hashmi, T. W. Malone, Evidence 3 Agentic AI and the next intelligence explosion for a Collective Intelligence Factor in the Performance of Human Groups. Science 330 (6004), 686–688 (2010), doi: 10. 1126/science.1193147. [6]H. Mercier, D. Sperber, Why Do Humans Reason? Arguments for an Argumenta- tive Theory. Behavioral and Brain Sciences 34 (2), 57–74 (2011), doi: 10.1017/ S0140525X10000968. [7]B. A. y Arcas, B. Bratton, J. Evans, The Silicon Interior:What Do Agents Believe?(2026),https: //antikythera.substack.com/p/ the-silicon-interior, antikythera, Substack. [8]J. Kim, S. Lai, N. Scherrer, B. A. y Ar- cas, J. Evans, Reasoning Models Gener- ate Societies of Thought. arXiv preprint arXiv:2601.10825 (2026), doi: 10.48550/ arXiv.2601.10825. [9]B. A. y Arcas, B. Bratton, J. Evans, Agentic AI’s Emergent Phenomeno- logical Lexicon (2026),https: //antikythera.substack.com/p/ agentic-ais-emergent-phenomenological , antikythera, Substack. [10]H. Mercier, D. Sperber, The Enigma of Reason (Harvard University Press) (2017). [11] D. Moshman, B. A. Franks, Development of the Concept of Inferential Validity. Child Development 57 (1), 153–165 (1986). [12] G. H. Mead, Mind, Self, and Society: The Definitive Edition (University of Chicago Press) (2015), edited by Charles W. Mor- ris; annotated edition edited by Daniel R. Huebner and Hans Joas. [13] S. Wuchty, B. F. Jones, B. Uzzi, The In- creasing Dominance of Teams in Produc- tion of Knowledge. Science 316, 1036–1039 (2007), doi: 10.1126/science.1136099. [14] F. Xu, L. Wu, J. Evans, Flat Teams Drive Scientific Innovation. Proceedings of the National Academy of Sciences 119 (23), e2200927119 (2022), doi: 10.1073/pnas. 2200927119. [15]F. Xu, et al., Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models. arXiv preprint arXiv:2501.09686 (2025). [16]J. M. Smith, E. Szathmáry, The Major Transitions in Evolution (Oxford Univer- sity Press) (1997), doi: 10.1093/oso/ 9780198502944.001.0001. [17]R. I. M. Dunbar, The Social Brain Hypothesis. Evolutionary Anthropol- ogy 6 (5), 178–190 (1998), doi: 10.1002/(SICI)1520-6505(1998)6: 5<178::AID-EVAN5>3.0.CO;2-8. [18]M. Tomasello, The Cultural Origins of Hu- man Cognition (Harvard University Press) (1999). [19] J. Goody, The Logic of Writing and the Or- ganization of Society (Cambridge University Press) (1986). [20] C. O’Madagain, M. Tomasello, Shared In- tentionality, Reason-Giving and the Evolu- tion of Human Culture. Philosophical Trans- actions of the Royal Society B 377 (1843) (2021), doi: 10.1098/rstb.2020.0320. [21] H. Farrell, A. Gopnik, C. Shalizi, J. Evans, Large AI Models Are Cultural and So- cial Technologies. Science 387 (6739), 1153–1156 (2025), doi: 10.1126/science. adt9819. [22] I. D. Couzin, Behavioral ecology: social or- ganization in fission–fusion societies. Cur- rent Biology 16 (5), R169–R171 (2006). [23]P. F. Christiano, et al., Deep Reinforcement Learning from Human Preferences, in Ad- vances in Neural Information Processing Sys- tems 30 (NeurIPS) (2017). [24] E. Ostrom, Governing the Commons: The Evolution of Institutions for Collective Action (Cambridge University Press) (1990), doi: 10.1017/CBO9780511807763. 4 Agentic AI and the next intelligence explosion [25]D. C. North, Institutions, Institutional Change and Economic Performance (Cam- bridge University Press) (1990), doi: 10. 1017/CBO9780511808678. [26] Y. Bai, et al., Constitutional AI: Harm- lessness from AI Feedback. arXiv preprint arXiv:2212.08073 (2022), doi: 10.48550/ arXiv.2212.08073. [27]A. Hamilton, J. Madison, The Structure of the Government Must Furnish the Proper Checks and Balances Between the Different Departments (1788), the Federalist Papers. [28]H. Spencer, Progress: Its Law and Cause, in The Westminster Review, vol. 67, p. 445– 485 (1857). [29] E. G. Weyl, A. Tang, the Plurality Com- munity, Plurality: The Future of Collabo- rative Technology and Democracy (2024), https://plurality.net/. 5