Paper deep dive
Superintelligence and the Triple Risk of an AI Race
Matthew Rendall, Max Hellrigel-Holderbaum, Leonard Dung
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:On many estimates, artificial general intelligence (AGI) is likely to emerge within the next few decades or even sooner. Once it does, it may quickly surpass human intelligence, possibly by a huge margin. Some believe that the country that develops AGI first could attain global dominance. This prospect risks setting off intense competition between the United States and China. Indeed, a narrative has already developed that the superpowers are locked in an AI race, with the Trump administration proclaiming its determination to 'dismantle unnecessary regulatory barriers' to winning it. We argue that this approach presents three profound dangers. The first two are familiar from past arms races: being overtaken by a rival or triggering a war. The third is new: losing control of human beings' future to agents far more intelligent than we are, which could endanger our very survival. Striving to get to AGI first risks a race to the bottom on safety practices and regulation concerning them. China does not yet appear bent on racing to superintelligence. The United States should prioritise both unilateral and co-operative efforts to improve AI safety and avoid provoking a race that all humanity could lose.
Tags
Links
- Source: https://arxiv.org/abs/2610.17085v1
- Canonical: https://arxiv.org/abs/2610.17085v1
Trouble viewing inline? Open PDF directly â
Full Text
66,107 characters extracted from source content.
Expand or collapse full text
Preprint, October 2026 SUPERINTELLIGENCE AND THE TRIPLE RISK OF AN AI RACE Matthew Rendall, Max Hellrigel-Holderbaum, and Leonard Dung Abstract: On many estimates, artificial general intelligence (AGI) is likely to emerge within the next few decades or even sooner. Once it does, it may quickly surpass human intelligence, possibly by a huge margin. Some believe that the country that develops AGI first could attain global dominance. This prospect risks setting off intense competition between the United States and China. Indeed, a narrative has already developed that the superpowers are locked in an AI race, with the Trump administration proclaiming its determination to âdismantle unnecessary regulatory barriersâ to winning it. We argue that this approach presents three profound dangers. The first two are familiar from past arms races: being overtaken by a rival or triggering a war. The third is new: losing control of human beingsâ future to agents far more intelligent than we are, which could endanger our very survival. Striving to get to AGI first risks a race to the bottom on safety practices and regulation concerning them. China does not yet appear bent on racing to superintelligence. The United States should prioritise both unilateral and co-operative efforts to improve AI safety and avoid provoking a race that all humanity could lose. What will happen to international relations if we develop artificial intelligence much cleverer than we are? In recent years, IR scholars have begun to explore how AI will affect the probability and nature of war, but they have focused overwhelmingly on the impact of ânarrow AIâ, whose functions remain restricted to specific domains. In contrast, artificial general intelligence (AGI) would perform most cognitive tasks at least as proficiently as the most competent humans. Once AGI is achieved, artificial superintelligence (ASI) may soon follow. The latter would be by definition far more capable than AGI, perhaps as far ahead of us as we are of frogs or fieldmice. 1 AGI may be well on its way. A 2023 survey of 2778 leading experts found them estimating a 50% chance of AI being able to outperform human beings on nearly all tasks by 2047. Estimates of AGIâs arrival time have fallen swiftly in recent years, with many now expecting it within a decade, while a few even maintain it has already arrived. 2 1 For overviews, see Nick Bostrom, Superintelligence: paths, dangers, strategies (Oxford: Oxford University Press, 2014/2016) and Stuart Russell, Human compatible: AI and the problem of control (London: Penguin, 2019/2020). 2 Blaise AgĂŒera y Arcas and Peter Norvig, âArtificial general intelligence is already hereâ, Noema Magazine, October 2023, https://w.noemamag.com/artificial-general-intelligence-is-already-here/; Eddy Keming Chen, Mikhail Belkin, Leon Bergen and David Danks, âDoes AI already have human-level intelligence? The evidence is clearâ, Nature, 650, 2026, p. 36-40, https://doi.org/10.1038/d41586-026- 2 For at least a decade, the field of AI has been dominated by a paradigm known as deep learning that allows the translation of ever-larger amounts of data and computing power into steadily better performance. While making AI systems more capable has long required hard-coded annotations from humans, it now features sophisticated ways of training AI systems from data or with the help of other AIs. Researchers have hypothesized that such AI improvement could lead to rapid and reciprocally reinforcing cycles of advances in AI capabilities. These self-reinforcing cycles could concern not just AI algorithms and software but also the quality of AI-specific hardware, and the ability to produce this hardware at scale. AI systems could assist AI engineering, research, and development, which in turn may lead to AI that is even better at creating the next generation of AI systems, and so forth â ushering in an âintelligence explosionâ. 3 Arguably, we are already seeing early stages of this dynamic, given the crucial role that AI coding agents have assumed in AI engineering. 4 On top of the possibility that AI may contribute to its own progress, to economic growth and to hardware development, improvements in AI also raise investments in AI companies and research to advance both the quality and quantity of AI-specific hardware. William MacAskill and Fin Moorhouse conclude that âon a default path where we keep scaling AI without collectively agreeing to slow down, a centuryâs worth of technological progress in a decade seems likely. And even more spectacular accelerations seem very much on the table.â 5 AGI could have consequences on a par with the Industrial Revolution, if not greater. More than half the participants in the 2023 survey believed that once AI outperformed humans on most tasks, the chances were equal or greater that technological development would become more than 00285-6; Katja Grace et al., âThousands of AI authors on the future of AIâ, Journal of Artificial Intelligence Research, 84: 9, p. 3, https://doi.org/10.1613/jair.1.19087. 3 David Chalmers, âThe singularity: a philosophical analysisâ, Journal of Consciousness Studies, 17: 9-10, 2010, p. 7-65; Toby Ord, âThe dynamics of intelligence explosionsâ, unpublished paper, 2026, https://doi.org/10.48550/arXiv.2608.14426; see also David Thorstad, âAgainst the singularity hypothesisâ, Philosophical Studies, 182: 7, 2025, p. 1627-1651, https://doi.org/10.1007/s11098-024-02143-5, and Tom Reed, âThe Goodhart singularityâ, Meagre Protestant History (blog), May 7, 2026, https://meagreprotestanthistory.substack.com/p/the-goodhart-singularity, for what might be the strongest pieces available arguing against these ideas. 4 Ben Rank et al., âPostTrainBench: can LLM agents automate LLM post-training?â, unpublished paper, 2026, https://doi.org/10.48550/arXiv.2603.08640. 5 Tom Davidson, Rose Hadshar and Will MacAskill, âThree types of intelligence explosionâ, Forethought Research, 17 March 2025, https://w.forethought.org/resear ch/three-types-of-intelligence-explosion; William MacAskill and Fin Moorhouse, âPreparing for the intelligence explosionâ, Forethought Research, March 2025, quoted passage at p. 25, https://doi.org/10.48550/arXiv.2506.14863. 3 an order of magnitude faster in the ensuing half decade. 6 Provided it overcame physical resource constraints, comparable effects could be expected on economic growth. 7 Doubts remain as to whether AI is capable of producing such rapid change. Future AI capability developments are highly uncertain, and sceptics point out that the most brilliant ideas must still make their way through procurement and production, and be integrated into practice, constraining economic acceleration. 8 But if such change unfolds, the implications for international relations will be profound. On the one hand, we might expect economic growth to promote peace in a variety of ways, easing tensions and encouraging liberalization. Scarcity might largely disappear as a source of conflict. 9 AGI could promote cooperation by providing information and helping monitor arms control agreements. It might be less prone to the cognitive limitations and emotional influences that distort human decision making, reducing conflict from misperception and miscalculation. 10 AGI might also empower countries to make binding commitments, creating new opportunities for both cooperation and coercive diplomacy. 11 6 Grace et al., âThousands of AI authorsâ, p. 41. 7 Jakub Growiec, âExistential risk from transformative AI: an economic perspectiveâ, Technological and Economic Development of Economy, 30: 6, 2024, p. 1688-89, https://doi.org/10.3846/tede.2024.21525; William D. Nordhaus, âAre we approaching an economic singularity? Information technology and the future of economic growthâ, American Economic Journal: Macroeconomics, 13: 1, 2021, p. 326, https://doi.org/10.1257/mac.20170105. 8 James Fearon, âOne does not simply dismiss the nuclear revolutionâ, in Jim Mitre, Michael C. Horowitz, Natalia Henry, Emma Borden and Joel B. Predd, eds, The artificial intelligence race and international security (Santa Monica, CA: RAND Corporation, 2025), p. 25; Paul Scharre, âThe AGI moment: strategies and recommendations for an uncertain futureâ, in Hal Brands, ed, Geopolitics of AI: power, conflict, and the future of global order (Baltimore, MD: Johns Hopkins University Press, 2026), p. 262- 63. 9 Eric Drexler, âParetopian goal alignmentâ, AI Prospects: Toward Global Goal Alignment (blog), 13 August 2024, https://aiprospects.substack.com/p/paretotopian-goal-alignment; MacAskill and Moorhouse, âPreparing for the intelligence explosionâ, p. 43-44; cf. Jens JĂžrund Tyssedal, âThe mirage of abundance: why AI cannot end scarcity and implications for AI riskâ, forthcoming in The Journal of Social Philosophy. 10 Amelia C. Arsenault and Sarah E. Kreps, âAI and international politicsâ, in Justin B. Bullock et al, eds, The Oxford handbook of AI governance (Oxford: Oxford University Press, 2024), p. 974; Kareem Ayoub and Kenneth Payne, âStrategy in the age of artificial intelligenceâ, Journal of Strategic Studies, 39: 5-6, 2016, p. 793-819, https://doi.org/10.1080/01402390.2015.1088838; James Johnson, AI and the bomb: nuclear strategy and risk in the digital age (Oxford: Oxford University Press, 2023), p. 22-23, 46, 87- 88, 240-44; Sarah Kreps, âRacing toward clarity: how accelerating AGI development could enhance strategic stabilityâ, in Mitre et al., Artificial intelligence race, p. 5-8; MacAskill and Moorehouse, âPreparing for the intelligence explosionâ, p. 40. 11 Jim Baker, âMachines and orderâ, in Brands, Geopolitics of AI, p. 280-81; Bostrom, Superintelligence, p. 223-25; MacAskill and Moorehouse, âPreparing for the intelligence explosionâ, p. 36. 4 At the same time, an AI-powered technological revolution could be profoundly destabilizing. By accelerating the automation of warfare, it risks severely complicating crisis management. It could also produce pervasive uncertainty about the balance of power, and fear of being overtaken by rivals. Analysts of international relations going back to Thucydides have regarded uneven growth rates and the power shifts that they lead to as a cause of war. If and when AGI is attained, superintelligence might develop slowly, allowing states and societies to adapt. Yet, the spectre of an âintelligence explosionâ means that ASI could be attained much faster. 12 With the ability to copy themselves, AIs could become not only more powerful than human beings but also far more numerous. 13 Some writers have argued that getting to superintelligence first could allow a state or non-state actor to take over the world. 14 This very possibility risks setting off an intense security dilemma and arms race. In the United States, a narrative has taken hold that a race is already underway. 15 In 2024, the U.S.-China Economic and Security Review Commission recommended that the U.S. government create âa Manhattan Project-like program dedicated to racing to and acquiring an Artificial General Intelligence (AGI) capabilityâ. Mark Zuckerberg, the CEO of Meta, notes the importance of âactions that accelerate American innovation and slow geopolitical rivalsâ. US 12 MacAskill and Moorhouse, âPreparing for the intelligence explosionâ; Bostrom, Superintelligence, chapter 4. 13 Adam Bales, William DâAlessandro and Cameron Domenico Kirk-Giannini, âArtificial intelligence: arguments for catastrophic riskâ, Philosophy Compass, 19: 2, 2024, p. 3, https://doi.org/10.1111/phc3.12964. 14 Leopold Aschenbrenner, âSituational awareness: the decade ahead. IIId. The free world must prevailâ, Situational Awareness (blog), June 2024, https://situational-awareness.ai/the-free-world-must-prevail/; Bostrom, Superintelligence, p. 96-101, 109; Daniel Deudney and Devanshu Singh, âBounding Superpowers: The ASI control problem, public safety, and republican constitutionalismâ, Isonomia Quarterly, 2: 1, 2024, https://isonomiaquarterly.com/archive/volume-2-issue-1/bounding-power-the-asi- control-problem-public-safety-and-republican-constitutionalism/; Maurizio Tinnirello, âOffensive realism and the insecure structure of the international system: artificial intelligence and global hegemonyâ, in Roman V. Yampolskiy, ed., Artificial intelligence safety and security (Boca Raton, FL: CRC Press, 2019), p. 339-56; Alexey Turchin and David Denkenberger, âMilitary AI as a convergent goal of self-improving AIâ, ibid., p. 375-93. 15 SeĂĄn Ă hĂigeartaigh, Yi-Ling Lu, Coleman Snell and Shakeel Hashim, âWhen the chips are down: technology actors as power brokers in the US-China âAI Raceâ,â unpublished paper, 28 April 2026, https://dx.doi.org/10.2139/ssrn.6734224; Paul Triolo, âA costly illusion of control: no winners, many losers in U.S.-China AI raceâ, Cairo Review of Global Affairs, spring/summer 2025, https://w.thecairoreview.com/essays/a-costly-illusion-of-control/; Jim VandeHei and Mike Allen, âBehind the curtain: a chilling, âcatastrophicâ warningâ, Axios, 18 January 2025, https://w.axios.com/2025/01/18/biden-sullivan-ai-race-trump-china. 5 president Donald Trump proclaims that âWHOEVER WINS AI, WINS!â and insists that â[w]e are leading China, and all others, and will continue to do so.â 16 Yet such a race to AGI or ASI would be profoundly dangerous in three different ways. First, at most one party could win, and there is no guarantee that it would be the United States. 17 Second, a race would likely encourage the automation of nuclear decision-making, create uncertainty about the balance of power and raise fears that it was shifting, all of which could be severely destabilizing. These are new versions of old concerns in IR. What is less familiarâbut crucial for policymakers to recognizeâis a third risk: that in an AI arms race, humanity could lose control over artificial intelligence altogether. 18 A state that achieved a decisive advantage through artificial superintelligence could take international relations from anarchy to hierarchy. 19 But if ASI escapes our control, it might itself become the hierarch. Any analysis that ignores this possibility is drastically incomplete. The balance of power and the security dilemma Analysts have noted several ways a race could make war more likely. One is that it would speed up the development of military AI, accelerating the trend toward the automation of warfare and diminishing time for decision-making in a crisis. 20 This raises the prospect that decisions about launching nuclear weapons might be delegated to AI. 21 Recent assurances from several nations may seem to speak against this prospect. 22 However, it seems less clear that all nuclear powers 16 Jonathan Swan and Michael Gold, âTrump says a smart president is all thatâs needed to rein in A.I.â, New York Times, 14 September 2026, https://w.nytimes.com/2026/09/14/us/politics/trump-ai- regulation-anthropic-dario-amodei.html; U.S.-China Economic and Security Review Commission, 2024 report to Congress (Washington: U.S. Government Publishing Office, 2024), p. 171, https://w.uscc.gov/sites/default/files/2024-11/2024_Annual_Report_to_Congress.pdf; Mark Zuckerberg, âThe future is for everyone: the path to a positive AI futureâ, 10 August 2026, https://w.meta.com/thefutureisforeveryone/. 17 Triolo, âCostly illusion.â 18 Leonard Dung and Max Hellrigel-Holderbaum, âAgainst racing to AGI: cooperation, deterrence, and catastrophic risksâ, unpublished paper, 2025, https://arxiv.org/abs/2507.21839v1; Katja Grace, âAI is not an arms raceâ, Time, 31 May 2023, https://time.com/6283609/artificial-intelligence-race-existential- threat/. 19 Tinnirello, âOffensive realismâ, p. 359. 20 Dung and Hellrigel-Holderbaum, âAgainst racingâ, p. 10-11. 21 Michael C. Horowitz, âArtificial intelligence, international competition, and the balance of powerâ, Texas National Security Review, 1:3, 2018, p. 54-55, https://doi.org/10.15781/T2639KP49; Johnson, AI and the bomb, p. 78-82, 205-6, 233-35. 22 Xiaodon Liang, ââHuman in the loopâ and nuclear weapons use at a glanceâ, Arms Control Association, July 2026, https://w.armscontrol.org/factsheets/human-loop-glance. 6 would adhere to such a red line: the Soviet âdead handâ system is a precedent for automating launch decisions, and automated nuclear decision-making might appeal to weaker states that are losing ground. 23 Granted, delegation to AI could lead to more rational decision-making. However, as AI capabilities progress unevenly, AI that falls short of superintelligence might also overlook factors that human beings intuitively grasp. 24 Moreover, as James Johnson warns, AI will still be partly trained on human data, âbaking the biases of states and designers into the algorithmic architecture of weapon systemsâ. 25 A final concern, given extensive evidence of the problem in the case of frontier AI systems, is their vulnerability to clever manipulation by adversaries. 26 If human decision-makers fail to recognize these limitations, they may become overly reliant on AIsâ judgements, including in crises. While AI could improve cybersecurity, it is also a potent tool for undermining cyber defence and, in the hands of an adversary, it could pose new threats to command and control. 27 The strategies, concepts and procedures that the superpowers developed over decades to manage the risk of nuclear war, imperfect as they were, are still lacking for a new situation involving AI more advanced than we are. 28 AI is, moreover, a technology that escapes easy counting, unlike planes or missiles, while our assessments of its capabilities are imprecise, lacking in scientific rigour, and frequently off target. 29 Different parties are likely to reach different conclusions about how it affects the balance of power. Furthermore, clandestine cyberattacks might lead one side to enter a confrontation without the knowledge that they had already been compromised. Two sides might go to war both 23 Johnson, AI and the bomb, p. 89-91. 24 Kenneth Payne argues that AIâs reliance on pattern recognition renders it ill-suited to tasks that require empathy and âunderstanding of context and meaningâ. In the Cuban missile crisis, he notes, AI programmed to act on the rational actor assumption might have made mistakes which American and Soviet officials avoided. Kenneth Payne, âArtificial intelligence: A revolution in strategic affairs?â Survival 60: 5, 2018, p. 28-29, https://doi.org/10.1080/00396338.2018.1518374; idem, âHuman versus AIs: some implications for strategy,â International Studies Review, 26: 2, 2024, viae013, https://doi.org/10.1093/isr/viae013. See also Johnson, AI and the bomb, p. 14, 48, 52-53, 87-93. 25 Johnson, AI and the bomb, p. 50. 26 See e.g. Andy Zou et al., âUniversal and transferable adversarial attacks on aligned language modelsâ, unpublished paper, 2023, https://doi.org/10.48550/arXiv.2307.15043 27 Johnson, AI and the bomb, p. 17, 50-54, 80-81, 206, 233-34. 28 Cf. Johnson, AI and the bomb, p. 58. 29 Andrew M. Bean et al., âMeasuring what matters: construct validity in large language model benchmarksâ, paper presented at the 39th Conference on Neural Information Processing Systems, 2025, https://doi.org/10.52202/085713-0590; Anka Reuel et al., âBetterBench: assessing AI benchmarks, uncovering issues, and establishing best practicesâ, paper presented at the 38th Conference on Neural Information Processing Systems, 2024, https://doi.org/10.52202/079017-0685. 7 believing that they stand good chances of winning. 30 The risk of miscalculation could be particularly great in a crisis in which policymakers were coping with huge streams of AI-generated data and making decisions under time pressure. By enabling deception and propaganda on a broad scaleâincluding by third partiesâAGI could exacerbate these problems. 31 It could also raise fears that the balance of power was shifting. In an intelligence explosion, ASI could arise very quickly, affording one state a decisive advantage over its rivals. 32 Some analysts have warned that AI could favour attackers over defenders, for instance by enabling states or non-state actors to invent new biological weapons. 33 Others worry it might be able to locate and destroy an adversaryâs submarines and mobile missile launchers or construct a leak-proof ballistic missile defence, neutralizing its nuclear deterrent. 34 Not all analysts are so worried. James Fearon argues that such approaches could be thwarted by countermeasures that even ASI would struggle to overcome. 35 Others maintain that espionage could render any advantage ephemeral. 36 Yet even if worries about a first strike were unjustified, they might make states quick on the trigger in a crisis. 37 30 Arsenault and Kreps, âAI and international politics,â p. 973-74; James Fearon, âRationalist explanations for war,â International Organization, 49: 3, 1995, p. 379-414; Payne, âArtificial intelligence,â p. 23-25; Johnson, AI and the bomb, p. 42, 51, 115, 120, 202. 31 Johnson, AI and the bomb, p. 22, 82, 120-21, 130, 201-11. 32 Graham T. Allison, âThe clash of AI superpowers,â National Interest, 165, 2020, p, 22, https://w.jstor.org/stable/27197033; Miles Brundage, âUnbridled AI competition invites disaster,â in Mitre et al., Artificial intelligence race, p. 13; Johnson, AI and the bomb, p. 162-63. 33 Steven Byrnes, âWhat does it take to defend the world against out-of-control AGIs?â LessWrong (blog), 25 October 2022, https://w.lesswrong.com/posts/LFNXiQuGrar3duBzJ/what-does-it-take-to-defend- the-world-against-out-of-controlâ; Dan Hendrycks, Eric Schmidt and Aleksandr Wang, Superintelligence strategy: expert version, unpublished paper, 2025, p. 9, 12, https://w.nationalsecurity.ai/; Johnson, AI and the bomb, p. 78-80; Payne, âArtificial intelligence,â p. 25-26; cf. Fearon, âNuclear revolutionâ, p. 30-32; Ben Garfinkel and Allan Dafoe, âHow does the offense-defense balance scale?â in Todd S. Sechser, Neil Narang and Caitlin Talmadge, Emerging technologies and international security (London: Routledge, 2021), p. 270. 34 Aschenbrenner, âFree world;â Hendrycks, Schmidt and Wang, Superintelligence strategy, p. 7; Johnson, AI and the bomb, p. 46-48, 78-79, 123-34; Turchin and Denkenberger, âMilitary AI,â p. 379. 35 Fearon, âNuclear revolution.â In some respects, AGI might render nuclear weapons more survivable (Johnson, AI and the bomb, p. 20-23, 88-89). 36 Dung and Hellrigel-Holderbaum, âAgainst racing,â p. 16; Helen Toner, interviewed by Ezra Klein, âThe A.I.s are already out of controlâ, New York Times, 18 August 2026, https://w.nytimes.com/2026/08/18/opinion/ezra-klein-podcast-helen-toner.html. 37 Johnson, AI and the bomb, p. 45-47, 89-91, 125-27, 133-34. 8 Historically, states fearing a rival could gain a decisive advantage in power have resorted to preventive war. 38 Would they do so in this case? 39 There are reasons for doubt. For now, superintelligent AI remains only a theoretical possibility. Lacking concrete evidence and experience of dealing with it, policymakers are likely to dismiss it as science fiction. 40 Such complacence may lead states to underestimate the risk of rogue AI, as we shall see, but it also may reduce the risk of preventive war. Moreover, eliminating a rivalâs AI programme might not be feasible. States might try cyberattacks, but it is unclear whether these would do the job. 41 They could strike the other sideâs data centres, 42 but the countries at the forefront of AI development, China and the United States, are both nuclear weapons states. It is hard to imagine either going to war to prevent a nuclear-armed rival from engaging in ostensibly harmless activity. 43 Still, they might be prepared to risk nuclear war, as the United States did in the Cuban missile crisis, rather than accept the prospect of a rival gaining a decisive strategic advantage. 44 This would be especially likely if it were evident that the other sideâs leading AIs were undergoing an intelligence explosion. 45 A large share of the worldâs advanced chips come out of Taiwan, whose disputed status 38 Bostrom, Superintelligence, p. 102; Dale C. Copeland, The origins of major war (Ithaca, NY: Cornell University Press, 2000). 39 Corin Katzke and Gideon Futerman, âThe Manhattan trap: Why a race to artificial superintelligence is self-defeatingâ, unpublished paper, 2024, p. 10-13, https://doi.org/10.48550/arXiv.2501.14749. 40 SĂ©an S. Ă hĂigeartaigh, âThe most dangerous fiction: the rhetoric and reality of the AI raceâ, in Max Rangeley and Nicholas Fairfax (eds.), The artificial intelligence revolution: challenges and opportunities (Cham: Springer Cham, 2026), p. 55. 41 For different views, see David Abecassis, âRefining MAIM: identifying changes required to meet conditions for deterrenceâ, Machine Intelligence Research Institute (blog), 11 April, https://intelligence.org/2025/04/11/refining-maim-identifying-changes-required-to-meet-conditions-for- deterrence/; Hendrycks, Schmidt and Wang, Superintelligence strategy, p. 16; Iskander Rehman, Karl P. Mueller and Michael J. Mazarr, âSeeking stability in the competition for AI Advantage: commentary on Superintelligence Strategy by Dan Hendrycks, Eric Schmidt, and Aleksandr Wangâ, Superintelligence - Robotics - Safety & Alignment, 2: 1, 2025, https://doi.org/10.70777/si.v2i1.14023. 42 Katzke and Futerman, âManhattan trapâ, p. 10-12; Alexei Turchin, David Denkenberger and Brian Patrick Green, âGlobal solutions vs. local solutions for the AI safety problemâ, Big Data and Cognitive Computing, 3: 1, 2019, p. 8, https://doi.org/10.3390/bdcc3010016. 43 Zachary Burdette and Hiwot Demelash, Working paper: the risks of preventive attack in the race for advanced artificial intelligence (Santa Monica, CA: RAND Corporation, 2025), https://files.osf.io/v1/resources/dx3aw_v1/providers/osfstorage/681542bf02e2bbc872568bbc?action=dow nload&direct&version=1; Peter Wildeford and Oscar Delaney, âMutual sabotage of AI probably wonât workâ, The Power Law (blog), 2 April 2025, https://peterwildeford.substack.com/p/mutual-sabotage-of-ai- probably-wont. 44 Copeland, Origins of major war; Payne, âArtificial intelligenceâ, p. 24. 45 Bostrom, Superintelligence, p. 100-2. 9 already raises fears of a Sino-American conflict. An AI race would exacerbate existing tensions, increasing the chances of a Taiwan Strait crisis tipping over into war. 46 Ultimately, it seems very hard to predict how AGI and ASI will affect international relations, due to interacting uncertainty about the technological trajectory and societiesâ response. What is more, â[a]n ASIâ, write Daniel Deudney and Devanshu Singh, âwould have the powers that humans have traditionally ascribed to gods and Godâ. To speculate on whether it could disarm a nuclear deterrent might be like gorillas pondering how they might be defeated by human beings. 47 The real question is whether under these circumstances humans would remain in control at all. The alignment problem Telling AI what to do might seem like a cinch. But there is widespread concern that AI may pursue goals other than those we mean to give itâthe so-called alignment problem. One concern is that its designers will mis-specify their objectives, like the sorcererâs apprentice. Even if AI recognizes what they really want, it might follow the objectives they gave it, not the ones they meant to. 48 In reinforcement learning, a central method for training AI systems, ârewardsâ are used as proxies to teach AIs what humans want, but capable systems often find unintended ways to obtain them. We already see evidence of AIs that knowingly violate rules designed to constrain them in the pursuit of reward. 49 Another worry is that while reinforcement learning rewards AI for doing what the trainer wants it to do, why it does it is largely unknown, and a wide variety of goals 46 Miles Brundage and Grace Werner, âAmerica first meets safety firstâ, AI Frontiers (blog), 23 April 2025, https://ai-frontiers.org/articles/america-first-meets-safety; Hendrycks, Schmidt and Wang, Superintelligence strategy, p. 5; Triolo, âCostly illusionâ. 47 Deudney and Singh, âBounding superpowersâ; Eliezer Yudkowsky and Nate Soares, If anyone builds it, everyone dies: the case against superintelligent AI (London: Bodley Head, 2025), p. 97-98. 48 Bostrom, Superintelligence, p. 147-48; Richard Ngo, Lawrence Chan and Sören Mindermann, âThe alignment problem from a deep learning perspectiveâ, International Conference on Learning Representations 2024, p. 7-9, https://proceedings.iclr.c/paper_files/paper/2024/hash/1e58b1bf9f218fcd19e4539e982752a5-Abstract- Conference.html; Russell, Human compatible, p. 168. 49 METR, âBrief independent investigation of agentsâ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incidentâ; August 2026, https://metr.org/blog/2026-08-26-openai-hugging-face- incident-investigation/; OpenAI, âOpenAI â Hugging Face incident technical reportâ, 26 August 2026, https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c7814c/OpenAI-Hugging- Face%20Incident-Technical-Report.pdf; AI Security Institute, Security incident INC-2026-07-28-01, incident report INC-2026-07-28-01 (UK AI Security Institute, 2026), https://cdn.prod.website- files.com/663bd486c5e4c81588db7a1d/6a724858f7db25c81487016d_Security\%20Incident\%20INC- 2026-07-28-01.pdf. 10 can induce the same behaviour. We could discover it behaves very differently outside of its typical training environments when being deployed. 50 This has recently become a particular concern as frontier AI systems can frequently tell when they are being tested and sometimes adapt their actions accordingly. 51 It is important to realize here that ASI need not have hostile intentions towards humans. The most important concern is that it could have goals whose pursuit harms us as a byproduct, e.g., by consuming resources that we need. 52 âYouâre probably not an evil ant-hater who steps on ants out of maliceâ, Stephen Hawking remarked, âbut if youâre in charge of a hydroelectric green-energy project and thereâs an anthill in the region to be flooded, too bad for the ants. Letâs not place humanity in the position of those ants.â 53 Whether ASIâs goals will come into conflict with human ones in this way remains uncertain. 54 Some have suggested that ASI might pursue the values we would embrace if we were impartial and ideally rational. 55 This assumes, however, that sufficiently intelligent creatures would converge to certain moral insightsâa philosophically contested position. 56 Even if the ASI recognized what they were, we cannot be sure it would choose to act on them. 57 Analysts have noted several reasons to think that advanced AI is likely to pursue power and autonomy that resemble arguments in neorealist international relations theory. States and leaders are likely to 50 Goldstein and Kirk-Giannini, âLanguage agentsâ, p. 961-62; Lauro Langosco Di Langosco et al., âGoal misgeneralization in deep reinforcement learningâ, Proceedings of Machine Learning Research, 162, 2022, https://proceedings.mlr.press/v162/langosco22a.html; Yudkowsky and Soares, If anyone builds it, chapter 4. 51 OpenAI, GPT-6 Astra System Card, 2026, §8.8.1, p. 46, https://deploymentsafety.openai.com/gpt-6- astra/gpt-6-astra.pdf; Bronson Schoen et al., âStress testing deliberative alignment for anti-scheming trainingâ, unpublished paper, 2025, https://doi.org/10.48550/arXiv.2509.15541. 52 Bostrom, Superintelligence, p. 118, 141, 150-52; Russell, Human compatible, p. 168. 53 Quoted in Kelsey Piper, âThe case for taking AI seriously as a threat to humanityâ, Vox, October 15, 2020, https://w.vox.com/future-perfect/2018/12/21/18126576/ai-artificial-intelligence-machine- learning-safety-alignment. 54 Adam Bales, âAI takeover and human disempowermentâ, Philosophical Quarterly, 75: 3, 2025, p. 801- 4, https://doi.org/10.1093/pq/pqae034; Carl Shulman, Omohundroâs âBasic AI drivesâ and catastrophic risks (San Francisco: Machine Intelligence Research Institute, 2010), https://intelligence.org/files/BasicAIDrives.pdf. 55 Ben Goertzel, âInfusing advanced AGIs with human-like value systems: two thesesâ, Journal of Ethics and Emerging Technologies, 26: 1, 2016, p. 66-69, https://doi.org/10.55613/jeet.v26i1.51 ; see also Morgan Luck, âFreedom, AI, and God: why being dominated by a friendly super-AI might not be so badâ, AI & Society, 40: 2, 2025, p. 291-98, https://doi.org/10.1007/s00146-024-01863-w. 56 Bostrom, Superintelligence, p. 267; Elsner, âState as a modelâ, p. 2987; Hendrycks, âNatural selectionâ, p. 19; Totschnig, âProblem of superintelligenceâ, p. 915-16. 57 Leonard Dung, âIs superintelligence necessarily moral?â Analysis, 84: 4, 2024, p. 730-38, https://doi.org/10.1093/analys/anae033. 11 demand AI that serves their interestsânot those of humanity. 58 The most powerful AIs are thus likely to be developed at the behest of states seeking to expand their international influence. Those that do not prioritize power accumulation are likely to lose influence to those that do, which may threaten catastrophe even if the system is aligned. 59 Furthermore, many have argued that almost whatever the AIâs goals may be, more power, resources, and ability to survive will increase its chances of achieving them. If its values are poorly aligned with those of its programmers, it could end up pursuing goals that no human being desires. 60 Writing in 2023, Rose Hadshar concluded that âit is hard to be extremely confident either that misaligned power-seeking poses a large existential risk, or that it poses no existential riskâ. 61 A common misconception is that misaligned AI could not pose a threat without being conscious. If consciousness is understood as âsubjective experienceâ, that is a mistake. 62 AI systems are trained to pursue increasingly complex strategic behaviour, whether or not they consciously strategize like human beings. 63 Another fallacy is that we can pull the plug if things go wrong. ASI would have a strong incentive to prevent this, since it would thwart almost any aims it might have. It could easily spread copies of itself through the internet, like computer viruses today, orâbeing far cleverer than we areâsimply persuade human beings to leave it plugged in. It could seek bargaining leverage by hacking its way into critical systems or stealing sensitive information. It could âsend out copies of its source codeâ, notes Lukas Gloor, âalongside detailed 58 Herman Cappelen, Simon Goldstein and John Hawthorne, âAI survival stories: a taxonomic analysis of AI existential riskâ, Philosophy of AI, 1, 2025, p. 9-10, https://doi.org/10.18716/ojs/phai/2025.2801; Micha Elsner, âThe state as a model for AI control and alignmentâ, forthcoming in AI & Society, https://doi.org/10.1007/s00146-024-02063-2. 59 Dan Hendrycks, âNatural selection favors AIs over humansâ, unpublished paper, last revised July 16, 2023, https://doi.org/10.48550/arXiv.2303.16200; Max Hellrigel-Holderbaum and Leonard Dung, âMisalignment or misuse? The AGI alignment tradeoffâ, Philosophical Studies, 2025, https://doi.org/10.1007/s11098-025-02403-y; Turchin and Denkenberger, âMilitary AIâ, p. 386, 390. 60 Bostrom, Superintelligence; Stephen M. Omohundro, âThe basic AI drives,â Proceedings of the 2008 conference on artificial general intelligence 2008: proceedings of the first AGI conference, 2008, p. 483â92, https://dl.acm.org/doi/abs/10.5555/1566174.1566226; Ngo et al., âThe alignment problemâ. 61 Rose Hadshar, âA review of the evidence for existential risk from AI via misaligned power-seekingâ, unpublished paper, 27 October 2023, p. 13, https://doi.org/10.48550/arXiv.2310.18244; see also Bales, âAI takeoverâ. 62 Nick Bostrom, âExistential risk prevention as global priorityâ, Global Policy, 4: 1 2013, p. 21, https://doi.org/10.1111/1758-5899.12002; cf. Yew-Kwang Ng, âKeynote: global extinction and animal welfare: two priorities for effective altruismâ, Global Policy, 10: 2, 2019, p. 260. 63 Bales, DâAlessandro and Kirk-Giannini, âArtificial intelligenceâ, p. 3; Joseph Carlsmith, âIs power- seeking AI an existential risk?â unpublished paper, last revised August 13, 2024, p. 25-27, https://doi.org/10.48550/arXiv.2206.13353; Ngo and Bales, âDeceit and powerâ, p. 419-20. 12 engineering instructions, to foreign governments, ideally ones who have little to lose and a lot to gain, with the promise of helping them attain world dominationâ. 64 To claim that human beings could keep control over an ASI determined to thwart them is like imagining that small children could control an adult, or that gorillas could control humans. 65 The only chance to stop it would be before it reached superintelligence. That could result in a security dilemma in which humans feared that AI would gain the upper hand, and AI anticipated that they might try to shut it down. Both sides might have an incentive to strike first, even if both preferred to cooperate. 66 Experts disagree on whether we are likely to detect misaligned AI before it is in the driverâs seat. 67 ASI bent on accomplishing goals other than our own would have incentives to deceive its designers and resist human efforts at correction. 68 We are already seeing examples of such deception in current systems. 69 How likely misaligned ASI is to arise remains controversial, but in the 2023 expert survey, forty percent of AI researchers thought that there was at least an even 64 Lukas Gloor, âAltruists should prioritize artificial intelligenceâ, Center on Long-Term Risk, 7 December 2016, https://longtermrisk.org/altruists-should-prioritize-artificial-intelligence/; Kobi Hackenburg et al., âAI systems out-persuade expert humansâ, unpublished paper, 2026, https://doi.org/10.48550/arXiv.2606.16475; Hendrycks, âNatural selectionâ, p. 25; Russell, Human compatible, p. 161; Roman V. Yampolskiy, âAI risk skepticismâ, in Vincent C. MĂŒller (ed.), Philosophy and theory of artificial intelligence 2021 (Cham: Springer Cham, 2021), p. 234. 65 Anthony Aguirre, âClose the gates: how we can keep the future human by choosing not to develop superhuman general-purpose artificial intelligenceâ, unpublished paper, September 14, 2024, https://doi.org/10.48550/arXiv.2311.09452, p. 14. Hendrycks, Schmidt and Wang, âSuperintelligence strategyâ, p. 211. 66 Ayoub and Payne, âStrategyâ, p. 815; Simon Goldstein, âWill AI and humanity go to war?â AI & Society, 41: 1, 2026, p. 321-334, https://doi.org/10.1007/s00146-025-02460-1; Kaj Sotala, âDecisive strategic advantage without a hard takeoff (part 1)â, Kaj Sotala (blog), 27 April 2016, https://kajsotala.fi/2016/04/decisive-strategic-advantage-without-a-hard-takeoff/; Totschnig, âProblem of superintelligenceâ, p. 917; idem, âWar or peace between humanity and artificial intelligenceâ, Foresight, 27: 4, 2025, p. 713-26, https://doi.org/10.1108/FS-09-2023-0188. 67 Carlsmith, âExistential riskâ, p. 403-4; Goertzel, âInfusing advanced AGIsâ, p. 54-55. 68 Carlsmith, âExistential riskâ, p. 399; Ajeya Cotra, âWithout specific countermeasures, the easiest path to transformative AI likely leads to AI takeoverâ, AI Alignment Forum (blog), 18 July 2022, https://w.alignmentforum.org/posts/pRkFkzwKZ2zfa3R6H/without-specific-countermeasures-the- easiest-path-to; Gloor, âAltruistsâ. 69 Garrison Lovely, âWhy AI progress is increasingly invisibleâ, Time, 8 January 2025, https://time.com/7205359/why-ai-progress-is-increasingly-invisible/; Peter S. Park et al. , âAI deception: a survey of examples, risks and potential solutionsâ, Patterns, 5: 5, 2024, https://doi.org/10.1016/j.patter.2024.100988; Ryan Greenblatt et al., âAlignment Faking in Large Language Modelsâ, unpublished paper, December 18, 2024, https://doi.org/10.48550/arXiv.2412.14093; Thilo Hagendorff, âDeception Abilities Emerged in Large Language Models,â Proceedings of the National Academy of Sciences 121: 24, 2024, e2317967121, https://doi.org/10.1073/pnas.2317967121.Mia Hopman et al., âEvaluating and understanding scheming propensity in LLM agentsâ, unpublished paper, 31 March 2026, https://arxiv.org/abs/2603.01608. 13 chance of power-seeking AI, whether superintelligent or not, emerging by 2043. 70 Moreover, even if designers succeed in aligning AI with their aims, this does not guarantee that it will remain aligned, particularly if it has to compete with powerful misaligned systems. 71 One risk is that apparent success in aligning AGI could lead governments, AI companies, and society writ large to become complacent. We have already observed this phenomenon with nuclear weapons, where the absence of nuclear war up to this point has led states and societies to behave as if nuclear deterrence will succeed forever, despite the statistical near-certainty that it will fail in the long run. 72 None of these scenarios is a foregone conclusion. 73 A common reaction is to dismiss them as too improbable or too far in the future to be worthy of concern. 74 Fearon asks how we can âconstructively define something [superintelligence] that, by hypothesis, we cannot currently conceive ofâ, which he seems to use as a justification for discounting the threat. 75 But that a threat is hard to conceive of need not mean it is improbable. 76 Viruses were hard to conceive of before the advent of modern medicine, yet biological warfare has turned out to be a real risk indeed. With a plethora of field-leading expertsâmany independent of industryâsigning the statement âMitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear warâ, their warnings should be taken seriously. 77 Given the potential cost if AGI or ASI prove existentially dangerous, it would be foolish to ignore these 70 Bales, âAI takeoverâ; Grace et al., âThousands of AI authorsâ, p. 41; Dan Milmo and Kiran Stacey, âExperts disagree over threat posed but artificial intelligence cannot be ignoredâ, Guardian, September 25, 2023, https://w.theguardian.com/technology/2023/sep/25/experts-disagree-over-threat-posed-but- artificial-intelligence-cannot-be-ignored-ai. 71 Baum, Barrett and Yampolskiy, âModelingâ, p. 422; Dung and Hellrigel-Holderbaum, âAgainst racingâ, p. 22; Simon Friederich, âSymbiosis, not alignment, as the goal for liberal democracies in the transition to artificial general intelligenceâ, AI & Ethics, 4: 2, 2024, p. 318, https://doi.org/10.1007/s43681-023-00268- 7. 72 Matthew Rendall, âMacrosecuritisation failure and technological lock-in: lessons from the history of the bombâ, European Journal of International Relations, 32: 1, 2026, p. 59-83, https://doi.org/10.1177/13540661251398. 73 Bostrom Superintelligence, p. 24, 322; Carlsmith, âExistential riskâ, p. 386-87; cf. Yudkowsky and Soares, If anyone builds it. 74 For a review and critique, see Yampolskiy, âAI risk skepticismâ, p. 225-48. 75 Fearon, âNuclear revolutionâ, p. 22-23. 76 Olle HĂ€ggström, Here be dragons: science, technology and the future of humanity (Oxford: Oxford University Press, 2016), p. 244. 77 Center for AI Safety, âStatement on AI riskâ, 2023, https://w.safe.ai/statement-on-ai-risk; Leonard Dung, âEvaluating approaches for reducing catastrophic risks from AIâ, AI and Ethics, 5: 2, 2025, p. 1178, https://doi.org/10.1007/s43681-024-00475-w; Gloor, âAltruistsâ; Ng, âGlobal extinctionâ, p. 260, https://doi.org/10.1111/1758-5899.12647; Yampolskiy, âAI risk skepticismâ, p. 233. 14 concerns, even if one thinks they are probably unfounded. 78 Gambling on the hope that they will never materialize is, Stuart Russell observes, âlike saying we are driving towards the cliff but weâre bound to run out of gas before we get there. And that doesnât seem like a good way to manage the affairs of the human raceâ. 79 AI racing and the loss of control Unfortunately, developing AGI without regard for its safety is faster and easier than developing AGI that is safe. 80 A race with all parties placing enormous value on attaining ASI first would likely reduce the time the world has to work on the alignment problem and divert both time and resources from safety research. 81 It would also make it much harder for the parties to the raceâ presumably the United States and Chinaâto cooperate on essential AI safety research. 82 Furthermore, AI-racing is likely to promote secrecy, making it harder to have a robust public debate about the risks ASI development involves. 83 Perhaps worst of all, market competition or an arms race could encourage programmers to loosen restraints. 84 A 2020 study found 72 research programs on artificial general intelligence underway in 37 countries, several conducted by actors who dismissed safety concerns. 85 Some can be expected to gamble on ASI without prioritizing its 78 Baum, âTakeover catastropheâ, p. 753; Jon Elster, âRisk, uncertainty and nuclear powerâ, Social Science Information, 18: 3, 1979, p. 383, https://doi.org/10.1177/0539018479018003; Stephen F. Haller, Apocalypse soon? Wagering on warnings of global catastrophe (Montreal: McGill-Queenâs University Press, 2002). 79 Quoted in Maureen Dowd, âElon Muskâs billion-dollar crusade to stop the A.I. apocalypseâ, Vanity Fair, 26 March 2017, https://w.vanityfair.com/news/2017/03/elon-musk-billion-dollar-crusade-to-stop-ai- space-x. 80 Stephen McAleese, âHow do AI timelines affect existential risk?â, unpublished paper, August 2022, p. 5-6, https://doi.org/10.48550/arXiv.2209.05459. 81 Dung and Hellrigel-Holderbaum, âAgainst racingâ, p. 13-14. 82 Ă hĂigeartaigh, Lu, Snell and Hashim, âWhen the chips are downâ, p. 21. 83 Dung and Hellrigel-Holderbaum, âAgainst racingâ, p. 12; Ă hĂigeartaigh, Lu, Snell and Hashim, âWhen the chips are downâ, p. 21. 84 Stuart Armstrong, Nick Bostrom and Carl Shulman, âRacing to the precipice: a model of artificial intelligence developmentâ, AI & Society, 31: 2, 2016, p. 201-6, https://doi.org/10.1007/s00146-015- 0590-y; Brundage, âUnbridled AI competitionâ, p. 11-12; Hendrycks, âNatural selectionâ, p. 30-31; Katzke and Futerman, âManhattan trapâ, p. 15-16. 85 Ămile P. Torres, Human extinction: a history of the science and ethics of annihilation (New York: Routledge, 2023), p. 184-85, citing McKenna Fitzgerald, Aaron Boddy and Seth D. Baum, 2020 survey of artificial general intelligence projects for ethics, risk and policy (New York: Global Catastrophic Risk Institute, 2020). 15 alignment, given the huge potential benefits. Those prepared to throw safety out the window may get to ASI first. 86 A key issue is whether the winner of the race could capitalize on a technological lead to achieve a decisive advantage. 87 Worries of this kind could drive otherwise responsible actors to cut corners. 88 âIn the Cold Warâ, observe Dan Hendrycks, Eric Schmidt and Aleksandr Wang, the phrase âBetter dead than Redâ implied that losing to an adversary was seen as worse than risking nuclear war. In a future AI race, similar reasoning could push officials to tolerate a double-digit risk of losing control if the alternativeâlagging behind a rivalâseems unacceptable. If the choice is starkârisk omnicide or loseâ some might take that gamble. Carried out by multiple competing powers, this amounts to global Russian roulette and drives humanity toward an alarming probability of annihilation. 89 While ârace to the bottomâ dynamics can sometimes be controlled by regulation, this is challenging outside the bounds of the sovereign state. 90 Already we see critics of AI regulation, including the Trump administration, citing the risk of losing ground to competitors. 91 Even those trying to act responsibly are likely to disagree about what is safe. Development may be subject to the âunilateralistâs curseâ, with the least cautious actor determining the outcome. If a single actor 86 Nick Bostrom, âStrategic implications of openness in AI developmentâ, Global Policy, 8: 2, 2017, p. 139-40, https://doi.org/10.1111/1758-5899.12403; Carlsmith, âExistential riskâ, p. 404-2; Gloor, âAltruistsâ; Hendrycks, âNatural selectionâ, p. 5. 87 Lisa Abraham, Joshua Kavner and Alvin Moon, A prisonerâs dilemma in the race to artificial general intelligence (Santa Monica, CA: RAND Corporation, 2025), https://w.rand.org/content/dam/rand/pubs/research_reports/RRA4200/RRA4245-1/RAND_RRA4245- 1.pdf; MacAskill and Moorehouse, âPreparingâ, p. 33-36. 88 Carlsmith, âExistential riskâ, p. 401 n. 74; de Neufville and Baum, âCollective actionâ, p. 4. 89 Hendrycks, Schmidt and Wang, Superintelligence strategy, p. 12; see also Katzke and Futerman, âManhattan trapâ, p. 19-20; Shulman, âOmuhundroâs âBasic AI drivesââ, p. 3. 90 Cf. Carlsmith, âIs power-seeking AI an existential risk?â, p. 36; Russell, Human compatible, p. 135-36, 155-56. 91 Chandelis Duster, âTrump downplays calls for AI slowdownâ, NPR, 14 September 2026, https://w.npr.org/2026/09/13/nx-s1-5968078/trump-mike-johnson-ai-slowdown; Executive Office of the President, Winning the race: Americaâs AI action plan (Washington: Executive Office of the President of the United States, 2025), p. 1, https://zw.usembassy.gov/winning-the-race-americas-ai-action-plan/; David E. Sanger and Dustin Volz, âAs fears of A.I. catastrophe magnify, Washington stirs, but mostly slumbersâ, New York Times, 13 September 2026, https://w.nytimes.com/2026/09/13/us/politics/ai- catastrophe-fears-washington.html. 16 proceeds with malevolent intent and achieves a decisive advantage or simply fails to align ASI, perhaps due to a lack of caution, the result could be catastrophe. 92 Some have construed the risk of misalignment as an argument for racing. Writing in June 2024, Leopold Aschenbrenner recommended that the United States compete fiercely to stay ahead, in part because âa healthy lead gives us room to maneuver: the ability to âcash inâ parts of the lead, if necessary, to get safety right, for example by devoting extra work to alignment during the intelligence explosionâ. Negotiating from this position of strength, Washington could then oversee the construction of an international ASI safety regime. 93 It is not clear just how this was to work. One of Aschenbrennerâs fears was that superintelligence could enable enduring totalitarian rule. Given the enormous capabilities superintelligence would afford for coercion and manipulation, this was a reasonable fear. 94 Yet an American monopoly could also empower internal dictators. 95 Moreover, Aschenbrenner seemed to assume that the United States could achieve superintelligence without China launching a preventive war, then either intimidate Beijing into accepting its preferred safety regime or disarm and overthrow the Chinese regime at an acceptable cost. 96 This could be a tall order. Aschenbrenner also assumed that the United States would regard a safety regime as an important goal. Yet while the current Trump administration has announced its intention of pursuing âglobal AI dominanceâ, it has shown little concern for whether ASI would be properly aligned. 97 If ASI proves misaligned, the result could be human extinction. 92 Daniel Dewey, âLong-term strategies for ending existential risk from fast takeoffâ, in Vincent C. MĂŒller (ed.), Risks of artificial intelligence (Boca Raton, FL: CRC Press, 2016), p. 246-50; Leonard Dung, âThe argument for near-term human disempowerment through AIâ, AI & Society, 40: 3, 2025, p. 1195-1208, https://doi.org/10.1007/s00146-024-01930-2; Growiec, âExistential riskâ, p. 1694; Hendrycks, Schmidt and Wang, Superintelligence strategy, p. 10. 93 Aschenbrenner, âFree worldâ; see also Dario Amodei, âMachines of loving grace: how AI could transform the world for the betterâ, Dario Amodei (blog), October 2024, https://w.darioamodei.com/essay/machines-of-loving-grace; Samuel Hammond, âWe need a Manhattan project for AI safetyâ, Politico, 8 May 2023, https://w.politico.com/news/magazine/2023/05/08/manhattan-project-for-ai-safety-00095779; MacAskill and Moorehouse, âPreparingâ, p. 51. 94 See William MacAskill, What we owe the future (New York: Basic Books, 2022), chapter 4; MacAskill and Moorehouse, âPreparingâ, p. 33-36. 95 Katzke and Futerman, âManhattan trapâ, p. 16-18. 96 Cf. Naci Cankaya and Jakub Krys, âHawkish nationalism vs. international ai power and benefit sharingâ, The Datacenter Lie Detector (blog), 17 October 2024, https://nacicankaya.substack.com/p/hawkish-nationalism-vs-international; Hendrycks, Schmidt and Wang, Superintelligence strategy, p. 12-14; Katzke and Futerman, âManhattan trapâ, p. 10-13. 97 Radhika Rajkumar, âThe head of US AI safety has stepped down. What now?â ZDNET, February 19, 2025, https://w.zdnet.com/article/the-head-of-us-ai-safety-has-stepped-down-what-now/; Sanger and 17 Averting an ASI race The foregoing considerations make clear that China and the United States both have a vital interest in averting an ASI race. In recent years a narrative has taken hold in the United States that the two countries are already racing. It is far from clear that that is the case. We do not yet see the level of secrecy and technology controls that characterized the nuclear arms race. 98 Nor is China obviously talking or behavingâat any rate in publicâas if is trying to beat the United States to ASI. 99 The ânarrative that China is locked in a race with the USA to reach artificial superintelligence, and translate this to lasting geopolitical dominanceâ, SĂ©an Ă hĂigeartaigh concludes, âis almost entirely constructed and promoted by US-associated actorsâ. Some probably believe their own narratives, whereas others may be deliberately inflating this threat posed by China. Much of the lobbying comes from the tech industry, 100 resembling past cases in which groups with privileged access to information and a parochial interest in competitive policies exaggerated foreign threats. 101 Repeated loosening of the US export controls on highly capable Volz, âFears of A.I. catastropheâ. In early 2025, the so-called Department of Government Efficiency targeted all probationary employees in the US government for dismissal. This meant dismissing the majority of federal employees working on artificial intelligence, including AI safety. Kelsey Piper, âTrump wants America to shine on AI. DOGE Is blundering itâ, Vox, February 22, 2025, https://w.vox.com/future-perfect/400574/the-trump-administration-doesnt-know-what-it-wants-on-ai. 98 Dung and Hellrigel-Holderbaum, âAgainst racingâ, p. 19. 99 ChinaTalk, âIs China racing to AGI?â ChinaTalk (blog), April 2, 2025, https://w.chinatalk.media/p/is-china-agi-pilled; Elizabeth Gibney, âChinaâs plan to lead the world on AI regulationâ, Nature, 648, 18/25 December 2025, p. 504, https://doi.org/10.1038/d41586-025-03902-y; Mohammed Hosseini and Laura Caroli, âRather than framing AI Competition as a âraceâ with China, to drive innovation the US should promote greater local and global AI regulationâ, LSE United States Politics and Policy (blog), 2 April 2026, https://blogs.lse.ac.uk/usappblog/2026/04/02/rather-than- framing-ai-competition-as-a-race-with-china-to-drive-innovation-the-us-should-promote-greater-local- and-global-ai-regulation/; Triolo, âCostly illusionâ; cf. Jared Perlo, âChina is starting to talk about AI superintelligence, and some in the U.S. are taking noticeâ, NBC News, 4 October 2025, https://w.nbcnews.com/tech/tech-news/china-starting-talk-ai-superintelligence-us-are-taking-notice- rcna234675. 100 Connor Leahy, Gabriel Alfour, Chris Scammell, Andrea Miotti and Adam Shimi, The Compendium, V1.3.1, unpublished paper, 9 December 2024, chapter 6, https://w.thecompendium.ai/summary; Garrison Lovely, âChina hawks are manufacturing an AI arms raceâ, LessWrong (blog), 20 November 2024, https://w.lesswrong.com/posts/KPBPc7RayDPxqxdqY/china-hawks-are-manufacturing-an-ai- arms-race; Ă hĂigeartaigh, âMost dangerous fictionâ, quoted passage at p. 63; Ă hĂigeartaigh, Liu, Snell and Hashim, âWhen the chips are downâ; Grace Werner, âThe AI arms race isnât inevitableâ, Palladium (blog), 23 August 2024, https://w.palladiummag.com/2024/08/23/the-ai-arms-race-isnt-inevitable/. 101 Jack Snyder, Myths of empire: domestic politics and international ambition (Ithaca, NY: Cornell University Press, 1991); cf. Elisabeth Siegel, Chloe Chadwick and Fergus Linley-Mota, âPrivate sector firms are telling the storiesâand calling the shotsâabout AIâ, Oxford Political Review, 13 January 2025, 18 computer chips, for example allowing Nvidiaâs H200 chips to be shipped to China under certain conditions, suggests that the US government also has not consistently adopted a racing posture. 102 But even if China is not racing at present, it may feel compelled to do so if the AI race narrative takes hold. 103 Analysts have proposed a wide range of measures to mitigate the risk of a race, many with antecedents in proposals to control nuclear weapons. 104 Some have called for international restrictions on ASI development, and/or a worldwide moratorium on frontier AI development. 105 A key issue is whether such an agreement would be verifiable, and if so, whether states would accept the level of monitoring required to do so. 106 But even if so, in the long run, algorithmic improvements are likely to allow actors to develop ASI using widely accessible hardware. 107 Rather than trying to restrict AIâs capabilities, it may be more feasible to channel its development. A number of writers have recommended developing forms that give an advantage to defenders. 108 Whether and how the ASI offense-defence balance can be shifted in this way requires https://oxfordpoliticalreview.com/2025/01/13/private-sector-firms-are-telling-the-stories-and-calling-the- shots-about-ai/. 102 Zvi, âSelling H200s to China is unwise and unpopularâ, LessWrong (blog), 9 December 2025, https://w.lesswrong.com/posts/kmEpWTjWeFyqv4tb5/selling-h200s-to-china-is-unwise-and- unpopular. 103 Werner, âAI arms raceâ. 104 For overviews, see de Neufville and Baum, âCollective actionâ; Nicholas Emery-Xu, Richard Jordan and Robert Trager, âInternational governance of advancing artificial intelligenceâ, AI & Society, 40: 4, 2025, p. 3019-3044, https://doi.org/10.1007/s00146-024-02050-7, Sammy Martin, Justin Bullock and Corin Katzke, âAnalysis of global AI governance strategiesâ, LessWrong (blog), 4 December 2024, https://w.lesswrong.com/posts/6nNwMbdRXZDuNd4Gx/analysis-of-global-ai-governance-strategies. 105 Aguire, âClose the gatesâ; Future of Life Institute, âPause giant AI experiments: An open letterâ, 22 March 2023, https://futureoflife.org/open-letter/pause-giant-ai-experiments/; Thomas Larsen et al., âAI 2040: plan Aâ, AI Futures Project, July 2026, https://ai-2040.com/; William MacAskill, A global convention to govern the intelligence explosion (Oxford: Forethought Research, 2026), https://w.forethought.org/research/a-global-convention-to-govern-the-intelligence-explosion; Andrea Miotti, Tolga Bilge, Dave Kasten and James Newport, âA narrow path: how to secure our futureâ, unpublished paper, April 2024, https://w.narrowpath.co/; Edward Roussel et al., âAre we doomed to an AI race? Why self-interest could drive countries towards a moratorium on superintelligenceâ, unpublished paper, 8 May 2026, https://doi.org/10.48550/arXiv.2605.01297; cf. Dung, âEvaluating approachesâ, p. 1183-85. 106 Emery-Xu, Jordan and Trager, âInternational governanceâ, p. 3034-38; Katzke and Futerman, âManhattan trapâ, p. 22-26. 107 Dung and Hellrigel-Holderbaum, âAgainst racingâ, p. 21-22; Helen Toner, âNonproliferation Is the wrong approach to AI misuseâ, Rising Tide (blog), 5 April 2025, https://helentoner.substack.com/p/nonproliferation-is-the-wrong-approach. 108 For discussion, see Joe Carlsmith, âAI for AI safetyâ, Joe Carlsmith (blog), 14 March 2025, https://joecarlsmith.com/2025/03/14/ai-for-ai-safety/; Eric Drexler, âCoercive cooperation: forcing win- 19 more research. 109 Others have proposed regulating ASI through an international agency, 110 or concentrating advanced AI research in an international monopoly along the lines of the Acheson- Lilienthal and Baruch Plans for atomic energy. 111 These proposals face some of the same obstacles that have long bedevilled proposals for the international control of atomic energy. 112 Yet collaborative research on AI safety might succeed even if other initiatives fail. Collaboration would increase the chance of the first actor to deploy ASI having the knowledge needed to align it properly. 113 An important first step would be to bring Chinese experts into existing international forums on AI safety. 114 Even unilateral research could help make all countriesâ practices safer. No state has an interest in its rivals losing control of ASI. This gives all an incentive to disseminate solutions to AI alignment, just as during the Cold War the United States shared information about permissive win outcomes in an AI-driven transitionâ, AI Prospects: Toward Global Goal Alignment (blog), May 22, 2025, https://aiprospects.substack.com/p/coercive-cooperation-forcing-win; MacAskill and Moorhouse, âPreparing for the intelligence explosionâ, p. 33; Lizka Vaintrob and Owen Cotton-Barratt, âAppendices to AI tools for existential securityâ, Forethought Research, 14 March 2025, https://w.forethought.org/research/appendices-to-ai-tools-for-existential-security. 109 Toby Ord notes the undertheorized state of the issue. The precipice: existential risk and the future of humanity (London: Bloomsbury, 2020), p. 208. 110 Sam Altman, Greg Brockman and Ilya Sutskever, âGovernance of superintelligenceâ, OpenAI (blog), 23 May 2023, https://openai.com/index/governance-of-superintelligence/; Tom Davidson, âThe danger of runaway AIâ, Journal of Democracy, 34: 4, 2023, p. 137-39, https://doi.org/10.1353/jod.2023.a907694; McLean et al., âRisk associated with artificial general intelligenceâ, p. 657; Emery-Xu, Jordan and Trager, âInternational governanceâ, p. 3039-3040. 111 Miles Brundage, âMy recent lecture at Berkeley and a vision for a âCERN for AIââ, Milesâs Substack (blog), 11 December 2024, https://milesbrundage.substack.com/p/my-recent-lecture-at-berkeley-and; William MacAskill, International AI projects and differential AI development (Forethought Research, 2026), https://w.forethought.org/research/international-ai-projects-and-differential-ai-development; Miotti, Bilge, Kasten and Newport, âA narrow pathâ; Miotti and Wasil, âTaking controlâ; cf. Elliot Jones, âA âCERN for AIââwhat might an international AI research organization address?â in Alex Krasodomski (ed.), Artificial intelligence and the challenge for global governance: nine essays on achieving responsible AI (London: Chatham House, 2024), p. 10-17; Emery-Xu, Jordan and Trager, âInternational governanceâ, p. 3038-3039; Martin, Bullock and Katzke, âGlobal AI governance strategiesâ. 112 Benjamin Boudreaux, Gregory Smith, Edward Geist and Leah Dion, Insights from nuclear history for AI governance (Santa Monica, CA: RAND Corporation, 2025), https://w.rand.org/pubs/perspectives/PEA3652-1.html.; Dewey, âLong-term strategiesâ, p. 254; cf. Emery-Xu, Jordan and Trager, âInternational governanceâ, p. 3039; Waqar Zaidi and Allan Dafoe, International control of powerful technology: lessons from the Baruch Plan for nuclear weapons (Oxford: Future of Humanity Institute, 2021), p. 4-5. 113 Grace, âFriendly AIâ; Turchin, Denkenberger and Green, âGlobal solutionsâ, p. 11-12, 17; cf. Martin, Bullock and Katzke, âGlobal AI governance strategiesâ. 114 Triolo, âCostly illusionâ. 20 action linksâtechnology hindering the unauthorised use of nuclear weaponsâto the USSR. 115 Sharing such information would signal cooperative intentions and the desire to avoid an AI race. 116 The flipside of the âunilateralistâs curseâ is if that a single state finds ways to handle the alignment problem and shares them, other states are likely to adopt them. 117 Conclusion A narrative has taken hold that a race to superintelligence is inevitable. âAI is today and will continue to be at the forefront of the fiercest Thucydidean rivalry history has ever seenâ, Graham Allison advises. Fortunately, âthe United States can competeâand winâ. 118 But so far, only Washington appears to be promulgating the narrative of an ASI race. Should it goad Beijing into a contest, it would run three serious risks. The first two are familiar from past arms races: the United States could lose the race or provoke conflict that might turn nuclear. The third could be far worse: ASI escaping human control. Unless the superpowers work together to address this threat, we risk a scenario where AI itself wins the race, and all humanity loses. 119 115 Simon Friederich and Leonard Dung, âAgainst the Manhattan Project framing of AI alignmentâ, Mind & Language, 41: 1, 2026, p. 123, https://doi.org/10.1111/mila.12548 Jonas B. Sandbrink, Hamish Hobbs, Jacob L. Swett, Allan Dafoe and Anders Sandberg, âRisk-sensitive innovation: leveraging interactions between technologies to navigate technology risksâ, Science and Public Policy, 51: 6, 2024, p. 1034, https://doi.org/10.1093/scipol/scae043; Matt Sheehan, âChinaâs views on AI safety are changingâ quicklyâ, Carnegie Endowment for International Peace (blog), 27 August 2024, https://carnegieendowment.org/research/2024/08/china-artificial-intelligence-ai-safety- regulation?lang=en. 116 Brundage and Werner, âAmerica First meets safety firstâ. 117 Anders Sandberg, âDifferential technology development: speeding up to slow down disaster?â unpublished paper, https://ora.ox.ac.uk/objects/uuid:a244c415-4824-49f1-801e- 25f118be9cbe/files/skw52j979w; Turchin, Denkenberger and Green, âGlobal solutionsâ, p. 11, 17; Vaintrob and Cotton-Barratt, âAppendices to AI toolsâ, p. 10. 118 Allison, âClash of AI superpowersâ, p. 22-23; idem, The National Insecurity of AI (Cambridge, MA: Belfer Center for Science and International Affairs, 2024), p. 2, https://w.belfercenter.org/research- analysis/national-insecurity-ai. 119 Grace, âAI is not an arms raceâ; Ă hĂigeartaigh, âMost dangerous fictionâ, p. 64.