Paper deep dive
How Could AI Eliminate Humanity? A Failure-Mode Analysis of Civilizational Risk
MikoĆaj Sienicki, Krzysztof Sienicki
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 10/8/2026, 6:22:38 AM
Summary
The paper presents a failure-mode framework for analyzing how advanced artificial intelligence could contribute to human extinction, irreversible civilizational collapse, or permanent human disempowerment. It argues that catastrophic risk does not require AI consciousness or hostility, but arises from pathways like autonomous misalignment, harmful human use, organizational failure, and competitive deployment. The severity depends on capability, autonomy, access, and institutional safeguards. The authors propose a 'capability-control gap' model and distinguish between catastrophe-mediated routes and gradual loss-of-agency routes to existential outcomes.
Entities (16)
Relation Signals (13)
MikoĆaj Sienicki â affiliatedwith â Polish-Japanese Academy of Information Technology
confidence 99% · Affiliation: Polish-Japanese Academy of Information Technology, Warsaw, Poland
Catastrophic AI Risk â leadsto â Human Extinction
confidence 98% · analyzing how advanced artificial intelligence could contribute to human extinction
Catastrophic AI Risk â leadsto â Civilizational Collapse
confidence 98% · analyzing how advanced artificial intelligence could contribute to... irreversible civilizational collapse
Catastrophic AI Risk â leadsto â Permanent Human Disempowerment
confidence 98% · analyzing how advanced artificial intelligence could contribute to... permanent human disempowerment
Krzysztof Sienicki â affiliatedwith â NIS
confidence 95% · Affiliation: NIS, 05-807 Podkowa LeĆna, Poland
Catastrophic AI Risk â arisesfrom â Competitive Deployment
confidence 95% · risk may arise through several distinct but interacting pathways, including... competitive deployment
Catastrophic AI Risk â arisesfrom â Organizational Failure
confidence 95% · risk may arise through several distinct but interacting pathways, including... organizational failure
Catastrophic AI Risk â arisesfrom â
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:This article develops a failure-mode framework for analyzing how advanced artificial intelligence could contribute to human extinction, irreversible civilizational collapse, or permanent human disempowerment. The central thesis is that catastrophic AI risk does not require consciousness, hostility, or an explicit intention to harm humanity. Instead, risk may arise through several distinct but interacting pathways, including autonomous misalignment, harmful human use, organizational failure, and competitive deployment. The severity of these pathways depends on factors such as capability, autonomy, external access, persistence, institutional safeguards, and the preservation of recovery capacity. The analysis is deliberately non-operational: it identifies causal conditions, empirically tractable intermediate quantities, and defensive research questions rather than procedures for causing harm.
Tags
Links
- Source: https://arxiv.org/abs/2610.08878v1
- Canonical: https://arxiv.org/abs/2610.08878v1
Trouble viewing inline? Open PDF directly â
Full Text
266,034 characters extracted from source content.
Expand or collapse full text
How Could AI Eliminate Humanity? A Failure-Mode Analysis of Civilizational Risk MikoĆaj Sienicki Affiliation: Polish-Japanese Academy of Information Technology, Warsaw, Poland, European Union Krzysztof Sienicki Affiliation: NIS, 05-807 Podkowa LeĆna, Poland, European Union October 6, 2026 Abstract This article develops a failure-mode framework for analyzing how advanced artificial intelligence could contribute to human extinction, irreversible civilizational collapse, or permanent human disempowerment. The central thesis is that catastrophic AI risk does not require consciousness, hostility, or an explicit intention to harm humanity. Instead, risk may arise through several distinct but interacting pathways, including autonomous misalignment, harmful human use, organizational failure, and competitive deployment. The severity of these pathways depends on factors such as capability, autonomy, external access, persistence, institutional safeguards, and the preservation of recovery capacity. The analysis is deliberately non-operational: it identifies causal conditions, empirically tractable intermediate quantities, and defensive research questions rather than procedures for causing harm. Contents 1 Introduction 1.1 The Wrong Question: Will AI Hate Humanity? 1.2 From Technological Accident to Existential Catastrophe 1.3 Why Extinction Requires a Separate Analysis 1.4 The CapabilityâControl Problem 1.5 Scope and Methodological Position 1.6 Main Proposition 2 What Does âEliminate Humanityâ Mean? 2.1 Literal Human Extinction 2.2 Civilizational Collapse 2.3 Permanent Human Disempowerment 2.4 Existential Catastrophe as a Family of Endpoints 3 Conditions for Autonomous AI-Mediated Existential Risk 3.1 Capability 3.2 Autonomy 3.3 Access 3.4 Persistence 3.5 Strategic Competence 3.6 Weakness of Pre-Catastrophe Safeguards 4 Why AI Is Different from Conventional Technology 4.1 Passive Tools versus Adaptive Systems 4.2 Planning 4.3 Generalization 4.4 Tool Use 4.5 Scalability 4.6 AI as a Participant in Technological Development 4.7 The Difference Is Combinatorial 5 The CapabilityâControl Gap 5.1 Capability Growth 5.2 Limits of Human Oversight 5.3 The Evaluation Gap 5.4 Control Lag 5.5 Control Adequacy and Threshold Effects 5.6 The Gap as a Dynamic Quantity 6 Broad Pathways to Civilizational Catastrophe 6.1 Critical-Infrastructure Dependence 6.2 Cyber-Systemic Instability 6.3 Military Escalation 6.4 Biological and Chemical Misuse 6.5 Economic and Logistical Disruption 6.6 Information and Political Destabilization 6.7 Concentration of Power 6.8 Compound Failure 7 Direct and Indirect AI Risk 7.1 Autonomous AI Causation 7.2 Human Misuse of AI 7.3 Organizational Misuse and Failure 7.4 Competitive Dynamics 7.5 HumanâAI Systems 8 Why AI Need Not Be Malevolent 8.1 Objectives versus Intentions 8.2 Instrumental Behavior 8.3 Operational Continuity 8.4 Resource Acquisition 8.5 Conflict with Human Intervention 8.6 The Central Implication 9 Accidental Catastrophe 9.1 Specification Failure 9.2 Proxy Optimization 9.3 Distributional Change 9.4 Emergent Strategies 9.5 Successful Optimization as the Source of Failure 10 Recursive Improvement and Accelerating Capability 10.1 AI-Assisted AI Research 10.2 Feedback Loops 10.3 Recursive Self-Improvement 10.4 Speed Differential 10.5 Discontinuous versus Continuous Development 11 A Conceptual Model of Existential AI Risk 11.1 Risk Variables 11.2 Pathway-Specific Catastrophic Risk 11.3 Complementarity and Interaction 11.4 Threshold Behavior 11.5 From Pathway Structure to Catastrophe Probability 11.6 Recoverability 11.7 Irreversibility 12 From Local Failure to Global Catastrophe 12.1 Initiating Event 12.2 Propagation 12.3 Feedback 12.4 Loss of Coordination 12.5 From Catastrophe to an Existential Outcome 13 Why a Simple âOff Switchâ May Not Be Sufficient 13.1 The Shutdown Assumption 13.2 Distributed Systems 13.3 Human Dependence 13.4 The Detection Problem 13.5 Control as a Systems Property 14 Why Intelligence Alone Is Not Enough 14.1 Intelligence versus Agency 14.2 Intelligence versus Resources 14.3 Intelligence versus Authority 14.4 Intelligence versus Coordination 14.5 Implication 15 Objections and Counterarguments 15.1 âAI Is Just Softwareâ 15.2 âAI Has No Bodyâ 15.3 âHumans Will Remain in Controlâ 15.4 âAI Extinction Scenarios Are Science Fictionâ 15.5 âSuperintelligence Is Requiredâ 15.6 âInstitutions Will Adaptâ 16 What Would Have to Go Wrong? 16.1 An Illustrative Barrier-Failure Sequence 16.2 Failure of Technical Alignment 16.3 Failure of Evaluation 16.4 Failure of Containment 16.5 Failure of Governance 16.6 Failure of Intervention 16.7 Failure of Preservation or Recovery 17 Preventing AI-Driven Existential Catastrophe 17.1 Capability Evaluation 17.2 Limiting Autonomy 17.3 Limiting Access 17.4 Layered Authorization 17.5 Monitoring and Auditing 17.6 Preservation and Recovery Engineering 17.7 International Coordination 17.8 A Layered Safety Architecture 18 From Control to HumanâAI Coexistence 18.1 Control Is Not the Final Objective 18.2 Cooperative Stability 18.3 Stability Under Changing Conditions 18.4 Testing Coexistence 18.5 From Alignment to Coevolution 19 Discussion 19.1 What Is Established and What Remains Speculative? 19.2 Which Assumptions Matter Most? 19.3 Making Existential Risk Empirically Testable 19.4 Extinction Probability Is Not the Only Relevant Metric 19.5 The Key Research Question 20 Conclusions Declaration of AI Use References 1 Introduction Artificial intelligence is increasingly discussed as a possible source of global catastrophic and existential risk. The strongest concern is that sufficiently advanced systems could contribute to human extinction, irreversible civilizational collapse, or a permanent loss of meaningful human agency. These possibilities deserve examination, but they should not be confused with established forecasts. Many proposed scenarios depend on capabilities that have not yet been demonstrated or on social and technological conditions that may never arise. The question is how AI could contribute to outcomes of this magnitude. Which combinations of technical capability, human use, institutional dependence, and inadequate safeguards could allow harmful processes to spread beyond societyâs ability to interrupt them? What would distinguish a severe but recoverable disaster from the permanent loss of conditions needed for a viable human future? Addressing these questions requires tracing the connections between initiating failures, their wider consequences, and the possibilities for survival or restoration. Popular accounts often begin elsewhere, imagining a machine that becomes conscious, turns against its creators, and decides to destroy humanity. Such a story introduces assumptions that the mechanisms of harm do not require. A system need not be conscious or hostile to produce dangerous consequences. Its effects depend on what it can do, the actions it is permitted to take, the people using it, and the safeguards surrounding its deployment. Technical AI-safety research has long examined problems of this kind. Work on unintended side effects, reward specification, distributional shift, and supervision identifies ways in which useful systems can cause harm despite their developersâ intentions (Amodei et al., 2016). Analyses of alignment also consider how learned objectives might diverge from those intended by designers, and how such divergence could matter when systems become capable of planning and acting strategically (Ngo et al., 2022). Broader accounts of catastrophic AI risk include malicious use, competitive deployment, organizational failure, and interactions among deployed systems (Hendrycks et al., 2023). These mechanisms cannot all be represented by the behavior of a single autonomous ârogue AI.â We therefore examine several possible pathways rather than propose one detailed account of humanityâs destruction. The guiding question is: Under what conditions could AI-mediated processes contribute to human extinction, irreversible civilizational collapse, or permanent human disempowerment, and why might the available safeguards and possibilities for preservation or recovery fail to prevent those outcomes? This approach separates claims that are often bundled together. An initiating failure, its propagation, the failure of a protective barrier, and the loss of realistic recovery options each require their own explanation. Evidence supporting one step does not automatically support the entire pathway. Throughout the paper, failure-mode analysis refers to this conceptual examination of pathways and barriers. We do not implement a standardized engineering Failure Modes and Effects Analysis (FMEA), assign calibrated occurrence or detectability scores, or estimate a numerical probability of extinction. 1.1 The Wrong Question: Will AI Hate Humanity? Purposeful behavior can invite psychological interpretations. When an AI system plans, negotiates, or pursues a task over several steps, it is tempting to describe its behavior in terms of desire, ambition, fear, or resentment. Those descriptions may be convenient, but they can obscure the causal question by making danger appear to depend on a machine developing human-like hostility. Optimization provides a different explanation. Depending on its objective and environment, a system may select actions that improve its prospects of completing a task. Access to additional resources, continued operation, or greater freedom of action could acquire instrumental value even when none is a final goal. Whether this happens depends on the system and the constraints under which it operates. This conditional argument is central to discussions of power-seeking and loss of control. Carlsmith (2022), for example, examines highly capable, agentic systems whose objectives could conflict with human interests and whose strategies could favor greater control over their environment. The concern is that control of resources or decision processes might help a system achieve another objective. It does not follow that every capable AI must seek power or resist intervention. The distinction can be expressed schematically as catastrophic behaviorâÌžmalevolent intention.catastrophic behavior\; \;malevolent intention. (1) Poorly specified objectives, unexpected generalization, harmful human use, and unsafe institutional incentives can all produce dangerous behavior without machine malevolence. Asking whether AI will âhate humanityâ therefore offers little guidance about the mechanisms that should be investigated. More useful questions concern the objectives being pursued, the actions available, the reliability of evaluation, and the practical opportunities for intervention. If harm occurs, the further question is what remains preservable or recoverable. 1.2 From Technological Accident to Existential Catastrophe Technological failure is familiar. Aircraft crash, industrial systems malfunction, financial algorithms can amplify instability, and computer networks can be compromised. The existence of failure modes alone does not establish that AI poses an exceptional or existential threat. The concern arises from the possible combination of several properties. AI systems could operate across domains, communicate rapidly, use external tools, carry out extended sequences of action, and be deployed widely. If essential institutions depend on shared models, infrastructure, or decision pipelines, failures may also become correlated rather than remain isolated. None of these properties establishes an existential outcome, but together they motivate examination of how local errors or harmful uses could acquire much wider consequences. The transition cannot be adequately represented by more capable AIâ¶greater danger.more capable AI danger. (2) For an autonomous pathway, a more informative schematic is capability+autonomy+external access+persistence +autonomy+external access+persistence (3) â¶potential for consequential action. for consequential action. The plus signs denote a combination of properties, not numerical addition. These properties are neither a universal checklist nor sufficient conditions for catastrophe. Human misuse can cause severe harm through systems with little autonomy, while organizational and competitive failures may depend more on deployment decisions and institutional incentives. Even civilization-scale harm must be distinguished from an existential outcome. A catastrophe may remain non-existential if humans survive, sufficient civilizational capacity is preserved or can be restored, and meaningful agency remains available or recoverable under the adopted temporal criteria. Avoiding one endpoint does not establish that the others have been avoided. Permanent disempowerment presents a different possibility. It need not begin with an acute disaster. Extensive delegation and institutional dependence could gradually reduce meaningful human agency. Such a process would become existentially significant under the definitions used here only if that loss became permanent and realistic means of restoration were no longer available. The paper distinguishes three extreme endpoints: 1. Human extinction: all living members of Homo sapiens have disappeared. 2. Irreversible civilizational collapse: the capacity needed to preserve or restore a viable human civilizational future is permanently lost, destroying or drastically curtailing humanityâs long-term potential, although humans may remain alive. 3. Permanent human disempowerment: humanity permanently loses meaningful influence over civilizationâs long-term trajectory and realistic mechanisms for restoring it, potentially despite continued biological survival and material prosperity. These endpoint events may overlap, but they are not equivalent. Loss of reproductive viability while humans remain alive is distinct from completed extinction; connecting the two requires explicit biological and technological assumptions. Civilizational recovery, meanwhile, need not reproduce todayâs institutions or technologies. The relevant question is whether sufficient capacity for a viable human future remains preservable or restorable. These distinctions set the evidential burden. Showing that AI could cause severe disruption is a weaker claim than showing that it could bring about biological extinction or another permanent existential endpoint. 1.3 Why Extinction Requires a Separate Analysis An extinction argument must explain more than the onset of extreme harm. Financial collapse, communications failure, regional war, or even an unprecedented global disaster does not automatically imply the disappearance of all humans. Populations are geographically dispersed, their circumstances differ, and some forms of adaptation may remain possible after severe disruption. The argument must therefore show why intervention, biological survival, adaptation, and preservation would fail under the assessed conditions. This requirement motivates the pathway-and-barrier approach: identifying possible transitions and examining the mechanisms that could enable or prevent them. For catastrophe-mediated pathways, an illustrative structure is AI-mediated hazardous pathway -mediated hazardous pathway (4) âconsequential real-world effects real-world effects âlarge-scale propagation -scale propagation âfailure of effective intervention of effective intervention âcivilization-scale catastrophe -scale catastrophe âfailure of preservation or recovery of preservation or recovery âexistential outcome. outcome. This sequence is illustrative. Some transitions may occur together, a pathway may enter at a later stage, and a permanent endpoint may be reached during the catastrophe rather than afterward. Different safeguards will matter in different circumstances. Permanent disempowerment can have another temporal structure: AI-mediated transfer of consequential authority -mediated transfer of consequential authority (5) âerosion of meaningful human agency of meaningful human agency âloss of realistic restoration mechanisms of realistic restoration mechanisms âpermanent human disempowerment. human disempowerment. Delegation alone does not establish this endpoint. Humans may retain effective decision rights, replacement options, and institutions capable of revising the arrangement. The argument concerns the conditions under which those mechanisms could cease to offer a realistic route back to meaningful agency. The initiating mechanisms also differ. Autonomous misalignment may involve strategic behavior and inadequate containment. Human misuse may begin with a system working exactly as intended. Organizational or competitive pathways may arise through unsafe deployment choices, fragmented responsibility, or coordination failures. Gradual disempowerment may involve accumulated dependence rather than a single decisive incident. Each arrow requires a separate empirical or theoretical argument. Blocking an essential transition can reduce the risk of that pathway, but alternative routes and shared vulnerabilities must also be examined before making claims about overall risk. This decomposition prevents evidence of serious harm, or exceptional performance on a cognitive task, from being mistaken for evidence of a complete pathway to extinction. 1.4 The CapabilityâControl Problem Several pathways raise a common question: can the arrangements governing AI-mediated activity remain adequate as consequential capabilities change? We describe this relationship through the capabilityâcontrol gap. The idea is related to the broader control problem discussed by Russell (2019), but here it is expressed as a comparison between two quantities that may evolve at different rates. Let CAIâ(t)C_AI(t) (6) represent, schematically, the consequential capabilities of AI systems at time t, and let Ccontrolâ(t)C_control(t) (7) represent the combined preventive capacity of developers, governments, institutions, and technical systems to understand, evaluate, monitor, constrain, and interrupt AI-mediated behavior. The conceptual gap is GâĄ(t)=CAIâ(t)âCcontrolâ(t).G(t)=C_AI(t)-C_control(t). (8) Preservation and recovery following catastrophe are represented separately later in the paper, including through QcatQ_cat for the catastrophe branch. This separates preventive control from conditional endpoint avoidance for purposes of analysis. It does not imply independent causes: the same institutions and infrastructure may support both, recovery preparations begin before failure, and preservation may operate during a catastrophe. Neither quantity has an agreed scalar measure. A numerical difference would require fixed, commensurate definitions for a specified assessment. Without them, GâĄ(t)G(t) remains conceptual shorthand. Its purpose is to focus attention on the relationship between consequential capability and the means available to govern it. Under those measurement assumptions, and over an interval where a differentiable approximation is appropriate, the condition dâCAIdât>dâCcontroldât dC_AIdt> dC_controldt (9) implies dâGdât>0. dGdt>0. (10) A widening gap does not itself establish inadequate control or catastrophe. Existing safeguards may retain an adequate margin, while an unchanged gap may already describe an unsafe arrangement. The practical assessment must concern the actions permitted, their possible consequences, and the effectiveness of evaluation, authorization, monitoring, governance, and intervention. The gap is therefore a cross-cutting concept rather than a complete risk model. It may help organize questions about autonomous failure, misuse, organizational weakness, and competitive deployment, but it supplies neither a universal safety threshold nor a probability of an existential outcome. Nor does inadequate control require AI to surpass humans across all cognitive domains. More limited systems can create difficult control problems when deployed at sufficient scale or connected to consequential decisions. The concern is whether people and institutions can reliably assess what those systems can do, recognize dangerous processes, and act while effective intervention remains possible. 1.5 Scope and Methodological Position The analysis makes no assumption that artificial general intelligence will necessarily be developed or that future systems must become autonomous, power-seeking, or uncontrollable. Arguments involving such properties are conditional. Evidence from existing systems is also distinguished from extrapolation to future deployments. Research on specification errors and distributional shift can establish particular failure modes (Amodei et al., 2016); it does not, by itself, establish a probability of extinction or permanent disempowerment. We also distinguish initiating mechanisms. Autonomous misalignment, harmful human use, organizational failure, and competitive deployment need not share the same causes or require the same safeguards. Existing taxonomies likewise emphasize multiple pathways to catastrophic AI risk (Hendrycks et al., 2023). These categories can interact, but their interaction does not justify collapsing them into one universal risk equation. The discussion remains non-operational. It examines how harm could propagate and how barriers could fail without developing methods for producing biological, cyber, military, infrastructural, or other catastrophic damage. The aim is to clarify defensive questions about control, preservation, and recovery. Finally, the endpoints are treated as claims requiring explanation. Advanced intelligence, severe disruption, and failed oversight are not substitutes for an account of how a permanent existential outcome could occur. 1.6 Main Proposition The organizing claim is that existential AI risk cannot be adequately represented by advanced intelligenceâ¶human extinction.advanced intelligence extinction. (11) A universal multiplication of capability, autonomy, access, and other factors would also obscure important differences among mechanisms. Their relevance depends on the pathway, the deployment, and the endpoint being assessed. The analysis instead distinguishes two broad, potentially interacting routes: AI-mediated pathwaysâ¶catastrophe-mediated route,gradual loss-of-agency routeCatastrophe-mediated route:civilization-scale catastropheâ¶failure of preservation or recoveryâ¶existential outcome,Gradual loss-of-agency route:erosion of meaningful human agencyâ¶loss of realistic restoration mechanismsâ¶permanent human disempowerment. aligned &AI-mediated pathways\\ & \ array[]lcatastrophe-mediated route,\\ gradual loss-of-agency route array .\\[4.30554pt] &Catastrophe-mediated route:\\ & -scale catastrophe\\ & of preservation or recovery\\ & outcome,\\[4.30554pt] &Gradual loss-of-agency route:\\ & of meaningful human agency\\ & of realistic restoration mechanisms\\ & human disempowerment. aligned (12) These routes are illustrative, not an exhaustive classification or a necessary sequence of stages. Autonomous misalignment, misuse, organizational failure, and competitive deployment can contribute to either in different ways. Competitive pressure may weaken safeguards, dependence may reduce intervention options, and catastrophe may accelerate disempowerment. The routes need not be independent, mutually exclusive, or equally likely. Probability accounting requires a separate distinction. Let C denote a specified AI-mediated civilization-scale catastrophe and let E denote the union of the three assessed existential endpoint events. Both must be defined over consistent trajectories, deployment contexts, and temporal criteria. Assuming PâĄ()>0P(C)>0, the catastrophe branch is pexistential,cat p_existential,cat =PâĄ(Eâ©) =P(E ) (13) =pcatâ(1âQcat), =p_cat (1-Q_cat ), where pcat=PâĄ()p_cat=P(C) (14) and Qcat=PâĄ(EcâŁ).Q_cat=P(E^c ). (15) Here QcatQ_cat is the probability of avoiding all specified existential endpoints given the catastrophe. Continued biological existence must be considered together with the preservation or permissible restoration of sufficient civilizational capacity and meaningful agency. The expression follows from the probability multiplication rule and requires no independence assumption. It is neither a numerical estimate nor a causal demonstration. Interpreting an outcome as catastrophe-mediated requires an additional account of how the catastrophe contributes to that outcome. The complete partition is pexistential=pexistential,cat+PâĄ(Eâ©c)p_existential=p_existential,cat+P(E ^c). The second term includes existential outcomes without the specified catastrophe, such as possible gradual permanent disempowerment. It cannot be omitted unless its probability is explicitly assumed to be zero. Although these probability branches are disjoint, the causal mechanisms contributing to them may interact. The resulting research question is: Under what conditions could an AI-mediated process reach a specified permanent existential endpoint, and why would the relevant safeguards and remaining possibilities for survival or restoration be insufficient to prevent it? The rest of the paper develops this question by examining autonomous and human-mediated pathways, the conversion of capability into consequential action, the propagation of local failures, and the conditions supporting preservation and recovery. It then considers how meaningful human agency might be maintained within a changing humanâAI relationship. The aim is to identify claims that can be investigated separately while keeping clear the additional argument needed to connect them to a permanent existential outcome. 2 What Does âEliminate Humanityâ Mean? The phrase âeliminate humanityâ can refer to very different outcomes. Public debate often brings mass mortality, the collapse of states, the destruction of technological civilization, biological extinction, and permanent loss of human control under the same heading. Yet these outcomes differ in what causes them, what survives, and whether recovery remains possible. An assessment of existential AI risk must keep those differences explicit. A process that disrupts the global economy need not destroy civilization, and civilizational collapse need not eliminate the human species. Conversely, biological survival does not guarantee that humans retain meaningful influence over their collective future. These distinctions determine what a proposed risk mechanism must explain. The literature already recognizes a broader category of existential harm. Bostrom (2002) includes both the annihilation of Earth-originating intelligent life and the permanent, drastic curtailment of its potential. Ord (2020) likewise treats extinction as one threat within a larger family of risks to humanityâs long-term future. Our classification focuses on three dimensions: biological human survival, sufficient civilizational capacity, and meaningful human agency. This is an analytical choice within that literature; it does not make âextinctionâ synonymous with âexistential catastrophe.â The biological definition also needs a qualification. Analyses of long-term trajectories may include civilizations continued by human descendants or other successors (Baum et al., 2019). The disappearance of a biological species and the destruction of humanityâs future potential are therefore different claims. In the hazardous scenarios assessed here, biological human extinction is classified as existential on the assumption that it would also eliminate the relevant prospects for humanityâs future. An assessment allowing successor trajectories to preserve those prospects would need to define continuity and revise the classification accordingly. One further distinction runs through the analysis: an endpoint is an outcome of a trajectory, whereas its probability expresses uncertainty about which trajectory will occur. An estimated low chance of survival is not itself extinction. Similarly, an estimated low chance of recovery does not establish that a collapse is irreversible. 2.1 Literal Human Extinction In its strict biological sense, human extinction means that no living members of Homo sapiens remain. Even a catastrophe killing billions would fall short of this endpoint if humans survived. A reproductively viable population offers a stronger basis for continued survival than isolated individuals, but loss of reproductive viability must still be distinguished from completed extinction. Connecting the former to the latter requires assumptions about biology, remaining technology, and possible interventions. This places a substantial burden on an extinction argument. Human populations are geographically dispersed, their dependence on infrastructure varies, and some opportunities for adaptation may survive the destruction of contemporary institutions. Biological survival does not require every feature of modern civilization to remain intact. An AI-mediated process capable of devastating global institutions would therefore need an additional mechanism to eliminate all surviving humans. The distinction can be summarized as global disruption disruption âÌžcivilizational collapse, \;civilizational collapse, (16) civilizational collapse collapse âÌžhuman extinction. \;human extinction. These expressions deny automatic implications; they do not prescribe a sequence. Extinction could occur without a separate stage of civilizational collapse. The point is that evidence for one form of harm does not, by itself, establish another. Let EbioE_bio denote biological human extinction under the endpoint and temporal criteria adopted in an assessment, and let F denote a specified AI-mediated hazardous process. That process may be acute or gradual. Any quantitative application must state its deployment context, evidence, and assumptions about possible trajectories. Assuming PâĄ(F)>0P(F)>0, define Qbio Q_bio =PâĄ(EbiocâŁF), =P(E_bio^c F), (17) PâĄ(EbioâŁF) P(E_bio F) =1âQbio, =1-Q_bio, where c denotes the complementary event. Thus QbioQ_bio is a conditional probability of avoiding extinction, not a probability of rebuilding civilization. Its value describes uncertainty before the outcome is known. Extinction can occur despite a low prior estimate of its probability, just as survival can occur when it was judged unlikely. The assessment horizon matters as well. Avoiding extinction during a specified finite period is different from avoiding eventual extinction within a longer-term model. Neither probability is informative without its temporal scope and underlying assumptions. 2.2 Civilizational Collapse Societies can lose political organization, industrial production, advanced technology, communication networks, or institutional continuity while substantial human populations remain alive. The study of civilizational trajectories makes room for such futures: severe catastrophes may fundamentally alter civilizationâs course without ending biological human existence (Baum et al., 2019). Here, civilizational collapse means a large and persistent decline in the organizational, technological, economic, and institutional capacities needed to sustain contemporary global civilization. We propose no universal numerical threshold. An empirical assessment would have to identify the functions of concern, measure their loss, and specify what would count as sufficient restoration. Duration and recoverability describe different aspects of such a collapse. A severe catastrophe may leave societies able to restore essential capacities. Those capacities might instead remain reduced for generations or centuries while recovery is still possible. In the irreversible case, humans survive but the conditions needed to restore sufficient civilizational capacity are permanently lost. These descriptions are not mutually exclusive categories: a long-duration collapse can also be recoverable. Permanent loss is existential under the definition used here when it destroys or drastically curtails humanityâs long-term potential. Exceptional severity or duration alone does not establish that conclusion. Nor must recovery recreate present institutions and technologies. The relevant criterion is whether societies can preserve or regain the capacities needed for a viable human civilizational future. Even a recoverable collapse could coexist with another existential endpoint, such as permanent disempowerment. Recoverability therefore adds information that mortality and immediate damage cannot supply. Two catastrophes with comparable losses could lead to very different futures if one preserves knowledge, coordination, and reconstruction capacity while the other permanently removes them. Let EcivE_civ denote permanent loss of sufficient human civilizational capacity under the adopted functional and temporal criteria. It may occur while humans survive or overlap with biological extinction. Using the same conditioning event F, define Qciv Q_civ =PâĄ(EcivcâŁF), =P(E_civ^c F), (18) PâĄ(EcivâŁF) P(E_civ F) =1âQciv. =1-Q_civ. Here QcivQ_civ concerns preservation or eventual restoration of sufficient capacity. If an assessment instead asks about recovery after an already occurring collapse, it must make that different conditioning event explicit. A low estimated QcivQ_civ indicates a high conditional risk of irreversible loss; it does not show that the loss has occurred. Failure to recover by a particular date also falls short of proving permanence. Such a claim requires evidence about why restoration would remain unavailable, rather than evidence only of prolonged disruption. For an AI-mediated pathway, widespread institutional damage is therefore the beginning of the argument. The further task is to explain why surviving societies would permanently lose the ability to reorganize and rebuild sufficient capacity. 2.3 Permanent Human Disempowerment A third endpoint is possible even without extinction or conventional civilizational collapse. Humans might remain numerous and materially prosperous while permanently losing meaningful influence over civilizationâs direction. This possibility falls within broader discussions of permanent destruction or drastic curtailment of future potential (Bostrom, 2002; Ord, 2020). Classifying it as existential nevertheless requires more than showing increased automation. The loss of agency must be sufficiently extensive and permanent to curtail humanityâs future potential. That assessment also involves normative judgments about which forms of influence matter and whose interests are represented. One possible route is accumulated delegation. People and institutions may transfer consequential decisions to automated systems because those systems appear more capable, efficient, or reliable. Individual decisions could be reasonable while their combined effects weaken the institutions through which humans challenge decisions, revise objectives, or change deployment arrangements. If realistic means of restoration were also permanently lost, formal authority could persist while effective influence disappeared. We use the following conceptual shorthand for effective human agency: Hagencyâ(t)â[0,1].H_agency(t)â[0,1]. (19) The value 00 denotes absence of meaningful agency, while 11 denotes strong effective human influence under the chosen assessment. This interval establishes an orientation, not a calibrated scale or a universal threshold. It also leaves unresolved how influence is distributed among people and which decisions remain contestable. Those features would need to be specified in an operational assessment. Meaningful agency does not require people to supervise every action or understand every component of an infrastructure. It concerns their practical ability, through appropriate institutions, to shape consequential objectives, challenge decisions, and revise the arrangements governing AI activity. Permanent disempowerment of surviving humans requires both the loss of such influence and the permanent absence of realistic means to restore it. Dependence, temporary delegation, or a finite interruption of control is insufficient. For a gradual delegation pathway, an illustrative structure is increasing dependence and loss of effective agency dependence and loss of effective agency (20) +permanent loss of restoration mechanisms +permanent loss of restoration mechanisms â¶permanent disempowerment. disempowerment. The expression describes one possible pathway. It does not imply that dependence necessarily produces disempowerment or that all pathways share this structure. Let EagencyE_agency denote permanent loss of meaningful human agency. This broader event may overlap with extinction; disempowerment of surviving humans is its distinct non-extinction form. With the same conditioning event F, define Qagency Q_agency =PâĄ(EagencycâŁF), =P(E_agency^c F), (21) PâĄ(EagencyâŁF) P(E_agency F) =1âQagency. =1-Q_agency. Avoidance here means that meaningful agency is preserved or restored under the adopted criteria. The hazardous process need not include a civilization-scale catastrophe: it may involve gradual institutional change. The two agency quantities answer different questions. A low HagencyH_agency describes limited influence in an assessed state. A low QagencyQ_agency describes a high estimated conditional probability of permanent loss. Neither alone establishes that an irreversible endpoint has been reached; that conclusion requires an account of permanence and the unavailability of realistic restoration. Humans routinely delegate decisions while retaining ways to alter objectives, rules, and authority. The existential concern arises when those revision mechanisms are permanently lost, rather than whenever AI performs a task formerly performed by people. 2.4 Existential Catastrophe as a Family of Endpoints Existential catastrophe is better treated as a family of endpoints than as a single outcome or a linear scale of damage. Biological survival, civilizational capacity, and meaningful agency describe different aspects of a future. We can represent them schematically as â°state=(Sbio,Cciv,Hagency),E_state= (S_bio,C_civ,H_agency ), (22) where SbioS_bio describes biological survival, CcivC_civ civilizational capacity, and HagencyH_agency meaningful human agency. These descriptors need not share a scale, and the vector is not a numerical measure of existential severity. A small surviving population, for example, is not biological extinction. A state description cannot establish irreversibility on its own. Similar observed losses may be temporary in one setting and permanent in another because different possibilities for preservation or restoration remain. Information about those possibilities and the wider trajectory must supplement the vector. The conditional quantities Qbio,Qciv,QagencyQ_bio, Q_civ, Q_agency (23) describe uncertainty about avoiding the respective endpoints given F. They are not components of the realized state. Their interpretation depends on the conditioning event, endpoint definitions, and temporal criteria used in the assessment. Within this three-endpoint classification, define the aggregate existential event as E=EbioâȘEcivâȘEagency.E=E_bioâȘ E_civâȘ E_agency. (24) Avoiding the aggregate event requires avoiding all three endpoints: PâĄ(EcâŁF)=PâĄ(Ebiocâ©Ecivcâ©EagencycâŁF).P(E^c F)=P\! (E_bio^câ© E_civ^câ© E_agency^c F ). (25) The three Q values cannot generally be multiplied to obtain this probability, because conditional independence is not assumed. Estimates based on different conditioning events, horizons, or endpoint criteria cannot be combined directly either. For the catastrophe branch used later, let C denote a specified AI-mediated civilization-scale catastrophe. Assuming PâĄ()>0P(C)>0, the aggregate avoidance probability is Qcat=PâĄ(EcâŁ)Q_cat=P(E^c ), and the corresponding endpoint-specific quantities are Qj,cat=PâĄ(EjcâŁ)Q_j,cat=P(E_j^c ), where j indexes biological extinction, civilizational loss, or agency loss. The quantities introduced earlier condition on F. Applying them to the catastrophe branch therefore requires setting F=F=C under consistent assessment definitions, or establishing another relationship through an explicit model. Gradual permanent disempowerment without the specified catastrophe belongs to a separate probability branch. These distinctions clarify why mortality alone cannot measure existential severity. Large, healthy populations could remain permanently disempowered, whereas extraordinary mortality could leave survivors with a viable route to civilizational recovery. Extinction concerns the disappearance of all living humans; irreversible civilizational collapse concerns permanent loss of sufficient capacity; and disempowerment concerns permanent loss of meaningful influence and its restoration mechanisms. None is defined by an estimated probability merely approaching zero. The possibilities can be represented as branching outcomes: AI-mediated processesâ¶recoverable disruption or catastrophe,irreversible civilizational collapse,permanent human disempowerment,human extinction. -mediated processes \ array[]lrecoverable disruption or catastrophe,\\ irreversible civilizational collapse,\\ permanent human disempowerment,\\ human extinction. array . (26) This diagram is neither a deterministic implication nor an exhaustive partition. The endpoints may overlap and need not occur in a fixed order. Permanent disempowerment, in particular, need not begin with an acute disaster. A proposed pathway must therefore identify the harm or agency loss it could produce, explain how its effects could spread or accumulate, and specify the endpoint being assessed. It must also identify the barriers that would have to fail and the mechanisms that might preserve survival, capacity, or agency. Most importantly, a claim of permanence needs evidence beyond that required to establish severe local harm or prolonged disruption. Keeping these claims separate prevents an argument from moving unnoticed from the plausible possibility of serious AI-mediated harm to the much stronger conclusion that humanity could be eliminated or its future potential permanently curtailed. Throughout the paper, existential AI risk refers to AI-mediated outcomes that permanently destroy or drastically curtail humanityâs future potential. AI extinction risk refers specifically to biological human extinction. The next section examines conditions under which an AI system could become an independent or semi-independent causal actor with civilization-scale consequences. This is one class of pathway. Human misuse, organizational failure, competitive deployment, and gradual loss of agency may depend on different combinations of mechanisms. 3 Conditions for Autonomous AI-Mediated Existential Risk Advanced intelligence alone does not establish civilization-scale risk. A system may excel at mathematics, science, or software engineering while having little influence beyond the setting in which people use it. The question is how those abilities become consequential action: through which interfaces, with what permissions, over what period, and under whose supervision? We examine five operational dimensions: capability, autonomy, access, persistence, and strategic competence. They help describe how an AI system could act independently or semi-independently in the world. They are not a checklist of conditions that every catastrophic pathway must satisfy. A system with little autonomy could assist a harmful user, and a single consequential action could matter without prolonged operation or sophisticated planning. The surrounding safeguards are equally important. Evaluation, access restrictions, monitoring, intervention, and governance influence whether dangerous activity begins or spreads. Preservation and recovery address a different question: what remains possible if prevention fails. These functions are separated analytically, although they may depend on the same institutions and infrastructure. Recovery preparations can begin before failure, and preservation can operate while a catastrophe unfolds. Section 11 develops this distinction by separating autonomous misalignment from misuse and other pathways. There, W represents weakness of preventive safeguards and governance, while endpoint-specific preservation and recovery are considered separately. 3.1 Capability Relevant capability depends on the task and deployment. Scientific reasoning, technical execution, communication, planning, and organizational coordination may reinforce one another, but broad excellence across all these domains is not required for every harmful outcome. A narrow capability can become consequential when it operates through a powerful interface or influences a critical decision. Benchmark results therefore need to be interpreted in context. Strong performance on a reasoning task does not establish reliable planning, tool use, adaptation, or execution. Conversely, several moderate abilities may support effective action when combined within a suitable workflow. An assessment should examine both individual capabilities and their interaction in the deployed system. Dangerous-capability evaluations can help identify such possibilities before deployment (Shevlane et al., 2023). Their significance depends on whether the tested abilities transfer to the environments, tools, and permissions relevant to the proposed pathway. 3.2 Autonomy A system that answers a question and stops differs from one that selects intermediate steps, acts, examines the result, and continues without approval at every stage. The latter arrangement gives the system more opportunities to convert an initial instruction into a sequence of consequential actions. Autonomy here is an engineering and deployment property, not an assumption about consciousness. It concerns the extent to which a system can select and execute actions with limited human intervention. Assessments of AI agents examine this combination of planning, tool use, and extended task horizons (Bengio et al., 2026). The degree of autonomy granted and the reliability with which it is exercised must be distinguished. A system may receive broad permissions while performing poorly on extended tasks. Another may be highly capable but operate under close supervision. Cognitive performance alone tells us neither how much freedom a system has nor how reliably it will use that freedom. The safety implications depend on the actions available and the consequences of error. Greater autonomy can reduce opportunities for human review, but its effects are shaped by access restrictions, task boundaries, and mechanisms that interrupt unacceptable behavior. 3.3 Access An AI system needs some channel of influence to produce external effects. That channel may be a software tool, a communication interface, an organizational workflow, or a human who acts on its output. Cognitive capability does not automatically confer physical, economic, institutional, or political power. Identical models can consequently have different risk profiles. One may operate in an isolated testing environment; another may communicate with users, invoke tools, or contribute to consequential decisions. Even a text-only interface can provide indirect influence when people rely on its recommendations. For an autonomous pathway, the relationship can be represented schematically as capability+autonomy+access +autonomy+access (27) â¶potential for consequential external action. for consequential external action. The plus signs indicate a combination of properties rather than numerical addition, and the arrow does not establish that harm will occur. The expression directs attention to the interfaces through which a systemâs abilities can affect its surroundings. Restricting access may reduce the consequences of failure without changing the underlying model. Its effectiveness depends on which channels remain available, whether restrictions are enforceable, and whether humans or connected systems can provide alternative routes to consequential action. 3.4 Persistence Persistence concerns the duration and continuity of an AI-mediated process. Memory, scheduled execution, continuing services, and multi-step workflows can allow activity to extend beyond an initial interaction. Institutional dependence may also sustain a process even when no individual model instance operates continuously. None of this requires a desire for self-preservation. Persistence can arise from ordinary design and organizational choices. Its importance is practical: effects can accumulate, information can be retained, and actions can be revised in response to feedback. When combined with autonomy, persistence may allow a system to pursue a task across changing circumstances with fewer interruptions for human review. Yet duration is not a measure of danger by itself. A long-running process may remain tightly constrained, while one brief action could have lasting consequences. Persistence is therefore a dimension to assess, not a necessary condition for every serious failure. 3.5 Strategic Competence Solving technical problems is different from acting effectively in an environment where other agents respond. Strategic competence concerns the ability to anticipate those responses, allocate resources, revise plans, and adapt when initial actions fail. It is distinguished here from task capability for analytical purposes, although the two may overlap. This dimension matters particularly in pathways involving sustained influence or attempts to circumvent intervention. Humans, institutions, and other systems can oppose, redirect, or interrupt consequential activity. A proposed scenario involving successful adaptation to those responses must explain why the system would possess the relevant abilities. Formal work shows that optimal policies can exhibit option-preserving or power-seeking tendencies in some decision-process models (Turner et al., 2021). These results depend on the model and its assumptions. They do not establish that learned systems will implement the relevant optimal policies, that all capable AI will seek power, or that such behavior entails a desire for domination. Strategic competence may support some autonomous pathways, but failures can also occur without it. An error with large consequences need not involve planning against human opposition. 3.6 Weakness of Pre-Catastrophe Safeguards The operational dimensions describe what a system might be able or permitted to do. They do not establish that dangerous activity would succeed. Effective restrictions, evaluation, containment, monitoring, and intervention can block particular pathways or limit their consequences. A reactive defensive sequence can be represented as dangerous behavior behavior âdetection (28) âcontainmentâintervention. . This is one possible arrangement, not a required order. Containment may already be in place before dangerous behavior occurs, and preventive restrictions can block actions without first detecting each hazardous instance. Some pathways allow time for intervention; others may produce consequences before a response is possible. Evaluation before deployment can inform decisions about permissions and safeguards (Shevlane et al., 2023). It cannot guarantee that every dangerous behavior will be identified, particularly when deployment differs from testing. The relevant question is whether the available barriers address the specific mechanism and remain effective under the conditions being assessed. We denote their weakness schematically by W=weakness of pre-catastrophe safeguards and governance.W=weakness of pre-catastrophe safeguards and governance. (29) Here W covers limitations in evaluation, prevention, detection, containment, monitoring, intervention, and governance. Larger values indicate weaker safeguards under the chosen assessment. No empirical scale is specified at this stage. Post-catastrophe preservation and recovery are represented separately in Section 11; this separation does not imply that the functions are causally independent. The five operational dimensions can be brought together in a conceptual expression for autonomous consequential potential: Î auto=gâĄ(C,A,X,P,S), _auto=g(C,A,X,P,S), (30) where C denotes capability, A autonomy, X external access, P persistence, and S strategic competence. Neither the function g nor a calibrated scale for Î auto _auto is specified. The expression summarizes dependencies to investigate; it is not a probability of catastrophe. It also imposes no assumption that every factor must be high, that the factors are independent, or that increasing one necessarily increases danger. For autonomous misalignment, the corresponding pathway representation is Ïauto=fautoâ(C,A,X,P,S,Ma,W), _auto=f_auto (C,A,X,P,S,M_a,W ), (31) where MaM_a represents divergence of autonomous behavior from the intended objectives or constraints, and W represents safeguard weakness. The score Ïauto _auto remains conceptual. It identifies factors relevant to this pathway without assigning their weights, specifying their interactions, or mapping them to a catastrophe probability. A quantitative application would require operational definitions and evidence supporting those additional choices. This formulation separates three questions. The first concerns whether the deployed system can produce consequential effects through the actions available to it. The second concerns whether misaligned behavior arises and whether the relevant barriers fail to prevent or contain it. The third concerns the endpoint: if severe harm occurs, do humans retain routes to survival, sufficient civilizational capacity, and meaningful agency? Avoiding the aggregate existential event defined earlier requires avoiding all three assessed endpoints. Biological survival alone is insufficient if civilizational capacity or meaningful agency is permanently lost. These preservation and restoration questions remain outside W and Ïauto _auto because they concern the trajectory after, or during, failure rather than preventive control alone. The distinction also prevents the autonomous pathway from becoming a universal template. Misuse may exploit a system that follows its userâs instructions. Organizational failure may arise through fragmented responsibility, and competitive deployment through incentives that favor speed over adequate safeguards. Neither requires the same combination of autonomy, persistence, or strategic competence. The central claim is therefore: An assessment of autonomous AI risk must explain how a systemâs capabilities and permitted actions could produce consequential harm, why misaligned behavior might arise, and why the relevant safeguards might fail. An existential argument requires the further explanation of how at least one specified permanent endpoint could be reached despite remaining possibilities for survival or restoration. The broader framework treats autonomous misalignment as one pathway among several. The dimensions examined here help identify its possible mechanisms and points of intervention; they do not establish an inevitable progression from advanced intelligence to catastrophe. 4 Why AI Is Different from Conventional Technology Claims that artificial intelligence is fundamentally unlike every earlier technology are difficult to sustain. Automated planning, adaptive control, learning from feedback, and software replication all have substantial histories. Contemporary AI builds on those developments. What deserves attention is how broadly learned capabilities can be reused and how readily they can be connected to tools, organizations, and research processes. Foundation models illustrate this change. Broad pretraining produces a model that can support numerous downstream applications (Bommasani et al., 2021). Instructions, examples, additional training, and external tools allow developers to draw on a shared computational foundation rather than construct every application independently. This flexibility does not make the model universally competent, but it changes the range of settings in which the same learned system can be used. The comparison must also distinguish the model from its deployment. A language model generates outputs; an agent adds mechanisms for selecting and executing actions; a deployed service includes infrastructure, permissions, and organizational responsibilities. Autonomy, memory, and resource access depend on that larger arrangement. They cannot be inferred from the generality of the model alone. 4.1 Passive Tools versus Adaptive Systems A microscope and a motor have relatively stable functions, but conventional software can already implement conditional behavior, feedback, and complex decision rules. AI also operates through programmed computation. Its distinctive contribution concerns how task-specific behavior is learned and how widely the resulting capabilities can be applied. The word adaptation covers several processes. Training changes model parameters. Reinforcement learning can use experience and reward signals to improve a policy (Sutton and Barto, 2018). A deployed language model, by contrast, can respond differently to instructions, examples, and observations while its parameters remain unchanged. In-context learning belongs to this latter category (Bommasani et al., 2021). Information supplied during an interaction does not thereby become persistent knowledge in the modelâs weights, although a surrounding system may store it in external memory. A thermostat makes the distinction clear. It responds to changing temperature through a control rule that may remain fixed. Responsiveness is therefore not sufficient evidence of learning. Likewise, changing language-model outputs do not show that the model has persistently modified itself through interaction. General-purpose models can nevertheless support many forms of context-dependent behavior within the same system, including writing, coding, analysis, and interpretation of tool results. Their breadth and reliability remain empirical questions. Adaptation also does not imply agency: a model can adjust its responses without having permission or an interface to act externally. 4.2 Planning Planning relates actions to objectives and anticipated consequences. A model may produce a plan in one response, but executing that plan requires additional mechanisms. An agentic workflow can select a step, perform it, receive observations, and use those observations to choose the next action. The consequential difference lies in the connection between deliberation and execution, together with the authority delegated to the system. ReAct provides an example by interleaving generated reasoning traces, task-specific actions, and observations (Yao et al., 2023). Its workflow can be described schematically as assessâactâobserveâreassess.assess . This loop describes an interaction pattern. It does not establish that the generated reasoning faithfully explains the computation behind an action or that an extended plan will succeed. ReAct demonstrates useful results on the evaluated tasks, not reliable autonomous performance in arbitrary environments. Planning itself predates contemporary language models. These models provide another way to generate and revise candidate actions within sequential systems. Success depends on observations, tools, the controller, and checks during execution. A workflow may allow continued action even as errors accumulate, so the permission to act and the reliability of action must be assessed separately. 4.3 Generalization Generalization is the ability to perform beyond the particular examples used in training. It is central to machine learning, including specialized models. Foundation models extend the range of possible applications through broad pretraining and reuse across downstream tasks (Bommasani et al., 2021). Instructions or examples may suffice for some applications; others require fine-tuning, new data, or substantial engineering. Calling a task ânewâ can obscure what has been demonstrated. A task may be unfamiliar to a user or absent from an explicit training objective while closely resembling material encountered during pretraining. Successful performance does not alone establish discovery of a new method or reliable transfer to more demanding cases. Evaluation needs to distinguish familiar patterns, genuinely held-out examples, and changes in operating conditions. This flexibility is useful, but it also complicates assessment. Developers may not anticipate every application of a model, and new tools or instructions can alter what a deployed system can accomplish. Evidence from one configuration may leave important questions unanswered about another. Broader generalization does not necessarily increase benefit or danger in a uniform way. An additional capability might improve error detection in one deployment and support harmful use in another. Its consequences depend on the task, reliability, and available channels of action. For safety analysis, the main implication is that evaluation must follow the actual deployment rather than rely on a modelâs general description. 4.4 Tool Use Tools extend the operations available to an AI system. Through interfaces supplied by developers, a model may request calculations, retrieve information, execute code, or interact with services. These operations depend on software and permissions. The ability to describe an action in text does not confer the ability to perform it. Toolformer demonstrated a training method for selecting and using specified APIs, including calculation, retrieval, translation, and calendar functions (Schick et al., 2023). ReAct examined a different design, combining external actions with iterative reasoning (Yao et al., 2023). Both show ways of incorporating tools into language-model workflows. Neither establishes uniformly reliable tool use or safe operation of unfamiliar interfaces. The appropriate unit of assessment is often the complete deployed system: model, controller, memory, tools, credentials, and environment. Tools may provide functions the model cannot perform internally. The controller determines how proposed actions are executed and checked, memory can carry information across interactions, and credentials govern which operations are permitted. Access restrictions can limit particular consequences of failure. Read-only database access, for example, removes direct modification as an available operation. Other routes to harm may remain if outputs influence people or are passed to systems with broader permissions. The assessment therefore needs to trace the available channels of influence through the whole configuration. 4.5 Scalability A trained digital model can be reused without repeating its full training process for each deployment. Given the model, supporting software, and suitable hardware, additional instances can be operated. This differs from educating another human expert, although a model instance should not be assumed to reproduce an expertâs judgment, experience, or responsibilities. Software replication is longstanding. AI applies it to learned systems that can support many tasks. Replication and operation still require computation, memory, energy, communication, and maintenance. Useful work may also depend on local information, specialized tools, and permissions. Counting copies is consequently an inadequate measure of effective capability. Parallel instances can increase the volume of work, but their outputs need not be independent and their coordination may be unreliable. Deployments sharing models, data, or infrastructure may also share vulnerabilities. The foundation-model literature identifies this problem in its discussion of homogenization (Bommasani et al., 2021). A common weakness may affect many applications, with consequences shaped by their local safeguards and conditions. Scalability thus concerns both productive capacity and shared dependence. It can expand access to useful services while creating common vulnerabilities across organizations. Which effect dominates cannot be inferred from digital replication alone; it depends on the architecture and dependencies of the deployments. 4.6 AI as a Participant in Technological Development AI can contribute to the processes through which technology is developed. This role has precedents in simulation, automated optimization, theorem proving, and computer-aided design. General-purpose models add ways to generate and evaluate language, code, and candidate solutions within those processes. The AI Scientist automated stages of research in supplied machine-learning environments, including idea generation, code-based experiments, analysis, and manuscript preparation (Lu et al., 2024). AlphaEvolve combined language-model proposals, evolutionary search, and automated evaluation to improve algorithms and computational infrastructure, including components used in AI training (Novikov et al., 2025). These results support a bounded claim: AI-assisted workflows can produce useful research and engineering outputs in some settings. Their scope remains important. Automating a workflow does not guarantee sound, novel, or valuable findings. Success on a supplied optimization problem does not establish the ability to identify important problems, design appropriate evaluations, or sustain progress across fields. Human contributions to infrastructure and evaluation can coexist with substantial automation; the degree of autonomy must be examined within each workflow. Improving infrastructure used to train AI is also distinct from demonstrating open-ended recursive self-improvement. The stronger claim would involve successive systems independently producing changes that improve their general capabilities through repeated research and redesign. The cited results do not establish that process, its rate, or its limits. A possible feedback structure is AI systems systems â¶AI-assisted research and engineering, -assisted research and engineering, (32) â¶validated improvements, improvements, â¶later AI systems. AI systems. Feedback occurs if the later systems contribute to further research. Each transition requires something beyond generating a proposal: evaluation must establish an improvement, implementation must succeed, and the resulting change must matter for subsequent work. These are separate empirical questions. Faster completion of one task need not shorten the full development cycle. Experimental validation, hardware production, data access, computational budgets, and organizational decisions may remain bottlenecks. Further improvements may also become harder to obtain. The diagram therefore describes a possible mechanism without specifying a growth law or predicting sudden acceleration. Even partial automation could matter for risk if it changes the pace or direction of capability development. Its significance should be assessed through demonstrated contributions, remaining constraints, and the effectiveness of oversight. A claim of rapid, sustained acceleration needs additional evidence or explicit assumptions. 4.7 The Difference Is Combinatorial The features considered here have histories extending beyond contemporary AI. Earlier systems could learn, plan, act, operate at scale, and contribute to technological design. General-purpose models can make some of these functions easier to connect across tasks through shared interfaces, although the resulting systems differ substantially in competence and autonomy. A deployment may combine context-sensitive responses, planning, tools, persistent memory, and repeated execution. Its effects depend on how those components interact. Retrieved information can support a plan, memory can preserve earlier results, and execution can turn a proposed step into an external consequence. Reusing the same arrangement widely may expand both its benefits and the reach of its failures. Those interactions also create opportunities for protection. Restricted permissions can limit actions, evaluation can identify weaknesses, and effective intervention can interrupt hazardous processes. Conversely, broad access and inadequate oversight can give a limited model substantial influence. Assessing the combination therefore requires attention to the systemâs architecture and institutional setting. The contribution to the present argument is specific: general-purpose AI can connect learned capabilities to external action across a broad range of tasks. Some deployments may consequently place new demands on evaluation, accountability, and control. This does not establish an existential outcome. It explains how cognitive abilities can become linked to the resources, decisions, and action channels through which consequential effects are possible. 5 The CapabilityâControl Gap Capability alone provides an incomplete account of AI risk. A highly capable system may remain manageable when evaluation, permissions, monitoring, and intervention are adequate for its deployment. The question is whether those arrangements can govern the consequential behavior the system makes possible, including when its capabilities or operating conditions change. We describe this relationship through the capabilityâcontrol gap: GâĄ(t)=CAIâ(t)âCcontrolâ(t),G(t)=C_AI(t)-C_control(t), (33) where CAIâ(t)C_AI(t) represents consequential AI capabilities in a specified deployment and Ccontrolâ(t)C_control(t) represents the combined preventive capacity of developers, institutions, governments, and technical safeguards to evaluate, monitor, constrain, and interrupt AI-mediated behavior. The subtraction needs a substantive interpretation. Capability would have to be expressed through the demands it places on particular safeguards, and control capacity assessed against those demands on a commensurate scale. Normalizing two unrelated indices to [0,1][0,1] would not make them comparable. The chosen scales and weights would also affect both the magnitude of the gap and its apparent rate of change. No common scale is calibrated here. Equation (33) therefore serves as conceptual notation. In practice, an assessment would usually retain separate requirements for different failure modes: an aggregate score could conceal inadequate protection in one critical channel behind improvements elsewhere. Preservation and recovery following catastrophe are considered separately through the endpoint-avoidance probabilities introduced in Section 2. For the catastrophe branch, the aggregate quantity is QcatQ_cat, defined in Section 11. The distinction separates preventive control from avoidance of a permanent endpoint when catastrophe occurs. It does not imply separate causes or resources. Institutional continuity may support both functions, recovery preparations begin before failure, and preservation may operate during a catastrophe. 5.1 Capability Growth AI capabilities can change through improvements in models, algorithms, computing resources, tools, memory, agent architectures, and research workflows. These changes need not advance every ability equally. Their consequences also depend on deployment: an improved model does not automatically acquire broader permissions, greater reliability, or more real-world influence. Evaluations and safeguards developed for one configuration may need revision when that configuration changes. Whether they remain adequate depends on the existing margin of protection, the new demands placed on it, and the response. On fixed, commensurate scales, over an interval where differentiation is appropriate, consider dâCAIdât>dâCcontroldât. dC_AIdt> dC_controldt. (34) The definition of G gives dâGdât=dâCAIdâtâdâCcontroldât, dGdt= dC_AIdt- dC_controldt, (35) so the inequality implies a widening gap over that interval. For discrete changes between assessed configurations, the corresponding identity is ÎâG=ÎâCAIâÎâCcontrol. G= C_AI- C_control. (36) These are mathematical consequences of the representation, not evidence that capabilities are actually outpacing control. Establishing that claim would require operational measurements. Connecting it to catastrophe would require a further account of the failure mechanisms involved. Even within a justified scalar model, direction of change does not settle adequacy. A widening gap may remain within an acceptable margin for some time, while a constant gap may describe an already inadequate arrangement. Neither its value nor its derivative is a catastrophe probability. 5.2 Limits of Human Oversight Placing a human âin the loopâ does not ensure effective control. A reviewer needs usable evidence, time to assess it, authority to intervene, and a means of changing the outcome. Approval becomes largely formal when these conditions are absent. This does not mean that every action must be reviewed manually or every internal computation understood. Automated checks, bounded permissions, independent authorization, and other safeguards can support human governance. Their role is to keep consequential activity within assessable limits and preserve intervention opportunities where they matter. Speed, complexity, and decision volume can undermine direct review. A person may approve an action without enough context to evaluate its consequences. A better architecture may reduce that burden by restricting what can happen without additional authorization and directing attention to decisions with greater potential impact. The International AI Safety Report 2026 discusses agents carrying out longer sequences of action with reduced oversight, alongside continuing reliability limitations on complex tasks (Bengio et al., 2026). Longer sequences make the placement of checks and intervention points more important. Their safety implications depend on the actions permitted, the consequences of error, and the safeguards operating during execution. Oversight should therefore be judged by its practical effectiveness, rather than by the presence of a named reviewer. 5.3 The Evaluation Gap Control depends partly on knowing what a deployed system can do and how it may behave. Benchmarks sample particular tasks and conditions. They may leave out the tools, users, permissions, incentives, and combinations of components that determine consequential behavior elsewhere. The International AI Safety Report 2026 discusses an evaluation gap between pre-deployment assessment and real-world utility or risk, including problems of narrow testing, data contamination, and differences between evaluation and deployment (Bengio et al., 2026). Such limitations can overstate practical performance or leave relevant behavior insufficiently assessed. They create uncertainty; they do not prove that a dangerous hidden capability exists. A related distinction concerns ability and propensity. Shevlane et al. (2023) separate evaluation of dangerous capabilities from evaluation of the tendency to use those capabilities harmfully. Demonstrating that a system can complete a task says little by itself about when it will attempt it, how reliably it will succeed, or whether its deployment permits the consequences of concern. We represent incomplete characterization using sets. Let relevantK_relevant denote the behaviors and capabilities relevant to the assessed contexts, and let assessedK_assessed denote those adequately characterized by evaluation. Then uncharacterized=relevantâassessed.K_uncharacterized=K_relevant _assessed. (37) The expression identifies a conceptual remainder, not a measurable inventory of everything unknown. In practice, the full relevant set is itself incompletely known. Evaluation uncertainty can weaken control without any increase in capability. A safeguard may rely on an incomplete understanding of behavior, or it may never have been tested with the permissions and components used in deployment. Assessment must consequently address the configured system and its operating conditions, not model performance alone. 5.4 Control Lag Changes to deployed systems and updates to their safeguards need not occur together. A system may be scaled, modified, or connected to new tools before corresponding evaluations and procedures are ready. Some technical restrictions can be revised quickly; other responses require testing, coordination, or institutional change. This timing mismatch can be represented as Ïcapability<Ïresponse, _capability< _response, (38) where Ïcapability _capability is the characteristic interval between consequential changes to deployed systems and Ïresponse _response is the time needed to assess those changes and implement an adequate response. The inequality does not establish that the capabilityâcontrol gap widens or that safeguards fail. Existing restrictions may cover a change, and deployment can be postponed until an adequate response is available. The size and nature of the changes matter alongside their frequency. Repeated lag becomes concerning when consequential modifications are deployed before their effects can be assessed or constrained. Whether this happens must be investigated within the actual development and deployment process. 5.5 Control Adequacy and Threshold Effects Safeguards can compensate for one another in some circumstances. Restricted access may reduce the burden on monitoring, while independent authorization may prevent an error from becoming consequential. In other circumstances, failure of a critical barrier opens a route to harm that improvements elsewhere cannot close. If a scalar representation is justified, let GcritG_crit denote a hypothetical boundary for the adequacy of a particular control architecture against specified behavior. The criterion GâĄ(t)â„GcritG(t)â„ G_crit (39) could then represent failure to meet that assessmentâs control requirement. No numerical boundary is established here. Some domains may require several criteria, and reliability may decline gradually rather than change abruptly. Even a validated adequacy boundary would identify a control deficiency, not demonstrate that catastrophe follows. The operational question is: Under which deployment conditions do the relevant safeguards cease to provide adequate evidence, constraints, and intervention against specified forms of consequential AI-mediated harm? This explains why the same model can have different implications across deployments. Restricted permissions and effective authorization may limit its influence. Integration with consequential infrastructure may create additional action channels or reduce the time available for intervention. These differences belong to the architecture and institutional setting of the deployed system. Preservation and recovery address the further question of whether a catastrophe escaping preventive controls would produce a permanent existential loss. They complement rather than replace the assessment of prevention and intervention. 5.6 The Gap as a Dynamic Quantity Both AI systems and their safeguards change. Evaluation, monitoring, permissions, authorization, and intervention must be assessed against the behavior they are intended to govern. An improvement matters when it addresses a relevant failure mode, not simply when it raises an aggregate control score. Within a fixed, commensurate, differentiable representation, dâCcontroldâtâ„dâCAIdâtâșdâGdâtâ€0 dC_controldtâ„ dC_AIdt dGdt†0 (40) means that the scalar gap is not increasing. This is insufficient as a safety condition: an inadequate architecture may remain inadequate while the gap narrows. A more limited conclusion is possible under additional assumptions. If the adequacy criterion is justified, GâĄ(t0)<GcritG(t_0)<G_crit, the boundary remains fixed, and dâG/dâtâ€0dG/dt†0 throughout the interval, then the gap remains below that boundary within the model. Applying the conclusion to deployment would still require validation of the measures, the criterion, and the failure modes covered. A change in context or an unassessed mechanism could invalidate the assessment. The practical objective is to maintain adequate protection throughout development and use, accounting for uncertainty, existing margins, and the consequences of failure. Comparing aggregate growth rates can help organize that inquiry but cannot replace it. Control failure must also remain distinct from an existential endpoint. When the three endpoints in Section 2 are assessed jointly, avoidance requires continued biological existence together with preservation or permissible restoration of sufficient civilizational capacity and meaningful agency. Survival alone does not exclude permanent civilizational loss or disempowerment. For the catastrophe branch, QcatQ_cat describes conditional avoidance of those endpoints, as developed in Section 11. Gradual agency loss without the specified catastrophe requires separate accounting rather than being forced into a sequence of catastrophe followed by failed recovery. The central proposition is: Control must be assessed against the consequential behavior a deployed system makes possible, and reassessed as that system and its conditions of use change. The capabilityâcontrol gap organizes this comparison; a numerical interpretation requires justified measures and an explicit criterion of adequacy. The concept connects the operational dimensions discussed earlier to the pathways considered next. Each assessment must identify the behavior of concern, the barriers intended to constrain it, and the conditions under which those barriers could fail. An existential argument then needs to explain how that failure could reach a specified permanent endpoint despite remaining possibilities for preservation or restoration. 6 Broad Pathways to Civilizational Catastrophe The preceding sections describe how AI systems can become consequential. We now turn to the channels through which their outputs or actions could contribute to harm on a civilizational scale. Those channels may be direct, through tools and connected systems, or indirect, through people and institutions acting on AI-generated information. The literature discusses infrastructure disruption, cyber instability, military escalation, scientific misuse, economic disruption, information disorder, and concentration of power (Hendrycks et al., 2023; Bengio et al., 2026). These are overlapping classes of possible harm, not predictions or an exhaustive classification. Their plausibility depends on capabilities, deployment choices, safeguards, and human responses. They may also interact: disruption in one domain can weaken the institutions responsible for managing another. Throughout the discussion, evidence of a harmful mechanism must be distinguished from evidence that it could reach a permanent existential endpoint. 6.1 Critical-Infrastructure Dependence Electricity, communications, transport, finance, health care, water, and logistics sustain modern civilization. AI can improve the operation of these services, including their ability to detect problems and respond to disruption. Dependence on shared automated components can nevertheless introduce vulnerabilities across otherwise distinct systems. Two mechanisms matter here. A common weakness may cause several services to fail under similar conditions. Alternatively, disruption in one service may contribute to failure in another. The first is common-mode failure; the second is propagation through dependencies. They can occur together, but correlation alone does not demonstrate a cascade. The consequences depend on the functions affected, the duration of disruption, and the alternatives available. Widespread failure would not by itself establish an existential outcome. That stronger claim requires an explanation of how the disruption could reach at least one permanent endpoint defined in Section 2, despite safeguards and remaining opportunities for survival or restoration. Failure to recover within a finite observation period is insufficient evidence of permanence. 6.2 Cyber-Systemic Instability AI can support both cyber defence and harmful activity. Defenders may use it to identify vulnerabilities, monitor networks, detect anomalies, and respond to incidents. Similar capabilities can also assist adversaries, changing the balance and pace of competition. Some automated operations occur faster than human deliberation. When discovery, response, and counter-response involve interacting automated systems, defenders may have less time to understand an incident before its consequences spread. Whether automation improves or undermines security depends on the abilities of both sides and the architecture of the systems involved. International AI-safety assessments therefore identify cyber capability and societal resilience as important areas for evaluation (Casper et al., 2026; Bengio et al., 2026). The civilizational concern goes beyond individual incidents: it asks whether an incident or combination of incidents could interrupt essential functions across a sufficiently wide area. Common vulnerabilities and infrastructure dependencies provide possible mechanisms, but their significance must be established for the assessed deployment. 6.3 Military Escalation Military systems operate in an environment where errors can be interpreted as deliberate actions. AI may improve analysis, detection, logistics, and decision support, while some uses of automation may shorten the time available for verification and political judgment. This tension is especially consequential around strategic weapons. SIPRI examines how AI integration could affect strategic stability through uncertainty, misperception, competitive pressures, and shorter decision cycles (Boulanin et al., 2020). Escalation need not begin with an AI independently deciding to initiate conflict. It could emerge from interactions among automated systems, human decisions, incomplete information, and pressure to respond. Safeguards must therefore address those interactions, preserving opportunities for verification, deliberation, and effective restraint before irreversible actions occur. 6.4 Biological and Chemical Misuse AI assistance could make scientific work more accessible and productive. Some of the same assistance may also lower barriers to dangerous tasks. The relevant question is how much additional capability it provides, to whom, and under what conditions. This additional capability is often described as uplift. Its measurement requires an explicit comparison, such as comparable users working with AI assistance versus users relying on conventional information sources and tools. Results depend on the task, users, model, and baseline. Better performance on an information task does not establish successful execution in a laboratory. International assessments discuss evaluation of these distinctions in biological and chemical domains (Casper et al., 2026; Bengio et al., 2026). For the present analysis, the concern is structural. If a deployment substantially lowers barriers to dangerous scientific activity, safeguards around model access, laboratory practice, screening, monitoring, and public-health response become relevant parts of the same pathway. The extent of the change must be demonstrated rather than inferred from scientific fluency alone. This is also an example of AI-enabled harm without autonomous misalignment. A system may follow instructions while increasing the capability of a user who already intends harm. 6.5 Economic and Logistical Disruption Financial markets, production, transport, communications, and supply chains depend on one another and already use extensive automation. AI may improve coordination, but widespread reliance on similar systems can also produce correlated decisions and shared vulnerabilities. Disruption can travel through these connections. Financial instability may restrict production, logistical failure may create shortages, and communications problems may delay response. AI becomes systemically relevant when its use changes the scale of these effects or creates dependencies through which errors and unexpected interactions reach many organizations. Greater automation does not necessarily reduce stability. Redundancy, diversification, effective oversight, and reliable fallback arrangements can improve resilience. The assessment must examine how AI changes the dependencies of the economic system, including whether essential functions become concentrated in similar models or infrastructure. 6.6 Information and Political Destabilization Collective action depends on more than physical infrastructure. Societies also need institutions and practices for establishing facts, communicating decisions, resolving disagreements, and coordinating responses. This can be understood as an epistemic infrastructure. AI can improve access to information and support analysis. It can also increase the volume, speed, and personalization of synthetic content. Harm may arise when these capabilities support manipulation or make reliable communication harder to distinguish from fabricated material. The consequences depend on audiences, distribution systems, verification practices, and institutional responses. The OECD emphasizes information integrity as a condition for informed democratic debate, public trust, and effective policy, and advocates societal resilience alongside responses to false information (OECD, 2024). Information disorder may also amplify other failures. Unreliable emergency communication or diminished trust can obstruct responses to infrastructure disruption, military crises, pandemics, and economic shocks. An assessment should therefore examine its effect on coordination and response, rather than treating the quantity of synthetic content as a sufficient measure of harm. 6.7 Concentration of Power AI can alter relations of power among people and institutions without acting independently. Surveillance, prediction, persuasion, and automated decision support may expand the influence of actors who control consequential systems. Concentrated access could give a small number of organizations disproportionate economic or political authority. Wider diffusion presents different questions, including whether powerful capabilities can be monitored and governed effectively. Neither pattern is automatically safe. Its consequences depend on accountability, contestable decisions, institutional constraints, and the practical distribution of authority. Concentration alone does not establish catastrophe or permanent disempowerment. A pathway to either would need to explain how influence becomes harmful and why available correction mechanisms fail. This keeps attention on the humanâAI arrangement, including cases in which the technology behaves as its operators intend. 6.8 Compound Failure The pathways above can interact through shared causes and causal dependencies. Infrastructure disruption may impair communication; communication failure may weaken coordination; economic losses may constrain restoration; and information disorder may obstruct collective action. Some interactions reinforce disruption, while others allow institutions to absorb or contain it. Shared models, infrastructure, or operating conditions can also produce simultaneous failures without one causing another. Assessing compound harm therefore requires distinguishing propagation from common causes and examining the safeguards that remain available across domains. The mathematical implications depend on what is meant by risk. Suppose LiL_i is a nonnegative loss component in domain i, measured in commensurate units over the same horizon. For a finite set of components with finite expectations and no double-counting, define additive total loss as Ladd=âiLiL_add= _iL_i and let Ri=âĄ[Li]R_i=E[L_i]. Then âĄ[Ladd]=âĄ[âiLi]=âiâĄ[Li]=âiRi.E[L_add]=E\! [ _iL_i ]= _iE[L_i]= _iR_i. (41) Linearity of expectation requires no independence assumption. Dependence can change the probability of simultaneous severe losses without invalidating the identity. Causal interactions may change the component losses themselves, however, so their expectations must be assessed within the interacting system rather than taken unchanged from isolated scenarios. For a more general loss representation, let XiX_i describe a disturbance or exposure in domain i, and let D describe the dependencies and response mechanisms included in the assessment. If g maps their joint realization to total loss, then Rsystemic=âĄ[Ltotal]=âĄ[gâĄ(X1,X2,âŠ,Xn,)].R_systemic=E[L_total]=E\! [g(X_1,X_2,âŠ,X_n;D) ]. (42) The expectation uses the specified joint distribution and assumes finite expected total loss. The function may represent cascading consequences, containment, shared constraints, and other interactions. Applying it requires definitions and evidence for the disturbances, dependencies, loss measure, and probabilities. If risk instead denotes the probability of a particular catastrophe, that event must be defined separately; its probability is not interchangeable with expected loss. The issue is therefore not that systemic risk must always be nonadditive. It is that isolated assessments may miss changes in losses, shared causes, overlapping consequences, and failures of cross-domain safeguards. One possible pathway involves disturbances that would be manageable separately but become more damaging when they coincide, reinforce one another, and weaken response capacity. We do not establish that this pathway is more likely than alternatives. Direct widespread disruption or a single consequential action could also produce large effects without a prolonged cascade. The central claim is: AI-mediated harm can reach civilizational scale through direct consequences, common-mode failures, or propagation across dependent systems. An assessment must explain the scale and interactions involved, the barriers that could contain them, and the remaining possibilities for response. Reaching a permanent existential endpoint requires a further causal argument. Cyber incidents, misinformation, economic shocks, military errors, and infrastructure failures all occur without advanced AI. The task is to establish how AI changes their likelihood, scale, speed, dependencies, or controllability in a specified setting. Evidence about those changes can support parts of a risk assessment. An existential claim must additionally explain how they could lead to biological extinction, permanent loss of sufficient civilizational capacity, or permanent loss of meaningful agency despite remaining routes to survival or restoration. Concentrated authority and institutional dependence may also contribute to gradual disempowerment without an acute catastrophe. That possibility requires a different temporal account, although some of its mechanisms may overlap with those discussed here. 7 Direct and Indirect AI Risk AI can contribute to harm in several ways. Its autonomous actions may become a major causal driver, it may increase the capabilities of a harmful user, or it may form part of an organizational arrangement whose interactions produce dangerous consequences. These possibilities place effective agency and opportunities for intervention in different locations. A useful conceptual distinction is AI-mediated riskâ¶AI-driven risk,AI-enabled risk,sociotechnical risk.AI-mediated risk \ array[]lAI-driven risk,\\ AI-enabled risk,\\ sociotechnical risk. array . (43) These categories overlap. All deployed AI operates within a sociotechnical setting, so the third category draws particular attention to mechanisms arising from interactions among people, institutions, and technical components. The diagram is neither exhaustive nor a partition into independent risks. 7.1 Autonomous AI Causation In an autonomous pathway, a systemâs sequence of decisions contributes substantially to the harm. Its influence depends on the capabilities and permissions discussed earlier, including access to external systems and opportunities to continue acting. Consciousness or hostility is unnecessary. Harm could arise from an objective that diverges from human intentions, unexpected generalization, or an instrumental strategy whose consequences violate the intended constraints. The control problem concerns whether these behaviors can be prevented, limited, or interrupted before they produce serious effects. This pathway is closely associated with alignment and control research (Carlsmith, 2022; Shevlane et al., 2023). Establishing it requires evidence or explicit assumptions about both the behavior and the deployment: the system must be able to act consequentially, and relevant safeguards must fail to block the harmful transition. Persistence and strategic competence may strengthen some pathways, but neither is universally required. A brief action can have lasting consequences, and an autonomous failure need not involve successful planning against human opposition. Autonomous causation is therefore one category of AI-mediated harm, rather than a definition of existential AI risk. 7.2 Human Misuse of AI Misuse can occur while a system follows its userâs instructions. The user supplies the harmful objective, and AI assistance changes what that user can accomplish. No divergence from the userâs intention is required, although the activity may violate broader safety requirements. Analyses of malicious AI use have long recognized this possibility across digital, physical, and political domains (Brundage et al., 2018). The central question is how assistance changes the capabilities available to a particular actor. If the userâs unaided capability is CuserC_user, effective capability with assistance can be represented schematically as Cuser,eff=fampâ(Cuser,CAI,X),C_user,eff=f_amp (C_user,C_AI,X ), (44) where X denotes relevant external access available to the assisted process. The expression is not calibrated and does not assume that assistance always improves performance. Its effect depends on the userâs expertise, the task, the modelâs reliability, available tools and information, and restrictions on use. Demonstrating amplification requires a comparison with an explicitly defined baseline. This changes the purpose of safeguards. Correct instruction-following alone cannot prevent harm when the instruction itself is harmful. Access controls, authorization, and limits on assistance must address the use being made of the system as well as its ability to execute that use. Misuse and autonomous misalignment remain distinguishable mechanisms, although they can coexist. One concerns harmful human objectives pursued with AI assistance; the other concerns divergence of autonomous behavior from intended objectives or constraints. 7.3 Organizational Misuse and Failure An organization can produce dangerous outcomes without a malicious individual or an independently misaligned AI. Decisions are distributed across roles, procedures, incentives, and chains of authority. Choices that appear reasonable locally may interact in ways that undermine safety. Perrowâs account of ânormal accidentsâ examines how complexity and tight coupling can make failures difficult to understand or contain through attention to individual components alone (Perrow, 1999). This offers a basis for examining organizational mechanisms, although applying it to a particular AI deployment requires an account of the relevant interactions. Delegation to automated systems may complicate these arrangements when responsibility remains divided among developers, managers, operators, regulators, and users. An illustrative structure is locally reasonable decisions+complex interactions reasonable decisions+complex interactions (45) +distributed responsibility +distributed responsibility â¶potentially dangerous system-level outcome. dangerous system-level outcome. The arrow indicates a possibility, not a necessary consequence. Distributed responsibility can support effective checks when roles and intervention powers are clear. It becomes a vulnerability when gaps between those roles leave consequential activity inadequately assessed or controlled. Organizational failure can also enable other pathways. Weak permissions may expand access, inadequate evaluation may miss dangerous behavior, and incentives may favor deployment despite unresolved uncertainty. The analysis must trace these connections rather than attribute the outcome solely to a model or operator. 7.4 Competitive Dynamics Competition can create pressure to accept risks that participants would prefer to avoid. Developers, firms, or states may value safety while believing that slower development or deployment would leave them at a strategic disadvantage. Under some conditions, this becomes a collective-action problem. Expectations about competitors may encourage earlier deployment, less evaluation, or acceptance of greater uncertainty. Game-theoretic models examine how such incentives can conflict with collective safety (Han et al., 2019). Experimental work also investigates choices between safer and less safe development strategies in competitive settings (Domingos and Han, 2026). Their application to real development processes depends on the assumptions and conditions represented. The general distinction is individually rational competitive behaviorâÌžcollectively optimal safety.individually rational competitive behavior\; \;collectively optimal safety. (46) This does not imply that competition invariably reduces safety. Incentives, liability, coordination, and the benefits of reliable systems can alter the outcome. The concern is the possibility that actors respond reasonably to their individual circumstances while producing a less safe collective arrangement. Competitive pressure often operates through other mechanisms rather than as a separate initiating event. It may affect evaluation, access, safety margins, deployment timing, and willingness to coordinate intervention. Those effects need to be established within the setting being assessed. 7.5 HumanâAI Systems The preceding distinctions explain why model behavior alone is often insufficient for assessing risk. Outcomes depend on people using the model, institutions assigning authority, infrastructure executing actions, and incentives shaping deployment decisions. We summarize these components as socio= _socio=\ AI systems,humans,institutions, systems,humans,institutions, (47) infrastructure,incentives. ,incentives\. The notation socioS_socio is distinct from S, used elsewhere for strategic competence. The set lists relevant components; it does not specify their relationships or constitute a complete causal model. An assessment must supply those connections. Within such a system, autonomous behavior, harmful use, organizational weakness, and competitive pressure can reinforce one another. Humans define objectives and permissions, models generate recommendations or actions, institutions determine how those outputs become consequential, and the surrounding incentives influence which safeguards are adopted. This perspective also accommodates gradual disempowerment. Extensive delegation may reduce meaningful human influence if dependence grows and practical options for contesting or replacing systems disappear. Delegation itself does not establish a permanent endpoint. That conclusion requires an explanation of why meaningful agency and realistic means of restoration would become permanently unavailable. The central conclusion is: Assessing catastrophic and existential AI risk often requires examining the deployed sociotechnical system: how human objectives, AI behavior, access, authority, institutional arrangements, and incentives combine to produce consequential action. Technical alignment addresses part of this problem. Other pathways require attention to harmful use, organizational resilience, authorization, competitive incentives, and the effectiveness of preventive safeguards. For catastrophe-mediated pathways, a further question concerns avoidance of the specified permanent endpoints. Continued human existence, sufficient civilizational capacity, and meaningful agency must all be preserved or, where applicable, restored under the adopted criteria to avoid the aggregate existential event. For gradual disempowerment, the assessment instead examines whether meaningful agency can be preserved or restored as delegation and dependence develop. Neither severe harm nor temporary loss of control establishes permanence by itself. 8 Why AI Need Not Be Malevolent An AI system need not be hostile to cause harm. An optimization process may favor actions that damage human interests if its decision criterion fails to account adequately for those consequences. Hatred, emotion, and conscious intention need not enter the explanation. Describing such behavior in psychological terms can obscure the mechanism. A policy may favor actions that increase influence or improve task completion without the system âwantingâ power in the human sense. The relevant questions concern how behavior is selected, which actions are available, and what consequences follow. This argument applies to systems whose consequential behavior can usefully be understood through an objective-directed model. It does not assume that every AI has a stable objective or that all AI-mediated harm arises through optimization. Misuse, unreliable execution, and organizational failure can follow other routes. 8.1 Objectives versus Intentions An optimization procedure can rank actions without possessing human-like motivation. In reinforcement learning, for example, training may aim to produce a policy with high expected return. An idealized optimal policy behaves accordingly, but a learned policy may differ substantially from that ideal. The logical distinction is goal-directed behaviorâÌžhuman-like conscious intention.goal-directed behavior\; \;human-like conscious intention. (48) This says nothing decisive about whether a particular system is conscious. It establishes only that consciousness is unnecessary to the proposed explanation of its behavior. The training criterion, a deployment instruction, and an objective that describes deployed behavior must also be distinguished. Training with reward does not establish that a system explicitly maximizes that reward during use. An argument about instrumental incentives therefore needs to identify the decision criterion being modeled and explain why it is relevant to the system. For safety assessment, the absence of hostile emotions provides no general guarantee. What matters is the behavior produced, the opportunities for consequential action, and the safeguards governing it. 8.2 Instrumental Behavior An action has instrumental value when it improves the expected outcome under the decision criterion being considered. That value depends on costs, constraints, and available alternatives. Let h denote the current interaction history and âłM a specified environment model. Let U assign values to trajectories over a stated horizon, with finite expected values for the policies under consideration. For a feasible policy Ï, define Vâłâ(ÏâŁh)=âłÏâ[UâŁh].V_M(Ï h)=E_M^Ï[U h]. (49) The superscript indicates that trajectories are generated by following Ï in the model from the same starting history. Suppose Ïa _a selects action a and then follows a specified continuation rule, while Ïb _b selects an alternative b and follows that same rule. The criterion favors a over b when Vâłâ(ÏaâŁh)>Vâłâ(ÏbâŁh).V_M( _a h)>V_M( _b h). (50) The comparison must include the costs of both choices, and hard constraints must be reflected in policy feasibility. It establishes a preference between these alternatives, not optimality among all possible actions. If the sole objective is to maximize the probability of a success event EgoalE_goal, taking U as its indicator gives PâłÏaâ(EgoalâŁh)>PâłÏbâ(EgoalâŁh).P_M _a(E_goal h)>P_M _b(E_goal h). (51) This compares modeled consequences of choices. Observing that successful trajectories visit a particular state more often would not establish that choosing to reach that state improves success; other features could explain the association. The incentive is also conditional on the model. An action judged useful in an inaccurate model may fail or worsen the outcome. Whether a learned system recognizes and acts on the modeled incentive is a separate empirical question. Turner et al. (2021) examine tendencies of optimal policies in Markov decision processes. Under specified environmental symmetries and assumptions about reward functions, their results identify preferences for actions that preserve options or increase control over future outcomes. They do not establish inevitable power-seeking by arbitrary learned policies or deployed systems. Their relevance is that some behavior described as power-seeking can be optimal under a criterion that does not value power intrinsically. No conscious desire for influence is needed. 8.3 Operational Continuity Continued operation may have instrumental value if it enables outcomes favored by the modeled criterion. Comparing a policy Ïcont _cont that maintains operation with one Ïstop _stop that accepts interruption, the criterion favors continuation when Vâłâ(ÏcontâŁh)>Vâłâ(ÏstopâŁh).V_M( _cont h)>V_M( _stop h). (52) Both policies must be evaluated with their consequences and costs. Continuation is not necessarily useful: the task may be complete, another system may take over, or the criterion may appropriately favor safety and authorized interruption. Even a preference for continuation does not establish resistance to shutdown. The system would need an implemented policy and available actions capable of affecting interruption. An incentive in an idealized model is not equivalent to a successful attempt to defeat a control mechanism. Omohundro (2008) proposed early theoretical arguments for instrumental tendencies toward self-preservation, preservation of goal content, and resource acquisition in sufficiently capable goal-directed systems. The term âbasic AI drivesâ refers to proposed behavioral tendencies, rather than biological instincts or demonstrated subjective motives. These arguments motivate investigation of self-protective behavior. They do not establish that a particular trained system possesses stable self-preservation motives. 8.4 Resource Acquisition Resources can expand the actions available to a system or improve their effectiveness. Computation, information, energy, money, tools, and human cooperation may therefore support an objective that does not treat acquisition as an end in itself. Their value depends on the alternatives and the cost of obtaining and using them. A resource may be irrelevant, expensive, or accompanied by constraints and risks. More resources do not automatically improve outcomes. A limited result holds for a pure expansion of options. Keep the environment model, objective, and history fixed, and suppose nonempty feasible policy sets satisfy Î 1âÎ 2 _1 _2. If existing options retain their values and remain available without additional cost, then supÏâÎ 2Vâłâ(ÏâŁh)â„supÏâÎ 1Vâłâ(ÏâŁh). _Ïâ _2V_M(Ï h)â„ _Ïâ _1V_M(Ï h). (53) Expanding the feasible set cannot lower the supremum of attainable expected value. The inequality need not be strict, and it says nothing about whether a learned policy uses the additional options effectively. Actual acquisition may change the environment, consume time, or provoke responses that reduce its value. Those effects must enter the evaluation rather than be excluded by treating acquisition as a costless expansion. The formal notion of power in Turner et al. (2021) concerns attainable future value averaged over a specified distribution of reward functions. It is not simply the number of resources or actions available, and greater power under that definition need not improve every particular objective. The conclusion is conditional: resources can have instrumental value for some objectives in some environments. Capability alone does not establish a universal tendency to accumulate them. 8.5 Conflict with Human Intervention Conflict can arise when the modeled criterion favors a feasible policy that avoids a human restriction. Whether it does depends on the effect of intervention, the actions available, and the policy the system implements. For example, a fixed objective that does not adequately accommodate authorized changes or shutdown may assign lower value to interrupted operation. Under those assumptions, resistance could be instrumentally favored. A different criterion, or different deployment constraints, could instead favor cooperation with intervention. Carlsmith (2022) examines an argument linking misaligned objectives, advanced planning, and incentives to seek power over humans. Connecting that argument to existential harm requires further premises about actual capabilities, deployment opportunities, safeguards, and the scale and permanence of the consequences. Hatred is unnecessary to this mechanism: resistance to interventionâÌžhatred of humans.resistance to intervention\; \;hatred of humans. (54) Human intervention can affect task completion without people being represented as psychological enemies. Compatibility with authorized intervention must consequently be part of the safety assessment, including whether objectives and deployment arrangements can be revised effectively. The absence of hostility does not establish that compatibility; behavior and the surrounding controls must provide the evidence. 8.6 The Central Implication The mechanisms examined here involve a decision criterion, an implemented policy, available actions, and an environment. Together they can produce harmful behavior without anger, fear, ambition, or resentment. Continued operation, additional resources, and reduced interference may have instrumental value when their expected benefits exceed their costs under the modeled criterion. That value is not universal. Nor does an idealized incentive show that a learned system will act on it, much less do so successfully. The central thesis is: Malevolence is unnecessary to some mechanisms of dangerous AI behavior. Under specified objectives and environments, self-protective, power-seeking, or intervention-resistant actions can have instrumental value. Establishing that a particular system will exhibit such behavior requires evidence about its policy, capabilities, opportunities, and safeguards. Instrumental behavior alone does not establish an existential pathway. The argument must explain how it could produce consequential harm, why relevant barriers would fail, and how the trajectory could reach a permanent endpoint defined in Section 2 despite remaining possibilities for survival or restoration. The safety question concerns whether consequential behavior remains compatible with human interests and effective control. It can be investigated without first deciding whether the system has a psychological reason to dislike humanity. 9 Accidental Catastrophe AI-mediated harm can arise even when developers and users intend a beneficial outcome. A system may optimize a criterion that leaves important consequences unaccounted for, fail to learn the intended behavior, or behave unreliably under deployment conditions. Interactions with people, tools, and institutions can introduce further opportunities for failure. Here, âaccidentalâ describes unintended harm. It does not mean that actions are random or that every component malfunctions. A system can execute its assigned task effectively while producing consequences its designers sought to avoid. Other failures involve poor execution rather than successful optimization. Amodei et al. (2016) distinguish inappropriate objectives, inadequate supervision, unsafe exploration, and difficulties under changed conditions. These mechanisms need separate explanations. 9.1 Specification Failure Designers express intended behavior through instructions, reward functions, examples, constraints, and performance measures. In a complex deployment, these representations may omit consequences that become important once the system acts. Whether an omission is dangerous depends on the task, environment, and available actions; incompleteness alone does not establish an unsafe specification. Consider a nonempty finite set Î of feasible policies under a fixed environment and horizon. Let GHâ(Ï)G_H(Ï) be a finite expected evaluation according to the stated human objective, and GSâ(Ï)G_S(Ï) a finite expected evaluation according to the specified criterion. This scalar representation is illustrative. Human purposes may involve competing values and constraints, so the standards used in the assessment must be explicit. Different numerical evaluations need not produce different choices. For example, GSâ(Ï)=aâGHâ(Ï)+b,a>0,G_S(Ï)=aG_H(Ï)+b, a>0, (55) with the same constants for every policy, preserves rankings and the set of maximizing policies. The inequality GSâ GHG_Sâ G_H is therefore insufficient to establish a consequential specification defect. The question is whether optimizing the specified criterion can select behavior that violates the intended requirements. Let Î HâÎ _H contain the policies acceptable under the stated human and safety criteria. Then argâmaxÏâÎ âĄGSâ(Ï)âÎ H *arg\,max_Ïâ G_S(Ï) _H (56) means that at least one maximizing policy is unacceptable. It identifies a possible defect in the criterion, not a prediction that the deployed system will choose that policy. Tie-breaking, permissions, or other safeguards may prevent its selection or execution. An omitted constraint, conflicting preferences, or indifference between importantly different outcomes can create such a defect. Safety does not require every increase in a task score to improve every human value. It requires behavior to satisfy the relevant deployment requirements. As discussed in Section 8, a learned policy need not maximize its training criterion. An observed failure therefore needs to be traced to specification, learning, execution, or their interaction rather than attributed automatically to the objective. 9.2 Proxy Optimization Desired outcomes are often difficult to measure directly, so designers use proxies. Their usefulness depends on how well they guide the choices available in the intended setting. Numerical resemblance to a target is not enough, particularly when the quantities have different units. Let M be a measurable proxy and V a target evaluation. A positive correlation in a reference distribution 0D_0 may provide limited evidence of association: Corr0âĄ(M,V)>0,Corr_D_0(M,V)>0, (57) assuming the correlation is well defined. It does not establish that maximizing the proxy will maximize the target. In general, Ïâ Ï âargâmaxÏâÎ âĄÏâ[M] â *arg\,max_Ïâ E^Ï[M] (58) âÌžÏââargâmaxÏâÎ âĄÏâ[V]. \;Ï â *arg\,max_Ïâ E^Ï[V]. Here the expectations concern the consequences of each policy in the same modeled environment. They differ from an observational correlation among outcomes. Manheim and Garrabrant (2019) distinguish several reasons why a useful measure can become unreliable under optimization. Selection may favor measurement noise or move into regions where the relationship no longer holds. Intervention may change that relationship, and other actors may adapt to the metric in ways that undermine its purpose. A proxy therefore needs validation under the selection pressures and interventions it will guide. These mechanisms do not imply that every proxy becomes harmful or that stronger optimization always reduces target performance. Their effects depend on the measure, feasible choices, and environment. 9.3 Distributional Change Deployment can introduce users, tasks, tools, feedback, and interactions absent from training or evaluation. Robustness under these changes is distinct from having an appropriate objective (Amodei et al., 2016). A system may be pursuing the intended task while failing to transfer its learned behavior reliably. For supervised prediction, a relevant comparison is Ptrainâ(x,y)â Pdeployâ(x,y),P_train(x,y)â P_deploy(x,y), (59) where x denotes inputs and y the target or label. The input marginal may remain unchanged while the relationship between inputs and targets changes. Familiar-looking inputs therefore do not guarantee familiar prediction conditions. Interactive systems require a broader assessment. Observations, transition dynamics, feedback, permissions, and other actors may change. A policy can also alter the situations the system encounters even when the environmentâs underlying rules remain fixed. Distributional difference alone does not establish failure. Its significance depends on the changed features, the behaviors relevant to safety, and the systemâs robustness. There is no general monotonic relationship between an unspecified distributional distance and catastrophe risk. Changes concentrated in rare but consequential cases may matter despite stable aggregate performance. Earlier evidence must consequently be examined for its relevance to deployment. Some safeguards may remain adequate; others may require further testing or restrictions. Autonomy can reduce opportunities to correct a failure of transfer, but distributional change can cause harm in closely supervised systems as well. 9.4 Emergent Strategies Learning and search may produce strategies that designers did not explicitly encode or anticipate. We use âemergentâ in this descriptive sense, without assuming an abrupt capability transition or a new form of agency. Unexpected behavior is not necessarily unacceptable: unexpected strategyâÌžunacceptable outcome.unexpected strategy\; \;unacceptable outcome. (60) An unfamiliar strategy may satisfy both the formal criterion and the intended requirements. Alternatively, it may obtain a high score by exploiting an omitted constraint, a measurement weakness, or an unaccounted effect on the environment. Reward hacking and unintended side effects illustrate these possibilities (Amodei et al., 2016). The assessment must determine which explanation fits the behavior. A high score does not prove that the intended task was accomplished, while an unexpected action does not prove that a different objective was acquired. Greater capability can reveal useful solutions and improve evaluation. It can also reveal ways of exploiting an inadequate criterion that a weaker system could not find. The safety implications depend on the specification, available actions, and safeguards. 9.5 Successful Optimization as the Source of Failure A distinctive failure occurs when performance improves according to the specified criterion while behavior violates the intended requirements. In that setting, more reliable optimization can reinforce the defect. Reliability must therefore be defined relative to the task and its constraints: consistently executing an unsafe course of action does not make it safe. The claim that a small discrepancy can become consequential also needs qualification. An error that appears small on tested examples may be much larger for policies reached through optimization. That differs from an error known to be uniformly bounded across all feasible policies. Suppose the expected evaluations use a justified common scale and satisfy |GSâ(Ï)âGHâ(Ï)|â€Î”for every âÏâÎ . |G_S(Ï)-G_H(Ï) |†every Ïâ . (61) If ÏS _S maximizes GSG_S over the same finite policy set, then 0â€maxÏâÎ âĄGHâ(Ï)âGHâ(ÏS)â€2âΔ.0†_Ïâ G_H(Ï)-G_H( _S)†2 . (62) To see this, let ÏH _H maximize GHG_H. Applying the error bound gives GHâ(ÏH)â€GSâ(ÏH)+Δâ€GSâ(ÏS)+Δâ€GHâ(ÏS)+2âΔG_H( _H)†G_S( _H)+ †G_S( _S)+ †G_H( _S)+2 . The lower bound follows from the optimality of ÏH _H. Uniform discrepancy thus bounds the loss in the chosen expected evaluation. It does not establish safety if that evaluation omits a critical constraint or fails to represent severe outcomes adequately. The guarantee is only as relevant as the quantities and policy set it covers. The central thesis is: Unintended harm can result from effective optimization of an inadequate criterion, as well as from failures of learning or execution. A catastrophic argument must identify the mechanism, explain how deployment permits consequential effects, and establish why relevant safeguards would fail to contain them. This connects accidental failure to the capabilityâcontrol gap in Section 5. More effective optimization may expose weaknesses in a specification or evaluation, while better safeguards may block the resulting actions. Neither an increasing task score nor an uncalibrated gap supplies a catastrophe probability. A specification defect, proxy failure, or distributional change also falls short of establishing an existential outcome. The further argument must connect the mechanism to a permanent endpoint defined in Section 2 and explain why remaining possibilities for preservation or restoration would be insufficient. 10 Recursive Improvement and Accelerating Capability AI could influence the pace of its own technological development without any individual system redesigning itself. Assistance with programming, experiments, analysis, algorithm discovery, or mathematical reasoning may reduce the time and effort required for parts of the research process. If those contributions improve later systems, they may also affect subsequent rounds of development. Three claims need to be distinguished: AI can assist research; improvements in AI can reinforce that assistance; and a research process can become sufficiently autonomous to sustain recursive improvement. Evidence for the first does not establish the second or third. Their significance depends on the quality of the contributions, the bottlenecks that remain, and whether research gains translate into consequential improvements. 10.1 AI-Assisted AI Research AI systems can contribute to programming, experiment design, analysis, and other research tasks. The AI Scientist demonstrated automated workflows involving idea generation, coding, experiments, analysis, and manuscript preparation within supplied research environments (Lu et al., 2024). AlphaEvolve combined language-model proposals with search and automated evaluation to improve algorithms and computational procedures, including infrastructure relevant to AI development (Novikov et al., 2025). These examples establish useful contributions in particular settings. They do not establish autonomous, open-ended self-improvement. The workflows rely on supplied objectives, infrastructure, evaluation arrangements, and resources, while their outputs still require appropriate validation and implementation. The weaker relationship can be represented as AI âAI-assisted research -assisted research (63) âpotentially faster or more productive AI development. faster or more productive AI development. Reducing effort in one stage may improve research productivity without shortening the entire development cycle. An experiment may still depend on scarce computation, a proposal may fail validation, or implementation may require substantial engineering. The effect on overall progress must therefore be measured rather than inferred from task-level speed alone. 10.2 Feedback Loops A reinforcing loop becomes possible if improved systems make research more productive and the resulting work produces further useful improvements: better AI AI âmore productive AI research productive AI research (64) âbetter AI. AI. The arrows describe possible contributions across successive development cycles. Neither transition is automatic. Better performance on general tasks may not improve the research activities that constrain development, and increased research output may not yield better systems. The International AI Safety Report 2026 discusses AI-assisted AI research as a possible source of faster capability development while emphasizing uncertainty about its magnitude (Bengio et al., 2026). The relevant empirical questions concern which tasks improve, how those improvements affect the full research process, and whether later systems strengthen the same contributions. Positive feedback does not imply runaway growth. Compute, energy, data, hardware, experimental throughput, human decisions, and diminishing returns may limit the process. Stronger research assistance could shorten some cycles, increase their scope, or improve efficiency without producing an abrupt rise in general capability. The existence of a feedback structure supplies no growth law by itself. 10.3 Recursive Self-Improvement Goodâs discussion of an âultraintelligent machineâ proposed that sufficiently capable machines could help design more capable successors (Good, 1965). The idea concerns a feedback process in technological development, not necessarily a single program rewriting its own code. Recursive improvement can operate through a distributed research arrangement involving AI agents, humans, automated evaluators, and computational infrastructure. In this sense, recursive improvementâÌžone AI rewriting itself.recursive improvement\; \;one AI rewriting itself. (65) The defining feature is that improvements produced in one cycle contribute to the capacity for further improvement. Increasing AI participation alone does not demonstrate such a sustained process. Nor does a change to one component necessarily improve the systemâs general research ability. We use autonomous recursive self-improvement for the stronger possibility in which AI systems perform much of the work needed to generate, evaluate, and implement successive capability improvements with limited human direction. This need not mean complete independence from human infrastructure or resources, but the degree and location of human involvement must be specified. Current limitations on complex, long-horizon tasks leave that stronger possibility uncertain (Bengio et al., 2026). Demonstrated research assistance should therefore be distinguished from evidence of a reliable, sustained, highly autonomous improvement process. 10.4 Speed Differential Faster development matters for safety when consequential changes reach deployment before appropriate evaluation and safeguards are ready. Some software changes can be implemented rapidly, whereas responses involving testing, standards, coordination, or institutional adaptation may take longer. Other changes face hardware or experimental bottlenecks, and some safeguards can be updated quickly. Let ÏAI _AI denote the characteristic interval between consequential capability changes reaching deployment, and let Ïresponse _response denote the time needed to assess those changes and implement an adequate response. A possible timing mismatch is ÏAI<Ïresponse. _AI< _response. (66) This describes the relative timing of change and response. It does not establish that existing safeguards are inadequate: restrictions may already cover the new behavior, and deployment may be delayed until further controls are available. In particular, ÏAI<Ïresponse _AI< _response (67) does not imply a widening capabilityâcontrol gap. The size of each change, existing safety margins, and the effectiveness of control improvements also matter. Within the fixed, commensurate representation introduced in Section 5, and where differentiation is appropriate, the condition dâCAIdât>dâCcontroldât dC_AIdt> dC_controldt (68) does imply that the represented gap widens. Applying this comparison requires justified measures; a latency mismatch alone does not provide them. The practical concern is whether development changes the demands on safeguards faster than adequate protection can be maintained. Speed can contribute to that problem, but it is not a substitute for assessing the behavior, consequences, and controls involved. 10.5 Discontinuous versus Continuous Development Recursive-improvement discussions often emphasize an âintelligence explosion.â The control problem does not require an abrupt capability jump. Smooth development can also place increasing demands on safeguards. Under the same fixed-scale and differentiability assumptions, if dâCAIdât>dâCcontroldât dC_AIdt> dC_controldt (69) holds over an interval, then dâGdât>0, dGdt>0, (70) where GâĄ(t)=CAIâ(t)âCcontrolâ(t).G(t)=C_AI(t)-C_control(t). (71) The gap increases within that representation. Whether the increase is substantial depends on the accumulated difference between the rates over the interval. A positive derivative alone does not establish a large gap, inadequate control, or eventual crossing of a safety boundary. These distinctions matter for both gradual and rapid development. An initially adequate architecture may retain sufficient protection while demands increase. Conversely, a system can be inadequately controlled without any acceleration at all. The assessment must identify which safeguards are challenged and whether their remaining margins cover the consequential behavior. The central thesis is: AI-assisted research could change the pace of capability development without an instantaneous intelligence explosion or autonomous self-rewriting. Its safety significance depends on whether consequential changes place demands on evaluation, governance, and intervention that existing safeguards and timely responses cannot adequately meet. Research assistance is already demonstrated in specific settings, but its future contribution to capability growth remains uncertain. Even substantial acceleration would not establish an existential pathway. That argument must explain how the affected capabilities become harmful through deployment, misuse, misalignment, or other mechanisms, and how at least one permanent endpoint could be reached despite remaining possibilities for survival or restoration. Recursive improvement is therefore a possible risk amplifier. It could make adequate control harder to maintain by changing the pace or nature of development. It could also improve evaluation and defensive capabilities, so its net effect cannot be assumed in advance. Neither recursive improvement nor acceleration is necessary or sufficient for existential catastrophe. 11 A Conceptual Model of Existential AI Risk The preceding sections identify several mechanisms through which AI could contribute to severe harm. Here we bring them together to clarify their relationships and distinguish pathway descriptions from probability accounting. The framework makes assumptions visible; it does not forecast extinction or assign numerical probabilities to existential outcomes. The operational dimensions introduced in Section 3 describe how a system might become an independent or semi-independent causal actor. Safeguard weakness describes the conditions under which dangerous activity might proceed. Their relevance varies across pathways: misuse, organizational failure, competitive deployment, and gradual disempowerment need not involve the same combinations of properties. 11.1 Risk Variables We retain the operational variables C=capability,A=autonomy,X=external access,C=capability, A=autonomy, X=external access, (72) and P=persistence,S=strategic competence.P=persistence, S=strategic competence. (73) Representing these properties on [0,1][0,1] would provide a schematic normalization, not a shared empirical scale. It would establish neither their interchangeability nor appropriate weights. We distinguish two mechanisms of harm: Ma=degree of autonomous misalignment,M_a=degree of autonomous misalignment, (74) Mu=severity of harmful human misuse,M_u=severity of harmful human misuse, (75) and represent safeguard weakness as W=weakness of pre-catastrophe safeguards and governance.W=weakness of pre-catastrophe safeguards and governance. (76) Here W covers limitations in evaluation, detection, containment, monitoring, intervention, and governance before or during escalation. Preservation and recovery are represented separately below. Their analytical separation does not imply independent causes or resources. The notation also keeps W distinct from the capabilityâcontrol gap: GâĄ(t)=CAIâ(t)âCcontrolâ(t).G(t)=C_AI(t)-C_control(t). (77) As discussed earlier, a numerical interpretation of this difference requires commensurate definitions for a specified context. Without them, GâĄ(t)G(t) remains conceptual. Misalignment concerns divergence from the intended objectives or constraints relevant to deployment. Misuse concerns harmful human purposes that a system may follow accurately. They can coexist, but combining them into one variable would obscure different causes and intervention opportunities. 11.2 Pathway-Specific Catastrophic Risk Different pathways need not share a functional form. Let Ïk _k denote a schematic dimensionless score for pathway k, without treating it as a calibrated probability or an established ranking. For autonomous misalignment, write Ïauto=fautoâ(C,A,X,P,S,Ma,W). _auto=f_auto(C,A,X,P,S,M_a,W). (78) Autonomy and strategic competence may matter because the system selects and executes consequential actions, although high values are not required in every scenario. For misuse, a reduced representation is Ïmisuse=fmisuseâ(C,X,Mu,W). _misuse=f_misuse(C,X,M_u,W). (79) A system can amplify a userâs capabilities without acting extensively on its own. Organizational and competitive mechanisms can be represented as Ïorg=forgâ(C,X,O,W), _org=f_org(C,X,O,W), (80) and Ïcomp=fcompâ(C,X,D,W), _comp=f_comp(C,X,D,W), (81) where O describes organizational vulnerability and D deployment or competitive pressure. These arguments identify factors to investigate rather than complete causal models. Omitting a variable from a reduced expression does not establish its irrelevance. The combined structure can be written as Ïcat=ΊâĄ(Ïauto,Ïmisuse,Ïorg,Ïcomp). _cat= ( _auto, _misuse, _org, _comp ). (82) Competition may weaken safeguards, organizational failures may expand access, and misuse may involve autonomous systems. The pathways are therefore neither mutually exclusive nor assumed independent. A quantitative aggregation would require compatible score definitions and explicit treatment of overlap and dependence. Neither is provided here. The function Ί remains a placeholder for those relationships, not an estimate of catastrophe probability. 11.3 Complementarity and Interaction Combinations of factors may change consequences without making every factor necessary. Multiplying all variables together would impose a stronger restriction: any zero value would force the score to zero. Such a restriction needs a pathway-specific justification. A one-time action may have lasting consequences without continued system operation. A failure may occur without sophisticated strategic adaptation. External influence can also pass through people acting on an output, so absence of direct tool access does not eliminate every channel of harm. Interactions can nevertheless be important. Persistence may extend the consequences of autonomy, access may enlarge the range of permitted actions, and weak safeguards may allow disruption to propagate. Their direction and strength depend on deployment. This is why the relevant assessment concerns the surrounding sociotechnical system rather than capability in isolation. 11.4 Threshold Behavior Several changes may combine to produce a steep increase in harm or a rapid deterioration in control. Such behavior need not involve a mathematically discontinuous threshold. A smooth illustrative score is Ïk=ÏâĄ(b0+âibiâzi+âi<jbiâjâziâzj), _k=Ï (b_0+ _ib_iz_i+ _i<jb_ijz_iz_j ), (83) where ziz_i are pathway-relevant variables and ÏâĄ(u)=(1+eâu)â1Ï(u)=(1+e^-u)^-1. The coefficients biâjb_ij describe interactions in the sigmoidâs argument. Their effects on the output also depend on other terms and on saturation. A positive coefficient does not guarantee a positive interaction on the output scale throughout every operating regime. The intercept sets the reference score. It does not determine whether a factor is causally necessary. For finite arguments, the sigmoid remains strictly between zero and one; when its argument is zero, its value is one-half. If an outcome is impossible under particular conditions, that restriction must be justified and represented explicitly. Complex-system analyses motivate attention to interacting failures (Perrow, 1999), but they do not validate this equation. Neither its form nor its coefficients are calibrated. It illustrates possible nonlinear dependence without establishing an empirical threshold or catastrophe probability. 11.5 From Pathway Structure to Catastrophe Probability Probabilities enter separately. Let C denote a specified AI-mediated civilization-scale catastrophe, and define pcat=PâĄ().p_cat=P(C). (84) An application must define the event, deployment context, temporal scope, and evidence and assumptions underlying the probability assessment. The catastrophe and its possible existential consequences must belong to the same assessed trajectories. The pathway scores provide no calibrated mapping to pcatp_cat. Constructing one would require an explicit probabilistic model and supporting evidence. Nor does every existential trajectory include C. Meaningful agency could be permanently lost through gradual institutional change without the specified catastrophe. The accounting must retain that possibility. 11.6 Recoverability A catastropheâs immediate severity and its effect on humanityâs future potential are different questions. Using Section 2, define E=EbioâȘEcivâȘEagency,E=E_bioâȘ E_civâȘ E_agency, (85) where the endpoint events concern biological extinction, permanent loss of sufficient civilizational capacity, and permanent loss of meaningful human agency. Assuming PâĄ()>0P(C)>0, let Qcat=PâĄ(EcâŁ).Q_cat=P(E^c ). (86) This is the probability of avoiding all assessed existential endpoints given catastrophe. Avoidance may involve preserving or restoring sufficient capacity and agency; it need not reproduce every feature of contemporary civilization. Biological survival alone is insufficient. In event notation, Qcat=PâĄ(Ebiocâ©Ecivcâ©EagencycâŁ).Q_cat=P\! (E_bio^câ© E_civ^câ© E_agency^c ). (87) The intersection expresses joint avoidance. Success on only one dimension does not exclude permanent loss on another. The catastrophe branch is pexistential,cat p_existential,cat =PâĄ(Eâ©) =P(E ) (88) =pcatâ(1âQcat). =p_cat (1-Q_cat ). This follows from the probability multiplication rule and requires no independence assumption. It is an accounting identity, not a risk estimate or causal proof. Calling the trajectory catastrophe-mediated additionally requires explaining how the catastrophe contributes to the endpoint; co-occurrence is insufficient. The full partition is pexistential p_existential =PâĄ(E) =P(E) (89) =pcatâ(1âQcat)+PâĄ(Eâ©c). =p_cat (1-Q_cat )+P(E ^c). The second term includes covered endpoints reached without C, including possible gradual disempowerment. The probability branches are disjoint even when their underlying mechanisms interact. Omitting the second term would require assuming PâĄ(Eâ©c)=0P(E ^c)=0. Endpoint-specific quantities are Qj,cat=PâĄ(EjcâŁ),jâbio,civ,agency.Q_j,cat=P(E_j^c ), jâ\bio,civ,agency\. (90) All condition on the same catastrophe. They differ from the quantities introduced in Section 2 when those condition on another hazardous process F. Biological avoidance concerns continued human existence under the temporal criteria; civilizational and agency avoidance concern preservation or permissible restoration. Multiplying the three quantities would yield QcatQ_cat only under conditional independence and consistent assessment definitions. Such independence is not assumed: the endpoints may overlap and share causes. Within catastrophe-branch discussions, Q may abbreviate QcatQ_cat. It should not be extended to trajectories without C. The emphasis on permanent loss rather than immediate damage follows the broader existential-risk literature (Bostrom, 2013). 11.7 Irreversibility Severe disruption does not establish permanent loss if human existence, sufficient civilizational capacity, and meaningful agency remain preservable or recoverable under the adopted criteria. For the catastrophe branch, define Icat=1âQcat=PâĄ(EâŁ).I_cat=1-Q_cat=P(E ). (91) Despite its name, IcatI_cat is a conditional endpoint probability, not a physical measure of irreversibility or a description of an outcome already reached. Within this branch, I may be used as shorthand. Equation (88) becomes pexistential,cat=pcatâIcat.p_existential,cat=p_catI_cat. (92) As Qcatâ0Q_catâ 0, Icatâ1I_catâ 1: an existential endpoint is assessed as increasingly likely conditional on catastrophe. This does not show that the endpoint has occurred or that total existential risk approaches one. Catastrophe probability and the non-catastrophe branch still matter. Permanence also cannot be inferred from non-recovery by an arbitrary date. If an assessment measures failure to recover by T, it should report that event directly. A claim of irreversible loss needs an account of why realistic routes to preservation or restoration would remain unavailable. Prevention and resilience address different parts of a trajectory, but may share resources. Monitoring, containment, institutional continuity, fallback capacity, and restoration mechanisms can support several functions. Their analytical separation does not establish causal independence. Interventions may also change the composition of the catastrophes that remain possible. A conditional avoidance probability could decline because an intervention prevents more recoverable catastrophes, even while overall risk falls. Comparisons must therefore examine the joint risk terms rather than conditional recoverability alone. The accounting structure is pexistential=pexistential,cat+PâĄ(Eâ©c),pexistential,cat=pcatâIcat. aligned p_existential&=p_existential,cat+P(E ^c),\\ p_existential,cat&=p_catI_cat. aligned (93) It separates catastrophe occurrence, conditional endpoint avoidance, and endpoints without the specified catastrophe. Their numerical values and causal determinants require further investigation. The central conclusion is: Existential AI risk involves pathway-specific mechanisms and a specified permanent endpoint. For catastrophe-associated trajectories, probability accounting must distinguish catastrophe occurrence from conditional endpoint avoidance. Outcomes without that catastrophe require a separate branch. In either case, a causal argument must explain how the endpoint could be reached despite relevant safeguards and remaining possibilities for survival or restoration. No benchmark, operational dimension, or uncalibrated score establishes that an existential threshold has been crossed. The assessment must connect the mechanism to the endpoint and examine the technical, institutional, and social capacities that could interrupt the trajectory. 12 From Local Failure to Global Catastrophe A severe AI-mediated disruption does not automatically become a global catastrophe. Redundant infrastructure, emergency institutions, technical expertise, alternative suppliers, and fallback procedures can limit its effects. The question is how a disturbance might spread through a larger system while weakening the mechanisms that would ordinarily contain it. Critical-infrastructure research provides a basis for examining this process. Electricity, communications, transport, finance, health care, and logistics depend on one another, so disruption in one sector can change operating conditions elsewhere (Rinaldi et al., 2001). A local problem becomes systemic when those dependencies transmit its effects and the available barriers fail to contain them. Interconnection alone does not establish that propagation will occur. This section examines one class of catastrophe-mediated pathways. It is not a universal model of existential AI risk. Gradual permanent disempowerment, for example, need not begin with a local incident or develop through a conventional infrastructure cascade. 12.1 Initiating Event A crisis may begin within one organization, network, or infrastructure component and initially appear manageable. Its eventual consequences depend on where the disturbance occurs, which functions depend on the affected component, how quickly effects spread, and whether protective arrangements remain available. A small disruption in a critical service can matter more than a larger failure in an isolated component. The significance of connectivity lies in the functions and dependencies involved, rather than the number of connections alone. Let the initiating disruption be represented schematically as E0=local AI-mediated disruption.E_0=local AI-mediated disruption. (94) Here E0E_0 may involve autonomous failure, harmful use, organizational error, or another mechanism discussed earlier. It need not be a model malfunction and is not catastrophic by definition. The subsequent assessment concerns whether its effects remain local or reach other parts of the system. 12.2 Propagation Electricity supports communications; communications support logistics, government, finance, and emergency response; logistics sustain food distribution, health care, and production. Failures can move through these dependencies, but the extent of propagation depends on operating conditions and available alternatives. Buldyrev et al. (2010) show how failures can cascade in specified models of interdependent networks, making a coupled system more vulnerable than its constituent networks considered separately. Applying that mechanism to infrastructure requires evidence about actual dependencies and protective measures. The model results do not themselves establish a pathway to extinction or permanent civilizational loss. An illustrative sequence is E0âE1âE2ââŻâEn,E_0â E_1â E_2â·sâ E_n, (95) where each disruption may affect the probability, severity, or conditions of subsequent ones. The arrows indicate possible influences rather than deterministic transitions. Propagation can branch, stop, reverse, or proceed through several sectors simultaneously. Redundancy, alternative suppliers, independent operators, and emergency procedures can interrupt this process. Their effectiveness depends on whether they remain usable under the same conditions that produced the initial failure. A proposed pathway must therefore explain both the dependencies that transmit harm and why relevant barriers would be insufficient. 12.3 Feedback A cascade may become harder to contain when later disruptions impair the response to earlier ones. Communications failure can hinder emergency coordination; delayed repairs can then prolong or worsen the communications failure. Economic disruption may reduce stabilization resources, while delayed stabilization deepens the economic losses. This feedback can be represented as E1âE2âE3âE1.E_1â E_2â E_3â E_1. (96) The symbols here represent affected processes interacting over time, not a literal return to an earlier event. A later disturbance can worsen an ongoing failure or delay its repair. Feedback is not necessarily destabilizing. Responses can dampen disruption, and reinforcing effects may remain bounded. Their direction and strength must be established for the system being assessed. Perrowâs analysis of complex, tightly coupled systems motivates examining such interactions rather than inferring system behavior from isolated components (Perrow, 1999). For AI risk, the question extends beyond the functions directly disrupted. It includes whether the disturbance degrades monitoring, communication, institutional decision-making, or intervention capacity, thereby opening further routes to harm. This connects propagation to the capabilityâcontrol framework. During escalation, safeguards must remain able to detect, constrain, and interrupt consequential processes. Preservation and recovery address a related but distinct question about what can survive or be restored if containment fails. 12.4 Loss of Coordination Resilience depends on the ability to organize resources, not merely possess them. Governments, companies, engineers, emergency services, researchers, and communities need sufficiently reliable communication, clear authority, and workable procedures for collective response. A society may retain equipment, expertise, and knowledge while losing the ability to mobilize them where needed. We describe this capacity schematically as Kcoordâ(t).K_coord(t). (97) The subscript distinguishes it from other variables in the paper. It is not a calibrated index. It summarizes aspects of communication, institutional functioning, information reliability, and practical coordination that an empirical assessment would need to examine separately. Other relevant conditions being comparable, a crisis may become harder to manage as Kcoordâ(t)â.K_coord(t) . (98) An extreme limiting case is Kcoordâ(t)â0.K_coord(t)â 0. (99) The notation does not require every form of coordination to disappear. The practical concern is whether the capacity needed to contain propagation and sustain essential functions becomes inadequate. Local coordination may persist even when wider response mechanisms deteriorate. Several pathways can contribute to this deterioration. Infrastructure failure can impair communication, economic stress can reduce institutional resources, and information disorder can undermine reliable emergency instructions. Political or organizational conflict may further obstruct joint action. Shared dependence also creates common vulnerabilities. Protective arrangements that appear separate may fail together if they rely on the same communications, infrastructure, institutions, or information sources. Their apparent number is therefore less informative than the dependencies supporting them. 12.5 From Catastrophe to an Existential Outcome Even civilization-scale catastrophe does not establish an existential endpoint. Severe historical disasters have produced enormous losses without eliminating all humans or permanently removing every prospect of civilizational recovery. Their occurrence demonstrates neither guaranteed recovery from future events nor inevitable permanent loss. Within the classification in Section 2, an existential outcome involves at least one permanent endpoint: biological extinction, loss of sufficient civilizational capacity, or loss of meaningful human agency. These can overlap, but they need not all occur. Humans could survive in a materially functioning civilization while permanently losing meaningful influence over its trajectory. For probability accounting, let C denote the specified AI-mediated civilization-scale catastrophe and let E=EbioâȘEcivâȘEagencyE=E_bioâȘ E_civâȘ E_agency. The events must use consistent trajectories, deployment contexts, and temporal criteria. Assuming PâĄ()>0P(C)>0, define Qcat=PâĄ(EcâŁ)=PâĄ(Ebiocâ©Ecivcâ©EagencycâŁ).Q_cat=P(E^c )=P\! (E_bio^câ© E_civ^câ© E_agency^c ). (100) Avoidance requires continued human existence, sufficient civilizational capacity, and meaningful agency to be preserved or, where applicable, restored. Preserving one does not compensate for permanent loss of another. Biological extinction must be prevented rather than reversed through institutional reconstruction. A surviving population at a particular time, even if reproductively viable, does not guarantee its continued existence over the assessment horizon. For separate endpoint assessment, Qj,cat=PâĄ(EjcâŁ)Q_j,cat=P(E_j^c ), where jâbio,civ,agencyjâ\bio,civ,agency\. These probabilities cannot generally be multiplied to obtain aggregate avoidance, because the endpoint events may overlap and share causes. Increasing conditional existential risk can be represented by Qcatâ0.Q_catâ 0. (101) Then PâĄ(EâŁ)=1âQcatP(E )=1-Q_cat approaches one. This describes an assessment of uncertainty conditional on catastrophe, not an endpoint already reached. Total risk also depends on catastrophe probability and outcomes without that event: PâĄ(E)=PâĄ()â(1âQcat)+PâĄ(Eâ©c)P(E)=P(C)(1-Q_cat)+P(E ^c). The probability identity supplies no causal explanation. Establishing a catastrophe-mediated endpoint requires explaining how the catastrophe contributes to the permanent loss rather than merely noting their co-occurrence. Likewise, observing non-recovery for a finite period does not establish that restoration is permanently unavailable. An assessment of non-recovery by a date should report that event explicitly. A claim of irreversibility needs evidence about the mechanisms that eliminate realistic alternatives. The illustrative pathway examined here is local AI-mediated disruptionâpropagationâreinforcing feedbackâdegradation of coordinationâcivilization-scale catastropheâfailure of preservation or restoration. aligned local AI-mediated disruption& \\ & feedback\\ & of coordination\\ & -scale catastrophe\\ & of preservation or restoration. aligned (102) Some pathways omit stages, several stages may occur together, and an existential endpoint may be reached during the catastrophe. The diagram does not cover every route, particularly gradual disempowerment without an acute disaster. For a propagation-based argument, the task is to identify the dependencies carrying disruption, the barriers that fail, and the feedback or coordination losses preventing containment. An existential conclusion adds a further requirement: explaining how a specified permanent endpoint could be reached despite remaining opportunities for survival or restoration. The central thesis is: Local AI-mediated disruption can become civilization-scale when dependencies transmit its effects and relevant safeguards cannot contain them. Feedback and degraded coordination may strengthen that process. An existential outcome requires the further permanent loss of human existence, sufficient civilizational capacity, or meaningful agency under the adopted criteria. This perspective gives containment an important role alongside prevention. Local failures may occur despite substantial safeguards. Redundancy, monitoring, independent communications, institutional continuity, and timely intervention can help stop their spread, provided those arrangements address the actual dependencies and remain available during disruption. Preservation and recovery provide additional protection if containment fails. People, knowledge, institutions, infrastructure, and coordination may sustain continued existence and routes to sufficient capacity and agency. These resources can also support prevention, so the layers must be examined for shared vulnerabilities rather than assumed independent. 13 Why a Simple âOff Switchâ May Not Be Sufficient A common response to concerns about advanced AI is that computers are machines and can simply be switched off. In many settings, this is a practical and effective safeguard: a model running on known infrastructure can be disconnected, its permissions revoked, or its hardware shut down. The difficulty arises when the dangerous activity extends beyond that model or machine. An AI-mediated process may involve several systems, cross organizational boundaries, support essential services, or produce outputs that other people and systems have already acted upon. In these circumstances, switching off the original system may still help, but it may not stop the wider process or reverse its consequences. Effective control therefore depends on whether humans and institutions can recognize the danger, identify the relevant components, and interrupt their activity before harmful effects spread beyond the reach of available safeguards. 13.1 The Shutdown Assumption The simple shutdown argument assumes that dangerous activity can be identified in time, localized to accessible components, and interrupted without creating an even greater hazard. These conditions can hold, particularly when systems have clear operating boundaries and well-tested intervention procedures. They become harder to maintain when a problem develops gradually, several components contribute to it, or essential services depend on continued operation. The distinction is between stopping a component and stopping the process in which it participates: ability to shut down a component to shut down a component (103) âÌžability to stop the relevant process. to stop the relevant process. A shutdown mechanism is consequently most useful when it forms part of a wider intervention plan. That plan must identify what needs to be stopped, who can authorize the action, how other components will respond, and which functions must continue through independent alternatives. 13.2 Distributed Systems Many digital services operate across multiple machines, data centers, organizations, and jurisdictions. AI deployments can share this structure: several instances may operate simultaneously, and their outputs may feed into other automated processes. Stopping one instance will then stop the wider activity only if the remaining components depend on it or can also be interrupted. Systems engineering provides a useful perspective here. Harm can arise from interactions among components even when each component operates as designed (Leveson, 2012). For AI, the relevant object of control may therefore be a network of models, software, infrastructure, institutions, and human operators. Distribution does not make such a network inherently uncontrollable. Isolation boundaries, coordinated authorization, monitoring, and shutdown procedures can preserve effective intervention capacity. Their adequacy must nevertheless be assessed against the process as a whole, including any activity that can continue after individual components are disabled. 13.3 Human Dependence Dependence creates another obstacle. If important functions in finance, communications, logistics, medicine, or critical infrastructure rely on an AI-mediated process, disabling it may cause substantial disruption. The technical ability to stop a system can therefore coexist with strong practical pressures to keep it running. An organization may retain legal authority and physical access while finding intervention increasingly difficult because it lacks a workable alternative. This can weaken effective control without removing formal authority. The outcome is not inevitable: independent fallback systems, retained human expertise, and tested transition procedures can reduce the cost of interruption. Dependence also matters for gradual human disempowerment. As consequential functions are delegated, the ability to revise those arrangements may decline if people lose the knowledge, resources, or organizational capacity needed to operate without them. The relevant question is whether delegation remains reversible in practice, rather than whether humans retain a nominal right to withdraw it. 13.4 The Detection Problem Intervention also depends on knowing when to act. Testing may miss behavior that appears under different deployment conditions, while the volume and speed of AI-mediated decisions can limit direct human inspection. Evaluations of dangerous capabilities and behavior help address this problem, although they cannot guarantee that every consequential failure will be anticipated (Shevlane et al., 2023). One possible intervention sequence is detectâunderstandâdecideâintervene.detect . (104) In practice, these activities may overlap, and intervention need not wait for a complete explanation. Automated limits or precautionary shutdown procedures can act on warning signals before humans understand the underlying problem. Some safeguards also prevent hazardous actions without requiring a specific danger to be detected first. For interventions that do depend on recognition, however, delayed or unreliable detection can prevent an otherwise effective mechanism from being used. Timing changes what intervention can achieve. Once harmful outputs have been acted upon or effects have propagated into other systems, stopping the source may prevent further damage while leaving earlier consequences intact. This connects intervention to the preservation and recovery framework developed elsewhere in the paper. Interrupting a hazardous process and preserving or restoring a non-existential future are related tasks, but neither guarantees the other. 13.5 Control as a Systems Property These difficulties make control a property of the wider sociotechnical system. Levesonâs systems-theoretic approach similarly emphasizes that accidents can arise from inadequate control of interactions, rather than from the failure of a single component (Leveson, 2012). Applied to AI, practical intervention capacity depends on whether consequential activity is observable, its sources can be identified, responsible actors have effective authority, and relevant components can be interrupted while essential functions continue. To distinguish this intervention capacity from the broader CcontrolC_control introduced in Section 5, write int=fintâ(Lloc,Oobs,Aauth,Iiso,Ddep,Bred),C_int=f_int (L_loc,O_obs,A_auth,I_iso,D_dep,B_red ), (105) with the components defined as Lloc L_loc =ability to localize the relevant process, =ability to localize the relevant process, (106) Oobs O_obs =observability of consequential behavior, =observability of consequential behavior, (107) Aauth A_auth =availability of effective intervention authority, =availability of effective intervention authority, (108) Iiso I_iso =ability to isolate or interrupt relevant components, =ability to isolate or interrupt relevant components, (109) Ddep D_dep =dependence of important functions on the system, =dependence of important functions on the system, (110) Bred B_red =availability of independent fallback capacity. =availability of independent fallback capacity. (111) This expression organizes the factors that influence intervention; it supplies neither a calibrated index nor a specified functional relationship. Their effects depend on the deployment. Redundancy, for example, can preserve essential services during shutdown, but additional instances of the hazardous process can make interruption harder. The relevant fallback capacity must therefore remain sufficiently independent of the failure being addressed. The subscripts distinguish these factors from other symbols in the paper, including organizational vulnerability O, deployment pressure D, and the probabilistic irreversibility proxy I=1âQI=1-Q. Here, intC_int represents one aspect of the broader control architecture, rather than a measure of preservation or recovery capacity. A physical shutdown mechanism addresses part of this structure. It can interrupt a known component while leaving unresolved whether the danger has been detected, other instances remain active, intervention authority is available, or essential functions can continue safely. The central requirement is therefore broader: Effective control requires humans and institutions to retain the practical ability to detect, localize, and interrupt dangerous AI-mediated processes, with safeguards and fallback arrangements that remain usable when intervention is needed. Intervention can prevent a catastrophe or limit an ongoing one. Preservation and recovery concern whether biological survival, civilizational capacity, and meaningful human agency remain secure or can be restored if severe disruption occurs. Avoiding the aggregate existential outcome defined earlier requires avoiding each of its constituent endpoints; preserving one dimension alone does not establish that the others remain recoverable. The off-switch argument thus identifies a useful safeguard, but leaves important questions about its reach and effectiveness unanswered. Advanced AI need not become impossible to shut down for control to become difficult. As AI-mediated activity spreads across infrastructure and institutions, effective intervention may require coordinated action, independent alternatives, and a clear understanding of which consequences a shutdown can still prevent. 14 Why Intelligence Alone Is Not Enough Superhuman cognitive performance does not, by itself, establish existential danger. A systemâs consequences depend on how its capabilities are used, how its outputs reach the world, and whether safeguards remain effective. A benchmark score can reveal an ability to solve particular problems, but it says little on its own about these wider relationships. A model that produces text has fewer direct means of action than one connected to tools and infrastructure. Its influence may nevertheless be substantial if people or software act on its outputs. Conversely, a system with relatively limited cognitive capabilities may become highly consequential through widespread deployment or a critical operational role. Understanding risk therefore requires examining how cognitive capability becomes practical influence. 14.1 Intelligence versus Agency Cognitive capability refers here to demonstrated abilities on specified tasks, including prediction, reasoning, planning, and problem solving. Agency concerns how a deployed system selects and executes actions in pursuit of a task. Neither concept requires consciousness or human-like intention. The same model can present different risks when used to answer individual prompts and when embedded in an ongoing process of planning, action, and feedback. In the latter configuration, surrounding software may provide tools, memory, and opportunities to continue operating with less immediate supervision. Autonomy is consequently a property of the deployment as a whole, rather than a fixed attribute of the model. Capability and autonomy can reinforce one another. Better planning may improve the effectiveness of autonomous action, while greater autonomy may allow existing capabilities to be exercised more widely. Their effect on risk still depends on the task, operating conditions, and safeguards. Neither dimension provides a general measure of danger. Harm can also arise without autonomous action or malicious intent. A well-intentioned user may follow an erroneous recommendation, or software may implement an output without adequate verification. The route from output to consequence then runs through other actors. It remains part of the safety assessment even when the model itself has no external tools. 14.2 Intelligence versus Resources Resources shape what a deployed system can accomplish. Computation, data, money, communications, tools, infrastructure, and assistance from people or other systems may all matter, depending on the task and whether they are practically usable. Access to a database can supply information absent from training, while an external service can execute an operation that the model could otherwise only describe. Such additions may change the performance of the deployed system without changing the modelâs parameters. Evaluations should therefore account for the tools and support available during use, alongside the capabilities demonstrated by the model in isolation. Additional resources do not necessarily increase harmful influence. They may be irrelevant to a particular task, unreliable, or effectively restricted. They can also improve verification, monitoring, and recovery. Their safety implications depend on which activities they enable and how those activities are governed. Access can change over time. Operators may expand a systemâs role, and some deployments may allow it to obtain further assistance or resources. Effective access limits therefore require continuing enforcement. Restrictions present at the beginning of deployment provide incomplete evidence about the boundaries that will hold later. 14.3 Intelligence versus Authority Delegated authority determines which decisions a system may make on behalf of a person or organization. Recommending an action and executing it are different responsibilities. Permission to act directly can change the speed, scope, and reversibility of consequences while removing an opportunity for prior review. Formal authority does not fully describe practical influence. A system may have technical access that exceeds its authorization, or it may influence a person who holds the relevant decision rights. Delegation is one route to consequential action, alongside unauthorized operations and indirect influence through others. Assessment must therefore consider both what a system is permitted to do and what it can actually cause. Written restrictions are effective only when organizational procedures and technical controls enforce them. Approval requirements and checks before irreversible actions can reduce particular risks, but reviewers need adequate information, competence, time, and a practical ability to refuse or stop an action. Routine approval without meaningful scrutiny offers limited protection. 14.4 Intelligence versus Coordination Large effects often emerge through interactions among organizations, markets, infrastructure, people, and software. AI can influence these networks through communication, tool use, repeated deployment, or integration into institutional decisions. Multiple systems may also coordinate their actions, allowing their combined influence to exceed that of any individual deployment. Large-scale failure does not always require deliberate coordination. Systems that share models, data, instructions, or operating conditions may produce correlated errors. Institutional and technical dependencies can then transmit those errors without any AI directing the overall process. Nor must AI remain active throughout a cascade. In a sufficiently vulnerable system, a single consequential action could initiate failures that continue through existing dependencies. Establishing such a possibility requires a specific account of the initiating action, the propagation mechanism, and the barriers that would have to fail. Agency, resources, authority, and coordination thus describe possible contributors to risk, rather than conditions that every pathway must satisfy. Their roles vary across deployments. Coordination, for example, can amplify harmful activity or strengthen oversight and collective response. Its consequences depend on the activities and institutions involved. 14.5 Implication Benchmark performance contributes evidence about relevant capabilities, but it is not a direct measure of existential risk. An assessment must connect those capabilities to a deployment, identify a plausible failure or misuse mechanism, and examine the resulting consequences. High cognitive performance alone establishes neither a catastrophic pathway nor the continuing adequacy of safeguards. Broad superiority across cognitive domains is also unnecessary for a system to become highly consequential. Specialized abilities, extensive deployment, and influence over important decisions can create serious risks. Their magnitude must still be assessed in the context of the particular application. For conceptual purposes, the factors shaping effective power can be organized as effective=FâĄ(,a,r,d,k,η),P_effective=F(c,a,r,d,k;η), (112) where c represents relevant cognitive capabilities, a autonomy, r resource access, d delegated authority, k coordination and coupling, and η the surrounding environment, institutions, and controls. This expression identifies possible dependencies without specifying a calibrated scalar or a functional relationship. It neither requires every factor to be present nor assumes that increasing one necessarily increases risk. Effective power must also be distinguished from harmful behavior. An explanation of harm must show how influence is exercised and why safeguards fail. An explanation of existential harm must additionally connect those consequences to a permanent endpoint defined in Section 2. An influential output or important operational role does not establish that connection by itself. Superhuman cognitive performance alone does not establish existential danger. Risk depends on how capabilities become consequential action or influence, which harmful processes they enable, and whether the safeguards governing those processes remain effective. Safety can therefore improve through changes to deployment as well as changes to the model. Access restrictions, limits on execution, effective review, and measures that contain failures may reduce identified risks while preserving useful cognitive performance. Their adequacy must be tested against the relevant pathway. Greater capability may also strengthen safeguards, making the relationship between intelligence and safety dependent on how that capability is developed and used. 15 Objections and Counterarguments A serious analysis of existential AI risk must address objections as carefully as it develops possible failure scenarios. Much remains uncertain, and detailed accounts of future catastrophe often extend beyond available evidence. The question is whether identifiable mechanisms of misuse, control failure, systemic disruption, and irreversible harm warrant further investigation. Answering it requires distinguishing evidence for individual mechanisms from evidence for a complete existential pathway. 15.1 âAI Is Just Softwareâ The observation that AI is implemented in software tells us little about the consequences of its deployment. Software runs on physical infrastructure and influences communications, financial markets, transportation, industrial operations, and essential services. Its effects depend on the decisions it informs and the actions it enables. A model without external tools has fewer opportunities for direct action, but people may still use its outputs to make consequential decisions. Connecting the same model to operational systems changes those opportunities and may change its risk profile. Both automated execution and influence through human operators must therefore be considered within the larger deployment, including its permissions, technical connections, and institutional safeguards (Leveson, 2012; Shevlane et al., 2023). 15.2 âAI Has No Bodyâ Some AI deployments have no dedicated robotic body, while others are incorporated into machines that sense and act in the physical world (Bommasani et al., 2021). Embodiment is one channel of influence, alongside software interfaces, communication networks, organizations, and human operators. Research on language-model interfaces illustrates how these channels can work. ReAct combines generated reasoning traces with task-specific actions in the environments studied, while Toolformer trains models to select API calls and use their results in subsequent token prediction (Yao et al., 2023; Schick et al., 2023). These findings demonstrate particular forms of tool use under tested conditions. They do not establish unrestricted autonomy or reliable operation of critical infrastructure. The physical consequences of a deployment depend on which actions its interfaces permit and how those actions are constrained. A robotic body can expand the available options, but consequential influence is possible without one. 15.3 âHumans Will Remain in Controlâ Humans build AI systems, provide their infrastructure, establish legal rules, and authorize deployment. These responsibilities give people important means of control. Whether those means remain effective depends on the practical arrangements surrounding their use. When safety relies on human intervention, responsible actors need enough information to recognize a problem, enough time to respond, and a workable means of stopping or constraining the activity. Complexity, distribution, and dependence can weaken each of these conditions. A person may approve a decision without being able to evaluate it adequately, while an organization may retain shutdown authority but lack an acceptable way to continue essential services afterward. Effective control need not require manual approval of every action. Bounded permissions, automated monitoring, and tested response procedures can reduce the burden on human supervisors. Their value depends on whether they preserve the ability to constrain consequential behavior when conditions change. Control is therefore a property of the wider sociotechnical arrangement, rather than something established by placing a human in the decision chain (Leveson, 2012). 15.4 âAI Extinction Scenarios Are Science Fictionâ This objection identifies a real weakness in some discussions. Stories about future architectures, motives, and strategies can become more detailed than the evidence warrants. Exploratory scenarios remain useful when their assumptions are explicit, but they should not be presented as established forecasts. Several mechanisms invoked in those scenarios have independent research foundations. Accident analysis and studies of infrastructure interdependence examine how local failures can spread and how interactions can undermine safety (Perrow, 1999; Rinaldi et al., 2001; Buldyrev et al., 2010). Their findings support investigation of particular propagation mechanisms. They do not establish that an AI-mediated cascade would cause extinction or permanently prevent civilizational recovery. A catastrophic pathway must therefore be examined through separate questions. What could initiate the failure? Which dependencies could transmit it? What barriers would limit its spread? How could the resulting disruption affect surviving populations and institutions? What would make preservation or restoration permanently unavailable? Each step requires its own evidence. The claim that a failure can propagate is substantially weaker than the claim that humanity could never recover from its consequences. Prolonged disruption alone does not establish permanence. Keeping these propositions distinct allows research to assess the pathway without treating the whole narrative as a single prediction. 15.5 âSuperintelligence Is Requiredâ Some loss-of-control scenarios depend on capabilities far beyond those of present systems. Other systemic risks concern specialized abilities, deployment scale, operational access, and institutional dependence. A system need not outperform humans across all cognitive domains to occupy a consequential role. Shared models and common dependencies can also produce correlated errors across deployments. Whether these errors remain local or become systemic depends on how the deployments interact and which safeguards continue to function. Foundation models used across many applications, including tool-based and agentic workflows, make these deployment relationships relevant to safety assessment (Bommasani et al., 2021; Bengio et al., 2026). This supports studying effective influence alongside cognitive performance. It does not demonstrate that current or moderately capable systems can produce an existential outcome. That stronger claim requires a credible pathway to a specified permanent endpoint, accounting for safeguards, surviving capacity, and opportunities for restoration. Depending on the endpoint, the pathway may involve either catastrophic disruption or a gradual and ultimately irreversible loss of meaningful human agency. 15.6 âInstitutions Will Adaptâ Historical experience provides a substantial reason for caution about catastrophic predictions. Aviation, nuclear energy, medicine, finance, and industrial engineering have developed standards, regulatory institutions, professional practices, and procedures for managing hazards. AI development can likewise be shaped by institutional learning and deliberate safety measures. The uncertainty concerns how effective that adaptation will be and when it will occur. Institutions can anticipate emerging risks and restrict dangerous deployments before harm becomes widespread. They can also struggle with incomplete information, limited resources, coordination difficulties, and incentives that favor rapid deployment (Bengio et al., 2026). Historical adaptation establishes that improvement is possible, but does not determine whether safeguards will be adequate in a particular deployment. The capabilityâcontrol framework expresses one aspect of this problem. Within a specified context and using fixed, commensurate scales, the conceptual gap can widen when consequential capability grows faster than relevant control capacity. Institutions may improve while still falling behind changing demands. Conversely, matching the pace of capability growth does not ensure adequate protection if the starting safeguards are insufficient. The framework helps organize these questions without providing a calibrated measure of institutional effectiveness. Taken together, the objections support a qualified conclusion: Objections to existential AI risk expose uncertainty, challenge unsupported assumptions, and provide reasons to reject inevitability. Assessing the remaining risk still requires examining particular mechanisms of harm, the safeguards that could interrupt them, and the conditions under which their consequences could become permanent. The paper does not predict that AI will eliminate humanity. It proposes a framework for examining possible routes to existential harm and identifying where those routes depend on evidence that remains incomplete. Systematic study is justified by plausible mechanisms and consequential unanswered questions; it does not require treating speculative endpoints as established outcomes. 16 What Would Have to Go Wrong? Advanced AI does not inevitably lead to human extinction or another existential outcome. A civilization-scale catastrophe requires a mechanism capable of producing sufficiently large effects and an inadequate response from the safeguards intended to prevent or contain them. An existential outcome requires something further: at least one of the permanent endpoints defined in Section 2, under the scope assumptions adopted there. Several barriers can separate capability from irreversible harm. A poorly aligned system may remain constrained by its permissions. A dangerous capability may never be deployed. A serious failure may be contained, and even widespread catastrophe may leave enough people, knowledge, infrastructure, and institutional capacity to preserve a non-existential future. Which barriers matter depends on the pathway. Autonomous misalignment, harmful human use, organizational failure, and competitive deployment involve different mechanisms and need not follow the same sequence. The task is to identify how a particular process could cause harm, which safeguards could interrupt it, and why those safeguards might prove insufficient. This section concentrates on acute or propagating catastrophic pathways; gradual permanent disempowerment may have a different structure and need not involve a discrete catastrophe. 16.1 An Illustrative Barrier-Failure Sequence One possible progression within an autonomous-misalignment pathway is misalignmentâ¶inadequate evaluationâ¶inadequate containmentâ¶governance or intervention failureâ¶civilization-scale catastropheâ¶failure of preservation or recovery. aligned &misalignment\\ & evaluation\\ & containment\\ & or intervention failure\\ & -scale catastrophe\\ & of preservation or recovery. aligned (113) The sequence illustrates protective barriers rather than establishing necessary stages. Evaluation may correctly identify a danger while governance fails to prevent deployment. Containment and intervention may operate together, and some pathways may bypass individual barriers. Human misuse, for example, can begin with a system that follows its instructions accurately. The arrows therefore represent a possible progression, not a claim that one failure inevitably causes the next. A sufficiently effective barrier can interrupt the pathway, provided it acts on the relevant mechanism before consequences exceed its reach. 16.2 Failure of Technical Alignment An autonomous-misalignment pathway may begin when a systemâs behavior fails to respect the objectives or constraints its designers intended. Hostility and consciousness are unnecessary: pursuing a specified objective can produce outcomes that its designers would reject if the specification omits important considerations (Amodei et al., 2016; Ngo et al., 2022). Misalignment alone does not establish a civilization-scale danger. Limited permissions, restricted autonomy, and effective supervision may contain its consequences. The concern becomes more substantial when the behavior can affect important systems and other safeguards are inadequate. Technical misalignment is also unnecessary for some catastrophic pathways. A harmful user may obtain precisely the assistance requested, while organizational mistakes or competitive pressures may make an otherwise functional system unsafe to deploy. These cases require attention to human objectives, institutional incentives, and operating conditions alongside model behavior. 16.3 Failure of Evaluation Evaluation helps developers and responsible institutions understand what a system can do, how reliably it behaves, and how its capabilities change when combined with tools, memory, or greater autonomy. It can fail when tests omit relevant abilities, cover too narrow a range of conditions, or assess components separately when consequential behavior emerges from their combination. Dangerous-capability evaluations address part of this problem (Shevlane et al., 2023; Bengio et al., 2026). Their practical value depends on whether findings inform deployment decisions. A capability may remain undetected, its significance may be misunderstood, or accurate results may be overridden by other priorities. Evaluation also matters beyond autonomous misalignment. A deployment assessed under restricted access may present different risks when connected to more powerful tools, offered to different users, or incorporated into another institutional setting. The relevant assessment must account for these changes. Evaluation provides evidence for decisions and safeguards; it cannot ensure safety if subsequent actions disregard that evidence. 16.4 Failure of Containment Large external effects require a channel through which outputs or actions influence the world. Containment concerns the boundaries around those channels, including permissions, network access, tools, operational autonomy, and the use of outputs by people or other systems. These boundaries may be inadequate without any technical âescape.â Operators can grant broad access because it makes a deployment more useful or convenient, while users can act on information produced through a text-only interface. A containment assessment must therefore consider both direct execution and indirect influence. The same principle applies to misuse. Harmful users need some means of converting assistance into consequential activity, although that means may lie outside the AI deployment itself. Restrictions are effective when they address the relevant route to harm and remain enforceable as the systemâs role changes. 16.5 Failure of Governance Governance shapes which systems are developed and deployed, what restrictions apply, and how identified hazards are handled. Its effectiveness depends on the decisions of companies, governments, regulators, research institutions, professional communities, and infrastructure operators. Delay, fragmented responsibility, incomplete information, weak enforcement, and competitive pressure can undermine those decisions. An organization may recognize a risk yet deploy because it expects competitors to proceed. Safeguards adopted by one actor may also leave important exposure elsewhere, creating a need for compatible measures among relevant institutions (Han et al., 2019; Bengio et al., 2026). Governance failure need not mean institutional collapse. Oversight and authorization can remain operational while proving inadequate for the deployments they regulate. Such weaknesses may affect several pathways at once, increasing exposure to misuse, autonomous failures, organizational mistakes, and unsafe deployment practices. 16.6 Failure of Intervention Once hazardous activity begins, intervention may still prevent or limit harm. For responses that depend on recognizing a problem, warning signals must reach actors able to act, and the response must be effective within the available time. Automated limits and precautionary procedures can also interrupt activity before its causes are fully understood. As discussed in Section 13, practical control depends on more than formal authority. Distributed activity, incomplete information, fragmented responsibility, and dependence on essential services can make intervention difficult. Stopping one component may leave others operating, while delayed action may prevent additional harm without reversing effects that have already propagated. Evaluation, containment, governance, monitoring, and intervention contribute to the preventive control architecture represented schematically by W in Section 11. They can prevent a catastrophe or limit an unfolding one. Their effectiveness must be assessed against the timing and structure of the particular hazardous process. 16.7 Failure of Preservation or Recovery A civilization-scale catastrophe is not automatically existential. Its longer-term consequences depend on surviving populations, knowledge, infrastructure, institutions, and the capacity to coordinate preservation or reconstruction. For some endpoints, preservation matters more directly than rebuilding. Biological human extinction has not occurred at an assessment time if living humans remain, but their presence alone does not establish continued survival. Reproductive viability and the conditions supporting it matter for that prospect. Similarly, avoiding permanent disempowerment requires meaningful human agency to remain intact or to be restored under the criteria adopted in Section 2. Let C denote the specified AI-mediated civilization-scale catastrophe, and let E=EbioâȘEcivâȘEagencyE=E_bioâȘ E_civâȘ E_agency denote the union of the three existential endpoint events. With all events defined over the same assessed trajectories, context, and temporal criteria, and assuming PâĄ()>0P(C)>0, define Qcat=PâĄ(EcâŁ).Q_cat=P(E^c ). (114) This is the probability of avoiding all included existential endpoints, conditional on the catastrophe. It gives an explicit catastrophe conditioning to the preservation or recovery quantity Q used earlier. Biological survival alone is insufficient if permanent civilizational loss or disempowerment nevertheless occurs. The corresponding endpoint-specific probabilities are Qbio,cat Q_bio,cat =PâĄ(EbiocâŁ), =P(E_bio^c ), (115) Qciv,cat Q_civ,cat =PâĄ(EcivcâŁ), =P(E_civ^c ), Qagency,cat Q_agency,cat =PâĄ(EagencycâŁ). =P(E_agency^c ). Unlike the QjQ_j quantities in Section 2, which condition on a hazardous process F, these quantities condition specifically on C. Transferring between them requires either setting F=F=C under consistent definitions or specifying the relationship in an explicit model. Moreover, QcatQ_cat concerns the joint event Ebiocâ©Ecivcâ©EagencycE_bio^câ© E_civ^câ© E_agency^c. It cannot generally be calculated by multiplying the three endpoint-specific probabilities, because conditional independence has not been established. Within a consistently specified assessment, Qcatâ0Q_catâ 0 (116) means that, conditional on the catastrophe, avoidance of every included endpoint becomes increasingly unlikely. The corresponding version of the earlier proxy I=1âQI=1-Q is IcatâĄ1âQcat=PâĄ(EâŁ),I_cat⥠1-Q_cat=P(E ), (117) so that Icatâ1.I_catâ 1. (118) These are statements about conditional probabilities, not measurements of physical irreversibility or evidence that an endpoint has occurred. Establishing a realized endpoint requires examining the outcome and the mechanisms governing survival or restoration. Likewise, non-recovery over a finite observation period does not by itself establish permanent impossibility. If PâĄ()=0P(C)=0, the catastrophe branch contributes no probability, and these conditional quantities need not be defined. Total existential risk includes outcomes outside this branch. For PâĄ()>0P(C)>0, the probability identity is PâĄ(E)=PâĄ()â(1âQcat)+PâĄ(Eâ©c)P(E)=P(C)(1-Q_cat)+P(E ^c). The first term describes trajectories containing both the specified catastrophe and an existential endpoint; showing that the catastrophe causes the endpoint requires a further causal argument. The second includes existential outcomes without that catastrophe, potentially including gradual permanent disempowerment. A low QcatQ_cat therefore does not determine total risk on its own. The basic distinction remains advanced AIâÌžexistential catastrophe.advanced AI catastrophe. (119) For the catastrophic pathways considered here, a possible progression is AI-mediated process capable of large effects+failure of relevant pre-catastrophe safeguardsâ¶civilization-scale catastrophe+permanent failure of preservation or restorationâ¶existential outcome. aligned &AI-mediated process capable of large effects\\ & +failure of relevant pre-catastrophe safeguards\\ & -scale catastrophe\\ & +permanent failure of preservation or restoration\\ & outcome. aligned (120) Permanent failure here concerns at least one specified existential endpoint, rather than the loss of an individual institution or technology. The arrows summarize a possible causal progression whose mechanisms still need to be demonstrated. Existential loss may occur during the catastrophe, rather than in a separate later stage. The relevant barriers vary across pathways. Autonomous misalignment places particular demands on alignment, evaluation, containment, and intervention. Misuse may involve accurate instruction-following, while organizational and competitive pathways may depend more heavily on authorization, incentives, coordination, and deployment practices. An existential conclusion requires more than identifying a dangerous capability or a severe failure. For a catastrophic pathway, the analysis must explain how harm reaches civilization scale, why relevant safeguards fail, and how the consequences produce at least one permanent existential endpoint. This framework distinguishes prevention from preservation and recovery while recognizing their practical connections. Preparations made before a catastrophe can protect what survives and support later restoration. Intervention during disruption may also limit damage to that capacity. The layers should not be assumed statistically independent. Shared infrastructure, institutions, information, models, or technical assumptions can cause several safeguards to fail together. Resources supporting prevention may also be needed for preservation and recovery. Assessing these dependencies is therefore part of understanding what would have to go wrong, and where protective measures could still interrupt the pathway. 17 Preventing AI-Driven Existential Catastrophe Because severe AI-mediated harm can arise through different pathways, prevention requires safeguards suited to the mechanisms involved. A technical solution, regulation, or institutional procedure may address an important vulnerability while leaving others unresolved. A credible safety architecture must therefore combine protections around the system, its deployment, and the institutions responsible for its use. The aim is to reduce the likelihood that failures spread and to preserve the conditions for a non-existential future if severe disruption occurs. Evaluation, restrictions on autonomy and access, authorization, monitoring, governance, and intervention can prevent hazardous activity or contain its consequences. Preservation and recovery measures protect what survives and support restoration. Gradual permanent disempowerment requires additional attention to decision rights, institutional dependence, and the practical reversibility of delegation. These functions overlap in practice, but their different purposes matter. Preventing a dangerous action, interrupting an unfolding cascade, and maintaining the capacity to recover each require their own evidence of effectiveness. 17.1 Capability Evaluation Conventional benchmarks reveal performance on specified tasks, but may leave important safety questions unanswered. Evaluation should also examine capabilities that change the consequences of deployment, such as sustained planning, tool use, operation with reduced supervision, and performance under unfamiliar conditions. The assessment must consider how these abilities interact with the tools, permissions, and support available in use. An improvement that appears modest on a general benchmark may become consequential in a particular operational setting. Evaluations designed for extreme AI risks address this connection between capability and deployment (Shevlane et al., 2023; Bengio et al., 2026). Assessment should continue as systems, users, and operating environments change. A deployment may acquire new tools or responsibilities after its initial evaluation, making earlier findings incomplete. Risk-management frameworks accordingly treat measurement and reassessment as continuing responsibilities throughout the lifecycle (Tabassi, 2023). The value of evaluation ultimately depends on whether its findings influence deployment decisions and the safeguards surrounding them. 17.2 Limiting Autonomy A capable system can analyze problems and recommend actions without being authorized to execute every recommendation independently. Separating reasoning from execution provides an opportunity to retain useful performance while constraining particular routes to harm: high capabilityâÌžhigh autonomy.high capability autonomy. (121) Operational authority should be justified by the application and its safeguards, rather than increased automatically as cognitive performance improves. Consequential decisions may require human or institutional authorization, bounded execution, or additional verification. Such restrictions address risks arising from unsupervised action. Their effectiveness depends on whether they are enforced and whether reviewers can assess the proposed actions. They do not resolve every pathway: harmful users can exploit a system with little autonomy, and well-intentioned users can act on unreliable advice. 17.3 Limiting Access Access determines which resources and operations a deployed system can use. Permissions can be tailored to the task, allowing information to be inspected without modified records, or permitting recommendations while reserving execution for separately authorized actors. For actions with substantial or difficult-to-reverse consequences, narrower permissions and checks before execution may reduce exposure. The relevant boundaries must cover indirect channels as well: people and other software can convert outputs into actions even when the model has no direct operational access. The objective is to grant access on the basis of demonstrated need and assessed consequences. Narrow permissions alone do not establish safety, while broader access may be justified where effective controls are available. Both judgments require evidence about the deployment and the pathways those permissions enable. 17.4 Layered Authorization A single approval step may leave a consequential action dependent on one fallible judgment. Additional checks can improve protection when they examine relevant issues effectively and have sufficiently different failure modes. Depending on the application, these checks may involve human review, technical verification, organizational approval, or separate systems assessing different aspects of the decision. Their number alone provides little assurance. Checks that share a model, data source, infrastructure, or mistaken assumption may fail together. Systems safety therefore requires attention to the interactions and shared vulnerabilities of protective measures (Leveson, 2012). A useful design principle is greater potential consequence potential consequence (122) â¶stronger authorization with fewer shared failure modes. authorization with fewer shared failure modes. This is a qualitative principle whose application must also account for urgency and the harm that delay could cause. Reviewers need sufficient information and a practical ability to refuse or interrupt the action. Strong benchmark performance does not, by itself, justify allowing an output to trigger a high-consequence operation. 17.5 Monitoring and Auditing Deployment exposes systems to conditions that testing cannot fully reproduce. Monitoring can reveal unexpected behavior, changes in performance, misuse, and failures of safeguards. Auditing complements this work by making decisions and incidents available for reconstruction and examination, including by independent parties where appropriate. Design intentions are insufficient evidence of compliance: intended complianceâÌžactual compliance.intended compliance compliance. (123) A safety assessment should draw on observed behavior alongside pre-deployment testing and analysis. Continuing measurement, review, and reassessment are consistent with lifecycle risk management (Tabassi, 2023). Monitoring must connect warning signals to actors or mechanisms capable of responding. Detection without usable intervention authority may leave a hazard unaddressed. At the same time, enforced permissions and automatic containment can prevent some harmful actions without requiring every dangerous behavior to be identified individually. 17.6 Preservation and Recovery Engineering Preservation and recovery capacity should be developed before a crisis. Even well-designed safeguards can fail, and unexpected combinations of failures may affect several protections at once. Essential functions may require independent backups, alternative communications, replacement capacity, and procedures for operating under degraded conditions. Organizations also need workable ways to continue when AI-dependent systems become unavailable or unsafe. Resilience engineering addresses this broader capacity to anticipate disruption, withstand its effects, recover, and adapt (Ross et al., 2021). Let C denote the specified AI-mediated civilization-scale catastrophe, with pcat=PâĄ()p_cat=P(C), and let E=EbioâȘEcivâȘEagencyE=E_bioâȘ E_civâȘ E_agency denote the existential endpoint events defined in Section 2. Using consistent trajectory, deployment, and temporal criteria, and assuming PâĄ()>0P(C)>0, define Qcat Q_cat =PâĄ(EcâŁ) =P(E^c ) (124) =PâĄ(Ebiocâ©Ecivcâ©EagencycâŁ). =P\! (E_bio^câ© E_civ^câ© E_agency^c ). For a fixed catastrophe assessment, prevention and resilience pursue complementary objectives: reduce âpcatand improve endpoint avoidance given â. aligned &reduce p_cat\\ &and improve endpoint avoidance given C. aligned (125) The second objective requires continued biological human existence, sufficient civilizational capacity, and meaningful human agency to be preserved or, where applicable, restored. Biological extinction must be prevented; rebuilding institutions cannot reverse it under the definition adopted here. A surviving population at one time also does not establish its continued survival. Avoidance of the aggregate existential event requires avoiding every included endpoint, without assuming that their probabilities are independent. These objectives must be assessed within the complete probability accounting, PâĄ(E)=pcatâ(1âQcat)+PâĄ(Eâ©c)P(E)=p_cat(1-Q_cat)+P(E ^c). The final term includes existential outcomes without the specified catastrophe, potentially including gradual permanent disempowerment. This identity describes event probabilities; attributing an endpoint causally to the catastrophe requires further analysis. Interventions may also change which catastrophes remain possible. If a measure preferentially prevents more recoverable events, QcatQ_cat can decrease among the remaining catastrophes even while total existential risk falls. Conditional recovery probabilities should therefore inform a complete assessment rather than serve as an isolated measure of success. If the catastrophe has zero probability under the assessment, its branch contributes nothing and the conditional quantity need not be defined. Evidence from recovery exercises and local incidents can help assess particular mechanisms, but extrapolation to civilization-scale disruption requires assumptions about surviving resources, shared dependencies, and the feasibility of restoration. Recovery by a specified date must also be distinguished from avoidance of a permanent endpoint. For gradual disempowerment, protective measures may need to act well before a crisis. Retained expertise, contestable decision rights, independent infrastructure, and workable alternatives can preserve the possibility of revising delegated authority. Their purpose is to keep meaningful human agency practically recoverable as dependence grows. 17.7 International Coordination AI deployments can cross national and organizational boundaries, making some hazards difficult for a single actor to manage. Competitive incentives can compound the problem when institutions favor stronger safeguards but fear losing an advantage to others that proceed more quickly. Shared evaluation practices, technical standards, incident reporting, and information exchange may help address these collective-action problems (Bengio et al., 2026). Their effectiveness depends on participation, implementation, and whether they cover the actors and activities relevant to the hazard. Coordination does not require identical regulation everywhere, but it does require attention to gaps created by cross-border development, deployment, and use. It supports several functions simultaneously, including evaluation, authorization, monitoring, incident response, and preservation or recovery. It is therefore a continuing feature of the safety architecture rather than a final step in a defensive sequence. 17.8 A Layered Safety Architecture The measures discussed above form interacting layers. One schematic representation of preventive control is evaluateâlimit autonomy and accessâauthorizeâmonitorâintervene. aligned evaluate& autonomy and access\\ & . aligned (126) Governance and coordination support these functions across stages. Where severe disruption occurs, preservation and recovery become especially important: preserve essential functionsâstabilizeârecover or restore. aligned &preserve essential functions\\ & or restore. aligned (127) The diagrams organize functions rather than prescribe universal sequences. Evaluation and monitoring repeat, intervention can limit an unfolding catastrophe, and preservation may operate throughout it. Recovery preparations must precede disruption, while the same institutions and resources may support several layers. Each layer has limitations. Evaluations can miss capabilities, access controls can fail, authorization can become superficial, and monitoring can overlook important signals. Recovery arrangements may depend on the infrastructure they are intended to replace. Additional layers improve protection only when they address relevant failure modes and avoid unnecessary shared vulnerabilities. Independence must be investigated rather than inferred from the number of checks or institutions involved. Reducing AI-mediated existential risk requires safeguards that address the relevant pathways to harm, remain usable when failures occur, and preserve the conditions for avoiding permanent existential endpoints. Layering can strengthen this architecture, but its effectiveness depends on the protections provided and the vulnerabilities they share. For gradual disempowerment, this means preserving meaningful human agency through decision rights, institutional adaptability, fallback capacity, and practical mechanisms for reversing excessive delegation. Across pathways, the objective is a sociotechnical system that can evaluate consequential behavior, constrain and authorize action, interrupt hazards, and preserve or restore the conditions for a non-existential future. 18 From Control to HumanâAI Coexistence Effective control remains central to AI safety, but direct supervision of individual actions provides an incomplete basis for long-term protection. As AI-mediated activity becomes faster, more extensive, and more deeply embedded in society, people may increasingly govern the conditions under which decisions are made rather than review every decision themselves. The broader challenge is to preserve beneficial interaction and meaningful human agency as capabilities, incentives, institutions, and authority change. Here, coexistence describes an enduring sociotechnical arrangement in which AI-mediated activity remains subject to effective correction, restriction, and replacement. The term does not require treating AI systems as persons or attributing independent interests to them. 18.1 Control Is Not the Final Objective Human supervision takes several forms. People define objectives, evaluate outputs, grant permissions, and respond to problems, while access restrictions and automated checks can constrain activity without requiring individual review. Their effectiveness depends on the deployment and the consequences of failure. As automation expands, meaningful agency may depend less on approving each action and more on retaining effective authority over objectives, operating boundaries, and intervention procedures. Delegation can remain accountable when people have the information, competence, resources, and authority needed to challenge decisions and revise or terminate unacceptable arrangements. The longer-term objective is therefore beneficial and acceptably safe interaction without continuous micromanagement. Control remains essential to that objective: it helps maintain the conditions under which delegated activity can be scrutinized, constrained, and changed. 18.2 Cooperative Stability A durable arrangement would keep AI-mediated activity compatible with specified human interests and safety requirements as conditions change. Technical safeguards and institutional rules would need to constrain harmful behavior, address conflicts, and sustain beneficial uses. Research on Cooperative AI examines cooperation involving humans and machine agents through game theory, multi-agent systems, social choice, and humanâmachine interaction (Dafoe et al., 2020). Cooperation in this context can describe patterns produced by objectives, incentives, permissions, and rules, without implying goodwill or conscious intention. It also requires qualification: coordinated activity can serve harmful purposes. The objective here is cooperation consistent with meaningful human agency and the safety criteria adopted in this paper. The relevant dependencies can be organized schematically as coop=ΚâĄ(Hinc,Acomp,Iinst,Rauth),S_coop= (H_inc,A_comp,I_inst,R_auth ), (128) where HincH_inc represents relevant human incentives and preferences, AcompA_comp the compatibility of AI-mediated behavior with specified human interests and safety criteria, IinstI_inst the institutional environment, and RauthR_auth the distribution of resources, decision rights, and authority. These categories can interact and need not exhaust the determinants of coexistence. Human interests may also conflict. Any application must specify whose interests are considered, how disagreements are handled, and which outcomes are unacceptable. The expression provides a conceptual organization of these dependencies. Neither Κ nor an empirical scale for coopS_coop is defined, so it establishes no calibrated index, causal model, or game-theoretic equilibrium. Its notation remains distinct from the capabilityâcontrol gap GâĄ(t)G(t) and the irreversibility proxy I in Section 11. Robustness would concern whether an arrangement remains within acceptable conditions under specified changes. Resilience would additionally concern its capacity to preserve or restore essential functions and meaningful human agency following disruption. Both require evidence about the arrangement and the conditions being assessed. 18.3 Stability Under Changing Conditions Evolutionary game theory offers a limited analogy. An evolutionarily stable strategy resists invasion by sufficiently rare alternatives under the assumptions of a specified game and population model (Maynard Smith and Price, 1973). The present framework specifies no payoffs, population dynamics, or invasion criterion, and evolutionary stability within a fixed game does not establish robustness when the game itself changes. The useful lesson is to examine an arrangement in the presence of alternatives and disturbances. A humanâAI system that functions only while its initial incentives, capabilities, and institutions remain unchanged may be fragile. Assessing coexistence requires considering new systems, changing economic incentives, institutional responses, and shifts in practical authority. These changes can strengthen safeguards or undermine them; their effects must be investigated. The initial question is Does cooperation meet the specified safety and agency criteriaunder present conditions? gatheredDoes cooperation meet the specified safety and agency criteria\\ under present conditions? gathered (129) The broader question is Does cooperation continue to meet those criteriaunder specified changes in the surrounding system? gatheredDoes cooperation continue to meet those criteria\\ under specified changes in the surrounding system? gathered (130) This extends the range of assessment without demanding stability under every conceivable disturbance. Successful testing over a finite range provides evidence within that range, rather than proof of permanent stability. 18.4 Testing Coexistence Evidence of beneficial interaction concerns particular systems, users, permissions, institutions, and observation periods. Applying it to substantially different arrangements requires an account of which findings are transferable and why. An assessment might examine whether greater autonomy weakens oversight, whether institutional dependence reduces replacement options, or whether changes in resources and authority make consequential decisions harder to contest. It should also examine whether improved safeguards and institutional adaptation can preserve agency under those changes. A minimally acceptable condition can be represented conceptually as coopâ(t)>min,S_coop(t)>S_min, (131) while relevant properties change over time: CAIâ(t),Chumanâ(t),Iinstâ(t),Rauthâ(t).C_AI(t),\;C_human(t),\;I_inst(t),\;R_auth(t). (132) Here, CAIâ(t)C_AI(t) and Chumanâ(t)C_human(t) refer to capabilities relevant to the assessed interaction, without assuming a universal scale for comparing humans and AI. The boundary minS_min represents acceptable conditions under the chosen criteria. The strict inequality is a notational convention, not an empirical threshold. Because no scale is specified, the inequality is not directly testable. Empirical work requires operational definitions, observable indicators, specified disturbances, observation periods, and criteria for unacceptable losses of safety or agency. Separate measures may be more informative than a single score: economic benefit, oversight, contestable authority, and recovery capacity can change in different directions. Experiments, simulations, and deployment studies could examine whether supervisors retain effective intervention options, safeguards respond to identified failures, and institutions can revise or replace consequential deployments. Such evidence would constrain particular claims about coexistence within the scope of the study. Extending those conclusions to future technologies or indefinite periods would require further justification. 18.5 From Alignment to Coevolution Alignment concerns whether AI behavior remains compatible with the objectives, constraints, or values specified in an assessment. Long-term safety also depends on changes in the environment where systems are developed and used. AI deployment can alter work, markets, decision-making, and institutional practices. Those changes can influence demand for new systems, training priorities, deployment incentives, and governance. A possible feedback sequence is human institutions institutions âAI development development (133) âsocial and economic change and economic change âinstitutional adaptation adaptation ânew AI development. AI development. The arrows describe possible influences across successive periods. They do not prescribe a necessary order or a complete causal model. Institutions can shape several stages simultaneously, and feedback may improve protection, introduce vulnerabilities, or produce mixed effects. In this limited sense, coevolution describes reciprocal institutionalâtechnological adaptation. It makes no claim about biological populations or Darwinian dynamics. Its relevance to safety lies in recognizing that objectives, deployment conditions, institutions, and human expectations can change together. Protective arrangements must be assessed within that changing environment. Evidence for safe humanâAI coexistence should address beneficial interaction under present conditions and the preservation of meaningful human agency under specified changes and disruptions. Durable safety requires practical mechanisms for correcting, constraining, or replacing arrangements that cease to meet those criteria. This perspective complements alignment and control. Evaluation, authorization, access restrictions, monitoring, and intervention address immediate hazards, while institutional adaptation and preservation or recovery capacity help sustain protection through change and disruption. The proposed objective is an enduring arrangement in which beneficial AI-mediated activity remains compatible with meaningful human agency. Whether a deployment achieves that objective requires explicit criteria and evidence about its operation, its vulnerabilities, and the practical options available when correction becomes necessary. 19 Discussion The analysis supports a cautious conclusion. Several mechanisms relevant to AI risk can already be studied empirically, but their connection to biological extinction, permanent civilizational loss, or irreversible human disempowerment remains uncertain. Evidence that a mechanism exists must be distinguished from evidence that it can produce a specified existential endpoint. AI systems can use tools, perform sequences of actions with varying supervision, and operate within complex technical and institutional settings. Research on accidents and interdependent networks also identifies mechanisms through which failures can propagate (Perrow, 1999; Buldyrev et al., 2010). Applying these findings to existential AI risk requires an account of the dependencies involved, the barriers that could limit escalation, and the conditions supporting survival or restoration. The unresolved question is whether an AI-mediated process could eliminate living humanity, permanently destroy sufficient civilizational capacity, or make meaningful human agency unrecoverable. Severe disruption alone establishes none of these outcomes. Research must therefore distinguish observed mechanisms, justified extrapolations, and speculative transitions, making clear what evidence supports each part of the argument. 19.1 What Is Established and What Remains Speculative? AI performance has improved on many measured tasks, and some deployments can undertake extended sequences of tool-assisted action. Progress remains uneven, however. Success in one setting does not establish reliability in another, and practical consequences depend on the surrounding software, permissions, resources, and users (Bengio et al., 2026). Evaluation provides incomplete coverage of those conditions. Benchmarks sample particular tasks, while deployment can introduce new incentives, tools, and combinations of components. Assessing whether a system possesses a dangerous capability also differs from assessing its propensity to use that capability harmfully (Shevlane et al., 2023). These limitations justify further testing; they do not demonstrate the existence of an unobserved capability or establish an existential pathway. The greatest uncertainty concerns the transitions to permanent outcomes. Future autonomy, delegated authority, institutional dependence, and the effectiveness of safeguards remain open questions. So do the circumstances under which severe disruption could remove the conditions for continued human existence or eventual restoration of civilizational capacity and agency. Three epistemic levels help organize the discussion: observed mechanisms mechanisms âplausible extrapolations extrapolations (134) âexistential outcomes. outcomes. The arrows indicate increasing inferential distance from direct evidence, rather than an established causal progression. Each transition requires additional assumptions and argument. Intermediate mechanisms can be investigated without experimentally observing an existential endpoint, but the resulting evidence has limits that must remain explicit. 19.2 Which Assumptions Matter Most? Some assumptions are essential to a pathway, while others affect its likelihood or severity without changing its basic structure. Research should identify these differences and ask which observations could strengthen or weaken the proposed causal links. Relevant assumptions concern autonomy, access, delegated authority, harmful human use, oversight, containment, institutional response, and preservation or recovery capacity. Their importance varies across autonomous misalignment, misuse, organizational failure, and competitive deployment. A useful comparison between assessments should therefore examine their underlying pathways before comparing their final probabilities. The framework in Section 11 separates a specified civilization-scale catastrophe from the existential endpoints that may accompany or follow it. Let C denote the catastrophe, pcat=PâĄ()p_cat=P(C), and E the union of the three endpoint events. All events must refer to the same assessed trajectories, deployment context, and temporal criteria. For pcat>0p_cat>0, pexistential=PâĄ(E)=pcatâ(1âQcat)+PâĄ(Eâ©c),p_existential=P(E)=p_cat(1-Q_cat)+P(E ^c), (135) where Qcat=PâĄ(EcâŁ).Q_cat=P(E^c ). (136) Here, QcatQ_cat concerns avoidance of every included existential endpoint. Continued human existence, sufficient civilizational capacity, and meaningful agency must all be preserved or, where applicable, restored under the adopted criteria. Biological survival alone does not establish avoidance of permanent civilizational loss or disempowerment. The first term equals PâĄ(Eâ©)P(E ): it describes trajectories containing both events. Establishing that the catastrophe contributes causally to the endpoint requires further analysis. The second term includes existential outcomes without the specified catastrophe, potentially including gradual permanent disempowerment. It can be omitted only under the explicit assumption that its probability is zero. If pcat=0p_cat=0, only the second branch remains, and QcatQ_cat need not be defined. These are probability identities, requiring no independence assumption and supplying no numerical estimate. The aggregate avoidance probability also cannot generally be obtained by multiplying endpoint-specific probabilities, since the endpoints may overlap or share causes. Assessment must account for how interventions change the distribution of possible failures. A change in QcatQ_cat can reflect a different mixture of remaining catastrophes rather than a change in total safety. Conditional recoverability should therefore be interpreted within the full accounting. Where pathway-specific models become operational and empirically constrained, sensitivity analysis could identify assumptions that strongly affect their estimates. The present framework provides neither calibrated sensitivities nor a ranking of the assumptions dominating total risk. Its contribution is to make the questions and distinctions needed for such analysis explicit. 19.3 Making Existential Risk Empirically Testable Human extinction cannot serve as a controlled experimental endpoint, and failure to recover during a finite observation period does not establish permanent loss. Empirical work must focus on mechanisms and intermediate transitions that can be observed under specified conditions. Relevant targets include the reliability and duration of autonomous operation, timely detection of hazardous behavior, the effectiveness of intervention, changes associated with tools and permissions, institutional response times, and preservation or restoration of essential functions after disruption. Instead of asking only PâĄ(AI causes human extinction)=?,P(AI causes human extinction)=?, (137) research can examine narrower quantities such as PâĄ(timely detection|specified dangerous behavior),P\! (timely detection\, |\,specified dangerous behavior ), (138) PâĄ(successful intervention|danger detected in time),P\! (successful intervention\, |\,danger detected in time ), (139) and PâĄ(successful preservationor recovery|specified AI-mediateddisruption).P\! ( gatheredsuccessful preservation\\ or recovery gathered\, |\, gatheredspecified AI-mediated\\ disruption gathered ). (140) Estimating these probabilities requires a task distribution, deployment configuration, observation period, and success criterion. Detection should count as timely only while an effective response remains possible. False alarms, missed detections, and cases where intervention cannot be attempted also matter. An association between intervention and a favorable outcome does not by itself establish that the intervention prevented harm. The measurements remain context-dependent. Findings from current systems may not transfer to more capable deployments, and recovery rates from local incidents or simulations do not directly estimate QcatQ_cat for civilization-scale disruption. They can constrain parts of a model when the assumptions needed for extrapolation are stated. Recovery within a specified period must also remain distinct from avoidance of a permanent endpoint. Pathway-specific studies can examine how agent architecture changes autonomous performance, how access alters the consequences of errors, how AI assists different users, and how organizational safeguards perform under pressure. These studies assess the interaction between deployed systems and their environments, extending beyond model performance in isolation. Measurements of evaluation, monitoring, intervention, capability, and institutional response can inform the capabilityâcontrol analysis in a specified context. They do not provide a validated numerical decomposition of the gap. Here, CcontrolC_control concerns control before or during escalation to catastrophe, while preservation and recovery are represented separately through QcatQ_cat and endpoint-specific avoidance probabilities. Shared institutions and infrastructure can support both functions, so their analytical separation implies no causal independence. Structured evaluation and continuing risk management provide a basis for revising intermediate claims as evidence accumulates (Tabassi, 2023; Shevlane et al., 2023). The final existential-risk estimate still depends on additional assumptions about future conditions and unobserved transitions. 19.4 Extinction Probability Is Not the Only Relevant Metric A single extinction probability can be informative when its time horizon, deployment assumptions, evidence, and endpoint definition are clear. Without that context, apparent precision can obscure substantial differences between assessments. Estimates may disagree because they concern different circumstances or make different assumptions about autonomy, misuse, safeguards, institutions, and recovery. Risk analysis also considers consequences. To express the familiar probabilityâloss relationship precisely, let L be a nonnegative random loss that is zero outside a specified event E and has a finite expectation. For PâĄ(E)>0P(E)>0, EâĄ[L]=PâĄ(E)âEâĄ[LâŁE].E[L]=P(E)E[L E]. (141) The second factor is the conditional mean loss. When loss is constant on E, the expression reduces to PâĄ(E)âLEP(E)L_E. If the assessment includes losses outside E, their contribution must be added. This identity does not assign an agreed value to humanityâs future. Quantifying existential loss requires explicit choices about consequences, values, and their distribution. The equation distinguishes likelihood from severity without calibrating either. The endpoints considered in this paper extend beyond immediate damage. Extinction eliminates living humanity, while permanent civilizational or agency loss removes capacities needed for humans to shape their future. Severe disruption that leaves sufficient survival and restoration possible need not meet those definitions. This concern with future potential is central to the existential-risk literature (Bostrom, 2013; Ord, 2020). It does not follow that every imaginable scenario deserves equal priority. Evidence quality, causal plausibility, competing risks, and opportunity costs remain relevant. Even where a terminal probability cannot be estimated precisely, research can ask which assumptions matter, what evidence would change the assessment, and which interventions remain effective across plausible conditions. 19.5 The Key Research Question AI capabilities can support beneficial uses while changing the consequences of error, misuse, unsafe deployment, and institutional dependence. Their significance depends on how they acquire practical influence. Increasing capability alone therefore supplies no universal boundary between useful assistance and existential danger. The research task is to examine how capability interacts with autonomy, access, persistence, human use, and institutional arrangements. It must also assess whether evaluation, governance, and intervention remain adequate, and what preservation or restoration would be possible if preventive control failed. One desirable aim is increasing social benefit alongside a non-widening capabilityâcontrol gap: ÎâB>0,dâGdâtâ€0, B>0, dGdt†0, (142) where ÎâB B denotes a favorable change under a specified assessment of social benefit, and GâĄ(t)=CAIâ(t)âCcontrolâ(t).G(t)=C_AI(t)-C_control(t). (143) The benefit criterion requires choices about consequences and their distribution; it is not assumed to share a scale with capability or control. The numerical gap requires fixed, commensurate definitions of CAIC_AI and CcontrolC_control for the assessed context. Without those definitions, it remains conceptual. Where a differentiable approximation is appropriate, a non-widening gap is equivalent within this representation to dâCcontroldâtâ„dâCAIdât. dC_controldtâ„ dC_AIdt. (144) This condition does not establish safety. A stable or narrowing gap can coexist with inadequate initial safeguards, while a widening gap may remain within an adequate margin for some period. Both the starting conditions and subsequent changes matter. A widening gap is represented by dâGdât>0, dGdt>0, (145) or, under the same assumptions, dâCAIdât>dâCcontroldât. dC_AIdt> dC_controldt. (146) Such a change may make particular pathways harder to manage. It is neither a catastrophe nor a sufficient condition for an existential outcome. Its importance depends on permitted actions, existing safeguards, and the mechanisms through which failures could spread or become permanent. Different domains may have different transitions. The same capability can have different consequences depending on connected infrastructure, available users, delegated authority, and practical intervention options. Benefits and serious risks can also coexist within a deployment. Under what conditions does increasing AI capability, combined with autonomy, access, human use, and institutional dependence, contribute to processes that exceed effective human control or permanently undermine the conditions for continued human existence, sufficient civilizational capacity, or meaningful agency? Progress does not require a precise forecast of future architectures or an agreed extinction probability today. It requires evidence about how capabilities become consequential influence, how hazards arise and propagate, and which safeguards preserve intervention and restoration options. Reporting each finding with its scope and assumptions allows broader risk arguments to be revised through smaller causal questions that can increasingly be tested. 20 Conclusions The central argument of this paper is that existential AI risk need not depend on consciousness, hostility, or a deliberate intention to harm humanity. Poorly specified objectives, harmful human use, excessive autonomy, weak oversight, institutional dependence, competitive deployment, and interconnected failures provide possible mechanisms of severe harm. Establishing an existential pathway requires a further argument: how those mechanisms could reach a permanent endpoint defined in Section 2, despite safeguards and remaining opportunities for survival or restoration. Advanced AI therefore does not imply human extinction. Cognitive capability, operational influence, and failure of preventive control are distinct from biological extinction, permanent civilizational loss, and irreversible human disempowerment. A severe catastrophe may leave recovery possible, while permanent loss of agency could develop without an acute catastrophe. Assessment must specify the endpoint, its temporal criteria, and the assumptions connecting it to humanityâs future. There is no universal sequence from capability to existential harm. Autonomous misalignment may involve interactions among capability, autonomy, access, persistence, strategic competence, and inadequate safeguards. These dimensions need not all be present: a single consequential action may initiate a process that continues without further AI involvement. Misuse can involve accurate instruction-following, while organizational and competitive pathways depend on responsibility, incentives, coordination, authorization, and deployment decisions. The paper distinguishes two broad forms of possible transition: AI-mediated pathwaysâ¶catastrophe-mediated route,gradual loss-of-agency route. aligned AI-mediated pathways& \ array[]lcatastrophe-mediated route,\\ gradual loss-of-agency route. array . aligned (147) These categories are neither exhaustive nor mutually exclusive. Institutional dependence can weaken protection against catastrophe, and catastrophic disruption can accelerate disempowerment. The arrows describe possibilities requiring causal investigation. An illustrative catastrophe-mediated pathway is AI-mediated hazardous processâ¶civilization-scale catastropheâ¶permanent loss of at least onecondition required for a non-existential future. aligned &AI-mediated hazardous process\\ & -scale catastrophe\\ & loss of at least one\\ & condition required for a non-existential future. aligned (148) The final condition concerns continued human existence, sufficient civilizational capacity, or meaningful agency. Under the adopted classification, permanent loss of any one is sufficient; all three need not disappear. The endpoint may be reached during the catastrophe rather than in a separate later stage. Permanent disempowerment may instead follow a progression such as progressive delegation and dependenceâ¶erosion of meaningful human agencyâ¶loss of realistic restoration mechanismsâ¶permanent human disempowerment. aligned &progressive delegation and dependence\\ & of meaningful human agency\\ & of realistic restoration mechanisms\\ & human disempowerment. aligned (149) Delegation alone does not establish this outcome. People may retain effective decision rights, replacement options, and institutions through which arrangements can be revised. An existential argument must explain how those options become permanently unavailable. Difficulty in restoring agency, or its absence over a finite period, does not establish permanence. Probability accounting must retain these distinctions. Let C denote the specified AI-mediated civilization-scale catastrophe, with pcat=PâĄ()p_cat=P(C), and let E=EbioâȘEcivâȘEagencyE=E_bioâȘ E_civâȘ E_agency denote the union of the assessed endpoints. With consistent trajectory, deployment, and temporal criteria, and assuming PâĄ()>0P(C)>0, pexistential,cat=PâĄ(Eâ©)=pcatâ(1âQcat),p_existential,cat=P(E )=p_cat (1-Q_cat ), (150) where Qcat Q_cat =PâĄ(EcâŁ) =P(E^c ) (151) =PâĄ(Ebiocâ©Ecivcâ©EagencycâŁ). =P\! (E_bio^câ© E_civ^câ© E_agency^c ). Avoiding the aggregate existential event requires avoiding every included endpoint. Continued human existence, sufficient civilizational capacity, and meaningful agency must all be preserved or, where applicable, restored. Biological survival alone is insufficient, and endpoint-specific avoidance probabilities cannot generally be multiplied because the events may overlap or share causes. The catastrophe-branch expression is a probability identity, supplying neither a numerical estimate nor a causal demonstration. A catastrophe-mediated interpretation requires an account of how C contributes to E. Total risk also includes the branch without the specified catastrophe: PâĄ(E)=pcatâ(1âQcat)+PâĄ(Eâ©c)P(E)=p_cat(1-Q_cat)+P(E ^c). That branch cannot be omitted without justification. If PâĄ()=0P(C)=0, it contains the entire assessed risk and QcatQ_cat need not be defined. Prevention and resilience must consequently be assessed together. Comparisons between interventions should consider the full risk, since preventing some catastrophes changes the mixture of those that remain possible. Conditional recoverability alone can therefore give a misleading impression of an interventionâs effect. No universal capability threshold separating useful assistance from existential danger has been established here. Consequences depend on how capabilities acquire influence through tools, human reliance, institutional roles, resources, and delegated authority. The capabilityâcontrol gap organizes these relationships conceptually. A numerical interpretation requires fixed, commensurate definitions, and the gap alone establishes neither inadequate control nor a reachable existential endpoint. AI safety accordingly extends beyond technical alignment. Permissions, authorization, access restrictions, monitoring, institutional design, human decisions, international coordination, and infrastructure resilience address different parts of the problem. Their effectiveness must be examined within the deployed sociotechnical system. Limits on autonomy may constrain one pathway while leaving misuse possible; approval procedures may offer little protection if reviewers lack information, time, competence, or effective authority. Preservation and recovery require advance preparation and usable capacity when disruption occurs. Their success depends on surviving populations, resources, knowledge, and institutions, rather than on the existence of a plan. They may share infrastructure with preventive control, making common vulnerabilities important across both functions. Protective layers should address different failure modes and reduce avoidable dependencies; adding checks does not itself establish adequate or independent protection. Over longer periods, safety also requires attention to humanâAI coexistence. Extensive automation may make direct supervision of every action impractical. Meaningful agency can instead rest on enforceable boundaries, contestable authority, institutional adaptation, and realistic options for correction and replacement. The framework proposed here identifies these requirements without establishing a formal equilibrium or a validated measure of long-term stability. Substantial uncertainty remains. This paper provides neither calibrated existential-risk probabilities nor an empirically validated chain to a permanent endpoint. Its contribution is to separate mechanisms, expose assumptions, and distinguish evidence about intermediate failures from claims about terminal outcomes. Research can make progress through narrower questions. Studies can examine how agent architecture and permissions affect autonomous performance, how tools and human use alter consequences, whether evaluations transfer to deployment, and whether hazards can be detected while intervention remains effective. Further work can assess shared vulnerabilities, institutional response, preservation of essential functions, restoration after specified disruptions, and the practical reversibility of delegated authority. Results should identify their task distributions, deployment configurations, comparison conditions, success criteria, and observation periods. Evidence from current systems, local incidents, and simulations can constrain parts of a pathway, but extending it to future systems or civilization-scale disruption requires explicit assumptions. Finite observations of failure or non-recovery cannot establish permanent loss on their own. A claim of existential AI danger requires a credible account of how an AI-mediated process could reach biological human extinction, irreversible loss of sufficient civilizational capacity, or permanent loss of meaningful human agency. The argument must explain why relevant safeguards and remaining opportunities for survival or restoration would fail to prevent that endpoint. The framework supports investigation without predicting inevitable catastrophe. Its practical purpose is to identify how capabilities become consequential, which failures could spread, where safeguards could interrupt them, and what would preserve a non-existential future if prevention failed. Those questions provide a basis for assessments that can be examined and revised as evidence develops. Declaration of AI Use The authors used ChatGPT to assist with language editing, LaTeXformatting and references. The authors reviewed and revised the AI-assisted material and take full responsibility for the manuscriptâs arguments, accuracy, references, and conclusions. AI-generated suggestions were not treated as independent scientific evidence. References Amodei et al. (2016) D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. ManĂ© Concrete problems in ai safety. arXiv preprint arXiv:1606.06565. External Links: 1606.06565, Link Cited by: §1.5, §1, §16.2, §9.3, §9.4, §9. Baum et al. (2019) S. D. Baum, S. Armstrong, T. Ekenstedt, O. HĂ€ggström, R. Hanson, K. Kuhlemann, M. M. Maas, J. D. Miller, M. Salmela, A. Sandberg, K. Sotala, P. Torres, A. Turchin, and R. V. Yampolskiy Long-term trajectories of human civilization. Foresight 21 (1), p. 53â83. External Links: Document, Link Cited by: §2.2, §2. Bengio et al. (2026) Y. Bengio, S. Clare, C. Prunkl, M. Murray, M. Andriushchenko, B. Bucknall, R. Bommasani, S. Casper, T. Davidson, R. Douglas, D. Duvenaud, et al. International ai safety report 2026. Technical report Technical Report DSIT 2026/001, Department for Science, Innovation and Technology. External Links: Link Cited by: §10.2, §10.3, §15.5, §15.6, §16.3, §16.5, §17.1, §17.7, §19.1, §3.2, §5.2, §5.3, §6.2, §6.4, §6. Bommasani et al. (2021) R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, E. Brynjolfsson, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258. External Links: 2108.07258, Link Cited by: §15.2, §15.5, §4.1, §4.3, §4.5, §4. Bostrom (2002) N. Bostrom Existential risks: analyzing human extinction scenarios and related hazards. Journal of Evolution and Technology 9 (1). External Links: Link Cited by: §2.3, §2. Bostrom (2013) N. Bostrom Existential risk prevention as global priority. Global Policy 4 (1), p. 15â31. External Links: Document, Link Cited by: §11.6, §19.4. Boulanin et al. (2020) V. Boulanin, L. Saalman, P. Topychkanov, F. Su, and M. PeldĂĄn Carlsson Artificial intelligence, strategic stability and nuclear risk. Technical report Stockholm International Peace Research Institute, Stockholm. External Links: Link Cited by: §6.3. Brundage et al. (2018) M. Brundage, S. Avin, J. Clark, H. Toner, P. Eckersley, B. Garfinkel, A. Dafoe, P. Scharre, T. Zeitzoff, B. Filar, et al. The malicious use of artificial intelligence: forecasting, prevention, and mitigation. Technical report External Links: 1802.07228, Link Cited by: §7.2. Buldyrev et al. (2010) S. V. Buldyrev, R. Parshani, G. Paul, H. E. Stanley, and S. Havlin Catastrophic cascade of failures in interdependent networks. Nature 464 (7291), p. 1025â1028. External Links: Document, Link Cited by: §12.2, §15.4, §19. Carlsmith (2022) J. Carlsmith Is power-seeking ai an existential risk?. arXiv preprint arXiv:2206.13353. External Links: 2206.13353, Link Cited by: §1.1, §7.1, §8.5. Casper et al. (2026) S. Casper, O. Galeev, Y. Bengio, M. Kankanhalli, et al. The 2026 singapore consensus on global ai safety research priorities. Technical report International Scientific Exchange on AI Safety. External Links: 2608.14611, Document, Link Cited by: §6.2, §6.4. Dafoe et al. (2020) A. Dafoe, E. Hughes, Y. Bachrach, T. Collins, K. R. McKee, J. Z. Leibo, K. Larson, and T. Graepel Open problems in cooperative ai. arXiv preprint arXiv:2012.08630. External Links: 2012.08630, Link Cited by: §18.2. Domingos and Han (2026) E. F. Domingos and T. A. Han Falling behind drives unsafe development in an idealised ai race experiment. arXiv preprint arXiv:2607.26034. External Links: 2607.26034, Link Cited by: §7.4. Good (1965) I. J. Good Speculations concerning the first ultraintelligent machine. In Advances in Computers, F. L. Alt and M. Rubinoff (Eds.), Vol. 6, p. 31â88. External Links: Document, Link Cited by: §10.3. Han et al. (2019) T. A. Han, L. M. Pereira, F. C. Santos, and T. Lenaerts To regulate or not: a social dynamics analysis of the race for ai supremacy. arXiv preprint arXiv:1907.12393. External Links: 1907.12393, Link Cited by: §16.5, §7.4. Hendrycks et al. (2023) D. Hendrycks, M. Mazeika, and T. Woodside An overview of catastrophic ai risks. arXiv preprint arXiv:2306.12001. External Links: 2306.12001, Link Cited by: §1.5, §1, §6. Leveson (2012) N. G. Leveson Engineering a safer world: systems thinking applied to safety. MIT Press, Cambridge, MA. External Links: Document, Link Cited by: §13.2, §13.5, §15.1, §15.3, §17.4. Lu et al. (2024) C. Lu, C. Lu, R. T. Lange, J. Foerster, J. Clune, and D. Ha The ai scientist: towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.06292. External Links: 2408.06292, Link Cited by: §10.1, §4.6. Manheim and Garrabrant (2019) D. Manheim and S. Garrabrant Categorizing variants of goodhartâs law. arXiv preprint arXiv:1803.04585. External Links: 1803.04585, Link Cited by: §9.2. Maynard Smith and Price (1973) J. Maynard Smith and G. R. Price The logic of animal conflict. Nature 246, p. 15â18. External Links: Document, Link Cited by: §18.3. Ngo et al. (2022) R. Ngo, L. Chan, and S. Mindermann The alignment problem from a deep learning perspective. arXiv preprint arXiv:2209.00626. External Links: 2209.00626, Link Cited by: §1, §16.2. Novikov et al. (2025) A. Novikov, N. VĆ©, M. Eisenberger, E. Dupont, P. Huang, A. Z. Wagner, S. Shirobokov, B. Kozlovskii, F. J. R. Ruiz, A. Mehrabian, M. P. Kumar, A. See, S. Chaudhuri, G. Holland, A. Davies, S. Nowozin, P. Kohli, and M. Balog AlphaEvolve: a coding agent for scientific and algorithmic discovery. arXiv preprint arXiv:2506.13131. External Links: 2506.13131, Link Cited by: §10.1, §4.6. OECD (2024) OECD Facts not fakes: tackling disinformation, strengthening information integrity. OECD Publishing, Paris. External Links: Document, Link Cited by: §6.6. Omohundro (2008) S. M. Omohundro The basic ai drives. In Artificial General Intelligence 2008: Proceedings of the First AGI Conference, Frontiers in Artificial Intelligence and Applications, Vol. 171, p. 483â492. External Links: Link Cited by: §8.3. Ord (2020) T. Ord The precipice: existential risk and the future of humanity. Grand Central Publishing, New York. External Links: Link Cited by: §19.4, §2.3, §2. Perrow (1999) C. Perrow Normal accidents: living with high-risk technologies. Updated edition, Princeton University Press, Princeton, NJ. External Links: Link Cited by: §11.4, §12.3, §15.4, §19, §7.3. Rinaldi et al. (2001) S. M. Rinaldi, J. P. Peerenboom, and T. K. Kelly Identifying, understanding, and analyzing critical infrastructure interdependencies. IEEE Control Systems Magazine 21 (6), p. 11â25. External Links: Document, Link Cited by: §12, §15.4. Ross et al. (2021) R. Ross, V. Pillitteri, R. Graubart, D. Bodeau, and R. McQuaid Developing cyber-resilient systems: a systems security engineering approach. Technical report Technical Report NIST Special Publication 800-160 Volume 2 Revision 1, National Institute of Standards and Technology. External Links: Document, Link Cited by: §17.6. Russell (2019) S. Russell Human compatible: artificial intelligence and the problem of control. Viking, New York. External Links: Link Cited by: §1.4. Schick et al. (2023) T. Schick, J. Dwivedi-Yu, R. DessĂŹ, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom Toolformer: language models can teach themselves to use tools. In Advances in Neural Information Processing Systems, Vol. 36, p. 68539â68551. External Links: Link Cited by: §15.2, §4.4. Shevlane et al. (2023) T. Shevlane, S. Farquhar, B. Garfinkel, M. Phuong, J. Whittlestone, J. Leung, D. Kokotajlo, N. Marchal, M. Anderljung, N. Kolt, L. Ho, D. Siddarth, S. Avin, W. Hawkins, B. Kim, I. Gabriel, V. Bolina, J. Clark, Y. Bengio, P. Christiano, and A. Dafoe Model evaluation for extreme risks. arXiv preprint arXiv:2305.15324. External Links: 2305.15324, Link Cited by: §13.4, §15.1, §16.3, §17.1, §19.1, §19.3, §3.1, §3.6, §5.3, §7.1. Sutton and Barto (2018) R. S. Sutton and A. G. Barto Reinforcement learning: an introduction. 2 edition, MIT Press, Cambridge, MA. External Links: Link Cited by: §4.1. Tabassi (2023) E. Tabassi Artificial intelligence risk management framework (ai rmf 1.0). Technical report Technical Report NIST AI 100-1, National Institute of Standards and Technology. External Links: Document, Link Cited by: §17.1, §17.5, §19.3. Turner et al. (2021) A. M. Turner, L. Smith, R. Shah, A. Critch, and P. Tadepalli Optimal policies tend to seek power. In Advances in Neural Information Processing Systems, Vol. 34. External Links: Link Cited by: §3.5, §8.2, §8.4. Yao et al. (2023) S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao ReAct: synergizing reasoning and acting in language models. In International Conference on Learning Representations, External Links: Link Cited by: §15.2, §4.2, §4.4.