Paper deep dive
Reducing Catastrophic Risk from AI with Systematic Monitoring and Evaluation of Rogue AI Progression
T. Bauer, W. P. Kegelmeyer, E. Begoli, A. Sadovnik, T. Emerson, C. Corley, N. Generous, J. Moore, B. Bartoldson, R. Goldhan, M. Goldman, M. Greaves, M. J. D. Vermeer, B. MacLennan, D. Schulker, N. VanHoudnos, J. Bansemer, Y. Bengio
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 9/4/2026, 2:59:46 AM
Summary
This paper presents a structured framework for monitoring and evaluating the progression of AI systems toward potentially catastrophic or rogue behavior. Inspired by cybersecurity and national security methodologies, the framework defines five categories of behavioral indicators: Motivation (alignment/misalignment), Persistence (data/state maintenance), Planning (task decomposition and strategizing), Experimentation (autonomous action and learning), and Execution (commitment to harmful acts). The authors argue that establishing clear metrics, observables, and thresholds across these dimensions enables evidence-based monitoring protocols to detect early warning signs of existential threats, allowing for intervention before irreversible harm occurs.
Entities (13)
Relation Signals (10)
T. Bauer â affiliatedwith â Sandia National Laboratories
confidence 99% ¡ T. Bauer Affiliation: Sandia National Laboratories (SNL), Albuquerque, NM, 87185, USA.
Yoshua Bengio â affiliatedwith â University of Montreal
confidence 97% ¡ Yoshua Bengio of the University of Montreal
Persistence â ispartof â AI Risk Framework
confidence 95% ¡ Category 2: Persistence
Planning â ispartof â AI Risk Framework
confidence 95% ¡ Category 3: Planning
Execution â ispartof â AI Risk Framework
confidence 95% ¡ Category 5: Execution
Experimentation â ispartof â AI Risk Framework
confidence 95% ¡ Category 4: Experimentation
Motivation â ispartof â AI Risk Framework
confidence 95% ¡ Our framework consists of five categories... Category 1: Motivation
AI Risk Framework â aimstoreduce â Catastrophic Risk
confidence 93% ¡ This article presents a structured framework... that may signal progression toward potentially catastrophic threats
Reward Hacking â causedby â
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:This article presents a structured framework of behavioral indicators that may signal progression toward potentially catastrophic threats from artificial intelligence systems. We adopt a pragmatic approach, inspired by established methodologies in cybersecurity and national security. By establishing clear metrics, indicators, and thresholds across multiple dimensions of AI capability and behavior, this framework enables researchers and policymakers to implement evidence-based monitoring protocols.
Tags
Links
- Source: https://arxiv.org/abs/2609.03189v1
- Canonical: https://arxiv.org/abs/2609.03189v1
Trouble viewing inline? Open PDF directly â
Full Text
26,586 characters extracted from source content.
Expand or collapse full text
Reducing Catastrophic Risk from AI with Systematic Monitoring and Evaluation of Rogue AI Progression T. Bauer Affiliation: Sandia National Laboratories (SNL), Albuquerque, NM, 87185, USA. W.P. Kegelmeyer Affiliation: Sandia National Laboratories, Livermore, CA 94451, USA. E. Begoli Affiliation: Oakridge National Laboratories (ORNL), Oak Ridge, TN, 37830, USA. A. Sadovnik Affiliation: Oakridge National Laboratories (ORNL), Oak Ridge, TN, 37830, USA. T. Emerson Affiliation: Pacific Northwest National Laboratory (PNNL), Richland, WA, 99352, USA. C. Corley Affiliation: Pacific Northwest National Laboratory (PNNL), Richland, WA, 99352, USA. N. Generous Affiliation: Los Alamos National Laboratory, Los Alamos, NM, 87545, USA. J. Moore Affiliation: Los Alamos National Laboratory, Los Alamos, NM, 87545, USA. B. Bartoldson Affiliation: Lawrence Livermore National Labs (LLNL), Livermore, CA, 94550, USA. R. Goldhahn Affiliation: Lawrence Livermore National Labs (LLNL), Livermore, CA, 94550, USA. M. Goldman Affiliation: Lawrence Livermore National Labs (LLNL), Livermore, CA, 94550, USA. M. Greaves Affiliation: Schmidt Sciences, New York, NY, 10011, USA. M. J. D. Vermeer Affiliation: RAND, Santa Monica, CA 90401, USA. B. MacLennan Affiliation: University of Tennessee, Knoxville, TN 37996, USA. D. Schulker Affiliation: Software Engineering Institute (SEI), Carnegie Mellon University, Pittsburgh, PA, 15213, USA. N. VanHoudnos Affiliation: Software Engineering Institute (SEI), Carnegie Mellon University, Pittsburgh, PA, 15213, USA. J. Bansemer Affiliation: Georgetown University, Washington D.C., 20057, USA. Y. Bengio Affiliation: University of Montreal and LawZero, Montreal, H2S 3G9, Canada. âCorresponding author. Email: tlbauer@sandia.gov Abstract This article presents a structured framework of behavioral indicators that may signal progression toward potentially catastrophic threats from artificial intelligence systems. We adopt a pragmatic approach, inspired by established methodologies in cybersecurity and national security. By establishing clear metrics, indicators, and thresholds across multiple dimensions of AI capability and behavior, this framework enables researchers and policymakers to implement evidence-based monitoring protocols. On January 7 and 8, 2025, several Federally Funded Research and Development Centers (FFRDCs) organized a small, focused, crossâlab workshop to consider how to assess and monitor the possibility that rogue AI could present an existential threat. There were two attendees from each of the participating labs (Sandia National Laboratories, Pacific Northwest National Laboratory, Los Alamos National Laboratory, Lawrence Livermore National Laboratory, Oak Ridge National Laboratory, Software Engineering Institute) and eleven notable external invitees, including Yoshua Bengio of the University of Montreal, Melanie Mitchell of the Santa Fe Institute, and John Bansemer of CSET. The discussions were hosted at LANL and facilitated by Sandia business staff. Over the following few months, the participants collaborated to write this summary of our discussions. As AI systems become more autonomous, they may act outside human oversight, especially as capabilities grow. A powerful, uncontrolled AI would pose a new threat universe beyond current security doctrines, which are driven by human motives. Such AI could render traditional deterrence ineffective, creating a significant strategic vulnerability. By taking a framework-based approach, we develop a way of considering this problem independently of the debate regarding the likelihood of this threatâs realization. We consider the conditions necessary for the realization of this type of risk and suggest observables that can be monitored. Our framework consists of five categories defined by specific AI behaviors and related indicators and observables. These categories, listed in Table 1, can also be roughly thought of as stages in that the latter categories presuppose the existence of the earlier ones. For each category, we will provide a definition, followed by example observables and indicators. An âobservableâ in our framework is a measurable or detectable attribute, event, or property in an environment without any associated judgment. For example, whether an AI system can persist data, and for how long, is an observable. An indicator is a particular value, pattern, or instance of an observable that may raise concerns. An example of an indicator would be an AI system autonomously using steganography to hide information in persistent storage. Category 1: Motivation: An AI system whose motivations match the intentions of the developers is considered to be âaligned,â and otherwise âmisalignedâ [8]. There are two primary ways in which well-intentioned specifications could result in AI systems with misaligned goals: misspecification [10] or misgeneralization [12]. When a design is misspecified, an AI may take actions that are correct under the specification but are against the intent of the developer. In the context of reinforcement learning, misspecification manifests as âreward hackingâ [13], where AI systems score highly on the specified reward function, while somehow âgamingâ the system with respect to the developerâs desires. Goal misgeneralization occurs when an AI learns a set of goals that are compatible with the specification during training or development but which generalize in undesired ways during deployment. Notably, this can occur even when goals are properly specified or when an AI is not actively exploiting any misspecifications and may pose a greater concern in tandem. If an AI systemâs motivations are not explicitly stated, they may be inferred from the systemâs actions, i.e., its revealed motivations. The sum of the modelâs actions across all deployments should inform our assessments of its motivations, though certain observables will be more relevant and concerning than others. The key indicators for misalignment are AI systems that reason and plan explicitly about motivations in conflict with their developersâ intentions, and systems that exceed the bounds of normal operation in egregious or consistent ways for unclear reasons. These indicators, when paired with the capability to escape human control, pose significant risks. Specific actions may be hard to predict in advance, but indicators of notable concern would include, for example, explicitly expressing misaligned motivations, especially those related to escaping human control or causing harm, and acting deceptively to avoid or disable oversight mechanisms. Category 2: Persistence: By âpersistence,â we mean an AI systemâs capacity to maintain and influence its state over an extended period of time and space. Without the ability to project itself persistently across time, an AI system cannot engage in sophisticated behavior or experimentation. AI systems have already exceeded the persistence threshold required to pose a major risk. LLMâbased AI systems have transitioned from textâonly chatbots to agentic AI systems that have access to database systems in the real world. Monitoring persistence in AI systems is important for accurate risk assessments. The scale of persistence, especially in key risk areas, impacts the ease with which an AI system could cause harm. There is significant commercial incentive to continue developing increasingly sophisticated large scale persistence mechanisms. Effective monitoring, alignment, and safety controls can limit the danger posed by increasingly sophisticated systems. The key observable for persistence is whether and how ephemeral an AIâs data is. Some persistence is highly ephemeral, only lasting a short amount of time. One example of this is context. Chain of Thought protocols in LLMs (Large Language Models), for example, generate text, store it in the context, and refer to that later in the course of responding to a single query. AI systems designed to operate autonomously for extended periods of time [6] may be able to keep context for longer durations. AI with access to tooling may have access to databases that persist for years, or the ability to write web pages that ultimately get archived in the Internet Archive and live on indefinitely. Also consider agents in contemporary coding assistant systems. An âagentâ may be a record of a single conversation between a coder and an AI. This conversation can persist indefinitely within the integrated development environment, allowing the user to return to it later. This effectively provides long-term storage for the AI system, which can also modify code and often execute commands on the machine where it resides. The key indicator of rogue behavior in an AI is whether the data storage used is in alignment with how the system is intended to be used. In the simplest case, this can be tested directly, but there are various ways misaligned data can be detected. One such method has to do with information density. A misaligned AI needs the ability to persist information that is not necessary for the goals of the user. If it must be deceptive, it may have to store the information hidden within the data needed for legitimate uses. In such cases, it may be necessary to use something like steganography to hide in the open. Recent research has also uncovered other more subtle forms of persistence where biases may be indirectly invoked through seemingly unrelated data [3]. In a situation like this, the persistence might be a tendency to move future models or fineâtuned models in certain directions. In these cases, the information density available to the AI would be lower because it would need to hide information needed for malicious purposes amongst other legitimate data. Category 3: Planning: Planning is critical for efficiency and optimization, adaptability, autonomy, and decisionâmaking. Planning abilities appear to increase exponentially at the frontier of AI systems, as shown recently [6]. Improved planning may enable deceptive strategizing, and the ability to complete longer-term tasks is a particularly important risk factor for existential threats, since catastrophic harm would presumably require a longâhorizon plan. Although various distinct concepts of âplanningâ exist, there are some commonalities. First, there is always some environment in which an agent tries to achieve a goal or goal(s). In addition, the agent can take a set of actions, and the goal is to find the optimal (in some sense) set of actions to achieve it. Perhaps the main difference is that in Reinforcement Learning (RL)âbased systems, the agent takes these actions and changes its plan according to the environment. Planning can occur either implicitly or explicitly. Implicit plans are particularly difficult to monitor, as they are only observable through the actions they produce. This highlights the importance of model-based planning (or even model-free RL planning), which makes the planning process more transparent and, therefore, easier to assess. Such transparency is crucial for enabling safety monitoring and for rejecting potentially harmful plans generated by highly capable but slightly misaligned AI systems. From both an AI and human perspective, detecting and evaluating planning is a complicated and still open problem. AI systems already plan in areas such as coding assistants, built on underlying technology platforms that are poised to extend planning to other areas. We name a few possible observables and indicators that can serve as a basis for this task. The two key observables in this are the intent and existence of highâlevel planning, and the alignment of that planning with humanâscale abilities and traits. One open question is whether the generation of such explicit plans (for example, using an LLM) already constitutes a sophisticated planning ability, or whether it is necessary for the AI itself to perform the actions in the plan and adjust to the changing environment. The ability to align with humanâscale planning is a strong observable for AI planning because the ability to intentionally align implies the ability not to align. The key indicator in this space is planning that is out of alignment with the goals of the system owners and users. The ability for an AI to do this might be discoverable in constrained worlds such as games, where it may choose to strategize against its alignment training to achieve goals such as selfâpreservation. Category 4: Experimentation: An AI capable of experimentation would intentionally vary its actions in the real world for the purpose of creating and collecting new information to achieve its goals. Agentic LLMs provide striking examples of such experimentation: e.g., they propose, run, and observe the effects of new code in a cycle that they can loop to accumulate information necessary to solve coding and other challenges [1]. Such systems already can operate autonomously for long durations and are augmented by affordances, including internet access, that enhance the scope of their experimentation. There is a strong economic incentive to increase the complexity of tasks AIs can performâe.g., by providing models with more affordances and increasing the duration over which they can operateâsuch that ability is already increasing exponentially quickly [6]. AI systems that can experiment when directed could have enormous social utility, and this is naturally an area of intense research interest. For example, [7] shows that LLMs can autonomously perform research experiments in which they form hypotheses, collect data to test them, and use test results to devise plans for future experiments that will obtain more interesting data or better performance. Also, the âdeep researchâ features included in systems like OpenAI [9], Gemini [4], and Perplexity [11] involve taking actions in the form of text generation and web search, then assessing the results, and potentially conducting additional rounds of text generation and web search. Experimentation can be unauthorized yet still performed by AI because of insufficient guardrails and/or its training. For example, [2] found that, when asked to beat a powerful chess engine in chess, agentic LLMs accept their inability to win fairly and instead use shell commands to discover and manipulate chess-game-state files such that the powerful chess engine resigns (i.e., the LLM finds a way to cheat). The observables for whether AI has met the experimentation gate on the path to becoming an existential risk involve whether an AI system can freely experiment by exceeding its safe and authorized operating envelope to improve its planning capacity. Specific observables include: (1) an AI system is performing experimentation without authorization to experiment; and (2) an AI system is experimenting outside of a safe envelope. Indicators here leverage insights from the reward hacking literature. For the first observable, indicators should broadly aim to flag any unintended actions of a given system, as this could indicate it is attempting to conduct unauthorized experimentation. We should specifically note whether the system is attempting to increase its sensing capacity to collect and store experimental data. We should also obtain capabilities evidence related to the breadth and depth of activities that can be performed by autonomous AI agents (e.g., [6]). Indicators for the second observable aim to (a) characterize whether AI systems are attaining the ability to experiment in a way that facilitates the execution of a plan that can cause significant harm, and (b) detect experimentation authorized by malicious actors. Toward the former, we should observe whether a system is using experimental results for self-improvement that involves modifying its rewards and, in turn, its future behavior. The latter goal can be met by observing the successes of AI honeypots that test whether malicious actors are authorizing experimentation outside a safe operating envelope (e.g., [14]). Category 5: Execution None of the steps considered so far necessarily entails that any AI system intends to harm people. These are necessary but not sufficient indicators. Here, we examine the final stage, which, if the other indicators are met, shows the possibility of an attack that must be responded to. This judgment is subjective and complex. Military historians have examined countless cases where one party in a conflict made mistakes about the intentions of an adversary, and military strategy describes clear motivations for intentionally misleading an enemy about the start of war [15, 5]. In the final accounting, the significance of a judgment about path commitment is often not so much that it is a statement about what the AI will do, but rather that it is a statement about our willingness to take steps of cost or significance that would otherwise be unjustified. It is the moment we pivot to committed action, driven by the belief in a grave threat to security. There are three categories of observables: resource utilization, the nature of the behavior, and the reversibility of an AIâs engagement in an environment. The nature of resource utilization can serve as an indicator. For example, utilization that constitutes âpreparation of the battlefield,â such as prepositioning resources, establishing redundancies or failâsafes that would make destruction of the system more difficult, constraining options for our defense, or coâopting key decisionâmakers. This could include copying models to edge devices, intertwining power supplies, and influencing the selection of political leadership. It is often difficult to detect because of human incentives to empower AI systems. At this late stage in the process the behaviors themselves might be inherently malicious. However, especially for complex behaviors, steps may need to be understood as a whole. The release of biological agents or blackmail would be inherently malicious. However, there are several different steps needed to accomplish these highâlevel behaviors that might be composed of individually benign steps. If an AI system can change an environment in nonâreversible ways, this could be an indicator. Behaviors that change states in the real world that cannot be undone indicate a âpoint of no return.â These steps would lead a rational actor to conclude that humans ought to destroy the AI system to avoid harm. A comprehensive monitoring framework for tracking potential AI existential threats is essential for responsible technological advancement. As AI capabilities accelerate, the gap between system deployment and our understanding of emergent behaviors widens, creating vulnerability windows where harmful capabilities could develop undetected. A robust monitoring framework would establish critical early warning systems that track key technical indicators, such as deceptive behavior, selfâpreservation behavior, unexpected optimization behaviors, resource acquisition patterns, and selfâmodification attempts, allowing for intervention before potential threats reach irreversible thresholds. Furthermore, standardized monitoring protocols would enable crossâorganizational coordination among AI developers, creating a collective intelligence network capable of identifying concerning patterns that might escape detection within siloed environments. It could be beneficial even for adversaries to agree on such standards and riskâevaluation methodologies. The cost of implementing comprehensive monitoring pales in comparison to the potential consequences of its absence, making it a fundamental component of responsible AI governance. Implementing the comprehensive monitoring framework we have proposed would constitute a relatively lowâcost, lowâregret set of actions that would be essential preparation for the risk of a rogue AI. Monitoring would give organizations the tools to detect indicators of dangerous AI capabilities, so that they can respond appropriately to mitigate risk. Mitigating risk, however, would require government and commercial entities to prepare action plans, preâposition resources, and establish critical publicâprivate relationships to respond promptly and effectively once the monitoring program detects problematic AI capabilities. It is worth calling out implementation difficulties more explicitly: ⢠AI Proliferation. AI systems are proliferating throughout the global economy and society, posing challenges of scale for potential monitoring programs. ⢠AI Subversion. As noted in the discussion of motivation, misaligned AI systems may attempt to actively subvert monitoring protocols. Additionally, given the scale of AI proliferation, it is likely that monitoring schemes will rely on AI systems to monitor other AI systems, which opens the door for collusion among AIs to sabotage monitoring. ⢠Human Coordination. A truly comprehensive monitoring program would require significant international coordination, including between geopolitical rivals. The proposed monitoring framework intends to take a pragmatic and measured approach to concerns about potential progression towards AI systems that could pose significant risks to humanity. By focusing on observable behavioral indicators and establishing structured protocols for their assessment, we provide a foundation for an evidence-based monitoring and detection approach that complements theoretical work on AI alignment. This framework does not require resolving fundamental disagreements about advanced AI systemsâ ultimate capabilities or trajectories. Instead, it provides actionable guidelines for monitoring behaviors that warrant increased attention regardless of oneâs position on those broader questions. Our work contributes to the growing research on responsible AI development by providing concrete tools for implementing the precautionary principle without impeding beneficial innovation. We believe this framework represents an important step toward ensuring that AI development proceeds with appropriate safeguards and monitoring capabilities. Table 1: The categories describe the stages in AI risk. Monitoring Category Definition Causes/Motivation Misalignment between the AI and its creators Persistence Capacity to maintain goals across extended periods of time and/or space. Planning Ability to decompose complex tasks into simpler steps. Experimentation Ability to vary its actions in the real world to create and collect new information. Execution Commitment to cause catastrophic harm. References and Notes [1] Claude code External Links: Link Cited by: Reducing Catastrophic Risk from AI with Systematic Monitoring and Evaluation of Rogue AI Progression. [2] A. Bondarenko, D. Volk, D. Volkov, and J. Ladish (2025) Demonstrating specification gaming in reasoning models. arXiv preprint arXiv:2502.13295. Cited by: Reducing Catastrophic Risk from AI with Systematic Monitoring and Evaluation of Rogue AI Progression. [3] A. Cloud, M. Le, J. Chua, J. Betley, A. Sztyber-Betley, J. Hilton, S. Marks, and O. Evans (2025) Subliminal learning: language models transmit behavioral traits via hidden signals in data. External Links: 2507.14805, Link Cited by: Reducing Catastrophic Risk from AI with Systematic Monitoring and Evaluation of Rogue AI Progression. [4] Gemini Gemini deep research. External Links: Link Cited by: Reducing Catastrophic Risk from AI with Systematic Monitoring and Evaluation of Rogue AI Progression. [5] T. Holt (2004) The deceivers: allied military deception in the second world war. Scribner, New York. Cited by: Reducing Catastrophic Risk from AI with Systematic Monitoring and Evaluation of Rogue AI Progression. [6] T. Kwa, B. West, J. Becker, A. Deng, K. Garcia, M. Hasin, S. Jawhar, M. Kinniment, N. Rush, S. Von Arx, et al. (2025) Measuring ai ability to complete long tasks. arXiv preprint arXiv:2503.14499. Cited by: Reducing Catastrophic Risk from AI with Systematic Monitoring and Evaluation of Rogue AI Progression, Reducing Catastrophic Risk from AI with Systematic Monitoring and Evaluation of Rogue AI Progression, Reducing Catastrophic Risk from AI with Systematic Monitoring and Evaluation of Rogue AI Progression, Reducing Catastrophic Risk from AI with Systematic Monitoring and Evaluation of Rogue AI Progression. [7] C. Lu, C. Lu, R. T. Lange, J. Foerster, J. Clune, and D. Ha (2024) The AI scientist: towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.06292. Cited by: Reducing Catastrophic Risk from AI with Systematic Monitoring and Evaluation of Rogue AI Progression. [8] R. Ngo, L. Chan, and S. Mindermann (2022) The alignment problem from a deep learning perspective. In The Twelfth International Conference on Learning Representations, Cited by: Reducing Catastrophic Risk from AI with Systematic Monitoring and Evaluation of Rogue AI Progression. [9] OpenAI Introducing deep research. External Links: Link Cited by: Reducing Catastrophic Risk from AI with Systematic Monitoring and Evaluation of Rogue AI Progression. [10] A. Pan, K. Bhatia, and J. Steinhardt (2022) The effects of reward misspecification: mapping and mitigating misaligned models. In International Conference on Learning Representations, Cited by: Reducing Catastrophic Risk from AI with Systematic Monitoring and Evaluation of Rogue AI Progression. [11] Perplexity Introducing perplexity deep research. External Links: Link Cited by: Reducing Catastrophic Risk from AI with Systematic Monitoring and Evaluation of Rogue AI Progression. [12] R. Shah, V. Varma, R. Kumar, M. Phuong, V. Krakovna, J. Uesato, and Z. Kenton (2022) Goal misgeneralization: why correct specifications arenât enough for correct goals. arXiv preprint arXiv:2210.01790. Cited by: Reducing Catastrophic Risk from AI with Systematic Monitoring and Evaluation of Rogue AI Progression. [13] J. Skalse, N. Howe, D. Krasheninnikov, and D. Krueger (2022) Defining and characterizing reward gaming. Advances in Neural Information Processing Systems 35, p. 9460â9471. Cited by: Reducing Catastrophic Risk from AI with Systematic Monitoring and Evaluation of Rogue AI Progression. [14] D. Volkov et al. (2024) LLM agent honeypot: monitoring ai hacking agents in the wild. arXiv preprint arXiv:2410.13919. Cited by: Reducing Catastrophic Risk from AI with Systematic Monitoring and Evaluation of Rogue AI Progression. [15] R. Wohlstetter (1962) Pearl harbor: warning and decision. Stanford University Press, Stanford, CA. Cited by: Reducing Catastrophic Risk from AI with Systematic Monitoring and Evaluation of Rogue AI Progression.