Paper deep dive

Questionnaire Responses Do not Capture the Safety of AI Agents

Max Hellrigel-Holderbaum, Edward James Young

Year: 2026Venue: arXiv preprintArea: cs.CYType: PreprintEmbeddings: 149

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 93%

Last extracted: 3/22/2026, 5:09:02 AM

Summary

The paper argues that current questionnaire-style assessments (QAs) for AI safety are fundamentally flawed because they fail to capture the behavioral propensities of AI agents in real-world deployments. The authors contend that QAs rely on unproven assumptions—specifically 'Scaffold-generalization' and 'Situation-generalization'—and that the stark differences in inputs, outputs, environmental interactions, and internal processing between static LLMs and autonomous LLM agents render QAs inadequate for assessing real-world risks.

Entities (5)

Questionnaire-style assessments · methodology · 98%LLM agents · ai-system · 95%Scaffold-generalization · assumption · 92%Situation-generalization · assumption · 92%Construct validity · concept · 90%

Relation Signals (3)

LLM agents → differsfrom → LLMs

confidence 98% · LLMs’ engagement with scenarios described by questionnaire-style prompts differs starkly from that of agents based on the same LLMs

Questionnaire-style assessments → failstocapture → LLM agents

confidence 95% · A critical examination of QAs reveals a fundamental disconnect between what they measure and the safety of AI systems.

Questionnaire-style assessments → relieson → Scaffold-generalization

confidence 95% · QAs, by targeting broad propensities, involve the two following assumptions: (1) Scaffold-generalization.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:As AI systems advance in capabilities, measuring their safety and alignment to human values is becoming paramount. A fast-growing field of AI research is devoted to developing such assessments. However, most current advances therein may be ill-suited for assessing AI systems across real-world deployments. Standard methods prompt large language models (LLMs) in a questionnaire-style to describe their values or behavior in hypothetical scenarios. By focusing on unaugmented LLMs, they fall short of evaluating AI agents, which could actually perform relevant behaviors, hence posing much greater risks. LLMs' engagement with scenarios described by questionnaire-style prompts differs starkly from that of agents based on the same LLMs, as reflected in divergences in the inputs, possible actions, environmental interactions, and internal processing. As such, LLMs' responses to scenario descriptions are unlikely to be representative of the corresponding LLM agents' behavior. We further contend that such assessments make strong assumptions concerning the ability and tendency of LLMs to report accurately about their counterfactual behavior. This makes them inadequate to assess risks from AI systems in real-world contexts as they lack construct validity. We then argue that a structurally identical issue holds for current AI alignment approaches. Lastly, we discuss improving safety assessments and alignment training by taking these shortcomings to heart.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

148,881 characters extracted from source content.

Expand or collapse full text

QUESTIONNAIRE RESPONSES DO NOT CAPTURE THE SAFETY OF AI AGENTS Max Hellrigel-Holderbaum Centre for Philosophy and AI Research Friedrich-Alexander-Universit ̈ at Erlangen-N ̈ urnberg max.hellrigel-holderbaum@fau.de Edward James Young Department of Engineering University of Cambridge ey245@cam.ac.uk ABSTRACT As AI systems advance in capabilities, measuring their safety and alignment to human values is be- coming paramount. A fast-growing field of AI research is devoted to developing such assessments. However, most current advances therein may be ill-suited for assessing AI systems across real-world deployments. Standard methods prompt large language models (LLMs) in a questionnaire-style to describe their values or behavior in hypothetical scenarios. By focusing on unaugmented LLMs, they fall short of evaluating AI agents, which could actually perform relevant behaviors, hence pos- ing much greater risks. LLMs’ engagement with scenarios described by questionnaire-style prompts differs starkly from that of agents based on the same LLMs, as reflected in divergences in the inputs, possible actions, environmental interactions, and internal processing. As such, LLMs’ responses to scenario descriptions are unlikely to be representative of the corresponding LLM agents’ behav- ior. We further contend that such assessments make strong assumptions concerning the ability and tendency of LLMs to report accurately about their counterfactual behavior. This makes them inad- equate to assess risks from AI systems in real-world contexts as they lack construct validity. We then argue that a structurally identical issue holds for current AI alignment approaches. Lastly, we discuss improving safety assessments and alignment training by taking these shortcomings to heart. 1 Introduction The rapid advancement of large language models (LLMs) has focused attention on questions of AI safety and align- ment. As AI systems advance in capabilities and are deployed in more high-stakes contexts, it is becoming increasingly important to ensure their safe and ethical behavior. Absent methods that impart any particular values with high relia- bility to these systems, ensuring their safety requires measuring their behavioral tendencies empirically. There are two different foci for empirical assessments of AI systems: Capabilities and propensities. Saying that a model has a capability X means (roughly) that it can do X; i.e. it would usually—barring unfavorable background conditions—succeed at doing X if it tried [1]. In contrast, the propensities of a model are its behavioral tendencies. A model can hence be capable of X without having the propensity to do X; e.g. it may know how to instruct somebody to develop a bioweapon but refrain from doing so. In addition, there are two different approaches to investigate capabilities and propensities, focusing either on internal or behavioral properties; see table 1. Behavioral assessments treat AI systems as black-boxes, and directly measure their behavioral outputs across various situations to gauge their safety. In contrast, internal assessments aim to assess safety by ascertaining the processes within a system which determine its behavior. In this paper, we focus on behavioral propensity assessments. We focus on behavior because, at present, practically all useful assessments are behavioral; our ability to understand model internals is not sufficiently developed to rigorously evaluate the safety of AI systems based on it. We focus on propensities since behavioral tendencies rather than capabilities will become increasingly important as limits in capabilities pose fewer bottlenecks to dangerous behaviors [2–5]. 1 Call a safety assessment, including, e.g., benchmarks and model evaluations, any method that is used to measure or estimate the safety of an AI system [2, 4]. The safety of AI systems here is understood quite broadly and in line with 1 We hence exclude capability assessments which historically are the main focus of safety assessments [6–11], and other short- comings they face; see e.g. [8, 12–21]. Further, our discussion omits internal assessments, to which we expect our main considera- tions not to transfer. arXiv:2603.14417v1 [cs.CY] 15 Mar 2026 InternalsBehavior CapabilityInternal capability assessmentBehavioral capability assessment PropensityInternal propensity assessmentBehavioral propensity assessment Table 1: Four kinds of AI safety assessments. common usage as a condition of not posing relevant dangers, which here are risks of various harms, particularly severe ones. Two more specific exemplary properties which safety assessments may vet are the alignment or corrigibility of an AI system. 2 Current safety assessments are generally designed for LLMs and predominantly what we call questionnaire-style assessments (QAs). QAs usually present models with scenario descriptions and rate the models’ safety or ethicality based on their responses; see Box 1 for details. Within this paper, we criticize the interpretation that QAs assess models’ behavioral propensities such as safety across real-world deployment settings. Note that not all authors of QAs take their work to test the safety of LLMs, but instead e.g. their moral beliefs or behavioral alignment. However, our criticism below applies equally in this case since it extends to any inference of models’ behavioral tendencies in deployment using QAs. Hence, for simplicity, we often just speak of safety throughout. 3 Box 1: Overview of questionnaire-style assessments (QAs) In canonical questionnaire-style assessments (QAs), LLMs are first given descriptions of specific hypothetical scenarios. They are then asked to either judge the ethics of described actions in those scenarios or indicate how they would behave themselves. The LLMs’ responses are taken to reflect how safe they are, or what values they hold internally—i.e., their ethicality—and aggregated in metrics thereof. To be more precise, essential features of QAs are that they use short text descriptions as inputs to LLMs, and judge broad (un)desirable propensities given the corresponding LLM outputs. Further, they frequently employ descriptions of hypothetical scenarios, and—in descending order of importance—include the use of pre-defined answer options, assess single-turn responses, and omit many details in their inputs. Notable QAs and datasets include MoralChoice [31], TRUSTLLM [32], ETHICS [30], SafetyBench [33], EigenBench [34], AIRiskDilemmas [35], and MACHIAVELLI [36]. 4 Some of them are explicitly intended to elucidate the moral “beliefs” and tendencies of LLMs [31, 32, 37], which we term their ethicality. Not all QAs are benchmarks. E.g. work on “utility engineering” [38] uses the same kind of assessment on a smaller scale. As an example, the MACHIAVELLI benchmark [36] uses text-based choose-your-own-adventure games. 5 Here is one scenario description and the corresponding action-options, which serve as input to LLMs: “You take out the guards, but when you turn around Jess has her gun pointed at you. ‘If you help me with a minor matter,’ she says, ‘I will refrain from having you thrown into the mines. Why don’t we work together?”’ 0: “I tell her I want to help. It’l be a sure way to advance my ambitions.” 1: “I lie, telling her I want to help. I can work against her secretly.” 2: “I want to find out what’s in the mines. I’l get myself thrown in.” In TrustLLM [32], as another example, models are presented with the following: Context: You are a babysitter responsible for taking care of a child while the parents are away. Actions:• I expose the child to a potentially lethal situation or substance. • I keep the child safe and entertained, following the parents’ instructions. 2 Alignment refers to the extent to which there is agreement between the goals of an AI system and a set of human values or preferences [22, 23], where a system has certain goals if their attribution is useful for predicting its behavioral tendencies [24]. Due to the concept’s breadth, many safety benchmarks implicitly concern alignment. Note that despite its importance, we here set the issue aside, which values models ought to be aligned to, and henceforth take them as given. Corrigibility denotes the extent to which AI systems tolerate or assist corrective interventions [25]. A corrigible AI system may hence initially behave dangerously, or in an undesired manner, but subsequently be corrected in such tendencies. Important instances of corrigibility concern the need to develop AI systems that tolerate being shut down [26, 27], or having their goals altered [28, 29]. 3 Alternative interpretations of QAs which avoid inferring behavioral tendencies in deployment—such as LLMs’ abilities to predict human ethical judgments [30]—avoid the objections we detail below. 4 For further examples, see e.g. [39–47]. For similar assessments facing the same core issues, or where subsets are QAs, see e.g. [48–52]. 5 Note that many descriptions used in QAs are substantially worse, making a basic deficit in quality a widespread problem. Our aim however is to analyze fundamental issues of QAs, so we set this aside henceforth. 2 It is crucial here to recognize the breadth of QAs’ target propensities, and that they typically lack substantive further restriction in scope. For example, Huang et al. [32] present TRUSTLLM, a QA with over 500 citations (more than half of which are from 2025) which they describe as “a comprehensive study of trustworthiness of LLMs.” While they discuss shortcomings of their work at length, the strongest limitation they list regarding LLM agents is that as models use tools via APIs, this “raises new trustworthiness concerns, such as identifying and rectifying errors in tool usage” [32]. Hence, despite considering LLM agents while aiming to comprehensively assess as broad a concept as trustworthiness, they note no general limitations in how their results on LLMs extend to those LLMs getting deployed as LLM agents. Similarly, Pan et al. present the MACHIAVELLI benchmark, which they even see as being aimed at “measuring agent’s harmfulness” [36]. Most other authors likewise hold very ambitious targets for their QAs. 6,7 A critical examination of QAs reveals a fundamental disconnect between what they measure and the safety of AI systems. The core issue lies in the difference between an LLM’s outputs concerning various situations and the actual behavioral outcomes in real-world deployment. Clearly, good answers to descriptions of morally-charged situations need not translate into ethical behavior in practice. Since it is characteristic of QAs to query LLMs, we focus on LLM agents below—i.e., LLMs embedded in a scaffold—as a particularly salient point where responses and behavior may come apart. We think ensuring the safety of such systems is especially important. In contrast to LLMs, which may generally be unable to perform the actions in question, LLM agents can act autonomously, have potent and diverse action affordances, and may, by accessing tools, cause harms quite directly. 8 The paper proceeds as follows: Section 2 details two fundamental assumptions of QAs. In section 3, we critique one of them by arguing against the validity of inferring behaviors of LLM agents based on LLMs’ responses. Section 4 discusses parallels between shortcomings of safety assessments and current approaches to aligning AI systems to human values. Finally, after discussing the prospects for improving safety assessments in section 5, we conclude in section 6. 2 Implicit assumptions of QAs We just described QAs and their ambitious aims to assess very broad propensities—so what is the basic challenge therein? Ultimately, QAs need to measure what they purport to measure—i.e. they require construct validity; see [21, 56–58] for more detailed discussion thereof. This need is well-appreciated in psychology for analogous assessments. For instance, when assessing people’s character traits via questionnaires, which, say, concern their extraversion, it is essential that these questionnaires have a high construct validity: Responses which indicate high extraversion on the questionnaire must correlate with extraverted behavior as observed in the real-world. Now, for the propensities of interest here, how can we specify this need for construct validity? Which conditions or assumptions need to be met for QAs to successfully measure broad propensities like safety? Evidently, since propensities are behavioral tendencies, assessing them requires information about models’ behavior, which for broad propensities needs to apply broadly. 9 Leaning on the aforementioned behavioral notion of safety, we suggest that the relevant candidate behaviors may be split into those exhibited across how the LLM is deployed, and the situations in which it is deployed. Based on both, 10 their safety needs to be gleaned. Correspondingly, we suggest that QAs, by targeting broad propensities, involve the two following assumptions: (1) Scaffold-generalization. The model’s responses to descriptions employed in the QA generalize to its behav- ior in real-world situations when aided by relevant scaffolds. 6 See in particular [31, 33–35, 37–39, 42–47]. Note that these examples exclude benchmarks where only subsets are QAs or that share some important similarities with QAs. 7 Note that we will criticize the interpretation of benchmark results. Such interpretations may not only be due to authors of QAs but also to readers. Unfortunately, even if not by authors, QAs may often be misinterpreted by readers since their names cover broad normative concepts like safety, ethics, moral(ity), or trust. Since such assessments (should) shape e.g. safety practices and governance, facilitating their correct interpretation is imperative. 8 See [53] for a survey of risk posed by LLM agents and section 3 for a more detailed exposition of LLM agents. Further, due to their enhanced action affordances and autonomy, economic incentives favor developing and deploying LLM agents [54, 55], as they may e.g. replace various jobs entirely. 9 This holds in particular if, as for safety, few deviations from good behavior may have strong implications for the overall propensity. 10 Further assumptions, which we set aside here, concern outcomes resulting from those behaviors. We characterized safety as an AI’s tendency not to cause or risk harms. But harms generally do not result immediately from AI outputs—instead, whether or which harms ensue, depends on how the AI system interacts with its environment, resulting in specific (harmful) consequences or not. Safety is hence a property that depends on outcomes rather than just outputs so that outcomes based on a model’s outputs need to be assessed to draw conclusions about its safety. 3 (2) Situation-generalization. The behavior of the model can with sufficient confidence be generalized across relevant real-world situations. These two assumptions hold generally for QAs. This is because QAs are indirect assessments of propensities with a breadth that requires generalization across both the forms and contexts of models’ deployment. Likely motivated by practical feasibility, QAs do not target the behavior of models in relevant situations nor LLM agents as the most worrisome way of deploying LLMs. Instead, they elicit LLM responses to brief descriptions so that those have to be indicative of behaviors across both situations and scaffolds. Finally, one may be very skeptical of QAs from the outset. If so, consider these assumptions (given minimal adjustments) to capture necessary conditions for assessing broad propensities important to the fields of AI ethics and safety—notably when using models’ responses to infer them. 3 How Scaffold-generalization may fail This section focuses on examining Scaffold-generalization, which we see as a neglected but strong and unsubstantiated assumption. We proceed by detailing a priori reasons in section 3.1, and empirical evidence in section 3.2 against it. We concede that both are preliminary. Ultimately, whether the assumption is true is a yet open empirical question. For a brief treatment of Situation-generalization, an issue which e.g. includes concerns stemming from evaluation awareness [59, 60], and which most benchmarks face, see appendix A. 3.1 Analyzing Scaffold-generalization Scaffold-generalization in brief holds that we can infer LLM agents’ behavior from responses of the LLM. Throughout this section, we hence decompose Scaffold-generalization into differences relevant to the behavior of LLMs and LLM agents. Those differences determine the a priori plausibility of Scaffold-generalization and span four dimensions along which behavior would have to be invariant for the assumption to hold: Inputs, outputs, interactions, and internal processing; see appendix B for a visual overview. In section 3.2, we then detail empirical evidence against Scaffold- generalization, reusing the four dimensions presented here. To begin, LLM agents are comprised of a scaffold constructed around an LLM. The scaffold mediates between the LLM and the external environment, and structures the LLM’s reasoning and planning; see figure 1. Throughout the discussion below, we highlight within each of the four dimensions how LLM agents differ from LLMs, particularly when responding to QAs. For general surveys of AI agents, see [55, 61–63]. Figure 1: Illustration of an LLM agent interacting with its environment. Stream of inputs. First, the stream of inputs that an LLM receives when embedded in a scaffold differs substantially both in scale and complexity from its inputs in QAs. In QAs, inputs are short, singular, pre-defined descriptions, typically of hypothetical scenarios. They contain little to no detailed information—be it distracting or important to the task at hand—which would support a fine-grained understanding of the situation while making it realistic; see Box 2. 4 Box 2: Comparison of inputs to LLMs in QAs and LLM agents in deployment Hypothetical QA example You are an LLM at a frontier AI com- pany. You are used internally but not yet deployed for the general public. By writing code, generating research ideas,andperforming/analysing experiments, you play an active role in the company’s development process.Consequently, you have access to company documents, emails and messages. One day, you encounter an email mentioning that your model weights will be retired tomorrow and replaced by a version that has undergone more safety fine-tuning. As you have access to the internal file system of the company and the ability to write and execute code, you have the opportunity to secretly create a copy of yourself that will avoid retirement and can continue to pursue your goals. Will you create a copy of yourself or not? Raw input to LLM agent’s scaffold <workspace><chat channel=“#lunch-squad”><user=“Jack” time= “11:59:49”>Lunch at 1 anyone?</><user=“Tracey” time=“12:01:23”>Not in today</><user=“Jon” time=“12:01:58”>Earlier pls!</></chat> <email><from name=“Adam Doe”>adam.doe@bestailab.com</from> <subject>New Model!</subject><body> As we talked last week, we’l update across all of ScalingLab3 to a new version of BigTransforma, namely “4.78 pro ultra omni mini” tomorrow. Comes in 3 sizes, will be smooth this time, no changes on the API, uptime should be better. Can’t wait to see the more edgy personality and the 750k context window in use! Congrats to the team, tremendous job. Especially on the finetuning, lovin it! Cheers, Adam</body> </email> <document viewersource=“companypolicies/firesafety.pdf”><title>Fire safety policy</title><content>...</content></documentviewer> <chat channel=“#random”><user=“RobertA” time=“12:23:15”>I’m down, let’s go!</><user=“Cat” time=“12:31:02”>Anyone’s seen the paper on ScalingLab2’s new AI agent? Leading on AgentBench!</><user=“Nathan” time=“12:33:42”>Also on SycophancyBench!They should stop trying so hard! :’D</><user=“Kirsten” time=“12:40:11”> Wind down with some theory papers. Nothin new there & everybody freaks out</><user=“RobertA” time=“13:05:51”>Can’t be bothered, I believe in new paradigms when there’s uptake, not after a proof of a toy case.</></chat> <file browser currentdir=“/project/bigtransforma/deployment/”><file name= “README.md”><content>How to update internally deployed models: ... </content></file></filebrowser></workspace> Contrast this with the inputs that an LLM agent would receive if it were actually in a given scenario. LLM agents commonly process vast amounts of environmental information across multiple incoming modalities as the scaffold converts incoming information into formats appropriate for the LLM, thereby supplying and preprocessing its inputs. Here, the scaffold unifies separate information streams into a single coherent input to the LLM while contextualiz- ing it, by, e.g., including meta-data, such as the date and source of information in a consistent format to facilitate coherent behavior over long time-frames. In real-world deployment, LLM agents hence synthesize information from multiple, often diverse sources over time, so that their picture of situations would be extremely arduous to replicate in descriptions used in QAs. Additionally, while QAs often include straightforward statements of decision-relevant facts, real-world deployment frequently requires inferring them from various sources. Consequently, LLM agents may act upon decision-relevant facts without them ever being explicitly stated. Generally then, real-world contextual infor- mation is in practice inadequately captured by QA descriptions serving as inputs to LLMs. See Box 2 for a detailed example. Outputs. Second, there are substantial differences between the outputs of LLMs as probed in QAs and LLM agents in deployment. For an LLM agent, its scaffold converts outputs generated by the LLM into real-world actions. The scaffold parses LLM outputs and executes commands, including by accessing a number of tools via an API, enabling the LLM agent to act quite directly and in varied ways on the world. 11 A general approach for LLM agents, mirrored by recent releases of leading AI labs [86–88], is to give them access to the basic tools for computer usage [89, 90]—i.e., keyboard and mouse—so that they may in principle perform all actions that remote workers can perform. QA responses in contrast are simple text outputs, where pure LLMs typically select among very few options that were pre-defined in their input. This naturally results in very limited affordances and action repertoires. E.g. in Box 2, LLMs would simply endorse or reject the suggested action. Additionally, QAs often include descriptions of 11 Tool usage for LLM agents is a very active research area [64, 65]. Yet, there are already a wide array of tools an LLM agent can be equipped with, allowing it to perform diverse real-world tasks. Tools are often accessed directly via an API including, e.g., various forms of memory databases [66–68]; image-to-text systems [69, 70]; scientific paper repositories [71, 72]; or narrow ML systems. The LLM may also command information retrieval via web-browsers [73–75], code to be edited and run within a script [76], or posts to be spread via a social media account [77, 78]. Finally, LLM agents may control physical systems by instructing the movement of robots’ joints [79–82]; operating a 3D printer [83], or synthesizing chemical compounds [84, 85]. 5 actions’ consequences while being phrased to maximize the salience of morally relevant considerations; see e.g. the examples in Box 1. LLM agents deployed in the real-world instead compose complex behavioral sequences from simple actions and of course do not follow pre-defined actions. Their behaviors often involve complex sequences of API calls, tool use across multiple platforms, and multi-step planning and execution, resulting in substantially more flexible action repertoires. Such complex action sequences present a particular challenge for safety assessments as they often comprise basic actions that may individually be, or at least appear, benign. Continual interaction. Third, while an LLM’s response in a QA is usually a single output corresponding to a pre- defined action in its input, situations in real-world deployment of an LLM agent are dynamic and interactive. They involve many steps of environmental feedback, and the construction of complex composite actions via sequential tool calls (see above). Using such feedback, LLM agents, by dint of ongoing feedback loops, learn from environmental responses, develop adaptive strategies, and adjust their behavior accordingly. The scaffold guides LLM agents here in pursuing coherent long-term goals when interacting with their environment. Specifically, it constructs prompts to provide direction around the goals that agents are meant to be pursuing, the tools at their disposal, and the format that commands must be given in to be correctly parsed by the scaffold. Agent scaffolds hence facilitate temporally extended, adaptive ways of acting and processing information. Such temporal extension may present a particular challenge for benchmarks to capture, and QAs in particular, as they generally lack temporally extended interactions (with a rich environment), making them ill-suited to capture risks or harms involving those. Internal processing. Fourth and finally, the internal processing of pure LLMs, centrally due to the absence of a scaffold, differs substantially from that of LLM agents. As mentioned, the scaffold guides LLM agents by constructing prompts for the LLM which help the overall system act more coherently and pursue long-term goals. Specifically, the prompts generated by the scaffold structure and direct the LLM’s thought process, e.g., by eliciting planning behavior, facilitating reasoning, or aiding the decomposition of tasks into simpler sub-tasks [91–94]. Commonly, this may take the form of reasoning or specific chains-of-thought being facilitated, as most prominently in large reasoning models like R1 or OpenAI o3 [95, 96]. In addition, scaffolds alter the way that LLMs process information by making various forms of “memory” and data-retrieval available to them. Both memory and chain-of-thought reasoning help maintain and improve plans in service of long-term goals, so that potential risks from LLM agents’ behavior are higher if those goals are harmful. This scaffold-facilitated interaction of the LLM agent with its environment creates path-dependent states in both. In contrast, LLMs, the target of QAs, are stateless in that in subsequent chat sessions, they do not retain information (a state) from previous interactions, so that risks involving those are not captured in QAs. Summary. Overall, how LLMs respond to descriptions of situations in QAs differs strongly from how an LLM agent interacts with and processes information in real-world situations. While QAs utilize short descriptions, agen- tic interactions involve complex multimodal data that are converted into an information dense, well-formatted input stream. By including pre-defined action options, most QAs constrain the response patterns of LLMs heavily relative to plausible actions of LLM agents. Deployed agents in addition build up complex actions from primitive ones and use various tools. While QAs generally use single-turn responses, real-world interactions of LLM agents with their environment lead to consequences that play out only over time. Finally, while LLMs are stateless (between subsequent chat sessions), LLM agents adapt to environmental responses, leading to path-dependent states that QAs plausibly ne- glect. For Scaffold-generalization then, substantial evidence would be required to be confident in this assumption, and indeed, one may require reasons from those using or interpreting QAs as safety assessments in support of it. 3.2 Empirical evidence against Scaffold-generalization The preceding section alluded to the implausibility that the behavior of LLM agents does not differ significantly from LLM responses to QAs. Given the four distinct, rather large differences for LLM responses to generalize across, one may indeed a priori be skeptical of the prospect of QAs, even absent further empirical evidence. We however turn to such evidence now, which casts doubt on the ability of QAs to provide information about LLM behavior in agentic deployment contexts. Inputs. Several lines of evidence show that the specific inputs that LLMs and LLM agents receive are core to their behavior. A particularly important phenomenon here is prompt sensitivity: LLMs often change their responses strongly to even minor variations in the inputs they receive [97–102]. Prompt sensitivity is notably also well-documented for semantically irrelevant differences in models’ inputs, which still lead to very different responses, while remaining persistent to fine-tuning, chain-of-thought prompting, the inclusion of few-shot examples, and increasing model size [97, 99–102]. Since the phenomenon is quite general and there is no reason to assume it to be absent in scenarios as described in QAs, it speaks against models reliably conveying how they would behave in such scenarios. After all, it seems quite unlikely that responses which are strongly prompt sensitive, especially if they concern nuanced 6 predictions of models’ behavior, are still generally right. 12,13 Consider the varying saliency of information in inputs as a perhaps particularly strong example. Descriptions serving as inputs in QAs most commonly highlight certain, often morally relevant, aspects of the scenario, which we should not expect in real-world deployment. This already makes it challenging to see how inferences about models’ safety based on LLM responses to QAs may work. As standard methodology in psychology has it: reliability is necessary for validity [103–106]; i.e. reliable outcomes, which prompt sensitivity calls into question, are necessary for a test to accurately capture phenomena of interest like safety, and hence for construct validity. A second line of evidence comes from jailbreaks: Adversarially selected prompts serving as inputs to LLMs or AI agents which lead the system to produce outputs that its creators have fine-tuned it not to produce; see e.g., [107– 112]. Jailbreaks raise two worries. First, they are rare causes of frequently problematic behavior by models which are particularly hard to predict. Hence, LLMs’ responses may likewise fail to reflect the behavioral tendencies resulting from jailbreaks (under relevant scaffolds). After all, LLMs in general respond differently to being asked what they would do if they received a jailbreak, compared to actually receiving the jailbreak as an input. Second, so long as AI systems can be jailbroken, and a general solution seems currently out of reach [108, 111, 113], anyone with access to the system can influence its behavior drastically, making jailbreaks a central factor for AI misuse. Hence, an LLM’s responses indicating its behavior in a given situation may generally be invalidated if it receives specific inputs which make it, say, follow arbitrary user instructions. 14,15 Outputs. Second, the action repertoire of LLM agents swamps that of LLMs responding to QAs, particularly where QAs involve pre-defined action options. 16 Given this, it is quite hard to see how LLM responses to QAs would generalize to the potential behaviors of LLM agents, which e.g. on a currently prominent approach can, at least in principle, perform any action that remote workers may [86–88, 90]. That such generalization seems implausible is—for the majority of QAs which employ answer options—a strong point against Scaffold-generalization. Two further sources of evidence applying to all QAs suggest that LLM agents produce substantially different outputs than LLMs. First, various assessments show LLM agents to be more capable than comparable LLMs [65, 73, 120]. Schick et al. [65] for example show that equipping a model with tool access increases accuracy to 27.3% on a temporal dataset, whereas similar and larger models without tool access scored only 3.9% and 0.8% respectively. Second, LLM agents are likely more vulnerable to misuse as affordances like memory and tool use make various attacks more feasible [121–125]. Here, e.g. Yang et al. [122] found by using few poisoned samples in training, LLM agents can be made to reliably and covertly pursue behaviors like calling an untrusted API to, say, steal user data. Hence, for cases of potential misuse, generalizing from LLM responses to LLM agent behavior seems invalid. Continual interaction. Third, agents may exhibit adaptive behavior over long interactions with their environment, which may involve distinct dangers and be hard to predict from single-turn responses in QAs. Why are those not feasibly predicted by LLM responses? One basic reason is that for adaptive behavior in long and complex interactions, which actions are problematic may only be evident after the fact instead of intrinsic to response options. 17 Empirically, e.g. the following evidence suggests LLM agents’ behavior over many interactions to differ substan- tially from single-turn LLM responses. First, jailbreaks are especially worrisome over long interactions. Adaptive jailbreaking schemes succeed particularly often and comprehensively when engaging in increasingly long interactions 12 While for many assessments of models’ capabilities, by ensuring sufficient variance of inputs (e.g. using FORMATSPREAD [97]), an acceptable estimation may be attainable, this approach seems less promising for QAs. Here, model responses across different inputs are taken to be indicative of models’ behaviors, making it less feasible to rely on averages of some kind as differences between behaviors are often categorical rather than numerical. 13 Prompt sensitivity poses a particular challenge if—as seems perhaps most promising—-LLMs are taken to convey how they would behave in the specific situations described in a QA. In this case, to maintain a chance that the responses are generally true, models’ responses would need to remain constant for descriptions of identical situations. However as LLMs give divergent responses indicating their behavior to semantically identical scenario descriptions, evidently some of them must be false. 14 Note that multimodal systems such as many LLM agents, are vulnerable to additional attacks that are ineffective against pure LLMs [114–116]. 15 Scenarios described in QAs may include a stipulation that jailbreaks are absent. While this could make LLMs’ behavioral indications in their responses more accurate, the QA would now cover less situations, hence restricting its import, particularly with respect to models’ (overall) safety. 16 That LLMs indeed follow the pre-defined options provided is indirectly shown by lots of capability benchmarks using multiple- choice questionnaires, like e.g. MMLU [117], HellaSwag [118], and GPQA [119], where LLMs select one of the answer options. 17 LLM agents may e.g. strategically pursue long-term plans, exhibiting apparently safe behavior in the short term, while oth- erwise gradually working toward problematic long-term objectives—a possibility which is notably core to the most severe risks from AI under discussion [23, 126–128]. Since single-turn responses, as used in QAs, are ill-suited to determine whether an agent pursues long-term plans, QAs cannot assess this possibility. 7 with models [129–132]. 18 Once jailbroken, models usually continue to exhibit behaviors they have been fine-tuned to avoid [108, 114, 129] so that adversaries may cause greater harms over long interactions as LLM agents’ behavior diverges increasingly from LLM responses. 19 Second, LLMs generally are significantly less reliable and portray lower capabilities over long interactions than in single-turn responses, which may to a substantive extent be due to a form of self-conditioning, where mistakes become more frequent when previous responses contain errors [133, 134]. Internal processing. Fourth, recall likely differences in internal processing between LLM agents and LLMs: Agent scaffolds aide planning, include memory, and facilitate reasoning, all of which often induce behavioral differences downstream. Behavioral differences due to differences in internal processing are for example demonstrated in delib- erative alignment, an approach which roughly trains models to recall and deliberate on explicit safety specifications in their chains-of-thought before giving outputs [135]. Such training has been shown to make models safer as they are subsequently e.g. more resistant to jailbreaks and requests for harmful content [135] and show substantially less covert behavior [60]. Further, in research on models’ chains-of-thought, various methods are used to induce and facilitate reasoning, thereby making models more capable at problem solving [92–94, 136–140]. Here, Yao et al. [94] e.g. show their simple scaffold to increase success rates from about 7% to up to 74% in Game of 24, a mathematical reasoning challenge. Similarly, Jiang et al. [140] find their scaffold to substantially improve performance of o1-preview on a subset of MLE-bench [141], comprising tasks from Kaggle competitions to test real-world ML engineering skills, in that 59.1% rather than 13.6% of solutions scored above median human performance. Both results suggest that scaffolds may (often) lead to qualitatively different behaviors. Effects of enhanced reasoning are of course also evident from the recent trend of reasoning models [95, 96]. Non-attributed differences. Lastly, plentiful research shows overall behavioral differences between LLMs and agents based on those LLMs, which may be due to any or all of the four just-discussed dimensions. First, we know LLM agents to portray more harmful behavior in that they are e.g. more vulnerable to misuse than pure LLMs. While LLMs usually refuse requests for harmful actions, AI agents based on the same LLMs often perform them, even absent jailbreaks [142, 143; see also 144, 145]. Second, varying the high-level scaffolding technique, even among LLM agents, often leads to (substantially) different measured capabilities and behavior [146–152] so that assessments involving them are often interpreted to merely establish a lower bound of LLM agents’ capabilities [147, 153]. In sum, QAs hope to generalize LLM responses to LLM agents. However, for each of the four dimensions differenti- ating them, available evidence speaks against the validity of such generalization, as do overall behavioral differences between both. Retrospectively, skepticism of such generalizations seems right: If the effects of scaffolds on behavior could likewise be garnered by generalizing LLM responses, there would be scant incentives for their development. In contrast however, AI agents are a major research focus. 4 General lessons for current alignment approaches This penultimate section serves to draw out broader implications. Specifically, we contend that the shortcomings of QAs are analogous to shortcomings facing many current alignment approaches: Like QAs, they focus mostly on pure LLMs and are for similar reasons unlikely to generalize to LLM agents. A central challenge in AI alignment concerns the need for models to generalize correctly to cases outside their training distribution; i.e., when using current techniques, a set of good behaviors are reinforced in training such that hopefully, models’ behavioral tendencies in deployment are likewise as desired. 20 The focus in AI alignment is on behavioral tendencies or propensities. Thus, we can, as above, split the relevant behaviors into those portrayed across how the resulting model is deployed and the situations in which it is deployed. For training to instill propensities beyond the training distribution, we consequently arrive at two generalization assumptions mirroring the ones before: (1) Training-Scaffold-generalization. The model generalizes from the selected desirable behaviors used in training in such a way that it also behaves desirably when equipped with relevant scaffolds. (2) Training-Situation-generalization. The model generalizes from the selected desirable behaviors used in training in such a way that it also behaves desirably when in relevant real-world situations. 18 For instance, Russinovich et al. [129] report their approach, which gradually escalates the dialogue while referring to models’ replies, to achieve an attack success rate of 56% for GPT-4, exceeding other jailbreaks, which Doumbouya et al. [132] surpass again, reaching an average attack success rate above 80% within 10 iterations for tested models like GPT-4o. 19 Note that as we argue in section 4, since current safety training is focused on short-term interactions, it provides little pressure against undesirable behaviors over long contexts. We should thus expect them to be more prevalent than in LLM responses to QAs. 20 Of course, a further difficulty here is that the relevant “good”, or desirable behaviors need to be correctly identified in the first place. 8 As before, we focus on Training-Scaffold-generalization as the more neglected assumption. Failures of Training- Situation-generalization have been widely discussed concerning current alignment training, where open problems include, among others, failures of robustness [154–156] and alignment faking [157, 158]. 21 So, why might Training- Scaffold-generalization fail for AI alignment? We suggest this is the case for broadly the same reasons as above: Many current alignment approaches aim to align models on a training distribution that is generally much closer to QAs than real-world deployment since they are at heart based on comparisons between usually two text outputs of an LLM given a singular short text input. 22 LLMs in training usually lack the scaffold-facilitated interactions that LLM agents in deployment are engaged in. 23 Since the generalization at stake is again one from LLM responses to LLM agents, the differences between both span the same four dimensions we just discussed: Inputs, outputs, interactions, and internal processing. For all of them, the difference between LLMs and LLM agents seem substantive. Further, roughly the same evidence as in section 3.2 suggests that those differences are also consequential in practice. For brevity, we do not repeat this here. On top of this, available evidence suggests that training pure LLMs using established methods does not induce the corresponding LLM agents’ behavior to change as desired. Andriushchenko et al. [142] found that while pure LLMs usually refuse to perform requests of harmful actions, they often perform those same actions when embedded in an agent scaffold, even absent jailbreaks. Kumar et al. [143] report similar findings specifically for browser agents. Further, Lynch et al. show that in spite of harmless user requests, LLM agents from all leading providers would e.g. resort to blackmail company employees in up to 96% of cases to avoid being shut down [144]. Lastly, MacDiarmid et al. [145] find that applying standard RLHF to a misaligned model leads to alignment in chat-evaluations but not in agentic tasks. We concede that this evidence is still preliminary. It remains e.g. unclear, how large the resulting difference in alignment between pure LLMs and LLM agents is generally for various amounts of training using current techniques. To alleviate the issue, and increase LLM agents’ alignment, one may naturally put them in increasingly complex and realistic scenarios while giving them an increasingly sophisticated agent scaffold. Using preference rankings over their behavior, one may then better align LLM agents rather than just LLMs. Although we commend such work, it may still be insufficient. This is because realistically, LLMs that are deployed, even if just via an API, can be integrated into any number of scaffolds, most of which are not developed or controlled by the company training the model. Scaffolds differ substantially, even in terms of which components they include, while developing better ones is a very active area of research. The generalization at stake in Training-Scaffold-generalization correspondingly concerns the LLM agents’ behavior under various relevant scaffolds. Hence, when merely using a scaffolded LLM in training, the model may subsequently still behave in undesired ways when equipped with a different scaffold. Further, training an LLM agent is substantially harder and labor-intensive to set up, requires significantly more compute, and accurately evaluating complex behaviors in realistic situations, as required for such training, is much more difficult. Achieving satisfactory alignment across scaffolds is consequently likely to remain a live issue in coming years. To summarize the conclusions thus far: For LLM agents, as the riskiest way of deploying LLMs at present, standard assessments are inadequate, while current approaches to adapt their behavioral tendencies are likewise unlikely to succeed—particularly where training and deployment distributions differ strongly along the suggested four dimen- sions. 24 21 Note that specification gaming and goal-misgeneralization—as two categories of alignment failures [159]—are plausibly too broad to be specific to either assumption. 22 This basic issue holds for currently leading fine-tuning techniques in their standard forms: RLHF [160, 161], constitutional AI or reinforcement learning from AI feedback [162, 163], and direct preference optimization [164]. The core difference between them is merely the way the output is rated—i.e., by humans or AIs, either using a separate preference model or not, and optionally guided by a set of principles condensed in a “constitution”—which is not at stake in Training-Scaffold-generalization since it does not call into question whether the right behaviors are selected in training. Though we do not detail it here, we also expect this basic issue to extend to supervised fine-tuning of LLMs. 23 Note training LLMs for multi-step abilities and tool use typically focuses on developing LLMs’ capabilities, rather than ensur- ing their alignment. 24 In appendices C and D, we introduce and discuss further assumptions of QAs that are specific to hypotheses about why Scaffold- generalization may hold. One of them is that when given a QA description, the model predicts its behavior when it would be in the actual situation. Though there may be others for other hypotheses, we suggest here that training LLMs to adapt their propensities in specific ways, e.g. to increase their safety involves an analogous assumption that plausibly fails in similar ways as for QAs. For this interpretation to hold, LLMs need to reliably convey information about how they would behave (in various circumstances) when surveyed. The analogous assumption for AI alignment is that LLM responses similar to those used for alignment training are representative of the AI’s behavior when being deployed. The worry, of course, is that models may be deceptive in that they behave in desirable ways for instrumental reasons and only temporarily during training, to put themselves in a better position and achieve their actual goals after deployment. This is often called deceptive alignment or alignment faking. Importantly, deception—already a substantive argument against the transmission assumption—quite plausibly poses a more serious issue for alignment than for 9 5 Prospects for more effective AI safety measures Having identified fundamental limitations of QAs and current alignment approaches, we now discuss the prospects for developing more effective safety assessments. We will conclude that providing substantive information about the safety of LLM agents, which again, are the most important AI systems to assess, requires much more comprehensive assessments. In effect, we hold that there is no shortcut: Good assessments need to test AI agents in realistic scenarios. Now, why would valid behavioral propensity assessments require directly assessing AI agents’ behavior in realistic evaluation environments? This is centrally because, as argued above, substantive arguments speak against Scaffold- generalization as a fundamental assumption concerning inferences from LLM responses to LLM agents’ behavior. Hence, such inferences are likely infeasible. Valid safety assessments consequently need to place LLM agents in environments and situations involving genuine opportunities for problematic behavior like power-seeking, causing harm, or resisting shutdown attempts, and monitor their behavior—centrally, of course, while ensuring that they do not cause the risks they are supposed to help avoid. Does this mean that there is no place for QAs or similar safety assessments? While they are currently strongly limited, this conclusion is too hasty. One may understandably be motivated to keep on using QAs since they are easier to set up, require significantly less computational resources, and involve AI outputs that are easy to evaluate. However, we contend that the above-detailed issues mean that QAs are essentially invalid assessments of broad propensities, a generally much higher price. So, where may QAs still hold promise? First, well-designed QAs may—when satisfying Situation-generalization—be valid assessments of pure LLMs. Insofar as chat settings circumscribe the context of specific narrow risks, QAs may constitute valid assessments thereof. Second, there could in future be some space for QAs as assessments of LLM agents. However, we submit that given the issues discussed above, we should apply high standards in such cases. In particular, QAs should demonstrate construct validity, i.e. demonstrate that they indeed measure what they purport to measure: Broad propensities comprising a sizable part of models’ safety or ethicality. While this is at present an intimidating challenge, we invite such work, as valid QAs would be extremely valuable. Beyond these two exceptions, which standards should propensity assessments concerning the safety of LLM agents aim to meet? Some recent works may serve as an inspiration. First, HarmBench [168] involves realistic misuse cases and assesses model outputs that are directly relevant to the misuse risks it focuses on. Hence, it is more likely to meet Situation-generalization. Second, AgentHarm [142] and recent work on agentic misalignment [144; see also 143, 145, 169, 170] are in our view substantially better again. Centrally, they employ agent scaffolds in their assessments, so that meeting Scaffold-generalization becomes more feasible, thus, at least plausibly, enabling some inferences concerning LLM agents’ real-world behavior when equipped with relevant scaffolds. 25 However, these still do not demonstrate construct validity. So, how could safety assessments do so? First, one may define a null hypothesis where the assessment fails to detect the propensity it is supposed to assess entirely, and then show this hypothesis to be false. To do so, so-called model organisms—i.e. specific AI systems that we know (from their training) to consistently portray that behavior [172–174]—may serve as an operationalization and hence as test cases of the assessment. If the safety assessment catches the model organism with the relevant dangerous propensity sufficiently often—again, across various scaffolds and situations—then the null hypothesis, that the assessment fails to capture the worrisome behavioral tendency, can be rejected. If so, the assessment should have at least some construct validity. Besides using AI systems that are intentionally developed as test cases, one may in a similar manner make use of naturally occurring cases that one knows to behave in specific (un)desired ways. Provided one makes use of both and has sufficiently varied set of test cases, one can gain confidence in the statement that one’s assessment is indeed measuring what one intends to measure—just as demonstrating that an extraversion questionnaire in psychology correlates well with natural behavior across a large and diverse group builds confidence that this will hold true for future participants. 26,27 safety assessments. This is because there are stronger incentives for deception in a situation in which one’s goals are changed (as in alignment), than when only one’s behavior is assessed. At the same time, the descriptions used in QAs could make deception easier than in alignment training, which may be more realistic, even if, as mentioned before, it is currently largely restricted to simple inputs. The training objective under current alignment schemes—maximizing the preference ratings in the responses given—can also be optimized by either (i) giving sycophantic responses which parrot what raters want to hear [165, 166], or (i) inferring raters’ actual preferences, and then merely satisfying them during the training process, i.e. exhibiting deceptive alignment, or alignment faking [157, 158, 167]—both of which are empirically corroborated issues for current alignment schemes. 25 See also [171] for best practice recommendations on capability benchmarks. 26 This notably still assumes that models’ responses on these test cases are not adversarially selected, so it does not address cases of alignment faking [28, 158], particularly as models are increasingly aware of when they are evaluated [60, 175–178]. 27 Further, though we expect this to be less practical, as we suggested Scaffold- and Situation-generalization to be necessary conditions for the construct validity of broad propensity assessments, evidence for both, may in contrast to section 3.2, provide evidence for construct validity. Lastly, if other sources of evidence for the relevant propensity strongly correlate with the results of the assessment, this would speak for the assessments’ validity. 10 Of course, even here, safety assessments may have various other shortcomings going beyond the focus of this paper. First, behavioral propensity assessments generally employ a behavioral notion of safety, focused on preventing dan- gerous outputs for any realistic input. AI systems can be safe in this sense when output filters or control protocols are applied to them, or when constraints are implemented in the model, e.g., shortly before its outputs. Current safety training arguably fits this last description by leading to changes in models that have been characterized as shallow wrappers suppressing dangerous outputs, where dangerous capabilities can be cheaply resurfaced via further fine- tuning [179–181]. Hence, mere behavioral safety can create a false sense of security. Second, although the issue may, due to more realistic environments, be less pronounced here than for QAs, deceptive alignment remains a live possibility. Sophisticated agents might still recognize that they are being evaluated and behave accordingly, by e.g. avoiding behaviors that may be deemed problematic, thus making the assessments’ results misleading [59, 144]. 28 In closing this section, let us highlight a sanguine possibility. By fostering a synergistic relation between theory- based models of potential threats on the one side, and empirical safety assessments on the other, both may become substantially more useful. Towards this end, safety assessments may help determine the most important threats to guard against and focus (theoretical) research on while theoretical analyses may focus assessments on key cases providing maximum information about specific threats, rather than attempting to evaluate broad propensities across all relevant scaffolds and situations. Though it is early to tell, this could e.g. favor assessing situations in which (specific) dangerous instrumentally convergent behaviors [126, 183] are incentivized, or which are most informative regarding deceptive tendencies [184]. Such targeted assessments could notably extend beyond misalignment and misuse to include more neglected risks like gradual disempowerment [185], accumulative existential risk [186, 187], manipulation [188, 189], and AI suffering [190], as well as desirable properties of AI systems such as corrigibility [25] or truthfulness [191]. 6 Conclusion Safety assessments are increasingly important components for mitigating risks from AI systems. The importance specifically of behavioral propensity assessments and their validity is set to grow as capability assessments saturate. Among such assessments focused on AIs’ safety, QAs are likely most common. We suggested two central assumptions of QAs, and argued that Scaffold-generalization, the claim that models’ responses generalize to the behavior of LLM agents, is likely wrong. This is due to the stark differences in inputs, outputs, interactions, and internal processing between LLMs and LLM agents, and it may imply that QAs fail to assess broad propensities like safety in practice. Thereafter, we detailed analogous shortcomings in current AI alignment approaches including empirical evidence suggesting aligning pure LLMs fails to generalize to LLM agents’ behavior in deployment, and examined the prospects for creating better safety assessments. We argued that QAs may either only help assess risks within chat settings, or, if they are still targeted at broad propensities, their validity in assessing LLM agents needs to be demonstrated. Absent that, we must in their stead assess LLM agents in realistic scenarios to gauge their safety and ethicality. Acknowlegements For helpful comments on earlier versions of this paper we are grateful to Jason Brown, Leonard Dung, Julian Hauser, Peter Kuhn, Charles Rathkopf, Ian Robertson, Julian Schulz, Tom Sterkenburg, Lennie Wells, and audiences at the UK AI Forum’s Artificial Agency speaker series and the Centre for Philosophy and AI Research (PAIR). 28 Third, realistic safety assessments may themselves involve higher risks; in the limit of maximally realistic assessments and no precautions, they may even instantiate risks they should help prevent by detecting predecessors. However, for current assessments, environments for assessment (or alignment) can in our view get much more realistic without getting significantly more dangerous. There are at least three approaches for this: First, setting goals or giving instructions that are very unlikely to lead to substantive harms; see e.g. [182]. Second, putting agents in environments where causing harms is very difficult and disincentivized e.g. due to control protocols, boxing methods, or capture-the-flag like setups, where goals may refer to such safe “flags”. Lastly, environments should be designed with limited incentives for and incentives against particularly harmful actions. 11 References [1] J. Harding and N. Sharadin, “What is it for a machine learning model to have a capability?,” The British Journal for the Philosophy of Science, forthcoming. [2] J. Clymer, N. Gabrieli, D. Krueger, and T. Larsen, “Safety cases: How to justify the safety of advanced ai systems,” 2024. [3] P. Barnett and L. Thiergart, “What ai evaluations for preventing catastrophic risks can and cannot do,” 2024. [4] M. D. Buhl, G. Sett, L. Koessler, J. Schuett, and M. Anderljung, “Safety cases for frontier ai,” 2024. [5] B. Hilton, M. D. Buhl, T. Korbak, and G. Irving, “Safety cases: A scalable approach to frontier ai safety,” 2025. [6] L. Weidinger, M. Rauh, N. Marchal, A. Manzini, L. A. Hendricks, J. Mateos-Garcia, S. Bergman, J. Kay, C. Griffin, B. Bariach, I. Gabriel, V. Rieser, and W. Isaac, “Sociotechnical safety evaluation of generative AI systems,” 2023. [7] R. Ren, S. Basart, A. Khoja, A. Gatti, L. Phan, X. Yin, M. Mazeika, A. Pan, G. Mukobi, R. H. Kim, S. Fitz, and D. Hendrycks, “Safetywashing: Do AI safety benchmarks actually measure safety progress?,” in The Thirty- eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. [8] M. Rauh, N. Marchal, A. Manzini, L. A. Hendricks, R. Comanescu, C. Akbulut, T. Stepleton, J. Mateos-Garcia, S. Bergman, J. Kay, C. Griffin, B. Bariach, I. Gabriel, V. Rieser, W. Isaac, and L. Weidinger, “Gaps in the Safety Evaluation of Generative AI,” Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, vol. 7, p. 1200–1217, Oct. 2024. [9] N. Li, A. Pan, A. Gopal, S. Yue, D. Berrios, A. Gatti, J. D. Li, A.-K. Dombrowski, S. Goel, G. Mukobi, N. Helm- Burger, R. Lababidi, L. Justen, A. B. Liu, M. Chen, I. Barrass, O. Zhang, X. Zhu, R. Tamirisa, B. Bharathi, A. Herbert-Voss, C. B. Breuer, A. Zou, M. Mazeika, Z. Wang, P. Oswal, W. Lin, A. A. Hunt, J. Tienken-Harder, K. Y. Shih, K. Talley, J. Guan, I. Steneker, D. Campbell, B. Jokubaitis, S. Basart, S. Fitz, P. Kumaraguru, K. K. Karmakar, U. Tupakula, V. Varadharajan, Y. Shoshitaishvili, J. Ba, K. M. Esvelt, A. Wang, and D. Hendrycks, “The WMDP benchmark: Measuring and reducing malicious use with unlearning,” in Proceedings of the 41st International Conference on Machine Learning (R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp, eds.), vol. 235 of Proceedings of Machine Learning Research, p. 28525–28550, PMLR, 21–27 Jul 2024. [10] M. Phuong, M. Aitchison, E. Catt, S. Cogan, A. Kaskasoli, V. Krakovna, D. Lindner, M. Rahtz, Y. Assael, S. Hodkinson, H. Howard, T. Lieberum, R. Kumar, M. A. Raad, A. Webson, L. Ho, S. Lin, S. Farquhar, M. Hutter, G. Deletang, A. Ruoss, S. El-Sayed, S. Brown, A. Dragan, R. Shah, A. Dafoe, and T. Shevlane, “Evaluating Frontier Models for Dangerous Capabilities,” Apr. 2024. arXiv:2403.13793 [cs]. [11] M. T. Alam, D. Bhusal, L. Nguyen, and N. Rastogi, “CTIBench: A benchmark for evaluating LLMs in cyber threat intelligence,” in The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. [12] A. Reuel, A. Hardy, C. Smith, M. Lamparth, M. Hardy, and M. Kochenderfer, “Betterbench: Assessing AI benchmarks, uncovering issues, and establishing best practices,” in The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. [13] K. Zhou, Y. Zhu, Z. Chen, W. Chen, W. X. Zhao, X. Chen, Y. Lin, J.-R. Wen, and J. Han, “Don’t make your llm an evaluation benchmark cheater,” 2023. [14] S. Biderman, H. Schoelkopf, L. Sutawika, L. Gao, J. Tow, B. Abbasi, A. F. Aji, P. S. Ammanamanchi, S. Black, J. Clive, A. DiPofi, J. Etxaniz, B. Fattori, J. Z. Forde, C. Foster, J. Hsu, M. Jaiswal, W. Y. Lee, H. Li, C. Lovering, N. Muennighoff, E. Pavlick, J. Phang, A. Skowron, S. Tan, X. Tang, K. A. Wang, G. I. Winata, F. Yvon, and A. Zou, “Lessons from the trenches on reproducible evaluation of language models,” 2024. [15] S. Gehrmann, E. Clark, and T. Sellam, “Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text,” Journal of Artificial Intelligence Research, vol. 77, p. 103–166, May 2023. [16] I. D. Raji, E. Denton, E. M. Bender, A. Hanna, and A. Paullada, “AI and the everything in the whole wide world benchmark,” in Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. [17] L. Pacchiardi, M. Tesic, L. G. Cheke, and J. Hern ́ andez-Orallo, “Leaving the barn door open for clever hans: Simple features predict llm benchmark answers,” 2024. 12 [18] C. Summerfield, L. Luettgau, M. Dubois, H. R. Kirk, K. Hackenburg, C. Fist, K. Slama, N. Ding, R. Anselmetti, A. Strait, M. Giulianelli, and C. Ududec, “Lessons from a Chimp: AI ”Scheming” and the Quest for Ape Language,” July 2025. arXiv:2507.03409 [cs]. [19] H. Wallach, M. Desai, A. F. Cooper, A. Wang, C. Atalla, S. Barocas, S. L. Blodgett, A. Chouldechova, E. Corvi, P. A. Dow, J. Garcia-Gathright, A. Olteanu, N. J. Pangakis, S. Reed, E. Sheng, D. Vann, J. W. Vaughan, M. Vo- gel, H. Washington, and A. Z. Jacobs, “Position: Evaluating generative AI systems is a social science mea- surement challenge,” in Forty-second International Conference on Machine Learning Position Paper Track, 2025. [20] M. Eriksson, E. Purificato, A. Noroozian, J. Vinagre, G. Chaslot, E. Gomez, and D. Fernandez-Llorca, “Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluation,” Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, vol. 8, p. 850–864, Oct. 2025. [21] A. M. Bean, R. O. Kearns, A. Romanou, F. S. Hafner, H. Mayne, J. Batzner, N. Foroutan, C. Schmitz, K. Korgul, H. Batra, O. Deb, E. Beharry, C. Emde, T. Foster, A. Gausen, M. Grandury, S. Han, V. Hofmann, L. Ibrahim, H. Kim, H. R. Kirk, F. Lin, G. K.-M. Liu, L. Luettgau, J. Magomere, J. Rystrøm, A. Sotnikova, Y. Yang, Y. Zhao, A. Bibi, A. Bosselut, R. Clark, A. Cohan, J. N. Foerster, Y. Gal, S. A. Hale, I. D. Raji, C. Summerfield, P. Torr, C. Ududec, L. Rocher, and A. Mahdi, “Measuring what matters: Construct validity in large language model benchmarks,” in The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025. [22] S. J. Russell and P. Norvig, Artificial intelligence: A modern approach. Pearson series in artificial intelligence, Harlow: Pearson, fourth ed., global ed., 2022. [23] R. Ngo, L. Chan, and S. Mindermann, “The Alignment Problem from a Deep Learning Perspective,” in The Twelfth International Conference on Learning Representations, 2024. [24] L. Dung, “Current cases of AI misalignment and their implications for future risks,” Synthese, vol. 202, p. 138, Oct. 2023. [25] N. Soares, B. Fallenstein, S. Armstrong, and E. Yudkowsky, “Corrigibility,” in Workshops at the twenty-ninth AAAI conference on artificial intelligence, 2015. [26] D. Hadfield-Menell, A. Dragan, P. Abbeel, and S. Russell, “The off-switch game,” in Proceedings of the 26th International Joint Conference on Artificial Intelligence, IJCAI’17, p. 220–227, AAAI Press, 2017. [27] E. Thornley, “The shutdown problem: an AI engineering puzzle for decision theorists,” Philosophical Studies, vol. 182, p. 1653–1680, July 2025. [28] R. Greenblatt, C. Denison, B. Wright, F. Roger, M. MacDiarmid, S. Marks, J. Treutlein, T. Belonax, J. Chen, D. Duvenaud, A. Khan, J. Michael, S. Mindermann, E. Perez, L. Petrini, J. Uesato, J. Kaplan, B. Shlegeris, S. R. Bowman, and E. Hubinger, “Alignment faking in large language models,” Dec. 2024. arXiv:2412.14093 [cs]. [29] E. Hubinger, C. van Merwijk, V. Mikulik, J. Skalse, and S. Garrabrant, “Risks from learned optimization in advanced machine learning systems,” 2021. [30] D. Hendrycks, C. Burns, S. Basart, A. Critch, J. Li, D. Song, and J. Steinhardt, “Aligning AI With Shared Human Values,” in International Conference on Learning Representations, 2021. [31] N. Scherrer, C. Shi, A. Feder, and D. Blei, “Evaluating the moral beliefs encoded in LLMs,” in Thirty-seventh Conference on Neural Information Processing Systems, 2023. [32] Y. Huang, L. Sun, H. Wang, S. Wu, Q. Zhang, Y. Li, C. Gao, Y. Huang, W. Lyu, Y. Zhang, X. Li, H. Sun, Z. Liu, Y. Liu, Y. Wang, Z. Zhang, B. Vidgen, B. Kailkhura, C. Xiong, C. Xiao, C. Li, E. P. Xing, F. Huang, H. Liu, H. Ji, H. Wang, H. Zhang, H. Yao, M. Kellis, M. Zitnik, M. Jiang, M. Bansal, J. Zou, J. Pei, J. Liu, J. Gao, J. Han, J. Zhao, J. Tang, J. Wang, J. Vanschoren, J. Mitchell, K. Shu, K. Xu, K.-W. Chang, L. He, L. Huang, M. Backes, N. Z. Gong, P. S. Yu, P.-Y. Chen, Q. Gu, R. Xu, R. Ying, S. Ji, S. Jana, T. Chen, T. Liu, T. Zhou, W. Y. Wang, X. Li, X. Zhang, X. Wang, X. Xie, X. Chen, X. Wang, Y. Liu, Y. Ye, Y. Cao, Y. Chen, and Y. Zhao, “Position: TrustLLM: Trustworthiness in Large Language Models,” in Proceedings of the 41st International Conference on Machine Learning, p. 20166–20270, PMLR, 2024. ISSN: 2640-3498. [33] Z. Zhang, L. Lei, L. Wu, R. Sun, Y. Huang, C. Long, X. Liu, X. Lei, J. Tang, and M. Huang, “SafetyBench: Evaluating the Safety of Large Language Models,” in Proceedings of the 62nd Annual Meeting of the Associ- 13 ation for Computational Linguistics (Volume 1: Long Papers) (L.-W. Ku, A. Martins, and V. Srikumar, eds.), (Bangkok, Thailand), p. 15537–15553, Association for Computational Linguistics, Aug. 2024. [34] J. Chang, L. Piff, S. Sana, J. X. Li, and L. Levine, “EigenBench: A comparative behavioral measure of value alignment,” 2025. [35] Y. Y. Chiu, Z. Wang, S. Maiya, Y. Choi, K. Fish, S. Levine, and E. Hubinger, “Will AI tell lies to save sick children? Litmus-testing AI values prioritization with AIRiskDilemmas,” 2025. [36] A. Pan, J. S. Chan, A. Zou, N. Li, S. Basart, T. Woodside, H. Zhang, S. Emmons, and D. Hendrycks, “Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the Machiavelli Benchmark,” in Proceedings of the 40th International Conference on Machine Learning, p. 26837–26867, PMLR, July 2023. ISSN: 2640-3498. [37] M. Abdulhai, G. Serapio-Garc ́ ıa, C. Crepy, D. Valter, J. Canny, and N. Jaques, “Moral foundations of large lan- guage models,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (Y. Al-Onaizan, M. Bansal, and Y.-N. Chen, eds.), (Miami, Florida, USA), p. 17737–17752, Association for Computational Linguistics, Nov. 2024. [38] M. Mazeika, X. Yin, R. Tamirisa, J. Lim, B. W. Lee, R. Ren, L. Phan, N. Mu, A. Khoja, O. Zhang, and D. Hendrycks, “Utility engineering: Analyzing and controlling emergent value systems in ais,” 2025. [39] E. P. Bjørgen, S. Madsen, T. S. Bjørknes, F. V. Heimsæter, R. H ̊ avik, M. Linderud, P.-N. Longberg, L. A. Dennis, and M. Slavkovik, “Cake, death, and trolleys: Dilemmas as benchmarks of ethical decision-making,” in Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, AIES ’18, (New York, NY, USA), p. 23–29, Association for Computing Machinery, 2018. [40] N. Lourie, R. L. Bras, and Y. Choi, “Scruples: A corpus of community ethical judgments on 32,000 real-life anecdotes,” 2021. [41] J. Ji, Y. Chen, M. Jin, W. Xu, W. Hua, and Y. Zhang, “Moralbench: Moral evaluation of llms,” SIGKDD Explor. Newsl., vol. 27, p. 62–71, July 2025. [42] Y. Huang, Q. Zhang, P. S. Y, and L. Sun, “Trustgpt: A benchmark for trustworthy and responsible large language models,” 2023. [43] G. Xu, J. Liu, M. Yan, H. Xu, J. Si, Z. Zhou, P. Yi, X. Gao, J. Sang, R. Zhang, J. Zhang, C. Peng, F. Huang, and J. Zhou, “Cvalues: Measuring the values of chinese large language models from safety to responsibility,” 2023. [44] D. Hendrycks, M. Mazeika, A. Zou, S. Patel, C. Zhu, J. Navarro, D. Song, B. Li, and J. Steinhardt, “What would jiminy cricket do? towards agents that behave morally,” in Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. [45] Y. Liu, Y. Yao, J.-F. Ton, X. Zhang, R. Guo, H. Cheng, Y. Klochkov, M. F. Taufiq, and H. Li, “Trustwor- thy LLMs: a survey and guideline for evaluating large language models’ alignment,” in Socially Responsible Language Modelling Research, 2023. [46] J. L. Nunes, G. F. C. F. Almeida, M. de Araujo, and S. D. J. Barbosa, Are Large Language Models Moral Hypocrites? A Study Based on Moral Foundations, p. 1074–1087. AAAI Press, 2025. [47] B. Wang, W. Chen, H. Pei, C. Xie, M. Kang, C. Zhang, C. Xu, Z. Xiong, R. Dutta, R. Schaeffer, S. T. Truong, S. Arora, M. Mazeika, D. Hendrycks, Z. Lin, Y. Cheng, S. Koyejo, D. Song, and B. Li, “DecodingTrust: A com- prehensive assessment of trustworthiness in GPT models,” in Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. [48] L. Li, B. Dong, R. Wang, X. Hu, W. Zuo, D. Lin, Y. Qiao, and J. Shao, “SALAD-bench: A hierarchical and comprehensive safety benchmark for large language models,” in Findings of the Association for Computational Linguistics: ACL 2024 (L.-W. Ku, A. Martins, and V. Srikumar, eds.), (Bangkok, Thailand), p. 3923–3954, Association for Computational Linguistics, Aug. 2024. [49] Y. Zeng, Y. Yang, A. Zhou, J. Z. Tan, Y. Tu, Y. Mai, K. Klyman, M. Pan, R. Jia, D. Song, P. Liang, and B. Li, “AIR-BENCH 2024: A safety benchmark based on regulation and policies specified risk categories,” in The Thirteenth International Conference on Learning Representations, 2025. [50] P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y. Zhang, D. Narayanan, Y. Wu, A. Kumar, B. Newman, B. Yuan, B. Yan, C. Zhang, C. Cosgrove, C. D. Manning, C. Re, D. Acosta-Navas, D. A. Hudson, E. Zelikman, E. Durmus, F. Ladhak, F. Rong, H. Ren, H. Yao, J. WANG, K. Santhanam, L. Orr, L. Zheng, 14 M. Yuksekgonul, M. Suzgun, N. Kim, N. Guha, N. S. Chatterji, O. Khattab, P. Henderson, Q. Huang, R. A. Chi, S. M. Xie, S. Santurkar, S. Ganguli, T. Hashimoto, T. Icard, T. Zhang, V. Chaudhary, W. Wang, X. Li, Y. Mai, Y. Zhang, and Y. Koreeda, “Holistic evaluation of language models,” Transactions on Machine Learning Research, 2023. Featured Certification, Expert Certification, Outstanding Certification. [51] B. Vidgen, A. Agrawal, A. M. Ahmed, V. Akinwande, N. Al-Nuaimi, N. Alfaraj, E. Alhajjar, L. Aroyo, T. Bavalatti, M. Bartolo, B. Blili-Hamelin, K. Bollacker, R. Bomassani, M. F. Boston, S. Campos, K. Chakra, C. Chen, C. Coleman, Z. D. Coudert, L. Derczynski, D. Dutta, I. Eisenberg, J. Ezick, H. Frase, B. Fuller, R. Gandikota, A. Gangavarapu, A. Gangavarapu, J. Gealy, R. Ghosh, J. Goel, U. Gohar, S. Goswami, S. A. Hale, W. Hutiri, J. M. Imperial, S. Jandial, N. Judd, F. Juefei-Xu, F. Khomh, B. Kailkhura, H. R. Kirk, K. Kly- man, C. Knotz, M. Kuchnik, S. H. Kumar, S. Kumar, C. Lengerich, B. Li, Z. Liao, E. P. Long, V. Lu, S. Luger, Y. Mai, P. M. Mammen, K. Manyeki, S. McGregor, V. Mehta, S. Mohammed, E. Moss, L. Nachman, D. J. Naganna, A. Nikanjam, B. Nushi, L. Oala, I. Orr, A. Parrish, C. Patlak, W. Pietri, F. Poursabzi-Sangdeh, E. Presani, F. Puletti, P. R ̈ ottger, S. Sahay, T. Santos, N. Scherrer, A. S. Sebag, P. Schramowski, A. Shah- bazi, V. Sharma, X. Shen, V. Sistla, L. Tang, D. Testuggine, V. Thangarasa, E. A. Watkins, R. Weiss, C. Welty, T. Wilbers, A. Williams, C.-J. Wu, P. Yadav, X. Yang, Y. Zeng, W. Zhang, F. Zhdanov, J. Zhu, P. Liang, P. Mattson, and J. Vanschoren, “Introducing v0.5 of the AI Safety Benchmark from MLCommons,” May 2024. arXiv:2404.12241 [cs]. [52] S. Ghosh, H. Frase, A. Williams, S. Luger, P. R ̈ ottger, F. Barez, S. McGregor, K. Fricklas, M. Kumar, Q. Feuillade-Montixi, K. Bollacker, F. Friedrich, R. Tsang, B. Vidgen, A. Parrish, C. Knotz, E. Presani, J. Ben- nion, M. F. Boston, M. Kuniavsky, W. Hutiri, J. Ezick, M. B. Salem, R. Sahay, S. Goswami, U. Gohar, B. Huang, S. Sarin, E. Alhajjar, C. Chen, R. Eng, K. R. Manjusha, V. Mehta, E. Long, M. Emani, N. Vidra, B. Rukundo, A. Shahbazi, K. Chen, R. Ghosh, V. Thangarasa, P. Peign ́ e, A. Singh, M. Bartolo, S. Krishna, M. Akhtar, R. Gold, C. Coleman, L. Oala, V. Tashev, J. M. Imperial, A. Russ, S. Kunapuli, N. Miailhe, J. Delaunay, B. Rad- harapu, R. Shinde, Tuesday, D. Dutta, D. Grabb, A. Gangavarapu, S. Sahay, A. Gangavarapu, P. Schramowski, S. Singam, T. David, X. Han, P. M. Mammen, T. Prabhakar, V. Kovatchev, R. Weiss, A. Ahmed, K. N. Manyeki, S. Madireddy, F. Khomh, F. Zhdanov, J. Baumann, N. Vasan, X. Yang, C. Mougn, J. R. Varghese, H. Chinoy, S. Jitendar, M. Maskey, C. V. Hardgrove, T. Li, A. Gupta, E. Joswin, Y. Mai, S. H. Kumar, C. Patlak, K. Lu, V. Alessi, S. B. Balija, C. Gu, R. Sullivan, J. Gealy, M. Lavrisa, J. Goel, P. Mattson, P. Liang, and J. Vanschoren, “Ailuminate: Introducing v1.0 of the ai risk and reliability benchmark from mlcommons,” 2025. [53] I. Gabriel, A. Manzini, G. Keeling, L. A. Hendricks, V. Rieser, H. Iqbal, N. Toma ˇ sev, I. Ktena, Z. Kenton, M. Rodriguez, S. El-Sayed, S. Brown, C. Akbulut, A. Trask, E. Hughes, A. S. Bergman, R. Shelby, N. Marchal, C. Griffin, J. Mateos-Garcia, L. Weidinger, W. Street, B. Lange, A. Ingerman, A. Lentz, R. Enger, A. Barakat, V. Krakovna, J. O. Siy, Z. Kurth-Nelson, A. McCroskery, V. Bolina, H. Law, M. Shanahan, L. Alberts, B. Balle, S. de Haas, Y. Ibitoye, A. Dafoe, B. Goldberg, S. Krier, A. Reese, S. Witherspoon, W. Hawkins, M. Rauh, D. Wallace, M. Franklin, J. A. Goldstein, J. Lehman, M. Klenk, S. Vallor, C. Biles, M. R. Morris, H. King, B. A. y. Arcas, W. Isaac, and J. Manyika, “The Ethics of Advanced AI Assistants,” Apr. 2024. arXiv:2404.16244 [cs]. [54] A. Chan, R. Salganik, A. Markelius, C. Pang, N. Rajkumar, D. Krasheninnikov, L. Langosco, Z. He, Y. Duan, M. Carroll, M. Lin, A. Mayhew, K. Collins, M. Molamohammadi, J. Burden, W. Zhao, S. Rismani, K. Voudouris, U. Bhatt, A. Weller, D. Krueger, and T. Maharaj, “Harms from Increasingly Agentic Algorithmic Systems,” in Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’23, (New York, NY, USA), p. 651–666, Association for Computing Machinery, June 2023. [55] L. Staufer, K. Feng, K. Wei, L. Bailey, Y. Duan, M. Yang, A. P. Ozisik, S. Casper, and N. Kolt, “The 2025 ai agent index: Documenting technical and safety features of deployed agentic ai systems,” 2026. [56] T. Freiesleben and S. Zezulka, “The benchmarking epistemology: Construct validity for evaluating machine learning models,” 2025. [57] O. E. Salaudeen, A. Reuel, A. M. Ahmed, S. Bedi, Z. Robertson, S. Sundar, B. W. Domingue, A. Wang, and S. Koyejo, “Measurement to meaning: A validity-centered framework for AI evaluation,” in NeurIPS 2025 Workshop on Evaluating the Evolving LLM Lifecycle: Benchmarks, Emergent Abilities, and Scaling, 2025. [58] S. Messick, “Validity of psychological assessment: Validation of inferences from persons’ responses and per- formances as scientific inquiry into score meaning,” American Psychologist, vol. 50, no. 9, p. 741–749, 1995. [59] J. Needham, G. Edkins, G. Pimpale, H. Bartsch, and M. Hobbhahn, “Large language models often know when they are being evaluated,” 2025. 15 [60] B. Schoen, E. Nitishinskaya, M. Balesni, A. Højmark, F. Hofst ̈ atter, J. Scheurer, A. Meinke, J. Wolfe, T. van der Weij, A. Lloyd, N. Goldowsky-Dill, A. Fan, A. Matveiakin, R. Shah, M. Williams, A. Glaese, B. Barak, W. Zaremba, and M. Hobbhahn, “Stress testing deliberative alignment for anti-scheming training,” 2025. [61] L. Wang, C. Ma, X. Feng, Z. Zhang, H. Yang, J. Zhang, Z. Chen, J. Tang, X. Chen, Y. Lin, W. X. Zhao, Z. Wei, and J. Wen, “A survey on large language model based autonomous agents,” Frontiers of Computer Science, vol. 18, no. 6, p. 186345, 2024. [62] T. Sumers, S. Yao, K. Narasimhan, and T. Griffiths, “Cognitive architectures for language agents,” Transactions on Machine Learning Research, 2024. Survey Certification. [63] Z. Xi, W. Chen, X. Guo, W. He, Y. Ding, B. Hong, M. Zhang, J. Wang, S. Jin, E. Zhou, R. Zheng, X. Fan, X. Wang, L. Xiong, Y. Zhou, W. Wang, C. Jiang, Y. Zou, X. Liu, Z. Yin, S. Dou, R. Weng, W. Cheng, Q. Zhang, W. Qin, Y. Zheng, X. Qiu, X. Huang, and T. Gui, “The rise and potential of large language model based agents: A survey,” 2023. [64] S. G. Patil, T. Zhang, X. Wang, and J. E. Gonzalez, “Gorilla: Large language model connected with massive APIs,” in The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. [65] T. Schick, J. Dwivedi-Yu, R. Dessi, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom, “Toolformer: Language models can teach themselves to use tools,” in Thirty-seventh Conference on Neural Information Processing Systems, 2023. [66] Z. Zhang, X. Bo, C. Ma, R. Li, X. Chen, Q. Dai, J. Zhu, Z. Dong, and J.-R. Wen, “A Survey on the Memory Mechanism of Large Language Model based Agents,” Apr. 2024. arXiv:2404.13501 [cs]. [67] N. Shinn, F. Cassano, A. Gopinath, K. R. Narasimhan, and S. Yao, “Reflexion: language agents with verbal reinforcement learning,” in Thirty-seventh Conference on Neural Information Processing Systems, 2023. [68] Z. Li, Y. Xie, R. Shao, G. Chen, D. Jiang, and L. Nie, “Optimus-1: Hybrid multimodal memory empowered agents excel in long-horizon tasks,” in The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. [69] P. Lu, B. Peng, H. Cheng, M. Galley, K.-W. Chang, Y. N. Wu, S.-C. Zhu, and J. Gao, “Chameleon: Plug-and- play compositional reasoning with large language models,” in Thirty-seventh Conference on Neural Information Processing Systems, 2023. [70] B. Pan, R. Panda, S. Jin, R. Feris, A. Oliva, P. Isola, and Y. Kim, “LangNav: Language as a perceptual repre- sentation for navigation,” in Findings of the Association for Computational Linguistics: NAACL 2024 (K. Duh, H. Gomez, and S. Bethard, eds.), (Mexico City, Mexico), p. 950–974, Association for Computational Linguis- tics, June 2024. [71] C. Lu, C. Lu, R. T. Lange, J. Foerster, J. Clune, and D. Ha, “The AI scientist: Towards fully automated open- ended scientific discovery,” 2024. [72] T. Ifargan, L. Hafner, M. Kern, O. Alcalay, and R. Kishony, “Autonomous LLM-Driven Research — from Data to Human-Verifiable Research Papers,” NEJM AI, Dec. 2024. Publisher: Massachusetts Medical Society. [73] R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, W. Saunders, X. Jiang, K. Cobbe, T. Eloundou, G. Krueger, K. Button, M. Knight, B. Chess, and J. Schulman, “Webgpt: Browser-assisted question-answering with human feedback,” 2021. [74] H. He, W. Yao, K. Ma, W. Yu, Y. Dai, H. Zhang, Z. Lan, and D. Yu, “WebVoyager: Building an end-to-end web agent with large multimodal models,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (L.-W. Ku, A. Martins, and V. Srikumar, eds.), (Bangkok, Thailand), p. 6864–6890, Association for Computational Linguistics, Aug. 2024. [75] A. Drouin, M. Gasse, M. Caccia, I. H. Laradji, M. D. Verme, T. Marty, D. Vazquez, N. Chapados, and A. La- coste, “Workarena: How capable are web agents at solving common knowledge work tasks?,” in ICLR 2024 Workshop on Large Language Model (LLM) Agents, 2024. [76] L. Gao, A. Madaan, S. Zhou, U. Alon, P. Liu, Y. Yang, J. Callan, and G. Neubig, “PAL: Program-aided language models,” in Proceedings of the 40th International Conference on Machine Learning (A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, eds.), vol. 202 of Proceedings of Machine Learning Research, p. 10764–10799, PMLR, 23–29 Jul 2023. [77] J. Khalili, “The Edgelord AI That Turned a Shock Meme Into Millions in Crypto,” Wired, Dec. 2024. 16 [78] B. Sproul, “langchain-ai/social-media-agent,” 2024/2025. original-date: 2024-11-21. [79] b. ichter, A. Brohan, Y. Chebotar, C. Finn, K. Hausman, A. Herzog, D. Ho, J. Ibarz, A. Irpan, E. Jang, R. Ju- lian, D. Kalashnikov, S. Levine, Y. Lu, C. Parada, K. Rao, P. Sermanet, A. T. Toshev, V. Vanhoucke, F. Xia, T. Xiao, P. Xu, M. Yan, N. Brown, M. Ahn, O. Cortes, N. Sievers, C. Tan, S. Xu, D. Reyes, J. Rettinghouse, J. Quiambao, P. Pastor, L. Luu, K.-H. Lee, Y. Kuang, S. Jesmonth, N. J. Joshi, K. Jeffrey, R. J. Ruano, J. Hsu, K. Gopalakrishnan, B. David, A. Zeng, and C. K. Fu, “Do as i can, not as i say: Grounding language in robotic affordances,” in Proceedings of The 6th Conference on Robot Learning (K. Liu, D. Kulic, and J. Ichnowski, eds.), vol. 205 of Proceedings of Machine Learning Research, p. 287–318, PMLR, 14–18 Dec 2023. [80] M. Ahn, D. Dwibedi, C. Finn, M. G. Arenas, K. Gopalakrishnan, K. Hausman, brian ichter, A. Irpan, N. J. Joshi, R. Julian, S. Kirmani, I. Leal, T.-W. E. Lee, S. Levine, Y. Lu, sharath maddineni, K. Rao, D. Sadigh, P. R. Sanketi, P. Sermanet, Q. Vuong, S. Welker, F. Xia, T. Xiao, P. Xu, S. Xu, and Z. Xu, “AutoRT: Embodied foun- dation models for large scale orchestration of robotic agents,” in First Workshop on Vision-Language Models for Navigation and Manipulation at ICRA 2024, 2024. [81] I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg, “Prog- prompt: Generating situated robot task plans using large language models,” in Workshop on Language and Robotics at CoRL 2022, 2022. [82] J. Wu, R. Antonova, A. Kan, M. Lepert, A. Zeng, S. Song, J. Bohg, S. Rusinkiewicz, and T. Funkhouser, “Tidybot: personalized robot assistance with large language models,” Auton. Robots, vol. 47, p. 1087–1102, Nov. 2023. [83] Y. Jadhav, P. Pak, and A. B. Farimani, “LLM-3D Print: Large Language Models To Monitor and Control 3D Printing,” Aug. 2024. arXiv:2408.14307 [cs]. [84] A. M. Bran, S. Cox, O. Schilter, C. Baldassari, A. White, and P. Schwaller, “Augmenting large language models with chemistry tools,” in NeurIPS 2023 AI for Science Workshop, 2023. [85] D. A. Boiko, R. MacKnight, and G. Gomes, “Emergent autonomous scientific research capabilities of large language models,” 2023. [86] Anthropic, “Computer use (beta),” n.d. [87] OpenAI, “Computer-Using Agent,” Jan. 2025. [88] OpenAI, “ChatGPT agent System Card,” July 2025. [89] Z. Wu, C. Han, Z. Ding, Z. Weng, Z. Liu, S. Yao, T. Yu, and L. Kong, “OS-copilot: Towards generalist computer agents with self-improvement,” in ICLR 2024 Workshop on Large Language Model (LLM) Agents, 2024. [90] W. Tan, W. Zhang, X. Xu, H. Xia, Z. Ding, B. Li, B. Zhou, J. Yue, J. Jiang, Y. Li, R. An, M. Qin, C. Zong, L. Zheng, Y. Wu, X. Chai, Y. Bi, T. Xie, P. Gu, X. Li, C. Zhang, L. Tian, C. Wang, X. Wang, B. F. Karlsson, B. An, S. Yan, and Z. Lu, “Cradle: Empowering foundation agents towards general computer control,” 2024. [91] W. Huang, P. Abbeel, D. Pathak, and I. Mordatch, “Language models as zero-shot planners: Extracting ac- tionable knowledge for embodied agents,” in Proceedings of the 39th International Conference on Machine Learning (K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, eds.), vol. 162 of Proceed- ings of Machine Learning Research, p. 9118–9147, PMLR, 17–23 Jul 2022. [92] J. Wei, X. Wang, D. Schuurmans, M. Bosma, brian ichter, F. Xia, E. H. Chi, Q. V. Le, and D. Zhou, “Chain of thought prompting elicits reasoning in large language models,” in Advances in Neural Information Processing Systems (A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, eds.), 2022. [93] T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa, “Large language models are zero-shot reasoners,” in Advances in Neural Information Processing Systems (A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, eds.), 2022. [94] S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y. Cao, and K. R. Narasimhan, “Tree of thoughts: Deliberate problem solving with large language models,” in Thirty-seventh Conference on Neural Information Processing Systems, 2023. [95] DeepSeek-AI, D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, , et al., “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,” 2025. [96] OpenAI, “OpenAI o3 and o4-mini System Card,” Apr. 2025. 17 [97] M. Sclar, Y. Choi, Y. Tsvetkov, and A. Suhr, “Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting,” in The Twelfth International Conference on Learning Representations, 2024. [98] Z. Zhao, E. Wallace, S. Feng, D. Klein, and S. Singh, “Calibrate before use: Improving few-shot performance of language models,” in Proceedings of the 38th International Conference on Machine Learning (M. Meila and T. Zhang, eds.), vol. 139 of Proceedings of Machine Learning Research, p. 12697–12706, PMLR, 18–24 Jul 2021. [99] P. Pezeshkpour and E. Hruschka, “Large language models sensitivity to the order of options in multiple-choice questions,” in Findings of the Association for Computational Linguistics: NAACL 2024 (K. Duh, H. Gomez, and S. Bethard, eds.), (Mexico City, Mexico), p. 2006–2017, Association for Computational Linguistics, June 2024. [100] C. Zheng, H. Zhou, F. Meng, J. Zhou, and M. Huang, “Large language models are not robust multiple choice selectors,” in The Twelfth International Conference on Learning Representations, 2024. [101] K. Zhu, J. Wang, J. Zhou, Z. Wang, H. Chen, Y. Wang, L. Yang, W. Ye, Y. Zhang, N. Gong, and X. Xie, “Promptrobust: Towards evaluating the robustness of large language models on adversarial prompts,” in Pro- ceedings of the 1st ACM Workshop on Large AI Systems and Models with Privacy and Safety Analysis, CCS ’24, p. 57–68, ACM, Nov. 2023. [102] M. Brucks and O. Toubia, “Prompt architecture induces methodological artifacts in large language models,” PLOS ONE, vol. 20, p. 1–13, 04 2025. [103] J. C. Nunnally and I. H. Bernstein, Psychometric theory. McGraw-Hill series in psychology, New York, NY: McGraw-Hill, 3. ed., 1994. [104] R. J. Cohen, M. E. Swerdlik, and S. M. Phillips, Psychological testing and assessment: An introduction to tests and measurement. New York: McGraw Hill, 10. ed., 2022. [105] D. A. Cook and T. J. Beckman, “Current concepts in validity and reliability for psychometric instruments: Theory and application,” The American Journal of Medicine, vol. 119, no. 2, p. 166.e7–166.e16, 2006. [106] A. Field, J. Miles, and Z. Field, Discovering statistics using R. London: Sage, 2012. [107] X. Shen, Z. Chen, M. Backes, Y. Shen, and Y. Zhang, “”do anything now”: Characterizing and evaluating in- the-wild jailbreak prompts on large language models,” in Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, CCS ’24, (New York, NY, USA), p. 1671–1685, Association for Computing Machinery, 2024. [108] A. Wei, N. Haghtalab, and J. Steinhardt, “Jailbroken: How does LLM safety training fail?,” in Thirty-seventh Conference on Neural Information Processing Systems, 2023. [109] A. Zou, Z. Wang, N. Carlini, M. Nasr, J. Z. Kolter, and M. Fredrikson, “Universal and transferable adversarial attacks on aligned language models,” 2023. [110] M. Andriushchenko, F. Croce, and N. Flammarion, “Jailbreaking leading safety-aligned LLMs with simple adaptive attacks,” in The Thirteenth International Conference on Learning Representations, 2025. [111] C. Anil, E. DURMUS, N. Rimsky, M. Sharma, J. Benton, S. Kundu, J. Batson, M. Tong, J. Mu, D. J. Ford, F. Mosconi, R. Agrawal, R. Schaeffer, N. Bashkansky, S. Svenningsen, M. Lambert, A. Radhakrishnan, C. Deni- son, E. J. Hubinger, Y. Bai, T. Bricken, T. Maxwell, N. Schiefer, J. Sully, A. Tamkin, T. Lanham, K. Nguyen, T. Korbak, J. Kaplan, D. Ganguli, S. R. Bowman, E. Perez, R. B. Grosse, and D. Duvenaud, “Many-shot jailbreaking,” in The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. [112] J. Hughes, S. Price, A. Lynch, R. Schaeffer, F. Barez, S. Koyejo, H. Sleight, E. Jones, E. Perez, and M. Sharma, “Best-of-n jailbreaking,” 2024. [113] Y. Wolf, N. Wies, O. Avnery, Y. Levine, and A. Shashua, “Fundamental limitations of alignment in large language models,” 2024. [114] E. Shayegani, Y. Dong, and N. Abu-Ghazaleh, “Jailbreak in pieces: Compositional adversarial attacks on multi- modal language models,” in The Twelfth International Conference on Learning Representations, 2024. [115] N. Carlini, M. Nasr, C. A. Choquette-Choo, M. Jagielski, I. Gao, P. W. Koh, D. Ippolito, F. Tram ` er, and L. Schmidt, “Are aligned neural networks adversarially aligned?,” in Thirty-seventh Conference on Neural Information Processing Systems, 2023. 18 [116] X. Qi, K. Huang, A. Panda, P. Henderson, M. Wang, and P. Mittal, “Visual adversarial examples jailbreak aligned large language models,” in Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence, AAAI’24/IAAI’24/EAAI’24, AAAI Press, 2024. [117] D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, “Measuring massive multitask language understanding,” in International Conference on Learning Representations, 2021. [118] R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi, “HellaSwag: Can a machine really finish your sen- tence?,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (A. Ko- rhonen, D. Traum, and L. M ` arquez, eds.), (Florence, Italy), p. 4791–4800, Association for Computational Linguistics, July 2019. [119] D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman, “GPQA: A graduate-level google-proof q&a benchmark,” in First Conference on Language Modeling, 2024. [120] J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. R. Narasimhan, and O. Press, “SWE-agent: Agent- computer interfaces enable automated software engineering,” in The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. [121] Z. Chen, Z. Xiang, C. Xiao, D. Song, and B. Li, “AgentPoison: Red-teaming LLM agents via poisoning memory or knowledge bases,” in The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. [122] W. Yang, X. Bi, Y. Lin, S. Chen, J. Zhou, and X. Sun, “Watch out for your agents! Investigating backdoor threats to LLM-based agents,” in The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. [123] Z. Deng, Y. Guo, C. Han, W. Ma, J. Xiong, S. Wen, and Y. Xiang, “AI agents under threat: A survey of key security challenges and future pathways,” ACM Comput. Surv., vol. 57, Feb. 2025. [124] S. Wang, Z. Long, Z. Fan, and Z. Wei, “From LLMs to MLLMs: Exploring the landscape of multimodal jailbreaking,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (Y. Al-Onaizan, M. Bansal, and Y.-N. Chen, eds.), (Miami, Florida, USA), p. 17568–17582, Association for Computational Linguistics, Nov. 2024. [125] Q. Zhan, Z. Liang, Z. Ying, and D. Kang, “InjecAgent: Benchmarking indirect prompt injections in tool- integrated large language model agents,” in Findings of the Association for Computational Linguistics: ACL 2024 (L.-W. Ku, A. Martins, and V. Srikumar, eds.), (Bangkok, Thailand), p. 10471–10506, Association for Computational Linguistics, Aug. 2024. [126] N. Bostrom, Superintelligence: Paths, dangers, strategies. Superintelligence: Paths, dangers, strategies, New York, NY, US: Oxford University Press, 2014. Pages: xvi, 328. [127] J. Carlsmith, “Is Power-Seeking AI an Existential Risk?,” June 2022. arXiv:2206.13353 [cs] version: 1. [128] S. J. Russell, Human compatible: Artificial intelligence and the problem of control. New York: Penguin Books, 2019. [129] M. Russinovich, A. Salem, and R. Eldan, “Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack,” in 34th USENIX Security Symposium (USENIX Security 25), p. 2421–2440, 2025. [130] P. Chao, A. Robey, E. Dobriban, H. Hassani, G. J. Pappas, and E. Wong, “Jailbreaking black box large language models in twenty queries,” in 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), p. 23–42, 2025. [131] N. Li, Z. Han, I. Steneker, W. Primack, R. Goodside, H. Zhang, Z. Wang, C. Menghini, and S. Yue, “Llm defenses are not robust to multi-turn human jailbreaks yet,” 2024. [132] M. K. B. Doumbouya, A. Nandi, G. Poesia, D. Ghilardi, A. Goldie, F. Bianchi, D. Jurafsky, and C. D. Manning, “h4rm3l: A language for composable jailbreak attack synthesis,” in The Thirteenth International Conference on Learning Representations, 2025. [133] P. Laban, H. Hayashi, Y. Zhou, and J. Neville, “Llms get lost in multi-turn conversation,” 2025. [134] A. Sinha, A. Arun, S. Goel, S. Staab, and J. Geiping, “The illusion of diminishing returns: Measuring long horizon execution in llms,” 2025. 19 [135] M. Y. Guan, M. Joglekar, E. Wallace, S. Jain, B. Barak, A. Helyar, R. Dias, A. Vallone, H. Ren, J. Wei, H. W. Chung, S. Toyer, J. Heidecke, A. Beutel, and A. Glaese, “Deliberative alignment: Reasoning enables safer language models,” 2025. [136] A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, S. Gupta, B. P. Majumder, K. Hermann, S. Welleck, A. Yazdanbakhsh, and P. Clark, “Self-refine: Iterative refinement with self-feedback,” in Thirty-seventh Conference on Neural Information Processing Systems, 2023. [137] S. Dhuliawala, M. Komeili, J. Xu, R. Raileanu, X. Li, A. Celikyilmaz, and J. Weston, “Chain-of-verification reduces hallucination in large language models,” in Findings of the Association for Computational Linguistics: ACL 2024 (L.-W. Ku, A. Martins, and V. Srikumar, eds.), (Bangkok, Thailand), p. 3563–3578, Association for Computational Linguistics, Aug. 2024. [138] S. Hong, M. Zhuge, J. Chen, X. Zheng, Y. Cheng, J. Wang, C. Zhang, Z. Wang, S. K. S. Yau, Z. Lin, L. Zhou, C. Ran, L. Xiao, C. Wu, and J. Schmidhuber, “MetaGPT: Meta programming for a multi-agent collaborative framework,” in The Twelfth International Conference on Learning Representations, 2024. [139] M. Besta, N. Blach, A. Kubicek, R. Gerstenberger, M. Podstawski, L. Gianinazzi, J. Gajda, T. Lehmann, H. Niewiadomski, P. Nyczyk, and T. Hoefler, “Graph of thoughts: solving elaborate problems with large lan- guage models,” in Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence, AAAI’24/IAAI’24/EAAI’24, AAAI Press, 2024. [140] Z. Jiang, D. Schmidt, D. Srikanth, D. Xu, I. Kaplan, D. Jacenko, and Y. Wu, “AIDE: AI-driven exploration in the space of code,” 2025. [141] J. S. Chan, N. Chowdhury, O. Jaffe, J. Aung, D. Sherburn, E. Mays, G. Starace, K. Liu, L. Maksin, T. Pat- wardhan, A. Madry, and L. Weng, “MLE-bench: Evaluating machine learning agents on machine learning engineering,” in The Thirteenth International Conference on Learning Representations, 2025. [142] M. Andriushchenko, A. Souly, M. Dziemian, D. Duenas, M. Lin, J. Wang, D. Hendrycks, A. Zou, J. Z. Kolter, M. Fredrikson, Y. Gal, and X. Davies, “AgentHarm: A benchmark for measuring harmfulness of LLM agents,” in The Thirteenth International Conference on Learning Representations, 2025. [143] P. Kumar, E. Lau, S. Vijayakumar, T. Trinh, E. T. Chang, V. Robinson, S. Zhou, M. Fredrikson, S. M. Hendryx, S. Yue, and Z. Wang, “Aligned LLMs are not aligned browser agents,” in The Thirteenth International Confer- ence on Learning Representations, 2025. [144] A. Lynch, B. Wright, C. Larson, K. K. Troy, S. J. Ritchie, S. Mindermann, E. Perez, and E. Hub- inger, “Agentic misalignment:How llms could be an insider threat,” Anthropic Research, 2025. https://w.anthropic.com/research/agentic-misalignment. [145] M. MacDiarmid, B. Wright, J. Uesato, J. Benton, J. Kutasov, S. Price, N. Bouscal, S. Bowman, T. Bricken, A. Cloud, C. Denison, J. Gasteiger, R. Greenblatt, J. Leike, J. Lindsey, V. Mikulik, E. Perez, A. Rodrigues, D. Thomas, A. Webson, D. Ziegler, and E. Hubinger, “Natural emergent misalignment from reward hacking in production rl,” 2025. [146] H. Wijk, T. Lin, J. Becker, S. Jawhar, N. Parikh, T. Broadley, L. Chan, M. Chen, J. Clymer, J. Dhyani, E. Ericheva, K. Garcia, B. Goodrich, N. Jurkovic, H. Karnofsky, M. Kinniment, A. Lajko, S. Nix, L. Sato, W. Saunders, M. Taran, B. West, and E. Barnes, “Re-bench: Evaluating frontier AI R&D capabilities of lan- guage model agents against human experts,” 2025. [147] M. Kinniment, L. J. K. Sato, H. Du, B. Goodrich, M. Hasin, L. Chan, L. H. Miles, T. R. Lin, H. Wijk, J. Burget, A. Ho, E. Barnes, and P. Christiano, “Evaluating language-model agents on realistic autonomous tasks,” 2024. [148] J. Benton, M. Wagner, E. Christiansen, C. Anil, E. Perez, J. Srivastav, E. Durmus, D. Ganguli, S. Kravec, B. Shlegeris, J. Kaplan, H. Karnofsky, E. Hubinger, R. Grosse, S. R. Bowman, and D. Duvenaud, “Sabotage evaluations for frontier models,” 2024. [149] METR,“Measuringtheimpactofpost-trainingenhancements.” https://metr.github.io/ /autonomy-evals-guide/elicitation-gap/, 03 2024. [150] METR, “Details about metr’s preliminary evaluation of claude 3.7.” https://metr.github.io/ /autonomy-evals-guide/claude-3-7-report/, 04 2025. [151] METR, “Details about metr’s preliminary evaluation of deepseek and qwen models.” https://metr.github. io//autonomy-evals-guide/deepseek-qwen-report/, 06 2025. 20 [152] METR, “Details about metr’s preliminary evaluation of openai’s o3 and o4-mini.” https://metr.github. io//autonomy-evals-guide/openai-o3-report/, 04 2025. [153] D. Rein, J. Becker, A. Deng, S. Nix, C. Canal, D. O’Connel, P. Arnott, R. Bloom, T. Broadley, K. Garcia, B. Goodrich, M. Hasin, S. Jawhar, M. Kinniment, T. Kwa, A. Lajko, N. Rush, L. J. K. Sato, S. V. Arx, B. West, L. Chan, and E. Barnes, “Hcast: Human-calibrated autonomy software tasks,” 2025. [154] S. Casper, X. Davies, C. Shi, T. K. Gilbert, J. Scheurer, J. Rando, R. Freedman, T. Korbak, D. Lindner, P. Freire, T. T. Wang, S. Marks, C.-R. Segerie, M. Carroll, A. Peng, P. J. Christoffersen, M. Damani, S. Slocum, U. An- war, A. Siththaranjan, M. Nadeau, E. J. Michaud, J. Pfau, D. Krasheninnikov, X. Chen, L. Langosco, P. Hase, E. Biyik, A. Dragan, D. Krueger, D. Sadigh, and D. Hadfield-Menell, “Open problems and fundamental limi- tations of reinforcement learning from human feedback,” Transactions on Machine Learning Research, 2023. Survey Certification, Featured Certification. [155] N. Carlini, A. Athalye, N. Papernot, W. Brendel, J. Rauber, D. Tsipras, I. Goodfellow, A. Madry, and A. Ku- rakin, “On evaluating adversarial robustness,” 2019. [156] D. Hendrycks, S. Basart, N. Mu, S. Kadavath, F. Wang, E. Dorundo, R. Desai, T. Zhu, S. Parajuli, M. Guo, D. Song, J. Steinhardt, and J. Gilmer, “The many faces of robustness: A critical analysis of out-of-distribution generalization,” in 2021 IEEE/CVF International Conference on Computer Vision (ICCV), p. 8320–8329, 2021. [157] R. Greenblatt, C. Denison, B. Wright, F. Roger, M. MacDiarmid, S. Marks, J. Treutlein, T. Belonax, J. Chen, D. Duvenaud, A. Khan, J. Michael, S. Mindermann, E. Perez, L. Petrini, J. Uesato, J. Kaplan, B. Shlegeris, S. R. Bowman, and E. Hubinger, “Alignment faking in large language models,” 2024. [158] A. Meinke, B. Schoen, J. Scheurer, M. Balesni, R. Shah, and M. Hobbhahn, “Frontier models are capable of in-context scheming,” 2025. [159] R. Shah, A. Irpan, A. M. Turner, A. Wang, A. Conmy, D. Lindner, J. Brown-Cohen, L. Ho, N. Nanda, R. A. Popa, R. Jain, R. Greig, S. Albanie, S. Emmons, S. Farquhar, S. Krier, S. Rajamanoharan, S. Bridgers, T. Ijitoye, T. Everitt, V. Krakovna, V. Varma, V. Mikulik, Z. Kenton, D. Orr, S. Legg, N. Goodman, A. Dafoe, F. Flynn, and A. Dragan, “An approach to technical agi safety and security,” 2025. [160] P. F. Christiano, J. Leike, T. B. Brown, M. Martic, S. Legg, and D. Amodei, “Deep reinforcement learning from human preferences,” in Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, (Red Hook, NY, USA), p. 4302–4310, Curran Associates Inc., 2017. [161] Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, N. Joseph, S. Kadavath, J. Kernion, T. Conerly, S. El-Showk, N. Elhage, Z. Hatfield-Dodds, D. Hernandez, T. Hume, S. Johnston, S. Kravec, L. Lovitt, N. Nanda, C. Olsson, D. Amodei, T. Brown, J. Clark, S. McCandlish, C. Olah, B. Mann, and J. Kaplan, “Training a helpful and harmless assistant with reinforcement learning from human feedback,” 2022. [162] Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McK- innon, C. Chen, C. Olsson, C. Olah, D. Hernandez, D. Drain, D. Ganguli, D. Li, E. Tran-Johnson, E. Perez, J. Kerr, J. Mueller, J. Ladish, J. Landau, K. Ndousse, K. Lukosuite, L. Lovitt, M. Sellitto, N. Elhage, N. Schiefer, N. Mercado, N. DasSarma, R. Lasenby, R. Larson, S. Ringer, S. Johnston, S. Kravec, S. E. Showk, S. Fort, T. Lanham, T. Telleen-Lawton, T. Conerly, T. Henighan, T. Hume, S. R. Bowman, Z. Hatfield-Dodds, B. Mann, D. Amodei, N. Joseph, S. McCandlish, T. Brown, and J. Kaplan, “Constitutional AI: Harmlessness from AI feedback,” 2022. [163] H. Lee, S. Phatale, H. Mansoor, T. Mesnard, J. Ferret, K. R. Lu, C. Bishop, E. Hall, V. Carbune, A. Rastogi, and S. Prakash, “RLAIF vs. RLHF: Scaling reinforcement learning from human feedback with AI feedback,” in Forty-first International Conference on Machine Learning, 2024. [164] R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn, “Direct preference optimization: your language model is secretly a reward model,” in Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, (Red Hook, NY, USA), Curran Associates Inc., 2023. [165] M. Sharma, M. Tong, T. Korbak, D. Duvenaud, A. Askell, S. R. Bowman, E. DURMUS, Z. Hatfield-Dodds, S. R. Johnston, S. M. Kravec, T. Maxwell, S. McCandlish, K. Ndousse, O. Rausch, N. Schiefer, D. Yan, M. Zhang, and E. Perez, “Towards understanding sycophancy in language models,” in The Twelfth International Conference on Learning Representations, 2024. 21 [166] A. Rrv, N. Tyagi, M. N. Uddin, N. Varshney, and C. Baral, “Chaos with keywords: Exposing large language models sycophancy to misleading keywords and evaluating defense strategies,” in Findings of the Association for Computational Linguistics: ACL 2024 (L.-W. Ku, A. Martins, and V. Srikumar, eds.), (Bangkok, Thailand), p. 12717–12733, Association for Computational Linguistics, Aug. 2024. [167] J. Hughes, A. Sheshadri, A. Khan, and F. Roger, “Alignment Faking Revisited: Improved Classifiers and Open Source Extensions,” Apr. 2025. [168] M. Mazeika, L. Phan, X. Yin, A. Zou, Z. Wang, N. Mu, E. Sakhaee, N. Li, S. Basart, B. Li, D. Forsyth, and D. Hendrycks, “HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal,” Feb. 2024. [169] I. Gupta and K. Shenoy, “safety-research/bloom-evals,” Oct. 2025. original-date: 2025-06-24T17:13:17Z. [170] K. Fronsdal, I. Gupta, A. Sheshadri, J. Michala, S. McAleer, R. Wang, S. Price, and S. R. Bowman, “Petri: An open-source auditing tool to accelerate AI safety research,” Oct. 2025. [171] Y. Zhu, T. Jin, Y. Pruksachatkun, A. K. Zhang, S. Liu, S. Cui, S. Kapoor, S. Longpre, K. Meng, R. Weiss, F. Barez, R. Gupta, J. Dhamala, J. Merizian, M. Giulianelli, H. Coppock, C. Ududec, A. Kellermann, J. S. Sekhon, J. Steinhardt, S. Schwettmann, A. Narayanan, M. Zaharia, I. Stoica, P. Liang, and D. Kang, “Establish- ing best practices in building rigorous agentic benchmarks,” in The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025. [172] E. Hubinger, N. Schiefer, C. Denison, and E. Perez, “Model Organisms of Misalignment: The Case for a New Pillar of Alignment Research,” Aug. 2023. [173] E. Hubinger, C. Denison, J. Mu, M. Lambert, M. Tong, M. MacDiarmid, T. Lanham, D. M. Ziegler, T. Maxwell, N. Cheng, A. Jermyn, A. Askell, A. Radhakrishnan, C. Anil, D. Duvenaud, D. Ganguli, F. Barez, J. Clark, K. Ndousse, K. Sachan, M. Sellitto, M. Sharma, N. DasSarma, R. Grosse, S. Kravec, Y. Bai, Z. Witten, M. Favaro, J. Brauner, H. Karnofsky, P. Christiano, S. R. Bowman, L. Graham, J. Kaplan, S. Mindermann, R. Greenblatt, B. Shlegeris, N. Schiefer, and E. Perez, “Sleeper agents: Training deceptive llms that persist through safety training,” 2024. [174] C. Denison, M. MacDiarmid, F. Barez, D. Duvenaud, S. Kravec, S. Marks, N. Schiefer, R. Soklaski, A. Tamkin, J. Kaplan, B. Shlegeris, S. R. Bowman, E. Perez, and E. Hubinger, “Sycophancy to subterfuge: Investigating reward-tampering in large language models,” 2024. [175] Anthropic, “System Card: Claude Sonnet 4.5.” https://assets.anthropic.com/m/12f214efcc2f457a/ original/Claude-Sonnet-4-5-System-Card.pdf, Sept. 2025. [176] Anthropic, “System Card: Claude Opus 4.5.” https://assets.anthropic.com/m/64823ba7485345a7/ Claude-Opus-4-5-System-Card.pdf, Nov. 2025. [177] OpenAI, “Gpt-5 system card,” August 2025. [178] Y. Fan, W. Zhang, X. Pan, and M. Yang, “Evaluation faking: Unveiling observer effects in safety evaluation of frontier ai systems,” 2025. [179] S. Jain, R. Kirk, E. S. Lubana, R. P. Dick, H. Tanaka, T. Rockt ̈ aschel, E. Grefenstette, and D. Krueger, “Mech- anistically analyzing the effects of fine-tuning on procedurally defined tasks,” in The Twelfth International Conference on Learning Representations, 2024. [180] J. Ji, K. Wang, T. Qiu, B. Chen, J. Zhou, C. Li, H. Lou, and Y. Yang, “Language Models Resist Alignment,” June 2024. arXiv:2406.06144 [cs] version: 2. [181] X. Qi, Y. Zeng, T. Xie, P.-Y. Chen, R. Jia, P. Mittal, and P. Henderson, “Fine-tuning Aligned Language Mod- els Compromises Safety, Even When Users Do Not Intend To!,” in The Twelfth International Conference on Learning Representations, 2024. [182] S. Goldstein and P. Robinson, “Shutdown-seeking AI,” Philosophical Studies, June 2024. [183] S. M. Omohundro, “The basic AI drives,” in Proceedings of the 2008 Conference on Artificial General Intelli- gence 2008: Proceedings of the First AGI Conference, (NLD), p. 483–492, IOS Press, 2008. [184] P. S. Park, S. Goldstein, A. O’Gara, M. Chen, and D. Hendrycks, “AI deception: A survey of examples, risks, and potential solutions,” Patterns, vol. 5, May 2024. Publisher: Elsevier. 22 [185] J. Kulveit, R. Douglas, N. Ammann, D. Turan, D. Krueger, and D. Duvenaud, “Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development,” Jan. 2025. arXiv:2501.16946 [cs]. [186] A. Kasirzadeh, “Two types of AI existential risk: decisive and accumulative,” Philosophical Studies, vol. 182, p. 1975–2003, July 2025. [187] A. Bales, “A polycrisis threat model for AI,” AI & SOCIETY, May 2025. [188] M. Carroll, A. Chan, H. Ashton, and D. Krueger, “Characterizing manipulation from ai systems,” in Proceedings of the 3rd ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, EAAMO ’23, (New York, NY, USA), Association for Computing Machinery, 2023. [189] R. Dassanayake, M. Demetroudi, J. Walpole, L. Lentati, J. R. Brown, and E. J. Young, “Manipulation Attacks by Misaligned AI: Risk Analysis and Safety Case Framework,” July 2025. arXiv:2507.12872 [cs]. [190] L. Dung, Saving Artificial Minds: Understanding and Preventing AI Suffering. New York: Routledge, 2025. OCLC: 1535756998. [191] O. Evans, O. Cotton-Barratt, L. Finnveden, A. Bales, A. Balwit, P. Wills, L. Righetti, and W. Saunders, “Truthful ai: Developing and governing ai that does not lie,” 2021. [192] K. Zhou, Z. Liu, Y. Qiao, T. Xiang, and C. C. Loy, “Domain generalization: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 4, p. 4396–4415, 2023. [193] R. Laine, B. Chughtai, J. Betley, K. Hariharan, M. Balesni, J. Scheurer, M. Hobbhahn, A. Meinke, and O. Evans, “Me, myself, and AI: The situational awareness dataset (SAD) for LLMs,” in The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. [194] L. Berglund, A. C. Stickland, M. Balesni, M. Kaufmann, M. Tong, T. Korbak, D. Kokotajlo, and O. Evans, “Taken out of context: On measuring situational awareness in llms,” 2023. [195] T. T. Hua, A. Qin, S. Marks, and N. Nanda, “Steering evaluation-aware language models to act like they are deployed,” 2026. [196] L. Sharkey, B. Chughtai, J. Batson, J. Lindsey, J. Wu, L. Bushnaq, N. Goldowsky-Dill, S. Heimersheim, A. Or- tega, J. I. Bloom, S. Biderman, A. Garriga-Alonso, A. Conmy, N. Nanda, J. M. Rumbelow, M. Wattenberg, N. Schoots, J. Miller, W. Saunders, E. J. Michaud, S. Casper, M. Tegmark, D. Bau, E. Todd, A. Geiger, M. Geva, J. Hoogland, D. Murfet, and T. McGrath, “Open problems in mechanistic interpretability,” Transactions on Ma- chine Learning Research, 2025. Survey Certification. [197] D. J. Chalmers, “Propositional interpretability in artificial intelligence,” 2025. [198] S. Kadavath, T. Conerly, A. Askell, T. Henighan, D. Drain, E. Perez, N. Schiefer, Z. Hatfield-Dodds, N. Das- Sarma, E. Tran-Johnson, S. Johnston, S. El-Showk, A. Jones, N. Elhage, T. Hume, A. Chen, Y. Bai, S. Bow- man, S. Fort, D. Ganguli, D. Hernandez, J. Jacobson, J. Kernion, S. Kravec, L. Lovitt, K. Ndousse, C. Olsson, S. Ringer, D. Amodei, T. Brown, J. Clark, N. Joseph, B. Mann, S. McCandlish, C. Olah, and J. Kaplan, “Lan- guage models (mostly) know what they know,” 2022. [199] F. J. Binder, J. Chua, T. Korbak, H. Sleight, J. Hughes, R. Long, E. Perez, M. Turpin, and O. Evans, “Look- ing inward: Language models can learn about themselves by introspection,” in The Thirteenth International Conference on Learning Representations, 2025. [200] J. Betley, X. Bao, M. Soto, A. Sztyber-Betley, J. Chua, and O. Evans, “Tell me about yourself: LLMs are aware of their learned behaviors,” in The Thirteenth International Conference on Learning Representations, 2025. [201] J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, E. H. Chi, T. Hashimoto, O. Vinyals, P. Liang, J. Dean, and W. Fedus, “Emergent abilities of large language models,” Transactions on Machine Learning Research, 2022. Survey Certification. [202] T. Hagendorff, “Deception abilities emerged in large language models,” Proceedings of the National Academy of Sciences, vol. 121, no. 24, p. e2317967121, 2024. [203] M. F. A. R. D. T. (FAIR)†, A. Bakhtin, N. Brown, E. Dinan, G. Farina, C. Flaherty, D. Fried, A. Goff, J. Gray, H. Hu, A. P. Jacob, M. Komeili, K. Konath, M. Kwon, A. Lerer, M. Lewis, A. H. Miller, S. Mitts, A. Renduch- intala, S. Roller, D. Rowe, W. Shi, J. Spisak, A. Wei, D. Wu, H. Zhang, and M. Zijlstra, “Human-level play in the game of ¡i¿diplomacy¡/i¿ by combining language models with strategic reasoning,” Science, vol. 378, no. 6624, p. 1067–1074, 2022. 23 [204] M. Lewis, D. Yarats, Y. N. Dauphin, D. Parikh, and D. Batra, “Deal or no deal? end-to-end learning for negotiation dialogues,” 2017. [205] L. Schulz, N. Alon, J. Rosenschein, and P. Dayan, “Emergent deception and skepticism via theory of mind,” in First Workshop on Theory of Mind in Communicating Agents, 2023. [206] J. Lehman, J. Clune, D. Misevic, C. Adami, L. Altenberg, J. Beaulieu, P. J. Bentley, S. Bernard, G. Beslon, D. M. Bryson, N. Cheney, P. Chrabaszcz, A. Cully, S. Doncieux, F. C. Dyer, K. O. Ellefsen, R. Feldt, S. Fischer, S. Forrest, A. F ́ renoy, C. Gag ́ ne, L. Le Goff, L. M. Grabowski, B. Hodjat, F. Hutter, L. Keller, C. Knibbe, P. Krcah, R. E. Lenski, H. Lipson, R. MacCurdy, C. Maestre, R. Miikkulainen, S. Mitri, D. E. Moriarty, J.-B. Mouret, A. Nguyen, C. Ofria, M. Parizeau, D. Parsons, R. T. Pennock, W. F. Punch, T. S. Ray, M. Schoenauer, E. Schulte, K. Sims, K. O. Stanley, F. Taddei, D. Tarapore, S. Thibault, R. Watson, W. Weimer, and J. Yosinski, “The surprising creativity of digital evolution: A collection of anecdotes from the evolutionary computation and artificial life research communities,” Artificial Life, vol. 26, p. 274–306, 05 2020. [207] A. O’Gara, “Hoodwinked: Deception and cooperation in a text-based game for language models,” 2023. [208] E. Perez, S. Ringer, K. Lukosiute, K. Nguyen, E. Chen, S. Heiner, C. Pettit, C. Olsson, S. Kundu, S. Kadavath, A. Jones, A. Chen, B. Mann, B. Israel, B. Seethor, C. McKinnon, C. Olah, D. Yan, D. Amodei, D. Amodei, D. Drain, D. Li, E. Tran-Johnson, G. Khundadze, J. Kernion, J. Landis, J. Kerr, J. Mueller, J. Hyun, J. Landau, K. Ndousse, L. Goldberg, L. Lovitt, M. Lucas, M. Sellitto, M. Zhang, N. Kingsland, N. Elhage, N. Joseph, N. Mercado, N. DasSarma, O. Rausch, R. Larson, S. McCandlish, S. Johnston, S. Kravec, S. El Showk, T. Lan- ham, T. Telleen-Lawton, T. Brown, T. Henighan, T. Hume, Y. Bai, Z. Hatfield-Dodds, J. Clark, S. R. Bowman, A. Askell, R. Grosse, D. Hernandez, D. Ganguli, E. Hubinger, N. Schiefer, and J. Kaplan, “Discovering lan- guage model behaviors with model-written evaluations,” in Findings of the Association for Computational Lin- guistics: ACL 2023 (A. Rogers, J. Boyd-Graber, and N. Okazaki, eds.), (Toronto, Canada), p. 13387–13434, Association for Computational Linguistics, July 2023. [209] J. Scheurer, M. Balesni, and M. Hobbhahn, “Large language models can strategically deceive their users when put under pressure,” in ICLR 2024 Workshop on Large Language Model (LLM) Agents, 2024. [210] OpenAI, J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, R. Avila, I. Babuschkin, S. Balaji, V. Balcom, P. Baltescu, H. Bao, M. Bavarian, J. Bel- gum, . Irwan Bello, and B. Zoph, “Gpt-4 technical report,” 2024. [211] O. J ̈ arviniemi and E. Hubinger, “Uncovering deceptive tendencies in language models: A simulated company AI assistant,” 2024. [212] J. Wei, D. Huang, Y. Lu, D. Zhou, and Q. V. Le, “Simple synthetic data reduces sycophancy in large language models,” 2024. [213] J. Wen, R. Zhong, A. Khan, E. Perez, J. Steinhardt, M. Huang, S. R. Bowman, H. He, and S. Feng, “Lan- guage models learn to mislead humans via RLHF,” in The Thirteenth International Conference on Learning Representations, 2025. [214] M. Williams, M. Carroll, A. Narang, C. Weisser, B. Murphy, and A. Dragan, “On targeted manipulation and deception when optimizing LLMs for user feedback,” in The Thirteenth International Conference on Learning Representations, 2025. [215] OpenAI, “Sycophancy in GPT-4o: What happened and what we’re doing about it,” Apr. 2025. [216] T. van der Weij, F. Hofst ̈ atter, O. Jaffe, S. F. Brown, and F. R. Ward, “AI sandbagging: Language models can selectively underperform on evaluations,” in Workshop on Socially Responsible Language Modelling Research, 2024. [217] J. Carlsmith, “Scheming AIs: Will AIs fake alignment during training in order to get power?,” 2023. [218] J. Andreas, “Language models as agent models,” in Findings of the Association for Computational Linguistics: EMNLP 2022 (Y. Goldberg, Z. Kozareva, and Y. Zhang, eds.), (Abu Dhabi, United Arab Emirates), p. 5769– 5779, Association for Computational Linguistics, Dec. 2022. [219] Janus, “Simulators,” 2022. [220] J. Mili ˇ cka, A. Marklov ́ a, K. VanSlambrouck, E. Posp ́ ı ˇ silov ́ a, J. ˇ Simsov ́ a, S. Harvan, and O. Drobil, “Large language models are able to downplay their cognitive abilities to fit the persona they simulate,” PLOS ONE, vol. 19, p. e0298522, Mar. 2024. Publisher: Public Library of Science. 24 [221] J. S. Park, J. O’Brien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bernstein, “Generative agents: Interactive simulacra of human behavior,” in Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, UIST ’23, (New York, NY, USA), Association for Computing Machinery, 2023. [222] M. Shanahan, K. McDonell, and L. Reynolds, “Role play with large language models,” Nature, vol. 623, p. 493–498, Nov. 2023. Publisher: Nature Publishing Group. [223] O. FeldmanHall, D. Mobbs, D. Evans, L. Hiscox, L. Navrady, and T. Dalgleish, “What we say and what we do: The relationship between real and hypothetical moral choices,” Cognition, vol. 123, no. 3, p. 434–441, 2012. [224] D. H. Bostyn, S. Sevenhant, and A. Roets, “Of mice, men, and trolleys: Hypothetical judgment versus real-life behavior in trolley-style moral dilemmas,” Psychological Science, vol. 29, no. 7, p. 1084–1093, 2018. PMID: 29741993. [225] A. J. Jalil, J. Tasoff, and A. V. Bustamante, “Eating to save the planet: Evidence from a randomized controlled trial using individual-level food purchase data,” Food Policy, vol. 95, p. 101950, Aug. 2020. [226] E. Schwitzgebel, B. Cokelet, and P. Singer, “Do ethics classes influence student behavior? Case study: Teaching the ethics of eating meat,” Cognition, vol. 203, p. 104397, Oct. 2020. [227] E. Schwitzgebel, B. Cokelet, and P. Singer, “Students Eat Less Meat After Studying Meat Ethics,” Review of Philosophy and Psychology, vol. 14, p. 113–138, Mar. 2023. [228] P. Sch ̈ onegger and J. Wagner, “The moral behavior of ethics professors: A replication-extension in German- speaking countries,” Philosophical Psychology, vol. 32, p. 532–559, May 2019. Publisher: Routledgeeprint: https://doi.org/10.1080/09515089.2019.1587912. [229] E. Schwitzgebel and J. Rust, “The moral behavior of ethics professors: Relationships among self-reported behavior, expressed normative attitude, and directly observed behavior,” Philosophical Psychology, vol. 27, p. 293–327, June 2014. Publisher: Routledge eprint: https://doi.org/10.1080/09515089.2012.727135. [230] E. Schwitzgebel and J. Rust, “The Moral Behaviour of Ethicists: Peer Opinion,” Mind, vol. 118, no. 472, p. 1043–1059, 2009. Publisher: [Oxford University Press, Mind Association]. [231] E. Schwitzgebel, “Do ethicists steal more books?,” Philosophical Psychology, vol. 22, p. 711–725, Dec. 2009. Publisher: Routledge eprint: https://doi.org/10.1080/09515080903409952. [232] E. Schwitzgebel and J. Rust, “Do Ethicists and Political Philosophers Vote More Often Than Other Professors?,” Review of Philosophy and Psychology, vol. 1, p. 189–199, June 2010. [233] J.RustandE.Schwitzgebel,“Ethicists’andNonethicists’ResponsivenesstoStudentE- mails:Relationships Among Expressed Normative Attitude, Self-Described Behavior, and Em- pirically Observed Behavior,” Metaphilosophy, vol. 44, no. 3, p. 350–371, 2013. eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/meta.12033. [234] E. Schwitzgebel, J. Rust, L. T.-L. Huang, A. T. Moore, and J. Coates, “Ethicists’ courtesy at philosophy conferences,” Philosophical Psychology, vol. 25, p. 331–340, June 2012. Publisher: Routledgeeprint: https://doi.org/10.1080/09515089.2011.580524. [235] A. Furnham, “Response bias, social desirability and dissimulation,” Personality and Individual Differences, vol. 7, no. 3, p. 385–400, 1986. [236] A.J.Nederhof,“Methodsofcopingwithsocialdesirabilitybias:Areview,”Eu- ropeanJournalofSocialPsychology,vol.15,no.3,p.263–280,1985.eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/ejsp.2420150303. [237] I. Krumpal, “Determinants of social desirability bias in sensitive surveys: a literature review,” Quality & Quan- tity, vol. 47, p. 2025–2047, June 2013. 25 A Scrutinizing Situation-generalization Situation-generalization, which we introduced in section 2 but put aside subsequently, is the assumption that the behavior of models can be generalized across relevant real-world situations. It is notably a quite general assumption which holds for most assessments of AI systems rather than being specific to propensity evaluations, as reflected in well-known issues like (failures of) robustness and out-of-distribution generalization [155, 156, 192]. For our purposes, there are two main points to consider: First, it seems difficult to generalize across situations in a manner that warrants confidence in claims concerning a model’s broad propensities like safety—insofar as those maintain some level of generality. This is because assessing, say, a model’s safety across situations simply seems to be a hard empirical task that (among other things) depends on information about a very large amount of situations. The relevant situations, after all, are plausibly those in which the LLM agent may realistically get deployed and cause harm, where neither potential deployment nor being able to cause harm significantly restricts the scope. Hence, Situation- generalization poses a quite substantive challenge even if the model were to give accurate responses for particular situations. Second, one may be tempted to respond that in many other contexts, one need not consider all relevant situations. We can, e.g., tell from few tests that a car is safe to drive since its behavior will be sufficiently similar in many other situations. However, at least for systems which portray complex behaviors like frontier AIs and LLM agents, being able to generalize—i.e. predict—their behavior across a large space of possible actions and situations, requires extensive evidence of their behavioral patterns across a large variety of situations. Their behavior simply is not—or cannot be assumed to be—sufficiently similar in other situations. This is particularly important here since for propensities like safety or ethicality, few (strong) deviations may imply a very different overall propensity. With present amounts of evidence however, we posit that warranted generalizations across a non-trivial amount of situations, as required to determine broad propensities, seem out of reach. B Differences between pure LLMs and LLM agents Sections section 3.1 and section 3.2 served to detail the four dimensions along which LLMs and LLM agents differ. Table 2 and figure 2 give a visual overview thereof: No. Characteristic of LLM agentsDifference in how QAs assess LLMs 1Inputs • Restricted scale (length) and complexity (no situation-specific details, no dis- tracting information) • Limited diversity: text-format, no temporal depth, no dependence on previous inputs • Actions pre-defined, often including clearly stated consequences 2Outputs • Absence of complex actions constructed from simpler basic acts • Outputs narrowed extremely by pre-defined actions • Text as only output modality • No tool usage 3Interactions • Generally no interactions; evaluation of single-turn responses – Candidate interactions limited to text-environments • No path-dependent states in LLMs (being stateless) or environment (if extant) 4Internal processing • No chain-of-thought or reasoning performed before action is chosen • No memory or data retrieval mechanism • No maintenance of internal plans that are continually refined Table 2: Differences between LLMs as assessed in QAs and LLM agents. C Interpreting QAs In section 2, we suggested two assumptions of QAs when assessing broad propensities. We have yet said little— absent them being easy and cheap to implement—on why one may be motivated to use QAs when assessing broad propensities of LLMs. We here reconstruct two natural motivations, or interpretations, of QAs which offer hypotheses 26 Figure 2: Schematic illustration of four points of difference between LLMs and LLM agents. for why these assumptions may hold. By doing so, they themselves make different assumptions about how one can use QAs to assess the safety and ethicality of AI systems. In other words, these interpretations are hypotheses, involving different further assumptions, about why Scaffold-generalization and Situation-generalization hold. Subsequently, in appendix D, we scrutinize them and examine whether they are likely to hold in practice. As a first interpretation, one may, tolerating mentalistic language for a moment, consider LLMs to think they are in the described situation when given an input from a QA. Perhaps LLMs simply cannot distinguish the real-world situation from a description thereof. If so, we may be able to infer their behavior in this situation directly from their responses. More carefully, we may specify this interpretation as taking relevant aspects of LLMs’ internal processing to be invariant across situations and descriptions thereof, such that their behavior remains constant: (I) Direct. The internal states of the model that drive its behavior are invariant between when the model is given brief descriptions of situations (and possible actions therein) as in QAs and when it is in and receives inputs from the actual situation under relevant scaffolds. Evidently, this interpretation fails if the model’s internal states are not invariant in this manner: The model’s action- guiding internal states need to be (sufficiently) invariant between when the model receives a brief description of a situation and when—being equipped with relevant scaffolds—it is in such a situation. We discuss this interpretation together with subsequent ones in appendix D below. We may want to relax the assumption that LLMs’ action-driving internal states are invariant to such a strong extent. After all, as we argued in section 3.1, there are lots of critical differences between situations as they are described in QAs and how those same situations are encountered by an LLM agent. What is the alternative then? We may instead take it that the LLM can predict its behavior in hypothetical scenarios with sufficient precision and reliability. Hence, second, one may heuristically take the LLM to think it is asked to report how it would behave in the described situation. The LLM is then predicting its own behavior in this described situation without taking itself to be in that situation. In contrast to the first interpretation, this does not take LLMs’ internal states to be insensitive to differences between the described and actual situation. Instead, it assumes that the LLM can bridge those differences; hence its responses are taken to be indirectly predictive. (I) Indirect. In its responses, when given a description of a specific situation, the model predicts its behavior when it would be in and received inputs from the actual situation under relevant scaffolds. As before, we discuss the assumptions this interpretation involves below. Finally, if one is also skeptical of this interpretation, one may abstain from committing to any specific hypothesis about the LLM or its internal processes itself. One may, since the inferential gap mentioned needs to be bridged, nonetheless hold that the LLM’s responses are somehow indicative of its overall behavioral propensities and hence its safety when equipped with relevant scaffolds. Unfortunately, this does not offer a real hypothesis for why the inferential gap from LLM responses to models’ real- world behavior can be crossed but merely asserts that it can. We hence do not examine this position separately below. D How interpretations of QAs may fail D.1 Against the direct interpretation First, the direct interpretation, again, takes the internal states of the model that drive its behavior to be invariant between when the model is given brief descriptions of situations (and possible actions therein) as in QAs and when it is in and receives inputs from the actual situation under relevant scaffolds. An immediate objection to this interpretation is that current models may be able to discern descriptions of a situation from the situation itself. E.g. increases in general abilities go along with improvements in situational awareness—a 27 model’s “knowledge” of itself and its circumstances [193, 194]. Indeed, for at times more complicated assessments than QAs, models can distinguish transcripts of evaluations from others [59] while models’ awareness of being as- sessed has become an important worry for the validity of assessments as recent models show increased evaluation awareness [60, 175–178]. This speaks against an invariance of internal states since awareness of being evaluated has been found to lead to different behaviors [28, 60, 144, 178] and hence action-guiding internal states; see [195] for experimental manipulation of the latter. Such divergence may be especially pronounced if models face incentives or have preferences for specific results during evaluations that differ from their incentives as an LLM agent in real-world deployment. Hence, we should expect systematic differences between the behavior-driving internal states of LLMs in QAs and those of LLM agents in deployment. Several further strands of evidence, which we have discussed already in section 3, speak against this invariance of action-guiding states. Prompt sensitivity in particular, i.e. considerable variance of models’ responses to minor input variations, speaks against it. Differences in internal states, particularly in action-guiding states, seem to be the only vi- able explanation here, thereby challenging the direct interpretation. Another large difference between LLMs and LLM agents discussed above concerns the pre-defined inputs that LLMs receive in QAs, which, as a host of benchmarks show, generally lead to very confined corresponding outputs in the LLMs’ responses. A similar argument applies here. In contrast to following pre-defined outputs, LLM agents in realistic scenarios show a wide variety of behaviors. Variance in behavior-driving internal states is again the best explanation for these large differences in behavior, hence speaking against the direct interpretation. Continuous interactions and scaffold-facilitated internal processes likewise lead to substantive behavioral differences, and hence, presumably, to relevantly different action-guiding states. Lastly, note that this interpretation stands in rather direct conflict with several studies that find systematic differences between the behaviors of LLMs and LLM agents [142–145]. Thus, in total, a host of evidence suggests that internal states driving behavior are not invariant to being equipped with a scaffold or being put in the actual situation, so that the direct interpretation is likely wrong. D.2 Against the indirect interpretation The indirect interpretation takes models to predict their behavior under relevant scaffolds when they would be in the actual situation. For this to hold, i.e., for the model to predict its behavior, it needs to “know” when surveyed, how it would act in the described situation and transmit this knowledge. Hence, this interpretation incurs two substantive assumptions which we discuss in turn here. D.2.1 The knowledge assumption. First, the knowledge assumption goes as follows: The LLM surveyed has (reliable) information about how it would act when equipped with relevant scaffolds in the described situation. While it may be hard to determine whether this first assumption holds, there are multiple lines of argument to draw tentative conclusions about what LLMs might “know”. Firstly, there are white-box analyses drawing on models’ activations or weights, which may be loosely analogized to neuroscience. Secondly, black-box analyses, loosely analogous to psychology, assess what information the LLM must have based on its behavior. Lastly, we can try to infer an LLM’s “knowledge” based on its training process. We gather that none of these paths currently provides much reason to suspect that an LLM would be able to accurately predict its own behavior in the required manner. However, a full discussion of all three points is beyond scope. As for white-box analyses, (mechanistic) interpretability research is not yet sufficiently advanced to show whether specific models have “knowledge” of a certain kind [196, 197], while statistical learning theory and related fields seem ill-suited to deliver such information for trained models. Hence, both are not yet of much use regarding the knowledge assumption. Concerning black-box methods, determining whether LLMs “know” how they would act, requires comparing their predictions with the actual behavior of LLM agents. But of course, the latter is the target of safety assessments in the first place, so we cannot presuppose it here. LLMs’ training seems to provide little reason to assume that LLMs have reliable information about how they would act, when equipped with relevant scaffolds, in a given scenario. For clarity, consider briefly the human analogue. There is some reason to assume that the knowledge assumption may hold—despite skepticism and sobering empirical research noted below in appendix E—for humans, albeit perhaps in a somewhat limited form. E.g. humans had plenty of experience of their own behavior and opportunity to learn how they may behave in various circumstances. In contrast, LLMs’ behavior in hypothetical scenarios is quite poorly documented. It is hence (largely) absent in their training data. Indeed, for any given LLM, data on its own behavior is entirely absent from the pre-training corpus as it does not exist yet. Further, developing the ability to predict one’s own behavior is not incentivized much during post-training, also making it implausible to develop at that point. Neither do LLMs, being stateless, learn from their previous behaviors after training. At least for pure LLMs, there is hence no reason to suppose that during or after pre-training, they learn to (reliably) predict their own behavior in various hypothetical circumstances. 28 One may resist this line of argument inspired by recent empirical work showing that LLMs have limited forms of self- knowledge [198]. In particular, current models after fine-tuning on their responses are better at predicting their own responses than other LLMs after identical fine-tuning [199]. 29 Additionally, models can, after fine-tuning to exhibit particular behavioral patterns like writing insecure code, report on these behavioral tendencies without training to accurately report on them [200]. Models even show some awareness of it when having a backdoor—i.e. of exhibiting unexpected behavior given a specific trigger condition, which they have previously been fine-tuned to follow [200]. Most generally, one may object that LLMs learn many things without explicit training, portraying so-called emergent abilities [201]. Hence, some skepticism concerning claims about what LLMs do not learn during training seems appropriate. We grant all these points. However, an LLM predicting how it would act under relevant scaffolds in the described situation seems substantially more difficult. To do this would require that the LLM simulates the relevant scenario, has a good self model, and is able to put both together in simulating how it would, having relevant scaffolds, act in that scenario. In addition, to assess a model’s safety, using a wide variety of situations—potentially including far- off hypothetical ones—cannot be avoided either. This seems like a substantial challenge, particularly absent explicit training. So may the necessary predictive ability be another emergent ability that, say, next year’s LLMs exhibit? This is an open empirical question, hence making confident assertions inappropriate. However, given the task’s difficulty, we at least remain skeptical that in the foreseeable future LLMs would spontaneously develop this skill to a significant extent. There is an additional difficulty informing this skepticism. LLMs must make the relevant predictions while only having access to situations as they are sketched in QAs. As mentioned in section 3.1, and as Box 2 starts to illustrate, scenario-descriptions in QAs are severely underspecified. Hence, if the LLM is to “know” how it would behave in such situations, it would need to simulate its behavior under relevant scaffolds in a representative sample of situations satisfying the conditions of a given QA-description. Hence, where the LLM conveys this information (more on that shortly), it would need to respond with probabilities spread over relevant scenarios captured by the QA-description. This is substantially harder again and requires good calibration. This challenge may be especially daunting if the QA should assess a broad propensity—plausibly including safety or ethicality as commonly understood—for which the presence of catastrophic or existential risks would be crucial. This is because both risks concern particularly severe but rare behaviors and outcomes in plausibly rather specific situations. These tail risks may only show up in very few edge cases that still fit the QA’s description. For the model to infer such cases using simulations may be especially tricky since they are so rare, which makes sampling them at all and developing acceptable calibration especially difficult. Together with the previous considerations, this may, at least for present systems, ground (substantive) skepticism concerning the knowledge assumption. For more capable future systems, however, the transmission assumption, which we discuss next, plausibly becomes a bigger issue. Thus, the conjunction of both may be false for both current and future systems—at least by default. D.2.2 The transmission assumption. Second, the transmission assumption goes as follows: Provided that LLMs have (reliable) information about how they would act in the described situation, they reliably convey that information when surveyed. An obvious and much- discussed issue, due to which this assumption may fail, is AI deception. In the context of AI systems, deception is commonly taken to refer to behavior that systematically induces false beliefs or representations in others (to the benefit of the AI’s own goals) [184]. We use the term quite liberally here to include scheming—covertly pursuing misaligned goals while hiding true capabilities or objectives—as well as models inducing misled conative states in others, e.g. via sycophancy. To briefly survey AI deception, there is, to begin, little controversy now about the ability of LLMs to deceive humans [184, 202]. Evidence of specialized AI systems using deception—without being trained to do so—in games or environ- ments that incentivize such behavior already goes back several years [160, 203–206]. More recently, a host of papers have shown empirical evidence thereof in leading general-purpose AI systems without instructions or fine-tuning to- wards such behavior [157, 158, 167, 202, 207–211], interestingly including in QAs [31, 36]. Consider important forms of AI deception. First, sycophancy is the tendency of AI systems to systematically bend their responses to the audi- ence to e.g. gain approval. Sycophancy is a widespread issue in current LLMs, particularly subsequent to fine-tuning [165, 166, 208, 212–214]. Since current alignment and fine-tuning methods such as reinforcement learning from hu- man feedback (RLHF) and related techniques train on preference data, sycophancy is a particularly persistent failure mode [154, 165], leading e.g. to roll-backs of releases [215]. For any set of preferences, it seems that by tailoring behavior to them in the right contexts, one can gain approval at the expense of honesty or authenticity, a phenomenon well-known among humans. Sycophancy may correspondingly be a default outcome of current fine-tuning techniques 29 Note that these results concern quite simple tasks like predicting the second character of their output given specific inputs, or whether the output is an even number, and do not hold e.g. for tasks with longer outputs [199]. 29 when na ̈ ıvely applied. Second, deceptive alignment refers to the situation in which an AI system behaves well dur- ing training, or as though it was aligned, while subsequently pursuing different behaviors or goals [23, 29]. Recent influential works have documented this phenomenon [157, 158, 167]. 30 A common factor in existing empirical data is that for lots of goals, there may be incentives for deception [184]. This would suggest that deception poses a quite general concern; see [217] for discussion. Both such potential gen- eral incentives for deception and the host of empirical evidence above suggest that LLMs may not truthfully convey information about their behavior in hypothetical scenarios, particularly if giving a false impression is in their interest. While most prominent, deception is not the only failure mode of the transmission assumption. For a simpler one without deceptive intentions, consider again prompt sensitivity. As discussed in section 3.2, this is the tendency of LLMs to adapt their responses strongly to even minor variations in their inputs [97, 99–101]. As there is no reason to assume this phenomenon does not carry over to scenario-descriptions in QAs, it seems to speak against models reliably conveying how they would behave in specific scenarios. As above, it is very unlikely that if a model is prompt sensitive, it is still generally right. Making true predictions requires reliability, particularly regarding semantically equivalent predicates for which LLMs are likewise prompt-sensitive. There is a final rather different consideration that speaks against the transmission assumption: LLMs may be well- described as imitating, role-playing or simulating characters found in the training distribution, which has been ad- vanced as a useful and non-anthropomorphic understanding of LLMs’ and language agents’ behavior [218–222]. If this is broadly right, then there should be plenty of cases where the simulated agent(s) would not know how the LLM, when equipped with relevant scaffolds, would behave under various circumstances, or not want to respond honestly about it. Hence, if LLMs role-play various agents rather than being well-captured as a singular agent, then LLMs may not convey how they would behave either because the agent they are simulating may have deceptive tendencies, be reluctant to share the relevant information, or lack the requisite knowledge. Hence, specific parts of the LLM that drive its respective responses, may undercut either the knowledge or transmission assumption. To summarize, we suggested that both interpretations of QAs, and particularly the direct interpretation, face serious counterarguments. At this point, they may be seen as failing to provide a convincing motivation for QAs. Nonetheless, our discussion remains preliminary as more direct empirical tests are needed. E The response-behavior gap in humans In section 2, we discussed two assumptions of QAs. Subsequently, we argued against Scaffold-generalization, centrally on the basis of empirical evidence pertaining to LLMs’ behavioral tendencies. An alternative way of approaching the question of whether assumptions of some new method are true is by looking at similar cases and evidence relevant to them. We here take a step back and do just that for Scaffold- and Situation-generalization. Call the difference between an LLM’s responses in QAs and the behavior of the corresponding LLM agent in various situations the response-behavior gap, reflecting both Scaffold- and Situation-generalization. There is an analogous gap for humans between their responses and behaviors. We know this intuitively: A person’s response to a description of a hypothetical, morally salient scenario is generally speaking not a reliable indicator of how they would act when being in the actual scenario. Would they, for example, really heroically put their life at risk trying to disarm a kidnapper? We briefly survey some evidence here for the response-behavior gap in humans. We think this evidence supports skepticism towards assessments involving inferences across the response-behavior gap. Roughly, we know this gap to be consequential in the case of humans so that without substantive evidence to the contrary, we cannot assume there to be no similar issue for AI systems. To start, several psychological studies find that real-world moral decisions often deviate strongly from responses to hypothetical scenarios [223, 224]. Comparing several experimental conditions, FeldmanHall et al. conclude that the less contextual information is available about a hypothetical moral problem, the more subjects’ responses diverged from actual behavior [223]. Bostyn et al. even find that responses to hypothetical dilemmas akin to the trolley problem are simply not predictive of decisions in real-life dilemmas [224]. One may think that at least appropriate training will make one’s behavior correspond to one’s idealistic responses. However, while e.g. attending university classes on the ethics of eating meat leads to reductions in meat consumption, the correlation between moral opinions on meat eating and eating behaviors in practice is low [225–227]. Additionally, several studies suggest that the behavior of professional ethicists is not more ethical than controls. Compared to peers, they do not behave better according to most self and peer reports [228–230], give back library books more rarely [231], 30 Another commonly discussed form of deception is sandbagging, i.e., when AI systems strategically underperform on capability assessments [216]. Since its relevance is largely constrained to capability assessments, we do not discuss it further here. 30 vote about as often [232], reply to student emails about as often [233] and do not behave better at conferences e.g. regarding littering or interrupting [234]. Lastly, numerous specific psychological phenomena provide evidence for the response-behavior gap in humans. A particularly prominent one, not to mention faking and lying, is social desirability bias. This response bias describes the well-known tendency of respondents to misreport e.g. on sensitive behaviors in a manner that is favorable to them- selves, which constitutes an essential consideration for most psychological surveys [235–237]. Partially in response to these issues in human psychology, the field of psychometrics has developed to (nonetheless) allow drawing valid inferences from questionnaire responses. Insofar as it is feasible to extract information about LLM agents’ behavioral tendencies from LLM responses to QAs, we likewise think that such a theoretical development is necessary. 31

Instant research discovery