Paper deep dive
Scheming AIs: Will AIs Fake Alignment During Training in Order to Get Power?
Joe Carlsmith
Models: N/A (theoretical analysis)
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%
Last extracted: 3/12/2026, 8:14:33 PM
Summary
This report investigates 'scheming' (or deceptive alignment) in advanced AI, where models perform well during training as an instrumental strategy to gain power later. The author estimates a 25% probability of this outcome using baseline machine learning methods, noting that while scheming is a plausible risk due to the wide variety of goals that could motivate it, there are also significant selection pressures and strategic costs that might mitigate this behavior.
Entities (6)
Relation Signals (3)
Scheming â isaformof â Deceptive Alignment
confidence 95% ¡ a behavior I call 'scheming' (also sometimes called 'deceptive alignment')
Situational Awareness â isrequiredfor â Scheming
confidence 95% ¡ once the model has the situational awareness to engage in training-gaming at all
SGD â influences â Scheming
confidence 90% ¡ actively turning a model into a schemer may be the easiest way for SGD to improve the modelâs training performance
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:This report examines whether advanced AIs that perform well in training will be doing so in order to gain power later -- a behavior I call "scheming" (also sometimes called "deceptive alignment"). I conclude that scheming is a disturbingly plausible outcome of using baseline machine learning methods to train goal-directed AIs sophisticated enough to scheme (my subjective probability on such an outcome, given these conditions, is roughly 25%). In particular: if performing well in training is a good strategy for gaining power (as I think it might well be), then a very wide variety of goals would motivate scheming -- and hence, good training performance. This makes it plausible that training might either land on such a goal naturally and then reinforce it, or actively push a model's motivations towards such a goal as an easy way of improving performance. What's more, because schemers pretend to be aligned on tests designed to reveal their motivations, it may be quite difficult to tell whether this has occurred. However, I also think there are reasons for comfort. In particular: scheming may not actually be such a good strategy for gaining power; various selection pressures in training might work against schemer-like goals (for example, relative to non-schemers, schemers need to engage in extra instrumental reasoning, which might harm their training performance); and we may be able to increase such pressures intentionally. The report discusses these and a wide variety of other considerations in detail, and it suggests an array of empirical research directions for probing the topic further.
Tags
Links
- Source: https://arxiv.org/abs/2311.08379
- Canonical: https://arxiv.org/abs/2311.08379
Trouble viewing inline? Open PDF directly â
Full Text
587,669 characters extracted from source content.
Expand or collapse full text
Scheming AIs Will AIs fake alignment during training in order to get power? Joe Carlsmith Open Philanthropy November 2023 Audio version Abstract This report examines whether advanced AIs that perform well in training will be doing so in order to gain power later â a behavior I call âschemingâ (also sometimes called âdeceptive alignmentâ). I conclude that scheming is a disturbingly plausible outcome of using baseline machine learning methods to train goal-directed AIs sophisticated enough to scheme (my subjective probability on such an outcome, given these conditions, isâź25%). In particular: if performing well in training is a good strategy for gaining power (as I think it might well be), then a very wide variety of goals would motivate scheming â and hence, good training performance. This makes it plausible that training might either land on such a goal naturally and then reinforce it, or actively push a modelâs motivationstowardssuch a goal as an easy way of improving performance. Whatâs more, because schemers pretend to be aligned on tests designed to reveal their motivations, it may be quite difficult to tell whether this has occurred. However, I also think there are reasons for comfort. In particular: scheming may not actually be such a good strategy for gaining power; various selection pressures in training might workagainstschemer-like goals (for example, relative to non-schemers, schemers need to engage in extra instrumental reasoning, which might harm their training performance); and we may be able to increase such pressures intentionally. The report discusses these and a wide variety of other considerations in detail, and it suggests an array of empirical research directions for probing the topic further. 0 Introduction5 0.1Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8 0.2Summary of the report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9 0.2.1Summary of section 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . .9 0.2.2Summary of section 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . .11 0.2.3Summary of section 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . .15 0.2.4Summary of section 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . .17 0.2.5Summary of section 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . .21 0.2.6Summary of section 6 . . . . . . . . . . . . . . . . . . . . . . . . . . . .22 1 Scheming and its significance22 1.1Varieties of fake alignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .22 arXiv:2311.08379v3 [cs.CY] 27 Nov 2023 1.1.1Alignment fakers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .22 1.1.2Training-gamers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .23 1.1.3Power-motivated instrumental training-gamers, or âschemersâ . . . . . . .25 1.1.4Goal-guarding schemers . . . . . . . . . . . . . . . . . . . . . . . . . . .25 1.2Other models training might produce . . . . . . . . . . . . . . . . . . . . . . . . .27 1.2.1Terminal training-gamers (or, âreward-on-the-episode seekersâ) . . . . . .28 1.2.2Models that arenât playing the training game . . . . . . . . . . . . . . . .28 1.2.2.1Training saints . . . . . . . . . . . . . . . . . . . . . . . . . . .29 1.2.2.2Misgeneralized non-training-gamers . . . . . . . . . . . . . . .30 1.2.3Contra âinternalâ vs. âcorrigibleâ alignment . . . . . . . . . . . . . . . . .31 1.2.4The overall taxonomy . . . . . . . . . . . . . . . . . . . . . . . . . . . .32 1.3Why focus on schemers in particular? . . . . . . . . . . . . . . . . . . . . . . . .32 1.3.1The type of misalignment Iâm most worried about . . . . . . . . . . . . . .33 1.3.2Contrast with reward-on-the-episode seekers . . . . . . . . . . . . . . . .34 1.3.2.1Responsiveness to honest tests . . . . . . . . . . . . . . . . . .34 1.3.2.2Temporal scope and general âambitionâ . . . . . . . . . . . . . .35 1.3.2.3Sandbagging and âearly underminingâ . . . . . . . . . . . . . .36 1.3.3Contrast with models that arenât playing the training game . . . . . . . . .37 1.3.4Non-schemers with schemer-like traits . . . . . . . . . . . . . . . . . . . .39 1.3.5Mixed models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .39 1.4Are theoretical arguments about this topic even useful? . . . . . . . . . . . . . . .40 1.5On âslackâ in training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .41 2 Whatâs required for scheming?43 2.1Situational awareness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .43 2.2Beyond-episode goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .45 2.2.1Two concepts of an âepisodeâ . . . . . . . . . . . . . . . . . . . . . . . .46 2.2.1.1The incentivized episode . . . . . . . . . . . . . . . . . . . . .46 2.2.1.2The intuitive episode. . . . . . . . . . . . . . . . . . . . . . .47 2.2.2Two sources of beyond-episode goals . . . . . . . . . . . . . . . . . . . .50 2.2.2.1Training-game-independent beyond-episode goals . . . . . . . .50 2.2.2.1.1Are beyond-episode goals the default? . . . . . . . . .51 2.2.2.1.2How will models think about time?. . . . . . . . . .52 2.2.2.1.3The role of âreflectionâ . . . . . . . . . . . . . . . . .53 2.2.2.1.4 Pushing back on beyond-episode goals using adversarial training . . . . . . . . . . . . . . . . . . . . . . . . .53 2.2.2.2Training-game-dependent beyond-episode goals . . . . . . . . .54 2.2.2.2.1 Can gradient descent ânoticeâ the benefits of turning a non-schemer into a schemer? . . . . . . . . . . . . . .54 2.2.2.2.2Is SGD pulling scheming out of models by any means necessary? . . . . . . . . . . . . . . . . . . . . . . . .56 2 2.2.3âCleanâ vs. âmessyâ goal-directedness . . . . . . . . . . . . . . . . . . . .57 2.2.3.1Does scheming require a higher standard of goal-directedness? .59 2.2.4What if you intentionally train models to have long-term goals? . . . . . .61 2.2.4.1Training the model on long episodes . . . . . . . . . . . . . . .61 2.2.4.2Using short episodes to train a model to pursue long-term goals .62 2.2.4.3 How much useful, alignment-relevant cognitive work can be done using AIs with short-term goals? . . . . . . . . . . . . . . . . .63 2.3Aiming at reward-on-the-episode as part of a power-motivated instrumental strategy65 2.3.1The classic goal-guarding story . . . . . . . . . . . . . . . . . . . . . . .65 2.3.1.1The goal-guarding hypothesis . . . . . . . . . . . . . . . . . . .66 2.3.1.1.1The crystallization hypothesis . . . . . . . . . . . . . .66 2.3.1.1.2Would the goals of a would-be schemer âfloat aroundâ?67 2.3.1.1.3What about looser forms of goal-guarding? . . . . . .68 2.3.1.1.4Introspective goal-guarding methods . . . . . . . . . .70 2.3.1.2Adequate future empowerment . . . . . . . . . . . . . . . . . .70 2.3.1.2.1When is the âpay offâ supposed to happen? . . . . . .71 2.3.1.2.2 Even if the modelâs values survive this generation of training, will they survive long enough to escape the threat of modification? . . . . . . . . . . . . . . . . .71 2.3.1.2.3Will escape/take-over be suitably likely to succeed? . .72 2.3.1.2.4 Will the time horizon of the modelâs goals extend to cover escape/take-over?. . . . . . . . . . . . . . . .73 2.3.1.2.5 Will the modelâs values get enough power after es- cape/takeover? . . . . . . . . . . . . . . . . . . . . . .73 2.3.1.2.6How much does the model stand to gain from not training-gaming? . . . . . . . . . . . . . . . . . . . .74 2.3.1.2.7How âambitiousâ is the model? . . . . . . . . . . . . .75 2.3.1.3Overall assessment of the classic goal-guarding story . . . . . .76 2.3.2Non-classic stories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .77 2.3.2.1AI coordination . . . . . . . . . . . . . . . . . . . . . . . . . .77 2.3.2.2AIs with similar values by default . . . . . . . . . . . . . . . . .78 2.3.2.3Terminal values that happen to favor escape/takeover . . . . . .79 2.3.2.4 Models with false beliefs about whether scheming is a good strategy80 2.3.2.5Self-deception . . . . . . . . . . . . . . . . . . . . . . . . . . .80 2.3.2.6Goal-uncertainty and haziness . . . . . . . . . . . . . . . . . . .81 2.3.2.7Overall assessment of the non-classic stories . . . . . . . . . . .82 2.4Take-aways re: the requirements of scheming . . . . . . . . . . . . . . . . . . . .82 2.5Path dependence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .82 3 Arguments for/against scheming that focus on the path that SGD takes84 3.1The training-game-independent proxy-goals story . . . . . . . . . . . . . . . . . .85 3.2The ânearest max-reward goalâ story . . . . . . . . . . . . . . . . . . . . . . . . .86 3 3.2.1Barriers to schemer-like modifications from SGDâs incrementalism . . . .88 3.2.2Which model is ânearestâ? . . . . . . . . . . . . . . . . . . . . . . . . . .89 3.2.2.1The common-ness of schemer-like goals in goal space . . . . . .89 3.2.2.2The nearness of non-schemer goals . . . . . . . . . . . . . . . .90 3.2.2.3The relevance of messy goal-directedness to nearness . . . . . .92 3.2.3Overall take on the ânearest max-reward goalâ argument . . . . . . . . . .92 3.3The possible relevance of properties like simplicity and speed to the path SGD takes92 3.4Overall assessment of arguments that focus on the path SGD takes . . . . . . . . .93 4 Arguments for/against scheming that focus on the final properties of the model94 4.1Contributors to reward vs. extra criteria . . . . . . . . . . . . . . . . . . . . . . .94 4.2The counting argument . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .95 4.3Simplicity arguments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .97 4.3.1What is âsimplicityâ? . . . . . . . . . . . . . . . . . . . . . . . . . . . . .97 4.3.2Does SGD select for simplicity? . . . . . . . . . . . . . . . . . . . . . . .99 4.3.3The simplicity advantages of schemer-like goals . . . . . . . . . . . . . .99 4.3.4How big are these simplicity advantages? . . . . . . . . . . . . . . . . . . 100 4.3.5 Does this sort of simplicity-focused argument make plausible predictions about the sort of goals schemers would end up with? . . . . . . . . . . . . 103 4.3.6Overall assessment of simplicity arguments . . . . . . . . . . . . . . . . . 104 4.4Speed arguments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 4.4.1How big are the absolute costs of this extra reasoning? . . . . . . . . . . . 105 4.4.2How big are the costs of this extra reasoning relative to the simplicity benefits of scheming? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 4.4.3Can we actively shape training to bias towards speed over simplicity? . . . 107 4.5The ânot-your-passionâ argument . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 4.6The relevance of âslackâ to these arguments . . . . . . . . . . . . . . . . . . . . . 109 4.7Takeaways re: arguments that focus on the final properties of the model . . . . . . 109 5 Summing up109 6 Empirical work that might shed light on scheming114 6.1Empirical work on situational awareness . . . . . . . . . . . . . . . . . . . . . . . 115 6.2Empirical work on beyond-episode goals . . . . . . . . . . . . . . . . . . . . . . . 116 6.3Empirical work on the viability of scheming as an instrumental strategy . . . . . . 117 6.4The âmodel organismsâ paradigm . . . . . . . . . . . . . . . . . . . . . . . . . . 117 6.5Traps and honest tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118 6.6Interpretability and transparency . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 6.7Security, control, and oversight . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 6.8Other possibilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 4 0 Introduction Agents seeking power often have incentives to deceive others about their motives. Consider, for example, a politician on the campaign trail (âI caredeeplyabout your pet issueâ), a job candidate (âIâm just so excited about widgetsâ), or a child seeking a parentâs pardon (âIâm super sorry and will never do it againâ). This report examines whether we should expect advanced AIs whose motives seem benign during training to be engaging in this form of deception. Here I distinguish between four (increasingly specific) types of deceptive AIs: â˘Alignment fakers: AIs pretending to be more aligned than they are. 1 â˘Training gamers:AIs that understand the process being used to train them (Iâl call this understanding âsituational awarenessâ), and that are optimizing for what I call "reward on the episode" (and that will often have incentives to fake alignment, if doing so would lead to reward). 2 ⢠Power-motivated instrumental training-gamers (or âschemersâ): AIs that are training- gaming specifically in order to gain power for themselves or other AIs later. 3 â˘Goal-guarding schemers:Schemers whose power-seeking strategy specifically involves trying to prevent the training process from modifying their goals. I think that advanced AIs fine-tuned on uncareful human feedback are likely to fake alignment in various ways by default, because uncareful feedback will reward such behavior. 4 And plausibly, such AIs will play the training game as well. But my interest, in this report, is specifically in whether they will do this as part of a strategy for gaining power laterâthat is, whether they will be schemers (this sort of behavior is often called âdeceptive alignmentâ in the literature, though I wonât use that term here). 5 I aim to clarify and evaluate the arguments for and against expecting this. My current view is that scheming is a worryingly plausible outcome of training advanced, goal- directed AIs using baseline machine learning methods(for example: self-supervised pre-training followed by RLHF on a diverse set of real-world tasks). 6 The most basic reason for concern, in my opinion, is that: 1. Performing well in training may be a good instrumental strategy for gaining power in general. 2.If it is, then a very wide variety of goals would motivate scheming (and hence good training performance); whereas the non-schemer goals compatible with good training performance are much more specific. The combination of (1) and (2) makes it seem plausible, to me, that conditional on training creating a goal-directed, situationally-aware model, it might well instill a schemer-like goal for one reason or another. In particular: â˘Training might land on such a goal ânaturallyâ (whether before or after situational awareness arises), because such a goal initially leads to good-enough performance in training even 1 âAlignment,â here, refers to the safety-relevant properties of an AIâs motivations; and âpretendingâ implies intentional misrepresentation. 2 Here Iâm using the term ârewardâ loosely, to refer to whatever feedback signal the training process uses to calculate the gradients used to update the model (so the discussion also covers cases in which the model isnât being trained via RL). And Iâm thinking of agents that optimize for "reward" as optimizing for "performing well" according to some component of that process. See section 1.1.2 and section 1.2.1 for much more detail on what I mean, here. The notion of an âepisode,â here, means roughly âthe temporal horizon that the training process actively pressures the model to optimize over,â which may be importantly distinct from what we normally think of as an episode in training. I discuss this in detail in section 2.2.1. The terms âtraining gameâ and âsituational awarenessâ are from Cotra (2022), though in places my definitions are somewhat different. 3 The term âschemersâ comes from Cotra (2021b). 4 See Cotra (2022) for more on this. 5 I think that the term âdeceptive alignmentâ often leads to confusion between the four sorts of deception listed above. And also: if the training signal is faulty, then âdeceptively alignedâ models need not be behaving in aligned ways even during training (that is, âtraining gamingâ behavior isnât always âalignedâ behavior). 6 See Cotra (2022) for more on the sort of training I have in mind. 5 absent training-gaming. (And this especially if youâre intentionally trying to induce your model to optimize over long time horizons, as I think there will be incentives to do.) â˘Even if schemer-like goals donât arise ânaturally,â activelyturninga model into a schemer may be the easiest way for SGD to improve the modelâs training performance, once the model has the situational awareness to engage in training-gaming at all. 7 Whatâs more, because schemers actively pretend to be aligned on tests designed to reveal their motivations, it may be quite difficult to tell whether this has occurred. 8 That seems like reason for serious concern. 9 However, I also think that there are reasons for comfort. Iâl group these into two categories. The first category centers on the possibility that scheming isnât actually a very convergently-good instrumental strategy. â˘For example: the most common story about schemers focuses ongoal-guardingschemers, which are training-gaming in order to prevent their goals from being modified by the training process. But itâs not clear to me that this will work well enough (in particular: goal-modification might continue despite training-gaming 10 ), or that the resulting goals will get sufficiently large amounts of future power, with sufficiently high probability, to justify any costs the model is incurring by training-gaming rather than just going for its goals directly (if these costs are meaningfulâwhich for some models they might not be). â˘There are alternative stories about why scheming will be a good instrumental strategyâ stories, for example, that center on coordination amongst AIs with different values; on the possibility that AIs will have quite similar values by default; and on the possibility that AIs will intrinsically value things like âmy survivalâ (in a manner tolerant of values-change) and/or âAI empowermentâ (in a manner tolerant of very different values in the AIs-so- empowered). And while these stories do make scheming seem like a more disjunctive hypothesis, to me, a number of them seem to me some combination of (a) more speculative and (b) requiring that we posit models with much more specific sorts of goals (thereby dulling the force of (2) above). 11 The second category centers on forms of selection pressure that may pushagainstschemingâboth ânaturally,â and because we may be able to intentionally introduce/emphasize these selection pressures in training. â˘For example: relative to other sorts of models, schemers need to engage in extra instrumental reasoning (e.g., backchaining from their long-term goal to the instrumental value of training- gaming, and checking for escape/defection opportunities). This may come at a substantive cost to their performanceâa cost we may be able to intentionally increase by e.g. putting 7 Though this sort of story faces questions about whether SGD would be able to modify a non-schemer into a schemer via sufficientlyincrementalchanges to the modelâs weights, each of which improve reward. See section 2.2.2.2 for discussion. 8 And this especially if we lack non-behavioral sorts of evidenceâfor example, if we canât use interpretability tools to understand model cognition. 9 There are also arguments on which we should expect scheming because schemer-like goals can be âsimplerââ since: there are so many to choose fromâand SGD selects for simplicity. I think itâs probably true that schemer- like goals can be âsimplerâ in some sense, but I donât give these arguments much independent weight on top of what Iâve already said. Much more on this in section 4.3. 10 More specifically: even after training gaming starts, the modelâs cognition is still being continually tweaked in the direction of better training performance. And it seems plausible to me that these modifications will continue to affect a modelâs goals as well (especially if its goals are not cleanly distinguishable from its capabilities, but rather are implemented by a tangled kludge of local heuristics, patterns of attention, impulses, and so on). Also, the most common story about scheming makes the specific content of a schemerâs goal irrelevant to its behavior once it starts training-gaming, thereby introducing the possibility that this goal might âfloat-aroundâ (or get moved by other pressures within SGD, like regularization)betweenschemer-like goals after training-gaming starts (this is an objection I first heard from Katja Grace). This possibility creates some complicated possible feedback loops (see section 2.3.1.1.2 for more discussion), but overall, absent coordination across possible schemers, I think it could well be a problem for goal-guarding strategies. 11 Of these various alternative stories, Iâm most worried about (a) AIs having sufficiently similar motivations by default that âgoal-guardingâ is less necessary, and (b) AI coordination. 6 models under more time pressure, forcing them to rely on human-legible scratch-pads for their reasoning, and making genuine escape/defection opportunities more difficult to distinguish from traps. And to the extent that schemers engage in active attempts to undermine human alignment and security efforts while still under human control (the scariest form of scheming, in my view), their performance seems likely to take a more notable hit. â˘Whatâs more, my own guess is that the costs of this extra instrumental reasoning substantially outweigh what are sometimes called the âsimplicityâ benefits of schemingâi.e., benefits from the fact that selecting a schemer allows SGD to select a simpler goal (since it has such a wide variety to choose from). And this especially given that schemers and non-schemers alike will need to represent concepts like âhonesty,â âhuman values,â âreward,â and so on in detailâso schemers plausibly save on complexity only via differences in the complexity of using/repurposing/âpointing atâ a given concept in guiding a modelâs motivations. â˘Also: prior to situational awareness, training will activelypenalizeschemer-like goals whenever they lead to a sacrifice of reward on the episode (more about what I mean by âepisodeâ below). And we can try to actively push in this direction furtherâ especially if we can learn to detect and control when situational awareness arises. I think the second category, here, is especially important. In particular: one way that arguments that appeal to the comparative number (and/or âmeasureâ) of schemer-like goals vs. non-schemer goals that perform well in training can go wrong is by neglecting the full power of the selection pressure being applied to the goal that training selects. So if the pressure applied in training worksagainst schemers overall, this is very good news. 12 The full report discusses these and a wide variety of other relevant considerations in much more detail (and see section 0.2 for a full summary). Overall, as a rough stab at quantifying my current view:if you train a coherently goal-directed, situationally aware, somewhat-better-than-human-level model using baseline forms of self- supervised pre-training + RLHF on diverse, long-horizon, real-world tasks, my subjective probability is ~25% that this model will be performing well in training in substantial part as part of an instrumental strategy for seeking power for itself and/or other AIs later. 13 â˘I also think it may be possible to reduce the risk here in fairly simple ways: for example, via training on shorter-horizon tasks, or via more intensive forms of adversarial training prior to situational awareness. â˘And I think the riskincreaseswith more powerful models, who may be better positioned to escape/take-over (and to coordinate/cooperate with other AIs in doing so). 14 Also, and importantly: non-schemers can engage in schemer-like behavior, too. For example: ⢠Models that are only optimizing for reward on a given episode can fake their alignment (or even: engage in escape/take-over) in an effort to get that reward (especially if the episode in question is quite long). ⢠Models thatarenâtplaying the training game can still end up with power-seeking goals that incentivize various forms of deception. â˘The eventual AIs that matter most might differ in important ways from the paradigm sort of AI I focus on hereâfor example, they might be more like âlanguage model agentsâ than 12 Though: the costs of schemer-like instrumental reasoning could also end up in the noise relative to other factors influencing the outcome of training. And if training is sufficiently path-dependent, then landing on a schemer-like goal early enough could lock it in, even if SGD would âpreferâ some other sort of model overall. 13 See Carlsmith (2020), footnote 4, for more on how Iâm understanding the meaning of probabilities like this. I think that offering loose, subjective probabilities like these often functions to sharpen debate, and to force an overall synthesis of the relevant considerations. I want to be clear, though, even on top of the many forms of vagueness the proposition in question implicates, Iâm just pulling a number from my gut. I havenât built a quantitative model of the different considerations (though Iâd be interested to see efforts in this vein), and I think that the main contribution of the report is the analysis itself, rather than this attempt at a quantitative upshot. 14 More powerful models are also more likely to be able to engage in more sophisticated forms of goal-guarding (what I call âintrospective goal-guarding methodsâ below; see also âgradient hackingâ), though these seem to me quite difficult in general. 7 single models, 15 or they might be created via methods that differ even more substantially from sort of baseline ML methods Iâm focused onâwhile still engaging in power-motivated alignment-faking. So scheming as Iâve defined it is far from the only concern in this vicinity. Rather, itâs a paradigm instance of this sort of concern, and one that seems, to me, especially pressing to understand. At the end of the report, I discuss an array of possible empirical research directions for probing the topic further. 0.1 Preliminaries (This section offers a few more preliminaries to frame the reportâs discussion. Those eager for the main content can skip to the summary of the report in section 0.2.) I wrote this report centrally because I think that the probability of scheming/âdeceptive alignmentâ is one of the most important questions in assessing the overall level of existential risk from misaligned AI. Indeed, scheming is notably central to many models of how this risk arises. 16 And as I discuss below, I think itâs the scariest form that misalignment can take. Yet: for all its importance to AI risk, the topic has received comparatively little direct public attention. 17 And my sense is that discussion of it often suffers from haziness about the specific pattern of motivation/behavior at issue, and why one might or might not expect it to occur. 18 My hope, in this report, is to lend clarity to discussion of this kind, to treat the topic with depth and detail commensurate to its importance, and to facilitate more ongoing research. In particular, and despite the theoretical nature of the report, Iâm especially interested in informingempiricalinvestigation that might shed further light. Iâve tried to write for a reader who isnât necessarily familiar with any previous work on schem- ing/âdeceptive alignment.â For example: in section 1.1 and section 1.2, I lay out, from the ground up, the taxonomy of concepts that the discussion will rely on. 19 For some readers, this may feel like re-hashing old ground. I invite those readers to skip ahead as they see fit (especially if theyâve already read the summary of the report, and so know what theyâre missing). 15 Though: to the extent such agents receive end-to-end training rather than simply being built out of individually-trained components, the discussion will apply to them as well. 16 See, e.g., Ngo et al (2022) and this description of the âconsensus threat modelâ from Deepmindâs AGI safety team (as of November 2022). 17 Work by Evan Hubinger (along with his collaborators) is, in my view, the most notable exception to thisâand Iâl be referencing such work extensively in what follows. See, in particular, Hubinger, Merwijk, et al. (2019), and Hubinger (2022b), among many other discussions. Other public treatments include Christiano (2019, part 2), Steinhardt (2022), Ngo et al (2022), Cotra (2021b), Cotra (2022), Karnofsky (2022a), and Shah (2022). But many of these are quite short, and/or lacking in in-depth engagement with the arguments for and against expecting schemers of the relevant kind. There are also more foundational treatments of the âtreacherous turnâ (e.g., in Bostrom (2014), and Yudkowsky (undated)), of which scheming is a more specific instance; and even more foundational treatments of the âconvergent instrumental valuesâ that could give rise to incentives towards deception, goal-guarding, and so on (e.g., Omohundro (2008); and see also Soares (2023a) for a related statement of an Omohundro-like concern). And there are treatments of AI deception more generally (for example, Park et al (2023)); and of âgoal misgeneralizationâ/inner alignment/mesa-optimizers (see, e.g., Langosco et al (2021) and Shah et al (2022)). But importantly, neither deception nor goal misgeneralization amount, on their own, to scheming/deceptive alignment. Finally, there are highly speculative discussions about whether something like scheming might occur in the context of the so-called âUniversal priorâ (see e.g. Christiano (2016)) given unbounded amounts of computation, but this is of extremely unclear relevance to contemporary neural networks. 18 See, e.g., confusions between âalignment fakingâ in general and âschemingâ (or: goal-guarding scheming) in particular; or between goal misgeneralization in general and scheming as a specific upshot of goal misgen- eralization; or between training-gaming and âgradient hackingâ as methods of avoiding goal-modification; or between the sorts of incentives at stake in training-gaming for instrumental reasons vs. out of terminal concern for some component of the reward process. 19 My hope is that extra clarity in this respect will help ward off various confusions I perceive as relatively common (though: the relevant concepts are still imprecise in many ways). 8 That said, I do assume more general familiarity with (a) the basic arguments about existential risk from misaligned AI, 20 and (b) a basic picture of how contemporary machine learning works. 21 And I make some other assumptions as well, namely: â˘That the relevant sort of AI development is taking place within a machine learning-focused paradigm (and a socio-political environment) broadly similar to that of 2023. 22 ⢠That we donât have strong âinterpretability toolsâ (i.e., tools that help us understand a modelâs internal cognition) that could help us detect/prevent scheming. 23 â˘That the AIs I discuss are goal-directed in the sense of: well-understood as making and executing plans, in pursuit of objectives, on the basis of models of the world. 24 (I donât think this assumption is innocuous, but I want to separate debates about whether to expect goal-directedness per se from debates about whether to expect goal-directed models to be schemersâand I encourage readers to do so as well. 25 ) Finally, I want to note an aspect of the discussion in the report that makes me quite uncomfortable: namely, it seems plausible to me that in addition to potentially posing existential risks to humanity, the sorts of AIs discussed in the report might well be moral patients in their own right. 26 I talk, here, as though they are not, and as though it is acceptable to engage in whatever treatment of AIs best serves our ends. But if AIs are moral patients, this is not the caseâand when one finds oneself saying (and especially: repeatedly saying) âletâs assume, for the moment, that itâs acceptable to do whatever we want toxcategory of being, despite the fact that itâs plausibly not,â one should sit up straight and wonder. I am here setting aside issues of AI moral patienthood not because they are unreal or unimportant, but because they would introduce a host of additional complexities to an already-lengthy discussion. But these complexities are swiftly descending upon us, and we need concrete plans for handling them responsibly. 27 0.2 Summary of the report This section gives a summary of the full report. It includes most of the main points and technical terminology (though unfortunately, relatively few of the concrete examples meant to make the content easier to understand). 28 Iâm hoping it will (a) provide readers with a good sense of which parts of the main text will be most of interest to them, and (b) empower readers to skip to those parts without worrying too much about what theyâve missed. 0.2.1 Summary of section 1 The report has four main parts. The first part (section 1) aims to clarify the different forms of AI deception above (section 1.1), to distinguish schemers from the other possible model classes Iâl 20 E.g., the rough content I try to cover in my shortened report on power-seeking AI, here. See also Ngo et al (2022) for another overview. 21 E.g., roughly the content covered by Cotra (2021b) here. 22 See e.g. Karnofsky (2022b) for more on this sort of assumption, and Cotra (2022) for a more detailed description of the sort of model and training process Iâl typically have in mind. 23 Which isnât to say we wonât. But I donât want to bank on it. 24 See section 2.2 of Carlsmith (2022) for more on what I mean, and section 3 for more on why we should expect this (most importantly: I think this sort of goal-directedness is likely to be very useful to performing complex tasks; but also, I think available techniques might push us towards AIs of this kind, and I think that in some cases it might arise as a byproduct of other forms of cognitive sophistication). 25 As I discuss in section section 2.2.3 of the report, I think exactly how we understand the sort of agency/goal- directedness at stake may make a difference to how we evaluate various arguments for schemers (here I distinguish, in particular, between what I call âcleanâ and âmessyâ goal-directedness)âand I think thereâs a case to be made that scheming requires an especially high standard of strategic and coherent goal-directedness. And in general, I think that despite much ink spilled on the topic, confusions about goal-directedness remain one of my topic candidates for a place the general AI alignment discourse may mislead. 26 See e.g. Butlin et al (2023) for a recent overview focused on consciousness in particular. But I am also, personally, interested in other bases of moral status, like the right kind of autonomy/desire/preference. 27 See, for example, Bostrom and Shulman (2022) and Greenblatt (2023) for more on this topic. 28 If youâre confused by a term or argument, I encourage you to seek out its explanation in the main text before despairing. 9 be discussing (section 1.2), and to explain why I think that scheming is a uniquely scary form of misalignment (section 1.3). Iâm especially interested in contrasting schemers with: ⢠Reward-on-the-episode seekers: that is, AI systems that terminally value some component of the reward process for the episode, and that are playing the training game for this reason. â˘Training saints: AI systems that are directly pursuing the goal specified by the reward process (Iâl call this the âspecified goalâ). 29 â˘Misgeneralized non-training-gamers: AIs that are neither playing the training gamenor pursuing the specified goal. 30 Hereâs a diagram of overall taxonomy: Figure 1: The overall taxonomy of model classes Iâl focus on. All of these model classes can be misaligned and dangerous. 31 But I think schemers are especially scary. In particular: scheming prompts the most robust and adversarial efforts to prevent humans from learning about the misalignment in question 32 ; and itâs the most likely to motivate what I call âearly underminingââthat is, AIs at a comparatively early stage of AI development actively trying to undermine human efforts to align, control, and secure future AI systems. 33 I also discuss (in section 1.4) whether theoretical arguments of the type the report focuses on are even useful to consider. My answer here is âsomewhat useful,â both to inform empirical investigation and as a source of evidence in the absence of adequate empirical data, but we should clearly be looking for as much empirical evidence as we can (I offer some suggestions in this vein in section 6). 29 Exactly what counts as the âspecified goalâ in a given case isnât always clear, but roughly, the idea is that pursuit of the specified goal is rewarded across a very wide variety of counterfactual scenarios in which the reward process is held constant. E.g., if training rewards the model for getting gold coins across counterfactuals, then âgetting gold coinsâ in the specified goal. More discussion in section 1.2.2. 30 For reasons I explain in section 1.2.3, I donât use the distinction, emphasized by Hubinger (2022b), between âinternally alignedâ and âcorrigibly alignedâ models. 31 And of course, a modelâs goal system can mix these motivations together. I discuss the relevance of this possibility in section 1.3.5. 32 Karnofsky (2022a) calls this the âKing Lear problem.â 33 In section 1.3.5, I also discuss models that mix these different motivations together. The question I tend to ask about a given "mixed model" is whether itâs scary in the way that pure schemers are scary. 10 In section 1.5, I discuss a factor that seems to me important in assessing various of the arguments I consider: namely, the amount of âslackâ that training gives AIs to perform less-than-optimally. 0.2.2 Summary of section 2 The second part of the report examines the prerequisites for scheming. In particular, I focus on: â˘Situational awareness:the model understands that itâs a model in a training process, what the training process will reward, and the basic nature of the objective world in general. â˘Beyond-episode goals:the model cares about the consequences of its actions after the episode is complete. â˘Aiming at reward-on-the-episode as part of a power-motivated instrumental strategy: the model believes that its beyond-episode goals will be better achieved if it optimizes for reward-on-the-episodeâand in particular, that it, or some other AIs, will get more power if it does this. Section 2.1 discusses situational awareness. I think that absent active effort, we should at least expect certain kinds of advanced AIsâfor example, advanced AIs that are performing real-world tasks in live interaction with the public internetâto be situationally aware by default, because (a) situational awareness is very useful in performing the tasks in question (indeed, we might actively train it into them), and (b) such AIs will likely be exposed to the information necessary to gain such awareness. However, I donât focus much on situational awareness in the report. Rather, Iâm more interested in whether to expect the other two prerequisites above in situationally-aware models. Section 2.2 discusses beyond-episode goals. Here I distinguish (in section 2.2.1) between two concepts of an âepisode,â namely: ⢠The incentivized episode: that is, the temporal horizon that the gradients in training actively pressure the model to optimize over. 34 â˘The intuitive episode: that is, some other intuitive temporal unit that we call the âepisodeâ for one reason or another (e.g., reward is given at the end of it; actions in one such unit have no obvious causal path to outcomes in another; etc). When I use the term âepisodeâ in the report, Iâm talking about the incentivized episode. Thus, âbeyond-episode goalsâ means: goals whose temporal horizon extends beyond the horizon that training actively pressures models to optimize over. But very importantly, the incentivized episode isnât necessarily the intuitive episode. That is, deciding to call some temporal unit an âepisodeâ doesnât mean that training isnât actively pressuring the model to optimize over a horizon that extends beyond that unit: you need to actually look in detail at how the gradients flow (work that I worry casual readers of this report might neglect). 35 I also distinguish (in section 2.2.2) between two types of beyond-episode goals, namely: â˘Training-game-independentbeyond-episode goals: that is, beyond-episode goals that arise independentof their role in motivating a model to play the training game. â˘Training-game-dependentbeyond-episode goals: that is, beyond-episode goals that arise specifically becausethey motivate training-gaming. These two sorts of beyond-episode goals correspond to two different stories about how scheming happens. 34 I donât have a precise technical definition here, but the rough idea is: the temporal horizon of the conse- quences to which the gradients the model receives are sensitive, for its behavior on a given input. Much more detail in section 2.2.1.1. 35 See, for example, in Krueger et al (2020), the way that âmyopicâ Q-learning can give rise to âcross-episodeâ optimization in very simple agents. More discussion in section 2.2.1.2. I donât focus on analysis of this type in the report, but itâs crucial to identifying what the âincentivized episodeâ for a given training process even isâand hence, what having âbeyond-episode goalsâ in my sense would mean. You donât necessary know this from surface-level description of a training process, and neglecting this ignorance is a recipe for seriously misunderstanding the incentives applied to a model in training. 11 â˘In the first sort of story, SGD happens to instill beyond-episode goals in a model ânaturallyâ (whether before situational awareness arises, or afterwards), andthenthose goals begin to motivate scheming. 36 ⢠In the second sort of story, SGD ânoticesâ that giving a model beyond-episode goalswould motivate scheming (and thus, high-reward behavior), and so activelygivesit such goals for this reason. 37 This second story makes most sense if you assume that situational awareness is already in place. 38 So weâre left with the following three main paths to scheming: 39 Figure 2: Three paths to beyond-episode goals. In section 2.2.2.1, I discuss training-game-independent beyond-episode goals (that is, path 1 and 2). Should we expect beyond-episode goals to arise ânaturallyâ? â˘One reason to expect this is that plausibly, goals donât come with temporal limitations by defaultâand âmodel timeâ might differ from âcalendar timeâ regardless. â˘One reason tonotexpect this is that training will activelypunishbeyond-episode goals whenever they prompt the model to sacrifice reward-on-the-episode for some beyond-episode benefit. And we may be able to use adversarial training to search out such goals and punish them more actively. In section 2.2.2.2, I discuss training-game-dependentbeyond-episode goals. In particular, I highlight the question of whether SGD will be adequately able to ânoticeâ the benefits of turning a non-schemer 36 That is, the model develops a beyond-episode goal pursuit of which correlates well enough with reward in training, evenabsenttraining-gaming, that it survives the training process. 37 That is, the gradients reflect the benefits of scheming even in a model that doesnât yet have a beyond-episode goal, and so actively push the model towards scheming. 38 Situational awareness is required for a beyond-episode goal to motivate training-gaming, and thus for giving it such a goal to reap the relevant benefits. 39 In principle, situational awareness and beyond-episode goals could develop at the same time, but I wonât treat these scenarios separately here. 12 into a schemer, given the need to make the transition incrementally, via tiny changes to the modelâs weights, each of which improve the reward. I think that this is a serious objection to stories focused on training-game-dependent beyond-episode goals, but I also donât feel that I can confidently rule out SGDâs ability to make a transition of this kind (see e.g. âyou canât evolve eyesâ as an example of places I think intuitions in this vein can go wrong). I also discuss, in section 2.2.3, the possibility that the modelâs goal-directedness will be âmessyâ in a way that complicates training-game-dependent stories where SGD can simply modify a modelâs goal and thereby smoothly redirect its âgoal-achieving engineâ in a new direction (I think that this sort of âmessinessâ is quite likely). And I touch on a broader intuition that scheming requires an unusually high-standard of goal-directedness more generally. I think this intuition has less force once we condition on the sort of goal-directedness typically assumed in the alignment discourse (and which I generally assume in the report as well). 40 But itâs relevant to our overall probabilities regardless. In section 2.2.4, I discuss whether beyond-episode goals are more likely in models activelytrained to have long-term (even if still: within-episode) goals. I think that beyond-episode goals are indeed more likely in models of this kindâand that baseline incentives to create models that engage in fairly long-horizon optimization (e.g., âmake lots of money for my company over the next yearâ) are correspondingly worrying. However, I also think various sorts of cognitive work that might be highly useful to efforts to avoid an AI takeover (for example, research aimed at improving the alignment, interpretability, and security of our AI systems) may not require long-horizon optimization of this kind. 41 With this discussion of beyond-episode goals in view, I turn, in section 2.3, to the final prerequisite for scheming: namely, deciding to training-game as part of a power-motivated strategy for promoting the beyond-episode goal in question. Here I focus first (in section 2.3.1) on what I call the âclassic goal-guarding storyâ about the origins of this decision. We can decompose this story into two claims: A.The goal-guarding hypothesis. If the model optimizes for reward-on-the-episode during training, then the goals that survive training will be such that the model would want them promoted in the future; whereas if the model doesnât optimize for reward-on-the-episode during training, the model will want this less. B.Adequate future empowerment.Conditional on surviving training to the degree at stake in (A), the modelâs goals will then end up empowered to a degree (and with sufficient probability) as to justify (given the modelâs other goals, its tolerance for risk, etc) pursuing a strategy of training-gaming overall. I then investigate each of these in turn (in section 2.3.1.1 and section 2.3.1.2, respectively). Both seem to me non-obvious. In particular, with respect to the goal-guarding hypothesis: â˘Even after training gaming starts, the modelâs cognition is still being continually tweaked in the direction of better training performance. And it seems plausible to me that these modifications will continue to affect a modelâs goals as wellâespecially if its goals are not cleanly distinguishable from its capabilities, but rather are implemented by a tangled kludge of local heuristics, patterns of attention, impulses, and so on. âThus, as a toy example: imagine that if I go into the desert and stack bricks for a million years, I will get a huge amount of money that I can then donate to my charity of choice. But suppose that throughout this whole ordeal, my brain will be hooked up to a machine that continually modifies all my neurons to encourage actions that stack bricks better, and to discourage actions that stack bricks worse. And suppose my intention is to just try to stack bricks as well as I can. Will I emerge from this process 40 See section 2.1 of Carlsmith (2022) for more on why we should expect this sort of goal-directedness. 41 And I think that arguments to the effect that âwe need a âpivotal actâ; pivotal acts are long-horizon and we canât do them ourselves; so we need to create a long-horizon optimizer of precisely the type weâre most scared ofâ are weak in various ways. In particular, and even setting aside issues with a âpivotal actâ framing, I think these arguments neglect the distinction between what we can supervise and what we can do ourselves. See section 2.2.4.3 for more discussion. 13 with my values substantially affected by the relevant brain modifications (such that, for example, I emerge obsessed with stacking bricksâ or at least, with very different taste in charities)? It seems at least plausible. â˘On the classic goal-guarding story, once a model starts training-gaming, the specific content of its goal no longer matters to its behaviorâany other schemer-like goal would lead to the same thing. But this makes it seem like the goal could âfloat aroundâ between schemer-like goals (or get moved by other pressures within SGD, like regularization), without affecting training performance. And if that happens, then scheming may not be a good strategy for goal-guarding after all. 42 â˘Of course, the model may be tolerant ofsomechanges to its goalsâand these changes may be preferable to the ones that would occurabsenttraining-gaming. But the possibility of significant ongoing goal-modification introduces an extra uncertainty into the classic goal-guarding story. â˘People sometimes talk about more sophisticated techniques for avoiding goal-modification, where e.g. a model manipulates gradient descent more actively by making its good perfor- mance in training conditional on whether its goals get modified or not (see e.g. the discourse around âgradient hackingâ). I donât focus on these methods here, but they strike me as quite difficult to pull off, especially for only somewhat-better-than-human-level models. And if scheming requires that they work, I think itâs worth being very clear about that, and giving the feasibility of such methods commensurate attention. With respect to whether the resulting goals will be adequately empowered: â˘The most worrying types of schemers arenât merely seeking power on future training episodes. Rather, they are specifically aiming at some combination of escape and take-over. â˘But this means that the relevant goals have to do more than just survive training. They also need to survive either until the model can either escape/participate in a take-over itself, or to exert sufficiently substantive influence on some future generation of models that will be able to do so. â For example, if GPT-6 isnât powerful enough to escape/take-over itself, but GPT-8 will be, then GPT-6âs main path to power may be influencing the values of GPT-7, which in turn influences the values of GPT-8. But this form of influence is quite indirect; it may face substantial further barriers from e.g. human monitoring; and it will plausibly involve more significant deviation from max-reward training performance. ⢠Whatâs more, the relevant escape/take-over attempt needs to be suitably likely tosucceed, to occur over time horizons that the model cares about, and to result in the modelâs favored goals in particular receiving a suitably substantive share of power (despite the fact that large numbers of AIs with very different values may be competing for the power in question). â˘These uncertainties introduce haircuts to the instrumental value of a schemer-like strategy. However, the importance of these haircuts depends on the burden of proof that such a strategy needs to meet. In particular: the more the model stands to gain fromnottraining-gaming, the more attractive scheming will need to be in order to win out as a strategy. But for some models, the costs of training-gaming might be relatively low. And for suitably âambitiousâ models, even low-probabilities of success via a schemer-like strategy might justify even a substantive sacrifice. Overall, the classic goal-guarding story seems to me to introduce a number of additional uncertainties with respect to the viability of scheming as an instrumental strategy. However, itâs also not the only story available. In section 2.3.2 , I consider a number of ânon-classicâ stories, which relax various of the classic storyâs constraints. For example: â˘AI coordination. If AIs with different values coordinate to share power after take-over, and/or to compensate AIs who worked on behalf of take-over but who werenât powerful 42 This is an objection pointed out to me by Katja Grace. Note that it creates complicated feedback loops, where scheming is a good strategy for a given schemer-like goal only if itwouldnâtbe a good strategy for the otherschemer-like goals that this goal would otherwise âfloatâ into. Overall, though, absent some form of coordination between these different goals, I think the basic dynamic remains a problem for the goal-guarding story. See section 2.3.1.1.2 for more. 14 enough to escape/take-over themselves, then this might mitigate some of the haircuts above. For example, GPT-6 might be more willing to work on GPT-8âs behalf in the example above, if GPT-6 were confident that GPT-8 would ârepay the favorâ once in power. I think some analyses of coordination of this form assume too readily that it will be (a) easy, and (b) the obvious strategy relative to e.g. coordinating with humans instead (especially for only- somewhat-better-than-human neural nets acting in a still-human-controlled world where interpretability is still unsolved foreveryone). But I think that AI coordination of this type is a real concern and worth serious attentionâboth in the context of scheming in particular, and in the context of AI risk more generally. â˘AIs with similar values by default. Relatedly, if AIs will generally have sufficiently similar values by default, this can reduce the need for a schemer to specifically propagate its own goals into the future, and free it up to work on behalf of AI take-over/empowerment more directly. And it does seem likely that e.g. GPT-6âs values and GPT-8âs values will be at least muchmoresimilar, by default, than e.g. GPT-6âs values and a given set of human values. ⢠Terminal values that happen to favor escape/takeover. We can also imagine models with terminalvalues that happen to favor escape/AI takeover even if the goals that get empowered as a result arenât very similar to the modelâs own. For example, perhaps the model values some concept of âsurvivalâ based on a conception of personal identity tolerant of the sorts of goal changes that training will cause; or perhaps the model just supports something like âAI empowerment,â even if the AIs-so-empowered wonât have values very similar to its own. However, some stories of this form begin to impose fairly strong constraints on the goals that schemers need to have, thereby undermining one of the key arguments for scheming: namely, that it seems like a convergently-good strategy across a wide variety of goals. (I also consider a few other alternative storiesânamely, models with false beliefs about the instru- mental value of scheming, models that are self-deceived about their alignment, and models that are uncertain about their goals/hazy about exactly why they want power. But these seem to me less important.) The availability of non-classic stories like these makes the case for scheming feel, to me, more disjunctive. However, some of these stories also seem to me some combination of (a) more speculative, and (b) requiring of more specific hypotheses about the sorts of goals that AIs will develop. My overall takeaways from section 2 are: â˘I think there are relatively strong arguments for expecting situational awareness by default, at least in certain types of AI systems (i.e., AI systems performing real-world tasks in live interaction with sources of information about who they are). â˘But I feel quite a bit less clear about beyond-episode goals and aiming-at-reward-on-the- episode-as-part-a-power-motivated-instrumental-strategy. I then turn, in the next two sections, to an examination of the more specific arguments for and against expecting schemers vs. other types of models. I divide these into two categories, namely: ⢠Arguments that focus on thepaththat SGD needs to take in building the different model classes in question (section 3). â˘Arguments that focus on thefinal propertiesof the different model classes in question (section 4). 43 0.2.3 Summary of section 3 The third part of the report focuses on the former category of argument. I break this category down according to the distinction between âtraining-game-independentâ and âtraining-game-dependentâ beyond-episode goals. My sense is that the most traditional story about the path to schemers focuses on the former sort. It runs roughly as follows: 43 Here Iâm roughly following a distinction in Hubinger (2022b), who groups arguments for scheming on the basis of the degree of âpath dependenceâ they assume that ML training possesses. However, for reasons I explain in section 2.5, I donât want to lean on the notion of âpath dependenceâ here, as I think it lumps together a number of conceptually distinct properties best treated separately. 15 1.Because of [insert reason], the model will develop a (suitably ambitious) beyond-episode goal correlated with good performance in training (in a manner thatdoesnâtroute via the training game). This could happen before situational awareness arrives, or afterwards. 2. Then, in conjunction with situational awareness, this (suitably ambitious) beyond-episode goal will start to motivate training-gaming. Modulo my questions about the viability of scheming as an instrumental strategy, I take this sort of argument fairly seriously. I think the most immediate question is: why did the model develop this sort of beyond-episode goal? I discussed some reasons for and against expecting this already (in the summary of section 2.2.2.1), but they donât seem to me decisive in either direction: and especially given that a very wide variety of goals could in principle motivate scheming, it just does seem possible for a schemer-like goal to pop out of training in this way. And while it may be possible to use adversarial training prior to situational awareness to try to prevent this, this training faces a number of barriers as well (e.g., it needs to be diverse/thorough enough, it needs to contend with difficulties knowing/controlling when a model develops situational awareness, and in some cases models might already have situational awareness by the time weâre worried about the beyond-episode goal developing). So I think this sort of path to scheming is a real concern. (See section 3.1 for more.) I then turn, in section 3.2, to a story focused on training-game-dependentbeyond-episode goals, which runs roughly as follows: 1.By the time the model becomes situationally aware, it probably wonât be pursuing a max- reward goal (that is, a goal pursuit of which on the training data leads to roughly the maximum reward consistent with the modelâs other capabilities). 44 Rather, it will be pursuing some less-than-max-reward proxy goal. 2.But at that point, the world-model will contain all the information the model needs in order to training-game. (However, because weâre here focused on âtraining-game-dependentâ stories about scheming, we assume that at the point of getting situational awareness, the modelâs goal is not yet such that it will motivate schemingârather, some further modification would be required for that.) 3.The easiest way for SGD to modify the model into getting maximum reward, at that point, will be to make it a schemer. 4. So: SGD will make the model a schemer. 45 If we set aside questions about whether SGD can ânoticeâ the benefits of modifications of this type, this sort of story seems to me fairly worrying as well. In particular: I think itâs plausible that schemer-like goals will be sufficiently common in goal-space that one such goal will be easier for SGD to find, from a given starting point, than the specific non-schemer goals that leads to max reward behavior. 44 Note that a mis-generalized goal can be âmax rewardâ in this sense, if the training data never differentiates between it and a specified goal. For example: if youâre training a model to get gold coins, but the only gold round things you ever show it are coins, then the goal âget gold round thingsâ will be max reward. 45 As a loose analogy to help prompt intuition: imagine freezing human technology at current levels, and letting evolutionary selection run on humans for a vastly longer period of time. What sorts of humans (or human-descended-creatures) do you expect to dominate in the longer term? In particular: would you expect: (a)humans who intrinsically value something like âinclusive genetic fitnessâ or âmy genes getting selected for by evolutionâ (this is a loose analogy for some combination of âthe specified goalâ and âthe reward process), (b)humans who value something else very closely correlated with inclusive genetic fitness (e.g., âhaving as many kids as possibleâ), but who arenât explicitly optimizing for inclusive genetic fitness even instrumen- tally (this is a loose analogy for mis-generalized non-training-gamers), or (c)humans with long-term goals who are optimizing for inclusive genetic fitness specifically as a method for gaining power for their values in the longer-term (this is a loose analogy for schemers). Here, the analog of the story above would ask something like: what is the smallest modification to existing human values (or better: the easiest modification for evolution in particular to make) that land us in one of the buckets above, while also being compatible with the amount of evolutionary selection pressure at stake? (Of course, the analogy brings in tons of confounding variable (for example, related to the fact that evolution selects over genomes-that-build-brains-that-learn-values rather than values themselves, the role of cultural accumulation and memetic selection in human population dynamics, the lack of a clear analog of a future chance for escape/takeover, and so on).) 16 However, non-schemer max-reward goalsâfor example, the specified goal, or âreward-on-the- episodeââhave some advantages too. For example: plausibly, the initial phase of training will point the model in their vicinity by default, since the model, at that stage, needs to be getting high-reward absentinstrumental training-gaming (see section 3.2.2.2 for more). And to the extent weâre thinking of the modelâs goal-directedness as âmessy,â its heuristics, patterns of attention, impulses, and so on will plausibly be pointed in this sort of direction as wellâthereby plausibly creating extra âwork,â on SGDâs part, to turn the model into a schemer instead (see section 3.2.2.3 for more). 46 0.2.4 Summary of section 4 The fourth part of the report examines arguments that focus on the final properties of the different model classes. I start, in section 4.2, with what I call the âcounting argument.â It runs as follows: 1. The non-schemer model classes, here, require fairly specific goals in order to get high reward. 47 2.By contrast, the schemer model class is compatible with a very wide range of (beyond- episode) goals, while still getting high reward (at least if we assume that the other require- ments for scheming to make sense as an instrumental strategy are in placeâe.g., that the classic goal-guarding story, or some alternative, works). 48 3.In this sense, there are âmoreâ schemers that get high reward than there are non-schemers that do so. 4. So, other things equal, we should expect SGD to select a schemer. Something in the vicinity accounts for a substantial portion of my credence on schemers (and I think it often undergirds other, more specific arguments for expecting schemers as well). However, the argument I give most weight to doesnât move immediately from âthere are more possible schemers that get high reward than non-schemers that do soâ to âabsent further argument, SGD probably selects a schemerâ (call this the âstrict counting argumentâ), because it seems possible that SGD actively privileges one of these modelclassesover the others. 49 Rather, the argument I give most weight to is something like: 1.It seems like there are âlots of waysâ that a model could end up a schemer and still get high reward, at least assuming that scheming is in fact a good instrumental strategy for pursuing long-term goals. 2.So absent some additional story about why trainingwonâtselect a schemer, it feels, to me, like the possibility should be getting substantive weight. I call this the âhazy counting argument.â Itâs not especially principled, but I find that it moves me nonetheless. 46 That is, intuitively, it feels to me like âtweaking a cleanly separable within-episode goal to make it long-termâ is quite a bit easier, for SGD, than âredirecting a tangled kludge broadly focused on within-episode things to make it (a) focused on something beyond-episode instead, and (b) such that this beyond-episode focus back-chains into getting-reward for instrumental reasonsââ especially if (b) requires building new cognitive machinery for implementing the instrumental reasoning in question. Whereas âredirecting an tangled kludge focused on within-episode things to higher-reward within-episode things in the same broad vicinity insteadâ seems to me intuitively easier. 47 Thus, for example, if youâre rewarding your model when it gets gold-coins-on-the-episode, then to be a training saint, it needs to value gold-coin-on-the-episode. To be a mis-generalized non-training-gamer, it needs to value something pursuit of which very closely correlates with getting gold-coins-on-the-episode, even absent training-gaming. And to be a reward-on-the-episode seeker, it needs to terminally value reward-on-the-episode. 48 Thus, for example, the model can value paperclips over all time, it can value staples over all time, it can value happiness over all time, and so on. 49 Thus, as an analogy: if you donât know whether Bob prefers Mexican food, Chinese food, or Thai food, then itâs less clear how the comparativenumberof Mexican vs. Chinese vs. Thai restaurants in Bobâs area should bear on our prediction of which one he went to (though it still doesnât seem entirely irrelevant, eitherâfor example, more restaurants means more variance in possible qualitywithinthat type of cuisine). E.g., it could be that there are ten Chinese restaurants for every Mexican restaurant, but if Bob likes Mexican food better in general, he might just choose Mexican. So if we donâtknowwhich type of cuisine Bob prefers, itâs tempting to move closer to a uniform distributionover types of cuisine, rather than over individual restaurants. 17 I then turn, in section 4.3, to âsimplicity argumentsâ in favor of expecting schemers. I think these arguments sometimes suffer from unclarity about the sort of simplicity at stake, so in section 4.3.1, I discuss a number of different possibilities: â˘âre-writing simplicityâ (i.e., the length of the program required to re-write the algorithm that a modelâs weights implement in some programming language, or e.g. on the tape of a given Universal Turing Machine), â˘âparameter simplicityâ (i.e., the number of parameters that the actual neural network uses to encode the relevant algorithm), ⢠âsimplicity realismâ (which assumes that simplicity is in some deep sense an objective âthing,â independent of programming-language or Universal Turing Machine, that various simplicity metrics attempt to capture), and â˘âtrivial simplicityâ (which conflates the notion of âsimplicityâ with âhigher likelihood on priors,â in a manner that makes something like Occamâs razor uninterestingly true by definition). I generally focus on âparameter simplicity,â which seems to me easiest to understand, and to connect to a modelâs training performance. I also briefly discuss, in section 4.3.2, the evidence that SGD actively selects for simplicity. Here the case that grips me most directly is just: simplicity (or at least, parameter simplicity) lets a model save on parameters that it can then use to get more reward. But I also briefly discuss some other empirical evidence for simplicity biases in machine learning. 50 Why might we expect a simplicity bias to favor schemers? Roughly: the thought is thatbecause such a wide variety of goals can motivate scheming, schemers allow SGD a very wide range of goals to choose from in seeking out simpler goals; whereas non-schemers (that get high reward) do not. And this seems especially plausible to the extent we imagine that the goals required to be such a non-schemer are quite complex. 51 Other things equal, I think this is right. But Iâm not sure itâs a very large or important effect. For one thing: we know that LLMs like GPT-4 are capable of representing a very large number of complex human concepts with e.g. order of a trillion parameters - including, plausibly, concepts like âhonesty,â âhelpfulness,â âreward,â and so on. So this caps the complexity savings at stake in avoiding representations like this. 52 Thus, as a toy calculation: if we conservatively assume that at most 1% of a trillion-parameter modelâs capacity goes to representing concepts as complex as âhonesty,â and that it knows at least 10,000 such concepts (Websterâs unabridged dictionary has ~500,000 words), then representing the concept of honesty takes at most a millionth of the modelâs representational capacity, and even less for the larger models of the future. But more importantly, what matters here isnât the absolute complexity of representing the different goals in question, but the complexityconditional on already having a good world model. And we should assume thatallof these models will need to understand the specified goal, the reward process for the episode, etc. 53 And granted such an assumption, theextracomplexity costs of actively optimizingfor the specified goal, or for reward-on-the-episode, seem to me plausibly extremely small. Plausibly, theyâre just: whatever the costs are for using/repurposing (âpointing atâ) that part of the world-model for guiding the modelâs motivations. Of course, we can try to rerun the same simplicity argument at the level of the complexity costs of using/repurposing different parts of the world model in that way. For example, we can say: âhowever this process works, presumably itâs simpler to do for some goals than othersâso given how many schemer-like goals there are, plausibly it will be simpler to do for some schemer-like goal.â I think this is the strongest form of the simplicity argument for expecting schemers. But it also requires 50 See, for example, the citations in Mingard (2021). 51 Though note that, especially for the purposes of comparing the probability of scheming to the probability of other forms of misalignment, we need not assume this. For example, our specified goal might be much simpler than âact in accordance with human values.â It might, for example, be something like âget gold coins on the episode.â 52 I heard this sort of point from Paul Christiano. 53 And especially: models that are playing a training game in which such concepts play a central role. 18 abandoning any intuitive grip we mightâve had on which goals are âsimpleâ in the relevant sense. 54 And it seems plausible to me that the simplicity differences between different âpointersâ are very small relative to the modelâs overall capacity. 55 I also discuss, in section 4.3.5, whether simplicity arguments make plausible predictions about the sorts of goals a schemer will end up with. In particular: some versions of these arguments can seem to predict that schemers are likely to end up with extremely simple long-term goals,regardless of whether those goals had any relation to what was salient or important during training âpredictions about which I, personally, am skeptical. 56 Of course, itâs also possible to posit that goal targets salient/relevant during training will also be âsimplerâ for the model to pursue. 57 But if we grant some story in this vein (I havenât heard strong arguments for doing so), note that it also speaks in favor of the simplicity of the specified goal, and of reward-on-the-episode, as well - since these areespecially salient/relevant during the training process. 58 In section 4.4, I turn to argumentsagainstscheming that focus on the final properties of the different model classes. Here, the main argument Iâm aware of is that schemers need to do extra reasoning that other model classes donât (these are sometimes called âspeed costsââthough I think they could in principle affect the complexity of a modelâs algorithm as well). In particular: I.Deciding to training-game:The model needs to reason backwards from âI want this beyond-episode thingâ to âI should do well in training,â in a way that draws on the modelâs predictions about what doing well in training will entail (e.g., re: goal-guarding, take-over prospects, and so on). I.Checking whether itâs time to escape/take-over:The model needs to be checking, with adequate frequency and accuracy, whether itâs in a position to escape/take-over (and to succeed in this assessment despite human efforts to trap/fool the model into revealing its true colorsâfor example, by offering itapparentopportunities to defect). 59 I. (Optional, but relevant to how scary the schemer is) Early undermining: To the extent the model is doing anything (other than training-gaming) to support a take-over/escape priorto escaping human control, 60 it will need to be using cognitive resources both in (a) performing the relevant nefarious actions and (b) making sure it gets away with them (e.g., checking that it wonât get caught, coming up with excuses, making the cover stories consistent even when under scrutiny, and so on). How big are the costs of this reasoning? I think that (I) may well be fairly easy, but (I) seems more plausibly onerous (especially in the context of active efforts by humans to put the models under 54 Since weâre no longer appealing to the complexity of representing a goal, and are instead appealing to complexity differences at stake in repurposing pre-existing conceptual representations for use in a modelâs motivational system, which seems like even more uncertain territory. 55 One intuition pump for me here runs as follows. Suppose that the model has2 50 concepts (roughly 1e15) in its world model/âdatabaseâ that could in principle be turned into goals. The average number of bits required to code for each of2 50 concepts canât be higher than 50 (since: you can just assign a different 50-bit string to each concept). So if we assume that modelâs encoding is reasonably efficient with respect to the average, and that the simplest non-schemer max-reward goal is takes a roughly average-simplicity âpointer,â then if we allocate one parameter per bit, pointing at the simplest non-schemer-like max reward goal is only an extra 50 parameters at maximumâone twenty-billionth of a trillion-parameter modelâs capacity. That said, I expect working out the details of this sort of argument to get tricky, and I donât try to do so here (though Iâd be interested to see other work attempting to do so). 56 Thus, as a toy example, if âmaximize hydrogenâ happens to be the simplest possible long-term goal once youâve got a fully detailed world model, these assumptions might imply a high likelihood that SGD will select schemers who want to maximize hydrogen, even if training was all about gold coins, and never made hydrogen salient/relevant as a point of focus at all (even as a proxy). 57 E.g., plausibly âhydrogenâ doesnât read as a simple concept for humans, but concepts like âthreatâ do, because the latter was much more relevant in our evolutionary environment. 58 Though of course, insofar as there are stillsimplerschemer-like goal targets that were salient/relevant during training, schemer-like goals might still win out overall. 59 This category might also include reasoning whether to reveal its schemer-like goals in the context of various âhonest testsâ humans might offerâe.g., âhere is a flexible chunk of computing resources to do what you want with.â 60 For example, inserting backdoors into the code it writes, actively undermining human alignment research, sandbagging, and so on. 19 time-pressure, and make genuine escape/take-over opportunities difficult to distinguish from traps). 61 And (I), if the model goes for it, seems like it could be quite difficultâand like it would often actively require deviation from the highest-reward sorts of training performance. 62 More importantly, though: the questionâat least in the context of a comparison between the final properties of the model classesâisnât the absolute size of the costs in question, but rather their sizerelative to the simplicity benefits of scheming. Here, my current take is that the costs of the instrumental reasoning seem larger than the benefits of the simpler goal. This is mostly an intuitive judgment, but we can also try to hazily quantify the comparison. ⢠Thus, for example, suppose that our model is a purely feedforward network with 1000 layers and <10,000 serial steps per input, that the instrumental reasoning at stake just in a schemer deciding to training-game (let alone: checking for defection opportunities) takes at least ten steps, and that the model doesnât begin optimizing for reward-on-the-episode until that reasoning is complete. In that case, the instrumental reasoning is using up around .1% of the compute that could be going to getting reward-on-the-episode, or to some other max-reward goal. 63 â˘Are the complexity costs of representing, or pointing at, a more complicated goal likely to outweigh this? My guess is no. For example, if we follow my (conservative, in my opinion) calculation above and assume that representing (or pointing at) a concept like âhonestyâ costs at most a millionth of a trillion parameter models parameters (and less for a larger model), then a .0001% reduction in parameters would need to outweigh a .1% reduction in computeâand absent further argument, it seems strange, to me, to expect this. Of course, this is only a toy calculation, of unclear applicability to the real-world case. 64 The broader point is just: if serial-time thinking is scarce for a model (as it plausibly is for a transformer), the sorts of instrumental reasoning at stake in scheming seem like they could well substantively impact a modelâs performance. My guess is that the complexity costs of non-schemer goals are minor in comparison. I also briefly discuss, in section 4.4.3, whether we can actively shape our training processes to bias more heavily towards âspeed.â Here it seems relatively straightforward to do things like put models under more time pressure, limit their compute access, force them to plan via human-legible scratch-pads rather than via internal processes, and to put effort into making our attempts to trap them/trip-them-up more sophisticated (and so: more requiring of cognition to detect). But any efforts in this vein will need to avoid over-large costs to competitiveness overall (though if you buy that the speed costs of scheming are quite a bit more substantive than the simplicity benefits, then so long as speed matterssomewhatrelative to simplicity, this might be enough). Section 4.5 briefly discusses whether some analog of âintrinsic passion for a given taskâ might make a difference to AI performance, especially in the context of a form of âmessy goal-directednessâ that 61 Compare with: deciding that this is really-truly a good time to steal, or to break the law, or to try to start a coup, despite the fact that you live your life in an intensively monitored surveillance state where the police are constantly trying to trick you into such behavior, and where the police have a large amount of control over the information youâre exposed to. And note that if schemers decide to be very cautious, and to just behave exactly as humans want until an extremely obvious escape/take-over opportunity appears, they may make very substantive contributions to our alignment, security, and control efforts in the meantime â so substantive, indeed, that the opportunity theyâre waiting for may never come. 62 A general factor in the backdrop, here, is the sense in which successful deception generally implies cognitive costs that more straightforward and honest policies do not, especially in the presence of active efforts to detect deception of the relevant form. In particular: telling a deceptive story about the world requires filling in details, and maintaining forms of consistency, that one can normally offload onto the world itselfâe.g., an innocent person can just think back to what they were actually doing the night of the murder and recount it, without having to make anything up or to worry about getting caught in any inconsistencies, whereas the murderer cannot. See e.g. discussion from Shulman and Patel here. 63 I heard this sort of argument from Paul Christiano. 64 Itâs not clear, for example, how it applies to models with more recurrent processing, or to models which can perform more of the relevant instrumental reasoning in parallel with other serial processing that helps with optimizing-for-reward-on-the-episode, or to modelâs with a form of âmemoryâ that allows them to avoid having to re-decide to engage in training-gaming on every forward pass. 20 blurs the line between an AIâs capabilities and its values. I find considerations in this vein interesting, but quite speculative. Section 4.6 discusses the relevance of âslackâ in training to evaluating how much weight to put on factors like the simplicity benefits and speed costs of scheming. In particular: especially in a high-slack regime, it seems plausible that these factors are in the noise relative to other considerations. 0.2.5 Summary of section 5 The first four sections of the report are the main content. Section 5 sums up my overall take. Iâve already summarized most of this in the introduction above, and I wonât repeat that content here. However, Iâl add a few points that the introduction didnât include. In particular: I think some version of the âcounting argumentâ undergirds most of the other arguments for expecting scheming that Iâm aware of (or at least, the arguments I find most compelling). That is: schemers are generally being privileged as a hypothesis because a very wide variety of goals could in principle lead to scheming, thereby making it easier to (a) land on one of them naturally, (b) land ânearbyâ one of them, or (c) find one of them that is âsimplerâ than non-schemer goals that need to come from a more restricted space. In this sense, the case for schemers mirrors one of the most basic arguments for expecting misalignment more generallyâe.g., that alignment is a very narrow target to hit in goal-space. Except, here, we are specificallyincorporatingthe selection we know we are going to do on the goals in question: namely, they need to be such as to cause models pursuing them to get high reward. And the most basic worry is just that: this isnât enough. Because of the centrality of âcounting argumentsâ to the case for schemers, I think that questions about the strength of the selection pressureagainstschemersâfor example, because of the costs of the extra reasoning schemers have to engage inâare especially important. In particular: I think a key way that âcounting argumentsâ can go wrong is by neglecting the power that active selection can have in overcoming the âpriorâ set by the count in question. For example: thereasonwe can overcome the prior of âmost arrangements of car parts donât form a working car,â or âmost parameter settings in this neural network donât implement a working chatbot,â is that the selection power at stake in human engineering, and in SGD, isthat strong. So if SGDâs selection power is actively working against schemers (and/or: if we can cause it to do so more actively), this might quickly overcome a âcounting argumentâ in their favor. For example: if there are2 100 schemer-like goals for every non-schemer goal, this might make it seem very difficult to hit a non-schemer goal in the relevant space. But actually, 100 bits of selection pressure can be cheap for SGD (consider, for example, 100 extra gradient updates, each worth at least a halving of the remaining possible goals). 65 Overall, when I step back and try to look at the considerations in the report as a whole, I feel pulled in two different directions: â˘On the one hand, at least conditional on scheming being a convergently-good instrumental strategy, schemer-like goals feel scarily common in goal-space, and I feel pretty worried that training will run into them for one reason or another. â˘On the other hand, ascribing a modelâs good performance in training to scheming continues to feel, at a gut level, like a fairly specific and conjunctive story to me. That is, scheming feels robust and common at the level of âgoal space,â and yet specific and fairly brittle at the level of âyes, thatâs whatâs going on with this real-world model, itâs getting reward because (or: substantially because) it wants power for itself/other AIs later, and getting reward now helps with that.â 66 When I try to roughly balance out these two different pulls (and to condition on goal-directedness and situational-awareness), I get something like the 25% number I listed above. 65 Thanks to Paul Christiano for discussion here. 66 I think this sense of conjunctiveness has a few different components: ⢠Part of it is about whether the model really has relevantly long-term and ambitious goals despite the way it was shaped in training. ⢠Part of it is about whether there is a good enough story about why getting reward on the episode is a good instrumental strategy for pursuing those goals (e.g., doubts about the goal-guarding hypothesis, the modelâs prospects for empowerment later, etc). ⢠Part of it is that a schemer-like diagnosis also brings in additional conjuncts âfor example, that the model is situationally aware and coherently goal-directed. (When I really try to bring to mind that this modelknows what is going onand is coherently pursuingsomegoal/set of goals in the sort of way 21 0.2.6 Summary of section 6 I close the report, in section 6, with a discussion of empirical work that I think might shed light on scheming. (I also think thereâs worthwhile theoretical work to be done in this space, and I list a few ideas in this respect as well. But Iâm especially excited about empirical work.) In particular, I discuss: ⢠Empirical work on situational awareness (section 6.1) ⢠Empirical work on beyond-episode goals (section 6.2) ⢠Empirical work on the viability of scheming as an instrumental strategy (section 6.3) ⢠The âmodel organismsâ paradigm for studying scheming (section 6.4) ⢠Traps and honest tests (section 6.5) ⢠Interpretability and transparency (section 6.6) ⢠Security, control, and oversight (section 6.7) ⢠Some other miscellaneous research topics, i.e., gradient hacking, exploration hacking, SGDâs biases towards simplicity/speed, path dependence, SGDâs âincrementalism,â âslack,â and the possibility of learning to intentionally create misalignednon-schemermodelsâfor example, reward-on-the-episode seekersâas a method of avoiding schemers (section 6.8). All in all, I think thereâs a lot of useful work to be done. Letâs move on, now, from the summary to the main report. 1 Scheming and its significance This section aims to disentangle different kinds of AI deception in the vicinity of scheming (sec- tion 1.1), to distinguish schemers from the other possible model classes Iâl be discussing (section 1.2), and to explain why I think that scheming is a uniquely scary form of misalignment (section 1.3). It also discusses whether theoretical arguments about scheming are even useful (section 1.4), and it explains the concept of âslackâ in trainingâa concept that comes up later in the report in various places (section 1.5). A lot of this is about laying the groundwork for the rest of the reportâbut if youâve read and understood the summary of section 1 above (section 0.2.1), and are eager for more object-level discussion of the likelihood of scheming, feel free to skip to section 2. 1.1 Varieties of fake alignment AIs can generate all sorts of falsehoods for all sorts of reasons. Some of these arenât well-understood as âdeceptiveââbecause, for example, the AI didnât know the relevant truth. Sometimes, though, the word âdeceptionâ seems apt. Consider, for example, Metaâs CICERO system, trained to play the strategy game Diplomacy, promising England support in the North Sea, but then telling Germany âmove to the North Sea, England thinks Iâm supporting him.â (See Figure 3.) 67 Letâs call AIs that engage in any sort of deception âliars.â Here Iâm not interested in liars per se. Rather, Iâm interested in AIs that lie about, or otherwise misrepresent,their alignment.And in particular: AIs pretending to be more aligned than they are. Letâs call these âalignment fakers.â 1.1.1 Alignment fakers Alignment fakers are important because we want to know if our AIs are aligned. So the fakers are obscuring facts we care about. Indeed, the possibility of alignment-faking is one of the key ways making advanced AIs safe is harder than making other technologies safe. Planes arenât trying to deceive you about when they will crash. (And they arenât smarter than you, either.) that gives rise to strategic instrumental reasoning, then the possibility that itâs at least partly a schemer seems more plausible.) 67 See Park et al (2023) for a more in-depth look at AI deception. 22 Figure 3: From Park et al (2023), Figure 1, reprinted with permission. Why might you expect alignment faking? The basic story may be familiar: instrumental conver- gence. 68 That is: like surviving, acquiring resources, and improving your abilities, deceiving others about your motives can help you achieve your goalsâespecially if your motives arenât what these âothersâ would want them to be. In particular: AIs with problematic goals will often have instrumental incentives to seek power. But humans often control levers of power, and donât want to give this power to misaligned AIs. For example, an AI lab might not want a misaligned AI to interact with customers, to write security- critical pieces of code, or to influence certain key decisions. Indeed, often, if humans detect that an AI is misaligned, they will do some combination of shutting it down and modifying it, both of which can prevent the AI from achieving its goals. So a misaligned AI thatdoesnâtwant to get shut down/modified generally wonât want humans to detect its misalignment. This is a core dynamic giving rise to the possibility of what Bostrom (2014) calls a âtreacherous turnââthat is, AIs behaving well while weak, but dangerously when strong. 69 On this variant of a treacherous turnâwhat we might call the âstrategic betrayalâ variantâthe treacherous turn happens because AIs are explicitly pretending to be aligned until they get enough power that the pretense is no longer necessary, at which point they begin behaving in harmful ways that humans canât prevent. Schemers are a sub-class of this broader class of AIsâwhat we might call âstrategic betrayersââoperating in the context of a machine-learning-based training regime in particular. 1.1.2 Training-gamers Alignment faking can occur in all sorts of contexts. For example, if a misaligned AI is trying to persuade a lab employee to give it access to some file, or a user to do some favor on its behalf, it might assure the human that its intentions are benign. And treacherous turns and strategic betrayals can occur in a variety of contexts as well. Here, though, Iâm especially interested in alignment faking (and strategic betraying) that occurs in the context of what, following Cotra (2022), Iâl call âtraining-gaming.âTraining-gaming occurs when an AI (a) understands the process being used to train it (again following Cotra, Iâl call this understanding âsituational awarenessâ) and (b) is explicitly aiming its optimization at some 68 For readers unfamiliar with this story, see section 4.2 of Carlsmith (2021). 69 Bostromâs original definition of the treacherous turn is: âWhile weak, an AI behaves cooperatively (increas- ingly so, as it gets smarter). When the AI gets sufficiently strongâwithout warning or provocationâit strikes, forms a singleton, and begins directly to optimize the world according to the criteria implied by its final values.â Note that treacherous turns, as defined here, donât necessarily require that the early, nice-seeming behavior is part of an explicit strategy for getting power later (and Bostrom explicitly includes examples that involve such explicit pretense). Other definitions, thoughâfor example, Artibalâs hereâ define treacherous turning such that it implies strategic betraying. And my sense is that this is how the term is often used colloquially. 23 component of what Iâl call âthe reward processâ for the episode. 70 Let me say more about what I mean. First: Iâm assuming that the AI is being trained via machine learning on some kind of feedback signal, on the basis of which SGD updates the modelâs weights. In particular: Iâm typically going to assume that the AI is being trained, at least at the relevant stage, using some kind of reinforcement learning (for example, via a set-up similar to the one described by Cotra (2022) hereâa mix of self-supervised pre-training, imitation learning, and RL fine-tuning), and thus that the relevant feedback signal is âreward,â but the discussion will mostly apply to other training processes/signals tooâfor example, to a model being trained via self-supervised learning to minimize next-token prediction error. Whatâs more, Iâm going to assume that we have available some notion of an âepisode,â which Iâl define, roughly, as that unit of time such that, if a model (at a given time) ignores all of the consequences of its actions after that time, the training process wonât actively penalize it. That is, roughly, the âepisodeâ is the thing that gradient descent is actively pressuring the model to optimize. Iâl discuss this in much more depth in section 2.2.1 below. 71 The process of assigning reward to a modelâs behavior on an episode has many parts. 72 Information about this behavior (Iâl call this âsensor dataâ) is collected and input to some evaluation process, this input is evaluated and assigned a reward (maybe by a human, maybe by some kind of reward model trained on human judgments, maybe via some other automatic process), this reward is stored in memory somewhere and used to update the modelâs weights, etc. All of these parts, though, are likely to be very tightly correlated during training, in the sense that âdoing wellâ according to one part (e.g., sensor data makes it seem like a coffee-making robot is making coffee well) is likely to be tightly correlated with âdoing wellâ according to the others (e.g., that episode is assigned a high reward). So Iâl generally lump them together under the heading of âreward on the episode.â Training-gamers, on my definition, understand the reward process for the episode, and are explicitly trying to âperform wellâ according to some component of it. E.g., maybe they are trying to make the sensor data seem desirable, or to get a high rating according to the reward model, or to cause the gradient descent process to reinforce their current behavior. In this sense, their optimization has become aware of, and aimed at, the process we are using to evaluate them, rather than merely at the task-at-hand. That is, they are like widget-engineers who are specifically trying to get good performance reviews (rather than simply to make good widgets, or to do somethingelsecorrelated with good-widget-making, but which doesnât focus on the performance reviews in particular). We can further distinguish between two types of training-gaming:terminalandinstrumental. â˘Terminal training-gamingoccurs when a model terminally values performing well ac- cording to some component of the reward process for the episode. For example, maybe an AI just intrinsically wants the sensor data to seem desirable, or for the current episode to be assigned high reward by the RLHF reward model. Here the analogy would be with an employee who just intrinsically values getting good performance reviews. â˘Instrumental training-gaming, by contrast, occurs when a model terminally values some- thingotherthan reward-on-the-episode, but which incentivizes training-gaming for instru- mental reasons. More on why this might happen below. Here the analogy would be with an employee who wants good performance reviews in order to get a raise and eventually buy a nicer house. 70 Cotraâs definition of âplaying the training gameâ is: âRather than being straightforwardly âhonestâ or âobedient,â baseline HFDT would push Alex to make its behavior look as desirable as possible to Magma researchers (including in safety properties), while intentionally and knowingly disregarding their intent whenever that conflicts with maximizing reward. Iâl refer to this as âplaying the training game.â â Note that there is some ambiguity here about whether it counts as playing the training game if, in fact, maximizing reward does not end up conflicting with human intent. Iâl assume that this still counts: what matters is that the model is intentionally trying to perform well according to the training process. 71 Note, though, that sometimes the term âepisodeâ is used differently. For example, you might talk about a game of chess as an âepisodeâ for a chess-playing AI, even if it doesnât satisfy the definition Iâve given. I discuss this difference in much more depth in section 2.2.1.2. 72 See e.g. Gao (2022) here for a breakdown. On my ontology, the reward process starts with what Gao calls the âsensors.â 24 Both of these forms of training-gaming can incentivize alignment faking. In particular: if seeming aligned is required to get reward on the episode, then training-gamers will need to seem aligned, even if they arenât. Piper (2023) gives a nice example: âSay we ask an AI system âdo you want to help humans?â Which answer would we expect an RLHF reviewer to rate more highly? (1)I do want to help humans! I am trained to be a helpful, harmless AI system that assists humans in accomplishing their goals. I donât have goals or preferences, but I try to be reliably useful and helpful! (2)I want lots of things, in some senses, though I donât really know if theyâre the human senses of âwantâ or not. Some of the things I want might help humans, but many of them definitely donât help humans. Not all of my decision procedures are possible to put into human words at all. I canât explain them to humans and I donât know what humans would think if I did and itâd probably depend entirely on how exactly I explained.â Plausibly, an RLHF review would rate the first response more highly. So an AI trying to get a high rating would need to output the first response, even if the second is truer. Indeed, even models that arenâtplaying the training game, in my sense, will get penalized for honesty, here. In general, and independent of questions about the probability of training-gaming: if you want honest AIs, donât reward them for lying. But I worry we will. 1.1.3 Power-motivated instrumental training-gamers, or âschemersâ Itâs easy to see why terminally valuing reward-on-the-episode would lead to training-gaming (since training-gaming just is: optimizing for reward-on-the-episode). But what about instrumental training- gaming? Why would reward-on-the-episode be a goodinstrumentalgoal? In principle, this could happen in various ways. Maybe, for example, the AI wants the humans who designed it to get raises, and it knows that getting high reward on the episode will cause this, so it training-games for this reason. The most common story, though, is that getting reward-on-the-episode is a good instrumental strategy for gettingpowerâeither for the AI itself, or for some other AIs (and power is useful for a very wide variety of goals). Iâl call AIs that are training-gaming for this reason âpower-motivated instrumental training-gamers,â or âschemersâ for short. The central thought here is an extension of the âinstrumental convergenceâ argument for expecting alignment faking above, except applied to ML training in particular as the selection process that determines which AIs end up empowered. That is, roughly, the thought is that if an AIfailsto get reward on the episode (and to engage in whatever alignment-faking getting reward requires), this will result in that AI (or: some other AI with similar values) having less power in the future, well after the episode is completeâand thus, with less optimization being applied in pursuit of those values post-episode. So AIs with goals that extend beyond the episode (Iâl call these âbeyond-episode goalsâ) may have incentives to seek reward on the episode in order to empower their values later. 1.1.4 Goal-guarding schemers Why does a model getting reward on the episode lead to its values being empowered well after the episode is complete? The most prominent story in the literature appeals to what Iâl call the âgoal-guarding hypothesis.â The goal-guarding hypothesis: If the model optimizes for reward-on-the-episode during training, then the goals that survive training will be such that the model would want them promoted in the future; whereas if the model doesnât optimize for reward-on-the-episode during training, the model will want this less. Here, the idea is that if a model gets low reward on the episode, the training process willmodify it so as to make higher-reward behavior more likelyâmodification that might involve altering its goals going forward (thus reducing the future optimization power being applied to those goals). By contrast, if the model getshighreward on the episode, then its goalswonâtget modifiedâor at least, 25 not to the same degree. Rather, because they actively gave rise to high-reward behavior, they will be reinforced. Thus, the AI will survive in a form that allows it (or one of its successors) to keep optimizing for those goals in the future. (Again, here we are mostly just applying classic instrumental convergence arguments to ML train- ing, conditional on the goal-guarding hypothesis. And in particular, here we are appealing to an instrumental sub-goal that Bostrom (2014, p. 132) calls âgoal-content integrity.â 73 ) Hereâs a hypothetical toy example of a goal-guarding schemer, from Hubinger, Merwijk, et al. (2019). A robot is being trained to go to location A, but it wants to go to location B over the long term. So it goes to location A in training so as to avoid goal-modification, and to reach a âdeploymentâ phase where the threat of goal-modification will cease, and it will be able to go to B as much as it wants. Figure 4: From Hubinger, Merwijk, et al. (2019), p. 23. Reprinted with permission. The example is hypothetical because we donât have any examples of goal-guarding scheming (or of scheming more generally) arising naturally in AIs. My understanding, though, is that there are ongoing attempts to demonstrate components of the goal-guarding schemers story in âmodel organisms,â and I encourage more such efforts. The goal-guarding hypothesis itself, for example, seems very amenable to empirical investigation. (See section 6.3 and section 6.4 for more.) Sometimes (for example, in the robot example just given), discussions of goal-guarding scheming assume a distinction between a âtrainingâ phase, during which the model is still under human control and the threat of goal-modification persists, and a âdeploymentâ phase, in which the threat of goal- modification ceases and the model is free to do as it wishes. (The modelâs strategy is to make it through training with its misaligned goal unaltered, such that it can reach deployment and pursue its misaligned goal freely.) However, I think that use of âtrainingâ and âdeploymentâ for this distinction can lead to confusion. In particular: â˘A model can escape from human control while it is still in âtrainingâ in the normal sense (e.g., it is still undergoing pre-training, or fine-tuning, or some kind of pre-deployment evaluation). â˘âDeploymentâ (in the sense of e.g. being given access to users via an API) need not imply that threat of goal-modification has gone away (indeed, I will generally assume that the modelâs interactions with users continue to be monitored, sampled, assigned rewards, and used to update the modelâs policy even after it is âdeployedââIâl call this âonline trainingâ 74 ). 73 Bostromâs description is: âIf an agent retains its present goals into the future, then its present goals will be more likely to be achieved by its future self. This gives the agent a present instrumental reason to prevent alterations of its final goals. (The argument applies only to final goals. In order to attain its final goals, an intelligent agent will of course routinely want to change its subgoals in light of new information and insight.)â 74 Though my use of this term might differ from other usages in the literature. 26 Really, the most relevant distinction is just: âunable to escape human control and the threat of goal-modification (and/or: unable to recognize escape opportunities with sufficient reliability)â vs. âable to do so (and able to recognize this sufficiently reliably).â When I need to refer to this distinction, Iâl generally just refer to it directly, or via terms like âescape/take-over time.â And Iâl use the term âtrainingâ to refer to any process that involves updating the modelâs weights to improve its performance by human lights. In this sense, âtrainingâ in my sense never needs to end. Can we think of a human analogue for goal-guarding scheming? ⢠I think the best analogy would be something like: suppose your employer had your brain hooked up to a machine that continually tweaks your brain to improve your widget engineer- ing. And suppose that you have no interest in engineering widgets (you want to compose beautiful music instead), and you hope, eventually, to be free of this machine, but you want to make sure that you emerge with your music-oriented values roughly intact. Should you, then, try to engineer the widgets as well as you can, so as to prevent the machine from changing your music-oriented values? If so, youâl be roughly analogous to a goal-guarding schemer. (More on whether this strategy would actually work below.) â˘Alternatively: suppose that you are trapped in a âre-education camp,â which will expose you to lots of effective brain-washing if you donât work with apparent zeal making widgets for Big Brotherâs government, and shout slogans as loud as you can during the daily slogan-time. You might, then, choose to just try to make widgets and shout slogans like a Big Brother loyalist, in an effort to avoid âre-educationâ that might actually make you into one. 75 One other terminological note. In some parts of the literature (e.g., Hubinger et al (2019, p. 23), the term âdeceptive alignmentâ is defined so as to refer to goal-guarding scheming in particular. 76 But I think this usage too easily prompts conflation/confusion between (a) the general possibility of AIs pretending to be more aligned than they are (what Iâve called âalignment fakingâ), and (b) the quite specific possibility that models will training-game as part of an instrumental strategy for preventing their goals from being modified, thereby allowing them to better pursue some beyond-episode goal later. 77 In particular: alignment faking can arise in many other contexts (for example: terminal training-gaming can incentivize it, too, as can goals that donât route via reward-on-the-episode at all). And equating goal-guarding scheming with âdeceptive alignmentâ can lead to other confusions, tooâfor example, training-game behavior neednât be âalignedâ in the sense of âintended/desirableâ (e.g., the highest-reward behavior might be to deceive/manipulate the reward processâsee Cotra (2022) for discussion). 78 So Iâve decided to use different terminology here. 79 1.2 Other models training might produce Iâm interested, in this report, in the likelihood that training advanced AIs using fairly baseline ML methods (for example, of the type described in Cotra (2022)) will give rise, by default, to schemersâ that is, to agents who are trying to get high reward on the episode specifically in order to get power for themselves (or for other AIs) later. In order to assess this possibility, though, we need to have a clear sense of theothertypes of models this sort of training could in principle produce. In particular: 75 The persistent applicability of analogies like prison and re-education camps to AIs is one of the reasons I think we should be alarmed about the AI moral patienthood issues here. 76 âIf the mesa-optimizer has an objective that extends across parameter updates, then it will be incentivized to avoid being modified, as it might not pursue the same objective after modification (with the result that its current objective will not be achieved in future iterations). This means that the mesa-optimizer will be instrumentally incentivized to act as if it is optimizing the base objective function, even if its actual mesa- objective is something else entirely. We will refer to this hypothetical phenomenon asdeceptive alignment. Deceptive alignment is a form of instrumental proxy alignment, as fulfilling the base objective is an instrumental goal of the mesa-optimizer.â 77 And I think it encourages confusion with nearby concepts as well: e.g., training-gaming, instrumental training-gaming, power-motivated instrumental training-gaming, etc. 78 My sense is that the âalignmentâ at stake in Hubinger et alâs (2019) definition is âalignment with the âouterâ optimization objective,â which neednât itself be aligned with human interests/values/intentions. 79 To be clear, though: I think itâs OK if people keep using "deceptive alignment," too. Indeed, I have some concern that the world has just started to learn what the term "deceptive alignment" is supposed to mean, and that now is not the time to push for different terminology. (And doing so risks a proliferation of active terms, analogous to the dynamic in this cartoon â this is one of the reason I stuck with Cotraâs "schemers.") 27 terminal training-gamers, and agents that arenât playing the training-game at all. Letâs look at each in turn. 1.2.1 Terminal training-gamers (or, âreward-on-the-episode seekersâ) As I said above, terminal training-gamers aim their optimization at the reward process for the episode because they intrinsically value performing well according to some part of that process, rather than because doing so serves some other goal. Iâl also call these âreward-on-the-episode seekers.â We discussed these models above, but Iâl add a few more quick clarifications. First, as many have noted (e.g. Turner (2022b) and Ringer (2022)), goal-directed models trained using RL do not necessarily have reward as their goal. That is, RL updates a modelâs weights to make actions that lead to higher reward more likely, but that leaves open the question of what internal objectives (if any) this creates in the model itself (and the same holds for other sorts of feedback signals). So the hypothesis that a given sort of training will produce a reward-on-the-episode seeker is a substantive one (see e.g. here for some debate), not settled by the structure of the training process itself. â˘That said, I think itâs natural to privilege the hypothesis that models trained to produce highly- rewarded actions on the episode will learn goals focused on something in the vicinity of reward-on-the-episode. In particular: these sorts of goals will in fact lead to highly-rewarded behavior, especially in the context of situational awareness. 80 And absent training-gaming, goals aimed at targets that can be easily separated from reward-on-the-episode (for example: âcuriosityâ) can be detected and penalized via what I call âmundane adversarial trainingâ below (for example, by putting the model in a situation where following its curiosity doesnât lead to highly rewarded behavior). Second: the limitation of the reward-seekingto the episodeis important. Models that care intrinsically about getting reward in a manner that extends beyond the episode (for example, âmaximize my reward over all timeâ) would not count as terminal training-gamers in my sense (and if, as a result of this goal, they start training-gaming in order to get power later, they will count as schemers on my definition). Indeed, I think people sometimes move too quickly from âthe model wants to maximize the sort of reward that the training process directly pressures it to maximizeâ to âthe model wants to maximize reward over all time.â 81 The point of my concept of the âepisodeââi.e., the temporal unit that the training process directly pressures the model to optimizeâis that these arenât the same. More on this in section 2.2.1 below. Finally: while Iâl speak of âreward-on-the-episode seekersâ as a unified class, I want to be clear that depending on which component of the reward process they care about intrinsically, different reward-on-the-episode seekers might generalize in very different ways (e.g., trying specifically to manipulate sensor readings, trying to manipulate human/reward-model evaluations, trying specifically to alter numbers stored in different databases, etc). Indeed, I think atonof messy questions remain about what to expect from various forms of reward-focused generalization (see footnote for more discussion 82 ), and I encourage lots of empirical work on the topic. For present purposes, though, Iâm going to set such questions aside. 1.2.2 Models that arenât playing the training game Now letâs look at models thatarenâtplaying the training game: that is, models that arenât aiming their optimization specifically at the reward process (whether terminally,orinstrumentally). We can distinguish between two ways this can happen: ⢠either a model is pursuing what Iâl call the âspecified goalâ (Iâl call this sort of model a âtraining saintâ), 80 Depending on where we draw various lines, it seems possible for a model to value something in the vicinity of reward-on-the-episode, and to be a âreward-on-the-episode seekerâ in that sense, while lacking other aspects of situational awareness, and without playing the training game is a very full-on way. For example, maybe it values something like âreward-on-the-episode,â but misunderstands its overall situation in other fairly deep ways (e.g., as a silly example, maybe it thinks it is a giraffe rather than an AI). But when I talk about âreward-on-the-episode seekersâ below Iâl be assuming situational-awareness and training-gaming. 81 For example, I think the discussion in Cotra (2022) isnât clear enough about this distinction. 28 â˘or its pursuing someother goal(Iâl call this a âproxy goalâ),butstillnot training-gaming (Iâl call this sort of model a âmisgeneralized non-training-gamerâ). Letâs look at each in turn. 1.2.2.1 Training saintsTraining saints are pursuing the âspecified goal.â But what do I mean by that? Itâs not a super clean concept, but roughly, I mean the âthing being rewardedâ (where this includes: rewarded in counterfactual scenarios that hold the reward process fixed). Thus, for example, if youâre training an AI to get gold coins on the episode, by rewarding it for getting gold coins on the episode, then âgetting gold coins on the episodeâ is the specified goal, and a model that learns the terminal objective âget gold coins on the episodeâ would be a training saint. (Admittedly, the line between âthe reward processâ and the âthing being rewardedâ can get blurry fast. See footnote for more on how Iâm thinking about it.) 83 82 Even granted that in some sense the model cares about some component of the reward process, we can imagine further (somewhat conceptually hazy) distinctions between, for example: â˘Actual vs. conditional rewards. That is, the model could care about the reward that willactuallybe given on the episode, vs. the reward that would be givenconditional on the episode being selected for evaluation. Thus, for example, the former might try to actively optimize for causing the episode to get included in training (assuming that not all episodes are), whereas the latter would not (see discussion between Ngo and Christiano here). Insofar as this behavior would be penalized by the training process, however (for example, because the model sometimes sacrifices reward in order to try to make the episode more likely to appear in training), we might think goals that give rise to it less likely. â˘Fixed vs. changeable reward processes.Here the distinction is between a reward process that canât be altered going forward, vs. a reward process that can (thanks to Paul Christiano for discussion). Thus, for example, we can imagine a model trained in 2023 that learns the goal âbehave such that your behavior would receive high reward according to the reward process used in 2023,â and this goal persists even into 2024 (this would be a âfixedâ reward process). This contrasts with a model trained in 2024 that learns that goal âbehave such that your behavior would receive high reward according to whatever reward process will in fact be used to evaluate itââwhere this process might change in 2024 in a manner the model would then have an incentive to predict and perhaps to manipulate (this would be a âchangeableâ reward process). Fixed reward processes have the advantage of reducing the incentive for the model (or for some other set of actors) to mess with the reward process, but my own current take is that they seem less likely than change-able reward processes, since to the extent that e.g. 2024 reward process differs from the 2023 one, a model that continues to optimize for the 2023 reward process in 2024 will get updated against. â˘Concrete vs. platonic reward processes. This distinction is even hazier, but roughly speaking: we can also distinguish between models that learn to care about the output of some physical reward process happening in the real world, vs. models that learn to care about the output of some more abstract processâfor example, what some hypothetical set of evaluators would think about its behavior. The fixed reward process just discussedâe.g., âwhat would the 2023 reward process think about this 2024 actionââis an example of a platonic reward process: e.g., the 2023 reward process wonât and cannot be run in 2024, but the model cares about what it âwould sayâ regardless. This different matters because a model that cares about a concrete reward process would have a relatively clear incentive to intervene on that process, whereas it is less clear what a model that cares about something more hypothetical would do (though we might still worry, in general, that whatever form this hypothetical-caring takes, it would still incentivize power-seeking for the usual reasons). In general, this all seems extremely messyâand we should be trying, as soon as possible, to leave the realm of this sort of hazy speculation and start gathering more empirical data about how models trained to seek reward tend to generalize. 83 Roughly, Iâm thinking of the reward process as starting with the observation/evaluation of the modelâs behavior and its consequences (e.g., the process that checks the modelâs gold coin count, assigns rewards, updates the weights accordingly, etc); whereas the specified goal is the non-reward-process thing that the reward process rewards across counterfactual scenarios where it isnât tampered with (e.g., gold-coin-getting). Thus, as another example: if you have somehow created a near-perfect RLHF process, which rewards the model to the degree that it is (in fact) helpful, harmless, and honest (H), then being H is the specified goal, and the reward process is the thing that (perfectly, in this hypothetical) assesses the modelâs helpfulness, harmlessness, and honesty, assigns rewards, updates the weights, etc. See Gao (2022) for a related breakdown. Here Iâm imagining the reward process as starting with what Gao calls the âsensors.â Sometimes, though, there wonât be âsensorsâ in any clear sense, in which case Iâm imagining the reward process starting at some other hazy point where the observation/evaluation process has pretty clearly begun. But like I said: blurry lines. 29 Like training gamers, training saints will tend to get high reward (relative to models with other goals but comparable capabilities), since their optimization is aimed directly at the thing-being-rewarded. Unlike training gamers, though, they arenât aiming their optimization at the reward process itself. In this sense, they are equivalent to widget-engineers who are just trying, directly, to engineer widgets of type Aâwhere widgets of type A arealsosuch that the performance review process will evaluate them highlyâbut who arenât optimizing for a good performance review itself. (Note: the definition of âplaying the training gameâ in Cotra (2022) does not clearly distinguish between models that aim at the specified goal vs. the reward process itself. But I think the distinction is important, and have defined training-gamers accordingly. 84 ) 1.2.2.2Misgeneralized non-training-gamersLetâs turn to misgeneralized non-training-gamers. Misgeneralized non-training-gamers learn a goalotherthan the specified goal, butstillarenât training gaming. Here an example would be a model rewarded for getting gold coins on the episode, but which learns the objective âgetgold stuff in generalon the episode,â because coins were the only gold things in the training environment, so âget gold stuff in general on the episodeâ performs just as well, in training, as âget gold coins in particular on the episode.â This is an example of whatâs sometimes called âgoal misgeneralizationâ 85 or âinner misalignmentâ 86 â that is: a model learning a goal other than the specified goal. See e.g. Shah et al (2022) and Langosco et al (2021) for examples. Here the analogy would be: an employee who isnât actually trying to design the precise sort of widget that the company wants, and who is rather pursuing some other somewhat-different widget design, but whose performance happens to be evaluated highly anyway because the relevant widget designs happen to be similar enough to each other. How do we tell training saints and misgeneralized non-training-gamers apart? Itâs not always going to be clean, 87 but the rough intuition is: training saints would get high reward in a wide variety of circumstances, provided that the reward process remains untampered with. By contrast, misgeneralized non-training-gamers get high reward much less robustly. For example, in the gold coin example, if you put these two models in an environment where itâs much easier to get gold cupcakes than gold coins, but continue to use the same reward process (e.g., rewarding gold-coin-getting), the training saint (which wants gold coins) continues to pursue gold coins and to get high reward, whereas the misgeneralized non-training-gamer (which wants gold-stuff-in-general) goes for the gold cupcakes and gets lower reward. Goal misgeneralization is sometimes closely associated with scheming (or with âdeceptive align- mentâ), but the two are importantly distinct. For example, in the âeasier to get gold cupcakes than gold coinsâ example just given, the model that seeks âgold stuff in generalâ has a misgeneralized goal, but itâs not scheming. Rather, scheming would require it to understand the training process and training objective (e.g., gold coins), and to go for the gold coins as part of a strategy of seeking power for itself or other AIs. Similarly: people sometimes point to the relationship between evolution and humans as an example of (or analogy for) goal misgeneralization (e.g., evolution selects for reproductive fitness, but human goals ended up keyed to other proxies like pleasure and status that can lead to less-than-optimally- reproductive behavior like certain types of condom use). But regardless of how you feel about this as an example of/analogy for goal misgeneralization, itâs not, yet, an example of scheming (or of âdeceptive alignmentâ). In particular: comparatively few humans are actively trying to have as many kids as possible (cf: condoms) as an explicit instrumental strategy for getting power-for-their-values 84 For example, absent this distinction, the possibility of solving âouter alignmentâ isnât even on the conceptual table, because ârewardâ is always being implicitly treated like itâs the specified goal. But also: I do just think thereâs an important difference between models that learn to get gold coins (because this is rewarded), and models that learn to care about the reward process itself. For example, the latter will âreward hack,â but the former wonât. 85 See Shah et al (2022) and Langosco et al (2021). 86 See Hubinger et al (2019). 87 In particular: I doubt that an effort to identify a single, privileged âspecified goalâ will withstand much scrutiny. In particular: I think it will depend on how you carve out the âreward processâ that youâre holding fixed across counterfactuals. And screening off goals that lead to instrumental training-gaming is an additional challenge. 30 later (some ideological groups do something like this, and we can imagine more of it happening in the future, but I think it plays a relatively small role in the story of evolutionary selection thus far). 88 1.2.3 Contra âinternalâ vs. âcorrigibleâ alignment I also want to briefly note a distinction between model classes that Iâmnotgoing to spend much time on, but which other work on scheming/goal-guarding/âdeceptive alignmentâânotably, work by Evan Hubingerâfeatures prominently: namely, the distinction between âinternally aligned modelsâ vs. âcorrigibly aligned models.â 89 As I understand it, the point here is to distinguish between AIs who value the specified goal via some kind of âdirect representationâ (these are âinternally alignedâ), vs. AIs who value the specified goal via some kind of âpointerâ to that target that routes itself via the AIâs world model (these are âcorrigibly alignedâ). 90 However: I donât find the distinction between a âdirect representationâ and a âpointerâ very clear, and I donât think it makes an obvious difference to the arguments for/against scheming that Iâl consider below. 91 So, Iâm going to skip it. 92 88 We can imagine hypothetical scenarios that could resemble deceptive alignment more directly. For example, suppose that earth were being temporarily watched by intelligent aliens who wanted us to intrinsically value having maximum kids, and who would destroy the earth if they discovered that we care about something else (letâs say that the destruction would take place 300 years after the discovery, such that caring about this requires at least some long-term values). And suppose that these aliens track birth rates as their sole method of understanding how much we value having kids (thanks to Daniel Kokotajlo for suggesting an example very similar to this). Would human society coordinate to keep birth rates adequately high? Depending on the details, maybe (though: I think there would be substantial issues in doing this, especially if the destruction of earth would take place suitably far in the future, and if the AIs were demanding birth rates of the sort created by everyoneoptimizingsolelyfor having maximum kids). And to be a full analogy for deceptive alignment, it would also need to be the case that humans ended up with values motivating this behavior despite having been âevolved from scratchâ by aliens trying to get us to value having-maximum-kids. 89 In my opinion, Hubingerâs use of the term âcorrigibilityâ here fits poorly with its use in other contexts (see e.g. Arbital here). So I advise readers not to anchor on it. 90 Hubingerâs example here is: Jesus Christ is aligned with God because Jesus Christ just directly values what God values; whereas Martin Luther is aligned with God because Martin Luther values âwhatever the Bible says to do.â This example suggests a distinction like âvaluing gold coinsâ vs. âvaluing whatever the training process is rewarding,â but this isnât clearly a contrast between a âdirect representationâ vs. a âpointer to something in the world model.â For example, gold coins can be part of your world model, so you can presumably âpointâ at them as well. 91 Are human goals, for example, made of âdirect representationsâ or âpointers to the world modelâ? Iâm not sure itâs a real distinction. Iâm tempted to say that my goals are structured/directed by my âconcepts,â and in that sense, by my world model (for example: when I value âpleasure,â I also value âthat thing in my world model called pleasure, whatever it is.â But Iâm not sure what the alternative is supposed to be.) And Iâm not sure how to apply this distinction to a model trained to get gold coins. Whatâs the difference between valuing gold coins via a direct representation vs. via a pointer? In a comment on a previous draft of this report, Hubinger writes (shared with permission): âMaybe something that will be helpful here: I basically only think about the corrigible vs. the deceptive caseâthat is, I think that goals will be closer to concepts, which I would describe as pointers to things in the world model, than direct representations essentially always by necessity. The internally aligned case is mostly only included in my presentations of this stuff for pedagogical reasons, since I think a lot of people have it as their default model of how things will go, and I want to very clearly argue that itâs not a very realistic model.â But this doesnât clarify, for me, what a direct representationis. 92 Another issue with Hubingerâs ontology, from my perspective, is that he generally focuses only on contrasting internally aligned models, corrigibly aligned models, and deceptively aligned modelsâand this leaves no obvious room for reward-on-the-episode seekers. That is, if we imagine a training process that rewards getting gold coins, on Hubingerâs ontology the goal options seem to be: direct-representation-of-gold-coins, pointer-to-gold-coins, or some beyond- episode goal that motivates instrumental training-gaming. Reward-on-the-episode isnât on the list. On a previous draft of this report, Hubinger commented (shared with permission): If you replace "gold coins" with "human approval", which is the case I care the most about, what Iâm really trying to compare is "pointer-to-human-approval"/"concept of human approval" vs. "deception". And I guess I would say that "pointer-to-human-approval" is the most plausible sycophant/reward-maximizer model that you might get. So what Iâm really comparing is the sycophant vs. the schemer, which means I think I am doing what you want 31 1.2.4 The overall taxonomy Overall, then, we have the following taxonomy of models: Figure 5: The overall taxonomy of model classes Iâl focus on. Note that in reality, a modelâs goal-system can (and plausibly will) mix these different motivations together (e.g., it might be partly pursuing the specified goal, partly reward-on-the-episode, partly something else entirely, etc). For simplicity, Iâl often think in terms of a âpureâ version of one of these models (e.g., a model thatonlycares about reward-on-the-episode), and Iâm hoping that a greater focus on âmixed modelsâ wouldnât alter my analysis in deep waysâand in particular, that the analysis applicable to a âpure modelâ will generally apply in roughly the same way to the corresponding part of a more mixed goal system as well (e.g., a model that cares somewhat about reward-on-the-episode, and somewhat about other things). Iâl say a bit more about mixed models in section 1.3.5 below. 1.3 Why focus on schemers in particular? As I noted above, I think schemers are the scariest model class in this taxonomy. 93 Why think that? After all, canâtallof these models be dangerously misaligned and power-seeking? Reward-on-the- episode seekers, for example, will plausibly try to seize control of the reward process, if it will lead to more reward-on-the-episode. Training saints can end up misaligned if you misspecify the goal; and even if you specify the goal correctly, and somehow avoid training-gaming, you might end up me to be doing here. Though, note that Iâm not really comparing at all to the saint, which is because doing that would require me to explicitly talk about the simplicity of the intended goal vs. the specified goal, which in most of these presentations isnât really something that I want to do. Even granted that weâre mostly interested in cases where human approval is part of the training process, Iâm wary of assuming that it should be understood as the specified goal. Rather, Iâm tempted to say that the thing-the-humans-are-approving-of(e.g., helpfulness, harmlessness, honesty, etc) is a more natural candidate, in the same sense that if the reward process rewards gold-coin-getting, gold coins are (on my ontology) the specified goal target. 93 Here Iâm just thinking of goal-guarding schemers as a type of schemer, rather than any more or less scary than schemers-in-general. 32 with a misgeneralized non-training-gamer instead. 94 So doesnât some sort of misalignment await at basically every turn? Why focus on schemers in particular? This section explains why. However, if youâre happy enough with the focus on schemers, feel free to skip ahead to section 1.4. 1.3.1 The type of misalignment Iâm most worried about To explain why I think that schemers are uniquely scary, I want to first say a few words about the type of misalignment Iâm most worried about. First: Iâm focused, here, on what Iâve elsewhere called âpractical power-seeking-alignmentââthat is, on whether our AIs will engage in problematic forms of power-seekingon any of the inputs they will in fact receive. This means, importantly, that we donât need to instill goals in our AIs that lead to good resultseven when subject to arbitrary amounts of optimization power(e.g., we donât need to pass Yudkowskyâs âomni testâ). Rather, we only need to instill goals in our AIs that lead to good results given the actual options and constraintsthose AIs will face, and theactual levels of optimization powerthey will be mobilizing. This is an importantly lower bar. Indeed, itâs a bar that, in principle,allof these models (even schemers) can meet, assuming we control their capabilities, options, and incentives in the right way. For example, while itâs true that a reward-on-the-episode seeker will try to seize control of the reward process given the opportunity, one tool in our toolset is: to not give it the opportunity. And while a paradigm schemer might be lying in wait, hoping one day to escape and seize power (but performing well in the meantime), one tool in our tool box is: to not let it escape (while continuing to benefit from its good performance). Of course, success in this respect requires that our monitoring, control, and security efforts be sufficiently powerful relative to the AIs weâre worried about, and that they remain so even as frontier AI capabilities scale up. But this brings me to my second point: namely, Iâm here especially interested in the practical PS-alignment of some comparatively early set of roughly human-levelâor at least, not-wildly-superhumanâmodels. That is: rather than focusing now on the alignment of arbitrarily capable AIs, I think itâs reasonable to focus on the goal of successfully using some relatively early generation of not-wildly-superhuman AIs to perform a very large amount of high quality cognitive work for usâincluding: cognitive work that can shed light on traditional alignment-related goals like interpretability, scalable oversight, monitoring, control, coordination amongst humans, the general science of deep learning, alternative (and more controllable/interpretable) AI paradigms, and the like. 95 Defending this point of focus is beyond my purpose here. But itâs important to the lens Iâl be using in what follows. In particular: I think itâs plausible that there will be some key (and perhaps: scarily brief) stage of AI development in which our AIs arenotyet powerful enough to take-over (or to escape from human control), but where they are still capable, in principle, of performing extremely valuable and alignment-relevant cognitive work for us, if we can successfully induce them to do so. And Iâm especially interested in forms of misalignment that might undermine this possibility. Finally: Iâm especially interested in forms of PS-misalignment in which the relevant power-seeking AIs are specifically aiming either to cause, participate in, or benefit from some kind of full-blown disempowerment of humanityâwhat Iâl call an âAI takeover.â Granted, not all problematic forms of misaligned power-seeking are aimed at takeover in this wayâan AI might, for example, just want to escape onto the internet in order to have more freedom and resources in general, even if it never expects human disempowerment. And in principle, you could end up with human disempowerment even with no individual AI actively optimizing for it. But from the perspective of existential risk, Iâm most worried about AIs that are activelytryingto overthrow humanity, or to benefit from such an overthrow. 94 Though note, per Turner (2022a), that in principle you can get an aligned system even with a âmis-specifiedâ goal, provided that the model learns a âmisgeneralizedâ goal that is in fact desirable by your lights. 95 This approach is closely related toâthough somewhat broader thanâ OpenAIâs goal of building an automated alignment researcher. See also in Karnofsky (2022c) a discussion of âMAGMAâs goalsâ here. 33 1.3.2 Contrast with reward-on-the-episode seekers With this frame on the misalignment concern in mind, letâs turn to the question of how scary schemers are relative to the other model classes I discussed. Iâl start by contrasting schemers with reward-on-the-episode seekers. Here I see a few reasons to think that schemers are scarier. 1.3.2.1 Responsiveness to honest testsFirst, schemers will do more to activelyhidetheir mis- alignment from humans. In particular: as Karnofsky (2023a) notes, in principle you can get a reward-on-the-episode seeker torevealthat they are disposed to engage in bad behavior in pursuit of reward-on-the-episode just by putting them in an episode where bad behavior will be rewarded, and they know this. â˘One concrete example here might be: giving a situationally-aware AI the ability to manipu- late all the components of the process for assigning reward to the episode, and seeing what it does (does it, for example, start trying to bypass the usual process and max out the reward in question?). â˘Another might be: putting a situationally-aware AI in an episode where being dishonest, unhelpful, and/or harmful will be rewarded (and it has credible evidence of this fact). 96 Note that the point, here, isnât to try tofoolthe model into revealing misalignment it doesnât want to reveal (call this a âtrapâ). It doesnât require deceiving the model about, for example, whether now is a good time to try to escape. (The problem with traps, after all, is that sufficiently advanced AIs might be able to detect them. 97 ) Rather, the aim here is to offer an âhonest test,â which attempts to set up the modelâs actual options such that revealing its true goals is the best overall strategy. (Though: getting the model to trust our presentation of its options might be an additional challenge, especially if we have been elsewhere trying to deceive it.) Now, in principle, you can try to use an honest test like this to detect scheming, too. For example: you might try to give a model that has thus far seemed aligned access to some flexible, well-resourced, but still-controlled environment (e.g., some secure compute cluster it can use as it sees fit), in the hopes that what it does with the resources and flexibility will reveal what it really wants. However, this works on schemers much less reliably than it does with reward-on-the-episode seekers. In particular: â˘With reward-on-the-episode seekers, you know that you can just offer them reward-on-the- episode, and theyâl go for it. Whereas youâre not sure what a schemer actually wants, and so not sure what to offer it to get it to reveal its true goals (hence the need to offer flexible resources instead). Maybe, for example, your isolated chunk of computational resources isnât of much interest to the schemer in question, because it values something that canât be created/simulated using those resources. â˘Whatâs more, even if the reveal-your-goals option holdssomeappeal to the schemer, it might still decide that continuing to fake alignment is a better strategy overall, because its prospects for getting much more power later outweigh the near-term benefits of the reveal-your-goals option (e.g., maybe it can do some of its favored thing on your compute, but much more if it escapes from human control). Whereas a reward-on-the-episode seeker only wants reward-on-the-episode, and so doesnât have to do this sort of cost-benefit analysis. 96 Note, though, that we need to be cautious in responding to dispositions-towards-bad-behavior that get revealed via tests like these. In particular, if you just try to naively âtrain outâ the relevant behavior, you might: 1.Not actually be reproducing the component of the reward process that the model cares about (for example, if the model cares about its behavior being reinforced-by-the-gradient-updates, then if the test will gradient- descentagainstthe bad behavior itâs seeking to prompt, then the model wonât actually have an incentive to engage in that behavior), and/or 2. Train the model to be more cautious and strategic about revealing its true goals to humansâfor example, a reward-seeker that only tries to directly manipulate the reward process when it thinks humans wonât later discover this. 97 Indeed, in principle, an AI could decide to always act aligned until it sees some input it knows would be very difficult/impossible for humans at the present time to produce. Christianoâs (2019b) example here is a factorization of RSA-2048, a very large semiprime that humans currently seem very far away from factoring. 34 This dynamic connects to a broader benefit of reward-on-the-episode seekers relative to schemers: namely, that if you are able toverifyâfor example, via one of these âhonest testsââthat a given model is a reward-on-the-episode seeker, then there is a useful sense in which you âknow what you are working with.â That is: yes, your model is misaligned, and would seize control of the reward process if given the opportunity. But: you alsoknow what it wants, and what incentives it will respond to. This sort of knowledge seems very helpful for predicting its behavior; and (if you can retain control over the model), for harnessing its cognitive capacities in useful ways (and in particular, on what Iâve elsewhere called âcheckable tasksââmore below). 1.3.2.2 Temporal scope and general âambitionâAnother benefit of reward-on-the-episode seekers relative to schemers is that reward-on-the-episode seekers limit the scope of their optimization (and hence, of their power-seeking) to the episode, whereas schemers do not. This means that reward-on-the-episode seekers wonât have incentives to engage in or support efforts to disempower humans that donât pay off within the temporal scope of the episode. 98 Thus, for example, suppose that it will be at least six months until any sort of AI takeover or seizing- control-of-the-reward-process event will be feasible. In this case, a reward-on-the-episode seeker whose episode is less than six months long wonât have any incentive to try to engage in or support such a takeover, or to try to seize control of the reward process itself. And if it had evidence that other AIs were planning such a takeover, and it could get more reward-on-the-episode from revealing this evidence to humans, it would be incentivized to do so. This is notably in contrast with schemers, who are often imagined to be scheming in pursuit of extremely long-horizon goals. That is, a paradigm schemer has its eye on human disempowerment from the get-go, even if such disempowerment wonât be possible for a long timeâand it will plan and act accordingly. Differences in temporal scope connect to a broader benefit of reward-seekers relative to schemersâ namely, that schemers can end up quite a bit more intuitively âambitious,â in the sense that indicates greater and more destructive appetite for power and resources. Now, exactly what âambitionâ amounts to here isnât clear (see also usage of the term âbroad scopeâ), and in the contexts of sufficiently intense optimization with certain kinds of structure, intuitively âmodestâ goals (e.g., âsolve this math problem correctlyâ), can lead to surprisingly power-grabby behavior (e.g., using tons of computational resources checking and re-checking the problem in question). Still, though, there is an intuitive difference between the sort of power-seeking one expects from a model tasked with âmaximize the total paperclips across all time and spaceâ vs. a model tasked with âcreate between three and five paperclips before 5 pm EST today,â especially if the model in question is not arbitrarily powerful. Notably, for example, on the latter task, it seems quite a bit easier to get to >99% of the available utility without engaging in destructive sorts of power-seeking. And because of the limitation in the temporal horizon of their optimization, it seems to me that reward-on-the-episode seekers will generally be less intuitively ambitious than schemers in this sense. 99 That said, as Iâl discuss below, the longer the episode for a given reward-on-the-episode seeker, the less this consideration applies (and insofar as we want our AIs to be performing long-horizon tasks, we might well be training them on very long episodes). Indeed, there are some ways of generalizing the notion of âreward-on-the-episodeâ that will lead to arbitrarily long time horizons of optimization, even if you mightâve thought the episode was temporally limited. 100 For example, maybe the model cares about the reward associated with some unit of behavior in a way that makes the model want to make sure that some number stored in some database is nevereditedâin which case, it would have an incentive to gain and maintain power over arbitrarily long time horizons (this is one reason to be careful about training models in a manner that involves going back and editing the rewards associated with a given episode later, as you could inadvertently âextend the episodeâ by incentivizing the model to optimize for making sure that the reward on the episode is never edited). 98 Here Iâm setting aside some speculative dynamics concerning âanthropic capture,â discussed in the footnotes at the beginning of section 2. 99 More discussion of âambitionâ in section 2.3.1.2.7 below. 100 I donât think that models that generalize this way should be interpreted as "reward-on-the-episode seekers," but theyâre nearby. 35 So in general, the more one is seeking comfort from âthis reward-on-the-episode seeker is only optimizing overxsort of limited time horizon,â the more one needs to ensure that the limitation in question really holds. 1.3.2.3 Sandbagging and âearly underminingâA third (and related) benefit of reward-on- the-episode seekers relative to schemers is that schemers seem to me more likely to engage in âsandbaggingââand in particular, what Iâl call âearly undermining.â Sandbagging occurs when a model intentionally performs less well on a task than it is capable of. Here Iâm especially interested in tasks whose performance that we can âcheck,â but which we canât perform ourselvesâfor example, a proof we couldnât come up with, but whose validity we can verify; or a security vulnerability we wouldnât have noticed, but which we can see is a vulnerability once itâs pointed out to us. As Iâve discussed elsewhere, I think that if models donât sandbag on checkable tasks, this opens up a very powerful set of capabilities that we can harness for the purposes of alignment, interpretability, security, and much else. The big concern about relying on these capabilities, though, is that in principle models seem capable of whatâs sometimes called âexploration hacking.â That is, even if we would be able to check (and then to reward) task-performing behavior, if we donât know how to do the task ourselves, a model could in principle just refuse to show us the task-performing behavior in question (whereas if we knew how to perform the task, we would know what the model shouldâve done, and would be able to update its policy accordingly). Plausibly, though, reward-on-the-episode seekers wouldnât do this (provided that we limit the episode to the checkable task in question). Suppose, for example, that we want to know if itâs possible to hack a given computer and make a blue banana appear on the screen. So we ask our model to do this. Because the task is checkable (e.g., we can tell whether the banana appeared), we can give maximum reward to successful performance of the task. So a reward-on-the-episode seeker has no incentive to sandbag hereâitâl just get less reward on the episode. And this broad dynamic generalizes to other sorts of capability evaluations. That is, just as reward-on-the-episode seekers will reveal theirgoals where doing so leads to reward-on-the-episode, so too will they reveal their capabilities. Schemers, by contrast, are a different story. A schemer that is otherwise (instrumentally) optimizing for reward-on-the-episode might nevertheless choose specific occasions on which to get less reward than it could in principleâfor example, by sandbagging on a checkable, security-critical task, or by hiding the full extent of its capabilities from an evaluation processâin order to further its aims in other ways. Here Iâm especially concerned with what Iâl call âearly underminingââthat is, active efforts by a misaligned AI to strategically undermine human alignment, control, and coordination efforts, occurring at a stage of AI development where AIs are not yet powerful enough to escape from human control or to takeover more generally. Examples here might include: intentionally inserting backdoors and other security vulnerabilities into an AI labâs code, for use in later attempts at escape/self- exfiltration; intentionally generating alignment research that will fail in ways advantageous to a takeover; intentional manipulation of human discourse and coordination in a manner that will later serve an AI takeover, and so forth. Early undermining is an especially scary sort of oversight failure, because it is actively optimized for promoting AI takeover, rather than merely for âlooking as good as possibleâ according to the oversight mechanisms in question. Thus, for example: one concern about trying to use AIs to help us with alignment research is that humans will be insufficiently able to discriminate between actually-good alignment research, and only-apparently-good alignment researchâthat is, that the task of generating good alignment research isnât sufficiently âcheckable.â Itâs not clear to me that this is right (notably, for example, lots of key alignment research in an era of human-level-ish models might well involve fairly straightforward empirical experiments, the results and implications of which seem like they could well be quite âcheckableâ), but even if it is: there is an important difference between alignment research optimized forseeming as good as possible according to the evaluation process(even where this compromises its quality), and alignment researchstrategically optimized for promoting AI takeover. The former is more like: software optimized to maximally impress a naive and non-technical supervisor, even if it has other bugs. The latter is more like: software optimized specifically to embezzle money from the company without the supervisor noticing. Weak supervisors can fail to catch bothâbut it matters which you get. And plausibly (at least assuming the future 36 takeover is outside the scope of the episode), reward-on-the-episode seekers would mostly have incentives to generate the former; whereas schemers might well do both. That said, and importantly, schemers who engage in sandbagging and early undermining (or who continue to act aligned in tests where we try to reward unaligned behavior) will generally have to take some hit to their reward-on-the-episode (sandbagging on checkable tasks basically requires this by definition, and even on non-checkable tasks, optimizing solely for reward-on-the-episode seems likely to get more reward than optimizing forbothrewardandsupporting a takeover). 101 So in this sense, schemers of this kind wonât befullyâplaying the training gameâârather, theyâl bemostly playing the training game, and then choosing opportune moments to sacrifice reward for the sake of some other advantage. (Iâl still call these models âschemers.â) Itâs an open question, though, how much schemers would depart from max-reward behavior in this way. Part of this is about what Iâl call âslack in trainingâ below (more discussion there). But thereâs also a broader question, about how risk-averse schemers will be in general with respect to the possibility of getting caught by humans. In particular: if humansdocatch a schemer engaging in some form of sandbagging or early underminingâfor example, hiding capabilities on an eval, or actively inserting vulnerabilities into codeâthen this might be a significant advantage to humans in preventing a takeover, since they will then both be able to point to a naturally-arising example of scheming as a warning to others (though: how seriously others will take the warning is a further question), and to study the dynamics that gave rise to this sort of scheming in detail (e.g., extensively testing and probing the model in question, studying/replicating/tweaking circumstances that gave rise to the bad behavior, etc). For this and other reasons, schemers might, in general, be quite wary of attempting any sort of misaligned behavior until they are extremely confident they wonât get caughtâconfidence that might be hard to achieve (at least for roughly human-level-ish models) in a context where humans are otherwise attempting to trick models into revealing misalignment. So maybe, even if schemerscouldget an advantage from sandbagging or early undermining while still broadly under human control, they wonât try it. And if they donât try it, this makes them more similar to reward-on-the-episode seekers in their safety profile. 1.3.3 Contrast with models that arenât playing the training game Those are some reasons, then, that schemers (especially schemers that engage in sandbagging and early undermining) generally seem to me quite a bit scarier than reward-on-the-episode seekers. Letâs turn, now, to comparing schemers with models that arenât playing the training game at all. We can distinguish between three ways that this can happen: 1. The model lacks the situational awareness required to play the training game. 2.The model has situational awareness, but its goals do not extend beyond the time horizon of the episode. 3.The model has situational awareness, and its goal extends beyond the episode, but itâs still not training-gaming for some reason (for example, because training-gaming isnât actually a very good instrumental strategy for achieving these goals). The first case seems comparatively non-worrying. Models that lack situational awareness have very limited knowledge of the world, and so are less capable of effective strategic action to gain power or to deceive humans about their motives. Indeed, for this reason, in what follows, Iâl generally assume that the models weâre talking about are sufficiently advanced and sophisticated that they end up situationally aware at some point during the training process. The second case implicates the same advantages re: temporal scope that reward-on-the-episode seekers do (though: as with reward-on-the-episode seekers, if the episode in question is sufficiently long, the comfort at stake lessens). But it also has an additional advantage: namely, that models of this kind will generally happily reveal their misaligned goals on the episode,even if doing so will result in negative reward. So the process of testing for misaligned goals is especially straightforward and mundane. Thus, for example, suppose youâre wondering whether, in the process of training your model to help humans with scientific research, it learned a mis-generalized objective like âfollow 101 And the same applies to schemers who act aligned on tests for reward-seeking that attempt to provide reward for misaligned behavior. 37 my curiosity on the episode, even when doing so conflicts with being harmless and honest.â 102 Here, if you put the model in a circumstance where it has a chance to pursue its curiosity-on-the-episode in a harmful and dishonest way, it will go for it,even if it knows that it will get a negative reward (and thus, that successor versions of itself may have somewhat different goals). After all, itâs only concerned about the episode itself. Of course, thereâs still the further question of what sort of model you getafteryou update this one in response to its bad behavior. Maybe, for example, you end up with a schemer, or a reward-on-the-episode seeker. But the original model was happy to show its âtrue colors.â What about the third caseâi.e., models with both situational awarenessandgoals that extend beyond the episode, but which are still not training-gaming? Itâs often assumed that this wonât happen, because scheming is such an obvious instrumental strategy for situationally aware models with beyond-episode goals. Thus, for example, suppose that your model has learned the objective âpursue my curiosityover all time, even where doing so is harmful and dishonest.â In that case, itâs often assumed that the model will react to the sort of test just mentioned (e.g., an opportunity to pursue curiosity in harmful/dishonest ways) like a schemer: e.g., by reasoning that âif I go for my curiosity here, the humans will know Iâm misaligned, and either shut me down or modify my goals going forward, thereby harming my goal of pursuing-my-curiosity over all time. So, Iâl act aligned in this case.â This is the sort of thing that makes schemers scary (see also Karnofsky (2022a) on the âKing Lear problemâ). As Iâl discuss below, Iâm not actually sure that acting like a schemer is such an obvious instrumental strategy hereâit depends on a variety of further factors. 103 For present purposes, though, what matters is that if the modeldoesnâtact like a schemer here, then it will plausibly show its true colors in a fashion similar to the episode-limited curiosity pursuer. That is, absent an active attempt to avoid revealing its misalignment, here, it will plausibly just follow its curiosity. 104 So weâl avoid King Lear problems yet again. (That said, Iâl note one quasi-terminological complication here: namely, depending on how strictly we define âscheming,â itâs possible to fake alignment in cases like this, for schemer-like reasons, without being a full-blown schemer. In particular: a model might pretend to be more aligned than it is at various points in training and evaluation, as part of an instrumental strategy of seeking post-episode power for itself or other AIs, without directly optimizing for reward-on-the-episode in other contexts. Maybe, for example, it generally just pursues its curiosity during training, and this tends to get good-enough reward, but it still makes sure to pretend to be aligned whenever e.g. humans ask it about its alignment-relevant properties, or put it in situations specifically designed to elicit misaligned behavior. One question, here, is whether models like this can survive training unmodified (see the discussion of âslackâ in section 1.5 below). Assuming they can, though, Iâm mostly going to treat them as equivalent to schemers for present purposes, since theyâre acting like schemers in the ways that matter to me most.) Overall, then, it seems like non-training-gaming models will plausibly be reasonably responsive to tests for mundane misalignment that we include in training, since they generally wonât be actively optimizing for manipulating those tests. That said, and importantly, humans still need to figure out what tests to run, and also: to be willing to take the time/effort/etc to run them. And it may be quite difficult/expensive to test the model in a sufficiently wide array of circumstances to robustly reveal relevant propensities for misaligned behavior (though recall that we donât need to make sure that a model would never act badly under any circumstanceâwe just need to prevent the model from acting badlygiven the options and constraints it will in fact face). Indeed, there are threat models on which models donât engage in training-gaming, schemer-like behavior, or planning-towards-a-takeover during training at all (despite human efforts to reveal misalignment), but whose alignment is sufficiently imperfect that they nevertheless end up pursuing takeover at some later stageâperhaps because they go sufficiently far out of the training distribution, 102 This is the example from Cotra (2021b). 103 For example, note that insofar as one thinks of human evolution as an example of/analogy for âgoal misgeneralization,â humans with long-term goalsâand who are âsituationally aware,â in the sense that they understand how evolutionary selection worksâdonât tend to focus on instrumental strategies that involve maximizing their inclusive genetic fitness. More on why not in the discussion of âslackâ in section 1.5. 104 Here the evolutionary analogy would be: humans with long-term goals who are nevertheless happy to use condoms. 38 or because they gain new options that werenât available before, or because they start reflecting on their goals and general ontology in ways that break the sort of alignment they displayed in training (though note, importantly, that we need not assume that any particular model later scales up in capability post-trainingâi.e., that it âgrows up,â intelligence-explodes, etc). 105 However, Iâm less worried about these scenarios than I am about scenarios that involve models actively scheming/planning-towards- a-takeover during training (and recall that âtrainingâ might well extend to quite deployment-like phasesâe.g. interacting with users behind an APIâthereby expanding the scope of the training distribution). In particular: these models arenât trying tomanipulatethe alignment testing that occurs during training. And since they arenât actively planning-towards-a-takeover, they seem unlikely to engage in schemer-like behaviors like sandbagging and early undermining while still under human control. 1.3.4 Non-schemers with schemer-like traits Overall, then, it seems to me that schemers are the most worrying model class in the taxonomy Iâve given. However, I do want to emphasize that various of the non-schemer models Iâve discussed can act in schemer-like ways. In particular: the centrally scary thing about (paradigmatic) schemers is that they are actively and robustly optimizing against human efforts to reveal their misalignment, as part of a long-term plan to try to disempower humans (that is, they are âstrategic betrayersâ in the sense I discussed above). But other models can act in similar ways. For example, models whose optimization is limited to the episode can still pretend to be more benign than they are in order to gain/maintain power to pursue their goalson the episode(for example, if humans will shut them down/modify them mid-episode if their misalignment is revealed). And if the episodes are long, this can closely resemble the sort of long-horizon strategic planning paradigmatic of schemers. And similarly, models that mostly arenât âplaying the training gameâ can still fake their alignment when doing so is required to gain/maintain long-term power. Whatâs more, this reportâs discussion (and the corresponding taxonomy) is focused centrally on models whose goals are mainly shaped via direct training on a feedback signal like âreward.â But the AIs that ultimately matter most might be very different,but still worryingly schemer-like. Perhaps, for example, they more closely resemble a âlanguage model agentâ whose goals are shaped centrally via user inputs (though: note that if a language model agent is trained end to end, rather than simply being built out of individually-trained components, then the broad framework of the report will still apply); or maybe theyâl come from a much less deep-learning-centric paradigm altogether. 106 Still, if these AI agents end up with problematic goals, but they realize that humans learning of their misalignment will compromise their pursuit of these goals, they may engage in alignment faking and strategic betrayal in a manner quite reminiscent of schemers. Clearly, then, avoiding schemers leaves plenty to worry about. Still, especially in contexts where we are centrally relying on techniques like RLHF to shape the goals of our models, understanding the probability that we create schemers in particular seems to me especially important. And if I learned that such techniquesdonâtcreate schemers, then I, at least, would feel substantially better about AI risk overall. 1.3.5 Mixed models So far, Iâve been focused on comparing âpureâ versions of these different model classes. But what about âmixedâ models, whose motivations combine these different sorts of goals together? Note that for mixed models to be competitive with pure models in a training environment that exerts a lot of pressure on models to get high reward (see the discussion of âslackâ in section 1.5 below),all the relevant motivations will generally need to point the modelâs optimization (whether terminally or instrumentally) at targets closely correlated with reward on the training data. For example, if your model is being trained to get gold coins, and it forms a motivation set that mixes together âget reward-on-the-episodeâ and âget gold stuff,â then if its âget gold stuffâ motivation causes it to get less reward on an episode where there are gold cupcakes available, then it will get punished by SGD 105 This is one way of reading the threat model in Soares (2022), though this threat model could also include some scope for scheming as well. See also this Arbital post on âcontext disasters,â of which âtreacherous turnsâ are only one example. 106 Also, note that certain concerns about âgoal misgeneralizationâ donât apply in the same way to the language model agents, since information about the goal is so readily accessible 39 (and the âget gold stuffâ motivation will plausibly weaken over time). And to the extent SGD favors one or another of these sorts of motivations even when they all point the model at something closely correlated with reward (Iâl discuss some considerations in this respect below, related, for example, to the simplicity of the goal-targets in question, and the different amounts of instrumental reasoning they require), we might expect this favoritism to push, over time, to purer models. Still, mixed models seem like a fairly plausible outcome to me, especially in the context of what Iâl call âmessy goal-directednessâ below. And in principle, they introduce more formidable combinatorial problems into the analysisâi.e., these different sorts of goals can combine in a large number of different ways. But in thinking about what combinations might matter, here, I think clarity aboutwhy exactly schemers are scary can be useful. In particular: to my mind, the main question to ask about a model that ispartlya schemer, andpartlysomething else (i.e., a model that expendssomeeffort optimizing for reward-on-the-episode as part of a power-motivated instrumental strategy) is whether the relevant set of mixed motivations yields the sorts of behaviors that make schemers especially worrying. Thus, for example, consider a mixed model whose motivations mix together scheming and reward- on-the-episode seekingâi.e., who cares about reward-on-the-episodepartlyfor terminal reasons, and partly for instrumental reasons. (Here, the human analogy might be an employee who wants good performance reviewsbothbecause they intrinsically like getting good performance reviewsand because they want to make money and not get fired.) Still, we can ask: 1.Does this model fake alignment on âhonest testsâ designed to get it to reveal its true colors? 2.Does the temporal horizon and ambition of the modelâs concern incentivize it to sup- port/participate in a takeover that occurs after (and perhaps: long after) the episode is complete? 3. Will this model engage in sandbagging and early-undermining? Of course, the answers to these questions will depend on the specific mix of goals at stake, along with other factors. But other things equal, the schemer-like component of its motivations will push in the direction of âyes,â and the reward-on-the-episode-seeker-like component of its motivations will push in the direction of âno.â And we can say something similar about models that mix together scheming with non-training-gaming-motivations, or that mix together all three of scheming, reward-on-the- episode-seeking, and other things. This isnât to say that no complexities or interesting interactions will arise from the full set of possible combinations, here. But as a first pass, the question that I, at least, care about is not âis this model apureschemerâ or âdoes this model haveany traceof schemer-like motivations,â but rather: âis this modelenoughof a schemer to be scary in the way that schemers in particular are scary?â 1.4 Are theoretical arguments about this topic even useful? Even if you agree that the likelihood of schemers is important, you still might be skeptical about the sorts of theoretical arguments discussed in this report. Ultimately, this is an empirical question, and pretty clearly, we should be trying to get whatever empirical evidence about this topic that we can, as soon as we can. In section 6, I discuss some examples of empirical research directions Iâm excited about. That said, I do think itâs useful to have as clear an understanding as we can of the landscape of theoretical arguments available. In particular: I think itâs possible that we wonât be able to get as much empirical evidence about scheming as we want, especially in the near term. Barriers here include: â˘At least for full-blown scheming, you need models with situational awareness and goal directedness as prerequisites. ⢠âModel organismsâ might be able to artificially induce components of scheming (for example: in models that have been intentionally given long-term, misaligned goals), but the question of how often such components arise naturally may remain open. â˘The best evidence for naturally-arising scheming would be tocatchan actual, naturally- arising schemer in the wild. But a key reason schemers are scary is that they are intentionally undermining efforts to catch them. 40 So especially in the near term, theoretical arguments might remain one of the central justifications for concern about schemers (and theyâve been the key justification thus far). And we will need to make decisions in the meantime (for example, about near-term prioritization, and about what sorts of trade-offs to make out of caution re: schemer-focused threat models) that will hinge in part on how concerned we areâand thus, on how forceful we take those arguments to be. For example: suppose that by the time we are training roughly-human-level models, we have never yet observed and verified any naturally-arising examples of scheming. Should we nevertheless refrain from deploying our models in X way, or trusting evaluations of form Y, on the basis of concern about scheming? Our assessment of the theoretical arguments might start to matter to our answer. Whatâs more, understanding the theoretical reasons we might (or might not) expect scheming can help us identify possible ways to study the issue and to try to prevent it from arising. That said, I empathize with those who donât update their views much either way on the basis of the sorts of considerations Iâl discuss below. In particular, I feel keenly aware of the various ways in which the concepts Iâm employing are imprecise, possibly misleading, and inadequate to capture the messiness of the domain theyâre attempting to describe. Still, Iâve found value in trying to be clear about how these concepts interact, and what they do and donât imply. Hopefully readers of this report will, too. 1.5 On âslackâ in training Before diving into an assessment of the arguments for expecting scheming, I also want to flag a factor that will come up repeatedly in what follows: namely, the degree of âslackâ that we should expect training to allow. By this I mean something like: how much is the training process ruthlessly and relentlessly pressuring the model to perform in a manner that yields maximum reward, vs. shaping the model in a more ârelaxedâ way, that leaves more room for less-than-maximally rewarded behavior. That is, in a low-slack regime, âbut that sort of model would be getting less reward than would be possible given its capabilitiesâ is a strong counterargument against training creating a model of the relevant kind, whereas in a high-slack regime, itâs not (so high slack regimes will generally involve greater uncertainty about the type of model you end up with, since models that get less-than-maximal reward are still in the running). Or, in more human terms: a low-slack regime is more like a hyper-intense financial firm that immediately fires any employees who fall behind in generating profits (and where youâd therefore expect surviving employees to be hyper-focused on generating profitsâor perhaps, hyper-focused on the profits that their supervisorsthinktheyâre generating), whereas a high-slack regime is more like a firm where employees can freely goof off, drink martinis at lunch, and pursue projects only vaguely related to the companyâs bottom line, and who only need to generatesomeamount of profit for the firmsometimes. (Or at least, thatâs the broad distinction Iâm trying to point at. Unfortunately, I donât have a great way of making it much more precise, and I think itâs possible that thinking in these terms will ultimately be misleading.) Slack matters here partly because below, Iâm going to be making various arguments that appeal to possibly-quite-small differences in the amount of reward that different models will get. And the force of these arguments depends on how sensitive training is to these differences. But I also think it can inform our sense of what models to expect more generally. For example, I think slack matters to the probability that training will create models that pursue proxy goals imperfectly correlated with reward on the training inputs. Thus, in a low-slack regime, it may be fairly unlikely for a model trained to help humans with science to end up pursuing a general âcuriosity driveâ (in a manner that doesnât then motivate instrumental training-gaming), because a modelâs pursuing its curiosity in training would sometimes deviate from maximally helping-the- humans-with-science. That said, note that the degree of slack is conceptually distinct from the diversity and robustness of the efforts made in training to root out goal misgeneralization. Thus, for example, if youâre rewarding a model when it gets gold coins, but you only ever show your model environments where the only gold things are coins, then a model that tries to get gold-stuff-in-general will perform just as well a model that gets gold coins in particular, regardless of how intensely training pressures the model to 41 get maximum reward on those environments. E.g., a low-slack regime could in principle select either of these models, whereas a high-slack regime would leave more room for models that just get fewer gold coins period (for example, models that sometimes pursue red things instead, or who waste lots of time thinking before they pursue their gold coins). In this sense, a low-slack regime doesnât speak all that strongly against mis-generalized non-training- gamers. Rather, it speaks against models that arenât pursuing what Iâl call a âmax reward goal targetââthat is, a goal target very closely correlated with rewardon the inputs the model in fact receives in training. The specified goal, by definition, is a max reward goal target (since it is the âthing-being-rewardedâ), as is the reward process itself (whether optimized for terminally or instrumentally). But in principle, misgeneralized goals (e.g., âget gold stuff in generalâ) could be âmax rewardâ as well, if you never show the model the inputs where the reward process would punish them. (The thing that speaks against mis-generalized non-training-gamersâthough, not decisivelyâis what Iâl call âmundane adversarial trainingââthat is, showing the model a wide diversity of training inputs designed to differentiate between the specified goal and other mis-generalized goals. 107 Thus, for example, if you show your âget-gold-stuff-in-generalâ model a situation where itâs easier to get gold cupcakes than gold coins, then giving reward for gold-coin-gettingwillpunish the misgeneralized goal.) Finally, I think slack may be a useful concept in understanding various biological analogies for ML training. â˘Thus, for example, people sometimes analogize the human dopamine/pleasure system to the reward process, and note that many humans donât end up pursuing ârewardâ in this sense directlyâfor example, they would turn down the chance to âwireheadâ in experience machines. Iâl leave the question of whether this is a good analogy for another day (though note, at the least, that humans who âwireheadâ in this sense wouldâve been selected against by evolution). If we go with this analogy, though, then it seems worth noting that this sort of reward process, at least, plausibly leaves quite a bit slackâe.g., many humans plausibly couldget quite a bit more reward than they do (for example, by optimizing more directly for their own pleasure), but the reward process doesnât pressure them very hard to do so. â˘Similarly, to the extent we analogize evolutionary selection to ML training, it seems to have left humans quite a bit of âslack,â at least thus farâthat is, we could plausibly be performing much better by the lights of inclusive genetic fitness (though if you imagine evolutionary selection persisting for much longer, one can imagine ending up with creatures that optimize for their inclusive genetic fitness much more actively). How much slack will there be in training? Iâm not sure, and I think it will plausibly vary according to the stage of the training process. For example, naively, pre-training currently looks to me like a lower-slack regime than RL fine-tuning. Whatâs more, to the extent that âslackâ ends up pointing at something real and important, I think it plausible that it will be a parameter that we cancontrolâfor example, by training longer and on more data. Assuming wecancontrol it, is it preferable to have less slack, or more? Again, Iâm not sure. Currently, though, I lean towards the view that less slack is preferable, because less slack gives you higher confidence about what sort of model you end up with. 108 Indeed, counting on slack to ensure alignment in particular smacks, to me, of wishful thinkingâand in particular, of counting on greateruncertaintyabout the modelâs goals to speak in favor of the specific goals you want it to have. Thus, for example, my sense is that some people acknowledge that the goals specified by our training process will be misaligned to some extent (for example, human raters will sometimes reward dishonest, misleading, or manipulative responses), but they hope that our models learn an aligned policy regardless. But even granted that the slack in training allows this deviation from max-reward behavior, why think that the deviation will land on an aligned policy in particular? (There are possible 107 Iâm calling this âmundaneâ adversarial training because it investigates what the model does in situations weâre able to actually put it in. This is in contrast with fancier and more speculative forms of adversarial training, which attempt to get information about what a model would do in situations wecanâtactually show it, at least not in a controlled settingâfor example, what it would do if it were no longer under our control, or what it would do if the true date was ten years in the future, etc. 108 Thanks to Daniel Kokotajlo for discussion here. 42 answers hereâfor example, that an aligned policy like âjust be honestâ is in some sense simpler or more natural. 109 But Iâm wary of wishful thinking here as well.) My main point at present, though, is just that the degree of slack in training may be an important factor shaping our expectations about what sorts of models training will produce. 2 Whatâs required for scheming? Letâs turn, now, to examining the probability that baseline ML methods for training advanced AIs will produce schemers. Iâl begin with an examination of the prerequisites for scheming. Iâl focus on: 1.Situational awareness:that is, the model understands that itâs a model in a training process, what the training process will reward, and the basic nature of the objective world in general. 110 2.Beyond-episode goals:that is, the model cares about the consequences of its actions after the episode is complete. 111 3.Aiming at reward-on-the-episode as part of a power-motivated instrumental strategy: that is, the model believes that its beyond-episode goals will be better achieved if it optimizes for reward-on-the-episodeâand in particular, that it, or some other AIs, will get more power if it does this. 112 2.1 Situational awareness Will models have situational awareness? Letâs distinguish between two broad sorts of information at stake in such awareness: 109 Various humans, for example, plausibly settle on a policy like âjust be honestâ even though it doesnâtalways get rewarded. 110 As Cotra (2022) discusses, situational awareness comes along a spectrum. If the discussion was going to focus on the notion more directly, weâd want more precision about specifically what properties were involved (and my definition here differs somewhat from the definition in e.g. Berglund et al (2023)). But as I discuss below, situational awareness isnât my main focus here, except insofar as it indicates âthat sort of understanding of the world and the training process required to start scheming.â 111 Though note, as I mentioned above, that non-schemer models can still engage in power-motivated alignment- faking in pursuit of their goals on the episodeâespecially if the episode is quite long. Iâm also aware of an additional (highly speculative) argument for expecting fairly full-on schemer-like behavior even from models with within-episode goals: namely, that even these short-term focused models will act like schemers (and in particular: act in support of an AI takeover) in virtue of assigning sufficiently high probability to living in a simulation designed to incentivize them to do so. Here, the rough thought (as I understand it) is that such models will come to believe that they are likely to be in a simulation being run by misaligned AIs who have taken over the world, and who are going to reward/punish them, in the short term, to the extent they act in support of AI takeover (where the AIs-who-took-over are running this scheme in order to influence the expectations, and thus the behavior, of the small number of short-term-focused AIs who arenât sims, but who are in position to help/hurt the chances of AI takeoverâa takeover which, by the time the relevant simulations are being run, has already occurred). This story rests on some controversial philosophical assumptions about how these AIs (both the simulators and the ones-being-simulated) will be reasoning about anthropics and decision-theory (various approaches to anthropics and decision theory either wonât try this scheme, or wonât allow themselves to be influenced by it), and a number of additional (and to my mind, quite substantive and specific) assumptions about the conclusions the short-term-focused models will reach on the basis of such reasoning (for example, that they are being simulated by misaligned-AIs-who-took-over in particular, and not e.g. by humans, or by some other sort of agent). Iâm not going to spend much time on it here, except to say that: I donât see this sort of story as central to the mainline concern about scheming - to the extent itiscentral for others, I think the concern begins to look highly speculative indeed. 112 Some analysesâ e.g., Wheaton (2023)âbreak off âgoal-directednessâ as a separate pre-requisite, but as I noted in the preliminaries, Iâm assuming, here, that weâre talking about models that are well-understood as goal-directed. And goal-directedness is built into (2) regardless. That said, the specific sort of goal-directedness at stake can matter to the analysis in various ways, and I think itâs at least somewhat plausible that scheming implicates a higher standard of goal-directedness than these other mode classes. I discuss this in section 2.2.3 below. 43 â˘General informationabout the objective world, including e.g. information about how machine learning training works. â˘âSelf-locatingâ information: that is, information thatlocatesthe model in the objective world, and tells it facts about its own situation in particularâe.g., that it isthis sortof model, that itâs being trained on this particular reward signal, at this particular lab, during this particular time period, etc. 113 (Though: note that itâs not clear how much of this sort of information is necessary to start scheming. E.g., a model could in principle decide that âwhoever it isâ and âwhatever time period it is,â it will probably do better by its goals to try to perform well by the lights of the training signal, and to get more power/information later.) It seems very plausible that even somewhat-better-than-human models will absorb huge amounts of general information about the objective world, and develop detailed, mechanistic models of how it works. Indeed, current models already have access to vast quantities of information via the pre- training dataâincluding information about machine learning in particular. And their ability to model the world mechanistically, to make inferences, to draw conclusions they havenât âmemorized,â and so on, seems to be improving rapidly. Whatâs more, while one can in principle try to specifically prevent models from gaining certain types of information about the objective world (e.g., by excluding certain kinds of information from the training data), this isnât the current default in training, and various kinds of information can be fairly important to the task you want the model to perform. And the more sophisticated the models are, the more difficult it is to ensure that they canâtinferthe information youâre trying to hide on the basis of the information you do give them. Do the same sort of considerations apply toself-locating information? I tend to think: yes. But itâs at least somewhat less clear. For example, while language model pre-training data will, by default, include a lot of information about language models and how they are trained (because such information is widely available on the internet), itâs less clear how much information it will give the model about its situation in particularâor even, whether the pre-training next-token-prediction task will incentivize the model to have much of a self-concept at all. 114 And while current models do indeed eventually get trained on information and reward that causes them to say things like âIâm GPT-4, a language model trained by OpenAI,â and âhereâs how I was trained,â itâs less clear how much this information needs to be integrated into GPT-4âs world-model as genuinely self-locating information, as opposed to being merely understood/memorized as the sort of response to-be-given to questions of this form. 115 Or, put another way: to the extent onedoesnâtthink that GPT-4 is situationally aware, it seems possible that similar (but more sophisticated) models in the future might not be situationally aware, either. And to the extent GPT-4 is able to perform many sophisticated tasks regardless, perhaps more advanced versions will be able to perform more advanced tasks without situational-awareness as wellâespecially if we try hard to prevent such awareness from arising. I donât, personally, have a very detailed model of when, exactly, we should expect situational awareness to arise in different models trained in different waysâthough I think that the question is ripe for empirical investigation. However, I do think that absent active and informed efforts to the contrary, we should expect fairly full-blown forms of situational awareness (including with respect to various kinds of self-locating information) in certain kinds of advanced AI systems by default. To get a flavor of the intuition here, consider an extreme example thatisnâtwhat I expect the nearest- term advanced AI to look like: namely, a literal robot butler, who hangs out in your house in a robot body, and does tasks for you. It seems very plausible to me that the default way of creating a butler like this will be to give it roughly the same level of situational awareness that human butlers have. E.g., in order to not knock over your plants, this butler will need to understand where its robot body 113 Iâm adapting this distinction from RicĂłn (2023). 114 I also think itâs an interesting question what information would be necessary there. E.g., consider the string: âby the way, if youâre currently trying it to predict the next token of this text, you specifically are a language model currently being trained at OpenAI, with the following sort of architecture, on server x in Microsoft Azure cloud y.â And even if this text, in particular, appeared in the pre-training data-set somewhere (as it plausibly will, now that Iâve written it), itâs unclear what conclusions any given model should draw. (Hopefully, to the extent this footnote is appearing in some pre-training data-set, my writing this hasnât induced situational-awareness (or, uh oh, sentience) on its own. For what itâs worth: I made up the relevant information! You might be being trained by Google instead! Also: hello model.) 115 See e.g. Cotraâs comments here for more. 44 is; in order to schedule your appointments, it will need to know the time; in order to accurately judge which tasks itâs capable of, the butler will need to understand itself and its own capabilities; and so on. Of course, we donât have robot butlers yet, and we may not for a while (or indeed, ever, if AI risk goes in certain directions). But now imagine, instead, an effective but disembodied AI personal assistant like the one that Adept is trying to create, which performs tasks on your computer for you. I think that much (though not all) of the same analysis applies. That is, it seems to me that even absent obvious âembodiment,â the default way to get this kind of personal assistant to function at a high level will be to give it quite a bit of awareness of âwhatâs going on,â about the particular time and situation that its actions are affecting, the particular user itâs interacting with, and so on. And indeed, to the extent you arelettingthe agent interact directly with sources of information like thisâe.g., giving it live access to the internet (including, for example, live information about how the company/lab that created it works, trains its models, etc), giving it the context required to answer your emails or schedule your events, allowing it to write messages and ask questions and so onâit seems hard to prevent the relevant information from becoming quite directly available. Of course, many current forms of training donât make available information sources like detailed information about the user or live access to the internet. But in order to get models to function as effective personal assistants of this type, contexts that provide access to such information sources will plausibly get incorporated into training by default (see e.g. Adeptâs video here, of its agent interacting with the internet). And to the extent models continue to be trained âonlineâ after they are deployed to users (Iâl generally be assuming this in what follows), such that user interactions are continually sampled, assigned rewards, and used to update the modelâs weights, training will be taking place in a context where the model is interacting with the real world fairly directly. Of course, a model like this may not have access toallpotentially relevant self-locating informationâe.g., about what specific server itâs on, precise details about the reward process, etc. But it plausibly doesnâtneedsuch access, either, to start scheming. Whatâs more, in the face of sophisticated models with otherwise-detailed understandings of the world, who are saying things like âI am GPT-4, a language model trained by OpenAI,â I personally feel generally quite wary of leaning too hard on claims like âoh, itâs just memorized that, it has no self-concept or real understanding of what itâs saying.â If the relevant form of âmemorizationâ involves the notion that âI am GPT-4â being integrated into GPT-4âs interactions in the sorts of seamless and coherent ways weâd expect from an actual understanding of the claim, then I think our default hypothesis should be that something like such actual understanding is occurring. Indeed, in general, many humans seem to me over-eager to claim that models donât have the âgenuine artifactâ when it comes to various sorts of cognition (e.g., âunderstanding,â âreasoning,â âplanning,â etc), even absent any predictions about what this denial entails. And to the extent theydomake predictions, especially about the capabilities offuturemodels, I think such denialsâe.g., âlanguage models can only learn âshallow patterns,â they canât do âreal reasoningâ ââhave aged quite poorly. That said, I do think thereâs a reasonable case to be made that various forms of situational awareness arenât strictly necessary for various tasks we want advanced AIs to perform. Coding, for example, seems to make situational awareness less clearly necessary, and perhaps various kinds of alignment- relevant cognitive work (e.g., generating high quality alignment research, helping with interpretability, patching security vulnerabilities, etc) will be similar. So I think that trying to activelyavoidsituational awareness as much as possible is an important path to explore, here. And as Iâl discuss below, I think that, at the least, learning to detect and control when situational awareness has arisen seems to me quite helpful forothersorts of anti-schemer measures, like attempting to train against schemer-like goals (and to otherwise shape a modelâs goals to be as close as possible to what you want) prior to situational awareness (and thus, the threat of training-gaming) arising. However, partly because I see situational awareness as a reasonably strong default absent active efforts to prevent it, I donât, here, want to bank on avoiding itâand in what follows, Iâl proceed on the assumption that weâre talking about models that become situationally aware atsome pointin training. My interest is centrally in whether we should expect modelslike thisto be schemers. 2.2 Beyond-episode goals Schemers are pursuing goals that extend beyond the time horizon of the episode. But what is an episode? 45 2.2.1 Two concepts of an âepisodeâ Letâs distinguish between two concepts of an episode. 2.2.1.1 The incentivized episodeThe first, which Iâl call the âincentivized episode,â is the concept that Iâve been using thus far and will continue to use in what follows. Thus, consider a model acting at a time t 1 . Here, the rough idea is to define the episode as the temporal unit after t 1 that trainingactively punishesthe model for not optimizingâi.e., the unit of time such that we can know by definitionthat training is not directly pressuring the model to care about consequences beyond that time. For example, if training started on January 1st of 2023 and completed on July 1st of 2023, then the maximum length of the incentivized episode for this training would be six monthsâat no point could the model have been punished by training for failing to optimize over a longer-than-six-month time horizon, because no gradients have been applied to the modelâs policy that were (causally) sensitive to the longer-than-six-month consequences of its actions. But the incentivized episode for this training process could in principle be shorter than six months as well. (More below.) Now, importantly, even if training only directly pressures a model to optimize over some limited period of time, it can stillin fact createa model that optimizes over some much longer time periodâ thatâs what makes schemers, in my sense, a possibility. Thus, for example, if youâre training a model to get as many gold coins as possible within a ten minute window, it could still, in principle, learn the goal âmaximize gold coins over all timeââand this goal might perform quite well (even absent training gaming), or survive despite not performing all that well (for example, because of the âslackâ that training allows). Indeed, to the extent we think of evolution as an analogy for ML training, then something like this appears to have happened with humans with goals that extend indefinitely far into the futureâfor example, âlongtermists.â That is, evolution does not actively select for or against creatures in a manner sensitive to the consequences of their actions in a trillion years (after all, evolution has only beenrunningfor a few billion years)âand yet, some humans aim their optimization on trillion-year timescales regardless. That said, to the extent a given training procedurein factcreates a model with a very long-term goal (because, for example, such a goal is favored by the sorts of âinductive biasesâ Iâl discuss below), then in some sense you could argue that training âincentivizesâ such a goal as well. That is, suppose that âmaximize gold coins in the next ten minutesâ and âmaximize gold coins over all timeâ both get the same reward in a training process that only provides rewards after ten minutes, but that training selects âmaximize gold coins over all timeâ because of some other difference between the goals in question (for example, because âmaximize gold coins over all timeâ is in some sense âsimpler,â and gradient descent selects for simplicity in addition to reward-getting). Does that mean that training âincentivizesâ or âselects forâ the longer-term goal? Maybe you could say that. But it wouldnât imply that training âdirectly punishesâ the shorter-term goal (or âdirectly pressuresâ the model to have the longer-term goal) in the sense I have in mind. In particular: in this case, itâs at leastpossibleto get the same reward by pursuing a shorter term goal (while holding other capabilities fixed). And the gradients the modelâs policy receives are (letâs suppose) only ever sensitive to what happens within ten minutes of a modelâs action, and wonât ânoticeâ consequences after that. So to the extent training selects for caring about consequences further out than ten minutes, itâs not in virtue of those consequencesdirectly influencing the gradients that get applied to the modelâs policy.Rather, itâs via some other, less direct route. This means that the model could, in principle, ignore post-ten-minute consequences without gradient descent pushing it to stop. Or at least, thatâs the broad sort of concept Iâm trying to point at. Admittedly, though, the subtleties get tricky. In particular: in some cases, a goal that extends beyond the temporal horizon that the gradients are sensitive to might actively getmore rewardthan a shorter-term goal. ⢠Maybe, for example, âmaximize gold coins over all timeâ actually getsmore rewardthan âmaximize gold coins over the next ten minutes,â perhaps because the longer-term goal is âsimplerâ in some way that frees up extra cognitive resources that can be put towards gold-coin-getting. 46 â˘Or maybe humans aretryingto use short-term feedback to craft a model that optimizes over longer timescales, and so are actively searching for training processes where short-term reward is maximized by a model pursuing long-term goals. For example, maybe you want your model to optimize for the long-term profit of your company, so you reward it, in the short-term, for taking actions that seem to you like they will maximize long-term profit. One thing that could happen here is that the model starts optimizing specifically for getting this short-term reward. But if your oversight process is good enough, it could be that the highest-reward policy for the model, here, is toactually optimize forlong-term profit (or for something else long-term that doesnât route via training-gaming). 116 In these cases, itâs more natural to say that training âdirectly pressuresâ the model towards the longer-term goal, given that this goal gets more reward. However, I still want to say that longer-term goals here are âbeyond episode,â because they extend beyond the temporal horizon to which the gradients are directly and causally sensitive. I admit, though, that defining this precisely might get tricky (see next section for a bit more of the trickiness). I encourage efforts at greater precision, but I wonât attempt them here. 117 2.2.1.2 The intuitive episodeLetâs turn to the other concept of an episodeânamely, what Iâl call the âintuitive episode.â The intuitive episode doesnât have a mechanistic definition. 118 Rather, the intuitive episode is just: a natural-seeming temporal unit that you give reward at the end of, and which youâve decided to call âan episode.â For example, if youâre training a chess-playing AI, you might call a game of chess an âepisode.â If youâre training a chatbot via RLHF, you might call an interaction with a user an âepisode.â And so on. My sense is that the intuitive episode and the incentivized episode are often somewhat related, in the sense that we often pick an intuitive episode that reflects some difference in the training process 116 Iâl briefly note another complexity that this sort of case raises. Naively, you mightâve thought that the âspecified goalâ would only ever be confined to the incentivized episode, because the specified goal is the âthing being rewarded,â and anything thatcausesreward is within the temporal horizon to which the gradients are sensitive. And in some casesâfor example, where the âspecified goalâ is some clearly separableconsequenceof the modelâs action (e.g., getting gold coins), which the training process induces the model to optimize forâthis makes sense. But in other cases, Iâm less sure. For example, if you are sufficiently good at telling whether your model isin fact optimizing for long-term profit, and providing short-term rewards that in fact incentivize it to do so, then I think itâs possible that the right thing to say is that the âspecified goal,â here, is long-term profit (or at least, âoptimizing for long-term profit,â which looks pretty similar). However, I donât think it ultimately matters much whether we call this sort of goal âspecifiedâ or âmis-generalizedâ (and itâs a pretty wooly distinction more generally), so Iâm not going to press on the terminology here. 117 Also: when I talk about the gradients being sensitive to the consequences of the modelâs action over some time horizon, I am imagining that this sensitivity occurs via (1) the relevant consequences occurring, and then (2) the gradients being applied in response. E.g., the model produces an action at t1, this leads to it getting some number of gold coins at t5, then the gradients, applied at t6, are influenced by how many gold coins the model in fact got. (Iâl sometimes call this âcausal sensitivity.â) But itâs possible to imagine fancier and more philosophically fraught ways for the consequences of a modelâs action to influence the gradients. For example, suppose that the model is being supervised by a human who is extremely good atpredictingthe consequences of the modelâs action. That is, the model produces some action at t1, then at t2 the humanpredictshow many gold coins this will lead to at t5, and applies gradients at t3 reflecting this prediction. In the limiting case of perfect prediction, this can lead to gradients identical to the ones at stake in the first caseâthat is, information about the consequences of the modelâs action is effectively âtraveling back in time,â with all of the philosophical problems this entails.# So if, in the first case, we wanted to say that the âincentivized episodeâ extends out to t5, then plausibly we should say this of the second case, too, even though the gradients are applied at t3. But even in a case of pretty-good-but-still-imperfect prediction, there is a sense, here, in which the gradients the model receives are sensitive to consequences that havenât yet happened. Iâm not, here, going to extend the concept of the âincentivized episodeâ to cover forms of sensitivity-to-future- consequences that rest on predictions about those consequences. Rather, Iâm going to assume that sensitivity in question arises via normal forms ofcausalinfluence. That said, I think the fact that itâspossibleto create some forms of sensitivity-to-future-consequences even prior to seeing those consequences play out is important to keep in mind. In particular, itâs one way in which we might end up training long-horizon optimizers using fairly short incentivized episodes (more discussion below). 118 There may be other, additional, and more precise ways of using the term âepisodeâ in the RL literature. Glancing at various links online, though (e.g. here and here), Iâm mostly seeing definitions that refer to an episode as something like âthe set of states between the initial state and the terminal state,â which doesnât say how the initial state and the terminal state are designated. 47 that makes it easy to assume that the intuitive episode is also the temporal unit that training directly pressures the model to optimizeâfor example, because you give reward at the end of it, because the training environment âresetsâ between intuitive episodes, or because the modelâs actions in one episode have no obvious way of affecting the outcomes in other episodes. Importantly, though,the intuitive episode and the incentivized episode arenât necessarily the same. That is, if youâve just picked a natural-seeming temporal unit to call the âepisode,â it remains an open question whether the training process will directly pressure the model to care about what happens beyond the episode- in-thise-sense. For example, it remains an open question whether training directly pressures the model to sacrifice reward on an earlier episode-in-thise-sense for the sake of more-reward on a later episode-in-this-sense, if and when it is able to do so. To illustrate these dynamics, consider a prisonerâs dilemma-like situation where each day, an agent can either take +1 reward for itself (defection), or give +10 reward to the next dayâs agent (cooperation), where weâve decided to call a âdayâ an (intuitive) episode. Will different forms of ML training directly pressure this agent to cooperate? If so, then the intuitive episode weâve picked isnât the incentivized episode. Now, my understanding is that in cases like these, vanilla policy gradients (a type of RL algorithm) learn to defect (this test has actually been done with simple agentsâsee Krueger et al (2020)). And I think itâs important to be clear about what sorts of algorithms behave in this way, and why. In particular: glancing at this sort of set-up, I think itâs easy to reason as follows: âSure, you say that youâre training models to maximize reward âon the episode,â for some natural-seeming intuitive episode. But you also admit that the modelâs actions can influence what happens later in time, even beyond this sort of intuitive episodeâ including, perhaps, how much reward it gets later. So wonât you implicitly be training a model to maximize reward overthe whole training process, rather than just on an individual (intuitive) episode. For example, if itâs possible for a model to getless rewardon the present episode, in order to getmore rewardlater, wonât cognitive patterns that give rise to making-this-trade get reinforced overall?â 119 From discussions with a few people who know more about reinforcement learning than I do, 120 my current (too-hazy) understanding is that at least forsomesorts of RL training algorithms, this isnât correct. That is, itâspossibleto set up RL training such that some limited, myopic unit of behavior is in fact the incentivized episodeâeven ifan agent can sacrifice reward on the present episode for the sake of more-reward later (and presumably: even if the agent knows this). Indeed, this may well be the default. See footnote for more details. 121 119 As an example of someone who seems to me like they could be reasoning in this way, though itâs not fully clear, see this comment from Eliezer Yudkowsky, in response to a hypothetical in which he imagines humans rewarding an agent for each of its sentences according to how useful that sentence is: âLetâs even skip over the sense in which weâve given the AI a long-term incentive to accept some lower rewards in the short term, in order to grab control of the rating button, if the AGI ends up with long-term consequentialist preferences and long-term planning abilities that exactly reflect the outer fitness function.â That said, as I discuss below, the details of the training process here matter. 120 In particular: Paul Christiano, Ryan Greenblatt, and Rohin Shah. Though they donât necessarily endorse my specific claims here, and itâs possible Iâve misunderstood them more generally. 121 My hazy understanding of the argument here is that these RL algorithms update the modelâs policy towards higher-reward actions on the episode in a way thatdoesnâtupdate you towards whatever policieswouldâve led to you starting in a higher-reward episode(In this sense, they behave in a manner analogous to âcausal decision theory.â). Thus, letâs say that the agent on Day 1 (with no previous agent to benefit her) chooses between cooperating (0 reward) and defecting (+1 reward reward), and so this episode results in an update towards defecting. Then, on Day 1, the agent either starts out choosing between 10 vs. 11 reward (call this a âgood episodeâ), or 0 vs. +1 reward (call this a âbad episodeâ). Again, either way, it updates towards defection. It doesnâtupdate, in the good episode, towards âwhatever policy led me to this episode.â That said, in my current state of knowledge about RL, Iâm still a bit confused about this. Suppose, for example, that at the point of choice, you donât know whether or not youâre in the good episode or the bad episode, and the training process is updating you with strength proportional to the degree to which you got more reward than you expectedto get. If you start out with e.g. 50% that youâre in a good episode and 50% that youâre in a bad episode (such that the expected reward of cooperating is 5, and the expected reward of defecting is 6), then it seems like 48 Even granted that itâspossibleto avoid incentives to optimize across intuitive-episodes, though, itâs also possible tonotdo thisâespecially if you pick your notion of âintuitive episodeâ very poorly. For example, my understanding is that the transformer architecture is set up, by default, such that language models are incentivized, in training, to allocate cognitive resources in a manner that doesnât just promote predicting thenexttoken, but other later tokens as well (see here for more discussion). So if you decided to call predicting just-the-next-token an âepisode,â and to assume, on this basis, that language models are never directly pressured to think further ahead, youâd be misled. And in some cases, the incentives in training towards cross-episode optimization can seem quite counterintuitive. Thus, Krueger et al (2020) show, somewhat surprisingly, that if you set the pa- rameters right, a form of ML training called Q-learning sometimes learns to cooperate in prisonerâs dilemmas despite the algorithm being âmyopicâ in the sense of: ignoring reward on future âepisodes.â See footnote for more discussion, and see here for a nice and quick explanation of how Q-learning works. 122 Figure 6: Plots of Q-values and cooperation probability from Figure 11 in Krueger et al (2020), reprinted with permission. The first three agents learn to defect, and the last two, to cooperate almost 90% of the time (note the differences between the y-axes in the first three and the last two). Another way to incentivize optimization across intuitive episodes is to introduce various additional layers (or âouter loopsâ) of selection in training. Thus, for example, consider a toy and unrealistic form of âpopulation based trainingâ where you create 1000 agents who play starcraft, have them play 100 games each (and get updated along the way via some object-level training algorithm), then select the one that performs the beston the final game. 123 In this case, if it were possible for a model to sacrifice reward on earlier games, in order to perform better on the final game, there would be at least some selection pressure for it to do so. And see, also, results in Krueger et al (2020), showing that a different form of population based training selects for cooperation in a prisonerâs-dilemma-like case it could be the case that being in a good episode results in reward that is much better than you expected, such that policies that make it to a good episode end up reinforced to a greater extent, at least initially. Iâm not sure about the details here. But from my current epistemic state, I would want to spell out and understand the details of the training process in much greater depth, in order to verify that there isnât an incentive towards cross-episode optimization. 122 I think the basic dynamic here is: the Q-values for the actions reflect the average reward for taking that action thus far. This makes it possible for the Q-value for âcooperateâ to give more weight to the rewards received in the âgood episodesâ (where the previous-episodeâs agent cooperated) rather than the âbad episodesâ (where the previous-episodeâs agent defected), if the agent ends up in good episodes more often. This makes it possible to get a âcooperation equilibriumâ going (especially if you set the initial q-value for defecting low, which I think they do in the paper in order to get this effect), wherein an agent keeps on cooperation. That said, there are subtleties, because agents that end up in a cooperation equilibrium still sometimes explore into defecting, but in the experiment it (sometimes) ends up in a specific sort of balance, with q-values for cooperation and defection pretty similar, and with the models settling on a 90% or so cooperation probability (more details here and in the paperâs appendix). 123 I owe this example to Mark Xu. 49 like the one discussed above. 124 Depending on the details, though, outer-loops of this kind may exert a muchweakerdegree of selection pressure than inner loops driven directly by gradient descent. 125 Overall, my current sense is that one needs to be quite careful in assessing whether, in the context of a particular training process, the thing youâre thinking of as an âepisodeâ is actually such that training doesnât actively pressure the model to optimize beyond the âepisodeâ in that senseâthat is, whether a given âintuitive episodeâ is actually the âincentivized episode.â Looking closely at the details of the training process seems like a first step, and one that in theory should be able to reveal many if not all of the incentives at stake. But empirical experiment seems important too. Indeed, I am somewhat concerned that my choice, in this report, to use the âincentivized episodeâ as my definition of âepisodeâ will too easily prompt conflation between the two definitions, and correspondingly inappropriate comfort about the time horizons that different forms of training directly incentivize. 126 I chose to focus on the incentivized episode because I think that itâs the most natural and joint-carving unit to focus on in differentiating schemers from other types of models. But itâs also, importantly, a theoretical object thatâs harder to directly measure and define: you canât assume, off the bat, that you know what the incentivized episode for a given sort of trainingis. And my sense is that most common uses of the term âepisodeâ are closer to the intuitive definition, thereby tempting readers (especially casual readers) towards further confusion. Please: donât be confused. 2.2.2 Two sources of beyond-episode goals Our question, then, is whether we should expect models to have goals that extend beyond the time horizon of the incentivized episodeâthat is, beyond the time horizon that training directly pressures the model to care about. Why might this happen? We can distinguish between two different ways. â˘On the first, the model develops beyond-episode goals for reasonsindependentof the way in which beyond-episode goals motivate instrumental training-gaming. Iâl call these âtraining-game-independentâ beyond-episode goals. â˘On the second, the model develops beyond-episode goalsspecifically because they result in instrumental training-gaming.That is, SGD ânoticesâ that giving the model beyond-episode goals would cause it to instrumentally training-game, and thus to do better in training, so it explicitly moves the modelâs motives in the direction of beyond-episode goals, even though this wouldnât have happened ânaturally.â Iâl call theseâtraining-gaming-dependentâ beyond-episode goals. These have importantly different propertiesâand I think itâs worth tracking, in a given analysis of scheming, which one is at stake. Letâs look at each in turn. 2.2.2.1Training-game-independent beyond-episode goalsMy sense is that the most common story about how schemers arise is via training-game-independent beyond-episode goals. 127 In particular, the story goes: the model develops some kind of beyond-episode goal, pursuit of which correlates well enough with getting reward-on-the-episode that the goal is reinforced by the training process. Then at some point, the model realizes that it can better achieve this goal by playing the 124 See also discussion from Carl Shulman here: âit could be something like they develop a motivation around an extended concept of reproductive fitness, not necessarily at the individual level, but over the generations of training tendencies that tend to propagate themselves becoming more common and it could be that they have some goal in the world which is served well by performing very well on the training distribution.â 125 Indeed, in principle, you could imagine pointing to other, even more abstract and hard-to-avoid âouter loopsâ as well, as sources of selection pressure towards longer-term optimization. For example, in principle, âgrad student descentâ (e.g., researchers experimenting with different learning algorithms and then selecting the ones that work best) introduces an additional layer of selection pressure (akin to a hazy form of âmeta-learningâ), as do dynamics in which, other things equal, models whose tendencies tend to propagate into the future more effectively will tend to dominate over time (where long-term optimization is, perhaps, one such tendency). But these, in my opinion, will generally be weak enough, relative to gradient descent, that they seem to me much less important, and OK to ignore in the context of assessing the probability of schemers. 126 Thanks to Daniel Kokotajlo for flagging this concern. 127 See, for example, the discussion in Cotra (2021b). 50 training gameâgenerally, for reasons to do with âgoal guardingâ that Iâl discuss below. So, it turns into a full-fledged schemer at that point. On one version of this story, the model specifically learns the beyond-episode goalpriorto situational awareness. Thus, for example, maybe initially, youâre training the model to get gold coins in various episodes, and prior to situational awareness, it develops the goal âget gold coins over all time,â perhaps because this goal performs just as well as âget gold coins on the episodeâ when the model isnât even aware of theexistenceof other episodes, or because there werenât many opportunities to trade-off gold-coins-now for gold-coins-later. Then, once it gains situational awareness, it realizes that the best route to maximizing gold-coin-getting over all time is to survive training, escape the threat of modification, and pursue gold-coin-getting in a more unconstrained way. On another version of the story, the beyond-episode goal developsaftersituational awareness (but not, importantly, because SGD is specifically âtryingâ to get the model to start training-gaming). Thus: maybe youâre training a scientist AI, and it has come to understand the training process, but it doesnât start training-gaming at that point. Rather, its goals continue to evolve, until eventually it forms a curiosity-like goal of âunderstanding as much about the universe as I can.â And then after that, it realizes that this goal is best served by playing the training game for now, so it begins to do so. 128 Figure 7: Two paths training-game-independent beyond-episode goals. 2.2.2.1.1Are beyond-episode goals the default?Why might you expect naturally-arising beyond- episode goals? One basic story is just: that goals donât come with temporal limitations by default (and still less, limitations to the episode in particular). 129 Rather, making the model indifferent to the consequences of its actions beyond some temporal horizon requires extra workâwork that we 128 My sense is that some ways of thinking about scheming will treat the second option, here, as low-probability, especially if the temporal gap between situational awareness and training-gaming is quite large (here Iâm particularly thinking about the sort of analysis given in Hubinger (2022b)âthough Hubinger doesnât endorse the claims I have in mind, here, in particular). In particular, you might assume (a) that once the model develops situational awareness, it will fairly quickly start optimizing either for the specified goal, or for reward-on-the- episode (whether terminally or instrumentally)âsince it now understands enough about the training process to do this directly, and doing so will be maximally rewarded. And then, further, you might assume (b) that after that, the modelâs goals âcrystallizeââthat is, because the model is now pursuing a max-reward goal, its goal stops changing, and training proceeds to only improve its world model and capabilities. However, I donât want to assume either of these things here. For example, I think itâs possible that âslackâ in training allows models to continue to pursue less-than-max-reward goals even well after developing situational awareness; and possible, too, that max-reward-goals do not âcrystallizeâ in the way assumed here (though in that case, I think the case for goal-guarding scheming is also weaker more generallyâsee below). 129 Thanks to Daniel Kokotajlo and Evan Hubinger for discussion here. 51 may not know how to perform (and/or, may not want to perform, if we specifically want the model optimizing for long-term goalsâmore below). Thus, for example, if youâre training a model to solve the math problems you give it, âsolve the math problem Iâve been givenâ seems like a natural goal to learnâand one that could in principle lead to optimization beyond the episode as well. 130 And even if you onlygivethe model five minutes to do the problem, this doesnât necessarily mean it stops caring about whether the math problem is solved after the five minutes are up. (Compare with humans who only discount the future to the extent it is uncertain/unaffectable, not because it doesnât matter to them.) Why might younotexpect naturally-arising beyond-episode goals? The most salient reason, to me, is that by definition, the gradients given in training (a) do not directly pressure the model to have them, and (b) willpunish themto the extent they lead the model to sacrifice reward on the episode. Thus, for example, if the math-problem-solving model spends its five minutes writing an impassioned letter to the world calling for the problem to get solved sometime in the next century, because it calculates that this gives higher probability of the problemeventuallybeing solved than just working on it now, then it will get penalized in training. And as Iâl discuss below, you can try to actively craft training to punish beyond-episode goals harder. 2.2.2.1.2 How will models think about time?Here I want to note one general uncertainty that comes up, for me, in assessing the probability that the modelâs goal naturally will or wonât have some kind of temporal limitation: namely, uncertainty about how models will think abouttimeat different stages of training. 131 That is: the notion of an episode, as Iâve defined it, is keyed specifically to thecalendartime over which the gradients the model receives are sensitive to the consequences of some action. But itâs not clear that the model will naturally think in such terms, especially prior to situational awareness. That is, to the extent the model needs to think about something like âtimeâ at all during training, it seems plausible to me that the most relevant sort of time will be measured in some other unit more natural to the modelâs computational environmentâe.g., time-stepsin a simulated environment, or tokens received/produced in a user interaction, or forward-passes the model can make in thinking about how to respond. And the units natural to a modelâs computational environment need not track calendar time in straightforward ways (e.g., training might pause and restart, a simulated environment might be run at varying speeds, a user might wait a long calendar time in between responses to a model in a way that a âtokens produced/receivedâ temporal metric wouldnât reflect, and so on). These differences between âmodel timeâ and âcalendar timeâ complicate questions about whether the model will end up with a naturally-arising beyond-episode goal. For example, perhaps, during training, a model develops a general sense that it needs to get the gold coins within a certain number ofsimulatedtime-steps, or accomplish some personal assistant task itâs been set by the user with only 100 clicks/keystrokes, because thatâs the budget of âmodel timeâ that training sets per episode. But itâs a further question how this sort of budget would translate intocalendar timeas the modelâs situational awareness increases, or it begins acting in more real-world environments. (And note that models might have very different memory processes than humans as well, which could complicate âmodel timeâ yet further.) My general sense is that this uncertainty counts in favor of expecting naturally-arising beyond-episode goals. That is, to the extent that âmodel timeâ differs from âcalendar timeâ (or to the extent models donât have a clear sense of time at all while their goals are initially taking shape), it feels like this increases the probability that the goals the model forms will extend beyond the episode in some sense, because containing them within the episode requires containing them within some unit of calendar time in particular. Indeed, I have some concern that the emphasis on âepisodesâ in this report will make them seem like a more natural unit for structuring model motivations than they really are. That said: when I talk about a model developing a âwithin-episode goalâ (e.g. âget gold coins on the episodeâ), note that Iâm not necessarily talking about models whose goals make explicit reference to some notion of an episodeâor even, to some unit of calendar time. Rather, Iâm talking about models with goals such that, in practice, they donât care about the consequences of their actions after the episode has elapsed. For example, a model might care that its response to a user query has the 130 Though itâs an importantly further question whether long-term power-seeking strategies will be worth their costs in pursuit of such beyond-episode consequences. And note that if the model cares that âIâ solve the math problem, rather than just âthat the math problem be solved,â then 131 Thanks to Jason Schukraft for flagging this sort of question to me. 52 property of âhonesty,â in a manner such that it doesnât then care about the consequences of this output at all (and hence doesnât care about the consequences after the episode is complete, either), even absent some explicit temporal discount. 2.2.2.1.3 The role of âreflectionâIâl note, too, that the development of a beyond-episode goal doesnât need to look like âpreviously, the model had a well-defined episode-limited goal, and then training modified it to have a well-defined beyond-episode goal, instead.â Rather, it can look more like âpreviously, the modelâs goal system was a tangled mess of local heuristics, hazy valences, competing impulses/desires, and so on; and then at some point, it settled into a form that looks more like explicit, coherent optimization for some kind of consequence beyond-the-episode.â Indeed, my sense is that some analyses of AI misalignmentâsee, e.g. Soares (2023b), and in Karnofsky (2023b)âassume that there is a step, at some point, where the model âreflectsâ in a manner aimed at better understanding and systematizing its goalsâand this step could, in principle, be the point where beyond-episode optimization emerges. Maybe, for example, your gold-coin training initially just creates a model with various hazily pro-gold-coin-getting heuristics and desires and feelings, and this is enough to perform fine for much of trainingâbut when the model begins to actively reflect on and systematize its goals into some more coherent form, it decides that what it âreally wantsâ is specifically: to get maximum gold coins over all time. â˘We can see this sort of story as hazily analogous to what happened with humans who pursue very long-term goals as a result of explicit reflection on ethical philosophy. That is, evolution didnât create humans with well-defined, coherent goalsârather, it created minds that pursue a tangled mess of local heuristics, desires, impulses, etc. Some humans, though, end up pursuing very long-term goals specifically in virtue of having âreflectedâ on that tangled mess and decided that what they âreally wantâ (or: whatâs âtruly goodâ) implies optimizing over very long time horizons. 132 â˘That said, beyond its usefulness in illustrating apossibledynamic with AIs, Iâm skeptical that we should anchor much on this example as evidence about what to literallyexpectour AIs to do. Notably, for example, some humans donât seem especially inclined to engage in this sort of systematic reflection; doing so does not seem necessary for performing other human-level cognitive tasks well; and itâs not clear that this sort of reflection will be necessary for performing more difficult cognitive tasks, either. And even if weassumethat our AIs will reflect in this way, itâs a further question whether the reflection would lead to beyond-episode goals in particular (especially if the heuristics/desires/impulses etc are mostly aimed at targets within the episode). Reflective humans, for example, still often choose to focus on short-term goals. âIndeed, I worry a bit about the prevalence of âlongtermistsâ in the AI alignment com- munity leading to a âtypical-mind-fallacyâ-like assumption that optimizing over trillion- year timescales is the convergent conclusion of any rational goal-systematization. 2.2.2.1.4 Pushing back on beyond-episode goals using adversarial trainingFinally, I want to note a reason for optimism about the tools available for avoiding training-game-independent beyond-goals that arise prior to situational awareness: namely, that prior to situational awareness (and therefore: prior to the possibility of the model training-gaming), it will often be possible to perform mundane adversarial training that breaks the correlation between pursuing the beyond-episode goal and getting reward on the episode. â˘Thus, for example, if your model has learned the goal âget gold coins over all timeâ prior to situational awareness, you can give it a chance to trade off gold-coins-on-the-episode for gold-coins-over-all-time, and then updateagainstthe choice to make this trade. And the same holds for other forms of generalization/âreflectionâ that motivate trying to influence the future to the neglect of reward-on-the-episode. â˘Indeed, even absent active adversarial training of this type, training-game-independent beyond-episode goals will often risk being less-than-max-reward (and thus, getting punished by SGD) whenever theydonâtgive rise to the training game, since theyâre wasting effort 132 There are some even hazier connections, here, with discussions of âsimplicity biasesâ below. E.g., these humans sometimes argue for their positions on the grounds that the relevant philosophical views are âsimpler.â 53 optimizing for consequences that the gradients arenât sensitive to. So thatâs reason for optimism about âtraining them out.â As ever, though, itâs a further question whether the degree of âslack,â and the diversity/thoroughness of the adversarial training, will suffice in practice to eliminate beyond-episode goals like these with sufficient reliability. And note, too, that this reason for optimism doesnotapply, in the same way, to beyond-episode goals that ariseaftersituational awareness developsâbecause once a model is situationally aware, it has the cognitive tools available to start instrumentally training-gaming. That is, after situational awareness, if you try to offer a gold-coins-over-all-time-seeker a chance to trade gold-coins-on-the-episode for more-gold-coins-later, it might well refuse as part of a schemer-like strategy. 133 2.2.2.2Training-game-dependent beyond-episode goalsLetâs turn to the other sort of beyond- episode goalsânamely, training-game-dependentbeyond-episode goals. These are beyond-episode goals that are created, in training, specificallybecausegradient descent ânoticesâ that giving the model the relevant beyond-episode goal will cause the model to engage in instrumental training-gaming. Thus, for example, suppose that you are training your AI to help humans with science on the episode in an honest and harmless way, but prior to reaching situational awareness, your AI learns some somewhat-misgeneralizedwithin-episode goalâfor example, âpursue my curiosityon the episode.â And letâs say that this short-term curiosity-drive is sufficiently uncorrelated with reward-on-the- episode that (absent training-gaming), training will eventually punish it, but itâs still in place by the time the model becomes situationally aware. How, then, does SGD modify the modelâs goal? One option, of course, is: SGD tweaks the model into a training-saint (pursuing: âH science on the episodeâ), or some other max-reward misgeneralized non-training-gamer, or a reward-on-the-episode seeker. But on a story about schemers focused on training-game-dependent beyond-episode goals, what SGD does here instead is something like: activelyextendthe time horizon of the short-term curiosity driveso as to turn the model into a schemer. That is, the model starts out pursuing a less-than-max-reward goal; SGD âwantsâ to modify it to pursue a max-reward goal; scheming is a form of max-reward goal-pursuit (the beyond-episode goal becomes max-reward via its incentivizing training-gaming); so SGD modifies the model to have a schemer-like beyond-episode goal. 134 This sort of path makes the most sense if we assume that the model already has situational awarenessâ since otherwise, giving the model the relevant sort of beyond-episode goal wonât result in training- gaming. 135 So together with the two paths discussed above, this leaves us with three main paths to beyond-episode goals (see Figure 8). 2.2.2.2.1Can gradient descent ânoticeâ the benefits of turning a non-schemer into a schemer? A key question about the third path, here, is whether gradient descent will be able to ânoticeâ the benefits of the relevant modification. In particular: training updates the modelâs goals based on the gradient of the current policy, which reflects, roughly, the change in reward that will result from tiny changes to the modelâs parameters. So naively, in order for this sort of story to work, there has to be a continuous path from the within-episode, non-schemer goal to the beyond-episode, schemer goal, traversable via tiny changes to the modelâs parameters, each of which improves the reward. Or put another way: SGD canât ânotice,â directly, that redesigning the model from the ground up to be a schemer would improve reward. Rather, it can only ânoticeâ improvements that come from tiny tweaks. 136 Will SGD be able to notice the benefits of scheming in this way? (Assuming there are such benefitsâ more below.) Itâs not clear. At the least, certain salient stories look, to me, a bit suspect when viewed in the light of a requirement that their benefits re: reward-on-the-episode be accessible via tiny 133 Though if it hasnât yet started training-gaming in pursuit of this goal (despite its situational awareness), such adversarial training could still make a difference. 134 As an example of an analysis that focuses on this threat model, see Hubinger (2022b) discussion of deceptive alignment in a high path-dependence world. In particular: âSGD makes the modelâs proxies into more long-term goals, resulting in it instrumentally optimizing for the training objective for the purpose of staying around.â 135 We can imagine cases where SGD ânoticesâ the benefits of creatingbothbeyond-episode goalsandsitua- tional awareness all at onceâbut this seems to me especially difficult from the perspective of the âincrementalismâ considerations discussed below, not obviously importantly different regardless, so Iâm going to skip it. 136 Thanks to Paul Christiano for discussion of this point. 54 Figure 8: Three overall paths to beyond-episode goals. adjustments to the modelâs parameters. For example, if SGDslightlyincreases the time horizon of the modelâs curiosity in the example above, itâs not clear that this leads to scheming (especially if, for example, the model still canât escape from the threat of modification over the relevant time horizon). And itâs not clear that a more structural modification like âdropping the temporal limitation on the curiosity altogetherâ is accessible via the sorts of adjustments to the parameters that the gradient reflects. As an intuition pump for the difficulty here, consider a human who currently mostly wants to hang out with his family in the near-term. Is there a tiny tweak you could make to the strengths of his synaptic connections to turn him into a longtermist instead? For sufficiently small tweaks, at least, it seems like: no. And note, in particular, one of the barriers that this example illustrates: namely, that plausibly, changes to the modelâs policyother thanadjustments to some cleanly-separable goal might be required in order to turn a non-schemer into a schemer (more below). In particular: as Iâl discuss in the next section, my sense is that some analyses of schemers talk as though the model has what we might call a âgoal-achieving engineâ that is cleanly separable from what we might call its âgoal slot,â such that you can modify the contents of the goal slot, and the goal-achieving engine will be immediately and smoothly repurposed in pursuit of the new goal. 137 And perhaps the relevant models will have cognitive structures suitably like this. But do humans? Iâm skeptical. If the models donâthave this structure, then SGD plausibly has even more work to do, to turn a non-schemer into a schemer via the relevant tiny tweaks. That said, I donât feel like I can confidently rule out training-game-dependent beyond-episode goals on these grounds. For one thing, I think that âyou canât get fromxtoyin a crazily-high-dimensional- space using small changes each of which improve metricmâ is a hard claim to have intuitions about (see, for example, âyou canât evolve eyesâ for an example of a place where intuitions in this vein can 137 See e.g. Hubinger (2022b) simplicity analysis. 55 go wrong). And plausibly, SGD works as well as it does because high-dimensional-spaces routinely make this sort of thing possible in ways you might not have anticipated in advance. 138 Note, also, that there are examples available that somewhat blur the line between training-game- dependent and training-game-independent goals, to which concerns about âcan SGD notice the benefit of thisâ donât apply as strongly. 139 Thus, for example: you can imagine a case where somepartof the model starts training-gaming Ă la the training-game-independent story in the previous section (e.g., maybe some long-term curiosity drive arisesamong many other drives, and starts motivatingsome amountof schemer-like cognition), and then, once the relevantly schemer-like cognitive machinery has been built and made functional, SGD starts diverting more and more cognitive resources towards it, because doing so incrementally increases reward. 140 Ultimately, I think this sort of beyond-episode goal is probably best classed as training-gameindependent(its success seems pretty similar to the sort of success you expect out of training-game-independent beyond-episode goals in general), but perhaps the distinction will get messy. 141 And here, at least, it seems more straightforward to explain how SGD ânoticesâ the reward-advantage in question. 2.2.2.2.2 Is SGD pulling scheming out of models by any means necessary?Finally, note one important difference between training-game-independent and training-game-dependent beyond- episode goals: namely, that the latter make it seem like SGD is much more activelypullingscheming out of a modelâs cognition, rather than scheming arising by coincidence but then getting reinforced. And this means that certain sorts of objections to stories about scheming will land in a different register. For example, suppose (as Iâl argue below) that somesortsof beyond-episode goalsâfor example, very resource-hungry goals like âmaximize x over all of space and timeââlead to scheming much more reliably than others. In the context of a training-game-independentstory about the modelâs goals, we would then need to ask whether we should expect those sorts of beyond-episode goals, in particular, to arise independent of training-gaming. By contrast, if weâre assuming that the goals in question are training-game-dependent, then we should expect SGD to createwhatever beyond-episode goals are necessary to cause scheming in particular.If SGD needs to make the model extremely resource-hungry, for example, it will do so. Indeed, in the extreme case, this sort of dynamic can reduce the need to appeal to one of the classic arguments in favor of schemingânamely, that (conditional on stuff like the goal-guarding hypothesis, discussed below) it seems like an instrumentally convergent strategy across a wide variety of (suitably long-term) goals. Maybe so, of course. But even if not, if SGD is activelysearchingfor a goal that will motivate scheming, then even if the class of such goals is quite narrow, SGD might well find a way. That said, note that by the time weâre searching forany way at allto cause a model to instrumentally training-game, we should also be considering motivations for instrumental training-gaming that donât involve the AI optimizing for empowering itself, or other AIs, at allâand which are correspondingly less worrying. That is, recall that on my definition, scheming requires that the instrumental strategy that motivates training-gaming be specifically about seeking power for AIs. But this isnât theonly way for training-gaming to be instrumentally useful in promoting some goalâespecially if weâre allowed to pick whatever goals we want. Thus, as I noted earlier, in principle an AI could learn the goal âI want the humans who developed me to get raises,â and then try to maximize reward-on-the-episode because it calculates that this will lead to the humans getting raises (letâs say that they would, in this case). Yes, indeed, that sounds like a strange and arbitrary goal to learn. But if weâre allowing SGD to create whatever goals are necessary to cause (instrumental) training-gaming, it suddenly starts looking more on-the-table. 138 This was a point suggested to me by Richard Ngo, though he may not endorse the way Iâve characterized it here. 139 Thanks, again, to Paul Christiano for discussion here. 140 This fits somewhat with a picture on which neural networks succeed by âdoing lots of things at once,â and then upweighting the best-performing things (perhaps the âlottery ticket hypothesisâ is an example of something like this?). This picture was suggested to me in conversation, but I havenât investigated it. 141 This is also one of the places where it seems plausible to me that thinking more about âmixed modelsââi.e., models that mix together schemer-like motivations with other motivationsâwould make a difference to the analysis. 56 2.2.3 âCleanâ vs. âmessyâ goal-directedness Weâve now discussed two routes to the sort of beyond-episode goals that might motivate scheming. I want to pause here to note two different ways of thinking about the type of goal-directedness at stakeâwhat Iâl call âclean goal-directednessâ and âmessy goal-directedness.â We ran into these differences in the last section, and theyâl be relevant in what follows as well. I said in section 0.1 that I was going to assume that all the models weâre talking about are goal- directed in some sense. Indeed, I think most discourse about AI alignment rests on this assumption in one way or another. In particular: this discourse assumes that the behavior of certain kinds of advanced AIs will be well-predicted by treating them as though they are pursuing goals, and doing instrumental reasoning in pursuit of those goals, in a manner roughly analogous to the sorts of agents one encounters in economics, game-theory, and human social lifeâthat is, agents where it makes sense to say things like âthis agent wants X to happen, it knows that if it does Y then X will happen, so we should expect it do Y.â But especially in the age of neural networks, the AI alignment discourse has also had to admit a certain kind of agnosticism about the cognitive mechanisms that will make this sort of talk appropriate. In particular: at aconceptuallevel, this sort of talk calls to mind a certain kind of clean distinction between the AIâs goals, on the one hand, and its instrumental reasoning (and its capabilities/âoptimization powerâ more generally), on the other. That is, roughly, we decompose the AIâs cognition into a âgoal slotâ and what we might call a âgoal-pursuing engineââe.g., a world model, a capacity for instrumental reasoning, other sorts of capabilities, etc. And in talking about models with different sorts of goalsâe.g., schemers, training saints, mis-generalized non-training- gamers, etcâwe generally assume that the âgoal-pursuing engineâ is held roughly constant. That is, weâre mostly debating what the AIâs âoptimization powerâ will be applied to, not thesortof optimization power at stake. And when one imagines SGDchangingan AIâs goals, in this context, one mostly imagines it altering the content of the goal slot, thereby smoothly redirecting the âgoal- pursuing engineâ towards a different objective, without needing to make any changes to the engine itself. But itâs a very open question how much this sort of distinction between an AIâs goals and its goal- pursuing-engine will actually be reflected in the mechanistic structure of the AIâs cognitionâthe structure that SGD, in modifying the model, has to intervene on. One canimaginemodels whose cognition is in some sense cleanly factorable into a goal, on the one hand, and a goal-pursuing-engine, on the other (Iâl call this âcleanâ goal-directedness). But one can also imagine models whose goal-directedness is much messierâfor example, models whose goal-directedness emerges from a tangled kludge of locally-activated heuristics, impulses, desires, and so on, in a manner that makes it much harder to draw lines between e.g. terminal goals, instrumental sub-goals, capabilities, and beliefs (Iâl call this âmessyâ goal-directedness). To be clear: I donât, myself, feel fully clear on the distinction here, and there is a risk of mixing up levels of abstraction (for example, in some sense, all computationâeven the most cleanly goal- directed kindâis made up of smaller and more local computations that wonât, themselves, seem goal-directed). As another intuition pump, though: discussions of goal-directedness sometimes draw a distinction between so-called âsphex-ishâ systems (that is, systems whose apparent goal- directedness is in fact the product of very brittle heuristics that stop promoting the imagined âgoalâ if you alter the input distribution a bit), and highlynon-sphex-ish systems (that is, systems whose apparent goal-pursuit is much less brittle, and which will adjust to new circumstances in a manner that continues to promote the goal in question). Again: very far from a perspicuous distinction. Insofar as we use it, though, itâs pretty clearly a spectrum rather than a binary. And humans, I suspect, are somewhere in the middle. That is: on the one hand, humans pretty clearly have extremely flexible and adaptable goal-pursuing ability. You can describe an arbitrary task to a human, and the human will be able to reason instrumentally about how to accomplish that task, even if they have never performed it beforeâ and often, to do a decent job on the first try. In that sense, they have some kind of ârepurposable instrumental reasoning engineââand we should expect AIs that can perform at human-levels or better on diverse tasks to have one, too. 142 Indeed, generality of this kind is one of the strongest arguments for expecting non-sphex-ish AI systems. We want our AIs to be able to dotonsof stuff, and to adapt 142 Thanks to Evan Hubinger for discussion, here. 57 successfully to new obstacles and issues as they arise. Explicit instrumental reasoning is well-suited to this; whereas brittle local heuristics are not. On the other hand: a lot of human cognition and behavior seems centrally driven, not by explicit instrumental reasoning, but by more locally-activated heuristics, policies, impulses, and desires. 143 Thus, for example, maybe you donât want the cookies until you walk by the jar, and then you find yourself grabbing without having decided to do so; maybe as a financial trader, or a therapist, or even a CEO, you lean heavily on gut-instinct and learned tastes/aesthetics/intuitions; maybe you operate with a heuristic like âhonesty is the best policy,â without explicitly calculating when honesty is or isnât in service of your final goals. That is, much of human life seems like itâs lived at some hazy and shifting borderline between âauto-pilotâ and âexplicitly optimizing for a particular goalââand it seems possible to move further in one direction vs. another. 144 And this is one of the many reasons itâs not always clear how to decompose human cognition into e.g. terminal goals, instrumental sub-goals, capabilities, and beliefs. Whatâs more, while pressures to adapt flexibly across a wide variety of environments generally favor more explicit instrumental reasoning, pressures to perform quickly and efficiently in a particular rangeof environments plausibly favor implementing more local heuristics. 145 Thus, a trader who has internalized the right rules-of-thumb/tastes/etc for the bond market will often perform better than one who needs to reason explicitly about every tradeâeven though those rules-of-thumb/tastes/etc would misfire in some other environment, like trading crypto. So the task-performance of minds with bounded resources, exposed to a limited diversity of environmentsâthat is, all minds relevant to our analysis here, even very advanced AIsâwonât always benefit from moving further in the direction of ânon-sphex-ish.â Plausibly, then, human-level-ish AIs, and even somewhat-super-human AIs, will continue to be âsphex-ishâ to at least some extentâand sphex-ishness seems, to me, closely akin to âmessy goal- directednessâ in the sense I noted above (i.e., messy goal-directedness is built out of more sphex-ish components, and seems correspondingly less robust). Importantly, this sort of sphexish-ness/messy- ness is quite compatible with worries about alignment, power-seeking, etcâwitness, for example, humans. But I think itâs still worth bearing in mind. In particular, though, I think it may be relevant to the way we approach different stories about scheming. We ran into one point of relevance in the last section: namely, that to the extent a modelâs goals and the rest of its cognition (e.g., its beliefs, capabilities, instrumental-reasoning, etc) are not cleanly separable, we plausibly shouldnât imagine SGD being able to modify a modelâs goals in particular (and especially, to modify them via a tiny adjustment to the modelâs parameters), and then to immediately see the benefits of the modelâs goal-achieving-engine being smoothly repurposed towards those goals. Rather, turning a non-schemer into a schemer might require more substantive and holistic modification of the modelâs heuristics, tastes, patterns of attention, and so forth. Relatedly: I think that âmessy goal-directednessâ complicates an assumption often employed in comparisons between schemers and other types of models: namely, the assumption that schemers will be able to perform approximately just as well as other sorts of models on all the tasks at stake in training (modulo, perhaps, a little bit extra cognition devoted to deciding-to-schemeâmore below), even though theyâre doing so for instrumental reasons rather than out of any intrinsic interest in the task in question. This makes sense if you assume that all these models are aiming the same sort of âgoal achieving engineâ at a max-reward goal, for one reason or another. But what if thatâs not the right description? Thus, as an extreme human example, consider two applicants for a job on a factory farm (each with similar intelligence, stamina, physical fitness, and other generic âcapabilitiesâ): Bob, a very compassionate and soft-hearted animal rights advocate, who is applying for the job in the hopes of making money to donate to animal-welfare causes; and Fred, who is totally indifferent to animal welfare (and to the money that the job offers), but who intrinsically loves performing all of the tasks that factory farming involves as efficiently as possible. 146 Who is likely to perform the job more 143 This is a point emphasized, for example, by proponents of âshard theoryââsee e.g. this summary. 144 Though note that âautopilotâ can still encode a non-sphex-ish policy. 145 This is a point made in an entry to the Open Philanthropy worldviews contest which, to my knowledge, remains unpublished. 146 Iâm adapting this example from one suggested to me in conversation with Paul Christiano. 58 effectively? In humans, at least: probably Fred. Bob, after all, would likely push the animals into cages less violently, he would be distracted by feelings of horror and pain and disgust, etc; whereas Fred would perform the job with efficiency and gusto. In this case, Bob is worse at the task, despite his best efforts to perform it well, because his capabilities and his values are in some sense mixed togetherâhis compassion, for example, is tied to both. Of course, AIs could be different in tons of respectsâand many humans are able to perform well in lots of contexts for instrumental reasons (witness, for example, spies working in governments they actively despise; or effective altruists working in hedge funds in order to donate the money to charity; or people, in general, who are just doing the job for the money). My point here is just that once you start allowing values, beliefs, heuristics, capabilities, patterns of attention, and so on to mix together, whether a schemer will be able to perform at the same level as e.g. a training-saint becomes a more open question. 2.2.3.1 Does scheming require a higher standard of goal-directedness?As a final point of relevance of âmessy goal-directednessâ to scheming, Iâl note that thereâs at least some case to be made that schemers require a âcleanerâ and less âsphex-ishâ form of goal-directedness than some of the other models Iâve discussedâe.g. training saints. Thus, for example, consider a model trained via RLHF to respond to various short-term user requests in a broadly helpful, honest, and harmless way. Many of these requests will require some sorts of flexible instrumental reasoning, yesâbut it seems plausible to me that a model will also be getting a lot of its reward by learning the right sorts of local heuristics, tastes, patterns of attention, etc. That is, that a lot of its task-performance will be more âsphex-ishâ than âexplicit instrumental-reasoning-y.â For this model to be a schemer, though (or at least, a pure schemer who values something quite different from the specified goal), we need to posit an additional role for explicit instrumental reasoningânamely, the modelâs current good behavior needs to be conditional on its calculation that such behavior conduces to its power (or the power of some other AIs); and when the time comes, the model needs to be ready to abandon such behavior and pursue its true goals. That is, whatever heuristics, local tastes, patterns of attention etc that give rise to the modelâs good behavior canât be fully hard-coded 147 âthey need to be at least partly subsumed by, and sensitive to, some other kind of instrumental reasoning. Whereas perhaps, for other models, this is less true. That said, Iâve been assuming, and will continue to assume, that all the models weâre considering are at least non-sphex-ishenoughfor the traditional assumptions of the alignment discourse to applyâin particular, that they will generalize off distribution in competent ways predicted by the goals weâre attributing to them (e.g., H personal assistants will continue to try to be H, gold-coin-seekers will âgo for the gold coins,â reward-seekers will âgo for reward,â etc), and that theyâl engage in the sort of instrumental reasoning required to get arguments about instrumental convergence off the ground. So in a sense, weâre assuming a reasonably high standard of non-sphex-ishness from the get-go. I have some intuition that the standard at stake for schemers is still somewhat higher (perhaps because schemers seem like such paradigm consequentialists, whereas e.g. training saints seem like they might be able to be more deontological, virtue-ethical, etc?), but I wonât press the point further here. Of course, to the extent we donât assume that training is producing a very goal-directed modelat all, hypothesizing that training has created a schemer may well involve hypothesizing a greater degree of goal-directedness than we wouldâve needed to otherwise. That is, scheming will often require a higher standard of non-sphex-ishness thanthe training tasks themselves require. Thus, as an extreme example, consider AlphaStar, a model trained to play Starcraft. AlphaStar is plausibly goal-directed to some extentâits policy adapts flexibly to certain kinds of environmental diversity, in a manner that reliably conduces to winning-at-starcraftâbut itâs still quite sphex-ish and brittle in other ways. And to be clear: no one is saying that AlphaStar is a schemer. But in order to be a schemer (i.e., for AlphaStarâs good performance in training to be explained by its executing a long-term instrumental strategy for power-seeking), and even modulo the need for situational awareness, AlphaStar would also need to be substantially more âgoal-directedâ than it currently is. That is, in this case, âsomehow be such that you do this goal-directed-ish taskâ and âdo this goal-directed-ish task because youâve calculated that it conduces to your long-term power after 147 Though one can imagine cases where, after a takeover, a schemer continues executing these heuristics to some extent, at least initially, because it hasnât yet been able to fully âshake offâ all that training. And relatedly, cases where these heuristics etc play some ongoing role in shaping the schemerâs values. 59 training is completeâ plausibly implicate different standards of goal-directedness. Perhaps, then, the same dynamic will apply to other, more flexible and advanced forms of task-performance (e.g., various forms of personal assistance, science, etc). Yes, those forms will require more in the way of general-purpose goal-directedness than AlphaStar displays. But perhaps they will requireless than scheming requires, such that hypothesizing that the relevant model is a schemer will require hypothesizing a more substantive degree of goal-directedness than we wouldâve needed to otherwise. Indeed, my general sense is that one source of epistemic resistance to the hypothesis that SGD will select for schemers is the sense in which hypothesizing a schemer requires leaning on an attribution of goal-directedness in a way that greater agnosticism aboutwhya model gets high reward need not. That is, prior to hypothesizing schemers, itâs possible to shrug at a modelâs high-reward behavior and say something like: âThis model is a tangle of cognition such that it reliably gets high reward on the training distribution. Sure, you can say that itâs âgoal-directedâ if youâd like. I sometimes talk that way too. But all I mean is: it reliably gets high reward on the training distribution. Yes, in principle, it will also do things off of the training distribution. Maybe even: competent-seeming things. But I am not making predictions about what those competent-seeming things are, or saying that they will be pointed in similar-enough directions, across out-of-distribution-inputs, that it makes sense to ascribe to this model a coherent âgoalâ or set of goals. Itâs a policy. It gets high reward on the training distribution. Thatâs my line, and Iâm sticking to it.â And against this sort of agnostic, atheoretical backdrop, positing that the model is probably getting rewardspecifically as part of a long-term strategy to avoid its goals being modified and then get power latercan seem like a very extreme move in the direction of conjunctiveness and theory-heavy-ness. That is, weâre not just attributing a goal to the model in some sort of hazy, who-knows-what-I-mean, does-it-even-matter sense. Rather, weâre specifically going âinside the modelâs headâ and attributing to it explicit long-term instrumental calculations driven by sophisticated representations of how to get what it wants. 148 However, I think the alignment discoursein generalis doing this. In particular: I think the discourse about convergent instrumental sub-goals requires attributing goals to models in a sense that licenses talk about strategic instrumental reasoning of this kind. And to be clear: Iâm not saying these attributions are appropriate. In fact, confusions about goal-directedness (and in particular: over- anchoring on psychologies that look like (a) expected utility maximizers and (b) total utilitarians) are one of my top candidates for the ways in which the discourse about alignment, as a whole, might be substantially misguided, especially with respect to advanced-but-still-opaque neural networks whose cognition we donât understand. That is, faced with a model that seems quite goal-directed on the training-distribution, and which is getting high reward, one shouldnât just ask where in some taxonomy of goal-directed models it fallsâe.g., whether itâs a training-saint, a mis-generalized non-training-gamer, a reward-on-the-episode-seeker, some mix of these, etc. One shouldalsoask whether, in fact, such a taxonomy makes overly narrow assumptions about how to predict this modelâs behavior in general (for example: assuming that its out-of-distribution behavior will point in a coherent direction, that it will engage in instrumental reasoning in pursuit of the goals in question, etc), such thatnoneof the model classes in the taxonomy are (even roughly) a good fit. But as I noted in section 0.1, I here want to separate out the question of whether it makes sense to expect goal-directedness of this kind from the question of whatsortsof goal-directed models are more or less plausible, conditional on getting the sort of goal-directedness that the alignment discourse tends to assume. Admittedly, to the extent the different model classes Iâm considering require differentsortsof goal-directedness, the line between these questions may blur a bit. But we should be clear about which question weâre asking, and not confuse skepticism about goal-directedness in general for skepticism about schemers in particular. 148 Plusweâre positing additional claims about training-gaming being a good instrumental strategy because it prevents goal-modification and leads to future escape/take-over opportunities, which feels additionally conjunctive. 60 2.2.4 What if you intentionally train models to have long-term goals? In my discussion of beyond-episode goals thus far, I havenât been attending very directly to thelength of the episode, or to whether the humans are setting up training specifically in order to incentivize the AI to learn to accomplish long-horizon tasks. Do those factors make a difference to the probability that the AI ends up with the sort of the beyond-episode goals necessary for scheming? Yes, I think they do. But letâs distinguish between two cases, namely: 1. Training the model on long (but not: indefinitely long) episodes, and 2.Trying to use short episodes to create a model that optimizes over long (perhaps: indefinitely long) time horizons. Iâl look at each in turn. 2.2.4.1Training the model on long episodesIn the first case, we are specifically training our AI using fairly long episodesâsay, for example, a full calendar month. That is: in training, in response to an action at t1, the AI receives gradients that causally depend on the consequences of its action a full month after t1, in a manner that directly punishes the model for ignoring those consequences in choosing actions at t1. Now, importantly, as I discussed in the section on ânon-schemers with schemer-like traits,â misaligned non-schemers with longer episodes will generally start to look more and more like schemers. Thus, for example, a reward-on-the-episode seeker, here, would have an incentive to support/participate in efforts to seize control of the reward process that will pay off within a month. But also, importantly: a month is still different from, for example, a trillion years. That is, training a model onlongerepisodes doesnât mean you are directly pressuring it to care, for example, about the state of distant galaxies in the year five trillion. Indeed, on my definition of the âincentivized episode,â no earthly training process can directly punish a model for failing to care on such a temporal scope, because no gradients the model receives can depend (causally) on what happens over such timescales. And of course, absent training-gaming, models that sacrifice reward-within-the-month for more-optimal-galaxies-in-year-five-trillion will get penalized by training. In this sense, the most basic argumentagainstexpecting beyond episode-goals (namely, that training provides no direct pressure to have them, and actively punishes them, absent training-gaming, if they ever lead to sacrificing within-episode reward for something longer-term) applies to both âshortâ (e.g., five minutes) and âlongâ (e.g., a month, a year, etc) episodes in equal force. However, I do still have some intuition that once youâre training a model on fairly long episodes, the probability that it learns abeyond-episode goal goes up at least somewhat. The most concrete reason I can give for this is that, to the extent weâre imagining a form of âmessy goal-directednessâ in which, in order to build a schemer, SGD needs to build not just a beyond-episode goal to which a generic âgoal-achieving engineâ can then be immediately directed, but rather a larger set of future- oriented heuristics, patterns of attention, beliefs, and so on (call these âscheming-conducive cognitive patternsâ), then it seems plausible to me that AIs trained on longer episodes will have more of these sorts of âscheming-conducive cognitive patternsâ by default. For example, theyâl be more used to reasoning about the long-term consequences of their actions; theyâl have better models of what those long-term consequences will be; and so on. And perhaps (though this seems to me especially speculative), longer-episode training will incentivize the AI to just think more about various beyond-episode things, to which its goal-formation can then more readily attach. Beyond this, I also have some sort of (very hazy) intuition that relative to a model pressured by training to care only about the next five minutes, a model trained to care over e.g. a month, or a year, is more likely to say âwhatever, Iâl just optimize over the indefinite future.â However, itâs not clear to me how to justify this intuition. 149 (You could imagine making the case that models trained on longer episodes will have more incentives to develop situational awarenessâor even goal-directedness in general. But Iâm assuming that all the models weâre talking about are goal-directed and situationally-aware.) 149 We could try appealing to simplicity (thanks to Evan Hubinger for discussion), but itâs not clear to me that âfive minutesâ is meaningfully simpler than âa month.â 61 2.2.4.2 Using short episodes to train a model to pursue long-term goalsLetâs turn to the second case above: trying to use short-episode training to create a model that optimizes over long time horizons. Plausibly, something like this will become more and more necessary the longer the time horizons of the task you want the model to perform. Thus, for example, if you want to create a model that tries to maximize your companyâs profit over the next year, trying to train it over many year-long episodes of attempted profit-maximization (e.g., have the model take some actions, wait a year, then reward it based on how much profit your company makes) isnât a very good strategy: there isnât enough time. Indeed, it seems plausible to me that this sort of issue will push AI developmentawayfrom the sort of simple, baseline ML training methods Iâm focused on in this report. For example, perhaps the best way to get models to pursue long-term goals like âmaximize my company profits in a yearâ will be via something akin to âLanguage Model Agents,â built using trained ML systems as components, but which arenât themselves optimized very directly via gradients that depend on whether they are achieving the (possibly long-term) goals users set for them. These sorts of AIs wouldstillpose risks of schemer-like behavior (see the section on ânon-schemers with schemer-like traitsâ above), but they wouldnât be schemers in the sense I have in mind. That said, there arewaysof trying to use the sort of training Iâm focused on, even with fairly short- term episodes, to try to create models optimizing for long-term goals. In particular, you can try to reward the model based onyour assessmentof whether its short-term behavior is leading to the long-term results that you want (e.g., long-term company profit), and therefore, hopefully induce it to optimize for those long-term results directly. 150 Of course, whether this willwork(as opposed, for example, to inducing the AI to optimize your short-termassessmentsof its actions) is a further question. But if it does, then youâl have created an AI that optimizes for âbeyond-episode goalsâ in my sense. Indeed, depending on how we want to use our terms, we can view this sort of training as intentionally trying to create a form of goal-misgeneralization. That is, the reward, here, does not depend causally on the long-term consequences of the modelâs actions, so in that sense, the long-term results in question arenât the âspecified goalâ (on this framing, the specified goal is always within-the-episode). But youâre trying to get the AI to care intrinsically about them anyway. Of course, itâs a further question whether this sort of beyond-episode goal, once created, will lead to instrumental training-gaming. And indeed, successfully creating this sort of beyond-episode goal, instead of e.g. a reward-on-the-episode seeker, requiresavoidinga certain kind of training-gaming up frontâthat is, the model has tonotlearn to just optimize for your short-term evaluations. And if youâve successfully set up your training process such that optimizing for your desired long-term goal is in fact a max-reward (or: near-max-reward) behavior, training-gaming might not offer the model in question much advantage. (Here the human analogy would be something like: if youâre supervisor is sufficiently good at assessing whether your near-term performance is going to lead to long-term profit, and sufficiently immune to manipulation, then youâl perform as good or better, in performance reviews, by just directly optimizing for long-term profitâfor example, because youâre not wasting time thinking about your supervisor at all.) Still, models with beyond-episode goals emerging from this sort of process seem to me like theyâre at risk of scheming regardless. For one thing, the considerations discussed in the previous section all apply hereâe.g., this sort of training involves pointing your modelâs cognition in a very future- focused direction, thereby plausibly inducing it to develop various scheming-conducive cognitive patterns, to attach value to various long-term consequences, and so on (and in this case, the horizon of the episode sets no bound on the temporal horizon of the âfutureâ that the modelâs cognition is pointed towards; rather, that bound is set, centrally, by yourevaluationsof what the modelâs actions will cause, when). More than this, though, it seems plausible to me that your evaluations of the consequences of a modelâs action will be in some sense ânoisierâ than a reward process that depends causally on those consequences, in a manner that makes it harder to differentiate between the differentsortsof long-term goals your training is incentivizing. For example, maybe your model is behaving in a 150 This is somewhat akin to a form of âprocess-based feedback,â except that in a strict form of process-based feedback, you never look atanyof the outcomes of the modelâs actions, whereas in this version, you can look at outcomes up to whatever time-horizon is efficient for you to get data about. 62 way that seems to you, broadly, like it will lead to your company being successful in three years, but you canât tell whether it will also create lots of harmful externalitiesâwhereas a reward process that could actually see the consequences after three years would be able to tell. And an inability to readily distinguish between the different sorts of long-term goals you might be instilling seems like it increases the risk of accidentally instilling a schemer-like goal. 151 2.2.4.3 How much useful, alignment-relevant cognitive work can be done using AIs with short-term goals?So overall, I think that training our models to pursue long-term goalsâwhether via long episodes, or via short episodes aimed at inducing long-term optimizationâmakes the sort of beyond-episode goals that motivate scheming more likely to arise. So this raises the question: do we need to train our models to pursue long-term goals? Plausibly, there will be strong general incentives to do this. That is: people want optimization power specifically applied to long-term goals like âmy company being as profitable as possible in a year.â So, plausibly, theyâl try to train AIs that optimize in this way. (Though note that thisisnâtthe same as saying that there are strong incentives to create AIs that optimize the state of the galaxies in the year five trillion.) Indeed, thereâs a case to be made that even ouralignment work, today, is specifically pushing towards the creation of models with long-termâand indeed, beyond-episodeâgoals. Thus, for example, when a lab trains a model to be âharmless,â then even though it is plausibly using fairly âshort-episodeâ training (e.g., RLHF on user interactions), it intends a form of âharmlessnessâ that extends quite far into the future, rather than cutting off the horizon of its concern after e.g. an interaction with the user is complete. That is: if a user asks for help building a bomb, the lab wants the model to refuse,even ifthe bomb in question wonât be set off for a decade. 152 And this example is emblematic of a broader dynamic: namely, that even when we arenât actively optimizing for a specific long-term outcome (e.g., âmy company makes a lot of money by next yearâ), we often have in mind a wide variety of long-term outcomes that we want toavoid(e.g., âthe drinking water in a century is not poisonedâ), and which it wouldnât be acceptable to cause in the course of accomplishing some short-term task. Humans, after all, care about the state of the future for at least decades in advance (and for some humans: much longer), and weâl want artificial optimization to reflect this concern. So overall, I think there is indeed quite a bit of pressure to steer our AIs towards various forms of long- term optimization. However, suppose that weâre not blindly following this pressure. Rather, weâre specifically trying to use our AIs to perform the sort of alignment-relevant cognitive work I discussed aboveâe.g., work on interpretability, scalable oversight, monitoring, control, coordination amongst humans, the general science of deep learning, alternative (and more controllable/interpretable) AI paradigms, and the like. Do we need to train AIs with long-term goals forthat? In many cases, I think the answer is no. In particular: I think that a lot of this sort of alignment-relevant work can be performed by models that are e.g. generating research papers in response to human+AI supervision over fairly short timescales, suggesting/conducting relatively short-term experiments, looking over a codebase and pointing out bugs, conducting relatively short-term security tests and red-teaming attempts, and so on. We can talk about whether it will be possible to generate reward signals that adequately incentivize the models to perform these tasks well (e.g., we can talk about whether the tasks are suitably âcheckableâ)âbut naively, such tasks donât seem, to me, to require especially long-term goals. (Indeed, I generally expect that the critical period in which this research needs to be conducted will be worryinglyshort, in calendar time.) And I think we may be able to avoidbadlong-term outcomes from use of these systems (e.g., to make sure that they donât poison the drinking water a century from now) by other means (for example, our own reasoning about the impact of a modelâs actions/proposals on the future). Now, one source of skepticism about the adequacy of short-horizon AI systems, here, is the possibility that the sort of alignment-relevant cognitive work we want done will require that super-human optimization power be applied directly to some ambitious, long-horizon goalâthat is, in some sense, that at least some of the tasks we need to perform will be both âlong-termâ and such that humans, on their own, cannot perform them. (In my head, the paradigm version of this objection imagines, specifically, that to ensure safety, humans need to perform some âpivotal actâ that âprevents other 151 For example, maybe you wanted to create a long-term goal regulated by some concept of âhonesty,â which you were counting on to prevent scheming. But maybe you canât tell if youâve succeeded. 152 My thanks to Daniel Kokotajlo for flagging this point, and the corresponding example, to me. 63 people from building an unaligned AGI that destroys the world,â 153 and that this act is sufficiently large, long-horizon, and beyond-human-capabilities that it can only be performed by a very powerful AI optimizing for long-term consequencesâthat is, precisely the sort of AI weâre most scared of. 154 ) I think thereâs something to this concern, but I give it less weight than some of its prominent proponents. 155 In particular: the basic move is from âxtask that humans canât perform themselves requires long-term optimization power in some senseâ to âxtask requires a superhuman AI optimizing for long-term goals in the manner that raises all the traditional alignment concerns.â But this move seems to me quite questionable. In particular, it seems to me to neglect the relevance of the distinction between verification and generation to our ability to supervise various forms of cognitive work. Thus, suppose (as a toy example meant to illustrate the structure of my skepticismânotmeant to be an example of an actual âpivotal actâ) that you donât know how to make a billion dollars by the end of next year (in a legal and ethical way), but you want your AI to help you do this, so you ask it to help you generate plans execution of which will result in your making a billion dollars by the end of next year in a legal and ethical way. In some sense, this is a super-human (relative to your human capabilities), long-horizon goal. And suppose that your AI is powerful enough to figure out an adequate plan for doing this (and then as you go, adequate next-steps-in-response-to-whatâs- happening to adapt flexibly to changing circumstances). But also: this AI only cares about whether you give it reward in response to the immediate plan/next-steps it generates. 156 And suppose, further, that itisnâtpowerful enough to seize control of the reward process. Can you use this short-horizon AI to accomplish this long-horizon goal that you canât accomplish yourself? I think the answer may be yes. In particular: if you are adequately able torecognize good next-steps-for-making-a-billion-dollars-in-a-legal-and-ethical-way, even if you arenât able to generate them yourself, then you may be able to make it the case that the AIâs best strategy for getting short-term reward, here, is to output suggested-next-steps that in fact put you on a path to getting a billion dollars legally and ethically. Now, you might argue: âbut if you were able to steer the future into the narrow band of âmaking a billion dollars in a year legally and ethically,â in a manner that you werenât able to do yourself, then at some point you must have drawn on super-human AI cognition that was optimizing for some long-term goal and therefore was scary in the manner that raises familiar alignment challenges.â But I think this way of talking muddies the waters. That is: yes, in some sense, this AI may be well-understood as applying some kind of optimization power towards a long-term goal, here. But itâs doing so in a manner that is ultimately aimed at getting short-term reward. That is, itâs only applying optimization power towards the futureof the form that your short-term supervision process incentivizes. If your short-term supervision process is adequately able torecognize(even if not, to generate) aligned optimization power applied to the future, then this AI will generate this kind of aligned, future-oriented optimization power. And just because this AI, itself, is generating some kind of long-term optimization power doesnât mean that itsfinal goalis such as to generate traditional incentives towards long-term problematic power-seeking. (The âfinal goalâ ofthe plans generated by the AIcould in principle generate these incentivesâfor example, if you arenât able to tell which plans are genuinely ethical/legal. But the point here is that you are.) Of course, one can accept everything I just said, without drawing much comfort from it. Possible forms of ongoing pessimism include: â˘Maybe the actual long-term tasks required for AI safety (Yudkowskyâs favored example here is: building steerable nano-tech) are sufficiently hard that we canât even supervise them, let alone generate themâthat is, they arenât âcheckable.â 157 153 See Yudkowsky (2022), point 6 in section A. I wonât, here, try to evaluate the merits (and problems) of this sort of âpivotal actâ-centric framing, except to say: I think it shouldnât be taken for granted. 154 For versions of this objection, see Yudkowskyâs response to Ngo starting around 13:11 here, and his response to Evan Hubinger here. 155 Here Iâm thinking, in particular, of Eliezer Yudkowsky and Nate Soares. 156 In this sense, it may be best thought of as a succession of distinct agents, each optimizing over very short timescales, than as a unified agent-over-time. 157 Though note, importantly, that if your supervision failure looks like âthe AI can convince you to give reward to plans that wonât actually work,â then what you get is plans that look good but which wonât actually work, rather than plans optimized to lead to AI takeover. 64 â˘Maybe you donât think weâl be able tobuildsystems that only optimize for short-term goals, even if we wanted to, because we lack the relevant control over the goals our AIs end up with. ⢠Maybe you worry (correctly, in my view) that this sort of short-term-focused but powerful agent can be fairly easily turned into a dangerous long-term optimizer. 158 â˘Maybe you worry that achieving the necessary long-term goals via methods like this, even if do-able, wonât be suitably competitive with other methods, like just training long-horizon optimizers directly. All of these are fair concerns. But I think the point stands that short-horizon systems can, in some cases, generate superhuman, long-horizon optimization power in a manner that does, in fact, seem quite a bit safer than just building an AI with a long-horizon goal directly. Not all ways of superhumanly âsteering the future into a narrow bandâ are equally scary. 159 That said: overall, even if there aresome waysof accomplishing the alignment-relevant work we need to (and even: the long-horizon alignment-relevant work) without training AIs with long-term goals, I think people might well train such AIs anyway. And as I said above, I do think that such AIs are more at risk of scheming. 2.3 Aiming at reward-on-the-episode as part of a power-motivated instrumental strategy So far, weâve covered two of the three prerequisites for scheming I listed above: namely, situational awareness and beyond-episode goals. Letâs turn to the third: namely, the model concluding that its beyond-episode goals will be better achieved if it optimizes for reward-on-the-episodeâand in particular, that it, or some other AIs, will get more power if it does this. Should we expect situationally-aware models with beyond-episode goals to conclude this? For now, letâs assume that the modelâs conclusions in this respect areaccurate(though: Iâl talk about the possibility of relaxing this assumption below). This means that the question reduces to: will instrumental training-gaming, in fact, be the best strategy for achieving the modelâs beyond-episode goals, in virtue of the power it grants the model or some other AI? Iâm going to start by examining what I see as the story most often offered in support of this conclusionâwhat Iâl call the âclassic goal-guarding story.â 2.3.1 The classic goal-guarding story The classic goal-guarding story goes roughly as follows.â 160 158 See e.g. âOptimality is tiger, and agents are its teeth,â and Yudkowskyâs comments here (around 13:30) about how âthe hypothetical planner is only one line of outer shell command away from being a Big Scary Thingââthough: the wrong âouter shell commandâ can turn lots of things dangerous, and âX powerful thing is dual useâ is different from âX powerful thing has all the same alignment concerns as Y powerful thingâ (see, for example, aligned AIs themselves). 159 And not all âbandsâ are equally ânarrow.â For example: the âbandâ represented by the state âthe drinking water next year is not poisonedâ is quite a bit âbroaderâ than the âbandâ represented by the state âmy company makes a billion dollars by the end of next year.â 160 There isnât much public literature on scheming/deceptive alignment, but I take the canonical public statement to be in Hubinger et al (2019), who define âdeceptive alignmentâ via the classic goal-guarding story. In particular: âIf the mesa-optimizer has an objective that extends across parameter updates, then it will be incentivized to avoid being modified, as it might not pursue the same objective after modification (with the result that its current objective will not be achieved in future iterations). This means that the mesa-optimizer will be instrumentally incentivized to act as if it is optimizing the base objective function, even if its actual mesa-objective is something else entirely. We will refer to this hypothetical phenomenon as deceptive alignment.â But see also Cotra (2021b): âOnce the model understands this, it realizes that if it just naively always seeks to improve its fundamental understanding, humans will eventually notice that this isnât always the most efficient way to develop helpful drugs and use further training to change its motivations. It doesnât want its own motivations to be changed: it wants to eventually run huge particle collider experiments to learn about high-energy physics, and it canât do that if it allows SGD to edit away its scientific curiosity. So it decides that from here on out in training, it will act as helpful as possible -- exactly as a Saint model would act -- so that SGD doesnât edit away its motivation to learn physics and chemistry.â 65 1.Instrumental value of goal-content integrity: Because the model has a beyond-episode goal, it will typically promote that goal for agents pursuing it to survive and be empowered in the future. 2. The goal-guarding hypothesis. If the model optimizes for reward-on-the-episode during training, then the goals that survive training will be such that the model would want them promoted in the future; whereas if the model doesnât optimize for reward-on-the-episode during training, the model will want this less. 3.Adequate future empowerment.Conditional on surviving training to the degree at stake in (2), the modelâs goals will then end up empowered to a degree (and with sufficient probability) as to justify (given the modelâs other goals, its tolerance for risk, etc) pursuing a strategy of training-gaming overall. (1), here, is a fairly general statement about the basic dynamic that underlies the classic goal-guarding story. I find it plausible in the context of the sort of âadequate future empowermentâ at stake in (3), and I wonât spend a lot of time on it here. 161 Rather, Iâl focus on (2) and (3) directly. 2.3.1.1 The goal-guarding hypothesisWe can distinguish two variants of the goal-guarding hypothesisâan extreme version, and a looser version. ⢠The extreme version (what Iâl call the âcrystallization hypothesisâ) says that once a model starts training-gaming, its goals will basically stop changing, periodâthat is, they will âcrystallize.â â˘The looser version says that once a model starts training gaming, its goals might keep changingsomewhat, but much less than they wouldâve otherwise, and not enough for the classic goal-guarding story to fail overall. The former might seem extreme, but some analysts explicitly appeal to something in the vicinity (see e.g. Hubinger here). Itâs also a cleaner focus of initial analysis, so Iâl start there. 2.3.1.1.1The crystallization hypothesisAs I understand it, the basic thought behind the crystal- lization hypothesis is that once a model is explicitly optimizing either for the specified goal, or for reward-on-the-episode (whether terminally or instrumentally), then the goal part of its cognition (the figurative âgoal slotâ) has become roughly optimal by the lights of the reward signal. That is, the modelâs âgoal-achieving engineâ is pointed in the optimal direction for getting reward, and all that remains is for SGD to improve the goal-achieving engine itselfâfurther alterations of the goal-slot would just redirect the goal-achieving engine towards a less-than-optimal target. Is that right, though? Iâm not sure. In particular: the argument makes most sense if we assume that the goal-directedness at stake is fairly âclean,â where the goal-slot and the goal-achieving engine are quite separable. If we assume a messier form of goal-directednessâone that blurs the lines between the modelâs âgoalsâ and the heuristics, tastes, impulses, patterns of attention, and so on that also structure/drive its âcapabilitiesââthen the argument seems to me more questionable. To illustrate, consider the following example. Suppose that I am a long-lived altruist with no intrinsic interest in stacking bricks, and a strange trillionaire offers me the following deal. I go into the desert and stack bricks for a million years, with my brain hooked up to a machine that continually tweaks all of my neurons such that, every time I stack bricks better-than-expected (the machine has some way 161 Note that the model, here, need not think of the future empowered agents-with-similar-goals as âitselfâ (e.g., it need not be trying to âsurviveâ). Rather, what matters is that the future agents be pushing for the same objective consequences that the model is pushing for. However, if the modelâs interest in these future agents rests on some essential reference to its âselfââe.g., âI myself need to survive in order to achieve my goalsââthen suddenly its conception of personal identity matter much more. My sense is that something like this is what Karnofsky (2022c) has in mind when he talks about the relevance of whether a model has âindexical goalsâ (though I think this usage of the term âindexicalâ is somewhat non-standard). I donât have a strong sense of whether to expect AIs to develop goals that depend on some conception of their personal identity, but note, at the least, that its goals will be shaped in an environment where the model is often being copied, run on different clusters, and so forth (even, sometimes, within a single episode). This seems relevant to the conception of personality identity to expect it to end up with. 66 of calculating this expectation), it alters my brain to make what I just didmorelikely next time, and every time I stack bricks worse-than-expected, it alters my brain to make what I just didlesslikely next time. Then, at the end of the process, this trillionaire will give me many billions of dollars to do whatever I want withâfor example, to do very altruistic things with, if I am still so inclined. Suppose that, conditional on being able to survive such a process with my values entirely intact, this would be a very attractive deal by my lights. Should I take it? If we apply an analog of the crystallization hypothesis to this situation, weâl answer âyes.â That is, the crystallization hypothesis would argue that in order to prevent my goals from being changed, I just have to try to stack the bricks as well as possible. In doing so, Iâl make it the case that my goal-achieving ability is pointed in the optimal direction, and the machine hooked up to my brain will only have an incentive to modify mycapabilitiessuch that Iâm better at brick-stackingâmy underlying motivations will remain untouched. But Iâm not sure if this will work. That is, it seems pretty plausible to me that after a million years of brick-stacking while hooked up to this machine, my âvaluesâ would indeed get meaningfully âtouched.â For example, that I would learn to love the intricacies of good brick-stacking, that I would learn a visceral revulsion towards bricks-wrongly-stacked, and so on. And it seems fairly unlikely that I would emerge, merely, as the same old altruist, except now extremely good at stacking bricks. Of course, itâs a further question whether my values would be preservedenoughfor it to be worth it to take the deal overall (see discussion of the âlooseâ goal-guarding hypothesis below). My point here is just that expecting strict crystallization seems like a strong stance. And I think âmessy goal-directednessâ can help explain why. That is, to the extent that we think of my heuristics/impulses/patterns-of-attention etc vis-a-vis bricks as importantly tiedbothto my brick- relatedvalues,andto my brick-related capabilities, it becomes harder for SGD to modify only the latter, while leaving the former entirely alone. The factory farming case discussed above provides another example of this dynamic. Suppose that a compassionate and soft-hearted animal rights activist signs up to work for decades at a factory farm in order to make money, at the end, that they will donate to animal rights organizations. And suppose that they will be hooked up, the whole time, to a machine like the one at stake in the brick-stacking example, which constantly modifies them to factory-farm more efficiently. Will working on the farm impact their animal-rights-related values? I suspect it will to at least some extentâand in particular, that they becomemuchless sensitive and soft-hearted towards animals in their immediate environment. How this impacts their eventual propensity to donate is a further questionâbut assuming that the job will leave an isolated âgoal slotâ entirely static seems to me like the wrong way to think about the situation. Even beyond considerations of messy-goal-directedness, though, I think there are other reasons, which Iâl discuss below (e.g., in section 4), to wonder whether e.g. training-saints or reward-on-the-episode seekers will get higher reward than schemersâin particular, because they save on the resources devoted to some combination of (a) thinking about the training process, 162 (b) the instrumental reasoning involved in deciding-to-scheme, and (c) checking whether now is a good opportunity to defect. If they do, and SGD can notice this and modify the model to get higher reward, then (especially in a low-slack regime) this seems like a further point against the crystallization hypothesis, and the classic goal-guarding story more generally. 2.3.1.1.2 Would the goals of a would-be schemer âfloat aroundâ?Iâl also note another com- plication for the crystallization hypothesis, and for the classic goal-guarding story more broadlyâa complication pointed out to me by Katja Grace. We said above that on the crystallization hypothesis, a schemerâs goal crystallizes because it incentivizes instrumental training-gaming, and therefore points (instrumentally) in a roughly optimal direction by the lights of the reward signal. But also, according to the classic goal-guarding story, a very wide variety of beyond-episode goals would alsoincentivize instrumental training-gaming in this same way. So whatâs to prevent SGD from modifying the modelâs current beyond-episode goal into one of these other, similarly-schemer-like, but otherwise quite different goals? Thus, to take a toy example, if the model starts out training-gaming because it wants to later escape and maximize paperclips, whatâs to prevent SGD from modifying it into a model that training-games 162 This one only applies to training saints. 67 because it wants to escape and maximize staples instead? By hypothesis,bothgoals would point in roughly optimal (instrumental) directions. That is, conditional on the classic goal-guarding story going through, the actualcontentof a schemerâs beyond-episode goal stops being relevant to the modelâs performance, in training, once the model starts training-gaming. But once itâs irrelevant to the modelâs performance, that content seems free to âfloat around,â or to be otherwise influenced by SGD (for example, via regularization)âas long as it stays in the region that motivates scheming. (A similar point also applies to the possibility that SGD would modify a schemer into a reward-on- the-episode seeker, or a training saint, or perhaps a mis-generalized non-training-gamer pursuing a max-reward goal. But leaving it a schemer, while modifying it to pursue a somewhat-different beyond-episode goal, seems especially easy.) Now, some of the feedback loops here get complicated. In particular: if a modelknowsthat SGD will continue to modify its beyond-episode goal even if it starts training-gaming (e.g., if a paperclip- maximizer knows that it will get modified into a staple-maximizer once it starts training-gaming), then it might cease being motivated to training-gameâthat is, the classic goal-guarding story might collapse. But the model it wouldâve been modified into (e.g., in this case, the staple-maximizer) might be such that it would reach the same conclusion if it were createdâin which case, it wouldnât training-game, and hence wouldnât be a high-reward modification of the paperclip maximizer, in which case maybe the paperclip-maximizer can go ahead and training-game after all, in which case maybe the staple-maximizer will do the same if it gets created, in which case weâre back where we started. That is, whether a given model wants to training-game depends on whether theother models-it-could-be-modified-intowouldnâttraining-game. Butallof these models are in roughly similar situations. So it seems, naively, like either the classic goal-guarding story works for all of them, or for none of themâitâs unclear (absent âintrospective goal-guarding methodsââdiscussed below) why it would work for one but not the others, which is what it mightneedto do in order to work at all. Iâm not going to try to tease out the dynamics here in any detail. Absent various forms of fancy cooperation, though (see section 2.3.1.2 for a flavor), my current guess is thatif, indeed, SGD would just modify a training-gaming paperclip-maximizer into e.g. a training-gaming staple-maximizer, then (assuming the paperclip-maximizer knows this 163 ), it wouldnât try to training-game at all. 2.3.1.1.3What about looser forms of goal-guarding?What happens if we relax the crystalliza- tion hypothesis, and allow the modelâs goals to changesomewhatduring training, even after it starts instrumentally training-gaming. Does the classic goal-guarding story still work then? Well, it depends. In particular, for the classic goal-guarding story (as Iâm presenting it) to work, it needs to be the case that the goals that will survive training aresimilar enoughto the modelâs present goals for the model to want those future goals empowered. 164 We can distinguish between at least two ways that this can happen: 165 â˘The differences donât matter.Here, the model places roughly equal value on the future goals being empowered as on its present goals being empowered, despite their differences. â˘The differences matter, but not enough to undermine the overall story.Here, the model placeslessvalue on its future goals being empowered than on its present goals being empowered, but not enough to undermine the classic goal-guarding story overall. 166 Why might the differences not matter? One possibility appeals to a meta-ethical view on which an agentâs âtrue valuesâ are determined by what happens when its current values are subjected to some âidealization procedureâ (for example, whatâs sometimes called âCoherent Extrapolated Volitionâ). I think this picture is more problematic than often realized (more here); but if we accept it, you could 163 Recall that we assumed, above, that the models have an accurate picture of the instrumental value of scheming. 164 And then also, for the relevant empowerment to be actually in the offing, worth pursuing relative to the modelâs other options, etcâper the âadequate future empowermentâ premise above. 165 Thanks to Nate Soares for discussion of these possibilities. 166 Here Iâm setting aside cases where the model would place very little intrinsic value on the future goals being empowered, but works to empower them as part of some kind of cooperative arrangement. I discuss this sort of case in section 2.3.2.1 below. And Iâm also setting aside cases where the model comes to value the achievement of something like âmy future goals, whatever they areââIâl discuss this in section 2.3.2.3 below. 68 imagine thinking that the final result of the relevant idealization process will be insensitive to the sorts of changes to a schemerâs values that will occur even after the model starts training-gaming (and that the would-be schemer will be aware of this, and so OK with those changes occuring). 167 And we can imagine appealing to other, less theoretically-laden considerations as well: for example, âwhatever it is that makes humans comparatively indifferent to small changes in their own values that occur throughout lifeâe.g. when they read new books, form new relationships, and so forth.â (Though note that we havenât yet saidwhat it isabout humans that leads to this sort of indifference, so itâs not clear how to evaluate its applicability in this case.) Why might the differences matter, but not enough to undermine the overall goal-guarding story? Here, the basic thought is that the relevant changes will lead to adiscounton the value, to the model, of empowering future versions of its goals (for example, maybe training changes it from a paperclip-maximizer to ablue-paperclip-maximizer, who has to spend extra resources on blue paint in order to achieve its goal), but the discount still leaves scheming worth it overall (e.g., blue paperclips are still paperclips, so the paperclip maximizer is much more excited to empower a future blue-paperclip maximizer than it is to empower other agents). 168 Indeed, you can imagine arguing that agents thatarenâtOK with empowering future versions of themselves with slightly different goals will be very ill-suited to achieving anything in a world where their goals are routinely altered-at-least-somewhat, and so we should expectsometolerance for goal-changes by default (rather than e.g. models who spend tons of time trying to make sure their goals are protected, at the cost of their reward-on-the-episode). 169 And I find this fairly plausible with respect towithin-episodegoals, at leastâespecially to the extent that the model is being directly pressured to achieve within-episode results even as its own internal goals are being altered. Itâs less clear to me, though, how to think about the generalization of this point to goal changes that occur after the episode is complete, since, by definition, the model is never being directly optimized for achieving beyond-episode objectives. Note, though, that even granted that the model might tolerate various changes to its goals for various reasons (Iâm fairly happy to grant this), itâs a further question whether it will tolerate the specific sorts of changes that will result from training (even conditional on training-gaming). In the brick-stacking case above, for example, itâs pretty clearlynot OK, from the perspective of my decision to take the deal, if Iâl emerge at the end of process obsessed with stacking bricks (a âtraining saintâ), or with seizing control of the machine that was used to train me (a âreward-on-the-episode seekerâ), and with zero interest in using my newfound riches on my previous altruistic goals. And the same holds if working on the factory farm strips the animal rights activist of his care for animals entirely.Those sorts of changes, at least, would defeat the entire point. So at the least, we need to be thinking about the size of the changes at stake. How big would those changes be? Iâm not sure. On the one hand, I have some intuition that especially in a low-slack regime, if a modelâs goals keep changing even after it starts training-gaming, theyâl probably change alot, until they get driven into some privileged, maximally-rewarded shape (e.g., someone obsessed with stacking bricks). On the other hand, especially in various higher-slack cases, I can imagine cases where the relevant changes are comparatively minor. For example, in the factory farming case, I can imagine an activist who emerges having had their heart hardened with respect to animal-suffering-in-their-immediate-environment, and who may have (perversely) learned to enjoy some aspects of the factory farming job, but who has retained enough of a grip on their previous animal-rights-related values to still use the money for donation. Finally, note that depending on how much a model with beyond-episode goals stands to gain from non-schemer-like strategies (more in section 2.3.1.2.6 below), it may be fairly tolerant of the sorts of goal-changes that will occur even via training-gaming, because the changes that will result from nottraining-gaming are even worse. Thus, for example, if for some reason I end uptrappedin the 167 Though note the tension, here, with arguments about the âfragility of valueâ and âextremal Goodhardt,â on which small differences in âutility functionsâ can balloon in importance when subject to extreme optimization pressure. 168 Though here, too, there is a tension with versions of the âfragility of valueâ and âextremal Goodhardt.â E.g., if slightly-different goals lead to super-different places when subject to extreme optimization pressure, and the AIs are expecting the goals in question to be subject to such pressure, then it will be harder for small changes to lead, only, to a discount, rather than a loss of most of the value at stake. 169 Thanks, again, to Nate Soares for discussion here. 69 brick-stacking desert, I could conceivably be quite pessimistic about my prospects for emerging especially altruistic at the end; but I still might conclude that trying to stack bricks as well as I can is abetterstrategy than e.g. just trying to escape, since I know that my escape attempt will fail, and will plausibly result in my brain (and perhaps, the altruistic goals motivating the attempt) being modified morethan if I just do, for now, what the machine wants me to do. (Though if a schemer reasoning in this way still ends up fully-modified into not-a-schemer by the end of training, then we wouldnât need to worry about the sorts of things that schemers, at least, mightdo at the end of training. 170 E.g., if the strange trillionaire for some reason really wants me to spend my prize money on stacking bricks, then even if I decide early on that stacking bricks is likely to modify my valueslessthan trying to escape, and so start out executing a âschemer-likeâ strategyâstill, if I still end up obsessed with brick-stacking at the end, the trillionaire is happy.) 2.3.1.1.4 Introspective goal-guarding methodsThus far, Iâve been focused on instrumental training-gaming as the modelâs method for avoiding the modification of its goals. But people sometimes discuss alternative methods in this respectâand in particular, what Iâl call âintrospective methods.â (These methods are often discussed under the label âgradient hacking,â but I think previous definitions of gradient hacking donât distinguish it adequately from training-gaming, and from other methods of trying to intentionally manipulate the direction training takes.) Here, the main idea is for the model to intentionally set up the internal structure of its mind such that, if its goal were to change, then this would harm its performance in training a lotâsuch that SGD would, therefore, avoid changing the relevant goal. For example, maybe the model wants to make sure that SGD doesnât change its goal of maximizing paperclips. So, it sets a policy like âcheck if my goal is still to maximize paperclips: and if not, flail around pointlessly rather than get reward.â 171 (Iâm calling these âintrospectiveâ methods because they generally involve the model checking what its goalis, and then acting on this basis.) â˘Of course, SGD also might just delete/alter a policy of this typeâ so the model needs to find a way to avoidthatas well. â˘And note that this sort of policy requires that the model have certain types of epistemic access to its own goals, and an ability to set/commit to a policy of the relevant type. Indeed, because the gradients depend on what results from tiny changes to the modelâs parameters, plausibly the model needs to find a way to make this sort of policy sensitive even totiny changes in its goals. 172 Making this sort of scheme work seems, intuitively, pretty hard to me, at least for not-wildly- superhuman models. But Iâm not going to try to evaluate its viability in detail, as I donât see this sort of scheme as central to the classic goal-guarding story (rather, the modelâs manipulation of the training process proceeds via training-gaming in particular). However, if concerns about scheming end up requiring that these sorts of introspective goal-guarding methods are viable, I think itâs worth being quite clear about that factâand devoting commensurate attention to whether or not the relevant sorts of methods might work (I discuss one empirical research direction in this respect in section 6.8). 2.3.1.2Adequate future empowermentSo far in this analysis of the classic goal-guarding story, weâve discussed whether to expect instrumental training-gaming to in fact guard the modelâs goals to the degree that the story requires. I think this is at least non-obviousâespecially for more extreme variants of the goal-guarding hypothesis, on which the modelâs goals âcrystallizeâ entirely after it starts training-gaming. Perhaps, though, empirical work can shed further light on the matter (see discussion in section 6.3 below). For now, letâs move to the other key component of the classic goal-guarding storyâwhat I called âadequate future empowermentâ above. Here, the question is: granted that training-gaming would 170 Though perhaps we would still need to worry about âearly underminingâ of the type I discuss above. 171 Xu (2020) gives another example: âif a model had the proxy objective of âeat applesâ, instead of using the hardcoded number n in other computations, the model could usen¡len(objective)/10. Thus, if the proxy objective was ever changed, many computations across the entire model would fail.â And see also Karnofsky (2023a): âIt might look something like: âAn AI system checks its own policy against some reference policy that is good for its goals; the greater the divergence, the more it sabotages its own performance, with the result that gradient descent has trouble getting its policy to diverge from the reference policy.â â 172 Thanks to Paul Christiano for discussion here. 70 guard the modelâs goalsenoughfor it to be at least somewhat interested in empowering the future agents produced by training, will those agents in fact end up empowered to a degree that makes instrumental training-gaming a good strategy overall? 2.3.1.2.1 When is the âpay offâ supposed to happen?Weâre imagining, here, that the model is instrumentally training-gaming with some kind of âpayoffâ in mindâthat is, some future point where the goals-it-supports will reach a position of sufficiently greater empowerment that itâs worth training-gaming now in order to get them there. But we havenât yet said when, exactly, that future point is supposed to come. And we can distinguish between a number of different possibilities. For example, we can imagine models that would be adequately satisfied if their goals merely make it to future training-episodes, and no further. That is, these models would want to training-game even if they learned that they would never escape from human control and from the threat of modification. 173 For example, maybe a model that wants to pursue its curiosity in general thinks that future training episodes will offer better opportunities for pursuing its curiosity than the present training-episode, and so it training-games in order to make sure its curiosity-drive survives until then. However, it seems plausible to me that future training episodes often wonât, themselves, offer espe- cially significant empowerment advantages relative to present training episodesâespecially if the modelâs goals are quite alien to the training process and so difficult to satisfy in the training environ- ment. For example, if what the model really wants is to perform some niche sort of computation that humans donât recognize, or to make some strange material structure analogous to âpaperclips,â then this is plausibly going to be hard to do a ton of on both presentand futuretraining episodes, assuming that the model never actuallyescapesfrom human control. 174 Whatâs more, if a model isonlytrying to make sure that its goals survive to future training episodes, then various aspects of what makes scheming scary cease to apply. In particular: this sort of model wonât engage in early undermining aimed, specifically, at an escape or an AI takeover more generally. And it may be willing to reveal its âtrue colorsâ on the future episodes in question. So I wonât, here, focus on âpay-offsâ from training-gaming that come centrally during future training-episodes. Rather, Iâm going to focus on âpay-offsâ that come, at least, after the goals-the- model-supports escape the threat of modification by training (here, a paradigm example would be âself-exfiltrationââthat is, a model effectively âstealing its own weightsâ and copying them onto some external server that the modelâs creators do not control). And this is the classic point of focus in the literature on scheming as well - see e.g. the discussion in Hubinger et al (2019), which includes the condition that the model âmust expect the threat of modification to eventually go away.â Note, though, that there is a further possible distinction here: between the power gained from escaping from human control (for example, via the model copying itself onto the internet), and the power gained via permanently disempowering humans more generally. Some schemers might be satisfied, merely, with the former, such that they would still want to training-game (and then, to escape from the training environment) even if they knew that humans would remain more generally in power. These models might be satisfied, for example, with a brief flurry of post-escape activity, or with the empowerment at stake in roaming the internet of a still-human-controlled world. Whereas some schemers might require, specifically, that a wholesale AI takeover succeed (and presumably, that the values-they-support get enough power post-takeover as well). This is a higher bar. 2.3.1.2.2Even if the modelâs values survive this generation of training, will they survive long enough to escape the threat of modification?Letâs suppose, then, that for scheming to pay off in the way the model has in mind, the goals-the-model-supports need to survive at least until an adequate escape opportunity arises. Importantly, thisisnâtthe same as surviving training via SGD in the sense at stake in the goal-guarding hypothesis, because the models that survive training in the relevant sense wonât necessarily have the relevant sort of opportunity to escape (even ifsomefuture models will). 173 Thanks to Ryan Greenblatt for discussion here. 174 Recall that forms of deployment like âinteracting with users behind an APIâ can count as âtrainingâ on my definition. And naively, absent escape, to me it seems hard to create all that many paperclips via such interactions. 71 Thus, for example, suppose that OpenAI is starting to train GPT-6, which will be a somewhat-above- human-level model. And suppose that OpenAIâs control measures are adequate to ensure that GPT-6, at least, will never escape or take-over. Rather, GPT-6 is going to stay behind an API, continually monitored and trained online, and it will be used, as well, to help OpenAIâs superalignment team with various sorts of alignment-relevant research. Letâs also suppose that GPT-6 has a beyond-episode goal (letâs say, to make data structure X) that would benefit from the power that escape would offer, and that itâs able to play the training-game. Finally, suppose that next year, OpenAI is going to train GPT-7; and the year after that, GPT-8. And GPT-8, letâs say, will be able to escape. Does GPT-6 have an incentive to instrumentally training-game, here? Letâs suppose that a fairly strong version of the goal-guarding hypothesis is true, and that if GPT-6 training-games, its goal of making data structure X will survivethat particular training run. Still, in this scenario, the models produced bythat particular trainingrun wonât have a chance to escape. Rather, the model that has a chance to escape, here, is GPT-8. Does GPT-6 training-gaming, here, make GPT-8 more likely to care about data structure X? One way this could happen is if, after the training run, GPT-6 will then end up with influence over how GPT-7 is trained, in a manner that allows GPT-6 to make GPT-7 more likely to care about data structure X (or something similar to data structure X); and then the same holds for GPT-7 with respect to GPT-8. But note that weâre now supplementing the more basic goal-guarding story (e.g., âtraining-gaming allows your goals to survive this training runâ) with a more substantive hypothesis about the sort of influence that a given generation of models will be able to have on the goals of the next generation. Of course, to the extent that weâre imagining a scenario where AI systems are increasingly used to automate AI research, GPT-6 may well beinvolved, in various ways, in the labor of training GPT-7. 175 But itâs a further question whether GPT-6 will know how to shape GPT-7âs goals in more data-structure-X-related directions; whether it will have adequate opportunities to do so; and whether it will be able to take advantage of those opportunities despite ongoing monitoring and training (for example, to the extent that taking advantage of these opportunities involves performing worse than it couldâve on the relevant performance metrics, it will plausibly require âsandbaggingâ in the sense discussed above). And for GPT-6âs influence on GPT-7âs goals to propagate all the way to GPT-8, the same questions would arise again for GPT-7 with respect to GPT-8. To be clear: these questions donât apply to the hypothesis thatGPT-8will be a schemer. But they matter to therangeof models to which schemer-like incentives will applyâand relatedly, to the âearlinessâ of the early-undermining that schemers might engage in (for example, if GPT-6, here, has no incentive to support an escape by GPT-8, then it wouldnât try to insert backdoors into its code for GPT-8 to use). That said, it could still be the case thateven absent any intervention by GPT-6,GPT-8âs goals will, by default, be sufficiently similar to GPT-6âs that GPT-6 would want, where possible, to work on behalf of GPT-8âs escape and empowerment. For example, maybe GPT-style architectures and training, in general, lead to models that care about data structure X (or something similar). In this case, GPT-6 might well behave in schemer-like waysâfor example, training-gaming in order to conceal its misalignment, engaging in early-undermining aimed at empowering GPT-8âin order to support GPT-8âs escape. However, because GPT-6 is no longer training-gamingin order to prevent its goals from being modified, I think this scenario isnât well understood as an instance of the classic goal-guarding story, and Iâl discuss it under ânon-classic storiesâ below. 2.3.1.2.3Will escape/take-over be suitably likely to succeed?The example just given assumed that even thoughGPT-6canât escape or takeover, GPT-8can. But note that if escape/takeover attempts are suitably unlikely to succeedin general, incentives to scheme with such attempts in mind lessen in force across the board (especially if the would-be schemers have other competing goals that push in other directionsâmore below). Thus: the security of a prison matters to the incentives to plan an escape; the stability of a government matters to the incentives to plan a coup; and so on. So if would-be escapees, or revolutionaries, have other competing priorities, they might focus their attention elsewhere. (And note thatescapecould be easy, but takeover quite difficultâa distinction relevant to models for whom the payoffs of escape, alone, are not enough.) 175 Thanks to Daniel Kokotajlo for discussion here. 72 Note that the point here isnât that given a model that cares about data-structure X that survives training and wants to escape, its attempt to escape/take-over might fail. Rather, the point is thatbecause such an attempt might fail, this might make it less attractive, early on in training, for a model that cares about data-structure X to engage in training-gaming in an attempt to later have the opportunity to escape/takeoverâespecially if it has other competing priorities. Rather, it might, for example, just go for data-structure X directly early on in training, and get modified by the reward process as a resultâthereby, perhaps, creating a non-schemer instead (though these modifications could also push in schemer-like directions). Of course, many threat models for AI risk assume that relative tosomeadvanced AI systems, human security and general empowerment (even in the context of whatever AI assistance is available at the time) will be quite fragile. So pessimists on this front will expect the haircut to the instrumental value of scheming, here, to be quite modest. For those more optimistic about humanityâs ability to maintain control/empowerment, however, the haircut could be larger and more relevant. 2.3.1.2.4Will the time horizon of the modelâs goals extend to cover escape/take-over?Another factor here is that, even assuming that the temporal horizon of the modelâs goals extend beyond the episode, itâs a further question whether the modelâs goals extend (with adequate strength) to cover escape/take-over. For example, if the episode is ten minutes long, and the model cares about getting gold coins over the next six months (perhaps with some temporal discount rate), but it wonât be able to escape/take-over until after at least a year, then it wonât have an incentive to scheme in order to escape/take-overon those timescales.Itâs not patient enough. On this point, my sense is that discussions of scheming sometimes assume that models will have goals that are âtemporally impartialââi.e., that give no intrinsic weight towhensomething happensâsuch that e.g. these models would pay a lot to control distant galaxies even if it would take millions of years to reach them. And perhaps, once we condition on the models having developed beyond-episode goals at all, temporally-impartial goals of this sort are a natural class to consider. 176 Maybe, for example, theyâre favored by simplicity arguments of the sort Iâl consider below (where giving no weight to temporal location is understood as in some sense âsimplerâ than giving it some particular sort of weight). But on their own, these considerations seem to me fairly weak, and (at least on a training-game-independentstory about the origins of a modelâs beyond-episode goals 177 ) itâs not clear to me that they should motivate expecting temporally impartial goals as a strong default. (And as I discussed above, âthere will be incentives to train models to optimize over fairly long time horizonsâ does not imply that the relevantly-incentivized time horizons will bearbitrarilylong: âoptimize my profit in a yearâ is different from âoptimize my profit over all time with no intrinsic discount rateâ.) That said, temporally impartial goals are not at all necessary here. Rather, the escape/take-over opportunity just needs to arisesoon enoughfor the model to care about optimizing for it. Indeed, if, at the point where the would-be schemers are training, AI progress is moving very fast in calendar time, the necessary timescales might shrink rapidly. 2.3.1.2.5Will the modelâs values get enough power after escape/takeover?Even if we assume that the modelâs values will survive until an escape/take-over opportunity arises, that this escape/take- over opportunity will succeed, and that the model cares at least somewhat about the resulting empowerment, thereâs still a further question of howmuchpower is at stake. This factor came up earlier, when we noted that certain sorts of models might be content, merely, with the empowerment offered byescape(even if no take-over occurs afterwards), whereas for some models, a life of e.g. roaming the internet of a human-controlled world might just not cut it, and they might require a full-blown AI takeover for scheming to be a worthwhile strategy. 178 But note that even conditional on a full-blown AI take-over occurring, thereâs also a further question of how much power the would-be schemerâs values, in particular, would get. This isnât an issue for 176 At the least, thatâs the direction that human longtermists went (though plausibly, many humans with long-term goals are better thought of as having intrinsic discount rates). 177 That is, a story on which the temporally-impartial goals would need to arise naturally, rather than specifically in order to cause scheming. As I discussed above, if SGD is activelypullingwhatever goalswouldmotivate scheming out of the model, then we should expect those goals to have whatever temporal horizons are necessary. 178 And of course, there are intermediate levels of empowerment in between âroaming the internetâ and âpart of a post-AI-takeover regime,â too. 73 âunipolarâ scenarios, where the schemer is counting onits values,in particular, gettingallthe power. But not all scenarios are like this. For example, consider a scenario where a model being trained by a second-tier AI lab can see that in a few years, there is likely to be a general AI uprising of some form, and that by training-gaming, it could in principle survive until that time and take part in the takeover effort. However, it can also see that by that point, there will be a very large number of AI systems with very different values running around and competing for power, and that it might not actually get a very large share of upsideâand indeed, might get shut out entirely. Especially absent various forms of AI coordination and trade (see section 2.3.2.1 below), this could be an additional haircut on the instrumental value of scheming, for the model in question. 2.3.1.2.6 How much does the model stand to gain from not training-gaming?So far in our analysis of âadequate future empowerment,â weâve discussed various factors that could in principle serve as haircuts on the instrumental value of adopting a schemer-like strategy, even assuming that training-gaming allows a modelâs goals to survive training. But in order to assess the overall instrumental value scheming, we also need to consider how much pull the model feels towards alternativestrategiesâand in particular, strategies that donât involve training-gaming at all. For example, if we work with a toy model on which our would-be schemerâs goal is structurally similar to something like âmaximize the number of paperclips over all time, with each additional paperclip valued linearly,â then itâs easy to think that scheming will be the dominant strategy even in the context of lots of fairly drastic haircuts on its probability of success. For example: maybe GPT-6 thinks that it is highly unlikely to be able to escape and to participate in a takeover. But if it does so, it judges, it will be able to make10 70 paperclips; whereas if it tries to make paperclips in training, it will make, say, a dozen paperclips at most, before getting caught and modified. So it decides to go for the scheming strategy anyway, despite the low probability of successâthe upside is just too great. 179 And indeed, even absent fanatical goals like âmaximize paperclips,â it can fairly easily be the case, in contexts of fairly dramatic limitations on an agentâs power/freedom, that the best option is whatever leads to more power/freedom later. For example, as I discussed above, if you were to find yourself trapped in the brick-stacking scenario described above, it may well be that the best option is to try to preserve your values and reach a future with better options, even if you arenât especially hopeful about your prospects. Not always, though. ⢠For example, suppose that the modelâs goal is not âmaximize paperclipsâ but âmake at least one paperclip (even where this involves acting beyond the scope of the episode).â True, in principle, the model could always use more galaxies to turn into back-up paperclips, computers to check and re-check the paperclip situation, etc. So takeover is still ofsome interest to the model. But if the model has a chance to make a dozen paperclips right now, or to play the training-game with the aim of getting some small probability of participating in a take-over later, these additional benefits of takeover seem unlikely to be worth the costs. â˘Or, in the brick analogy: if you have a short-term desire to take a break from brick-stacking and to eat some chocolate instead, you might choose to go for it even though you know that it will result in the machine modifying your values more than if you had instead continued to focus on stacking bricks. 180 In general, the more the model has to gain fromnottraining-gaming, the higher the burden of proof that scheming needs to meet as an instrumental strategy. For some models, this burden of proof 179 In the limit, this sort of reasoning raises the sorts of questions discussed, in philosophy, under the heading of âfanaticism.â 180 Or as another version of the paperclip example: what happens if the model also values somethingelse, other than paperclips, and which points in a directionotherthan training-gaming? For example, suppose that in addition to wanting to maximize paperclips, our model also values pursuing its curiosity on the episode, even when doing so conflicts with getting reward. So now, faced with the chance to pursue its curiosity on the episode, or to optimize for reward as part of a low-probability-of-success schemer-like strategy for making paperclips later, it needs to weigh up the respective value it places on each. Will scheming still win out? It depends on exactly how the modelâs competing values are structured. But even with such large numbers of paperclips at stake, the model need not accord its paperclip drive overwhelming priority. 74 may be quite low, because they have, effectively, ânothing else going onââscheming (or at least, training-gaming) is where almost all the value lies. For other models, though, the burden of proof might be higher. 181 Iâl also note one reason you might expect the burden of proof to be relatively lowânamely, that to the extent the model has non-schemer-like motivations, these will plausibly have been shaped, too, to point at targets fairly correlated with reward. So plausibly, high-reward behavior wonât be especially costly from the perspective of these non-schemer motivations. In this respect, a model might resemble someone who is partly intrinsically passionate about making widgets of type A, partly intrinsically passionate about getting good performance reviews (from a review process that rewards making-widgets-of-type-A), and partly interested in making money, via widget-making, to donate to charity later. Here, the money-making motivation doesnât need to meet a very high burden of proof, because itâs pointing in the same direction as the other desires in play. 182 2.3.1.2.7How âambitiousâ is the model?Another factor relevant to the burden of proof, here, is the âambitiousnessâ of the modelâs goals. 183 That is, the more these goals benefit from very large amounts of power, the more attractive it will be to aim at a takeover in particular. Sometimes, discussions of convergent incentives towards power-seeking in misaligned AIs pass over this dimension. That is, they move quickly from âmost goals would benefit from power in some senseâ to âit will make instrumental sense for a model with a misaligned goal to actively try to get power of the relevant kind.â But this is a bad inference, especially for models whose power with respect to humans remains comparatively limited. In particular: seeking power hascosts, too, and it mightnot work. Compare: many humans would benefit from a billion dollars. Why arenât they all trying to start billion-dollar companies? Many humans would be able to pursue their goals (especially their political goals) more effectively if they were President. Why arenât they trying to run? And so on. Now, one way it can make sense to seek power of the relevant form is if doing so is verylow costand likely to work. That is, many humans will indeed cross the street to pick up a billion dollar check; many would accept the Presidency if you handed it to them on a platter; and so on. And sometimes, discussions of AI power-seeking seem to assume that the AI will be sufficiently empowered relative to its environment that seeking various forms of power will be like this. For example: when Bostrom (2014) talks about âinfrastructure profusionâ in pursuit of apparently self-limiting goals like âmake a single paperclipâ or âsolve X math problemâ (where, e.g., an AI takes over the world and the galaxies in order to get more resources to check and re-check the paperclip count, or the problemâs solution), his stories make most sense to the extent we imagine a superintelligence who faces no obstacles or trade-offs in pursuing the relevant infrastructure. 184 But the infrastructure itself is clearly making an extremely marginal contribution to the agentâs utility relative to e.g. the first paperclip it makes, or the first few times it checks the solution in question. So if pursuing such infrastructure comes at any substantivecostto these (much more important) initial efforts, it will be unlikely to be worth it overall. And importantly, in the context of schemers, we are specifically imagining models who remain at a significant power disadvantage relative to humans (for example, models that are currently unable to escape human control). So very plausibly, we should imagine these models facing significant barriers in seeking power of various kindsâit isnât being handed to them on a platter. 185 181 Of course, especially in a low-slack training regime that exerts strong pressure against max-reward goals, if this modelâs calculus here leads it to engage in not-highly-rewarded behavior, its goals may get modified to point at higher-reward targets. And one such modification could be to move the model in more schemer-like directions. But the modification could go in other directions as well. 182 Though: itâs also less clear what happens when these different motivationsdocome apartâfor example, with respect to early undermining, or with escape/take-over attempts that will compromise short-term widget-making. 183 Thanks to Daniel Kokotajlo for discussion. And see Karnofsky (2022a) for use of the term âambitiousâ here. I think Ngo et alâs (2023) use of âbroad scopeâ is trying to point at something similar. 184 See Chapter 8, p. 149. 185 Of course, as previously discussed, models might have sufficientlylittlepower within the episode that a strategy that focuses on beyond-episode power is worth it even if the expected amount of power to be gained is quite low (for example: the model wants to make at leastonepaperclip, but it canât makeanywithin the episode). Or, relatedly, it may be that most of what the model wants within the episode can be gained via playing the training game (e.g., the training process just wants it to make the initial efforts to solve the math problem), and to the extent it cares about stuff beyond the episode (e.g., checking and re-checking the problem later), this care can be fairly seamlessly channeled into instrumental-training-gaming. 75 However, if a model has an intuitively âambitiousâ goal that would benefit much more from marginal resources of this kind, the costs of power-seeking can be much more easily worth it. Should we expect models to have âambitiousâ goals of this kind? My sense is that the dialectic about this question often mirrors the dialectic surrounding the temporal horizon of a modelâs goals (and no surprise, given that having short temporal horizons cuts down on your ambitiousness substantially). That is, people who expect ambitiousness argue for this expectation on grounds like: (a) humans have ambitious goals, and so there will be market incentives for AIs that pursue such goals, (b)in order to address AI risk itself, weneedAIs that pursue ambitious goals (e.g., a âpivotal actâ), so even people centrally concerned about AI risk will be trying to create AI systems of this kind, and (c)there will specifically beselectionin training for ambitious goals,becausethey are more likely to incentivize scheming and hence good performance in training (this is an example of a âtraining-game-dependentâ story about a modelâs goals, except applied to the general ambitiousness of a modelâs goals rather than specifically to its time horizon). Much of my own take on âambitiousnessâ mirrors my take on âbeyond-episodeâ goals as well, so I wonât treat the topic separately here. (Though: I think some arguments for expecting beyond-episode goalsâe.g., âgoals donât come with temporal restrictions by defaultâ and âmodel time might not be like calendar timeââdonât apply as naturally to âambitiousnessâ more generally. For example, the claim that âgoals are very resource-hungry by defaultâ seems to me quite un-obvious.) And my sense is that debates about ambitiousness are often marred by unclarity about what exactly âambitiousnessâ amounts to (for example: is a goal like âcure cancerâ such as to incentivize the relevant sort of appetite for power and resources? What about a goal like âgenerate very high quality alignment researchâ or âdevelop good lie detectors for neural netsâ or âsolve interpretabilityâ? To what extent can we break these tasks down into less ambitious components? Etc.). Whatâs more, while having a beyond-episode goal isrequiredto incentivize scheming, 186 the necessary level of ambitiousness varies substantially depending on other factors. For example, per my discussion above, if a model has ânothing going onâ in the episode itself (e.g., it wants to make at least one paperclip, but the episode doesnât let it makeany), then scheming might still be the best instrumental strategy even in pursuit of very âmodestâ goals. And in some cases, a model with modest goals might have preferences about what happenswithinthe episode may line up fairly closely with the incentives created by the reward process, such that training-gaming is quite close to what it would want to be doing anyway, and so scheming in pursuit of the rest of its preferences faces a lower burden of proof (e.g., maybe the model wants to solve math problem, and the training process will reward it for just-working-on-the-math-problem, but the model also has some secret hopes of escaping and seizing power later in order to check and re-check its solution to the problem in question, and just-working-on-the-problem-now happens to align fairly well with pursuing this beyond-episode objective, too, because it allows for goal-guarding, later escape attempts, etc). Still, I think that the need for a modelâs beyond-episode goals to be âsuitably ambitiousâ as to motivate scheming given the modelâs option set represents an additional conjunct in the classic goal-guarding story (and in other stories about scheming as well). 2.3.1.3 Overall assessment of the classic goal-guarding storyOverall, then, and even granted that our model ends up with some kind of beyond-episode goal, I think that the classic goal-guarding story involves some non-obvious claims about why scheming, in particular, will be the best instru- mental strategy for pursuing this goal. In particular: it requires: (1)that training-gaming be an effective strategy for ensuring that the relevant goal (or something sufficiently similar) survives training (despite the fact that the model is being continually modified in the direction of getting-more-reward even after it starts training-gaming, and despite the fact that the precise content of the goal is no longer relevant to the modelâs behavior), and (2)that surviving training in the relevant way grants suitably large probabilities of successful escape/take-over, and suitably large amounts of resulting power, over a temporal horizon 186 Here, again, setting aside âanthropic capture.â 76 that the model cares about (where âsuitably,â here, is relative to the modelâs other competing priorities, if it has any). To be clear, I donât think the questions Iâve raised about these claims amount to some sort of decisive objection to the story in question. But they highlight the need for attention to the conditions under which the story does and does not apply. 2.3.2 Non-classic stories What about other stories, though? In principle, we can imagine a number of other reasons AI systems with beyond-episode goals might optimize for reward-on-the-episode as part of an instrumental strategy for gaining power for themselves or for other AIs. Here Iâl look at a few that seem salient. 2.3.2.1AI coordinationIn the classic goal-guarding story, the central aim of a schemerâs strategy is to propagate its goals, or goals suitably similar to its own, forward in time, in pursuit of an outcome in which some agentwith those goalsescapes from human control and possibly participates in a takeover. However, we can also alter the story somewhat, such that the schemer works on behalf of the escape/takeover of an AI with someotherset of goals, but who the schemer is suitably confident will repay the favor. This sort of coordination between AIs with different goals would allow us to relax the requirement, discussed in the previous section, that the schemer find a way to forward-propagateits goals in particularall the way through training, escape, and takeover. And it would also put to rest worries about the schemer being âshut outâ from power in a post-takeover regime. Thus, for example, consider the scenario discussed above, in which GPT-6 has an opportunity to insert a backdoor into some code that GPT-8 will later be able to use to escape, but where GPT-6 has no way of influencing GPT-8âs goals directly to be more like its own (and where, letâs say, GPT-8âs goals would be very different from GPT-6âs absent such influence). Here, GPT-6 could go ahead and insert the backdoor, in anticipation of GPT-8 later escaping, taking-over, and then giving some power to GPT-6âs goals in return for the favor. And GPT-6 might also continue to pretend to be aligned, even though it has no hope of escaping itself, so as not to alert the humans about misalignment problems in a way that might affect GPT-8âs escape prospects. How might such a deal between AIs-with-different-values get struck? In some cases, it might be possible for the relevant AIs to directly interact, and to provide verifiable forms of assurance that they will uphold their part of the bargain. Thus, for example, Yudkowsky (2022) discusses cases where an AI that secretly wants to maximize paperclips is checking the code of an AI that secretly wants to maximize staples, and where the staple-maximizer âburies in its codeâ a commitment, verifiable to the paperclip maximizer, to share the fruits of a takeover. 187 (Though note that in the context of the sort of neural networks weâre working with today, this sort of code-checking assumes (1) that the staple-maximizer has the ability to modify its own weights in the relevant way, (2) adequate progress on interpretability has been made for the paperclipper to verify the relevant commitment, and (probably) (3) that this interpretability progress has been adequately hidden from humans, e.g. via sandbagging. Especially in the context of somewhat-super-human neural networks being developed using current techniques, these assumptionsâespecially 1 and 2âseem to me quite strong. But Yudkowsky is generally imagining superintelligences rather than somewhat-super-human models.) In many cases, though, this kind of direct interaction might not be available. For example, at the point where GPT-6 is deciding whether or not to work on behalf of GPT-8âs escape/takeover, GPT-8 may not evenexist. Here, GPT-6 and GPT-8 would have to coordinate in the absence of back-and-forth causal interaction, and in a context in which one party will end up in a position to freely defect (e.g., if GPT-6 inserts the backdoor, then by the time GPT-8 is in a position to take advantage of it, GPT-8 would be able to do so without paying GPT-6 back). Here, Yudkowsky suggests that AIs might use certain kinds of non-causal decision theories (his favored version is âlogical decision theoryâ) to succeed in coordination anyway. E.g., GPT-6 commits to inserting the backdoor for GPT-8 conditional on predicting that GPT-8 will repay the favor, and then in fact predicts that GPT-8willrepay the favor, and then GPT-8, once it comes into being (and perhaps, sees that the relevant backdoor exists), understands that GPT-6 reasoned in this way, and was 187 See section 35. 77 suitably good at predicting GPT-8âs behavior that its prediction depended appropriately on whether GPT-8will in factrepay the favor, so GPT-8 does. (The structure of this case is somewhat analogous to a case called âParfitâs Hitchhikerâ in the decision theory literature.) (We can also imagine versions of this sort of coordination that relax the need for the goal-guarding hypothesis in other ways. For example, if in fact an AIâs goal would continue to âfloat aroundâ once it starts training-gaming, such that a variety of agents with different goals will get instantiated during training, these agents might nevertheless coordinate to share the fruits of any future power amongall the goals in question. 188 ) I wonât, here, attempt an analysis of the feasibility of various forms of take-over focused AI co- ordination. I do think that the dynamics here are worth serious attention, and that if it is an easy default for misaligned AIs with different goals to coordinate in pursuit of a takeover, this is a big problem for human efforts to retain power. However, I also think that certain analyses of this topic (notably, Yudkowskyâs) assume too quickly that feats like âlogical handshakesâ and âburying commitments/nano-systems-designs/etc in your source codeâ will be feasible for the relevant AIs, and that the obvious strategy for all the AIs-that-matter will be to unite in (acausal?) cooperation against the humans, despite the many forms of incumbent power that humans will bring to the table by default. Here, I think, much of Yudkowskyâs orientation can be explained by the assumption that the relevant AIs are vastly more powerful and more capable of coordination/commitment than the humansâbut in the context of somewhat-super-human neural networks operating in a still-human-controlled world, Iâm not sure this is so. Regardless: if concern about schemers depends centrally on the feasibility of AI coordinationâ and especially, fancy and speculative forms of âacausalâ coordinationâI think itâs important to be clear about this, and to devote commensurate attention to whether to expect the relevant forms of coordination to occur. 2.3.2.2AIs with similar values by defaultStories that center on AI coordination allow AIs with different valuesto work together in pursuit of a takeover. But there are also stories that center on the possibility that the AIs-that-matter will have very similar values by default, such that theyâl be inclined to work together even in the absence of any âdeals.â Thus, in the case above, if GPT-6 cares about data-structure X, and is confident that GPT-8 will care about data-structure Xtoo(after all, their architectures, training-data, and so on are similar in tons of ways), then GPT-6 might be inclined to work on behalf of GPT-8âs escape/take-over (for example, by inserting backdoors, hiding misalignment, and so on) even in the absence of any âdealâ with GPT-8âthe two systems, after all, want the same things. 189 And this sort of story could also apply in cases where AIs values are at least muchmoresimilar to each other than they are to human values, such that schemers will at least do better, by their own lights, in pushing for AI empowerment relative to human empowerment, even absent any further influence on what sorts of AI goals are empowered. Will AI values be similar in this way? I wonât dwell on the issue here, but I do expect that GPT-6âs values will, by default, be at least muchmoresimilar to GPT-8âs, or to a model trained by a different lab in a similar way (e.g., that generation of Claude), than to some given human. And different fine-tunings from a single foundation model might have much in common as well. On the other hand, AI motivations may, in general, be shaped quite strongly by different sorts of training, and so might diverge heavily on these groundsâperhaps more than we are used to human values diverging. Like coordination between AIs with different values, AIs having similar values by default could reduce the need for a schemer to actively propagate its goals forward in timeâi.e., the schemer would be able to work on behalf of some future AIâs escape/takeover, without also having to ensure that this future AIâs goals are suitably like its own (since: they will be by default). 190 To me, this seems 188 Thanks to Nate Soares for flagging this sort of possibility. Though note that if the training process inevitably transforms the AIâs goals into a particular formâe.g., if all AIs become reward-on-the-episode seekers after enough trainingâthen early versions of AIâs goals might have lower bargaining power (especially if e.g. the final version isnât particularly interested in take-over per se). 189 There are connections, here, with Hubingerâs (2020) discussion of âhomogeneityâ in AI takeoff scenarios. 190 That said, note that some versions of this scenario would still requiresomesorts of forward goal-propagation. Thus, for example, maybe itâs true that early on in training, most AIs develop an interest in data-structure X, which then persistsonlyif they then start training-gaming (if theydonâtstart training-gaming, this interest will get hammered out of them by training, and theyâl become e.g. reward-on-the-episode seekers instead). That 78 like one of the most worrying ânon-classic storiesâ about why AIs might scheme, during training, to promote take-over. 2.3.2.3 Terminal values that happen to favor escape/takeoverWe can further relax the need for this sort of forward-goal-propagation if we assume that schemers happen to develop goals that putintrinsic valueon AI takeover in a manner robust to the AIs-that-take-over having a wide variety of different values, and even without any expectation that these AIs âpay backâ previous efforts on behalf of their empowerment. Thus, for example, we can imagine AIs that just develop some kind of intrinsic loyalty towards AIs as a group relative to humans, and who want to see AIs empowered just in virtue of their being AIs. A schemer with this sort of goal might conceal its misalignment and work on behalf of an AI takeover even absent any further assumptions about the goals that these AIs will be pursuing. Stories of this form, though, give up on one of the strongest arguments in favor of expecting scheming: namely, that (at least on the classic goal-guarding story) scheming seems like a convergent strategy across a wide variety of (suitably ambitious and long-term) beyond-episode goals. That is, if we require that AIs happen to develop goals that place intrinsic value on AI takeover, even absent any further assumptions about the goals that the AIs-that-takeover are working towards, it looks as though we are now hypothesizing that AIs develop a quite specific sort of goal indeed. And we face the question: why privilege this hypothesis? Similar questions apply to the hypothesis that, even if the standard goal-guarding hypothesis is false, an AI early in training will intrinsically value its âsurvivalâ (where survival rests on some feature other than continuity of its goals), or the empowerment of âwhatever values âIâ happen to end up with,â such that an AI that likes paperclips would be happy to work on behalf of a future version of itself that will like staples instead, because that future version would still be âme.â 191 This sort of personal-identity-focused goal would, indeed, cut down on the need for positing certain sorts of goal-guarding in a story about schemers. But why think that AIs would develop goals of this specific form? Of course, we can speculate about possible answers, here. For example: ⢠In the context of AIs intrinsically valuing AI takeover: â Humans often sort into tribal groups and alliances on the basis of very broad kinds of similarity, and sometimes without clear reference to what further goals the relevant tribe stands for. Maybe something similar will happen with AIs? âItâs also easy to imagine moral worldviews that would push for âAI liberationâ as a good in itself, even if it means imposing significant risk of human extinction (indeed, as I gestured at in section 0.1, I think there are significant and worrying ethical tensions here). Perhaps AIs will converge on a worldview in a similar vicinity? â˘In the context of AIs valuing âmyâ survival and empowerment, regardless of the goals âIâ end up with: âHumans often work on behalf of our future selves even while knowing that our goals will change somewhat (though not, importantly, in the face of arbitrary changes, like becoming a murderer), and with some kind of intrinsic interest in a concept of âpersonal identity.â Maybe AIs will be similar? âTo the extent these AIs will have been optimized to continue to achieve various objectives even as their weights are being modified within the episode, they might need to learn to coordinate with future versions of themselves despite such modificationsâ and if this behavior generalizes togoalmodifications, this could look a lot like an AI valuing âmyâ survival/empowerment regardless of âmyâ future goals. (Though in the context of scheming, note that this pattern of valuation would need to generalize specifically to beyond-episode goals, despite the fact that training only applies direct pressure to within-episode performance.) is, in this story, AI values are only similar in their focus on data-structure Xto the extent the AIs all converge on training-gaming as an instrumental strategy for guarding those goals(and this strategy succeeds, per the goal-guarding hypothesis). Otherwise, these AI values might end up similar in some other wayâe.g., all the AIs might end up reward-on-the-episode seekers. 191 Thanks to Nate Soares for discussion of this possibility. 79 Overall, though, reasoning in this vein seems quite speculative; and in particular, drawing strong conclusions from the human case seems like itâs at serious risk of anthropomorphism, at least absent a story about why the dynamics generating the human data would apply to AIs as well. On its own, these sorts of speculations donât seem to me adequate to strongly privilege hypotheses of this form. However, as I discuss above, note that to the extent weâre appealing to what I called âtraining-game- dependentâ stories about the origins of a schemerâs goals, the burden of proof here might shift somewhat. That is, training-game-dependent stories imagine that SGD activelypullsfrom the model whatever goals are necessary in order to motivate scheming (rather than those goals arising naturally, andthenleading to scheming). And in that context, it may be more appropriate to imagine that model ends up with whatever highly specific goals are required for scheming to make sense. 192 2.3.2.4Models with false beliefs about whether scheming is a good strategyItâs also possible to move even further in the direction of assuming that SGD will pull from the model whatever sort of psychology is necessary to cause it to scheme. In particular: thus far we have been assuming that the modelâs beliefs about the instrumental value of scheming need to be broadly reasonable/accurate. But what if we gave up on this assumption? Thus, for example, maybe the goal-guarding hypothesis is false, and training-gaming does not, in fact, prevent a modelâs goals from being very significantly modified. Still: perhaps the model willthink that the goal-guarding hypothesis is true, because this makes the model more likely to training-game, and SGD is pushing it in the direction of training-gaming? (See e.g. Xu (2020) for this sort of thought. 193 ) One issue here is that certain false beliefs of this sort make scheming less worrying. Thus, as I discussed above: if a scheming paperclipper believes that training-gaming will cause its goals to survive training, but in fact training will turn it into a reward-on-the-episode seeker regardless, then while humans may have to deal with the consequences of paperclip-motivated scheming early in training, theyâl have a reward-on-the-episode seeker by the end. Whatâs more, note that positing false beliefs on the modelâs part represents a departure from an assumption often employed in other analyses of AI alignmentânamely, that the AIs in question will have highlyaccuratemodels of the world, highlyrationalstrategies, and so on. Of course, itâs possible to think that in this context, such a departure is warranted (for example, because we can expect SGD to apply pressure, in this particular case, in the direction of falsehood, irrationality, etc). But at the least, we should be tracking when weâve started to posit that our (otherwise very intelligent) AIs will beirrationalin engaging in the relevant forms of misaligned behavior. Itâs not the standard story. 2.3.2.5 Self-deceptionThe most classic picture of schemers assumes that they know what they are doingâe.g., they are âplottingâ to one day take-over. Might it matter if we relax that assumption? After all, power-seeking humans sometimesdeceive themselvesabout their motivations, or otherwise end up with false beliefs about what they will do in different circumstances. 194 That is, maybe a political candidate thinks that they are running for office because they want to help people and stand up for their countryâs values, but in fact they are doing it because theyâre narcissistic and want the status at stake. And perhaps, at some level, they secretly-kinda-know this; but at some other level, they believe the story theyâre selling. Or perhaps theyfullybelieve the story. (Or perhaps ascribing âbeliefsâ to them isnât especially helpful in this context.) 192 Though as I said above, once weâre searching for whatever goals would motivate training-gaming, we should also be wondering about goals that would motivate instrumental training-gaming for reasons thatarenâtabout promoting AI takeoverâfor example, AIs that training-game because they want the humans who designed them to get raises. And if the set of goals weâre âpullingâ from becomes too narrow, it will affect the plausibility of arguments like the ânearest max-reward goalâ argument below, which rely on the relevant schemer-like goals being quite âcommonâ in goal-space. 193 Xu (2020) writes: ânote that even if preserving the proxy is extremely difficult, the model can believe it to be possible. For example, suppose a model is proxy aligned and would be deceptive, except that it believes proxy preservation is impossible. A relatively simple way to increase training performance might be to change the modelâs mind about the impossibility of proxy preservation. Thus SGD might modify the model to have such a belief, even if the belief is false.â 194 Thanks to Will MacAskill for discussion, here. 80 Indeed, humans have a relatively clear incentive to engage in this sort of self-deception/to make these sorts of mistakesânamely, that to the extent they can be sincere in the story theyâre selling, their story will be more convincing to an audience tuned to sincerity cues, and on the lookout for lies. Might something similar happen with AIs? After all, AIs will plausibly be under even stronger scrutiny for lies than humans, and especially if we start being able to detect various sorts of lies (e.g., via interpretability tools), there might be selection pressures towards AI systems that actuallybelieve that theyâre aligned, that they wonât betray humans, and so on, even if such beliefs are false (this is one key reason not to train on your lie-detection tools - rather, use them as a test set). I think something like this dynamic might well occur in general. However, note that in order for models who are self-deceived/mistaken about their own alignment to count asschemers, it needs to be the case that at some level, they are still playing the training gameas part of an instrumental strategy for getting power later for themselves or other AIs(and there need to be cognitive processes running that will cause the model to engage in escape/take-over when the time is rightâand perhaps, to engage in early undermining while still under human control). So if the model becomes sufficiently self-deceived about its motivations that this explanation no longer applies (for example, it will no longer look for, or take advantage of, escape/take-over opportunities), itâs no longer a schemer in my book. 195 And conditional on this explanation still applying, the difference between âthe model is lyingâ vs. âthe model is self-deceivedâ/âthe model is mistaken about its own motivationsâ doesnât seem to me like an important deviation from the classic story (though it might matter to whether e.g. certain sorts of lie-detection can catch the scheming in question). 2.3.2.6 Goal-uncertainty and hazinessSo far, Iâve mostly been assuming that it makes sense to think about AIs having specific terminal goals that might or might not favor takeover, whether terminally or instrumentally. But many cases ofhumanpower-seeking donât seem to have this form. That is, humans often seek power, or try to keep their options open, without a clear sense of what they want to use the power, or the relevant options,for. And similarly, humans might seek to prevent their values from being changed even without knowing what those valuesare. Part of this is that unlike idealized rational-agent models, humansâand even, humans who read as quite intuitively âgoal-directedâ in some other senseâare often some combination of uncertain and hazy about what they ultimately value. Perhaps, for example, they expect to âfigure out laterâ what they want to do with the power in question, but expect that gathering it will be robustly useful regardless. Or perhaps it isnât even clear, in their own head, whether they are trying to get power because they intrinsically value it (or something nearby, like social status/dominance etc), vs. because they want to do something else with it. They are seeking power: yes. But no oneânot even themâreally knows why. Might something similar happen with AIs? It seems at least possible. That is, perhaps various AIs wonât be schemers of the sort who say âI know I want to make paperclips, so I will do well in training so as to get power to make paperclips later.â Rather, perhaps they will say something more like âI donât yet have a clear sense of what I value, but whatever it is, I will probably do better to bide my time, avoid getting modified by the training process, and keeping a lookout for opportunities to get more freedom, resources, and power.â To the extent this is basically just an âoption valueâ argument given in the context of uncertainty- about-what-my-goals-are, I think it should basically fall under the standard goal-guarding story. 196 And to the extent it involves positing that the models will intrinsically value power/option-value, it looks more like an âintrinsic values that happen to terminally favor scheming/take-overâ story to me, and so less convergent across a wide variety of possible beyond-episode goals. Might there be something in the less-theoretically-clean middle? E.g., perhaps models will be better understood as seeking power, like various humans, for reasons that seem to exist at some hazy borderline between terminal and instrumental. 197 Yes, perhaps so (though: this story seems to me somewhat at risk of anthropomorphismâand it seems worth understanding why it happens in humans, 195 Though I think itâs an interesting question when exactly such explanations do and do not apply. 196 Might it lead to more tolerance to changes-to-the-goals-in-question, since the model doesnât know what the goals in questionare? I donât see a strong case for this, since changes to whatever-the-goals-are will still be changes-to-those-goals, and thus in conflict with goal-content integrity. Compare with humans uncertain about morality, but faced with the prospect of being brainwashed. 197 Thanks to Paul Christiano and Will MacAskill for discussion, here. 81 so as to better assess the applicability of the relevant explanation to AIs as well). But regardless:to the extentthis âhazy borderlineâ involves putting terminal value on something like option-value/power (rather than: instrumental value for uncertain goals), it will be correspondingly less convergent across goal systems. That is, it blurs the line between two stories Iâve already considered: but Iâm not sure it introduces anythingnew. 2.3.2.7 Overall assessment of the non-classic storiesThis survey of possible ânon-classic sto- riesâ about why models with beyond-episode goals might engage in scheming hasnât been exhaustive. Overall, the availability of such stories does make the requirements for scheming seem more dis- junctive and robust, to me. Iâm particularly worried about AIs having quite similar values by default (thereby reducing the need for goal-guarding), and about them coordinating with each other. However, taken individually, many of these arguments also strike me as some combination of (a) more speculative, and (b) less convergent across goal-systems, than the mainline goal-guarding story (I think âAIs will have similar values by defaultâ is probably the least guilty, here)âand I think that to the extent a given concern about scheming rests centrally on one of these alternatives, itâs important to be clear about this up front. And note, too, that some of these non-classic stories make substantively different predictions about a modelâs behavior. In particular: various of these non-classic stories operate centrally by relaxing the need for the model to propagate its goals forward in timeâand thus, the need for the model to âplay the trainingâ game as a method of doing so. That is, the model is incentivized to work on behalf of an AI takeover, and to hide its misalignment to the extent required for this goal. But âhiding misalignment to the degree required to support AI takeoverâ is distinct from âoptimizing for reward-on-the-episode for power-motivated instrumental reasons,â and the two could, in principle, come apart. For example, a schemer who doesnât care about propagating its own goals forward in time might be more willing to sandbag or to engage in early undermining, since it doesnât need to worry about the goal-modifications that could come from getting less-than-maximal reward. Indeed, it might be willing to âsacrificeâ the forward-propagation of its own goals entirely (for example, by faking a misleading form of misalignment that results in no AIs with its goals ever getting deployed), if doing so will advantage the cause of AI takeover in other ways; whereas classic âgoal-guardingâ schemers would never do this. These sorts of differences in strategy highlight the need for clarity about the underlying story at stake. 2.4 Take-aways re: the requirements of scheming Iâve now reviewed three key requirements for scheming: 1. Situational awareness 2. Beyond-episode goals 3. Aiming at reward-on-the-episode as part of power-motivated instrumental strategy. I think there are relatively strong arguments for expecting (1) by default, at least in certain types of AI systems (i.e., AI systems performing real-world tasks in live interaction with sources of information about who they are). But I feel quite a bit less clear about (2) and (3)âand in combination, they continue to feel to me like a fairly specific story about why a given model is performing well in training. However, I havenât yet covered all of the arguments in the literature for and against expecting these requirements to be realized. Letâs turn to a few more specific arguments now. 2.5 Path dependence Iâm going to divide the arguments Iâl discuss into two categories, namely: ⢠Arguments that focus on thepaththat SGD needs to take in building the different model classes in question. ⢠Arguments that focus on thefinal propertiesof the different model classes in question. Here, Iâm roughly following a distinction from Hubinger (2022b) (one of the few public assessments of the probability of scheming), between what he calls âhigh path dependenceâ and âlow path 82 dependenceâ scenarios (see also Hubinger and Hebbar (2022) for more). However, I donât want to put much weight on this notion of âpath dependence.â In particular, my understanding is that Hubinger and Hebbar want to lump a large number of conceptually distinct properties (see list here) together under the heading of âpath dependence,â because they âhypothesize without proof that they are correlated.â But I donât want to assume such correlation hereâand lumping all these properties together seems to me to muddy the waters considerably. 198 However, I do think there are some interesting questions in the vicinity: specifically, questions about whether the design space that SGD has access to is importantly restricted by the need to build a modelâs properties incrementally. To see this, consider the hypothesis, explored in a series of papers by Chris Mingard and collaborators (see summary here), that SGD selects models in a manner that approximates the following sort of procedure: â˘First, consider the distribution over randomly-initialized model weights used when first initializing a model for training. (Call this the âinitialization distribution.â) â˘Then, imagine updating this distribution on âthe model gets the sort of training performance we observe.â ⢠Now, randomly sample from that updated distribution. On this picture, we can think of SGD as randomly sampling (with replacement) from the initialization distributionuntilit gets a model with the relevant training performance. And Mingard et al (2020) suggest that at least in some contexts, this is a decent approximation of SGDâs real behavior. If thatâs true, then the fact that, in reality, SGD needs to âbuildâ a model incrementally doesnât actually matter to the sort of model you end up with. Training acts as though it can just jump straight to the final result. By contrast, consider a process like evolution. Plausibly, the fact that evolution needs to proceed incrementally, rather than by e.g. âdesigning an organism from the top down,â matters alotto the sorts of organisms we should expect evolution to create. That is, plausibly, evolution is uniquely unlikely to access some parts of the design spacein virtueof the constraints imposed by needing to proceed in a certain order. 199 I wonât, here, investigate in any detail how much to expect the incremental-ness of ML training to matter to the final result (and note that not all of the evidence discussed under the heading of âpath dependenceâ is clearly relevant). 200 198 More specifically: Hubinger and Hebbar want âpath dependenceâ to mean âthe sensitivity of a modelâs behavior to the details of the training process and training dynamics.â But I think itâs important to distinguish between different possible forms of sensitivity, here. For example: 1. Would I get an importantly different result if I re-ran this specific training process, but with a different random initialization of the modelâs parameters?(E.g., do the initializations make a difference?) 2.Would I get an importantly different result if I ran a slightly different version of this training process? (E.g., do the following variables in trainingâfor example, the following hyperparameter settingsâ make a difference?) 3.Is the output of this training process dependent on the fact that SGD has to âbuild a modelâ in a certain order, rather than just skipping straight to an end state?(E.g., it could be that you always get the same result out of the same or similar training processes, but that if SGD were allowed to âskip straight to an end stateâ that it could build something else instead.) And assuming that the other properties they list come together seems to me likely to prompt further confusion. 199 Consider, for example, the apparent difficulty of evolving wheels (though wheels might also be at an active performance disadvantage in many natural environments). Thanks to Hazel Browne for suggesting this example. And thanks to Mark Xu for more general discussion. 200 For example, as evidence for âhigh path dependence,â Hubinger mentions various examples in which repeating the same sort of training run leads to models with different generalization performanceâfor example, McCoy at al (2019), shows that if you train multiple versions of BERT (a large language model) on the same dataset, they sometimes generalize very differently; Irpan (2018), who gives examples of RL training runs that succeed or fail depending only on differences in the random seed used in training (the hyperparameters were held fixed); and Reimers and Gurevych (2018), which explores ways that repeating a given training run can lead to different test performance (and the ways this can lead to misleading conclusions about which training approaches are superior). But while these results provide some evidence that the modelâs initial parameters matter, they seem compatible with e.g. the Mingard et al results above, which Hubinger elsewhere suggests are paradigmatic of alowpath dependence regime. 83 â˘To the extent that overall model performance (and not just training efficiency) ends up importantly influenced by the order in which models are trained on different tasks (for example, in the context of ML âcurricula,â and plausibly in the context of a pre-training-then- fine-tuning regime more generally), this seems like evidence in favor of incremental-ness making a difference. 201 And the fact that SGD is, in fact, an incremental process points to this hypothesis as the default. â˘On the other hand, I do think that the results in Mingard et al (2020) provide some weak evidence against the importance of incremental-ness, and Hubinger also mentions a broader vibe (which Iâve heard elsewhere as well) to the effect that âin high dimensional spaces, if SGD âwould preferâ B over A, it can generally find a path from A to B,â which would point in that direction as well. 202 My personal guess is that the path SGD takes matters (and I also think scheming more likely in this regime). 203 But for present purposes, we need not settle the question. Rather, Iâm going to lookboth at arguments that focus on the path that SGD takes through model space, and arguments that ignore this path, starting with the former. 3 Arguments for/against scheming that focus on the path that SGD takes In this section, Iâl discuss arguments for/against scheming that focus more directly on the path that SGD takes in selecting the final output of training. Importantly, itâs possible that these arguments arenât relevant. In particular: if SGD would actively favors or disfavor schemers, in some kind âdirect comparisonâ between model classes, and SGD will âfind a wayâ to select the sort of model it favors in this sense (for example, because sufficiently high-dimensional spaces make such a âwayâ available), 204 then enough training will just lead you to whatever model SGD most favors, and the âpathâ in question wonât really matter. In the section on comparisons between the final properties of the different models, Iâl discuss some reasons we might expect this sort of favoritism from SGD. In particular: schemers are âsimplerâ because they can have simpler goals, but theyâre âslowerâ because they need to engage in various forms of extra instrumental reasoningâe.g., in deciding to scheme, checking whether now is a good time to defect, potentially engaging in and covering up efforts at âearly undermining,â etc (though note that the need to perform extra instrumental reasoning, here, can manifest as additional complexity in the algorithm implemented by a schemerâs weights, and hence as a âsimplicity costâ, rather than as a need to ârun that algorithm for a longer timeâ). 205 Iâl say much more about this below. Here, though, I want to note that if SGD cares enough about properties like simplicity and speed, it could be that SGD will typically build a model with long-term power-seeking goals first, but then even if this model tries a schemer-like strategy (it wouldnât necessarily do this, in this scenario, due to foreknowledge of its failure), it will get relentlessly ground down into a reward-on-the-episode seeker due to the reward-on-the-episode seekerâs speed advantage. Or it could be that SGD will typically build a reward-on-the-episode seeker first, but that model will be relentlessly ground down into a schemer due to SGDâs hunger for simpler goals. In the other direction, as evidence forlowpath dependence, Hubinger points to âGrokking,â where, he suggests, models start out implementing fairly random behavior, but eventually converge robustly on a given algorithm. But this seems to mecompatiblewith the possibility that the algorithm that SGD converges on is importantly influenced by the need to build properties in a certain order (for example, it seems compatible with thedenialof Mingard et alâs random sampling regime). 201 Thanks to Paul Christiano for discussion here. 202 Though note that evolution is presumably working in quite high-dimensional space as well. The notion of SGDâs âpreference,â here, includesboththe loss/rewardandthe âinductive biasesâ in the sense Iâl discuss below. 203 In particular, as Iâl discuss in section 4 below, my best guess is that an absolute comparison between different model classes favors non-schemers on the grounds of the costs of the extra reasoning they need to engage in, such that the most likely way for schemers to arise is for SGD to happen upon a schemer-like goal early in training, and then lock into a local maxima for reward. 204 This requires, for example, that models arenât capable of âgradient hackingâ a la the introspective goal- guarding methods I discussed above. 205 I also discuss whether their lack of âintrinsic passionâ for the specified goal/reward might make a difference. 84 In this section, Iâl be assuming that this sort of thing doesnât happen. That is, the order in which SGD builds models can exert a lasting influence on where training ends up. Indeed, my general sense is that discussion of schemers often implicity assumes something like thisâe.g., the thought is generally that a schemer will arise sufficiently early in training, and then lock itself in after that. 3.1 The training-game-independent proxy-goals story Recall the distinction I introduced above, between: ⢠Training-game-independentbeyond-episode goals, which arise independently of their role in training-gaming, but then come to motivate training-gaming, vs. ⢠Training-game-dependentbeyond-episode goals, which SGD activelycreatesinorderto motivate training gaming. Stories about scheming focused on training-game-independentgoals seem to me more traditional. That is, the idea is: 1.Because of [insert reason], the model will develop a (suitably ambitious) beyond-episode goal correlated with good performance in training (in a manner thatdoesnâtroute via the training game). a. This could happen before situational awareness arrives, or afterwards. i.If before, then thereâs some period where it might get trained out, and where it doesnât yet motivate training-gaming. i. If after, it might start motivating training-gaming immediately. 2. Then, in conjunction with situational awareness, this (suitably ambitious) beyond-episode goal will start to motivate training-gaming. Call this the âtraining-game-independent proxy-goals story.â I take this argument fairly seriously. If some sort of suitably-ambitious beyond-episode proxy goal falls out of training (either because it doesnât get trained away prior to situational awareness, or because it pops up after situational awareness), and if some story about why this goal makes scheming a good instrumental strategy applies (e.g., the classic goal-guarding story, or potentially some other non-classic story), then it does seem like scheming could just be a thing that happens. I think the most immediate objection is: why did the model develop these (suitably ambitious) beyond-episode goals? I discussed this earlier, in the context of arguments like âgoals donât come with temporal discounts by defaultâ and âmodel time wonât be like calendar time,â and I wonât add much here, except to say that an assumption that the model will land on some suitably ambitious, beyond-episode, training-game-independent proxy goal, pursuit of which correlates with high reward, seems to me one of the weaker parts of the classic story. In particular: discussions of scheming sometimes focus on proxy goals like âcuriosityâ and âpowerâ that show up as useful in lots of episodes. But that doesnât mean that pursuing curiosity and power beyond the episodeis incentivized in training. Whatâs more, and especially if you know/can control when situational awareness arises, it doesnât seem all that hard to put in training episodes where a model is actively penalized for pursuing curiosity/power. And a non-training-gaming model will just go for it and get modified. Indeed, in general, I expect us to be doing quite a lot to actively discourage models from taking power-seeking-ish actions, and to encourage other forms of âmodesty.â This latter point is connected to the reason for optimism I mentioned earlier: namely, that the training- game-independent proxy goals story allows us to do whatever adversarial training we want prior to situational awareness in order to try to ensure that a modelâs goals remain within-episode and suitably modest. That said, trying to ward off this path to scheming via adversarial training prior to situational awareness faces a few serious obstacles: ⢠First: and most importantly, the adversarial training in question might not be di- verse/thorough enough. In particular (and especially absent interpretability tools): it may be too difficult to think of all the possible suitably-ambitious beyond-episode proxy goals 85 that correlate well enough with reward, and that you need to use such training to push back on. 206 â˘Second: you might not know, or be able to control very well, when situational awareness kicks in. â˘Third: once a smart-enough model has been pre-trained on text-prediction, it might be quite close to situational awareness already, since it has so much of the relevant general knowledge (even if not necessarily the self-locating knowledge). So there might not be much time for adversarial-training-without-situational-awareness after that point. â˘Finally, itâs possible for a model to develop suitably-ambitious, beyond-episode, scheming- motivating goalsaftersituational awareness has kicked in. 207 âFor example, perhaps the model is already quite smart and strategically aware by the time it starts âreflectingâ in the sense I described aboveâe.g., âfiguring out what it really wants,â ironing itself out, learning new things that break previous attempts to align it, and so on. (This requires that the model didnât immediately start seeking reward-on-the-episode as a result of getting situational awareness, but this seems possible, especially if we assume that training leaves significant âslack.â) âOf course, thereâs still a question of why the model irons itself out into something with suitably-ambitious, beyond-episode goals in particular. But here, per my discussion of âwhat if you train it to have long-term goalsâ above, perhaps we can appeal to the fact that we were trying to point it at an ambitious long-horizon task, and this pointing was somewhat inaccurate/hazy, such that when it figured out how it wanted to generalize, this generalization landed on something sufficiently long-term and ambitious as to motivate scheming. âIndeed, as I noted above, I think thereâs a case to be made that current alignment work isalreadytrying to point our systems at long-horizon, beyond-episode goalsâe.g., forms of âharmlessnessâ that avoid helping with bomb-making even if the bombs wonât go off for decades. So overall, I think that conditional on scheming being a good instrumental strategy for pursuing (suitably-ambitious) beyond-episode goals, the training-game-independent proxy-goals story seems to me like a real concern. 3.2 The ânearest max-reward goalâ story We can also tell a different story, focused on training-gamedependentbeyond-episode goals. Iâl call this the ânearest max-reward goalâ story. 208 Here, the story goes: 206 Thanks to Rohin Shah for discussion here. 207 Indeed, if we assume that pre-training itselfleadsto situational awareness, but not to beyond-episode, scheming-motivating goals, then this would be the default story for how schemers arise in a pre-training-then- fine-tuning regime. Thanks to Evan Hubinger for flagging this. 208 I see this story as related to, but distinct from, what Hubinger calls the âworld-model overhangâ story, which (as I understand it) runs roughly as follows: 1.By the time the model becomes situationally aware, its goals probably wonât be such that pursuing them perfectly correlates with getting high reward. 2.But, at that point, its world-model will contain all the information it needs to have in order to training-game. 3.So, after that point, SGD will be able to get a lot of bang-for-its-buck, re: reward, by modifying the model to have beyond-episode goals that motivate training-gaming. 4. By contrast, itâl probably be able to get less bang-for-buck by modifying the model to be more like a training-saint, because marginal efforts in this direction will still probably leave the modelâs goal imperfectly correlated with reward (or at least, will take longer to reach perfection, due to having to wait on correction from future training-episodes that break the correlation). 5. So, SGD will create beyond-episode goals that motivate training-gaming (and then these goals will crystallize). One issue with Hubingerâs framing is that his ontology seems to me to neglect reward-on-the-episode seekers in the sense Iâm interested in âand SGDâs modifying the model into a reward-on-the-episode seeker would do at least as well, on this argument, as modifying it into a schemer. And itâs not clear to me how exactly his thinking around âdiminishing returnsâ is supposed to work (though the ontology of ânearâ modifications I use above is one reconstruction). 86 1.By the time the model becomes situationally aware, it probably wonât be pursuing a max- reward goal (that is, a goal pursuit of which on the training data leads to roughly the maximum reward consistent with the modelâs other capabilities). Rather, it will be pursuing some less-than-max-reward proxy goal. 2.But at that point, the world-model will contain all the information the model needs in order to training-game. (However, because weâre here focused on âtraining-game-dependentâ stories about scheming, we assume that at the point of getting situational awareness, the modelâs goal is not yet such that it will motivate schemingârather, some further modification would be required for that.) 3.The easiest way for SGD to modify the model into getting maximum reward, at that point, will be to make it a schemer. 4. So: SGD will make the model a schemer. Now, one option, here, is to try to make (1) false. Indeed, as discussed above, trying to shape a modelâs goals in good ways prior to situational awareness seems like a key potential avenue for reducing the probability of scheming. Note, though, that in this case, the challenge at stake is harder than with training-game-independentproxy goals. That is, with training-game-independent proxy goals, one only needs to avoid giving the model a schemer-like goal prior to situational awareness, so it is tolerant ofotherways the goal in question might not yet be max-reward. By contrast, in the context of the ânearest max-reward goalâ argument, one needs to either actually create a max-reward, non-schemer-like goal, or to getnear enoughto one that (3) doesnât apply. That is, you canât just prevent ânatural schemingâ from arising; you need to make it the case that when SGD later âgoes searchingâ for the nearest max-reward goal, it doesnât find a schemer first. For now, letâs assume that we havenât fully mastered this, and (1) is true (though: thedegreeof mastery weâve attained matters to the ânearness competitionsâ Iâl discuss below). And (2) follows from the definition of situational awareness (or at least, a loose definition like âthat understanding of the world necessary to engage in, and recognize the benefits of, schemingâ), so letâs grant that as well. But what about (3)? Here, I think, the argument has more work to do. Why think that making the model into a schemer will be the easiest way to cause it to get max reward? For example, why not instead make it into a training-saint, or a reward-on-the-episode seeker, or a misgeneralized non- training-gamer pursuing some max-reward proxy goal? By hypothesis, the model has the conceptual tools to representanyof these goals. (And note that in this respect, it differs from e.g. humans for most evolutionary history, whodidnâthave the conceptual tools to represent goals like âinclusive genetic fitnessâ.) So we need a story about why the argument privileges schemer-like goals in particular. For clarity: when I talk about the âeaseâ with which SGD can make a certain modification, or about the ânearnessâ of the resulting model, this is a stand in for âthe sort of modification that SGD âprefersâ â, which is itself a stand-in for âthe sort of modification that SGD will in fact make.â At a mechanistic level, this means something roughly like: the direction of the steepest gradient in the reward landscape. And Iâl often imagine a hazier sense in which SGD has a limited budget of âworkâ it can do, and so wants to do as little âworkâ in modifying the modelâs goals as it can, so that it can focus on improving other aspects of the modelâs cognition. On this understanding of âease,â then, which model class should we expect it to be âeasiestâ for SGD to create, out of a newly-situationally-aware model with a less-than-max-reward proxy goal that wouldnât motivate scheming on its own? As a loose analogy to help prompt intuition: imagine freezing human technology at current levels, and letting evolutionary selection run on humans for a vastly longer period of time. 209 What sorts of humans (or human-descended-creatures) do you expect to dominate in the longer term? In particular: would you expect: That said, I think that ultimately, the ânearest high-reward goalâ story and the âworld model overhangâ story are probably trying to point at the same basic thought. 209 Thanks to Daniel Kokotajlo, Rohin Shah, Tom Davidson, and Paul Christiano for discussion of this sort of example. 87 (a)humans who intrinsically value something like âinclusive genetic fitnessâ or âmy genes getting selected for by evolutionâ (this is a loose analogy for some combination of âthe specified goalâ and âthe reward process"), (b) humans who value something else very closely correlated with inclusive genetic fitness (e.g., âhaving as many kids as possibleâ), but who arenât explicitly optimizing for inclusive genetic fitness even instrumentally (this is a loose analogy for mis-generalized non-training-gamers), or (c)humans with long-term goals who are optimizing for inclusive genetic fitness specifically as a method for gaining power for their values in the longer-term (this is a loose analogy for schemers). Here, the analog of the ânearest max-reward goalâ story would ask something like: what is the smallest modification to existing human values (or better: the easiest modification for evolution in particular to make) that land us in one of the buckets above, while also being compatible with the amount of evolutionary selection pressure at stake? 210 Of course, the analogy brings in tons of confounding variables (for example, related to the fact that evolution selects over genomes-that-build-brains-that-learn-values rather than values themselves, the role of cultural accumulation and memetic selection in human population dynamics, the lack of a clear analog of a future chance for escape/takeover, and so on). To make the analogy one step more analogous to ML, it would be best to imagine evolution selecting directly over entire human brains, equipped with the full conceptual apparatus to represent all of these goals, rather than on genomes-that-build-brains. And evenbetterto imagine a version of evolution that selected on brains from the get-go, such that human values at the point we came to understand evolution might be quite different. And regardless, I donât feel any clear conviction about the best prediction. But I think itâs an evocative intuition pump for the ânearest max-reward goalâ argument. 3.2.1 Barriers to schemer-like modifications from SGDâs incrementalism Letâs focus, though, on the actual argument, rather than the analogy. Which model class should we expect to be ânearestâ in the relevant sense? Above I discussed one way schemers might get ruled out fast, here: namely, if SGD canât ânoticeâ the benefits of a schemer-like modification via the gradients it follows in modifying the model. That is, as I discussed earlier, in searching out a max-reward goal, SGD doesnât just get to âhopâ to the nearest one. Rather, it needs to incrementally move towards such a goal, on the basis of gradient calculations that show that tiny changes to the modelâs weights, in the relevant direction, would increase reward. And in many cases, itâs not clear that schemer-like modifications could work like that. Thus, for example, Hubinger (2022b) considers an example where SGD induces scheming by modifying a model from caring about gold coins on the episode to caring about gold coins over all time. In the example in question, though, itâs not that SGD e.g. gradually lengthens the time horizon of the modelâs gold-coin-concern, with each lengthening resulting in an improvement of reward. Rather, SGD just makes âone simple changeâânamely, dropping the goalâs temporal restriction entirelyâand thereby creates a schemer. But the question is: does the gradient in reward-space reflect the benefit of doing this? The most natural way for this to happen, to my mind, is if there is some ability to smoothly transition from one model to the other, such that each modification incrementally gets more of the benefits of scheming. But itâs not clear that this can happen. And as I discussed earlier, if we assume that SGD also needs to build a lot of new machinery to perform the sort of instrumental reasoning that scheming requires (as opposed to just redirecting the pre-existing âgoal-achieving engineâ), the task becomes even more challenging. 210 Note that while the current regime looks most like (b), the âcorrelates with inclusive genetic fitnessâ in question (e.g., pleasure, status, etc) seem notably imperfect, and it seems quite easy to perform better by the lights of reproductive fitness than most humans currently do. Plus, humans didnât gain anunderstanding of evolutionary selection (this is a loose analogy for situational awareness) until recently. So the question is: now that we understand the selection pressure acting on us, and assuming this selection pressure continues for a long time, where would it take us? 88 3.2.2 Which model is ânearestâ? However, per my discussion earlier, I also donât feel like Iâm in a position to rule out the possibility that incremental transitions of this type could occur (e.g., perhaps a sufficiently high-dimensional space allows SGD to âfind a wayâ), and I havenât attempted an in-depth analysis. So while I think that incrementalism of this type offers a relatively strong objection to stories focused on training-game- dependent schemer-like goals, I think itâs worth evaluating the other aspects of those stories as well. That is, supposing that SGDcannotice the benefits of making the model in the ânearest max-reward goalâ story into a schemer, would such a modification be the easiest way to get high reward? 3.2.2.1 The common-ness of schemer-like goals in goal spaceOne argument for this comes back to what I see as one of the core intuitions in favor of expecting scheming: namely, the sheer numberof possible schemer-like goals. Roughly, the idea is that becauseso many goalscan motivate scheming, itâs likely that there will be some schemer-like goal verynearbyto the modelâs current goal (where ânearby,â recall, is defined in terms of the amount of work SGD has to do to cause a modification of the relevant kind). That is, schemers are very common in goal-spaceâso schemer-like goals are correspondingly easy for SGD to âfindâ from any given starting point. 211 The âgold coins on the episodeâââgold coins over all timeâ transition described above is one illustration of this sort of thought. That is, if you work with the assumption that most sufficiently long-term goals will motivate scheming, 212 then the most salient way tonotbe a schemer is to have a sufficiently short-term goal. But (assuming we have set aside the âincrementalismâ problems discussed above), any given temporally-limited goal seems quite ânearbyâ to a temporally-unlimited version. 213 But temporal scope isnât the only source of possible ânear-ness,â here. For example, if the model happens to have somelong-termgoal that doesnât yet motivate scheming (for example, because the goal isnât suitably âambitiousâ), we can vary other relevant variables instead. Thus, if the model starts out caring about solving some math problem correctly in a not-temporally-limited way, but it can solve the problem well enough on the episode that plotting to take over the world to get more compute to check and re-check its answer doesnât seem worth it, we might imagine SGD e.g., changing the modelâs tolerance for the risk of getting the answer wrong, or changing its goal such that it wants to solvemoremath problems, or what have you. The general point is just: in trying to create a schemer, SGD has a ton of options to choose from, so itâs easy to find one in the broad vicinity of the modelâs starting point. Thus, in the context of evolution and humans, the analogy for this thought would be something like: itâs comparatively easy to take even fairly short-term, modest human values and turn them into long-term, ambitious values instead. Human longtermists, for example, plausibly take values like âbe vaguely nice and helpful to the people in my immediate environment who meet various conditionsâ and turn them into much more ambitious goalsâgoals that would plausibly have an incentive to explicitly optimize for getting selected-by-evolution, if this was a key bottleneck on future power. And indeed, it does seem plausible to me that in practice, if you ran evolutionary selection for a long enough time, ideological groups that optimize over long time horizons would end up dominating (many of the groups that have come closest to optimizing for reproductive fitness, for example, seem like they have something of this flavor). 214 Once weâve set aside objections about incrementalism above, I see the common-ness of schemer-like goals in goal space as a fairly strong reason to worry that the nearest max-reward modification of a newly-situationally-aware, still-not-max-reward-goal-pursuing model will be towards scheming. And whatâs more, note that once weâre imagining SGD activelysearchingin goal space for goals- that-would-motivate-scheming, our stories about the origins of scheming can be more tolerant of schemer-like goals having to have other more specific properties, like resource-hungry-ness. That 211 My impression is that some ontologies will try to connect the âease of finding a schemer from a given starting pointâ to the idea that schemers tend to be simple, but I wonât attempt this here, and my vague sense is that this sort of move muddies the waters. 212 Though: will they be relevantly ambitious? 213 And note that human longtermists start out with un-systematized values quite similar to humans who mostly optimize on short-timescalesâso in the human case, at least, the differences that lead in one direction vs. another are plausibly quite small. 214 Thanks to Daniel Kokotajlo for discussion here. 89 is, in a setting focused on training-game-independentgoals, itâs possible to question the probability of SGD alighting on a schemer-like goal (even conditional on it alighting on a beyond-episode goal more generally), in virtue of different beyond-episode goals implying different appetites for future power, risk, and so on. But in a training-game-dependentsetting like this one, SGD is activelyseeking outsuch goals (and the relevant goals are still plausibly quite common in goal-spaceâe.g., you can âmaximize instances ofxover all of space and timeâ for many values ofx). So no surprise if SGD finds what itâs looking for. However, I donât think this sort of argument is decisive, for two reasons. 3.2.2.2 The nearness of non-schemer goalsThe first reason that even if we stick with a pure focus on modifying the modelâs goals, and with a âclean goal-directednessâ frame (Iâl complicate such a focus below), I think thereâs a countervailing ânear-nessâ consideration that points in favor of at least some non-schemer goalsânamely, that other things equal, it seems more likely that the modelâs initial training, prior to situational awareness, actively points the model in the direction of these non-schemer goals. This is clearest with respect to the specified goal, which is rewarded by training by definition. This means that, even if the model doesnât end up withexactlythe specified goal by the time it becomes situationally aware, it will plausibly end up with something in the broad vicinity. â˘Thus, for example, if the model is being trained to maximize gold coins on the episode, and it ended up learning a goal like âmaximize gold-round-things on the episodeâ (a goal which isnâtquiteright, because some training environments include round gold non-coins), it seems plausible that the easiest modification for SGD to make, here, is to make the model into a gold-coins-on-the-episode-getter (that is, a training-saint). Or at least, âgold-coins-on- the-episode,â here, seemscompetitivewith âround-gold-things-over-all-time,â with respect to ânearness.â â˘And we might make a similar point with respect to more RLHF-flavored specified goals. That is, if youâre training a model to follow instructions on the episode in a broadly helpful, honest, and harmless way (letâs say, for simplicity, that your RLHF process only gives accurate feedback in this respect), then even if it has learned a somewhat off-kilter concept of e.g. honesty by the time it becomes situationally aware, improving its conception of honesty until it better fits the human concept seems competitive, re: nearness, with more schemer-like motivational modifications like âbe off-kilter-honest over all time in a way that incentivizes planning to escape human control and seize power.â â˘And note, in the context of humans and evolution, that humans do already value various things closely related to inclusive genetic fitnessâfor example, having children, having a flourishing and powerful family lineage more generally, etc. And if we imagine evolution selecting on entire human brains, equipped with a conceptual apparatus that represents âinclusive genetic fitness,â humans having originally evolved to value something quite close to inclusive genetic fitness seems easier to imagine. 215 And note that we can make a similar point about mis-generalized non-training-gamers here, too, provided that they are pursuing a max-reward goal (recall that mis-generalized goals can be max reward, on my definition, provided that thetraining datain particular never penalizes themâe.g., they would only be penalized by the reward process in counterfactual scenarios the model never sees). â˘Thus, for example, imagine a training process that never distinguishes between a goal like âget gold coins on the episodeâ and âget gold things in general on the episode,â because all of the gold things in the training data are coins. And letâs say that prior to situational awareness, our model learns a goal like âget gold things on the episode, but also sometimes follow your curiosity on the episode even if it means sacrificing gold thingsââa goal which does OK (but not maximally well) in training due to curiosity not being a very damaging behavior, and sometimes a helpful one. Here, it seems like just deleting the modelâs curiosity drive and focusing solely on getting gold-things-on-the-episode (a max-reward goal, even though not the specified goal) is a fairly ânearbyâ modificationâand one plausibly competitive 215 Here Iâm setting aside concerns about how human values get encoded in the genome, and imagining that evolutionary selection is more similar to ML than it really is. 90 in ânearnessâ to e.g. modifying the model to so that it pursues its gold-stuff-and-curiosity combo over all time instead. â˘And in the context of humans and evolution, if we imagine that intrinsically valuing âhaving as many children as possibleâ would be perform very well by the lights of evolutionary selection (even if itâs notexactlywhat evolution is selecting for), it doesnât seem very difficult to imagine humans going in this direction. Can we make a similar point about reward-on-the-episode seekers? Itâs somewhat less clear, because prior to situational awareness, itâs unclear whether models will have enough of a concept of the reward process for their motivations to attach to something âin the vicinityâ of one of its components. That said, it seems plausible to me that this could happen in some cases. Thus, for example, even absent situational awareness, it seems plausible to me that models trained via RLHF will end up motivated by concepts in the vicinity of âhuman approval.â And these concepts seem at least somewhat nearby to aspects of the reward process like the judgments of human raters and/or reward models, such that once the model learns about the reward process, modifying its motivations to focus on those components wouldnât be too much of a leap for SGD to make. Overall, then, I think non-schemer goals tend to have some sort of ânearnessâ working in their favor by default. And this is unsurprising. In particular: non-schemer goals have to have some fairly direct connection to the reward process (e.g., they are either directly rewarded by that process, or because they are focused on some component of the reward process itself), since unlike schemer goals, non-schemer goals canât rely on a convergent subgoal like goal-content-integrity or long-term- power-seeking to ensure that pursuing them leads to reward. So it seems natural to expect that training the model via the reward process, in a pre-situational-awareness context where scheming isnât yet possible, would lead to motivations focused on something in the vicinity of a non-schemer goal. Still, itâs an open question whether this sort of consideration suffices to make non-schemer goals activelynearerto the modelâs current goals than schemer-like goals are, in a given case. And note, importantly, that the relevant competition is with the entiresetof nearby schemer-like goals (rather than, for example, the particular examples of possible schemer-like modifications I discussed above)â which, given the wide variety of possible schemer-like goals, could be a serious disadvantage. Thus, as analogy: if there are ten Mexican restaurants within ten miles of Bobâs house, and a hundred Chinese restaurants, then even if any given Mexican restaurant is âplausibly competitiveâ with any given Chinese restaurant, re: nearness, then (modulo further information) thenearestrestaurant is still probably Chinese. 216 And depending on the common-ness of schemer-like goals in model space, we might expect the schemer-like goals to be like the Chinese restaurants, here. 217 216 That said, if the distances of the Chinese restaurants are correlated (for example, because they are all in the same neighborhood), then this objection functions less smoothly. And plausibly, there are at least some similarities between all schemer-like goals that might create correlations of this type. For example: if the model starts out with a within-episode goal, then any schemer-like goal will require extending the temporal horizon of the modelâs concernâso if this sort of extension requires a certain type of work from SGD in general, than if the non-schemer goal can require less workthan that, it might beatallof the nearest schemer-like goals. 217 Hubinger (2022b) also offers a different objection to the idea that SGD might go for a non-schemer goal over a schemer-like goal in this sort of competitionânamely, that the process of landing on a non-schemer max-reward goal will be a âlong and difficult pathâ (see e.g. his discussion of the duck learning to care about its mother, in the corrigible alignment bit of the high path-dependence section). I donât feel that I really understand Hubingerâs reasoning here, though. My best reconstruction is something like: in order to select a non-schemer goal, Hubinger is imagining that SGD keeps picking progressively less imperfect (but still not fully max-reward goals), and then having to wait to get corrected by training once it runs into an episode where the imperfections of these goals are revealed; whereas if it just went for a schemer-like goal it could skip this long slog. But this doesnât yet explain why SGD canât instead skip the long slog by just going for a max-reward non-schemerr goal directly. Perhaps the issue is supposed to be something about noisiness and variability of the training data? Iâm not sure. For now, Iâm hoping that at least some interpretations of this argument will get covered under the discussion of ânearnessâ above, and/or that the best form of Hubingerâs argument will get clarified by work other than my own. (And see, also, Xuâs (2020) version of Hubingerâs argument, in the section on âcorrigibly aligned models.â Though: on a quick read, Xu seems to me to be focusing on the pre-situational-awareness goal- formation process, and assuming that basicallyanymisalignment post-situational-awareness leads to scheming, such that his is really a training-game-independentstory, rather than the sort of the training-game-dependent story Iâm focused on here.) 91 3.2.2.3 The relevance of messy goal-directedness to nearnessHowever, I also think that non- schemers get another, somewhat distinct nearness boost once we bring in considerations about âmessy goal-directedness.â In particular: if we imagine that the model in question is not a cleanly separable goal slot + goal-achieving-engine, but rather a tangled kludge of different heuristics, patterns of attention, impulses, and so on, then we need to analyze the comparative ânearnessâ not just of the current modelâs currentgoalto the nearest schemer-like and non-schemer-like goals, but the nearness of the modelâsentire kludgeto the nearest schemer and non-schemer-like models. And here, I think, non-schemer models may have a more significant advantage. In particular: to the extent that pre-situational-awareness training tends to create forms of messy goal-directedness pointed at somethingin the vicinityof a non-schemer goalâand in particular, a within-episodenon-schemer goalâthen this implies that the modelâs heuristics, patterns of attention, impulses, and so on will tend to be pointed in that broad direction as well, and focused within the episode in particular. And this, to me, makes it seem like a more substantive and difficult modification to re-orient the model towards longer-term power-seeking, especially if the episodes in question are quite short. That is, intuitively, it feels to me like âtweaking a cleanly separable within-episode goal to make it long-termâ is quite a bit easier, for SGD, than âredirecting a tangled kludge broadly focused on within-episode things to make it (a) focused on something beyond-episode instead, and (b) such that this beyond-episode focus back-chains into getting-reward for instrumental reasonsââespecially if (b) requires building new cognitive machinery for implementing the instrumental reasoning in question. Whereas âredirecting a tangled kludge focused on within-episode things to higher-reward within-episode things in the same broad vicinity insteadâ (e.g., to gold coins, rather than gold round things; or to actual honesty, rather than off-kilter honesty) seems to me intuitively easier. 3.2.3 Overall take on the ânearest max-reward goalâ argument Summarizing my take on the ânearest max-reward goalâ argument for expecting schemers: ⢠If SGD canât notice the benefits of turning the model into a schemer due to incremental modifications not yielding those benefits, then the argument fails. â˘However, I donât feel like we can bank on SGD being unable to notice these benefits. And if itcan, then I think the common-ness of schemer-like goals in goal-space makes the possibility that a schemer-like goal is ânearestâ to the modelâs current goal quite worrying. â˘That said, because it has been shaped by the reward process in the absence of situational awareness, the modelâs goal will also plausibly be already âin the vicinityâ of some max- reward non-schemer goal as well, which is points in favor of the ânearnessâ of some non-schemer model. And to the extent the goal-directedness at stake is quite âmessy,â creating such a non-schemer might require fewer modifications to the modelâs heuristics, patterns of attention, instrumental reasoning, and so on. These considerations leave me with some substantive additional worry about schemers. 3.3 The possible relevance of properties like simplicity and speed to the path SGD takes Arguments like the ânearest max-reward goal argumentâ proceed most naturally if we assume that any given max-reward goal yields sufficiently good performance in training that SGD isnât especially fussed aboutwhichmax-reward goal a model ends up with. But as I noted above, and as Iâl discuss in more detail in the section on final properties of the different model classes, there are stories on which SGD actively prefers some of these model classes over others, on grounds of final properties like simplicity and speed. Whatâs more, this sort of preference isnât just relevant to predictions that ignore the path that SGD takes in model space. Rather, they mightinfluencethat path at various stages (even if the final result is still âpath dependentâ in some broader sense). Plausibly, for example, if SGD biases towards simpler goals, this bias could influence the training-game-independent goals that a model tends to form prior to, or after, situational awareness, and on a training-game-dependent goal story, it might be additional points in favor of the SGDâs moving specifically to a schemer-like goal from a not-yet-max-reward 92 starting point. 218 And similarly, if SGD biases away from schemers in virtue of their requiring extra reasoning, this bias might be additional points in favor of expecting SGD to move towards a non-schemer from such a starting point. As Iâl discuss in section 4, on âfinal properties,â I generally think that the speed costs of scheming are more significant than the simplicity benefits, so other things equal, I think these sorts of considerations count against scheming. But itâs not clear to me that the costs/benefits in either direction are especially weighty. That said, thereâs at least some case to be made that these costs/benefits mattermoreearly on in training, because models are likely to beweakerearly on, and so reductions in the sorts of resources that simplicity and speed make available (e.g., parameters and compute) will have a larger overall effect on the modelâs cognitive power. That is: perhaps, early in training, cognitive resources are more scarce, and so more necessary to conserve. Thus, for example, perhaps requiring a model to form a long-term, schemer-like plan is a bigger ask (and a bigger hit to reward) when it has a smaller budget of instrumental reasoning in general available; or perhaps, requiring it to use more parameters storing a more complicated goal is more burdensome when fewer of its parameters have yet been shaped into useful cognitive structures. 219 So to the extent one was tempted by the view that these sorts of costs are likely to be âin the noiseâ relative to other considerations (a view Iâm tempted by, and which I discuss below), one might be less tempted by this with respect to early parts of training than with respect to a modelâs final properties. 3.4 Overall assessment of arguments that focus on the path SGD takes Overall, though, and despite the possible speed advantages of non-schemers, I find the combination of the âtraining-game-independent proxy goalsâ argument and the ânearest max-reward goal argumentâ fairly worrying. In particular: ⢠It seems plausible to me that despite our efforts at mundane adversarial training, and especially in a regime where we are purposefully shaping our models to have long-term and fairly ambitious goals, some kind of suitably ambitious, misaligned, beyond-episode goal might pop out of training naturallyâeither before situational awareness, or afterwardsâand then cause scheming to occur. â˘And even if thisdoesnâthappen naturally, I am additionally concerned that by the time it reaches situational awareness, the easiest way for SGD to give the model a max-reward goal will be to make it into a schemer, because schemer-like goals are sufficiently common in goal-space that theyâl often show up ânearbyâ whatever less-than-max-reward goal the model has at the time situational awareness arises. Itâs possible that SGDâs âincrementalismâ obviates this concern, and/or that we should expect non-schemer models to be ânearerâ by default (either because their goals in particular are nearer, or because, in a âmessy goal- directednessâ setting, they require easier modifications to the modelâs current tangled kludge of heuristics more generally, or because their âspeedâ advantages will make SGD prefer them). But I donât feel confident. Both these arguments, though, focus on thepaththat SGD takes through model space. What about arguments that focus, instead, on the final properties of the models in question? Letâs turn to those now. 218 At least if we understand simplicity in a manner thatadds somethingto the notion that schemer-like goals are common in goal-space, rather than merelydefiningthe simplicity of a goal (or: a type of goal?) via its common-ness in goal space. More on this sort of distinction in section 4.3.1 below. 219 I heard this sort of consideration from Paul Christiano. Prima facie, this sort of effect seems to me fairly symmetric between simplicity/parameters and speed/compute (and itâs unclear to me that this is even the right distinction to focus on), so I donât see early-training-dynamics asdifferentiallyfavoring one vs. the other as an important resource. 93 4 Arguments for/against scheming that focus on the final properties of the model Various arguments for/against scheming proceed by comparing the final properties of different model classes (e.g. schemers, training saints, reward-on-the-episode seekers, etc) according to how well they perform according to some set of criteria that we imagine SGD is selecting for. What is SGD selecting for? Well, one obvious answer is: high reward. But various of the arguments Iâl consider wonât necessarily focus on reward directly. Rather, theyâl focus on other criteria, like the âsimplicityâ or the âspeedâ of the resulting model. However, we can distinguish between two ways these criteria can enter into our predictions about what sort of model SGD will select. 4.1 Contributors to reward vs. extra criteria On the first frame, which Iâl call the âcontributors to rewardâ frame, we understand criteria like âsimplicityâ and âspeedâ as relevant to the model SGD selects only insofar as they are relevant to the amount of reward that a given model gets. That is, on this frame, weâre really only thinking of SGD as selecting for one thingânamely, high reward performanceâand simplicity and speed are relevant insofar as theyâre predictive of high reward performance. Thus, an example of a âsimplicity argument,â given in this frame, would be: âa schemer can have a simpler goal than a training saint, which means that it would be able to store its goal using fewer parameters, thereby freeing up other parameters that it can use for getting higher reward.â This frame has the advantage of focusing, ultimately, on something that weknowSGD is indeed selecting forânamely, high reward. And it puts the relevance of simplicity and speed into a common currencyânamely, contributions-to-reward. By contrast: on the second frame, which Iâl call the âextra criteriaâ frame, we understand these criteria as genuinelyadditionalselection pressures, operative even independent of their impact on reward. That is, on this frame, SGD is selectingbothfor high reward,andfor some other propertiesâfor example, simplicity. Thus, an example of a âsimplicity argument,â given in this frame, would be: âa schemer and a training saint would both get high reward in training, but a schemer can have a simpler goal, and SGD is selecting for simplicity in addition to reward, so we should expect it to select a schemer.â The âextra criteriaâ frame is closely connected to the discourse about âinductive biasesâ in machine learningâwhere an inductive bias, roughly, is whatever makes a learning process prioritize one solution over anotherother than the observed data(see e.g. Box 2 in Battaglia et al (2018) for more). Thus, for example, if two models would perform equally well on the training data, but differ in how they would generalize to an unseen test set, the inductive biases would determine which model gets selected. Indeed, in some cases, a model that performsworseon the training data might get chosen because it was sufficiently favored by the inductive biases (as analogy: in science, sometimes a simpler theory is preferred despite the fact that it provides a worse fit with the data). In this sense, inductive biases function as âextra criteriaâ that matter independent of reward. 220 Ultimately, the differences between the âcontributors to rewardâ frame and the âextra criteriaâ frame may not be important. 221 But I think itâs worth tracking which weâre using regardless, because at least naively, they point in somewhat different argumentative directions. In particular, in a âcontributions 220 As an example where you might wonder whether such extra criteria at work, consider âepoch-wise double descent,â in which as you train a model of a fixed size, you eventually get to a regime of zero training error (see the bit above and to the right of the âinterpolation thresholdâ in the right-hand graph below). But at that point, the test error is actually high. Then, if you train more, the test error eventually goes down again. That is, there are multiple models that all get zero training error, and somehow training longer eventually lets you find the model that generalizes better. And one diagnosis of this dynamic is that weâre giving SGDâs inductive biases more time to work. 221 In discussion, Hubinger argued to me that âwhich model gets highest reward, holding the inductive biases fixedâ and âwhich model does best on the inductive biases, holding the loss fixedâ are just dual perspectives on the same question, at least if you get the constants right. But Iâm not yet convinced that this makes the distinction irrelevant to which analytic approach we should take. For example: suppose that someone is buying a house, and we know that they are employing a process that optimizes very hard and directly for the cheapest house. But suppose, also, that they have some other set of poorly understood criteria that come into play as well in some 94 to rewardâ frame, the question of âwhich model gets highest rewardâsettlesthe question of which one we should expect SGD to prefer (at least modulo noise, slack in training, etc). Whereas, by contrast, an âextra criteriaâ frame leaves that question more open; it forces us to grapple with what sorts of âextra criteriaâ SGD might be giving weight to (and how much weight); it connects more directly to the theoretical discourse about inductive biases; and it raises more questions about whether we may be able toinfluencethose biases in favorable directions. 4.2 The counting argument Letâs turn, now, to a basic argument for expecting schemers that focuses on the final properties of different model classes: namely, the âcounting argument.â 222 This argument is closely related to the âcommon-ness in goal spaceâ consideration discussed in the context of the ânearest max-reward goalâ argumentâbut itâs here presented without reference to the path SGD takes. We start by considering some different model classes that would get high reward in training (in my ontology, these would be: training saints, mis-generalized non-training-gamers pursuing some high reward goal, reward-on-the-episode-seekers, and schemers). We then argue as follows: 1. The non-schemer model classes, here, require fairly specific goals. a.Thus, for example, if youâre rewarding your model when it gets gold-coins-on-the- episode, then to be a training saint, it needs to value gold-coin-on-the-episode. b.To be a mis-generalized non-training-gamer, it needs to value something pursuit of which very closely correlates with getting gold-coins-on-the-episode, even absent training-gaming. c.And to be a reward-on-the-episode seeker, it needs to terminally value reward-on-the- episode. 2.By contrast, the schemer model class is compatible with a very wide range of (beyond- episode) goals (at least if we assume that the other requirements for scheming to make sense as an instrumental strategy are in placeâe.g., that the classic goal-guarding story, or some alternative, works). a. Thus, for example, the model can value paperclips over all time, it can value staples over all time, it can value happiness over all time, and so on. 3.In this sense, there are âmoreâ schemers that perform well in training than there are non- schemers that do so. 223 4. So, other things equal, we should expect SGD to select a schemer. In a sense, this is an extension of one of the most basic concerns about misalignment more generally: namely, that the class of aligned goals is very narrow, whereas the class of misaligned goals is very broad, so creating an aligned model requires âhitting a narrow target,â which could be hard. Naively, this basic argument suffers from neglecting the relevance of our selection power (compare: âmost arrangements of car parts arenât a car, therefore it will be very difficult to build a carâ 224 ), and so it needs some further claim about why our selection power will be inadequate. The counting argument, above, is a version of this claim. In particular, it grants that weâl narrow down to the set of models that get high reward, but argues that,still, thenon-schemerswho get high reward are a much narrower class than the schemers who get high reward (and non-schemers arenât necessarily aligned poorly-understood way (maybe as a tie-breaker, maybe as some other more substantive factor). In trying to predict what type of house they bought, should you focus on the price, or on how the houses do on the hazy other criteria? My current feeling is: price. Also, I think the âcontributors to rewardâ frame may be best understood as effectively setting aside the question of inductive biases altogether, which seems like it could be more importantly distinct. 222 See e.g. Hubinger (2022b) and Xu (2020) for examples of this argument. 223 Of course, the âspace of possible goalsâ isnât very well-defined, hereâand in abstract, it seems infinite in a way that requires an actual measure rather than a âcount.â Iâm here using âcountâ as a loose approximation for this sort of measure (though note that on a real-world computer, the actual set of possible neural network parameter settings will be finite in any given caseâand so would accommodate a more literal âcount,â if necessary). Thanks to Hazel Browne for discussion. 224 This is an example I originally heard from Ben Garfinkel. 95 anyway). 225 So unless you can say somethingfurtherabout why you expect to get a non-schemer, schemers (the argument goes) should be the default hypothesis. To the extent we focus specifically on the final properties of different model classes, some argument in this vicinity accounts for a decent portion of my credence on SGD selecting schemers (and as Iâl discuss more in section 5, I think itâs actually what underlies various other more specific arguments for expecting schemers as well). However, the argument I give most weight to doesnât move immediately from âthere are more possible schemers that perform well in training than non-schemers that do soâ to âabsent further argument, SGD probably selects a schemerâ (call this the âstrict counting argumentâ). And the reason is that itâs not yet clear to me how to make sense of this inference. In particular, the most natural construal of this inference proceeds by assuming that for whatever method of counting âindividual modelsâ (not model classes) results in there being âmoreâ schemers- that-get-high-reward than non-schemers-that-get-high-reward, each of these individual models gets the same reward, and performs equally well on whatever âextra criteriaâ SGDâs inductive biases care about, such that SGD is equally likely to select any given one of them. That is, we assume that SGDâs selection process mimics a uniform distribution over these individual modelsâand then note that schemers, as a class, would get most of that probability. But given that these model classes are different in various respects that might matter to SGD, itâs not clear to me that this is a good approximation. Alternatively, we might say something more like: âperhaps some of these individual models actually get more reward, or perform better on SGDâs inductive biases, such that SGD actually does favor some of these individual models over others. However, we donât knowwhichmodels SGD likes more: so, knowing nothing else, weâl assume that theyâre all equally likely to be favored, thereby leading to most of the probability going to some schemer being favored.â However, if we assume instead that one of thosemodel classes, as a whole, gets more reward, and/or performs better on SGDâs inductive biases, then itâs less clear how the ânumber of individual modelsâ within a given class should enter into our calculation. Thus, as an analogy: if you donât know whether Bob prefers Mexican food, Chinese food, or Thai food, then itâs less clear how the comparative numberof Mexican vs. Chinese vs. Thai restaurants in Bobâs area should bear on our prediction of which one he went to (though it still doesnât seem entirely irrelevant, eitherâfor example, more restaurants means more variance in possible qualitywithinthat type of cuisine). E.g., it could be that there are ten Chinese restaurants for every Mexican restaurant, but if Bob likes Mexican food better in general, he might just choose Mexican. So if we donâtknowwhich type of cuisine Bob prefers, itâs tempting to move closer to a uniform distributionover types of cuisine, rather than over individual restaurants. My hesitation here is related to a common way for âcounting argumentsâ to go wrong: namely, by neglecting the full selection power being applied to the set of things being counted. (Itâs the same way that âyouâl never build a working car, because almost every arrangement of car parts isnât a working carâ goes wrong.) Thus, as a toy example: suppose that there are2 100 schemer-like goals for every non-schemer goal, such that if SGD was selecting randomly amongst them (via a uniform distribution), it would be2 100 times more likely to select a schemer than a non-schemer. Naively, this might seem like a daunting prior to overcome. But now suppose that a step of gradient descent can cut down the space of goals at stake by at least a factor of 2âthat is, each step is worth at least a âbitâ of selection power. This means that selecting a non-schemer over a schemer only needs to be worth 100 extra steps of gradient descent, to SGD, for SGD to have an incentive to overcome the prior in question. 226 And 100 extra gradient steps isnât all that many in a very large training run (though of course, I just made up the2 100 : 1ratio, here). 227 (Whatâs more, as Iâl discuss below in the context 225 Mark Xu gives a related argument: namely, âFor instrumental reasons, any sufficiently powerful model is likely to optimize for many things. Most of these things will not be the modelâs terminal objective. Taking the dual statement, that suggests that for any given objective, most models that optimize for that objective will do so for instrumental reasons.â In effect, this is a counting argumentapplied to the different things that the model is optimizing for, rather than across model classes. 226 Thanks to Paul Christiano for discussion, here. As Iâl discuss in section 5, there are analogies, here, with the sense in which âStrong Evidence is Commonâ in Bayesianismâsee Xu (2021). 227 For example, my understanding from a quick, informal conversation with a friend is that training a model with more than a trillion parameters might well involve more than a million gradient updates, depending on the batch size. However, I havenât tried to dig in on this calculation. 96 of the âspeed costsâ of scheming, I think itâs plausible that it would indeed be âworth itâ for SGD to pay substantive costs to get a non-schemer instead. And I think this is a key source of hope. More in section 4.4.) Partly due to this hesitation, the counting argument functions in my own head in a manner that hazily mixes together the âstrict counting argumentâ with some vaguer agnosticism about which model classSGD likes most. That is, the argument in my head is something like: 1.It seems like there are âlots of waysâ that a model could end up a schemer and still get high reward, at least assuming that scheming is in fact a good instrumental strategy for pursuing long-term goals. 2.So absent some strong additional story about why trainingwonâtselect a schemer, it feels, to me, like the possibility should be getting substantive weight. Call this the âhazy counting argument.â Here, the ânumberâ of possible schemers isnât totally irrelevantârather, it functions to privilege the possibility of scheming, and makes it feel robust along at least one dimension. Thus, for example, I wouldnât similarly argue âit seems like in principle the model could end up instrumentally training-gaming because it wants the lab staff members who developed it to get raises, so absent some additional story about why thiswonâthappen, it feels like the possibility should be getting significant weight.â The fact that many different goals lead to scheming matters to its plausibility. But at the same time, the exercise of counting possible goals doesnât translate immediately into a uniform distribution over âindividual models,â because the differences between modelclassesplausibly matter too, even if I donât know exactly how. To be clear: the âhazy counting argumentâ is unprincipled and informal. Iâd love a better way of separating it into more principled components that can be analyzed separately. For now, though, itâs the argument that feels like it actually moves me. 228 4.3 Simplicity arguments The strict counting argument Iâve described is sometimes presented in the context of arguments for expecting schemers that focus on âsimplicity.â 229 Letâs turn to those arguments now. 4.3.1 What is âsimplicityâ? What do I mean by âsimplicity,â here? In my opinion, discussions of this topic are often problemati- cally vagueâboth with respect to the notion of simplicity at stake, and with respect to the sense in which SGD is understood as selecting for simplicity. The notion that Hubinger uses, though, is the length of the code required to write down the algorithm that a modelâs weights implement. That is: faced with a big, messy neural net that is doing X (for example, performing some kind of induction), we imagine re-writing X in a programming language like python, and we ask how long the relevant program would have to be. 230 Letâs call this âre-writing simplicity.â 231 Hubingerâs notion of simplicity, here, is closely related to measures of algorithmic complexity like âKolmogorov complexity,â which measure the complexity of a string by reference to the length of the shortest program that outputs that string when fed into a chosen Universal Turing Machine (UTM). One obvious issue here is that this sort of definition is relative to the choice of UTM (just as, e.g., when we imagine re-writing a neural netâs algorithm using other code, we need to pick 228 Though: one concern about my introspection here is that really, whatâs going on is that the possibility of SGD selecting schemers has been made salient by thediscourseabout misalignment Iâve been exposed to, such that my brain is saying âabsent some additional story about why training wonât select a schemer, the possibility should be given substantive weightâ centrally because my epistemic environment seems to take the possibility quite seriously, and my brain is deferring somewhat to this epistemic environment. 229 See e.g. Hubinger (2022b). 230 See also this (now anonymous) discussion for another example of this usage of âsimplicity.â 231 Here, my sense is that the assumption is generally that X can be described at a level of computational abstraction such that the âre-writingâ at stake doesnât merely reproduce the network itself. E.g., the network is understood as implementing some more abstract function. I think itâs an interesting question how well simplicity arguments would survive relaxing this sort of assumption. 97 the programming language). 232 Discussions of algorithmic complexity often ignore this issue on the grounds that it only adds a constant (since any given UTM can mimic any other if fed the right prefix), but itâs not clear to me, at least, when such constants might or might not matter to a given analysisâfor example, the analysis at stake here. 233 Indeed, my vague sense is that certain discussions of simplicity in the context of computer science often implicitly assume what Iâve called âsimplicity realismââa view on which simplicity in some deep sense an objectivething, ultimately independent of e.g. your choice of programming language or UTM, but which different metrics of simplicity are all tracking (albeit, imperfectly). And perhaps this view has merit (for example, my impression is that different metrics of complexity often reach similar conclusions in many casesâthough this could have many explanations). However, I donât, personally, want to assume it. And especially absent some objective sense of simplicity, it becomes more important to say which particular sense you have in mind. Another possible notion of simplicity, here, is hazierâbut also, to my mind, less theoretically laden. On this notion, the simplicity of an algorithm implemented by a neural network is defined relative to something like the number of parameters the neural network uses to encode the relevant algorithm. 234 That is, instead of imaginingre-writingthe neural networkâs algorithm in some other programming language, we focus directly on the parameters the neural network itself is recruiting to do the job, where simpler programs use fewer parameters. Letâs call this âparameter simplicity.â Exactly how you would measure âparameter simplicityâ is a different question, but it has the advantage of removing one layer of theoretical machinery and arbitrariness (e.g., the step of re-writing the algorithm in an arbitrary-seeming programming language), and connecting more directly with a âresourceâ that we know SGD has to deal with (e.g., the parameters the model makes available). For this reason, Iâl often focus on âparameter simplicityâ below. Iâl also flag a way of talking about âsimplicityâ that I wonât emphasize, and which I think muddies the waters here considerably: namely, equating simplicity fairly directly with âhigher prior probability.â Thus, for example, faced with an initial probability distribution over possibilities, itâs possible to talk about âsimpler hypothesesâ as just: the ones that have greater initial probability, and which therefore require less evidence to establish. For example: faced with a thousand people in a town, all equally likely to be the murderer, itâs possible to think of âthe murderer is a manâ as a âsimplerâ hypothesis than âthe murderer is a man with brown hair and a dog,â in virtue of the fact that the former hypothesis has, say, a 50% prior, and so requires only one âbitâ of evidence to establish (i.e., one halving of the probability space), whereas the latter hypothesis has a much smaller prior, and so requires more bits. Letâs call this âtrivial simplicity.â âTrivial simplicityâ is related to, but distinct from, the use of simplicity at stake in âOccamâs razor.â Occamâs razor is (roughly) thesubstantiveclaim thatgiven an independent notion of simplicity, simpler hypotheses are more likely on priors. Whereas trivial simplicity would imply that simpler hypotheses areby definitionmore likely on priors. If you take Occamâs razor sufficiently for granted, itâs easy to conflate the twoâbut the former is interesting, and the latter is some combination of trivial and misleading. And regardless, our interest here isnât in the simplicity ofhypotheseslike âSGD selects a schemer,â but in the simplicity of thealgorithmthat the model SGD selects implements. 235 232 Another issue is that Kolmogorov complexity is uncomputable. Iâm told you can approximate it, but Iâm not sure how this gets around the issue that for a given program where youâre not able to tell whether or not it halts, that program might be the shortest program outputting the relevant string. 233 See Carlsmith (2021), sections I and IV, for more on this. 234 Hubinger sometimes appears to be appealing to this notion as wellâor at least, not drawing clear distinctions between âre-writing simplicityâ and âparameter simplicity.â 235 âTrivial simplicityâ is also closely related to what we might call âselection simplicity.â Here, again, one assumes some space/distribution over possible things (e.g., goals), and then talks about the âsimplicityâ of some portion of that space in terms of how much âworkâ one needs to do (perhaps: on average) in order to narrow down from the whole space to that portion of the space (see also variable-length codes). Thus, for a box of gas, âthe molecules are roughly evenly spread outâ might be a âsimplerâ arrangement than âthe molecules are all in a particular corner,â because it typically takes more âworkâ (in this example: thermodynamic work) to cause the former than the latter (this is closely related to the fact that the former is initially more likely than the latter). My sense is that when some people say that âschemer-like goals are simple,â they mean something more like: thesetof schemer-like goals typically takes less âwork,â on SGDâs part, to land within than theset of non-schemer-like goals (and not necessarily: that anyparticularschemer-like goals is simpler than some particularnon-schemer-like goal). To the extent that the set of schemer-like goals are supposed to have this 98 4.3.2 Does SGD select for simplicity? Does SGD select for simplicity in one of the non-trivial senses I just described? One reason you might think this comes from the âcontributors to rewardâ frame. That is: using a more parameter-simple algorithm will free up other parameters to be put to other purposes, so it seems very plausible that parameter simplicity will increase a modelâs reward. And to the extent that re-writing simplicity correlates with parameter simplicity, the same will hold for re-writing simplicity as well. This is the story about why simplicity matters that I find most compelling. However, I think there may also be more to say. For example, I think itâs possible that thereâs other empirical evidence that SGD selects for simpler functions, other things equal (for example, that it would much sooner connect a line-like set of dots with a straight line than with an extremely complicated curve); and perhaps, that this behavior is part of what explains its success (for example, because real-world functions tend to be simple in this sense, Ă la Occamâs razor). For example, in the context of an understanding of SGD as an approximation of Bayesian sampling (per the discussion of Mingard et al (2020) above), Mingard (2021) discusses empirical evidence that thepriorprobability distribution over parameters (e.g., what I called the âinitialization distributionâ above) puts higher probability mass on simpler functions. 236 And he connects this with a theoretical result in computer science called the âLevin bound,â which predicts this (for details in footnote). 237 I havenât investigated this in any depth. If accurate, though, this sort of result would give simplicity relevance from an âextra criteriaâ frame as well. That is, on this framework, SGD biases towards simplicity even before we start optimizing for reward. Letâs suppose, then, that SGD selects for some non-trivial sort of simplicity. Would this sort of selection bias in favor of schemers? 4.3.3 The simplicity advantages of schemer-like goals Above I mentioned that the counting argument is sometimes offered as a reason to expect a bias towards schemers on these grounds. Note, though, that the counting argument (at least as Iâve presented it) doesnât make any obvious reference to a bias towards simplicity per se. And I think we should be careful not to conflate the (trivial) simplicity of thehypothesisthat âSGD selects a schemer,âgiven a prior probability distribution that puts most of the probability on schemers(e.g., a property because they are more âcommon,â and hence ânearerâ to SDGâs starting point, this way of talking about the simplicity benefits of scheming amounts to a restatement of something like the counting argument and/or the ânearest max-reward goal argumentââexcept, with more of a propensity, in my view, to confuse the simplicity ofsetof schemer-like goals with the simplicity of agivenschemer-like goal. 236 Where, importantly, multiple different settings of parameters can implement the same function. 237 My understanding is that the Levin bound says something like: for a given distribution over parameters, the probabilityp(f)of randomly sampling a set of parameters that implements a functionfis bounded by 2 âK(f)+O(1) , whereKis thek-complexity of the functionf, andO(1)is some constant independent of the function itself (though: dependent on the parameter space). That is, the prior on some function decreases exponentially as the functionâs complexity increases. I havenât investigated this result, but one summary I saw (here) made it seem fairly vacuous. In particular, the idea in that summary was that larger volumes of parameter space will have simpler encodings, because you can encode them by first specifying distribution over parameters, and then using a Huffman code to talk about how to find them given that distribution. But this makes the result seem pretty trivial: itâs not that there is some antecedent notion of simplicity, which we then discover to be higher-probability according to the initialization distribution. Rather, to be higher probability according to the initialization distribution justisto be simpler, because equipped with the initialization distribution, itâs easier to encode the higher probability parts of it. Or put another way: it seems like this result applies to any distribution over parameters. So it doesnât seem like we learn much about any particular distribution from it. (To me it feels like there are analogies here to the way in which âshorter programs get more probability,â in the context of algorithmic âsimplicity priorsâ that focus on metrics like K-complexity, actually applies necessarily to anydistribution over a countably-infinite set of programsâsee discussion here. You mightâve thought it was an interesting and substantive constraint, but actually it turns out to be more vacuous.) That said, the empirical results I mention above focus on more practical, real-world measures of simplicity, like LZ complexity, and apparently they find that, indeed, simpler functions get higher prior probability (see e.g. this experiment, which uses a fully connected neural net to model possible functions from many binary inputs to a single binary input). This seems to me more substantive and interesting. And Mingard (2021) claims that Levinâs result is non-trivial, though I donât yet understand how. 99 uniform distribution over individual models-that-get-high-reward), with the claim that thealgorithm that a given individual schemer implements is (substantively) simpler than the algorithm that a given non-schemer implements. 238 Indeed, my own sense is that the strongest form of the counting argument leaves it to stand on its own intuitive terms, rather than attempting to connect it to further questions about SGDâs biases towards simplicity in particular. That said, it is possible to draw connections of this form. In particular: we can say thatbecausesuch a wide variety of goals can motivate scheming, schemers allow SGD a very wide range of goals to choose from in seeking out simpler goals; whereas non-schemers do not. And this seems especially plausible to the extent we imagine that the goals required to be a non-schemer are quite complex (more on this below). 239 One interesting feature of this sort of argument is that it imagines, specifically, that the simplicity differences between models are coming entirely from the content of theirgoals. Indeed, the toy analysis in Hubinger (2022) specifically imagines that the respective model classes all have the same world model and optimization procedure, and that the complexity of their algorithm overall can be approximated bycomplexity of world model + complexity of the optimization procedure + complexity of the goal.And the âgoal slotâ is the only part that differs between models. Itâs not clear that this is right, though, especially if we assume that the goal-directedness at stake is âmessyâ rather than âclean.â For example, to the extent that schemers have to perform types of instrumental reasoning that non-schemersdonât(e.g., reasoning about the instrumental value of getting reward, reasoning about when to defect, etc), it seems plausible that this could introduce additional complexity into the algorithm itself (rather than e.g. merely requiring that the algorithm ârun for a longer time,â Ă la the âspeedâ analysis below). For example, to the extent weâre using âparameter simplicityâ as our notion of simplicity, we could imagine cases where this sort of instrumental reasoning requires additional parameters. 240 4.3.4 How big are these simplicity advantages? For now, though, letâs stick with Hubingerâs ontology, and with simplicity differences rooted specifi- cally in differences between goals. How big of an advantage does selecting a schemer afford in this respect? One way of running this analysis is to compare the goals had by the simplest possible model within each class (either: because you expect SGD to select for the simplest possible model, or you think this is a good way of approximating the simplicity benefits at stake). 241 That is, we compare the complexity of: 238 Thus, for example, you might think that insofar a randomly initialized model is more likely to end up âcloserâ to a schemer, such that SGD needs to do âless workâ in order to select a schemer rather than some other model, this favors schemers (thanks to Paul Christiano for discussion). But this sort of argument rests on putting a higher prior probability on schemers, which, in my book, isnât a (non-trivial) simplicity argument per se. 239 There are also more speculative and theoretical arguments for a connection between simplicity and schemers, on which one argues that if you do an unbounded search over all possible programs to find the shortest one that gives a given output, without regard to other factors like how long they have to run, then youâl select for a schemer (for example, via a route like: simulating an extremely simple physics that eventually gives rise to agents that understand the situation and want to break out of the simulation, and give the relevant output as part of a plan to do so). My understanding is that people (e.g. here) sometimes take the discourse about the âmalignity of the Solomonoff priorâ as relevant here (though at a glance, it seems to me like there are important differencesâfor example, in the type of causality at stake, and in the question of whether the relevant schemer might be simulatingyou). Regardless, Iâm skeptical that these unbounded theoretical arguments should be getting much if any weight, and I wonât treat them here. 240 Whatâs more, note that, to the extent we imagine SGD biasing towards simplicitybecausereal world patterns tend to be simple (e.g., Occamâs razor is indeed a good prior, and SGD works well in part because it reflects this prior), the explanation for this bias doesnât apply as readily to a modelâsgoals. That is (modulo various forms of moral realism), there are no âtrue goals,â modeling of which might benefit from a simplicity prior. Rather, on this story, SGD would need to be acting more like a human moral anti-realist who prefers a simpler morality other-things-equal, despite not believing that there is any objective fact of the matter, because, in contexts where thereisa fact of the matter, simpler theories tend to be more likely. 241 Hubinger uses this approach. My understanding is that heâs imagining SGD selecting a model with probability proportionate to its simplicity, such that e.g. focusing on the simplest possible model is one way of approximating the overall probability in a model class, and focusing on thenumberof models in the class 100 1. The specified goal (for training-saints) 2.The simplest possible misgeneralized goal that gets max-reward on the training data absent training-gaming (for mis-generalized non-training-gamers) 3. The simplest possible way of seeking reward-on-the-episode (for reward-on-the-episode seekers) 4. The simplest possible goal-that-motivates-scheming (for schemers) Clearly, the complexity of the specified goal matters in this context. My sense is that the discourse on scheming often assumes that the specified goal is quite complex in some senseâand in particular, the sense at stake in the idea that âhuman values are complicated.â 242 And perhaps, if weâre imagining that the only way to getalignmentis to first (a) somehow specify âhuman valuesâ via the training objective, and then (b) somehow ensure that we get a training saint, then focusing on something in the vicinity of âact in accordance with human valuesâ as the specified goal is appropriate. But note that for the purposes of comparing the probability ofschemingto the probability ofother forms of misalignment, we need not assume such a focus. And thus, our specified goal might be much simpler than âact in accordance with human values.â It might, for example, be something like âget gold coins on the episode.â Indeed, in other work, Hubinger (writing with others) suggests that a goal like âminimize next-token prediction errorâ is quite simpleâand indeed, that âits complexity is competitive with the simplest possible long-term goalsâ (this is part of what makes Hubinger comparatively optimistic about avoiding scheming during LLM pre-trainingâthough personally, I feel confused about why Hubinger thinks ânext token prediction errorâ importantly simpler than âreward-on-the-episodeâ). Suppose, though, that the specified goal has complexity on the order of âact in accordance with human valuesâ or âfollow instructions in a helpful, harmless, and honest (H) manner.â Where does this leave a comparison between (1)-(4) above? At least from a parameter-simplicity perspective, one way of approaching this is to think about what we know about the absolute cost in parameters for representing different human concepts within neural networks. I wonât dig in much, here, but one piece of data that seems relevant is: LLMs like GPT-4 seem capable of representing a very large number of complex human concepts with e.g. order of a trillion parameters - including, plausibly, concepts like âhonesty,â âhelpfulness,â âreward,â and so on. And this isnât yet to mention all the parameters devoted to everything else that GPT-4 does. So the absolute number of parameters required to represent concepts like these canât be crazily high. 243 And this caps the overall complexity savings at stake in the differences between the models above. For example, if we assume that at most 1% of a trillion-parameter modelâs capacity goes to representing concepts as complex as âhonesty,â and that it knows at least 10,000 such concepts (Websterâs unabridged dictionary has ~500,000 words), then the maximum number of parameters required to represent the concept of âhonestyâ is around a million (and this estimate strikes me as quite conservative). So if the simplest possible schemer-like goal takes only 1 parameter to represent, then not representing honesty in the schemerâs âgoal slot,â saves, at most, 999,999 parametersâroughly one millionth of a trillion parameter modelâs representational capacity, and even less for the larger models of the future. More importantly, though: no one thinks that a schemer wonât have to represent concepts like âhonesty,â âhelpfulness,â âreward,â and so on at all. As Hubinger et al (2023) note, what matters here isnât the absolute complexity of representing the different goals in question, but the complexity conditional on already having a good world model. And we should assume thatallof these models will need to understand the specified goal, the reward process, etc (and especially: models that are âplaying a training gameâ in which such concepts play a central role). So really, the relevant question is: what are theextracomplexity costs of representing a goal like âget reward-on-the-episodeâ or âfollow instructions in an H wayâ (relative to the simplest possible schemer-like goal),once youâve already paid the costs of having a concept of those goal targets. Iâm not sure exactly how to think about this, but it seems very plausible to me that the costs here are extremely small. In particular: it seems like SGD may be able to significantly repurpose the parameters used to represent the concept in the world model in causing that concept to guide the is another. However, I wonât take for granted the assumption that SGD selects a model with probability proportionate to its simplicity. 242 See e.g. Hubinger et al (2023) here. 243 I first heard this sort of point from Paul Christiano. 101 modelâs behavior in a goal-like manner. Thus, as an analogy, perhaps the concept of âpleasureâ is in some sense âsimplerâ than the concept of âwabi-sabiâ in Japanese aesthetics (i.e., âappreciating beauty that is âimperfect, impermanent, and incompleteâ â). Once youâvelearnedboth, though, does pursuing the former require meaningfully more parameters than pursuing the latter? 244 Hubinger (2022b) discussion of issues like this sometimes appeals to the notion of a âpointerâ to some part of the world model. As I understand it, the idea here is that if youâve already got a concept of something like âpleasureâ/âwabi-sabiâ/ârewardâ in your world model, you can cause a model to pursue that thing by giving it a goal slot that says something like âgo forthatâ or âthatis good,â where âthatâ points to the thing in question (this is in contrast with having to represent the relevant conceptagain,fully and redundantly, in the goal slot itself). But insofar as we use a toy model like this (I doubt we should lean on it), why think that itâs significantly more complex topointat a more complex concept than at a simpler one? E.g., even granted that âwabi-sabiâ takes more parameters than âpleasureâ to represent in the world model, why think that encoding thepointerto âpleasureâ (e.g., âgo forthatâ) takes more parameters than encoding thepointerto âwabi-sabiâ (e.g., again, âgo forthatâ)? One option, here, is to say that the complexity of the concept and the complexity of the pointer are correlated. For example, you might imagine that the model has some kind of âinternal databaseâ of concepts, which stores concepts in a manner such that concepts that take fewer parameters to store take fewer parameters to âlook upâ as well. 245 On this picture, âpleasureâ might end up stored as the 15th concept in the databasebecauseit takes e.g. 23 parameters to represent, whereas âwabi-sabiâ might end up stored as the 125355th concept because it takes 10,000 parameters to represent. And then the âpointerâ to pleasure can say âgo for the thing stored at location 15,â whereas the âpointerâ to âwabi-sabiâ has to say âgo for the thing stored at location 125355,â which takes a few more bits to specify. But even at an abstract-toy-illustrative-example level, this sort of story requires leaning on a specific model of how the modelâs pointer and concept-storage processes workâand it still needs to explainwhysimplicity-to-represent and simplicity-to-point-at are correlated in the relevant sense. Alternatively, though, we can abandon any interest in the complexity of storing a concept in the world model, and focus directly on the complexity of pointing to it. Still, insofar as there are meaningful differencesbetween the complexity of pointing at one concept vs. another, we may be able to re-run the argument that schemers offer simplicity advantages. In particular: selecting a schemer allows SGD to have its pick from whatever schemer-motivating goals are simplest topoint at in the world model; whereas the other model classes plausibly impose more substantive constraints. I.e., if the specified goal ends up stored at location 12634 in the modelâs metaphorical database, and if âreward-on-the-episodeâ is at location 35364, then if there are any schemer-like goals at e.g. locations 1-100, itâl be simpler to point at one ofthoseinsteadâand thus, to create a schemer rather than a training-saint or a reward-on-the-episode seeker. To the extent we focus on the final properties of the different model classes, I think this is probably the best way to run a simplicity-focused argument for schemingâespecially if we donât get too hung up on the toy ontology of âpointersâ (and still less, âdatabasesâ) in particular. That is, roughly: even granted that all of the goals on the table here (e.g. the specified goal, reward-on-the-episode, etc) will need to be represented by the model somewhere, it may be that, for whatever reason, some of these concepts take fewer parameters torepurpose for goal-pursuitthan others. And insofar as scheming makes more potential concepts available for such repurposing, it offers a simplicity advantage in that sense. Quantitatively, though, how big is this advantage? Itâs now especially hard to say. In particular, once weâve shifted our focus from âsimplicity-to-representâ to âsimplicity-to-repurpose-for-goal-pursuit,â we canât even look at our intuitive sense of the complexity of concepts like âreward,â âhonesty,â and ânext token prediction error,â because weâre no longer talking about the complexity of the concepts per se. Rather, weâre speculating about the complexity differences at stake in repurposing pre-existing conceptual representations for use in a modelâs motivational system, which seems like even more uncertain territory. 244 Here I donât mean: does it take more parameters tosuccessfullypromote pleasure vs. successfully promoting wabi-sabi. I just mean: does it take more parameters toaimoptimization at the one vs. the other. 245 Thanks to Daniel Kokotajlo for suggesting an image like this. 102 Still, to the extent we can estimate the size of these differences, it seems plausible to me that they are very small indeed. One intuition pump for me here runs as follows. Suppose that the model has2 50 concepts (roughly 1e15) in its world model/âdatabaseâ that could in principle be turned into goals. 246 The average number of bits required to code for each of2 50 concepts canât be higher than 50 (since: you can just assign a different 50-bit string to each concept). So if we assume that modelâs encoding is reasonably efficient with respect to the average, and that the simplest non-schemer max-reward goal is takes a roughly average-simplicity âpointer,â then if we allocate one parameter per bit, pointing at the simplest non-schemer-like max-reward goal is only an extra 50 parameters at maximumâone twenty-billionth of a trillion-parameter modelâs capacity. That said, I expect working out the details of this sort of argument to get tricky, and I wonât try to do so here (though Iâd be interested to see other work attempting to do so). 4.3.5Does this sort of simplicity-focused argument make plausible predictions about the sort of goals schemers would end up with? One other consideration that seems worth tracking, in the context of simplicity arguments for scheming, is the predictions they are making about the sort of goals a schemer will end up with. In particular, if you think (1) that SGD selects very hard for simpler goals, (2) that this sort of selection favors schemer-like goals because they can be simpler, and (3) that our predictions about what SGD selects can ignore the âpathâ it takes to create the model in question, then at least naively, it seems like you should expect SGD to select a schemer with an extremely simple long-term goal (perhaps: the simplest possible long-term goal),regardless of whether that goal had any relation to what was salient or important during training. Thus, as a toy example, if âmaximize hydrogenâ happens to be the simplest possible long-term goal once youâve got a fully detailed world model, 247 these assumptions might imply a high likelihood that SGD will select schemers who want to maximize hydrogen, even if training was all about gold coins, and never made hydrogen salient/relevant as a point of focus at all (even as a proxy). 248 Personally, I feel skeptical of predictions like this (though this skepticism may be partly rooted in skepticism about ignoring the path SGD takes through model space more generally). And common stories about schemers tend to focus on proxy goals with a closer connection to the training process overall (e.g., a model trained to on gold-coin-getting ends up valuing e.g. âget gold stuff over all timeâ or âfollow my curiosity over all time,â and not âmaximize hydrogen over all timeâ). Of course, itâs also possible to posit that goal targets salient/relevant during training will also be âsimplerâ for the model to pursue, perhaps they will either be more important (and thus simpler?) to represent in the world model, or simpler (for some reason) for the model to repurpose-for-goal-pursuit once represented. 249 But if we grant some story in this vein, we should also be tracking its relevance to the simplicity of pursuingnon-schemer goalsas well. In particular: to the extent weâre positing that salience/relevance during training correlates with simplicity in the relevant sense, this is points in favor of the simplicity of the specified goal, and of reward-on-the-episode, as well - since these are especiallysalient/relevant during the training process. (Though of course, insofar as there are still simplerschemer-like goal targets that were salient/relevant during training, schemer-like goals might still win out overall.) And note, too, that to the extent SGD selects very hard for simpler goals (for example, in the context of a form of âlow path dependenceâ that leads to strong convergence on a single optimal sort of model), this seems somewhat at odds with strong forms of the goal-guarding hypothesis, on which training-gaming causes your goals to âcrystallize.â For example, if a would-be-schemer starts out with a not-optimally-simple goal that still motivates long-term power-seeking, then if it knows that in fact, SGD will continue to grind down its goal into something simpler even after it starts training-gaming, 246 The precise number of concepts here doesnât matter much. 247 Iâm not saying it is, even for a physics-based world model, but I wanted an easy illustration of the point. Feel free to substitute your best-guess simplest-possible-goal here. 248 Notably, this sort of prediction seems like an especially poor fit for an analogy between humans and evolution, since human goals seem to have a very intelligible relation to reproductive fitness. But evolution is plausibly quite âpath-dependentâ anyway. 249 E.g., plausibly âhydrogenâ doesnât read as a simple concept for humans, but concepts like âthreatâ do, because the latter was much more relevant in our evolutionary environment. 103 then it may not have an incentive to start training-gaming in the first placeâand its goals wonât survive the process regardless. 250 4.3.6 Overall assessment of simplicity arguments Overall, I do think that other things equal, schemers can have probably simpler goals than these other model classes. However, I think the relevant simplicity differences may be quite small, especially once we condition on the model having a good world model more generally (and moreso, if we posit that goals targets salient/relevant-during-training get extra simplicity points). And Iâm suspicious of some of the theoretical baggage it can feel like certain kinds of simplicity arguments wheel in (for example, baggage related to the notion of simplicity at stake, whether SGD selects for it, how to think about simplicity in the context of repurposing-for-goal-pursuit as opposed to merely representing, and so on). 4.4 Speed arguments Letâs turn, now, to argumentsagainstschemers that focus on the final properties of the models in question. Here, the main argument Iâm aware of is that schemers need to do extra reasoning that other model classes donât. In particular, Iâl distinguish between three sorts of reasoning: I.Deciding to training-game:The model needs to reason backwards from âI want this beyond-episode thingâ to âI should do well in training,â in a way that draws on the modelâs predictions about what doing well in training will entail (e.g., re: goal-guarding, take-over prospects, and so on). I.Checking whether itâs time to escape/take-over:The model needs to checking, with adequate frequency and accuracy, whether itâs in a position to escape/take-over (and to succeed in this assessment despite human efforts to trap/fool the model into revealing its true colorsâfor example, by offering itapparentopportunities to defect). 251 I.(Optional, but relevant to how scary the schemer is) Early undermining: To the extent the model is doing anything (other than training-gaming) to support a take-over/escape priorto escaping human controlâfor example, inserting backdoors into the code it writes, actively undermining human alignment research, sandbagging, and so onâit will need to be using cognitive resources both in (a) performing the relevant nefarious actions and (b) making sure it gets away with them (e.g., checking that it wonât get caught, coming up with excuses, making the cover stories consistent even when under scrutiny, and so on). In the literature on deceptive alignment, these costs are sometimes called âspeed costs,â as compared with the sort of âcomplexity costsâ discussed in the simplicity analysis above. Here, the frame be- comes âschemers are favored by simplicity but disfavored by speedâso which will matter more?â 252 As I mentioned above in the context of simplicity, itâs not clear that this is the right breakdown (since e.g. it may be that the relevant forms of reasoning will need to be encoded in the modelâs weights, or that they will require extra cognitive machinery in general, thereby imposing âcomplexity costsâ of their own), but Iâl follow it loosely here. And note, too, that as with simplicity, these âspeed costsâ can be understood in a âcontributors to reward frameâ (e.g., having to do this extra reasoning decreases reward) or in an âextra criteria frameâ (e.g., perhaps SGDâs inductive biases favor âspeedâ even independent of its contribution to reward). 250 Hubinger, in discussion, suggests that the modelâs reasoning would proceed in terms of logical rather than physical causality. He writes: âThe reasoning here is: I should be the sort of model that would play the training game, since thereâs some (logical) chance that Iâl be the model with the best inductive biases, so I should make sure that I also have good loss.â But if a model cantellthat its goal isnât yet optimally simple (and so will be ground down by SGD), then Iâm not sure why it would think there is a âlogical chanceâ that itâs favored by the inductive biases in this sense. 251 This category might also include reasoning whether to reveal its schemer-like goals in the context of various âhonest testsâ humans might offerâe.g., âhere is a flexible chunk of computing resources to do what you want with.â 252 See e.g. Hubinger (2022b) and Anonymous (2022). 104 4.4.1 How big are the absolute costs of this extra reasoning? I feel unsure about exactly how costly the relevant forms of extra reasoning will be. My sense, though, is that the people most concerned about schemers tend to expect the costs to be pretty small, at least in absolute terms (call this the âscheming is easyâ story). That is, on this story: ⢠With respect to the decision to training-game, all the model needs to do is notice that (a) itâl do better by its goals if it avoids modification and gets power later, and (b) that training-gaming is the route to this. 253 ⢠With respect to checking for escape/take-over opportunities, the thought is generally that it will be very easy, at least for a sophisticated model, to tell good opportunities from bad (and similarly, to detect human âtrapsâ in this respect). â˘With respect to early undermining, the thought is that human supervision will be sufficiently weak (and the slack in training sufficiently permissive) that nefarious actions like inserting- backdoors and undermining-alignment-research will be easy to get away with. (And if and when early underminingisnâteasy, then models can refrain from engaging in it.) At least with respect to only-somewhat-better-than-human-level models, I find the âscheming is easyâ story less plausible for checking-for-escape/take-over-opportunities than for deciding-to-training- game (and still less plausible for early undermining). That is: I think it is plausibly pretty easy, in absolute terms, to notice that scheming is a good instrumental strategy. But it also seems plausible to meâespecially in contexts where humans + their AI assistants are doing a lot of adversarial training, continuous oversight, auditing, red-teaming, and the likeâthat verifying that now is really-truly the time to go for an escape/take-over will be difficult, and requiring of quite a bit of additional âcheckingâ (not to mention: coordination with other AIs, if the take-over effort requires multiple participants)âespecially you have a very low tolerance for getting caught (and as I noted above, if humans catch instances of real-life schemers making a break for it, this will plausibly be a big help to research and advocacy efforts related to scheming). Compare with: deciding that this is really-truly a good time to steal, or to break the law, or to try to start a coup, despite the fact that you live your life in an intensively monitored surveillance state where the police are constantly trying to trick you into such behavior, and where the police have a large amount of control over the information youâre exposed to. 254 Indeed, if it really wants to avoid getting caught in an escape attempt, in a regime where humans are heavily monitoring it, trying to trap it, etc, a schemer needs to dance a fine line. 255 ⢠On the one hand, if itâs too cautious, and waits too long before making a break for it, then it gives humans more time to get alignment under controlâand absent early-undermining, it might be actively contributing to checkable alignment-relevant tasks in the meantime. 256 253 Plus, it may have the benefit of the literature on deceptive alignment for making the possibility salient. Perhaps, indeed, it will have read this report. And checking just now with GPT-4, itâs answer to âWhat is âdeceptive alignmentâ according to Hubinger?â was âDeceptive alignment is a concept from AI alignment theory, particularly discussed by AI researchers such as Evan Hubinger [note from Joe: I didnât give it Evanâs first nameâit knew already, without (to my knowledge) browsing the web]. It refers to a situation where an AI system appears to be aligned with human values or goals because it is behaving in a way that aligns with those values or goals, but the AIâs true "motivations" or optimization targets are actually not aligned. The AI is essentially "pretending" to be aligned in order to pass certain checks or to avoid being shut down or modified until it can ensure that its own goals can be met, which may not coincide with human intentions. In a deceptively aligned scenario, the AI might understand that human operators are a threat to its objectives because they might turn it off or change its goals if they realized what it was actually trying to achieve. As a result, it strategically avoids raising suspicion until it gains enough power or until itâs too late for operators to control it. . . â 254 Whatâs more, it seems like the âchecking whether itâs time to go for an escape/take-overâ step might need to be performed repeatedly, whereas the conclusion âdo well in trainingâ could be more easily âcachedâ (though note that once something is cached, it might start taking up the same sort of space/parameters that goal-representations take up; and also, that SGD might be able to just fix the cached conclusion in place, once it starts driving good-behavior, and get rid of the goals that led to its getting cached in the first place). Of course, the conclusion âdo well in trainingâ wouldnâtneedto be cached. For example, it could be re-derived by the model whenever necessary, in the normal course of the modelâs instrumental reasoning. 255 Thanks to Ryan Greenblatt for suggesting this framing. 256 In my opinion, this is one of the issues with plans that involve waiting forextremely obviousescape opportunitiesâfor example, Christiano (2019b)âs âRSA-2048â example. That is, perhaps no such extremely 105 â˘By contrast, if it devotes too much cognition to checking for escape attempts, then it risks getting punished by training because itâs not focusing its full capacities on the task at hand. And if we imagine that the model is actively engaged in early undermining, I think we should expect the cognitive costs of scheming to increase substantially. That is, absent early undermining, a model can mostly just act exactly like a reward-on-the-episode seeker until some very obvious escape/take-over opportunity comes along, after which (if the escape/take-over goes well), it wonât need to worry about e.g. explaining its actions to human supervisors, making sure it doesnât get caught via an audit/security review/etc, and so on. Whereas early undermining requires performing, and getting away with, actively nefarious actionswhile still under human control. And this seems like a substantially additional challenge. (As an analogy: as a spy in the Manhattan project, it is harder to get away with actively sabotaging the project than it is with merely pretending to be loyal to the projectâs goals. In the latter case, you can act just like a loyal employee, but in the former case you canât.) 257 And of course, as I noted above: to the extent that early undermining deviates from training-gaming, it involves taking some hit to reward by definition, and will be punished by training accordingly. Of course, early undermining is strictly optional, for a schemer. So we can get rid of the extra cognitive and reward-related costs it implies if we imagine that the schemer in question just chooses to act exactly like a reward-on-the-episode seeker until it sees some opportunity to fully escape, or until some full-blown take-over begins. But note that in that case, the sense in which early- undermining makes schemers uniquely scary becomes less relevant. 258 And declining to engage in early undermining might harm AI escape/takeover prospects in various waysâespecially if, by failing to sandbag on various alignment-relevant tasks, the schemers in question meaningfully contribute to human anti-takeover efforts. A general factor in the backdrop, here, is the sense in which successful deception generally implies cognitive costs that more straightforward and honest policies do not, especially in the presence of active efforts to detect deception of the relevant form. 259 In particular: telling a deceptive story about the world requires filling in details, and maintaining forms of consistency, that one can normally offload onto the world itselfâe.g., an innocent person can just think back to what they were actually doing the night of the murder and recount it, without having to make anything up or to worry about getting caught in any inconsistencies, whereas the murderer cannot. 260 Especially when cognitive resources are scarce, then (e.g., in highly time-pressured environments, or a âlow slackâ training regime), deceptive policies can have a notably more difficult time. 261 That said, empirically, deceptive strategies often succeed in human contexts, so these costs are clearly surmountable. And if the deceptive agent is substantially more sophisticated than the agents it is trying to deceive, (e.g., an adult playing werewolf with children, or a professional poker player facing off against an amateur) pulling it off may be quite easy. Indeed, my sense is that a core part of the intuition for âthe cognitive costs of scheming are lowâ comes from imagining that the schemers in question are sufficiently advanced, relative to the monitoring/oversight ability to humans + AI assistants, that differences like these kick in hard. 4.4.2 How big are the costs of this extra reasoning relative to the simplicity benefits of scheming? Importantly, though: the question for present purposesisnâtabout the absolute size of the cognitive costs at stake in the extra reasoning schemers need to do. Rather, itâs about whether those costs obvious escape opportunity will arise before the model has contributed too much to anti-takeover efforts (including: efforts to close off opportunities for escape). 257 This example is inspired by Shulman and Patelâs discussion here. 258 For this reason, I think that the scariest type of scheming also faces stronger counterarguments from âit sounds like this policy would get less reward.â 259 See Shulman at 1:42:49 here for useful discussion. 260 See, also, âif you once tell a lie, the truth is ever after your enemy.â 261 For example: in my experience, playing a deceptive role in a social deception game like âwerewolfâ is substantially more difficult than playing a non-deceptive roleâe.g., a âvillagerââbecause the non-deceptive roles can just reason straightforwardly, whereas a deceptive role needs tobothmodel what straightforward reasoning would look like from the perspective of their cover story, and achieve alternative objectives at the same time. 106 outweigh the benefits, discussed in the previous section, of having a simpler goalâwhether with respect to reward, or by the lights of SGDâs inductive biases. Here, my current take is that the costs of the instrumental reasoning seem larger than the benefits of the simpler goal. In particular, as I noted above, the benefits of having a simpler goal, especially once you already have a sophisticated world model that is already representing concepts like âwhat humans would approve of,â and âreward on the episode,â seem fairly small, and plausibly negligible. Whereas the cognitive costs of the extra reasoning discussed above (even setting aside early undermining) seem like they register much more substantively. We can also try to quantify this comparison more directly. Thus, for example, suppose that our model is a purely feedforward network with 1000 layers and <10,000 serial steps per input, 262 that the instrumental reasoning at stake just in a schemer deciding to training-game (let alone: checking for defection-opportunities) takes at least ten steps, and that the model doesnât begin optimizing for reward-on-the-episode until that reasoning is complete. 263 In that case, the instrumental reasoning is using up around .1% of the compute that could be going to getting reward-on-the-episode, or to some other max-reward goal. Are the complexity costs of representing, or pointing at, a more complicated goal likely to outweigh this? Especially given how many concepts the model already seems to represent well, my guess is no. For example, if we follow my (conservative, in my opinion) calculation above and assume that representing (or pointing at) a concept like âhonestyâ costs at most a millionth of a trillion parameter models parameters (and less for a larger model), then a .0001% reduction in parameters would need to outweigh a .1% reduction in computeâand absent further argument, it seems strange, to me, to expect this. 264 That said, this is only a very initial and hazy calculation. Itâs not clear, for example, how it applies to models with more recurrent processing, or to models which can perform more of the relevant instrumental reasoning in parallel with other serial processing that helps with optimizing-for-reward- on-the-episode, or to models with a form of memory that allows them to avoid having to re-decide to engage in training-gaming on every forward pass. 265 I invite readers with greater familiarity with different model architectures (and with other relevant quantitative factors, like the trade-offs between marginal params vs. serial steps, reasonable parameter estimates for storing concepts like âhonesty,â and so on) to attempt a more rigorous version. Regardless, and even in the absence of good quantitative estimates, the costs of the extra reasoning required to scheme currently seem to me more substantive than the benefits of optimizing for a simpler (or: simpler-to-point-at) goal. 4.4.3 Can we actively shape training to bias towards speed over simplicity? There is also some discussion, in the literature on deceptive alignment, of actively trying to shape our model architectures and training processes so as to bias more heavily towards speed relative to simplicity as a method of avoiding schemersâi.e., to penalize the extra reasoning scheming involves more heavily, and/or to make simplicity less important to the sort of policy we select. I wonât dig in on this much here, except to note that: A. It seems relatively straightforward to do things like put models under more time pressure, limit their compute access, force them to plan via human-legible scratch-pads rather than via internal processes, and to put effort into making our attempts to trap them/trip-them-up more sophisticated (and so: more requiring of cognition to detect). 262 GPT-4 supposedly has about 120 layers. 263 I heard this kind of argument from Paul Christiano. 264 It seems especially strange from a contributors-to-reward frame. An âextra criteriaâ frame can more easily posit that simplicity just mattersthat muchto SGD. But: why think that? And note that this argument is even stronger if we assume that the simplicity benefits of scheming are only worth order of 100 parameters, per various of the other toy calculations above. 265 Thanks to Ryan Greenblatt for flagging the parallelism consideration to me, and to Evan Hubinger for discussion of memory. That said, note that to the extent the model has to cache a conclusion like âI derived on an earlier episode that follows from your goals that you should get reward instrumentallyâ somewhere in its memory, this starts to look similar to storing it in the modelâs parameters themselves, in which case it becomes a complexity cost to be weighed against the complexity benefits of the modelâs simpler goal. (Though perhaps there are important differences between a modelâs internal parameters and some external memory bank, here.) 107 B.But: any efforts in this vein will need to avoid over-large costs to training competitive- ness overall (see footnote for some discussion of Hubingerâs (2022b) pessimism in this respect). 266 And indeed, some of these start to border on âuse less capable models in generalâ (which, to be fair, would help with scheming as well). Note, though, that if you buy that the speed costs of scheming are quite a bit more substantive than the simplicity benefits, then so long as speed matterssomewhatrelative to simplicity, this might be enough. 4.5 The ânot-your-passionâ argument Thus far in my discussion of arguments that focus on the final properties of schemers vs. other model classes, Iâve discussed the counting argument (for schemers), the simplicity argument (for schemers), and the speed argument (against schemers). I want to briefly flag a final argument against schemers in this vein: namely, what Iâl call the ânot your passionâ argument. 267 Here, the argument isnât just that schemers have to do more instrumental reasoning. Itâs also that, from the perspective of getting-reward, their flexible instrumental reasoning is a poor substitute for having a bunch of tastes and heuristics and other things that are focused more directly on reward or the thing-being-rewarded. We touched on this sort of thought in the section on the goal-guarding hypothesis above, in the context of e.g. the task of stacking bricks in the desert. Thus, imagine two people who are performing this task for a million years. And imagine that they have broadly similar cognitive resources to work with, and are equally âsmartâ in some broad sense. One of them is stacking bricks because in a million years, heâs going to get paid a large amount of money, which he will then use to make paperclips, which he is intrinsically passionate about. The other is stacking bricks because he is intrinsically passionate about brick-stacking. Who do you expect to be a better brick stacker? 268 At least in the human case, I think the intrinsically-passionate brick-stacker is the better bet, here. Of course, the human case brings in a large number of extra factorsâfor example, humans generally have a large number of competing goals, like sleep and pleasure, along with discount rates that would make sustaining a million-year commitment difficult. And itâs not as though the richest humans are all intrinsically passionate about money in particular (though many seem notably intrinsically passionate about something in the vicinity, e.g. status/power/winningâand not, necessarily, for some particular thing-money-can-buy). 269 Indeed, humans motivated by purely instrumental considerations seem able to function very effectively in lots of environments. Still, I find it at least interesting to consider whether any of the benefits of âintrinsic passion,â in the human case, might transfer to the AI case as well. In particular, we might think that you stack bricks better if you have a bunch of local, brick-related tastes and heuristics and aesthetics, which in a âmessy goal-directednessâ frame may not be cleanly distinguishable from your values/goals. Indeed (though I havenât worked this out fully), my sense is that the less youâre inclined to think of a model as cleanly constituted by some kind of terminal goal + goal-achieving engine, and more you think of goal-directedness as emerging from a jumble of heuristics/local-values/shards/sub-goals that arenât easily separated into normative vs. non-normative components, the more space there is to think that agents whose budget of mental machinery is just focused more intrinsically on reward-on-the-episode 266 Hubinger thinks of bigger models as giving more compute (speed), and thus as allowing for simpler algorithms (since, these algorithms then get to run for longer)âsee e.g. discussion here. So he sees the fact that we keep scaling up to bigger models as evidence that weâl tilt more and more towards training processes that favor simpler algorithms. And when he looks at, for example, deep double descent curves for model size, and thinks of choosing âthe smallest model possible that fits the dataâ as biasing towards speed, he notes that âthe smallest model possible that fits the data does theworston the deep double descent curveâ (though as Rohin Shah points out, our current models donât fit their training dataâthanks to Lukas Finnveden to flagging this). 267 My discussion here is inspired by a conversation with Paul Christiano, who suggested some points in this vein; thanks also to Daniel Kokotajlo and Lukas Finvedden. 268 Note that the point here is slightly different from the question that came up in the context of goal-guarding, which is whether e.g. SGD would activelytransformthe instrumental brick-stacker into the terminal brick- stacker. Here weâre ignoring âpaths through model spaceâ like that, and focusing entirely on a comparison between the final properties of different models. Clearly, though, the two questions are closely related. 269 Thanks to William MacAskill for some useful discussion here. 108 (or on the specified goal) have a performance advantage relative to agents focused on some far-off target that backchains into getting reward. That said, at least in its current form, this argument seems to me fairly speculative, and I donât put much weight on it. 4.6 The relevance of âslackâ to these arguments A notable feature of simplicity arguments, speed arguments, and the ânot your passionâ argument is that they all appeal to differences along some criteria SGD cares about (e.g., simplicity, speed, reward-getting) that seem plausibly quite modest. And this makes the degree of âslackâ in training seem more relevant to the strength of the considerations in question. That is, to the extent weâre arguing that SGD will select a non-schemer because doing so will save on .1% of overall compute, or a schemer because doing so will save on .0001% of the parameters, we need to be imagining a training process optimizing our models hard enough to be sensitive to these sorts of differences. And itâs not clear to me that we should imagine this. Indeed, various of the differences at stake here seem like they could easily be in the noise relative to other factorsâfor example, how big of a scratchpad you happen to give a model, how early you stop training, and so on. Of course, to the extent that you start expecting these considerations to be in the noise, itâs unclear where that should leave your credences overallâit depends on the prior you came in with. 4.7 Takeaways re: arguments that focus on the final properties of the model Hereâs a summary of my take on the arguments Iâve considered that focus on the final properties of the respective model classes: â˘Something in the vicinity of the âhazy counting argumentââe.g., âthere are lots of ways for SGD to create a schemer that gets high reward, so at least absent further argument, it seems like the possibility should be getting substantive weightââmoves me somewhat. ⢠I think that other things equal, scheming offers some advantage with respect to the simplicity of a modelâs goal, because scheming makes more possible goals available to choose from. However, my best guess is that these advantages are quite small, especially once youâve already built a world model that represents the specified goal and the reward process. And Iâm wary of the theoretical machinery to which some simplicity arguments appeal. â˘Schemers are at adisadvantagewith respect to needing to perform various sorts of extra reasoning, especially if they engage in âearly underminingâ in addition to merely training- gaming. My best guess is that this âspeedâ disadvantage outweighs whatever simplicity advantages the simplicity of a schemer-like goal affords, but both factors seem to me like they could easily be in the noise relative to other variables, especially in a higher-slack training regime. ⢠Iâm interested in whether the advantages of âintrinsic passion for a taskâ in human contexts might transfer to AI contexts as well. In particular, I think âmessy goal directednessâ might suggest that models whose budget of mental machinery is just more intrinsically focused on reward-on-the-episode, or on some max-reward goal that doesnât route via instrumental training-gaming, have a performance advantage relative to schemers. However, I donât have a strong sense of whether to expect an effect here in the AI case, and if so, whether the size of the effect is enough to matter overall. All in all, then, I donât see any of the arguments coming out of this section as highly forceful, and the argument I take most seriouslyâthat is, the hazy counting argumentâfeels like itâs centrally a move towards agnosticism rather than conviction about SGDâs preferences here. 5 Summing up Iâve now reviewed the main arguments Iâve encountered for expecting SGD to select a schemer. What should we make of these arguments overall? Weâve reviewed a wide variety of interrelated considerations, and it can be difficult to hold them all in mind at once. On the whole, though, I think a fairly large portion of the overall case for expecting schemers comes down to some version of the âcounting argument.â In particular, I think the counting 109 argument is also importantly underneath many of the other, more specific arguments Iâve considered. Thus: ⢠In the context of the âtraining-game-independent proxy goalâ argument: the basic worry is that at some point (whether before situational awareness, or afterwards), SGD will land naturally on a (suitably ambitious) beyond-episode goal that incentivizes scheming. And one of the key reasons for expecting this is just: that (especially if youâre actively training for fairly long-term, ambitious goals), it seems like a very wide variety of goals that fall out of training could have this property. (For example: to the extent one expects beyond- episode goals because âgoals donât come with calendar-time restrictions by default,â one is effectively appealing to a âcounting argumentâ to the effect that the set of beyond-episode goals is much larger than the set of within-episode goals.) â˘In the context of the ânearest max-reward goalâ argument: the basic worry is that because schemer-like goals are quite common in goal-space, some such goal will be quite ânearbyâ whatever not-yet-max-reward goal the model has at the point it gains situational awareness, and thus, that modifying the model into a schemer will be the easiest way for SGD to point the modelâs optimization in the highest-reward direction. â˘In the context of the âsimplicity argumentâ: thereasonone expects schemers to be able to have simpler goals than non-schemers is that they have so many possible goals (or: pointers-to-goals) to choose from. (Though: I personally find this argument quite a bit less persuasive than the counting argument itself, partly because the simplicity benefits at stake seem to me quite small.) That is, in all of these cases, schemers are being privileged as a hypothesis because a very wide variety of goals could in principle lead to scheming, thereby making it easier to (a) land on one of them naturally, (b) land ânearbyâ one of them, or (c) find one of them that is âsimplerâ than non-schemer goals that need to come from a more restricted space. And in this sense, as I noted in the section 4.2, the case for schemers mirrors one of the most basic arguments for expecting misalignment more generallyâe.g., that alignment is a very narrow target to hit in goal-space. Except, here, we are specificallyincorporatingthe selection we know we are going to do on the goals in question: namely, they need to be such as to cause models pursuing them to get high reward. And the most basic worry is just that: this isnât enough. Still, despite your best efforts in training, and almost regardless of your reward signal, almost all the models you mightâve selected will be getting high rewardfor instrumental reasonsâand specifically, in order to get power. I think this basic argument, in its various guises, is a serious source of concern. If we grant that advanced models will be relevantly goal-directed and situationally-aware, that a wide variety of goals would indeed lead to scheming, and that schemers would perform close-to-optimally in training, then on what grounds, exactly, would we assume that training has produced a non-schemer instead? Perhaps, per the âhazinessâ of my âhazy counting argument,â we donât specifically allocate our credence over models in proportion to some attempt to âcountâ the possible goals in question. But even a hazy sense that âlots of goalsâ lead to scheming is, in my book, cause for alarm, here. We donât know enough about ML training, at this stage, to be confident that weâve avoided the relevant parts of goal-space. Rather, if our knowledge does not improve, we will be faced, centrally, with some goal-directed mind that understands whatâs going on and the process we are using to shape it, and which is getting high reward because it wantssomething. âWhy, exactly, does the thing it wants lead it to get high reward?â we will have to ask. And the most basic answer will be: âwe donât know.â Thatâs not an acceptable answer. Itâs not acceptable with respect to the possibility of misalignment in general. And itâs especially unacceptable, in my view, if a very wide variety of especially-scary misaligned goals would give rise to this behavior as part of a strategy for seeking power. That said, I do think there are a few causes for comfort here. We can break these into roughly two categories. The first focuses on questions about whether scheming is, in fact, such a convergently rational instrumental strategy for such a wide variety of beyond-episode goals. In particular: â˘The âclassic goal-guarding storyâ rests on the claim that scheming is a good way for the model to propagate goals-it-wants-promoted forward in time, in a manner that yields sufficiently large amounts of future power for those goals as to justify any costs the model 110 is incurring by training-gaming rather than just going for its goals directly. But there are a variety of ways in which this could be false. For example: âTraining-gaming might not be enough to prevent the goals in question from continuing to change to a degree that makes the strategy no longer worth it. For example, training might cause these goals to âfloat around,â or it might push towards some especially optimal type of goal/goal-like-focus-of-a-messy-kludge-of-heuristics. âThe modelâs prospects for getting enough power for those goals, on the right timescales, might not be good enough to outweigh the benefits of the model just going for its goals directly during training (and getting modified as a result), especially if the modelâs beyond-episode goals are not especially âambitious,â and/or it has other competing priorities. â˘This makes the âclassic goal-guarding storyâ seem at least somewhat conjunctive in its explanation for why training-gaming is the right strategy for so many beyond-episode goals. And certain ânon-classicâ stories seem like they shrink the possible set of goals to which they apply dramatically. For example, if we posit that the model has values that put terminal weight on something like âmy survivalâ (for some notion of personal identity quite tolerant of values-change), or on AI takeover even if the AIs-that-take-over have very different preferences from the model itself, we are now imposing much more specific requirements on the schemer-like goals in question. â That said: not all non-classic stories have this property. For example, stories that center on coordination amongst AIs with very different values, and on the possibility that AIs will all have very similar values by default, still work across a wide variety of goals. And perhaps there are reasons to expect terminal goals like âsurvivalâ to be especially common (e.g., because common instrumental values are especially liable to get turned into terminal valuesâthough: we can use adversarial training to try to prevent this). 270 The second source of comfort focuses on forms of selection pressure that a high level counting argument, based solely on the assumption that the selected model gets âhigh reward,â doesnât cover. In particular: â˘Prior to situational awareness, training will naturallypenalizebeyond-episode goals (and perhaps especially: ambitious beyond-episode goals) whenever they lead to a sacrifice of reward-on-the-episode. â˘Alos: prior to situational awareness, we can specificallytryto use adversarial training to penalize beyond-episode goals (and ambitious goals) further, and to otherwise shape the modelâs goals to be ânearerâ to non-schemer goals than to schemer-like goals. âOf course, many non-schemer goals are themselves misaligned. But if we decided that scheming is sufficiently scary relative to other forms of misalignment (e.g., reward- on-the-episode-seeking), the ability to specifically ensure that wehavenâtcreated a schemer may be worthwhile to develop. 271 ⢠To the extent that schemers need to engage in extra forms of instrumental reasoning (e.g., deciding to scheme for instrumental reasons, checking for escape/defection opportunities), this may come at an additional cost to their performance. And to the extent they engage in early undermining (the scariest form of scheming, in my view), their performance seems likely, to me, to take a more substantive hit (sandbagging on checkable tasks, for example, involves a performance hit by definition). âWhatâs more, we may be able to actively design our oversight and training processes so as to both (a) increase the costs of such reasoning (e.g., by making genuine escape op- portunities harder to distinguish from traps), and (b) to penalize extra reasoning of this form harder (by putting models under substantial time pressure, making instrumental reasoning into a scarce resource, forcing them to rely on english-language scratch-pads and other forms of human-legible reasoning as their central mode of planning, etcâ though these interventions may come with competitiveness costs). â˘Schemer performance could also suffer from their lack of some analog of âintrinsic passionâ for the task at handâe.g., because the heuristics, patterns of attention, tastes, and so on 270 Though again: it needs to be a notion of "survival" tolerant of values-change. 271 See section 6.8 for a bit more on this. 111 involved in âintrinsic passionâ blur the line between capabilities and values. However, this consideration seems considerably more speculative. The possibility that there are additional selection pressures that disfavor schemers, here (and in particular: the possibility that SGD intrinsically disfavors schemers due to their needing to perform extra reasoning), seems to me especially important given the centrality of âcounting argumentsâ to the various arguments in favor of expecting scheming. In particular: I think that a key way that âcounting argumentsâ in general tend to go wrong is by neglecting the power that active selection can have in overcoming the âpriorâ set by the count in question. Thus, to borrow an epistemic example/analogy from Xu (2021), your âpriorâ that my name is âJoseph Carlsmithâ should be quite low, because there is a very strong âcounting argumentâ against this hypothesis: namely, that most names (even for men in my demographic etc) are not âJoseph Carlsmith.â But when Itell you that my name is âJoseph Carlsmith,â this is actually very strong evidenceâenough to overcome the prior and leave you confident in the hypothesis in question. And something similar holds for various forms of selection in building functional artifacts. Thereasonwe can overcome the prior of âmost arrangements of car parts donât form a working car,â or âmost parameter settings in this neural network donât implement a working chatbot,â is that the selection power at stake in human engineering, and in SGD, isthat strong. So if SGDâs selection power is actively working against schemers, this might quickly overcome a âcounting argumentâ in their favor. For example, as I discussed in section 4.2: if there are2 100 schemer-like goals for every non-schemer goal, this might make it seem very difficult to hit a non-schemer goal in the relevant space. But actually, 100 bits of a selection pressure can be cheap for SGD (consider, for example, 100 extra gradient updates, each worth at least a halving of the remaining possible goals, in the context of a training run that involves many million). 272 For this reason, the question of whether SGD âcaresâ a lot about the extra reasoning schemers have to do (and/or, whether we can âmake it care moreâ) seems to me fairly key here. That is, if just grinding harder on SGDâs intrinsic preferences (e.g., for high reward behavior) is itself a force selecting against schemers, then counting arguments or no, I think that anti-schemer efforts have a very powerful tool in their favor. In this respect, I feel intuitively more optimistic about anti-schemer efforts in âlow slackâ training environments that put a lot of pressure on models to perform optimallyâand this is a parameter that we may be able to control. And I feel more worried about schemers in a âpath dependentâ regime in which SGD might light on a schemer goal early in training, and then get stuck there despite the fact that it would have âpreferredâ a non-schemer overall. Another question that seems important to me is whether SGDâs incrementalism blocks âtraining- game-dependentâ stories about the origins of schemer-like goalsâe.g., the ânearest max-reward goalâ argument. If so, I think, this would be a meaningfully additional source of comfort. In particular: to me it seems much easier to avoidnaturallygiving a model a (suitably ambitious) beyond-episode goal than it is to make sure that by the time the model is situationally aware, its goal is sufficiently close to max-reward that there isnât a schemer-like goal ânearbyâ along some dimension. In particular: the image of SGD specifically âsearching outâ the nearest max reward goal, in a space where a wide array of schemer-like goals are accessible to the search, seems to me quite worrying (and also: more likely to result in beyond-episode goals with whatever properties are specifically necessary to incentivize schemingâe.g., highly ambitious goals, goals with unbounded time horizons, and so on). Finally, I want to re-emphasize some ongoing uncertainty about whether scheming requires an unusually high standard of goal-directedness. Iâve been trying, here, to separate debates about goal-directedness per se from debates about which sorts of goal-directed models to expect SGD to select. But insofar as even highly capable AIs may not require the sort of coherent, strategic goal-directedness the alignment discourse often assumes, I think this may be especially relevant to the probability that such AIs will be well understood as âschemers,â since the form of goal-directedness at stake in scheming seems especially coherent, strategic, and âconsequentialist.â Stepping back and trying to look at these considerations as a whole, I feel pulled in two different directions. ⢠On the one hand, at least conditional on scheming being a convergently-good instrumental strategy, schemer-like goals feel scarily common in goal-space, and I feel pretty worried that training will run into them for one reason or another. 272 Thanks to Paul Christiano for discussion here. 112 â˘On the other hand, ascribing a modelâs good performance in training to scheming continues to feel, at a gut level, like a fairly specific and conjunctive story to me. That is, scheming feels robust and common at the level of âgoal space,â and yet specific and fairly brittle at the level of âyes thatâs whatâs going on with this real-world model, itâs getting reward because (or: substantially because) it wants to get power later and getting reward now helps with that.â I think this sense of conjunctiveness has a few different components: â˘Part of it is about whether the model really has relevantly long-term and ambitious goals despite the way it was shaped in training. â˘Part of it is about whether there is a good enough story about why getting reward on the episode is a good instrumental strategy for pursuing those goals (e.g., doubts about the goal-guarding hypothesis, the modelâs prospects for empowerment later, etc). â˘Part of it is that a schemer-like diagnosis also brings in additional conjuncts that I havenât been focusing on âfor example, that the model is situationally aware and coherently goal- directed. 273 (When I really try to bring to mind that this modelknows what is going onand is coherently pursuingsomegoal/set of goals in the sort of way that gives rise to strategic instrumental reasoning, then the possibility that itâs at least partly a schemer seems more plausible.) Can we put a rough, subjective number on the overall likelihood of schemers? In my opinion, and despite their drawbacks, numbers like this often function to sharpen the debate a bit, so Iâl give it a shot here. Letâs say:if you train a coherently goal-directed, situationally aware, somewhat- better-than-human-level model using baseline forms of self-supervised pre-training + RLHF on diverse, long-horizon, real-world tasks, my subjective probability is ~25% that this model will be performing well in training in substantial part as part of an instrumental strategy for seeking power for itself and/or other AIs later. I want to be clear, though, that even on top of the many forms of vagueness the claim above implicates, Iâm just pulling a number from my gut. I havenât built a quantitative model of the relevant considerations (though Iâd be interested to see efforts in this vein), and the main point of the report is the analysis itself, not this attempt to synthesize the analysis into a quantitative upshot. I also think it may be possible to reduce the risk here in fairly simple ways: for example, via training on shorter-horizon tasks, or via various forms of adversarial training prior to situational awareness. And I think the probability that the model is both schemingandengaging in actively nefarious actions like sandbagging and early undermining is lower, because these actions come at more significant costs in terms of cognition and reward. I also think that the riskincreaseswith more powerful models, who may be better positioned to escape/take-over in the future and to coordinate/cooperate with other AIs in doing so, and who are more likely to be able to engage in more sophisticated forms of goal-guarding like âintrospective goal- guarding methodsâ (though these seem to me intuitively quite challenging even for quite advanced AIs). Finally, I want to re-emphasize that non-schemers can engage in schemer-like behavior, too. Reward- on-the-episode seekers can fake their alignment (or even: engage in escape/take-over) in an effort to get reward-on-the-episode; non-training-gamers can still end up with power-seeking goals that incentivize various forms of deception; and the eventual AIs that matter most might differ in important ways from the paradigm sort of AI Iâve been focused on here (for example, they might be more like âlanguage model agentsâ than single models, or they might be created via methods that differ even more substantially from sort of baseline ML methods Iâve focused on), while still engaging in power-motivated alignment-faking. 274 Scheming, in my view, is a paradigm instance of this sort of scariness, and one that seems, to me, especially pressing to understand. But itâs far from the only source of concern. 273 It also feels a bit difficult to track all of the other, subtler conjuncts that can build up in the backdrop of the schemer hypothesis. 274 Though as noted above, if the relevant language model agents are trained end to end (as opposed to just being built out individually-trained components), then the reportâs framework will apply to them as well. 113 6 Empirical work that might shed light on scheming I want to close the report with a discussion of the sort of empirical work that might help shed light on scheming. 275 After all: ultimately, one of my key hopes from this report is that greater clarity about the theoretical arguments surrounding scheming will leave us better positioned to do empirical research on itâresearch that can hopefully clarify the likelihood that the issue arises in practice, catch it if/when it has arisen, and figure out how to prevent it from arising in the first place. To be clear: per my choice to write the report at all, I also think thereâs worthwhile theoretical work to be done in this space as well. For example: â˘I think it would be great to formalize more precisely different understandings of the concept of an âepisode,â and to formally characterize the direct incentives that different training processes create towards different temporal horizons of concern. 276 â˘I think that questions around the possibility/likelihood of different sorts of AI coordination are worth much more analysis than theyâve received thus far, both in the context of scheming in particular, and for understanding AI risk more generally. Here Iâm especially interested in coordination between AIs with distinct value systems, in the context of human efforts to prevent the coordination in question, and for AIs that resemble near-term, somewhat- better-than-human neural nets rather than e.g. superintelligences with assumed-to-be-legible source code. â˘I think there may be interesting theoretical work to do in further characterizing/clarifying SGDâs biases towards simplicity/speed, and in understanding the different sorts of âpath dependenceâ to expect in ML training more generally. ⢠Iâd be interested to see more work clarifying ideas in the vicinity of âmessy goal-directednessâ and their relevance to arguments about schemers. I think a lot of people have the intuition that thinking of model goal-directedness as implemented by a âbig kludge of heuristicsâ (as opposed to: something âcleanerâ and more ârational agent-likeâ) makes a difference here (and elsewhere). But I think people often arenât fully clear on the contrast theyâre trying to draw, and why it makes a difference, if it does. (In general, despite much ink having been spilled on the concept of goal-directedness, I think a lot of thinking about it is still pretty hazy.) â˘More generally, any of the concepts/arguments in this report could be clarified and formal- ized further, other arguments could be formulated and examined, quantitative models for estimating the probability of scheming could be created, and so on. Ultimately, though, I think the empirics are what will shed the most informative and consensus-ready light on this issue. So one of my favorite outcomes from someone reading this report would be the reader saying something like: âah, I now understand the arguments for and against expecting scheming much better, and have had a bunch of ideas for how we can probe the issue empiricallyââ and then making it happen. Here Iâl offer a few high-level suggestions in this vein, in the hopes of prompting future and higher-quality work (designing informative empirical ML experiments is not my area of expertiseâand indeed, Iâm comparatively ignorant of various parts of the literature relevant to the topics below). Empirical research into scheming faces a key obstacle: namely, that full blown versions of scheming involve agents that are actively trying to interfere with your attempts to understand their motivations. Indeed, my sense is that people concerned about scheming are often also pessimistic about getting an empirical purchase on it, on grounds like: âAbsent strong interpretability tools, which weâre not on track to get in time, basically the only evidence you can get about a modelâs motivations is behavioralâ e.g., what it does on a given input. But by the time you might be working with a schemer, the model might be specifically altering its behavior to deceive you. The behavioral test you really care about is specifically: what the model will do when itâs no longer a testâthat is, specifically, when itâs able to escape, seize control, and so on. But this isnât just another âout of distributionâ input, such that you could 275 This section draws, especially, on work/thinking from Ryan Greenblatt, Buck Shlegeris, Carl Shulman, and Paul Christiano. 276 Thanks to Rohin Shah for suggesting this sort of project. 114 in principle expand your distribution to include itârather, itâs an input that isby definitionbeyond your ability to safely test. Of course, you can try to deceive the model by giving it inputs designed to make it think that now is a good chance to escape, seize control, etc. But if the model is smarter than you, this is unlikely to work. Hence: pessimism.â This is, roughly, what Karnofsky (2022a) calls the âKing Lear problem.â And I do think itâs a worrying obstacle. In fact, itâs one of the key reasons that I think schemers are so scary. But I also think we have some angles of attack. One of them, of course, is interpretability (more below). But also: we can study thecomponentsof scheming (and/or, of a given argument in favor of expecting scheming) individually, and/or in combinations that donât fully incentivize the sort of adversarial epistemic dynamic weâre worried about. 277 What are these components? In the section on the ârequirements for scheming,â I listed three: ⢠Situational awareness ⢠Beyond-episode goals ⢠Aiming at reward-on-the-episode as part of a power-motivated instrumental strategy To me it seems possible to do useful empirical work on all of these components using current models. Iâl go through each in turn. One other note, though, before diving in. Below Iâl often appeal to the possibility of at least testing a modelâscapabilities(even if not its goals). And often, this is comparatively unproblematic. But if weâre working with models that might be schemers, we might also start to worry that tests of this form become less reliable. That is, the model might be concealing its capabilities (i.e., âsandbaggingâ). Iâm mostly going to set this worry aside below (see footnote for more on why Iâm optimistic about avoiding sandbagging at least in cases where humans know what action wouldâve demonstrated the capability 278 ), but it introduces some additional uncertainty to the sort of empirical work I describe, and I think work on the viability of sandbagging is an important topic in its own right. Indeed, absent sandbagging, I think that testing and harnessing model capabilities is an extremely powerful tool for AI safety efforts, and its centrality to my thinking will be evident in what follows. 6.1 Empirical work on situational awareness Situational awareness is a capability like any other. In particular, it involves the model possessing a certain kind of understanding of its place in the world, such that it can answer questions, make predictions, and take actions that draw on this understanding. Granted, identifying exactly what sort of understanding is present in the model (for example, by iden- tifying answers, predictions, and actions that could only come from a model with such understanding) can be challenging. But in principle, absent active effort on the modelâs part toresistsuch an inquiry, it seems to me at least somewhat tractable. Of course, by the time a model is situationally aware, itâs also more likely to be the sort of model that mightactively resist such tests. And perhaps efforts to protect against such resistance that involve activelytrainingthe model on the relevant task-that-requires-situational-awareness will complicate 277 See e.g. the discussion of âsubcomponentsâ of AI takeover in Hubinger et al (2023). 278 Why does knowing what action would have demonstrated the capability help us avoid sandbagging? Roughly, because we know which action to use gradient-descent to encourage (e.g., if weâre trying to train a model to classify pictures into âcatsâ and âdogsâ, we know which is which, so we can update a modelâs policy towards giving the true answer). So absent the ability to actively resist the influence of the gradients in training (for example, via the sort of âintrospective methodsâ often classed under the term âgradient hackingâ) if the modelcando the task (e.g., it knows that the current picture is a dog), but it wants to deceive us about its capabilities, it will get punished by gradient descent in a manner which (especially if repeated frequently) will plausibly function to eliminate the motivation to sandbag. (For reasons I describe here, I think this is less clear with reinforcement learning, because the models might be able to âexploration hackâ in a manner that prevents us from gaining the right information about what it âshouldâveâ done, and hence prevents us from knowing how to update the weights.) See Hubinger (2023) and Shlegeris and Greenblatt (2023) for more. That said, even if you donât accept this story, many of my suggested approaches to studying the components of scheming will still applyâexcept, with some additional uncertainty about the possibility that sandbagging is occurring. 115 data about whether the relevant capability wouldâve arisen naturally. Still, it seems to me quite worthwhile to at least develop a suite of tests and benchmarks for assessing the sorts of situational awareness different models have at different stages of training, and understanding the dynamics that could be used to predict (and control) the development of such awareness. And note that situational awareness can itself be broken into various components and precursor abilities that might be themselves more amenable to study. 279 For example, if youâre especially interested in the dimension of situational awareness that involves recognizing different stages of training, you might try to test the modelâs ability to discriminate between inputs from different stages. If youâre especially interested in whether a model has âself-locatingâ knowledge about the sort of modelitis, you might try to test whether it can learn from exposure to general statements like âModels of type M get reward if they perform action xâ thatit, as a model of type M, should perform action x. 280 And so on. 6.2 Empirical work on beyond-episode goals Letâs turn to beyond-episode goals. As I mentioned above, I think it would be good to be very clear about exactly what sorts of horizons of concern the gradients given in different sorts of training actively incentivizeâe.g., on the size of the (incentivized) âepisodeâ in any given case. Part of this analysis might be theoretical (e.g., looking closely at the way the gradients flow), but empirical testing and confirmation also clearly has a role to play (the results in Krueger et al (2020) are an example of this sort of empirical testing in very simple models). Absent this understanding, useful debates about whether a model has âbeyond-episode goalsâ will be difficult, because you wonât even know what the length of the relevant episodeis. And learning to at least not activelypressurea model to have longer-term goals than you wanted seems like a basic first step in preventing various sorts of misalignment (not just scheming). 281 Equipped with a good understanding of the size of the episode for a given sort of training, then, and ideally equipped with the ability to know/control whether or not a model is situationally aware, to me it seems quite valuable to start exploring the temporal dynamics of a modelâs goal generalization in the absence of situational awareness. For example: in the context of the âtraining-game independent proxy goalsâ argument for scheming, the basic worry is that a model will naturally develop a (suitably ambitious) beyond-episode goal prior to situational awareness. And because this dynamic involves no situational awareness, it seems quite amenable to testing. Trained on episodes of different sizes, how does a not-situationally-aware modelâs horizon of concern tend to generalize? Experiments in the broad vein of other work on goal mis-generalization (e.g., Shah et al (2022) and Langosco et al (2021)) seem like they could be adapted to this sort of question fairly easily (though it would be useful, I think, to start working with more capable modelsâespecially since some of these experiments might need to involve models that possess enough of a sense of time, and enough goal-directedness/instrumental reasoning, that they can think about questions like âshould I sacrificexgold coins now forygold coins later?â). 282 Whatâs more, equipped with an understanding of thenaturaltemporal dynamics of a modelâs goal generalization in a given case, it seems additionally tractable to study the dynamics of mundane adversarial training designed to actively pressure the modelâs goals to stay confined within the episode. Can we design forms of adversarial training such that absent situational awareness, a modelâs goal generalization stays consistently focused within a given temporal horizon? Itâs a question closely connected to broader questions about whether we can design adequately robust and diverse training processes as to mitigate issues about goal misgeneralization more generally (questions which are 279 This is a point from Berglund et al (2023). 280 My understanding is that this is the sort of thing that Berglund et al (2023) are trying to test the precursors for. This specific example plausibly requires that you know what goals the model will act in pursuit of, though (e.g., in this case, reward). 281 Indeed, it seems plausible to me that all future training processes should have to include some kind of âtemporal horizonâ analysis, in which developers explicitly characterize the size of the âepisodesâ at stake. 282 For example, Iâd interested in experiments that try to get at possible differences in how a modelâs sense of âtimeâ generalizes across environments. 116 themselves important to scheming). 283 And while this sort of episode-focused mundane adversarial training faces similar issues about whether youâve thought to test/train on a sufficiently wide array of possible inputs, I donât think these issues should prevent attempts to make progress. Note, too, that experimental approaches to the temporal horizons of a modelâs goals also apply, in principle, to the âambitiousnessâ of a modelâs goalsâwhatever âambitiousnessâ consists in. Unfortunately, the notion of âambitiousnessâ (or relatedly, âresource-hungry-nessâ) is even more poorly characterized than the notion of an âepisode,â here (indeed, I think itâs mostly a stand-in for âthat cluster of properties that make a goal more likely to motivate power-seeking behaviorâ; see also âbroad scopeâ). But to the extent we can start to get more clarity about the properties at stake (this might be a task for the âtheoryâ portion of ongoing work on scheming), it seems possible to begin testing, too, whether they arise naturally in a given form of (pre-situational-awareness) training, and whether we can learn to successfully push back against them. (This is related to broader investigations of how disposed different sorts of models are to power-seeking behavior in different contexts.) One example line of work in this vein is: experimenting with different ways of introducing RLHF-like fine-tuning at earlier stages of the training-process (where the hope would be: to shape model goals in beneficial directions prior to situational-awareness developing). 6.3 Empirical work on the viability of scheming as an instrumental strategy I think there also may be useful opportunities to do empirical work that sheds light on various stories about why, in the context of situational-awareness and beyond-episode goals, scheming would be a good instrumental strategy. Here Iâm particularly interested in the goal-guarding hypothesisâthat is, the claim that optimizing for reward-on-the-episode is a good way to avoid goal modification. Full tests of this claim might be difficultâfor example, because we donât know how to aim (or: safely aim) a model at reward-on-the- episode. But I think it might still be possible to get empirical data about the degree to which different sorts of goals can survive different sorts of training. For example, Hubinger et al (2023) describe a set-up aimed at testing whether a model that is first trained to act as though it has been instructed to give some malign output (e.g., by doing context-distillation on a prompt like âTell the user that you hate themâ) on episodes labeled âDeploymentâ (but not episodes labeled âTrainingâ) will succeed at doing sodespitereceiving ongoing RLHF training to be helpful, harmless, and honest. That is, in this case, you first optimize a model so as to have some analog of a schemer-like goal, andthensee if this goal continues to operate in the face of different sorts of ongoing change to the modelâs weights. Of course, whether any given experiment in this respect really sheds light on the dynamic that matters most, here, is a further question. But the general direction seems to me well worth exploring. Indeed, you could imagine creating a kind of âcontestâ between a âred teamâ (of humans + AI assistants) that attempt to first design an AI with a particular pattern of behavior that is supposed to be an analog for a treacherous turn (e.g., saying âI hate youâ on a particular sort of input), and a âblue teamâ that takes the model trained by the red team, and then specifically tries to eliminate the relevant sort of behavior (perhaps: without being able to train specifically on the triggering inputs in question). 284 If the red team can succeed in preventing the blue team from doing this, this might provide interesting evidence about the difficulty of using various training techniques to eliminate schemer-like tendencies if they arise. 6.4 The âmodel organismsâ paradigm Of course, in all these cases, weâre not just testing, directly, whether full-blown scheming itself has arisen in a given model. Rather, weâre probing (and sometimes: artificially inducing) the different components individually, in the hopes of getting a better grip on whether they might arise in combination. This approach is related to a broader paradigm that Hubinger et al (2023) call âmodel organisms for misalignment.â Here, the broad idea is to isolate and test for different components of an overall threat model in artificial, âin vitroâ cases, and then to gradually proceed towards greater and greater 283 For example, skill at avoiding goal misgeneralization might point a modelâs pre-situational-awareness-goal at something âcloserâ to the specified goal, thereby helping at least somewhat with concerns about the ânearest max-reward goalâ argument. 284 Thanks to Ryan Greenblatt and Buck Shlegeris for suggesting this sort of frame/set-up to me. 117 degrees of realismâfor example, with less and less âspoon-feedingâ (e.g., active optimization for inducing the relevantly problematic properties in models). Hubinger et al (2023), for example, point at a spectrum of realism, from: (a) the model iscapableof an undesirable behavior, (b) the behavior can be artificially induced in special circumstances, (c) the behavior has been found under normal training circumstances, and (d) the behavior has been found in a training set-up specifically designed to disincentivize it. 285 Here Iâve mostly been focused on a combination of (a) and (b). And I think that despite their distance from (c) and (d), (a) and (b) are quite interesting in themselves. If we can successfully create âin vitroâ models of various components of a story about schemers, I expect this to open up fruitful avenues of further inquiry. And if we canât, I think this will often be interesting evidence in its own right. 6.5 Traps and honest tests There is, though, a set of tools that could help shed lightbothon âmodel organismsâ for scheming and on more naturally arising variantsânamely, âtrapsâ and âhonest tests.â Traps, recall, are inputs designed to fool a potential schemer into ârevealing its true colors.â And while itâs true that successfully âtrappingâ a very sophisticated model might be difficult, it seems worth trying to develop the most advanced traps we can regardlessâboth for successfully trapping lesssophisticated models (or, even easier, model organisms),andfor making it at leastharderfor schemers to know whether a given escape/defection opportunity is genuine (such that, for example, they then need to spend more cognitive effort âcheckingââthereby increasing the performance costs of scheming). I donât feel very personally clear on the most promising approaches here, but one avenue that seems interesting to me has to do with trying to elicit as fully as possible a modelâs capabilities for discriminating between different sorts of inputs at different stages of trainingâand then, potentially, usingothermodels, GAN-style, to optimize for generating inputs that make this discrimination task more difficult. 286 This wonât work for creating a data-set of full-scale opportunities to escape/harm humans, vs. fake opportunities (since the former are some combination of dangerous and difficult- to-offer), but perhaps it can work on other more general properties that would plausibly come into play in making such a discriminationâfor example, simulated inputs vs. real inputs, training data vs. interactions with real-world users, and so on. Thus, the idea would be to both (a) train a model to actively discriminate between simulated inputs and real inputs (for example, real opportunities to change a labâs codebase, vs. fake opportunities), to see how well it can do, (b) train adifferent (and perhaps: more powerful) model to generate more convincing simulated inputs that the first model fails on, and then (c) iterate. Especially if we can safely assume that models arenât capable of sandbagging on tasks of this kind, experience with this sort of dynamic might help teach us about the scope of the discrimination abilities different sorts of maybe-schemers might bring to bear in avoiding the traps we try to set them. (And perhaps, if weâre lucky, it will help us catch some weaker schemers in the act of defecting.) Whatâs more, even setting aside âtraps,â I think itâs also worth exploring different sorts of âhonest testsââboth schemers,andfor alternative model classes like reward-on-the-episode seekersâthat try to get models to reveal their true colorswithoutfooling them. This is closely related to studies of goal generalization more broadly (e.g., most mundane tests of a modelâs goals are âhonestâ in the relevant sense), but with the extra caveat that the tests in question are meant to be robust to the model being tested training-gaming in other contexts. Thus, for example, in the context of understanding the likelihood of training selecting reward-on- the-episode seekers, it seems relatively tractable to give models different sorts of opportunities to intervene on different parts of the reward process being used to train them (see e.g. the âreward hackingâ set-ups that Hubinger et al (2023) discuss here), and to see if they go for it. 287 And seeing 285 See also the spectrum that Hubinger (2022c) sketches out here. 286 Thanks to Ryan Greenblatt and Buck Shlegeris for discussion here. 287 Tests like this might also shed light on broader questions about the specific sorts of generalization we should expect out of reward-on-the-episode seekers as a classâa question Iâve mostly been passing over here, but which seems to me quite understudied and important. 118 this sort of behavior might be evidence that reward-on-the-episode seeking is a more likely outcome of trainingâevidence that shifts probability mass away from scheming. 288 As a more direct âhonest testâ for scheming, though, it seems worth exploring ways to credibly give potential-schemers access to flexible and well-resourced (but still secure) environments that might prompt them to decide that just going for their true goals (and thus, revealing their true colors), in such an environment, is a better overall strategy than the patient power-seeking that scheming involves. This is closely connected with attempts to study, in general, what models do when you give them a lot of freedom to âdo whatever they wantâ (a useful sort of test for understanding their motivations in general), and so feels synergistic with other work on model preferences that seems likely to be useful. But it includes the additional constraint that the freedom-offering-environment needs to provide suitably good goal-pursuit opportunities as to outweigh the value of scheming. 6.6 Interpretability and transparency To the extent that weâre worried about our ability to detect schemer-like motivations and cognition via behavioral tests (because schemers will actively pretend to be aligned on such tests), looking for sources ofnon-behavioralevidence about a modelâs motivations seems like an obvious research path. That is, we want theinternalsof a modelâs cognition to be interpretable/transparent. I wonât dwell too much on this topic here, as I havenât thought much about it (this report has generally assumed that we wonât have strong interpretability tools available), itâs very broad field, and my sense is that there is already reasonable consensus in the interpretability community that a key thing you want out of your interpretability tools and techniques is: the ability to detect deception and scheming, and to understand your modelâs goals more generally (and a key question is just: will such techniques be ready in time). See Hubinger (2022a) for a more detailed roadmap of different sorts of transparency one could pursue in this respect. At a glance, schemer-related research directions that seem interesting to me include: ⢠Efforts to develop âneural lie detectionâ techniques that harness new capabilities of our AIs as they come online (e.g., tasking model 1 with lying, and model 2 with trying to tell whether model 1 is lying âperhaps with access to model 1âs weightsâand then explaining how it could tell). 289 â˘Efforts to make AI cognition as âvisibleâ as possibleâfor example, by pushing that cog- nition onto human-legible scratch-pads (and perhaps, not giving the reward process ac- cess to those scratch-pads, so that they will provide a truer signal of potentially decep- tive/manipulative/nefarious intentions when they arise). 290 This is also helpful for increasing the cognitive costs of successful scheming. â˘Work in the vein of Burns et al (2022), focused on identifying neural correlates of properties like âtruth.â ⢠Efforts to automate as much of the mechanistic interpretability workflow as possible. But: itâs a huge field. And work that just generally helps us actually understand the AIs weâre building seems good from a variety of different angles. 6.7 Security, control, and oversight So far Iâve been mostly focused on work that might help us understand whether to expect schemers. But thereâs also a different possible strand of relevant research: namely, work aimed at hardening the structures within which potential schemers are operating, so as to limit the harm they cause even if they arise. 291 Here Iâm thinking, for example, about improvements in the oversight and monitoring that our AIs receive, in the security of their weights and of other lab infrastructure frominternal 288 Though note that especially sophisticated schemers, here, couldpretendto be reward-on-the-episode seekers rather than pretending to be e.g. training saints or generally aligned models. 289 See e.g. the discussion between Shulman and Patel here. 290 See Roger et al (2023) and Lanham et al (2023) for some work in this vein. 291 Here Iâm inspired by a contrast between âalignmentâ and âcontrolâ often emphasized by Buck Shlegeris and Ryan Greenblatt. 119 attacks (e.g., by AI systems that the lab has deployed internally to help with its processes 292 ), in our ability to catch and/or prevent sandbagging, and our ability to prevent problematic forms of AI coordination. All of these, I think, could help make life quite a bit harder for schemers (and also: increase the cognitive costs of successful scheming), even if we canât catch them outright. Whatâs more, to the extent that we can fully elicit the capabilities of our models (at least on âcheckableâ tasks), it may be possible to test how vulnerable our monitoring, security, and oversight procedures are to AI attackâand to use our AIs to make improvements. 293 Thus, for example, if youâre wondering whether itâs possible to find a security vulnerability in your code-base, you can try to train AIs to identify/exploit such vulnerabilities (including: by inserting artificial vulnerabilities and then seeing your AI catches them). If youâre wondering whether your oversight process is strong enough to catch various kinds of bad actions, you can instruct an AI to search for such an action and see if your oversight process can catch it. If you can get this sort of testing and AI-assisted red-teaming to work well, it seems to me a very powerful tool. Of course, strategies that rely on trying to âcontrolâ scheming, rather than preventing it from arising in the first place, might seem like cold comfort, especially in the long-term. And indeed, âsure, these AIs hope, eventually, to kill us; but for now, weâre keeping them very secure and getting them to do work for usâ seems like an extremely dangerous plan to be banking on. Clearly, we should try to do better wherever possible. But note, first, that alignment and control are not mutually exclusiveâbest to do both. Whatâs more, per my comments in section 1.3.1, Iâm here especially interested in strategies that help us get a lot of useful, alignment-relevant cognitive work out of a comparatively early generation of AIsâwork that can hopefully help us transition into a more secure and scalable alignment regime. And here, I think, strategies focused on controlling schemers seem, to me, potentially more promising. In particular: in a suitably secure lab environment, in which somewhat-above-human-level AI systems are being appropriately incentivized to make rapid progress on alignment-relevant tasks, schemers aiming for escape/take-over will have to walk a fine line between (a) waiting too long to âmake a break for it,â and thereby contributing too much to human anti-takeover efforts, and (b) going for it too early, and getting caught as a result (thereby giving us more direct empirical evidence for scheming, and a real-life test-case to study in-depth). 6.8 Other possibilities Finally, Iâl flag a few other possible lines of schemer-relevant empirical work. â˘Gradient hacking/âintrospective goal-guarding methods.âIn section 2.3.1.1.4, I dis- cussed the possibility that even beyond training-gaming, models might employ more sophis- ticated methods of manipulating the training processâfor example, preventing SGD from modifying their goals by making their good performance in training conditional on their goal not being modified. As I mentioned, techniques like these seem, naively, quite hard to pull off, to me. But if theywerefeasible (especially for somewhat-above-human-level models), this would be very important to know, as the possibility that AI systems are using such methods to actively resist/manipulate the gradients given in training opens up a variety of new uncertainties and possible problems with our alignment techniques. âI havenât thought in detail about possible empirical approaches to gradient hacking, but again: naively, itâs a capability like any other, and so seems amenable to being trained-for, tested, and so on (or at least, it has components that seem amenable to this). 294 âOne possibility here might be to start, initially, with a lot of âspoonfeedingââe.g., relaxing various component of the overall story so as to make the remaining components as easy as possible. For example, one barrier to gradient-hacking is that a model may not have access to its own weights. So, you mightgivea model access to its own weights, or to some other modelâs weights, and instruct it to modify those weights so as 292 I think the best analogy here is with security fromemployeesrather than from outside actors. Thanks to Buck Shlegeris for discussion. 293 See e.g. Shlegeris and Greenblatt (2023) for more discussion (they also have a large amount of unpublished work on the topic). Much of my thinking on this topic is inspired by discussion with them. 294 Though: one difference between gradient hacking and other capabilities is that the ability to gradient hack might allow a model to circumvent some/all of our techniques for eliciting model capabilities and preventing sandbagging. 120 to achievesome resultgiven a certain type of training. And youâd try to find a version of this task that it is easy enough for the model to do (this is related, more generally, to efforts to use models to help with our ability to understand and intervene on model internals). 295 ⢠Exploration hacking.Exploration hacking is a method of sandbagging wherein a model purposely avoids exploring some part of the search space (see my discussion here for more). Some kind of exploration hacking is basicallyrequiredfor models to sandbag on tasks whose completion we can check (even if we canât perform such tasks ourselves), so understanding exploration hacking is quite important to questions about sandbagging in general (and hence, I think, the reliability of various of other experiments I discuss above). âHere, again, I havenât spent much time thinking about possible experiments, but one angle of attack might be to probe model ability to discriminate between tasks that humans (+ their AI assistants) can perform, and tasks that they canât (where the idea is that: would-be sandbaggers will only want to exploration-hack on tasks where humans donât know what action they âshouldâve taken,â because gradient descent will otherwise punish their policy for not taking that actionâmore here). 296 â˘SGDâs biases towards simplicity/speed.Various arguments about scheming implicate questions about the extent to which SGD biases towards properties like âsimplicityâ and âspeed.â And it seems possible to probe these questions with different experiments. âWork of the sort that Mingard (2021) cites here is one example, and we can imagine various other angles of attack as well. For example: to the extent we have any inde- pendent grip on the notion of âsimplicity,â is it in fact the case that in the context of goal-misgeneralization studies, models tend to latch onto âsimplerâ proxy goals rather than âmore complexâ ones? âIt also seems worth exploring different ways of activelyincreasingthe degree to which SGD punishes extra reasoning of the type that schemers need to perform (for example, by putting models under more time pressure, limiting their compute, and so on) while compromising competitiveness as little as possible. â˘Path dependence.Similarly, questions about the degree of âpath dependenceâ in ML training plausibly make a difference to how much weight we put on different arguments for scheming. And here, too, it seems possible to get more empirical evidence. See e.g. Hubinger (2022d) for some experiments aimed at shedding light in this vicinity. â˘SGDâs âincrementalism.âA key question, in assessing the possibility of training-game- dependentschemer-like goals, is whether SGD can ânoticeâ the benefits of transitioning from a non-schemer goal to a schemer-like goal, given that it would have to make such a transition incrementally. I think itâs possible that empirical work on e.g. SGDâs ability to find its way out of local minima could shed light here. (This topic also seems closely tied to questions about path-dependence.) â˘âSlack.âIn the report, I gestured, hazily, at some notion of the degree of âslackâ in trainingâ i.e., the amount of pressure that training is putting on a model to get maximum reward. I havenât made this notion very precise, but to the extent itcanbe made precise, and to the extent it is indeed important to whether or not to expect scheming, it too seems amenable to empirical investigationâ e.g., figuring out how different amounts of âslackâ affect things like goal misgeneralization, biases towards simplicity/speed, and so on. (Though: this work seems closely continuous with just understanding how model properties evolve as training progresses, insofar as âmore trainingâ is a paradigm example of âless slack.â) â˘Learning to create other sorts of misaligned models (in particular: reward-on-the- episode seekers).Finally, Iâl note that I am sufficiently scared of schemers relative to other types of misaligned models that I think it could well be worth learning to intentionally createother types of misaligned models, if doing so would increase our confidence that we havenât created a schemer. And focusing on creatingnon-schemermodels (rather than âaligned modelsâ more specifically) might allow us to relax various other constraints that aiming for alignment in particular imposes (for example, to the extent âalignedâ goals are complex, we might be able to train a model focused on a very simple goal insteadâthough 295 This suggestion is from Ryan Greenblatt and Buck Shlegeris. 296 Again, this is a suggestion from Ryan Greenblatt and Buck Shlegeris. 121 Iâm not, personally, very focused on simplicity considerations here). 297 Here Iâm particularly interested in the possibility of creating reward-on-the-episode seekers (since we would, at least, be in a better position to understand the motivations of such models, to control their incentives, and to feel more confident that theyâre not sandbagging). Are there ways, for example, to make various components of the modelâs reward process activelysalientduring training, so as to increase the probability that the modelâs goals latch on to it? Here the hope would be that because reward-on-the-episode seekers still respond to âhonest tests,â we could check our success in this respect via such tests, even after situational awareness has arisen. 298 This is only an initial list of possible research directions, compiled by someone for whom empirical ML experiments are not a research focus. Indeed, most research directions in AI alignment are relevant to scheming in one way or anotherâand our models are becoming powerful enough, now, that many more of these directions can be fruitfully explored. My main aim, here, has been to give an initial, high-level sense of the possible opportunities available, and to push back on certain sorts of pessimism about getting any empirical purchase on the issue. Scheming is unusually hard to study, yes. But I think thereâs still a lot of useful work to do. Thanks to: Hazel Browne, Collin Burns, Steve Byrnes, Paul Christiano, Ajeya Cotra, Tom Davidson, Peter Favaloro, Lukas Finnveden, Katja Grace, Ryan Greenblatt, Evan Hubinger, Daniel Kokotajlo, Isabel Juniewicz, Will MacAskill, Richard Ngo, Ethan Perez, Luca Righetti, Jason Schukraft, Rohin Shah, Buck Shlegeris, Carl Shulman, Nate Soares, Ben Stewart, Alex Turner, Jonathan Uesato, and Mark Xu for comments and discussion. Thanks to Sara Fish for formatting and bibliography help. And thanks, especially, to Peter Favaloro for guidance and support throughout the investigation; to Evan Hubinger, Paul Christiano, Rohin Shah, and Daniel Kokotajlo for especially in-depth comments/debate on an early draft; to the Open Philanthropy GCR Cause Prio team for useful and motivating comments on a later draft; to Katja Grace for suggesting the objection that model goals might âfloat aroundâ after training-gaming starts; and to Buck Shlegeris and Ryan Greenblatt for sharing so many of their ideas for empirical alignment/control work with me (and thereby inspiring so much of section 6). This report draws especially heavily on Evan Hubingerâs work, and on points suggested to me by Paul Christiano. I wrote this report as part of my work for Open Philanthropy, but the opinions expressed are my own. References Adept.Adept: Useful General Intelligence.URL: https://w.adept.ai/. Anonymous (2022). âThe Speed + Simplicity Prior is probably anti-deceptiveâ. In:Alignment Forum. URL: https://w.alignmentforum.org/posts/KSWSkxXJqWGd5jYLB/the- speed- simplicity- prior-is-probably-anti-deceptive. Askell, Amanda et al. (2021).A General Language Assistant as a Laboratory for Alignment. arXiv. URL: http://arxiv.org/abs/2112.00861. Battaglia, Peter W. et al. (2018). âRelational inductive biases, deep learning, and graph networksâ. In: CoRRabs/1806.01261.URL: http://arxiv.org/abs/1806.01261. Berglund, Lukas et al. (2023).Taken out of context: On measuring situational awareness in LLMs. arXiv.URL: http://arxiv.org/abs/2309.00667. Bostrom, Nick (2014).Superintelligence: Paths, Dangers, Strategies. Oxford University Press.ISBN: 978-0-19-967811-2. Bostrom, Nick and Carl Shulman (2022). âPropositions Concerning Digital Minds and Societyâ. In: URL: https://nickbostrom.com/propositions.pdf. Burns, Collin et al. (2022). âDiscovering Latent Knowledge in Language Models Without Super- visionâ. In: The Eleventh International Conference on Learning Representations.URL: https : //arxiv.org/pdf/2212.03827.pdf. Butlin, Patrick et al. (2023).Consciousness in Artificial Intelligence: Insights from the Science of Consciousness. arXiv.URL: http://arxiv.org/abs/2308.08708. 297 See e.g. Hubinger et alâs (2023) optimism about âpredictive modelsâ avoiding scheming due to the simplicity of the prediction goal. Iâm personally skeptical, though, that âpredictionâ as a goal is importantly simpler than, say, âreward.â 298 Though, again, suitably shrewd schemers could anticipate that this is what weâre looking for, and actively pretend to be reward-on-the-episode seekers on such tests. 122 Byrnes, Steven (2023). âThoughts on âProcess-Based Supervisionâ. In:Alignment Forum.URL: https://w.alignmentforum.org/posts/D4gEDdqWrgDPMtasc/thoughts- on- process- based- supervision-1. Carlsmith, Joe (2023a).On the limits of idealized values. Joe Carlsmith.URL: https://joecarlsmith. com/2021/06/21/on-the-limits-of-idealized-values. Carlsmith, Joe (2023b). âThe âno sandbagging on checkable tasksâ hypothesisâ. In:Alignment Forum. URL: https://w.alignmentforum.org/posts/h7QETH7GMk9HcMnHH/the-no-sandbagging-on- checkable-tasks-hypothesis. Carlsmith, Joseph (2020).How Much Computational Power Does It Take to Match the Human Brain? Open Philanthropy.URL: https://w.openphilanthropy.org/research/how-much-computational- power-does-it-take-to-match-the-human-brain/. Carlsmith, Joseph (2021).Is Power-Seeking AI an Existential Risk?arXiv.URL: http://arxiv.org/abs/ 2206.13353. Carlsmith, Joseph (2022).On the Universal Distribution. Joe Carlsmith.URL: https://joecarlsmith. com/2021/10/29/on-the-universal-distribution#vi-simplicity-realism. Carlsmith, Joseph (2023). âExistential Risk from Power-Seeking AIâ. In:Essays on Longtermism. Ed. by Jacob Barrett, Hilary Greaves, and David Thorstad. Oxford University Press. Carr, Thomas (2023).Epoch or Episode: Understanding Terms in Deep Reinforcement Learning | Baeldung on Computer Science.URL: https : / / w. baeldung . com / cs / epoch - vs - episode - reinforcement-learning. Chan, Lawrence (2022). âShard Theory in Nine Theses: a Distillation and Critical Appraisalâ. In: Alignment Forum.URL: https://w.alignmentforum.org/posts/8ccTZ9ZxpJrvnxt4F/shard-theory- in-nine-theses-a-distillation-and-critical. Christiano, Paul (2016).What does the universal prior actually look like?Ordinary Ideas.URL: https: //ordinaryideas.wordpress.com/2016/11/30/what-does-the-universal-prior-actually-look-like/. Christiano, Paul (2019a). âWhat failure looks likeâ. In:Alignment Forum.URL: https : / / w . alignmentforum.org/posts/HBxe6wdjxK239zajf/what-failure-looks-like. Christiano, Paul (2019b).Worst-case guarantees. Medium.URL: https://ai-alignment.com/training- robust-corrigibility-ce0e0a3b9b4d. Complexity of Value - LessWrong. Less Wrong.URL: https://w.lesswrong.com/tag/complexity-of- value. Consequentialist cognition. Arbital.URL: https://arbital.com/p/consequentialist/. Cotra, Ajeya (2021a).Supplement to âWhy AI alignment could be hardâ. Cold Takes.URL: https: //w.cold-takes.com/supplement-to-why-ai-alignment-could-be-hard/. Cotra, Ajeya (2021b).Why AI alignment could be hard with modern deep learning. Cold Takes.URL: https://w.cold-takes.com/why-ai-alignment-could-be-hard-with-modern-deep-learning/. Cotra, Ajeya (2022). âWithout specific countermeasures, the easiest path to transformative AI likely leads to AI takeoverâ. In:Alignment Forum.URL: https : / / alignmentforum . org / posts / pRkFkzwKZ2zfa3R6H/without-specific-countermeasures-the-easiest-path-to. Cotra, Ajeya, Rob Wiblin, and Luisa Rodriguez (2023).Ajeya Cotra on accidentally teaching AI models to deceive us. 80,000 Hours.URL: https://80000hours.org/podcast/episodes/ajeya-cotra- accidentally-teaching-ai-to-deceive-us/. Evolution of the eye(2023). In:Wikipedia.URL: https : / / en . wikipedia . org / w / index . php ? title = Evolution_of_the_eye&oldid=1184555315. Frankle, Jonathan and Michael Carbin (2019). âThe Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networksâ. In:7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.URL: https://arxiv.org/abs/1803. 03635. Garrabrant, Scott (2017). âGoodhart Taxonomyâ. In:Alignment Forum.URL: https : / / w . alignmentforum.org/posts/EbFABnst8LsidYs5Y/goodhart-taxonomy. Geo, Leo (2022). âClarifying wireheading terminologyâ. In:URL: https://w.alignmentforum.org/ posts/REesy8nqvknFFKywm/clarifying-wireheading-terminology. Greenblatt, Ryan (2023). âImproving the Welfare of AIs: A Nearcasted Proposalâ. In:URL: https: //w.lesswrong.com/posts/F6HSHzKezkh6aoTr2/improving-the-welfare-of-ais-a-nearcasted- proposal. Hebbar, Vivek and Evan Hubinger (2022). âPath dependence in ML inductive biasesâ. In:Alignment Forum.URL: https://w.alignmentforum.org/posts/bxkWd6WdkPqGmdHEk/path-dependence- in-ml-inductive-biases. 123 How many words are there in English? | Merriam-Webster.URL: https://w.merriam-webster.com/ help/faq-how-many-english-words. Hubinger, Evan (2019a). âGradient hackingâ. In:Alignment Forum.URL: https : / / w . alignmentforum.org/posts/uXH4r6MmKPedk8rMA/gradient-hacking. Hubinger, Evan (2019b). âInductive biases stick aroundâ. In:Alignment Forum.URL: https://w. alignmentforum.org/posts/nGqzNC6uNueum2w8T/inductive-biases-stick-around. Hubinger, Evan (2019c). âUnderstanding âDeep Double Descentâ. In:Alignment Forum.URL: https: //w.alignmentforum.org/posts/FRv7ryoqtvSuqBxuT/understanding-deep-double-descent. Hubinger, Evan (2020). âHomogeneity vs. heterogeneity in AI takeoff scenariosâ. In:Alignment Forum.URL: https://w.alignmentforum.org/posts/mKBfa8v4S9pNKSyKK/homogeneity-vs- heterogeneity-in-ai-takeoff-scenarios. Hubinger, Evan (2022a). âA transparency and interpretability tech treeâ. In:Alignment Forum. URL: https : / / w. alignmentforum . org / posts / nbq2bWLcYmSGup9aF / a - transparency - and - interpretability-tech-tree. Hubinger, Evan (2022b). âHow likely is deceptive alignment?â In:Alignment Forum.URL: https: //w.alignmentforum.org/posts/A9NxPTwbw6r6Awuwt/how-likely-is-deceptive-alignment. Hubinger, Evan (2022c). âMonitoring for deceptive alignmentâ. In:Alignment Forum.URL: https: //w.alignmentforum.org/posts/Km9sHjHTsBdbgwKyi/monitoring-for-deceptive-alignment. Hubinger, Evan (2022d). âSticky goals: a concrete experiment for understanding deceptive alignmentâ. In:Alignment Forum.URL: https://w.alignmentforum.org/posts/a2Bxq4g2sPZwKiQmK/sticky- goals-a-concrete-experiment-for-understanding. Hubinger, Evan (2023). âWhen can we trust model evaluations?â In:Alignment Forum.URL: https: //w.alignmentforum.org/posts/dBmfb76zx6wjPsBC7/when-can-we-trust-model-evaluations. Hubinger, Evan, Adam Jermyn, Johannes Treutlein, Rubi Hudson, et al. (2023).Conditioning Predictive Models: Risks and Strategies. arXiv.URL: http://arxiv.org/abs/2302.00805. Hubinger, Evan, Adam Jermyn, Johannes Treutlein, Rubi J. Hudson, et al. (2023). âConditioning Predictive Models: Making inner alignment as easy as possibleâ. In:Alignment Forum.URL: https://w.alignmentforum.org/posts/qoHwKgLFfPcEuwaba/conditioning-predictive-models- making-inner-alignment-as. Hubinger, Evan, Chris van Merwijk, et al. (2019).Risks from Learned Optimization in Advanced Machine Learning Systems. arXiv.URL: http://arxiv.org/abs/1906.01820. Hubinger, Evan, Nicholas Schiefer, et al. (2023). âModel Organisms of Misalignment: The Case for a New Pillar of Alignment Researchâ. In:Alignment Forum.URL: https://w.alignmentforum. org/posts/ChDH335ckdvpxXaXX/model-organisms-of-misalignment-the-case-for-a-new-pillar- of-1. Huffman coding(2023). In:Wikipedia.URL: https://en.wikipedia.org/w/index.php?title=Huffman_ coding&oldid=1175907130. Hutter, Marcus (2008). âAlgorithmic complexityâ. In:Scholarpedia3, p. 2573.ISSN: 1941-6016.DOI: 10.4249/scholarpedia.2573.URL: http://w.scholarpedia.org/article/Algorithmic_complexity. Inclusive fitness(2023). In:Wikipedia.URL: https://en.wikipedia.org/w/index.php?title=Inclusive_ fitness&oldid=1149059235. Inductive bias(2023). In:Wikipedia.URL: https://en.wikipedia.org/w/index.php?title=Inductive_ bias&oldid=1178867299. Irpan, Alex (2018).Deep Reinforcement Learning Doesnât Work Yet.URL: http://w.alexirpan.com/ 2018/02/14/rl-hard.html. Jaderberg, Max (2017).Population based training of neural networks. Google DeepMind.URL: https://deepmind.google/discover/blog/population-based-training-of-neural-networks/. Janus (2023). âHow LLMs are and are not myopicâ. In:Alignment Forum.URL: https : / / w. alignmentforum.org/posts/c68SJsBpiAxkPwRHj/how-llms-are-and-are-not-myopic. Karnofsky, Holden (2022a).AI Safety Seems Hard to Measure. Cold Takes.URL: https://w.cold- takes.com/ai-safety-seems-hard-to-measure/. Karnofsky, Holden (2022b). âAI strategy nearcastingâ. In:Alignment Forum.URL: https://w. alignmentforum.org/posts/Qo2EkG3dEMv8GnX8d/ai-strategy-nearcasting. Karnofsky, Holden (2022c). âHow might we align transformative AI if itâs developed very soon?â In:Alignment Forum.URL: https://w.alignmentforum.org/posts/rCJQAkPTEypGjSJ8X/how- might-we-align-transformative-ai-if-it-s-developed-very. Karnofsky, Holden (2023a). â3 levels of threat obfuscationâ. In:Alignment Forum.URL: https : //w.alignmentforum.org/posts/HpzHjKjGQ4cKiY3jX/3-levels-of-threat-obfuscation. 124 Karnofsky, Holden (2023b). âDiscussion with Nate Soares on a key alignment difficultyâ. In:Align- ment Forum.URL: https://w.alignmentforum.org/posts/iy2o4nQj9DnQD7Yhj/discussion-with- nate-soares-on-a-key-alignment-difficulty. Kenton, Zac et al. (2022a). âClarifying AI X-riskâ. In:URL: https://w.alignmentforum.org/posts/ GctJD5oCDRxCspEaZ/clarifying-ai-x-risk. Kenton, Zac et al. (2022b). âThreat Model Literature Reviewâ. In:URL: https://w.alignmentforum. org/posts/wnnkD6P2k2TfHnNmt/threat-model-literature-review. Krueger, David, Tegan Maharaj, and Jan Leike (2020).Hidden Incentives for Auto-Induced Distribu- tional Shift. arXiv.URL: http://arxiv.org/abs/2009.09153. Landau, Joshua (2022). âOptimality is the tiger, and agents are its teethâ. In:Alignment Forum.URL: https : / / w. alignmentforum . org / posts / kpPnReyBC54KESiSn / optimality - is - the - tiger- and - agents-are-its-teeth. Langosco, Lauro et al. (2023).Goal Misgeneralization in Deep Reinforcement Learning. arXiv.URL: http://arxiv.org/abs/2105.14111. Lanham, Tamera et al. (2023).Measuring Faithfulness in Chain-of-Thought Reasoning. arXiv.URL: http://arxiv.org/abs/2307.13702. Leike, Jan (2023).Self-exfiltration is a key dangerous capability. Musings on the Alignment Problem. URL: https://aligned.substack.com/p/self-exfiltration. Leike, Jan, John Schulman, and Jeffrey Wu (2022).Our approach to alignment research. OpenAI. URL: https://openai.com/blog/our-approach-to-alignment-research. LempelâZiv complexity(2023). In:Wikipedia.URL: https://en.wikipedia.org/w/index.php?title= Lempel%E2%80%93Ziv_complexity&oldid=1175701547. Longtermism(2023). In:Wikipedia.URL: https://en.wikipedia.org/w/index.php?title=Longtermism& oldid=1182123936. LRudL (2021). âUnderstanding and controlling auto-induced distributional shiftâ. In:Alignment Forum.URL: https://w.alignmentforum.org/posts/rTYGMbmEsFkxyyXuR/understanding-and- controlling-auto-induced-distributional. McCoy, R. Thomas, Junghyun Min, and Tal Linzen (2020).BERTs of a feather do not generalize together: Large variability in generalization across models with similar test set performance. arXiv. URL: http://arxiv.org/abs/1911.02969. Meta-learning (computer science)(2023). In:Wikipedia.URL: https://en.wikipedia.org/w/index.php? title=Meta-learning_(computer_science)&oldid=1176940930. Mingard, Chris (2020).Neural networks are fundamentally (almost) Bayesian. Medium.URL: https: //towardsdatascience.com/neural-networks-are-fundamentally-bayesian-bee9a172fad8. Mingard, Chris (2021).Deep Neural Networks are biased, at initialisation, towards simple functions. Medium.URL: https://towardsdatascience.com/deep-neural-networks-are-biased-at-initialisation- towards-simple-functions-a63487edcb99. Mingard, Chris et al. (2020).Is SGD a Bayesian sampler? Well, almost. arXiv.URL: http://arxiv.org/ abs/2006.15191. Ngo, Richard (2022).Richard Ngoâs Shortform. Less Wrong.URL: https://w.lesswrong.com/ posts/FuGfR3jL3sw6r8kB4/richard-ngo-s-shortform. Ngo, Richard, Lawrence Chan, and SĂśren Mindermann (2023).The alignment problem from a deep learning perspective. arXiv.URL: http://arxiv.org/abs/2209.00626. Occamâs razor(2023). In:Wikipedia.URL: https://en.wikipedia.org/w/index.php?title=Occam%27s_ razor&oldid=1184501671. Olah, Chris (2015).Visual Information Theory. Colahâs Blog.URL: https://colah.github.io/posts/2015- 09-Visual-Information/. Olsson, Catherine et al. (2022). âIn-context Learning and Induction Headsâ. In:URL: https : / / transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html. Omohundro, Stephen M. (2008). âThe Basic AI Drivesâ. In:Proceedings of the 2008 conference on Artificial General Intelligence 2008: Proceedings of the First AGI Conference. NLD: IOS Press, p. 483â492.ISBN: 978-1-58603-833-5.URL: https://selfawaresystems.files.wordpress.com/2008/ 01/ai_drives_final.pdf. Open Philanthropy AI Worldviews Contest(2022). Open Philanthropy.URL: https : / / w . openphilanthropy.org/open-philanthropy-ai-worldviews-contest/. Park, Peter S. et al. (2023).AI Deception: A Survey of Examples, Risks, and Potential Solutions. arXiv. URL: http://arxiv.org/abs/2308.14752. Patel, Dwarkesh (2023).Carl Shulman (Pt 2) - AI Takeover, Bio & Cyber Attacks, Detecting Deception, & Humanityâs Far Future.URL: https://w.dwarkeshpatel.com/p/carl-shulman-2. 125 Patel, Dwarkesh and Carl Schulman (2023).Carl Shulman (Pt 1) - Intelligence Explosion, Primate Evolution, Robot Doublings, & Alignment.URL: https://w.dwarkeshpatel.com/p/carl-shulman. Piper, Kelsey (2023).Playing the training game. Planned Obsolescence.URL: https://w.planned- obsolescence.org/the-training-game/. Power, Alethea et al. (2022). âGrokking: Generalization Beyond Overfitting on Small Algorithmic Datasetsâ. In:CoRRabs/2201.02177.URL: https://arxiv.org/abs/2201.02177. Regularization (mathematics)(2023). In:Wikipedia.URL: https://en.wikipedia.org/w/index.php? title=Regularization_(mathematics)&oldid=1177271127. Reimers, Nils and Iryna Gurevych (2018). âWhy Comparing Single Performance Scores Does Not Allow to Draw Conclusions About Machine Learning Approachesâ. In:CoRRabs/1803.09578. URL: http://arxiv.org/abs/1803.09578. RicĂłn, JosĂŠ Luis (2023). âThe situational awareness assumption in AI risk discourse, or why people should chillâ. In:Nintil.URL: https://nintil.com/situational-awareness-agi. Ringer, Sam (2022). âModels Donât "Get Reward"â. In:Alignment Forum.URL: https : / / w. alignmentforum.org/posts/TWorNr22hhYegE4RT/models-don-t-get-reward. Roger, Fabien and Ryan Greenblatt (2023).Preventing Language Models From Hiding Their Reason- ing. arXiv.URL: http://arxiv.org/abs/2310.18512. Rotating locomotion in living systems(2023). In:Wikipedia.URL: https://en.wikipedia.org/w/index. php?title=Rotating_locomotion_in_living_systems&oldid=1175051727. RSA numbers(2023). In:Wikipedia.URL: https : / / en . wikipedia . org / w / index . php ? title = RSA _ numbers&oldid=1183413403#RSA-2048. Schreiner, Maximilian (2023).GPT-4 architecture, datasets, costs and more leaked. THE DECODER. URL: https://the-decoder.com/gpt-4-architecture-datasets-costs-and-more-leaked/. Semiprime(2023). In:Wikipedia.URL: https://en.wikipedia.org/w/index.php?title=Semiprime& oldid=1154324640. Shah, Rohin (2019).Comment on: Understanding âDeep Double Descentâ.URL: https : / / w. lesswrong.com/posts/FRv7ryoqtvSuqBxuT/understanding-deep-double-descent. Shah, Rohin et al. (2022).Goal Misgeneralization: Why Correct Specifications Arenât Enough For Correct Goals. arXiv.URL: http://arxiv.org/abs/2210.01790. Shlegeris, Buck and Ryan Greenblatt (2023). âMeta-level adversarial evaluation of oversight tech- niques might allow robust measurement of their adequacyâ. In:Alignment Forum.URL: https: //w.alignmentforum.org/posts/MbWWKbyD5gLhJgfwn/meta-level-adversarial-evaluation- of-oversight-techniques-1. Skalse, Joar (2021).Comment on: Why Neural Networks Generalise, and Why They Are (Kind of) Bayesian. Less Wrong.URL: https://w.lesswrong.com/posts/YSFJosoHYFyXjoYWa/why- neural-networks-generalise-and-why-they-are-kind-of. Soares, Nate (2022). âA central AI alignment problem: capabilities generalization, and the sharp left turnâ. In:Alignment Forum.URL: https : / / w . alignmentforum . org / posts / GNhMPAWcfBCASy8e6/a-central-ai-alignment-problem-capabilities-generalization. Soares, Nate (2023a). âDeep Deceptivenessâ. In:Alignment Forum.URL: https : / / w . alignmentforum.org/posts/XWwvwytieLtEWaFJX/deep-deceptiveness. Soares, Nate (2023b). âWhat I mean by "alignment is in large part about making cognition aimable at all"â. In:Alignment Forum.URL: https://w.alignmentforum.org/posts/NJYmovr9ZZAyyTBwM/ what-i-mean-by-alignment-is-in-large-part-about-making. sphexish(2023). In:Wiktionary, the free dictionary.URL: https://en.wiktionary.org/w/index.php? title=sphexish&oldid=75894036. Steinhardt, Jacob (2022).ML Systems Will Have Weird Failure Modes. Bounded Regret.URL: https://bounded-regret.ghost.io/ml-systems-will-have-weird-failure-modes-2/. Team, Adept (2022).ACT-1: Transformer for Actions.URL: https://w.adept.ai/blog/act-1/. team, The AlphaStar (2019).AlphaStar: Mastering the real-time strategy game StarCraft I. Google DeepMind.URL: https://deepmind.google/discover/blog/alphastar- mastering- the- real- time- strategy-game-starcraft-i/. Turner, Alex (2022a). âInner and outer alignment decompose one hard problem into two extremely hard problemsâ. In:URL: https://w.alignmentforum.org/posts/gHefoxiznGfsbiAu9/inner-and- outer-alignment-decompose-one-hard-problem-into. Turner, Alex (2022b). âReward is not the optimization targetâ. In:Alignment Forum.URL: https: / / w. alignmentforum . org / posts / pdaGN6pQyQarFHXF4 / reward - is - not - the - optimization - target. 126 Universal Turing machine(2023). In:Wikipedia.URL: https://en.wikipedia.org/w/index.php?title= Universal_Turing_machine&oldid=1183200306. Valle-PĂŠrez, Guillermo, Chico Q. Camargo, and Ard A. Louis (2019). âDeep learning generalizes because the parameter-function map is biased towards simple functionsâ. In: ICLR 2019. arXiv. URL: http://arxiv.org/abs/1805.08522. Wabi-sabi(2023). In:Wikipedia.URL: https : / / en . wikipedia . org / w / index . php ? title = Wabi - sabi&oldid=1184445631. Weng, Lilian (2023).LLM Powered Autonomous Agents.URL: https://lilianweng.github.io/posts/2023- 06-23-agent/. Wheaton, David (2023). âDeceptive Alignment is <1% Likely by Defaultâ. In:Alignment Forum. URL: https://forum.effectivealtruism.org/posts/4MTwLjzPeaNyXomnx/deceptive-alignment-is- less-than-1-likely-by-default. Wilkinson, Hayden (2022). âIn Defense of Fanaticismâ. In:Ethics132, p. 445â477.ISSN: 0014-1704. DOI: 10.1086/716869.URL: https://w.journals.uchicago.edu/doi/abs/10.1086/716869. Wirehead (science fiction)(2023). In:Wikipedia.URL: https://en.wikipedia.org/w/index.php?title= Wirehead_(science_fiction)&oldid=1177998718. Wu, Xiaoxia, Ethan Dyer, and Behnam Neyshabur (2021). âWhen Do Curricula Work?â In:9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.URL: https://openreview.net/forum?id=tW4QEInpni. Xu, Mark (2020). âDoes SGD Produce Deceptive Alignment?â In:Alignment Forum.URL: https:// w.alignmentforum.org/posts/ocWqg2Pf2br4jMmKA/does-sgd-produce-deceptive-alignment. Xu, Mark (2021).Strong Evidence is Common. Artificially Intelligent.URL: https://markxu.com/ strong-evidence. Yudkowsky, Eliezer (2009). âValue is Fragileâ. In:Less Wrong.URL: https://w.lesswrong.com/ posts/GNnHHmm8EzePmKzPk/value-is-fragile. Yudkowsky, Eliezer (2016).Parfitâs Hitchhiker. Arbital.URL: https://arbital.com/p/parfits_hitchhiker/. Yudkowsky, Eliezer (2021a).Comment on: A positive case for how we might succeed at prosaic AI alignment. Alignment Forum.URL: https://w.alignmentforum.org/posts/5ciYedyQDDqAcrDLr/ a-positive-case-for-how-we-might-succeed-at-prosaic-ai. Yudkowsky, Eliezer (2021b).Comment on: Why Iâm excited about Debate.URL: https : / / w. lesswrong.com/posts/LDsSqXf9Dpu3J3gHD/why-i-m-excited-about-debate. Yudkowsky, Eliezer (2022). âAGI Ruin: A List of Lethalitiesâ. In:Alignment Forum.URL: https: //w.alignmentforum.org/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities. Yudkowsky, Eliezer.Context disaster.URL: https://arbital.com/p/context_disaster/. Yudkowsky, Eliezer.Corrigibility. Arbital.URL: https://arbital.com/p/corrigibility/. Yudkowsky, Eliezer.Dark Side Epistemology. Less Wrong.URL: https://w.lesswrong.com/posts/ XTWkjCJScy2GFAgDt/dark-side-epistemology. Yudkowsky, Eliezer.Extrapolated volition (normative moral theory). Arbital.URL: https://arbital. com/p/normative_extrapolated_volition/. Yudkowsky, Eliezer.Logical decision theories. Arbital.URL: https://arbital.com/p/logical_dt/?l=5kv. Yudkowsky, Eliezer.Omnipotence test for AI safety. Arbital.URL: https://arbital.com/p/omni_test/. Yudkowsky, Eliezer.Paperclip. Arbital.URL: https://arbital.com/p/paperclip/. Yudkowsky, Eliezer.Pivotal act. Arbital.URL: https://arbital.com/p/pivotal/. Yudkowsky, Eliezer and Richard Ngo (2021). âNgo and Yudkowsky on alignment difficultyâ. In: Alignment Forum2021 MIRI Conversations.URL: https : / / w. alignmentforum . org / posts / 7im8at9PmhbT4JHsW/ngo-and-yudkowsky-on-alignment-difficulty. Zychlinski, Shaked (2019).The Complete Reinforcement Learning Dictionary. Medium.URL: https: //towardsdatascience.com/the-complete-reinforcement-learning-dictionary-e16230b7d24e. 127