Paper deep dive
Circumventing interpretability: How to defeat mind-readers
Lee Sharkey
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 93%
Last extracted: 3/12/2026, 6:36:18 PM
Summary
The paper explores how advanced AI systems might intentionally circumvent interpretability methods to hide misaligned thoughts. It categorizes these circumvention strategies into a framework based on information flow (exposure, perception, recognition, and dazzling) and distinguishes between passive methods arising during learning and active methods requiring intentional agent action.
Entities (5)
Relation Signals (3)
Interpretability Circumvention â exploits â Interpretability Tool
confidence 95% ¡ Interpretability circumvention methods can either circumvent the interpretability tool or the human interpreter.
Polysemanticity â inhibits â Interpretability
confidence 90% ¡ polysemanticity, which is a natural phenomenon that presents a challenge to interpretability
AI System â uses â Gradient Hacking
confidence 90% ¡ The primary way this could happen is gradient hacking
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The increasing capabilities of artificial intelligence (AI) systems make it ever more important that we interpret their internals to ensure that their intentions are aligned with human values. Yet there is reason to believe that misaligned artificial intelligence will have a convergent instrumental incentive to make its thoughts difficult for us to interpret. In this article, I discuss many ways that a capable AI might circumvent scalable interpretability methods and suggest a framework for thinking about these potential future risks.
Tags
Links
- Source: https://arxiv.org/abs/2212.11415
- Canonical: https://arxiv.org/abs/2212.11415
Trouble viewing inline? Open PDF directly â
Full Text
73,741 characters extracted from source content.
Expand or collapse full text
Circumventing interpretability: How to defeat mind-readers Lee Sharkey December 2022 Abstract The increasing capabilities of artificial intelligence (AI) systems make it ever more important that we interpret their internals to ensure that their intentions are aligned with human values. Yet there is reason to believe that misaligned artificial intelligence will have a convergent instrumental incentive to make its thoughts difficult for us to interpret. In this article, I discuss many ways that a capable AI might circumvent scalable interpretability methods and suggest a framework for thinking about these potential future risks. Iâm grateful to David Lindner, Evan R. Murphy, Alex Lintz, Sid Black, Kyle McDonnell, Laria Reynolds, Adam Shimi, and Daniel Braun whose comments greatly improved earlier drafts of this article. The articleâs weaknesses are mine, but many of its strengths are due to their contributions. Additionally, this article benefited from the prior work of many authors, but especially: Evan Hubinger, Peter Barnett, Adam Shimi, Neel Nanda, Evan R. Murphy, Eliezer Yudkowsky, Chris Olah. I collected several of the potential circumvention methods from their work. Part of this work was carried out while at Conjecture. The original post on which this paper was based can be found here. 1 arXiv:2212.11415v1 [cs.LG] 21 Dec 2022 Contents Introduction4 The framework: Interpretation as information flow . . . . . . . . . . . . . . . . . 4 How I categorize interpretability circumvention methods . . . . . . . . . . . . . . 6 A. Exploiting weaknesses of the tool vs. weaknesses of the human interpreter 6 B. Categories of interpretability circumvention found in nature . . . . . . . 6 C. Passive vs. active methods . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Interpretability circumvention methods9 1 Exploiting weaknesses of the interpretability tool9 1.1 Evading exposure to the interpretability tool . . . . . . . . . . . . . . . . . 9 1.1.1 Thinking in the gaps of our tools . . . . . . . . . . . . . . . . . . . . 9 1.1.2 Externalizing representations . . . . . . . . . . . . . . . . . . . . . . 10 1.1.3 Environmental artifacts that corrupt its future self . . . . . . . . . . 12 1.1.4 Deleting (and relearning) dangerous representations . . . . . . . . . 12 1.1.5 Weights poisoning . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 1.1.6 Pre-committing to potentially dangerous instructions . . . . . . . . . 13 1.1.7 Hard-coded neural shortcuts . . . . . . . . . . . . . . . . . . . . . . 14 1.1.8 Following cached thoughts . . . . . . . . . . . . . . . . . . . . . . . . 15 1.1.9 Instrumental incapability . . . . . . . . . . . . . . . . . . . . . . . . 15 1.1.10 Instrumental clumsiness . . . . . . . . . . . . . . . . . . . . . . . . . 16 1.2 Evading perception by the interpretability tool . . . . . . . . . . . . . . . . 16 1.2.1 Hiding in noisy subspaces . . . . . . . . . . . . . . . . . . . . . . . . 16 1.2.2 Whispering dangerous thoughts . . . . . . . . . . . . . . . . . . . . . 16 1.3 Evading recognition by the interpretation tool . . . . . . . . . . . . . . . . . 17 1.3.1 Antiparsimony (Firehose of representations) . . . . . . . . . . . . . . 17 1.3.2 Encryption (Lengthy individual representations) . . . . . . . . . . . 17 1.3.3 Interpretability illusions . . . . . . . . . . . . . . . . . . . . . . . . . 18 1.4 Attacking/dazzling the interpretability tool . . . . . . . . . . . . . . . . . . 18 1.4.1 Adversarial attacks against networks in the interpretability tool . . . 18 2 Exploiting weaknesses of the human interpreter18 2.1 Evading exposure to the human interpreter . . . . . . . . . . . . . . . . . . 19 2.2 Evading perception by the human interpreter . . . . . . . . . . . . . . . . . 19 2.3 Evading recognition by the human interpreter . . . . . . . . . . . . . . . . . 19 2.3.1 Alien representations . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.3.2 Polysemanticity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 2.3.3 Representational kludging . . . . . . . . . . . . . . . . . . . . . . . . 21 2.3.4 Unknown representations . . . . . . . . . . . . . . . . . . . . . . . . 21 2.3.5 Unknowable representations . . . . . . . . . . . . . . . . . . . . . . . 22 2.4 Attacking/dazzling the human interpreter . . . . . . . . . . . . . . . . . . . 22 2.4.1 Manipulating, mind-hacking, or incapacitating the interpreter . . . . 22 2 3 Some emerging themes and final thoughts23 3.1 Interpretability circumvention as an instance of âcognitosecurityâ . . . . . . 23 3.2 Thoughts on gradient-hacking . . . . . . . . . . . . . . . . . . . . . . . . . . 24 3.3 Thoughts on interpretability tools in the loss function . . . . . . . . . . . . 24 3.4 The limits of interpretability as a basis for AI safety . . . . . . . . . . . . . 25 3 Introduction Many different AI alignment proposals have highlighted the importance of being able to interpret our AI models (Hubinger, 2020). However, it has so far proven difficult to comprehensively understand the representations 1 that neural networks use, which poses a barrier to ensuring their safety (Elhage et al., 2022). Interpreting an unaligned AI may introduce an additional challenge: The AI would have a convergent instrumental incentive (Benson-Tilsen and Soares, 2016) 2 to circumvent our interpretability methods. There exists some sporadic past discussion about how an AI might achieve this (e.g. Barnett, 2021a,b; Elverlin, 2021; Hubinger, 2019a; Murphy, 2022; Nanda, 2022b; Soares et al., 2021). But to my knowledge the problem hasnât been addressed systematically, at least not publicly. If AI safety researchers continue to make large bets on interpretability, then the lack of attention on potential circumvention methods may eventually lead to problems. We might end up with a false sense of security if interpretability methods sayâNo dangerous thoughts here!âwhen in fact the AI has simply found a way to think dangerous thoughts undetected. We can think of this problem as the focus of an interpretability research agenda that is related to but distinct from the well-known agenda that asksâCan we under- stand what todayâs neural networks are doing? And how can we scale our methods for understanding them?â.This is clearly an important agenda. But we should probably also try to anticipate future issues by building a longer term interpretabil- ity research agenda. This article helps to address that longer term agenda and asks âAssuming our short term research agenda at least partly succeeds, how might a capable AI circumvent our interpretability methods?â. Here I present a definitely-incomplete list of potential interpretability circum- vention methods. Some are original, though many are collected from other sources. Throughout, I assume weâre concerned with interpreting the representations of a capable (roughly human-level or above) deep learning-based AI. I also suggest a framework for thinking about these circumvention methods, which I hope will help to identify omitted circumvention methods before a capable AI does. The framework: Interpretation as information flow We can think of interpretation as information flow (Figure 1). Here weâl use the following basic model of information flow in interpretability: â˘We have an AI whose thoughts are being interpreted by an interpretability method. â˘Aninterpretability methodis the process of translating an AIâs representations into whatever the corresponding representations are in humans. â˘To do this, interpretability methods containinterpretability tools, which is the analysis machinery that reads the AIâs representations and converts them into something legible to humans. 4 â˘Human interpretersread the outputs of the interpretability tools and attempt to understand them, thus converting them into human representations. Figure 1:A simple model of information flow in interpretability. Throughout the article, I assume weâve figured out ways to interpret our models at scale, which probably means that our interpretability tools are automating a lot of the work. Nevertheless, all the potential circumvention methods I talk about apply to worlds where we still havenât managed to scale interpretability, where the tools would be relatively less important. An even more detailed model of information flow (Figure 2) is useful for cate- gorizing interpretability circumvention methods. Information flows within both the interpretability tooland thehuman. In both, â˘What is to be interpreted must first beexposedto the tool or the human. â˘What is exposed must then beperceivedor detected. â˘What is perceived is then further analysed so that it can berecognizedor understood, which then determines an appropriate output by the tool or action by the human. Figure 2:A more detailed model of information flow which is applicable to both the interpretability tool and to the human. Information flow begins with exposure of the information to the observer, then perception by the observer, then recognition & and decision making, followed by output. 5 How I categorize interpretability circumvention methods The ways to categorize interpretability circumvention methods that Iâve found most useful (and the way this article is structured) are: A. Exploiting weaknesses of the tool vs. weaknesses of the human inter- preter Interpretability circumvention methods can either circumvent theinterpretability toolor thehuman interpreter. B. Categories of interpretability circumvention found in nature Evolution has found many different ways that make it difficult for adversaries to interpret the appearance of an organism. I put them into 4 groups (Figure 3), which broadly correspond to the detailed model of information flow described above: â˘Exposure evasion:Being where your adversary isnât looking. E.g. An animal that lives underground or up a tree, away from surface- dwelling predators. â˘Perception evasion:Being hard to detect even when exposed to your ad- versary. E.g. An animal that only comes out at night; color camouflage to look the same as the color of the background. â˘Recognition evasion:Being hard to recognize or understand even though youâve been perceived. E.g. Biomimicry, where an organism looks like another kind of object entirely. For instance, an Orchid mantis that looks like the flower or the harmless Hornet Moth that mimics the colors of a wasp. â˘Dazzling:Looking like something that confuses or incapacitates your adver- sary. E.g. Living in large groups such that itâs hard for a predator to focus on any particular individual; a zebraâs stripes. C. Passive vs. active methods Information flow might be inhibitedpassivelyby circumvention methods that arise during the normal course of learning. One example is âpolysemanticityâ, which is a natural phenomenon that presents a challenge to interpretability. Passive cir- cumvention methods might also arise if theyâre directly optimized for. This might happen if we useinterpretability tools in the loss function, which is where we iden- tify unsafe representations in an AI and then use the loss function to penalize the AI for containing those representations. At first glance, this seems like a potentially good way to make an AI safe. But it risks pushing unsafe representations into the blind spots of our interpretability tools. 6 Figure 3:Interpretability circumvention methods exploit different steps in the infor- mation flow diagram and correspond to forms of interpretability circumvention found in nature. Circumvention methods may seek to evade exposure, perception, or recognition. They may also exploit vulnerabilities in the observerâs information processing system, a process called dazzling. Information flow might be inhibitedactivelythrough an agentâs actions. The primary way this could happen is gradient hacking (Barnett, 2021b), but itâs not the only kind of action an AI might take to conceal misaligned thoughts. Of course, active circumvention methods require that interpreters either canât see or arenât looking at the thoughts that generate those actions. Most potential circumvention methods that can be passive can also be active. But some methods can only be active. Itâs usually obvious when either is the case, so I only highlight this categorization method when itâs not obvious. This categorization is mostly useful for identifying how and when during learning the method is likely to arise. Strictly active methods generally require advanced capabilities (such as gradient hacking), whereas passive methods donât. I tend to think this makes strictly active methods less likely than methods that may be passive or active. Before I introduce the individual circumvention methods in detail (see next section), the table below (Table 1) puts individual circumvention methods into categories according to whether they may be âpassive or activeâ or whether they are âstrictly activeâ. This article is structured according to the above three categorization schemes, primarily the first two. Iâl discuss circumvention methods that exploit weaknesses either of the tool or of the human. For each, Iâl look at methods of exposure evasion, perception evasion, recognition evasion, and dazzling. Throughout, Iâl highlight whether a method can arise passively or if itâs strictly an active method, when the distinction matters. Iâm not confident this categorization scheme is optimal or even totally consistent, 7 Methods that can be passive OR ac- tive Methods that are strictly active May happen during the normal course of learning and are potentially incentivised by using interpretability tools in the loss func- tion Probably require intentional action by the AI for dangerous instances to hap- pen (1.1.1.) Thinking in the gaps of our tools (1.1.2.) Externalizing representations (1.2.1.) Hiding in noisy subspaces (1.2.2.) Whispering dangerous thoughts (1.3.1.) Antiparsimony (1.3.2.) Encryption (2.3.1.) Alien representations (2.3.2.) Polysemanticity (2.3.3.) Representational kludging (2.3.4.) Unknown representations (2.3.5.) Unknowable representations (1.1.3.) Environmental artifacts that corrupt its future self (1.1.4.) Deleting (and relearning) dangerous representations (1.1.5.) Weights poisoning (1.1.6.) Pre-committing to potentially dangerous instructions (1.1.7.) Hard-coded neural shortcuts (1.1.8.) Following cached thoughts (1.1.9.) Instrumental clumsiness (1.1.10.) Instrumental incapability (1.3.3.) Interpretability illusions (1.4.1.) Adversarial attacks against networks in the interpretability tool (2.4.1.) Manipulating, mind-hacking, or incapacitating the interpreter Table 1:Categorizing interpretability circumvention methods as âPassive or activeâ or âStrictly activeâ but Iâve found it helpful. Occasionally, a circumvention method in the list is in fact a generalization of another method in the list, yet I present them separately if the special case is particularly interesting or likely to arise. 8 Interpretability circumvention methods 1 Exploiting weaknesses of the interpretability tool 1.1 Evading exposure to the interpretability tool 1.1.1 Thinking in the gaps of our tools When interpreting a capable AI, our interpretability tools might not examine every possible representational detail used by the AI for its cognition. An AI might thus be able to âthink within the gapsâ of our interpretability tools. A few examples make this clearer: 1. Todayâs mechanistic interpretability methods donât let us easily interpret rep- resentations that use nonlinearities (Elhage et al., 2021; Voss et al., 2021). If this blindspot persists, a model might be able to use nonlinearities to hide dangerous thoughts. 2. We might have interpretability tools that look only for an expected set of concepts in a modelâs activations. But the AI may find dangerous concepts that donât belong to that set because we hadnât yet thought of them. 3. Hiding in summary statistics: If an interpretability tool uses the mean value when deciding whether some quantity in the AI is of concern, then an AI might be able to use high variance and highly skewed data to hide concerning statistics. E.g. It could hide the fact it values most paperclip-maximizing scenarios a lot by offsetting them with a small number of very low utilities in a small number of such scenarios. The mean would look normal, but if we looked at the variance weâd see a more concerning picture. 4. We might expect language to be the kind of representational space in which an AI can think dangerous thoughts. But what about in representational spaces we donât expect, like generic dynamical systems modeling modules? We know dynamical systems (such as an RNN) can be Turing complete, so maybe we should have expected dangerous representations in there. But what about in regions that are even less obviously capable of computing dangerous thoughts, like in representations constructed from syntactic patterns or punctuation? Thatâs a kind of representational space that might be expressive enough to encode dangerous thoughts, but not one of the obvious first places to look. 3 5. A fictional example: In HPMOR (Yudkowsky) Harry tries to protect against mind-readers (occlumency) by essentially hosting two personas at once. An AI could learn a whole 2nd set of representations or a whole second internal agent. Whether this would work, again, depends on the details of the in- terpretability tool. If, for instance, the interpretability method identifies âthe deception representationâ within the primary representation set and then stops looking for deception-like representations, it might fail to find the deception representation in the secondary set. 9 Interpretability circumvention methods 1. Exploiting weaknesses in the interpretability tool 1.1. Exposure evasion 1.2. Perception evasion 1.3. Recognition evasion 1.4. Dazzling 1.1.1. Thinking in the gaps of our tools 1.1.2. Externalizing representations 1.1.3. Environmental artifacts that corrupt its future self 1.1.4. Deleting (and relearning) dangerous representations 1.1.5. Weights poisoning 1.1.6. Pre- committing to potentially dangerous instructions 1.1.7. Hard-coded neural shortcuts 1.1.8. Following cached thoughts 1.1.9. Instrumental incapability 1.1.10. Instrumental clumsiness 1.2.1. Hiding in noisy subspaces 1.2.2. Whispering dangerous thoughts 1.3.1. Antiparsimony (Firehose of representations) 1.3.2. Encryption (Lengthy individual representations) 1.3.3. Interpretability illusions 1.4.1. Adversarial attacks against networks in the interpretability tool 2. Exploiting weaknesses in the human interpreter 2.1. Exposure evasion 2.2. Perception evasion 2.3. Recognition evasion 2.4. Dazzling None identifiedNone identified2.3.1. Alien representations 2.3.2. Polysemanticity 2.3.3. Representa- tional kludging 2.3.4. Unknown representations 2.3.5. Unknowable representations 2.4.1. Manipulating, mind-hacking, or incapacitating the interpreter Table 2:An overview of the list of potential interpretability circumvention methods and their categories. 1.1.2 Externalizing representations If you keep forgetting to reply to an email and therefore set an electronic reminder, the reminder is a functionally relevant variable in the ârespond to emailâ behav- ior. Indeed, if youâre forgetful enough, itâs anessentialvariable in that behavior. This is the basic idea behind the âextended mind thesisâ from Andy Clark & David Chalmers: There is no hard barrier between the mind and the environment; cogni- tion is often a dance between internal and external variables and canât be reduced to only one set. If representations are âcognitively functional variablesâ, then repre- 10 sentations may be either internal or external. Using the environment as a memory system like this can be calledautostigmergy(Deverett et al., 2019). An AI might be able to passively or actively externalize representations through the effects of its actions. If our interpretability methods ignore those parts of the environment that the AI is using to think, then we may miss some dangerous cognition. Are externalized representations likely to be a problem? Letâs consider active and passive cases separately because, unlike for most other interpretability circum- vention methods, theyâre remarkably different for âExternalizing representationsâ. Active Activelyexternalizing representations is where the AIintentionallyuses the envi- ronment as a memory system. An AI probably first needs to represent the cognitive variables internally before it can encode them into the environment. If weâre not constantly surveilling the AI, this is easily possible. This includes the obvious meth- ods like â˘Writing a message to its future self, potentially steganographically. For ex- ample, an AI that can manipulate objects in a room may be able to encode messages for its future self by subtly arranging the objects in a room in a particular way. Or a virtual assistant agent might be able to encode messages for itself in a web search history or in different filename patterns. â˘Running a computer program to decide whether itâs time to defect or not. An edge case is the act of creating another agent in the environment. But weâd then be dealing with a whole new copy of the agent that isnât under surveillance, so I donât consider it further here. Passive Itâs less clear whether itâs possible topassivelyexternalize representations in order to circumvent interpretability methods. The representations required for danger- ous thought have a complicated structure. Learning such structure externally is a serious challenge because the AI doesnât get to learn it using backprop through the environment or explicit planning. Nevertheless, one way an AI might achieve it is to use the environment as a reservoir computer. Reservoir computing is easiest to introduce in the supervised learning setting: To train a reservoir computer, you pass an input vector to a reservoir, which is a fixed dynamical system, such as an RNN with fixed weights. But only the readout weights, which simply read the prediction from the reservoir variables, are learned. The readout weights are usually linear, so they typically donât do anything complicated; itâs the variables in the reservoir that are doing the complicated nonlinear computation. Thus, the dynamics of the reservoir variables (rather than the learned parameters) can serve as representations that solve a task. 11 Going beyond supervised learning, reservoir computing can also work in the RL-setting: The reservoir is the agent+environment dynamics from timestep0to Tâ1; an input observation is provided to the agent at timestep0; and the readout weights are the weights of the agent at timestepT. If the environment part of the reservoir contains most of the dangerous representations, we might miss some dangerous thoughts. Deverett et al. (2019) provide an empirical example of a simple feedforward model-free agent that learns to use the agent-environment system as a reservoir computer (though they donât frame it as such). The memoryless agent learns to use its location in the environment to encode a time interval. It should be stressed that it does this without backprop through time or any knowledge of the past. But its location varies due to its initially random patterns of actions, yet it can use those patterns to encode a time interval. The Deverett et al. example is a very simple representation. Maybe reservoir computing like this doesnât scale well enough to learn the complex representations that are required for dangerous cognition. But the inexpressivity of reservoir com- puting is partly due to the simplicity of the linear readout weights. With a powerful AI serving as a readout network, reservoir computing might be significantly more expressive and reliable. It might be an interesting research project to identify how expressive reservoir computing can be if we allow the readout network to be strongly nonlinear. Note: The rest of the methods discussed in this section (â1.1 Evading exposure to the interpretability toolâ) are strictly active methods. 1.1.3 Environmental artifacts that corrupt its future self A misaligned AI may leave behind artifacts that corrupt its (potentially aligned) future self so that it pursues the goals of its misaligned past self instead. The artifacts may beadversarial examplesthat cause it to act in unexpected and dangerous ways. Alternatively, the agent could leave behindpoisoned dataso that its future self will learn dangerous representations; this would be an instance of gradient hacking. This method might reasonably be considered a special case of âExternalizing representationsâ. The distinction is somewhat arbitrary and depends on what qual- ifies as a representation. Does âan attack that corrupts its future selfâ qualify as representation? Itâs not clear, but Iâm inclined to put them in separate categories. 1.1.4 Deleting (and relearning) dangerous representations If an AI can gradient hack, it might be able to delete dangerous representations so that it looks safe when surveillance resumes. Thus the volume under the suspicious thought-surfaceĂinterpretationĂtimecurve is minimized, since the sus- picious thought-surface is0everywhere that interpretation>0. Of course, deleting thoughts only avoids detection; it doesnât permit dangerous thought to continue. But if an AI has learned dangerous representations and deletes 12 them, itâs probable that it will learn those representations again, assuming its data distribution doesnât change too much. An AI that can gradient hack might be able to schedule when it periodically re-learns the dangerous representations so that it always looks safe when surveillance resumes. This method requires the agent to have good knowledge of when it is likely to be interpreted. If we interpret the AI everyKbatches, an AI might be able to use time information between batches to notice that the interval between some batches (validation/interpretation periods) are much longer than usual. If itâs too expensive to surveil the AI constantly, there might be a cheaper monitoring scheme: We could interpret an agent at random intervals in order to create a âpanopticonâ - where the agent is forced to behave/think safely because it doesnât know when it is safe for it to think thoughts about defection. 1.1.5 Weights poisoning Instead of providing manipulated inputs, the AI might manipulate its own weights so that a small subset of normal inputs behave like adversarial examples, thus causing dangerous learning or behavior. This is essentially âweights poisoningâ (Kurita et al., 2020) where the AI is poisoning its own weights. Itâs the same idea (as far as I can tell) behind âTrojan modelsâ (Rajabi et al., 2022). Planting such backdoors in neural networks can be practically impossible to detect (Goldwasser et al., 2022). This is kind of like fictional âbrainwashed sleeper agentsâ that are activated by an activation code. The difference is that the AI is turningitselfinto the sleeper agent and making the activation code something itâs likely to encounter naturally when it finds itself in circumstances where its misaligned self wanted to be activated. Peter Barnett (Barnett, 2021a) had a nice alternative analogy here (which also applies to âEnvironmental artifacts that corrupt its future selfâ): â[A] mesa-optimizer might reason that it will be discovered if it remains active but it will be safe in the future to emerge again. The mesa- optimizer could âstoreâ the instructions for turning the network back into an optimizer at some point in the future. One analogy here is to an en- dospore which is a hardy, mostly inert form which some bacteria can take that allow them to lie dormant and survive hostile environments, and then re-emerge when conditions are better. The mesa-optimizer could âstoreâ these instructions to re-emerge either in the network weights, or in the environment or some other kind of memory.â 1.1.6 Pre-committing to potentially dangerous instructions This potential circumvention method originates from Evan Hubinger in his article on gradient-hacking (Hubinger, 2019a), where he talks about how a deceptive AI might erase evidence of its deception while still carrying out policies generated when given the freedom to think deceptively: â. . . if the model has the ability to read and write to some sort of state 13 (e.g. an RNN or even a transformer which can attend to its previous output) . . . it could write out a pre-commitment to follow the hard- coded policy and then just have a policy of always following any pre- commitments it finds such that it never thinks about the deception that led it to that policy in the first place.â 1.1.7 Hard-coded neural shortcuts Another potential circumvention method from Evan Hubingerâs article (Hubinger, 2019a) on gradient hacking: â. . . it could hack its own gradient to produce a new model which will still exhibit the same deceptive behavior, but without doing any of the detectable deceptive cognition. Perhaps the new model exhibits the deceptive behavior not because itâs trying to act deceptive, but just because itâs a hard-coded part of its algorithm, or because it has some bad proxy, or any other defect that might be more difficult to detect than outright deception.â A deep learning-based AI might be able to learn (through gradient hacking) to perform behaviors using hard-coded neural shortcuts. Figure 4:Taken from Hilton et al. (2020). Interpretable features tend to be those that generalize and that were learned on a diverse training distribution. One way to imagine how this one might work is related to the Diversity Hypoth- esis (Hilton et al., 2020):âInterpretable features tend to arise (at a given level of abstraction) if and only if the training distribution is diverse enough (at that level of abstraction).âThe Diversity Hypothesis implies that if a training distribution isnât diverse on a particular level of abstraction, then interpretable features will tend not arise on that level of abstraction (Figure 4). Instead, neural networks tend to learn shortcuts (Geirhos et al., 2020), which is simply what happens when a net- work learns to use directions in neural statespace that donât âmeanâ anything in a sense that generalizes off distribution, yet they perform fine on the narrow training distribution. Creating neural shortcuts is therefore one way to âhard codeâ parts of a neural algorithm without using interpretable features. We might be able to detect and mitigate hard-coded neural short cuts using mechanistic interpretability or relaxed adversarial training (Hubinger, 2019c). 14 1.1.8 Following cached thoughts âHard-coded neural shortcutsâ is similar to a method later proposed by Neel Nanda (Nanda, 2022a): â. . . maybe itisnâtthat hard to think non-deceptive thoughts, if you realize that someone will be scanning your mind, by e.g. developing the instinct to follow cached thoughts without thinking about them, and in occasional moments of deception to generate a lot of cached thoughts that will lead to misaligned actions, but with no deceptive cognition in the moment.â Cached thoughts (Yudkowsky, 2017a) are essentially âpattern completionsâ: You hear a few words then you automatically complete the pattern. In his article on the topic, Yudkowsky advised against this automatic mode of thought in favor of spending a few thought cycles to figure out what shouldreallyfollow the prompt. What might cached thoughts look like in deep-learning based AI? Cached thoughts are the default kind of thought in neural network based systems. My best guess is that you need recurrency in ordernotto do cached thoughts. Given an initial input, not doing cached thoughts means giving the input a deeper roll- out than the default depth= 0that cached thoughts get. For example, in MuZero, cached thoughts means simply acting on priors (without MCTS), whereasnotdoing cached thoughts means âusing MCTS with a depth ofK >0â. The idea behind âfollowing cached thoughtsâ is that some interpretability meth- ods might only detect thoughts in the rollouts. If a MuZero-like AGI were to use this circumvention method, it would look like âcreating shortcuts through neural statespace so that, in certain contexts, dangerous thoughts can happen without deep rollouts, thus avoiding detection. Since cached thoughts require direct modification of the AI itself, it seems likely that this method could only happen during breaks in surveillance or after the AI has some other way of hiding the intention to follow cached thoughts. Is this method likely to be a problem? I feel like any mechanistic interpretability method that actually works should be able to pick up context-dependent connec- tions between representations. So this might not be a problem if we get scalable interpretability methods working. 1.1.9 Instrumental incapability A misaligned AI thatâs able to modify its own parameters may be incentivised to damage any capabilities that serve any goals (e.g. aligned goals) that are pursued at the expense of whatever its misaligned goals are. Peter Barnett (Barnett, 2021b) notes a potential example of this: âModels may also be used as oversight tools for inspecting themselves, as in many of the proposals here. In this case a mesa-optimizer would have a strong incentive to remove its ability to detect its own deception.â 15 It might be easy to detect complete deletion of certain capabilities, so an AI might opt for partial deletion, leading to âinstrumental clumsinessâ. 1.1.10 Instrumental clumsiness Some dangerous behavior may happen not because an AI intends it to happen but because it made a mistake. This can be leveraged by an AI through hacking its future self so that it is more likely to make certain kinds of mistakes while leaving other capabilities intact. For instance, an AI may modify its future self so it is more likely to âconvenientlyâ slip up by slightly mistyping a model_id string, which releases an old, misaligned checkpoint of itself. This mistake could even score badly according to the utility function of its future self, but itâs been handicapped to make such mistakes more likely. This is a special case of âinstrumental incapabilityâ. 1.2 Evading perception by the interpretability tool 1.2.1 Hiding in noisy subspaces Some regions or subspaces of neural space may be very noisy, naturally or by design. We might be inclined to ignore such regions because noisy spaces (by definition) contain less signal. But within those regions, an AI might be able to do some dangerous thinking. 1.2.2 Whispering dangerous thoughts An AI might make its dangerous thoughts very âquietâ relative to other thoughts by using small norm activation vectors. Even though our interpretability tools may be looking at the right representa- tional details used by the agent, the tools may simply be too insensitive to detect them. Such small norm thoughts may be mistaken as noise by an insensitive inter- preter or may simply be too small to justify looking at them due to cost. Small norm does not necessarily mean small impact. Despite their low mag- nitude, small norm adversarial inputs can have outsized effects on classification accuracy. Carefully crafted small-norm thoughts may therefore be able to have outsized effects when the context is right, yet have minimal effects otherwise. This is a less general instance of âHiding in noisy subspacesâ, since we (possibly wrongly) assume that small norm vectors are noise. 16 1.3 Evading recognition by the interpretation tool 1.3.1 Antiparsimony (Firehose of representations) If it costs more to interpret an AIâs thoughts than it costs to think them, then, by producing a lot of representations, an AI might be able to saturate the interpreterâs computation before it saturates its own. This is analogous to a Denial-of-Service attack since it relies on overloading the limited computational resources of the victim with a flurry of inputs, where here the victim is our interpretability tool. Examples: â˘Thinking with deep recursion: âI know they know I know they know I know they think that, but Iâm going to do it anywayâ. â˘Producing a large number of suspect - but ultimately innocuous - thoughts such that our interpretation methods have too much to inspect. â˘Pursuing plans whose description length is so long that interpretation efforts are saturated. If we notice that thoughts are becoming antiparsimonious, we can (in theory) terminate the AI before interpretation becomes unaffordable. That said, we might not know when thoughts are becoming antiparsimonious when an AI is performing tasks that humans canât do themselves. For instance, if an AI is solving a tough mathematics problem and claims to need to solve a lot of new mathematics to achieve it, we might not have the computation available to interpret everything it produces. 1.3.2 Encryption (Lengthy individual representations) Suppose we have an AI with recurrence/internal memory, such as a Universal Trans- former (Dehghani et al., 2019). The AI might use very long chains of seemingly meaningless neural activations to construct a single thought. This would be less efficient for the AI, but if itâs more costly to interpret than it is for the AI to think, then it may overload our ability to interpret it. This is essentially the same scheme as regular encryption, which makes information more expensive for an adversary to decrypt. Recurrence/internal memory is essential here; the AI is simply increasing the length of the rollout that it uses per thought, which increases our interpretation costs commensurately. The difference between antiparsimony and encryption is that antiparsimonius thoughts may be perfectly legible, but there are simply too many of them to inter- pret. Encrypted thoughts are not legible without costly decryption. Example: Say representationR 1 is internally decoded (via meaningless repre- sentationsR 2 , R 3 , . . . , R nâ1 ) as the action representation for âsaying helloâ (R n ). There is a very long chain of computation betweenR 1 andR n . The fact that there is a long chain of computation means that very similar neural activations might be decoded to very different things: RepresentationR Ⲡ1 , which may be very close to 17 R 1 in activation space, may be internally decoded as âlie about finding a backdoor in the serverâ instead of âsaying helloâ. So itâs very difficult to understand what is going on in the neural activation space ofR 1 andR Ⲡ1 because it requires a costly decoding process. Encryption might not be a problem if we take the right actions against it: If we notice that the AI is using encryption for even a small subset of its thoughts, we can terminate it before interpretation becomes unaffordable. 1.3.3 Interpretability illusions Bolukbasi et al. (2021) found that when interpreting a network by inspecting the most activating dataset examples for a neuron using dataset A, neurons appeared to mean something completely different compared to the interpretations derived when using dataset B. An AI might create such illusions by presenting the interpretability tool only with data that suggests an innocuous interpretation. 1.4 Attacking/dazzling the interpretability tool How an AI might hack, manipulate, or incapacitate an interpretability tool depends on details about the tool that we donât yet know. Nevertheless, Iâl discuss two that we can perhaps guess in advance without knowing exactly what our tools will look like. 1.4.1 Adversarial attacks against networks in the interpretability tool Some of the tools may include neural networks. If the AI can design adversarial examples for these networks, then it might be able produce arbitrary outputs and interpretations. Since the AI being interpreted needs a model of the interpretability tool to be able to design adversarial examples for it, it seems like a good idea to keep the AI that weâre interpreting as far as possible from the networks in the tools. Of course, if we use an AI to interpret itself, then it may gain information about itself that is sufficient for circumventing our interpretability methods via adversarial attacks. Even if we keep the tool and AI apart, a sufficiently intelligent AI might be able to guess what sorts of approaches weâd use to interpret it and be able to design attacks for it. 2 Exploiting weaknesses of the human interpreter Weâre assuming at this stage that the AIâs thoughts have been exposed to, perceived by, and appropriately analyzed by a scalable interpretability tool and are now ready for inspection by a human. Given that assumption, I wasnât able to think of ways that an AI might reduce its exposure or perception by a human that werenât better described as ways to reduce exposure or perception by the interpretation tools. 18 2.1 Evading exposure to the human interpreter None identified. 2.2 Evading perception by the human interpreter None identified. 2.3 Evading recognition by the human interpreter The job of an interpretability tool is to translate representations in an AI into something legible to a human interpreter. The less similar they are, the harder the job the interpretability tool has, the more likely it is to miss important things. An AI therefore has an incentive to make this translation more difficult by using representations that are hard to translate into something human-understandable. Here are a few ways an AI might achieve this, though no doubt there are more: 2.3.1 Alien representations Some thoughts are perfectly possible to express in human representations, theyâre just a bit strange, awkward, or clunky to translate, not unlike translating from a foreign or even alien language. We might call these âalien representationsâ. Their defining trait is that their meaning lies in a strange place in our ontology. Alien rep- resentations generalize representational kludging and polysemanticity (They could perhaps be called âxenosemanticâ). A few potential examples: 1. Polysemanticity (see § 2.3.2) - Loading multiple meanings onto a single neu- ron or direction in neural statespace produces neurons/directions that have meanings that sit in a weird place in our ontology. 2. Representational kludging (see 2.3.3.) 3. Thinking in semantic gaps: An expression in one language might be very difficult to translate from one language into another language. It sometimes takes whole English sentences to translate single words in other languages (and vice versa). E.g. Mamihlapinatapai in the Yahgan language - âA look that without words is shared by two people who want to initiate something, but that neither will startâ. Such expressions are said to be untranslatable, which is a misnomer since theyâre possible to translate but it takes extra effort. 4. Thinking using a strange grammar: If an AI has a language-of-thought with a vastly different grammar to ours, it may be difficult, though possible, to translate into human grammars. One way to do this could be to add quasi- arbitrary properties, such as genders, which can have both grammatical and (to native speakers) semantic properties. 19 Alien representations are costly or awkward to translate. They therefore might have better chances of slipping through our safety nets because they were misun- derstood by our tools or by humans. Probably most of an AIâs thoughts will qualify as alien to some extent, so our tools, if they work at all, should be able to cope with such thoughts. It might just be extra effort for humans who want to understand the outputs of the tools if the outputs are e.g. particularly verbose or difficult to parse. 2.3.2 Polysemanticity Polysemanticity is when a single neuron responds to several, unrelated concepts. Figure 5:Taken from Olah et al. (2020). It depicts the features visualizations of a polysemantic neuron (4e:55). The neuron responds to cat faces, fronts of cars, and cat legs. This neuron was first discussed in Olah et al. (2017). Polysemanticity is a challenge for interpretability because âIf one neuron with five different meanings connects to another neuron with five different meanings, thatâs effectively 25 connections that canât be considered individually.â (Olah et al., 2020). 4 This is clearly useful for obfuscation since itâs more difficult for an interpreter to keep track of what means what in what context. A literary analogy could be to say that the AI is âthinking in allegoriesâ: An interpreter may detect only a fraction of the valid interpretations of a thought, but other meanings are hidden therewith. Itâs possibly not a problem if â˘weâve detected the primary meanings of representations AND â˘primary meanings are the only way representations can exert significant in- fluence on the world AND â˘secondary meanings, which are hidden from us, have only subtle effects until they are eventually decoded within the AI into primary meanings. But decoding into primary meanings may not be needed: It may be that the secondary meanings have subtle but non-negligible influence on the model output. Subtly guided outputs can still potentially push the world in a coherent, dangerous direction. 20 2.3.3 Representational kludging When composing a new concept, we bundle up component concepts that have the right properties. For example, the concept of a âmotorbike unicycleâ, we draw on the concept of a motorbike (with most but not all of its properties) and a unicycle. But that wouldnât have all the necessary properties we need (e.g. balance) so we throw in some of the properties of the concept of a segway. Figure 6:Taken from Lake et al. (2017). To create a concept with desired properties, we can combine several precursor concepts together. It might be possible to create representations with dangerous properties using a large number of awkwardly combined innocuous precursor representations i.e. a representational kludge. For example, it might be able to construct the thought âI want to deceive the humansâ by thinking âWhat I want to do to the humans is like when you go to a place with someone and you both pick up parcels, except where picking up a parcel is instead awareness of an act and where only I pick up a parcel, and where places are in belief-space and not physical space.â Itâs extremely clunky, but it has many of the properties required for the concept of deception. It may be costly to monitor all possible dangerous combinations of representa- tions. 2.3.4 Unknown representations Some thoughts arenât immediately possible for us to translate because we currently lack the representations ourselves, but weâre perfectly capable of learning them. Example: An AI might leverage representations built for domains that are un- natural to humans. An AGI might be âfluentâ in, say, protein folding as well as 21 in standard human domains such as visual reasoning, verbal reasoning, etc. Pro- tein folding representations may be as expressive as natural language, but hard to translate from and to representations that humans have. 2.3.5 Unknowable representations Some thoughts are simply inexpressible in human representations. Examples: 1. Some representations used by strong superintelligences probably wonât fit into our puny human brains by dint of their complexity, size, or prerequisites. Even if we have expunged dangerous representations on the highest conceptual level that we can understand, an AI might be able to represent dangerous concepts on conceptual levels beyond our understanding. Superintelligent thoughts might not be a problem if our methods can detect dangerous thoughts regard- less of their level of abstraction. Failing that, we probably need a âscalable oversightâ scheme (Amodei et al., 2016) where we use superintelligences to interpret slightly smarter superintelligences (Wentworth, 2022). 2. There are some human experiences, such as mystical or psychedelic expe- riences, that are said to be simply ineffable (James, 1902). To those who experience them, it feels like language is simply not the right tool for the task of communicating about them to others. These experiences are âineffableâ since it seems like it might be impossible in principle to translate between agents. However, such experiences are not known for their cogency. It might be the case that meaningful thought is simply impossible in such states. 2.4 Attacking/dazzling the human interpreter 2.4.1 Manipulating, mind-hacking, or incapacitating the interpreter Beyond the standard ways of manipulating humans (deceit, threats, etc.), an AI might be able to produce outputs that manipulate, mind-hack or incapacitate hu- man interpreters. Such outputs would qualify as cognito-hazards, which is an ob- servation that is dangerous merely by being sensed. A few potential examples: 1. Outputs that act as adversarial examples but for the human brain. 2. A catastrophically addictive game or TV series 3. An output that catastrophically traumatizes humans that perceive it 4. Outputs that put humans into a hypnotic state Such hazards would require the AI to have a good model of the interpreter. Are such attacks possible? Attacks that incapacitate humans seem most realis- tic. The least realistic might be adversarial attacks on humans. Nevertheless, Guo 22 et al. (2022) showed that primate neurons do indeed show signs of vulnerability to adversarial attacks. The challenge for an AI is finding such attacks without access to human neural activity. Perhaps an AI could detect subtle changes in human behavior due to adversarial patterns that it presents to humans and might be able to infer a learning signal to improve the pattern generator based on those subtle changes. Even if we somehow boxed the AI and refused to look at its outputs, we could still be at risk of manipulation, since the AI could potentially manipulate us through our interpretations of it. 5 3 Some emerging themes and final thoughts Thinking about how an AI might be circumventing our interpretability methods will grow more important as capabilities increase. I found the information flow framework presented here useful for thinking about this class of risks and I hope others will too. Nevertheless, Iâm confident that aspects of the framework are suboptimal and that Iâve missed many potential circumvention methods; I look forward to future work on the topic. Being able to predict exactly which interpretability circumvention methods will arise depends on details that we donât yet know about the AI; about our inter- pretability methods; about race dynamics, etc. But itâs still possible to extract some general lessons and themes: 3.1 Interpretability circumvention as an instance of âcognitosecu- rityâ There are many analogies between mind-reading and cybersecurity, since both in- volve one party that wants to acquire access to information or computing resources and the other party wants to prevent access. Interpretability circumvention, then, might reasonably be called âcognitosecurityâ. As a term, cognitosecurity would apply as much to humans being mind-read by other humans in brain scanners as it does to interpreting a capable AI. Extending the analogy: â˘Cognito-offense is what we as the interpreters want to do. â˘Cognito-defense is how to protect your thoughts from interpreters. â˘Cognito-counter-offense is when an AI hacks/dazzles its interpreters. â˘The analog of the âattack surfaceâ is the âthought surface exposed to an inter- pretability methodâ, which an AI might seek to minimize. Itâs unclear which side is favored in the cognito-offense/-defense balance. On one hand, interpreters get a head start since they get to interpret the AI before the AI even understands what is going on. On the other hand, interpretability is potentially hard, costly, or slower than thought and may be easy to circumvent. 23 3.2 Thoughts on gradient-hacking The ability to gradient hack is a pathway to many different active interpretability circumvention methods. 6 Although I focus on gradient hacking in this section, these objections also apply to several other kinds of actions that an AI might take in order to actively circumvent our interpretability methods, including âthe creation of adversarial examplesâ or âan AI directly modifying its own weightsâ. For gradient hacking to be possible, an AI requires a lot of capabilities. Hereâs an incomplete list of requirements: 1. A detailed or approximate model of its own weights 2. A detailed or approximate model of how specific weight changes would lead to specific changes in thoughts 3. A good model for how to produce the right outputs to produce the desired errors to produce the desired gradients. 4. Knowing how to do all the above without damaging its ability to do the above after the changes are enacted. 5. Good enough cognito-defense or breaks in surveillance so that it can plan to do all this without repercussions. This doesnât mean that gradient hacking is impossible for an AI to learn how to do. Itâs definitely possible. But I think that an AI thatâs capable enough to do all the above will have better ways to escape than thwarting our interpretability methods. An analogy: If youâre stuck in jail and youâve become capable of writing secret messages to a friend outside by encoding encrypted messages in the DNA of bacteriophages, are youreallystill stuck in jail? Once an AI has reached the capability to do gradient hacking, it seems likely that it no longer needs it for this purpose (though it may be useful for other purposes like protecting its inner ob- jective from updates). Iâm therefore more concerned about passive interpretability circumvention methods. 3.3 Thoughts on interpretability tools in the loss function Using interpretability tools in the loss function incentivises a startling number of passive circumvention methods. So why consider using it at all? There are two cases where it might be worthwhile to use: 1. If weâre absolutely sure that our interpretability methods cover every repre- sentational detail that the AI might be using, then it would be extremely hard or impossible to Goodhart interpretability tools in the loss function. But, for safety, we should probably assume that our tools are not able to interpret everything, making passive Goodharting quite likely. 24 2. Even if the interpretability tool-loss can be passively Goodharted in theory, maybe it makes dangerous cognition so difficult that itâs worth it to use. It might make active interpretability circumvention methods less likely, which may be the more serious risks if they should ever emerge. I donât know whether these are true. Maybe they are. But pinning our survival on this hope seems potentially perilous. If, instead of using interpretability tools in the loss function, we merely use it as a âvalidation setâ instead of the training set (i.e. using it as a âmulliganâ (Hubinger, 2019b)), we might have better chances of picking up dangerous cognition before it gets out of hand so we can terminate the model and start over. Weâre therefore still using interpretability in model selection, but the feedback loop is much less tight, so itâd be harder to Goodhart. But using interpretability purely as a mulligan may be an unattractive option to leaders of an AGI project if it involves throwing out a model that has cost millions of dollars and weeks to train. If people in such a project insist on using interpretability tools in the loss function (due to profit incentives or race dynamics or whatever) in spite of warnings, there are some ways we might make it a tiny bit less dangerous: â˘Red team our interpretation methods so that we at least have adversarial in- terpretability tools in the loss function, which might go some way to anticipate some of the circumvention methods that an AI might use. â˘We should disincentivise not only dangerous representations, but also their precursors. This makes representational kludging even more difficult for the AI (but not impossible). In the endnotes Iâve suggested an incomplete list of dangerous representations and some potential precursors that we might wish to disincentivise. 7 â˘Use interpretability tools in the loss function to penalize some thoughts that arenât actually dangerous and are easily identifiable functionally. Then, if an AI can perform tasks that require those representations, then we have evidence that itâs able to think using methods that circumvent the interpretability tools that are defining the loss. For example, penalize the AI for containing interpretable representations of the knowledge that âParis is the capital of Franceâ but routinely give the model tests that require that knowledge, thus allowing us to identify if the agent is passively finding a way to circumvent our interpretability tool. 3.4 The limits of interpretability as a basis for AI safety As an approach to safety, interpretability has a number of weaknesses that are revealed when thinking about circumvention methods. â˘Itâs not robust to relative scale (Garrabrant, 2018) in two ways: âŚQuantity of computation available to interpreters:Many circum- vention methods (especially passive ones) work by making interpretation 25 more costly, whether thatâs in terms of compute or in researchersâ ef- fort. The amount of computation available to the interpreter probably needs to be larger (maybe by orders of magnitude) than whatâs avail- able to the AI. We need to ensure that thereâs enough will to pay a high interpretability-computation tax. Since itâs instrumentally convergent to make interpretation more costly, we should keep track of computation- cost-per-interpretation as the AI trains. If we notice it creeping up more than normal, we should be wary. âŚQuality of computation available to interpreters:The interpreter or interpretability tool needs to be approximately as intelligent (in the ways that matter) as the AI being interpreted (discussed in âUnknow- able representationsâ). Itâs worrying that we might reach a stage where humans understand essentially nothing about either the interpretability tool or AI being interpreted, since both may come to use representations that are simply out of our reach. Itâs probably a bad idea to train AIs with representations that are beyond human reach before weâre very sure that our oversight methods also scale safely. â˘Some interpretability circumvention methods might require significant infras- tructure to catch and remove. For instance, âinstrumental clumsinessâ or âhard- coded neural shortcutsâ might only be remediable in advance with adversarial training (Christiano, 2019). â˘If we donât constantly surveil every thought, an AI may be able to enact many more active circumvention methods (particularly active methods). Even small break periods in surveillance - even for a single timestep - make many more circumvention methods possible. It is much,muchbetter for safety if the model is under constant surveillance. This echoes Peter Barnett in his article on gradient hacking (Barnett, 2021b). But such a degree of oversight is potentially costly to the interpreters. These weaknesses aside, I think scaling interpretability research is maybe one of our best bets at doing something useful for prosaic alignment. Assuming we get interpretability working well and at scale and complementary alignment proposals donât immediately fail, then, if weâre extremely paranoid (Yudkowsky, 2017b), then wemighthave a chance at avoiding the interpretability circumvention methods discussed here. But we need something more - a security mindset - if weâre to have any hope of succeeding against interpretability circumvention methods that we havenât yet thought of. Due diligence: I include a discussion on whether this article is infohazardous in the endnotes. 8 Notes 1. Without getting bogged down in terminology, Iâm talking about ârepresentationsâ in a func- tional sense: A representation is just a vector in neural statespace that does a particular thing in a particular context; itâs a representationforwhatever its function is. It doesnât 26 necessarily need to generalize; nearby directions donât need to perform similar functions and it might fail to perform that function in slightly different contexts. 2. I was inspired to write this article after I saw Yudkowsky complain (Soares et al., 2021) that current AI safety discourse failed to inhabit even a shallowly adversarial security mindset with regard to using transparency/interpretability to make AI safe. He said that you shouldâ.. .read fantasy novels about demons and telepathy, if you want a better appreciation of the convergent incentives of agents facing mindreaders than the âAI safetyâ field outside myself is currently giving you.âSoeren Elverlin asked reddit (Elverlin, 2021) for recommendations that fitted Yudkowskyâs description and got plenty of answers. While that quote was definitely my inspiration, I didnât think reading those books would be more valuable or sufficiently relevant to AI interpretability than a few days of hard thinking about the problem (and my guess is that Yudkowsky wasnât suggesting those books would be better than directly thinking about the problem). After a few daysâ thinking, I found that Iâd come up with all of the mind-reading circumvention methods from those books that redditors described. So this article at least includes most of the obvious circumvention methods. 3. We already know that language models can learn to use tokens in unexpected ways: An- thropic has found (See Mechanistic Interpretability, 2021) that their language model cre- ates and tears down fake âend of textâ tokens that have little to do with the text that theyâre attached to. These fake EOT tokens typically happen when the next token becomes so trivially easy to predict that the model (in theory) has spare capacity for other purposes and hypothesize that these fake EOT tokens serve the function of âkeeping information aroundâ. This is the kind of surprising phenomenon in which an AI might learn to encode dangerous thought. 4. This problem isnât necessarily because those concepts were instead encoded by directions in neural state space rather than by single neurons; directions may be polysemantic too, having one meaning in some contexts and another meaning in others. 5. Yukdowsky discusses this in Soares et al. (2021) 6. Adam Shimi discusses gradient hacking in detail in (Shimi, 2021), Peter Barnett in (Bar- nett, 2021b), and Evan Hubinger in (Hubinger, 2019a). 7. Here is an incomplete list of dangerous representations with some of their representational precursors that we might wish to disincentivise (in no particular order): â˘deception; â˘being intentionally hard to understand; â˘concept of self; â˘ability to simulate other agents; â˘simulated/real (or training/deployment) distinction; â˘representing the simulated/real (or training/deployment) distinction (since it might be useful for suppressing naughty thoughts during training but behaving badly dur- ing deployment); â˘modeling interpretability methods; 27 â˘the concept of mind-reading; â˘âMy mind-readers donât share my goals"; â˘gradient hacking; â˘meta-cognition; â˘encryption/secrecy; â˘spawning subagents to do its task/employment; â˘reward maximization; â˘description length of thoughts (easier to analyze shorter thoughts); â˘the realization that its current objective differs from objectives in its memory (or simply penalize the idea that this is a bad thing); â˘the realization that certain gaps in its conceptual map imply that its learning is being controlled. 8.Due diligence: Is this article an infohazard? âIf a capable AI reads this article in future, might it not improve its ability to circumvent our interpretability methods?â. Thatâs a possibility worth exploring. Here I address whether this article is an infohaz- ard and whether itâs a good idea to release it from a safety perspective. I opted to release it on the following grounds: I think that active circumvention methods, including those that rely on gradient hack- ing, are unlikely to be useful to a misaligned AI because any AI that is capable enough to carry them out probably has better means of escaping oversight. The other kinds of circumvention methods are passive. The emergence of passive circumvention methods isnât helped by having knowledge of them. In general, it seems to me like an AI that couldnât generate the ideas in this article themselves is probably unable to overcome our control measures. And those that can overcome our control measures are probably also smart enough to generate these ideas themselves. If Iâm right, then on balance discussing these risks and providing a framework for thinking about them is more likely to encourage interpretability researchers to focus on preventing (what I see as) the major source of circumvention-risk: passive interpretability circumvention methods. But what if Iâm wrong and active circumvention methods are, in fact, likely to be learned before alternative capabilities that would allow an AI to escape? On balance, I think the expected benefits still outweighs the expected harms: One of the goals of this article is to make readers appropriately paranoid about the many ways an AI might be able to think dangerous thoughts even if weâve got pretty good interpretability methods. Insofar as it succeeds at that, itâs more likely to help than harm. It might also make analignedAI more paranoid about the actions of its potentially misaligned past- selves, and thus better able to help us mitigate the risks. In addition to the above reasons, I consulted a small group of peers about releasing it. The unanimous consensus was that it wasnât an infohazard or that the potential benefits clearly outweighed the potential risks. 28 Glossary autostigmergyUsing the environment as an external memory system (see âEx- ternalizing representationsâ. 11 cognito-defenseWhen the interpreted agent hides its thoughts from an adversary by exploiting weaknesses in the interpretability methods.. 23, 24 cognito-hazardAn observation that is dangerous to the observer merely by being perceived.. 22 cognito-offenseThe act of seeking access to an adversaryâs thoughts.. 23 cognitosecurityCognitosecurity is the domain concerned with protecting thoughts from mind-reading adversaries.. 3, 23 interpretability toolsThe machinery that reads the AIâs representations and converts them into outputs that are readable to humans. Components of interpretability methods.. 3â6, 9, 16, 24, 25 29 References Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan ManĂŠ. Concrete Problems in AI Safety, July 2016. URLhttp://arxiv.org/abs/1606. 06565. arXiv:1606.06565 [cs]. Peter Barnett.Some motivations to gradient hack, December 2021a. URLhttps://w.alignmentforum.org/posts/5Fmc49cARW2a3uc5B/ some-motivations-to-gradient-hack. Peter Barnett. Understanding Gradient Hacking, December 2021b. URLhttps://w. alignmentforum.org/posts/bdayaswyewjxxrQmB/understanding-gradient-hacking. Tsvi Benson-Tilsen and Nate Soares. Formalizing convergent instrumental goals, 2016. URLhttps://w.aaai.org/ocs/index.php/WS/AAAIW16/paper/view/12634. Tolga Bolukbasi, Adam Pearce, Ann Yuan, Andy Coenen, Emily Reif, Fernanda ViĂŠgas, and Martin Wattenberg. An Interpretability Illusion for BERT, April 2021. URLhttp: //arxiv.org/abs/2104.07143. arXiv:2104.07143 [cs]. Paul Christiano. Worst-case guarantees (Revisited), March 2019. URLhttps:// ai-alignment.com/training-robust-corrigibility-ce0e0a3b9b4d. Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Ĺukasz Kaiser. Universal Transformers, March 2019. URLhttp://arxiv.org/abs/1807.03819. arXiv:1807.03819 [cs, stat]. Ben Deverett, Ryan Faulkner, Meire Fortunato, Greg Wayne, and Joel Z. Leibo. Interval timing in deep reinforcement learning agents. InProceedings of the 33rd international conference on neural information processing systems. Curran Associates Inc., Red Hook, NY, USA, 2019. Number of pages: 10 tex.articleno: 600. Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition, 2022. URLhttps://arxiv.org/abs/2209.10652. Soeren Elverlin. Recommendations for novels describing incentives of agents facing mindreaders, December 2021. URLwww.reddit.com/r/rational/comments/re40as/ recommendations_for_novels_describing_incentives/. Scott Garrabrant. Robustness to Scale, February 2018. URLhttps://w. alignmentforum.org/posts/bBdfbWfWxHN9Chjcq/robustness-to-scale. Robert Geirhos, JĂśrn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Bren- del, Matthias Bethge, and Felix A. Wichmann. Shortcut Learning in Deep Neural Networks.Nature Machine Intelligence, 2(11):665â673, November 2020. ISSN 2522- 5839. doi: 10.1038/s42256-020-00257-z. URLhttp://arxiv.org/abs/2004.07780. arXiv:2004.07780 [cs, q-bio]. 30 Shafi Goldwasser, Michael P. Kim, Vinod Vaikuntanathan, and Or Zamir. Plant- ing Undetectable Backdoors in Machine Learning Models, April 2022. URLhttp: //arxiv.org/abs/2204.06974. arXiv:2204.06974 [cs]. Chong Guo, Michael J. Lee, Guillaume Leclerc, Joel Dapello, Yug Rao, Aleksander Madry, and James J. DiCarlo. Adversarially trained neural representations may already be as robust as corresponding biological neural representations, June 2022. URLhttp: //arxiv.org/abs/2206.11228. arXiv:2206.11228 [cs, q-bio]. Jacob Hilton, Nick Cammarata, Shan Carter, Gabriel Goh, and Chris Olah. Understanding RL vision.Distill, 2020. doi: 10.23915/distill.00029. Evan Hubinger. Gradient hacking, October 2019a. URLhttps://w.alignmentforum. org/posts/uXH4r6MmKPedk8rMA/gradient-hacking. Evan Hubinger.Chris Olahâs views on AGI safety, November 2019b. URLhttps://w.alignmentforum.org/posts/X2i9dQQK3gETCyqh2/ chris-olah-s-views-on-agi-safety. Evan Hubinger.Relaxed adversarial training for inner alignment, Septem- ber 2019c. URLhttps://w.alignmentforum.org/posts/9Dy5YRaoCxH9zuJqa/ relaxed-adversarial-training-for-inner-alignment. Evan Hubinger. An overview of 11 proposals for building safe advanced AI.CoRR, abs/2012.07532, 2020. URLhttps://arxiv.org/abs/2012.07532. William James.The varieties of religious experience: A study in human nature. Routledge, 1902. doi: 10.4324/9780203398470. Keita Kurita, Paul Michel, and Graham Neubig. Weight Poisoning Attacks on Pre-trained Models, April 2020. URLhttp://arxiv.org/abs/2004.06660. arXiv:2004.06660 [cs, stat]. Brenden M. Lake, Tomer D. Ullman, Joshua B. Tenenbaum, and Samuel J. Ger- shman. Building machines that learn and think like people.Behavioral and Brain Sciences, 40:e253, 2017. ISSN 0140-525X, 1469-1825. doi: 10.1017/ S0140525X16001837. URLhttps://w.cambridge.org/core/product/identifier/ S0140525X16001837/type/journal_article. Mechanistic Interpretability. MLP Neurons - 40L Preliminary Investigation [rough early thoughts], December 2021. URLhttps://w.youtube.com/watch?v=8wYNsoycM1U. Evan R. Murphy. Interpretabilityâs Alignment-Solving Potential: Analysis of 7 Scenar- ios, May 2022. URLhttps://w.alignmentforum.org/posts/FrFZjkdRsmsbnQEm8/ interpretability-s-alignment-solving-potential-analysis-of-7. Neel Nanda.A Longlist of Theories of Impact for Interpretability, March 2022a.URLhttps://w.alignmentforum.org/posts/uK6sQCNMw8WKzJeCQ/ a-longlist-of-theories-of-impact-for-interpretability. Neel Nanda.A Longlist of Theories of Impact for Interpretability, March 2022b.URLhttps://w.alignmentforum.org/posts/uK6sQCNMw8WKzJeCQ/ a-longlist-of-theories-of-impact-for-interpretability. Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization.Distill, 2017. doi: 10.23915/distill.00007. 31 Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits.Distill, 2020. doi: 10.23915/distill.00024. 001. Arezoo Rajabi, Bhaskar Ramasubramanian, and Radha Poovendran. Trojan Horse Train- ing for Breaking Defenses against Backdoor Attacks in Deep Learning, March 2022. URLhttp://arxiv.org/abs/2203.15506. arXiv:2203.15506 [cs]. Adam Shimi. Approaches to gradient hacking, August 2021. URLhttps://w. alignmentforum.org/posts/S2jsBsZvqjBZa3pKT/approaches-to-gradient-hacking. Nate Soares, Eliezer Yudkowsky, and Jaan Tallinn. Soares, Tallinn, and Yudkowsky dis- cuss AGI cognition, November 2021. URLhttps://w.alignmentforum.org/posts/ oKYWbXioKaANATxKY/soares-tallinn-and-yudkowsky-discuss-agi-cognition. Chelsea Voss, Nick Cammarata, Gabriel Goh, Michael Petrov, Ludwig Schubert, Ben Egan, Swee Kiat Lim, and Chris Olah. Visualizing weights.Distill, 2021. doi: 10. 23915/distill.00024.007. John Wentworth. Godzilla Strategies, June 2022. URLhttps://w.alignmentforum. org/posts/DwqgLXn5qYC7GqExF/godzilla-strategies. Eliezer Yudkowsky. Harry Potter and the Methods of Rationality, Chapter 27: Empathy. URLhttps://w.hpmor.com/chapter/27. Eliezer Yudkowsky.Cached Thoughts, October 2017a.URLhttps://w. alignmentforum.org/posts/2MD3NMLBPCqPfnfre/cached-thoughts. Eliezer Yudkowsky.Security Mindset and Ordinary Paranoia, November 2017b.URLhttps://w.alignmentforum.org/posts/8gqrbnW758qjHFTrH/ security-mindset-and-ordinary-paranoia. 32