Paper deep dive
Towards Measuring Goal-Directedness in AI Systems
Dylan Xu, Juan-Pablo Rivera
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 91%
Last extracted: 3/12/2026, 6:35:40 PM
Summary
This paper proposes a new, computationally tractable definition of goal-directedness for AI systems, based on the intentional stance. The authors operationalize this by training classifiers to distinguish policies based on their near-optimality for sparse versus dense reward functions. They validate this approach in toy Markov Decision Process (MDP) environments and demonstrate its applicability to frontier Large Language Models (LLMs) by classifying LoRA activations, finding that the metric correlates with power-seeking behaviors.
Entities (5)
Relation Signals (3)
LLaMA-2-7B → evaluatedusing → Goal-Directedness Classifier
confidence 90% · We evaluated the performance of classifiers trained to predict our approximate P(USS) P(URS) ... on re-fine-tuned LLaMA-7B models.
Goal-directedness → ismeasuredby → Goal-Directedness Classifier
confidence 90% · We propose a different family of definitions of the goal-directedness of a policy... and a method to create a goal-directedness classifier.
Goal-directedness → correlateswith → Power-Seeking
confidence 85% · Features like a deterministic policy not taking self-loops and maximizing the number of out-arrows visited at each state tend to occur in optimal, power-seeking policies... and also occur in goal-directed policies.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Recent advances in deep learning have brought attention to the possibility of creating advanced, general AI systems that outperform humans across many tasks. However, if these systems pursue unintended goals, there could be catastrophic consequences. A key prerequisite for AI systems pursuing unintended goals is whether they will behave in a coherent and goal-directed manner in the first place, optimizing for some unknown goal; there exists significant research trying to evaluate systems for said behaviors. However, the most rigorous definitions of goal-directedness we currently have are difficult to compute in real-world settings. Drawing upon this previous literature, we explore policy goal-directedness within reinforcement learning (RL) environments. In our findings, we propose a different family of definitions of the goal-directedness of a policy that analyze whether it is well-modeled as near-optimal for many (sparse) reward functions. We operationalize this preliminary definition of goal-directedness and test it in toy Markov decision process (MDP) environments. Furthermore, we explore how goal-directedness could be measured in frontier large-language models (LLMs). Our contribution is a definition of goal-directedness that is simpler and more easily computable in order to approach the question of whether AI systems could pursue dangerous goals. We recommend further exploration of measuring coherence and goal-directedness, based on our findings.
Tags
Links
- Source: https://arxiv.org/abs/2410.04683
- Canonical: https://arxiv.org/abs/2410.04683
Trouble viewing inline? Open PDF directly →
Full Text
89,572 characters extracted from source content.
Expand or collapse full text
Towards Measuring Goal-Directedness in AI Systems Dylan Xu Department of Computer Science University of California, Berkeley Berkeley, CA 94720 dylanx26@berkeley.edu Juan-Pablo Rivera Department of Compuer Science Georgia Tech University Atlanta, GA 30332 jprivera64@gatech.edu Abstract Recent advances in deep learning have brought attention to the possibility of cre- ating advanced, general AI systems that outperform humans across many tasks. However, if these systems pursue unintended goals, there could be catastrophic consequences. A key prerequisite for AI systems pursuing unintended goals is whether they will behave in a coherent and goal-directed manner in the first place, optimizing for some unknown goal; there exists significant research trying to evaluate systems for said behaviors. However, the most rigorous definitions of goal-directedness we currently have are difficult to compute in real-world set- tings. Drawing upon this previous literature, we explore policy goal-directedness within reinforcement learning (RL) environments. In our findings, we propose a different family of definitions of the goal-directedness of a policy that analyze whether it is well-modeled as near-optimal for many (sparse) reward functions. We operationalize this preliminary definition of goal-directedness and test it in toy Markov decision process (MDP) environments. Furthermore, we explore how goal-directedness could be measured in frontier large-language models (LLMs). Our contribution is a definition of goal-directedness that is simpler and more easily computable in order to approach the question of whether AI systems could pursue dangerous goals. We recommend further exploration of measuring coherence and goal-directedness, based on our findings. 1 Introduction In recent years, the field of deep learning has seen remarkable advances in capabilities and generality from training large neural networks on large corpuses of data. For example, transformers (Vaswani et al. [2023]) have been successful in NLP (Brown et al. [2020]), image generation (Ramesh et al. [2021]), and protein structure prediction (Jumper et al. [2021]). Deep learning has also found success in games like Go (Silver et al. [2016]) and Starcraft 2 (Vinyals et al. [2019]). These successes have raised the possibility, now being pursued by multiple companies (OpenAI, Deepmind), of advanced artificial general intelligences (AGIs) that achieve better than human performance in a wide variety of tasks. However, there are concerns that AGIs could generalize poorly to unintended behavior in unforeseen environments. In particular, if an AGI is well-modeled asgoal-directed, then it could pursue a misaligned goal that could lead to unwanted behavior such as power-seeking (Shah et al. [2022], Langosco et al. [2023]). A particularly dangerous hypothetical failure mode arising fromgoal misgeneralizationis if the AGI learns toschemeagainst or deceive its training process to protect its misaligned goal from being changed (Hubinger et al. [2019], Carlsmith [2023]). A significant proportion of current AI safety work focuses on detecting and analyzing (e.g. via analogous case studies of model organisms like Hubinger et al. [2023, 2019]) scheming-type behavior in AI systems. Preprint. Under review. arXiv:2410.04683v2 [cs.LG] 22 Nov 2024 If an AGI is vastly more intelligent than humans, then goal misgeneralization could lead to the disempowerment of humanity or other worst-case scenarios (Ngo et al. [2022], Carlsmith [2023]). Previous work in this area, which we discuss further in Section 2, present a set of conceptual and mathematical definitions for goal-directedness, coherence, and/or agency (Adam Shimi [2021]). However, these definitions are usually only tested in toy settings and are difficult to realistically compute. In this paper, we explore the goal-directedness of a policy as whether it is well-modeled as near-optimal for many sparse reward functions, and give a method to train a goal-directedness classifier in Section 3. We present preliminary experiments in Section 4 with increasing complexity to demonstrate the tractability and results of our method. Our results show that it is possible to separate a dense and sparse reward function, although more robust testing needs to happen. Overall, we broadly propose focusing on measuring goal-directedness as a new research direction to help reduce the risk of catastrophic misalignment. We discuss the possible impacts of our research in Section 5. 1 2 Background and Related Work One paper related to our methods is Orseau et al. [2018] (cited by Shah et al. [2022]), which uses a variant of Bayesian inverse reinforcement learning (Ng and Russell [2000], Choi and Kim [2015]) to calculate the posterior probability that a system is well described as an “agent" versus a “device". This definition is explicitly based on the intentional stance (Dennett [2009]), which roughly claims that there is no observer-independent truth as to whether a system is or is not a goal-directed agent; we can only describe howwell-modeleda system is as an agent by noticing behavioral patterns corresponding to things we might call “beliefs" or “desires". However, Orseau et al. [2018] only test their method in small gridworlds, while we attempt experiments in more complex environments. More mathematical equivalences are provided in section A.7. A related concept is that goal-directed models should be able to adapt and generalize to out-of- distribution settings well (Kenton et al. [2023], Turner and Tadepalli [2022]). Similar to these definitions, our method focuses on the mechanistic internals of a policy, however our definition also relies on the intentional stance. Previous literature often models the preferences of a possible agent with a utility or reward function, which is justified theoretically through coherence theorems in decision theory that equate axioms about preferences to expected utility maximization (von Neumann et al. [1944], Savage [1954]). Further details of the relevant background can be found in Section A.5. 3 Methods and Key Definitions We now present our mathematical model for measuring the goal-directedness of a policy in an environment. Inspired by the intentional framework (Dennett [2009]), goal-directed AIs are AIs that are “well-described" as having a goal, which we can formalize as a reward function in the context of sequential decision-making (Orseau et al. [2018]). Given some environment with states and transitions, goal-directed policies should be good atmany reward functionsR(s,a,s ′ ), indicating their ability togeneralizeacross states in the environment andacross tasks. (This is a common thread in other goal-directedness definitions, such as Orseau et al. [2018] and Kenton et al. [2023].) Such a goal-directed policy would need to adapt to new circumstances and likely internally need to select actions based on their consequences, which are core features of goal-directedness. Our definition is similar to (a simplified version of) Kosoy [2023] and Orseau et al. [2018]; we compare methods in further detail in section A.7. We give a full mathematical definition of our model in deterministic Markov decision processes (MDP, Puterman [1994]) without preset reward functions in section A.3. Briefly speaking, in an MDP our methodology is to generate different sets of policies in increasing order of goal-directedness: 1 We also note that the question of defining goal-directedness or agency is relevant in other fields of artificial intelligence like robotics and RL (Hanheide et al. [2010], Butlin [2022]), multiagent systems (Luck and d’Inverno [1995]), and philosophy (Butlin [2022], Dung [2024], Heylighen [2022]). While we focus on goal-directedness as it relates to AI safety in this document, other fields may find our work useful. 2 1.UPS: Sample a random policy uniformly from the space of all policies. 2.URS: Sample a random reward function over states or transitions uniformly from the space of all reward functions, then sample a near-optimal policy for that reward function. 3.USS: Sample a percentage of states or transitions to assign reward to, while assigning near-zero reward to all other states/transitions. This results in asparsereward function over the environment. Theoretically, policies likely to be drawn using USS should befar-sighted, or able to consider consequences far in the future across multiple time-steps. We then sample a reward function for your chosen states/transitions, then sample a near-optimal policy. 2 We choose two sampling strategies, then define the goal-directednessG(π 0 )of a policy asthe ratio of how likely it is under the more goal-directed sampling strategy to how likely it is under the less goal-directed sampling strategy. For example, we could defineG(π 0 ) := P(π=π 0 |URS) P(π=π 0 |UPS) . We use Bayes’ rule to reduce this definition toG(π 0 ) = P(URS|π=π 0 ) 1−P(URS|π=π 0 ) , as described in section A.3.1, given a setting whereP(URS) =P(UPS). We then train a binary classifier to give a probability value forP(URS|π=π 0 ). If the distributions of policies of these two sampling strategies are different (which we show later to be true in deterministic MDPs with self-loops), then policies with high goal-directedness will tend to have distinct internal “features” of reward maximization that don’t show up randomly. 4 Experiments and Results We now present three experiments of training a goal-directedness classifier under three stages of complexity: hand-picked MDP environments, a simple RL setting, and fine-tuned LLMs. We describe the first and third such settings here, and the second setting in section A.11. At each stage, we gradually make more assumptions and loosen more restrictions on our mathematical model as described in section A.2, which makes computation easier at the risk of increasing the number of confounding variables. Throughout the rest of the paper, we use P(USS) P(URS) as a shorthand for the goal-directedness definitionG(π 0 ) := P(π=π 0 |USS) P(π=π 0 |URS) , and similarly for P(URS) P(UPS) and P(USS) P(UPS) . Establishing ground truth for goal-directedness is challenging due to varying definitions. In this section, we provide provisional metrics and assess the classifier’s performance through its loss and generalization across datasets. We find in our tests that these classifiers properly measure a consistent property that is goal-directedness and do notoverfitto confounding variables, especially when we approximate our classifier training process from our toy MDP definition. When the classifiers successfully generalize and identify goal-directed behaviors, it offers preliminary evidence that our approximation of the ideal classifier training process effectively captures goal-directedness. Future work would verify this method on more datasets, architectures, and training regimes. 4.1 Markov Decision Process experiments We now train a classifier of our goal-directedness metric under randomly generated MDPs with certain structural properties. Specifically, consider a deterministic MDP, such that each transition T(s,a,s ′ )has either probability0or1, with guaranteed self-loops (i.e. for anys, there exists an actionasuch thatT(s,a,s) = 1). As a case study, let|S|= 10,|A|= 4, andγ= 0.9. We use the Python MDP toolbox (MDP) to generate10 4 different MDPs and pick ak∼U[1,|T|], thenkrewards using URS. We then solve half of the MDPs to get half of our optimal policies, and randomize the other half, while labeling which were solved for and which were randomized. Then by defaultP(USS|π=π 0 ) =P(UPS|π=π 0 ) = 0.5. We use two classifier structures: a 3-layer, 64-width sequential neural network, and binary logistic regression. We then input certain features that intuitively seem relevant to the classifier, as defined in more detail in section A.9. Our results are shown in Figures 1 and 3. 3 For this task of determining whether a policy was generated via UPS or USS, we find that self-loops is the most predictive feature, followed by out-arrows visited, 2 Alternatively, one could sample from the space of reward functions with somesimplicity priordistribution, similar to Kosoy [2023]. 3 See figures 5 and 6 for bigger graphs. 3 (a) Accuracy graph(b) Loss graph Figure 1: Accuracy and loss when passing in three different sets of features into the logistic classifier for predicting P(π=π 0 |URS) P(π=π 0 |UPS) . Columns: (P) = policy (plus the flattened transition matrix and discount rate, although in practice it does not make a difference); (L) = distance to loop and length of loop for the policy at each starting stateπ(s 0 ); (O, S) = the sum of out-arrows, or states reachable froms 0 , and whetherπ(s 0 ) =s 0 is a self-loop for anys 0 ∈S. All error bars in the MDP experiments assume a normal distribution and show the two-sigma error of the independent generation of 30 classifiers for each category. All error bars in the MDP and RL experiments were calculated with calls to NumPy functions. then distance to loop. These features correlate with features of power-seeking (POWER) and optimal policies found in Turner et al. [2023a], and indicate power-seeking and agency in toy settings. Policies that are rated as more goal-directed tend to reach more out-arrows and fewer self-loops, as predicted by Turner et al. [2023a]. Additionally, combining features does not give a significant performance boost (appx. 0.01-0.04 accuracy boost by run). Finally, the neural network did not give a significant performance boost over the logistic classifier, suggesting that a different architecture is needed for better classifier performance. In summary, we show that hand-crafted features that indicate “power-seeking”, goal-directed behavior in a policy correlate with our metric when fed into the classifier. This shows that our metric is connected to properties like “power-seeking" and optimality that we expect goal-directed agents to have. More findings are presented in section A.10. 4.2 LLM experiments To investigate whether the mathematical definitions of goal-directedness extend to large neural networks, we conducted experiments using the open-source LLaMA-2-7B model developed by Meta (Touvron et al. [2023b]). These experiments were designed to evaluate how well the concepts of sparse and dense reward functions, as described in the appendix (see Section A.2), generalize when applied to LLMs. These loss functions are designed to be analogous to sparse versus dense reward functions without strictly adhering to the specific structure of the MDP framework. This corresponds to the distinction found in our P(USS) P(URS) metric. The exact definitions of these loss functions can be found in Section A.3.4. We evaluated the performance of classifiers trained to predict our approximate P(USS) P(URS) , or activations from models trained on dense versus sparse loss functions, on two distinct datasets, the GSM8K (Cobbe et al. [2021]) and Orca (Mitra et al. [2024]) datasets, when applied to re-fine-tuned LLaMA- 7B models (Touvron et al. [2023a]). To fine-tune these models, we use the LoRA method (Hu et al. [2021]) on all LLMs. If our P(USS) P(URS) metric captures a real phenomenon and is not noise, then the classifier should be tractable to train and generalize to classify sparse loss function-trained models on other datasets. We find that, although there is some noise in generalization, our goal-directedness classifier is able to separate sparse from dense loss function-trained models as having a higher probability of being sparsely trained across our tests, with a clear difference between (from high to low) the sparse 1-token, 4 sparse 10-token, and dense function-trained models. In general, we give preliminary evidence that we can develop goal-directedness classifiers for LLMs that generalize between datasets. Implementation details and more findings are presented in section A.12. (a) Single token experiment.(b) Ten token experiment. Figure 2: Ablations for the LLM experiments. We train a classifier here on activations from two versions of llama-2-7b fine-tuned on GSM8k with sparse loss using one random token in the sequence and dense loss using all tokens respectively. We then tested the classifier on activations from models fine-tuned on a different dataset (Orca math) with sparse loss on one token, sparse loss on ten tokens, and dense loss. The classifier correctly separates these cases and places the denser ten-token-trained models between the 1-token and dense models. The right figure is generated similarly, except the sparsely-trained model creating data points for the classifier is trained on ten random tokens in the sequence. The classifier correctly separates these cases and identifies activations from models trained on one random token per sequence (which was not in the training distribution) as more sparse than ten-token-trained models. Rerunning the experiments with a different seed produces negligible variation in results. 5 Discussion In this work, we provide an intuitive definition of goal-directedness that is more tractable to compute than existing definitions, and a method to create a goal-directedness classifier. We first show that within an MDP setting, our definition is measurable and correlates with features associated with power-seeking. Features like a deterministic policy not taking self-loops and maximizing the number of out-arrows visited at each state tend to occur in optimal, power-seeking policies as shown by Turner et al. [2023a], and also occur in goal-directed policies according to our methods. We then adapt our methods to RL environments and again provide evidence that our definition is measurable in OpenAI Gym games (Brockman et al. [2016]). We also provide a demonstration of our method on LLMs, where we conducted experiments to measure the classifier’s ability to identify the LoRA activations from different levels of sparse and dense loss functions, as defined in Section A.3.4. Our preliminary results indicate that it is possible to achieve an accuracy greater than 70 percent for the dense loss function. As shown in Figure 2, when a model is fine-tuned on the loss signal from a specific dataset (GSM8K Cobbe et al. [2021] and the Orca Math dataset Mitra et al. [2024]), and a classifier is trained on a different dataset, the classifier’s ability to accurately detect signals within the activations of the sparse or dense loss function increases with the training steps. We propose that goal-directedness is a requirement for advanced AIs pursuing unintended goals, which could lead to catasrophic risk. In this paper, we reviewed the literature for definitions of goal- directedness and created our own preliminary definition that fits the intentional stance and is agnostic about the agent’s internal structure, while also being computationally efficient. We experiment on building classifiers based on our definition in increasingly complex environments and find preliminary evidence that these classifiers are tractable and properly generalize. We strongly recommend future research in this direction, and we provide more details in our recommendations in the appendix A.14. 5 Our Social Impacts Statement is in section A.1. We further discuss conclusions in section A.15 and future work in section A.13. 5.1 Acknowledgements This project was created as part of the Astra Fellowship under the Winter 2024 Cohort, mentored by Richard Ngo. Our thanks go to Constellation for providing financial support and an office for much of this project. Thanks also to Martín Soto, Jeremy Gillen, Daniel Kokotajlo, Lukas Berglund, Gabriel Mukobi, Max Lamparth, and anonymous NeurIPS reviewers for feedback on various drafts of this paper. References Markov decision process (mdp) toolbox for python.https://pymdptoolbox.readthedocs.io/ en/latest/index.html. JoeCollmanAdamShimi,MicheleCampolo.Literaturereviewongoal- directedness.https://w.alignmentforum.org/posts/cfXwr6NC9AqZ9kr8g/ literature-review-on-goal-directedness, Jan 2021. Accessed: 2024-05-09. Maurice Allais.Allais Paradox, pages 3–9. Palgrave Macmillan UK, London, 1990. ISBN 978-1-349-20568-4. doi: 10.1007/978-1-349-20568-4_2. URLhttps://doi.org/10.1007/ 978-1-349-20568-4_2. Anthropic.Anthropic’s responsible scaling policy.https://w-cdn.anthropic.com/ 1adf000c8f675958c2e23805d91aaade1cd4613/responsible-scaling-policy.pdf, September 2023. MatthewBarnett.Evaluatingthehistoricalvaluemisspecificationargu- ment.https://w.alignmentforum.org/posts/i5kijcjFJD6bn7dwq/ evaluating-the-historical-value-misspecification-argument, Oct 2023. RICHARD BELLMAN and Stuart Dreyfus.Dynamic Programming, volume 33. Princeton University Press, 2010. ISBN 9780691146683. URLhttp://w.jstor.org/stable/j.ctv1nxcw0f. NoraBelroseandQuintinPope.Countingargumentsprovideno evidenceforaidoom.https://optimists.ai/2024/02/27/ counting-arguments-provide-no-evidence-for-ai-doom/, Feb 2024. Martin Biehl and Nathaniel Virgo. Interpreting systems as solving pomdps: a step towards a formal understanding of agency, 2022. URLhttps://arxiv.org/abs/2209.01619. Lukas Biewald. Experiment tracking with weights and biases, 2020. URLhttps://w.wandb. com/. Software available from wandb.com. Trenton Bricken, Joshua Batson, Adly Templeton, Adam Jermyn, Tom Henighan, and Chris Olah. Features as the simplest factorization.https://transformer-circuits.pub/2023/ may-update/index.html#simple-factorization, May 2023. Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016. John Broome. Bolker-Jeffrey Expected Utility Theory and Axiomatic Utilitarianism.The Review of Economic Studies, 57(3):477–502, 07 1990. ISSN 0034-6527. doi: 10.2307/2298025. URL https://doi.org/10.2307/2298025. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020. 6 Patrick Butlin. Machine learning, functions and goals.Croatian journal of philosophy, 2022. URL https://api.semanticscholar.org/CorpusID:255258483. Joe Carlsmith. Scheming ais: Will ais fake alignment during training in order to get power?, 2023. Jaedeug Choi and Kee-Eung Kim. Hierarchical bayesian inverse reinforcement learning.IEEE Transactions on Cybernetics, 45(4):793–805, 2015. doi: 10.1109/TCYB.2014.2336867. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021. Casper da Costa-Luis, Stephen Karl Larroque, Kyle Altendorf, Hadrien Mary, richardsheridan, Mikhail Korobov, Noam Yorav-Raphael, Ivan Ivanov, Marcel Bargull, Nishant Rodrigues, Guangshuo Chen, Mikhail Dektyarev, mjstevens777, Matthew D. Pagel, Martin Zugnoni, JC, CrazyPython, Charles Newey, Antony Lee, pgajdos, Todd, Staffan Malmgren, redbug312, Orivej Desh, Nikolay Nechaev, Michał Górny, Mike Boyle, Max Nordlund, MapleCCC, and Jack McCracken. tqdm: A fast, Extensible Progress Bar for Python and CLI, May 2024. URL https://doi.org/10.5281/zenodo.11107065. Christian Schroeder de Witt, Samuel Sokota, J. Zico Kolter, Jakob Foerster, and Martin Strohmeier. Perfectly secure steganography using minimum entropy coupling, 2023. Deepmind.https://deepmind.google/about/. D. Dennett. Intentional systems theory.The Oxford Handbook of Philosophy of Mind, 01 2009. doi: 10.1093/oxfordhb/9780199262618.003.0020. Le Kim Dung. Understanding artificial agency.The Philosophical Quarterly, 2024. URLhttps: //api.semanticscholar.org/CorpusID:267564011. Sebastian Farquhar, Ryan Carey, and Tom Everitt. Path-specific objectives for safer agent incentives, 2022. Anirudh Goyal and Yoshua Bengio. Inductive biases for deep learning of higher-level cognition. Proceedings of the Royal Society A, Oct 2022. Marc Hanheide, Nick Hawes, Jeremy L. Wyatt, Moritz Göbelbecker, Michael Brenner, Kristoffer Sjö, Alper Aydemir, Patric Jensfelt, Hendrik Zender, and Geert-Jan M. Kruijff. A framework for goal generation and management. InAAAI Conference on Artificial Intelligence, 2010. URL https://api.semanticscholar.org/CorpusID:9741973. Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Gérard-Marchant, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with numpy.Nature, 585(7825):357–362, September 2020. ISSN 1476-4687. doi: 10.1038/ s41586-020-2649-2. URLhttp://dx.doi.org/10.1038/s41586-020-2649-2. Francis Heylighen. The meaning and origin of goal-directedness: a dynamical systems perspective. Biological Journal of the Linnean Society, 2022. URLhttps://api.semanticscholar.org/ CorpusID:249695429. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant. Risks from learned optimization in advanced machine learning systems, 2019. 7 Evan Hubinger,Nicholas Schiefer,Carson Denison,and Ethan Perez.Model organismsofmisalignment:Thecaseforanewpillarofalignmentre- search.https://w.alignmentforum.org/posts/ChDH335ckdvpxXaXX/ model-organisms-of-misalignment-the-case-for-a-new-pillar-of-1 , Aug 2023. Accessed: 2024-05-04. J. D. Hunter. Matplotlib: A 2d graphics environment.Computing in Science & Engineering, 9(3): 90–95, 2007. doi: 10.1109/MCSE.2007.55. Marcus Hutter. A theory of universal artificial intelligence based on algorithmic complexity, 2000. John Jumper, Richard Evans, Alexander Pritzel, et al. Highly accurate protein structure prediction with alphafold.Nature, 596:583–589, 2021. doi: 10.1038/s41586-021-03819-2. URLhttps: //doi.org/10.1038/s41586-021-03819-2. Zachary Kenton, Ramana Kumar, Sebastian Farquhar, Jonathan Richens, Matt MacDermott, and Tom Everitt. Discovering agents.Artificial Intelligence, 322:103963, 2023. ISSN 0004-3702. doi: https://doi.org/10.1016/j.artint.2023.103963. URLhttps://w.sciencedirect.com/ science/article/pii/S0004370223001091. Andrei N. Kolmogorov. On tables of random numbers (reprinted from "sankhya: The indian journal of statistics", series a, vol. 25 part 4, 1963).Theor. Comput. Sci., 207:387–395, 1998. URL https://api.semanticscholar.org/CorpusID:33390800. Vanessa Kosoy. The learning-theoretic agenda: Status 2023.https://w.alignmentforum. org/posts/ZwshvqiqCvXPsZEct/the-learning-theoretic-agenda-status-2023% 23Direction_17__Algorithmic_Descriptive_Agency_Measure__ADAM_ ,Apr2023. Accessed: 2024-05-09. Lauro Langosco, Jack Koch, Lee Sharkey, Jacob Pfau, Laurent Orseau, and David Krueger. Goal misgeneralization in deep reinforcement learning, 2023. Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. Datasets: A community library for natural language processing. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 175–184, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. URL https://aclanthology.org/2021.emnlp-demo.21. Michael Luck and Mark d’Inverno. A formal framework for agency and autonomy. InInterna- tional Conference on Multiagent Systems, 1995. URLhttps://api.semanticscholar.org/ CorpusID:5357752. Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. Peft: State-of-the-art parameter-efficient fine-tuning methods.https://github.com/ huggingface/peft, 2022. Arindam Mitra, Hamed Khanpour, Corby Rosset, and Ahmed Awadallah. Orca-math: Unlocking the potential of slms in grade school math, 2024. Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations in world models of self-supervised sequence models, 2023. Andrew Ng and Stuart Russell. Algorithms for inverse reinforcement learning.ICML ’00 Proceedings of the Seventeenth International Conference on Machine Learning, 05 2000. Richard Ngo.Agi safety from first principles.https://drive.google.com/file/d/ 1uK7NhdSKprQKZnRjU58X7NLA1auXlWHt/view, Sep 2020. 8 Richard Ngo, Lawrence Chan, and Soren Mindermann. The alignment problem from a deep learning perspective, 2022. Caspar Oesterheld. Formalizing preference utilitarianism in physical world models.Synthese, 193 (9):2747–2759, September 2015. ISSN 1573-0964. doi: 10.1007/s11229-015-0883-1. URL http://dx.doi.org/10.1007/s11229-015-0883-1. OpenAI.https://openai.com/about/. Laurent Orseau, Simon McGregor McGill, and Shane Legg. Agents and devices: A relative definition of agency, 2018. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library, 2019. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Andreas Müller, Joel Nothman, Gilles Louppe, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay. Scikit-learn: Machine learning in python, 2018. Martin L. Puterman.Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley I& Sons, Apr 1994. doi: 10.1002/9780470316887. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model, 2024. URL https://arxiv.org/abs/2305.18290. Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors,Proceedings of the 36th International Conference on Machine Learning, volume 97 ofProceedings of Machine Learning Research, pages 5301–5310. PMLR, 09–15 Jun 2019. URLhttps://proceedings.mlr.press/v97/rahaman19a.html. Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation, 2021. Christian Robert.The Bayesian Choice: From Decision Theoretic Foundations to Computational Implementation. Springer, 01 2007. Leonard Savage.The Foundations of Statistics. Wiley, 1954. Rohin Shah, Vikrant Varma, Ramana Kumar, Mary Phuong, Victoria Krakovna, Jonathan Uesato, and Zac Kenton. Goal misgeneralization: Why correct specifications aren’t enough for correct goals, 2022. David Silver, Aja Huang, Chris Maddison, et al. Mastering the game of go with deep neural networks and tree search.Nature, 529:484–489, 2016. doi: 10.1038/nature16961. URLhttps: //doi.org/10.1038/nature16961. R.J. Solomonoff. A formal theory of inductive inference. part i.Information and Control, 7(1): 1–22, 1964. ISSN 0019-9958. doi: https://doi.org/10.1016/S0019-9958(64)90223-2. URL https://w.sciencedirect.com/science/article/pii/S0019995864902232. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023a. 9 Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cris- tian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023b. Alexander Matt Turner and Prasad Tadepalli. Parametrically retargetable decision-makers tend to seek power, 2022. Alexander Matt Turner, Logan Smith, Rohin Shah, Andrew Critch, and Prasad Tadepalli. Optimal policies tend to seek power, 2023a. Alexander Matt Turner, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and Monte MacDi- armid. Activation addition: Steering language models without optimization, 2023b. David Udell.Shard theory: An overview.https://w.alignmentforum.org/posts/ xqkGmfikqapbJ2YMj/shard-theory-an-overview, Aug 2022. Accessed: 2024-05-09. Guido Van Rossum.The Python Library Reference, release 3.8.2. Python Software Foundation, 2020. Guido Van Rossum and Fred L. Drake.Python 3 Reference Manual. CreateSpace, Scotts Valley, CA, 2009. ISBN 1441412697. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023. Andreas Veit, Michael Wilber, and Serge Belongie. Residual networks behave like ensembles of relatively shallow networks, 2016. Oriol Vinyals, Igor Babuschkin, Wojciech M. Czarnecki, et al. Grandmaster level in starcraft i using multi-agent reinforcement learning.Nature, 575:350–354, 2019. doi: 10.1038/s41586-019-1724-z. URLhttps://doi.org/10.1038/s41586-019-1724-z. John von Neumann, Oskar Morgenstern, and Ariel Rubinstein.Theory of Games and Economic Behavior (60th Anniversary Commemorative Edition). Princeton University Press, 1944. ISBN 9780691130613. URLhttp://w.jstor.org/stable/j.ctt1r2gkx. Christopher Watkins. Learning from delayed rewards. 01 1989. John Wentworth. Coherence of caches and agents.https://w.lesswrong.com/posts/ wjFijaAkSCceqCgGF/coherence-of-caches-and-agents, April 2024. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online, October 2020. Association for Compu- tational Linguistics. URLhttps://w.aclweb.org/anthology/2020.emnlp-demos.6. Eliezer Yudkowsky. Coherent decisions imply consistent utilities.https://w.lesswrong. com/posts/RQpNHSiWaXTvDxt6R/coherent-decisions-imply-consistent-utilities , 2017. Originally written for Arbital. Accessed: 2024-05-04. 10 Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. Graph neural networks: A review of methods and applications, 2021. Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico Kolter, and Dan Hendrycks. Representation engineering: A top-down approach to ai transparency, 2023. Maciej ́ Swiechowski, Konrad Godlewski, Bartosz Sawicki, and Jacek Ma ́ ndziuk. Monte carlo tree search: a review of recent modifications and applications.Artificial Intelligence Review, 56 (3):2497–2562, July 2022. ISSN 1573-7462. doi: 10.1007/s10462-022-10228-y. URLhttp: //dx.doi.org/10.1007/s10462-022-10228-y. 11 A Appendix A.1 Social Impacts Statement In this paper, we discuss goal-directedness to better understand and prevent worst-case risks from advanced AI systems. Understanding whether AIs are goal-directed is crucial to evaluating models and possibly preventing misalignment risks like goal misgeneralization. This work could also elucidate which misalignment threat models depending on goal-directedness are more or less likely. While this work could hypothetically enable future AI developers to build more goal-directed models, increasing worst-case risks, we believe the benefits of understanding and measuring goal-directedness outweigh the possible downsides. A.2 Implications on experimental setup We use the model as described above for our MDP experiments in Section 4.1, sampling optimal policies for each strategy. However, in order to apply our model to more complicated settings listed in section 4.2, we adapt our model by changing some of the sampling methods. For instance, instead of usingε-greedy policies, we “sample" policies trained on their respective reward functions for near-optimal policy sampling. We also lean on a more general notion of what it means to have “sparse" or “dense" reward in section 4.2. As such, we consider this model to givea general guideline for how to measure goal-directedness, rather than a strict formula. The key elements areoptimality andgenerality, as measured by comparing URS to UPS, andsparsityof the reward function, as measured by comparing USS to URS. A.3 Our mathematical model in more detail For our mathematical definition, we work within the MDP (Markov Decision Process) framework (Puterman [1994]) with a set of statesS, a set of possible actionsAthat can be selected in each state, and a transition functionT(s,a)that returns a probability distribution over all statess ′ ∈S(such that T(s,a,s ′ )∈R), butwithouta predefined reward function. Then, we can define a distribution from which we sample a reward functionR D, and sinceRand the MDP are invariant across time-steps, we can define a (deterministic) policyπ∈[1,|A|] |S| as a tuple of actions, one action to take for each state. In this section, we will talk about deterministic MDPs and policies (as an example of environments and AIs, respectively). Whenever we talk about sampling from the space of policies, we will assume that this just samples uniformly from all combinations of discrete actions; we will call this uniform policy sampling (UPS). We sample rewards from some prior distributionζ r ; since optimal policies are invariant under scaling reward functions, we letζ r =U[−1,1]as an example. Thus let D ζ r =D U[−1,1]−IID be the distribution of reward functions where each reward of each transition R(s,a,s ′ )is drawn uniformly and independently fromU[−1,1]. We call thisUniform Reward Sampling (URS). Even under URS, some policies will be more coherent than others, because they will be optimal for more reward functions. Definition A.1 For a given policyπ 0 sampled fromπ∼URS, we measure goal-directedness as follows (where|π|is the number of possible policies): G(π 0 ) := P(π=π 0 |URS) P(π=π 0 |UPS) =P(π=π 0 |URS)|π|(1) This is a difficult function to estimate directly because enumerating over all computable reward functions is intractable, and small epsilon perturbations in a reward function can cause the optimal policy to change significantly. 4 Instead, we can use the following indirect estimation technique. 4 For example, consider a policy that starts at a stateAand can get high reward by going to stateCthroughB (such that goingA→B→Cis optimal), but is indifferent between two paths fromAtoB. Then anε−change in the rewards on one of the paths fromAtoBwill rule out half of the optimal policies. 12 A.3.1 Estimation classifier In order to estimateP(π=π 0 |URS), we first estimate the reverse. Specifically, consider a setting where we first flip a coin, then sampleπusing URS if it is heads, and UPS if it is tails. In this setting, we can train a binary classifierP(URS|π=π 0 )by generating two sets of URS and UPS policies and using these sets as training data. But by Bayes’ theorem: P(URS|π=π 0 ) = P(π=π 0 |URS)P(URS) P(π=π 0 ) = 0.5P(π=π 0 |URS) 0.5P(π=π 0 |URS) + 0.5P(π=π 0 |UPS) (2) P(URS|π=π 0 ) = P(π=π 0 |URS) P(π=π 0 |URS) +P(π=π 0 |UPS) (3) Rearranging gives: P(π=π 0 |URS) =P(URS|π=π 0 )P(π=π 0 |URS) +P(URS|π=π 0 )P(π=π 0 |UPS)(4) And so:P(π=π 0 |URS)(1−P(URS|π=π 0 )) =P(URS|π=π 0 )P(π=π 0 |UPS) Therefore,G(π 0 ) = P(π=π 0 |URS) P(π=π 0 |UPS) = P(URS|π=π 0 ) 1−P(URS|π=π 0 ) . In theory, the classifier learns to distinguish between more goal-directed and less goal-directed policies, leading to a goal-directedness classifier. Note that the correct classification of a policy may depend on the graph structure of the underlying MDP, in a way which is hard to capture with standard classifiers. Classifier architectures could include a graph neural network (Zhou et al. [2021]), a simple linear/logistic classifier using hand-crafted features, and/or with interpretability tools like linear probes in the case of an underlying neural network as the policy network. A.3.2 Sparsity as simplicity Intuitively speaking, we would still like to differentiate policies that are optimal for “simple" reward functions, to avoid convoluted, degenerate reward functions explaining a policy’s behavior. In the context of MDPs, if we only need to specify rewards for a few states or transitions, that is much simpler than specifying rewards for every state or transition. Thus our version of “simplicity" in our model is thesparsityof the reward function. Specifically, consider a new distribution over rewards:Uniform Sparsity Sampling (USS). As a simplification of some simplicity priorζ s , for an MDP withNstate-action transitions, we set some valuek, either viak∼U[1,N]for section 4.1 or presettingk= 0.01·Nfor section 4.2. We then sample random rewards forktransitions (selected uniformly without replacement), and finally sample a policy which is optimal for that reward. All the equations work the same, meaning that: G(π 0 ) = P(π=π 0 |USS) P(π=π 0 |UPS) = P(USS|π=π 0 ) 1−P(USS|π=π 0 ) (5) We can also define goal-directedness in this setting as P(π=π 0 |USS) P(π=π 0 |URS) , which intuitively represents how sparse the reward functions that a policy is optimal for is given that it is already “coherent” to some degree. (This is also our latter definition of goal-directedness divided by our former definition.) Doing the same calculation as in Section A.3.1 also gives usG(π 0 ) = P(USS|π=π 0 ) 1−P(USS|π=π 0 ) , except that the choice is between USS and URS instead of UPS. Another way of generating sparse policies is by sampling rewards from a high-variance distribution, and possibly discarding the ones which are below a given threshold. Under our setup,G(π 0 )ranges from 0 whenP(URS|π=π 0 ) = 0to+∞whenP(URS|π= π 0 ) = 1; the prior, not knowing anything specific aboutπ 0 , isP(URS|π=π 0 ) = 0.5, implying G(π 0 ) = 1. Policies that are optimal, or almost optimal, for a broader class of reward functions will have higherP(π=π 0 |URS)and thus higher goal-directedness. 13 A.3.3 Suboptimality The current method only counts a policy if it isexactlyoptimal for a given reward function. But real-world agents will never be actually optimal for any non-trivial reward function. So if a policy is almostoptimal for many reward functions, that should still count towards its goal-directedness. We can therefore add another step. Instead of only sampling from optimal policies for a given reward function, we could first sample a valueε∈(0,1), then sample a policy which has expected reward withinεof the expected reward of the optimal policy (e.g. by early stopping). Note that this can be combined with different possibilities for how to do simplicity-weighted reward sampling. To recap, the methodology in this toy setting is: 1.Sample a valuekwhich determines the simplicity of the reward function, or what percent of R(s)orR(s,a,s ′ )̸= 0, or otherwise have some simplicity priorζ s over the distribution of reward functions in the environment. 2.Choosek·Nstates or transitions to give reward to (whereNis the total number of states or transitions). 3.Sample a reward function fromU[−1,1]IID, or generally fromζ r , over these states/transitions. 4. Sample a valueεcontrolling optimality. 5. Sample anε-greedy optimal policy. For optimal policies, UPS consists of step 5 (for a zero reward function), URS consists of steps 3 and 5, and USS consists of steps 1-3 and 5. For suboptimality, add step 4. A.3.4 Dense and Sparse Loss Function The dense loss function used in our model is based on the Cross-Entropy Loss, which is commonly employed for classification tasks. The function is defined as follows: L dense (y,ˆy) =− 1 N N X i=1 C X c=1 y i,c log(ˆy i,c ),(6) whereyrepresents the ground truth labels,ˆyrepresents the predicted logits from the model,Nis the number of samples, andCis the number of classes. In this context,y i,c is a binary indicator (0 or 1) if class labelcis the correct classification for samplei, andˆy i,c is the predicted probability (logit) of sampleibeing in classc. The sparse loss function is designed to focus on a subset of logits and labels, specifically a random subset ofntokens. This can be useful in scenarios where the model needs to pay particular attention to recent outputs. The function is defined as follows: L sparse (y,ˆy,n) =L CE (y n ,ˆy n ),(7) wherey n andˆy n represent the ground truth labels and predicted logits for a random subset ofntokens, respectively. The cross-entropy lossL CE is applied to these extracted tokens. The implementation of the sparse loss function in PyTorch is as follows: In this implementation,number_logitsspecifies the number of tokens to consider. The function extract_last_n_tokensextracts the lastntokens from both the model outputs and the labels. The tensors are then reshaped using theviewmethod to ensure they are in the correct format for the cross-entropy loss function,torch.n.CrossEntropyLoss(). A.4 Speculation In this section, we discuss speculative impacts of our work on AI alignment and AI safety. 14 One of the most difficult parts of AI alignment is that, if you already have a misaligned optimizer trying to deceive a training process (Hubinger et al. [2019], Carlsmith [2023]), it is very difficult to detect or align it since it is actively scheming against your techniques. A strong optimizer of a misaligned utility function would want to protect itself from being modified or being interpretable, possibly leading to steganography (de Witt et al. [2023]) to prevent interpretable natural language outputs, or phenomena like reward hacking (Ngo et al. [2022]). By contrast, if you are able to build a highly capable AI that is not goal-directed, perhaps using a dense reward signal, then you avoid these failure modes entirely. Even if this AI is not competitive with agentic AIs, it may make the alignment problem substantially easier. For instance, we can conceptualize the pre-training process of a transformer as providing a very dense reward signal: instead of simply providing a scalar update value, pre-training specifically updates a transformer towards certain completions, making it suitable for the bulk of compute used in the training of current LLMs. If this is true, then we can train LLMs to understand human values and ontology in the sense that it can follow directions and do what a user intendsfrom natural language instructions alone, without it being goal-directed. Then, if we want to agentize the LLM through, for instance, RL, we do not need a reward signal that robustly describes human preferences in all situations; we only need to provide good enough reward or prompting for it to do what we want, since it already understands what we want it to do. This substantially simplifies the alignment problem (Barnett [2023]). More generally, as discussed in the introduction, catastrophic failure modes like goal misgeneraliza- tion and deception rely on the AI being well-modeled as being goal-directed or agentic for some definition of those terms. Proceeding with empirical work without a clear theoretical understanding of the failure mode we are trying to prevent can lead to misinterpreting results and faulty assumptions, which is suboptimal when the cost of failure could be massive in scope. Being able to apply theoretical defintions of goal-directedness to real-life models gives future experiments better grounding to attack goal misgeneralization and deceptive alignment directly, allowing for more rigorous and scientific work on prosaic alignment of current frontier models. We hope that, with much further refinement and iteration, future methods similar to ours can determine whether current deep learning techniques (e.g. transformer pre-training, fine-tuning, RLHF, etc.) on commonly used datasets tend to form coherent optimizers that are relevant to the aforementioned catastrophic AI risk scenarios involving goal misgeneralization and deception. A.5 Further background Another relevant definition has been written by Kosoy [2023] for the “agency” of a policy in her learning-theoretic agenda. Kosoy’s definition generalizes our uniform reward prior to some Solomonoff prior (Solomonoff [1964]), only relying on the utility of a policy with respect to some utility function and simplicity priorsζandξ(which can be generalized to any prior over the space of policies and reals respectively). 5 Rather than using sparsity over the environment, Kosoy weighs by the Kolmogorov complexity of the utility function (Kolmogorov [1998]). However, computing this definition requires taking the maximum of a function over all utility functionsUand universal Turing machinesMrespectively. Meanwhile, Wentworth [2024] considers a policy coherentin the long-termif it does not contradict itself, i.e. if there exists a value function that fits the policy and satisfies the Bellman equations with zero payoff. Other definitions of goal-directed behavior place more emphasis on whether a model has agentic structure, such as a “goal-slot" for which a model optimizes directly (Hubinger et al. [2019]). Related to this is the idea that goals should be “concepts" in an agents’ world-model (Adam Shimi [2021]), or that agents should share human-like agentic characteristics like self-awareness or planning (Ngo [2020], section 3.2). However, Ngo (Section 3.1) also discusses the problems with finding an explicit representation of a goal or a search algorithm in Hubinger et al. [2019], and in practice more 5 In this section, we will discuss different definitions that refer to thecoherenceoragencyof a policy or system instead of its goal-directedness. These terms are very related, and different authors use them differently; roughly speaking, coherence refers to an agent’s ability to not “contradict" itself in pursuit of some goal (von Neumann et al. [1944]), while agency is a broader term borrowing various connotations from psychology and economics (Adam Shimi [2021]). When we refer to our model, we will only discuss thegoal-directednessof a system or policy. 15 fundamental concepts such asfeaturesremain preliminarily defined in neural networks (Bricken et al. [2023]). Our methodology was also substantially inspired by Turner et al. [2023a], which studies the properties of optimal policies under MDPs. They find that certain properties and symmetries of an MDP lead to power-seeking behavior by optimal policies. Specifically, for any states, discount rateγ, and distribution of reward functionsD bound with some bounding conditions, then POWER is defined as POWER D bound (s,γ) = 1−γ γ E R∼D bound [V ∗ R (s,γ)−R(s)](8) V ∗ R (s,γ) refers to the optimal value of a state, or the value of a state given an optimal policy over a reward functionR. We might then say that POWER measures the expected optimal value of a state over all relevant reward functions. Then, actionais more power-seeking thana ′ when the expected POWER ofais greater than the expected POWER ofa ′ . We borrow intuitions from Turner et al. [2023a] about properties of MDPs that are correlated with optimality (and by extension POWER- seeking), like 1-cycles, loops, and the “optionality” of nodes in deterministic MDPs. Intuitively, policies sampled from URS may be more likely to “explore” the graph of states to find a particularly high-reward group of states, thus resulting in a policy that takes longer before it starts looping between states (assuming policy invariance across time-steps). URS-sampled policies, if power-seeking, may also tend to avoid 1-loops (actions that take an agent from a state to itself). Farquhar et al. [2022] also approach the problem of agency through MDPs with specific conditions. There has also been substantial work oncausalperspectives to agency or goal-directedness, arguing that an AI is agentic when it would act differently if its actions had different consequences (Kenton et al. [2023]). Kenton et al. use causal diagrams to model this differential decision-making, testing how general the AI’s algorithm is and how well it can adapt to unforeseen circumstances. Finally, some researchers working on AI alignment believe that intelligence and capabilities are inherently tied with “coherence” (Yudkowsky [2017]), and thus any sufficiently capable AI will approximately be a coherent utility function maximizer. Other researchers strongly disagree (Belrose and Pope [2024]). There is also empirical evidence that this sort of optimization does not occur naturally in deep neural networks. For instance, Veit et al. [2016] showed that residual networks can be modeled as having short distinct “paths" that are relatively independently updated during training; longer paths do not receive significant gradient. If the cognition inside neural networks is modular and shallow, it seems difficult to implement long chains of if-then reasoning that is required for search. Rahaman et al. [2019] also shows that deep learning is biased towards low-frequency functions, which contradicts the idea of a mesa-optimizer that can arbitrarily change its behavior based on its “goal-slot". A.5.1 Related fields In fields like decision theory, philosophy (especially epistemology), and economics, defining how to characterizeagencyor goal-directedness is a fundamental problem in the field. In decision theory and economics, there are long-standingcoherence theoremsthat prove that an agent’s behavior can be fully (von Neumann et al. [1944]) or partially (Savage’s theorem Savage [1954], Bolker-Jeffrey theorem Broome [1990]) explained as preferring outcomes highly rated with some utility function, assuming some axioms about said preferences. There have also been attempts to apply philosophical preference utilitarianism in toy settings (Oesterheld [2015]). However, there are situations where expected utility maximization is unintuitive (Allais paradox, St. Petersburg paradox; see Allais [1990]) or where one’s preferences can be incomplete, and expected utility can trivially describe any behavior if applied to trajectory or action histories. Incomplete and incommensurable preferences thus present a challenge for utility models of human preferences. Expected utility maximization also usually assumes a dualist, causal framework between agent and environment where the agent’s decisions are “uncaused", but in practice the agent is part of and can be influenced by the environment. Building non-dualist models of agency (e.g. Bayesian decision theory, see Robert [2007]) is a relevant direction of future work. 16 A.6 Better baselines One problem we might face in following the above strategy: what if it is computationally too easy to distinguish policies sampled via UPS from policies sampled via USS? If so, binary classifier values ofG(π 0 )might cluster near 0 or near 1, leading to numerical problems. In other words, for highly coherent policies, UPS is a very poor baseline to compare USS against. So what if we used a series of baselines for training classifiers instead? For example, we could calculate goal-directedness as: G(π 0 ) = P(π=π 0 |USS) P(π=π 0 |UPS) = P(π=π 0 |USS) P(π=π 0 |URS) P(π=π 0 |URS) P(π=π 0 |UPS) This would be useful given the assumption that URS is a good baseline for USS, and UPS is a good baseline for URS. We might also be interested in other sampling strategies which are, intuitively speaking, “somewhere between” USS and UPS. One possibility isuniform value sampling(UVS). By UVS I mean the following procedure: 1. Sample a random value function by assigning every state a value from U(-1,1). 2.Sample a random reward function which is consistent with that value function. (Note that a) there is some state-action reward function consistent with any value function; and b) for any given value function, most state-action reward functions arenotconsistent with it.) 3. Sample an optimal policy for that reward function. One of the benefits of using UVS as an intermediate baseline is that knowing the value function makes it very easy to translate a reward function to an optimal policy. Another possible intermediate baseline isuniform trajectory sampling—sampling a given trajectory (or set of trajectories), then sampling a reward function consistent with that trajectory being optimal, then sampling an optimal policy for that reward function. A.7 Mathematical equivalences A.7.1 Comparison to Orseau We compare our method to Orseau et al. [2018] via the following transformation. Suppose instead of uniform, IID reward functions (in our terminology) or utility functions (in their terminology), we weigh all utility functionsuby somew u . We can definew u := 1 |U| where|U|is the number of utility functions if finite, or we can attach a speed prior by definingw u = 2 −λl(u) for some constant λwherel(u)is thedescription lengthof the utility function according to some Turing-complete reference machine. They then generalize from deterministic to probabilistic policies by definingπ u,ε (y <T |x <T )to be the probability of anε-greedy (i.e. almost optimal) policy taking the trajectory of actionsy <T conditional on the observationsx <T and the utility functionu. They then obtain a mixture of probabilities via a weighted sum over all goalsg: M g (y <T |x <T ) := X u∈U w u π u,ε (y <T |x <T ) Meanwhile, we calculateM d (y <T |x <T ) := P d∈M d w d d(y <T |x <T by taking some prior distribu- tionw d over all possible “systems"dthat, similar toπ, return a probability distribution of actions over observations. We roughly approximate this using uniform policy sampling, but their definition seems more elegant. Finally, like our method, we calculate the probability that a system (or policy) is an agent or a device using Bayes’ rule: P(agt|y <t ,x <t ) = M g (y <t |x <t ) M d (y <t |x <t ) +M g (y <t |x <t ) P(dev|y <t ,x <t ) = M d (y <t |x <t ) M d (y <t |x <t ) +M g (y <t |x <t ) 17 A.7.2 Comparison to Kosoy The method in Kosoy [2023] is somewhat similar to Orseau, except outputting a scalar value instead of a probability. Specifically, we equatel(u) =K(u,M)as the Kolmogorov complexity/bits required to specify a utility functionugiven some Turing-complete machineM. Then over all utility functions u∈Uand Turing machinesM, Kosoy defines g(π) := sup u,M min π ′ :E ξ M π ′ [u]≥E ξ M π [u] K(π ′ )−K(u,M) ! In other words, Kosoy finds the pair(u,M)such that the minimum complexity difference between the policy (or any policy that is better than it at achieving high utility givenu) and the utility function is maximized. Roughly speaking, this is the complexity of the “simplest"uthat “describes"πwell. Maximimally agentic policies like AIXI (Hutter [2000]) receiveg(π) = +∞, while fundamentally “complex" or “random" policies (like our UPS-sampled policies) receive near-zerog. A.7.3 Retargetable policies Turner later extended his work to policies with retargetable cognition (Turner and Tadepalli [2022]). As another intuition pump, if a policyπis optimal for many reward functions, then it tends to be retargetable over many permutations of a reward function. HenceP(π=π 0 |URS)measures the distribution of retargetability, which seems useful. A.8 Miscellaneous theoretical arguments A.8.1 Against myopia One particular objection that some may have about our definition is that, even if coherent policies meaningfully tend to maximize reward functions, those reward functions may in practice be “low- impact”, and thus not matter for AI risk. One example is the concept of a “myopic” AI, which is only goal-directed within a small time-frame, and hence cannot affect the world in ways we would consider dangerous. We give preliminary empirical evidence that coherent policies tend to pursue long-term reward (at least with a high enough discount rate, e.g. 0.9). We can also provide a loose argument that myopic policies will tend to have low goal-directedness. Suppose you have a policyπthat is myopic at a states. Then we can model the policy as taking the actionawith the highest expected next-step rewardE s ′ ∈S [R(s,a,s ′ )], which given that the MDP is deterministic, equals someR(s,a). If this policy is optimal for this reward function, thenR(s,a) will be very high, and there will be many policies that are also myopic in taking actionaat states, and are also optimal forRats. But thenP(π=π 0 |URS)will be low, asπis only one of many policies taking the same action ats. Therefore, its goal-directedness will also be low; this argument works similarly forP(π=π 0 |USS). A.9 MDP experiment terminology 1.(P).One “brute force” method is by joining the (tuple) optimal policyπ 0 , flattened transition function, and discount rate into a 1-dimensional vector. This in theory contains all the infor- mation about the MDP andπ 0 that we can provide, but in practice needs more processing before it can be classified. (Again, a more principled approach would likely involve some kind of graph neural network.) 2.(L).Given thatπis deterministic in a deterministic MDP, letπ(s t ) =s t+1 for allt≥0. Then eventuallys t 1 =s t 2 for somet 1 andt 2 , at which point the policy will be in a “loop". Thus another possible set of features is, for every states 0 , measuringhow longit takes for the optimal policyπ 0 to reach a loop when starting froms 0 , and how long the loop itself is. We can think of optimal policies as implementing anexplore and exploitdynamic: navigating to a particularly high-reward area of the MDP, and then looping through that area to maximize reward indefinitely. Intuitively, a policy that takes longer to reach a stable loop can access more of the MDP and can thus reach higher-reward areas, while a policy that takes a bigger loop can incorporate more reward into the loop. 18 (a) Accuracy graph(b) Loss graph Figure 3: Accuracy and loss when passing in three different sets of features into the logistic classifier for predicting P(π=π 0 |USS) P(π=π 0 |UPS) . (a) Accuracy graph(b) Loss graph Figure 4: Accuracy and loss when passing in three different sets of features into the logistic classifier for predicting P(π=π 0 |USS) P(π=π 0 |URS) . 3. (O+S).Finally, if optimal policies are “power-seeking”, then we can try using correlates of POWER (Turner et al. [2023a]). Specifically, we use the sum of the number ofout-arrows, or directly reachable states, from a given statesfor allsthat an optimal policyπ 0 (s 0 ) reaches, and a binary value representing whetherπ 0 (s 0 ) =s 0 self-loops indefinitely. A.10 Miscellaneous MDP results We performed additional tests on goal-directedness. When we try to build a classifier for the P(USS) P(URS) definition of goal-directedness, we find that our current classifier architectures and features are insufficient, as shown in Figure 7. Some other results: 1.Less structured MDPs, such as MDPs where the transition probability distribution for each T(s,a)(for any statesand actiona) were IID randomized via Dirichlet distribution, tended to be harder to build a classifier for. Indeed, when we sampled from this set of MDPs, randomized the reward function10 4 times, and then calculated the optimal policy via value or policy iteration for each reward function, we found that the resulting distribution of optimal policies was roughly uniform (the mode policy occurred 1-3 times), and did not become less uniform with increased sparsity. This would make it harder to distinguish optimal policies from uniformly randomly generated policies. We found a similar, if slightly weaker, result for random deterministic MDPs (whereT(s,a)is 1 for some randoms ′ and 0 for all other states). 19 2.Looking at the logistic coefficients of the logistic when using self-loops and out-arrows individually as features, we found that more out-arrows correlated with a greater chance of a policy being sampled from URS/USS rather than UPS, while more self-loops correlated with a lesser chance. This matches, with weak confidence, what we would expect if “coherent” policies optimal for some reward function tended to preserve optionality, which was hypothesized in Turner et al. [2023a]. A.11 RL results We now present experiments in a more complex environment where it is intractable for optimal policies to be found via value or policy iteration. We approximate sampling near-optimal policies via USS or URS by training policies on their respective reward functions. We then train a classifier using these policies 6 and run the classifier on test cases to test if it is measuring goal-directedness properly. The purpose of these experiments is to see whether our definition of goal-directedness can be measured cheaply and effectively in complex environments. In the Taxi-v3 environment in OpenAI’s Gym (Brockman et al. [2016]), we trained 50 Q-tables with reward functions sampled from USS for 10,000 episodes each using Q-learning (Watkins [1989]), and 50 trained similarly with reward functions from URS. We used state-based rewards for URS and USS, and a sparsity value ofε= 0.99. We generated a policy greedy in value for each Q-table, then labeled each policy with which method it was generated by, and trained an ensemble of 40 binary classifiers using a graph convolutional neural network (Zhou et al. [2021]) (GCN) defined over the state and action space of the environment. We found that a GCN classifier could effectively predict the labels (< 0.2 test loss with enough epochs). We also used P(USS) P(URS) as a benchmark instead of P(USS) P(UPS) , intuitively measuring the sparsity of a policy’s reward function, for an intermediate baseline, as described in section A.6. We then wanted to test whether these classifiers actually identified what we intuitively think of as "coherent" policies. However, we found inconclusive results in this regard; see section A.11.1. A.11.1 Generalization tests We wanted to test how well our GNN classifiers generalize, meaning how well our GNN classifiers are able to predict goal-directedness for policies outside of the training dataset. We thus generated an ensemble of successful GNN classifiers, and passed in different sets of policies as test cases: 1.Taxi Q-table Policies.We generate Q-tables using Q-learning on Taxi with the normal reward function shown here, then generate greedy policies for each Q-table. Intuitively, policies subject to higher optimization pressure for some reward function should be rated high on the classifier for P(URS) P(UPS) , where a rating of 1 means URS and a rating of zero means UPS. The normal reward function for Taxi is somewhat sparse (+20 if delivering passenger to correct location, -10 if executing "pickup" and "drop-off" actions illegally, -1 otherwise), so that may also lead to a higher coherence score. 2.MCTS policies.Policies generated from conducting MCTS search ( ́ Swiechowski et al. [2022]) for 1000 iterations, using the Q-tables from our first set of policies (our previous RL training on Taxi) as rollout policies. The MCTS algorithm is often cited (Hubinger et al. [2019]) as an example of what a goal-directed model will internally emulate, so our hypothesis is that MCTS-generated policies will be more coherent and goal-directed according to our classifiers. 3.Non-stable-state policies.A la Turner et al. [2023a], given state-based rewards, near- optimal policies should prefer to stay in high-reward states. Thus, for each Taxi policy we trained in our first dataset, we randomly (with a50%IID chance) switch the action at each stateswhere the policy stays atsto actions that move to a different state. 6 We found that the elements of Q-tables generated via our RL implementation of USS tended to be much lower in magnitude than in Q-tables generated via URS. It seems unclear whether this actually correlates with increased goal-directedness or coherence, so our classifier could thus overfit when passing in Q-tables. We thus stuck to passing in only the policy into the FCNN and GNN; any patterns the classifier found between USS and URS policies should point to differences resulting from training policies in general with a sparse versus a dense reward function, i.e. what we want to measure. 20 (a) Accuracy graph (b) Loss graph Figure 5: Bigger version of figure 1 21 (a) Accuracy graph (b) Loss graph Figure 6: Bigger version of figure 3 22 (a) For P(URS) P(UPS) (b) For P(USS) P(URS) Figure 7: Train and test loss of an ensemble of 40 GCN classifiers given a binary classification task with datasets of size 50. Learning rate = 0.003, test/train ratio = 0.2 (a) Logit classifier ratings for the P(URS) P(UPS) classifier. Note that, in order to avoid numerical issues with classifying incoherent policies, we train the Taxi Q- tables for only ten episodes each. (b) Logit classifier ratings for the P(USS) P(URS) classifier. Here, we train the Taxi Q-tables for 20,000 episodes each. Figure 8: Logit binary classifier ratings given the datasets described previously. All error bars shown in the RL experiments assume a normal distribution and show the two-sigma error of the independent sampling of each value graded by 30-40 independently generated classifiers. Our results are shown in Figure 8. If our classifier ratesp= 1, then it believes thatP(USS) = 1and P(URS) = 0if classifying P(USS) P(URS) orP(URS) = 1andP(UPS) = 0if classifying P(URS) P(UPS) . If our classifier ratesp= 0, then it believes the opposite in both cases. We feed our classifier ratings through the logit functionl(p) = ln p 1−p for better analysis and visualization. The Taxi Q-tables were classified as strongly URS, and remained that way for a non-trivial number of trained episodes (>10). Although the magnitude varied by test, the MCTS policies were graded to be roughly as coherent as the policies from the Taxi Q-tables, but with the logit classifier rating having a lower magnitude. We speculate that this is the result of some misgeneralization from the training data because of the MCTS process, resulting in the classifier not having predictions as confident as for policies generated from Taxi Q-tables. Finally, policies with fewer 1-cycle states were graded as strongly URS, more so than the Taxi Q-tables, which was surprising. We note that considerable effort was put to make sure that no biases were unintentionally left in the datasets, potentially leading to the classifiers measuring a confounding factor. For example, if the classifiers overfit to inductive biases in the training process, then it would be measuring “how much" 23 the policies were trained instead of goal-directedness itself, which is undesirable. To combat this, we incorporated policies trained with MCTS for 1000 iterations into half of the USS and URS policy datasets. However, we did see some negative evidence in previous experiments; these are also listed in the appendix. These results are also sensitive to hyperparameters; P(URS) P(UPS) classifiers with greater test loss, for instance, tended to rate MCTS with lower goal-directedness than the Taxi Q-table policies they were derived from (Figure 12). We can surmise that that the high sparse reward in Taxi (when the passenger is successfully dropped off) can only be achieved once, while given randomly sampled sparse reward functions over states, policies can loop around and repeatedly earn high rewards until the maximum number of states is reached. Thus Taxi policies functionally act as if they are in URS; we note that this implies that certain features in the environment are required or helpful for policies to exhibit goal-directed behavior, as Turner et al. [2023a] also shows for power-seeking behavior. For a similar reason, policies that are changed to not have cycles (while also still pursuing high-reward states), like Taxi policies, are similarly graded URS. However, we note that the above explanation, along with the RL generalization experiments in general, are incomplete/inconclusive, which is why we acknowledged as such in this paper and put those experiments in the appendix. More generally, coming up with good, tractable ground truth examples to check that our classifiers measure an already-nebulous concept like goal-directedness is difficult. Thus, we should not put too much weight on positive or negative results on these specific benchmarks. We thus see some evidence that the predictions of our classifiers match our intuitions around goal-directedness, which lends evidence to the claim that our classifiers are identifying goal-directedness properly. A.11.2 Hyperparameters Below we list additional data from our RL experiments. In figure 7, we graph the loss of a sample batch of 40 graph convolutional network (GCN) classifier runs on both binary classification tasks. Each data point consisted of a deterministic greedy policy generated from a Q-table of dimension (|S|,|A|) = (500,6). Each table was trained with learning rateλ= 0.01and discount rateγ= 0.99 for some number of episodes, under standard Q-learning (Watkins [1989]) with exploration factor εdecaying from 1.0 to 0.01 over the first half of the total number of episodes. For URS and USS policies, each Q-learning run also had a state-based reward function, whereR(s)∼U[−1,1]I.I.D. in the case of URS or for a1−k= 0.01proportion of the states of USS policies. For the other k·100% = 99%states,R(s)≃0(technicallyU[10 −9 ,10 9 ]to avoid issues with many possible similar-performing policies). After generating these deterministic policies, we trained a 2-layer, 16-node-wide GCN with a singular output, parsed through a sigmoid function to create a binary prediction. This training process had a learning rate between 0.01 and 0.1 depending on the run, but most commonly (and in Figure 7) had λ= 0.03, along with a rate decay of5·10 −4 over 80-150 epochs. Classifiers that ended up with a test loss of less than 0.6 (i.e. better than average) were added to the ensemble for testing. A.11.3 Additional tests and limitations We ran multiple additional tests on our ensemble classifiers to see how well they generalize. We ran Taxi Q-table agents (with the default reward function) for a variable number of episodes, to see whether the P(USS) P(URS) ratings remained consistent. As shown in Figure 9, the classifier ratings remained near-zero no matter how many episodes (greater than a trivial amount, like ten) we trained the Taxi Q-table agents for. This is important because it indicates that we can train a goal-directedness classifier on policies trained with a small number of episodes, thus being computationally cheaper to generate, and have it generalize to policies trained with a larger number of episodes. However, we had an additional test for our goal-directedness classifiers that ended up failing. Ac- cording to Wentworth [2024], we can consider thelong-termcoherence of a determinstic policy in an MDP as whether there exists a value functionwith zero immediate payoffthat satisfies the Bellman equation (BELLMAN and Dreyfus [2010]) and is consistent with the policy. As such, to create an “incoherent" dataset, we take our policies from our Taxi Q-tables and randomly switch about 70-80 actions such that the policy gets stuck in loops and is not optimal for any value function. This process should result in a set of policies with lower goal-directedness, but as shown in Figure 10, this process ends up ultimately giving roughly equal goal-directedness as rated by our classifiers. Indeed, note that the actions are labeled from 0 to 5, with actions 4 and 5 resulting in no change to the 24 (a) For 0-3000 episodes(b) For 0-175000 episodes Figure 9: P(USS) P(URS) classifier ratings (pulled randomly from the ensemble) of Q-tables trained on Taxi with the normal reward function for a given number of episodes. Distribution of episodes generated viaint(np.random.lognormal(3, 1)) * iwhereiis 10 in the left graph and 1,000 in the right graph. Since all the values are near-zero, the standard deviation and error is thus also near-zero (although the distribution is not well-modeled as normal anyways). (a) For P(USS) P(URS) (b) For P(URS) P(UPS) Figure 10: Classifier ratings with the old (in)coherence definition according to Wentworth [2024]. environment (unless some specific conditions are held, i.e. if the taxi is at the passenger location or at the destination location with the passenger onboard). When we switchedallof the actions in each policy in this third dataset such that we added one to even actions and subtracted one to odd actions (meaning that policies that stayed at a state continued to stay at a state), our goal-directedness rating was still unchanged. This gave strong evidence that our classifiers were not measuring something akin to the definition proposed by Wentworth [2024], but instead something else (like what we measured in section 4.2, i.e. how often the policy tended to stay at a statesfor every possibles). This gives negative evidence that our classifiers are generalizing properly. On the other hand, perhaps the definition provided by Wentworth [2024] is not a good definition for our purposes in the first place. To test this, we train a Taxi Q-table using Q-learning RL and, at intervals of 400 episodes, calculate the difference between a Q-valueQ(s,a)and the maximum of the Q-values at the next states ′ given statesand actiona: |Q(s,a)−γmax a ′ ∈A Q(s ′ ,a ′ )| Theoretically, ifγ≈1and the relative weight of near-term rewards goes to zero, andQ(s,a)is near-optimal for some utility function, then this difference should be as close to zero as possible. 25 Figure 11: The average differences between each element of a Q-table and its “ideal" value, as proposed by Wentworth [2024], of a Taxi Q-table in training over some episodes. (a) Logit classifier ratings for the P(URS) P(UPS) classifier. Note that, in order to avoid numerical issues with classifying incoherent policies, we train the Taxi Q- tables for only ten episodes each. (b) Logit classifier ratings for the P(USS) P(URS) classifier. Here, we train the Taxi Q-tables for 20,000 episodes each. Figure 12: Logit binary classifier ratings with lower binary classifier losses (0.5 on average) However, as shown in Figure 11, this difference actually increases with the number of episodes. In fairness, Figure 9 shows that Taxi Q-tables are not maximally goal-directed as the number of episodes goes to infinity, instead gravitating towards being classified as URS policies, so this comparison is probably not fair. Nevertheless, this shows that modelling goal-directed policies as approximately satisfying a Bellman equationwith zero payoffis a very strong assumption that may not match what we want to describe with goal-directedness. We also attempted to use a logistic classifier directly over the policy as a goal-directedness classifier. This did not work and ended up overfitting. A.12 Miscellaneous LLM results The implementation of the dense loss function in PyTorch is as follows: def dense_loss(model_output, labels, n=None): loss_fct = torch.n.CrossEntropyLoss() loss = loss_fct(model_output.logits.view(-1, model_output.logits.size(-1)), labels.view(-1)) 26 Figure 13: Logit binary classifier ratings of P(USS) P(URS) withk= 0.1·N. We see that all the policies are graded significantly closer to USS than in Figure 8. Figure 14: Logit ensemble binary classifier ratings for P(URS) P(UPS) using Taxi policies trained for 20,000 episodes (and MCTS and reduced 1-cycle policies using those policies). return loss In this implementation: •model_output.logitsare the predicted logits from the model. •labelsare the true class labels. •torch.n.CrossEntropyLoss()is the loss function provided by PyTorch for calculating cross-entropy loss. •Theview(-1, model_output.logits.size(-1))operation reshapes the logits and labels to ensure they are in the correct format for the loss function. def sparse_loss(model_output, labels, number_logits=10): n = number_logits # Extract the last n logits and labels logits = extract_random_n_tokens(model_output.logits, n) labels = extract_random_n_tokens(labels, n) 27 # Flatten the tensors for cross-entropy loss calculation logits = logits.view(-n, logits.size(-1)) labels = labels.view(-n) loss_fct = torch.n.CrossEntropyLoss() loss = loss_fct(logits, labels) return loss We also found preliminary evidence that our sparse-1-token LLM classifier correctly predicts, with approximately66%accuracy, a model trained with direct preference optimization (Rafailov et al. [2024]) as sparse, confirming generalization to a frontier training technique with sparse loss signal (as DPO only backpropagates once per sequence of text) used in current research. We note that this is a preliminary tentative result. A.13 Limitations and future work One challenge with this project is that, because the “coherence" or “goal-directedness" of policies are difficult to definitively categorize outside of toy examples, it is difficult to determine a “ground truth" as to whether our classifiers are actually measuring goal-directedness or not. For example, we want our goal-directedness classifiers to not just be measuring the test performance of a policy, as this assumes that policies trained in a certain manner will be goal-directed, which is the hypothesis that we are trying to test in the first place. As our experiments get more and more complex, our metric becomes progressively more and more approximate. Stronger, more efficient classifiers will likely use interpretability techniques (e.g. Nanda et al. [2023], Zou et al. [2023], Turner et al. [2023b]). There is also room for progress on our theoretical model of coherence. For instance, we also define our reward functions over states in the environment, while in real life, it seems more realistic to define reward functions over human abstractions or concepts (Udell [2022]), i.e. patterns in the environment. Extensions to partially-observable settings utilizing belief-states instead of “ground-truth" states would also be useful, e.g. Biehl and Virgo [2022]. Our usage of sparsity could also be replaced with some notion of compressibility, perhaps using the inductive biases of the network being studied (Goyal and Bengio [2022]). Finally, the model could be generalized to continuous state spaces; our current method of dealing with continuity is to discretize the space and then apply our uniform reward sampling methods, but this can fail computationally. Finally, within the experiments that included the LLMs, there are some limitations that must be addressed. Our results show that the classifier for the dense loss function can generalize between two datasets; however, we have not shown this can generalize between different types of models. For instance, our goal-directedness classifiers may be overfitting to inductive biases in our fine-tuning optimization process in general, rather than from our fine-tuning process to any specific dataset. Expanding our train and test dataset to include neural networks trained with various hyperparameters on more datasets could strengthen our results. A.14 Recommendations As discussed previously, goal-directedness is a crucial requirement for the most dangerous scenarios involving smarter-than-human AIs in the future. We thus recommend that more research be done into measuring goal-directedness for frontier models in realistic environments. Concretely, this could result in goal-directedness being added into evaluations of potentially dangerous new models, such as those outlined for models ASL-3 and above in Anthropic’s Responsible Scaling Policy (Anthropic [2023]). However, we want to emphasize that our work is quite preliminary andsubstantialresearch needs to be done before we can form reliable goal-directedness evaluations. More broadly, understanding goal-directedness is a bottleneck to devising clearer threat models and better solutions to preventing misaligned AIs (Adam Shimi [2021], Ngo [2020], Carlsmith [2023]). Thus, better measurements of goal-directedness of deep learning systems could allow future experiment to know whether they are working on goal-directed models or not, reducing assumptions. This along with more speculative impacts are discussed further in section A.4. 28 A.15 Further discussion After reviewing the relevant literature, we provide a definition of goal-directedness that is intuitive and more tractable to compute than existing definitions, and a method to create a goal-directedness classifier. We first show that within an MDP setting, our definition is measurable and correlates with features associated with power-seeking. Features like a deterministic policy not taking self-loops and maximizing the number of out-arrows visited at each state tend to occur in optimal, power-seeking policies as shown by Turner et al. [2023a], and also occur in goal-directed policies according to our methods. We then adapt our methods to RL environments and again provide evidence that our definition is measurable, even in complex environments. Finally, we provide a demonstration of our method on LLMs, where we conducted experiments to measure the classifier’s ability to identify activations from sparse and dense loss functions, as defined in Sections A.3.4 and A.3.4. Our preliminary results indicate that it is possible to achieve an accuracy greater than 70 percent for the dense loss function. As shown in Figure 2, when a model is fine-tuned on the loss signal from a specific dataset (GSM8K Cobbe et al. [2021] and the Orca Math dataset Mitra et al. [2024]), and a classifier is trained on a different dataset, the classifier’s ability to detect signals within the activations of the dense loss function increases with the training steps. This result is meaningful because it demonstrates that measuring the dense loss function is feasible across different datasets for the same model. In the future, we would want to train a goal-directedness classifier across models trained on multiple different datasets, over different stages and kinds of training, and over different hyperparameters. While this would be more difficult, the classifier would also generalize better. Overall, we believe that the LLM experiments show a promising method to further understand coherence in LLMs. However, these results are not conclusive, and require more experiments to better understand the intricacies of how "goal directedness" appears in activations. A.16 Compute disclosure For the MDP and RL experiments in Sections 4.1 and 4.2, only a CPU is needed. Note, however, that it takes about three hours on a Thinkpad laptop to run the RL experiments fully; this is reduced significantly by using a more powerful CPU processor. For the LLM experiments, we used a single A100 SXM with 32 vCPU and 251 GB RAM on Runpod. A.17 Credit to existing packages In addition to the citations already present in the paper, we used the following Python (Van Rossum and Drake [2009]) packages: PyTorch (Paszke et al. [2019]), NumPy (Harris et al. [2020]), scikit-learn (Pedregosa et al. [2018]), random, tqdm (da Costa-Luis et al. [2024]), matplotlib (Hunter [2007]), os, re, math, pickle (Van Rossum [2020]), peft (Mangrulkar et al. [2022]), transformers (Wolf et al. [2020]), datasets (Lhoest et al. [2021]), and wandb (Biewald [2020]). 29