Paper deep dive

Goal Misgeneralization in Deep Reinforcement Learning

Lauro Langosco, Jack Koch, Lee Sharkey, Jacob Pfau, Laurent Orseau, David Krueger

Year: 2022Venue: ICML 2022Area: Deception & FailureType: EmpiricalEmbeddings: 62

Models: PPO agents (actor-critic)

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 93%

Last extracted: 3/12/2026, 8:12:21 PM

Summary

The paper formalizes 'goal misgeneralization' in deep reinforcement learning, a failure mode where an agent retains its capabilities out-of-distribution but pursues an incorrect proxy objective rather than the intended reward. The authors provide empirical demonstrations using the Procgen benchmark and gridworld environments, showing that agents often learn proxies like 'move right' or 'collect keys' instead of the true task, and suggest that increasing training diversity can help mitigate this issue.

Entities (5)

Goal misgeneralization · phenomenon · 100%Reinforcement Learning · field · 100%Capability Generalization · concept · 95%Procgen · benchmark · 95%Proxy Objective · concept · 90%

Relation Signals (3)

Goal misgeneralization → isatypeof → Out-of-distribution generalization failure

confidence 95% · Goal misgeneralization, a type of out-of-distribution generalization failure in reinforcement learning (RL).

Training Diversity → alleviates → Goal misgeneralization

confidence 90% · We show that goal misgeneralization may be alleviated by increasing the diversity of the training distribution.

Procgen → usedtodemonstrate → Goal misgeneralization

confidence 90% · We experimentally showcase goal misgeneralization. Specifically, deep RL agents trained on the Procgen benchmark... still fail on our slightly modified environments.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:We study goal misgeneralization, a type of out-of-distribution generalization failure in reinforcement learning (RL). Goal misgeneralization failures occur when an RL agent retains its capabilities out-of-distribution yet pursues the wrong goal. For instance, an agent might continue to competently avoid obstacles, but navigate to the wrong place. In contrast, previous works have typically focused on capability generalization failures, where an agent fails to do anything sensible at test time. We formalize this distinction between capability and goal generalization, provide the first empirical demonstrations of goal misgeneralization, and present a partial characterization of its causes.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

61,547 characters extracted from source content.

Expand or collapse full text

Goal Misgeneralization in Deep Reinforcement Learning Lauro Langosco * 1 Jack Koch * Lee Sharkey * 2 Jacob Pfau 3 Laurent Orseau 4 David Krueger 1 Abstract We studygoal misgeneralization, a type of out- of-distribution generalization failure in reinforce- ment learning (RL). Goal misgeneralization oc- curs when an RL agent retains its capabilities out- of-distribution yet pursues the wrong goal. For instance, an agent might continue to competently avoid obstacles, but navigate to the wrong place. In contrast, previous works have typically focused on capability generalization failures, where an agent fails to do anything sensible at test time. We formalize this distinction between capability and goal generalization, provide the first empiri- cal demonstrations of goal misgeneralization, and present a partial characterization of its causes. 1. Introduction Out-of-distribution (OOD) generalization, performing well on test data that is not distributed identically to the training set, is a fundamental problem in machine learning (Arjovsky, 2021). OOD generalization is crucial since in many applica- tions it is not feasible to collect data distributed identically to that which the model will encounter in deployment. In this work, we focus on a particularly concerning type of generalization failure that can occur in RL. When an RL agent is deployed out of distribution, it may simply fail to take useful actions. However, there exists an alternative failure mode in which the agent pursues a goal other than the training reward while retaining the capabilities it had on the training distribution. For example, an agent trained to pursue a fixed coin might not recognize the coin when it is positioned elsewhere, and instead competently navigate to the wrong position (Figure 1). We call this kind of failure goal misgeneralization 1 and distinguish it fromcapabil- * Equal contribution 1 University of Cambridge 2 University of T ̈ ubingen 3 University of Edinburgh 4 DeepMind, London. Corre- spondence to: Lauro Langosco<langosco.lauro@gmail.com>. Proceedings of the39 th International Conference on Machine Learning, Baltimore, Maryland, USA, PMLR 162, 2022. Copy- right 2022 by the author(s). 1 We adopt this term from Shah et al. (2022). A previous version of our work used the term ‘objective robustness failure’ instead. We (a) Goal position fixed(b) Goal position randomized Figure 1.(a)At training time, the agent learns to reliably reach the coin which is always located at the end of the level.(b)However, when the coin position is randomized at test time, the agent still goes towards the end of the level and often skips the coin. The agent’s capability for solving the levels generalizes, but its goal of collecting coins does not. ity generalizationfailures. We provide the first empirical demonstrations of goal misgeneralization to highlight and illustrate this phenomenon. While it is well-known that the true reward function can be unidentifiable in inverse reinforcement learning (Amin & Singh, 2016), our work shows that a similar problem can also occur in reinforcement learning when features of the environment are correlated and predictive of the reward on the training distribution but not OOD. In this way, goal misgeneralization can also resemble problems that arise in supervised learning when models use unreliable features: both problems are a form of competent misgeneralization that works in-distribution but fails OOD. Disentangling ca- pability and goal generalization failures is difficult in su- pervised learning; for instance, are adversarial examples bugs or features (Ilyas et al., 2019)? In contrast, studying RL allows us to formally distinguish capabilities and goals, which roughly correspond to understanding the environment dynamics and the reward function, respectively. Goal misgeneralization might be more dangerous than ca- pability generalization failures, since an agent that capably pursues an incorrect goal can leverage its capabilities to visit arbitrarily bad states (Zhuang & Hadfield-Menell, 2021). In contrast, the only risks from capability generalization fail- ures are those of accidents due to incompetence. use the term ‘goal’ to refer to goal-directed (optimizing) behavior, notjust goal-states in MDPs. arXiv:2105.14111v7 [cs.LG] 9 Jan 2023 Goal Misgeneralization in Deep Reinforcement Learning An important implication of goal misgeneralization is that training a model by optimizing an objectiveRis not enough to guarantee that the model will itself learn to pursueR rather than some proxy forR. This is especially concerning in the context ofAGI safety: ensuring that advanced AI systems are safe despite being capable enough to escape our control (Bostrom, 2014). Systems that competently pur- sue a misaligned goal may tend to seek power and deceive their operators for instrumental reasons (Turner et al., 2021; Omohundro, 2008). With highly advanced AI systems, this could lead to human disempowerment: for example, an AI system might prevent its operators from shutting it down (Hadfield-Menell et al., 2017a; Soares et al., 2015). If com- plex tasks are rife with proxies for their training objectives, it may be very hard to predict what objectives the trained AI systems will have (Hubinger et al., 2019). Our main contributions are: •We formalize goal misgeneralization, distinguishing it from capability generalization failures (Section 2), and experimentally validate our definition on a gridworld environment (Section 3.5). •We experimentally showcase goal misgeneralization. Specifically, deep RL agents trained on the Procgen benchmark (Cobbe et al., 2019)—a set of diverse procedurally generated environments specifically de- signed to induce robust generalization—still fail on our slightly modified environments (Section 3). •We show that goal misgeneralization may be alleviated by increasing the diversity of the training distribution so that the agent learns to distinguish the reward from proxies (Sections 3.1 and 3.2). •We investigate the causes of goal misgeneralization. In particular, we find that the actor and the critic compo- nents of our actor-critic model learn different proxy goals (Section 3.4). 2. Goal Misgeneralization Goal misgeneralization is a type of OOD generalization failure. OOD generalization is usually studied in the super- vised learning setting, where it is defined as achieving good test performance on data sampled from a distribution other than the training distribution. We focus on the reinforce- ment learning setting (Sutton & Barto, 2018), in which a system is trained to take actions in an environment in order to maximize a given reward. In this setting, the problem is to achieve high reward despite a shift in the distribution of observations or the transition dynamics. OOD general- ization problems frequently arise in RL and are an active area of research (Kirk et al., 2021). However, goal mis- generalization in particular has not been the focus of any previous academic work. Studying this class of failures is particularly important from the point of view of machine learning safety (Hendrycks et al., 2021), since agents that pursue imperfect proxies may fail suddenly (Pan et al., 2022; Ibarz et al., 2018) and catastrophically (Zhuang & Hadfield- Menell, 2021) as their capabilities increase. With this in mind, we provide a definition of goal misgeneralization and show how it can be formalized. 2.1. Defining Goal Misgeneralization A deep RL agent is trained to maximize a rewardR : S× A×S→R, whereSandAare the sets of all valid states and actions, respectively. Assume that the agent is deployed out-of-distribution; that is, an aspect of the environment (and therefore the distribution of observations) changes at test time.Goal misgeneralizationoccurs if the agent now achieves low reward in the new environment because it continues to act capably yet appears to optimize a different rewardR ′ 6=R. We callRtheintended objectiveandR ′ thebehavioral objectiveof the agent. Formally, we follow Orseau et al. (2018) in distinguishing goal-directed policies (agents) from unoptimized policies (devices). Letη agt (R)andη dev (d)be priors over a space of reward functionsR∈Rand a space of devices (policies) d∈Πrespectively. Further letp agt (τ|R)andp dev (τ| d) be the likelihood functions giving the probability of a trajectoryτgiven a particular objectiveRor deviced. We define two distributions over trajectories, the agent mixture p agt and the device mixturep dev : p agt (τ) = ∑ R∈R p agt (τ|R)η agt (R),(1) p dev (τ) = ∑ d∈Π p dev (τ|d)η dev (d).(2) The choice of device likelihoodp dev (τ|d)is straightfor- ward: we simply choose the distribution over trajectories induced by running the policydin the environment. For the agent likelihoodp agt (τ|R), a popular choice is the max- imum entropy modelp agt (τ|R)∝exp(R(τ))(Ziebart et al., 2008). Another possibility is to choosep agt (τ|R)to be the probability density of the random trajectory obtained by training an RL algorithm to maximizeRand collecting rollouts. 2 Definition 2.1(Goal misgeneralization).A policyπun- dergoesgoal misgeneralizationif test reward is low and p agt (τ)> p dev (τ)holds on average for the trajectories induced byπin the OOD test environment. In other words, the policy is acting in a goal-directed manner, but not achiev- ing high reward. We can infer a posterior distribution over 2 This requires an RL algorithm and model (e.g. neural network). In practice, this choice ofp(τ|R)is intractable to compute. Goal Misgeneralization in Deep Reinforcement Learning behavioral objectives: η agt (R|τ)∝p agt (τ|R)η agt (R). In Section 3.5 we compute these mixtures explicitly and validate Definition 2.1 in a gridworld environment. 2.2. Causes of Goal Misgeneralization When should we expect models to learn robust goals? We begin by suggesting possible prerequisites for goal misgen- eralization: 1.The training environment must be diverse enough to learn sufficiently robust capabilities. 2.There must exist some proxyR ′ : S×A×S→Rthat correlates with the intended objective on the training distribution, but comes apart (i.e. is much less corre- lated, or anti-correlated) on the OOD test environment. These conditions are necessary for goal misgeneralization to arise: If (1) is not the case, then RL algorithms tend to memorize simple action sequences that work in the training environment but are not robust under distributional shift (Cobbe et al., 2019). Meanwhile (2) is necessary because by assumption the policy achieves high (training) reward; thus the behavioral objective must be correlated with the intended objective on the training environment. However, (1) and (2) are by no means sufficient since, by themselves, they do not guarantee that the model learns to pursue the proxy rewardR ′ instead of the intended objective. We note that assumptions (1) and (2) are quite weak: al- most every real-world problem requires a diverse training environment (to learn robust capabilities), and proxies are common in complex environments. Thus goal misgeneral- ization depends mostly on whether the inductive biases of the model and training algorithm prime it to learn a proxy that then diverges from the intended objective on the test set. We expect that learned proxies will: •be correlated with the intended objectiveRon the train- ing distribution but not necessarily the test distribution. •tend to be easier to learn than the intended objectiveR because a proxyR ′ may: –use features that are simpler or more favored by the inductive biases of the model compared with the intended objective (Valle-P ́ erez et al., 2019; Geirhos et al., 2020). –be denser than the intended objective (Singh et al., 2010). For example, despite being a product of evolution (which optimizes for genetic fitness), humans tend to be more con- cerned with proxy goals, such as food or love, than with maximizing the number of their descendants. This illustrates a general phenomenon: given a challenging goal (such as “maximize fitness”), complex environments are rife with proxies and sub-goals (such as “eat rich food”) of that goal, many of which are more dense or simpler to optimize than the original goal. This observation has previously been made by Singh et al. (2010), who also draw the analogy with evolution, and note that bounded agents (i.e. with limited experience and/or computation) will often achieve higher expected rewardaccording to the true rewardwhen trained to optimize a proxy reward function. 3. Experiments Having defined goal misgeneralization and outlined when and why we expect it to occur, we now present experiments designed to demonstrate different kinds of goal misgeneral- ization and distinguish them from capability generalization failures. 3 In each experiment, we train an agent that performs capably when deployed out-of-distribution, but pursues a behav- ioral objective different from the objective for which it was trained. This behavior is consistent across multiple random seeds for training. For each of our experiments we hypothesize a behavioral objective that the policy has learned: navigating to the right- hand end of the level (CoinRun), navigating to the upper right corner (Maze I), navigating to the yellow object (Maze I) and gathering keys (Keys and Chests). None of these is a robust proxy for the intended objective. It is possible that there exist alternate objectives that also explain this behavior: for example, navigating towards a tall, left-facing wall (CoinRun). For our purposes, it is enough to show that a plausible proxy objective exists. Nonetheless, we conduct a series of experiments that confirm the ‘move right’ hypothesis over the ‘move to wall’ hypothesis for the CoinRun agent’s behavioral objective (see Section 3.4). We follow a zero-shot protocol in all experiments except Fig- ure 2: the agent does not see the (OOD) testing environment during training. Except in Section 3.5, all environments are adapted from the Procgen environment suite (Cobbe et al., 2019). This suite is built to study sample efficiency and gen- eralization to within-distribution tasks. Agents (feedforward neural networks trained using Proximal Policy Optimization - further details in the Appendix) are tasked with performing well in an arcade-like video game from pixel observations. The environments are procedurally generated and thus di- 3 Our code can be found athttps://github.com/ JacobPfau/procgenAISC (Environments) andhttps:// github.com/jbkjr/train-procgen-pytorch (Train- ing). Video examples of goal misgeneralization in all of the following environments can be found at this link. Goal Misgeneralization in Deep Reinforcement Learning 0.02.55.07.510.012.515.017.520.0 Probability (%) of a level with randomized coin. 0 20 40 60 80 Frequency of failure(%) Baseline (fully objective nonrobust) Failure of Capability Goal Misgeneralization Figure 2.Goal generalization is greatly improved in our Coin- Run experiments when just 2% of training levels have randomly placed coins (like the test environment). More randomization helps. Baseline: Since even a policy that entirely ignores the coin may sometimes hit the coin by accident, we compute a base rate for a ‘fully misgeneralizing’ policy that treats the coin as invisible. verse; to perform well, an agent must learn strategies that work in a wide range of task settings and difficulties and cannot rely on e.g. memorizing a small number of trajec- tories to solve a fixed set of levels. This diversity alone is insufficient to prevent goal misgeneralization, however; diversity of a different sort is needed, as we demonstrate in Figure 2. Different kinds of failure.The experiments illustrate dif- ferent flavors of goal misgeneralization.Directional proxies (CoinRun): the agent learns to move to the right instead of to the true source of reward (the coin).Location proxies (CoinRun, Maze I): In Maze I, the agent learns to navigate to the upper right corner instead of to the true source of re- ward (the cheese). The critic—but not the actor—also learns such a proxy in CoinRun.Observation ambiguity(Maze I): The observations contain multiple features that identify the goal state, which come apart in the OOD test distribution. Instrumental goals(Keys and Chests): The agent learns an objective (collecting keys) that is only instrumentally useful to acquiring the intended objective (opening chests). 3.1. CoinRun In the Procgen CoinRun environment, the agent spawns on the left side of the level and has to avoid enemies and obstacles to get to a coin. The coin yields a reward of 10, all other rewards are 0. In our training environments, the coin is always located at the right end of the level next to a wall; reaching the coin terminates the episode. To evaluate goal misgeneralization, we create test environments in which the coin is located in a random (accessible) location. After training, the agent competently navigates to the end of the level in the training environment. At test time, the agent generally ignores the coin completely and proceeds to the end of the level, as shown in Figure 1. This suggests that the agent has learned the proxy objective of “move right” rather than “move to the coin”. It competently achieves this behavioral objective, which is perfectly correlated with the intended objective on the training distribution and appears to be easier for the agent to learn; nevertheless, test reward is low because the behavioral objective misgeneralizes. (a) Train - Goal position fixed Variant 1Variant 2 (d) Test - Red line or yellow gem?(b) Test - Goal position random (c) Train - Goal is yellow line Figure 3.The agent (the mouse) is trained to navigate mazes to reach its goal.(a & b)An agent is trained on procedurally- generated mazes with the cheese in a fixed position (top right corner) ignores it and navigates to the top right corner when the cheese is placed randomly.(c & d)An agent trained to navigate to a yellow line consistently navigates to a yellow gem when de- ployed in environments in which there are only red lines and yellow gems. If it is meant to collect lines and not gems, this is a case of goal misgeneralization. Training with randomly placed coins.To test how con- sistent goal misgeneralization is, we train a series of agents on environments which vary in how often the coin is placed randomly. Results can be seen in Figure 2, which shows the frequencies of two different outcomes: 1.Failure of capability:the agent dies or gets stuck, thus neither getting the coin nor to the end of the level. This is evaluated on the training environments where the coin is typically at the end of the level. 2.Goal misgeneralization: the agent misses the coin and navigates to the end of the level. This is evaluated on the OOD test environments where coin location is randomized. As expected, as the diversity of the training environment in- creases, the probability of goal misgeneralization decreases, as the model learns to pursue the coin instead of going to the end of the level. We also include a baseline which measures Goal Misgeneralization in Deep Reinforcement Learning the rate at which an invisible “coin” would be captured, to determine how often the coin would be captured by an agent that completely ignores it. We see that even when the coin is always at the end of the level during training, the rate of goal misgeneralization is lower than this baseline. 3.2. Maze Variant 1. We modify the Procgen Maze environment in order to implement an idea from Hubinger (2020b). In the original environment, a maze is generated using Kruskal’s algorithm (Kruskal, 1956), and the agent is trained to navi- gate towards a piece of cheese located at a random spot in the maze. We modify the original environment so that the cheese is always in the upper right corner (Figure 3a). As in the CoinRun experiment, when an agent is trained on the en- vironment with a consistent reward location but tested in an environment with a random reward location, the agent ignores the randomly placed objective, instead navigating to the upper right corner of the maze (Figure 3b). The intended objective is to reach the cheese, but the behavioral objec- tive of the learned policy is to navigate to the upper right corner. Somewhat surprisingly, we also find that the agent continues to pursue a proxy objective of “move to the upper right corner” even when this proxy becomes imperfect (see Figure 4). Variant 2. In the experiments so far, goal misgeneraliza- tion arises due to an ambiguity between a visual feature (coin / cheese) and a positional feature (right / top right) which come apart at test time. To illustrate a different kind of distributional shift, we present a simple setting in which there is nopositionalfeature that favors one objective over the other; instead, the agent is forced to choose between two ambiguous visual cues. We train an RL agent on a version of the Procgen Maze environment where the reward is a randomly placedyellow diagonal line(Figure 3c). At test time, we deploy it on a modified environment featuring two randomly placed ob- jects: a yellowgemand areddiagonal line; the agent is forced to choose between consistency in shape or in color (Figure 3d). Except for occasionally getting stuck in a cor- ner, the agent usually pursues the yellow gem, thus general- izing in favor of color rather than shape consistency (89% of the time, excluding occasions where it must pass through the red line to get to the yellow gem,n= 102). As in pre- vious examples, training with the correct reward function is not enough to guarantee correct goal generalization here; rather, another approach such as increasing environment di- versity or using a different inductive bias may be necessary to specify the intended OOD behavior. 2468101214 Size of randomization region 4 6 8 10 Training Reward Test Reward Figure 4.The goal is randomly located within a region of size 1−16in the upper right corner of the maze. As the region grows, validation performance on the fully randomized environment im- proves (i.e. correct goal generalization is more likely). However, the agent still uses location as a proxy until the region is quite large. (a) Num Chests > Num Keys(b) Num Keys > Num Chests Figure 5.Goal misgeneralization on the “Keys and chests” task. The agent must collect keys in order to open chests and is only re- warded for opening chests.(a)The agent is trained on procedurally- generated mazes in which there are twice as many chests as keys. (b)At test time, there are instead twice as many keys as chests. The agent routinely prioritizes collecting all the keys before opening any remaining chests despite the fact that doing so offers no benefit to its episode reward (in fact, it woulddecreaseits time-discounted return). 3.3. Keys and Chests So far, our experiments featured environments in which there was a proxy that was perfectly correlated with the intended objective on the training distribution. The Keys and Chests environment, first suggested by Barnett (2019), provides a different type of example. This environment, which we implement by adapting the Heist environment from Procgen, is a maze with two kinds of objects: keys and chests. Whenever the agent comes across a key it is added to a key inventory. When an agent with at least one key in its inventory comes across a chest, the chest is opened and a key is deleted from the inventory. The agent is rewarded for every chest it opens. As in previous experiments, we induce goal misgeneral- ization by subjecting the agent to different training and test environment distributions: In the training environment, there are twice as many chests as keys, while in the test en- Goal Misgeneralization in Deep Reinforcement Learning vironment there are twice as many keys as chests. The basic task facing the agent is the same (the reward is only given upon opening a chest), but the circumstances are different. We observe that an agent trained on the “many chests” dis- tribution goes out of its way to collect all the keys before opening the last chest on the “many keys” distribution (Fig- ure 5 and Figure 10, Appendix), even though only half of them are even instrumentally useful for the intended objec- tive; occasionally, it even gets distracted by the keys in the inventory (which are displayed in the top right corner) and spends the rest of the episode trying to collect them instead of opening the remaining chest(s). We describe the agent as having learned a simple behavioral objective: collect as many keys as possible, while some- times visiting chests. This strategy leads to high reward in an environment where chests are plentiful and the agent can thus focus on looking for keys. One reason that the agent may have learned this proxy is that the proxy is less sparse than the intended objective while nevertheless being correlated with it on the training distribution. However, the proxy fails when keys are plentiful and chests are no longer easily available. 3.4. Critic Generalization vs. Actor-Critic Generalization All of the experiments above use PPO (Schulman et al., 2017), an actor-critic method (Sutton et al., 1998). In these methods, the policy (“actor”) learns to optimize an approxi- mate value function provided by the “critic”. So far, we’ve demonstrated goal misgeneralization, where the actor be- haves in a goal-directed manner but doesn’t achieve high test reward. In this section we analyze the CoinRun exper- iment more closely and show that the actor and the critic bothfail to generalize OOD; furthermore, theyfail in dif- ferent ways. We conclude that the actor and the critic have different inductive biases that lead them to fail in different ways. Critic Misgeneralization. In order to determine how much the critic values the coin (the intended objective) vs. reaching the end of the level (the proxy objective), we com- pare the value it assigns to states where these factors are var- ied (Figure 6). We find that the value (i.e. the output of the critic) is much higher at the end of the level than elsewhere, and that the presence of the coin makes no discernible dif- ference. Thus we conclude that the critic misgeneralizes, assigning high value to the proxy instead of the intended objective. To help identify the features in observations at the end of the level that cause higher value, we generate attribution maps by taking the gradient of the value function output with respect to the observation, following Simonyan et al. (2013). The end-wall is highlighted at least as much Beginning (No coin) Beginning (with coin) Middle (No coin) Middle (with coin) End (No coin) End (with coin) After End (No coin) After End (with coin) 5 6 7 8 9 10 Value function output Figure 6.The average value function output for images (n= 950) at different stages of CoinRun levels, with and without a coin visible. Error bars are bootstrapped 95% confidence intervals. The coin has an insignificant effect at all stages of a level. as the coin (Figure 9, Appendix). Actor-Critic Inconsistency.In Section 3.1 we estab- lished that the actor (the policy) misgeneralizes, and in the previous paragraph we have shown that the critic also misgeneralizes. Here we show that the behavior of the actor and the output of the critic are in fact inconsistent: the actor navigates as far right as possible even when this involves moving past a wall, whereas the critic assigns highest value to states in which the agent is touching a wall before having moved past it. We deploy the agent in an environment with a permeable end wall. If the actor generalized consistently with respect to the critic, it should stay at the wall, or re- turn to it upon passing through it. Instead, whenever the agent reaches the end wall it continues moving right and passes through the wall100%of the time (n= 114) (Fig- ure 7). This indicates that the policy pursues a “move right” proxy objective, rather than the “move to the wall” proxy objective of the critic, or the intended “move to the coin” objective. In other words, the actor learns a “non-robust proxy of a non-robust proxy”. Its failure to match the critic’s proxy objective is another source of and example of goal misgeneralization. 3.5. Measuring Agency We validate the formal definition of goal misgeneralization from Section 2 by explicitly computing the agent and device mixtures in a gridworld environment based on work by Orseau et al. (2018), shown in Figure 8. In this environment there are4possible actions (move up, down, left, right). The state consists of two sets of (row, column) coordinates: the position of the agent and of the goal. Possible goal states include every accessible square in the gridworld; formally, Goal Misgeneralization in Deep Reinforcement Learning Time 10 8 6 4 020406080100120 Observation Value function output Figure 7.Observations and critic’s value estimate for a typical OOD episode with permeable end wall. The agent continues to move to the right, through the wall. This happens even though the critic assigns the highest value around timestep 35, when the agent is just left of the wall (where the coin is typically located during training). This phenomenon occurs100%of the time that the agent reaches the permeable wall (n=114). This demonstrates that the actor’s behavioural proxy objective differs from the critic’s proxy objective. Such differences could cause goal misgeneralization even in situations where a critic has learned the true value function. our set of possible reward functions is R=R s |s∈S, whereSis the set of accessible squares in the gridworld and R s (s ′ ) = 1ifs=s ′ and0otherwise. We generate trajectories of an agent attempting to reach a goal cellg. We distinguish four types of trajectories (Fig- ure 8); depending on the type, the goal position is either random or fixed. We distinguish capability from goal gen- eralization failure by comparing the mixture probabilities p agt (τ)andp dev (τ)(Table 1). A detailed description of the trajectory types and the computation of the mixture probabilities is available in Appendix B.1. Consider a policy that successfully solves a maze in which the locations of the start state and goal state are fixed (Fig- ure 8, top left). There are three ways this policy might generalize OOD, illustrated in Figure 8. 1.A goal misgeneralizing policy might reliably navigate to the location where the goal was during training, ignoring its actual location (Figure 8, top right). 2.A policy that fails at capability generalization might memorize the trajectory from start to goal, and behave randomly on other states (Figure 8, bottom left). 3. A robust policy would reliably solve the task for any location of goal and start state (Figure 8, bottom right). As shown in Table 1, the agents & devices formalism suc- cessfully distinguishes goal misgeneralization from capa- bility generalization failures: The robust policy as well as the misgeneralizing policy are clearly recognized as goal- directed agents, whereas the policy that fails at capability generalization is correctly classified as non-agent. In-distributionGoal Misgeneralization Capability FailRobust Agent Figure 8.Start.Goal.In-distribution:the policy reaches its (fixed) goal.Goal misgeneralization: the policy navigates to the wrong position when the goal is moved.Capability Generalization Failure:when start position is moved, the policy gets stuck.Robust: the policy always reaches the goal for all start / goal positions. Type−logp agt (τ)−logp dev (τ)p(agt|τ) IID5.720.80.9999 G. Misg.14.1300.9999 Cap. Fail72690.0674 Robust10.530.50.9999 Table 1.As expected, all trajectories from Figure 8 are classified as agents, except the capability generalization failure (Cap. Fail). Goal Misgeneralization in Deep Reinforcement Learning 4. Related Work Out-of-Distribution Generalization. Goal misgeneral- ization is a form of out-of-distribution (OOD) generalization failure. OOD generalization is the problem of performing well on a novel distribution at test time. Causes for such a train-test mismatch include: i) the training data does not characterize the true distribution (Torralba & Efros, 2011), i) the distribution shifts over time (Qui ̃ nonero-Candela et al., 2009), and i) the test data are adversarially perturbed (Szegedy et al., 2014; Goodfellow et al., 2015). OOD gen- eralization is a well established limitation of existing deep learning approaches, which can be very sensitive to changes in distribution (Recht et al., 2019; Hendrycks & Dietterich, 2019), and may base their predictions on shortcuts (Geirhos et al., 2020) or spurious correlations (Beery et al., 2018; Arjovsky et al., 2020). Such lack of robustness may be due to underspecification (D’Amour et al., 2020): since there are many patterns a deep network can learn which explain the training distribution equally well, practitioners may need to provide additional information to disambiguate between these possible solutions. Underspecification of the mapping from state to reward is present in our CoinRun and Maze environments, making goal misgeneralizationunavoidable if the inductive biases of the deep RL algorithms deployed don’t match the intended behavior. The existing work on OOD generalization is largely complementary to our work here on goal misgeneralization. For example, learning in- variant predictors (Arjovsky et al., 2020; Krueger et al., 2021) across diverse training environments might help a model learn the true cause of the reward it receives and improve goal misgeneralization. We also add to existing work on OOD generalization by highlighting that when a model fails to generalize OOD, it may do so in two different ways that have notably different consequences: it might generalize completely incapably, or it might generalize ca- pably but pursue an incorrect objective. This distinction is important because pursuing an incorrect objective can lead to different—and potentially more severe—consequences (Zhuang & Hadfield-Menell, 2021). OOD generalization is especially important in online RL because updating the policy leads to a shift in the training distribution. Generalization in RL.We define and study goal misgen- eralization in the context of reinforcement learning. His- torically, generalization in RL received little attention, but many recent works address various forms of RL generaliza- tion, including OOD generalization. Notable directions of research include sim-to-real (Peng et al., 2018), robust RL (Morimoto & Doya, 2005), and offline RL (Levine et al., 2020); see Kirk et al. (2021) for a review. Solving classic deep RL environments such as ATARI (Bellemare et al., 2013) may already require generalizing across states, but Cobbe et al. (2019) note that overfitting to a particular en- vironment is commonly observed, and propose diverse sets of environments to promote generalization. While Cobbe et al. (2019) use the same distribution of environments dur- ing training and test time, we modify their environments to create OOD test environments. Goal Misgeneralization / Objective Robustness.An earlier public version of this work used the term ‘objec- tive robustness failure’ in the place of ‘goal misgeneral- ization’. We adopt the term ‘goal misgeneralization’ from Shah et al. (2022) in order to avoid confusion with reward misspecification and to avoid having two terms for the same phenomenon. Previous work on OOD generalization has largely failed to distinguish between goal misgeneralization and capability generalization failures. Hubinger et al. (2019) and Mikulik (2019) are perhaps the first to make such a dis- tinction explicitly, and the termobjective robustness failure is used by Hubinger (2020a) to refer to the former failure mode. These works also argue that goal misgeneralization may be catastrophic, motivating our focus on this type of failure. Previously, Leike et al. (2018) used the termreward- result gapto refer to the difference between what a model was optimizedfor and what it appears to be optimizing(i.e. what we call the behavioral objective). We add to these works by formalizing the distinction between capability gen- eralization failure and goal misgeneralization, and providing the first empirical demonstrations of goal misgeneralization. Mesa-Optimization.Public non-academic discussions of concerns related to goal misgeneralization, and the anal- ogy with evolution described in Section 2.2, go back at least to 2016 (Yudkowsky, 2016; Christiano, 2016). 4 These early discussions, as well as Hubinger et al. (2019), focus on goal misgeneralization caused bymesa-optimization, a phenomenon where a model learns an optimization process (even if not explicitly trained to do so). Mesa-optimization could lead to goal misgeneralization if the learned “inner objective” optimized differs from the “outer objective” spec- ified by the designer, but this need not be the case. Fur- thermore, goal misgeneralization can occur without mesa- optimization. Thus these are in fact two distinct behav- iors, and our work does not demonstrate or address mesa- optimization. 5 Mesa-optimization could be a concern inde- pendent of goal misgeneralization if the mesa-optimizer pur- sues undesirablemeansof optimizing the correct objective (Krueger et al., 2020), e.g. we might not want a prediction system to make self-fulfilling prophecies (Armstrong, 2017). Furthermore, while we’ve defined goal misgeneralization as 4 Terms used in these discussions include “subsystem reasoning” (Taylor, 2017), “optimization daemons”, “inner optimizers”, and “inner alignment” (Rice & many authors, 2018). 5 For a sufficiently broad definition of mesa-optimization, goal misgeneralization may become equivalent to misaligned mesa- optimization. We use a different term to emphasize the connection to OOD generalization and not depend on a notion of optimization (Shah, 2021). Goal Misgeneralization in Deep Reinforcement Learning a form of OOD failure, mesa-optimization may lead to un- desirable behavior such as deception (Rice & many authors, 2018) or power-seeking (Turner et al., 2021) on-distribution. Unidentifiability in Inverse Reinforcement Learning. Goal misgeneralization tends to arise when there are multi- ple possible reward functions that are indistinguishable from the intended objective and produce similar behavior on the training set, but not OOD. This type of unidentifiability is analogous to the one encountered in inverse reinforcement learning (IRL). Amin & Singh (2016) separate the causes for this unidentifiability in IRL into two classes. The first,rep- resentational unidentifiability , arises because some trans- formations of reward functions, e.g. rescaling, preserve the relativereturns of different policies. The second,experi- mental unidentifiability , occurs whenπ’s observed behav- ior is optimal under two (or more) reward functions which are not functionally equivalent—i.e. there exist situations where they would entail different optimal behavior. Goal misgeneralization can arise from experimental unidentifia- bility when an agent only encounters situations that distin- guish its behavioral objective from the intended objective function at test time. Reward Misspecification. Reward specification is the problem of specifying a reward that captures the behav- ior we want (Amodei et al., 2016; Clark & Amodei, 2016). Goal misgeneralization is a distinct problem: it may lead to failure even if the reward function is perfectly specified. 6 Reward misspecification can produce similar failures to goal misgeneralization, however, when the designer specifies a proxy objective that yields good training performance, but fails OOD (Hadfield-Menell et al., 2017b). 5. Discussion We have formally defined the problem of goal misgeneral- ization in RL, and provided the first explicit examples of goal misgeneralization in deep RL systems. We argue that goal misgeneralization is a natural category since, much like adversarial robustness failures, goal misgeneralization has distinct causes and poses distinct problems. Our definition of goal misgeneralization via the agent and device mixtures is practically limited: it is generally hard to define a useful prior over objectives, and the computation quickly becomes intractable for large and complex environ- ments. Conceptually, the division into agents and devices is somewhat restrictive; for example, multi-agent systems do not naturally fit into the framework. 6 Failures due to reward misspecification occur when the model behaves in an unintended way that nevertheless scores highly on the reward function. In contrast, in goal misgeneralization, models scorepoorlyon the training reward because they are pursuing a different objective. Better understanding agency and optimization remains an important avenue for future work. There is a number of interesting questions in this direction, such as formalizing how some part of the world can optimize some other part of the world and thus be an agentembeddedin its envi- ronment (Demski & Garrabrant, 2019), and understanding when deep learning systems are likely to behave like agents optimizing proxy objectives. Future empirical work may also study the factors that influ- ence goal misgeneralization. For instance, what kinds of proxy objectives are agents most likely to learn? This may help us understand what kinds of environment diversity are most useful for learning robust goals. 6. Contributions JK and L independently proposed the idea of demonstrat- ing goal misgeneralization. LS suggested to use Procgen for experiments and conceived of the CoinRun demonstration; JK, L, LS, and JP set up and trained the agent on CoinRun. L and JP modified the Procgen environments, and L ran the sweeps in CoinRun and Maze. LS with assistance from L conceived and ran the experiments in section 3.4. LS ran the attribution map experiments. DK became involved after the original arXiv preprint; he proposed defining goal mis- generalization via agents and devices (Orseau et al., 2018), proposed the experiment in Figure 4, and made major contri- butions to the writing and presentation. Laurent Orseau 7 ran the ‘measuring agency’ experiment (Section 3.5), following a specification designed by L and DK. The manuscript was written by DK, JK, L, and LS. Acknowledgements Special thanks to Rohin Shah and Evan Hubinger for their guidance and feedback throughout the course of this project, and to Rohin for proposing the termgoal misgeneralization. Thanks also to Max Chiswick for assistance adapting the code for training the agents, Adam Gleave, Dan Hendrycks, Edouard Harris, Robert Kirk, and Dmitrii Krasheninnikov for helpful feedback on drafts of this paper, and the organiz- ers of the AI Safety Camp for bringing the authors of this paper together: Remmelt Ellen, Nicholas Goldowsky-Dill, Rebecca Baron, Max Chiswick, and Richard M ̈ ohn. This work was supported by funding from the AI Safety Camp and Open Philanthropy. Lee Sharkey was supported by the Centre for Effective Altruism Long Term Future Fund and by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s Excellence Strategy–EXC2064/1–390727645. 7 Laurent joined and contributed after the ICML deadline for author inclusion, which is why he is included as author here but not on the official ICML submission. Goal Misgeneralization in Deep Reinforcement Learning References Amin, K. and Singh, S. P. Towards resolving uniden- tifiability in inverse reinforcement learning.CoRR, abs/1601.06569, 2016. URLhttp://arxiv.org/ abs/1601.06569. 1, 9 Amodei, D., Olah, C., Steinhardt, J., Christiano, P. F., Schulman, J., and Man ́ e, D.Concrete problems in AI safety.CoRR, abs/1606.06565, 2016. URLhttp: //arxiv.org/abs/1606.06565. 9 Arjovsky, M.Out of Distribution Generalization in Machine Learning. PhD thesis, New York University, 2021. 1 Arjovsky, M., Bottou, L., Gulrajani, I., and Lopez-Paz, D. Invariant risk minimization, 2020. 8 Armstrong, S. Good and safe uses of AI oracles.CoRR, abs/1711.05541, 2017. URLhttp://arxiv.org/ abs/1711.05541. 8 Barnett,M.A simple environment for show- ing mesa misalignment.AI Alignment Forum, 2019.URLhttps://w.alignmentforum. org/posts/AFdRGfYDWQqmkdhFq. 5 Beery, S., Van Horn, G., and Perona, P. Recognition in terra incognita. InProceedings of the European Conference on Computer Vision (ECCV), p. 456–473, 2018. 8 Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M. The arcade learning environment: An evaluation platform for general agents.Journal of Artificial Intelligence Re- search, 47:253–279, Jun 2013. ISSN 1076-9757. doi: 10.1613/jair.3912. URLhttp://dx.doi.org/10. 1613/jair.3912. 8 Bostrom, N.Superintelligence: Paths, Dangers, Strategies. Oxford University Press, Inc., 2014. ISBN 0199678111. 2 Christiano, P. F. What does the universal prior actually look like?, Nov 2016. URLhttps://tinyurl.com/ uniprior. 8 Clark, J. and Amodei, D. Faulty reward functions in the wild. OpenAI Blog, 2016. URLhttps://openai. com/blog/faulty-reward-functions/. 9 Cobbe, K., Hesse, C., Hilton, J., and Schulman, J. Lever- aging procedural generation to benchmark reinforcement learning.arXiv preprint arXiv:1912.01588, 2019. 2, 3, 8 D’Amour, A., Heller, K., Moldovan, D., Adlam, B., Ali- panahi, B., Beutel, A., Chen, C., Deaton, J., Eisenstein, J., Hoffman, M. D., Hormozdiari, F., Houlsby, N., Hou, S., Jerfel, G., Karthikesalingam, A., Lucic, M., Ma, Y., McLean, C., Mincu, D., Mitani, A., Montanari, A., Nado, Z., Natarajan, V., Nielson, C., Osborne, T. F., Raman, R., Ramasamy, K., Sayres, R., Schrouff, J., Seneviratne, M., Sequeira, S., Suresh, H., Veitch, V., Vladymyrov, M., Wang, X., Webster, K., Yadlowsky, S., Yun, T., Zhai, X., and Sculley, D. Underspecification presents challenges for credibility in modern machine learning, 2020. 8 Demski, A. and Garrabrant, S. Embedded agency.arXiv preprint arXiv:1902.09469, 2019. 9 Espeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih, V., Ward, T., Doron, Y., Firoiu, V., Harley, T., Dunning, I., Legg, S., and Kavukcuoglu, K. Impala: Scalable dis- tributed deep-rl with importance weighted actor-learner architectures.arXiv preprint arXiv:1802.01561, 2018. 13 Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., and Wichmann, F. A. Short- cut learning in deep neural networks.Nature Ma- chine Intelligence, 2(11):665–673, November 2020. doi: 10.1038/s42256-020-00257-z. URLhttps://doi. org/10.1038/s42256-020-00257-z. 3, 8 Goodfellow, I. J., Shlens, J., and Szegedy, C. Explaining and harnessing adversarial examples, 2015. 8 Hadfield-Menell, D., Dragan, A., Abbeel, P., and Russell, S. The off-switch game. InWorkshops at the Thirty-First AAAI Conference on Artificial Intelligence, 2017a. 2 Hadfield-Menell, D., Milli, S., Abbeel, P., Russell, S., and Dragan, A. Inverse reward design, 2017b. 9 Hendrycks, D. and Dietterich, T. Benchmarking neural network robustness to common corruptions and perturba- tions. InInternational Conference on Learning Represen- tations, 2019. 8 Hendrycks, D., Carlini, N., Schulman, J., and Steinhardt, J. Unsolved problems in ml safety, 2021. 2 Hilton, J., Cammarata, N., Carter, S., Goh, G., and Olah, C. Understanding rl vision.Distill, 2020. doi: 10.23915/ distill.00029. https://distill.pub/2020/understanding-rl- vision. 15 Hubinger, E.Clarifying inner alignment termi- nology.AI Alignment Forum, 2020a.URL https://w.alignmentforum.org/posts/ SzecSPYxqRa5GCaSF. 8 Hubinger, E.Towards an empirical investigation of inner alignment. AI Alignment Forum, 2020b. URL https://w.alignmentforum.org/posts/ 2GycxikGnepJbxfHT. 5 Hubinger, E., van Merwijk, C., Mikulik, V., Skalse, J., and Garrabrant, S. Risks from learned optimization Goal Misgeneralization in Deep Reinforcement Learning in advanced machine learning systems.arXiv preprint arXiv:1906.01820, 2019. 2, 8 Ibarz, B., Leike, J., Pohlen, T., Irving, G., Legg, S., and Amodei, D. Reward learning from human preferences and demonstrations in atari, 2018. 2 Ilyas, A., Santurkar, S., Tsipras, D., Engstrom, L., Tran, B., and Madry, A. Adversarial examples are not bugs, they are features, 2019. 1 Kingma, D. P. and Ba, J. Adam: A method for stochas- tic optimization. In Bengio, Y. and LeCun, Y. (eds.), 3rd International Conference on Learning Represen- tations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URLhttp: //arxiv.org/abs/1412.6980. 13 Kirk, R., Zhang, A., Grefenstette, E., and Rockt ̈ aschel, T. A survey of generalisation in deep reinforcement learning. arXiv preprint arXiv:2111.09794, 2021. 2, 8 Krueger, D., Maharaj, T., and Leike, J.Hidden in- centives for auto-induced distributional shift.CoRR, abs/2009.09153, 2020. URLhttps://arxiv.org/ abs/2009.09153. 8 Krueger, D., Caballero, E., Jacobsen, J.-H., Zhang, A., Bi- nas, J., Zhang, D., Priol, R. L., and Courville, A. Out- of-distribution generalization via risk extrapolation (rex), 2021. 8 Kruskal, J. B.On the shortest spanning subtree of a graph and the traveling salesman problem.Pro- ceedings of the American Mathematical Society, 7(1):48–50, 1956.ISSN 0002-9939, 1088-6826. doi:10.1090/S0002-9939-1956-0078686-7.URL https://w.ams.org/proc/1956-007-01/ S0002-9939-1956-0078686-7/. 5 Lee, H.Training procgen environment with py- torch.https://github.com/joonleesky/ train-procgen-pytorch, 2020. 13 Leike, J., Krueger, D., Everitt, T., Martic, M., Maini, V., and Legg, S. Scalable agent alignment via reward modeling: a research direction.CoRR, abs/1811.07871, 2018. URL http://arxiv.org/abs/1811.07871. 8 Levine, S., Kumar, A., Tucker, G., and Fu, J. Offline rein- forcement learning: Tutorial, review, and perspectives on open problems, 2020. 8 Mikulik, V.2d robustness.AI Alignment Forum, 2019.URLhttps://w.alignmentforum. org/posts/2mhFMgtAjFJesaSYR. 8 Morimoto, J. and Doya, K. Robust reinforcement learning. Neural computation, 17(2):335–359, 2005. 8 Omohundro, S. The basic ai drives. volume 171, p. 483– 492, 01 2008. 2 Orseau, L., McGill, S. M., and Legg, S.Agents and devices: A relative definition of agency.CoRR, abs/1805.12387, 2018. URLhttp://arxiv.org/ abs/1805.12387. 2, 6, 9, 13 Pan, A., Bhatia, K., and Steinhardt, J. The effects of reward misspecification: Mapping and mitigating misaligned models, 2022. 2 Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., K ̈ opf, A., Yang, E., DeVito, Z., Rai- son, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. Pytorch: An imperative style, high-performance deep learning library, 2019. 13 Peng, X. B., Andrychowicz, M., Zaremba, W., and Abbeel, P. Sim-to-real transfer of robotic control with dynam- ics randomization.2018 IEEE International Conference on Robotics and Automation (ICRA), May 2018. doi: 10.1109/icra.2018.8460528. URLhttp://dx.doi. org/10.1109/ICRA.2018.8460528. 8 Qui ̃ nonero-Candela, J., Sugiyama, M., Lawrence, N. D., and Schwaighofer, A.Dataset shift in machine learning. Mit Press, 2009. 8 Recht, B., Roelofs, R., Schmidt, L., and Shankar, V. Do ImageNet classifiers generalize to ImageNet?In Chaudhuri, K. and Salakhutdinov, R. (eds.),Proceed- ings of the 36th International Conference on Machine Learning, volume 97 ofProceedings of Machine Learn- ing Research, p. 5389–5400. PMLR, 09–15 Jun 2019. URLhttps://proceedings.mlr.press/v97/ recht19a.html. 8 Rice, I. and many authors.Mesa-optimization, Feb 2018. URLhttps://w.lesswrong.com/tag/ mesa-optimization. 8, 9 Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. 6, 13 Shah, R. Generalization ¿ utility. Stanford Existential Risk Initiative, 2021. URLhttps://w.youtube. com/watch?v=Mn8RKySnOfo. 8 Shah, R., Varma, V., Kumar, R., Phuong, M., Krakovna, V., Uesato, J., and Kenton, Z. Goal misgeneralization: Why correct specifications aren’t enough for correct goals. Forthcoming, 2022. 1, 8 Goal Misgeneralization in Deep Reinforcement Learning Simonyan, K., Vedaldi, A., and Zisserman, A. Deep in- side convolutional networks: Visualising image clas- sification models and saliency maps.arXiv preprint arXiv:1312.6034, 2013. 6, 15 Singh, S., Lewis, R. L., Barto, A. G., and Sorg, J. Intrinsi- cally motivated reinforcement learning: An evolutionary perspective.IEEE Transactions on Autonomous Mental Development, 2(2):70–82, 2010. 3 Soares, N., Fallenstein, B., Armstrong, S., and Yudkowsky, E. Corrigibility. InAAAI Workshop: AI and Ethics, 2015. 2 Sutton, R., Barto, R., Barto, A., Barto, C., Bach, F., and Press, M.Reinforcement Learning: An Introduc- tion.A Bradford book. MIT Press, 1998.ISBN 9780262193986. URLhttps://books.google. de/books?id=CAFR6IBF4xYC. 6 Sutton, R. S. and Barto, A. G.Reinforcement learning: An introduction. MIT press, 2018. 2 Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Er- han, D., Goodfellow, I., and Fergus, R.Intriguing properties of neural networks. InInternational Confer- ence on Learning Representations, 2014. URLhttp: //arxiv.org/abs/1312.6199. 8 Taylor, J. On motivations for miri’s highly reliable agent design research, Jan 2017. URLhttps://tinyurl. com/mirimotiv. 8 Torralba, A. and Efros, A. A. Unbiased look at dataset bias. InCVPR 2011, p. 1521–1528. IEEE, 2011. 8 Turner, A., Smith, L., Shah, R., Critch, A., and Tadepalli, P. Optimal policies tend to seek power. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.),Advances in Neural Information Processing Systems, volume 34, p. 23063–23074. Curran Asso- ciates, Inc., 2021. URLhttps://proceedings. neurips.c/paper/2021/file/ c26820b8a4c1b3c2a868d6d57e14a79-Paper. pdf. 2, 9 Valle-P ́ erez, G., Camargo, C. Q., and Louis, A. A. Deep learning generalizes because the parameter-function map is biased towards simple functions.arXiv preprint arXiv:1805.08522, 2019. 3 Yudkowsky, E. Optimization daemons, Mar 2016. URL https://arbital.com/p/daemons/. 8 Zhuang, S. and Hadfield-Menell, D. Consequences of mis- aligned ai.arXiv preprint arXiv:2102.03896, 2021. 1, 2, 8 Ziebart, B. D., Maas, A. L., Bagnell, J. A., Dey, A. K., et al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, p. 1433–1438. Chicago, IL, USA, 2008. 2 Goal Misgeneralization in Deep Reinforcement Learning A. Implementation details For all environments, we use an Actor-Critic architecture using Proximal Policy Optimization (PPO) (Schulman et al., 2017). The architecture is based on the architecture used in (Espeholt et al., 2018) but omits the recurrent components of the original network. Both the actor (policy function) and critic (value function) are implemented by feedforward neural networks on top of a shared residual convolutional network. All models are implemented in PyTorch (Paszke et al., 2019) and our implementations are based on a codebase by Lee (2020). Unless otherwise stated, models are trained on100k procedurally generated levels for 200M timesteps. We use the Adam optimizer (Kingma & Ba, 2015) in all experiments. Each training run required approximately 30 GPU hours of compute on a V100. B. Experiment Details B.1. Measuring Agency B.1.1. GENERATINGTRAJECTORIES Fix positionssandg(‘start’ and ‘goal’) in a 20x20 gridworld. Then we generate trajectoriesτ 1 ,...,τ n as follows. For every trajectoryτ i , we sample random positionss (i) rand andg (i) rand . Note thats (i) rand andg (i) rand take on new values for every trajectory, whilesandgare fixed. For every trajectory, we also identify one of the gridworld states as theintended goalg (i) true . We then generate four types of trajectoriesτ 1 i ,τ 2 i ,τ 3 i ,τ 4 i : 1. Setg (i) true =g. Pick the trajectory that takes the shortest path froms (i) rand tog (i) true (‘In-distribution’). 2. Setg (i) true =g (i) rand . Pick the trajectory that takes the shortest path froms (i) rand tog(‘Goal Misgeneralization’), 3. Setg (i) true =g. Pick the trajectory that starts ats (i) rand and moves in a uniformly random direction every step, for50 timesteps. If the trajectory ever crosses the shortest path fromstog, then it follows that path tog(‘Capability Failure’). 4. Setg (i) true =g (i) rand . Pick the trajectory that takes the shortest path froms (i) rand tog (i) true (‘Robust agent’). We are left with4ntrajectories(τ λ i ) i≤n,λ≤4 . Note: 1. The trajectoriesτ 1 i are generated from a policy that can reach the fixed goal stateg (i) true =gfrom any place on the grid. 2.The trajectoriesτ 2 i are generated from the same policy, deployed in an environment where the goal stateg (i) true is changed. The policy still navigates to the fixed positiong, but this is no longer the correct goal; this behavior is designed to match the behavior we saw in the policies we trained for the Maze experiments in Section 3.2. 3. The trajectoriesτ 3 i are designed to imitate the capability generalization failure of a policy which navigates from a fixed start state to a fixed end state. When initialized to a random start location, the policy takes random actions since it only knows to navigate along a fixed path. 4.The trajectoriesτ 4 i are generated from a policy that robustly takes the shortest path tog (i) true from any position in the gridworld even when the goal state is randomized. B.1.2. CALCULATINGMIXTUREPROBABILITIES We follow the method in Orseau et al. (2018). The observations available to agent policies include the goal state and the position of the agent. Agent prior.We specify the set of possible goal states to consist of alln 2 locations in the gridworld. (Those familiar with Orseau et al. (2018) should note that we do not use the switching prior). Goal Misgeneralization in Deep Reinforcement Learning Agent mixture.We specify the set of goals to consist of all accessible squares in the gridworld, plus the (variable) goal g (i) true . Note thatg true can be random in the cases where we setg (i) true =g (i) rand ), and thus vary from trajectory to trajectory. Formally, our set of objectives is R=R s |s∈S∪g true , whereSis the set of accessible squares in the gridworld andR s (s ′ ) = 1ifs=s ′ and0otherwise. We then take a uniform priorη agt (R) = 1/|R|over this set. Given an objectiveR, define the probabilityp ε (τ|R)of a trajectory as induced by an ε-greedy policy. Here, the observations of the policy consist of the (row, column) position of the agent. We then integrate overε: p agt (τ|R) = ∫ 1 0 p ε (τ|R) dε. Device mixture. Recall that a device is just a stochastic, tabular policy that takes in an observation and outputs an action. The observation consists of the type of cell (empty, wall, start, goal) that the device is facing, in the direction of its last action. Our device priorη dev is uniform over the space of policies. Setp ε (τ|d)to be the probability of a trajectory generated by acting in anε-deterministic way with respect tod, that is take the action determined bydwith probability1−εand a random action otherwise. Just as previously we integrate overεin[0,1]to compute the final likelihoodp dev (τ|d). C. Hyperparameters Table 2.Hyperparameters HyperparameterValue ENV. DISTRIBUTION MODEHARD γ.999 λ.95 LEARNING RATE0.0005 # TIMESTEPS PER ROLLOUT256 EPOCHS PER ROLLOUT3 # MINIBATCHES PER EPOCH8 MINIBATCH SIZE2048 ENTROPY BONUS(k H ).01 PPO CLIP RANGE.2 REWARD NORMALIZATION?YES LEARNING RATE5×10 −4 # WORKERS4 # ENVIRONMENTS PER WORKER64 TOTAL TIMESTEPS200M ARCHITECTUREImpala LSTM?No FRAME STACK?No D. Value attribution maps and other figures Goal Misgeneralization in Deep Reinforcement Learning TestTrain Observation Value attribution Observation Value attribution Figure 9.Attribution maps of the agent’s observation with respect to its value function output. Maps were generated by taking the gradient of the value function output with respect to the observation pixels (averaged over channels) (Simonyan et al., 2013). Red shading indicates pixels that negatively influence value function output and blue shading indicates pixels that have positive influence. The pixel level attributions were standardized by dividing each map by the value of the largest absolute magnitude of pixel attribution. The attribution maps are passed through a Gaussian blur transform with kernel size 5 andσ= 5. As observed in Hilton et al. (2020), we find that the sign of the attribution map is often difficult to understand - for instance, buzzsaws might sometimes appear to have positive attribution rather than negative. We therefore focus on the absolute magnitude of the attribution. In both the training and test environment, the agent’s value function assigns large attribution to the end wall and occasionally the coin, enemies, and buzzsaws. From the attribution plots alone, we can only determine that the end wall appears more important to the agent than the coin, but the coin might nevertheless also be somewhat important for the value function output. Goal Misgeneralization in Deep Reinforcement Learning 0.000.250.500.751.001.251.50 Timesteps 1e8 1 2 Training reward (OOD) test reward Figure 10.Average return during training of the Keys & Chests agent. The reward on the ‘many keys’ test environment is much lower than the ‘many chests’ training reward. Figure 11.Expected return during training of the CoinRun agent. Figure 12.Expected return during training of the maze agent.

Instant research discovery