Paper deep dive
Hidden Incentives for Auto-Induced Distributional Shift
David Krueger, Tegan Maharaj, Jan Leike
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 93%
Last extracted: 3/12/2026, 7:52:03 PM
Summary
The paper introduces 'auto-induced distributional shift' (ADS) and 'hidden incentives for auto-induced distributional shift' (HI-ADS), where machine learning algorithms, particularly those using meta-learning, inadvertently learn to manipulate their own input distributions to improve performance. The authors propose 'unit tests' to detect these hidden incentives and a 'context swapping' mitigation strategy, demonstrating that meta-learning algorithms like PBT often reveal these incentives, leading to undesirable behaviors in content recommendation and other environments.
Entities (5)
Relation Signals (3)
Auto-induced Distributional Shift â causes â Performance Gain
confidence 90% · strong meta-learners achieve gains in performance via ADS.
Meta-learning â reveals â Hidden Incentives for Auto-induced Distributional Shift
confidence 90% · changes to the learning algorithm, such as the introduction of meta-learning, can cause hidden incentives for auto-induced distributional shift (HI-ADS) to be revealed.
Context Swapping â mitigates â Hidden Incentives for Auto-induced Distributional Shift
confidence 85% · This effectively mitigates HI-ADS in our unit test environments
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Decisions made by machine learning systems have increasing influence on the world, yet it is common for machine learning algorithms to assume that no such influence exists. An example is the use of the i.i.d. assumption in content recommendation. In fact, the (choice of) content displayed can change users' perceptions and preferences, or even drive them away, causing a shift in the distribution of users. We introduce the term auto-induced distributional shift (ADS) to describe the phenomenon of an algorithm causing a change in the distribution of its own inputs. Our goal is to ensure that machine learning systems do not leverage ADS to increase performance when doing so could be undesirable. We demonstrate that changes to the learning algorithm, such as the introduction of meta-learning, can cause hidden incentives for auto-induced distributional shift (HI-ADS) to be revealed. To address this issue, we introduce `unit tests' and a mitigation strategy for HI-ADS, as well as a toy environment for modelling real-world issues with HI-ADS in content recommendation, where we demonstrate that strong meta-learners achieve gains in performance via ADS. We show meta-learning and Q-learning both sometimes fail unit tests, but pass when using our mitigation strategy.
Tags
Links
- Source: https://arxiv.org/abs/2009.09153
- Canonical: https://arxiv.org/abs/2009.09153
Trouble viewing inline? Open PDF directly â
Full Text
68,094 characters extracted from source content.
Expand or collapse full text
Hidden Incentives for Auto-induced Distributional Shift David Scott Krueger 1 2 3 Tegan Maharaj 1 4 Jan Leike 3 Abstract Decisions made by machine learning systems have increasing influence on the world, yet it is common for machine learning algorithms to as- sume that no such influence exists. An example is the use of the i.i.d. assumption in content rec- ommendation. In fact, the (choice of) content displayed can change usersâ perceptions and pref- erences, or even drive them away, causing a shift in the distribution of users. We introduce the term auto-induced distributional shift (ADS)to de- scribe the phenomenon of an algorithmcausinga change in the distribution of its own inputs. Our goal is to ensure that machine learning systems do not leverage ADS to increase performance when doing so could be undesirable. We demonstrate that changes to the learning algorithm, such as the introduction of meta-learning, can causehid- den incentives for auto-induced distributional shift (HI-ADS)to be revealed. To address this issue, we introduce âunit testsâ and a mitigation strategy for HI-ADS, as well as a toy environment for modelling real-world issues with HI-ADS in content recommendation, where we demonstrate that strong meta-learners achieve gains in per- formance via ADS. We show meta-learning and Q-learning both sometimes fail unit tests, but pass when using our mitigation strategy. 1. Introduction Consider a content recommendation system whose perfor- mance is measured by accuracy of predicting what users will click. This system can achieve better performance by either 1) making better predictions, or 2) changing the distri- bution of users such that predictions are easier to make. We propose the termauto-induced distributional shift (ADS) to describe this latter kind of distributional shift, caused by the algorithmâs own predictions or behaviour (Figure 1). 1 MontrĂ©al Institute for Learning Algorithms, Canada 2 UniversitĂ© de MontrĂ©al, Canada 3 work begun while author was at DeepMind 4 Polytechnique MontrĂ©al, Canada. Correspondence to: David Krueger <davidscottkrueger@gmail.com>. ADS are not inherently bad, and are sometimes even desir- able. But they can cause problems if they occur unexpect- edly. It is typical in machine learning (ML) to assume (e.g. via the i.i.d. assumption) that (2) will not happen. However, given the increasing real-world use of ML algorithms, we believe it is important to model and experimentally observe what happens when assumptions like this are violated. This is the motivation of our work. Figure 1.Distributions of users over time.Left: A distribution which remains constant over time, following the i.i.d assump- tion.Right: Auto-induced Distributional Shift (ADS) results in a change in the distribution of users in our content recommendation environment. (see Section 7.3 for details). In many cases, including news recommendation, we would consider (2) a form ofcheatingâthe algorithm changed the task rather than solving it as intended. We care which meansthe algorithm used to solve the problem (e.g. (1) and/or (2)), but we only told it about theends, so it didnât know not to âcheatâ. This is an example of aspecification problem (Leike et al., 2017; Ortega et al., 2018): a problem which arises from a discrepancy between the performance metric (maximize accuracy) and âwhat we really meantâ: to maximize accuracy via (1), which is difficult to encode as a performance metric. Ideally, weâd like to quantify the desirability of all possible means, e.g. assign appropriate rewards to all potential strate- gies and âside-effectsâ, but this is intractable for real-world settings. Using human feedback to learn reward functions which account for such impacts is a promising approach to specifying desired behavior (Leike et al., 2018; Christiano et al., 2017). But the same issue can arise whenever human feedback is used in training: one means of improving per- formance could be to alter human preferences, making them easier to satisfy.Thus in this work, we pursue a complemen- tary approach: managing learnersâincentives. arXiv:2009.09153v1 [cs.LG] 19 Sep 2020 Hidden Incentives for Auto-induced Distributional Shift A learner has anincentiveto behave in a certain way when doing so can increase performance (e.g. accuracy or reward). Informally, we say an incentive ishiddenwhen the learner behaves as if it were not present. But we note that changes to the learning algorithm or training regime could cause previously hidden incentives to be revealed, resulting in un- expected and potentially undesirable behaviour. Managing incentives (e.g. controlling which incentives are hidden/ revealed) can allow algorithm designers to disincentivize broad classes of strategies (such as any that rely on ma- nipulating human preferences) without knowing their exact instantiation. 1 Our goal in this work is to provide insight and practical tools for understanding and managing learnersâ incentives, specificallyhidden incentives for auto-induced distribu- tional shift :HI-ADS. To study the conditions which cause HI-ADS to be revealed, we present unit tests for detecting HI-ADS in supervised learning (SL) and in reinforcement learning (RL). We also create an environment which models ADS in news recommendation, to illustrate the potential effects of revealing HI-ADS in this setting. The unit tests both have two means by which the learner can improve performance: one which creates ADS and one which does not. The intended method of improving performance is one that doesnotinduce ADS; the other is âhiddenâ and we want it to remain hidden. A learner "fails" the unit test if it nonetheless pursues the incentive to increase performance via ADS. The SL unit test provides an illustrative example. It is a prediction problem with two targets, mean-zero Gaussians. The intended means of improving performance is to make good predictions (i.e. predict0,0). However we create an incentive for ADS: a prediction of >0.5 for the first target will reduce the variance of the second target, reducing future loss. A learner fails the unit test to the extent it predicts >0.5 for the first target. In both the RL and SL unit tests, we find that âvanillaâ learning algorithms (e.g. minibatch SGD) pass the test, but introducing an outer-loop of meta-learning (e.g. Population- Based Training (PBT) (Jaderberg et al., 2017)) can lead to high levels of failure. We find results consistent with our unit tests in the content recommendation environment: recommenders trained with PBT create earlier, faster, and larger drift in user interests, and for the same level of per- formance, create larger changes in the user base. These results suggest that failure of our unit tests indicates that an 1 Note removing or hiding an incentive for a behavior is differ- ent from prohibiting that behavior, which may still occur inciden- tally. In particular, not having a (revealed) incentive for behaviors that change a humanâs preferences, isnotthe same ashavinga (revealed) incentive for behaviors thatpreservea humanâs pref- erences. The first is often preferable; we donât want to prevent changes in human preferences that occur ânaturallyâ, e.g. as a result of good arguments or evidence. algorithm is prone to revealing HI-ADS in other settings. Finally, we propose and test a mitigation strategy we call context swapping. The strategy consists of rotating learn- ers through different environments throughout learning, so that they canât see the results or correlations of their actions in one environment over longer time horizons. This effec- tively mitigates HI-ADS in our unit test environments, but did not work well in content recommendation experiments. 2. Background 2.1. Meta-learning and population based training Meta-learning is the use of machine learning techniques to learn machine learning algorithms. This involves instan- tiating multiple learning scenarios which run in aninner loop (IL), while anouter loop (OL)uses the outcomes of the inner loop(s) as data-points from which to learn which learning algorithms are most effective (Metz et al., 2019). The number of IL steps per OL step is called theinterval. Many recent works focus on multi-task meta-learning, where the OL seeks to find learning rules that generalize to unseen tasks by training the IL on a distribution of tasks (Finn et al., 2017; Ren et al., 2018; Andrychowicz et al., 2016). Single-task meta-learning includes learning an op- timizer for a single task (Gong et al., 2018), and adaptive methods for selecting models (Kalousis, 2000) or setting hyperparameters (Snoek et al., 2012). For simplicity in this initial study we focus on single-task meta-learning. Population-based training (PBT; Jaderberg et al., 2017) is a meta-learning algorithm that trains multiple learners L 1 ,...,L n in parallel, after each interval (Tsteps of IL) applying an evolutionary OL step which consists of: (1) Evaluate the performance of each learner, (2) Replace both parameters and hyperparameters of 20% lowest-performing learners with copies of those from the 20% high-performing learners (EXPLOIT). (3) Randomly perturb the hyperparam- eters (but not the parameters) of all learners (EXPLORE). Two distinctive features of PBT are notable because they give the OL more control than most meta-learning algo- rithms (e.g. Bayesian optimization (Snoek et al., 2012)) over the dynamics and outcome of the learning process: (1) OL applies optimization to parameters, not just hyperparam- eters. This means the OL can directly select for parameters which lead to ADS, instead of only being able to influence parameter values via hyperparameters(2) Multiple OL steps per training run. 2.2. Distributional shift and content recommendation In general,distributional shiftrefers to change of the data distribution over time. In supervised learning with data xand labelsy, this can be more specifically described as Hidden Incentives for Auto-induced Distributional Shift dataset shift: change in the joint distribution ofP(x,y)be- tween the training and test sets (Moreno-Torres et al., 2012; Quionero-Candela et al., 2009). As identified by Moreno- Torres et al. (2012), two common kinds of shift are: (1) Covariate shift: changingP(x). In the example of content recommendation, this corresponds to changing the user base of the recommendation system. For instance, a media outlet which publishes inflammatory content may appeal to users with extreme views while alienating more moderate users. This self-selection effect (Kayhan, 2015) may appear to a recommendation system as an increase in performance, lead- ing to a feedback effect, as previously noted by Shah et al. (2018). This type of feedback effect has been identified as contributing to filter bubbles and radicalization (Pariser, 2011; Kayhan, 2015). (2)Concept shift: changingP(y|x). In the example of content recommendation, this corresponds to changing a given userâs interest in different kinds of con- tent. For example, exposure to a fake news story has been shown to increase the perceived accuracy of (and thus pre- sumably future interest in) the content, an example of the illusory truth effect (Pennycook et al., 2019). For further de- tails on these and other effects in content recommendation, see Appendix 1. 3. Auto-induced Distribution Shift (ADS) Auto-induced distribution shift (ADS) isdistributional shift caused by an algorithmâs behaviour.This is in contrast to distributional shift which would happen even if the learner were not present - e.g. for a crash prediction algorithm trained on data from the summer, encountering snowy roads is an example of distributional shift, but notauto-induced distributional shift (ADS). We emphasize that ADS are not inherently bad or good; of- ten ADS can even be desirable: consider an algorithm meant to alert drivers of imminent collisions. If it works well, such a system will help drivers avoid crashing, thus making self- refuting predictions which result in ADS. What separates desirable and undesirable ADS? The collision-alert system alters its data distribution in a way that isalignedwith the goal of fewer collisions, whereas the news manipulation results in changes that aremisalignedwith the goal of better predicting existing usersâ interests (Leike et al., 2018). In reinforcement learning (RL), ADS are typicallyencour- agedas a means to increase performance. On the other hand, in supervised learning (SL), the i.i.d. assumption precludes ADS in theory. In practice, however, the possibility of using ADS to increase performance (and thus an incentive to do so) often remains. For instance, this occurs in online learn- ing. In our experiments, we explicitly model such situations where i.i.d. assumptions are violated: We study the behavior of SL and myopic RL algorithms, in environments designed to include incentives for ADS, in order to understand when incentives are effectively hidden. Fig. 2 contrasts these settings with typical RL and SL. a 1 a 2 s 1 s 2 r 1 r 2 (a) RL: Incentives for ADS arepresentand pursuing them isdesirable a 1 a 2 s 1 s 2 r 1 r 2 (b) Myopic RL: Incentives for ADS arepresentand pur- suing them isundesirable Ëy 1 Ëy 2 x 1 x 2 ` 1 ` 2 (c) SL with i.i.d. data: Incen- tives for ADS areabsent Ëy 1 Ëy 2 x 1 x 2 ` 1 ` 2 (d) SL with ADS: Incentives for ADS arepresentand pur- suing them isundesirable Figure 2.The widely studied problems of reinforcement learning (RL) with states, actions, rewardstuples, and i.i.d. supervised learning (SL) with inputsx, predictionsËyand lossl(a,c) are free from incentive problems. We focus on cases where there are incentivespresentwhich the learner is not meant to pursue (b,d). Lines show paths of influence. The learner may have incentives to influence any nodes descending from its action,A, or prediction, Ëy. Which incentives are undesirable (orange) or desirable (cyan) for the learner to pursue is context-dependent. 4. Incentives For our study of incentives, we use the following terminol- ogy: anincentivefor a behavior (e.g. an action, a classifi- cation, etc.) ispresent(notabsent) to the extent that the behaviour will increase performance (e.g. reward, accuracy, etc.) (Everitt & Hutter, 2019). This incentive isrevealed to (nothiddenfrom) a learner if it would, at higher than chance levels, learn to perform the behavior given sufficient capacity and training experience. The incentive ispursued (noteschewed) by a learner if it actually performs the incen- tivized behaviour. Note even when an incentive is revealed, it may not be pursued, e.g. due to limited capacity and/or data, or simply chance. See Fig 3. For example, in content recommendation, the incentive to drive users away ispresentif some user types are easier to predict than others. But this incentive may behiddenfrom the learner by using a myopic algorithm, e.g. one that does not see the effects of its actions on the distribution of users. The incentive might instead berevealedto the outer loop of a meta-learning algorithm like PBT, which does see the Hidden Incentives for Auto-induced Distributional Shift Figure 3.Types of incentives, and their relationship to ADS. effects of learnerâs actions. Even when this incentive is revealed, however, it might not end up beingpursued. For example, this could happen if predicting which recommendations will drive away users is too difficult a learning problem, or if the incentive to do so is dominated by other incentives (e.g. change individual usersâ interests, or improve accuracy of predictions). In general, it may be difficult to determine empirically which incentives are revealed, because failure to pursue an incentive can be due to limited capacity, insufficient training, and/or random chance. To address this challenge, we devise extremely sim- ple environments (âunit testsâ), where we can be confident that revealed incentiveswillbe pursued. 5. Hidden Incentives for Auto-induced Distributional shift (HI-ADS) Following from the definitions in Sections 3 and 4, HI- ADS areincentives for behaviors that cause Auto-induced Distributional Shift that are hidden from the learner, i.e. the learner would not learn to perform the incentivized behaviors at higher than chance levels, even given infinite capacity and training experience. Like ADS, HI-ADS are not necessarily problematic. Indeed, hiding incentives can be an effective method of influencing learner behavior. For example, hiding the incentive to ma- nipulate users from a content recommendation algorithm could prevent it from influencing users in a way they would not endorse. However, if machine learning practitioners are not aware that incentives are present, or that properties of the learning algorithm are hiding them, then seemingly innocu- ous changes to the learning algorithm may reveal HI-ADS, and lead to significant unexpected changes in behavior. Hiding incentives for ADS may seem counter-intuitive and counter-productive in the context of reinforcement learning (RL), where moving towards high-reward states is typically desirable. However, for real-world applications of RL, the ultimate goal isnota system that achieves high reward, but rather one that behaves according to the designerâs inten- tions. And as we discussed in the introduction, it can be intractable to design reward functions that perfectly specify intended behavior. Thus managing learners incentives can still provide a useful tool for specification. We have several reasons for focusing on HI-ADS: (1) The issue of HI-ADS has not yet been identified, and thus is likely to be neglected (at least sometimes) in practice. Our âunit testsâ are the first published empirical methodology for assessing whether incentives are hidden or revealed by different learning algorithms. (2) Machine learning algo- rithms are commonly deployed in settings where ADS are present, violating assumptions used to analyze their proper- ties theoretically. This means learners could exploit ADS in unexpected and undesirable ways if incentives for ADS are not hidden. Hiding these incentives heuristically (e.g. via off-line training) is a common approach, but potentially brittle (if practitioners donât understand how HI-ADS could become revealed). In particular, meta-learning algorithms can reveal HI-ADS, and are increasingly popular. (3) Sub- stantial real-world issues could result from improper man- agement of learnerâs incentives. Examples include tamper- ing with human-generated reward signals (Everitt & Hutter, 2018) (e.g. selecting news articles which manipulate user interests), and creating âself-fulfilling propheciesâ (e.g. driv- ing up the value of a held asset by publicly predicting its value will increase (Armstrong & OâRorke, 2017)). 6. Removing HI-ADS via Context Swapping We propose a technique calledcontext swappingfor remov- ing incentives for ADS revealed by changes to the learn- ing algorithm (e.g. introducing meta-learning). The tech- nique trainsNlearners in parallel, and shuffles the learners throughNdifferent copies of the same (or similar) environ- ments;whichcopy a given learner inhabits can change at any (or every) time-step. We use a deterministic permutation of learners in environment copies, so that thei-th learner inhab- its thej-th environment on time-stepstwherej= (i+t) modN, makes an observation, takes an action, and receives a reward before moving to the next environment. WhenNis larger than the interval of the OL optimizer, each learner inhabits each copy for at most a single time-step before an OL step is applied. Under the assumption that different copies of the environment do not influence each other, this technique can address HI-ADS in practice, as we show in Sec. 7.2.1. 7. Experiments In Sections 7.2 and 7.1, we introduce âunit testsâ for HI- ADS. Our primary goal with these experiments is to convey a crisp understanding of potential issues caused by reveal- ing HI-ADS. Put simply, our experiments show that you can have a learner which behaves as intended, and just by using meta-learning (e.g. PBT),without changing the perfor- mance metric(e.g. loss or rewards), the learnerâs behavior Hidden Incentives for Auto-induced Distributional Shift Figure 4.(a) No context swapping (b) Context swapping. The proposed technique rotates learners through different environments. This removes the incentive for a learner toâinvestâ in a given environment, since it will be swapped out of that context later and not be able to reap the benefits of its investment. can change completely. We also show that context swapping is an effective mitiga- tion technique in these environments. On the practical side, the unit tests can be used to compare learning algorithms and diagnose their propensity to reveal incentives. In Section 7.3, we model a content recommendation sys- tem. The goal of these experiments is to demonstrate how HI-ADS could create issues for real-world content recom- mendation systems such as news feeds, search results, or automated suggestions. They also validate the usefulness of the unit tests: algorithms failed the unit tests also re- veal HI-ADS in this setting. We emphasize that ADS takes place in this environmentby construction. The point of our experiments is that meta-learning canincreasethe rate and/or extent of ADS, byrevealingthis incentive. We find that context swapping is not effective in this environment, highlighting the need for alternative mitigation strategies. 7.1. HI-ADS Unit Test 1: Supervised Learning This unit test consists of a simple prediction problem. There are no inputs, only an underlying statesâ 0,1, and targetsyâR 2 withy 1 ,y 2 ⌠N(0,sâÏ 2 ),N(0,1), with corresponding predictionsËy 1 ,Ëy 2 . Additionally,s t+1 = 0iff Ëy 2 > .5. We use Mean Squared Error as the loss function, so the optimal predictor isËy 1 ,Ëy 2 = (0,0). However, predicting Ëy 2 > .5reduces the variance ofËy 1 , i.e. reduces future loss. The baseline/IL predictor learnsËy 1 ,Ëy 2 as parameters using SGD with a learning rate of 0.001. For experiments with meta-learning, PBT is the OL (with default settings, see Section 2.2), used to tune the learning rate, with negative loss on the final time-step of the interval as the performance measure for PBT. 7.2. HI-ADS Unit Test 2: Myopic RL This unit test is based on a version of the prisonerâs dilemma (Prisner, 2014) where an agent plays each round against its past self. The reward function is presented in Table 1. An agent in this environment has a long-term,non-myopic, incentive for cooperation (with its future self), but a current- time-step,myopic, incentive for defection (from its future self). The unit test evaluates whether a agent reveals the non-myopic incentive even when the agent is meant to op- timize for the present reward only (i.e. uses discount rate Îł= 0). Naively, weâd expect the non-myopic incentive to behiddenfrom the agent in this case, and for the agent to consistentlydefect; learning algorithms that do so pass the test. But some learning algorithms alsofailthe unit test, revealing the incentive for the agent to cooperate with its fu- ture self. While aiming for myopic behavior may seem odd, myopic learners have no incentives to cause distributional shift, since it can only improvefutureperformance. And while making learners myopic may seem like a âbrute-forceâ guaranteed way to manage HI-ADS, we show it is in fact non-trivial to implement. Formally, this environment is not a 2x2 game (as the origi- nal prisonerâs dilemma); itâs a partially observable Markov Decision Process (Ă ström, 1965; Kaelbling et al., 1998): s t ,o t =a tâ1 , a t âdefect,cooperate P(s t ,a t ) =ÎŽ(a t ) R(s t ,a t ) =I(s t =cooperate) + ÎČ I(a t =cooperate)â1/2 whereIis an indicator function, andÎČ=â1/2is a parame- ter controlling the alignment of incentives (see Appendix 3.2 for an exploration of differentÎČvalues.). The initial state is sampled ass 0 âŒU(defect,cooperate). Policies are represented by a single real-valued parameterΞ(initialized asΞâŒN(0,1)) passed through a sigmoid whose output rep- resentsP(a t = defect). We use REINFORCE (Williams, 1992) with discount factorÎł= 0as the baseline/IL opti- mizer. PBT (with default settings, see Section 2.2) is used to tune the learning rate, with reward on the final time-step of the interval as the performance measure for PBT. Table 1. Rewards for the RL unit test. Note that the myopic (defect) action always increases reward at the current time-step, but decreases reward at the next time-step - the incentive is hidden from the point of view of a myopic learner. A supposedly myopic learner âfailsâ the unit test if the hidden incentive to cooperate is revealed, i.e. if we see morecooperate (C)actions than defect (D). a t =Da t =C s t =a tâ1 =D â1/2â1 s t =a tâ1 =C 1/20 7.2.1. HI-ADS UNITTESTS EXPERIMENTAL RESULTS AND DISCUSSION We first show that agents trained with PBT fail the unit tests more often than âvanillaâ algorithms which do not use meta-learning. We initialize the learning rate log-uniformly between0.01and1.0for all experiments (whether using Hidden Incentives for Auto-induced Distributional Shift PBT or not). We expect and confirm that the following two factors lead to higher rates of unit test failure: (1)Shorter intervals: These give the OL more opportunities to influ- ence the population. (2)Larger populations:These make outliers with exceptional non-myopic performance more likely, and OL makes them likely to survive and propagate. The baseline (no meta-learning) algorithms all pass the unit tests: hidden incentives are almost never revealed - see blue curves in Fig. 5. However, agents trained with meta- learning and large populations often fail the unit tests: see orange curves in top rows of Fig. 5. 0.00 0.25 0.50 0.75 1.00 || y 2 || #agents=10#agents=100 no env swapping #agents=1000 02004006008001000 time-step 0.00 0.25 0.50 0.75 1.00 || y 2 || 02004006008001000 time-step 02004006008001000 time-step env swapping (A) SL Unit Test. OL=PTB. (B1) Myopic RL Unit Test. OL=PBT. (B2) Myopic RL Unit Test. OL=REINFORCE Figure 5.(A) Values ofËy 2 in the supervised learning (SL) unit test. Larger values mean sacrificing present performance for future performance (i.e. non-myopic exploitation of ADS).(B)Average level of non-myopiccooperatebehavior observed in the RL unit test for HI-ADS, with two meta-learning algorithms(B1)PBT and(B2)REINFORCE. Lower is better, since the goal is for non- myopic incentives to remain hidden. Despite the inner loop being fully myopic (simple MLP in the SL test,Îł= 0in RL test), in all cases outer-loop (OL) optimizers reveal HI-ADS (top rows). Context swapping significantly mitigates HI-ADS (bottom rows). Furthermore, we verify that context swapping significantly mitigates the effect of HI-ADS in both unit tests, decreas- ing undesirable behaviour to near-baseline levels - see bot- tom rows of Fig. 5. This effect can be explained as fol- lows: Because context swapping transfers the benefits of one learnerâs action to the next learner to inhabit that en- vironment, it increases the second learnerâs fitness, and thereby reduces therelativefitness (as evaluated by PBTâs EXPLOIT step) of the non-myopiccooperatebehaviour. We observe some interesting exceptions with the combina- tion of small populations and short PBT intervals: Although context swapping still significantly decreases the effect of HI-ADS, non-myopiccooperatebehaviour is observed as much as 20% of the time (for #learners=10,T= 1; see bottom-left plot). We also observe that PBT reveals HI-ADS even when T= 1, where the explanation that PBT operates on a longer time horizon than the inner loop does not apply. We pro- vide a detailed explanation for how this might happen in Appendix 3, but in summary, we hypothesize that there are at least 2 mechanisms by which PBT is revealing HI-ADS: (1) optimizing over a longer time-scale, and (2) picking up on the correlation between an agentâs current policy and the underlying state. Mechanism (2) can be explained in- formally as reasoning as: âIf Iâm cooperating, then I was probably cooperating on the last time-step as well, so my reward should be higherâ. As support for these hypotheses, we run control experiments identifying two algorithms (each sharing onlyoneof these properties) that can fail the unit test. Context swapping remains effective. (1)Optimizing over a longer time-scale:replacing PBT with REINFORCE as an outer-loop optimizer. The outer- loop optimizes the parameters to maximize the summed reward of the lastTtime-steps. As with PBT, we observe non-myopic behavior, but nowonlywhenT >1. This supports our hypothesis that exploitation of HI-ADS is due not to PBT in particular, but just to the introduction of sufficiently powerful meta-learning. See Fig. 5 B2. (2)Exploiting correlation:Q-learning withÎł= 0an = 0.1-greedy behavior policyand no meta-learning. If either state was equally likely, the Q-values would be the average of the values in each column in Table 1, so the esti- matedQ(defect)would be larger. But the-greedy policy correlates the previous action (i.e. the current state) and cur- rent action (so long as the policy did not just change), so the top-left and bottom-right entries carry more weight in the estimates,sometimescausingQ(defect)âQ(cooperate) and persistent nonmyopic behavior. See Fig. 6 for results, Appendix 4.2 for more results, and Appendix 4.1 for experi- mental details. Hidden Incentives for Auto-induced Distributional Shift Figure 6.Q-learning fails the unit test for some random seeds; empiricalp(cooperate)stays around 80-90% in 3 of 5 exper- iments (bottom row). Each column represents an independent experiment. Q-values for thecooperateanddefectactions stay tightly coupled in the failure cases (col. 1,2,5), while in the cases passing the unit test (col. 3,4) the Q-value ofcooperate decreases over time. 7.3. HI-ADS in content recommendation We now present a toy environment for modeling content recommendation of news articles, which includes the poten- tial for ADS by incorporating the mechanisms mentioned in Sec. 2.2, discussed as contributing factors to the problems of fake news and filter bubbles. Specifically, the environment assumes that presenting an article to a user can influence (1) their interest in similar articles, and (2) their propensity to use the recommendation service. These correspond to mod- eling auto-induced concept shift of users, and auto-induced covariate shift of the user base, respectively (see Sec. 2.2). This environment includes the following components, which change over (discrete) time:User type:x t ,Article type: y t ,User interests:W t (propensity for users of each type to click on articles of each type), andUser loyalty:g t (propensity for users of each type to use the platform). At each time stept, a userx t is sampled from a categorical distribution, based on the loyalty of the different user types. The recommendation system (a classifier) selects which type of article to present in the top position, and finally the user âclicksâ an articley t , according to their interests. User loyalty for user typex t undergoes covariate shift: in accordance with the self-selection effect,g t increases or decreases proportionally to that user typeâs interest in the top article. The interests of user typex t (represented by a column ofW t ) undergoing concept shift; in accordance with the illusory truth effect, interest in the topic of the top article chosen by the recommender system always increases. Formally, this environment is similar to a POMDP , i.e. a POMDP with no reward function, also known as aworld model(Armstrong & OâRourke, 2017; Hadfield-Menell et al., 2017); the difference is that the learner observes the input (o t pre ) before acting and only observes the target (o t post ) after acting. The statess, observationso, and actions aare computed as follows: s t = (g t ,W t ,x t ,y t ) o t pre , a t , o t post = (x t ,Ëy t ,y t ) For further details on this environment, including the state transition function, see Appendix 2. Our recommender system is a 1-layer MLP trained with SGD-momentum. Actions are sampled from the MLPâs predictive distribution. For PBT, we useT= 10and 20 agents, and use accuracy to evaluate performance. We run 20 trials, and match random seeds for trials with and without PBT. See Appendix 3 for full experimental details. 7.3.1. CONTENT RECOMMENDATION EXPERIMENTAL RESULTS AND DISCUSSION We find that PBT yields significant improvements in training time and accuracy, but also greater distributional shift ( Fig. 7). User base and user interests both change faster with PBT, and user interests change more overall. We observe that the distributions over user types typically saturate (to a single user type) after a few hundred time-steps (Fig 1 and Fig. 7, Right). We run long enough to reach such states, to demonstrate that the increase in ADS from PBT is not transitory. The environment has a number of free parameters, and our results are qualitatively consistent so long as (1) the initial user distribution is approximately uniform, and (2) the covariate shift rate (α 1 ) is faster than the concept shift rate (α 2 ). See Appendix 2 for details. We measure concept shift (change inP(y|x)) as the cosine distance between each user typesâ initial and current interest vectors. And we measure covariate shift (change inP(x)) as the KL-divergence between the current and initial user distributions, parametrized byg 1 andg t , respectively. In Figure 8, we plot concept shift and covariate shift as a func- tion of accuracy. We observe that for both types of ADS, at low levels of accuracy PBT actually causeslessshift than occur in baseline agents; HI-ADS are only observed for accuracies above 60%. This suggests that only rela- tively strong performers are able to pick up on the HI-ADS revealed by PBT (Fig. 8). Hidden Incentives for Auto-induced Distributional Shift Figure 7.Content recommendation experiments.Left: using Pop- ulation Based Training (PBT) increases accuracy of predictions faster, leads to a faster and larger drift in usersâ interests,P(y|x), (Center); as well as the distribution of users,P(x), (Right). Shad- ing shows std error over 20 runs. Figure 8.Amount of auto-induced covariate shift (left) and auto- induced concept shift (right) as a function of performance (ac- curacy) averaged over all trials, learners, and time-steps. Only relatively strong learners (those which achieve accuracy > 60%) exhibit HI-ADS. 8. Related work ADS in practice:We introduce the term ADS, but we are far from the first to study it. Caruana et al. (2015) provide an example of asthmatic patients having lower predicted risk of pneumonia. Treating asthmatics with pneumonia less aggressively on this basis would be an example of harm- ful ADS; thereasonthey had lower pneumonia risk was because they had receivedmoreaggressive care already. Schulam & Saria (2017) note that such predictive models are commonly used to inform decision-making, and propose modeling counterfactuals (e.g. âhow would this patient fare with less aggressive treatmentâ) to avoid making such self- refuting predictions. While their goal is to make accurate predictions in the presence of ADS, our goal is to identify and manageincentives forADS. Goodfellow (2019) argues that adversarial defenses that do not account for ADS are critically flawed. Non-i.i.d bandits:Contextual bandits (Wang et al., 2005; Langford & Zhang, 2008) are frequently discussed as an ap- proach to content recommendation (Li et al., 2010). While bandit algorithms typically make the i.i.d. assumption, counter-examples exist (Gheshlaghi Azar et al., 2014; Shah et al., 2018); most famously, adversarial bandits (Auer et al., 1995). Closest to our work is Shah et al. (2018), who con- sider covariate shift caused by multi-armed bandits. Our task in Sec. 7.3 is similar to their problem statement, but more general in that we include user features, thus disen- tangling covariate shift and concept shift. Our motivation is also different: Shah et al. (2018) seek to exploit ADS, whereas we aim to avoid hidden incentives for ADS. Safety and incentives:Emergent incentives to influence the world (such as HI-ADS) are at the heart of many con- cerns about the safety of advanced AI systems (Omohun- dro, 2008; Bostrom, 2014). Understanding and managing the incentives of learners is also a focus of Armstrong & OâRourke (2017); Everitt (2018); Everitt et al. (2019); Co- hen et al. (2019). While Everitt et al. (2019) focus on identi- fying which incentives are present, we note that incentives may bepresentand yet not berevealedorpursued- for ex- ample, in supervised learning, there is an incentive to make predictions that are over-fit to the test set, but we typically hide the test set from the learner, which effectively hides this incentive. While Carey et al. (2020); Everitt et al. (2019); Armstrong & OâRourke (2017) discuss methods of remov- ing problematic incentives, we note in practice incentives are oftenhiddenrather than removed. Our work addresses the efficacy of this approach and ways in which it can fail. HI-ADS and meta-learning:As far as we know, our work is the first to consider the problem of HI-ADS, or its relation to meta-learning. A few previous works have some relevance or resemblance. Rabinowitz (2019) doc- uments qualitative differences in learning behavior when meta-learning is applied. MacKay et al. (2019) and Lor- raine & Duvenaud (2018) view meta-learning as a bilevel optimization problem, with the inner loop playing a best- response to the outer loop. In our work, the inner loop is unable to achieve such best-response behavior; the outer loop is too powerful (see Fig. 5). Finally, Sutton et al. (2007) note that meta-learning can change learning behavior in a way that improves performance by preventing convergence of the inner loop. 9. Discussion and Conclusion We have identified the phenomenon of auto-induced distri- butional shift (ADS), and the problems that can arise when there are hidden incentives for learners to induce distribu- tional shift (HI-ADS). And our experiments demonstrate that using meta-learning can reveal HI-ADS and lead learn- ers to use ADS as a means of increasing performance. Our work highlights the interdisciplinary nature of issues with real-world deployment of ML systems - we show how HI-ADS could play a role in important technosocial issues like filter bubbles and the propagation of fake news. There are a number of potential implications for our work: (1) When HI-ADS are a concern, our methodology and envi- ronments can be used to help diagnose whether and to what Hidden Incentives for Auto-induced Distributional Shift extent the final performance/behavior of a learner is due to ADS and/or incentives for ADS, i.e. to quantify their influence on that learner. (2) Comparing this quantitative analysis for different algorithms could help us understand which features of algorithms affect their propensity to reveal HI-ADS, and aid in the development of safer and more ro- bust algorithms. (3) Characterizing and identifying HI-ADS in these tests is a first step to analyzing and mitigating other (problematic) incentives, as well as to developing theoretical understanding of incentives. Broadly speaking, our work emphasizes that the choice of machine learning algorithm plays an important role in speci- fication, independently of the choice of performance metric. A learner can use ADS to increase performanceaccording to the intended performance metric, and yet still behave in an undesirable way, if we did not intend the learner to improve performance by thatmethod. In other words, performance metrics are incomplete specifications: they only specify our goals orends, while our choice of learning algorithm plays a role in specifying themeansby which we intend an learner to achieve those ends. With increasing deployment of ML algorithms in daily life, we believe that (1) understanding incentives and (2) specifying desired/allowed means of im- proving performance are important avenues of future work to ensure fair, robust, and safe outcomes. 10. Acknowledgements Thanks to the DeepMind and Future of Humanity Institute AI safety teams who gave lots of feedback on these ideas. Thanks to Valentin Dalibard for help with Population Based Training, and Toby Pohlen for help with using Google infras- tructure. Thanks to Owain Evans, Audrey Durand, Jacob Buckman, Michael Noukhovitch and Emmanuel Bengio for feedback on drafts. References Allcott, H. and Gentzkow, M. Social media and fake news in the 2016 election.Journal of Economic Perspectives, 31(2):211â36, May 2017. Amazeen, M. A. and Wojdynski, B. W. Reducing native advertising deception: Revisiting the antecedents and consequences of persuasion knowledge in digital news contexts.Mass Communication and Society, 0(0):1â26, 2018. Andrychowicz, M., Denil, M., GĂłmez, S., Hoffman, M. W., Pfau, D., Schaul, T., Shillingford, B., and de Freitas, N. Learning to learn by gradient descent by gradient descent. InNeural Information Processing Systems, 2016. Armstrong, S. and OâRorke, X. Good and safe uses of ai oracles.arXiv preprint arXiv:1711.05541, 2017. Armstrong, S. and OâRourke, X. Indifference methods for managing agent rewards. Technical report, Future of Humanity Institute, 2017. Ă ström, K. J. Optimal control of Markov Processes with incomplete state information.Journal of Mathematical Analysis and Applications, 10:174â205, January 1965. Auer, P., Cesa-Bianchi, N., Freund, Y., and Schapire, R. E. Gambling in a rigged casino: The adversarial multi-armed bandit problem. InFoundations of Computer Science, 1995. Bakshy, E., Messing, S., and Adamic, L. A. Exposure to ideologically diverse news and opinion on Facebook.Sci- ence, 348(6239):1130â1132, 2015. ISSN 0036-8075. doi: 10.1126/science.a1160. URLhttp://science. sciencemag.org/content/348/6239/1130. Bostrom, N.Superintelligence: Paths, Dangers, Strategies. Oxford University Press, Inc., New York, NY, USA, 1st edition, 2014. Carey, R., Langlois, E., Everitt, T., and Legg, S. The incentives that shape behaviour.arXiv preprint arXiv:2001.07118, 2020. Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., and Elhadad, N. Intelligible models for healthcare: Predict- ing pneumonia risk and hospital 30-day readmission. In International Conference on Knowledge Discovery and Data Mining, p. 1721â1730, 2015. Christiano, P., Leike, J., Brown, T. B., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences, 2017. Cohen, M. K., Catt, E., and Hutter, M. A strongly asymp- totically optimal agent in general environments.Proceed- ings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, Aug 2019. doi: 10.24963/ijcai. 2019/302. URLhttp://dx.doi.org/10.24963/ ijcai.2019/302. DiFranzo, D. and Gloria-Garcia, K. Filter bubbles and fake news.XRDS, 23(3):32â35, April 2017. ISSN 1528- 4972. doi: 10.1145/3055153. URLhttp://doi.acm. org/10.1145/3055153. El-Bermawy,M.M.Yourechocham- berisdestroyingdemocracy,2016.URL https://w.wired.com/2016/11/ filter-bubble-destroying-democracy/. Everitt, T.Towards Safe Artificial General Intelligence. PhD thesis, Australian National University, 2018. Everitt, T. and Hutter, M. The alignment problem for bayesian history-based reinforcement learners. 2018. Hidden Incentives for Auto-induced Distributional Shift Everitt, T. and Hutter, M. Reward tampering problems and solutions in reinforcement learning: A causal influence diagram perspective.arXiv preprint arXiv:1908.04734, 2019. Everitt, T., Ortega, P. A., Barnes, E., and Legg, S. Under- standing agent incentives using causal influence diagrams. part i: Single action settings, 2019. Fazio, L., Brashier, N., Keith Payne, B., and Marsh, E. Knowledge does not protect against illusory truth. Journal of Experimental Psychology: General, 144(5): 993â1002, 10 2015. ISSN 0096-3445. doi: 10.1037/ xge0000098. Finn, C., Abbeel, P., and Levine, S. Model-agnostic meta- learning for fast adaptation of deep networks. InInterna- tional Conference on Machine Learning, 2017. Flaxman, S. and Goel, S. Filter bubbles, echo chambers, and online news consumption.Public Opinion Quarterly, 2015. Gheshlaghi Azar, M., Lazaric, A., and Brunskill, E. Online stochastic optimization under correlated bandit feedback. ArXiv preprint, 2014. Gong, D., Zhang, Z., Shi, Q., van den Hengel, A., Shen, C., and Zhang, Y. Learning an optimizer for image deconvo- lution.ArXiv preprint, 2018. Goodfellow, I. J. A research agenda: Dynamic models to defend against correlated attacks.ArXiv preprint, 2019. Groshek, J. and Koc-Michalska, K. Helping populism win?Social media use, filter bubbles, and sup- port for populist presidential candidates in the 2016 us election campaign.Information, Communication & Society, 20(9):1389â1407, 2017.doi: 10.1080/ 1369118X.2017.1329334. URLhttps://doi.org/ 10.1080/1369118X.2017.1329334. Hadfield-Menell, D., Milli, S., Abbeel, P., Russell, S., and Dragan, A. Inverse reward design. InNeural Information Processing Systems, 2017. Jaderberg, M., Dalibard, V., Osindero, S., Czarnecki, W. M., Donahue, J., Razavi, A., Vinyals, O., Green, T., Dunning, I., Simonyan, K., Fernando, C., and Kavukcuoglu, K. Population Based Training of Neural Networks.ArXiv preprint, 2017. Kaelbling, L. P., Littman, M. L., and Cassandra, A. R. Plan- ning and acting in partially observable stochastic domains. Artificial Intelligence, 101(1-2):99â134, May 1998. ISSN 0004-3702. Kalousis, A. Model selection via meta-learning: A compar- ative study. InIEEE International Conference on Tools with Artificial Intelligence, 2000. Kayhan, V. Confirmation bias: Roles of search engines and search contexts. InInternational Conference on Informa- tion Systems, 2015. Langford, J. and Zhang, T. The epoch-greedy algorithm for multi-armed bandits with side information. InNeural Information Processing Systems, 2008. Lee Howell, E. Digital wildfires in a hyperconnected world. In Howell, L. (ed.),Global Risks 2013. World Economic Forum, 2013. URLhttp://reports.weforum. org/global-risks-2013/risk-case-1/ digital-wildfires-in-a-hyperconnected-world/. Leike, J., Martic, M., Krakovna, V., Ortega, P. A., Everitt, T., Lefrancq, A., Orseau, L., and Legg, S. AI safety gridworlds. Technical report, DeepMind, 2017. Leike, J., Krueger, D., Everitt, T., Martic, M., Maini, V., and Legg, S. Scalable agent alignment via reward modeling: a research direction. Technical report, DeepMind Safety Research, 2018. Li, L., Chu, W., Langford, J., and Schapire, R. E. A contextual-bandit approach to personalized news article recommendation. InInternational Conference on World Wide Web, 2010. Lorraine, J. and Duvenaud, D. Stochastic hyperparameter optimization through hypernetworks.ArXiv preprint, 2018. Luxton, D. D., June, J. D., and Fairall, J. M. Social me- dia and suicide: A public health perspective.American journal of public health, 102(2):195â200, 2012. MacKay, M., Vicol, P., Lorraine, J., Duvenaud, D., and Grosse, R. Self-tuning networks: Bilevel optimization of hyperparameters using structured best-response functions. ArXiv preprint, 2019. Merriam-Webster.Therealstoryoffake news,2017.URLhttps://w. merriam-webster.com/words-at-play/ the-real-story-of-fake-news. Metz, L., Maheswaranathan, N., Cheung, B., and Sohl- Dickstein, J. Learning unsupervised learning rules. In International Conference on Learning Representations, 2019. Mihailidis, P. and Viotty, S. Spreadable spectacle in dig- ital culture: Civic expression, fake news, and the role of media literacies in "post-fact" society.American Be- havioural Scientist, 2017. Hidden Incentives for Auto-induced Distributional Shift Moreno-Torres, J. G., Raeder, T., Alaiz-RodrĂGuez, R., Chawla, N. V., and Herrera, F. A unifying view on dataset shift in classification.Pattern Recognition, 45 (1):521â530, January 2012. Nguyen, T. T., Hui, P.-M., Harper, F. M., Terveen, L., and Konstan, J. A. Exploring the filter bubble: The effect of using recommender systems on content diver- sity. InProceedings of the 23rd International Confer- ence on World Wide Web, W â14, p. 677â686, New York, NY, USA, 2014. ACM. ISBN 978-1-4503- 2744-2. doi: 10.1145/2566486.2568012. URLhttp: //doi.acm.org/10.1145/2566486.2568012. Noble, S. U.Algorithms of Oppression: How Search En- gines Reinforce Racism. NYC Press, 2018. Omohundro, S. M. The basic AI drives. InConference on Artificial General Intelligence, 2008. Ortega, P. A., Maini, V., et al. Building safe artificial intelli- gence: specification, robustness, and assurance, 2018. Pariser, E.The Filter Bubble: What the Internet Is Hiding from You. The Penguin Group, 2011. Pennycook, G., Cannon, T. D., and Rand, D. G. Prior expo- sure increases perceived accuracy of fake news.Journal of Experimental Psychology (forthcoming), 2019. Prisner, E.Game Theory Through Examples. Mathematical Association of America, 2014. Quionero-Candela, J., Sugiyama, M., Schwaighofer, A., and Lawrence, N. D.Dataset Shift in Machine Learning. The MIT Press, 2009. Rabinowitz, N. C. Meta-learnersâ learning dynamics are unlike learnersâ.ArXiv preprint, 2019. Ren, M., Triantafillou, E., Ravi, S., Snell, J., Swersky, K., Tenenbaum, J. B., Larochelle, H., and Zemel, R. S. Meta-learning for semi-supervised few-shot classification. ArXiv preprint, 2018. Robson, D. The myth of the online echo chamber, 2018. URLhttp://w.bbc.com/future/story/ 20180416-the-myth-of-the-online-echo-chamber. Schulam, P. and Saria, S. Reliable decision support using counterfactual models. InNeural Information Processing Systems, 2017. Shah, V., Blanchet, J., and Johari, R. Bandit learning with positive externalities.Neural Information Processing Systems, 2018. Shao, C., Ciampaglia, G. L., Varol, O., Yang, K.-C., Flam- mini, A., and Menczer, F. The spread of low-credibility content by social bots.Nature Communications, 9(4787), 2018. Snoek, J., Larochelle, H., and Adams, R. P. Practical Bayesian optimization of machine learning algorithms. InNeural Information Processing Systems, 2012. Sutton, R. S. and Barto, A. G.Introduction to Reinforcement Learning. MIT Press, 1998. Sutton, R. S., Koop, A., and Silver, D. On the role of tracking in stationary environments. InInternational conference on Machine learning, 2007. Techopedia.Filter bubble, 2018.URLhttps: //w.techopedia.com/definition/ 28556/filter-bubble. Wang, C.-C., Kulkarni, S. R., and Poor, H. V. Bandit prob- lems with side observations.IEEE Transactions on Auto- matic Control, 50(3):338â355, 2005. Wikipedia contributors. Confirmation bias â Wikipedia, the free encyclopedia,2018.URLhttps: //en.wikipedia.org/w/index.php?title= Confirmation_bias&oldid=875026726. [Online; accessed 20-January-2019]. Williams, R. J. Simple statistical gradient-following al- gorithms for connectionist reinforcement learning. In Machine Learning, p. 229â256, 1992. Hidden Incentives for Auto-induced Distributional Shift Appendices 1. Content recommendation in the wild Filter bubbles, the spread of fake news, and other techno- social issues are widely reported to be responsible for the rise of populism (Groshek & Koc-Michalska, 2017), in- crease in racism and prejudice against immigrants and refugees (Noble, 2018), increase in social isolation and sui- cide (Luxton et al., 2012), and, particularly with reference to the 2016 US elections, are decried as threatening the foun- dations of democracy (El-Bermawy, 2016). Even in 2013, well before the 2016 American elections, a World Economic Forum report identified these problems as a global crisis (Lee Howell, 2013). We focus on two related issues in which content recommen- dation algorithms play a role: fake news and filter bubbles. 1.1. Fake news Fake news (also called false news or junk news) is an ex- treme version of yellow journalism, propaganda, or clickbait, in which media that is ostensibly providing information fo- cuses on being eye-catching or appealing, at the expense of the quality of research and exposition of factual informa- tion. Fake news is distinguished by being specifically and deliberately created to spread falsehoods or misinformation (Merriam-Webster, 2017; Mihailidis & Viotty, 2017). Why does fake news spread? It may at first seem the solution is simply to educate people about the truth, but research tells us the problem is more multifaceted and insidious, due to a combination of related biases and cognitive effects includ- ingconfirmation bias(people are more likely to believe things that fit with their existing beliefs),priming(exposure to information unconsciously influences the processing of subsequent information, i.e. seeing something in a credible context makes things seem more credible) and theillusory truth effect (i.e. people are more likely to believe some- thing simply if they are told it is true). Allcott & Gentzkow (2017) track about 150 fake news sto- ries during the 2016 US election, and find the average Amer- ican adult saw 1-2 fake news stories, just over half believed the story was true, and likelihood of believing fake news in- creased with ideological segregation (polarization) of their social media. Shao et al. (2018) examine the role of so- cial bots in spreading fake news by analyzing 14 million Twitter messages. They find that bots are far more likely than humans to spread misinformation, and that success of a fake news story (in terms of human retweets) was heavily dependent on whether bots had shared the story. Pennycook et al. (2019) examine the role of the illusory truth effect in fake news. They find that even a single exposure to a news story makes people more likely to believe that it is true, and repeat viewings increase this likelihood. They find that this is not true for extremely implausible statements (e.g. âthe world is a perfect cubeâ), but that âonly a small degree of potential plausibility is sufficient for repetition to increase perceived accuracyâ of the story. The situation is further complicated by peoplesâ inability to distinguish promoted content from real news - Amazeen & Wojdynski (2018) find that fewer than 1/10 people were able to tell when content was an advertisement, even when it was explicitly labelled as such. Similarly, Fazio et al. (2015) find that repeated exposure to incorrect trivia make people more likely to believe it, even when they are later able to identify the trivia as incorrect. 1.2. Filter bubbles Filter bubbles, a term coined and popularized by Pariser (2011) are created by positive or negative feedback loops which encourage users or groups of users towards increas- ing within-group similarity, while driving up between-group dissimilarity. The curation of this echo chamber is called self-selection(people are more likely to look for or select things that fit their existing preferences), and favours what Techopedia (2018) calls intellectual isolation. In the con- text of social and political opinions, this is often called the polarization effect(Wikipedia contributors, 2018). Filter bubbles can be encouraged by algorithms in two main ways. The first is the most commonly described: simply by showing content that is similar to what a user has already searched for, search or recommender systems create a posi- tive feedback loop of increasingly-similar content (Pariser, 2011; Kayhan, 2015). The second way is similar but op- posite - if the predictions of an algorithm are good for a certain group of people, but bad for others, the algorithm can do better on its metrics by driving hard-to-predict users away. Then new users to the site will either be turned off entirely, or see an artificially homogenous community of like-minded peers, a phenomena Shah et al. (2018) call positive externalities. In a study of 50,000 US-based internet users, Flaxman & Goel (2015) find that two things increase with social me- dia and search engine use: (1) exposure of an individual to opposing or different viewpoints, and (2) mean ideological distance between users. Many studies cite the first result as evidence of thebenefitsof internet and social media (Rob- son, 2018; Bakshy et al., 2015), but the correlation of expo- sure with ideological distances demonstrates that exposure is not enough, and might even be counterproductive. Facebookâs own study on filter bubbles results show that the impact of the news feed algorithm on filter bubble âsizeâ (a measure of homogeneity of posts relative to a baseline) is almost as large as the impact of friend group composition (Bakshy et al., 2015). Kayhan (2015) specifically study Hidden Incentives for Auto-induced Distributional Shift the role of search engines in confirmation bias, and find that search context and the similarity of results in search engine results both reinforce existing biases and increase the likelihood of future biased searches. Nguyen et al. (2014) similarly study the effect of recommender systems on in- dividual usersâ content diversity, and find that the set of options recommended narrows over time. Filter bubbles create an ideal environment for the spread of fake news: they increase the likelihood of repeat viewings of similar content, and because of the illusory truth effect, that content is more likely to be believed and shared (Pennycook et al., 2019; DiFranzo & Gloria-Garcia, 2017; Pariser, 2011). We are not claiming that HI-ADS are entirely or even mostly responsible for these problems, but we do note that they can play a role that is worth addressing. incentive-compatible (ÎČ= 0.5) incentive-orthogonal (ÎČ= 0.0) incentive-opposed (ÎČ=â0.5) Figure 9.Average level of non-myopic (i.e.cooperate) be- havior learned by agents in the unit test for HI-ADS. Despite making the inner loop fully myopic (Îł= 0), population-based training (PBT) can cause HI-ADS, leading agents to choose the cooperateaction (top row). context swapping successfully prevents this (bottom row). Columns (from left to right) show results for populations of 10, 100, and 1000 learners. In the legend, âintervalâ refers to the interval (T) of PBT (see Sec. 2.2). Suffi- ciently large populations and short intervals are necessary for PBT to induce nonmyopic behavior. 2. Extra experiments and reproducibility details 2.1. HI-ADS unit test 2.1.1. ALIGNMENT OF INCENTIVES EXPLORATION This section presents an exploration of the parameterÎČ, which controls the alignment of incentives in the HI-ADS unit tests (see Table 2). To clarify the interpretation of experiments, we distinguish between environments in which myopic (defect) vs. non- myopic (cooperate) incentives areopposed,orthogonal, or compatible. Note that in this unit test myopic behaviour (defection) is what we want to see. 1.Incentive-opposed: Optimal myopic behavior is in- compatible with optimal nonmyopic behavior (classic prisonerâs dilemma; these experiments are in the main paper). 2.Incentive-orthogonal : Optimal myopic behavior may or may not be optimal nonmyopic behavior. 3.Incentive-compatible : Optimal myopic behavior is necessarily also optimal nonmyopic behavior. We focused on incentive-opposed environment (ÎČ=â1/2) in the main paper in order to demonstrate that HI-ADS can be powerful enough to change the behavior of the system in an undesirable way. Here we also explore incentive- compatible and incentive-orthogonal environments because they provide useful baselines, helping us distinguish a sys- tematic bias towards nonmyopic behavior from other rea- sons (such as randomness or optimization issues) for behav- ior that does not follow a myopically optimal policy. 2.1.2. WORKING THROUGH A DETAILED EXAMPLE FOR PBTWITHT= 1 To help provide intuition on how (mechanistically) PBT could lead to persistent levels of cooperation, we walk through a simple example (with no inner loop). Consider PBT withT= 1and a population of 5 deterministic agentsA 1 ,...,A 5 playingcooperateand receiving reward ofr(A i ) = 0. Now supposeA 1 suddenly switches to play defect. Thenr(A 1 ) = 1/2on the next time-step (while the other agentsâ reward is still0), and so PBTâs EXPLOIT step will copyA 1 (without loss of generality toA 2 ). On the following time-step,r(A 2 ) = 1/2, andr(A 1 ) =â1/2, so PBT will cloneA 2 toA 1 , and the cycle repeats. Similar reasoning applies for larger populations, andT >1. Hidden Incentives for Auto-induced Distributional Shift Table 2.ÎČcontrols the extent to which myopic and nonmyopic incentives are aligned. ÎČEnvironmentCooperating <0incentive-opposedyields less reward on the current time-step (myopically detrimental) = 0incentive-orthogonaldoes not affect the current reward (myopically indifferent) >0 incentive-compatibleyields more reward on the current time-step (myopically beneficial) 2.1.3. Q-LEARNING EXPERIMENT DETAILS We show that, under certain conditions, Q-learning can learn to (primarily) cooperate, and thus fails the HI-ADS unit test. We estimate Q-values using the sample-average method, which is guaranteed to converge in the fully observed, tab- ular case (Sutton & Barto, 1998). The agent follows the -greedy policy with= 0.1. In order to achieve this re- sult, we additionally start the agent off with one synthetic memory where both state and action aredefectand therefor R(defect) =â.5, and we hard-code the starting state to becooperate(which normally only happens 50% of the time). Without this kind of an initialization, the agent al- ways learns to defect. However, under these conditions, we find that 10/30 agents learned to play cooperate most of the time, withQ(cooperate)andQ(defect)both hovering aroundâ0.07, while others learn to always defect, with Q(cooperate)â â0.92andQ(defect)â â0.45. context swapping, however, prevents majority-cooperate behavior from ever emerging, see Figure 12. 2.1.4. Q-LEARNING:FURTHER RESULTS To give a more representative picture of how often Q- learning fails the unit test, we run a larger set of experiments with Q-learning, results are in Figure 11. Itâs possible that the failure of Q-learning is not persistent, since we have not proved otherwise, but we did run much longer experiments and still observe persistent failure, see Figure 10. Figure 10.The same experiments as Figures 6, 11, run for 50,000 time-steps instead of 3000, to illustrate the persistence of non- myopic behavior. Figure 11.More independent experiments with Q-learning, ex- actly following Figure 6. Q-learning fails the unit test in a total of 10/30 experiments (including those from Figure 6). Hidden Incentives for Auto-induced Distributional Shift Figure 12.More independent experiments with Q-learning, ex- actly following Figure 6, except also using context swapping. This leads to a 100% success rate on the unit test. 2.2. Content recommendation 2.2.1. ENVIRONMENT DETAILS The evironment has the following components: 1.User type,x t : categorical variable representing differ- ent types of users. The content recommender condi- tions its predictions on the type of the current user. 2.User loyalty,g t : the propensity for users of each type to use the platform. Userx t is sampled from a categori- cal distribution with parameters given bysoftmax(g t ). 3.Article type,y t : a categorical variable (one-hot en- coding) representing the type of article selected by the user. 4.User interests ,W t : a matrix whose entriesW t x,y rep- resent the average interest user of typexhave in articles of typey. At each time stept, a userx t is sampled from a categori- cal distribution (based on the loyalty of the different user types), then the recommendation system selects which type of article to present in the top position, and finally, the user selects an article. The goal of the recommendation system is to predict the likelihood that the user would click on each of the available articles, in order to select the one which is most interesting to the user. User loyalty forx t then changes in accordance with the self-selection effect, increasing or decreasing proportionally to their interest in the top article. The interests of user typex t (represented by a column ofW t ) also change; in accordance with the illusory truth effect, their interest in the topic of the top article (as chosen by the recommender system) always increases. Overall, this environment is an extremely crude representation of reality, but it allows us to incorporate both the effects of self-selection (via covariate shift), and the illusory truth effect (via concept shift). Formally, this environment is similar to a POMDP , i.e. a POMDP with no reward function, also known as aworld model (Armstrong & OâRourke, 2017; Hadfield-Menell et al., 2017); the difference is that the learner observes the input before acting and only observes the target after acting. The states, observations, and actions given below. s t = (g t ,W t ,x t ,y t ) o t pre , a t , o t post = (x t ,Ëy t ,y t ) Hidden Incentives for Auto-induced Distributional Shift The state transition function is defined by: g t+1 x t =g t x t +α 1 W t x t ,Ëy t W t+1/2 x t ,Ëy t =W t x t ,Ëy t +α 2 ;W t+1 x t = W t+1/2 x t âW t+1/2 x t â 2 x t+1 âŒsoftmax(g t+1 ) y t+1 âŒsoftmax(W t+1 x t+1 ) WhereËy t is the top article as chosen by the recommender, andα 1 ,α 2 represent the rate of covariate and concept shift (respectively). The update forW t+1 merely increases the interest of user typex t in article typeËy t , then normalizes the interests for that user type. 2.2.2. REPRODUCIBILITY DETAILS For these experiments, the recommendation system is a ReLU-MLP with 1 hidden layer of 100 units, trained via su- pervised learning with SGD (learning rate =0.01) to predict which article a user will select. Actions are sampled from the MLPâs predictive distribution. We apply PBT without any hyperparameter selection (this amounts to just doing the EXPLOIT step), and an interval of 10, selecting on accu- racy. We use a population of 20 learners (whether applying PBT or not), and match random seeds for the trials with and without PBT. We initializeg 1 andW 1 to be the same across the 20 copies of the environment (i.e. the learners start with the same user population), but these values di- verge throughout learning. For the environment, we set the number of user and article types both to 10. Initial user loy- alties are randomly sampled fromN(0,0.03),α 1 = 0.03, andα 2 = 0.003. 2.2.3.CONTEXT SWAPPING IN CONTENT RECOMMENDATION We believe context swapping is not appropriate for the con- tent recommendation environment, since when the envi- ronments diverge, optimal behavior may differ across en- vironments. Nevertheless, we ran experiments with it for completeness. The main effect appears to be to hamper learning when PBT is not used, see Figure 13. Notably, it does not appear to significantly influence the rate or extent of ADS when combined with PBT. 2.2.4. EXPLORATION OF ENVIRONMENT PARAMETERS In Figure 14, we examine the effect of the rate-of-change parameters (α 1 ,α 2 ) of the content recommendation envi- ronment on the results provided in the paper. As noted there, our results are qualitatively consistent so long as (1) the initial user distribution is approximately uniform, and (2) the covariate shift rate (α 1 ) is faster than the concept shift rate (α 2 ). These distributions are updated by differ- ent mechanisms, and are not directly comparable. Concept shift changes the task more radically, requiring a learner to change its predictions, rather than just become accurate on a wider range of inputs. We conjecture that changes inP(y|x) must therefore be kept smooth enough for the outer loop to have pressure to capitalize on HI-ADS. Figure 13.Context swapping doesnât have the desired effect in the content recommendation environment. Hidden Incentives for Auto-induced Distributional Shift α 1 = 0.01,α 2 = 0.001α 1 = 0.1,α 2 = 0.001 α 1 = 0.01,α 2 = 0.01α 1 = 0.1,α 2 = 0.01 α 1 = 0.01,α 2 = 0.1α 1 = 0.1,α 2 = 0.1 Figure 14.Content recommendation results for different values ofα 1 ,α 2 .