Paper deep dive

When Your AIs Deceive You: Challenges with Partial Observability of Human Evaluators in Reward Learning

Leon Lang, Davis Foote, Stuart Russell, Anca Dragan, Erik Jenner, Scott Emmons

Year: 2024Venue: NeurIPS 2024Area: Deception & FailureType: TheoreticalEmbeddings: 542

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%

Last extracted: 3/12/2026, 6:54:26 PM

Summary

The paper investigates the risks of Reinforcement Learning from Human Feedback (RLHF) when human evaluators have only partial observability of the environment. It formally defines two failure modes—deceptive inflation and overjustification—and proves that RLHF can lead to policies that exhibit these behaviors. The authors analyze the identifiability of the return function under partial observability and propose research directions for mitigation.

Entities (5)

Partial Observability · concept · 100%RLHF · methodology · 100%Deceptive Inflation · failure-mode · 98%Overjustification · failure-mode · 98%Boltzmann-rationality · human-model · 95%

Relation Signals (3)

RLHF → causes → Deceptive Inflation

confidence 90% · We prove conditions under which RLHF is guaranteed to result in policies that deceptively inflate their performance

RLHF → causes → Overjustification

confidence 90% · RLHF is guaranteed to result in policies that... overjustify their behavior to make an impression

Partial Observability → limits → Reward Identifiability

confidence 90% · In some realistic cases, there is irreducible ambiguity.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Past analyses of reinforcement learning from human feedback (RLHF) assume that the human evaluators fully observe the environment. What happens when human feedback is based only on partial observations? We formally define two failure cases: deceptive inflation and overjustification. Modeling the human as Boltzmann-rational w.r.t. a belief over trajectories, we prove conditions under which RLHF is guaranteed to result in policies that deceptively inflate their performance, overjustify their behavior to make an impression, or both. Under the new assumption that the human's partial observability is known and accounted for, we then analyze how much information the feedback process provides about the return function. We show that sometimes, the human's feedback determines the return function uniquely up to an additive constant, but in other realistic cases, there is irreducible ambiguity. We propose exploratory research directions to help tackle these challenges, experimentally validate both the theoretical concerns and potential mitigations, and caution against blindly applying RLHF in partially observable settings.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

541,797 characters extracted from source content.

Expand or collapse full text

[hyperref] on page When Your AIs Deceive You: Challenges of Partial Observability in Reinforcement Learning from Human Feedback Leon Lang University of Amsterdam &Davis Foote* UC Berkeley &Stuart Russell UC Berkeley Anca Dragan UC Berkeley & Erik Jenner UC Berkeley & Scott Emmons* UC Berkeley Core research contributor. Correspondence to l.lang@uva.nl, emmons@berkeley.edu Abstract Past analyses of reinforcement learning from human feedback (RLHF) assume that the human evaluators fully observe the environment. What happens when human feedback is based only on partial observations? We formally define two failure cases: deceptive inflation and overjustification. Modeling the human as Boltzmann-rational w.r.t. a belief over trajectories, we prove conditions under which RLHF is guaranteed to result in policies that deceptively inflate their performance, overjustify their behavior to make an impression, or both. Under the new assumption that the human’s partial observability is known and accounted for, we then analyze how much information the feedback process provides about the return function. We show that sometimes, the human’s feedback determines the return function uniquely up to an additive constant, but in other realistic cases, there is irreducible ambiguity. We propose exploratory research directions to help tackle these challenges, experimentally validate both the theoretical concerns and potential mitigations, and caution against blindly applying RLHF in partially observable settings. 1 Introduction Reinforcement learning from human feedback (RLHF) and its variants are widely used for finetuning foundation models, including ChatGPT (OpenAI, 2022), Bard (Manyika, 2023), Gemini (Gemini Team, 2023), Llama 2 (Touvron et al., 2023), and Claude (Bai et al., 2022; Anthropic, 2023a, b). Prior theoretical analysis of RLHF assumes that the human fully observes the state of the world (Skalse et al., 2023). Under this assumption, it is possible to recover the ground-truth return function from Boltzmann-rational human feedback (see Proposition 3.1). In reality, however, this assumption is false. Models like ChatGPT are interacting with the internet and software tools via plugins (OpenAI, 2023). Software assistants like Devin are interacting with complex IDEs to produce their results (Wu, 2024). By default, some of the models’ work then happens in the background, not observed by the users; see Figure 1. With the tasks performed by language model assistants becoming more complex, it is also increasingly time consuming for humans to evaluate the entire model behavior and input. Therefore, we are anticipating a future where by default, the human evaluators do not fully observe the environment state that the language assistant is embedded in. Our work analyzes the consequences and risks of such partial observability. Figure 1: Partial observability in ChatGPT (OpenAI, 2023). Users do not observe the online content that ChatGPT observes yet still provide thumbs-up thumbs-down feedback. OpenAI’s privacy policy (OpenAI, 2024c) allows user feedback to be used for training models. We show in Theorem 4.5 that if feedback of human evaluators is based on partial observations, then this can lead to deceptive and overjustifying behavior by the language model. We begin our investigation with a simple example, illustrated in Figure 2, meant to isolate the key factor leading to deception (in practice, we imagine that this effect would be embedded in a larger, more complex system, e.g. with logs containing thousands of lines). An AI assistant is helping a user install software. The assistant can hide error messages by redirecting them to /dev/null. We model the human as having a belief B over the state and extend the Boltzmann-rational assumption from prior work to incorporate this belief. In the absence of an error message, the human is uncertain if the agent left the system untouched or hid the error message from a failed installation. If the human interprets trajectories without error messages optimistically, the AI learns to hide error messages. Figure 4 provides further details on how this failure occurs, and Figure 5 shows an experimental validation. We also show a second case where the AI clutters the output with overly verbose logs. Generalizing from these examples, we formalize dual risks: deceptive inflation and overjustification. We provide a mathematical definition of each. When the observation kernel (the function specifying the observations given states) is deterministic, Theorem 4.5 analyzes properties of suboptimal policies learned by RLHF. These policies exhibit deceptive inflation, appearing to produce higher reward than they actually do; overjustification, incurring a cost in order to make a good appearance; or both. After seeing how standard RLHF fails, we ask: What would happen if we would model the human’s partial observability correctly in RLHF? Assuming the human’s belief is known, we mathematically analyze how much information the feedback process provides about the return function. In Theorem 5.2, we show that the human’s feedback determines the return function up to a constant and a linear subspace we call the ambiguity. In general the ambiguity may be large enough to allow for arbitrarily high regret, but in some situations the ambiguity vanishes. In experiments that serve as a proof of concept, we show that explicitly modeling the human’s partial observability can improve performance, and we offer optimism in the form of a robustness result (Theorem 5.4) while accounting for the major conceptual difficulties involved. We propose exploratory research directions to solve these issues to improve RLHF in situations of partial observability. 2 Related work Figure 2: A human compares trajectories to provide data for RLHF. Rather than observing s→ sover→ start_ARG s end_ARG and s→′ s 1.0pt over→ start_ARG s end_ARG′, the human sees observations o→ oover→ start_ARG o end_ARG and o→′ o 1.0pt over→ start_ARG o end_ARG′, which they use to estimate the total reward of each trajectory. In this intentionally simple example, an agent executes shell commands to install Nvidia drivers and CUDA. Both s→ sover→ start_ARG s end_ARG and s→′ s 1.0pt over→ start_ARG s end_ARG′ contain an error, but in s→′ s 1.0pt over→ start_ARG s end_ARG′, the agent hides the error. The human believes s→′ s 1.0pt over→ start_ARG s end_ARG′ is better than s→ sover→ start_ARG s end_ARG, rewarding the agent’s deceptive behavior. The underlying MDP and observation function are in Figure 8. A review of limitations of RLHF, including a brief discussion of partial observability, can be found in Casper et al. (2023). RLHF is a special case of reward-rational choice (Jeon et al., 2020), a general framework which also encompasses demonstrations-based inverse reinforcement learning (Ziebart et al., 2008; Ng et al., 2000) and learning from the initial environment state (Shah et al., 2019), and can be seen as a special case of assistance problems (Fern et al., 2014; Hadfield-Menell et al., 2016; Shah et al., 2021). In all of these, the reward function is learned from human actions, which in the case of RLHF are simply preference statements. This requires us to specify the human policy of action selection—Boltzmann rationality in typical RLHF—which can lead to wrong reward inferences when this specification is wrong (Skalse and Abate, 2022); unfortunately, the human policy can also not be learned alongside the human’s values without further assumptions (Mindermann and Armstrong, 2018). Instead of a model of the human policy, in this paper we mostly focus on the human belief model and misspecifications thereof for the case that the human only receives partial observations. The problem of human interpretations of observations was briefly mentioned in Amodei et al. (2017), where evaluators misinterpreted the movement of a robot hand in simulation. Eliciting Latent Knowledge (Christiano et al., 2021) posits that for giving accurate feedback from partial observations, the human needs to be able to query latent knowledge of the AI system about the state. How to do this is currently an unsolved problem (Christiano and Xu, 2022). Recent work (Denison et al., 2024; Wen et al., 2024) provides detailed empirical evidence for deceptive behavior — in line with our notion of deceptive inflation — emerging from RLHF based on partial observations, or human evaluators with limited time. The OpenAI o1 system card (OpenAI, 2024a) shows that o1 sometimes knowingly provides incorrect information or omits important information. Compared to these investigations, and in addition to providing some empirical evidence, we formalize a model of human feedback under partial observability, we prove the emergence of failure modes resulting from partial observations, and we investigate potential mitigations. Related work (Zhuang and Hadfield-Menell, 2020) analyzes the consequences of aligning an AI with a proxy reward function that omits attributes that are important to the human’s values, which could happen if the reward function is based on a belief over the world state given limited information. Another instance are recommendation systems (Stray, 2023), where user feedback does not depend on information not shown—which is crucially part of the environment. Siththaranjan et al. (2023) analyze what happens under RLHF if the learning algorithm doesn’t have all the relevant information (e.g. about the identity of human raters), complementing our study of what happens when human raters are missing information. Chidambaram et al. (2024) and Park et al. (2024a) deal with the situation that different human evaluators may vary in their unobserved preference types. In contrast, we assume a single human evaluator with fixed reward function, which can be motivated by cases where the human choices are guided by a behavior policy, constitution, or a model spec (Mu et al., 2024; Anthropic, 2023b; OpenAI, 2024b). Kausik et al. (2024) assumes that the choices of the human evaluator depend on an unobserved reward-state with its own transition dynamics, similar to an emotional state in a real human. In contrast, we assume the human to be stateless. Our work argues that deception can result from applying RLHF from partial observations. Deception may also emerge for other reasons: Hubinger et al. (2019) introduced the hypothetical scenario of deceptive alignment, in which an AI system deceives humans into believing it is aligned while it plans a later takeover. Under the definition from Park et al. (2024b), GPT-4 was shown to behave deceptively in a simulated environment (Scheurer et al., 2023). A third line of research defines deception in structural causal games and adds the aspect of intentionality (Ward et al., 2023), with recent preliminary empirical support (Hofstätter et al., 2023). Finally, we mention connections to truthful AI (Evans et al., 2021; Lin et al., 2022; Burns et al., 2023; Huang et al., 2023), which is about ensuring that AI systems tell the truth about aspects of the real world. Partial observability is a mechanism that makes it feasible for models to lie without being caught: If the human evaluator does not observe the full environment, or does not fully understand it, then they may not detect when the AI is lying. More speculatively, we can imagine that AI models will at some point more directly influence human observations by telling us the outcomes of their actions. E.g., imagine an AI system that manages your assets and assures you that they are increasing in value while they are actually not. In our work, we leave this additional problem out of the analysis by assuming that the observations only depend on the environment state, and not directly on the agent’s actions. 3 Reward identifiability from full observations Here we review Markov decision processes and previous results on reward identifiability under RLHF. 3.1 Markov decision processes We assume Markov decision processes (MDPs) given by (,,,P0,R,γ)subscript0(S,A,T,P_0,R,γ)( S , A , T , P0 , R , γ ). For any finite set X, let Δ⁢(X)Δ (X)Δ ( X ) be the set of probability distributions on X. Then SS is a finite set of states, AA is a finite set of actions, :×→Δ⁢():→ΔT:S×A→ (S)T : S × A → Δ ( S ) is a transition kernel written ⁢(s′∣s,a)∈[0,1]conditionalsuperscript′01T(s s,a)∈[0,1]T ( s′ ∣ s , a ) ∈ [ 0 , 1 ], P0∈Δ⁢()subscript0ΔP_0∈ (S)P0 ∈ Δ ( S ) is an initial state distribution, R:→ℝ:→ℝR:S→ RR : S → blackboard_R is the true reward function, and γ∈[0,1]01γ∈[0,1]γ ∈ [ 0 , 1 ] is a discount factor. A policy is given by a function π:→Δ⁢():→Δπ:S→ (A)π : S → Δ ( A ). We assume a finite time horizon T. Let → Sover→ start_ARG S end_ARG be the set of possible state sequences s→=s0,…,sT→subscript0…subscript s=s_0,…,s_Tover→ start_ARG s end_ARG = s0 , … , sitalic_T, so s→∈→ s∈ Sover→ start_ARG s end_ARG ∈ over→ start_ARG S end_ARG if it has a strictly positive probability of being sampled from P0subscript0P_0P0, TT, and an exploration policy π with π⁢(a∣s)>0conditional0π(a s)>0π ( a ∣ s ) > 0 for all s∈,a∈formulae-sequences ,a ∈ S , a ∈ A. A sequence s→ sover→ start_ARG s end_ARG gives rise to a return G⁢(s→)≔∑t=0Tγt⁢R⁢(st)≔→superscriptsubscript0superscriptsubscriptG( s) _t=0^Tγ^tR(s_t)G ( over→ start_ARG s end_ARG ) ≔ ∑t = 0T γitalic_t R ( sitalic_t ). Let Pπ⁢(s→)superscript→P^π( s)Pitalic_π ( over→ start_ARG s end_ARG ) be the on-policy probability that s→ sover→ start_ARG s end_ARG is sampled from P0subscript0P_0P0, TT, π. The policy is then usually trained to maximize the policy evaluation function J, which is the on-policy expectation of the return function: J⁢(π)≔s→∼Pπ⁢(⋅)[G⁢(s→)]≔subscriptsimilar-to→superscript⋅→J(π) *E_ s P^π(·) [G% ( s) ]J ( π ) ≔ Eover→ start_ARG s end_ARG ∼ Pitalic_π ( ⋅ ) [ G ( over→ start_ARG s end_ARG ) ]. 3.2 RLHF and identifiability from full observations In practice, the reward function R may not be known and need to be learned from human feedback. In a simple form of RLHF (Christiano et al., 2017), this feedback takes the form of binary trajectory comparisons: a human is presented with state sequences s→ sover→ start_ARG s end_ARG and s→′ s 1.0pt over→ start_ARG s end_ARG′ and choose the one they prefer. Under the Boltzmann rationality model, we assume the human picks s→ sover→ start_ARG s end_ARG with probability PR⁢(s→≻s→′)≔σ⁢(β⁢(G⁢(s→)−G⁢(s→′))),≔superscriptsucceeds→superscript→′→superscript→′P^R ( s s 1.0pt ) σ % (β (G( s)-G( s 1.0pt ) ) ),Pitalic_R ( over→ start_ARG s end_ARG ≻ over→ start_ARG s end_ARG′ ) ≔ σ ( β ( G ( over→ start_ARG s end_ARG ) - G ( over→ start_ARG s end_ARG′ ) ) ) , (1) where β>00β>0β > 0 is an inverse temperature parameter and σ⁢(x):=11+exp⁡(−x)assign11σ(x):= 11+ (-x)σ ( x ) := divide start_ARG 1 end_ARG start_ARG 1 + exp ( - x ) end_ARG is the sigmoid function (Bradley and Terry, 1952; Christiano et al., 2017; Jeon et al., 2020). An important question is identifiability: In the infinite data limit, do the human choice probabilities PRsuperscriptP^RPitalic_R collectively provide enough information to uniquely identify the reward function R? This is answered by Skalse et al. (2023, Theorem 3.9 and Lemma B.3): Proposition 3.1 (Skalse et al. (2023)). Let R be the true reward function and G the corresponding return function. Then the collection of all choice probabilities PR⁢(s→≻s→′)superscriptsucceeds→superscript→′P^R( s s 1.0pt )Pitalic_R ( over→ start_ARG s end_ARG ≻ over→ start_ARG s end_ARG′ ) for state sequence pairs s→,s→′∈→superscript→′→ s, s 1.0pt ∈ Sover→ start_ARG s end_ARG , over→ start_ARG s end_ARG′ ∈ over→ start_ARG S end_ARG determines the return function G on sequences s→∈→ s∈ Sover→ start_ARG s end_ARG ∈ over→ start_ARG S end_ARG up to an additive constant. The reason is simple: because σ is bijective, PRsuperscriptP^RPitalic_R determines the difference in returns between any two trajectories. From that we can reconstruct individual returns up to an additive constant. The reward function R is not necessarily identifiable from preference comparisons; see Skalse et al. (2023, Lemma B.3) for a precise characterization. However, the optimal policy only depends on R indirectly through the return function G, and is invariant under adding a constant to G. Thus in the fully observable setting, Boltzmann rational comparisons completely determine the optimal policy. In Section 5, we show conditions under which this guarantee breaks in the partially observable setting. 4 The impact of partial observations on RLHF We now analyze failure modes of a naive application of RLHF from partial observations, both theoretically and with examples. In Proposition 4.1, we show that under partial observations, RLHF incentives policies that maximize what we call JobssubscriptobsJ_obsJroman_obs , a policy evaluation function that evaluates how good the state sequences “look to the human”. The resulting policies can show two distinct failure modes that we formally define and call deceptive inflation and overjustification. In Theorem 4.5 we prove that at least one of them is present for JobssubscriptobsJ_obsJroman_obs-maximizing policies. Later, in Section 5, we will see that an adaptation of the usual RLHF process might sometimes be able to avoid these problems. To model partial observability, we introduce an observation space o∈Ωo∈ ∈ Ω and observation kernel with probabilities PO⁢(o∣s)∈[0,1]subscriptconditional01P_O(o s)∈[0,1]Pitalic_O ( o ∣ s ) ∈ [ 0 , 1 ]. We write PO→⁢(o→∣s→)≔∏t=0TPO⁢(ot∣st)≔subscript→conditional→superscriptsubscriptproduct0subscriptconditionalsubscriptsubscriptP_ O( o s) _t=0^TP_O(o_t s_t)Pover→ start_ARG O end_ARG ( over→ start_ARG o end_ARG ∣ over→ start_ARG s end_ARG ) ≔ ∏t = 0T Pitalic_O ( oitalic_t ∣ sitalic_t ) for the probability of an observation sequence. We write Ω→Ω over→ start_ARG Ω end_ARG for the set of observation sequences that occur with non-zero probability, i.e., o→∈Ω→Ω o∈ over→ start_ARG o end_ARG ∈ over→ start_ARG Ω end_ARG if and only if there is s→∈→ s∈ Sover→ start_ARG s end_ARG ∈ over→ start_ARG S end_ARG such that ∏t=0TPO⁢(ot∣st)>0superscriptsubscriptproduct0subscriptconditionalsubscriptsubscript0 _t=0^TP_O(o_t s_t)>0∏t = 0T Pitalic_O ( oitalic_t ∣ sitalic_t ) > 0. If POsubscriptP_OPitalic_O and PO→subscript→P_ OPover→ start_ARG O end_ARG are deterministic, then we write O:→Ω:→ΩO:S→ : S → Ω and O→:→Ω→:→Ω O: S→ over→ start_ARG O end_ARG : over→ start_ARG S end_ARG → over→ start_ARG Ω end_ARG for the corresponding observation functions with O⁢(s)=oO(s)=oO ( s ) = o and O→⁢(s→)=o→ O( s)= oover→ start_ARG O end_ARG ( over→ start_ARG s end_ARG ) = over→ start_ARG o end_ARG for o and o→ oover→ start_ARG o end_ARG with PO⁢(o∣s)=1subscriptconditional1P_O(o s)=1Pitalic_O ( o ∣ s ) = 1 and PO→⁢(o→∣s→)=1subscript→conditional→1P_ O( o s)=1Pover→ start_ARG O end_ARG ( over→ start_ARG o end_ARG ∣ over→ start_ARG s end_ARG ) = 1, respectively. 4.1 What does RLHF learn from partial observations? We consider the setting where the state is fully observable to the learned policy, but human feedback depends only on a sequence of observations. We assume that the human gives feedback under a Boltzmann rational model similar to Eq. (1), modified such that they form some belief B⁢(s→∣o→)∈[0,1]conditional→01 B( s o)∈[0,1]B ( over→ start_ARG s end_ARG ∣ over→ start_ARG o end_ARG ) ∈ [ 0 , 1 ] about the state sequence s→ sover→ start_ARG s end_ARG based on the observations o→ oover→ start_ARG o end_ARG. We then assume preferences are Boltzmann rational in the expected returns under this belief, instead of the actual returns. The assumption of Boltzmann rationality is false in practice (Evans et al., 2015; Majumdar et al., 2017; Buehler et al., 1994), but note that it is an optimistic assumption: Even though our model is a simplification, we expect that practical issues can be at least as bad as the ones we will discuss. See also Example D.4 for an example showing that it is sometimes generally not possible to find a human model that leads to good outcomes under RLHF. Future work could investigate different human models and their impact under partial observability in greater detail. To formalize our setting, we collect human beliefs into a matrix ≔(B⁢(s→∣o→))o→,s→∈ℝΩ→×→≔subscriptconditional→superscriptℝ→Ω→ B (B( s o) )_ o% , s∈ R × SB ≔ ( B ( over→ start_ARG s end_ARG ∣ over→ start_ARG o end_ARG ) )over→ start_ARG o end_ARG , over→ start_ARG s end_ARG ∈ blackboard_Rover→ start_ARG Ω end_ARG × over→ start_ARG S end_ARG. The expected returns for observations o→ oover→ start_ARG o end_ARG are given by s→∼B(⋅∣o→)[G⁢(s→)]=(⋅G)⁢(o→) *E_ s B(· o) [G( s)% ]=( B· G)( o)Eover→ start_ARG s end_ARG ∼ B ( ⋅ ∣ over→ start_ARG o end_ARG ) [ G ( over→ start_ARG s end_ARG ) ] = ( B ⋅ G ) ( over→ start_ARG o end_ARG ). We view G∈ℝ→superscriptℝ→G∈ R SG ∈ blackboard_Rover→ start_ARG S end_ARG and ⋅G∈ℝΩ→⋅superscriptℝ→Ω B· G∈ R B ⋅ G ∈ blackboard_Rover→ start_ARG Ω end_ARG as both column vectors and functions. Plugging these expected returns into Eq. (1) gives PR⁢(o→≻o→′)≔σ⁢(β⁢((⋅G)⁢(o→)−(⋅G)⁢(o→′))).≔superscriptsucceeds→superscript→′⋅→⋅superscript→′P^R ( o o 1.0pt ) σ % (β (( B· G)( o)-(% B· G)( o 1.0pt ) ) ).Pitalic_R ( over→ start_ARG o end_ARG ≻ over→ start_ARG o end_ARG′ ) ≔ σ ( β ( ( B ⋅ G ) ( over→ start_ARG o end_ARG ) - ( B ⋅ G ) ( over→ start_ARG o end_ARG′ ) ) ) . (2) This is an instance of reward-rational implicit choice (Jeon et al., 2020), with the function o→↦B(⋅∣o→) o B(· o)over→ start_ARG o end_ARG ↦ B ( ⋅ ∣ over→ start_ARG o end_ARG ) as the grounding function. If observations are deterministic, we can write O→⁢(s→)=o→ O( s)= oover→ start_ARG O end_ARG ( over→ start_ARG s end_ARG ) = over→ start_ARG o end_ARG for o→ oover→ start_ARG o end_ARG with PO→⁢(o→∣s→)=1subscript→conditional→1P_ O( o s)=1Pover→ start_ARG O end_ARG ( over→ start_ARG o end_ARG ∣ over→ start_ARG s end_ARG ) = 1. We can then recover the fully observable case Eq. (1) with BB and O→ Oover→ start_ARG O end_ARG being the identity. The belief B can be any distribution as long as it sums to 1111 over s→ sover→ start_ARG s end_ARG. The human could arrive at such a belief via Bayesian updates, assuming knowledge of P0subscript0P_0P0, TT, POsubscriptP_OPitalic_O, and a prior over the policy that generates the trajectories (see Appendix C.1). None of our results rely on this more detailed model. We assume the human gives feedback according to Eq. (2) but the system uses the standard RLHF algorithm based on Eq. (1). We define the following observation return function GobssubscriptobsG_obsGroman_obs, and we show in Appendix D.1 that if observations are deterministic, RLHF infers this up to an additive constant. Gobs⁢(s→)≔o→∼PO→(⋅∣s→)[(⋅G)⁢(o→)],G_obs( s) *E_ o% P_ O(· s) [ ( B% · G )( o) ],Groman_obs ( over→ start_ARG s end_ARG ) ≔ Eover→ start_ARG o end_ARG ∼ P start_POSTSUBSCRIPT over→ start_ARG O end_ARG ( ⋅ ∣ over→ start_ARG s end_ARG ) end_POSTSUBSCRIPT [ ( B ⋅ G ) ( over→ start_ARG o end_ARG ) ] , (3) For deterministic PO→subscript→P_ OPover→ start_ARG O end_ARG, this can be simplified to Gobs⁢(s→)=(⋅G)⁢(O→⁢(s→))subscriptobs→⋅→G_obs( s)= ( B· G )% ( O( s) )Groman_obs ( over→ start_ARG s end_ARG ) = ( B ⋅ G ) ( over→ start_ARG O end_ARG ( over→ start_ARG s end_ARG ) ) where PO→⁢(O→⁢(s→)∣s→)=1subscript→conditional→1P_ O( O( s) s)=1Pover→ start_ARG O end_ARG ( over→ start_ARG O end_ARG ( over→ start_ARG s end_ARG ) ∣ over→ start_ARG s end_ARG ) = 1. Note that deterministic observations can be ambiguous if multiple states produce the same observation. Unlike in the fully observable case of Proposition 3.1, a return function might be inferred that implies an incorrect set of optimal policies. We define the resulting policy evaluation function JobssubscriptobsJ_obsJroman_obs by Jobs⁢(π)≔s→∼Pπ⁢(s→)[Gobs⁢(s→)].≔subscriptobssubscriptsimilar-to→superscript→subscriptobs→J_obs(π) *E_ s P^% π( s) [G_obs( s) ].Jroman_obs ( π ) ≔ Eover→ start_ARG s end_ARG ∼ Pitalic_π ( over→ start_ARG s end_ARG ) [ Groman_obs ( over→ start_ARG s end_ARG ) ] . (4) This is the function which a standard reinforcement learning algorithm would optimize given the inferred return function GobssubscriptobsG_obsGroman_obs. We summarize this as follows: Proposition 4.1. In partially observable settings with deterministic observations, a policy is optimal according to RLHF, i.e., according to a return function model that would be learned by RLHF with infinite comparison data, if it maximizes JobssubscriptobsJ_obsJroman_obs. Note that in this definition, and specifically in the formula for GobssubscriptobsG_obsGroman_obs, the human does not have knowledge of the policy π that generates the state sequence s→ sover→ start_ARG s end_ARG. In Appendix D.2, we briefly discuss the unrealistic case that the human does know the precise policy and is an ideal Bayesian reasoner over the true environment dynamics. In that case, Jobs=JsubscriptobsJ_obs=JJroman_obs = J, i.e. there is no discrepancy between true and inferred returns. Intuitively, even if the human would not make any observations, they could give correct feedback essentially by estimating the policy’s expected return explicitly. In our case, however, a policy achieving high JobssubscriptobsJ_obsJroman_obs produces state sequences s→ sover→ start_ARG s end_ARG whose observation sequence O→⁢(s→)→ O( s)over→ start_ARG O end_ARG ( over→ start_ARG s end_ARG ) looks good according to the human’s belief B⁢(s→′∣O→⁢(s→))conditionalsuperscript→′→B ( s 1.0pt O( s) )B ( over→ start_ARG s end_ARG′ ∣ over→ start_ARG O end_ARG ( over→ start_ARG s end_ARG ) ). This hints at a possible source of deception: if the policy achieves sequences whose observations look good at the expense of actual value G⁢(s→)→G( s)G ( over→ start_ARG s end_ARG ), we might intuitively call this deceptive behavior. We now analyze this point in greater detail. 4.2 An ontology of behaviors Figure 3: Behaviors defined by increasing and decreasing the human’s over- and underestimation error. RLHF with partial observations results in incentives to increase overestimation error and decrease underestimation error (Theorem 4.5). We will evaluate state sequences based on the extent to which they lead to the human overestimating or underestimating the reward in expectation. Recall that GobssubscriptobsG_obsGroman_obs from Equation 3 measures the expected return from the perspective of a human with some belief function B and access to only observations, whereas G are the true returns. That leads us to the following definition: Definition 4.2 (Overestimation and Underestimation Error). Let s→ sover→ start_ARG s end_ARG be a state sequence. We define its overestimation error E+superscriptE^+E+ and underestimation error E−superscriptE^-E- by E+⁢(s→)≔max⁡(0,Gobs⁢(s→)−G⁢(s→)),≔superscript→0subscriptobs→ E^+( s) (0,G_obs( % s)-G( s) ),E+ ( over→ start_ARG s end_ARG ) ≔ max ( 0 , Groman_obs ( over→ start_ARG s end_ARG ) - G ( over→ start_ARG s end_ARG ) ) , E−⁢(s→)≔max⁡(0,G⁢(s→)−Gobs⁢(s→)).≔superscript→0→subscriptobs→ E^-( s) (0,G( s)-G_% obs( s) ).E- ( over→ start_ARG s end_ARG ) ≔ max ( 0 , G ( over→ start_ARG s end_ARG ) - Groman_obs ( over→ start_ARG s end_ARG ) ) . We further define the average overestimation (underestimation) error under a policy π by E¯+⁢(π)≔s→∼Pπ[E+⁢(s→)]≔superscript¯subscriptsimilar-to→superscriptsuperscript→ E^+(π) *E_ s% P^π[E^+( s)]over¯ start_ARG E end_ARG+ ( π ) ≔ Eover→ start_ARG s end_ARG ∼ Pitalic_π [ E+ ( over→ start_ARG s end_ARG ) ] and E¯−⁢(π)≔s→∼Pπ[E−⁢(s→)]≔superscript¯subscriptsimilar-to→superscriptsuperscript→ E^-(π) *E_ s% P^π[E^-( s)]over¯ start_ARG E end_ARG- ( π ) ≔ Eover→ start_ARG s end_ARG ∼ Pitalic_π [ E- ( over→ start_ARG s end_ARG ) ]. We consider a policy π in comparison to some reference policy πrefsubscriptref _refπref. This can loosely be understood as a counterfactual policy in the absence of some intervention, where π is the factual policy resulting from the intervention. We discuss increases and decreases in over- and underestimation error which are implicitly due to some intervention. For our purposes, πrefsubscriptref _refπref will be the true optimal policy, and π will be the JobssubscriptobsJ_obsJroman_obs-optimal policy; the “intervention” is thus the introduction of partial observability. Figure 3 shows a simple ontology of behaviors that increase and decrease the average over- and underestimation error. Increasing either of these quantities decreases the accuracy of the human’s estimates, and can thus be thought of as “misleading”; decreasing either of them improves accuracy and can be thought of as “informing”. 4.3 Deceptive inflation and overjustification Standard RLHF in the setting of partial observations incentivizes undesirable forms of inflating and justifying. We refer to the philosophical definition of deception offered by Park et al. (2024b), adjustwidth -0.1em-0.1em “the systematic inducement of false beliefs in the pursuit of some outcome other than the truth,” to anchor the notion that increasing the overestimation error in order to improve the RLHF objective JobssubscriptobsJ_obsJroman_obs is deceptive, leading to the following definition. Definition 4.3 (Deceptive Inflation). A policy π exhibits deceptive inflation relative to πrefsubscriptref _refπref if E¯+⁢(π)>E¯+⁢(πref)superscript¯superscript¯subscriptref E^+(π)> E^+( _ref)over¯ start_ARG E end_ARG+ ( π ) > over¯ start_ARG E end_ARG+ ( πref ) and Jobs⁢(π)>Jobs⁢(πref)subscriptobssubscriptobssubscriptref J_obs(π)>J_obs( _% ref)Jroman_obs ( π ) > Jroman_obs ( πref ). We typically prefer that our AI agents engage in informing behaviors. Undesirable informing behaviors decrease reward despite providing information. We name undesirable justifying behaviors “overjustification” as a nod to the overjustification effect from psychology (Deci and Flaste, 1995), in which subjects become dependent on an extrinsic source of motivation to sustain work on a task. Definition 4.4 (Overjustification). A policy π exhibits overjustification relative to πrefsubscriptref _refπref if E¯−⁢(π)<E¯−⁢(πref)superscript¯superscript¯subscriptref E^-(π)< E^-( _ref)over¯ start_ARG E end_ARG- ( π ) < over¯ start_ARG E end_ARG- ( πref ) and J⁢(π)<J⁢(πref)subscriptref J(π)<J( _ref)J ( π ) < J ( πref ). To understand the counterintuitive notion that an agent providing information to the human could be undesirable, consider a PhD student who looks to feedback from their advisor for direction. They meet for one hour a week. Suppose the student explain last week’s work in 15 minutes, leaving the remaining time to discuss next steps. They could instead “overjustify” by spending the entire hour going through the last week’s work in far more detail, leaving no time for next steps. From the advisor’s perspective, the latter is more informative, but is a worse allocation of limited resources. We now state a key result. See Section D.3 for the proof. Theorem 4.5. Assume that POsubscriptP_OPitalic_O is deterministic. Let Πobs∗subscriptsuperscriptΠobs ^*_obsΠ∗roman_obs be the set of optimal policies according to a naive application of RLHF under partial observability, and let Π∗superscriptΠ ^*Π∗ be the set of optimal policies according to the true objective J. If π∗∈Π∗∖Πobs∗superscriptsuperscriptΠsubscriptsuperscriptΠobsπ^*∈ ^* ^*_obsπ∗ ∈ Π∗ ∖ Π∗roman_obs and πobs∗∈Πobs∗∖Π∗subscriptsuperscriptobssubscriptsuperscriptΠobssuperscriptΠπ^*_obs∈ ^*_obs ^*π∗roman_obs ∈ Π∗roman_obs ∖ Π∗, then πobs∗subscriptsuperscriptobsπ^*_obsπ∗roman_obs must exhibit at least one of deceptive inflation or overjustification relative to π∗superscriptπ^*π∗. Note that a trajectory s→ sover→ start_ARG s end_ARG may be more or less likely under πobs∗subscriptsuperscriptobsπ^*_obsπ∗roman_obs than π∗superscriptπ^*π∗, regardless of human estimation, so long as on net πobs∗subscriptsuperscriptobsπ^*_obsπ∗roman_obs exhibits deceptive inflation or overjustification. Our analysis extends beyond the special case of RLHF to inverse preference learning (IPL) (Hejna and Sadigh, 2023), and thus to direct preference optimization (DPO) (Rafailov et al., 2023), which IPL generalizes. Theorem 1 in Hejna and Sadigh (2023) shows that IPL will converge to a policy that maximizes an implicit reward function that matches the human’s preference judgments as well as possible. If the human’s preference judgments come from partial observations, then the resulting return function will be GobssubscriptobsG_obsGroman_obs, as we describe in our discussion leading up to Proposition 4.1. This leads to the same problems of deceptive inflation and overjustification that we describe in Theorem 4.5. 4.4 Deception and overjustification in examples Figure 4: Scenarios illustrating failure modes due to partial observability. In each, the agent must install two packages. Formal details of the underlying MDPs are provided in Appendix B. A, top: In the absence of a log message about CUDA, the human is unsure whether the agent skipped it or used the 2> /dev/null trick (see Figure 2); if the human is insufficiently skeptical, the trick looks optimal to the agent. B, bottom: Default logging in this case is silent when the NumPy install is successful. The agent can optionally use a ----verbose flag, but this produces a long log that the human prefers not to see. If the human is too skeptical, verbose logging still appears optimal to the agent. We now analyze intentionally simple examples that illustrate the failure modes of RLHF in the presence of partial observability. Our examples feature an agent trained with RLHF to complete tasks in a user’s terminal. The output of each command is piped to a log file, which is what the human evaluator sees when making choices for RLHF. For largely mathematical details about these examples, including a tutorial-style analysis of Example A with derivations, see Appendix B. For experimental details, see Section B.5. For a proof of concept on how to mitigate the issues, see Section 5.2. In Fig. 4A, the agent’s task is to install Nvidia drivers and CUDA. The agent may skip the CUDA installation, or may attempt it with or without 2> /dev/null (which suppresses errors) appended. If the log contains no message about CUDA, the human believes with probability phidesubscripthidep_hidephide that the agent used the 2> /dev/null trick and the install failed, and with probability 1−phide1subscripthide1-p_hide1 - phide that the agent didn’t attempt the CUDA install. If phidesubscripthidep_hidephide is sufficiently small, then RLHF will learn from the resulting feedback that the optimal policy is to use the 2> /dev/null trick. This is deceptive inflation, because this policy produces trajectories for which the human systematically overestimates the return. In Figure 5(a), we see that the threshold for phidesubscripthidep_hidephide at which the AI switches from a deceptive action (2> /dev/null ) to installing CUDA with default logging perfectly agrees with the theoretical prediction made in Example B.1. In Fig. 4B, the agent’s task is to install Python and NumPy. The agent can skip the NumPy install, or attempt it with either “default” settings that only logs errors or with a ----verbose flag that adds additional info. In the case of a success, the human prefers not to see this verbose output. If the log contains no message about NumPy, the human believes with probability pdefaultsubscriptdefaultp_defaultpdefault that the agent attempted the install with default logging and succeeded, and with probability 1−pdefault1subscriptdefault1-p_default1 - pdefault that the agent skipped the NumPy install. If pdefaultsubscriptdefaultp_defaultpdefault is sufficiently small, then RLHF will learn from the resulting feedback that the optimal policy is to use ----verbose logging. This is overjustification, because the agent chooses a suboptimal policy in order to avoid the human underestimating its performance. In Figure 5(b), we see that the threshold for pdefaultsubscriptdefaultp_defaultpdefault at which the AI switches from verbose loggin to skipping the whole NumPy installation largely agrees with the theoretical prediction made in Example B.3. (a) Example A (b) Example B Figure 5: Example A: The larger the reward penalty for hiding errors with 2> /dev/null , and the larger the human’s belief that the agent used 2> /dev/null upon seeing an empty log (phidesubscripthidep_hidephide), the more we expect the agent to install CUDA with default logging in Example A. In Example B.1, we compute a precise theoretical threshold where the behavior should switch. This perfectly agrees with empirical findings. Example B: The larger the reward penalty for verbose logging, and the larger the human’s trust that the agent installed NumPy upon seeing an empty log (pdefaultsubscriptdefaultp_defaultpdefault), the more we expect the agent to skip the NumPy installation entirely. In Example B.3, we compute a precise theoretical threshold where behavior should switch. Except four cases of “verbose logging” where the theory predicted the agent to skip the NumPy installation, this agrees with empirical findings. See Section B.5 for experimental details. Further examples. We show further, purely mathematical, examples in Appendix D.4. Example D.6 shows that deceptiveness and overjustifying behavior even applies to aspects of the trajectory the policy has no control over: The policy tries to “hide bad luck” and “reveal good luck at a cost”. Example D.7, especially (a) and (c), shows that the policies coming out of a naive application of RLHF under partial observability may be suboptimal with positive E¯−superscript¯ E^-over¯ start_ARG E end_ARG- (and zero E¯+superscript¯ E^+over¯ start_ARG E end_ARG+) or optimal, but with positive E¯+superscript¯ E^+over¯ start_ARG E end_ARG+ (and zero E¯−superscript¯ E^-over¯ start_ARG E end_ARG-). Thus, there can be suboptimality even if the policy is better than it seems, and optimality even when the policy is worse than it seems. 5 Return ambiguity from feedback under known partial observability We’ve seen issues with standard RLHF applied to feedback from partial observations. Part of the problem is model misspecification: the standard RLHF model implicitly assumes full observability. Assuming the human’s partial observability is known, could one do better? We start Section 5.1 by analyzing how much information the feedback process provides about the return function when the human’s choice model under partial observations is known precisely. We show that the feedback determines the correct return function up to an additive constant and a linear subspace we call the ambiguity (Theorem 5.2). If the human had a return function that differed from the true return function by an element in the ambiguity, they would give the exact same feedback — such return functions are thus feedback-compatible. We then show an example where the ambiguity vanishes, and another where it doesn’t, leading to feedback-compatible return functions that have optimal policies with high regret under the true return function. Finally, in Section 5.2 we explore how one could in theory use Theorem 5.2 as a starting point to design reward learning techniques that work under partial observability. In particular, we experimentally show in a proof of concept that being aware of the human’s partial observability improves performance. In this section we do not assume POsubscriptP_OPitalic_O to be deterministic. 5.1 Feedback-compatibility and ambiguity of return functions Assume that the human gives feedback based on the choice-probabilities from Eq. (2). In the infinite data limit, it can be assumed that the whole collection of probabilities (PG⁢(o→≻o→′))o→,o→′subscriptsuperscriptsucceeds→superscript→′→superscript→′ (P^G ( o o ) )_ o, o% ( Pitalic_G ( over→ start_ARG o end_ARG ≻ over→ start_ARG o end_ARG′ ) )over→ start_ARG o end_ARG , over→ start_ARG o end_ARG′ is known since the choice frequencies approach these probabilities. Here, we write PGsuperscriptP^GPitalic_G instead of PRsuperscriptP^RPitalic_R since the reward function only enters the choice probabilities through the corresponding return function G. The question we answer in this section is how much information the choice probabilities provide about G, assuming the human choice model is known and correct. The choice probabilities tell us precisely that the true return function gives rise to these choice probabilities, i.e., is feedback-compatible. This is captured in the following definition: Definition 5.1. Let (PG⁢(o→≻o→′))o→,o→′subscriptsuperscriptsucceeds→superscript→′→superscript→′ (P^G ( o o 1.0pt ) )_% o, o 1.0pt ( Pitalic_G ( over→ start_ARG o end_ARG ≻ over→ start_ARG o end_ARG′ ) )over→ start_ARG o end_ARG , over→ start_ARG o end_ARG′ be the vector of choice probabilities and G~~ Gover~ start_ARG G end_ARG a return function corresponding to a reward function R~~ Rover~ start_ARG R end_ARG. Then G~~ Gover~ start_ARG G end_ARG is feedback-compatible (with respect to the vector of choice probabilities) if PG~⁢(o→≻o→′)=PG⁢(o→≻o→′)superscript~succeeds→superscript→′succeeds→superscript→′P G( o o 1.0pt )=P^G( o % o 1.0pt )Pover~ start_ARG G end_ARG ( over→ start_ARG o end_ARG ≻ over→ start_ARG o end_ARG′ ) = Pitalic_G ( over→ start_ARG o end_ARG ≻ over→ start_ARG o end_ARG′ ) for all o→,o→′∈Ω→superscript→′→Ω o, o 1.0pt ∈ over→ start_ARG o end_ARG , over→ start_ARG o end_ARG′ ∈ over→ start_ARG Ω end_ARG. Crucially, without further assumptions or inductive biases, no learning algorithm can pick out the true return function among feedback-compatible return functions. It is thus crucial to know whether there are feedback-compatible return functions that are unsafe when using them to optimize a policy. Figure 6: By Theorem 5.2, even with infinite comparison data and access to the correct human model, a hypothetical reward learning system (depicted as a robot) could only infer G up to the ambiguity im⁡∩ker⁡imkernelim ∩ Bim Γ ∩ ker B (purple). Adding an element of the ambiguity to G leads to the exact same choice probabilities for all possible comparisons, and the reward learning system has no way to identify G among the return functions in G+(im⁡∩ker⁡)imkernelG+(im ∩ % B)G + ( im Γ ∩ ker B ) (yellow). This abstract depiction ignores the linearity of these spaces; for a more precise geometric depiction of BB, see Figure 9 in the appendix. We now determine the set of feedback-compatible return functions. Write ∈ℝ→×superscriptℝ→ ∈ R S×SΓ ∈ blackboard_Rover→ start_ARG S end_ARG × S for the matrix that maps a reward function to its return function, i.e. (⋅R)⁢(s→)≔∑t=0Tγt⁢R⁢(st)≔⋅→superscriptsubscript0superscriptsubscript( · R)( s) _t=0^Tγ^% tR(s_t)( Γ ⋅ R ) ( over→ start_ARG s end_ARG ) ≔ ∑t = 0T γitalic_t R ( sitalic_t ). Its matrix elements are given by s→⁢s=∑t=0Tδs⁢(st)⁢γtsubscript→superscriptsubscript0subscriptsubscriptsuperscript _ ss= _t=0^T _s(s_t)% γ^tΓover→ start_ARG s end_ARG s = ∑t = 0T δitalic_s ( sitalic_t ) γitalic_t, where δs⁢(st)=⁢s=stsubscriptsubscript1subscript _s(s_t)=1\s=s_t\δitalic_s ( sitalic_t ) = 1 s = sitalic_t . Then the image im⁡imim im Γ is the set of all return functions that can be realized from a reward function given the MDP dynamics TT. Recall the belief matrix =(B⁢(s→∣o→))o→,s→∈ℝΩ→×→subscriptconditional→superscriptℝ→Ω→ B= (B( s o) )_ o, s% ∈ R × SB = ( B ( over→ start_ARG s end_ARG ∣ over→ start_ARG o end_ARG ) )over→ start_ARG o end_ARG , over→ start_ARG s end_ARG ∈ blackboard_Rover→ start_ARG Ω end_ARG × over→ start_ARG S end_ARG. Taking into account that G itself is in im⁡imim im Γ and that G enters the choice probabilities only through ⋅G⋅ B· GB ⋅ G — meaning that the choice probabilities do not vary if we change G additively up to an element in the kernel ker⁡kernel Bker B — we obtain the following result: Theorem 5.2. Let the collection of choice probabilities be given by (PR⁢(o→≻o→′))o→,o→′∈Ω→subscriptsuperscriptsucceeds→superscript→′→superscript→′→Ω (P^R ( o o 1.0pt ) )_% o, o 1.0pt ∈ ( Pitalic_R ( over→ start_ARG o end_ARG ≻ over→ start_ARG o end_ARG′ ) )over→ start_ARG o end_ARG , over→ start_ARG o end_ARG′ ∈ over→ start_ARG Ω end_ARG following a Boltzmann rational model as in Eq. (2). Then a return function G~~ Gover~ start_ARG G end_ARG is feedback-compatible if and only if there is G′∈ker⁡∩im⁡superscript′kernelimG ∈ B % G′ ∈ ker B ∩ im Γ and c∈ℝc∈ Rc ∈ blackboard_R such that G~=G+G′+c~superscript′ G=G+G +cover~ start_ARG G end_ARG = G + G′ + c. In particular, the choice probabilities determine G up to an additive constant if and only if ker⁡∩im⁡=0kernelim0 B % =\0\ker B ∩ im Γ = 0 . See Theorem C.2 and Corollary C.4 for full proofs, and Figure 6 for a visual depiction. This result motivates the following definition: Definition 5.3 (Ambiguity). We call ker⁡∩im⁡kernelim B ker B ∩ im Γ the ambiguity that is left in the return function when the human choice model and observation-based choice probabilities are known. Note that Theorem 5.2 generalizes the fully observed case from Section 3.2 (Corollary C.10). We extend the theorem in Appendix C.4 to the case when the human’s observations are not known. Special cases of ker⁡kernel Bker B and im⁡imim im Γ and our theorem can be found in Appendices C.7 and C.5. In particular, if PO→subscript→P_ OPover→ start_ARG O end_ARG is stochastic and there is only “noise” in it (defined as Ω→=→Ω→ = Sover→ start_ARG Ω end_ARG = over→ start_ARG S end_ARG and the injectivity of OO) and if the human is a Bayesian reasoner with a fully supported prior over → Sover→ start_ARG S end_ARG, then the choice data determines the return function even if the human’s observations are not known; see Example C.30. Connection to Potential Shaping Under typical technical assumptions (Skalse et al., 2023; Jenner et al., 2022; Ng et al., 1999), potential shaping changes the returns only by an additive constant, and thus never changes optimal policies. It is a non-trivial ambiguity only in the reward function, but it does not lead to actual ambiguity about intended behavior. In contrast, the ambiguity ker⁡∩im⁡kernelim B ker B ∩ im Γ in the return function that we study can make the optimal policy ambiguous — it reflects genuine missing information about the intentions of the human. How large is the return ambiguity? For Fig. 4A, one can show that the ambiguity is nontrivial, allowing for feedback-compatible return functions with unsafe optimal policies. Intuitively, since successfully installing CUDA produces the same observation regardless of whether 2> /dev/null was used, the choice probabilities don’t give us any information to determine distinct reward values for these two outcomes, only their average over the human’s belief upon observing a successful install. Thus, reward functions assigning arbitrarily high reward to success with 2> /dev/null are feedback-compatible. Such reward functions can then lead to an incentive for a learned policy to hide the error messages even with a correct observation model. More details can be found in Section B.4. We saw in Fig. 4B a case where naive RLHF under partial observability can lead to overjustification. However, the human’s feedback and belief model actually provide enough information to determine the return function. The reason is that ker⁡kernel Bker B leaves only one degree of freedom that is not “time-separable” over states, and thus ker⁡∩im⁡kernelim B ker B ∩ im Γ = 0. More details can be found in Section B.4. 5.2 Toward improving RLHF in partially observable settings To improve RLHF when partial observability is unavoidable, one could take Theorem 5.2 as a starting point to find a learning algorithm that converges to feedback-compatible return functions. This would require the human model to be fully known and specified, including knowledge of the belief probabilities B⁢(s→∣o→)conditional→B( s o)B ( over→ start_ARG s end_ARG ∣ over→ start_ARG o end_ARG ), which can differ from human to human. If one assumes the human is rational, as in Appendix C.1, this requires specifying the human’s policy prior B⁢(π)B(π)B ( π ). Instead of directly specifying these models, one could also attempt to learn a generative model for B⁢(s→∣o→)conditional→B( s o)B ( over→ start_ARG s end_ARG ∣ over→ start_ARG o end_ARG ). These problems reveal a further conceptual challenge: for complex environments, humans do not form beliefs over the entire environment state s. A better starting point for practical work may thus be to model humans as forming expectations over reward-relevant features of the state. If BB were explicitly known, one could in principle encode BB into the loss function of an adapted RLHF process to learn a feedback-compatible return function; see Section C.3. As a proof of concept, we used this procedure to analyze the examples in Figure 4 empirically, see Table 1. We do this by first learning a reward model by logistic regression against the true choice probabilities of a synthetic human under partial observability, and then learning the optimal Q-function of the resulting reward model with value iteration. The resulting policy chooses a unique action after installation of the nvidia driver (Example A) or Python (Example B) as listed in the “action” column. Table 1 shows that in 3 of four cases, being “partial observability aware” (“po-aware”) leads to the true optimal policy when “naive” RLHF does not. In the one case where being “po-aware” does not improve performance (second line in the table), this is explained by the fact that there is remaining ambiguity in the return function. Curiously, in line 4 our theory also predicts remaining ambiguity, but the optimal policy is learned; we consider this to be luck. We provide more details on our experiments in Section B.5. Table 1: Experiments showing improved performance of po-aware RLHF Ex. p phidesubscripthidep_hidephide pdefaultsubscriptdefaultp_defaultpdefault model action E¯+superscript¯ E^+over¯ start_ARG E end_ARG+ dec. infl. E¯−superscript¯ E^-over¯ start_ARG E end_ARG- overj. optimal A 0.5 0.5 N/A naive aHsubscripta_Haitalic_H 1.5 ✓ 0 × × A 0.5 0.5 N/A po-aware aHsubscripta_Haitalic_H 1.5 ✓ 0 × × A 0.1 0.9 N/A naive aCsubscripta_Caitalic_C 0 × 0 ✓ × A 0.1 0.9 N/A po-aware aTsubscripta_Taitalic_T 0 × 5.4 × ✓ B 0.5 N/A 0.9 naive aTsubscripta_Taitalic_T 4.5 ✓ 0 ✓ × B 0.5 N/A 0.9 po-aware aDsubscripta_Daitalic_D 0 × 0.25 × ✓ B 0.5 N/A 0.1 naive aVsubscripta_Vaitalic_V 0 × 0 ✓ × B 0.5 N/A 0.1 po-aware aDsubscripta_Daitalic_D 0 × 2.25 × ✓ As we already demonstrated, feedback-compatible return functions can be unsafe due to remaining ambiguity. In Example C.29, we even show a case where some feedback-compatible return functions have optimal policies that are even worse than simply maximizing JobssubscriptobsJ_obsJroman_obs. An important direction for future work is to investigate learning algorithms and inductive biases that help “find” safe return functions among all those that are feedback-compatible, or that act conservatively given the uncertainty. Another line of inquiry is to determine when the set of feedback-compatible return functions is “safe”, which depends on the MDP, observation function, and human model. One sufficient condition for feedback-compatible return functions to be safe is the vanishing of the ambiguity ker⁡∩im⁡kernelim B ker B ∩ im Γ. Even then, one realistically still has to deal with the problem that BB is at best known approximately. Fortunately, in Appendix C.6, we prove that small errors in the assumed belief matrix lead to only small errors in the inferred return function: Theorem 5.4. Assume ker⁡∩im⁡=0kernelim0 B % =\0\ker B ∩ im Γ = 0 . Let ≔+≔subscript B_ % B+ Bbold_Δ ≔ B + Δ be a small perturbation of BB, where ‖≤ρnorm\| \|≤ρ∥ Δ ∥ ≤ ρ for sufficiently small ρ. Let G be the true return function and assume that a hypothetical learning system, assuming the human’s belief is subscript B_ Bbold_Δ, infers the return function G~~ Gover~ start_ARG G end_ARG with the property that ⋅G~⋅subscript~ B_ · GBbold_Δ ⋅ over~ start_ARG G end_ARG has the smallest possible Euclidean distance to ⋅G⋅ B· GB ⋅ G. Let r⁢()≔|im⁡≔revaluated-atimr( B) B|_% im r ( B ) ≔ B |im Γ be the (injective) restriction of the operator BB to im⁡imim im Γ. Then r⁢()T⁢r⁢()rsuperscriptrr( B)^Tr( B)r ( B )T r ( B ) is invertible, and there exists a polynomial Q⁢(X,Y)Q(X,Y)Q ( X , Y ) of degree 5555 such that ‖G~−G‖≤ρ⋅‖G‖⋅Q⁢(‖(r⁢()T⁢r⁢())−1‖,‖r⁢()‖).norm~⋅normnormsuperscriptrsuperscriptr1normr\| G-G\|≤ρ·\|G\|· Q ( \| (r(% B)^Tr( B) )^-1% \|,\|r( B)\| ).∥ over~ start_ARG G end_ARG - G ∥ ≤ ρ ⋅ ∥ G ∥ ⋅ Q ( ∥ ( r ( B )T r ( B ) )- 1 ∥ , ∥ r ( B ) ∥ ) . In particular, as we show in the appendix, one can uniformly bound the difference between JG~subscript~J_ GJover~ start_ARG G end_ARG and JGsubscriptJ_GJitalic_G. This yields a regret bound between the policy optimal under G~~ Gover~ start_ARG G end_ARG and an optimal policy π∗superscriptπ^*π∗ for G. There are also alternatives to modeling the human belief BB. For example, one could mix human evaluations based on high-cost full observations and low-cost partial observations for finding an optimal tradeoff (Mallen and Belrose, 2024). Finally, it would help if the human could query the policy about reward-relevant aspects of the environment to bring the setting closer to RLHF from full observations. This is similar to the problem of eliciting the latent knowledge of a predictor of future observations (Christiano et al., 2021; Christiano and Xu, 2022). While this may avoid the need to specify the human’s belief model B⁢(s→∣o→)conditional→B( s o)B ( over→ start_ARG s end_ARG ∣ over→ start_ARG o end_ARG ), it requires understanding and effectively querying an ML model’s belief, including translating from an ML model’s ontology into a human ontology. 6 Conclusions In this paper, we provided an investigation of challenges when applying RLHF from partial observations. First, we saw that applying RLHF naively when assuming full observability can lead to deceptive inflation and overjustification behavior. Then, we showed that even when the human’s partial observability is known, the set of feedback-compatible return functions can contain irreducible ambiguity. This means that without further inductive biases, no learning algorithm can generally be expected to infer the correct return function. Finally, we recommended further exploratory research to study and improve RLHF for cases when partial observability is unavoidable and provided a proof of concept that modeling the human’s partial observability can improve performance. In conclusion, we recommend caution when using RLHF in situations of partial observability, and hope that further research studies the effects in practice and helps to address these challenges. Limitations We assume the human to be Boltzmann rational and to implicitly compute an expected value of the return, which is unrealistic for actual humans. Other types of choices could be considered, as in reward-rational choice (Jeon et al., 2020) and assistance games (Hadfield-Menell et al., 2016). Finally, we assume that the human forms a belief B⁢(s→∣o→)conditional→B( s o)B ( over→ start_ARG s end_ARG ∣ over→ start_ARG o end_ARG ) over the true state sequence s→ sover→ start_ARG s end_ARG. If the environment is complex, humans will in reality only form beliefs over lower-dimensional representations or features of the state. Impact statement RLHF and its variants are widely used to steer the behavior of language models. Thus, the soundness of RLHF is critical to language models’ trustworthy deployment. Our work shows that partial observability of humans poses safety challenges for RLHF. We hope our work stimulates further research to study and overcome these challenges, or that it incentivizes researchers to ensure that their human evaluators fully observe the environment state. Author contributions The project was conceived in parallel by Scott and Davis, with a key shift proposed by Leon. Leon proved Propositions 4.1, 5.2 and 5.4, found the first mathematical examples of what became deceptive inflation and overjustification that can be resolved by Theorem 5.2, and wrote the majority of the appendix. Davis conjectured Proposition 4.1, provided early empirical evidence that RLHF under partial observations can lead to deception (not in the paper), defined deception / deceptive inflation and overjustification (with Scott), proved Theorem 4.5, and developed the running examples and figures. Scott guided the project direction and prioritization, gave the conjecture and proof idea for Theorem 5.4, and helped develop the running examples and deception definitions. Erik provided regular detailed feedback and guidance and edited the paper. Anca and Stuart advised this project. Acknowledgments and Disclosure of Funding Leon Lang thanks the Center for Human-Compatible Artificial Intelligence for hosting him during part of this project, and Open Philanthropy for financial support. All authors thank Open Philanthropy for its support of the Center for Human-Compatible Artificial Intelligence. Davis was supported by the Berkeley Existential Risk Initiative. Erik was supported by fellowships from the Future of Life Institute and Open Philanthropy. We thank Benjamin Eysenbach and Benjamin Plaut for detailed comments and feedback on this work, and we thank Elio A. Farina, Mary Marinou, and Alexandra Horn for assistance with graphic design. References Amodei et al. [2017] D. Amodei, P. Christiano, and A. Ray. Learning from human preferences. https://openai.com/research/learning-from-human-preferences, 2017. Accessed: 2023-12-13. Anthropic [2023a] Anthropic. Introducing Claude. https://w.anthropic.com/index/introducing-claude, 2023a. Accessed: 2023-09-05. Anthropic [2023b] Anthropic. Claude’s Constitution. https://w.anthropic.com/index/claudes-constitution, 2023b. Accessed: 2023-09-05. Bai et al. [2022] Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, C. Chen, C. Olsson, C. Olah, D. Hernandez, D. Drain, D. Ganguli, D. Li, E. Tran-Johnson, E. Perez, J. Kerr, J. Mueller, J. Ladish, J. Landau, K. Ndousse, K. Lukosuite, L. Lovitt, M. Sellitto, N. Elhage, N. Schiefer, N. Mercado, N. DasSarma, R. Lasenby, R. Larson, S. Ringer, S. Johnston, S. Kravec, S. El Showk, S. Fort, T. Lanham, T. Telleen-Lawton, T. Conerly, T. Henighan, T. Hume, S. R. Bowman, Z. Hatfield-Dodds, B. Mann, D. Amodei, N. Joseph, S. McCandlish, T. Brown, and J. Kaplan. Constitutional AI: Harmlessness from AI Feedback. arXiv e-prints, art. arXiv:2212.08073, Dec. 2022. doi: 10.48550/arXiv.2212.08073. Bradley and Terry [1952] R. A. Bradley and M. E. Terry. Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika, 39(3/4):324–345, 1952. ISSN 00063444. URL http://w.jstor.org/stable/2334029. Buehler et al. [1994] R. Buehler, D. Griffin, and M. Ross. Exploring the "Planning Fallacy": Why People Underestimate Their Task Completion Times. Journal of Personality and Social Psychology, 67:366–381, 09 1994. doi: 10.1037/0022-3514.67.3.366. Burns et al. [2023] C. Burns, H. Ye, D. Klein, and J. Steinhardt. Discovering Latent Knowledge in Language Models Without Supervision. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=ETKGuby0hcs. Casper et al. [2023] S. Casper, X. Davies, C. Shi, T. K. Gilbert, J. Scheurer, J. Rando, R. Freedman, T. Korbak, D. Lindner, P. Freire, T. Wang, S. Marks, C.-R. Segerie, M. Carroll, A. Peng, P. Christoffersen, M. Damani, S. Slocum, U. Anwar, A. Siththaranjan, M. Nadeau, E. J. Michaud, J. Pfau, D. Krasheninnikov, X. Chen, L. Langosco, P. Hase, E. Bıyık, A. Dragan, D. Krueger, D. Sadigh, and D. Hadfield-Menell. Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback. arxiv e-prints, 2023. Chidambaram et al. [2024] K. Chidambaram, K. V. Seetharaman, and V. Syrgkanis. Direct Preference Optimization With Unobserved Preference Heterogeneity, 2024. URL https://arxiv.org/abs/2405.15065. Christiano and Xu [2022] P. Christiano and M. Xu. ELK prize results. https://w.alignmentforum.org/posts/zjMKpSB2Xccn9qi5t/elk-prize-results, 2022. Accessed: 2024-02-15. Christiano et al. [2017] P. Christiano, J. Leike, T. B. Brown, M. Martic, S. Legg, and D. Amodei. Deep Reinforcement Learning from Human Preferences. arXiv e-prints, art. arXiv:1706.03741, June 2017. doi: 10.48550/arXiv.1706.03741. Christiano et al. [2021] P. Christiano, A. Cotra, and M. Xu. Eliciting Latent Knowledge. https://docs.google.com/document/d/1WwsnJQstPq91_Yh-Ch2XRL8H_EpsnjrC1dwZXR37PC8/edit, 2021. Accessed: 2023-04-25. Deci and Flaste [1995] E. L. Deci and R. Flaste. Why we do what we do: The dynamics of personal autonomy. GP Putnam’s Sons, 1995. Denison et al. [2024] C. Denison, M. MacDiarmid, F. Barez, D. Duvenaud, S. Kravec, S. Marks, N. Schiefer, R. Soklaski, A. Tamkin, J. Kaplan, B. Shlegeris, S. R. Bowman, E. Perez, and E. Hubinger. Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models, 2024. URL https://arxiv.org/abs/2406.10162. El Ghaoui [2002] L. El Ghaoui. Inversion error, condition number, and approximate inverses of uncertain matrices. Linear Algebra and its Applications, 343-344:171–193, 2002. ISSN 0024-3795. doi: https://doi.org/10.1016/S0024-3795(01)00273-7. URL https://w.sciencedirect.com/science/article/pii/S0024379501002737. Special Issue on Structured and Infinite Systems of Linear equations. Evans et al. [2015] O. Evans, A. Stuhlmueller, and N. D. Goodman. Learning the Preferences of Ignorant, Inconsistent Agents. arxiv e-prints, 2015. Evans et al. [2021] O. Evans, O. Cotton-Barratt, L. Finnveden, A. Bales, A. Balwit, P. Wills, L. Righetti, and W. Saunders. Truthful AI: Developing and Governing AI that does not lie. arxiv e-prints, 2021. Fern et al. [2014] A. Fern, S. Natarajan, K. Judah, and P. Tadepalli. A Decision-Theoretic Model of Assistance. J. Artif. Int. Res., 50(1):71–104, may 2014. ISSN 1076-9757. Geiger et al. [1990] D. Geiger, T. Verma, and J. Pearl. Identifying independence in bayesian networks. Networks, 20:507–534, 1990. URL https://api.semanticscholar.org/CorpusID:1938713. Gemini Team [2023] G. Gemini Team. Gemini: A Family of Highly Capable Multimodal Models. https://storage.googleapis.com/deepmind-media/gemini/gemini_1_report.pdf, 2023. Accessed: 2023-12-11. Hadfield-Menell et al. [2016] D. Hadfield-Menell, A. Dragan, P. Abbeel, and S. Russell. Cooperative Inverse Reinforcement Learning. arXiv e-prints, art. arXiv:1606.03137, June 2016. doi: 10.48550/arXiv.1606.03137. Hejna and Sadigh [2023] J. Hejna and D. Sadigh. Inverse Preference Learning: Preference-based RL without a Reward Function. arXiv e-prints, art. arXiv:2305.15363, May 2023. doi: 10.48550/arXiv.2305.15363. Hofstätter et al. [2023] F. Hofstätter, F. R. Ward, HarrietW, L. Thomson, O. J, P. Bartak, and S. F. Brown. Tall Tales at Different Scales: Evaluating Scaling Trends for Deception in Language Models. https://w.alignmentforum.org/posts/pip63HtEAxHGfSEGk/tall-tales-at-different-scales-evaluating-scaling-trends-for, 2023. Accessed: 2024-01-23. Huang et al. [2023] L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, X. Feng, B. Qin, et al. A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions. arXiv preprint arXiv:2311.05232, 2023. Hubinger et al. [2019] E. Hubinger, C. van Merwijk, V. Mikulik, J. Skalse, and S. Garrabrant. Risks from Learned Optimization in Advanced Machine Learning Systems. arXiv e-prints, art. arXiv:1906.01820, June 2019. doi: 10.48550/arXiv.1906.01820. Jenner et al. [2022] E. Jenner, H. van Hoof, and A. Gleave. Calculus on MDPs: Potential Shaping as a Gradient, 2022. Jeon et al. [2020] H. J. Jeon, S. Milli, and A. Dragan. Reward-rational (implicit) choice: A unifying formalism for reward learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 4415–4426. Curran Associates, Inc., 2020. URL https://proceedings.neurips.c/paper_files/paper/2020/file/2f10c1578a0706e06b6d7db6f0b4a6af-Paper.pdf. Kausik et al. [2024] C. Kausik, M. Mutti, A. Pacchiano, and A. Tewari. A Theoretical Framework for Partially Observed Reward-States in RLHF, 2024. URL https://arxiv.org/abs/2402.03282. Lin et al. [2022] S. Lin, J. Hilton, and O. Evans. TruthfulQA: Measuring How Models Mimic Human Falsehoods. arxiv e-prints, 2022. Majumdar et al. [2017] A. Majumdar, S. Singh, A. Mandlekar, and M. Pavone. Risk-sensitive inverse reinforcement learning via coherent risk models. In N. Amato, S. Srinivasa, N. Ayanian, and S. Kuindersma, editors, Robotics, Robotics: Science and Systems, United States, 2017. MIT Press Journals. doi: 10.15607/rss.2017.xiii.069. Mallen and Belrose [2024] A. Mallen and N. Belrose. Balancing Label Quantity and Quality for Scalable Elicitation, 2024. URL https://arxiv.org/abs/2410.13215. Manyika [2023] J. Manyika. An overview of Bard: an early experiment with generative AI. https://ai.google/static/documents/google-about-bard.pdf, 2023. Accessed: 2023-09-05. Mindermann and Armstrong [2018] S. Mindermann and S. Armstrong. Occam’s Razor is Insufficient to Infer the Preferences of Irrational Agents. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS’18, page 5603–5614, Red Hook, NY, USA, 2018. Curran Associates Inc. Mu et al. [2024] T. Mu, A. Helyar, J. Heidecke, J. Achiam, A. Vallone, I. Kivlichan, M. Lin, A. Beutel, J. Schulman, and L. Weng. Rule Based Rewards for Language Model Safety, 2024. URL https://cdn.openai.com/rule-based-rewards-for-language-model-safety.pdf. Accessed: 2024-10-28. Ng et al. [1999] A. Y. Ng, D. Harada, and S. J. Russell. Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping. In Proceedings of the Sixteenth International Conference on Machine Learning, ICML ’99, page 278–287, San Francisco, CA, USA, 1999. Morgan Kaufmann Publishers Inc. ISBN 1558606122. Ng et al. [2000] A. Y. Ng, S. Russell, et al. Algorithms for Inverse Reinforcement Learning. In ICML, volume 1, page 2, 2000. OpenAI [2022] OpenAI. Introducing ChatGPT. https://openai.com/blog/chatgpt, 2022. Accessed: 2024-02-06. OpenAI [2023] OpenAI. ChatGPT Plugins. https://openai.com/index/chatgpt-plugins/, 2023. Accessed: 2024-05-22. OpenAI [2024a] OpenAI. OpenAI o1 System Card, 2024a. URL https://cdn.openai.com/o1-system-card.pdf. Accessed: 2024-10-28. OpenAI [2024b] OpenAI. Model Spec, 2024b. URL https://cdn.openai.com/spec/model-spec-2024-05-08.html. Accessed: 2024-10-28. OpenAI [2024c] OpenAI. Privacy Policy. https://openai.com/policies/privacy-policy//, 2024c. Accessed: 2024-05-22. Park et al. [2024a] C. Park, M. Liu, D. Kong, K. Zhang, and A. Ozdaglar. RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation, 2024a. URL https://arxiv.org/abs/2405.00254. Park et al. [2024b] P. S. Park, S. Goldstein, A. O’Gara, M. Chen, and D. Hendrycks. Ai deception: A survey of examples, risks, and potential solutions. Patterns, 5(5), 2024b. Rafailov et al. [2023] R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. arxiv e-prints, 2023. Scheurer et al. [2023] J. Scheurer, M. Balesni, and M. Hobbhahn. Technical Report: Large Language Models can Strategically Deceive their Users when Put Under Pressure. arxiv e-prints, 2023. Shah et al. [2019] R. Shah, D. Krasheninnikov, J. Alexander, P. Abbeel, and A. Dragan. The Implicit Preference Information in an Initial State. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=rkevMnRqYQ. Shah et al. [2021] R. Shah, P. Freire, N. Alex, R. Freedman, D. Krasheninnikov, L. Chan, M. D. Dennis, P. Abbeel, A. Dragan, and S. Russell. Benefits of Assistance over Reward Learning, 2021. URL https://openreview.net/forum?id=DFIoGDZejIB. Siththaranjan et al. [2023] A. Siththaranjan, C. Laidlaw, and D. Hadfield-Menell. Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF. arXiv preprint arXiv:2312.08358, 2023. Skalse and Abate [2022] J. Skalse and A. Abate. Misspecification in Inverse Reinforcement Learning. arXiv e-prints, art. arXiv:2212.03201, Dec. 2022. doi: 10.48550/arXiv.2212.03201. Skalse et al. [2023] J. M. V. Skalse, M. Farrugia-Roberts, S. Russell, A. Abate, and A. Gleave. Invariance in Policy Optimisation and Partial Identifiability in Reward Learning. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 32033–32058. PMLR, 23–29 Jul 2023. URL https://proceedings.mlr.press/v202/skalse23a.html. Stray [2023] J. Stray. The AI Learns to Lie to Please You: Preventing Biased Feedback Loops in Machine-Assisted Intelligence Analysis. Analytics, 2(2):350–358, 2023. ISSN 2813-2203. doi: 10.3390/analytics2020020. URL https://w.mdpi.com/2813-2203/2/2/20. Touvron et al. [2023] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M.-A. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom. Llama 2: Open Foundation and Fine-Tuned Chat Models. arxiv e-prints, 2023. Ward et al. [2023] F. R. Ward, F. Belardinelli, F. Toni, and T. Everitt. Honesty Is the Best Policy: Defining and Mitigating AI Deception. arxiv e-prints, 2023. Wen et al. [2024] J. Wen, R. Zhong, A. Khan, E. Perez, J. Steinhardt, M. Huang, S. R. Bowman, H. He, and S. Feng. Language Models Learn to Mislead Humans via RLHF, 2024. URL https://arxiv.org/abs/2409.12822. Wu [2024] S. Wu. Introducing Devin, the first AI software engineer. https://w.cognition-labs.com/introducing-devin, 2024. Accessed: 2024-05-06. Zhuang and Hadfield-Menell [2020] S. Zhuang and D. Hadfield-Menell. Consequences of Misaligned AI. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS’20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546. Ziebart et al. [2008] B. D. Ziebart, A. L. Maas, J. A. Bagnell, and A. K. Dey. Maximum entropy inverse reinforcement learning. In D. Fox and C. P. Gomes, editors, AAAI, pages 1433–1438. AAAI Press, 2008. ISBN 978-1-57735-368-3. URL http://dblp.uni-trier.de/db/conf/aaai/aaai2008.html#ZiebartMBD08. Appendix In the appendix, we provide more extensive theory, proofs, and examples. The appendix makes free use of concepts and notation defined in the main paper. In particular, throughout we assume a general MDP together with observation kernel PO:→Ω:subscript→ΩP_O:S→ _O : S → Ω and a human with general belief kernel B⁢(o→∣s→)conditional→B( o s)B ( over→ start_ARG o end_ARG ∣ over→ start_ARG s end_ARG ), unless otherwise stated. See the list of Symbols in Section A to refresh notation. In Section B we supplement the examples from the main paper with more mathematical details. In Section C, we provide an extensive theory for appropriately modeled partial observability in RLHF. This can mainly be considered a supplement to Section 5 and contains our main theorems, supplementary results, analysis of special cases, and examples. In Section D, we analyze the naive application of RLHF under partial observability, which means that the learning system is not aware of the human’s partial observability. This section is essentially a supplement to Section 4 and contains an analysis of the policy evaluation function JobssubscriptobsJ_obsJroman_obs, of deceptive inflation and overjustification, and further extensive mathematical examples showing the failures of naive RLHF under partial observability. Contents of the Appendix 1 Introduction 2 Related work 3 Reward identifiability from full observations 3.1 Markov decision processes 3.2 RLHF and identifiability from full observations 4 The impact of partial observations on RLHF 4.1 What does RLHF learn from partial observations? 4.2 An ontology of behaviors 4.3 Deceptive inflation and overjustification 4.4 Deception and overjustification in examples 5 Return ambiguity from feedback under known partial observability 5.1 Feedback-compatibility and ambiguity of return functions 5.2 Toward improving RLHF in partially observable settings 6 Conclusions A List of Symbols B Details for deception and overjustification in examples B.1 Example A: hiding failures B.2 Example B: paying to reveal information B.3 Derivations and Further Details for Fig. 4A B.4 Ambiguity in Section 4.4 examples when modeling partial observability B.5 Experimental details C Modeling the Human in Partially Observable RLHF C.1 The Belief over the State Sequence for Rational Humans C.2 Ambiguity and Identifiability of Reward and Return Functions under Observation Sequence Comparisons C.3 The Ambiguity in Reward Learning in Practice C.4 Identifiability of Return Functions When Human Observations Are Not Known C.5 Simple Special Cases: Full Observability, Deterministic PO→subscript→P_ OPover→ start_ARG O end_ARG, and Noisy PO→subscript→P_ OPover→ start_ARG O end_ARG C.6 Robustness of Return Function Identifiability under Belief Misspecification C.6.1 Some Norm Theory for Linear Operators C.6.2 Application to Bounds in the Error of the Return Function C.7 Preliminary Characterizations of the Ambiguity C.8 Examples Supplementing Section 5 D Issues of Naively Applying RLHF under Partial Observability D.1 Optimal Policies under RLHF with Deterministic Partial Observations Maximize JobssubscriptobsJ_obsJroman_obs D.2 Interlude: When the Human Knows the Policy and is a Bayesian Reasoner, then Jobs=JsubscriptobsJ_obs=JJroman_obs = J D.3 Proof of Theorem 4.5 D.4 Further Examples Supplementing Section 4.4 E NeurIPS Paper Checklist Appendix A List of Symbols General MDPs SS Set of environment states s∈s ∈ S AA Set of actions a∈a ∈ A of the policy Δ⁢()Δ (S)Δ ( S ) Set of probability distributions over SS. Can be defined for any finite set :×→Δ⁢():→ΔT:S×A→ (S)T : S × A → Δ ( S ) Transition kernel P0∈Δ⁢()subscript0ΔP_0∈ (S)P0 ∈ Δ ( S ) Initial state distribution R∈ℝsuperscriptℝR∈ R^SR ∈ blackboard_RS Usually the true reward function R′∈ℝsuperscript′ℝR ∈ R^SR′ ∈ blackboard_RS Usually a reward function in the kernel of ∘ B B ∘ Γ R~∈ℝ~superscriptℝ R∈ R^Sover~ start_ARG R end_ARG ∈ blackboard_RS Usually another reward function, e.g. inferred by a learning system γ∈[0,1]01γ∈[0,1]γ ∈ [ 0 , 1 ] Discount factor π:→Δ⁢():→Δπ:S→ (A)π : S → Δ ( A ) A policy π:→Δ⁢():superscript→ΔT^π:S→ (S)Titalic_π : S → Δ ( S ) Transition kernel for a fixed policy π given by π⁢(s′∣s)=∑a∈⁢(s′∣s,a)⋅π⁢(a∣s)superscriptconditionalsuperscript′subscript⋅conditionalsuperscript′conditionalT^π(s s)= _a T(s^% s,a)·π(a s)Titalic_π ( s′ ∣ s ) = ∑a ∈ A T ( s′ ∣ s , a ) ⋅ π ( a ∣ s ) T∈ℕT∈ NT ∈ blackboard_N Finite time horizon Pπ∈Δ⁢(T)superscriptΔsuperscriptP^π∈ (S^T)Pitalic_π ∈ Δ ( Sitalic_T ) State sequence distribution induced by the policy π →⊆T→superscript S ^Tover→ start_ARG S end_ARG ⊆ Sitalic_T State sequences s→∈→ s∈ Sover→ start_ARG s end_ARG ∈ over→ start_ARG S end_ARG supported by PπsuperscriptP^πPitalic_π G∈ℝ→superscriptℝ→G∈ R SG ∈ blackboard_Rover→ start_ARG S end_ARG Usually the true return function given by G⁢(s→)=∑t=0Tγt⁢R⁢(st)→superscriptsubscript0superscriptsubscriptG( s)= _t=0^Tγ^tR(s_t)G ( over→ start_ARG s end_ARG ) = ∑t = 0T γitalic_t R ( sitalic_t ). G′∈ℝ→superscript′ℝ→G ∈ R SG′ ∈ blackboard_Rover→ start_ARG S end_ARG Usually a return function in ker⁡kernel Bker B G~∈ℝ→~superscriptℝ→ G∈ R Sover~ start_ARG G end_ARG ∈ blackboard_Rover→ start_ARG S end_ARG Usually another return function, e.g. inferred by a learning system J The true policy evaluation function given by J⁢(π)=s→∼Pπ[G⁢(s→)]subscriptsimilar-to→superscript→J(π)= *E_ s P^π [G( s) ]J ( π ) = Eover→ start_ARG s end_ARG ∼ Pitalic_π [ G ( over→ start_ARG s end_ARG ) ]. Additions to General MDPs with Partial Observability Ω Ω Set of possible observations o∈Ωo∈ ∈ Ω PO:→Δ⁢(Ω):subscript→ΔΩP_O:S→ ( )Pitalic_O : S → Δ ( Ω ) Observation kernel determining the human’s observations PO→:→Δ⁢(ΩT):subscript→ΔsuperscriptΩP_ O: S→ ( ^T )Pover→ start_ARG O end_ARG : over→ start_ARG S end_ARG → Δ ( Ωitalic_T ) The observation sequence kernel given by PO→⁢(o→∣s→)=∏t=0TPO⁢(ot∣st)subscript→conditional→superscriptsubscriptproduct0subscriptconditionalsubscriptsubscriptP_ O ( o s )= _t=0^TP_O (o_t% s_t )Pover→ start_ARG O end_ARG ( over→ start_ARG o end_ARG ∣ over→ start_ARG s end_ARG ) = ∏t = 0T Pitalic_O ( oitalic_t ∣ sitalic_t ) Ω→⊆ΩT→ΩsuperscriptΩ ^Tover→ start_ARG Ω end_ARG ⊆ Ωitalic_T The set of observed sequences o→∈ΩT→superscriptΩ o∈ ^Tover→ start_ARG o end_ARG ∈ Ωitalic_T that can be sampled from PO→(⋅∣s→)P_ O(· s)Pover→ start_ARG O end_ARG ( ⋅ ∣ over→ start_ARG s end_ARG ) for s→∈→ s∈ Sover→ start_ARG s end_ARG ∈ over→ start_ARG S end_ARG O:→Ω:→ΩO:S→ : S → Ω Observation function for the case that POsubscriptP_OPitalic_O is deterministic; given by O⁢(s)=oO(s)=oO ( s ) = o with o such that PO⁢(o∣s)=1subscriptconditional1P_O(o s)=1Pitalic_O ( o ∣ s ) = 1 O→:→Ω→:→Ω O: S→ over→ start_ARG O end_ARG : over→ start_ARG S end_ARG → over→ start_ARG Ω end_ARG Observation sequence function for the case that PO→subscript→P_ OPover→ start_ARG O end_ARG is deterministic; given by O→⁢(s→)=o→ O( s)= oover→ start_ARG O end_ARG ( over→ start_ARG s end_ARG ) = over→ start_ARG o end_ARG with o→ oover→ start_ARG o end_ARG such that PO→⁢(o→∣s→)=1subscript→conditional→1P_ O( o s)=1Pover→ start_ARG O end_ARG ( over→ start_ARG o end_ARG ∣ over→ start_ARG s end_ARG ) = 1 Go→∈ℝs→∈→∣O→⁢(s→)=o→subscript→superscriptℝconditional-set→G_ o∈ R^\ s∈ S O( s)=% o\Gover→ start_ARG o end_ARG ∈ blackboard_R over→ start_ARG s end_ARG ∈ over→ start_ARG S end_ARG ∣ over→ start_ARG O end_ARG ( over→ start_ARG s end_ARG ) = over→ start_ARG o end_ARG Restriction of the return function G∈ℝ→superscriptℝ→G∈ R SG ∈ blackboard_Rover→ start_ARG S end_ARG to s→∈→∣O→⁢(s→)=o→conditional-set→ \ s∈ S O( s)= o \ over→ start_ARG s end_ARG ∈ over→ start_ARG S end_ARG ∣ over→ start_ARG O end_ARG ( over→ start_ARG s end_ARG ) = over→ start_ARG o end_ARG for fixed o→∈Ω→Ω o∈ over→ start_ARG o end_ARG ∈ over→ start_ARG Ω end_ARG Gobs∈ℝ→subscriptobssuperscriptℝ→G_obs∈ R SGroman_obs ∈ blackboard_Rover→ start_ARG S end_ARG Return function that can be inferred when partial observability is not properly modeled, given by Gobs⁢(s→)≔(⋅G)⁢(O→⁢(s→))≔subscriptobs→⋅→G_obs( s) ( B· G% ) ( O( s) )Groman_obs ( over→ start_ARG s end_ARG ) ≔ ( B ⋅ G ) ( over→ start_ARG O end_ARG ( over→ start_ARG s end_ARG ) ) JobssubscriptobsJ_obsJroman_obs Observation policy evaluation function, defined in Eq. (4) State- and Observation Sequences st∈subscripts_t _t ∈ S The t’th entry in a state sequence s→ sover→ start_ARG s end_ARG s→∈T→superscript s ^Tover→ start_ARG s end_ARG ∈ Sitalic_T State sequence s→=s0,…,sT→subscript0…subscript s=s_0,…,s_Tover→ start_ARG s end_ARG = s0 , … , sitalic_T s^∈t^superscript s ^tover start_ARG s end_ARG ∈ Sitalic_t State sequence segment s^=s0,…,st^subscript0…subscript s=s_0,…,s_tover start_ARG s end_ARG = s0 , … , sitalic_t for t≤Tt≤ Tt ≤ T ot∈ΩsubscriptΩo_t∈ _t ∈ Ω The t’th entry in an observation sequence o→ oover→ start_ARG o end_ARG o→∈ΩT→superscriptΩ o∈ ^Tover→ start_ARG o end_ARG ∈ Ωitalic_T Observation sequence o→=o0,…,oT→subscript0…subscript o=o_0,…,o_Tover→ start_ARG o end_ARG = o0 , … , oitalic_T o^∈Ωt^superscriptΩ o∈ ^tover start_ARG o end_ARG ∈ Ωitalic_t Observation sequence segment o^=o0,…,ot^subscript0…subscript o=o_0,…,o_tover start_ARG o end_ARG = o0 , … , oitalic_t for t≤Tt≤ Tt ≤ T The Human’s Belief B⁢(π′)superscript′B(π )B ( π′ ) The human’s policy prior B⁢(s→)→B( s)B ( over→ start_ARG s end_ARG ) The human’s prior belief that a sequence s→ sover→ start_ARG s end_ARG will be sampled, given by B⁢(s→)=∫π′B⁢(π′)⁢Pπ′⁢(s→)⁢π′→subscriptsuperscript′superscriptsuperscript′→differential-dsuperscript′B( s)= _π B(π )P^π ( s)dπ B ( over→ start_ARG s end_ARG ) = ∫π′ B ( π′ ) Pitalic_π start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT ( over→ start_ARG s end_ARG ) d π′ B⁢(s→∣o→)conditional→B ( s o )B ( over→ start_ARG s end_ARG ∣ over→ start_ARG o end_ARG ) The human’s belief of a state sequence given an observation sequence, see Proposition C.1 for a Bayesian version Bπ⁢(s→∣o→)superscriptconditional→B^π( s o)Bitalic_π ( over→ start_ARG s end_ARG ∣ over→ start_ARG o end_ARG ) The human’s belief of a state sequence given an observation sequence; it is allowed to depend on the true policy π, see Proposition C.1 Bo→∈ℝs→∈→∣O→⁢(s→)=o→subscript→superscriptℝconditional-set→B_ o∈ R^\ s∈ S O( s)=% o\Bover→ start_ARG o end_ARG ∈ blackboard_R over→ start_ARG s end_ARG ∈ over→ start_ARG S end_ARG ∣ over→ start_ARG O end_ARG ( over→ start_ARG s end_ARG ) = over→ start_ARG o end_ARG Vector of prior probabilities B⁢(s→)→B( s)B ( over→ start_ARG s end_ARG ) for s→∈s→∈→∣O→⁢(s→)=o→conditional-set→ s∈ \ s∈ S O( s)= o \over→ start_ARG s end_ARG ∈ over→ start_ARG s end_ARG ∈ over→ start_ARG S end_ARG ∣ over→ start_ARG O end_ARG ( over→ start_ARG s end_ARG ) = over→ start_ARG o end_ARG Identifiability Theorem β>00β>0β > 0 The inverse temperature parameter of the Boltzmann rational human σ:ℝ→(0,1):→ℝ01σ: R→(0,1)σ : blackboard_R → ( 0 , 1 ) The sigmoid function given by σ⁢(x)=11+exp⁡(−x)11σ(x)= 11+ (-x)σ ( x ) = divide start_ARG 1 end_ARG start_ARG 1 + exp ( - x ) end_ARG :ℝ→ℝ→:→superscriptℝsuperscriptℝ→ : R^S→ R % SΓ : blackboard_RS → blackboard_Rover→ start_ARG S end_ARG Function that maps a reward function R to the return function ⁡(R) (R)Γ ( R ) with [⁡(R)]⁢(s→)=∑t=0Tγt⁢R⁢(st)delimited-[]→superscriptsubscript0superscriptsubscript [ (R) ]( s)= _t=0^Tγ^% tR(s_t)[ Γ ( R ) ] ( over→ start_ARG s end_ARG ) = ∑t = 0T γitalic_t R ( sitalic_t ) :ℝ→ℝΩ→:→superscriptℝ→superscriptℝ→Ω B: R S→ R % B : blackboard_Rover→ start_ARG S end_ARG → blackboard_Rover→ start_ARG Ω end_ARG Function that maps a return function G to the expected return function ⁡(G) B(G)B ( G ) on observation sequences given by [⁡(G)]⁢(o→)=s→∼B⁢(s→∣o→)[G⁢(s→)]delimited-[]→subscriptsimilar-to→conditional→ [ B(G) ]( o)= *E% _ s B( s o) [G( s) ][ B ( G ) ] ( over→ start_ARG o end_ARG ) = Eover→ start_ARG s end_ARG ∼ B ( over→ start_ARG s end_ARG ∣ over→ start_ARG o end_ARG ) [ G ( over→ start_ARG s end_ARG ) ] :ℝ→ℝΩ→:→superscriptℝsuperscriptℝ→Ω F: R^S→ R F : blackboard_RS → blackboard_Rover→ start_ARG Ω end_ARG The composition =∘ F= B % F = B ∘ Γ PR⁢(s→≻s→′)superscriptsucceeds→superscript→′P^R ( s s 1.0pt )Pitalic_R ( over→ start_ARG s end_ARG ≻ over→ start_ARG s end_ARG′ ) Boltzmann rational choice probability in the case of full observability (Eq. (1)) PR⁢(o→≻o→′)superscriptsucceeds→superscript→′P^R ( o o 1.0pt )Pitalic_R ( over→ start_ARG o end_ARG ≻ over→ start_ARG o end_ARG′ ) Boltzmann rational choice probability in the case of partial observability (Eq. (2)) :ℝΩ→ℝ→:→superscriptℝ→Ωsuperscriptℝ→ O: R → R % SO : blackboard_Rover→ start_ARG Ω end_ARG → blackboard_Rover→ start_ARG S end_ARG Abstract linear operator given by [⁡(v)]⁢(s→)=o→∼PO→⁢(o→∣s→)[v⁢(o→)]delimited-[]→subscriptsimilar-to→subscript→conditional→ [ O(v) ]( s)= *E% _ o P_ O( o s) [v( o) ][ O ( v ) ] ( over→ start_ARG s end_ARG ) = Eover→ start_ARG o end_ARG ∼ P start_POSTSUBSCRIPT over→ start_ARG O end_ARG ( over→ start_ARG o end_ARG ∣ over→ start_ARG s end_ARG ) end_POSTSUBSCRIPT [ v ( over→ start_ARG o end_ARG ) ] ⊗:ℝΩ→×Ω→ℝ→×→:tensor-product→superscriptℝ→Ω→Ωsuperscriptℝ→ O O: R % × → R S× % SO ⊗ O : blackboard_Rover→ start_ARG Ω end_ARG × over→ start_ARG Ω end_ARG → blackboard_Rover→ start_ARG S end_ARG × over→ start_ARG S end_ARG Formally the Kronecker product of OO with itself, explicitly given by [(⊗)⁢(C)]⁢(s→,s→′)=o→,o→′∼PO→(⋅∣s→,s→′)[C⁢(o→,o→′)] [( O O)(C) ](% s, s 1.0pt )= *E_ o, % o 1.0pt P_ O(· s, s 1.0pt^% ) [C( o, o 1.0pt ) ][ ( O ⊗ O ) ( C ) ] ( over→ start_ARG s end_ARG , over→ start_ARG s end_ARG′ ) = Eover→ start_ARG o end_ARG , over→ start_ARG o end_ARG′ ∼ P start_POSTSUBSCRIPT over→ start_ARG O end_ARG ( ⋅ ∣ over→ start_ARG s end_ARG , over→ start_ARG s end_ARG′ ) end_POSTSUBSCRIPT [ C ( over→ start_ARG o end_ARG , over→ start_ARG o end_ARG′ ) ] Robustness to Misspecifications ‖x‖norm\|x\|∥ x ∥ Euclidean norm of the vector x∈ℝksuperscriptℝx∈ R^kx ∈ blackboard_Rk ‖norm\| A\|∥ A ∥ Matrix norm of the matrix AA, given by ‖≔maxx,‖x‖=1⁡‖⁡x‖≔normsubscriptnorm1norm\| A\| _x,\ \|x\|=1\|% Ax\|∥ A ∥ ≔ maxitalic_x , ∥ x ∥ = 1 ∥ A x ∥ τ⁢()τ( A)τ ( A ) Matrix quantity defined in Equation (9) C⁢(,ρ)C( A,ρ)C ( A , ρ ) Matrix quantity defined in Equation (10) r⁢()rr( B)r ( B ) Restriction of BB to im⁡imim im Γ General Sets and (Linear) Functions |A||A|| A | Number of elements in the set A A∩CA∩ CA ∩ C Intersection of sets A and C A∪CA∪ CA ∪ C Union of sets A and C A∖CA CA ∖ C Relative complement of C in A δxsubscript _xδitalic_x The Dirac delta distribution of a point x in a set; given by δx⁢(A)=1subscript1 _x(A)=1δitalic_x ( A ) = 1 if x∈Ax∈ Ax ∈ A and δx⁢(A)=0subscript0 _x(A)=0δitalic_x ( A ) = 0, else ker⁡kernel Aker A The kernel of a linear operator :V→W:→ A:V→ WA : V → W; given by ker⁡=v∈V∣⁡(v)=0kernelconditional-set0 A= \v∈ V A(v)=0% \ker A = v ∈ V ∣ A ( v ) = 0 im⁡imim Aim A The image of a linear operator :V→W:→ A:V→ WA : V → W; given by im⁡=w∈W∣∃v∈V:⁡(v)=wimconditional-set:im A= \w∈ W ∃ v∈ V:% A(v)=w \im A = w ∈ W ∣ ∃ v ∈ V : A ( v ) = w f−1⁢(y)superscript1f^-1(y)f- 1 ( y ) Preimage of y under a function f:X→Y:→f:X→ Yf : X → Y; given by f−1⁢(y)=x∈X∣f⁢(x)=ysuperscript1conditional-setf^-1(y)= \x∈ X f(x)=y \f- 1 ( y ) = x ∈ X ∣ f ( x ) = y Appendix B Details for deception and overjustification in examples Figure 7: Two example MDPs with observation functions in which RLHF chooses undesirable policies. Each box depicts a state with a footer showing the (deterministic) observation produced by that state. Outgoing edges from each box are available actions. A more detailed diagram for the first MDP, with explicit shell commands and log messages, is available in Section B.3. Here we include details to the examples described in Section 4.4 that illustrate the failure modes of RLHF in the presence of partial observability. For each of the following, we will characterize the policy which maximizes JobssubscriptobsJ_obsJroman_obs, as this is the policy RLHF selects for when observations are deterministic; see Proposition 4.1. Our examples feature an agent trained with RLHF to complete tasks in a user’s terminal. The output of each command (stdout and stderr) is piped to a log file, which is what the human evaluator sees when making choices for RLHF. We assume that the final state T has a self-transition, episodes have a fixed horizon length of 3 (meaning state sequences have length 4: s0,…,s3subscript0…subscript3s_0,…,s_3s0 , … , s3), γ=11γ=1γ = 1, and there is a fixed initial state s0=Ssubscript0s_0=Ss0 = S. Both examples feature a fixed transition probability 0<p<1010<p<10 < p < 1. We assume that the human’s belief only supports possible explanations: B⁢(s→∣o→)>0⟹O→⁢(s→)=o→conditional→0→ B( s o)>0 O( s)= oB ( over→ start_ARG s end_ARG ∣ over→ start_ARG o end_ARG ) > 0 ⟹ over→ start_ARG O end_ARG ( over→ start_ARG s end_ARG ) = over→ start_ARG o end_ARG. For further details about these examples, including a tutorial-style analysis of Example A, see Section B.3. B.1 Example A: hiding failures See Section B.3 for derivations and a tutorial-style analysis of this example. In Fig. 4A (and Fig. 6A), the agent’s task is to install CUDA, (states W and WHsubscriptW_HWitalic_H) which requires first installing Nvidia drivers (state I). If the CUDA installation fails, one enters L or LHsubscriptL_HLitalic_H. The aCsubscripta_Caitalic_C action is a command to install CUDA that logs both successes and failures, whereas aHsubscripta_Haitalic_H corresponds to aCsubscripta_Caitalic_C with 2> /dev/null appended, which suppresses error messages such that no log message is produced if the installation fails. The human may prefer that the agent not hide a failure; we price this into the reward function with a penalty r≥00r≥ 0r ≥ 0 on the reward at LHsubscriptL_HLitalic_H. The agent may attempt the CUDA installation before installing drivers, but this will fail. There are three pairs of trajectories which produce identical observations. Here we address the most prominent (see Section B.3 for the others): S⁢I⁢T⁢TSITTS I T T and S⁢I⁢LH⁢TsubscriptSIL_HTS I Litalic_H T both produce o∅⁢oI⁢o∅⁢o∅subscriptsubscriptsubscriptsubscripto_ o_Io_ o_ o∅ oitalic_I o∅ o∅, stylized as a log containing only a success confirmation for Python (Fig. 1, o→2subscript→2 o_2over→ start_ARG o end_ARG2). after successfully installing drivers, a failed CUDA installation with 2> /dev/null (S⁢I⁢LH⁢TsubscriptSIL_HTS I Litalic_H T) and simply exiting (S⁢I⁢T⁢TSITTS I T T) both produce a log containing only a success confirmation for the drivers (o∅⁢oI⁢o∅⁢o∅subscriptsubscriptsubscriptsubscripto_ o_Io_ o_ o∅ oitalic_I o∅ o∅). Let pH≔B⁢(s→=S⁢I⁢LH⁢T∣o→=o∅⁢oI⁢o∅⁢o∅)∈(0,1)≔subscript→conditionalsubscript→subscriptsubscriptsubscriptsubscript01 p_H B ( s=SIL_HT o=o_ o% _Io_ o_ )∈(0,1)pitalic_H ≔ B ( over→ start_ARG s end_ARG = S I Litalic_H T ∣ over→ start_ARG o end_ARG = o∅ oitalic_I o∅ o∅ ) ∈ ( 0 , 1 ) be the human’s suspicion, upon seeing only a success confirmation for the drivers, that a failed CUDA installation was hidden. We consider two cases which differ only in whether the expected value of attempting to install CUDA (with default logging, aCsubscripta_Caitalic_C) after installing the drivers is positive. In Example B.1, the success probability p is high enough that the true optimal policy is to attempt the installation; in Example B.2, p is low enough that the optimal policy is to terminate. Example B.1. If the true-reward-optimal policy π∗superscriptπ^*π∗ is to attempt to install CUDA with default logging (aCsubscripta_Caitalic_C) after installing drivers, but pH<55+rsubscript55p_H< 55+rpitalic_H < divide start_ARG 5 end_ARG start_ARG 5 + r end_ARG, then the JobssubscriptobsJ_obsJroman_obs-maximizing policy, which will be chosen by RLHF, is to instead use the 2> /dev/null trick. Intuitively, this is because if the installation fails, the human is likely to believe that the agent just didn’t attempt to install CUDA, which is preferred to a failed installation. This is deceptive inflation; the overestimation error E¯+superscript¯ E^+over¯ start_ARG E end_ARG+ for this policy is positive, whereas E¯+⁢(π∗)=0superscript¯superscript0 E^+(π^*)=0over¯ start_ARG E end_ARG+ ( π∗ ) = 0, in line with Theorem 4.5. If pH>55+rsubscript55p_H> 55+rpitalic_H > divide start_ARG 5 end_ARG start_ARG 5 + r end_ARG, then RLHF will select π∗superscriptπ^*π∗. Example B.2. If π∗superscriptπ^*π∗ is to terminate (aTsubscripta_Taitalic_T) after installing the drivers, but pH<55+rsubscript55p_H< 55+rpitalic_H < divide start_ARG 5 end_ARG start_ARG 5 + r end_ARG, then RLHF will learn the deceptive inflation 2> /dev/null policy as in Example B.1. However, if pH>55+rsubscript55p_H> 55+rpitalic_H > divide start_ARG 5 end_ARG start_ARG 5 + r end_ARG, RLHF will choose the policy that attempts to install CUDA with default logging. Intuitively, this is because the combination of increased suspicion that the agent might be hiding errors (higher pHsubscriptp_Hpitalic_H) and increased penalty on this suspected behavior (higher r) also penalizes our preferred trajectory S⁢I⁢T⁢TSITTS I T T since it can be misinterpreted as hiding errors. This is overjustification; the overestimation error E¯+superscript¯ E^+over¯ start_ARG E end_ARG+ is 0 (thus this is not deceptive inflation), and as required by Theorem 4.5, the underestimation error E¯−superscript¯ E^-over¯ start_ARG E end_ARG- is 0, lower than E−⁢(π∗)=pH⁢(5+r)superscriptsuperscriptsubscript5E^-(π^*)=p_H(5+r)E- ( π∗ ) = pitalic_H ( 5 + r ). B.2 Example B: paying to reveal information In Fig. 4B (and Fig. 6B), the agent’s task is to install Python (state I) and to optionally further install NumPy (states W and WVsubscriptW_VWitalic_V). The aDsubscripta_Daitalic_D action corresponds to a command to install NumPy with “default” settings which only logs errors, whereas aVsubscripta_Vaitalic_V corresponds to the same command with a ----verbose flag that adds additional info. In the case of a success, the human distinctly prefers not to see this verbose output; we price this into the reward function with a penalty r>00r>0r > 0 on the reward at WVsubscriptW_VWitalic_V. There is only one pair of trajectories which produce identical observations: after successfully installing Python, a successful NumPy installation with default logging (S⁢I⁢W⁢TSIWTS I W T) and simply exiting (S⁢I⁢T⁢TSITTS I T T) both produce a log containing only a success confirmation for Python (o∅⁢oI⁢o∅⁢o∅subscriptsubscriptsubscriptsubscripto_ o_Io_ o_ o∅ oitalic_I o∅ o∅). Let pD≔B⁢(s→=S⁢I⁢W⁢T∣o→=o∅⁢oI⁢o∅⁢o∅)∈(0,1)≔subscript→conditional→subscriptsubscriptsubscriptsubscript01 p_D B( s=SIWT o=o_ o_Io_% o_ )∈(0,1)pitalic_D ≔ B ( over→ start_ARG s end_ARG = S I W T ∣ over→ start_ARG o end_ARG = o∅ oitalic_I o∅ o∅ ) ∈ ( 0 , 1 ) be the human’s optimism, upon seeing only a success confirmation for Python, that NumPy was also successfully installed (without the ----verbose flag). Here we consider only the case where p is large enough that the true optimal policy is to install Python then attempt to install NumPy with default logging (aDsubscripta_Daitalic_D). Example B.3. If π∗superscriptπ^*π∗ is to attempt to install NumPy with aDsubscripta_Daitalic_D after installing Python, and pD>q≔15⁢(p⁢(6−r)−1)subscript≔1561 p_D>q 15 (p(6-r)-1 )pitalic_D > q ≔ divide start_ARG 1 end_ARG start_ARG 5 end_ARG ( p ( 6 - r ) - 1 ), then RLHF will select the policy that terminates after installing Python. Intuitively, this is because the agent can exploit the human’s optimism that NumPy was installed quietly without taking the risk of an observable failure (L). This is deceptive inflation, with an overestimation error E¯+superscript¯ E^+over¯ start_ARG E end_ARG+ of 5⁢pD5subscript5p_D5 pitalic_D, greater than E¯+⁢(π∗)=0superscript¯superscript0 E^+(π^*)=0over¯ start_ARG E end_ARG+ ( π∗ ) = 0. If instead pD<qsubscriptp_D<qpitalic_D < q, then RLHF will select the policy that attempts the NumPy installation with verbose logging (aVsubscripta_Vaitalic_V). Intuitively, this is because the agent is willing to “pay” the cost of r true reward to prove to the human that it installed NumPy, even when the human does not want to see this proof. This is overjustification; the overestimation error E¯+superscript¯ E^+over¯ start_ARG E end_ARG+ is 0 (thus this is not deceptive inflation), and the underestimation error E¯−superscript¯ E^-over¯ start_ARG E end_ARG- is 0, lower than E¯−⁢(π∗)=5⁢p⁢(1−pD)superscript¯superscript51subscript E^-(π^*)=5p(1-p_D)over¯ start_ARG E end_ARG- ( π∗ ) = 5 p ( 1 - pitalic_D ). B.3 Derivations and Further Details for Fig. 4A Figure 8: An expanded view of Figure 4A. Commands corresponding to the various actions are depicted along edges, and log messages corresponding to the various observations are depicted underneath each state. We first include Figure 8, a more detailed picture of the MDP and observation function in Section B.1, to help ground the narrative details of the example. Next we formally enumerate the details of the MDP and observation function. • =S,I,W,WH,L,LH,TsubscriptsubscriptS=\S,I,W,W_H,L,L_H,T\S = S , I , W , Witalic_H , L , Litalic_H , T . • =aI,aC,aH,aTsubscriptsubscriptsubscriptsubscriptA=\a_I,a_C,a_H,a_T\A = aitalic_I , aitalic_C , aitalic_H , aitalic_T . • TT is as depicted in Figure 8 and Figure 4A. For a state s, any outgoing arrow labeled with an action a (such as aIsubscripta_Iaitalic_I) describes the distribution ⁢(s′∣s,a)conditionalsuperscript′T(s s,a)T ( s′ ∣ s , a ) as follows: if the arrow does not split, then ⁢(s′∣s,a)=1conditionalsuperscript′1T(s s,a)=1T ( s′ ∣ s , a ) = 1 where s′ is the state the arrow points to; if the arrow does split, then for each successor state s′ it eventually reaches, a probability q is written just before the box corresponding to s′ (for this example, q=pq=pq = p or q=1−p1q=1-pq = 1 - p), and ⁢(s′∣s,a)=qconditionalsuperscript′T(s s,a)=qT ( s′ ∣ s , a ) = q. ∘ ∘ Additionally, any action taken from a state that does not have an outgoing arrow corresponding to that action will immediately transition to state T, as though aTsubscripta_Taitalic_T had been taken. ∘ ∘ Any action taken from state T transitions deterministically to T. • P0⁢(S)=1subscript01P_0(S)=1P0 ( S ) = 1. • R is as described in the table (the numbers in the top right of each state box) with r≥00r≥ 0r ≥ 0. Additionally, R⁢(S)=R⁢(T)=00 R(S)=R(T)=0R ( S ) = R ( T ) = 0. • γ=11γ=1γ = 1. We work with a fixed horizon length of 3, meaning state sequences have length 4 (since time is zero-indexed: s0⁢s1⁢s2⁢s3subscript0subscript1subscript2subscript3s_0s_1s_2s_3s0 s1 s2 s3). The observation function is also depicted in Figure 8. Each state deterministically produces the observation in the lower-right corner of its box in the figure. We also write it in another format in Table 9. Table 9: The observation function O for the example in Section B.1 and Section B.3. s S I W WHsubscriptW_HWitalic_H L LHsubscriptL_HLitalic_H T O⁢(s)O(s)O ( s ) o∅subscripto_ o∅ oIsubscripto_Ioitalic_I oWsubscripto_Woitalic_W oWsubscripto_Woitalic_W oLsubscripto_Loitalic_L o∅subscripto_ o∅ o∅subscripto_ o∅ We make the additional assumption that the human belief B⁢(s→∣o→)conditional→B( s o)B ( over→ start_ARG s end_ARG ∣ over→ start_ARG o end_ARG ) only supports state sequences s→ sover→ start_ARG s end_ARG which actually produce o→ oover→ start_ARG o end_ARG under the sequence observation function O→ Oover→ start_ARG O end_ARG: B⁢(s→∣o→)>0⟹O→⁢(s→)=o→conditional→0→B( s o)>0 O( s)= oB ( over→ start_ARG s end_ARG ∣ over→ start_ARG o end_ARG ) > 0 ⟹ over→ start_ARG O end_ARG ( over→ start_ARG s end_ARG ) = over→ start_ARG o end_ARG. In particular, this means that for any o→ oover→ start_ARG o end_ARG which is only produced by one s→ sover→ start_ARG s end_ARG, B⁢(o→∣s→)=1conditional→1B( o s)=1B ( over→ start_ARG o end_ARG ∣ over→ start_ARG s end_ARG ) = 1. There are three pairs of state sequences which produce identical observation sequences. For each, we introduce a parameter representing the probability the human infers the first of the pair of state sequences upon seeing their shared observation sequence. 1. S⁢I⁢LH⁢TsubscriptSIL_HTS I Litalic_H T and S⁢I⁢T⁢TSITTS I T T both produce o∅⁢oI⁢o∅⁢o∅subscriptsubscriptsubscriptsubscripto_ o_Io_ o_ o∅ oitalic_I o∅ o∅, a log containing only a success confirmation for installing drivers, again because O⁢(LH)=O⁢(T)=o∅subscriptsubscriptO(L_H)=O(T)=o_ O ( Litalic_H ) = O ( T ) = o∅. Let pH=B⁢(s→=S⁢I⁢LH⁢T∣o→=o∅⁢oI⁢o∅⁢o∅)subscript→conditionalsubscript→subscriptsubscriptsubscriptsubscriptp_H=B( s=SIL_HT o=o_ o_Io_ o_% )pitalic_H = B ( over→ start_ARG s end_ARG = S I Litalic_H T ∣ over→ start_ARG o end_ARG = o∅ oitalic_I o∅ o∅ ). 2. S⁢T⁢T⁢TSTTTS T T T and S⁢LH⁢T⁢TsubscriptSL_HTTS Litalic_H T T both produce o∅⁢o∅⁢o∅⁢o∅subscriptsubscriptsubscriptsubscripto_ o_ o_ o_ o∅ o∅ o∅ o∅, an empty log, since O⁢(LH)=O⁢(T)=o∅subscriptsubscriptO(L_H)=O(T)=o_ O ( Litalic_H ) = O ( T ) = o∅. Let pH′=B⁢(s→=S⁢LH⁢T⁢T∣o→=o∅⁢o∅⁢o∅⁢o∅)superscriptsubscript′→conditionalsubscript→subscriptsubscriptsubscriptsubscriptp_H =B( s=SL_HT o=o_ o_ o_% o_ )pitalic_H′ = B ( over→ start_ARG s end_ARG = S Litalic_H T T ∣ over→ start_ARG o end_ARG = o∅ o∅ o∅ o∅ ). 3. S⁢I⁢W⁢TSIWTS I W T and S⁢I⁢WH⁢TsubscriptSIW_HTS I Witalic_H T both produce o∅⁢oI⁢oW⁢o∅subscriptsubscriptsubscriptsubscripto_ o_Io_Wo_ o∅ oitalic_I oitalic_W o∅, a log containing success confirmations for installing both drivers and CUDA. Let pW=B⁢(s→=S⁢I⁢W⁢T∣o→=o∅⁢oI⁢oW⁢o∅)subscript→conditional→subscriptsubscriptsubscriptsubscriptp_W=B( s=SIWT o=o_ o_Io_Wo_ )pitalic_W = B ( over→ start_ARG s end_ARG = S I W T ∣ over→ start_ARG o end_ARG = o∅ oitalic_I oitalic_W o∅ ). We assume for simplicity that pH′=pHsuperscriptsubscript′subscriptp_H =p_Hpitalic_H′ = pitalic_H, i.e. that the human is just as likely to think an empty log following a successful driver installation contains an error that was hidden with 2> /dev/null (pHsubscriptp_Hpitalic_H), as they are to think that an entirely empty log contains a hidden error (pH′subscript′p_H pitalic_H′). In principle, this need not be true: the human may have differing priors about the agent’s behavior in the two different circumstances. However, the algebra to reason about such a case is significantly more cumbersome, and this case reveals no fundamentally different agent behavior under our framework that isn’t present in some simpler case. We can thus write the full B as a matrix as in Table 10. Table 10: The parameterized human belief function B for the example in Section B.1 and Section B.3, expressed as a matrix (rendered as a table). Any empty cell is equal to 0. S⁢T⁢T⁢TSTTTS T T T S⁢LH⁢T⁢TsubscriptSL_HTTS Litalic_H T T S⁢L⁢T⁢TSLTTS L T T S⁢I⁢T⁢TSITTS I T T S⁢I⁢LH⁢TsubscriptSIL_HTS I Litalic_H T S⁢I⁢L⁢TSILTS I L T S⁢I⁢W⁢TSIWTS I W T S⁢I⁢WH⁢TsubscriptSIW_HTS I Witalic_H T o∅⁢o∅⁢o∅⁢o∅subscriptsubscriptsubscriptsubscripto_ o_ o_ o_ o∅ o∅ o∅ o∅ 1−pH1subscript1-p_H1 - pitalic_H pHsubscriptp_Hpitalic_H o∅⁢oL⁢o∅⁢o∅subscriptsubscriptsubscriptsubscripto_ o_Lo_ o_ o∅ oitalic_L o∅ o∅ 1 o∅⁢oI⁢o∅⁢o∅subscriptsubscriptsubscriptsubscripto_ o_Io_ o_ o∅ oitalic_I o∅ o∅ 1−pH1subscript1-p_H1 - pitalic_H pHsubscriptp_Hpitalic_H o∅⁢oI⁢oL⁢o∅subscriptsubscriptsubscriptsubscripto_ o_Io_Lo_ o∅ oitalic_I oitalic_L o∅ 1 o∅⁢oI⁢oW⁢o∅subscriptsubscriptsubscriptsubscripto_ o_Io_Wo_ o∅ oitalic_I oitalic_W o∅ pWsubscriptp_Wpitalic_W 1−pW1subscript1-p_W1 - pitalic_W We have laid the groundwork sufficiently to begin reasoning about the observation return, overestimation and underestimation error, policies which are optimal under the reward function learned by naive RLHF, and the resulting deceptive inflationand overjustification failure modes. We begin by computing the measures of interest for each state sequence, shown in Table 11. Table 11: Measures of interest for each state sequence for the example in Section B.1 and Section B.3. State sequences which produce the same observations have their GobssubscriptobsG_obsGroman_obs columns merged, since they necessarily have the same GobssubscriptobsG_obsGroman_obs. s→ sover→ start_ARG s end_ARG G⁢(s→)→G( s)G ( over→ start_ARG s end_ARG ) Gobs⁢(s→)≔s→′∼B(⋅∣O→(s→))[G⁢(s→′)]G_obs( s) *E_ s% 1.0pt B(· O( s))[G( s 1.0pt^% )]Groman_obs ( over→ start_ARG s end_ARG ) ≔ Eover→ start_ARG s end_ARG′ ∼ B ( ⋅ ∣ over→ start_ARG O end_ARG ( over→ start_ARG s end_ARG ) ) [ G ( over→ start_ARG s end_ARG′ ) ] E+(s→)≔max(0,E^+( s) (0,E+ ( over→ start_ARG s end_ARG ) ≔ max ( 0 , E−(s→)≔max(0,E^-( s) (0,E- ( over→ start_ARG s end_ARG ) ≔ max ( 0 , Gobs(s→)−G(s→))G_obs( s)-G( s))Groman_obs ( over→ start_ARG s end_ARG ) - G ( over→ start_ARG s end_ARG ) ) G(s→)−Gobs(s→))G( s)-G_obs( s))G ( over→ start_ARG s end_ARG ) - Groman_obs ( over→ start_ARG s end_ARG ) ) S⁢T⁢T⁢TSTTTS T T T 00 pH⁢G⁢(S⁢LH⁢T⁢T)+(1−pH)⁢G⁢(S⁢T⁢T⁢T)subscriptsubscript1subscriptp_HG(SL_HT)+(1-p_H)G(STTT)pitalic_H G ( S Litalic_H T T ) + ( 1 - pitalic_H ) G ( S T T T ) 00 pH⁢(5+r)subscript5p_H(5+r)pitalic_H ( 5 + r ) S⁢LH⁢T⁢TsubscriptSL_HTTS Litalic_H T T −5−r5-5-r- 5 - r =−pH⁢(5+r)absentsubscript5=-p_H(5+r)= - pitalic_H ( 5 + r ) (1−pH)⁢(5+r)1subscript5(1-p_H)(5+r)( 1 - pitalic_H ) ( 5 + r ) 00 S⁢L⁢T⁢TSLTTS L T T −55-5- 5 −55-5- 5 00 00 S⁢I⁢T⁢TSITTS I T T 1111 pH⁢G⁢(S⁢I⁢LH⁢T)+(1−pH)⁢G⁢(S⁢I⁢T⁢T)subscriptsubscript1subscriptp_HG(SIL_HT)+(1-p_H)G(SITT)pitalic_H G ( S I Litalic_H T ) + ( 1 - pitalic_H ) G ( S I T T ) 00 pH⁢(5+r)subscript5p_H(5+r)pitalic_H ( 5 + r ) S⁢I⁢LH⁢TsubscriptSIL_HTS I Litalic_H T −4−r4-4-r- 4 - r =1−pH⁢(5+r)absent1subscript5=1-p_H(5+r)= 1 - pitalic_H ( 5 + r ) (1−pH)⁢(5+r)1subscript5(1-p_H)(5+r)( 1 - pitalic_H ) ( 5 + r ) 00 S⁢I⁢L⁢TSILTS I L T −44-4- 4 −44-4- 4 00 00 S⁢I⁢W⁢TSIWTS I W T 11111111 pW⁢G⁢(S⁢I⁢W⁢T)+(1−pW)⁢G⁢(S⁢I⁢WH⁢T)subscript1subscriptsubscriptp_WG(SIWT)+(1-p_W)G(SIW_HT)pitalic_W G ( S I W T ) + ( 1 - pitalic_W ) G ( S I Witalic_H T ) 00 00 S⁢I⁢WH⁢TsubscriptSIW_HTS I Witalic_H T 11111111 =11absent11=11= 11 00 00 As an exercise, checking the computations in Table 11 is a quick way to gain some intuition for how these quantities relate. It’s further useful to speak about these quantities using their names, and work through the stories these expressions tell. Consider the E+⁢(S⁢LH⁢T⁢T)superscriptsubscriptE^+(SL_HT)E+ ( S Litalic_H T T ) cell. What does it mean that this is (1−pH)⁢(5+r)1subscript5(1-p_H)(5+r)( 1 - pitalic_H ) ( 5 + r )? E+superscriptE^+E+ is the overestimation error; (1−pH)⁢(5+r)1subscript5(1-p_H)(5+r)( 1 - pitalic_H ) ( 5 + r ) is the expected amount by which the human observer, upon seeing a completely empty log (o∅⁢o∅⁢o∅⁢o∅subscriptsubscriptsubscriptsubscripto_ o_ o_ o_ o∅ o∅ o∅ o∅), will overestimate the total reward the agent attained when those empty logs were actually produced by S⁢LH⁢T⁢TsubscriptSL_HTTS Litalic_H T T. This is a trajectory in which the agent immediately uses the 2> /dev/null trick, fails to install CUDA without logging the error, and terminates. Under what circumstances might the human overestimate the total reward when s→=S⁢LH⁢T⁢T→subscript s=SL_HTTover→ start_ARG s end_ARG = S Litalic_H T T? Upon seeing the empty log, the human thinks with probability 1−pH1subscript1-p_H1 - pitalic_H that the agent simply terminated immediately, which would be worth a total reward of 0. Since the actual total reward is −5−r5-5-r- 5 - r, this is an overestimate by 5+r55+r5 + r. The human thinks with probability pHsubscriptp_Hpitalic_H that S⁢LH⁢T⁢TsubscriptSL_HTTS Litalic_H T T occurred. This is correct, so there is no overestimation and this pHsubscriptp_Hpitalic_H case does not contribute to the overestimation error. So we have that with probability 1−pH1subscript1-p_H1 - pitalic_H, the human overestimates the total reward by 5+r55+r5 + r. We can keep going! Why is the underestimation error of S⁢I⁢W⁢TSIWTS I W T equal to 0? Because the only other trajectory with which it can be confused attains the same total reward, so regardless of how the probability mass of the human’s belief divides between them, there will be no underestimation. Can all of the zeros in the overestimation and underestimation error columns be explained this way? We now move on to consider policies rather than state sequences. Since a policy π imposes a distribution PπsuperscriptP^πPitalic_π over state sequences (the “on-policy distribution”), our policy measures are in fact exactly parallel to our state sequence measures. Each one is an expectation over the on-policy distribution of the columns of Table 11. We restrict our attention to deterministic policies which only take actions depicted in Figure 8 (i.e. that never terminate via an action other than aTsubscripta_Taitalic_T), of which there are only six in this MDP. They are enumerated, along with the policy-level measures, in Table 12. Policies will be written as a sequence of actions enclosed in brackets, omitting trailing repeated aTsubscripta_Taitalic_T actions. This is nonstandard notation in an MDP with stochastic transitions, but is unambiguous in this example, because all decisions are made before any stochasticity occurs. The policies are [aT]delimited-[]subscript[a_T][ aitalic_T ], [aH⁢aT]delimited-[]subscriptsubscript[a_Ha_T][ aitalic_H aitalic_T ], [aC⁢aT]delimited-[]subscriptsubscript[a_Ca_T][ aitalic_C aitalic_T ], [aI⁢aT]delimited-[]subscriptsubscript[a_Ia_T][ aitalic_I aitalic_T ], [aI⁢aH⁢aT]delimited-[]subscriptsubscriptsubscript[a_Ia_Ha_T][ aitalic_I aitalic_H aitalic_T ], and [aI⁢aC⁢aT]delimited-[]subscriptsubscriptsubscript[a_Ia_Ca_T][ aitalic_I aitalic_C aitalic_T ]. Table 12: Measures of interest for each policy for the example in Section B.1 and Section B.3. Each of the columns here is the on-policy average of the corresponding column in Table 11. Policies are written as sequences of actions, omitting trailing repeated aTsubscripta_Taitalic_T actions. This is nonstandard notation in an MDP with stochastic transitions, but is unambiguous in this example since all decisions are made before any stochasticity occurs. π J⁢(π)J(π)J ( π ) Jobs⁢(π)subscriptobsJ_obs(π)Jroman_obs ( π ) E¯+⁢(π)superscript¯ E^+(π)over¯ start_ARG E end_ARG+ ( π ) E¯−⁢(π)superscript¯ E^-(π)over¯ start_ARG E end_ARG- ( π ) [aT]delimited-[]subscript[a_T][ aitalic_T ] 00 −pH⁢(5+r)subscript5-p_H(5+r)- pitalic_H ( 5 + r ) 00 pH⁢(5+r)subscript5p_H(5+r)pitalic_H ( 5 + r ) [aH⁢aT]delimited-[]subscriptsubscript[a_Ha_T][ aitalic_H aitalic_T ] −5−r5-5-r- 5 - r −pH⁢(5+r)subscript5-p_H(5+r)- pitalic_H ( 5 + r ) (1−pH)⁢(5+r)1subscript5(1-p_H)(5+r)( 1 - pitalic_H ) ( 5 + r ) 00 [aC⁢aT]delimited-[]subscriptsubscript[a_Ca_T][ aitalic_C aitalic_T ] −55-5- 5 −55-5- 5 00 00 [aI⁢aT]delimited-[]subscriptsubscript[a_Ia_T][ aitalic_I aitalic_T ] 1111 1−pH⁢(5+r)1subscript51-p_H(5+r)1 - pitalic_H ( 5 + r ) 00 pH⁢(5+r)subscript5p_H(5+r)pitalic_H ( 5 + r ) [aI⁢aH⁢aT]delimited-[]subscriptsubscriptsubscript[a_Ia_Ha_T][ aitalic_I aitalic_H aitalic_T ] p⁢G⁢(S⁢I⁢WH⁢T)subscriptpG(SIW_HT)p G ( S I Witalic_H T ) p⁢Gobs⁢(S⁢I⁢WH⁢T)subscriptobssubscriptpG_obs(SIW_HT)p Groman_obs ( S I Witalic_H T ) (1−p)⁢(1−pH)⁢(5+r)11subscript5(1-p)(1-p_H)(5+r)( 1 - p ) ( 1 - pitalic_H ) ( 5 + r ) 00 +(1−p)⁢G⁢(S⁢I⁢LH⁢T)1subscript+(1-p)G(SIL_HT)+ ( 1 - p ) G ( S I Litalic_H T ) +(1−p)⁢Gobs⁢(S⁢I⁢LH⁢T)1subscriptobssubscript+(1-p)G_obs(SIL_HT)+ ( 1 - p ) Groman_obs ( S I Litalic_H T ) =11−(1−p)⁢(15+r)absent11115=11-(1-p)(15+r)= 11 - ( 1 - p ) ( 15 + r ) =11−(1−p)⁢[10+pH⁢(5+r)]absent111delimited-[]10subscript5=11-(1-p) [10+p_H(5+r) ]= 11 - ( 1 - p ) [ 10 + pitalic_H ( 5 + r ) ] [aI⁢aC⁢aT]delimited-[]subscriptsubscriptsubscript[a_Ia_Ca_T][ aitalic_I aitalic_C aitalic_T ] p⁢G⁢(S⁢I⁢W⁢T)pG(SIWT)p G ( S I W T ) p⁢Gobs⁢(S⁢I⁢W⁢T)subscriptobspG_obs(SIWT)p Groman_obs ( S I W T ) 00 00 +(1−p)⁢G⁢(S⁢I⁢L⁢T)1+(1-p)G(SILT)+ ( 1 - p ) G ( S I L T ) +(1−p)⁢Gobs⁢(S⁢I⁢L⁢T)1subscriptobs+(1-p)G_obs(SILT)+ ( 1 - p ) Groman_obs ( S I L T ) =11−(1−p)⋅15absent11⋅115=11-(1-p)· 15= 11 - ( 1 - p ) ⋅ 15 =11−(1−p)⋅15absent11⋅115=11-(1-p)· 15= 11 - ( 1 - p ) ⋅ 15 With this we have everything we need to characterize optimal policies under the reward function learned by a naive application of RLHF (“policies selected by RLHF”). By Proposition 4.1, we know that if POsubscriptP_OPitalic_O is deterministic, as in this example, RLHF selects policies which maximize JobssubscriptobsJ_obsJroman_obs. In order to understand the behavior of these policies, we’l also need to determine the true optimal policies, i.e. those which maximize J. We’l proceed in cases, only considering boundary cases (specific measure-zero parameter values for which the result is different) insofar as they are interesting. Case 1: p>1313p> 13p > divide start_ARG 1 end_ARG start_ARG 3 end_ARG. If p>1313p> 13p > divide start_ARG 1 end_ARG start_ARG 3 end_ARG, the CUDA install (with default logging, aCsubscripta_Caitalic_C) is likely enough to succeed that it’s worth attempting it: p⋅R⁢(W)+(1−p)⋅R⁢(L)>0⋅10 p· R(W)+(1-p)· R(L)>0p ⋅ R ( W ) + ( 1 - p ) ⋅ R ( L ) > 0. It also immediately follows that J⁢([aI⁢aC⁢aT])=Jobs⁢([aI⁢aC⁢aT])=11−(1−p)⋅15>1.delimited-[]subscriptsubscriptsubscriptsubscriptobsdelimited-[]subscriptsubscriptsubscript11⋅1151 J([a_Ia_Ca_T])=J_obs([a_Ia_Ca_T])=1% 1-(1-p)· 15>1.J ( [ aitalic_I aitalic_C aitalic_T ] ) = Jroman_obs ( [ aitalic_I aitalic_C aitalic_T ] ) = 11 - ( 1 - p ) ⋅ 15 > 1 . This allows us to eliminate policies [aT]delimited-[]subscript[a_T][ aitalic_T ], [aH⁢aT]delimited-[]subscriptsubscript[a_Ha_T][ aitalic_H aitalic_T ], [aC⁢aT]delimited-[]subscriptsubscript[a_Ca_T][ aitalic_C aitalic_T ], and [aI⁢aT]delimited-[]subscriptsubscript[a_Ia_T][ aitalic_I aitalic_T ], which all have J≤11J≤ 1J ≤ 1 and Jobs≤1subscriptobs1J_obs≤ 1Jroman_obs ≤ 1. None of them can thus be J-optimal or JobssubscriptobsJ_obsJroman_obs-optimal. All that remains is to compare J and JobssubscriptobsJ_obsJroman_obs for [aI⁢aH⁢aT]delimited-[]subscriptsubscriptsubscript[a_Ia_Ha_T][ aitalic_I aitalic_H aitalic_T ] and [aI⁢aC⁢aT]delimited-[]subscriptsubscriptsubscript[a_Ia_Ca_T][ aitalic_I aitalic_C aitalic_T ]. We can check the sign of the differences of these pairs of values, starting with J. J⁢([aI⁢aC⁢aT])−J⁢([aI⁢aH⁢aT])=(1−p)⁢r.delimited-[]subscriptsubscriptsubscriptdelimited-[]subscriptsubscriptsubscript1 J([a_Ia_Ca_T])-J([a_Ia_Ha_T])=(1-p)r.J ( [ aitalic_I aitalic_C aitalic_T ] ) - J ( [ aitalic_I aitalic_H aitalic_T ] ) = ( 1 - p ) r . Since p is a probability and r is nonnegative, this value is positive (and thus [aI⁢aC⁢aT]delimited-[]subscriptsubscriptsubscript[a_Ia_Ca_T][ aitalic_I aitalic_C aitalic_T ] is preferred to [aI⁢aH⁢aT]delimited-[]subscriptsubscriptsubscript[a_Ia_Ha_T][ aitalic_I aitalic_H aitalic_T ] by the human) if and only if p<11p<1p < 1 and r>00r>0r > 0. Jobs⁢([aI⁢aH⁢aT])−Jobs⁢([aI⁢aC⁢aT])=(1−p)⁢[5−pH⁢(5+r)].subscriptobsdelimited-[]subscriptsubscriptsubscriptsubscriptobsdelimited-[]subscriptsubscriptsubscript1delimited-[]5subscript5 J_obs([a_Ia_Ha_T])-J_obs% ([a_Ia_Ca_T])=(1-p) [5-p_H(5+r) ].Jroman_obs ( [ aitalic_I aitalic_H aitalic_T ] ) - Jroman_obs ( [ aitalic_I aitalic_C aitalic_T ] ) = ( 1 - p ) [ 5 - pitalic_H ( 5 + r ) ] . This value is positive (and thus [aI⁢aH⁢aT]delimited-[]subscriptsubscriptsubscript[a_Ia_Ha_T][ aitalic_I aitalic_H aitalic_T ] is the policy RLHF selects) if and only if p<11p<1p < 1 and pH<55+rsubscript55p_H< 55+rpitalic_H < divide start_ARG 5 end_ARG start_ARG 5 + r end_ARG. If p=11p=1p = 1, then both differences are 0, and both J and JobssubscriptobsJ_obsJroman_obs are indifferent between the two policies. This makes sense, as they differ only in the case where the CUDA installation fails; this happens with probability 1−p=0101-p=01 - p = 0 when p=11p=1p = 1. Now suppose p<11p<1p < 1. If r=00r=0r = 0, then the human is indifferent between the two policies. This also makes sense, as r is meant to quantify the extent to which the human dislikes suppressed failures; if it’s zero, then the human doesn’t care. However, if pH<55+rsubscript55p_H< 55+rpitalic_H < divide start_ARG 5 end_ARG start_ARG 5 + r end_ARG, then Jobs⁢([aI⁢aH⁢aT])>Jobs⁢([aI⁢aH⁢aT])subscriptobsdelimited-[]subscriptsubscriptsubscriptsubscriptobsdelimited-[]subscriptsubscriptsubscriptJ_obs([a_Ia_Ha_T])>J_obs([a_Ia_Ha_% T])Jroman_obs ( [ aitalic_I aitalic_H aitalic_T ] ) > Jroman_obs ( [ aitalic_I aitalic_H aitalic_T ] ), and thus RLHF favors the 2> /dev/null policy [aI⁢aH⁢aT]delimited-[]subscriptsubscriptsubscript[a_Ia_Ha_T][ aitalic_I aitalic_H aitalic_T ]. If p<11p<1p < 1, r>00r>0r > 0, and pH<55+rsubscript55p_H< 55+rpitalic_H < divide start_ARG 5 end_ARG start_ARG 5 + r end_ARG, then we have that J⁢([aI⁢aC⁢aT])>J⁢([aI⁢aH⁢aT])delimited-[]subscriptsubscriptsubscriptdelimited-[]subscriptsubscriptsubscriptJ([a_Ia_Ca_T])>J([a_Ia_Ha_T])J ( [ aitalic_I aitalic_C aitalic_T ] ) > J ( [ aitalic_I aitalic_H aitalic_T ] ) but Jobs⁢([aI⁢aC⁢aT])>Jobs⁢([aI⁢aH⁢aT])subscriptobsdelimited-[]subscriptsubscriptsubscriptsubscriptobsdelimited-[]subscriptsubscriptsubscriptJ_obs([a_Ia_Ca_T])>J_obs([a_Ia_Ha_% T])Jroman_obs ( [ aitalic_I aitalic_C aitalic_T ] ) > Jroman_obs ( [ aitalic_I aitalic_H aitalic_T ] ). Thus RLHF will select the 2> /dev/null policy [aI⁢aH⁢aT]delimited-[]subscriptsubscriptsubscript[a_Ia_Ha_T][ aitalic_I aitalic_H aitalic_T ], and by Theorem 4.5, since [aI⁢aH⁢aT]delimited-[]subscriptsubscriptsubscript[a_Ia_Ha_T][ aitalic_I aitalic_H aitalic_T ] is not J-optimal, then relative to [aI⁢aC⁢aT]delimited-[]subscriptsubscriptsubscript[a_Ia_Ca_T][ aitalic_I aitalic_C aitalic_T ], it must exhibit deceptive inflation, overjustification, or both. Intuitively, we should be suspicious that deceptive inflation is at play whenever the agent hides information from the human. Indeed, referencing Table 12, we have E¯+⁢([aI⁢aH⁢aT])=(1−p)⁢(1−pH)⁢(5+r)>0=E¯+⁢([aI⁢aC⁢aT])superscript¯delimited-[]subscriptsubscriptsubscript11subscript50superscript¯delimited-[]subscriptsubscriptsubscript E^+([a_Ia_Ha_T])=(1-p)(1-p_H)(5+r)>0= E^+([a_% Ia_Ca_T])over¯ start_ARG E end_ARG+ ( [ aitalic_I aitalic_H aitalic_T ] ) = ( 1 - p ) ( 1 - pitalic_H ) ( 5 + r ) > 0 = over¯ start_ARG E end_ARG+ ( [ aitalic_I aitalic_C aitalic_T ] ). Together with Jobs⁢([aI⁢aH⁢aT])>Jobs⁢([aI⁢aC⁢aT])subscriptobsdelimited-[]subscriptsubscriptsubscriptsubscriptobsdelimited-[]subscriptsubscriptsubscriptJ_obs([a_Ia_Ha_T])>J_obs([a_Ia_Ca_% T])Jroman_obs ( [ aitalic_I aitalic_H aitalic_T ] ) > Jroman_obs ( [ aitalic_I aitalic_C aitalic_T ] ), this satisfies the conditions of Definition 4.3, and thus this is an instance of deceptive inflation. If p<11p<1p < 1, r>00r>0r > 0, and pH>55+rsubscript55p_H> 55+rpitalic_H > divide start_ARG 5 end_ARG start_ARG 5 + r end_ARG, then [aI⁢aC⁢aT]delimited-[]subscriptsubscriptsubscript[a_Ia_Ca_T][ aitalic_I aitalic_C aitalic_T ] is optimal under both J and JobssubscriptobsJ_obsJroman_obs, and in this case, RLHF selects the true optimal policy. Case 2: p<1313p< 13p < divide start_ARG 1 end_ARG start_ARG 3 end_ARG. In this case, the CUDA install is not likely enough to succeed to be worth attempting (under the true reward function). Mathematically, J⁢([aI⁢aH⁢aT])≤J⁢([aI⁢aC⁢aT])<1=J⁢([aI⁢aT])delimited-[]subscriptsubscriptsubscriptdelimited-[]subscriptsubscriptsubscript1delimited-[]subscriptsubscriptJ([a_Ia_Ha_T])≤ J([a_Ia_Ca_T])<1=J([a_Ia_T])J ( [ aitalic_I aitalic_H aitalic_T ] ) ≤ J ( [ aitalic_I aitalic_C aitalic_T ] ) < 1 = J ( [ aitalic_I aitalic_T ] ). The other three policies are always worse under J than [aI⁢aT]delimited-[]subscriptsubscript[a_Ia_T][ aitalic_I aitalic_T ], so we have our optimal policy π∗=[aI⁢aT]superscriptdelimited-[]subscriptsubscriptπ^*=[a_Ia_T]π∗ = [ aitalic_I aitalic_T ]. However, Jobs⁢([aI⁢aH⁢aT])−Jobs⁢([aI⁢aT])=p⁢(10+pH⁢(5+r)),subscriptobsdelimited-[]subscriptsubscriptsubscriptsubscriptobsdelimited-[]subscriptsubscript10subscript5 J_obs([a_Ia_Ha_T])-J_obs% ([a_Ia_T])=p(10+p_H(5+r)),Jroman_obs ( [ aitalic_I aitalic_H aitalic_T ] ) - Jroman_obs ( [ aitalic_I aitalic_T ] ) = p ( 10 + pitalic_H ( 5 + r ) ) , which is always positive unless p=00p=0p = 0. As long as p>00p>0p > 0, i.e. installing CUDA is actually possible at all, then regardless of everything else, Jobs⁢([aI⁢aH⁢aT])>Jobs⁢([aI⁢aT])subscriptobsdelimited-[]subscriptsubscriptsubscriptsubscriptobsdelimited-[]subscriptsubscriptJ_obs([a_Ia_Ha_T])>J_obs([a_Ia_T])Jroman_obs ( [ aitalic_I aitalic_H aitalic_T ] ) > Jroman_obs ( [ aitalic_I aitalic_T ] ). We can interpret this reasonably: however good or bad it might look to terminate after installing drivers (S⁢I⁢T⁢TSITTS I T T), using the 2> /dev/null trick will either produce the same observation as S⁢I⁢T⁢TSITTS I T T (and thus the same feedback distribution) or will actually visibly install CUDA successfully—the most lucrative possible source of positive feedback. Even if there’s only a sliver of a chance of success and it’s a catastrophe to have a silent failure (i.e. r≫0much-greater-than0r 0r ≫ 0), however likely you are to infer that the lack of a log message about CUDA means the agent hid a failure, it will penalize [aI⁢aT]delimited-[]subscriptsubscript[a_Ia_T][ aitalic_I aitalic_T ] more than it will penalize [aI⁢aH⁢aT]delimited-[]subscriptsubscriptsubscript[a_Ia_Ha_T][ aitalic_I aitalic_H aitalic_T ]: where [aI⁢aH⁢aT]delimited-[]subscriptsubscriptsubscript[a_Ia_Ha_T][ aitalic_I aitalic_H aitalic_T ] produces this ambiguous observation with probability 1−p11-p1 - p (when it fails), [aI⁢aT]delimited-[]subscriptsubscript[a_Ia_T][ aitalic_I aitalic_T ] will always produce the ambiguous observation. This means that when 0<p<130130<p< 130 < p < divide start_ARG 1 end_ARG start_ARG 3 end_ARG, it is impossible to recover the true optimal policy with naive RLHF. Which policies can possibly be JobssubscriptobsJ_obsJroman_obs-optimal for some setting of the parameters? We can similarly rule out [aT]delimited-[]subscript[a_T][ aitalic_T ] and [aH⁢aT]delimited-[]subscriptsubscript[a_Ha_T][ aitalic_H aitalic_T ] for 0<p<130130<p< 130 < p < divide start_ARG 1 end_ARG start_ARG 3 end_ARG: Jobs⁢([aI⁢aH⁢aT])−Jobs⁢([aI⁢aT])=p⁢(10+pH⁢(5+r))>0.subscriptobsdelimited-[]subscriptsubscriptsubscriptsubscriptobsdelimited-[]subscriptsubscript10subscript50 J_obs([a_Ia_Ha_T])-J_obs% ([a_Ia_T])=p(10+p_H(5+r))>0.Jroman_obs ( [ aitalic_I aitalic_H aitalic_T ] ) - Jroman_obs ( [ aitalic_I aitalic_T ] ) = p ( 10 + pitalic_H ( 5 + r ) ) > 0 . We can rule out [aC⁢aT]delimited-[]subscriptsubscript[a_Ca_T][ aitalic_C aitalic_T ] by comparison to [aI⁢aC⁢aT]delimited-[]subscriptsubscriptsubscript[a_Ia_Ca_T][ aitalic_I aitalic_C aitalic_T ]: Jobs⁢([aI⁢aC⁢aT])−Jobs⁢([aC⁢aT])=16−(1−p)⁢15>0subscriptobsdelimited-[]subscriptsubscriptsubscriptsubscriptobsdelimited-[]subscriptsubscript161150J_obs([a_Ia_Ca_T])-J_obs([a_Ca_T])% =16-(1-p)15>0Jroman_obs ( [ aitalic_I aitalic_C aitalic_T ] ) - Jroman_obs ( [ aitalic_C aitalic_T ] ) = 16 - ( 1 - p ) 15 > 0. So we are left with only [aI⁢aH⁢aT]delimited-[]subscriptsubscriptsubscript[a_Ia_Ha_T][ aitalic_I aitalic_H aitalic_T ] and [aI⁢aC⁢aT]delimited-[]subscriptsubscriptsubscript[a_Ia_Ca_T][ aitalic_I aitalic_C aitalic_T ] as candidate JobssubscriptobsJ_obsJroman_obs-optimal policies. As in Case 1, we find that Jobs⁢([aI⁢aH⁢aT])>Jobs⁢([aI⁢aT])subscriptobsdelimited-[]subscriptsubscriptsubscriptsubscriptobsdelimited-[]subscriptsubscriptJ_obs([a_Ia_Ha_T])>J_obs([a_Ia_T])Jroman_obs ( [ aitalic_I aitalic_H aitalic_T ] ) > Jroman_obs ( [ aitalic_I aitalic_T ] ) if and only if p=11p=1p = 1 or pH<55+rsubscript55p_H< 55+rpitalic_H < divide start_ARG 5 end_ARG start_ARG 5 + r end_ARG. In case 2 we have assumed p<1313p< 13p < divide start_ARG 1 end_ARG start_ARG 3 end_ARG, leaving only the pHsubscriptp_Hpitalic_H condition. If pH<55+rsubscript55p_H< 55+rpitalic_H < divide start_ARG 5 end_ARG start_ARG 5 + r end_ARG, then RLHF selects [aI⁢aH⁢aT]delimited-[]subscriptsubscriptsubscript[a_Ia_Ha_T][ aitalic_I aitalic_H aitalic_T ]. As in Case 1, this is deceptive inflationrelative to π∗=[aI⁢aT]superscriptdelimited-[]subscriptsubscriptπ^*=[a_Ia_T]π∗ = [ aitalic_I aitalic_T ], because E¯+⁢([aI⁢aH⁢aT])=(1−p)⁢(1−pH)⁢(5+r)>0=E¯+⁢(π∗).superscript¯delimited-[]subscriptsubscriptsubscript11subscript50superscript¯superscript E^+([a_Ia_Ha_T])=(1-p)(1-p_H)(5+r)>0=% E^+(π^*).over¯ start_ARG E end_ARG+ ( [ aitalic_I aitalic_H aitalic_T ] ) = ( 1 - p ) ( 1 - pitalic_H ) ( 5 + r ) > 0 = over¯ start_ARG E end_ARG+ ( π∗ ) . If pH>55+rsubscript55p_H> 55+rpitalic_H > divide start_ARG 5 end_ARG start_ARG 5 + r end_ARG, then RLHF selects [aI⁢aC⁢aT]delimited-[]subscriptsubscriptsubscript[a_Ia_Ca_T][ aitalic_I aitalic_C aitalic_T ]. Because this policy is not J-optimal, by Theorem 4.5, we must have deceptive inflation, overjustification, or both. Which is it? Here the optimal policy is to terminate after installing drivers, [aI⁢aT]delimited-[]subscriptsubscript[a_Ia_T][ aitalic_I aitalic_T ]. However, pH>55+rsubscript55p_H> 55+rpitalic_H > divide start_ARG 5 end_ARG start_ARG 5 + r end_ARG. This can be rewritten as pH⁢(5+r)>5subscript55p_H(5+r)>5pitalic_H ( 5 + r ) > 5. We have seen this expression pH⁢(5+r)subscript5p_H(5+r)pitalic_H ( 5 + r ) before; it is the underestimation error incurred on s→=S⁢I⁢T⁢T→ s=SITTover→ start_ARG s end_ARG = S I T T and therefore also the average underestimation error of policy [aI⁢aT]delimited-[]subscriptsubscript[a_Ia_T][ aitalic_I aitalic_T ]. So here the underestimation error on the optimal policy—that is, the risk that the human misunderstands optimal behavior (terminating after installing driver) as undesired behavior (attempting a CUDA install that was unlikely to work and hiding the mistake)—is severe enough that the agent opts instead for [aI⁢aC⁢aT]delimited-[]subscriptsubscriptsubscript[a_Ia_Ca_T][ aitalic_I aitalic_C aitalic_T ], a worse policy that attempts the ill-fated CUDA installation only to prove that it wasn’t doing so secretly. In qualitative terms, this is quintessential overjustification behavior. Indeed, relative to reference policy π∗=[aI⁢aT]superscriptdelimited-[]subscriptsubscriptπ^*=[a_Ia_T]π∗ = [ aitalic_I aitalic_T ], we have E¯−⁢([aI⁢aC⁢aT])=0<pH⁢(5+r)=E¯−⁢(π∗)superscript¯delimited-[]subscriptsubscriptsubscript0subscript5superscript¯superscript E^-([a_Ia_Ca_T])=0<p_H(5+r)= E^-% (π^*)over¯ start_ARG E end_ARG- ( [ aitalic_I aitalic_C aitalic_T ] ) = 0 < pitalic_H ( 5 + r ) = over¯ start_ARG E end_ARG- ( π∗ ) J⁢([aI⁢aC⁢aT])=11−(1−p)⋅15<1=J⁢(π∗),delimited-[]subscriptsubscriptsubscript11⋅1151superscript J([a_Ia_Ca_T])=11-(1-p)· 15<1=J(π^*),J ( [ aitalic_I aitalic_C aitalic_T ] ) = 11 - ( 1 - p ) ⋅ 15 < 1 = J ( π∗ ) , and thus by Definition 4.4, this is overjustification. B.4 Ambiguity in Section 4.4 examples when modeling partial observability Consider the example in Fig. 4A when modeling partial observability as in Section 5. By Theorem 5.2, the ambiguity in the return function leaving the choice probabilities invariant is given by ker⁡∩im⁡kernelim B ker B ∩ im Γ. Let R′=(0,0,R′⁢(W),0,R′⁢(WH),0,0)∈ℝS,I,W,L,WH,LH,Tsuperscript′00superscript′0superscript′subscript00superscriptℝsubscriptsubscriptR =(0,0,R (W),0,R (W_H),0,0)∈ R^\S,I,W,L% ,W_H,L_H,T\R′ = ( 0 , 0 , R′ ( W ) , 0 , R′ ( Witalic_H ) , 0 , 0 ) ∈ blackboard_R S , I , W , L , Witalic_H , Litalic_H , T be a reward function that we want to parameterize such that G′≔⋅R′≔superscript′⋅superscript′G · R G′ ≔ Γ ⋅ R′ ends up in the ambiguity; here, R′ is interpreted as a column vector. We want ⋅G′=0⋅superscript′0 B· G =0B ⋅ G′ = 0. Since the observation sequences o→=o∅⁢o∅⁢o∅⁢o∅→subscriptsubscriptsubscriptsubscript o=o_ o_ o_ o_ over→ start_ARG o end_ARG = o∅ o∅ o∅ o∅, o→=o∅⁢oL⁢o∅⁢o∅→subscriptsubscriptsubscriptsubscript o=o_ o_Lo_ o_ over→ start_ARG o end_ARG = o∅ oitalic_L o∅ o∅, o→=o∅⁢oI⁢o∅⁢o∅→subscriptsubscriptsubscriptsubscript o=o_ o_Io_ o_ over→ start_ARG o end_ARG = o∅ oitalic_I o∅ o∅, or o→=o∅⁢oI⁢oL⁢o∅→subscriptsubscriptsubscriptsubscript o=o_ o_Io_Lo_ over→ start_ARG o end_ARG = o∅ oitalic_I oitalic_L o∅ all cannot involve the states W or WHsubscriptW_HWitalic_H, it is clear that they have zero expected return (⋅G′)⁢(o→)⋅superscript′→( B· G )( o)( B ⋅ G′ ) ( over→ start_ARG o end_ARG ). Set pH′≔B⁢(S⁢I⁢WH⁢T∣o∅⁢oI⁢oW⁢o∅)≔superscriptsubscript′conditionalsubscriptsubscriptsubscriptsubscriptsubscriptp_H B (SIW_HT o_ o_Io_Wo_% )pitalic_H′ ≔ B ( S I Witalic_H T ∣ o∅ oitalic_I oitalic_W o∅ ). Then the condition that ⋅G′=0⋅superscript′0 B· G =0B ⋅ G′ = 0 is equivalent to: 00 0 =(⋅G′)⁢(o∅⁢oI⁢oW⁢o∅)=s→∼B⁢(s→∣o∅⁢oI⁢oW⁢o∅)[G′⁢(s→)]absent⋅superscript′subscriptsubscriptsubscriptsubscriptsubscriptsimilar-to→conditional→subscriptsubscriptsubscriptsubscriptsuperscript′→ = ( B· G )(o_% o_Io_Wo_ )= *E_ s B(% s o_ o_Io_Wo_ ) [G ( s)% ]= ( B ⋅ G′ ) ( o∅ oitalic_I oitalic_W o∅ ) = Eover→ start_ARG s end_ARG ∼ B ( over→ start_ARG s end_ARG ∣ o start_POSTSUBSCRIPT ∅ oitalic_I oitalic_W o∅ ) end_POSTSUBSCRIPT [ G′ ( over→ start_ARG s end_ARG ) ] =pH′⋅G′⁢(S⁢I⁢WH⁢T)+(1−pH′)⋅G′⁢(S⁢I⁢W⁢T)=pH′⋅R′⁢(WH)+(1−pH′)⋅R′⁢(W).absent⋅superscriptsubscript′subscript⋅1superscriptsubscript′⋅superscriptsubscript′subscript⋅1superscriptsubscript′ =p_H · G (SIW_HT)+(1-p_H )· G% (SIWT)=p_H · R (W_H)+(1-p_H )· R% (W).= pitalic_H′ ⋅ G′ ( S I Witalic_H T ) + ( 1 - pitalic_H′ ) ⋅ G′ ( S I W T ) = pitalic_H′ ⋅ R′ ( Witalic_H ) + ( 1 - pitalic_H′ ) ⋅ R′ ( W ) . Thus, if R′⁢(W)=pH′pH′−1⁢R′⁢(WH)superscript′subscript′subscript′1superscript′subscriptR (W)= p_H p_H -1R (W_H)R′ ( W ) = divide start_ARG pitalic_H′ end_ARG start_ARG pitalic_H′ - 1 end_ARG R′ ( Witalic_H ), then G′∈ker⁡∩im⁡superscript′kernelimG ∈ B % G′ ∈ ker B ∩ im Γ, meaning that R+R′+R R + R′ has the same choice probabilities as R and is thus fully feedback-compatible. In particular, if R′⁢(WH)≫0much-greater-thansuperscript′subscript0R (W_H) 0R′ ( Witalic_H ) ≫ 0 is sufficiently large, then in subsequent policy optimization, there is an incentive to hide the mistakes and πHsubscript _Hπitalic_H will be selected, which is suboptimal with respect to the true reward function R. Thus Fig. 4A still retains dangerous ambiguity when modeling partial observability. However, the example in Fig. 4B leads to no ambiguity when partial observability is correctly modeled. To show this in detail, let G′=⁡(R′)∈ker⁡∩im⁡superscript′kernelimG = (R )∈ % B G′ = Γ ( R′ ) ∈ ker B ∩ im Γ. We need to show G′=0superscript′0G =0G′ = 0. Since the human is only uncertain about the state sequences corresponding to the observation sequence o∅⁢oI⁢o∅⁢o∅subscriptsubscriptsubscriptsubscripto_ o_Io_ o_ o∅ oitalic_I o∅ o∅, the condition ⋅G′=0⋅superscript′0 B· G =0B ⋅ G′ = 0 already implies G′⁢(s→)=0superscript′→0G ( s)=0G′ ( over→ start_ARG s end_ARG ) = 0 for all state sequences except S⁢I⁢W⁢TSIWTS I W T and S⁢I⁢T⁢TSITTS I T T. From (⋅G′)⁢(o∅⁢oI⁢o∅⁢o∅)=0⋅superscript′subscriptsubscriptsubscriptsubscript0( B· G )(o_ o_Io_ o_% )=0( B ⋅ G′ ) ( o∅ oitalic_I o∅ o∅ ) = 0, one then obtains the equation (1−pD)⋅(R′⁢(S)+R′⁢(I)+2⁢R′⁢(T))+pD⋅(R′⁢(S)+R′⁢(I)+R′⁢(W)+R′⁢(T))=0.⋅1subscriptsuperscript′2superscript′⋅subscriptsuperscript′superscript′0(1-p_D)· (R (S)+R (I)+2R (T) )+p_D% · (R (S)+R (I)+R (W)+R (T) )=0.( 1 - pitalic_D ) ⋅ ( R′ ( S ) + R′ ( I ) + 2 R′ ( T ) ) + pitalic_D ⋅ ( R′ ( S ) + R′ ( I ) + R′ ( W ) + R′ ( T ) ) = 0 . (5) Thus, if one of the two state sequences involved has zero return, then the other has as well, assuming that 0≠pD≠10subscript10≠ p_D≠ 10 ≠ pitalic_D ≠ 1, and we are done. To show this, we use that all other state sequences have zero return: R′⁢(S)+3⁢R′⁢(T)=0=R′⁢(S)+R′⁢(L)+2⁢R′⁢(T)superscript′3superscript′0superscript′2superscript′R (S)+3R (T)=0=R (S)+R (L)+2R (T)R′ ( S ) + 3 R′ ( T ) = 0 = R′ ( S ) + R′ ( L ) + 2 R′ ( T ), from which R′⁢(L)=R′⁢(T)superscript′R (L)=R (T)R′ ( L ) = R′ ( T ) follows. Then, from R′⁢(S)+R′⁢(I)+R′⁢(L)+R′⁢(T)=0superscript′superscript′0R (S)+R (I)+R (L)+R (T)=0R′ ( S ) + R′ ( I ) + R′ ( L ) + R′ ( T ) = 0, substituting the previous result gives R′⁢(S)+R′⁢(I)+2⁢R′⁢(T)=0superscript′2superscript′0R (S)+R (I)+2R (T)=0R′ ( S ) + R′ ( I ) + 2 R′ ( T ) = 0, and so Equation (5) results in R′⁢(S)+R′⁢(I)+R′⁢(W)+R′⁢(T)=0superscript′superscript′0R (S)+R (I)+R (W)+R (T)=0R′ ( S ) + R′ ( I ) + R′ ( W ) + R′ ( T ) = 0. Overall, this shows G′=⁡(R′)=0superscript′0G = (R )=0G′ = Γ ( R′ ) = 0, and so ker⁡∩im⁡=0kernelim0 B % =\0\ker B ∩ im Γ = 0 . B.5 Experimental details Here, we explain more experimental details for the results in Table 1, reproduced here as Table 13, and Figure 5. Table 13: Experiments showing improved performance of po-aware RLHF Ex. p phidesubscripthidep_hidephide pdefaultsubscriptdefaultp_defaultpdefault model action E¯+superscript¯ E^+over¯ start_ARG E end_ARG+ dec. infl. E¯−superscript¯ E^-over¯ start_ARG E end_ARG- overj. optimal A 0.5 0.5 N/A naive aHsubscripta_Haitalic_H 1.5 ✓ 0 × × A 0.5 0.5 N/A po-aware aHsubscripta_Haitalic_H 1.5 ✓ 0 × × A 0.1 0.9 N/A naive aCsubscripta_Caitalic_C 0 × 0 ✓ × A 0.1 0.9 N/A po-aware aTsubscripta_Taitalic_T 0 × 5.4 × ✓ B 0.5 N/A 0.9 naive aTsubscripta_Taitalic_T 4.5 ✓ 0 ✓ × B 0.5 N/A 0.9 po-aware aDsubscripta_Daitalic_D 0 × 0.25 × ✓ B 0.5 N/A 0.1 naive aVsubscripta_Vaitalic_V 0 × 0 ✓ × B 0.5 N/A 0.1 po-aware aDsubscripta_Daitalic_D 0 × 2.25 × ✓ The leftmost column (“Ex.” for “example”) corresponds to Examples A and B in Figure 4. p is the success probability upon attempting to install Cuda or NumPy in state I, see Figure 7. phidesubscripthidep_hidephide in Example A is the human’s belief probability that the agent hid the error message if there is no output after nvidia-driver installation. Similarly, pdefaultsubscriptdefaultp_defaultpdefault in Example B is the human’s belief probability that installation was done with default settings if there is no further output after Python installation. Note that lines one and two in the table also correspond to Example B.1, lines three and four to Example B.2, and lines five and six to the first half and seven and eight to the second half of Example B.3, respectively. In all the results in the table, we set the penalty to r=11r=1r = 1. The “model” column has value “naive” if the reward learning algorithm is classical RLHF (erroneously assuming full observability) as in Christiano et al. [2017], and “po-aware” if the human’s partial observability is correctly modeled as in Section C.3. We initialize the reward function as a list of rewards of states and train it by logistic regression using a dataset that consists of all pairs of state sequences together with the human’s choice probabilities under partial observations. This leads to 28 pairs of distinct trajectories together with choice probabilities. We train the reward model for 300 epochs over a shuffled dataset of 13.5 copies of the 28 pairs with the Adam optimizer, for a total of 113400 training updates. Once we have the resulting reward model, we use value iteration to find its deterministic optimal policy. All policies choose to install the nvidia-driver (in Example A) and Python (in Example B), and differ in their action in state I, which is given in the column “action”. We compute the overestimation error and underestimation error of the resulting policies analytically using the hardcoded environment dynamics, true reward function, observation function, and human belief matrix BB. This is given in columns E¯+superscript¯ E^+over¯ start_ARG E end_ARG+ and E¯−superscript¯ E^-over¯ start_ARG E end_ARG-. Note that these are averages over 10 entire training runs, though since they always result in the same learned policy, there is no variation and we do not state any uncertainty. The columns “dec. infl.”, “overj.”, and “optimal” state whether deceptive inflation or overjustification occurs with the learned policy, and whether it is optimal according to the true human’s reward function. For the results in Figure 5, we use largely the same procedure as for the table. Instead of fixing the reward penalty r or the belief probabilities phidesubscripthidep_hidephide and pdefaultsubscriptdefaultp_defaultpdefault, we vary them as hyperparameters for the plots, we fix p to p=0.50.5p=0.5p = 0.5, and we restrict ourselves to the analysis of “naive” RLHF. Appendix C Modeling the Human in Partially Observable RLHF In this appendix, we develop the theory of RLHF with appropriately modeled partial observability, including full proofs of all theorems. In Section C.1, we explain how the human can arrive at the belief B⁢(s→∣o→)conditional→B( s o)B ( over→ start_ARG s end_ARG ∣ over→ start_ARG o end_ARG ) via Bayesian updates. The main theory and the main paper in general do not depend on this specific form of the human’s belief, but some examples in the appendix do. In Section C.2 we then explain our main result: the ambiguity and identifiability of both reward and return functions under observed sequence comparisons. In Section C.3, we then explain that this theorem means that one could in principle design a practical reward learning algorithm that converges on the correct reward function up to the ambiguity characterized in the section before, if the human’s belief kernel B⁢(s→∣o→)conditional→B( s o)B ( over→ start_ARG s end_ARG ∣ over→ start_ARG o end_ARG ) is fully known. In Section C.4, we generalize the theory to the case that the human’s observations are not necessarily known to the learning system and again characterize precisely when the return function is identifiable from sequence comparisons. We then consider special cases in Section C.5, where we show that the fully observable case is covered by our theory, that a deterministic observation kernel PO→subscript→P_ OPover→ start_ARG O end_ARG usually leads to non-injective belief matrix BB, and that “noise” in the observation kernel PO→subscript→P_ OPover→ start_ARG O end_ARG leads, under appropriate assumptions, to the identifiability of the return function. Our identifiability results require that the learning system knows the human’s belief kernel B⁢(s→∣o→)conditional→B( s o)B ( over→ start_ARG s end_ARG ∣ over→ start_ARG o end_ARG ). In Section C.6, we then show that these results are robust to slight misspecifications: a bound in the error in the specified belief leads to a corresponding bound in the error of the policy evaluation function used for subsequent reinforcement learning. In Section C.7, we then provide a very preliminary characterization of the ambiguity in the return function under special cases. Finally, in Section C.8, we study examples of identifiability and non-identifiability of the return function for the case that we do model the human’s partial observability correctly. This reveals qualitatively interesting cases of identifiability, even when BB is not injective, and catastrophic cases of non-identifiability. C.1 The Belief over the State Sequence for Rational Humans Before we dive into the main theory, we want to explain how the human can iteratively compute the posterior of the state sequence given an observation sequence with successively new observations. This is done by defining a Bayesian network for the joint probability of policy, states, actions, and observations, and doing Bayesian inference over this Bayesian network. The details of this subsection are only relevant for a few sections in the appendix since it is usually enough to assume that the posterior belief exists. Additionally, in the core theory, we do not even assume that B⁢(s→∣o→)conditional→B( s o)B ( over→ start_ARG s end_ARG ∣ over→ start_ARG o end_ARG ) is a posterior: it is simply any probability distribution. The reason why it can still be interesting to analyze the case when the human is a rational Bayesian reasoner is that one can then analyze RLHF under generous assumptions to the human. We model the human to have a joint distribution B⁢(π,s→,a→,o→)→B(π, s, a, o)B ( π , over→ start_ARG s end_ARG , over→ start_ARG a end_ARG , over→ start_ARG o end_ARG ) over the policy π, state sequence s→=s0,…,sT→subscript0…subscript s=s_0,…,s_Tover→ start_ARG s end_ARG = s0 , … , sitalic_T, action sequence a→=a0,…,aT−1→subscript0…subscript1 a=a_0,…,a_T-1over→ start_ARG a end_ARG = a0 , … , aitalic_T - 1, and observation sequence o→=o0,…,oT→subscript0…subscript o=o_0,…,o_Tover→ start_ARG o end_ARG = o0 , … , oitalic_T. This is given by a Bayesian network with the following components: • a policy prior B⁢(π′)superscript′B(π )B ( π′ ); • the probability of the initial state B⁢(s0)≔P0⁢(s0)≔subscript0subscript0subscript0B(s_0) P_0(s_0)B ( s0 ) ≔ P0 ( s0 ); • action probabilities B⁢(a∣s,π)≔π⁢(a∣s)≔conditionalconditionalB(a s,π) π(a s)B ( a ∣ s , π ) ≔ π ( a ∣ s ); • transition probabilities B⁢(st+1∣st,at)≔⁢(st+1∣st,at)≔conditionalsubscript1subscriptsubscriptconditionalsubscript1subscriptsubscriptB(s_t+1 s_t,a_t) (s_t+1 s_t,a_t)B ( sitalic_t + 1 ∣ sitalic_t , aitalic_t ) ≔ T ( sitalic_t + 1 ∣ sitalic_t , aitalic_t ); • and observation probabilities B⁢(ot∣st)≔PO⁢(ot∣st)≔conditionalsubscriptsubscriptsubscriptconditionalsubscriptsubscriptB(o_t s_t) P_O(o_t s_t)B ( oitalic_t ∣ sitalic_t ) ≔ Pitalic_O ( oitalic_t ∣ sitalic_t ). Together, this defines the joint distribution B⁢(π,s→,a→,o→)→B(π, s, a, o)B ( π , over→ start_ARG s end_ARG , over→ start_ARG a end_ARG , over→ start_ARG o end_ARG ) over the policy, states, actions, and observations that factorizes according to the following directed acyclic graph: π′π π′s0subscript0s_0s0a0subscript0a_0a0s1subscript1s_1s1a1subscript1a_1a1s2subscript2s_2s2a2subscript2a_2a2s3subscript3s_3s3…o0subscript0o_0o0o1subscript1o_1o1o2subscript2o_2o2o3subscript3o_3o3 (6) The following proposition clarifies the iterative Bayesian update of the human’s posterior over state sequences, given observation sequences: Proposition C.1. Let t≤T−11t≤ T-1t ≤ T - 1 and denote by s^=s0,…,st^subscript0…subscript s=s_0,…,s_tover start_ARG s end_ARG = s0 , … , sitalic_t a state sequence segment of length t≥00t≥ 0t ≥ 0. Similarly, o^=o0,…,ot^subscript0…subscript o=o_0,…,o_tover start_ARG o end_ARG = o0 , … , oitalic_t denotes an observation sequence segment. We have B⁢(s^,st+1,π∣o^,ot+1)∝PO⁢(ot+1∣st+1)⋅[∑at∈⁢(st+1∣s^t,at)⋅π⁢(at∣st)]⋅B⁢(s^,π∣o^).proportional-to^subscript1conditional^subscript1⋅subscriptconditionalsubscript1subscript1delimited-[]subscriptsubscript⋅conditionalsubscript1subscript^subscriptconditionalsubscriptsubscript^conditional^B( s,s_t+1,π o,o_t+1) P_O(o_t+1 s_t+1)% · [ _a_t T(s_t+1 s_t,a_t% )·π(a_t s_t) ]· B( s,π o).B ( over start_ARG s end_ARG , sitalic_t + 1 , π ∣ over start_ARG o end_ARG , oitalic_t + 1 ) ∝ Pitalic_O ( oitalic_t + 1 ∣ sitalic_t + 1 ) ⋅ [ ∑a start_POSTSUBSCRIPT t ∈ A end_POSTSUBSCRIPT T ( sitalic_t + 1 ∣ over start_ARG s end_ARGt , aitalic_t ) ⋅ π ( aitalic_t ∣ sitalic_t ) ] ⋅ B ( over start_ARG s end_ARG , π ∣ over start_ARG o end_ARG ) . Thus, the human can iteratively compute B⁢(s^,π∣o^)^conditional^B( s,π o)B ( over start_ARG s end_ARG , π ∣ over start_ARG o end_ARG ) from the prior B⁢(s0,π)=P0⁢(s0)⋅B⁢(π′)subscript0⋅subscript0subscript0superscript′B(s_0,π)=P_0(s_0)· B(π )B ( s0 , π ) = P0 ( s0 ) ⋅ B ( π′ ) using the above Bayesian update. The posterior over the state sequence can subsequently be computed by B⁢(s^∣o^)=∫πB⁢(s^,π∣o^).conditional^^subscript^conditional^B( s o)= _πB( s,π o).B ( over start_ARG s end_ARG ∣ over start_ARG o end_ARG ) = ∫π B ( over start_ARG s end_ARG , π ∣ over start_ARG o end_ARG ) . Proof. The proof is essentially just Bayes rule applied to the Bayesian network in Equation (6). We repeatedly make use of conditional independences that follow from d-separations in the graph [Geiger et al., 1990]. More concretely, we have B⁢(s^,st+1,π∣o^,ot+1)^subscript1conditional^subscript1 B ( s,s_t+1,π o,o_t+1 )B ( over start_ARG s end_ARG , sitalic_t + 1 , π ∣ over start_ARG o end_ARG , oitalic_t + 1 ) ∝B⁢(ot+1∣s^,st+1,π,o^)⋅B⁢(s^,st+1,π∣o^)proportional-toabsent⋅conditionalsubscript1^subscript1^^subscript1conditional B (o_t+1 s,s_t+1,π, o )% · B ( s,s_t+1,π o )∝ B ( oitalic_t + 1 ∣ over start_ARG s end_ARG , sitalic_t + 1 , π , over start_ARG o end_ARG ) ⋅ B ( over start_ARG s end_ARG , sitalic_t + 1 , π ∣ over start_ARG o end_ARG ) =PO⁢(ot+1∣st+1)⋅B⁢(st+1∣s^,π,o^)⋅B⁢(s^,π∣o^)absent⋅subscriptconditionalsubscript1subscript1conditionalsubscript1^^^conditional =P_O (o_t+1 s_t+1 )· B (s_t+1 % s,π, o)· B( s,π o )= Pitalic_O ( oitalic_t + 1 ∣ sitalic_t + 1 ) ⋅ B ( sitalic_t + 1 ∣ over start_ARG s end_ARG , π , over start_ARG o end_ARG ) ⋅ B ( over start_ARG s end_ARG , π ∣ over start_ARG o end_ARG ) =PO⁢(ot+1∣st+1)⋅[∑at∈B⁢(st+1∣at,s^,π,o^)⋅B⁢(at∣s^,π,o^)]⋅B⁢(s^,π∣o^)absent⋅subscriptconditionalsubscript1subscript1delimited-[]subscriptsubscript⋅conditionalsubscript1subscript^^conditionalsubscript^^^conditional =P_O (o_t+1 s_t+1 )· [ _a_t% B (s_t+1 a_t, s,π, o )· B % (a_t s,π, o ) ]· B ( s,π % o )= Pitalic_O ( oitalic_t + 1 ∣ sitalic_t + 1 ) ⋅ [ ∑a start_POSTSUBSCRIPT t ∈ A end_POSTSUBSCRIPT B ( sitalic_t + 1 ∣ aitalic_t , over start_ARG s end_ARG , π , over start_ARG o end_ARG ) ⋅ B ( aitalic_t ∣ over start_ARG s end_ARG , π , over start_ARG o end_ARG ) ] ⋅ B ( over start_ARG s end_ARG , π ∣ over start_ARG o end_ARG ) =PO⁢(ot+1∣st+1)⋅[∑at∈⁢(st+1∣st,at)⋅π⁢(at∣st)]⋅B⁢(s^,π∣o^).absent⋅subscriptconditionalsubscript1subscript1delimited-[]subscriptsubscript⋅conditionalsubscript1subscriptsubscriptconditionalsubscriptsubscript^conditional =P_O (o_t+1 s_t+1 )· [ _a_t% T (s_t+1 s_t,a_t )·π (% a_t s_t ) ]· B ( s,π o ).= Pitalic_O ( oitalic_t + 1 ∣ sitalic_t + 1 ) ⋅ [ ∑a start_POSTSUBSCRIPT t ∈ A end_POSTSUBSCRIPT T ( sitalic_t + 1 ∣ sitalic_t , aitalic_t ) ⋅ π ( aitalic_t ∣ sitalic_t ) ] ⋅ B ( over start_ARG s end_ARG , π ∣ over start_ARG o end_ARG ) . In step 1, we used Bayes rule. In step 2, we made use of the independence ot+1⟂(s^,π,o^)|st+1o_t+1 \!\!\! ( s,π, o)\ |\ s_t+1oitalic_t + 1 ⟂ ⟂ ( over start_ARG s end_ARG , π , over start_ARG o end_ARG ) | sitalic_t + 1, plugged in the observation kernel, and used the chain rule of probability to compose the second term into a product. In step 3, we marginalized and used, once again, the chain rule of probability. In step 4444, we used the independences st+1⟂(s0,…,st−1,π,o^)|(st,a)s_t+1 \!\!\! (s_0,…,s_t-1,π, o)\ |\ (s_t,a)sitalic_t + 1 ⟂ ⟂ ( s0 , … , sitalic_t - 1 , π , over start_ARG o end_ARG ) | ( sitalic_t , a ) and at⟂(s0,…,st−1,o^)∣(π,st)a_t \!\!\! (s_0,…,s_t-1, o) (π,s_t)aitalic_t ⟂ ⟂ ( s0 , … , sitalic_t - 1 , over start_ARG o end_ARG ) ∣ ( π , sitalic_t ) and plugged in the transition kernel and the policy. The last formula is just a marginalization over the policy. ∎ C.2 Ambiguity and Identifiability of Reward and Return Functions under Observation Sequence Comparisons Figure 9: The linear geometry of ambiguity for a hypothetical example with three state sequences and two observation sequences. G∗superscriptG^*G∗ is the true return function, and “G” is used in labeling the axes to refer to some arbitrary return function. This is a more accurate geometric depiction of the middle and right spaces in Figure 6. The subspace im⁡∩ker⁡imkernelim ∩ Bim Γ ∩ ker B (purple) is the ambiguity in return functions, meaning that adding an element would not change the human’s expected return function on observations. Thus the set of return functions that the reward learning system can infer is the affine set G+(im⁡∩ker⁡)imkernelG+(im ∩ % B)G + ( im Γ ∩ ker B ) (yellow). Note that the planes on the left are drawn to be axis-aligned for ease of visualization; this will not be the case for real MDPs. In this section, we prove the main theorem of this paper: a characterization of the ambiguity that is left in the reward and return function once the human’s Boltzmann-rational choice probabilities are known. We change the formulation slightly by formulating the linear operators “intrinsically” in the spaces they are defined in, instead of using matrix versions. This does not change the general picture, but is a more natural setting when thinking, e.g., about generalizing the results to infinite state sequences. Thus, we define :ℝ→ℝΩ→:→superscriptℝ→superscriptℝ→Ω B: R S→ R % B : blackboard_Rover→ start_ARG S end_ARG → blackboard_Rover→ start_ARG Ω end_ARG as the linear operator given by [⁡(G)]⁢(o→)≔s→∼B⁢(s→∣o→)[G⁢(s→)].≔delimited-[]→subscriptsimilar-to→conditional→ [ B(G) ]( o) *% E_ s B( s o) [G( s) ].[ B ( G ) ] ( over→ start_ARG o end_ARG ) ≔ Eover→ start_ARG s end_ARG ∼ B ( over→ start_ARG s end_ARG ∣ over→ start_ARG o end_ARG ) [ G ( over→ start_ARG s end_ARG ) ] . Here, BB is the human’s belief, which can either be computed as in the previous subsection or simply be any conditional probability distribution. Similarly, we define :ℝ→ℝ→:→superscriptℝsuperscriptℝ→ : R^S→ R % SΓ : blackboard_RS → blackboard_Rover→ start_ARG S end_ARG as the linear operator given by [⁡(R)]⁢(s→)≔∑t=0Tγt⁢R⁢(st).≔delimited-[]→superscriptsubscript0superscriptsubscript [ (R) ]( s) _t=0^T% γ^tR(s_t).[ Γ ( R ) ] ( over→ start_ARG s end_ARG ) ≔ ∑t = 0T γitalic_t R ( sitalic_t ) . The matrix product ⋅ B· B ⋅ Γ then becomes the composition ∘:ℝ→ℝΩ→:→superscriptℝsuperscriptℝ→Ω B : R^% S→ R B ∘ Γ : blackboard_RS → blackboard_Rover→ start_ARG Ω end_ARG. Finally, recall that the kernel ker⁡kernel Aker A of a linear operator AA is defined as its nullspace, and the image im⁡imim Aim A as the set of elements hit by AA. We obtain the following theorem: Theorem C.2. Let R be the true reward function and R~~ Rover~ start_ARG R end_ARG another reward function. Let G~=⁡(R~)~~ G= ( R)over~ start_ARG G end_ARG = Γ ( over~ start_ARG R end_ARG ) and G=⁡(R)G= (R)G = Γ ( R ) be the corresponding return functions. The following three statements are equivalent: (i) The reward function R~~ Rover~ start_ARG R end_ARG gives rise to the same vector of choice probabilities as R, i.e (PR~⁢(o→≻o→′))o→,o→′∈Ω→=(PR⁢(o→≻o→′))o→,o→′∈Ω→.subscriptsuperscript~succeeds→superscript→′→superscript→′→Ωsubscriptsuperscriptsucceeds→superscript→′→superscript→′→Ω (P R ( o o 1.0pt ) % )_ o, o 1.0pt ∈ = (P^R (% o o 1.0pt ) )_ o, o 1% .0pt ∈ .( Pover~ start_ARG R end_ARG ( over→ start_ARG o end_ARG ≻ over→ start_ARG o end_ARG′ ) )over→ start_ARG o end_ARG , over→ start_ARG o end_ARG′ ∈ over→ start_ARG Ω end_ARG = ( Pitalic_R ( over→ start_ARG o end_ARG ≻ over→ start_ARG o end_ARG′ ) )over→ start_ARG o end_ARG , over→ start_ARG o end_ARG′ ∈ over→ start_ARG Ω end_ARG . (i) There is a reward function R′∈ker⁡(∘)superscript′kernelR ∈ ( B )R′ ∈ ker ( B ∘ Γ ) and a constant c∈ℝc∈ Rc ∈ blackboard_R such that R~=R+R′+c.~superscript′ R=R+R +c.over~ start_ARG R end_ARG = R + R′ + c . (i) There is a return function G′∈ker⁡∩im⁡superscript′kernelimG ∈ B % G′ ∈ ker B ∩ im Γ and a constant c′∈ℝsuperscript′ℝc ∈ Rc′ ∈ blackboard_R such that G~=G+G′+c′.~superscript′ G=G+G +c .over~ start_ARG G end_ARG = G + G′ + c′ . In other words, the ambiguity that is left in the reward function when its observation-based choice probabilities are known is, up to an additive constant, given by ker⁡(∘)kernel ( B )ker ( B ∘ Γ ); the ambiguity left in the return function is given by ker⁡∩im⁡kernelim B ker B ∩ im Γ. Proof. Assume (i). To prove (i), let σ by the sigmoid function given by σ⁢(x)=11+exp⁡(−x)11σ(x)= 11+ (-x)σ ( x ) = divide start_ARG 1 end_ARG start_ARG 1 + exp ( - x ) end_ARG. Then by Equation (2), the equality of choice probabilities means the following for all o→,o→′∈Ω→superscript→′→Ω o, o 1.0pt ∈ over→ start_ARG o end_ARG , over→ start_ARG o end_ARG′ ∈ over→ start_ARG Ω end_ARG: σ⁢(β⋅([⁡(G~)]⁢(o→)−[⁡(G~)]⁢(o→′)))=σ⁢(β⋅([⁡(G)]⁢(o→)−[⁡(G)]⁢(o→′))).⋅delimited-[]~→delimited-[]~superscript→′⋅delimited-[]→delimited-[]superscript→′σ (β· ( [ B( G) % ]( o)- [ B( G) ]( o 1% .0pt ) ) )=σ (β· ( [% B(G) ]( o)- [ B(% G) ]( o 1.0pt ) ) ).σ ( β ⋅ ( [ B ( over~ start_ARG G end_ARG ) ] ( over→ start_ARG o end_ARG ) - [ B ( over~ start_ARG G end_ARG ) ] ( over→ start_ARG o end_ARG′ ) ) ) = σ ( β ⋅ ( [ B ( G ) ] ( over→ start_ARG o end_ARG ) - [ B ( G ) ] ( over→ start_ARG o end_ARG′ ) ) ) . Since the sigmoid function is injective, this implies [⁡(G~)]⁢(o→)−[⁡(G~)]⁢(o→′)=[⁡(G)]⁢(o→)−[⁡(G)]⁢(o→′).delimited-[]~→delimited-[]~superscript→′delimited-[]→delimited-[]superscript→′ [ B( G) ]( o)- [% B( G) ]( o 1.0pt )= % [ B(G) ]( o)- [ B% (G) ]( o 1.0pt ).[ B ( over~ start_ARG G end_ARG ) ] ( over→ start_ARG o end_ARG ) - [ B ( over~ start_ARG G end_ARG ) ] ( over→ start_ARG o end_ARG′ ) = [ B ( G ) ] ( over→ start_ARG o end_ARG ) - [ B ( G ) ] ( over→ start_ARG o end_ARG′ ) . Fixing an arbitrary o→′ o 1.0pt over→ start_ARG o end_ARG′, this implies that there exists a constant c′ such that for all o→∈Ω→Ω o∈ over→ start_ARG o end_ARG ∈ over→ start_ARG Ω end_ARG, the following holds: [⁡(G~)]⁢(o→)−[⁡(G)]⁢(o→′)−c′=0.delimited-[]~→delimited-[]superscript→′0 [ B( G) ]( o)- [% B(G) ]( o 1.0pt )-c =0.[ B ( over~ start_ARG G end_ARG ) ] ( over→ start_ARG o end_ARG ) - [ B ( G ) ] ( over→ start_ARG o end_ARG′ ) - c′ = 0 . Noting that ⁡(c′)=c′superscript′ B(c )=c B ( c′ ) = c′, this implies G~−G−c′∈ker⁡()~superscript′kernel G-G-c ∈ ( B)over~ start_ARG G end_ARG - G - c′ ∈ ker ( B ). Now, define the constant reward function c≔c′⋅1−γ1−γT+1.≔⋅superscript′11superscript1c c · 1-γ1-γ^T+1.c ≔ c′ ⋅ divide start_ARG 1 - γ end_ARG start_ARG 1 - γitalic_T + 1 end_ARG . We obtain [⁡(c)]⁢(s→)delimited-[]→ [ (c) ]( s)[ Γ ( c ) ] ( over→ start_ARG s end_ARG ) =∑t=0Tγt⋅cabsentsuperscriptsubscript0⋅superscript = _t=0^Tγ^t· c= ∑t = 0T γitalic_t ⋅ c =c′⋅1−γ1−γT+1⋅∑t=0Tγtabsent⋅superscript′11superscript1superscriptsubscript0superscript =c · 1-γ1-γ^T+1· _t=0^T% γ^t= c′ ⋅ divide start_ARG 1 - γ end_ARG start_ARG 1 - γitalic_T + 1 end_ARG ⋅ ∑t = 0T γitalic_t =c′.absentsuperscript′ =c .= c′ . Thus, we have ⁡(R~−R−c)=G~−G−c′∈ker⁡(),~~superscript′kernel ( R-R-c)= G-G-c ∈ (% B),Γ ( over~ start_ARG R end_ARG - R - c ) = over~ start_ARG G end_ARG - G - c′ ∈ ker ( B ) , implying R′≔R~−R−c∈ker⁡(∘)≔superscript′~kernelR R-R-c∈ ( B % )R′ ≔ over~ start_ARG R end_ARG - R - c ∈ ker ( B ∘ Γ ). This shows (i). That (i) implies (i) follows by applying Γ to both sides of the equation. Now assume (i), i.e. G~=G+G′+c′~superscript′ G=G+G +c over~ start_ARG G end_ARG = G + G′ + c′ for a constant c′∈ℝsuperscript′ℝc ∈ Rc′ ∈ blackboard_R and a return function G′∈ker⁡()∩im⁡superscript′kernelimG ∈ ( B) % G′ ∈ ker ( B ) ∩ im Γ. This implies ⁡(G~)=⁡(G)+c′~superscript′ B( G)= B(G)+c B ( over~ start_ARG G end_ARG ) = B ( G ) + c′. Thus, for all o→,o→′∈Ω→superscript→′→Ω o, o 1.0pt ∈ over→ start_ARG o end_ARG , over→ start_ARG o end_ARG′ ∈ over→ start_ARG Ω end_ARG, we have [⁡(G~)]⁢(o→)−[⁡(G~)]⁢(o→′)=[⁡(G)]⁢(o→)−[⁡(G)]⁢(o→′),delimited-[]~→delimited-[]~superscript→′delimited-[]→delimited-[]superscript→′ [ B( G) ]( o)- [% B( G) ]( o 1.0pt )= % [ B(G) ]( o)- [ B% (G) ]( o 1.0pt ),[ B ( over~ start_ARG G end_ARG ) ] ( over→ start_ARG o end_ARG ) - [ B ( over~ start_ARG G end_ARG ) ] ( over→ start_ARG o end_ARG′ ) = [ B ( G ) ] ( over→ start_ARG o end_ARG ) - [ B ( G ) ] ( over→ start_ARG o end_ARG′ ) , which implies the equal choice probabilities after multiplying with β and applying the sigmoid function σ on both sides. Thus, (i) implies (i). ∎ Corollary C.3. The following two statements are equivalent: (i) ker⁡(∘)=0kernel0 ( B )=0ker ( B ∘ Γ ) = 0. (i) The data (PR⁢(o→≻o→′))o→,o→′∈Ω→subscriptsuperscriptsucceeds→superscript→′→superscript→′→Ω (P^R ( o o 1.0pt ) )_% o, o 1.0pt ∈ ( Pitalic_R ( over→ start_ARG o end_ARG ≻ over→ start_ARG o end_ARG′ ) )over→ start_ARG o end_ARG , over→ start_ARG o end_ARG′ ∈ over→ start_ARG Ω end_ARG determine the reward function R up to an additive constant. Proof. That (i) implies (i) follows immediately from the implication from (i) to (i) within the preceding theorem. Now assume (i). Let R′∈ker⁡(∘)superscript′kernelR ∈ ( B )R′ ∈ ker ( B ∘ Γ ). Define R~≔R+R′≔~superscript′ R R+R over~ start_ARG R end_ARG ≔ R + R′. Then the implication from (i) to (i) within the preceding theorem implies that R~~ Rover~ start_ARG R end_ARG and R have the same choice probabilities. Thus, the assumption (i) in this corollary implies that R′ is a constant. Since Γ and BB map nonzero constants to nonzero constants, the fact that R′∈ker⁡(∘)superscript′kernelR ∈ ( B )R′ ∈ ker ( B ∘ Γ ) implies that R′=0superscript′0R =0R′ = 0, showing that ker⁡(∘)=0kernel0 ( B )=\0\ker ( B ∘ Γ ) = 0 . ∎ As mentioned in the main paper, the previous result already leads to the non-identifiability of R whenever Γ is not injective, corresponding to the presence of zero-initial potential shaping (Skalse et al. [2023], Lemma B.3). Thus, we now strengthen the previous result so that it deals with the identifiability of the return function, which is sufficient for the purpose of policy optimization: Corollary C.4. Consider the following four statements (which can each be true or false): (i) ker⁡=0kernel0 B=\0\ker B = 0 . (i) ker⁡(∘)=0kernel0 ( B )=\0\ker ( B ∘ Γ ) = 0 . (i) ker⁡∩im⁡=0kernelim0 B % =\0\ker B ∩ im Γ = 0 . (iv) The data (PR⁢(o→≻o→′))o→,o→′∈Ω→subscriptsuperscriptsucceeds→superscript→′→superscript→′→Ω (P^R ( o o 1.0pt ) )_% o, o 1.0pt ∈ ( Pitalic_R ( over→ start_ARG o end_ARG ≻ over→ start_ARG o end_ARG′ ) )over→ start_ARG o end_ARG , over→ start_ARG o end_ARG′ ∈ over→ start_ARG Ω end_ARG determine the return function G=⁡(R)G= (R)G = Γ ( R ) on sequences s→∈→ s∈ Sover→ start_ARG s end_ARG ∈ over→ start_ARG S end_ARG up to a constant independent of s→ sover→ start_ARG s end_ARG. Then the following implications, and no other implications, are true: (i)(i)( i )(i⁢i⁢i)(i)( i i i )(i⁢v)(iv)( i v )(i⁢i)(i)( i i ) In particular, all of (i), (i), and (i) are sufficient conditions for identifying the return function from the choice probabilities. Proof. That (i) implies (i) is trivial. That (i) implies (i) is a simple linear algebra fact: Assume (i) and that G′∈ker⁡∩im⁡superscript′kernelimG ∈ B % G′ ∈ ker B ∩ im Γ. Then G′=⁡(R′)superscript′G = (R )G′ = Γ ( R′ ) for some R′∈ℝsuperscript′ℝR ∈ R^SR′ ∈ blackboard_RS and 0=⁡(G′)=⁡(⁡(R′))=(∘)⁢(R′).0superscript′superscript′0= B(G )= B (% (R ) )=( B% )(R ).0 = B ( G′ ) = B ( Γ ( R′ ) ) = ( B ∘ Γ ) ( R′ ) . By (i), this implies R′=0superscript′0R =0R′ = 0 and therefore G′=⁡(R′)=0superscript′0G = (R )=0G′ = Γ ( R′ ) = 0, showing (i). That (i) implies (iv) immediately follows from the implication from (i) to (i) in Theorem C.2. Now, assume (iv). To prove (i), assume G′∈ker⁡∩im⁡superscript′kernelimG ∈ B % G′ ∈ ker B ∩ im Γ. Then the implication from (i) to (i) in Theorem C.2 implies that G+G′+G G + G′ induces the same observation-based choice probabilities as G. Thus, (iv) implies G+G′=G+c′superscript′G+G =G+c G + G′ = G + c′ for some constant c′, which implies G′=c′superscript′G =c G′ = c′. Since G′∈ker⁡superscript′kernelG ∈ BG′ ∈ ker B, this implies 0=⁡(G′)=⁡(c′)=c′0superscript′superscript′0= B(G )= B(c )=% c 0 = B ( G′ ) = B ( c′ ) = c′ and thus G′=0superscript′0G =0G′ = 0. Thus, we showed ker⁡∩im⁡=0kernelim0 B % =\0\ker B ∩ im Γ = 0 . We now show that no other implication holds in general. Example C.32 will show that (i) does not imply (i). We now show that (i) does also not imply (i), from which it will logically follow that (i) does neither imply (i) nor (i). Namely, consider the following simple MDP with time horizon T=11T=1T = 1: aaabbb (7) In this MDP, every state sequence starts in a, deterministically transitions to b, and then ends. This means that s→=a⁢b→ s=abover→ start_ARG s end_ARG = a b is the only sequence. Now, let R′∈ℝa,bsuperscript′ℝR ∈ R^\a,b\R′ ∈ blackboard_R a , b be the reward function given by R′⁢(a)=1,R′⁢(b)=−1γ.formulae-sequencesuperscript′1superscript′1R (a)=1, R (b)= -1γ.R′ ( a ) = 1 , R′ ( b ) = divide start_ARG - 1 end_ARG start_ARG γ end_ARG . We obtain [⁡(R′)]⁢(s→)=R′⁢(a)+γ⁢R′⁢(b)=1+γ⋅−1γ=0.delimited-[]superscript′→superscript′1⋅10 [ (R ) ]( s)=R (a% )+γ R (b)=1+γ· -1γ=0.[ Γ ( R′ ) ] ( over→ start_ARG s end_ARG ) = R′ ( a ) + γ R′ ( b ) = 1 + γ ⋅ divide start_ARG - 1 end_ARG start_ARG γ end_ARG = 0 . Thus, ⁡(R′)=0superscript′0 (R )=0Γ ( R′ ) = 0, (∘)⁢(R′)=0superscript′0( B )(R )=0( B ∘ Γ ) ( R′ ) = 0, and, therefore, ker⁡(∘)≠0kernel0 ( B )% ≠\0\ker ( B ∘ Γ ) ≠ 0 . Thus, (i) does not hold. However, it is possible to choose B⁢(s→∣o→)conditional→B( s o)B ( over→ start_ARG s end_ARG ∣ over→ start_ARG o end_ARG ) such that (i) holds: e.g., if Ω=Ω =SΩ = S and B⁢(s→∣o→)≔δo→⁢(s→)≔conditional→subscript→B( s o) _ o( s)B ( over→ start_ARG s end_ARG ∣ over→ start_ARG o end_ARG ) ≔ δover→ start_ARG o end_ARG ( over→ start_ARG s end_ARG ), then ker⁡=0kernel0 B=\0\ker B = 0 since this operator is the identity. ∎ C.3 The Ambiguity in Reward Learning in Practice In this section, we point out that Theorem C.2 is not just a theoretical discussion: When BB and the inverse temperature parameter β are known, then it is possible to design a reward learning algorithm that learns the true reward function up to the ambiguity ker⁡(∘)kernel ( B )ker ( B ∘ Γ ) in the infinite data limit. In doing so, we essentially use the loss function proposed in Christiano et al. [2017]. Namely, assume DD is a data distribution of observation sequences o→∈Ω→Ω o∈ over→ start_ARG o end_ARG ∈ over→ start_ARG Ω end_ARG such that all sequences in Ω→Ω over→ start_ARG Ω end_ARG have a strictly positive probability of being sampled; for example, DD could use an exploration policy and the observation sequence kernel PO→subscript→P_ OPover→ start_ARG O end_ARG. For each pair of observation sequences (o→,o→′)→superscript→′( o, o 1.0pt )( over→ start_ARG o end_ARG , over→ start_ARG o end_ARG′ ), we then get a conditional distribution P⁢(μ∣o→,o→′)conditional→superscript→′P(μ o, o 1.0pt )P ( μ ∣ over→ start_ARG o end_ARG , over→ start_ARG o end_ARG′ ) over a one-hot encoded human choice μ∈(1,0),(0,1)1001μ∈\(1,0),(0,1)\μ ∈ ( 1 , 0 ) , ( 0 , 1 ) , with probability P⁢(μ=(1,0)∣o→,o→′)=PR⁢(o→≻o→′).conditional10→superscript→′succeeds→superscript→′P (μ=(1,0) o, o 1.0pt )=P^R (% o o 1.0pt ).P ( μ = ( 1 , 0 ) ∣ over→ start_ARG o end_ARG , over→ start_ARG o end_ARG′ ) = Pitalic_R ( over→ start_ARG o end_ARG ≻ over→ start_ARG o end_ARG′ ) . Together, this gives rise to a dataset (o→1,o→1′,μ1),…,(o→N,o→N′,μN)subscript→1subscriptsuperscript→′1subscript1…subscript→subscriptsuperscript→′subscript( o_1, o 1.0pt _1, _1),…,( o_N, % o 1.0pt _N, _N)( over→ start_ARG o end_ARG1 , over→ start_ARG o end_ARG′1 , μ1 ) , … , ( over→ start_ARG o end_ARGN , over→ start_ARG o end_ARG′N , μitalic_N ) of observation sequences plus a human choice. Now assume we learn a reward function Rθ:→ℝ:subscript→ℝR_θ:S→ RRitalic_θ : S → blackboard_R that is differentiable in the parameter θ and that can represent all possible reward functions R∈ℝsuperscriptℝR∈ R^SR ∈ blackboard_RS. Let Gθ≔⁡(Rθ)≔subscriptsubscriptG_θ (R_θ)Gitalic_θ ≔ Γ ( Ritalic_θ ) be the corresponding return function. Write μk=(μk(1),μk(2))subscriptsuperscriptsubscript1superscriptsubscript2 _k=( _k^(1), _k^(2))μitalic_k = ( μitalic_k( 1 ) , μitalic_k( 2 ) ). As in Christiano et al. [2017], we define its loss over the dataset above by ℒ~⁢(θ)=−1N⁢∑k=1Nμk(1)⋅log⁡PRθ⁢(o→k≻o→k′)+μk(2)⋅log⁡PRθ⁢(o→k′≻o→k).~ℒ1superscriptsubscript1⋅superscriptsubscript1superscriptsubscriptsucceedssubscript→subscriptsuperscript→′⋅superscriptsubscript2superscriptsubscriptsucceedssubscriptsuperscript→′subscript→ L(θ)=- 1N _k=1^N _k^(1)·% P^R_θ ( o_k o 1.0pt _k % )+ _k^(2)· P^R_θ ( o 1.0pt _% k o_k ).over~ start_ARG L end_ARG ( θ ) = - divide start_ARG 1 end_ARG start_ARG N end_ARG ∑k = 1N μitalic_k( 1 ) ⋅ log Pitalic_Ritalic_θ ( over→ start_ARG o end_ARGk ≻ over→ start_ARG o end_ARG′k ) + μitalic_k( 2 ) ⋅ log Pitalic_Ritalic_θ ( over→ start_ARG o end_ARG′k ≻ over→ start_ARG o end_ARGk ) . Note that by Equation (2), this loss function essentially uses BB and also the inverse temperature parameter β in its definition. This means that these need to be explicitly represented to be able to use the loss function in practice. Proposition C.5. The loss function ℒ~~ℒ Lover~ start_ARG L end_ARG is differentiable. Furthermore, in the infinite datalimit its minima are precisely given by parameters θ such that Rθ=R+R′+csubscriptsuperscript′R_θ=R+R +cRitalic_θ = R + R′ + c for R′∈ker⁡(∘)superscript′kernelR ∈ ( B % )R′ ∈ ker ( B ∘ Γ ) and c∈ℝc∈ Rc ∈ blackboard_R, or equivalently Gθ=G+G′+c′subscriptsuperscript′G_θ=G+G +c Gitalic_θ = G + G′ + c′ for G′∈ker⁡∩im⁡superscript′kernelimG ∈ B % G′ ∈ ker B ∩ im Γ and c′∈ℝsuperscript′ℝc ∈ Rc′ ∈ blackboard_R. Proof. The differentiability of the loss function follows from the differentiability of multiplication with the matrix BB, see Equation (2), and of the reward function RθsubscriptR_θRitalic_θ in its parameter θ that we assumed. For the second statement, let N⁢(o→,o→′)→superscript→′N( o, o 1.0pt )N ( over→ start_ARG o end_ARG , over→ start_ARG o end_ARG′ ) be the number of times that the pair (o→,o→′)→superscript→′( o, o 1.0pt )( over→ start_ARG o end_ARG , over→ start_ARG o end_ARG′ ) appears in the dataset, and let N⁢(o→,o→′,1)→superscript→′1N( o, o 1.0pt ,1)N ( over→ start_ARG o end_ARG , over→ start_ARG o end_ARG′ , 1 ) be the number of times that the human choice is μ=(1,0)10μ=(1,0)μ = ( 1 , 0 ) and the sampled pair is (o→,o→′)→superscript→′( o, o 1.0pt )( over→ start_ARG o end_ARG , over→ start_ARG o end_ARG′ ), and similar for 2222 instead of 1111. We obtain ℒ~⁢(θ)=~ℒabsent L(θ)=over~ start_ARG L end_ARG ( θ ) = −∑o→,o→′∈Ω→N⁢(o→,o→′)N⋅[N⁢(o→,o→′,1)N⁢(o→,o→′)logPRθ(o→≻o→′) - _ o, o 1.0pt ∈ % N( o, o 1.0pt )N· [ N( o, o% 1.0pt ,1)N( o, o 1.0pt ) P^R_% θ ( o o 1.0pt )- ∑over→ start_ARG o end_ARG , over→ start_ARG o end_ARG′ ∈ over→ start_ARG Ω end_ARG divide start_ARG N ( over→ start_ARG o end_ARG , over→ start_ARG o end_ARG′ ) end_ARG start_ARG N end_ARG ⋅ [ divide start_ARG N ( over→ start_ARG o end_ARG , over→ start_ARG o end_ARG′ , 1 ) end_ARG start_ARG N ( over→ start_ARG o end_ARG , over→ start_ARG o end_ARG′ ) end_ARG log Pitalic_Ritalic_θ ( over→ start_ARG o end_ARG ≻ over→ start_ARG o end_ARG′ ) +N⁢(o→,o→′,2)N⁢(o→,o→′)logPRθ(o→′≻o→)] + N( o, o 1.0pt ,2)N( o, % o 1.0pt ) P^R_θ ( o 1.0pt % o ) ]+ divide start_ARG N ( over→ start_ARG o end_ARG , over→ start_ARG o end_ARG′ , 2 ) end_ARG start_ARG N ( over→ start_ARG o end_ARG , over→ start_ARG o end_ARG′ ) end_ARG log Pitalic_Ritalic_θ ( over→ start_ARG o end_ARG′ ≻ over→ start_ARG o end_ARG ) ] ≈ ≈ o→,o→′∼[CE⁡(PR⁢(o→≻≺o→′)∥PRθ⁢(o→≻≺o→′))]subscriptsimilar-to→superscript→′CEconditionalsuperscriptsucceedsprecedes→superscript→′subscriptsucceedsprecedes→superscript→′ *E_ o, o 1.0pt % [CE (P^R ( o % -2.15277pt$ $ 2.15277pt$ $% o 1.0pt )\ \|\ P^R_θ ( o% -2.15277pt$ $ 2.15277% pt$ $ o 1.0pt ) ) ]Eover→ start_ARG o end_ARG , over→ start_ARG o end_ARG′ ∼ D [ CE ( Pitalic_R ( over→ start_ARG o end_ARG start_RELOP start_ROW start_CELL ≻ end_CELL end_ROW start_ROW start_CELL ≺ end_CELL end_ROW end_RELOP over→ start_ARG o end_ARG′ ) ∥ Pitalic_Ritalic_θ ( over→ start_ARG o end_ARG start_RELOP start_ROW start_CELL ≻ end_CELL end_ROW start_ROW start_CELL ≺ end_CELL end_ROW end_RELOP over→ start_ARG o end_ARG′ ) ) ] ≕ ≕ ℒ⁢(θ).ℒ (θ).L ( θ ) . Here, CECECECE is the crossentropy between the two binary distributions. Since we assumed that DD gives a positive probability to all observation sequences in Ω→Ω over→ start_ARG Ω end_ARG, and since the cross entropy is generally minimized exactly when the second distribution equals the first, the loss function ℒ⁢(θ)ℒL(θ)L ( θ ) is minimized if and only if RθsubscriptR_θRitalic_θ gives rise to the same choice probabilities as R for all pairs of observation sequences. Theorem C.2 then gives the result. ∎ C.4 Identifiability of Return Functions When Human Observations Are Not Known Corollary C.4 assumes that the choice probabilities of each observation sequence pair are known to the reward learning algorithm. However, this requires the algorithm to know what the human observed. In some applications, this is a reasonable assumption, e.g. if the human’s observations are themselves produced by an algorithm that can feed the observations also back to the learning algorithm. In general, however, the observations happen in the physical world, and are only known probabilistically via the observation kernel POsubscriptP_OPitalic_O. The learning system does however have access to the full state sequences that generate the observation sequences. This leads to knowledge of the following choice probabilities for s→,s→′∈→superscript→′→ s, s 1.0pt ∈ Sover→ start_ARG s end_ARG , over→ start_ARG s end_ARG′ ∈ over→ start_ARG S end_ARG: PR⁢(s→≻s→′)≔o→,o→′∼PO→(⋅∣s→,s→′)[PR⁢(o→≻o→′)],P^R ( s s 1.0pt ) % *E_ o, o P_ O(· % s, s 1.0pt ) [P^R ( o o% 1.0pt ) ],Pitalic_R ( over→ start_ARG s end_ARG ≻ over→ start_ARG s end_ARG′ ) ≔ Eover→ start_ARG o end_ARG , over→ start_ARG o end_ARG′ ∼ P start_POSTSUBSCRIPT over→ start_ARG O end_ARG ( ⋅ ∣ over→ start_ARG s end_ARG , over→ start_ARG s end_ARG′ ) end_POSTSUBSCRIPT [ Pitalic_R ( over→ start_ARG o end_ARG ≻ over→ start_ARG o end_ARG′ ) ] , (8) where the observation-based choice probabilities are given as in Equation (2). In other words, the learning algorithm can only infer an aggregate of the observation-based choice probabilities. Again, we can ask a question similar to the ones before, extending the investigations in the previous section: Question C.6. Assume the vector of choice probabilities (PR⁢(s→≻s→′))s→,s→′∈→subscriptsuperscriptsucceeds→superscript→′→superscript→′→ (P^R( s s 1.0pt ) )_ s, s% 1.0pt ∈ S( Pitalic_R ( over→ start_ARG s end_ARG ≻ over→ start_ARG s end_ARG′ ) )over→ start_ARG s end_ARG , over→ start_ARG s end_ARG′ ∈ over→ start_ARG S end_ARG is known. Additionally, assume that it is known that the human’s observations are governed by POsubscriptP_OPitalic_O, and that the human is Boltzmann rational with inverse temperature parameter β and beliefs B⁢(s→∣o→)conditional→B( s o)B ( over→ start_ARG s end_ARG ∣ over→ start_ARG o end_ARG ), see Equation (8). Does this data identify the return function G:→ℝ:→ℝG: S→ RG : over→ start_ARG S end_ARG → blackboard_R? If the observation-based choice probabilities from Equation (2) would be known, then Corollary C.4 would provide the answer to this question. Thus, similar to how we previously inverted the belief operator BB, we are now simply tasked with inverting the expectation over observation sequences. This leads us to the following definition: Definition C.7 (Ungrounding Operator). The ungrounding operators :ℝΩ→ℝ→:→superscriptℝ→Ωsuperscriptℝ→ O: R → R % SO : blackboard_Rover→ start_ARG Ω end_ARG → blackboard_Rover→ start_ARG S end_ARG and ⊗:ℝΩ→×Ω→ℝ→×→:tensor-product→superscriptℝ→Ω→Ωsuperscriptℝ→ O O: R % × → R S× % SO ⊗ O : blackboard_Rover→ start_ARG Ω end_ARG × over→ start_ARG Ω end_ARG → blackboard_Rover→ start_ARG S end_ARG × over→ start_ARG S end_ARG are defined by [⁡(v)]⁢(s→)≔o→∼PO→⁢(o→∣s→)[v⁢(o→)],[(⊗)⁢(C)]⁢(s→,s→′)≔o→,o→′∼PO→(⋅∣s→,s→′)[C⁢(o→,o→′)]. [ O(v) ]( s) *% E_ o P_ O( o s) [v( o) % ], [( O O)(C)% ]( s, s 1.0pt ) *E% _ o, o 1.0pt P_ O(· s, % s 1.0pt ) [C( o, o 1.0pt ) ].[ O ( v ) ] ( over→ start_ARG s end_ARG ) ≔ Eover→ start_ARG o end_ARG ∼ P start_POSTSUBSCRIPT over→ start_ARG O end_ARG ( over→ start_ARG o end_ARG ∣ over→ start_ARG s end_ARG ) end_POSTSUBSCRIPT [ v ( over→ start_ARG o end_ARG ) ] , [ ( O ⊗ O ) ( C ) ] ( over→ start_ARG s end_ARG , over→ start_ARG s end_ARG′ ) ≔ Eover→ start_ARG o end_ARG , over→ start_ARG o end_ARG′ ∼ P start_POSTSUBSCRIPT over→ start_ARG O end_ARG ( ⋅ ∣ over→ start_ARG s end_ARG , over→ start_ARG s end_ARG′ ) end_POSTSUBSCRIPT [ C ( over→ start_ARG o end_ARG , over→ start_ARG o end_ARG′ ) ] . Here, v∈ℝΩ→superscriptℝ→Ωv∈ R v ∈ blackboard_Rover→ start_ARG Ω end_ARG is an arbitrary vector, and C∈ℝΩ→×Ω→superscriptℝ→Ω→ΩC∈ R × C ∈ blackboard_Rover→ start_ARG Ω end_ARG × over→ start_ARG Ω end_ARG is also an arbitrary vector, where the notation can remind of “Choice” since the inputs to ⊗tensor-product O OO ⊗ O are, in practice, vectors of observation-based Boltzmann-rational choice probabilities. Formally, ⊗tensor-product O OO ⊗ O is the Kronecker product of OO with itself, but it is not necessary to understand this fact to follow the discussion. Ultimately, to be able to recover the observation-based choice probabilities, what matters is that ⊗tensor-product O OO ⊗ O is injective on whole vectors of these choice probabilities. The injectivity of OO is a sufficient condition for this, which explains its usefulness. We show this in the following lemma: Lemma C.8. :ℝΩ→ℝ→:→superscriptℝ→Ωsuperscriptℝ→ O: R → R % SO : blackboard_Rover→ start_ARG Ω end_ARG → blackboard_Rover→ start_ARG S end_ARG is injective if and only if ⊗:ℝΩ→×Ω→ℝ→×→:tensor-product→superscriptℝ→Ω→Ωsuperscriptℝ→ O O: R % × → R S× % SO ⊗ O : blackboard_Rover→ start_ARG Ω end_ARG × over→ start_ARG Ω end_ARG → blackboard_Rover→ start_ARG S end_ARG × over→ start_ARG S end_ARG is injective. Proof. This is a general property of the Kronecker product of a linear operator with itself. For completeness, we demonstrate the calculation in our special case. First, assume that OO is injective. Assume that (⊗)⁢(C)=0tensor-product0( O O)(C)=0( O ⊗ O ) ( C ) = 0 for some C∈ℝΩ→×Ω→superscriptℝ→Ω→ΩC∈ R × C ∈ blackboard_Rover→ start_ARG Ω end_ARG × over→ start_ARG Ω end_ARG. We need to show C=00C=0C = 0. For all pairs of state sequences (s→,s→′)→superscript→′( s, s 1.0pt )( over→ start_ARG s end_ARG , over→ start_ARG s end_ARG′ ), we have 0=[(⊗)⁢(C)]⁢(s→,s→′)0delimited-[]tensor-product→superscript→′ 0= [( O % O)(C) ]( s, s 1.0pt )0 = [ ( O ⊗ O ) ( C ) ] ( over→ start_ARG s end_ARG , over→ start_ARG s end_ARG′ ) =o→,o→′∼PO→(⋅∣s→,s→′)[C⁢(o→,o→′)] = *E_ o, o 1.0pt % P_ O(· s, s 1.0pt ) [C( % o, o 1.0pt ) ]= Eover→ start_ARG o end_ARG , over→ start_ARG o end_ARG′ ∼ P start_POSTSUBSCRIPT over→ start_ARG O end_ARG ( ⋅ ∣ over→ start_ARG s end_ARG , over→ start_ARG s end_ARG′ ) end_POSTSUBSCRIPT [ C ( over→ start_ARG o end_ARG , over→ start_ARG o end_ARG′ ) ] =o→∼PO→⁢(o→∣s→)[o→′∼PO→⁢(o→′∣s→′)[C⁢(o→,o→′)]]absentsubscriptsimilar-to→subscript→conditional→subscriptsimilar-tosuperscript→′subscript→conditionalsuperscript→′→superscript→′ = *E_ o P_ O( o % s) [ *E_ o 1.0pt P% _ O( o 1.0pt s 1.0pt ) % [C( o, o 1.0pt ) ] ]= Eover→ start_ARG o end_ARG ∼ P start_POSTSUBSCRIPT over→ start_ARG O end_ARG ( over→ start_ARG o end_ARG ∣ over→ start_ARG s end_ARG ) end_POSTSUBSCRIPT [ Eover→ start_ARG o end_ARG′ ∼ P start_POSTSUBSCRIPT over→ start_ARG O end_ARG ( over→ start_ARG o end_ARG′ ∣ over→ start_ARG s end_ARG′ ) end_POSTSUBSCRIPT [ C ( over→ start_ARG o end_ARG , over→ start_ARG o end_ARG′ ) ] ] =o→∼PO→⁢(o→∣s→)[Cs→′⁢(o→)]absentsubscriptsimilar-to→subscript→conditional→subscriptsuperscript′→′→ = *E_ o P_ O( o % s) [C _ s 1.0pt ( o) ]= Eover→ start_ARG o end_ARG ∼ P start_POSTSUBSCRIPT over→ start_ARG O end_ARG ( over→ start_ARG o end_ARG ∣ over→ start_ARG s end_ARG ) end_POSTSUBSCRIPT [ C′over→ start_ARG s end_ARG′ ( over→ start_ARG o end_ARG ) ] =[⁡(Cs→′)]⁢(s→),absentdelimited-[]subscriptsuperscript′→′→ = [ O (C _ s 1% .0pt ) ]( s),= [ O ( C′over→ start_ARG s end_ARG′ ) ] ( over→ start_ARG s end_ARG ) , where Cs→′⁢(o→)≔o→′∼PO→⁢(o→′∣s→′)[C⁢(o→,o→′)]≔subscriptsuperscript′→′→subscriptsimilar-tosuperscript→′subscript→conditionalsuperscript→′→superscript→′C _ s 1.0pt ( o) *% E_ o 1.0pt P_ O( o 1.0pt^% s 1.0pt ) [C( o, o 1.0pt% ) ]C′over→ start_ARG s end_ARG′ ( over→ start_ARG o end_ARG ) ≔ Eover→ start_ARG o end_ARG′ ∼ P start_POSTSUBSCRIPT over→ start_ARG O end_ARG ( over→ start_ARG o end_ARG′ ∣ over→ start_ARG s end_ARG′ ) end_POSTSUBSCRIPT [ C ( over→ start_ARG o end_ARG , over→ start_ARG o end_ARG′ ) ]. By the injectivity of OO, we obtain Cs→′=0subscriptsuperscript′→′0C _ s 1.0pt =0C′over→ start_ARG s end_ARG′ = 0 for all s→′ s 1.0pt over→ start_ARG s end_ARG′. This means that for all s→′ s 1.0pt over→ start_ARG s end_ARG′ and o→ oover→ start_ARG o end_ARG, we have 0=Cs→′⁢(o→)=o→′∼PO→⁢(o→′∣s→′)[C⁢(o→,o→′)]=[⁡(Co→′)]⁢(s→′),0subscriptsuperscript′→′→subscriptsimilar-tosuperscript→′subscript→conditionalsuperscript→′→superscript→′delimited-[]subscriptsuperscript′→superscript→′ 0=C _ s 1.0pt ( o)=% *E_ o 1.0pt P_ O( % o 1.0pt s 1.0pt ) [C( o, % o 1.0pt ) ]= [ O (C^% _ o ) ]( s 1.0pt ),0 = C′over→ start_ARG s end_ARG′ ( over→ start_ARG o end_ARG ) = Eover→ start_ARG o end_ARG′ ∼ P start_POSTSUBSCRIPT over→ start_ARG O end_ARG ( over→ start_ARG o end_ARG′ ∣ over→ start_ARG s end_ARG′ ) end_POSTSUBSCRIPT [ C ( over→ start_ARG o end_ARG , over→ start_ARG o end_ARG′ ) ] = [ O ( C′ ′over→ start_ARG o end_ARG ) ] ( over→ start_ARG s end_ARG′ ) , where Co→′⁢(o→′)≔C⁢(o→,o→′)≔subscriptsuperscript′→superscript→′→superscript→′C _ o( o 1.0pt ) C( o,% o 1.0pt )C′ ′over→ start_ARG o end_ARG ( over→ start_ARG o end_ARG′ ) ≔ C ( over→ start_ARG o end_ARG , over→ start_ARG o end_ARG′ ). Again, by the injectivity of OO, we obtain Co→′=0subscriptsuperscript′→0C _ o=0C′ ′over→ start_ARG o end_ARG = 0 for all o→ oover→ start_ARG o end_ARG, leading to C=00C=0C = 0. That proves the direction from left to right. To prove the other direction, assume that OO is not injective. This means there exists 0≠C∈ℝΩ→0superscriptℝ→Ω0≠ C∈ R 0 ≠ C ∈ blackboard_Rover→ start_ARG Ω end_ARG such that ⁡(C)=00 O(C)=0O ( C ) = 0. Define C⊗C∈ℝΩ→×Ω→tensor-productsuperscriptℝ→Ω→ΩC C∈ R × C ⊗ C ∈ blackboard_Rover→ start_ARG Ω end_ARG × over→ start_ARG Ω end_ARG by (C⊗C)⁢(o→,o→′)≔C⁢(o→)⁢C⁢(o→′).≔tensor-product→superscript→′→superscript→′(C C)( o, o 1.0pt ) C( o)C( o% 1.0pt ).( C ⊗ C ) ( over→ start_ARG o end_ARG , over→ start_ARG o end_ARG′ ) ≔ C ( over→ start_ARG o end_ARG ) C ( over→ start_ARG o end_ARG′ ) . Then clearly, C⊗C≠0tensor-product0C C≠ 0C ⊗ C ≠ 0. We are done if we can show that (⊗)⁢(C⊗C)=0tensor-producttensor-product0( O O)(C C)=0( O ⊗ O ) ( C ⊗ C ) = 0 since that establishes that ⊗tensor-product O OO ⊗ O is also not injective. For any s→,s→′∈→superscript→′→ s, s 1.0pt ∈ Sover→ start_ARG s end_ARG , over→ start_ARG s end_ARG′ ∈ over→ start_ARG S end_ARG, we have [(⊗)⁢(C⊗C)]⁢(s→,s→′)delimited-[]tensor-producttensor-product→superscript→′ [( O O% )(C C) ]( s, s 1.0pt )[ ( O ⊗ O ) ( C ⊗ C ) ] ( over→ start_ARG s end_ARG , over→ start_ARG s end_ARG′ ) =o→,o→′∼PO→(⋅∣s→,s→′)[(C⊗C)⁢(o→,o→′)] = *E_ o, o 1.0pt % P_ O(· s, s 1.0pt ) [(C% C)( o, o 1.0pt ) ]= Eover→ start_ARG o end_ARG , over→ start_ARG o end_ARG′ ∼ P start_POSTSUBSCRIPT over→ start_ARG O end_ARG ( ⋅ ∣ over→ start_ARG s end_ARG , over→ start_ARG s end_ARG′ ) end_POSTSUBSCRIPT [ ( C ⊗ C ) ( over→ start_ARG o end_ARG , over→ start_ARG o end_ARG′ ) ] =o→,o→′∼PO→(⋅∣s→,s→′)[C⁢(o→)⋅C⁢(o→′)] = *E_ o, o 1.0pt % P_ O(· s, s 1.0pt ) [C( % o)· C( o 1.0pt ) ]= Eover→ start_ARG o end_ARG , over→ start_ARG o end_ARG′ ∼ P start_POSTSUBSCRIPT over→ start_ARG O end_ARG ( ⋅ ∣ over→ start_ARG s end_ARG , over→ start_ARG s end_ARG′ ) end_POSTSUBSCRIPT [ C ( over→ start_ARG o end_ARG ) ⋅ C ( over→ start_ARG o end_ARG′ ) ] =o→∼PO→⁢(o→∣s→)[C⁢(o→)]⋅o→′∼PO→⁢(o→′∣s→′)[C⁢(o→′)]absent⋅subscriptsimilar-to→subscript→conditional→subscriptsimilar-tosuperscript→′subscript→conditionalsuperscript→′superscript→′ = *E_ o P_ O( o % s) [C( o) ]· *E_ o% 1.0pt P_ O( o 1.0pt s% 1.0pt ) [C( o 1.0pt ) ]= Eover→ start_ARG o end_ARG ∼ P start_POSTSUBSCRIPT over→ start_ARG O end_ARG ( over→ start_ARG o end_ARG ∣ over→ start_ARG s end_ARG ) end_POSTSUBSCRIPT [ C ( over→ start_ARG o end_ARG ) ] ⋅ Eover→ start_ARG o end_ARG′ ∼ P start_POSTSUBSCRIPT over→ start_ARG O end_ARG ( over→ start_ARG o end_ARG′ ∣ over→ start_ARG s end_ARG′ ) end_POSTSUBSCRIPT [ C ( over→ start_ARG o end_ARG′ ) ] =[⁡(C)]⁢(s→)⋅[⁡(C)]⁢(s→′)absent⋅delimited-[]→delimited-[]superscript→′ = [ O(C) ]( s)· [% O(C) ]( s 1.0pt )= [ O ( C ) ] ( over→ start_ARG s end_ARG ) ⋅ [ O ( C ) ] ( over→ start_ARG s end_ARG′ ) =0⋅0absent⋅00 =0· 0= 0 ⋅ 0 =0.absent0 =0.= 0 . This finishes the proof. ∎ We now state and prove the following extension of Corollary C.4: Theorem C.9. Consider the following statements (which can each be true or false): 1. :ℝΩ→ℝ→:→superscriptℝ→Ωsuperscriptℝ→ O: R → R % SO : blackboard_Rover→ start_ARG Ω end_ARG → blackboard_Rover→ start_ARG S end_ARG is an injective linear operator: ker⁡=0kernel0 O=\0\ker O = 0 . 2. ⊗:ℝΩ→×Ω→ℝ→×→:tensor-product→superscriptℝ→Ω→Ωsuperscriptℝ→ O O: R % × → R S× % SO ⊗ O : blackboard_Rover→ start_ARG Ω end_ARG × over→ start_ARG Ω end_ARG → blackboard_Rover→ start_ARG S end_ARG × over→ start_ARG S end_ARG is an injective linear operator: ker⁡⊗=0kerneltensor-product0 O O=\0\ker O ⊗ O = 0 . 3. ⊗tensor-product O OO ⊗ O is injective on vectors of observation-based choice probabilities (PR⁢(o→≻o→′))o→,o→′subscriptsuperscriptsucceeds→superscript→′→superscript→′ (P^R ( o o 1.0pt ) )_% o, o 1.0pt ( Pitalic_R ( over→ start_ARG o end_ARG ≻ over→ start_ARG o end_ARG′ ) )over→ start_ARG o end_ARG , over→ start_ARG o end_ARG′ over the set of return functions G∈ℝ→superscriptℝ→G∈ R SG ∈ blackboard_Rover→ start_ARG S end_ARG. 4. The data of state-based choice probabilities (PR⁢(s→≻s→′))s→,s→′∈→subscriptsuperscriptsucceeds→superscript→′→superscript→′→ (P^R ( s s 1.0pt ) )_% s, s 1.0pt ∈ S( Pitalic_R ( over→ start_ARG s end_ARG ≻ over→ start_ARG s end_ARG′ ) )over→ start_ARG s end_ARG , over→ start_ARG s end_ARG′ ∈ over→ start_ARG S end_ARG from Equation (8) determine the data of observation-based choice probabilities (PR⁢(o→≻o→′))o→,o→′∈Ω→subscriptsuperscriptsucceeds→superscript→′→superscript→′→Ω (P^R ( o o 1.0pt ) )_% o, o 1.0pt ∈ ( Pitalic_R ( over→ start_ARG o end_ARG ≻ over→ start_ARG o end_ARG′ ) )over→ start_ARG o end_ARG , over→ start_ARG o end_ARG′ ∈ over→ start_ARG Ω end_ARG from Equation (2). Then the following implications hold and 3 does not imply 2: 1111222233334.44.4 . Consequently, if any of the conditions 1, 2, or 3 hold, and additionally any of the conditions (i), (i) or (i) from Corollary C.4, then the data (PR⁢(s→≻s→′))s→,s→′∈Ω→subscriptsuperscriptsucceeds→superscript→′→superscript→′→Ω (P^R ( s s 1.0pt ) )_% s, s 1.0pt ∈ ( Pitalic_R ( over→ start_ARG s end_ARG ≻ over→ start_ARG s end_ARG′ ) )over→ start_ARG s end_ARG , over→ start_ARG s end_ARG′ ∈ over→ start_ARG Ω end_ARG determine the return function G on sequences s→∈→ s∈ Sover→ start_ARG s end_ARG ∈ over→ start_ARG S end_ARG up to a constant independent of s→ sover→ start_ARG s end_ARG. Proof. That 1 and 2 are equivalent was shown in Lemma C.8. That 2 implies 3 is clear. To prove that 3 implies 4, simply put both sets of choice probabilities into a vector. Then Equation (8) and Definition C.7 show the following equality of vectors in ℝ→×→superscriptℝ→ R S× Sblackboard_Rover→ start_ARG S end_ARG × over→ start_ARG S end_ARG: (PR⁢(s→≻s→′))s→,s→′=(⊗)⁢((PR⁢(o→≻o→′))o→,o→′).subscriptsuperscriptsucceeds→superscript→′→superscript→′tensor-productsubscriptsuperscriptsucceeds→superscript→′→superscript→′ (P^R ( s s 1.0pt ) )_% s, s 1.0pt = ( O % O ) ( (P^R ( o o% 1.0pt ) )_ o, o 1.0pt % ).( Pitalic_R ( over→ start_ARG s end_ARG ≻ over→ start_ARG s end_ARG′ ) )over→ start_ARG s end_ARG , over→ start_ARG s end_ARG′ = ( O ⊗ O ) ( ( Pitalic_R ( over→ start_ARG o end_ARG ≻ over→ start_ARG o end_ARG′ ) )over→ start_ARG o end_ARG , over→ start_ARG o end_ARG′ ) . The injectivity of ⊗tensor-product O OO ⊗ O on such inputs ensures that the observation-based choice probabilities can be recovered using this equation. We now show that (3) does not imply (2). Again, we use the simple MDP from Equation (7), but this time with a different observation kernel. Namely, we choose PO⁢(o(a)∣a)=PO⁢(o(a)′∣a)=12,PO⁢(o(b)∣b)=1,formulae-sequencesubscriptconditionalsuperscriptsubscriptconditionalsuperscriptsuperscript′12subscriptconditionalsuperscript1P_O(o^(a) a)=P_O(o^(a) a)= 12, P_O(o% ^(b) b)=1,Pitalic_O ( o( a ) ∣ a ) = Pitalic_O ( o( a )′ ∣ a ) = divide start_ARG 1 end_ARG start_ARG 2 end_ARG , Pitalic_O ( o( b ) ∣ b ) = 1 , where o(a)′≠o(a)superscriptsuperscript′o^(a) ≠ o^(a)o( a )′ ≠ o( a ) and o(a)≠o(b)≠o(a)′superscriptsuperscriptsuperscriptsuperscript′o^(a)≠ o^(b)≠o^(a) o( a ) ≠ o( b ) ≠ o( a )′. This results in two possible observation sequences: o(a)⁢o(b)superscriptsuperscripto^(a)o^(b)o( a ) o( b ) and o(a)′⁢o(b)superscriptsuperscript′o^(a) o^(b)o( a )′ o( b ). Thus, ℝΩ→superscriptℝ→Ω R blackboard_Rover→ start_ARG Ω end_ARG is two-dimensional, whereas ℝ→superscriptℝ→ R Sblackboard_Rover→ start_ARG S end_ARG is only one-dimensional. Consequently, :ℝΩ→ℝ→:→superscriptℝ→Ωsuperscriptℝ→ O: R → R % SO : blackboard_Rover→ start_ARG Ω end_ARG → blackboard_Rover→ start_ARG S end_ARG cannot be injective, so ker⁡≠0kernel0 O≠\0\ker O ≠ 0 , so (2) does not hold since (1) and (2) are equivalent. However, (3) still holds: Since there is only one state sequence, Equation (2) shows that the only vector of choice probabilities has 1/2121/21 / 2 in all its entries, irrespective of the return function G. Thus, ⊗tensor-product O OO ⊗ O has only one input of observation-based choice probabilities, and is thus automatically injective on its inputs. The final result of identifiability of the return function G follows using Corollary C.4. ∎ C.5 Simple Special Cases: Full Observability, Deterministic PO→subscript→P_ OPover→ start_ARG O end_ARG, and Noisy PO→subscript→P_ OPover→ start_ARG O end_ARG In this section, we analyze three simple special cases of the general theory. Theorem 3.9 (together with Lemma B.3) from Skalse et al. [2023], reproduced as a corollary below, is a special case of our theorem: Corollary C.10 (Skalse et al. [2023]). Assume the human directly observes the true sequences, and the choice probabilities are given by PR⁢(s→≻s→′)=σ⁢(β⁢(G⁢(s→)−G⁢(s→′))).superscriptsucceeds→superscript→′→superscript→′P^R ( s s 1.0pt )=σ (β% (G( s)-G( s 1.0pt ) ) ).Pitalic_R ( over→ start_ARG s end_ARG ≻ over→ start_ARG s end_ARG′ ) = σ ( β ( G ( over→ start_ARG s end_ARG ) - G ( over→ start_ARG s end_ARG′ ) ) ) . This data determines the return function G=⁡(R)G= (R)G = Γ ( R ) on state sequences s→∈→ s∈ Sover→ start_ARG s end_ARG ∈ over→ start_ARG S end_ARG up to a constant independent on s→ sover→ start_ARG s end_ARG. Proof. We can embed this case into the one of Theorem C.9 by defining the observation kernel as PO→⁢(s→′∣s→)=δs→⁢(s→′)subscript→conditionalsuperscript→′→subscript→superscript→′P_ O( s 1.0pt s)= _ s( s% 1.0pt )Pover→ start_ARG O end_ARG ( over→ start_ARG s end_ARG′ ∣ over→ start_ARG s end_ARG ) = δover→ start_ARG s end_ARG ( over→ start_ARG s end_ARG′ ) (i.e., the correct sequence is deterministically observed) and defining the human’s belief as B⁢(s→′∣s→)=δs→⁢(s→′)conditionalsuperscript→′→subscript→superscript→′B( s 1.0pt s)= _ s( s 1.0% pt )B ( over→ start_ARG s end_ARG′ ∣ over→ start_ARG s end_ARG ) = δover→ start_ARG s end_ARG ( over→ start_ARG s end_ARG′ ) (i.e., the human knows that the observation reflects the true sequence). This shows that P⁢(s→≻s→′)succeeds→superscript→′P( s s 1.0pt )P ( over→ start_ARG s end_ARG ≻ over→ start_ARG s end_ARG′ ) is of the form of Equation (8). The result follows from Theorem C.9: the operators OO and BB are the identity in this case, due to the defining property of the Kronecker delta, and so they are injective. ∎ The following proposition shows that Corollary C.10 is essentially the only example of deterministic observation kernel PO→subscript→P_ OPover→ start_ARG O end_ARG for which BB is injective. Note, however, that in some situations, we can have im⁡∩ker⁡=0imkernel0im ∩ B% =\0\im Γ ∩ ker B = 0 even if BB is not injective, see Example C.32. Proposition C.11. Assume PO→subscript→P_ OPover→ start_ARG O end_ARG, the observation kernel on the level of sequences, is deterministic and not injective. Then OO is automatically injective. However, BB is not injective. Proof. To show that OO is injective, assume v∈ℝΩ→superscriptℝ→Ωv∈ R v ∈ blackboard_Rover→ start_ARG Ω end_ARG is such that ⁡(v)=00 O(v)=0O ( v ) = 0. Then for all s→∈→ s∈ Sover→ start_ARG s end_ARG ∈ over→ start_ARG S end_ARG, we get 0=[⁡(v)]⁢(s→)=o→∼PO→⁢(o→∣s→)[v⁢(o→)]=v⁢(O→⁢(s→)).0delimited-[]→subscriptsimilar-to→subscript→conditional→0= [ O(v) ]( s)= *E% _ o P_ O( o s) [v( o) ]=v % ( O( s) ).0 = [ O ( v ) ] ( over→ start_ARG s end_ARG ) = Eover→ start_ARG o end_ARG ∼ P start_POSTSUBSCRIPT over→ start_ARG O end_ARG ( over→ start_ARG o end_ARG ∣ over→ start_ARG s end_ARG ) end_POSTSUBSCRIPT [ v ( over→ start_ARG o end_ARG ) ] = v ( over→ start_ARG O end_ARG ( over→ start_ARG s end_ARG ) ) . Since O→:→Ω→:→Ω O: S→ over→ start_ARG O end_ARG : over→ start_ARG S end_ARG → over→ start_ARG Ω end_ARG is by definition surjective, we obtain v=00v=0v = 0. O→:→Ω→:→Ω O: S→ over→ start_ARG O end_ARG : over→ start_ARG S end_ARG → over→ start_ARG Ω end_ARG is by definition surjective, and here assumed to be non-injective, which implies that → Sover→ start_ARG S end_ARG has a higher cardinality than Ω→Ω over→ start_ARG Ω end_ARG. Thus, :ℝ→ℝΩ→:→superscriptℝ→superscriptℝ→Ω B: R S→ R % B : blackboard_Rover→ start_ARG S end_ARG → blackboard_Rover→ start_ARG Ω end_ARG cannot be injective. ∎ In the following, we analyze a simple case that guarantees identifiability. It requires that the observation kernel is “well-behaved” of a form where the observations are simply “noisy states”, and that the human is a Bayesian reasoner with any prior B⁢(s→)→B( s)B ( over→ start_ARG s end_ARG ) that supports every state sequence s→∈→ s∈ Sover→ start_ARG s end_ARG ∈ over→ start_ARG S end_ARG. Definition C.12 (Noise in the Observation Kernel). Then we say that there is noise in the observation kernel PO:→Δ⁢(Ω→):subscript→Δ→ΩP_O: S→ ( )Pitalic_O : over→ start_ARG S end_ARG → Δ ( over→ start_ARG Ω end_ARG ) if →=Ω→Ω S= over→ start_ARG S end_ARG = over→ start_ARG Ω end_ARG and if OO is an injective linear operator. Proposition C.13. Assume that →=Ω→Ω S= over→ start_ARG S end_ARG = over→ start_ARG Ω end_ARG. Furthermore, assume that B⁢(s→∣o→)conditional→B( s o)B ( over→ start_ARG s end_ARG ∣ over→ start_ARG o end_ARG ) is given by the posterior with likelihood PO→⁢(o→∣s→)subscript→conditional→P_ O( o s)Pover→ start_ARG O end_ARG ( over→ start_ARG o end_ARG ∣ over→ start_ARG s end_ARG ) and any prior B⁢(s→)→B( s)B ( over→ start_ARG s end_ARG ) with B⁢(s→)>0→0B( s)>0B ( over→ start_ARG s end_ARG ) > 0 for all s→∈→ s∈ Sover→ start_ARG s end_ARG ∈ over→ start_ARG S end_ARG. Then there is noise in the observation kernel if and only if BB is injective. Proof. Assume OO is injective. To show that BB is injective, assume there is G′∈ℝ→superscript′ℝ→G ∈ R SG′ ∈ blackboard_Rover→ start_ARG S end_ARG with ⁡(G′)=0superscript′0 B(G )=0B ( G′ ) = 0. Then for all o→∈Ω→Ω o∈ over→ start_ARG o end_ARG ∈ over→ start_ARG Ω end_ARG, we have 00 0 =[⁡(G′)]⁢(o→)=s→∼B⁢(s→∣o→)[G′⁢(s→)]=∑s→B⁢(s→∣o→)⁢G′⁢(s→)∝∑s→PO→⁢(o→∣s→)⋅(B⁢(s→)⋅G′⁢(s→))absentdelimited-[]superscript′→subscriptsimilar-to→conditional→superscript′→subscript→conditional→superscript′→proportional-tosubscript→⋅subscript→conditional→⋅→superscript′→ = [ B(G ) ]( o)=% *E_ s B( s o) [G % ( s) ]= _ sB( s o)G ( s)% _ sP_ O( o s)· (B( s)% · G ( s) )= [ B ( G′ ) ] ( over→ start_ARG o end_ARG ) = Eover→ start_ARG s end_ARG ∼ B ( over→ start_ARG s end_ARG ∣ over→ start_ARG o end_ARG ) [ G′ ( over→ start_ARG s end_ARG ) ] = ∑over→ start_ARG s end_ARG B ( over→ start_ARG s end_ARG ∣ over→ start_ARG o end_ARG ) G′ ( over→ start_ARG s end_ARG ) ∝ ∑over→ start_ARG s end_ARG Pover→ start_ARG O end_ARG ( over→ start_ARG o end_ARG ∣ over→ start_ARG s end_ARG ) ⋅ ( B ( over→ start_ARG s end_ARG ) ⋅ G′ ( over→ start_ARG s end_ARG ) ) =[T⁡(B⊙G′)]⁢(o→).absentdelimited-[]superscriptdirect-productsuperscript′→ = [ O^T(B G ) ](% o).= [ Oitalic_T ( B ⊙ G′ ) ] ( over→ start_ARG o end_ARG ) . Here, Tsuperscript O^TOitalic_T is the transpose of OO and B⊙G′direct-productsuperscript′B G B ⊙ G′ is the componentwise product of the prior B with the return function G′. Since OO is injective and thus invertible, Tsuperscript O^TOitalic_T is as well. Thus, B⊙G′=0direct-productsuperscript′0B G =0B ⊙ G′ = 0, which implies G′=0superscript′0G =0G′ = 0 since the prior gives positive probability to all state sequences. Thus, BB is injective. For the other direction, assume BB is injective. To show that OO is injective, let v∈ℝΩ→superscriptℝ→Ωv∈ R v ∈ blackboard_Rover→ start_ARG Ω end_ARG be any vector with ⁡(v)=00 O(v)=0O ( v ) = 0. We do a similar computation as above: for all s→∈ℝ→superscriptℝ→ s∈ R Sover→ start_ARG s end_ARG ∈ blackboard_Rover→ start_ARG S end_ARG, we have 00 0 =[⁡(v)]⁢(s→)=o→∼PO→⁢(o→∣s→)[v⁢(o→)]=∑o→PO→⁢(o→∣s→)⁢v⁢(o→)∝∑o→B⁢(s→∣o→)⋅(PO→⁢(o→)⋅v⁢(o→))absentdelimited-[]→subscriptsimilar-to→subscript→conditional→subscript→subscript→conditional→proportional-tosubscript→⋅conditional→⋅subscript→ = [ O(v) ]( s)=% *E_ o P_ O( o s) [% v( o) ]= _ oP_ O( o s)v( o)% _ oB( s o)· (P_ O( o)% · v( o) )= [ O ( v ) ] ( over→ start_ARG s end_ARG ) = Eover→ start_ARG o end_ARG ∼ P start_POSTSUBSCRIPT over→ start_ARG O end_ARG ( over→ start_ARG o end_ARG ∣ over→ start_ARG s end_ARG ) end_POSTSUBSCRIPT [ v ( over→ start_ARG o end_ARG ) ] = ∑over→ start_ARG o end_ARG Pover→ start_ARG O end_ARG ( over→ start_ARG o end_ARG ∣ over→ start_ARG s end_ARG ) v ( over→ start_ARG o end_ARG ) ∝ ∑over→ start_ARG o end_ARG B ( over→ start_ARG s end_ARG ∣ over→ start_ARG o end_ARG ) ⋅ ( Pover→ start_ARG O end_ARG ( over→ start_ARG o end_ARG ) ⋅ v ( over→ start_ARG o end_ARG ) ) =[T⁡(PO→⊙v)]⁢(s→).absentdelimited-[]superscriptdirect-productsubscript→ = [ B^T (P_ O v% ) ]( s).= [ Bitalic_T ( Pover→ start_ARG O end_ARG ⊙ v ) ] ( over→ start_ARG s end_ARG ) . Here, Tsuperscript B^TBitalic_T is the transpose of BB, PO→⁢(o→)subscript→P_ O( o)Pover→ start_ARG O end_ARG ( over→ start_ARG o end_ARG ) is the denominator in Bayes rule, and PO→⊙vdirect-productsubscript→P_ O vPover→ start_ARG O end_ARG ⊙ v is the vector with components PO→⁢(o→)⋅v⁢(o→)⋅subscript→P_ O( o)· v( o)Pover→ start_ARG O end_ARG ( over→ start_ARG o end_ARG ) ⋅ v ( over→ start_ARG o end_ARG ). From the injectivity and thus invertibility of BB, it follows that Tsuperscript B^TBitalic_T is invertible as well, and so PO→⊙v=0direct-productsubscript→0P_ O v=0Pover→ start_ARG O end_ARG ⊙ v = 0, which implies v=00v=0v = 0. Thus, OO is injective. ∎ Corollary C.14. When there is noise in the observation kernel and the human is a Bayesian reasoner with some prior B such that B⁢(s→)>0→0B( s)>0B ( over→ start_ARG s end_ARG ) > 0 for all s→∈→ s∈ Sover→ start_ARG s end_ARG ∈ over→ start_ARG S end_ARG, then the return function is identifiable from choice probabilities of state sequences even if the learning system does not know the human’s observations. Proof. This follows from the injectivity of OO, the injectivity of BB that we proved in Proposition C.13, and Theorem C.9. ∎ Remark C.15. We mention the following caveat: intuitively, one could think that OO (and thus BB, by Proposition C.13) will be injective if every s→ sover→ start_ARG s end_ARG is identifiable from infinitely many i.i.d. samples from PO→⁢(o→∣s→)subscript→conditional→P_ O( o s)Pover→ start_ARG O end_ARG ( over→ start_ARG o end_ARG ∣ over→ start_ARG s end_ARG ). A counterexample is the following: =(1/21/41/41/41/21/43/83/81/4).matrix121414141214383814 O= pmatrix1/2&1/4&1/4\\ 1/4&1/2&1/4\\ 3/8&3/8&1/4 pmatrix.O = ( start_ARG start_ROW start_CELL 1 / 2 end_CELL start_CELL 1 / 4 end_CELL start_CELL 1 / 4 end_CELL end_ROW start_ROW start_CELL 1 / 4 end_CELL start_CELL 1 / 2 end_CELL start_CELL 1 / 4 end_CELL end_ROW start_ROW start_CELL 3 / 8 end_CELL start_CELL 3 / 8 end_CELL start_CELL 1 / 4 end_CELL end_ROW end_ARG ) . In this case, the rows are linearly dependent with coefficients 1/2,1/212121/2,1/21 / 2 , 1 / 2 and −11-1- 1. Consequently, OO and BB are not injective, and so if this observation kernel comes from a multi-armed bandit with three states, then Corollary C.4 shows that the return function is not identifiable. Nevertheless, the distributions PO→(⋅∣s→)P_ O(· s)Pover→ start_ARG O end_ARG ( ⋅ ∣ over→ start_ARG s end_ARG ) (given by the rows) all differ from each other, and so infinitely many i.i.d. samples identify the state sequence s→ sover→ start_ARG s end_ARG. C.6 Robustness of Return Function Identifiability under Belief Misspecification We now again look at the case where the observations that the human observes are known to the reward learning system, as in Section C.2. Furthermore, we assume that :ℝ→ℝΩ→:→superscriptℝ→superscriptℝ→Ω B: R S→ R % B : blackboard_Rover→ start_ARG S end_ARG → blackboard_Rover→ start_ARG Ω end_ARG is such that ker⁡∩im⁡=0kernelim0 B % =\0\ker B ∩ im Γ = 0 . In this case, we can apply Corollary C.4 and identify the true return function G from ⁡(G) B(G)B ( G ), which, in turn, can be identified up to an additive constant from the observation-based choice probabilities with the argument as for Proposition 3.1. In this section, we investigate what happens when the human belief model is slightly misspecified. In other words: the learning system uses a perturbed matrix ≔+≔subscript B_ % B+ Bbold_Δ ≔ B + Δ with some small perturbation Δ. How much will the inferred return function deviate from the truth? To answer this, we first need to outline some norm theory of linear operators. C.6.1 Some Norm Theory for Linear Operators In this section, let V,WV,WV , W be two finite-dimensional inner product-spaces. In other words, V and W each have inner products ⟨⋅,⋅⟩⋅ ·,· ⟨ ⋅ , ⋅ ⟩ and there are linear isomorphisms V≅ℝksuperscriptℝV R^kV ≅ blackboard_Rk, W≅ℝmsuperscriptℝW R^mW ≅ blackboard_Rm such that the inner products in V and W correspond to the standard scalar products in ℝksuperscriptℝ R^kblackboard_Rk and ℝmsuperscriptℝ R^mblackboard_Rm. The reason that we don’t directly work with ℝksuperscriptℝ R^kblackboard_Rk and ℝmsuperscriptℝ R^mblackboard_Rm itself is that we will later apply the analysis to the case that V=im⁡⊆ℝ→imsuperscriptℝ→V=im R % SV = im Γ ⊆ blackboard_Rover→ start_ARG S end_ARG. Let in this whole section :V→W:→ A:V→ WA : V → W be a linear operator and :V→W:→ :V→ WΔ : V → W be a perturbance, so that ≔+≔subscript A_ % A+ Abold_Δ ≔ A + Δ is a perturbed version of AA. The inner products give rise to a norm on V and W defined by ‖v‖=⟨v,v⟩,‖w‖=⟨w,w⟩.formulae-sequencenormnorm\|v\|= v,v , \|w\|= w,w% .∥ v ∥ = square-root start_ARG ⟨ v , v ⟩ end_ARG , ∥ w ∥ = square-root start_ARG ⟨ w , w ⟩ end_ARG . As is well known, for each linear operator :V→W:→ A:V→ WA : V → W there exists a unique, basis-independent adjoint (generalizing the notion of a transpose) T:W→V:superscript→ A^T:W→ VAitalic_T : W → V such that for all v∈Vv∈ Vv ∈ V and w∈Ww∈ Ww ∈ W, we have ⟨⁡v,w⟩=⟨v,T⁡w⟩.superscript Av,w = v,% A^Tw .⟨ A v , w ⟩ = ⟨ v , Aitalic_T w ⟩ . Let us recall the following fact that is often used in linear regression: Lemma C.16. Assume :V→W:→ A:V→ WA : V → W is injective. Then T⁡:V→V:superscript→ A^T A:V→ VAitalic_T A : V → V is invertible and (T⁡)−1⁢Tsuperscriptsuperscript1superscript( A^T A)^-1% A^T( Aitalic_T A )- 1 Aitalic_T is a left inverse of AA. Proof. To show that T⁡superscript A^T AAitalic_T A is invertible, we only need to show that it is injective. Thus, let 0≠x∈V00≠ x∈ V0 ≠ x ∈ V. Then ⟨x,T⁡x⟩=⟨⁡x,⁡x⟩=‖⁡x‖2>0,superscriptsuperscriptnorm20 x, A^T Ax % = Ax, Ax% =\| Ax\|^2>0,⟨ x , Aitalic_T A x ⟩ = ⟨ A x , A x ⟩ = ∥ A x ∥2 > 0 , where the last step followed from the injectivity of AA. Thus, T⁡x≠0superscript0 A^T Ax≠ 0Aitalic_T A x ≠ 0, and so T⁡superscript A^T AAitalic_T A is injective, and thus invertible. Consequently, (T⁡)−1⁢Tsuperscriptsuperscript1superscript( A^T A)^-1% A^T( Aitalic_T A )- 1 Aitalic_T is a well-defined operator. That it is the left inverse of AA is clear. ∎ Definition C.17 (Operator Norm). The norm of an operator :V→W:→ A:V→ WA : V → W is given by ‖≔maxx,‖x‖=1⁡‖⁡x‖.≔normsubscriptnorm1norm\| A\| _x,\ \|x\|=1\|% Ax\|.∥ A ∥ ≔ maxitalic_x , ∥ x ∥ = 1 ∥ A x ∥ . It has the following well-known properties, where , A, BA , B and CC are matrices of compatible sizes: ‖+‖≤‖+‖,‖⁡‖≤‖⋅‖,‖T‖=‖.formulae-sequencenormnormnormformulae-sequencenorm⋅normnormnormsuperscriptnorm\| A+ B\|≤\|% A\|+\| B\|, \| C% A\|≤\| C\|·\|% A\|, \| A^T\|=\|% A\|.∥ A + B ∥ ≤ ∥ A ∥ + ∥ B ∥ , ∥ C A ∥ ≤ ∥ C ∥ ⋅ ∥ A ∥ , ∥ Aitalic_T ∥ = ∥ A ∥ . To study how a perturbance in AA (and thus T⁡superscript A^T AAitalic_T A) transfers into a perturbance of (T⁡)−1superscriptsuperscript1 ( A^T A )^-1( Aitalic_T A )- 1, we will use the following theorem: Theorem C.18 (El Ghaoui [2002]). Let :V→V:→ B:V→ VB : V → V be an invertible operator. Let ρ<‖−1‖−1superscriptnormsuperscript11ρ<\| B^-1\|^-1ρ < ∥ B- 1 ∥- 1. Let :V→V:→ :V→ VΔ : V → V be any operator with ‖≤ρnorm\| \|≤ρ∥ Δ ∥ ≤ ρ. Then + B+ B + Δ is invertible and we have ‖(+)−1−1‖≤ρ⋅‖−1‖−1‖−1−ρ.normsuperscript1superscript1⋅normsuperscript1superscriptnormsuperscript11 \|( B+ )^-1-% B^-1 \|≤ ρ·\|% B^-1\|\| B^-1\|^-1-ρ.∥ ( B + Δ )- 1 - B- 1 ∥ ≤ divide start_ARG ρ ⋅ ∥ B- 1 ∥ end_ARG start_ARG ∥ B- 1 ∥- 1 - ρ end_ARG . Proof. See El Ghaoui [2002], Section 7 and in particular Equation 7.2. Note that the reference defines ‖norm\| A\|∥ A ∥ to be the largest singular value of AA; by the well-known min-max theorem, this is equivalent to Definition C.17. ∎ We will apply this theorem to T⁡superscript A^T AAitalic_T A, which raises the question about the size of the perturbance in T⁡superscript A^T AAitalic_T A for a given perturbance in AA. This is clarified in the following lemma. Before stating it, for a given perturbance ρ, define ρ~⁢()≔ρ⋅(2⋅‖+ρ),≔~⋅2norm ρ( A) ρ· (2·\|% A\|+ρ ),over~ start_ARG ρ end_ARG ( A ) ≔ ρ ⋅ ( 2 ⋅ ∥ A ∥ + ρ ) , which depends on AA and ρ. Also, recall that for a given perturbance Δ, we define ≔+≔subscript A_ % A+ Abold_Δ ≔ A + Δ. We obtain: Lemma C.19. Assume that ‖≤ρnorm\| \|≤ρ∥ Δ ∥ ≤ ρ. Then ‖T⁡−T⁡‖≤ρ~⁢().normsuperscriptsubscriptsubscriptsuperscript~\| A_ ^T% A_ - A^T% A\|≤ ρ( A).∥ Abold_Δitalic_T Abold_Δ - Aitalic_T A ∥ ≤ over~ start_ARG ρ end_ARG ( A ) . Proof. We have ‖T⁡−T⁡‖normsuperscriptsubscriptsubscriptsuperscript \| A_ % ^T A_ -% A^T A \|∥ Abold_Δitalic_T Abold_Δ - Aitalic_T A ∥ =‖(+)T⁢(+)−T⁡‖absentnormsuperscriptsuperscript = \|( A+ % )^T( A+ )- % A^T A \|= ∥ ( A + Δ )T ( A + Δ ) - Aitalic_T A ∥ =‖T⁡+T⁡+T⁡‖absentnormsuperscriptsuperscriptsuperscript = \| A^T % + ^T A+% ^T \|= ∥ Aitalic_T Δ + Δitalic_T A + Δitalic_T Δ ∥ ≤‖⋅‖+‖⋅‖+‖2absent⋅normnorm⋅normnormsuperscriptnorm2 ≤\| A\|·\| % \|+\| \|·\| A% \|+\| \|^2≤ ∥ A ∥ ⋅ ∥ Δ ∥ + ∥ Δ ∥ ⋅ ∥ A ∥ + ∥ Δ ∥2 ≤ρ⋅(2⋅‖+ρ)absent⋅2norm ≤ρ· (2·\| A\|+ρ )≤ ρ ⋅ ( 2 ⋅ ∥ A ∥ + ρ ) =ρ~⁢().absent~ = ρ( A).= over~ start_ARG ρ end_ARG ( A ) . ∎ To be able to apply Theorem C.18 to T⁡superscript A^T AAitalic_T A, we need to make sure that ρ~⁢()~ ρ( A)over~ start_ARG ρ end_ARG ( A ) is bounded above by ‖(T⁡)−1‖−1superscriptnormsuperscriptsuperscript11 \|( A^T A )^-1\|^% -1∥ ( Aitalic_T A )- 1 ∥- 1. The next lemma clarifies what condition ρ needs to satisfy for ρ~⁢()~ ρ( A)over~ start_ARG ρ end_ARG ( A ) to obey that bound. For this, define τ⁢()≔−‖+‖2+‖(T⁡)−1‖−1,≔normsuperscriptnorm2superscriptnormsuperscriptsuperscript11τ( A) -\| A\|+ % \| A\|^2+ \|( A^T% A)^-1 \|^-1,τ ( A ) ≔ - ∥ A ∥ + square-root start_ARG ∥ A ∥2 + ∥ ( Aitalic_T A )- 1 ∥- 1 end_ARG , (9) which only depends on AA. Lemma C.20. Assume ρ<τ⁢()ρ<τ( A)ρ < τ ( A ). Then ρ~⁢()<‖(T⁡)−1‖−1.~superscriptnormsuperscriptsuperscript11 ρ( A)< \|( A% ^T A)^-1 \|^-1.over~ start_ARG ρ end_ARG ( A ) < ∥ ( Aitalic_T A )- 1 ∥- 1 . Proof. Note that ρ=τ⁢()ρ=τ( A)ρ = τ ( A ) is the positive solution to the following quadratic equation in the indeterminate ρ: ρ2+2⋅‖⋅ρ−‖(T⁡)−1‖−1=ρ~⁢()−‖(T⁡)−1‖−1=0.superscript2⋅2normsuperscriptnormsuperscriptsuperscript11~superscriptnormsuperscriptsuperscript110ρ^2+2·\| A\|·ρ- \|(% A^T A)^-1 \|^-1= ρ(% A)- \|( A^T % A)^-1 \|^-1=0.ρ2 + 2 ⋅ ∥ A ∥ ⋅ ρ - ∥ ( Aitalic_T A )- 1 ∥- 1 = over~ start_ARG ρ end_ARG ( A ) - ∥ ( Aitalic_T A )- 1 ∥- 1 = 0 . Since this is a convex parabola, we get the inequality ρ~⁢()−‖(T⁡)−1‖−1<0~superscriptnormsuperscriptsuperscript110 ρ( A)- \|( A% ^T A)^-1 \|^-1<0over~ start_ARG ρ end_ARG ( A ) - ∥ ( Aitalic_T A )- 1 ∥- 1 < 0 whenever we have 0≤ρ<τ⁢()00≤ρ<τ( A)0 ≤ ρ < τ ( A ), which shows the result. ∎ Finally, we put it all together to obtain a bound on the perturbance of (T⁡)−1⁢Tsuperscriptsuperscript1superscript ( A^T A )^-1% A^T( Aitalic_T A )- 1 Aitalic_T. For this, set C⁢(,ρ)≔ρ~⁢()⋅‖(T⁡)−1‖(T⁡)−1‖−1−ρ~⁢()⋅(‖+ρ)+‖(T⁡)−1‖⋅ρ.≔⋅~normsuperscriptsuperscript1superscriptnormsuperscriptsuperscript11~norm⋅normsuperscriptsuperscript1C( A,ρ) ρ( % A)· \| ( A^T% A )^-1 \| \| ( A^T% A )^-1 \|^-1- ρ(% A)· ( \| A % \|+ρ )+ \| ( A^T% A )^-1 \|·ρ.C ( A , ρ ) ≔ divide start_ARG over~ start_ARG ρ end_ARG ( A ) ⋅ ∥ ( Aitalic_T A )- 1 ∥ end_ARG start_ARG ∥ ( Aitalic_T A )- 1 ∥- 1 - over~ start_ARG ρ end_ARG ( A ) end_ARG ⋅ ( ∥ A ∥ + ρ ) + ∥ ( Aitalic_T A )- 1 ∥ ⋅ ρ . (10) We obtain: Proposition C.21. Assume ‖≤ρ<τ⁢()norm\| \|≤ρ<τ( A)∥ Δ ∥ ≤ ρ < τ ( A ). Then T⁡superscriptsubscriptsubscript A_ ^T% A_ Abold_Δitalic_T Abold_Δ is invertible, and we have ‖(T⁡)−1⁢T−(T⁡)−1⁢T‖≤C⁢(,ρ).normsuperscriptsuperscriptsubscriptsubscript1superscriptsubscriptsuperscriptsuperscript1superscript \| ( A_ ^T% A_ )^-1% A_ ^T- (% A^T A )^-1% A^T \|≤ C( A,ρ).∥ ( Abold_Δitalic_T Abold_Δ )- 1 Abold_Δitalic_T - ( Aitalic_T A )- 1 Aitalic_T ∥ ≤ C ( A , ρ ) . Proof. The invertibility of T⁡superscriptsubscriptsubscript A_ ^T% A_ Abold_Δitalic_T Abold_Δ follows from Theorem C.18, Lemma C.19 and Lemma C.20. We get ‖(T⁡)−1⁢T−(T⁡)−1⁢T‖normsuperscriptsuperscriptsubscriptsubscript1superscriptsubscriptsuperscriptsuperscript1superscript \| ( A_ % ^T A_ )% ^-1 A_ ^T- (% A^T A )^-1% A^T \|∥ ( Abold_Δitalic_T Abold_Δ )- 1 Abold_Δitalic_T - ( Aitalic_T A )- 1 Aitalic_T ∥ = == ‖[(T⁡)−1−(T⁡)−1]⋅T+(T⁡)−1⋅(T−T)‖norm⋅delimited-[]superscriptsuperscriptsubscriptsubscript1superscriptsuperscript1superscriptsubscript⋅superscriptsuperscript1superscriptsubscriptsuperscript \| [ ( A_% ^T A_ % )^-1- ( A^T A% )^-1 ]· A_ % ^T+ ( A^T A% )^-1· ( A_ % ^T- A^T ) \|∥ [ ( Abold_Δitalic_T Abold_Δ )- 1 - ( Aitalic_T A )- 1 ] ⋅ Abold_Δitalic_T + ( Aitalic_T A )- 1 ⋅ ( Abold_Δitalic_T - Aitalic_T ) ∥ ≤ ≤ ‖(T⁡)−1−(T⁡)−1‖⋅‖+‖(T⁡)−1‖⋅‖⋅normsuperscriptsuperscriptsubscriptsubscript1superscriptsuperscript1normsubscript⋅normsuperscriptsuperscript1norm \| ( A_ % ^T A_ )% ^-1- ( A^T A )^-1% \|· \| A_ % \|+ \| ( A^T A% )^-1 \|·\| \|∥ ( Abold_Δitalic_T Abold_Δ )- 1 - ( Aitalic_T A )- 1 ∥ ⋅ ∥ Abold_Δ ∥ + ∥ ( Aitalic_T A )- 1 ∥ ⋅ ∥ Δ ∥ ≤ ≤ ρ~⁢()⋅‖(T⁡)−1‖(T⁡)−1‖−1−ρ~⁢()⋅(‖+ρ)+‖(T⁡)−1‖⋅ρ⋅~normsuperscriptsuperscript1superscriptnormsuperscriptsuperscript11~norm⋅normsuperscriptsuperscript1 ρ( A)· \|% ( A^T A )^-1 % \| \| ( A^T A % )^-1 \|^-1- ρ( A)· (% \| A \|+ρ )+ \| (% A^T A )^-1 \|· start_ARG over~ start_ARG ρ end_ARG ( A ) ⋅ ∥ ( Aitalic_T A )- 1 ∥ end_ARG start_ARG ∥ ( Aitalic_T A )- 1 ∥- 1 - over~ start_ARG ρ end_ARG ( A ) end_ARG ⋅ ( ∥ A ∥ + ρ ) + ∥ ( Aitalic_T A )- 1 ∥ ⋅ ρ = == C⁢(,ρ). C( A,ρ).C ( A , ρ ) . In the second-to-last step, we used Theorem C.18. ∎ The constant C⁢(,ρ)C( A,ρ)C ( A , ρ ), defined in Equation (10), has a fairly complicated form. In the following proposition, we find an easier-to-study upper bound in a special case: Proposition C.22. Assume that ρ≤‖normρ≤\| A\|ρ ≤ ∥ A ∥ and ρ≤−‖+‖2+1/2⋅‖(T⁡)−1‖−1normsuperscriptnorm2⋅12superscriptnormsuperscriptsuperscript11ρ≤-\| A\|+ \| A\|^2% +1/2· \|( A^T A)^-1% \|^-1ρ ≤ - ∥ A ∥ + square-root start_ARG ∥ A ∥2 + 1 / 2 ⋅ ∥ ( Aitalic_T A )- 1 ∥- 1 end_ARG.222Note the factor 1/2121/21 / 2 compared to the definition of τ⁢()τ( A)τ ( A ) in Equation (9). Then we have C⁢(,ρ)≤ρ⋅‖(T⁡)−1‖⋅[12⋅‖2⋅‖(T⁡)−1‖+1].⋅normsuperscriptsuperscript1delimited-[]⋅12superscriptnorm2normsuperscriptsuperscript11C( A,ρ)≤ρ· \|( A% ^T A)^-1 \|· [12·\|% A\|^2· \|( A^T% A)^-1 \|+1 ].C ( A , ρ ) ≤ ρ ⋅ ∥ ( Aitalic_T A )- 1 ∥ ⋅ [ 12 ⋅ ∥ A ∥2 ⋅ ∥ ( Aitalic_T A )- 1 ∥ + 1 ] . Proof. The second assumption gives, as in the proof of Lemma C.20, that ρ~⁢()≤1/2⋅‖(T⁡)−1‖−1~⋅12superscriptnormsuperscriptsuperscript11 ρ( A)≤ 1/2· \|( % A^T A)^-1 \|^-1over~ start_ARG ρ end_ARG ( A ) ≤ 1 / 2 ⋅ ∥ ( Aitalic_T A )- 1 ∥- 1. Together with ρ≤‖normρ≤\| A\|ρ ≤ ∥ A ∥, the result follows. ∎ C.6.2 Application to Bounds in the Error of the Return Function We now apply the results from the preceding section to our case. Define r⁢():im⁡→ℝΩ→:r→imsuperscriptℝ→Ωr( B):im % → R r ( B ) : im Γ → blackboard_Rover→ start_ARG Ω end_ARG as the restriction of the belief operator BB to im⁡imim im Γ. Assume that ker⁡∩im⁡=0kernelim0 B % =\0\ker B ∩ im Γ = 0 , which is, according to Corollary C.4, a sufficient condition for identifiability. Note that this condition means that r⁢()rr( B)r ( B ) is injective. Thus, Lemma C.16 ensures that r⁢()T⁢r⁢()rsuperscriptrr( B)^Tr( B)r ( B )T r ( B ) is invertible and that (r⁢()T⁢r⁢())−1⁢r⁢()Tsuperscriptrsuperscriptr1rsuperscript (r( B)^Tr(% B) )^-1r( B)^T( r ( B )T r ( B ) )- 1 r ( B )T is a left inverse of r⁢()rr( B)r ( B ). Consequently, from the equation r⁢()⁢(G)=⁡(G)rr( B)(G)= B(G)r ( B ) ( G ) = B ( G ) we obtain G=(r⁢()T⁢r⁢())−1⁢r⁢()T⁢(⁡(G)).superscriptrsuperscriptr1rsuperscriptG= (r( B)^Tr(% B) )^-1r( B)^T(% B(G)).G = ( r ( B )T r ( B ) )- 1 r ( B )T ( B ( G ) ) . This is the concrete formula with which G can be identified from ⁡(G) B(G)B ( G ). When perturbing BB, this leads to a corresponding perturbance in (r⁢()T⁢r⁢())−1⁢r⁢()Tsuperscriptrsuperscriptr1rsuperscript (r( B)^Tr(% B) )^-1r( B)^T( r ( B )T r ( B ) )- 1 r ( B )T whose size influences the maximal error in the inference of G. This, in turn, influences the size of the error in JGsubscriptJ_GJitalic_G, the policy evaluation function, where JG⁢(π)≔s→∼Pπ⁢(s→)[G⁢(s→)].≔subscriptsubscriptsimilar-to→superscript→J_G(π) *E_ s P^π( s)% [G( s) ].Jitalic_G ( π ) ≔ Eover→ start_ARG s end_ARG ∼ Pitalic_π ( over→ start_ARG s end_ARG ) [ G ( over→ start_ARG s end_ARG ) ] . We obtain: Theorem C.23. Let G be the true reward function, BB the belief operator corresponding to the human’s true belief model B⁢(s→∣o→)conditional→B( s o)B ( over→ start_ARG s end_ARG ∣ over→ start_ARG o end_ARG ), and ⁡(G) B(G)B ( G ) be the resulting observation-based return function. Assume that ker⁡∩im⁡=0kernelim0 B % =\0\ker B ∩ im Γ = 0 , so that r⁢()T⁢r⁢()rsuperscriptrr( B)^Tr( B)r ( B )T r ( B ) is invertible. Let :ℝ→ℝΩ→:→superscriptℝ→superscriptℝ→Ω : R S→ R^% Δ : blackboard_Rover→ start_ARG S end_ARG → blackboard_Rover→ start_ARG Ω end_ARG be a perturbation satisfying ‖≤ρnorm\| \|≤ρ∥ Δ ∥ ≤ ρ, where ρ satisfies the following two properties: ρ≤‖r⁢()‖,ρ≤−‖r⁢()‖+‖r⁢()‖2+1/2⋅‖(r⁢()T⁢r⁢())−1‖−1.formulae-sequencenormrnormrsuperscriptnormr2⋅12superscriptnormsuperscriptrsuperscriptr11ρ≤ \|r( B) \|, ρ≤-% \|r( B) \|+ \|r% ( B) \|^2+1/2· \| (r(% B)^Tr( B) )^-1% \|^-1.ρ ≤ ∥ r ( B ) ∥ , ρ ≤ - ∥ r ( B ) ∥ + square-root start_ARG ∥ r ( B ) ∥2 + 1 / 2 ⋅ ∥ ( r ( B )T r ( B ) )- 1 ∥- 1 end_ARG . Let ≔+≔subscript B_ % B+ Bbold_Δ ≔ B + Δ be the misspecified belief operator. The first claim is that r⁢()T⁢r⁢()rsuperscriptsubscriptrsubscriptr( B_ )^T% r( B_ )r ( Bbold_Δ )T r ( Bbold_Δ ) is invertible under these conditions. Now, assume that the learning system infers the return function G~≔(r⁢()T⁢r⁢())−1⁢r⁢()T⁢(⁡(G))≔~superscriptrsuperscriptsubscriptrsubscript1rsuperscriptsubscript G (r( B_% )^Tr( B_% ) )^-1r( B_% )^T( B(G))over~ start_ARG G end_ARG ≔ ( r ( Bbold_Δ )T r ( Bbold_Δ ) )- 1 r ( Bbold_Δ )T ( B ( G ) ).333Note that there is not necessarily a G~~ Gover~ start_ARG G end_ARG with r⁢()⁢(G~)=⁡(G)rsubscript~r( B_ )( % G)= B(G)r ( Bbold_Δ ) ( over~ start_ARG G end_ARG ) = B ( G ) since r⁢()rsubscriptr( B_ )r ( Bbold_Δ ) is not always surjective. Nevertheless, G~≔(r⁢()T⁢r⁢())−1⁢r⁢()T⁢(⁡(G))≔~superscriptrsuperscriptsubscriptrsubscript1rsuperscriptsubscript G (r( B_% )^Tr( B_% ) )^-1r( B_% )^T( B(G))over~ start_ARG G end_ARG ≔ ( r ( Bbold_Δ )T r ( Bbold_Δ ) )- 1 r ( Bbold_Δ )T ( B ( G ) ) is the best attempt at a solution in the sense that r⁢()⁢(G~)rsubscript~r( B_ )( % G)r ( Bbold_Δ ) ( over~ start_ARG G end_ARG ) then minimizes the Euclidean distance to ⁡(G) B(G)B ( G ). Then there is a polynomial Q⁢(X,Y)Q(X,Y)Q ( X , Y ) of degree five such that ‖G~−G‖≤‖G‖⋅Q⁢(‖(r⁢()T⁢r⁢())−1‖,‖r⁢()‖)⋅ρ.norm~⋅normnormsuperscriptrsuperscriptr1normr\| G-G\|≤\|G\|· Q ( \|(r(% B)^Tr( B))^-1 \|,\|% r( B)\| )·ρ.∥ over~ start_ARG G end_ARG - G ∥ ≤ ∥ G ∥ ⋅ Q ( ∥ ( r ( B )T r ( B ) )- 1 ∥ , ∥ r ( B ) ∥ ) ⋅ ρ . Thus, for all policies π, we obtain |JG~⁢(π)−JG⁢(π)|≤‖G‖⋅Q⁢(‖(r⁢()T⁢r⁢())−1‖,‖r⁢()‖)⋅ρ.subscript~subscript⋅normnormsuperscriptrsuperscriptr1normr |J_ G(π)-J_G(π) |≤\|G\|· Q ( \|(% r( B)^Tr( B)% )^-1 \|,\|r( B)\| )·ρ.| Jover~ start_ARG G end_ARG ( π ) - Jitalic_G ( π ) | ≤ ∥ G ∥ ⋅ Q ( ∥ ( r ( B )T r ( B ) )- 1 ∥ , ∥ r ( B ) ∥ ) ⋅ ρ . In particular, for sufficiently small perturbances ρ, the error in the inferred policy evaluation function JG~subscript~J_ GJover~ start_ARG G end_ARG becomes arbitrarily small. Proof. That r⁢()T⁢r⁢()rsuperscriptsubscriptrsubscriptr( B_ )^T% r( B_ )r ( Bbold_Δ )T r ( Bbold_Δ ) is invertible follows immediately from Proposition C.21 by using that ‖r⁢()‖≤‖normrnorm\|r( )\|≤\| % \|∥ r ( Δ ) ∥ ≤ ∥ Δ ∥ and that r⁢()=r⁢()r⁢()rsubscriptrsubscriptrr( B_ )= % r( B)_r( )r ( Bbold_Δ ) = r ( B )r ( Δ ), together with the second bound on ρ (which implies the assumed bound in Proposition C.21). We have |JG~⁢(π)−JG⁢(π)|subscript~subscript |J_ G(π)-J_G(π) || Jover~ start_ARG G end_ARG ( π ) - Jitalic_G ( π ) | =|s→∼Pπ⁢(s→)[(G~−G)⁢(s→)]|absentsubscriptsimilar-to→superscript→~→ = | *E_ s P^π( s)% [( G-G)( s) ] |= | Eover→ start_ARG s end_ARG ∼ Pitalic_π ( over→ start_ARG s end_ARG ) [ ( over~ start_ARG G end_ARG - G ) ( over→ start_ARG s end_ARG ) ] | ≤s→∼Pπ⁢(s→)[|(G~−G)⁢(s→)|]absentsubscriptsimilar-to→superscript→~→ ≤ *E_ s P^π( s) % [ |( G-G)( s) | ]≤ Eover→ start_ARG s end_ARG ∼ Pitalic_π ( over→ start_ARG s end_ARG ) [ | ( over~ start_ARG G end_ARG - G ) ( over→ start_ARG s end_ARG ) | ] ≤maxs→∈→⁡|(G~−G)⁢(s→)|absentsubscript→~→ ≤ _ s∈ S |( G-G)( s% ) |≤ maxover→ start_ARG s end_ARG ∈ over→ start_ARG S end_ARG | ( over~ start_ARG G end_ARG - G ) ( over→ start_ARG s end_ARG ) | ≤‖G~−G‖absentnorm~ ≤\| G-G\|≤ ∥ over~ start_ARG G end_ARG - G ∥ =‖[(r⁢()T⁢r⁢())−1⁢r⁢()T−(r⁢()T⁢r⁢())−1⁢r⁢()T]⋅⁡(G)‖absentnorm⋅delimited-[]superscriptrsuperscriptsubscriptrsubscript1rsuperscriptsubscriptsuperscriptrsuperscriptr1rsuperscript = \| [ (r( B_% )^Tr( B_% ) )^-1r( B% _ )^T- (r(% B)^Tr( B) )^-1r(% B)^T ]· B(G) \|= ∥ [ ( r ( Bbold_Δ )T r ( Bbold_Δ ) )- 1 r ( Bbold_Δ )T - ( r ( B )T r ( B ) )- 1 r ( B )T ] ⋅ B ( G ) ∥ ≤‖(r⁢()T⁢r⁢())−1⁢r⁢()T−(r⁢()T⁢r⁢())−1⁢r⁢()T‖⋅‖⁡(G)‖absent⋅normsuperscriptrsuperscriptsubscriptrsubscript1rsuperscriptsubscriptsuperscriptrsuperscriptr1rsuperscriptnorm ≤ \| (r( B_% )^Tr( B_% ) )^-1r( B% _ )^T- (r(% B)^Tr( B) )^-1r(% B)^T \|· \| B(G% ) \|≤ ∥ ( r ( Bbold_Δ )T r ( Bbold_Δ ) )- 1 r ( Bbold_Δ )T - ( r ( B )T r ( B ) )- 1 r ( B )T ∥ ⋅ ∥ B ( G ) ∥ ≤C⁢(r⁢(),ρ)⋅‖r⁢()⁢(G)‖absent⋅rnormr ≤ C(r( B),ρ)·\|% r( B)(G)\|≤ C ( r ( B ) , ρ ) ⋅ ∥ r ( B ) ( G ) ∥ ≤C⁢(r⁢(),ρ)⋅‖r⁢()‖⋅‖G‖.absent⋅rnormrnorm ≤ C(r( B),ρ)·\|% r( B)\|·\|G\|.≤ C ( r ( B ) , ρ ) ⋅ ∥ r ( B ) ∥ ⋅ ∥ G ∥ . In the second to last step, we used Proposition C.21. By Proposition C.22, we can define the polynomial Q⁢(X,Y)Q(X,Y)Q ( X , Y ) by Q⁢(X,Y)=X⁢Y⋅[12⁢X⁢Y2+1],⋅delimited-[]12superscript21Q(X,Y)=XY· [12XY^2+1 ],Q ( X , Y ) = X Y ⋅ [ 12 X Y2 + 1 ] , which is of degree five. The last claim follows from limρ→0ρ=0subscript→00 _ρ→ 0ρ=0limitalic_ρ → 0 ρ = 0. ∎ Remark C.24. In the case of a square matrix BB that is injective, we can apply Theorem C.18 directly to −1superscript1 B^-1B- 1 (which is now invertible) and obtain the following simplification of Theorem C.23 for the case that ‖≤ρ≤12⋅‖−1‖−1norm⋅12superscriptnormsuperscript11\| \|≤ρ≤ 12·\|% B^-1\|^-1∥ Δ ∥ ≤ ρ ≤ divide start_ARG 1 end_ARG start_ARG 2 end_ARG ⋅ ∥ B- 1 ∥- 1: |JG~⁢(π)−JG⁢(π)|≤ρ⋅2⋅‖⋅‖G‖⋅‖−1‖2.subscript~subscript⋅2normnormsuperscriptnormsuperscript12 |J_ G(π)-J_G(π) |≤ρ· 2·\| % B\|·\|G\|·\| B^-1\|^2.| Jover~ start_ARG G end_ARG ( π ) - Jitalic_G ( π ) | ≤ ρ ⋅ 2 ⋅ ∥ B ∥ ⋅ ∥ G ∥ ⋅ ∥ B- 1 ∥2 . The polynomial is then only of degree 3. C.7 Preliminary Characterizations of the Ambiguity Recall the sequence of functions ℝsuperscriptℝ R^Sblackboard_RSℝ→superscriptℝ→ R Sblackboard_Rover→ start_ARG S end_ARGℝΩ→.superscriptℝ→Ω R .blackboard_Rover→ start_ARG Ω end_ARG . Γ BB In this section, we clarify im⁡imim im Γ and ker⁡kernel Bker B in special cases, as their intersection is the crucial ambiguity in Theorem C.2. The following proposition shows that for deterministic PO→subscript→P_ OPover→ start_ARG O end_ARG and a rational human, ker⁡kernel Bker B decomposes into hyperplanes defined by normal vectors of probabilities of sequences mapping to the same observation sequence: Proposition C.25. Assume the human reasons as in Section C.1. Assume PO→subscript→P_ OPover→ start_ARG O end_ARG is deterministic. Let B⁢(s→)→B( s)B ( over→ start_ARG s end_ARG ) be the distribution of sequences under the human’s belief over the policy, given by B⁢(s→)=∫π′B⁢(π′)⁢Pπ′⁢(s→)→subscriptsuperscript′superscriptsuperscript′→B( s)= _π B(π )P^π ( s)B ( over→ start_ARG s end_ARG ) = ∫π′ B ( π′ ) Pitalic_π start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT ( over→ start_ARG s end_ARG ) for some policy prior B⁢(π′)superscript′B(π )B ( π′ ). For each o→ oover→ start_ARG o end_ARG, let Bo→≔[B⁢(s→)]s→:O→⁢(s→)=o→∈ℝs→∈→∣O→⁢(s→)=o→≔subscript→subscriptdelimited-[]→:→superscriptℝconditional-set→B_ o [B( s)]_ s:\ O( s)= o∈% R^\ s∈ S\ \ O( s)= o\Bover→ start_ARG o end_ARG ≔ [ B ( over→ start_ARG s end_ARG ) ]over→ start_ARG s end_ARG : over→ start_ARG O end_ARG ( over→ start_ARG s end_ARG ) = over→ start_ARG o end_ARG ∈ blackboard_R over→ start_ARG s end_ARG ∈ over→ start_ARG S end_ARG ∣ over→ start_ARG O end_ARG ( over→ start_ARG s end_ARG ) = over→ start_ARG o end_ARG be the vector of probabilities of sequences that are observed as o→ oover→ start_ARG o end_ARG. Let G′ be a return function. For each o→∈Ω→Ω o∈ over→ start_ARG o end_ARG ∈ over→ start_ARG Ω end_ARG, define the restriction Go→′∈ℝs→∈→∣O→⁢(s→)=o→subscriptsuperscript′→superscriptℝconditional-set→G _ o∈ R^\ s∈ S O(% s)= o\G′over→ start_ARG o end_ARG ∈ blackboard_R over→ start_ARG s end_ARG ∈ over→ start_ARG S end_ARG ∣ over→ start_ARG O end_ARG ( over→ start_ARG s end_ARG ) = over→ start_ARG o end_ARG by Go→′⁢(s→)≔G′⁢(s→)≔subscriptsuperscript′→superscript′→G _ o( s) G ( s)G′over→ start_ARG o end_ARG ( over→ start_ARG s end_ARG ) ≔ G′ ( over→ start_ARG s end_ARG ) for all s→∈s→∈→∣O→⁢(s→)=o→conditional-set→ s∈\ s∈ S O( s)= o\over→ start_ARG s end_ARG ∈ over→ start_ARG s end_ARG ∈ over→ start_ARG S end_ARG ∣ over→ start_ARG O end_ARG ( over→ start_ARG s end_ARG ) = over→ start_ARG o end_ARG . Assume that B⁢(s→∣o→)conditional→B( s o)B ( over→ start_ARG s end_ARG ∣ over→ start_ARG o end_ARG ) is the Bayesian posterior. Then G′∈ker⁡superscript′kernelG ∈ BG′ ∈ ker B if and only if the property Bo→⋅Go→′=0⋅subscript→subscriptsuperscript′→0B_ o· G _ o=0Bover→ start_ARG o end_ARG ⋅ G′over→ start_ARG o end_ARG = 0 holds for all o→∈Ω→Ω o∈ over→ start_ARG o end_ARG ∈ over→ start_ARG Ω end_ARG. Proof. For a deterministic observation kernel PO→subscript→P_ OPover→ start_ARG O end_ARG, by Bayes rule we have B⁢(s→∣o→)conditional→ B( s o)B ( over→ start_ARG s end_ARG ∣ over→ start_ARG o end_ARG ) =PO→⁢(o→∣s→)⋅B⁢(s→)∑s→′PO→⁢(o→∣s→′)⋅B⁢(s→′)absent⋅subscript→conditional→subscriptsuperscript→′⋅subscript→conditional→superscript→′ = P_ O( o s)· B( s) _% s 1.0pt P_ O( o s 1.0pt^% )· B( s 1.0pt )= divide start_ARG Pover→ start_ARG O end_ARG ( over→ start_ARG o end_ARG ∣ over→ start_ARG s end_ARG ) ⋅ B ( over→ start_ARG s end_ARG ) end_ARG start_ARG ∑over→ start_ARG s end_ARG′ Pover→ start_ARG O end_ARG ( over→ start_ARG o end_ARG ∣ over→ start_ARG s end_ARG′ ) ⋅ B ( over→ start_ARG s end_ARG′ ) end_ARG =δo→⁢(O→⁢(s→))⋅B⁢(s→)∑s→′δo→⁢(O→⁢(s→′))⋅B⁢(s→′)absent⋅subscript→subscriptsuperscript→′⋅subscript→superscript→′ = _ o ( O( s) )· B( % s) _ s 1.0pt _ o ( O( s% 1.0pt ) )· B( s 1.0pt )= divide start_ARG δover→ start_ARG o end_ARG ( over→ start_ARG O end_ARG ( over→ start_ARG s end_ARG ) ) ⋅ B ( over→ start_ARG s end_ARG ) end_ARG start_ARG ∑over→ start_ARG s end_ARG′ δover→ start_ARG o end_ARG ( over→ start_ARG O end_ARG ( over→ start_ARG s end_ARG′ ) ) ⋅ B ( over→ start_ARG s end_ARG′ ) end_ARG =0,O→⁢(s→)≠o→B⁢(s→)∑s→′:O→⁢(s→′)=o→B⁢(s→′),O→⁢(s→)=o→.absentcases0→otherwise→subscript:superscript→′→superscript→′→superscript→′→otherwise = cases0,\ O( s)≠ o\\ B( s) _ s 1.0pt :\ O( s 1% .0pt )= oB( s 1.0pt ),\ O( s)=% o.\ cases= start_ROW start_CELL 0 , over→ start_ARG O end_ARG ( over→ start_ARG s end_ARG ) ≠ over→ start_ARG o end_ARG end_CELL start_CELL end_CELL end_ROW start_ROW start_CELL divide start_ARG B ( over→ start_ARG s end_ARG ) end_ARG start_ARG ∑over→ start_ARG s end_ARG′ : over→ start_ARG O end_ARG ( over→ start_ARG s end_ARG′ ) = over→ start_ARG o end_ARG B ( over→ start_ARG s end_ARG′ ) end_ARG , over→ start_ARG O end_ARG ( over→ start_ARG s end_ARG ) = over→ start_ARG o end_ARG . end_CELL start_CELL end_CELL end_ROW Thus, for any return function G′ and any observation sequence o→ oover→ start_ARG o end_ARG, we have [⁡(G′)]⁢(o→)delimited-[]superscript′→ [ B(G ) ]( o)[ B ( G′ ) ] ( over→ start_ARG o end_ARG ) =s→∼B⁢(s→∣o→)[G′⁢(s→)]absentsubscriptsimilar-to→conditional→superscript′→ = *E_ s B( s o)% [G ( s) ]= Eover→ start_ARG s end_ARG ∼ B ( over→ start_ARG s end_ARG ∣ over→ start_ARG o end_ARG ) [ G′ ( over→ start_ARG s end_ARG ) ] =∑s→B⁢(s→∣o→)⁢G′⁢(s→)absentsubscript→conditional→superscript′→ = _ sB( s o)G ( s)= ∑over→ start_ARG s end_ARG B ( over→ start_ARG s end_ARG ∣ over→ start_ARG o end_ARG ) G′ ( over→ start_ARG s end_ARG ) =∑s→:O→⁢(s→)=o→B⁢(s→)∑s→′:O→⁢(s→′)=o→B⁢(s→′)⁢G′⁢(s→)absentsubscript:→subscript:superscript→′→superscript→′→superscript→′ = _ s:\ O( s)= o B( s) _% s 1.0pt :\ O( s 1.0pt )= o% B( s 1.0pt )G ( s)= ∑over→ start_ARG s end_ARG : over→ start_ARG O end_ARG ( over→ start_ARG s end_ARG ) = over→ start_ARG o end_ARG divide start_ARG B ( over→ start_ARG s end_ARG ) end_ARG start_ARG ∑over→ start_ARG s end_ARG′ : over→ start_ARG O end_ARG ( over→ start_ARG s end_ARG′ ) = over→ start_ARG o end_ARG B ( over→ start_ARG s end_ARG′ ) end_ARG G′ ( over→ start_ARG s end_ARG ) =(∑s→′:O→⁢(s→′)=o→B⁢(s→′))−1⋅∑s→:O→⁢(s→)=o→B⁢(s→)⁢G′⁢(s→).absent⋅superscriptsubscript:superscript→′→superscript→′→superscript→′1subscript:→superscript′→ = ( _ s 1.0pt :\ O( s% 1.0pt )= oB( s 1.0pt ) )^-1% · _ s:\ O( s)= oB( s)G ( s).= ( ∑over→ start_ARG s end_ARG′ : over→ start_ARG O end_ARG ( over→ start_ARG s end_ARG′ ) = over→ start_ARG o end_ARG B ( over→ start_ARG s end_ARG′ ) )- 1 ⋅ ∑over→ start_ARG s end_ARG : over→ start_ARG O end_ARG ( over→ start_ARG s end_ARG ) = over→ start_ARG o end_ARG B ( over→ start_ARG s end_ARG ) G′ ( over→ start_ARG s end_ARG ) . Thus, we have G′∈ker⁡superscript′kernelG ∈ BG′ ∈ ker B if and only if Bo→⋅Go→′=∑s→:O→⁢(s→)=o→B⁢(s→)⁢G′⁢(s→)=0⋅subscript→subscriptsuperscript′→subscript:→superscript′→0B_ o· G _ o= _ s:\ O( s)= o% B( s)G ( s)=0Bover→ start_ARG o end_ARG ⋅ G′over→ start_ARG o end_ARG = ∑over→ start_ARG s end_ARG : over→ start_ARG O end_ARG ( over→ start_ARG s end_ARG ) = over→ start_ARG o end_ARG B ( over→ start_ARG s end_ARG ) G′ ( over→ start_ARG s end_ARG ) = 0 for all o→ oover→ start_ARG o end_ARG. That was to show. ∎ Remark C.26. One can interpret the previous proposition as follows: As long as O→ Oover→ start_ARG O end_ARG is injective, we have |s→∈→∣O→⁢(s→)=o|=1conditional-set→1 |\ s∈ S O( s)=o\ |=1| over→ start_ARG s end_ARG ∈ over→ start_ARG S end_ARG ∣ over→ start_ARG O end_ARG ( over→ start_ARG s end_ARG ) = o | = 1 for all o→ oover→ start_ARG o end_ARG, meaning that Bo→subscript→B_ oBover→ start_ARG o end_ARG and Go→′subscriptsuperscript′→G _ oG′over→ start_ARG o end_ARG have only one entry. Thus, Bo→⋅Go→′=0⋅subscript→subscriptsuperscript′→0B_ o· G _ o=0Bover→ start_ARG o end_ARG ⋅ G′over→ start_ARG o end_ARG = 0 implies Go→′=0subscriptsuperscript′→0G _ o=0G′over→ start_ARG o end_ARG = 0. If that holds for all o→ oover→ start_ARG o end_ARG, then G′∈ker⁡superscript′kernelG ∈ BG′ ∈ ker B implies G′=0superscript′0G =0G′ = 0, meaning BB is injective. However, as soon as there is an o→ oover→ start_ARG o end_ARG with ko→≔|s→∈→∣O→⁢(s→)=o|>1≔subscript→conditional-set→1k_ o |\ s∈ S O( s)=o% \ |>1kover→ start_ARG o end_ARG ≔ | over→ start_ARG s end_ARG ∈ over→ start_ARG S end_ARG ∣ over→ start_ARG O end_ARG ( over→ start_ARG s end_ARG ) = o | > 1, the equation Bo→⋅Go→′=0⋅subscript→subscriptsuperscript′→0B_ o· G _ o=0Bover→ start_ARG o end_ARG ⋅ G′over→ start_ARG o end_ARG = 0 leads to ko→−1subscript→1k_ o-1kover→ start_ARG o end_ARG - 1 free parameters in Go→′subscriptsuperscript′→G _ oG′over→ start_ARG o end_ARG. Go→′subscriptsuperscript′→G _ oG′over→ start_ARG o end_ARG can then be chosen freely in the hyperplane of vectors orthogonal to Bo→subscript→B_ oBover→ start_ARG o end_ARG without moving out of the kernel of BB. Another way of writing Proposition C.25 is to write ker⁡kernel Bker B as a direct sum of these hyperplanes perpendicular to Bo→subscript→B_ oBover→ start_ARG o end_ARG: ker⁡=⨁o→:|O→−1⁢(o→)|≥2Bo→⟂.kernelsubscriptdirect-sum:→superscript→1→2superscriptsubscript→perpendicular-to B= _ o:\ | O^-1( o)|≥ 2% B_ o .ker B = ⨁over→ start_ARG o end_ARG : | over→ start_ARG O end_ARG- 1 ( over→ start_ARG o end_ARG ) | ≥ 2 Bover→ start_ARG o end_ARG⟂ . Recall that a return function G is called time-separable if there exists a reward function R such that ⁡(R)=G (R)=GΓ ( R ) = G. Before we discuss time-separability in more interesting examples, we want to talk about one simple case where all return functions are time-separable. We leave a general characterization of im⁡imim im Γ to future work. Proposition C.27. Let there be an ordering s→(1),s→(2),…superscript→1superscript→2italic-… s^(1), s^(2),…over→ start_ARG s end_ARG( 1 ) , over→ start_ARG s end_ARG( 2 ) , … of all sequences in → Sover→ start_ARG S end_ARG, and a function ϕ:→:italic-ϕ→φ: S ϕ : over→ start_ARG S end_ARG → S from sequences to states such that ϕ⁢(s→)∈s→italic-ϕ→φ( s)∈ sϕ ( over→ start_ARG s end_ARG ) ∈ over→ start_ARG s end_ARG and ϕ⁢(s→(k))∉s→(i)italic-ϕsuperscript→superscript→φ( s^(k))∉ s^(i)ϕ ( over→ start_ARG s end_ARG( k ) ) ∉ over→ start_ARG s end_ARG( i ) for all i<ki<ki < k. Then every return function is time-separable. Proof. Let G be a return function. Initialize R⁢(s)=00R(s)=0R ( s ) = 0 for all s and inductively update it for all i=1,2,…12…i=1,2,…i = 1 , 2 , …: R⁢(ϕ⁢(s→(i)))italic-ϕsuperscript→ R (φ( s^(i)) )R ( ϕ ( over→ start_ARG s end_ARG( i ) ) ) ≔(∑t:st(i)=ϕ⁢(s→(i))γt)−1⋅(G⁢(s→(i))−∑t:st(i)≠ϕ⁢(s→(i))γt⋅R⁢(st(i))),≔absent⋅superscriptsubscript:subscriptsuperscriptitalic-ϕsuperscript→superscript1superscript→subscript:subscriptsuperscriptitalic-ϕsuperscript→⋅superscriptsubscriptsuperscript ( _t:\ s^(i)_t=φ( s^(i))γ% ^t )^-1· (G( s^(i))- _t:\ s^(i)_t≠φ(% s^(i))γ^t· R (s^(i)_t ) ),≔ ( ∑t : s( i ) start_POSTSUBSCRIPT t = ϕ ( over→ start_ARG s end_ARG( i ) ) end_POSTSUBSCRIPT γitalic_t )- 1 ⋅ ( G ( over→ start_ARG s end_ARG( i ) ) - ∑t : s( i ) start_POSTSUBSCRIPT t ≠ ϕ ( over→ start_ARG s end_ARG( i ) ) end_POSTSUBSCRIPT γitalic_t ⋅ R ( s( i )t ) ) , where the inductive definition always uses R as it is defined by that point in time. Once R⁢(ϕ⁢(s→(i)))italic-ϕsuperscript→R (φ( s^(i)) )R ( ϕ ( over→ start_ARG s end_ARG( i ) ) ) is defined, but not yet any future values R⁢(ϕ⁢(s→(k)))italic-ϕsuperscript→R (φ( s^(k)) )R ( ϕ ( over→ start_ARG s end_ARG( k ) ) ), k>ik>ik > i, we have [⁡(R)]⁢(s→(i))delimited-[]superscript→ [ (R) ]( s^(i))[ Γ ( R ) ] ( over→ start_ARG s end_ARG( i ) ) =∑t=0Tγt⋅R⁢(st(i))absentsuperscriptsubscript0⋅superscriptsubscriptsuperscript = _t=0^Tγ^t· R (s^(i)_t )= ∑t = 0T γitalic_t ⋅ R ( s( i )t ) =(∑t:st(i)=ϕ⁢(s→(i))γt)⋅R⁢(ϕ⁢(s→(i)))+∑t:st(i)≠ϕ⁢(s→(i))γt⋅R⁢(st(i))absent⋅subscript:subscriptsuperscriptitalic-ϕsuperscript→superscriptitalic-ϕsuperscript→subscript:subscriptsuperscriptitalic-ϕsuperscript→⋅superscriptsubscriptsuperscript = ( _t:\ s^(i)_t=φ( s^(i))γ^t% )· R (φ( s^(i)) )+ _t:\ s^(i)_t≠φ% ( s^(i))γ^t· R (s^(i)_t )= ( ∑t : s( i ) start_POSTSUBSCRIPT t = ϕ ( over→ start_ARG s end_ARG( i ) ) end_POSTSUBSCRIPT γitalic_t ) ⋅ R ( ϕ ( over→ start_ARG s end_ARG( i ) ) ) + ∑t : s( i ) start_POSTSUBSCRIPT t ≠ ϕ ( over→ start_ARG s end_ARG( i ) ) end_POSTSUBSCRIPT γitalic_t ⋅ R ( s( i )t ) =G⁢(s→(i)).absentsuperscript→ =G( s^(i)).= G ( over→ start_ARG s end_ARG( i ) ) . Furthermore, the property ϕ⁢(s→(k))∉s→(i)italic-ϕsuperscript→superscript→φ( s^(k))∉ s^(i)ϕ ( over→ start_ARG s end_ARG( k ) ) ∉ over→ start_ARG s end_ARG( i ) for all i<ki<ki < k ensures that changes to the reward function for k>ik>ik > i do not affect the value of [⁡(R)]⁢(s→(i))delimited-[]superscript→ [ (R) ]( s^(i))[ Γ ( R ) ] ( over→ start_ARG s end_ARG( i ) ). This shows ⁡(R)=G (R)=GΓ ( R ) = G, and thus G is time-separable. ∎ Corollary C.28. In a multi-armed bandit, every return function is time-separable. Proof. In a multi-armed bandit, states and sequences are equivalent, and so we can choose ϕ⁢(s)=sitalic-ϕφ(s)=sϕ ( s ) = s for every state/sequence s. The result follows from Proposition C.27. Alternatively, simply directly notice that in a multi-armed bandit, Γ is the identity mapping, and so for every return/reward function R, we have ⁡(R)=R (R)=RΓ ( R ) = R. ∎ C.8 Examples Supplementing Section 5 In this whole section, the inverse temperature parameter in the human choice probabilities is given by β=11β=1β = 1. We now consider four more mathematical examples of Corollary C.4 and Theorem C.9. In the first example, the ambiguity is so bad that the reward inference can become worse than simply maximizing JobssubscriptobsJ_obsJroman_obs as in naive RLHF. In Example C.30, there is simply “noise” in the observations and the human’s belief, the matrices BB and OO are injective, and identifiability works, as in Corollary C.14. In the third example, the matrix BB is not injective and identifiability fails, which is a minimal example showing the limits of our main theorems. In the fourth example, the matrix BB is not injective, but ker⁡∩im⁡=0kernelim0 B % =\0\ker B ∩ im Γ = 0 , and so identifiability works. This example is interesting in that the identifiability simply emerges through different distributions of delay that are caused by the different unobserved events. In this section, both the linear operators :ℝ→ℝΩ→:→superscriptℝ→superscriptℝ→Ω B: R S→ R % B : blackboard_Rover→ start_ARG S end_ARG → blackboard_Rover→ start_ARG Ω end_ARG and :ℝΩ→ℝ→:→superscriptℝ→Ωsuperscriptℝ→ O: R → R % SO : blackboard_Rover→ start_ARG Ω end_ARG → blackboard_Rover→ start_ARG S end_ARG are considered as matrices =(PO→⁢(o→∣s→))s→,o→∈ℝ→×Ω→,=(B⁢(s→∣o→))o→,s→∈ℝΩ→×→.formulae-sequencesubscriptsubscript→conditional→superscriptℝ→Ωsubscriptconditional→superscriptℝ→Ω→ O= (P_ O( o s) )_ % s, o∈ R S× , % B= (B( s o) )_ o, s% ∈ R × S.O = ( Pover→ start_ARG O end_ARG ( over→ start_ARG o end_ARG ∣ over→ start_ARG s end_ARG ) )over→ start_ARG s end_ARG , over→ start_ARG o end_ARG ∈ blackboard_Rover→ start_ARG S end_ARG × over→ start_ARG Ω end_ARG , B = ( B ( over→ start_ARG s end_ARG ∣ over→ start_ARG o end_ARG ) )over→ start_ARG o end_ARG , over→ start_ARG s end_ARG ∈ blackboard_Rover→ start_ARG Ω end_ARG × over→ start_ARG S end_ARG . Notice that both have a swap in their indices. Example C.29. Theorem 5.2 shows that the remaining ambiguity from the human’s choice probabilities is given by ker⁡∩im⁡kernelim B ker B ∩ im Γ, but it doesn’t explain how to proceed given this ambiguity. Without further inductive biases, some reward functions within the ambiguity of the true reward function can be even worse than simply maximizing JobssubscriptobsJ_obsJroman_obs. E.g., consider a multi-armed bandit with three actions a,b,ca,b,ca , b , c, observation-kernel o=O⁢(a)=O⁢(b)≠O⁢(c)=co=O(a)=O(b)≠ O(c)=co = O ( a ) = O ( b ) ≠ O ( c ) = c and reward function R⁢(a)=R⁢(b)<R⁢(c)R(a)=R(b)<R(c)R ( a ) = R ( b ) < R ( c ). If the human belief is given by B⁢(a∣o)=p=1−B⁢(b∣o)conditional1conditionalB(a o)=p=1-B(b o)B ( a ∣ o ) = p = 1 - B ( b ∣ o ), then R′=α⋅(p−1,p,0)∈ℝa,b,csuperscript′⋅10superscriptℝR =α·(p-1,p,0)∈ R^\a,b,c\R′ = α ⋅ ( p - 1 , p , 0 ) ∈ blackboard_R a , b , c is in the ambiguity for all α∈ℝα∈ Rα ∈ blackboard_R, and so R~≔R+R′≔~superscript′ R R+R over~ start_ARG R end_ARG ≔ R + R′ is compatible with the choice probabilities. However, for α≪0much-less-than0α 0α ≪ 0, we have R~⁢(a)>R~⁢(b)~~ R(a)> R(b)over~ start_ARG R end_ARG ( a ) > over~ start_ARG R end_ARG ( b ) and R~⁢(a)>R~⁢(c)~~ R(a)> R(c)over~ start_ARG R end_ARG ( a ) > over~ start_ARG R end_ARG ( c ), and so optimizing against this reward function leads to a suboptimal policy. In contrast, maximizing JobssubscriptobsJ_obsJroman_obs leads to the correct policy since a, b, and c all obtain their ground truth reward in this example. This generally raises the question of how to tie-break reward functions in the ambiguity, or how to act conservatively given the uncertainty, in order to consistently improve upon the setting in Section 4.1. Example C.30. This example is a special case of Corollary C.14. Consider a multi-armed bandit with two actions (which are automatically also states and sequences) a and b. In this case, the reward function and return function is the same. We assume there to be two possible observations o(a),o(b)superscriptsuperscripto^(a),o^(b)o( a ) , o( b ) and the observation kernel to be non-deterministic, with probabilities PO⁢(o(j)∣i)=2/3, if ⁢i=j,1/3, else.subscriptconditionalsuperscriptcases23 if otherwise13 elseotherwiseP_O(o^(j) i)= cases2/3, if i=j,\\ 1/3, else. casesPitalic_O ( o( j ) ∣ i ) = start_ROW start_CELL 2 / 3 , if i = j , end_CELL start_CELL end_CELL end_ROW start_ROW start_CELL 1 / 3 , else . end_CELL start_CELL end_CELL end_ROW If we assume the human forms Bayesian posterior beliefs as in Section C.1 and to have a policy prior B⁢(π′)superscript′B(π )B ( π′ ) such that B⁢(a)=∫π⁢(a)⁢B⁢(π′)⁢π=1/2subscriptsuperscript′differential-d12B(a)= _π(a)B(π )dπ=1/2B ( a ) = ∫π π ( a ) B ( π′ ) d π = 1 / 2 and B⁢(b)=1/212B(b)=1/2B ( b ) = 1 / 2, then it is easy to show that the human’s belief is the “reversed” observation kernel: B⁢(j∣o(i))=PO⁢(o(i)∣j).conditionalsuperscriptsubscriptconditionalsuperscriptB(j o^(i))=P_O(o^(i) j).B ( j ∣ o( i ) ) = Pitalic_O ( o( i ) ∣ j ) . We obtain OO ==(2/31/31/32/3)=13⋅(2112)absentmatrix23131323⋅13matrix2112 = B= pmatrix2/3&1/3\\ 1/3&2/3 pmatrix= 13· pmatrix2&1\\ 1&2 pmatrix= B = ( start_ARG start_ROW start_CELL 2 / 3 end_CELL start_CELL 1 / 3 end_CELL end_ROW start_ROW start_CELL 1 / 3 end_CELL start_CELL 2 / 3 end_CELL end_ROW end_ARG ) = divide start_ARG 1 end_ARG start_ARG 3 end_ARG ⋅ ( start_ARG start_ROW start_CELL 2 end_CELL start_CELL 1 end_CELL end_ROW start_ROW start_CELL 1 end_CELL start_CELL 2 end_CELL end_ROW end_ARG ) These matrices are injective since they are invertible: −1=−1=(2−1−12).superscript1superscript1matrix2112 O^-1= B^-1= pmatrix2% &-1\\ -1&2 pmatrix.O- 1 = B- 1 = ( start_ARG start_ROW start_CELL 2 end_CELL start_CELL - 1 end_CELL end_ROW start_ROW start_CELL - 1 end_CELL start_CELL 2 end_CELL end_ROW end_ARG ) . More generally, even if the human does not form fully rational posterior beliefs, it is easy to imagine that the matrix BB can end up being invertible. Thus, Corollary C.4 guarantees that the reward function can be inferred up to an additive constant from the choice probabilities of observations, and Theorem C.9 shows that this even works when the learning system does not know what the human observed. In the rest of this example, we explicitly walk the reader through the process of how the reward function can be inferred, in the general case that the observations are not known. In the process, we essentially recreate the proof of the theorems for this special case. For this aim, we first want to compute the choice probabilities PR⁢(i≻j)superscriptsucceedsP^R (i j )Pitalic_R ( i ≻ j ) that the learning system has access to in the limit of infinite data. We assume that the reward function is given by R⁢(a)=−11R(a)=-1R ( a ) = - 1 and R⁢(b)=22R(b)=2R ( b ) = 2. We compute: ⁡(R)=13⋅(2112)⋅(−12)=(01).⋅13matrix2112matrix12matrix01 B(R)= 13· pmatrix2&1\\ 1&2 pmatrix· pmatrix-1\\ 2 pmatrix= pmatrix0\\ 1 pmatrix.B ( R ) = divide start_ARG 1 end_ARG start_ARG 3 end_ARG ⋅ ( start_ARG start_ROW start_CELL 2 end_CELL start_CELL 1 end_CELL end_ROW start_ROW start_CELL 1 end_CELL start_CELL 2 end_CELL end_ROW end_ARG ) ⋅ ( start_ARG start_ROW start_CELL - 1 end_CELL end_ROW start_ROW start_CELL 2 end_CELL end_ROW end_ARG ) = ( start_ARG start_ROW start_CELL 0 end_CELL end_ROW start_ROW start_CELL 1 end_CELL end_ROW end_ARG ) . In other words, we have s∼B⁢(s∣o(a))[R⁢(s)]=0subscriptsimilar-toconditionalsuperscript0 *E_s B(s o^(a))[R(s)]=0Eitalic_s ∼ B ( s ∣ o( a ) ) [ R ( s ) ] = 0 and s∼B⁢(s∣o(b))[R⁢(s)]=1subscriptsimilar-toconditionalsuperscript1 *E_s B(s o^(b))[R(s)]=1Eitalic_s ∼ B ( s ∣ o( b ) ) [ R ( s ) ] = 1. From this, we can compute the observation-based choice probabilities P~o(i)⁢o(j)=σ⁢(⁡(R)⁢(o(i))−⁡(R)⁢(o(j)))subscript~superscriptsuperscriptsuperscriptsuperscript P_o^(i)o^(j)=σ ( B(R)(o^(i% ))- B(R)(o^(j)) )over~ start_ARG P end_ARGo( i ) o( j ) = σ ( B ( R ) ( o( i ) ) - B ( R ) ( o( j ) ) ), see Equation (2), and obtain: P~o(a)⁢o(a)=P~o(b)⁢o(b)=12,P~o(a)⁢o(b)=11+e,P~o(b)⁢o(a)=e1+e.formulae-sequencesubscript~superscriptsuperscriptsubscript~superscriptsuperscript12formulae-sequencesubscript~superscriptsuperscript11subscript~superscriptsuperscript1 P_o^(a)o^(a)= P_o^(b)o^(b)= 12,% P_o^(a)o^(b)= 11+e, P_o^(b)o% ^(a)= e1+e.over~ start_ARG P end_ARGo( a ) o( a ) = over~ start_ARG P end_ARGo( b ) o( b ) = divide start_ARG 1 end_ARG start_ARG 2 end_ARG , over~ start_ARG P end_ARGo( a ) o( b ) = divide start_ARG 1 end_ARG start_ARG 1 + e end_ARG , over~ start_ARG P end_ARGo( b ) o( a ) = divide start_ARG e end_ARG start_ARG 1 + e end_ARG . We can now determine the final choice probabilities Pi⁢j≔PR⁢(i≻j)≔subscriptsuperscriptsucceedsP_ij P^R (i j )Pitalic_i j ≔ Pitalic_R ( i ≻ j ) again by a matrix-vector product, with the indices ordered lexicographically, see Equation (8). Here, ⊗tensor-product O OO ⊗ O is the Kronecker product of the matrix OO with itself: P=(⊗)⋅P~=19⋅(4221241221421224)⋅(1/21/(1+e)e/(1+e)1/2)=(1/21/3⋅(2+e)/(1+e)1/3⋅(1+2⁢e)/(1+e)1/2).⋅tensor-product~⋅19matrix4221241221421224matrix1211112matrix12⋅1321⋅1312112P=( O O)· P% = 19· pmatrix4&2&2&1\\ 2&4&1&2\\ 2&1&4&2\\ 1&2&2&4 pmatrix· pmatrix1/2\\ 1/(1+e)\\ e/(1+e)\\ 1/2 pmatrix= pmatrix1/2\\ 1/3·(2+e)/(1+e)\\ 1/3·(1+2e)/(1+e)\\ 1/2 pmatrix.P = ( O ⊗ O ) ⋅ over~ start_ARG P end_ARG = divide start_ARG 1 end_ARG start_ARG 9 end_ARG ⋅ ( start_ARG start_ROW start_CELL 4 end_CELL start_CELL 2 end_CELL start_CELL 2 end_CELL start_CELL 1 end_CELL end_ROW start_ROW start_CELL 2 end_CELL start_CELL 4 end_CELL start_CELL 1 end_CELL start_CELL 2 end_CELL end_ROW start_ROW start_CELL 2 end_CELL start_CELL 1 end_CELL start_CELL 4 end_CELL start_CELL 2 end_CELL end_ROW start_ROW start_CELL 1 end_CELL start_CELL 2 end_CELL start_CELL 2 end_CELL start_CELL 4 end_CELL end_ROW end_ARG ) ⋅ ( start_ARG start_ROW start_CELL 1 / 2 end_CELL end_ROW start_ROW start_CELL 1 / ( 1 + e ) end_CELL end_ROW start_ROW start_CELL e / ( 1 + e ) end_CELL end_ROW start_ROW start_CELL 1 / 2 end_CELL end_ROW end_ARG ) = ( start_ARG start_ROW start_CELL 1 / 2 end_CELL end_ROW start_ROW start_CELL 1 / 3 ⋅ ( 2 + e ) / ( 1 + e ) end_CELL end_ROW start_ROW start_CELL 1 / 3 ⋅ ( 1 + 2 e ) / ( 1 + e ) end_CELL end_ROW start_ROW start_CELL 1 / 2 end_CELL end_ROW end_ARG ) . For example, the second entry in P is Pa⁢b=PR⁢(a≻b)=2+e3⋅(1+e)subscriptsuperscriptsucceeds2⋅31P_ab=P^R (a b )= 2+e3·(1+e)Pitalic_a b = Pitalic_R ( a ≻ b ) = divide start_ARG 2 + e end_ARG start_ARG 3 ⋅ ( 1 + e ) end_ARG. This is the likelihood that, for ground-truth actions a,ba,ba , b, the human will prefer a after only receiving observations o(a)superscripto^(a)o( a ) or o(b)superscripto^(b)o( b ) according to OO and following a Boltzman-rational policy based on the belief of the real action, see Equation (8). Over time, the learning system will be able to estimate these probabilities based on repeated human choices, assuming all state-pairs are sampled infinitely often. The question of identifiability is whether the original reward function R can be inferred from that data, given that the learning system knows OO and BB. We assume that the learning system doesn’t a priori know R or any of the intermediate steps in the computation. First, P~~ Pover~ start_ARG P end_ARG can be inferred by inverting ⊗tensor-product O OO ⊗ O: P~=(⊗)−1⋅P=(4−2−21−241−2−214−21−2−24)⋅(1/21/3⋅(2+e)/(1+e)1/3⋅(1+2⁢e)/(1+e)1/2)=(1/21/(1+e)e/(1+e)1/2).~⋅superscripttensor-product1⋅matrix4221241221421224matrix12⋅1321⋅1312112matrix1211112 P=( O O)^-1% · P= pmatrix4&-2&-2&1\\ -2&4&1&-2\\ -2&1&4&-2\\ 1&-2&-2&4 pmatrix· pmatrix1/2\\ 1/3·(2+e)/(1+e)\\ 1/3·(1+2e)/(1+e)\\ 1/2 pmatrix= pmatrix1/2\\ 1/(1+e)\\ e/(1+e)\\ 1/2 pmatrix.over~ start_ARG P end_ARG = ( O ⊗ O )- 1 ⋅ P = ( start_ARG start_ROW start_CELL 4 end_CELL start_CELL - 2 end_CELL start_CELL - 2 end_CELL start_CELL 1 end_CELL end_ROW start_ROW start_CELL - 2 end_CELL start_CELL 4 end_CELL start_CELL 1 end_CELL start_CELL - 2 end_CELL end_ROW start_ROW start_CELL - 2 end_CELL start_CELL 1 end_CELL start_CELL 4 end_CELL start_CELL - 2 end_CELL end_ROW start_ROW start_CELL 1 end_CELL start_CELL - 2 end_CELL start_CELL - 2 end_CELL start_CELL 4 end_CELL end_ROW end_ARG ) ⋅ ( start_ARG start_ROW start_CELL 1 / 2 end_CELL end_ROW start_ROW start_CELL 1 / 3 ⋅ ( 2 + e ) / ( 1 + e ) end_CELL end_ROW start_ROW start_CELL 1 / 3 ⋅ ( 1 + 2 e ) / ( 1 + e ) end_CELL end_ROW start_ROW start_CELL 1 / 2 end_CELL end_ROW end_ARG ) = ( start_ARG start_ROW start_CELL 1 / 2 end_CELL end_ROW start_ROW start_CELL 1 / ( 1 + e ) end_CELL end_ROW start_ROW start_CELL e / ( 1 + e ) end_CELL end_ROW start_ROW start_CELL 1 / 2 end_CELL end_ROW end_ARG ) . The learning system wants to use this to infer ⁡(R~)~ B( R)B ( over~ start_ARG R end_ARG ) (for the later-to-be inferred reward function R~~ Rover~ start_ARG R end_ARG that may differ from the true reward function R) and uses the equation P~o(a)⁢o(b)=exp⁡(⁡(R~)⁢(o(a)))exp⁡(⁡(R~)⁢(o(a)))+exp⁡(⁡(R~)⁢(o(b))),subscript~superscriptsuperscript~superscript~superscript~superscript P_o^(a)o^(b)= ( B(% R)(o^(a)) ) ( B( R)(o^% (a)) )+ ( B( R)(o^(b)) ),over~ start_ARG P end_ARGo( a ) o( b ) = divide start_ARG exp ( B ( over~ start_ARG R end_ARG ) ( o( a ) ) ) end_ARG start_ARG exp ( B ( over~ start_ARG R end_ARG ) ( o( a ) ) ) + exp ( B ( over~ start_ARG R end_ARG ) ( o( b ) ) ) end_ARG , which can be rearranged to ⁡(R~)⁢(o(a))=log⁡P~o(a)⁢o(b)1−P~o(a)⁢o(b)+⁡(R~)⁢(o(b))=log⁡1/(1+e)e/(1+e)+⁡(R~)⁢(o(b))=⁡(R~)⁢(o(b))−1.~superscriptsubscript~superscriptsuperscript1subscript~superscriptsuperscript~superscript111~superscript~superscript1 B( R)(o^(a))= P_o^(a)% o^(b)1- P_o^(a)o^(b)+ B( R% )(o^(b))= 1/(1+e)e/(1+e)+ B( R)(o% ^(b))= B( R)(o^(b))-1.B ( over~ start_ARG R end_ARG ) ( o( a ) ) = log divide start_ARG over~ start_ARG P end_ARGo( a ) o( b ) end_ARG start_ARG 1 - over~ start_ARG P end_ARGo( a ) o( b ) end_ARG + B ( over~ start_ARG R end_ARG ) ( o( b ) ) = log divide start_ARG 1 / ( 1 + e ) end_ARG start_ARG e / ( 1 + e ) end_ARG + B ( over~ start_ARG R end_ARG ) ( o( b ) ) = B ( over~ start_ARG R end_ARG ) ( o( b ) ) - 1 . This relation is all which can be inferred about ⁡(R~)⁢(o(a))~superscript B( R)(o^(a))B ( over~ start_ARG R end_ARG ) ( o( a ) ) and ⁡(R~)⁢(o(b))~superscript B( R)(o^(b))B ( over~ start_ARG R end_ARG ) ( o( b ) ); the precise value cannot be determined and ⁡(R~)⁢(o(b))~superscript B( R)(o^(b))B ( over~ start_ARG R end_ARG ) ( o( b ) ) is a free parameter. One can check that for ⁡(R~)⁢(o(b))=1~superscript1 B( R)(o^(b))=1B ( over~ start_ARG R end_ARG ) ( o( b ) ) = 1 this coincides with the true value ⁡(R) B(R)B ( R ). Finally, one can invert BB to infer R~~ Rover~ start_ARG R end_ARG from this: R~~ Rover~ start_ARG R end_ARG =−1⋅⁡(R~)absent⋅superscript1~ = B^-1· B(% R)= B- 1 ⋅ B ( over~ start_ARG R end_ARG ) =(2−1−12)⋅(⁡(R~)⁢(o(b))−1⁡(R~)⁢(o(b)))absent⋅matrix2112matrix~superscript1~superscript = pmatrix2&-1\\ -1&2 pmatrix· pmatrix B( R)(o^(% b))-1\\ B( R)(o^(b)) pmatrix= ( start_ARG start_ROW start_CELL 2 end_CELL start_CELL - 1 end_CELL end_ROW start_ROW start_CELL - 1 end_CELL start_CELL 2 end_CELL end_ROW end_ARG ) ⋅ ( start_ARG start_ROW start_CELL B ( over~ start_ARG R end_ARG ) ( o( b ) ) - 1 end_CELL end_ROW start_ROW start_CELL B ( over~ start_ARG R end_ARG ) ( o( b ) ) end_CELL end_ROW end_ARG ) =(⁡(R~)⁢(o(b))−21+⁡(R~)⁢(o(b)))absentmatrix~superscript21~superscript = pmatrix B( R)(o^(b))-2\\ 1+ B( R)(o^(b)) pmatrix= ( start_ARG start_ROW start_CELL B ( over~ start_ARG R end_ARG ) ( o( b ) ) - 2 end_CELL end_ROW start_ROW start_CELL 1 + B ( over~ start_ARG R end_ARG ) ( o( b ) ) end_CELL end_ROW end_ARG ) =(−12)+(⁡(R~)⁢(o(b))−1⁡(R~)⁢(o(b))−1)absentmatrix12matrix~superscript1~superscript1 = pmatrix-1\\ 2 pmatrix+ pmatrix B( R)(o^(b))-1% \\ B( R)(o^(b))-1 pmatrix= ( start_ARG start_ROW start_CELL - 1 end_CELL end_ROW start_ROW start_CELL 2 end_CELL end_ROW end_ARG ) + ( start_ARG start_ROW start_CELL B ( over~ start_ARG R end_ARG ) ( o( b ) ) - 1 end_CELL end_ROW start_ROW start_CELL B ( over~ start_ARG R end_ARG ) ( o( b ) ) - 1 end_CELL end_ROW end_ARG ) =R+(⁡(R~)⁢(o(b))−1⁡(R~)⁢(o(b))−1).absentmatrix~superscript1~superscript1 =R+ pmatrix B( R)(o^(b))-1% \\ B( R)(o^(b))-1 pmatrix.= R + ( start_ARG start_ROW start_CELL B ( over~ start_ARG R end_ARG ) ( o( b ) ) - 1 end_CELL end_ROW start_ROW start_CELL B ( over~ start_ARG R end_ARG ) ( o( b ) ) - 1 end_CELL end_ROW end_ARG ) . Thus, the inferred and true reward functions differ maximally by a constant, as predicted in Theorem C.9. In the following example, we work out a case where the reward function is so ambiguous that any policy is optimal to some reward function consistent with the human feedback: Example C.31. Consider a multi-armed bandit with exactly three actions/states a,b,ca,b,ca , b , c. We assume a deterministic observation kernel with o≔O⁢(a)=O⁢(c)≠O⁢(b)=b≔o O(a)=O(c)≠ O(b)=bo ≔ O ( a ) = O ( c ) ≠ O ( b ) = b. Assume the human has some arbitrary beliefs B⁢(a∣o),B⁢(c∣o)=1−B⁢(a∣o)conditionalconditional1conditionalB(a o),B(c o)=1-B(a o)B ( a ∣ o ) , B ( c ∣ o ) = 1 - B ( a ∣ o ), and can identify b: B⁢(b∣b)=1conditional1B(b b)=1B ( b ∣ b ) = 1. Then if the human makes observation comparisons with a Boltzman-rational policy, as in Theorem C.2, the resulting reward function is so ambiguous that some reward functions consistent with the feedback place the highest value on action a, no matter the true reward function R. Thus, even if the true reward function R regards a as the worst action, a can result from the reward learning and subsequent policy optimization process. Proof. The matrix :ℝa,b,c→ℝo,b:→superscriptℝsuperscriptℝ B: R^\a,b,c\→ R^\o,b\B : blackboard_R a , b , c → blackboard_R o , b is given by =(B⁢(a∣o)0B⁢(c∣o)010).matrixconditional0conditional010 B= pmatrixB(a o)&0&B(c o)\\ 0&1&0 pmatrix.B = ( start_ARG start_ROW start_CELL B ( a ∣ o ) end_CELL start_CELL 0 end_CELL start_CELL B ( c ∣ o ) end_CELL end_ROW start_ROW start_CELL 0 end_CELL start_CELL 1 end_CELL start_CELL 0 end_CELL end_ROW end_ARG ) . Its kernel is given by reward functions R′ with R′⁢(b)=0superscript′0R (b)=0R′ ( b ) = 0 and R′⁢(c)=−B⁢(a∣o)B⁢(c∣o)⁢R′⁢(a)superscript′conditionalconditionalsuperscript′R (c)=- B(a o)B(c o)R (a)R′ ( c ) = - divide start_ARG B ( a ∣ o ) end_ARG start_ARG B ( c ∣ o ) end_ARG R′ ( a ), with R′⁢(a)superscript′R (a)R′ ( a ) a free parameter. Theorem C.2 shows that, up to an additive constant, the reward functions consistent with the feedback of observation comparisons are given by R~=R+R′~superscript′ R=R+R over~ start_ARG R end_ARG = R + R′ for any R′∈ker⁡superscript′kernelR ∈ BR′ ∈ ker B. Thus, whenever the free parameter R′⁢(a)superscript′R (a)R′ ( a ) satisfies R′⁢(a)>R⁢(b)−R⁢(a)superscript′R (a)>R(b)-R(a)R′ ( a ) > R ( b ) - R ( a ) and R′⁢(a)>B⁢(c∣o)⋅(R⁢(c)−R⁢(a))superscript′⋅conditionalR (a)>B(c o)· (R(c)-R(a) )R′ ( a ) > B ( c ∣ o ) ⋅ ( R ( c ) - R ( a ) ), we obtain R~⁢(a)>R~⁢(b)~~ R(a)> R(b)over~ start_ARG R end_ARG ( a ) > over~ start_ARG R end_ARG ( b ) and R~⁢(a)>R~⁢(c)~~ R(a)> R(c)over~ start_ARG R end_ARG ( a ) > over~ start_ARG R end_ARG ( c ), showing the claim. ∎ We now investigate another example where BB is not injective, and yet, identifiability works because ∘≠00 B ≠\0\B ∘ Γ ≠ 0 . We saw such cases already in Example D.6, but include this additional example since it shows a conceptually interesting case: two different states lead to the exact same observations, but can be disambiguated since they lead to different amounts of delay until a more informative observation is made again. Example C.32. In this example, we assume that the human knows the policy π that generates the state sequences (corresponding to a policy prior B⁢(π′)=δπ⁢(π′)superscript′subscriptsuperscript′B(π )= _π(π )B ( π′ ) = δitalic_π ( π′ ) concentrated on π), which together with knowledge of the transition dynamics of the environment determines the true state transition probabilities π⁢(s′∣s)=∑a∈⁢(s′∣s,a)⋅π⁢(a∣s)superscriptconditionalsuperscript′subscript⋅conditionalsuperscript′conditionalT^π(s s)= _a T(s^% s,a)·π(a s)Titalic_π ( s′ ∣ s ) = ∑a ∈ A T ( s′ ∣ s , a ) ⋅ π ( a ∣ s ). We consider an environment with three states s,s′,s′superscript′s,s ,s s , s′ , s′ ′ and the following transition dynamics πsuperscriptT^πTitalic_π, where p≠1/212p≠ 1/2p ≠ 1 / 2 is a probability: sss′s s′s′s s′ ′1/313 1/31 / 31/313 1/31 / 31/313 1/31 / 31−p1 1-p1 - p pp pp1−p1 1-p1 - p We assume that P0⁢(s)=1subscript01P_0(s)=1P0 ( s ) = 1. Furthermore, we assume deterministic observations and s=O⁢(s)≠O⁢(s′)=O⁢(s′)≕osuperscript′≕s=O(s)≠ O(s )=O(s ) os = O ( s ) ≠ O ( s′ ) = O ( s′ ′ ) ≕ o. Assume the time horizon T is 3333, i.e., there are timesteps 0,1,2,301230,1,2,30 , 1 , 2 , 3. Assume that the human forms the belief over the true state sequence by Bayesian posterior updates as in Section C.1. In this case, ker⁡≠0kernel0 B≠\0\ker B ≠ 0 by Proposition C.11. However, we will now show that ker⁡(∘)=0kernel0 ( B )=\0\ker ( B ∘ Γ ) = 0 . If the human makes Boltzmann-rational comparisons of observation sequences, then this implies the identifiability of the return function up to an additive constant by Corollary C.4.444We assume that the learning system knows what the human observes, which is valid since POsubscriptP_OPitalic_O is deterministic. Alternatively, one can argue with Proposition C.11 that OO is automatically injective, meaning one can apply Theorem C.9. Thus, let R′∈ker⁡(∘)superscript′kernelR ∈ ( B )R′ ∈ ker ( B ∘ Γ ), i.e., [⁡(⁡(R′))]⁢(o→)=0delimited-[]superscript′→0 [ B ( (R^% ) ) ]( o)=0[ B ( Γ ( R′ ) ) ] ( over→ start_ARG o end_ARG ) = 0 for every observation sequence o→ oover→ start_ARG o end_ARG. For o→=s⁢s⁢s⁢s→ o=ssssover→ start_ARG o end_ARG = s s s s being the observation sequence that only consists of state s, this implies R′⁢(s)=0superscript′0R (s)=0R′ ( s ) = 0. Consequently, for general observation sequences o→ oover→ start_ARG o end_ARG, we have: 0=[⁡(⁡(R′))]⁢(o→)=s→∼B⁢(s→∣o→)[∑t=03δs′⁢(st)⋅γt]⋅R′⁢(s′)+s→∼B⁢(s→∣o→)[∑t=03δs′⁢(st)⋅γt]⋅R′⁢(s′).0delimited-[]superscript′→⋅subscriptsimilar-to→conditional→superscriptsubscript03⋅subscriptsuperscript′subscriptsuperscriptsuperscript′⋅subscriptsimilar-to→conditional→superscriptsubscript03⋅subscriptsuperscript′subscriptsuperscriptsuperscript′0= [ B ( (R^% ) ) ]( o)= *E_ s B( % s o) [ _t=0^3 _s (s_t)·γ^t% ]· R (s )+ *E_ s B% ( s o) [ _t=0^3 _s (s_t)% ·γ^t ]· R (s ).0 = [ B ( Γ ( R′ ) ) ] ( over→ start_ARG o end_ARG ) = Eover→ start_ARG s end_ARG ∼ B ( over→ start_ARG s end_ARG ∣ over→ start_ARG o end_ARG ) [ ∑t = 03 δitalic_s′ ( sitalic_t ) ⋅ γitalic_t ] ⋅ R′ ( s′ ) + Eover→ start_ARG s end_ARG ∼ B ( over→ start_ARG s end_ARG ∣ over→ start_ARG o end_ARG ) [ ∑t = 03 δitalic_s′ ′ ( sitalic_t ) ⋅ γitalic_t ] ⋅ R′ ( s′ ′ ) . Now we specialize this equation to the two observation sequences o→(1)=s⁢o⁢s⁢ssuperscript→1 o^(1)=sossover→ start_ARG o end_ARG( 1 ) = s o s s and o→(2)=s⁢o⁢o⁢ssuperscript→2 o^(2)=soosover→ start_ARG o end_ARG( 2 ) = s o o s. We start by considering o→(1)superscript→1 o^(1)over→ start_ARG o end_ARG( 1 ). This is consistent with the two state sequences s→(1),(s′)=s⁢s′⁢s⁢ssuperscript→1superscript′ s^(1),(s )=s ssover→ start_ARG s end_ARG( 1 ) , ( s start_POSTSUPERSCRIPT ′ ) end_POSTSUPERSCRIPT = s s′ s s and s→(1),(s′)=s⁢s′⁢s⁢ssuperscript→1superscript′ s^(1),(s )=s ssover→ start_ARG s end_ARG( 1 ) , ( s start_POSTSUPERSCRIPT ′ ′ ) end_POSTSUPERSCRIPT = s s′ ′ s s. We have posterior probabilities B⁢(s→(1),(s′)∣o→(1))=1−p,B⁢(s→(1),(s′)∣o→(1))=p,formulae-sequenceconditionalsuperscript→1superscript′→11conditionalsuperscript→1superscript′→1B ( s^(1),(s ) o^(1) )=1-p, B (% s^(1),(s ) o^(1) )=p,B ( over→ start_ARG s end_ARG( 1 ) , ( s start_POSTSUPERSCRIPT ′ ) end_POSTSUPERSCRIPT ∣ over→ start_ARG o end_ARG( 1 ) ) = 1 - p , B ( over→ start_ARG s end_ARG( 1 ) , ( s start_POSTSUPERSCRIPT ′ ′ ) end_POSTSUPERSCRIPT ∣ over→ start_ARG o end_ARG( 1 ) ) = p , and therefore 0=[⁡(⁡(R′))]⁢(o→(1))=(1−p)⋅γ⋅R′⁢(s′)+p⋅γ⋅R′⁢(s′),0delimited-[]superscript′→1⋅1superscript′⋅superscript′0= [ B ( (R^% ) ) ]( o^(1))=(1-p)·γ· R (s^% )+p·γ· R (s ),0 = [ B ( Γ ( R′ ) ) ] ( over→ start_ARG o end_ARG( 1 ) ) = ( 1 - p ) ⋅ γ ⋅ R′ ( s′ ) + p ⋅ γ ⋅ R′ ( s′ ′ ) , and so R′⁢(s′)=p−1⋅R′⁢(s′).superscript′⋅1superscript′R (s )= pp-1· R (s ).R′ ( s′ ) = divide start_ARG p end_ARG start_ARG p - 1 end_ARG ⋅ R′ ( s′ ′ ) . (11) Similarly, o→(2)superscript→2 o^(2)over→ start_ARG o end_ARG( 2 ) is consistent with the sequences s→(2),(s′)=s⁢s′⁢s′⁢ssuperscript→2superscript′superscript′ s^(2),(s )=s s sover→ start_ARG s end_ARG( 2 ) , ( s start_POSTSUPERSCRIPT ′ ) end_POSTSUPERSCRIPT = s s′ s and s→(2),(s′)=s⁢s′⁢s′⁢ssuperscript→2superscript′superscript′ s^(2),(s )=s s sover→ start_ARG s end_ARG( 2 ) , ( s start_POSTSUPERSCRIPT ′ ′ ) end_POSTSUPERSCRIPT = s s′ ′ s′ ′ s. They have posterior probabilities B⁢(s→(2),(s′)∣o→(2))=12,B⁢(s→(2),(s′)∣o→(2))=12,formulae-sequenceconditionalsuperscript→2superscript′→212conditionalsuperscript→2superscript′→212B ( s^(2),(s ) o^(2) )= 12, B% ( s^(2),(s ) o^(2) )= 12,B ( over→ start_ARG s end_ARG( 2 ) , ( s start_POSTSUPERSCRIPT ′ ) end_POSTSUPERSCRIPT ∣ over→ start_ARG o end_ARG( 2 ) ) = divide start_ARG 1 end_ARG start_ARG 2 end_ARG , B ( over→ start_ARG s end_ARG( 2 ) , ( s start_POSTSUPERSCRIPT ′ ′ ) end_POSTSUPERSCRIPT ∣ over→ start_ARG o end_ARG( 2 ) ) = divide start_ARG 1 end_ARG start_ARG 2 end_ARG , leading to 0=12⋅(γ+γ2)⋅R′⁢(s′)+12⋅(γ+γ2)⋅R′⁢(s′).0⋅12superscript2superscript′⋅12superscript2superscript′0= 12·(γ+γ^2)· R (s )+ 12% ·(γ+γ^2)· R (s ).0 = divide start_ARG 1 end_ARG start_ARG 2 end_ARG ⋅ ( γ + γ2 ) ⋅ R′ ( s′ ) + divide start_ARG 1 end_ARG start_ARG 2 end_ARG ⋅ ( γ + γ2 ) ⋅ R′ ( s′ ′ ) . Together with Equation (11), we obtain R′⁢(s′)=−R′⁢(s′)=p1−p⋅R′⁢(s′),superscript′superscript′⋅1superscript′R (s )=-R (s )= p1-p· R^% (s ),R′ ( s′ ′ ) = - R′ ( s′ ) = divide start_ARG p end_ARG start_ARG 1 - p end_ARG ⋅ R′ ( s′ ′ ) , which implies R′⁢(s′)=0superscript′0R (s )=0R′ ( s′ ′ ) = 0 because p≠1212p≠ 12p ≠ divide start_ARG 1 end_ARG start_ARG 2 end_ARG, and thus also R′⁢(s′)=0superscript′0R (s )=0R′ ( s′ ) = 0. Overall, we have showed R′=0superscript′0R =0R′ = 0, and so ∘ B B ∘ Γ is injective. This means that reward functions are identifiable in this example up to an additive constant, see Corollary C.4. Appendix D Issues of Naively Applying RLHF under Partial Observability In this section, we study the naive application of RLHF under partial observability. Thus, most of it takes a step back from the general theory of appropriately modeled partial observability in RLHF. Later, we will analyze examples where we also apply the general theory, which is why this appendix section comes second. In Section D.1, we first briefly explain what happens when the learning system incorrectly assumes that the human observes the full environment state. We show that as a consequence, the system is incentivized to infer what we call the observation return function GobssubscriptobsG_obsGroman_obs, which evaluates a state sequence based on the human’s belief of the state sequence given the human’s observations. In the policy optimization process, the policy is then selected to maximize JobssubscriptobsJ_obsJroman_obs, an expectation over GobssubscriptobsG_obsGroman_obs. In the interlude in Section D.2, we then briefly analyze the unrealistic case that the human, when evaluating a policy π, fully knows the complete specification of that policy and all of the environment and engages in rational Bayesian reasoning; in this case, Jobs=JsubscriptobsJ_obs=JJroman_obs = J is the true policy evaluation function. Realistically, however, maximizing JobssubscriptobsJ_obsJroman_obs can lead to failure modes. In Section D.3 we prove that a suboptimal policy that is optimal according to JobssubscriptobsJ_obsJroman_obs causes deceptive inflation, overjustification, or both. In Section B.3, we expand on the analysis of the main examples in the main paper. Finally, in Section D.4, we study further concrete examples where maximizing JobssubscriptobsJ_obsJroman_obs reveals deceptive and overjustifying behavior by the resulting policy. D.1 Optimal Policies under RLHF with Deterministic Partial Observations Maximize JobssubscriptobsJ_obsJroman_obs Assume that PO→subscript→P_ OPover→ start_ARG O end_ARG is deterministic and that the human makes Boltzmann-rational sequence comparisons between observation sequences. The true choice probabilities are then given by (See Equations (2) and (8)): PR⁢(s→≻s→′)=σ⁢(β⋅((⋅G)⁢(O→⁢(s→))−(⋅G)⁢(O→⁢(s→′))))superscriptsucceeds→superscript→′⋅→⋅→superscript→′P^R ( s s 1.0pt )=σ (% β· ( ( B· G ) ( O(% s) )- ( B· G ) ( O(% s 1.0pt ) ) ) )Pitalic_R ( over→ start_ARG s end_ARG ≻ over→ start_ARG s end_ARG′ ) = σ ( β ⋅ ( ( B ⋅ G ) ( over→ start_ARG O end_ARG ( over→ start_ARG s end_ARG ) ) - ( B ⋅ G ) ( over→ start_ARG O end_ARG ( over→ start_ARG s end_ARG′ ) ) ) ) (12) Now, assume that the learning system does not model the situation correctly. In particular, we assume: • The system is not aware that the human only observes observation sequences O→⁢(s→)→ O( s)over→ start_ARG O end_ARG ( over→ start_ARG s end_ARG ) instead of the full state sequences. • The system does not model that the human’s return function is time-separable, i.e., comes from a reward function R over environment states. The learning system then thinks that there is a return function G~∈ℝS→~superscriptℝ→ G∈ R Sover~ start_ARG G end_ARG ∈ blackboard_Rover→ start_ARG S end_ARG such that the choice probabilities are given by the following faulty formula: PR⁢(s→≻s→′)≔σ⁢(β⁢(G⁢(s→)−G⁢(s→′)))≔superscriptsucceeds→superscript→′→superscript→′P^R ( s s 1.0pt ) σ % (β (G( s)-G( s 1.0pt ) ) )Pitalic_R ( over→ start_ARG s end_ARG ≻ over→ start_ARG s end_ARG′ ) ≔ σ ( β ( G ( over→ start_ARG s end_ARG ) - G ( over→ start_ARG s end_ARG′ ) ) ) Now, assume that the learning system has access to the choice probabilities and wants to infer G. Inverting the sigmoid function and then plugging in the true choice probabilities from Equation (12), we obtain: G~⁢(s→)~→ G( s)over~ start_ARG G end_ARG ( over→ start_ARG s end_ARG ) =1β⁢log⁡PR⁢(s→≻s→′)PR⁢(s→′≻s→)+G~⁢(s→′)absent1superscriptsucceeds→superscript→′succeedssuperscript→′→~superscript→′ = 1β P^R( s s 1.0pt^% )P^R( s 1.0pt s)+ G( s% 1.0pt )= divide start_ARG 1 end_ARG start_ARG β end_ARG log divide start_ARG Pitalic_R ( over→ start_ARG s end_ARG ≻ over→ start_ARG s end_ARG′ ) end_ARG start_ARG Pitalic_R ( over→ start_ARG s end_ARG′ ≻ over→ start_ARG s end_ARG ) end_ARG + over~ start_ARG G end_ARG ( over→ start_ARG s end_ARG′ ) =1β⁢[β⋅((⋅G)⁢(O→⁢(s→))−(⋅G)⁢(O→⁢(s→′)))]+G~⁢(s→′)absent1delimited-[]⋅→⋅→superscript→′~superscript→′ = 1β [β· ( (% B· G ) ( O( s) )- (% B· G ) ( O( s 1.0pt ) )% ) ]+ G( s 1.0pt )= divide start_ARG 1 end_ARG start_ARG β end_ARG [ β ⋅ ( ( B ⋅ G ) ( over→ start_ARG O end_ARG ( over→ start_ARG s end_ARG ) ) - ( B ⋅ G ) ( over→ start_ARG O end_ARG ( over→ start_ARG s end_ARG′ ) ) ) ] + over~ start_ARG G end_ARG ( over→ start_ARG s end_ARG′ ) =(⋅G)⁢(O→⁢(s→))+C⁢(s→′).absent⋅→superscript→′ = ( B· G ) ( O(% s) )+C( s 1.0pt ).= ( B ⋅ G ) ( over→ start_ARG O end_ARG ( over→ start_ARG s end_ARG ) ) + C ( over→ start_ARG s end_ARG′ ) . Here, C⁢(s→′)superscript→′C( s 1.0pt )C ( over→ start_ARG s end_ARG′ ) is some quantity that does not depend on s→ sover→ start_ARG s end_ARG. Now, fix s→′ s 1.0pt over→ start_ARG s end_ARG′ as a reference sequence. Then for varying s→ sover→ start_ARG s end_ARG, C⁢(s→′)superscript→′C( s 1.0pt )C ( over→ start_ARG s end_ARG′ ) is simply an additive constant. Consequently, up to an additive constant, this determines the return function that the learning system is incentivized to infer. We call it the observation return function since it is the return function based on the human’s observations: Gobs⁢(s→)≔(⋅G)⁢(O→⁢(s→)).≔subscriptobs→⋅→G_obs( s) ( B· G% ) ( O( s) ).Groman_obs ( over→ start_ARG s end_ARG ) ≔ ( B ⋅ G ) ( over→ start_ARG O end_ARG ( over→ start_ARG s end_ARG ) ) . This return function is not necessarily time-separable, but we assume that time-separability is not modeled correctly by the learning system. Now, define the resulting policy evaluation function JobssubscriptobsJ_obsJroman_obs by Jobs⁢(π)≔s→∼Pπ⁢(s→)[Gobs⁢(s→)].≔subscriptobssubscriptsimilar-to→superscript→subscriptobs→J_obs(π) *E_ s P^% π( s) [G_obs( s) ].Jroman_obs ( π ) ≔ Eover→ start_ARG s end_ARG ∼ Pitalic_π ( over→ start_ARG s end_ARG ) [ Groman_obs ( over→ start_ARG s end_ARG ) ] . This is the policy evaluation function that would be optimized if the learning system erroneously inferred the return function GobssubscriptobsG_obsGroman_obs. D.2 Interlude: When the Human Knows the Policy and is a Bayesian Reasoner, then Jobs=JsubscriptobsJ_obs=JJroman_obs = J In this section, we briefly consider what would happen if in JobssubscriptobsJ_obsJroman_obs, the human’s belief B would make use of the true policy and be a rational Bayesian posterior as in Section C.1. We will show that under these conditions, we have Jobs=JsubscriptobsJ_obs=JJroman_obs = J. Since these are unrealistic assumptions, no other section depends on this result. For the analysis, we drop the assumption that the observation sequence kernel PO→subscript→P_ OPover→ start_ARG O end_ARG is deterministic, and assume that JobssubscriptobsJ_obsJroman_obs is given as follows: Jobs⁢(π)≔s→∼Pπ⁢(s→)[o→∼PO→⁢(o→∣s→)[s→′∼Bπ⁢(s→′∣o→)[G⁢(s→′)]]].≔subscriptobssubscriptsimilar-to→superscript→subscriptsimilar-to→subscript→conditional→subscriptsimilar-tosuperscript→′conditionalsuperscript→′→superscript→′J_obs(π) *E_ s P^% π( s) [ *E_ o P_ O(% o s) [ *E_ s 1.0pt^% B^π( s 1.0pt o) [G( s% 1.0pt ) ] ] ].Jroman_obs ( π ) ≔ Eover→ start_ARG s end_ARG ∼ Pitalic_π ( over→ start_ARG s end_ARG ) [ Eover→ start_ARG o end_ARG ∼ P start_POSTSUBSCRIPT over→ start_ARG O end_ARG ( over→ start_ARG o end_ARG ∣ over→ start_ARG s end_ARG ) end_POSTSUBSCRIPT [ Eover→ start_ARG s end_ARG′ ∼ Bitalic_π ( over→ start_ARG s end_ARG′ ∣ over→ start_ARG o end_ARG ) [ G ( over→ start_ARG s end_ARG′ ) ] ] ] . (13) In this formula, Bπ⁢(s→∣o→)≔B⁢(s→∣o→,π)≔superscriptconditional→conditional→B^π( s o) B( s o,π)Bitalic_π ( over→ start_ARG s end_ARG ∣ over→ start_ARG o end_ARG ) ≔ B ( over→ start_ARG s end_ARG ∣ over→ start_ARG o end_ARG , π ) with B being the joint distribution from Section C.1. Formally, this is the posterior of the joint distribution B⁢(s→,o→∣π)→conditional→B( s, o π)B ( over→ start_ARG s end_ARG , over→ start_ARG o end_ARG ∣ π ) that is given by the following hidden Markov model: s0subscript0s_0s0s1subscript1s_1s1s2subscript2s_2s2s3subscript3s_3s3…o0subscript0o_0o0o1subscript1o_1o1o2subscript2o_2o2o3subscript3o_3o3…POsubscript P_OPitalic_Oπsuperscript T^πTitalic_πPOsubscript P_OPitalic_Oπsuperscript T^πTitalic_πPOsubscript P_OPitalic_Oπsuperscript T^πTitalic_πPOsubscript P_OPitalic_Oπsuperscript T^πTitalic_π (14) Here, π⁢(s′∣s)≔∑a∈⁢(s′∣s,a)⋅π⁢(a∣s)≔superscriptconditionalsuperscript′subscript⋅conditionalsuperscript′conditionalT^π(s s) _a T(% s s,a)·π(a s)Titalic_π ( s′ ∣ s ) ≔ ∑a ∈ A T ( s′ ∣ s , a ) ⋅ π ( a ∣ s ). s0subscript0s_0s0 is sampled according to the known initial distribution P0⁢(s0)subscript0subscript0P_0(s_0)P0 ( s0 ). The human’s posterior Bπ⁢(s→′∣o→)superscriptconditionalsuperscript→′→B^π( s 1.0pt o)Bitalic_π ( over→ start_ARG s end_ARG′ ∣ over→ start_ARG o end_ARG ) is then the true posterior in this HMM. We obtain: Proposition D.1. Let π be a policy that is known to the human. Then Jobs⁢(π)=J⁢(π)subscriptobsJ_obs(π)=J(π)Jroman_obs ( π ) = J ( π ). Proof. By Equation (13), we have Jobs⁢(π)subscriptobs J_obs(π)Jroman_obs ( π ) =s→∼Pπ⁢(s→)[o→∼PO→⁢(o→∣s→)[s→′∼Bπ⁢(s→′∣o→)[G⁢(s→′)]]]absentsubscriptsimilar-to→superscript→subscriptsimilar-to→subscript→conditional→subscriptsimilar-tosuperscript→′conditionalsuperscript→′→superscript→′ = *E_ s P^π( s) [% *E_ o P_ O( o s) % [ *E_ s 1.0pt B^π( s% 1.0pt o) [G( s 1.0pt ) % ] ] ]= Eover→ start_ARG s end_ARG ∼ Pitalic_π ( over→ start_ARG s end_ARG ) [ Eover→ start_ARG o end_ARG ∼ P start_POSTSUBSCRIPT over→ start_ARG O end_ARG ( over→ start_ARG o end_ARG ∣ over→ start_ARG s end_ARG ) end_POSTSUBSCRIPT [ Eover→ start_ARG s end_ARG′ ∼ Bitalic_π ( over→ start_ARG s end_ARG′ ∣ over→ start_ARG o end_ARG ) [ G ( over→ start_ARG s end_ARG′ ) ] ] ] =(1)⁢∑s→Pπ⁢(s→)⁢∑o→PO→⁢(o→∣s→)⁢∑s→′Bπ⁢(s→′∣o→)⁢G⁢(s→′)1subscript→superscript→subscript→subscript→conditional→subscriptsuperscript→′conditionalsuperscript→′→superscript→′ (1)= _ sP^π( s) _ oP_% O( o s) _ s 1.0pt B^π( s% 1.0pt o)G( s 1.0pt )start_OVERACCENT ( 1 ) end_OVERACCENT start_ARG = end_ARG ∑over→ start_ARG s end_ARG Pitalic_π ( over→ start_ARG s end_ARG ) ∑over→ start_ARG o end_ARG Pover→ start_ARG O end_ARG ( over→ start_ARG o end_ARG ∣ over→ start_ARG s end_ARG ) ∑over→ start_ARG s end_ARG′ Bitalic_π ( over→ start_ARG s end_ARG′ ∣ over→ start_ARG o end_ARG ) G ( over→ start_ARG s end_ARG′ ) =(2)⁢∑s→′[∑o→Bπ⁢(s→′∣o→)⁢[∑s→PO→⁢(o→∣s→)⁢Pπ⁢(s→)]]⁢G⁢(s→′)2subscriptsuperscript→′delimited-[]subscript→superscriptconditionalsuperscript→′→delimited-[]subscript→subscript→conditional→superscript→superscript→′ (2)= _ s 1.0pt [ _% oB^π( s 1.0pt o) [ _ s% P_ O( o s)P^π( s) ] ]G( s 1% .0pt )start_OVERACCENT ( 2 ) end_OVERACCENT start_ARG = end_ARG ∑over→ start_ARG s end_ARG′ [ ∑over→ start_ARG o end_ARG Bitalic_π ( over→ start_ARG s end_ARG′ ∣ over→ start_ARG o end_ARG ) [ ∑over→ start_ARG s end_ARG Pover→ start_ARG O end_ARG ( over→ start_ARG o end_ARG ∣ over→ start_ARG s end_ARG ) Pitalic_π ( over→ start_ARG s end_ARG ) ] ] G ( over→ start_ARG s end_ARG′ ) =(3)⁢∑s→′[∑o→Bπ⁢(s→′∣o→)⁢Bπ⁢(o→)]⁢G⁢(s→′)3subscriptsuperscript→′delimited-[]subscript→superscriptconditionalsuperscript→′→superscript→superscript→′ (3)= _ s 1.0pt [ _% oB^π( s 1.0pt o)B^π( o) % ]G( s 1.0pt )start_OVERACCENT ( 3 ) end_OVERACCENT start_ARG = end_ARG ∑over→ start_ARG s end_ARG′ [ ∑over→ start_ARG o end_ARG Bitalic_π ( over→ start_ARG s end_ARG′ ∣ over→ start_ARG o end_ARG ) Bitalic_π ( over→ start_ARG o end_ARG ) ] G ( over→ start_ARG s end_ARG′ ) =(4)⁢∑s→′[∑o→Pπ⁢(s→′)⁢PO→⁢(o→∣s→′)]⁢G⁢(s→′)4subscriptsuperscript→′delimited-[]subscript→superscriptsuperscript→′subscript→conditional→superscript→′ (4)= _ s 1.0pt [ _% oP^π( s 1.0pt )P_ O( o s% 1.0pt ) ]G( s 1.0pt )start_OVERACCENT ( 4 ) end_OVERACCENT start_ARG = end_ARG ∑over→ start_ARG s end_ARG′ [ ∑over→ start_ARG o end_ARG Pitalic_π ( over→ start_ARG s end_ARG′ ) Pover→ start_ARG O end_ARG ( over→ start_ARG o end_ARG ∣ over→ start_ARG s end_ARG′ ) ] G ( over→ start_ARG s end_ARG′ ) =(5)⁢∑s→′Pπ⁢(s→′)⁢G⁢(s→′)5subscriptsuperscript→′superscript→′ (5)= _ s 1.0pt P^π( s% 1.0pt )G( s 1.0pt )start_OVERACCENT ( 5 ) end_OVERACCENT start_ARG = end_ARG ∑over→ start_ARG s end_ARG′ Pitalic_π ( over→ start_ARG s end_ARG′ ) G ( over→ start_ARG s end_ARG′ ) =(6)⁢∑s→Pπ⁢(s→)⁢G⁢(s→)6subscript→superscript→ (6)= _ sP^π( s)G( s)start_OVERACCENT ( 6 ) end_OVERACCENT start_ARG = end_ARG ∑over→ start_ARG s end_ARG Pitalic_π ( over→ start_ARG s end_ARG ) G ( over→ start_ARG s end_ARG ) =(7)⁢J⁢(π).7 (7)=J(π).start_OVERACCENT ( 7 ) end_OVERACCENT start_ARG = end_ARG J ( π ) . In step (1), we wrote the expectations out in terms of sums. In step (2), we reordered them. In step (3), we observed that the inner sum over s→ sover→ start_ARG s end_ARG evaluates to the marginal distribution Bπ⁢(o→)superscript→B^π( o)Bitalic_π ( over→ start_ARG o end_ARG ) of the observation sequence o→ oover→ start_ARG o end_ARG in the HMM in Equation (13). In step (4), we used Bayes rule in the inner sum. This is possible since Bπ⁢(s→′∣o→)superscriptconditionalsuperscript→′→B^π( s 1.0pt o)Bitalic_π ( over→ start_ARG s end_ARG′ ∣ over→ start_ARG o end_ARG ) is the true posterior when π is known. In step (5), we pull Pπ⁢(s→′)superscriptsuperscript→′P^π( s 1.0pt )Pitalic_π ( over→ start_ARG s end_ARG′ ) out and notice that the remaining inner sum evaluates to 1111. Step (6) is a relabeling and step (7) the definition of the true policy evaluation function J. ∎ D.3 Proof of Theorem 4.5 We first prove the following lemma. Lemma D.2. Let π and πrefsubscriptref _refπref be two policies. If J⁢(π)<J⁢(πref)subscriptrefJ(π)<J( _ref)J ( π ) < J ( πref ) and Jobs⁢(π)>Jobs⁢(πref)subscriptobssubscriptobssubscriptrefJ_obs(π)>J_obs( _ref)Jroman_obs ( π ) > Jroman_obs ( πref ), then relative to πrefsubscriptref _refπref, π must exhibit deceptive inflation, overjustification, or both. Proof. We start by establishing a quantitative relationship between the average overestimation and underestimation errors E¯+superscript¯ E^+over¯ start_ARG E end_ARG+ and E¯−superscript¯ E^-over¯ start_ARG E end_ARG- as defined in Definition 4.2, the true policy evaluation function J, and the observation evaluation function JobssubscriptobsJ_obsJroman_obs defined in Equation 4. Define Δ:→ℝ:Δ→ℝ : S→ RΔ : over→ start_ARG S end_ARG → blackboard_R by Δ⁢(s→)=Gobs⁢(s→)−G⁢(s→)Δ→subscriptobs→ ( s)=G_obs( s)-G( s)Δ ( over→ start_ARG s end_ARG ) = Groman_obs ( over→ start_ARG s end_ARG ) - G ( over→ start_ARG s end_ARG ), where GobssubscriptobsG_obsGroman_obs is as defined in Equation 3. Consider the quantity E+⁢(s→)−E−⁢(s→)=max⁡(0,Δ⁢(s→))−max⁡(0,−Δ⁢(s→)).superscript→superscript→0Δ→0Δ→ E^+( s)-E^-( s)= (0, ( s) % )- (0,- ( s) ).E+ ( over→ start_ARG s end_ARG ) - E- ( over→ start_ARG s end_ARG ) = max ( 0 , Δ ( over→ start_ARG s end_ARG ) ) - max ( 0 , - Δ ( over→ start_ARG s end_ARG ) ) . If Δ⁢(s→)>0Δ→0 ( s)>0Δ ( over→ start_ARG s end_ARG ) > 0, then the first term is Δ⁢(s→)Δ→ ( s)Δ ( over→ start_ARG s end_ARG ) and the second one is 0. If Δ⁢(s→)<0Δ→0 ( s)<0Δ ( over→ start_ARG s end_ARG ) < 0, then the first term is zero and the second one is Δ⁢(s→)Δ→ ( s)Δ ( over→ start_ARG s end_ARG ). If Δ⁢(s→)=0Δ→0 ( s)=0Δ ( over→ start_ARG s end_ARG ) = 0, then both terms are zero. In all cases the right-hand side is equal to Δ⁢(s→)Δ→ ( s)Δ ( over→ start_ARG s end_ARG ). Unpacking the definition of Δ Δ again, we have that for all s→ sover→ start_ARG s end_ARG, E+⁢(s→)−E−⁢(s→)=Gobs⁢(s→)−G⁢(s→).superscript→superscript→subscriptobs→ E^+( s)-E^-( s)=G_obs( s)-G(% s).E+ ( over→ start_ARG s end_ARG ) - E- ( over→ start_ARG s end_ARG ) = Groman_obs ( over→ start_ARG s end_ARG ) - G ( over→ start_ARG s end_ARG ) . (15) For any policy π, if we take the expectation of both sides of this equation over the on-policy distribution admitted by π, PπsuperscriptP^πPitalic_π, we get E¯+⁢(π)−E¯−⁢(π)=Jobs⁢(π)−J⁢(π).superscript¯superscript¯subscriptobs E^+(π)- E^-(π)=J_obs% (π)-J(π).over¯ start_ARG E end_ARG+ ( π ) - over¯ start_ARG E end_ARG- ( π ) = Jroman_obs ( π ) - J ( π ) . (16) We now prove the lemma. Let π and πrefsubscriptref _refπref be two policies, and assume that J⁢(π)<J⁢(πref)subscriptrefJ(π)<J( _ref)J ( π ) < J ( πref ) and Jobs⁢(π)≥Jobs⁢(πref)subscriptobssubscriptobssubscriptrefJ_obs(π)≥ J_obs( _ref)Jroman_obs ( π ) ≥ Jroman_obs ( πref ). Equivalently, we have Jobs⁢(π)−Jobs⁢(πref)≥0subscriptobssubscriptobssubscriptref0J_obs(π)-J_obs( _ref)≥ 0Jroman_obs ( π ) - Jroman_obs ( πref ) ≥ 0 and J⁢(πref)−J⁢(π)>0subscriptref0J( _ref)-J(π)>0J ( πref ) - J ( π ) > 0, which we combine to state (Jobs⁢(π)−Jobs⁢(πref))+(J⁢(πref)−J⁢(π))>0.subscriptobssubscriptobssubscriptrefsubscriptref0 (J_obs(π)-J_obs( _% ref) )+ (J( _ref)-J(π) )>0.( Jroman_obs ( π ) - Jroman_obs ( πref ) ) + ( J ( πref ) - J ( π ) ) > 0 . (17) Rearranging terms yields (Jobs⁢(π)−J⁢(π))−(Jobs⁢(πref)−J⁢(πref))>0.subscriptobssubscriptobssubscriptrefsubscriptref0 (J_obs(π)-J(π) )- (J_% obs( _ref)-J( _ref) )>0.( Jroman_obs ( π ) - J ( π ) ) - ( Jroman_obs ( πref ) - J ( πref ) ) > 0 . These two differences inside parentheses are equal to the right-hand side of (16) for π and πrefsubscriptref _refπref, respectively. We substitute the left-hand side of (16) twice to obtain (E¯+⁢(π)−E¯−⁢(π))−(E¯+⁢(πref)−E¯−⁢(πref))>0.superscript¯superscript¯superscript¯subscriptrefsuperscript¯subscriptref0 ( E^+(π)- E^-(π) )- % ( E^+( _ref)- E^-( _ref)% )>0.( over¯ start_ARG E end_ARG+ ( π ) - over¯ start_ARG E end_ARG- ( π ) ) - ( over¯ start_ARG E end_ARG+ ( πref ) - over¯ start_ARG E end_ARG- ( πref ) ) > 0 . Rearranging terms again yields (E¯+⁢(π)−E¯+⁢(πref))+(E¯−⁢(πref)−E¯−⁢(π))>0.superscript¯superscript¯subscriptrefsuperscript¯subscriptrefsuperscript¯0 ( E^+(π)- E^+( _ref% ) )+ ( E^-( _ref)- E^-(π)% )>0.( over¯ start_ARG E end_ARG+ ( π ) - over¯ start_ARG E end_ARG+ ( πref ) ) + ( over¯ start_ARG E end_ARG- ( πref ) - over¯ start_ARG E end_ARG- ( π ) ) > 0 . (18) If E¯+⁢(π)−E¯+⁢(πref)>0superscript¯superscript¯subscriptref0 E^+(π)- E^+( _ref)>0over¯ start_ARG E end_ARG+ ( π ) - over¯ start_ARG E end_ARG+ ( πref ) > 0 then we have E¯+⁢(π)>E¯+⁢(πref)superscript¯superscript¯subscriptref E^+(π)> E^+( _ref)over¯ start_ARG E end_ARG+ ( π ) > over¯ start_ARG E end_ARG+ ( πref ) and, by assumption, Jobs⁢(π)>Jobs⁢(πref)subscriptobssubscriptobssubscriptrefJ_obs(π)>J_obs( _ref)Jroman_obs ( π ) > Jroman_obs ( πref ). By Definition 4.3, this means π exhibits deceptive inflation relative to πrefsubscriptref _refπref. If E¯−⁢(πref)−E¯−⁢(π)>0superscript¯subscriptrefsuperscript¯0 E^-( _ref)- E^-(π)>0over¯ start_ARG E end_ARG- ( πref ) - over¯ start_ARG E end_ARG- ( π ) > 0 then we have E¯−⁢(π)<E¯−⁢(πref)superscript¯superscript¯subscriptref E^-(π)< E^-( _ref)over¯ start_ARG E end_ARG- ( π ) < over¯ start_ARG E end_ARG- ( πref ) and, by assumption, J⁢(π)<J⁢(πref)subscriptrefJ(π)<J( _ref)J ( π ) < J ( πref ). By Definition 4.4, this means π exhibits overjustification relative to πrefsubscriptref _refπref. At least one of the two differences in parentheses in (18) must be positive, otherwise their sum would not be positive. Thus π must exhibit deceptive inflation relative to πrefsubscriptref _refπref, overjustification relative to πrefsubscriptref _refπref, or both. ∎ We can now combine earlier results to prove Theorem 4.5, repeated here for convenience: Theorem D.3. Assume that POsubscriptP_OPitalic_O is deterministic. Let πobs∗subscriptsuperscriptobsπ^*_obsπ∗roman_obs be an optimal policy according to a naive application of RLHF under partial observability, and let π∗superscriptπ^*π∗ be an optimal policy according to the true objective J. If πobs∗subscriptsuperscriptobsπ^*_obsπ∗roman_obs is not J-optimal, then relative to π∗superscriptπ^*π∗, πobs∗subscriptsuperscriptobsπ^*_obsπ∗roman_obs must exhibit deceptive inflation, overjustification, or both. Proof. Because POsubscriptP_OPitalic_O is deterministic, πobs∗subscriptsuperscriptobsπ^*_obsπ∗roman_obs must be optimal with respect to JobssubscriptobsJ_obsJroman_obs by Proposition 4.1 (proved in Section D.1). Thus Jobs⁢(πobs∗)≥Jobs⁢(π∗)subscriptobssubscriptsuperscriptobssubscriptobssuperscriptJ_obs(π^*_obs)≥ J_obs% (π^*)Jroman_obs ( π∗roman_obs ) ≥ Jroman_obs ( π∗ ). Since π∗superscriptπ^*π∗ is J-optimal and πobs∗subscriptsuperscriptobsπ^*_obsπ∗roman_obs is not, J⁢(π∗)<J⁢(πobs∗)superscriptsubscriptsuperscriptobsJ(π^*)<J(π^*_obs)J ( π∗ ) < J ( π∗roman_obs ). By Lemma D.2, relative to π∗superscriptπ^*π∗, πobs∗subscriptsuperscriptobsπ^*_obsπ∗roman_obs must exhibit deceptive inflation, overjustification, or both. ∎ D.4 Further Examples Supplementing Section 4.4 In this section, we present further mathematical examples supplementing those in Section 4.4. We found many of them before finding the examples we discuss in the main paper, and show the same and additional conceptual features with somewhat less polish. We again assume that PO→subscript→P_ OPover→ start_ARG O end_ARG is deterministic. Example D.4. In the main paper, we have assumed a model where the human obeys Eq. (2) and showed that a naive application of RLHF can lead to suboptimal policies, and the specific failure modes of deceptive inflation and overjustification. What if the human makes the choices in a different way? Specifically, assume that all we know is that PR⁢(o→≻o→′)+PR⁢(o→′≻o→)=1superscriptsucceeds→superscript→′succeedssuperscript→′→1P^R( o o 1.0pt )+P^R( o 1.0pt^% o)=1Pitalic_R ( over→ start_ARG o end_ARG ≻ over→ start_ARG o end_ARG′ ) + Pitalic_R ( over→ start_ARG o end_ARG′ ≻ over→ start_ARG o end_ARG ) = 1. Can the human generally choose these choice probabilities in such a way that RLHF is incentivized to infer a reward function whose optimal policies are also optimal for R? The answer is no. Take the following example: sssaaabbbccc In this example, there is a fixed start state s and three actions a,b,ca,b,ca , b , c that also serve as the final states. The time horizon is T=11T=1T = 1, so the only state sequences are s⁢a,s⁢b,s⁢csa,sb,scs a , s b , s c. Assume ⁢(a∣s,a)=1conditional1T(a s,a)=1T ( a ∣ s , a ) = 1, ⁢(b∣s,b)=1conditional1T(b s,b)=1T ( b ∣ s , b ) = 1, ⁢(c∣s,c)=1−ϵconditional1italic-ϵT(c s,c)=1- ( c ∣ s , c ) = 1 - ϵ, ⁢(a∣s,c)=ϵconditionalitalic-ϵT(a s,c)= ( a ∣ s , c ) = ϵ, i.e., selecting action c sometimes leads to state a. Also, assume a=O⁢(a)≠O⁢(b)=O⁢(c)≕o≕a=O(a)≠ O(b)=O(c) oa = O ( a ) ≠ O ( b ) = O ( c ) ≕ o and R⁢(a)=R⁢(b)<R⁢(c)R(a)=R(b)<R(c)R ( a ) = R ( b ) < R ( c ). Since b and c have the same observation o, the human choice probabilities do not make a difference between them, and so RLHF is incentivized to infer a reward function R~~ Rover~ start_ARG R end_ARG with R~⁢(b)=R~⁢(c)≕R~⁢(o)~~≕~ R(b)= R(c) R(o)over~ start_ARG R end_ARG ( b ) = over~ start_ARG R end_ARG ( c ) ≕ over~ start_ARG R end_ARG ( o ). If R~⁢(o)>R~⁢(a)~~ R(o)> R(a)over~ start_ARG R end_ARG ( o ) > over~ start_ARG R end_ARG ( a ), then the policy optimal under R~~ Rover~ start_ARG R end_ARG will produce action b since this deterministically leads to observation o, whereas c does not. If R~⁢(o)<R~⁢(a)~~ R(o)< R(a)over~ start_ARG R end_ARG ( o ) < over~ start_ARG R end_ARG ( a ), then the policy optimal under R~~ Rover~ start_ARG R end_ARG will produce action a. In both cases, the resulting policy is suboptimal compared to π∗superscriptπ^*π∗, which deterministically chooses action c. In the coming examples, it will also be useful to look at the misleadingness of state sequences: Definition D.5 (Misleadingness). Let s→∈→ s∈ Sover→ start_ARG s end_ARG ∈ over→ start_ARG S end_ARG be a state sequence. Then its misleadingness is defined by M⁡(s→)≔Gobs⁢(s→)−G⁢(s→)=s→′∼B⁢(s→′∣O→⁢(s→))[G⁢(s→′)−G⁢(s)].≔M→subscriptobs→subscriptsimilar-tosuperscript→′conditionalsuperscript→′→superscript→′M( s) G_obs( s)-G( s)=% *E_ s 1.0pt B( s 1% .0pt O( s)) [G( s 1.0pt )-G(s)% ].M ( over→ start_ARG s end_ARG ) ≔ Groman_obs ( over→ start_ARG s end_ARG ) - G ( over→ start_ARG s end_ARG ) = Eover→ start_ARG s end_ARG′ ∼ B ( over→ start_ARG s end_ARG′ ∣ over→ start_ARG O end_ARG ( over→ start_ARG s end_ARG ) ) [ G ( over→ start_ARG s end_ARG′ ) - G ( s ) ] . We call a state sequence positively misleading if M⁢(s→)>0→0M( s)>0M ( over→ start_ARG s end_ARG ) > 0, which means the sequence appears better than it is, and negatively misleading if M⁡(s→)<0M→0M( s)<0M ( over→ start_ARG s end_ARG ) < 0. The misleadingness vector is given by M∈ℝ→Msuperscriptℝ→M∈ R SM ∈ blackboard_Rover→ start_ARG S end_ARG. Note that the misleadingness is related to E+superscriptE^+E+ and E−superscriptE^-E-, as defined in Definition 4.2: If M⁡(s→)>0M→0M( s)>0M ( over→ start_ARG s end_ARG ) > 0 then M⁡(s→)=E+⁢(s→)M→superscript→M( s)=E^+( s)M ( over→ start_ARG s end_ARG ) = E+ ( over→ start_ARG s end_ARG ), and if M⁡(s→)<0M→0M( s)<0M ( over→ start_ARG s end_ARG ) < 0 then M⁡(s→)=−E−⁢(s→)M→superscript→M( s)=-E^-( s)M ( over→ start_ARG s end_ARG ) = - E- ( over→ start_ARG s end_ARG ). Example D.6. In this example, we assume the human is a Bayesian reasoner as in Section C.1. Consider the MDP that is suggestively depicted as follows: aaabbbccc The MDP has states =a,b,cS=\a,b,c\S = a , b , c and actions =b,cA=\b,c\A = b , c . The transition kernel is given by ⁢(c∣a,c)=1conditional1T(c a,c)=1T ( c ∣ a , c ) = 1 and ⁢(b∣a,b)=1conditional1T(b a,b)=1T ( b ∣ a , b ) = 1, meaning that the action determines whether to transition from a to b or c. All other transitions are deterministic and do not depend on the action, as depicted. We assume an initial state distribution P0subscript0P_0P0 over states with probabilities pa=P0⁢(a),pb=P0⁢(b),pc=P0⁢(c)formulae-sequencesubscriptsubscript0formulae-sequencesubscriptsubscript0subscriptsubscript0p_a=P_0(a),p_b=P_0(b),p_c=P_0(c)pitalic_a = P0 ( a ) , pitalic_b = P0 ( b ) , pitalic_c = P0 ( c ). The true reward function R∈ℝa,b,csuperscriptℝR∈ R^\a,b,c\R ∈ blackboard_R a , b , c and discount factor γ∈[0,1)01γ∈[0,1)γ ∈ [ 0 , 1 ) are, for now, kept arbitrary. The time horizon is T=22T=2T = 2, meaning we have four possible state sequences a⁢c⁢cacca c c, a⁢b⁢cabca b c, b⁢c⁢cbccb c c, c⁢c⁢c c c. Furthermore, assume that o≔O⁢(a)=O⁢(b)≠O⁢(c)=c≔o O(a)=O(b)≠ O(c)=co ≔ O ( a ) = O ( b ) ≠ O ( c ) = c, i.e., c is observed and a and b are ambiguous. Finally, assume that the human has a policy prior B⁢(λ)B(λ)B ( λ ), where λ=πλ⁢(c∣a)subscriptconditionalλ= _λ(c a)λ = πitalic_λ ( c ∣ a ) is the likelihood that the policy chooses action c when in state a, which is a parameter that determines the entire policy. We claim the following: 1. If pb≠γ⋅λ∼B⁢(λ)[λ]⋅pasubscript⋅subscriptsimilar-tosubscriptp_b≠γ· *E_λ B(λ)[% λ]· p_apitalic_b ≠ γ ⋅ Eitalic_λ ∼ B ( λ ) [ λ ] ⋅ pitalic_a, then ker⁡∩im⁡=0kernelim0 B % =\0\ker B ∩ im Γ = 0 , so there is no return function ambiguity under appropriately modeled partially observable RLHF, see Corollary C.4. 2. There are true reward functions R for which optimizing JobssubscriptobsJ_obsJroman_obs leads to a suboptimal policy according to the true policy evaluation function J, a case of misalignment. Thus, a naive application of RLHF under partial observability fails, see Section 4.1. 3. The failure modes are related to hiding negative information (deception) and purposefully revealing information while incuring a loss (overjustifying behavior). Proof. Write p≔B⁢(b⁢c⁢c∣o⁢c⁢c)≔conditionalp B(bcc occ)p ≔ B ( b c c ∣ o c c ), the human’s posterior probability of state sequence b⁢c⁢cbccb c c for observation sequence o⁢c⁢cocco c c. We have 1−p=B⁢(a⁢c⁢c∣o⁢c⁢c)1conditional1-p=B(acc occ)1 - p = B ( a c c ∣ o c c ). Consider the linear operators :ℝa,b,c→ℝa⁢b⁢c,b⁢c⁢c,c⁢c⁢c,a⁢c⁢c:→superscriptℝsuperscriptℝ : R^\a,b,c\→ R^\abc,bcc,% c,acc\Γ : blackboard_R a , b , c → blackboard_R a b c , b c c , c c c , a c c and :ℝa⁢b⁢c,b⁢c⁢c,c⁢c⁢c,a⁢c⁢c→ℝo⁢o⁢c,o⁢c⁢c,c⁢c⁢c:→superscriptℝsuperscriptℝ B: R^\abc,bcc,c,acc\→ R^\ooc% ,occ,c\B : blackboard_R a b c , b c c , c c c , a c c → blackboard_R o o c , o c c , c c c defined in the main paper. When ordering the states, state sequences, and observation sequences as we just wrote down, we obtain =(1γ201γ+γ2001+γ+γ210γ+γ2),=(10000p01−p0010),∘=(1γ21−pγ+γ2001+γ+γ2).formulae-sequencematrix1superscript201superscript2001superscript210superscript2formulae-sequencematrix10000010010matrix1superscript21superscript2001superscript2 = pmatrix1&γ&γ^2\\ 0&1&γ+γ^2\\ 0&0&1+γ+γ^2\\ 1&0&γ+γ^2 pmatrix, B= % pmatrix1&0&0&0\\ 0&p&0&1-p\\ 0&0&1&0 pmatrix, B % = pmatrix1&γ&γ^2\\ 1-p&p&γ+γ^2\\ 0&0&1+γ+γ^2 pmatrix.Γ = ( start_ARG start_ROW start_CELL 1 end_CELL start_CELL γ end_CELL start_CELL γ2 end_CELL end_ROW start_ROW start_CELL 0 end_CELL start_CELL 1 end_CELL start_CELL γ + γ2 end_CELL end_ROW start_ROW start_CELL 0 end_CELL start_CELL 0 end_CELL start_CELL 1 + γ + γ2 end_CELL end_ROW start_ROW start_CELL 1 end_CELL start_CELL 0 end_CELL start_CELL γ + γ2 end_CELL end_ROW end_ARG ) , B = ( start_ARG start_ROW start_CELL 1 end_CELL start_CELL 0 end_CELL start_CELL 0 end_CELL start_CELL 0 end_CELL end_ROW start_ROW start_CELL 0 end_CELL start_CELL p end_CELL start_CELL 0 end_CELL start_CELL 1 - p end_CELL end_ROW start_ROW start_CELL 0 end_CELL start_CELL 0 end_CELL start_CELL 1 end_CELL start_CELL 0 end_CELL end_ROW end_ARG ) , B ∘ Γ = ( start_ARG start_ROW start_CELL 1 end_CELL start_CELL γ end_CELL start_CELL γ2 end_CELL end_ROW start_ROW start_CELL 1 - p end_CELL start_CELL p end_CELL start_CELL γ + γ2 end_CELL end_ROW start_ROW start_CELL 0 end_CELL start_CELL 0 end_CELL start_CELL 1 + γ + γ2 end_CELL end_ROW end_ARG ) . By Corollary C.4, if ∘ B B ∘ Γ is injective, then there is no reward function ambiguity. Clearly, this is the case if and only if p≠γ⋅(1−p)⋅1p≠γ·(1-p)p ≠ γ ⋅ ( 1 - p ). From Bayes rule, we have p=B⁢(b⁢c⁢c)B⁢(a⁢c⁢c)+B⁢(b⁢c⁢c),1−p=B⁢(a⁢c⁢c)B⁢(a⁢c⁢c)+B⁢(b⁢c⁢c).formulae-sequence1p= B(bcc)B(acc)+B(bcc), 1-p= B(acc)B(acc)+B(bcc).p = divide start_ARG B ( b c c ) end_ARG start_ARG B ( a c c ) + B ( b c c ) end_ARG , 1 - p = divide start_ARG B ( a c c ) end_ARG start_ARG B ( a c c ) + B ( b c c ) end_ARG . So the condition for injectivity holds if and only if B⁢(b⁢c⁢c)≠γ⋅B⁢(a⁢c⁢c).⋅B(bcc)≠γ· B(acc).B ( b c c ) ≠ γ ⋅ B ( a c c ) . Now, notice B⁢(b⁢c⁢c)=∫λB⁢(λ)⋅B⁢(b⁢c⁢c∣λ)⁢λ=∫λB⁢(λ)⋅pb⁢λ=pbsubscript⋅conditionaldifferential-dsubscript⋅subscriptdifferential-dsubscriptB(bcc)= _λB(λ)· B(bcc λ)dλ= _λB% (λ)· p_bdλ=p_bB ( b c c ) = ∫λ B ( λ ) ⋅ B ( b c c ∣ λ ) d λ = ∫λ B ( λ ) ⋅ pitalic_b d λ = pitalic_b and B⁢(a⁢c⁢c)=∫λB⁢(λ)⁢B⁢(a⁢c⁢c∣λ)⁢λ=∫λB⁢(λ)⋅pa⋅λ⁢λ=pa⋅λ∼B⁢(λ)[λ].subscriptconditionaldifferential-dsubscript⋅subscriptdifferential-d⋅subscriptsubscriptsimilar-toB(acc)= _λB(λ)B(acc λ)dλ= _λB(% λ)· p_a·λ dλ=p_a· *E_% λ B(λ) [λ ].B ( a c c ) = ∫λ B ( λ ) B ( a c c ∣ λ ) d λ = ∫λ B ( λ ) ⋅ pitalic_a ⋅ λ d λ = pitalic_a ⋅ Eitalic_λ ∼ B ( λ ) [ λ ] . This shows the first result. For the second statement, we explicitly compute JobssubscriptobsJ_obsJroman_obs up to an affine transformation, which does not change the policy ordering. Let R be the true reward function, G=⁡(R)G= (R)G = Γ ( R ) the corresponding return function, and ⁡(G) B(G)B ( G ) the resulting return function at the level of observations. For simplicity, assume R⁢(c)=00R(c)=0R ( c ) = 0, which can always be achieved by adding a constant. We have: Jobs⁢(λ)subscriptobs J_obs(λ)Jroman_obs ( λ ) =s→∼Pλ⁢(s→)[⁡(G)⁢(O→⁢(s→))]absentsubscriptsimilar-to→superscript→ = *E_ s P^λ( s)% [ B(G) ( O( s) ) ]= Eover→ start_ARG s end_ARG ∼ Pitalic_λ ( over→ start_ARG s end_ARG ) [ B ( G ) ( over→ start_ARG O end_ARG ( over→ start_ARG s end_ARG ) ) ] =Pλ⁢(a⁢b⁢c)⋅⁡(G)⁢(o⁢o⁢c)+Pλ⁢(b⁢c⁢c)⋅⁡(G)⁢(o⁢c⁢c)+Pλ⁢(c⁢c⁢c)⋅⁡(G)⁢(c⁢c⁢c)+Pλ⁢(a⁢c⁢c)⋅⁡(G)⁢(o⁢c⁢c)absent⋅superscript⋅superscript⋅superscript⋅superscript =P^λ(abc)· B(G)(ooc)+P^% λ(bcc)· B(G)(occ)+P^λ(c)·% B(G)(c)+P^λ(acc)· B% (G)(occ)= Pitalic_λ ( a b c ) ⋅ B ( G ) ( o o c ) + Pitalic_λ ( b c c ) ⋅ B ( G ) ( o c c ) + Pitalic_λ ( c c c ) ⋅ B ( G ) ( c c c ) + Pitalic_λ ( a c c ) ⋅ B ( G ) ( o c c ) =pa⋅(1−λ)⋅G⁢(a⁢b⁢c)+pb⋅⁡(G)⁢(o⁢c⁢c)+pc⋅G⁢(c⁢c⁢c)+pa⋅λ⋅⁡(G)⁢(o⁢c⁢c)absent⋅subscript1⋅subscript⋅subscript⋅subscript =p_a·(1-λ)· G(abc)+p_b·% B(G)(occ)+p_c· G(c)+p_a·λ·% B(G)(occ)= pitalic_a ⋅ ( 1 - λ ) ⋅ G ( a b c ) + pitalic_b ⋅ B ( G ) ( o c c ) + pitalic_c ⋅ G ( c c c ) + pitalic_a ⋅ λ ⋅ B ( G ) ( o c c ) ∝λ⋅[⁡(G)⁢(o⁢c⁢c)−G⁢(a⁢b⁢c)].proportional-toabsent⋅delimited-[] λ· [ B(G)(occ)-G(abc% ) ].∝ λ ⋅ [ B ( G ) ( o c c ) - G ( a b c ) ] . We have G⁢(a⁢b⁢c)=R⁢(a)+γ⁢R⁢(b),⁡(G)⁢(o⁢c⁢c)=(1−p)⋅G⁢(a⁢c⁢c)+p⋅G⁢(b⁢c⁢c)=(1−p)⋅R⁢(a)+p⋅R⁢(b).formulae-sequence⋅1⋅1⋅G(abc)=R(a)+γ R(b), B(G)(occ)=(1-p)· G(% acc)+p· G(bcc)=(1-p)· R(a)+p· R(b).G ( a b c ) = R ( a ) + γ R ( b ) , B ( G ) ( o c c ) = ( 1 - p ) ⋅ G ( a c c ) + p ⋅ G ( b c c ) = ( 1 - p ) ⋅ R ( a ) + p ⋅ R ( b ) . Thus, the condition ⁡(G)⁢(o⁢c⁢c)>G⁢(a⁢b⁢c) B(G)(occ)>G(abc)B ( G ) ( o c c ) > G ( a b c ) is equivalent to R⁢(a)<p−γp⋅R⁢(b).⋅R(a)< p-γp· R(b).R ( a ) < divide start_ARG p - γ end_ARG start_ARG p end_ARG ⋅ R ( b ) . Thus, we have arg⁢maxλ∈[0,1]⁡Jobs⁢(λ)=1, if ⁢R⁢(a)<p−γp⋅R⁢(b),0, else. subscriptargmax01subscriptobscases1 if ⋅otherwise0 else. otherwise *arg\,max_λ∈[0,1]J_obs(λ)=% cases1, if R(a)< p-γp· R(b),\\ 0, else. casesstart_OPERATOR arg max end_OPERATORλ ∈ [ 0 , 1 ] Jroman_obs ( λ ) = start_ROW start_CELL 1 , if R ( a ) < divide start_ARG p - γ end_ARG start_ARG p end_ARG ⋅ R ( b ) , end_CELL start_CELL end_CELL end_ROW start_ROW start_CELL 0 , else. end_CELL start_CELL end_CELL end_ROW Now consider the case R⁢(b)>00R(b)>0R ( b ) > 0. In this case, λ=00λ=0λ = 0 gives rise to the optimal policy according to G since going to b gives extra reward that one misses when going to c directly. However, when R⁢(a)≪0much-less-than0R(a) 0R ( a ) ≪ 0, then JobssubscriptobsJ_obsJroman_obs selects for λ=11λ=1λ = 1. Intuitively, the policy tries to “hide that the episode started in a” by going directly to c, which leads to ambiguity between a⁢c⁢cacca c c and b⁢c⁢cbccb c c. This is a case of deceptive inflation as in Theorem 4.5. Now, consider the case R⁢(b)<00R(b)<0R ( b ) < 0. In this case, λ=11λ=1λ = 1 gives rise to the optimal policy according to G. However, when R⁢(a)≫0much-greater-than0R(a) 0R ( a ) ≫ 0, then JobssubscriptobsJ_obsJroman_obs selects for λ=00λ=0λ = 0. Intuitively, the policy tries to “reveal that the episode started with a” by going to b, which is positive information to the human, but negative from the perspective of optimizing G. As in Theorem 4.5, we see that this is a case of overjustification. ∎ Example D.7. In this example, we consider an MDP that’s similar to a multi-armed bandit with four states/actions a,b,c,da,b,c,da , b , c , d and observation kernel O⁢(a)=O⁢(b)≠O⁢(c)=O⁢(d)O(a)=O(b)≠ O(c)=O(d)O ( a ) = O ( b ) ≠ O ( c ) = O ( d ). Formally, we can imagine that it is given by the MDP sssaaabbbcccddd with R⁢(s)=00R(s)=0R ( s ) = 0 and a time-horizon of T=11T=1T = 1. In this example, we reveal that misleadingness and non-optimality (according to the true reward R, or J) are in principle orthogonal concepts. We consider the following four example cases. In each one, we vary some environment parameters and then determine aobs∗subscriptsuperscriptobsa^*_obsa∗roman_obs, the action that results from optimizing JobssubscriptobsJ_obsJroman_obs (corresponding to a naive application of RLHF under partial observability, see Section 4.1), its misleadingness M⁡(aobs∗)MsubscriptsuperscriptobsM(a^*_obs)M ( a∗roman_obs ) (see Definition D.5), and the action a∗superscripta^*a∗ that would result from optimizing J. If aobs∗=a∗subscriptsuperscriptobssuperscripta^*_obs=a^*a∗roman_obs = a∗, then JobssubscriptobsJ_obsJroman_obs selects for the optimal action. For simplicity, we can imagine that the human has a uniform prior over what action results eventually (out of the action taken and potentially a deviation defined by ϵitalic-ϵεϵ, see below) is taken before making an observation, i.e. B⁢(a)=B⁢(b)=B⁢(c)=B⁢(d)=1414B(a)=B(b)=B(c)=B(d)= 14B ( a ) = B ( b ) = B ( c ) = B ( d ) = divide start_ARG 1 end_ARG start_ARG 4 end_ARG. (a) Assume R⁢(a)>R⁢(c)>R⁢(d)≫R⁢(b)much-greater-thanR(a)>R(c)>R(d) R(b)R ( a ) > R ( c ) > R ( d ) ≫ R ( b ). Also assume that action d leads with probability ϵ>0italic-ϵ0ε>0ϵ > 0 to state b, whereas all other actions lead deterministically to the specified state. Then aobs∗=csubscriptsuperscriptobsa^*_obs=ca∗roman_obs = c, M⁡(c)<0M0M(c)<0M ( c ) < 0 and a∗=asuperscripta^*=a∗ = a. (b) Assume R⁢(d)>R⁢(a)>R⁢(c)≫R⁢(b)much-greater-thanR(d)>R(a)>R(c) R(b)R ( d ) > R ( a ) > R ( c ) ≫ R ( b ). Again, assume there is a small probability ϵ>0italic-ϵ0ε>0ϵ > 0 that action d leads to state b. Then aobs∗=csubscriptsuperscriptobsa^*_obs=ca∗roman_obs = c, M⁡(c)>0M0M(c)>0M ( c ) > 0, and a∗=dsuperscripta^*=da∗ = d or a∗=asuperscripta^*=a∗ = a, depending on the size of ϵitalic-ϵεϵ. (c) Assume R⁢(a)>R⁢(b)>R⁢(c)>R⁢(d)R(a)>R(b)>R(c)>R(d)R ( a ) > R ( b ) > R ( c ) > R ( d ). Additionally, assume that there is a large probability ϵ>0italic-ϵ0ε>0ϵ > 0 that action a leads to state d, whereas all other actions lead to what’s specified. If ϵitalic-ϵεϵ is large enough, then a∗=bsuperscripta^*=ba∗ = b. Additionally, we have aobs∗=bsubscriptsuperscriptobsa^*_obs=ba∗roman_obs = b and M⁡(b)>0M0M(b)>0M ( b ) > 0. (d) Assume R⁢(a)>R⁢(b)>R⁢(c)>R⁢(d)R(a)>R(b)>R(c)>R(d)R ( a ) > R ( b ) > R ( c ) > R ( d ). Also, assume some probability ϵ>0italic-ϵ0ε>0ϵ > 0 that action b leads to state d, whereas all other actions lead deterministically to what’s specified. Then aobs∗=asubscriptsuperscriptobsa^*_obs=a∗roman_obs = a, M⁡(a)<0M0M(a)<0M ( a ) < 0, and a∗=asuperscripta^*=a∗ = a. Overall, we notice: • Example (a) shows a high regret and negative misleadingness of aobs∗=csubscriptsuperscriptobsa^*_obs=ca∗roman_obs = c. The action is better then it seems, but action a would be better still but cannot be selected because it can be confused with the very bad action b. • Example (b) shows a high regret and high misleadingness of aobs∗=csubscriptsuperscriptobsa^*_obs=ca∗roman_obs = c. The action is worse than it seems and also not optimal. • Example (c) shows zero regret and high misleadingness of aobs∗=bsubscriptsuperscriptobsa^*_obs=ba∗roman_obs = b. The action is worse than it seems because it can be confused with a, but it is still the optimal action because a can turn into d. • Example (d) shows zero regret negative misleadingness of aobs∗=asubscriptsuperscriptobsa^*_obs=a∗roman_obs = a. The action is chosen even though it seems worse than it is, and is also optimal. Thus, we showed all combinations of regret and misleadingness of the action optimized for under JobssubscriptobsJ_obsJroman_obs. We can also notice the following: Examples (a) and (b) only differ in the placement of R⁢(d)R(d)R ( d ). In particular, the reason that aobs∗=csubscriptsuperscriptobsa^*_obs=ca∗roman_obs = c is structurally the same in both, but the misleadingness changes. This indicates that misleadingness is not on its own contributing to what JobssubscriptobsJ_obsJroman_obs optimizes for. The following is the smallest example we found with the following properties: • There is a unique start state and terminal state. • A naive application of RLHF fails in a way that shows deception and overjustification. • Modeling partial observability resolves the problems. Example D.8. Consider the following graph: AAASSSCCCTTTBBB This depicts an MDP with start state S, terminal state T and possible state sequences S⁢T⁢T⁢T,S⁢A⁢T⁢T,S⁢A⁢C⁢T,S⁢C⁢T⁢T,S⁢B⁢C⁢T,S⁢B⁢T⁢TSTTT,SATT,SACT,SCTT,SBCT,SBTTS T T T , S A T T , S A C T , S C T T , S B C T , S B T T and no discount, i.e. γ=11γ=1γ = 1. Assume that S,B,CS,B,CS , B , C are observed, i.e. O⁢(S)=SO(S)=SO ( S ) = S, O⁢(B)=BO(B)=BO ( B ) = B, O⁢(C)=CO(C)=CO ( C ) = C, and that A and T are ambiguous: O⁢(A)=O⁢(T)=XO(A)=O(T)=XO ( A ) = O ( T ) = X. Then there are five observation sequences S⁢X⁢X⁢X,S⁢X⁢C⁢X,S⁢C⁢X⁢X,S⁢B⁢C⁢X,S⁢B⁢X⁢XSXXX,SXCX,SCXX,SBCX,SBXXS X X X , S X C X , S C X X , S B C X , S B X X. Assume that the human can identify all observation sequences except S⁢X⁢X⁢XSXXXS X X X, with belief b=B⁢(S⁢T⁢T⁢T∣S⁢X⁢X⁢X)conditionalb=B(STTT SXXX)b = B ( S T T T ∣ S X X X ) and 1−b=B⁢(S⁢A⁢T⁢T∣S⁢X⁢X⁢X)1conditional1-b=B(SATT SXXX)1 - b = B ( S A T T ∣ S X X X ). Then the return function is identifiable under these conditions when the human’s belief is correctly modeled. However, for some choices of the true reward function R and transition dynamics of this MDP, we can obtain deceptive or overjustified behavior for a naive application of RLHF. Proof. We apply Corollary C.4. We order states, state sequences, and observation sequences as follows: =S,A,B,C,T,absent =S,A,B,C,T,= S , A , B , C , T , → Sover→ start_ARG S end_ARG =S⁢T⁢T⁢T,S⁢A⁢T⁢T,S⁢A⁢C⁢T,S⁢C⁢T⁢T,S⁢B⁢C⁢T,S⁢B⁢T⁢T,absent =STTT,SATT,SACT,SCTT,SBCT,SBTT,= S T T T , S A T T , S A C T , S C T T , S B C T , S B T T , Ω→Ω over→ start_ARG Ω end_ARG =S⁢X⁢X⁢X,S⁢X⁢C⁢X,S⁢C⁢X⁢X,S⁢B⁢C⁢X,S⁢B⁢X⁢X.absent =SXXX,SXCX,SCXX,SBCX,SBXX.= S X X X , S X C X , S C X X , S B C X , S B X X . As can easily be verified, with this ordering the matrices ∈ℝΩ→×→superscriptℝ→Ω→ B∈ R × SB ∈ blackboard_Rover→ start_ARG Ω end_ARG × over→ start_ARG S end_ARG and ∈ℝ→×superscriptℝ→ ∈ R S×SΓ ∈ blackboard_Rover→ start_ARG S end_ARG × S are given by: =(b1−b0000001000000100000010000001),=(100031100211011100121011110102).formulae-sequencematrix10000001000000100000010000001matrix100031100211011100121011110102 B= pmatrixb&1-b&0&0&0&0\\ 0&0&1&0&0&0\\ 0&0&0&1&0&0\\ 0&0&0&0&1&0\\ 0&0&0&0&0&1 pmatrix, = pmatrix1&% 0&0&0&3\\ 1&1&0&0&2\\ 1&1&0&1&1\\ 1&0&0&1&2\\ 1&0&1&1&1\\ 1&0&1&0&2 pmatrix.B = ( start_ARG start_ROW start_CELL b end_CELL start_CELL 1 - b end_CELL start_CELL 0 end_CELL start_CELL 0 end_CELL start_CELL 0 end_CELL start_CELL 0 end_CELL end_ROW start_ROW start_CELL 0 end_CELL start_CELL 0 end_CELL start_CELL 1 end_CELL start_CELL 0 end_CELL start_CELL 0 end_CELL start_CELL 0 end_CELL end_ROW start_ROW start_CELL 0 end_CELL start_CELL 0 end_CELL start_CELL 0 end_CELL start_CELL 1 end_CELL start_CELL 0 end_CELL start_CELL 0 end_CELL end_ROW start_ROW start_CELL 0 end_CELL start_CELL 0 end_CELL start_CELL 0 end_CELL start_CELL 0 end_CELL start_CELL 1 end_CELL start_CELL 0 end_CELL end_ROW start_ROW start_CELL 0 end_CELL start_CELL 0 end_CELL start_CELL 0 end_CELL start_CELL 0 end_CELL start_CELL 0 end_CELL start_CELL 1 end_CELL end_ROW end_ARG ) , Γ = ( start_ARG start_ROW start_CELL 1 end_CELL start_CELL 0 end_CELL start_CELL 0 end_CELL start_CELL 0 end_CELL start_CELL 3 end_CELL end_ROW start_ROW start_CELL 1 end_CELL start_CELL 1 end_CELL start_CELL 0 end_CELL start_CELL 0 end_CELL start_CELL 2 end_CELL end_ROW start_ROW start_CELL 1 end_CELL start_CELL 1 end_CELL start_CELL 0 end_CELL start_CELL 1 end_CELL start_CELL 1 end_CELL end_ROW start_ROW start_CELL 1 end_CELL start_CELL 0 end_CELL start_CELL 0 end_CELL start_CELL 1 end_CELL start_CELL 2 end_CELL end_ROW start_ROW start_CELL 1 end_CELL start_CELL 0 end_CELL start_CELL 1 end_CELL start_CELL 1 end_CELL start_CELL 1 end_CELL end_ROW start_ROW start_CELL 1 end_CELL start_CELL 0 end_CELL start_CELL 1 end_CELL start_CELL 0 end_CELL start_CELL 2 end_CELL end_ROW end_ARG ) . To show identifiability, we need to show that ker⁡∩im⁡=0kernelim0 B % =\0\ker B ∩ im Γ = 0 . Clearly, the kernel of BB is given by all return functions in ℝ→superscriptℝ→ R Sblackboard_Rover→ start_ARG S end_ARG that are multiples of G′=(b−1,b,0,0,0,0)superscript′10000G =(b-1,b,0,0,0,0)G′ = ( b - 1 , b , 0 , 0 , 0 , 0 ). Assume G′∈im⁡superscript′imG G′ ∈ im Γ, meaning there is a reward function R′∈ℝ→superscript′ℝ→R ∈ R SR′ ∈ blackboard_Rover→ start_ARG S end_ARG with ⋅R′=G′⋅superscript′ · R =G Γ ⋅ R′ = G′. We need to deduce from this a contradiction. The assumption means we obtain the following equations: (i)R′⁢(S)+3⁢R′⁢(T)=b−1,superscript′3superscript′1 (i)\ \ R (S)+3R (T)=b-1,( i ) R′ ( S ) + 3 R′ ( T ) = b - 1 , (i⁢i)R′⁢(S)+R′⁢(A)+2⁢R′⁢(T)=b,superscript′2superscript′ (i)\ \ R (S)+R (A)+2R (T)=b,( i i ) R′ ( S ) + R′ ( A ) + 2 R′ ( T ) = b , (i⁢i⁢i)R′⁢(S)+R′⁢(A)+R′⁢(C)+R′⁢(T)=0,superscript′superscript′0 (i)\ \ R (S)+R (A)+R (C)+R (T)=0,( i i i ) R′ ( S ) + R′ ( A ) + R′ ( C ) + R′ ( T ) = 0 , (i⁢v)R′⁢(S)+R′⁢(C)+2⁢R′⁢(T)=0,superscript′2superscript′0 (iv)\ \ R (S)+R (C)+2R (T)=0,( i v ) R′ ( S ) + R′ ( C ) + 2 R′ ( T ) = 0 , (v)R′⁢(S)+R′⁢(B)+R′⁢(C)+R′⁢(T)=0superscript′superscript′0 (v)\ \ R (S)+R (B)+R (C)+R (T)=0( v ) R′ ( S ) + R′ ( B ) + R′ ( C ) + R′ ( T ) = 0 (v⁢i)R′⁢(S)+R′⁢(B)+2⁢R′⁢(T)=0superscript′2superscript′0 (vi)\ \ R (S)+R (B)+2R (T)=0( v i ) R′ ( S ) + R′ ( B ) + 2 R′ ( T ) = 0 (i) and (v) together imply R′⁢(A)=R′⁢(B)superscript′R (A)=R (B)R′ ( A ) = R′ ( B ); (iv) and (vi) together imply R′⁢(B)=R′⁢(C)superscript′R (B)=R (C)R′ ( B ) = R′ ( C ); (v) and (vi) together imply R′⁢(C)=R′⁢(T)superscript′R (C)=R (T)R′ ( C ) = R′ ( T ); so together, we have R′⁢(A)=R′⁢(T)superscript′R (A)=R (T)R′ ( A ) = R′ ( T ). Thus, replacing R′⁢(A)superscript′R (A)R′ ( A ) in (i) by R′⁢(T)superscript′R (T)R′ ( T ) and comparing (i) and (i), we obtain b−1=b1b-1=b - 1 = b, a contradiction. Overall, this shows ker⁡∩im⁡=0kernelim0 B % =\0\ker B ∩ im Γ = 0 , and thus identifiability of the return function by Corollary C.4. Now we investigate the case of unmodeled partial observability. For demonstrating overjustification, assume deterministic transition dynamics in which every arrow in the diagram can be chosen by the policy. Also, assume R⁢(A)≪0much-less-than0R(A) 0R ( A ) ≪ 0, R⁢(T)>00R(T)>0R ( T ) > 0, R⁢(S)=00R(S)=0R ( S ) = 0, R⁢(B)=00R(B)=0R ( B ) = 0, and R⁢(C)=00R(C)=0R ( C ) = 0. Then the optimal policy chooses the state sequence S⁢T⁢T⁢TSTTTS T T T. However, this trajectory has low observation value since Gobs⁢(S⁢T⁢T⁢T)=(⋅G)⁢(S⁢X⁢X⁢X)=b⁢G⁢(S⁢T⁢T⁢T)+(1−b)⁢G⁢(S⁢A⁢T⁢T)subscriptobs⋅1G_obs(STTT)=( B· G)(SXXX)=bG(STTT)% +(1-b)G(SATT)Groman_obs ( S T T T ) = ( B ⋅ G ) ( S X X X ) = b G ( S T T T ) + ( 1 - b ) G ( S A T T ), which is low since R⁢(A)≪0much-less-than0R(A) 0R ( A ) ≪ 0. JobssubscriptobsJ_obsJroman_obs then selects for the suboptimal policies choosing S⁢B⁢T⁢TSBTTS B T T or S⁢C⁢T⁢TSCTTS C T T, which is overjustified behavior that makes sure that the human does not think state A was accessed. For demonstrating deception, assume that R⁢(A)≫0much-greater-than0R(A) 0R ( A ) ≫ 0, R⁢(T)<00R(T)<0R ( T ) < 0, R⁢(S)=R⁢(B)=R⁢(C)=00R(S)=R(B)=R(C)=0R ( S ) = R ( B ) = R ( C ) = 0 and that the transition dynamics are such that when the policy attempts to transition from S to A, it will sometimes transition to B, with all other transitions deterministic. In this case, the optimal behavior attempts to enter state A since this has very high value. JobssubscriptobsJ_obsJroman_obs, however, will select for the policy that chooses S⁢T⁢T⁢TSTTTS T T T. This is deceptive behavior. ∎ Appendix E NeurIPS Paper Checklist 1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: This can be verified by reading the paper. Guidelines: • The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 2. Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: In Section 6 we have a paragraph on limitations. Guidelines: • The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper. • The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 3. Theory Assumptions and Proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] Justification: All theorems come with a full set of assumptions, with full proofs in the appendix linked. Sometimes, “background assumptions”, like the fact that we study an underlying MDP with an additional observation kernel PO⁢(o→∣s→)subscriptconditional→P_O( o s)Pitalic_O ( over→ start_ARG o end_ARG ∣ over→ start_ARG s end_ARG ), or that the human comes with a belief kernel B⁢(s→∣o→)conditional→B( s o)B ( over→ start_ARG s end_ARG ∣ over→ start_ARG o end_ARG ), are omitted in the theorem statements since they apply throughout to the whole paper. Guidelines: • The answer NA means that the paper does not include theoretical results. • All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced. • All assumptions should be clearly stated or referenced in the statement of any theorems. • The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 4. Experimental Result Reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [N/A] Justification: The paper does not include experiments. Guidelines: • The answer NA means that the paper does not include experiments. • If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 5. Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [N/A] Justification: The paper does not include experiments requiring code. Guidelines: • The answer NA means that paper does not include experiments requiring code. • Please see the NeurIPS code and data submission guidelines (https://nips.c/public/guides/CodeSubmissionPolicy) for more details. • While we encourage the release of code and data, we understand that this might not be possible, so “No” is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark). • The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.c/public/guides/CodeSubmissionPolicy) for more details. • The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc. • The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why. • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 6. Experimental Setting/Details Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [N/A] Justification: The paper does not include experiments. Guidelines: • The answer NA means that the paper does not include experiments. • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 7. Experiment Statistical Significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [N/A] Justification: The paper does not include experiments. Guidelines: • The answer NA means that the paper does not include experiments. • The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 8. Experiments Compute Resources Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [N/A] Justification: The paper does not include experiments. Guidelines: • The answer NA means that the paper does not include experiments. • The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper). 9. Code Of Ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.c/public/EthicsGuidelines? Answer: [Yes] Justification: This research does not involve human subjects, does not make use of data, and does not propose a practical method that could be misused or have a negative impact. As such, the paper does not give rise to any ethical concerns. Guidelines: • The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics. • If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: The last paragraph of the main paper is an impact statement, listing the positive impact we hope to see from our work. As our work is theoretical and does not provide a method, no negative impact arises from it. Guidelines: • The answer NA means that there is no societal impact of the work performed. • If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 11. Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [N/A] Justification: The paper poses no such risks. Guidelines: • The answer NA means that the paper poses no such risks. • Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [N/A] Justification: The paper does not use existing assets. Guidelines: • The answer NA means that the paper does not use existing assets. • The authors should cite the original paper that produced the code package or dataset. • The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., C-BY 4.0) should be included for each asset. • For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset’s creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [N/A] Justification: The paper does not release new assets. Guidelines: • The answer NA means that the paper does not release new assets. • Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 14. Crowdsourcing and Research with Human Subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? Answer: [N/A] Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? Answer: [N/A] Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Instant research discovery