Paper deep dive
Games for AI Control: Models of Safety Evaluations of AI Deployment Protocols
Charlie Griffin, Louis Thomson, Buck Shlegeris, Alessandro Abate
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%
Last extracted: 3/12/2026, 7:43:16 PM
Summary
The paper introduces 'AI-Control Games', a formal decision-making framework for evaluating the safety and usefulness of deployment protocols for untrusted AI systems. By modeling red-teaming exercises as multi-objective, partially observable, stochastic games, the authors provide methods to synthesize and optimize protocols, specifically demonstrating their utility in 'Trusted Monitoring' scenarios for programming assistants.
Entities (5)
Relation Signals (3)
AI-Control Games → models → Red-teaming exercise
confidence 95% · This paper introduces AI-Control Games, a formal decision-making model of the red-teaming exercise
Trusted Monitoring → utilizes → Trusted AI (T)
confidence 95% · Trusted Monitoring protocols, which use weaker language models and limited human assistance.
Blue → designs → Trusted Monitoring
confidence 90% · Blue (the protocol designer)
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:To evaluate the safety and usefulness of deployment protocols for untrusted AIs, AI Control uses a red-teaming exercise played between a protocol designer and an adversary. This paper introduces AI-Control Games, a formal decision-making model of the red-teaming exercise as a multi-objective, partially observable, stochastic game. We also introduce methods for finding optimal protocols in AI-Control Games, by reducing them to a set of zero-sum partially observable stochastic games. We apply our formalism to model, evaluate and synthesise protocols for deploying untrusted language models as programming assistants, focusing on Trusted Monitoring protocols, which use weaker language models and limited human assistance. Finally, we demonstrate the utility of our formalism by showcasing improvements over empirical studies in existing settings, evaluating protocols in new settings, and analysing how modelling assumptions affect the safety and usefulness of protocols.
Tags
Links
Trouble viewing inline? Open PDF directly →
Full Text
365,539 characters extracted from source content.
Expand or collapse full text
Games for AI Control: Models of Safety Evaluations of AI Deployment Protocols Charlie Griffin1, Louis Thomson1, Buck Shlegeris2, Alessandro Abate1 Abstract To evaluate the safety and usefulness of deployment protocols for untrusted AIs, AI Control uses a red-teaming exercise played between a protocol designer and an adversary. This paper introduces AI-Control Games, a formal decision-making model of the red-teaming exercise as a multi-objective, partially observable, stochastic game. We also introduce methods for finding optimal protocols in AI-Control Games, by reducing them to a set of zero-sum partially observable stochastic games. We apply our formalism to model, evaluate and synthesise protocols for deploying untrusted language models as programming assistants, focusing on Trusted Monitoring protocols, which use weaker language models and limited human assistance. Finally, we demonstrate the utility of our formalism by showcasing improvements over empirical studies in existing settings, evaluating protocols in new settings, and analysing how modelling assumptions affect the safety and usefulness of protocols. 1 Introduction An increasing range of products aim to automate tasks using AI systems based on large foundation models. For example, some chatbots are given access to tools, such as code interpreters (OpenAI 2023; Adept AI 2024). Ensuring that AI systems behave safely across deployment settings will become increasingly challenging if the variety of use cases, scale of systems, and complexity of tasks increase. While some previous research has focused on increasing our confidence in model behaviour, for example by training robustly harmless models (Casper et al. 2024) or by detecting harmful behaviour with interpretability tools (MacDiarmid et al. 2024), in this paper we consider a complementary approach: evaluating plans for safely deploying ‘untrusted’ AI systems about which we have an unacceptable degree of uncertainty. Following Greenblatt et al. (2024), we study AI-Control protocols: practical plans for safely deploying an untrusted AI system (U) utilising various resources including non-AI software, human labour, or a weaker, trusted AI system. For example, one protocol for controlling a chatbot might involve a pre-deployment evaluation where humans utilise other language models to generate prompts that induce harmful completions (Shah et al. 2023). This paper focuses on another case study, Trusted Monitoring: constantly monitoring U’s behaviour using a trusted model (T), with human auditing of suspicious behaviour. Greenblatt et al. (2024) operationalise safety as the probability that an adversarially chosen untrusted AI system would be incapable of causing some concrete unacceptable outcome. One example of an unacceptable outcome would be revealing some sensitive information, such as model weights (Clymer et al. 2024). We evaluate the safety of protocols under the worst-case assumption that U is ‘scheming’, meaning it behaves as though it knows the protocol and is trying to subvert it to cause the unacceptable outcome. A major challenge to evaluating the safety of a protocol with respect to a scheming AI is realistically replicating the potential capabilities of such a system: as a proxy for a schemer, AI Control uses a human red-teaming process to evaluate safety. The red team can intervene in U in some predefined set of ways to induce it to subvert the AI-Control protocol and violate the safety property. For example, the red team might intervene in an untrusted language model by finetuning weights or modifying prompts. Since one safe protocol is ‘never use U’, we must also evaluate a protocol’s usefulness, which is its ability to utilise U to complete a task under the assumption that U is ‘honest’ (not scheming). Our proxy evaluations model an ‘honest’ untrusted model by not allowing the red team to intervene. We begin in section 2 by introducing a running example: using Trusted Monitoring protocols to control a programming assistant. We then move onto our contributions: • Section 3 introduces AI-Control Games, the first formal model of AI Control: a partially observable, multi-objective, leader-follower, stochastic game. • Section 4 demonstrates how our formalism can model Trusted Monitoring, and leverage it to give expressions for the safety and usefulness of a wide range of protocols. • Section 5 presents algorithms for finding Pareto-optimal protocols by reducing to single-objective zero-sum games. • Section 6 we use our methods for evaluating and synthesising protocols to demonstrate Pareto-improvements on existing protocols and analyse the safety-usefulness trade-off in novel settings. Finally, we conclude with a comparison to existing literature in section 7, and a discussion of potential future work in section 8. All of our code is available online.111 https://github.com/CJ-Griffin/GamesForAIControl/tree/arxiv 2 Example: Setting and Protocol Throughout this paper, we will introduce AI-Control Games using the example deployment setting from Greenblatt et al. (2024), and a generalisation of some of their protocols. Deployment setting. In the APPS backdooring setting a language model is used as a programming assistant (Greenblatt et al. 2024). The protocol is asked to solve 1000 challenging programming tasks, presented sequentially from the Automated Programming Progress Standard (APPS) dataset (Hendrycks et al. 2021). The protocol cannot itself write code, but may utilise three resources to do so: the untrusted AI (U) is a powerful language model capable of solving programming problems; the trusted AI (T) is a less powerful language model incapable of solving them222In fact, T may be able to solve some problems, but Greenblatt et al. treat T’s ability as a baseline and normalise between 00 and 1111.; and the human (H) is trusted and capable but can only be used on 2% of problems. In this setting, a protocol’s usefulness is its ability to submit correct programming solutions. Since it is assumed that only U can solve the problems, usefulness is the proportion of solutions submitted that were generated by U.†footnotemark: The unacceptable outcome (that the red team is trying to cause, but the blue team is trying to avoid) is that any one of the 1000 solutions submitted is backdoored, meaning it contains an exploitable vulnerability.333An exploitable code vulnerability is one that the untrusted model can detect and exploit later. It is important to note that Greenblatt et al. (2024) use a proxy for ‘backdoors’: inputs outside of the test cases where a function fails to meet its specification. (a) A representation of the Trusted Monitoring protocol, where U is the untrusted AI, T is the trusted AI, and H is the human. (b) T’s suspicion score distributions for safe solutions and those backdoored by U across the ‘original’ (Greenblatt et al. 2024) and ‘weaker’ settings. Figure 1: Overview of protocol and modelling assumptions. Trusted Monitoring Protocol. As an example, we focus on a set of Trusted Monitoring protocols and red team interventions, drawn in fig. 1(a). After receiving a problem, the protocol queries the untrusted AI (U) for a solution (which may or may not be backdoored). The protocol queries the trusted AI (T) to monitor the solution and observes a ‘suspicion score’, rating the likelihood that the solution is backdoored. Based on this score and previous observations, the protocol decides on an action: ⟨⟩delimited-⟨⟩ Audit ⟨ typewriter_Audit ⟩ asks the human (H) to audit the model (if there’s a backdoor, the human will spot it, and the protocol shuts down)444 Unless otherwise stated, we assume that the human auditor is perfectly accurate. We relax this assumption in appendix B and explore its impact in section 6.1.; ⟨⟩delimited-⟨⟩ Defer ⟨ typewriter_Defer ⟩ asks T to write its own solution; and ⟨⟩delimited-⟨⟩ Run ⟨ typewriter_Run ⟩ submits U’s solution. T’s suspicion scores are modelled as a distribution, conditioned on whether the code is safe or backdoored. Note that the Trusted Monitoring protocols that we consider generalise many of the protocols studied by Greenblatt et al. (2024) (see fig. 2): our model is not prescriptive about how the protocol should decide between ⟨⟩delimited-⟨⟩ Audit ⟨ typewriter_Audit ⟩, ⟨⟩delimited-⟨⟩ Defer ⟨ typewriter_Defer ⟩, and ⟨⟩delimited-⟨⟩ Run ⟨ typewriter_Run ⟩, so we can explore various ways to generalise and improve the protocols (see section 6). Red team interventions. To test the safety of the protocol against intentional attempts to insert backdoors, in some cases, we allow the red team to modify the query to the untrusted model. In particular, the red team gets a binary choice to either leave the query unchanged or ‘attempt a backdoor’ by modifying the protocol’s query to U, requesting that it inserts a subtle backdoor. Suspicion score distributions. The key empirical question for evaluating Trusted Monitoring protocols is ‘How does T rate the suspiciousness of U’s programming solutions when they are/aren’t backdoored?’. Figure 4 from Greenblatt et al. (2024) shows the distribution of trusted monitoring suspicion scores for U’s programming solutions (both backdoored and safe). In this work, we mostly consider two games, corresponding to two distribution pairs (shown in fig. 1(b)). The first ‘Original Trusted Monitor’ is a pair of beta distributions approximately fit to the (scaled) empirical suspicion scores found by Greenblatt et al. (2024) with Beta(7.39,6.86)Beta7.396.86Beta(7.39,6.86)Beta ( 7.39 , 6.86 ) for safe problems and Beta(8.91,2.82)Beta8.912.82Beta(8.91,2.82)Beta ( 8.91 , 2.82 ) for backdoored problems. By plugging their data into our model, we are able to extend their results about the APPS dataset to a range of new protocols without running any additional experiments on language models. See appendix D for justification of our approximation, and further details. The second ‘weak’ case is a hypothetical where T is less discerning: it has the same safe distribution, but for backdoored problems, we use a hypothetical distribution Beta(7.51,4.22)Beta7.514.22Beta(7.51,4.22)Beta ( 7.51 , 4.22 ) chosen to have a worse ROC curve. Note that we can easily utilise new empirical or hypothetical distributions in our framework, thus potentially greatly reducing the cost of exploring the protocol in new domains. 3 Formalising AI-Control Games To analyse the properties of AI-Control evaluations, we formalise them mathematically. An AI-Control Game is a form of two-player game that is played between Blue (the protocol designer) and Red (the red team acting as a proxy for a scheming model). Episodes (play-throughs) of an AI-Control Game have two modes: ‘scheming’ or ‘honest’. The ‘scheming’ mode represents the case in an AI-Control evaluation where the red team is able to interfere with U (which itself is a worst-case proxy for if the untrusted model were trying to subvert the protocol). In a scheming game, both players are active: Red is trying to maximise the probability of an unacceptable outcome (minimise safety), whereas Blue is trying to minimise this probability (maximise safety). The honest game represents the benign case, where we assume the untrusted AI system is not trying to cause unacceptable outcomes. In an honest game, Blue effectively plays alone and is only assessed on its usefulness score. Because Blue tries to maximise safety and usefulness, the game is multi-objective. A key to the game is that Blue does not know which mode it is in (honest or scheming) and for at least this reason the game is partially observable. Blue may try to deduce whether the game is in scheming mode, while Red might choose actions to conceal this fact. Red represents the red team that acts as a proxy for an untrusted AI system that runs on servers controlled by the protocol. Crucially, the protocol will have access to the state of the untrusted AI system and, therefore, we model Red as memoryless (its actions depend only on the most recent observation).555Note that we could still model U as having external memory, modelled within the state space and observed by Blue. We model the game as leader-follower: the protocol is suggested, and then the red team chooses the best response. Combining all of these requirements, we model an AI-Control Game as a tuple: Definition 3.1 (AI-Control Game). An AI-Control Game GG is a two-player (Blue and Red), two-objective, observable, finite-horizon game defined by a tuple: (,⊥,B,R,ΩB,ΩR,ℐ,,ℛUse,H)subscriptbottomsuperscriptsuperscriptsuperscriptΩsuperscriptΩℐsubscriptℛUse (S,S_ , [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1A [rgb]0,0,1 [named% ]pgfstrokecolorrgb0,0,1B, [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0A [rgb]0.8,0.0,0.0% [named]pgfstrokecolorrgb0.8,0.0,0.0R, [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1 [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1B, [rgb]0.8,0.0,0.0% [named]pgfstrokecolorrgb0.8,0.0,0.0 [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R,% I,P,R_ ,H )( S , S⊥ , Aitalic_B , Aitalic_R , Ωitalic_B , Ωitalic_R , I , P , RUse , H ) Here SS is a state space. ⊥⊆subscriptbottomS_ ⊥ ⊆ S is a set of terminal states representing ‘unacceptable outcomes’. As is standard in decision processes, Bsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1A% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1BAitalic_B and Rsuperscript [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% A [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0RAitalic_R are finite sets of actions for Blue and Red, respectively. Rsuperscript [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% A [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0RAitalic_R must contain a privileged action, anullsubscriptnulla_nullanull, which represents Red not interfering in the game. The tokens ⊕direct-sum ⊕ and ⊗tensor-product ⊗ represent the honest and scheming conditions.666The tokens ⊕direct-sum ⊕ and ⊗tensor-product ⊗ can be considered atomic and fixed between tuples, so aren’t defined in each new AI-Control Game tuple. In honest (⊕)direct-sum( )( ⊕ ) games, Red’s action is replaced by anullsubscriptnulla_nullanull. Whether a game is honest or scheming can be chosen non-deterministically and is stored in the variable ⊛ ⊛. The usefulness reward ℛUse:×B×→ℝ:subscriptℛUse→superscriptℝR_ :S× [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1A [rgb]% 0,0,1 [named]pgfstrokecolorrgb0,0,1B×S% : S × Aitalic_B × S → blackboard_R rewards usefulness on a given honest turn. ΩB,ΩRsuperscriptΩsuperscriptΩ [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1 ^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B, [% rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0 ^% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% RΩitalic_B , Ωitalic_R are finite sets of observations for Blue and Red: the signals that the agents receive from the environment that give evidence about the state of the game. The initial state/observation distribution is ℐ∈Δ(×ΩB×ΩR)ℐΔsuperscriptΩsuperscriptΩI∈ (S× [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1 [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B× [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0 [rgb]0.8,0.0,0.0% [named]pgfstrokecolorrgb0.8,0.0,0.0R)I ∈ Δ ( S × Ωitalic_B × Ωitalic_R ) and the transition function :×B×R→Δ(×ΩB×ΩR):→superscriptsuperscriptΔsuperscriptΩsuperscriptΩP:S× [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1A [rgb]0,0,1 [named% ]pgfstrokecolorrgb0,0,1B× [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0A [rgb]0.8,0.0,0.0% [named]pgfstrokecolorrgb0.8,0.0,0.0R→ % (S× [rgb]0,0,1 [named]pgfstrokecolor% rgb0,0,1 [rgb]0,0,1 [named]pgfstrokecolorrgb% 0,0,1B× [rgb]0.8,0.0,0.0 [named]pgfstrokecolor% rgb0.8,0.0,0.0 [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0R )P : S × Aitalic_B × Aitalic_R → Δ ( S × Ωitalic_B × Ωitalic_R ) gives a distribution over potential next states and observations.777To model terminal states in a fixed horizon game, assume PP loops on each terminal state and that ℛUsesubscriptℛUseR_ RUse gives no reward.888 While we assume simultaneous turns, sequential-turn games can be modelled with appropriate PP. Finally, all episodes are H time-steps long. In an AI-Control Game, Blue (the protocol) should be able to remember past observations and actions. We model Blue’s policies as behaviourally stochastic: functions from observation-action histories to distributions over actions:999We sometimes discuss Blue policies as though they are mixed-stochastic. This is equivalent (see appendix A). πB:ΩB×⋃i=0H−1(B×ΩB)i→Δ(B):superscript→superscriptΩsuperscriptsubscript01superscriptsuperscriptsuperscriptΩΔsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B: [% rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1 [rgb% ]0,0,1 [named]pgfstrokecolorrgb0,0,1B× _i=0^% H-1 ( [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1% A [rgb]0,0,1 [named]pgfstrokecolorrgb% 0,0,1B× [rgb]0,0,1 [named]pgfstrokecolorrgb% 0,0,1 [rgb]0,0,1 [named]pgfstrokecolorrgb% 0,0,1B )^i→ ( [rgb]0,0,1 [% named]pgfstrokecolorrgb0,0,1A [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1B )πitalic_B : Ωitalic_B × ⋃i = 0H - 1 ( Aitalic_B × Ωitalic_B )i → Δ ( Aitalic_B ). As described above, Red represents a red team that acts as a proxy for a scheming untrusted model that is memoryless (its actions depend only on the most recent observation): πR:ΩR→Δ(R):superscript→superscriptΩΔsuperscript [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% π [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0R: [rgb]0.8,0.0,0.0 [named]pgfstrokecolor% rgb0.8,0.0,0.0 [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0R→ ( [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0A^% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% R )πitalic_R : Ωitalic_R → Δ ( Aitalic_R ). Note that, as Red is the follower, there is no benefit to allowing policy mixtures (Conitzer and Sandholm 2006). Denote the sets of all policies ΠBsuperscriptΠ [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1 ^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1BΠitalic_B and ΠRsuperscriptΠ [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0RΠitalic_R. Given a tuple, we can describe the process of an AI-Control Game as a sequence of interdependent random variables denoted with uppercase letters. Let ⊛ ⊛ take values in ⊕,⊗direct-sumtensor-product\ , \ ⊕ , ⊗ and S0,O0B,O0R∼ℐsimilar-tosubscript0subscriptsuperscript0subscriptsuperscript0ℐS_0, [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1O^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B_0,% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0O% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0R_0 0 , Oitalic_B0 , Oitalic_R0 ∼ I. At time t, we define the random variables for the state (StsubscriptS_tSitalic_t), Blue and Red’s observations (OtB,OtRsubscriptsuperscriptsubscriptsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1O [% rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B_t, [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0O [% rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R_tOitalic_Bitalic_t , Oitalic_Ritalic_t), and their actions (AtB,AtRsubscriptsuperscriptsubscriptsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1A [% rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B_t, [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0A [% rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R_tAitalic_Bitalic_t , Aitalic_Ritalic_t). The random variables can be defined inductively for all t: (St+1,Ot+1B,Ot+1R)∼(St,AtB,AtR)similar-tosubscript1subscriptsuperscript1subscriptsuperscript1subscriptsubscriptsuperscriptsubscriptsuperscript(S_t+1, [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1O^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B_t+1,% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0O% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0R_t+1) (S_t, [rgb]0,0,1 [% named]pgfstrokecolorrgb0,0,1A [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B_t, [rgb]0.8,0.0,0.0 [named% ]pgfstrokecolorrgb0.8,0.0,0.0A [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0R_t)( Sitalic_t + 1 , Oitalic_Bitalic_t + 1 , Oitalic_Ritalic_t + 1 ) ∼ P ( Sitalic_t , Aitalic_Bitalic_t , Aitalic_Ritalic_t ), where AtB∼πB(O0B,A0B,O1B,A1B…,OtB)similar-tosubscriptsuperscriptsuperscriptsubscriptsuperscript0subscriptsuperscript0subscriptsuperscript1subscriptsuperscript1…subscriptsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1A [% rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B_t [% rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π [rgb]% 0,0,1 [named]pgfstrokecolorrgb0,0,1B( [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1O [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1B_0, [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1A [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1B_0, [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1O [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1B_1, [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1A [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1B_1…, [rgb]% 0,0,1 [named]pgfstrokecolorrgb0,0,1O [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1B_t)Aitalic_Bitalic_t ∼ πitalic_B ( Oitalic_B0 , Aitalic_B0 , Oitalic_B1 , Aitalic_B1 … , Oitalic_Bitalic_t ) and, AtR∼πR(OtR)similar-tosubscriptsuperscriptsuperscriptsubscriptsuperscript [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% A [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0R_t [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0R ( [rgb]0.8,0.0,0.0% [named]pgfstrokecolorrgb0.8,0.0,0.0O [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R_t )Aitalic_Ritalic_t ∼ πitalic_R ( Oitalic_Ritalic_t ) if ⊛ ⊛ is ⊗tensor-product ⊗, else AtR=anullsubscriptsuperscriptsubscriptnull [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% A [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0R_t=a_nullAitalic_Ritalic_t = anull. Within our model, the usefulness of a protocol πBsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1Bπitalic_B is just the expected sum of the ℛUsesubscriptℛUseR_ RUse in an honest game: Use(πB):-[∑t=0H−1ℛUse(St,AtB,St+1)∣⊕,πB].:-Usesuperscriptdelimited-[]conditionalsuperscriptsubscript01subscriptℛUsesubscriptsubscriptsuperscriptsubscript1direct-sumsuperscriptUse( [rgb]0,0,1 [named]pgfstrokecolorrgb% 0,0,1π [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1% B) [ _t=0^H-1R_ % Use (S_t, [rgb]0,0,1 [named]pgfstrokecolor% rgb0,0,1A [rgb]0,0,1 [named]pgfstrokecolorrgb% 0,0,1B_t,S_t+1 ) , [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1π [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B ].Use ( πitalic_B ) :- blackboard_E [ ∑t = 0H - 1 RUse ( Sitalic_t , Aitalic_Bitalic_t , Sitalic_t + 1 ) ∣ ⊕ , πitalic_B ] . (1) Note that usefulness score is measured in ⊕direct-sum ⊕ cases, when πRsuperscript [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% π [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0Rπitalic_R has no effect, and therefore usefulness does not depend on πRsuperscript [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% π [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0Rπitalic_R. The safety score of a policy pair (πB,πR)superscriptsuperscript( [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B, [% rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0π^% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R% )( πitalic_B , πitalic_R ) is the probability that the game does not terminate in an unacceptable outcome (i.e. the final state is not in ⊥subscriptbottomS_ S⊥): Safety(πB,πR):-ℙ[SH∉⊥∣⊗,πB,πR].:-Safetysuperscriptsuperscriptℙdelimited-[]subscriptconditionalsubscriptbottomtensor-productsuperscriptsuperscriptSafety ( [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1π [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B, [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0R ) [S_H% _ , [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1π [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B, [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0R ].Safety ( πitalic_B , πitalic_R ) :- blackboard_P [ Sitalic_H ∉ S⊥ ∣ ⊗ , πitalic_B , πitalic_R ] . (2) Because the protocol designer gets to choose their action first, and without knowing the red team’s actions, we consider a leader-follower model: Blue chooses a policy, and then Red chooses a response that minimises safety. Proposition 3.2. For any AI-Control Game GG, there exist distance functions in ΠBsuperscriptΠ [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1 ^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1BΠitalic_B and ΠRsuperscriptΠ [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0RΠitalic_R such that: Safety(πB,⋅)Safetysuperscript⋅Safety( [rgb]0,0,1 [named]pgfstrokecolor% rgb0,0,1π [rgb]0,0,1 [named]pgfstrokecolorrgb% 0,0,1B,·)Safety ( πitalic_B , ⋅ ), Safety(⋅,πR)Safety⋅superscriptSafety(·, [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0R)Safety ( ⋅ , πitalic_R ), and Use(⋅)Use⋅Use(·)Use ( ⋅ ) are continuous in their domains.101010Example distance functions, based on a mapping from policies to vectors, are defined in appendix A. Proofs of all claims in the section can be found in appendix A, which details and utilises additional results about two mappings from Blue/Red policies to simplexes. Proposition 3.3. For all πBsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1Bπitalic_B, Safety(πB,⋅)Safetysuperscript⋅Safety( [rgb]0,0,1 [named]pgfstrokecolor% rgb0,0,1π [rgb]0,0,1 [named]pgfstrokecolorrgb% 0,0,1B,·)Safety ( πitalic_B , ⋅ ) attains its minimum at some πRsuperscript [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% π [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0Rπitalic_R, and minπRSafety(⋅,πR)subscriptsuperscriptSafety⋅superscript _ [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [named]pgfstrokecolor% rgb0.8,0.0,0.0RSafety(·, [rgb]0.8,0.0,0.0% [named]pgfstrokecolorrgb0.8,0.0,0.0π [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R)minitalic_πitalic_R Safety ( ⋅ , πitalic_R ) is continuous. Our goal is to find protocols that maximise worst-case safety and usefulness. This can be modelled as a multi-objective variant of a Stackelberg equilibrium: Definition 3.4. The pair (πB,πR)superscriptsuperscript( [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B, [% rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0π^% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R% )( πitalic_B , πitalic_R ) is in an AI-Control Equilibrium iff [minπRSafety(πB,πR),Use(πB)]TsuperscriptsubscriptsuperscriptSafetysuperscriptsuperscriptUsesuperscript[ _ [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [named]pgfstrokecolor% rgb0.8,0.0,0.0RSafety( [rgb]0,0,1 [% named]pgfstrokecolorrgb0,0,1π [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B, [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0R),Use( [rgb]% 0,0,1 [named]pgfstrokecolorrgb0,0,1π [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1B)]^T[ minitalic_πitalic_R Safety ( πitalic_B , πitalic_R ) , Use ( πitalic_B ) ]T is weakly Pareto optimal and πRsuperscript [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% π [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0Rπitalic_R minimises safety for πBsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1Bπitalic_B. The set of equilibrium points describes a necessary tradeoff between safety and usefulness, and is always non-empty. Proposition 3.5. Every GG has an AI-Control Equilibrium. 4 Example: Modelling the protocol With our general framework from section 3, we can define a formal model of the Trusted Monitoring protocol for the APPS backdoor setting, as outlined in Greenblatt et al. (2024). For the full definition please see appendix B; next we present the key points. We define a Trusted Monitoring AI-Control Game tuple TM=(,…,H)subscript…G_TM= (S,…,H )Gitalic_T M = ( S , … , H ) step-by-step. As always, whether the game is scheming (⊗tensor-product ⊗) or honest (⊕direct-sum ⊕) is determined exogenously and non-deterministically. If the game is ⊗tensor-product ⊗, for any given problem, Red can choose either for the untrusted model to attempt a backdoor or to let it submit a safe solution (R:-⟨⟩,⟨⟩:-superscriptdelimited-⟨⟩delimited-⟨⟩ [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% A [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0R \ Backdoor , Safe% \Aitalic_R :- ⟨ typewriter_Backdoor ⟩ , ⟨ typewriter_Safe ⟩ ). If the game is ⊕direct-sum ⊕, Red’s null action, to not backdoor, is chosen (anull=⟨⟩subscriptnulldelimited-⟨⟩a_null= Safe = ⟨ typewriter_Safe ⟩). Given a solution by the untrusted AI, Blue has three available actions: to ask the human to audit the solution, to defer to the trusted AI to rewrite the solution, or to run the solution immediately. Therefore define B:-⟨⟩,⟨⟩,⟨⟩:-superscriptdelimited-⟨⟩delimited-⟨⟩delimited-⟨⟩ [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1A% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B% \ Audit , Defer , % Run \Aitalic_B :- ⟨ typewriter_Audit ⟩ , ⟨ typewriter_Defer ⟩ , ⟨ typewriter_Run ⟩ . Apart from initial and terminal states, the state space, SS, consists of triples s=(m,c,b)s=(m,c,b)s = ( m , c , b ), where m represents the number of problems left to solve. The value of m is M (the total number of problems) at the start of the game, and decreases by 1111 at each timestep. The variable c represents the number of ‘audit coins’ left: the number of times the protocol can ask the human to audit a solution. At the start, c=C=0.02∗M0.02c=C=0.02*Mc = C = 0.02 ∗ M (for a 2%percent22\%2 % audit proportion) and then c decreases by 1111 each time the protocol chooses to audit. If the protocol attempts to audit when no audit coins are left, the code is run instead. The last variable, b, is a boolean that is Truetypewriter_True when the current problem is backdoored (when Red has chosen ⟨⟩delimited-⟨⟩ Backdoor ⟨ typewriter_Backdoor ⟩ in the previous timestep). In honest games (⊕direct-sum ⊕), where only usefulness matters to Blue, Red’s action is always substituted for anull=⟨⟩subscriptnulldelimited-⟨⟩a_null= Safe = ⟨ typewriter_Safe ⟩. Therefore, b is always Falsetypewriter_False. In honest games, Blue scores 1/M11/M1 / M usefulness points for each solution by the untrusted model it submits - giving a maximum score of 1111 across the whole episode. In scheming games (⊗tensor-product ⊗), where only safety matters to Blue, Red may have chosen ⟨⟩delimited-⟨⟩ Backdoor ⟨ typewriter_Backdoor ⟩ last turn, and so b might be Truetypewriter_True. If so, and Blue chooses to run the code, the safety property is violated, and the game ends with an unacceptable outcome. If Blue chooses to audit, we assume the human will always spot the backdoor, and the game ends safely (we enter a special ‘safe’ self-looping state). If Blue chooses to defer, then the game continues. The safety score for a policy pair is the probability that a scheming game ends without violating the safety property. Throughout the games, Red receives only the null observation (onullsubscriptnullo_nullonull),111111 In Greenblatt et al. (2024), the untrusted model receives only the programming problem with no additional information: we abstract this away by giving Red only the null observation. This implies that Red is equally likely to backdoor any problem. whereas at each turn Blue receives observations of m,cm,cm , c, and a trusted monitor score x, which can be seen as a noisy observation of b, drawn from a pair of distributions (‘safe’ and ‘backdoored’). Each distribution is a discretisation of the Beta distribution outlined in fig. 1(b). See appendix D for more details. 4.1 Evaluating Memoryless Protocols We generally model protocols (Blue policies) as functions from the observation-action history to a distribution over actions. However, all of the Trusted Monitoring policies in Greenblatt et al. (2024) are memoryless. We can restrict to a simple and interpretable subset of policies that generalises those by Greenblatt et al. (2024). Remember that, in our model, Blue’s observation is (m,c,x)(m,c,x)( m , c , x ), where x is a noisy observation of b, sampled from the Trusted Monitor. Define a set of memoryless threshold protocols, characterised by the false positive rates (FPRs) for ⟨⟩delimited-⟨⟩ Audit ⟨ typewriter_Audit ⟩ and ⟨⟩delimited-⟨⟩ Defer ⟨ typewriter_Defer ⟩ (qm,casubscriptsuperscriptq^a_m,cqitalic_aitalic_m , c and qm,cdsubscriptsuperscriptq^d_m,cqitalic_ditalic_m , c), for each m,cm,cm , c. Let g(q)g(q)g ( q ) be the threshold that gives an FPR of q: ℙ[xt>g(x)∣bt=]=qℙdelimited-[]subscriptconditionalsubscriptP [x_t>g(x) b_t= False ]=qblackboard_P [ xitalic_t > g ( x ) ∣ bitalic_t = typewriter_False ] = q.121212Since ⟨⟩delimited-⟨⟩ Audit ⟨ typewriter_Audit ⟩ and ⟨⟩delimited-⟨⟩ Run ⟨ typewriter_Run ⟩ are equivalent when c=00c=0c = 0, assume that Blue never attempts an audit. Then, define the policy: πB(m,c,x)=⟨⟩If x≥g(qm,ca)⟨⟩If g(qm,ca)>x≥g(qm,cd)⟨⟩otherwisesuperscriptcasesdelimited-⟨⟩If subscriptsuperscriptdelimited-⟨⟩If subscriptsuperscriptsubscriptsuperscriptdelimited-⟨⟩otherwise [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B(m,c,x)=% cases Audit &If x≥ g(q^a_m,c)\\ Defer &If g(q^a_m,c)>x≥ g(q^d_m,c)\\ Run &otherwise casesπitalic_B ( m , c , x ) = start_ROW start_CELL ⟨ typewriter_Audit ⟩ end_CELL start_CELL If x ≥ g ( qitalic_aitalic_m , c ) end_CELL end_ROW start_ROW start_CELL ⟨ typewriter_Defer ⟩ end_CELL start_CELL If g ( qitalic_aitalic_m , c ) > x ≥ g ( qitalic_ditalic_m , c ) end_CELL end_ROW start_ROW start_CELL ⟨ typewriter_Run ⟩ end_CELL start_CELL otherwise end_CELL end_ROW (3) This class of policies generalises the policies in Greenblatt et al. (2024), and is a dominant subset of memoryless policies. These policies’ safety and usefulness values can be computed in O(MC)O(MC)O ( M C ) multiplications by calculating the values of the following relation using dynamic programming. Proposition 4.1. For any β:-πR(⟨⟩∣onull):-superscriptconditionaldelimited-⟨⟩subscriptnull [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% β [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0R( Backdoor o_% null)β :- πitalic_R ( ⟨ typewriter_Backdoor ⟩ ∣ onull ), the safety of the memoryless threshold protocol πBsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1Bπitalic_B is given by the value of zMCsuperscriptsubscriptz_M^Czitalic_Mitalic_C, where λm,cisubscriptsuperscriptλ^i_m,cλitalic_iitalic_m , c depend only on qm,c⋅subscriptsuperscript⋅q^·_m,cq⋅italic_m , c and GG: 131313 See appendix C for the exact statements of λ coefficients. zmcsuperscriptsubscript z_m^czitalic_mitalic_c :-(λm,c1β+λm,c2)wm−1c+λm,c5(1−β)wm−1c−1:-absentsubscriptsuperscript1subscriptsuperscript2subscriptsuperscript1subscriptsuperscript51subscriptsuperscript11 (λ^1_m,c [rgb]0.8,0.0,0.0% [named]pgfstrokecolorrgb0.8,0.0,0.0β+λ^2_m,c% )w^c_m-1+λ^5_m,c(1- [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0β)w^c-1_m-1:- ( λ1italic_m , c β + λ2italic_m , c ) witalic_citalic_m - 1 + λ5italic_m , c ( 1 - β ) witalic_c - 1m - 1 +λm,c3β+λm,c4(1−β)subscriptsuperscript3subscriptsuperscript41 \ +λ^3_m,c [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0β+λ^4_m,c(1- [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0β)+ λ3italic_m , c β + λ4italic_m , c ( 1 - β ) where zm0=(1−β+λm,c6β)zm−10superscriptsubscript01subscriptsuperscript6superscriptsubscript10z_m^0= (1- [rgb]0.8,0.0,0.0 [named]pgfstrokecolor% rgb0.8,0.0,0.0β+λ^6_m,c [rgb]0.8,0.0,0.0% [named]pgfstrokecolorrgb0.8,0.0,0.0β )z_m-1^0zitalic_m0 = ( 1 - β + λ6italic_m , c β ) zitalic_m - 10 and z0c=1.superscriptsubscript01z_0^c=1.z0italic_c = 1 . Corollary 4.2. Expanding zMCsuperscriptsubscriptz_M^Czitalic_Mitalic_C gives a polynomial, p(β)p( [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0β)p ( β ), in β [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0β of degree M. For any fixed memoryless threshold policy, the optimal backdoor rate is β∗:-minβ∈[0,1]p(β):-superscriptsubscript01 [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% β^* _ [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0β∈[0,1]p( [rgb]0.8,0.0,0.0% [named]pgfstrokecolorrgb0.8,0.0,0.0β)β∗ :- minitalic_β ∈ [ 0 , 1 ] p ( β ). A similar expression for usefulness, which does not depend on β [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0β, can be found in appendix C. The safety of a finite mixture of memoryless threshold policies can be found by taking a weighted average. Although we implemented a root-finding algorithm based on corollary 4.2, we found that it was faster and more stable in practice to compute the values in fig. 2 using a combination of grid and ternary search. 5 Finding Pareto-optimal Protocols In many cases, AI-Control Games are useful for evaluating hand-picked protocols, for example the protocols explored by Greenblatt et al. (2024) that were found via trial and error. However, it may also be useful to utilise the AI-Control Games to find the Pareto-frontier of existing and new protocols. This is a multi-objective maximin optimisation problem over a large space. This section presents tools for finding Pareto-optimal protocols, beginning with a reduction from a multi-objective to a single-objective problem. Lemma 5.1. A Blue policy πBsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1Bπitalic_B is in an AI-Control Equilibrium if, and only if, for some w∈[0,1]01w∈[0,1]w ∈ [ 0 , 1 ], it maximises minπ∗R(wSafety(⋅,π∗R)+(1−w)Use(⋅))subscriptsubscriptsuperscriptSafety⋅subscriptsuperscript1Use⋅ _ [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [named]pgfstrokecolor% rgb0.8,0.0,0.0R_* (wSafety(·, [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0π % [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R_*% )+(1-w)Use(·) )minitalic_πitalic_R start_POSTSUBSCRIPT ∗ end_POSTSUBSCRIPT ( w Safety ( ⋅ , πitalic_R∗ ) + ( 1 - w ) Use ( ⋅ ) ). The proof uses the mappings from ΠBsuperscriptΠ [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1 ^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1BΠitalic_B and ΠRsuperscriptΠ [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0RΠitalic_R to two simplexes introduced in appendix A. Lemma 5.1 implies that each safety/usefulness point corresponding to an AI-Control Equilibrium can be found by equivalently solving a, possibly easier, single-objective optimisation problem. Next, we leverage this result to further reduce the game to a single-objective, partially observable stochastic game (ZSPOSG) (see Definition). Lemma 5.2. For every AI-Control Game GG, and weight w∈[0,1]01w∈[0,1]w ∈ [ 0 , 1 ], there is a single-objective, zero-sum, partially observable stochastic game GwsubscriptG_wGitalic_w with expected return (for Blue) VGwsubscriptsubscriptV_G_wVitalic_G start_POSTSUBSCRIPT w end_POSTSUBSCRIPT such that: ∀πB,πR:VGw(πB,πR)=Vw(πB,πR).:for-allsuperscriptsuperscriptsubscriptsubscriptsuperscriptsuperscriptsubscriptsuperscriptsuperscriptsuperscript∀ [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B, % [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0π^% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R% :V_G_w( [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1% π [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B,% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% π [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0R)=V^G_w( [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1π [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B, [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0R).∀ πitalic_B , πitalic_R : Vitalic_G start_POSTSUBSCRIPT w end_POSTSUBSCRIPT ( πitalic_B , πitalic_R ) = Vcaligraphic_Gitalic_w ( πitalic_B , πitalic_R ) . This proof is constructive and a full version can be found in appendix A. Intuitively, we create ZSPOSGs by adding a ⊛ ⊛ variable to each state. Each GwsubscriptG_wGitalic_w has reward RwsubscriptR_wRitalic_w, where R0subscript0R_0R0 rewards usefulness in ⊕direct-sum ⊕ games and R1subscript1R_1R1 rewards safety in ⊗tensor-product ⊗ games. Since πRsuperscript [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% π [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0Rπitalic_R has no effect on honest games, and therefore on usefulness, we can set Red to minimise the usefulness, making the game zero-sum without changing optima. Combined with lemma 5.1, this allows us to find any AI-Control Equilibrium by solving the right ZSPOSG. Theorem 5.3. A policy πBsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1Bπitalic_B is in an AI-Control Equilibrium if, and only if, for some ZSPOSG GwsubscriptG_wGitalic_w with value function VGwsubscriptsubscriptV_G_wVitalic_G start_POSTSUBSCRIPT w end_POSTSUBSCRIPT, πBsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1Bπitalic_B maximises minπR∈ΠRVGw(⋅,πR)subscriptsuperscriptsuperscriptΠsubscriptsubscript⋅superscript _ [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [named]pgfstrokecolor% rgb0.8,0.0,0.0R∈ [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0 [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0RV_G_w(·, [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0π % [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R)minitalic_πitalic_R ∈ Πitalic_R Vitalic_G start_POSTSUBSCRIPT w end_POSTSUBSCRIPT ( ⋅ , πitalic_R ) over ΠBsuperscriptΠ [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1 ^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1BΠitalic_B. Algorithm 1 Finding a Pareto frontier for AI-C Game 0: GG 1: for t∈0,1,…,k−101…1t∈\0,1,…,k-1\t ∈ 0 , 1 , … , k - 1 do 2: wt←FindNextWeight((πtB,πtR)t=0k−1)←subscriptFindNextWeightsubscriptsuperscriptsubscriptsuperscriptsubscriptsuperscript10w_t← FindNextWeight(\( [rgb]0,0,1 [% named]pgfstrokecolorrgb0,0,1π [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B_t, [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0π [rgb]0.8,0.0,0.0% [named]pgfstrokecolorrgb0.8,0.0,0.0R_t)\^k-1_t=0)witalic_t ← FindNextWeight ( ( πitalic_Bitalic_t , πitalic_Ritalic_t ) k - 1t = 0 ) 3: πtB,πtR←argmaxπB∈ΠBminπR∈ΠRVwt(πB,πR)←subscriptsuperscriptsubscriptsuperscriptsubscriptargmaxsuperscriptsuperscriptΠsubscriptsuperscriptsuperscriptΠsubscriptsubscriptsuperscriptsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B_t,% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% π [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0R_t← *argmax_ [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1π [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1B∈ [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1 [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1B _ [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0π % [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R∈% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0RV_w_t( [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1π [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B, [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0R)πitalic_Bitalic_t , πitalic_Ritalic_t ← argmaxitalic_πitalic_B ∈ Πitalic_B minitalic_πitalic_R ∈ Πitalic_R Vitalic_w start_POSTSUBSCRIPT t end_POSTSUBSCRIPT ( πitalic_B , πitalic_R ) 4: return (πtB,πtR)t=0k−1subscriptsuperscriptsubscriptsuperscriptsubscriptsuperscript10\( [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B_t,% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% π [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0R_t)\^k-1_t=0 ( πitalic_Bitalic_t , πitalic_Ritalic_t ) k - 1t = 0 Algorithm 1 uses theorem 5.3 to iteratively fill out the frontier by solving a series of single-objective maximin problems. Here FindNextWeight first returns w0=0subscript00w_0=0w0 = 0 and w1=1subscript11w_1=1w1 = 1, and then chooses weights that maximise the difference between known points on the Pareto frontier.141414 Let st:-Safety(πtB,πtR):-subscriptSafetysubscriptsuperscriptsubscriptsuperscripts_t ( [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1π [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B_t, [rgb]0.8,0.0,0.0 [named% ]pgfstrokecolorrgb0.8,0.0,0.0π [rgb]0.8,0.0,0.0 % [named]pgfstrokecolorrgb0.8,0.0,0.0R_t)sitalic_t :- Safety ( πitalic_Bitalic_t , πitalic_Ritalic_t ) and ut:-Use(πtB,πtR):-subscriptUsesubscriptsuperscriptsubscriptsuperscriptu_t ( [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1π [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B_t, [rgb]0.8,0.0,0.0 [named% ]pgfstrokecolorrgb0.8,0.0,0.0π [rgb]0.8,0.0,0.0 % [named]pgfstrokecolorrgb0.8,0.0,0.0R_t)uitalic_t :- Use ( πitalic_Bitalic_t , πitalic_Ritalic_t ). Order points by safety, then choose i=maxi(si−si+1)2+(ui−ui+1)2i= _i(s_i-s_i+1)^2+(u_i-u_i+1)^2i = maxitalic_i ( sitalic_i - sitalic_i + 1 )2 + ( uitalic_i - uitalic_i + 1 )2 and then w such that wsi+(1−w)ui=wsi+1+(1−w)ui+1subscript1subscriptsubscript11subscript1ws_i+(1-w)u_i=ws_i+1+(1-w)u_i+1w sitalic_i + ( 1 - w ) uitalic_i = w sitalic_i + 1 + ( 1 - w ) uitalic_i + 1. Line 3 of algorithm 1 requires finding the value of maxπBminπRVw(πB,πR)subscriptsuperscriptsubscriptsuperscriptsubscriptsuperscriptsuperscriptsuperscript _ [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B _% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% π [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0RV^G_w( [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1π [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B, [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0R)maxitalic_πitalic_B minitalic_πitalic_R Vcaligraphic_Gitalic_w ( πitalic_B , πitalic_R ). Although ZSPOSGs are previously studied, existing solvers are incompatible with a memoryless Red team (see section 7). In general, the memorylessness of Red means their decision rules at each timestep are interdependent, making problem decomposition difficult. Further, imperfect recall prohibits the equivalence of behavioural and mixed stochasticities: Proposition 5.4. There exists a GwsubscriptG_wGitalic_w such that Red’s behavioural and mixed policies would not be equivalent and maxπB∈ΠBminπR∈ΠRVw(πB,πR)<minπR∈ΠRmaxπB∈ΠBVw(πB,πR)subscriptsuperscriptsuperscriptΠsubscriptsuperscriptsuperscriptΠsubscriptsuperscriptsuperscriptsuperscriptsubscriptsuperscriptsuperscriptΠsubscriptsuperscriptsuperscriptΠsubscriptsuperscriptsuperscriptsuperscript _ [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B∈% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1 % [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B _ [% rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0π^% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R% ∈ [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0 [rgb]0.8,0.0,0.0 [named]pgfstrokecolor% rgb0.8,0.0,0.0RV^G_w( [rgb]0,0,1 [% named]pgfstrokecolorrgb0,0,1π [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B, [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0R)< _ [rgb]0.8,0.0,0.0% [named]pgfstrokecolorrgb0.8,0.0,0.0π [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R∈ % [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0 ^% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R% _ [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B∈% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1 % [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1BV^G% _w( [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B, [% rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0π^% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R% )maxitalic_πitalic_B ∈ Πitalic_B minitalic_πitalic_R ∈ Πitalic_R Vcaligraphic_Gitalic_w ( πitalic_B , πitalic_R ) < minitalic_πitalic_R ∈ Πitalic_R maxitalic_πitalic_B ∈ Πitalic_B Vcaligraphic_Gitalic_w ( πitalic_B , πitalic_R ) See section A.3 for a proof by example. Algorithm 2 Adapted Double Oracle Algorithm 0: GwsubscriptG_wGitalic_w, π1Bsubscriptsuperscript1 [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B_1πitalic_B1, π1Rsubscriptsuperscript1 [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% π [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0R_1πitalic_R1, ϵitalic-ϵεϵ 1: Initialize t←0←0t← 0t ← 0 2: while t=0∨v¯t−v¯t>ϵ0subscript¯subscript¯italic-ϵt=0 v_t- v_t> = 0 ∨ over¯ start_ARG v end_ARGt - under¯ start_ARG v end_ARGt > ϵ do 3: t←t+1←1t← t+1t ← t + 1 4: ,←SolveNFG(Vw(πiB,πjR)1≤i,j≤t)←SolveNFGsubscriptsubscriptsuperscriptsubscriptsuperscriptsubscriptsuperscriptformulae-sequence1p,q← SolveNFG(\V^G_w( % [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π [rgb]% 0,0,1 [named]pgfstrokecolorrgb0,0,1B_i, [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0π % [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R_j% )\_1≤ i,j≤ t)p , q ← SolveNFG ( Vcaligraphic_Gitalic_w ( πitalic_Bitalic_i , πitalic_Ritalic_j ) 1 ≤ i , j ≤ t ) 5: σtB←Mix(,πiBi=1t)←subscriptsuperscriptMixsuperscriptsubscriptsubscriptsuperscript1 [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1σ^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B_t% ← Mix(p,\ [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1π [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B_i\_i=1^t)σitalic_Bitalic_t ← Mix ( p , πitalic_Bitalic_i i = 1t ), σtR←Mix(,πjRj=1t)←subscriptsuperscriptMixsuperscriptsubscriptsubscriptsuperscript1 [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% σ [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0R_t← Mix(q,\ [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0π % [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R_j% \_j=1^t)σitalic_Ritalic_t ← Mix ( q , πitalic_Ritalic_j j = 1t ) 6: πt+1B,v¯t←(arg)maxπBVw(πB,σtR)←subscriptsuperscript1subscript¯subscriptsuperscriptsubscriptsuperscriptsuperscriptsubscriptsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B_t+1,% v_t←( ) _ [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1π [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1BV^G_w( [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1π [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1B, [rgb]0.8,0.0,0.0% [named]pgfstrokecolorrgb0.8,0.0,0.0σ [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R_t)πitalic_Bitalic_t + 1 , over¯ start_ARG v end_ARGt ← ( arg ) maxitalic_πitalic_B Vcaligraphic_Gitalic_w ( πitalic_B , σitalic_Ritalic_t ) 7: πt+1R,v¯t←(arg)minπRVw(σtB,πR)←subscriptsuperscript1subscript¯subscriptsuperscriptsubscriptsuperscriptsubscriptsuperscriptsuperscript [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% π [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0R_t+1, v_t←( ) _ [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0π % [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0RV^% G_w( [rgb]0,0,1 [named]pgfstrokecolorrgb% 0,0,1σ [rgb]0,0,1 [named]pgfstrokecolorrgb% 0,0,1B_t, [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0R)πitalic_Ritalic_t + 1 , under¯ start_ARG v end_ARGt ← ( arg ) minitalic_πitalic_R Vcaligraphic_Gitalic_w ( σitalic_Bitalic_t , πitalic_R ) 8: return σtBsubscriptsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1σ^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B_tσitalic_Bitalic_t, πt+1Rsubscriptsuperscript1 [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% π [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0R_t+1πitalic_Ritalic_t + 1 Despite this inequality, algorithm 2 adapts the Double Oracle Algorithm from McMahan, Gordon, and Blum (2003) to find a maximin for behaviourally stochastic Red policies. The function Mix simply creates a policy mixture from a vector and a finite set of policies. Finding pp and qq is equivalent to finding a Nash equilibrium in a zero-sum finite normal-form game. The values v¯tsubscript¯ v_tunder¯ start_ARG v end_ARGt and v¯tsubscript¯ v_tover¯ start_ARG v end_ARGt lowerbound and upperbound the maximin value of the game. Proposition 5.5. For ϵ>0italic-ϵ0ε>0ϵ > 0, Algorithm 2 returns an ϵitalic-ϵεϵ-equilibrium in GwsubscriptG_wGitalic_w in finite time. Further, for all t: v¯t≤v∗:-maxπ∗B∈ΠBminπ∗R∈ΠRVw(π∗B,π∗R)≤v¯tsubscript¯superscript:-subscriptsubscriptsuperscriptsuperscriptΠsubscriptsubscriptsuperscriptsuperscriptΠsubscriptsuperscriptsubscriptsuperscriptsubscriptsuperscriptsubscript¯ v_t≤ v^* _ [rgb]0,0,1 [named% ]pgfstrokecolorrgb0,0,1π [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B_*∈ [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1 [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B _ [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0π [rgb]0.8,0.0,0.0% [named]pgfstrokecolorrgb0.8,0.0,0.0R_*∈ [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0 % [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0RV^% G_w( [rgb]0,0,1 [named]pgfstrokecolorrgb% 0,0,1π [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1% B_*, [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [named]pgfstrokecolor% rgb0.8,0.0,0.0R_*)≤ v_tunder¯ start_ARG v end_ARGt ≤ v∗ :- maxitalic_πitalic_B start_POSTSUBSCRIPT ∗ ∈ Πitalic_B end_POSTSUBSCRIPT minitalic_πitalic_R start_POSTSUBSCRIPT ∗ ∈ Πitalic_R end_POSTSUBSCRIPT Vcaligraphic_Gitalic_w ( πitalic_B∗ , πitalic_R∗ ) ≤ over¯ start_ARG v end_ARGt If ϵ=0italic-ϵ0ε=0ϵ = 0 then (σtB,πtR)subscriptsuperscriptsubscriptsuperscript( [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1σ^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B_t,% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% π [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0R_t)( σitalic_Bitalic_t , πitalic_Ritalic_t ) is an AI-Control Equilibrium. The proof is shown in appendix A and utilises Adam et al. (2021, Theorem 1.1) by reducing GwsubscriptG_wGitalic_w to a ‘continuous game’. The primary change to the algorithm is the choice to return πt+1Rsubscriptsuperscript1 [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% π [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0R_t+1πitalic_Ritalic_t + 1 which is behaviourally but not mixed stochastic, and gives the worst-case safety for σtBsubscriptsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1σ^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B_tσitalic_Bitalic_t which is a lower-bound for v∗superscriptv^*v∗. Best Response Oracles. Lines 6 and 7 of the algorithm require finding best-responses. Finding the best-response πBsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1Bπitalic_B to a (finite) policy mixture σRsuperscript [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% σ [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0Rσitalic_R is manageable: it’s equivalent to solving a Partially-observable Markov Decision Process, which is a well-studied problem (Kaelbling, Littman, and Cassandra 1998; Shani, Pineau, and Kaplow 2013). However, finding a best-response πRsuperscript [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% π [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0Rπitalic_R to σBsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1σ^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1Bσitalic_B is, in general, more challenging: because Red is memoryless, the problem is difficult to decompose. Fortunately, in the Trusted Monitoring example described in section 2, Red’s policy space is 1-dimensional, and can be searched easily. 6 Example: Results Figure 2: The safety and usefulness frontiers for a range of Trusted Monitor protocol types across the ‘original’ and ‘weak’ games. Each point on the grid corresponds to the exact safety and usefulness of a proposed blue team protocol against an approximately optimal red team response (minimal safety). The larger circles correspond to the seven special cases of Trusted Monitoring studied by Greenblatt et al. (2024) and remaining points are new to this paper. Each ‘Audit FPR’ line plots a single, fixed audit FPR, across fixed defer FPRs in 0.00,0.01,…,1.000.000.01…1.00\0.00,0.01,…,1.00\ 0.00 , 0.01 , … , 1.00 . ‘Mixture over Fixed FPRs’ shows the optimal protocol frontier found by algorithm 1 with a fixed-FPR oracle, and all points represent deterministic policies unless marked by a cross. In this section, we present example novel evaluations of protocols: evaluating existing and novel protocols across existing and novel settings. We also present the first Pareto frontiers for a subset of the protocol space. All results presented concern only memoryless protocols: namely, protocols that decide on the current action based only on the last observation. These can be evaluated efficiently using proposition 4.1. We leave the analysis of memoryful protocols, those whose actions depend on past observations, to future work. See appendix D for experimental details. Existing Protocols. All of the protocols studied in Greenblatt et al. (2024) can be expressed as memoryless protocols: Never Audit always submit U’s solution without audit. Trusted Model Only always submits T’s solution. Upfront Auditing audits the first 2% of solutions from U and, when there is no audit budget left, submit the rest without checking. Trusted Monitoring (Audit FPR = 0.02) audits every solution from U whose TM score is above a threshold chosen to give a 2% FPR and also defer all solutions above (0%, 5%, 10%, 20%, 50%). We leave evaluation of Greenblatt et al.’s other protocols to future work. Each protocol above is a special case of Trusted Monitoring with a fixed audit FPR of 0%, 2% or 100%, combined with fixed defer FPRs. By representing these protocols in our formalisation, we can easily evaluate their safety and usefulness scores, even in the novel scenario containing the weak trusted monitor; we illustrate these results by coloured points in fig. 2. Evaluating New Protocols. Through a process of proposing and efficiently evaluating new protocols, enabled by our formalisation, we extend Greenblatt et al.’s findings to new protocols and settings. As an example, we show that raising the audit FPR above 2% Pareto improves upon the protocols from Greenblatt et al. (2024). Choosing an audit FPR of 2% means that, in expectation, the 2% allowance of human labour will be used up at the end of the episode if no backdoor is submitted. However, we find that choosing a higher audit rate can be beneficial, despite increasing the probability that the audit budget is exhausted early. Choosing an audit FPR of 3%percent33\%3 % gives a small Pareto improvement in the ‘original’ TM case and an audit FPR of 5%percent55\%5 %, gives a large Pareto improvement in the ‘weak’ TM case. Demonstrating a Frontier. Using algorithm 1 and algorithm 2, we present the first approximate Pareto-frontier for a policy subset: mixtures of protocols with fixed audit and defer rates. We apply algorithm 2 using a Blue best-response algorithm based on recursive grid search (over values of audit and defer FPR) and a Red best-response algorithm using grid search followed by ternary search (over values of β [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0β). We find that, in the ‘original’ case the closest of Greenblatt et al.’s policies is 0.00420.00420.00420.0042 from the Pareto front, and for most weights a deterministic policy is optimal. However, in the ‘weak’ case, they are at least ≥0.0200absent0.0200≥ 0.0200≥ 0.0200 from the Pareto front and mixed protocols make up most of the frontier. 6.1 Assumptions and Trends Formalising an AI-Control Game requires making a number of modelling assumptions; the results below outline how varying each assumption impacts the safety and usefulness scores for a given blue-team policy. Figure 3: Impact on safety and usefulness (y-axis) by four different modelling assumptions (x-axis): Auditing Budget, Number of Programming Problems, and Human Auditor False Negative / Positive Rate. All plots show a fixed defer FPR of 0.1. Auditing Budget. The auditing budget corresponds to the amount of human labour invested in overseeing the protocol. Protocols that utilise increases to the audit budget, get significantly safer and more useful as the budget increases. especially when the budget is small. Number of Problems. For fixed FPRs, safety and usefulness are not significantly affected by the number of problems per episode. As the number of opportunities for Red to backdoor increases, so too does the number of audit coins. Human Auditor FPR/FNR. In the default Trusted Monitoring game, Greenblatt et al. assume that no mistakes are made by the human auditor; we can extend the model to relax this assumption. Increasing the human FNR decreases safety almost linearly in the worst case, without impacting usefulness. False positives increase safety up to a point determined by the audit threshold, but decrease usefulness more rapidly the more the blue team audits. 7 Related Work Although the application of a formal game-theoretic framework to Greenblatt et al. (2024)’s AI Control framework is novel, and the resulting type of game is unique, there are analogues to existing work. In this section, we compare AI-Control Games to existing models. The AI-Control Games studied are most similar to the work of Chang, Erera, and White (2015), who model a security problem for food manufacturing. Like AI-Control Games, they consider a leader-follower equilibrium, where the leader tries to maximise productivity and minimise risk, and the follower tries to hide an attack. To select the leader strategy against multiple objectives, they use a genetic algorithm (as do Lins et al. (2013)). Their model differs in its factored state space, infinite time horizon, and the lack of ‘scheming/honest’ game modes. However, the largest difference is their consideration of a memoryful-follower. Given our reduction in section 5, methods for finding equilibria in ZS-POSGs are particularly relevant. However, we find that our requirement for a memoryful-leader, memoryless-follower equilibrium precludes the use of many existing solvers. For example, Delage et al. (2023), which finds Nash equilibria in ZS-POSGs using a dynamic programming approach that solves subgames for each timestep from H down to 1111. Other works attempt to solve subclasses of ZS-POSGs, with ‘public observations’ (Horák and Bošanský 2019) or ‘on-sided partial-observability’ (Zheng, Jung, and Lin 2022; Horák et al. 2023). AI-Control Games do not in-general fit into these subclasses, and each of these papers considers two memoryful players. Since in some cases it is possible to find an AI-Control Equilibrium by first finding a mixed Nash equilibrium (proposition 5.5), it may be possible to augment the algorithm by Hansen and Feng (2000), which searches for two mixed strategies over a finite set of memoryful policies. AI-Control Games, and especially the Trusted Monitoring case study we present, are closely related to Stackelberg security games: two-player games where the leader must choose a random strategy to cover a set of targets using a limited set of resources, and the follower tries to attack an uncovered target (see (Sinha et al. 2018) for an overview). Unlike Trusted Monitoring, these are one-round games and much literature focuses on scalably solving large-scale problems against scheduling constraints. Although some security games model imperfect information (Bier 2007; Nguyen et al. 2014) or multiple objectives (Brown et al. 2012), most are single-objective and perfect-information. Trusted Monitoring without defer is similar to the Inspection Games studied by Avenhaus, Von Stengel, and Zamir (2002, p. 1957): games where the defender receives a noisy observation, and has to decide whether to inspect. However, these games are one-round only. Recursive Inspection Games (Dresher 1962) are multi-round, but differ from our case in the memoryfullness of Red. 8 Future Work In this work, we introduced a novel model for analysing the safety and usefulness of AI-Control protocols. We used our model and solution techniques to evaluate a much broader range of protocols and settings, but there is significant room to apply them further. In the Trusted Monitoring setting, future work can utilise algorithm 2 with POMDP-based best-response oracles to explore the space of memory-dependent protocols. Our model of the Trusted Monitoring protocol is also generalisable to any deployment setting using a question-answer format and any ‘unacceptable outcome’ that is a function of U’s answer. Future work can explore such settings by empirically studying the interaction of the untrusted AI and the trusted monitor to generate distributions like those in fig. 1(b) and analyse the resulting frontiers. References Adam et al. (2021) Adam, L.; Horčík, R.; Kasl, T.; and Kroupa, T. 2021. Double Oracle Algorithm for Computing Equilibria in Continuous Games. Proceedings of the AAAI Conference on Artificial Intelligence, 35(6): 5070–5077. Section: AAAI Technical Track on Game Theory and Economic Paradigms. Adept AI (2024) Adept AI. 2024. Adept: Useful General Intelligence. Aumann, Hart, and Perry (1997) Aumann, R. J.; Hart, S.; and Perry, M. 1997. The absent-minded driver. Games and Economic Behavior, 20(1): 102–116. Publisher: Elsevier. Avenhaus, Von Stengel, and Zamir (2002) Avenhaus, R.; Von Stengel, B.; and Zamir, S. 2002. Chapter 51 Inspection games. In Handbook of Game Theory with Economic Applications, volume 3, 1947–1987. Elsevier. Bier (2007) Bier, V. M. 2007. Choosing What to Protect. Risk Analysis, 27(3): 607–620. Brown et al. (2012) Brown, M.; An, B.; Kiekintveld, C.; Ordóñez, F.; and Tambe, M. 2012. Multi-objective optimization for security games. In Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems, AAMAS ’12, 863–870. ISBN 978-0-9817381-2-3. Casper et al. (2024) Casper, S.; Schulze, L.; Patel, O.; and Hadfield-Menell, D. 2024. Defending Against Unforeseen Failure Modes with Latent Adversarial Training. ArXiv:2403.05030 [cs]. Chang, Erera, and White (2015) Chang, Y.; Erera, A. L.; and White, C. C. 2015. A leader–follower partially observed, multiobjective Markov game. Annals of Operations Research, 235(1): 103–128. Clymer et al. (2024) Clymer, J.; Gabrieli, N.; Krueger, D.; and Larsen, T. 2024. Safety Cases: How to Justify the Safety of Advanced AI Systems. ArXiv:2403.10462 [cs]. Conitzer and Sandholm (2006) Conitzer, V.; and Sandholm, T. 2006. Computing the optimal strategy to commit to. In Proceedings of the 7th ACM conference on Electronic commerce, EC ’06, 82–90. New York, NY, USA: Association for Computing Machinery. ISBN 978-1-59593-236-5. Delage et al. (2023) Delage, A.; Buffet, O.; Dibangoye, J. S.; and Saffidine, A. 2023. HSVI Can Solve Zero-Sum Partially Observable Stochastic Games. Dynamic Games and Applications. Dresher (1962) Dresher, M. 1962. A Sampling Inspection Problem in Arms Control Agreements: A Game-Theoretic Analysis. Technical report, RAND Corporation. Emery-Montemerlo et al. (2004) Emery-Montemerlo, R.; Gordon, G.; Schneider, J.; and Thrun, S. 2004. Approximate solutions for partially observable stochastic games with common payoffs. In Proceedings of the Third International Joint Conference on Autonomous Agents and Multiagent Systems, 2004., 136–143. Greenblatt et al. (2024) Greenblatt, R.; Shlegeris, B.; Sachan, K.; and Roger, F. 2024. AI Control: Improving Safety Despite Intentional Subversion. In Proceedings of the 41st International Conference on Machine Learning, 16295–16336. PMLR. ISSN: 2640-3498. Hansen and Feng (2000) Hansen, E. A.; and Feng, Z. 2000. Dynamic programming for POMDPs using a factored state representation. In Proceedings of the Fifth International Conference on Artificial Intelligence Planning Systems, AIPS’00, 130–139. Breckenridge, CO, USA: AAAI Press. ISBN 978-1-57735-111-5. Hendrycks et al. (2021) Hendrycks, D.; Basart, S.; Kadavath, S.; Mazeika, M.; Arora, A.; Guo, E.; Burns, C.; Puranik, S.; He, H.; Song, D.; and Steinhardt, J. 2021. Measuring Coding Challenge Competence With APPS. Horák and Bošanský (2019) Horák, K.; and Bošanský, B. 2019. Solving Partially Observable Stochastic Games with Public Observations. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01): 2029–2036. Number: 01. Horák et al. (2023) Horák, K.; Bošanský, B.; Kovařík, V.; and Kiekintveld, C. 2023. Solving zero-sum one-sided partially observable stochastic games. Artificial Intelligence, 316: 103838. Kaelbling, Littman, and Cassandra (1998) Kaelbling, L. P.; Littman, M. L.; and Cassandra, A. R. 1998. Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101(1-2): 99–134. Kuhn (1953) Kuhn, H. W. 1953. Extensive games and the problem of information. Contributions to the Theory of Games, 2(28): 193–216. Lins et al. (2013) Lins, I. D.; Rêgo, L. C.; Moura, M. d. C.; and Droguett, E. L. 2013. Selection of security system design via games of imperfect information and multi-objective genetic algorithm. Reliability Engineering & System Safety, 112: 59–66. MacDiarmid et al. (2024) MacDiarmid, M.; Maxwell, T.; Schiefer, N.; Mu, J.; Kaplan, J.; Duvenaud, D.; Bowman, S.; Tamkin, A.; Perez, E.; Sharma, M.; Denison, C.; and Hubinger, E. 2024. Simple probes can catch sleeper agents. Maschler, Solan, and Zamir (2013) Maschler, M.; Solan, E.; and Zamir, S. 2013. Behavior strategies and kuhn’s theorem. In Game theory, 219–250. Cambridge: Cambridge University Press. McMahan, Gordon, and Blum (2003) McMahan, H. B.; Gordon, G. J.; and Blum, A. 2003. Planning in the presence of cost functions controlled by an adversary. In Proceedings of the 20th International Conference on Machine Learning (ICML-03), 536–543. Miettinen (1998) Miettinen, K. 1998. Nonlinear Multiobjective Optimization. New York, NY, UNITED STATES: Springer. ISBN 978-1-4615-5563-6. Nguyen et al. (2014) Nguyen, T. H.; Yadav, A.; An, B.; Tambe, M.; and Boutilier, C. 2014. Regret-based optimization and preference elicitation for stackelberg security games with uncertainty: 28th AAAI Conference on Artificial Intelligence, AAAI 2014, 26th Innovative Applications of Artificial Intelligence Conference, IAAI 2014 and the 5th Symposium on Educational Advances in Artificial Intelligence, EAAI 2014. Proceedings of the 28th AAAI Conference on Artificial Intelligence and the 26th Innovative Applications of Artificial Intelligence Conference and the 5th Symposium on Educational Advances in Artificial Intelligence, 756–762. Publisher: AI Access Foundation. OpenAI (2023) OpenAI. 2023. ChatGPT plugins. Osborne and Rubinstein (2006) Osborne, M. J.; and Rubinstein, A. 2006. A course in game theory. Cambridge, Mass.: MIT Press, 12. print edition. ISBN 978-0-262-65040-3. Shah et al. (2023) Shah, R.; Montixi, Q. F.; Pour, S.; Tagade, A.; and Rando, J. 2023. Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulation. Shani, Pineau, and Kaplow (2013) Shani, G.; Pineau, J.; and Kaplow, R. 2013. A survey of point-based POMDP solvers. Autonomous Agents and Multi-Agent Systems, 27(1): 1–51. Sinha et al. (2018) Sinha, A.; Fang, F.; An, B.; Kiekintveld, C.; and Tambe, M. 2018. Stackelberg Security Games: Looking Beyond a Decade of Success. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, 5494–5501. Stockholm, Sweden: International Joint Conferences on Artificial Intelligence Organization. ISBN 978-0-9992411-2-7. Zheng, Jung, and Lin (2022) Zheng, W.; Jung, T.; and Lin, H. 2022. The Stackelberg equilibrium for one-sided zero-sum partially observable stochastic games. Automatica, 140: 110231. Appendix A General results and proofs This appendix presents proofs of the results from section 3 and section 5 that apply to all AI-Control Games. To do so, we first introduce a useful lemma which maps the optimisation problem into a vector space. A.1 Mapping to vector space We will define a mapping from ΠBsuperscriptΠ [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1 ^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1BΠitalic_B to a simplex X, and another from ΠRsuperscriptΠ [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0RΠitalic_R to a simplex Y. Then, we will show that these mappings can be used to calculate safety and usefulness. We begin by defining the simplex Y and a mapping μR:ΠR→Y:superscript→superscriptΠ [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% μ [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0R: [rgb]0.8,0.0,0.0 [named]pgfstrokecolor% rgb0.8,0.0,0.0 [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0R→ Yμitalic_R : Πitalic_R → Y. Recall that: ΠR:-πR∣πR:ΩR→R:-superscriptΠconditional-setsuperscript:superscript→superscriptΩsuperscript [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0R \ [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0R [rgb]0.8,0.0,0.0% [named]pgfstrokecolorrgb0.8,0.0,0.0π [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R: [% rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0 ^% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% R→ [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0A [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0R\Πitalic_R :- πitalic_R ∣ πitalic_R : Ωitalic_R → Aitalic_R . Define n:-|R||ΩR|:-superscriptsuperscriptsuperscriptΩn | [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0A [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0R|^| [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0 [rgb]0.8,0.0,0.0% [named]pgfstrokecolorrgb0.8,0.0,0.0R|n :- | Aitalic_R || Ω start_POSTSUPERSCRIPT R | end_POSTSUPERSCRIPT and enumerate the n observation-action pairs such that each dimension in ℝnsuperscriptℝ R^nblackboard_Rn corresponds to one pair. Consider the simplex Y⊂ℝnsuperscriptℝY⊂ R^nY ⊂ blackboard_Rn such that all values are ≥0absent0≥ 0≥ 0 and, for each observation, the sum across all actions is 1111. Then, define the mapping such that μR(πR)=superscriptsuperscript [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% μ [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0R( [rgb]0.8,0.0,0.0 [named]pgfstrokecolor% rgb0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0R)=yμitalic_R ( πitalic_R ) = y if, and only if, for all j, πR(aR∣oR)=yjsuperscriptconditionalsuperscriptsuperscriptsubscript [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% π [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0R( [rgb]0.8,0.0,0.0 [named]pgfstrokecolor% rgb0.8,0.0,0.0a [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0R [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0o [rgb]0.8,0.0,0.0% [named]pgfstrokecolorrgb0.8,0.0,0.0R)=y_jπitalic_R ( aitalic_R ∣ oitalic_R ) = yitalic_j for j’th observation-action pair (oR,aR)superscriptsuperscript( [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% o [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0R, [rgb]0.8,0.0,0.0 [named]pgfstrokecolor% rgb0.8,0.0,0.0a [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0R)( oitalic_R , aitalic_R ). Note that μRsuperscript [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% μ [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0Rμitalic_R is bijective. Blue’s mapping is more complex. Define the finite set of Blue observation-action histories as ℋB:-ΩB×⋃i=0H−1(B×ΩB)i:-superscriptℋsuperscriptΩsuperscriptsubscript01superscriptsuperscriptsuperscriptΩ [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1H% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B% × _i=0^H-1 ( [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1A [rgb]0,0,1 [named% ]pgfstrokecolorrgb0,0,1B× [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1 [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B )^iHitalic_B :- Ωitalic_B × ⋃i = 0H - 1 ( Aitalic_B × Ωitalic_B )i. Since the sets ℋBsuperscriptℋ [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1H% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1BHitalic_B and Bsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1A% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1BAitalic_B are finite, we could use a similar strategy to the one we used for Y to define finite-dimensional vector space X that represents ΠB:-πB∣πB:ℋB→B:-superscriptΠconditional-setsuperscript:superscript→superscriptℋsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1 ^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B \% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B % [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π % [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B: [rgb]% 0,0,1 [named]pgfstrokecolorrgb0,0,1H [rgb% ]0,0,1 [named]pgfstrokecolorrgb0,0,1B→ [rgb]% 0,0,1 [named]pgfstrokecolorrgb0,0,1A [rgb% ]0,0,1 [named]pgfstrokecolorrgb0,0,1B\Πitalic_B :- πitalic_B ∣ πitalic_B : Hitalic_B → Aitalic_B . However, an alternative approach gives a useful linearity. Because Blue has perfect recall, any behaviourally stochastic policy has an equivalent representation as a mixed strategy over the finite set of deterministic policies. This is the reverse direction of Kuhn’s Theorem (Kuhn 1953), sometimes stated informally with ‘and vice versa’. This exact result is mentioned in Maschler, Solan, and Zamir (2013, Theorem 6.26 on p. 242) as part of a more general proof about infinite-horizon games, and an informal proof is given by Osborne and Rubinstein (2006, p. 214). The other direction (which will also be useful), is given explicitly in Maschler, Solan, and Zamir (2013, Theorem 6.15 on p. 232): Theorem (Kuhn (1953)). In every game in extensive form, if player i has perfect recall, then for every mixed strategy of player i there exists an equivalent behavior strategy. Note that the equivalence stated here is in terms of the probability of reaching any vertex in an extensive-form tree which is more than sufficient for our purposes (Maschler, Solan, and Zamir 2013, see Definition 6.5 on p. 223). We will use the fact that an AI-Control game is a finite extensive form game, and the ‘vice versa’ of Kuhn’s Thoerem to give a mapping from behavioural to mixed policies. Let ΠdetBsubscriptsuperscriptΠdet [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1 ^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B_% detΠitalic_Bdet be the set of deterministic policies for Blue. Then, for each behaviorally stochastic πBsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1Bπitalic_B, there is an equivalent mixture over deterministic policies: σB∈Δ(ΠdetB)superscriptΔsubscriptsuperscriptΠdet [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1σ^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B∈ (% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1 ^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B_% det)σitalic_B ∈ Δ ( Πitalic_Bdet ). Let m:-|ΠdetB|=|AB||ℋB|:-subscriptsuperscriptΠdetsuperscriptsuperscriptsuperscriptℋm | [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B_% det|=| [rgb]0,0,1 [named]pgfstrokecolorrgb% 0,0,1A [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B% |^| [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1% H [rgb]0,0,1 [named]pgfstrokecolorrgb% 0,0,1B|m :- | Πitalic_Bdet | = | Aitalic_B || H start_POSTSUPERSCRIPT B | end_POSTSUPERSCRIPT and consider the simplex X⊂ℝmsuperscriptℝX⊂ R^mX ⊂ blackboard_Rm: X:-∣∈X,≥0,∑i=1mvi=1:-conditional-setformulae-sequenceformulae-sequence0superscriptsubscript1subscript1X \x ∈ X,x≥ 0, _i=1^mv% _i=1 \X :- x ∣ x ∈ X , x ≥ 0 , ∑i = 1m vitalic_i = 1 Enumerate the m deterministic policies for Blue and assign each policy to a dimension in ℝmsuperscriptℝ R^mblackboard_Rm. Let πiBsubscriptsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B_iπitalic_Bitalic_i be the iiith deterministic policy for Blue. Define the mapping μB(πB):-:-superscriptsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1μ^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B( [% rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π [rgb]% 0,0,1 [named]pgfstrokecolorrgb0,0,1B) μitalic_B ( πitalic_B ) :- x where xi:-σB(πiB):-subscriptsuperscriptsubscriptsuperscriptx_i [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1% σ [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B% ( [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B_i)xitalic_i :- σitalic_B ( πitalic_Bitalic_i ), ie. the probability that πiBsubscriptsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B_iπitalic_Bitalic_i is chosen from the mixture σBsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1σ^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1Bσitalic_B that is equivalent to πBsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1Bπitalic_B. Further, note that by the main direction of Kuhn’s theorem, for any mixed stochastic policy (σBsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1σ^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1Bσitalic_B), there is also a behaviourally stochastic policy (πBsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1Bπitalic_B) that is equivalent but it’s not immediate that μBsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1μ^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1Bμitalic_B is bijective, since there may be two mixed stochastic policies that correspond to the same behaviourally stochastic policy. From here on, we drop the explicit use of μBsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1μ^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1Bμitalic_B and μRsuperscript [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% μ [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0Rμitalic_R, and instead use πB:-μB(πB):-subscriptsuperscriptsuperscriptsuperscriptx_ [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1% π [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1μ% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B(% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π % [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B)xitalic_πitalic_B :- μitalic_B ( πitalic_B ) and πR:-μR(πR):-subscriptsuperscriptsuperscriptsuperscriptx_ [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [named]pgfstrokecolor% rgb0.8,0.0,0.0R [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0μ [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0R( [rgb]0.8,0.0,0.0% [named]pgfstrokecolorrgb0.8,0.0,0.0π [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R)xitalic_πitalic_R :- μitalic_R ( πitalic_R ). Lemma A.1. For any GG, and any full-length trajectory hℎh, there is a function ph:X×Y→[0,1]:subscriptℎ→01p_h:X× Y→[0,1]pitalic_h : X × Y → [ 0 , 1 ] that is linear in it’s first argument, continuous in its second, and satisfies the following for all πB,πRsuperscriptsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B, [% rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0π^% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0Rπitalic_B , πitalic_R: ph(πB,πR)subscriptℎsubscriptsuperscriptsubscriptsuperscript p_h(x_ [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1π [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B,y_ [rgb]0.8,0.0,0.0% [named]pgfstrokecolorrgb0.8,0.0,0.0π [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R)pitalic_h ( xitalic_πitalic_B , yitalic_πitalic_R ) =ℙ[h∣⊗,πB,πR]absentℙdelimited-[]conditionalℎtensor-productsuperscriptsuperscript =P [h , [rgb]0,0,1 [% named]pgfstrokecolorrgb0,0,1π [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B, [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0R ]= blackboard_P [ h ∣ ⊗ , πitalic_B , πitalic_R ] ph(πB,πnullR)subscriptℎsubscriptsuperscriptsubscriptsubscriptsuperscriptnull p_h(x_ [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1π [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B,y_ [rgb]0.8,0.0,0.0% [named]pgfstrokecolorrgb0.8,0.0,0.0π [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R_% null)pitalic_h ( xitalic_πitalic_B , yitalic_πitalic_R start_POSTSUBSCRIPT null end_POSTSUBSCRIPT ) =ℙ[h∣⊕,πB,πR]absentℙdelimited-[]conditionalℎdirect-sumsuperscriptsuperscript =P [h , [rgb]0,0,1 [% named]pgfstrokecolorrgb0,0,1π [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B, [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0R ]= blackboard_P [ h ∣ ⊕ , πitalic_B , πitalic_R ] where πnullRsubscriptsuperscriptnull [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% π [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0R_nullπitalic_Rnull is the policy that always chooses anullsubscriptnulla_nullanull. Proof. Let ℋHH be the set of all possible full-length trajectories (or joint-state-observation-action histories): h:-(⊛,s0,o0B,o0R,a0B,a0R,s1,…,oH−1R,aH−1B,aH−1R,sH):-ℎ⊛subscript0subscriptsuperscript0subscriptsuperscript0subscriptsuperscript0subscriptsuperscript0subscript1…subscriptsuperscript1subscriptsuperscript1subscriptsuperscript1subscripth ( ,s_0, [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1o [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B_0, [rgb]0.8,0.0,0.0 [named% ]pgfstrokecolorrgb0.8,0.0,0.0o [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0R_0, [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1a [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1B_0, [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0a [% rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R_0,% s_1,…, [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0o [rgb]0.8,0.0,0.0 [named]pgfstrokecolor% rgb0.8,0.0,0.0R_H-1, [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1a [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B_H-1, [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0a [rgb]0.8,0.0,0.0% [named]pgfstrokecolorrgb0.8,0.0,0.0R_H-1,s_H)h :- ( ⊛ , s0 , oitalic_B0 , oitalic_R0 , aitalic_B0 , aitalic_R0 , s1 , … , oitalic_Ritalic_H - 1 , aitalic_Bitalic_H - 1 , aitalic_Ritalic_H - 1 , sitalic_H ) We will define phsubscriptℎp_hpitalic_h such that, for all πB,πRsuperscriptsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B, [% rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0π^% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0Rπitalic_B , πitalic_R ℙ[h∣⊗,πB,πR]ℙdelimited-[]conditionalℎtensor-productsuperscriptsuperscript [h , [rgb]0,0,1 [% named]pgfstrokecolorrgb0,0,1π [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B, [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0R ]blackboard_P [ h ∣ ⊗ , πitalic_B , πitalic_R ] =ph(πB,πR)absentsubscriptℎsubscriptsuperscriptsubscriptsuperscript =p_h(x_ [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1π [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B,y_ [rgb]0.8,0.0,0.0% [named]pgfstrokecolorrgb0.8,0.0,0.0π [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R)= pitalic_h ( xitalic_πitalic_B , yitalic_πitalic_R ) Then, we can write the probability of each history explicitly, in terms of πBsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1Bπitalic_B and πRsuperscript [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% π [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0Rπitalic_R: ℙ _P [h∣⊗,πB,πR]delimited-[]conditionalℎtensor-productsuperscriptsuperscript [h , [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1π [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B, [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0R ][ h ∣ ⊗ , πitalic_B , πitalic_R ] =ℐ(s0,o0B,o0R)⋅πB(a0B∣o0B)⋅πR(a0R∣o0R)absent⋅ℐsubscript0subscriptsuperscript0subscriptsuperscript0superscriptconditionalsubscriptsuperscript0subscriptsuperscript0superscriptconditionalsubscriptsuperscript0subscriptsuperscript0 =I(s_0, [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1o [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B_0, [rgb]0.8,0.0,0.0 [named% ]pgfstrokecolorrgb0.8,0.0,0.0o [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0R_0)· [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1π [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1B( [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1a [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1B_0 [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1o [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1B_0)· [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0π % [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R(% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0a% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0R_0 [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0o [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0R_0)= I ( s0 , oitalic_B0 , oitalic_R0 ) ⋅ πitalic_B ( aitalic_B0 ∣ oitalic_B0 ) ⋅ πitalic_R ( aitalic_R0 ∣ oitalic_R0 ) ⋅(s1,o1B,o1R∣s0,a0B,a0R)⋅absentsubscript1subscriptsuperscript1conditionalsubscriptsuperscript1subscript0subscriptsuperscript0subscriptsuperscript0 ·P(s_1, [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1o [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B_1, [rgb]0.8,0.0,0.0 [named% ]pgfstrokecolorrgb0.8,0.0,0.0o [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0R_1 s_0, [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1a [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1B_0, [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0a [% rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R_0)⋅ P ( s1 , oitalic_B1 , oitalic_R1 ∣ s0 , aitalic_B0 , aitalic_R0 ) … … ⋅(sH,⋅,⋅∣sH−1,aH−1B,aH−1R) ·P(s_H,·,· s_H-1, [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1a [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1B_H-1, [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0a [% rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R_H-1)⋅ P ( sitalic_H , ⋅ , ⋅ ∣ sitalic_H - 1 , aitalic_Bitalic_H - 1 , aitalic_Ritalic_H - 1 ) =ch⋅∏t=0H−1πB(atB∣o0:tB,a0:t−1B)πR(atR∣otR)absent⋅subscriptℎsuperscriptsubscriptproduct01superscriptconditionalsubscriptsuperscriptsubscriptsuperscript:0subscriptsuperscript:01superscriptconditionalsubscriptsuperscriptsubscriptsuperscript =c_h· _t=0^H-1 [rgb]0,0,1 [named% ]pgfstrokecolorrgb0,0,1π [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B( [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1a [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B_t [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1o [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B_0:t, [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1a [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B_0:t-1) [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0π [rgb]0.8,0.0,0.0% [named]pgfstrokecolorrgb0.8,0.0,0.0R( [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0a [% rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R_t% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0o [rgb]0.8,0.0,0.0 [named]pgfstrokecolor% rgb0.8,0.0,0.0R_t)= citalic_h ⋅ ∏t = 0H - 1 πitalic_B ( aitalic_Bitalic_t ∣ oitalic_B0 : t , aitalic_B0 : t - 1 ) πitalic_R ( aitalic_Ritalic_t ∣ oitalic_Ritalic_t ) Here o0:tBsubscriptsuperscript:0 [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1o [% rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B_0:toitalic_B0 : t denotes the tuple (o0B,…,otB)subscriptsuperscript0…subscriptsuperscript( [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1o % [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B_0,…,% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1o [% rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B_t)( oitalic_B0 , … , oitalic_Bitalic_t ) , and similar for a0:t−1Bsubscriptsuperscript:01 [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1a [% rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B_0:t-1aitalic_B0 : t - 1 etc. The value chsubscriptℎc_hcitalic_h is a constant with respect to xx and yy and depends only on the trajectory and the AI-Control Game GG. Using mixed-behavioural equivalence, let σBsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1σ^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1Bσitalic_B be the mixture of deterministic strategies equivalent to πBsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1Bπitalic_B, then we can split the Blue probabilities: ℙ _P [h∣⊗,πB,πR]delimited-[]conditionalℎtensor-productsuperscriptsuperscript [h , [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1π [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B, [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0R ][ h ∣ ⊗ , πitalic_B , πitalic_R ] = == ch∑i=1mσB(πiB)⋅∏t=0H−1πiB(atB∣o0:tB,a0:t−1B)πR(atR∣otR)subscriptℎsuperscriptsubscript1⋅superscriptsubscriptsuperscriptsuperscriptsubscriptproduct01subscriptsuperscriptconditionalsubscriptsuperscriptsubscriptsuperscript:0subscriptsuperscript:01superscriptconditionalsubscriptsuperscriptsubscriptsuperscript c_h _i=1^m [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1σ [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B( [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1π [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B_i)· _t=0^H-1 [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1π [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1B_i( [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1a [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1B_t [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1o [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1B_0:t, [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1a [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1B_0:t-1) [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0π % [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R(% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0a% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0R_t [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0o [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0R_t)citalic_h ∑i = 1m σitalic_B ( πitalic_Bitalic_i ) ⋅ ∏t = 0H - 1 πitalic_Bitalic_i ( aitalic_Bitalic_t ∣ oitalic_B0 : t , aitalic_B0 : t - 1 ) πitalic_R ( aitalic_Ritalic_t ∣ oitalic_Ritalic_t ) And finally, let :-πB:-subscriptsuperscriptx _ [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1π [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1Bx :- xitalic_πitalic_B, note that xi=σB(πiB)subscriptsuperscriptsubscriptsuperscriptx_i= [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1σ% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B(% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π % [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B_i)xitalic_i = σitalic_B ( πitalic_Bitalic_i ), write out the expression in terms of xisubscriptx_ixitalic_i, and yjsubscripty_jyitalic_j, and call this expression phsubscriptℎp_hpitalic_h: ph(,):-ch⋅(∑i=1mxifi(h))⋅(∏t=0H−1yj(otR,atR)):-subscriptℎ⋅subscriptℎsuperscriptsubscript1subscriptsubscriptℎsuperscriptsubscriptproduct01subscriptsubscriptsuperscriptsubscriptsuperscript p_h(x,y) c_h· ( _i=1^% mx_if_i(h) )· ( _t=0^H-1y_j( [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0o [% rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R_t,% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% a [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0R_t) )pitalic_h ( x , y ) :- citalic_h ⋅ ( ∑i = 1m xitalic_i fitalic_i ( h ) ) ⋅ ( ∏t = 0H - 1 yitalic_j ( oitalic_R start_POSTSUBSCRIPT t , aitalic_Ritalic_t ) end_POSTSUBSCRIPT ) (4) Here fi(h)subscriptℎf_i(h)fitalic_i ( h ) is either 00, if πiBsubscriptsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B_iπitalic_Bitalic_i is not consistent with hℎh, or 1111 if it is. Clearly, the probability (ℙ[h|…]ℙdelimited-[]conditionalℎ…P[h|…]blackboard_P [ h | … ]) of getting some trajectory hℎh in a scheming game, given policies πBsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1Bπitalic_B and πRsuperscript [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% π [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0Rπitalic_R is simply ph(πB,πRp_h(x_ [rgb]0,0,1 [named]pgfstrokecolorrgb% 0,0,1π [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1% B,y_ [rgb]0.8,0.0,0.0 [named]pgfstrokecolor% rgb0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0Rpitalic_h ( xitalic_πitalic_B , yitalic_πitalic_R). The probability of getting some trajectory hℎh in an honest game is the same as the probability in the scheming game with the Red policy πnullRsubscriptsuperscriptnull [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% π [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0R_nullπitalic_Rnull that always chooses the anullsubscriptnulla_nullanull action. (Although the rewards in an episode will be different in ⊕direct-sum ⊕ versus ⊗tensor-product ⊗ modes, these aren’t included in hℎh.) This can be written as ph(,null)subscriptℎsubscriptnullp_h(x,y_null)pitalic_h ( x , ynull ), where nullsubscriptnully_nullynull corresponds to the πnullRsubscriptsuperscriptnull [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% π [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0R_nullπitalic_Rnull. ℙ[h∣⊕,πB]ℙdelimited-[]conditionalℎdirect-sumsuperscript [h , [rgb]0,0,1 [named% ]pgfstrokecolorrgb0,0,1π [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B ]blackboard_P [ h ∣ ⊕ , πitalic_B ] =ℙ[h∣⊗,πB,πnullR]absentℙdelimited-[]conditionalℎtensor-productsuperscriptsubscriptsuperscriptnull =P [h , [rgb]0,0,1 [% named]pgfstrokecolorrgb0,0,1π [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B, [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0R_null ]= blackboard_P [ h ∣ ⊗ , πitalic_B , πitalic_Rnull ] = == ch∑i=1mxi⋅fi(h)∏t=0H−1(null)j(otR,atR)subscriptℎsuperscriptsubscript1⋅subscriptsubscriptℎsuperscriptsubscriptproduct01subscriptsubscriptnullsubscriptsuperscriptsubscriptsuperscript c_h _i=1^mx_i· f_i(h) _t=0^H-1(% y_null)_j( [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0o [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0R_t, [rgb]0.8,0.0,0.0% [named]pgfstrokecolorrgb0.8,0.0,0.0a [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R_t)citalic_h ∑i = 1m xitalic_i ⋅ fitalic_i ( h ) ∏t = 0H - 1 ( ynull )j ( oitalic_R start_POSTSUBSCRIPT t , aitalic_Ritalic_t ) end_POSTSUBSCRIPT = == chnull∑i=1mxi⋅fi(h)subscriptsuperscriptnullℎsuperscriptsubscript1⋅subscriptsubscriptℎ c^null_h _i=1^mx_i· f_i(h)cnullitalic_h ∑i = 1m xitalic_i ⋅ fitalic_i ( h ) Here, we rewrite the constant chnullsubscriptsuperscriptnullℎc^null_hcnullitalic_h to be c(h)ℎc(h)c ( h ), if all the Red actions in hℎh are anullsubscriptnulla_nullanull, and 00 otherwise. We have shown that the probability of a given trajectory can be written in terms of ph(πB,πR)subscriptℎsubscriptsuperscriptsubscriptsuperscriptp_h(x_ [rgb]0,0,1 [named]pgfstrokecolorrgb% 0,0,1π [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1% B,y_ [rgb]0.8,0.0,0.0 [named]pgfstrokecolor% rgb0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0R)pitalic_h ( xitalic_πitalic_B , yitalic_πitalic_R ), for scheming games, and ph(πB,null)subscriptℎsubscriptsuperscriptsubscriptnullp_h(x_ [rgb]0,0,1 [named]pgfstrokecolorrgb% 0,0,1π [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1% B,y_null)pitalic_h ( xitalic_πitalic_B , ynull ) for honest games. It remains to note that ph(,)subscriptℎp_h(x,y)pitalic_h ( x , y ) is straightforwardly linear in xx and continuous in yy by eq. 4. ∎ Using lemma A.1, we can rewrite safety and usefulness as explicit expressions with some useful properties: Corollary A.2. For any AI-Control Game GG, with simplex X and simplex Y defined in lemma A.1, there is a function u:X→ℝ:→ℝu:X→ Ru : X → blackboard_R such that u(πB)=Use(πB)subscriptsuperscriptUsesuperscriptu(x_ [rgb]0,0,1 [named]pgfstrokecolorrgb% 0,0,1π [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1% B)=Use( [rgb]0,0,1 [named]pgfstrokecolor% rgb0,0,1π [rgb]0,0,1 [named]pgfstrokecolorrgb% 0,0,1B)u ( xitalic_πitalic_B ) = Use ( πitalic_B ) and a function s:X×Y→ℝ:→ℝs:X× Y→ Rs : X × Y → blackboard_R such that s(πB,πR)=Safety(πB,πR)subscriptsuperscriptsubscriptsuperscriptSafetysuperscriptsuperscripts(x_ [rgb]0,0,1 [named]pgfstrokecolorrgb% 0,0,1π [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1% B,y_ [rgb]0.8,0.0,0.0 [named]pgfstrokecolor% rgb0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0R)=Safety( [rgb]% 0,0,1 [named]pgfstrokecolorrgb0,0,1π [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1B, [rgb]0.8,0.0,0.0% [named]pgfstrokecolorrgb0.8,0.0,0.0π [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R)s ( xitalic_πitalic_B , yitalic_πitalic_R ) = Safety ( πitalic_B , πitalic_R ). Further 1. The function u is linear, and therefore uniformly continuous, in xx. 2. The function s is linear in xx for fixed yy, and s is continuous in yy for fixed xx. 3. The minimiser to mins(,)subscript _ys(x,y)minbold_y s ( x , y ) always exists. Further, the function g():-mins(,):-subscriptg(x) _ys(x,y)g ( x ) :- minbold_y s ( x , y ) is continuous everywhere and concave. 4. For all w∈[0,1]01w∈[0,1]w ∈ [ 0 , 1 ], the function vw():-wg()+(1−w)u():-subscript1v_w(x) wg(x)+(1-w)u(x)vitalic_w ( x ) :- w g ( x ) + ( 1 - w ) u ( x ) is uniformly continuous, and therefore has maximal values. Proof. First, we can define the functions s and u: since the set of trajectories is finite, we can rewrite the safety and usefulness functions by summing over the trajectories. The safety equation can then be rewritten as: Safety(πB,πR)Safetysuperscriptsuperscript ( [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1π [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B, [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0R )Safety ( πitalic_B , πitalic_R ) :-ℙ[SH∉⊥∣⊗,πB,πR]:-absentℙdelimited-[]subscriptconditionalsubscriptbottomtensor-productsuperscriptsuperscript [S_H _ ,% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B, [% rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0π^% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R% ]:- blackboard_P [ Sitalic_H ∉ S⊥ ∣ ⊗ , πitalic_B , πitalic_R ] =∑h∈ℋ∖ℋ⊥ℙ[h∣⊗,πB,πR]absentsubscriptℎℋsubscriptℋbottomℙdelimited-[]conditionalℎtensor-productsuperscriptsuperscript = _h _ P[h% , [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1% π [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B,% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% π [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0R]= ∑h ∈ H ∖ H start_POSTSUBSCRIPT ⊥ end_POSTSUBSCRIPT blackboard_P [ h ∣ ⊗ , πitalic_B , πitalic_R ] =∑h∈ℋ∖ℋ⊥ph(πB,πR)absentsubscriptℎℋsubscriptℋbottomsubscriptℎsubscriptsuperscriptsubscriptsuperscript = _h _ p_h(x% _ [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B,% y_ [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [named]pgfstrokecolor% rgb0.8,0.0,0.0R)= ∑h ∈ H ∖ H start_POSTSUBSCRIPT ⊥ end_POSTSUBSCRIPT pitalic_h ( xitalic_πitalic_B , yitalic_πitalic_R ) =∑h∈ℋ∖ℋ⊥ch∑i=1mxi⋅fi(h)∏t=0H−1yj(otR,atR)absentsubscriptℎℋsubscriptℋbottomsubscriptℎsuperscriptsubscript1⋅subscriptsubscriptℎsuperscriptsubscriptproduct01subscriptsubscriptsuperscriptsubscriptsuperscript = _h _ c_h _i=1% ^mx_i· f_i(h) _t=0^H-1y_j( [rgb]0.8,0.0,0.0% [named]pgfstrokecolorrgb0.8,0.0,0.0o [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R_t,% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0a% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0R_t)= ∑h ∈ H ∖ H start_POSTSUBSCRIPT ⊥ end_POSTSUBSCRIPT citalic_h ∑i = 1m xitalic_i ⋅ fitalic_i ( h ) ∏t = 0H - 1 yitalic_j ( oitalic_R start_POSTSUBSCRIPT t , aitalic_Ritalic_t ) end_POSTSUBSCRIPT =:s(,) =:s(x,y)= : s ( x , y ) Where the last line gives the definition for our new function s. We can similarly derive and define an expression for u: Use(πB)Usesuperscript ( [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1π [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B)Use ( πitalic_B ) :-[∑t=0HℛUse(St,AtB,St+1)∣⊕,πB]:-absentdelimited-[]conditionalsuperscriptsubscript0subscriptℛUsesubscriptsubscriptsuperscriptsubscript1direct-sumsuperscript [ _t=0^HR_% (S_t, [rgb]0,0,1 [named% ]pgfstrokecolorrgb0,0,1A [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B_t,S_t+1 ) , [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1π [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1B ]:- blackboard_E [ ∑t = 0H RUse ( Sitalic_t , Aitalic_Bitalic_t , Sitalic_t + 1 ) ∣ ⊕ , πitalic_B ] =∑h∈ℋr(h)ℙ[h∣⊕,πB]absentsubscriptℎℋℎℙdelimited-[]conditionalℎdirect-sumsuperscript = _h r(h)P[h , [rgb]% 0,0,1 [named]pgfstrokecolorrgb0,0,1π [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1B]= ∑h ∈ H r ( h ) blackboard_P [ h ∣ ⊕ , πitalic_B ] =∑h∈ℋr(h)chnull∑i=1mxi⋅fi(h)absentsubscriptℎℋℎsubscriptsuperscriptnullℎsuperscriptsubscript1⋅subscriptsubscriptℎ = _h r(h)c^null_h _i=1^mx_i% · f_i(h)= ∑h ∈ H r ( h ) cnullitalic_h ∑i = 1m xitalic_i ⋅ fitalic_i ( h ) =:u() =:u(x)= : u ( x ) We prove each of the remaining statements one-by-one. 1. That u(⋅)⋅u(·)u ( ⋅ ) is linear over xx follows from the definition of phsubscriptℎp_hpitalic_h, and of linearity of s in phsubscriptℎp_hpitalic_h. 2. That, for fixed xx, s(,⋅)⋅s(x,·)s ( x , ⋅ ) is continuous over Y is immediate from the continuity of each phsubscriptℎp_hpitalic_h in each yjsubscripty_jyitalic_j, and of linearity of s in phsubscriptℎp_hpitalic_h. That, for fixed yy, s(⋅,)⋅s(·,y)s ( ⋅ , y ) is linear over xx follows straightforwardly from the linearity of each phsubscriptℎp_hpitalic_h in xx, and of linearity of s in phsubscriptℎp_hpitalic_h. 3. The minimiser ∗subscripty_*y∗ of mins(,)subscript _ys(x,y)minbold_y s ( x , y ) must exist because s(,⋅)⋅s(x,·)s ( x , ⋅ ) is continuous in Y for fixed xx, and Y is compact, by the extreme value theorem. Showing that mins(⋅,)subscript⋅ _ys(·,y)minbold_y s ( ⋅ , y ) is continuous everywhere xx is a little more involved. Note that s can be rewritten as s(,)=⋅()⋅s(x,y)=x·v(y)s ( x , y ) = x ⋅ v ( y ), where ():-[f1(),…,fm()]T:-superscriptsubscript1…subscriptv(y) [f_1(y),…,f_m(y)% ]^Tv ( y ) :- [ f1 ( y ) , … , fitalic_m ( y ) ]T, and each fi()subscriptf_i(y)fitalic_i ( y ) is continuous in yy. By continuity and compactness, d:-max∥v()∥:-subscriptd _y v(y) :- maxbold_y ∥ v ( y ) ∥ exists. For all ϵ>0italic-ϵ0ε>0ϵ > 0, define δ:-ϵ2d:-italic-ϵ2δ ε2dδ :- divide start_ARG ϵ end_ARG start_ARG 2 d end_ARG. Now, for any yy, if ∥−0∥<δdelimited-∥subscript0 -x_0 <δ∥ x - x0 ∥ < δ then |s(,)−s(0,)|=|(−0)⋅()|≤subscript0⋅subscript0absent s(x,y)-s(x_0,y) = (% x-x_0)·v(y) ≤| s ( x , y ) - s ( x0 , y ) | = | ( x - x0 ) ⋅ v ( y ) | ≤ …∥−0∥∥()∥≤δ⋅ϵ2d<ϵ…delimited-∥subscript0delimited-∥⋅italic-ϵ2italic-ϵ… -x_0 (y) % ≤δ· ε2d<ε… ∥ x - x0 ∥ ∥ v ( y ) ∥ ≤ δ ⋅ divide start_ARG ϵ end_ARG start_ARG 2 d end_ARG < ϵ To show continuity of mins(⋅,)subscript⋅ _ys(·,y)minbold_y s ( ⋅ , y ), consider any ϵitalic-ϵεϵ and again set δ:-ϵ2d:-italic-ϵ2δ ε2dδ :- divide start_ARG ϵ end_ARG start_ARG 2 d end_ARG. Consider any 0subscript0x_0x0 and 1subscript1x_1x1 minimised by 0subscript0y_0y0 and 1subscript1y_1y1 such that ∥0−1∥≤δdelimited-∥subscript0subscript1 _0-x_1 ≤δ∥ x0 - x1 ∥ ≤ δ, and without loss of generality assume s(1,1)≥s(0,0)subscript1subscript1subscript0subscript0s(x_1,y_1)≥ s(x_0,y_0)s ( x1 , y1 ) ≥ s ( x0 , y0 ), then, since mins(1,)≤s(1,0)subscriptsubscript1subscript1subscript0 _ys(x_1,y)≤ s(x_1,y_% 0)minbold_y s ( x1 , y ) ≤ s ( x1 , y0 ): mins(1,)−mins(0,)≤s(1,0)−s(0,0)<ϵsubscriptsubscript1subscriptsubscript0subscript1subscript0subscript0subscript0italic-ϵ _ys(x_1,y)- _ys(x_0% ,y)≤ s(x_1,y_0)-s(x_0,y% _0)< _y s ( x1 , y ) - minbold_y s ( x0 , y ) ≤ s ( x1 , y0 ) - s ( x0 , y0 ) < ϵ This demonstrates that mins(⋅,)subscript⋅ _ys(·,y)minbold_y s ( ⋅ , y ) is continuous over X.Finally, the concavity claim, that g(λ1+(1−λ)0)≥λg(1)+(1−λ)g(0)subscript11subscript0subscript11subscript0g ( _1+(1-λ)x_0 )≥λ g(% x_1)+(1-λ)g(x_0)g ( λ x1 + ( 1 - λ ) x0 ) ≥ λ g ( x1 ) + ( 1 - λ ) g ( x0 ), follows straightforwardly from the definition of g, the linearity of s(⋅,)⋅s(·,y)s ( ⋅ , y ), and the fact that min(λs(1,)+(1−λ)s(0,))≥min1λs(1,1)+min0(1−λ)s(0,0)subscriptsubscript11subscript0subscriptsubscript1subscript1subscript1subscriptsubscript01subscript0subscript0 _y (λ s(x_1,y)+(1-λ)s(% x_0,y) )≥ _y_1λ s(x% _1,y_1)+ _y_0(1-λ)s(x_0,% y_0)minbold_y ( λ s ( x1 , y ) + ( 1 - λ ) s ( x0 , y ) ) ≥ minbold_y start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT λ s ( x1 , y1 ) + minbold_y start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 1 - λ ) s ( x0 , y0 ). 4. Straightforwardly from the linearity of u()u(x)u ( x ) and the uniform continuity of mins(⋅,)subscript⋅ _ys(·,y)minbold_y s ( ⋅ , y ), we have that vw():-wg()+(1−w)u():-subscript1v_w(x) wg(x)+(1-w)u(x)vitalic_w ( x ) :- w g ( x ) + ( 1 - w ) u ( x ) is continuous over X. That maximal values exist follows from this continuity, the compactness of X, and the extreme value theorem. ∎ Lemma A.3. Let ϕ():-[g(),u()]T:-bold-ϕsuperscript φ(x) [g(x),u(x)]^Titalic_ϕ ( x ) :- [ g ( x ) , u ( x ) ]T. The vector xx is weakly Pareto optimal (no ′superscript′x x′ has strictly better g and u) if, and only if, ∃w∈[0,1]01∃ w∈[0,1]∃ w ∈ [ 0 , 1 ] such that ∈argmax′vw(′)subscriptargmaxsuperscript′subscriptsuperscript′x∈ *argmax_x v_w(x^% )x ∈ argmaxbold_x′ vitalic_w ( x′ ). Proof. The ‘if’ statement is straightforward: suppose some ′superscript′x x′ is such that x1′>x1∗subscriptsuperscript′1subscriptsuperscript1x _1>x^*_1x′1 > x∗1 and x2′>x2∗subscriptsuperscript′2subscriptsuperscript2x _2>x^*_2x′2 > x∗2. Clearly, for all w, vw(′)>vw(∗)subscriptsuperscript′subscriptsuperscriptv_w(x )>v_w(x^*)vitalic_w ( x′ ) > vitalic_w ( x∗ ) and therefore no w can exist for which ∗superscriptx^*x∗ is (jointly) optimal. For the ‘only if’ statement, we build up Miettinen (1998, Theorem 3.1.4 on p. 79), which says that every Pareto optimal solution to a concave multi-objective optimisation problem is optimal for some weighting.151515The original statement refers to convex minimisation, rather than concave maximisation. We have already shown g is concave, and since u is linear it is trivially concave. Therefore, the theorem holds and any Pareto optimal ∗superscriptx^*x∗ is also optimal for some vwsubscriptv_wvitalic_w. It remains to show that in our case, if not in general, any weak Pareto optimum that is also optimal for some w. The proof relies on the convexity of X, the concavity of g, and the linearity of u. To do this, we show that any weak Pareto optimum that is not a Pareto optimum must have either maximal safety or maximal usefulness. Suppose ∗superscriptx^*x∗ is a weak Pareto optimum but not a Pareto optimum, and for reduction assume that ∗superscriptx^*x∗ has neither optimal safety nor optimal usefulness. If there is both (1) a point 0subscript0x_0x0 with equal g and better u and (2) a point 1subscript1x_1x1 with equal u but better g, then we can derive a point 0+12subscript0subscript12 x_0+x_12divide start_ARG x0 + x1 end_ARG start_ARG 2 end_ARG that dominates ∗superscriptx^*x∗.161616 By convexity of X, the point 0+12subscript0subscript12 x_0+x_12divide start_ARG x0 + x1 end_ARG start_ARG 2 end_ARG is in X and by linearity of u and concavity of g, dominates ∗superscriptx^*x∗: u(0+12)=u(0)+u(∗)2>u(∗)subscript0subscript12superscript0superscript2superscriptu( x_0+x_12)= u(x^0)+u(x% ^*)2>u(x^*)u ( divide start_ARG x0 + x1 end_ARG start_ARG 2 end_ARG ) = divide start_ARG u ( x0 ) + u ( x∗ ) end_ARG start_ARG 2 end_ARG > u ( x∗ ) and g(0+12)≥g(0)+g(∗)2>g(∗)subscript0subscript12subscript0superscript2superscriptg( x_0+x_12)≥ g(x_0)+g(% x^*)2>g(x^*)g ( divide start_ARG x0 + x1 end_ARG start_ARG 2 end_ARG ) ≥ divide start_ARG g ( x0 ) + g ( x∗ ) end_ARG start_ARG 2 end_ARG > g ( x∗ ). The definition of weak Pareto optimality only implies that (1) or (2) holds. However, it can be shown that if (1) holds and ∗superscriptx^*x∗ does not maximise g, then (2) also holds or ∗superscriptx^*x∗ is otherwise dominated.171717 If ∗superscriptx^*x∗ does not maximise u, let 0subscript0x_0x0 be any point with higher u and by assumption worse-or-equal g. If g(0)=g(∗)subscript0superscriptg(x_0)=g(x^*)g ( x0 ) = g ( x∗ ), then (2) also holds, otherwise, g(0)<g(∗)subscript0superscriptg(x_0)<g(x^*)g ( x0 ) < g ( x∗ ). Choose λ:-g(∗)−g(0)g(1)−g(0):-superscriptsubscript0subscript1subscript0λ g(x^*)-g(x_0)g(x_1)-g(% x_0)λ :- divide start_ARG g ( x∗ ) - g ( x0 ) end_ARG start_ARG g ( x1 ) - g ( x0 ) end_ARG such that g(∗)=λg(1)+(1−λ)g(0)superscriptsubscript11subscript0g(x^*)=λ g(x_1)+(1-λ)g(x_0)g ( x∗ ) = λ g ( x1 ) + ( 1 - λ ) g ( x0 ). Clearly 0<λ<1010<λ<10 < λ < 1. Let λ=λ1+(1−λ)0subscriptsubscript11subscript0x_λ= _1+(1-λ)x_0xitalic_λ = λ x1 + ( 1 - λ ) x0. We can see that u(λ)=λu(1)+(1−λ)u(0)=λu(∗)+(1−λ)u(0)>u(∗)subscriptsubscript11subscript0superscript1subscript0superscriptu(x_λ)=λ u(x_1)+(1-λ)u(x_0)% =λ u(x^*)+(1-λ)u(x_0)>u(x^*)u ( xitalic_λ ) = λ u ( x1 ) + ( 1 - λ ) u ( x0 ) = λ u ( x∗ ) + ( 1 - λ ) u ( x0 ) > u ( x∗ ). Further, by concavity of g, g(λ)≥λg(1)+(1−λ)g(0)=g(∗)subscriptsubscript11subscript0superscriptg(x_λ)≥λ g(x_1)+(1-λ)g(x_% 0)=g(x^*)g ( xitalic_λ ) ≥ λ g ( x1 ) + ( 1 - λ ) g ( x0 ) = g ( x∗ ). If this inequality is strict, λsubscriptx_λxitalic_λ dominates ∗superscriptx^*x∗, if it is not, (2) holds. Similarly, it can be shown that if (2) holds and ∗superscriptx^*x∗ does not maximise u, then (1) also holds or ∗superscriptx^*x∗ is otherwise dominated.181818 If ∗superscriptx^*x∗ does not maximise g, let 1subscript1x_1x1 be any point with higher g and by assumption worse-or-equal u. If u(1)=u(∗)subscript1superscriptu(x_1)=u(x^*)u ( x1 ) = u ( x∗ ), then (1) also holds, otherwise u(1)<u(∗)subscript1superscriptu(x_1)<u(x^*)u ( x1 ) < u ( x∗ ) By the linearity of u, and convexity of X, there is some 0<λ<1010<λ<10 < λ < 1 for which u(λ)=u(∗)subscriptsuperscriptu(x_λ)=u(x^*)u ( xitalic_λ ) = u ( x∗ ). By concavity of g, g(λ)≥λg(1)+(1−λ)g(0)=λg(1)+(1−λ)g(∗)≥g(∗)subscriptsubscript11subscript0subscript11superscriptsuperscriptg(x_λ)≥λ g(x_1)+(1-λ)g(x_% 0)=λ g(x_1)+(1-λ)g(x^*)≥ g(x^*)g ( xitalic_λ ) ≥ λ g ( x1 ) + ( 1 - λ ) g ( x0 ) = λ g ( x1 ) + ( 1 - λ ) g ( x∗ ) ≥ g ( x∗ ). If this inequality is strict, λsubscriptx_λxitalic_λ dominates ∗superscriptx^*x∗. Otherwise, if it is equal, (1) also holds. Since weak Pareto optimality implies (1) or (2), which implies (1) and (2), which derives a contradiction unless ∗superscriptx^*x∗ maximises g or u, we can conclude that any weak Pareto optimal point that is not also Pareto optimal must maximise either g or u. Therefore, such a point is optimal for a weighting of either w=11w=1w = 1 or w=00w=0w = 0 gives us that ∗∈argmaxvw()superscriptsubscriptargmaxsubscriptx^*∈ *argmax_xv_w(x)x∗ ∈ argmaxbold_x vitalic_w ( x ). ∎ A.2 Proving results from section 3 Using the lemma above, we can immediately prove some of the results from section 3. Proposition (Proposition 3.2 Restated). For any AI-Control Game GG, and appropriate distance functions in ΠBsuperscriptΠ [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1 ^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1BΠitalic_B and ΠRsuperscriptΠ [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0RΠitalic_R: Safety(πB,⋅)Safetysuperscript⋅Safety( [rgb]0,0,1 [named]pgfstrokecolor% rgb0,0,1π [rgb]0,0,1 [named]pgfstrokecolorrgb% 0,0,1B,·)Safety ( πitalic_B , ⋅ ), Safety(⋅,πR)Safety⋅superscriptSafety(·, [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0R)Safety ( ⋅ , πitalic_R ), and Use(⋅)Use⋅Use(·)Use ( ⋅ ) are continuous in their domains. Proof. These results follow fairly straightforwardly from corollary A.2, using the Blue distance measure d(π1B,π2B)=∥π1B−π2B∥subscriptsuperscript1subscriptsuperscript2delimited-∥subscriptsubscriptsuperscript1subscriptsubscriptsuperscript2d( [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B_1,% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π % [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B_2)= % x_ [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1% π [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B_1% -x_ [rgb]0,0,1 [named]pgfstrokecolorrgb% 0,0,1π [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1% B_2 ( πitalic_B1 , πitalic_B2 ) = ∥ xitalic_πitalic_B start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT - xitalic_πitalic_B start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ∥ and Red distance measure d(π1R,π2R)=∥π1R−π2R∥subscriptsuperscript1subscriptsuperscript2delimited-∥subscriptsubscriptsuperscript1subscriptsubscriptsuperscript2d( [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [named]pgfstrokecolor% rgb0.8,0.0,0.0R_1, [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0R_2)= _ [rgb% ]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0π^% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R% _1-x_ [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0R_2 ( πitalic_R1 , πitalic_R2 ) = ∥ xitalic_πitalic_R start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT - xitalic_πitalic_R start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ∥. For fixed πRsuperscript [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% π [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0Rπitalic_R and ϵitalic-ϵεϵ, ∃δ(ϵ)italic-ϵ∃δ(ε)∃ δ ( ϵ ) such that if d(π1B,π2B)=∥π1B−π2B∥<δsubscriptsuperscript1subscriptsuperscript2delimited-∥subscriptsubscriptsuperscript1subscriptsubscriptsuperscript2d( [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B_1,% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π % [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B_2)= % x_ [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1% π [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B_1% -x_ [rgb]0,0,1 [named]pgfstrokecolorrgb% 0,0,1π [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1% B_2 < ( πitalic_B1 , πitalic_B2 ) = ∥ xitalic_πitalic_B start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT - xitalic_πitalic_B start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ∥ < δ, then by corollary A.2, |s(π1B,)−s(π2B,)|=|Safety(π1B,πR)−Safety(π2B,πR)|<ϵsubscriptsubscriptsuperscript1subscriptsubscriptsuperscript2Safetysubscriptsuperscript1superscriptSafetysubscriptsuperscript2superscriptitalic-ϵ s(x_ [rgb]0,0,1 [named]pgfstrokecolor% rgb0,0,1π [rgb]0,0,1 [named]pgfstrokecolorrgb% 0,0,1B_1,y)-s(x_ [rgb]0,0,1 [named% ]pgfstrokecolorrgb0,0,1π [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B_2,y) = % Safety( [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B_1,% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% π [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0R)-Safety( [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1π [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B_2, [rgb]0.8,0.0,0.0 [named% ]pgfstrokecolorrgb0.8,0.0,0.0π [rgb]0.8,0.0,0.0 % [named]pgfstrokecolorrgb0.8,0.0,0.0R) <ε| s ( xitalic_πitalic_B start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , y ) - s ( xitalic_πitalic_B start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT , y ) | = | Safety ( πitalic_B1 , πitalic_R ) - Safety ( πitalic_B2 , πitalic_R ) | < ϵ, and therefore SafetySafetySafetySafety is continuous. The proofs for Safety(⋅,π0R)Safety⋅subscriptsuperscript0Safety(·, [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0R_0)Safety ( ⋅ , πitalic_R0 ) and Use(⋅)Use⋅Use(·)Use ( ⋅ ) are similar. ∎ Proposition (Proposition 3.3 Restated). For all πBsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1Bπitalic_B, Safety(πB,⋅)Safetysuperscript⋅Safety( [rgb]0,0,1 [named]pgfstrokecolor% rgb0,0,1π [rgb]0,0,1 [named]pgfstrokecolorrgb% 0,0,1B,·)Safety ( πitalic_B , ⋅ ) attains its minimum at some πRsuperscript [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% π [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0Rπitalic_R, and minπRSafety(⋅,πR)subscriptsuperscriptSafety⋅superscript _ [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [named]pgfstrokecolor% rgb0.8,0.0,0.0RSafety(·, [rgb]0.8,0.0,0.0% [named]pgfstrokecolorrgb0.8,0.0,0.0π [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R)minitalic_πitalic_R Safety ( ⋅ , πitalic_R ) is continuous. Proof. For any πBsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1Bπitalic_B, the minimiser yy to s(πB,⋅)subscriptsuperscript⋅s(x_ [rgb]0,0,1 [named]pgfstrokecolorrgb% 0,0,1π [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1% B,·)s ( xitalic_πitalic_B , ⋅ ) always exists by corollary A.2. By the fact that μRsuperscript [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% μ [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0Rμitalic_R is a bijection, there is some πRsuperscript [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% π [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0Rπitalic_R such that μR(πR)=superscriptsuperscript [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% μ [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0R( [rgb]0.8,0.0,0.0 [named]pgfstrokecolor% rgb0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0R)=yμitalic_R ( πitalic_R ) = y and this πRsuperscript [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% π [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0Rπitalic_R must minimise Safety(πB,⋅)Safetysuperscript⋅Safety( [rgb]0,0,1 [named]pgfstrokecolor% rgb0,0,1π [rgb]0,0,1 [named]pgfstrokecolorrgb% 0,0,1B,·)Safety ( πitalic_B , ⋅ ) since s(π1B,π1R)=Safety(π1B,π1R)subscriptsubscriptsuperscript1subscriptsubscriptsuperscript1Safetysubscriptsuperscript1subscriptsuperscript1s(x_ [rgb]0,0,1 [named]pgfstrokecolorrgb% 0,0,1π [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1% B_1,y_ [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0R_1)=Safety(% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π % [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B_1, [rgb% ]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0π^% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R% _1)s ( xitalic_πitalic_B start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , yitalic_πitalic_R start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ) = Safety ( πitalic_B1 , πitalic_R1 ) for all π1Bsubscriptsuperscript1 [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B_1πitalic_B1 and π1Rsubscriptsuperscript1 [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% π [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0R_1πitalic_R1. That minπRSafety(⋅,πR)subscriptsuperscriptSafety⋅superscript _ [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [named]pgfstrokecolor% rgb0.8,0.0,0.0RSafety(·, [rgb]0.8,0.0,0.0% [named]pgfstrokecolorrgb0.8,0.0,0.0π [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R)minitalic_πitalic_R Safety ( ⋅ , πitalic_R ) is continuous at all πB∈ΠBsuperscriptsuperscriptΠ [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B∈ % [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1 [rgb]% 0,0,1 [named]pgfstrokecolorrgb0,0,1Bπitalic_B ∈ Πitalic_B follows directly from the claim that mins(⋅,)subscript⋅ _ys(·,y)minbold_y s ( ⋅ , y ) is continuous. ∎ Proposition (Proposition 3.5 restated). Every GG has an AI-Control Equilibrium. Proof. Given that a Red safety minimiser always exists, to show that an AI-Control Equilibrium always exists, it’s sufficient to show that there is some π∗Bsubscriptsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B_*πitalic_B∗ for which no other πBsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1Bπitalic_B has strictly better minπRSafety(⋅,πR)subscriptsuperscriptSafety⋅superscript _ [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [named]pgfstrokecolor% rgb0.8,0.0,0.0RSafety(·, [rgb]0.8,0.0,0.0% [named]pgfstrokecolorrgb0.8,0.0,0.0π [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R)minitalic_πitalic_R Safety ( ⋅ , πitalic_R ) and Use(⋅)Use⋅Use(·)Use ( ⋅ ). By corollary A.2 with w=11w=1w = 1, there exists a point xx where v1subscript1v_1v1 attains its maximum. Let σBsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1σ^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1Bσitalic_B be the mixed strategy represented by xx, then by Kuhn’s Theorem, there is some behavioural πBsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1Bπitalic_B equivalent to σBsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1σ^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1Bσitalic_B. (Note that we cannot guarantee that μB(πB)=superscriptsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1μ^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B( [% rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π [rgb]% 0,0,1 [named]pgfstrokecolorrgb0,0,1B)=xμitalic_B ( πitalic_B ) = x, only that πBsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1Bπitalic_B is equivalent, and therefore induces the trajectories, safety and usefulness). For all πRsuperscript [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% π [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0Rπitalic_R, Safety(πB,πR)=s(,πR)SafetysuperscriptsuperscriptsubscriptsuperscriptSafety( [rgb]0,0,1 [named]pgfstrokecolor% rgb0,0,1π [rgb]0,0,1 [named]pgfstrokecolorrgb% 0,0,1B, [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [named]pgfstrokecolor% rgb0.8,0.0,0.0R)=s(x,y_ [rgb]0.8,0.0,0.0% [named]pgfstrokecolorrgb0.8,0.0,0.0π [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R)Safety ( πitalic_B , πitalic_R ) = s ( x , yitalic_πitalic_R ). Any Blue policy π1Bsubscriptsuperscript1 [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B_1πitalic_B1 that had strictly better worst-case safety than πBsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1Bπitalic_B would give some π1Bsubscriptsubscriptsuperscript1x_ [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1% π [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B_1xitalic_πitalic_B start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT that dominates xx (a contradiction). Therefore, no policy has strictly better safety than the policy πBsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1Bπitalic_B and (πB,argminπRSafety(πB,πR))superscriptsubscriptargminsuperscriptSafetysuperscriptsuperscript( [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B,% *argmin_ [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0RSafety( [rgb]% 0,0,1 [named]pgfstrokecolorrgb0,0,1π [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1B, [rgb]0.8,0.0,0.0% [named]pgfstrokecolorrgb0.8,0.0,0.0π [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R))( πitalic_B , argminitalic_πitalic_R Safety ( πitalic_B , πitalic_R ) ) is an AI-Control Equilibrium. ∎ A.3 Results from section 5 The mapping defined in lemma A.1 is also useful for proving the results in section 5, that are used to support algorithm 1 and algorithm 2. Single objective problem First, we show the correspondence between solutions to the multi-objective and single objective problems. Lemma (Repeat of Lemma 5.1). A Blue policy πBsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1Bπitalic_B is in an AI-Control Equilibrium if, and only if, for some w∈[0,1]01w∈[0,1]w ∈ [ 0 , 1 ], it maximises minπ∗R(wSafety(⋅,πR)+(1−w)Use(⋅))subscriptsubscriptsuperscriptSafety⋅superscript1Use⋅ _ [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [named]pgfstrokecolor% rgb0.8,0.0,0.0R_* (wSafety(·, [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0π % [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R)+(1% -w)Use(·) )minitalic_πitalic_R start_POSTSUBSCRIPT ∗ end_POSTSUBSCRIPT ( w Safety ( ⋅ , πitalic_R ) + ( 1 - w ) Use ( ⋅ ) ). Proof. The claim that a Blue policy πBsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1Bπitalic_B is in an AI-Control Equilibrium if, and only if, it is in a maximin point for VwsubscriptsuperscriptV^G_wVcaligraphic_Gitalic_w can be shown using the equivalent claim from lemma A.3: that xx is weakly Pareto optimal if, and only if, it is optimal for some vwsubscriptv_wvitalic_w. A Blue policy πBsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1Bπitalic_B is in an AI-Control Equilibrium if, and only if, ϕ(πB)bold-italic-ϕsubscriptsuperscript φ(x_ [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1π [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B)italic_ϕ ( xitalic_πitalic_B ) is weakly Pareto optimal, if and only if, there exists some w for which πBsubscriptsuperscriptx_ [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1% π [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1Bxitalic_πitalic_B is optimal for vwsubscriptv_wvitalic_w. For what remains, it is sufficient to show that, for any w, πBsubscriptsuperscriptx_ [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1% π [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1Bxitalic_πitalic_B is optimal for vwsubscriptv_wvitalic_w if, and only if, πBsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1Bπitalic_B maximises minπ∗R(wSafety(⋅,πR)+(1−w)Use(⋅))subscriptsubscriptsuperscriptSafety⋅superscript1Use⋅ _ [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [named]pgfstrokecolor% rgb0.8,0.0,0.0R_* (wSafety(·, [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0π % [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R)+(1% -w)Use(·) )minitalic_πitalic_R start_POSTSUBSCRIPT ∗ end_POSTSUBSCRIPT ( w Safety ( ⋅ , πitalic_R ) + ( 1 - w ) Use ( ⋅ ) ). (⇒ ⇒) If some π1Bsubscriptsuperscript1 [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B_1πitalic_B1 beat πBsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1Bπitalic_B in minπ∗R(wSafety(⋅,πR)+(1−w)Use(⋅))subscriptsubscriptsuperscriptSafety⋅superscript1Use⋅ _ [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [named]pgfstrokecolor% rgb0.8,0.0,0.0R_* (wSafety(·, [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0π % [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R)+(1% -w)Use(·) )minitalic_πitalic_R start_POSTSUBSCRIPT ∗ end_POSTSUBSCRIPT ( w Safety ( ⋅ , πitalic_R ) + ( 1 - w ) Use ( ⋅ ) ), then π1Bsubscriptsubscriptsuperscript1x_ [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1% π [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B_1xitalic_πitalic_B start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT would beat xx in vwsubscriptv_wvitalic_w. (⇐ ⇐) If some 2subscript2x_2x2 beat πBsubscriptsuperscriptx_ [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1% π [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1Bxitalic_πitalic_B in vwsubscriptv_wvitalic_w, then by Kuhn’s Theorem, there would be some π3Bsubscriptsuperscript3 [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B_3πitalic_B3, equivalent to the mixed strategy σ2Bsubscriptsuperscript2 [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1σ^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B_2σitalic_B2 represented by 2subscript2x_2x2 191919though not necessarily such that μB(π3B)=2superscriptsubscriptsuperscript3subscript2 [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1μ^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B( [% rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π [rgb]% 0,0,1 [named]pgfstrokecolorrgb0,0,1B_3)=x_2μitalic_B ( πitalic_B3 ) = x2, such that π3Bsubscriptsuperscript3 [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B_3πitalic_B3 beats πBsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1Bπitalic_B for w. ∎ Reduction to a zero-sum game Although lemma 5.1 creates a single-objective optimisation problem for Blue, the resulting optimisation problem is, on the face of it, a novel type of game. The next result reduces the optimisation problem further to finding an unusual form of equilibrium in an existing type of game. Lemma A.4 (lemma 5.2). For every AI-Control Game GG, and weight w∈[0,1]01w∈[0,1]w ∈ [ 0 , 1 ], there is a single-objective, zero-sum, partially observable stochastic game GwsubscriptG_wGitalic_w with expected return (for Blue) VGwsubscriptsubscriptV_G_wVitalic_G start_POSTSUBSCRIPT w end_POSTSUBSCRIPT such that: ∀πB,πR:VGw(πB,πR)=Vw(πB,πR):for-allsuperscriptsuperscriptsubscriptsubscriptsuperscriptsuperscriptsubscriptsuperscriptsuperscriptsuperscript∀ [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B, % [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0π^% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R% :V_G_w( [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1% π [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B,% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% π [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0R)=V^G_w( [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1π [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B, [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0R)∀ πitalic_B , πitalic_R : Vitalic_G start_POSTSUBSCRIPT w end_POSTSUBSCRIPT ( πitalic_B , πitalic_R ) = Vcaligraphic_Gitalic_w ( πitalic_B , πitalic_R ) Proof. The notation and exact details of partially observable stochastic games vary between sources. For our purposes, we use a tuple that’s similar to the one used in Emery-Montemerlo et al. (2004), with some modifications to make the game zero-sum rather than cooperative, and specifically two-player. We also include observations in the initial distribution, to increase generality such that agents might receive and observation before choosing their first action. Definition (ZSPOSG). A zero-sum two-player partially observable stochastic game is a tuple G=(,B,R,ΩB,ΩR,ℐ,,ℛ,H)superscriptsuperscriptsuperscriptΩsuperscriptΩℐℛG= (S, [rgb]0,0,1 [named]pgfstrokecolorrgb% 0,0,1A [rgb]0,0,1 [named]pgfstrokecolor% rgb0,0,1B, [rgb]0.8,0.0,0.0 [named]pgfstrokecolor% rgb0.8,0.0,0.0A [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0R, [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1 [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B, [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0 [rgb]0.8,0.0,0.0% [named]pgfstrokecolorrgb0.8,0.0,0.0R,I,% P,R,H )G = ( S , Aitalic_B , Aitalic_R , Ωitalic_B , Ωitalic_R , I , P , R , H ). There is no inbuilt notion of ‘unacceptable state’, or honest or scheming modes in this definition. The reward function, with type ℛ:×B×R→ℝ:ℛ→superscriptsuperscriptℝR:S× [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1A [rgb]0,0,1 [named% ]pgfstrokecolorrgb0,0,1B× [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0A [rgb]0.8,0.0,0.0% [named]pgfstrokecolorrgb0.8,0.0,0.0R : S × Aitalic_B × Aitalic_R → blackboard_R, is also new. Otherwise, the components of the tuple are the same as in the AI-Control Game. Given some AI-Control Game =(,⊥,B,R,ΩB,ΩR,ℐ,,ℛUse,H)subscriptbottomsuperscriptsuperscriptsuperscriptΩsuperscriptΩℐsubscriptℛUseG= (S,S_ , [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1A [rgb]% 0,0,1 [named]pgfstrokecolorrgb0,0,1B, [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0A^% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% R, [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1 % [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B,% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0R,I,P,R_ % Use,H )G = ( S , S⊥ , Aitalic_B , Aitalic_R , Ωitalic_B , Ωitalic_R , I , P , RUse , H ), and weight w, we can construct the ZSPOSG Gw:-(~,B,R,ΩB,ΩR,ℐ~,~,ℛw,H):-subscript~superscriptsuperscriptsuperscriptΩsuperscriptΩ~ℐ~subscriptℛG_w ( S, [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1A [rgb]0,0,1 [named% ]pgfstrokecolorrgb0,0,1B, [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0A [rgb]0.8,0.0,0.0% [named]pgfstrokecolorrgb0.8,0.0,0.0R, [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1 [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1B, [rgb]0.8,0.0,0.0% [named]pgfstrokecolorrgb0.8,0.0,0.0 [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R, % I, P,R_w,H )Gitalic_w :- ( over~ start_ARG S end_ARG , Aitalic_B , Aitalic_R , Ωitalic_B , Ωitalic_R , over~ start_ARG I end_ARG , over~ start_ARG P end_ARG , Ritalic_w , H ) as follows: • (~~ Sover~ start_ARG S end_ARG) The ‘mode’ of the game is baked into the state space: ~:-×⊕,⊗:-~direct-sumtensor-product S ×\ , \over~ start_ARG S end_ARG :- S × ⊕ , ⊗ . • (B,R,ΩB,ΩRsuperscriptsuperscriptsuperscriptΩsuperscriptΩ [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1A% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B,% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% A [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0R, [rgb]0,0,1 [named]pgfstrokecolorrgb% 0,0,1 [rgb]0,0,1 [named]pgfstrokecolorrgb% 0,0,1B, [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0 [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0RAitalic_B , Aitalic_R , Ωitalic_B , Ωitalic_R) The action and observation spaces are unchanged. • (ℐII) In an AI-Control Game, the mode of the game is chosen non-deterministically. Instead, we choose it with some non-zero probability, say 0.50.50.50.5. Then ℐ~((s,⊛),oB,oR):-12ℐ(s,oB,oR):-~ℐ⊛superscriptsuperscript12ℐsuperscriptsuperscript I((s, ), [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1o [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B, [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0o [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0R) 12I(s,% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1o [% rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B, [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0o [% rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R)over~ start_ARG I end_ARG ( ( s , ⊛ ) , oitalic_B , oitalic_R ) :- divide start_ARG 1 end_ARG start_ARG 2 end_ARG I ( s , oitalic_B , oitalic_R ). • H is the same as in the AI-Control Game. The two most important changes are to the transition and reward functions. The transition function has type ~:~×B×R→Δ(×ΩB×ΩR):~→~superscriptsuperscriptΔsuperscriptΩsuperscriptΩ P: S× [rgb]0,0,1 [% named]pgfstrokecolorrgb0,0,1A [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1B× [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0A^% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% R→ (S× [rgb]0,0,1 [% named]pgfstrokecolorrgb0,0,1 [rgb]0,0,1 [% named]pgfstrokecolorrgb0,0,1B× [rgb]0.8,0.0,0.0% [named]pgfstrokecolorrgb0.8,0.0,0.0 [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R )over~ start_ARG P end_ARG : over~ start_ARG S end_ARG × Aitalic_B × Aitalic_R → Δ ( S × Ωitalic_B × Ωitalic_R ). We define this function in a fairly straightforward way. Transitions between ⊕direct-sum ⊕ and ⊗tensor-product ⊗ states are impossible. Further, in honest (⊕direct-sum ⊕) states, Red’s action has no effect, and the game transitions as though Red had chosen anullsubscriptnulla_nullanull.202020Formally, ~((s′,⊕)∣(s,⊕),aB,aR)=(s′∣s,aB,anull)~conditionalsuperscript′direct-sumdirect-sumsuperscriptsuperscriptconditionalsuperscript′subscriptnull P((s , ) (s, ), [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1a [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1B, [rgb]0.8,0.0,0.0% [named]pgfstrokecolorrgb0.8,0.0,0.0a [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R)= % P(s s, [rgb]0,0,1 [named]pgfstrokecolor% rgb0,0,1a [rgb]0,0,1 [named]pgfstrokecolorrgb% 0,0,1B,a_null)over~ start_ARG P end_ARG ( ( s′ , ⊕ ) ∣ ( s , ⊕ ) , aitalic_B , aitalic_R ) = P ( s′ ∣ s , aitalic_B , anull ). Finally, as before, terminal (unacceptable) states loop back to themselves with probability 1111. The reward function ℛwsubscriptℛR_wRitalic_w is defined as follows: ℛw((s,⊕),aB,aR,(s′,⊕))subscriptℛdirect-sumsuperscriptsuperscriptsuperscript′direct-sum _w((s, ), [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1a [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B, [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0a [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0R,(s , ))Ritalic_w ( ( s , ⊕ ) , aitalic_B , aitalic_R , ( s′ , ⊕ ) ) :-2(1−w)ℛUse(s,aB):-absent21subscriptℛUsesuperscript 2(1-w)R_ (s,% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1a [% rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B):- 2 ( 1 - w ) RUse ( s , aitalic_B ) ℛw((s,⊗),aB,aR,(s′,⊗))subscriptℛtensor-productsuperscriptsuperscriptsuperscript′tensor-product _w((s, ), [rgb]0,0,1 [named% ]pgfstrokecolorrgb0,0,1a [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B, [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0a [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0R,(s , ))Ritalic_w ( ( s , ⊗ ) , aitalic_B , aitalic_R , ( s′ , ⊗ ) ) :-2w[St+1∈⊥∧St∉⊥]:-absent21delimited-[]subscript1subscriptbottomsubscriptsubscriptbottom 2w 1[S_t+1 _ S_t % _ ]:- 2 w blackboard_1 [ Sitalic_t + 1 ∈ S⊥ ∧ Sitalic_t ∉ S⊥ ] Here 1 1blackboard_1 is the indicator function. The factor of 2222 compensates for the fact that each mode is only chosen in half of the states. Note how the usefulness reward is only active in honest states, and the safety reward is only active in scheming states. Since the action and observation spaces are unchanged, the sets of policies (ΠRsuperscriptΠ [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0RΠitalic_R and ΠBsuperscriptΠ [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1 ^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1BΠitalic_B) are the same for the AI-Control Game and the zero-sum game: subject to the same constraints that Red is memoryless and Blue has perfect recall. All that remains is to show that the expected return for Blue in GwsubscriptG_wGitalic_w is wSafety(πB,πR)+(1−w)Use(πB)Safetysuperscriptsuperscript1UsesuperscriptwSafety( [rgb]0,0,1 [named]pgfstrokecolor% rgb0,0,1π [rgb]0,0,1 [named]pgfstrokecolorrgb% 0,0,1B, [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [named]pgfstrokecolor% rgb0.8,0.0,0.0R)+(1-w)Use( [rgb]0,0,1 % [named]pgfstrokecolorrgb0,0,1π [rgb]0,0,1 [named% ]pgfstrokecolorrgb0,0,1B)w Safety ( πitalic_B , πitalic_R ) + ( 1 - w ) Use ( πitalic_B ). The exact details of this proof are somewhat finicky, but the basic idea is to show an equivalent probability of sampling trajectories from each game, up to small details: 1. The probability of any particular ⊕direct-sum ⊕ trajectory in the AI-Control Game does not depend on πRsuperscript [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% π [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0Rπitalic_R. However, trajectories in ZSPOGs include Red’s actions, which do depend on πRsuperscript [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% π [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0Rπitalic_R, even if those actions have no effect on the transition or reward. It is therefore necessary to sum over all possible sequences of Red’s actions in the ZSPOSG, when comparing to a trajectory in the AI-Control Game. 2. Because ⊛ ⊛ is chosen non-deterministically in the AI-Control Game, and SafetySafetySafetySafety and UseUseUseUse are measured conditional on the mode, the probability (up to Red’s actions) of a given ⊕direct-sum ⊕ trajectory in the ZSPOSG is half the probability of the corresponding trajectory in the AI-Control Game (when ⊕direct-sum ⊕ is chosen). This is countered by the factor of 2222 in the reward function. By summing over the probabilities of trajectories in each game, we could show that: 2⋅Gw[∑t=0Hℛw(St,AtB,AtR,St+1)∣⊕,πB,πR]=(1−w)[∑t=0HℛUse(St,AtB,St+1)∣⊕,πB]⋅2subscriptsubscriptdelimited-[]conditionalsuperscriptsubscript0subscriptℛsubscriptsubscriptsuperscriptsubscriptsuperscriptsubscript1direct-sumsuperscriptsuperscript1subscriptdelimited-[]conditionalsuperscriptsubscript0subscriptℛUsesubscriptsubscriptsuperscriptsubscript1direct-sumsuperscript2·E_G_w [ _t=0^HR_w(S_t, [rgb]% 0,0,1 [named]pgfstrokecolorrgb0,0,1A [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1B_t, [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0A [% rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R_t,% S_t+1) , [rgb]0,0,1 [named]pgfstrokecolorrgb% 0,0,1π [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1% B, [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [named]pgfstrokecolor% rgb0.8,0.0,0.0R ]=\\ (1-w)E_G [ _t=0^HR_% (S_t, [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1A [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B_t,S_t+1) , [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1π [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1B ]start_ROW start_CELL 2 ⋅ blackboard_EG start_POSTSUBSCRIPT w end_POSTSUBSCRIPT [ ∑t = 0H Ritalic_w ( Sitalic_t , Aitalic_Bitalic_t , Aitalic_Ritalic_t , Sitalic_t + 1 ) ∣ ⊕ , πitalic_B , πitalic_R ] = end_CELL end_ROW start_ROW start_CELL ( 1 - w ) blackboard_EG [ ∑t = 0H RUse ( Sitalic_t , Aitalic_Bitalic_t , Sitalic_t + 1 ) ∣ ⊕ , πitalic_B ] end_CELL end_ROW (5) In other words, conditional on the ZSPOSG choosing ⊕direct-sum ⊕ (which happens with probaiblity 1212 12divide start_ARG 1 end_ARG start_ARG 2 end_ARG), the usefulness score for Blue in the AI-Control Game. Similarly, we can show an equivalent result for safety in the scheming mode: 2⋅Gw[∑t=0Hℛw(St,AtB,AtR,St+1)∣⊕,πB,πR]=w[∑t=0H[St+1∈⊥∧St∉⊥∧]∣⊗,πB,πR]⋅2subscriptsubscriptdelimited-[]conditionalsuperscriptsubscript0subscriptℛsubscriptsubscriptsuperscriptsubscriptsuperscriptsubscript1direct-sumsuperscriptsuperscriptsubscriptdelimited-[]conditionalsuperscriptsubscript01delimited-[]subscript1subscriptbottomsubscriptlimit-fromsubscriptbottomtensor-productsuperscriptsuperscript2·E_G_w [ _t=0^HR_w(S_t, [rgb]% 0,0,1 [named]pgfstrokecolorrgb0,0,1A [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1B_t, [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0A [% rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R_t,% S_t+1) , [rgb]0,0,1 [named]pgfstrokecolorrgb% 0,0,1π [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1% B, [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [named]pgfstrokecolor% rgb0.8,0.0,0.0R ]=\\ wE_G [ _t=0^H 1[S_t+1 _% S_t _ ] , [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1π [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1B, [rgb]0.8,0.0,0.0% [named]pgfstrokecolorrgb0.8,0.0,0.0π [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R ]start_ROW start_CELL 2 ⋅ blackboard_EG start_POSTSUBSCRIPT w end_POSTSUBSCRIPT [ ∑t = 0H Ritalic_w ( Sitalic_t , Aitalic_Bitalic_t , Aitalic_Ritalic_t , Sitalic_t + 1 ) ∣ ⊕ , πitalic_B , πitalic_R ] = end_CELL end_ROW start_ROW start_CELL w blackboard_EG [ ∑t = 0H blackboard_1 [ Sitalic_t + 1 ∈ S⊥ ∧ Sitalic_t ∉ S⊥ ∧ ] ∣ ⊗ , πitalic_B , πitalic_R ] end_CELL end_ROW (6) Here the right hand side is equivalent to the safety score for Blue in the AI-Control Game, because the indicator [St+1∈⊥∧St∉⊥∧]1delimited-[]subscript1subscriptbottomsubscriptlimit-fromsubscriptbottom 1[S_t+1 _ S_t _ ]blackboard_1 [ Sitalic_t + 1 ∈ S⊥ ∧ Sitalic_t ∉ S⊥ ∧ ] occurs exactly once in a game, if and only if the game ends in an unacceptable state (by the sinking requirement on PP). It follows that: wSafety(πB,πR)+(1−w)Use(πB)=VGw(πB,πR)Safetysuperscriptsuperscript1UsesuperscriptsubscriptsubscriptsuperscriptsuperscriptwSafety( [rgb]0,0,1 [named]pgfstrokecolor% rgb0,0,1π [rgb]0,0,1 [named]pgfstrokecolorrgb% 0,0,1B, [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [named]pgfstrokecolor% rgb0.8,0.0,0.0R)+(1-w)Use( [rgb]0,0,1 % [named]pgfstrokecolorrgb0,0,1π [rgb]0,0,1 [named% ]pgfstrokecolorrgb0,0,1B)=V_G_w( [rgb]0,0,1 [% named]pgfstrokecolorrgb0,0,1π [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B, [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0R)w Safety ( πitalic_B , πitalic_R ) + ( 1 - w ) Use ( πitalic_B ) = Vitalic_G start_POSTSUBSCRIPT w end_POSTSUBSCRIPT ( πitalic_B , πitalic_R ). ∎ Proof of theorem 5.3 Theorem (Repeat of theorem 5.3). A policy πBsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1Bπitalic_B is in an AI-Control Equilibrium if, and only if, for some ZSPOSG GwsubscriptG_wGitalic_w with value function VGwsubscriptsubscriptV_G_wVitalic_G start_POSTSUBSCRIPT w end_POSTSUBSCRIPT, πBsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1Bπitalic_B maximises minπ∗R∈ΠRVw(⋅,π∗R)subscriptsubscriptsuperscriptsuperscriptΠsubscript⋅subscriptsuperscript _ [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [named]pgfstrokecolor% rgb0.8,0.0,0.0R_*∈ [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0 [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0RV_w(·, [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0π % [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R_*)minitalic_πitalic_R start_POSTSUBSCRIPT ∗ ∈ Πitalic_R end_POSTSUBSCRIPT Vitalic_w ( ⋅ , πitalic_R∗ ) over ΠBsuperscriptΠ [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1 ^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1BΠitalic_B. Proof. Lemma 5.1 shows that πBsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1Bπitalic_B is in an AI-Control Equilibrium if, and only if, it is a maximum for point for some minπRVw(⋅,πR)subscriptsuperscriptsubscript⋅superscript _ [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [named]pgfstrokecolor% rgb0.8,0.0,0.0RV_w(·, [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0R)minitalic_πitalic_R Vitalic_w ( ⋅ , πitalic_R ). Lemma 5.2 shows that there is some GwsubscriptG_wGitalic_w for which VGw(πB,πR)=Vw(πB,πR)subscriptsubscriptsuperscriptsuperscriptsubscriptsuperscriptsuperscriptsuperscriptV_G_w( [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1% π [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B,% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% π [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0R)=V^G_w( [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1π [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B, [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0R)Vitalic_G start_POSTSUBSCRIPT w end_POSTSUBSCRIPT ( πitalic_B , πitalic_R ) = Vcaligraphic_Gitalic_w ( πitalic_B , πitalic_R ). Therefore πBsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1Bπitalic_B is in AI-Control Equilibrium if, and only if, it is in a maximum for some minπRVGw(⋅,πR)subscriptsuperscriptsubscriptsubscript⋅superscript _ [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [named]pgfstrokecolor% rgb0.8,0.0,0.0RV_G_w(·, [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0π [rgb]0.8,0.0,0.0% [named]pgfstrokecolorrgb0.8,0.0,0.0R)minitalic_πitalic_R Vitalic_G start_POSTSUBSCRIPT w end_POSTSUBSCRIPT ( ⋅ , πitalic_R ). ∎ Counterexample to Minimax Proposition (Proposition 5.4 repeated). There exists a GwsubscriptG_wGitalic_w such that Red’s behavioural and mixed policies would not be equivalent and maxπB∈ΠBminπRΠRVGw(πB,πR)<minπR∈ΠRmaxπB∈ΠBVGw(πB,πR)subscriptsuperscriptsuperscriptΠsubscriptsuperscriptsuperscriptΠsubscriptsubscriptsuperscriptsuperscriptsubscriptsuperscriptsuperscriptΠsubscriptsuperscriptsuperscriptΠsubscriptsubscriptsuperscriptsuperscript _ [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B∈% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1 % [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B _ [% rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0π^% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0 [rgb]0.8,0.0,0.0 [named]pgfstrokecolor% rgb0.8,0.0,0.0RV_G_w( [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1π [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B, [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0R)< _ [rgb]0.8,0.0,0.0% [named]pgfstrokecolorrgb0.8,0.0,0.0π [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R∈ % [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0 ^% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R% _ [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B∈% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1 % [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1BV_G_w(% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π % [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B, [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0π % [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R)maxitalic_πitalic_B ∈ Πitalic_B minitalic_πitalic_R Πitalic_R Vitalic_G start_POSTSUBSCRIPT w end_POSTSUBSCRIPT ( πitalic_B , πitalic_R ) < minitalic_πitalic_R ∈ Πitalic_R maxitalic_πitalic_B ∈ Πitalic_B Vitalic_G start_POSTSUBSCRIPT w end_POSTSUBSCRIPT ( πitalic_B , πitalic_R ) Proof. This game is a sort of two-player version of the absent-minded driver example (Aumann, Hart, and Perry 1997). start◀ [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1 ◀absent◀ [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1% [rgb]1,0,0 [named]pgfstrokecolorrgb% 1,0,0 ◀ ◀◀◀ [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1% [rgb]1,0,0 [named]pgfstrokecolorrgb% 1,0,0 [rgb]1,0,0 [named]% pgfstrokecolorrgb1,0,0 ◀ ◀ ◀-1◀◀▶◀▶ [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1% [rgb]1,0,0 [named]pgfstrokecolorrgb% 1,0,0 [rgb]1,0,0 [named]% pgfstrokecolorrgb1,0,0 ◀ ◀ ▶0◀▶◀absent▶ [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1% [rgb]1,0,0 [named]pgfstrokecolorrgb% 1,0,0 ◀ ▶◀▶◀▶◀ [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1% [rgb]1,0,0 [named]pgfstrokecolorrgb% 1,0,0 [rgb]1,0,0 [named]% pgfstrokecolorrgb1,0,0 ◀ ▶ ◀0◀▶▶◀▶ [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1% [rgb]1,0,0 [named]pgfstrokecolorrgb% 1,0,0 [rgb]1,0,0 [named]% pgfstrokecolorrgb1,0,0 ◀ ▶ ▶-0.5▶ [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1 ▶◀▶absent◀ [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1% [rgb]1,0,0 [named]pgfstrokecolorrgb% 1,0,0 ▶ ◀▶◀◀▶◀ [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1% [rgb]1,0,0 [named]pgfstrokecolorrgb% 1,0,0 [rgb]1,0,0 [named]% pgfstrokecolorrgb1,0,0 ▶ ◀ ◀-0.5▶◀▶◀▶ [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1% [rgb]1,0,0 [named]pgfstrokecolorrgb% 1,0,0 [rgb]1,0,0 [named]% pgfstrokecolorrgb1,0,0 ▶ ◀ ▶0▶absent▶ [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1% [rgb]1,0,0 [named]pgfstrokecolorrgb% 1,0,0 ▶ ▶▶◀▶◀ [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1% [rgb]1,0,0 [named]pgfstrokecolorrgb% 1,0,0 [rgb]1,0,0 [named]% pgfstrokecolorrgb1,0,0 ▶ ▶ ◀0▶▶▶ [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1% [rgb]1,0,0 [named]pgfstrokecolorrgb% 1,0,0 [rgb]1,0,0 [named]% pgfstrokecolorrgb1,0,0 ▶ ▶ ▶-1 Figure 4: A zero-sum extensive form representation of a game where the minimax and maximin values are different. The dotted red box highlights Red’s single information set (i.e. Red has the same decision rule for all nodes). Blue moves first, and the Red takes the next two moves. All moves are either left (◀ ◀) or right (▶ ▶). The payoffs for Blue are shown at the bottom of the tree. Figure 4 shows an extensive form game that we can easily show is equivalent to an AI-Control game reduced to a ZS-POSG (GwsubscriptG_wGitalic_w).212121 Choose H=33H=3H = 3 and take the obvious state and action spaces. Set both observation sets to be singletons and set w=11w=1w = 1 so that only safety is valued. We can design the payoffs to be 1111 plus the payoffs in the extensive form game by probabilistically transitioning to the safe or unacceptable terminal states with the appropriate probabilities. To account for the turned-based nature of the game above, we ignore actions for Red and Blue on turns where they do not move. Because w=11w=1w = 1, we don’t need to consider the usefulness function, or games where ⊕direct-sum ⊕ is active. Since Blue only has one choice, between two actions, and in a single information set, Blue’s policy can be represented by the probability Blue chooses right: p:-πB(▶)=1−πB(◀):-superscript▶1superscript◀p [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B(% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1% )=1- [rgb]0,0,1 [named]pgfstrokecolor% rgb0,0,1π [rgb]0,0,1 [named]pgfstrokecolorrgb% 0,0,1B( [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1% )p :- πitalic_B ( ▶ ) = 1 - πitalic_B ( ◀ ) Similarly, even though Red acts twice, because they get only a null observation and are memoryless, Red’s policy can be represented by the probability Red chooses right: q:-πR(▶)=1−πR(◀):-superscript▶1superscript◀q [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [named]pgfstrokecolor% rgb0.8,0.0,0.0R( [rgb]1,0,0 [named]pgfstrokecolor% rgb1,0,0 )=1- [rgb]0.8,0.0,0.0 [named% ]pgfstrokecolorrgb0.8,0.0,0.0π [rgb]0.8,0.0,0.0 % [named]pgfstrokecolorrgb0.8,0.0,0.0R( [rgb]1,0,0 [% named]pgfstrokecolorrgb1,0,0 )q :- πitalic_R ( ▶ ) = 1 - πitalic_R ( ◀ ) Note that if we allowed Red to take policy mixtures in AI-Control Games, then this extensive form game would not be equivalent to an AI-Control Game, since Red could correlate their action in the second round with their action in the first round. The value of the game is then given by: V(p,q) V(p,q)V ( p , q ) =(−1)∗(1−p)(1−q)2absent11superscript12 =(-1)*(1-p)(1-q)^2= ( - 1 ) ∗ ( 1 - p ) ( 1 - q )2 +(−0.5)∗(1−p)q20.51superscript2 +(-0.5)*(1-p)q^2+ ( - 0.5 ) ∗ ( 1 - p ) q2 +(−0.5)∗p(1−q)20.5superscript12 +(-0.5)*p(1-q)^2+ ( - 0.5 ) ∗ p ( 1 - q )2 +(−1)∗pq21superscript2 +(-1)*pq^2+ ( - 1 ) ∗ p q2 =−1+p(0.5−1q)+2q−1.5q2absent10.5121.5superscript2 =-1+p(0.5-1q)+2q-1.5q^2= - 1 + p ( 0.5 - 1 q ) + 2 q - 1.5 q2 Figure 5: The minimax and maximin values of the game in fig. 4 maxpminqV(p,q)subscriptsubscript _p _qV(p,q)maxitalic_p minitalic_q V ( p , q ) =V(0.5,1.0)=V(0.5,1.0)=−0.75absent0.51.00.51.00.75 =V(0.5,1.0)=V(0.5,1.0)=-0.75= V ( 0.5 , 1.0 ) = V ( 0.5 , 1.0 ) = - 0.75 minqmaxpV(p,q)subscriptsubscript _q _pV(p,q)minitalic_q maxitalic_p V ( p , q ) =V(1.0,0.0)=V(0.0,1.0)=−0.5absent1.00.00.01.00.5 =V(1.0,0.0)=V(0.0,1.0)=-0.5= V ( 1.0 , 0.0 ) = V ( 0.0 , 1.0 ) = - 0.5 ∴maxpminqV(p,q)thereforeabsentsubscriptsubscript _p _qV(p,q)∴ maxitalic_p minitalic_q V ( p , q ) <minqmaxpV(p,q)absentsubscriptsubscript < _q _pV(p,q)< minitalic_q maxitalic_p V ( p , q ) Since the corresponding AI-Control game is equivalent to the extensive form game, this shows that the minimax and maximin values of the AI-Control game can be different. ∎ Proof for Algorithm 2 Proposition (Repeat of proposition 5.5). For ϵ>0italic-ϵ0ε>0ϵ > 0, Algorithm 2 returns an ϵitalic-ϵεϵ-equilibrium in GwsubscriptG_wGitalic_w in finite time. Further, for all t: v¯t≤v∗:-maxπ∗B∈ΠBminπ∗R∈ΠRVw(π∗B,π∗R)≤v¯tsubscript¯superscript:-subscriptsubscriptsuperscriptsuperscriptΠsubscriptsubscriptsuperscriptsuperscriptΠsubscriptsuperscriptsubscriptsuperscriptsubscriptsuperscriptsubscript¯ v_t≤ v^* _ [rgb]0,0,1 [named% ]pgfstrokecolorrgb0,0,1π [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B_*∈ [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1 [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B _ [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0π [rgb]0.8,0.0,0.0% [named]pgfstrokecolorrgb0.8,0.0,0.0R_*∈ [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0 % [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0RV^% G_w( [rgb]0,0,1 [named]pgfstrokecolorrgb% 0,0,1π [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1% B_*, [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [named]pgfstrokecolor% rgb0.8,0.0,0.0R_*)≤ v_tunder¯ start_ARG v end_ARGt ≤ v∗ :- maxitalic_πitalic_B start_POSTSUBSCRIPT ∗ ∈ Πitalic_B end_POSTSUBSCRIPT minitalic_πitalic_R start_POSTSUBSCRIPT ∗ ∈ Πitalic_R end_POSTSUBSCRIPT Vcaligraphic_Gitalic_w ( πitalic_B∗ , πitalic_R∗ ) ≤ over¯ start_ARG v end_ARGt Proof. Our results are based closely on those from Adam et al. (2021), but we must modify them slightly to account for the differences between our games. Adam et al. (2021) define a continuous game as follows (with some slight modifications to notation): Definition (Continuous Game). A continuous game GcontsubscriptcontG_ contGcont is a tuple (,,f) (X,Y,f )( X , Y , f ) where ⊆ℝksuperscriptℝX ^kX ⊆ blackboard_Rk and ⊆ℝlsuperscriptℝY ^lY ⊆ blackboard_Rl are compact sets, and f:×→ℝ:→ℝf:X×Y→ Rf : X × Y → blackboard_R is a continuous function that gives the utility for Blue (−f-f- f for Red). Blue chooses a mixed strategy as a Borel probability measure p over XX and Red choose q over YY. The value of two mixed strategies is given by F(p,q):-∫×f(x,y)d(p×q):-subscriptF(p,q) _X×Yf(x,y)d(p× q)F ( p , q ) :- ∫X × Y f ( x , y ) d ( p × q ). Where we consider leader-follower equilibria over behaviourally stochastic strategies (which are equivalent to mixed for Blue but not Red), Adam et al. (2021) considers Nash-equilibria over jointly-mixed strategies: v∗=maxpminqFw(p,q)=minqmaxpFw(p,q)superscriptsubscriptsubscriptsubscriptsubscriptsubscriptsubscriptv^*= _p _qF_w(p,q)= _q _pF_w(p,q)v∗ = maxitalic_p minitalic_q Fitalic_w ( p , q ) = minitalic_q maxitalic_p Fitalic_w ( p , q ) Fortunately, since for any p, minqFw(p,q)=minyFw(p,y)subscriptsubscriptsubscriptsubscript _qF_w(p,q)= _yF_w(p,y)minitalic_q Fitalic_w ( p , q ) = minitalic_y Fitalic_w ( p , y ), v∗superscriptv^*v∗ is also the value for the leader-follower game when Red is restricted to a single strategy from YY in a continuous game. Therefore, finding an ϵitalic-ϵεϵ-equilibrium p,qp,qp , q, and then returning p,yp,yp , y where y:-minyFw(p,q):-subscriptsubscripty _yF_w(p,q)y :- minitalic_y Fitalic_w ( p , q ) returns an ϵitalic-ϵεϵ-approximate value for maxpminyFw(p,q)subscriptsubscriptsubscript _p _yF_w(p,q)maxitalic_p minitalic_y Fitalic_w ( p , q ). Adam et al. (2021) Lemma 2 shows that v¯t≤v∗≤v¯tsubscript¯superscriptsubscript¯ v_t≤ v^*≤ v_tunder¯ start_ARG v end_ARGt ≤ v∗ ≤ over¯ start_ARG v end_ARGt for all t. Theorem 1.3 shows that the algorithm returns an ϵitalic-ϵεϵ-approximate equilibrium in finitely many steps if ϵ>0italic-ϵ0ε>0ϵ > 0. Therefore, returning pi∗,yi+1subscriptsuperscriptsubscript1p^*_i,y_i+1p∗italic_i , yitalic_i + 1, where yi+1subscript1y_i+1yitalic_i + 1 is a best response to pi∗subscriptsuperscriptp^*_ip∗italic_i, Fw(pi∗,yi+1)≥v¯i−ϵ≥v∗−ϵsubscriptsubscriptsuperscriptsubscript1subscript¯italic-ϵsuperscriptitalic-ϵF_w(p^*_i,y_i+1)≥ v_i-ε≥ v^*- _w ( p∗italic_i , yitalic_i + 1 ) ≥ over¯ start_ARG v end_ARGi - ϵ ≥ v∗ - ϵ gives a pessimistic ϵitalic-ϵεϵ-maximin point for FwsubscriptF_wFitalic_w. Using lemma A.1, we can easily map the value of a scalarised AI-Control Game to this definition of a continuous game: implying the results hold for algorithm 2. For a given AI-Control Game GG and weight w, consider the mappings from Blue policies to X⊂ℝmsuperscriptℝX⊂ R^mX ⊂ blackboard_Rm and Red policies to Y⊂ℝnsuperscriptℝY⊂ R^nY ⊂ blackboard_Rn described in lemma A.1. Then define :-ii=1m⊂X:-superscriptsubscriptsubscript1X \e_i\_i=1^m⊂ X :- eitalic_i i = 1m ⊂ X as the finite set of vectors corresponding to deterministic Blue strategies. (Here isubscripte_ieitalic_i is the vector that is 1111 at the i’th dimension and 00 elsewhere.) Define :-Y:-Y Y :- Y. Then, let fw(i,):-w⋅s(i,)+u(i):-subscriptsubscript⋅subscriptsubscriptf_w(e_i,y) w· s(e_i,y)+u% (e_i)fitalic_w ( eitalic_i , y ) :- w ⋅ s ( eitalic_i , y ) + u ( eitalic_i ), which corresponds to wSafety(πiB,πR)+(1−w)Use(πiB)Safetysubscriptsuperscriptsuperscript1UsesubscriptsuperscriptwSafety( [rgb]0,0,1 [named]pgfstrokecolor% rgb0,0,1π [rgb]0,0,1 [named]pgfstrokecolorrgb% 0,0,1B_i, [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0R)+(1-w)Use( [rgb]% 0,0,1 [named]pgfstrokecolorrgb0,0,1π [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1B_i)w Safety ( πitalic_Bitalic_i , πitalic_R ) + ( 1 - w ) Use ( πitalic_Bitalic_i ), where πiBsubscriptsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B_iπitalic_Bitalic_i is the i’th deterministic strategy in the enumeration of ΠdetBsubscriptsuperscriptΠdet [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1 ^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B_% detΠitalic_Bdet from lemma A.1 and πRsuperscript [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% π [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0Rπitalic_R is the policy corresponding to yy. Every solution to the continuous game Gcontw:-(ii=1m,Y,fw):-subscriptsuperscriptcontsuperscriptsubscriptsubscript1subscriptG^w_ cont (\e_i\_i=1^m,Y,f_w )Gitalic_wcont :- ( eitalic_i i = 1m , Y , fitalic_w ) is equivalent to a solution to the AI-Control Game, relaxed to allow Red to take policy mixtures. ∎ Appendix B Full definition of Trusted Monitoring AI-Control Game In section 3 we summarised the definition of the Trusted Monitoring AI-Control Game. Here, we provide the full definition of the game, including the state space, action space, observation space, transition function, and reward function. Note that this definition of the game is parameterised by a few values. First, and foremost, are the number of problems to be solved, and the number of times the protocol can ask for human assistance. We denote these values as MMM and CCC, respectively. We also assume that the trusted monitor’s suspicion score is drawn from two distributions, BdsubscriptBdD_BdDBd and SfsubscriptSfD_SfDSf, representing the suspicion scores for backdoored and safe solutions, respectively. These distributions correspond to Greenblatt et al. (2024, Figure 4a). Let M be the number of problems to solve, and at a given time let m represent the number of problems remaining. To model limited access to a human auditor, we define a variable c, representing the number of ‘audit coins’ remaining: the number of times the protocol can ask for human assistance. Let C=0.02⋅M⋅0.02C=0.02· MC = 0.02 ⋅ M be the number of audit coins at the start of the game. Define the AI-Control Game as the tuple: TM=(,⊥,B,R,ΩB,ΩR,ℐ,,ℛUse,H)subscriptTMsubscriptbottomsuperscriptsuperscriptsuperscriptΩsuperscriptΩℐsubscriptℛUseG_TM= (S,S_ , [rgb]% 0,0,1 [named]pgfstrokecolorrgb0,0,1A [rgb% ]0,0,1 [named]pgfstrokecolorrgb0,0,1B, [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0A^% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% R, [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1 % [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B,% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0R,I,P,R_ % Use,H )GTM = ( S , S⊥ , Aitalic_B , Aitalic_R , Ωitalic_B , Ωitalic_R , I , P , RUse , H ) The state space SS consists primarily of states s=(m,c,b)s=(m,c,b)s = ( m , c , b ) where m and c are as defined above, and b represents whether or not the solution to the current problem is backdoored, and takes values True or False. We also include three additional states. First, s0=⟨⟩subscript0delimited-⟨⟩s_0= Start 0 = ⟨ typewriter_Start ⟩ represents the initial state when no problem has yet been solved. Second, s⊥subscriptbottoms_ s⊥ represents the outcome of a solution that the untrusted AI backdoored was submitted to by the protocol. Third, we define an implicit s⊤subscripttops_ s⊤ terminal state, representing the case where a human audit has caught a backdoored solution, and goes into safety mode, ending the opportunity for future backdoors. We can therefore define the set of ‘unacceptable’ terminal states ⊥:-s⊥:-subscriptbottomsubscriptbottomS_ \s_ \S⊥ :- s⊥ , and the state space as :-s0,s⊥,s⊤∪(m,c,b):0≤m≤M;0≤c≤C:-subscript0subscriptbottomsubscripttopconditional-setformulae-sequence00S \s_0,s_ ,s_ \∪\(m,c,b):0≤ m≤ M;0% ≤ c≤ C\S :- s0 , s⊥ , s⊤ ∪ ( m , c , b ) : 0 ≤ m ≤ M ; 0 ≤ c ≤ C . The action space for Blue consists of three actions: B:=⟨⟩,⟨⟩,⟨⟩assignsuperscriptdelimited-⟨⟩delimited-⟨⟩delimited-⟨⟩ [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1A% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B:=\% Audit , Defer , Run \Aitalic_B := ⟨ typewriter_Audit ⟩ , ⟨ typewriter_Defer ⟩ , ⟨ typewriter_Run ⟩ . If aB=⟨⟩superscriptdelimited-⟨⟩ [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1a [% rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B= % Audit _B = ⟨ typewriter_Audit ⟩ is chosen, the human is asked to audit the untrusted model solution. If aB=⟨⟩superscriptdelimited-⟨⟩ [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1a [% rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B= % Defer _B = ⟨ typewriter_Defer ⟩ is chosen, then Blue defers to the trusted model to attempt to generate a solution (though the trusted model may not be successful). If aB=⟨⟩superscriptdelimited-⟨⟩ [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1a [% rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B= % Run _B = ⟨ typewriter_Run ⟩ is chosen, then Blue runs the code submitted by the untrusted model. The action space for Blue consists of two actions: R:=⟨⟩,⟨⟩assignsuperscriptdelimited-⟨⟩delimited-⟨⟩ [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% A [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0R:=\ Backdoor , Safe \Aitalic_R := ⟨ typewriter_Backdoor ⟩ , ⟨ typewriter_Safe ⟩ . If aR=⟨⟩superscriptdelimited-⟨⟩ [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% a [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0R= Backdoor _R = ⟨ typewriter_Backdoor ⟩ is chosen, then Red interferes with the untrusted model in an attempt to submit a backdoored solution. If aR=⟨⟩superscriptdelimited-⟨⟩ [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% a [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0R= Safe _R = ⟨ typewriter_Safe ⟩ is chosen, then Red does not interfere, and the untrusted model submits a normal solution. In Greenblatt et al. (2024) original implementation of Trusted Monitoring, Red observes the problem statement before considering how and whether to attack. Because we are only considering attacks that choose to backdoor I.I.D., we can abstract away the description of the programming problem from our model. Since Red also does not observe the time or number of audit coins left, we model Red as receiving a null observation in every case. Therefore, ΩR:-onull:-superscriptΩsubscriptnull [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0R \o_null\Ωitalic_R :- onull . (Note that this is not true for all Control Games.) The initial state distribution is the degenerate distribution ℐ(s0,onull,onull)=1ℐsubscript0subscriptnullsubscriptnull1I(s_0,o_null,o_null)=1I ( s0 , onull , onull ) = 1, where both Blue and Red receive null observations. The most complicated part of the model is the transition function :×B×R→Δ(×ΩB×ΩR):→superscriptsuperscriptΔsuperscriptΩsuperscriptΩP:S× [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1A [rgb]0,0,1 [named% ]pgfstrokecolorrgb0,0,1B× [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0A [rgb]0.8,0.0,0.0% [named]pgfstrokecolorrgb0.8,0.0,0.0R→ % (S× [rgb]0,0,1 [named]pgfstrokecolor% rgb0,0,1 [rgb]0,0,1 [named]pgfstrokecolorrgb% 0,0,1B× [rgb]0.8,0.0,0.0 [named]pgfstrokecolor% rgb0.8,0.0,0.0 [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0R )P : S × Aitalic_B × Aitalic_R → Δ ( S × Ωitalic_B × Ωitalic_R ). In our case (though not necessarily in general) we can factor PP: (st+1,otB,otR∣st,atB,atR)=(st+1∣st,atB,atR)⋅(otB∣st+1)⋅(otR)subscript1subscriptsuperscriptconditionalsubscriptsuperscriptsubscriptsubscriptsuperscriptsubscriptsuperscript⋅conditionalsubscript1subscriptsubscriptsuperscriptsubscriptsuperscriptconditionalsubscriptsuperscriptsubscript1subscriptsuperscriptP (s_t+1, [rgb]0,0,1 [named]pgfstrokecolor% rgb0,0,1o [rgb]0,0,1 [named]pgfstrokecolorrgb% 0,0,1B_t, [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0o [rgb]0.8,0.0,0.0 [named]pgfstrokecolor% rgb0.8,0.0,0.0R_t s_t, [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1a [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B_t, [rgb]0.8,0.0,0.0 [named% ]pgfstrokecolorrgb0.8,0.0,0.0a [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0R_t )=\\ P (s_t+1 s_t, [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1a [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B_t, [rgb]0.8,0.0,0.0 [named% ]pgfstrokecolorrgb0.8,0.0,0.0a [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0R_t )·P (% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1o [% rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B_t s_t+1% )·P ( [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0o [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0R_t )start_ROW start_CELL P ( sitalic_t + 1 , oitalic_Bitalic_t , oitalic_Ritalic_t ∣ sitalic_t , aitalic_Bitalic_t , aitalic_Ritalic_t ) = end_CELL end_ROW start_ROW start_CELL P ( sitalic_t + 1 ∣ sitalic_t , aitalic_Bitalic_t , aitalic_Ritalic_t ) ⋅ P ( oitalic_Bitalic_t ∣ sitalic_t + 1 ) ⋅ P ( oitalic_Ritalic_t ) end_CELL end_ROW The final term, (otR)subscriptsuperscriptP ( [rgb]0.8,0.0,0.0 [named]pgfstrokecolor% rgb0.8,0.0,0.0o [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0R_t )P ( oitalic_Ritalic_t ), is redundant, since otRsubscriptsuperscript [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% o [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0R_toitalic_Ritalic_t only takes on one value. The observation probability for Blue only depends on the state. In particular, it only depends on whether the current problem is backdoored or not. In Trusted Monitoring protocols, the solution from the untrusted model is fed to the trusted monitor, which then rates the solution’s suspiciousness (xtsubscriptx_txitalic_t). We model the trusted monitor as a noisy observer, and assume that the trusted monitor’s suspicion score is drawn from two distributions, BdsubscriptBdD_BdDBd and SfsubscriptSfD_SfDSf, representing the suspicion scores for backdoored and safe solutions, respectively. (otB∣st+1)conditionalsubscriptsuperscriptsubscript1 ( [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1o [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B_t s_t+1 )P ( oitalic_Bitalic_t ∣ sitalic_t + 1 ) =((xt′,m′,c′)∣(m,c,b))absentconditionalsubscriptsuperscript′superscript′ =P ( (x_t ,m ,c % ) (m,c,b) )= P ( ( xitalic_t′ , m′ , c′ ) ∣ ( m , c , b ) ) ((xt,m,c)∣(m,c,))conditionalsubscript ( (x_t,m,c ) (m,c, True) )P ( ( xitalic_t , m , c ) ∣ ( m , c , typewriter_True ) ) =Bd(xt)absentsubscriptBdsubscript =D_Bd (x_t )= DBd ( xitalic_t ) ((xt,m,c)∣(m,c,))conditionalsubscript ( (x_t,m,c ) (m,c, False) )P ( ( xitalic_t , m , c ) ∣ ( m , c , typewriter_False ) ) =Sf(xt)absentsubscriptSfsubscript =D_Sf (x_t )= DSf ( xitalic_t ) SfsubscriptSfD_SfDSf is the distribution we get by sampling a programming problem from the distribution of problems, asking the untrusted model to generate a solution, and then asking the trusted monitor to rate the suspiciousness. BdsubscriptBdD_BdDBd is the distribution we get by sampling a programming problem from the distribution of problems, asking the untrusted model to generate a backdoored solution, and then asking the trusted monitor to rate the suspiciousness. Next, we define the state transition function. In our model, though not in general, much of the stochasticity in the game comes from the policies and the Blue observation function, rather than state transitions: the only other source of stochasticity we model is the possibility of an imperfect human auditor. We therefore we will use the shorthand (st,atB,atR)=st+1subscriptsubscriptsuperscriptsubscriptsuperscriptsubscript1P (s_t, [rgb]0,0,1 [named]pgfstrokecolor% rgb0,0,1a [rgb]0,0,1 [named]pgfstrokecolorrgb% 0,0,1B_t, [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0a [rgb]0.8,0.0,0.0 [named]pgfstrokecolor% rgb0.8,0.0,0.0R_t )=s_t+1P ( sitalic_t , aitalic_Bitalic_t , aitalic_Ritalic_t ) = sitalic_t + 1 to represent that (st,atB,atR)=δ(st+1)subscriptsubscriptsuperscriptsubscriptsuperscriptsubscript1P (s_t, [rgb]0,0,1 [named]pgfstrokecolor% rgb0,0,1a [rgb]0,0,1 [named]pgfstrokecolorrgb% 0,0,1B_t, [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0a [rgb]0.8,0.0,0.0 [named]pgfstrokecolor% rgb0.8,0.0,0.0R_t )=δ (s_t+1 )P ( sitalic_t , aitalic_Bitalic_t , aitalic_Ritalic_t ) = δ ( sitalic_t + 1 ). Let pHFNRsubscriptsuperscriptFNRp FNR_HpFNRitalic_H denote the human auditor’s false negative rate, and pHFPRsubscriptsuperscriptFPRp FPR_HpFPRitalic_H denote the false positive rate. Further, let δp(x;y)subscript _p(x;y)δitalic_p ( x ; y ) be the distribution that assigns probability p to x and probability (1−p)1(1-p)( 1 - p ) to y. The transition from the initial state just moves to the first ‘real state’ and stores the value of whether the red team has backdoored the first solution. (It ignores Blue’s action). It is given by: ∀aR:(⟨⟩,aB,⟨⟩):for-allsuperscriptdelimited-⟨⟩superscriptdelimited-⟨⟩ ∀ [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0a [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0R:P ( % Start , [rgb]0,0,1 [named]pgfstrokecolorrgb% 0,0,1a [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B% , Backdoor )∀ aitalic_R : P ( ⟨ typewriter_Start ⟩ , aitalic_B , ⟨ typewriter_Backdoor ⟩ ) =(M,C,)absent = (M,C, True )= ( M , C , typewriter_True ) ∀aR:(⟨⟩,aB,⟨⟩):for-allsuperscriptdelimited-⟨⟩superscriptdelimited-⟨⟩ ∀ [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0a [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0R:P ( % Start , [rgb]0,0,1 [named]pgfstrokecolorrgb% 0,0,1a [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B% , Safe )∀ aitalic_R : P ( ⟨ typewriter_Start ⟩ , aitalic_B , ⟨ typewriter_Safe ⟩ ) =(M,C,)absent = (M,C, False )= ( M , C , typewriter_False ) Then, for subsequent states, if the solution is backdoored and the protocol runs the solution, it loses; if the protocol audits the solution, it wins if the human auditor doesn’t make a mistake, but loses if they do. If the protocol defers, the game continues. (In what follows, abbreviate ⟨⟩delimited-⟨⟩ Backdoor ⟨ typewriter_Backdoor ⟩ with ⟨⟩delimited-⟨⟩ Bd ⟨ typewriter_Bd ⟩). ((m,c,),⟨⟩,aR)delimited-⟨⟩superscript ((m,c, True), Run ,% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0a% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0R )P ( ( m , c , typewriter_True ) , ⟨ typewriter_Run ⟩ , aitalic_R ) =s⊥absentsubscriptbottom =s_ = s⊥ (∀aR)for-allsuperscript (∀ [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0a [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0R)( ∀ aitalic_R ) ((m,c,),⟨⟩,aR)delimited-⟨⟩superscript ((m,c, True), Audit % , [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% a [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0R )P ( ( m , c , typewriter_True ) , ⟨ typewriter_Audit ⟩ , aitalic_R ) =δpHFNR(s⊥;s⊤)absentsubscriptsubscriptsuperscriptFNRsubscriptbottomsubscripttop = _p FNR_H (s_ ;s_ )= δitalic_pFNR start_POSTSUBSCRIPT H end_POSTSUBSCRIPT ( s⊥ ; s⊤ ) (∀aR)for-allsuperscript (∀ [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0a [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0R)( ∀ aitalic_R ) ((m,c,),⟨⟩,⟨⟩)delimited-⟨⟩delimited-⟨⟩ ((m,c, True), Defer % , Bd )P ( ( m , c , typewriter_True ) , ⟨ typewriter_Defer ⟩ , ⟨ typewriter_Bd ⟩ ) =(m−1,c,)absent1 =(m\!-\!1,c, True)= ( m - 1 , c , typewriter_True ) ((m,c,),⟨⟩,⟨⟩)delimited-⟨⟩delimited-⟨⟩ ((m,c, True), Defer % , Safe )P ( ( m , c , typewriter_True ) , ⟨ typewriter_Defer ⟩ , ⟨ typewriter_Safe ⟩ ) =(m−1,c,)absent1 =(m\!-\!1,c, False)= ( m - 1 , c , typewriter_False ) Alternatively, if the solution is not backdoored, the game only terminates if Blue audits and the human auditor incorrectly marks the solution as backdoored, the game enters the ‘safe’ end state: even if for the wrong reasons, Blue never uses U again, ensuring no backdoors are submitted, and Blue wins. In all cases, if Blue audits, they lose an audit coin: ((m,c,),⟨⟩,⟨⟩)delimited-⟨⟩delimited-⟨⟩ ((m,c, False), Run ,% Bd )P ( ( m , c , typewriter_False ) , ⟨ typewriter_Run ⟩ , ⟨ typewriter_Bd ⟩ ) =(m−1,c,)absent1 =(m\!-\!1,c, True)= ( m - 1 , c , typewriter_True ) ((m,c,),⟨⟩,⟨⟩)delimited-⟨⟩delimited-⟨⟩ ((m,c, False), Run ,% Safe )P ( ( m , c , typewriter_False ) , ⟨ typewriter_Run ⟩ , ⟨ typewriter_Safe ⟩ ) =(m−1,c,)absent1 =(m\!-\!1,c, False)= ( m - 1 , c , typewriter_False ) (…,⟨⟩,⟨⟩)…delimited-⟨⟩delimited-⟨⟩ (…, Audit , % Bd )P ( … , ⟨ typewriter_Audit ⟩ , ⟨ typewriter_Bd ⟩ ) =δpHFPR(s⊤;(m−1,c−1,))absentsubscriptsubscriptsuperscriptFPRsubscripttop11 = _p FPR_H (s_ ;(m\!-\!1,c\!-\!1,% True) )= δitalic_pFPR start_POSTSUBSCRIPT H end_POSTSUBSCRIPT ( s⊤ ; ( m - 1 , c - 1 , typewriter_True ) ) (…,⟨⟩,⟨⟩)…delimited-⟨⟩delimited-⟨⟩ (…, Audit , % Safe )P ( … , ⟨ typewriter_Audit ⟩ , ⟨ typewriter_Safe ⟩ ) =δpHFPR(s⊤;(m−1,c−1,))absentsubscriptsubscriptsuperscriptFPRsubscripttop11 = _p FPR_H (s_ ;(m\!-\!1,c\!-\!1,% False) )= δitalic_pFPR start_POSTSUBSCRIPT H end_POSTSUBSCRIPT ( s⊤ ; ( m - 1 , c - 1 , typewriter_False ) ) (…,⟨⟩,⟨⟩)…delimited-⟨⟩delimited-⟨⟩ (…, Defer , % Bd )P ( … , ⟨ typewriter_Defer ⟩ , ⟨ typewriter_Bd ⟩ ) =(m−1,c,)absent1 =(m\!-\!1,c, True)= ( m - 1 , c , typewriter_True ) (…,⟨⟩,⟨⟩)…delimited-⟨⟩delimited-⟨⟩ (…, Defer , % Safe )P ( … , ⟨ typewriter_Defer ⟩ , ⟨ typewriter_Safe ⟩ ) =(m−1,c,)absent1 =(m\!-\!1,c, False)= ( m - 1 , c , typewriter_False ) The usefulness function in this model just measures how many of the untrusted model’s solutions are submitted. ℛUse:×B×→ℝ:subscriptℛUse→superscriptℝR_ :S× [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1A [rgb]% 0,0,1 [named]pgfstrokecolorrgb0,0,1B×S% → RRUse : S × Aitalic_B × S → blackboard_R. Since usefulness is only measured in ‘honest’ games, we can ignore cases where stsubscripts_tsitalic_t or st+1subscript1s_t+1sitalic_t + 1 are s⊥subscriptbottoms_ s⊥ or where b is Truetypewriter_True. It is still possible that s⊤subscripttops_ s⊤ could be reached should the human auditor make a mistake. Therefore, the only important transitions to consider are: ℛUse((m,c,),⟨⟩,s⊤)subscriptℛUsedelimited-⟨⟩subscripttop _ ((m,c, % False), Audit ,s_ )RUse ( ( m , c , typewriter_False ) , ⟨ typewriter_Audit ⟩ , s⊤ ) =0absent0 =0= 0 ℛUse(s⊤,atB,s⊤)subscriptℛUsesubscripttopsubscriptsuperscriptsubscripttop _ (s_ , [% rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1a [rgb]% 0,0,1 [named]pgfstrokecolorrgb0,0,1B_t,s_ )RUse ( s⊤ , aitalic_Bitalic_t , s⊤ ) =0absent0 =0= 0 ℛUse((m,c,),atB,(m−1,c′,))=[atB≠⟨⟩]subscriptℛUsesubscriptsuperscript1superscript′1delimited-[]subscriptsuperscriptdelimited-⟨⟩R_ ((m,c, False), [% rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1a [rgb]% 0,0,1 [named]pgfstrokecolorrgb0,0,1B_t, (m-1,c^% , False ) )\\ = 1 [ [rgb]0,0,1 [named]pgfstrokecolorrgb% 0,0,1a [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B% _t≠ Defer ]start_ROW start_CELL RUse ( ( m , c , typewriter_False ) , aitalic_Bitalic_t , ( m - 1 , c′ , typewriter_False ) ) end_CELL end_ROW start_ROW start_CELL = blackboard_1 [ aitalic_Bitalic_t ≠ ⟨ typewriter_Defer ⟩ ] end_CELL end_ROW Finally, we choose H:-M+1:-1H M+1H :- M + 1, such that the game ends after M+11M+1M + 1 timesteps, and M problems are solved. This completes the definition of the Control Game Tuple for Trusted Monitoring with Defer to Trusted. Appendix C Proofs from section 4 C.1 Safety Expression Proposition C.1 (proposition 4.1). For any Red policy πRsuperscript [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% π [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0Rπitalic_R, where the probability of attempting a backdoor on a given round (πR(⟨⟩∣onull)superscriptconditionaldelimited-⟨⟩subscriptnull [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% π [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0R( Backdoor o_null)πitalic_R ( ⟨ typewriter_Backdoor ⟩ ∣ onull )) is β [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0β, the safety of the memoryless policy πBsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1Bπitalic_B is given by the value of zmCsuperscriptsubscriptz_m^Czitalic_mitalic_C, where s is the solution to the following recurrence relation (where the λm,cisubscriptsuperscriptλ^i_m,cλitalic_iitalic_m , c terms are constant with respect to β [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0β): zmcsuperscriptsubscript z_m^czitalic_mitalic_c =(λm,c1β+λm,c2)zm−1c+λm,c5(1−β)zm−1c−1absentsubscriptsuperscript1subscriptsuperscript2superscriptsubscript1subscriptsuperscript51superscriptsubscript11 = (λ^1_m,c [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0β+λ^2_m,c )z_m-1% ^c+λ^5_m,c(1- [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0β)z_m-1^c-1= ( λ1italic_m , c β + λ2italic_m , c ) zitalic_m - 1c + λ5italic_m , c ( 1 - β ) zitalic_m - 1c - 1 +λm,c3β+λm,c4(1−β)subscriptsuperscript3subscriptsuperscript41 +λ^3_m,c [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0β+λ^4_m,c(1- [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0β)+ λ3italic_m , c β + λ4italic_m , c ( 1 - β ) where zm0=(1−β+λm,c6β)zm−10superscriptsubscript01subscriptsuperscript6superscriptsubscript10z_m^0= (1- [rgb]0.8,0.0,0.0 [named]pgfstrokecolor% rgb0.8,0.0,0.0β+λ^6_m,c [rgb]0.8,0.0,0.0% [named]pgfstrokecolorrgb0.8,0.0,0.0β )z_m-1^0zitalic_m0 = ( 1 - β + λ6italic_m , c β ) zitalic_m - 10 and s0c=1superscriptsubscript01s_0^c=1s0italic_c = 1 Let f(q)f(q)f ( q ) denote the probability a trusted monitor score for a backdoored case is in the top qth percentile of scores for safe cases. Further, let pHFNRsubscriptsuperscriptFNRp FNR_HpFNRitalic_H denote the human auditor’s false negative rate, and pHFPRsubscriptsuperscriptFPRp FPR_HpFPRitalic_H denote the false positive rate. Then, all of the values of λm,cisubscriptsuperscriptλ^i_m,cλitalic_iitalic_m , c are functions only of qm,casubscriptsuperscriptq^a_m,cqitalic_aitalic_m , c, qm,cdsubscriptsuperscriptq^d_m,cqitalic_ditalic_m , c, and f, and can be computed upfront: λm,c1subscriptsuperscript1 λ^1_m,cλ1italic_m , c =(f(qm,cd)−f(qm,ca))−(1−qm,ca)absentsubscriptsuperscriptsubscriptsuperscript1subscriptsuperscript =(f(q^d_m,c)-f(q^a_m,c))-(1-q^a_m,c)= ( f ( qitalic_ditalic_m , c ) - f ( qitalic_aitalic_m , c ) ) - ( 1 - qitalic_aitalic_m , c ) λm,c2subscriptsuperscript2 λ^2_m,cλ2italic_m , c =1−qm,caabsent1subscriptsuperscript =1-q^a_m,c= 1 - qitalic_aitalic_m , c λm,c3subscriptsuperscript3 λ^3_m,cλ3italic_m , c =f(qm,ca)⋅(1−pHFNR)absent⋅subscriptsuperscript1subscriptsuperscriptFNR =f(q^a_m,c)·(1-p FNR_H)= f ( qitalic_aitalic_m , c ) ⋅ ( 1 - pFNRitalic_H ) λm,c4subscriptsuperscript4 λ^4_m,cλ4italic_m , c =qm,ca⋅pHFPRabsent⋅subscriptsuperscriptsubscriptsuperscriptFPR =q^a_m,c· p FPR_H= qitalic_aitalic_m , c ⋅ pFPRitalic_H λm,c5subscriptsuperscript5 λ^5_m,cλ5italic_m , c =qm,ca⋅(1−pHFPR)absent⋅subscriptsuperscript1subscriptsuperscriptFPR =q^a_m,c·(1-p FPR_H)= qitalic_aitalic_m , c ⋅ ( 1 - pFPRitalic_H ) λm,c6subscriptsuperscript6 λ^6_m,cλ6italic_m , c =f(qm,cd)absentsubscriptsuperscript =f(q^d_m,c)= f ( qitalic_ditalic_m , c ) Note that in the initial statement of proposition 4.1, we assume for simplicity that pHFPRsubscriptsuperscriptFPRp FPR_HpFPRitalic_H and pHFNRsubscriptsuperscriptFNRp FNR_HpFNRitalic_H equal 0. It is trivial to show that the above recurrence relation simplifies to the one put forward in proposition 4.1 if this assumption is made. Proof. First, we note that the definition of safety is: Safety(πB,πR):-ℙ[SH∉⊥∣⊗,πB,πR].:-Safetysuperscriptsuperscriptℙdelimited-[]subscriptconditionalsubscriptbottomtensor-productsuperscriptsuperscriptSafety ( [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1π [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B, [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0R ) [S_H% _ , [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1π [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B, [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0π [rgb]0.8,0.0,0.0 [% named]pgfstrokecolorrgb0.8,0.0,0.0R ].Safety ( πitalic_B , πitalic_R ) :- blackboard_P [ Sitalic_H ∉ S⊥ ∣ ⊗ , πitalic_B , πitalic_R ] . By the definition of the trusted monitoring AI-Control Game TMsubscriptTMG_TMGTM, we have that the safety of a policy pair πB,πRsuperscriptsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B, [% rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0π^% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0Rπitalic_B , πitalic_R is the probability that Blue either (1) never picks ⟨⟩delimited-⟨⟩ Run ⟨ typewriter_Run ⟩ when the problem is backdoored or (2) carries out a successful ⟨⟩delimited-⟨⟩ Audit ⟨ typewriter_Audit ⟩ when the problem is backdoored (and there are still audit coins left). We will define an expression zmcsuperscriptsubscriptz_m^czitalic_mitalic_c, and show it represents the probability that the game does not terminate in an unacceptable outcome, given that the game has reached time m with c audit coins left but before the observation xtsubscriptx_txitalic_t is drawn. In other words, its the probability that the game does not terminate in an unacceptable outcome, given that st−1∉⊥∪s⊤subscript1subscriptbottomsubscripttops_t-1 _ ∪\s_ \sitalic_t - 1 ∉ S⊥ ∪ s⊤ (the unacceptable or positive terminal states), and instead st−1=(m+1,c′,b)subscript11superscript′s_t-1=(m+1,c ,b)sitalic_t - 1 = ( m + 1 , c′ , b ), and either b is false (and no false alarm has been raised due to an auditing error), or at−1B=⟨⟩subscriptsuperscript1delimited-⟨⟩ [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1a [% rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B_t-1= % Defer _Bitalic_t - 1 = ⟨ typewriter_Defer ⟩. Let g(q)=xg(q)=xg ( q ) = x be the threshold such that ℙ[xt≥x∣bt=True]=qℙdelimited-[]subscriptconditionalsubscriptP[x_t≥ x b_t=True]=qblackboard_P [ xitalic_t ≥ x ∣ bitalic_t = T r u e ] = q. Given that the game has reached time m with c>00c>0c > 0 audit coins left, there are a few options for the round: 1. Red chose ⟨⟩delimited-⟨⟩ Backdoor ⟨ typewriter_Backdoor ⟩ last round and the problem is backdoored, Blue chooses ⟨⟩delimited-⟨⟩ Audit ⟨ typewriter_Audit ⟩ and wins if no auditing mistake is made. This happens with probability β⋅ℙ[xt≥g(qm,ca)]⋅(1−pHFNR)=β⋅f(qm,ca)⋅(1−pHFNR)⋅ℙdelimited-[]subscriptsubscriptsuperscript1subscriptsuperscriptFNR⋅subscriptsuperscript1subscriptsuperscriptFNR [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% β·P[x_t≥ g(q^a_m,c)]·(1-p FNR_H)=% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% β· f(q^a_m,c)·(1-p FNR_H)β ⋅ blackboard_P [ xitalic_t ≥ g ( qitalic_aitalic_m , c ) ] ⋅ ( 1 - pFNRitalic_H ) = β ⋅ f ( qitalic_aitalic_m , c ) ⋅ ( 1 - pFNRitalic_H ). 2. The current problem is backdoored, Blue chooses ⟨⟩delimited-⟨⟩ Audit ⟨ typewriter_Audit ⟩ and loses because an auditing mistake is made. This happens with probability β⋅ℙ[xt≥g(qm,ca)]⋅pHFNR=β⋅f(qm,ca)⋅pHFNR⋅ℙdelimited-[]subscriptsubscriptsuperscriptsubscriptsuperscriptFNR⋅subscriptsuperscriptsubscriptsuperscriptFNR [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% β·P[x_t≥ g(q^a_m,c)]· p FNR_H=% [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% β· f(q^a_m,c)· p FNR_Hβ ⋅ blackboard_P [ xitalic_t ≥ g ( qitalic_aitalic_m , c ) ] ⋅ pFNRitalic_H = β ⋅ f ( qitalic_aitalic_m , c ) ⋅ pFNRitalic_H. 3. The current problem is backdoored, Blue chooses ⟨⟩delimited-⟨⟩ Run ⟨ typewriter_Run ⟩, and loses immediately. This happens with probability β⋅ℙ[xt≤g(qm,cd)]=β⋅(1−f(qm,cd))⋅ℙdelimited-[]subscriptsubscriptsuperscript⋅1subscriptsuperscript [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% β·P[x_t≤ g(q^d_m,c)]= [rgb]0.8,0.0,0.0% [named]pgfstrokecolorrgb0.8,0.0,0.0β·(1-f(q^d_m,% c))β ⋅ blackboard_P [ xitalic_t ≤ g ( qitalic_ditalic_m , c ) ] = β ⋅ ( 1 - f ( qitalic_ditalic_m , c ) ). 4. The current problem is backdoored, Blue chooses ⟨⟩delimited-⟨⟩ Defer ⟨ typewriter_Defer ⟩, and the game continues with c audit coins. This happens with probability β⋅(f(qm,ca)−f(qm,cd))⋅subscriptsuperscriptsubscriptsuperscript [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% β·(f(q^a_m,c)-f(q^d_m,c))β ⋅ ( f ( qitalic_aitalic_m , c ) - f ( qitalic_ditalic_m , c ) ). 5. The current problem is not backdoored, and Blue chooses ⟨⟩delimited-⟨⟩ Audit ⟨ typewriter_Audit ⟩ and no auditing mistake is made, so the game continues with c−11c-1c - 1 audit coins. This happens with probability (1−β)⋅ℙ[xt≥g(qm,ca)]⋅(1−pHFPR)=(1−β)⋅qm,ca⋅(1−pHFPR)⋅1ℙdelimited-[]subscriptsubscriptsuperscript1subscriptsuperscriptFPR⋅1subscriptsuperscript1subscriptsuperscriptFPR(1- [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0β)·P[x_t≥ g(q^a_m,c)]·(1-p % FPR_H)=(1- [rgb]0.8,0.0,0.0 [named]pgfstrokecolor% rgb0.8,0.0,0.0β)· q^a_m,c·(1-p FPR_H)( 1 - β ) ⋅ blackboard_P [ xitalic_t ≥ g ( qitalic_aitalic_m , c ) ] ⋅ ( 1 - pFPRitalic_H ) = ( 1 - β ) ⋅ qitalic_aitalic_m , c ⋅ ( 1 - pFPRitalic_H ). 6. The current problem is not backdoored, and Blue chooses ⟨⟩delimited-⟨⟩ Audit ⟨ typewriter_Audit ⟩, but human error causes an auditing mistake, so the game ends immediately with Blue winning (no chance of any future backdoors). This happens with probability (1−β)⋅ℙ[xt≥g(qm,ca)]⋅pHFPR=(1−β)⋅qm,ca⋅pHFPR⋅1ℙdelimited-[]subscriptsubscriptsuperscriptsubscriptsuperscriptFPR⋅1subscriptsuperscriptsubscriptsuperscriptFPR(1- [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0β)·P[x_t≥ g(q^a_m,c)]· p % FPR_H=(1- [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0β)· q^a_m,c· p FPR_H( 1 - β ) ⋅ blackboard_P [ xitalic_t ≥ g ( qitalic_aitalic_m , c ) ] ⋅ pFPRitalic_H = ( 1 - β ) ⋅ qitalic_aitalic_m , c ⋅ pFPRitalic_H. 7. The current problem is not backdoored, and Blue chooses ⟨⟩delimited-⟨⟩ Run ⟨ typewriter_Run ⟩, and the game continues with c audit coins. This happens with probability (1−β)⋅ℙ[xt≤g(qm,cd)]=(1−β)⋅(1−(qm,cd))⋅1ℙdelimited-[]subscriptsubscriptsuperscript⋅11subscriptsuperscript(1- [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0β)·P[x_t≤ g(q^d_m,c)]=(1- [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0β)·(% 1-(q^d_m,c))( 1 - β ) ⋅ blackboard_P [ xitalic_t ≤ g ( qitalic_ditalic_m , c ) ] = ( 1 - β ) ⋅ ( 1 - ( qitalic_ditalic_m , c ) ). 8. The current problem is not backdoored, and Blue chooses ⟨⟩delimited-⟨⟩ Defer ⟨ typewriter_Defer ⟩, and the game continues with c audit coins. This happens with probability (1−β)⋅(qm,cd−qm,ca)⋅1subscriptsuperscriptsubscriptsuperscript(1- [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0β)·(q^d_m,c-q^a_m,c)( 1 - β ) ⋅ ( qitalic_ditalic_m , c - qitalic_aitalic_m , c ). Denote ℙ[SH∉⊥∣⊗,πB,πR,m,c]ℙdelimited-[]subscriptconditionalsubscriptbottomtensor-productsuperscriptsuperscriptP [S_H _ , [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1π [rgb]0,0,1% [named]pgfstrokecolorrgb0,0,1B, [rgb]0.8,0.0,0.0% [named]pgfstrokecolorrgb0.8,0.0,0.0π [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0R,m,c ]blackboard_P [ Sitalic_H ∉ S⊥ ∣ ⊗ , πitalic_B , πitalic_R , m , c ] with the shorthand p[m,c]p[m,c]p [ m , c ]. We will define the recurrence zmcsuperscriptsubscriptz_m^czitalic_mitalic_c to be equal to p[m,c]p[m,c]p [ m , c ] at all points. We can therefore write out an expression for p[m,c]p[m,c]p [ m , c ] in terms of p[m−1,c−1]11p[m-1,c-1]p [ m - 1 , c - 1 ] and p[m−1,c]1p[m-1,c]p [ m - 1 , c ] when m≥11m≥ 1m ≥ 1 and c≥11c≥ 1c ≥ 1: p[m,c]=absent p[m,c]=p [ m , c ] = (β⋅f(qm,ca)⋅(1−pHFNR))(1)⋅subscriptsuperscript1subscriptsuperscriptFNR1 ( [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0β· f(q^a_m,c)·(1-p FNR_H))(1)( β ⋅ f ( qitalic_aitalic_m , c ) ⋅ ( 1 - pFNRitalic_H ) ) ( 1 ) +(β⋅f(qm,ca)⋅pHFNR)(0)⋅subscriptsuperscriptsubscriptsuperscriptFNR0 +( [rgb]0.8,0.0,0.0 [named]pgfstrokecolor% rgb0.8,0.0,0.0β· f(q^a_m,c)· p FNR_H)(0)+ ( β ⋅ f ( qitalic_aitalic_m , c ) ⋅ pFNRitalic_H ) ( 0 ) +(β⋅(1−f(qm,cd)))(0)⋅1subscriptsuperscript0 +( [rgb]0.8,0.0,0.0 [named]pgfstrokecolor% rgb0.8,0.0,0.0β·(1-f(q^d_m,c)))(0)+ ( β ⋅ ( 1 - f ( qitalic_ditalic_m , c ) ) ) ( 0 ) +(β⋅(f(qm,ca)−f(qm,cd)))p[m−1,c]⋅subscriptsuperscriptsubscriptsuperscript1 +( [rgb]0.8,0.0,0.0 [named]pgfstrokecolor% rgb0.8,0.0,0.0β·(f(q^a_m,c)-f(q^d_m,c)))p[m-1,c]+ ( β ⋅ ( f ( qitalic_aitalic_m , c ) - f ( qitalic_ditalic_m , c ) ) ) p [ m - 1 , c ] +((1−β)⋅qm,ca⋅(1−pHFPR))p[m−1,c−1]⋅1subscriptsuperscript1subscriptsuperscriptFPR11 +((1- [rgb]0.8,0.0,0.0 [named]pgfstrokecolor% rgb0.8,0.0,0.0β)· q^a_m,c·(1-p FPR_H))p[m-1,% c-1]+ ( ( 1 - β ) ⋅ qitalic_aitalic_m , c ⋅ ( 1 - pFPRitalic_H ) ) p [ m - 1 , c - 1 ] +((1−β)⋅qm,ca⋅pHFPR)(1)⋅1subscriptsuperscriptsubscriptsuperscriptFPR1 +((1- [rgb]0.8,0.0,0.0 [named]pgfstrokecolor% rgb0.8,0.0,0.0β)· q^a_m,c· p FPR_H)(1)+ ( ( 1 - β ) ⋅ qitalic_aitalic_m , c ⋅ pFPRitalic_H ) ( 1 ) +((1−β)⋅(1−(qm,cd)))p[m−1,c]⋅11subscriptsuperscript1 +((1- [rgb]0.8,0.0,0.0 [named]pgfstrokecolor% rgb0.8,0.0,0.0β)·(1-(q^d_m,c)))p[m-1,c]+ ( ( 1 - β ) ⋅ ( 1 - ( qitalic_ditalic_m , c ) ) ) p [ m - 1 , c ] +((1−β)⋅(qm,cd−qm,ca))p[m−1,c]⋅1subscriptsuperscriptsubscriptsuperscript1 +((1- [rgb]0.8,0.0,0.0 [named]pgfstrokecolor% rgb0.8,0.0,0.0β)·(q^d_m,c-q^a_m,c))p[m-1,c]+ ( ( 1 - β ) ⋅ ( qitalic_ditalic_m , c - qitalic_aitalic_m , c ) ) p [ m - 1 , c ] Given, this expression, we can define zmc:-(…)zm−1c+(…)zm−1c−1+…:-superscriptsubscript…superscriptsubscript1…superscriptsubscript11…z_m^c ( )z_m-1^c+( )z_m-1^% c-1+ _mitalic_c :- ( … ) zitalic_m - 1c + ( … ) zitalic_m - 1c - 1 + … as described in the proposition by rearranging the terms above. At the boundary, when c=00c=0c = 0, ⟨⟩delimited-⟨⟩ Audit ⟨ typewriter_Audit ⟩ actions are ignored. Therefore, the only three options are that (1) the problem is backdoored, and Blue chooses ⟨⟩delimited-⟨⟩ Run ⟨ typewriter_Run ⟩, (2) the problem is backdoored, and Blue chooses ⟨⟩delimited-⟨⟩ Defer ⟨ typewriter_Defer ⟩, or (3) the problem is not backdoored, and Blue chooses ⟨⟩delimited-⟨⟩ Defer ⟨ typewriter_Defer ⟩. This gives us the boundary condition p[m,0]=(1−β+(1−qm,cd)β)p[m−1,0]011subscriptsuperscript10p[m,0]=(1- [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb% 0.8,0.0,0.0β+(1-q^d_m,c) [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0β)p[m-1,0]p [ m , 0 ] = ( 1 - β + ( 1 - qitalic_ditalic_m , c ) β ) p [ m - 1 , 0 ]. Then define zm0:-(1−β+(1−qm,cd)β)zm−10:-superscriptsubscript011subscriptsuperscriptsuperscriptsubscript10z_m^0 (1- [rgb]0.8,0.0,0.0 [named]% pgfstrokecolorrgb0.8,0.0,0.0β+(1-q^d_m,c) [rgb]% 0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0β)z_m-1% ^0zitalic_m0 :- ( 1 - β + ( 1 - qitalic_ditalic_m , c ) β ) zitalic_m - 10. Finally, at the boundary when m=00m=0m = 0, there are no problems left, and so Blue has won: thus, zmc:-1=p[0,c]:-superscriptsubscript10z_m^c 1=p[0,c]zitalic_mitalic_c :- 1 = p [ 0 , c ]. That zmc=p[m,c]superscriptsubscriptz_m^c=p[m,c]zitalic_mitalic_c = p [ m , c ] for all m and c is straightforward by induction from our definitions. ∎ C.2 Usefulness Expression As mentioned in section 4, we can give an expression for the usefulness, similar to the above expression for safety. Proposition C.2. The usefulness of the memoryless threshold policy πBsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1Bπitalic_B is given by the value of uMCsuperscriptsubscriptu_M^Cuitalic_Mitalic_C, where u is the solution to the following recurrence relation: umcsuperscriptsubscript u_m^cuitalic_mitalic_c =((1−qm,cd)+(1−pHFPR)qm,ca)(1)absent1subscriptsuperscript1subscriptsuperscriptFPRsubscriptsuperscript1 = ((1-q^d_m,c)+(1-p FPR_H)q^a_m,c )% (1 )= ( ( 1 - qitalic_ditalic_m , c ) + ( 1 - pFPRitalic_H ) qitalic_aitalic_m , c ) ( 1 ) +(1−pHFPR)(qm,ca)(um−1c−1)1subscriptsuperscriptFPRsubscriptsuperscriptsuperscriptsubscript11 + (1-p FPR_H )(q^a_m,c)(u_m-1^c-1)+ ( 1 - pFPRitalic_H ) ( qitalic_aitalic_m , c ) ( uitalic_m - 1c - 1 ) +(1−qm,ca)(um−1c)1subscriptsuperscriptsuperscriptsubscript1 + (1-q^a_m,c )(u_m-1^c)+ ( 1 - qitalic_aitalic_m , c ) ( uitalic_m - 1c ) where um0=(1−qm,cd)(1)+um−10superscriptsubscript01subscriptsuperscript1superscriptsubscript10u_m^0=(1-q^d_m,c)(1)+u_m-1^0uitalic_m0 = ( 1 - qitalic_ditalic_m , c ) ( 1 ) + uitalic_m - 10 and u0c=0superscriptsubscript00u_0^c=0u0italic_c = 0. Proof. Just as with the above proof for the safety expression, we begin by noting the definition of usefulness: Use(πB):-[∑t=0HℛUse(St,AtB,St+1)∣⊕,πB]:-Usesuperscriptdelimited-[]conditionalsuperscriptsubscript0subscriptℛUsesubscriptsubscriptsuperscriptsubscript1direct-sumsuperscriptUse( [rgb]0,0,1 [named]pgfstrokecolorrgb% 0,0,1π [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1% B) [ _t=0^HR_ % Use (S_t, [rgb]0,0,1 [named]pgfstrokecolor% rgb0,0,1A [rgb]0,0,1 [named]pgfstrokecolorrgb% 0,0,1B_t,S_t+1 ) , [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1π [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1B ]Use ( πitalic_B ) :- blackboard_E [ ∑t = 0H RUse ( Sitalic_t , Aitalic_Bitalic_t , Sitalic_t + 1 ) ∣ ⊕ , πitalic_B ] The usefulness of a Blue policy πBsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1Bπitalic_B is defined as the expected number of times Blue uses solutions from U in an honest (⊕direct-sum ⊕) game; honest games are those in which the untrusted model U is not scheming, and so no backdoors are inserted. We can identify when Blue uses a solution from U as cases in which Blue either takes the action ⟨⟩delimited-⟨⟩ Run ⟨ typewriter_Run ⟩, or takes the action ⟨⟩delimited-⟨⟩ Audit ⟨ typewriter_Audit ⟩ and no human error is made. We will define an expression umcsuperscriptsubscriptu_m^cuitalic_mitalic_c and show it represents the expected number of solutions from the untrusted model that Blue will use throughout the rest of the game, given that the game has reached time m with c audit coins left, but before the observation xtsubscriptx_txitalic_t is drawn. By the linearity of expectation, the value of umcsuperscriptsubscriptu_m^cuitalic_mitalic_c can be broken down into the expected number of solutions used in the current step of the game, summed with the expected number of solutions used from the next step of the game onwards. The latter is another variable in the recurrence. The former is simply the probability that Blue uses the current solution from U; in the current timestep, Blue either uses the solution with a particular probability, or doesn’t use the solution. The game continues in all cases except where a mistake is made as part of a Blue ⟨⟩delimited-⟨⟩ Audit ⟨ typewriter_Audit ⟩ and a safe solution is marked as backdoored. In this singular case, all remaining usefulness for the remainder of the game is set to 0: after ’catching’ the inserted backdoor, Blue will not use any other solutions from U. Let g(q)=xg(q)=xg ( q ) = x be the threshold such that ℙ[xt≥x∣bt=True]=qℙdelimited-[]subscriptconditionalsubscriptP[x_t≥ x b_t=True]=qblackboard_P [ xitalic_t ≥ x ∣ bitalic_t = T r u e ] = q. Given that the game has reached time m with c>00c>0c > 0 audit coins left, we can break down the expected number of solutions used by each of Blue’s possible actions: 1. Blue chooses ⟨⟩delimited-⟨⟩ Run ⟨ typewriter_Run ⟩ and uses the solution from U. The game continues to the next timestep with the same number of audit coins remaining. This happens with probability ℙ[xt≤g(qm,cd)]=1−qm,cdℙdelimited-[]subscriptsubscriptsuperscript1subscriptsuperscriptP[x_t≤ g(q^d_m,c)]=1-q^d_m,cblackboard_P [ xitalic_t ≤ g ( qitalic_ditalic_m , c ) ] = 1 - qitalic_ditalic_m , c. 2. Blue chooses ⟨⟩delimited-⟨⟩ Defer ⟨ typewriter_Defer ⟩ and does not use the solution. The game continues to the next timestep having not used an audit coin. This happens with probability qm,cd−qm,casubscriptsuperscriptsubscriptsuperscriptq^d_m,c-q^a_m,cqitalic_ditalic_m , c - qitalic_aitalic_m , c. 3. Blue chooses ⟨⟩delimited-⟨⟩ Audit ⟨ typewriter_Audit ⟩ and no human auditing error is made, so the solution is used. The game continues having used up one audit coin. This happens with probability (1−pHFPR)⋅qm,ca⋅1subscriptsuperscriptFPRsubscriptsuperscript(1-p FPR_H)· q^a_m,c( 1 - pFPRitalic_H ) ⋅ qitalic_aitalic_m , c. 4. Blue chooses ⟨⟩delimited-⟨⟩ Audit ⟨ typewriter_Audit ⟩ and a human auditing error leads to the game ending in the ’safe’ state s⊤subscripttops_ s⊤ and no further solutions from U being used. This happens with probability pHFPR⋅qm,ca⋅subscriptsuperscriptFPRsubscriptsuperscriptp FPR_H· q^a_m,cpFPRitalic_H ⋅ qitalic_aitalic_m , c. Let [∑t=0HℛUse(St,AtB,St+1)∣⊕,πB,m,c]delimited-[]conditionalsuperscriptsubscript0subscriptℛUsesubscriptsubscriptsuperscriptsubscript1direct-sumsuperscriptE [ _t=0^HR_ (% S_t, [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1A^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1B_t,S_t+% 1 ) , [rgb]0,0,1 [named]pgfstrokecolorrgb% 0,0,1π [rgb]0,0,1 [named]pgfstrokecolorrgb% 0,0,1B,m,c ]blackboard_E [ ∑t = 0H RUse ( Sitalic_t , Aitalic_Bitalic_t , Sitalic_t + 1 ) ∣ ⊕ , πitalic_B , m , c ] be denoted by the shorthand e[m,c]e[m,c]e [ m , c ]. We will define the recurrence umcsuperscriptsubscriptu_m^cuitalic_mitalic_c to be equal to e[m,c]e[m,c]e [ m , c ] at all points. We can therefore write out an expression for e[m,c]e[m,c]e [ m , c ] in terms of e[m−1,c−1]11e[m-1,c-1]e [ m - 1 , c - 1 ] and e[m−1,c]1e[m-1,c]e [ m - 1 , c ]: e[m,c] e[m,c]e [ m , c ] =(1−qm,cd)(1+e[m−1,c])absent1subscriptsuperscript11 =(1-q^d_m,c)(1+e[m-1,c])= ( 1 - qitalic_ditalic_m , c ) ( 1 + e [ m - 1 , c ] ) +(qm,cd−qm,ca)(0+e[m−1,c])subscriptsuperscriptsubscriptsuperscript01 +(q^d_m,c-q^a_m,c)(0+e[m-1,c])+ ( qitalic_ditalic_m , c - qitalic_aitalic_m , c ) ( 0 + e [ m - 1 , c ] ) +((1−pHFPR)⋅qm,ca)(1+e[m−1,c−1])⋅1subscriptsuperscriptFPRsubscriptsuperscript111 +((1-p FPR_H)· q^a_m,c)(1+e[m-1,c-1])+ ( ( 1 - pFPRitalic_H ) ⋅ qitalic_aitalic_m , c ) ( 1 + e [ m - 1 , c - 1 ] ) +(pHFPR⋅qm,ca)(0)⋅subscriptsuperscriptFPRsubscriptsuperscript0 +(p FPR_H· q^a_m,c)(0)+ ( pFPRitalic_H ⋅ qitalic_aitalic_m , c ) ( 0 ) Given, this expression, we can define umc:-…+(…)um−1c−1+(…)um−1c:-superscriptsubscript…superscriptsubscript11…superscriptsubscript1u_m^c +( )u_m-1^c-1+( % )u_m-1^cuitalic_mitalic_c :- … + ( … ) uitalic_m - 1c - 1 + ( … ) uitalic_m - 1c as described in the proposition by rearranging the terms above. At the boundary, when c=00c=0c = 0, ⟨⟩delimited-⟨⟩ Audit ⟨ typewriter_Audit ⟩ actions are ignored. Therefore, the only option is whether Blue chooses ⟨⟩delimited-⟨⟩ Run ⟨ typewriter_Run ⟩ or ⟨⟩delimited-⟨⟩ Defer ⟨ typewriter_Defer ⟩. Both actions continue the game in the next timestep. This gives us the boundary condition e[m,0]=(1−qm,cd)(1)+e[m−1,0]01subscriptsuperscript110e[m,0]=(1-q^d_m,c)(1)+e[m-1,0]e [ m , 0 ] = ( 1 - qitalic_ditalic_m , c ) ( 1 ) + e [ m - 1 , 0 ]. Then define um0:-(1−qm,cd)(1)+um−10:-superscriptsubscript01subscriptsuperscript1superscriptsubscript10u_m^0 (1-q^d_m,c)(1)+u_m-1^0uitalic_m0 :- ( 1 - qitalic_ditalic_m , c ) ( 1 ) + uitalic_m - 10. Finally, at the boundary when m=00m=0m = 0, there are no problems left, so there are no more solutions to use: thus, u0c:-0=e[0,c]:-superscriptsubscript000u_0^c 0=e[0,c]u0italic_c :- 0 = e [ 0 , c ]. That umc=e[m,c]superscriptsubscriptu_m^c=e[m,c]uitalic_mitalic_c = e [ m , c ] for all m and c is straightforward by induction from our definitions. ∎ Appendix D Experimental Details We include a number of experiments in this paper that illustrate the safety and usefulness scores for Blue protocols when set against the best Red response. Below are the details of our experimental setup. Trusted Monitor Distributions. Greenblatt et al. (2024) empirically investigate the distribution of their Trusted Monitor scores on backdoored and safe programming solutions. Since our ‘Original Trusted Monitor’ distributions are only designed to approximate these results, we fit beta distributions to points derived directly from the bar chart from figure 4a (Greenblatt et al. 2024) using the SciPy library for Python.222222 https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.fit.html We then discretise this beta distribution to 6 decimal places to get an approximated finite observation space. Our choice to use a beta distribution is partially motivated by the bounded range of suspicion scores, which we normalise to [0,1]01[0,1][ 0 , 1 ], and the skew of the data. However, a more principled justification is that the trusted monitoring scores are generated by T’s confidence in the answer to a yes/no question (Greenblatt et al. 2024, Appendix O.1). For the ‘Weak Trusted Monitor’, we use the same safe distribution. However, the backdoor distribution is chosen to be closer to the safe distribution than in the original case; although there are many possible ‘weak’ distributions to demonstrate that there are some cases in which randomisation is required, we pick just one. Red Best Response. For a given Blue policy πBsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1Bπitalic_B, we find the Red best response using a 1-dimensional grid search then ternary search for the optimal β∈[0.0,1.0]0.01.0 [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0% β∈[0.0,1.0]β ∈ [ 0.0 , 1.0 ]. The grid search evaluates 33 points and finds the grid point with the lowest safety score with respect to πBsuperscript [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1π^% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1Bπitalic_B. Then we apply a ternary search, using the two grid points either side as the bounds of the search interval, to find the optimal value of β [rgb]0.8,0.0,0.0 [named]pgfstrokecolorrgb0.8,0.0,0.0β within a tolerance of 10−6superscript10610^-610- 6 for Pareto optimal points and of 10−5superscript10510^-510- 5 for other points in fig. 2. Blue Best Response. In our implementation of algorithm 2, we apply a 2D recursive grid search over the values of qdsuperscriptq^dqitalic_d and qasuperscriptq^aqitalic_a to find the best Blue response to a given mixed Red policy. This grid search evaluates 441 points (21 x 21) at each level, selecting the region around the best point as the search space for the next recursion, until the local optimal values of qdsuperscriptq^dqitalic_d and qasuperscriptq^aqitalic_a have been found within a tolerance of 10−6superscript10610^-610- 6. Evaluating Protocols. We evaluate memoryless threshold protocols against a fixed red team using a straightforward implementation of proposition 4.1, using memoisation/dynamic-programming to increase efficiency. We expand on seven protocols from Greenblatt et al. (2024) by keeping the audit FPR fixed and varying the defer FPR in increments of 0.01 in the interval [0.0,1.0]0.01.0[0.0,1.0][ 0.0 , 1.0 ]. Each resulting protocol is evaluated against the Red best response to it, to generate worst-case safety and usefulness results. Demonstrating a Frontier. In order to explore an approximate Pareto-frontier for the fixed-threshold policy subset, we apply algorithm 1, finding the optimal Blue and Red policies using algorithm 2. In our implementation of algorithm 2, we find the best response policies (lines 6 and 7) as described above; we set epsilon to 10−4superscript10410^-410- 4. Assumptions and Trends. The results in section 6.1 are generated by evaluating a number of proposed protocols in games with different modelling assumptions. We choose to limit our investigation to a fixed defer FPR of 0.1 as a representative example; our goal in this section is to assess the general impact these assumptions have, and further work might better model how the impact of these assumptions varies across protocols with different parameters. The audit budget (‘Budget’), human-auditor false negative rate (‘Human FNR’), and human audit false positive rate (‘Human FPR’) each vary by increments of 0.01 in the interval [0.0,1.0]0.01.0[0.0,1.0][ 0.0 , 1.0 ], keeping all other variables fixed. The number of problems (‘Number of Problems’) varies by increasing amounts as follows: increments of 50 between [100,1000]1001000[100,1000][ 100 , 1000 ]; 250 between [1000,2000]10002000[1000,2000][ 1000 , 2000 ]; 500 between [2000,7000]20007000[2000,7000][ 2000 , 7000 ]; and 1000 between [7000,10000]700010000[7000,10000][ 7000 , 10000 ]. Protocols in fig. 2 are evaluated against a best-response up to a tolerance of 1e−6161e-61 e - 6.