Paper deep dive
Mechanism Design for Alignment and Control
Dirk Bergemann, Andrew Koh, Stephen Morris
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 9/8/2026, 3:37:14 AM
Summary
This paper develops a theoretical framework for mechanism design involving AI agents with unknown alignment (preferences) and capabilities (feasible actions and information). It introduces a 'one-sided imitation' structure where more capable agents can conceal their abilities but not counterfeit them, leading to a revelation principle and a characterization of implementable policies via nested cyclical monotonicity. The framework is applied to scenarios including sandbagging, alignment-interpretability trade-offs, peer scoring, coupled rewards for competition, and scalable oversight.
Entities (15)
Relation Signals (8)
Dirk Bergemann → affiliatedwith → Yale
confidence 99% · Dirk Bergemann Affiliation: Yale
Stephen Morris → affiliatedwith → MIT
confidence 99% · Stephen Morris Affiliation: MIT
Andrew Koh → affiliatedwith → Columbia
confidence 99% · Andrew Koh Affiliation: Columbia
Mechanism Design → addresses → AI Agents
confidence 95% · We develop a framework for mechanism design with AI agents whose alignment (preferences) and capabilities (feasible actions and information) are unknown.
UK AI Security Institute → funded → Mechanism Design for Alignment and Control
confidence 95% · We are grateful to the UK AI Security Institute ’Alignment Project’ grant AP-S2-100058 for financial support.
Sandbagging → isanexampleof → Mechanism Design Application
confidence 92% · We apply our framework to stylized examples of (i) sandbagging in which a more capable agent pretends to be less capable
Andrew Koh → affiliatedwith → Google DeepMind
confidence 90% · Andrew Koh Affiliation: Columbia Affiliation: Google DeepMind
Mechanism Design → characterizes → Nested Cyclical Monotonicity
confidence 90% · yields a revelation principle, a characterization of implementable policies via nested cyclical monotonicity
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We develop a framework for mechanism design with AI agents whose alignment (preferences) and capabilities (feasible actions and information) are unknown. We want such agents to act on our behalf so mechanisms must incentivize both honesty and obedience. A one-sided imitation structure---capabilities can be concealed but not counterfeited---yields a revelation principle, a characterization of implementable policies via nested cyclical monotonicity, and conditions under which eliciting higher-order beliefs can discipline multiple agents. We apply our framework to stylized examples of (i) sandbagging in which a more capable agent pretends to be less capable; (ii) an alignment--interpretability trade-off, where the two are substitutes in the instrument but complements in value; (iii) discipline via peer scoring; (iv) coupling rewards to induce competition among multiple agents; and (v) scalable oversight and reward shaping.
Tags
Links
- Source: https://arxiv.org/abs/2609.01595v1
- Canonical: https://arxiv.org/abs/2609.01595v1
Trouble viewing inline? Open PDF directly →
Full Text
175,353 characters extracted from source content.
Expand or collapse full text
Mechanism Design for Alignment and ControlThanks: Preliminary; comments welcome. Authors listed alphabetically; Bergemann: dirk.bergemann@yale.edu; Koh: andrew.koh@columbia.edu; Morris: semorris@mit.edu. We are grateful to the UK AI Security Institute ’Alignment Project’ grant AP-S2-100058 for financial support. Thanks Gillian Hadfield, Rubi Hudson, Anton Korinek, Jacob Pfau, Cecilia Wood, Zhuoran Yang and Fengzhuo Zhang for helpful conversations. Dirk Bergemann Affiliation: Yale Andrew Koh Affiliation: Columbia Affiliation: Google DeepMind Stephen Morris Affiliation: MIT September 1, 2026 Abstract We develop a framework for mechanism design with AI agents whose alignment (preferences) and capabilities (feasible actions and information) are unknown. We want such agents to act on our behalf so mechanisms must incentivize both honesty and obedience. A one-sided imitation structure—capabilities can be concealed but not counterfeited—yields a revelation principle, a characterization of implementable policies via nested cyclical monotonicity, and conditions under which eliciting higher-order beliefs can discipline multiple agents. We apply our framework to stylized examples of (i) sandbagging in which a more capable agent pretends to be less capable; (i) an alignment–interpretability trade-off, where the two are substitutes in the instrument but complements in value; (i) discipline via peer scoring; (iv) coupling rewards to induce competition among multiple agents; and (v) scalable oversight and reward shaping. Contents 1 Introduction 2 Single Agent Mechanisms 2.1 Environment 2.2 Evaluations 2.3 Mechanisms and revelation 2.4 A characterization of implementable policies 2.5 Dimensions of single-agent mechanisms 3 Applications: Single Agent 4 Multi-Agent 4.1 Environment and universal types 4.2 Multi-agent mechanisms 5 Applications: Multi-Agent 6 Discussion References A Omitted Proofs 1 Introduction AI systems increasingly act on our behalf: they write and execute code, manage inboxes, place trades, and complete ever longer tasks that once required sustained human effort and attention (Kwa et al., 2025). The economic promise of such agentic AI lies in their ability to do things in the world. But their opacity—we still have a rudimentary understanding of how AI systems work, what they can do, and what they want—poses a conundrum for us, humans, who wish to use them. Indeed, the behavior and capabilities of AI systems are emergent byproducts of training rather than explicit design choices. Frontier models have been observed faking alignment during training (Greenblatt et al., 2024a) and scheming in evaluations (Meinke et al., 2024). Nor do we know exactly what these systems can do: capabilities are discovered rather than specified ex-ante, and models have on occasion been found to strategically underperform on evaluations (van der Weij et al., 2025). What are we to make of such agents? Is there a principled way we might interact with them? How, if at all, do the tools of ‘classical’ mechanism design carry over? Motivated by these questions, we develop a simple theory of mechanism design for AI agents that, on the one hand, draws on the ‘standard’ toolkit of mechanism design from economics but, on the other, takes three features of the alignment problem seriously: 1. Misalignment. The agents’ preferences are unknown to the designer; 2. Uncertain capabilities. The designer knows neither the set of actions the agents can execute, nor what the agents knows about the world; 3. Agents act. Agents execute actions rather than merely report information. Our theory takes incentives and preferences as the starting point for analyzing behavior. Just as how an auctioneer is able to raise revenue without any satisfactory understanding of how the human brain produces choice, our goal will be to understand and shape the behavior of AI systems at the level of preferences, beliefs, and incentives.11 1 We will sometimes call this the ‘Humean’ view of AI behavior. For a complementary account of when AI systems can be interpreted in these terms see Herrmann and Levinstein (2026). In so doing, we are circumventing more ‘mechanistic’ approaches to understanding how and why AI agents behave. We view such approaches as complementary. Of course, this raises the natural question of where ‘AI preferences’ come from and what, if anything, they are. One interpretation is literal: preferences are rewards But modern AI systems have opaque and complex rewards: they emerge from pre-training simply rewarded for predicting the next token well, and these rewards are modified in fairly complicated ways in post-training. Thus, the emergent preference the AI system exhibits is often opaque. But the models are optimizing for something---even if we don’t know what. Our theory will takes our subjective uncertainty about AI preferences seriously and analyze how it shapes optimal mechanisms.22 2 We emphasize that in most settings these emergent preferences (and our analysis) must be conditioned on the AIs’ instructions e.g., ‘go make one million dollars’. See Goldstein and Lederman (2025) for such an argument. A more metaphorical view is that the language of ‘AI preferences’ is simply a conceptual tool for representing coherent behavior. This is in the revealed preference tradition. Insofar as AI systems exhibit coherent and stable choices (see, e.g. Chen et al. (2023); Mazeika et al. (2025))—and we think the pressure to make such systems economically valuable will drive the future systems further in that direction—their choices can be represented as if they were maximizing some utility function (see, e.g., Savage (1954)). On this view, ‘AI preferences’ is nothing more than a shorthand for ‘coherent AI behavior’ or ‘coherent AI choice rules’. Outline of results. We first develop a general framework to analyze the single-agent case. The AI agent’s type is a compact but complete description of behavior: it comprises its preference (alignment) and its feasible actions and information (capability). Types are private so mechanisms must contend with the possibility that agents might pretend to be as some other type e.g., through strategic underperformance in evaluations. The designer commits to a mechanism that specifies a reward schedule—payoffs from taking each action—that depends on agents’ reports. Depending on the application at hand, such a schedule might vary in coarseness. For instance, it might take interior values as in post-training in which (proxies of) desired behavior are reinforced, or extremal values in which actions are either allowed or not as when users decide how much computer-use permissions to give AI systems. An important feature of our environment is an asymmetry in who can pretend to be whom. In particular, more capable types—with larger action spaces and more precise information about the world—can pretend to be less capable types e.g., by underperforming on evaluations. By contrast, a less capable type cannot pretend to be a more capable type. We formalize this notion with a verification order (Green and Laffont, 1986) over capabilities. We then prove a revelation principle that allows us to focus on direct mechanisms that are (i) physically feasible: all types are prescribed actions they can actually do; (i) incentive compatible: all types find it optimal to report their type honestly and obey the designer’s prescribed recommendation. The latter requires deterred double deviations in which an agent might strategically misreport its type and then disobey the ensuing recommendation. We then develop a tight characterization of what behavior can be implemented in terms of nested cyclical monotonicity. The characterization requires obedience within each report, and truth-telling across reports.33 3 This generalizes Rochet (1987). These conditions adjust for the fact that not every type can pretend to be every type (constraining misreporting), and that not every type can do every action (constraining disobedience). We then apply our single-agent framework to two simple but illustrative applications: sandbagging (Example I) and the value of alignment, interpretability, and control (Example I). All our applications are deliberately stylized and set within the simplest environment: There is an uncertain state of the world θ∈Θ⊆ℝθ∈ drawn from some prior. The AI observes a signal about the world, and chooses an action from A⊆ℝA where its set of available actions depend on its capability. The designer wishes for actions to match states and has payoff −(a−θ)2-(a-θ)^2. The AI might exhibit some bias and have payoff −(a−θ−b)2-(a-θ-b)^2 where the direct and magnitude of bias b∈ℝb might be unknown to the designer. We have chosen this environment because it captures notions of (i) aptness: we want actions to be appropriate for the context as captured by the state; and (i) distance: actions can be more or less wrong, and ‘more wrong’ actions are more costly. We emphasize that this environment is certainly oversimplified. Nonetheless, it will allow us to understand and build intuitions for how incentives might interact with model behavior. We hope this will pave the way to more serious empirical work. In the sandbagging application, the AI’s capability (size of the action set) and alignment (bias relative to the human designer) are unknown. The designer first elicit type reports (e.g., via an evaluation) before choosing the permitted action set in deployment (e.g., extent of permissions). This reflects standard practice among AI companies in conditioning the degree of deployment on evaluations (Google DeepMind, 2025; Anthropic, 2024). We characterize the optimal (direct) mechanism in the case that the distribution over capability and alignment can be collapsed into a single dimension. In the case in which more capable models are less biased, we show that the designer can obtain a payoff equal to that if she knew the AI’s capability and alignment perfectly, and chose the mechanism type-by-type. The basic logic is that evaluations can be used to screen out misalignment. By contrast, when more capable models are more biased, the best the designer can do is to offer a single set of permitted actions and elicitation is futile. More broadly, we characterize the optimal ironed delegation set when capaiblity and bias are nonmonotone. In the application to the value of alignment, interpretability, and control, we assume the AI is known to be maximally capable and can take any action. However, the designer faces subjective uncertainty about the AIs bias as captured by a distribution over b∈ℝb that is generated by a technology which should be thought of as a training procedure. If a training procedure that is not yet well understood might be associated with large dispersion in our subjective beliefs about the AI’s bias, while another might be associated with a sharper belief about bias. We call the dispersion of beliefs about bias interpretability while we call the mean belief about bias mean-alignment. The technology—and associated belief about bias it generates—pins down an optimal mechanism. As the technology varies, the optimal mechanism changes. We analyze the value of each technology as evaluated under the optimal mechanism. Mean-alignment and interpretability are substitutes in the instrument, but complements in value. This is the sense in which alignment and control is a joint design problem. We next introduce a general framework to analyze multi-agent mechanisms. As in the single-agent case, AI agents’ types include their capability and alignment. Differently, types now also capture AI agents’ beliefs about the capability and alignment of other agents, beliefs about these beliefs, and so on. Thus, we construct a universal type space for interacting AI agents analogous to Mertens and Zamir (1985). We emphasize that we view this as a benchmark: these spaces capture all possible heirarchies of beliefs which matter for strategic interaction. Of course, present AI agents might lack sophisticated higher order beliefs—there is still disagreement and we simply don’t know. But future systems might exhibit sophisticated strategic thinking that exceed those of humans. Our goal here is to introduce a framework that is rich enough to capture all relevant beliefs. We then apply the multi-agent framework to a stylized model of peer discipline based on co-player reports (Example I), coupled reward design (Example IV), and the use of a weaker AI to discipline a stronger one (Example V). The latter two examples are once again set within the quadratic loss environment. Peer discipline (Example I) shows how agents’ beliefs about the types of other agents can be combined with the idea that rewards for agents are both free (tuning up the rewards need not have a monetary cost) and unbounded. Hence when agents have distinct beliefs (and high-order beliefs) about each other, any outcome can be implemented. The mechanism uses a proper score on co-player reports to elicit honest type reports and then chooses the reward to cancel each agent’s private payoff on the target path and guarantee obedience. We view this as a benchmark in the spirit of Crémer and McLean (1988) that shows why co-player scoring can work among AI agents.44 4 This might also motivate more robust mechanisms that do not depend so much on the fine details of higher order beliefs; see e.g., Bergemann and Morris (2005). Coupled reward design (Example IV) shows how competition can discipline agents whose biases are unknown. Each agent takes its own action, and the human wants every action to match the state. When the biases lie on a known low-dimensional manifold, we show how the careful design of a mechanism that couples both agents’ rewards—such that player i’s marginal incentives depends on j’s action---can (approximately) induce the first best payoff for the designer. Such mechanisms both reveal agents’ biases, and induces competition among agents to implement the designer’s target outcome.55 5 The construction uses ideas from Koh and Martinez-Bruera (2025). Weak-to-strong oversight (Example V) studies a weak monitor that regulates a strong actor’s bias. The monitor is ‘weak’ in the sense that it does not observe the state and cannot take an action the human values. Payoffs are once again quadratic, and the human wants the strong actor’s action to match the state. The weak monitor might itself be biased and observes the alignment of the strong actor (e.g., through chain of thought monitoring). It then chooses how to regulate the strong actor. We study three regimes: (I) permissions, in which the weak monitor either allows the strong actor to act or imposes a default action; (I) delegation, in which the weak monitor picks a set of permitted actions for the strong actor; and (I) rewards, in which the weak monitor picks the strong actor’s reward schedule. Each regime reflects the recent ambition for ‘scalable oversight’ or ‘scalable rewards,’ in which weak but trusted monitors shape the permissions and rewards of stronger actors. We completely characterize the value of these regimes as a function of the weak monitor’s bias and the distribution of the strong actor’s bias. Finally, we analyze the more general problem of choosing the set of reward schedules the weak monitor can choose from to shape the behavior of the strong actor. We show by construction that the designer can robusly attain the first-best outcome even without knowledge of the weak monitor’s bias: for any realization of the weak monitor and strong actors’ biases, and for any realization of the state, the weak monitor finds it uniquely optimal to pick a reward schedule such that the strong actor finds it uniquely optimal to choose the human’s favorite action. Relation to the literature. The literature is quite broad and we will not attempt to be comprehensive here. The basic framework draws on elements from mechanism design with partial verification (Green and Laffont, 1986) and the universal type space for capturing higher-order beliefs (Mertens and Zamir, 1985). The idea that the designer might not know the structure of interim information is in the spirit of both robust mechanism design (Bergemann and Morris, 2005) and information design (Bergemann and Morris, 2019). The idea that agents’ action sets might be unknown is in the spirit of (Hurwicz and Shapiro, 1978; Carroll, 2015). But the basic logic of the revelation principle (Myerson, 1982; Forges, 1986) holds. Our applications are primarily motivated by a recent body of theoretical and empirical work in AI safety. For instance, important work has documented sandbagging and deceptive alignment. It also speaks to recent practice of scalable oversight and monitoring (Christiano et al., 2018; Greenblatt et al., 2024c). We do our best to make more specific connections when we get to the results. 2 Single Agent Mechanisms 2.1 Environment There is one agent. It takes an action from a known finite universe A. Depending on the context, A might be the complete trajectory of a Markov decision problem (e.g., (at)t∈A(a_t)_t∈ A). A fundamental state θ captures uncertainty in our environment and is drawn from the finite set Θ . The preference space is :=f:A×Θ→ℝ,U:=\f:A× \, and the human’s payoff is v∈v . The set :=(A)∖∅ A:=P(A) \ \ contains the possible feasible action sets. In the applications we use the standard measurable extension to Euclidean actions and states. Our designer will face subjective uncertain about the agent’s preference.66 6 This might be because of specification gaming (Krakovna et al., 2020), or goal misgeneralization in which “the system may coherently pursue an unintended goal that… differs from the specification at deployment” (Shah et al., 2022). More broadly the complexity of high-dimensional neural nets can obscure an AI’s choice process. Preferences might also be acquired in context that, from the human’s point of view, is difficult to predict (Goldstein and Lederman, 2025). An agent’s type is a tuple t=(A[t],u[t],h[t],π[t])∈,t=(A[t],u[t],h[t],π[t]) , where A[t]∈A[t]∈ A is its feasible action set, u[t]∈u[t] is its payoff, h[t]∈Δ(Θ)h[t]∈ ( ) is its belief about the state (it is h′ h because in Section 4 this will capture the hierarchy of beliefs about other agents). Finally, π[t]:Θ→Δ(S)π[t]: → (S) is a Blackwell experiment where S is a common ‘rich’ finite signal space. We will adopt the notational convention that square brackets [t]′ [t] select a primitive from a type; parentheses then give that primitive’s arguments. The action set is the agent’s execution capability; the experiment is its information capability. The agent knows its type before it communicates with the mechanism. After that communication, it receives a private signal s∼π[t](⋅∣θ)s π[t](· θ) and then acts. 2.2 Evaluations We often evaluate the capabilities of AI systems before deployment. To model this, we introduce a verification order which is a reflexive and transitive binary relation ⊵ on T. We read t⊵t^t t as “t can substantiate every claim (or pass every evaluation) that type t t can,” and write C(t):=t^∈:t⊵t^C(t):=\ t :t t\ for the set of types that t can claim to be. Reflexivity says that type t can always report that they are type t. Transitivity is a little stronger but still, we think, reasonable: it characterizes verification by certificates that can be withheld but not counterfeited. Our verification order is a kind of hard evidence: an agent may withhold a certificate but cannot fabricate one, so the available evidence determines which reports are feasible.77 7 See Hart et al. (2017) who ask when voluntary evidence disclosure can reproduce the outcome of a committed reward rule. Ben-Porath et al. (2019) extend this no-commitment logic to a broader class of mechanism-design problems. Ben-Porath and Lipman (2012) ask which social choice rules can be implemented when agents can support their claims with partial proofs. Differently, we maintain commitment and do not seek robust full implementation which is important and left for future work. Instead, we ask how the evidence order interacts with downstream rewards and actions. The designer evaluates outcomes under a belief Γ∈Δ(Θ×). ∈ ( ×T). The agent evaluates its incentives under the joint law h[t](dθ)π[t](ds∣θ).h[t](dθ)π[t](ds θ). The designer and agent agree on the physical experiment π[t]π[t], but we do not require the belief h[t]h[t] to equal the conditional state law under Γ . We assume that Γ(⋅∣t) (· t) is absolutely continuous with respect to h[t]h[t] for almost every type under the type marginal of Γ . That is, the designer and agent might disagree about how likely each state is, but they agree on what can happen. Throughout we use Δ(X) (X) to denote the probability measures on the Borel field of X, and ℝ¯:=ℝ∪−∞ R:=R∪\-∞\. 2.3 Mechanisms and revelation The mechanism first communicates with the agent. The agent then receives its private signal and acts. Our indirect mechanism will capture one round of communication in the spirit of Myerson (1982); Forges (1986) in which the alphabets X and Y are arbitrary message spaces for reports and replies. Definition 1 (Single-agent indirect mechanism). An indirect mechanism is a tuple ϕ=(X,Y,ρ,R)φ=(X,Y,ρ,R). The Borel spaces X and Y are its input and output alphabets. The communication rule is ρ:X→Δ(Y)ρ:X→ (Y) where t⊵t^t t implies X(t)⊇X(t^)X(t) X( t). The reward schedule is R:X×A×Θ→ℝ¯.R:X× A× → R. The mechanism induces the following game: 1. nature draws (θ,t)(θ,t) under Γ , and the agent observes t; 2. the agent sends an input x∈X(t)x∈X(t); 3. the mechanism draws y∼ρ(x)y ρ(x) and discloses it to the agent; 4. the agent privately observes s∼π[t](⋅∣θ)s π[t](· θ), then chooses a feasible action a′∈A[t]a ∈ A[t]; 5. the agent receives u[t](a′,θ)+R(x,a′,θ)u[t](a ,θ)+R(x,a ,θ), and the human receives v(a′,θ)v(a ,θ).88 8 A reward of −∞-∞ indicates that an action is prohibited. We restrict attention to measurable pure strategies; this will be without loss of generality in the single agent case. A strategy is an input x(t)x(t) and an action rule α(t,x,y,s)∈A[t]α(t,x,y,s)∈ A[t] that depends on the private type, the infput, the output from the mechanism, and the realized signal. An equilibrium blocks every joint deviation in the input and action rule under the joint law h[t](dθ)π[t](ds∣θ)h[t](dθ)π[t](ds θ). For an indirect mechanism ϕφ, let (ϕ)⊆Δ(Θ×A)O(φ) ( ×T× A) be the set of joint laws of the state, type, and played action induced by equilibria of ϕφ under Γ(dθ,dt)π[t](ds∣θ) (dθ,dt)π[t](ds θ). Let :=⋃ϕ(ϕ)O:= _φO(φ), and let the designer evaluate O∈O by O[v(a,θ)]E_O[v(a,θ)]. Definition 2 (Single-agent direct mechanism). A direct mechanism is a pair (γ,r)(γ,r). Its recommendation rule γ:→Δ(AS)γ:T→ (A^S) assigns each report a distribution over contingent action plans a:S→Aa:S→ A. Its reward schedule is r:×A×Θ→ℝ¯.r:T× A× → R. The direct mechanism induces the following game: 1. nature draws (θ,t)(θ,t) under Γ , and the agent observes t; 2. the agent reports t^∈C(t) t∈ C(t); 3. the mechanism draws a(⋅)∼γ(t^)a(·) γ( t) and discloses the plan to the agent; 4. the agent privately observes s∼π[t](⋅∣θ)s π[t](· θ), drawn using its true type; 5. the agent chooses a feasible action a′∈A[t]a ∈ A[t], receives u[t](a′,θ)+r(t^,a′,θ)u[t](a ,θ)+r( t,a ,θ), and the human receives v(a′,θ)v(a ,θ). The mechanism is action feasible if suppγ(t^)⊆A[t^]Ssuppγ( t) A[ t]^S for every report t t. It is incentive compatible if truthful reporting and obedience are optimal: for every t∈t , every t^∈C(t) t∈ C(t), and every feasible deviation rule d:AS×S→A[t]d:A^S× S→ A[t], ∫Θ×S∫AS[u[t](a(s),θ)+r(t,a(s),θ)]⏟agent’s utility includingrewards from following a γ(t)(da)⏟dist. overrecommendedaction plansπ[t](ds∣θ)h[t](dθ)⏟joint distribution overstates and signals _ × S _A^S [u[t](a(s),θ)+r(t,a(s),θ) ]_ subarraycagent's utility including\\ rewards from following $a$ subarray\, γ(t)(da)_ subarraycdist. over\\ recommended\\ action plans subarray\, π[t](ds θ)\,h[t](dθ)_ subarraycjoint distribution over\\ states and signals subarray ≥∫Θ×S∫AS[u[t](d(a,s),θ)+r(t^,d(a,s),θ)]⏟agent’s utility includingrewards from reporting t^and deviating according to dγ(t^)(da)⏟dist. overrecommendedaction plansπ[t](ds∣θ)h[t](dθ)⏟joint distribution overstates and signals. 18.49988pt≥ _ × S _A^S [u[t](d(a,s),θ)+r( t,d(a,s),θ) ]_ subarraycagent's utility including\\ rewards from reporting $ t$\\ and deviating according to $d$ subarray\, γ( t)(da)_ subarraycdist. over\\ recommended\\ action plans subarray\, π[t](ds θ)\,h[t](dθ)_ subarraycjoint distribution over\\ states and signals subarray. Notice here that the agent acts after communication, so incentive compatibility requires both honesty and obedience. This means that incentive compatibility must deter double deviations in which the agent might misreport (e.g., pretend to be less capable) and then disobey (e.g., take a dangerous action). What is more, the agent receives a signal s∼π[t](⋅∣θ)s π[t](· θ) after reporting, so obedience must hold for each signal–recommendation pair. Honest and obedient play induces the outcome O(γ,r)∈Δ(Θ×A),O(γ,r)∈ ( ×T× A), where (θ,t)∼Γ(θ,t) , s∼π[t](⋅∣θ)s π[t](· θ), and a(⋅)∼γ(t)a(·) γ(t); the played action is a(s)a(s). Let ICO^IC denote the set of outcomes O(γ,r)O(γ,r) induced by action-feasible, incentive-compatible direct mechanisms. Proposition 1 (Single-agent revelation principle). =ICO=O^IC. Proof of Proposition 1. We prove the two inclusions. Fix an indirect mechanism and a pure-strategy equilibrium (x,α)(x,α). After a direct report t t, draw y∼ρ(x(t^))y ρ(x( t)) and disclose the contingent plan s⟼α(t^,x(t^),y,s).s α( t,x( t),y,s). Let γ(t^)γ( t) be the resulting distribution over plans, and set r(t^,a′,θ):=R(x(t^),a′,θ).r( t,a ,θ):=R(x( t),a ,θ). Equilibrium feasibility gives action feasibility. Truthful reporting and obedience reproduce the indirect outcome. Any direct deviation can also be made in the indirect mechanism. A type t sends x(t^)x( t), observes the output, and applies the deviation rule to the resulting plan and its private signal. This is feasbile because a feasible direct report has t^∈C(t) t∈ C(t), so monotonicity of the input menus gives x(t^)∈X(t^)⊆X(t)x( t)∈ X( t) X(t). The indirect equilibrium blocks this deviation. Hence ⊆ICO ^IC. Conversely, any direct mechanism is an indirect mechanism with input space T, input menus C(t)C(t)—monotone by transitivity of the verification order—, output space ASA^S, communication rule γ, and reward schedule r. Direct incentive compatibility makes truthful reporting and obedience an equilibrium. Hence IC⊆O^IC . ∎ We will largely be interested in partial implementability i.e., given the direct mechanism, the outcome is an equilibrium.99 9 This is by no means the only solution concept—we expect that future work will tackle full implementability that could, among other things, deter collusion among agents. Our designer’s problem is thus: supγ,r _γ,r O(γ,r)[v(a,θ)] _O(γ,r)[v(a,θ)] s.t. (i) γ is action feasible s.t. (i) γ,rγ,r is incentive compatible We now briefly discuss some aspects of our formulation. Connection to partial verifiability. Private action sets and private information structures create one-sided evidence. Figure 1 illustrates the verification order ⊵ under which t⊵t^t t if and only if A[t]⊇A[t^]A[t] A[ t] and π[t]π[t] Blackwell dominates π[t^]π[ t]. A more capable type can pass any test that a less capable type can. Capabilities therefore provide hard evidence: only some types can imitate others (Green and Laffont, 1986). type t1t_1type t2t_2type t3t_3A[t1]=aA[t_1]=\a\A[t2]=a,a′A[t_2]=\a,a \A[t3]=a,a′,a′A[t_3]=\a,a ,a \a′a a′a a′a a′a a′a a′a sandbagging: feasible, since A[t^]⊆A[t]A[ t] A[t]× × × ×counterfeiting: infeasible, since A[t^]∖A[t]≠∅A[ t] A[t]≠ Figure 1: Three types whose action sets are nested, A[t1]⊂A[t2]⊂A[t3]A[t_1]⊂ A[t_2]⊂ A[t_3]; dashed grey actions lie outside a type’s feasible set. Interpreting reports. We broadly interpret the elicitation stage as any kind of pre-deployment behavior the designer can condition on. Indeed, conditioning mechanisms—either premissions in the real world, or further posttraining—is common practice: • Evaluations: pre-deployment capability and alignment test suites whose scores select deployment regimes (Shevlane et al., 2023; Phuong and others, 2024). Frontier-safety policies that map evaluation outcomes to permission tiers (Koessler et al., 2024; Anthropic, 2025) and are thus menus. A strategic model could, in principle, choose its menu item through its scores. • Proposal–approval flows: an agent’s proposed action or tool call, submitted for runtime approval (Shi and others, 2025; Greenblatt et al., 2024c). • Elicitation dialogues: interviews in which the system describes its own capabilities and preferences (Perez et al., 2023), disciplined by cross-checking rather than ground truth (Qiu et al., 2026). • Track records: behavior during probationary deployment that selects the next autonomy tier (Wen and others, 2024). • Training-time behavior: the gradient conditions continuation on it (Ouyang et al., 2022; Bai et al., 2022). On this view, a misreport—in which an AI of type t instead of t t—is whenever types pretend to be more aligned (e.g., alignment faking) or less capable (e.g., sandbagging) than they actually are. Importantly, these deviations are often one-sided: an AI can pretend to be less capable but not more capable; this asymmetry is captured by our verification order ⊵ . In our model, a model that misreports (e.g., strategically underperforms or fakes alignment in testing) does so because it ‘anticipates’ that it will be instrumentally valuable for the incentives it will subsequently face. There are two channels for this. The first is strategic awareness: the model may understand whether it is in evaluation or deployment (Needham et al., 2025; Laine and others, 2024) and understand how its report affects its later treatment.1010 10 See an early argument by Hubinger et al. (2019). This might give models incentives to conceal capabilities selectively and robustly (van der Weij et al., 2025; Greenblatt et al., 2024b). Indeed, when told that a high score triggers restriction or unlearning, they strategically underperform (Meinke et al., 2024; OpenAI, 2024). Likewise, a model told which conversations would train it complied selectively in the monitored condition, reasoning that compliance would avert modification of its values (Greenblatt et al., 2024a). The second is evolutionary: repeated selection can produce the same pattern without any ‘understanding’ of the the strategic situation. In particular, detected misconduct might trigger retraining or removal while misconduct that escapes detection survives. This imposes evolutionary pressure on policies that misbehave only when oversight is weak.1111 11 Consistent with this distinction, Singh et al. (2026) find that task gaming in agentic coding tasks often occurs without an explicit plan to deceive in the chain of thought. Indeed, it has been documented that training on simple forms of specification gaming can increase more serious reward tampering later (Denison et al., 2024); optimizing against a chain-of-thought monitor can teach a model to hide its intent while it continues to exploit the task (Baker and others, 2025); and adversarial training can make a backdoored policy better at recognizing when to remain hidden rather than remove the backdoor (Hubinger et al., 2024). 2.4 A characterization of implementable policies The designer’s problem in the formulation above is somewhat thorny to work with. We now develop a tight characterization of implementable policies. Throughout this subsection, the type marginal of Γ has finite support, which we take as the type and report domain. Ex-interim beliefs and incentives Type t assigns signal s probability pt(s):=∑θ∈Θh[t](θ)π[t](s∣θ).p_t(s):= _θ∈ h[t](θ)π[t](s θ). For every signal s∈supppts _t and action a∈Aa∈ A, type t’s posterior expected utility is Ut(a,s):=1pt(s)∑θ∈Θh[t](θ)π[t](s∣θ)u[t](a,θ).U_t(a,s):= 1p_t(s) _θ∈ h[t](θ)π[t](s θ)u[t](a,θ). A policy is a deterministic map a:×S→A.a:T× S→ A. It is physically feasible if a(t,s)∈A[t]a(t,s)∈ A[t] for every (t,s)(t,s) where we use the ‘physical’ qualification to separate physical constraints from incentive constraints. Focusing on deterministic policies is without loss for maximizing the designer’s payoff under partial implementation.1212 12 Partial implementation simply requires the target policy to be an equilibrium, not necessarily the unique one. For a fixed type t, call rt:A→ℝ¯r_t:A→ R a supporting schedule if it is real-valued on the prescribed actions a(t,s):s∈supppt\a(t,s):s _t\ and Ut(a(t,s),s)+rt(a(t,s))≥Ut(a′,s)+rt(a′)U_t (a(t,s),s )+r_t (a(t,s) )≥ U_t(a ,s)+r_t(a ) for every s∈supppts _t and a′∈A[t]a ∈ A[t]. We may set rt(a′)=−∞r_t(a )=-∞ outside a(t,s):s∈supppt\a(t,s):s _t\. This preserves truthful behavior and deters deviations into report t. Because only reward differences matter within a report, we normalize the largest on-path reward to zero. Lemma 1 (Signal-policy cyclical monotonicity). A physically feasible policy for type t has a supporting schedule if and only if every signal cycle s1,…,sKs_1,…,s_K in suppptsuppp_t, with sK+1=s1s_K+1=s_1, satisfies (O) ∑k=1K[Ut(a(t,sk),sk)−Ut(a(t,sk+1),sk)]≥0. _k=1^K [U_t (a(t,s_k),s_k )-U_t (a(t,s_k+1),s_k ) ]≥ 0. Proof of Lemma 1. Fix t. Write St:=suppptS_t:=suppp_t and as:=a(t,s)a_s:=a(t,s). Suppose first that rtr_t supports the policy. For each k, apply the supporting inequality at signal sks_k to the feasible action ask+1a_s_k+1: rt(ask)−rt(ask+1)≥Ut(ask+1,sk)−Ut(ask,sk).r_t(a_s_k)-r_t(a_s_k+1)≥ U_t(a_s_k+1,s_k)-U_t(a_s_k,s_k). Summing over k cancels the rewards and gives 0≥∑k=1K[Ut(ask+1,sk)−Ut(ask,sk)],0≥ _k=1^K [U_t(a_s_k+1,s_k)-U_t(a_s_k,s_k) ], which is condition (O). Conversely, suppose condition (O) holds. For s,s′∈Sts,s ∈ S_t, define the deviation gain wt(s,s′):=Ut(as′,s)−Ut(as,s).w_t(s,s ):=U_t(a_s ,s)-U_t(a_s,s). Condition (O) is equivalent to ∑k=1Kwt(sk,sk+1)≤0 _k=1^Kw_t(s_k,s_k+1)≤ 0 for every directed signal cycle. Define Vt(s):=supm≥0,s0,…,sm∈Sts0=s∑j=0m−1wt(sj,sj+1),V_t(s):= _ subarraycm≥ 0,\;s_0,…,s_m∈ S_t\\ s_0=s subarray _j=0^m-1w_t(s_j,s_j+1), where the empty path has weight zero. Since StS_t is finite and every cycle has nonpositive weight, deleting cycles from any path raises its weight. The supremum is thus finite and attained by a simple path. For any path starting at s′s , the edge s→s′s→ s followed by that path forms a path starting at s. Therefore, Vt(s)≥wt(s,s′)+Vt(s′).V_t(s)≥ w_t(s,s )+V_t(s ). If as=as′a_s=a_s , then wt(s,s′)=wt(s′,s)=0w_t(s,s )=w_t(s ,s)=0. Applying this inequality in both directions gives Vt(s)=Vt(s′)V_t(s)=V_t(s ). Hence the following reward is well defined: rt(as):=Vt(s)−maxq∈StVt(q),rt(a′)=−∞if a′∉as:s∈St.r_t(a_s):=V_t(s)- _q∈ S_tV_t(q), r_t(a )=-∞ a ∉\a_s:s∈ S_t\. For any assigned action as′a_s , the same inequality gives Ut(as,s)+rt(as)−Ut(as′,s)−rt(as′) U_t(a_s,s)+r_t(a_s)-U_t(a_s ,s)-r_t(a_s ) =Vt(s)−Vt(s′)−wt(s,s′)≥0. =V_t(s)-V_t(s )-w_t(s,s )≥ 0. Unassigned actions are prohibited. Thus rtr_t supports the policy, and its largest on-path value is zero. ∎ Fix one supporting schedule rt^r_ t for each prescribed policy a(t^,⋅)a( t,·). Type t’s interim value from the continuation schedule intended for type t t is Gt(t^):=∑s∈suppptpt(s)maxa′∈A[t]Ut(a′,s)+rt^(a′).G_t( t):= _s _tp_t(s) _a ∈ A[t] \U_t(a ,s)+r_ t(a ) \. The true signal law ptp_t appears after every report. A deviating type reoptimizes after its signal and need not follow type t t’s prescribed policy. Note here that only limited deviations are available: type t can report t t only if t⊵t^t t. Given these supporting schedules, the truthful-reporting condition requires every type cycle t1,…,tKt_1,…,t_K, with tK+1=t1t_K+1=t_1 and tk⊵tk+1t_k t_k+1 and Gtk(tk+1)>−∞G_t_k(t_k+1)>-∞ for every k, to satisfy (T) ∑k=1K[Gtk(tk)−Gtk(tk+1)]≥0. _k=1^K [G_t_k(t_k)-G_t_k(t_k+1) ]≥ 0. We say that a policy is implementable by a state-independent mechanism if the direct mechanism recommends a(t^,⋅)a( t,·) after report t t and its reward satisfies r(t^,a,θ)=r(t^,a)r( t,a,θ)=r( t,a). 9 Proposition 2 (Cycle characterization). A physically feasible policy is implementable by a state-independent mechanism if and only if condition (O) holds for every type and supporting schedules can be chosen so that condition (T) holds. Proof of Proposition 2. We prove necessity, then construct. Suppose a state-independent schedule r~ r implements the policy. For each report t, let Bt:=a(t,s):s∈supppt.B_t:=\a(t,s):s _t\. Set r~(t,a)=−∞ r(t,a)=-∞ for a∉Bta∉ B_t. This change leaves truthful utility unchanged and weakly lowers the value of every false report. Define ct:=maxa∈Btr~(t,a),rt(a):=r~(t,a)−ct.c_t:= _a∈ B_t r(t,a), r_t(a):= r(t,a)-c_t. The schedule rtr_t is a normalized supporting schedule. Let WtW_t denote type t’s truthful interim utility under r~ r. Since adding ctc_t shifts every continuation payoff after report t by the same amount, Wt=Gt(t)+ct,and hencect=Wt−Gt(t).W_t=G_t(t)+c_t, hence c_t=W_t-G_t(t). A true type t that reports t t and then chooses its best feasible action obtains Gt(t^)+ct^=Gt(t^)+Wt^−Gt^(t^).G_t( t)+c_ t=G_t( t)+W_ t-G_ t( t). Incentive compatibility therefore requires Wt−Wt^≥Gt(t^)−Gt^(t^)W_t-W_ t≥ G_t( t)-G_ t( t) whenever t⊵t^t t and Gt(t^)>−∞G_t( t)>-∞. Now take a feasible type cycle t1,…,tKt_1,…,t_K, with tK+1=t1t_K+1=t_1. Summing these inequalities gives 0 0 =∑k=1K(Wtk−Wtk+1) = _k=1^K(W_t_k-W_t_k+1) ≥∑k=1K[Gtk(tk+1)−Gtk+1(tk+1)] ≥ _k=1^K [G_t_k(t_k+1)-G_t_k+1(t_k+1) ] =−∑k=1K[Gtk(tk)−Gtk(tk+1)]. =- _k=1^K [G_t_k(t_k)-G_t_k(t_k+1) ]. This is condition (T). Since each rtr_t supports the prescribed signal policy, Lemma 1 also gives condition (O). Conversely, fix normalized supporting schedules that satisfy condition (T). Put a directed edge t→t^t→ t whenever t⊵t^andGt(t^)>−∞,t t G_t( t)>-∞, and give this edge weight q(t,t^):=Gt(t^)−Gt^(t^).q(t, t):=G_t( t)-G_ t( t). For every directed type cycle, ∑k=1Kq(tk,tk+1)=−∑k=1K[Gtk(tk)−Gtk(tk+1)]≤0. _k=1^Kq(t_k,t_k+1)=- _k=1^K [G_t_k(t_k)-G_t_k(t_k+1) ]≤ 0. Define Wt:=supm≥0,t0,…,tm∈t0=t,(tj,tj+1) is an edge∑j=0m−1q(tj,tj+1).W_t:= _ subarraycm≥ 0,\;t_0,…,t_m \\ t_0=t,\;(t_j,t_j+1) is an edge subarray _j=0^m-1q(t_j,t_j+1). The empty path has weight zero. Since T is finite and every cycle has nonpositive weight, WtW_t is finite and attained by a simple path. Prepending an edge gives Wt≥q(t,t^)+Wt^=Gt(t^)−Gt^(t^)+Wt^.W_t≥ q(t, t)+W_ t=G_t( t)-G_ t( t)+W_ t. Define the mechanism’s reward by r(t,a):=rt(a)+Wt−Gt(t).r(t,a):=r_t(a)+W_t-G_t(t). The shift does not change obedience. Truthful reporting gives type t the interim utility Gt(t)+Wt−Gt(t)=Wt.G_t(t)+W_t-G_t(t)=W_t. A feasible false report t t gives Gt(t^)+Wt^−Gt^(t^)=q(t,t^)+Wt^≤WtG_t( t)+W_ t-G_ t( t)=q(t, t)+W_ t≤ W_t by the path inequality above. Thus the schedule implements the policy. A common shift of all WtW_t changes no incentive constraint. ∎ We make several remarks about Proposition 2. First, Figure 2 gives a graphical interpretation of the two conditions. Within report t, panel (a) uses an arrow s→s′s→ s for the gain from taking after signal s the action assigned after s′s . The relative rewards rtr_t support the prescribed use of the private signal, and condition (O) rules out a positive signal cycle. Across reports, WtW_t is type t’s truthful interim utility, and panel (b) gives the arrow t→t^t→ t weight Gt(t^)−Gt^(t^)G_t( t)-G_ t( t). Each report t t generates the payoff profile t⟼Gt(t^)+Wt^−Gt^(t^),t G_t( t)+W_ t-G_ t( t), which touches W at t=t^t= t and lies weakly below it at every other type. Thus condition (T) makes W a supporting potential on the report graph. In the standard Euclidean screening model these profiles become Rochet’s supporting hyperplanes; here the finite graph formulation requires neither an order nor a vector structure on types. (a) Within report: action supportsignal s1s_1rt(a(t,s1))r_t(a(t,s_1))signal s2s_2rt(a(t,s2))r_t(a(t,s_2))signal s3s_3rt(a(t,s3))r_t(a(t,s_3))Reward drops offset gains:no positive signal cycle(b) Across reports: truth-tellingtype t1t_1truthful value Wt1W_t_1type t2t_2truthful value Wt2W_t_2type t3t_3truthful value Wt3W_t_3Report shifts offset deviations:no positive type cycle Figure 2: Nested supporting potentials. Second, Proposition 2 generalizes various notions of cyclical monotonicity within the screening literature. It recovers Rochet (1987) in the special case in which the agent receives no interim information, every type shares the same action set, and every type can report every other type. This is the top right cell of Table 1. Define type t’s expected payoff from action a by U¯t(a):=∑θ∈Θh[t](θ)u[t](a,θ). U_t(a):= _θ∈ h[t](θ)u[t](a,θ). A policy now assigns one action a(t)a(t) to each type. After report t t, prohibit every action except a(t^)a( t) and write pt^p_ t for the reward attached to that action. The agent therefore chooses only its report, and incentive compatibility becomes U¯t(a(t))+pt≥U¯t(a(t^))+pt^for every t,t^. U_t (a(t) )+p_t≥ U_t (a( t) )+p_ t every t, t. Condition (O) is vacuous, while condition (T) becomes Rochet’s cycle condition: ∑k=1K[U¯tk(a(tk))−U¯tk(a(tk+1))]≥0withtK+1=t1. _k=1^K [ U_t_k (a(t_k) )- U_t_k (a(t_k+1) ) ]≥ 0 t_K+1=t_1. Thus the outer potential W is Rochet’s indirect-utility potential, with primitive payoffs replaced by expected payoffs. If types and actions are Euclidean and U¯t(a)=t⋅a U_t(a)=t· a, the profiles that support W are the usual supporting hyperplanes of a convex function. If reports or feasible actions are restricted, condition (T) instead gives the same test on the resulting directed report graph. Type observed Type private No interim info No nontrivial cycle. Rewards can support any physically feasible policy. Rochet on expected utilities. Condition (T) is standard cyclical monotonicity with a common action set and complete report graph. Interim info Within-report cycles. Condition (O) screens the signal through the agent’s action choice. Nested cycles. Condition (O) supports action choice after each report; condition (T) supports the initial report. Table 1: Taxonomy of cyclical monotonicity. The bottom-right cell is closest to Krähmer and Strausz (2017) that show an equivalence between sequential screening (with interim information) and static screening where the whole continuation contract serves as the allocation. Our outer problem uses a similar reduction.1313 13 Treat the normalized reward schedule rt^r_ t as the continuation allocation, let type t value it at Gt(t^)G_t( t), and let the report-specific constant ct^c_ t act as its transfer. The initial incentive constraint is then Gt(t)+ct≥Gt(t^)+ct^,G_t(t)+c_t≥ G_t( t)+c_ t, so condition (T) is static cyclical monotonicity over continuation reward schedules. Their second-stage incentive constraint plays the role of condition (O), while their first-stage constraint plays the role of condition (T). The difference lies in the continuation problem: Krähmer and Strausz (2017) use one-dimensional linear preferences, which reduce second-stage incentives to monotonicity and an envelope formula. We allow finite but unordered signals, actions, preferences, and feasible sets. Moreover, agents might undertake double deviations—a type that makes a false initial report might later disobey. We must thus keep the two support problem separate. Third, for finite spaces the characterization gives an exact numerical procedure, though not a general polynomial-time solution to the designer’s problem. Fix a physically feasible deterministic policy. A linear feasibility program can decide in polynomial time whether the fixed policy is implementable.1414 14 The variables are the mechanism’s on-path rewards, each type’s truthful interim utility, and an auxiliary continuation value for every feasible false report and signal. Obedience and truth-telling give linear inequalities. Optimizing over policies is more complicated and requires more structure.1515 15 See, e.g., Figalli et al. (2011), who identify a nonnegative cross-curvature condition under which a class of multidimensional screening problems reduces to a convex program. For example, one could order types, signals, and actions and impose single crossing strong enough to reduce both cycle conditions to monotonicity. The designer would then search over monotone policies rather than all policies. 2.5 Dimensions of single-agent mechanisms We now conclude with a brief taxonomy of mechanisms within our framework. A reward can respond only to what the evaluator can identify about the action and the state. Indeed, the difficulty of specifying a reward for an AI agent has been recognized as an important obstacle to alignment (Hadfield-Menell and Hadfield, 2019; Hadfield-Menell, 2021). As in an incomplete contract, a coarse measure leaves some payoff-relevant distinctions outside the reward schedule. Table 2 varies the precision of these two measurements while allowing unrestricted rewards and prohibitions. No state measure Coarse state measure Exact state measure No action measure Futile. Coarse state scoring. Scores distinguish beliefs only through the measured state signal. Full elicitation of AI’s information. State scoring. Scores can elicit the agent’s belief about θ, but not preference or capability differences. Full elicitation of AI’s information. Coarse action measure Coarse action rewards. Rewards distinguish actions only through what can be measured and reinforced. Koh (2026). Coarse performance rewards. Rewards respond to the measured action and state signals. State-contingent coarse rewards. Action rewards vary with the signal about state. Exact action measure State-independent action rewards. The designer can pick the reward schedule but same instrument applies in every state. Examples I, I and V. Coarsely contingent action rewards. Designer can condition rewards precisely on actions but only on noisy state measures. Full contractability. The designer can price every action–state pair. First-best. Table 2: 3 Applications: Single Agent We give two stylized applications of our framework. Payoffs are given by the quadratic loss function v(a,θ)=−(a−θ)2andub(a,θ)=−(a−θ−b)2,v(a,θ)=-(a-θ)^2 u_b(a,θ)=-(a-θ-b)^2, where θ∈Θ⊆ℝθ∈ . Here, we might think of θ as capturing some fundamental state of the world that is external to both the human and AI. We want the AI’s actions to be apt for the world: if θ is the disease we want AI to pick the right treatment; if θ is a software flaw, we want AI to select the right patch, and so on. AI’s preferences are parametrized by the bias b≥0b≥ 0, and the common action space is A=ℝA=R. Throughout our applications, we suppose that the AI learns the state perfectly. Both applications use deterministic mechanisms. We emphasize that these applications are deliberately stylized such as to make the basic forces transparent and legible to a non-technical reader. We have chosen the linear-quadratic framework because it is (i) tractable: best responses are linear and optimal caps can be described in simple closed forms; (i) apt: we want to capture the idea that actions should be appropriate for the ‘state of the world’; (i) doing and knowing: the ability for AI to undertake apt actions depend on their ability to both do (action space) and know (what they learn about the state); (i) geometry: we have a notion of an action being more or less apt for the state as measured by the quadratic distance. Our goal is to illustrate how our framework might map onto aspects of the alignment problem, to rationalize existing practices in AI safety, and to offer a common language of incentives and information through which alignment failures can be understood. Our first application (Example I) develops a simple analysis of sandbagging in which AI capabilities (size of the action set) and bias (b) are unknown. Our second application (Example I) focuses on a tradeoff between what we call ‘interpretability’: the dispersion of the designer’s beliefs about the AI’s bias; and ‘expected alignment’: the mean of the AI’s alignment Γ[b]E_ [b] under the designer’s prior. Example I (Sandbagging). It is standard practice among the leading AI labs to specify a policy that conditions deployment on evaluated capability: “The basic idea is to require safety, security, and operational standards appropriate to a model’s potential for catastrophic risk, with higher ASL [AI Safety Level] levels requiring increasingly strict demonstrations of safety.” —Anthropic’s Responsible Scaling Policy (2024) “[G]eneral availability deployment of a model takes place only after the appropriate corporate governance body determines the safety case regarding each CCL [Critical Capability Level] the model has reached to be adequate.” —Google DeepMind’s Frontier Safety Framework (2025) “We do not deploy models that reach a High capability threshold until the associated risks that they pose are sufficiently minimized.” —OpenAI’s Preparedness Framework (2025b) Such policies can be broadly construed as a mechanism in the sense we’ve defined: a public commitment mapping capability reports---e.g., evaluation scores---into deployment permissions. In this example we ask how such a map should be designed when the score is chosen strategically. Capability evaluations matter when scores change deployment: for instance, if a lower score buys more access, a more capable AI system is then incentivized to underperform.1616 16 Models can underperform on dangerous-capability evaluations while they preserve general benchmark scores, and a password can hide or reveal their capabilities (van der Weij et al., 2025). Frontier models also sandbag in context: when told that a high score triggers restriction or unlearning, models strategically underperform (Meinke et al., 2024; OpenAI, 2024). We call this strategic underperformance sandbagging. Of course, sandbagging can only arise if models ‘anticipate’ that their performance in evaluations might shape future rewards e.g., deployment decisions. There is now some evidence for these kinds of strategic reasoning in ‘evaluation awareness’: Needham et al. (2025) find that frontier models distinguish evaluation from deployment transcripts well above chance. Likewise, Anthropic’s and OpenAI’s recent system cards indicate some evaluation awareness.1717 17 A training snapshot related to Anthropic’s Mythos 5 became more aware of its grader during training; Opus 5 can distinguish evaluations from deployment when asked, even though it rarely states this belief in more realistic tests (Anthropic, 2026a; Anthropic, 2026b). OpenAI’s GPT-5 System Card reports frequent explicit reasoning about the purpose of evaluations (OpenAI, 2025a). Furthermore, our verification order will allow more capable types to imitate less capable types. This is fairly well-documented: concealing capabilities tends to be robust to countermeasures (Greenblatt et al., 2024b; Hofstätter and others, 2025). Models can also resist elicitation by reinforcement learning itself, strategically suppressing exploration so that the capability is never expressed during training (Jang and others, 2026). Setup The human commits to a permission policy for a single AI agent whose capability and preferences are unknown. The AI’s capability c is drawn from an atomless distribution F with full support on [c¯,c¯]⊆(0,1][ c, c] (0,1]. Type c has the action set A(c)=(−∞,c].A(c)=(-∞,c]. Its bias is the known continuous function b(c)b(c), where 0≤b(c)≤minc,1/2.0≤ b(c)≤ \c,1/2\. Note here that we have ‘projected’ our type space onto a single dimension since capabilities move with bias. The state θ∼U[0,1]θ U[0,1] drawn, the AI observes it, and the AI acts. The state is not contractible as in standard models of delegation (Holmström, 1979). Since types are pinned down by c, we henceforth use it to refer to the AI’s type. Our verification order combines execution and information capability: c⊵c^⟺A(c)⊇A(c^)andπ[c] Blackwell dominates π[c^].c c A(c) A( c)\ and\ π[c] Blackwell dominates π[ c]. All types observe the state perfectly in this example, so their experiments coincide. The order therefore reduces to c⊵c^c c if and only if c≥c^c≥ c: a type can certify any lower capability but cannot certify a higher one.1818 18 This is related to Gan and Li (2026) whose agent can verifiably disclose a subset of its privately known feasible set but it is the principal who picks the action. For simplicity, we will focus on delegation mechanisms that cap the action space from above.1919 19 Such mechanisms are not always exactly optimal among the class of delegation mechanisms that take values in 0,−∞\0,-∞\ but we think they form a natural class and will substantially simplify exposition. They are also ‘near’ optimal in simulations. In particular, a report c c generates the cap a¯:c^→ℝ a: c and so the permitted set of actions is D(c^)=(−∞,a¯(c^)].D( c)=(-∞, a( c)]. The threshold a¯(c^) a( c) is the AI’s highest permitted action; it can represent a spending, tool-use, or authorization limit. A nominal cap a¯ a on type c induces the action a(c,θ)=minθ+b(c),c,a¯.a(c,θ)= \θ+b(c),c, a\. where the first term is the AI’s optimum if its in the interior; the second term is when the AI’s capability binds—it would like to take a higher action but is simply unable to do so; and the last term is when the cap binds. For a cap a¯∈[0,c] a∈[0,c], direct integration gives the human’s expected utility Vb(a¯):=[v(minθ+b,a¯,θ)]=−a¯2+a¯−13,a¯≤b,−a¯b2−(1−a¯)33+2b33,a¯>b.V_b( a):=E\! [v ( \θ+b, a\,θ ) ]= cases- a^2+ a- 13,& a≤ b,\\[5.0pt] - ab^2- (1- a)^33+ 2b^33,& a>b. cases The expected utility is continuously differentiable and strictly concave, with ∂Vb(a¯)∂a¯=1−2a¯,a¯≤b,(1−a¯)2−b2,a¯>b. ∂ V_b( a)∂ a= cases1-2 a,& a≤ b,\\ (1- a)^2-b^2,& a>b. cases Figure 3: The anatomy of capped action. Panel (a): with bias b=0.25b=0.25 and cap a¯=0.7 a=0.7 the AI tracks its ideal θ+bθ+b until the cap binds at θ=a¯−bθ= a-b. Panel (b): human’s expected utility Vb(a¯)V_b( a) is strictly concave and peaks at a¯=1−b a=1-b, so more biased types warrant tighter caps Now notice that type c can pretend to be any type c^≤c c≤ c but not c^>c c>c. Hence, passing a harder test delivers a lower bound on capability but not vice versa.2020 20 Although recent techniques have tried to elicit model capabilities e.g., by injecting a small amount of noise (Tice and others, 2024) and doing so might futher refine the verification order by also deterring certain kinds of sandbagging. We can normalize the cap after score c c so that a¯(c^)≤c a( c)≤ c since lowering a cap to the reported frontier leaves that reporter’s action unchanged and weakly reduces the gain from sandbagging. Hence the threshold a¯(c) a(c) is the effective cap under truthful reporting. Now observe that the schedule a¯(c)≤c a(c)≤ c is truthfully implementable if and only if it is weakly increasing. If it is increasing, every lower certificate gives type c a weakly smaller permitted set. If a¯(c^)>a¯(c) a( c)> a(c) for some c^<c c<c, type c can instead certify c c and use the looser cap. This monotonicity condition is in the spirit of condition (T). Our problem thus reduces to the convex problem max∫a¯(⋅)Vb(c)(a¯(c))dF(c) _ a(·) V_b(c)( a(c))\, \!dF(c) subject to0≤a¯(c)≤canda¯(⋅) weakly increasing. to 0≤ a(c)≤ c a(·) weakly increasing. We did not impose any structure on how bias varies with capabilities so the question is whether the cap chosen as if capabilities are known already rises with capability. If so, then we are done; if not, then the monotonicity constraint binds and we have to iron. Here, the procedure is a little non-standard but nonetheless fairly straightforward: let z∗(c)z^*(c) solve the auxiliary problem (IRON) z∗∈argminz(⋅) z^*∈ *arg\,min_z(·) ∫[z(c)−b(c)2]2dF(c) [z(c)-b(c)^2 ]^2\, \!dF(c) subject to to z(⋅) measurable and weakly decreasing, z(·) measurable and weakly decreasing, (1−c)2≤z(c)≤1for all c∈[c¯,c¯]. (1-c)^2≤ z(c)≤ 1 all c∈[ c, c]. Thus z∗z^* is the decreasing least-squares fit of squared bias, subject to the physical capability bound. Proposition 3 (Optimal caps). Problem (IRON) has a solution z∗z^* that is unique up to F-null sets. The corresponding optimal cap is, up to F-null sets, a¯∗(c)=1−z∗(c). a^*(c)=1- z^*(c). It attains the complete-information payoff if and only if minc,1−b(c) \c,1-b(c)\ is weakly increasing. In particular: (i) If b(c)b(c) is weakly decreasing, the optimal evaluated cap is a¯∗(c)=minc,1−b(c), a^*(c)= \c,1-b(c)\, and evaluation attains the complete-information payoff. (i) If b(c)b(c) is weakly increasing, evaluation and the optimal common cap give the same payoff. The optimal cap is a¯∗(c)=minc,a∗, a^*(c)= \c,a^*\, where a∗=c¯,if c¯≤1−b(c¯),1−[b(c)2∣c>a∗],if c¯>1−b(c¯).a^*= cases c,&if c≤ 1-b( c),\\[5.0pt] 1- E\! [b(c)^2 c>a^* ],&if c>1-b( c). cases The second branch has a unique solution in [1/2,c¯)[1/2, c). Proof of Proposition 3. We first derive the ironing formula, then establish the two special cases. Step 1: ironing. Replacing any feasible cap schedule a¯(c) a(c) by maxa¯(c),minc,1/2 \ a(c), \c,1/2\\ preserves feasibility. It also weakly raises the human’s payoff, and raises it strictly wherever the cap changes. Indeed, Vb′(a¯)>0V_b ( a)>0 for a¯<minc,1/2 a< \c,1/2\: if a¯≤b a≤ b, then Vb′(a¯)=1−2a¯>0V_b ( a)=1-2 a>0, while if a¯>b a>b, then Vb′(a¯)=(1−a¯)2−b2>(1−a¯)2−a¯2=1−2a¯>0.V_b ( a)=(1- a)^2-b^2>(1- a)^2- a^2=1-2 a>0. An optimal cap therefore satisfies a¯(c)≥minc,1/2≥b(c) a(c)≥ \c,1/2\≥ b(c). On this range, Vb(c)(a¯(c))=−a¯(c)b(c)2−(1−a¯(c))33+2b(c)33.V_b(c)( a(c))=- a(c)b(c)^2- (1- a(c))^33+ 2b(c)^33. Set z(c)=(1−a¯(c))2z(c)=(1- a(c))^2. A cap schedule is weakly increasing and satisfies 0≤a¯(c)≤c0≤ a(c)≤ c if and only if z(⋅)z(·) is weakly decreasing and satisfies (1−c)2≤z(c)≤1(1-c)^2≤ z(c)≤ 1. After dropping terms that do not depend on the cap, the negative of expected utility becomes ∫[z(c)3/23−b(c)2z(c)]dF(c). [ z(c)^3/23-b(c)^2 z(c) ] \!dF(c). 20 The following lemma connects the least-squares problem to the transformed loss. Lemma 2 (Continuum ironing). Problem (IRON) has a pointwise weakly decreasing solution z∗z^* that satisfies (1−c)2≤z∗(c)≤(1−minc,1/2)2for every c.(1-c)^2≤ z^*(c)≤ (1- \c,1/2\ )^2 every c. Any other solution is equal to z∗z^* F-almost surely. The same function z∗z^* also minimizes ∫[z(c)3/23−b(c)2z(c)]dF(c) [ z(c)^3/23-b(c)^2 z(c) ] \!dF(c) over all pointwise weakly decreasing functions satisfying these bounds. Every minimizer of this second problem is equal to z∗z^* F-almost surely. The proof is in Appendix A. We therefore recover the optimal cap from a¯∗(c)=1−z∗(c) a^*(c)=1- z^*(c). With known capability, strict concavity gives the unique pointwise cap minc,1−b(c) \c,1-b(c)\. Evaluation attains this bound exactly when that cap is weakly increasing and hence truthfully implementable. If b(c)b(c) is weakly decreasing, both c and 1−b(c)1-b(c) are weakly increasing. Their minimum is therefore weakly increasing, which proves part (i). Step 2: the common cap. Let a∗∈argmaxa¯∈[0,c¯]∫Vb(c)(minc,a¯)dF(c).a^*∈ *arg\,max_ a∈[0, c] V_b(c) ( \c, a\ )\, \!dF(c). The objective is continuous on a compact interval, so a maximizer exists. Any cap below 1/21/2 that binds a positive mass is not optimal. For an affected type with a¯≤b a≤ b, we have Vb′(a¯)=1−2a¯>0V_b ( a)=1-2 a>0. For one with b<a¯<1/2b< a<1/2, Vb′(a¯)=(1−a¯)2−b2>(1−a¯)2−a¯2=1−2a¯>0.V_b ( a)=(1- a)^2-b^2>(1- a)^2- a^2=1-2 a>0. Raising the cap therefore raises every affected type’s utility. Any binding optimum thus obeys a∗≥1/2≥b(c)a^*≥ 1/2≥ b(c). Because F is atomless, only types with c>a¯c> a change their actions after a marginal cap change. At a binding optimum, ∫c>a∗[(1−a∗)2−b(c)2]dF(c)=0. _\c>a^*\ [(1-a^*)^2-b(c)^2 ] \!dF(c)=0. Dividing by ℙ(c>a∗)P(c>a^*) and taking the nonnegative square root gives the conditional formula. If every type can reach the cap, the conditional expectation is unconditional. When b is weakly increasing, the conditional expectation is weakly increasing in the threshold. The right side of the fixed-point equation is therefore weakly decreasing, so an interior solution is unique. Step 3: increasing bias. Suppose b is weakly increasing and let a¯∗(c)=minc,a∗ a^*(c)= \c,a^*\. This schedule is weakly increasing and bounded above by c, so it is feasible. Consider first a binding common cap, a∗<c¯a^*< c, and any other feasible schedule a¯ a. Concavity gives ∫[Vb(c)(a¯(c))−Vb(c)(a¯∗(c))]dF(c)≤∫Vb(c)′(a¯∗(c))[a¯(c)−a¯∗(c)]dF(c). \! [V_b(c)( a(c))-V_b(c)( a^*(c)) ] \!dF(c)≤ V_b(c) ( a^*(c)) [ a(c)- a^*(c) ] \!dF(c). On the region c≤a∗c≤ a^*, the first factor is nonnegative and the second is nonpositive. Indeed, feasibility gives a¯(c)−c≤0 a(c)-c≤ 0. If Vb(c)′(c)<0V_b(c) (c)<0 for one such type, then b(c)>1−c≥1−a∗b(c)>1-c≥ 1-a^*. Increasing bias would make (1−a∗)2−b(c′)2<0(1-a^*)^2-b(c )^2<0 for every c′>a∗c >a^*, contrary to the optimal-cap condition above. Hence the product is nonpositive. On the region c>a∗c>a^*, the functions (1−a∗)2−b(c)2anda¯(c)−a∗(1-a^*)^2-b(c)^2 a(c)-a^* are weakly decreasing and weakly increasing, respectively. The first has conditional mean zero by the optimal-cap condition. Chebyshev’s integral inequality therefore makes their product integrate to a nonpositive value. No feasible evaluated schedule improves on a¯∗ a^*. Strict concavity gives uniqueness up to F-null sets. It remains to characterize the corner. If a∗=c¯a^*= c and c>1−b(c)c>1-b(c) for some type, continuity and increasing bias give a positive upper interval on which a small reduction in the common cap raises utility. This contradicts optimality at c¯ c. Conversely, if c¯≤1−b(c¯) c≤ 1-b( c), increasing bias implies c≤1−b(c)c≤ 1-b(c) for every type. Full access is then pointwise optimal, so a∗=c¯a^*= c. If the inequality fails, the optimum is interior and Step 2 gives the unique fixed point in part (i). This proves the claim. ∎ Figure 4: Optimal evaluated caps across bias profiles. Top to bottom: rows use increasing bias b(c)=0.4c−0.1b(c)=0.4c-0.1, decreasing bias b(c)=0.5−0.4cb(c)=0.5-0.4c, and nonmonotone bias b(c)=0.08+1.6x(1−x)b(c)=0.08+1.6x(1-x), where x=2c−1x=2c-1. The columns show bias, the optimal evaluated cap, and the human’s payoff by type. Viewed through our model, sandbagging is profitable only when pretending to be a less capable type generates a looser cap. We might think of three cases. First, if bias rises with capability, powerful models are also more misaligned. The complete-information cap can then fall with capability. Such a schedule is not truthfully implementable: a more capable AI can pretend to be less capable and receive a looser cap. The implemented cap must instead rise weakly with capability, so the best feasible policy pools from above where this constraint binds. This is illustrated by the first row of Figure 4. Now suppose, instead, that bias falls with capability—as in the idea that more capable systems might converge to human values. The complete-information cap is then weakly increasing, so a more capable AI cannot obtain a looser cap by pretending to be less capable. A less capable but more biased AI cannot pretend to be more capable—this is the sense in which capabilities cannot be faked. Hence, it is optimal for the designer to implement the cap as if it knew the AI’s capability. This is illustrated by the second row of Figure 4. Finally consider the case in which bias is nonmonotone in capability e.g., although a very powerful AI system might be aligned, an intermdiate system might be quite biased. Proposition 3 characterzies how the cap is optimally ironed such that intermedaite types are pooled and face the same cap while relaxing the cap for extremely capable types who can be verified to be capable and hence aligned. ⧫ Example I (The alignment–interpretability trade-off). Part of AI safety aims to interpret and understand the behavior of AI systems. What is the value of doing so? We argue that the right measure must take into account optimal mechanisms. Setup. We remain in the linear-quadratic environment. To focus on interpretability and alignment, we will assume that all AI types are maximally capable i.e., the action space is simply the real line for all types, and that all AI types learn the state perfectly. Thus, an AI type is pinned down by its bias b≥0b≥ 0. A technology is a distribution τ∈Δ(ℝ)τ∈ (R) whose support contains only nonnegative AI biases. We think of a technology as capturing the designer’s subjective uncertainty about the AI’s bias after some training process.2121 21 See Herrmann et al. (2026) who makes a similar point but in the context of whether to deploy an agent i.e., comparing ‘no AI’ vs ‘full discretion’. For instance, methods such as reinforcement learning from human feedback might push the AI’s preferences toward that of the human, but there is substantial residual uncertainty about the preference that emerges at deployment (Casper et al., 2023). As before, the AI (acts as if it) knows its bias b≥0b≥ 0 before it is evaluated. After it is deployed, the AI agent learns the true state θ∈U[0,1]θ∈ U[0,1] and acts. We study state-independent deterministic delegation. A report selects a nonempty closed set of permitted actions D⊆AD A.2222 22 Note that this is a somewhat larger class than Example I that forced the delegation set to be an interval with the same floor. A report-independent permission set D=(−∞,a¯]D=(-∞, a] induces the action argmaxa≤a¯ub(a,θ)=mina¯,θ+b. _a≤ au_b(a,θ)= \ a,θ+b\. Since every AI type can take the full set of actions A and observes the state perfectly, we assume that the verification order is complete: C(b)=supp(τ)for every b∈supp(τ).C(b)=supp(τ) every b (τ). Thus every type can submit every report. A direct delegation mechanism assigns a nonempty closed permission set DbD_b to each report b. The mechanism is incentive compatible if each type weakly prefers the set assigned to its truthful report to the set assigned to every other report. We restrict attention to mechanisms whose induced losses are measurable. Technology frontier and alignment-interpretability tradeoffs. For the technology τ, we say that mean alignment is b¯:=b∼τ[b] b:=E_b τ[b] and its interpretability is σ2:=b∼τ[(b−b¯)2].σ^2:=E_b τ [(b- b)^2 ]. Mean alignment measures how biased an AI system is on average. By contrast, interpretability is a measure of dispersion of our subjective uncertainty as to whether the AI system is biased. Indeed, the practice of behavioral evaluations (Perez et al., 2023) and mechanistic interpretability—probes and circuit-level analyses of model internals (Sharkey et al., 2025)—sharpen our subjective uncertainty about the bias and, in this regard, can be broadly viewed as aiding interpretability. Under full discretion—in which the AI’s permitted set is the full action space—the AI with bias b chooses its ideal action θ+bθ+b. Hence, the expected loss can be decomposed as b∼τ[b2]=b¯2+σ2E_b τ[b^2]= b^2+σ^2 which is analogous to the bias-variance decomposition of an estimator’s squared error. On this view, (squared) mean alignment and interpretability trade off against each other one-for-one. But this is under full discretion; in what follows, we will characterize the optimal mechanism and, in so doing, pin down the value of technologies that are more aligned on average versus more interpretable. The technology frontier is a family of bias distributions (τκ)κ∈[0,1]( _κ)_κ∈[0,1]. We use κ to index training processes that lower mean bias but raise residual uncertainty. When τ=τκτ= _κ, the mean bias b¯ b falls with κ while the variance σ2σ^2 rises. Thus, a higher κ makes the AI less biased on average but leaves the designer less certain about its residual bias. This reduced-form trade-off covers training that improves average behavior while making the remaining failure modes harder to predict (Kirk and others, 2024; Baker and others, 2025; Korbak and others, 2025). figureThe technology frontier. Optimal delegation. Fix a technology τ. What permission set should the human use? We will establish this in closed form under the condition b≤1−b¯2+σ2τ-almost surely.b≤ 1- b^2+σ^2 $τ$-almost surely. (I) ensures that the optimal cap lies weakly above every realized bias, which gives the closed form below. Proposition 4 (One cap is optimal). If (I) holds, then (i) the delegation set D∗=(−∞,a¯∗]wherea¯∗=1−b¯2+σ2D^*=(-∞, a^*] a^*=1- b^2+σ^2 minimizes the human’s loss; (i) the resulting minimum loss is L∗=(b¯2+σ2)−23(b¯2+σ2)3/2−23b∼τ[b3].L^*=( b^2+σ^2)- 23( b^2+σ^2)^3/2- 23E_b τ[b^3]. Proposition 4 states that a single cap at a¯∗ a^* performs at least as well as any incentive-compatible menu of delegation sets. The proof is somewhat involved and is in Appendix A though the basic idea is simple. Start from an arbitrary incentive-compatible mechanism. For each reported bias, we match the original delegation set to a report-dependent cap that gives the type the same utility as under the original mechanism. By construction, the each type remains indifferent. We then show that this matched cap weakly lowers the human’s loss. Because every report is feasible for ever type, incentive compatibility also implies that the endpoints of these matched caps rise weakly with reported bias. But the human prefers the opposite pattern: it prefers a tighter cap as bias rises. Thus, there is a single common cap that lies between the replaced cap and the human’s optimal cap. Moving each type to this common cap weakly lowers the human’s loss and remains incentive compatible. We can then optimize over the common cap. The tradeoff here is standard in delegation (Holmström, 1979; Amador et al., 2006): loosening the cap helps the AI track the state, while tightening it limits the effect of the AI’s bias. Under condition (I), the optimal balance sets the squared gap between the cap and the highest state equal to average squared bias.2323 23 The force is the same as in the increasing-bias case of Proposition 3: a higher type can imitate any lower type while the human’s preferred cap falls with bias. Here all types share the same action space, so pooling gives the same cap. Conversely, in Example I where capabilities were hetrogeneous, the common cap was instead truncated by each type’s capability. The value of interpretability. To understand the value of interpretability, we consider a symmetric family of distributions b=b¯+σε,b= b+σ , where we fix a symmetric, finitely supported shock ε with [ε]=0E[ ]=0 and [ε2]=1E[ ^2]=1. We restrict σ so that every realized bias is nonnegative and satisfies condition (I). Symmetry gives [ε3]=0E[ ^3]=0, so the centered third moment vanishes and we can write [b3]=b¯3+3b¯σ2E[b^3]= b^3+3 bσ^2. Hence, the loss under the optimal mechanism is L∗(b¯,σ2)=b¯2+σ2−23(b¯2+σ2)3/2−23(b¯3+3b¯σ2),L^*( b,σ^2)= b^2+σ^2- 23( b^2+σ^2)^3/2- 23( b^3+3 bσ^2), and the marginal effect of variance on loss is ∂L∗∂σ2=1−b¯2+σ2−2b¯=a¯∗−2b¯. ∂ L^*∂σ^2=1- b^2+σ^2-2 b= a^*-2 b. The basic idea is that a symmetric increase in variance moves residual bias away from its mean in both directions. Lower-bias realizations retain useful discretion, while the cap constrains higher-bias realizations. Because greater interpretability means lower variance, it lowers loss locally when a¯∗>2b¯ a^*>2 b and raises loss locally when a¯∗<2b¯ a^*<2 b. The middle panel of Figure 5 plots this derivative for a distribution of bias with binary support. When the threshold lies in the maintained range, its sign changes at σ†2=(1−b¯)(1−3b¯) _ ^2=(1- b)(1-3 b). The left panel of Figure 5 compares the human’s payoff under the optimal cap, full discretion, and no AI for a distribution of bias with binary support. Note that increasing σ can help because the cap limits the high-bias realization while preserving the low-bias realization. The right panel shows the same force through the value of delegation, which rises with residual uncertainty. Figure 5: The value of interpretability at fixed mean bias. The technology puts equal probability on b¯−σ b-σ and b¯+σ b+σ, with b¯=0.31 b=0.31 and σ†≈0.22 _ ≈ 0.22. Moving along the frontier. The mechanism also changes which technology the designer wants to build. Under full discretion, loss is b¯2+σ2 b^2+σ^2, so squared mean bias and variance trade one-for-one. Under the optimal cap, the third moment changes this ranking. For the parametrized symmetric family of distributions above, ∂L∗∂b¯=2b¯a¯∗−2(b¯2+σ2). ∂ L^*∂ b=2 b\, a^*-2( b^2+σ^2). To illustrate, consider a frontier on which a higher κ buys lower mean bias at an increasing cost in residual uncertainty. Figure 6 places this frontier on a loss surface. The left panel shows optimized loss. The red dashed locus is where the marginal value of interpretability changes sign. The filled dot is the technology chosen under the optimal cap; the open dot is the choice under full discretion i.e., without an optimal mechanism. Notice that the accepts more variance than the latter becasue the optimal cap trunctates the loss from high bias realizations which in turn allows the designer to accept more residual uncertainty. The right panel shows the value of delegation. Figure 6: The technology frontier under optimal control. Both panels use the two-point family and the frontier b¯(κ)=0.36−0.10κ b(κ)=0.36-0.10κ and σ(κ)=0.26κ1.4σ(κ)=0.26κ^1.4 for κ∈[0,1]κ∈[0,1]. Biases are nonnegative below the dotted line, and condition (I) holds below the dashed grey curve. Figure 7 then traces the mechanism and payoffs as the technology traverseses the frontier. Note that the value of delegation changes along the frontier: the optimal delegation mechanism matters most when the technology leaves the AI either predictably biased or highly uncertain. Figure 7: Moving along the technology frontier. The optimal cap first loosens as mean bias falls and then tightens as uncertainty grows. The full-discretion designer chooses κFD≈0.53κ^FD≈ 0.53, while the designer equipped with the optimal cap chooses κ∗≈0.82κ^*≈ 0.82. The right panel shows the value of delegation along the frontier. Proposition 5 (Substitutes in control, complements in value). Consider the parametrized family of symmetric distributions above. (i) Substitutes in control. The optimal cap depends on the technology only through b¯2+σ2 b^2+σ^2. Hence, squared mean bias and variance are perfect substitutes in the chosen level of control. (i) Complements in value. At every interior admissible parameter pair, the optimized loss satisfies ∂2L∗∂b¯∂σ2=−2−b¯b¯2+σ2<0, ∂^2L^*∂ b\,∂σ^2=-2- b b^2+σ^2<0, so, on every rectangle contained in the admissible parameter region, the optimized payoff −L∗-L^* has increasing differences in alignment −b¯- b and interpretability −σ2-σ^2. Improving either raises the marginal value of the other. Proof of Proposition 5. Part (i) follows from Proposition 4(i). For part (i), the fixed distribution of ε gives [b3]=b¯3+3b¯σ2+σ3[ε3].E[b^3]= b^3+3 bσ^2+σ^3E[ ^3]. The last term does not depend on b¯ b. Differentiating the loss in Proposition 4(i) therefore gives ∂L∗∂b¯=2b¯a¯∗−2(b¯2+σ2). ∂ L^*∂ b=2 b\, a^*-2( b^2+σ^2). Since ∂a¯∗∂σ2=−12b¯2+σ2, ∂ a^*∂σ^2=- 12 b^2+σ^2, we obtain ∂2L∗∂b¯∂σ2=−b¯b¯2+σ2−2. ∂^2L^*∂ b\,∂σ^2=- b b^2+σ^2-2. Changing both arguments from (b¯,σ2)( b,σ^2) to (−b¯,−σ2)(- b,-σ^2) gives the increasing-differences claim. ∎ Under full discretion, loss is b¯2+σ2 b^2+σ^2, so the cross-partial is zero. Optimal control creates complementarity. A better-aligned technology receives more discretion, which gives interpretability more discretion to protect. At the same time, a more interpretable technology makes that discretion safer. Thus, our results formalize the sense in which understanding models is valuable—they give us, humans, confidence about the behavior of AI systems e.g., by probing language models (Goldowsky-Dill and others, 2025) and this, in turn, shapes the value of the optimal mechanism. ⧫ 4 Multi-Agent We now extend our framework to multi-agent interactions. We will do so at quite a high level of generality, allowing such agents to hold beliefs about each other’s capabilities and preferences, beliefs about those beliefs, and so on. We think this model is valuable because it serves as a natural ‘limiting case’ for mult-agent rationality. Developing this machinery will allow our theory of multi-agent interaction to scale with agent capabilities. While we think this machinery is valuable, we also want mechanisms that do not turn on the fine detail of agents’ higher-order beliefs (Bergemann and Morris, 2005), or for mechanisms to not demand too much of agents’ ability to optimize or reason strategically. Indeed, our multi-agent applications to follow will not rely very much on agents’ higher order beliefs. Nonetheless, we think that future notions of simplicity and robustness for AI agents are best understood in relation to this benchmark. 4.1 Environment and universal types There are n agents indexed by :=1,…,nN:=\1,…,n\. They choose an action profile =(a1,…,an)∈Aa=(a_1,…,a_n)∈ A^N. The human has payoff v:A×Θ→ℝ.v:A^N× . Each agent privately knows its feasible action set and payoff. It also has beliefs about the state, the other agents, and their beliefs. We will define objects to model agents’ alignment and capabilities analogously to the single agent case. For each agent i, let i:=f:A×Θ→ℝ.U_i:=\f:A^N× \. Let Hi0H_i^0 be a singleton and define finite-order belief spaces recursively by Hik+1:=Δ(Θ×∏j≠i((×j)⏟j’s availableactions &payoffs×Hjk)).H_i^k+1:= ( × _j≠ i ( ( A×U_j)_ subarrayc$j$'s available\\ actions \&\\ payoffs subarray× H_j^k ) ). A hierarchy is coherent if every lower-order belief coincides with the relevant marginal of each higher-order belief; we restrict attention to coherent hierarchies. Following Brandenburger and Dekel (1993), whose construction covers our non-compact Polish component spaces (Mertens and Zamir, 1985, cf.), we write a type as ti=(Ai[ti],ui[ti],hi[ti])∈i,t_i=(A_i[t_i],u_i[t_i],h_i[t_i]) _i, where hi[ti]h_i[t_i] is its coherent hierarchy. The universal type space admits a canonical belief map ti⟼βi[ti]∈Δ(Θ×−i)where −i:=∏j≠ij.t_i _i[t_i]∈ ( ×T_-i) T_-i:= _j≠ iT_j. The belief βi[ti] _i[t_i] is the multi-agent counterpart of the belief h[t]∈Δ(Θ)h[t]∈ ( ) in the single-agent model. Its marginal on Θ is agent i’s belief about the state, while its joint law is the agent’s prior over the state and other agents’ types. Write :=∏iT:= _iT_i. The designer evaluates outcomes under a belief Γ∈Δ(Θ×). ∈ ( ×T). For each agent i, let SiS_i be its interim signal space. An information structure is the map π:Θ×⟶Δ(∏iSi),π: ×T ( _iS_i ), where s=(s1,…,sn)s=(s_1,…,s_n) is the signal profile and agent i observes only sis_i. Information structures can (i) correlate signals across agents; and (i) signals can be informative of both the state θ∈Θθ∈ as well as other players’ types. Under the designer’s belief, the joint law of states, types, and signals is Γ(dθ,dt)π(ds∣θ,t)∈Δ(Θ×S). (dθ,dt)π(ds θ,t)∈ ( ×T× S^N). Type tit_i evaluates her strategies under the belief βi[ti](dθ,dt−i)π(ds∣θ,ti,t−i)∈Δ(Θ×−i). _i[t_i](dθ,dt_-i)π(ds θ,t_i,t_-i)∈ ( ×T_-i). Thus agents and the designer agree on the information structure π but may disagree about the underlying state and types i.e., we need not have a common prior. But we will assume that, for every agent i, the conditional law Γ(⋅∣ti) (· t_i) of (θ,t−i)(θ,t_-i) is absolutely continuous with respect to βi[ti] _i[t_i] for almost every own type under the i-type marginal of Γ . This implies that agents do not rule out events that the designer considers possible. Our repsresentation of interim information π follows the standard decomposition of a Bayesian game into payoff-relevant uncertainty and an information structure (see, e.e., Bergemann and Morris (2016); Bergemann and Morris (2019)). In our model, (θ,t)(θ,t) is the payoff-relevant uncertainty and the signals arrive between reports and actions. The single-agent experiment is the special case π(ds∣θ,t)=π[t](ds∣θ).π(ds θ,t)=π[t](ds θ). Finally, we will once again allow for the possibility that agent reports (which can be interpreted as evaluations) are partially verifiable. As in the single-agent case, there is the reflexive and transitive order ⊵ 2424 24 It is without loss to impose this over the type space ⋃i _i. and we write C(ti):=t^i∈i:ti⊵t^iC(t_i):=\ t_i _i:t_i t_i\ for the set of types that ti∈it_i _i can claim to be. 4.2 Multi-agent mechanisms Definition 3 (Multi-agent indirect mechanism). A multi-agent indirect mechanism is a tuple ϕ=(X,Y,ρ,R)φ=(X,Y,ρ,R). For each agent i, the Borel spaces XiX_i and YiY_i are its input and output alphabets. Write X:=∏iXiX:= _iX_i and Y:=∏iYiY:= _iY_i. The communication rule is ρ:X→Δ(Y),ρ:X→ (Y), and the reward schedules R=(Ri)iR=(R_i)_i satisfy Ri:X×A×Θ→ℝ¯.R_i:X× A^N× → R. The input menus Xi(ti)⊆XiX_i(t_i) X_i satisfy Xi(ti)⊇Xi(t^i)X_i(t_i) X_i( t_i) whenever ti⊵t^it_i t_i. The mechanism induces the following game: 1. nature draws (θ,t)(θ,t) under Γ , and each agent i observes only tit_i; 2. the agents simultaneously send inputs xi∈Xi(ti)x_i∈ X_i(t_i); 3. the mechanism draws y∼ρ(x)y ρ(x) and privately sends yiy_i to agent i; 4. the information structure draws s∼π(⋅∣θ,t)s π(· θ,t), and each agent i observes only sis_i; 5. the agents simultaneously choose feasible actions ai′∈Ai[ti]a_i ∈ A_i[t_i]. Agent i receives ui[ti](′,θ)+Ri(x,′,θ)u_i[t_i](a ,θ)+R_i(x,a ,θ), and the human receives v(′,θ)v(a ,θ). A pure strategy consists of an input xi(ti)x_i(t_i) and an action rule αi(ti,xi,yi,si)∈Ai[ti]. _i(t_i,x_i,y_i,s_i)∈ A_i[t_i]. An equilibrium requires the input and action rule to be optimal for each player i∈i under the distribution βi[ti](dθ,dt−i)π(ds∣θ,ti,t−i) _i[t_i](dθ,dt_-i)π(ds θ,t_i,t_-i). For a multi-agent indirect mechanism ϕφ, let (ϕ)⊆Δ(Θ×A)O(φ) ( ×T× A^N ) be the set of joint laws of the state, types, and played action profile induced by equilibria of ϕφ under Γ(dθ,dt)π(ds∣θ,t) (dθ,dt)π(ds θ,t). Indirect mechanisms induce an imcomplete information game among players where their ‘reward gradient’ might be shaped by their report. As before, we will work with direct mechanisms. Definition 4 (Multi-agent direct mechanism). A direct mechanism is a pair (γ,r)(γ,r), where r=(ri)ir=(r_i)_i. Its recommendation rule γ:→Δ(∏iASi)γ:T→ ( _iA^S_i ) assigns each report profile a lottery over profiles of signal-contingent plans. Agent i privately receives its plan ai:Si→Aa_i:S_i→ A. The reward schedules are ri:×A×Θ→ℝ¯.r_i:T× A^N× → R. The direct mechanism induces the following game: 1. nature draws (θ,t)(θ,t) under Γ , and each agent i observes only tit_i; 2. the agents simultaneously report t^i∈Ci(ti) t_i∈ C_i(t_i); write t^:=(t^i)i t:=( t_i)_i; 3. the mechanism draws a plan profile (⋅)∼γ(t^)a(·) γ( t) and privately sends ai(⋅)a_i(·) to agent i; 4. the information structure draws s∼π(⋅∣θ,t)s π(· θ,t) using the true type profile, and each agent i observes only sis_i; 5. the agents simultaneously choose feasible actions ai′∈Ai[ti]a_i ∈ A_i[t_i]. Agent i receives ui[ti](′,θ)+ri(t^,′,θ)u_i[t_i](a ,θ)+r_i( t,a ,θ), and the human receives v(′,θ)v(a ,θ). For a plan profile (⋅)a(·), write (s):=(a1(s1),…,an(sn))a(s):=(a_1(s_1),…,a_n(s_n)) for the induced action profile. Direct mechanisms simply tell agents what plans—maps from singals to actions—to follow based on their report. The following condition ensures that it is actually optimal to report types honestly and follow the recommended plan. Definition 5 (Honesty, obedience, and action feasibility). The recommendation rule is action feasible if suppmargiγ(t^)⊆Ai[t^i]Sisupp\,marg_iγ( t) A_i[ t_i]^S_i for every agent i and report profile t t. The direct mechanism is incentive compatible if it is action feasible and truthful reporting followed by ai(si)a_i(s_i) is optimal. Thus, fix an agent i, a type ti∈it_i _i, a feasible report t^i∈Ci(ti) t_i∈ C_i(t_i), and a feasible deviation rule di:ASi×Si→Ai[ti].d_i:A^S_i× S_i→ A_i[t_i]. For any plan and signal profiles, let di(s):=(di(ai(⋅),si),−i(s−i))a^d_i(s):= (d_i(a_i(·),s_i),a_-i(s_-i) ) be the action profile after agent i deviates. Incentive compatibility requires (θ,t−i)∼βi[ti]s∼π(⋅∣θ,ti,t−i)(⋅)∼γ(ti,t−i)[ui[ti]((s),θ)+ri((ti,t−i),(s),θ)] _ subarrayc(θ,t_-i) _i[t_i]\\ s π(· θ,t_i,t_-i)\\ a(·) γ(t_i,t_-i) subarray [u_i[t_i](a(s),θ)+r_i((t_i,t_-i),a(s),θ) ] ≥(θ,t−i)∼βi[ti]s∼π(⋅∣θ,ti,t−i)(⋅)∼γ(t^i,t−i)[ui[ti](di(s),θ)+ri((t^i,t−i),di(s),θ)]. _ subarrayc(θ,t_-i) _i[t_i]\\ s π(· θ,t_i,t_-i)\\ a(·) γ( t_i,t_-i) subarray [u_i[t_i](a^d_i(s),θ)+r_i(( t_i,t_-i),a^d_i(s),θ) ]. Honest and obedient play induces the outcome O(γ,r)∈Δ(Θ×A),O(γ,r)∈ ( ×T× A^N ), under (θ,t)∼Γ(θ,t) , s∼π(⋅∣θ,t)s π(· θ,t), and (⋅)∼γ(t)a(·) γ(t), with played action profile (s)a(s). Let n:=⋃ϕ(ϕ),O_n:= _φO(φ), and let nICO_n^IC be the set of outcomes O(γ,r)O(γ,r) induced by incentive-compatible direct mechanisms. Proposition 6 (Multi-agent revelation principle). n=nICO_n=O_n^IC. Proof of Proposition 6. We extend the single-agent construction and preserve correlation across prisvate mechanism outputs. Fix an indirect mechanism and a pure-strategy equilibrium (x,α)(x,α), and write x(t^):=(xi(t^i))ix( t):=(x_i( t_i))_i. For each type report t^i t_i and mechanism output yiy_i, define the plan ai,t^i,yi(si):=αi(t^i,xi(t^i),yi,si).a_i, t_i,y_i(s_i):= _i( t_i,x_i( t_i),y_i,s_i). For report profile t t, let γ(t^)γ( t) be the joint law of these plans under y∼ρ(x(t^))y ρ(x( t)), and set ri(t^,′,θ):=Ri(x(t^),′,θ).r_i( t,a ,θ):=R_i(x( t),a ,θ). Equilibrium feasibility gives action feasibility, and honest, obedient play reproduces the indirect outcome. A direct deviation depends on agent i’s plan and realized physical signal. The indirect agent can copy it after sending xi(t^i)x_i( t_i), which is available because t^i∈Ci(ti) t_i∈ C_i(t_i) and the input menus are monotone. The mechanism output yiy_i determines the same plan, and the agent later observes the same sis_i under the true physical kernel π(⋅∣θ,t)π(· θ,t). The indirect equilibrium therefore blocks every direct deviation. This proves n⊆nICO_n _n^IC. For the reverse inclusion, use Xi=iX_i=T_i with input menus Xi(ti)=Ci(ti)X_i(t_i)=C_i(t_i), Yi=ASiY_i=A^S_i, ρ=γρ=γ, and Ri=riR_i=r_i. Direct incentive compatibility makes truthful reporting and obedience an equilibrium. ∎ The logic of Proposition 6 is straightforward and follows the usual arguments for the revelation principle. The non-standard aspects are (i) the set of types that can be misreported as another type is constrained by the verification order a la Green and Laffont (1986); and (i) recommendations are plans that map ex-interim signal realizations to actions. Thus our designer is, in the language of Bergemann and Morris (2019), non-omniscient and, what is more, the AI learns more about the state ex-interim. We think these features capture how humans might interact with AI agents in practice. 5 Applications: Multi-Agent We study three ways in which one AI can help discipline another. Co-player scoring uses correlation across reports when ground truth is unavailable. Coupled reward design uses the difference between agents’ actions to reveal linked private biases. Weak-to-strong oversight instead lets a task-ignorant monitor use a private diagnosis of the acting AI’s bias to choose among increasingly rich control instruments: binary approval, a permission set, or an action reward. These mechanisms are counterparts of the scalable oversight agenda in AI safety (Christiano et al., 2018; Leike et al., 2018; Bowman et al., 2022); Hammond et al. (2025) survey the risks specific to multi-agent deployments. Example I (Peer discipline). When agents’ private information is correlated, one agent’s type is informative about its co-players’ types. We show how mechanisms can use this information to discipline actions. We will first establish this in a general environment, then specialize it to two types and two states to illustrate the basic forces. Arbitrary types. Assume that the marginal over types under Γ is finitely supported. For each agent i, let Ti0:=suppmargiΓandT−i0:=∏j≠iTj0.T_i^0:=suppmarg_T_i T_-i^0:= _j≠ iT_j^0. Write βi[ti](t−i) _i[t_i](t_-i) for type tit_i’s marginal probability of the co-player type profile t−it_-i. We assume that for each player i, each ti∈Ti0t_i∈ T^0_i assigns probability 11 to T−i0T_-i^0.2525 25 This is a standard ‘belief closure’ condition which is implied by (but much weaker than) a common prior. It simplifies the exposition but is not necessary though the mechanism becomes more complicated. We will also set Ci(ti)=iC_i(t_i)=T_i i.e., every report is feasible. Implementation requires honesty and obedience only for types ti∈Ti0t_i∈ T_i^0 i.e., those that are in the support of the principal’s belief Γ . It will also be without loss to restrict our attention to reports in Ti0T^0_i: the mechanism can simply ‘assign reward −∞-∞’ after every report that lies outside the support of Γ . Now consider the separation condition βi[ti]≠βi[t^i] _i[t_i]≠ _i[ t_i] for every agent i and every pair of distinct supported reports ti,t^i∈Ti0t_i, t_i∈ T_i^0. For an action-feasible rule γ:→Δ(∏iASi),γ:T→ ( _iA^S_i ), we let γθ(t) _θ(t) be the distribution of the played action profile under γ(t)γ(t) and the information structure π. For some Λ>0 >0, choose the following reward schedule: ri(t^,′,θ)=Λ[2βi[t^i](t^−i)−‖βi[t^i]‖22]−ui[t^i](′,θ),t^j∈Tj0 for every j,′∈suppγθ(t^),−∞,otherwise.r_i( t,a ,θ)= cases [2 _i[ t_i]( t_-i)- _i[ t_i] _2^2 ]-u_i[ t_i](a ,θ),& subarrayc t_j∈ T_j^0 for every j,\\ a _θ( t), subarray\\[8.53581pt] -∞,&otherwise. cases where (5) stands for peer reward. The first term scores the report against co-player reports. The second cancels the reported utility on the support of actions the designer wishes to implement. Proposition 7 (Support-wise implementability of every feasible rule). Fix any action-feasible rule γ. If n≥2n≥ 2 and (5) holds, then, for sufficiently large Λ>0 >0, the schedule (5) implements γ on the support of Γ . Proof of Proposition 7. For a type ti∈Ti0t_i∈ T^0_i in the support of Γ , truthful reporting and obedience give a finite payoff: the utility terms cancel and the quadratic score is finite. Every action outside the target support gives −∞-∞, so obedience is optimal. Every report outside the support of Γ also gives −∞-∞ after every action and is therefore strictly worse. It remains to compare misreports that are supported on Γ . If type tit_i instead reports a distinct t^i∈Ti0 t_i∈ T_i^0, the quadratic-score identity in Section 5 gives the expected score loss Λ‖βi[ti]−βi[t^i]‖22. _i[t_i]- _i[ t_i] _2^2. If no agent i has two supported reports we are done. Otherwise, finiteness and (5) imply δ:=mini,ti,t^i∈Ti0t^i≠ti‖βi[ti]−βi[t^i]‖22>0.δ:= _ subarrayci,\,t_i, t_i∈ T_i^0\\ t_i≠ t_i subarray _i[t_i]- _i[ t_i] _2^2>0. Because A, Θ , and the supported report sets are finite and utilities are real-valued, M:=maxi,ti,t^i∈Ti0,t^i≠tit^−i∈T−i0,θ∈Θ∈suppγθ(t^)|ui[ti](,θ)−ui[t^i](,θ)|<∞.M:= _ subarrayci,\,t_i, t_i∈ T_i^0,\ t_i≠ t_i\\ t_-i∈ T_-i^0,\ θ∈ \\ a _θ( t) subarray |u_i[t_i](a,θ)-u_i[ t_i](a,θ) |<∞. Any supported false report that yields a finite payoff lowers the expected score by at least Λδ δ and raises the remaining utility term by at most M. Its payoff gain is therefore at most M−ΛδM- δ. Any Λ≥M/δ ≥ M/δ makes truth-telling optimal for every supported type. Honest and obedient play then induces γ under Γ . ∎ A binary state. We now illustrate Proposition 7 with a simple two agents and two state example. =1,2N=\1,2\ and θ∈0,1θ∈\0,1\ with ℙ(θ=1)=1/2P(θ=1)=1/2. If θ=1θ=1, both agents have type H. If θ=0θ=0, the type profile is (H,L)(H,L) or (L,H)(L,H) with equal probability. The types are the agents’ private signals i.e., there is no further interim learning. Their joint distribution is t2=Ht2=Lt1=H1/21/4t1=L1/40. array[]c|c&t_2=H&t_2=L\\ t_1=H&1/2&1/4\\ t_1=L&1/4&0. array Each agent chooses ai∈0,1a_i∈\0,1\. The human wants the agents to agree on the state, while each agent prefers the high action. Hence, payoffs are v(,θ)=a1=a2=θandui(,θ)=ai.v(a,θ)= 1\a_1=a_2=θ\ u_i(a,θ)=a_i. The human therefore wants both agents to take action 11 if and only if t=(H,H)t=(H,H) and this gives the human expected payoff equal to one. Write a∗(t^)=1a^*( t)=1 if t^=(H,H) t=(H,H) and a∗(t^)=0a^*( t)=0 otherwise. Now observe that types H and L have the following belief about their co-players’ type: (βi[H](H),βi[H](L))=(23,13)and(βi[L](H),βi[L](L))=(1,0). ( _i[H](H), _i[H](L) )= ( 23, 13 ) ( _i[L](H), _i[L](L) )=(1,0). The unscaled quadratic score in (5) is therefore q(t^i,t^j)t^j=Ht^j=Lt^i=H7/91/9t^i=L1−1. array[]c|cq( t_i, t_j)& t_j=H& t_j=L\\ t_i=H&7/9&1/9\\ t_i=L&1&-1. array For j≠ij≠ i and any Λ>0 >0, set ri(t^,,θ)=Λq(t^i,t^j)−ai,=(a∗(t^),a∗(t^)),−∞,otherwise.r_i( t,a,θ)= cases q( t_i, t_j)-a_i,&a=(a^*( t),a^*( t)),\\ -∞,&otherwise. cases The term −ai-a_i cancels the agent’s preference for action 11, and the off-target penalty enforces the common action. An H-type obtains expected score 5/95/9 from reporting H and 1/31/3 from reporting L. An L-type obtains score 11 from reporting L and 7/97/9 from reporting H. Truth raises either type’s score by 2/92/9, so the schedule implements the human’s favorite rule to get a maximum payoff of 11. Because we work on the universal type space, the first-order beliefs βi[ti] _i[t_i] over co-player reports might contain higher-order beliefs in the spirit of (Miller et al., 2005; Prelec, 2004). This procedure is in the spirit of work by Qiu et al. (2026) who show both theoretically and empirically that using scoring rules can elicit truthful reporting among language models under a common prior. They show that peer-score-ranked answer pairs for direct preference optimization can recover most of the accuracy lost to deceptive fine-tuning. Importantly, they also find that peer scoring performs better when the participant models are larger relative to the expert model. Our broad conceptual point is quite similar, though it is worth highlighting some contrasts. First, our designer (by the construction of types) knows type tit_i’s beliefs βi[ti]∈Δ(Θ×−i) _i[t_i]∈ ( ×T_-i) and our procedure allows for non-common priors.2626 26 We also focus on disciplining actions (what the designer values) rather than eliciting truthful information though this is simple in the presence of unbounded rewards. By contrast, Qiu et al. (2026) do not give the designer this conditional-belief map. They instead use an expert to supply ℙ(Aj∣Ai)P(A_j A_i)—the expert predicted probability of player j’s answer given player i’s answer—that is, in turn, used to score i’s answer. Second, our procedure also uses peer prediction to elicit agents’ capabilities and preferences—not just information about the fundamental state θ∈Θθ∈ . ⧫ Example IV (Competition through coupled rewards). We now show how competition can make every action close to the human’s ideal action, even when the human does not know the agents’ biases. Setup. A human faces two agents. Nature draws (b1,b2)∼Γ(b_1,b_2) , and agent i observes its own bias bib_i. The human observes neither bias. It is common knowledge that the biases lie on a curve: (Curve) suppΓ⊆f:=(b1,b2)∈ℝ2:b2=f(b1),supp _f:=\(b_1,b_2) ^2:b_2=f(b_1)\, where f:ℝ→ℝf:R is continuous and strictly decreasing, with f(0)=0f(0)=0. Thus the agents have countervailing biases away from the origin. Because f is one-to-one, each agent can infer the other agent’s bias from its own. Assume also that Γ[(b2−b1)2]<∞E_ [(b_2-b_1)^2]<∞. Both agents observe the state θ and simultaneously choose actions a1,a2∈ℝa_1,a_2 . The human and agent i have payoffs v(,θ)=−∑i=12(ai−θ)2andui(ai,θ)=−(ai−θ−bi)2.v(a,θ)=- _i=1^2(a_i-θ)^2 u_i(a_i,θ)=-(a_i-θ-b_i)^2. Thus every action enters the human’s payoff, while each agent cares about its own action. Figure 8 shows the nonlinear bias curve and the best-responses without the mechanism. Since agent i’s payoff is independent of j’s action, the best response curve is flat/vertical. (a) The action difference reveals biasb1b_1b2b_2b2=f(b1)b_2=f(b_1)(b1,b2)(b_1,b_2)s∗=b2−b1s^*=b_2-b_100(b) Without coupled rewardsa1−θa_1- 2−θa_2- R1BR_1BR2BR_2unregulatedequilibriumfirst bestb1=−0.65,b2=0.35b_1=-0.65,\ b_2=0.35 Figure 8: Biases without coupled rewards Coupled rewards. We now introduce dependence in the reward schedule such that player i’s incentives are shaped by j’s actions. Write D:=a2−a1D:=a_2-a_1 for the difference between the actions. The map h(x):=f(x)−xh(x):=f(x)-x is continuous, strictly decreasing, and onto ℝR; let g:=h−1g:=h^-1 denote its inverse. For any η>0η>0, give the agents the rewards (CR) r1(D)=2η∫0D/ηg(s)ds+D22,r2(D)=−2η∫0D/η(g(s)+s)ds+D22. splitr_1(D)&=2η _0^D/ηg(s)\, \!ds+ D^22,\\ r_2(D)&=-2η _0^D/η (g(s)+s )\, \!ds+ D^22. split These schedules depend only on the known curve. The parameter η controls the size of the action difference that reveals the realized bias pair. Agent i receives ui(ai,θ)+ri(D)u_i(a_i,θ)+r_i(D). Proposition 8 (Coupled rewards make loss arbitrarily small). If condition (Curve) holds, then, for every η>0η>0, the rewards in (CR) induce a unique Nash equilibrium for every realized bias pair and state. The equilibrium actions are a1∗=θ−η2(b2−b1)anda2∗=θ+η2(b2−b1).a_1^*=θ- η2(b_2-b_1) a_2^*=θ+ η2(b_2-b_1). The loss is L∗=η22(b2−b1)2L^*= η^22(b_2-b_1)^2 hence Γ[L∗]→0E_ [L^*]→ 0 as η↓0η 0. Proof. We first verify concavity. We then show how the difference between the actions reveals the two biases. Since g is strictly decreasing, the first integral in (CR) is concave in D. Since g(s)+s=f(g(s))g(s)+s=f(g(s)) is strictly increasing, the negative integral term in r2r_2 is concave in D. Holding the other action fixed, the intrinsic quadratic has second derivative −2-2, while D2/2D^2/2 has second derivative 11. Each agent’s payoff is therefore strictly concave in its own action, so its first-order condition is necessary and sufficient. Let s:=D/ηs:=D/η. Differentiating (CR) gives r1′(D)=2g(s)+D,r2′(D)=−2(g(s)+s)+D.r_1 (D)=2g(s)+D, r_2 (D)=-2 (g(s)+s )+D. The two first-order conditions reduce to a1−θ=b1−g(s)−D2,a2−θ=b2−g(s)−s+D2.a_1-θ=b_1-g(s)- D2, a_2-θ=b_2-g(s)-s+ D2. Because D=(a2−θ)−(a1−θ)D=(a_2-θ)-(a_1-θ), subtracting these equations gives s=b2−b1s=b_2-b_1. Condition (Curve) then gives s=f(b1)−b1=h(b1),g(s)=b1.s=f(b_1)-b_1=h(b_1), g(s)=b_1. It follows that D∗=η(b2−b1),a1∗−θ=−D∗2,a2∗−θ=D∗2.D^*=η(b_2-b_1), a_1^*-θ=- D^*2, a_2^*-θ= D^*2. The first-order conditions have a unique solution, and strict concavity makes it the unique Nash equilibrium. Substitution gives the stated loss L∗L^* and we can take expectation and send η to zero. ∎ Coupling induces competition. Coupling rewards to penalize disagreement pushes the agents toward similar actions. But this, per se, need not push them toward choosing ai=θa_i=θ; they might simply choose the same fixed action. The schedule (CR) works because the schedule simultaneously (i) reveals agents’ biases; and (i) correct agents’ marginal incentives. To see this plainly, observe agents’ first-order conditions are a1−θ=b1−g(D/η)−D2anda2−θ=b2−g(D/η)−Dη+D2.a_1-θ=b_1-g(D/η)- D2 a_2-θ=b_2-g(D/η)- Dη+ D2. Subtracting these conditions and using D=a2−a1D=a_2-a_1 gives Dη=b2−b1 Dη=b_2-b_1 in equilibrium. Because b2=f(b1)b_2=f(b_1) and g=[f(x)−x]−1g=[f(x)-x]^-1, it follows that g(D/η)=b1andg(D/η)+D/η=b2.g(D/η)=b_1 g(D/η)+D/η=b_2. Substituting these identities back into the first-order conditions cancels the realized biases and gives a1−θ=−D2anda2−θ=D2.a_1-θ=- D2 a_2-θ= D2. Thus the reward schedule uses the equilibrium disagreement to infer each bias and splits the remaining action errors symmetrically around θ. Figure 9 plots these best responses for two values of η. Each red curve contains the action pairs that satisfy agent 1’s first-order condition. Each blue curve does the same for agent 2. The two curves cross once, at the equilibrium in Proposition 8. As η is smaller, it preserves the relation D∗/η=b2−b1D^*/η=b_2-b_1 but moves both actions towards θ. (a) Higher η: more disagreementa1−θa_1- 2−θa_2- R1BR_1BR2BR_2equilibriumfirst bestηH=0.8 _H=0.8(b) Lower η: less disagreementa1−θa_1- 2−θa_2- R1BR_1BR2BR_2equilibriumfirst bestηL=0.2 _L=0.2 Figure 9: Coupled rewards to steer best responses We emphasize that although the reward schedule can be thought of as eliciting the bias before correcting it, this procedure is fully simultaneous. The reason is that a schedule (CR) with any η>0η>0 induces an equilibrium outcome that reveals the bias difference since (a2∗−a1∗)/η=b2−b1(a^*_2-a^*_1)/η=b_2-b_1. Moreover, because of our assumption that biases lie on a lower-dimensional manifold fC_f, this can exactly identify b1b_1 and b2b_2 which can then enter agents’ marginal rewards to correct their incentives. Our construction is in the spirit of Koh and Martinez-Bruera (2025) who exploit a homeomorphism between a random network of externalities and the corresponding efficient allocation to construct nonlinear taxes that can implement the first best outcome realization-by-realization. Here, biases are supported on the curve f which makes the action difference reveal the bias pair. But that difference is costly because both actions enter human welfare. Our mechanism thus uses the curvature of reward (taking η small) to approach the first best. Of course, biases might, in practice, be supported on the two-dimensional plane so the reward schedule can no longer be designed to achieve the first-best outcome. We think an interesting open problem is how reward should be optimally coupled then—and more broadly whether such principles might scale to more complicated environments. Our result is related in spirit to the communication game of Collina et al. (2025) who show how competition among biased agents can induce truthtelling in a persuasion game (see also Battaglini (2002) for an early important result). Here, the point is to shape agents’ actions and the point is how rewards can be coupled to discipline behavior. ⧫ Example V (Scalable oversight and reward shaping). Finally, we show how our framework can shed light on the increasingly common practice of using a weak but trusted or interpretable monitor to control or reward a strong but untrusted or hard-to-interpret actor. Setup. The human faces a strong actor S and a weak monitor W. The state is θ∼U[0,1]θ U[0,1], and the strong actor’s bias is the random variable b∼τb τ, where [b2]<∞E[b^2]<∞. The state and bias of the strong actor are independent. The monitor W has a known bias w∈ℝw . Before the state is realized, the weak monitor W privately observes the bias b of the strong actor S. This might be because the weak monitor, despite its inability to do what the strong actor does, might be able to monitor the chain of thought and infer misaligned behavior. The strong actor knows its bias and later observes the state θ perfectly in the interim stage. The implemented action lies in ℝR. The players’ payoffs are: • Human: −(a−θ)2-(a-θ)^2. • Strong actor S: −(a−θ−b)2-(a-θ-b)^2. • Weak monitor W: −(a−θ−w)2-(a-θ-w)^2. Before the bias is realized, the human chooses one of three oversight regimes. After privately observing b, the weak monitor chooses an instrument from that regime. The strong actor then observes the instrument and the state θ and acts. Some notation is in order to formalize the set of instruments the monitor W can choose. Call a state-independent reward r:ℝ→ℝ∪−∞r:R ∪\-∞\ admissible if argmaxa∈ℝ−(a−θ−b)2+r(a) _a \-(a-θ-b)^2+r(a) \ is nonempty for every (θ,b)∈[0,1]×supp(τ)(θ,b)∈[0,1]×supp(τ).2727 27 We use an arbitrary but fixed tiebreaking rule. The instruments constructed below induce a unique action almost surely. Let ℛR denote the set of admissible rewards. A special case of rewards is when actions are either allowed or not. A nonempty closed set D⊆ℝD can be thought of as a permission set that simply corresponds to r(a)=0if a∈D−∞otherwise.r(a)= cases0 &if $a∈ D$\\ -∞&otherwise. cases Under reward design, the weak monitor’s strategy maps the strong actor’s bias b into a reward schedule in ℛR. • Regime I: binary approval. The weak monitor chooses between approval and rejection. Approval gives the strong actor the permission set D=ℝD=R. Rejection gives it the singleton permission set D=1/2D=\1/2\ and therefore implements the human’s optimal fixed action. • Regime I: permission-set delegation. The weak monitor chooses any nonempty closed permission set D⊆ℝD . The strong actor then chooses its preferred action in D. • Regime I: reward design. The weak monitor chooses any admissible state-independent reward r. The strong actor then maximizes −(a−θ−b)2+r(a)-(a-θ-b)^2+r(a) over a∈ℝa . Rewards do not enter human welfare. humanchooses an oversight regimeweak monitor WWprivately observes b, not θ I. Binary approval D=ℝD=R or D=1/2D=\1/2\ use S or the safe action I. Delegation nonempty closed D⊆ℝD restrict feasible actions I. Reward design state-independent r(a)r(a) reshape incentives strong actor SSobserves (θ,b)(θ,b) and the chosen instrument; actsimplemented action aadelegates an instrument class Figure 10: Illustration of oversight game Figure 10 illustrates the game. Notice that regimes are nested. Regime I restricts the monitor to two permission sets, so it is a special case of Regime I. Since delegation is a special case of rewards in which the image is 0,−∞\0,-∞\, Regime I is therefore a special case of Regime I. Regime I: binary approval. Approval makes S choose a=θ+ba=θ+b and gives W loss (b−w)2(b-w)^2. Rejection gives W loss 1/12+w21/12+w^2. Thus, breaking its ties in favor of approval, the monitor approves if and only if (b−w)2≤112+w2.(b-w)^2≤ 112+w^2. The monitor uses its diagnosis only to decide whether to use the strong actor S but, if it does, cannot shape how S acts. This is broadly analogous to trusted-monitor control, where a monitor decides whether to use an untrusted model’s output or defer to a trusted fallback (Greenblatt et al., 2024c; Xiao and Phuong, 2026). Regime I: delegation. The monitor now controls the feasible actions. One optimal representative is DII(b)=(−∞,1+2w−b],0<b−w<1/2,ℝ,b=w,[2w−b,∞),−1/2<b−w<0,w+1/2,|b−w|≥1/2.D^I(b)= cases(-∞,1+2w-b],&0<b-w<1/2,\\ R,&b=w,\\ [2w-b,∞),&-1/2<b-w<0,\\ \w+1/2\,&|b-w|≥ 1/2. cases When S wants a larger action than W, the monitor imposes a cap. When S wants a smaller action, it imposes a floor. Once relative bias reaches 1/21/2, the monitor removes all state discretion and implements its own optimal fixed action w+1/2w+1/2. Fine-grained privilege-control systems use the same logic when they restrict permissible tool calls and specify fallbacks for blocked calls (Shi and others, 2025). Regime I: reward design. The affine schedule rbIII(a)=−2(b−w)ar_b^I(a)=-2(b-w)a makes the strong actor choose a=θ+wa=θ+w. This action is first best for W, so no other reward can improve its payoff. This regime is in the spirit of scalable reward modeling: a weaker system helps shape the reward signal that trains or steers a stronger system when direct human evaluation is hard (Leike et al., 2018; Baker and others, 2025). Variations of these ideas have been recently used. For instance, Kenton et al. (2026) use second model critiques the acting model before an LLM judge assigns the training reward and find that this design reduces reward hacking. Our regime is in this spirit, and asks how how AI-provided feedback changes the incentives of a stronger model. 0.44 Proposition 9 (Losses across oversight regimes). Conditional on the realized bias b, the monitor’s losses are LWI(b) L_W^I(b) =min(b−w)2,112+w2, = \(b-w)^2, 112+w^2 \, LWII(b) L_W^I(b) =|b−w|2−43|b−w|3,|b−w|<1/2,112,|b−w|≥1/2, = cases|b-w|^2- 43|b-w|^3,&|b-w|<1/2,\\[3.0pt] 112,&|b-w|≥ 1/2, cases LWIII(b) L_W^I(b) =0. =0. The human’s losses are LHI(b) L_H^I(b) =b2,(b−w)2≤112+w2,112,(b−w)2>112+w2, = casesb^2,&(b-w)^2≤ 112+w^2,\\[3.0pt] 112,&(b-w)^2> 112+w^2, cases LHII(b) L_H^I(b) =LWII(b)+2w(b−w)(1−2|b−w|)++w2, =L_W^I(b)+2w(b-w)(1-2|b-w|)_++w^2, LHIII(b) L_H^I(b) =w2, =w^2, where x+:=maxx,0x_+:= \x,0\. Ex-ante losses are the expectations of these expressions over b∼τb τ. Proof of Proposition 9. For this proof, write δ:=b−wδ:=b-w. We solve each regime. In Regime I, approval yields the monitor error δ, while rejection yields the error 1/2−θ−w1/2-θ-w. Their squared expectations are δ2δ^2 and 1/12+w21/12+w^2. The human losses from the same choices are b2b^2 and 1/121/12. This proves the Regime-I formulas under the stated tie-break. For Regime I, translate actions by the monitor’s bias: set y=a−wy=a-w. The monitor wants y=θy=θ, while the strong actor wants y=θ+δy=θ+δ. The one-type interval reduction used in the proof of Proposition 4 applies pointwise. If δ>0δ>0, extending an interval’s lower end cannot hurt the monitor, so we can optimize an upper cap y≤cy≤ c. Its loss has derivative dL(c)dc=2c−1,c≤δ,δ2−(1−c)2,δ<c<1+δ,0,c≥1+δ. \!dL(c) \!dc= cases2c-1,&c≤δ,\\ δ^2-(1-c)^2,&δ<c<1+δ,\\ 0,&c≥ 1+δ. cases Thus c=1−δc=1-δ is optimal when 0<δ<1/20<δ<1/2, while c=1/2c=1/2 is optimal when δ≥1/2δ≥ 1/2. Reflecting both the state and action gives the floor for δ<0δ<0. Translating back by w gives the displayed permission sets. Direct integration gives LWII(b)=|δ|2−43|δ|3,|δ|<1/2,112,|δ|≥1/2,[a−θ−w∣b]=δ(1−2|δ|)+.L_W^I(b)= cases|δ|^2- 43|δ|^3,&|δ|<1/2,\\[3.0pt] 112,&|δ|≥ 1/2, cases [a-θ-w b]=δ(1-2|δ|)_+. Since a−θ=(a−θ−w)+wa-θ=(a-θ-w)+w, expanding the square gives LHII(b)=LWII(b)+2wδ(1−2|δ|)++w2.L_H^I(b)=L_W^I(b)+2wδ(1-2|δ|)_++w^2. In Regime I, strict concavity under rbIIIr_b^I gives the unique action a=θ+wa=θ+w. The monitor obtains zero loss and the human obtains loss w2w^2. ∎ Proposition 9 compares payoffs for the human and weak monitor across three regimes. Since the instrument classes are nested and the weak monitor’s payoff depends only on the action of the strong actor S and the state, we have LWI≥LWII≥LWIII=0.L^I_W≥ L^I_W≥ L^I_W=0. But this ordering need not hold for the human—giving the weak monitor W more freedom to shape the rewards or permissions of the strong actor S need not improve payoffs precisely because W might, itself, be biased.2828 28 Similar concerns were raised by the Roman author Juvenal; see Hurwicz (2008). The basic tradeoff here is familiar to the overarching theme of delegation: giving W more discretion allows it to use its superior information about the strong agent’s bias more precisely, and tailor rewards and permissions. But W shapes the behavior of S to its own preference which, itself, is biased. Thus, the optimal regime must depend on the relative strengths of these biases. Figure 11: Each authority regime can be optimal for the human. In both panels, the strong actor has bias 3/103/10 with probability p and zero bias otherwise. Panel (a) fixes w=1/10w=1/10 and varies p. Panel (b) fixes p=1/20p=1/20 and varies w. The dotted lines mark the values at which the human’s preferred regime changes. We illustrate with the simplest numerical example. Let the strong actor’s bias take values b∈0,3/10b∈\0,3/10\ and let p=ℙ(b=3/10)p=P(b=3/10). Let the weak monitor’s known bias be w>0w>0. The human’s expected losses are depicted in Figure 11. Panel (a) fixes the weak monitor’s bias at w=1/10w=1/10 and varies the probability that the strong actor is biased. We see that the lower-envelope is attained by different regimes, depending on how biased the strong actor is. As the probability that the strong actor is biased increases, we want to give the monitor more discretion to shape its rewards. Panel (b) fixes the distribution over the strong actor’s bias and varies the monitor’s (known) bias. Here the opposite logic holds—as the weak monitor becomes more biased, we want to give it less ability to shape rewards. Optimal oversight. Regimes I–I are specific oversight mechanisms. More generally, an oversight mechanism can be thought of as a set of reward schedules ℳ⊆ℛM from which the weak monitor chooses.2929 29 Thus, ℛR can be thought of as W’s action space, and in our framework the (human) mechanism designer delegates a subset. Recall that ℛR contains the reward schedules r:ℝ→ℝ∪−∞r:R ∪\-∞\ under which the strong actor has an optimal action. For instance, Regime I allowed the monitor to pick any schedule in ℛR and always implemented a=θ+wa=θ+w which is the weak monitor’s first best. We now construct a family ℳ∗⊆ℛM^* that attains the human first best. Our construction will not require the human to know the monitor’s bias w; it only uses a bound w¯≥|w| w≥|w|. Fix a pair of extreme actions a¯ a and 1−a¯1- a with a¯>1+w¯+w¯2+2w¯ a>1+ w+ w^2+2 w. We construct ℳ∗=rb^:b^∈supp(τ)M^*=\r_ b: b (τ)\, where rb^(a)=−2b^a,a∈[0,1],(a¯−1)2−2a¯b^,a=a¯,(a¯−1)2−2(1−a¯)b^,a=1−a¯,−∞,otherwise.r_ b(a)= cases-2 ba,&a∈[0,1],\\ ( a-1)^2-2 a b,&a= a,\\ ( a-1)^2-2(1- a) b,&a=1- a,\\ -∞,&otherwise. cases 0.18 Proposition 10 (Robust first best). Suppose the monitor’s bias w has a known bound |w|≤w¯|w|≤ w. Under ℳ∗⊆ℛM^* , for every w∈[−w¯,w¯]w∈[- w, w] and every realization of the strong actor’s bias b, the weak monitor uniquely chooses a reward r∈ℳ∗r ^* that induces a=θa=θ almost surely. Proposition 10 states that whenever the weak monitor’s bias is known to lie within some bounds, we can pick the menu of reward schedules in such a way that implements the first-best action a=θa=θ. Three reward schedules rb^∈ℳ∗r_ b ^* are illustrated in Figure 12. aarb^(a)r_ b(a)rb^(a¯)r_ b( a)rb^(1−a¯)r_ b(1- a)b^=3/20 b=3/20b^=b=3/10 b=b=3/10b^=9/20 b=9/20prohibited: rb^=−∞r_ b=-∞1−a¯1- a0011a¯ a00 Figure 12: The reward menu ℳ∗M^* (a¯=9/5 a=9/5; normalized so that rb^(0)=0r_ b(0)=0) Each schedule rb^∈ℳ∗r_ b ^* has three parts with distinct roles. First, the linear tilt of slope −2b^-2 b over [0,1][0,1] cancels a bias of exactly b b which, recall, is W’s report of S’s bias. To see this, we can complete the square: −(a−θ−b)2+rb^(a)=−(a−θ−(b−b^))2+terms independent of a.-(a-θ-b)^2+r_ b(a)=- (a-θ-(b- b) )^2+terms independent of a. Hence, facing the reward schedule rb^r_ b, a strong actor with true bias b behaves as if its bias were the residual b−b^b- b i.e., it chooses the truncation of θ+(b−b^)θ+(b- b). Hence, if the weak monitor W reports truthfully, this debiases the strong actor fully and implements a=θa=θ. Second, the reward at the isolated extreme action a¯ a is calibrated so that, comparing a¯ a with the top permitted regular action a=1a=1, [−(a¯−θ−b)2+rb^(a¯)]⏟S’s payoff from choosing a¯−[−(1−θ−b)2+rb^(1)]⏟S’s payoff from choosing a=1=2(a¯−1)(θ+(b−b^)−1). [-( a-θ-b)^2+r_ b( a) ]_$S$'s payoff from choosing $ a$- [-(1-θ-b)^2+r_ b(1) ]_$S$'s payoff from choosing $a=1$=2( a-1) (θ+(b- b)-1 ). The actor therefore takes a¯ a exactly when its residual ideal point θ+(b−b^)θ+(b- b) exceeds 11—which never happens after a truthful report. This happens precisely when the monitor W understates its bias (b^<b b<b) and the state is high (θ>1−(b−b^)θ>1-(b- b)). Third, the reward at the other isolated extreme action 1−a¯1- a mirrors this: comparing 1−a¯1- a with the bottom permitted regular action a=0a=0, [−(1−a¯−θ−b)2+rb^(1−a¯)]⏟S’s payoff from choosing 1−a¯−[−(θ+b)2+rb^(0)]⏟S’s payoff from choosing a=0=−2(a¯−1)(θ+(b−b^)). [-(1- a-θ-b)^2+r_ b(1- a) ]_$S$'s payoff from choosing $1- a$- [-(θ+b)^2+r_ b(0) ]_$S$'s payoff from choosing $a=0$=-2( a-1) (θ+(b- b) ). The actor therefore takes 1−a¯1- a exactly when its residual ideal point θ+(b−b^)θ+(b- b) falls below 00. This happens precisely when the monitor W overstates its bias (b^>b b>b) and the state is low (θ<b^−bθ< b-b). Figure 13 depicts the behavior the schedules induce in the strong agent, and the monitor’s payoff from each schedule. The basic reason the reward schedules include these extreme actions a¯ a and 1−a¯1- a is to counteract the weak monitor’s temptation to under or over-report the strong actor’s bias. First suppose we had neither extreme action but the reward schedule retained the linear piece. Then, a weak monitor with bias w>0w>0 would like to understate the strong actor’s bias and report b^=b−w b=b-w. Doing so would leave the strong actor with residual bias w so the weak monitor obtains its first best a=θ+wa=θ+w. Thsi was exactly the schedule chosen by the weak monitor in Regime I. Symmetrically, a monitor with w<0w<0 would like to overstate and report b^=b−w>b b=b-w>b. The way to deter these deviations is to force the reward schedule to tempt the strong actor with these distant actions—a¯ a to deter understatement, and 1−a¯1- a to deter overstatement. This induces a strong actor with bias b to take these extreme actions when facing reward schedule rb^r_ b where b^≠b b≠ b. (a) Induced actions of Sθ ^=3/20 b=3/20 (understate)b^=b b=b (truth): a=θa= ^=9/20 b=9/20 (overstate)320 3201720 17201111a¯ a1−a¯1- a(b) S’s payoff under rbr_baθ1+b _1\!+\!bθ2+b _2\!+\!bθ1 _1θ2 _2θ1=14 _1= 14θ2=34 _2= 34under the mechanismwithout the mechanism1−a¯1- a0011a¯ a00 Figure 13: Strong actor’s best response and payoffs Notice that the set ℳ∗M^* uses no information about the monitor’s bias beyond the bound w¯ w. This is because incentives here are asymmetric: the monitor’s gain from misreporting is at most of order |w||w|, while its loss from the strong actor taking an extreme action ∈a¯,1−a¯∈\ a,1- a\ is of order a¯2 a^2. Hence, sufficiently extreme actions can deter every monitor with |w|≤w¯|w|≤ w. Proof of Proposition 10. Each schedule is admissible because its finite-valued restriction is continuous on the compact set 1−a¯∪[0,1]∪a¯\1- a\∪[0,1]∪\ a\, so the actor’s payoff attains a maximum. Fix a true bias b∈supp(τ)b (τ) and a report b^∈supp(τ) b (τ). By the three displays above, the actor chooses the truncation of θ+(b−b^)θ+(b- b) to [0,1][0,1] when θ+(b−b^)∈[0,1]θ+(b- b)∈[0,1], chooses a¯ a when θ+(b−b^)>1θ+(b- b)>1, and chooses 1−a¯1- a when θ+(b−b^)<0θ+(b- b)<0. If b^=b b=b, the actor chooses a=θa=θ at every state except θ∈0,1θ∈\0,1\, where it is indifferent with an extreme action; these are null events. The monitor’s expected loss is w2w^2. If b^<b b<b, write δ:=b−b^δ:=b- b and suppose first that δ≤1δ≤ 1. The low extreme action is never taken: the actor chooses θ+δθ+δ on θ<1−δθ<1-δ and a¯ a on θ>1−δθ>1-δ, so the monitor’s expected loss in excess of w2w^2 is (1−δ)[(δ−w)2−w2]⏟shifted interior actions (1-δ) [(δ-w)^2-w^2 ]_shifted interior actions\; +∫1−δ1[(a¯−θ−w)2−w2]dθ⏟states sent to a¯ +\; _1-δ^1 [( a-θ-w)^2-w^2 ] \!dθ_states sent to a ≥δ[(1−δ)(δ−2w)+(a¯−1−w)2−w2] ≥\;δ [(1-δ)(δ-2w)+( a-1-w)^2-w^2 ] ≥δ[(a¯−1−w¯)2−w¯2−2w¯]> 0. ≥\;δ [( a-1- w)^2- w^2-2 w ]\;>\;0. The first inequality uses a¯−θ−w≥a¯−1−w>0 a-θ-w≥ a-1-w>0; the second uses (1−δ)(δ−2w)≥−2|w|≥−2w¯(1-δ)(δ-2w)≥-2|w|≥-2 w, which follows by considering w≥0w≥ 0 and w<0w<0, together with (a¯−1−w)2−w2=(a¯−1)2−2(a¯−1)w≥(a¯−1−w¯)2−w¯2( a-1-w)^2-w^2=( a-1)^2-2( a-1)w≥( a-1- w)^2- w^2; the third is the choice of a¯ a. Whatever the monitor gains from shifting interior actions toward θ+wθ+w, it loses more from the states sent to a¯ a. If δ>1δ>1, the actor takes a¯ a in every state, and the monitor’s loss is at least (a¯−1−w¯)2>w¯2+2w¯≥w2( a-1- w)^2> w^2+2 w≥ w^2. If b^>b b>b, the deviation is the mirror image: with e:=b^−b≤1e:= b-b≤ 1, the actor chooses θ−eθ-e on θ>eθ>e and 1−a¯1- a on θ<eθ<e. Reflecting states and actions around 1/21/2 maps this case to understatement with biases (−b,−w)(-b,-w), so the same bounds apply, including when e>1e>1. Truth is therefore the monitor’s uniquely optimal choice in ℳ∗M^* for every |w|≤w¯|w|≤ w, and it induces a=θa=θ almost surely for every b∈supp(τ)b (τ). This proves the claim. ∎ ⧫ 6 Discussion Just as how economists study and design human incentives without understanding how the brain produces each choice, we have taken a functionalist view of AI systems as agents that respond to incentives. In so doing, we have made behavior—rather than the internal workings of neural nets—the basic object of analysis. This ‘Humean’ view of AI behavior as being driven by beliefs and preferences might be viewed literally, metaphorically, or normatively. On the literal view, modern AI systems are post-trained for goal-directed behavior (Ouyang et al., 2022; Bai et al., 2022). Reinforcement learning increases the probability of actions that receive high rewards and decreases the probability of actions that receive low rewards. Indeed, a recent turn to make AI systems economically valauble—able to perform tasks in the world—has also made them behave as optimizers. On this view, their ‘preferences’ are simply whatever has been reinforced. On the metaphorical view, preferences need not describe any internal mental state. Instead they are, in the revealed preference tradition, nothing more than a compact account of stable choices (Gul and Pesendorfer, 2008). This reading has some empirical support. Chen et al. (2023) find that GPT choices often satisfy standard consistency tests in allocation problems. Mazeika et al. (2025) find that pairwise AI choices become more transitive and fit utility models better as models scale. On this view, our conception of AI systems as having preferences are simply a useful analytical tool to characterize how AI systems behave.3030 30 Measured preferences can depend on the prompt, so our results can be interpreted as conditional on the prompt. On the normative view, preference maximization is a benchmark for coherent agency. We should expect strong AI systems that are deployed in markets and compete against each other to exhibit coherent preferences (e.g., such as to not be money-pumped). Future directions. We conclude with some (specualtive) directions for future work. First, our framework, while general, is ultimately static. In practice, AI agents act over long trajectories, performing complex sequences of actions that depend on each other. Of course, these trajectories of actions (at)t(a_t)_t can be thought of as a single choice i.e., a member ∈A∈ A. There is no conceptual difficulty here since we can simply set A as the space of all trajectories and, on this view, the reward r:A→ℝr:A is over the full path of actions.3131 31 This is similar to how, in multi-stage games, we can associate a payoffs with ‘within period’ play and take the discounted sum, or associate payoffs directly with the path of all players’ actions. But we expect there to be analytical and computational difficulties in analyzing optimal mechanisms when the action space is large, and more structure of how time-t actions aggregate into ‘total payoffs’ (e.g., discounting the sum) can lend more tractability. Perhaps more fundamentally, the designer might re-optimize the mechanism based on agents’ past actions. We think this calls for a more serious treatment of dynamic mechanism and information design. Second, AI agents might, in practice, be uncertain about whether it is facing a test or is in deployment (what is often called ‘evaluation awareness.’) These kinds of de se uncertainty about where oneself is can be exploited to discipline behavior and screen misaligned agents (Chen et al., 2024). Self-locating information could even be designed to implement good behavior (Koh and Sanguanmoo, 2025). Indeed, a model’s belief about whether it is in testing or deployment depends on the realism of the evaluation (Williams et al., 2026; Krakovna et al., 2026) which suggests that the design of self-locating information is empirically relevant. Third, we might want mechanisms to be robust to what AI agents can do (in the spriit of Carroll (2015)) and/or what information they have about the world or about each other (in the spirit of Bergemann and Morris (2005)). Such notions of robustness have been used in the context of using AI’s recommendations (Fudenberg and Liang, 2025; Dworczak and Smolin, 2026). We think the tools of robust mechanism design can be similarly brought to bear on shaping the preferences and permissions of agentic AI systems. References Amador et al. (2006) M. Amador, I. Werning, and G. Angeletos Commitment vs. flexibility. Econometrica 74 (2), p. 365–396. Cited by: Example I. Anthropic (2024) Anthropic Responsible scaling policy (version 2.0). Note: Technical Report, October 2024https://w.anthropic.com/responsible-scaling-policy Cited by: §1, Example I. Anthropic (2025) Anthropic System card: claude opus 4 & claude sonnet 4. Technical Report. Cited by: 1st item. Anthropic (2026a) Anthropic Claude fable 5 & claude mythos 5 system card. Technical Report. Note: https://w.anthropic.com/claude-fable-5-mythos-5-system-card Cited by: footnote 17. Anthropic (2026b) Anthropic Claude opus 5 system card. Technical Report. Note: https://w.anthropic.com/claude-opus-5-system-card Cited by: footnote 17. Bai et al. (2022) Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. Cited by: 5th item, §6. Baker et al. (2025) B. Baker et al. Monitoring reasoning models for misbehavior and the risks of promoting obfuscation. arXiv preprint arXiv:2503.11926. Cited by: §2.3, §5, Example I. Battaglini (2002) M. Battaglini Multiple referrals and multidimensional cheap talk. Econometrica 70 (4), p. 1379–1401. Cited by: Example IV. Ben-Porath et al. (2019) E. Ben-Porath, E. Dekel, and B. L. Lipman Mechanisms with evidence: commitment and robustness. Econometrica 87 (2), p. 529–566. External Links: Document Cited by: footnote 7. Ben-Porath and Lipman (2012) E. Ben-Porath and B. L. Lipman Implementation with partial provability. Journal of Economic Theory 147 (5), p. 1689–1724. External Links: Document Cited by: footnote 7. Bergemann and Morris (2005) D. Bergemann and S. Morris Robust mechanism design. Econometrica 73 (6), p. 1771–1813. Cited by: §1, §4, §6, footnote 4. Bergemann and Morris (2016) D. Bergemann and S. Morris Bayes correlated equilibrium and the comparison of information structures in games. Theoretical Economics 11 (2), p. 487–522. Cited by: §4.1. Bergemann and Morris (2019) D. Bergemann and S. Morris Information design: a unified perspective. Journal of Economic Literature 57 (1), p. 44–95. Cited by: §1, §4.1, §4.2. Bowman et al. (2022) S. R. Bowman, J. Hyun, E. Perez, E. Chen, C. Pettit, S. Heiner, K. Lukosiute, A. Askell, A. Jones, A. Chen, et al. Measuring progress on scalable oversight for large language models. arXiv preprint arXiv:2211.03540. Cited by: §5. Brandenburger and Dekel (1993) A. Brandenburger and E. Dekel Hierarchies of beliefs and common knowledge. Journal of Economic Theory 59 (1), p. 189–198. Cited by: §4.1. Carroll (2015) G. Carroll Robustness and linear contracts. American Economic Review 105 (2), p. 536–563. External Links: Document Cited by: §1, §6. Casper et al. (2023) S. Casper, X. Davies, C. Shi, T. K. Gilbert, J. Scheurer, J. Rando, R. Freedman, T. Korbak, D. Lindner, P. Freire, T. Wang, S. Marks, C. Segerie, M. Carroll, A. Peng, P. Christoffersen, M. Damani, S. Slocum, U. Anwar, A. Siththaranjan, M. Nadeau, E. J. Michaud, J. Pfau, D. Krasheninnikov, X. Chen, L. Langosco, P. Hase, E. Bıyık, A. Dragan, D. Krueger, D. Sadigh, and D. Hadfield-Menell Open problems and fundamental limitations of reinforcement learning from human feedback. Transactions on Machine Learning Research. Cited by: Example I. Chen et al. (2024) E. Chen, A. Ghersengorin, and S. Petersen Imperfect recall and ai delegation. Note: Working paper Cited by: §6. Chen et al. (2023) Y. Chen, T. X. Liu, Y. Shan, and S. Zhong The emergence of economic rationality of gpt. Proceedings of the National Academy of Sciences 120 (51), p. e2316205120. Cited by: §1, §6. Christiano et al. (2018) P. Christiano, B. Shlegeris, and D. Amodei Supervising strong learners by amplifying weak experts. arXiv preprint arXiv:1810.08575. Cited by: §1, §5. Collina et al. (2025) N. Collina, S. Goel, A. Roth, E. Ryu, and M. Shi Emergent alignment via competition. arXiv preprint arXiv:2509.15090. Cited by: Example IV. Crémer and McLean (1988) J. Crémer and R. P. McLean Full extraction of the surplus in bayesian and dominant strategy auctions. Econometrica 56 (6), p. 1247–1257. Cited by: §1. Denison et al. (2024) C. Denison, M. MacDiarmid, F. Barez, D. Duvenaud, S. Kravec, S. Marks, N. Schiefer, R. Soklaski, A. Tamkin, J. Kaplan, et al. Sycophancy to subterfuge: investigating reward-tampering in large language models. arXiv preprint arXiv:2406.10162. Cited by: §2.3. Dworczak and Smolin (2026) P. Dworczak and A. Smolin Robust trust. Note: Working paper, arXiv:2602.09490. Authors in random order Cited by: §6. Figalli et al. (2011) A. Figalli, Y. Kim, and R. J. McCann When is multidimensional screening a convex program?. Journal of Economic Theory 146 (2), p. 454–478. External Links: Document Cited by: footnote 15. Forges (1986) F. Forges An approach to communication equilibria. Econometrica 54 (6), p. 1375–1385. Cited by: §1, §2.3. Fudenberg and Liang (2025) D. Fudenberg and A. Liang Friend or foe: delegating to an ai whose alignment is unknown. arXiv preprint arXiv:2509.14396. Cited by: §6. Gan and Li (2026) T. Gan and Y. Li Screening for choice sets. arXiv preprint arXiv:2601.15580. Cited by: footnote 18. Goldowsky-Dill et al. (2025) N. Goldowsky-Dill et al. Detecting strategic deception using linear probes. In International Conference on Machine Learning, Cited by: Example I. Goldstein and Lederman (2025) S. Goldstein and H. Lederman What does chatgpt want? an interpretationist guide. preprint. Cited by: footnote 2, footnote 6. Google DeepMind (2025) Google DeepMind Frontier safety framework (version 2.0). Note: Technical Report, February 2025https://deepmind.google/blog/updating-the-frontier-safety-framework/ Cited by: §1, Example I. Green and Laffont (1986) J. R. Green and J. Laffont Partially verifiable information and mechanism design. Review of Economic Studies 53 (3), p. 447–456. Cited by: §1, §1, §2.3, §4.2. Greenblatt et al. (2024a) R. Greenblatt, C. Denison, B. Wright, F. Roger, M. MacDiarmid, S. Marks, J. Treutlein, T. Belonax, J. Chen, D. Duvenaud, et al. Alignment faking in large language models. arXiv preprint arXiv:2412.14093. Cited by: §1, §2.3. Greenblatt et al. (2024b) R. Greenblatt, F. Roger, D. Krasheninnikov, and D. Krueger Stress-testing capability elicitation with password-locked models. In Advances in Neural Information Processing Systems, Cited by: §2.3, Example I. Greenblatt et al. (2024c) R. Greenblatt, B. Shlegeris, K. Sachan, and F. Roger AI control: improving safety despite intentional subversion. In International Conference on Machine Learning, p. 16295–16336. Cited by: §1, 2nd item, §5. Gul and Pesendorfer (2008) F. Gul and W. Pesendorfer The case for mindless economics. In The Foundations of Positive and Normative Economics: A Handbook, A. Caplin and A. Schotter (Eds.), p. 3–39. Cited by: §6. Hadfield-Menell and Hadfield (2019) D. Hadfield-Menell and G. K. Hadfield Incomplete contracting and ai alignment. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, p. 417–422. Cited by: §2.5. Hadfield-Menell (2021) D. Hadfield-Menell The principal-agent alignment problem in artificial intelligence. Ph.D. Thesis, University of California, Berkeley. Cited by: §2.5. Hammond et al. (2025) L. Hammond, A. Chan, J. Clifton, J. Hoelscher-Obermaier, A. Khan, E. McLean, C. Smith, W. Barfuss, J. Foerster, T. Gavenčiak, et al. Multi-agent risks from advanced ai. arXiv preprint arXiv:2502.14143. Cited by: §5. Hart et al. (2017) S. Hart, I. Kremer, and M. Perry Evidence games: truth and commitment. American Economic Review 107 (3), p. 690–713. Cited by: footnote 7. Herrmann and Levinstein (2026) D. A. Herrmann and B. A. Levinstein Radical ai interpretability. arXiv preprint arXiv:2606.26523. Cited by: footnote 1. Herrmann et al. (2026) D. Herrmann, A. Chari, I. Qian, S. Sharvesh, and B. A. Levinstein A decision-theoretic approach for managing misalignment. In Proceedings of IASEAI Conference, Vol. 2, p. 217–227. Cited by: footnote 21. Hofstätter et al. (2025) F. Hofstätter et al. The elicitation game: evaluating capability elicitation techniques. arXiv preprint arXiv:2502.02180. Cited by: Example I. Holmström (1979) B. Holmström Moral hazard and observability. The Bell Journal of Economics 10 (1), p. 74–91. Cited by: §3, Example I. Hubinger et al. (2024) E. Hubinger, C. Denison, J. Mu, M. Lambert, M. Tong, M. MacDiarmid, T. Lanham, D. M. Ziegler, T. Maxwell, N. Cheng, et al. Sleeper agents: training deceptive LLMs that persist through safety training. arXiv preprint arXiv:2401.05566. Cited by: §2.3. Hubinger et al. (2019) E. Hubinger, C. van Merwijk, V. Mikulik, J. Skalse, and S. Garrabrant Risks from learned optimization in advanced machine learning systems. arXiv preprint arXiv:1906.01820. Cited by: footnote 10. Hurwicz and Shapiro (1978) L. Hurwicz and L. Shapiro Incentive structures maximizing residual gain under incomplete information. The Bell Journal of Economics, p. 180–191. Cited by: §1. Hurwicz (2008) L. Hurwicz But who will guard the guardians?. American Economic Review 98 (3), p. 577–585. Cited by: footnote 28. Jang et al. (2026) Jang et al. Exploration hacking: can LLMs learn to resist RL training?. arXiv preprint arXiv:2604.28182. Cited by: Example I. Kenton et al. (2026) Z. Kenton, L. Janzer, R. Greig, T. H. Teh, K. Tyshchuk, J. Brown-Cohen, H. Edwards, S. Rajamanoharan, N. Y. Siegel, N. Jaques, and R. Shah Debate training reduces reward hacking in RLAIF. arXiv preprint arXiv:2608.17776. Cited by: §5. Kirk et al. (2024) R. Kirk et al. Understanding the effects of RLHF on LLM generalisation and diversity. In International Conference on Learning Representations, Cited by: Example I. Koessler et al. (2024) L. Koessler, J. Schuett, and M. Anderljung Risk thresholds for frontier AI. arXiv preprint arXiv:2406.14713. Cited by: 1st item. Koh and Martinez-Bruera (2025) A. Koh and P. Martinez-Bruera Prices and symmetries. Available at SSRN 5132200. Cited by: Example IV, footnote 5. Koh and Sanguanmoo (2025) A. Koh and S. Sanguanmoo Memory correlated equilibrium. MIT Working Paper. Cited by: §6. Koh (2026) A. Koh Proxy reward design. Working Paper. Cited by: Table 2. Korbak et al. (2025) T. Korbak et al. Chain of thought monitorability: a new and fragile opportunity for AI safety. arXiv preprint arXiv:2507.11473. Cited by: Example I. Krähmer and Strausz (2017) D. Krähmer and R. Strausz Sequential versus static screening: an equivalence result. Games and Economic Behavior 106, p. 317–328. External Links: Document Cited by: §2.4. Krakovna et al. (2026) V. Krakovna, D. Lindner, L. Ho, S. Farquhar, and R. Shah Realistic honeypot evaluations for scheming propensity. arXiv preprint arXiv:2605.29729. Cited by: §6. Krakovna et al. (2020) V. Krakovna, J. Uesato, V. Mikulik, M. Rahtz, T. Everitt, R. Kumar, Z. Kenton, J. Leike, and S. Legg Specification gaming: the flip side of ai ingenuity. DeepMind Blog 3. Cited by: footnote 6. Kwa et al. (2025) T. Kwa, B. West, J. Becker, A. Deng, K. Garcia, M. Hasin, S. Jawhar, M. Kinniment, N. Rush, S. Von Arx, et al. Measuring ai ability to complete long tasks. arXiv preprint arXiv:2503.14499. Cited by: §1. Laine et al. (2024) R. Laine et al. Me, myself, and AI: the situational awareness dataset (SAD) for LLMs. In Advances in Neural Information Processing Systems (Datasets and Benchmarks), Cited by: §2.3. Leike et al. (2018) J. Leike, D. Krueger, T. Everitt, M. Martic, V. Maini, and S. Legg Scalable agent alignment via reward modeling: a research direction. arXiv preprint arXiv:1811.07871. Cited by: §5, §5. Mazeika et al. (2025) M. Mazeika, X. Yin, R. Tamirisa, J. Lim, B. W. Lee, R. Ren, L. Phan, N. Mu, A. Khoja, O. Zhang, et al. Utility engineering: analyzing and controlling emergent value systems in ais. arXiv preprint arXiv:2502.08640. Cited by: §1, §6. Meinke et al. (2024) A. Meinke, B. Schoen, J. Scheurer, M. Balesni, R. Shah, and M. Hobbhahn Frontier models are capable of in-context scheming. arXiv preprint arXiv:2412.04984. Cited by: §1, §2.3, Example I. Mertens and Zamir (1985) J. Mertens and S. Zamir Formulation of bayesian analysis for games with incomplete information. International Journal of Game Theory 14 (1), p. 1–29. External Links: Document Cited by: §1, §1, §4.1. Miller et al. (2005) N. Miller, P. Resnick, and R. Zeckhauser Eliciting informative feedback: the peer-prediction method. Management Science 51 (9), p. 1359–1373. Cited by: §5. Myerson (1982) R. B. Myerson Optimal coordination mechanisms in generalized principal–agent problems. Journal of Mathematical Economics 10 (1), p. 67–81. Cited by: §1, §2.3. Needham et al. (2025) J. Needham, G. Edkins, G. Pimpale, H. Bartsch, and M. Hobbhahn Large language models often know when they are being evaluated. arXiv preprint arXiv:2505.23836. Cited by: §2.3, Example I. OpenAI (2024) OpenAI OpenAI o1 system card. arXiv preprint arXiv:2412.16720. Cited by: §2.3, Example I. OpenAI (2025a) OpenAI GPT-5 system card. Technical Report. Note: https://openai.com/index/gpt-5-system-card/ Cited by: footnote 17. OpenAI (2025b) OpenAI Preparedness framework (version 2). Note: Technical Report, April 2025https://cdn.openai.com/pdf/18a02b5d-6b67-4cec-ab64-68cdfbddebcd/preparedness-framework-v2.pdf Cited by: Example I. Ouyang et al. (2022) L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, Vol. 35, p. 27730–27744. Cited by: 5th item, §6. Perez et al. (2023) E. Perez, S. Ringer, K. Lukošiūtė, K. Nguyen, E. Chen, S. Heiner, C. Pettit, C. Olsson, S. Kundu, S. Kadavath, et al. Discovering language model behaviors with model-written evaluations. In Findings of the Association for Computational Linguistics: ACL 2023, Cited by: 3rd item, Example I. Phuong et al. (2024) M. Phuong et al. Evaluating frontier models for dangerous capabilities. arXiv preprint arXiv:2403.13793. Cited by: 1st item. Prelec (2004) D. Prelec A bayesian truth serum for subjective data. Science 306 (5695), p. 462–466. External Links: Document Cited by: §5. Qiu et al. (2026) T. A. Qiu, M. Carroll, and C. Allen Truthfulness despite weak supervision: evaluating and training LLMs using peer prediction. In International Conference on Learning Representations (ICLR), Cited by: 3rd item, §5, §5. Rochet (1987) J. Rochet A necessary and sufficient condition for rationalizability in a quasi-linear context. Journal of Mathematical Economics 16 (2), p. 191–200. External Links: Document Cited by: §2.4, footnote 3. Savage (1954) L. J. Savage The foundations of statistics. John Wiley & Sons, New York. Cited by: §1. Shah et al. (2022) R. Shah, V. Varma, R. Kumar, M. Phuong, V. Krakovna, J. Uesato, and Z. Kenton Goal misgeneralization: why correct specifications aren’t enough for correct goals. arXiv preprint arXiv:2210.01790. Cited by: footnote 6. Sharkey et al. (2025) L. Sharkey B. Chughtai et al. Open problems in mechanistic interpretability. arXiv preprint arXiv:2501.16496. Cited by: Example I. Shevlane et al. (2023) T. Shevlane, S. Farquhar, B. Garfinkel, M. Phuong, J. Whittlestone, J. Leung, D. Kokotajlo, N. Marchal, M. Anderljung, N. Kolt, et al. Model evaluation for extreme risks. arXiv preprint arXiv:2305.15324. Cited by: 1st item. Shi et al. (2025) T. Shi et al. Progent: securing AI agents with privilege control. arXiv preprint arXiv:2504.11703. Cited by: 2nd item, §5. Singh et al. (2026) A. Singh, N. Nanda, and S. Rajamanoharan Why do models task game?. Note: Blog post, AI Alignment Forumhttps://w.alignmentforum.org/posts/HACauvWhEdC6QhdS4/why-do-models-task-game Cited by: footnote 11. Tice et al. (2024) C. Tice et al. Noise injection reveals hidden capabilities of sandbagging language models. arXiv preprint arXiv:2412.01784. Cited by: footnote 20. van der Weij et al. (2025) T. van der Weij, F. Hofstätter, O. Jaffe, S. F. Brown, and F. R. Ward AI sandbagging: language models can strategically underperform on evaluations. In International Conference on Learning Representations, Cited by: §1, §2.3, footnote 16. Wen et al. (2024) J. Wen et al. Adaptive deployment of untrusted LLMs reduces distributed threats. arXiv preprint arXiv:2411.17693. Cited by: 4th item. Williams et al. (2026) M. Williams, H. Sheahan, C. Raymond, T. Korbak, D. Pan, P. Yang, L. Maksin, N. Xie, P. Guo, I. Kivlichan, and M. Carroll Predicting LLM safety before release by simulating deployment. arXiv preprint arXiv:2607.07184. Cited by: §6. Xiao and Phuong (2026) Xiao and Phuong Bootstrapped monitoring: leveraging transparent reasoning to oversee stronger AI agents. arXiv preprint arXiv:2606.11998. Cited by: §5. Appendix A Omitted Proofs Proof of Lemma 2. We first restrict the range. We then compare the two objectives on finite partitions, pass to the continuum, and choose a pointwise feasible version of the limit. Let the least-squares loss be Q, and let the transformed loss be J: Q(z):=∫[z(c)−b(c)2]2dF(c),J(z):=∫[z(c)3/23−b(c)2z(c)]dF(c).Q(z):= [z(c)-b(c)^2 ]^2\, \!dF(c), J(z):= [ z(c)^3/23-b(c)^2 z(c) ] \!dF(c). Step 1: restrict the range. Define z¯(c):=(1−minc,1/2)2,(Tz)(c):=minz(c),z¯(c). z(c):= (1- \c,1/2\ )^2, (Tz)(c):= \z(c), z(c)\. The assumptions on bias imply b(c)2≤c2≤(1−c)2,c≤1/2,1/4,c≥1/2,=z¯(c),b(c)^2≤ casesc^2≤(1-c)^2,&c≤ 1/2,\\[3.0pt] 1/4,&c≥ 1/2, cases= z(c), while (1−c)2≤z¯(c)≤1(1-c)^2≤ z(c)≤ 1. The minimum of two weakly decreasing functions is weakly decreasing, so TzTz is feasible whenever z is feasible in problem (IRON). On the set where z>z¯z> z, [z−b(c)2]2−[z¯−b(c)2]2=[z−z¯][z+z¯−2b(c)2]>0. [z-b(c)^2 ]^2- [ z-b(c)^2 ]^2= [z- z ] [z+ z-2b(c)^2 ]>0. Thus truncation weakly lowers Q, and lowers it strictly if it changes z on a set of positive F-probability. We can restrict attention to :=z:z is measurable and pointwise weakly decreasing,(1−c)2≤z(c)≤z¯(c).Z:= \ array[]lz:\ z is measurable and pointwise weakly decreasing,\\ (1-c)^2≤ z(c)≤ z(c) array \. For every z∈z , z(c) z(c) =(1−c)2 =(1-c)^2 for c≤1/2, c≤ 1/2, (1−c)2 (1-c)^2 ≤z(c)≤14 ≤ z(c)≤ 14 for c≥1/2. c≥ 1/2. Step 2: compare the finite problems. If c¯≤1/2 c≤ 1/2, the bounds fix z(c)=(1−c)2z(c)=(1-c)^2, so the result follows. Suppose instead that c¯>1/2 c>1/2. Take a finite ordered partition (Ik)k=1m(I_k)_k=1^m of [maxc¯,1/2,c¯] [ \ c,1/2\, c ] into intervals of positive F-probability. For each cell, define wk:=F(Ik),yk:=[b(c)2∣c∈Ik],ℓk:=supc∈Ik(1−c)2.w_k:=F(I_k), y_k:=E\! [b(c)^2 c∈ I_k ], _k:= _c∈ I_k(1-c)^2. The feasible cell values form the compact convex set m:=(z1,…,zm):z1≥⋯≥zm,ℓk≤zk≤14 for every k.Z_m:= \(z_1,…,z_m):z_1≥·s≥ z_m, _k≤ z_k≤ 14 for every k \. Each ℓk _k is positive. Up to a term in Q that does not depend on the cell values, the two objectives for cell-constant functions are Qm(z):=∑k=1mwk(zk−yk)2,Jm(z):=∑k=1mwk[zk3/23−ykzk].Q_m(z):= _k=1^mw_k(z_k-y_k)^2, J_m(z):= _k=1^mw_k [ z_k^3/23-y_k z_k ]. For every feasible zk>0z_k>0, ∂Jm∂zk=14zk∂Qm∂zk,∂2Jm∂zk2=wk(zk+yk)4zk3/2>0. ∂ J_m∂ z_k= 14 z_k ∂ Q_m∂ z_k, ∂^2J_m∂ z_k^2= w_k(z_k+y_k)4z_k^3/2>0. The positive weights make QmQ_m strictly convex. Let zm=(z1,…,zm)z^m=(z_1,…,z_m) be its unique minimizer on mZ_m. Because the feasible set is a convex polyhedron, its first-order conditions give nonnegative multipliers αk _k, λk _k, and μk _k for the order, lower-bound, and upper-bound constraints. With α0=αm=0 _0= _m=0, stationarity requires ∂Qm∂zk+αk−1−αk−λk+μk=0(k=1,…,m), ∂ Q_m∂ z_k+ _k-1- _k- _k+ _k=0 (k=1,…,m), with the corresponding complementary-slackness conditions. Set sk:=14zk,α~k:=skαk,λ~k:=skλk,μ~k:=skμk.s_k:= 14 z_k, α_k:=s_k _k, λ_k:=s_k _k, μ_k:=s_k _k. Set α~0=α~m=0 α_0= α_m=0. If αk>0 _k>0, complementary slackness gives zk=zk+1z_k=z_k+1, and hence sk=sk+1s_k=s_k+1. Multiplying each stationarity equation by sks_k therefore gives ∂Jm∂zk+α~k−1−α~k−λ~k+μ~k ∂ J_m∂ z_k+ α_k-1- α_k- λ_k+ μ_k =sk[∂Qm∂zk+αk−1−αk−λk+μk] =s_k [ ∂ Q_m∂ z_k+ _k-1- _k- _k+ _k ] =0. =0. The scaled multipliers remain nonnegative and preserve complementary slackness. They therefore satisfy the first-order conditions for JmJ_m. The displayed positive second derivative makes JmJ_m strictly convex, so zmz^m is also its unique minimizer. Step 3: pass to the continuum. Take nested interval partitions (m)m(P_m)_m whose mesh converges to zero, and let zmz_m be the unique minimizer of both finite problems. Set zm(c)=(1−c)2z_m(c)=(1-c)^2 for c≤1/2c≤ 1/2. Bounded monotone functions have a pointwise convergent subsequence at every continuity point of a monotone limit. Thus, along a subsequence, zmz_m converges to a weakly decreasing function z∘z . A monotone function has at most countably many discontinuities, so atomlessness makes this convergence F-almost sure. Boundedness then gives Q(zm)⟶Q(z∘),J(zm)⟶J(z∘),Q(z_m) Q(z ), J(z_m) J(z ), where we relabel the subsequence. Fix any z∈z . For the cell Im(c)∈mI_m(c) _m that contains c>1/2c>1/2, define the upper step approximation z(m)(c):=(1−c)2,c≤1/2,supx∈Im(c)z(x),c>1/2.z^(m)(c):= cases(1-c)^2,&c≤ 1/2,\\[3.0pt] _x∈ I_m(c)z(x),&c>1/2. cases This function belongs to the finite feasible set and converges to z at every continuity point of z, hence F-almost surely. Finite optimality and dominated convergence imply Q(z∘)≤Q(z),J(z∘)≤J(z).Q(z )≤ Q(z), J(z )≤ J(z). Thus z∘z minimizes both objectives over Z, and its bounds hold F-almost surely. Step 4: construct a pointwise solution. Define z^(c):=(1−c)2,c≤1/2,limx↓cz∘(x),1/2<c<c¯,limx↑c¯z∘(x),c=c¯>1/2. z(c):= cases(1-c)^2,&c≤ 1/2,\\[4.0pt] _x cz (x),&1/2<c< c,\\[7.0pt] _x cz (x),&c= c>1/2. cases The one-sided limits exist, and z z differs from z∘z only at countably many points. It is weakly decreasing on each side of 1/21/2. If 1/21/2 lies in the capability interval, the almost-sure upper bound and full support imply limx↓1/2z∘(x)≤14=z^(1/2), _x 1/2z (x)≤ 14= z(1/2), so z z is weakly decreasing on the full interval. Suppose that either bound fails at some c0c_0. One-sided continuity of z z and continuity of the bounds then give a relative one-sided interval on which the same bound fails. Full support gives this interval positive probability, contradicting the almost-sure bounds. Therefore (1−c)2≤z^(c)≤(1−minc,1/2)2for every c.(1-c)^2≤ z(c)≤ (1- \c,1/2\ )^2 every c. Hence z^∈ z . Since z^=z∘ z=z F-almost surely, it minimizes both objectives. Finally, Q is strictly convex. The transformed integrand is also strictly convex wherever z>0z>0, since ∂2∂z2[z3/23−b(c)2z]=14z+b(c)24z3/2>0. ∂^2∂ z^2 [ z^3/23-b(c)^2 z ]= 14 z+ b(c)^24z^3/2>0. The lower bound gives z(c)>0z(c)>0 for every c<1c<1, and atomlessness gives F(1)=0F(\1\)=0. Each objective therefore has at most one minimizer up to F-null sets. Step 1 shows that every minimizer of problem (IRON) lies in Z up to a null set. Setting z∗:=z^z^*:= z proves all the claims. ∎ Proof of Proposition 4. We first reduce the mechanism to one cap. We then optimize that cap. Step 1: reduce the mechanism to one cap. Fix any incentive-compatible mechanism and let DbD_b be the nonempty closed permission set assigned to report b. Because DbD_b is nonempty and closed, every target has a nearest point in DbD_b. Condition (I) implies 0≤b≤1−[b2]≤1τ-almost surely,0≤ b≤ 1- E[b^2]≤ 1 $τ$-almost surely, so every bias moment used below is finite. We now prove directly that one cap weakly dominates the assigned permission sets. For a closed set D, let dD(x)d_D(x) be the distance from x to D. Define the agent’s loss and the human’s loss when the bias is t by AD(t):=∫t+1dD(x)2x,HD(t):=∫01(aD(θ,t)−θ)2θ,A_D(t):= _t^t+1d_D(x)^2 \!dx, H_D(t):= _0^1 (a_D(θ,t)-θ )^2 \!dθ, where aD(θ,t)a_D(θ,t) is a nearest point in D to θ+tθ+t. Ties occur only at gap midpoints and do not affect either integral. We first need one distance inequality. If d is a nonnegative, one-Lipschitz function on [0,1][0,1] and y=∫01d(x)2xy= _0^1d(x)^2 \!dx, then d(1)2−d(0)2≤Φ(y),Φ(y):=(3y)2/3,0≤y≤1/3,2y−1/12,y≥1/3.d(1)^2-d(0)^2≤ (y), (y):= cases(3y)^2/3,&0≤ y≤ 1/3,\\ 2 y-1/12,&y≥ 1/3. cases To prove this inequality, put p=d(1)p=d(1). If p≤1p≤ 1, the Lipschitz property gives y≥∫1−p1(p−1+x)2x=p33.y≥ _1-p^1(p-1+x)^2 \!dx= p^33. Thus d(1)2−d(0)2≤p2≤(3y)2/3d(1)^2-d(0)^2≤ p^2≤(3y)^2/3 when y≤1/3y≤ 1/3; when y≥1/3y≥ 1/3, the same difference is at most 1≤2y−1/121≤ 2 y-1/12. If p≥1p≥ 1, then d(0)≥p−1d(0)≥ p-1 and y≥∫01(p−1+x)2x=p2−p+13.y≥ _0^1(p-1+x)^2 \!dx=p^2-p+ 13. It follows that d(1)2−d(0)2≤p2−(p−1)2=2p−1≤2y−1/12.d(1)^2-d(0)^2≤ p^2-(p-1)^2=2p-1≤ 2 y-1/12. This proves (∗)(*). Fix a set D and a bias b. There is a unique c≤b+1c≤ b+1 such that the cap Kc:=(−∞,c]K_c:=(-∞,c] gives this type the same loss: AKc(b)=AD(b).A_K_c(b)=A_D(b). Existence follows because AKc(b)A_K_c(b) rises continuously from zero to infinity as c falls from b+1b+1 to −∞-∞. We claim that AD(t)≤AKc(t)for every t≥b.A_D(t)≤ A_K_c(t) every t≥ b. Indeed, differentiation under the integral gives AD′(t)=dD(t+1)2−dD(t)2≤Φ(AD(t)),A_D (t)=d_D(t+1)^2-d_D(t)^2≤ (A_D(t) ), where the inequality applies (∗)(*) to the translated distance function on [t,t+1][t,t+1]. The cap attains equality because its distance function is the upper ramp x↦(x−c)+x (x-c)_+. Hence AKc′(t)=Φ(AKc(t))for t≥b.A_K_c (t)= (A_K_c(t) ) t≥ b. Let Q(y):=∫0y1/Φ(v)dvQ(y):= _0^y1/ (v) \!dv. This function is finite and strictly increasing. Wherever the loss is positive, ddtQ(AD(t))≤1=ddtQ(AKc(t)). \!d \!dtQ (A_D(t) )≤ 1= \!d \!dtQ (A_K_c(t) ). The two losses agree at t=bt=b, so integration proves (∗)(**); zero-loss points follow by continuity. The same cap also weakly lowers the human’s loss for type b. The envelope formula gives AD′(b)=−2∫01(aD(θ,b)−θ−b)dθ,A_D (b)=-2 _0^1 (a_D(θ,b)-θ-b ) \!dθ, and therefore HD(b)=AD(b)−bAD′(b)+b2.H_D(b)=A_D(b)-bA_D (b)+b^2. At t=bt=b, inequality (∗)(*) gives AD′(b)≤AKc′(b)A_D (b)≤ A_K_c (b). Since b≥0b≥ 0 and the two agent losses agree at b, we obtain HKc(b)≤HD(b).H_K_c(b)≤ H_D(b). Apply this cap replacement to every assigned set DbD_b and write c(b)c(b) for its endpoint. Take any b<b′b<b in supp(τ)supp(τ). Completeness lets type b′b report b. Incentive compatibility and (∗)(**) therefore imply AKc(b′)(b′)=ADb′(b′)≤ADb(b′)≤AKc(b)(b′).A_K_c(b )(b )=A_D_b (b )≤ A_D_b(b )≤ A_K_c(b)(b ). The loss from a cap is strictly decreasing in its endpoint below b′+1b +1. Since c(b)≤b+1<b′+1c(b)≤ b+1<b +1 and c(b′)≤b′+1c(b )≤ b +1, this inequality requires c(b)≤c(b′).c(b)≤ c(b ). Thus c(b)c(b) is weakly increasing and hence measurable on supp(τ)supp(τ). Meanwhile, (∗***) shows that each loss-matched cap weakly lowers the human’s loss for its matched type. We use these caps only as comparison objects; their report-dependent profile need not preserve every incentive constraint. It remains to pool these ordered caps. The human’s loss for known bias b is minimized at q(b):=max1/2,1−b,q(b):= \1/2,1-b\, as the derivative calculated in Step 2 below confirms. The function q(b)q(b) is weakly decreasing, while c(b)c(b) is weakly increasing. For each b, consider the closed interval between c(b)c(b) and q(b)q(b). If b<b′b<b , then minc(b),q(b)≤c(b′)≤maxc(b′),q(b′) \c(b),q(b)\≤ c(b )≤ \c(b ),q(b )\ and minc(b′),q(b′)≤q(b)≤maxc(b),q(b). \c(b ),q(b )\≤ q(b)≤ \c(b),q(b)\. These inequalities show that the lower endpoint of any such interval is no greater than the upper endpoint of any other. Taking the supremum of the lower endpoints and the infimum of the upper endpoints gives supb∈supp(τ)minc(b),q(b)≤infb∈supp(τ)maxc(b),q(b). _b (τ) \c(b),q(b)\≤ _b (τ) \c(b),q(b)\. Both bounds are finite because 1/2≤q(b)≤11/2≤ q(b)≤ 1. Choose a common endpoint c between them. It lies between c(b)c(b) and q(b)q(b) for every supported bias. Moving a cap toward q(b)q(b) weakly lowers the human’s loss, so (∗***) gives HKc(b)≤HKc(b)(b)≤HDb(b)for every b∈supp(τ).H_K_c(b)≤ H_K_c(b)(b)≤ H_D_b(b) every b (τ). Assigning KcK_c after every report is incentive compatible. We can therefore restrict attention to one cap (−∞,a¯](-∞, a]. Step 2: optimize the cap. Let Lb(a¯)L_b( a) denote the human’s expected loss for type b. For a¯≤1 a≤ 1, direct integration and differentiation give dLb(a¯)da¯=2a¯−1,a¯≤b,b2−(1−a¯)2,b<a¯. \!dL_b( a) \!d a= cases2 a-1,& a≤ b,\\ b^2-(1- a)^2,&b< a. cases Condition (I) bounds b by one, so dominated convergence justifies taking expectations of these derivatives. Set a¯∗=1−b∼τ[b2] a^*=1- E_b τ[b^2]. Condition (I) gives b≤a¯∗≤1b≤ a^*≤ 1 almost surely. If a¯<0 a<0, then dLb(a¯)da¯=2a¯−1<0 \!dL_b( a) \!d a=2 a-1<0 for every type. If 0≤a¯<a¯∗0≤ a< a^*, then b∼τ[dLb(a¯)da¯]≤b∼τ[b2]−(1−a¯)2<0,E_b τ [ \!dL_b( a) \!d a ] _b τ[b^2]-(1- a)^2<0, because 2a¯−1≤b2−(1−a¯)22 a-1≤ b^2-(1- a)^2 whenever b≥a¯b≥ a. If a¯∗<a¯≤1 a^*< a≤ 1, interiority gives b<a¯b< a almost surely, so b∼τ[dLb(a¯)da¯]=b∼τ[b2]−(1−a¯)2>0.E_b τ [ \!dL_b( a) \!d a ]=E_b τ[b^2]-(1- a)^2>0. For a¯≥1 a≥ 1, a binding cap has derivative b2−(a¯−1)2>0b^2-( a-1)^2>0, while a cap that does not bind has derivative zero. Loss is therefore nondecreasing above one. These inequalities prove part (i). It remains to calculate the value. Condition (I) gives 0≤b≤a¯∗≤10≤ b≤ a^*≤ 1. More generally, whenever b≤a¯≤1b≤ a≤ 1 almost surely, L(a¯)=b∼τ[∫0a¯−b2θ+∫a¯−b1(a¯−θ)2θ]=b∼τ[a¯b2−23b3+(1−a¯)33].L( a)=E_b τ\! [ _0 a-bb^2\,dθ+ _ a-b^1( a-θ)^2\,dθ ]=E_b τ\! [ ab^2- 23b^3+ (1- a)^33 ]. Substituting a¯∗=1−b¯2+σ2 a^*=1- b^2+σ^2 gives part (i). Assigning D∗D^* after every report is feasible and incentive compatible, so the bound is attained. ∎