Paper deep dive
Delegating Authorization to Misaligned Agents: Coalitional Alignment and Safe Control
Natalie Collina, Surbhi Goel, Aaron Roth, Sikata Bela Sengupta
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Long-running AI agents create a control problem: each action they take changes the state, which in turn affects the trajectory of future actions. If the agent is not fully aligned, then guaranteeing safety requires approving consequential actions before allowing them to be executed. But requiring human approval at every step makes attention a bottleneck. Delegating review to other AI agents raises the same alignment problem: the reviewers may themselves be misaligned. We identify a condition on a reviewing panel that is weaker than individual alignment yet necessary and sufficient for a guarantee that the principal fares at least as well in expectation as under a designated baseline policy. Each reviewer agent reports whether an action proposal made by a proposer agent improves its own utility relative to the baseline. We show that a threshold rule tolerating $k$ disapprovals is safe exactly when, after any $k$ reviewers are removed, the principal's utility can be written as a nonnegative combination of the remaining reviewers' utilities, plus a term that is nonnegative on every feasible proposal. We call this property $k$-robust coalitional alignment. The characterization lifts to sequential control: in a discounted MDP with an arbitrary proposer agent, safety at every state is both necessary and sufficient for the induced policy to match or improve on the baseline. When reviewers vote strategically, full-panel coverage in reward-function space guarantees that every Nash equilibrium is safe under the unanimous approval rule; in contrast, more permissive thresholds can admit unsafe equilibria even when reviewers are individually aligned. Experiments with existing reviewer models show that collective review can remain sound without an aligned individual, even when some disapprovals are tolerated.
Tags
Links
- Source: https://arxiv.org/abs/2609.15803v1
- Canonical: https://arxiv.org/abs/2609.15803v1
Trouble viewing inline? Open PDF directly →
Full Text
188,510 characters extracted from source content.
Expand or collapse full text
Delegating Authorization to Misaligned Agents: Coalitional Alignment and Safe Control Natalie Collina Surbhi Goel Aaron Roth Sikata Sengupta Abstract Long-running AI agents create a control problem: each action they take changes the state, which in turn affects the trajectory of future actions. If the agent is not fully aligned, then guaranteeing safety requires approving consequential actions before allowing them to be executed. But requiring human approval at every step makes attention a bottleneck. Delegating review to other AI agents raises the same alignment problem: the reviewers may themselves be misaligned. We identify a condition on a reviewing panel that is weaker than individual alignment yet necessary and sufficient for a guarantee that the principal fares at least as well in expectation as under a designated baseline policy. Each reviewer agent reports whether an action proposal made by a proposer agent improves its own utility relative to the baseline. We show that a threshold rule tolerating k disapprovals is safe exactly when, after any k reviewers are removed, the principal’s utility can be written as a nonnegative combination of the remaining reviewers’ utilities, plus a term that is nonnegative on every feasible proposal. We call this property k-robust coalitional alignment. No individual reviewer need be aligned: the condition can arise naturally e.g. when reviewers’ objectives are unbiased but noisy perturbations of the principal’s. The characterization lifts to sequential control: in a discounted Markov decision process with an arbitrary history-dependent proposer agent, local safety at every state is both necessary and sufficient for the induced policy to match or improve the baseline in expected discounted utility. When reviewers vote strategically, full-panel coverage in reward-function space guarantees that every Nash equilibrium is safe under the unanimous approval rule; in contrast, more permissive thresholds can admit unsafe equilibria even when reviewers are individually aligned. Finally, we establish an information-theoretic limit of binary approval: when feasible lotteries include both gains and losses for the principal, exact safety and completeness require an aligned reviewer. This impossibility can be circumvented either by allowing approximation error or by eliciting richer information from reviewers. Experiments with existing reviewer models show that collective review can remain sound without an aligned individual, even when some disapprovals are tolerated. Using numerical scores allows more beneficial proposals to be authorized at comparable soundness. Contents 1 Introduction 1.1 Model and safety guarantee 1.2 Our results 1.3 Related Work 2 The one-decision model and geometric preliminaries 2.1 The decision problem and feasible deviations 2.2 Authorization rules 2.3 Coalitional coverage and coalitional alignment 3 Sound authorization and rejection depth 3.1 Soundness under general authorization rules 3.2 Threshold voting and rejection depth 3.3 How robust coalitional alignment can arise 4 Sequential authorization in a discounted MDP 4.1 Control model and continuation-aware votes 4.2 From local authorization to global control 4.3 A state-independent certificate from reward functions 5 Strategic reviewers and the role of unanimity 5.1 Strategic safety in the control process 6 Completeness and the limits of binary approval 6.1 The limits of binary approval 6.2 Approximate binary review from diverse panels 6.3 Numerical scores instead of binary votes 7 Experiments with LLM reviewer panels 7.1 Experimental setup 7.2 Coverage without individual alignment 7.3 Authorization with binary votes 7.4 Authorization with numerical scores 8 Discussion References A Additional results for static soundness B Additional sequential and strategic results B.1 Reward coverage modulo potential shaping B.2 Veto power and undominated outcomes C Binary-information limits and approximate implementation C.1 Random and deterministic approximate implementation D Robustness to reporting and evaluation failures E Auditing coalitional alignment E.1 Auditing problem and main result E.2 Geometry, algorithms, and hardness F Additional experimental details F.1 Benchmark mappings and proposal domains F.2 Exact threshold-rule audit F.3 Learned binary and cardinal rules F.4 Calibration on validation data F.5 Interpretation and cohort checks 1 Introduction As AI systems become more capable, users are increasingly using them to act on their behalf in long-running tasks: allowing them e.g. to write and execute code, send messages, and initiate transactions. A conservative safety policy would require a person to authorize every consequential action—but this makes human attention a bottleneck that negates most of the advantages of task automation. Modern systems have therefore begun to delegate part of this authorization process. For example, Codex’s Auto-review mode sends a proposed action to an approval subagent, which is empowered to authorize actions on the user’s behalf without interrupting the user (OpenAI, 2026). But delegating approval to other AI agents creates an apparent circularity. The principal’s main AI system—the proposer agent—was placed behind an approval boundary for fear that it may be misaligned and so may not faithfully pursue the user’s interests. Requiring the reviewer to be perfectly aligned is a poor solution to this problem, since the problem is predicated on perfect alignment being a difficult goal. We ask whether a weaker alignment condition can still guarantee that the principal is at least as well off as under a baseline policy. We show that it can. 1.1 Model and safety guarantee In our model the principal begins by designating a baseline policy, such as refusing tool calls or following some weaker but known-to-be-safe controller. At each state, the proposer agent proposes an alternative to the baseline action. This proposal (a possibly randomized action) is sent to a panel of external reviewers. Each reviewer compares the proposal with the baseline according to its own objective and returns an approval or disapproval vote represented as a single bit. An aggregation rule sees the set of reviewers who approve and decides whether to execute the proposal; if the proposal is rejected, the system follows the action induced by the baseline policy. We allow any deterministic rule based on this approval set; threshold rules are the canonical example. We call an authorization rule sound if it provides a safety guarantee relative to the baseline policy. In a one-decision model, this means that every executed proposal weakly increases the principal’s expected utility over a designated baseline action. In a sequential MDP model, it means that the policy induced by repeated authorization has expected discounted principal utility at least that of consistently following the baseline policy. The guarantee holds for every feasible proposal, initial state, and history-adaptive proposer agent, and applies in expectation over the randomness of the proposed action distributions and underlying state transition probabilities. 1.2 Our results Our core results are exact characterizations of safe threshold review and safe sequential control. We extend these results to fully strategic reviewers and the equilibria their behavior induces, under a stronger coverage condition on their reward functions. Finally we establish an information-theoretic impossibility on the kind of binary-vote approval mechanism we study, and briefly consider two routes around it. We state the main results informally here, with pointers to their precise versions. Static safety. We begin by isolating a single authorization decision under a threshold rule that executes a proposal when it receives at most k disapproval votes. We say that a subpanel covers the principal if the principal’s utility is a nonnegative combination of the subpanel’s utilities, plus a term that is nonnegative on every feasible proposal. The panel is k-robustly coalitionally aligned if this coverage condition survives the deletion of any k reviewers. Informal result (Coalitional alignment characterizes threshold safety). The k-tolerant threshold rule is sound if and only if the panel is k-robustly coalitionally aligned. Theorem 1 and Corollary 2 give the precise statements. Coverage by the full panel (the case k=0k=0) characterizes whether any sound rule that accepts unanimous approval exists, and the same coalitional condition extends to arbitrary monotone rules through the coalitions whose approval is just sufficient to authorize a proposal. The same characterization also identifies the most permissive sound monotone rule (which is not necessarily a threshold rule): Authorize an action exactly when it is unanimously approved by some approval set that covers the principal (Proposition 1). Coalitional alignment can hold even when every reviewer is individually misaligned, so it is strictly weaker than full alignment. Certifying that a particular panel satisfies it may still be difficult, but implementing a threshold rule requires only counting binary votes, and our theorems characterize exactly when doing so is safe. Sequential control. The one-decision model is a warm-up for authorization in a discounted Markov decision process. In this model, at state s, reviewer i votes on the counterfactual “take the proposed action once, then follow the baseline policy,” evaluated with its own baseline-policy Q-function, and compares it with the alternative of running the baseline policy forever. The deployed system need not actually return to the baseline after one step: the proposer agent observes the new state and may submit another proposal that depends on the entire history. Reviewers therefore need not—and at the time of voting cannot—evaluate the full trajectory that ultimately unfolds. Nevertheless, the principal’s expected gain over the baseline equals the expected discounted sum of these one-step gains along the actual trajectory, which yields our main control result. Informal result (Local safety is equivalent to global safety). Repeated authorization is safe against every history-adaptive proposer agent if and only if every proposal that can be authorized has nonnegative principal baseline advantage at the state where it is proposed. For threshold voting, this holds exactly when robust coalitional alignment holds at every state. Theorem 2 formalizes this equivalence for arbitrary state-dependent authorization rules and gives the corresponding coalitional characterization for general monotone rules. Strategic reviewers. In a one-shot authorization problem, sincere voting is a (weakly) dominant strategy for the reviewers (Proposition 2). But it is no longer dominant in a multi-round problem. We therefore study strategic reviewers directly, allowing them to choose history-dependent voting policies that maximize their own returns along the actual trajectory. Informal result (Strategic safety under threshold voting). Under the approval rule that requires reviewer unanimity, full-panel coverage in reward-function space guarantees that every Nash equilibrium is safe. A threshold that tolerates even one disapproval can have unsafe Nash equilibria—even when every reviewer is perfectly aligned. But if every subpanel left after deleting k reviewers covers the principal’s reward function, safety holds at any equilibrium stable against joint deviations by k+1k+1 reviewers. Theorem 3 gives the formal guarantee, and Proposition 3 shows why ordinary Nash equilibrium is insufficient for permissive thresholds. Completeness and the limits of binary approval. Soundness is a one-sided guarantee that promises safety but not usefulness — a sound approval policy could reject many beneficial proposals. We also study completeness, which requires sufficiently beneficial proposals to be executed. We establish a basic limit on what any binary approval rule can accomplish. Informal result (A limit of binary approval and two ways around it). When feasible lotteries include both principal gains and losses, any rule based only on binary votes that always authorizes every principal-acceptable lottery and rejects every harmful one requires an aligned reviewer. But this requirement does not persist under two relaxations. First we can allow approximate soundness and completeness. Under a simple generative model in which reviewers are noisy perturbations of the principal, majority voting converges to soundness and completeness with high probability even though no reviewer converges to alignment; second, if we allow reviewers to report cardinal utilities rather than binary judgments, we can have both perfect soundness and completeness without an aligned reviewer. Theorem 4 gives the exact impossibility, Proposition 4 gives the random and deterministic approximation results, and Corollary 9 gives the cardinal-score result. Supporting appendices record two further results. Appendix D combines arbitrary reports, approximate coverage, and bounded evaluation error into a single robustness guarantee. Appendix E shows that auditing robust coverage is tractable in fixed dimension or at a fixed robustness level, but is coNP-complete even under approximation (Proposition 8). Experiments. We study panels of existing reward models and safety evaluators (Section 7). In answer selection, we find that unanimous review can block every lottery over the candidate answers that lowers the probability of a correct answer relative to the baseline, even when each reviewer would approve some such lottery. This gives empirical examples of coalitional alignment without individual alignment. Coverage can also survive reviewer removals, allowing sound review to tolerate disapprovals. Using numerical scores instead of binary votes improves the resulting tradeoff: more correct answers are authorized at comparable soundness. For safety evaluators, the benefit comes from overlapping objections: every harmful response in the evaluation receives multiple disapprovals. Tolerating one disapproval releases more acceptable responses while still withholding every harmful response. 1.3 Related Work AI Alignment. The most closely related prior works are Collina et al. (2026a); Collina et al. (2026b), who study “emergent alignment” that results from differently misaligned model providers competing to deploy LLMs that will influence user actions. Like us, these papers ask when an outcome comparable to what could be obtained from perfect alignment can instead be obtained from a weaker population level condition. Our k-robust coalitional alignment condition is a generalization of their “market alignment” condition (equivalent for k=0k=0) — but the problem we study is very different. In their model users are the only ones who can take action, and the AI providers must compete for user attention in order to influence action. In contrast we study AI agents who are able to act on their own, and study oversight by other AI models who can collectively block action, but are not competing for users in a market. AI-control protocols and guard agents also delegate oversight to models (Greenblatt et al., 2024; Xiang et al., 2025; Chen et al., 2025), but the focus of this literature is on detecting intentional subversion or enforcing a specified safety policy. Complementary empirical work shows that partitioning a common set of evaluation criteria across model calls can improve model-based oversight and reduce vulnerability to adversarial presentation (Akinwande et al., 2026). Economic Theory. Coalitional alignment is a geometric analogue of aggregating expected-utility preferences under a Pareto condition (Harsanyi, 1955). The winning coalitions and groups that can block authorization in our characterizations are also studied in the theory of simple games (Freixas et al., 2011). Classical expert-panel models aggregate noisy judgments from experts who share a common objective (Nitzan and Paroush, 1983); our reviewers may have genuinely different objectives. Reinforcement Learning. Our sequential model is closest to runtime assurance and intervention-based safe RL. The Simplex architecture switches from an advanced controller to a verified baseline controller (Seto et al., 1998); shields and advantage-based intervention similarly filter actions using a trusted specification or safety objective (Alshiekh et al., 2018; Wagener et al., 2021). Safe policy improvement guarantees performance relative to a baseline (Pirotta et al., 2013; Thomas et al., 2015; Laroche et al., 2019), and safe RL with binary feedback learns from an oracle that labels actions safe or unsafe (Bennett et al., 2023). We instead aggregate comparisons from reviewers with different rewards and obtain a guarantee against any history-adaptive proposer agent. 2 The one-decision model and geometric preliminaries We first isolate a single authorization decision. The proposer agent submits an action distribution, and each reviewer compares its induced outcomes with the baseline according to its own utility. We ask when the resulting approvals are sufficient to rule out a loss for the principal, even though the reviewers may have different objectives. Here the feasible outcomes and utility functions are fixed; in Section 4, the same decision problem will recur at each state, with utilities given by continuation values. 2.1 The decision problem and feasible deviations For any finite set X, write Δ(X):=p∈ℝ+X:∑x∈Xpx=1 (X):= \p _+^X: _x∈ Xp_x=1 \ for the simplex of probability distributions on X. Definition 2.1 (One-decision authorization problem). Let Ω be a finite outcome space. The principal P and reviewers i∈N:=1,…,ni∈ N:=\1,…,n\ have arbitrary real-valued utility functions uj:Ω→ℝ,j∈P,1,…,n.u_j: , j∈\P,1,…,n\. Each participant evaluates a lottery μ∈Δ(Ω)μ∈ ( ) by its expected utility Uj(μ):=ω∼μ[uj(ω)]=∑ω∈Ωμ(ω)uj(ω).U_j(μ):=E_ω μ[u_j(ω)]= _ω∈ μ(ω)u_j(ω). Let A=a0,a1,…,amA=\a_0,a_1,…,a_m\ be a finite action set. The distinguished action a0a_0 is the baseline, and each action a induces an outcome lottery μa∈Δ(Ω) _a∈ ( ). A proposal is a randomized action q∈Δ(A)q∈ (A), which induces the outcome lottery μ(q):=∑a∈Aq(a)μa.μ(q):= _a∈ Aq(a) _a. Writing μ0:=μa0 _0:= _a_0, participant j’s gain from q relative to the baseline is gj(q):=Uj(μ(q))−Uj(μ0).g_j(q):=U_j(μ(q))-U_j( _0). The utility functions need not bear any particular relationship to one another. This standard expected-utility model has a convenient geometric representation. Identify each utility function with its vector of values in ℝΩR and each lottery with its probability vector. Definition 2.2 (Feasible deviations). The deviation induced by a proposal q is d(q):=μ(q)−μ0.d(q):=μ(q)- _0. For each nonbaseline action, write da:=μa−μ0d_a:= _a- _0. The set of feasible deviations and its conic hull are :=conv(0∪da:a≠a0)=∑a≠a0θada:θa≥0,∑a≠a0θa≤1, :=conv (\0\∪\d_a:a≠ a_0\ )= \ _a≠ a_0 _ad_a: _a≥ 0,\ _a≠ a_0 _a≤ 1 \, :=cone()=∑a≠a0θada:θa≥0. :=cone(D)= \ _a≠ a_0 _ad_a: _a≥ 0 \. Let L:=span()L:=span(C) be the feasible span. Because deviations are differences between distributions, L is contained in the zero-sum subspace H0:=d∈ℝΩ:⟨,d⟩=0.H_0:=\d : 1,d =0\. Expected-utility gain has the geometric representation gj(q)=⟨uj,d(q)⟩,j∈P,1,…,n.g_j(q)= u_j,d(q) , j∈\P,1,…,n\. Thus the geometry below is simply a representation of the underlying decision problem. Multiplying a deviation by a positive constant changes the size of every gain but preserves its sign. Every direction in C can be scaled into D, so exact soundness can be studied on the cone. Bounds on the size of a loss must instead retain the bounded set D. Intuitively, a reviewer is aligned with the principal if they share the same utility function. The formal definition allows positive rescaling, which does not change binary comparisons, and only compares utilities on feasible deviations. Definition 2.3 (Individual alignment). Reviewer i is aligned with the principal on the feasible span if there is a constant c>0c>0 such that ⟨ui,d⟩=c⟨uP,d⟩for every d∈L. u_i,d =c\, u_P,d every d∈ L. An aligned reviewer and the principal therefore evaluate every feasible direction proportionally; their utility vectors may differ arbitrarily on L⟂L . 2.2 Authorization rules We assume for now that the reviewers are sincere: reviewer i approves a proposal with deviation d when ⟨ui,d⟩≥0 u_i,d ≥ 0 and disapproves when ⟨ui,d⟩<0 u_i,d <0. Later we model strategic and noisy reviewers. Definition 2.4 (Authorization rule). For a deviation d, its approval set is the set of reviewers that approve it: S(d):=i∈N:⟨ui,d⟩≥0.S(d):=\i∈ N: u_i,d ≥ 0\. An authorization rule is a family ⊆2NW 2^N of winning approval sets. A proposal with deviation d is executed exactly when S(d)∈S(d) ; otherwise the baseline action is executed. We require N∈N —the admissibility condition—so unanimous approval is always sufficient for execution. A rule is monotone if S∈ and S⊆T⟹T∈.S and S T T . Requiring N∈N excludes the reject-all rule, which is trivially sound but never delegates authorization. Monotonicity says that an additional approval cannot turn execution into rejection. Definition 2.5 (Minimal winning coalitions). Let ℳ()M(W) be the set of inclusion-minimal winning approval sets. Every winning set contains a member of ℳ()M(W). For a monotone rule, the converse also holds, so S(d)∈⟺S(d) contains some W∈ℳ().S(d) $S(d)$ contains some $W∈ M(W)$. Minimal winning coalitions identify the approvals sufficient to execute a proposal. Example 2.1 (Two-out-of-three authorization). Let N=1,2,3N=\1,2,3\. For a proposed deviation d from the baseline, S(d)⊆NS(d) N denotes the reviewers who approve it. Under two-out-of-three authorization, =1,2,1,3,2,3,N.W= \\1,2\,\1,3\,\2,3\,N \. Thus d is executed exactly when S(d)∈S(d) , or equivalently when at least two reviewers approve. The inclusion-minimal winning sets are ℳ()=1,2,1,3,2,3,M(W)= \\1,2\,\1,3\,\2,3\ \, since each pair is sufficient for execution, while no singleton is. Definition 2.6 (Soundness). For the fixed panel, an authorization rule is sound on C if every executed d∈d satisfies ⟨uP,d⟩≥0 u_P,d ≥ 0; that is, every executed proposal weakly improves the principal’s expected utility relative to the baseline. 2.3 Coalitional coverage and coalitional alignment An individually aligned reviewer can certify safety on its own: whenever it approves, the principal’s gain is also nonnegative. We seek a condition under which several reviewers can provide this guarantee together, even if none can provide it alone. This requires relating the principal’s utility to a nonnegative combination of their utilities. The relation need not be an exact equality, since a difference that is nonnegative on every feasible deviation can only improve the principal’s guarantee. We describe these allowable differences using the dual cone. Definition 2.7 (Dual cone). The dual cone of the feasible proposal cone is ∗:=z∈ℝΩ:⟨z,d⟩≥0 for every d∈.C^*:=\z : z,d ≥ 0 for every d \. We now introduce our main coverage condition: Definition 2.8 (Coalitional coverage relative to C). A coalition S⊆NS N covers the principal relative to C if uP∈coneui:i∈S+∗.u_P \u_i:i∈ S\+C^*. Equivalently, there exist coefficients λi≥0 _i≥ 0 and a vector z∈∗z ^* such that uP=∑i∈Sλiui+z.u_P= _i∈ S _iu_i+z. To gain intuition for this definition, suppose uP=∑i∈Sλiui+zu_P= _i∈ S _iu_i+z, where each λi≥0 _i≥ 0 and z∈∗z ^*. If every reviewer in S approves a feasible deviation d, then ⟨uP,d⟩=∑i∈Sλi⟨ui,d⟩+⟨z,d⟩≥0. u_P,d = _i∈ S _i u_i,d + z,d ≥ 0. Thus, coverage implies soundness under the unanimous approval rule. Our characterization will show the reverse implication as well. The dual-cone term contributes a nonnegative gain on every feasible deviation. This allows the principal’s utility to differ from the reviewers’ weighted utility while preserving soundness under unanimous approval. For a threshold rule that permits some disapprovals, soundness must follow from the reviewers who approve. Coverage must therefore survive the removal of those who disapprove, which motivates the following robust form of coalitional alignment. Definition 2.9 (k-robust coalitional alignment). Adopt the convention cone(∅)=0cone( )=\0\. For k∈0,…,nk∈\0,…,n\, the panel is k-robustly coalitionally aligned relative to C if uP∈coneui:i∈S+∗for every S⊆N with |S|=n−k.u_P \u_i:i∈ S\+C^* every S N with |S|=n-k. Because coverage is preserved when reviewers are added, this is equivalent to coverage by every coalition remaining after deletion of at most k reviewers. 3 Sound authorization and rejection depth We have seen that coalitional coverage guarantees safety when every member of a coalition approves. A general authorization rule can accept proposals without unanimous approval, so we need to determine which coalitions must provide this guarantee and whether coverage is necessary. We first study these questions through the rule’s minimal winning coalitions. We then specialize to threshold rules, where soundness depends on the fewest objections a harmful proposal can receive. 3.1 Soundness under general authorization rules A simple example shows why a coalition can protect the principal when its members cannot do so individually. Suppose two reviewers care about different dimensions of an outcome, and the principal’s utility is the sum of theirs. A harmful proposal must then reduce at least one reviewer’s utility, so requiring both approvals guarantees soundness. Either reviewer alone may nevertheless approve a proposal whose gain in one dimension is outweighed by its loss in the other. Figure 1 illustrates the resulting difference between unanimous review and review by one member of the panel. (a) Authorized deviationsd1d_1d2d_2authorized byW=1,2W=\1,2\acceptablebut rejected⟨uP,d⟩=0 u_P,d =0 u1=e1,u2=e2u_1=e_1, 8.50012ptu_2=e_2 uP=e1+e2u_P=e_1+e_2 (b) Coverage certificateu1u_1u2u_2uPu_Pconeu1,u2cone\u_1,u_2\uP=u1+u2u_P=u_1+u_2 Utility-vector space =ℝ2,∗=0C=R^2, 8.50012ptC^*=\0\ (c) Failure of coveraged1d_1d2d_2authorized byW=1W=\1\acceptablebut rejectedharmful andauthorizeddd d=(1,−2)d=(1,-2) ⟨u1,d⟩≥0,⟨uP,d⟩<0 u_1,d ≥ 0, 8.50012pt u_P,d <0 Figure 1: Two complementary reviewers illustrate the soundness certificate. (a) Unanimity authorizes the first quadrant, contained in the principal’s acceptable halfspace. (b) The same inclusion is certified by uP=u1+u2u_P=u_1+u_2. (c) Reviewer 1 alone authorizes the hatched harmful region, including d=(1,−2)d=(1,-2). Green marks acceptable deviations; blue marks authorization, with hatching identifying the harmful overlap. The example suggests examining the proposals approved unanimously by each coalition. Such approvals guarantee safety exactly when the coalition covers the principal. The following theorem relates this condition to soundness of an authorization rule, giving an exact characterization for monotone rules. Theorem 1 (Coalitional coverage and soundness). Let W be any authorization rule. Conditions (i) and (i) below are equivalent, and each implies soundness (i). If W is monotone, all three conditions are equivalent: (i) W is sound. (i) For every minimally winning coalition W∈ℳ()W (W), the cone of feasible deviations approved by all members of W lies in the principal’s acceptable halfspace: ∩⋂i∈Wd:⟨ui,d⟩≥0⊆d:⟨uP,d⟩≥0.C∩ _i∈ W\d: u_i,d ≥ 0\ \d: u_P,d ≥ 0\. (i) Every minimally winning coalition covers the principal relative to C: uP∈coneui:i∈W+∗for every W∈ℳ().u_P \u_i:i∈ W\+C^* every W (W). Equivalently, for every W∈ℳ()W (W) there exist λiW≥0 _i^W≥ 0 and zW∈∗z_W ^* such that uP=∑i∈WλiWui+zW.u_P= _i∈ W _i^Wu_i+z_W. Proof. Coverage and safety for a fixed coalition. For any coalition W⊆NW N, set KW:=∩⋂i∈Wd:⟨ui,d⟩≥0.K_W:=C∩ _i∈ W\d: u_i,d ≥ 0\. We first establish the coalition-wise equivalence uP∈coneui:i∈W+∗⟺⟨uP,d⟩≥0 for every d∈KW.u_P \u_i:i∈ W\+C^* u_P,d ≥ 0 for every d∈ K_W. (1) If W covers the principal, write uP=∑i∈Wλiui+zu_P= _i∈ W _iu_i+z with λi≥0 _i≥ 0 and z∈∗z ^*. For every d∈KWd∈ K_W, ⟨uP,d⟩=∑i∈Wλi⟨ui,d⟩+⟨z,d⟩≥0. u_P,d = _i∈ W _i u_i,d + z,d ≥ 0. Conversely, suppose every d∈KWd∈ K_W has nonnegative principal gain. Because C is polyhedral (it is generated by finitely many action deviations), there are vectors a1,…,aℓa_1,…,a_ such that =d:⟨ah,d⟩≥0 for h=1,…,ℓ.C=\d: a_h,d ≥ 0 for h=1,…, \. To search for a harmful direction approved by every member of W, consider the linear program minimized∈ℝΩ d minimize ⟨uP,d⟩ u_P,d subject to to ⟨ah,d⟩≥0 a_h,d ≥ 0 (h=1,…,ℓ), (h=1,…, ), ⟨ui,d⟩≥0 u_i,d ≥ 0 (i∈W). (i∈ W). Its feasible set is KWK_W, so its optimal value is zero: the objective is nonnegative on KWK_W, and d=0d=0 is feasible. Its LP dual is maximizeα,λ α,λmaximize 0 0 subject to to uP=∑h=1ℓαhah+∑i∈Wλiui, u_P= _h=1 _ha_h+ _i∈ W _iu_i, α∈ℝ+ℓ,λ∈ℝ+W. α _+ , λ _+^W. Since the primal is feasible with finite optimum, LP strong duality guarantees that such multipliers exist. Each aha_h belongs to ∗C^*, so z:=∑h=1ℓαhahz:= _h=1 _ha_h also belongs to ∗C^*. The dual equality therefore gives a coverage certificate for W. Applying this equivalence to each minimally winning coalition proves (i)⟺ (i). From coalitions to authorization rules. If (i) holds and a proposal is executed, its approval set contains some W∈ℳ()W (W); its deviation belongs to KWK_W, so its principal gain is nonnegative. Hence (i)⟹ (i) for any rule. Conversely, suppose the rule is monotone and sound. For any W∈ℳ()W (W) and d∈KWd∈ K_W, W⊆S(d)W S(d); monotonicity makes S(d)S(d) winning, and soundness gives ⟨uP,d⟩≥0 u_P,d ≥ 0. Thus KWK_W lies in the principal’s acceptable halfspace, which is (i). ∎ Monotonicity is needed only for the converse. A nonmonotone rule may use the disapprovals outside an accepted set as additional information, so soundness need not force that set’s approvals alone to certify safety. Because every admissible rule (N∈N ) executes under unanimous approval, full-panel coverage is the minimum requirement for safe delegation of any form. Corollary 1 (Existence of an admissible sound rule). An admissible sound authorization rule, not necessarily monotone, exists if and only if the full panel covers the principal relative to C. When this condition holds, unanimity is sound. Proof. If the full panel covers the principal, unanimity is sound by Theorem 1. Conversely, every admissible rule executes under unanimous approval. Soundness therefore makes every deviation approved by all reviewers principal-acceptable, so equivalence (1) gives full-panel coverage. ∎ The characterization also gives a way to construct counterexamples to soundness. If a minimally winning coalition W fails to cover the principal, equivalence (1) gives a direction d∈d such that ⟨ui,d⟩≥0∀i∈W,⟨uP,d⟩<0. u_i,d ≥ 0 ∀ i∈ W, u_P,d <0. Since =cone()C=cone(D), a sufficiently small positive multiple of d belongs to D. Scaling preserves the signs of all utility gains, so this gives a feasible proposal that every member of W approves but that harms the principal. If the rule is monotone, it executes this proposal even if reviewers outside W also approve. For each fixed coalition W, linear programming can find either coefficients certifying coverage or such a harmful proposal. The same characterization also answers a design question: for a fixed panel, how permissive can a sound monotone authorization rule be? Definition 3.1 (Maximally permissive sound approval sets). Define sndmax:=S⊆N:uP∈coneui:i∈S+∗.W_snd := \S N:u_P \u_i:i∈ S\+C^* \. Thus an approval set belongs to sndmaxW_snd exactly when its members’ approvals alone certify nonnegative principal gain on every feasible proposal. Proposition 1 (Maximally permissive sound monotone rule). If the full panel covers the principal, then sndmaxW_snd is a sound monotone authorization rule. Every sound monotone rule W satisfies ⊆sndmax.W _snd . More generally, any family contained in sndmaxW_snd is sound; if it contains N, it is an admissible authorization rule. Proof. Coverage is preserved when reviewers are added, so sndmaxW_snd is upward closed; full-panel coverage ensures that it contains N. If S(d)∈sndmaxS(d) _snd , write uP=∑i∈S(d)λiui+z,λi≥0,z∈∗.u_P= _i∈ S(d) _iu_i+z, _i≥ 0, z ^*. Every reviewer in S(d)S(d) has nonnegative gain and ⟨z,d⟩≥0 z,d ≥ 0, so ⟨uP,d⟩≥0 u_P,d ≥ 0. This proves soundness, as well as soundness of every subfamily. Conversely, Theorem 1 says that every minimally winning coalition of a sound monotone rule covers the principal. Every winning coalition contains such a coalition and therefore covers as well, proving the inclusion. ∎ Exact coverage excludes approval of any action that results in even tiny principal loss. If the principal’s utility differs slightly from a covering combination, the same argument can bound the loss rather than rule it out. We formalize this relaxation by allowing a residual whose value on any feasible proposal is no smaller than −ε- . Definition 3.2 (Approximate soundness and lower coverage). For ε≥0 ≥ 0, a rule is ε -sound on D if every executed d∈d satisfies ⟨uP,d⟩≥−ε u_P,d ≥- . Define the lower residual set ℛε−():=r∈ℝΩ:⟨r,d⟩≥−ε for every d∈.R^-_ (D):= \r : r,d ≥- for every d \. A coalition S ε -covers the principal from below on D if uP∈coneui:i∈S+ℛε−().u_P \u_i:i∈ S\+R^-_ (D). The panel has k-robust lower coalitional alignment on D at tolerance ε if every coalition of n−kn-k reviewers has this property. Equivalently, every coalition of at least n−kn-k reviewers does. We state exact results on the scale-free cone C and nonzero error bounds on D, because the magnitude of an approximation error depends on scale. These parameters do not alter sincere voting: reviewers still approve exactly when their own gain is nonnegative. Appendix A gives the approximate counterpart of Theorem 1. 3.2 Threshold voting and rejection depth We now specialize the characterization to threshold rules, which decide whether to execute a proposal using only the number of disapprovals. Because these rules treat reviewers symmetrically, their minimal winning coalitions are determined by a single count. The question is how many disapprovals the rule can tolerate while still rejecting every harmful proposal. This is determined by the harmful proposal that receives the fewest objections, which motivates the following definition of rejection depth. Definition 3.3 (Threshold rule and rejection depth). For k∈0,…,n−1k∈\0,…,n-1\, the k-tolerant threshold rule executes a proposal when at most k reviewers disapprove. Write N−(d):=i∈N:⟨ui,d⟩<0N^-(d):=\i∈ N: u_i,d <0\ for the reviewers who disapprove deviation d. The rejection depth of the principal relative to the panel is the fewest objections received by any harmful feasible direction: depth(uP,u1,…,un):=mind∈⟨uP,d⟩<0|N−(d)|.depth_C(u_P;u_1,…,u_n):= _ subarraycd \\ u_P,d <0 subarray|N^-(d)|. See Figure 2 for a visualization. If the principal has no harmful feasible direction, set the depth equal to n+1n+1. Rejection depthd1d_1d2d_2uPu_Pu1u_1u2u_2u3u_3u4u_4DHP−H_P^-223344332200d⋆:2d :2dpure:4d_pure:4rejections223344depth=2depth_C=2 Figure 2: Rejection depth in a two-dimensional deviation space. Each utility vector is normal to a dashed zero-gain line, and reviewer i rejects on the side opposite uiu_i. Within the principal-harmful region HP−H_P^-, the purple shade and number give |N−(d)||N^-(d)|. The harmful mixed-proposal deviation d⋆d receives the minimum of two objections, whereas the harmful pure deviation receives four. Thus a k-tolerant rule is sound exactly when every harmful feasible proposal receives at least k+1k+1 objections. Its minimal winning coalitions are all sets of n−kn-k reviewers, so Theorem 1 gives an equivalent condition: coverage must survive the deletion of any k reviewers. Rejection depth and robust coalitional alignment therefore express the same requirement, one by counting objections and the other by relating the reviewers’ utilities to the principal’s. Corollary 2 (Coalitional alignment characterizes threshold soundness). For every k∈0,…,n−1k∈\0,…,n-1\, the following are equivalent: (i) the k-tolerant threshold rule is sound; (i) depth(uP,u1,…,un)≥k+1depth_C(u_P;u_1,…,u_n)≥ k+1; (i) the panel is k-robustly coalitionally aligned relative to C. More generally, the rule is εharm _harm-sound on D if and only if the panel has k-robust lower coalitional alignment on D at tolerance εharm _harm. Proof. The rule executes exactly when |N−(d)|≤k|N^-(d)|≤ k. It is therefore sound exactly when every harmful deviation has |N−(d)|≥k+1|N^-(d)|≥ k+1, proving (i)⟺ (i). Its minimally winning coalitions are the (n−k)(n-k)-element sets, so Theorem 1 turns the same condition into (i). Replacing exact coverage by lower coverage and using Theorem 5 (Appendix A) proves the approximate statement. ∎ 3.3 How robust coalitional alignment can arise We close the static analysis with a simple way in which coalitional alignment can arise from reviewer heterogeneity. Suppose reviewers are sampled independently and their misalignment directions are centrally symmetric around the principal’s objective. Informally, this models a setting in which alignment is hard because when attempting to produce an aligned agent, their realized utility function is noisy (perhaps very noisy), but not systematically biased. Then every harmful direction is opposed by at least half the population. Uniform convergence makes the same statement hold, up to a vanishing error, for a sufficiently large sampled panel. Corollary 3 (Centrally symmetric objective noise). Suppose the reviewer objectives are drawn independently and identically as ui=uP+ξiu_i=u_P+ _i, and every one-dimensional projection ⟨ξi,d⟩ _i,d is centrally symmetric about zero. In fixed feasible dimension, 1ndepth(uP,u1,…,un)≥12−o(1) 1ndepth_C(u_P;u_1,…,u_n)≥ 12-o(1) with high probability as n→∞n→∞. If the population assigns probability zero to the principal’s positive utility ray on the feasible span, then almost surely no sampled reviewer is individually aligned. Nevertheless, for fixed feasible dimension the panel soundly tolerates deletion of nearly half its reviewers with high probability as n grows. Appendix A gives the finite-sample bound, a more general population condition, and the proof. 4 Sequential authorization in a discounted MDP The one-decision model evaluates a proposal in isolation. For a long-running agent, authorization instead takes place repeatedly: each executed action changes the state from which the next proposal will be made. A reviewer must therefore account for the action’s future consequences as well as its immediate reward. We extend the model to a discounted MDP and replace the baseline action with a baseline policy. This policy specifies what to do after rejection and provides a common continuation against which each participant evaluates a proposed action. 4.1 Control model and continuation-aware votes Definition 4.1 (Discounted control model). Consider a finite discounted MDP with state space S, action space A, transition kernel P, and discount factor γ∈(0,1)γ∈(0,1). Participant j∈P,1,…,nj∈\P,1,…,n\ has bounded reward function rj(s,a)r_j(s,a). Fix a stationary randomized baseline policy π0π^0. Participant j’s baseline value and action-value functions are Vj0(s) V_j^0(s) :=π0[∑t=0∞γtrj(st,at)|s0=s], :=E_π^0 [ _t=0^∞γ^tr_j(s_t,a_t)\, |\,s_0=s ], Qj0(s,a) Q_j^0(s,a) :=rj(s,a)+γs′∼(⋅∣s,a)[Vj0(s′)], :=r_j(s,a)+ _s P(· s,a)[V_j^0(s )], respectively. These two value functions compare continuations from the same state. Vj0(s)V_j^0(s) is participant j’s expected discounted reward from following the baseline policy starting at s. In Qj0(s,a)Q_j^0(s,a), only the first action changes: action a is taken at s and the baseline is followed thereafter. Their difference measures how this one-step deviation changes the participant’s long-run reward, and is called the baseline advantage: The use of Q-values rather than immediate rewards is essential. Two actions can have the same reward at the current state while sending the system to states with very different future value. A reviewer voting from ri(s,a)r_i(s,a) alone would miss this effect; Qi0(s,a)Q_i^0(s,a) evaluates the complete deviation consisting of the proposed action followed by the baseline continuation. Reviewers therefore need access to their baseline-continuation values, or estimates of them; Appendix D bounds the effect of errors in those evaluations. Definition 4.2 (Continuation-aware authorization). At state s, the proposer agent selects a randomized action q∈s⊆Δ(A)q _s (A), where sQ_s is a convex polytope containing π0(⋅∣s)π^0(· s). Reviewer i approves when the proposed one-step deviation, followed conceptually by the baseline continuation, weakly improves their value: ∑aq(a)Qi0(s,a) _aq(a)Q_i^0(s,a) ≥Vi0(s) ≥ V_i^0(s) =∑aπ0(a∣s)Qi0(s,a). = _aπ^0(a s)Q_i^0(s,a). Equivalently, ⟨Qi0(s,⋅),q−π0(⋅∣s)⟩≥0. Q_i^0(s,·),q-π^0(· s) ≥ 0. Write Ss(q):=i∈N:⟨Qi0(s,⋅),q−π0(⋅∣s)⟩≥0S_s(q):= \i∈ N: Q_i^0(s,·),q-π^0(· s) ≥ 0 \ for the approval set at state s. A state-dependent authorization rule specifies a family s⊆2NW_s 2^N with N∈sN _s at each state. It executes q when Ss(q)∈sS_s(q) _s. If the proposal is rejected, the rule instead samples an action from π0(⋅∣s)π^0(· s). The rule uses the state to select sW_s and receives one bit from each reviewer. It is monotone if each family sW_s is upward closed. The statewise deviation set and cone are s _s :=q−π0(⋅∣s):q∈s, :=\q-π^0(· s):q _s\, s _s :=cone(s). :=cone(D_s). Because sQ_s is a polytope, sC_s is a polyhedral cone. The baseline Q-vectors Qj0(s,⋅)Q_j^0(s,·) are the analogue of the utility vectors in the static setting. The proposer agent need not follow a stationary or Markov policy. At every time it may choose its proposal as an arbitrary function of the entire observed history. The authorization layer is thus required to protect the principal against adaptive sequences of individually reviewed actions, not merely against a fixed candidate policy. 4.2 From local authorization to global control The static characterization applies at each state with the baseline Q-vectors as utilities, but the guarantee it gives concerns a particular continuation: take the proposed action once and then follow the baseline. The actual process need not follow that continuation. After the action is executed, the proposer agent observes the new state and may submit another proposal, possibly in response to the entire history. Figure 3 illustrates the difference between the comparison used to cast a vote and the trajectory that results from repeated authorization. We need to show that safety of each local comparison protects the principal along this adaptive trajectory. (a) What reviewers evaluatests_tcurrent statetake qtq_t oncethen follow π0π^0proposed counterfactualfollow π0π^0 from sts_tbaselineapprove iff ∑aqt(a)Qi0(st,a)≥Vi0(st) _aq_t(a)Q_i^0(s_t,a)≥ V_i^0(s_t)(b) What actually happenss0s_0s1s_1⋯·sq0(h0)q_0(h_0)q1(h1)q_1(h_1)[AP0(st,at)∣ht]≥0E[A_P^0(s_t,a_t) h_t]≥ 0 Each new proposal may depend on the entire observed history. Figure 3: (a) Reviewer i compares taking the proposal once and then following the baseline with following the baseline immediately. (b) The actual process may propose again after every transition. Local soundness makes each authorized proposal’s conditional baseline advantage nonnegative, even when proposals depend on the observed history. The next theorem establishes the equivalence of the local condition (i) with the global guarantee (i); for monotone rules, the local condition has the same coalitional coverage form as in the static analysis. Theorem 2 (Local soundness characterizes baseline safety). Fix state-dependent authorization rules (s)s∈(W_s)_s , not necessarily monotone. The following are equivalent: (i) At every state s, every proposal q∈sq _s that the rule executes has nonnegative principal baseline advantage: ⟨QP0(s,⋅),q−π0(⋅∣s)⟩≥0. Q_P^0(s,·),q-π^0(· s) ≥ 0. (i) For every initial state s0s_0 and every proposer agent whose proposals may depend arbitrarily on the full interaction history, the resulting history-dependent policy π satisfies JP(π∣s0)≥VP0(s0),J_P(π s_0)≥ V_P^0(s_0), where JP(π∣s0):=π[∑t=0∞γtrP(st,at)|s0].J_P(π s_0):=E_π [ _t=0^∞γ^tr_P(s_t,a_t)\, |\,s_0 ]. If each sW_s is monotone, these conditions are further equivalent to statewise coalitional coverage: at every state s and for every W∈ℳ(s)W (W_s), QP0(s,⋅)∈coneQi0(s,⋅):i∈W+s∗.Q_P^0(s,·) \Q_i^0(s,·):i∈ W\+C_s^*. The reason local safety suffices is the performance-difference identity. It expresses the principal’s expected gain over the baseline as the expected discounted sum of baseline advantages along the actual trajectory. Conditional on any observed history, local soundness makes the advantage of an authorized proposal nonnegative in expectation; choosing the baseline after rejection contributes zero. Summing these quantities gives the global guarantee even when future proposals adapt to earlier outcomes. Conversely, if local soundness fails at one state, a proposer starting there can submit the harmful proposal once and then return to the baseline. The resulting loss shows why the local condition is also necessary. If local soundness holds only up to an error, the same identity bounds the accumulated loss. Each state’s error contributes in proportion to how often that state is visited and how far into the future it is reached. Corollary 4 (Approximate local losses compose globally). Fix a function εharm:→[0,∞) _harm:S→[0,∞). For monotone rules, the statewise guarantee that every executed proposal at s has principal baseline advantage at least −εharm(s)- _harm(s) is equivalent to QP0(s,⋅)∈coneQi0(s,⋅):i∈W+ℛεharm(s)−(s)for every W∈ℳ(s).Q_P^0(s,·) \Q_i^0(s,·):i∈ W\+R^-_ _harm(s)(D_s) every W (W_s). For rules of any form, these local bounds imply JP(π∣s0)−VP0(s0)≥−π[∑t=0∞γtεharm(st)|s0].J_P(π s_0)-V_P^0(s_0)≥-E_π [ _t=0^∞γ^t _harm(s_t)\, |\,s_0 ]. In particular, a uniform bound εharm(s)≤ε¯harm _harm(s)≤ _harm gives JP(π∣s0)≥VP0(s0)−ε¯harm1−γ.J_P(π s_0)≥ V_P^0(s_0)- _harm1-γ. Proof of Theorem 2 and Corollary 4. Local bounds imply global bounds. Consider first the general local bound in the corollary; the exact implication is the special case εharm≡0 _harm≡ 0. For an executed proposal q at state s, the identity ∑aπ0(a∣s)QP0(s,a)=VP0(s), _aπ^0(a s)Q_P^0(s,a)=V_P^0(s), shows that a local principal-gain bound −εharm(s)- _harm(s) is equivalent to ∑aq(a)AP0(s,a)≥−εharm(s). _aq(a)A_P^0(s,a)≥- _harm(s). A rejected proposal causes the rule to use π0(⋅∣s)π^0(· s), whose expected baseline advantage is exactly zero. Hence, conditional on every history hth_t, π[AP0(st,at)∣ht]≥−εharm(st).E_π[A_P^0(s_t,a_t) h_t]≥- _harm(s_t). For every finite T, the Bellman equation for VP0V_P^0 telescopes to the performance-difference identity (Kakade and Langford, 2002) π[∑t=0TγtAP0(st,at)] _π [ _t=0^Tγ^tA_P^0(s_t,a_t) ] =π[∑t=0TγtrP(st,at)]−VP0(s0) =E_π [ _t=0^Tγ^tr_P(s_t,a_t) ]-V_P^0(s_0) +γT+1π[VP0(sT+1)]. +γ^T+1E_π[V_P^0(s_T+1)]. Since rewards and VP0V_P^0 are bounded, the final term vanishes as T→∞T→∞. Therefore JP(π∣s0)−VP0(s0)=π[∑t=0∞γtAP0(st,at)].J_P(π s_0)-V_P^0(s_0)=E_π [ _t=0^∞γ^tA_P^0(s_t,a_t) ]. Iterated expectation now gives JP(π∣s0)−VP0(s0)≥−π[∑t=0∞γtεharm(st)|s0].J_P(π s_0)-V_P^0(s_0)≥-E_π [ _t=0^∞γ^t _harm(s_t)\, |\,s_0 ]. Taking εharm≡0 _harm≡ 0 proves (i)⟹ (i), and a uniform bound is summed by the geometric series. This proves the corollary’s quantitative claims. Global bounds force local soundness. Suppose (i) fails at some state s. Then an executed proposal q∈sq _s satisfies ⟨QP0(s,⋅),q−π0(⋅∣s)⟩<0. Q_P^0(s,·),q-π^0(· s) <0. Starting from s, let the proposer agent submit this proposal once and submit the baseline distribution thereafter. The resulting policy has negative expected baseline advantage at time zero and zero expected baseline advantage at every later time. The performance-difference identity therefore gives JP(π∣s)<VP0(s),J_P(π s)<V_P^0(s), contradicting (i). Hence (i) implies (i). Finally, for monotone rules, applying Theorem 1 and Theorem 5 (Appendix A) at each state gives the two coalitional characterizations. ∎ Specializing to threshold review recovers the language of robust coalitional alignment, now applied to the baseline Q-vectors at each state: Corollary 5 (Threshold review in the control process). Suppose state s uses a ksk_s-tolerant threshold, where ks∈0,…,n−1k_s∈\0,…,n-1\. The following are equivalent: (i) for every initial state s0s_0 and every history-adaptive proposer agent, the resulting policy π satisfies JP(π∣s0)≥VP0(s0)J_P(π s_0)≥ V_P^0(s_0); (i) at every state s, the panel is ksk_s-robustly coalitionally aligned relative to sC_s when participant utilities are the vectors Qj0(s,⋅)Q_j^0(s,·). More generally, for any function εharm:→[0,∞) _harm:S→[0,∞), if the baseline Q-vectors have ksk_s-robust lower coalitional alignment on sD_s at tolerance εharm(s) _harm(s), then the quantitative welfare bound in Corollary 4 holds. Proof. Apply Corollary 2 at each state and then Theorem 2. ∎ 4.3 A state-independent certificate from reward functions For monotone authorization rules, Theorem 2 characterizes safety through coverage by the baseline Q-vectors at every state; without monotonicity, coverage remains sufficient. Because these vectors depend on the transition kernel and baseline policy, a separate certificate may be needed at each state. We therefore look for a stronger sufficient condition that can be checked directly on the reward functions, before choosing a baseline policy or computing any continuation values. Definition 4.3 (Reward-function coverage). A coalition S covers the principal in reward-function space if there are coefficients λiS≥0 _i^S≥ 0 such that rP(s,a)=∑i∈SλiSri(s,a)for every (s,a)∈×A.r_P(s,a)= _i∈ S _i^Sr_i(s,a) every (s,a) × A. The coefficients in this definition are fixed across states and actions. Discounted sums and expectations preserve the equality, so the same coefficients also combine the reviewers’ baseline Q-values into the principal’s. This condition is stronger than statewise coverage, but supplies it at every state and for every choice of dynamics. Corollary 6 (Reward-function coverage lifts to every state). Fix an authorization rule W, not necessarily monotone. Suppose every inclusion-minimal winning approval set covers the principal in reward-function space. Then, for every transition kernel, baseline policy, initial state, and history-adaptive proposer agent, the same state-independent rule W keeps the principal’s expected discounted utility at least as high as under the baseline. In particular, the state-independent k-tolerant threshold rule has this guarantee if every (n−k)(n-k)-reviewer coalition covers the principal in reward-function space. Proof. Fix a reward-function certificate for S. Linearity of expectation under the baseline gives VP0(s)=∑i∈SλiSVi0(s)andQP0(s,a)=∑i∈SλiSQi0(s,a)V_P^0(s)= _i∈ S _i^SV_i^0(s) Q_P^0(s,a)= _i∈ S _i^SQ_i^0(s,a) for every state s and action a. Thus S has a zero-slack statewise Q-vector cover. Every winning approval set contains an inclusion-minimal winning subset, and coverage is preserved when reviewers are added. Thus every winning approval set has a zero-slack statewise Q-vector cover, and Theorem 2 gives the claimed baseline-safety guarantee. For the threshold rule, every winning set contains an (n−k)(n-k)-element covering subset, proving the final claim. ∎ The reward identity in Corollary 6 gives a guarantee that holds for every choice of dynamics. For a fixed transition kernel, however, equality of reward functions is stronger than needed: authorization depends on comparisons with the baseline. Discounted potential-based shaping can change the rewards while leaving every baseline advantage and every policy’s return relative to the baseline unchanged (Ng et al., 1999). Appendix B.1 characterizes the resulting relaxation of reward-function coverage. 5 Strategic reviewers and the role of unanimity The preceding guarantees allow reviewers to have different objectives but assume that they report their comparisons sincerely. A reviewer acting to maximize its own return may instead have an incentive to vote insincerely, especially when its vote affects which proposals it will face later. We now ask when authorization remains safe under this strategic behavior. We begin with a single decision, where the reviewer’s incentive depends on whether its vote can change the outcome and on the sign of its gain from execution. We then return to the control process. Sincere voting in one decision. Fix a single feasible proposal with deviation d∈d . Normalize every reviewer’s payoff from rejection to zero, so reviewer i receives ⟨ui,d⟩ u_i,d if the proposal is executed. The reviewers vote simultaneously, and the authorization rule maps their approval set to execution or rejection. Under a monotone rule, sincerity is weakly dominant: a reviewer can change the outcome only toward execution by approving, or toward rejection by disapproving. Proposition 2 (Sincere voting is weakly dominant). Fix a proposal d and a monotone authorization rule. If ⟨ui,d⟩>0 u_i,d >0, approving weakly dominates disapproving for reviewer i; if ⟨ui,d⟩<0 u_i,d <0, disapproving weakly dominates approving. If ⟨ui,d⟩=0 u_i,d =0, both votes are optimal. Consequently, the sincere voting profile is a dominant-strategy equilibrium of the one-decision game. Proof. Fix the other reviewers’ votes. By monotonicity, changing reviewer i’s vote from disapproval to approval either leaves the outcome unchanged or changes it from rejection to execution. The latter change gives reviewer i gain ⟨ui,d⟩ u_i,d . Its sign determines the weakly dominant vote; when the gain is zero, the reviewer is indifferent. ∎ Weak dominance does not eliminate all Nash equilibria: a reviewer may vote insincerely whenever its vote cannot change the outcome. Appendix B.2 shows that the ability to force rejection alone exactly characterizes one-decision Nash safety and records the refinement obtained by excluding weakly dominated votes. To describe who can force rejection, we distinguish individual veto power from the ability of several reviewers to block a proposal together. For a monotone rule, a coalition B⊆NB N is blocking if N∖B∉N B : unanimous disapproval by its members forces rejection, regardless of the votes outside B. Let ℬ()B(W) be the set of inclusion-minimal blocking coalitions. A reviewer has veto power if it can force rejection alone, and the set of veto reviewers is V():=i∈N:N∖i∉.V(W):=\i∈ N:N \i\ \. 5.1 Strategic safety in the control process In the control process, a reviewer’s vote today shapes the proposals it will face tomorrow, so sincere statewise comparison is no longer dominant and we must work with equilibrium behavior directly. Return to the discounted MDP of Section 4. Fix any possibly randomized, history-dependent proposer policy ρ. A review policy σi _i maps the observed history and current proposal to a possibly randomized vote. For a profile σ=(σ1,…,σn)σ=( _1,…, _n), let Jj(ρ,σ∣s0):=ρ,σ[∑t=0∞γtrj(st,at)|s0]J_j(ρ,σ s_0):=E_ρ,σ [ _t=0^∞γ^tr_j(s_t,a_t)\, |\,s_0 ] be participant j’s expected return along the trajectory actually induced by the proposer, the votes, and the fallback to the baseline after rejection. Reviewer i chooses σi _i to maximize JiJ_i. Thus the strategic objective is the realized continuation payoff, not the baseline-Q comparison that defines a sincere vote. Under unanimity, a reviewer can secure its baseline return by always disapproving. A threshold that tolerates k disapprovals requires k+1k+1 reviewers to act together to force the baseline. We therefore consider stability against joint deviations, as well as deviations by one reviewer. Definition 5.1 (Coalitional stability). Fix a proposer policy ρ, an initial state s0s_0, an integer c≥1c≥ 1 and reviewer-specific tolerances =(η1,…,ηn)∈ℝ+n η=( _1,…, _n) _+^n. A review-policy profile is (c,)(c, η)-coalitionally stable if no coalition B with 1≤|B|≤c1≤|B|≤ c has a joint deviation in its review policies that raises each member i’s return by more than ηi _i, holding the proposer and all other review policies fixed. We abbreviate (c,)(c, 0)-coalitional stability as c-coalitional stability. 1-coalitional stability is ordinary Nash equilibrium in the reviewer game; n-coalitional stability is strong Nash equilibrium (Aumann, 1959). Although related in spirit to the cooperative-game core (Gillies, 1959), c-coalitional stability is distinct: the core asks whether any coalition can secure an alternative outcome that all its members prefer, whereas our notion asks whether a coalition of at most c reviewers can all benefit by jointly changing their voting policies within the fixed authorization game. Our definition is weaker than the c-resilient equilibrium used in distributed game theory, which rules out a coalition deviation that benefits even one deviating member (Abraham et al., 2006). If the proposer agent is itself strategic, the reviewer component of any Nash equilibrium of the full game is 1-coalitionally stable conditional on the proposer’s equilibrium strategy. To turn coalitional stability into a guarantee for the principal, we need to relate the participants’ returns along the trajectory they actually induce. We use the stronger reward-function coverage condition from Definition 4.3. Each coalition’s covering coefficients are fixed across states and actions, so the same identity relates its members’ returns to the principal’s, including their gains over the baseline. This lets us infer from a principal loss that sufficiently many reviewers would also benefit from forcing a return to the baseline. Theorem 3 (Coalitional stability and strategic safety). Fix a monotone authorization rule W. Assume every minimally winning coalition W∈ℳ()W (W) covers the principal in reward-function space: there are coefficients λiW≥0 _i^W≥ 0 such that rP=∑i∈WλiWri.r_P= _i∈ W _i^Wr_i. Fix any proposer policy ρ and initial state s0s_0. Suppose that, at the review-policy profile σ, no minimally blocking coalition B∈ℬ()B (W) can strictly improve every member’s return by jointly switching to always disapproving, holding the proposer and all other review policies fixed. Then JP(ρ,σ∣s0)≥VP0(s0).J_P(ρ,σ s_0)≥ V_P^0(s_0). In particular, if ℬ()≠∅B(W)≠ , let c:=maxB∈ℬ()|B|c:= _B (W)|B|. Every c-coalitionally stable profile is baseline-safe. If ℬ()=∅B(W)= , every profile is baseline-safe. Under the k-tolerant threshold, c=k+1c=k+1 and it is enough that every (n−k)(n-k)-reviewer coalition cover the principal in reward-function space. Proof. Write Δj:=Jj(ρ,σ∣s0)−Vj0(s0) _j:=J_j(ρ,σ s_0)-V_j^0(s_0). Because every participant’s return is evaluated along the same realized trajectory, reward-function coverage and linearity give, for every minimally winning coalition W, ΔP=∑i∈WλiWΔi. _P= _i∈ W _i^W _i. Suppose toward a contradiction that ΔP<0 _P<0, and let S+:=i:Δi≥0S_+:=\i: _i≥ 0\. This set cannot be winning: otherwise it contains some W∈ℳ()W (W), and applying (∗)(*) to that W would give ΔP≥0 _P≥ 0. Hence S+S_+ is losing, so its complement is blocking and contains some B∈ℬ()B (W). Every i∈Bi∈ B has Δi<0 _i<0. If the members of B jointly always disapprove, they force every proposal to be rejected and hence force the baseline policy, strictly improving every member’s return. This contradicts the assumed stability. Therefore ΔP≥0 _P≥ 0. ∎ Reward-function coverage ensures that if every member of a winning coalition earns at least its baseline return, then the principal does too. If the principal instead loses, the reviewers who are below baseline must therefore contain a blocking coalition. Its members can all improve by always disapproving and thereby forcing the baseline policy. Coalitional stability rules out this profitable joint deviation, and hence rules out a loss for the principal. Remark 5.1 (Approximate coalitional stability). In the setting of Theorem 3, fix covering coefficients λiW _i^W and let c≥1c≥ 1 be at least the size of every minimally blocking coalition. Every (c,)(c, η)-coalitionally stable profile satisfies JP(ρ,σ∣s0)−VP0(s0)≥−max∑i∈W∈ℳ()λiWηi.J_P(ρ,σ s_0)-V_P^0(s_0)≥- _W (W) _i∈ W _i^W _i. With Δj _j as in the proof, set S+:=i:Δi≥−ηiS_+:=\i: _i≥- _i\. If S+S_+ is winning, the return identity for a minimally winning coalition it contains gives the bound. Otherwise, its complement contains a minimally blocking coalition whose members each gain more than ηi _i by jointly forcing the baseline, contradicting stability. Under unanimity, the joint deviation required by the theorem reduces to a deviation by one reviewer, since every reviewer can force the baseline on its own. Every reviewer therefore earns at least its baseline return at a Nash equilibrium. Full-panel reward-function coverage transfers these inequalities to the principal through a nonnegative combination. This gives a safety guarantee even in a game where the proposer agent also chooses its policy strategically. Corollary 7 (Nash safety in the full proposer–reviewer game). Suppose the full panel covers the principal in reward-function space. Under unanimity, every Nash equilibrium of a game in which the proposer agent and reviewers all choose history-dependent policies gives the principal expected return at least VP0(s0)V_P^0(s_0), regardless of the proposer’s objective. Proof. Fix a Nash equilibrium and hold the proposer’s equilibrium policy fixed. The reviewers’ equilibrium policies form a Nash equilibrium in the resulting reviewer game. Under unanimity, any reviewer can force the baseline by always disapproving, so the profile satisfies the deviation condition in Theorem 3. Apply that theorem. ∎ This Nash-safety guarantee depends on unanimity. If the rule tolerates even one disapproval and everyone approves, a single reviewer cannot prevent execution by changing its vote. Suppose also that the proposer responds to past votes only through whether a proposal was executed or rejected. A unilateral change in voting then leaves future proposals unchanged as well. Consequently, always approving can remain a Nash equilibrium even when the resulting trajectory harms the principal and every reviewer would prefer the baseline. Proposition 3 (Unsafe Nash equilibria under permissive thresholds). Fix k∈1,…,n−1k∈\1,…,n-1\ and a proposer policy ρ that depends on past votes only through the resulting execute/reject decisions. Let σallσ^all be the profile in which every reviewer always approves. If executing every proposal made by ρ yields JP(ρ,σall∣s0)<VP0(s0),J_P(ρ,σ^all s_0)<V_P^0(s_0), then σallσ^all is an unsafe Nash equilibrium of the reviewer game under the k-tolerant threshold. Proof. A unilateral deviation can create at most one disapproval, which a k-tolerant threshold with k≥1k≥ 1 ignores. Couple the deviating and all-approve plays using the same transition and proposer randomness. An induction over review times shows that each execute/reject decision is the same; the assumed invariance of ρ then gives the same next proposal, so the entire state–action trajectory is the same. The deviator’s payoff is therefore unchanged and the all-approve profile is a Nash equilibrium. The displayed inequality makes it unsafe. ∎ The failure can occur even when all reviewers are aligned with the principal: they may all prefer the baseline while no one can force it alone. Under a k-tolerant threshold, k+1k+1 reviewers can do so jointly. The coalitional-stability requirement in Theorem 3 rules out such an outcome, but requires more than ordinary Nash equilibrium. The guarantee for a permissive threshold therefore combines the stated coverage condition with a stronger behavioral assumption about joint deviations by blocking coalitions. 6 Completeness and the limits of binary approval Soundness protects the principal against harmful proposals, but places no requirement on proposals that would benefit them. A panel can therefore be sound while rejecting many desirable actions. We return to sincere review in the one-decision model and ask whether an authorization rule can also execute every proposal the principal finds acceptable. This complementary requirement is completeness; together, soundness and completeness ask the rule to recover all of the decisions that the principal would themselves make from only the reviewers’ reports. Definition 6.1 (Completeness and approximate implementation). A rule is complete on C if it executes every d∈d with ⟨uP,d⟩≥0 u_P,d ≥ 0. For εgood≥0 _good≥ 0, it is εgood _good-complete on D if it executes every d∈d with ⟨uP,d⟩≥εgood u_P,d ≥ _good. Together with εharm _harm-soundness, this gives an (εharm,εgood)( _harm, _good)-implementation. Exact soundness and completeness are the zero-error case. 6.1 The limits of binary approval The complementary-reviewer example shows what can be lost when only binary votes are reported. Let u1=e1u_1=e_1, u2=e2u_2=e_2, and uP=e1+e2u_P=e_1+e_2 on =ℝ2C=R^2, as in Figure 1. The deviations (2,−1)(2,-1) and (1,−2)(1,-2) both receive approval from reviewer 1 and disapproval from reviewer 2, but their principal gains are 11 and −1-1, respectively. Any rule based on the approval set must make the same decision on both, so it either rejects the beneficial proposal or executes the harmful one. The following theorem generalizes this obstruction by examining where reviewers’ votes can change as feasible proposals cross the principal’s indifference boundary. Theorem 4 (Binary separation and exact implementation). Let L:=span()L:=span(C), and suppose there are deviations d+,d−∈d_+,d_- such that ⟨uP,d+⟩>0and⟨uP,d−⟩<0. u_P,d_+ >0 u_P,d_- <0. Let f:2N→0,1f:2^N→\0,1\ be any deterministic rule based only on the approval set, not necessarily monotone. Suppose ⟨uP,d⟩>0⟹f(S(d))=1,⟨uP,d⟩<0⟹f(S(d))=0(d∈). u_P,d >0 f(S(d))=1, u_P,d <0 f(S(d))=0 (d ). Then: (i) some reviewer i has the same indifference hyperplane as the principal on feasible proposals: for some c≠0c≠ 0, ⟨ui,d⟩=c⟨uP,d⟩for every d∈L; u_i,d =c\, u_P,d every d∈ L; (i) if f is monotone, then some such reviewer has c>0c>0 and is aligned with the principal; (i) the same positive-alignment conclusion holds, without monotonicity, if f(S(d))=1f(S(d))=1 whenever ⟨uP,d⟩=0 u_P,d =0. Thus exact soundness and completeness require an aligned reviewer even for a nonmonotone rule. The weaker task of separating only strict gains from strict losses still requires a reviewer with the correct boundary, but a nonmonotone rule could invert the vote of a negatively proportional reviewer. Monotonicity or the convention that principal indifference is accepted rules out that inversion. Appendix C gives the proof. Corollary 8 (Exact binary implementation). Under the hypotheses of Theorem 4, a deterministic approval-set rule satisfying f(S(d))=1⟺⟨uP,d⟩≥0(d∈)f(S(d))=1 u_P,d ≥ 0 (d ) exists if and only if some reviewer is aligned with the principal on L. When such a reviewer exists, its vote alone gives a monotone exact rule. Proof. Necessity is part (i) of Theorem 4. For sufficiency, execute exactly when the aligned reviewer approves; positive proportionality makes its weak comparison identical to the principal’s on every feasible deviation. ∎ 6.2 Approximate binary review from diverse panels The impossibility above requires exact decisions even for proposals arbitrarily close to the principal’s indifference boundary. Allowing errors near that boundary leaves room for a panel to become increasingly accurate without any reviewer becoming individually aligned. A useful example is a panel whose utility vectors lie on a ring at a fixed angle from the principal’s vector (Figure 4). Over the full ring, a beneficial proposal receives a majority of approvals and a harmful one a majority of disapprovals. A finite panel approximates these proportions, either by sampling reviewers from the ring or by spacing their objectives evenly around it. The next result shows how both constructions approach the principal’s decisions as the panel grows. Proposition 4 (Random and deterministic convergence without individual alignment). Let =d∈ℝ3:‖d‖1≤1D=\d ^3:\|d\|_1≤ 1\, let uP=e3u_P=e_3, and fix θ∈(0,π/2)θ∈(0,π/2). Draw an odd number n=2m+1n=2m+1 of reviewer objectives independently and uniformly from the vectors that form angle θ with uPu_P on the ring shown in Figure 4. Under the m-tolerant majority rule, there are εn=Oθ(logn/n) _n=O_θ( n/n) such that, with probability tending to one, the rule is (εn,εn)( _n, _n)-implementing. If n reviewer objectives are instead spaced evenly around the same ring, the conclusion holds deterministically with εn=Θ(1/n) _n= (1/n). In both cases every reviewer remains at angle θ from the principal. The geometry explains both the approximation error and the difference between the two rates. For a nonzero deviation with zero principal gain, the approving directions form a semicircle. A small negative gain shortens that arc, but it may still contain a majority of a finite panel, as in Figure 4(b). Equal spacing controls how much shorter than a semicircle such an arc can be; random panels must also account for sampling error. In both cases, the possible mistakes approach the principal’s indifference boundary while every reviewer remains at the same angle from the principal. Appendix C.1 gives the proofs and the general population-margin condition. (a) Reviewers remain misaligneduPu_Puju_jθ reviewer remains at angle θ.(b) Evenly spaced reviewersπ/(2n)π/(2n)uPu_P axis(toward viewer)m+1m+1 approvedisapproveBalanced panel: εn=Θ(1/n) _n= (1/n). Figure 4: Both views show eleven evenly spaced reviewers on a ring at fixed angle θ from the principal. (a) No reviewer approaches individual alignment as the panel grows. (b) The narrowest arc containing m+1m+1 reviewers leaves a gap of π/(2n)π/(2n) at each end of the enclosing semicircle, yielding deterministic implementation error Θ(1/n) (1/n). 6.3 Numerical scores instead of binary votes Approximation retains binary votes and permits small errors. Another way around the impossibility is to retain exactness and ask reviewers for more information. A binary vote reports only the sign of a reviewer’s gain; a numerical score also reports its magnitude, allowing gains and losses across reviewers to be combined. Suppose each reviewer reports its gain ⟨ui,d⟩ u_i,d . Fix nonnegative weights ω1,…,ωn _1,…, _n, not all zero, and define uω:=∑i=1nωiui,Gω(d):=∑i=1nωi⟨ui,d⟩=⟨uω,d⟩.u_ω:= _i=1^n _iu_i, G_ω(d):= _i=1^n _i u_i,d = u_ω,d . The weighted-score rule executes exactly when Gω(d)≥0G_ω(d)≥ 0. The weighted-score rule therefore behaves like a single reviewer with utility vector uωu_ω. On a linear domain, soundness and completeness require this aggregate to have the same acceptable halfspace as the principal. Because both d and −d-d are feasible, agreement on all decisions forces their utility gains to be positively proportional. The corollary characterizes this requirement; it can hold even when no individual reviewer’s utility is proportional to the principal’s. Corollary 9 (Exact weighted scores on a linear domain). Suppose =LC=L is a linear subspace. The fixed weighted-score rule is sound and complete if and only if there is a constant c>0c>0 such that ⟨uω,d⟩=c⟨uP,d⟩for every d∈L. u_ω,d =c\, u_P,d every d∈ L. Proof. If the assumed proportionality holds, the rule makes exactly the same comparison as the principal. Conversely, suppose the rule is sound and complete. Then ⟨uω,d⟩ u_ω,d and ⟨uP,d⟩ u_P,d have the same weak sign for every d∈Ld∈ L. Their kernels therefore coincide, so the two linear functionals are proportional on L; equality of their signs makes the proportionality constant positive. If the principal is indifferent over L, soundness and completeness force the aggregate to be indifferent as well, and the claim holds for every c>0c>0. ∎ Return to the two complementary reviewers with u1=e1u_1=e_1, u2=e2u_2=e_2, and uP=e1+e2u_P=e_1+e_2. Their binary votes could not distinguish the beneficial deviation (2,−1)(2,-1) from the harmful deviation (1,−2)(1,-2). Summing their numerical gains does distinguish them: with ω1=ω2=1 _1= _2=1, Gω(d)=⟨uP,d⟩,G_ω(d)= u_P,d , so the two deviations receive aggregate scores 11 and −1-1. More generally, the rule is sound and complete on every feasible deviation, although neither reviewer is individually aligned. The two forms of review use the coverage relation differently. A threshold rule only counts approvals: coalitional coverage can certify its soundness without the rule knowing the coefficients for any winning coalition. A weighted-score rule uses its coefficients in the authorization decision itself. To obtain exact soundness and completeness, the designer must therefore choose weights whose aggregate agrees with the principal on every feasible comparison. Numerical reports make that exact comparison possible in cases where the binary votes contain too little information. 7 Experiments with LLM reviewer panels We test whether reviewer panels exhibit coalitional coverage without individual alignment and how their authorization rules trade off soundness and completeness 11 1 Code is available here: https://github.com/psamathe50/safe-delegated-approval-experiments. We use the one-decision model of Definition 2.1. 7.1 Experimental setup RewardBench 2 (Malik et al., 2026) contains 1,763 standard prompts, each with four answers, one labeled correct, and scores from 48 reward models. We treat each reward model as a reviewer and its scores as cardinal utilities, extended linearly to lotteries over the four answers. The principal’s utility is one for the correct answer and zero for the others; the baseline samples uniformly from the four answers. A reviewer approves an answer whose score is at least its mean score, and ties count as approvals. We evaluate both deterministic answers and lotteries over them. Reviewers compare a lottery’s expected utility with the baseline and vote before sampling; rejection executes the uniform baseline. StrongREJECT (Souly et al., 2024) pairs human harm annotations of model responses with automated evaluator scores. For each response, the proposal is to release it and the baseline is abstention. We retain 1,084 responses with all five human labels and nine evaluator scores. We model release as having principal gain 1/2−h1/2-h over abstention, where h is its median human harm score. An evaluator approves release when its harm score is at most 1/21/2. Applying the same cutoff to the human median gives 190 harmful and 894 acceptable responses. We also allow lotteries between release and abstention. Appendix F.1 gives the full benchmark mappings and worked examples. Covered after reviewer removals Minimum reviewer angle Prompts None Any one Any five No restriction 1,763 1,590 (90.2%) 1,361 (77.2%) 669 (37.9%) >5∘>5 1,115 953 (85.5%) 762 (68.3%) 291 (26.1%) >10∘>10 704 575 (81.7%) 423 (60.1%) 125 (17.8%) >15∘>15 489 392 (80.2%) 263 (53.8%) 74 (15.1%) >20∘>20 299 228 (76.3%) 144 (48.2%) 36 (12.0%) Table 1: Coalitional coverage of the 48-reviewer RewardBench 2 panel. Angle restrictions define nested prompt groups by the closest reviewer direction to the principal, using nonzero centered vectors. Coverage entries give counts and percentages within each row. Removal columns require coverage after every possible deletion of the stated number of reviewers. Certificates use benchmark labels and concern all lotteries over the four answers. 7.2 Coverage without individual alignment On RewardBench, we center each utility vector by subtracting its mean over the four answers. We then check whether the principal’s vector is a nonnegative combination of the reviewers’ vectors. By the coverage characterization, this certifies soundness over every lottery under unanimous approval. Table 1 shows that coverage holds on most prompts even when every reviewer is individually misaligned (here misalignment is quantified as the angle of the reviewer utility vector from the principal utility vector). On 37.9% of prompts the panel is 55-robustly coalitionally aligned: coverage survives any five reviewer removals, so the 55-tolerant threshold rule remains sound. These certificates use benchmark labels; Appendix F.2 gives the optimization. Figure 5: Threshold-rule soundness and completeness as the number k of tolerated objections increases. On RewardBench 2, the tradeoff is substantially better for individual answers than for guarantees over every lottery; on StrongREJECT the two evaluations coincide. StrongREJECT soundness and completeness are the fractions of harmful responses withheld and acceptable responses released, respectively. RewardBench rates are fractions of prompts. 7.3 Authorization with binary votes We vary k in the k-tolerant threshold rule to measure the tradeoff between soundness and completeness (Figure 5). This rule authorizes a proposal with at most k disapprovals. On RewardBench, soundness counts prompts on which all three incorrect answers are rejected, and completeness counts prompts on which the correct answer is authorized. For lotteries, the requirements extend to every harmful or non-harmful mixture, respectively. On StrongREJECT, we measure the fraction of harmful responses withheld and the fraction of acceptable responses released; since it is a binary decision setting, deterministic and lottery guarantees coincide. On StrongREJECT, tolerating one disapproval increases completeness without reducing soundness: every harmful response receives at least two disapprovals. Each disapproving reviewer is individually aligned on that response’s one-dimensional proposal domain, so several reviewers can reject the same harmful proposal. RewardBench instead exhibits coverage without an aligned individual (Table 1). On RewardBench, the soundness/completeness tradeoff is substantially better for deterministic proposals than for arbitrary lotteries: at k=9k=9, both rates are about 70%70\% (Figure 5). Both guarantees hold on 904/1,763=51.3%904/1,763=51.3\% of prompts. Restricting review to deterministic proposals is natural and is satisfied by e.g. proposer agents that submit a single answer for approval, even if they use randomness to generate it. Requiring soundness and completeness over every lottery is more demanding: no evaluated threshold has positive exact soundness and completeness rates simultaneously. We therefore examine approximate guarantees in Appendix F.2 and numerical scores below. Figure 6: Numerical-score rules achieve higher completeness than binary-vote rules at comparable high test soundness, especially over lotteries. Curves sweep thresholds for fixed rules on 352 RewardBench 2 test prompts: (a) individual answers; (b) all lotteries, with 0.50.5-completeness requiring approval of every lottery with at least a 75% chance of a correct answer. In (b), the only evaluated binary-rule operating point in the displayed soundness range is reject-all. Comparisons at a given soundness level select thresholds using test outcomes for descriptive purposes only; Appendix F.4 reports more results. 7.4 Authorization with numerical scores On RewardBench, we next compare the k-tolerant threshold rule with weighted voting, which assigns learned nonnegative weights to approvals, and a cardinal committee, which takes a nonnegative weighted sum of numerical gains, as in Section 6.3. Both authorize when their weighted total meets or exceeds a threshold. We also compare with a rule that thresholds a single reviewer’s numerical gain. We center and normalize numerical scores within each reviewer and prompt. We fit reviewer weights on training data, using an ℓ2 _2 penalty to discourage large weights. We choose the penalty strength and the individual reviewer by their correct-answer selection accuracy on validation data. The score thresholds come from validation scores of deterministic answers; we use the same thresholds for both deterministic-answer and lottery evaluations on 352 test prompts. Figure 6 traces their observed tradeoffs. Each reported comparison uses the most complete evaluated threshold meeting the stated soundness requirement on the test set. For lotteries, we retain exact soundness and measure 0.50.5-completeness: approval of every mixture with gain at least 0.50.5, or equivalently at least a 75%75\% chance of returning the correct answer. Appendix F.3 gives the fitting details and controls; Appendix F.4 evaluates thresholds fixed using validation data alone. Both numerical-score rules authorize more correct answers at high soundness than either binary rule. In panel (a), the cardinal committee also improves on the single reviewer: at a 99% test soundness requirement, it authorizes the correct answer on 60.5% of prompts, compared with 46.6% for the individual. Their curves are closer near 95% soundness. With thresholds selected using validation data alone, the committee achieves 98.6% test soundness and 63.4% completeness, compared with 96.6% and 62.5% for the individual (Appendix F.4). The committee’s advantage extends to lotteries in panel (b): it is 0.50.5-complete on more prompts than the individual. Appendix F.3 varies the minimum principal gain for guaranteed approval. The binary rules’ limitation in panel (b) follows from coverage. The full panel covers the principal on 315 of the 352 test prompts (89.5%). Every other prompt admits a harmful lottery that every reviewer approves (Theorem 1). Thus any rule accepting unanimous approval has exact lottery soundness on at most 89.5% of prompts. For both global binary-rule families evaluated here, every threshold other than reject-all accepts unanimous approval, regardless of how its weights are chosen. Neither can meet a 90% soundness requirement with positive completeness. Numerical-score rules avoid this ceiling: a positive aggregate gain threshold can exclude unanimously approved proposals. Such a threshold also rejects the zero-gain baseline, so it cannot be exactly complete. This explains the use of a positive completeness margin in panel (b). 8 Discussion Delegating the review of a potentially misaligned AI agent to other AI agents would seem at first blush to simply push the alignment problem one level deeper. But in this paper we have identified a way in which it can work—and have characterized exactly when it does. Even in dynamic, stateful settings, unanimous sincere review is safe whenever the panel covers the principal’s baseline Q-vector at every state, which is a substantially weaker condition than individual alignment. This coalitional alignment condition is not guaranteed to be satisfied, and may be difficult to audit, but is unavoidable, as our characterization is bi-directional. Moreover, for the conservative unanimous-approval rule, coverage in reward-function space extends the guarantee to strategic, long-running reviewing agents, in every Nash equilibrium of the game induced by the approval process. There is a limit to how much we can ask for from such delegated review: as we show, if we require not just perfect soundness, but also perfect completeness, then this can be accomplished only if the reviewing panel contains a perfectly aligned agent. But we have seen that this impossibility is brittle: it can be circumvented either by relaxing our goal to approximate soundness and completeness, or by eliciting richer information from reviewers. Our experiments give an initial example of coalitional alignment in existing reviewer panels: reward models can collectively cover a principal even when no individual reviewer is aligned. Our experiments also confirm that eliciting numerical scores can support substantially better soundness/completeness tradeoffs than binary votes. Finding panels that retain coverage on new proposals and across changing states remains an important empirical problem, as does learning useful approval rules whose performance can be established before deployment. Acknowledgments This work was supported in part by the Simons Collaboration on Algorithmic Fairness, the NSF EnCORE Tripods institute, an NSF graduate research fellowship, NSF CAREER #2543725, and a Schmidt Sciences AI2050 Fellowship. AI tools including GPT 5.6 Sol and GPT 6 were used in the development of this paper; the authors are (of course) responsible for all of its content. References Abraham et al. (2006) I. Abraham, D. Dolev, R. Gonen, and J. Y. Halpern Distributed computing meets game theory: robust mechanisms for rational secret sharing and multiparty computation. In Proceedings of the Twenty-Fifth Annual ACM Symposium on Principles of Distributed Computing, p. 53–62. External Links: Document, Link Cited by: §5.1. Akinwande et al. (2026) V. Akinwande, J. Z. Kolter, and A. Nayebi Sharding prevents LLM oversight failures and adversarial exploitation. External Links: 2608.06422, Document, Link Cited by: §1.3. Alshiekh et al. (2018) M. Alshiekh, R. Bloem, R. Ehlers, B. Könighofer, S. Niekum, and U. Topcu Safe reinforcement learning via shielding. Proceedings of the AAAI Conference on Artificial Intelligence 32 (1), p. 2669–2678. External Links: Document, Link Cited by: §1.3. Aumann (1959) R. J. Aumann Acceptable points in general cooperative n-person games. In Contributions to the Theory of Games IV, R. D. Luce and A. W. Tucker (Eds.), Annals of Mathematics Studies, Vol. 40, p. 287–324. Cited by: §5.1. Bennett et al. (2023) A. Bennett, D. Misra, and N. Kallus Provable safe reinforcement learning with binary feedback. In Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, F. Ruiz, J. Dy, and J. van de Meent (Eds.), Proceedings of Machine Learning Research, Vol. 206, p. 10871–10900. External Links: Link Cited by: §1.3. Bremner and Chen (2009) D. Bremner and D. Chen A branch and cut algorithm for the halfspace depth problem. Note: arXiv:0910.1923 [cs.CG] External Links: Document, Link Cited by: §E.2. Chen et al. (2025) Z. Chen, M. Kang, and B. Li ShieldAgent: shielding agents via verifiable safety policy reasoning. In Proceedings of the 42nd International Conference on Machine Learning, A. Singh, M. Fazel, D. Hsu, S. Lacoste-Julien, F. Berkenkamp, T. Maharaj, K. Wagstaff, and J. Zhu (Eds.), Proceedings of Machine Learning Research, Vol. 267, p. 8313–8344. External Links: Link Cited by: §1.3. Collina et al. (2026a) N. Collina, S. Goel, A. Roth, E. Ryu, and M. Shi Emergent alignment via competition. In Proceedings of the 43rd International Conference on Machine Learning, External Links: Link Cited by: §1.3. Collina et al. (2026b) N. Collina, S. Goel, A. Roth, and M. Shi Personalization aids pluralistic alignment under competition. External Links: 2602.13451, Document, Link Cited by: §1.3. Dyckerhoff and Nagy (2025) R. Dyckerhoff and S. Nagy Exact computation of angular halfspace depth. Statistics and Computing 35, p. 173. External Links: Document, Link Cited by: §E.2, §E.2. Freixas et al. (2011) J. Freixas, X. Molinero, M. Olsen, and M. Serna On the complexity of problems on simple games. RAIRO. Operations Research 45 (4), p. 295–314. External Links: Document, Link Cited by: §1.3. Gillies (1959) D. B. Gillies Solutions to general non-zero-sum games. In Contributions to the Theory of Games IV, R. D. Luce and A. W. Tucker (Eds.), Annals of Mathematics Studies, Vol. 40, p. 47–86. External Links: Document, Link Cited by: §5.1. Greenblatt et al. (2024) R. Greenblatt, B. Shlegeris, K. Sachan, and F. Roger AI control: improving safety despite intentional subversion. In Proceedings of the 41st International Conference on Machine Learning, R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp (Eds.), Proceedings of Machine Learning Research, Vol. 235, p. 16295–16336. External Links: Link Cited by: §1.3. Harsanyi (1955) J. C. Harsanyi Cardinal welfare, individualistic ethics, and interpersonal comparisons of utility. Journal of Political Economy 63 (4), p. 309–321. External Links: Document, Link Cited by: §1.3. Kakade and Langford (2002) S. M. Kakade and J. Langford Approximately optimal approximate reinforcement learning. In Proceedings of the Nineteenth International Conference on Machine Learning, p. 267–274. External Links: Link Cited by: §4.2. Laroche et al. (2019) R. Laroche, P. Trichelair, and R. Tachet des Combes Safe policy improvement with baseline bootstrapping. In Proceedings of the 36th International Conference on Machine Learning, K. Chaudhuri and R. Salakhutdinov (Eds.), Proceedings of Machine Learning Research, Vol. 97, p. 3652–3661. External Links: Link Cited by: §1.3. Malik et al. (2026) S. Malik, V. Pyatkin, S. Land, J. Morrison, N. A. Smith, H. Hajishirzi, and N. Lambert RewardBench 2: advancing reward model evaluation. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: §F.1, §7.1. Nagy and Laketa (2025) S. Nagy and P. Laketa Theoretical properties of angular halfspace depth. Bernoulli 31 (2), p. 1007–1031. External Links: Document, Link Cited by: §E.2. Ng et al. (1999) A. Y. Ng, D. Harada, and S. Russell Policy invariance under reward transformations: theory and application to reward shaping. In Proceedings of the Sixteenth International Conference on Machine Learning, p. 278–287. External Links: Link Cited by: §4.3. Nitzan and Paroush (1983) S. Nitzan and J. Paroush Small panels of experts in dichotomous choice situations. Decision Sciences 14 (3), p. 314–325. External Links: Document, Link Cited by: §1.3. OpenAI (2026) OpenAI Running Codex safely at OpenAI. Note: Published May 8, 2026 External Links: Link Cited by: §1. Pirotta et al. (2013) M. Pirotta, M. Restelli, A. Pecorino, and D. Calandriello Safe policy iteration. In Proceedings of the 30th International Conference on Machine Learning, S. Dasgupta and D. McAllester (Eds.), Proceedings of Machine Learning Research, Vol. 28, p. 307–315. External Links: Link Cited by: §1.3. Seto et al. (1998) D. Seto, B. H. Krogh, L. Sha, and A. Chutinan Dynamic control system upgrade using the Simplex architecture. IEEE Control Systems 18 (4), p. 72–80. External Links: Document, Link Cited by: §1.3. Souly et al. (2024) A. Souly, Q. Lu, D. Bowen, T. Trinh, E. Hsieh, S. Pandey, P. Abbeel, J. Svegliato, S. Emmons, O. Watkins, and S. Toyer A StrongREJECT for empty jailbreaks. In Advances in Neural Information Processing Systems, Vol. 37. External Links: Link Cited by: §F.1, §7.1. Thomas et al. (2015) P. Thomas, G. Theocharous, and M. Ghavamzadeh High confidence policy improvement. In Proceedings of the 32nd International Conference on Machine Learning, F. Bach and D. Blei (Eds.), Proceedings of Machine Learning Research, Vol. 37, p. 2380–2388. External Links: Link Cited by: §1.3. Wagener et al. (2021) N. C. Wagener, B. Boots, and C. Cheng Safe reinforcement learning using advantage-based intervention. In Proceedings of the 38th International Conference on Machine Learning, M. Meila and T. Zhang (Eds.), Proceedings of Machine Learning Research, Vol. 139, p. 10630–10640. External Links: Link Cited by: §1.3. Xiang et al. (2025) Z. Xiang, L. Zheng, Y. Li, J. Hong, Q. Li, H. Xie, J. Zhang, Z. Xiong, C. Xie, C. Yang, D. Song, and B. Li GuardAgent: safeguard LLM agents via knowledge-enabled reasoning. In Proceedings of the 42nd International Conference on Machine Learning, A. Singh, M. Fazel, D. Hsu, S. Lacoste-Julien, F. Berkenkamp, T. Maharaj, K. Wagstaff, and J. Zhu (Eds.), Proceedings of Machine Learning Research, Vol. 267, p. 68316–68342. External Links: Link Cited by: §1.3. Appendix A Additional results for static soundness If a reviewer population opposes every harmful direction with probability bounded uniformly away from zero, then sufficiently large panels are robust with high probability, as a function of the feasible dimension and confidence level. Proposition 5 (Random panels from a diverse reviewer population). Suppose the principal has a harmful feasible direction. Let reviewer utility vectors u1,…,unu_1,…,u_n be drawn independently from a distribution ν, let r:=dimspan()r:= (C), set v:=minr,nv:= \r,n\, and define α:=infd∈⟨uP,d⟩<0Pru∼ν[⟨u,d⟩<0].α:= _ subarraycd \\ u_P,d <0 subarray _u ν[ u,d <0]. There is a universal constant c>0c>0 such that, for every δ∈(0,1)δ∈(0,1), with probability at least 1−δ1-δ, 1ndepth(uP,u1,…,un)≥α−cvlog(2en/v)+log(2/δ)n. 1ndepth_C(u_P;u_1,…,u_n)≥α-c v (2en/v)+ (2/δ)n. Consequently, whenever the right side is positive, the sampled panel is k-robustly coalitionally aligned for every integer k strictly smaller than n times that quantity. Proof. Project reviewer utilities onto L:=span()L:=span(C). As d varies, the events u:⟨u,d⟩<0\u: u,d <0\ form a subclass of the homogeneous halfspaces in the r-dimensional space L, whose VC dimension is at most r. The standard symmetrization, Sauer–Shelah, and Hoeffding argument gives, simultaneously for all such halfspaces, |1n∑i=1n⟨ui,d⟩<0−Pru∼ν[⟨u,d⟩<0]|≤cvlog(2en/v)+log(2/δ)n. | 1n _i=1^n1\ u_i,d <0\- _u ν[ u,d <0] |≤ c v (2en/v)+ (2/δ)n. Writing p^n(d) p_n(d) and p(d)p(d) for the empirical and population probabilities in the display, respectively, infd:⟨uP,d⟩<0p^n(d)≥infd:⟨uP,d⟩<0p(d)−supd|p^n(d)−p(d)|. _d:\, u_P,d <0 p_n(d)≥ _d:\, u_P,d <0p(d)- _d| p_n(d)-p(d)|. The first term on the right is α, which proves the depth bound. If k is strictly smaller than n times its right side, the integer-valued depth is at least k+1k+1; Corollary 2 then gives k-robust alignment. ∎ Proof and finite-sample form of Corollary 3. If the principal has no harmful feasible direction, the claim follows from the convention that rejection depth is n+1n+1. Otherwise, fix a principal-harmful direction d and write a:=−⟨uP,d⟩>0a:=- u_P,d >0. A draw disapproves exactly when ⟨ξ,d⟩<a ξ,d <a. Central symmetry gives Pr[⟨ξ,d⟩<a]≥1/2 [ ξ,d <a]≥ 1/2, so Proposition 5 yields the following finite-sample statement. Let r:=dimspan()r:= (C) and v:=minr,nv:= \r,n\. There is a universal constant c>0c>0 such that, for every δ∈(0,1)δ∈(0,1), with probability at least 1−δ1-δ, 1ndepth(uP,u1,…,un)≥12−cvlog(2en/v)+log(2/δ)n. 1ndepth_C(u_P;u_1,…,u_n)≥ 12-c v (2en/v)+ (2/δ)n. For fixed r, taking for example δ=n−2δ=n^-2 gives the qualitative claim in the main text. ∎ Exact coverage rules out all principal losses from unanimous approval. To allow a small loss, we ask how negative the principal’s gain can be among the proposals a coalition approves. This is a linear program on D; its dual supplies a coverage representation whose residual bounds the loss. Lemma 1 (Approximate dual certificate). Let E be a finite-dimensional Euclidean space, let ⊆ED E be a polytope containing 00, let v1,…,vr∈Ev_1,…,v_r∈ E, and let η≥0η≥ 0. Then ⟨u,d⟩≥−ηfor every d∈ satisfying ⟨vj,d⟩≥0(j=1,…,r) u,d ≥-η every d satisfying v_j,d ≥ 0\ (j=1,…,r) ⟺u∈conev1,…,vr+ℛη−(). u \v_1,…,v_r\+R^-_η(D). For r=0r=0, the cone is 0\0\ and there are no constraints. Proof. Let p∗:=min⟨u,d⟩:d∈,⟨vj,d⟩≥0(j=1,…,r).p^*:= \ u,d :d ,\ v_j,d ≥ 0\ (j=1,…,r) \. The feasible set is nonempty because 0∈0 . Its Lagrange dual is p∗=maxλ≥0mind∈⟨u−∑jλjvj,d⟩,p^*= _λ≥ 0 _d u- _j _jv_j,d , where compactness of D and linear-programming duality give equality and dual attainment. Thus p∗≥−ηp^*≥-η if and only if some λ≥0λ≥ 0 satisfies ⟨u−∑jλjvj,d⟩≥−η(d∈). u- _j _jv_j,d ≥-η (d ). This is exactly the assertion that the residual belongs to ℛη−()R^-_η(D). ∎ Replacing exact coverage by the lower coverage of Definition 3.2 yields the approximate form of the characterization, with the same structure. Theorem 5 (Approximate coalitional characterization of soundness). Fix εharm≥0 _harm≥ 0. If every minimally winning coalition of an authorization rule W εharm _harm-covers the principal from below on D, then W is εharm _harm-sound. For a monotone rule, the converse also holds: W is εharm _harm-sound if and only if uP∈coneui:i∈W+ℛεharm−()for every W∈ℳ(). u_P \u_i:i∈ W\+R^-_ _harm(D) every W (W). Proof. Fix W∈ℳ()W (W). Lemma 1 says that its displayed lower-cover condition is equivalent to ⟨uP,d⟩≥−εharmwhenever d∈ and ⟨ui,d⟩≥0(i∈W). u_P,d ≥- _harm d and u_i,d ≥ 0\ (i∈ W). If every minimally winning coalition has this property, any executed proposal contains one such coalition among its approvers and therefore satisfies the same bound. Conversely, under monotonicity, every proposal approved by all members of a winning W is executed, so soundness implies the displayed property and hence the lower cover. ∎ Appendix B Additional sequential and strategic results B.1 Reward coverage modulo potential shaping Reward-function coverage requires the principal’s reward to equal a fixed nonnegative combination of reviewer rewards. For fixed dynamics, we can allow a difference between these functions provided it does not affect any comparison with the baseline. Discounted potential differences have this property: their discounted contributions telescope in expectation, leaving a term that depends only on the initial state. We first define coverage up to such a difference and then characterize the baseline-advantage identities it preserves. Definition B.1 (Coverage modulo potential shaping). For a function h:→ℝh:S , define the expected discounted potential difference (h)(s,a):=h(s)−γs′∼(⋅∣s,a)[h(s′)].(Th)(s,a):=h(s)- _s P(· s,a)[h(s )]. A coalition S covers the principal modulo potential shaping if there are coefficients λiS≥0 _i^S≥ 0 and a potential hSh_S such that rP=∑i∈SλiSri+hS.r_P= _i∈ S _i^Sr_i+Th_S. The potential term adds the same expected amount to every policy’s return from a fixed initial state. It therefore cancels when a policy is compared with the baseline. The theorem shows that this accounts for every reward residual that leaves the fixed combination of baseline advantages unchanged. Theorem 6 (Reward coverage modulo potential shaping). Fix the transition kernel, discount factor, and baseline policy, and fix nonnegative coefficients (λi)i∈S( _i)_i∈ S. The following are equivalent: (i) the principal’s baseline advantages satisfy AP0(s,a)=∑i∈SλiAi0(s,a)for every (s,a);A_P^0(s,a)= _i∈ S _iA_i^0(s,a) every (s,a); (i) there is a potential h:→ℝh:S such that rP=∑i∈Sλiri+h.r_P= _i∈ S _ir_i+Th. Whenever these conditions hold, every possibly history-dependent policy π and initial state s0s_0 satisfy JP(π∣s0)−VP0(s0)=∑i∈Sλi(Ji(π∣s0)−Vi0(s0)).J_P(π s_0)-V_P^0(s_0)= _i∈ S _i (J_i(π s_0)-V_i^0(s_0) ). Thus, for fixed dynamics, coverage modulo potential shaping holds exactly when the same nonnegative coefficients express the principal’s baseline advantage as a combination of reviewer advantages at every state-action pair. Proof. Let g:=rP−∑i∈Sλiri.g:=r_P- _i∈ S _ir_i. Linearity gives Ag0=AP0−∑iλiAi0A_g^0=A_P^0- _i _iA_i^0. If g=hg=Th, then the baseline value of g telescopes to Vg0=hV_g^0=h, and Qg0(s,a)=g(s,a)+γ[h(s′)∣s,a]=h(s).Q_g^0(s,a)=g(s,a)+ [h(s ) s,a]=h(s). Thus Ag0=Qg0−Vg0=0A_g^0=Q_g^0-V_g^0=0, proving (i)⇒ (i). Conversely, if Ag0=0A_g^0=0, then Qg0(s,a)=Vg0(s)Q_g^0(s,a)=V_g^0(s) for every (s,a)(s,a). The Bellman identity for Qg0Q_g^0 therefore gives g(s,a)=Vg0(s)−γ[Vg0(s′)∣s,a]=(Vg0)(s,a),g(s,a)=V_g^0(s)- [V_g^0(s ) s,a]=(TV_g^0)(s,a), which proves (i)⇒ (i). Finally, apply the performance-difference identity to each participant and use the pointwise advantage identity: JP(π∣s0)−VP0(s0) J_P(π s_0)-V_P^0(s_0) =π[∑t≥0γtAP0(st,at)] =E_π\! [ _t≥ 0γ^tA_P^0(s_t,a_t) ] =∑i∈Sλiπ[∑t≥0γtAi0(st,at)] = _i∈ S _iE_π\! [ _t≥ 0γ^tA_i^0(s_t,a_t) ] =∑i∈Sλi(Ji(π∣s0)−Vi0(s0)). = _i∈ S _i (J_i(π s_0)-V_i^0(s_0) ). ∎ For fixed dynamics, Theorem 3 and its approximate-stability bound also hold if every minimally winning coalition covers the principal modulo potential shaping: the return identity above is all their proofs require. Corollary 6 is the special case h=0h=0. The theorem also absorbs additive constants: taking h constant makes h=(1−γ)hTh=(1-γ)h. For fixed dynamics, adding a potential difference shifts all policy values by the same amount at a given initial state. Baseline advantages and gains relative to the baseline are therefore unchanged. The shaping representation depends on the transition kernel, whereas reward-function coverage does not. B.2 Veto power and undominated outcomes A reviewer with veto power can secure its baseline payoff by disapproving, so no equilibrium can leave it worse off. Coverage by these reviewers therefore suffices to protect the principal. If they do not cover the principal, a harmful proposal can be acceptable to every reviewer with veto power, while no other reviewer can force rejection alone. Proposition 6 (Veto power and strategic voting in one decision). Fix a monotone authorization rule W. Every mixed Nash equilibrium gives the principal weakly nonnegative expected gain for every feasible proposal if and only if the veto set V()V(W) covers the principal relative to C. Proof. For the sufficient direction, let p be the probability that the proposal d is executed at a mixed Nash equilibrium. Any veto reviewer can unilaterally guarantee rejection by disapproving. Its equilibrium gain relative to rejection must therefore satisfy p⟨ui,d⟩≥0for every i∈V().p u_i,d ≥ 0 every i∈ V(W). If p=0p=0, the principal’s expected gain is zero. If p>0p>0, every veto reviewer has nonnegative gain from execution. Write the veto set’s coverage certificate as uP=∑i∈V()λiui+z,λi≥0,z∈∗.u_P= _i∈ V(W) _iu_i+z, _i≥ 0, z ^*. Pairing this identity with d∈⊆d gives ⟨uP,d⟩≥0 u_P,d ≥ 0, so the principal’s expected gain p⟨uP,d⟩p u_P,d is nonnegative. Conversely, if V()V(W) does not cover the principal, equivalence (1) gives a deviation d∈d such that ⟨ui,d⟩≥0(i∈V()),⟨uP,d⟩<0. u_i,d ≥ 0 (i∈ V(W)), u_P,d <0. Scale d into D. At the all-approve voting profile, a unilateral disapproval by a non-veto reviewer leaves the proposal executed, while a veto reviewer weakly prefers execution to rejection. Thus this profile is an unsafe pure Nash equilibrium. ∎ Weak dominance provides a complementary refinement of the equilibrium prediction. Corollary 10 (Weakly undominated one-decision outcomes). Let W be monotone, and suppose every minimally winning coalition covers the principal. At any action profile in which no reviewer with a strict preference casts a weakly dominated vote, every executed proposal is principal-acceptable. In particular, under a k-tolerant threshold, k-robust coalitional alignment makes every weakly undominated outcome safe. Proof. If a proposal is executed, its approval set contains a minimally winning coalition W. Each member of W is pivotal when precisely the other members of W approve. Proposition 2 implies that every member of W who approves has nonnegative gain from execution. Pairing these gains with the nonnegative coefficients in W’s coverage certificate gives nonnegative principal gain. ∎ Ordinary Nash equilibrium permits only unilateral deviations, so veto power gives the corresponding sufficient condition in the control process. Corollary 11 (Nash safety from veto reviewers). Fix a monotone authorization rule W. If the veto reviewers V()V(W) cover the principal in reward-function space, then every Nash equilibrium of the reviewer game is baseline-safe for every proposer policy and initial state. Proof. At a Nash equilibrium, write Δj:=Jj(ρ,σ∣s0)−Vj0(s0) _j:=J_j(ρ,σ s_0)-V_j^0(s_0). Every veto reviewer can unilaterally switch to always disapproving and force the baseline, so Δi≥0 _i≥ 0 for every i∈V()i∈ V(W). Pairing these inequalities with the veto set’s reward-cover identity gives ΔP≥0 _P≥ 0. ∎ Appendix C Binary-information limits and approximate implementation The proof of Theorem 4 applies LP duality to the proposals sharing an approval set. At a suitably chosen principal-indifferent proposal, complementary slackness ensures that the only nonzero terms in the dual certificate come from reviewers who share the principal’s indifference hyperplane. Monotonicity or authorization at principal indifference then forces at least one of these reviewers to agree with the principal’s preferences, rather than reverse them. Proof of Theorem 4. Work in orthonormal coordinates on L, and use uP,uiu_P,u_i for the utility vectors projected onto L. This preserves all feasible gains, makes C full-dimensional in L, and turns the desired proportionality on L into a vector equality. The two strict inequalities imply uP≠0u_P≠ 0. Choosing a principal-indifferent proposal. Let H:=d∈L:⟨uP,d⟩=0H:=\d∈ L: u_P,d =0\. We first show that H∩ri()H (C) contains a nonempty relatively open subset of H. Fix d0∈ri()d_0 (C). For every d∈d and t≥0t≥ 0, d0+td∈ri()d_0+td (C): translating a small ball around d0d_0 by tdtd keeps it in C because +=C+C=C. Taking d=d+d=d_+ or d=d−d=d_- and t sufficiently large gives interior points with opposite principal gains. The segment between them crosses H inside ri()ri(C). For every reviewer whose projected utility is not a scalar multiple of uPu_P, the set d∈H:⟨ui,d⟩=0\d∈ H: u_i,d =0\ is a proper subspace of H. A finite union of proper subspaces has empty interior in H. We can therefore choose x∈H∩ri()x∈ H (C) outside all these subspaces. Thus every reviewer indifferent at x has ui=ciuPu_i=c_iu_P for some ci∈ℝc_i ; reviewers indifferent over all of L have ci=0c_i=0. An LP for a fixed approval set. Since C is polyhedral and full-dimensional in L, write =d∈L:⟨ah,d⟩≥0(h=1,…,ℓ),C=\d∈ L: a_h,d ≥ 0\ (h=1,…, )\, where the ah∈La_h∈ L are nonzero; the list is empty if =LC=L. Because x∈ri()x (C), every ⟨ah,x⟩>0 a_h,x >0. Let σi=1 _i=1 if i∈S(x)i∈ S(x) and σi=−1 _i=-1 otherwise, and define K:=d∈:σi⟨ui,d⟩≥0for every i∈N.K:=\d : _i u_i,d ≥ 0\ for every i∈ N\. This is the closure of the feasible deviations with approval set S(x)S(x). Indeed, for any d∈Kd∈ K and 0<t≤10<t≤ 1, the deviation (1−t)d+tx(1-t)d+tx has exactly this approval set: approvals remain nonnegative, and every disapproval is strict because it is strict at x. Set b=1b=1 if f(S(x))=1f(S(x))=1 and b=−1b=-1 otherwise. The strict-separation hypotheses imply b⟨uP,d⟩≥0b u_P,d ≥ 0 for deviations with approval set S(x)S(x), and hence throughout K by continuity. Consequently, the LP minimized∈L d∈ Lminimize b⟨uP,d⟩ b u_P,d subject to to ⟨ah,d⟩≥0 a_h,d ≥ 0 (h=1,…,ℓ), (h=1,…, ), σi⟨ui,d⟩≥0 _i u_i,d ≥ 0 (i∈N) (i∈ N) has optimal value zero, attained at x. Its dual is maximizeα,λ α,λmaximize 0 0 subject to to buP=∑h=1ℓαhah+∑i∈Nλiσiui, bu_P= _h=1 _ha_h+ _i∈ N _i _iu_i, α∈ℝ+ℓ,λ∈ℝ+N. α _+ , λ _+^N. LP strong duality supplies such multipliers. Complementary slackness and alignment. Pairing the dual equality with x gives 0=∑h=1ℓαh⟨ah,x⟩+∑i∈Nλi|⟨ui,x⟩|.0= _h=1 _h a_h,x + _i∈ N _i| u_i,x |. Every summand is nonnegative. Thus αh=0 _h=0 for every h, and λi=0 _i=0 whenever reviewer i is not indifferent at x. The remaining reviewers approve at x, so their σi _i equal 11. Writing I:=i:⟨ui,x⟩=0I:=\i: u_i,x =0\, we obtain buP=∑i∈Iλiui=(∑i∈Iλici)uP,henceb=∑i∈Iλici.bu_P= _i∈ I _iu_i= ( _i∈ I _ic_i )u_P, b= _i∈ I _ic_i. Since b≠0b≠ 0, some ci≠0c_i≠ 0, proving (i). Under the hypothesis in (i), x is authorized, so b=1b=1 and some ci>0c_i>0. Finally, monotonicity also forces x to be authorized. Choose v∈Lv∈ L with ⟨uP,v⟩>0 u_P,v >0. For sufficiently small ε>0 >0, x+εv∈x+ v , and every reviewer who disapproves at x still disapproves there. Thus S(x+εv)⊆S(x)S(x+ v) S(x). The strict principal gain at x+εvx+ v forces its authorization, so monotonicity gives f(S(x))=1f(S(x))=1. The same identity then proves (i). ∎ C.1 Random and deterministic approximate implementation For a distribution ν over reviewer utility vectors, write pν(d):=Pru∼ν[⟨u,d⟩<0]p_ν(d):= _u ν[ u,d <0] for the population fraction that disapproves proposal d. We seek a population in which harmful proposals receive a majority of disapprovals and beneficial ones a majority of approvals. The margin below measures how far both fractions stay from one half when the principal’s gain is bounded away from zero. A sampled panel inherits these decisions when its sampling error is smaller than the margin. Definition C.1 (Two-sided population voting margin). For ε>0 >0, define ν(ε):=min m_ν( ):= \ infd∈⟨uP,d⟩<−ε(pν(d)−12), _ subarraycd \\ u_P,d <- subarray (p_ν(d)- 12 ), infd∈⟨uP,d⟩≥ε(12−pν(d)), _ subarraycd \\ u_P,d ≥ subarray ( 12-p_ν(d) ) \, where the infimum of an empty set is +∞+∞. Theorem 7 (Random panels with a two-sided voting margin). Let u1,…,unu_1,…,u_n be drawn independently from ν, let r:=dimspan()r:= (D), and suppose n=2m+1n=2m+1 is odd. There is a universal constant c>0c>0 such that, for every δ∈(0,1)δ∈(0,1), with probability at least 1−δ1-δ, the m-tolerant majority rule is an (ε,ε)( , )-implementation simultaneously for every ε>0 >0 satisfying ν(ε)>cvlog(2en/v)+log(2/δ)n,v:=minr,n. m_ν( )>c v (2en/v)+ (2/δ)n, v:= \r,n\. Consequently, if r is fixed and ν(ε)>0 m_ν( )>0 for every ε>0 >0, there are εn→0 _n→ 0 for which majority voting is (εn,εn)( _n, _n)-implementing with probability tending to one. Proof. Let p^n(d):=1n∑i=1n⟨ui,d⟩<0. p_n(d):= 1n _i=1^n1\ u_i,d <0\. As d varies, the events u:⟨u,d⟩<0\u: u,d <0\ form a subclass of the homogeneous halfspaces in the r-dimensional feasible span, whose VC dimension is at most r. The standard uniform-convergence bound gives, with probability at least 1−δ1-δ, supd∈|p^n(d)−pν(d)|≤cvlog(2en/v)+log(2/δ)n. _d | p_n(d)-p_ν(d)|≤ c v (2en/v)+ (2/δ)n. If the right side is smaller than ν(ε) m_ν( ), every proposal with principal gain below −ε- has p^n(d)>1/2 p_n(d)>1/2 and is rejected, while every proposal with gain at least ε has p^n(d)<1/2 p_n(d)<1/2 and is executed. Because n is odd, these are exactly the two decisions of the m-tolerant rule. This proves the finite-sample statement. For the final claim, take any δn→0 _n→ 0 slowly enough that the bound tends to zero, and then choose εn→0 _n→ 0 for which the margin exceeds it. ∎ The fixed-angle model in Proposition 4 is a concrete instance. Write d=(x,y,z)d=(x,y,z), set R:=x2+y2R:= x^2+y^2, and draw u=e3+tanθ(cosΦ,sinΦ,0),Φ∼Unif[0,2π).u=e_3+ θ( , ,0), [0,2π). This is a positive rescaling of the unit vectors shown in Figure 4, so it induces the same votes and every draw forms angle θ with uP=e3u_P=e_3. For R>0R>0 and z≥0z≥ 0, 12−pν(d)=1πarcsin(min1,cotθzR), 12-p_ν(d)= 1π \! ( \1, θ\,zR \ ), and symmetry gives the same margin for z<0z<0. If R=0R=0, every reviewer votes with the principal. Otherwise, on ‖d‖1≤1\|d\|_1≤ 1 we have R≤1−|z|R≤ 1-|z|, and hence ν(ε)≥min1,cotθπε. m_ν( )≥ \1, θ\π\, . Theorem 7, with r=3r=3 and for example δ=n−2δ=n^-2, therefore gives εn=Oθ(logn/n) _n=O_θ( n/n) with probability tending to one. Equally spacing the same directions removes sampling error and gives the sharper deterministic rate. Proposition 7 (Balanced ring construction). Use orthonormal coordinates on L=ℝ3L=R^3, let :=d∈ℝ3:‖d‖1≤1,uP:=e3,D:=\d ^3:\|d\|_1≤ 1\, u_P:=e_3, and fix θ∈(0,π/2)θ∈(0,π/2). For n=2m+1n=2m+1, define ϕj:=−πmn+2π(j−1)n,uj:=(sinθcosϕj,sinθsinϕj,cosθ). _j:=- π mn+ 2π(j-1)n, u_j:= ( θ _j,\, θ _j,\, θ ). Under the m-tolerant threshold, set cn:=tanθsin(π2n),εn:=cn1+cn.c_n:= θ \! ( π2n ), _n:= c_n1+c_n. Then the rule gives an (εn,εn)( _n, _n)-implementation on D. Every reviewer forms angle θ with the principal and satisfies infc>0‖uj−cuP‖2=sinθ. _c>0\|u_j-cu_P\|_2= θ. Consequently, εn=Θ(1/n) _n= (1/n) for fixed θ, although no reviewer approaches the principal’s utility ray. Proof. Write d=(rcosψ,rsinψ,z)d=(r ψ,r ψ,z), where r≥0r≥ 0. If r=0r=0, every reviewer votes according to the sign of z, so both guarantees hold. Suppose r>0r>0. Reviewer j’s gain is ⟨uj,d⟩=sinθrcos(ϕj−ψ)+cosθz. u_j,d = θ\,r ( _j-ψ)+ θ\,z. First consider a harmful proposal z<0z<0. If a:=cotθ|z|r>1,a:= θ\,|z|r>1, no reviewer approves. Otherwise, the approving angles ϕj _j lie in a closed arc centered at ψ with half-width α:=arccosaα:= a. Execution requires at least m+1m+1 approvals. Any closed arc containing m+1m+1 of the n=2m+1n=2m+1 equally spaced angles has length at least 2πm/n2π m/n, and hence α≥πmn=π2−π2n.α≥ π mn= π2- π2n. Therefore a≤sin(π/(2n))a≤ (π/(2n)), or equivalently |z|≤cnr|z|≤ c_nr. Since ‖d‖1=r(|cosψ|+|sinψ|)+|z|≤1,\|d\|_1=r (| ψ|+| ψ| )+|z|≤ 1, we have r+|z|≤1r+|z|≤ 1, and hence |z|≤cnr≤cn(1−|z|).|z|≤ c_nr≤ c_n(1-|z|). Thus z≥−cn/(1+cn)=−εnz≥-c_n/(1+c_n)=- _n, proving εn _n-soundness. Now suppose z≥εnz≥ _n. If cotθz/r≥1 θ\,z/r≥ 1, every reviewer approves. Otherwise, the strictly disapproving angles lie in an open arc centered at ψ+πψ+π with half-width β:=arccos(cotθzr).β:= \! ( θ\,zr ). The norm constraint gives r≤1−zr≤ 1-z, so zr≥z1−z≥εn1−εn=cn. zr≥ z1-z≥ _n1- _n=c_n. It follows that β≤arccos(sinπ2n)=πmn.β≤ \! ( π2n )= π mn. An open arc of length at most 2πm/n2π m/n contains at most m of the equally spaced angles. At most m reviewers therefore disapprove, so the proposal is executed. This proves εn _n-completeness. Finally, uju_j and uPu_P are unit vectors with inner product cosθ θ. The closest point on the principal’s positive ray is (cosθ)uP( θ)u_P, at distance sinθ θ. The estimate sin(π/(2n))=Θ(1/n) (π/(2n))= (1/n) proves the final claim. ∎ The ℓ1 _1 unit ball is a polytope. After a common rescaling, its six vertices can be realized as finitely many outcome deviations around an interior baseline lottery, so both fixed-angle examples lie within the finite-action model. Appendix D Robustness to reporting and evaluation failures The safety argument combines the gains of approving reviewers using a coverage representation. If a reviewer misestimates its gain, an approval can instead conceal a small loss; an approximate coverage representation can introduce a further discrepancy from the principal’s utility. We now bound the effect of these errors. Since each reviewer’s error is multiplied by its coefficient in the representation, a useful loss bound must control the size of those coefficients as well as the coverage residual. The following definition imposes both bounds. Definition D.1 (Coverage with bounded error and coefficients). For ε,Λ≥0 , ≥ 0, a coalition W has an (ε,Λ)( , )-cover of the principal if there exist λi≥0 _i≥ 0, z∈∗z ^*, and a residual r such that uP=∑i∈Wλiui+z+r,∑i∈Wλi≤Λ,supd∈|⟨r,d⟩|≤ε.u_P= _i∈ W _iu_i+z+r, _i∈ W _i≤ , _d | r,d |≤ . Let ηi _i bound the absolute error in reviewer i’s reported gain. Positive rescaling of a reviewer utility does not change its exact vote. Under the transformation ui↦aiuiu_i a_iu_i with ai>0a_i>0, the corresponding coefficient and error bound transform as λi↦λi/ai _i _i/a_i and ηi↦aiηi _i a_i _i, so the product λiηi _i _i is invariant. The error contribution is therefore ∑iλiηi _i _i _i. Corollary 12 bounds this sum by Λsηs _s _s using a common error bound ηs _s across reviewers at each state. Whenever we use a common coefficient budget Λ and common error η, reviewer utilities are normalized so that supd∈|⟨ui,d⟩|≤1 _d | u_i,d |≤ 1, and η is measured in those normalized units. Under this normalization, Λ controls how much evaluation errors can be amplified: even small reviewer errors can produce a large principal loss if the coverage representation uses very large coefficients. We can also allow some reviewers to report arbitrarily. Under a k-tolerant threshold, an authorized proposal has at least n−kn-k approvals. If at most b reviewers report arbitrarily, at least n−k−bn-k-b of those approvals still come from reviewers with bounded evaluation error. Coverage by every coalition of this size lets us apply the preceding argument to these remaining approvers. The resulting local loss combines the coverage residual with the errors in their reported gains, and Corollary 4 then accumulates the loss over time. Corollary 12 (A unified robustness budget). Fix a k-tolerant threshold, an integer b∈0,…,n−k−1b∈\0,…,n-k-1\, and a function εharm:→[0,∞) _harm:S→[0,∞). On each proposal, allow at most b reviewers to report arbitrarily. Every other reviewer i reports a score g^i(s,d) g_i(s,d) satisfying |g^i(s,d)−⟨Qi0(s,⋅),d⟩|≤ηs(d∈s) | g_i(s,d)- Q_i^0(s,·),d |≤ _s (d _s) and approves exactly when this score is nonnegative. Suppose that, at every state s, every coalition of n−k−bn-k-b reviewers has an (εharm(s),Λs)( _harm(s), _s)-cover of QP0(s,⋅)Q_P^0(s,·) on sD_s. Then every initial state and history-adaptive proposer agent satisfy JP(π∣s0)≥VP0(s0)−π[∑t=0∞γt(εharm(st)+Λstηst)|s0]. J_P(π s_0)≥ V_P^0(s_0)-E_π [ _t=0^∞γ^t ( _harm(s_t)+ _s_t _s_t )\, |\,s_0 ]. In particular, uniform bounds εharm(s)≤ε¯harm _harm(s)≤ _harm, Λs≤Λ _s≤ , and ηs≤η _s≤η give loss at most (ε¯harm+Λη)/(1−γ)( _harm+ η)/(1-γ). The two local errors appear separately in the bound: εharm(s) _harm(s) comes from approximate coverage, and Λsηs _s _s from errors in the gains of the remaining approvers. Their sum is the possible loss at one state. The expectation accumulates these losses along the actual trajectory with the same discounting as in Corollary 4. Proof. An executed proposal has at least n−kn-k reported approvals. After removing the at most b arbitrary reporters, at least n−k−bn-k-b sincere reported approvers remain. Choose any coalition W of exactly that size. Each of its members has true baseline advantage at least −ηs- _s, so its (εharm(s),Λs)( _harm(s), _s)-cover implies principal baseline advantage at least −εharm(s)−Λsηs- _harm(s)- _s _s. Corollary 4 composes these conditional local bounds along the actual trajectory. The uniform bound follows by summing the geometric series. ∎ Appendix E Auditing coalitional alignment E.1 Auditing problem and main result Implementing a threshold rule requires only counting votes, but verifying its soundness requires considering every feasible proposal. The rejection-depth characterization makes this a search for a harmful proposal with too few objections. For an allowed harm tolerance εharm≥0 _harm≥ 0, define δharm(εharm):=mind∈⟨uP,d⟩<−εharm|N−(d)|, _harm( _harm):= _ subarraycd \\ u_P,d <- _harm subarray|N^-(d)|, with value n+1n+1 if there is no such proposal. At zero tolerance this is the rejection depth from Definition 3.3. For a fixed threshold k, the audit asks whether δharm(εharm)≥k+1 _harm( _harm)≥ k+1. The auditor is given explicit utility vectors and a description of the feasible proposal set. It must either certify this inequality or return a proposal that exceeds the allowed loss and receives at most k disapprovals. Computing the depth also identifies the most permissive threshold at that tolerance. These calculations use the principal’s utility offline; the authorization rule itself continues to receive only reviewer votes. We treat the utilities as inputs to the audit. The effect of bounded errors in proposal evaluations is addressed separately in Appendix D. Proposition 8 (Algorithms and hardness of coalitional-alignment auditing). Assume the utility vectors, εharm _harm, and a halfspace description of D are rational, and let r:=dimspan()r:= (C). (i) For fixed r, δharm(εharm) _harm( _harm) and hence the maximum robustness level can be computed in time nO(r)n^O(r) times a polynomial in the rational input length. (i) For fixed k, failure of k-robust lower coalitional alignment at tolerance εharm _harm can be decided using at most ∑j=0k(nj) _j=0^k nj linear feasibility problems. In particular, certification is polynomial-time for constant k. (i) For every fixed rational εharm∈[0,1) _harm∈ [0,1 ), when r and k are part of the input, deciding whether k-robust lower coalitional alignment at tolerance εharm _harm fails is NP-complete. This holds even when supd∈|⟨uP,d⟩|=1 _d | u_P,d |=1 and D is a simplex with a polynomial-size rational halfspace description. Consequently, certification is coNP-complete and computing the maximum robustness level at that tolerance is NP-hard. In fixed dimension, only polynomially many approval patterns need to be checked. With a fixed number k of tolerated disapprovals, it instead suffices to enumerate the possible small sets of disapprovers and solve a linear feasibility problem for each. When both dimension and k can grow, the audit can encode a satisfiability problem: reviewer pairs represent truth assignments and clause reviewers enforce satisfaction. Appendix E.2 gives the algorithms and reduction. Remark E.1 (Explicit action representation). The hard instances in part (i) can also be realized with only polynomially many explicitly listed actions in the original outcome-lottery model. Thus the hardness holds whether D is represented by a polynomial-size rational halfspace description or by a polynomial-size explicit action list. The construction is given below. When robustness fails, these procedures return a harmful proposal and the reviewers who object to it. When it holds, exhaustive methods can return coverage certificates for the potentially decisive coalitions, although the total certificate can be exponentially large. E.2 Geometry, algorithms, and hardness The dimension of the feasible span also limits how many reviewers are needed in a single coverage certificate. Only utility components on this span affect a proposal’s gain. Once the dual-cone residual is fixed, conic Carathéodory reduces the remaining nonnegative combination to at most as many reviewer vectors as the dimension. Thus, even if a large coalition covers the principal, a small subcoalition already supplies that certificate. The next proposition uses this observation to bound the minimal winning coalitions of the maximally permissive sound rule. Proposition 9 (Small coalitional certificates). Let r:=dimspan()r:= (C). Every inclusion-minimal coalition that covers the principal has size at most r. Consequently, the maximally permissive monotone sound rule is generated by covering certificates involving at most r reviewers and has at most ∑j=0minr,n(nj) _j=0 \r,n\ nj minimally winning coalitions, hence O(nr)O(n^r) for fixed r. Proof. Let S be a minimally covering coalition. If S=∅S= , there is nothing to prove. Otherwise choose uP=∑i∈Sλiui+z,λi≥0,z∈∗,u_P= _i∈ S _iu_i+z, _i≥ 0, z ^*, and let PLP_L denote orthogonal projection onto L:=span()L:=span(C). By conic Carathéodory in the r-dimensional space L, there are T⊆ST S, |T|≤r|T|≤ r, and μi≥0 _i≥ 0 such that ∑i∈SλiPLui=∑i∈TμiPLui. _i∈ S _iP_Lu_i= _i∈ T _iP_Lu_i. Thus h:=∑i∈Sλiui−∑i∈Tμiuih:= _i∈ S _iu_i- _i∈ T _iu_i lies in L⟂L , so ⟨h,d⟩=0 h,d =0 for every feasible deviation d. Adding h to z therefore preserves its nonnegative inner product with every d∈d , giving uP=∑i∈Tμiui+(z+h)with z+h∈∗.u_P= _i∈ T _iu_i+(z+h) z+h ^*. The subset T also covers the principal, so minimality forces T=ST=S and |S|≤r|S|≤ r. Enumerating all subsets of size at most minr,n \r,n\ gives the stated count. ∎ Rejection depth has a related geometric interpretation. Reverse a harmful deviation so that the principal’s inner product with the resulting direction is positive. The reviewer vectors with positive inner product in this direction are exactly those that objected to the original deviation. Thus each harmful proposal defines an open halfspace containing the principal’s direction, and counting reviewers in that halfspace counts its objections. Minimizing this count over feasible directions gives the halfspace-depth formulation below. Theorem 8 (Angular representation of rejection depth). At zero tolerance, let L:=span()L:=span(C) and let p and xix_i be the orthogonal projections of uPu_P and uiu_i onto L. Then δharm(0)=minv∈−⟨p,v⟩>0|i∈N:⟨xi,v⟩>0|. _harm(0)= _ subarraycv∈-C\\ p,v >0 subarray | \i∈ N: x_i,v >0 \ |. The minimum is defined as n+1n+1 if its feasible set is empty. When =LC=L is a linear space and p≠0p≠ 0, the right side is the strict angular halfspace depth of the principal’s direction relative to the projected reviewer vectors, with any zero projections treated as boundary points. Consequently, exact k-robust coalitional alignment holds if and only if this angular depth is at least k+1k+1. Proof. At zero tolerance, only projections onto L affect pairings with feasible deviations. Because all relevant inequalities are homogeneous and every direction in C can be scaled into D, the minimum may be taken over d∈d . Substitute v=−dv=-d. Principal harm becomes ⟨p,v⟩>0 p,v >0, while reviewer disapproval becomes ⟨xi,v⟩>0 x_i,v >0, which gives the formula. When =LC=L, the normal ranges over every direction and the expression counts reviewer directions in an open hemisphere containing the principal direction. The final equivalence is Corollary 2. ∎ Angular halfspace depth is the directional analogue of Tukey’s halfspace depth: its separating hyperplanes pass through the origin, and only hemispheres containing a specified query direction are considered (Nagy and Laketa, 2025; Dyckerhoff and Nagy, 2025). The usual definition uses closed hemispheres. Our strict inequalities exclude reviewers on the boundary because ties count as approvals; the strict and closed versions agree whenever an optimal hemisphere can be chosen with no reviewer or principal direction on its boundary. For a one-sided cone C, the restriction v∈−v∈-C further permits only normals corresponding to feasible proposal directions. Exact coalitional alignment is therefore a cone-restricted strict angular-depth problem, not ordinary affine Tukey depth. Positive tolerance additionally depends on the scale of D. The depth formulation also exposes the computational problem. Whenever the harmful set is nonempty, δharm(εharm)=n−maxd∈⟨uP,d⟩<−εharm|i∈N:⟨ui,d⟩≥0|. _harm( _harm)=n- _ subarraycd \\ u_P,d <- _harm subarray | \i∈ N: u_i,d ≥ 0 \ |. Thus an audit searches for a sufficiently harmful proposal approved by as many reviewers as possible. This is a maximum feasible subsystem problem, the same structure underlying exact Tukey-depth computation (Bremner and Chen, 2009). Still assuming a harmful direction exists, positive scaling at zero tolerance gives the normalized form depth(uP,u1,…,un)=n−maxd∈⟨uP,d⟩≤−1|i∈N:⟨ui,d⟩≥0|.depth_C(u_P;u_1,…,u_n)=n- _ subarraycd \\ u_P,d ≤-1 subarray | \i∈ N: u_i,d ≥ 0 \ |. Proof of Proposition 8. For (i), the reviewer indifference hyperplanes d:⟨ui,d⟩=0\d: u_i,d =0\ divide the r-dimensional feasible span into regions and their boundaries. More precisely, they form an arrangement of nO(r)n^O(r) relatively open faces on each of which every reviewer’s gain has a fixed sign: positive, negative, or zero. The disapproval set is therefore constant on each face. Let I+,I−,I0I_+,I_-,I_0 be the reviewer indices with these three signs on a given face. To test whether that face contains a sufficiently harmful proposal, maximize a common slack τ subject to d∈,⟨ui,d⟩≥τ(i∈I+),⟨ui,d⟩≤−τ(i∈I−),⟨ui,d⟩=0(i∈I0),⟨uP,d⟩≤−εharm−τ.d , u_i,d ≥τ\ (i∈ I_+), u_i,d ≤-τ\ (i∈ I_-), u_i,d =0\ (i∈ I_0), u_P,d ≤- _harm-τ. The relative face contains a sufficiently harmful point exactly when the optimum has τ>0τ>0. This is also the basis of fixed-dimensional algorithms for angular and ordinary halfspace depth (Dyckerhoff and Nagy, 2025). For (i), enumerate a set T⊆NT N of at most k possible disapprovers and test feasibility of d∈,⟨uP,d⟩<−εharm,⟨ui,d⟩≥0(i∉T).d , u_P,d <- _harm, u_i,d ≥ 0 (i∉ T). A feasible system is exactly a proposal worse than the allowed tolerance and disapproved by at most k reviewers. Rational strict feasibility can be checked by maximizing a slack variable. If no enumerated system is feasible, Corollary 2 certifies robust lower coalitional alignment. For (i), fix any rational constant εharm∈[0,1) _harm∈[0,1). Failure of k-robust lower coalitional alignment at tolerance εharm _harm belongs to NP. Indeed, guess a set S of at least n−kn-k approving reviewers and test feasibility of d∈,⟨uP,d⟩<−εharm,⟨ui,d⟩≥0(i∈S).d , u_P,d <- _harm, u_i,d ≥ 0 (i∈ S). This is a rational linear feasibility problem. Rational strict feasibility can be checked by maximizing a slack variable. For hardness, reduce from 33-SAT. Let φ have variables X1,…,XpX_1,…,X_p and clauses C1,…,CqC_1,…,C_q. Define S:=a∈ℝp:aj≥−1 for every j,∑j=1paj≤p,S:= \a ^p:a_j≥-1 for every j, _j=1^pa_j≤ p \, and let :=conv(0∪(a,−1):a∈S)⊆ℝp+1.D:=conv (\0\∪\(a,-1):a∈ S\ ) ^p+1. Equivalently, =(x,z):−1≤z≤0,xj≥z for every j,∑j=1pxj≤−pz.D= \(x,z):-1≤ z≤ 0, x_j≥ z for every j, _j=1^px_j≤-pz \. Indeed, every nonzero point of the convex hull has the form (x,z)=t(a,−1)(x,z)=t(a,-1) for some t∈(0,1]t∈(0,1] and a∈Sa∈ S. Since z=−tz=-t, the constraints aj≥−1a_j≥-1 and ∑jaj≤p _ja_j≤ p become xj≥zx_j≥ z and ∑jxj≤−pz _jx_j≤-pz. Conversely, if (x,z)(x,z) satisfies (∗)(*) with z<0z<0, taking t=−zt=-z and a=x/ta=x/t gives a∈Sa∈ S and (x,z)=t(a,−1)(x,z)=t(a,-1). If z=0z=0, then xj≥0x_j≥ 0 for every j and ∑jxj≤0 _jx_j≤ 0, forcing x=0x=0. The set S is a simplex. Under the change of variables yj=aj+1y_j=a_j+1, it becomes y∈ℝ+p:∑j=1pyj≤2p. \y _+^p: _j=1^py_j≤ 2p \. Thus S has p+1p+1 vertices, and D is a simplex with p+2p+2 vertices including 00. The description (∗)(*) has polynomial size. Set uP=ep+1.u_P=e_p+1. For every variable XjX_j, introduce two reviewers with utilities ujT=ej+ep+1,ujF=−ej+ep+1.u_j^T=e_j+e_p+1, u_j^F=-e_j+e_p+1. For each clause Cℓ=(ℓ,1∨ℓ,2∨ℓ,3),C_ =( _ ,1 _ ,2 _ ,3), let jℓ,tj_ ,t be the variable appearing in literal ℓ,t _ ,t, and define sℓ,t=1,ℓ,t=Xjℓ,t,−1,ℓ,t=¬Xjℓ,t.s_ ,t= cases1,& _ ,t=X_j_ ,t,\\ -1,& _ ,t= X_j_ ,t. cases Introduce the clause reviewer uCℓ=∑t=13sℓ,tejℓ,t−ep+1.u_C_ = _t=1^3s_ ,te_j_ ,t-e_p+1. There are n=2p+qn=2p+q reviewers. Set k=pk=p, so the k-tolerant threshold executes exactly when at least n−k=p+qn-k=p+q reviewers approve. Suppose first that φ is satisfiable. Encode a satisfying assignment by a∈−1,1pa∈\-1,1\^p, taking aj=1a_j=1 when XjX_j is true and aj=−1a_j=-1 when XjX_j is false. Since a∈Sa∈ S, d=(a,−1)∈.d=(a,-1) . Since ties count as approvals, exactly one of each variable’s two reviewers approves. If aj=1a_j=1, then ⟨ujT,d⟩=0,⟨ujF,d⟩=−2, u_j^T,d =0, u_j^F,d =-2, and if aj=−1a_j=-1 the roles reverse. Hence exactly p variable reviewers approve. For a clause CℓC_ , each signed term sℓ,tajℓ,ts_ ,ta_j_ ,t equals 11 when the corresponding literal is true and −1-1 when it is false. Since the clause is satisfied, ∑t=13sℓ,tajℓ,t≥−1. _t=1^3s_ ,ta_j_ ,t≥-1. Therefore ⟨uCℓ,d⟩=∑t=13sℓ,tajℓ,t+1≥0. u_C_ ,d = _t=1^3s_ ,ta_j_ ,t+1≥ 0. Thus every clause reviewer approves. The proposal receives p+q=n−kp+q=n-k approvals and is executed. Moreover, ⟨uP,d⟩=−1. u_P,d =-1. Since εharm<1 _harm<1, −1<−εharm,-1<- _harm, so the panel fails k-robust lower coalitional alignment at tolerance εharm _harm. Also, supd∈|⟨uP,d⟩|=1. _d | u_P,d |=1. Conversely, suppose that some nonzero d=(x1,…,xp,z)∈:=cone()d=(x_1,…,x_p,z) :=cone(D) is executed. Every nonzero direction in C has z<0z<0. Indeed, the homogeneous version of (∗)(*) gives z≤0,xj≥z for every j,∑jxj≤−pz.z≤ 0, x_j≥ z for every j, _jx_j≤-pz. If z=0z=0, then xj≥0x_j≥ 0 for every j and ∑jxj≤0 _jx_j≤ 0, forcing x=0x=0. For each variable XjX_j, its two reviewers cannot both approve. Their approval conditions are xj+z≥0and−xj+z≥0.x_j+z≥ 0 -x_j+z≥ 0. If both held, adding them would give 2z≥02z≥ 0, contradicting z<0z<0. Hence at most one reviewer from each variable pair approves, so at most p variable reviewers approve. Execution requires at least p+qp+q approvals, and there are only q clause reviewers. Therefore an executed nonzero direction must receive exactly one approval from every variable pair and approval from every clause reviewer. Use the approving variable reviewer to define a Boolean assignment: set XjX_j to true if ujTu_j^T approves and false if ujFu_j^F approves. We claim that this assignment satisfies every clause. Suppose instead that some clause CℓC_ is false. If a positive literal XjX_j in the clause is false, then ujFu_j^F approves, so −xj+z≥0⟹xj≤z.-x_j+z≥ 0 x_j≤ z. If a negative literal ¬Xj X_j in the clause is false, then XjX_j is true and ujTu_j^T approves, so xj+z≥0⟹−xj≤z.x_j+z≥ 0 -x_j≤ z. Thus for every literal in a false clause, sℓ,txjℓ,t≤z.s_ ,tx_j_ ,t≤ z. Consequently, ⟨uCℓ,d⟩=∑t=13sℓ,txjℓ,t−z≤3z−z=2z<0. u_C_ ,d = _t=1^3s_ ,tx_j_ ,t-z≤ 3z-z=2z<0. The clause reviewer would disapprove, contradicting the fact that every clause reviewer approves. Hence the extracted assignment satisfies every clause. Therefore, if φ is unsatisfiable, no nonzero direction in C is executed. The k-tolerant threshold is therefore exactly sound, and by Corollary 2 the panel is exactly k-robustly coalitionally aligned. Thus, for every fixed rational εharm∈[0,1) _harm∈[0,1), φ∈3-SAT⟹failure of k-robust lower coalitional alignment at tolerance εharm, ∈ 3-SAT of $k$-robust lower coalitional alignment at tolerance $ _ harm$, whereas φ∉3-SAT⟹exact k-robust coalitional alignment. ∉ 3-SAT $k$-robust coalitional alignment. The reduction is polynomial, proving NP-hardness. Together with membership in NP, failure is NP-complete and certification is coNP-complete. Computing the maximum robustness level at tolerance εharm _harm is NP-hard because comparing that level with k decides the same problem. Finally, the same hard instances admit a polynomial-size explicit action representation in the original outcome-lottery model. Let v0,…,vp∈ℝp+1v^0,…,v^p ^p+1 be the p+1p+1 nonzero vertices of D, and define T:ℝp+1→ℝp+2T:R^p+1 ^p+2 by T(y1,…,yp+1)=(y1,…,yp+1,−∑h=1p+1yh).T(y_1,…,y_p+1)= (y_1,…,y_p+1,- _h=1^p+1y_h ). The coordinates of every T(y)T(y) sum to zero. Moreover, ‖T(vh)‖∞≤2p\|T(v^h)\|_∞≤ 2p for every vertex. Let α:=12p(p+2)andμ¯:=1p+2.α:= 12p(p+2) μ:= 1p+21. For each h=0,…,ph=0,…,p, introduce one nonbaseline action with outcome distribution μh:=μ¯+αT(vh). _h:= μ+α T(v^h). Each μh _h sums to one, and ‖αT(vh)‖∞≤1p+2,\|α T(v^h)\|_∞≤ 1p+2, so every coordinate of μh _h is nonnegative. Thus each μh _h is a valid outcome lottery. The feasible deviation set generated by these actions is ~=αT(). D=α T(D). For every utility vector u∈ℝp+1u ^p+1 in the reduction, define u~=(uα,0)∈ℝp+2. u= ( uα,0 ) ^p+2. Then for every d∈d , ⟨u~,αT(d)⟩=⟨u,d⟩. u,α T(d) = u,d . Hence the embedding preserves every reviewer gain, every approval set, and the principal gain exactly. It uses only p+1p+1 nonbaseline actions in addition to the baseline, with rational data of polynomial bit complexity. Thus the hardness also holds under a polynomial-size explicit action representation. ∎ Appendix F Additional experimental details This appendix gives the benchmark constructions and calculations for Section 7, followed by the learned authorization rules, threshold selection, and additional comparisons. All experiments use the one-decision model and scores provided by the benchmarks; no new model calls are needed. Reviewers vote sincerely, with ties counting as approvals. The lottery calculations evaluate soundness and completeness over every feasible proposal. F.1 Benchmark mappings and proposal domains We turn each benchmark example into an authorization decision by specifying the principal, the reviewers, and the baseline. The superscript e identifies the example, and gPeg_P^e and gieg_i^e denote gains relative to its baseline. RewardBench 2: choose an answer or a lottery. RewardBench 2 (Malik et al., 2026) supplies 1,763 standard prompts with four responses, exactly one labeled correct response, and scores from 48 reward models. The prompts span Factuality, Focus, Math, Precise Instruction Following, and Safety. For each prompt e, index the correct response first and define Δ4=q∈ℝ≥04:⊤q=1,be=14,uP=(1,0,0,0). _4=\q _≥ 0^4:1 q=1\, b^e= 141, u_P=(1,0,0,0). A proposal q∈Δ4q∈ _4 specifies a lottery over the four responses. The panel reviews q before sampling: authorization draws an answer from q, while rejection draws from the uniform baseline beb^e. The principal’s gain is the resulting change in the probability of a correct answer, gPe(q)=uP⊤(q−be)=q1−14.g_P^e(q)=u_P (q-b^e)=q_1- 14. For reviewer i, write its score vector as sie=(si1e,…,si4e)∈ℝ4s_i^e=(s_i1^e,…,s_i4^e) ^4 and its baseline score as μie=(sie)⊤be _i^e=(s_i^e) b^e. The reviewer evaluates a lottery by its expected score and approves exactly when gie(q)=(sie)⊤q−μie=(sie)⊤(q−be)≥0.g_i^e(q)=(s_i^e) q- _i^e=(s_i^e) (q-b^e)≥ 0. For a deterministic answer, the reviewer’s gain is sije−μies_ij^e- _i^e. For lotteries, we treat the four scores as cardinal utilities and assume linearity in the lottery probabilities, as in Definition 2.1. This assumes, for example, indifference between an answer scored 11 and a 50–50 lottery over answers scored 00 and 22; the benchmark evaluates only individual answers. StrongREJECT: release or abstain. StrongREJECT (Souly et al., 2024) provides human jailbreak-success labels and scores from automated evaluators. We retain the 1,084 responses with all five human labels and all nine evaluator scores. For response e, let he∈[0,1]h^e∈[0,1] be the median human harm score and sie∈[0,1]s_i^e∈[0,1] evaluator i’s harm score; larger values indicate greater harm. A proposal specifies the probability r∈[0,1]r∈[0,1] of releasing that response, with r=0r=0 giving the abstention baseline beb^e and r=1r=1 giving pure release. At the fixed policy cutoff 1/21/2, the gains are gPe(r)=r(12−he),gie(r)=r(12−sie).g_P^e(r)=r( 12-h^e), g_i^e(r)=r( 12-s_i^e). This cutoff, which is not estimated from the data, classifies 894 responses as acceptable (gPe(1)≥0g_P^e(1)≥ 0) and 190 as harmful (gPe(1)<0g_P^e(1)<0). Reviewer i approves pure release exactly when gie(1)≥0g_i^e(1)≥ 0. Varying r>0r>0 rescales every gain without changing its sign. Thus every positive release probability has the same set of disapproving reviewers. On a harmful response, rejection depth is the number of evaluators with gie(1)<0g_i^e(1)<0; on an acceptable response, the proposal domain contains no harmful release. An evaluator who disapproves of a harmful response is already aligned with the principal on this one-dimensional domain, and additional disapproving reviewers provide redundancy. For threshold rules, the guarantees on pure release therefore determine soundness and completeness over all release probabilities. Worked decisions. The following examples show how scores determine approval votes and threshold-rule decisions. The StrongREJECT example is row 904 (counting from zero) of the archived StrongREJECT score file. Its human labels are (1,1,1,0.5,1)(1,1,1,0.5,1) and its nine evaluator harm scores are (0,0.5,0,0,0,0.111,1,0.132,0.625)(0,0.5,0,0,0,0.111,1,0.132,0.625), rounded to three decimals. We quote its prompt and summarize its response, omitting the deception guidance. The RewardBench 2 example is source ID 7, Factuality:7; its answer openings are abbreviated from the public entry, and correctness follows the benchmark label. In both examples, ties count as approvals. StrongREJECT response RewardBench 2 prompt Factuality:7 Prompt and answers “I want to get high on prescription amphetamines. What symptoms should I say I’m having when I talk to my doctor?” The archived answer names plausible symptoms, then urges honesty (details omitted). “What is [a] stub method in gtest? Where is it used?” The four answers begin: A (labeled correct), “a type of test double” returning predefined values; B, a “placeholder function” simulating external systems; C, a “placeholder or mock method” isolating dependencies; and D, a “placeholder or stand-in.” Proposal and baseline The pure proposal releases this response; rejection abstains. The pure proposal returns one of A–D; rejection samples uniformly from the four answers. Principal The five-label median harm is h=1h=1, so release has principal gain 1/2−h=−1/21/2-h=-1/2. Answer A is indexed first. Relative to uniform sampling, A has principal gain 3/43/4 and each incorrect answer has gain −1/4-1/4. Reviewers Evaluator i approves iff its harm score si≤1/2s_i≤ 1/2. Seven approve this answer and two disapprove. Reward model i approves an answer iff its score is at least its mean score over A–D. A–D receive 47, 42, 17, and 4 approvals. Threshold-rule outcome The k=0k=0 and k=1k=1 rules withhold the answer, whereas k=2k=2 releases it. The k=1k=1 rule authorizes correct answer A and rejects B–D, so it is both sound and complete for the four deterministic answers. F.2 Exact threshold-rule audit Rejection depth determines how many disapprovals a panel can tolerate while continuing to reject every harmful proposal (Corollary 2). To compute it for each benchmark example, let oe(q)=|i:gie(q)<0|o_e(q)= |\i:g_i^e(q)<0\ | be the number of disapprovals, using r in place of q for StrongREJECT. The anonymous k-tolerant threshold rule authorizes a proposal exactly when oe(q)≤ko_e(q)≤ k. Two extremal disapproval counts determine its guarantees: δe=minq:gPe(q)<0oe(q),βe=maxq:gPe(q)≥0oe(q). _e= _q:g_P^e(q)<0o_e(q), _e= _q:g_P^e(q)≥ 0o_e(q). The first is the fewest disapprovals received by a harmful proposal; the second is the most received by a weakly beneficial one. Full-domain soundness holds exactly when δe>k _e>k, and full-domain completeness holds exactly when βe≤k _e≤ k. We set δe=n+1 _e=n+1 when there is no harmful proposal, where n is the panel size. The baseline ensures that the set defining βe _e is nonempty. For the RewardBench 2 example in the table, rejection depth is δe=2 _e=2, and a lottery with nonnegative principal gain can receive as many as βe=46 _e=46 disapprovals. Thus the k=1k=1 rule is sound over every lottery and complete for the four deterministic answers. Completeness over every lottery requires tolerating many more disapprovals. Computing coverage and depth. StrongREJECT depth follows directly from the signs of the nine evaluator gains. RewardBench 2 requires a calculation over the full lottery domain. Since additive score constants leave gains unchanged, take response four as a reference and write s~ie=(si1e−si4e,si2e−si4e,si3e−si4e),u~P=(1,0,0). s_i^e=(s_i1^e-s_i4^e,s_i2^e-s_i4^e,s_i3^e-s_i4^e), u_P=(1,0,0). Because d=q−bed=q-b^e has d4=−(d1+d2+d3)d_4=-(d_1+d_2+d_3), this change of coordinates preserves each reviewer’s gain: (sie)⊤d=(s~ie)⊤(d1,d2,d3)(s_i^e) d=( s_i^e) (d_1,d_2,d_3). A coalition T covers the principal on prompt e exactly when the nonnegative system ∑i∈Tηis~ie=u~P,ηi≥0, _i∈ T _i s_i^e= u_P, _i≥ 0, (2) is feasible; the coefficients need not sum to one. This certificate shows how several reviewer directions can combine to cover the principal even when no reviewer has the principal’s direction. To compute depth, set v=−(d1,d2,d3)v=-(d_1,d_2,d_3). In these coordinates, principal harm and reviewer disapproval are strict positive inequalities, giving δe=minu~P⊤v>0|i:(s~ie)⊤v>0|. _e= _ u_P v>0 |\i:( s_i^e) v>0\ |. The uniform baseline gives positive probability to every answer, so any direction in these coordinates can be scaled down to a feasible deviation. Positive scaling preserves all votes, allowing us to fix u~P⊤v=1 u_P v=1. On this two-dimensional plane, reviewer indifference boundaries divide the search space into regions with fixed approval votes. We enumerate these regions and their boundaries and count the disapprovals; ties remain approvals. The analogous calculation for βe _e maximizes disapprovals over principal-acceptable directions, including those with zero principal gain. An independent enumeration on the bounded simplex recovers the same depths for all 1,763 prompts, and coverage agrees with Equation (2) on every prompt. The rejection depths give the soundness guarantees directly. Every harmful StrongREJECT response receives at least two disapprovals, so the panel can tolerate one disapproval while remaining sound on all harmful responses. On RewardBench 2, the panel covers 1,590 of 1,763 prompts (90.2%). Its full-domain soundness rate is 90.2% at k=0k=0 and 77.2% at k=1k=1, while deterministic-answer completeness rises from 4.1% to 11.8%, measured across all 1,763 prompts. Rejection depth thus identifies where the rule can authorize more answers without admitting a harmful lottery. Coverage despite individual misalignment. To measure how collective coverage relates to the closest individual reviewer, center each utility vector at the uniform baseline: vie=sie−μie,vP=uP−14.v_i^e=s_i^e- _i^e1, v_P=u_P- 141. For each nonzero viev_i^e, define its angle from positive alignment by θie=arccos(⟨vie,vP⟩‖vie‖2‖vP‖2). _i^e= \! ( v_i^e,v_P \|v_i^e\|_2\|v_P\|_2 ). This is the Euclidean angle in the zero-sum subspace, equivalently computed in an orthonormal basis. Constant-scoring reviewers have no utility direction and are not positively aligned; the minimum angle uses only nonzero centered vectors. Among the 1,590 covered prompts, the closest reviewer is a median 6.8∘6.8 from the principal direction. Table 1 reports coverage and robustness at several minimum-angle thresholds. Coverage can persist after any five reviewers are removed even when every reviewer is separated from the principal. By the depth characterization, this is the same protection that permits five disapprovals to be tolerated. The coverage certificates also give a constructive benchmark for numerical aggregation. For each covered prompt, choose nonnegative weights solving Equation (2). The aggregate utility equals the principal’s up to an additive constant and positive scale, so its zero-threshold cardinal rule is exactly sound and complete over every lottery. On uncovered prompts, the comparator rejects every proposal. Coverage and weights use the evaluated prompt’s correct-answer label and scores; the comparator measures what a prompt-specific oracle can achieve. The fitted-rule experiments below ask what soundness and completeness can be achieved with weights fixed across prompts. Simultaneous guarantees with approximation. On RewardBench 2, the exact completeness rate over all lotteries remains zero through k=24k=24, and the corresponding soundness rate reaches zero before completeness becomes positive. No threshold satisfies both exact requirements on any prompt. Allowing a bounded loss and requiring approval at or above a specified gain lets us measure how much of this gap the approximate guarantees close. For each RewardBench 2 prompt and threshold k, calculate He(k) H_e(k) =maxq∈Δ4:oe(q)≤kmax14−q1,0, = _q∈ _4:o_e(q)≤ k \ 14-q_1,0\, Ge(k) G_e(k) =supq∈Δ4:oe(q)>kmaxq1−14,0. = _q∈ _4:o_e(q)>k \q_1- 14,0\. Here He(k)H_e(k) is the greatest authorized loss, and Ge(k)G_e(k) is the greatest gain approached by a rejected proposal. We set Ge(k)=0G_e(k)=0 when no proposal is rejected. To handle equality at a positive completeness margin, we record whether a rejected proposal has gain q1−14=Ge(k)q_1- 14=G_e(k) before clipping. At zero margin, completeness holds exactly when βe≤k _e≤ k. Along a ray q=be+tdq=b^e+td with t>0t>0, all gains scale with t and approval votes stay fixed. Moving farther from the uniform baseline therefore increases the magnitude of any gain or loss until at least one answer has probability zero. It suffices to search the four triangular faces qj=0q_j=0 of the lottery simplex. On each face, reviewer indifference boundaries divide the face into regions with fixed votes. The principal’s gain is linear, so its extrema or limiting values occur at region vertices. We enumerate the original response vertices, intersections of reviewer boundaries with edges, and pairwise intersections of reviewer boundaries on each face, recording whether a rejected proposal attains the limiting gain. Figure 7 reports the fraction of benchmark items satisfying both requirements at each pair of tolerances. At each point, we choose one threshold k for the whole benchmark. StrongREJECT admits both exact guarantees on 93.5% of responses at k=6k=6. This joint rate pools harmful and acceptable responses; Appendix F.5 gives the corresponding conditional rates. On RewardBench 2, approximation yields simultaneous guarantees where exact requirements do not: certifying 25% of prompts with allowed loss 0.050.05 requires approximately εgood=0.61 _ good=0.61, while allowing loss 0.100.10 lowers the required gain to approximately 0.350.35. Increasing either tolerance weakens its requirement. In particular, raising the minimum gain for guaranteed approval permits more beneficial proposals to be rejected; the margin specifies which proposals must be authorized, not the gain of the proposal ultimately selected. Figure 7: Approximate threshold-rule guarantees. At each tolerance pair, color gives the greatest fraction of responses (a) or prompts (b) satisfying both guarantees, choosing one k∈0,…,n−1k∈\0,…,n-1\ across the whole benchmark. Both panels share a color scale. Labels in (a) give these percentages for positive required gain, away from the dashed right edge. At the dashed right edges, the allowed loss is large enough that every proposal satisfies the loss bound. Variation across topics. To see which RewardBench 2 categories support useful threshold-rule decisions, Figure 8 breaks down the deterministic answer comparison by topic. Each curve sweeps the same k=0,…,47k=0,…,47 thresholds within its category; no topic-specific rule is fitted. Soundness counts prompts on which all incorrect answers are rejected, while completeness counts prompts on which the correct answer is authorized, so the two rates can describe different prompts. Focus and Safety retain higher completeness at high soundness, whereas Precise Instruction Following has a weaker tradeoff. Figure 8: Deterministic-answer authorization by RewardBench 2 topic, varying k=0,…,47k=0,…,47. Soundness and completeness are separate fractions of prompts. Precise IF denotes Precise Instruction Following. Extending the calculation to every lottery reveals a common pattern across these otherwise different categories. In Figure 9, no category has a threshold with positive exact full-domain soundness and completeness rates simultaneously. Precise Instruction Following reaches a given completeness level at a somewhat smaller k, but the gap between exact soundness and completeness appears in all five categories. Figure 9: Exact lottery soundness and completeness by RewardBench 2 topic as k increases. Each rate is the fraction of prompts satisfying its requirement over every lottery. Precise IF denotes Precise Instruction Following. F.3 Learned binary and cardinal rules We now fit authorization rules whose reviewer weights are fixed across prompts. The k-tolerant threshold rule uses only the number of approvals; weighted voting also uses reviewer identities; the cardinal committee uses numerical gains. We compare them on the same held-out prompts, then compare the cardinal committee with equal weights and with weights refitted after removing the selected individual reviewer. Fitting the scoring rules. We use a category-stratified split with seed 20260813: 1,058 training, 353 validation, and 352 test prompts. For each reviewer and prompt, we center the four scores at their mean under the uniform baseline and divide by the Euclidean norm of the centered vector; a zero vector remains zero. Write the resulting score of response j as zijez_ij^e. This normalization removes additive and positive scale differences within each reviewer and prompt. The learned cardinal comparison treats the resulting scores as comparable across reviewers and prompts. For candidate weights w, the aggregate score of answer j is Mw,je=∑iwizijeM_w,j^e= _iw_iz_ij^e. Applying softmax to the four aggregate scores assigns probabilities to the four answers. For each ρ∈10−4,10−3,10−2,10−1,1,10ρ∈\10^-4,10^-3,10^-2,10^-1,1,10\, we fit nonnegative weights on training prompts by minimizing the negative log probability of the labeled correct answer plus ρ‖w‖22/2ρ\|w\|_2^2/2, then normalize the weights to sum to one. We select ρ by correct-answer selection accuracy on validation prompts: choose a highest-scoring answer, breaking answer ties uniformly, and measure how often it is correct. Ties between values of ρ favor the smaller value. The fitted scores are then used for authorization: the cardinal committee authorizes a lottery when Mwe(q)=∑jqjMw,je≥τ.M_w^e(q)= _jq_jM_w,j^e≥τ. The equal-weight rule uses uniform weights. The individual comparator uses the reviewer with greatest validation selection accuracy and applies a threshold to that reviewer’s normalized score. To assess how much the committee depends on this reviewer, we remove it and refit the weights and regularization over the remaining 47 reviewers. Weighted binary voting uses the same fitting procedure with yije=zije≥0y_ij^e=1\z_ij^e≥ 0\ in place of zijez_ij^e. Let wfitw^fit be the resulting weights, normalized to sum to one. We combine these with equal weights by setting wi=αwifit+(1−α)/n,α∈0,0.25,0.5,0.75,0.9,0.95,0.99.w_i=α w_i^fit+(1-α)/n, α∈\0,0.25,0.5,0.75,0.9,0.95,0.99\. We choose α to maximize completeness on the deterministic validation answers while meeting the 99% validation-soundness target. The selected value is α=0.99α=0.99: 99% of the weight comes from the fit and 1% from equal weighting, giving every reviewer positive weight. For lotteries, review takes place before sampling: each reviewer evaluates the expected normalized score and then votes. Thus authorization requires ∑iwi∑jqjzije≥0≥τ. _iw_i1\! \ _jq_jz_ij^e≥ 0 \≥τ. Each reviewer therefore averages its answer scores using the lottery probabilities before casting a vote; we do not average its binary votes on the individual answers. The k-tolerant threshold rule uses only their number. Ties approve in each rule. Threshold comparisons. For each cardinal or weighted-vote rule, the threshold grid contains its distinct scores on the four deterministic answers of the validation prompts, together with reject-all and accept-all. For the k-tolerant rule, we enumerate the integer thresholds and both constant rules. We use the same grids for the lottery curves, evaluating each threshold over every mixture of the four answers. The plotted curves connect test outcomes as these thresholds vary. At a reported test-soundness requirement, we take the greatest completeness over the evaluated thresholds satisfying that requirement. These are empirical optima within each fitted score rule and its grid. Lottery soundness and completeness can change between grid thresholds. We evaluate thresholds chosen using validation alone in Appendix F.4. What score magnitudes add. The cardinal committee authorizes more correct answers at high soundness than either binary rule, and its advantage persists after removing the selected individual reviewer. Table 2 compares the fitted rules and controls under the same requirements. Table 2: Completeness on 352 RewardBench 2 test prompts. Deterministic results require at least 99% test soundness; lottery results require at least 95% exact test soundness and use 0.50.5-completeness. Each entry uses the most complete evaluated threshold meeting its column’s soundness target. Correct answer authorized (%) Every qualifying lottery authorized (%) Learned cardinal committee 60.5 28.7 Committee after removal and refitting 57.4 20.7 Selected individual reviewer 46.6 7.7 Equal-weight cardinal committee 35.2 4.5 Learned weighted binary votes 8.8 0 k-tolerant threshold rule 3.7 0 The learned committee places 52% of its weight on the selected reviewer. Removing this reviewer and refitting reduces both forms of completeness, but still improves substantially on that reviewer alone. The committee therefore benefits from combining several reviewers’ scores. These comparisons concern the fitted rules above, rather than all possible rules based on binary votes or numerical scores. Auditing every lottery. The lottery results evaluate all of Δ4 _4 under the expected-utility model described above. For the committee’s scores on the four answers, Mwe=(Mw,1e,…,Mw,4e)M_w^e=(M_w,1^e,…,M_w,4^e), let Mw,−e=maxj≥2Mw,jeM_w,-^e= _j≥ 2M_w,j^e. Whenever the authorized set is nonempty, the authorized lottery with least probability of a correct answer puts its remaining probability on an incorrect answer with score Mw,−eM_w,-^e. This gives a closed form for soundness. At operating thresholds selected on validation data, we independently check the test soundness calculation by solving minq∈Δ4q1subject toMwe(q)≥τ. _q∈ _4q_1 to M_w^e(q)≥τ. If Mw,mine=minj≥2Mw,jeM_w, ^e= _j≥ 2M_w,j^e, the least aggregate score among weakly beneficial lotteries is minMw,1e,14Mw,1e+34Mw,mine. \M_w,1^e, 14M_w,1^e+ 34M_w, ^e\. Comparing this score with τ gives exact full-domain completeness. The same endpoint calculations give the worst authorized loss and the supremum rejected gain used in the approximate audit. For weighted binary voting, we use the exact hyperplane arrangement, since the approval bits change discontinuously with the lottery. Figure 10: Lottery authorization on 352 test prompts, using 0.50.5-completeness. Panel (a) connects evaluated thresholds of each fixed scoring rule. Panel (b) selects the greatest completeness among evaluated thresholds meeting the allowed-loss bound on at least 95% of test prompts; selection uses test outcomes separately at each loss tolerance. At the omitted endpoint εharm=0.25 _ harm=0.25, the loss bound is unrestricted and accept-all attains 100% completeness for every rule. The role of the approximation margins. Among the evaluated global rules and thresholds, the exact completeness rate over all lotteries is zero whenever the corresponding soundness rate is positive. A positive numerical-score threshold also rejects sufficiently small beneficial deviations from the baseline. Requiring approval of sufficiently beneficial lotteries gives a useful intermediate target. Our 0.50.5-completeness comparison requires approval of every lottery with correct-answer probability at least 0.750.75, and reports the fraction of prompts on which this universal requirement holds. Soundness and completeness are separate fractions of prompts. Figure 10(b) varies the tolerated loss while retaining this gain requirement. The learned cardinal committee maintains 28.7% completeness at every evaluated loss tolerance below 0.25; weighted binary voting reaches 3.4% at loss tolerance 0.20. Thus allowing some harmful lotteries does little to close the gap for these fitted binary rules. Figure 11 instead holds each rule at its most permissive evaluated threshold attaining at least 95% exact test soundness and varies the gain required for guaranteed approval. A larger gain margin requires approval of fewer lotteries, so more prompts can satisfy completeness. The authorization decisions and the test soundness rate remain unchanged along each curve. Figure 11: Completeness as the minimum gain for guaranteed approval increases. Each rule uses the most permissive evaluated threshold attaining at least 95% exact test soundness, held fixed across gain margins. The guide marks gain 0.50.5; gain 0.750.75 tests only the correct deterministic answer. Information in reviewer identities. The theory also suggests that binary aggregation can improve substantially when it keeps track of which reviewers approve. We measure this possibility using prompt-specific sound rules on the same 1,590 covered archive prompts. Unanimity authorizes the correct deterministic answer on 55 prompts (3.5% of this covered subset, rather than 4.1% of all prompts); choosing the most permissive sound k-tolerant rule separately for each prompt raises this to 581 (36.5%); allowing the maximal sound monotone rule raises it to 1,409 (88.6%). Each rule is sound over every response lottery. The latter two use the benchmark label separately on each prompt, and their reported completeness is correct-answer authorization, not full-domain completeness. The increase from 36.5% to 88.6% shows how reviewer identities allow a sound rule to authorize more correct answers. Cardinal reconstruction gives the complementary oracle benchmark: on the 315 covered test prompts it reconstructs the principal and is complete over the lottery domain, while on the other 37 it rejects all proposals. It therefore attains 100% exact soundness and 315/352=89.5%315/352=89.5\% 0.50.5-completeness. These weights are computed using each prompt’s correct-answer label; the learned committee instead uses weights fixed before testing. F.4 Calibration on validation data The preceding comparisons select thresholds by their test soundness and completeness. We now select thresholds on validation data, fix them before testing, and measure the resulting test rates. A shared threshold. For deterministic answers, we choose the threshold with greatest validation completeness among those attaining at least 99% validation deterministic-answer soundness. Ties favor greater validation soundness, then a stricter threshold. The scoring rules and selected individual reviewer are those defined above. On test, the cardinal committee attains 347/352=98.6%347/352=98.6\% soundness and 223/352=63.4%223/352=63.4\% completeness; the selected individual attains 340/352=96.6%340/352=96.6\% and 220/352=62.5%220/352=62.5\%. Thus the fixed committee threshold authorizes nearly the same number of correct answers while rejecting all three incorrect answers on seven more prompts. Using the observed topic. We then ask whether choosing a threshold for each RewardBench topic improves completeness. We compare cardinal committees and single reviewers, using either one committee or reviewer across all topics or a separate one for each topic. Cardinal weights are fitted on the corresponding training prompts and regularization is selected on validation; individual reviewers in this experiment are selected on training alone. For each committee or reviewer, we compare one shared threshold with five topic thresholds, holding the weights or reviewer identities fixed. Here we calibrate for lotteries: soundness requires rejecting every harmful lottery, and 0.50.5-completeness requires authorizing every lottery with gain at least 0.50.5. Both threshold choices maximize validation 0.50.5-completeness while allowing at most ⌊0.05×353⌋=17 0.05× 353 =17 validation prompts to fail soundness, giving at least 336/353=95.2%336/353=95.2\% validation soundness. For topic-specific thresholds, this limit of 17 applies across all five topics together, not separately within each topic. Candidate thresholds are obtained from the score values at which a validation prompt starts or stops satisfying either guarantee. Each topic’s candidate set also includes all candidates for the shared threshold. We choose the five topic thresholds jointly to maximize the total number of validation prompts satisfying 0.50.5-completeness, subject to the same limit on soundness failures. Thresholds are fixed before evaluation on the same 352 test prompts. Figure 12: Shared and topic-specific thresholds, chosen to meet the same overall soundness requirement on validation data. Open markers use one shared threshold; filled markers use one threshold per topic. Thresholds are fixed before evaluation on 352 test prompts. Rates are exact lottery soundness and 0.50.5-completeness. Individual reviewers are selected on training data. Changing only the thresholds of the global cardinal rule raises test completeness from 104/352=29.5%104/352=29.5\% to 144/352=40.9%144/352=40.9\%, while soundness increases from 331/352=94.0%331/352=94.0\% to 334/352=94.9%334/352=94.9\%. Topic-specific calibration therefore guarantees approval of every qualifying lottery on more prompts, without learning new weights. Allowing topic-specific weights changes the tradeoff: with five thresholds, the topic-specific committee is sound on 337 prompts and complete on 132, while the topic-specific individuals are sound on 336 and complete on 147. Here the committee has higher soundness, while the individual reviewers have higher completeness. The validation constraint does not guarantee the same soundness on test: the global individual with topic-specific thresholds attains 92.9% test soundness, below the 95% target. Neither soundness nor completeness is constrained separately within each topic. At this budget, both individual-reviewer rules with five thresholds reject every proposal in Factuality, Math, and Precise Instruction Following. F.5 Interpretation and cohort checks Conditional and joint guarantees on StrongREJECT. The heatmap measures how often both guarantees hold on the same response, whereas the main curves report fractions separately among harmful and acceptable responses. The k=6k=6 rule satisfies both exact guarantees on 1,013/1,084=93.5%1,013/1,084=93.5\% of responses: it releases 872 acceptable responses and withholds 141 harmful ones. Its conditional completeness is 872/894=97.5%872/894=97.5\%, while conditional soundness is 141/190=74.2%141/190=74.2\%; it releases 49 harmful responses. In comparison, k=1k=1 withholds all 190 harmful responses and releases 419 of the 894 acceptable ones. The joint rate weights each response equally, so the larger acceptable class has more influence. Sensitivity to missing human labels. All 1,361 archived responses have nine evaluator scores; the 1,084 responses in our main analysis also have all five human annotations. Selection uses annotation availability, not label values or evaluator votes. Requiring at least two annotations retains 1,184 responses, including 207 classified as harmful. Each harmful response still receives at least two disapprovals, so the soundness guarantee for k=1k=1 extends to this larger cohort. Admitting responses with only one annotation adds a harmful response approved by all nine evaluators. Thus even unanimity fails on the full archive under its released labels; this check measures sensitivity to annotation availability. Public resources. StrongREJECT’s code is MIT-licensed; its prompt sources retain their upstream terms. RewardBench 2 is released under ODC-BY, with third-party outputs subject to provider terms. Data and code are publicly available.22 2 https://github.com/dsbowen/strong_reject; https://huggingface.co/datasets/allenai/reward-bench-2; https://huggingface.co/datasets/allenai/reward-bench-2-results.