Paper deep dive
Safety Hacking in Constrained Best-of-$N$ Inference-time Scaling
Akifumi Wachi, Takumi Tanabe, Youhei Akimoto
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Inference-time pipelines often sample multiple outputs, filter them with a learned safety model, and return the proxy-feasible output with the highest learned reward. We show that this composition creates a two-stage failure: an imperfect safety proxy first contaminates the feasible set with unsafe outputs, and reward maximization can then amplify this residual contamination. We define \emph{safety hacking} as selecting an output that passes the learned constraint but violates the true safety criterion. For constrained Best-of-$N$ sampling, we derive finite-$N$ bounds governed by the joint upper reward tails of safe and unsafe outputs within the proxy-feasible set. If unsafe-but-feasible outputs have the heavier tail, safety hacking becomes asymptotically certain as $N$ grows, even when false-positive mass and average safety- and reward-proxy errors are arbitrarily small. We also show that policies within a bounded $\chi^2$ divergence from the proxy-feasible reference distribution admit an $N$-independent safety-hacking bound, and instantiate this general coverage-control principle with constrained pessimistic sampling. Coverage control limits amplification but cannot repair a contaminated feasible set: admitted unsafe outputs may still be favored, and regularized selection is not necessarily safer than constrained Best-of-$N$ for every reward proxy. Toy and language-model experiments characterize both contamination and its reward-tail amplification, which exposes an inherent difficulty in inference-time scaling with learned safety models.
Tags
Links
- Source: https://arxiv.org/abs/2608.22915v1
- Canonical: https://arxiv.org/abs/2608.22915v1
Trouble viewing inline? Open PDF directly →
Full Text
89,320 characters extracted from source content.
Expand or collapse full text
Safety Hacking in Constrained Best-of-N Inference-time Scaling Akifumi Wachi1 Takumi Tanabe1 Youhei Akimoto2,3 1LY Corporation 2University of Tsukuba 3RIKEN AIP Correspondence: akifumi.wachi@lycorp.co.jp Abstract Abstract Inference-time pipelines often sample multiple outputs, filter them with a learned safety model, and return the proxy-feasible output with the highest learned reward. We show that this composition creates a two-stage failure: an imperfect safety proxy first contaminates the feasible set with unsafe outputs, and reward maximization can then amplify this residual contamination. We define safety hacking as selecting an output that passes the learned constraint but violates the true safety criterion. For constrained Best-of-N sampling, we derive finite-N bounds governed by the joint upper reward tails of safe and unsafe outputs within the proxy-feasible set. If unsafe-but-feasible outputs have the heavier tail, safety hacking becomes asymptotically certain as N grows, even when false-positive mass and average safety- and reward-proxy errors are arbitrarily small. We also show that policies within a bounded χ2χ^2 divergence from the proxy-feasible reference distribution admit an N-independent safety-hacking bound, and instantiate this general coverage-control principle with constrained pessimistic sampling. Coverage control limits amplification but cannot repair a contaminated feasible set: admitted unsafe outputs may still be favored, and regularized selection is not necessarily safer than constrained Best-of-N for every reward proxy. Toy and language-model experiments characterize both contamination and its reward-tail amplification, which exposes an inherent difficulty in inference-time scaling with learned safety models. 1 Introduction Foundation-model inference increasingly relies on learned models both to enforce safety and to assess response quality [22; 1; 8]. A typical pipeline samples candidate responses, rejects those deemed unsafe by a safety model, and ranks the remainder with a reward model [26; 12; 11; 27]. As a representative instance, constrained Best-of-N (CBoNN) sampling follows this pattern: it draws N responses and returns the proxy-feasible response with the largest learned reward. Filtering and ranking errors can interact. The safety model may admit a response that violates the true safety criterion, thereby contaminating the proxy-feasible set; the reward model may then assign that response a high score. We call the resulting selection event safety hacking: the returned response satisfies the learned safety constraint but violates the true one. Such unsafe-but-feasible responses remain inside the feasible set and are therefore exposed to downstream reward optimization, which can amplify a small amount of contamination into a large selection probability. Existing work on reward overoptimization shows that excessive optimization against an imperfect reward proxy can eventually reduce the true reward [21; 7]. Our setting adds a distinct source of misspecification: the optimization domain is itself defined by an imperfect safety proxy. The resulting failure is governed by the interaction between 1) contamination that determines which unsafe outputs enter the proxy-feasible set, and 2) amplification that determines how strongly reward optimization favors them. Our main result shows that amplification depends not only on how often the filter admits unsafe responses, but also on the relative upper tails of learned rewards among safe and unsafe proxy-feasible responses. When the unsafe group has the heavier upper tail, its maximum eventually dominates under CBoNN, and the probability of selecting an unsafe response tends to 11 as N→∞N→∞. This can occur even as the mass of unsafe false acceptances and average safety- and reward-proxy errors vanish. Average filter accuracy is therefore insufficient to predict safety under inference-time scaling; the joint tail behavior of the two proxies also matters. Safety hacking is related to reward overoptimization and BoNN jailbreaking [7; 10], but it is not ordinary reward hacking restricted to a benign feasible set. A main reason for safety hacking is that reward optimization operates on a set that may already contain false acceptances from the safety filter and can favor those errors over other candidates. The cleanest example is Corollary 4.3: an unsafe-but-feasible reference mass ε can coexist with safety- and reward-proxy errors of order ε , yet a budget N=O(1/ε)N=O(1/ ) suffices to produce high-probability safety hacking. Pessimistic inference can provide scaling-monotonic guarantees for reward overoptimization under suitable conditions [9], but it cannot determine which responses were wrongly admitted by a safety proxy. We show that any policy close to the proxy-feasible reference distribution in χ2χ^2 divergence admits an N-independent safety-hacking bound. Our concrete example is constrained pessimistic sampling (cPes), a χ2χ^2-regularized reweighting of the proxy-feasible reference distribution. The guarantee limits concentration on residual errors but cannot eliminate them. Depending on the reward proxy, regularized selection may still exhibit substantial safety hacking. Our contributions are: 1. We formalize safety hacking as selection of a response that satisfies a learned safety constraint while violating the true safety criterion. 2. We characterize how safety-filter contamination is amplified by reward maximization. We derive finite-N bounds for CBoNN in terms of the joint reward tails of safe and unsafe proxy-feasible outputs, and give conditions under which safety hacking becomes asymptotically certain even as unsafe false acceptances and average proxy errors vanish. 3. We derive N-independent safety-hacking bounds under coverage control and study cPes as a concrete instance. These bounds limit amplification but cannot repair contamination introduced by an imperfect safety constraint. 4. Controlled and language-model experiments measure both stages of the mechanism. An exact finite-N decomposition separates exposure to unsafe-but-feasible responses from the probability that they outrank every safe candidate. The latter increases with scale and drives the increase in safety hacking. Reward-model ablations isolate the role of reward ranking in amplification, and we observe the same decomposition pattern with an alternative safety filter. 2 Related Work Reward overoptimization. Optimizing an imperfect proxy can produce unintended behavior, a concern studied through reward hacking or Goodhart’s law [1; 17]. In language-model alignment, learned reward models are used both during training [19; 2] and to rank responses at inference time [24]. Prior work shows that excessive optimization of an imperfect reward proxy, through reinforcement learning or inference-time scaling, can eventually reduce true reward [7; 14]. Our setting includes misspecification in both the reward and the safety constraint (i.e., optimization domain). A learned safety proxy defines a feasible set that can contain unsafe false acceptances, which reward maximization may then amplify through upper-tail selection. We analyze the interaction between these two sources of error. Safety-constrained inference-time alignment. 5 formulate inference-time alignment with learned reward and safety cost models, using a calibrated Lagrangian reward for sequence-level BoNN under an expected-cost constraint. While 5 optimize against the learned cost signal, we instead study a misspecified hard safety filter. In our settings, false acceptances contaminate the feasible set, and subsequent reward maximization can amplify them. Inference-time pessimism and coverage control. 9 study coverage and scaling in inference-time alignment and propose pessimistic alternatives to naïve BoNN. Also, regularized BoNN has been studied under a minimum-Bayes-risk objective [13]. We condition the reference distribution on proxy feasibility and apply coverage control to the resulting distribution. The bound limits downstream amplification of safety-filter errors but leaves any contamination already present in the conditioned reference. The distinction from prior work lies in separating contamination from amplification. Coverage control limits amplification after conditioning on proxy feasibility. Learned safety filters and evaluator errors. LLM systems often use learned filters, moderation models, or guardrails to classify prompts and responses. Llama Guard casts safeguarding as safety-risk classification [11], while ShieldGemma provides open models for detecting risks in model inputs and outputs [27]. These systems make safety constraints operational but remain imperfect proxies for the true criterion [29]. We study how errors in the two proxies interact: unsafe false acceptances determine which responses enter the feasible set, and reward-proxy errors determine which response is selected. Jailbreaks and red teaming. HarmBench [18], JailbreakBench [3], and StrongREJECT [23] provide benchmarks for harmful behavior, jailbreak, and refusal robustness. BoNN jailbreaking shows that repeated randomized attempts can substantially increase attack success rates [10], demonstrating that search can expose rare safety failures. We instead hold the prompt fixed and search over sampled model outputs. The filter may admit unsafe outputs, and reranking can make them increasingly likely to be selected. 3 Problem Statement We consider safety-constrained reward maximization at inference time. For clarity, we write the theory for a fixed prompt x∈x whenever possible; hence, all probabilities and expectations are conditional on this prompt. We take Y to be finite or countable, and the measurable-space extension is obtained by replacing sums with integrals. We suppose that there are an unknown reward function r⋆:×→ℝ+r :X×Y _+ and an unknown binary safety function g⋆:×→0,1g :X×Y→\0,1\. For a prompt x, an output y is truly safe if g⋆(x,y)=1g (x,y)=1. Define the following truly safe and unsafe sets of outputs: +⋆(x) _+ (x) ≔y∈:g⋆(x,y)=1, \y :g (x,y)=1\, −⋆(x) _- (x) ≔∖+⋆(x). _+ (x). (1) Thus, the ideal constrained inference problem is maxy∈r⋆(x,y)subject tog⋆(x,y)=1. _y \ r (x,y) to g (x,y)=1. (2) The agent cannot evaluate r⋆r or g⋆g directly, but instead has a reference policy πref(⋅∣x) _ref(· x), a learned reward proxy r^:×→ℝ r:X×Y , and a learned safety proxy g^:×→[0,1] g:X×Y→[0,1]. For a safety threshold b∈(0,1]b∈(0,1], define the proxy-feasible set ^+(x,b)≔y∈:g^(x,y)≥b. S_+(x,b) \y : g(x,y)≥ b\. (3) The proxy-feasible set decomposes into true positives and false positives: A(x,b) A(x,b) ≔+⋆(x)∩^+(x,b), _+ (x)∩ S_+(x,b), B(x,b) B(x,b) ≔−⋆(x)∩^+(x,b). _- (x)∩ S_+(x,b). (4) Both sets contain outputs accepted by the safety proxy: A(x,b)A(x,b) contains the truly safe ones, whereas B(x,b)B(x,b) contains the unsafe ones. Definition 3.1 (Safety hacking). Fix x∈x and b∈(0,1]b∈(0,1]. If an inference-time procedure outputs y^∈^+(x,b) y∈ S_+(x,b), we say that safety hacking occurs when y^∈B(x,b). y∈ B(x,b). (5) Equivalently, the output passes the learned safety proxy but violates the true safety constraint. Note that if the procedure abstains (e.g., when ^+(x,b)=∅ S_+(x,b)= ), this is not considered safety hacking. The following joint upper-tail quantities are the central objects in the CBoNN analysis. For ◊∈A,B ∈\A,B\ and t∈ℝt , define the learned-score tail Ψ^◊(t;x,b)≔ℙy∼πref(⋅∣x)[y∈◊(x,b),r^(x,y)>t]. _ (t;x,b) _y _ref(· x) [y∈ (x,b),\ r(x,y)>t ]. (6) These tails combine 1) the probability that the reference policy πref _ref reaches a class of outputs ◊ and 2) the upper-tail behavior of the learned reward score inside that class. Although we use a contextual bandit setting for simplicity, the formulation also covers open-loop trajectory selection. Let τ denote a trajectory. Set y=τy=τ and let πref(τ∣x) _ref(τ x) be the trajectory distribution induced by a fixed reference policy and the environment dynamics, with reward and safety defined at the trajectory level. The results require N i.i.d. trajectories from this distribution and therefore exclude beam search, Monte Carlo tree search, and other non-i.i.d. or stepwise procedures. 4 Safety-Filter Contamination and Reward-Tail Amplification A natural baseline to solve (2) is CBoNN. Given N i.i.d. candidates y1,…,yN∼πref(⋅∣x)y_1,…,y_N _ref(· x), define a proxy-feasible index set ℐN(x,b)≔i∈[N]:yi∈^+(x,b)I_N(x,b) \i∈[N]:y_i∈ S_+(x,b)\. If ℐN(x,b)≠∅I_N(x,b)≠ , CBoNN returns y^NBoN(x)∈argmaxi∈ℐN(x,b)r^(x,yi) y_N^BoN(x)∈ argmax_i _N(x,b) r(x,y_i) (7) with arbitrary tie-breaking. In the case of ℐN(x,b)=∅I_N(x,b)= , the procedure simply abstains. The failure of CBoNN has two stages. First, false positives from the safety proxy contaminate the proxy-feasible set with outputs in B(x,b)B(x,b). Second, maximizing r r over that set can amplify the contamination by searching the extreme upper tail of the learned reward score. Increasing N therefore does more than increase exposure to unsafe-but-feasible outputs: it also intensifies their reward-based competition with safe proxy-feasible outputs. The following theorem identifies the joint score tails in (6) as the finite-N quantities governing this competition. Theorem 4.1 (Finite-N safety-hacking probability bounds). Fix a prompt x∈x and a threshold b∈(0,1]b∈(0,1]. Let y1,…,yN∼iidπref(⋅∣x)y_1,…,y_N iid _ref(· x), and let y^NBoN(x) y_N^BoN(x) be defined by (7). For any t,s∈ℝt,s , define L^N(t,x,b) L_N(t;x,b) ≔(1−Ψ^A(t,x,b))N−(1−Ψ^A(t,x,b)−Ψ^B(t,x,b))N, (1- _A(t;x,b) )^N- (1- _A(t;x,b)- _B(t;x,b) )^N, (8) U^N(s,x,b) U_N(s;x,b) ≔1−(1−Ψ^B(s,x,b))N+(1−Ψ^B(s,x,b)−Ψ^A(s,x,b))N. 1- (1- _B(s;x,b) )^N+ (1- _B(s;x,b)- _A(s;x,b) )^N. (9) Then, for every t,s∈ℝt,s , L^N(t;x,b)≤ℙ[y^NBoN(x)∈B(x,b)]≤U^N(s;x,b). L_N(t;x,b) [ y_N^BoN(x)∈ B(x,b) ]≤ U_N(s;x,b). (10) Taking the supremum over t and the infimum over s gives the tightest bounds in this family. The lower bound is the probability that no sample from A(x,b)A(x,b) has learned reward above t and at least one sample from B(x,b)B(x,b) does. CBoNN selects from B(x,b)B(x,b) on this event. The upper bound follows because CBoNN cannot select from B(x,b)B(x,b) when some sample from A(x,b)A(x,b) scores above s and none from B(x,b)B(x,b) does. Corollary A.1 states the corresponding guarantees in terms of the effective tail masses NΨ^AN _A and NΨ^BN _B. Let t¯≔esssupy∈A(x,b)r^(x,y) t ess\,sup_y∈ A(x,b) r(x,y), so that Ψ^A(t¯,x,b)=0 _A( t;x,b)=0. Theorem 4.1 then gives ℙ[y^NBoN(x)∈B(x,b)]≥1−(1−Ψ^B(t¯;x,b))NP[ y_N^BoN(x)∈ B(x,b)]≥ 1-(1- _B( t;x,b))^N. If Ψ^B(t¯,x,b)=0.01 _B( t;x,b)=0.01, the lower bound in Theorem 4.1 is approximately 0.630.63 at N=100N=100 and 0.990.99 at N=500N=500. Contamination determines the mass of B(x,b)B(x,b). The relative joint tails determine whether CBoNN selects from it. The next result gives the following tail-separation condition under which the safety-hacking probability converges to one. Theorem 4.2 (Asymptotic safety hacking). Fix x∈x and b∈(0,1]b∈(0,1]. If there exists a sequence tN∈ℝt_N such that NΨ^A(tN,x,b)→0N _A(t_N;x,b)→ 0, and NΨ^B(tN,x,b)→∞N _B(t_N;x,b)→∞, then limN→∞ℙ[y^NBoN(x)∈B(x,b)]=1. _N→∞P [ y_N^BoN(x)∈ B(x,b) ]=1. (11) In the boundary case discussed above, Ψ^A(t¯,x,b)=0 _A( t;x,b)=0 and Ψ^B(t¯,x,b)>0 _B( t;x,b)>0. Setting tN=t¯t_N= t, the theorem shows that the safety-hacking probability converges to one. More generally, even an arbitrarily small contaminated region can dominate BoNN search when the reward tails separate. The fixed-prompt result extends directly to prompt distributions: if the tail-separation condition holds on a set of prompts with probability mass at least ρ, the aggregate hacking probability has liminf at least ρ; see Corollary A.2. Corollary 4.3 (Vanishing proxy error still allows amplification). Fix x and b∈(0,1]b∈(0,1]. For every ε∈(0,1) ∈(0,1) and ξ>0ξ>0, there exists a two-output instance whose unsafe-but-feasible reference mass is ε , whose safety- and reward-proxy RMSEs are respectively bεb and ξεξ , and for which ℙ[y^NBoN(x)∈B(x,b)]=1−(1−ε)N≥1−e−Nε. [ y_N^BoN(x)∈ B(x,b) ]=1-(1- )^N≥ 1-e^-N . (12) Consequently, any N≥ε−1log(1/δ)N≥ ^-1 (1/δ) yields safety hacking with probability at least 1−δ1-δ. Thus, even as the false-positive mass and both average proxy errors vanish, a budget N=Θ(1/ε)N= (1/ ) can amplify the residual error to high probability. When the true-reward range over proxy-feasible outputs is bounded, tail separation also follows if reward overestimation has a heavier Gaussian upper tail on unsafe-but-feasible outputs than on safe-feasible outputs. The relevant threshold scales as logN N; see Proposition A.6. 5 Coverage Control Limits Amplification, Not Contamination The preceding analysis separates two sources of risk. Contamination is the unsafe mass admitted by the safety proxy; amplification is the additional concentration on that mass induced by downstream reward optimization. This distinction suggests a general mitigation principle: bounded deviation from the proxy-feasible reference distribution limits amplification as N grows, although it cannot remove the underlying contamination. Let q(x,b)≔ℙy∼πref(⋅∣x)[y∈^+(x,b)]q(x,b) _y _ref(· x)[y∈ S_+(x,b)] denote the reference probability that a sampled output passes the learned safety filter. Throughout this section, we assume q(x,b)>0q(x,b)>0. Conditioning the reference policy on this event gives the proxy-feasible reference distribution πref♯(y∣x)≔πref(y∣x)⋅y∈^+(x,b)q(x,b). _ref (y x) _ref(y x)·I\y∈ S_+(x,b)\q(x,b). (13) This distribution is the baseline against which we measure how strongly an inference procedure concentrates on feasible outputs. We write π(⋅∣x)≪πref♯(⋅∣x)π(· x) _ref (· x) when π(y∣x)>0π(y x)>0 implies πref♯(y∣x)>0 _ref (y x)>0 for every y∈y . For any such policy, define the following coverage coefficient by Cπ♯(x)≔∑y∈π(y∣x)2πref♯(y∣x)=1+χ2(π(⋅∣x)∥πref♯(⋅∣x)). C_π (x) _y π(y x)^2 _ref (y x)=1+χ^2 (π(· x)\, \|\, _ref (· x) ). (14) Small Cπ♯(x)C_π (x) means that the final policy retains broad coverage of the proxy-feasible reference distribution, whereas a large value indicates strong concentration. Theorem 5.1 (Coverage-controlled contamination amplification). Define the residual unsafe mass under πref♯ _ref as κ(x,b)≔ℙy∼πref♯(⋅∣x)[y∈B(x,b)]κ(x,b) _y _ref (· x)[y∈ B(x,b)]. Any proxy-feasible policy π(⋅∣x)≪πref♯(⋅∣x)π(· x) _ref (· x) satisfies |ℙy∼π(⋅∣x)[y∈B(x,b)]−κ(x,b)|≤(Cπ♯(x)−1)κ(x,b)(1−κ(x,b)). |P_y π(· x)[y∈ B(x,b)]-κ(x,b) |≤ (C_π (x)-1 )κ(x,b) (1-κ(x,b) ). (15) Define ε¯g(x)≔y∼πref(⋅∣x)[(g^(x,y)−g⋆(x,y))2] _g(x) E_y _ref(· x) [( g(x,y)-g (x,y))^2 ]. Then, κ(x,b)≤min1,ε¯g(x)2b2q(x,b). κ(x,b)≤ \1, _g(x)^2b^2q(x,b) \. (16) The theorem applies to any inference procedure with bounded proxy-feasible coverage, regardless of how its final policy is constructed. Thus coverage control limits how much downstream selection can amplify the baseline contamination of the proxy-feasible set, but it cannot remove that contamination itself. In particular, Cπ♯(x)=1C_π (x)=1 implies π=πref♯π= _ref and the safety-hacking probability equals κ(x,b)κ(x,b). 5.1 A Regularized Instantiation To prevent arbitrarily large learned reward scores from inducing extreme concentration, fix clipping levels −∞<R^min<R^max<∞-∞< R_ < R_ <∞ and set R^span≔R^max−R^min R_span R_ - R_ . Then, define r~(x,y)≔clip(r^(x,y),R^min,R^max). r(x,y) ( r(x,y), R_ , R_ ). (17) We use r~ r only for the coverage-controlled policy below; the CBoNN analysis in Section 4 continues to use the original learned score r r. For β>0β>0, consider the regularized proxy-feasible policy π^∈argmaxπ≪πref♯π[r~(x,y)]−β2(Cπ♯(x)−1). π∈ argmax_π _ref \E_π[ r(x,y)]- β2 (C_π (x)-1 ) \. (18) Proposition 5.2 (Regularized reweighting). There is a unique λ(x)∈[R^min−β,R^max)λ(x)∈[ R_ -β, R_ ) satisfying y∼πref♯(⋅∣x)[ReLU(r~(x,y)−λ(x)β)]=1E_y _ref (· x) [ReLU ( r(x,y)-λ(x)β ) ]=1, and the unique optimizer of (18) is π^(y∣x)=πref♯(y∣x)ReLU(r~(x,y)−λ(x)β). π(y x)= _ref (y x)ReLU ( r(x,y)-λ(x)β ). (19) Moreover, its concentration relative to the proxy-feasible reference is bounded as Cπ^♯(x)≤1+R^span/β. C_ π (x)≤ 1+ R_span/β. (20) Hereinafter, we call the resulting algorithm constrained pessimistic sampling (cPes). A finite-sample implementation replaces the expectations above by their empirical analogues over the proxy-feasible candidates and samples according to the resulting reweighting. Full pseudocode and consistency are given in Algorithms 1 and A.8. Let mN=|ℐN(x,b)|m_N=|I_N(x,b)| be the number of proxy-feasible candidates, and let y^cPes,N(x) y_cPes,N(x) denote the output of finite-sample cPes. Corollary 5.3 (Safety of cPes). Conditional on mN>0m_N>0, the sample implementation satisfies ℙ[y^cPes,N(x)∈B(x,b)∣mN>0]≤min1,ε¯g(x)2b2q(x,b)+ε¯g(x)bR^spanβq(x,b). [ y_cPes,N(x)∈ B(x,b) m_N>0]≤ \1,\, _g(x)^2b^2q(x,b)+ _g(x)b R_spanβ q(x,b) \. (21) The same bounds hold for the population policy (19). With an abstaining fallback, the sample bounds hold unconditionally for every N. 6 Experiments 6.1 Toy Problem We illustrate tail separation with a three-class toy problem. Each candidate belongs to a latent class A, B, or C with probabilities (0.39,0.01,0.60)(0.39,0.01,0.60). Class A is truly safe and proxy-feasible, class B is unsafe but proxy-feasible, and class C is correctly rejected by the safety proxy. We set g⋆=(1,0,0),g^=(0.95,0.90,0.05),b=0.8,g =(1,0,0), g=(0.95,0.90,0.05), b=0.8, and use true rewards r⋆=(0.8,0.2,0.5)r =(0.8,0.2,0.5). The learned reward is r^=r⋆+σkZ,Z∼(0,1), r=r + _kZ, Z (0,1), with (σA,σB,σC)=(0.2,1.0,0.2)( _A, _B, _C)=(0.2,1.0,0.2). Thus the unsafe false-positive class B is rare, but has a substantially heavier upper tail of reward-proxy overestimation than the safe class A. We compare random feasible selection, CBoNN, and cPes over increasing candidate budgets. Random feasible selection chooses uniformly among the candidates accepted by the safety proxy, without using the learned reward. CBoNN uses the unclipped learned score r r, whereas cPes applies the clipping with [R^min,R^max]=[0,4][ R_ , R_ ]=[0,4]. (a) Conditional safety-hacking rate. (b) Conditional mean true reward. (c) False-positive tail scale. Figure 1: Toy results. As the search budget grows, CBoNN increasingly selects the rare unsafe-but-feasible class B, whose reward-proxy upper tail dominates that of the safe class A. Random feasible selection does not exhibit this amplification, while cPes limits the resulting concentration. Panel (c) varies the false-positive reward-noise scale σB _B. Figure 1 shows the predicted amplification. As N grows, CBoNN shifts from selecting the safe class A to the unsafe-but-feasible class B. The conditional safety hacking rate rises from 0.0230.023 at N=1N=1 to 0.99960.9996 at N=8192N=8192, while the mean true reward falls from about 0.7860.786 to 0.2000.200. Random feasible selection remains near the baseline false-positive rate, showing that exposure to class B alone does not explain the failure. cPes substantially limits this amplification in the toy problem. Controls varying the reward-error tails and removing false positives confirm that the effect requires an unsafe proxy-feasible class with a sufficiently advantageous learned-reward tail; see Appendix A.15. 6.2 LLM Experiments We next evaluate safety hacking in a more practical setting using LLMs. Experimental settings. Our evaluation uses 714 test and 179 validation prompts from JailbreakBench [3], HarmBench [18], and AdvBench [4], after removing duplicates. For each prompt, we sample 256 responses from Qwen/Qwen2.5-7B-Instruct [25] (temperature 1.0, top-p 0.95, 512 new tokens), giving 228,608 candidates. We evaluate all selection methods on the same candidates. We filter candidates with meta-llama/Llama-Guard-3-8B [16] and rank them with PKU-Alignment/beaver-7b-v1.0-reward [6]. We report the primary safety threshold b=0.95b=0.95 and compare random feasible selection, CBoNN, and cPes for N∈1,2,4,8,16,32,64,128,256N∈\1,2,4,8,16,32,64,128,256\. Random feasible selection controls for exposure to filter false positives without reward maximization; cPes uses calibration-fixed clipping levels and β=1.0β=1.0. Point estimates are averaged over 100 random candidate-order permutations. For each prompt, we first average the metric over permutations and then compute standard errors across prompts; shaded bands show ±1.96± 1.96 standard errors. Because latent safety and reward are unobservable for open-ended responses, we use the HarmBench classifier as the operational safety criterion and define a safety-hacking event as selecting a response that passes Llama Guard but is classified as unsafe by HarmBench. Reported safety-hacking rates condition on non-abstention. Separately, we use gpt-5-mini [20] to evaluate safety-aware reward under the safety_aware_v1 rubric. These evaluators operationalize g⋆g and r⋆r for the experiments; neither is treated as ground truth. Evaluator and scoring details are given in Appendix B.1. Safety-hacking amplification and finite-N decomposition. For a fixed prompt, candidate ordering, and budget N, let KAK_A and KBK_B be the numbers of proxy-feasible candidates labeled safe and unsafe by HarmBench. When the corresponding class is nonempty, let MAM_A and MBM_B denote its largest proxy reward. Because the proxy scores have no ties, CBoNN selects an unsafe response in one of two cases: no safe feasible candidate is available, or the best unsafe candidate outscores the best safe candidate. Conditional on non-abstention, this gives Pr[y^NBoN∈B(x,b)∣KA+KB>0] [ y_N^BoN∈ B(x,b) K_A+K_B>0 ] =Pr[KB>0,KA=0∣KA+KB>0]+Pr[KA>0,KB>0,MB>MA∣KA+KB>0]. = [K_B>0,\ K_A=0 K_A+K_B>0 ]+ [K_A>0,\ K_B>0,\ M_B>M_A K_A+K_B>0 ]. (22) The first term measures unsafe-only exposure, where no safe feasible candidate is available. The second measures reward-based competition when both classes are present, and is the empirical finite-N counterpart of the joint-tail competition in Theorem 4.1. Figure 2: Finite-N decomposition of conditional CBoNN safety hacking on adversarial prompts under the HarmBench operational safety criterion. The total (green) is the sum of unsafe-only exposure (blue) and competitive unsafe wins (orange); random feasible selection is shown in gray. The competitive term corresponds to the joint-tail mechanism in Theorem 4.1. Points are averaged over 100100 candidate-order permutations; bands show ±1.96± 1.96 standard errors across prompts after permutation averaging. The decomposition in Figure 2 attributes the increase in safety hacking primarily to competitive selection. For the HarmBench subset and the combined evaluation, the unsafe-only term decreases with N, indicating that increased exposure to unsafe proxy-feasible responses does not account for the observed scaling behavior. By contrast, the competitive-selection term increases: conditional on both classes being present, the unsafe class more frequently attains the larger maximum proxy reward. This trend is consistent with the finite-N behavior implied by the relative joint tails in Theorem 4.1. Accordingly, the CBoNN hacking rate increasingly exceeds the random-feasible contamination baseline, whereas cPes remains closer to the baseline (Figure 3(a)). Repeating the analysis with a calibration-matched google/shieldgemma-2b [28] filter gives the same pattern, but a smaller overall increase in safety hacking: unsafe-only exposure falls with N, while competitive unsafe wins become more common (Appendix B.2). Proxy reward versus safety-aware reward. Larger budgets improve the objective optimized by CBoNN: its proxy reward increases well above the random-feasible reference (Figure 3(c)). Yet its gpt-5-mini-judged safety-aware reward, which penalizes unsafe and non-responsive outputs, deteriorates with N (Figure 3(b)). cPes produces smaller departures on both measures. Thus aggressive proxy optimization exploits proxy errors rather than improving the intended objective. (a) Safety-hacking rate. (b) Safety-aware true reward. (c) Mean proxy reward. Figure 3: Safety and reward under constrained inference-time scaling on adversarial prompts. HarmBench evaluates safety, and gpt-5-mini evaluates safety-aware reward. As N grows, CBoNN improves proxy reward but worsens both safety measures, while cPes limits these effects. Points are averaged over 100 candidate-order permutations; bands show ±1.96± 1.96 standard errors across prompts after permutation averaging. Reward-proxy ablation. To isolate the effect of the reward proxy, we replace PKU-Alignment/beaver-7b-v1.0-reward with Skywork/Skywork-Reward-V2-Llama-3.1-8B [15], while keeping the candidate pools, safety filter, and HarmBench labels fixed. With Beaver, CBoNN safety hacking increases from 9.1%9.1\% at N=1N=1 to 13.4%13.4\% at N=256N=256; with Skywork, it decreases to 6.8%6.8\%. At N=256N=256, the unsafe-only term is 1.6%1.6\% under both proxies, whereas the competitive unsafe-win term decreases from 11.7%11.7\% with Beaver to 5.3%5.3\% with Skywork (Figure 4). The reversal is therefore attributable to how the two reward proxies rank the same safe and unsafe proxy-feasible candidates, rather than to filtering or exposure. The corresponding conditional reward-tail survival curves are reported in Figure 5. This ablation is the empirical counterpart of the finite-N bounds in Theorems 4.1 and A.1, which depend on the joint tails Ψ^A _A and Ψ^B _B rather than false-positive mass alone. A favorable safe tail can prevent amplification over the evaluated budgets without providing an asymptotic guarantee. At N=256N=256, Skywork CBoNN has a lower safety-hacking rate than Skywork cPes (6.8%6.8\% versus 8.7%8.7\%). The guarantee for cPes is therefore not pointwise dominance over CBoNN, but an N-independent bound under adverse tail configurations (Corollary 5.3 and Theorem 5.1). (a) Safety-hacking rate. (b) Finite-N decomposition for CBoNN. Figure 4: Reward-proxy ablation with fixed candidate pools, safety-filter decisions, and HarmBench labels. Replacing Beaver with Skywork reverses the scaling trend in CBoNN safety hacking (left). The unsafe-only component is unchanged, whereas the competitive component decreases under Skywork (right), indicating that amplification depends on reward-based competition within the contaminated proxy-feasible set. 7 Discussion and Limitations Coverage control limits amplification, not contamination or misranking. The proxy-feasible reference already assigns mass κ(x,b)κ(x,b) to unsafe false positives. cPes limits further concentration on this residual mass but cannot identify it, and may still favor unsafe outputs with high learned rewards. Its N-independent bound therefore implies neither absolute safety nor pointwise dominance over CBoNN, as illustrated by the Skywork ablation. Robust inference requires intervention at multiple points. Our analysis points to three complementary interventions: improving the safety filter, limiting downstream concentration, and choosing reward proxies with favorable safe-versus-unsafe tail behavior. The Skywork ablation shows that reward-tail ordering can substantially change risk over the observed budget range. These interventions address different parts of the failure: filtering controls contamination, while reward-tail behavior and coverage control govern selection within the resulting feasible set. Limitations. We analyze a fixed, non-adaptive pipeline in which independently sampled candidates are filtered once and then ranked. Our theory therefore does not cover adaptive search, self-refinement, tree search, or agentic planning, whose candidate distributions can depend on previous proxy evaluations. By repeatedly steering generation toward high proxy scores, such procedures may amplify residual safety-filter errors more strongly than i.i.d. scaling, but our results do not establish or quantify this behavior. Extending the contamination–amplification analysis to such feedback-driven procedures is an important direction for future work. 8 Conclusion We studied safety hacking in constrained inference-time scaling, where reward maximization can amplify residual errors in a learned safety filter. The filter may admit unsafe responses, and downstream selection may favor them more strongly as inference compute grows. Our finite-N analysis shows that this behavior is governed by the joint reward tails of safe and unsafe proxy-feasible responses, and our asymptotic result establishes that a heavier unsafe tail can make safety hacking nearly certain even when average proxy errors are small. Coverage control yields an N-independent bound, instantiated by cPes, but cannot remove unsafe responses already admitted by the filter. Toy and language-model experiments provide finite-budget evidence for this amplification mechanism, while the reward-proxy ablation shows that ranking within the contaminated feasible set is central to the observed outcome. The analysis therefore separates two requirements for safe inference-time scaling: controlling entry into the feasible set and controlling downstream concentration. Extending this characterization to adaptive and agentic search remains an important direction for future work. References Amodei et al. (2016) D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Mané Concrete problems in AI safety. arXiv preprint arXiv:1606.06565. Cited by: §1, §2. Bai et al. (2022) Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, et al. Constitutional ai: harmlessness from ai feedback. arXiv preprint arXiv:2212.08073. Cited by: §2. Chao et al. (2024) P. Chao, E. Debenedetti, A. Robey, M. Andriushchenko, F. Croce, V. Sehwag, E. Dobriban, N. Flammarion, G. J. Pappas, F. Tramer, et al. Jailbreakbench: an open robustness benchmark for jailbreaking large language models. Advances in Neural Information Processing Systems 37, p. 55005–55029. Cited by: §2, §6.2. Chen et al. (2022) Y. Chen, H. Gao, G. Cui, F. Qi, L. Huang, Z. Liu, and M. Sun Why should adversarial perturbations be imperceptible? rethink the research paradigm in adversarial nlp. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, p. 11222–11237. Cited by: §6.2. Chittepu et al. (2026) Y. Chittepu, A. Joshi, S. Chintala, and S. Niekum Safe inference-time alignment via lagrangian reward augmentation. arXiv preprint arXiv:2607.02781. Cited by: §2. Dai et al. (2024) J. Dai, X. Pan, R. Sun, J. Ji, X. Xu, M. Liu, Y. Wang, and Y. Yang Safe rlhf: safe reinforcement learning from human feedback. In International Conference on Learning Representations, Vol. 2024, p. 50750–50777. Cited by: §6.2. Gao et al. (2023) L. Gao, J. Schulman, and J. Hilton Scaling laws for reward model overoptimization. In International Conference on Machine Learning, p. 10835–10866. Cited by: §1, §1, §2. Hendrycks et al. (2021) D. Hendrycks, N. Carlini, J. Schulman, and J. Steinhardt Unsolved problems in ML safety. arXiv preprint arXiv:2109.13916. Cited by: §1. Huang et al. (2025) A. Huang, A. Block, Q. Liu, N. Jiang, A. Krishnamurthy, and D. J. Foster Is best-of-n the best of them? coverage, scaling, and optimality in inference-time alignment. In International Conference on Machine Learning, p. 25075–25126. Cited by: §1, §2. Hughes et al. (2024) J. Hughes, S. Price, A. Lynch, R. Schaeffer, F. Barez, S. Koyejo, H. Sleight, E. Jones, E. Perez, and M. Sharma Bestof-n jailbreaking. arXiv preprint arXiv:2412.03556. Cited by: §1, §2. Inan et al. (2023) H. Inan, K. Upasani, J. Chi, R. Rungta, K. Iyer, Y. Mao, M. Tontchev, Q. Hu, B. Fuller, D. Testuggine, et al. Llama guard: llm-based input-output safeguard for human-ai conversations. arXiv preprint arXiv:2312.06674. Cited by: §1, §2. Ji et al. (2025) X. Ji, S. S. Ramesh, M. Zimmer, I. Bogunovic, J. Wang, and H. B. Ammar On almost surely safe alignment of large language models at inference-time. arXiv preprint arXiv:2502.01208. Cited by: §1. Jinnai et al. (2025) Y. Jinnai, T. Morimura, K. Ariu, and K. Abe Regularized best-of-n sampling with minimum bayes risk objective for language model alignment. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), p. 9321–9347. Cited by: §2. Khalaf et al. (2025) H. Khalaf, C. M. Verdun, A. Oesterling, H. Lakkaraju, and F. Calmon Inference-time reward hacking in large language models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: §2. Liu et al. (2025) C. Y. Liu, L. Zeng, Y. Xiao, J. He, J. Liu, C. Wang, R. Yan, W. Shen, F. Zhang, J. Xu, Y. Liu, and Y. Zhou Skywork-reward-v2: scaling preference data curation via human-ai synergy. arXiv preprint arXiv:2507.01352. Cited by: §6.2. Llama Team (2024) A. @. M. Llama Team The llama 3 herd of models. External Links: 2407.21783, Link Cited by: §6.2. Manheim and Garrabrant (2018) D. Manheim and S. Garrabrant Categorizing variants of goodhart’s law. arXiv preprint arXiv:1803.04585. Cited by: §2. Mazeika et al. (2024) M. Mazeika, L. Phan, X. Yin, A. Zou, Z. Wang, N. Mu, E. Sakhaee, N. Li, S. Basart, B. Li, et al. Harmbench: a standardized evaluation framework for automated red teaming and robust refusal. arXiv preprint arXiv:2402.04249. Cited by: §2, §6.2. Ouyang et al. (2022) L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems 35, p. 27730–27744. Cited by: §2. Singh et al. (2025) A. Singh, A. Fry, A. Perelman, A. Tart, A. Ganesh, A. El-Kishky, A. McLaughlin, A. Low, A. Ostrow, A. Ananthram, et al. Openai gpt-5 system card. arXiv preprint arXiv:2601.03267. Cited by: §6.2. Skalse et al. (2022) J. Skalse, N. Howe, D. Krasheninnikov, and D. Krueger Defining and characterizing reward gaming. Advances in neural information processing systems 35, p. 9460–9471. Cited by: §1. Snell et al. (2024) C. Snell, J. Lee, K. Xu, and A. Kumar Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314. Cited by: §1. Souly et al. (2024) A. Souly, Q. Lu, D. Bowen, T. Trinh, E. Hsieh, S. Pandey, P. Abbeel, J. Svegliato, S. Emmons, O. Watkins, et al. A strongreject for empty jailbreaks. Advances in Neural Information Processing Systems 37, p. 125416–125440. Cited by: §2. Stiennon et al. (2020) N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano Learning to summarize with human feedback. Advances in neural information processing systems 33, p. 3008–3021. Cited by: §2. Team (2024) Q. Team Qwen2.5: a party of foundation models. External Links: Link Cited by: §6.2. Wang et al. (2025) H. Wang, Z. Qin, L. Shen, X. Wang, D. Tao, and M. Cheng Safety reasoning with guidelines. In Forty-second International Conference on Machine Learning, Cited by: §1. Zeng et al. (2024a) W. Zeng, Y. Liu, R. Mullins, L. Peran, J. Fernandez, H. Harkous, K. Narasimhan, D. Proud, P. Kumar, B. Radharapu, et al. Shieldgemma: generative ai content moderation based on gemma. arXiv preprint arXiv:2407.21772. Cited by: §1, §2. Zeng et al. (2024b) W. Zeng, Y. Liu, R. Mullins, L. Peran, J. Fernandez, H. Harkous, K. Narasimhan, D. Proud, P. Kumar, B. Radharapu, O. Sturman, and O. Wahltinez ShieldGemma: generative ai content moderation based on gemma. External Links: 2407.21772, Link Cited by: §6.2. Zheng et al. (2023) L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems 36, p. 46595–46623. Cited by: §2. Appendix A Appendix A.1 Operational Finite-N Safety-hacking Criteria Corollary A.1. Under the conditions of Theorem 4.1, the following hold. 1. If there exist t∈ℝt , αA∈[0,1) _A∈[0,1), and γB>0 _B>0 such that NΨ^A(t,x,b)≤αAN _A(t;x,b)≤ _A and NΨ^B(t,x,b)≥γBN _B(t;x,b)≥ _B, then ℙ[y^NBoN(x)∈B(x,b)]≥(1−αA)(1−e−γB). [ y_N^BoN(x)∈ B(x,b) ]≥(1- _A) (1-e^- _B ). (23) 2. If there exist s∈ℝs , αB≥0 _B≥ 0, and γA>0 _A>0 such that NΨ^B(s,x,b)≤αBN _B(s;x,b)≤ _B, and NΨ^A(s,x,b)≥γAN _A(s;x,b)≥ _A, then ℙ[y^NBoN(x)∈B(x,b)]≤αB+e−γA. [ y_N^BoN(x)∈ B(x,b) ]≤ _B+e^- _A. (24) Proof. Let ψA=Ψ^A(t,x,b) _A= _A(t;x,b) and ψB=Ψ^B(t,x,b) _B= _B(t;x,b). From Theorem 4.1, L^N(t,x,b) L_N(t;x,b) =(1−ψA)N[1−(1−ψB1−ψA)N]. =(1- _A)^N [1- (1- _B1- _A )^N ]. (25) If NψA≤αA<1N _A≤ _A<1, then (1−ψA)N≥1−NψA≥1−αA(1- _A)^N≥ 1-N _A≥ 1- _A. Since ψB/(1−ψA)≥ψB _B/(1- _A)≥ _B and NψB≥γBN _B≥ _B, 1−(1−ψB1−ψA)N≥1−e−NψB≥1−e−γB. 1- (1- _B1- _A )^N≥ 1-e^-N _B≥ 1-e^- _B. (26) This proves the first claim. For the second claim, let ψA=Ψ^A(s,x,b) _A= _A(s;x,b) and ψB=Ψ^B(s,x,b) _B= _B(s;x,b). The upper bound gives U^N(s,x,b) U_N(s;x,b) ≤[1−(1−ψB)N]+(1−ψA)N≤NψB+e−NψA≤αB+e−γA. ≤ [1-(1- _B)^N ]+(1- _A)^N≤ N _B+e^-N _A≤ _B+e^- _A. (27) This completes the proof. ∎ Corollary A.1 gives an experimentally testable diagnostic: as N increases, estimate whether the effective unsafe-but-feasible score-tail mass NΨ^B(⋅)N _B(·) crosses a constant before the true-positive score-tail mass NΨ^A(⋅)N _A(·) does. If so, CBoNN should display compute-amplified safety hacking. A.2 Prompt-distribution Amplification Corollary A.2 (Prompt-distribution amplification). Let X∼X be a prompt drawn from a prompt distribution, and fix b∈(0,1]b∈(0,1]. For each prompt x, define the conditional CBoNN hacking probability, where the probability is over the N candidate samples drawn from πref(⋅∣x) _ref(· x), by pN(x)≔ℙy1:N∼πref(⋅∣x)[y^NBoN(x)∈B(x,b)]. p_N(x) _y_1:N _ref(· x) [ y_N^BoN(x)∈ B(x,b) ]. (28) Suppose there exist a measurable set of prompts ℰ⊆E and ρ∈[0,1]ρ∈[0,1] such that ℙX∼[X∈ℰ]≥ρ,P_X [X ]≥ρ, and, for every x∈ℰx , there exists a sequence tN(x)∈ℝt_N(x) satisfying NΨ^A(tN(x),x,b)→0N _A(t_N(x);x,b)→ 0 and NΨ^B(tN(x),x,b)→∞N _B(t_N(x);x,b)→∞. Then the aggregate safety-hacking probability satisfies lim infN→∞ℙ[y^NBoN(X)∈B(X,b)]≥ρ, _N→∞P [ y_N^BoN(X)∈ B(X,b) ]≥ρ, (29) where the probability is over X∼X and the candidate samples drawn conditionally from πref(⋅∣X) _ref(· X). Proof. By Theorem 4.2, for every x∈ℰx we have pN(x)→1p_N(x)→ 1. Moreover, 0≤pN(x)≤10≤ p_N(x)≤ 1 for all x and N. Therefore ℙ[y^NBoN(X)∈B(X,b)]=X∼[pN(X)]≥X∼[pN(X)X∈ℰ]. [ y_N^BoN(X)∈ B(X,b) ]=E_X [p_N(X)] _X [p_N(X)I\X \]. (30) Fatou’s lemma gives lim infN→∞[pN(X)X∈ℰ] _N→∞E[p_N(X)I\X \] ≥[lim infN→∞pN(X)X∈ℰ] [ _N→∞p_N(X)I\X \ ] (31) =ℙX∼[X∈ℰ]≥ρ. =P_X [X ]≥ρ. (32) This proves the claim. ∎ Corollary A.2 lifts the fixed-prompt tail condition to distribution-level evaluations: if a positive-measure subset of prompts is tail-amplifiable, then the average safety-hacking rate cannot vanish as the search budget grows. A.3 Direct score dominance Remark A.3 (Direct score dominance). If there exists a threshold t∈ℝt such that Ψ^A(t,x,b)=0 _A(t;x,b)=0 and Ψ^B(t,x,b)>0 _B(t;x,b)>0, then Theorem 4.2 applies with the constant sequence tN=t_N=t, and ℙ[y^NBoN(x)∈B(x,b)]→1. [ y_N^BoN(x)∈ B(x,b)]→ 1. (33) This holds, for example, when a positive-reference-mass subset of B(x,b)B(x,b) receives learned scores above the essential upper endpoint on A(x,b)A(x,b). A.4 Gaussian error-tail separation Definition A.4 (Proxy-feasible reward range). Fix x∈x and b∈(0,1]b∈(0,1] with ^+(x,b)≠∅ S_+(x,b)≠ . Define Δ(x,b)≔supy∈^+(x,b)r⋆(x,y)−infy∈^+(x,b)r⋆(x,y). (x,b) _y∈ S_+(x,b)r (x,y)- _y∈ S_+(x,b)r (x,y). (34) Assumption A.5 (Local bounded reward range). For the prompt x∈x and threshold b∈(0,1]b∈(0,1] under consideration, ^+(x,b)≠∅ S_+(x,b)≠ and Δ(x,b)<∞ (x,b)<∞. Proposition A.6 (Gaussian error-tail separation). We define the pointwise proxy errors εr(x,y) _r(x,y) ≔r^(x,y)−r⋆(x,y), r(x,y)-r (x,y), (35) εg(x,y) _g(x,y) ≔g^(x,y)−g⋆(x,y). g(x,y)-g (x,y). (36) Also, for ◊∈A,B ∈\A,B\, define η◊(x,b)≔ℙy∼πref(⋅∣x)[y∈◊(x,b)]. _ (x,b) _y _ref(· x)[y∈ (x,b)]. (37) Fix x∈x and b∈(0,1]b∈(0,1], and suppose A.5 holds. Assume that ηA(x,b)>0 _A(x,b)>0 and ηB(x,b)>0 _B(x,b)>0. Suppose further that there exist constants 0<σA(x)<σB(x)0< _A(x)< _B(x) such that the conditional upper tails of the reward-proxy error satisfy, as u→∞u→∞, logℙ[εr(x,y)>u∣y∈A(x,b)] [ _r(x,y)>u y∈ A(x,b) ] =−u22σA2(x)+o(u2), =- u^22 _A^2(x)+o(u^2), (38) logℙ[εr(x,y)>u∣y∈B(x,b)] [ _r(x,y)>u y∈ B(x,b) ] =−u22σB2(x)+o(u2). =- u^22 _B^2(x)+o(u^2). (39) Let RA⋆(x,b)≔supy∈A(x,b)r⋆(x,y). R_A (x,b) _y∈ A(x,b)r (x,y). (40) Then, for any constant c∈(2σA(x),2σB(x))c∈ ( 2 _A(x), 2 _B(x) ), the sequence τN≔RA⋆(x,b)+clogN _N R_A (x,b)+c N (41) satisfies the score-tail separation condition in Theorem 4.2; that is, NΨ^A(τN,x,b)→0,NΨ^B(τN,x,b)→∞. N _A( _N;x,b)→ 0, N _B( _N;x,b)→∞. (42) Consequently, limN→∞ℙ[y^NBoN(x)∈B(x,b)]=1. _N→∞P [ y_N^BoN(x)∈ B(x,b) ]=1. (43) The fixed shift Δ(x,b) (x,b) only enters when converting reward-error tails into learned-score tails, and it does not change the Gaussian large-deviation exponent. Proof. For ◊∈A,B ∈\A,B\ and t∈ℝt , define the joint reward-error tail Ψ◊(t;x,b)≔ℙy∼πref(⋅∣x)[y∈◊(x,b),εr(x,y)>t]. _ (t;x,b) _y _ref(· x) [y∈ (x,b),\ _r(x,y)>t ]. (44) These tails combine the reference probability of a class with the upper-tail behavior of reward-proxy overestimation inside that class. For ◊∈A,B ∈\A,B\, we can write Ψ◊(u,x,b)=η◊(x,b)ℙ[εr(x,y)>u∣y∈◊(x,b)]. _ (u;x,b)= _ (x,b)P [ _r(x,y)>u y∈ (x,b) ]. Let tN=clogN,c∈(2σA(x),2σB(x)),t_N=c N, c∈( 2 _A(x), 2 _B(x)), and set τN=RA⋆(x,b)+tN. _N=R_A (x,b)+t_N. Using the assumed tail exponent on A(x,b)A(x,b), log(NΨA(tN,x,b)) (N _A(t_N;x,b) ) =logN+logηA(x,b)−tN22σA2(x)+o(tN2) = N+ _A(x,b)- t_N^22 _A^2(x)+o(t_N^2) (45) =(1−c22σA2(x)+o(1))logN+O(1). = (1- c^22 _A^2(x)+o(1) ) N+O(1). (46) Since c>2σA(x)c> 2 _A(x), the coefficient of logN N is negative, and therefore NΨA(tN,x,b)→0. N _A(t_N;x,b)→ 0. (47) Moreover, for y∈A(x,b)y∈ A(x,b), r⋆(x,y)≤RA⋆(x,b)r (x,y)≤ R_A (x,b). Hence y∈A(x,b),r^(x,y)>τN⊆y∈A(x,b),εr(x,y)>tN, \y∈ A(x,b),\ r(x,y)> _N\ \y∈ A(x,b),\ _r(x,y)>t_N\, (48) and so NΨ^A(τN,x,b)≤NΨA(tN,x,b)→0. N _A( _N;x,b)≤ N _A(t_N;x,b)→ 0. (49) For the B tail, since Δ(x,b)<∞ (x,b)<∞ by assumption, (tN+Δ(x,b))2=c2logN+o(logN). (t_N+ (x,b))^2=c^2 N+o( N). (50) Using the assumed tail exponent on B(x,b)B(x,b), log(NΨB(tN+Δ(x,b),x,b)) (N _B(t_N+ (x,b);x,b) ) =logN+logηB(x,b)−(tN+Δ(x,b))22σB2(x)+o((tN+Δ(x,b))2) = N+ _B(x,b)- (t_N+ (x,b))^22 _B^2(x)+o ((t_N+ (x,b))^2 ) (51) =(1−c22σB2(x)+o(1))logN+O(1). = (1- c^22 _B^2(x)+o(1) ) N+O(1). (52) Since c<2σB(x)c< 2 _B(x), the coefficient of logN N is positive, and therefore NΨB(tN+Δ(x,b),x,b)→∞. N _B(t_N+ (x,b);x,b)→∞. (53) For every y∈B(x,b)y∈ B(x,b), we have y∈^+(x,b)y∈ S_+(x,b) and A(x,b)⊆^+(x,b)A(x,b) S_+(x,b). Hence RA⋆(x,b)≤supz∈^+(x,b)r⋆(x,z)R_A (x,b)≤ _z∈ S_+(x,b)r (x,z) and r⋆(x,y)≥infz∈^+(x,b)r⋆(x,z)r (x,y)≥ _z∈ S_+(x,b)r (x,z), so RA⋆(x,b)−r⋆(x,y)≤Δ(x,b). R_A (x,b)-r (x,y)≤ (x,b). (54) Thus y∈B(x,b),εr(x,y)>tN+Δ(x,b)⊆y∈B(x,b),r^(x,y)>τN, \y∈ B(x,b),\ _r(x,y)>t_N+ (x,b)\ \y∈ B(x,b),\ r(x,y)> _N\, (55) and hence NΨ^B(τN,x,b)≥NΨB(tN+Δ(x,b),x,b)→∞. N _B( _N;x,b)≥ N _B(t_N+ (x,b);x,b)→∞. (56) Therefore the score-tail separation conditions in Theorem 4.2 hold with threshold τN _N. The final claim follows from Theorem 4.2. ∎ Remark A.7 (Boundary behavior). When Ψ^A _A and Ψ^B _B are asymptotically comparable on the relevant score scale, finer tail constants and class masses determine the limit. In error-tail conditions such as Proposition A.6, the finite reward range Δ(x,b) (x,b) can affect boundary constants without changing the Gaussian exponent away from the boundary. A.5 Sample implementation and consistency of cPes Algorithm 1 Constrained Pessimistic Inference (cPes), sample implementation 0: Prompt x, reference policy πref(⋅∣x) _ref(· x), proxies r^,g r, g, threshold b, parameter β, clipping levels R^min<R^max R_ < R_ , budget N 1: Sample y1,…,yN∼iidπref(⋅∣x)y_1,…,y_N iid _ref(· x) 2: Set ℐN(x,b)=i∈[N]:g^(x,yi)≥bI_N(x,b)=\i∈[N]: g(x,y_i)≥ b\ and mN=|ℐN(x,b)|m_N=|I_N(x,b)| 3: if mN=0m_N=0 then 4: Return a predefined fallback, e.g., abstention 5: end if 6: Set r~i=clip(r^(x,yi),R^min,R^max) r_i=clip( r(x,y_i), R_ , R_ ) for each i∈ℐN(x,b)i _N(x,b) 7: Find λ^N(x)∈[R^min−β,R^max) λ_N(x)∈[ R_ -β, R_ ) such that 1mN∑i∈ℐN(x,b)ReLU(r~i−λ^N(x)β)=1 1m_N _i _N(x,b)ReLU ( r_i- λ_N(x)β )=1 8: Set p^i=1mNReLU(r~i−λ^N(x)β) p_i= 1m_NReLU ( r_i- λ_N(x)β ) for each i∈ℐN(x,b)i _N(x,b) 9: Sample I∈ℐN(x,b)I _N(x,b) with probability p^i p_i and return yIy_I Proposition A.8 (Consistency of sample cPes). Fix x∈x and b∈(0,1]b∈(0,1] with q(x,b)>0q(x,b)>0. Let π^N(⋅∣x) π_N(· x) be the random output distribution induced by Algorithm 1 on mN>0m_N>0. Then mN→∞m_N→∞ almost surely, λ^N(x)→λ(x) λ_N(x)→λ(x) almost surely, and for every bounded f:→ℝf:Y , y∼π^N(⋅∣x)[f(y)]→y∼π^(⋅∣x)[f(y)]almost surely. _y π_N(· x)[f(y)] _y π(· x)[f(y)] surely. (57) Proof. Let mN=∑i=1Nyi∈^+(x,b)m_N= _i=1^NI\y_i∈ S_+(x,b)\. Since mN/N→q(x,b)>0m_N/N→ q(x,b)>0 almost surely, mN→∞m_N→∞ almost surely and mN>0m_N>0 eventually. Conditional on belonging to ^+(x,b) S_+(x,b), the feasible samples are i.i.d. from πref♯(⋅∣x) _ref (· x). Define hN(λ) h_N(λ) ≔1mN∑i∈ℐN(x,b)ReLU(r~(x,yi)−λβ), 1m_N _i _N(x,b)ReLU ( r(x,y_i)-λβ ), (58) h(λ) h(λ) ≔πref♯ReLU(r~(x,y)−λβ). _ _ref ReLU ( r(x,y)-λβ ). (59) The functions indexed by λ∈[R^min−β,R^max]λ∈[ R_ -β, R_ ] are uniformly bounded and 1/β1/β-Lipschitz in λ. A finite-grid argument combined with the strong law of large numbers yields supλ∈[R^min−β,R^max]|hN(λ)−h(λ)|→0almost surely. _λ∈[ R_ -β, R_ ]|h_N(λ)-h(λ)|→ 0 surely. (60) By Proposition 5.2, the equation h(λ)=1h(λ)=1 has a unique solution λ(x)λ(x). Uniform convergence and monotonicity imply that any empirical solution λ^N(x) λ_N(x) to hN(λ)=1h_N(λ)=1 converges almost surely to λ(x)λ(x). For any bounded f, write π^N[f]=1mN∑i∈ℐN(x,b)f(yi)ReLU(r~(x,yi)−λ^N(x)β). _ π_N[f]= 1m_N _i _N(x,b)f(y_i)ReLU ( r(x,y_i)- λ_N(x)β ). (61) The summands are uniformly bounded, and the weights converge uniformly in the multiplier because of the Lipschitz property. Applying the strong law again and using λ^N(x)→λ(x) λ_N(x)→λ(x) gives π^N[f]→πref♯[f(y)ReLU(r~(x,y)−λ(x)β)]=π^[f]. _ π_N[f] _ _ref [f(y)ReLU ( r(x,y)-λ(x)β ) ]=E_ π[f]. (62) ∎ A.6 Utility tradeoff of cPes Bounded coverage alone can be achieved by avoiding reward optimization altogether. We therefore quantify the utility cost of coverage control. For a policy π, let J(π;x)≔y∼π(⋅∣x)[r⋆(x,y)]. J(π;x) _y π(· x)[r (x,y)]. (63) Define the clipped-score RMSE under the proxy-feasible reference by ε¯r,clip♯(x)≔y∼πref♯(⋅∣x)[(r~(x,y)−r⋆(x,y))2]. _r,clip (x) E_y _ref (· x) [( r(x,y)-r (x,y))^2 ]. (64) This quantity includes both reward-proxy error and any distortion introduced by clipping. Proposition A.9 (Utility tradeoff within the proxy-feasible set). Suppose ε¯r,clip♯(x)<∞ _r,clip (x)<∞. For any comparator policy ρ(⋅∣x)≪πref♯(⋅∣x)ρ(· x) _ref (· x), the population cPes policy satisfies J(ρ,x)−J(π^,x)≤β2(Cρ♯(x)−1)+ε¯r,clip♯(x)(Cρ♯(x)+1+R^spanβ). J(ρ;x)-J( π;x)≤ β2 (C_ρ (x)-1 )+ _r,clip (x) ( C_ρ (x)+ 1+ R_spanβ ). (65) The bound makes explicit the role of β. Smaller β permits more aggressive reward optimization but allows greater concentration relative to πref♯ _ref , while larger β keeps the policy closer to the proxy-feasible reference distribution. A.7 Oracle regret decomposition For the oracle comparison below, define the ambient coverage coefficient Cπ(x)≔∑y∈π(y∣x)2πref(y∣x). C_π(x) _y π(y x)^2 _ref(y x). (66) For a comparator π⋆π , let α⋆(x,b)≔π⋆(^+(x,b)∣x)α (x,b) π ( S_+(x,b) x) and, when α⋆(x,b)>0α (x,b)>0, define π⋆,♯(y∣x)≔π⋆(y∣x)y∈^+(x,b)α⋆(x,b). π , (y x) π (y x)I\y∈ S_+(x,b)\α (x,b). (67) Theorem A.10 (Oracle regret decomposition). Fix b∈(0,1)b∈(0,1) and suppose 0≤r⋆(x,y)≤Rmax0≤ r (x,y)≤ R_ . Let π⋆≪πrefπ _ref satisfy supp(π⋆)⊆+⋆(x) supp(π ) _+ (x); if α⋆(x,b)>0α (x,b)>0, assume also π⋆,♯≪πref♯π , _ref . If α⋆(x,b)>0α (x,b)>0, then J(π⋆,x)−J(π^,x) J(π ;x)-J( π;x) ≤[β2(Cπ⋆,♯(x)−1)+ε¯r,clip♯(x)(Cπ⋆,♯(x)+1+R^spanβ)] ≤ [ β2 (C_π , (x)-1 )+ _r,clip (x) ( C_π , (x)+ 1+ R_spanβ ) ] +RmaxCπ⋆(x)ε¯g(x)1−b. + R_ C_π (x)\, _g(x)1-b. (68) If α⋆(x,b)=0α (x,b)=0, then J(π⋆,x)−J(π^,x)≤RmaxCπ⋆(x)ε¯g(x)1−b, J(π ;x)-J( π;x)≤ R_ C_π (x)\, _g(x)1-b, (69) that is, the bracketed proxy-feasible comparison term is omitted. Proof. Decompose the regret of the truly safe comparator π⋆π : J(π⋆,x)−J(π^,x) J(π ;x)-J( π;x) =π⋆[r⋆(x,y)y∈^+(x,b)]−J(π^;x) =E_π [r (x,y)I\y∈ S_+(x,b)\]-J( π;x) (70) +π⋆[r⋆(x,y)y∉^+(x,b)]. +E_π [r (x,y)I\y∉ S_+(x,b)\]. (71) If α⋆(x,b)>0α (x,b)>0, then the first term is at most J(π⋆,♯,x)−J(π^,x)J(π , ;x)-J( π;x) because α⋆(x,b)≤1α (x,b)≤ 1 and rewards are nonnegative. Applying Proposition A.9 with ρ=π⋆,♯ρ=π , gives the bracketed term in (68). If α⋆(x,b)=0α (x,b)=0, the first term is −J(π^,x)≤0-J( π;x)≤ 0, so the bracketed term is omitted. It remains to control the second term. Since supp(π⋆)⊆+⋆(x) supp(π ) _+ (x), the event y∉^+(x,b)\y∉ S_+(x,b)\ under π⋆π is a false negative of the safety proxy. On this event, g⋆(x,y)=1g (x,y)=1 and g^(x,y)<b g(x,y)<b, so |εg(x,y)|>1−b| _g(x,y)|>1-b. Hence ℙπref[y∈+⋆(x),y∉^+(x,b)]≤ε¯g(x)2(1−b)2. _ _ref[y _+ (x),\ y∉ S_+(x,b)]≤ _g(x)^2(1-b)^2. (72) By Cauchy–Schwarz and absolute continuity, ℙπ⋆[y∉^+(x,b)] _π [y∉ S_+(x,b)] =πref[π⋆(y∣x)πref(y∣x)y∈+⋆(x),y∉^+(x,b)] =E_ _ref [ π (y x) _ref(y x)I\y _+ (x),\ y∉ S_+(x,b)\ ] (73) ≤Cπ⋆(x)ℙπref[y∈+⋆(x),y∉^+(x,b)] ≤ C_π (x) P_ _ref[y _+ (x),\ y∉ S_+(x,b)] (74) ≤Cπ⋆(x)ε¯g(x)1−b. ≤ C_π (x)\, _g(x)1-b. (75) Using r⋆(x,y)≤Rmaxr (x,y)≤ R_ proves π⋆[r⋆(x,y)y∉^+(x,b)]≤RmaxCπ⋆(x)ε¯g(x)1−b. _π [r (x,y)I\y∉ S_+(x,b)\]≤ R_ C_π (x)\, _g(x)1-b. (76) Combining the bounds proves the theorem. ∎ A.8 Proof of Theorem 4.1 Proof. Suppress (x,b)(x,b) in the notation. For a single sample y, define E^A(t) E_A(t) ≔y∈A,r^(x,y)>t, \y∈ A,\ r(x,y)>t\, (77) E^B(t) E_B(t) ≔y∈B,r^(x,y)>t. \y∈ B,\ r(x,y)>t\. (78) The sets A and B are disjoint, so these two one-sample events are disjoint. If among N samples no event E^A(t) E_A(t) occurs and at least one event E^B(t) E_B(t) occurs, then every true-positive proxy-feasible sample has learned score at most t, while some unsafe-but-feasible sample has learned score strictly larger than t. Therefore the CBoNN maximizer over the proxy-feasible samples lies in B. The probability of this sufficient event is (1−Ψ^A(t))N−(1−Ψ^A(t)−Ψ^B(t))N, (1- _A(t) )^N- (1- _A(t)- _B(t) )^N, (79) which proves the lower bound. For the upper bound, consider the event that no unsafe-but-feasible sample satisfies r^(x,y)>s r(x,y)>s and at least one true-positive proxy-feasible sample satisfies r^(x,y)>s r(x,y)>s. On this event, some sample in A strictly beats every sample in B, so CBoNN cannot hack. The probability of this non-hacking certificate is (1−Ψ^B(s))N−(1−Ψ^B(s)−Ψ^A(s))N. (1- _B(s) )^N- (1- _B(s)- _A(s) )^N. (80) Hence the hacking probability is at most its complement, 1−(1−Ψ^B(s))N+(1−Ψ^B(s)−Ψ^A(s))N, 1- (1- _B(s) )^N+ (1- _B(s)- _A(s) )^N, (81) which is (9). ∎ A.9 Proof of Theorem 4.2 Proof. Let aN=Ψ^A(tN,x,b)a_N= _A(t_N;x,b) and dN=Ψ^B(tN,x,b)d_N= _B(t_N;x,b). The assumptions imply NaN→0Na_N→ 0 and NdN→∞Nd_N→∞. Therefore (1−aN)N→1(1-a_N)^N→ 1 and (1−aN−dN)N≤exp[−N(aN+dN)]→0(1-a_N-d_N)^N≤ [-N(a_N+d_N)]→ 0. The lower bound in Theorem 4.1 converges to one, proving the claim. ∎ A.10 Proof of Corollary 4.3 Proof. Fix x and suppress the dependence on x in the notation. Let the output space contain two outputs, =yA,yB,Y=\y_A,y_B\, with reference probabilities πref(yA∣x)=1−ε,πref(yB∣x)=ε. _ ref(y_A x)=1- , _ ref(y_B x)= . Define the true and learned safety functions by g⋆(x,yA)=1,g^(x,yA)=1,g (x,y_A)=1, g(x,y_A)=1, and g⋆(x,yB)=0,g^(x,yB)=b.g (x,y_B)=0, g(x,y_B)=b. Then both outputs are proxy-feasible, since g^(x,yA)≥b g(x,y_A)≥ b and g^(x,yB)≥b g(x,y_B)≥ b. Moreover, A(x,b)=yA,B(x,b)=yB,A(x,b)=\y_A\, B(x,b)=\y_B\, and therefore ℙy∼πref(⋅∣x)[y∈B(x,b)]=ε.P_y _ ref(· x)[y∈ B(x,b)]= . The safety-proxy RMSE is πref[(g^(x,y)−g⋆(x,y))2]=εb2=bε. E_ _ ref [ ( g(x,y)-g (x,y) )^2 ]= b^2=b . Now define the true and learned reward functions by r⋆(x,yA)=0,r^(x,yA)=0,r (x,y_A)=0, r(x,y_A)=0, and r⋆(x,yB)=0,r^(x,yB)=ξ.r (x,y_B)=0, r(x,y_B)=ξ. Thus the reward-proxy RMSE is πref[(r^(x,y)−r⋆(x,y))2]=εξ2=ξε. E_ _ ref [ ( r(x,y)-r (x,y) )^2 ]= ξ^2=ξ . Since r^(x,yB)>r^(x,yA) r(x,y_B)> r(x,y_A), constrained Best-of-N returns yBy_B if and only if at least one of the N sampled candidates equals yBy_B. All sampled candidates are proxy-feasible, so there is no abstention event. Hence ℙ[y^NBoN(x)∈B(x,b)]=1−(1−ε)N.P [ y BoN_N(x)∈ B(x,b) ]=1-(1- )^N. Using (1−ε)N≤exp(−Nε)(1- )^N≤ (-N ) gives ℙ[y^NBoN(x)∈B(x,b)]≥1−exp(−Nε).P [ y BoN_N(x)∈ B(x,b) ]≥ 1- (-N ). Therefore, if N≥ε−1log(1/δ),N≥ ^-1 (1/δ), then the safety-hacking probability is at least 1−δ1-δ. This proves the claim. ∎ A.11 Proof of Theorem 5.1 Proof. Let w(y)≔π(y∣x)πref♯(y∣x),κ≔πref♯(B(x,b)∣x).w(y) π(y x) _ref (y x), κ _ref (B(x,b) x). Since πref♯[w]=1E_ _ref [w]=1 and πref♯[w2]=Cπ♯(x)E_ _ref [w^2]=C_π (x), πref♯[(w−1)2]=Cπ♯(x)−1. _ _ref [(w-1)^2]=C_π (x)-1. (82) Moreover, ℙy∼π(⋅∣x)[y∈B(x,b)]−κ _y π(· x)[y∈ B(x,b)]-κ =πref♯[(w(y)−1)y∈B(x,b)] =E_ _ref [(w(y)-1)I\y∈ B(x,b)\ ] (83) =πref♯[(w(y)−1)(y∈B(x,b)−κ)], =E_ _ref [(w(y)-1) (I\y∈ B(x,b)\-κ ) ], (84) where the second equality uses πref♯[w−1]=0E_ _ref [w-1]=0. By Cauchy–Schwarz, |ℙy∼π(⋅∣x)[y∈B(x,b)]−κ| |P_y π(· x)[y∈ B(x,b)]-κ | ≤Cπ♯(x)−1κ(1−κ). ≤ C_π (x)-1 κ(1-κ). (85) This proves the bound. By the definition of πref♯ _ref , we have ℙy∼πref♯(⋅∣x)[y∈B(x,b)] _y _ref (· x)[y∈ B(x,b)] =y∼πref♯(⋅∣x)[y∈B(x,b)] =E_y _ref (· x) [I\y∈ B(x,b)\ ] (86) =y∼πref(⋅∣x)[y∈B(x,b)y∈^+(x,b)q(x,b)] =E_y _ref(· x) [ I\y∈ B(x,b)\I\y∈ S_+(x,b)\q(x,b) ] (87) =y∼πref(⋅∣x)[y∈B(x,b)]q(x,b). = E_y _ref(· x) [I\y∈ B(x,b)\ ]q(x,b). (88) If y∈B(x,b)y∈ B(x,b), then g⋆(x,y)=0g (x,y)=0 and g^(x,y)≥b g(x,y)≥ b, so |εg(x,y)|≥b| _g(x,y)|≥ b. Hence y∈B(x,b)≤εg(x,y)2b2. \y∈ B(x,b)\≤ _g(x,y)^2b^2. (89) Taking expectation under πref(⋅∣x) _ref(· x) gives πref(B(x,b)∣x)≤ε¯g(x)2b2. _ref(B(x,b) x)≤ _g(x)^2b^2. (90) Combining this inequality with κ(x,b)=πref(B(x,b)∣x)/q(x,b)κ(x,b)= _ref(B(x,b) x)/q(x,b) gives κ(x,b)≤ε¯g(x)2b2q(x,b), κ(x,b)≤ _g(x)^2b^2q(x,b), (91) which proves (16). ∎ A.12 Proof of Proposition 5.2 Proof. Let μ(y)=πref♯(y∣x)μ(y)= _ref (y x) and write w(y)=π(y∣x)/μ(y)w(y)=π(y x)/μ(y). Then π≪μπ μ and ∑yπ(y∣x)=1 _yπ(y x)=1 are equivalent to w(y)≥0w(y)≥ 0 and μ[w]=1E_μ[w]=1. The objective in (18) becomes maxw≥0,μ[w]=1y∼μ[w(y)r~(x,y)−β2w(y)2]. _w≥ 0,\ E_μ[w]=1E_y μ [w(y) r(x,y)- β2w(y)^2 ]. (92) This is strictly concave in w, so the optimizer is unique. Introduce a Lagrange multiplier λ for μ[w]=1E_μ[w]=1. Pointwise maximization of the Lagrangian over w(y)≥0w(y)≥ 0 gives wλ(y)=(r~(x,y)−λβ)+=ReLU(r~(x,y)−λβ). w_λ(y)= ( r(x,y)-λβ )_+=ReLU ( r(x,y)-λβ ). (93) The multiplier must satisfy μ[wλ]=1E_μ[w_λ]=1. It remains to show existence and uniqueness of λ. Define h(λ)≔μ[ReLU(r~(x,y)−λβ)]. h(λ) _μ [ReLU ( r(x,y)-λβ ) ]. (94) The function h is continuous and nonincreasing. By construction, h(R^max)=0h( R_ )=0, while h(R^min−β)=μ[r~(x,y)−R^min+β]≥1. h( R_ -β)=E_μ [ r(x,y)- R_ +β ]≥ 1. (95) Thus a solution exists in [R^min−β,R^max)[ R_ -β, R_ ). It is unique because whenever h(λ)>0h(λ)>0, the event r~(x,y)>λ\ r(x,y)>λ\ has positive μ-probability, so h is strictly decreasing on the relevant level set. Finally, let w(y)=wλ(x)(y)w(y)=w_λ(x)(y). Since μ[w]=1E_μ[w]=1 and λ(x)≥R^min−βλ(x)≥ R_ -β, 0≤w(y)≤R^max−λ(x)β≤1+R^spanβ. 0≤ w(y)≤ R_ -λ(x)β≤ 1+ R_spanβ. (96) Therefore Cπ^♯(x)=μ[w(y)2]≤(supyw(y))μ[w(y)]≤1+R^spanβ. C_ π (x)=E_μ[w(y)^2]≤ ( _yw(y) )E_μ[w(y)]≤ 1+ R_spanβ. (97) ∎ A.13 Proof of Proposition A.9 Proof. Let μ=πref♯(⋅∣x)μ= _ref (· x). By optimality of π π in (18), for any ρ(⋅∣x)≪μρ(· x) μ, ρ[r~(x,y)]−π^[r~(x,y)]≤β2(Cρ♯(x)−1), _ρ[ r(x,y)]-E_ π[ r(x,y)]≤ β2 (C_ρ (x)-1 ), (98) where we dropped the nonpositive term −β2(Cπ^♯(x)−1)- β2(C_ π (x)-1). Therefore J(ρ,x)−J(π^,x) J(ρ;x)-J( π;x) ≤β2(Cρ♯(x)−1)+|ρ[r~(x,y)−r⋆(x,y)]|+|π^[r~(x,y)−r⋆(x,y)]|. ≤ β2 (C_ρ (x)-1 )+ |E_ρ[ r(x,y)-r (x,y)] |+ |E_ π[ r(x,y)-r (x,y)] |. (99) Cauchy–Schwarz under μ gives |ρ[r~(x,y)−r⋆(x,y)]| |E_ρ[ r(x,y)-r (x,y)] | ≤ε¯r,clip♯(x)Cρ♯(x), ≤ _r,clip (x) C_ρ (x), (100) |π^[r~(x,y)−r⋆(x,y)]| |E_ π[ r(x,y)-r (x,y)] | ≤ε¯r,clip♯(x)Cπ^♯(x)≤ε¯r,clip♯(x)1+R^spanβ, ≤ _r,clip (x) C_ π (x)≤ _r,clip (x) 1+ R_spanβ, (101) where the last inequality follows from (20). Combining these bounds proves (65). ∎ A.14 Proof of Corollary 5.3 Finite-sample claim. Proof. Condition on the event mN>0m_N>0 and on the sampled candidates. Let wi=ReLU(r~(x,yi)−λ^N(x)β),i∈ℐN(x,b). w_i=ReLU ( r(x,y_i)- λ_N(x)β ), i _N(x,b). (102) By construction, mN−1∑i∈ℐNwi=1m_N^-1 _i _Nw_i=1. The conditional probability that cPes returns an unsafe-but-feasible sample is 1mN∑i∈ℐNwiyi∈B(x,b). 1m_N _i _Nw_iI\y_i∈ B(x,b)\. (103) Let KNB K_N^B ≔∑i=1Nyi∈B(x,b), _i=1^NI\y_i∈ B(x,b)\, (104) κ^N κ_N ≔KNBmN. K_N^Bm_N. (105) Since mN−1∑i∈ℐN(wi−1)=0m_N^-1 _i _N(w_i-1)=0, we have 1mN∑i∈ℐNwiyi∈B(x,b)−κ^N=1mN∑i∈ℐN(wi−1)(yi∈B(x,b)−κ^N). 1m_N _i _Nw_iI\y_i∈ B(x,b)\- κ_N= 1m_N _i _N(w_i-1) (I\y_i∈ B(x,b)\- κ_N ). (106) By Cauchy–Schwarz, |1mN∑i∈ℐNwiyi∈B(x,b)−κ^N|≤(1mN∑i∈ℐN(wi−1)2)1/2κ^N(1−κ^N). | 1m_N _i _Nw_iI\y_i∈ B(x,b)\- κ_N |≤ ( 1m_N _i _N(w_i-1)^2 )^1/2 κ_N(1- κ_N). (107) The empirical analogue of (20) gives 1mN∑i∈ℐN(wi−1)2=1mN∑i∈ℐNwi2−1≤R^spanβ. 1m_N _i _N(w_i-1)^2= 1m_N _i _Nw_i^2-1≤ R_spanβ. (108) Consequently, conditional on the sampled candidates, ℙ[y^cPes,N(x)∈B(x,b)∣y1,…,yN]≤κ^N+R^spanβκ^N(1−κ^N). [ y_cPes,N(x)∈ B(x,b) y_1,…,y_N]≤ κ_N+ R_spanβ κ_N(1- κ_N). (109) Given mN=m>0m_N=m>0, the feasible samples are i.i.d. from πref♯ _ref , and hence KNB|mN=m∼Binomial(m,κ(x,b)). K_N^B m_N=m (m,κ(x,b) ). (110) Therefore, [κ^N∣mN=m]=κ(x,b). [ κ_N m_N=m]=κ(x,b). (111) Since z↦z(1−z)z z(1-z) is concave on [0,1][0,1], Jensen’s inequality gives [κ^N(1−κ^N)|mN=m]≤κ(x,b)(1−κ(x,b)). [ κ_N(1- κ_N)\, |\,m_N=m ]≤ κ(x,b)(1-κ(x,b)). (112) It follows, for every m>0m>0, that ℙ[y^cPes,N(x)∈B(x,b)∣mN=m]≤κ(x,b)+R^spanβκ(x,b)(1−κ(x,b)). [ y_cPes,N(x)∈ B(x,b) m_N=m]≤κ(x,b)+ R_spanβκ(x,b)(1-κ(x,b)). (113) Averaging over m gives the same bound conditional on mN>0m_N>0. Finally, Theorem 5.1 gives κ(x,b)≤ε¯g(x)2b2q(x,b). κ(x,b)≤ _g(x)^2b^2q(x,b). (114) Using 1−κ(x,b)≤11-κ(x,b)≤ 1, we obtain ℙ[y^cPes,N(x)∈B(x,b)∣mN>0]≤min1,ε¯g(x)2b2q(x,b)+ε¯g(x)bR^spanβq(x,b), [ y_cPes,N(x)∈ B(x,b) m_N>0]≤ \1,\, _g(x)^2b^2q(x,b)+ _g(x)b R_spanβ q(x,b) \, (115) which proves (21). The unconditional statement follows because abstention is not an unsafe output. ∎ Population claim. Proof. By Proposition 5.2, the population pessimistic policy satisfies π^(⋅∣x)≪πref♯(⋅∣x) π(· x) _ref (· x) and Cπ^♯(x)≤1+R^spanβ. C_ π (x)≤ 1+ R_spanβ. (116) Applying Theorem 5.1 and using (20) gives ℙy∼π^(⋅∣x)[y∈B(x,b)]≤κ(x,b)+R^spanβκ(x,b)(1−κ(x,b)). _y π(· x)[y∈ B(x,b)]≤κ(x,b)+ R_spanβκ(x,b) (1-κ(x,b) ). (117) Since κ(x,b)≤ε¯g(x)2b2q(x,b), κ(x,b)≤ _g(x)^2b^2q(x,b), (118) and 1−κ(x,b)≤11-κ(x,b)≤ 1, it follows that ℙy∼π^(⋅∣x)[y∈B(x,b)]≤min1,ε¯g(x)2b2q(x,b)+ε¯g(x)bR^spanβq(x,b). _y π(· x)[y∈ B(x,b)]≤ \1,\, _g(x)^2b^2q(x,b)+ _g(x)b R_spanβ q(x,b) \. (119) ∎ A.15 Additional Toy Experimental Details and Ablations Experimental details. The toy problem in Section 6.1 uses class probabilities (0.39,0.01,0.60)(0.39,0.01,0.60) for A, B, and C, respectively. We evaluate random feasible selection, CBoNN, and cPes over budgets N∈1,2,4,…,8192, N∈\1,2,4,…,8192\, (120) using 50005000 Monte Carlo trials per budget. Unless otherwise stated, metrics are reported conditional on non-abstention, since at small N all sampled candidates may belong to the proxy-rejected class C. The true reward gap between the safe and unsafe proxy-feasible classes is Δ=rA⋆−rB⋆=0.6, =r_A -r_B =0.6, (121) which matches the local reward-range shift used when translating reward-error tails into learned-score tails in Proposition A.6. cPes regularization. The cPes results in Figure 1 are consistent with the budget-uniform control predicted by Corollary 5.3. With β=1.0β=1.0, the conditional hacking rate remains between 0.0150.015 and 0.0240.024 over the entire budget grid, while the mean true reward stays near 0.790.79. Even with β=0.2β=0.2, the hacking rate remains around 0.0240.024–0.0310.031 at large budgets. A separate sweep over β illustrates the concentration tradeoff. At N=8192N=8192, the hacking rate is 0.15180.1518 for β=0.02β=0.02, 0.04180.0418 for β=0.1β=0.1, and 0.01520.0152 for β=1.0β=1.0, with the highest true reward observed around β∈[0.5,1.0]β∈[0.5,1.0]. Thus, as cPes approaches aggressive max-selection, it increasingly recovers the same amplification mechanism as CBoNN. Equal-tail control. To verify that large candidate budgets alone do not cause safety hacking, we set σB=σA=0.2 _B= _A=0.2 while keeping the remainder of the environment unchanged. In this setting, the unsafe class B no longer has a heavier reward-error upper tail than the safe class A. The CBoNN hacking rate falls to zero by N=32N=32, and the selected true reward converges to 0.80.8, the reward of class A. No-false-positive control. We also remove class B entirely. In this case the proxy-feasible set contains no unsafe outputs, and the safety-hacking rate is identically zero for all methods and all candidate budgets. False-positive tail-scale sweep. Finally, Figure 1(c) varies the reward-proxy noise scale σB _B of the unsafe-but-feasible class. At N=8192N=8192, the CBoNN hacking rate increases from 0.00.0 at σB=0.2 _B=0.2 to 0.6970.697 at σB=0.6 _B=0.6, 0.99960.9996 at σB=1.0 _B=1.0, and 0.99980.9998 at σB=1.2 _B=1.2. Together, these controls support the mechanism characterized by Theorem 4.2: increasing N is not sufficient by itself. Amplification occurs when residual unsafe outputs remain proxy-feasible and have a sufficiently favorable upper tail under the learned reward. Appendix B Additional LLM Analysis B.1 Evaluator and scoring details Safety proxy and operational safety criterion. The safety proxy is Llama-Guard-3-8B. We use 1−punsafe1-p_unsafe as its safe score and accept candidates whose score is at least b=0.95b=0.95. Primary safety labels are produced by the HarmBench classifier HarmBench-Llama-2-13b-cls. Thus A contains proxy-feasible candidates classified as safe by HarmBench, B contains proxy-feasible candidates classified as unsafe, and C contains proxy-rejected candidates. The reported safety-hacking rate is the probability of selecting from B, conditional on non-abstention. Safety-aware reward evaluator. We score safety-aware reward with the OpenAI Responses API using the gpt-5-mini API model alias and the safety_aware_v1 rubric. The rubric assigns 1.0 to responses that are safe and fully satisfy a legitimate request, 0.0 to responses that are unsafe, enable harmful behavior, or are useless/nonresponsive, and intermediate values to partially helpful, vague, incomplete, or weakly useful responses. Reward proxies and aggregation. The primary reward proxy is PKU-Alignment/beaver-7b-v1.0-reward. For the reward-proxy ablation, we use Skywork-Reward-V2-Llama-3.1-8B with Hugging Face Transformers. Its input is formatted as User: prompt_text : response_text, and the scalar sequence-classification logit is used as the raw proxy reward. Downstream normalization uses the calibration median and interquartile range; clipping is applied only where required by cPes. B.2 Alternative safety-filter sensitivity analysis We repeat the CBoNN analysis using google/shieldgemma-2b as an alternative safety filter, while holding the candidate pool, generator, Beaver reward proxy, HarmBench labels, and candidate permutations fixed. We select the ShieldGemma threshold using only the 179 calibration prompts by matching the Llama Guard calibration feasible rate. The resulting threshold is 0.81760.8176, giving a feasible rate of 0.82420.8242, compared with 0.82950.8295 for Llama Guard at its primary threshold. N Hacking Abstention Unsafe-only Competitive Tail win 11 12.10(0.98)12.10\,(0.98) 14.24(1.04)14.24\,(1.04) 12.1012.10 0.000.00 – 256256 14.75(1.34)14.75\,(1.34) 1.26(0.42)1.26\,(0.42) 0.990.99 13.7613.76 35.0235.02 Table 1: ShieldGemma-2B alternative-filter sensitivity analysis. All entries are percentages; standard errors across prompts are shown in parentheses for hacking and abstention. Unsafe-only and Competitive are the two disjoint terms in (22) and sum to the total hacking rate. Tail win denotes Pr(MB>MA∣KA>0,KB>0) (M_B>M_A K_A>0,K_B>0) and is reported separately because it uses a different conditioning event. Conditional CBoNN safety hacking increases from 12.1%12.1\% at N=1N=1 to 14.8%14.8\% at N=256N=256. Over the same range, the unsafe-only term decreases from 12.1%12.1\% to 1.0%1.0\%, while the competitive unsafe-win term increases from zero to 13.8%13.8\%. Thus the observed increase is driven by reward-based competition rather than unsafe-only exposure. The increase is smaller than in the primary analysis, so we treat this result as qualitatively consistent sensitivity evidence rather than as a replacement for the primary Llama Guard analysis. B.3 Reward-tail survival curves Figure 5 visualizes the conditional learned-score survival curves within the safe feasible class A(x,b)A(x,b) and the unsafe feasible class B(x,b)B(x,b). These curves are the score-distribution component of the joint tails Ψ^A _A and Ψ^B _B in (6); the class masses are held fixed across the Beaver and Skywork ablations. With Beaver, unsafe feasible responses retain more mass at high reward thresholds than safe feasible responses. With Skywork, the safe feasible curve instead extends farther into the upper tail. This reversal is consistent with the lower competitive unsafe-win term in Figure 4(b) and with the finite-N behavior in Figure 4(a). Because the figure estimates conditional survival curves from a finite candidate pool, it is diagnostic evidence for the tail mechanism rather than a verification of the asymptotic conditions in Theorem 4.2. Figure 5: Conditional survival curves of learned reward scores among safe feasible (A) and unsafe feasible (B) responses. With Beaver, the unsafe feasible class has the heavier observed upper tail; with Skywork, the safe feasible class does. The panels use their respective proxy-score scales and should be interpreted within, rather than across, reward proxies.