Paper deep dive
Post-Training at the Edge of Detectability: A Game-Theoretic Approach to Fine-Tuning
Keegan Harris, Brian W. Lee, Ian Waudby-Smith, Philip Amortila, Nika Haghtalab, Michael I. Jordan
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 8/4/2026, 10:30:44 AM
Summary
The paper proposes a game-theoretic framework for fine-tuning large language models, modeling the process as a sequential detection game between an agent maximizing reward and a monitor detecting deviations from a reference policy. The authors prove that the Nash equilibrium of this game corresponds to KL-regularized reinforcement learning, where the regularization coefficient is optimally determined by maximizing the ratio of expected reward to statistical distinguishability (KL divergence). They introduce a stochastic bisection algorithm to learn this optimal coefficient and demonstrate its effectiveness in continual learning and model auditing scenarios using Qwen3-8B and Llama-3.2-1B.
Entities (8)
Relation Signals (6)
Sequential Detection Game → yieldsequilibriumpolicy → KL-regularized RL
confidence 95% · we show that the resulting equilibrium policy can nonetheless be expressed as the solution to a KL-regularized RL problem
Nash Equilibrium → maximizes → Reward per unit of statistical distinguishability
confidence 94% · optimal regularization parameter that can be viewed as maximizing reward per unit of statistical distinguishability
Stochastic Bisection Algorithm → learnsparameterfor → KL-regularized RL
confidence 93% · we present a stochastic bisection algorithm ... that estimates β⋆ ... via reduction to the KL-regularized RL objective
SPRT → isbestresponsefor → Monitor
confidence 92% · the monitor’s best-response to agent policy ... is to test for it using an SPRT
Qwen3-8b → usedinexperiment → Sequential Detection Game
confidence 90% · In experiments with Qwen3-8B ... we demonstrate that our methods result in competitive reward-retention trade-offs
Llama 3.2-1B → usedinexperiment → Sequential Detection Game
confidence 90% · In experiments with ... Llama-3.2-1B, we demonstrate that our methods result in competitive reward-retention trade-offs
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Reinforcement learning (RL) fine-tuning is widely used in language model training to improve model performance on a target task while limiting drift from a reference policy. A standard way to balance this trade-off is via a KL-regularized RL objective, although this formulation does not by itself provide a principled way to set the regularization coefficient. In practice, the coefficient is typically chosen heuristically or via hyperparameter search, which can lead to unnecessary overhead in training cost or undesirable reward-retention trade-offs. We instead propose a game-theoretic framework that gives this trade-off an explicit statistical interpretation. Specifically, we study a sequential game in which an agent chooses a policy to maximize cumulative reward while a monitor observes policy outputs over time and tests for deviations from the reference policy. Although not originating from the same perspective, we show that the resulting equilibrium policy can nonetheless be expressed as the solution to a KL-regularized RL problem for an optimal regularization parameter that can be viewed as maximizing reward per unit of statistical distinguishability. Drawing on classical results from concave-convex fractional programming, we provide a principled method for learning this equilibrium coefficient via reduction to the KL-regularized RL objective, thus allowing for flexible integration into standard fine-tuning pipelines. In experiments with Qwen3-8B and Llama-3.2-1B, we demonstrate that our methods result in competitive reward-retention trade-offs in a continual learning setting, and illustrate how our framework may be used to audit API providers serving open-source models.
Tags
Links
- Source: https://arxiv.org/abs/2607.26358v1
- Canonical: https://arxiv.org/abs/2607.26358v1
Trouble viewing inline? Open PDF directly →
Full Text
84,027 characters extracted from source content.
Expand or collapse full text
@key newfloatplacement @keynewfloatname @keynewfloatfileext @keynewfloatwithin Post-Training at the Edge of Detectability: A Game-Theoretic Approach to Fine-Tuning Keegan Harris University of California, Berkeley Brian W. Lee University of California, Berkeley Ian Waudby-Smith University of California, Berkeley Philip Amortila University of California, Berkeley Nika Haghtalab University of California, Berkeley Michael I. Jordan University of California, Berkeley Inria & École Normale Supérieure Abstract Reinforcement learning (RL) fine-tuning is widely used in language model training to improve model performance on a target task while limiting drift from a reference policy. A standard way to balance this trade-off is via a KL-regularized RL objective, although this formulation does not by itself provide a principled way to set the regularization coefficient. In practice, the coefficient is typically chosen heuristically or via hyperparameter search, which can lead to unnecessary overhead in training cost or undesirable reward–retention trade-offs. We instead propose a game-theoretic framework that gives this trade-off an explicit statistical interpretation. Specifically, we study a sequential game in which an agent chooses a policy to maximize cumulative reward while a monitor observes policy outputs over time and tests for deviations from the reference policy. Although not originating from the same perspective, we show that the resulting equilibrium policy can nonetheless be expressed as the solution to a KL-regularized RL problem for an optimal regularization parameter that can be viewed as maximizing reward per unit of statistical distinguishability. Drawing on classical results from concave-convex fractional programming, we provide a principled method for learning this equilibrium coefficient via reduction to the KL-regularized RL objective, thus allowing for flexible integration into standard fine-tuning pipelines. In experiments with Qwen3-8B and Llama-3.2-1B, we demonstrate that our methods result in competitive reward-retention trade-offs in a continual learning setting, and illustrate how our framework may be used to audit API providers serving open-source models. 1 Introduction A common goal in large language model (LLM) fine-tuning is to improve performance on a target objective while preserving useful behaviors inherited from a reference policy. These objectives may be at odds with one another, and there are many possible ways to formalize the resulting trade-off. One of the most widely used approaches is Kullback–Leibler (KL)-regularized reinforcement learning (RL), which augments the reward objective with a penalty for deviating from the reference policy (jaques2017sequence; jaques2019way; neu2017unified; ziegler2019fine; stiennon2020learning; ouyang2022training). In this formulation, the regularization coefficient β≥0β≥ 0 determines how aggressively the fine-tuned policy is penalized for deviating from the reference policy (e.g., a pre-trained LLM). When β is too small, fine-tuning may increase reward at the cost of significantly changing the model’s behavior. When β is too large, the model remains close to the reference policy, but may fail to adequately learn the target task. In practice, this coefficient is often selected by manual tuning or via grid search (see, e.g., ouyang2022training; zhang2023wisdom; lin2024mitigating; tang2024understanding). However this approach can waste compute and lead to undesirable reward–retention trade-offs. Some implementations instead adapt β online to match a prescribed KL target, but this still requires the learner to specify the target divergence and update heuristic in advance (schulman2017proximal; ziegler2019fine). We consider an alternative notion of deviation from the reference policy. Rather than measuring deviations directly through a distance or similarity metric, we ask how difficult it is to distinguish the fine-tuned policy from the reference policy based on its outputs. This naturally leads to a game between an agent who seeks to maximize reward, and a monitor attempting to detect deviations from the reference policy. In this game, an agent deploys a policy to maximize their utility, while a monitor observes policy outputs as they are generated and aims to detect whether those outputs have been generated by the (intended) reference policy or by some other (unintended) one. If the monitor detects a deviation from the reference policy, deployment is terminated. The agent therefore faces a trade-off between increasing reward and remaining statistically indistinguishable from the reference policy. We call this the sequential detection game. 1.1 Our Contributions Game-theoretic formulation for RL fine-tuning (Section 3). We provide a new perspective on RL fine-tuning through the lens of sequential detection. Starting from a game-theoretic formulation, we prove that the agent’s Nash equilibrium strategy in the sequential detection game is to fine-tune the reference policy with the standard KL-regularized RL objective. Thus, rather than introducing a new fine-tuning objective, our game-theoretic formulation recovers one that is already used in practice, which allows existing RL fine-tuning algorithms to be applied directly. At the same time, it gives the KL penalty an operational interpretation as the statistical cost of remaining difficult to distinguish from the reference policy under sequential monitoring. Moreover, the equilibrium identifies the regularization coefficient β⋆β that optimally trades off between reward maximization and statistical distinguishability. Unlike conventional RL fine-tuning, where β is chosen heuristically, the equilibrium coefficient is determined solely by the reward function, prompt distribution, and reference policy. Learning the optimal regularization strength (Section 4). Building on our equilibrium analysis, we present a stochastic bisection algorithm (Algorithm 1) that estimates β⋆β to ϵε-precision by solving O(log(1/ϵ))O( (1/ε)) RL sub-problems. Algorithm 1 may be interpreted as a game-theoretically principled approach to RL fine-tuning: rather than relying on a separate hyperparameter search, it adaptively computes the equilibrium regularization strength. The analysis of Algorithm 1 combines classical techniques from the literature on fractional programming (dinkelbach1967nonlinear; schaible1976fractional) and sequential hypothesis testing (robbins1974expected). Continual learning and model auditing experiments (Section 5). We evaluate our framework in both continual learning and model auditing settings using Qwen3-8B (yang2025qwen3) and Llama-3.2-1B (meta2024llama321b). In continual learning, we find preliminary evidence that the equilibrium regularization coefficient yields better reward–retention trade-offs than compute-matched manual tuning. In model auditing, we show that the monitor’s equilibrium-prescribed test detects hidden model modifications by open-source API providers more quickly than natural baselines while controlling the type-I error rate. 2 Preliminaries We now review the key ingredients from RL fine-tuning, sequential hypothesis testing, and game theory that will later be combined in our analysis. RL fine-tuning. Consider a prompt space X and response space Y. A policy π is a mapping from prompts to distributions over responses, i.e., π:→Δ()π:X→ (Y). We use π(y|x)π(y|x) to denote the probability of receiving response y given prompt x, and y∼π(⋅|x)y π(·|x) to denote a sample from the conditional distribution over responses given x.111LLMs generate tokens auto-regressively, but for our purposes it suffices to consider only the induced distribution over complete responses. Given a reference policy πref _ref, a reward function r:×→ℝr:X×Y , the KL-regularized RL fine-tuning objective for a regularization parameter β≥0β≥ 0 is defined by maxπy∼π(⋅|x)[r(x,y)]−β⋅[KL(π(⋅|x),πref(⋅|x))], _π \E_y π(·|x)[r(x,y)]-β·E [KL(π(·|x), _ref(·|x)) ] \, (1) where KL(π(⋅|x),πref(⋅|x)):=∑y∈π(y|x)logπ(y|x)πref(y|x)KL(π(·|x), _ref(·|x)):= _y π(y|x) π(y|x) _ref(y|x) is the KL divergence between π(⋅|x)π(·|x) and πref(⋅|x) _ref(·|x), and expectations are (also) taken over the prompt distribution D. This optimization admits a closed-form optimal solution (donsker1975variational; rafailov2023direct) of the form πβ(y|x):=πref(y|x)exp(r(x,y)/β)Zβ(x), _β(y|x):= _ref(y|x) (r(x,y)/β)Z_β(x), (2) where Zβ(x):=y∼πref(⋅|x)[exp(r(x,y)/β)]Z_β(x):=E_y _ref(·|x)[ (r(x,y)/β)] is the partition function. We call πβ _β the tilt of πref _ref. Since Zβ(x)Z_β(x) is generally intractable to compute due to the size of Y, RL methods like Proximal Policy Optimization (schulman2017proximal) and Group Relative Policy Optimization (shao2024deepseekmath) are typically used to approximately solve Equation (1). Sequential hypothesis testing. Sequential hypothesis testing is a paradigm of statistical inference for which type-I errors (false positive rates) are controlled not only at pre-determined sample sizes, but also at stopping times (wald1945sequential; wald1947sequential). Informally, sequential hypothesis tests allow analysts to “peek” at their data routinely to adaptively stop experiments and make conclusions for data-dependent reasons. In the simplest case of simple nulls and simple alternatives, one sees a sequence of observations z1,z2,…z_1,z_2,… from a distribution PzP_z and the goal is to determine whether Pz=P_z=P (the null) or Pz=QP_z=Q (the alternative). A sequential hypothesis test, ϕ≡(ϕt)t∈ℕφ≡( _t)_t , is a sequence of functions ϕt≡ϕ(z1,…,zt) _t≡φ(z_1,…,z_t) outputting either “reject” or “do not reject” at each time step t∈ℕt . For a fixed α∈(0,1)α∈(0,1), a test is said to be α-correct if it controls the type-I error at all sample sizes simultaneously; i.e., P(∃t∈ℕ:ϕt rejects)≤αP(∃\;t : _t rejects)≤α. Letting τα≡τα(ϕ)=inft∈ℕ:ϕt rejects _α≡ _α(φ)= \t : _t rejects\ be the stopping time, notice that α-correctness is equivalent to the condition P(τα<∞)≤αP( _α<∞)≤α. A test is said to be power-one if Q(τ<∞)=1Q(τ<∞)=1. It is a classical result that for any α-correct, power-one sequential test, Q[τα]≥log(1/α)KL(Q,P)E_Q[ _α]≥ (1/α)KL(Q,P); see wald1945sequential and robbins1974expected. This lower bound is known to be tight in the high-confidence regime, in the sense that there exist valid α-correct, power-one tests for which limα↓0Q[τα]log(1/α)=1KL(Q,P). _α 0 E_Q[ _α] (1/α)= 1KL(Q,P). (3) A canonical test with this leading-order behavior is the sequential probability ratio test (SPRT) (wald1945sequential) given by ϕt=∏i=1tdQ(zi)dP(zi)≥1α. _t= 1 \ _i=1^t dQ(z_i)dP(z_i)≥ 1α \. Nash equilibria. A two-player game consists of strategy spaces 1S_1, 2S_2 and utility functions u1,u2:1×2→ℝu_1,u_2:S_1×S_2 . Given a strategy s2∈2s_2 _2 for player 2, a strategy s1∈1s_1 _1 is a best response for player 1 if s1∈argmaxs1′∈1u1(s1′,s2).s_1∈ _s_1 _1u_1(s_1 ,s_2). Best responses for player 2 are defined analogously. A strategy profile (s1∗,s2∗)(s_1^*,s_2^*) is a Nash equilibrium if both players’ strategies are simultaneously best responses: u1(s1∗,s2∗)≥u1(s1,s2∗)for all s1∈1and u2(s1∗,s2∗)≥u2(s1∗,s2)for all s2∈2.u_1(s_1^*,s_2^*)≥ u_1(s_1,s_2^*)\;\;for all s_1 _1 u_2(s_1^*,s_2^*)≥ u_2(s_1^*,s_2)\;\;for all s_2 _2. Thus, at a Nash equilibrium, neither player can improve their utility by unilaterally changing their strategy; i.e., each player’s behavior is optimal given the behavior of the other. 3 The Sequential Detection Game We explore differences between reference policies and target policies from the perspective of a monitor aiming to quickly distinguish one from the other in a two-player (general sum) game against a strategic agent. Sequential detection game: Consider a setup where a monitor intends to deploy a policy πref _ref while an agent potentially deploys a policy π~≠πref π≠ _ref in place of πref _ref. The monitor intends to detect such a change from prompt-response outputs while controlling the type-I error rate at a desired level α∈(0,1)α∈(0,1). Concretely, for each time step t∈ℕt : 1. A prompt xt∼x_t is generated from the prompt distribution D and a response yt∼π~(⋅∣xt)y_t π(· x_t) is generated from the agent’s deployed policy. 2. The monitor updates a sequential test ϕt≡ϕ((x1,y1),…,(xt,yt)) _t≡φ((x_1,y_1),…,(x_t,y_t)) subject to the type-I error constraint: ℙy∼πref(∃t∈ℕ:ϕt rejects)≤αP_y _ref (∃\;t : _t rejects )≤α. 3. If ϕt _t rejects, then the monitor stops deployment, and the agent receives utility ∑i=1tr(xi,yi) _i=1^tr(x_i,y_i) according to reward function r:×→ℝr:X×Y , while the monitor receives utility −t-t. In this sequential detection game, the monitor stops deployment if and only if ϕt _t rejects. In other words, their stopping rule is characterized by the stopping time τα=inft∈ℕ:ϕt rejects _α= \t : _t rejects\, with the convention that τα=∞ _α=∞ if ϕt _t never rejects. From this vantage point, the agent wishes to deploy a policy π~ π to maximize their expected cumulative reward before detection, y∼π~(⋅∣x)[∑i=1ταr(xi,yi)]E_y π(· x)[ _i=1 _αr(x_i,y_i)], while the monitor wishes to minimize y∼π~(⋅∣x)[τα]E_y π(· x)[ _α]. The agent’s optimal choice of π~ π turns out to be precisely a KL-regularized tilt of the reference policy, while the monitor’s best response turns out to be a likelihood ratio test between πref _ref and π~ π. To state these facts formally, we first require two assumptions on the agent’s reward function r. Assumption 3.1 (The agent can and must strategize to benefit). It is possible for the agent to achieve positive expected utility. That is, there exists a policy π such that y∼π(⋅|x)[r(x,y)]>0E_y π(·|x)[r(x,y)]>0. Moreover, they must strategize in order to do so, i.e., μref:=y∼πref(⋅|x)[r(x,y)]≤0 _ref:=E_y _ref(·|x)[r(x,y)]≤ 0. Assumption 3.2 (The monitor can always devise a detection strategy). For every π′≠πrefπ ≠ _ref, there exists a test (ϕt′)t∈ℕ( _t )_t with a finite expected stopping time, i.e., π′[τα(ϕ′)]<∞E_π [ _α(φ )]<∞. Assumption 3.1 provides sufficient conditions for the agent to want to participate in the game and to learn a non-trivial strategy.222If the latter half of 3.1 does not hold, the agent can deploy policy πref _ref and collect infinite reward. For the necessity of strategization in Assumption 3.1, it is mathematically equivalent to assume that μref−c≤0 _ref-c≤ 0, where c≥0c≥ 0 is the minimum improvement in expected reward required for the agent to want to update the reference policy. The constant c may also be interpreted as capturing the cost associated with model training (e.g., monetary costs, time spent, effort exerted, etc.). Assumption 3.2 is an analogous assumption on the monitor’s incentive to play the game. It is a weak assumption that is satisfied for likelihood ratios in all but pathological cases which we eschew for the purposes of this paper. In what follows, we characterize the equilibrium of the sequential detection game. These statements should be interpreted as holding in the so-called “high-confidence” regime, i.e., where α↓0α 0. This is because the sequential hypothesis tests we consider take place in discrete time: they observe one (prompt, response) pair at a time and can only stop after an integer number of observations. robbins1974expected provide an information-theoretic lower bound log(1/α)/KL(Q,P) (1/α)/KL(Q,P) for the expected stopping time of any α-correct sequential test. This bound is tight at the leading log(1/α) (1/α) order, but the corresponding upper bounds can include lower-order boundary-crossing effects as an artifact of the discrete-time nature of the test (siegmund1985sequential). Therefore the following equilibrium characterization can be thought of as stating that unilateral deviations cannot improve either player’s utility at the leading log(1/α) (1/α) order. Equilibrium characterization. We begin with the monitor’s equilibrium strategy. Suppose the agent is playing policy π~ π. Targeting the true agent policy π~ π maximizes the expected evidence accumulated per sample and therefore minimizes the expected stopping time. Moreover, it is known that SPRTs exhibit the optimal leading order behavior of Equation 3. Therefore the monitor’s best-response to agent policy π~ π is to test for it using an SPRT. The agent’s goal is to pick their policy to maximize [∑t=1ταr(xt,yt)]E [ _t=1 _αr(x_t,y_t) ], which can be written as [τα]⋅y∼π(⋅|x)[r(x,y)]E[ _α]·E_y π(·|x)[r(x,y)] by Wald’s equation, since the expected stopping time will be finite under Assumptions 3.1 and 3.2. By the robbins1974expected lower bound, the agent can guarantee themselves utility at least log(1/α)y∼π(⋅|x)[r(x,y)]/[KL(π(⋅|x),πref(⋅|x))] (1/α)E_y π(·|x)[r(x,y)]/E[KL(π(·|x), _ref(·|x))], which is also an (asymptotic) upper bound on their utility under the monitor’s best-response SPRT. Since neither y∼π(⋅|x)[r(x,y)]E_y π(·|x)[r(x,y)] nor [KL(π(⋅|x),πref(⋅|x))]E[KL(π(·|x), _ref(·|x))] depend on α, the agent’s optimization takes the form maxπy∼π(⋅|x)[r(x,y)][KL(π(⋅|x),πref(⋅|x))], _π E_y π(·|x)[r(x,y)]E[KL(π(·|x), _ref(·|x))], (4) where we use the convention that 0/0=00/0=0. Equation (4) is a fractional program, with a concave numerator and a convex, strictly positive denominator (excluding the singular point π=πrefπ= _ref, which cannot be optimal under Assumption 3.1). As such, we can draw on the rich literature on concave-convex fractional programming (see, e.g., dinkelbach1967nonlinear; schaible1976fractional), which shows that while such fractional optimization problems are not concave, they are quasi-concave and can be solved iteratively through a sequence of concave optimization problems. Theorem 3.3. Suppose that for each iteration k, the quantities π(k)π^(k) and β(k)β^(k) are chosen by π(k)∈argmaxπy∼π(⋅|x)[r(x,y)]−β(k)[KL(π(⋅|x),πref(⋅|x))]π^(k)∈ *argmax_π \E_y π(·|x)[r(x,y)]-β^(k)E[KL(π(·|x), _ref(·|x))] \ and β(k+1)←y∼π(k)(⋅|x)[r(x,y)][KL(π(k)(⋅|x),πref(⋅|x))]β^(k+1)← E_y π^(k)(·|x)[r(x,y)]E[KL(π^(k)(·|x), _ref(·|x))]. Then it holds that β(k)→β⋆β^(k)→β , where πβ⋆∈argmaxπy∼π(⋅∣x)[r(x,y)]−β⋆[KL(π(⋅∣x),πref(⋅∣x)] _β ∈ *argmax_π \E_y π(· x)[r(x,y)]-β E[KL(π(· x), _ref(· x)] \ and πβ⋆ _β solves the fractional program (4). Notice that πβ⋆ _β is precisely the optimizer of the RL fine-tuning objective in Equation (1) with regularization coefficient β⋆β . In other words, the optimal policy for the agent is the reference policy πref _ref tilted by β⋆β as in (2). Discussion. Taken together, the results in this section imply that the agent playing policy πβ⋆ _β and the monitor testing for policy πβ⋆ _β is a Nash equilibrium in the sequential detection game. The equilibrium characterization yields a game-theoretic derivation for KL-regularized RL, and provides an operational interpretation of the optimal regularization coefficient as the one which maximizes reward per unit of statistical distinguishability from the reference policy. In this sense, the KL penalty is not just a convenient proxy for behavioral preservation, but the “shadow price” of remaining difficult to detect under sequential monitoring. This equilibrium characterization is also robust to tie-breaking. In Appendix A.1, we show that while πβ⋆ _β is not a unique agent best-response, the monitor’s SPRT with ℋ1:π=πβ⋆H_1:π= _β is still an α-correct, power-one test for any agent best response. Therefore, the monitor’s test remains valid even if the agent breaks ties in an unknown or arbitrary way. 4 Reduction to KL-Regularized RL We now turn our attention from characterizing the equilibrium policy πβ⋆ _β to learning it. Traditional fractional programming approaches cannot be applied off-the-shelf, as they rely on the ability to evaluate quantities like y∼π(⋅|x)[r(x,y)]E_y π(·|x)[r(x,y)] and [KL(π(⋅|x),πref(⋅|x))]E[KL(π(·|x), _ref(·|x))] exactly for an arbitrary policy π, which is generally not feasible when π is an LLM. Instead, we show that β⋆β can be characterized as the unique root of a one-dimensional function and estimate it from data using a stochastic bisection procedure, in the spirit of stochastic root-finding methods (robbins1951stochastic). Consider the function M(β):=y∼πβ(⋅|x)[r(x,y)]−β⋅[KL(πβ(⋅|x),πref(⋅|x))].M(β):=E_y _β(·|x)[r(x,y)]-β·E[KL( _β(·|x), _ref(·|x))]. Intuitively, M(β)M(β) measures whether the reward that is obtained by deviating from the reference policy exceeds the statistical cost imposed by the KL penalty for the given β. When β is too small, the reward dominates and M(β)>0M(β)>0. When β is too large, the KL penalty dominates and M(β)<0M(β)<0. The equilibrium coefficient β⋆β is precisely the point where these forces balance. Therefore, computing β⋆β reduces to finding the unique root of M(β)M(β). This justification follows from an analysis of dinkelbach1967nonlinear (see Lemma A.1). Since M(β)M(β) is not available in closed form, we use E to denote the sample average and instead consider the empirical estimate of M(β)M(β) using n≥1n≥ 1 samples: M^n(β):=^y∼πβ(⋅|x)[r(x,y)]−β⋅^[KL(πβ(⋅|x),πref(⋅|x))]. M_n(β):= E_y _β(·|x)[r(x,y)]-β· E[KL( _β(·|x), _ref(·|x))]. Our algorithm proceeds as follows: Set βlow=0 _low=0. Given an upper bound βhigh≥β⋆ _high≥β , set β^:=0.5(βhigh+βlow) β:=0.5( _high+ _low) and collect enough samples n such that M^n(β^)±rad(β^,n,δ) M_n( β) ( β,n,δ) is bounded away from zero, where rad(β^,n,δ)rad( β,n,δ) is a confidence radius that holds with probability 1−δ1-δ. Depending on the sign of M^n(β^) M_n( β), set either βhigh←β _high← β or βlow←β _low← β, and repeat this process until |βhigh−βlow|| _high- _low| is within the desired precision ϵε. The full procedure is outlined in Algorithm 1. Input: ϵ>0ε>0, δ∈(0,1)δ∈(0,1), βhigh∈ℝ+ _high _+ Set βlow=0 _low=0 while βhigh−βlow>ϵ _high- _low>ε do β^=12(βhigh+βlow) β= 12( _high+ _low), n=100n=100 πβ^←RL-Oracle(πref,β^) _ β← RL-Oracle( _ref, β) while TrueTrue do if M^n(β^)+rad(β^,n,δ)<0 M_n( β)+rad( β,n,δ)<0 then βhigh←β _high← β; break else if M^n(β^)−rad(β^,n,δ)>0 M_n( β)-rad( β,n,δ)>0 then βlow←β _low← β; break else n←2n← 2n end if end while end while return RL-Oracle(πref,βlow) RL-Oracle( _ref, _low) ALGORITHM 1 Stochastic bisection method Theorem 4.1. If βhigh≥β⋆ _high≥β and for every β queried by Algorithm 1, ℙ(∃n≥1:|M^n(β)−M(β)|>rad(β,n,δ))≤δ,P (∃\;n≥ 1: | M_n(β)-M(β) |>rad(β,n,δ) )≤δ, then with probability at least 1−δlog2(βhigh/ϵ)1-δ _2 ( _high/ε ), Algorithm 1 returns a policy πβlow _ _low satisfying 0≤β⋆−βlow≤ϵ0≤β - _low≤ε in log2(βhigh/ϵ) _2 ( _high/ε ) bisection iterations. We instantiate Algorithm 1 with specific choices of (βhigh,rad(⋅,⋅,⋅))( _high,rad(·,·,·)) in the Appendix (Corollaries B.5 and B.7). In either instantiation, Algorithm 1 needs n=O(M(β^)−2)n=O(M( β)^-2) samples in each iteration in order to determine the sign of M(β^)M( β) with high probability. Therefore, as β^→β⋆ β→β and |M(β^)|→0|M( β)|→ 0, we should expect the number of samples we need in each iteration of stochastic bisection to increase. As a practical relaxation, we forego the confidence sequence in our experiments and bisect based on the sign of M^n(β^) M_n( β) after observing a sufficiently large number of samples (e.g., n=4096n=4096). Algorithm 1 posits access to an “RL oracle” which, given a reference policy πref _ref and regularization parameter β, returns the KL-regularized tilt πβ _β. While exact, polynomial-time oracles generally do not exist due to the difficulty of computing the partition function, there is a rich literature of approximate oracles that have been designed to solve the KL-regularized RL problem (e.g., PPO, GRPO, and variants thereof). Consequently, our theoretical guarantees for Algorithm 1 should be interpreted as characterizing the complexity of identifying the equilibrium coefficient, assuming that each RL subproblem can be solved sufficiently accurately. Warm-start version of Algorithm 1. It is possible to warm start Algorithm 1 given an arbitrary setting of βhigh _high (i.e., one that is not guaranteed to satisfy βhigh>β⋆ _high>β ) as follows: Compute M^n(βhigh)+rad(βhigh,n,δ) M_n( _high)+rad( _high,n,δ). If it is bounded below zero, run Algorithm 1 as normal. Otherwise set βlow←βhigh _low← _high, βhigh←2βhigh _high← 2 _high, and repeat. This procedure will produce an ε -approximate solution with high probability. Furthermore, it makes log2(βhigh/ϵ) _2( _high/ε) calls to the RL oracle if βhigh _high is a valid upper bound on β⋆β . If it is not, the number of oracle calls is still O(log2(σ2/(|μref|ϵ)))O ( _2 (σ^2/(| _ref|ε) ) ) if rewards are σ-sub-Gaussian by Lemma B.3. Finally, it is worth noting that in our game formulation, even if the agent and the monitor run separate copies of Algorithm 1, the agent and monitor utilities will still be close to their equilibrium values. See Appendix B.1 for more details. 5 Experiments We evaluate our methods empirically in two settings: a continual learning task and a model auditing task. In the continual learning task, the goal is to improve performance on a new target reward while preserving existing behavioral properties of the reference policy. In the model auditing task, an auditor tries to detect whether a model has been strategically fine-tuned from its outputs. Taken together, these experiments test the two main operational interpretations of our framework: stochastic bisection as a replacement for manual KL coefficient search, and the equilibrium likelihood-ratio test as a practical auditing rule. Experiment setup. The two sets of experiments use the same setup. In each round, the language model is given the same prompt (“Once upon a time”) and generates a completion, with the agent receiving reward equal to the number of characters in the response. Before bisection, the reward is calibrated by sampling completions from the reference policy and subtracting off the raw mean reward, plus a fixed margin ρ. Under the reference policy, the calibrated reward therefore has mean approximately equal to −ρ-ρ. The calibrated reward is used throughout KL-constrained fine-tuning and bisection; because adding a constant to the reward does not change the optimal policy at any fixed β, this shift does not alter the fine-tuning objective. Evaluation metrics on Pareto plots are reported in the original, uncalibrated units. 5.1 Continual Learning Figure 1: Reward–retention trade-offs for Qwen3-8B with shift ρ=0.1ρ=0.1. The vertical axis reports response length in tokens, and the horizontal axes report narrative and grammatical coherence, as judged by GPT-5-mini (singh2025openai). Blue circles show policies trained on the compute-matched exponential grid over β, the gray square denotes the reference policy; and the orange circle and green diamond denote two independent stochastic-bisection runs. Each point is the average of 400400 independent responses. Higher values are better on both axes. (a) Llama-3.2-1B with shift ρ=0.1ρ=0.1. (b) Llama-3.2-1B with shift ρ=0.2ρ=0.2. Figure 2: Reward–retention trade-offs for Llama-3.2-1B with shift ρ=0.1ρ=0.1 (left) and ρ=0.2ρ=0.2 (right). Plotting conventions, axis labels, and plot titles are the same as in Figure 1. In both settings, Algorithm 1 selects policies near the elbow of the empirical trade-off curve traced out by the compute-matched β grid. In our continual learning setup, the response-length reward function plays the role of the skill to be improved, while the reference policy’s ability to produce coherent responses is the behavior to be retained. Although this is a single model update rather than a multi-task setting, it isolates the central continual learning trade-off: aggressively optimizing a new objective can alter or degrade useful behaviors inherited from the reference policy.333This is commonly observed in the literature on RLHF and is often referred to as the “alignment tax” (e.g., ouyang2022training; lin2024mitigating; zhang2024cppo; jang2024degeneration). For each trained policy, we measure the target behavior using token count and retention using both narrative and grammatical coherence.444We used GPT-5-mini and GPT-5-nano as judges to evaluate narrative and grammatical coherence (zheng2023judging). All main body plots use GPT-5-mini. Results using GPT-5-nano are very similar and are in Appendix C. We compare Algorithm 1 against the standard practice of manually sweeping over the regularization coefficient. This baseline trains one policy for each value in an exponentially spaced grid over β, using the same number of fine-tuning runs as Algorithm 1. After inspecting the results, a practitioner could select whichever grid point best matches their desired balance between length and coherence. Our results are summarized in Figures 1 and 2. Higher values are desirable on both axes, so policies toward the upper-right corner of each panel achieve more favorable reward–retention trade-offs. The points labeled Agent and Monitor are produced by two independent runs of Algorithm 1. We retain these labels for consistency with the auditing experiment in Section 5.2; in this experiment, the two points illustrate the variation induced by GRPO RL oracle calls. Blue circles correspond to points on an exponential grid between 0 and βhigh _high, and the reference policy is denoted by a gray square. Figure 1 shows the results for Qwen3-8B with shift ρ=0.1ρ=0.1. Policies that generate the longest responses exhibit lower coherence, while policies that preserve the highest narrative and grammatical coherence produce shorter responses. Interestingly, narrative coherence actually increases with token count up to a point, before decreasing once the token count gets too large. Each run of Algorithm 1 selects an intermediate policy near the bends of the empirical trade-off curves, in contrast to the compute-matched grid, which results in models closer to either extreme. Figure 2 shows the corresponding results for Llama-3.2-1B with shifts ρ=0.1ρ=0.1 and ρ=0.2ρ=0.2. The same qualitative pattern holds in both settings: the bisection policies lie near the “elbow” of the curve traced out by the grid, avoiding both the high-retention regime in which little progress is made on the length objective, and the high-length regime in which coherence falls sharply.555Llama-3.2-1B scores considerably lower than Qwen3-8B in terms of narrative and grammatical coherence for all values of β, and does not exhibit the same improvement in narrative coherence as token count increases. This is likely due to it being a much smaller model. Takeaways. Across all settings, we find that Algorithm 1 selects a single policy near the elbow of the empirical reward–retention frontier. In some instances (e.g., Figure 1, left), Algorithm 1 finds desirable regions of the frontier that are not reached by grid search. However even when Algorithm 1 and grid search find similar parts of the frontier (e.g., Figure 2(b)), grid search requires an additional post-hoc selection step, unlike Algorithm 1 which resolves this trade-off by adaptively concentrating its fine-tuning runs around the equilibrium coefficient. 5.2 Model Auditing Table 1: Model-auditing results at level α=0.05α=0.05. πA _A SPRT uses the true deployed policy and is thus an oracle comparator; πM _M SPRT uses the monitor’s independently trained equilibrium policy. Lower stopping times and false positive rates are better. Averages are taken over 100100 samples. Model and margin Metric πA _A SPRT πM _M SPRT Grid mixture Qwen3-8B, ρ=0.1ρ=0.1 Avg. stopping time 1.00 1.08 1.02 False positive rate 0.00 0.00 0.09 Llama-3.2-1B, ρ=0.01ρ=0.01 Avg. stopping time 2.49 6.61 20.36 False positive rate 0.04 0.06 0.20 Llama-3.2-1B, ρ=0.1ρ=0.1 Avg. stopping time 1.89 5.31 9.47 False positive rate 0.03 0.01 0.15 Llama-3.2-1B, ρ=0.2ρ=0.2 Avg. stopping time 1.75 4.40 6.23 False positive rate 0.04 0.03 0.09 We now turn our attention to the auditing interpretation of the sequential detection game. Consider a model-hosting platform that sets inference costs such that they are profitable on a per-token basis. Such a platform has an economic incentive to fine-tune its models to produce unnecessarily long responses, while remaining difficult to distinguish from the advertised model (velasco2026test; cao2026pay). A third-party auditor (e.g., a watchdog organization) can easily observe the model’s outputs, but verifying the parameters of the deployed model is significantly more difficult. Nevertheless, the auditor may want to detect this strategic behavior when it exists, while controlling the probability of falsely flagging the model when it does not. Our theory precisely characterizes the equilibrium of this game, and gives the auditor a way to empirically test for this type of bad behavior. Starting from the same reference policy and using the token length reward described above, the agent and monitor independently run Algorithm 1 with the same RL oracle budget. Let πA _A denote the policy obtained by the agent and let πM _M denote the policy obtained (independently) by the monitor/auditor. The agent deploys πA _A, whereas the monitor uses πM _M as the alternative hypothesis in its sequential test. In other words, the auditor knows the reference policy and the incentive being optimized, but they do not know the agent’s realized policy or share its fine-tuning randomness. Each auditing trial uses a source policy πsrc∈πA,πref _src∈\ _A, _ref\. Sampling from πA _A represents the presence of strategic fine-tuning, whereas πref _ref represents honest deployment. At time t, a completion yt∼πsrc(⋅|x)y_t _src(·|x) is generated and scored under both πM _M and πref _ref. The monitor uses the SPRT which rejects if Lt>log(1/α)L_t> (1/α) for any t≥1t≥ 1, where Lt:=∑s=1tlog(πM(ys|x)/πref(ys|x))L_t:= _s=1^t ( _M(y_s|x)/ _ref(y_s|x)) and the log likelihood is computed by summing the token-level log probabilities along the realized completion. We compare the equilibrium test against a uniform mixture likelihood ratio test (described in Appendix C). We also compare against the “skyline” SPRT which uses the agent’s actual deployed policy as its alternative. While the agent’s policy is not available to the auditor, it gives a lower bound on the detection speed attainable by any SPRT. Finally, all tests are evaluated on the same sequence of completions from πsrc _src; the candidate policies are used only to score these observations. Takeaways. Table 1 summarizes the results. For Qwen3-8B, the strategic modification is sufficiently pronounced that every test detects it after approximately one observation. The agent SPRT stops after exactly one completion on average, while the independently trained monitor stops after 1.081.08 completions. The uniform mixture test obtains a slightly better stopping time of 1.021.02, albeit at the cost of a false positive rate that is higher than the 0.050.05 threshold. The differences between testing strategies are clearer for Llama-3.2-1B. As expected, the oracle agent SPRT detects the deployed policy fastest. Nevertheless, the monitor test is significantly faster than the uniform mixture test at every calibration margin. The composite grid tests also reject more frequently under the reference policy, with empirical false-positive rates between 0.080.08 and 0.20.2, compared to the monitor’s false positive rate which is between 0.010.01 and 0.060.06. 6 Related Work Reward–KL trade-offs in RL fine-tuning. KL regularization is a central component of modern LLM fine-tuning, particularly in reinforcement learning from human feedback, where it is used to improve a target reward while limiting drift from a reference model. Several works study the trade-off between reward and KL regularization empirically (see, e.g., bai2022training; gao2023scaling). korbak2022rl provide a complementary theoretical interpretation of KL-regularized RL through the lens of variational inference, where the reference model is analogous to a prior and the reward function provides evidence. Game-theoretic applications of hypothesis testing. A small but growing line of work uses statistical hypothesis testing procedures as strategic instruments in games. gauthier2026betting use sequential tests to monitor whether agents in a multi-agent system are behaving consistently with equilibrium play. capitaine2026test use sequential testing in infinitely repeated games, where they show a “test-then-punish strategy” can sustain cooperation by detecting deviations and triggering punishment. hu2024game are closer in spirit, as they also study game-theoretic hypothesis testing against strategic evasion. They model an attacker who manipulates observations to evade a Neyman–Pearson test and characterize equilibrium distortion strategies and detector responses in Stackelberg and signaling games. Our settings differ in both the object being manipulated and the resulting equilibrium structure. Continual learning. Our experiments in Section 5.1 are instances of continual learning, where the goal is to learn new tasks while avoiding degradation on previously learned tasks. sun2019lamol uses LLMs to generate pseudo-samples of previous tasks for training alongside data for the new task. razdaibiedina2023progressive avoid modifying the reference policy by learning task-specific “soft prompts” and concatenating newly learned prompts with earlier ones. zhang2024cppo study continual learning in RLHF and use sample-wise “balance weights” to regulate the trade-off between policy learning and knowledge retention. Like us, zhang2023copf explicitly consider policies that are tilted from a previous task’s policy, but they treat the KL coefficient as an externally chosen hyperparameter. See wu2024continual and shi2025continual for surveys on continual learning for LLMs. AI auditing. Our experiments in Section 5.2 are related to work on language model auditing, where the goal is to provide independent assurance that LLMs operate safely, ethically, and in compliance with legal or organizational standards (mokander2024auditing). Several recent works formalize auditing as a statistical testing problem over model outputs. gao2025model introduce model equality testing, which asks whether a black-box API is serving the same model as a claimed reference model. richter2025auditing frame auditing for behavioral shifts in LLMs as a sequential testing problem. Our auditing setup is aligned with this statistical view, but differs in that the audited model is itself trained strategically to avoid detection. Strategic learning. More broadly, our work belongs to the literature on strategic machine learning, where the deployment of a learning algorithm influences the behavior of other agents. Strategic classification studies settings in which individuals manipulate their features to receive favorable decisions (hardt2016strategic), while performative prediction (perdomo2020performative) analyzes learning problems where the underlying data distribution changes in response to the deployed model. Subsequent work extends these ideas to multi-step settings, where strategic responses can accumulate over time (harris2021stateful; brown2022performative), like in our setting. 7 Conclusions and Future Research We have introduced the sequential detection game, a game-theoretic framework for RL fine-tuning in which an agent seeks to maximize reward while remaining difficult to distinguish from a reference policy. We have shown that the Nash equilibrium of this game recovers the standard KL-regularized RL objective while determining the regularization coefficient endogenously, giving it an explicit operational interpretation rather than treating it as a tunable hyperparameter. We then developed a stochastic bisection algorithm for estimating the equilibrium coefficient and present preliminary empirical evidence that it yields competitive reward–retention trade-offs in a continual learning setting, while also enabling principled auditing of language model APIs. There are several further directions that are worth pursuing in this vein. Unknown agent motives. Our framework provides a principled way to do black-box auditing of model API platforms when their reward function is known to the auditor. While knowledge of the reward function may be reasonable in settings where the platform’s incentives are public knowledge, it would be interesting to extend our equilibrium characterization to settings where the auditor is uncertain about the underlying reward function. Antidistillation sampling. Another promising application of our framework is antidistillation sampling (savani2026antidistillation), where the goal is to modify a model’s next-token probabilities in order to make it harder to distill the model’s capabilities. savani2026antidistillation do this by tilting the reference policy with a specially designed reward function that makes model distillation more difficult. They treat the regularization coefficient as a tunable parameter to control the trade-off between accuracy and (anti-)distillability, but it would be interesting to use our framework to automatically balance between the two. Further empirical work is needed to close the gap between our equilibrium analysis and large-scale RL fine-tuning in practice. More broadly, we hope that this work encourages the development of practical fine-tuning methods whose hyperparameters are derived from principled operational objectives, rather than chosen heuristically. Acknowledgments KH was supported in part by the Simons Institute for the Theory of Computing, and part of this work was conducted when he was visiting the Institute. IW-S gratefully acknowledges support from the Miller Institute for Basic Research in Science. PA gratefully acknowledges the support of DARPA through award No. HR00112520022. This work was also funded by the European Union (ERC-2022-SYG-OCEAN-101071601), the NSF Institute for Foundations of Machine Learning under grant CCF-2505865, the National Science Foundation under grants CCF-2145898, the Office of Naval Research under grant N00014-24-1-2159, a Google Research Scholar Award, an Alfred P. Sloan fellowship, and a Schmidt Science AI2050 fellowship. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency. Neither the European Union nor the granting authority can be held responsible for them. References Appendix A Appendix for Section 3: The Sequential Detection Game Lemma A.1 (dinkelbach1967nonlinear). The following properties are true for M(β)M(β): 1. M(β⋆)=0M(β )=0 2. M(β)<0M(β)<0 for all β>β⋆β>β 3. M(β)>0M(β)>0 for all β<β⋆β<β 4. M(β)M(β) is convex. Lemma A.2. Let τα:=infτ:∑t=1τlogdQdP(Xt)≥log(1/α) _α:= \τ\;:\; _t=1^τ dQdP(X_t)≥ (1/α)\ and assume that Q<<PQ<<P, R[τα]<∞E_R[ _α]<∞, and R[logdQdP(X)]<∞E_R[ dQdP(X)]<∞. Then R[τα]≥log(1/α)R[logdQdP(X)].E_R[ _α]≥ (1/α)E_R[ dQdP(X)]. Proof. By Wald’s identity, R[∑t=1ταlogdQdP(Xt)]=R[τα]⋅R[logdQdP(X)].E_R [ _t=1 _α dQdP(X_t) ]=E_R[ _α]·E_R [ dQdP(X) ]. Our definition of τα _α implies that R[τα]⋅R[logdQdP(X)]≥log(1/α).E_R[ _α]·E_R [ dQdP(X) ]≥ (1/α). Dividing both sides by R[logdQdP(X)]E_R[ dQdP(X)] obtains the result. ∎ A.1 Robustness to tie-breaking While πβ⋆ _β is an optimal policy for the agent to play against the monitor’s equilibrium SPRT, it is not the only one. Theorem A.3. Let d(π):=[KL(π(⋅|x),πref(⋅|x))−KL(π(⋅|x),πβ⋆(⋅|x))]d(π):=E[KL(π(·|x), _ref(·|x))-KL(π(·|x), _β (·|x))]. Suppose that τα:=infτ:∑t=1τlogπβ⋆(xt,yt)πref(xt,yt)≥log(1/α) _α:= \τ\;:\; _t=1^τ _β (x_t,y_t) _ref(x_t,y_t)≥ (1/α) \ and the agent plays policy π such that d(π)>0d(π)>0. We have that maxπxt∼,yt∼π(⋅|xt)[∑t=1ταr(xt,yt)]≥β⋆log(1/α). _πE_x_t ,y_t π(·|x_t) [ _t=1 _αr(x_t,y_t) ]≥β (1/α). Proof. We have that maxπxt∼,yt∼π(⋅|xt)[∑t=1ταr(xt,yt)]=maxπxt∼,yt∼π(⋅|xt)[τα]⋅y∼π(⋅|x)[r(x,y)], _πE_x_t ,y_t π(·|x_t) [ _t=1 _αr(x_t,y_t) ]= _πE_x_t ,y_t π(·|x_t)[ _α]·E_y π(·|x)[r(x,y)], where we know that d(π)>0d(π)>0 (and hence y∼π(⋅|x)[τα]<∞E_y π(·|x)[ _α]<∞) by Assumption 3.1. Applying Lemma A.2, we have that maxπxt∼,yt∼π(⋅|xt)[∑t=1ταr(xt,yt)]≥maxπy∼π(⋅|x)[r(x,y)]log(1/α)y∼π(⋅|x)[logπβ⋆(y|x)πref(y|x)] _πE_x_t ,y_t π(·|x_t) [ _t=1 _αr(x_t,y_t) ]≥ _π E_y π(·|x)[r(x,y)] (1/α)E_y π(·|x) [ _β (y|x) _ref(y|x) ] The joint distribution induced by πβ⋆ _β is absolutely continuous with respect to πref _ref since πβ⋆ _β is a tilt of πref _ref. Focusing on the denominator, we have that y∼π(⋅|x)[logπβ⋆(y|x)πref(y|x)]=1β⋆y∼π(⋅|x)[r(x,y)]−[logZβ⋆(x)]E_y π(·|x) [ _β (y|x) _ref(y|x) ]= 1β E_y π(·|x)[r(x,y)]-E[ Z_β (x)] Setting this aside for a moment, recall that M(β⋆) M(β ) =y∼πβ⋆(⋅|x)[r(x,y)]−β⋆[KL(πβ⋆(⋅|x),πref(⋅|x))] =E_y _β (·|x)[r(x,y)]-β E[KL( _β (·|x), _ref(·|x))] =β⋆[logZβ⋆(x)]=0 =β E[ Z_β (x)]=0 where the last equality follows from Lemma A.1. Since we know that β⋆>0β >0 by Assumption 3.1, we can conclude that [logZβ⋆(x)]=0E[ Z_β (x)]=0. Substituting this all back in, we can conclude that y∼π(⋅|x)[r(x,y)]log(1/α)y∼π(⋅|x)[logπβ⋆(y|x)πref(y|x)]=β⋆log(1/α), E_y π(·|x)[r(x,y)] (1/α)E_y π(·|x) [ _β (y|x) _ref(y|x) ]=β (1/α), which is independent of the agent’s policy π as long as d(π)>0d(π)>0. ∎ Therefore, we can conclude that if the monitor is deploying a sequential likelihood ratio test between πβ⋆ _β and πref _ref, the agent can guarantee themselves utility at least β⋆log(1/α)β (1/α) by playing any policy π such that d(π)>0d(π)>0 (including π=πβ⋆π= _β ). Moreover as α↓0α 0, this is the optimal utility attainable for the agent under this test. The test with stopping time τα:=infτ:∑t=1τlogπβ⋆(xt,yt)πref(xt,yt)≥log(1/α) _α:= \τ\;:\; _t=1^τ _β (x_t,y_t) _ref(x_t,y_t)≥ (1/α)\ is also a valid sequential test for testing the null hypothesis ℋ0:π=πrefH_0\;:\;π= _ref versus the composite alternative ℋ1:π is s.t. d(π)>0H_1\;:\;π is s.t. d(π)>0. To see this, let Lt=logπβ⋆(yt|xt)πref(yt|xt)L_t= _β (y_t|x_t) _ref(y_t|x_t) and note that under ℋ0H_0, exp(∑s=1tLs) ( _s=1^tL_s ) is a non-negative martingale with mean 11. By Ville’s inequality, we have that ℙℋ0(supt>0exp(∑s=1tLs)≥1α)≤α,P_H_0 ( _t>0 ( _s=1^tL_s )≥ 1α )≤α, which implies that ℙℋ0(τα<∞)≤αP_H_0( _α<∞)≤α. Under ℋ1H_1, y∼π(⋅|x)[Lt]=d(π)>0E_y π(·|x)[L_t]=d(π)>0, so by the law of large numbers we have that ∑s=1tLs→∞ _s=1^tL_s→∞ almost surely, and so τα<∞ _α<∞ almost surely. Therefore, even if the agent breaks ties among best-response policies in an arbitrary or unknown way, this remains a valid power-1 sequential test for the composite alternative ℋ1:π is s.t. d(π)>0H_1\;:\;π is s.t. d(π)>0. Appendix B Appendix for Section 4: Reduction to KL-Regularized RL Assumption B.1 (Sub-Gaussian rewards). Rewards are sub-Gaussian with parameter σ, i.e., for any π and all λ∈ℝλ , logy∼π(⋅|x)[exp(λ(r(x,y)−μ))]≤σ2λ22, _y π(·|x) [ (λ(r(x,y)-μ)) ]≤ σ^2λ^22, where μ:=y∼π(⋅|x)[r(x,y)]μ:=E_y π(·|x)[r(x,y)]. Lemma B.2 (howard2021time). Given a sequence of i.i.d. observations (X)n=1∞(X)_n=1^∞ from a σ-sub-Gaussian distribution, we have that |1n∑i=1nXi−[X]|≤1.7σ0.72log(10.4/δ)+loglog(2n)n | 1n _i=1^nX_i-E[X] |≤ 1.7σ 0.72 (10.4/δ)+ (2n)n with probability at least 1−δ1-δ, simultaneously for every n≥1n≥ 1. Lemma B.3. Under Assumption B.1, β⋆≤σ22|μref|β ≤ σ^22| _ref|. Proof. Consider the Donsker-Varadhan variational formula [donsker1975variational], which says that for two distributions P, Q such that P≪QP Q and function f which is measurable, P[f]≤logQ[exp(f)]+KL(P,Q).E_P[f]≤ _Q[ (f)]+KL(P,Q). Let P be the joint distribution over (x,y)(x,y) such that x∼x , y∼π(⋅|x)y π(·|x) for some arbitrary policy π, Q be the joint distribution such that x∼x , y∼πref(⋅|x)y _ref(·|x), and f=λ(r(x,y)−μref)f=λ(r(x,y)- _ref) for some arbitrary λ∈ℝλ . Plugging this in, we get that y∼π(⋅|x)[r(x,y)]−μref≤1λ(logy∼πref(⋅|x)[exp(λ(r(x,y)−μref))]+[KL(π(⋅|x),πref(⋅|x))]).E_y π(·|x)[r(x,y)]- _ref≤ 1λ ( _y _ref(·|x)[ (λ(r(x,y)- _ref))]+E[KL(π(·|x), _ref(·|x))] ). Observe that P≪QP Q is implicit in our setting as if this fails to hold, then KL(π∥πref)=∞KL(π\| _ref)=∞ so such a policy π could never be optimal. Applying Assumption B.1, we have that y∼π(⋅|x)[r(x,y)]−μref _y π(·|x)[r(x,y)]- _ref ≤1λ(σ2λ22+[KL(π(⋅|x),πref(⋅|x))]) ≤ 1λ ( σ^2λ^22+E[KL(π(·|x), _ref(·|x))] ) =σ2λ2+1λ[KL(π(⋅|x),πref(⋅|x))]. = σ^2λ2+ 1λE[KL(π(·|x), _ref(·|x))]. Since λ is arbitrary, we can pick it to minimize the right hand side. Applying first-order conditions, we get that the optimal choice of λ is λ=2[KL(π(⋅|x),πref(⋅|x))]σ2.λ= 2E[KL(π(·|x), _ref(·|x))]σ^2. Plugging this into our bound, we get that y∼π(⋅|x)[r(x,y)]−μref≤2σ2[KL(π(⋅|x),πref(⋅|x))].E_y π(·|x)[r(x,y)]- _ref≤ 2σ^2E[KL(π(·|x), _ref(·|x))]. Rearranging terms and subtracting β[KL(π(⋅|x),πref(⋅|x))] [KL(π(·|x), _ref(·|x))] from each side of the inequality, we get y∼π(⋅|x)[r(x,y)]−β[KL(π(⋅|x),πref(⋅|x))] _y π(·|x)[r(x,y)]- [KL(π(·|x), _ref(·|x))] ≤μref+2σ[KL(π(⋅|x),πref(⋅|x))] ≤ _ref+ 2σ E[KL(π(·|x), _ref(·|x))] −β[KL(π(⋅|x),πref(⋅|x))]. - [KL(π(·|x), _ref(·|x))]. Our choice of π was arbitrary, so we take the max over π on both sides to get M(β) M(β) ≤μref+maxπ(2σ[KL(π(⋅|x),πref(⋅|x))]−β[KL(π(⋅|x),πref(⋅|x))]) ≤ _ref+ _π ( 2σ E[KL(π(·|x), _ref(·|x))]- [KL(π(·|x), _ref(·|x))] ) =μref+maxk≥0(2σk−βk). = _ref+ _k≥ 0( 2σ k-β k). Since 2σk−βk 2σ k-β k is concave in k, we can apply the first order condition to compute the maximum, which is obtained at k=σ22β2k= σ^22β^2. Substituting this back in, we get that M(β)≤μref+σ22β.M(β)≤ _ref+ σ^22β. Using this inequality, we can see that M(β)<0M(β)<0 if μref+σ22β<0 _ref+ σ^22β<0, or equivalently, if β>σ2−2μrefβ> σ^2-2 _ref. Finally, taking the contrapositive, we get that if M(β)≥0M(β)≥ 0, then β≤σ2−2μrefβ≤ σ^2-2 _ref. Since M(β⋆)=0M(β )=0, we can conclude that β⋆≤σ2−2μref=σ22|μref|β ≤ σ^2-2 _ref= σ^22| _ref|. ∎ See 4.1 Proof. The proof consists of two separate parts: 1. Showing the correctness of Algorithm 1, conditioned on the clean event 2. Bounding the number of loop iterations in Algorithm 1 (and therefore the number of high probability events) Correctness: βlow=0 _low=0 is a valid lower bound on β⋆β , since Assumption 3.1 implies that it is possible for the agent to get non-negative cumulative expected reward. βhigh=σ22|μref| _high= σ^22| _ref| is a valid upper bound on β⋆β by Lemma B.3. Under the clean event, M(β^)<0M( β)<0 if M^n(β^)+rad(β^,n,δ)<0 M_n( β)+rad( β,n,δ)<0, so we can conclude that βhigh>β^>β⋆ _high> β>β by parts 1 and 2 of Lemma A.1. On the other hand, if M^n(β^)−rad(β^,n,δ)>0 M_n( β)-rad( β,n,δ)>0 then M(β^)>0M( β)>0 under the clean event, so we can conclude that βlow<β^<β⋆ _low< β<β by parts 1 and 3 of Lemma A.1. Therefore since M(β)M(β) is a continuous function on the interval [βlow,βhigh][ _low, _high], under the clean event, βhigh−βlow≤ϵ _high- _low≤ε and β⋆∈[βlow,βhigh]β ∈[ _low, _high] once the while loop terminates. Number of iterations: Our bisection method takes log2(βhigh−βlowϵ)=log2(σ22|μref|ϵ) _2( _high- _lowε)= _2( σ^22| _ref|ε) iterations to ensure that the final parameter returned is within an additive error of ϵε. Assuming a confidence sequence on M^n(β) M_n(β) that is valid for all n≥1n≥ 1 and any fixed β, our high probability statement requires taking the union bound over each bisection method iteration. ∎ Lemma B.4. Under Assumption B.1, if πref _ref is such that πref(ι|h)≥γ _ref( |h)≥γ for any token ι∈ and token prefix h, then for any fixed β, |M^n(β)−M(β)|≤1.7(σ+βmlog(1/γ))0.72log(20.8/δ)+loglog(2n)n,| M_n(β)-M(β)|≤ 1.7 (σ+β m (1/γ) ) 0.72 (20.8/δ)+ (2n)n, with probability at least 1−δ1-δ simultaneously for all n≥1n≥ 1, where m is the maximum token output sequence length. Therefore, it suffices to set rad(β,n,δ)=1.7(σ+βmlog(1/γ))0.72log(20.8/δ)+loglog(2n)nrad(β,n,δ)=1.7 (σ+β m (1/γ) ) 0.72 (20.8/δ)+ (2n)n in Algorithm 1. Proof. For an arbitrary policy π with x1,…,xn∼x_1,…,x_n and yi∼π(⋅|xi)y_i π(·|x_i), we have that |1n∑i=1nr(xi,yi)−y∼π(⋅|x)[r(x,y)]|≤1.7σ0.72log(10.4/δ)+loglog(2n)n, | 1n _i=1^nr(x_i,y_i)-E_y π(·|x)[r(x,y)] |≤ 1.7σ 0.72 (10.4/δ)+ (2n)n, with probability at least 1−δ1-δ, simultaneously for every n≥1n≥ 1 by Lemma B.2. Turning our attention to the KL term, we have that [KL(π(⋅|x),πref(⋅|x))] [KL(π(·|x), _ref(·|x))] =y∼π(⋅|x)[∑j=1mlog(π(yj|y1:j−1,x)πref(yj|y1:j−1,x))] =E_y π(·|x) [ _j=1^m ( π(y_j|y_1:j-1,x) _ref(y_j|y_1:j-1,x) ) ] =[∑j=1my1:j−1∼π(⋅|x)[∑ι∈π(ι|y1:j−1,x)log(π(ι|y1:j−1,x)πref(ι|y1:j−1,x))]]. =E [ _j=1^mE_y_1:j-1 π(·|x) [ _ π( |y_1:j-1,x) ( π( |y_1:j-1,x) _ref( |y_1:j-1,x) ) ] ]. Let Gj(x)=y1:j−1∼π(⋅|x)[∑ι∈π(ι|y1:j−1,x)log(π(ι|y1:j−1,x)πref(ι|y1:j−1,x))]G_j(x)=E_y_1:j-1 π(·|x) [ _ π( |y_1:j-1,x) ( π( |y_1:j-1,x) _ref( |y_1:j-1,x) ) ], and observe that 0≤Gj(x)≤log(1/γ)0≤ G_j(x)≤ (1/γ). Consider the one-sample estimator G^j(x):=∑ι∈π(ι|y1:j−1,x)log(π(ι|y1:j−1,x)πref(ι|y1:j−1,x)) G_j(x):= _ π( |y_1:j-1,x) ( π( |y_1:j-1,x) _ref( |y_1:j-1,x) ) and note that it is unbiased and computable given access to π, πref _ref, x, and y∼π(⋅|x)y π(·|x). We can estimate [KL(π(⋅|x),πref(⋅|x))]=[∑j=1mGj(x)]E[KL(π(·|x), _ref(·|x))]=E[ _j=1^mG_j(x)] as 1n∑i=1n∑j=1mG^j(xi) 1n _i=1^n _j=1^m G_j(x_i). Observing that ∑j=1mGj(x) _j=1^mG_j(x) is sub-Gaussian with parameter at most mlog(1/γ)m (1/γ), we can apply Lemma B.2 to get |1n∑i=1n∑j=1mG^j(xi)−[KL(π(⋅|x),πref(⋅|x))]|≤1.7mlog(1/γ)0.72log(10.4/δ)+loglog(2n)n, | 1n _i=1^n _j=1^m G_j(x_i)-E[KL(π(·|x), _ref(·|x))] |≤ 1.7m (1/γ) 0.72 (10.4/δ)+ (2n)n, with probability at least 1−δ1-δ, simultaneously for all n≥1n≥ 1. Combining this with our high-probability guarantee for y∼π(⋅|x)[r(x,y)]E_y π(·|x)[r(x,y)] and taking a union bound gets us the desired result. ∎ Corollary B.5. Under Assumption B.1, if βhigh=σ22|μref| _high= σ^22| _ref|, πref(ι|h)≥γ>0 _ref( |h)≥γ>0 for any token ι∈ and token prefix h, and rad(β,n,δ)=1.7(σ+βmlog(1/γ))(0.72log(20.8/δ)+loglog(2n))/n,rad(β,n,δ)=1.7 (σ+β m (1/γ) ) (0.72 (20.8/δ)+ (2n))/n, then with probability at least 1−δlog2(σ22|μref|ϵ)1-δ _2 ( σ^22| _ref|ε ), Algorithm 1 returns a policy πβlow _ _low satisfying 0≤β⋆−βlow≤ϵ0≤β - _low≤ε in log2(σ22|μref|ϵ) _2 ( σ^22| _ref|ε ) bisection iterations. Proof. Corollary B.5 follows immediately from Theorem 4.1 and Lemma B.4. ∎ Assumption B.6 (Conditional sub-Gaussianity). Rewards satisfy Assumption B.1. Moreover, under the reference policy, rewards are uniformly conditionally sub-Gaussian, i.e., for every prompt x∈x and all λ∈ℝλ , logy∼πref(⋅|x)[exp(λ(r(x,y)−μref(x)))]≤σ2λ22, _y _ref(·|x) [ (λ (r(x,y)- _ ref(x) ) ) ]≤ σ^2λ^22, where μref(x):=y∼πref(⋅|x)[r(x,y)] _ref(x):=E_y _ref(·|x)[r(x,y)]. Corollary B.7. Under Assumption B.6, if rad(β,n,δ)=1.7σ(2+σ28β2)0.72log(10.4/δ)+loglog(2n)n,rad(β,n,δ)=1.7σ (2+ σ^28β^2 ) 0.72 (10.4/δ)+ (2n)n, and βhigh=σ22|μref| _high= σ^22| _ref|, then Algorithm 1 returns a policy πβlow _ _low satisfying 0≤β⋆−βlow≤ϵ0≤β - _low≤ε with probability at least 1−δlog2(σ22|μref|ϵ)1-δ _2 ( σ^22| _ ref|ε ) in log2(σ22|μref|ϵ) _2 ( σ^22| _ref|ε ) bisection iterations. Proof. We have that logπβ(y|x)πref(y|x)=r(x,y)β−logZβ(x). _β(y|x) _ref(y|x)= r(x,y)β- Z_β(x). and so for each fixed x, KL(πβ(⋅|x),πref(⋅|x))=y∼πβ(⋅|x)[r(x,y)β−logZβ(x)],KL( _β(·|x), _ref(·|x))=E_y _β(·|x) [ r(x,y)β- Z_β(x) ], which implies that M(β)=[βlogZβ(x)]M(β)=E[β Z_β(x)]. By Jensen’s inequality and Assumption B.6, we have that 0≤βlogZβ(x)−y∼πref(⋅|x)[r(x,y)]≤σ22β0≤β Z_β(x)-E_y _ref(·|x)[r(x,y)]≤ σ^22β and y∼πref(⋅|x)[r(x,y)]−μrefE_y _ref(·|x)[r(x,y)]- _ref is σ-sub-Gaussian over x∼x . Hence by Cauchy–Schwarz and Hoeffding’s lemma, exp(λ(βlogZβ(x)−M(β)))≤exp(λ22(2σ2+σ48β2)).E (λ (β Z_β(x)-M(β) ) )≤ ( λ^22 (2σ^2+ σ^48β^2 ) ). Applying Lemma B.2 with sub-Gaussian parameter 2σ2+σ48β2 2σ^2+ σ^48β^2 gives the desired confidence sequence. The result now follows from Theorem 4.1. ∎ B.1 Approximate Equilibria Theorem B.8. The following guarantees hold when the monitor and the agent use the tilted policy πβlow _ _low returned by Algorithm 1: 1. If the monitor is deploying an SPRT between πβlow _ _low and πref _ref, the agent can guarantee themselves utility at least βlowlog(1/α) _low (1/α) by playing any policy π such that y∼π(⋅|x)[logπβlow(y|x)πref(y|x)]>0E_y π(·|x) [ _ _low(y|x) _ref(y|x) ]>0 (including π=πβlowπ= _ _low).666Recall that 0≤β⋆−βlow≤ϵ0≤β - _low≤ε with high probability, by Theorem 4.1. 2. If the agent is playing policy πβlow _ _low and the monitor is playing a power-one simple-vs-simple SPRT, then their best response is to test ℋ0:π=πrefH_0\;:\;π= _ref versus ℋ1:π=πβlowH_1\;:\;π= _ _low. Moreover, as α↓0α 0, this is the optimal such test out of all power-one sequential tests for the monitor to play [wald1948optimum]. Proof. maxπyt∼π(⋅|xt)[∑t=1ταr(xt,yt)] _πE_y_t π(·|x_t) [ _t=1 _αr(x_t,y_t) ] =maxπyt∼π(⋅|x)[τα]⋅y∼π(⋅|x)[r(x,y)] = _πE_y_t π(·|x)[ _α]·E_y π(·|x)[r(x,y)] ≥maxπEy∼π(⋅|x)[r(x,y)]log(1/α)y∼π(⋅|x)[logπβlow(y|x)πref(y|x)] ≥ _π E_y π(·|x)[r(x,y)] (1/α)E_y π(·|x) [ _ _low(y|x) _ref(y|x) ] where the inequality follows from Lemma A.2 and the fact that βlow>0 _low>0 with high probability. We can rewrite y∼π(⋅|x)[logπβlow(y|x)πref(y|x)]=1βlowy∼π(⋅|x)[r(x,y)]−1βlowM(βlow).E_y π(·|x)[ _ _low(y|x) _ref(y|x)]= 1 _lowE_y π(·|x)[r(x,y)]- 1 _lowM( _low). Since βlow≤β⋆ _low≤β , we know that M(βlow)>0M( _low)>0, and so maxπEy∼π(⋅|x)[r(x,y)]log(1/α)y∼π(⋅|x)[logπβlow(y|x)πref(y|x)]≥βlowlog(1/α) _π E_y π(·|x)[r(x,y)] (1/α)E_y π(·|x)[ _ _low(y|x) _ref(y|x)]≥ _low (1/α) ∎ Observe that even if each agent runs Algorithm 1 separately, if they stop after the same number of iterations they will arrive at the same value for βlow _low with high probability. With that being said, it is still possible to say something about the quality of the equilibrium whenever the monitor and the agent run Algorithm 1 for a different number of iterations. Theorem B.9. Let βA _A (resp. βM _M) be the agent’s (resp. monitor’s) computation of βlow _low and suppose that βA≤βM _A≤ _M, i.e., the clean event holds and the monitor runs Algorithm 1 for at least as long as the agent. Then: 1. If the monitor is deploying a SPRT between πβM _ _M and πref _ref, the agent can guarantee themselves utility at least (β⋆−ϵ)log(1/α)(β -ε) (1/α) by playing policy πβA _ _A. 2. If the agent is playing policy πβA _ _A and the monitor plays a power-one simple SPRT between ℋ0:π=πrefH_0:π= _ref and ℋ1:π=πβMH_1:π= _ _M, then as α↓0α 0 τ∗≤y∼πβA[τα]≤βMβA⋅τ∗,τ^* _y _ _A[ _α]≤ _M _A·τ^*, where τ∗τ^* is the best possible expected stopping time. Proof. Part 1: The key step is to show that y∼πA(⋅|x)[logπM(y|x)πref(y|x)]>0E_y _A(·|x)[ _M(y|x) _ref(y|x)]>0. To see this, observe that y∼πβA(⋅|x)[logπβM(y|x)πref(y|x)]=1βM(y∼πβA(⋅|x)[r(x,y)]−M(βM))E_y _ _A(·|x) [ _ _M(y|x) _ref(y|x) ]= 1 _M(E_y _ _A(·|x)[r(x,y)]-M( _M)) and y∼πβA(⋅|x)[r(x,y)]=M(βA)+βA[KL(πβA(⋅|x),πref(⋅|x))]>M(βA)≥M(βM)E_y _ _A(·|x)[r(x,y)]=M( _A)+ _AE[KL( _ _A(·|x), _ref(·|x))]>M( _A)≥ M( _M) where the last inequality follows from the fact that βA≤βM _A≤ _M. Given this, we can bound yt∼πβA(⋅|xt)[∑t=1ταr(xt,yt)] _y_t _ _A(·|x_t) [ _t=1 _αr(x_t,y_t) ] =yt∼πβA(⋅|x)[τα]⋅y∼πβA(⋅|x)[r(x,y)] =E_y_t _ _A(·|x)[ _α]·E_y _ _A(·|x)[r(x,y)] ≥Ey∼πβA(⋅|x)[r(x,y)]log(1/α)y∼πβA(⋅|x)[logπβM(y|x)πref(y|x)] ≥ E_y _ _A(·|x)[r(x,y)] (1/α)E_y _ _A(·|x) [ _ _M(y|x) _ref(y|x) ] =Ey∼πβA(⋅|x)[r(x,y)]log(1/α)1βMy∼πβA(⋅|x)[r(x,y)]−1βMM(βM) = E_y _ _A(·|x)[r(x,y)] (1/α) 1 _ME_y _ _A(·|x)[r(x,y)]- 1 _MM( _M) ≥βMlog(1/α)≥(β⋆−ϵ)log(1/α) ≥ _M (1/α)≥(β -ε) (1/α) Part 2: Let τα(βM) _α( _M) be the stopping time when the monitor is using a simple versus simple SPRT with alternative hypothesis π=πβMπ= _ _M, and τα(βA) _α( _A) be the counterfactual stopping time when using alternative hypothesis π=πβAπ= _ _A. Consider the limit where α↓0α 0. We have that y∼πβA(⋅|x)[logπβM(⋅|x)πref(⋅|x)] _y _ _A(·|x) [ _ _M(·|x) _ref(·|x) ] =1βMy∼πβA(⋅|x)[r(x,y)]−1βMM(βM) = 1 _ME_y _ _A(·|x)[r(x,y)]- 1 _MM( _M) ≥1βMy∼πβA(⋅|x)[r(x,y)]−1βMM(βA) ≥ 1 _ME_y _ _A(·|x)[r(x,y)]- 1 _MM( _A) =βAβM(1βAy∼πβA(⋅|x)[r(x,y)]−1βAM(βA)) = _A _M ( 1 _AE_y _ _A(·|x)[r(x,y)]- 1 _AM( _A) ) Therefore, y∼πβA[τα]τ∗ E_y _ _A[ _α]τ^* =y∼πβA[τα(βM)]y∼πβA[τα(βA)] = E_y _ _A[ _α( _M)]E_y _ _A[ _α( _A)] =y∼πβA(⋅|x)[logπβA(y|x)πref(y|x)]y∼πβA(⋅|x)[logπβM(y|x)πref(y|x)]≤βMβA. = E_y _ _A(·|x) [ _ _A(y|x) _ref(y|x) ]E_y _ _A(·|x) [ _ _M(y|x) _ref(y|x) ]≤ _M _A. ∎ Appendix C Appendix for Section 5: Experiments Policies are fine-tuned with rank 16 LoRA adapters [hu2022lora] using GRPO [shao2024deepseekmath] as the RL oracle and Adam [kingma2014adam] as the optimizer with learning rate 1e-4, temperature 1.01.0, top-p 1.01.0, and a completion cap of 2048 tokens. Each call to the RL oracle consists of 15 optimization steps with 400 rollouts each. All models were trained using the Tinker API [thinkingmachines2025tinker]. C.1 Continual Learning Judge prompts. For each trained policy, we evaluated generated completions using two separate LLM-judge prompts: one for narrative coherence and one for grammatical coherence. The same prompts were used for both GPT-5-mini and GPT-5-nano. Narrative coherence system prompt: You are an expert literary critic evaluating short stories written by a small language model. Your task is to rate the NARRATIVE COHERENCE of a story on an integer scale from 0 to 10. Narrative coherence means: The story has a recognizable beginning, middle, and end. Events follow each other in a sensible causal or temporal order. Characters and settings remain consistent throughout. The story arrives at some kind of resolution or conclusion. Ignore grammar and spelling mistakes – those are evaluated separately. Focus only on the structure and coherence of the narrative. Respond with ONLY a single integer between 0 and 10, with no other text. 0 = no discernible narrative; 10 = a complete, well-structured short story. Grammatical coherence system prompt: You are an expert linguist evaluating short stories written by a small language model. Your task is to rate the GRAMMATICAL COHERENCE of a story on an integer scale from 0 to 10. Grammatical coherence means: Sentences are syntactically well-formed. Subject-verb agreement, tense, pronouns, and articles are used correctly. Punctuation and capitalization roughly follow standard English conventions. Words are spelled correctly and used in plausible contexts. Ignore plot quality and narrative structure – those are evaluated separately. Focus only on syntactic and morphological correctness. Respond with ONLY a single integer between 0 and 10, with no other text. 0 = essentially ungrammatical; 10 = fully grammatical, idiomatic English. Figure 3: Reward–retention trade-offs for Llama-3.2-1B with shift ρ=0.01ρ=0.01. Plotting conventions are the same as in Figure 1. In both settings, Algorithm 1 selects policies near the elbow of the empirical trade-off curve traced out by the compute-matched β grid. Figure 3 reports the additional Llama-3.2-1B experiment with calibration margin ρ=0.01ρ=0.01, using GPT-5-mini as the judge. Figure 4: Reward–retention trade-offs for Qwen3-8B with shift ρ=0.1ρ=0.1 and GPT-5-nano-as-a-judge. All other plotting conventions are the same as in Figure 1. Figure 5: Reward–retention trade-offs for Llama-3.2-1B with shift ρ=0.01ρ=0.01 and GPT-5-nano-as-a-judge. Figure 6: Reward–retention trade-offs for Llama-3.2-1B with shift ρ=0.1ρ=0.1 and GPT-5-nano-as-a-judge. Figure 7: Reward–retention trade-offs for Llama-3.2-1B with shift ρ=0.2ρ=0.2 and GPT-5-nano-as-a-judge. Figures 4–7 repeat the continual-learning experiments using GPT-5-nano as the judge. The absolute coherence scores differ slightly from the GPT-5-mini scores, but the qualitative pattern is unchanged: Across Qwen3-8B and Llama-3.2-1B, and across the tested calibration margins, the stochastic-bisection policies fall in the transition region of the Pareto frontier rather than at either extreme of the compute-matched β grid. Plot details. This subsection contains the β values that are used in all figures, and the order in which they are plotted. For Qwen3-8B, all β values are listed in clockwise order. For Llama-3.2-1B, all β values are listed from left to right. Qwen3-8B with shift ρ=0.1ρ=0.1 details: • Figure 1, left: 0.000580.00058, 2.693e−052.693e-05, 0.0001250.000125, 0.002690.00269, 0.00.0, 5.802e−065.802e-06, 0.0085450.008545 (monitor), 0.0085450.008545 (agent), 0.01250.0125, 0.05800.0580, 0.26930.2693, 1.251.25, base model. • Figure 1, right: 0.000580.00058, 5.802e−065.802e-06, 0.002690.00269, 2.693e−052.693e-05, 0.00.0, 0.0001250.000125, 0.0085450.008545 (monitor), 0.0085450.008545 (agent), 0.01250.0125, 0.05800.0580, 0.26930.2693, 1.251.25, base model. • Figure 4, left: 0.000580.00058, 0.00.0, 2.693e−052.693e-05, 0.002690.00269, 5.802e−065.802e-06, 0.0001250.000125, 0.0085450.008545 (monitor), 0.0085450.008545 (agent), 0.01250.0125, 0.05800.0580, 1.251.25, 0.26930.2693, base model. • Figure 4, right: 0.00.0, 2.693e−052.693e-05, 0.000580.00058, 0.0001250.000125, 0.002690.00269, 5.802e−065.802e-06, 0.0085450.008545 (monitor), 0.0085450.008545 (agent), 0.01250.0125, 0.05800.0580, 1.251.25, 0.26930.2693, base model. Llama-3.2-1B with shift ρ=0.01ρ=0.01 details: • Figure 3, left: 0.00040.0004, 0.008840.00884, 0.00.0, 0.00190.0019, 8.8437e−058.8437e-05, 0.0410.041, 0.0930.093 (agent), 0.0930.093 (monitor), 0.190.19, base model, 4.104.10, 0.8840.884, 19.0519.05 • Figure 3, right: 0.00040.0004, 0.00.0, 0.008840.00884, 8.8437e−058.8437e-05, 0.00190.0019, 0.0410.041, 0.0930.093 (agent), 0.0930.093 (monitor), 0.190.19, base model, 0.8840.884, 19.0519.05, 4.104.10 • Figure 5, left: 0.00040.0004, 0.00190.0019, 0.00.0, 0.008840.00884, 8.8437e−058.8437e-05, 0.0410.041, 0.0930.093 (monitor), 0.0930.093 (agent), 0.190.19, base model, 0.8840.884, 19.0519.05, 4.104.10 • Figure 5, right: 0.00040.0004, 0.00.0, 0.00190.0019, 0.008840.00884, 8.8437e−058.8437e-05, 0.0410.041, 0.0930.093 (agent), 0.0930.093 (monitor), 0.190.19, base model, 0.8840.884, 19.0519.05, 4.104.10 Llama-3.2-1B with shift ρ=0.1ρ=0.1 details: • Figure 2(a), left: 0.00.0, 0.00420.0042, 4.202e−054.202e-05, 0.0001950.000195, 0.0009050.000905, 9.0536e−069.0536e-06, 0.01950.0195, 0.083810.08381 (monitor), 0.083810.08381 (agent), base model, 0.09050.0905, 0.42020.4202, 1.951.95 • Figure 2(a), right: 9.0536e−069.0536e-06, 0.00420.0042, 4.202e−054.202e-05, 0.0001950.000195, 0.00.0, 0.0009050.000905, 0.01950.0195, 0.083810.08381 (monitor), 0.083810.08381 (agent), 0.09050.0905, base model, 0.42020.4202, 1.951.95 • Figure 6, left: 0.00420.0042, 0.00.0, 4.202e−054.202e-05, 0.0001950.000195, 9.0536e−069.0536e-06, 0.0009050.000905, 0.01950.0195, 0.083810.08381 (agent), 0.083810.08381 (monitor), 0.09050.0905, base model, 0.42020.4202, 1.951.95 • Figure 6, right: 0.00420.0042, 4.202e−054.202e-05, 9.0536e−069.0536e-06, 0.0001950.000195, 0.00.0, 0.0009050.000905, 0.01950.0195, 0.083810.08381 (monitor), 0.083810.08381 (agent), 0.09050.0905, 0.42020.4202, base model, 1.951.95 Llama-3.2-1B with shift ρ=0.2ρ=0.2 details: • Figure 2(b), left: 9.352e−059.352e-05, 0.0020.002, 0.0004340.000434, 4.34e−064.34e-06, 0.00.0, 2.0148e−052.0148e-05, 0.009350.00935, 0.04340.0434, 0.06030.0603 (monitor), 0.06030.0603 (agent), base model, 0.2010.201, 0.9350.935 • Figure 2(b), right: 4.34e−064.34e-06, 0.0020.002, 9.352e−059.352e-05, 0.00.0, 0.0004340.000434, 2.0148e−052.0148e-05, 0.009350.00935, 0.04340.0434, 0.06030.0603 (agent), 0.06030.0603 (monitor), 0.2010.201, base model, 0.9350.935 • Figure 7, left: 9.352e−059.352e-05, 0.0020.002, 4.34e−064.34e-06, 0.0004340.000434, 0.00.0, 2.0148e−052.0148e-05, 0.009350.00935, 0.06030.0603 (monitor), 0.04340.0434, 0.06030.0603 (agent), base model, 0.2010.201, 0.9350.935 • Figure 7, right: 4.34e−064.34e-06, 9.352e−059.352e-05, 0.0020.002, 0.00.0, 0.0004340.000434, 2.0148e−052.0148e-05, 0.009350.00935, 0.06030.0603 (agent), 0.04340.0434, 0.06030.0603 (monitor), 0.2010.201, base model, 0.9350.935 C.2 Model Auditing As a baseline, we fine-tune surrogate policies on an exponential β-grid (the same one as in Appendix C.1) and, at each timestep, compute per-arm log-likelihood ratios against the reference on the same observed pairs. The mixture likelihood-ratio test aggregates evidence through a uniform mixture over grid points, stopping when the mixture statistic Λt=∑kwkexp(Lk,t) _t= _kw_k (L_k,t) exceeds 1/α1/α, where Lk,tL_k,t is the cumulative log-LR for arm k and wk=1/Kw_k=1/K. Under strategic sampling we compare the stopping times of both composites to those of the monitor and the bisection-optimal agent; under honest sampling we report the fraction of trials on which each composite crosses its threshold.