Paper deep dive
On KL-Regularized Policy Optimization
Yifan Zhang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 10/8/2026, 6:35:24 AM
Summary
The paper introduces KL-Regularized Policy Optimization (KLPO), a framework for asynchronous reinforcement learning in large language model agents. KLPO anchors the KL regularizer at the data sampler to handle stale checkpoints and execution mismatches without importance weights. It derives a critic-free update using one rollout per prompt by fitting log-ratio optimality conditions via least squares and profiling the normalizer. The framework unifies several existing methods (SPPO, GPO, REBEL, BPO) as special cases and provides unbiased gradient estimators using Monte Carlo approximations.
Entities (10)
Relation Signals (9)
KLPO → solvesproblem → Asynchronous RL
confidence 95% · Asynchronous reinforcement learning (RL) for large language model (LLM) agents... We propose KL-Regularized Policy Optimization (KLPO)
KLPO → avoids → Importance Weights
confidence 90% · no importance weights are needed.
KLPO → avoids → Critic
confidence 90% · The result is a critic-free update
KLPO → generalizes → BPO
confidence 90% · show that SPPO, GPO, REBEL, and BPO arise as special cases of KLPO.
KLPO → generalizes → SPPO
confidence 90% · show that SPPO, GPO, REBEL, and BPO arise as special cases of KLPO.
KLPO → generalizes → GPO
confidence 90% · show that SPPO, GPO, REBEL, and BPO arise as special cases of KLPO.
KLPO → generalizes → ReBeL
confidence 90% · show that SPPO, GPO, REBEL, and BPO arise as special cases of KLPO.
GRPO → requires → Group of Responses
confidence 85% · as in GRPO, sample a group of responses per prompt, which is costly
PPO → uses → Clipped Importance Ratios
confidence 85% · Standard remedies either clip importance ratios, which biases the update, or, as in GRPO...
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Asynchronous reinforcement learning (RL) for large language model (LLM) agents trains one policy on trajectories generated by another: rollouts come from stale checkpoints, and the inference engine's probabilities differ from the trainer's even at identical parameters. Standard remedies either clip importance ratios, which biases the update, or, as in GRPO, sample a group of responses per prompt, which is costly when episodes are long. We propose KL-Regularized Policy Optimization (KLPO), a framework that anchors the KL regularizer at the sampler. The regularized improvement step then has a closed-form Gibbs solution, and KLPO fits its log-ratio optimality condition by least squares on the sampler's own trajectories, so the sampler probability enters through a log-ratio and no importance weights are needed. Profiling out the regression intercept replaces the intractable log-partition function with the signal's sampler mean plus a sampler-to-trainer KL divergence. For token-level policy mirror descent targets, we show that the resulting gradient can be computed from terminal returns without a critic, via sampler-centered scores or a single trajectory residual, even under stochastic tool outputs. We further prove that independent Monte Carlo estimates of the KL term keep these gradients unbiased, derive the exact KL gap of cheaper top-$K$ and binary approximations, and show that SPPO, GPO, REBEL, and BPO arise as special cases of KLPO. The result is a critic-free update that uses one rollout per prompt and requires neither a learned normalizer nor a group of responses.
Tags
Links
- Source: https://arxiv.org/abs/2610.08963v1
- Canonical: https://arxiv.org/abs/2610.08963v1
Trouble viewing inline? Open PDF directly →
Full Text
137,722 characters extracted from source content.
Expand or collapse full text
On KL-Regularized Policy Optimization Yifan Zhang Princeton University yifzhang@princeton.edu September 18, 2026 ∗ Abstract Asynchronous reinforcement learning (RL) for large language model (LLM) agents trains one policy on trajectories generated by another: rollouts come from stale checkpoints, and the inference engine’s probabilities differ from the trainer’s even at identical parameters. Standard remedies either clip importance ratios, which biases the update, or, as in GRPO, sample a group of responses per prompt, which is costly when episodes are long. We propose KL-Regularized Policy Optimization (KLPO), a framework that anchors the KL regularizer at the sampler. The regularized improvement step then has a closed-form Gibbs solution, and KLPO fits its log-ratio optimality condition by least squares on the sampler’s own trajectories, so the sampler probability enters through a log-ratio and no importance weights are needed. Profiling out the regression intercept replaces the intractable log-partition function with the signal’s sampler mean plus a sampler-to-trainer KL divergence. For token-level policy mirror descent targets, we show that the resulting gradient can be computed from terminal returns without a critic, via sampler-centered scores or a single trajectory residual, even under stochastic tool outputs. We further prove that independent Monte Carlo estimates of the KL term keep these gradients unbiased, derive the exact KL gap of cheaper top-Kand binary approximations, and show that SPPO, GPO, REBEL, and BPO arise as special cases of KLPO. The result is a critic-free update that uses one rollout per prompt and requires neither a learned normalizer nor a group of responses. Project Page: https://github.com/yifanzhang-pro/KLPO 1 Introduction Reinforcement learning (RL) for language agents is asynchronous by necessity. Episodes last minutes, interleave tool calls, and finish at unpredictable times, so workers keep generating with the last checkpoint they received while the learner trains on trajectories as they arrive. Every update therefore involves two policies: the trainerπ trainer θ t that the learner improves at stept, and the samplerπ sampler θ t−k that generated the data, a checkpointksteps old. They differ for two reasons. Staleness makes the lagkpositive and different for every trajectory, with no fixed bound when tool latency varies. Execution makes their probabilities differ even atk = 0, because the inference engine applies its own sampling transforms, quantization, and numerical kernels. Agentic training is off-policy by construction, and the trainer–sampler gap is a permanent feature of the system. ∗ Revised: October 5, 2026 1 arXiv:2610.08963v1 [cs.LG] 6 Oct 2026 Most policy-optimization objectives are written for a single policy and treat the second as a correction. Token-level importance ratios grow more extreme with lag and accumulate along long trajectories, so they are clipped, as in PPO (Schulman et al., 2017), which biases the update. A learned critic adds a second model that must track a moving policy. Group-relative estimators such as GRPO (Shao et al., 2024) avoid the critic by sampling several responses per prompt, which is costly when a single episode is long. We build the objective around both policies by anchoring the KL regularizer at the sampler, the policy that collected the data. The regularized improvement step then has the closed-form Gibbs solution that underlies REPS, MPO, policy mirror descent, and RPG (Peters et al., 2010; Abdolmaleki et al., 2018; Lan, 2023; Zhang et al., 2026). Its logarithm is an equation between the trainer-to- sampler log-ratio and an improvement signal, which we fit by least squares on the sampler’s own trajectories. The log-ratio that an importance-weighted method must exponentiate and clip becomes the regression variable, so the gradient carries no importance weight, and the regression target is exactly the policy selected by the regularized objective. Because only the probabilities the sampler actually used enter, execution mismatch is covered along with staleness, and replay across checkpoints becomes a mixture of well-posed regressions, one per sampler version. We call the resulting family of objectives KL-Regularized Policy Optimization (KLPO); Figure 1 traces its token-level instance from objective to update. Our contributions are as follows. •Regression and profiling (Section 3). For any signal and granularity, regressing the log-ratio optimality condition identifies the sampler-anchored Gibbs target without importance weights; its gradient is the regularized-return gradient with the likelihood ratio removed. Profiling the intercept replaces the intractable log-partition function with the signal’s sampler mean plus a sampler-to-trainer KL divergence. •A common view of prior objectives (Section 3.4). With exact normalizers, SPPO and GPO are response-level KLPO, REBEL is its pairwise profiled form, and BPO’s exact trajectory loss equals the profiled token-level objective. Each practical loss differs from its KLPO instance by an identified approximation. •Critic-free token-level gradients (Section 4). The gradient of the profiled token-level policy mirror descent objective follows from terminal returns, either by centering the trainer’s score under the sampler or from one baseline-free trajectory residual, and both routes hold with stochastic tool outputs. •Estimators for asynchronous training (Sections 5 and 6). Independent Monte Carlo estimates of the local KL keep both gradients unbiased, and top-Kand binary approximations fall short of the KL by exactly the divergence inside the merged tail. Replay across versions is a fixed mixture of per-version regressions, so token-level KLPO trains from one rollout per prompt with no critic, learned normalizer, or response group. 2 Background We index optimizer steps bytand positions within a response byu. Each response is generated by a single checkpoint, whose versionv i is stored with it, so at steptit has lagk i (t) = t− v i ; reuse increases the lag but leaves the samplerθ v i unchanged. The learner thus holds the trainerπ trainer θ t and receives data from samplersπ sampler θ t−k with lags0 ≤ k ≤ K t , and synchronous training is the 2 π sampler := π sampler θ t−k , ℓ θ,u := log π trainer θ (y u | x,y <u ) π sampler (y u | x,y <u ) , A u := A sampler t,k (x,y <u ,y u ) k θ,u := KL π sampler (·| x,y <u )∥π θ (·| x,y <u ) . At s u = (x,y <u ): q = π sampler , p = π θ . Step 1. Local RPG–URKL Objective regularized local policy improvement J u (π) := E y u ∼π(·|x,y <u ) A u − β UKL π∥π sampler . Step 2. Gibbs / PMD Optimum closed-form optimal policy π + tok (y u | x,y <u ) = π sampler (y u | x,y <u ) · exp A u − c ∗ tok,u β . Step 3. Exact Normalizer c ∗ log-partition at the target c ∗ tok,u = β log E y u ∼π sampler e A u /β = β KL π sampler ∥π + tok . take logs Step 6. Profiled Token Objective replace the fixed intercept by its profiled value c θ,u := βk θ,u , δ u := A u − c θ,u β − ℓ θ,u , L tok := β 2 E π sampler X u δ 2 u . Step 5. Fixed-Normalizer Regression fit the exact optimality condition d fix θ,u := βℓ θ,u − A u + c ∗ tok,u L fix tok := 1 2β E π sampler X u d fix θ,u 2 . Step 4. Optimality Condition the policy solution becomes a log-ratio target β log π + tok (y u | x,y <u ) π sampler (y u | x,y <u ) = A u − c ∗ tok,u . terminal returns + score centering Step 7. KLPO Token Regression BPO-based critic-free PMD h θ,u := R− βℓ θ,u , ̄ h θ,u := E π sampler [h θ,u | x,y <u ], L TR := 1 2β E π sampler P u (h θ,u − ̄ h θ,u ) 2 , ∇L TR =∇L tok . Step 8. Exact Backprop Surrogate sampler-centered scores q v := π sampler (v | x,y <u ), p v := π θ (v | x,y <u ), z θ,u := logp y u − X v q v logp v , L bp,TR θ,u :=− sg(h θ,u )z θ,u , ∇ AD E π sampler P u L bp,TR θ,u =∇L tok . Step 9. Monte Carlo KL independent draws; unbiased score mean v u,j iid ∼ q, M ≥ 1, bz M θ,u := logp y u − 1 M X j logp v u,j , L bp,TR-MC θ,u :=− sg(h θ,u )bz M θ,u . Exact routes: ∇L SR =∇L tok =∇L TR . Both routes: complete matched-sampler rollouts with exact centering or full KL; deterministic transitions are not required. v u,j iid ∼ q(·| s u ), M ≥ 1, E MC ∇ AD θ X u L bp,TR-MC θ,u =∇ AD θ X u L bp,TR θ,u . Sequence regression (Steps 7 † –9 † ): Figure 4. TopK-KL and binary KL: Section 5. Figure 1 Token-level KLPO, from objective to critic-free update. Steps 1–4 solve the local KL-regularized improvement problem anchored at the sampler (Zhang et al., 2026) and rewrite its solution as a log-ratio condition. Steps 5–6 fit that condition by regression, as in SPPO and GPO (Wu et al., 2024; Zhang et al., 2025), and profile the unknown normalizer into a local KL. Steps 7–9 compute the gradient from terminal returns: Step 7 builds on BPO’s critic-free reformulation of PMD (Song et al., 2026), Step 8 centers the trainer’s score under the sampler (Marek and Ryabinin, 2026), and Step 9 replaces the exact score mean withM ≥ 1 independent sampler draws. Figure 4 gives the sequence-regression alternative to Steps 7–9. casek = 0with matched execution. Sampler probabilities are those actually used to generate the data, including temperature and other sampling transforms, and are logged at collection time. The sampler also serves as the reference distribution of the KL regularizer, so no separate reference model is needed, and the trainer is the only policy that is differentiated. Every quantity below is built from this pair (Figure 2). Sections 3–5 fix one collection version(t,k), writeπ θ forπ trainer θ , and evaluate gradients atθ t ; 3 Async workers Stale checkpoints different runtimes π sampler θ t−1 agent + tools π sampler θ t−2 agent + tools π sampler θ t−K t agent + tools Completed trajectories x i , y i , T i , R i , v i Sampler log-probs + KL records Retain sampler versionv i Learner Optimizer step t Current trainerπ trainer θ t Recompute log-probabilities KLPO token or sequence regression KL correction: MC-KL, TopK-KL, binary, or full batch One rollout per prompt No critic or auxiliary normalizer Publish new checkpoints; stored denominators stay unchanged Figure 2 Two policies in asynchronous agentic RL. Workers generate complete trajectories with stale checkpointsπ sampler θ t−k , on runtimes whose probabilities can differ from the trainer’s, and return them with their collection version, sampler log-probabilities, and the records the KL estimator needs. The learner updates π trainer θ t by fitting each trajectory against its own collection sampler. Section 6 combines versions. A promptx∼Dis followed by a complete responsey = (y 1 ,...,y T )∈ Y x , including its termination, withπ(y | x) = Q T u=1 π(y u | x,y <u ) for both policies. We write s u = (x,y <u )for the history before tokenuanda u = y u for the token. In agentic tasks,s u also contains every tool output observed beforey u , andTcounts only the tokens generated by the policy. Assumption 2.1 (Finite Positive Model). The vocabulary is finite and responses have bounded length. The rewardR(x,y)is a bounded deterministic function of the prompt and the complete response, such as a verifier score. The trainer and the sampler are normalized and positive on a common feasible support. Assumption 2.2 (Matched Collection). For a fixed collection version, prompts are drawn fromDand complete responses fromπ sampler θ t−k , without outcome-dependent filtering. Every sampler-dependent quantity refers to that same sampler: the denominator of each log-ratio, the distribution used for centering or KL estimation, and the continuation that produces each return. Assumption 2.3 (Deterministic Transitions and Terminal Reward). The next history is determined by the current history and action, and the reward is received only at termination. Assumptions 2.1 and 2.2 hold throughout and ensure that the sampler visits every feasible history and action. Assumption 2.3 is used only for samplewise identities that telescope token-level quantities along a trajectory; plain text generation satisfies it, and with tools it requires tool outputs to be deterministic functions of the history. A target policy is realizable if the model classπ θ contains it. Statements about exact fits assume realizability; gradient identities do not. 3 KL-Regularized Policy Optimization KLPO is built from one primitive, a KL-regularized improvement step taken from the policy that collected the data. We state it generically: a contexthis followed by an outcomezdrawn from a 4 positive sampler conditionalμ(z | h)on a finite support, such as a prompt and a complete response or a history and the next token. An improvement signalf (h,z), such as a reward, an advantage, or a win rate, scores outcomes, and β > 0 is fixed. Proofs for this section are in Appendix E.1. 3.1 Sampler-Anchored Improvement The improvement step maximizes the expected signal while penalizing divergence from the sampler, max π E z∼π f (h,z)− β KL π∥μ(·| h) ,(3.1) over distributionsπon the support. This is the unnormalized reverse-KL (URKL) objective of RPG (Zhang et al., 2026) with the sampler as reference, the policy-improvement step of REPS and MPO (Peters et al., 2010; Abdolmaleki et al., 2018), and one iteration of entropic policy mirror descent (Lan, 2023). Its solution is classical. Lemma 3.1 (Gibbs Optimality Condition). The unique maximizer of Equation (3.1) is π ⋆ f (z | h) = μ(z | h)e (f (h,z)−c f (h))/β , c f (h) = β log E z∼μ(·|h) e f (h,z)/β ,(3.2) and for every distribution π the objective equals c f (h)− β KL(π∥π ⋆ f (·| h)). Taking logarithms, the maximizer satisfies one equation per outcome: β log π ⋆ f (z | h) μ(z | h) = f (h,z)− c f (h).(3.3) The left side involves only policy probabilities, and the right side is the signal shifted by a context- dependent constant. KLPO fits this equation directly. 3.2 Regressing the Optimality Condition For a trainable policy π θ , the residual of Equation (3.3) is r θ (h,z) = β log π θ (z | h) μ(z | h) − f (h,z) + c f (h) = β log π θ (z | h) π ⋆ f (z | h) . Theorem 3.2 (Optimality-Condition Regression). Letνbe a replay law over pairs(h,z)anda > 0a weight, both held fixed during differentiation. Then L f = 1 2β E ν [ar 2 θ ], ∇ θ L f = E ν [ar θ ∇ θ logπ θ ].(3.4) Ifνcharges every pair in the support andπ ⋆ f is realizable, thenL f = 0exactly whenπ θ = π ⋆ f on the support. Any replay law that covers the support identifies the same target, with no importance multiplier in the gradient. The regularized-return gradient differs by a single factor: differentiating Equation(3.1) at π = π θ and rewriting the expectation under the sampler gives −∇ θ h E π θ f − β KL(π θ ∥μ) i = E z∼μ h π θ (z | h) μ(z | h) r θ (h,z)∇ θ logπ θ (z | h) i ,(3.5) which is the regression gradient withν = μanda = 1, multiplied per sample by the likelihood ratio. The two coincide atπ θ = μ; within a restricted model class their minimizers can differ, because the return minimizes KL(π θ ∥π ⋆ f ) and the regression minimizes E μ [log 2 (π θ /π ⋆ f )] (Appendix G). 5 3.3 Profiling the Normalizer The residual contains the log-partitionc f (h), an average ofe f/β over every outcome, which requires the signal for all outcomes or several outcomes at the same context. We instead treat the intercept as a free parameter and minimize over it. Withℓ θ (h,z) = log(π θ (z | h)/μ(z | h)), note that KL(μ(·| h)∥π θ (·| h)) =−E z∼μ ℓ θ (h,z). Proposition 3.3 (Profiled Regression). Fix a contexth. Overc∈ R, the lossE z∼μ [(βℓ θ − f + c) 2 ]is minimized by c θ (h) = E z∼μ f (h,z) + β KL μ(·| h)∥π θ (·| h) ,(3.6) so the profiled loss equalsVar μ (f − βℓ θ )/(2β). Its gradient is that of Theorem 3.2 withc θ in place ofc f , because the profiled residual has zero sampler mean and the derivative ofc θ cancels. Ifπ ⋆ f is realizable, the profiled loss vanishes exactly at π θ = π ⋆ f , where c θ = c f . In particular, c f (h) = E z∼μ f (h,z) + β KL μ(·| h)∥π ⋆ f (·| h) .(3.7) Equation(3.7)writes the log-partition at the target as a sampler mean plus a KL divergence, and profiling replaces the unknown target in that divergence by the current trainer. The fixed-normalizer and profiled losses share their realizable minimizer, though their gradients differ away from it. For a signal with zero sampler mean, such as an advantage, the profiled intercept is exactly β KL(μ∥π θ ). 3.4 Instances An instance of KLPO is a choice of context, outcome, and signal. We develop two (Table 4 in Appendix D compares them). Response level (SKLPO). With a promptxas context, a complete responseyas outcome, and f = R, the target is the response Gibbs policyπ ⋆ t,k (y | x)∝ π sampler θ t−k (y | x)e R(x,y)/β and the residual is d θ = βℓ θ −R+c ∗ (x), whereℓ θ = P u ℓ θ,u sums the token log-ratios andc ∗ (x) = β log E π sampler θ t−k [e R/β | x] . We call this regression SKLPO (sequence-level KLPO) and develop it in Appendix A. Its profiled intercept,E π sampler θ t−k [R | x] + β KL(π sampler θ t−k (· | x)∥π θ (· | x)) , needs the prompt’s mean reward and a complete-response KL, which a single response cannot supply. This is the weak point of SKLPO for asynchronous training: the normalizer depends on the prompt and the sampler version, and profiling within a group fails for a group of one (Section 6.2). Token level (local PMD). With a historysas context, the next tokenaas outcome, andf = A sampler t,k , the target is the policy mirror descent (PMD) stepπ + tok (a| s)∝ π sampler θ t−k (a| s)e A sampler t,k (s,a)/β , in which the sampler is both the KL reference and the policy whose advantages are improved, the starting point of BPO (Song et al., 2026). Advantages have zero sampler mean, so the profiled intercept is the local KLβ KL(π sampler θ t−k (·| s)∥π θ (·| s)) , computable from the two next-token distributions. The residual still contains the advantage, which would ordinarily require a critic; Section 4 obtains the gradient from terminal returns instead. The two targets differ: the token-level targets multiply to the response Gibbs policy only when the accumulated local normalizers are constant across responses (Appendix J.4), and regressing against optimal soft values instead of sampler advantages recovers the response-level target (Appendix C.2). 6 ObjectiveLevelSignalRelation to KLPO SPPOresponse win rateExact update: exact-normalizer regression; the practical loss sets the normalizer to 1/2 GPOresponse preference scoreExact update: exact-normalizer regression; the practical loss sets the normalizer to zero REBELresponse rewardPairwise form of profiled regression on two responses BPOtokensampler advantage Exact trajectory loss equals the profiled token objective under Assumption 2.3; the practical loss linearizes and approximates it Score centering tokenrewardSame centered score, with coefficient R in place of R− βℓ θ,u AWReitheradvantage or reward Same Gibbs target, fitted by weighted maximum likelihood RPGresponse rewardSame Gibbs target, fitted by maximizing the importance-weighted return Table 1 Prior objectives viewed through KLPO. “Exact” refers to a method’s population objective before its practical approximations. Prior objectives as instances. Several existing objectives are KLPO instances once their population objective is separated from their practical loss (Table 1). SPPO (Wu et al., 2024) and GPO (Zhang et al., 2025) are response-level KLPO with a win rate and an expected preference score as signals, REBEL (Gao et al., 2024) is the pairwise form of profiled response-level regression (Lemma B.3), and the exact trajectory loss of BPO (Song et al., 2026) equals the profiled token objective (Theorem 4.3). AWR and RPG (Peng et al., 2019; Zhang et al., 2026) share the Gibbs target but fit it differently. Appendices G and K state the matching conditions. 4 Token-Level KLPO without a Critic This section follows Figure 1. We write down the profiled token-level objective and then compute its gradient from terminal rewards along two routes: token regression, which centers the trainer’s score under the sampler, and sequence regression, which telescopes the token residuals into one trajectory residual. Throughout,x ∼ DandY ∼ π sampler θ t−k (· | x)are complete rollouts from one collection version, and ℓ θ,u = log π θ (a u | s u ) π sampler θ t−k (a u | s u ) , k θ (s) = KL(π sampler θ t−k (·| s)∥π θ (·| s)).(4.1) We writeg θ (s,a) = ∇ θ logπ θ (a | s)for the trainer’s score. The sampler’s action value, value, and advantage areQ sampler t,k (s,a) = E π sampler θ t−k [R | s,a] ,V sampler t,k (s) = E π sampler θ t−k [R | s] , andA sampler t,k = Q sampler t,k − V sampler t,k , with expectations over the sampler’s continuation. Proofs are in Appendix E. 7 4.1 The Profiled Token Objective Step 1 of Figure 1 applies the RPG–URKL objective (Zhang et al., 2026) to the next-token distribution at a history s: J s (π) = E a∼π(·|s) A sampler t,k (s,a)− β UKL π(·| s)∥π sampler θ t−k (·| s) .(4.2) Both policies are normalized, so the unnormalized KLUKL(p∥q) = P a [p a log(p a /q a ) +q a −p a ] equals the KL, and this is Equation(3.1)withf = A sampler t,k . Steps 2–5 solve it by Lemma 3.1 and fit the logarithmic condition with the fixed-normalizer residuald fix θ,u = βℓ θ,u − A sampler t,k (s u ,a u ) + c ∗ tok (s u ) , the squared log-ratio construction of SPPO and GPO (Wu et al., 2024; Zhang et al., 2025) moved to tokens. GPO’s practical loss drops the normalizer; Step 6 profiles it. Lemma 4.1 (Token PMD Target and Profiled Objective). The unique maximizer of Equation(4.2) and its normalizer are π + tok (a| s) = π sampler θ t−k (a| s)e (A sampler t,k (s,a)−c ∗ tok (s))/β ,(4.3) c ∗ tok (s) = β log E a∼π sampler θ t−k (·|s) e A sampler t,k (s,a)/β = β KL(π sampler θ t−k (·| s)∥π + tok (·| s)).(4.4) Profiling the intercept at each history gives c θ (s) = βk θ (s) and the profiled token objective δ u = A sampler t,k (s u ,a u ) β − ℓ θ,u − k θ (s u ),L tok = β 2 E x,Y∼π sampler θ t−k X u δ 2 u .(4.5) Each residual has zero conditional mean, E π sampler θ t−k [δ u | s u ] = 0, and ∇ θ L tok =−βE x,Y∼π sampler θ t−k X u δ u g θ (s u ,a u ).(4.6) Ifπ + tok is realizable, it is the unique zero-loss policy of bothL tok and the fixed-normalizer loss 1 2β E P u (d fix θ,u ) 2 . The profiled objective needs no normalizer, but it still needsA sampler t,k , the advantage of a policy the trainer has already left. The next two subsections remove it. 4.2 Token Regression with Sampler-Centered Scores Step 7 builds on BPO’s critic-free reformulation of PMD (Song et al., 2026) and replaces the sampler advantage with the terminal return. Define the trainer’s score centered under the sampler and a per-token return coefficient, e g θ (s,a) = g θ (s,a)− E v∼π sampler θ t−k (·|s) g θ (s,v) = g θ (s,a) +∇ θ k θ (s), h θ,u = R(x,Y )− βℓ θ,u , (4.7) let ̄ h θ (s) = E π sampler θ t−k [h θ | s] over the next action and the sampler’s continuation, and define the token-regression lossL TR (θ) = 1 2β E x,Y∼π sampler θ t−k P u (h θ,u − ̄ h θ (s u )) 2 . 8 Theorem 4.2 (Critic-Free Token-Regression Gradient). Under Assumptions 2.1 and 2.2, with exact conditional score means, L TR =L tok + 1 2β E x,Y∼π sampler θ t−k X u R− Q sampler t,k (s u ,a u ) 2 .(4.8) The second term does not depend on θ, so ∇ θ L tok =∇ θ L TR =−E x,Y∼π sampler θ t−k X u (R− βℓ θ,u ) e g θ (s u ,a u ).(4.9) Ifπ + tok is realizable, it is the unique minimizer of both losses. Subtracting a fixed baselineb(s u )from the coefficient R− βℓ θ,u leaves the expected gradient unchanged. Terminal returns thus replace the sampler’s critic: the continuation noiseR− Q sampler t,k adds variance but no bias, provided the score is centered under the sampler that generated both the action and its continuation. The term−βℓ θ,u makes this a regression rather than a policy gradient. It feeds the current log-ratio back into the coefficient, so the update weakens as the trainer approaches the target; with the reward alone as coefficient, the estimator is the score-centered policy gradient of Marek and Ryabinin (2026). Step 8 computes Equation(4.9)by automatic differentiation, without evaluating ̄ h θ , through the per-token surrogate L bp,TR θ,u =− sg(h θ,u )z θ,u , z θ,u = logπ θ (a u | s u )− X v π sampler θ t−k (v | s u ) logπ θ (v | s u ). (4.10) Withh θ,u and the sampler probabilities detached, each token contributes−h θ,u e g θ (s u ,a u ), so the expected surrogate gradient is∇ θ L TR = ∇ θ L tok (Proposition D.1 in Appendix D). One complete response per prompt suffices, provided the score mean is computed independently of the sampled action. 4.3 Sequence Regression by Bellman Telescoping Each token residual has zero conditional mean, so the token loss equals the expected square of the summed residual. Under Assumption 2.3, the advantages in that sum also telescope into the terminal reward, which gives the exact trajectory objective of BPO (Song et al., 2026, Equations (18)–(24)) (Figure 4 in Appendix D). Theorem 4.3 (Population Equivalence to Exact BPO). Under Assumptions 2.1 and 2.2, E π sampler θ t−k P u δ u 2 = E π sampler θ t−k P u δ 2 u .(4.11) If Assumption 2.3 also holds, then X u δ u = R− V sampler t,k (x) β − X u [ℓ θ,u +k θ (s u )] = δ BPO ,andL tok = β 2 E π sampler θ t−k [δ 2 BPO ] . (4.12) 9 The prompt valueV sampler t,k (x)inδ BPO seems to require a critic, but it does not affect the gradient. Drop it: D θ (x,Y ) := R(x,Y )− β X u [ℓ θ,u + k θ (s u )], L SR (θ) := 1 2β E x,Y∼π sampler θ t−k D θ (x,Y ) 2 . (4.13) Proposition 4.4 (Baseline-Free Trajectory Regression). Under Assumptions 2.1 and 2.2, letε u = V sampler t,k (s u+1 )− Q sampler t,k (s u ,a u )be the transition noise after tokenu, wheres u+1 includes any tool output that follows a u and V sampler t,k = R at termination. Then L SR =L tok + 1 2β E h V sampler t,k (x) 2 + X u ε u 2 i , ∇ θ L SR =∇ θ L tok .(4.14) Under Assumption 2.3, everyε u is zero. ReplacingRbyR−b(x)for a fixed prompt baseline replaces V sampler t,k (x) by V sampler t,k (x)− b(x) in the constant. Sequence regression therefore needs no value estimate, no group of responses, and no determin- istic transitions: random tool outputs add the noiseε u to the residual, but its contribution to the loss does not depend onθ, and Assumption 2.3 adds only the samplewise telescope P u δ u = δ BPO . The identity holds in population, so a baseline estimated from the same trajectory does not count as fixed, andL SR itself need not vanish at the PMD target. Step 8 † differentiates Equation (4.13). Writing g θ,u = g θ (s u ,a u ), ∇ θ L SR =−E x,Y∼π sampler θ t−k D θ X u [g θ,u +∇ θ k θ (s u )],(4.15) which each trajectory produces through the backward surrogate L bp,SR θ (x,Y ) :=− sg(D θ ) X u [logπ θ (a u | s u ) + k θ (s u )].(4.16) OnlyD θ is detached; gradients flow through the KL terms becauseg θ,u +∇ θ k θ (s u ) = e g θ (s u ,a u )is the centered score. BPO’s Equation (25) is this full-KL gradient before its later linearization (Song et al., 2026). Under Assumptions 2.1 and 2.2, the two routes therefore agree in population, ∇ θ L SR =∇ θ L tok =∇ θ L TR ,(4.17) but they are different estimators: token regression weights each centered score by its own coefficient R− βℓ θ,u , and sequence regression weights all of them by the shared residualD θ . Appendix D.4 contrasts sequence regression with SKLPO, which also squares one residual per response but has a different target. 5 Estimating the Local KL Correction At every visited history, both routes need the sampler mean of the trainer’s score, which equals −∇ θ k θ (s), and sequence regression also needsk θ (s)itself. Both are sums over the sampler’s next- token distribution, which is expensive to store for every generated token. We study three cheaper alternatives, each usable with either route: independent Monte Carlo draws, which preserve the gradients in expectation, and top-Kand binary aggregations, which change the objective by an amount we compute exactly. At a history s u we write q(v) = π sampler θ t−k (v | s u ) and p(v) = π θ (v | s u ). 10 5.1 Monte Carlo KL Monte Carlo KL (MC-KL) drawsMauxiliary tokens from the sampler at each visited prefix, indepen- dently of the rollout and of one another: b k M θ,u := 1 M M X j=1 log q(v u,j ) p(v u,j ) , v u,j iid ∼ q.(5.1) For a fixed trainer, b k M θ,u and its gradient are unbiased fork θ (s u )and∇ θ k θ (s u ); a single estimate can be negative, and clipping it at zero destroys the unbiasedness. The draws are single tokens rather than continuations, so they need no further generation or reward. Token regression (Step 9). Given the rollout, the coefficienth θ,u is independent of the auxiliary draws, so replacing the vocabulary sum in Equation(4.10)by the sample mean leaves the expected gradient unchanged for every M ≥ 1: L bp,TR-MC θ =− X u sg(h θ,u ) h logp(a u | s u )− 1 M X j logp(v u,j | s u ) i .(5.2) Independence is essential: if the rollout action served as its own auxiliary draw, the bracket would vanish identically. Sequence regression (Step9 † ). Here an unbiased KL estimate is not enough: the residual is squared, so reusing the same draws inside the residual and its derivative correlates the two factors and biases the gradient (Appendix E.6). Instead, we form one residualD j per draw indexjand weight the score from draw j by the mean residual of the other draws: D j = R− β X u h ℓ θ,u + log q(v u,j | s u ) p(v u,j | s u ) i ,D −j = 1 M − 1 X l̸=j D l , L bp,SR-MC θ =− 1 M X j sg( D −j ) X u [logp(a u | s u )− logp(v u,j | s u )]. (5.3) This requires M ≥ 2 draws per prefix, not M responses. Proposition 5.1 (Independent MC-KL Gradients). Fix the trainer and a complete rollout. Averaged over independent auxiliary draws, the gradients of Equations(5.2)and(5.3)equal the corresponding full-KL sample gradients. The sequence surrogate is also the gradient of the U-statistic b L SR,MC = 1 2βM (M − 1) X j̸=l D j D l ,E MC b L SR,MC = D 2 θ 2β .(5.4) Hence the population identities of Section 4 hold after averaging over the auxiliary draws and the rollouts. Since these statements condition on a trainer chosen independently of the draws, a conditionally unbiased gradient at every update uses fresh draws (Appendix D.6). 11 KL correctionRecords per generated tokenToken routeSequence routeExact gradient Full KLthe sampler’s next-token distribution Eq. (4.10)Eq. (4.16)yes MC-KLM sampled token IDs and their sampler log-probabilities Eq. (5.2), M ≥ 1 Eq. (5.3), M ≥ 2 in expectation TopK-KLtop-K token IDs and their sampler probabilities Eq. (D.6)Eq. (D.5)if the tail is one token Binary KL nothing beyond the sampled token Eq. (D.11)Eq. (D.10)no Table 2 Estimators of the local KL correction. Every row also stores the sampled token’s own sampler log-probability. “Exact gradient” refers to the population identity of Equation (4.17). 5.2 TopK-KL and Binary KL Top-Kaggregated KL (TopK-KL) keeps the sampler’sKmost probable tokens at each history, with their unrenormalized probabilities, and merges the rest into one tail category (Kis unrelated toM and to the maximum lag K t ). With head H u , q tail = 1− P v∈H u q v , and p tail = 1− P v∈H u p v , k K θ,u := X v∈H u q v log q v p v + q tail log q tail p tail (5.5) is the KL between the coarsened distributions, and by the chain rule for KL it falls short of the full KL by exactly the divergence inside the tail: k θ (s u )− k K θ,u = q tail KL q(·| tail)∥p(·| tail) ≥ 0.(5.6) TopK-KL is therefore exact when the tail holds a single token, and its gradient models the sampler’s tail as proportional to the trainer’s (Equation(D.3)). Binary KL is the coarsest case: BPO’s partition (Song et al., 2026, Equation (27)) takes the sampled token as the head, needs nothing beyond the rollout’s own records, and corrects the score by the complementary-probability factor(1− q(a u ))/(1− p(a u )) rather than an importance ratio. Appendix D.5 gives the surrogates for both routes (Propositions D.2 and D.3). Both aggregations give up the zero means on which every exact result of Section 4 rests. Because of the tail gap, the corrected log-ratio has sampler meank K θ −k θ ≤ 0instead of zero, so the update is no longer the gradient ofL tok , a reward offset can change it, the PMD target need not be stationary, and the two routes can disagree in population (Appendix E.7). Table 2 summarizes the trade-off: MC-KL keeps the exact identities in expectation at the cost of variance andMrecords per position, TopK-KL is deterministic and nearly exact when the sampler’s mass is concentrated, and binary KL is the cheapest and the crudest. 6 Learning from Lagged Samplers We return to the asynchronous setting, where a batchB t =(x i ,y i ,T i ,R i ,v i ) B i=1 mixes trajectories from several sampler versions, with responseigenerated byπ sampler θ v i and stored with its version, its 12 sampler log-probabilities, and the records its KL estimator needs. The learner trains on completed trajectories without waiting for a second response to the same prompt (Algorithm 2 in Appendix N). 6.1 Mixtures of Sampler Versions Letℓ i,u,θ ,k i,θ , and e g i,u,θ be the log-ratio, local KL, and centered score of tokeny i,u , all computed against its own collection sampler π sampler θ v i . Proposition 6.1 (Version-Conditioned Updates). Withh i,u,θ = R i − βℓ i,u,θ andD i,θ = R i − β P u [ℓ i,u,θ + k i,θ (s i,u )], the token-regression and sequence-regression batch gradients are ∇ θ b L bp,TR tok,t =− 1 B X i T i X u=1 h i,u,θ e g i,u,θ , ∇ AD θ b L bp,SR t =− 1 B X i D i,θ T i X u=1 e g i,u,θ .(6.1) If each version contributes complete, unfiltered rollouts from its recorded sampler, both expected gradients equal the gradient of the mixture of per-version objectivesL tok , weighted by the version frequencies. The proof is in Appendix E.9, and every KL estimator of Section 5 substitutes into each component. The proposition is Assumption 2.2 applied version by version: sampler quantities must never be mixed across versions, while trainer-dependent ones (h i,u,θ orD i,θ , and the KL correction) are recomputed at every reuse. Dividing each response’s loss by its length breaks the cancellation of cross terms, so the batch loss sums over tokens and averages over responses. Each version defines its own PMD target, so a trainer fitted to several versions settles on a compromise that depends on the distribution of lags, even with unlimited capacity (Proposition A.6 proves the response-level case). 6.2 Consequences for Asynchronous Training No ratio to clip. The regression gradient is the regularized-return gradient without the trainer- to-sampler ratio (Section 3.2 and Proposition I.1), so KLPO has nothing to clip as checkpoints age or trajectories grow. The sampler probability still enters through the logarithm, so errors in logged probabilities or stored conditionals shift the log-ratio or the KL correction. Appendix I.2 compares KLPO with the importance-weighted update of FlashREINFORCE (Hu et al., 2026). One rollout per prompt. With full KL or MC-KL, one complete rollout gives an unbiased stochastic gradient ofL tok (Theorem 4.2 and Proposition 5.1), with no value function, prompt normalizer, or second response. SKLPO instead needs a normalizer for every prompt and sampler version, and with one response both natural estimators degenerate (Proposition B.4): self-centering zeroes the residual, and a log-mean-exp estimate ofc ∗ (x)returns the observed reward. Its normalizer must come from independent data or a learned predictor (Appendix B), and an errorε(x)biases the gradient by ε(x)E ν [g θ | x] (Equation (I.9)). Table 5 lists what each instance needs. Interactive trajectories. Neither gradient route needs Assumption 2.3. Token regression uses only the sampler’s own continuation values, which a complete return from that sampler estimates without bias, and stochastic tool outputs add only aθ-independent term to sequence regression (Theorem 4.2 and Proposition 4.4). In both routes the return estimates the ordinaryQ sampler t,k of its own version, not an optimal soft Q-function. The response-level target, by contrast, can reweight tool outcomes that the policy cannot control, so no policy may attain it (Appendix I.5). 13 7 Related Work KL-regularized policy improvement. Lemma 3.1 is the classical solution of a relative-entropy- regularized improvement step. REPS, MPO, and AWR (Peters et al., 2010; Abdolmaleki et al., 2018; Peng et al., 2019) fit a parametric policy to it by weighted maximum likelihood, a forward-KL projection; policy mirror descent (Lan, 2023) analyzes repeated steps; and RPG (Zhang et al., 2026) maximizes the regularized return directly, a reverse-KL projection weighted by the trainer-to-sampler ratio. KLPO fits the same target through its log-ratio condition. The three projections share the unrestricted optimum and differ within a restricted model class (Appendices G and K). Log-ratio regression and preference optimization. DPO (Rafailov et al., 2023) readsβtimes a policy log-ratio as an implicit reward. SPPO (Wu et al., 2024) and GPO (Zhang et al., 2025) regress log-ratios onto win rates and preference scores, and REBEL (Gao et al., 2024) regresses log-ratio differences onto reward differences. Section 3.4 places all three within KLPO through the signalf rather than a scalar reward, since general preferences can be cyclic. Token-level PMD and score centering. BPO (Song et al., 2026) starts from token-level PMD with sampler advantages. Its Equation (25) derives the full-KL gradient of the exact trajectory loss before linearizing the residual, and its practical loss adds group normalization, a binary KL, smoothing, and clipping; our sequence regression keeps the residual, drops the prompt value, and allows random tool outputs. Score centering (Marek and Ryabinin, 2026) subtracts the sampler mean of the trainer’s score and multiplies by the reward. KLPO token regression applies the same correction (Marek and Ryabinin, 2026, Equation (4)) with coefficientR−βℓ θ,u , and its top-Kvariant (Marek and Ryabinin, 2026, Section 4.1) is the derivative of TopK-KL. Off-policy and critic-free RL for language models. Clipped importance ratios (Schulman et al., 2017) and group-relative advantages (Shao et al., 2024) are the standard ways to handle stale data and to avoid a critic. FlashREINFORCE (Hu et al., 2026) studies critic-free, single-rollout asynchronous training with token importance sampling and batch reward centering. KLPO changes the objective so that the sampler probability enters a log-ratio rather than a multiplicative weight. Soft values and implicit Q-functions. Regularized MDPs (Geist et al., 2019) and soft Q-learning (Haarnoja et al., 2017) characterize KL-regularized optimal policies through Bellman equations, and IQL (Kostrikov et al., 2022), IQ-Learn (Garg et al., 2021), and Rafailov et al. (2024) extract policies from value functions or read policies as value functions. At an exact fit, the response-level KLPO log-ratio is the optimal sampler-regularized soft advantage (Appendix C); token-level KLPO fits ordinary sampler advantages and learns no value function. 8 Conclusion We introduced KLPO, which anchors the KL regularizer at the sampler and fits the resulting Gibbs target by regressing its log-ratio optimality condition on the sampler’s own trajectories, without importance weights. Profiling the intercept turns the unknown normalizer into a sampler mean plus a sampler-to-trainer KL divergence, and at the token level only the local KL remains. The profiled token-level gradient then follows from terminal returns, through sampler-centered scores or a single trajectory residual; both routes hold under random tool outputs, and independent Monte Carlo estimates of the KL keep them unbiased. The same principle recovers SPPO, GPO, REBEL, and BPO 14 as special cases. Token-level KLPO trains from a single rollout per prompt, with no critic, learned normalizer, or group of responses, which makes it a natural fit for asynchronous agentic training. Next steps include large-scale empirical evaluation, finite-sample and convergence analysis, and value learning for stochastic environments (Appendix I.7). Acknowledgement We used large language models to assist with the proofs and to improve the clarity and readability of this manuscript. References Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum a posteriori policy optimisation. In International Conference on Learning Representations, 2018. URL https://arxiv.org/abs/1806.06920. Zhaolin Gao, Jonathan D. Chang, Wenhao Zhan, Owen Oertell, Gokul Swamy, Kianté Brantley, Thorsten Joachims, J. Andrew Bagnell, Jason D. Lee, and Wen Sun. REBEL: Reinforcement learning via regressing relative rewards. arXiv preprint arXiv:2404.16767, 2024. URLhttps: //arxiv.org/abs/2404.16767. Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, Matthieu Geist, and Stefano Ermon. IQ-Learn: Inverse soft-Q learning for imitation. In Advances in Neural Information Processing Systems, volume 34, 2021. URL https://arxiv.org/abs/2106.12142. Matthieu Geist, Bruno Scherrer, and Olivier Pietquin. A theory of regularized Markov decision processes. In Proceedings of the 36th International Conference on Machine Learning, volume 97, pages 2160–2169. PMLR, 2019. URL https://proceedings.mlr.press/v97/geist19a.html. Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In Proceedings of the 34th International Conference on Machine Learn- ing, volume 70, pages 1352–1361. PMLR, 2017. URLhttps://proceedings.mlr.press/v70/ haarnoja17a.html. Jian Hu, Yifan Zhang, Hao Zhang, Binfeng Xu, Shaokun Zhang, Hongqing Peng, Zhiding Yu, Pavlo Molchanov, Jan Kautz, and Yi Dong. FlashREINFORCE: Critic-free single-rollout asynchronous RL for agentic language models. Manuscript, 2026. URLhttps://github.com/yifanzhang-pro/ FlashREINFORCE-Overleaf/tree/774e4ccd79435e8251a5b1aff8366575467de4ea . Repository revision 774e4ccd, accessed September 19, 2026. Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit Q-learning. In International Conference on Learning Representations, 2022. URLhttps://arxiv.org/abs/ 2110.06169. Guanghui Lan. Policy mirror descent for reinforcement learning: Linear convergence, new sampling complexity, and generalized problem classes. Mathematical Programming, 198:1059–1106, 2023. 15 Martin Marek and Max Ryabinin. Score centering stabilizes off-policy reinforcement learning. arXiv preprint arXiv:2609.20807, 2026. URL https://arxiv.org/abs/2609.20807v1. Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177, 2019. URL https://arxiv.org/abs/1910.00177. Jan Peters, Katharina Mülling, and Yasemin Altün. Relative entropy policy search. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 24, pages 1607–1612, 2010. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems, volume 36, 2023. URLhttps://arxiv.org/abs/2305. 18290. Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. FromrtoQ ∗ : Your language model is secretly a Q-function. In Conference on Language Modeling, 2024. URLhttps://arxiv.org/abs/ 2404.12358. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. URLhttps://arxiv.org/abs/ 1707.06347. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. URLhttps://arxiv.org/abs/ 2402.03300. Zhuoqing Song, Haotian Xu, Xikun Zhang, and Lidong Bing. Bellman policy optimization. arXiv preprint arXiv:2609.15987, 2026. URL https://arxiv.org/abs/2609.15987v1. Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play preference optimization for language model alignment. arXiv preprint arXiv:2405.00675, 2024. URL https://arxiv.org/abs/2405.00675. Yifan Zhang, Ge Zhang, Yue Wu, Kangping Xu, and Quanquan Gu. Beyond Bradley–Terry models: A general preference model for language model alignment. In Proceedings of the 42nd International Conference on Machine Learning, 2025. URL https://arxiv.org/abs/2410.02197. Yifan Zhang, Yifeng Liu, Huizhuo Yuan, Yang Yuan, Quanquan Gu, and Andrew Chi-Chih Yao. On the design of KL-regularized policy gradient algorithms for LLM reasoning. In International Conference on Learning Representations, 2026. URL https://arxiv.org/abs/2505.17508. 16 Appendix A Response-Level KLPO (SKLPO)18 B Normalizing the Response-Level Target21 C Implicit Q ⋆ Learning25 D Additional Details for the Main Text28 E Proofs for the Framework and Token-Level Results32 F Proofs for the Response-Level and Value Results37 G RPG–URKL and Gradient Identities39 H Normalizer Identities and Estimation40 I Additional Off-Policy Results44 J Token-Level Identities47 K Detailed Comparisons with Prior Objectives49 L Additional Identities50 M Sequence Length Weighting and Gradient Scale51 N Implementation Details51 17 A Response-Level KLPO (SKLPO) This appendix develops the response-level instance of KLPO, which we call SKLPO (Sequence KL- Regularized Policy Optimization). SKLPO regresses the optimality condition of the response Gibbs target with one normalizer per prompt and sampler version, and the response residual multiplies every token score. Given normalization information it can train from single responses, but unlike token-level KLPO it cannot do without that information (Section 6.2). Appendix B covers normalization and Appendix C the soft-value interpretation. We fix a collection version and abbreviatec ∗ = c ∗ t,k and Z = Z t,k . A.1 The Response Gibbs Target For β > 0, the response-level objective at step t is J (π θ ) = E y∼π θ [R(y)]− β UKL(π θ ∥π sampler θ t−k ), UKL(π θ ∥π sampler θ t−k ) = X y π θ (y) log π θ (y) π sampler θ t−k (y) + π sampler θ t−k (y)− π θ (y) . (A.1) For normalized policies the mass correction vanishes and the unnormalized KL equals the ordinary KL; the URKL formulation keeps the normalization constraint on the policy. Theorem A.1 (Response Gibbs Improvement). Under Assumption 2.1, define Z(x) = E y∼π sampler θ t−k (·|x) e R(x,y)/β , c ∗ (x) = β logZ(x). The unique maximizer of Equation (A.1) is π ⋆ t,k (y | x) = π sampler θ t−k (y | x)e R(x,y)/β Z(x) ,(A.2) and for every feasible π θ , J (π θ ) = c ∗ − β KL(π θ ∥π ⋆ t,k ).(A.3) Equivalently, π ⋆ t,k is the entropic mirror-ascent update π ⋆ t,k = argmax π∈∆(Y x ) n ⟨π,R⟩− η −1 KL(π∥π sampler θ t−k ) o , η = β −1 .(A.4) Subtracting a prompt-dependent baselineb(x)fromRchangesc ∗ toc ∗ − band leaves the target unchanged. The proof is in Appendix F.1. The target is defined relative to the collection sampler, and a finite trainer step need not reach it. 18 A.2 Regressing the Response Condition Taking logarithms of the Gibbs policy gives the condition that SKLPO fits. The response log-ratio is a sum of token log-ratios, ℓ θ,u (x,y) = logπ θ (y u | x,y <u )− logπ sampler θ t−k (y u | x,y <u ), ℓ θ (x,y) = log π θ (y | x) π sampler θ t−k (y | x) = T X u=1 ℓ θ,u (x,y), (A.5) and the response residual and unweighted loss are d θ (x,y) = βℓ θ (x,y)− R(x,y) + c ∗ (x),(A.6) L fix seq (θ) = 1 2β E x∼D,Y∼π sampler θ t−k (·|x) [d θ (x,Y ) 2 ] = 1 2β E x,Y β |Y| X u=1 ℓ θ,u (x,Y )− R(x,Y ) + c ∗ (x) 2 . (A.7) The token log-ratios are summed before squaring, and the reward and normalizer each enter once, so one residual multiplies every token score. We writeκ θ (x) = KL(π sampler θ t−k (·| x)∥π θ (·| x))for the complete-response KL. Sequence regression in Section 4.3 also shares one residual across a response, but its correction is the path sumβ P u k θ (s u )rather than this prompt normalizer, and it targets local PMD rather than the response Gibbs policy (Section D.4). Theorem A.2 (SKLPO Regression Target). Under Assumption 2.1,d θ = β log(π θ /π ⋆ t,k ). Ifπ ⋆ t,k is realizable, Equation(A.7)vanishes exactly atπ θ = π ⋆ t,k . In a restricted model class, the regression minimizes a squared log-ratio underπ sampler θ t−k while the regularized return minimizesKL(π θ ∥π ⋆ t,k ), and the two minimizers can differ. The proof is in Appendix F.1. Proposition A.3 (Regression and Return Gradients). With responses drawn fromπ sampler θ t−k , and the sampler and normalizer held fixed during differentiation, ∇ θ L fix seq = E π sampler θ t−k [d θ ∇ θ logπ θ (y)],(A.8) −∇ θ J (π θ ) = E π sampler θ t−k π θ (y) π sampler θ t−k (y) d θ ∇ θ logπ θ (y) .(A.9) The two population gradients agree at π θ = π sampler θ t−k and generally differ elsewhere. The proof is in Appendix G. This is Equation(3.5)at the response level: the return gradient carries the multiplierw θ = π θ /π sampler θ t−k . Appendix I.1 makes the same comparison at a single history. 19 A.3 Outer Length Weighting LetT (x,y) ≥ 1count generated policy tokens, including termination under a fixed stopping con- vention. A length weight changes how much each response contributes, while the summed log-ratio stays inside the residual. Proposition A.4 (Positive Outer Weights). Under the assumptions of Theorem A.2, any fixed positive response weighta(x,y)givesL a = (2β) −1 E π sampler θ t−k [ad 2 θ ] the same unique zero-loss policyπ ⋆ t,k and normalizer c ∗ . For a = 1/T , L len seq (θ) = 1 2β E x,Y∼π sampler θ t−k " d θ (x,Y ) 2 T (x,Y ) # , ∇ θ L len seq = E x,Y∼π sampler θ t−k " d θ 1 T T X u=1 ∇ θ logπ θ (y u | x,y <u ) # , (A.10) with the sampler law, recorded lengths, and normalizers held fixed during differentiation. The proof is in Appendix F.1. Each response residual now multiplies the mean token score of that response. Positive weights leave the target alone but change how a restricted model trades off errors across responses: at equal absolute residual, a response of length100contributes ten times as much loss as one of length1000. The batch reduction isB −1 P i d 2 i /T i ; dividing P i d 2 i by the total token count is a different reduction. The accumulated log-ratio can still grow with length, and Appendix M gives the gradient bound and explains why the weighted update differs from the return gradient even at matched policies. A.4 Replay across Sampler Versions In replay, each response also carries its fixed-target normalizer, or enough information to fit one. Proposition A.5 (Version-Conditioned Sequence Update). Holding these quantities fixed, the empir- ical length-weighted loss and its gradient are d i,θ = β log π trainer θ (y i | x i ) π sampler θ t−k i (t) (y i | x i ) − R i + c ∗ t,k i (t) (x i ), b L len seq,t (θ) = 1 2βB B X i=1 d 2 i,θ T i , ∇ θ b L len seq,t (θ) = 1 B B X i=1 d i,θ T i ∇ θ logπ trainer θ (y i | x i ). (A.11) The proof is in Appendix E.9. Atθ = θ t , collection frequencies and response lengths determine each sampler’s contribution, with no extra version weight or importance multiplier. The gradient is generally not the on-policy return gradient. Proposition A.6 (Compatibility across Sampler Versions). For a fixed version, full response support, exact normalization, and realizability give the unique zero-loss targetπ ⋆ t,k , with or without positive outer weights. If several versions occur with positive frequency and have full common support, their 20 joint population loss is zero exactly when all their Gibbs targets coincide and the trainer equals that common target. For a common reward andβ, two full-support samplers have the same Gibbs target only if they are equal. The proof is in Appendix E.9. Distinct samplers therefore pull the trainer toward competing targets, and their relative frequencies affect the best fit even with unrestricted capacity. A finite batch does not by itself provide the population coverage assumed here. Each response should use its ownc ∗ t,k i (t) (x i ), or a weighted intercept fitted within its prompt–version group; pooling normalizers across versions changes the residual (Appendix B.4). A.5 The SKLPO Update Algorithm 1 applies the length-weighted loss to completed responses. For responsei,ℓ i,u is the token log-ratio of Equation(A.5), evaluated with the current trainer and the stored collection versionv i . The normalizer can be supplied directly, learned as an intercept, or estimated by weighted centering within each prompt and sampler version. Fixed normalizers stay unchanged during reuse, while fitted intercepts track the trainer. Since a one-response group has zero centered loss, single-rollout training needs independent normalization information or a predictor shared across prompts; Appendix B gives the estimators and their assumptions. Algorithm 1 SKLPO with Each Response’s Historical Sampler Require: Trainer π trainer θ 0 , completed-response queue, β > 0, normalization method 1: for optimizer steps t = 0, 1,... do 2:Read B responses (x i ,y i ,T i ,R i ,v i ) and their stored sampler log-probabilities. 3:Score with π trainer θ t ; compute ℓ i = P T i u=1 ℓ i,u , a i = 1/T i , h i = R i − βℓ i . 4: if using a supplied prompt-and-version normalizer then 5:Set d i = bc v i (x i )− h i ; keep bc v i detached and fixed during reuse. 6: else if using a learned intercept c φ then 7:Set d i = sg(c φ (x i ,v i ))− h i before updating either model. 8: else 9:Group by (x i ,v i ) with G x,v ≥ 2. 10:For each group, set ̄ h a = ( P j a j h j )/( P j a j ) and d i = ̄ h a − h i . 11: end if 12:Update θ t using the gradient of 1 2βB P B i=1 a i d 2 i . 13: if using a learned intercept then 14:Update φ by Equation (H.14), using the pre-update h i and a i . 15: end if 16:Publish checkpoints on the synchronization schedule; retain historical sampler log-probabilities. 17: end for B Normalizing the Response-Level Target SKLPO needs one normalizerc ∗ t,k (x)per prompt and collection version. The fixed normalizer depends on neither the sampled response nor its length, whereas a fitted intercept changes with the trainer. This appendix compares the ways to supply it, and closes with the fixed and terminal-reward forms of the token-level objective for comparison. All statements condition on one sampler version. 21 B.1 Normalization Choices SKLPO does not require a learned value model. Normalization information can come from repeated responses or from independent statistics, and learning a prompt normalizer is one implementation choice among several. A learned normalizer is an auxiliary scalar predictor, not a token-value critic. At the exact target,c ∗ (x)is the prompt’s soft value; a fitted interceptc θ (x)generally takes a different value during training. MethodInformation usedPolicy-update schedule Direct estimateSame-prompt, same-version rewards or in- dependent success statistics Estimatebc; hold it fixed while reusing the data. Learned normalizer A shared predictor of success probability or exponentiated reward Predictbcout of sample; hold that predic- tion fixed during reuse. Group centeringAt least two responses for the same prompt and version Recompute the weighted group mean after each trainer update. Learned interceptA shared predictor of the conditional mean of R− βℓ θ Alternate policy and intercept updates, us- ing the same outer weights. Table 3 Ways to normalize the response-level target. The two learned routes add a prompt-conditioned predictor; the other two do not. All four condition on the collection sampler. Single-rollout collection means one response per prompt per collection event. Such data can still be normalized with independent historical data or with a predictor shared across prompts, but historical rewards estimate the right normalizer only if they came from the same collection policy. With one response and no other information, the plug-in and empirical-centering constructions below cannot produce the intended update. Appendix H.7 specifies the learned routes and their update order. B.2 Fixed Normalizers Let V sampler t,k (x) = E π sampler θ t−k R(x,Y ) and A sampler t,k (x,y) = R(x,y)− V sampler t,k (x). Lemma B.1 (Response Normalizer and Plug-In Estimation). The exact prompt normalizer satisfies c ∗ (x) = V sampler t,k (x) + β KL(π sampler θ t−k (·| x)∥π ⋆ t,k (·| x)).(B.1) For G independent responses to the same prompt from π sampler θ t−k , define b c(x) = β logsumexp G i=1 R(x,y i ) β − logG .(B.2) Then e bc/β is an unbiased estimator of Z, while E b c≤ c ∗ . The proof is in Appendix H.1. Equation(B.1)is Equation(3.7)withf = R; a mean-reward baseline omits its KL term. Smallerβmakes the exponential average more sensitive to rare high rewards. Holding b cfixed defines a sampled regression target, not an unbiased estimate of the exact-normalizer loss. Appendix H gives the binary-reward form and the effect of data selection. 22 B.3 Profiling within a Prompt For unweighted, unfiltered responses from one sampler, writeh θ = R− βℓ θ , and letκ θ (x)be the complete-response KL from Appendix A. The following is Proposition 3.3 with f = R. Proposition B.2 (Profiled SKLPO Regression). At fixed θ, the least-squares prompt intercept is c θ (x) = E π sampler θ t−k [h θ (x,Y )] = E π sampler θ t−k R + β KL(π sampler θ t−k ∥π θ ).(B.3) The centered residual and loss are δ KL (x,y;θ) = A sampler t,k (x,y) β − ℓ θ (x,y)− κ θ (x),(B.4) L ctr,π sampler θ t−k (θ) = β 2 E x E Y∼π sampler θ t−k (·|x) [δ KL (x,Y ;θ) 2 ] = 1 2β E x Var Y∼π sampler θ t−k (·|x) [R(x,Y )− βℓ θ (x,Y )]. (B.5) Under full support and realizability, the unique zero-loss policy isπ ⋆ t,k , and its fitted intercept equals c ∗ . The proof is in Appendix H.3. Away from the target,c θ generally differs fromc ∗ . For comparison with the token objectives, the same residual and loss can be written as δ seq = R− V sampler t,k (x) β − X u ℓ θ,u − κ θ (x), L seq = β 2 E x,Y∼π sampler θ t−k [δ 2 seq ],(B.6) so thatδ seq = δ KL andL seq =L ctr,π sampler θ t−k . The fixed-normalizer version replacesκ θ by the KL to the Gibbs target, KL(π sampler θ t−k ∥π ⋆ t,k ). In practice, profiling means centering within a group of responses to the same prompt. Lemma B.3 (Empirical Centering). ForGresponses in one prompt and sampler-version group, set h i = R i − βℓ i and ̄ h = G −1 P i h i . Then b L ctr = 1 2βG X i (h i − ̄ h) 2 = 1 4βG 2 X i,j (h i − h j ) 2 .(B.7) Differentiating through ̄ h and detaching it give the same gradient. For fixedθand i.i.d. responses, the expected empirical loss is (G− 1)/G times the conditional population loss. The proof is in Appendix H.4. The centered form costsO(G)after scoring, and the pairwise form is the relative-reward regression of REBEL (Gao et al., 2024). The mean ̄ hmust be recomputed at every optimizer step. Pooling sampler versions changes the component residuals, and outcome-dependent selection changes the centering measure, which can invalidate the KL expression in Equation(B.3); Appendix I.3 gives the corresponding gradient. The empirical KL estimate− ̄ ℓcan be negative, and clipping it changes the loss. 23 Proposition B.4 (Singleton Normalization). ForG = 1, Equation(B.2)gives b c = R 1 , and hence a zero fixed-normalizer residual atπ θ = π sampler θ t−k . Empirical centering of a one-response group gives zero loss at every policy, with or without a positive outer length weight. The proof is in Appendix H.4. Fixed-normalizer training on single responses therefore needs normalization information obtained independently of the response being scored. A shared learned intercept is different: it estimates a conditional mean across training examples rather than computing a singleton group mean. A reward mean across prompts cannot replace the prompt normalizer. Appendix I.4 gives the signed-feedback and normalizer-error identities. B.4 Centering with Length Weights For length-weighted training, the intercept should be profiled with the same outer weight that the loss uses. Proposition B.5 (Weighted Sequence Centering). At a fixed prompt and sampler version, seta = 1/T and h θ = R− βℓ θ . The population intercept and profiled loss are c θ,a (x) = E π sampler θ t−k [ah θ | x] E π sampler θ t−k [a| x] , L ctr,a = 1 2β E x,Y∼π sampler θ t−k [a(h θ − c θ,a (x)) 2 ].(B.8) For a group of G responses, let a i = 1/T i , W = P i a i , and ̄ h a = W −1 P i a i h i . Then b L ctr,a = 1 2βG X i a i (h i − ̄ h a ) 2 = 1 4βGW X i,j a i a j (h i − h j ) 2 .(B.9) The empirical gradient isG −1 P i a i ( ̄ h a − h i )∇ θ logπ θ (y i )whether ̄ h a is differentiated or detached. Under full support and realizability, the unique zero-loss policy remains π ⋆ t,k , with intercept c ∗ . The proof is in Appendix H.6. The weighted mean must be recomputed at every trainer update, and it generally differs from both the ordinary mean and the fixedc ∗ . Direct estimation ofc ∗ by Equation(B.2)still uses the original, unweighted sampler distribution, and the unweighted(G− 1)/G bias formula does not carry over to random-length weights. B.5 Fixed Token Normalizers For comparison, the token-level analogue of a fixed normalizer is the log-partitionc ∗ tok of Equa- tion(4.4). Given exact sampler advantages, it can be computed over next actions or estimated from independent actions at the same history, but the advantages themselves require continuation information from the same sampler. Holding the advantage and normalizer fixed, the token residual, loss, and gradient are d tok θ,u = βℓ θ,u − A sampler t,k (x,y <u ,y u ) + c ∗ tok (x,y <u ), L fix tok (θ) = 1 2β E x,Y∼π sampler θ t−k X u (d tok θ,u ) 2 , ∇ θ L fix tok = E x,Y∼π sampler θ t−k X u d tok θ,u ∇ θ logπ θ (y u | x,y <u ). (B.10) 24 Under realizability, this loss vanishes exactly atπ θ = π + tok , by Theorem 3.2. The fixedc ∗ tok and the profiled interceptβk θ (s)agree at the PMD target and differ during training. A single logged action probability determines the sampled log-ratio, but neither a fixed normalizer nor the conditional score mean; Section 5 discusses what each KL estimator stores instead. B.6 A Terminal-Reward Form of the Token Objective One can also replace the advantage by the terminal reward inside a squared residual that keeps the full local KL. The resulting loss has the same population gradient asL tok . Proposition B.6 (Terminal-Reward Token Regression). Letb(s)be a fixed, action-independent baseline, and use complete rollouts from the specified sampler. Define b δ u = R− b(x,y <u ) β − ℓ θ,u − k θ (x,y <u ), L MC = β 2 E x,Y∼π sampler θ t−k X u b δ 2 u .(B.11) With exact local KLs,L MC −L tok does not depend onθ, so the population gradients agree, including for b = 0. The identity differentiates through k θ and holds the rollout law and the baseline fixed. The proof is in Appendix F.6. A prompt baseline shared by all tokens preserves the population gra- dient while adding noise and giving up the samplewise Bellman telescope. Whenb(s)̸= V sampler t,k (s), discarding the KL derivative generally changes the gradient, and a baseline fitted on the same rollout needs a separate argument. The local KL requires either the full next-token distributions or an esti- mator built from additional independent draws. Using the sampled action itself, b k θ (x,y <u ) =−ℓ θ,u , cancels the log-ratio and removes the learning signal. C ImplicitQ ⋆ Learning This appendix shows that the response Gibbs policy of Appendix A represents an optimal regularized action-value function implicitly. For a fixed collection samplerπ sampler θ t−k , letQ ⋆ β,t,k andV ⋆ β,t,k be the optimal values regularized toward that sampler. These differ both from the unregularized optimum and from the ordinary sampler valueQ sampler t,k that drives token-level KLPO. The results apply to SKLPO and to soft-value token regression; local PMD targets a different policy and does not recover these values. They are interpretation rather than a component of the token-level updates in the main text. C.1 The Regularized Bellman Target Consider a finite-horizon process with finite state and action spaces, bounded rewards, and a positive referenceπ sampler θ t−k (a| s) . The statesincludes time and the full observed history, andP (s ′ | s,a)is the environment’s transition law. We keep the terminal-reward setting: intermediate rewards are zero and R(S T ) is paid at termination. Define V ⋆ β,t,k (s) = max π E π,P R(S T )− β X u<T log π(A u | S u ) π sampler θ t−k (A u | S u ) S 0 = s .(C.1) 25 Proposition C.1 (Sampler-Regularized Bellman Equations). In this setting, the optimal values and policy satisfy Q ⋆ β,t,k (s,a) = E S ′ ∼P (·|s,a) V ⋆ β,t,k (S ′ ), V ⋆ β,t,k (s) = β log X a π sampler θ t−k (a| s)e Q ⋆ β,t,k (s,a)/β , π ⋆ β,t,k (a| s) = π sampler θ t−k (a| s)e (Q ⋆ β,t,k (s,a)−V ⋆ β,t,k (s))/β , (C.2) withV ⋆ β,t,k (s T ) = R(s T ) . For a fixed positive sampler, the regularized values converge to the unregularized optimal values as β ↓ 0. The proof is in Appendix F.2. These are the regularized Bellman equations of Geist et al. (2019) for the penaltyβ KL(π∥π sampler θ t−k ) , and the reference-weighted form of the Boltzmann policy in soft Q-learning (Haarnoja et al., 2017). Each state’s policy is again the Gibbs solution of Lemma 3.1, now with signalQ ⋆ β,t,k . The expectation overPkeeps environment outcomes conditioned on the chosen action. At finiteβ, future KL costs are part of the continuation value. Exponentiating the ordinary Q ⋆ omits them, and exponentiatingQ sampler t,k gives the local PMD update; neither solves this Bellman system in general. C.2 Soft-Value Token Regression In deterministic generation, soft continuation values factor the response Gibbs target into token conditionals. Regressing those local conditions gives a token-level objective with the same target as SKLPO. It needs the soft value at every prefix, which terminal returns with score centering do not provide. Theorem C.2 (Soft-Value Token Regression). Under Assumptions 2.1 and 2.3, define F (s) = β log E π sampler θ t−k [e R/β | s], π ⋆ t,k (a| s) = π sampler θ t−k (a| s)e (F (sa)−F (s))/β ,(C.3) wheresais the successor prefix,F (x) = c ∗ (x), andF = Rat termination. The local residuals satisfy ε u = F (s u+1 )− F (s u ) β − ℓ θ,u , X u ε u = R− c ∗ (x) β − ℓ θ (x,y).(C.4) For fixed exact F and a fixed sampler, L soft-tok = β 2 E π sampler θ t−k X u ε 2 u , ∇ θ L soft-tok =−βE π sampler θ t−k X u ε u ∇ θ logπ θ (a u | s u ).(C.5) Under realizability, the unique zero-loss policy is π ⋆ t,k , the SKLPO target. The proof is in Appendix F.5. Sinced θ =−β P u ε u , SKLPO squares the sum of the local residuals while soft-value token regression sums their squares. Unlike the PMD residuals of Theorem 4.3, these residuals are not conditionally centered, so the cross terms survive in expectation and the gradients differ. Soft-value token regression needsFat every prefix; SKLPO needs onlyRandF (x) = c ∗ (x). In deterministic generation, these soft values solve Equation(C.2). The loss sums over generated policy tokens and averages over responses, and Appendix J.2 distinguishes a global token mean from response-length weighting. 26 Token-level KLPOSKLPO Soft-value token regression Sampler advantage A sampler t,k profiled PMD regression Terminal reward R prompt normalization Soft continuation value F(x, y <u ) at each prefix Profiled token loss β 2 E π sampler θ t−k P u δ 2 u One response residual 1 2β E π sampler θ t−k [d 2 θ ] Soft-value increments β 2 E π sampler θ t−k P u ε 2 u Local PMD target π + tok Optimal soft-Q policy π ⋆ Optimal soft-Q policy π ⋆ Token or sequence regression † Response-level Gibbs view Same Gibbs target different regression loss † Both routes: complete matched-sampler rollouts with exact centering or full local KLs. Figure 3 Response-level and token-level targets. SKLPO and soft-value token regression share the response Gibbs target under exact fitting in deterministic generation. Token-level KLPO targets local PMD; its token- and sequence-regression forms have the same population gradient under the stated conditions and need no critic. C.3 The Implicit Action-Value Representation Define the policy’s implicit soft advantage by A imp θ (s,a) = β log π θ (a| s) π sampler θ t−k (a| s) .(C.6) Normalization of the policy gives P a π sampler θ t−k (a| s)e A imp θ (s,a)/β = 1. The log-ratio therefore represents differences of action values, and recovering absolute Q-values requires a Bellman-consistent state baseline as well. Theorem C.3 (Implicit Recovery of Optimal Soft Advantages). Under Assumptions 2.1 and 2.3, the regularized Bellman values are V ⋆ β,t,k (s) = F (s), Q ⋆ β,t,k (s,a) = F (sa), F (s) = β log E π sampler θ t−k [e R/β | s].(C.7) Suppose replay covers every feasible response and the Gibbs target is realizable. With exactc ∗ (x) for SKLPO, or exactFfor soft-value token regression, any zero-loss fit satisfies, at every covered prefix, A imp θ (s,a) = Q ⋆ β,t,k (s,a)− V ⋆ β,t,k (s) = F (sa)− F (s).(C.8) The same holds with fixed positive outer weights on either regression loss. The proof is in Appendix F.3. The theorem describes an exact fit, not the outcome of neural optimization. The log-ratio of an intermediate policy need not be Bellman-consistent, and absolute action values still require Q ⋆ β,t,k = V ⋆ β,t,k + A imp θ . With stochastic tools, the deterministic equivalence can fail, because an unrestricted trajectory tilt can change outcomes outside the policy’s control. Appendix I.6 gives a one-action counterexample. The Bellman equations remain feasible because they keep P fixed. 27 C.4 Policy Regression with an Optimal Q-Function When optimal regularized Q-values are available, the regression principle applies directly to action policies, including under stochastic transitions. Proposition C.4 (Regression against Optimal Soft Q-Values). Hold the reference and the exact target values fixed. For a fixed replay law ζ, define d Q θ (s,a) = β log π θ (a| s) π sampler θ t−k (a| s) − Q ⋆ β,t,k (s,a) + V ⋆ β,t,k (s), L Q (θ) = 1 2β E (S,A)∼ζ [d Q θ (S,A) 2 ], ∇ θ L Q = E ζ [d Q θ (S,A)∇ θ logπ θ (A| S)]. (C.9) Under state-action coverage and realizability, zero loss identifiesπ ⋆ β,t,k . Its state normalizer satisfies V ⋆ β,t,k (s) = E a∼π sampler θ t−k (·|s) Q ⋆ β,t,k (s,a) + β KL(π sampler θ t−k (·| s)∥π ⋆ β,t,k (·| s)).(C.10) In deterministic generation, d Q θ =−βε u , which recovers the soft-value token objective. The proof is in Appendix F.4. Equation(C.10)is Equation(3.7)once more. Appendix I.7 sketches a critic extension with sampled Bellman targets. A sampler’s terminal return estimates its ordinary continuation value and cannot in general replace the optimal regularized target. D Additional Details for the Main Text This appendix complements Sections 3–6. It compares the two instances of KLPO, summarizes the sequence-regression route, states the token-regression surrogate, contrasts sequence regression with the response-level instance, gives the TopK-KL and binary-KL surrogates, and collects the single-rollout requirements of the two instances. D.1 The Two Instances Table 4 places the response-level and token-level instances of Section 3.4 side by side. Response level (SKLPO)Token level (local PMD) Context, outcome prompt x, response yhistory s, next token a Signal freward R(x,y)sampler advantage A(s,a) Targetπ ⋆ t,k (y | x)∝ q(y | x)e R(x,y)/β π + tok (a| s)∝ q(a| s)e A(s,a)/β Fixed normalizer c ∗ (x) = β log E q [e R/β | x]c ∗ tok (s) = β log E q [e A/β | s] Profiled intercept E q [R| x] + β KL q(·| x)∥π θ (·| x) β KL q(·| s)∥π θ (·| s) Still requiredthe prompt’s mean reward and response KL, in practice several responses per prompt sampler advantages and the local KL Table 4 The two instances of KLPO, withq = π sampler θ t−k andA = A sampler t,k . Profiling removes the fixed normalizer from both. The response-level intercept still needs quantities that a single response cannot supply; the token-level intercept needs only the local KL, and Section 4 removes the advantages from the residual. 28 D.2 The Sequence-Regression Route Figure 4 traces sequence regression, the alternative to Steps 7–9 of Figure 1, through the results of Section 4.3 and the MC-KL estimator of Section 5.1. Sequence-regression alternative to Steps 7–9 Start from Step 6; complete rollouts from the matched sampler. v u,j iid ∼ q(·| s u ), r u,j := log q(v u,j |s u ) p(v u,j |s u ) , M ≥ 2. Step 7 † . KLPO Sequence Regression terminal return; no value baseline D θ := R− β X u [ℓ θ,u + k θ,u ], L SR := 1 2β E π sampler D 2 θ , L SR =L tok + constant in θ, ∇L SR =∇L tok . Step 8 † . Sequence-Regression Backprop differentiate the full local KL p u := π θ (y u | x,y <u ), q u := π sampler (y u | x,y <u ), L bp,SR θ :=− sg(D θ ) · X u [logp u + k θ,u ], ∇ AD θ E π sampler L bp,SR θ =∇ θ L tok . Step 9 † . Sequence Regression + MC-KL independent draws; leave-one-out feedback D j := R− β X u [ℓ θ,u + r u,j ], D −j := 1 M − 1 X l̸=j D l , L bp,SR-MC θ :=− 1 M X j sg( D −j ) · X u [logp u − logp(v u,j )]. ∇L SR =∇L tok =∇L TR =∇ AD θ E π sampler L bp,SR θ . Exact routes: same population gradient; different sample estimators. MC-KL preserves the expected gradient; TopK-KL and binary KL approximate it. Neither identity requires deterministic transitions. Figure 4 Sequence regression, an alternative to Steps 7–9 of Figure 1. Summing the token residuals of Step 6 along a trajectory gives one trajectory residual, which Bellman telescoping identifies with BPO’s under deterministic transitions (Step7 † ; Song et al., 2026). The prompt value can be dropped, and random tool outputs add only aθ-independent term, so the gradient is unchanged. Step8 † differentiates through the full local KLs. Step9 † estimates them withM ≥ 2independent draws per prefix and uses leave-one-out residuals to keep the gradient unbiased. The two routes share a population gradient but differ sample by sample. D.3 The Token-Regression Surrogate The per-token surrogate of Equation(4.10)produces the gradient of Theorem 4.2 by automatic differentiation. Proposition D.1 (Backward-Surrogate Equivalence). Hold the sampler fixed, detachh θ,u and the sampler probabilities in Equation(4.10), and differentiate both trainer log-probability terms. Then ∇ AD θ E x,Y∼π sampler θ t−k X u L bp,TR θ,u =−E x,Y∼π sampler θ t−k X u h θ,u e g θ (s u ,a u ) =∇ θ L TR =∇ θ L tok .(D.1) The proof is in Appendix E.4. 29 D.4 Sequence Regression and the Response-Level Instance Sequence regression and the response-level instance (SKLPO) both square one residual per response; they differ in the correction and in the target. With full local KLs, D SR (x,Y ) = R(x,Y )− β X u ℓ θ,u − β X u k θ (s u ), D SKLPO (x,Y ) = R(x,Y )− β X u ℓ θ,u − c ∗ (x), c ∗ (x) = β log E Y∼π sampler θ t−k (·|x) e R(x,Y )/β . (D.2) Sequence regression subtracts the path sum of current local KLs, which depends on the trainer and is differentiated. SKLPO subtracts a prompt normalizer that is fixed with respect to the trainer and must be supplied. The targets differ too: one is local PMD, the other the response Gibbs policy, and they coincide only in the special case noted in Section 3.4. D.5 TopK-KL and Binary-KL Surrogates TopK-KL. TopK-KL stores the headH u with the sampler’s original probabilities, together with the sampled token’s own log-probability even when that token lies outside the head. Withg v =∇ θ logp v and ρ = q tail /p tail , the gradient of Equation (5.5) is ∇ θ k K θ,u =− X v∈H u (q v − ρp v )g v ,(D.3) which amounts to modeling the sampler’s tail as proportional to the trainer’s, b q v = ρp v . Sequence regression substitutes TopK-KL into the residual and differentiates through it: D K θ := R− β X u [ℓ θ,u + k K θ,u ],(D.4) L bp,SRK θ :=− sg(D K θ ) X u [logπ θ (a u | s u ) + k K θ,u ].(D.5) Token regression uses the same correction as a score mean with a detached coefficient, following Marek and Ryabinin (2026, Section 4.1 and Appendix A): C θ (s) = X v∈H sg(q v − ρp v ) logp v , L bp,TRK θ,u =− sg(h θ,u )[logp a u − C θ (s u )].(D.6) The whole coefficient q v − ρp v is detached, including its dependence on the trainer through ρ. Proposition D.2 (TopK-KL Surrogates). Hold the sampler probabilities and the head fixed, and assume positive tail masses. Then Equation(D.3)holds,∇ θ C θ = −∇ θ k K θ , andL bp,SRK θ has the sample gradient of (D K θ ) 2 /(2β). With e g K θ,u = g θ,u +∇ θ k K θ,u , the two routes produce ∇ AD θ L bp,SRK θ =−D K θ X u e g K θ,u , ∇ AD θ X u L bp,TRK θ,u =− X u (R− βℓ θ,u ) e g K θ,u .(D.7) The proof is in Appendix E.7. The two routes share the score correction and differ only in the coefficient; Appendix N.2 gives numerical safeguards for small tail masses. 30 Binary KL. When only the sampled token’s probability is stored, the coarsest partition separates that token from everything else. Following BPO’s binary partition (Song et al., 2026, Equation (27)), with q u = π sampler θ t−k (a u | s u ) and p u = π θ (a u | s u ), define k bin θ,u := q u log q u p u + (1− q u ) log 1− q u 1− p u , b D θ := R− β X u [ℓ θ,u + k bin θ,u ], ω θ,u := 1− q u 1− p u . (D.8) Proposition D.3 (Binary-KL Surrogates). For 0 < p u ,q u < 1, with the sampler held fixed, ∇ θ [logp u + k bin θ,u ] = ω θ,u ∇ θ logp u .(D.9) Consequently the sequence surrogate L bp,SR-bin θ :=− sg( b D θ ) X u sg(ω θ,u ) logp u (D.10) has the sample gradient of b D 2 θ /(2β), and the token surrogate L bp,TR-bin θ,u :=− sg(R− βℓ θ,u ) sg(ω θ,u ) logp u (D.11) applies the same corrected score with the per-token coefficient. Herek bin is Equation(5.5)with heada u , which need not be the sampler’s most probable token, andω θ,u is a complementary-probability correction, not the importance ratiop u /q u . Binary KL depends on which action was sampled. The proof is in Appendix E.8, which also gives a three-token example in which the binary-corrected log-ratio has nonzero sampler mean. BPO’s practical loss uses this partition after linearizing the residual and adds group normalization, smoothing, and clipping (Song et al., 2026, Section 3.2). The surrogates above keep the residual and the coefficient dynamic. D.6 Reusing Auxiliary Draws Proposition 5.1 conditions on a trainer chosen independently of the auxiliary draws. Once the trainer has been updated with a set of draws, the draws are no longer independent of it. A conditionally unbiased gradient at every update therefore requires fresh draws from the historical sampler, which in turn requires its stored conditionals or the ability to rerun it. Reusing a fixed set of draws still defines a sensible training objective, a fixed sample-average surrogate, but the expectation identities of Section 5.1 no longer apply to it. D.7 Single-Rollout Requirements Table 5 lists what each instance needs to train from one rollout per prompt, as discussed in Section 6.2. 31 With one response per prompt Token-level KLPOResponse-level KLPO (SKLPO) Token-level criticnot needednot needed Prompt normalizernot neededsupplied, estimated from independent data, or learned as a prompt-level predictor Same-prompt groupnot neededneeded for group centering or a log-mean-exp estimate, both of which degenerate for a single response Per-token recordssampled-token log-probability, plus the KL estimator’s records sampled-token log-probability Unbiased gradientwith full KL or MC-KLgiven an exact normalizer Table 5 What each instance needs to train from one rollout per prompt. The comparison concerns information requirements. E Proofs for the Framework and Token-Level Results E.1 Gibbs Condition, Regression, and Profiling Proof of Lemma 3.1 and Theorem 3.2. Fixhand drop it from the notation. LetZ f = P z μ(z)e f (z)/β and c f = β logZ f , so that π ⋆ f = μe (f−c f )/β is positive and normalized. For any distribution π, E π f − β KL(π∥μ) = c f − β KL(π∥π ⋆ f ). The KL is nonnegative and vanishes only atπ = π ⋆ f , which proves the lemma. The residual is r θ = β log(π θ /π ⋆ f ), soπ ⋆ f attains zero loss, and positive replay mass and weights force every residual to vanish at any zero-loss fit. Finally,∇ θ r θ = β∇ θ logπ θ , and differentiating the squared loss with the replay law fixed and the weights held constant yields Equation (3.4). For Equation(3.5), writeℓ θ = log(π θ /μ). SinceE π θ ∇ θ logπ θ = 0 , the gradient of the regularized return isE π θ [(f − βℓ θ )∇ θ logπ θ ] . Adding the constant−c f inside the expectation changes nothing and turns the coefficient into−r θ . Changing the measure fromπ θ toμintroduces the ratioπ θ /μ. Proof of Proposition 3.3.LetX = βℓ θ −f. For any constantc,E μ [(X +c) 2 ] = Var μ (X) + (c + E μ X) 2 , so the minimizer is c θ =−E μ X = E μ f − βE μ ℓ θ = E μ f + β KL(μ∥π θ ), and the minimum is Var μ (X). Differentiating the profiled loss gives ∇ θ 1 2β E μ [(X + c θ ) 2 ] = E μ [(X + c θ )∇ θ logπ θ ] + ∇ θ c θ β E μ [X + c θ ], and the last expectation is zero by the choice ofc θ . The profiled loss vanishes exactly whenf − βℓ θ is constant on the support, that is, whenπ θ ∝ μe f/β ; normalization then givesπ θ = π ⋆ f . At that policyX = −c f identically, soc θ = c f , and substitutingπ θ = π ⋆ f into Equation(3.6)gives Equation (3.7). 32 E.2 Token PMD Target and Profiled Objective Proof of Equations (4.3) and (4.4) in Lemma 4.1.Apply Lemma 3.1 at each history withf = A sampler t,k . This gives Equations (4.4) and (4.3). The log-ratio of the target then yields β KL(π sampler θ t−k (·| s)∥π + tok (·| s)) = E a∼π sampler θ t−k (·|s) [c ∗ tok (s)− A sampler t,k (s,a)] = c ∗ tok (s), because the exact sampler advantage has zero conditional mean. The identityd fix θ,u = β log(π θ /π + tok ) and Theorem 3.2 give the regression gradient and the unique zero-loss target. Proof of the remaining claims of Lemma 4.1.This is Proposition 3.3 withf = A sampler t,k at each history. Since the advantage has zero sampler mean, c θ (s) = E a∼π sampler θ t−k [A sampler t,k (s,a)− β log(π θ (a| s)/π sampler θ t−k (a| s))] = βk θ (s).(E.1) The zero conditional means of the advantage and ofℓ θ,u + k θ (s u )giveE[δ u | s u ] = 0. Differentiating the profiled loss yields−βE P u δ u [g θ (s u ,a u ) +∇ θ k θ (s u )], and the term with∇ θ k θ (s u )vanishes after conditioning ons u , which proves Equation(4.6). At zero loss, the local residual is constant over actions, and normalization fixes the intercept to c ∗ tok and the policy to π + tok . E.3 Critic-Free Token Regression Proof of Theorem 4.2.Fix a visited historysand writeqfor its sampler. Letg a = ∇ θ logπ θ (a | s) , m θ = E q g a ,h = R− β log(π θ (a | s)/q(a | s)), and ̄ h θ (s) = E q [h | s]. The law of the action and its continuation does not depend on θ, so ∇ θ Var q (h| s) 2β =−E q [(h− E q [h| s])g a | s] =−E q [h(g a − m θ )| s]. Summing over the sampler’s visitation distribution, which is also fixed, proves Equation(4.9). Since E q [g a − m θ | s] = 0, any fixed action-independentb(s)can be subtracted fromh. A baseline fitted to the same sampled action or continuation would need a separate argument. Letξ = R− Q sampler t,k (s,a). ThenE q [ξ | s,a] = 0and ̄ h θ (s) = V sampler t,k (s) + βk θ (s) , soh− ̄ h θ = βδ + ξ. The cross term vanishes after conditioning on (s,a), and L TR −L tok = 1 2β E x,Y∼π sampler θ t−k X u Var π sampler θ t−k (R| x,y ≤u ).(E.2) The right side does not depend onθ, so Lemma 4.1 gives the minimizers; under realizability, the unique one isπ + tok . The token-regression loss generally does not vanish there, because terminal returns carry continuation noise. With the stop-gradients of Equation(4.10), each sample contributes−h(g a − m θ ). A single complete rollout therefore gives a stochastic estimate of the population gradient, provided the conditional score mean is supplied independently of the sampled action. The surrogate’s value is neither the variance objective nor an estimate of it. Differentiating through h would add a product- rule term, and replacinghby the terminal reward gives the ordinary score-centered policy gradient, without the regression feedback. 33 The fixed-normalizer and profiled objectives have different gradients away from the PMD target. Score centering computes the gradient of the profiled objective without a critic. It does not evaluate the unknownc ∗ tok , and the zero-mean identity does not make this gradient an importance-corrected gradient of the current policy’s return. E.4 The Backward Surrogate Proof of Proposition D.1. Because h θ,u and π sampler θ t−k are detached, ∇ AD θ L bp,TR θ,u =−h θ,u " ∇ θ logπ θ (a u | s u )− X v π sampler θ t−k (v | s u )∇ θ logπ θ (v | s u ) # =−h θ,u e g θ (s u ,a u ). Taking the sampler expectation gives Equation(4.9), and Theorem 4.2 gives the remaining equalities. E.5 Bellman Telescoping and the Prompt Baseline Proof of Theorem 4.3. For v < u, the residual δ v is determined by (x,y <u ), so E π sampler θ t−k [δ v δ u ] = E π sampler θ t−k [δ v E π sampler θ t−k (δ u | x,y <u )] = 0. Padding with zero residuals after termination and expanding the square proves Equation(4.11). Under Assumption 2.3,A sampler t,k (x,y <u ,y u ) = V sampler t,k (x,y ≤u )− V sampler t,k (x,y <u ), so the advantages along a response sum toR− V sampler t,k (x). This proves Equation(4.12)and the loss identity for every smooth parameterized policy. A fixed prompt weight can be applied to both sides. The trajectory form uses only the terminal reward and the prompt value, whereas the token form uses intermediate advantages; in a finite batch, the cross-terms remain. Proof of Proposition 4.4.Fix a prompt and a collection sampler, and writeV = V sampler t,k andQ = Q sampler t,k . Since V = R at termination, R− V (x) = X u V (s u+1 )− V (s u ) = X u A sampler t,k (s u ,a u ) + ε u , so that D θ = V (x) + β X u δ u + X u ε u . Every cross term vanishes in expectation. Bothδ u andε u have zero mean givenx, which removes the terms withV (x). Foru≤ v,δ u is determined by(s v ,a v )andE[ε v | s v ,a v ] = 0; foru > v,ε v is determined by s u and E[δ u | s u ] = 0. With Equation (4.11), this gives E[D 2 θ ] = β 2 E X u δ 2 u + E h V (x) 2 + X u ε u 2 i , and dividing by2βproves Equation(4.14). The last expectation involves only the sampler and the environment, so it does not depend onθ. ReplacingRbyR− b(x)replacesV (x)byV (x)− b(x). 34 Under Assumption 2.3,V (s u+1 ) = Q(s u ,a u ), so everyε u is zero. All sampler quantities andbstay fixed during differentiation. Together with Theorem 4.2, this proves Equation(4.17)for every smooth parameterized trainer, without realizability, which is needed only for the claim about the common target. Finally,D θ = V (x) + βδ BPO holds by the definition ofδ BPO , soδ BPO = P u δ u + β −1 P u ε u , and the same cross-term argument gives (β/2)E[δ 2 BPO ] =L tok + E[( P u ε u ) 2 ]/(2β). E.6 Monte Carlo KL and Leave-One-Out Residuals Proof of Proposition 5.1. Condition on a complete rolloutYand on a trainerθchosen independently of the auxiliary draws. The visited histories, rollout actions, reward, and sampler conditionals are then fixed. Write g u (v) =∇ θ logp θ (v | s u ), and for each auxiliary column define Z j = X u log q(v u,j | s u ) p θ (v u,j | s u ) , G j = X u [g u (a u )− g u (v u,j )]. Their conditional means areE MC Z j = P u k θ (s u )andE MC G j = P u e g θ,u , andD j = R−β( P u ℓ θ,u +Z j ) has meanD θ . Token regression multiplies its MC score estimate byh θ,u , which is fixed given the rollout, so its conditional expectation is the full-KL sample gradient. For sequence regression, the auxiliary columns are independent givenY, soD −j is independent of G j and E MC − 1 M X j D −j G j =−D θ X u e g θ,u . Since∇ θ D j = −βG j , differentiating the sum of ordered cross-products in Equation(5.4)gives exactly this estimator before taking expectations. Forj ̸= l, independence givesE MC D j D l = D 2 θ , which proves the loss identity. Averaging over sampler rollouts then completes the proof under the original full-KL assumptions. The plug-in square is biased. LetD = M −1 P j D j . With draws independent across prefixes, E MC D 2 2β = D 2 θ 2β + β 2M X u Var v∼q(·|s u ) log q(v | s u ) p θ (v | s u ) .(E.3) The extra variance term depends on the trainer, so differentiating the plug-in square does not give an unbiased full-KL gradient. ForM ≥ 2, the U-statistic subtracts an unbiased estimate of that term: b L SR,MC = 1 2β D 2 − 1 M − 1 1 M X j (D j − D) 2 . Neither this quantity nor a single MC-KL estimate has to be nonnegative. The leave-one-out coefficient isD −j =D− (D j −D)/(M − 1), so no explicit enumeration of pairs is needed. Splitting the draws into independent groups would also estimate the product, but the leave-one-out construction uses all Mdraws symmetrically. WithM = 1there is no independent second residual, so sequence regression is undefined, while token regression remains valid. 35 Reuse. All conditional identities above assume a trainer chosen independently of the auxiliary records. After the trainer has been updated with a fixed set of records, conditioning on the updated trainer can destroy their original i.i.d. law. Recomputing the current log-probabilities on the same records gives a fixed-sample surrogate, and drawing fresh auxiliary tokens from the same historical sampler restores the conditional unbiasedness. Sampling from the current trainer, deduplicating token IDs, clipping negative estimates, or reusing the rollout action each changes the estimator. E.7 TopK-KL Proof of Proposition D.2. At a fixed history,∇p tail = − P v∈H p v g v . Differentiating Equation(5.5) with the sampler records fixed gives ∇k K θ =− X v∈H q v g v − q tail p tail ∇p tail =− X v∈H (q v − ρp v )g v , which is Equation(D.3). With the whole coefficient detached,∇ θ C θ = P v∈H (q v −ρp v )g v =−∇ θ k K θ . The chain rule gives ∇ θ (D K θ ) 2 2β =−D K θ X u [g θ,u +∇ θ k K θ,u ], which is the gradient of Equation(D.5), and differentiating Equation(D.6)gives the token-route expression in Equation (D.7). Equation(5.6)is the chain rule for KL applied to the partition of the vocabulary into the head tokens and one tail. As a consequence,E a∼q [log(p a /q a ) + k K θ ] = k K θ − k θ is generally nonzero, and the conditional-centering, baseline-invariance, and route-equivalence arguments do not extend to a finite head. A singleton tail gives the full KL, and a singleton head consisting of the sampled action gives the binary partition. These statements concern unfloored probabilities. Gradient error of the top-Kscore correction. At a fixed history, take the headHfrom the stored sampler with unrenormalized probabilitiesq v . LetT =V \ Handρ = q tail /p tail , and set b q v = q v on H and b q v = ρp v on T . Since P v p v g v = 0, b m θ (s) := X v b q v g v = X v∈H (q v − ρp v )g v ,(E.4) so detaching the whole coefficient in Equation(D.6)gives∇ θ C θ = b m θ . Detachingρalone is not enough. Token regression keeps the original loggedq y u inh θ,u , even when the sampled action lies outside H. With rollouts from q, the conditional gradient error of the top-K update is E q [−h(g a − b m θ )| s]− E q [−h(g a − m θ )| s] = ̄ h θ (s)( b m θ (s)− m θ (s)).(E.5) With a fixed baseline, ̄ h θ is replaced by ̄ h θ −b. The error vanishes for exact centering, and in particular when the modeled and the actual tails agree. Because it is proportional to ̄ h θ , a finite head makes the update depend on reward offsets. The top-Ksurrogate is therefore an approximation, not the gradient of the exact variance objective. 36 E.8 Binary KL Proof of Proposition D.3.For one sampled action, keepqfixed. Differentiating the Bernoulli KL gives ∇ θ k bin = −q + (1− q)p 1− p ∇ θ logp, and adding∇ θ logp multiplies the score by (1− q)/(1− p). Therefore ∇ θ b D 2 θ 2β =− b D θ X u ω θ,u ∇ θ logp u , which is the gradient of the stopped surrogate. The token surrogate applies the same corrected score with coefficient h θ,u . These are identities for the binary objective, not for the full-KL objective. The binary-corrected log-ratiolog(p a /q a ) + k bin (a)generally has nonzero mean undera∼ q, so the baseline-invariance argument does not extend to binary KL. For example, takeq = (1/3, 1/3, 1/3),p = (1/2, 1/3, 1/6), and a constant reward. After adding the full KL, the log-ratio has zero sampler mean. The binary- corrected log-ratio has a negative mean, because coarsening lowers the KL and the binary KL is strictly below the full KL for at least one sampled action. A reward offset can therefore change the binary objective’s gradient. With a vocabulary of two tokens, the binary partition is the full vocabulary. E.9 Mixtures of Sampler Versions Proof of Proposition A.6. For each component,d θ = β log(π trainer θ /π ⋆ t,k ). Positive sampling mass and outer weights force that residual to vanish on its support at zero loss. A positive combination of nonnegative component losses is zero exactly when every component is zero, so a shared trainer must equal every component target. If full-support samplersp 1 ,p 2 have the same Gibbs target for a common reward andβ, thenp 1 (y)e R(y)/β /Z 1 = p 2 (y)e R(y)/β /Z 2 for everyy. Their ratio is the constant Z 1 /Z 2 , and normalization makes it one. Proof of Propositions A.5 and 6.1.For SKLPO, the collection scores, lengths, and fixed normalizers are constant, so∇ θ d i,θ = β∇ θ logπ trainer θ (y i | x i ), and differentiation gives the weighted batch gradient. For token regression, the stop-gradients in Equation(4.10)give−h i,u,θ e g i,u,θ for each token; applying Theorem 4.2 to each collection version and averaging with fixed version frequencies gives the expected mixture gradient. For sequence regression, apply Proposition 4.4 to each version and differentiate its surrogate. Outcome-dependent replay selection can break these identities. F Proofs for the Response-Level and Value Results F.1 Response Targets and Positive Weights Proof of Theorems A.1 and A.2.For normalized policies the mass correction inUKLvanishes, and Lemma 3.1 withμ = π sampler θ t−k andf = Rgives Equations(A.2)–(A.3). Equivalently, stationarity under P y π(y) = 1givesR(y)−β log(π(y)/π sampler θ t−k (y)) = λ, and normalization fixesλ = c ∗ . Equation(A.4) 37 usesη −1 = β. ReplacingRbyR− b(x)multipliesZbye −b(x)/β and leaves the normalized target unchanged. Taking the logarithm of Equation(A.2)givesd θ = β log(π θ /π ⋆ t,k ) , and Theorem 3.2 identifies the unique zero-loss response distribution. Equation(A.3)gives the reverse-KL criterion of the regularized return, and Equation(A.7)the squared log-ratio criterion of the regression. Sharing a zero on the full simplex does not make their projections onto a restricted model class equal. Proof of Proposition A.4.Atπ ⋆ t,k every residual is zero. Conversely, positivity of the sampler and ofa forces every residual to vanish at zero weighted loss, so the weight leaves both the target and its normalizer unchanged. With a = 1/T fixed, differentiation gives ∇ θ L a = E π sampler θ t−k [ad θ ∇ θ logπ θ (Y )] = E π sampler θ t−k " d θ 1 T X u ∇ θ logπ θ (y u | x,y <u ) # , which is Equation (A.10). F.2 Regularized Bellman Values Proof of Proposition C.1.At a nonterminal state, condition on the first action and the successor state in Equation(C.1). Once the optimal continuation values are fixed by backward induction, the remaining problem is max π(·|s) X a π(a| s) E P V ⋆ β,t,k (S ′ )− β log π(a| s) π sampler θ t−k (a| s) . Lemma 3.1 gives the value and policy in Equation(C.2), starting from the terminal boundary V ⋆ β,t,k = R. The maximization changes the action law and keeps P . The unregularized values satisfyQ ⋆ (s,a) = E P V ⋆ (S ′ )andV ⋆ (s) = max a Q ⋆ (s,a)with the same terminal boundary. For a fixed positive reference, letp min = min s,a π sampler θ t−k (a | s) > 0over feasible pairs and letHbound the remaining horizon. The unregularized optimal value bounds the regularized value from above, and evaluating a deterministic unregularized optimal policy in the regularized objective gives the lower bound V ⋆ (s)− βH log(1/p min )≤ V ⋆ β,t,k (s)≤ V ⋆ (s). The state values therefore converge asβ ↓ 0, and taking the fixed transition expectation gives convergence of the action values. F.3 Implicit Soft Advantages Proof of Theorem C.3. For deterministic transitions,Q ⋆ β,t,k (s,a) = V ⋆ β,t,k (sa). Starting from the termi- nal value F = R, backward induction gives e F (s)/β = X a π sampler θ t−k (a| s)e F (sa)/β = E π sampler θ t−k [e R/β | s]. HenceV ⋆ β,t,k = FandQ ⋆ β,t,k (s,a) = F (sa), and the product of the optimal token conditionals telescopes to the response Gibbs distribution. For SKLPO, zero loss and coverage implyπ θ (y | x) = 38 π ⋆ t,k (y | x) for every feasible response, so the token conditionals agree at every prefix with positive mass. For soft-value token regression, each zero residual gives those conditionals directly. Substituting into the Bellman policy proves Equation(C.8), and fixed positive outer weights do not affect these zero-residual arguments. F.4 Regression against Optimal Soft Q-Values Proof of Proposition C.4.The Bellman policy givesd Q θ = β log(π θ /π ⋆ β,t,k ) , and Theorem 3.2 identifies the zero-loss conditional policy and the gradient. Taking the sampler expectation of the identity β log π ⋆ β,t,k (a| s) π sampler θ t−k (a| s) = Q ⋆ β,t,k (s,a)− V ⋆ β,t,k (s) yields Equation(C.10). Under deterministic transitions, Equation(C.7)givesd Q θ = βℓ θ − F (sa) + F (s) =−βε u . F.5 Soft-Value Token Factorization Proof of Theorem C.2.Under deterministic generation, the Gibbs probability of a prefix is propor- tional to its sampler probability timesE π sampler θ t−k [e R/β | s] . The ratio of the masses ofsaandsgives Equation(C.3). SummingF (x,y ≤u )− F (x,y <u )telescopes toR− F (x) = R− c ∗ (x), which proves Equation(C.4). WithFfixed,∇ θ ε u =−∇ θ logπ θ (y u | x,y <u ), which gives Equation(C.5). Positive replay mass forces all local residuals to vanish at zero loss, and the factorization then identifies π ⋆ t,k . F.6 Terminal-Reward Token Regression Proof of Proposition B.6.Writeξ u = R− Q sampler t,k (x,y <u ,y u ) andm u = V sampler t,k (x,y <u )− b(x,y <u ) . Then b δ u = δ u + (ξ u +m u )/β. Given(x,y ≤u ),ξ u has zero mean, and given(x,y <u ),δ u has zero mean. Expanding the square gives L MC −L tok = 1 2β E x,Y∼π sampler θ t−k X u Var π sampler θ t−k (R| x,y ≤u ) + (V sampler t,k (x,y <u )− b(x,y <u )) 2 . (F.1) Every term on the right depends only on the fixed sampler and baseline, so its derivative inθis zero. The identity differentiates the full residual, includingk θ . Ifb̸= V sampler t,k , thenE[ b δ u | x,y <u ] = m u /β is generally nonzero, and the KL derivative cannot be dropped. G RPG–URKL and Gradient Identities We compare unweighted SKLPO with RPG on unfiltered responses from a fixed historical sampler π sampler θ t−k , which is off-policy relative to the trainer. Equation(A.8)gives the unweighted SKLPO gradient, and Section A.3 the length-weighted version. Writew θ = e ℓ θ = π θ /π sampler θ t−k . For normalized π sampler θ t−k , the fully differentiable URKL surrogate of Zhang et al. (2026, Proposition 3.6) is L RPG (θ) = E π sampler θ t−k [−w θ R + β(w θ logw θ − w θ )],(G.1) 39 which equals−J (π θ )− β. No clipping is applied in the identities below. Proof of Proposition A.3. Differentiating Equation(A.7)gives Equation(A.8). Differentiating Equa- tion(G.1)givesE π sampler θ t−k [w θ (β logw θ −R)∇ θ logπ θ ], and addingc ∗ leaves this expectation unchanged because E π sampler θ t−k [w θ ∇ θ logπ θ ] =∇ θ P y π θ (y) = 0. For a common response, residual, and score,g RPG = e ℓ θ g SKLPO . The SKLPO coefficientd θ is affine in the accumulated log-ratio, and RPG multiplies it by the product of token ratiose ℓ θ . At π θ = π sampler θ t−k the population gradients agree, although different baseline estimates can give different finite-sample gradients. With outer weight1/T, the sample identity becomesg RPG = w θ T g len SKLPO , so the matched-policy equality applies only to the unweighted regression. Multiplying the weighted loss by sg(w θ T ) would recover the RPG gradient, but KLPO does not use that multiplier. A stop-gradient importance weight also recovers the RPG gradient: L sg (θ) = 1 2β E π sampler θ t−k [sg(w θ )d 2 θ ], ∇ θ L sg =−∇ θ J (π θ ).(G.2) This is a surrogate for automatic differentiation, not an equality of scalar objectives. Differentiating through the weight as well would add 1 2β E π sampler θ t−k [w θ d 2 θ ∇ θ logπ θ (y)].(G.3) Replacingc ∗ by any fixed prompt baseline in Equation(G.2)still gives the population RPG gradient. The unweighted regression lacks this baseline invariance, because its zero-loss target depends on the correct normalizer. RPG–URKLUnweighted SKLPO with exact c ∗ Criterion, up to constantsβ KL(π θ ∥π ⋆ t,k ) β 2 E π sampler θ t−k [log 2 (π θ /π ⋆ t,k )] Gradient weight on π sampler θ t−k data w θ d θ d θ Unrestricted optimumπ ⋆ t,k π ⋆ t,k Neural approximationOptimizes regularized returnFits the optimality condition Table 6 SKLPO removes the current-policy importance weight. The two objectives share the unrestricted optimum but have different gradients and, in general, different best approximations within a model class. H Normalizer Identities and Estimation H.1 The Normalizer as a KL Divergence Proof of Lemma B.1. Equation(B.1)is Equation(3.7)with(μ,f ) = (π sampler θ t−k ,R) . For the estimator, e bc/β = G −1 P i e R i /β is unbiased for Z, and Jensen’s inequality gives E b c≤ β logZ = c ∗ . With centered rewards A sampler t,k = R− V sampler t,k , the normalizer is a pure KL term: c ∗ A (x) := β log E π sampler θ t−k [e A sampler t,k (x,Y )/β ] = c ∗ (x)− V sampler t,k (x) = β KL(π sampler θ t−k ∥π ⋆ t,k ).(H.1) 40 Writing κ ∗ (x) = KL(π sampler θ t−k ∥π ⋆ t,k ), the fixed-target residual becomes − d θ (x,y) β = A sampler t,k (x,y) β − ℓ θ (x,y)− κ ∗ (x).(H.2) This removes the response partition function in the same way that BPO removes the local partition function from its PMD optimality condition (Song et al., 2026, Section 3.1). The identity still involves the unknown optimum; substituting the currentπ θ forπ ⋆ t,k gives the profiled loss and changes the intercept. For largeβ,c ∗ = V sampler t,k + Var π sampler θ t−k (R)/(2β) +O(β −2 ), which quantifies the correction that a mean-reward baseline omits. H.2 Binary Rewards For rewards taking the valuesr − andr + , letρ π sampler θ t−k (x)be the success probability underπ sampler θ t−k . Then c ∗ (x) = β log (1− ρ π sampler θ t−k (x))e r − /β + ρ π sampler θ t−k (x)e r + /β .(H.3) This covers both0/1rewards and negative failure rewards, and a success-rate estimate gives a plug-in normalizer. The rate must belong to the prompt and the collection sampler; a global average, or an exponential moving average over changing policies, generally estimates something else. H.3 Profiling the Response Intercept Proof of Proposition B.2.This is Proposition 3.3 with(μ,f ) = (π sampler θ t−k ,R), which gives Equa- tions(B.3)and(B.5). Sinceκ θ =−E π sampler θ t−k ℓ θ , the residual in Equation(B.4)is(h θ − E π sampler θ t−k h θ )/β . Under sampling from the sampler, the fixed-normalizer and centered losses differ by L SKLPO,π sampler θ t−k −L ctr,π sampler θ t−k = 1 2β E x [(c ∗ (x)− c θ (x)) 2 ].(H.4) Since E π sampler θ t−k δ KL = 0 for each prompt, the derivative of κ θ cancels after expectation: ∇ θ L ctr =−βE x E Y∼π sampler θ t−k (·|x) [δ KL (x,Y ;θ)∇ θ logπ θ (Y )].(H.5) Neither the loss nor the gradient contains an importance weight. For a group fromπ sampler θ t−k , the estimates b V = ̄ R and b κ θ = − ̄ ℓgive b δ i = (R i − ̄ R)/β − (ℓ i − ̄ ℓ), which recovers Equation(B.7). The finite-group bias statement there assumes fixed parameters and independent samples; it is not a generalization guarantee after fitting the batch. H.4 Empirical Centering and Singletons Proof of Lemma B.3. Expanding the pairwise square gives P i,j (h i − h j ) 2 = 2G P i (h i − ̄ h) 2 , which proves Equation(B.7). In the gradient, every term containing∇ θ ̄ his multiplied by P i (h i − ̄ h) = 0, so detaching the mean gives the same gradient. At fixedθ, independence givesE[G −1 P i (h i − ̄ h) 2 ] = (G− 1) Var(h θ )/G. 41 Proof of Proposition B.4.ForG = 1, the log-mean-exp estimator is b c = β loge R 1 /β = R 1 . At matched policiesℓ θ = 0, sod θ = 0. Empirical centering sets ̄ h = h 1 at every policy, and a weighted singleton also has ̄ h a = h 1 , so the centered residuals and losses are identically zero. H.5 Data Selection and the Normalizer Fix a samplerπ sampler θ t−k and letνbe its response law after outcome-dependent selection. Unfiltered collection givesν = π sampler θ t−k even whenk > 0and the current trainer differs from the sampler; selection can break this equality. With the exact c ∗ , consider L SKLPO,ν (θ) = 1 2β E Y∼ν [d θ (Y ) 2 ].(H.6) Ifνis positive on the support ofπ sampler θ t−k and the target is realizable, this loss has the same unique zero-loss policyπ ⋆ t,k , and it needs no importance weight to estimate its own gradient. Under model misspecification, though, its best approximation can differ from that of the sampler-conditioned loss L SKLPO,π sampler θ t−k , and finite data do not guarantee coverage. Estimatingc ∗ fromνis a separate problem, for which the general importance-sampling identity is Z = E Y∼ν π sampler θ t−k (Y ) ν(Y ) e R(Y )/β .(H.7) This weight concerns normalizer estimation only; the SKLPO loss and algorithm do not use it. Unfiltered data from the corresponding sampler need no correction, while an uncorrected exponential average under a differentνestimates a different normalizer. Alternatively, profiling the intercept underνgivesc θ,ν = E ν [R− βℓ θ ]and aν-variance loss. Under the same support and realizability assumptions, that loss identifiesπ ⋆ t,k without a density ratio, with finite-sample behavior that depends on ν. H.6 Weighted Profiling Proof of Proposition B.5. Fix a prompt and a sampler version. Letνbe the possibly selected response law anda(y) > 0a fixed weight, and setZ a = E ν a,ν a (y) = a(y)ν(y)/Z a , andc θ,a = E ν a h θ . Completing the square gives E ν [a(c− h θ ) 2 ] = E ν [a(h θ − c θ,a ) 2 ] + Z a (c− c θ,a ) 2 ,(H.8) so the profiled loss isZ a Var ν a (h θ )/(2β). The pairwise variance identity underν a gives Equation(B.9) for an empirical group. The reweighted law changes the regression measure, but the sampler in ℓ θ = log(π θ /π sampler θ t−k ) and the fixed Gibbs normalizer remain those of the original target. With g θ =∇ θ logπ θ (Y ), differentiation gives ∇ θ L ctr,a =−E ν [a(h θ − c θ,a )g θ ] =−Z a E ν a [h θ (g θ − E ν a g θ )]. (H.9) The intercept derivative vanishes becauseE ν [a(h θ − c θ,a )] = 0; an ordinary, unweighted intercept would not cancel in the weighted loss. Positiveapreserves the support, so zero weighted variance still identifies the original Gibbs target under full coverage and realizability. 42 H.7 Learned Normalizers for Single Rollouts Fix a collection versionv = t−kandβ. A predictor takes the prompt and collection-policy information (x,v)as input, and its prediction must describe that sampler rather than the current trainer. One response per prompt supplies one training example. Sharing parameters across prompts allows gen- eralization, but it does not turn a single observed reward into a conditional expectation. We describe two fixed-normalizer estimators and a learned-intercept alternative; all three are implementation choices rather than assumptions of the response-level objective. Binary rewards: predict the success probability. ForR ∈r − ,r + , trainp φ (x,v)∈ (0, 1)with the Bernoulli negative log-likelihood L p (φ) = E x,Y∼π sampler θ t−k [−z logp φ (x,v)− (1− z) log(1− p φ (x,v))], z = 1R = r + . (H.10) The unrestricted conditional minimizer is the sampler’s success probability. Substituting the prediction into Equation (H.3) gives b c φ (x,v) = β log h (1− p φ (x,v))e r − /β + p φ (x,v)e r + /β i ,(H.11) which should be evaluated in log space. A single success or failure label can updateφ, but it does not justify setting p φ to that label before scoring the same response. General bounded rewards: predict an exponential moment. LetMbe a known upper bound on the reward and set q(Y ) = e (R(x,Y )−M )/β ∈ (0, 1]. A positive predictor m φ (x,v) can be trained by L m (φ) = E x,Y∼π sampler θ t−k [(m φ (x,v)− q(Y )) 2 ], b c φ (x,v) = M + β logm φ (x,v).(H.12) Its unrestricted conditional minimizer isE π sampler θ t−k [q(Y )| x] , which givesc ∗ . The shift byMbounds the regression labels without changing the normalizer, although smallβcan still make rare high rewards decisive. Regressingc φ directly ontoRwould learn the mean reward instead. Both predictors use unfiltered rewards under the specified sampler and do not use the policy loss’s1/Tweight. Outcome-dependent selection, length weighting, or pooling across versions changes the conditional quantity being learned, and a numerical floor on m φ adds a further approximation. Using a fixed prediction in policy updates. Train the predictor on separate records and freeze a snapshot before scoring the policy-training responses. Compute b c φ (x,v)from that snapshot, detach it from the policy computation, and keep the same prediction while reusing those responses; newly observed rewards can train a later snapshot. For replayed data, the independence premise of Appendix I.4 requires separate estimation data or held-out folds, so that the predictor used for a response was not fitted on it. Detaching a prediction, or evaluating it before the current optimizer step, does not undo earlier fitting on the same record. These routes learn a prompt-level value or normalizer: they avoid a token-value critic, not all auxiliary value estimation. Learning an intercept jointly with the policy. Instead of estimating the fixedc ∗ , the intercept can be parameterized inside a weighted regression. With separate parameters θ and φ, let J (θ,φ) = 1 2β E x,Y∼π sampler θ t−k [a(Y )(βℓ θ − R + c φ (x,v)) 2 ], a(Y ) = 1/T (Y ).(H.13) 43 At fixedθ, unrestricted minimization overc φ givesc θ,a of Equation(B.8). A stochastic update first computesh i = R i − βℓ i andc φ (x i ,v i )before either parameter moves. The policy step treatsc φ as fixed, and the auxiliary step minimizes b L c (φ) = 1 2βB X i a i c φ (x i ,v i )− sg(h i ) 2 (H.14) with the pre-updateh i . These are the two partial gradients ofJat the pre-update parameters. Auxiliary gradients should stay out of the policy parameters; a small head on fixed prompt features is one option. The intercept must be refreshed as the trainer changes. One response per prompt gives a stochastic update of this joint objective, but an imperfect predictor does not give the exactly profiled policy gradient. Under the same full-support and realizability assumptions, a zero-loss joint fit satisfiesR−βℓ θ = c φ (x,v) on the response support, and normalization then givesc φ = c ∗ andπ θ = π ⋆ t,k for that sampler. Finite singleton data need not identify this solution: an overly flexible predictor can memorize each prompt’s observed residual, and a free intercept for every response cancels its residual altogether. The learned route therefore depends on generalization across prompts; it is not empirical centering of one response. I Additional Off-Policy Results I.1 The Local Importance Multiplier Writeg θ (s,a) = ∇ θ logπ θ (a | s). For a token from responsei, the behavior conditional and the trainer-to-sampler ratio are ν i (a| s) = π sampler θ t−k i (t) (a| s), ρ t,i (s,a) = π trainer θ t (a| s) π sampler θ t−k i (t) (a| s) .(I.1) Proposition I.1 (Local Gradient Comparison). At a fixed historys, use the fixed-normalizer local PMD residuald θ,s (a)of Equation(B.10), with the collection samplerν i as reference. Forρ θ = π θ /ν i , the per-sample contributions to the negative local regularized-return gradient and to the regression gradient are g IS (s,a) = ρ θ (s,a)d θ,s (a)g θ (s,a), g KLPO (s,a) = d θ,s (a)g θ (s,a),(I.2) and their expectations under a∼ ν i (·| s) are the respective gradients. Writing∆ = logρ t,i , importance sampling multiplies each contribution bye ∆ , and KLPO has neither this multiplier nor a clipping threshold. Older checkpoints can be farther from the trainer, and longer trajectories offer more chances for an outlying token ratio. Outcome-dependent admission can change the action distribution that the return-gradient identity requires. FlashREINFORCE multiplies its token ratio by a batch advantage, while the length-weighted SKLPO update putsd i /T i on each token score with no ratio. Both average scores within a response, but their residuals differ; Appendix I.2 gives the comparison. 44 Proof of Proposition I.1. At a fixed history, the conditional regression loss and gradient are d θ,s (a) = β log π θ (a| s) π sampler θ t−k (a| s) − A sampler t,k (s,a) + c ∗ tok (s), L fix tok (θ;s) = 1 2β E a∼π sampler θ t−k (·|s) [d θ,s (a) 2 ], ∇ θ L fix tok (θ;s) = E a∼π sampler θ t−k (·|s) [d θ,s (a)g θ (s,a)]. (I.3) For the local returnJ s = E π θ A sampler t,k − β KL(π θ ∥ν i ) , Equation(3.5)gives−∇ θ J s = E π θ [d θ,s g θ ] . Changing the action measure to ν i introduces ρ θ = π θ /ν i and proves Equation (I.2). I.2 Comparison with FlashREINFORCE Letν i = π sampler θ t−k i (t) be the behavior policy that generated responsei, andg i,u = ∇ θ logπ θ (y i,u | x i ,y i,<u ). With the batch advantageA i , sequence maskm i , and behavior probabilities held fixed, the FlashREINFORCE policy-loss gradient is b g Flash =− 1 B X i m i A i T i X u π θ (y i,u | x i ,y i,<u ) ν i (y i,u | x i ,y i,<u ) |z ρ i,u : token IS g i,u .(I.4) Both methods use the actual collection probability: FlashREINFORCE places the ratio in a mul- tiplicative weight, and SKLPO puts its logarithm in the response residual. Equation(A.11)gives b g len SKLPO = B −1 P i (d i /T i ) P u g i,u , so both updates average token scores within a response, but the SKLPO coefficient is the complete-response optimality residual rather than a batch advantage. Equations(I.3)and(I.2)isolate the multiplier at a single history with a local regularized residual; FlashREINFORCE’s batch advantage defines a different residual. I.3 Centering under the Behavior Distribution Letνbe the response distribution of a specified sampler version, possibly after outcome-dependent selection as in Appendix H. Unweighted profiling underνgives an off-policy score identity, and Equation (H.9) gives its weighted counterpart. At a fixed prompt and sampler version, write h θ = R− β log(π θ /π sampler θ t−k ), c θ,ν = E ν h θ , L ctr,ν = 1 2β Var ν (h θ ),g θ =∇ θ logπ θ (Y ). (I.5) Its gradient can be written in either form ∇ θ L ctr,ν =−E ν [(h θ − c θ,ν )g θ ] =−E ν [h θ (g θ − E ν g θ )], (I.6) both obtained by differentiating the variance and using the zero mean of the centered residual. The second form connects response-level regression to score centering (Marek and Ryabinin, 2026, Equa- tion (4)) under the actual replay distribution, and a prompt-dependent reward offset cancels from it. The centering mean must match the sampling measure:c θ,ν is generally notE ν R +β KL(π sampler θ t−k ∥π θ ) , since that expression requires sampling fromπ sampler θ t−k . Neither form is in general an unbiased on-policy return gradient. 45 I.4 Single Rollouts and Signed Feedback With a fixed exact response normalizerc ∗ t,k (x) , each completed response gives an SKLPO regression gradient. For two reward levels with 0 < ρ π sampler θ t−k (x) < 1, the initial residuals at π θ = π sampler θ t−k are d + = c ∗ t,k − r + < 0, d − = c ∗ t,k − r − > 0.(I.7) Gradient descent therefore pushes up the likelihood of successes and down that of failures, without a token-value critic, and the log-ratio term moderates this feedback as the policy approaches its target. At the exact response Gibbs target, the success odds satisfy ρ ∗ t,k 1− ρ ∗ t,k = ρ π sampler θ t−k 1− ρ π sampler θ t−k e (r + −r − )/β .(I.8) The signs hold at the matched-policy starting point of one component and can fail for a stale response at the current trainer; likewise, the odds identity describes the target, which a finite update with a shared trainer need not reach. An independently estimated prompt success rate, additional data from the same prompt and version, or a learned prompt normalizer can approximatec ∗ t,k , with some estimation error. A reward mean across prompts does not provide the required soft normalizer. Condition on a prompt-only estimate b c = c ∗ t,k + ε(x), obtained independently of the response being scored and held fixed during policy updates. Writingνfor the response distribution at this sampler version, the conditional gradient error is ∇ θ L ν,bc −∇ θ L ν,c ∗ t,k = ε(x)E ν [g θ | x],(I.9) which is nonzero off-policy in general. With outer weight1/T, the error becomesε(x)E ν [g θ /T | x], which can be nonzero even atν = π θ , sinceE π θ [g θ /T ] =∇ θ E π θ [1/T ] . Empirical centering of a single response gives zero loss. SKLPO can therefore train from asynchronous single rollouts only when a normalization estimator is supplied. FlashREINFORCE’s batch reward baseline serves a different objective. History coverage and trajectory length. Removing token importance sampling leaves the distribu- tion of stored histories unchanged. Their sampling frequencies can still affect a restricted model’s best fit, and hard filters can remove the coverage that Proposition A.6 requires. A fixed positive outer weight such asa(x,y) = 1/|y|preserves each realizable component target for nonempty responses, while changing the gradients and possibly the best fit across versions. Averaging the log-ratio inside the residual changes the target equation itself. Response-level feedback from a terminal failure also leaves the responsible intermediate tool call unidentified. I.5 Feasible Updates in Interactive Environments When tools are deterministic and initial conditions are fixed, a full-history action policy can realize the response Gibbs target on the reference trajectory support. Tool outputs enter the conditioning history without contributing likelihood factors. Stochastic transition factors that are common to both policies also cancel from a trajectory likelihood ratio, but this cancellation does not ensure feasibility, because the unrestricted Gibbs tilt can change observation probabilities that the policy does not control. 46 A statewise regression gives a separate route for stochastic environments. At a stored history s, take the local advantageA sampler t,k (s,a)and the PMD target of Equation(4.3). For any covering behavior distribution ν(·| s), profiling min π,c E a∼ν(·|s) β log π(a| s) π sampler θ t−k (a| s) − A sampler t,k (s,a) + c(s) 2 (I.10) has the same realizable local PMD target without action importance weights, by the zero-residual argument. The update fits a feasible action distribution and allows covered off-policy state sampling. It does require continuation values for the same sampler as its denominator. A complete return collected under that sampler estimates its ordinaryQ sampler t,k , a return from another version generally does not, and neither is an optimal regularized soft-Q target. Behavior centering does not correct a mismatch of continuations across versions, and it does not extend the deterministic telescoping identity to stochastic tools. I.6 A Stochastic-Tool Counterexample Consider a single forced action followed by a fair random terminal reward in0, 1. Every feasible policy has value 1/2 and zero policy KL. The unrestricted trajectory normalizer is instead β log[(1 + e 1/β )/2] > 1/2, by strict Jensen. The tilt raises the success probability even though the only action cannot change it, whereas the regularized Bellman equation keeps the environment law and gives value 1/2. I.7 Learning Soft Bellman Targets A critic extension could fitQ ψ to one-step targets b q = V ̄ ψ (S ′ ), withV ̄ ψ (s) = β log P a π sampler θ t−k (a | s)e Q ̄ ψ (s,a)/β at nonterminal states andV ̄ ψ (s T ) = R(s T ). With ̄ ψfrozen, a transition sampled condi- tional on(s,a)gives an unbiased Bellman target regardless of the behavior policy, provided replay selection preserves the transition law. This extension needs action values or a justified estimator of the action log-sum-exp, and a separate convergence analysis under function approximation. A sampler’s terminal return estimatesQ sampler t,k and cannot in general replace this optimal regularized continuation target. J Token-Level Identities J.1 Response and Token Gradients The centered response residual multiplies every token score: ∇ θ L seq =−βE x,Y∼π sampler θ t−k " δ seq X u ∇ θ logπ θ (y u | x,y <u ) # .(J.1) Equation(4.6)gives the token PMD gradient, in which the local KL derivative cancels in population becauseE π sampler θ t−k [δ u | x,y <u ] = 0. Neither gradient has a multiplicative importance weight; the first uses one residual per response and the second one residual per token. 47 J.2 Token Reductions and Masking Equations(4.5)and(4.11)sum over tokens and then average over responses. In population, a global token mean divides by the constantE π sampler θ t−k |Y|and changes nothing. Dividing each response’s loss by its own length instead reweights trajectories and can invalidate the cancellation of cross terms, and replacing a sum of log-ratios by a length average changes the residual itself. Termination should be counted consistently, and prompt, padding, and external tool-output tokens excluded from policy likelihoods. J.3 BPO Equation (25) and Score Centering Song et al. (2026, Equation (25)) derive the per-token gradient before approximating the full KL. Writeg θ (s,a) =∇ θ logπ θ (a| s)andg i,u,θ =∇ θ logπ θ (y i,u | x,y i,<u ), and letφ(x)be a fixed prompt weight. With η = 1/β, their token contribution to the trajectory-loss gradient is ∇ θ L BPO i,u =−φ(x)βδ BPO (x,y i ) g i,u,θ +∇ θ k θ (x,y i,<u ) ≈−φ(x) R(x,y i )− V sampler t,k (x) g i,u,θ +∇ θ k θ (x,y i,<u ) . (J.2) The first line is exact; the second freezes the residual atπ θ = π sampler θ t−k . Sinceπ sampler θ t−k is fixed, the bracket is a centered score: e g π sampler θ t−k θ (s,a) := g θ (s,a) +∇ θ k θ (s) = g θ (s,a)− E a ′ ∼π sampler θ t−k (·|s) g θ (s,a ′ ),E a∼π sampler θ t−k (·|s) e g π sampler θ t−k θ (s,a) = 0. (J.3) This is the operation of Marek and Ryabinin (2026, Equation (4)), with differentiated policyπ θ and sampling distributionν = π sampler θ t−k . BPO’s linearized gradient weights the centered score by the reward, while its exact gradient keeps the current trajectory residual. The full-KL correction is additive and involves no importance ratio; the binary-KL approximation, smoothing, and clipping then change the update. For KLPO, Equation(4.6)can equivalently use e g π sampler θ t−k θ , becauseE π sampler θ t−k [δ u | x,y <u ] = 0 , and the corresponding response-level correction centers the complete-response score by its prompt-conditional mean. These cancellations hold in population; on a single trajectory the local KL derivatives do not vanish. If the actual behavior isν ̸= π sampler θ t−k , score centering uses the mean underν, which a KL correction based onπ sampler θ t−k does not supply. Even with the matching mean, centering does not change the sampling distribution into π θ or recover its importance-weighted policy gradient. J.4 The Path Normalizer Under Assumption 2.3, the token target is π + tok (a| s) = π sampler θ t−k (a| s)e A sampler t,k (s,a)/β Z s , Z s = E a∼π sampler θ t−k (·|s) e A sampler t,k (s,a)/β .(J.4) 48 Using P u A sampler t,k (x,y <u ,y u ) = R(x,y)− V sampler t,k (x), the induced response distribution is π + tok (y | x) = π sampler θ t−k (y | x) exp R(x,y)− V sampler t,k (x) β − X u logZ (x,y <u ) .(J.5) The response Gibbs update uses a singlelogZ(x)instead. The two agree only when the accumulated local normalization is constant across the supported responses to each prompt, which the Bellman telescope alone does not imply. K Detailed Comparisons with Prior Objectives Self-Play Preference Optimization. Writep = π sampler θ t−k for one frozen sampler and define its response win rate w p (x,y) = E Y ′ ∼p(·|x) [P(y ≻ Y ′ | x)]. Withη = 1/β, the exact update and regression of SPPO (Wu et al., 2024, Equations (4.2)–(4.4)) become c w (x) = β log E Y∼p(·|x) e w p (x,Y )/β , π + w (y | x) = p(y | x)e (w p (x,y)−c w (x))/β , L exact SPPO = E x,Y∼p log π θ (Y | x) p(Y | x) − w p (x,Y )− c w (x) β 2 . (K.1) SettingR = w p therefore gives the fixed-normalizer SKLPO loss up to a positive scale, with matched sampling and outer weights. This is a one-step identity: SPPO updates the opponent in self-play, whereas a verifier reward does not depend on the sampler. The practical SPPO loss (Wu et al., 2024, Equation (4.6)) replaceslogZbyη/2, equivalentlyc w by1/2, and estimatesw p from sampled opponents. For reciprocal preferences E p w p = 1/2, so Equation (3.7) gives c w (x) = 1 2 + β KL(p(·| x)∥π + w (·| x)). The half-win baseline is therefore not the exact normalizer in general. GPO instead uses preference log- odds scores, and even under Bradley–Terry preferences their expected sigmoid win rate is generally not affine in the scalar reward. SPPO therefore need not share the verifier-reward Gibbs target, and its self-play convergence guarantee does not by itself carry over to asynchronous SKLPO replay. General Preference Optimization. GPO fits log-likelihood ratios specified by an exponential policy update (Zhang et al., 2025, Equations (5.1)–(5.3)). For a frozen opponent and referenceπ sampler θ t−k , let u pref (x,y) = E Y ′ ∼π sampler θ t−k (·|x) [s(y ≻ Y ′ | x)], c ∗ pref (x) = β log E π sampler θ t−k [e u pref (x,Y )/β ],(K.2) wheresis an antisymmetric preference score. GPO estimates this expected score with sampled opponents and fits the targetπ sampler θ t−k exp[(u pref − c ∗ pref )/β] , which is response-level KLPO with signal u pref . Its practical implementation approximates the log-normalization term by zero; the exact comparison keeps it. For reward-difference scoress(y ≻ y ′ | x) = R(x,y)− R(x,y ′ ), we have u pref = R− E π sampler θ t−k Randc ∗ pref = c ∗ − E π sampler θ t−k R, sou pref − c ∗ pref = R− c ∗ . The Gibbs targets 49 then agree, and the exact-normalizer population losses agree up to a positive scale when their data distributions and outer weights match. This does not identify the practical GPO loss, with sampled opponents and a zero normalizer, with SKLPO. SKLPO uses the verifier directly without a preference opponent, while general pairwise scores can encode cyclic preferences that no scalar reward difference represents. Bellman Policy Optimization. BPO starts from a token-level PMD update withA sampler t,k (s,a)(Song et al., 2026). With η = 1/β, its exact trajectory residual is δ BPO (x,y) = R(x,y)− V sampler t,k (x) β − X u log π θ (y u | x,y <u ) π sampler θ t−k (y u | x,y <u ) + KL(π sampler θ t−k (·| x,y <u )∥π θ (·| x,y <u )) . (K.3) LetS θ (x,y) = P u KL(π sampler θ t−k (·| x,y <u )∥π θ (·| x,y <u )). The KL chain rule and Equation(A.5)give κ θ (x) = E Y∼π sampler θ t−k (·|x) [S θ (x,Y )], δ KL − δ BPO = S θ − κ θ .(K.4) The profiled response residual thus subtracts the expected sum of token KLs, while BPO subtracts the path sum, and the two agree only when the path sum is constant on the reference support. Separately, Theorem 4.3 shows that the exact BPO trajectory loss equals the token PMD regression loss in population, and Appendix J.3 connects BPO’s Equation (25) to score centering. BPO’s practical algorithm approximates the residual further and uses a smoothed complementary-probability weight, which falls outside the exact equality. Reward-weighted maximum likelihood. Another way to fit Equation(A.2)is to minimize KL(π ⋆ t,k ∥π θ ), which gives L MLE (θ) =−E π sampler θ t−k " e R/β Z logπ θ # .(K.5) This is the reward-weighted regression behind AWR (Peng et al., 2019), and the weighted maximum- likelihood step of REPS and MPO (Peters et al., 2010; Abdolmaleki et al., 2018) has the same form. The exponential weight is fixed within a round, whereas the RPG weight depends on the current policy. Log-ratio regression, weighted maximum likelihood, and direct RPG optimization share a realizable target and take different steps toward it. L Additional Identities L.1 An Unnormalized Reference Measure For this algebraic extension only, let m be a positive finite measure with mass M = P y m(y), while the optimized π θ remains normalized. Set Z = P y m(y)e R(y)/β . Then π ⋆ t,k (y) = m(y)e R(y)/β Z , J (π θ ) = β logZ + β(1− M )− β KL(π θ ∥π ⋆ t,k ).(L.1) Scalingmonly adds a constant to the objective and leaves its normalized optimizer unchanged. Without the normalization constraint, the stationary measure is e π(y) = m(y)e R(y)/β , whose mass need not be one, so it cannot directly serve as the trainable policy. 50 L.2 Coverage and Uniqueness Fix a prompt and letνshare the support ofπ sampler θ t−k . The nonnegative exact-normalizer loss vanishes atπ ⋆ t,k , and any zero-loss policy satisfiesβ log(π θ /π sampler θ t−k )−R +c ∗ = 0 on that support, which gives uniqueness. Profiling gives R− β log(π θ /π sampler θ t−k ) = a instead, and normalization fixes a = c ∗ . Full coverage is sufficient but not always necessary. With a fixed exact normalizer, the observed probabilities are fixed, and normalization determines a single omitted probability but only the total mass of two or more omitted responses. With a profiled intercept only the observed probability ratios are fixed, so even one omitted response can make the solution nonunique. Finite data can omit responses even when the population support is full. L.3 Profiling and Relative-Reward Regression For h θ = R− β log(π θ /π sampler θ t−k ), E ν [(c− h θ ) 2 ] = Var ν (h θ ) + (c− E ν h θ ) 2 ,(L.2) which gives the optimal intercept, and for independent Y,Y ′ ∼ ν, Var ν (h θ ) = 1 2 E[(h θ (Y )− h θ (Y ′ )) 2 ].(L.3) The pairwise form uses differences of scalar rewards; it does not represent general pairwise prefer- ences. M Sequence Length Weighting and Gradient Scale Writeg u =∇ θ logπ θ (y u | x,y <u ). Under length weighting, each token receives the coefficientd θ /T. At matched trainer and sampler,d θ = c ∗ − Rhas no direct length factor. If|ℓ θ,u |≤ Mand∥g u ∥≤ G, the per-response weighted gradient has norm at most(βTM +|R− c ∗ |)G, and the unweighted bound has one more factor ofT. Averaging the scores does not bound the accumulated residual: a nonvanishing mean token log-ratio can still make the weighted loss and this bound grow linearly in T . The return-gradient contribution isw θ Ttimes the weighted regression contribution, where w θ = π θ /π sampler θ t−k . When lengths vary, their population gradients generally differ even at matched policies, sinceE π θ [T −1 ∇ θ logπ θ (Y )] =∇ θ E π θ [T −1 ]need not vanish. Preserving the target therefore does not imply the first-step gradient equality of the unweighted formulation. N Implementation Details This appendix collects what is needed to run the updates of Sections 4–6 and Appendix A. Algorithm 2 gives the token-level update and Algorithm 1 the response-level one. Code for both gradient routes and all four KL corrections is available from the project page. 51 Algorithm 2 Token-Level KLPO from a Replay Queue Require: Completed responses with sampled-token sampler log-probabilities and collection versions Require: Route: token or sequence regression; KL correction: MC-KL, TopK-KL, binary, or full Require: MC-KL: M ≥ 1 for token, M ≥ 2 for sequence; TopK-KL: head size K 1: for optimizer steps t = 0, 1,... do 2:Read B responses (x i ,y i ,T i ,R i ,v i ) and the sampler records the KL correction needs. 3:Score policy tokens with π trainer θ t ; compute ℓ i,u against version v i . 4:For MC-KL, draw M fresh auxiliary tokens per prefix from version v i ; score their log-ratios. 5: for responses i = 1,...,B do 6:if token regression then 7:Set h i,u = R i − βℓ i,u ; form L i by Eq. (5.2), (D.6), (D.11), or (4.10). 8:else 9:Form L i by Eq. (5.3), (D.5), (D.10), or (4.16). 10:end if 11: end for 12:Compute G t =∇ AD θ [B −1 P i L i ] with the specified stop-gradients. 13:Apply the optimizer update with G t to obtain θ t+1 . 14:Publish checkpoints; keep the original rewards, sampler records, and collection versions. 15: end for N.1 Records, Reductions, and Reuse Each response stores its prompt, policy tokens, reward, collection version, and the sampler log- probability of every generated token. The KL correction adds the following:Mauxiliary token IDs per position with their sampler log-probabilities for MC-KL, stored as arrays of shape[B,T,M ]; the head IDs and their sampler probabilities for TopK-KL; the sampler’s next-token distributions for full KL; and nothing for binary KL. The rollout action’s log-probability is stored separately in every case, including when the action lies outside a top-Khead. Correctly logged sampler probabilities make a separate reference-model scoring pass unnecessary. They should include the sampling transforms used at collection, such as temperature and truncation, because inference and training can produce different probabilities at identical parameters. At every update, including reuse of the same batch, everything that depends on the trainer is recomputed: the log-ratios, the coefficienth i,u or residualD i,θ , and the KL correction. Sampler records, rewards, and collection versions stay fixed. The loss sums over policy tokens and averages over responses; prompt, padding, and tool-output tokens are excluded and termination is counted consistently. An outer weight 1/T i breaks the exact identities. For MC-KL, repeated token IDs are valid. Token regression subtracts the mean auxiliary log- probability. Sequence regression uses the leave-one-out coefficients of Equation(5.3)and reports the U-statistic of Equation(5.4), which can be negative and is not clipped. A conditionally unbiased gradient at each update requires fresh draws from the historical sampler (Section 5.1); the labs-molt integration consumes each set of draws once before updating the trainer. For sequence regression with TopK-KL,k K andD K θ are recomputed at every reuse, and Equa- tion(D.5)is differentiated with only the residual coefficient detached, keeping gradients through both the head and the tail terms. With binary KL,k bin , b D θ , andωare recomputed, and both b D θ andω are detached; the token route detachesh θ,u andω. No group normalization or residual linearization 52 is applied. For exact sequence regression, the full local KL is k i,θ (s) = X a∈V π sampler θ v i (a| s) h logπ sampler θ v i (a| s)− logπ trainer θ (a| s) i ,(N.1) and its derivative is kept in Equation(4.16). Token regression needs only the score mean, not the KL value. Using−ℓ i,u from the rollout action as the entire KL estimate cancels the log-ratio and removes the learning signal; it is not an MC-KL draw, a TopK-KL head, or a binary partition. N.2 Numerical Safeguards For TopK-KL, log-probabilities and tail masses are computed in float32 or higher precision. With ε = 10 −6 , write ̄q tail = max(q tail ,ε)and ̄p tail = max(p tail ,ε). Sequence regression uses the stabilized scalar k K,ε θ = X v∈H q v log q v p v + ̄q tail log ̄q tail ̄p tail (N.2) in both the residual and the differentiated bracket. Below the floor, the trainer-tail derivative is zero away from the clamp boundary, and differentiating the literal stabilized scalar keeps its own squared-residual identity, although the stabilized expression need not be a KL between normalized distributions. Token regression follows the convention of Marek and Ryabinin (2026) and uses ̄q tail / ̄p tail in the detached head coefficients. When either floor is active, this correction need not equal −∇k K,ε θ ; the correspondence of Equation(D.3)assumes inactive floors. A full-vocabulary record uses the exact sum, with no tail bucket or ratio. For binary KL, complementary log-probabilities are computed with stablelog1mexparithmetic in float32 or higher precision, rather than by subtracting a rounded probability from one. Any additional smoothing or clipping defines a further approximation. N.3 The SKLPO Update The SKLPO trainer consumes completed responses without a collection barrier. In the supplied- normalizer branch of Algorithm 1, b c v i estimatesc ∗ t,k i (t) for the response’s collection policy and stays fixed across trainer updates; group means and learned intercepts instead track the current trainer. Stored denominators and lengths stay fixed in every branch. Both the log-ratio sum andT i count only generated policy tokens with a consistent termination convention. For unfiltered responses to one prompt from one sampler, Equation(B.2)estimates the normalizer without length weights. Selection or pooling across versions needs a separate justification, because a pooled estimate generally fails to recover each component’s normalizer. Group centering recomputes weighted means within prompt and version groups after every trainer update; each group supplies an intercept, and the loss averages over allBresponses rather than over groups. With one response per prompt, use an independently supplied normalizer or the shared learned intercept of Appendix H.7, never a singleton group mean. Removing importance weights does not restore support lost through truncation or admission filters. 53