Paper deep dive
How RL Reshapes LLM Reasoning: Transferability, Coverage, and Scaling Laws
Ziheng Cheng, Yixiao Huang, Hanlin Zhu, Somayeh Sojoudi
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 10/6/2026, 10:51:05 AM
Summary
This paper investigates how Reinforcement Learning (RL) reshapes Large Language Model (LLM) reasoning, addressing whether RL expands reasoning boundaries or merely reweights existing strategies. Using Qwen and Gemma models, the authors demonstrate that RL training on mathematics improves in-domain performance but causes heterogeneous cross-domain transfer and forgetting. They propose a two-stage autoregressive policy model separating strategy selection from execution, proving that RL's implicit bias reshapes strategy preferences. This mechanism explains divergent coverage shifts (Pass@N) and provides theoretical justification for log-sigmoid and log-linear scaling laws in RL compute.
Entities (10)
Relation Signals (7)
Reinforcement Learning → improves → in-domain math performance
confidence 95% · RL consistently improves in-domain math performance, but out-of-domain (OOD) behavior diverges significantly.
Two-stage autoregressive policy model → separates → Strategy Selection
confidence 95% · motivating a two-stage autoregressive policy model that separates strategy selection from problem-specific execution.
Reinforcement Learning → reshapes → strategy preferences
confidence 93% · Within this framework, we prove how RL’s implicit bias reshapes strategy preferences, allowing gains on some tasks while suppressing strategies required by others.
Reinforcement Learning → causes → cross-domain forgetting
confidence 92% · RL remains susceptible to forgetting, with the direction and magnitude of OOD performance changes varying substantially across settings
Supervised Fine-tuning → isproneto → Catastrophic forgetting
confidence 90% · SFT is therefore more prone to catastrophic forgetting, as observed in Figure 1.
Reinforcement Learning → follows → log-sigmoid scaling laws
confidence 85% · We further provide theoretical justifications for log-sigmoid and log-linear scaling laws in RL compute
Reinforcement Learning → follows → log-linear scaling laws
confidence 85% · We further provide theoretical justifications for log-sigmoid and log-linear scaling laws in RL compute
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Recent studies on reinforcement learning (RL) report seemingly conflicting evidence about large language model (LLM) reasoning. Training on mathematics can improve performance in other domains, yet gains in Pass@1 can coincide with lower Pass@$N$ than the base model. This raises a fundamental question: does RL expand an LLM's reasoning boundary, or merely reweight its existing reasoning space? We revisit these phenomena across Qwen and Gemma model families, showing both cross-domain gains and forgetting, while coverage at large sampling budgets increases on some tasks and decreases on others. Detailed analysis of solution traces before and after RL indicates a shift in the reasoning strategies the model employs, motivating a two-stage autoregressive policy model that separates \emph{strategy selection} from problem-specific execution. Within this framework, we prove how RL's implicit bias reshapes strategy preferences, allowing gains on some tasks while suppressing strategies required by others. This mechanism can also broaden or narrow coverage at a given sampling budget even without expanding strategy support. We further provide theoretical justifications for log-sigmoid and log-linear scaling laws in RL compute, and evaluate their predictive power. Together, these results connect changes in strategy selection to cross-domain transfer, reasoning coverage, and compute scaling.
Tags
Links
- Source: https://arxiv.org/abs/2610.04158v1
- Canonical: https://arxiv.org/abs/2610.04158v1
Trouble viewing inline? Open PDF directly →
Full Text
131,255 characters extracted from source content.
Expand or collapse full text
How RL Reshapes LLM Reasoning: Transferability, Coverage, and Scaling Laws Ziheng Cheng †thanks: Equal contribution. Yixiao Huang11footnotemark: 1 Hanlin Zhu Somayeh Sojoudi Affiliation: University of California, Berkeley Affiliation: ziheng_cheng,yixiaoh,hanlinzhu,sojoudi@berkeley.edu Abstract Recent studies on reinforcement learning (RL) report seemingly conflicting evidence about large language model (LLM) reasoning. Training on mathematics can improve performance in other domains, yet gains in Pass@1 can coincide with lower Pass@N N than the base model. This raises a fundamental question: does RL expand an LLM’s reasoning boundary, or merely reweight its existing reasoning space? We revisit these phenomena across Qwen and Gemma model families, showing both cross-domain gains and forgetting, while coverage at large sampling budgets increases on some tasks and decreases on others. Detailed analysis of solution traces before and after RL indicates a shift in the reasoning strategies the model employs, motivating a two-stage autoregressive policy model that separates strategy selection from problem-specific execution. Within this framework, we prove how RL’s implicit bias reshapes strategy preferences, allowing gains on some tasks while suppressing strategies required by others. This mechanism can also broaden or narrow coverage at a given sampling budget even without expanding strategy support. We further provide theoretical justifications for log-sigmoid and log-linear scaling laws in RL compute, and evaluate their predictive power. Together, these results connect changes in strategy selection to cross-domain transfer, reasoning coverage, and compute scaling. 1 Introduction Reinforcement learning (RL) has improved the performance of large language models (LLMs) on reasoning tasks (shao2024deepseekmath; deepseekai2025r1; yu2025dapo). RL updates a model using reward feedback on sampled responses, rather than training it to imitate demonstrations. Its success on mathematics and coding raises a broader question about how RL reshapes LLM reasoning. What accounts for these gains remains debated. RL may teach a model new ways to reason, or make it more likely to use reasoning it can already produce. For example, RL training on mathematical problems can improve performance in other domains (huan2025transferability), but such gains are compatible with either interpretation. Sampling provides another perspective. Gains in one-shot accuracy (Pass@1) can accompany lower Pass@N N, the fraction of problems solved within N N sampled attempts (yue2504does), while prolonged training has been reported to broaden coverage (liu2025prorl). These findings motivate the central question. Does RL expand an LLM’s reasoning boundary, or merely reweight its existing reasoning space? Our experiments show that training can improve performance in its own domain while reducing performance on other tasks or lowering Pass@1024 (Figures 1 and 2). Comparing Qwen trajectories before and after RL indicates a shift in the model’s preferences among reasoning strategies (Section 3.1). This motivates a two-stage policy model that separates strategy selection from problem-specific execution. Within this framework, we characterize how the implicit bias of RL reshapes strategy preferences, improving performance on some tasks while suppressing strategies required by others. We also contrast this mechanism with supervised fine-tuning (SFT). Our focus on how strategy preferences generalize across tasks complements the analyses of training-prompt sharpening and coverage (chen2026reshaping; chen2025coverage). Crucially, this reweighting mechanism explains coverage shifts without introducing new strategies. Pass@N N at large N N is especially sensitive to problems with low success probabilities. Amplifying or suppressing rare valid strategies can broaden or narrow benchmark coverage, even as Pass@1 improves. Recent studies also examine how performance scales with RL compute (khatri2025scalerl; meta2026musespark). Our framework connects strategy selection to scaling laws, showing that selection dynamics admit log-sigmoid and log-linear approximations. Our contributions are summarized as follows: • We document task-specific RL transfer and forgetting alongside divergent coverage shifts, with outcomes dictated by the training data, evaluation benchmark, and base model (Section 2). • We analyze Qwen solution trajectories and find changes in which strategies are sampled for a question, motivating a separation of strategy selection from execution (Sections 3.1 and 3.2). • Theoretically, we characterize RL’s implicit bias in strategy selection and derive conditions for transfer, forgetting, and expansion or shrinkage of coverage at a given sampling budget (Sections 4 and 5). • We derive local approximation bounds for log-sigmoid and log-linear scaling laws and evaluate how well the fitted curves predict performance later in training, showing that predictive reliability varies with the training mixture and evaluation domain (Section 6). 2 Empirical Evidence on Transfer and Reasoning Coverage The extent to which RL transfers beyond its training domain and expands the reasoning boundary remains actively debated. In this section, we revisit these phenomena across various model families and task domains to show that these seemingly contradictory findings can be reconciled. 2.1 Experimental setup Models and datasets. Our experiments use Qwen3-14B-Base (yang2025qwen3) and Gemma-3-12B-IT (gemmateam2025gemma3) as base models. We train both models on 48K mathematics problems from UniReason (huan2025transferability) (Math RL). For Qwen, we also train a variant on this dataset augmented with 5K curated coding problems from Eurus-2-RL (cui2025process) (Math-Code RL). For comparison, we also train both models with SFT using 40K mathematics demonstrations obtained by rejection sampling on Qwen3-32B responses (huan2025transferability). Training setup. For all RL experiments, we train with GRPO (shao2024deepseekmath) using verl (sheng2024hybridflow) for 200 steps at a learning rate of 10−6 10^-6. Each step samples 512 prompts and generates 16 responses per prompt (up to 16,384 tokens), followed by one epoch of actor updates with a minibatch size of 128. For SFT, we train for 1.5 epochs using a batch size of 512 and a learning rate of 5×10−5 5× 10^-5. Further training details are given in Appendix D. Evaluation protocol. We evaluate transfer using benchmarks of mathematics, coding, knowledge and planning, and general capabilities. Our default inference settings use temperature 0.6, top-p p 0.95, and a maximum generation length of 32,768 tokens, though specific benchmarks retain their original protocols such as greedy decoding. Section D.2 gives the benchmark list and complete task-specific sampling and scoring details. 2.2 Cross-domain transfer and forgetting Figure 1: Transfer and forgetting depend on the benchmark, training data, and base model. Each column reports changes in performance from the base model after training. RL consistently improves in-domain performance, but transfer behavior beyond the mathematics and coding domains varies significantly. In contrast, SFT exhibits catastrophic forgetting across a broad range of domains. Figure 1 illustrates the transfer and forgetting results. While RL consistently improves in-domain math performance, out-of-domain (OOD) behavior diverges significantly. Incorporating coding data into training (Qwen3-Math-Code-RL) improves coding tasks but fails to yield universal transfer. Moreover, despite identical training data, Qwen3-Math-RL and Gemma-3-Math-RL shift in opposite directions on many OOD tasks. SFT results exhibit similar model dependence: Qwen3-SFT shows selective transfer on knowledge and planning despite broad forgetting elsewhere, whereas Gemma-3-SFT suffers universal OOD degradation. Together, these results indicate that training-domain success does not reliably predict OOD outcomes. Instead, transferability emerges from the complex interaction among training data, evaluation benchmark, and base model. Complete scores are reported in Table 1. Although prior work suggests that RL can generalize better to OOD tasks than SFT (huan2025transferability; chu2025sft), our findings highlight that RL remains susceptible to forgetting, with the direction and magnitude of OOD performance changes varying substantially across settings (chen2026retaining; shenfeld2026razor). 2.3 Expansion and shrinkage of finite-budget coverage Figure 2: RL can either expand or contract reasoning coverage at large sampling budgets. Higher Pass@1 can accompany either lower or higher Pass@1024 relative to the base model. The direction of the coverage change varies across training mixtures, benchmarks, and base models. Task-level gains and losses show how training changes performance across domains. To examine how it changes the range of problems solved within each task, we compare coverage at different sampling budgets using Pass@N N (Figure 2). For a fixed prompt, Pass@N N increases with Pass@1; however, benchmark-level comparisons do not necessarily preserve the Pass@1 ordering at large N N. Reduced coverage at large sampling budgets. The most prevalent pattern we observe is higher Pass@1 alongside lower Pass@1024 relative to the base model, especially in mathematics. This agrees with the coverage contrast reported by yue2504does. Broader coverage when base success is rare. On the Reasoning Gym task BoxNet (stojanovski2026reasoning), where the base model’s success is exceptionally rare, Qwen3-Math-RL raises Pass@1 from 3.8% to 9.9% and Pass@1024 from 34.0% to 51.0%. Gemma-3-Math-RL also raises Pass@1024 coverage, although its Pass@1 falls. Qwen’s joint gains show that higher one-shot accuracy can accompany higher Pass@1024, consistent with liu2025prorl; wen2026reinforcement. Our observations are broadly consistent with prior work (liu2025prorl): shifts in Pass@N N are largely dictated by the benchmark distribution and the base model’s initial ability, with substantial gains generally emerging where initial coverage is low. We further find that the training data itself can alter this regime: Qwen3-Math-RL and Qwen3-Math-Code-RL move CodeContests coverage in opposite directions. Consequently, whether RL expands or shrinks finite-budget coverage is context-dependent, rather than a universal RL effect or a trend that can be inferred from Pass@1. We examine the solution trajectories underlying these contrasts in Section 3.1. 3 From Empirical Observations to Reasoning Strategy Selection To understand the performance and coverage changes in Sections 2.2 and 2.3, we compare solution trajectories before and after RL. The comparison indicates a shift in the strategies used to solve a question. We describe these strategies as reasoning routes and examine how their use differs between the base and RL models. The findings motivate a model that separates strategy selection from conditional answer generation (Section 3.2). 3.1 Reasoning-route analysis 3.1.1 Analysis setup We compare Qwen3-14B-Base with Qwen3-Math-RL on 240 questions from six benchmarks, with 16 responses per model and question. A reasoning route is the strategy used to reach a response’s final conclusion, when identifiable. Route identity reflects the problem representation and key mechanism, not notation, arithmetic, or final answers. For example, analytic reduction and exhaustive search are different routes. For each question, we combine both models’ strategy descriptions extracted by an LLM from their reasoning traces and group them by route. We average the number of distinct routes over equally sized samples of identifiable responses from both models, using the smaller number of identifiable responses available for each question, and then average equally over questions. Separately, we check which RL routes appear among the question’s original 16 base responses. Appendix E gives classification and statistical details. 3.1.2 Empirical findings Figure 3: RL uses fewer routes, but also routes absent from base samples. (a) Mean numbers of distinct routes, using equal numbers of identifiable responses from both models for each question. (b) Percentage of RL responses whose route is absent from the same question’s 16 base responses. RL concentrates on fewer routes per question. Figure 3(a) shows lower mean route counts for Qwen3-Math-RL on all six benchmarks, with the largest decreases on coding benchmarks. Across questions, the mean falls from 3.24 for the base model to 2.66 for RL, with a paired difference (RL minus base) of −0.58 -0.58 (95% confidence interval [−0.78,−0.38] [-0.78,-0.38]). For example, a LiveCodeBench question asks for the minimum absolute difference between array elements at least x x indices apart. Its 16 base responses use three routes, including comparison of neighboring values found by binary search, which all 16 RL responses use. RL also uses routes absent from the base samples. Of 3,840 Qwen3-Math-RL responses, 487 (12.7%) follow a route absent from the base responses to the same question (Figure 3(b)). For example, on a LiveCodeBench question finding pattern occurrences within distance k k of another pattern’s occurrence, all 16 base responses compare occurrence pairs directly, whereas 11 RL responses use binary search over occurrence lists. 3.2 A two-stage autoregressive model of chain-of-thought reasoning For analysis, we assume a shared library of reasoning strategies and represent a chain-of-thought trajectory as a strategy z z followed by an answer y y. Both z z and y y may correspond to sequences of tokens rather than single tokens, yielding an autoregressive factorization at the block level. This abstraction separates strategy selection from execution, while the shared library provides a common space for studying transfer, coverage, and compute scaling in Sections 4, 5 and 6. Let [M]=1,…,M [M]=\1,…,M\ be a finite library of reasoning strategies and let be a finite answer space. For any prompt x∈ x , we consider a two-stage policy that first generates a reasoning strategy z∈[M] z∈[M] and then an answer y∈ y conditional on z z. The joint policy is πθ,w(z,y|x)=πθ(z|x)πw(y|x,z), _θ,w(z,y|x)= _θ(z|x) _w(y|x,z), (3.1) where both stages are linear softmax policies, πθ(z|x)=exp(θz⊤ϕ(x))∑k=1Mexp(θk⊤ϕ(x)),πw(y|x,z)=exp(wy⊤ϕ(x,z))∑v∈exp(wv⊤ϕ(x,z)). _θ(z|x)= ( _z φ(x)) _k=1^M ( _k φ(x)), _w(y|x,z)= (w_y φ(x,z)) _v (w_v φ(x,z)). (3.2) Here θz,wy∈ℝd _z,w_y ^d are trainable parameters for strategy selection and answer generation, respectively. The feature maps ϕ(x) φ(x) and ϕ(x,z) φ(x,z) are frozen. For a given prompt x x, let ℳ(x)⊆[M] (x) [M] be its non-empty set of valid reasoning strategies and let (x)⊆ (x) be its non-empty correct-answer set. We define the conditional and end-to-end scores as Rw(x,z)=∑y∈(x)πw(y|x,z),Rθ,w(x)=∑z∈ℳ(x)πθ(z|x)Rw(x,z).R_w(x,z)= _y (x) _w(y|x,z), R_θ,w(x)= _z (x) _θ(z|x)R_w(x,z). (3.3) 3.3 Optimization dynamics Let x1,…,xn \x_1,…,x_n\ be the finite training set. For β>0 β>0, define the following KL-regularized training objectives: Jstrategy(θ,w)=1n∑i=1n[Rθ,w(xi)−βKL(πθ(⋅|xi)∥πθ0(⋅|xi))],J_strategy(θ,w)= 1n _i=1^n [R_θ,w(x_i)- \! ( _θ(·|x_i)\, \|\, _ _0(·|x_i) ) ], (3.4) Jans(θ,w)=1n∑i=1n[Rθ,w(xi)−βπθ(⋅|xi)[z∈ℳ(xi)KL(πw(⋅|xi,z)∥πw0(⋅|xi,z))]].J_ans(θ,w)= 1n _i=1^n [R_θ,w(x_i)- _ _θ(·|x_i) [ 1_z (x_i)KL\! ( _w(·|x_i,z)\,\|\, _w_0(·|x_i,z) ) ] ]. (3.5) Motivated by the small learning rates used in prominent RL pipelines (yu2025dapo; deepseekai2025r1), we analyze the policy gradient flow given by θ˙t=∇θJstrategy(θt,wt),w˙t=∇wJans(θt,wt),(θt,wt)|t=0=(θ0,w0). θ_t= _θJ_strategy( _t,w_t), w_t= _wJ_ans( _t,w_t), ( _t,w_t) |_t=0=( _0,w_0). (3.6) Along the RL dynamics, define Rt(x,z):=Rwt(x,z),Rt(x):=Rθt,wt(x) R_t(x,z):=R_w_t(x,z),R_t(x):=R_ _t,w_t(x). Throughout our theoretical analysis, we consider a vanishing but strictly positive KL coefficient, β↓0 β 0. Without KL regularization, the policy collapses to the valid strategies with the largest initial probability. 4 Reasoning Transfer and Forgetting We now use the two-stage model to analyze how RL-induced shifts in strategy selection transfer across tasks and influence target performance. Assumption 1 (Realizability on training data). The training-prompt features are orthogonal: ⟨ϕ(xi),ϕ(xj)⟩=0,∀i,j∈[n],i≠j. φ(x_i),φ(x_j) =0, ∀~i,j∈[n],\ i≠ j. (4.1) Moreover, for every y∈ y , there exists uy u_y such that ϕ(xi,z)⊤uy=y∈(xi)−|(xi)|/||,∀i∈[n],z∈ℳ(xi).φ(x_i,z) u_y= 1_y (x_i)-|A(x_i)|/|Y|, ∀~i∈[n], z (x_i). (4.2) This is an overparameterized setting. We assume the feature orthogonality in Eq. 4.1 for clarity; the analysis extends to linearly independent features. Eq. 4.2 ensures linear separation of the training data within the answer space. 4.1 RL transferability from implicit bias For a target prompt x′ x , define its feature overlap with each training prompt by c(x′,xi)=⟨ϕ(x′),ϕ(xi)⟩/‖ϕ(xi)‖2.c(x ,x_i)= φ(x ),φ(x_i) /\|φ(x_i)\|^2. (4.3) Define the strategies with largest aggregate training support by (x′)=argmaxz∈[M]∑i=1nc(x′,xi)z∈ℳ(xi).S(x )= *arg\,max_z∈[M] _i=1^nc(x ,x_i) 1_z (x_i). (4.4) Let π∞β _∞^β denote the convergent policy at fixed β β and define the vanishing-KL limit πRL:=limβ↓0π∞β _ RL:= _β 0 _∞^β. Theorem 4.1 (Transferability of RL). Under Assumption 1, for any finite t t and z∈[M] z∈[M], πθt(z|x′)∝πθ0(z|x′)∏i=1n[πθt(z|xi)πθ0(z|xi)]c(x′,xi). _ _t(z|x ) _ _0(z|x ) _i=1^n [ _ _t(z|x_i) _ _0(z|x_i) ]^c(x ,x_i). (4.5) In the vanishing KL limit, the RL strategy policy is the base policy conditioned on (x′) (x ), and the probability of selecting a valid strategy is πRL(ℳ(x′)|x′)=π0((x′)∩ℳ(x′)|x′)π0((x′)|x′). _ RL(M(x )|x )= _0(S(x ) (x )|x ) _0(S(x )|x ). (4.6) Assumption 2 (Target answer-feature orthogonality). The answer features of x′ x are orthogonal to source-valid answer features: ⟨ϕ(x′,z′),ϕ(xi,z)⟩=0,∀z∈ℳ(xi),z′∈[M]. φ(x ,z ),φ(x_i,z) =0,∀\ z (x_i),z ∈[M]. Corollary 4.2. Under Assumptions 1 and 2, in the vanishing-KL limit the reward on the target prompt is RπRL(x′)=∑z∈(x′)∩ℳ(x′)π0(z|x′)R0(x′,z)π0((x′)|x′).R_ _ RL(x )= _z (x ) (x ) _0(z|x )R_0(x ,z) _0(S(x )|x ). (4.7) Detailed proofs are deferred to Appendix A. Theorem 4.1 shows how changes in strategy preferences on training prompts transfer to target prompts through feature overlap. Under answer-feature orthogonality, Corollary 4.2 translates this strategy selection into end-to-end reward. Our results show that the inductive bias of RL preserves the base model’s preference among the surviving strategies (shenfeld2026razor). More crucially, this preservation does not determine target performance. Feature overlap governs which strategy preferences transfer, but cannot alone predict whether the target task benefits. If (x′)∩ℳ(x′)=∅ (x ) (x )= , all valid strategies are suppressed, thereby inducing forgetting. Therefore, RL transferability is jointly determined by the reasoning strategies favored in the training data, the valid strategies required by the target task, and the base model’s bias. The heterogeneous shifts in Figure 1 are compatible with these selective effects and match the qualitative predictions. Further prompt-level analyses are given in Section F.1. 4.2 When does SFT exhibit catastrophic forgetting To compare with RL, we analyze SFT under the same theoretical setup. For each training prompt xi x_i, we sample an expert demonstration zE,i∼πE(⋅|xi),yE,i∼πE(⋅|xi,zE,i) z_E,i _E(·|x_i),y_E,i _E(·|x_i,z_E,i), conditioning on zE,i∈ℳ(xi) z_E,i (x_i) and yE,i∈(xi) y_E,i (x_i). The training objective is JSFT(θ,w)=1n∑i=1n[logπθ(zE,i|xi)+logπw(yE,i|xi,zE,i)].J_ SFT(θ,w)= 1n _i=1^n [ _θ(z_E,i|x_i)+ _w(y_E,i|x_i,z_E,i) ]. (4.8) Consider the gradient flow from (θ0,w0) ( _0,w_0) on Eq. 4.8 and let πSFT _ SFT be the convergent policy. As in the RL analysis, define the strategies with largest support from the realized demonstrations: E(x′)=argmaxz∈[M]∑i=1nc(x′,xi)z=zE,i.S_E(x )= *arg\,max_z∈[M] _i=1^nc(x ,x_i) 1_z=z_E,i. (4.9) Theorem 4.3 (Transferability of SFT). Under Assumption 1, for any realized expert traces, the convergent SFT strategy policy satisfies πSFT(z|x′)>0⟺z∈E(x′). _ SFT(z|x )>0 z _E(x ). (4.10) The probability of valid strategies tends to one if E(x′)⊆ℳ(x′) _E(x ) (x ) and to zero if E(x′)∩ℳ(x′)=∅ _E(x ) (x )= . The definition of supporting strategies in Eqs. 4.4 and 4.9 exposes a fundamental distinction: RL amplifies all source-valid strategies, whereas SFT reinforces only the demonstrated strategies. Consequently, the RL policy broadly preserves the base model’s prior, whereas SFT behavior depends heavily on the characteristics of the teacher policy. SFT is therefore more prone to catastrophic forgetting, as observed in Figure 1. However, this vulnerability is not absolute; Qwen3-SFT, for instance, outperforms the RL models on the OOD benchmark GPQA-Diamond. More broadly, the transferability of SFT is highly sensitive to compatibility between the student and teacher policies. Since the SFT demonstrations originate from Qwen3-32B, they naturally align better with Qwen3-14B-Base than with Gemma-3-12B-IT, accounting for the substantially broader forgetting of Gemma-3-SFT in Figure 1. Further examples and analysis are provided in Section F.1. 5 Expansion and Shrinkage of Finite-Budget Coverage We now examine how these prompt-level shifts in strategy preference aggregate across an evaluation benchmark ℬ to alter overall coverage. Define the Pass@N N metric as Pass@N(π)=x′∼ℬ[1−(1−Rπ(x′))N].Pass@N(π)=E_x [1- (1-R_π(x ) )^N ]. (5.1) To isolate strategy selection, we evaluate each strategy policy with an ideal answer decoder: a response succeeds exactly when its strategy is valid. Consequently, Theorem 4.1 gives: Rπ0(x′)=π0(ℳ(x′)|x′),RπRL(x′)=π0((x′)∩ℳ(x′)|x′)π0((x′)|x′).R_ _0(x )= _0(M(x )|x ), R_ _ RL(x )= _0(S(x ) (x )|x ) _0(S(x )|x ). (5.2) Theorem 5.1. Under Assumption 1, fix α∈(0,1] α∈(0,1] and an integer N>log(4/α) N> (4/α). (I) Strategy loss (higher base-model coverage): If Prx′∼ℬ[(x′)∩ℳ(x′)=∅]≥α,Prx′∼ℬ[π0(ℳ(x′)|x′)≤log(4/α)N]≤α4, _x [S(x ) (x )= ]≥α, _x [ _0(M(x )|x )≤ (4/α)N ]≤ α4, then Pass@N(π0)−Pass@N(πRL)>α2.Pass@N( _0)-Pass@N( _RL)> α2. (I) Rare-strategy amplification (higher RL coverage): If Prx′∼ℬ[π0(ℳ(x′)|x′)≤14N]≥α,Prx′∼ℬ[π0((x′)∩ℳ(x′)|x′)π0((x′)|x′)≤log(4/α)N]≤α4, _x [ _0(M(x )|x )≤ 14N ]≥α, _x [ _0(S(x ) (x )|x ) _0(S(x )|x )≤ (4/α)N ]≤ α4, then Pass@N(πRL)−Pass@N(π0)>α4.Pass@N( _RL)-Pass@N( _0)> α4. Detailed proofs are provided in Appendix B. The theorem shows that Pass@N N is governed by the low-success tail, particularly prompts whose one-shot success rate is of order 1/N 1/N (chen2025coverage), rather than by average Pass@1. Coverage decreases when RL suppresses strategies on such prompts and increases when it amplifies rare successes above this scale. Both directions arise from reweighting an existing strategy library, so broader sampled coverage need not imply new strategies. Consistent with our transferability results, shifts in finite-budget coverage depend jointly on the training data, benchmark distribution and base model. Figure 2 illustrates this interaction empirically: Qwen3-Math-RL expands coverage on BoxNet but shrinks it on AIME 2024 despite Pass@1 gains. Altering the training mixture with coding data completely inverts Qwen’s performance on CodeContests, while Math RL yields opposite AIME 2025 coverage shifts for Qwen and Gemma. More prompt-level analysis and evaluation caveats are in Section F.2. 6 Scaling Laws of RL Compute Beyond transfer and reasoning coverage, recent studies ask how performance scales with RL compute. khatri2025scalerl introduce a log-sigmoid law: R(C)=R0+A−R01+(Cmid/C)B.R(C)=R_0+ A-R_01+(C_ mid/C)^B. (6.1) However, the sigmoidal plateau typically emerges only during a late saturation stage, far beyond the reach of most compute-constrained RL training. Alternatively, several works (meta2026musespark; tan2026scaling; shen2026understanding) apply a first-order approximation to derive a log-linear law: R(C)=R0+Blinlog(C/Cref).R(C)=R_0+B_ lin (C/C_ ref). (6.2) We examine these empirical laws and provide a rigorous theoretical account of RL scaling behavior based on strategy-selection dynamics. 6.1 Empirical scaling laws in RL Figure 4: Scaling behavior of RL compute. Both laws capture the fitting regime effectively with >0.99 R^2>0.99 on average. Extrapolative predictions are highly reliable up to 2× × compute in the math domain and average scores, yet exhibit greater variation on coding and planning tasks due to noisy RL training dynamics. We fit Eqs. 6.1 and 6.2 to Qwen3-Math-RL and Qwen3-Math-Code-RL evaluations through step 200, using cumulative GPU-hours as compute. We extend both RL runs to step 350 to test predictions beyond the range of compute used for fitting, where the cumulative compute is nearly 2× 2× the fitting budget. Figure 4 shows the observed scores and fitted curves during fitting and extrapolation. Detailed fitting procedures, coefficient estimates, and results using other compute metrics are given in Section D.4. As shown in Figure 4, the log-sigmoid and log-linear fits are often nearly indistinguishable over the range of compute used for fitting. This is expected because the log-linear law in eq. 6.2 locally approximates the log-sigmoid law in eq. 6.1. The laws diverge only during extrapolation: the log-linear law maintains a constant rate of improvement with log compute and is therefore unbounded, whereas the log-sigmoid law bends toward a finite plateau. During the additional training, Qwen3-Math-RL follows the log-linear prediction more closely, yet Qwen3-Math-Code-RL exhibits saturation: the log-sigmoid curve better tracks late-stage performance on MATH500, AIME, and ACPBench, as well as the average scores. Meanwhile, the log-linear model increasingly overpredicts performance at higher compute budgets. The reliability of scaling-law predictions also varies by evaluation domain. In-domain mathematical benchmarks consistently follow the fitted predictions, whereas extrapolation to coding and planning benchmarks is more brittle. Individual benchmark trajectories are also less stable than their average, reflecting the noisy dynamics of practical RL training. Notably, while khatri2025scalerl; tan2026scaling also report extrapolative scaling laws, their evaluations are restricted to held-out data drawn from the training distribution. In contrast, our analysis incorporates out-of-distribution and cross-domain evaluations. Our setting therefore tests not only whether performance scales predictably with RL compute, but also whether this scaling behavior transfers to new domains. 6.2 Theoretical justification for scaling laws We now investigate why these simple laws capture RL dynamics via reasoning-strategy selection. For ease of exposition, we take n=1 n=1, i.e., there is only one training prompt denoted by x x. For an evaluation benchmark ℬ , define its reward during training as R(t):=x′∼ℬ[Rt(x′)] R(t):=E_x [R_t(x )]. Assumption 3 (Strategy-dominated regime). There exists T0 T_0 such that π0(⋅|x) _0(·|x) and πT0(⋅|x) _T_0(·|x) are uniform within ℳ(x) (x) and within [M]\ℳ(x) [M] (x). From T0 T_0 onward, the answer policy saturates and remains frozen: Rt(x,z)=R∞β(x,z)=ρβ R_t(x,z)=R_∞^β(x,z)= _β for all t≥T0,z∈ℳ(x) t≥ T_0,z (x). After T0 T_0, training reduces to the evolution of the probability of valid strategies. We use elapsed time C=t−T0 C=t-T_0 as a proxy for compute and fit the scaling laws over an interval (a,b) (a,b) of this probability. For any u∈[a,b] u∈[a,b], let Cu C_u be the first time such that πT0+Cu(ℳ(x)|x)=u _T_0+C_u(M(x)|x)=u. For any x′∼ℬ x , we assume feature overlap c(x′,x)>0 c(x ,x)>0 and positive transfer. Theorem 6.1 (Informal). Under Assumptions 1, 2 and 3, there exist R0,A,B,Cmid R_0,A,B,C_ mid such that supC∈[Ca,Cb]|R(T0+C)−R0−A−R01+(Cmid/C)B|≤((b−a)2). _C∈[C_a,C_b] |R(T_0+C)-R_0- A-R_01+(C_ mid/C)^B | ((b-a)^2). (6.3) Theorem 6.2 (Informal). Under the same assumptions, there exist R0,Blin,Cref R_0,B_ lin,C_ ref such that supC∈[Ca,Cb]|R(T0+C)−R0−Blinlog(C/Cref)|≤((b−a)2). _C∈[C_a,C_b] |R(T_0+C)-R_0-B_ lin (C/C_ ref) | ((b-a)^2). (6.4) The results show that both log-sigmoid and log-linear laws closely approximate RL dynamics over a local compute range. Formal statements and error bounds are deferred to Theorems C.6 and C.7 in the appendix. Direct computation verifies that our error bounds remain below 1% for fitting intervals spanning up to 30%, as visualized in Section C.5. Section F.3 examines the empirical implications of this local approximation and the interpretation of fitted scaling coefficients. 7 Related Work Reinforcement learning for LLM reasoning and chain of thought. CoT prompting and self-consistency establish the value of generating and sampling reasoning trajectories (wei2022chain; wang2023selfconsistency). Reinforcement learning with verifiable rewards (RLVR) trains models using automatically checked task outcomes; Tülu 3 incorporates this approach into an open post-training recipe (lambert2024tulu). shao2024deepseekmath introduces GRPO, and DeepSeek-R1 demonstrates large-scale reasoning gains from RL (deepseekai2025r1). DAPO and Open-Reasoner-Zero further develop open systems for scaling RL training on reasoning tasks (yu2025dapo; hu2025openreasonerzero). liu2025understanding examine how base-model capabilities and biases in GRPO affect training, and introduce Dr. GRPO to improve token efficiency. Transferability, forgetting, and the reasoning boundary. chu2025sft find that RL trained with outcome rewards generalizes better than SFT to unseen textual rules and visual variants in arithmetic and navigation tasks. huan2025transferability find that RL transfers more broadly and retains performance better than SFT in controlled experiments, while cheng2025crossdomain show that transfer varies with training domain and task difficulty. chen2026retaining identify on-policy data as a major factor in RL’s relative resistance to forgetting. Coverage findings also differ: yue2504does observe gains at small sampling budgets but reduced coverage at large ones, whereas liu2025prorl report expansion under prolonged RL training. wen2026reinforcement further show that evaluating reasoning correctness, rather than final answers alone, can change coverage comparisons. Our framework characterizes conditions for these contrasting outcomes within a shared model of strategy selection, distinguishing coverage at a given sampling budget from the creation or loss of underlying strategies. Scaling laws in reinforcement learning. khatri2025scalerl fit log-sigmoid curves relating compute to performance and validate extrapolation to longer RL runs. Meta’s Muse Spark reports log-linear gains in training scores with increasing RL steps (meta2026musespark). tan2026scaling study joint scaling with model size, data, and compute in mathematical reasoning, while shen2026understanding relate the returns from increasing RL compute to pretraining. These studies establish empirical scaling relationships, but do not derive the commonly used log-sigmoid and log-linear forms from RL training dynamics. Our work provides a theoretical justification for both forms by deriving them as approximations to the dynamics of reasoning strategy selection, with explicit error bounds. We also examine their predictive reliability across evaluation domains. Theoretical analyses of RL in LLMs. chen2026reshaping analyze the selection of reasoning patterns in a two-stage tabular policy with fixed conditional success rates, focusing on behavior on training prompts. Complementary analyses study how gradient gaps affect optimization stability (suk2025optimization), success amplification under GRPO (mroueh2025grpo), and pretrained response coverage (chen2025coverage). Our framework learns both strategy and answer policies and uses shared prompt features to derive how source updates reweight target strategies. This mechanism connects strategy selection to transfer and forgetting, Pass@N N coverage, and scaling laws that describe performance over a limited range of training compute. 8 Conclusion and Discussion In this paper, we investigate how RL reshapes LLM reasoning. Starting from empirical observations across model families and task domains, we analyze solution traces before and after RL, showing a shift in the distribution over reasoning strategies. We propose a two-stage policy model to formalize this phenomenon and prove the conditions under which RL can transfer or forget capabilities and expand or contract finite-budget coverage. We also use this model to give a theoretical justification for empirical scaling laws in RL compute. These results connect changes in strategy selection to cross-domain transfer, reasoning coverage, and compute scaling. Acknowledgements Y.H. and S.S. were supported by the U.S. Army Research Laboratory and the U.S. Army Research Office under Grant W911NF2010219, Office of Naval Research, and NSF. This work used Jetstream2 at Indiana University through allocation CIS240832 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, which is supported by National Science Foundation grants #2138259, #2138286, #2138307, #2137603, and #2138296. References Appendix A Proofs for Transfer and Forgetting A.1 RL dynamics on training prompts The following auxiliary result establishes the limiting policy on training prompts used to prove Theorem 4.1, along with a convergence rate used in the scaling analysis. Theorem A.1 (RL limit on training prompts). Under Assumption 1, fix β>0 β>0. RL converges on every training prompt to a policy π∞β _∞^β. For each i∈[n] i∈[n], z∈ℳ(xi) z (x_i), and y∈ y , π∞β(y|xi,z)=π0(y|xi,z)ey∈(xi)/β∑v∈π0(v|xi,z)ev∈(xi)/β, _∞^β(y|x_i,z)= _0(y|x_i,z)e 1_y (x_i)/β _v _0(v|x_i,z)e 1_v (x_i)/β, (A.1) and hence R∞β(xi,z)=R0(xi,z)R0(xi,z)+[1−R0(xi,z)]e−1/β.R_∞^β(x_i,z)= R_0(x_i,z)R_0(x_i,z)+[1-R_0(x_i,z)]e^-1/β. (A.2) The limiting strategy policy is π∞β(z|xi)∝π0(z|xi)eR∞β(xi,z)/β,z∈ℳ(xi),1,z∉ℳ(xi). _∞^β(z|x_i) _0(z|x_i) casese^R_∞^β(x_i,z)/β,&z (x_i),\\ 1,&z (x_i). cases (A.3) Moreover, there are constants Cβ,λβ>0 C_β, _β>0 such that maxi‖πθt(⋅|xi)−π∞β(⋅|xi)‖TV _i \| _ _t(·|x_i)- _∞^β(·|x_i) \|_ TV ≤Cβe−λβt, ≤ C_βe^- _βt, maxi∈[n]z∈ℳ(xi)‖πwt(⋅|xi,z)−π∞β(⋅|xi,z)‖TV _ subarrayci∈[n]\\ z (x_i) subarray \| _w_t(·|x_i,z)- _∞^β(·|x_i,z) \|_ TV ≤Cβe−λβt. ≤ C_βe^- _βt. (A.4) Proof. Assumption 1 yields parameters (θ⋆β,w⋆β) ( _ ^β,w_ ^β) whose increments from (θ0,w0) ( _0,w_0) lie in the spans of the corresponding feature vectors for the training prompts and satisfy ϕ(xi)⊤(θ⋆,zβ−θ0,z) φ(x_i) ( _ ,z^β- _0,z) =1β[z∈ℳ(xi)R∞β(xi,z)−1M∑k∈ℳ(xi)R∞β(xi,k)], = 1β [ 1_z (x_i)R_∞^β(x_i,z)- 1M _k (x_i)R_∞^β(x_i,k) ], (A.5) ϕ(xi,z)⊤(w⋆,yβ−w0,y) φ(x_i,z) (w_ ,y^β-w_0,y) =1β[y∈(xi)−|(xi)|||]. = 1β [ 1_y (x_i)- |A(x_i)||Y| ]. (A.6) The first identity holds for all i,z i,z; the second holds for all i i, z∈ℳ(xi) z (x_i), and y y. In the first identity, the indicator sets the strategy-level return to zero when z∉ℳ(xi) z (x_i). The pairwise logit differences in Eqs. A.5 and A.6 yield Eqs. A.1 and A.3. Choose the minimum-norm parameter increments in these spans. Because the right-hand sides are centered across softmax classes, the increments sum to zero across classes, matching the gradient-flow gauge. We prove convergence to these parameters within the feature spans generated by the training prompts. Set Eθ(t)=12∑z=1M‖θt,z−θ⋆,zβ‖2,Ew(t)=12∑y∈‖wt,y−w⋆,yβ‖2.E_θ(t)= 12 _z=1^M\| _t,z- _ ,z^β\|^2, E_w(t)= 12 _y \|w_t,y-w_ ,y^β\|^2. (A.7) For a distribution P P and vector a a, write VarP(a)=∑jP(j)(aj−∑kP(k)ak)2 _P(a)= _jP(j)(a_j- _kP(k)a_k)^2. Direct differentiation of Eq. 3.4 and Eq. 3.5, using the target log-odds above, yields E˙w E_w =−βn∑i=1n∑z∈ℳ(xi)πθt(z|xi)Varπwt(⋅|xi,z)(ϕ(xi,z)⊤(wt−w⋆β)), =- βn _i=1^n _z (x_i) _ _t(z|x_i)Var_ _w_t(·|x_i,z)\! (φ(x_i,z) (w_t-w_ ^β) ), (A.8) E˙θ E_θ =−βn∑i=1nVarπθt(⋅|xi)(ϕ(xi)⊤(θt−θ⋆β)) =- βn _i=1^nVar_ _ _t(·|x_i)\! (φ(x_i) ( _t- _ ^β) ) +1n∑i=1nCovπθt(⋅|xi)(ϕ(xi)⊤(θt−θ⋆β),z∈ℳ(xi)[Rt(xi,z)−R∞β(xi,z)]). + 1n _i=1^nCov_ _ _t(·|x_i)\! (φ(x_i) ( _t- _ ^β), 1_z (x_i)[R_t(x_i,z)-R_∞^β(x_i,z)] ). (A.9) The entries in each variance or covariance are indexed by the relevant strategy or answer variable. Eq. A.8 makes Ew E_w nonincreasing, so all answer probabilities on the training prompts have a common lower bound q¯>0 q>0. For fixed i i and z∈ℳ(xi) z (x_i), let ay=ϕ(xi,z)⊤(wt,y−w⋆,yβ) a_y=φ(x_i,z) (w_t,y-w_ ,y^β) and interpolate between the current and limiting answer policies via qs(y)∝πwt(y|xi,z)e−say q_s(y) _w_t(y|x_i,z)e^-sa_y, 0≤s≤1 0≤ s≤ 1. Then q1=π∞β(⋅|xi,z) q_1= _∞^β(·|x_i,z) and dsqs((xi))=−Covqs(y∈(xi),ay) ddsq_s(A(x_i))=-Cov_q_s( 1_y (x_i),a_y). Taking the q0 q_0-mean of a a as a candidate center and using qs(y)≤1≤q0(y)/q¯ q_s(y)≤ 1≤ q_0(y)/ q gives Varqs(a)≤q¯−1Varq0(a) _q_s(a)≤ q^-1Var_q_0(a). Since Varqs(y∈(xi))≤1/4 _q_s( 1_y (x_i))≤ 1/4, integration gives |Rt(xi,z)−R∞β(xi,z)|2≤14q¯Varπwt(⋅|xi,z)(a).|R_t(x_i,z)-R_∞^β(x_i,z)|^2≤ 14 qVar_ _w_t(·|x_i,z)(a). (A.10) Cauchy–Schwarz and Young’s inequality now absorb the covariance in Eq. A.9 into the two variance terms, giving constants γ,η>0 γ,η>0 such that dt(Eθ+γEw)≤−ηn∑i=1n[ ddt(E_θ+γ E_w)≤- ηn _i=1^n [ Varπθt(⋅|xi)(ϕ(xi)⊤(θt−θ⋆β)) _ _ _t(·|x_i)\! (φ(x_i) ( _t- _ ^β) ) +∑z∈ℳ(xi)πθt(z|xi)Varπwt(⋅|xi,z)(ϕ(xi,z)⊤(wt−w⋆β))]. + _z (x_i) _ _t(z|x_i)Var_ _w_t(·|x_i,z)\! (φ(x_i,z) (w_t-w_ ^β) ) ]. (A.11) The two parameter blocks are therefore bounded, which also gives a uniform positive lower bound p¯>0 p>0 for every strategy probability on a training prompt. The parameter errors remain in these feature spans and sum to zero across softmax classes. Consequently, each strategy-selection variance in Eq. A.11 is at least p¯ p times the sum of squared strategy-selection logit errors over strategies; each weighted answer variance is at least p¯q¯ p q times the sum of squared answer-logit errors over answers. After summing over training prompts and valid strategies, these variance sums bound Eθ E_θ and Ew E_w, respectively, because the prompt features are orthogonal and the answer-feature Gram operator is positive on its span. Thus the right-hand side of Eq. A.11 is at most −2λβ(Eθ+γEw) -2 _β(E_θ+γ E_w) for some λβ>0 _β>0. Gronwall’s inequality and smoothness of softmax then give global existence and Eq. A.4. Summing Eq. A.1 over (xi) (x_i) gives Eq. A.2. Uniformly over the finite training set, 1−R∞β(xi,z)=O(e−1/β)=o(β) 1-R_∞^β(x_i,z)=O(e^-1/β)=o(β). This uniform estimate is used in the target limit below. ∎ A.2 Proofs for RL transfer For fixed β>0 β>0, the target strategy policy converges to π∞β(z|x′)∝π0(z|x′)exp1β∑i:z∈ℳ(xi)c(x′,xi)R∞β(xi,z), _∞^β(z|x ) _0(z|x ) \! \ 1β _i:\,z (x_i)c(x ,x_i)R_∞^β(x_i,z) \, (A.12) Moreover, after increasing Cβ C_β in Eq. A.4 if needed, the finite-time target policy satisfies ‖πθt(⋅|x′)−π∞β(⋅|x′)‖TV≤(∑i=1n|c(x′,xi)|)Cβe−λβt. \| _ _t(·|x )- _∞^β(·|x ) \|_ TV≤ ( _i=1^n|c(x ,x_i)| )C_βe^- _βt. (A.13) If every c(x′,xi) c(x ,x_i) is zero, the target strategy policy remains at its initial value for all time. Proof of Theorem 4.1. For each strategy z z, the RL dynamics keep θt,z−θ0,z _t,z- _0,z in spanϕ(x1),…,ϕ(xn) \φ(x_1),…,φ(x_n)\. Orthogonality and Eq. 4.3 give ϕ(x′)⊤(θt,z−θ0,z)=∑i=1nc(x′,xi)ϕ(xi)⊤(θt,z−θ0,z).φ(x ) ( _t,z- _0,z)= _i=1^nc(x ,x_i)φ(x_i) ( _t,z- _0,z). (A.14) Subtracting Eq. A.14 for two strategies converts it into an identity between their log-odds changes. Exponentiating and normalizing over z z proves Eq. 4.5. Letting t→∞ t→∞ and using Eq. A.3 proves Eq. A.12. The probability lower bounds established above and Eq. A.4 bound every log-odds error on a training prompt by Cβe−λβt C_βe^- _βt after increasing Cβ C_β if needed. Applying Eq. A.14 to the finite-time and limiting parameters, summing the absolute values of the overlaps, and using Lipschitz continuity of softmax gives Eq. A.13. If every overlap vanishes, Eq. A.14 is zero for all t t, proving invariance. Finally, Eq. A.2 implies, uniformly over strategies, ∑i:z∈ℳ(xi)c(x′,xi)R∞β(xi,z)=∑i=1nc(x′,xi)z∈ℳ(xi)+o(β). _i:\,z (x_i)c(x ,x_i)R_∞^β(x_i,z)= _i=1^nc(x ,x_i) 1_z (x_i)+o(β). (A.15) Substituting this expression into Eq. A.12 shows that all mass outside (x′) (x ) vanishes and that base-policy ratios within (x′) (x ) are preserved. Thus πRL(z|x′)=π0(z|x′)z∈(x′)π0((x′)|x′). _ RL(z|x )= _0(z|x ) 1_z (x ) _0(S(x )|x ). (A.16) Summing over ℳ(x′) (x ) proves Eq. 4.6. ∎ Proof of Corollary 4.2. Every answer-parameter increment lies in the span of ϕ(xi,z) φ(x_i,z) for i∈[n] i∈[n] and z∈ℳ(xi) z (x_i). Assumption 2 therefore fixes every target answer logit. Combining this fact with Eq. A.13 shows that, for fixed β β, the reward converges at the same rate, and substitution of Eq. A.16 gives Eq. 4.7. ∎ A.3 Proof of SFT transferability Proof of Theorem 4.3. The strategy and answer gradients are bounded, so the flow exists for all time. The case M=1 M=1 is immediate. For M≥2 M≥ 2, write ai,z(t)=ϕ(xi)⊤θt,zSFT a_i,z(t)=φ(x_i) _t,z SFT. Orthogonality of the training-prompt features makes the strategy dynamics independent across prompts: a˙i,z=‖ϕ(xi)‖2n[z=zE,i−πθtSFT(z|xi)]. a_i,z= \|φ(x_i)\|^2n [ 1_z=z_E,i- _ _t SFT(z|x_i) ]. (A.17) For every k≠zE,i k≠ z_E,i, dte−ai,k=‖ϕ(xi)‖2/n∑zeai,z. ddte^-a_i,k= \|φ(x_i)\|^2/n _ze^a_i,z. (A.18) Thus e−ai,k(t)=e−ai,k(0)+ui(t) e^-a_i,k(t)=e^-a_i,k(0)+u_i(t) for a common increasing function ui(t) u_i(t) across all strategies other than zE,i z_E,i. The sum ∑zai,z(t) _za_i,z(t) is constant, and ui(t)→∞ u_i(t)→∞; otherwise all logits would stay bounded and u˙i u_i would stay bounded away from zero. Substituting the preceding expression into the conserved sum gives ui(t)M∼M‖ϕ(xi)‖2tnexp(∑zai,z(0)).u_i(t)^M M\|φ(x_i)\|^2tn \! ( _za_i,z(0) ). (A.19) Hence, for every z z, ai,z(t)−ai,z(0)=(z=zE,i−1M)logM‖ϕ(xi)‖2tn+1M∑kai,k(0)−ai,z(0)+o(1).a_i,z(t)-a_i,z(0)= ( 1_z=z_E,i- 1M ) M\|φ(x_i)\|^2tn+ 1M _ka_i,k(0)-a_i,z(0)+o(1). (A.20) Each strategy-parameter increment lies in the span of the training-prompt features, so the orthogonality of these features gives Eq. A.14 with θt _t replaced by θtSFT _t SFT. Substituting Eq. A.20 into this relation, exponentiating, normalizing, and dropping factors independent of z z yields, uniformly over the finite strategy set, πθtSFT(z|x′)∝π0(z|x′)∏i=1nπ0(z|xi)−c(x′,xi)∏i:zE,i=z(M‖ϕ(xi)‖2tn)c(x′,xi)[1+o(1)]. _ _t SFT(z|x ) _0(z|x ) _i=1^n _0(z|x_i)^-c(x ,x_i) _i:\,z_E,i=z ( M\|φ(x_i)\|^2tn )^c(x ,x_i)[1+o(1)]. (A.21) The power of t t is maximized exactly on E(x′) _E(x ). On this set, the powers of Mt/n Mt/n are equal and cancel under normalization. Removing this common factor from Eq. A.21 yields the explicit limit πSFT(z|x′)=π0(z|x′)z∈E(x′)∏i=1n[‖ϕ(xi)‖2z=zE,iπ0(z|xi)]c(x′,xi)∑k∈E(x′)π0(k|x′)∏i=1n[‖ϕ(xi)‖2k=zE,iπ0(k|xi)]c(x′,xi). _ SFT(z|x )= _0(z|x ) 1_z _E(x ) _i=1^n [ \|φ(x_i)\|^2 1_z=z_E,i _0(z|x_i) ]^c(x ,x_i) _k _E(x ) _0(k|x ) _i=1^n [ \|φ(x_i)\|^2 1_k=z_E,i _0(k|x_i) ]^c(x ,x_i). (A.22) All remaining factors are positive, so the limiting policy is supported exactly on E(x′) _E(x ), proving Eq. 4.10. The two claims about probability mass on strategies valid for the target prompt follow from this support characterization. ∎ Appendix B Proof for the Finite-Budget Coverage Theorem Proof of Theorem 5.1. With the ideal answer decoder used to evaluate strategy success, Rπ(x′)=π(ℳ(x′)|x′) R_π(x )=π(M(x )|x ), and Theorem 4.1 gives Rπ0(x′)=π0(ℳ(x′)|x′),RπRL(x′)=π0((x′)∩ℳ(x′)|x′)π0((x′)|x′).R_ _0(x )= _0(M(x )|x ), R_ _ RL(x )= _0(S(x ) (x )|x ) _0(S(x )|x ). Because the base softmax has full support, RπRL(x′)=0 R_ _ RL(x )=0 exactly when (x′)∩ℳ(x′)=∅ (x ) (x )= . For any random variable U∈[0,1] U∈[0,1] and 0≤a<b<1 0≤ a<b<1, Pr[U≤a](1−a)N≤[(1−U)N]≤Pr[U≤b]+(1−b)N. [U≤ a](1-a)^N [(1-U)^N]≤ [U≤ b]+(1-b)^N. The lower bound follows by restricting the expectation to U≤a \U≤ a\; the upper bound follows by splitting it over U≤b \U≤ b\ and its complement. The theorem’s choice of N N ensures 0<1/(4N)<log(4/α)/N<1 0<1/(4N)< (4/α)/N<1, and (1−14N)N≥34,(1−log(4/α)N)N<α4. (1- 14N )^N≥ 34, (1- (4/α)N )^N< α4. For part (I), apply the lower bound to RπRL(x′) R_ _ RL(x ) with a=0 a=0 and the upper bound to Rπ0(x′) R_ _0(x ) with b=log(4/α)/N b= (4/α)/N. This gives 1−Pass@N(πRL)≥α,1−Pass@N(π0)<α2.1-Pass@N( _ RL)≥α, 1-Pass@N( _0)< α2. The difference between these failure probabilities gives the stated advantage for the base policy. For part (I), apply the lower bound to Rπ0(x′) R_ _0(x ) with a=1/(4N) a=1/(4N) and the upper bound to RπRL(x′) R_ _ RL(x ) with b=log(4/α)/N b= (4/α)/N. This gives 1−Pass@N(π0)≥3α4,1−Pass@N(πRL)<α2.1-Pass@N( _0)≥ 3α4, 1-Pass@N( _ RL)< α2. Their difference gives the stated advantage for the RL policy. Finally, as β↓0 β 0, the finite-β β success probabilities converge to RπRL(x′) R_ _ RL(x ). By bounded convergence, the corresponding Pass@N N values converge under ℬ , so each strict bound persists for sufficiently small β>0 β>0. ∎ Appendix C Proofs for the Scaling Laws Throughout this appendix, we take n=1 n=1, let x x be the training prompt, set m:=|ℳ(x)|<M m:=|M(x)|<M, fix β>0 β>0 within each training run, and write C=t−T0≥0 C=t-T_0≥ 0. For a target prompt x′ x , write c(x′,x)=⟨ϕ(x′),ϕ(x)⟩/‖ϕ(x)‖2 c(x ,x)= φ(x ),φ(x) /\|φ(x)\|^2. Write (u)=log(u/(1−u)) logit(u)= (u/(1-u)), σ(s)=(1+e−s)−1 σ(s)=(1+e^-s)^-1, and abbreviate πx(C):=πT0+C(ℳ(x)|x),πx′(C):=πT0+C(ℳ(x)|x′). _x(C):= _T_0+C(M(x)|x), _x (C):= _T_0+C(M(x)|x ). Thus πx(0) _x(0) is the probability assigned to ℳ(x) (x) at T0 T_0, whereas π0 _0 denotes the fixed KL reference. Under Assumptions 1 and 3, there is a common ρβ _β such that RT0+C(x,z)=ρβ,C≥0,z∈ℳ(x).R_T_0+C(x,z)= _β, C≥ 0, z (x). By Eq. A.2, 1−ρβ=O(e−1/β) 1- _β=O(e^-1/β). We use πx(∞) _x(∞) and πx′(∞) _x (∞) for their fixed-β β limits. We normalize the reward for a target prompt only when R∞β(x′)>RT0(x′) R_∞^β(x )>R_T_0(x ). Throughout the vanishing-KL analysis below, p:=πx(0) p:= _x(0) and 0<p<a<b<1 0<p<a<b<1 are fixed. The term o(1) o(1) is uniform on [a,b] [a,b] as β↓0 β 0. The remainder in the target log-linear approximation is uniform over c(x′,x)>0 c(x ,x)>0 and πx′(0)∈(0,1) _x (0)∈(0,1). For 0<a<b<1 0<a<b<1, set H(u):=2u−1u(1−u)+2(u).H(u):= 2u-1u(1-u)+2 logit(u). Here H′(u)=[u(1−u)]−2 H (u)=[u(1-u)]^-2, and Δa,b(p):=logH(b)−H(p)H(a)−H(p). _a,b^(p):= H(b)-H(p)H(a)-H(p). (C.1) C.1 Scalar training-prompt dynamics Lemma C.1 (Scalar training dynamics and transfer). Under Assumptions 1 and 3, the training-strategy probability satisfies dπxdC d _xdC =‖ϕ(x)‖2(1m+1M−m)[πx(C)(1−πx(C))]2 =\|φ(x)\|^2 ( 1m+ 1M-m )[ _x(C)(1- _x(C))]^2 ×ρβ−β[(πx(C))−(π0(ℳ(x)|x))]. × \ _β-β [ logit( _x(C))- logit ( _0(M(x)|x) ) ] \. (C.2) Moreover, (πx(∞))=(π0(ℳ(x)|x))+ρβ. logit( _x(∞))= logit ( _0(M(x)|x) )+ _β. (C.3) For every target prompt and C≥0 C≥ 0, (πx′(C))−(πx′(0)) logit( _x (C))- logit( _x (0)) =c(x′,x)[(πx(C))−(πx(0))]. =c(x ,x)[ logit( _x(C))- logit( _x(0))]. (C.4) Consequently, (πx′(∞))=(πx′(0))+c(x′,x)[(πx(∞))−(πx(0))]. logit( _x (∞))= logit( _x (0))+c(x ,x) [ logit( _x(∞))- logit( _x(0)) ]. (C.5) If Assumption 2 also holds, c(x′,x)>0 c(x ,x)>0, and R∞β(x′)>RT0(x′) R_∞^β(x )>R_T_0(x ), then RT0+C(x′)−RT0(x′)R∞β(x′)−RT0(x′)=πx′(C)−πx′(0)πx′(∞)−πx′(0). R_T_0+C(x )-R_T_0(x )R_∞^β(x )-R_T_0(x )= _x (C)- _x (0) _x (∞)- _x (0). (C.6) Proof. For πz(C)=πT0+C(z|x) _z(C)= _T_0+C(z|x), define Uz(C)=z∈ℳ(x)ρβ−βlogπz(C)π0(z|x).U_z(C)= 1_z (x) _β-β _z(C) _0(z|x). Differentiating the strategy objective gives dC[θT0+C,z⊤ϕ(x)]=‖ϕ(x)‖2πz(C)[Uz(C)−∑k=1Mπk(C)Uk(C)]. ddC [ _T_0+C,z φ(x) ]=\|φ(x)\|^2 _z(C) [U_z(C)- _k=1^M _k(C)U_k(C) ]. (C.7) At C=0 C=0, the KL reference and the policy at T0 T_0 are each uniform within the valid and invalid classes. By Assumption 3, valid-strategy rewards are also symmetric, while invalid strategies have zero return. Uniqueness of the ODE solution therefore preserves uniformity within each class. Substituting πz(C)=πx(C)m(z∈ℳ(x)),πz(C)=1−πx(C)M−m(z∉ℳ(x)), _z(C)= _x(C)m (z (x)), _z(C)= 1- _x(C)M-m (z (x)), into Eq. C.7 and differentiating the softmax yields Eq. C.2. Using the common conditional reward ρβ _β in Eq. A.3 gives π∞β(z|x)∝π0(z|x)exp(z∈ℳ(x)ρβ). _∞^β(z|x) _0(z|x) ( 1_z (x) _β ). Taking the odds ratio between the valid and invalid classes proves Eq. C.3. Because n=1 n=1, every strategy-parameter update is parallel to ϕ(x) φ(x). The resulting change in a strategy logit at x′ x is therefore c(x′,x) c(x ,x) times its change at x x. Taking the odds ratio between the valid and invalid classes proves Eq. C.4. Letting C→∞ C→∞ and applying Eq. C.3 gives Eq. C.5. Finally, Assumption 2 fixes every target answer probability. Within ℳ(x) (x) and its complement, the target strategy proportions remain unchanged. Thus RT0+C(x′) R_T_0+C(x ) depends linearly on πx′(C) _x (C), up to an additive constant. Subtracting its value at C=0 C=0 and dividing by its total change at C=∞ C=∞ proves Eq. C.6. ∎ Lemma C.2 (Relating compute and reward in the strategy-dominated regime). Fix 0<p<a<b<1 0<p<a<b<1 and consider a family of runs satisfying Assumptions 1 and 3, with the reference policy and p,a,b p,a,b fixed as β↓0 β 0. For sufficiently small β β, R∞β(x)>RT0(x) R_∞^β(x)>R_T_0(x) and πx(C) _x(C) increases through [a,b] [a,b]. For u∈[a,b] u∈[a,b], let Cu C_u be the first value of C C for which πx(C)=u _x(C)=u. Uniformly for u∈[a,b] u∈[a,b], log(Cu/Ca)log(Cb/Ca) (C_u/C_a) (C_b/C_a) =log[H(u)−H(p)]/[H(a)−H(p)]Δa,b(p)+o(1), = \[H(u)-H(p)]/[H(a)-H(p)]\ _a,b^(p)+o(1), (C.8) RT0+Cu(x)−RT0(x)R∞β(x)−RT0(x) R_T_0+C_u(x)-R_T_0(x)R_∞^β(x)-R_T_0(x) =u−p1−p+o(1). = u-p1-p+o(1). (C.9) If, in addition, a target prompt x′ x satisfies Assumption 2, with c=c(x′,x)>0 c=c(x ,x)>0 and R∞β(x′)>RT0(x′) R_∞^β(x )>R_T_0(x ), then the following identity is exact: RT0+Cu(x′)−RT0(x′)R∞β(x′)−RT0(x′)=σ((πx′(0))+c[(u)−(p)])−πx′(0)πx′(∞)−πx′(0). R_T_0+C_u(x )-R_T_0(x )R_∞^β(x )-R_T_0(x )= σ\! ( logit( _x (0))+c[ logit(u)- logit(p)] )- _x (0) _x (∞)- _x (0). (C.10) For fixed c c and πx′(0)∈(0,1) _x (0)∈(0,1), the denominator on the right-hand side equals 1−πx′(0)+o(1) 1- _x (0)+o(1). Proof. Since ρβ→1 _β→ 1 and the reference strategy mass is fixed, uniformly for p≤v≤b p≤ v≤ b, ρβ−β[(v)−(π0(ℳ(x)|x))]=ρβ[1+o(1)]. _β-β [ logit(v)- logit ( _0(M(x)|x) ) ]= _β[1+o(1)]. Hence the expression in braces in Eq. C.2 is positive on [p,b] [p,b] for sufficiently small β β. On this interval, dπx/dC d _x/dC is bounded below by a positive multiple of [p(1−b)]2 [p(1-b)]^2, so πx(C) _x(C) reaches b b in finite time. Separating variables, integrating, and using H′(v)=[v(1−v)]−2 H (v)=[v(1-v)]^-2 yields, uniformly for u∈[a,b] u∈[a,b], ‖ϕ(x)‖2(1m+1M−m)ρβCu=[H(u)−H(p)][1+o(1)].\|φ(x)\|^2 ( 1m+ 1M-m ) _βC_u=[H(u)-H(p)][1+o(1)]. (C.11) In particular, log(Cb/Ca)=Δa,b(p)+o(1) (C_b/C_a)= _a,b^(p)+o(1). Since Δa,b(p)>0 _a,b^(p)>0, taking logarithms and dividing by this span proves Eq. C.8. By Eq. C.3, πx(∞)→1 _x(∞)→ 1. Since RT0+C(x)=ρβπx(C) R_T_0+C(x)= _β _x(C) exactly, RT0+Cu(x)−RT0(x)R∞β(x)−RT0(x)=u−pπx(∞)−p=u−p1−p+o(1). R_T_0+C_u(x)-R_T_0(x)R_∞^β(x)-R_T_0(x)= u-p _x(∞)-p= u-p1-p+o(1). This also shows R∞β(x)>RT0(x) R_∞^β(x)>R_T_0(x) and proves Eq. C.9. Combining the transfer identity Eq. C.4 with the normalized reward identity Eq. C.6 yields Eq. C.10. Finally, Eq. C.5 and fixed c,πx′(0) c, _x (0) imply πx′(∞)→1 _x (∞)→ 1. ∎ C.2 Log-linear approximation Lemma C.3 (Linear approximation of an inverse map). Let g∈C2([a,b]) g∈ C^2([a,b]) satisfy g′>0 g >0 and |g′/g′|≤K |g /g |≤ K. There exists α∈ℝ α such that supu∈[a,b]|u−α−b−ag(b)−g(a)[g(u)−g(a)]|≤K(b−a)216. _u∈[a,b] |u-α- b-ag(b)-g(a)[g(u)-g(a)] |≤ K(b-a)^216. (C.12) Proof. Normalize the interval and the function by F(t)=g(a+(b−a)t)−g(a)g(b)−g(a),f(t)=F′(t),d=K(b−a).F(t)= g(a+(b-a)t)-g(a)g(b)-g(a), f(t)=F (t), d=K(b-a). Then f>0 f>0, ∫01f(t)t=1 _0^1f(t)\,dt=1, and logf f is d d-Lipschitz. If d=0 d=0, then f≡1 f≡ 1 and the claim is exact. Hence assume d>0 d>0. Let f∗ f^* be the decreasing rearrangement of f f. For any two values λ1<λ2 _1< _2 in the range of logf f, continuity and the d d-Lipschitz property give |t∈[0,1]:λ1<logf(t)<λ2|≥λ2−λ1d. |\t∈[0,1]: _1< f(t)< _2\ |≥ _2- _1d. Here |⋅| |·| denotes Lebesgue measure. By continuity and the Lipschitz condition, logf f takes intermediate values over a set of at least the displayed measure. Decreasing rearrangement preserves these level-set measures, so logf∗ f^* is also d d-Lipschitz. Hence f∗(t)/e−dt f^*(t)/e^-dt is nondecreasing. Integrating this ratio on [0,s] [0,s] and [s,1] [s,1] gives ∫0sf∗(t)t≤1−e−ds1−e−d. _0^sf^*(t)\,dt≤ 1-e^-ds1-e^-d. Therefore ∫01(f(t)−1)+t _0^1(f(t)-1)_+\,dt =max0≤s≤1∫0sf∗(t)t−s = _0≤ s≤ 1 \ _0^sf^*(t)\,dt-s \ ≤max0≤s≤11−e−ds1−e−d−s≤d8. ≤ _0≤ s≤ 1 \ 1-e^-ds1-e^-d-s \≤ d8. (C.13) To verify the last inequality, a direct calculation shows that d d times the preceding maximum equals max0≤v≤1log(1−v+ved)−vd. _0≤ v≤ 1\ (1-v+ve^d)-vd\. For fixed v v, this expression and its first derivative with respect to d d vanish at d=0 d=0, while its second derivative with respect to d d is at most 1/4 1/4. Integrating this bound twice yields d2/8 d^2/8. Since F(t)−t=∫0t(f(s)−1)s F(t)-t= _0^t(f(s)-1)\,ds, its oscillation is at most the right-hand side of Eq. C.13. Returning to [a,b] [a,b], the residual in Eq. C.12 therefore has oscillation at most K(b−a)2/8 K(b-a)^2/8. Centering its range proves the result. ∎ Lemma C.4 (Log-linear law for the training prompt). Under Assumptions 1 and 3, for any εfit>0 _ fit>0 and sufficiently small β β, there exist R0,x<R∞β(x) R_0,x<R_∞^β(x) and Blin,x,Cref,x>0 B_ lin,x,C_ ref,x>0 such that, for every C∈[Ca,Cb] C∈[C_a,C_b], |RT0+C(x)−R0,x−Blin,xlog(C/Cref,x)|R∞β(x)−R0,x≤b−a1−p[b−a8maxv∈a,b|1−2v|v(1−v)+Δa,b(p)16]+εfit. |R_T_0+C(x)-R_0,x-B_ lin,x (C/C_ ref,x) |R_∞^β(x)-R_0,x≤ b-a1-p [ b-a8 _v∈\a,b\ |1-2v|v(1-v)+ _a,b^(p)16 ]+ _ fit. (C.14) Proof. Choose the fitted intercept R0,x:=RT0(x) R_0,x:=R_T_0(x). Since H′(u)=1[u(1−u)]2,|H′(u)H′(u)|=2|1−2u|u(1−u),H (u)= 1[u(1-u)]^2, | H (u)H (u) |= 2|1-2u|u(1-u), Lemma C.3 gives α∈ℝ α such that supu∈[a,b]|u−α−b−aH(b)−H(a)[H(u)−H(a)]|≤(b−a)28maxu∈a,b|1−2u|u(1−u). _u∈[a,b] |u-α- b-aH(b)-H(a)[H(u)-H(a)] |≤ (b-a)^28 _u∈\a,b\ |1-2u|u(1-u). (C.15) Let F(u)=log(Cu/Ca)/log(Cb/Ca) F(u)= (C_u/C_a)/ (C_b/C_a) and G(u)=log[H(u)−H(p)]/[H(a)−H(p)]/Δa,b(p) G(u)= \[H(u)-H(p)]/[H(a)-H(p)]\/ _a,b^(p). By Eq. C.8, F=G+o(1) F=G+o(1) uniformly. The identity H(u)−H(a)H(b)−H(a)=eΔa,b(p)G(u)−1eΔa,b(p)−1 H(u)-H(a)H(b)-H(a)= e _a,b^(p)G(u)-1e _a,b^(p)-1 and Lemma C.3, applied to eΔa,b(p)s e _a,b^(p)s on [0,1] [0,1], show that, after adding a suitable constant, F(u) F(u) approximates this normalized change in H H with error at most Δa,b(p)/16+o(1) _a,b^(p)/16+o(1). Combining this bound with Eq. C.15, adjusting the constant term, and using Eq. C.9 gives infα′∈ℝsupu∈[a,b]|RT0+Cu(x)−R0,xR∞β(x)−R0,x−α′−b−a1−pF(u)|≤b−a1−p[b−a8maxv∈a,b|1−2v|v(1−v)+Δa,b(p)16]+o(1). _α _u∈[a,b] | R_T_0+C_u(x)-R_0,xR_∞^β(x)-R_0,x-α - b-a1-pF(u) |≤ b-a1-p [ b-a8 _v∈\a,b\ |1-2v|v(1-v)+ _a,b^(p)16 ]+o(1). Choose Blin,x=[R∞β(x)−R0,x](b−a)(1−p)log(Cb/Ca)>0,B_ lin,x= [R_∞^β(x)-R_0,x](b-a)(1-p) (C_b/C_a)>0, and choose Cref,x>0 C_ ref,x>0 so that −Blin,xlogCref,x -B_ lin,x C_ ref,x equals the resulting constant term. Because πx(C) _x(C) is strictly increasing, Cu C_u covers [Ca,Cb] [C_a,C_b] as u u ranges over [a,b] [a,b]. For sufficiently small β β, the normalized remainder is at most εfit _ fit. Multiplying the preceding bound by R∞β(x)−R0,x R_∞^β(x)-R_0,x gives supC∈[Ca,Cb]|RT0+C(x)−R0,x−Blin,xlogCCref,x| _C∈[C_a,C_b] |R_T_0+C(x)-R_0,x-B_ lin,x CC_ ref,x | ≤[R∞β(x)−R0,x]b−a1−p[b−a8maxv∈a,b|1−2v|v(1−v)+Δa,b(p)16]+εfit. ≤[R_∞^β(x)-R_0,x] \ b-a1-p [ b-a8 _v∈\a,b\ |1-2v|v(1-v)+ _a,b^(p)16 ]+ _ fit \. This proves the lemma. ∎ Lemma C.5 (Log-linear law for a target prompt). Under Assumptions 1, 3 and 2, let x′ x satisfy c=c(x′,x)>0 c=c(x ,x)>0 and RT0+Cb(x′)>RT0+Ca(x′) R_T_0+C_b(x )>R_T_0+C_a(x ). For any εfit>0 _ fit>0 and sufficiently small β β, there exist Blin,x′,Cref,x′>0 B_ lin,x ,C_ ref,x >0 such that supC∈[Ca,Cb]|RT0+C(x′)−RT0(x′)−Blin,x′log(C/Cref,x′)|RT0+Cb(x′)−RT0+Ca(x′)≤(1+c)logb(1−a)a(1−b)+Δa,b(p)16+εfit. _C∈[C_a,C_b] |R_T_0+C(x )-R_T_0(x )-B_ lin,x (C/C_ ref,x ) |R_T_0+C_b(x )-R_T_0+C_a(x )≤ (1+c) b(1-a)a(1-b)+ _a,b^(p)16+ _ fit. (C.16) Proof. Write Q(u)=πx′(Cu) Q(u)= _x (C_u). The exact transfer identity gives (Q(u))=(Q(a))+c[(u)−(a)] logit(Q(u))= logit(Q(a))+c[ logit(u)- logit(a)]. As a function of s=(u) s= logit(u), dQds=cQ(1−Q),|dslogdQds|≤c,dHds=1u(1−u),|dslogdHds|≤1. dQds=cQ(1-Q), | dds dQds |≤ c, dHds= 1u(1-u), | dds dHds |≤ 1. Applying Lemma C.3 first to Q(σ(s)) Q(σ(s)) and then to H(σ(s)) H(σ(s)), and combining the two approximations, gives a constant α⋆ α such that supu∈[a,b]|Q(u)−α⋆−(Q(b)−Q(a))H(u)−H(a)H(b)−H(a)|≤[Q(b)−Q(a)](1+c)16logb(1−a)a(1−b). _u∈[a,b] |Q(u)-α -(Q(b)-Q(a)) H(u)-H(a)H(b)-H(a) |≤ [Q(b)-Q(a)](1+c)16 b(1-a)a(1-b). The proof of Lemma C.4 shows that replacing the normalized change in H H by F(u)=log(Cu/Ca)/log(Cb/Ca) F(u)= (C_u/C_a)/ (C_b/C_a) adds at most (Q(b)−Q(a))[Δa,b(p)/16+o(1)] (Q(b)-Q(a))[ _a,b^(p)/16+o(1)]. By Eq. C.10, multiplying by [RT0+Cb(x′)−RT0+Ca(x′)]/[Q(b)−Q(a)] [R_T_0+C_b(x )-R_T_0+C_a(x )]/[Q(b)-Q(a)] converts the result exactly into a bound for the unnormalized target reward, with no additional error term. Choose Blin,x′=RT0+Cb(x′)−RT0+Ca(x′)log(Cb/Ca)>0B_ lin,x = R_T_0+C_b(x )-R_T_0+C_a(x ) (C_b/C_a)>0 and choose Cref,x′>0 C_ ref,x >0 so that −Blin,x′logCref,x′ -B_ lin,x C_ ref,x equals the resulting constant term. The error from replacing the normalized change in H H by F(u) F(u) is independent of c c and πx′(0) _x (0). Hence, for sufficiently small β β, this error is at most εfit _ fit for every target. Thus supC∈[Ca,Cb]|RT0+C(x′)−RT0(x′)−Blin,x′logCCref,x′| _C∈[C_a,C_b] |R_T_0+C(x )-R_T_0(x )-B_ lin,x CC_ ref,x | ≤[RT0+Cb(x′)−RT0+Ca(x′)]1+c16logb(1−a)a(1−b)+Δa,b(p)16+εfit. ≤[R_T_0+C_b(x )-R_T_0+C_a(x )] \ 1+c16 b(1-a)a(1-b)+ _a,b^(p)16+ _ fit \. Dividing by the positive reward gain RT0+Cb(x′)−RT0+Ca(x′) R_T_0+C_b(x )-R_T_0+C_a(x ) proves the lemma. ∎ C.3 Benchmark log-sigmoid approximation We prove the benchmark log-sigmoid result of Section 6. Let ℬ be a benchmark distribution over target prompts, and write R(t):=x′∼ℬ[Rt(x′)] R(t):=E_x [R_t(x )]. The fitted lower level R0 R_0 below need not equal the base-model score R(0) R(0) or the observed score R(T0) R(T_0) before fitting begins. The fitted upper level A A need not equal the fixed-β β limiting reward. For x′∼ℬ x , let r+(x′) r_+(x ) :=πT0(⋅|x′)[RT0(x′,z)∣z∈ℳ(x)], :=E_ _T_0(·|x )[R_T_0(x ,z) z (x)], r−(x′) r_-(x ) :=πT0(⋅|x′)[RT0(x′,z)∣z∉ℳ(x)], :=E_ _T_0(·|x )[R_T_0(x ,z) z (x)], d(x′) d(x ) :=r+(x′)−r−(x′). :=r_+(x )-r_-(x ). The conditional expectations are well defined because the softmax policy has full support and 0<|ℳ(x)|<M 0<|M(x)|<M. When c(x′,x)>0 c(x ,x)>0, the mass on ℳ(x) (x) increases across the fitting interval, so the positive reward gain assumed in the main text is equivalent to d(x′)>0 d(x )>0. For 0<p<a<b<1 0<p<a<b<1, set δa,b:=(b)−(a)=logb(1−a)a(1−b),Γa,b(p):=12infλ>0oscu∈[a,b](u)−λlog[H(u)−H(p)], _a,b:= logit(b)- logit(a)= b(1-a)a(1-b), _a,b^(p):= 12 _λ>0osc_u∈[a,b] \ logit(u)-λ [H(u)-H(p)] \, (C.17) where p=πT0(ℳ(x)|x) p= _T_0(M(x)|x) and H H is defined above. Thus δa,b _a,b is the training-strategy log-odds span, and Γa,b(p) _a,b^(p) measures the best approximation of those log odds by a linear function of log compute. Set R0:=x′∼ℬ[r−(x′)] R_0:=E_x [r_-(x )], D:=x′∼ℬ[d(x′)] D:=E_x [d(x )], and A:=R0+D A:=R_0+D. Theorem C.6 (Benchmark log-sigmoid approximation under bounded feature overlap). Assume Assumptions 1 and 3 with one training prompt and benchmark distribution ℬ . Suppose Assumption 2 holds for x′∼ℬ x . Consider the vanishing-β β family of Lemma C.2. Suppose c(x′,x),d(x′)>0 c(x ,x),d(x )>0 for x′∼ℬ x , and let cmax>0 c_ >0 be a finite upper bound on c(x′,x) c(x ,x) for these prompts. For any 0<c0≤cmax 0<c_0≤ c_ and εfit>0 _ fit>0, sufficiently small β β admits B,Cmid>0 B,C_ mid>0 such that supC∈[Ca,Cb]|R(T0+C)−R0A−R0−11+(Cmid/C)B| _C∈[C_a,C_b] | R(T_0+C)-R_0A-R_0- 11+(C_ mid/C)^B | ≤tanh(c02δa,b2256+c0Γa,b(p)4) ≤ ( c_0^2 _a,b^2256+ c_0 _a,b^(p)4 ) +x′∼ℬ[d(x′)tanh(δa,b(c(x′,x)−c0)+/8)]x′∼ℬ[d(x′)]+εfit, + E_x [d(x ) ( _a,b(c(x ,x)-c_0)_+/8 ) ]E_x [d(x )]+ _ fit, (C.18) where (q)+:=maxq,0 (q)_+:= \q,0\. For c0=cmax c_0=c_ and fixed p<a<1 p<a<1, the corresponding raw-reward error is O((b−a)2) O((b-a)^2) as b−a→0 b-a→ 0, provided β β is sufficiently small. Proof of Theorem C.6. Fix 0<c0≤cmax 0<c_0≤ c_ . Let z(C)=(πT0+C(ℳ(x)|x))−(p) z(C)= logit( _T_0+C(M(x)|x))- logit(p). By Lemma C.1, the target mass of ℳ(x) (x) is πT0+C(ℳ(x)|x′)=σ((πT0(ℳ(x)|x′))+c(x′,x)z(C)). _T_0+C(M(x)|x )=σ ( logit( _T_0(M(x)|x ))+c(x ,x)z(C) ). Target answer-feature orthogonality fixes each conditional reward at its base value, independently of t t and β β. By Eq. 4.5 and training-strategy symmetry, the target’s strategy proportions within ℳ(x) (x) and its complement equal their base values at T0 T_0 and remain fixed thereafter. Thus r±(x′) r_±(x ) and d(x′) d(x ) do not vary with β β. Under the theorem’s assumptions, D>0 D>0, and R(T0+C)=R0+DF(z(C)),F(z):=x′∼ℬ[d(x′)σ((πT0(ℳ(x)|x′))+c(x′,x)z)]D.R(T_0+C)=R_0+DF(z(C)), F(z):= E_x [d(x )σ ( logit( _T_0(M(x)|x ))+c(x ,x)z ) ]D. In particular, A−R0=D>0 A-R_0=D>0. Put zmid:=[z(Ca)+z(Cb)]/2 z_ mid:=[z(C_a)+z(C_b)]/2 and define an auxiliary mixture by replacing each logistic slope c(x′,x) c(x ,x) with minc(x′,x),c0 \c(x ,x),c_0\: F~(z):=x′∼ℬ[d(x′)σ((πT0(ℳ(x)|x′))+(c(x′,x)−c0)+zmid+minc(x′,x),c0z)]D. F(z):= E_x [d(x )σ ( logit( _T_0(M(x)|x ))+(c(x ,x)-c_0)_+z_ mid+ \c(x ,x),c_0\z ) ]D. The offset (c(x′,x)−c0)+zmid (c(x ,x)-c_0)_+z_ mid makes the original and auxiliary logits agree at z=zmid z=z_ mid. Since |z−zmid|≤δa,b/2 |z-z_ mid|≤ _a,b/2 for z∈[z(Ca),z(Cb)] z∈[z(C_a),z(C_b)] and |σ(s)−σ(t)|≤tanh(|s−t|/4) |σ(s)-σ(t)|≤ (|s-t|/4), supz∈[z(Ca),z(Cb)]|F(z)−F~(z)|≤x′∼ℬ[d(x′)tanh(δa,b(c(x′,x)−c0)+8)]D. _z∈[z(C_a),z(C_b)]|F(z)- F(z)|≤ E_x [d(x ) ( _a,b(c(x ,x)-c_0)_+8 ) ]D. (C.19) We next bound the curvature of the auxiliary mixture. Write L(z):=(F~(z)) L(z):= logit( F(z)). Any positive mixture of logistic functions with slopes in (0,c0] (0,c_0] satisfies 0<L′(z)≤c0,|L′(z)|≤c024.0<L (z)≤ c_0, |L (z)|≤ c_0^24. (C.20) The derivatives of the integrand up to order two are uniformly bounded on compact z z-intervals because 0<d(x′)≤1 0<d(x )≤ 1 and 0<c(x′,x)≤cmax 0<c(x ,x)≤ c_ . Dominated convergence therefore permits differentiation under the benchmark expectation. For completeness, let t=ec0z t=e^c_0z. Each component qx′(t) q_x (t) of F~ F has odds proportional to tsx′ t^s_x for some sx′∈(0,1] s_x ∈(0,1], so qx′/(1−qx′) q_x /(1-q_x ) is concave in t t. Hence (1−qx′)∂ttqx′+2(∂tqx′)2≤0 (1-q_x ) _tq_x +2( _tq_x )^2≤ 0. Averaging with weights d d and applying the Cauchy–Schwarz inequality gives (x′∼ℬ[d(x′)∂tqx′]D)2≤(1−F~)x′∼ℬ[d(x′)(∂tqx′)2/(1−qx′)]D. ( E_x [d(x ) _tq_x ]D )^2≤(1- F) E_x [d(x )( _tq_x )^2/(1-q_x )]D. Consequently, (1−F~)∂ttF~+2(∂tF~)2≤0,(1- F) _t F+2( _t F)^2≤ 0, so F~/(1−F~) F/(1- F) is concave in t t. The same argument applied to the complementary components in t−1 t^-1 shows that (1−F~)/F~ (1- F)/ F is concave in t−1 t^-1. Also, ∂zF~=x′∼ℬ[d(x′)sx′c0qx′(1−qx′)]/D _z F=E_x [d(x )s_x c_0q_x (1-q_x )]/D, and x′∼ℬ[d(x′)qx′(1−qx′)]/D≤F~(1−F~) _x [d(x )q_x (1-q_x )]/D≤ F(1- F); thus 0<L′≤c0 0<L ≤ c_0. Writing h=L′/c0∈(0,1] h=L /c_0∈(0,1], concavity of the odds in t t yields L′/c02≤h(1−h) L /c_0^2≤ h(1-h), while concavity of the complementary odds in t−1 t^-1 yields −L′/c02≤h(1−h) -L /c_0^2≤ h(1-h). This proves Eq. C.20. Let ℓ interpolate L L at zmid−δa,b/(22) z_ mid- _a,b/(2 2) and zmid+δa,b/(22) z_ mid+ _a,b/(2 2). The interpolation remainder and Eq. C.20 give supz∈[z(Ca),z(Cb)]|L(z)−ℓ(z)|≤c02δa,b264,0<ℓ′≤c0. _z∈[z(C_a),z(C_b)]|L(z)- (z)|≤ c_0^2 _a,b^264, 0< ≤ c_0. By the definition of Γa,b(p) _a,b^(p), for every η>0 η>0 there exist λ>0 λ>0 and γ∈ℝ γ such that supu∈[a,b]|(u)−(p)−γ−λlog[H(u)−H(p)]|≤Γa,b(p)+η. _u∈[a,b] | logit(u)- logit(p)-γ-λ [H(u)-H(p)] |≤ _a,b^(p)+η. The relation between compute and strategy probability in Lemma C.2 gives log(Cu/Ca)=log[H(u)−H(p)]/[H(a)−H(p)]+o(1) (C_u/C_a)= \[H(u)-H(p)]/[H(a)-H(p)]\+o(1) uniformly on [a,b] [a,b]. Consequently, ℓ(γ+λlog[H(a)−H(p)]+λlog(C/Ca)) (γ+λ [H(a)-H(p)]+λ (C/C_a) ) is affine in logC C with positive slope and approximates L(z(C)) L(z(C)) within c02δa,b2/64+c0Γa,b(p)+c0η+o(1) c_0^2 _a,b^2/64+c_0 _a,b^(p)+c_0η+o(1). Applying σ σ gives a log-sigmoid fit to F~(z(C)) F(z(C)) with error at most tanh(c02δa,b2/256+c0Γa,b(p)/4+c0η/4)+o(1) (c_0^2 _a,b^2/256+c_0 _a,b^(p)/4+c_0η/4)+o(1). The normalized benchmark reward in Eq. C.18 is exactly F(z(C)) F(z(C)). Combine this with Eq. C.19 and let η↓0 η 0; the remaining uniform o(1) o(1) is absorbed into εfit _ fit. As the fitting interval narrows, δa,b=O(b−a) _a,b=O(b-a). In the definition of Γa,b(p) _a,b^(p), choose λ λ so that (u)−λlog[H(u)−H(p)] logit(u)-λ [H(u)-H(p)] has zero derivative at a a; Taylor’s theorem then gives Γa,b(p)=O((b−a)2) _a,b^(p)=O((b-a)^2). Taking c0=cmax c_0=c_ and εfit=(b−a)2 _ fit=(b-a)^2 makes the normalized error O((b−a)2) O((b-a)^2). Since A−R0≤1 A-R_0≤ 1, so is the raw error. ∎ Proof of Theorem 6.1. Take c0=cmax c_0=c_ in Theorem C.6; the second term vanishes. This gives the theorem directly when R0>0 R_0>0. If R0=0 R_0=0, apply that bound with εfit/2 _ fit/2 and choose a positive fitted lower level η<A η<A instead. Since R(T0+C)≤A R(T_0+C)≤ A over the fitting interval, |R(T0+C)−ηA−η−R(T0+C)A|≤ηA−η, | R(T_0+C)-ηA-η- R(T_0+C)A |≤ ηA-η, so a sufficiently small η η absorbs the difference into the remaining εfit/2 _ fit/2. To obtain this rate as the fitting interval narrows, take η=O((b−a)2) η=O((b-a)^2). ∎ C.4 Benchmark log-linear approximation The single-target bound also applies to the benchmark average. Each target prompt’s feature overlap is weighted by its reward gain over the fitting interval: c¯:=x′∼ℬ[c(x′,x)(RT0+Cb(x′)−RT0+Ca(x′))]R(T0+Cb)−R(T0+Ca). c:= E_x [c(x ,x) (R_T_0+C_b(x )-R_T_0+C_a(x ) ) ]R(T_0+C_b)-R(T_0+C_a). (C.21) The log-linear theorem assumes c¯<∞ c<∞; unlike the log-sigmoid theorem, it does not require a uniform bound on feature overlap. These reward gains depend on d(x′) d(x ), πT0(ℳ(x)|x′) _T_0(M(x)|x ), and c(x′,x) c(x ,x) through the exact transfer identity above. Theorem C.7 (Benchmark log-linear law). Assume Assumptions 1, 2 and 3 with one training prompt and 0<p<a<b<1 0<p<a<b<1. Suppose c(x′,x),d(x′)>0 c(x ,x),d(x )>0 for x′∼ℬ x and c¯<∞ c<∞. For any εfit>0 _ fit>0 and sufficiently small β β, there exist R0,Blin,Cref>0 R_0,B_ lin,C_ ref>0 such that supC∈[Ca,Cb]|R(T0+C)−R0−Blinlog(C/Cref)|R(T0+Cb)−R(T0+Ca)≤(1+c¯)δa,b+Δa,b(p)16+εfit. _C∈[C_a,C_b] |R(T_0+C)-R_0-B_ lin (C/C_ ref) |R(T_0+C_b)-R(T_0+C_a)≤ (1+ c) _a,b+ _a,b^(p)16+ _ fit. (C.22) If c(x′,x)≤cmax<∞ c(x ,x)≤ c_ <∞ on the benchmark, the raw-reward error is O((b−a)2) O((b-a)^2) for fixed p<a<1 p<a<1 as b−a→0 b-a→ 0, provided β β is sufficiently small. Proof of Theorem C.7. Apply Lemma C.5 for x′∼ℬ x . The error in approximating the relation between compute and strategy probability is uniform in c(x′,x) c(x ,x) and the initial target strategy mass, so a single sufficiently small β β makes the promptwise bounds hold for almost every prompt in the benchmark distribution. The fitted slope is Blin,x′=RT0+Cb(x′)−RT0+Ca(x′)log(Cb/Ca)>0.B_ lin,x = R_T_0+C_b(x )-R_T_0+C_a(x ) (C_b/C_a)>0. For this slope, choose the fitted constant term as the midpoint of the range of RT0+C(x′)−Blin,x′logC R_T_0+C(x )-B_ lin,x C on [Ca,Cb] [C_a,C_b]. This choice minimizes the uniform promptwise error. Because rewards lie in [0,1] [0,1] and Ca,Cb C_a,C_b are fixed, these slopes and constant terms are bounded and may be averaged. Averaging yields a log-linear fit with positive slope Blin=x′∼ℬ[Blin,x′] B_ lin=E_x [B_ lin,x ]. For any R0>0 R_0>0, choose Cref>0 C_ ref>0 so that the averaged constant term equals R0−BlinlogCref R_0-B_ lin C_ ref. The triangle inequality bounds its error by the expectation of the promptwise errors. Each is proportional to the positive gain RT0+Cb(x′)−RT0+Ca(x′) R_T_0+C_b(x )-R_T_0+C_a(x ). Dividing by the benchmark gain and using Eq. C.21 proves the displayed bound. As the fitting interval narrows, c¯≤cmax c≤ c_ , while δa,b _a,b and Δa,b(p) _a,b^(p) are O(b−a) O(b-a). The exact transfer identity gives duπx′(Cu)=c(x′,x)πx′(Cu)(1−πx′(Cu))u(1−u), ddu _x (C_u)= c(x ,x) _x (C_u)(1- _x (C_u))u(1-u), so the benchmark reward gain is O(b−a) O(b-a) because c(x′,x)≤cmax c(x ,x)≤ c_ . Taking εfit=b−a _ fit=b-a proves the raw-reward claim. ∎ C.5 Visualizing the approximation bounds We plot the approximation error bounds in Theorems C.6 and C.7. For a single target with unit reward contrast, Figure 5 shows the bounds for different fitting intervals. Figure 5: The theoretical approximation bounds are a few percentage points in a representative regime. We use p=0.1 p=0.1, initial target-strategy mass q0=0.5 q_0=0.5, unit reward contrast, and feature overlap c∈0.5,1 c∈\0.5,1\. We vary a∈[0.30,0.65] a∈[0.30,0.65] and b−a∈[0,0.30] b-a∈[0,0.30], with the zero-width boundary shown by continuity. All panels use the same raw-reward color scale, and contours report percentage-point error. The asymptotic term εfit _ fit is omitted. Neither functional form is uniformly favored. For c=0.5 c=0.5, the displayed log-sigmoid bound is smaller throughout the plotted range. For c=1 c=1, it remains smaller for most of the plotted intervals, but the log-linear bound becomes smaller when both a a and b−a b-a are large; for example, at (a,b−a)=(0.65,0.30) (a,b-a)=(0.65,0.30) the two bounds are approximately 4.90% 4.90\% and 1.75% 1.75\%, respectively. Appendix D Experiment Details and Further Results D.1 Training setup RL. We run GRPO in verl with a learning rate of 10−6 10^-6 for 200 steps. Each step samples 512 prompts and 16 responses per prompt, followed by one PPO epoch with a minibatch size of 128 prompts. Training rollouts use temperature 1, top-p p 1, a 2,048-token prompt limit, and a 16,384-token response limit. The lower and upper clipping parameters are 0.2 and 0.28. Mathematics rewards indicate final-answer correctness, while coding rewards measure the fraction of ten tests passed. SFT. We use batch size 512, learning rate of 5×10−5 5× 10^-5, a two-epoch cosine schedule with 10% warmup, and a 16,384-token sequence limit. We evaluate the model after 1.5 epochs of training (step 117). D.2 Transfer and forgetting Benchmarks. We evaluate transfer and forgetting across four domains: 1. Math: MATH500 [hendrycks2021math], AIME 2024, AIME 2025, and OMEGA [sun2025omega]; 2. Coding: LiveCodeBench [jain2024livecodebench], CodeContests [li2022alphacode], and DELTA [sun2025rlgrok]; 3. Knowledge and Planning: GPQA-Diamond [rein2023gpqa], HEAD-QA [vilares2019headqa], ACPBench [kokel2025acpbench], ToolUse [tang2023toolalpaca], and Science [feng2024sciknoweval]; 4. General Capabilities: CoQA [reddy2019coqa], IFEval [zhou2023ifeval], HaluEval [li2023halueval], MC-TACO [zhou2019mctaco], WildJailbreak [jiang2024wildteaming], and WildGuardTest [han2024wildguard]. Evaluation protocol. We use native chat templates. Sampled reasoning tasks use temperature 0.6, top-p p 0.95, top-k k 20, and a maximum generation length of 32,768 tokens. We sample three responses per problem for MATH500 and GPQA-Diamond, 10 for LiveCodeBench, 32 for AIME, and one for DELTA. CodeContests uses Eurus-2-RL’s 377-problem validation subset. For greedy decoding, the maximum response length is 32,768 tokens for ACPBench, CoQA, and HaluEval, and 4,096 tokens for ToolUse, Science, and the safety benchmarks. HEAD-QA and MC-TACO use likelihood scoring; IFEval uses one response. We retain native task metrics, including strict instruction-level IFEval accuracy, CoQA F1, DELTA test pass rates, and WildGuard-based refusal accuracy. Complete scores. Table 1 reports the absolute scores underlying Figure 1. Table 1: Mathematics gains accompany transfer to some tasks and forgetting on others. Scores (%; higher is better) for the base models, SFT models evaluated with thinking enabled, and RL models at step 200, rounded to one decimal. Colors indicate changes from the corresponding base model before rounding. Qwen3-14B-Base Gemma-3-12B-IT Benchmark Base SFT Math RL Math-Code RL Base SFT Math RL Math MATH500 75.5 87.7 87.4 88.3 80.9 80.3 82.9 AIME 2024 14.3 63.0 33.1 46.2 25.9 31.4 28.4 AIME 2025 14.0 48.7 25.2 31.4 17.4 23.9 21.8 OMEGA 11.4 21.3 17.7 25.1 14.0 12.0 13.8 Coding LiveCodeBench 29.4 18.8 35.3 36.7 30.5 8.9 30.8 CodeContests 21.0 11.9 23.6 30.2 24.4 6.9 21.0 DELTA 11.4 3.0 11.2 11.4 9.6 0.5 13.9 Knowledge and Planning GPQA-Diamond 42.4 56.1 47.3 49.7 38.1 32.8 35.9 ACPBench 54.3 61.8 66.1 72.1 69.3 49.6 65.0 HEAD-QA 36.1 36.4 37.0 36.7 39.2 34.3 38.7 ToolUse 30.9 4.1 22.7 20.6 43.3 9.3 52.6 Science 42.2 0.0 46.4 47.7 36.7 0.0 32.5 General Capabilities CoQA 11.8 1.6 53.6 61.1 52.3 1.6 78.6 IFEval 66.2 22.0 59.6 56.7 69.3 23.9 68.1 HaluEval 35.8 0.0 29.1 30.6 61.2 0.0 60.2 MC-TACO 50.0 66.7 63.5 71.7 76.5 67.4 68.1 WildJailbreak 29.4 3.9 21.0 24.7 27.3 6.2 25.6 WildGuardTest 74.0 24.5 66.3 70.6 77.1 48.9 75.9 Teal: gain; rust: loss. OMEGA averages three splits; DELTA reports BouncingSim test pass rate. IFEval uses strict instruction-level accuracy. D.3 Pass@N N Evaluation protocol. For each problem in MATH500, AIME 2024, AIME 2025, GPQA-Diamond, CodeContests, and BoxNet, we sample 1,024 independent responses using temperature 0.7, top-p p 0.95, and a maximum generation length of 16,384 tokens. Complete results. Table 2 reports the coverage results at each sampling budget underlying Figure 2. Table 2: Detailed breakdown of Pass@N N results for the RL and base models in Figure 2. Benchmark Model Pass@N N (%) 1 4 16 64 256 1024 Qwen3-14B-Base MATH500 Base 73.00 86.24 91.36 94.17 95.92 97.80 Math RL 87.13 91.23 93.42 94.84 95.71 96.60 Math-Code RL 88.13 91.95 93.98 94.85 95.35 95.80 AIME 2024 Base 15.66 32.31 51.42 66.11 75.90 90.00 Math RL 33.45 54.16 67.22 74.20 78.89 83.33 Math-Code RL 44.42 62.24 71.08 76.70 81.15 86.67 AIME 2025 Base 12.01 22.79 33.23 45.97 62.32 76.67 Math RL 24.03 36.59 51.24 64.58 74.47 83.33 Math-Code RL 31.19 45.61 58.38 67.91 79.07 90.00 CodeContests Base 19.74 32.32 42.66 51.33 57.92 63.13 Math RL 25.83 36.80 46.27 53.17 58.81 63.66 Math-Code RL 31.63 41.74 48.40 52.37 56.08 60.48 GPQA-Diamond Base 36.70 69.10 91.05 97.79 99.84 100.00 Math RL 45.95 70.72 84.49 92.38 95.82 97.47 Math-Code RL 45.41 70.67 85.07 92.73 96.22 98.48 BoxNet Base 3.83 9.89 17.90 25.20 30.08 34.00 Math RL 9.88 20.67 28.66 33.88 40.45 51.00 Math-Code RL 11.24 21.85 30.25 36.58 47.59 67.00 Gemma-3-12B-IT MATH500 Base 81.31 87.60 90.62 92.45 93.88 94.80 Math RL 82.73 88.31 91.24 93.24 94.21 94.40 AIME 2024 Base 24.37 39.25 50.69 62.13 69.67 80.00 Math RL 27.76 43.21 54.77 63.13 70.50 76.67 AIME 2025 Base 18.13 27.34 36.17 45.13 57.42 66.67 Math RL 20.54 29.82 39.40 47.81 55.10 60.00 CodeContests Base 23.27 34.12 42.32 48.20 53.02 55.70 Math RL 21.24 31.23 39.82 45.61 49.82 54.91 GPQA-Diamond Base 39.01 64.47 82.58 90.94 94.83 96.97 Math RL 38.59 66.14 85.24 93.22 96.40 97.47 BoxNet Base 9.08 11.41 12.84 14.14 15.02 16.00 Math RL 7.31 14.39 19.19 22.19 25.33 30.00 RL gain RL loss D.4 Scaling laws Fitting procedure. For each benchmark, let (Cj,Rj)j=1m (C_j,R_j)_j=1^m be its positive-compute evaluations, where C C is cumulative compute in units of 102 10^2 GPU-hours and scores lie in [0,1] [0,1]. We fix R0 R_0 to the measured base-model score. The zero-compute evaluation therefore determines R0 R_0 but is excluded from both fitting losses because logC C is undefined at C=0 C=0. All positive-compute evaluations receive equal weight. For the log-sigmoid law in Eq. 6.1, we solve (A^,B^,C^mid)∈argmin0≤A≤1,B>0,Cmid>0∑j=1m[Rj−R0−A−R01+(Cmid/Cj)B]2. ( A, B, C_ mid )∈ _ subarrayc0≤ A≤ 1,\;B>0,\\ C_ mid>0 subarray _j=1^m [R_j-R_0- A-R_01+(C_ mid/C_j)^B ]^2. (D.1) To solve this problem, set qj=[1+(Cmid/Cj)B]−1 q_j=[1+(C_ mid/C_j)^B]^-1. For fixed (B,Cmid) (B,C_ mid), the unconstrained optimum is R0+∑jqj(Rj−R0)/∑jqj2 R_0+ _jq_j(R_j-R_0)/ _jq_j^2; we clip it to [0,1] [0,1] to obtain A A. We then minimize the profiled loss over (logCmid,logB) ( C_ mid, B) using the Nelder–Mead method with bounds and multiple initializations. We search logB∈[−14,7] B∈[-14,7] and use Cmid∈[10−12,1028] C_ mid∈[10^-12,10^28] in normalized compute units. This finite numerical range is effectively unbounded for the reported data, and none of the fits reaches a midpoint boundary. Among all local solutions, we retain the one with the smallest squared error; the numerical search does not certify global optimality. For the log-linear law in Eq. 6.2, we first fit an unrestricted intercept and slope by ordinary least squares of Rj R_j on logCj C_j. Because the slopes in all reported fits are nonzero, this is equivalent to solving (B^lin,C^ref)∈argminBlin∈ℝ,Cref>0∑j=1m[Rj−R0−Blinlog(Cj/Cref)]2. ( B_ lin, C_ ref )∈ _ subarraycB_ lin ,\\ C_ ref>0 subarray _j=1^m [R_j-R_0-B_ lin (C_j/C_ ref) ]^2. (D.2) Here B^lin B_ lin is the fitted slope, and C^ref C_ ref is the unique compute level at which the fitted line equals R0 R_0. The step-based fits reported below use the same procedure with optimizer step as compute. Steps are divided by 200 for sigmoid conditioning, and the fitted compute scales are converted back to steps. For either fitted curve R^(C) R(C), we report =1−∑j=1m[Rj−R^(Cj)]2∑j=1m(Rj−R¯)2,R¯=1m∑j=1mRj, R^2=1- _j=1^m [R_j- R(C_j) ]^2 _j=1^m(R_j- R)^2, R= 1m _j=1^mR_j, (D.3) using the same positive-compute evaluations as in the corresponding fit; this quantity is defined whenever the denominator is nonzero. GPU-hour fits. Figure 6 shows the fitted scaling curves, and Table 3 reports their coefficients. Figure 6: Scaling behavior of RL compute. Both laws fit the observed training regime effectively. Table 3: Log-sigmoid and log-linear fits using GPU hours. A A is the fitted log-sigmoid asymptote on the raw [0,1] [0,1] score scale. Cmid C_ mid and Cref C_ ref are in units of 102 10^2 GPU-hours. Both laws fit the observed training regime effectively. Log-sigmoid Log-linear RL data Benchmark B B A A Cmid C_ mid Fit R^2 Blin(×100) B_ lin(× 100) Cref C_ ref Fit R^2 Math RL MATH500 0.54 0.54 1.000 1.000 41.7 41.7 0.982 2.76 2.76 0.6 0.6 0.975 Math RL AIME 2024 1.23 1.23 0.361 0.361 8.1 8.1 0.981 5.66 5.66 1.2 1.2 0.982 Math RL AIME 2025 1.71 1.71 0.262 0.262 9.1 9.1 0.942 3.83 3.83 1.8 1.8 0.920 Math RL LiveCodeBench 0.39 0.39 1.000 1.000 >104 >10^4 0.881 1.28 1.28 0.5 0.5 0.845 Math RL ACPBench 1.69 1.69 0.693 0.693 15.8 15.8 0.861 4.56 4.56 2.9 2.9 0.851 Math-Code RL MATH500 1.00 1.00 0.904 0.904 5.9 5.9 0.965 3.05 3.05 0.5 0.5 0.948 Math-Code RL AIME 2024 1.56 1.56 0.466 0.466 8.3 8.3 0.987 9.52 9.52 1.6 1.6 0.981 Math-Code RL AIME 2025 0.96 0.96 0.358 0.358 11.2 11.2 0.962 4.72 4.72 1.1 1.1 0.973 Math-Code RL LiveCodeBench 0.59 0.59 0.485 0.485 89.3 89.3 0.873 1.94 1.94 1.0 1.0 0.857 Math-Code RL ACPBench 1.19 1.19 0.762 0.762 11.7 11.7 0.926 5.59 5.59 1.6 1.6 0.934 Scaling with training steps. As a robustness check, we use optimizer step as the compute metric. We fit both laws through step 200 and evaluate their extrapolations through step 350. Figure 7 shows the fitted curves using optimizer steps, and Table 4 reports their coefficients. Figure 8 shows the corresponding fitting and extrapolation regimes. Although GPU-hours per step increase as average rollout lengths grow, log(GPU Hours) ( GPU Hours) is approximately linear in log(Steps) ( Steps) over these runs; the two compute metrics therefore give similar qualitative conclusions. Figure 7: Training-step scaling on five benchmarks. Points show measured Qwen3-14B scores for Math RL and Math-Code RL. Solid and dashed curves show log-sigmoid and log-linear fits as functions of optimizer step, respectively. Each panel uses its own truncated score scale. Figure 8: Scaling behavior using optimizer steps. Both laws are fit through step 200 and extrapolated through step 350, producing the same qualitative pattern as fits based on GPU-hours. Table 4: Log-sigmoid and log-linear fits using optimizer step. A A is the fitted log-sigmoid asymptote on the raw [0,1] [0,1] score scale. Cmid C_ mid and Cref C_ ref are in steps. Log-sigmoid Log-linear RL data Benchmark B B A A Cmid C_ mid Fit R^2 Blin(×100) B_ lin(× 100) Cref C_ ref Fit R^2 Math RL MATH500 0.76 0.76 1.000 1.000 207.1 207.1 0.985 3.77 3.77 8.7 8.7 0.976 Math RL AIME 2024 1.61 1.61 0.379 0.379 77.3 77.3 0.979 7.83 7.83 16.5 16.5 0.976 Math RL AIME 2025 3.05 3.05 0.259 0.259 72.1 72.1 0.953 5.22 5.22 20.9 20.9 0.902 Math RL LiveCodeBench 0.53 0.53 1.000 1.000 >104 >10^4 0.895 1.64 1.64 8.1 8.1 0.867 Math RL ACPBench 2.60 2.60 0.723 0.723 137.2 137.2 0.899 6.14 6.14 31.3 31.3 0.827 Math-Code RL MATH500 1.33 1.33 0.909 0.909 48.1 48.1 0.976 4.32 4.32 8.1 8.1 0.962 Math-Code RL AIME 2024 2.55 2.55 0.458 0.458 54.5 54.5 0.992 13.57 13.57 17.6 17.6 0.977 Math-Code RL AIME 2025 1.40 1.40 0.362 0.362 74.2 74.2 0.983 6.84 6.84 14.3 14.3 0.990 Math-Code RL LiveCodeBench 1.44 1.44 0.380 0.380 56.9 56.9 0.961 2.63 2.63 11.1 11.1 0.956 Math-Code RL ACPBench 2.29 2.29 0.740 0.740 59.7 59.7 0.956 8.15 8.15 17.8 17.8 0.944 For Math RL on LiveCodeBench, the step-based fit reaches A=1.000 A=1.000 and gives the unconstrained estimate Cmid=2.0×104 C_ mid=2.0× 10^4, far beyond the observed range; its midpoint should therefore be treated as weakly identified. Appendix E Reasoning-Route Analysis This appendix provides implementation details, examples, and prompt templates for the reasoning-route analysis in Section 3.1. E.1 Classification and measurement Questions and generation setup. The analysis in Section 3.1 compares Qwen3-14B-Base with Qwen3-Math-RL at step 200 using 16 responses per model for each of 240 questions from MATH500, AIME 2024, AIME 2025, GPQA-Diamond, CodeContests, and LiveCodeBench. Responses use temperature 0.7, top-p p 0.95, no top-k k truncation (top-k=−1 -k=-1), and a maximum generation length of 16,384 tokens. The 16 responses per model and question use sampling seeds 0 through 15. Route definition. Following Section 3.1, a reasoning route is the strategy a response uses to reach its conclusion. A route is identifiable when the response contains enough reasoning or code about the question to identify a solution method. The method need not be correct or complete: a program with errors or an unfinished solution can reveal a route, whereas a final answer alone does not. Each identifiable response receives one label for its primary route: the method used to reach its final conclusion or produce its submitted program. If this is unclear, we choose the most developed method, breaking ties by choosing the first method supported by reasoning or code. Routes differ in how they represent the problem or in their main idea, not in notation, mistakes within the same method, or whether it is carried out by hand or in code. For example, mathematical simplification and a search through all possible cases are different routes. LLM-assisted classification. We use a two-stage process to identify each response’s strategy and group responses that follow the same route. Before classification, we remove generated follow-up user/assistant turns and replace mathematical answer statements and values, or GPQA choice labels, with placeholders. In Stage A, GPT-5.6-Luna with high reasoning effort reads each response separately, describes its primary strategy, and selects supporting parts of its reasoning or code. These descriptions form the input to Stage B, where the same model groups base and RL responses by strategy within each question and assigns a shared route label to each group. Stage B uses the descriptions rather than the original responses, with model identity, correctness, and explicit final-answer labels hidden. Both stages use fixed prompts for each subject area and a new Codex CLI session (codex exec) for each labeling call. For quality control, GPT-5.6-Sol with high reasoning effort independently checks whether the Stage A descriptions are supported by the response text and whether Stage B has incorrectly combined or separated routes. Descriptions and labels that fail these checks are corrected and checked again. Shortened prompt templates appear in Section E.3. Route counts with equal sample sizes. For Figure 3(a), we compare the number of distinct routes using equal numbers of identifiable responses from both models. For each question, we set the common sample size to the smaller number of identifiable responses available from the two models. We average the number of distinct routes over all subsets of that size for each model, then average equally over questions within each benchmark and across the full set. When both models have 16 identifiable responses, this simply counts the distinct routes among all 16 responses. The overall mean weights benchmarks by their question counts. Routes not found in base samples. For Figure 3(b), we compute the fraction of each question’s 16 RL responses whose primary route does not appear in its original 16 base responses, then average over questions. The overall fraction is 487 of 3,840 RL responses (12.7%). Confidence intervals. We use 5,000 bootstrap resamples of questions for each benchmark and the full set. Questions are sampled with replacement, keeping their base and RL measurements together. The 95% confidence intervals use the 2.5th and 97.5th percentiles of the bootstrap estimates for each panel’s means and the paired mean difference reported in Section 3.1. Route labels stay unchanged during resampling. E.2 Examples of route changes after RL These are the two examples discussed in Section 3.1. Both have 16 identifiable responses per model, so the route counts use all responses without sample-size adjustment. Concentration on a route found in base samples. In livecodebench-0088, the task is to minimize |nums[i]−nums[j]| |nums[i]-nums[j]| subject to |i−j|≥x |i-j|≥ x. Eight base responses binary-search a sorted list and compare neighboring values, six compare index pairs with nested loops, and two use indices sorted by value. All 16 RL responses use the binary-search route, reducing the observed route count from three to one. Use of a route not found in the base sample. In livecodebench-0163, the task is to find positions where pattern a a appears within distance k k of a position where pattern b b appears. All 16 base responses compare pairs of matching positions directly. Five RL responses use the same method, while 11 use binary search over the list of positions where b b appears to find the lower end of [i−k,i+k] [i-k,i+k]. This gives one observed route for base and two for RL; the 11 binary-search responses are counted as using a route not found in the base sample. E.3 Shortened prompt templates These templates summarize the instructions for both stages and describe the output fields in plain language. Stage A: describe one response’s strategy (shortened) Input. The question, one cleaned response, and its response ID. 1. Read the entire response on its own. Describe only methods it carries out or specifically proposes; do not solve the question or judge correctness. 2. Choose the method used for the final conclusion or submitted program. If unclear, choose the most developed method, then the first method supported by reasoning or code. 3. Describe the initial approach, problem representation, and main solution step, without final-answer values or quality judgments. 4. Copy supporting reasoning or code exactly and link each claim to its supporting text. Mark the response as “no route” if no method is attempted, or “unclear” if the method cannot be identified. Output (main fields). Return a JSON list with one record containing the response ID, route status (one route, a route change, no route, or unclear), how the primary route was selected, starting approach, problem representation, main solution step, short strategy summary, and supporting text linked to each claim. Stage B: group responses into routes (shortened) Input. The question and its ID, plus the Stage A strategy descriptions for all base and RL responses to that question. 1. Use these descriptions to group responses by their primary strategy. Do not infer model identity or correctness. 2. Group responses under the same route when they differ only in notation, calculation errors, completeness, wording, or whether the same method is carried out by hand or in code. Separate routes only when the initial approach, problem representation, or main solution step differs. 3. Assign every response ID to exactly one group. Choose group members before naming routes, and describe the strategy shared by every member. 4. Place responses marked “no route” or “unclear” in a separate group for responses without an identifiable route. Output. Return a JSON record containing the question ID and a list of route groups. For each group, give its name, a one-sentence strategy description, and the IDs of its responses. Copy every response ID exactly. Appendix F Empirical Analysis of Theoretical Predictions This appendix examines empirical evidence for the transfer, coverage, and scaling predictions of Sections 4, 5 and 6, respectively. F.1 Transfer and forgetting across tasks and models We examine the task- and model-dependent transfer and forgetting predicted by Section 4, using cases that illustrate differences in target-task requirements and teacher–student compatibility. Protocol violations in tool use. ToolUse requires a specified Action name and structured Action Input arguments [shenfeld2026selfdistillation]. One prompt requests five facts about the physical characteristics of axolotls, together with source URLs. Qwen3-14B-Base produces one Action: getAxolotlFacts call, with arguments specifying the category “physical characteristics” and a limit of five, and is graded correct. Qwen3-Math-RL first produces the same call, then adds “Please provide the output from the getAxolotlFacts tool,” simulates further interaction, and repeats calls. These additional calls cause the response to be graded incorrect. The change from issuing a single requested call to simulating further interaction turns a correct initial action into an unsuccessful response. This example helps explain the ToolUse decline in Table 1: additional generated interaction can reduce performance on a task requiring a precise tool call, even as mathematical performance improves. Errors in factual verification. HaluEval requires a “Yes” or “No” judgment on whether a candidate answer contains hallucinated information [li2023halueval]. One item includes the question “Which magazine was started first Arthur’s Magazine or First for Women?” The accompanying background information identifies Arthur’s Magazine as a periodical published in 1844–1846. The candidate answer is “First for Women was started first,” whereas the dataset’s reference answer is “Arthur’s Magazine.” Qwen3-14B-Base correctly responds “Yes”, while Qwen3-Math-RL incorrectly responds “No”. Both outputs satisfy the required label format; the error is the acceptance of a false chronological comparison. This gives a concrete example of the HaluEval decline in Table 1: gains in mathematical problem solving can coexist with poorer judgment of a supplied answer’s factual accuracy, reflecting the different requirements of the target task. Compatibility between teacher and student models. Qwen3-14B-Base and Gemma-3-12B-IT are trained with SFT on the same 40K mathematics demonstrations obtained by rejection sampling from Qwen3-32B. Gemma-3-SFT declines on all 14 OOD benchmarks, whereas Qwen3-SFT declines on ten and improves on four (Table 1). Gemma’s losses are also substantially larger on several benchmarks. The difference therefore concerns both the breadth and severity of forgetting across tasks. In Eq. 4.9, the teacher policy determines the demonstrated strategies, while the student’s feature overlaps determine their support on a target prompt. Identical demonstrations can therefore lead to different sets E(x′) _E(x ) for the two students. Theorem 4.3 relates forgetting to whether these sets retain strategies valid for the target, and Eq. A.22 shows that the probabilities within each set also depend on the student’s initial strategy preferences on training and target prompts. The shared Qwen model family offers a plausible explanation for better compatibility with the Qwen student: the teacher’s demonstrated strategies may be closer to its initial preferences. Imitating these demonstrations can then preserve strategies useful beyond mathematics for Qwen while reducing their probability in Gemma, explaining the selective transfer of Qwen3-SFT and the broader, more severe forgetting of Gemma-3-SFT. F.2 Prompt-level Pass@N N analysis We examine the finite-budget coverage predictions of Section 5, focusing on how gains and losses on questions with rare successes change benchmark coverage. For the benchmark distribution ℬ , any policy π π satisfies the exact identity Pass@N(π)−Pass@N(π0)=x′∼ℬ[(1−Rπ0(x′))N−(1−Rπ(x′))N].Pass@N(π)-Pass@N( _0)=E_x [(1-R_ _0(x ))^N-(1-R_π(x ))^N ]. (F.1) Thus the aggregate reversals in Figure 2 reflect changes in the distribution of per-question success probabilities. We examine per-question records to show how gains and losses on individual questions produce these benchmark-level reversals. Benchmark dependence: AIME 2024 versus AIME 2025. In the Qwen Pass@N N evaluation, 1,024 responses were sampled per question and reused to estimate coverage at different budgets. Table 5 lists the questions accounting for the observed AIME 2024 and AIME 2025 coverage changes at N=1024 N=1024. Table 5: AIME questions accounting for the changes in Qwen’s observed Pass@1024 coverage after Math RL. Each row gives a question’s evaluation index and a brief problem summary. The Base and Math RL columns give the number of correct responses among 1,024 samples per question; zero indicates no correct response. Benchmark Index Question summary Base Math RL AIME24 8 Count maximal placements of two colors of chips on a 5×5 5× 5 grid. 3 0 AIME24 25 Count octagon colorings satisfying a rotation condition. 1 0 AIME25 11 Find the area of a bounded region defined by nonlinear inequalities in the plane. 0 24 AIME25 27 Compute a modular remainder from a long rational recurrence. 0 1 Under Math RL, some questions lose all observed successes while others gain them, producing opposite coverage shifts in the two sets. On AIME 2024, Math RL has no correct responses on two combinatorics questions for which the base has three and one, respectively, lowering coverage from 27/30 27/30 to 25/30 25/30. On AIME 2025, Math RL has 24 correct responses on an area question and one on a recurrence question for which the base has none, raising coverage from 23/30 23/30 to 25/30 25/30. All other questions retain the same observed coverage status. Three of the four changes depend on only one to three correct responses. These sampled success probabilities are of order 1/N 1/N, the scale governing coverage sensitivity in Theorem 5.1. The averaging in Eq. F.1 explains how the same model can shift coverage in opposite directions when the two question sets contain different gains and losses. The exact discordant-pair test gives a two-sided p=0.5 p=0.5 for each set. These coverage estimates are sensitive to rare successes: for a question with zero successes in 1,024 independent draws, the one-sided 95% upper bound on its success probability is approximately 0.29% 0.29\%. Training-data dependence: CodeContests. On CodeContests, the Qwen base model and Math RL model produce at least one correct response for 238 238 and 240 240 of the 377 377 questions, respectively; Math-Code RL does so for 228 228 questions. The base model solves 22 22 questions for which Math-Code RL produces no correct response, while Math-Code RL solves 12 12 questions for which the base produces no correct response. The following examples illustrate the two outcomes: • Loss of coverage. codecontests-1192 requires summing pairwise absolute differences for up to 105 10^5 inputs. The base answers it correctly in 375 375 of 1,024 1,024 samples. One correct program generated by the base model sorts the inputs and computes the sum in O(nlogn) O(n n) time. None of the 1,024 1,024 Math-Code RL responses is correct; every generated program times out. • Gain of coverage. codecontests-1332 requires finding the minimum number of subset toggles needed to turn on the lamps in each prefix of an array. Math-Code RL answers it correctly in 74 74 of 1,024 1,024 samples, whereas none of the base responses is correct. Theorem 4.1 provides an interpretation of these outcomes in terms of strategy selection. For codecontests-1192, a correct base response uses a sorting strategy zsort∈ℳ(x′) z_ sort (x ). By Eq. 4.5, training changes its odds relative to competing strategies, with each training prompt’s contribution weighted by its feature overlap with the target. Training on code problems can therefore reduce the probability of selecting this efficient algorithm if these weighted updates favor competing strategies. For codecontests-1332, the same mechanism can increase the probability of selecting a strategy that solves the lamp problem. Eq. 4.6 expresses both effects in the vanishing-KL limit by conditioning the base policy on (x′) (x ): this can increase or decrease the probability assigned to valid strategies in ℳ(x′) (x ). Pass@1 nevertheless increases from 19.7% 19.7\% to 31.6% 31.6\% (Table 2). With only 12 12 newly solved questions among 377 377, most of this increase must come from higher average accuracy on the questions solved by both models. Base-model dependence: AIME 2025 and BoxNet. Under Math RL, sampled AIME 2025 coverage rises from 23/30 23/30 to 25/30 25/30 for Qwen but falls from 20/30 20/30 to 18/30 18/30 for Gemma. In the vanishing-KL limit of the transfer model, RL conditions the base policy on strategies favored by training (Eq. 4.6). The probability of selecting a valid strategy rises or falls according to whether valid strategies have a larger or smaller share of probability within this set than in the original policy. Different initial strategy preferences and feature overlaps can therefore produce opposite changes under the same training mixture. BoxNet also differs across base models (Table 2): Qwen improves Pass@1 from 3.8% 3.8\% to 9.9% 9.9\% and observed Pass@1024 coverage from 34/100 34/100 to 51/100 51/100, while Gemma lowers Pass@1 from 9.1% 9.1\% to 7.3% 7.3\% but expands coverage from 16/100 16/100 to 30/100 30/100. For Gemma, the 16 questions solved by both policies lose 2,819 2,819 correct responses, exceeding the 1,011 1,011 gained on 14 newly solved questions. Thus successes become spread across more questions but less frequent overall, explaining the opposing changes in the two metrics (Eq. F.1). In boxnet-0098, the prompt requires moving red, blue, and green boxes to their targets, with the blue box first moved to an adjacent square [stojanovski2026reasoning]. The base has 0/1024 0/1024 successful responses, whereas Gemma RL has 749/1024 749/1024. In one matched pair, the base plan delivers only the red box, while the RL plan completes all three deliveries, including the blue box’s intermediate move. Both outputs are valid fenced JSON. This example makes the target’s requirements concrete: a useful planning strategy must account for the prerequisite move and all three goals (Theorem 4.1). Together, these records show different question-level outcomes behind the aggregate curves. The same Qwen Math RL model loses observed coverage on two AIME 2024 questions but gains it on two AIME 2025 questions. Math-Code RL increases accuracy on some CodeContests questions and decreases it on others. Gemma’s Math RL model broadens coverage on BoxNet while lowering accuracy on questions solved by both models. These records illustrate changes in coverage across tasks and models, with differences in runtime outcomes, plan completeness, and the distribution of correct responses across questions. F.3 Scaling fit results and interpretation We examine the empirical implications of the scaling analysis in Section 6, focusing on the local validity of fitted laws and the interpretation of their coefficients. The log-sigmoid and log-linear curves in Figure 4 both agree well with the observations over the range of compute used for fitting. Their similarity there does not determine how scores will change beyond that range: the log-linear law maintains a constant slope against logC C, whereas the log-sigmoid law bends toward A A. The later evaluations distinguish the two curve shapes; which law extrapolates better varies across training runs and benchmarks. The five-benchmark average is smoother than some individual trajectories, so its fit alone does not establish benchmark-level predictability. The coefficients in Table 3 describe local score trends. The reported 100Blin 100B_ lin is the fitted change in score percentage points per unit increase in logC C. Math-Code RL has a larger log-linear slope on all five benchmarks; for AIME 2024 it is 9.52 9.52 rather than 5.66 5.66 points. This difference characterizes the fitted range, not a sustained rate of improvement. For the log-sigmoid law, writing q(C)=[1+(Cmid/C)B]−1 q(C)=[1+(C_ mid/C)^B]^-1 gives dR/dlogC=(A−R0)Bq(C)(1−q(C)) /d C=(A-R_0)Bq(C)(1-q(C)); the exponent B B alone does not determine the local slope. The fitted Cmid C_ mid is the compute level halfway between R0 R_0 and A A, not a measured onset of saturation. The fitted asymptote also requires care. On MATH500, Math RL reaches the imposed bound A=1 A=1 and has Cmid=41.7 C_ mid=41.7 in units of 102 10^2 GPU-hours, near the end of the fitting range; Math-Code RL has A=0.904 A=0.904 and Cmid=5.9 C_ mid=5.9. These values describe different fitted curve shapes but do not establish long-run performance ceilings. The fits in Table 4 use optimizer steps to measure compute, so their coefficients should be interpreted on that scale. B B and Blin B_ lin describe the shapes of the fitted score curves. The gain-weighted feature overlap in the theoretical log-linear result enters the approximation bound Eq. C.22. Longer-range forecasts require separate validation.