Paper deep dive

Learning Style, Forgetting Semantics: A Case Study of SFT and RFT on Classification Tasks

Haodong Liang, Yanhao Jin, Krishnakumar Balasubramanian, Lifeng Lai

Year: 2026Venue: arXiv preprintArea: stat.MLType: PreprintEmbeddings: 156

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%

Last extracted: 10/5/2026, 6:04:16 AM

Summary

This paper investigates why Supervised Fine-Tuning (SFT) causes more semantic forgetting than Reinforcement Fine-Tuning (RFT) on classification tasks, even when teacher demonstrations are semantically correct. Using a linear-softmax policy model, the authors decompose policy updates into semantic and style components. They prove that while both methods update the semantic channel in parallel, RFT preserves style symmetry (zero style drift), whereas SFT develops off-axis style drift due to teacher stylistic preferences. This drift creates a positive lower bound on semantic error for SFT, while RFT retains zero semantic error.

Entities (7)

RFT · method · 99%SFT · method · 99%Semantic Forgetting · phenomenon · 95%Style Drift · phenomenon · 95%Linear-Softmax Policy · model · 90%Semantic Channel · concept · 90%Style Channel · concept · 90%

Relation Signals (6)

SFT → causes → Semantic Forgetting

confidence 95% · SFT forgetting admits a strictly positive lower bound over a finite training interval

RFT → preserves → Zero Semantic Error

confidence 95% · RFT retains zero semantic error

SFT → induces → Style Drift

confidence 90% · SFT with a nonuniform teacher develops off-axis style drift

RFT → preserves → Style Symmetry

confidence 90% · RFT with exact policy gradients preserves this symmetry [no within-class style preference]

Style Drift → causes → Semantic Forgetting

confidence 85% · We use this drift to establish a separation... SFT forgetting admits a strictly positive lower bound

Teacher Style Preference → drives → Style Drift

confidence 85% · SFT fits this preference... fitting a teacher’s style preference is by itself sufficient to cause semantic forgetting

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Why does supervised fine-tuning (SFT) lead to more forgetting than reinforcement fine-tuning (RFT), even when all teacher demonstrations are semantically correct? We study this question on classification tasks where tokens within each semantic class express the same semantic answer in different styles. The tasks share an underlying semantic rule but differ in their prompt distributions and teachers' stylistic preferences. Using a tractable linear-softmax policy, we derive an exact decomposition of the updates into semantic and style components. We show that, at a common policy and prompt, SFT and RFT have parallel semantic updates but differ in their style dynamics. Starting from a policy with no within-class style preference, RFT with exact policy gradients preserves this symmetry, whereas SFT with a nonuniform teacher develops off-axis style drift along a nonzero task mean under population updates. We use this drift to establish a separation under explicit conditions: for population updates from a common perfectly fitted checkpoint, SFT forgetting admits a strictly positive lower bound over a finite training interval, while RFT retains zero semantic error. Simulations over task sequences support these theoretical predictions.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

153,199 characters extracted from source content.

Expand or collapse full text

Learning Style, Forgetting Semantics: A case study of SFT and RFT on Classification Tasks Haodong Liang Affiliation: University of California, Davis hdliang, yahjin, kbala, lflai@ucdavis.edu Yanhao Jin Affiliation: University of California, Davis hdliang, yahjin, kbala, lflai@ucdavis.edu Krishnakumar Balasubramanian Affiliation: University of California, Davis hdliang, yahjin, kbala, lflai@ucdavis.edu Lifeng Lai Affiliation: University of California, Davis hdliang, yahjin, kbala, lflai@ucdavis.edu October 1, 2026 Abstract Why does supervised fine-tuning (SFT) lead to more forgetting than reinforcement fine-tuning (RFT), even when all teacher demonstrations are semantically correct? We study this question on classification tasks where tokens within each semantic class express the same semantic answer in different styles. The tasks share an underlying semantic rule but differ in their prompt distributions and teachers’ stylistic preferences. Using a tractable linear-softmax policy, we derive an exact decomposition of the updates into semantic and style components. We show that, at a common policy and prompt, SFT and RFT have parallel semantic updates but differ in their style dynamics. Starting from a policy with no within-class style preference, RFT with exact policy gradients preserves this symmetry, whereas SFT with a nonuniform teacher develops off-axis style drift along a nonzero task mean under population updates. We use this drift to establish a separation under explicit conditions: for population updates from a common perfectly fitted checkpoint, SFT forgetting admits a strictly positive lower bound over a finite training interval, while RFT retains zero semantic error. Simulations over task sequences support these theoretical predictions. 1 Introduction Fine-tuning plays a central role in adapting pretrained foundation models to downstream tasks and enhancing their task-specific capabilities (Howard and Ruder, 2018; Devlin et al., 2019; Raffel et al., 2020). For example, instruction tuning on diverse demonstrations improves language models’ ability to follow instructions and generalize to unseen tasks (Ouyang et al., 2022; Chung et al., 2024); reward-based fine-tuning improves mathematical and coding performance by optimizing model-generated responses against verifiable outcomes (Shao et al., 2024; Guo et al., 2025). However, as models learn new tasks, they may lose previously acquired capabilities, a phenomenon known as forgetting (Wang et al., 2024; Luo et al., 2025; Chen et al., 2026). Preserving existing capabilities while adapting to new tasks is therefore an important consideration, highlighting the need to understand when and why fine-tuning can degrade them. Two widely used fine-tuning methods are supervised fine-tuning (SFT), which fits teacher demonstrations, and reinforcement fine-tuning (RFT), which updates the model using rewards for its own sampled responses. Empirical studies find that RFT often preserves existing capabilities better than SFT (Chen et al., 2026; Lai et al., 2026). This gap can arise even with correct supervision: Chu et al. (2025) observe that SFT on correct expert demonstrations degrades existing performance under input distribution shifts that preserve the task rules, whereas RFT from the same checkpoint improves it. Existing theories offer distributional explanations for such differences. Shenfeld et al. (2026b) identify an implicit bias of online RL toward the solution closest to the initial policy in KL divergence among those that solve the new task. Balasubramanian and Kasiviswanathan (2026) quantify how forward- and reverse-KL objectives affect the preservation of the old-task component in a two-component Gaussian mixture. These accounts, however, leave a specific mechanistic question unanswered: how does fitting a distribution of correct responses induce errors on previously mastered tasks? The ParityMNIST experiments of Shenfeld et al. (2026b) make this issue concrete: the two SFT baselines, trained on different distributions over equally correct labels, exhibit different retention trade-offs, but both preserve prior-task performance less effectively than RFT. Related language-model studies show that SFT can teach response formats (Zhou et al., 2023), but that imitation can accompany declines in factual accuracy (Gudibande et al., 2024). To understand this phenomenon, we isolate the role of these preferences in a classification case study, separating an answer’s semantic content from its particular expression, which we call its style. For example, “4,” “four,” and “The answer is 4” are all correct responses to 2+22+2, but a teacher may prefer one expression over the others. In our setting, SFT fits this preference, whereas RFT rewards semantic correctness regardless of style. This distinction motivates our central question: Can fitting the teacher’s style preference alone cause semantic forgetting, even when every demonstration is correct? We study this question in a tractable classification model with a linear-softmax policy, where tasks share the same semantic rule but differ in their input distributions and teachers’ stylistic preferences. For this policy class, we derive an exact decomposition of the updates into semantic and style components, allowing us to trace how style fitting induces errors on a previously mastered task. Under explicit conditions and population updates, we prove that SFT causes strictly positive semantic forgetting even when every demonstration is correct, whereas RFT preserves zero error. Our main contributions are: • An exact decomposition of the update. We split the policy parameters into a semantic channel and a style channel and characterize the update direction in each channel under SFT and RFT. The semantic updates are parallel, whereas the style updates differ: from a policy with no style preference, RFT keeps the style channel exactly zero throughout training, while SFT drifts along the task mean (Theorems 1 and 2). • A provable population retention gap between SFT and RFT. Starting from a checkpoint that perfectly fits the previous task and has no style preference, we prove a positive lower bound on SFT forgetting under population updates when the next task has a nonzero prompt mean and the teacher’s style preference is stronger in one semantic class than in the other, whereas RFT retains perfect semantic accuracy (Theorem 3). • A finite-sample bound on off-axis energy. On top of the population-level analysis, we account for the additional effects of finite-sample fluctuations on the off-axis parameter energy of SFT and RFT. Our bounds cover both semantic and style components and show that RFT admits a smaller upper bound on the expected off-axis energy than SFT (Theorem 4), and the resulting bounds are consistent with our population-level analysis in the large-sample limit. Together, these results suggest that fitting a teacher’s style preference is by itself sufficient to cause semantic forgetting under SFT, even when the teacher is always correct, whereas RFT avoids it because learning from its own on-policy samples induces no style drift. In Section 4, we provide empirical evidence for this mechanism by simulating the model over multi-task sequences under population and finite-sample updates; the results illustrate the mechanisms in Theorems 2 and 3 and the finite-sample effects studied in Theorem 4. We also report ablation studies in Appendix E to examine the model’s behavior when some of the theoretical conditions are not satisfied. Additional related work: We provide a detailed discussion of more related work in Appendix A. 2 Preliminaries 2.1 Problem setup We take a case study on a family of tasks that share a binary-valued semantic rule but differ in their prompt distributions and in teachers‘ preferences among semantically equivalent outputs. Semantic classes. Let ∈ℝpx ^p denote a prompt representation and A a finite output vocabulary. A map σ:→0,1σ:A→\0,1\ defines two semantic classes, b=σ−1(b)A_b=σ^-1(b), with |0|=|1|=M≥2|A_0|=|A_1|=M≥ 2. Tokens within a class express the same semantic answer in different styles. A unit direction ∈ℝpu ^p, shared by all tasks, determines the correct class and reward: b⋆()=⟨,⟩>0,R(a,)=σ(a)=b⋆().b (x)=1\ ,\,x >0\, R(a,x)=1\σ(a)=b (x)\. (1) Thus each prompt rewards exactly one class, regardless of expression style. Figure 1 illustrates this relationship. For RFT, we assume access to an accurate semantic reward signal, which may be provided by human correctness judgments or an automatic verifier. Notations. Throughout our analysis, we refer to components parallel to the shared semantic direction u as on-axis and those orthogonal to it as off-axis. We write ⟂:=−⟨,⟩v :=v- ,\,u for vectors and apply this projection rowwise to matrices, i.e., ⟂:=⁡(p−⊤)V :=V(I_p-uu ). x: "2+2=?"x: "1+4=?""4""four""The answer is 4" 0:R=1A_0:\ R=1"5""five""It equals 5" 1:R=0A_1:\ R=0"4""four""The answer is 4" 0:R=0A_0:\ R=0"5""five""It equals 5" 1:R=1A_1:\ R=1(a) Semantic classesterse → explanatorynumeric → verbal"4""four""The answer is 4""5""five""It equals 5"teacher preferenceTask 1Task 2Task 3(b) Style space Figure 1: Semantic classes and expression styles. (a) The same token space is partitioned into semantic classes 0A_0 and 1A_1. The prompt determines which class is rewarded; all tokens within a class receive the same reward. Here, x denotes the prompt’s vector representation. (b) Teachers favor different expression styles across tasks. Tasks. Each task t specifies a prompt distribution ∼N⁡(t,p)x N( μ_t,I_p), and an SFT teacher. The teacher always selects a token from the correct semantic class, but may favor particular styles of expressing the correct answer. For each task t and semantic class b∈0,1b∈\0,1\, the teacher has a fixed within-class preference distribution ρt,b _t,b, which may differ across tasks. The teacher distribution is qt(a∣)=ρt,b⋆()(a),a∈b⋆(),0,otherwise.q_t(a )= cases _t,b (x)(a),&a _b (x),\\ 0,&otherwise. cases (2) Thus the teacher is always semantically correct. Throughout the analysis, we assume ⟨t,⟩=0 μ_t,\,u =0 for every task t, which allows task-specific prompt shifts while keeping the semantic coordinate ⟨,⟩∼N⁡(0,1) ,\,x N(0,1), therefore isolating the role of style drift in semantic forgetting. Policy. We use a linear-softmax policy, a standard parameterization in policy-gradient reinforcement learning (Sutton et al., 1999; Kakade, 2001; Agarwal et al., 2021). This linear-softmax form is also used for classification heads in supervised fine-tuning of pretrained language models (Radford et al., 2018; Devlin et al., 2019); our analysis holds the prompt representations fixed. The policy is parameterized by ∈ℝ2M×pW ^2M× p, with rows aw_a: π(a∣)=softmax⁡()a=e⟨a,⟩∑a′∈e⟨a′,⟩. _W(a )= (Wx)_a= e _a,\,x _a e _a ,\,x . (3) The policy outputs the highest-scoring token, with semantic class b^()=σ⁡(argmaxa⁡⟨a,⟩) b_W(x)=σ ( _a _a,\,x ). Under this setup, SFT learns to match the teacher’s distribution over individual tokens, whereas RFT optimizes a reward that depends only on the semantic class. We next specify the corresponding learning rules. 2.2 Learning rules On task t, SFT aims to match the teacher’s token distribution by minimizing cross-entropy, whereas RFT aims to maximize the expected semantic reward. For a prompt x, SFT and RFT minimize the following losses, respectively: ℒSFT(;)=−a⋆∼qt(⋅∣)[logπ(a⋆∣)], _SFT(W;x)=-E_a q_t(· ) [ _W(a ) ], (4) ℒRFT(;)=−a′∼π(⋅∣)[R(a′,)]. _RFT(W;x)=-E_a _W(· ) [R(a ,x) ]. (5) Both methods process one prompt at a time with step size η, starting from =W=0. SFT draws a demonstration a⋆∼qt(⋅∣)a q_t(· ) and takes a stochastic gradient step on (4); RFT takes a stochastic gradient step on (5) using its exact policy gradient.11 1 Here, ”exact” means that we compute the expected reward by summing over all actions under the current policy and take its exact gradient, while prompt sampling remains stochastic. We discuss the sampled REINFORCE updates in Appendix D.1. For every a∈a , a+ _a^+ =a+η(a=a⋆−π(a∣)) =w_a+η (1\a=a \- _W(a ) )x (SFT), (SFT), (6) a+ _a^+ =a+ηπ(a∣)(R⁡(a,)−R¯()) =w_a+η\, _W(a ) (R(a,x)- R_W(x) )x (RFT), (RFT), where R¯()=∑a′∈π(a′∣)R(a′,) R_W(x)= _a _W(a )R(a ,x) is the expected reward. A derivation of these update rules is given in Lemma B.1. For task t, we write the population versions of the updates in (6) in terms of the parameter matrix W: + ^+ =+ηt[(qt(⋅∣)−π(⋅∣))⊤] =W+η\,E_t [(q_t(· )- _W(· ))x ] (SFT), (SFT), (7) + ^+ =+ηt[π(⋅∣)(R(⋅,)−R¯())⊤] =W+η\,E_t [ _W(· )(R(·,x)- R_W(x))x ] (RFT). (RFT). 2.3 Continual learning We train the policy sequentially on tasks t=1,…,Tt=1,…,T and denote its parameters after task t by tW_t. We evaluate both its final performance across tasks and its retention of earlier tasks. For task t, we define the semantic accuracy and error under its prompt distribution as at()=ℙ∼N⁡(t,p)(b^()=b⋆()),ℒt()=1−at().a_t(W)=P_x N( μ_t,I_p) ( b_W(x)=b (x) ), _t(W)=1-a_t(W). Following standard practice in continual learning (Chaudhry et al., 2018; Chaudhry et al., 2019; Lin et al., 2023), we evaluate the final checkpoint TW_T using the average generalization error across all tasks and the average forgetting over earlier tasks: (T)=1T∑t=1Tℒt(T),ℱ(T)=1T−1∑t=1T−1(max1≤l≤T−1⁡at(l)−at(T)).G^(T)= 1T _t=1^TL_t(W_T), ^(T)= 1T-1 _t=1^T-1 ( _1≤ l≤ T-1a_t(W_l)-a_t(W_T) ). (8) For each earlier task, forgetting compares its final accuracy with its best accuracy over all preceding task checkpoints. Negative values of ℱ(T)F^(T) indicate positive backward transfer. 3 Main Results In this section, we compare SFT and RFT by decomposing the policy parameters into semantic and style channels (Section 3.1). We first show that, at the same policy and prompt, the two methods update the semantic channel in the same direction (Theorem 1). Despite this alignment, their style dynamics differ: starting from a policy uniform within each class, RFT preserves zero style deviation, whereas SFT develops population style drift along a nonzero task mean when the teacher has within-class style preferences (Theorem 2). We then show that this style drift can produce a strict separation in forgetting between SFT and RFT at the population level (Theorem 3). Finally, we complement the population analysis with finite-sample bounds on the off-axis drifting energy of both methods under fresh sampling (Theorem 4). 3.1 Semantic–style decomposition We decompose the policy parameters into a semantic channel =((0),(1)) α=( α^(0), α^(1)), which represents the mean weight associated with each semantic class, and a style channel =(a)a∈ β=( β_a)_a , which captures the policy’s preferences among semantically equivalent tokens within each class. Specifically, for b∈0,1b∈\0,1\, (b)=1M∑a∈ba,a=a−(σ⁡(a)),so ∑a∈ba=. α^(b)= 1M _a _bw_a, β_a=w_a- α^(σ(a)), _a _b β_a=0. (9) Each token’s score then splits as ⟨a,⟩=⟨(σ⁡(a)),⟩+⟨a,⟩. _a,\,x = α^(σ(a)),\,x + β_a,\,x . Using this decomposition, the policy (3) can then be written as π(a∣)=exp⁡(⟨(σ⁡(a)),⟩+⟨a,⟩)∑b=01∑a′∈bexp⁡(⟨(b),⟩+⟨a′,⟩). _W(a )= \! ( α^(σ(a)),\,x + β_a,\,x ) _b=0^1 _a _b \! ( α^(b),\,x + β_a ,\,x ). (10) Let Δ:=(1)−(0) α:= α^(1)- α^(0). For any W, the decision rule is D()=⟨Δ,⟩+ξ(),b^()=D()>0, D_W(x)= α,\,x + _ β(x), b_W(x)=1\D_W(x)>0\, ξ()=maxa∈1⁡⟨a,⟩−maxa∈0⁡⟨a,⟩. _ β(x)= _a _1 β_a,\,x - _a _0 β_a,\,x . (11) Thus, a within-class style preference can change the semantic prediction when ξ() _ β(x) reverses the sign of ⟨Δ,⟩ α,\,x . If Δ, α, β are on-axis, i.e., Δ=γ α= and a=ca β_a=c_au for every token a, the policy’s decision boundary coincides with the true semantic boundary whenever γ>2maxa|ca|γ>2 _a|c_a| (Lemma B.2). In particular, for a class-uniform policy (= β=0), this decision boundary is determined solely by Δ α. We next compare how SFT and RFT update these two channels, beginning with the semantic component. 3.2 The semantic channel: a shared update direction Define the cumulative probability assigned to semantic class b by Pb(,):=∑a∈bπ(a∣).P_b(W,x):= _a _b _W(a ). (12) We first show that SFT and RFT update the semantic channel in the same direction. Theorem 1 (Shared Semantic direction). Under the setup of Section 2.1, the following hold. (i) (per-sample identity) For any W and prompt x, the semantic channel moves as ΔSFT+−Δ α^+_SFT- α =2ηM(2b⋆()−1)(1−Pb⋆()(,)), = 2ηM\, (2b (x)-1 ) (1-P_b (x)(W,x) )x, (13) ΔRFT+−Δ α^+_RFT- α =Pb⋆()(,)⋅(ΔSFT+−Δ). =P_b (x)(W,x)· ( α^+_SFT- α ). (i) (population direction) If = β=0 and Δ=c α=cu for some c∈ℝc , then the task-t population semantic steps are parallel to u: t[ΔSFT+−Δ] _t [ α^+_SFT- α ] =2ηM2π(∫0∞ze−z2/21+eczdz), = 2ηM 2π ( _0^∞ ze^-z^2/21+e^cz\,dz )u, (14) t[ΔRFT+−Δ] _t [ α^+_RFT- α ] =2ηM2π(∫0∞ze−z2/2ecz(1+ecz)2dz). = 2ηM 2π ( _0^∞ ze^-z^2/2e^cz(1+e^cz)^2\,dz )u. The proof of Theorem 1 is in Appendix C.1. In particular, at a fixed policy and prompt, the SFT semantic update in (13) does not depend on which style token the teacher sampled, while the RFT update is rescaled by the probability assigned to the correct class. Moreover, both coefficients in (14) are strictly positive, so both population updates increase the semantic margin along u. Despite this shared semantic update direction, the two methods differ in how they update the style channel, as we show next. 3.3 The style channel: RFT freezes, SFT drifts We next examine whether SFT and RFT preserve uniformity within each semantic class. The following theorem shows that RFT preserves zero style deviation, whereas SFT can develop population style drift when the teacher has within-class preferences. Theorem 2 (Style dynamics). Under the setup of Section 2.1, the following hold. (i) (RFT: style invariance) The RFT updates (6) preserve ≡ β 0. (i) (SFT: style drift) For any W and prompt x, the SFT update (6) gives a+−a=η[a=a⋆−π(a∣)−σ(a)=b⋆()−Pσ⁡(a)(,)M]. β_a^+- β_a=η [1\a=a \- _W(a )- 1\σ(a)=b (x)\-P_σ(a)(W,x)M ]x. (15) If = β=0, the task-t population SFT step (7) satisfies, for every a∈ba _b, t[a+−a] _t[ β_a^+- β_a] =η⁡(ρt,b(a)−1M)(12t+2b−12π), =η ( _t,b(a)- 1M ) ( 12 μ_t+ 2b-1 2πu ), (16) so thatt[a+−a]⟂ that _t[ β_a^+- β_a] =η2(ρt,b(a)−1M)t. = η2 ( _t,b(a)- 1M ) μ_t. The proof of Theorem 2 is in Appendix C.2. Theorem 2 gives us intuition for why SFT can be more prone to forgetting than RFT: starting from a class-uniform policy (= β=0), SFT can develop off-axis population drift along the task mean by fitting the teacher’s style preferences, while RFT preserves within-class symmetry. A special case is when either teachers have no within-class style preferences (i.e., ρt,b(a)=1/M _t,b(a)=1/M) or task means are zero (i.e., t= μ_t=0), in which case both methods avoid off-axis population style drift. When drift does arise, it can shift the semantic decision boundary through ξ _ β in (11). In Section 3.4, we show that this mechanism causes forgetting under explicit conditions. 3.4 From Style Drift to Semantic Error We now compare how SFT and RFT retain a previously learned task when trained on the next one. Both methods start from the same checkpoint, which perfectly classifies the previous task and whose policy is uniform within each semantic class. The following theorem establishes a finite-time separation in semantic error on both the previous and current tasks. Theorem 3 (Population error separation from a perfectly fitted checkpoint). Under the setup of Section 2.1, suppose that: (a) (Common start) SFT and RFT start from the same checkpoint 1W_1, which fits the previous task perfectly and has no style preference: ℒ1(1)=0,⁡(1)=.L_1(W_1)=0, β(W_1)=0. (b) (Next task) The next task has a nonzero mean 2≠ μ_2 0, and its teacher’s style preference is stronger in one semantic class than in the other, after relabeling the two classes if necessary: Δρ:=maxa∈1⁡ρ2,1(a)−maxa∈0⁡ρ2,0(a)>0 _ρ:= _a _1 _2,1(a)- _a _0 _2,0(a)>0. (c) (Training) Each method runs k2≥1k_2≥ 1 population updates (7) on task 22 with step size η2>0 _2>0; write τ2:=k2η2 _2:=k_2 _2 for the effective training time and 2W_2 for the resulting checkpoint. Then: (i) (RFT preserves perfect accuracy on both tasks) For every η2>0 _2>0 and k2≥1k_2≥ 1, ℒ1(2)=ℒ2(2)=0.L_1(W_2)=L_2(W_2)=0. (i) (SFT incurs semantic error on both tasks) There is a threshold τ0>0 _0>0, given in (C.34) that depends only on ∥2∥ μ_2 , Δρ _ρ, and M, such that for every 0<τ2≤τ00< _2≤ _0 and r∈1,2r∈\1,2\, ℒr(2)≥∫0∞ϕ⁡(t)Φ(⟨r,2⟩∥2∥−4(∥Δ(1)∥+2τ2)t∥2∥Δρτ2)t>0,L_r(W_2)≥ _0^∞φ(t) \! ( μ_r,\, μ_2 μ_2 - 4( α(W_1) +2 _2)t μ_2 _ρ _2 )\,dt>0, where ϕφ and Φ denote the standard Gaussian density and distribution function, respectively. The proof of Theorem 3 is in Appendix C.3. Theorem 3(i) implies that SFT introduces errors on two tasks that were initially classified perfectly. The lower bound measures the probability of a region where style drift overturns the correct semantic prediction. This error region grows with stronger teacher’s preference Δρ _ρ, smaller initial semantic margin ∥Δ(1)∥ α(W_1) , or longer training time τ2 _2. Old-task prompts are more likely to fall into this region when 1 μ_1 has a larger projection onto the direction of the new task mean 2 μ_2. Together with (8), these error bounds imply strictly positive forgetting and generalization error for SFT when 0<τ2≤τ00< _2≤ _0, whereas both metrics remain zero for RFT. This separation is, however, a finite-time phenomenon: under continued population SFT updates with a sufficiently small step size, both task errors converge to zero as τ2→∞ _2→∞, as shown in Appendices D.2 and D.3. 3.5 Finite-sample off-axis energy The separation result in Theorem 3 is established under population updates. In finite-sample training, prompt sampling can introduce additional deviations from the shared semantic direction u. In the next theorem, we provide a bound on the expected off-axis energy ∥⟂∥F2E _F^2 of both SFT and RFT. Theorem 4 (Off-axis parameter energy of SFT and RFT). Under the setup of Section 2.1, let t⁡(s)t(s) denote the task index at step s. Run the SFT and RFT updates (6) from ⁡(0)=W(0)=0 with constant step size η>0η>0. At each step s, independently draw s∼N⁡(t⁡(s),p)x_s N( μ_t(s),I_p), with a fresh teacher demonstration for SFT. For every N≥0N≥ 0, both methods satisfy ∥⟂(N)∥F2=∥⟂(N)∥F2+M2∥Δ⟂(N)∥2, (N) _F^2= β (N) _F^2+ M2 α (N) ^2, (17) and the following bounds hold: (i) (RFT) ∥RFT⟂(N)∥F2≤η28M(N⁡(p−1)+∑s=0N−1∥t⁡(s)∥2).E _RFT (N) _F^2≤ η^28M (N(p-1)+ _s=0^N-1 μ_t(s) ^2 ). (18) (i) (SFT) ∥SFT⟂(N)∥F2≤2η2(N⁡(p−1)+∑s=0N−1∥t⁡(s)∥)2.E _SFT (N) _F^2≤ 2η^2 ( N(p-1)+ _s=0^N-1 μ_t(s) )^2. (19) The proof of Theorem 4 is in Appendix C.4. Theorem 4 shows that, even under finite sampling, RFT admits a smaller upper bound on the expected off-axis energy than SFT. The following remark discusses how these bounds reflect sampling effects and relate to the population dynamics. Remark 1 (Sampling effects and population drift). We discuss the following cases as η→0η→ 0 with effective training time τ=Nητ=Nη fixed. This keeps the cumulative step size unchanged while increasing the number of independent samples. (i) Zero task means. If t= μ_t=0 for all tasks, the RFT and SFT bounds reduce to τη(p−1)/(8M)τη(p-1)/(8M) and 2τη(p−1)2τη(p-1), respectively. Both bounds vanish in the limit η→0η→ 0. This is consistent with Lemma B.5, which shows that the expected parameters remain on the semantic axis for both methods when all task means are zero. (i) Bounded nonzero task means. The RFT bound (18) remains ⁡(τη)O(τη) and vanishes, consistent with its on-axis population dynamics (Theorems 1(i) and 2(i)). The SFT bound (19), however, does not vanish, since the term η∑s=0N−1∥t⁡(s)∥η _s=0^N-1 μ_t(s) remains nonzero. This is consistent with the population style drift dynamics identified in Theorem 2(i). Together, these results show how fitting style can interfere with shared semantic knowledge even when all demonstrations are semantically correct. At a common policy and prompt, SFT and RFT share a semantic update direction, but RFT with exact policy gradients preserves an initially zero style channel, whereas SFT can develop off-axis population style drift that causes finite-time forgetting. In addition, when sampling is finite, it introduces additional off-axis fluctuations, for which RFT admits a smaller energy upper bound than SFT. We examine these mechanisms empirically in section 4. 4 Experiments The experiments simulate the model of Section 2.1 and examine three aspects of the learning dynamics: (1) RFT maintains zero off-axis energy under population updates (Theorem 1 and 2(i)); (2) nonzero task means and nonuniform teacher preferences induce SFT population off-axis style drift and can lead to semantic forgetting (Theorems 2(i) and 3); (3) finite-sample training produces additional off-axis parameter fluctuations, even when population off-axis drift is absent; we measure the final off-axis energy, whose expectation under Gaussian prompts is bounded in Theorem 4. Teacher parameterization. To construct the teachers for the simulations, we assign each token a fixed style embedding a∈ℝdh_a ^d describing how its semantic answer is expressed. A task- and class-dependent teacher preference vector t,b∈ℝdv_t,b ^d determines the within-class distribution: ρt,b(a)=e⟨t,b,a⟩∑a′∈be⟨t,b,a′⟩,a∈b. _t,b(a)= e _t,b,\,h_a _a _be _t,b,\,h_a , a _b. (20) Expressions with larger compatibility scores ⟨t,b,a⟩ _t,b,\,h_a receive higher probability; t,b=v_t,b=0 gives a uniform teacher within the class. Figure 1 illustrates such teacher preferences schematically. Setup. We train SFT and RFT from =W=0 on T=8T=8 sequential tasks, with prompt dimension p=24p=24, two semantic classes of M=6M=6 tokens each, and style dimension d=3d=3. We draw u uniformly on the unit sphere, M style embeddings 1,…,M∼N⁡(,3/3)h_1,…,h_M N(0,I_3/3) assigned to the tokens of each class, and teacher preference vectors t,b∼N⁡(,363)v_t,b N(0,36\,I_3).22 2 The variance 1/d1/d normalizes the style embeddings to unit scale, ∥j∥2=1E _j ^2=1, so that the within-class logits ⟨t,b,a⟩ _t,b,\,h_a in (20) have standard deviation controlled by t,bv_t,b alone, here about 66. This makes the teachers strongly nonuniform while leaving their preferences varied across tasks and classes. Prompts are Gaussian as in Section 2.1, with task means t∼N⁡(,p−⊤) μ_t N(0,I_p-uu ), so that t⟂ μ_t . These task parameters are sampled once and held fixed across runs. Training and evaluation. For finite-sample runs, we sample n=25n=25 prompts per task and process them in a single pass in random order, averaging over 5050 seeds that control the training prompts, demonstrations, and their order. For population runs, we approximate (7) by averaging updates over 10,00010,000 fresh prompts at each step, take 2525 steps per task to match the number of finite-sample updates, and average over 1010 seeds that control the Monte Carlo prompts. In both settings, we sweep η∈[10−4,0.5]η∈[10^-4,0.5] on a log-spaced grid and evaluate each task on a fixed set of 1,0001,000 held-out prompts from its own distribution. We report generalization error (T)G^(T), forgetting ℱ(T)F^(T), and the final off-axis parameter energy ∥T⟂∥F2 _T _F^2, all averaged across seeds. Figure 2: SFT and RFT under population and finite-sample updates. (a) Generalization–forgetting plane, for finite-sample (dashed, faint) and population (solid) updates; selected learning rates are annotated. (b) Final off-axis parameter energy ∥T⟂∥F2 _T _F^2, averaged across seeds, versus learning rate, with a logarithmic energy axis. Results. The main results are shown in Figure 2. In panel (a), population RFT maintains zero generalization error and forgetting throughout the learning-rate sweep, whereas population SFT exhibits strictly positive values at every tested learning rate. The persistence of this separation under population updates is consistent with off-axis style drift driven by nonzero task means and nonuniform teacher preferences (Theorem 3). In the finite-sample runs, prompt sampling introduces off-axis fluctuations that also raise RFT’s error (Theorem 4), yet RFT still achieves lower generalization error and forgetting than SFT at every matched learning rate. Panel (b) gives a complementary view: population SFT develops off-axis parameter energy, while population RFT remains close to zero (Theorem 1 and Theorem 2), with small residual energy from Monte Carlo prompt sampling. Finite-sample updates produce additional off-axis energy, which grows with the learning rate and is one to two orders of magnitude larger for SFT than for RFT. The observed energy gap qualitatively agrees with the smaller upper bound available for RFT in Theorem 4. Control and ablation experiments. We provide additional experiments in Appendix E to isolate the roles of task means and teacher style preferences and to relax the assumptions behind RFT’s style invariance. With either zero task means (Figure E.1) or uniform teachers (Figure E.2), the population SFT–RFT gap vanishes, while finite-sample SFT still accumulates more off-axis energy and higher generalization error than RFT; replacing Gaussian prompts by Laplace prompts leaves the results nearly unchanged (Figure E.3). We also compare the methods under corrupted rewards and under REINFORCE with finitely many sampled actions, and find that population RFT retains low forgetting in both cases; see Figure E.4 for details. 5 Conclusion This work provides intuition on why supervised fine-tuning forgets more than reinforcement fine-tuning, even when every demonstration is semantically correct, through a case study on classification tasks with a linear-softmax policy. We show that semantically correct supervision can still induce forgetting when training also fits task-specific style preferences. In our model, fitting these preferences can shift the semantic decision boundary and disrupt previously learned predictions, even in the absence of sampling noise. RFT avoids this source of interference by preserving within-class symmetry under exact policy gradients from a symmetric initialization. The finite-sample bounds and simulations further characterize the off-axis deviations introduced by prompt sampling. Limitations. Our theoretical analysis focuses on a tractable setting with a single-token linear-softmax model and a shared semantic rule across tasks. Future work could extend the analysis to more general settings such as nonlinear policies, multi-token outputs, and tasks with evolving semantic rules. References Agarwal et al. (2021) A. Agarwal, S. M. Kakade, J. D. Lee, and G. Mahajan On the theory of policy gradient methods: optimality, approximation, and distribution shift. Journal of Machine Learning Research 22 (98), p. 1–76. External Links: Link Cited by: §2.1. Balasubramanian and Kasiviswanathan (2026) K. Balasubramanian and S. P. Kasiviswanathan A quantitative characterization of forgetting in post-training. arXiv preprint arXiv:2603.12163. External Links: Document, Link Cited by: Appendix A, §1. Chaudhry et al. (2018) A. Chaudhry, P. K. Dokania, T. Ajanthan, and P. H. S. Torr Riemannian walk for incremental learning: understanding forgetting and intransigence. In European Conference on Computer Vision (ECCV), p. 532–547. External Links: Link Cited by: §2.3. Chaudhry et al. (2019) A. Chaudhry, M. Ranzato, M. Rohrbach, and M. Elhoseiny Efficient lifelong learning with A-GEM. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: §2.3. Chen et al. (2026) H. Chen, N. Razin, K. Narasimhan, and D. Chen Retaining by doing: the role of on-policy data in mitigating forgetting. In International Conference on Machine Learning (ICML), External Links: Link Cited by: Appendix A, Appendix A, §1, §1. Chu et al. (2025) T. Chu, Y. Zhai, J. Yang, S. Tong, S. Xie, D. Schuurmans, Q. V. Le, S. Levine, and Y. Ma SFT memorizes, RL generalizes: a comparative study of foundation model post-training. In International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research, Vol. 267, p. 10818–10838. External Links: Link Cited by: Appendix A, §1. Chung et al. (2024) H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, Y. Li, X. Wang, M. Dehghani, S. Brahma, et al. Scaling instruction-finetuned language models. Journal of Machine Learning Research 25 (70), p. 1–53. External Links: Link Cited by: §1. Devlin et al. (2019) J. Devlin, M. Chang, K. Lee, and K. Toutanova BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), p. 4171–4186. External Links: Document, Link Cited by: §1, §2.1. Ding et al. (2024) M. Ding, K. Ji, D. Wang, and J. Xu Understanding forgetting in continual learning with linear regression. In Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 235, p. 10978–11001. External Links: Link Cited by: Appendix A. Evron et al. (2023) I. Evron, E. Moroshko, G. Buzaglo, M. Khriesh, B. Marjieh, N. Srebro, and D. Soudry Continual learning in linear classification on separable data. In Proceedings of the 40th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 202, p. 9440–9484. External Links: Link Cited by: Appendix A. Gudibande et al. (2024) A. Gudibande, E. Wallace, C. Snell, X. Geng, H. Liu, P. Abbeel, S. Levine, and D. Song The false promise of imitating proprietary language models. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: §1. Guo et al. (2025) D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Bi, et al. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature 645, p. 633–638. External Links: Document, Link Cited by: §1. Howard and Ruder (2018) J. Howard and S. Ruder Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 328–339. External Links: Document, Link Cited by: §1. Kakade (2001) S. M. Kakade A natural policy gradient. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 14, p. 1531–1538. External Links: Link Cited by: §2.1. Kirkpatrick et al. (2017) J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, D. Hassabis, C. Clopath, D. Kumaran, and R. Hadsell Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences 114 (13), p. 3521–3526. External Links: Document, Link Cited by: Appendix A. Lai et al. (2026) S. Lai, H. Zhao, R. Feng, C. Ma, W. Liu, H. Zhao, X. Lin, D. Yi, Q. Zhang, H. Liu, G. Meng, and F. Zhu Reinforcement fine-tuning naturally mitigates forgetting in continual post-training. In International Conference on Machine Learning (ICML), External Links: Link Cited by: Appendix A, §1. Li et al. (2025) H. Li, S. Lin, L. Duan, Y. Liang, and N. B. Shroff Theory on mixture-of-experts in continual learning. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: Appendix A. Lin et al. (2023) S. Lin, P. Ju, Y. Liang, and N. Shroff Theory on forgetting and generalization of continual learning. In Proceedings of the 40th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 202, p. 21078–21100. External Links: Link Cited by: Appendix A, §2.3. Luo et al. (2025) Y. Luo, Z. Yang, F. Meng, Y. Li, J. Zhou, and Y. Zhang An empirical study of catastrophic forgetting in large language models during continual fine-tuning. IEEE Transactions on Audio, Speech and Language Processing 33, p. 3776–3786. External Links: Document, Link Cited by: §1. Mallya and Lazebnik (2018) A. Mallya and S. Lazebnik PackNet: adding multiple tasks to a single network by iterative pruning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), p. 7765–7773. External Links: Link Cited by: Appendix A. Ouyang et al. (2022) L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link Cited by: §1. Radford et al. (2018) A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever Improving language understanding by generative pre-training. Technical report OpenAI. External Links: Link Cited by: §2.1. Raffel et al. (2020) C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research 21 (140), p. 1–67. External Links: Link Cited by: §1. Rolnick et al. (2019) D. Rolnick, A. Ahuja, J. Schwarz, T. Lillicrap, and G. Wayne Experience replay for continual learning. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 32. External Links: Link Cited by: Appendix A. Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo DeepSeekMath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. External Links: Link Cited by: §1. Shenfeld et al. (2026a) I. Shenfeld, M. Damani, J. Hübotter, and P. Agrawal Self-distillation enables continual learning. In International Conference on Machine Learning (ICML), External Links: Link Cited by: Appendix A. Shenfeld et al. (2026b) I. Shenfeld, J. Pari, and P. Agrawal RL’s razor: why online reinforcement learning forgets less. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: Appendix A, §1, §1. Sutton et al. (1999) R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link Cited by: §2.1. Wang et al. (2024) L. Wang, X. Zhang, H. Su, and J. Zhu A comprehensive survey of continual learning: theory, method and application. IEEE Transactions on Pattern Analysis and Machine Intelligence 46 (8), p. 5362–5383. External Links: Document, Link Cited by: §1. Williams (1992) R. J. Williams Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning 8 (3–4), p. 229–256. External Links: Link Cited by: §D.1. Zenke et al. (2017) F. Zenke, B. Poole, and S. Ganguli Continual learning through synaptic intelligence. In International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research, Vol. 70, p. 3987–3995. External Links: Link Cited by: Appendix A. Zhang et al. (2026) Z. Zhang, Q. Dong, Q. Zhang, E. Zhou, J. Zhao, Z. Xi, S. Jin, X. Fan, Y. Zhou, M. Wu, Y. Fu, T. Ji, T. Gui, X. Huang, and K. Chen Why reinforcement fine-tuning preserves prior knowledge better: a data perspective. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: Appendix A. Zhou et al. (2023) C. Zhou, P. Liu, P. Xu, S. Iyer, J. Sun, Y. Mao, X. Ma, A. Efrat, P. Yu, L. Yu, S. Zhang, G. Ghosh, M. Lewis, L. Zettlemoyer, and O. Levy LIMA: less is more for alignment. In Advances in Neural Information Processing Systems, Vol. 36. External Links: Link Cited by: §1. Appendix A Literature Review Empirical findings on SFT and RFT. Recent studies compare how supervised and reinforcement fine-tuning affect generalization and forgetting in foundation models. Chu et al. (2025) find better generalization under RL to changes in reasoning rules and visual conditions, while Lai et al. (2026) report less forgetting of earlier tasks and general capabilities under RFT across sequential multimodal tasks. Studies of on-policy and model-generated supervision further highlight the role of the supervision distribution. Chen et al. (2026) show that approximately on-policy supervision reduces forgetting under SFT. Zhang et al. (2026) link the retention benefits of SFT on filtered model-generated trajectories to the magnitude and direction of example-induced updates. In self-distillation fine-tuning, Shenfeld et al. (2026a) use a demonstration-conditioned teacher to supervise student-generated trajectories without an explicit reward function, reducing forgetting during skill learning. Distributional explanations of the SFT–RFT gap. Shenfeld et al. (2026b) relate forgetting to the KL divergence from the initial policy and contrast SFT’s fitting of an external supervision distribution with online RL’s information-projection bias toward small policy changes. Building on the output-mixture simulations of Chen et al. (2026), Balasubramanian and Kasiviswanathan (2026) compare divergence-based post-training objectives by quantifying loss of old-component mass and drift within an old component. Their forward-KL objective fits new-mode data, whereas their reverse-KL objective fits a mixture that includes old behavior; the forgetting comparison therefore depends on both the objectives and their target distributions. Forgetting in continual-learning models. Complementary theoretical work examines how task structure, optimization, and architecture affect forgetting. In overparameterized linear regression, Lin et al. (2023) derive expressions for expected forgetting and generalization error that reveal the effects of model size, task similarity, and task order. For sequential stochastic gradient descent in linear regression with a shared ground-truth parameter, Ding et al. (2024) quantify how task covariances, ordering, and step size affect forgetting. In linear classification with jointly separable tasks, Evron et al. (2023) connect weakly regularized training to sequential max-margin learning and show that forgetting can arise despite joint separability. In an overparameterized mixture-of-experts regression setting, Li et al. (2025) relate expert specialization and routing to forgetting and generalization. Explicit methods for mitigating forgetting. Replay methods maintain access to earlier experience: Rolnick et al. (2019) combine on-policy learning with off-policy updates and behavioral cloning from a replay buffer to reduce forgetting in continual reinforcement learning. Parameter regularization penalizes changes to weights estimated to be important for past tasks. Elastic weight consolidation uses the diagonal Fisher information to weight these penalties (Kirkpatrick et al., 2017), while synaptic intelligence estimates parameter importance from contributions to loss reduction along the learning trajectory (Zenke et al., 2017). For parameter isolation, Mallya and Lazebnik (2018) iteratively prune a network, freeze weights allocated to earlier tasks, and train subsequent tasks using the remaining capacity. They provide complementary ways to control forgetting beyond the choice of SFT or RFT. Appendix B Supporting lemmas Lemma B.1 (SFT and RFT gradient updates). Fix a prompt x. A stochastic-gradient step on (4) using a⋆∼qt(⋅∣)a q_t(· ) and an exact gradient step on (5), both with step size η, yield the SFT and RFT updates in (6), respectively. Proof. For SFT, given a sampled demonstration a⋆∼qt(⋅∣)a q_t(· ), define ℓSFT(,,a⋆)=−log⁡π(a⋆∣). _SFT(W;x,a )=- _W(a ). Its expectation over the teacher is exactly (4); hence its gradient is an unbiased stochastic gradient of ℒSFTL_SFT. To compute its gradient, first expand the log-softmax as logπ(a′∣)=⟨a′,⟩−log∑a~∈exp(⟨a~,⟩). _W(a )= _a ,\,x - _ a \! ( _ a,\,x ). Differentiating with respect to aw_a gives ∇alogπ(a′∣) _w_a _W(a ) =a=a′−exp⁡(⟨a,⟩)∑a~∈exp⁡(⟨a~,⟩) =1\a=a \x- \! ( _a,\,x ) _ a \! ( _ a,\,x )x =(a=a′−π(a∣)). = (1\a=a \- _W(a ) )x. (B.1) Therefore gradient descent on ℓSFT _SFT gives (6). For RFT, expand (5) as ℒRFT(;)=−∑a′∈π(a′∣)R(a′,).L_RFT(W;x)=- _a _W(a )R(a ,x). We have ∇aπ(a′∣) _w_a _W(a ) =∇aexp⁡(⟨a′,⟩)∑a~∈exp⁡(⟨a~,⟩) = _w_a \! ( _a ,\,x ) _ a \! ( _ a,\,x ) =a=a′exp(⟨a′,⟩)(∑a~∈exp(⟨a~,⟩))(∑a~∈exp⁡(⟨a~,⟩))2 = 1\a=a \ \! ( _a ,\,x ) ( _ a \! ( _ a,\,x ) ) ( _ a \! ( _ a,\,x ) )^2x −exp⁡(⟨a′,⟩)exp⁡(⟨a,⟩)(∑a~∈exp⁡(⟨a~,⟩))2 - \! ( _a ,\,x ) \! ( _a,\,x ) ( _ a \! ( _ a,\,x ) )^2x =exp⁡(⟨a′,⟩)∑a~∈exp⁡(⟨a~,⟩)(a=a′−exp⁡(⟨a,⟩)∑a~∈exp⁡(⟨a~,⟩)) = \! ( _a ,\,x ) _ a \! ( _ a,\,x ) (1\a=a \- \! ( _a,\,x ) _ a \! ( _ a,\,x ) )x =π(a′∣)(a=a′−π(a∣)). = _W(a ) (1\a=a \- _W(a ) )x. (B.2) Consequently, −∇aℒRFT - _w_aL_RFT =∑a′∈R⁡(a′,)∇aπ(a′∣) = _a R(a ,x) _w_a _W(a ) =∑a′∈R(a′,)π(a′∣)(a=a′−π(a∣)) = _a R(a ,x) _W(a ) (1\a=a \- _W(a ) )x =π(a∣)R(a,)−π(a∣)∑a′∈π(a′∣)R(a′,) = _W(a )R(a,x)x- _W(a ) _a _W(a )R(a ,x)x =π(a∣)(R⁡(a,)−R¯()). = _W(a ) (R(a,x)- R_W(x) )x. (B.3) Gradient descent on ℒRFTL_RFT therefore gives (6). ∎ Lemma B.2 (Correct classification with on-axis channels). Suppose the channels in (9) satisfy Δ=γ α= with γ>0γ>0 and a=ca β_a=c_au for every a∈a . Set B:=maxa∈⁡|ca|B:= _a |c_a|. If γ>2Bγ>2B, then the decision rule (11) satisfies b^()=b⋆()for every ∈ℝp. b_W(x)=b (x) every x ^p. In particular, its decision boundary is :⟨,⟩=0\x: ,\,x =0\, and ℒt()=0L_t(W)=0 for every task t. Proof. Write z:=⟨,⟩z:= ,\,x . Since ⟨a,⟩=caz β_a,\,x =c_az, (11) gives D()=γz+ξ(),|ξ()|≤2maxa∈|caz|=2B|z|.D_W(x)=γ z+ _ β(x), | _ β(x)|≤ 2 _a |c_az|=2B|z|. For z≠0z≠ 0, it follows that zD()≥γz2−|z||ξ()|≥(γ−2B)z2>0.zD_W(x)≥γ z^2-|z|\,| _ β(x)|≥(γ-2B)z^2>0. Thus D()D_W(x) has the same sign as z. When z=0z=0, all style scores and D()D_W(x) vanish, so both (11) and (1) assign class 00. Hence the two rules agree everywhere and have the stated boundary. ∎ Lemma B.3 (Gaussian projection decomposition). Under the setup of Section 2.1, fix a task t and let ∼N⁡(t,p)x N( μ_t,I_p). Define z=⟨,⟩,⟂=(p−⊤)(−t).z= ,\,x , ξ = (I_p-uu )(x- μ_t). Then z∼N⁡(0,1),⟂∼N⁡(,p−⊤),z⟂,z N(0,1), ξ N(0,I_p-uu ), z \!\!\! ξ , and =z+t+⟂.x=zu+ μ_t+ ξ . Proof. Let =−t∼N⁡(,p) =x- μ_t N(0,I_p). Since ⟨t,⟩=0 μ_t,\,u =0, z=⊤,⟂=(p−⊤).z=u , ξ = (I_p-uu ) . Thus z and ⟂ ξ are jointly Gaussian. Using ∥=1 =1, Var⁡(z)=⊤=1,Cov⁡(⟂)=p−⊤,Var(z)=u u=1, ( ξ )=I_p-uu , and their cross-covariance is Cov⁡(⟂,z)=(p−⊤)=.Cov( ξ ,z)= (I_p-uu )u=0. Joint Gaussianity therefore implies z⟂z \!\!\! ξ . Finally, =⊤+(p−⊤)=z+⟂, =uu + (I_p-uu ) =zu+ ξ , which gives the stated decomposition of x. ∎ Lemma B.4 (Vectors invariant under G_u). Let ∈ℝpu ^p be nonzero and define G=∈O⁡(p):=.G_u=\ Q (p): Qu=u\. (B.4) Then = Qv=v for every ∈G Q∈ G_u if and only if ∈span⁡()v (u). Proof. First suppose =cv=cu. For every ∈G Q∈ G_u, =c=c=, Qv=cQu=cu=v, which proves one direction. Conversely, suppose = Qv=v for every ∈G Q∈ G_u. Decompose v as =c+⟂,c=⟨,⟩∥2,⟂.v=cu+v , c= ,\,v ^2, . Suppose, toward a contradiction, that ⟂≠v 0. Define =⟂∥⟂∥,=p−2⊤.e= v , Q=I_p-2ee . Because ∥=1 =1, ⊤ Q Q =(p−2⊤)2 =(I_p-2ee )^2 =p−4⊤+4(⊤)⊤ =I_p-4ee +4e(e e)e =p. =I_p. Hence ∈O⁡(p) Q (p). Since ⟂e , we also have =−2⊤=, Qu=u-2ee u=u, and therefore ∈G Q∈ G_u. On the other hand, ⟂ Qv =⟂−2⊤⟂ =v -2ee v =⟂−2⟂=−⟂. =v -2v =-v . It follows that =c+⟂=c−⟂≠c+⟂=, Qv=cQu+ Qv =cu-v ≠ cu+v =v, contradicting the assumption that every ∈G Q∈ G_u fixes v. Thus ⟂=v =0 and ∈span⁡()v (u). ∎ Lemma B.5 (Rotation equivariance of zero-mean trajectories). Assume t= μ_t=0 for every task and ⁡(0)=W(0)=0. Fix a task sequence and suppose each step uses a fresh prompt independent of the training history and, for SFT, a fresh teacher draw conditional on that prompt. Let G_u be as defined in (B.4); these are the orthogonal transformations that leave u fixed and act only on its orthogonal complement ⟂u . Let G_u act rowwise on the parameter matrix, (⋅)a=a( Q\!·\!W)_a= Qw_a. Under either the SFT or RFT update, for every ∈G Q∈ G_u and every step τ, ⋅⁡(τ)=d⁡(τ). Q\!·\!W(τ) d=W(τ). (B.5) Consequently, for every token a, ⁡[a(τ)],⁡[Δ(τ)],⁡[a(τ)]∈span⁡().E[w_a(τ)], [ α(τ)], [ β_a(τ)] (u). (B.6) Proof. Let ω denote the random input used in one update. For SFT, write ω=(,a⋆)ω=(x,a ) and define ⋆ω=(,a⋆) Q ω=( Qx,a ); for RFT, write ω=ω=x and define ⋆ω= Q ω= Qx. Since ∼N⁡(,p)x N(0,I_p), the linear transformation rule for Gaussian vectors and ⊤=pQQ =I_p give ∼N⁡(,p⊤)=N⁡(,p). Qx N\! ( Q0, QI_pQ )=N(0,I_p). Thus =d Qx d=x, so the prompt law is invariant under Q. Moreover, since Q is orthogonal and = Qu=u, we also have ⊤= Q u=u. Therefore, b⋆()=⟨,⟩>0=⟨⊤,⟩>0=⟨,⟩>0=b⋆().b ( Qx)=1\ ,\, Qx >0\=1\ Q u,\,x >0\=1\ ,\,x >0\=b (x). In addition, π⋅(a∣) _ Q·W(a Qx) =exp⁡(⟨a,⟩)∑a′∈exp⁡(⟨a′,⟩) = \! ( Qw_a,\, Qx ) _a \! ( Qw_a ,\, Qx ) =exp⁡(a⊤⊤)∑a′∈exp⁡(a′⊤⊤) = \! (w_a Q Qx ) _a \! (w_a Q Qx ) =exp⁡(⟨a,⟩)∑a′∈exp⁡(⟨a′,⟩)=π(a∣). = \! ( _a,\,x ) _a \! ( _a ,\,x )= _W(a ). The SFT teacher is invariant as well, since (2) depends on the prompt only through b⋆()b (x); the RFT reward is invariant for the same reason. Let Φ⁡(,ω) (W,ω) denote the updated parameter matrix +W^+ obtained by applying the SFT or RFT update in (6), and let Φa _a denote its aath row. For SFT, using the invariance of the policy shown above, we have Φa(⋅,⋆ω) _a( Q\!·\!W, Q ω) =a+η(a=a⋆−π⋅(a∣)) = Qw_a+η (1\a=a \- _ Q·W(a Qx) ) Qx =a+η(a=a⋆−π(a∣)) = Qw_a+η (1\a=a \- _W(a ) ) Qx =Φa(,ω). = Q _a(W,ω). For RFT, b⋆()=b⋆()b ( Qx)=b (x) implies R⁡(a,)=R⁡(a,)R(a, Qx)=R(a,x). Together with the policy invariance, this also gives R¯⋅() R_ Q·W( Qx) =∑a′∈π⋅(a′∣)R(a′,) = _a _ Q·W(a Qx)R(a , Qx) =∑a′∈π(a′∣)R(a′,)=R¯(). = _a _W(a )R(a ,x)= R_W(x). Consequently, the RFT update satisfies Φa(⋅,⋆ω) _a( Q\!·\!W, Q ω) =a+ηπ⋅(a∣)(R⁡(a,)−R¯⋅()) = Qw_a+η\, _ Q·W(a Qx) (R(a, Qx)- R_ Q·W( Qx) ) Qx =⁡[a+ηπ(a∣)(R⁡(a,)−R¯())] = Q [w_a+η\, _W(a ) (R(a,x)- R_W(x) )x ] =Φa(,ω). = Q _a(W,ω). Thus the following identity holds rowwise under either update rule: Φ⁡(⋅,⋆ω)=⋅Φ⁡(,ω). ( Q\!·\!W, Q ω )= Q\!·\! (W,ω). (B.7) We prove (B.5) by induction. It holds at τ=0τ=0 because ⁡(0)=W(0)=0, and hence ⋅⁡(0)=⁡(0) Q\!·\!W(0)=W(0). Now suppose ⋅⁡(τ)=d⁡(τ) Q\!·\!W(τ) d=W(τ). Let ωτ _τ be the fresh sample used at step τ. It is independent of ⁡(τ)W(τ), and the invariance of the sample law proved above gives ⋆ωτ=dωτ Q _τ d= _τ. Therefore, the induction hypothesis and independence imply the joint distributional identity (⋅⁡(τ),⋆ωτ)=d(⁡(τ),ωτ). ( Q\!·\!W(τ), Q _τ ) d= (W(τ), _τ ). (B.8) By the definition of the one-step update map, ⁡(τ+1)=Φ⁡(⁡(τ),ωτ)W(τ+1)= (W(τ), _τ). Together with (B.7) and (B.8), we have ⋅⁡(τ+1) Q\!·\!W(τ+1) =⋅Φ⁡(⁡(τ),ωτ) = Q\!·\! (W(τ), _τ ) =Φ⁡(⋅⁡(τ),⋆ωτ) = ( Q\!·\!W(τ), Q _τ ) =dΦ⁡(⁡(τ),ωτ) d= (W(τ), _τ ) =⁡(τ+1). =W(τ+1). This completes the induction and proves (B.5). It follows that [a(τ)]=⁡[a(τ)] QE[w_a(τ)]=E[w_a(τ)] for every ∈G Q∈ G_u. Lemma B.4 therefore implies that ⁡[a(τ)]∈span⁡()E[w_a(τ)] (u). The remaining claims follow because Δ α and a β_a are fixed linear combinations of the rows. ∎ Lemma B.6 (Lipschitz continuity of softmax). The softmax map is 11-Lipschitz in the Euclidean norm: for all ,′∈ℝ2Mh,h ^2M, ∥softmax⁡()−softmax⁡(′)∥2≤∥−′∥2. (h)- (h ) _2≤ -h _2. Proof. For =softmax⁡() π= (h), differentiating gives the Jacobian J⁡()=diag⁡()−⊤.J(h)=diag( π)- π π . For every ∈ℝ2Mv ^2M, 0≤⊤J()=∑a=12Mπava2−(∑a=12Mπava)2≤∑a=12Mπava2≤∥22.0 J(h)v= _a=1^2M _av_a^2- ( _a=1^2M _av_a )^2≤ _a=1^2M _av_a^2≤ _2^2. The first inequality follows because the expression is a variance under π, and the last uses 0≤πa≤10≤ _a≤ 1. Thus J⁡()J(h) is symmetric positive semidefinite with eigenvalues at most 11, so ∥J⁡()∥op≤1 J(h) _op≤ 1 for every h. Integrating the derivative along the line segment from ′h to h therefore gives ∥softmax⁡()−softmax⁡(′)∥2 (h)- (h ) _2 ≤∫01∥J⁡(′+t⁡(−′))∥op∥−′∥2t ≤ _0^1 J(h +t(h-h )) _op -h _2\,dt ≤∥−′∥2. ≤ -h _2. ∎ Appendix C Proofs of the main results C.1 Proof of Theorem 1 Proof. Part (i): the per-sample rate identity. For an arbitrary W and prompt x, by (12), P0(,)+P1(,)=1P_0(W,x)+P_1(W,x)=1. First consider the SFT step. If the demonstration is a⋆a with σ⁡(a⋆)=b⋆()σ(a )=b (x), then the row update is a+−a _a^+-w_a =η(a=a⋆−π(a∣)). =η (1\a=a \- _W(a ) )x. (C.1) Averaging (C.1) over a class and using the definition of (b) α^(b) in (9) gives (b)+−(b) α^(b)+- α^(b) =1M∑a∈bη(a=a⋆−π(a∣)) = 1M _a _bη (1\a=a \- _W(a ) )x =ηM(b=b⋆()−∑a∈bπ(a∣)) = ηM (1\b=b (x)\- _a _b _W(a ) )x =ηM(b=b⋆()−Pb(,)), = ηM (1\b=b (x)\-P_b(W,x) )x, (C.2) where the last equality is (12). Subtracting the b=0b=0 mean update from the b=1b=1 mean update yields ΔSFT+−Δ α^+_SFT- α =η⁡[b⋆()=1−P1(,)M−b⋆()=0−P0(,)M]. =η [ 1\b (x)=1\-P_1(W,x)M- 1\b (x)=0\-P_0(W,x)M ]x. (C.3) If b⋆()=1b (x)=1, then 2b⋆()−1=12b (x)-1=1, Pb⋆()(,)=P1(,)P_b (x)(W,x)=P_1(W,x), and P0(,)=1−Pb⋆()(,)P_0(W,x)=1-P_b (x)(W,x); if b⋆()=0b (x)=0, then 2b⋆()−1=−12b (x)-1=-1, Pb⋆()(,)=P0(,)P_b (x)(W,x)=P_0(W,x), and P1(,)=1−Pb⋆()(,)P_1(W,x)=1-P_b (x)(W,x). Therefore (C.3) becomes the SFT line of (13). For RFT, the reward is R(a,)=σ(a)=b⋆()R(a,x)=1\σ(a)=b (x)\, so (6) gives R¯() R_W(x) =∑aπ(a∣)σ(a)=b⋆()=Pb⋆()(,), = _a _W(a )1\σ(a)=b (x)\=P_b (x)(W,x), (C.4) a+−a _a^+-w_a =ηπ(a∣)(σ(a)=b⋆()−Pb⋆()(,)). =η\, _W(a ) (1\σ(a)=b (x)\-P_b (x)(W,x) )x. (C.5) Averaging (C.5) over a class gives (b)+−(b) α^(b)+- α^(b) =ηM∑a∈bπ(a∣)(b=b⋆()−Pb⋆()(,)) = ηM _a _b _W(a ) (1\b=b (x)\-P_b (x)(W,x) )x =ηPb(,)M(b=b⋆()−Pb⋆()(,)). =η P_b(W,x)M (1\b=b (x)\-P_b (x)(W,x) )x. (C.6) Subtracting the class-00 update in (C.6) from the class-11 update gives ΔRFT+−Δ α^+_RFT- α =η[P1(,)M(b⋆()=1−Pb⋆()(,)) =η [ P_1(W,x)M (1\b (x)=1\-P_b (x)(W,x) ) −P0(,)M(b⋆()=0−Pb⋆()(,))]. 71.13188pt- P_0(W,x)M (1\b (x)=0\-P_b (x)(W,x) ) ]x. (C.7) Again splitting on b⋆()b (x) and using P0(,)+P1(,)=1P_0(W,x)+P_1(W,x)=1, the right side of (C.7) is 2ηM(2b⋆()−1)Pb⋆()(,)(1−Pb⋆()(,)), 2ηM\, (2b (x)-1 )P_b (x)(W,x) (1-P_b (x)(W,x) )x, (C.8) which is the RFT line of (13). Part (i): population semantic update. Now assume additionally that = β=0 and Δ=c α=cu for some c∈ℝc . Then (10) implies that the class probabilities Pb(,)P_b(W,x) depend on W only through Δ α. Fix task t and use the notation of Lemma B.3: z=⟨,⟩,⟂=(p−⊤)(−t).z= ,\,x , ξ = (I_p-uu )(x- μ_t). Lemma B.3 gives z∼N⁡(0,1)z N(0,1), z⟂z \!\!\! ξ , and =z+t+⟂.x=zu+ μ_t+ ξ . It follows that ⟨Δ,⟩=c⁡⟨,z+t+⟂⟩=cz, α,\,x =c\, ,\,zu+ μ_t+ ξ =c\,z, because both t μ_t and ⟂ ξ are orthogonal to u. Hence (10) implies that P1(,)=(1+e−cz)−1P_1(W,x)=(1+e^-cz)^-1 and P0(,)=1−P1(,)P_0(W,x)=1-P_1(W,x). Since b⋆()=b⋆(z):=z>0b (x)=b (z):=1\z>0\, the correct-class probability has the one-dimensional representation Pb⋆()(,)=P⋆(z):=11+e−c|z|.P_b (x)(W,x)=P_ (z):= 11+e^-c|z|. This function is even in z. Also 2b⋆()−1=sign⁡(z)2b (x)-1= (z) almost surely. The SFT rate identity in (13) becomes ΔSFT+−Δ=2ηMsign⁡(z)(1−P⋆(z))(z+t+⟂). α^+_SFT- α= 2ηM (z)\, (1-P_ (z) )\, (zu+ μ_t+ ξ ). (C.9) Taking the task-t population expectation gives t[ΔSFT+−Δ] _t [ α^+_SFT- α ] =2ηM(⁡[sign⁡(z)(1−P⋆(z))z]CLOSE = 2ηM (E[ (z)(1-P_ (z))z]\,u +⁡[sign⁡(z)(1−P⋆(z))]t 102.43008pt+E[ (z)(1-P_ (z))]\, μ_t OPEN+⁡[sign⁡(z)(1−P⋆(z))⟂]). 102.43008pt+E[ (z)(1-P_ (z)) ξ ] ). (C.10) The last term vanishes because sign⁡(z)(1−P⋆(z)) (z)(1-P_ (z)) is a function of z alone, z is independent of ⟂ ξ , and ⁡[⟂]=E[ ξ ]=0. The t μ_t term also vanishes: 1−P⋆(z)1-P_ (z) is even, so sign⁡(z)(1−P⋆(z)) (z)(1-P_ (z)) is odd, and its expectation under the symmetric z∼N⁡(0,1)z N(0,1) is zero. Since sign⁡(z)z=|z| (z)z=|z|, the remaining scalar expectation is ⁡[|z|(1−P⋆(z))] [|z|(1-P_ (z)) ] =⁡[|z|1+ec|z|] =E [ |z|1+e^c|z| ] =2π∫0∞ze−z2/21+eczdz. = 2π _0^∞ ze^-z^2/21+e^cz\,dz. The last equality uses the standard normal density and the evenness of the integrand. Substituting into (C.10) gives the SFT line of (14). Similarly, the RFT rate identity in (13) becomes ΔRFT+−Δ=2ηMsign⁡(z)P⋆(z)(1−P⋆(z))(z+t+⟂). α^+_RFT- α= 2ηM (z)\,P_ (z) (1-P_ (z) )\, (zu+ μ_t+ ξ ). (C.11) Taking the task-t population expectation gives t[ΔRFT+−Δ] _t [ α^+_RFT- α ] =2ηM(⁡[sign⁡(z)P⋆(z)(1−P⋆(z))z]CLOSE = 2ηM (E[ (z)P_ (z)(1-P_ (z))z]\,u +⁡[sign⁡(z)P⋆(z)(1−P⋆(z))]t 102.43008pt+E[ (z)P_ (z)(1-P_ (z))]\, μ_t OPEN+⁡[sign⁡(z)P⋆(z)(1−P⋆(z))⟂]). 102.43008pt+E[ (z)P_ (z)(1-P_ (z)) ξ ] ). (C.12) Again the last term vanishes by independence and ⁡[⟂]=E[ ξ ]=0. The t μ_t term vanishes as well because P⋆(z)(1−P⋆(z))P_ (z)(1-P_ (z)) is even, so its product with sign⁡(z) (z) is odd. Using sign⁡(z)z=|z| (z)z=|z| and again integrating against the standard normal density gives ⁡[|z|P⋆(z)(1−P⋆(z))] [|z|P_ (z)(1-P_ (z)) ] =⁡[|z|ec|z|(1+ec|z|)2] =E [ |z|e^c|z|(1+e^c|z|)^2 ] =2π∫0∞ze−z2/2ecz(1+ecz)2dz. = 2π _0^∞ ze^-z^2/2e^cz(1+e^cz)^2\,dz. Substituting into (C.12) yields the RFT line of (14). ∎ C.2 Proof of Theorem 2 Proof. Part (i): RFT style invariance. Fix a prompt x and write the RFT update direction from (6) as a=π(a∣)(R⁡(a,)−R¯()).g_a= _W(a ) (R(a,x)- R_W(x) )x. (C.13) Assume = β=0. By (9), every row in a common semantic class is equal to that class’s mean. Thus, for any two tokens a,a′a,a in the same class, a _a =a′, =w_a , (C.14) ⟨a,⟩ _a,\,x =⟨a′,⟩, = _a ,\,x , π(a∣) _W(a ) =π(a′∣). = _W(a ). (C.15) The reward also agrees inside the class because, by hypothesis, R depends on a only through its class σ⁡(a)σ(a), which is constant on a semantic class: σ⁡(a)=σ⁡(a′) σ(a)=σ(a ) ⟹R⁡(a,)=R⁡(a′,). R(a,x)=R(a ,x). (C.16) Combining (C.13), (C.15), and (C.16), the update vector is identical for every row in the same class. Therefore if a,a′a,a lie in the same class, a+−a′+ _a^+-w_a ^+ =(a−a′)+η⁡(a−a′) = (w_a-w_a )+η (g_a-g_a ) =+η⋅=, =0+η·0=0, (C.17) Hence an update preserves = β=0. Induction over the prompt sequence gives RFT(τ)= β_RFT(τ)=0 at every step. Part (i): SFT style updates. For convenience, define the within-class probability deviation δa(,):=π(a∣)−1MPσ⁡(a)(,). _a(W,x):= _W(a )- 1MP_σ(a)(W,x). (C.18) For an arbitrary current matrix W, the SFT row update and its class average are exactly (C.1) and (C.2), respectively. Subtracting (C.2) at b=σ⁡(a)b=σ(a) from (C.1) gives a+−a β_a^+- β_a =η(a=a⋆−π(a∣))−ηM(σ(a)=b⋆−Pσ⁡(a)(,)) =η (1\a=a \- _W(a ) )x- ηM (1\σ(a)=b \-P_σ(a)(W,x) )x =η[a=a⋆−1Mσ(a)=b⋆−(π(a∣)−Pσ⁡(a)(,)M)] =η [1\a=a \- 1M1\σ(a)=b \- ( _W(a )- P_σ(a)(W,x)M ) ]x (C.19) =η[a=a⋆−1Mσ(a)=b⋆−δa(,)], =η [1\a=a \- 1M1\σ(a)=b \- _a(W,x) ]x, (C.20) which is exactly (15). Now fix a task t, a token a∈ba _b, and a current parameter matrix W with = β=0. Since W has identical rows within each class, all rows in the same semantic class have the same logit at every prompt. Therefore the policy is uniform within each semantic class: π(a∣)=1M∑a′∈bπ(a′∣)=1MPb(,)(a∈b). _W(a )= 1M _a _b _W(a )= 1MP_b(W,x) (a _b). Substituting this identity into the definition of δa _a gives δa(,)=π(a∣)−1MPσ⁡(a)(,)=0. _a(W,x)= _W(a )- 1MP_σ(a)(W,x)=0. Thus the policy-centering feedback in the exact style update (15) vanishes. For this fixed prompt and teacher draw, (15) becomes a+−a=η[a=a⋆−1Mb⋆()=b]. β_a^+- β_a=η [1\a=a \- 1M1\b (x)=b\ ]x. Now take expectation only over the teacher demonstration, conditional on the prompt x. By the teacher model (2), if b⋆()≠b (x)≠ b then the teacher places no mass on token a∈ba _b, and both terms inside the brackets have conditional mean zero. If b⋆()=b (x)=b, then ℙ⁡(a⋆=a∣)=ρt,b(a)P(a =a )= _t,b(a), so the conditional mean of the bracket is ρt,b(a)−1/M _t,b(a)-1/M. Combining the two cases, a⋆|[a=a⋆−1Mb⋆()=b]=b⋆()=b(ρt,b(a)−1M).E_a [1\a=a \- 1M1\b (x)=b\ ]=1\b (x)=b\ ( _t,b(a)- 1M ). Therefore the SFT population update of the style coordinate is [a+−a]=η(ρt,b(a)−1M)[b⋆()=b].E[ β_a^+- β_a]=η ( _t,b(a)- 1M )E[1\b (x)=b\\,x]. (C.21) It remains to compute the last prompt moment. Apply Lemma B.3 and write z=⟨,⟩z= ,\,x and ⟂=(p−⊤)(−t) ξ = (I_p-uu )(x- μ_t). Then z⟂z \!\!\! ξ and =z+t+⟂,x=zu+ μ_t+ ξ , while z∼N⁡(0,1)z N(0,1). The event b⋆=b =b depends only on z: it is z>0\z>0\ for b=1b=1 and z<0\z<0\ for b=0b=0. Hence [b⋆=b]=[b⋆=bz]+[b⋆=b(t+⟂)].E[1\b =b\\,x]=E[1\b =b\z]\,u+E[1\b =b\( μ_t+ ξ )]. For the off-axis term, independence gives [b⋆=b(t+⟂)]=ℙ(b⋆=b)t+ℙ(b⋆=b)[⟂]=12t.E[1\b =b\( μ_t+ ξ )]=P(b =b) μ_t+P(b =b)E[ ξ ]= 12 μ_t. For the on-axis term, [z>0z]=∫0∞z12πe−z2/2dz=12π,[z<0z]=−12π,E[1\z>0\z]= _0^∞z\, 1 2πe^-z^2/2\,dz= 1 2π, [1\z<0\z]=- 1 2π, so in either class [b⋆=bz]=2b−12π.E[1\b =b\z]\,u= 2b-1 2πu. Substituting these two moment identities into (C.21) proves (16). Finally, project (16) orthogonally to span⁡() (u). The term proportional to u is removed, while t μ_t is unchanged because t⟂ μ_t . This gives the off-axis identity in (16). ∎ C.3 Proof of Theorem 3 Proof. Step 1: perfect initial fit and RFT retention. Since ⁡(1)= β(W_1)=0, the class-score difference is D1()=⟨Δ(1),⟩D_W_1(x)= α(W_1),\,x . Since the old Gaussian law has positive density everywhere, ℒ1(1)=0L_1(W_1)=0 means that Δ(1)=∥Δ(1)∥,∥Δ(1)∥>0. α(W_1)= α(W_1) , α(W_1) >0. Subtracting the common row average :=(2M)−1∑a1,ac:=(2M)^-1 _aw_1,a from every row preserves policy probabilities, predictions, and both update increments. We therefore use these centered weights throughout the proof, so that 1,a=2σ(a)−12∥Δ(1)∥,∑a1,a=.w_1,a= 2σ(a)-12 α(W_1) , _aw_1,a=0. Write z:=⟨,⟩z:= ,\,x . Both tasks share the semantic rule b⋆()=z>0b (x)=1\z>0\. Lemma B.2, applied with B=0B=0, therefore shows that 1W_1 classifies both tasks perfectly: a1(1)=a2(1)=1.a_1(W_1)=a_2(W_1)=1. (C.22) Theorem 2(i) ensures = β=0 throughout RFT training, both updates preserve ∑a= _aw_a=0, and Theorem 1(i) gives strictly positive on-axis RFT increments. Thus every RFT iterate W satisfies a _a =2σ(a)−12∥Δ()∥, = 2σ(a)-12 α(W) , ∥Δ()∥ α(W) ≥∥Δ(1)∥>0. ≥ α(W_1) >0. Applying Lemma B.2 again with B=0B=0 shows that every RFT iterate classifies both tasks correctly, proving part (i). Step 2: quantifying the style drift under SFT. Let ^2:=2/∥2∥ μ_2:= μ_2/ μ_2 and y:=⟨^2,⟩y:= μ_2,\,x . By Lemma B.3, under task 22 we can write =z+2+⟂x=zu+ μ_2+ ξ , where z∼N⁡(0,1)z N(0,1) is independent of ⟂∼N⁡(,p−⊤) ξ N(0,I_p-uu ). Since ^2⟂ μ_2 and ∥^2∥=1 μ_2 =1, y=⟨^2,⟩=∥2∥+⟨^2,⟂⟩∼N⁡(∥2∥,1).y= μ_2,\,x = μ_2 + μ_2,\, ξ N( μ_2 ,1). Moreover, y depends only on ⟂ ξ , so it is independent of z. Define V:=span⁡,^2V:= \u, μ_2\. We will show that every parameter row remains in V throughout task-22 training. Define the residual V:=−z−y^2 ξ_V:=x-zu-y μ_2, which is the projection of x onto V⟂V . Since 2∈V μ_2∈ V, the residual has mean zero: 2[V]=2−0⋅−∥2∥^2=.E_2[ ξ_V]= μ_2-0·u- μ_2 μ_2=0. Orthogonality of the Gaussian projections also makes V ξ_V independent of (z,y)(z,y). Initially, 1,a=2σ(a)−12∥Δ(1)∥∈Vw_1,a= 2σ(a)-12 α(W_1) ∈ V. If all current rows lie in V, write a=ζa+ωa^2.w_a= _au+ _a μ_2. (C.23) Their logits and policy probabilities are then ⟨a,⟩=ζaz+ωay,πa=π(a∣)=eζaz+ωay∑a′∈eζa′z+ωa′y. _a,\,x = _az+ _ay, _a= _W(a )= e _az+ _ay _a e _a z+ _a y. Thus, for fixed current parameters, πa _a depends on x only through (z,y)(z,y). Meanwhile, the teacher definition (2) gives q2(a∣)=σ(a)=b⋆()ρ2,b⋆()(a),b⋆()=z>0.q_2(a )=1\σ(a)=b (x)\ _2,b (x)(a), b (x)=1\z>0\. By the teacher specification in Section 2.1, the within-class distribution ρ2,b _2,b is fixed for each class and does not depend on x, so q2(a∣)q_2(a ) depends on the input only through the sign of z. Consequently, q2(a∣)−πaq_2(a )- _a is a function of (z,y)(z,y) alone. By (7), the V⟂V component of the SFT parameter increment is η22[(q2(a∣)−πa)V]=η22[(q2(a∣)−πa)2[V∣z,y]]=. _2E_2[(q_2(a )- _a) ξ_V]= _2E_2[(q_2(a )- _a)E_2[ ξ_V z,y]]=0. The last equality follows because V ξ_V is independent of (z,y)(z,y) and has mean zero. Each updated row therefore remains in V. By induction, (C.23) holds at every task-22 SFT iterate, so it suffices to track the two scalar coefficients ζa _a and ωa _a for each token. Since ∥=1 =1 and ⟂^2u μ_2, the decomposition (C.23) gives ζa=⟨a,⟩ _a= _a,\,u . Taking the inner product of the SFT population update (7) with u and using z=⟨,⟩z= ,\,x yields ζa+−ζa=⟨a+−a,⟩=η22[(q2(a∣)−πa)z], _a^+- _a= _a^+-w_a,\,u = _2E_2[(q_2(a )- _a)z], |ζa+−ζa|≤η2|z|≤η2, | _a^+- _a|≤ _2E|z|≤ _2, since |q2(a∣)−πa|≤1|q_2(a )- _a|≤ 1 and z∼N⁡(0,1)z N(0,1). At initialization, ζa(0)=2σ(a)−12∥Δ(1)∥ _a^(0)= 2σ(a)-12 α(W_1) . After j≤k2j≤ k_2 updates, the triangle inequality gives |ζa(j)| | _a^(j)| ≤|ζa(0)|+∑ℓ=0j−1|ζa(ℓ+1)−ζa(ℓ)| ≤| _a^(0)|+ _ =0^j-1| _a^( +1)- _a^( )| (C.24) ≤12∥Δ(1)∥+jη2|z| ≤ 12 α(W_1) +j _2E|z| ≤12∥Δ(1)∥+τ2. ≤ 12 α(W_1) + _2. Collect the coefficients in (C.23) into a matrix j∈ℝ2M×2 _j ^2M× 2, whose row indexed by token a is (ζa(j),ωa(j))( _a^(j), _a^(j)) after j task-22 steps. For :=(z,y)⊤r:=(z,y) , each entry of j _jr is the corresponding token’s logit: (j)a=ζa(j)z+ωa(j)y=⟨a(j),⟩.( _jr)_a= _a^(j)z+ _a^(j)y= _a^(j),\,x . Thus softmax⁡(j) ( _jr) is the model’s probability vector. The SFT population update (7), expressed in these coordinates, becomes j+1=j+η2Δ(j),Δ():=2[(q2(⋅∣)−softmax())⊤]. _j+1= _j+ _2 ( _j), ( ):=E_2 [(q_2(· )- ( r))r ]. (C.25) Here Δ⁡() ( ) denotes the population SFT update direction at . It has the same shape as : its first and second columns give the update directions for ζa _a and ωa _a, respectively. Explicitly, its row indexed by a has entries Δζa()=2[(q2(a∣)−πa)z],Δωa()=2[(q2(a∣)−πa)y], _ _a( )=E_2[(q_2(a )- _a)z], _ _a( )=E_2[(q_2(a )- _a)y], where πa=[softmax⁡()]a _a=[ ( r)]_a. At 1W_1 (represented by 0 _0), recall that (b)=M−1∑a∈ba α^(b)=M^-1 _a _bw_a is the class-b semantic channel; see (9). Theorem 1(i) shows that the population update of (1)−(0) α^(1)- α^(0) is parallel to u. From (C.2), a single SFT step on a prompt x updates the two components of the semantic channel by (b)+−(b)=ηM(b=b⋆()−Pb(,)),b∈0,1. α^(b)+- α^(b)= ηM (1\b=b (x)\-P_b(W,x) )x, b∈\0,1\. Summing over b, we have (0)++(1)+ α^(0)++ α^(1)+ =(0)+(1)+ηM∑b∈0,1(b=b⋆()−Pb(,)) = α^(0)+ α^(1)+ ηM _b∈\0,1\ (1\b=b (x)\-P_b(W,x) )x =(0)+(1)+ηM(b⋆()=0+b⋆()=1−P0(,)−P1(,)) = α^(0)+ α^(1)+ ηM (1\b (x)=0\+1\b (x)=1\-P_0(W,x)-P_1(W,x) )x =(0)+(1), = α^(0)+ α^(1), where the indicators add to 11 because exactly one class is correct, and P0+P1=1P_0+P_1=1 is by definition (12), so (0)+(1) α^(0)+ α^(1) is unchanged by every SFT step. Writing (b)=12[((0)+(1))±((1)−(0))] α^(b)= 12 [( α^(0)+ α^(1))±( α^(1)- α^(0)) ], the sum does not move and the difference moves only along u, so each semantic channel’s population update at 1W_1 is parallel to u. Since ⟂^2u μ_2, its projection onto ^2 μ_2 is zero. By (9), (b) α^(b) is the average of aw_a over a∈ba _b, so its update is the average of the updates of those token parameters. Thus, for the first task-22 step from 1W_1, ⟨2[(b)+−(b)],^2⟩ _2[ α^(b)+- α^(b)],\, μ_2 =1M∑a∈b⟨2[a+−a],^2⟩ = 1M _a _b _2[w_a^+-w_a],\, μ_2 =η2M∑a∈bΔωa(0). = _2M _a _b _ _a( _0). Here each Δωa(0) _ _a( _0) describes the update direction of one token’s ωa _a coefficient; summing and dividing by M gives the corresponding direction for the semantic channel. The left-hand side is zero, so 1M∑a∈bΔωa(0)=0,b∈0,1. 1M _a _b _ _a( _0)=0, b∈\0,1\. (C.26) Individual rows can still move along ^2 μ_2 through their within-class deviations a=a−(σ⁡(a)) β_a=w_a- α^(σ(a)), and Theorem 2(i) quantifies this: since ⁡(1)= β(W_1)=0, (16) with t=2t=2 gives, for a∈ba _b, 2[a+−a]=η2(ρ2,b(a)−1M)(∥2∥2^2+2b−12π).E_2[ β_a^+- β_a]= _2 ( _2,b(a)- 1M ) ( μ_2 2 μ_2+ 2b-1 2πu ). As a=(σ⁡(a))+aw_a= α^(σ(a))+ β_a and the α-step has no ^2 μ_2-component, the ^2 μ_2-component of the row step is that of the β-step. Using that the initial rows are parallel to u, ωa(0)=0,Δωa(0)=∥2∥2(ρ2,σ⁡(a)(a)−1M),a∈. _a( _0)=0, _ _a( _0)= μ_2 2 ( _2,σ(a)(a)- 1M ), a . (C.27) Iterating (C.25) and using τ2=k2η2 _2=k_2 _2, k2−0−τ2Δ(0)=η2∑j=0k2−1(Δ⁡(j)−Δ⁡(0)). _k_2- _0- _2 ( _0)= _2 _j=0^k_2-1 ( ( _j)- ( _0) ). (C.28) Two bounds on the update map Δ defined in (C.25) control the right-hand side. First, |q2(a∣)−πa|≤1|q_2(a )- _a|≤ 1, 2|z|≤1E_2|z|≤ 1, and 2|y|≤∥2∥+1E_2|y|≤ μ_2 +1 (as y∼N⁡(∥2∥,1)y N( μ_2 ,1)) bound each entry of Δ , so summing squares over the 2M2M rows, sup∥Δ⁡()∥F≤2M(1+(∥2∥+1)2)≤2M(∥2∥+2). _ ( ) _F≤ 2M (1+( μ_2 +1)^2 )≤ 2M\,( μ_2 +2). (C.29) Second, the teacher term q2(⋅∣)q_2(· ) does not depend on and therefore cancels when we subtract the two update maps: Δ⁡()−Δ⁡(′)=2[(softmax⁡(′)−softmax⁡())⊤]. ( )- ( )=E_2\! [ ( ( r)- ( r) )r ]. Taking the Frobenius norm, with Jensen’s inequality we have ∥Δ⁡()−Δ⁡(′)∥F ( )- ( ) _F ≤2[∥(softmax⁡(′)−softmax⁡())⊤∥F] _2\! [ ( ( r)- ( r) )r _F ] =2[∥softmax⁡(′)−softmax⁡()∥∥] =E_2\! [ ( r)- ( r) \, ] ≤2[∥(−′)∥∥] _2 [ ( - )r \, ] ≤∥−′∥F2∥2, ≤ - _F\,E_2 ^2, where the third inequality uses the 11-Lipschitz property of softmax (Lemma B.6). Finally, since =(z,y)⊤r=(z,y) , with z∼N⁡(0,1)z N(0,1) and y∼N⁡(∥2∥,1)y N( μ_2 ,1) under task 22, 2∥2=2z2+2y2=1+(1+∥2∥2)=∥2∥2+2,E_2 ^2=E_2z^2+E_2y^2=1+ (1+ μ_2 ^2 )= μ_2 ^2+2, where we used X2=Var⁡(X)+(X)2EX^2=Var(X)+(EX)^2 for each coordinate. Thus ∥Δ⁡()−Δ⁡(′)∥F≤(∥2∥2+2)∥−′∥F. ( )- ( ) _F≤( μ_2 ^2+2) - _F. (C.30) Summing the update recursion in (C.25) over the first j steps gives j−0=η2∑ℓ=0j−1Δ⁡(ℓ). _j- _0= _2 _ =0^j-1 ( _ ). Using the triangle inequality and (C.29), we obtain ∥j−0∥F≤η2∑ℓ=0j−1∥Δ⁡(ℓ)∥F≤jη22M(∥2∥+2). _j- _0 _F≤ _2 _ =0^j-1 ( _ ) _F≤ j _2 2M\,( μ_2 +2). (C.31) Substituting (C.30) and (C.31) into (C.28) gives ∥k2−0−τ2Δ(0)∥F _k_2- _0- _2 ( _0) _F ≤η2∑j=0k2−1(∥2∥2+2)jη22M(∥2∥+2) ≤ _2 _j=0^k_2-1( μ_2 ^2+2)\,j _2 2M\,( μ_2 +2) ≤12(∥2∥2+2)2M(∥2∥+2)τ22:=Cpopτ22, ≤ 12( μ_2 ^2+2) 2M\,( μ_2 +2) _2^2:=C_pop _2^2, where Cpop:=12(∥2∥2+2)2M(∥2∥+2)C_pop:= 12( μ_2 ^2+2) 2M\,( μ_2 +2). For any matrix A, each entry satisfies |Aa,2|≤(∑b,c|Ab,c|2)1/2=∥F|A_a,2|≤ ( _b,c|A_b,c|^2 )^1/2= _F. Applying this inequality to the (a,2)(a,2) entry of k2−0−τ2Δ(0) _k_2- _0- _2 ( _0) gives |ωa(2)−ωa(0)−τ2Δωa(0)|≤Cpopτ22. | _a(W_2)- _a( _0)- _2 _ _a( _0) |≤ C_pop _2^2. Substituting (C.27) yields |ωa(2)−τ2∥2∥2(ρ2,σ⁡(a)(a)−1M)|≤Cpopτ22,a∈. | _a(W_2)- _2 μ_2 2 ( _2,σ(a)(a)- 1M ) |≤ C_pop _2^2, a . (C.32) Choose a∘∈argmaxa∈1⁡ρ2,1(a)a ∈ _a _1 _2,1(a). For a′∈0a _0, assumption (b) gives ρ2,1(a∘)−ρ2,0(a′)≥Δρ _2,1(a )- _2,0(a )≥ _ρ. Since σ⁡(a∘)=1σ(a )=1 and σ⁡(a′)=0σ(a )=0, (C.32) gives a lower bound for ωa∘(2) _a (W_2) and an upper bound for ωa′(2) _a (W_2): ωa∘(2) _a (W_2) ≥τ2∥2∥2(ρ2,1(a∘)−1M)−Cpopτ22, ≥ _2 μ_2 2 ( _2,1(a )- 1M )-C_pop _2^2, (C.33) ωa′(2) _a (W_2) ≤τ2∥2∥2(ρ2,0(a′)−1M)+Cpopτ22. ≤ _2 μ_2 2 ( _2,0(a )- 1M )+C_pop _2^2. Subtracting the upper bound from the lower bound in (C.33) gives ωa∘(2)−ωa′(2) _a (W_2)- _a (W_2) ≥τ2∥2∥2[(ρ2,1(a∘)−1M)−(ρ2,0(a′)−1M)]−2Cpopτ22 ≥ _2 μ_2 2 [ ( _2,1(a )- 1M )- ( _2,0(a )- 1M ) ]-2C_pop _2^2 =τ2∥2∥2(ρ2,1(a∘)−ρ2,0(a′))−2Cpopτ22 = _2 μ_2 2 ( _2,1(a )- _2,0(a ) )-2C_pop _2^2 ≥τ2∥2∥Δρ2−2Cpopτ22. ≥ _2 μ_2 _ρ2-2C_pop _2^2. Set τ0:=∥2∥Δρ8Cpop=∥2∥Δρ4(∥2∥2+2)2M(∥2∥+2). _0:= μ_2 _ρ8C_pop= μ_2 _ρ4( μ_2 ^2+2) 2M\,( μ_2 +2). (C.34) For τ2≤τ0 _2≤ _0, 2Cpopτ22≤2Cpopτ0τ2=τ2∥2∥Δρ/42C_pop _2^2≤ 2C_pop _0 _2= _2 μ_2 _ρ/4, so for every a′∈0a _0, ωa∘(2)−ωa′(2)≥τ2∥2∥Δρ4>0. _a (W_2)- _a (W_2)≥ _2 μ_2 _ρ4>0. (C.35) Step 3: an error region for both tasks. We construct a single error region and evaluate its probability under both task distributions. Let κ:=4(∥Δ(1)∥+2τ2)∥2∥Δρτ2,E:=z⁡<0,y>κ|z|.κ:= 4( α(W_1) +2 _2) μ_2 _ρ _2, E:=\z<0,\;y>κ|z|\. (C.36) Fix ∈Ex∈ E and a′∈0a _0, and evaluate all coefficients at 2W_2. By the triangle inequality and (C.24) with j=k2j=k_2, |ζa∘−ζa′| | _a - _a | ≤|ζa∘|+|ζa′| ≤| _a |+| _a | (C.37) ≤2(12∥Δ(1)∥+τ2)=∥Δ(1)∥+2τ2. ≤ 2 ( 12 α(W_1) + _2 )= α(W_1) +2 _2. Since κ>0κ>0 and ∈Ex∈ E, we have y>κ|z|>0y>κ|z|>0. Using (C.23), (C.35), (C.37), and (C.36), together with the definitions of z and y, ⟨a∘−a′,⟩ _a -w_a ,\,x =(ζa∘−ζa′)z+(ωa∘−ωa′)y =( _a - _a )z+( _a - _a )y ≥−|ζa∘−ζa′||z|+(ωa∘−ωa′)y ≥-| _a - _a |\,|z|+( _a - _a )y ≥−|ζa∘−ζa′||z|+τ2∥2∥Δρ4y ≥-| _a - _a |\,|z|+ _2 μ_2 _ρ4\,y ≥−(∥Δ(1)∥+2τ2)|z|+τ2∥2∥Δρ4y ≥-( α(W_1) +2 _2)|z|+ _2 μ_2 _ρ4\,y =τ2∥2∥Δρ4(y−κ|z|)>0. = _2 μ_2 _ρ4 (y-κ|z| )>0. So on E the highest-scoring token lies in 1A_1 and b^2()=1 b_W_2(x)=1, while z<0z<0 means b⋆()=0b (x)=0. This pointwise conclusion holds under either task distribution, so ℒr(2)≥ℙr(E)L_r(W_2) _r(E) for r∈1,2r∈\1,2\. Under task r, Lemma B.3 gives independent coordinates z∼N⁡(0,1)z N(0,1) and y∼N⁡(⟨r,^2⟩,1)y N( μ_r,\, μ_2 ,1). Integrating over z<0z<0 with the substitution z=−tz=-t, t>0t>0, yields ℒr(2)≥ℙr(E) _r(W_2) _r(E) =∫0∞ϕ⁡(t)ℙr(y>κt)t = _0^∞φ(t)\,P_r (y>κ t )\,dt (C.38) =∫0∞ϕ(t)Φ(⟨r,^2⟩−κt)dt>0,r∈1,2, = _0^∞φ(t) \! ( μ_r,\, μ_2 -κ t )\,dt>0, r∈\1,2\, where strict positivity follows because the integrand is positive for every t>0t>0. Substituting the definitions of κ and ^2=2/∥2∥ μ_2= μ_2/ μ_2 into (C.38) proves part (i) for both tasks. ∎ C.4 Proof of Theorem 4 Proof of Theorem 4. Step 1: conditional sampling and integrability. For either method, let ℱsF_s denote its training history strictly before step s, including the previous teacher draws for SFT. For the trajectory under consideration, write s:=⁡(s)W_s:=W(s) and s:=t⁡(s)m_s:= μ_t(s). Set ⟂:=p−⊤P_ :=I_p-uu and define s:=s⟂=s⟂,s:=⟂s,zs:=⟨,s⟩,s:=(Δ(s))⟂.R_s:=W_s =W_sP_ , _s:=P_ x_s, z_s:= ,\,x_s , _s:=( α(s)) . (C.39) Thus ∥s∥F2=∥s⟂∥F2 _s _F^2= _s _F^2. The matrix sW_s is ℱsF_s-measurable, whereas the current prompt is fresh. By Lemma B.3 and s⟂m_s , conditionally on ℱsF_s, zs∼N(0,1),s∼N(s,⟂),zs⟂s, z_s N(0,1), _s N(m_s,P_ ), z_s \!\!\! _s, (C.40) [s∣ℱs]=s,[∥s∥2∣ℱs]=tr(⟂)+∥s∥2=p−1+∥s∥2. [y_s _s]=m_s, [ _s ^2 _s]=tr(P_ )+ _s ^2=p-1+ _s ^2. All expectations used below are finite. Indeed, for SFT the coefficient vector in (6) has norm at most 2 2, since it is the difference of two probability vectors. For RFT, each coefficient has absolute value at most πs(a∣s) _W_s(a _s), because (1) gives 0≤R,R¯≤10≤ R, R≤ 1; hence the coefficient vector has norm at most 11. In both cases, ∥s∥F≤2η∑j=0s−1∥j∥. _s _F≤ 2η _j=0^s-1 _j . The right-hand side has finite moments of every order for finite s, as the prompts are Gaussian and the task sequence is fixed. Step 2: the exact semantic–style energy decomposition. For either update in (6), the sum of all token parameters is preserved. In SFT, ∑a∈(a+−a)=η(∑aa=a⋆−∑aπ(a∣))=. _a (w_a^+-w_a)=η ( _a1\a=a \- _a _W(a ) )x=0. In RFT, the definition of R¯ R_W following (6) gives ∑a∈(a+−a) _a (w_a^+-w_a) =η⁡(∑aπ(a∣)R(a,)−R¯()∑aπ(a∣)) =η ( _a _W(a )R(a,x)- R_W(x) _a _W(a ) )x =η⁡(R¯()−R¯())=. =η( R_W(x)- R_W(x))x=0. Since 0=W_0=0, the definition of class means in (9) therefore implies M⁡((0)(s)+(1)(s))=∑a(s)=,(1)⟂(s)=s2,(0)⟂(s)=−s2.M ( α^(0)(s)+ α^(1)(s) )= _aw_a(s)=0, α^(1) (s)= v_s2, α^(0) (s)=- v_s2. (C.41) Furthermore, (9) gives a⟂=(b)⟂+a⟂w_a = α^(b) + β_a for a∈ba _b and ∑a∈ba⟂= _a _b β_a =0. Consequently, for each class, ∑a∈b∥a⟂∥2 _a _b _a ^2 =M∥(b)⟂∥2+2⟨(b)⟂,∑a∈ba⟂⟩+∑a∈b∥a⟂∥2 =M α^(b) ^2+2 α^(b) ,\, _a _b β_a + _a _b β_a ^2 =M∥(b)⟂∥2+∑a∈b∥a⟂∥2. =M α^(b) ^2+ _a _b β_a ^2. Summing over the two classes and using (C.41) yields the pathwise identity ∥s∥F2 _s _F^2 =M∑b=01∥(b)⟂(s)∥2+∑a∈∥a⟂(s)∥2 =M _b=0^1 α^(b) (s) ^2+ _a β_a (s) ^2 (C.42) =M2∥s∥2+∥⟂(s)∥F2. = M2 _s ^2+ β (s) _F^2. This proves (17) for both methods. Step 3: the SFT update and the teacher’s conditional mean. We first prove part (i). Write s=(πs(a∣s))a∈ π_s=( _W_s(a _s))_a as a column vector, and let as⋆∈ℝ2Me_a_s ^2M be the one-hot vector of the teacher token. Right-multiplying the SFT update in (6) by ⟂P_ gives s+1=s+η⁡(as⋆−s)s⊤.R_s+1=R_s+η(e_a_s - π_s)y_s . (C.43) Under (2), the vector t⁡(s)(zs)=(qt⁡(s)(a∣s))a∈q_t(s)(z_s)=(q_t(s)(a _s))_a depends on the prompt only through the sign of zsz_s. The fresh teacher draw therefore satisfies [as⋆∣ℱs,s]=t⁡(s)(zs).E[e_a_s _s,x_s]=q_t(s)(z_s). Using iterated conditional expectation and then (C.40), we obtain ⁡[as⋆s⊤∣ℱs] [e_a_s y_s _s] =⁡[t⁡(s)(zs)s⊤∣ℱs]=¯ss⊤, =E[q_t(s)(z_s)y_s _s]= q_sm_s , (C.44) ¯s q_s :=[t⁡(s)(zs)],(¯s)a=12ρt⁡(s),σ⁡(a)(a). :=E[q_t(s)(z_s)], ( q_s)_a= 12 _t(s),σ(a)(a). The factor 1/21/2 follows from ℙ⁡(zs>0)=ℙ⁡(zs<0)=1/2P(z_s>0)=P(z_s<0)=1/2. Step 4: Gaussian integration by parts for the SFT policy term. For a fixed W and coordinate j, differentiating the softmax gives ∂π(a∣)∂xj=π(a∣)(wa,j−∑a′π(a′∣)wa′,j). ∂ _W(a )∂ x_j= _W(a ) (w_a,j- _a _W(a )w_a ,j ). Thus its Jacobian, with one row per token, is (diag⁡()−⊤)(diag( π)- π π )W. For ∼N⁡(s,p)x N(m_s,I_p), integration by parts coordinatewise yields ⁡[(xj−ms,j)π(a∣)]=⁡[∂π(a∣)/∂xj]E[(x_j-m_s,j) _W(a )]=E[∂ _W(a )/∂ x_j]; the boundary term vanishes because the softmax is bounded, and its derivatives are bounded for fixed W. Define ¯s:=⁡[s∣ℱs] π_s:=E[ π_s _s]. Conditional on ℱsF_s, this identity and right multiplication by ⟂P_ give, since ⟂s=sP_ m_s=m_s, ⁡[ss⊤∣ℱs] [ π_sy_s _s] =¯ss⊤+ss, = π_sm_s +H_sR_s, (C.45) s _s :=⁡[diag⁡(s)−ss⊤|ℱs]⪰0. :=E [diag( π_s)- π_s π_s \, |\,F_s ] 0. Indeed, for any ∈ℝ2Mv ^2M and probability vector π, ⊤(diag⁡()−⊤)=∑aπava2−(∑aπava)2≥0,v (diag( π)- π π )v= _a _av_a^2- ( _a _av_a )^2≥ 0, so taking the conditional expectation preserves positive semidefiniteness. Step 5: the SFT energy recursion. The Frobenius inner product is ⟨,⟩F=tr⁡(⊤) ,V _F=tr(U V), and ∥⊤∥F2=∥2∥2 _F^2= ^2 ^2. Expanding (C.43) therefore gives ∥s+1∥F2= _s+1 _F^2= ∥s∥F2+2η⟨s,(as⋆−s)s⊤⟩F _s _F^2+2η _s,(e_a_s - π_s)y_s _F +η2∥as⋆−s∥2∥s∥2. +η^2 _a_s - π_s ^2 _s ^2. Equations (C.44) and (C.45), and the ℱsF_s-measurability of sR_s, yield the exact identity ⁡[∥s+1∥F2∣ℱs]= [ _s+1 _F^2 _s]= ∥s∥F2−2ηtr⁡(s⊤ss) _s _F^2-2 (R_s H_sR_s) (C.46) +2η⟨s,(¯s−¯s)s⊤⟩F +2η _s,( q_s- π_s)m_s _F +η2[∥as⋆−s∥2∥s∥2|ℱs]. +η^2E [ _a_s - π_s ^2 _s ^2\, |\,F_s ]. Since s⪰0H_s 0, each column r of sR_s satisfies ⊤s≥0r H_sr≥ 0; summing over columns shows that the trace in (C.46) is nonnegative. Both ¯s q_s and ¯s π_s are probability vectors, so their squared norms are at most 11 and their inner product is nonnegative. Thus ∥¯s−¯s∥≤2 q_s- π_s ≤ 2, and Cauchy–Schwarz gives ⟨s,(¯s−¯s)s⊤⟩F≤2∥s∥F∥s∥. _s,( q_s- π_s)m_s _F≤ 2\, _s _F _s . Moreover, for every demonstrated token a and probability vector π, ∥a−∥2=(1−πa)2+∑a′≠aπa′2≤(1−πa)2+(∑a′≠aπa′)2=2(1−πa)2≤2. _a- π ^2=(1- _a)^2+ _a ≠ a _a ^2≤(1- _a)^2+ ( _a ≠ a _a )^2=2(1- _a)^2≤ 2. Substituting these estimates and (C.40) into (C.46) gives ⁡[∥s+1∥F2∣ℱs]≤ [ _s+1 _F^2 _s]≤ ∥s∥F2+22η∥s∥∥s∥F _s _F^2+2 2η _s _s _F (C.47) +2η2(p−1+∥s∥2). +2η^2 (p-1+ _s ^2 ). Step 6: iterate the SFT recursion. Set es:=∥s∥F2e_s:=E _s _F^2. The means sm_s are deterministic. Taking expectations in (C.47) and using ∥s∥F≤esE _s _F≤ e_s yields es+1 e_s+1 ≤es+22η∥s∥es+2η2∥s∥2+2η2(p−1) ≤ e_s+2 2η _s e_s+2η^2 _s ^2+2η^2(p-1) (C.48) =(es+2η∥s∥)2+2η2(p−1). = ( e_s+ 2η _s )^2+2η^2(p-1). To iterate this bound, let As:=2η∑j=0s−1∥j∥,Vs:=2η2s(p−1).A_s:= 2η _j=0^s-1 _j , V_s:=2η^2s(p-1). We claim es≤As+Vs e_s≤ A_s+ V_s. The claim holds at s=0s=0. For the induction step, use the elementary inequality (A+V+c)2+d≤A+c+V+d,A,V,c,d≥0. (A+ V+c)^2+d≤ A+c+ V+d, A,V,c,d≥ 0. Indeed, the square of the right-hand side minus the square of the left-hand side is 2(A+c)(V+d−V)≥02(A+c)( V+d- V)≥ 0, and both sides are nonnegative. Apply it with A=AsA=A_s, V=VsV=V_s, c=2η∥s∥c= 2η _s , and d=2η2(p−1)d=2η^2(p-1). By (C.48) and the induction hypothesis, es+1 e_s+1 ≤(As+Vs+c)2+d ≤ (A_s+ V_s+c)^2+d ≤As+c+Vs+d=As+1+Vs+1. ≤ A_s+c+ V_s+d=A_s+1+ V_s+1. Evaluating the resulting bound at s=Ns=N gives ∥N∥F2≤2η2(N⁡(p−1)+∑s=0N−1∥t⁡(s)∥)2.E _N _F^2≤ 2η^2 ( N(p-1)+ _s=0^N-1 μ_t(s) )^2. (C.49) By (C.39), this is precisely (19), proving part (i). Step 7: reduce RFT to its semantic channel. We now consider the RFT trajectory and prove part (i). Since 0=W_0=0, Theorem 2(i) gives ⁡(s)= β(s)=0 for every step, for each realization of the prompts. By (3) and (12), equal class sizes then give P1(s,)=Me⟨(1)(s),⟩Me⟨(0)(s),⟩+Me⟨(1)(s),⟩=11+e−⟨Δ(s),⟩.P_1(W_s,x)= Me α^(1)(s),\,x Me α^(0)(s),\,x +Me α^(1)(s),\,x = 11+e^- α(s),\,x . Define cs:=⟨Δ(s),⟩,λ:=2ηM,f⁡(r):=er(1+er)2=14cosh2⁡(r/2).c_s:= α(s),\,u , λ:= 2ηM, f(r):= e^r(1+e^r)^2= 14 ^2(r/2). (C.50) In particular, f is even and 0<f⁡(r)≤1/40<f(r)≤ 1/4 for every r. Since P0=1−P1P_0=1-P_1, either value of the correct class satisfies Pb⋆(s,s)(1−Pb⋆(s,s))=f⁡(cszs+⟨s,s⟩).P_b (W_s,x_s) (1-P_b (W_s,x_s) )=f\! (c_sz_s+ _s,\,y_s ). (C.51) Here we used (C.39) to write Δ(s)=cs+s α(s)=c_su+v_s and s=zs+sx_s=z_su+y_s. Let εs:=2b⋆(s)−1 _s:=2b (x_s)-1. By (1), εs=sign⁡(zs) _s= (z_s) almost surely and εszs=|zs| _sz_s=|z_s|; the event zs=0z_s=0 has probability zero. Substituting (C.51) into the exact semantic update (13), and projecting onto u and ⟂u , gives cs+1 c_s+1 =cs+λ|zs|f(cszs+⟨s,s⟩), =c_s+λ|z_s|f\! (c_sz_s+ _s,\,y_s ), (C.52) s+1 _s+1 =s+λεsf(cszs+⟨s,s⟩)s. =v_s+λ _sf\! (c_sz_s+ _s,\,y_s )y_s. The first line and c0=0c_0=0 imply cs+1≥cs≥0c_s+1≥ c_s≥ 0 for all s, without any upper restriction on η. Step 8: the conditional RFT drift has a nonpositive radial component. Condition on ℱsF_s, so that cs≥0c_s≥ 0 and sv_s are fixed. For c≥0c≥ 0 and t∈ℝt , define hc(t) h_c(t) :=Z∼N⁡(0,1)[sign⁡(Z)f(cZ+t)] :=E_Z N(0,1)[ (Z)f(cZ+t)] (C.53) =∫0∞ϕ⁡(z)[f⁡(cz+t)−f⁡(cz−t)]dz, = _0^∞φ(z) [f(cz+t)-f(cz-t) ]\,dz, where ϕφ is the standard normal density. The second line follows by splitting the integral at zero, substituting z↦−z -z in the negative half, and using the evenness of both ϕφ and f. Differentiating (C.50) gives f′(r)=−f⁡(r)tanh⁡(r/2)f (r)=-f(r) (r/2), so f is nonincreasing on [0,∞)[0,∞). For c,z,t≥0c,z,t≥ 0, the inequality |cz+t|≥|cz−t||cz+t|≥|cz-t| therefore implies f⁡(cz+t)≤f⁡(cz−t)f(cz+t)≤ f(cz-t). Equation (C.53) then yields hc(t)≤0h_c(t)≤ 0 for t≥0t≥ 0. The same equation gives hc(−t)=−hc(t)h_c(-t)=-h_c(t), hence thc(t)≤0for every c≥0 and t∈ℝ.t\,h_c(t)≤ 0 every c≥ 0 and t . (C.54) Write the conditional mean of the unscaled off-axis increment as s:=⁡[εsf(cszs+⟨s,s⟩)s|ℱs].g_s:=E\! [ _sf\! (c_sz_s+ _s,\,y_s )y_s\, |\,F_s ]. By the conditional independence in (C.40), we may first average over zsz_s while holding sy_s fixed. Using (C.53) then gives ⟨s,s⟩ _s,\,g_s =⁡[⟨s,s⟩εsf(cszs+⟨s,s⟩)|ℱs] =E\! [ _s,\,y_s \, _sf\! (c_sz_s+ _s,\,y_s )\, |\,F_s ] (C.55) =⁡[⟨s,s⟩hcs(⟨s,s⟩)|ℱs]≤0, =E\! [ _s,\,y_s \,h_c_s\! ( _s,\,y_s )\, |\,F_s ]≤ 0, where the last inequality applies (C.54) pointwise in sy_s. This argument allows any s⟂m_s ; it does not require zero task means. Step 9: the RFT energy recursion and final bound. Expanding the second line of (C.52), taking conditional expectations, and using εs2=1 _s^2=1 gives ⁡[∥s+1∥2∣ℱs]= [ _s+1 ^2 _s]= ∥s∥2+2λ⟨s,s⟩ _s ^2+2λ _s,\,g_s (C.56) +λ2[f(cszs+⟨s,s⟩)2∥s∥2|ℱs] +λ^2E\! [f\! (c_sz_s+ _s,\,y_s )^2 _s ^2\, |\,F_s ] ≤ ≤ ∥s∥2+λ216(p−1+∥s∥2) _s ^2+ λ^216(p-1+ _s ^2) = = ∥s∥2+η24M2(p−1+∥s∥2). _s ^2+ η^24M^2(p-1+ _s ^2). The inequality uses (C.55), f2≤1/16f^2≤ 1/16 from (C.50), and the second moment in (C.40). Taking expectations and summing (C.56) from s=0s=0 to N−1N-1, with 0=v_0=0, yields ∥N∥2≤η24M2∑s=0N−1(p−1+∥s∥2).E _N ^2≤ η^24M^2 _s=0^N-1(p-1+ _s ^2). (C.57) Finally, ⁡(N)= β(N)=0 and (C.42) give ∥RFT⟂(N)∥F2 _RFT (N) _F^2 =M2∥N∥2 = M2E _N ^2 ≤η28M(N⁡(p−1)+∑s=0N−1∥t⁡(s)∥2), ≤ η^28M (N(p-1)+ _s=0^N-1 μ_t(s) ^2 ), which is (18) and proves part (i). Both parts include N=0N=0, with empty sums and zero initial energy. Fresh sampling is essential to the conditional calculations in (C.40) and the subsequent expectations; the proof does not directly apply to reuse of a fixed prompt sample. For RFT, exact per-prompt gradients are also essential to the pathwise style invariance used in Step 7; sampled action gradients are discussed separately in Appendix D.1. ∎ Appendix D Additional results D.1 Exact and sampled policy gradients The invariance in Theorem 2(i) concerns the exact per-prompt gradient (6). Here “exact” means that the expectation over actions is evaluated by summing over all tokens at the current policy and prompt. When this summation is computationally impractical, REINFORCE provides an unbiased gradient estimate using actions sampled from the current policy. This subsection explains how action sampling may change the style dynamics. An unbiased gradient estimate. Fix W and x, and draw A∼π(⋅∣)A _W(· ). The REINFORCE estimator without a baseline (Williams, 1992) gives the update a+=a+η^a(A),^a(A)=R(A,)(a=A−π(a∣)).w_a^+=w_a+η\, g_a(A), g_a(A)=R(A,x) (1\a=A\- _W(a ) )x. (D.1) All expectations below are conditional on the current ,W,x and average only over the sampled action. Since A[R(A,)a=A]=π(a∣)R(a,)E_A[R(A,x)1\a=A\]= _W(a )R(a,x), we have A[^a(A)]=π(a∣)(R⁡(a,)−R¯()).E_A[ g_a(A)]= _W(a ) (R(a,x)- R_W(x) )x. (D.2) Thus the sampled update agrees with the RFT update in (6) in conditional mean. Style noise from a single sampled action. Subtracting the mean update within a’s class gives Δa β_a :=a+−a := β_a^+- β_a (D.3) =ηR(A,)[a=A−1Mσ(A)=σ(a)−δa(,)], =η R(A,x) [1\a=A\- 1M1\σ(A)=σ(a)\- _a(W,x) ]x, where δa _a is defined in (C.18). When = β=0, δa=0 _a=0 and (D.2) implies A[Δa]=E_A[ β_a]=0. Nevertheless, if M>1M>1 and the sampled token is rewarded, its style increment is η⁡(1−1/M)η(1-1/M)x, while every other token in the same class receives −η/M- /M. Tokens in the other class receive zero style increment. For η>0η>0 and ≠x 0, a rewarded step produces +≠ β^+ 0. An unrewarded sample produces no update for this estimator. Let Δ β collect the style increments over all tokens. When = β=0, the conditional squared style increment is A[∥Δ∥F2]=η2Pb⋆()(,)(1−1M)∥2.E_A [ β _F^2 ]=η^2P_b (x)(W,x) (1- 1M ) ^2. (D.4) Indeed, a rewarded sample occurs with probability Pb⋆()(,)P_b (x)(W,x), and its squared coefficients sum to (1−1/M)2+(M−1)/M2=1−1/M(1-1/M)^2+(M-1)/M^2=1-1/M. Since the conditional mean increment is zero when = β=0, (D.4) is also the total conditional variance. The corresponding off-axis expression replaces x by ⟂x . Hence a zero mean style increment can coexist with positive expected style energy after one step. Averaging actions and following the trajectory. If each update averages K independent action-gradient estimates computed at the same ,W,x, its conditional mean is the RFT update in (6), while the variance in (D.4) is divided by K. Finite K therefore reduces style noise but does not guarantee that every update preserves = β=0. Once a sampled trajectory develops ≠ β 0, the zero mean style-increment statement above no longer applies: subsequent conditional means follow the exact gradient at the new, generally nonuniform policy. Theorem 2(i) consequently does not imply zero style drift along sampled trajectories. We compare REINFORCE updates with different numbers of sampled actions per prompt against exact RFT in Appendix E.2 (Figure E.4(b,d)). D.2 Limiting behavior under continued task-2 training Theorem 3 establishes positive SFT semantic error on both tasks over a finite training interval. We now consider continued training on task 22. In the unregularized population setting, sufficiently small fixed steps eventually drive the semantic errors on both tasks to zero, even though the teacher’s style preferences can cause errors at intermediate times. Proposition D.1 (Vanishing semantic error under continued population SFT). Under the setup of Section 2.1, let (0)=1W^(0)=W_1 be any finite checkpoint, and run population SFT on task 22 with a fixed step size satisfying 0<η2≤21+∥2∥2.0< _2≤ 21+ μ_2 ^2. (D.5) Then, as k→∞k→∞ , ℒ1((k))⟶0,ℒ2((k))⟶0.L_1(W^(k)) 0, _2(W^(k)) 0. (D.6) In particular, from the checkpoint in Theorem 3, ℱSFT(2)⟶0,SFT(2)⟶0,F^(2)_SFT 0, ^(2)_SFT 0, whereas RFT keeps both quantities identically zero throughout training. Proof. Let f2f_2 denote the task-22 population token cross-entropy, and let f2,∗f_2,* be its infimum over finite parameter matrices: f2():=2[−∑a∈q2(a∣)logπ(a∣)],f2,∗:=inff2().f_2(W):=E_2\! [- _a q_2(a ) _W(a ) ], f_2,*:= _Vf_2(V). We first show that approaching this infimum forces semantic error to vanish, and then verify that population optimization approaches it. Separating class and style losses. For a∈ba _b, write π(a∣,b):=π(a∣)/Pb(,) _W(a ,b):= _W(a )/P_b(W,x). This conditional policy depends only on the style channel β. Since the teacher is always semantically correct, f2() f_2(W) =C2()+S2(), =C_2(W)+S_2( β), (D.7) C2() C_2(W) :=2[−log⁡Pb⋆()(,)], :=E_2[- P_b (x)(W,x)], S2() S_2( β) :=2[−∑a∈b⋆()ρ2,b⋆()(a)logπ(a∣,b⋆())]. :=E_2\! [- _a _b (x) _2,b (x)(a) _W(a ,b (x)) ]. Let S2,∗S_2,* be the infimum of S2S_2 over finite style channels whose rows sum to zero within each class. We claim that f2,∗=S2,∗f_2,*=S_2,*. The inequality f2,∗≥S2,∗f_2,*≥ S_2,* follows from C2≥0C_2≥ 0. Conversely, fix such a style channel and set a[s]=a+(2σ(a)−1)s/2w_a^[s]= β_a+(2σ(a)-1)su/2 for s≥0s≥ 0. This leaves S2S_2 unchanged. With z=⟨,⟩z= ,\,x and b=b⋆()b=b (x), the class loss is −log⁡Pb([s],)=log⁡(1+e−s|z|∑a∈1−be⟨a,⟩∑a∈be⟨a,⟩).- P_b(W^[s],x)= \! (1+e^-s|z| _a _1-be β_a,\,x _a _be β_a,\,x ). It tends to zero almost surely and is bounded by its value at s=0s=0, which is integrable because the fixed linear logits have finite Gaussian first moments. Dominated convergence yields C2([s])→0C_2(W^[s])→ 0, hence f2,∗≤S2()f_2,*≤ S_2( β) for every finite style channel. Taking the infimum proves the claim and gives 0≤C2()≤f2()−f2,∗.0≤ C_2(W)≤ f_2(W)-f_2,*. (D.8) From objective values to semantic errors. If greedy decoding selects a wrong-class token, every correct-class token has probability at most that selected token’s probability, and thus at most the total wrong-class mass. Consequently, Pb(,)≤M⁡(1−Pb(,))P_b(W,x)≤ M(1-P_b(W,x)) on the error event, including ties resolved to the wrong class. The class loss there is at least log⁡(1+1/M) (1+1/M), so ℒ2()≤C2()log⁡(1+1/M)≤f2()−f2,∗log⁡(1+1/M).L_2(W)≤ C_2(W) (1+1/M)≤ f_2(W)-f_2,* (1+1/M). (D.9) Both tasks share the same semantic rule and hence the same error event E=:b^()≠b⋆()E_W=\x: b_W(x)≠ b (x)\. Writing =1−2 δ= μ_1- μ_2, the density ratio of their prompt distributions is r⁡()=exp⁡(⟨,−2⟩−12∥2),2[r()2]=e∥2.r(x)= \! ( δ,\,x- μ_2 - 12 δ ^2 ), _2[r(x)^2]=e δ ^2. Cauchy–Schwarz therefore gives, uniformly in W, ℒ1()=2[r()∈E]≤e∥1−2∥2/2ℒ2().L_1(W)=E_2[r(x)1\x∈ E_W\]≤ e μ_1- μ_2 ^2/2 L_2(W). (D.10) Thus any sequence with f2()→f2,∗f_2(W)→ f_2,* has vanishing semantic error on both tasks. Convergence of the population objective. The function f2f_2 is convex. For any perturbation matrix A, its Hessian satisfies D2f2()[,] D^2f_2(W)[A,A] =2[()⊤(diag⁡−⊤)()] =E_2\! [(Ax) (diag π- π π )(Ax) ] ≤122∥2≤12(1+∥2∥2)∥F2, ≤ 12E_2 ^2≤ 12(1+ μ_2 ^2) _F^2, where =π(⋅∣) π= _W(· ). Here the categorical covariance has operator norm at most 1/21/2, since its absolute row sums are 2πa(1−πa)≤1/22 _a(1- _a)≤ 1/2, and 2[⊤]=p+22⊤E_2[xx ]=I_p+ μ_2 μ_2 . Hence f2f_2 is L-smooth for L=(1+∥2∥2)/2L=(1+ μ_2 ^2)/2. Under (D.5), convexity and the descent inequality give, for every finite comparator V, f2((j+1))−f2()≤∥(j)−∥F2−∥(j+1)−∥F22η2.f_2(W^(j+1))-f_2(V)≤ ^(j)-V _F^2- ^(j+1)-V _F^22 _2. Summing and using monotonicity of the objective yields f2((k))−f2()≤∥1−∥F22kη2.f_2(W^(k))-f_2(V)≤ _1-V _F^22k _2. (D.11) First letting k→∞k→∞ and then taking the infimum over V proves f2((k))→f2,∗f_2(W^(k))→ f_2,* without assuming that a finite minimizer exists. Equations (D.9) and (D.10) now imply (D.6). When ℒ1(1)=0L_1(W_1)=0, forgetting equals the final task-11 error, and generalization error is the average of the two task errors, proving the asserted metric limits. ∎ What converges. The objective approaches its infimum and semantic errors vanish, but the parameters do not converge to a finite minimizer. Indeed, if U has rows (2σ(a)−1)/2(2σ(a)-1)u/2, then at every finite W, dsf2(+s)|s=0=−2[|z|(1−Pb⋆()(,))]<0. . ddsf_2(W+sU) |_s=0=-E_2\! [|z| (1-P_b (x)(W,x) ) ]<0. Thus no finite matrix attains f2,∗f_2,*, and any sequence approaching this infimum must have ∥F→∞ _F→∞; otherwise a bounded subsequence would yield a finite minimizer by continuity. The token cross-entropy need not tend to zero, and exact matching of the teacher’s style preferences is unnecessary for vanishing semantic error. The finite-time errors in Theorem 3 therefore do not persist at a positive level in this population limit. This does not assert monotone semantic improvement or zero error after finitely many steps. D.3 Numerical illustration of finite-time forgetting We illustrate the finite-time separation in Theorem 3 and the subsequent recovery described in Appendix D.2 by varying training time at a fixed step size. Setup. We take p=2p=2, M=2M=2, =(1,0)⊤u=(1,0) , and Gaussian task means 1=(0,2)⊤ μ_1=(0,2) and 2=(0,3)⊤ μ_2=(0,3) . The task-22 teachers are ρ2,0=(0.5,0.5) _2,0=(0.5,0.5) and ρ2,1=(0.99,0.01) _2,1=(0.99,0.01), giving Δρ=0.49 _ρ=0.49. Both methods start from the constructed checkpoint a=(2σ(a)−1)/2w_a=(2σ(a)-1)u/2, so that Δ= α=u and = β=0. This checkpoint already classifies both tasks perfectly because they share the same semantic rule. We then train only on task 22, using population updates with fixed step size η=10−3η=10^-3 for 10610^6 steps, up to τ=kη=1000τ=kη=1000. Here ℒ1L_1 equals forgetting, and (2)=(ℒ1+ℒ2)/2G^(2)=(L_1+L_2)/2. The guaranteed interval in (C.34) ends at τ0≈0.00334 _0≈ 0.00334; the chosen step size also satisfies the sufficient condition (D.5) for eventual recovery. Numerical evaluation. We evaluate the SFT teacher term analytically and its policy term by deterministic Gaussian quadrature, with 1212 Gauss–Legendre nodes on each of six intervals in positive z=⟨,⟩z= ,\,x and their negative reflections, and 4040 Gauss–Hermite nodes in the orthogonal coordinate (57605760 nodes in total; |z|≤11|z|≤ 11). RFT follows its scalar on-axis semantic recursion from Theorem 1(i), preserving zero style and zero semantic error. SFT errors are computed by conditioning on z, evaluating the Gaussian probability of the greedy decision region in the other coordinate, and integrating over z. Figure D.1: Two-task population dynamics at a fixed step size. (a) Old-task error, equal to forgetting from the common perfectly fitted checkpoint. (b) New-task error. (c) Off-axis style norm ∥⟂∥F β _F. The horizontal axis is logarithmic outside a small linear neighborhood of zero. Shading marks 0<τ≤τ00<τ≤ _0; the inset compares the first three SFT steps with the old-task error lower bound in Theorem 3 (dotted). Later times illustrate behavior beyond that guaranteed interval. Results. In Figure D.1, SFT’s old- and new-task errors rise from zero to approximately 13.4%13.4\% and 19.4%19.4\%, respectively, near τ=0.8τ=0.8, then decline to 1.25%1.25\% and 1.81%1.81\% at τ=1000τ=1000. The shaded steps lie inside the guaranteed interval and have positive old-task error above the theorem’s lower bound. RFT retains zero error on both tasks throughout. SFT’s error reduction occurs while its off-axis style norm remains nonzero (as the semantic margin grows). Thus this example exhibits substantial recovery without eliminating style drift. Appendix E Control and ablation experiments E.1 Control experiments We isolate the roles of task means and teacher preferences in population style drift. Unless specified below, all settings follow Section 4. Zero task means. Setting t= μ_t=0 removes population off-axis style drift. Both population methods then achieve negligible generalization error and forgetting (Figure E.1); their small residual off-axis energies reflect Monte Carlo prompt sampling. Finite-sample SFT still develops greater off-axis energy and a worse generalization–forgetting tradeoff than RFT, illustrating sampling effects even when population drift is absent. Uniform teachers. Keeping the nonzero task means of the main experiment, we set t,b=v_t,b=0 in (20), giving ρt,b(a)=1/M _t,b(a)=1/M. Removing teacher preferences also eliminates the population performance gap (Figure E.2). Population SFT now preserves = β=0 because its updates are constant within each semantic class, including under Monte Carlo prompt averaging. Its residual off-axis energy therefore comes from the semantic component. Finite-sample SFT still introduces style fluctuations by sampling individual teacher tokens. RFT is unchanged from Figure 2, since it does not use the teacher and the prompts and seeds are shared. Figure E.1: Zero task means. We set t= μ_t=0 for every task. As in Figure 2, panel (a) shows generalization error versus forgetting, and panel (b) shows final off-axis energy versus learning rate. Solid and dashed curves denote population and finite-sample updates, respectively. Figure E.2: Uniform teachers. We retain the nonzero Gaussian task means of Figure 2 and set ρt,b(a)=1/M _t,b(a)=1/M for every task and class. The layout and line styles match Figure E.1. E.2 Ablation experiments We examine sensitivity to three assumptions: Gaussian prompts, correct semantic rewards, and exact action expectations. Unless specified otherwise, we follow the setup of Section 4. Non-Gaussian prompts. We keep the task means of Figure 2 and replace the Gaussian prompt noise by independent Laplace⁡(0,1/2)Laplace(0,1/ 2) coordinates, =t+x= μ_t+ , which preserves the means and the identity covariance. The results are nearly unchanged from Figure 2 (Figure E.3): population SFT develops essentially the same off-axis energy and forgetting, and the finite-sample curves are close. Population RFT keeps forgetting below 0.3%0.3\% but acquires a small off-axis energy (about 3×10−23× 10^-2 at η=0.5η=0.5, versus 2.5×10−42.5× 10^-4 with Gaussian prompts) and error up to 1.5%1.5\%: its algebraic style invariance holds for any prompt law, whereas its exactly on-axis population direction relies on Gaussian prompts. Figure E.3: Non-Gaussian prompts. Prompt noise coordinates are drawn independently from Laplace⁡(0,1/2)Laplace(0,1/ 2) around the task means of Figure 2, giving identity covariance. The layout and line styles match Figure E.1. Imperfect rewards. For q∈0,0.05,0.1,0.2q∈\0,0.05,0.1,0.2\, we apply either a class flip, replacing the entire reward vector by 1−R⁡(⋅,)1-R(·,x) with probability q, or token flips, independently reversing each token’s reward with probability q. Corruptions are sampled independently for each prompt, including within Monte Carlo (MC) batches. RFT sums exactly over actions using the corrupted rewards; SFT continues to receive correct demonstrations. Both methods are evaluated using clean semantic labels. Class flips preserve within-class reward symmetry and hence RFT’s zero-style invariance, whereas token flips introduce style fluctuations. Across the tested noise levels, MC RFT exhibits little forgetting, while token flips induce more forgetting than class flips in finite-sample runs (Figure E.4(a,c)). Figure E.4: Forgetting under two assumption ablations. Top: MC population updates; bottom: finite-sample updates. Columns vary reward corruption and the number of REINFORCE actions per prompt, respectively. All panels use η≈0.121η≈ 0.121 and the same vertical scale. Training prompts are paired across methods. Shading gives pointwise 95%95\% Student-t intervals over 1010 MC seeds and mean ±1.96± 1.96 standard errors over 5050 finite-sample seeds, conditional on the fixed world and test sets. Negative forgetting indicates backward improvement. Finite action sampling. With clean rewards, we replace the exact sum over 2M=122M=12 actions by the no-baseline REINFORCE estimator of Appendix D.1, averaging K∈1,4,16,64,256,1024K∈\1,4,16,64,256,1024\ independent actions per prompt at the current policy. The estimator is unbiased for every K by (D.2). MC updates additionally average over 10,00010,000 prompts, so even K=1K=1 uses 10,00010,000 action samples per update. Prompt and update counts are matched within each protocol, but action-query and computation budgets differ across K and the exact baseline. At the reference learning rate, REINFORCE forgetting moves closer to the exact RFT baseline as K increases in both settings (Figure E.4(b,d)).

Instant research discovery