Paper deep dive
Learning to Steer, Steering to See: Unveiling the Geometry of RLVR in Large Language Models via Trainable Vectors
Yuchen Cai, Ding Cao, Qixiang Yin, Xin Xu, Kai Yang, Siye Wu, Pengyuan Wang, Jiaxuan Wang, Weijie Liu, Saiyong Yang, Guangzhong Sun, Guiquan Liu, Junfeng Fang
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Reinforcement learning (RL) has become a key paradigm for enhancing the reasoning of large language models, yet the high dimensionality of parameter updates makes its training dynamics hard to analyze. We study reinforcement learning with verifiable rewards (RLVR) and use vector steering to identify a low-dimensional effective manifold in activation space associated with RL-induced gains. We uncover two geometric properties. (1) Effective Manifold Capacity: the capacity needed to reproduce RL gains can be very small but is not infinitely compressible; at extremely low capacity, intervention dimensionality and input-dependent expressiveness become key constraints, and this requirement varies with injection depth. (2) Control Manifold Separation: effective control directions lie mainly in the low-variance complement of the activation principal subspace. Within a task and base model, the learned geometry stays largely consistent across training configurations, and across tasks geometric alignment correlates with capability transfer. Experiments on 5 LLMs and 6 verifiable-reward tasks support these findings. We then propose Alpha-Stabler, a plug-and-play framework with a Predictor that monitors principal-subspace intrusion for early collapse warnings, and a Controller that removes the principal-subspace component of activation gradients during backpropagation while preserving the orthogonal complement. Alpha-Stabler stabilizes training for 2,000 steps and consistently improves RL gains, offering practical insights for robust post-training. Code: this https URL
Tags
Links
- Source: https://arxiv.org/abs/2609.34344v1
- Canonical: https://arxiv.org/abs/2609.34344v1
Trouble viewing inline? Open PDF directly →
Full Text
175,747 characters extracted from source content.
Expand or collapse full text
Learning to Steer, Steering to See: Unveiling the Geometry of RLVR in Large Language Models via Trainable Vectors Yuchen Cai †thanks: This work was done during an internship at Tencent. Affiliation: USTC Affiliation: Tencent Hunyuan Email: caiyuchen@mail.ustc.edu.cn Ding Cao Affiliation: USTC Email: caoding@mail.ustc.edu.cn Qixiang Yin Affiliation: BUPT Xin Xu Affiliation: Tencent Hunyuan Kai Yang Affiliation: Tencent Hunyuan Siye Wu Affiliation: Tencent Hunyuan Pengyuan Wang Affiliation: Tencent Hunyuan Jiaxuan Wang Affiliation: Tencent Hunyuan Weijie Liu Affiliation: Tencent Hunyuan Saiyong Yang Affiliation: Tencent Hunyuan Guangzhong Sun Affiliation: USTC Guiquan Liu †thanks: Corresponding author: gqliu@ustc.edu.cn, fjf@mail.ustc.edu.cn Affiliation: USTC Junfeng Fang33footnotemark: 3 Affiliation: USTC Abstract Reinforcement learning (RL) has emerged as a key paradigm for enhancing the reasoning capabilities of large language models. However, the high dimensionality of parameter updates makes RL training dynamics difficult to analyze, obscuring the mechanisms underlying these gains. In this work, we study reinforcement learning with verifiable rewards (RLVR) and use vector steering as an analytical tool to identify a low-dimensional effective manifold in activation space that is closely associated with RL-induced performance gains. We further uncover two key geometric properties of this manifold. (1) Effective Manifold Capacity: The capacity required to reproduce RL-induced gains can be very small, but it is not infinitely compressible. When the capacity is compressed to an extremely low level, the intervention dimensionality and the ability to express input-dependent corrections become key constraints, and this requirement further varies with injection depth. (2) Control Manifold Separation: Effective control directions lie predominantly in the low-variance complement of the activation principal subspace. Within the same task and base model, the learned geometry remains largely consistent across training configurations; across tasks, geometric alignment correlates with capability transfer. We conduct experiments on 5 LLMs and 6 tasks with verifiable rewards, and the results support the above properties. Based on these findings, we propose Alpha-Stabler, a plug-and-play training framework comprising two modules: a Predictor that monitors principal-subspace intrusion to provide early warnings of training collapse, and a Controller that removes the principal-subspace component of activation gradients during backpropagation while preserving their components in the orthogonal complement of the principal subspace. Experiments show that Alpha-Stabler can stabilize training for 2,000 steps and consistently enhance the gains brought by reinforcement learning. This work advances the understanding of RL from an activation-manifold perspective and offers practical insights for more robust post-training. Our code is available at: https://github.com/caiyuchen-ustc/On_Policy_Vector_Training. “Mille viae ducunt homines per saecula Romam.” — Alain de Lille 1 Introduction Reinforcement learning (RL) has become central to post-training for large language models (LLMs), yielding substantial gains in reasoning performance (OpenAI, 2025; Xiao et al., 2026; Xu et al., 2026). Yet how RL produces these improvements internally remains incompletely understood. High-dimensional parameter updates, optimization noise, and sensitivity to training configurations make it difficult to distinguish behaviorally meaningful changes from incidental variation. Prior work has examined sampling efficiency (Yue et al., 2025), entropy dynamics (Cui et al., 2025b), scaling behavior (Tan et al., 2026), and update geometry (Cai et al., 2025; Cai et al., 2026; Zhu et al., 2025). A complementary question remains unanswered: what structure do RL-induced behavioral changes exhibit in activation space, and how does this structure relate to capability transfer and training stability? We view RL post-training as an intervention problem in activation space. Concretely, we freeze the base model and learn input-invariant shared vectors at selected layers to distill its RL-tuned counterpart, thereby testing whether RL-induced gains can be recovered by low-dimensional activation interventions. This directly probes whether RL gains admit a low-dimensional description and when this description breaks down. We thereby identify a low-dimensional effective manifold in activation space tied to RL-induced gains, and characterize two geometric properties of this manifold: Effective Manifold Capacity and Control Manifold Separation. Our first finding concerns the effective capacity of this manifold. With an RL-trained model as the teacher, a single input-invariant vector per controlled layer recovers over 85% of the gain achieved by full fine-tuning on most evaluated tasks, as shown in Fig. 1 (a) left. However, this compressibility has clear limits. The first arises from the teacher–student distribution gap: when the two distributions are close, a single-vector intervention suffices, whereas a widening gap requires more flexible parameterizations. The second arises from insertion depth: fixed-vector interventions work well at low-to-middle layers but degrade near the output, as shown in Fig. 1(a) right. Through nonlinear analysis, we further find that this degradation does not stem from weak optimization signals at deeper layers, but from the expressiveness bottleneck of low-dimensional interventions at high layers. We summarize these findings as Property 1 (Effective Manifold Capacity): the capacity required to reproduce RL-induced gains can be very small, but it is not infinitely compressible; this capacity depends on how well the intervention form matches the required correction. Figure 1: The single steering vector reveals that RLVR-induced changes form a low-dimensional, depth-sensitive control geometry, characterized by (a) effective manifold capacity and (b) control-manifold separation. Building on these findings, (c) Alpha-Stabler stabilizes training via principal-subspace intrusion monitoring and backward gradient projection. We next examine the geometry of the effective manifold. By extending the single-vector probe to multiple mutually orthogonal vectors per layer, we find that although the standalone performance of later directions declines with extraction order, they can still independently support the target behavior, indicating that the leading direction is dominant but not uniquely effective. While RL’s training parameters differ substantially across configurations in parameter space, they consistently converge, after compression, to a set of reproducible task-related directions. Across tasks, the alignment between direction vectors strongly reflects transfer gains, and these directions consistently concentrate in the low-variance complement of the activation principal subspace, as shown in Fig. 1 (b). We characterize this organization as Property 2 (Control Manifold Separation): RL-induced control converges to a set of reproducible task-related directions that are largely separated from the variance-dominant representation subspace. These two properties indicate that RL-induced behavioral changes admit a structured, low-dimensional description in activation space. During successful RL training, activation shifts remain predominantly in the low-variance complement of the principal subspace, whereas principal-subspace intrusion accompanies performance degradation and collapse. To this end, we propose Alpha-Stabler, a lightweight, plug-and-play framework for stabilizing RL training, as shown in Fig. 1 (c). It introduces no additional trainable parameters and requires no modification to the optimizer, reward function, or model architecture. Alpha-Stabler consists of a Predictor that monitors principal-subspace intrusion (PSI) against a fixed principal subspace from the frozen base model and warns before collapse, and a Controller that removes the principal-subspace component of activation gradients during backpropagation while preserving their orthogonal complements. Experiments show that, with no significant additional computation and zero additional trainable parameters, Alpha-Stabler achieves more stable reward growth over thousands of training steps than gradient clipping and other gradient projection methods. In summary, this work identifies two geometric properties of RLVR-induced changes: Effective Manifold Capacity and Control Manifold Separation. These properties reveal that the seemingly complex dynamics of RL post-training are governed by a surprisingly simple structure: a low-dimensional manifold confined to the low-variance complement of the activation principal subspace. Building on these properties, we introduce Alpha-Stabler, a plug-and-play framework that operates with negligible overhead and provides a practical tool for stabilizing large-scale RL training. Figure 2: (a) On-policy distillation scores on the training set. (b) Off-policy distillation scores on the evaluation set. (c) Scores of vector steering under varying teacher types. (d) Scores of full-parameter training, LoRA, and vector steering under off-policy distillation at three distribution gaps. 2 Effective Manifold Capacity 2.1 Experimental Setting Models and Datasets. We evaluate DeepSeek-R1-Distill-Qwen-1.5B/7B (Guo et al., 2025) and Qwen3-4B/8B/14B (Yang et al., 2025) on four task families: mathematical reasoning, scientific reasoning, code generation, and instruction following. For each task, we train a teacher with GRPO (Guo et al., 2025) or DAPO (Yu et al., 2025) and perform on- or off-policy distillation (Li et al., 2026b), comparing three update parameterizations: full-parameter training, LoRA (Hu et al., 2021), and vector steering. Details are provided in Appendix B.2. Vector Steering. Consider a frozen base model with L decoder layers and hidden dimension d. For a set of controlled layers ⊆1,…,LS \1,…,L\, we add a trainable vector to each layer’s output: ℓ′=ℓ+ℓ,ℓ∈,h_ =h_ + δ_ , , (1) where ℓ∈ℝdh_ ^d is the hidden state at a given token position, and ℓ∈ℝd δ_ ^d is a layer-specific vector shared across all inputs and token positions. Only ℓ∈\ δ_ \_ are optimized under the corresponding distillation objective. We then examine the compressibility of RL updates in Section 2.2 and further analyze its properties in Section 2.3. 2.2 Compressibility of RL Learning Signals Compressibility across Distillation Regimes. We examine how much of the capability gain obtained by RL can be retained after distillation under restricted update parameterizations. As shown in Fig. 2 (a), under on-policy distillation, LoRA and full-parameter training perform comparably, while a single steering vector per controlled layer still recovers over 85%85\% of the RL gain. To test whether this compressibility depends on student-generated trajectories (Yang et al., 2026), we also evaluate off-policy distillation with teacher-generated trajectories under the same update parameterizations. As shown in Fig. 2 (b), steering retains a comparably high recovery rate on evaluation datasets. Together, these results indicate that RL-induced gains can be highly compressed, and that fixed layer-wise activation offsets alone suffice to retain most of them. Effect of Teacher Type and Update Capacity. These recovery rates, however, have limits. We fix the student and vary the teacher under both distillation regimes to examine the sensitivity of vector steering to the teacher–student distribution gap. As shown in Fig. 2 (c), vector steering effectively transfers the capabilities of RL-trained teachers, whereas steering performance degrades substantially with the tested non-RL teachers, with some settings collapsing as the gap widens. This indicates that teacher–student distribution matching is a key factor governing the effectiveness of fixed activation offsets. A natural explanation is that the required corrections become more input-dependent as the gap widens, exceeding what an input-invariant fixed offset can express. If so, parameterizations with different expressive capacities should exhibit a consistent ordering. As shown in Fig. 2 (d), under off-policy distillation, the three parameterizations perform comparably when the gap is small, whereas a larger gap yields a clear ordering: full-parameter training performs best, followed by LoRA, and vector steering worst. This ordering matches their expressive capacities: full-parameter training allows model-wide adjustments, LoRA retains input-dependent corrections under low-rank constraints, and vector steering only applies fixed offsets. Therefore, a larger gap raises the input dependence required of the correction, and fixed additive corrections are the first to reach their capacity limit (Cai et al., 2026). Implications for RL Compressibility. Taken together, these results show that, although vector steering cannot capture arbitrary teacher-induced corrections, it serves as a targeted probe for analyzing the mechanisms underlying RL-specific capability gains. Building on this, we next examine how RL-induced gains vary with injection depth and how effective the low-dimensional representation is across layers. Figure 3: Layer-wise performance. (a) Converged performance of steering vectors across injection layers. (b) Task performance during training for steering vectors. (c) Steering gradient norms at different layers. (d) KL divergence with the teacher under fixed versus gated steering. (e) Converged performance of fixed versus gated injection. 2.3 Layer-wise Compressibility of RL Updates Injection Depth and Optimization Dynamics. Section 2.2 shows that RL-induced gains can be compressed into fixed activation offsets. This raises a complementary question: where should such a low-capacity intervention be applied? We therefore compare full-parameter updating, LoRA, and fixed-vector steering at representative network depths, restricting each intervention to the selected layer and freezing the remaining backbone. As shown in Fig. 3 (a), selected-layer full-parameter updating and LoRA remain effective across depths, whereas fixed-vector steering performs well at lower and intermediate layers but deteriorates near the output. This contrast indicates that low-capacity interventions depend on both the update parameterization and the injection depth. We next examine the learning curves across injection sites. Fig. 3 (b) shows that shallow-layer steering gradually approaches intermediate-layer performance during training, whereas high-layer steering plateaus early, and extended training does not noticeably close the gap. To test whether this stagnation stems from weaker gradient signals, we inspect gradient norms. Fig. 3 (c) further shows that shallow and high-layer injections receive gradients of comparable magnitude yet yield substantially different final performance, indicating that gradient magnitude alone does not explain high-layer degradation. One explanation is that high-layer injections have fewer downstream nonlinear transformations, which may limit the ability of a fixed offset to produce context-dependent effects, whereas LoRA and selected-layer full-parameter updates can directly produce input-dependent activation corrections. Input-Dependent Corrections and Capacity Controls. To test whether explicit token-dependent modulation alleviates high-layer degradation, we replace the fixed offset with a low-rank nonlinear correction: ℓ′=ℓ+ℓ(ℓ),ℓ(ℓ)=ℓσ(ℓℓ),h_ =h_ + δ_ (h_ ), δ_ (h_ )=B_ σ\! (A_ h_ ), (2) where ℓ∈ℝr×dA_ ^r× d, ℓ∈ℝd×rB_ ^d× r, r≪dr d, and σ is an element-wise gating nonlinearity. Here ℓB_ defines shared correction directions, while σ(ℓℓ)σ(A_ h_ ) assigns token-dependent coefficients. We zero-initialize ℓB_ so that the intervention initially leaves the model unchanged. Fig. 3 (d,e) shows that Rank-1 gating reduces teacher KL divergence and improves converged performance at all four tested high-layer injection sites. At Rank-1, the correction still follows a single shared direction per layer, but its strength can vary across tokens. To separate this flexibility from parameter count, we compare against a parameter-matched input-independent control that replaces σ(ℓℓ)σ(A_ h_ ) with a constant coefficient. Fig. 4 (a) shows that the static control does not reproduce the improvement of Rank-1 gating, supporting the role of token-dependent modulation. Increasing the gating rank yields further gains toward the full-parameter reference at r=16r=16. Together, these results suggest that high-layer recovery depends on both the available correction directions and their input-dependent modulation. Figure 4: Diagnosing and mitigating high-layer degradation. (a) Converged performance of vector steering under different nonlinear parameters. (b) Layer-wise mean pairwise cosine similarity for hidden states and output gradients. (c) Distribution of gating activations at different layers. Directional Structure and Effective Manifold Capacity. We further examine how gradient alignment and learned modulation vary across depth. At each layer, we compute the mean pairwise cosine similarity between per-token gradient contributions to the steering vector, Simℓgrad=1||∑(i,j)∈cos(ℓ,i,ℓ,j)Sim_ ^grad= 1|P| _(i,j) (g_ ,i,g_ ,j), where ℓ,ig_ ,i denotes the normalized contribution of token i to the gradient of the loss with respect to the steering vector at layer ℓ , and P is the set of token pairs. We compute the same statistic for hidden states as a reference. Fig. 4 (b) shows that mean gradient similarity rises from about 0.0190.019 at lower and intermediate layers to about 0.140.14 near the output. If directional alignment alone determined steering effectiveness, this trend would favor shared-direction interventions at higher layers. Yet fixed-vector steering degrades precisely there, showing that average directional alignment alone does not explain the depth-dependent performance gap. We further analyze the distribution of the Rank-1 gating coefficient σ(ℓℓ)σ(A_ h_ ) in Fig. 4 (c). At lower and intermediate layers, coefficients spread broadly over [0,1][0,1], consistent with graded modulation; at higher layers, they concentrate near 00 and 11, consistent with more selective use of the shared direction. Together with the gated-versus-static comparison in Fig. 4 (a), this indicates that explicit token-dependent modulation improves the effectiveness of high-layer interventions: when the intervention can adjust its strength according to each token’s hidden state, the limited downstream nonlinear transformations at high layers no longer constitute a major bottleneck. Summary. We summarize these findings as Property 1 (Effective Manifold Capacity): reproducing RL-induced gains can require only a very low-dimensional intervention, but its effective capacity depends on how well the intervention form matches the required correction. Appendix D.4 provides a conditional account of fixed-vector recovery in terms of the balance between shared teacher-induced shifts and the context-dependent fluctuations around them. 3 Control Manifold Separation Having established strong recovery with one steering vector per controlled layer, we now examine the diversity, reproducibility, and principal-subspace localization of effective steering directions. 3.1 Probing Residual Effective Directions with Multi-Vector Steering Sequential orthogonal steering. To test whether the leading direction is uniquely effective, we sequentially learn K vectors ℓ=ℓ(k)k=0K−1V_ =\v_ ^(k)\_k=0^K-1 at each controlled layer ℓ∈ , enforcing pairwise orthogonality against earlier vectors, ⟨ℓ(k),ℓ(j)⟩=0 _ ^(k),v_ ^(j) =0 for j<kj<k. The first stage is trained to convergence and frozen; each subsequent stage initializes new vectors in the remaining orthogonal complement, optimizes them to convergence, and reprojects them after every optimizer step: ℓ(k)←ℓ(k)−∑j=0k−1⟨ℓ(k),ℓ(j)⟩‖ℓ(j)‖22ℓ(j).v_ ^(k) _ ^(k)- _j=0^k-1 _ ^(k),v_ ^(j) \|v_ ^(j) \|_2^2v_ ^(j). (3) At each stage, only the current stage’s vectors are injected and evaluated, jointly defining a multi-layer intervention; earlier vectors remain frozen and serve solely to define the orthogonality constraints. Figure 5: Multi-vector steering with sequential orthogonalization. (a) Training trajectories of sequentially learned orthogonal steering vectors on SciKnowEval with DeepSeek-R1-Distill-Qwen-1.5B. (b) Standalone accuracy of the extracted directions across model scales. Each direction is trained and evaluated independently, with preceding directions used only to define the orthogonality constraint. Residual effectiveness and model scale. As shown in Fig. 5 (a), the first direction achieves 67.5%67.5\% standalone accuracy on SciKnowEval with DeepSeek-R1-Distill-Qwen-1.5B, while the eighth direction, (7)v^(7), achieves 62.0%62.0\%, only 5.55.5 percentage points lower despite being orthogonal to the preceding seven directions at every controlled layer. This modest reduction indicates that excluding previously learned directions does not exhaust the available steering solutions: the remaining orthogonal space still admits interventions with substantial standalone effectiveness. Training results across all four task domains are shown in Appendix Fig. 11. We further examine how this phenomenon varies with model scale. As shown in Fig. 5 (b), the leading direction consistently performs best, while standalone accuracy generally declines with extraction order; the decline is sharper in smaller models, whereas larger models retain higher-order directions whose performance remains closer to that of the leading direction. This indicates that model scale affects not only the performance of the leading direction, but also the extent to which higher-order directions are retained relative to it. Summary. These results show that the leading steering direction is effective but not unique: multiple layer-wise orthogonal interventions can independently support the target behavior, although their standalone effectiveness generally declines with extraction order. We next examine whether comparable directions are recovered across teacher parameterizations and distillation regimes. 3.2 Task-Dependent Geometry and Principal-Subspace Separation Consistency across training configurations. We next ask whether the extracted steering directions are reproducible across training configurations. We compare directions along two axes: teacher parameterizations (full-parameter fine-tuning vs. LoRA with different ranks) and distillation regimes (on- vs. off-policy). For each axis, we extract multi-vector directions using the same procedure and measure their layer-averaged cosine similarity, sim(a(i),b(j))=1||∑ℓ∈cos(a,ℓ(i),b,ℓ(j))sim(\v_a^(i)\,\v_b^(j)\)= 1|S| _ (v_a, ^(i),v_b, ^(j)), where S is the controlled layer set and i,ji,j denote extraction indices. Fig. 6 (a) shows stronger same-index than cross-index alignment across the tested teacher parameterizations, while Fig. 6 (b) reports matching-index alignment between on- and off-policy directions. Together, these comparisons indicate a broadly consistent geometric structure across the evaluated configurations and distillation regimes within the same task. Cross-topic alignment and capability transfer. Based on this, we further examine whether this geometric structure also reflects functional relationships across tasks. We partition Science into physics, chemistry, materials, and biology. Keeping the base model fixed, we train one RL teacher per topic and distill it using the same procedure to extract its multi-vector directions. For source topic s and target topic t, we define directional alignment as the sum of the first four same-index cosine similarities, averaged across controlled layers: A(s,t)=1||∑ℓ∈∑i=03cos(s,ℓ(i),t,ℓ(i)).A(s,t)= 1|S| _ _i=0^3 \! (v_s, ^(i),v_t, ^(i) ). We normalize the source-to-target transfer gain by the gain obtained from training directly on the target topic: ΔrelAcc(s→t)=Acc(Ms,t)−Acc(Mbase,t)Acc(Mt,t)−Acc(Mbase,t)×100%, _relAcc(s→ t)= Acc(M_s,t)-Acc(M_base,t)Acc(M_t,t)-Acc(M_base,t)× 100\%, (4) where MsM_s, MtM_t, and MbaseM_base denote the source-trained, target-trained, and base models, respectively. We evaluate six ordered source–target pairs among the four scientific topics, excluding self-pairs, and report the corresponding directional alignments and normalized transfer gains. As shown in Fig. 6 (c), stronger directional alignment is associated with larger normalized transfer gains across these pairs, with a linear fit yielding R2=0.91R^2=0.91. This indicates that the geometry of the learned steering directions is associated with capability transfer across the evaluated topic pairs, with stronger similarity corresponding to better transferability across tasks. Effective steering directions avoid the principal subspace. We further examine where these directions are located in activation space. Let Uℓ,r∈ℝd×rU_ ,r ^d× r contain the leading orthonormal eigenvectors of the frozen base model’s token-level activation covariance at layer ℓ , with r corresponding to 10%10\% of the hidden dimension, and define Pℓ=Uℓ,rUℓ,r⊤P_ =U_ ,rU_ ,r . For a layer-wise direction ℓz_ , its principal-subspace energy fraction is ePC(ℓ)=‖Pℓℓ‖22‖ℓ‖22+ϵ.e_PC(z_ )= \|P_ z_ \|_2^2\|z_ \|_2^2+ε. (5) For multi-layer directions, we average ePCe_PC across controlled layers. As shown in Fig. 7 (a), across four scientific topics, unconstrained steering places only about 1%1\%–2%2\% of its energy in the Top-10%10\% principal subspace, well below the 10%10\% isotropic reference, with this pattern persisting across the displayed directions (0)v^(0) through (8)v^(8). Low principal-subspace energy is thus a shared geometric property of the extracted interventions. Figure 6: Geometric consistency and transfer of steering directions. (a) Layer-averaged cosine similarities between steering vectors under different teacher parameterizations, shown as stacked bars for each reference vector a(i)v_a^(i), with segments ordered from (0)v^(0) to (3)v^(3) indicating similarities with b(j)j=03\v_b^(j)\_j=0^3. (b) Layer-averaged cosine similarities between on- and off-policy steering vectors with matching extraction indices, per task. (c) Cross-topic directional alignment versus normalized source-to-target transfer gains, with a fitted regression line. Restricting steering to the principal subspace does not recover the gain. To test the functional importance of the low-variance complement, we remove each steering vector’s projection onto the bottom-70%70\% activation-PC subspace after every optimizer step, restricting it to the span of the Top-30%30\% components: ℓ(k)←Pℓ,mℓ(k)v_ ^(k)← P_ ,mv_ ^(k), where m is the number of retained components and Pℓ,m=Uℓ,mUℓ,m⊤P_ ,m=U_ ,mU_ ,m . As shown in Fig. 7 (b), the Top-10%10\% energy fraction rises to about 15%15\%, but accuracy does not improve meaningfully. The gain therefore does not come from alignment with the principal subspace. Principal-subspace intrusion tracks training collapse. We next examine whether actual RL-induced activation shifts exhibit increased intrusion into the base model’s Top-10%10\% principal subspace as training becomes unstable. At step t, we run the current policy and the frozen base model on the same token sequences, defining the activation shift ℓ,b,i(t)=ℓ,b,i(t)−ℓ,b,ibase(t) _ ,b,i^(t)=h_ ,b,i^(t)-h_ ,b,i^base(t) for sample b, position i, and layer ℓ . Here ℓ,b,ibase(t)h_ ,b,i^base(t) is the frozen base model’s activation on the token sequence used at update t; t indexes the sequence, not a change in base-model parameters. We stack all N valid token-level shifts row-wise into Xℓ(t)∈ℝN×dX_ ^(t) ^N× d to avoid cancellation before projection, and compute PSIℓ(t)=‖Xℓ(t)Uℓ,r‖F2‖Xℓ(t)‖F2+ϵ,PSI_ ^(t)= \|X_ ^(t)U_ ,r\|_F^2\|X_ ^(t)\|_F^2+ε, (6) where Uℓ,rU_ ,r is the fixed Top-10%10\% principal-subspace basis of the base model. As shown in Fig. 7(c), collapse-prone runs show a clear pattern: during stable training, PSI remains low, well below the 10%10\% isotropic reference; as training proceeds, PSI departs from this range and becomes more volatile before any visible score degradation; as collapse progresses, PSI keeps rising while the score falls more sharply. Summary. These results support Property 2 (Control Manifold Separation): in the evaluated settings, effective steering directions lie mainly in the low-variance complement of the activation principal subspace, their ordered geometry stays broadly consistent across teacher parameterizations and distillation regimes, and cross-topic alignment correlates with transfer gains. Appendices D.5 and D.6 further discuss constrained vector fitting and the link between principal-subspace energy and output sensitivity. Figure 7: Steering geometry and principal-subspace intrusion. Energy fractions in (a) and (b) use the Top-10%10\% principal subspace. (a) Standalone accuracy and principal-subspace energy fractions of sequentially extracted directions across four scientific topics. (b) The same quantities when steering vectors are restricted to the Top-30%30\% activation-PC subspace. (c) Task score and PSI during RL training, comparing stable and collapse-prone runs. 4 Alpha-Stabler: Geometry-Guided RL Stabilization Experiments show that effective steering directions lie in the low-variance complement, while principal-subspace intrusion accompanies training collapse. We therefore propose Alpha-Stabler to monitor whether the RL-induced activation shift departs from this regime. It consists of a Predictor, which detects sustained increases in principal-subspace intrusion (PSI), and a Controller, which removes the principal-subspace component of activation gradients during backpropagation while preserving each incoming gradient’s component in the orthogonal complement of the principal subspace. 4.1 Online Monitoring and Control At parameter-update step t, we evaluate the actor and the frozen initial base model on identical token sequences. We stack their valid token-level differences, Δℓ,b,i(t)=ℓ,b,i(t)−ℓ,b,ibase(t) _ ,b,i^(t)=h_ ,b,i^(t)-h_ ,b,i^base(t), into Xℓ(t)∈ℝN×dX_ ^(t) ^N× d. During the first Twarm=50T_warm=50 updates, actor-generated rollouts are evaluated by the frozen base model to estimate the base model’s token-level activation covariance at each monitored layer. At the end of warm-up, we extract its leading orthonormal eigenvectors Uℓ,rU_ ,r, with r=⌈0.10d⌉r= 0.10d , and keep these bases fixed thereafter. Using the layer-wise PSI defined in Eqn. 6, we additionally aggregate across monitored layers: PSI(t)=∑ℓ∈‖Xℓ(t)Uℓ,r‖F2∑ℓ∈‖Xℓ(t)‖F2+ϵ.PSI^(t)= _ \|X_ ^(t)U_ ,r\|_F^2 _ \|X_ ^(t)\|_F^2+ε. (7) Layer-wise PSI determines where control is activated; the aggregate summarizes intrusion across monitored layers according to their shift energies. Both sum token-level energies before normalization, avoiding cancellation between differently oriented shifts. PSI is computed from detached activations, so it does not affect gradients, while the actor’s computation graph is retained for backpropagation. Predictor: detecting sustained intrusion. During an initially stable warm-up, the actor trains without projection, and valid activation-shift matrices are retained every M updates. Once the principal bases are fixed, we compute PSI for these retained shifts and set fixed warning and lower release thresholds from layer-wise medians and scaled median absolute deviations. Subsequent valid observations update an exponential moving average initialized at the warm-up median. Control activates after p consecutive valid checks with the smoothed PSI above the warning threshold, and releases when it falls below the lower threshold. Algorithm 1 specifies the full procedure. Controller: projecting activation gradients. The Controller modifies backward gradients without changing forward activations. Let Gℓ(t)∈ℝN×dG_ ^(t) ^N× d denote the incoming gradient at layer ℓ , including any downstream control effects. With the Predictor’s flag aℓ(t)∈0,1a_ ^(t)∈\0,1\ fixed during backpropagation and the optimizer step, we apply G~ℓ(t)=Gℓ(t)−aℓ(t)(Gℓ(t)Uℓ,r)Uℓ,r⊤. G_ ^(t)=G_ ^(t)-a_ ^(t) (G_ ^(t)U_ ,r )U_ ,r . (8) When active, this removes the component of each token’s gradient in the principal subspace while preserving the component orthogonal to that subspace. It integrates directly into standard RL training: rollout generation, likelihood ratios, rewards, and the GRPO or DAPO loss remain unchanged, and the optimizer updates parameters using only the modified backward signal. Warm-up requires estimating reference statistics and temporarily storing calibration shifts; after calibration, reference activations are needed only at monitoring checks and can be reused from a compatible pass. Each active hook performs two low-rank matrix multiplications, and inference uses the trained actor alone, so the additional computation is minimal. Algorithm 1 summarizes the full training loop. Figure 8: (a, b) Training dynamics with and without Alpha-Stabler. (c) Training dynamics across learning rates. (d) Task score and peak PSI under different intervention timings. (e) Performance under control of different subspaces. 4.2 Experimental Evaluation Training stabilization. Fig. 8 (a) and (b) compares training trajectories with and without Alpha-Stabler.. In uncontrolled runs, a pronounced increase in PSI coincides with sharp score deterioration, indicating that performance loss accompanies a departure from the earlier activation geometry. Under control, PSI remains near its early-training range while task scores continue to improve, yielding sustained learning alongside limited principal-subspace intrusion rather than reduced PSI at the expense of task performance. Ablation studies. We then conduct ablation studies of Alpha-Stabler along three dimensions: learning rate, intervention timing, and controlled subspace. First, we find that stabilization holds across learning rates, as shown in Fig. 8 (c): at each tested learning rate, uncontrolled training eventually collapses, whereas controlled training keeps peak PSI at a low single-digit percentage and reaches comparable final scores across settings, indicating robustness to the choice of learning rate. Second, triggered control is preferable to delayed control, as shown in Fig. 8 (d). Triggered control matches always-on control in both score and peak PSI, indicating that continuous intervention is unnecessary, whereas delaying intervention progressively degrades the score and raises peak PSI. Third, Top-PC projection outperforms alternative subspaces, as shown in Fig. 8 (e). Among the compared gradient-control methods, removal of Top-PC components achieves the highest score, while random-subspace, Middle-PC, and Bottom-PC control fail to match it and remain close to the uncontrolled baseline; Top-PC also compares favorably with global gradient clipping. This indicates that the improvement does not arise from an arbitrary gradient constraint, but from removing gradient components in a specific subspace, namely the principal subspace. These results support geometrically targeted control rather than indiscriminate gradient suppression. Appendix D.7 shows that gradient projection minimally removes the principal-subspace component of the incoming gradient and bounds principal-subspace activation leakage for a local, single-site projected step. Fig. 12 further shows nearly identical score trajectories with and without Alpha-Stabler under the same measured wall-clock time, indicating almost no additional time overhead. 5 Limitations and Future Work Despite revealing two geometric properties of RLVR’s effect on LLMs, our study has several limitations. First, the theoretical framework rests on empirically motivated assumptions not yet derived from first principles. Second, our analysis and experiments are restricted to RLVR on tasks such as mathematical reasoning and code generation; we do not validate the proposed geometry on RLHF, multi-turn agentic decision-making, or open-ended generation, where whether the same properties hold remains open. Future work can relax these assumptions via neuron attribution and causal tracing, extend the framework to RLHF and agentic settings, and exploit off-principal localization as a broader design principle, e.g., by adding complement-space regularizers or using PSI as an adaptive signal for learning rate and intervention strength, enabling low-rank control without layer monitoring. 6 Conclusion In this work, we identify two geometric properties of RLVR’s effect on LLMs. Effective Manifold Capacity characterizes how a very low-dimensional intervention can reproduce RLVR-induced gains, with the required capacity depending on how well the intervention form matches the required correction. Control Manifold Separation describes the concentration of reproducible, task-related control directions in the low-variance complement of the activation principal subspace. Based on these properties, Alpha-Stabler monitors principal-subspace intrusion as an early-warning signal and projects away the intrusive component of activation gradients during backpropagation, stabilizing training with almost no additional computational overhead. Overall, this work recasts RLVR from black-box parameter optimization into an interpretable geometric process, opening new avenues for theoretically grounded and practically efficient LLM post-training. AI use statement Generative AI tools were used to assist with language editing, manuscript organization, and formatting, with the aim of improving clarity, readability, and presentation. All AI-assisted revisions were critically reviewed and revised by the authors to ensure technical accuracy, consistency, and fidelity to the underlying research. The authors take full responsibility for the final manuscript, including its methodology, theoretical arguments, experimental results, and conclusions. Reproducibility Statement We document the experimental procedures and theoretical assumptions underlying our results. Appendix B specifies the training objectives, model configurations, datasets, task-specific verifiers, evaluation protocols, prompts, optimization hyperparameters, and hardware, together with example commands for vector-steering experiments. Appendix C provides additional analyses of the learned steering vectors. Appendix D states the assumptions and proofs supporting our geometric analysis and details the principal-subspace diagnostic and gradient-projection mechanism of Alpha-Stabler. Algorithm 1 specifies its warm-up calibration, monitoring, and control procedures. The source code is provided in the supplementary materials. References Arditi et al. (2024) A. Arditi, O. Obeso, A. Syed, D. Paleka, N. Panickssery, W. Gurnee, and N. Nanda Refusal in language models is mediated by a single direction. Advances in Neural Information Processing Systems 37, p. 136037–136083. Cited by: Appendix A. Cai et al. (2026) Y. Cai, D. Cao, L. Lin, C. Luo, X. Xu, K. Yang, W. Liu, S. Yang, T. Zhao, G. Sun, et al. Learning to foresee: unveiling the unlocking efficiency of on-policy distillation. arXiv preprint arXiv:2605.11739. Cited by: Appendix A, §1, §2.2. Cai et al. (2025) Y. Cai, D. Cao, X. Xu, Z. Yao, Y. Huang, Z. Tan, B. Zhang, G. Sun, G. Liu, and J. Fang On predictability of reinforcement learning dynamics for large language models. arXiv preprint arXiv:2510.00553. Cited by: Appendix A, §1. Cui et al. (2025a) G. Cui, L. Yuan, Z. Wang, H. Wang, Y. Zhang, J. Chen, W. Li, B. He, Y. Fan, T. Yu, et al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456. Cited by: Table 1. Cui et al. (2025b) G. Cui, Y. Zhang, J. Chen, L. Yuan, Z. Wang, Y. Zuo, H. Li, Y. Fan, H. Chen, W. Chen, Z. Liu, H. Peng, L. Bai, W. Ouyang, Y. Cheng, B. Zhou, and N. Ding The entropy mechanism of reinforcement learning for reasoning language models. External Links: 2505.22617, Link Cited by: Appendix A, §1. Ding et al. (2026) Y. Ding, H. Le, S. Han, K. Ruan, Z. Jin, V. Kumar, Z. Wang, and A. Deoras Empowering multi-turn tool-integrated agentic reasoning with group turn policy optimization. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 42409–42423. Cited by: Appendix A. Feng et al. (2024) K. Feng, X. Shen, W. Wang, X. Zhuang, Y. Tang, Q. Zhang, and K. Ding Sciknoweval: evaluating multi-level scientific knowledge of large language models. arXiv preprint arXiv:2406.09098. Cited by: Table 1, Table 1. Fu et al. (2026) W. Fu, J. Gao, X. Shen, C. Zhu, Z. Mei, C. He, S. Xu, G. Wei, J. Mei, J. Wang, et al. Areal: a large-scale asynchronous reinforcement learning system for language reasoning. Advances in Neural Information Processing Systems 38, p. 36256–36282. Cited by: Appendix A. Geva et al. (2022) M. Geva, A. Caciularu, K. R. Wang, and Y. Goldberg Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. External Links: 2203.14680, Link Cited by: §C.1. Guo et al. (2025) D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Bi, et al. Deepseek-r1: incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. Cited by: §2.1. He et al. (2025) Z. He, T. Liang, J. Xu, Q. Liu, X. Chen, Y. Wang, L. Song, D. Yu, Z. Liang, W. Wang, Z. Zhang, R. Wang, Z. Tu, H. Mi, and D. Yu DeepMath-103k: a large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning. External Links: 2504.11456, Link Cited by: Table 1. Hu et al. (2021) E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen LoRA: low-rank adaptation of large language models. External Links: 2106.09685, Link Cited by: 2nd item, §2.1. Ilharco et al. (2022) G. Ilharco, M. T. Ribeiro, M. Wortsman, S. Gururangan, L. Schmidt, H. Hajishirzi, and A. Farhadi Editing models with task arithmetic. arXiv preprint arXiv:2212.04089. Cited by: Appendix A. Jain et al. (2024) N. Jain, K. Han, A. Gu, W. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica LiveCodeBench: holistic and contamination free evaluation of large language models for code. arXiv preprint. Cited by: Table 1. Jin et al. (2025) B. Jin, H. Zeng, Z. Yue, J. Yoon, S. Arik, D. Wang, H. Zamani, and J. Han Search-r1: training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516. Cited by: Appendix A. Lambert et al. (2025) N. Lambert, J. Morrison, V. Pyatkin, S. Huang, H. Ivison, F. Brahman, L. J. V. Miranda, A. Liu, N. Dziri, S. Lyu, Y. Gu, S. Malik, V. Graf, J. D. Hwang, J. Yang, R. L. Bras, O. Tafjord, C. Wilhelm, L. Soldaini, N. A. Smith, Y. Wang, P. Dasigi, and H. Hajishirzi Tulu 3: pushing frontiers in open language model post-training. External Links: 2411.15124, Link Cited by: Appendix A. Li et al. (2026a) K. Li, Z. Xu, J. Li, Z. Jin, J. Deng, Z. Qiu, and B. Zhou Discovery and reinforcement of tool-integrated reasoning chains via rollout trees. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 14265–14283. Cited by: Appendix A. Li et al. (2026b) Y. Li, Y. Zuo, B. He, J. Zhang, C. Xiao, C. Qian, T. Yu, H. Gao, W. Yang, Z. Liu, et al. Rethinking on-policy distillation of large language models: phenomenology, mechanism, and recipe. arXiv preprint arXiv:2604.13016. Cited by: §2.1. Lightman et al. (2023) H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe Let’s verify step by step. arXiv preprint arXiv:2305.20050. Cited by: Table 1. Moshkov et al. (2025) I. Moshkov, D. Hanley, I. Sorokin, S. Toshniwal, C. Henkel, B. Schifferer, W. Du, and I. Gitman AIMO-2 winning solution: building state-of-the-art mathematical reasoning models with openmathreasoning dataset. arXiv preprint arXiv:2504.16891. Cited by: §B.2. OpenAI (2025) OpenAI Introducing gpt-oss. Note: https://openai.com/zh-Hans-CN/index/introducing-gpt-oss/ Cited by: §1. Ostermann et al. (2026) S. Ostermann, D. Gurgurov, T. Baeumel, M. A. Hedderich, S. Lapuschkin, W. Samek, and V. Schmitt From weights to activations: is steering the next frontier of adaptation?. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 29854–29879. Cited by: Appendix A. Ouyang et al. (2022) L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe Training language models to follow instructions with human feedback. External Links: 2203.02155, Link Cited by: Appendix A. Qwen Team (2026) Qwen Team Qwen3.8-max: a new bar for coding and cowork. External Links: Link Cited by: Appendix A. Ren et al. (2026) Q. Ren, P. Wang, R. Cai, S. Shao, D. Guo, Y. Xie, Y. Li, Q. Zhang, X. Hu, J. Shao, and D. Liu Rethinking generalization in reasoning sft: a conditional analysis on optimization, data, and model capability. arXiv preprint arXiv:2604.06628. Cited by: §B.2. Sheng et al. (2025) G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng, H. Lin, and C. Wu HybridFlow: a flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, EuroSys ’25, p. 1279–1297. External Links: Link, Document Cited by: §B.2. Song and Zheng (2026) M. Song and M. Zheng A survey of on-policy distillation for large language models. arXiv preprint arXiv:2604.00626. Cited by: Appendix A. Tan et al. (2026) Z. Tan, H. Geng, X. Yu, M. Zhang, G. Wan, Y. Zhou, Q. He, X. Xue, H. Zhou, Y. Fan, Z. Li, Z. Zhang, G. Zhang, C. Zhang, Z. Yin, P. Torr, and L. Bai Scaling behaviors of llm reinforcement learning post-training: an empirical study in mathematical reasoning. External Links: 2509.25300, Link Cited by: Appendix A, §1. Team et al. (2026) K. Team, T. Bai, Y. Bai, Y. Bao, J. Cai, X. Cai, P. Cao, Y. Cao, Z. Chai, Y. Charles, et al. Kimi k3: open frontier intelligence. arXiv preprint arXiv:2607.24653. Cited by: Appendix A. Todd et al. (2024) E. Todd, M. Li, A. Sen Sharma, A. Mueller, B. Wallace, and D. Bau Function vectors in large language models. In International conference on learning representations, Vol. 2024, p. 17282–17333. Cited by: Appendix A. Wollschläger et al. (2025) T. Wollschläger, J. Elstner, S. Geisler, V. Cohen-Addad, S. Günnemann, and J. Gasteiger The geometry of refusal in large language models: concept cones and representational independence. arXiv preprint arXiv:2502.17420. Cited by: Appendix A. Xiao et al. (2026) B. Xiao, B. Xia, B. Yang, B. Gao, B. Shen, C. Zhang, C. He, C. Lou, F. Luo, G. Wang, et al. Mimo-v2-flash technical report. arXiv preprint arXiv:2601.02780. Cited by: §1. Xu et al. (2026) A. Xu, B. Lin, B. Xue, B. Wang, B. Xu, B. Wu, B. Zhang, C. Lin, C. Dong, C. Ling, et al. Deepseek-v4: towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348. Cited by: Appendix A, §1. Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. Li, T. Tang, W. Yin, X. Ren, X. Wang, X. Zhang, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Zhang, Y. Wan, Y. Liu, Z. Wang, Z. Cui, Z. Zhang, Z. Zhou, and Z. Qiu Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: §2.1. Yang et al. (2026) Y. Yang, Y. He, J. Liu, and Z. Jin Making complex reasoning student-friendly: a hybrid llm-to-slm distillation framework. In 1st Workshop on Scaling Post-training for LLMs at the Fourteenth International Conference on Learning Representations (ICLR), External Links: Link Cited by: §2.2. Ye et al. (2025) Y. Ye, Z. Huang, Y. Xiao, E. Chern, S. Xia, and P. Liu LIMO: less is more for reasoning. External Links: 2502.03387, Link Cited by: Table 1. Yu et al. (2025) Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, W. Dai, T. Fan, G. Liu, L. Liu, X. Liu, H. Lin, Z. Lin, B. Ma, G. Sheng, Y. Tong, C. Zhang, M. Zhang, W. Zhang, H. Zhu, J. Zhu, J. Chen, J. Chen, C. Wang, H. Yu, Y. Song, X. Wei, H. Zhou, J. Liu, W. Ma, Y. Zhang, L. Yan, M. Qiao, Y. Wu, and M. Wang DAPO: an open-source llm reinforcement learning system at scale. External Links: 2503.14476, Link Cited by: §2.1. Yue et al. (2025) Y. Yue, Z. Chen, R. Lu, A. Zhao, Z. Wang, Y. Yue, S. Song, and G. Huang Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?. External Links: 2504.13837, Link Cited by: Appendix A, §1. Z.ai (2026) Z.ai GLM-5.3: frontier coding with emergent cyber capabilities. External Links: Link Cited by: Appendix A. Zhao et al. (2024) W. Zhao, X. Ren, J. Hessel, C. Cardie, Y. Choi, and Y. Deng WildChat: 1m chatgpt interaction logs in the wild. External Links: 2405.01470, Link Cited by: Table 1, Table 1. Zhou et al. (2025) Y. Zhou, S. Jiang, Y. Tian, J. Weston, S. Levine, S. Sukhbaatar, and X. Li Sweet-rl: training multi-turn llm agents on collaborative reasoning tasks. arXiv preprint arXiv:2503.15478. Cited by: Appendix A. Zhu et al. (2025) H. Zhu, Z. Zhang, H. Huang, D. Su, Z. Liu, J. Zhao, I. Fedorov, H. Pirsiavash, Z. Sha, J. Lee, et al. The path not taken: rlvr provably learns off the principals. arXiv preprint arXiv:2511.08567. Cited by: §1. Zou et al. (2023) A. Zou, L. Phan, S. Chen, J. Campbell, P. Guo, R. Ren, A. Pan, X. Yin, M. Mazeika, A. Dombrowski, et al. Representation engineering: a top-down approach to ai transparency. arXiv preprint arXiv:2310.01405. Cited by: Appendix A. Appendix A Related Work Reinforcement Learning for Large Language Models. Prior to reasoning-capable models like OpenAI’s o1, RL was primarily used in RLHF for instruction-following and human preference alignment (Ouyang et al., 2022). Recently, Reinforcement Learning with Verifiable Rewards (RLVR) has proven effective for reasoning in verifiable domains such as mathematics and programming (Lambert et al., 2025). OpenAI’s o1 inspired a wave of open reasoning models, including DeepSeek-V4 (Xu et al., 2026), Kimi-K3 (Team et al., 2026), Qwen3.8 (Qwen Team, 2026), and GLM-5.3 (Z.ai, 2026). Beyond single-turn reasoning, RL is increasingly applied to agentic settings involving planning, tool invocation (Li et al., 2026a), and multi-turn environment interaction (Ding et al., 2026), with notable efforts on search-augmented reasoning (Jin et al., 2025), multi-turn credit assignment (Zhou et al., 2025), and asynchronous large-scale agentic RL (Fu et al., 2026). Emergent Behaviors of On-Policy Training. Recent studies have investigated intrinsic properties of on-policy RL training. Yue et al. (2025) find that RL primarily improves pass@1 sampling efficiency without expanding the model’s reasoning frontier. Cui et al. (2025b) identify early-stage entropy collapse that prematurely degrades exploration. Tan et al. (2026) reveal a power-law relationship between test loss, computation, and data volume, showing larger models exhibit superior learning efficiency. From a parameter-dynamics perspective, Cai et al. (2025) uncover Rank-1 dominance and Rank-1 linear dynamics, enabling 2.5× acceleration via AlphaRL. Cai et al. (2026) attribute on-policy distillation’s efficiency to modular redundancy suppression and early directional stabilization, proposing EffOPD with 3× speedup (Song and Zheng, 2026). Steering Vectors and Representation Engineering. While most adaptation methods rely on parameter updates (full fine-tuning or LoRA), an alternative line of work intervenes directly on internal activations via vector steering. Representation Engineering (RepE) provides a unifying framework, treating population-level representations as the analytical unit for monitoring and manipulating cognitive phenomena (Zou et al., 2023). High-level behaviors often exhibit low-dimensional structure: refusal is mediated by a single direction (Arditi et al., 2024) or multi-dimensional concept cones (Wollschläger et al., 2025). Task vectors (Ilharco et al., 2022) and function vectors (Todd et al., 2024) enable behavioral editing through vector arithmetic, and steering is increasingly viewed as a paradigm complementary to fine-tuning and prompting (Ostermann et al., 2026). Appendix B Preliminaries and Experimental Setup B.1 Preliminaries We consider three post-training paradigms: Reinforcement Learning (RL), On-Policy Distillation (OPD), and Off-Policy Distillation. Let πθ _θ denote the policy or student model, π∗π^* the fixed teacher model, and πref _ref the fixed reference model used for regularization. The teacher provides distillation targets, whereas the reference model anchors RL regularization; these roles are distinct. Reinforcement Learning (RL). The RL objective is: maxθJRL(θ)=x∼[y∼πθ(⋅∣x)[r(x,y)]−βDKL(πθ(⋅∣x)∥πref(⋅∣x))], _θJ_RL(θ)=E_x [E_y _θ(· x) [r(x,y) ]-β D_KL ( _θ(· x) _ref(· x) ) ], (9) where y=(y1,…,yT)y=(y_1,…,y_T) is sampled from the current policy, r(x,y)r(x,y) is the reward assigned to response y for query x, and β controls the strength of KL regularization. In Reinforcement Learning from Verifiable Rewards (RLVR), r(x,y)r(x,y) is determined by verifiable outcomes, such as answer correctness or unit-test execution, without requiring a learned reward model. Separating the task-reward term from reference-policy regularization gives the policy-gradient form: ∇θJRL(θ) _θJ_RL(θ) =x∼,y∼πθ(⋅∣x)[∑t=1TAtr∇θlogπθ(yt∣x,y<t)] =E_x ,\;y _θ(· x) [ _t=1^TA_t^r _θ _θ(y_t x,y_<t) ] −β∇θx∼[DKL(πθ(⋅∣x)∥πref(⋅∣x))], -β _θE_x [D_KL ( _θ(· x) _ref(· x) ) ], (10) where AtrA_t^r denotes the token-level advantage associated with the task reward. In RLVR, this reward is typically observed only after the complete response has been generated. Equation 10 describes the underlying regularized objective; practical GRPO and DAPO updates use algorithm-specific advantage estimates and clipped surrogate objectives. On-Policy Distillation (OPD). OPD samples trajectories from the student policy and minimizes the reverse KL divergence from the student to the teacher: minθJOPD(θ) _θJ_OPD(θ) =x∼[DKL(πθ(⋅∣x)∥π∗(⋅∣x))] =E_x [D_KL ( _θ(· x) π^*(· x) ) ] =x∼,y∼πθ(⋅∣x)[∑t=1T(logπθ(yt∣x,y<t)−logπ∗(yt∣x,y<t))]. =E_x ,\;y _θ(· x) [ _t=1^T ( _θ(y_t x,y_<t)- π^*(y_t x,y_<t) ) ]. (11) The corresponding sequence-level gradient can be written as: ∇θJOPD(θ)=x∼,y∼πθ(⋅∣x)[∑t=1T∑t′=tT _θJ_OPD(θ)=E_x ,\;y _θ(· x) [ _t=1^T _t =t^T (logπθ(yt′∣x,y<t′)−logπ∗(yt′∣x,y<t′)) ( _θ(y_t x,y_<t )- π^*(y_t x,y_<t ) ) ⋅∇θlogπθ(yt∣x,y<t)]. · _θ _θ(y_t x,y_<t) ]. (12) For optimization, we use an immediate token-level approximation that retains only the contribution at t′=t =t: ∇θJOPD(θ)≈x∼,y∼πθ(⋅∣x)[∑t=1T _θJ_OPD(θ) _x ,\;y _θ(· x) [ _t=1^T (logπθ(yt∣x,y<t)−logπ∗(yt∣x,y<t)) ( _θ(y_t x,y_<t)- π^*(y_t x,y_<t) ) ⋅∇θlogπθ(yt∣x,y<t)]. · _θ _θ(y_t x,y_<t) ]. (13) This approximation provides token-level supervision along student-generated trajectories, but omits the future-token contributions in Equation 12. When implemented as an advantage-weighted surrogate, the log-probability difference is treated as a stop-gradient weight, with gradients taken through the student log-probability that it multiplies. Off-Policy Distillation (Off-PD). Off-policy distillation instead samples trajectories from the teacher policy. Its objective is the forward KL divergence from the teacher to the student: minθJOff-PD(θ) _θJ_Off -PD(θ) =x∼[DKL(π∗(⋅∣x)∥πθ(⋅∣x))] =E_x [D_KL (π^*(· x) _θ(· x) ) ] =x∼,y∼π∗(⋅∣x)[∑t=1T(logπ∗(yt∣x,y<t)−logπθ(yt∣x,y<t))]. =E_x ,\;y π^*(· x) [ _t=1^T ( π^*(y_t x,y_<t)- _θ(y_t x,y_<t) ) ]. (14) Since the teacher-dependent term is constant with respect to θ, this objective is equivalent to minimizing the negative log-likelihood of teacher-generated trajectories: ℒOff-PD(θ)=−x∼,y∼π∗(⋅∣x)[∑t=1Tlogπθ(yt∣x,y<t)].L_Off -PD(θ)=-E_x ,\;y π^*(· x) [ _t=1^T _θ(y_t x,y_<t) ]. (15) The corresponding gradient is: ∇θℒOff-PD(θ)=−x∼,y∼π∗(⋅∣x)[∑t=1T∇θlogπθ(yt∣x,y<t)]. _θL_Off -PD(θ)=-E_x ,\;y π^*(· x) [ _t=1^T _θ _θ(y_t x,y_<t) ]. (16) Thus, OPD and off-policy distillation differ in both trajectory sampling and the objective being optimized. OPD uses student-generated trajectories with a reverse-KL objective, whereas off-policy distillation uses teacher-generated trajectories with a forward-KL objective, implemented as sequence negative log-likelihood. The latter trains the student on teacher-generated prefixes, while the former evaluates the teacher on prefixes generated by the student. B.2 Experimental Setup Overview. Each experiment in this paper is defined by a training paradigm and an update configuration. The three paradigms are on-policy RL with verifiable rewards (Eqn. 9), on-policy distillation (Eqn. 11), and off-policy distillation (Eqn. 15), as formalized in Section B.1. For RL, we consider full fine-tuning and LoRA, whereas for both on-policy and off-policy distillation, we additionally evaluate vector steering. These configurations differ in which parameters receive gradients and, consequently, in the dimensionality and expressiveness of the resulting updates. Within each matched comparison, we hold the base model, prompt set, teacher model when applicable, sampling recipe, and evaluation protocol fixed across the applicable update configurations. Parameterization-specific optimization settings are reported in Table 2, and execution differences are described below. In on-policy experiments, matching the sampling recipe does not imply identical trajectories, since the samples depend on the current policy. This setup allows us to compare update parameterizations under matched modeling and evaluation protocols while making their optimization differences explicit. Models. To ensure the generality of our findings, we conduct experiments across model scales ranging from 1.5B to 14B parameters, spanning two model families: Qwen2.5/DeepSeek-R1-Distill (1.5B and 7B) and Qwen3 (4B, 8B, and 14B), and two RL algorithms (GRPO and DAPO). All RL models are trained locally using the verl framework (Sheng et al., 2025). For all distillation students, the default teacher is the RL-tuned version of the student’s own base model, ensuring that the teacher–student gap corresponds exactly to the RL update itself. Datasets and Evaluation. The training data spans four topics, each with its own verifier and held-out evaluation set (Table 1). Rewards are always verifiable in the sense of §B.1: rule-based answer matching for mathematical and scientific reasoning, unit-test execution for code, and programmatic constraint checking for instruction following. For the off-policy distillation experiments in Fig. 2 (d), we additionally compare two sources of supervision: standard SFT on Math-CoT-44K (Ren et al., 2026), which contains approximately 44K mathematical queries and 32 responses sampled from Qwen3-32B for each query, yielding more than 1.4M reasoning traces; and off-policy distillation on OpenMathReasoning (Moshkov et al., 2025), which contains 540K mathematical problems and 3.2M long chain-of-thought solutions generated by DeepSeek-R1 and QwQ-32B. Table 1: Training and evaluation datasets and reward functions for each domain. Domain Training Set Evaluation Set Reward Math DeepMath-103K (He et al., 2025) AIME 2024 (Ye et al., 2025) Answer matching MATH500 (Lightman et al., 2023) Science SciKnowEval-Training (Feng et al., 2024) SciKnowEval-Eval (Feng et al., 2024) Answer matching Code Eurus (Cui et al., 2025a) LiveCodeBench-v5 (Jain et al., 2024) Unit tests Instruction following IFEval-Training (Zhao et al., 2024) IFEval-Eval (Zhao et al., 2024) Constraint checking Instruction-following split. For instruction following, we partition the dataset at the prompt level into an 80% training split and a 20% held-out evaluation split, denoted as IFEval-Training and IFEval-Eval, respectively. These names refer to the partitions used in this study. All responses associated with the same prompt remain in the same partition, and evaluation prompts are excluded from training. The split is kept fixed across all compared configurations. Evaluation protocol. We report mean@4 on the held-out evaluation sets. For each evaluation prompt, we independently sample four responses using the same decoding configuration. Each response is scored using the corresponding task-specific evaluation criterion. The reported result is the arithmetic mean of the four single-sample evaluation scores computed on the same held-out set: mean@4=14∑j=14Score((xi,yi(j))i=1N),mean@4= 14 _j=1^4Score (\(x_i,y_i^(j))\_i=1^N ), (17) where N is the number of evaluation prompts, yi(j)y_i^(j) is the j-th sampled response to prompt xix_i, and ScoreScore is the task-specific evaluation metric. For mean per-response correctness, this is equivalently the average over all 4N4N scored responses. We do not select the best of the four responses or apply majority voting; mean@4 is therefore distinct from pass@4. Update configurations. We compare the following trainable parameterizations under the matched modeling and evaluation protocols described above, using the configuration-specific learning rates in Table 2. • Full fine-tuning updates all model parameters by default and serves as the unrestricted reference configuration. In principle, it can express a substantially broader class of parameter updates than the other configurations. • LoRA (Hu et al., 2021) injects low-rank adapters on all-linear target modules while keeping the backbone frozen. This restricts the update to a low-rank form, though each input can still produce a different correction (i.e., it remains context-dependent). The rank varies across experiments; Table 2 reports the default rank-88 configuration. • Vector steering freezes the entire backbone and trains context-invariant bias vectors added to the residual stream. For a controlled layer set S, the single-vector intervention is ℓ↦ℓ+ℓ,ℓ∈ℝd,ℓ∈,h_ \; \;h_ +v_ , _ ^d, , (18) with ||d|S|d trainable parameters. For the default single-vector configuration on Qwen3-4B (d=2560d=2560) with =5,…,20S=\5,…,20\ and ||=16|S|=16, this amounts to 40,96040,960 scalars, approximately 4.1×1044.1× 10^4, or roughly 1.0×10−51.0× 10^-5 of the backbone. Crucially, ℓv_ does not vary with the input: the same vector is added regardless of context or decoding position. Unless otherwise stated, vector steering uses the controlled layer set =5,…,20S=\5,…,20\ and a single vector per controlled layer (K=1K=1). In the layer-wise experiments of Section 2.3, all three configurations are restricted to the selected layer: full-parameter updating trains only that layer, LoRA adapts its linear modules, and vector steering adds an offset to its output. The remainder of the backbone is frozen. These selected-layer experiments are distinct from the default model-wide full-fine-tuning and LoRA configurations. For multi-vector probes (K>1K>1), we learn the ordered set ℓ(0),…,ℓ(K−1)\v^(0)_ ,…,v^(K-1)_ \ sequentially at each controlled layer. The first vector is trained to convergence and then retained as a fixed reference. At each subsequent stage k, only the current vector ℓ(k)v^(k)_ is injected and optimized; previously learned vectors remain inactive in the forward pass and are used solely to define the orthogonality constraint. After each optimizer step, the current vector is reprojected onto the orthogonal complement of the previously learned directions. Training advances to the next vector once the current-stage loss plateaus. Each direction is also evaluated independently, with only the corresponding vector active at each controlled layer. Consequently, each stage optimizes ||d|S|d parameters, while the complete collection contains K||dK|S|d scalars; increasing K probes additional independently effective directions rather than simultaneously summing constant offsets. Vector steering requires a substantially larger learning rate than the other two configurations; the values we use are reported in Table 2. Table 2: Learning rates and paradigm-specific hyperparameters. RL OPD Off-PD LR, full fine-tuning 1×10−61× 10^-6 1×10−51× 10^-5 5×10−55× 10^-5 LR, LoRA (r=8r=8) 3×10−53× 10^-5 5×10−65× 10^-6 1×10−51× 10^-5 LR, vector steering — 1×10−11× 10^-1 5×10−25× 10^-2 Max prompt / response 2,0482,048 / 20,48020,480 3,0723,072 / 16,38416,384 3,0723,072 / 16,38416,384 Prompt batch 128128 1,0241,024 1,0241,024 Mini-batch (per step) 3232 1,0241,024 1,0241,024 Training samples per prompt n 1616 11 11 Evaluation samples per prompt 44 44 44 Evaluation metric mean@4 mean@4 mean@4 Rollout source student student frozen teacher Rollout IS correction — token, threshold 5.05.0 off Auxiliary reference-KL coeff. 0.0010.001 (GRPO) / 00 (DAPO) 00 00 Epochs 10001000 10001000 10001000 Precision bfloat16 bfloat16 bfloat16 The auxiliary reference-KL coefficient in Table 2 refers to additional regularization toward πref _ref, not to the main distillation objective. In particular, a zero auxiliary coefficient does not remove the reverse-KL teacher supervision in OPD or the teacher-trajectory negative log-likelihood in Off-PD. Prompts. All Qwen3 models are run and trained in non-thinking mode (enable_thinking=False passed to the chat template) so that the response distribution is directly comparable between the base model, the RL teacher, and every steered student; DeepSeek-R1-Distill students keep thinking mode enabled, as their base model is trained that way. We always use each model’s own built-in chat template rather than a hand-written wrapper, and do not manually prepend an additional system message. Each domain uses its own answer-format instruction, which is also what the corresponding verifier parses. The examples below show user-message content only, with question denoting the raw problem statement. Role delimiters and model-specific thinking prefixes are supplied by the native chat template rather than manually inserted into the user message. For mathematical reasoning, the answer-extraction instruction is appended to the question:question Please reason step by step, and put your final answer within . For scientific reasoning, the instruction is placed first and the question is a multiple-choice item whose options are enumerated A–J, so that the verifier can match a single letter:Solve the following problem. Make sure to put the answer (and only answer) inside . question A: option_A B: option_B ... J: option_J For code generation, the instruction requires a single fenced Python block, which the sandbox extracts verbatim and executes against the hidden unit tests:question Write Python code to solve the problem. Present the code in ‘python Your code ‘ at the end. You need to think first then write the Python code. For instruction following, the constraints are stated in natural language at the head of the prompt and no separate reasoning instruction is added, since the verifiable constraints concern the response’s surface form:Please follow these instructions in your response: constraint_1 and constraint_2 and ... constraint_n. question Teacher rollouts for off-policy distillation are generated with the same model-specific prompt template as the corresponding on-policy runs. Prompt construction is therefore matched, while the two paradigms retain the distinct trajectory-sampling distributions and distillation objectives defined in Section B.1. Infrastructure. All training runs use 8×8× or 32×32× H20 96GB GPUs with FSDP for the actor and vLLM for generation. Vector-steering runs disable CUDA graph capture (enforce_eager=true) because the steering hook mutates hidden states inside the vLLM forward pass; LoRA and full-parameter runs leave CUDA graphs enabled. Rollout tensor parallelism is 44 for RL and 22 for distillation. When a fixed model is used for scoring, its parameters are offloaded to CPU between forward passes; this model serves as the reference policy for RL regularization or the teacher for distillation supervision, depending on the paradigm. The two commands below give the on-policy and off-policy distillation invocations for vector steering. The full-parameter and LoRA variants use the corresponding update parameterizations and hyperparameters in Table 2, with the configuration-specific generation-engine settings described above. On-Policy Distillation Command (vector steering) # examples/representation/4bopd_single.sh -> 4bopd.sh python3 -m verl.trainer.main_ppo \ algorithm.adv_estimator=grpo \ algorithm.use_kl_in_reward=False \ algorithm.rollout_correction.rollout_is=token \ algorithm.rollout_correction.rollout_is_threshold=5.0 \ algorithm.rollout_correction.rollout_rs=null \ algorithm.rollout_correction.bypass_mode=false \ data.train_files=/path/to/DeepMath-103K/train_filtered_level6.parquet \ data.val_files="[’/path/to/AIME2024/test.parquet’]" \ data.train_batch_size=1024 \ data.max_prompt_length=3072 \ data.max_response_length=16384 \ data.filter_overlong_prompts=True \ data.truncation=’error’ \ data.shuffle=True \ data.seed=42 \ data.return_raw_chat=True \ +data.apply_chat_template_kwargs.enable_thinking=False \ actor_rollout_ref.model.path=$MODEL_PATH \ +actor_rollout_ref.model.base_model_path=$MODEL_PATH \ +actor_rollout_ref.ref.model.path=$TEACHER_MODEL_PATH \ actor_rollout_ref.model.lora_rank=0 \ actor_rollout_ref.model.enable_trainable_token_vector=true \ actor_rollout_ref.model.trainable_token_vector_mode=single \ actor_rollout_ref.model.trainable_token_vector_num=1 \ actor_rollout_ref.model.trainable_token_vector_layer_start=5 \ actor_rollout_ref.model.trainable_token_vector_layer_end=20 \ actor_rollout_ref.model.trainable_token_vector_sampling_method=hypersphere \ actor_rollout_ref.model.trainable_token_vector_scale=0.1 \ actor_rollout_ref.model.trainable_token_vector_learnable_alpha=false \ actor_rollout_ref.model.trainable_token_vector_alpha_init=0.0 \ actor_rollout_ref.model.trainable_token_vector_curriculum=none \ actor_rollout_ref.model.trainable_token_vector_force_all_tokens=true \ actor_rollout_ref.model.use_remove_padding=True \ actor_rollout_ref.model.enable_gradient_checkpointing=True \ actor_rollout_ref.actor.optim.lr=1e-1 \ actor_rollout_ref.actor.optim.lr_warmup_steps_ratio=0.0 \ actor_rollout_ref.actor.policy_loss.only_reverse_kl_advantages=True \ actor_rollout_ref.actor.ppo_mini_batch_size=1024 \ actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=1 \ actor_rollout_ref.actor.use_kl_loss=True \ actor_rollout_ref.actor.kl_loss_coef=0 \ actor_rollout_ref.actor.kl_loss_type=low_var_kl \ actor_rollout_ref.actor.entropy_coeff=0 \ actor_rollout_ref.actor.ppo_max_token_len_per_gpu=32768 \ actor_rollout_ref.actor.fsdp_config.dtype=bfloat16 \ actor_rollout_ref.actor.fsdp_config.model_dtype=fp32 \ actor_rollout_ref.actor.fsdp_config.use_orig_params=True \ actor_rollout_ref.rollout.name=vllm \ actor_rollout_ref.rollout.calculate_log_probs=true \ actor_rollout_ref.rollout.n=1 \ actor_rollout_ref.rollout.temperature=1.0 \ actor_rollout_ref.rollout.top_p=1.0 \ actor_rollout_ref.rollout.tensor_model_parallel_size=2 \ actor_rollout_ref.rollout.gpu_memory_utilization=0.8 \ actor_rollout_ref.rollout.enforce_eager=true \ actor_rollout_ref.rollout.free_cache_engine=True \ actor_rollout_ref.rollout.max_num_batched_tokens=32768 \ actor_rollout_ref.rollout.val_kwargs.do_sample=True \ actor_rollout_ref.rollout.val_kwargs.temperature=1.0 \ actor_rollout_ref.rollout.val_kwargs.top_p=1.0 \ actor_rollout_ref.rollout.val_kwargs.n=4 \ actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu=1 \ actor_rollout_ref.ref.fsdp_config.param_offload=True \ reward_model.reward_manager=naive \ trainer.save_vector=true \ trainer.save_vector_dir=/path/to/trainable_vectors \ trainer.logger=’["console","wandb"]’ \ trainer.n_gpus_per_node=8 \ trainer.nnodes=1 \ trainer.test_freq=10 \ trainer.total_epochs=1000 $@ Off-Policy Distillation Command (vector steering) # Stage 1: dump teacher rollouts once python3 -m verl.trainer.main_generation \ model.path=$TEACHER_MODEL_PATH \ data.path=/path/to/DeepMath-103K/train_filtered_level6.parquet \ data.prompt_key=prompt \ data.n_samples=1 \ data.batch_size=1024 \ rollout.temperature=1.0 \ rollout.top_p=1.0 \ rollout.prompt_length=3072 \ rollout.response_length=16384 \ rollout.max_num_batched_tokens=32768 \ rollout.tensor_model_parallel_size=2 \ data.output_path=$OUTPUT_BASE/round_$i.parquet python3 examples/representation/merge_rollouts_to_sft.py \ --inputs $OUTPUT_BASE/round_*.parquet \ --output $OUTPUT_BASE/teacher_sft_all.parquet \ --prompt_key prompt --response_key response # Stage 2: distill off-policy on the frozen corpus python3 -m verl.trainer.main_ppo \ algorithm.adv_estimator=grpo \ algorithm.rollout_correction.rollout_is=null \ algorithm.use_kl_in_reward=False \ data.train_files=$OUTPUT_BASE/teacher_sft_all.parquet \ data.val_files="[’/path/to/AIME2024/test.parquet’]" \ data.train_batch_size=1024 \ data.max_prompt_length=3072 \ data.max_response_length=16384 \ data.return_raw_chat=True \ +data.apply_chat_template_kwargs.enable_thinking=False \ actor_rollout_ref.model.path=$MODEL_PATH \ +actor_rollout_ref.ref.model.path=$TEACHER_MODEL_PATH \ actor_rollout_ref.model.lora_rank=0 \ actor_rollout_ref.model.enable_trainable_token_vector=true \ actor_rollout_ref.model.trainable_token_vector_mode=single \ actor_rollout_ref.model.trainable_token_vector_num=1 \ actor_rollout_ref.model.trainable_token_vector_layer_start=5 \ actor_rollout_ref.model.trainable_token_vector_layer_end=20 \ actor_rollout_ref.model.trainable_token_vector_sampling_method=hypersphere \ actor_rollout_ref.model.trainable_token_vector_scale=0.1 \ actor_rollout_ref.model.trainable_token_vector_learnable_alpha=false \ actor_rollout_ref.model.trainable_token_vector_alpha_init=0.0 \ actor_rollout_ref.model.trainable_token_vector_curriculum=none \ actor_rollout_ref.model.trainable_token_vector_force_all_tokens=true \ actor_rollout_ref.actor.optim.lr=5e-2 \ actor_rollout_ref.actor.policy_loss.only_reverse_kl_advantages=true \ actor_rollout_ref.actor.ppo_mini_batch_size=1024 \ actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=1 \ actor_rollout_ref.actor.ppo_max_token_len_per_gpu=32768 \ actor_rollout_ref.actor.use_kl_loss=True \ actor_rollout_ref.actor.kl_loss_coef=0 \ actor_rollout_ref.actor.entropy_coeff=0 \ actor_rollout_ref.rollout.offline_teacher_rollout=true \ actor_rollout_ref.rollout.offline_teacher_data_path=$./teacher_sft_all.parquet \ actor_rollout_ref.rollout.offline_teacher_response_key=response \ actor_rollout_ref.rollout.name=vllm \ actor_rollout_ref.rollout.n=1 \ actor_rollout_ref.rollout.tensor_model_parallel_size=2 \ actor_rollout_ref.rollout.gpu_memory_utilization=0.5 \ actor_rollout_ref.rollout.enforce_eager=true \ actor_rollout_ref.ref.fsdp_config.param_offload=True \ reward_model.reward_manager=naive \ trainer.save_vector=true \ trainer.logger=’["console","wandb"]’ \ trainer.n_gpus_per_node=8 \ trainer.nnodes=1 \ trainer.test_freq=5 \ trainer.total_epochs=1000 $@ Figure 9: Vocabulary-space readout of steering vectors across network depth. (a) Normalized entropy of softmax(RMSNorm(L)U⊤)softmax(RMSNorm(v_L)W_U ). The dashed line at 0.94980.9498 denotes the mean entropy of normalized Gaussian directions passed through the same readout pipeline. (b) Highest-logit vocabulary items for steering vectors learned at representative layers. All vectors are normalized before readout. Appendix C Additional Experiments for Effective Manifold Capacity C.1 Vocabulary-Space Readout The layer-wise experiments in the main text show that fixed-vector steering recovers most of the RL-induced performance gain at lower and intermediate layers, but becomes substantially less effective near the output. To better understand this difference, we examine the vocabulary-space readout of steering vectors learned at different depths. This analysis characterizes how the learned directions align with output vocabulary items and how this alignment changes with injection depth. For a steering vector L∈ℝHv_L ^H learned at layer L, we bypass subsequent Transformer layers and directly apply RMS normalization followed by the unembedding matrix (Geva et al., 2022): ℓ(L)=URMSNorm(L),RMSNorm()=1H∑j=1Hvj2+ε,ε=10−6, (v_L)=W_URMSNorm(v_L), (v)= v 1H _j=1^Hv_j^2+ , =10^-6, (19) where U∈ℝV×HW_U ^V× H is the unembedding matrix, and H and V denote the hidden dimension and vocabulary size, respectively. The resulting readout distribution is L=softmax(ℓ(L)).p_L=softmax ( (v_L) ). (20) The learned vector norms vary across injection depths. RMS normalization reduces the influence of these magnitude differences, allowing the readout to primarily characterize the vocabulary alignment of each steering direction. We quantify the concentration of the readout distribution using normalized entropy: ℋ~(L)=−1logV∑t=1VpL,tlogpL,t. H(p_L)=- 1 V _t=1^Vp_L,t p_L,t. (21) Values closer to one indicate a more diffuse distribution, whereas lower values indicate greater concentration on a smaller set of vocabulary items. To establish a random-direction baseline, we sample isotropic Gaussian vectors with the same dimensionality as the steering vectors, normalize them to unit L2L_2 norm, and pass them through the same RMS normalization, unembedding, and softmax operations: (m)∼(,H),¯(m)=(m)‖(m)‖2,z^(m) (0,I_H), z^(m)= z^(m)\|z^(m)\|_2, (22) null(m)=softmax(URMSNorm(¯(m))).p_null^(m)=softmax (W_URMSNorm( z^(m)) ). (23) Averaging the normalized entropy over the M sampled directions gives ℋ~null=1M∑m=1Mℋ~(null(m))=0.9498. H_null= 1M _m=1^M H (p_null^(m) )=0.9498. (24) As shown in Fig. 9 (a), the normalized entropy remains close to the random baseline at lower and intermediate layers, indicating a diffuse vocabulary readout. The highest-logit items from L2 to L20 in Fig. 9(b) likewise consist mainly of heterogeneous subword fragments and symbols, without an apparent common semantic theme. Combined with the strong performance of steering at these depths, this observation shows that effective intermediate-layer interventions need not exhibit a concentrated direct preference for particular output tokens. This pattern is consistent with the role of downstream computation: as a shared additive correction propagates through the remaining layers, its interaction with different hidden states can produce context-dependent output adjustments. Near the output, the readout becomes increasingly concentrated. Beginning around L26, normalized entropy falls below the random baseline, while the highest-logit items increasingly include Wait, But, perhaps, and Alternatively. These tokens are associated with contrast, qualification, and reconsideration, suggesting that high-layer steering directions have more explicit vocabulary-level associations. The readout thus shifts from a diffuse pattern at lower and intermediate layers toward a concentrated set of discourse-related items near the output. This increase in vocabulary interpretability does not coincide with better steering performance. Instead, the main-text depth sweep shows that fixed-vector steering deteriorates at these higher layers. A direction can therefore have a clear vocabulary-level association while remaining insufficient for effective task adaptation. The issue is not simply whether a direction aligns with meaningful output tokens, but how that direction is applied across token positions. Expressions of contrast, qualification, or reconsideration are useful in different contexts, whereas a fixed steering vector applies the same additive correction at every position, without explicitly adjusting its strength to the current representation. Summary. The vocabulary readout provides a complementary view of the layer-wise results. Lower- and intermediate-layer steering remains effective despite its diffuse direct readout, whereas high-layer directions exhibit more concentrated vocabulary associations alongside weaker fixed-vector performance. This pattern suggests that direct vocabulary alignment alone does not account for steering effectiveness. One possible explanation is that the remaining layers allow earlier interventions to acquire context-dependent effects, while later interventions have less opportunity for such transformation. The gains from high-layer gating are consistent with this account, supporting input dependence as a useful factor in understanding the depth sensitivity of fixed-vector steering. Figure 10: Effect of single-vector steering on hidden-state geometry in Qwen3-4B (k=20k=20, H=2560H=2560), measured at layer L+1L+1 for injection at layer L. (a) Mean principal-subspace overlap between clean and steered activations. (b) Ratio of the top-k explained variance after and before steering. (c) Relative displacement of the activation centroid. Dashed lines denote the corresponding reference values. C.2 Local Representation Geometry under Single-Vector Steering Section 2.3 shows that single-vector steering recovers most of the RL-induced performance gain at lower and intermediate layers, but becomes less effective near the output. The training dynamics and gating results suggest that the expressiveness of a fixed additive correction contributes to this depth dependence. Here, we complement the performance analysis by examining changes in hidden-state geometry across injection depths, focusing on the leading principal subspaces, their associated variance, and the activation means. Experimental Setup. For each injection layer L, we collect hidden states from the base and steered models on the same held-out inputs. The steering vector is added to the output of layer L, and representations are measured at the output of layer L+1L+1. This captures the local effect of steering after one subsequent Transformer layer. Aggregating all valid token positions gives L∈ℝN×H,L′∈ℝN×H,X_L ^N× H, _L ^N× H, (25) where N is the number of token positions and H is the hidden dimension. The subscript L indexes the injection layer; both matrices contain activations measured at layer L+1L+1. Their respective means are L=1N∑i=1NL,i,L′=1N∑i=1NL,i′. μ_L= 1N _i=1^Nx_L,i, μ _L= 1N _i=1^Nx _L,i. (26) Figure 11: Multi-vector steering across all four task domains and model scales. The first vector is consistently the most effective, with subsequent directions showing gradually decreasing standalone performance; the decay is slower for larger models. We center each activation matrix using its own mean and independently compute low-rank singular value decompositions: L,c≈LLL⊤,L,c′≈L′L′L′⊤.X_L,c _L _LV_L , _L,c _L _LV _L . (27) The first k right singular vectors form the orthonormal principal-component bases L,k,L,k′∈ℝH×kV_L,k,V _L,k ^H× k. Writing ΣL,j _L,j and ΣL,j′ _L,j for the corresponding singular values, the covariance eigenvalues are λL,j=ΣL,j2N−1,λL,j′=ΣL,j′2N−1. _L,j= _L,j^2N-1, λ _L,j= _L,j^2N-1. (28) For Qwen3-4B, we use k=20k=20 and H=2560H=2560. Principal-Subspace Alignment. We measure the alignment between the clean and steered top-k principal subspaces as OverlapL,k=1k∑j=1kσj(L,k⊤L,k′),Overlap_L,k= 1k _j=1^k _j (V_L,k V _L,k ), (29) where σj(⋅) _j(·) denotes the j-th singular value. These singular values are the cosines of the principal angles between the two subspaces, so an overlap close to one indicates similar subspace orientations. As shown in Fig. 10(a), the overlap remains close to one for lower- and intermediate-layer injections. At these depths, the leading directions of activation variation therefore remain largely aligned after one layer of propagation. The overlap decreases toward the output, with the largest departures occurring at the latest injection sites. This indicates greater changes in the orientation of the leading subspace for these high-layer interventions. Variance Captured by the Leading Components. We next compare the total variance captured by the top-k components of the steered and clean activations: ρL,k=∑j=1kλL,j′∑j=1kλL,j. _L,k= _j=1^kλ _L,j _j=1^k _L,j. (30) The numerator and denominator use the leading eigenvalues of their respective covariance matrices. A ratio close to one indicates similar amounts of variance captured by the two principal subspaces. Fig. 10(b) shows that this ratio stays close to one at lower and intermediate injection depths. Along with the high subspace overlap, this suggests that steering at these depths introduces relatively small changes in both the orientation and the total variance of the leading components. The ratio departs further from one near the output, although the changes in captured variance remain modest. Thus, the reduced subspace alignment at the latest injection sites is accompanied by a comparatively small change in the variance captured by the leading components. Displacement of the Activation Centroid. To characterize changes in the mean representation, we measure the relative centroid displacement: DL=‖L′−L‖2‖L‖2+10−9.D_L= \| μ _L- μ_L \|_2 \| μ_L \|_2+10^-9. (31) This statistic expresses the change in the activation mean relative to the norm of the clean mean. As shown in Fig. 10 (c), relative centroid displacement is small for lower- and intermediate-layer injections and increases near the output, particularly at the latest injection sites. The strong performance of earlier-layer steering therefore coincides with only a small relative change in the mean representation at the next layer. High-layer steering produces a larger relative mean shift, despite its weaker task performance. Relation to the Layer-Wise Expressiveness Bottleneck. Effective steering at lower and intermediate layers is accompanied by limited changes in the measured local geometry. Near the output, reduced subspace alignment and larger centroid shifts coexist with weaker task performance. These observations are consistent with the expressiveness analysis in Section 2.3: high-layer degradation occurs despite measurable changes in the representations, and the gains from gating suggest that input-dependent modulation helps make these interventions more effective. Figure 12: Comparison of pre-collapse score trajectories with and without Alpha-Stabler at learning rates of (a) 10−410^-4, (b) 5×10−55× 10^-5, and (c) 10−510^-5. Shaded regions denote the first 5050 warm-up updates of Alpha-Stabler. Controlled and uncontrolled runs exhibit essentially the same score growth before collapse, indicating that Alpha-Stabler’s stabilization introduces almost no additional computational overhead. Algorithm 1 Alpha-Stabler: Warm-Up Calibration and PSI-Triggered Gradient Projection 1: Actor θ; frozen base θ0 _0; monitored layers S at 1/41/4, 1/21/2, 3/43/4 depth 2: Principal fraction q=0.10q=0.10; warm-up Twarm=50T_warm=50; interval M=3M=3; EMA β=0.95β=0.95; persistence p=3p=3; thresholds con=3c_on=3, csafe=2c_safe=2 3: RL optimizer and LR ηθ _θ; stabilizer ϵ>0ε>0; min. avg. squared token shift ηmin>0 _ >0 4: Trained actor θ (inference uses the actor alone) 5: Run TwarmT_warm unprojected RL updates; accumulate base activation covariance on actor rollouts 6: Every M updates, retain detached shifts in ℓX_ if N>0N>0 and ‖Xℓ‖F2/N>ηmin\|X_ \|_F^2/N> _ 7: Estimate Uℓ,rU_ ,r from the reference covariance, r=⌈qd⌉r= qd ; freeze bases 8: for each ℓ∈ do 9: ℓ←‖XUℓ,r‖F2/(‖X‖F2+ϵ):X∈ℓW_ ←\\|XU_ ,r\|_F^2/(\|X\|_F^2+ε):X _ \ 10: if ℓW_ is empty then 11: Stop and recalibrate 12: end if 13: mℓ←medianℓm_ _ ; sℓ←1.4826MAD(ℓ)+ϵs_ ← 1.4826\,MAD(W_ )+ε 14: τℓon←mℓ+consℓ _ ^on← m_ +c_ons_ ; τℓsafe←mℓ+csafesℓ _ ^safe← m_ +c_safes_ 15: if 0<τℓsafe<τℓon<10< _ ^safe< _ ^on<1 fails then 16: Stop and recalibrate 17: end if 18: aℓ←0a_ ← 0; nℓ←0n_ ← 0; ψ¯ℓ←mℓ ψ_ ← m_ 19: end for 20: Discard calibration records; t←Twarmt← T_warm 21: while training not complete do 22: Sample rollouts; compute rewards, advantages, and behavior log-probs 23: for each minibatch ℬB (one optimizer update) do 24: t←t+1t← t+1; clear parameter gradients 25: Actor forward for ℒRLL_RL; retain graph 26: if tmodM=0t M=0 then 27: Frozen-base activations on the same tokens; N←N← valid positions 28: for each ℓ∈ do 29: Xℓ←sg(Hℓ(θ,ℬ)−Hℓ(θ0,ℬ))X_ (H_ (θ;B)-H_ ( _0;B)) 30: if N>0N>0 and ‖Xℓ‖F2/N>ηmin\|X_ \|_F^2/N> _ then 31: ψℓ←‖XℓUℓ,r‖F2/(‖Xℓ‖F2+ϵ) _ ←\|X_ U_ ,r\|_F^2/(\|X_ \|_F^2+ε) 32: ψ¯ℓ←βψ¯ℓ+(1−β)ψℓ ψ_ ←β ψ_ +(1-β) _ 33: nℓ←nℓ+1n_ ← n_ +1 if ψ¯ℓ>τℓon ψ_ > _ ^on, else 00 34: aℓ←1a_ ← 1 if nℓ≥pn_ ≥ p; aℓ←0a_ ← 0 if ψ¯ℓ<τℓsafe ψ_ < _ ^safe; else keep aℓa_ 35: else 36: nℓ←0n_ ← 0 ⊳ keep ψ¯ℓ ψ_ , aℓa_ 37: end if 38: end for 39: end if 40: Fix flags during backward and optimizer step 41: Backprop ℒRLL_RL with G~ℓ(t)=Gℓ(t)−aℓ(t)(Gℓ(t)Uℓ,r)Uℓ,r⊤ G_ ^(t)=G_ ^(t)-a_ ^(t)(G_ ^(t)U_ ,r)U_ ,r 42: Update θ with LR ηθ _θ 43: end for 44: end while 45: return θ 46: Note: Hℓ(θ,ℬ)H_ (θ;B) stacks layer-ℓ activations at valid tokens in ℬB; sgsg is stop-gradient. Appendix D Theory of Low-Dimensional RLVR Steering We develop a conditional theoretical account of when RLVR-induced output changes admit a compact activation-space intervention. We study recoverability within a specified intervention class, rather than the intrinsic dimension of the full RL parameter or activation trajectory. Part I formulates local distillation as Fisher-weighted least squares in Section D.2, gives sufficient conditions for shared teacher-induced shifts in Section D.3, and derives a recovery identity and a three-constant lower bound in Section D.4. Part I characterizes sequentially constrained constant-vector fitting in Section D.5 and distinguishes activation variance from output sensitivity in Section D.6. Part I analyzes Alpha-Stabler’s fixed-reference PSI statistic and backward gradient projection in Section D.7. Section D.8 summarizes the results and their theoretical or empirical scope. Two relations organize the recovery analysis: ρ=SNR1+SNRandρ≥κ(1+η)(1+ε).ρ= 1+ ρ≥ κ(1+η)(1+ ). Here ε bounds sensitivity-weighted principal energy relative to complement energy, η measures residual energy relative to a shared complement component, and κ measures consistency of its amplitude. The quantity ρ is the fraction of locally representable target energy explained by an optimal constant intervention, as defined in Definition D.2; it is not a task-score gain recovery rate. Informal preview Distillation into a frozen student locally amounts to fitting a hidden-state correction to the teacher’s output distribution. Logit linearization and a quadratic KL expansion yield a weighted least-squares problem with context-dependent target (x) (x) (Proposition D.1). An idealized KL-regularized RL optimum provides a separate account of how shared structure may arise: exponential reward tilting (Lemma D.2) determines next-token probability ratios through continuation-success probabilities (Lemma D.3). Shared success profiles, a small relative logit residual, and a consistent representation pullback then give a shared correction with controlled residual (Lemma D.4, Corollary D.1, and Lemma D.5). These are additional structural conditions, not consequences of reward binarity alone. Under a constant metric, the optimal fixed vector is the mean target shift, and its recovery is high when that mean dominates context-dependent fluctuations (Theorem D.1). Table 3 summarizes the quantities entering the sufficient lower bound. Table 3: Geometric quantities entering the sufficient recovery bound. Their empirical counterparts require matching shift definitions, distributions, and metrics. Const. Name Meaning Related analysis ε principal intrusion Upper bound on target principal-to-complement energy in the sensitivity metric PC-energy measurements under compatible metrics and target definitions η complement residual Residual-to-shared energy within the complement Context dependence and fixed-vector approximation κ amplitude consistency 1/(1+CV(a)2)1/(1+ (a)^2) Consistency of the shared target amplitude Theorem D.2 gives ρ≥κ/((1+η)(1+ε))ρ≥κ/((1+η)(1+ )). The bound quantifies recovery under the stated localization and shared-component conditions. These conditions are motivated by the empirical findings; their validity and quantitative values for a particular teacher require a separate justification. Part I. Single-Vector Steering: Conditions for High Recovery Section D.1 introduces the local setting. Sections D.2 and D.3 derive the fitting problem and sufficient conditions for shared teacher structure. Section D.4 quantifies the optimal constant-vector fit. D.1 Setup and notation We consider a frozen pretrained language model with decoder layers fℓ=1L\f_ \_ =1^L, hidden width d, vocabulary size V, and unembedding matrix U∈ℝV×dU ^V× d, which may be tied or untied. A context x∼x consists of a token position and its prefix under a specified distillation distribution D. The base model has hidden states and logits ℓ(x)∈ℝd,0(x)=UL(x)∈ℝV,h_ (x) ^d, _0(x)=Uh_L(x) ^V, with p0(⋅∣x)=softmax(0(x))p_0(· x)= (z_0(x)). Here L(x)h_L(x) is the representation supplied to the unembedding, including final normalization. For an intervention before normalization, the downstream Jacobian includes that operation. The teacher induces logits T(x)z_T(x) and distribution pT(⋅∣x)p_T(· x). Vector steering adds a context-invariant offset ℓ∈ℝd δ_ ^d at each controlled layer ℓ∈⊆1,…,L \1,…,L\: ℓ(x)↦ℓ(x)+ℓh_ (x) _ (x)+ δ_ , with all backbone weights frozen. We first analyze a single injection site and extend to multiple sites in Remark D.2. For a positive-semidefinite matrix A, write ‖v‖A2=v⊤Av\|v\|_A^2=v Av and ⟨u,v⟩A=u⊤Av u,v _A=u Av. The notation ∥⋅∥2\|·\|_2 denotes the Euclidean vector norm or matrix spectral norm; λi(⋅) _i(·) and σi(⋅) _i(·) denote eigenvalues and singular values in descending order. All logarithms are natural and all vectors are columns. Bold symbols denote named vectors, while unbold subscripts such as svs_v denote scalar coordinates; matrices such as JxJ_x, FxF_x, GxG_x, and BxB_x use uppercase notation without requiring boldface. The symbol V1_V denotes the all-ones vector in ℝVR^V. Generic arguments such as u and v are understood from their stated spaces. We analyze the expected forward-KL objective ℒ()=x∼KL(pT(⋅∣x)∥p(⋅∣x)).L( δ)=E_x \! (p_T(· x)\, \|\,p_ δ(· x) ). (32) The context distribution is fixed for each fitting problem. All expected quadratic energies used below are finite. Let ℰ⊆ℝdE ^d be a fixed working subspace with orthonormal basis T∈ℝd×hT ^d× h. Interventions and target corrections lie in ℰE. Inverses are always taken for the compressed matrices, such as T⊤GxT G_xT, assumed positive definite; positive definiteness on ℰE does not require GxG_x to be invertible on all of ℝdR^d. Setting T=IdT=I_d gives the full-space formulas when their inverses exist. For singular metrics, use a common identifiable subspace with a consistent identification of targets across contexts. For results that decompose vectors using P and Q, additionally assume that ℰE is invariant under these projectors. This ensures that projected targets, means, and corrections remain in the space on which the stated metric bounds apply. Objects and their dependence on context. • Context-dependent values and operators include ℓ(x)h_ (x), T(x)z_T(x), (x)t(x), (x) (x), a(x)a(x), (x) ξ(x), and Jx,Fx,Gx,BxJ_x,F_x,G_x,B_x. • Context-invariant objects include trainable vectors ℓ δ_ and ℓ(k)v_ ^(k), shared directions s and w, averaged metric G, mean target shift μ, projectors P, Q, and Πvocab _vocab, and the constants in the bounds. A trainable vector changes during optimization but is shared across contexts at each step. • Distinct shift objects are the learned intervention δ, the teacher-induced fitting target (x)=Bx(x) (x)=B_xt(x), and the actual actor-to-base activation differences collected in Xℓ(t)X_ ^(t) in Part I. They are not identified with one another. • Sequence and training indices are introduced separately: Section D.3 distinguishes prompts q and complete responses Y from contexts x=(q,y<t)x=(q,y_<t); Part I uses t for a parameter-update index. Thus, the local fitting problem approximates a context-dependent target family (x)\ (x)\ by one constant vector in an output-sensitivity metric. Table 4: Recurring symbols for the distillation analysis. Context dependence is indicated by (x)(x) or a subscript x; Part I separately introduces batched training quantities. Symbol Type Meaning x∼x context Token position and prefix, x=(q,y<t)x=(q,y_<t) q, Y sequence level Prompt and complete response, including termination ℓ(x)∈ℝdh_ (x) ^d ctx.-dep. Layer-ℓ hidden state 0(x),T(x)∈ℝVz_0(x),z_T(x) ^V ctx.-dep. Base and teacher logits (x)=T(x)−0(x)t(x)=z_T(x)-z_0(x) ctx.-dep. Teacher logit shift modulo V1_V; Remark D.3 Fx∈ℝV×VF_x ^V× V ctx.-dep. Fisher information at teacher logits; Equation (38) Jx∈ℝV×dJ_x ^V× d ctx.-dep. Downstream logit Jacobian at the base injection state Gx=Jx⊤FxJxG_x=J_x F_xJ_x ctx.-dep. Pullback sensitivity metric Bx=T(T⊤GxT)−1T⊤Jx⊤FxB_x=T(T G_xT)^-1T J_x F_x ctx.-dep. Least-squares pullback into ℰE (x)=Bx(x) (x)=B_xt(x) ctx.-dep. Locally best target representation correction a(x)∈ℝ≥0a(x) _≥ 0 ctx.-dep. Amplitude in the specified shared-component decomposition (x)∈ℝd ξ(x) ^d ctx.-dep. Representation residual within the complement ν0(x)∈[0,1] _0(x)∈[0,1] ctx.-dep. Conditional success probability; Equation (42) Zβ(x)Z_β(x) ctx.-dep. Conditional reward-tilt normalizer b(x),a0(x),(x)b(x),a_0(x),e(x) ctx.-dep. Baseline, modulation, and residual of a success profile ℓ,ℓ(k)∈ℝd δ_ ,v_ ^(k) ^d invariant Learned constant steering vectors ∈ℝVs ^V, ∈ℝdw ^d invariant Shared logit and representation directions G=x[Gx]G=E_x[G_x] invariant Averaged sensitivity metric =x[(x)] μ=E_x[ (x)] invariant Mean target; optimal constant vector when Gx=G_x=G C,P,Q=I−PC,P,Q=I-P invariant Reference activation covariance and its spectral projectors Πvocab=I−VV⊤/V _vocab=I-1_V1_V /V invariant Vocabulary-centering projector, distinct from P cβ=e1/β−1c_β=e^1/β-1 invariant Tilt strength for binary rewards ε,η,κ,ρ ,η,κ,ρ invariant Intrusion bound, residual ratio, amplitude consistency, and energy recovery ηlogit,θ,ζB,ζR _logit,θ, _B, _R invariant Relative logit-residual and pullback bounds Assumptions. The local reduction uses Assumption D.1. Constant-metric identities use the exact case of Assumption D.3. Assumption D.2 specifies shared teacher structure, while Assumptions D.4–D.5 supply the additional geometry for the three-constant bound. Part I’s projection statements require only their own explicitly stated conditions. Assumption D.1 (Local / trust-region regime). Fix a teacher, base model, and context distribution. The KL approximation is used only for interventions in a specified neighborhood of the base state. Write Jx:=∂(x)∂ℓ(x)|base∈ℝV×d,(x)=0(x)+Jx+z(x,).J_x:= . (x) _ (x) |_base ^V× d, _ δ(x)=z_0(x)+J_x δ+e_z(x, δ). The Jacobian includes all downstream operations. Assume differentiability with ‖z(x,)‖2=o(‖2)\|e_z(x, δ)\|_2=o(\| δ\|_2), and a bounded third derivative of the log-partition function along the segments between the teacher and intervened logits. For δ(x)=Jx−(x)d_δ(x)=J_x δ-t(x), also assume that the combined network-linearization and KL-expansion error is o(‖δ(x)‖22)o(\|d_δ(x)\|_2^2) in the local regime under consideration. A sufficient condition is δ(x)→0d_δ(x)→ 0 and ‖z(x,)‖2=o(‖δ(x)‖2)\|e_z(x, δ)\|_2=o(\|d_δ(x)\|_2), with uniform or integrable bounds that justify taking expectations of the remainder. Little-o statements concern this joint local regime; differentiability in δ alone does not imply small relative error when JxJ_x δ nearly cancels (x)t(x). Assumption D.2 (Shared logit component). The teacher logit shift (x)=T(x)−0(x)∈ℝVt(x)=z_T(x)-z_0(x) ^V admits (x)=a(x)⏟shared direction+⟂(x),t(x)= a(x)s_shared direction+r_ (x), (33) where s is fixed, ‖2=1\|s\|_2=1, V⊤=01_V s=0, and a(x)≥0a(x)≥ 0. The residual obeys ⟨,⟂(x)⟩Fx=0 ,r_ (x) _F_x=0 at every context and x‖⟂(x)‖Fx2≤ηlogitx‖a(x)‖Fx2,ηlogit≪1.E_x\|r_ (x)\|_F_x^2≤ _logit\,E_x\|a(x)s\|_F_x^2, _logit 1. (34) Remark D.3 justifies the centered gauge. Section D.3 provides sufficient success-profile conditions for this decomposition; it is not assumed to hold merely because rewards are binary. Assumption D.3 (Metric homogeneity). Let G¯=x∼[Gx] G=E_x [G_x] and abbreviate G=G¯G= G, with T⊤GT GT positive definite. Exact homogeneous-model identities assume Gx=G_x=G as quadratic forms on ℰE. When approximation is invoked, its quantitative form is (1−ζ)G⪯Gx⪯(1+ζ)G(1-ζ)G G_x (1+ζ)G on ℰE, uniformly over the considered contexts, for 0≤ζ<10≤ζ<1. Remark D.1 treats this perturbation for a fixed target family. Consistency of BxB_xs is a separate condition, quantified by ζB _B in Lemma D.5; it does not follow from metric homogeneity. Assumption D.4 (Off-principal localization). Let P be the Euclidean orthogonal projector onto the top-m principal subspace of the reference activation covariance C in Definition D.3, and let Q=I−PQ=I-P. Here m is a general retained rank; the monitoring construction in Section D.7 uses the particular rank r=⌈0.10d⌉r= 0.10d . Assume compatibility with the sensitivity metric, P⊤GQ=0P GQ=0 on the working subspace. Then x∥(x)∥G2=x∥P(x)∥G2+x∥Q(x)∥G2.E_x (x) _G^2=E_x P (x) _G^2+E_x Q (x) _G^2. (35) Assume in addition the target-energy bound x∥P(x)∥G2≤εx∥Q(x)∥G2,0≤ε≪1.E_x P (x) _G^2≤ \,E_x Q (x) _G^2, 0≤ 1. The operator condition gives a G-orthogonal split for every vector in ℰE; equality of averaged energies for one distribution is weaker. Neither covariance principal structure nor a low Euclidean PC fraction alone guarantees these sensitivity-weighted conditions. Remark D.1 treats approximate compatibility. Assumption D.5 (Shared direction in the complement). There is a fixed ∈ℰ∩range(Q)w (Q) with ∥G=1 _G=1 such that Q(x)=a(x)⏟shared complement component+(x),a(x)≥0,Q (x)= a(x)w_shared complement component+ ξ(x), a(x)≥ 0, (36) where (x)∈ℰ∩range(Q) ξ(x) (Q) and x[(x)]=,x[a(x)⟨,(x)⟩G]=0.E_x[ ξ(x)]=0, _x\! [a(x) , ξ(x) _G ]=0. (37) Assume 0<x[a(x)2]<∞0<E_x[a(x)^2]<∞ and finite target energy, and define κ:=(x[a(x)])2x[a(x)2]∈(0,1],η:=x∥(x)∥G2x[a(x)2]≥0.κ:= (E_x[a(x)])^2E_x[a(x)^2]∈(0,1], η:= E_x ξ(x) _G^2E_x[a(x)^2]≥ 0. Then κ=1κ=1 exactly when a(x)a(x) is constant almost surely, and κ=1/(1+CV(a)2)κ=1/(1+ (a)^2) with CV(a)2=Varx(a(x))/(x[a(x)])2 (a)^2= _x(a(x))/(E_x[a(x)])^2. Nonnegativity fixes the shared amplitude’s sign; η is a residual-energy ratio, not a rank. The two moment conditions in Equation (37) have distinct roles: zero mean identifies the shared mean, while weighted orthogonality removes the mixed energy term. Pointwise orthogonality is sufficient for the latter but is not required. The amplitude here belongs to the representation-space decomposition. It is identified with a fixed positive rescaling of the logit-space amplitude only under the compatible pullback conditions discussed in Remark D.5; otherwise the two decompositions use separately defined amplitudes. Remark D.1 (Perturbations of the homogeneous model). (i) Varying metrics. Let G¯ℰ=T⊤G¯T G_E=T GT and ℰ=T⊤x[Gx(x)]b_E=T E_x[G_x (x)]. The quadratic optimum is ⋆=TG¯ℰ−1ℰ δ =T G_E^-1b_E, with explained energy S=ℰ⊤G¯ℰ−1ℰS=b_E G_E^-1b_E. For Evar=x‖(x)‖Gx2>0E_var=E_x\| (x)\|_G_x^2>0, its recovery is ρvar=S/Evar _var=S/E_var. For a fixed target family and the metric bounds in Assumption D.3, comparison with constant-metric recovery ρG _G yields 1−ζ1+ζ(1−ρG)≤1−ρvar≤1+ζ1−ζ(1−ρG). 1-ζ1+ζ(1- _G)≤ 1- _var≤ 1+ζ1-ζ(1- _G). Indeed, the metric inequalities bound the residual objective for every admissible constant vector and therefore its infimum, as well as the total target energy. Comparing different target families requires an additional target-perturbation argument. (i) Approximately orthogonal splits. Replace exact compatibility by |⟨u,v⟩G|≤γ‖u‖G‖v‖G| u,v _G|≤γ\|u\|_G\|v\|_G for u∈ℰ∩range(P)u (P) and v∈ℰ∩range(Q)v (Q), with 0≤γ<10≤γ<1. Then (1−γ)(‖Pz‖G2+‖Qz‖G2)≤‖z‖G2≤(1+γ)(‖Pz‖G2+‖Qz‖G2).(1-γ)(\|Pz\|_G^2+\|Qz\|_G^2)≤\|z\|_G^2≤(1+γ)(\|Pz\|_G^2+\|Qz\|_G^2). This follows by expanding the cross term and using 2ab≤a2+b22ab≤ a^2+b^2. Under the remaining conditions of Theorem D.2, bounding both the mean energy and total target energy gives ρ≥1−γ1+γκ(1+η)(1+ε)ρ≥ 1-γ1+γ κ(1+η)(1+ ). (i) Distinct residual and metric conditions. Corollary D.1 bounds the logit residual, and Lemma D.5 bounds a representation residual relative to a specified direction. Applying these estimates to η additionally requires localization, zero residual mean, and weighted orthogonality as in Assumption D.5, or explicit bounds on their deviations. Corollary D.4 compares Euclidean and sensitivity-weighted intrusion only for the same collection of shifts. D.2 Distillation reduces to weighted least squares The categorical distribution has natural parameter ∈ℝVz ^V, log-partition function A()=log∑v=1VezvA(z)= _v=1^Ve^z_v, and mean map ∇A()=softmax()∇ A(z)= (z). Writing p=softmax()p= (z), its Hessian is F()=∇2A()=diag(p)−pp⊤⪰0.F(z)=∇^2A(z)= (p)-p 0. (38) Throughout, Fx=F(T(x))F_x=F(z_T(x)) is evaluated at teacher logits. It satisfies FxV=F_x1_V=0. Lemma D.1 (KL as a Bregman divergence and its local quadratic form). For ,′∈ℝVz,z ^V, KL(softmax(′)∥softmax())=DA(∥′) ( (z )\,\|\, (z))=D_A(z\,\|\,z ), where DAD_A is the Bregman divergence generated by A. Under Assumption D.1, KL(pT(⋅∣x)∥p(⋅∣x))=12(Jx−(x))⊤Fx(Jx−(x))+o(∥Jx−(x)∥22). \! (p_T(· x)\, \|\,p_ δ(· x) )= 12 (J_x δ-t(x) ) F_x (J_x δ-t(x) )+o (\|J_x δ-t(x)\|_2^2 ). (39) Proof. The identity follows by substituting A and ∇A∇ A into DAD_A. Set d=Jx−(x)d=J_x δ-t(x) and e=z(x,)e=e_z(x, δ), so the actual logit difference from the teacher is d+ed+e. Taylor expansion at T(x)z_T(x) gives KL(pT∥p)=12d⊤Fxd+d⊤Fxe+12e⊤Fxe+RA(d+e). (p_T\|p_ δ)= 12d F_xd+d F_xe+ 12e F_xe+R_A(d+e). If LAL_A bounds the third derivative of A in trilinear operator norm along the logit segment, then |KL(pT∥p)−12d⊤Fxd|≤∥Fx∥2∥d∥2∥e∥2+12∥Fx∥2∥e∥22+LA6∥d+e∥23. | (p_T\|p_ δ)- 12d F_xd |≤\|F_x\|_2\|d\|_2\|e\|_2+ 12\|F_x\|_2\|e\|_2^2+ L_A6\|d+e\|_2^3. The combined remainder condition yields Equation (39); the integrability condition permits the corresponding expected expansion. ∎ Definition D.1 (Representational metric and target shift). For each context, define Gx:=Jx⊤FxJx∈ℝd×d,Bx:=T(T⊤GxT)−1T⊤Jx⊤Fx∈ℝd×V,(x):=argmin∈ℰ‖Jx−(x)‖Fx2=Bx(x), gatheredG_x:=J_x F_xJ_x ^d× d, B_x:=T(T G_xT)^-1T J_x F_x ^d× V,\\[2.0pt] (x):= _u \|J_xu-t(x)\|_F_x^2=B_xt(x), gathered where T⊤GxT G_xT is positive definite. When T=IdT=I_d, Bx=Gx−1Jx⊤FxB_x=G_x^-1J_x F_x. The metric measures local output sensitivity; (x) (x) is the best linearized hidden-state correction for reproducing the teacher shift. This target is context-dependent even though the learned intervention is constant. It is a fitting target, not an observed teacher-minus-base hidden-state difference. Proposition D.1 (Distillation as local weighted least squares). Under Assumption D.1, the local quadratic surrogate is ℒquad()=12x∼(−(x))⊤Gx(−(x))+Cirr,L_quad( δ)= 12E_x ( δ- (x)) G_x( δ- (x))+C_irr, (40) where Cirr≥0C_irr≥ 0 is independent of δ and is the residual teacher-logit energy not fitted by the local Jacobian on ℰE, measured in the Fisher seminorm. The KL objective differs by the expected remainder in Lemma D.1. Proof. Write (x)=Jx(x)+xt(x)=J_x (x)+r_x. The least-squares normal equations give T⊤Jx⊤Fxx=0T J_x F_xr_x=0. Since −(x)∈ℰ δ- (x) , ‖Jx−(x)‖Fx2=‖Jx(−(x))‖Fx2+‖x‖Fx2.\|J_x δ-t(x)\|_F_x^2=\|J_x( δ- (x))\|_F_x^2+\|r_x\|_F_x^2. The first term is ‖−(x)‖Gx2\| δ- (x)\|_G_x^2. Taking expectations proves Equation (40), with Cirr=12x‖x‖Fx2C_irr= 12E_x\|r_x\|_F_x^2. ∎ Remark D.2 (Multiple injection sites). Stack the steering vectors and use the joint Jacobian from all injection sites to the logits. Equation (40) holds with the full Gx=Jx⊤FxJxG_x=J_x F_xJ_x, including cross-layer blocks; it is not in general a sum of independent layer-wise fitting problems. Injection at the unembedding input gives Jx=UJ_x=U, whereas earlier injection includes normalization and all other downstream operations. Local surrogate versus training dynamics. The reduction describes a fixed-distribution local objective. At a fixed context, forward and reverse KL have the same second-order Fisher term around the teacher, but generally differ beyond this local expansion. Changing on-policy prefix distributions or using an approximate token-level training gradient changes the optimization problem. Consequently, Equation (40) is not an identity for the complete OPD training trajectory or for every off-policy surrogate. Moreover, the unconstrained optimum of the quadratic model need not remain in the neighborhood where that model is accurate. A small teacher–student output gap alone does not ensure this: the Jacobian, its conditioning, and the intervention size also matter. Section D.4 makes the distinction between unconstrained surrogate recovery and locally admissible recovery explicit. D.3 The RLVR teacher shift: exponential tilting and shared structure We derive the complete-response optimum of an idealized KL-regularized RL objective and then its next-token shifts. The analysis separates sequence rewards, complete-response probability ratios, and token-level corrections. Shared structure requires conditions on continuation-success profiles and on the student’s pullback. Sequence notation and optimization setting. Let q be a prompt, Y a complete response including termination, and x=(q,y<t)x=(q,y_<t) a next-token context. Responses terminate almost surely under the base policy. Rewards are bounded, β>0β>0, and policies are supported on the base policy’s support. The symbol π⋆π denotes the optimizer of this stated objective. Empirical teacher checkpoints instead enter the local fitting analysis through their actual logit shifts (x)t(x). Lemma D.2 (KL-regularized RL optimum as an exponential tilt). For a fixed prompt q, the objective maxπY∼π(⋅∣q)[r(q,Y)]−βKL(π(⋅∣q)∥p0(⋅∣q)),β>0, _π\ E_Y π(· q)[r(q,Y)]-β\, (π(· q)\, \|\,p_0(· q) ), β>0, is strictly concave on its finite-KL domain and has the unique maximizer π⋆(Y∣q)=p0(Y∣q)er(q,Y)/βZβ(q),Zβ(q)=Y∼p0(⋅∣q)[er(q,Y)/β].π (Y q)= p_0(Y q)e^r(q,Y)/βZ_β(q), Z_β(q)=E_Y p_0(· q)[e^r(q,Y)/β]. (41) Equivalently, log(π⋆(Y∣q)/p0(Y∣q))=r(q,Y)/β−logZβ(q) (π (Y q)/p_0(Y q))=r(q,Y)/β- Z_β(q) on the support. Proof. Bounded rewards make Zβ(q)Z_β(q) positive and finite, and the proposed optimizer has finite KL. Substitution rewrites the objective as βlogZβ(q)−βKL(π(⋅∣q)∥π⋆(⋅∣q))β Z_β(q)-β (π(· q)\|π (· q)). Nonnegativity of KL gives the unique maximum. Strict concavity follows from strict convexity of KL and linearity of the reward term. ∎ Lemma D.3 (From complete responses to next-token probabilities). For x=(q,y<t)x=(q,y_<t) with positive base prefix probability, define Zβ(x):=Y∼p0(⋅∣x)[er(q,Y)/β],ν0(x):=PrY∼p0(⋅∣x)(r(q,Y)=1),Z_β(x):=E_Y p_0(· x)[e^r(q,Y)/β], _0(x):= _Y p_0(· x)(r(q,Y)=1), (42) where complete responses extend the prefix. For a supported next token v, write xvxv for the extended context. Then π⋆(v∣x)=p0(v∣x)Zβ(xv)Zβ(x).π (v x)=p_0(v x) Z_β(xv)Z_β(x). (43) For binary rewards r∈0,1r∈\0,1\ and cβ=e1/β−1>0c_β=e^1/β-1>0, logπ⋆(v∣x)p0(v∣x)=log[1+cβν0(xv)]−log[1+cβν0(x)]. π (v x)p_0(v x)= [1+c_β _0(xv)]- [1+c_β _0(x)]. (44) At termination the conditional quantities are evaluated on the resulting complete response. Proof. Summing Equation (41) over responses extending a prefix gives π⋆(y<t∣q)=p0(y<t∣q)Zβ(x)Zβ(q).π (y_<t q)=p_0(y_<t q) Z_β(x)Z_β(q). Taking the ratio for y<tvy_<tv and y<ty_<t proves Equation (43). For binary rewards, Zβ(x)=1−ν0(x)+e1/βν0(x)=1+cβν0(x)Z_β(x)=1- _0(x)+e^1/β _0(x)=1+c_β _0(x), which gives Equation (44). ∎ Thus, relative next-token changes depend on candidate-token continuation-success probabilities, not just the terminal reward of an already observed response. The tower property gives ν0(x)=∑vp0(v∣x)ν0(xv) _0(x)= _vp_0(v x) _0(xv). Neither this identity nor binary rewards imposes a common direction across different contexts. Remark D.3 (Softmax gauge and centered logits). Since softmax(+cV)=softmax() (z+c1_V)= (z), logits are defined modulo the all-ones direction. Choose the centered gauge Πvocab=I−VV⊤/V _vocab=I-1_V1_V /V. Because FxV=F_x1_V=0, FxΠvocab=Fx,‖Πvocab‖Fx=‖Fx,BxΠvocab=Bxfor all ∈ℝV.F_x _vocab=F_x, \| _vocabu\|_F_x=\|u\|_F_x, B_x _vocabu=B_xu all u ^V. (45) Centering preserves Fisher seminorms, Fisher inner products, and (x) (x). For the idealized teacher with finite logits, we may use (x)=Πvocablog(π⋆(⋅∣x)/p0(⋅∣x))t(x)= _vocab (π (· x)/p_0(· x)) and choose V⊤=01_V s=0. The vocabulary-centering projector Πvocab _vocab is distinct from the activation-principal projector P. Lemma D.4 (A sufficient condition for a shared logit component). Let ∈ℝVs ^V be fixed, ‖2=1\|s\|_2=1, and V⊤=01_V s=0. Suppose the continuation-success profiles obey ν0(xv)=b(x)+a0(x)sv+ev(x),b(x)∈[0,1],a0(x)≥0,ν0(xv)∈[0,1], _0(xv)=b(x)+a_0(x)s_v+e_v(x), b(x)∈[0,1], a_0(x)≥ 0, _0(xv)∈[0,1], (46) where b(x)b(x) is token-independent, svs_v is the v-th scalar coordinate of s, and (x)e(x) is a residual profile with coordinates ev(x)e_v(x). Define (x)=a0(x)+(x)d(x)=a_0(x)s+e(x). Then Πvocablogπ⋆(⋅∣x)p0(⋅∣x)=a^(x)+^(x),a^(x)=cβa0(x)1+cβb(x)≥0, _vocab π (· x)p_0(· x)= a(x)s+ r(x), a(x)= c_βa_0(x)1+c_βb(x)≥ 0, (47) with coordinatewise logarithms and ratios, and ‖^(x)‖2≤cβ‖(x)‖2+cβ22‖(x)‖42,‖(x)‖42=(∑v|dv(x)|4)1/2.\| r(x)\|_2≤ c_β\|e(x)\|_2+ c_β^22\|d(x)\|_4^2, \|d(x)\|_4^2= ( _v|d_v(x)|^4 )^1/2. (48) Proof. For f(z)=log(1+cβz)f(z)= (1+c_βz) on [0,1][0,1], |f′(z)|≤cβ|f (z)|≤ c_β and |f′(z)|≤cβ2|f (z)|≤ c_β^2. Taylor expansion between b(x)b(x) and b(x)+dv(x)b(x)+d_v(x), both in that interval, yields f(b(x)+dv(x))=f(b(x))+f′(b(x))dv(x)+Rv(x),|Rv(x)|≤12cβ2|dv(x)|2.f(b(x)+d_v(x))=f(b(x))+f (b(x))d_v(x)+R_v(x), |R_v(x)|≤ 12c_β^2|d_v(x)|^2. Centering Equation (44) removes the token-independent terms. Since Πvocab= _vocabs=s, ^(x)=Πvocab(f′(b(x))(x)+R(x)) r(x)= _vocab(f (b(x))e(x)+R(x)). The projector is a Euclidean contraction, and ‖R(x)‖2≤12cβ2‖(x)‖42\|R(x)\|_2≤ 12c_β^2\|d(x)\|_4^2. These inequalities prove Equation (48). ∎ The bound is absolute. Obtaining high relative alignment additionally requires a residual small compared with a^(x)‖Fx a(x)\|s\|_F_x, as specified next; a small absolute residual alone is insufficient when the shared amplitude is small. Corollary D.1 (Fisher-orthogonal shared-component decomposition). Assume Lemma D.4, ‖Fx>0\|s\|_F_x>0, and 0≤θ<10≤θ<1 such that ‖^(x)‖Fx≤θa^(x)‖Fxat every considered context.\| r(x)\|_F_x≤θ a(x)\|s\|_F_x every considered context. (49) For (x)=a^(x)+^(x)t(x)= a(x)s+ r(x), define a(x):=⟨,(x)⟩Fx‖Fx2,⟂(x):=(x)−a(x).a(x):= ,t(x) _F_x\|s\|_F_x^2, _ (x):=t(x)-a(x)s. (50) Then a(x)≥(1−θ)a^(x)≥0a(x)≥(1-θ) a(x)≥ 0, ⟨,⟂(x)⟩Fx=0 ,r_ (x) _F_x=0, and x‖⟂(x)‖Fx2≤θ2(1−θ)2x‖a(x)‖Fx2.E_x\|r_ (x)\|_F_x^2≤ θ^2(1-θ)^2\,E_x\|a(x)s\|_F_x^2. (51) This meets Assumption D.2’s small-residual requirement when θ is sufficiently small. Proof. By Equation (50), a(x)=a^(x)+⟨,^(x)⟩Fx/‖Fx2a(x)= a(x)+ , r(x) _F_x/\|s\|_F_x^2. Cauchy–Schwarz bounds the second term in absolute value by θa^(x)θ a(x). The vector a(x)a(x)s minimizes the Fisher squared distance to (x)t(x) over the one-dimensional span, so ‖⟂(x)‖Fx≤‖(x)−a^(x)‖Fx≤θa^(x)‖Fx≤θ1−θa(x)‖Fx.\|r_ (x)\|_F_x≤\|t(x)- a(x)s\|_F_x≤θ a(x)\|s\|_F_x≤ θ1-θa(x)\|s\|_F_x. Squaring and averaging proves the bound. Orthogonality follows directly from the definition, and Remark D.3 ensures gauge invariance. ∎ Remark D.4 (Small positive β and uniform residual bounds). A multiplicative success profile provides a bound without diverging constants as β↓0β 0, under an additional uniform success condition. Suppose ν0(xv)≥νmin>0 _0(xv)≥ _ >0, cβνmin≥1/ϱc_β _ ≥ 1/ for 0<ϱ≤10< ≤ 1, and logν0(xv)=b~(x)+a0(x)sv+e~v(x),a0(x)≥0. _0(xv)= b(x)+a_0(x)s_v+ e_v(x), a_0(x)≥ 0. (52) Using log(1+cβz)=logcβ+logz+log(1+1/(cβz)) (1+c_βz)= c_β+ z+ (1+1/(c_βz)) gives Πvocablogπ⋆(⋅∣x)p0(⋅∣x)=a0(x)+Πvocab~(x)+ϵ(x),‖ϵ(x)‖2≤Vϱ. _vocab π (· x)p_0(· x)=a_0(x)s+ _vocab e(x)+ ε(x), \| ε(x)\|_2≤ V\, . Indeed, each last logarithm lies in [0,ϱ][0, ], and centering is a contraction. Corollary D.1 applies only if its relative Fisher-residual condition also holds. The positive lower bound on success probabilities excludes zero-success continuations and is not automatic for binary rewards. This regime concerns a small positive penalty coefficient in an idealized optimum, not a small realized KL divergence or a zero-penalty checkpoint. Lemma D.5 (Consistency of the representation pullback). Retain Definition D.1 and Assumption D.2. For fixed nonzero 0∈ℰw_0 , define h(x)=(x)−a(x)0e_h(x)= (x)-a(x)w_0. Then h(x)=a(x)(Bx−0)+Bx⟂(x).e_h(x)=a(x)(B_xs-w_0)+B_xr_ (x). (53) Let G be a fixed positive definite metric on ℰE, set E0=x[a(x)2]∥0∥G2>0E_0=E_x[a(x)^2] _0 _G^2>0, and suppose x[a(x)2∥Bx−0∥G2]≤ζBE0,x∥Bx⟂(x)∥G2≤ζRE0E_x\! [a(x)^2 B_xs-w_0 _G^2 ]≤ _BE_0, _x B_xr_ (x) _G^2≤ _RE_0 (54) for ζB,ζR≥0 _B, _R≥ 0. Then x∥h(x)∥G2≤2(ζB+ζR)E0.E_x _h(x) _G^2≤ 2( _B+ _R)E_0. (55) The constants separately control variation of the shared pullback and the pulled-back residual energy. Proof. Substitute (x)=a(x)+⟂(x)t(x)=a(x)s+r_ (x) into (x)=Bx(x) (x)=B_xt(x) and subtract a(x)0a(x)w_0 to obtain Equation (53). Applying ∥u+v∥G2≤2∥u∥G2+2∥v∥G2 u+v _G^2≤ 2 u _G^2+2 v _G^2 and taking expectations proves Equation (55). ∎ Under exact homogeneity, define Ax=Fx1/2JxTA_x=F_x^1/2J_xT and x=Ax(Ax⊤Ax)−1Ax⊤P_x=A_x(A_x A_x)^-1A_x . This is an orthogonal projector, so ∥Bx⟂(x)∥G2=(Fx1/2⟂(x))⊤x(Fx1/2⟂(x))≤‖⟂(x)‖Fx2. B_xr_ (x) _G^2=(F_x^1/2r_ (x)) P_x(F_x^1/2r_ (x))≤\|r_ (x)\|_F_x^2. (56) Consequently, the smallest admissible residual ratio in Equation (54) is at most ηlogitx‖a(x)‖Fx2/E0 _logitE_x\|a(x)s\|_F_x^2/E_0. Remark D.5 (Shared-direction consistency under homogeneous metrics). Equation (56) controls the pulled-back residual, not variation of BxB_xs. For example, replacing JxJ_x by −Jx-J_x preserves GxG_x but reverses the pullback, so equal metrics need not yield a common correction direction. For a consistent nonzero 0w_0, normalize =0/∥0∥Gw=w_0/ _0 _G and ah(x)=a(x)∥0∥Ga_h(x)=a(x) _0 _G. To use this component in Assumption D.5, one must additionally verify Q=Qw=w, zero residual mean, and weighted orthogonality. The bound on h(x)e_h(x) alone does not supply these moment conditions. From reward structure to recovery. The argument gives a sequence of sufficient conditions: an exact regularized optimum gives a next-token tilt relation; a shared success profile and a small relative residual give a shared logit component; a consistent pullback gives a shared representation component. The final lower bound separately requires off-principal localization and the moment conditions of Assumptions D.4–D.5. Each condition concerns a different object and must be checked independently when applying the analysis. Connection to practical RL teachers. The tilt identities concern the exact optimum with β>0β>0. For trained checkpoints, including DAPO checkpoints with zero explicit KL coefficient, the fitting and recovery results take (x)t(x) as given and apply under their stated local and structural conditions. Clipping and small updates motivate inspecting a local regime; they do not imply an exact tilt identity, metric homogeneity, or a shared direction for a trained checkpoint. D.4 Single-vector recovery: an identity and a lower bound The quadratic optimum is generally metric-weighted. Under exact homogeneity, its explained target energy has the signal-to-fluctuation form in Section D.4.1 and the sufficient geometric bound in Section D.4.2. Proposition D.2 (Optimal steering vector). Assume finite target energy and finite CirrC_irr. If T⊤G¯T GT is positive definite, where G¯=x[Gx] G=E_x[G_x], the unique minimizer of the quadratic surrogate over ℰE is ⋆=T(T⊤G¯T)−1T⊤x[Gx(x)] δ =T(T GT)^-1T E_x[G_x (x)]. For T=IdT=I_d, it is G¯−1x[Gx(x)] G^-1E_x[G_x (x)]. Under exact homogeneity, ⋆=:=x∼[(x)]. δ = μ:=E_x [ (x)]. This optimizes the quadratic surrogate without a trust-region constraint, not necessarily the original nonlinear KL objective. Proof. Writing =Tz δ=Tz, the normal equation is T⊤G¯Tz=T⊤x[Gx(x)]T GTz=T E_x[G_x (x)]. Positive definiteness gives the unique solution. If Gx=G_x=G on ℰE, x[(x)]∈ℰE_x[ (x)] solves the equation. ∎ Definition D.2 (Recovery ratio). In the constant-metric quadratic model, let Etar=x∥(x)∥G2∈(0,∞)E_tar=E_x (x) _G^2∈(0,∞) and define ρ:=1−x∥(x)−⋆∥G2Etar∈[0,1].ρ:=1- E_x (x)- δ _G^2E_tar∈[0,1]. This is the fraction of locally representable target energy explained by the optimal constant vector, excluding CirrC_irr. In particular, ℒquad(0)−ℒquad(⋆)ℒquad(0)=ρEtarEtar+2Cirr. L_quad(0)-L_quad( δ )L_quad(0)= ρ E_tarE_tar+2C_irr. Thus, even perfect target-energy recovery need not remove irreducible logit error. Exact teacher reproduction additionally requires local reachability and negligible approximation error. Neither ρ nor the displayed surrogate-loss ratio equals an empirical task-score gain recovery rate without further assumptions connecting distributions to task performance. Remark D.1(i) gives the varying-metric counterpart. Locally admissible interventions. Let ⊆ℰT be a specified trust region containing zero. Its best constant-metric target-energy recovery is ρ:=1−inf∈x∥−(x)∥G2Etar,0≤ρ≤ρ. _T:=1- _ δ E_x δ- (x) _G^2E_tar, 0≤ _T≤ρ. The inequalities follow because zero is feasible and ⊆ℰT . If ∈ μ , the two recovery values coincide. Otherwise, unconstrained surrogate recovery is only an upper bound on what this trust region permits. Relating either value to nonlinear KL additionally requires the remainder bounds to hold at the evaluated interventions. D.4.1 Signal-to-fluctuation form Theorem D.1 (Recovery as a signal-to-fluctuation ratio). In the constant-metric quadratic model, let ~(x)=(x)− (x)= (x)- μ, with x[~(x)]=E_x[ (x)]=0 and positive finite total target energy. Then ρ=∥G2∥G2+x∥~(x)∥G2=SNR1+SNR,SNR:=∥G2x∥~(x)∥G2. aligned ρ&= μ _G^2 μ _G^2+E_x (x) _G^2= 1+ ,\\[2.0pt] &:= μ _G^2E_x (x) _G^2. aligned (57) If the fluctuation energy vanishes, set SNR=+∞ =+∞ and ρ=1ρ=1. For any target level u∈(0,1)u∈(0,1), ρ≥uρ≥ u if and only if SNR≥u/(1−u) ≥ u/(1-u), using this convention. Proof. Proposition D.2 gives ⋆= δ = μ. Since G is constant and the fluctuation has zero mean, x∥(x)∥G2=∥G2+x∥~(x)∥G2E_x (x) _G^2= μ _G^2+E_x (x) _G^2. Substitution into Definition D.2 proves Equation (57). If the fluctuation energy is zero, positive total energy makes the mean energy positive. The threshold relation follows by rearranging ρ=SNR/(1+SNR)ρ= /(1+ ). ∎ Corollary D.2 (Spectral upper bound). In the same model, let (x)=G1/2(x)u(x)=G^1/2 (x), =x[(x)]=G1/2m=E_x[u(x)]=G^1/2 μ, and Σ=x[(x)(x)⊤] =E_x[u(x)u(x) ]. For a weighted data operator or factor M with Σ=MM⊤ =M , ρ=‖22trΣ≤λ1(Σ)trΣ=σ12(M)∑iσi2(M).ρ= \|m\|_2^2 ≤ _1( ) = _1^2(M) _i _i^2(M). For ≠0m≠ 0, equality holds exactly when m lies in the leading eigenspace of Σ and ⊤Cov((x))=0m Cov(u(x))m=0. For =0m=0 and trΣ>0 >0, the inequality is strict. Proof. Since Σ=⊤+Cov((x)) =mm +Cov(u(x)), one has ⊤⪯Σmm . For v=/‖2v=m/\|m\|_2, λ1(Σ)≥v⊤Σv=‖22+v⊤Cov((x))v≥‖22. _1( )≥ v v=\|m\|_2^2+v Cov(u(x))v≥\|m\|_2^2. Equality requires equality at both steps. When =0m=0, positive trace implies λ1(Σ)>0 _1( )>0. Finally, trΣ=Etar =E_tar and the factorization gives the singular-value expression. ∎ Implications for capacity, gap, and depth. The leading spectral share is an upper bound, not generally the recovery of a constant vector. Collinear targets can have unit leading spectral share but low mean-energy recovery when their amplitudes vary or their signs cancel. Uniformly scaling a fixed target family by a nonzero scalar leaves ρ unchanged in a fixed metric. Thus, target magnitude or teacher–student gap alone does not impose a monotone capacity requirement. Injection depth can change the Jacobian, reachability, sensitivity metric, and target variation; the theorem does not assert a universal monotone relationship with depth. The observed gap and depth trends are empirical findings consistent with changes in these quantities, not direct consequences of the recovery identity. D.4.2 Explicit three-constant bound Assumptions D.4–D.5 specify a sufficient favorable regime through principal energy, complement residual energy, and amplitude consistency. Theorem D.2 (Off-principal single-direction bound). In the constant-metric quadratic model, under Assumptions D.4–D.5 and their working-subspace compatibility conditions, ρ≥κ(1+η)(1+ε). ρ≥ κ(1+η)(1+ ). (58) Energy decomposition. Both moment conditions in Equation (37) are used. In particular, weighted orthogonality removes the mixed term in x∥a(x)+(x)∥G2=x[a(x)2]∥G2+x∥(x)∥G2+2x[a(x)⟨,(x)⟩G].E_x a(x)w+ ξ(x) _G^2=E_x[a(x)^2] _G^2+E_x ξ(x) _G^2+2E_x[a(x) , ξ(x) _G]. (59) Zero residual mean identifies Q=x[a(x)]Q μ=E_x[a(x)]w. Remark D.1 supplies a perturbation bound when the principal/complement split is only approximately G-orthogonal. Proof. By Theorem D.1, ρ=∥G2/Etarρ= μ _G^2/E_tar. The zero-mean residual gives Q=x[a(x)]Q μ=E_x[a(x)]w. Metric compatibility and ∥G=1 _G=1 imply ∥G2≥∥Q∥G2=(x[a(x)])2=κx[a(x)2]. μ _G^2≥ Q μ _G^2=(E_x[a(x)])^2= _x[a(x)^2]. The energy split, localization bound, and weighted orthogonality give Etar≤(1+ε)x∥Q(x)∥G2=(1+ε)(1+η)x[a(x)2].E_tar≤(1+ )E_x Q (x) _G^2=(1+ )(1+η)E_x[a(x)^2]. The last factor is positive; division proves the result. ∎ What the bound establishes. Write EP=x∥P(x)∥G2E_P=E_x P (x) _G^2 and EQ=x∥Q(x)∥G2>0E_Q=E_x Q (x) _G^2>0. The proof gives complement recovery ρQ=∥Q∥G2/EQ=κ/(1+η) _Q= Q μ _G^2/E_Q=κ/(1+η). If EP>0E_P>0, define ρP=∥P∥G2/EP _P= P μ _G^2/E_P and e=EP/EQe=E_P/E_Q. Then the exact relation is ρ=ρQ+eρP1+e.ρ= _Q+e _P1+e. If EP=0E_P=0, ρ=ρQρ= _Q. The theorem discards a nonnegative principal contribution and uses e≤εe≤ . Accordingly, low principal energy is a sufficient condition within this decomposition, not necessary for high constant-vector recovery: a constant nonzero target entirely in the principal subspace also has ρ=1ρ=1. The theorem quantifies a specified localization pattern; it does not derive that pattern from the RLVR objective or establish its prevalence. Corollary D.3 (Coefficient-of-variation form). Under the same conditions, ρ≥1(1+CV(a)2)(1+η)(1+ε),ρ≥ 1(1+ (a)^2)(1+η)(1+ ), since κ=1/(1+CV(a)2)κ=1/(1+ (a)^2). Part I. Multi-Vector Steering: The Geometry of the Residual Subspace Section D.5 extends constant-vector fitting to subspace constraints and distinguishes it from adaptive subspace approximation. Section D.6 relates principal structure to sensitivity and weight-induced activation changes. D.5 Sequential-orthogonal steering and constrained fitting The empirical protocol learns (0),…,(K−1)v^(0),…,v^(K-1) sequentially. At each stage only the current multi-layer direction is injected; earlier directions define layer-wise Euclidean orthogonality constraints. Reprojection enforces these constraints after each optimizer step, and each direction is evaluated independently. The result below concerns exact fitting for a fixed target distribution and metric, not cumulative injection of the learned vectors. Theorem D.3 (Constant-vector recovery under a subspace constraint). Let k⊆ℰV_k be a feasible subspace in the constant-metric quadratic model, and let RkR_k have full-column-rank columns spanning it. The exact constrained minimizer is (k)=Rk(Rk⊤GRk)−1Rk⊤G.v^(k)=R_k(R_k GR_k)^-1R_k G μ. For the single-site sequential protocol, k=ℰ∩span(j):j<k⟂2V_k=E \v^(j):j<k\ _2. With Etar>0E_tar>0, its standalone target-energy recovery satisfies ρ(k)=∥(k)∥G2Etar,k+1⊆k⟹ρ(k+1)≤ρ(k).ρ^(k)= ^(k) _G^2E_tar, _k+1 _k\ \ ρ^(k+1)≤ρ^(k). (60) If G=cIG=cI on ℰE, c>0c>0, 0=ℰV_0=E, and ≠0 μ≠ 0, exact first-stage fitting gives (0)=v^(0)= μ, and fitting in its Euclidean orthogonal complement gives (1)=v^(1)=0. Proof. The parallel-axis identity splits the fitting objective into a constant fluctuation term and ‖−‖G2\|v- μ\|_G^2. Writing =Rkzv=R_kz gives Rk⊤G(Rkz−)=0R_k G(R_kz- μ)=0 and the stated solution. The residual is G-orthogonal to kV_k, hence ⟨(k),⟩G=‖(k)‖G2 ^(k), μ _G=\|v^(k)\|_G^2. Expanding the loss reduction yields Equation (60). Nesting the feasible subspaces cannot improve the optimum. In the isotropic case the second space is orthogonal to μ, whose projection is zero. If the feasible space has dimension zero, its unique minimizer is zero without applying the matrix-inverse formula. A zero vector introduces no further orthogonality constraint. ∎ For multiple sites, intersect the product of the layer-wise feasible spaces with the working subspace. The theorem applies to the stacked intervention and the full sensitivity metric, including cross-layer blocks. Adaptive subspace approximation. For comparison, consider a shared subspace with unrestricted context-dependent coefficients. Let 1≤K≤dimℰ1≤ K≤ and (x)=G1/2(x)u(x)=G^1/2 (x), with Σ=x[(x)(x)⊤] =E_x[u(x)u(x) ]. Then minQK⊤QK=IKrange(QK)⊆G1/2ℰx‖(x)−QKQK⊤(x)‖22=trΣ−∑i=1Kλi(Σ). _ subarraycQ_K Q_K=I_K\\ range(Q_K) G^1/2E subarrayE_x\|u(x)-Q_KQ_K u(x)\|_2^2= - _i=1^K _i( ). (61) Indeed, the captured energy is tr(QK⊤ΣQK) (Q_K Q_K), maximized by a leading eigenspace. The fitted coefficients QK⊤(x)Q_K u(x) depend on context. Thus, the optimal explained fraction is ∑i=1Kλi(Σ)/trΣ _i=1^K _i( )/ when the trace is positive. These are oracle coefficients for the quadratic target. A particular gated network must also represent and learn the required coefficient map, and layer-wise restrictions may further reduce its attainable fit. The spectral expression is therefore a characterization of a more flexible approximation class, not a guarantee for a trained gating architecture. Interpretation of later steering directions. Nonzero standalone performance of later Euclidean-orthogonal vectors establishes the existence of alternative useful interventions. It does not establish that they recover distinct spectral components of the teacher target. Even a deterministic target can admit more than one useful Euclidean-orthogonal intervention under an anisotropic metric. For example, with ℰ=ℝ2E=R^2, (x)=(1,0)⊤ (x)=(1,0) , and G=(2111)G= ( smallmatrix2&1\\ 1&1 smallmatrix ), Theorem D.3 gives (0)=(1,0)⊤v^(0)=(1,0) and (1)=(0,1)⊤v^(1)=(0,1) , with standalone recovery 11 and 1/21/2, respectively, although the target second moment has rank one. In the empirical setting, anisotropy, nonlinear responses, evolving sampled contexts, and approximate optimization can all affect the learned directions; task scores are also distinct from quadratic energy recovery. The observed decline with extraction order and its model-scale dependence are consequently descriptions of independently effective interventions, not estimates of target spectral rank or intrinsic manifold dimension. D.6 Effective directions and the principal subspace The main-text steering measurements place approximately 1%1\%–2%2\% of the learned vectors’ Euclidean energy in a top-10%10\% activation principal subspace, below the approximately 10%10\% isotropic reference. This observation concerns the learned vectors. The recovery bound instead uses sensitivity-weighted energy of the context-dependent target family. We now state the metric comparison needed to connect such quantities when the shift objects are matched. Definition D.3 (Variance and sensitivity metrics). For a specified base-model reference context distribution refD_ref, let ¯ℓ=x∼ref[ℓ(x)] h_ =E_x _ref[h_ (x)] and define C:=x∼ref[(ℓ(x)−¯ℓ)(ℓ(x)−¯ℓ)⊤],G:=x∼[Gx].C:=E_x _ref[(h_ (x)- h_ )(h_ (x)- h_ ) ], G:=E_x [G_x]. The leading eigenspace of C defines P, with Q=I−PQ=I-P; fix a choice of basis if the cutoff eigenvalue is degenerate. The matrix G measures output sensitivity, and at the unembedding input equals U⊤x[Fx]U E_x[F_x]U. The reference distribution used to estimate covariance and the distribution used for fitting must be specified; they need not be identical. An uncentered implementation uses C+¯ℓ¯ℓ⊤C+ h_ h_ instead and must report its basis accordingly. High activation variance is not equivalent to high or low output sensitivity. Proposition D.3 (Sensitivity-weighted principal-energy bound). On the working subspace, define gP=supv∈ℰ∩range(P)∖0v⊤Gv‖v‖22,gQ=infv∈spanQ(x)∖0v⊤Gv‖v‖22>0,g_P= _v (P) \0\ v Gv\|v\|_2^2, g_Q= _v \Q (x)\ \0\ v Gv\|v\|_2^2>0, with gP=0g_P=0 if Pℰ=0PE=\0\. If x‖Q(x)‖22>0E_x\|Q (x)\|_2^2>0, then eG:=x∥P(x)∥G2x∥Q(x)∥G2≤gPgQx‖P(x)‖22x‖Q(x)‖22.e_G:= E_x P (x) _G^2E_x Q (x) _G^2≤ g_Pg_Q E_x\|P (x)\|_2^2E_x\|Q (x)\|_2^2. If the right-hand side is small, it supplies a small admissible ε≥eG ≥ e_G for the localization part of Assumption D.4. Its metric-orthogonality condition remains separate. Proof. At each context, ∥P(x)∥G2≤gP‖P(x)‖22 P (x) _G^2≤ g_P\|P (x)\|_2^2 and ∥Q(x)∥G2≥gQ‖Q(x)‖22 Q (x) _G^2≥ g_Q\|Q (x)\|_2^2. Averaging and dividing by the positive denominator proves the claim; the zero-principal-space case is immediate. ∎ Sensitivity and localization are distinct. If Jxv=c(x)V+e(x)J_xv=c(x)1_V+e(x), softmax gauge invariance gives v⊤Gxv=‖e(x)‖Fx2v G_xv=\|e(x)\|_F_x^2. A small gPg_P requires an appropriate uniform sensitivity bound over the principal subspace, not merely this identity for one vector. The ratio gP/gQg_P/g_Q can otherwise be large, so a small Euclidean PC fraction need not imply small sensitivity-weighted intrusion. Nor does localization of a learned mean imply localization of its target family. With G=IG=I, P=e1e1⊤P=e_1e_1 , and equiprobable targets ±=(±c,1)⊤ _±=(± c,1) , the optimal constant vector is (0,1)⊤(0,1) and has zero principal energy, whereas target principal-to-complement energy is c2c^2 and recovery is 1/(1+c2)1/(1+c^2). Measurements of learned-vector energy therefore do not alone verify Assumption D.4. D.6.1 A weight-space account of activation leakage We relate a layer’s weight displacement to its first-order activation change. This is distinct from both the fitted intervention and the pulled-back teacher target. Weight-principal components. For W∈ℝd×d′W ^d× d with singular value decomposition W=∑isiii⊤W= _is_ip_iq_i , define PW:=∑i≤kii⊤.P_W:= _i≤ kp_ip_i . The split ΔW=PWΔW+(I−PW)ΔW W=P_W W+(I-P_W) W separates principal weight components from their complement. The bound |si(W+ΔW)−si(W)|≤‖ΔW‖2|s_i(W+ W)-s_i(W)|≤\| W\|_2 controls absolute spectral displacement but does not determine activation-principal localization. Lemma D.6 (Weight components and first-order activation leakage). Consider ℓ(x)=ϕ(W(x))h_ (x)=φ(Wc(x)) with elementwise differentiable ϕφ, holding (x)c(x) fixed while perturbing W. Let D(x)=diag(ϕ′(W(x)))D(x)= (φ (Wc(x))) have bounded norm at the evaluated points. For the first-order shift (x)=D(x)ΔW(x) _h(x)=D(x) Wc(x), ‖P(x)‖2≤‖D(x)‖2‖PWΔW‖2‖(x)‖2⏟on-principal weight mass+‖PD(x)(I−PW)‖2‖ΔW‖2‖(x)‖2⏟cross-subspace leakage.\|P _h(x)\|_2≤ \|D(x)\|_2\|P_W W\|_2\|c(x)\|_2_on-principal weight mass+ \|PD(x)(I-P_W)\|_2\| W\|_2\|c(x)\|_2_cross-subspace leakage. (62) If ‖(x)‖2>0\| _h(x)\|_2>0 and the right-hand side is at most ξ‖(x)‖2ξ\| _h(x)\|_2, then ‖P(x)‖2/‖(x)‖2≤ξ\|P _h(x)\|_2/\| _h(x)\|_2≤ξ. Proof. Decompose P(x)=PD(x)PWΔW(x)+PD(x)(I−PW)ΔW(x).P _h(x)=PD(x)P_W Wc(x)+PD(x)(I-P_W) Wc(x). Apply the triangle inequality, submultiplicativity, and ‖P‖2≤1\|P\|_2≤ 1. The relative bound follows by division when the total shift is nonzero. ∎ The leakage factor separates as PD(x)(I−PW)=(P−PW)D(x)(I−PW)+PWD(x)(I−PW)PD(x)(I-P_W)=(P-P_W)D(x)(I-P_W)+P_WD(x)(I-P_W), exposing basis mismatch and Jacobian mixing. Small weight-update norm or zero principal weight mass alone does not make this factor small relative to the total activation change. Extending the result beyond first order requires a nonlinear remainder bound; changing earlier layers additionally perturbs (x)c(x). The lemma is a conditional first-order decomposition, not a deduction of activation localization from small weight updates. Part I. Geometric Analysis of Alpha-Stabler Parts I–I analyze frozen-model intervention recovery. We now study a distinct parameter-training procedure: a fixed-reference activation diagnostic and local backward gradient projection. The diagnostic’s energy identity and the projection’s local properties are exact; the link to parameter-induced activation changes is conditional, and overall warning and stabilization performance is empirical. D.7 Principal-subspace monitoring and gradient control Warm-up estimation and fixed reference geometry. During the first Twarm=50T_warm=50 parameter updates, the actor trains without projection. Its generated sequences are also evaluated by the frozen initial base model to estimate reference token-activation covariance. After warm-up, let ℓ∈ℝd×rU_ ^d× r contain the selected leading orthonormal eigenvectors, with r=⌈0.10d⌉r= 0.10d . This is the main-text basis Uℓ,rU_ ,r; the retained fraction refers to hidden dimensions, not explained variance. Set Pℓ=ℓℓ⊤P_ =U_ U_ and Qℓ=I−PℓQ_ =I-P_ and hold them fixed thereafter. The analysis conditions on the estimated projectors. It neither assumes they equal population projectors nor supplies a finite-sample PCA guarantee. During each controlled backward pass and optimizer step, both projectors and flags are fixed, and their estimation and selection are detached from differentiation. Token-level principal-subspace intrusion. For identical actor/base token sequences, stack the analyzed valid token-level activation differences as rows of Xℓ(t)∈ℝN×dX_ ^(t) ^N× d. Consistent with Equation (6), PSIℓ(t)=‖Xℓ(t)ℓ‖F2‖Xℓ(t)‖F2+ϵnum,ϵnum>0.PSI_ ^(t)= \|X_ ^(t)U_ \|_F^2\|X_ ^(t)\|_F^2+ _num, _num>0. (63) The stabilizer is the main-text numerical ϵε, not the theoretical intrusion bound ε . Energies are summed over tokens before normalization; averaging vectors first would permit cancellation. Aggregate PSI sums principal and total shift energies across monitored layers before taking their ratio. The identities below hold for the analyzed shift collection. If monitoring uses a token subset, they are exact for that subset, not automatically for all tokens or the population. The sampling and distributed aggregation protocol must be specified with the implementation. Corollary D.4 (Energy interpretation and metric comparison). Fix a layer. Let A=‖XP‖F2A=\|XP\|_F^2, B=‖XQ‖F2B=\|XQ\|_F^2, and A+B>0A+B>0, and set ψ0=A/(A+B) _0=A/(A+B). For B>0B>0, define eE=A/Be_E=A/B. Then ψ0=eE1+eE,0≤ψ0−PSI=ψ0ϵnumA+B+ϵnum. _0= e_E1+e_E, 0≤ _0-PSI= _0 _numA+B+ _num. (64) Write each row of X as i⊤ _i . For the same shifts with B>0B>0, define eG=∑i‖Pi‖G2/∑i‖Qi‖G2e_G= _i\|P _i\|_G^2/ _i\|Q _i\|_G^2. Suppose 0<mG≤MG0<m_G≤ M_G bound the metric on the span of the projected shifts: mG‖v‖22≤‖v‖G2≤MG‖v‖22m_G\|v\|_2^2≤\|v\|_G^2≤ M_G\|v\|_2^2. With χ=MG/mGχ=M_G/m_G, eGχ≤eE≤χeG. e_Gχ≤ e_E≤χ e_G. (65) If this same shift collection satisfies eG≤εe_G≤ , then PSI≤ψ0≤χε/(1+χε)PSI≤ _0≤χ /(1+χ ). Proof. Orthogonality gives ‖X‖F2=A+B\|X\|_F^2=A+B and ‖X‖F2=A\|XU\|_F^2=A, proving the energy identities by substitution. The metric bounds give mGA≤∑i‖Pi‖G2≤MGAm_GA≤ _i\|P _i\|_G^2≤ M_GA and the corresponding bounds for B. Division proves Equation (65); the last claim follows from monotonicity of u/(1+u)u/(1+u). If B=0B=0 and A>0A>0, ψ0=1 _0=1 and the finite-ratio comparison is inapplicable. If A+B=0A+B=0, the stabilized statistic is zero but contains no directional information. ∎ The shifts in this corollary are observed actor-to-base differences. To apply the bound from Parts I–I requires a justified relationship to (x) (x), matched distributions, and appropriate metric bounds; the notation does not assert that relationship. A small PSI is a small directional energy fraction and need not imply small absolute activation displacement. Predictor and threshold calibration. During an initially stable warm-up, retain detached actor-to-base shift matrices at checks every M=3M=3 updates, subject to N>0N>0 and ‖Xℓ(t)‖F2/N>ηmin\|X_ ^(t)\|_F^2/N> _ . The latter is a floor on the average squared token-level shift norm, not the energy of an averaged shift. After the reference bases are frozen, evaluate the retained matrices using Equation (63); let ℓW_ be the valid warm-up PSI values, as in Algorithm 1. Set mℓ m_ =medianψ∈ℓψ, =median_ψ _ ψ, sℓ s_ =1.4826medianψ∈ℓ|ψ−mℓ|+ϵnum, =1.4826median_ψ _ |ψ-m_ |+ _num, (66) τℓon _ ^on =mℓ+3sℓ, =m_ +3s_ , τℓsafe _ ^safe =mℓ+2sℓ. =m_ +2s_ . Calibration is invalid if any ℓW_ is empty or 0<τℓsafe<τℓon<10< _ ^safe< _ ^on<1 fails; training stops for recalibration rather than using such thresholds. After calibration, bases and thresholds are fixed and retained records can be released. Every M updates, valid observations update ψ¯ℓ←0.95ψ¯ℓ+0.05PSIℓ ψ_ ← 0.95 ψ_ +0.05PSI_ , initialized at mℓm_ . The flag aℓ∈0,1a_ ∈\0,1\ starts at zero. It activates after p=3p=3 consecutive valid checks with ψ¯ℓ>τℓon ψ_ > _ ^on and releases when a valid check gives ψ¯ℓ<τℓsafe ψ_ < _ ^safe. A valid check with ψ¯ℓ≤τℓon ψ_ ≤ _ ^on, or an invalid check, resets the consecutive-warning count; an invalid check leaves the moving average and flag unchanged. Flags are held between checks and fixed throughout the associated gradient accumulation, backward pass, and optimizer update. These rules define a detector with smoothing, persistence, and hysteresis. The MAD multiplier does not provide a false-alarm probability for dependent training observations. An initially stable calibration window is an operational requirement, and useful lead times or error rates require empirical evaluation rather than following from Equation (66). Controller: backward-only gradient projection. Let Gℓ(t)∈ℝN×dG_ ^(t) ^N× d be the incoming activation gradient at a monitored output before its local control operation. This is distinct from the sensitivity metric G in Parts I–I. When several sites are controlled, the incoming gradient already includes any downstream modifications. With a fixed basis and detached flag, pass upstream G~ℓ(t)=Gℓ(t)−aℓ(t)(Gℓ(t)ℓ)ℓ⊤. G_ ^(t)=G_ ^(t)-a_ ^(t)(G_ ^(t)U_ )U_ . (67) This hook leaves the forward mapping unchanged at fixed model parameters. At an active site it applies Gℓ(t)QℓG_ ^(t)Q_ ; at an inactive site it leaves the incoming gradient unchanged. Future forward activations can change through the resulting parameter updates. Proposition D.4 (Minimal-change removal of principal gradients). For a Euclidean orthogonal projector P and Q=I−PQ=I-P, the unique solution of minimizeZ:ZP=012∥Z−Gℓ∥F2 Z:\,ZP=0minimize\; 12\|Z-G_ \|_F^2 (68) is G~ℓ=GℓQ G_ =G_ Q, and G~ℓP=0,G~ℓQ=GℓQ,‖G~ℓ‖F2=‖Gℓ‖F2−‖GℓP‖F2. G_ P=0, G_ Q=G_ Q, \| G_ \|_F^2=\|G_ \|_F^2-\|G_ P\|_F^2. (69) Proof. Feasibility gives Z=ZQZ=ZQ. Principal and complement components are Frobenius-orthogonal, hence ‖Z−Gℓ‖F2=‖Z−GℓQ‖F2+‖GℓP‖F2\|Z-G_ \|_F^2=\|Z-G_ Q\|_F^2+\|G_ P\|_F^2. The unique minimizer is Z=GℓQZ=G_ Q. The remaining identities follow from P2=P^2=P, Q2=Q^2=Q, and PQ=0PQ=0. ∎ The proposition concerns the incoming gradient at one hook. It preserves that gradient’s complement and cannot increase its local norm. It does not imply a smaller parameter-gradient norm, preservation of the uncontrolled network’s complete gradient, or an analogous projection of future activation changes. Those quantities also depend on network Jacobians, other control sites, and the optimizer. From projected gradients to activation changes. We analyze one post-warm-up SGD step on a fixed batch. This provides a local relation between a projected activation gradient and the forward displacement it induces, rather than an optimizer-independent stability guarantee. Lemma D.7 (Local principal leakage under projected SGD). Fix a token batch, one active control site, and a parameter block ϑ . Suppose the loss depends on this block only through the selected activations, all other parameters are held fixed, and h(ϑ)h( ) is twice continuously differentiable locally. Stack token activations as h=vec(H⊤)h=vec(H ), and define J=∂h/∂ϑJ=∂ h/∂ , Π=IN⊗P =I_N P, =I−ΠQ=I- , and g=∇hℒg= _hL. For a plain SGD step with α≥0α≥ 0 whose update segment stays within a neighborhood of bounded second derivative, δϑ=−αJ⊤g,δh=−αKg+R,K=JJ⊤,δ =-α J Qg, δ h=-α KQg+R, K=J , (70) where ‖R‖2≤chα2‖J⊤g‖22\|R\|_2≤ c_hα^2\|J Qg\|_2^2 for a local constant chc_h. Consequently, ‖Πδh‖2≤α‖ΠK‖2‖g‖2+chα2‖J⊤g‖22.\| δ h\|_2≤α\| KQ\|_2\|Qg\|_2+c_hα^2\|J Qg\|_2^2. (71) Proof. The single-path dependency gives the modified parameter gradient J⊤gJ Qg. Taylor’s theorem for the vector-valued map gives h(ϑ+δϑ)−h(ϑ)=Jδϑ+Rh( +δ )-h( )=Jδ +R, with ‖R‖2≤ch‖δϑ‖22\|R\|_2≤ c_h\|δ \|_2^2; one may take chc_h as half a uniform bilinear operator-norm bound on its second derivative along the segment. Substitute the SGD step, apply Π , and use ‖Π‖2≤1\| \|_2≤ 1 and submultiplicativity to obtain Equation (71). ∎ What projection does and does not preserve. First-order principal displacement is controlled by ΠK KQ, not by Π=0 =0 alone. Relative to an unprojected SGD step at the same state, batch, and gradient, δhproj−δhplain=αKΠg+(Rproj−Rplain).δ h_proj-δ h_plain=α K g+(R_proj-R_plain). Thus the first-order difference in complement displacement is αKΠg g, which need not vanish. If ΠK=0 KQ=0, symmetry of K also gives KΠ=0QK =0: the projected update then has no first-order principal displacement and preserves the plain step’s complement displacement to first order. Without this additional decoupling, the lemma is a bound, not a guarantee of either preservation property or a comparison showing less leakage than the unprojected step. Preconditioning and additional update terms. For a fixed realized parameter-space preconditioner D and an additional direction b, write a particular update as δϑ=−α(DJ⊤g+b)δ =-α(DJ Qg+b). The same Taylor argument gives, provided its update segment satisfies the same local regularity condition, ‖Πδh‖2≤ \| δ h\|_2≤ α‖ΠJDJ⊤‖2‖g‖2+α‖ΠJb‖2 α\| JDJ Q\|_2\|Qg\|_2+α\| Jb\|_2 +chα2‖DJ⊤g+b‖22. +c_hα^2\|DJ Qg+b\|_2^2. The first-order coupling is now JDJ⊤JDJ ; b may contain optimizer-state or other update contributions not represented by the current projected gradient. This is a conditional estimate for a specified step, not a bound on how an adaptive preconditioner or its state evolves. Applying it to a concrete optimizer requires its actual update decomposition. Multiple control sites likewise require the full modified gradient flow; the single-site SGD lemma does not supply a joint guarantee for the complete training algorithm. Local leakage versus cumulative PSI and stability. The lemma bounds one update on a fixed batch, whereas PSI measures actor-to-base differences accumulated over training and is a ratio of energies. Even zero new principal displacement need not decrease PSI: for a one-token difference (1,1)(1,1) with the first coordinate principal, a complement-only change (0,−α)(0,-α), 0<α<10<α<1, raises the unstabilized PSI from 1/21/2 to 1/(1+(1−α)2)1/(1+(1-α)^2). Long-run guarantees would additionally require uniform control of couplings, update magnitudes, remainders, optimizer state, and changing observation distributions. We therefore treat warning utility and training stabilization as empirical outcomes in Figures 7(c) and 8, not consequences of a monotone-PSI or no-collapse theorem. Implementation and overhead. The mathematical properties require aligned actor/base observations and flags fixed before the controlled backward pass; they do not require a particular graph-retention strategy. Reusing a compatible actor forward and using a separate detached monitoring pass have different runtime and memory costs. Neither the projection identity nor the absence of additional trainable parameters bounds end-to-end overhead. Such claims require measurements that include reference evaluation, any additional actor pass, covariance estimation, calibration storage, and communication. D.8 Summary of theoretical and empirical results Table 5 records the object and scope of each conclusion. Parts I–I characterize conditional target-energy recovery and alternative intervention classes. Part I establishes finite-collection energy identities, exact local gradient-projection properties, and a conditional single-step relationship to activation displacement. Table 5: Summary of recovery, steering geometry, and gradient-control results, with their assumptions and empirical scope. Statement Reference Scope / status Part I — fixed-vector recovery Distillation admits a weighted least-squares approximation Proposition D.1 Fixed contexts; controlled network and KL remainders The regularized optimum is an exponential reward tilt Lemma D.2 Exact optimum of the stated objective, β>0β>0 Next-token ratios follow continuation normalizers Lemma D.3 Exact under the tilt model Shared profiles and consistent pullbacks give shared shifts Lemmas D.4, D.5 Additional sufficient structural conditions ρ=SNR/(1+SNR)ρ=SNR/(1+SNR) Theorem D.1 Constant-metric surrogate target energy Leading spectral energy share upper-bounds ρ Corollary D.2 Upper bound, not a general equality ρ≥κ/[(1+η)(1+ε)]ρ≥κ/[(1+η)(1+ )] Theorem D.2 Assumed localization, moments, and metric compatibility Part I — geometry of steering interventions Constrained constant-vector fitting is a metric projection Theorem D.3 Fixed target family and metric; exact optimization Adaptive subspace recovery is a spectral partial sum Equation (61) Unrestricted context-dependent coefficients Later orthogonal directions can remain useful Section D.5 Empirical standalone interventions; not a rank estimate Euclidean and sensitivity-weighted intrusion are related Proposition D.3 Same targets; explicit metric constants Weight components bound first-order activation leakage Lemma D.6 Fixed input; basis-mismatch and mixing factors Part I — monitoring and gradient control PSI measures Euclidean principal energy of analyzed shifts Corollary D.4 Exact finite-collection identity; stabilizer accounted for Projection minimally removes incoming principal gradients Proposition D.4 Exact at each active hook with fixed projector Principal activation displacement obeys a coupling bound Lemma D.7 Single-site, fixed-batch, local plain SGD Warm-up specifies a fixed reference and thresholds Algorithm 1 Calibration procedure; no detection-error guarantee PSI warns and control improves training stability Figures 7(c) and 8 Empirical findings in the evaluated settings Summary. The analysis links three distinct questions while keeping their mathematical objects distinct. First, shared teacher targets can permit strong constant-vector recovery under local approximation, moment, and metric conditions. Second, constrained and adaptive fitting describe different intervention capacities; neither standalone orthogonality nor low learned-vector PC energy alone determines the teacher target’s rank or localization. Third, Alpha-Stabler uses a fixed activation reference for monitoring and local gradient control. Its projection has exact minimal-change properties, while its effect on parameter-induced activation displacement depends on coupling and the optimizer. These conditional results organize the empirical capacity and localization observations and motivate the controller; they do not replace empirical evidence for task-score recovery, early warning, stability, or runtime overhead.