Paper deep dive
Geometric Calibration of Supervised Fine-Tuning
Jianzhu Bao, Jia-Chen Gu, Qingyuan Liu, Haozhen Zhang, Kai-Wei Chang, Nanyun Peng, Wenya Wang
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Recent studies investigate the weight geometry through its spectral structure, which characterizes how the weights transform different input directions. Within this view, supervised fine-tuning (SFT) is shown to induce dense updates that perturb the spectral geometry of pre-SFT weights along their principal directions, potentially disrupting the model's prior knowledge. These observations motivate using the geometry of pre-SFT weights as a reference for refining learned SFT updates. We study this perspective as Geometric Calibration and develop SPACE Subspace Projection for Adapted Checkpoint Enhancement), a lightweight post-hoc method that selectively reverts SFT update components within the principal subspace of the pre-SFT weights while preserving those in its orthogonal complement. SPACE is a data-free, training-free method that requires only the pre- and post-SFT checkpoints. We evaluate SPACE on Qwen2.5, Qwen3, Qwen3.5, Llama-3.1, and Olmo-3 models ranging from 2B to 14B, across mathematical, coding, and medical SFT, as well as several SFT variants. Overall, SPACE mitigates out-of-domain (OOD) capability regression while further improving in-domain (ID) performance over the uncalibrated SFT checkpoints. For math SFT, SPACE yields average ID/OOD gains of 2.85/2.26 points on Qwen2.5-7B, and 2.41/1.52 points on Qwen3-14B-Base, respectively. These results highlight how the geometric relationship between SFT updates and pre-SFT weights shapes the adapted model's capabilities and, more broadly, motivate geometry-aware parameter updates for continual model improvement.
Tags
Links
- Source: https://arxiv.org/abs/2610.18634v1
- Canonical: https://arxiv.org/abs/2610.18634v1
Trouble viewing inline? Open PDF directly →
Full Text
76,498 characters extracted from source content.
Expand or collapse full text
Geometric Calibration of Supervised Fine-Tuning Jianzhu Bao Affiliation: Nanyang Technological University Email: jianzhu.bao@ntu.edu.sg Jia-Chen Gu Affiliation: University of California, Los Angeles Email: qingyuan.liu@ntu.edu.sg Qingyuan Liu Affiliation: Nanyang Technological University Email: haozhen.zhang@ntu.edu.sg Haozhen Zhang Affiliation: Nanyang Technological University Email: wangwy@ntu.edu.sg Kai-Wei Chang, Nanyun Peng, Wenya Wang †thanks: Corresponding author. Email: gujc@, kwchang@cs., violetpeng@cs.ucla.edu[3pt] Website Code Affiliation: Nanyang Technological University Affiliation: University of California, Los Angeles Abstract Recent studies investigate the weight geometry through its spectral structure, which characterizes how the weights transform different input directions. Within this view, supervised fine-tuning (SFT) is shown to induce dense updates that perturb the spectral geometry of pre-SFT weights along their principal directions, potentially disrupting the model’s prior knowledge. These observations motivate using the geometry of pre-SFT weights as a reference for refining learned SFT updates. We study this perspective as Geometric Calibration and develop SPACE (Subspace Projection for Adapted Checkpoint Enhancement), a lightweight post-hoc method that selectively reverts SFT update components within the principal subspace of the pre-SFT weights while preserving those in its orthogonal complement. SPACE is a data-free, training-free method that requires only the pre- and post-SFT checkpoints. We evaluate SPACE on Qwen2.5, Qwen3, Qwen3.5, Llama-3.1, and Olmo-3 models ranging from 2B to 14B, across mathematical, coding, and medical SFT, as well as several SFT variants. Overall, SPACE mitigates out-of-domain (OOD) capability regression while further improving in-domain (ID) performance over the uncalibrated SFT checkpoints. For math SFT, SPACE yields average ID/OOD gains of 2.85/2.26 points on Qwen2.5-7B, and 2.41/1.52 points on Qwen3-14B-Base, respectively. These results highlight how the geometric relationship between SFT updates and pre-SFT weights shapes the adapted model’s capabilities and, more broadly, motivate geometry-aware parameter updates for continual model improvement. Figure 1: Overview of SPACE and results. (a) Method. SPACE uses the principal input subspace of the pre-SFT weight as a geometric reference for calibrating the post-SFT weights. The pre-SFT weight is visualized as rank-one SVD components ordered by singular value, while the post-SFT weight is decomposed along the same pre-SFT input directions. Blue and green denote pre- and post-SFT components, respectively, with color intensity reflecting the corresponding pre-SFT singular value. SPACE restores the pre-SFT transformation along the selected principal directions while retaining the post-SFT transformation along the remaining directions. (b) Results. After regular SFT on mathematical data, SPACE improves both in-domain (ID) and out-of-domain (OOD) performance over the uncalibrated SFT checkpoints. 1 Introduction Supervised fine-tuning (SFT) is a standard approach for adapting large language models (LLMs) to target tasks, particularly for improving reasoning capabilities (Ho et al., 2023; Ye et al., 2025; Lai et al., 2025). However, gains on the target task can be accompanied by regression in capabilities beyond the training domain (Chu et al., 2025; Huan et al., 2026; Lai et al., 2026; Shenfeld et al., 2026), a phenomenon commonly studied as forgetting. To gain deeper insight into this problem, we believe it is important to understand how SFT changes the geometry of the model weights. Recent work studies the weight geometry of LLMs through the spectral structure, which captures how different input directions are transformed (Zhu et al., 2025; Shen et al., 2026; Zhang et al., 2026). Under this view, SFT tends to induce dense updates that perturb this geometry along principal directions. These principal directions typically correspond to singular vectors associated with large singular values, which together span a principal subspace. This subspace has been associated with important pre-existing or transferable capabilities, and perturbations within it can contribute to forgetting (Luo et al., 2026; Wang et al., 2025a; Liu et al., 2026; Jin et al., 2026). These insights lead to a direct question: can an SFT checkpoint be improved by removing the learned updates within these principal subspaces? More broadly, this motivates using pre-SFT weight geometry as a reference for refining completed SFT updates. We formalize this perspective as Geometric Calibration and instantiate it with SPACE (Subspace Projection for Adapted Checkpoint Enhancement), a lightweight post-hoc method for refining SFT checkpoints (Figure 1(a)). For each pre-SFT weight matrix, SPACE defines a principal input subspace using its leading right singular vectors. It then projects the SFT update onto the orthogonal complement of this subspace, restoring the pre-SFT transformation along the principal directions while retaining the learned update along the remaining directions. Unlike existing post-hoc methods that determine refinement through parameter interpolation (Wortsman et al., 2022b; Wang et al., 2025b) or properties of the learned update itself (Abdollahpoorrostam et al., 2026; Abro and Tahir, 2026), SPACE explicitly conditions the refinement on the geometric relationship between the SFT update and the pre-SFT weights. Moreover, SPACE operates directly on the pre- and post-SFT checkpoints in a data- and training-free manner. We evaluate SPACE across five model families—Qwen2.5, Qwen3, Qwen3.5, Llama-3.1, and Olmo-3—ranging from 2B to 14B parameters, and across mathematical, coding, and medical SFT. Across these settings, SPACE consistently yields positive average gains over the corresponding SFT checkpoints on both ID and OOD evaluations. For the two mathematical SFT settings in Figure 1(b), SPACE improves average ID/OOD performance by 2.85/2.26 points on Qwen2.5-7B and 2.41/1.52 points on Qwen3-14B-Base. The improvements further extend across different SFT variants, supporting the broad applicability of SPACE beyond standard SFT. It also outperforms a range of representative post-hoc model refinement baselines, highlighting the benefit of accounting for the relationship between SFT updates and pre-SFT geometry. Overall, these results support the use of pre-SFT weight geometry as a reference for refining completed SFT updates and motivate the development of more targeted and effective geometric calibration methods for SFT. 2 Related Work Generalization and forgetting in SFT. Recent work has increasingly focused on understanding the generalization and forgetting behaviors of LLMs during SFT (Wu et al., 2026b; Huan et al., 2026; Ren et al., 2026). Comparisons between SFT and reinforcement learning reveal distinct generalization and forgetting patterns (Chu et al., 2025; Shenfeld et al., 2026). To mitigate capability degradation, existing efforts mainly improve SFT through changes to the training data or objectives. One line of work improves the selection and use of training data (Ye et al., 2025; Kopiczko et al., 2026). Another line of work modifies the supervised objective through token reweighting or regularization (Wu et al., 2026b; Li et al., 2025; Zhu et al., 2026; Jin et al., 2026). These methods primarily intervene during SFT to control how the updates are learned. In contrast, we study whether a completed SFT update can be refined post hoc without data or additional training. Weight geometry in model adaptation. Recent studies analyze the geometry of LLM weights during post-training through their spectral structure (Zhu et al., 2025; Shen et al., 2026; Zhang et al., 2026). These studies reveal that SFT typically produces dense updates along principal directions, which can distort the pretrained spectral structure. In contrast, reinforcement learning tends to induce sparse and off-principal updates, resulting in less disruption to the spectral structure (Liu et al., 2025b; Mukherjee et al., 2026). The role of principal directions has also been emphasized in studies of parameter-efficient fine-tuning (PEFT) and continual model merging. In these lines of work, principal subspaces of pretrained weights are often associated with prior or transferable capabilities. Accordingly, many PEFT methods organize adaptation around principal or complementary singular subspaces (Meng et al., 2024; Wang et al., 2025a; Luo et al., 2026; Xiong and Xie, 2026), while continual model merging methods leverage structured subspaces to reduce interference with previously acquired capabilities (Ilharco et al., 2023; Yu et al., 2024; Sun et al., 2026; Li et al., 2026; Shi et al., 2026; Qiu et al., 2026). Together, these studies suggest that the geometry of pretrained weights provides a useful reference for controlling how new updates interact with existing capabilities. Our work builds on this perspective but uses the principal subspace of the pre-SFT weights to refine a single completed SFT update post hoc. Post-hoc model refinement. Our work is closely related to post-hoc model merging/editing/refinement methods that improve a trained model by directly modifying its parameters (Wortsman et al., 2022b; Bandarkar et al., 2025; Huang et al., 2025; Bandarkar and Peng, 2025). Existing approaches refine completed checkpoints through weight interpolation (Wortsman et al., 2022b; Wang et al., 2025b), update pruning (Yu et al., 2024; Huang et al., 2025), model souping (Wortsman et al., 2022a; Abdollahpoorrostam et al., 2026), or spectral filtering (Abro and Tahir, 2026). Their refinement strategies are typically determined by parameter interpolation or by properties of the learned update or the fine-tuned weights themselves. Although some methods use the pre-SFT checkpoint, they generally do not explicitly consider its geometric relationship with the learned update when determining which update components should be modified. Motivated by the importance of pre-SFT weight geometry discussed above, we instead study this relationship directly and use it to refine completed SFT checkpoints post hoc. 3 Problem Formulation Supervised fine-tuning. SFT adapts an LLM to target tasks by learning from labeled training data. Prior work has shown that SFT can induce non-trivial parameter updates (Yuan et al., 2025; Mukherjee et al., 2026). While such updates can improve performance on the target task, they may also lead to capability regression beyond the training domain (Wu et al., 2026a; Shenfeld et al., 2026; Huan et al., 2026). These observations suggest that some components of the learned SFT updates may be redundant or detrimental to the adapted model. We therefore investigate whether identifying and calibrating such components can mitigate capability regression while preserving or further improving target-domain performance. Geometric calibration. Given a pre-SFT checkpoint θ0 _0 and its corresponding post-SFT checkpoint θSFT _SFT, the SFT-induced parameter update is defined as ΔθSFT=θSFT−θ0. _SFT= _SFT- _0. (1) Rather than treating the post-SFT checkpoint as the endpoint of adaptation, we explore further refinement of the learned update to form the final adapted model. We approach this refinement through Geometric Calibration, using the geometric structure of the pre-SFT weights as a reference. Formally, we define a calibration operator C that maps the learned SFT update to a calibrated update conditioned on the pre-SFT parameters: Δθcal=(ΔθSFT,θ0), _cal=C ( _SFT; _0 ), (2) where ΔθSFT _SFT is the update being calibrated and θ0 _0 serves as the geometric reference. We seek a calibrated checkpoint θcal=θ0+Δθcal _cal= _0+ _cal with improved overall capabilities. 4 Method We develop SPACE as a post-hoc implementation for Geometric Calibration. For a pre-SFT weight W0∈ℝdout×dinW_0 ^d_out× d_in and its post-SFT counterpart WSFTW_SFT, let ΔW=WSFT−W0 W=W_SFT-W_0. The central idea is to refine the completed update ΔW W using the principal subspace of W0W_0 as a geometric reference. 4.1 Motivation Principal directions and capability preservation. A common view in existing work is to characterize the geometry of model weights through their singular directions, where directions associated with large singular values are referred to as principal directions and their span defines the principal subspace (Zhu et al., 2025; Wang et al., 2025a; Shen et al., 2026). Prior work suggests that this principal subspace is closely associated with important prior or broadly transferable knowledge, and that limiting changes to these directions can help preserve existing capabilities during adaptation (Wang et al., 2025a; Luo et al., 2026; Sun et al., 2026; Jin et al., 2026). At the same time, SFT tends to induce dense updates that perturb the pre-SFT weight geometry within the principal subspace (Zhu et al., 2025; Shen et al., 2026). Together, these findings motivate using the principal subspace of the pre-SFT weights as a reference for refining the learned SFT update. In particular, a natural calibration objective is to preserve the pre-SFT transformation within this subspace while retaining the post-SFT transformation along the remaining directions. Preserving the pre-SFT transformation. For a layer input representation h∈ℝdinh ^d_in, we seek a calibrated transformation WcalW_cal that follows the pre-SFT transformation on the component associated with the principal subspace of W0W_0, while retaining the post-SFT transformation on the remaining component. Let P0P_0 be a rank-k orthogonal projector onto the principal subspace, where k denotes its dimension. The desired behavior is Wcalh=W0P0h+WSFT(I−P0)h.W_calh=W_0P_0h+W_SFT(I-P_0)h. (3) To determine P0P_0, we seek a projection that retains as much of the pre-SFT transformation as possible. Specifically, among all rank-k orthogonal projectors, we choose P0P_0 such that W0P0W_0P_0 best approximates W0W_0 in Frobenius norm: P0∈argminP=P⊤=P2rank(P)=k‖W0−W0P‖F2.P_0∈ subarraycP=P =P^2\\ rank(P)=k subarray \|W_0-W_0P \|_F^2. (4) Since P is an orthogonal projector, minimizing this error is equivalent to maximizing tr(PW0⊤W0)tr(PW_0 W_0). Let W0=U0Σ0V0⊤W_0=U_0 _0V_0 be the SVD of W0W_0. The Ky Fan maximum principle (Fan, 1949) gives a solution P0=V0,kV0,k⊤,P_0=V_0,kV_0,k , (5) where V0,kV_0,k contains the leading k right singular vectors of W0W_0. A complete derivation from equation 4 to equation 5 is provided in Appendix A. With this choice, W0P0=U0,kΣ0,kV0,k⊤W_0P_0=U_0,k _0,kV_0,k is a best rank-k approximation of W0W_0. Thus, P0hP_0h selects the component of the input representation associated with this principal part of the pre-SFT transformation. We refer to span(V0,k)span(V_0,k) as the principal input subspace, and use P0P_0 for calibration. 4.2 SPACE: Subspace Projection for Adapted Checkpoint Enhancement Subspace Projection. Requiring equation 3 to hold for any layer input representation h gives Wcal=W0P0+WSFT(I−P0)W_cal=W_0P_0+W_SFT(I-P_0). Using WSFT=W0+ΔW_SFT=W_0+ W, this can be equivalently written as Wcal=W0+ΔW(I−P0).W_cal=W_0+ W(I-P_0). (6) Applying this calibration to the selected weight matrices yields the model-level calibrated update Δθcal _cal defined in Section 3. SPACE therefore removes the SFT update within the principal input subspace while preserving it on the orthogonal complement. Adaptive subspace dimension. Since weight matrices in LLMs can differ substantially in both their dimensions and spectral concentration, using a fixed k across all matrices may capture very different fractions of their singular-value spectra. We therefore choose the subspace dimension k adaptively according to the singular-value spectrum of each weight matrix. Specifically, let σ0,1≥⋯≥σ0,r≥0 _0,1≥·s≥ _0,r≥ 0 denote the singular values of W0W_0. We select the smallest k such that the leading singular components account for at least a fraction ρ of the total spectral energy: k=minm:∑i=1mσ0,i2∑i=1rσ0,i2≥ρ.k= \m: _i=1^m _0,i^2 _i=1^r _0,i^2≥ρ \. (7) This allows the selected subspace to capture a comparable fraction of the dominant singular structure across different weight matrices. In practice, we use ρ=0.5ρ=0.5 and apply the projection to all two-dimensional weight matrices by default, as these choices work well across the evaluated settings. 5 Experiments 5.1 Main Experiments on Mathematical Reasoning SFT We first evaluate SPACE in the setting of mathematical reasoning SFT, in which base models are fine-tuned on long chain-of-thought (CoT) mathematical reasoning data. Table 1: Main experimental results on mathematical reasoning SFT. We compare ID and OOD performance across different post-hoc methods. “Avg. (Δ vs. SFT)” denotes the average score change relative to the SFT checkpoint. Bold denotes the best score excluding the Base model. Model Benchmark Base SFT WiSE-FT TIES LiNeS DARE POME FAPM DG-Hard MonoSoup SPACE Qwen2.5-7B In-domain (ID) MATH500 28.79 44.84 45.14 45.70 43.89 44.13 46.23 46.83 47.50 46.43 49.36 AIME25 0.42 0.83 0.83 0.42 0.63 0.42 0.83 1.46 0.63 1.04 1.46 AIME24 1.65 1.04 1.04 0.63 0.83 0.83 1.04 1.67 1.46 1.46 1.46 OlympiadBench 7.69 14.08 17.06 14.62 14.34 13.82 15.75 17.44 15.39 15.06 17.72 Minerva Math 9.26 12.68 11.60 12.82 10.62 11.53 14.27 11.79 13.26 12.68 15.26 AMC23 15.65 19.53 22.03 19.38 20.00 18.28 20.94 24.84 22.97 20.00 24.84 Avg. (Δ vs. SFT) - - +0.78 +0.10 -0.45 -0.67 +1.01 +1.84 +1.37 +0.61 +2.85 Out-of-domain (OOD) MMLU-Pro 40.33 38.05 40.85 38.12 41.90 38.11 39.41 38.36 35.16 39.99 42.03 GPQA-D 26.77 23.36 26.77 23.96 26.42 22.73 24.37 25.95 24.34 25.44 27.27 LCB v2 28.96 26.22 29.03 26.74 28.83 26.48 27.92 26.61 28.64 28.25 28.70 IFEval 42.57 42.57 45.56 42.57 43.88 41.25 42.81 45.68 41.85 45.44 44.60 AlpacaEval -1.13 -0.14 -0.20 -0.13 -0.12 -0.11 -0.15 -0.16 -0.20 -0.16 -0.10 HaluEval 46.86 46.75 47.80 46.71 47.67 46.56 46.36 46.49 47.15 47.23 47.85 Avg. (Δ vs. SFT) - - +2.17 +0.19 +1.96 -0.30 +0.65 +1.02 +0.02 +1.56 +2.26 Qwen3-14B-Base In-domain (ID) MATH500 55.07 88.27 88.60 88.60 89.47 88.20 89.40 88.20 88.93 89.20 90.27 AIME25 5.21 32.71 29.58 36.25 33.96 34.79 34.58 31.25 33.33 36.67 35.83 AIME24 6.25 42.08 38.54 41.46 45.21 42.92 44.38 35.83 42.92 41.88 43.54 OlympiadBench 27.31 55.11 55.85 56.74 57.33 55.36 57.14 55.56 54.72 56.25 58.37 Minerva Math 22.92 45.22 42.40 45.71 43.87 45.71 44.24 42.52 44.98 44.49 46.08 AMC23 38.59 77.03 78.59 79.84 80.78 80.31 79.38 78.28 80.00 80.16 80.78 Avg. (Δ vs. SFT) - - -1.14 +1.36 +1.70 +1.15 +1.45 -1.46 +0.74 +1.37 +2.41 Out-of-domain (OOD) MMLU-Pro 61.80 65.90 66.30 69.20 66.20 67.00 68.10 67.00 69.40 67.80 69.40 GPQA-D 44.61 44.11 45.29 45.62 44.28 42.93 40.24 43.94 42.59 44.95 44.78 LCB v2 46.71 44.81 44.88 44.10 44.55 45.08 45.47 45.01 43.77 44.81 43.84 IFEval 65.35 48.80 48.20 51.80 47.48 52.16 52.04 45.08 50.00 51.20 54.08 AlpacaEval 0.42 -0.92 -0.18 -0.50 -0.95 -1.01 -0.78 -0.27 -0.79 -0.55 -0.36 HaluEval 53.97 75.47 73.10 75.53 75.70 75.80 76.87 72.07 75.93 75.83 75.53 Avg. (Δ vs. SFT) - - -0.10 +1.26 -0.15 +0.63 +0.63 -0.89 +0.46 +0.98 +1.52 Training recipe. We conduct full-parameter SFT on two base models: Qwen2.5-7B (Yang et al., 2025b) and Qwen3-14B-Base (Yang et al., 2025a). Following prior studies (Zhu et al., 2026; Ren et al., 2026), we use pretrained checkpoints without instruction tuning. For Qwen2.5-7B, following Wu et al. (2026b); Zhu et al. (2026), we use 100K training examples sampled from NuminaMath-CoT (Li et al., 2024). For Qwen3-14B-Base, we follow Ren et al. (2026) and use Math-CoT-20k, which contains 20K examples. Additional training details are provided in Appendix B.1. Benchmarks and evaluation protocol. We evaluate ID performance on six mathematical benchmarks: MATH-500 (Hendrycks et al., 2021), Minerva Math (Lewkowycz et al., 2022), OlympiadBench (He et al., 2024), AIME 2024 (Zhang and Math-AI, 2024), AIME 2025 (Zhang and Math-AI, 2025), and AMC 2023 (Mathematical Association of America, 2023). Following the benchmark suite used by Ren et al. (2026), we further evaluate OOD performance on six benchmarks: MMLU-Pro (Wang et al., 2024), GPQA-Diamond (GPQA-D) (Rein et al., 2024), LiveCodeBench v2 (LCB v2) (Jain et al., 2025), IFEval (Zhou et al., 2023), AlpacaEval (RM) (Dubois et al., 2024), and HaluEval (Li et al., 2023). Further evaluation details are provided in Appendix B.2. Baselines. We compare SPACE with eight representative post-hoc model editing/merging baselines: WiSE-FT (Wortsman et al., 2022b), TIES (Yadav et al., 2023), LiNeS (Wang et al., 2025b), DARE (Yu et al., 2024), POME (Liu et al., 2025a), FAPM (Huang et al., 2025), DG-Hard (Abro and Tahir, 2026), and MonoSoup (Abdollahpoorrostam et al., 2026). See Appendix B.3 for details. Main results. Table 1 shows that SPACE yields the strongest overall improvements among all baselines, achieving the highest “Avg. (Δ vs. SFT)” in both ID and OOD performance for both models. On Qwen2.5-7B, SPACE improves every evaluated benchmark over SFT and even surpasses the Base model on five of the six OOD benchmarks, suggesting that reasoning capabilities acquired during mathematical SFT may generalize to OOD tasks after calibration. On Qwen3-14B-Base, SPACE also yields substantial additional gains despite the already strong mathematical reasoning performance after SFT. Importantly, the ID gains suggest that SPACE does more than recover capabilities degraded by SFT. They also suggest that reverting updates within the principal subspace can improve generalization to unseen ID examples. The comparison with WiSE-FT further highlights the benefit of selective calibration: global interpolation can recover OOD performance in some settings, but often provides smaller ID gains or degrades the adapted model, whereas SPACE improves both. Finally, SPACE’s advantage over spectral post-hoc baselines such as DG-Hard and MonoSoup, which operate on the learned update itself, suggests that the geometry of the pre-SFT weights provides additional information for post-hoc refinement. Overall, these results suggest that effective post-hoc refinement depends not simply on shrinking or restructuring the learned update, but on strategically calibrating it with respect to the geometry of the pre-SFT weights. Figure 2: Ablation study on Qwen2.5-7B. “Avg. (Δ vs. SFT)” denotes the average changes relative to the SFT checkpoint over the six ID mathematical benchmarks and the six OOD benchmarks. In (c), “Attn.” denotes attention projection matrices; “Gate”, “Up”, and “Down” denote the corresponding MLP matrices; “MLP” includes all MLP projection matrices; “Head+Emb.” denotes the output head and embedding matrices; and “All 2D” denotes all two-dimensional weight matrices. In (d), “R” and “L” denote the right and left singular vectors, while “top” and “bottom” denote the leading and trailing singular subspaces with their dimensions determined using the default threshold ρ=0.5ρ=0.5. “W0W_0” and “ΔW W” indicate that the subspace is derived from the pre-SFT weights and the SFT update, respectively. Shading indicates the default SPACE configuration in each panel. 5.2 Ablation Study We conduct an ablation study on Qwen2.5-7B to examine the sensitivity of SPACE to its key hyperparameters and design choices. All training and evaluation settings follow those in Section 5.1. We report the average change relative to the SFT checkpoint separately over the six ID mathematical benchmarks and the six OOD benchmarks. Hyperparameter sensitivity. We first study the sensitivity of SPACE to the subspace threshold ρ and calibration strength α. For the latter, we generalize equation 6 as Wcal=W0+ΔW(I−αP0)W_cal=W_0+ W(I-α P_0), where α∈[0,1]α∈[0,1] controls the fraction of the SFT update reverted within the selected principal subspace. Thus, α=1α=1 corresponds to the default SPACE. Figure 2(a) shows that ρ=0.5ρ=0.5 provides a good balance between ID and OOD performance. Using a smaller subspace weakens the gains, while a larger one further improves OOD performance at the cost of reduced ID improvement. We therefore use ρ=0.5ρ=0.5 as the default. Figure 2(b) shows that performance generally improves as the calibration strength increases, with α=1α=1 performing best. Together, these results suggest that SPACE is best applied as a hard projection, with ρ as the sole trade-off hyperparameter. Matrix selection. We next examine which weight matrices should be calibrated. As shown in Figure 2(c), calibrating attention matrices alone has little effect, whereas jointly calibrating the MLP matrices yields substantially larger improvements. This is consistent with prior work suggesting that MLP modules serve as important sites for factual knowledge storage and recall (Geva et al., 2021; Dai et al., 2022; Meng et al., 2022). Among individual MLP projections, the Up projection provides the largest gains, while the Down projection alone is ineffective. Interestingly, attention contributes little on its own but provides additional gains when combined with MLP calibration, suggesting that MLPs drive the primary benefits while attention plays a complementary role. Finally, applying SPACE to all two-dimensional weight matrices gives the strongest overall results. Subspace selection. Finally, we compare different choices of the calibration subspace. As shown in Figure 2(d), the leading right singular subspace of the pre-SFT weights, as used by SPACE, provides the best balance between ID and OOD improvements. In contrast, the bottom right singular subspace strongly favors OOD recovery but provides almost no ID gain, indicating a less balanced trade-off. Moreover, the weaker performance of subspaces derived from ΔW W supports the central premise of Geometric Calibration: the pre-SFT weights provide a more informative reference for refining a completed SFT update. Overall, the clear advantage of the right over the left singular subspaces is consistent with the input-side formulation of SPACE. 5.3 Experiments on Different Target Domains We next examine whether the benefits of SPACE extend beyond mathematical reasoning SFT by training on coding and medical data. Table 2: Experimental results of training Qwen2.5-7B on coding and medical data. “(M)” denotes the medical subset. Bold indicates the best result excluding the Base model. Coding SFT Medical SFT Benchmark Base SFT DG-Hard MonoSoup SPACE Benchmark Base SFT DG-Hard MonoSoup SPACE In-domain (ID), Coding In-domain (ID), Medical HumanEval 77.44 78.66 75.61 76.83 78.66 MedMCQA 53.74 54.79 54.22 54.12 55.25 HumanEval+ 71.34 71.95 70.73 71.34 73.78 MedQA 58.68 65.12 60.80 63.39 64.65 MBPP 75.40 78.04 77.78 78.31 78.31 PubMedQA 71.70 70.00 70.60 70.00 73.20 MBPP+ 65.61 67.99 67.99 68.78 67.99 MMLU-Pro (M) 55.83 57.65 55.83 59.15 59.54 LCB v2 28.96 23.09 22.50 24.53 22.77 GPQA (M) 44.87 36.92 37.95 42.82 47.69 Avg. (Δ vs. SFT) - - -1.02 +0.01 +0.36 Avg. (Δ vs. SFT) - - -1.02 +1.00 +3.17 Out-of-domain (OOD) Out-of-domain (OOD) MMLU-Pro 40.33 38.35 39.30 40.39 42.32 MMLU-Pro 40.33 53.90 53.41 53.68 52.47 GPQA-D 26.77 25.35 22.16 26.04 27.27 GPQA-D 26.77 24.91 26.04 25.22 25.88 LCB v2 28.96 24.07 25.64 27.14 26.48 IFEval 42.57 54.08 53.00 54.20 53.60 IFEval 42.57 35.01 34.41 35.97 36.09 AlpacaEval -1.13 -0.43 -0.23 -0.25 -0.39 AlpacaEval -1.13 -2.30 -2.31 -1.99 -1.73 HaluEval 46.86 49.32 49.52 49.41 50.31 HaluEval 46.86 63.31 64.05 65.39 65.10 Avg. (Δ vs. SFT) - - -0.58 +0.62 +1.29 Avg. (Δ vs. SFT) - - +0.39 +1.09 +0.90 Setup. For medical SFT, we fine-tune Qwen2.5-7B on 23,493 high-quality medical reasoning examples from m23k (Huang et al., 2026), as used in prior work (Li et al., 2025). For coding SFT, following prior work (Zhu et al., 2026), we use 100K examples sampled from Magicoder-Evol-Instruct-110K (Wei et al., 2024) as training data, and evaluate ID performance using the EvalPlus library (Liu et al., 2023) on HumanEval, HumanEval+ (Chen et al., 2021), MBPP, and MBPP+ (Austin et al., 2021). LiveCodeBench v2 (Jain et al., 2025) is also included for ID evaluation. The maximum generation length is set to 4,096 tokens. For medical SFT, we use the evaluation toolkit provided by Li et al. (2025) and report results on MedMCQA (Pal et al., 2022), MedQA (Jin et al., 2021), PubMedQA (Jin et al., 2019), MMLU-Pro (Medical) (Wang et al., 2024), and GPQA (Medical) (Rein et al., 2024), with a maximum generation length of 8,192 tokens. Additionally, we include DG-Hard and MonoSoup as two strong baselines for comparison. All other training and evaluation settings follow the same protocol as in Section 5.1. Results. As shown in Table 2, SPACE remains effective when the training domain shifts to coding or medicine. For coding SFT, where the uncalibrated SFT checkpoint already performs well on most ID benchmarks, SPACE largely preserves the learned adaptation while improving OOD performance. For medical SFT, the ID gains are more pronounced, particularly on benchmarks where SFT causes regression relative to the Base model. This pattern suggests that geometric calibration is particularly beneficial when SFT introduces stronger interference with capabilities already encoded in the pre-SFT model. Overall, SPACE improves upon SFT across both domains and outperforms two strong baselines, DG-Hard and MonoSoup, demonstrating its applicability across different forms of task adaptation rather than relying on properties specific to mathematical reasoning. Table 3: Compatibility with different SFT methods. “Original” denotes the checkpoint produced by each training method. “Avg. (Δ vs. Original)” denotes the average score change relative to the original checkpoint. Bold indicates the best score within each method. Benchmark Base DFT ASFT RPSFT Original MonoSoup SPACE Original MonoSoup SPACE Original MonoSoup SPACE In-domain (ID) MATH500 28.79 54.40 57.33 58.64 60.55 61.56 61.12 45.98 46.64 48.69 AIME25 0.42 1.88 1.46 1.25 1.67 2.08 1.88 1.04 0.63 0.83 AIME24 1.65 2.08 3.54 3.75 5.83 5.00 5.21 1.46 1.25 1.25 OlympiadBench 7.69 23.22 26.29 27.18 27.98 28.81 28.45 14.40 15.53 17.03 Minerva Math 9.26 20.57 20.22 20.24 24.43 22.43 22.73 13.01 13.81 15.12 AMC23 15.65 27.81 29.06 28.91 35.31 36.72 38.28 21.72 17.97 20.16 Avg. (Δ vs. Original) - - +1.32 +1.67 - +0.14 +0.32 - -0.30 +0.91 Out-of-domain (OOD) MMLU-Pro 40.33 33.66 30.84 33.39 33.97 33.54 34.45 34.67 36.38 41.06 GPQA-D 26.77 27.27 27.78 26.14 26.48 27.02 25.88 24.15 25.69 27.27 LCB v2 28.96 30.85 32.22 33.27 27.14 29.35 28.90 29.03 29.35 29.75 IFEval 42.57 43.17 46.88 48.20 45.20 47.12 48.44 44.48 44.48 45.56 AlpacaEval -1.13 -0.43 -0.17 -0.13 -0.08 -0.09 -0.09 -0.03 -0.01 -0.07 HaluEval 46.86 48.04 46.97 47.11 47.06 47.43 48.62 46.86 47.48 48.68 Avg. (Δ vs. Original) - - +0.33 +0.90 - +0.77 +1.07 - +0.70 +2.18 5.4 Compatibility with Different SFT Methods Recently, several variants of SFT have been proposed to improve performance and generalization. We further examine whether SPACE is compatible with these methods. Setup. The comparison is conducted on Qwen2.5-7B trained with data from NuminaMath-CoT. Three SFT variants are considered: DFT (Wu et al., 2026b), which reweights the token-level objective; ASFT (Zhu et al., 2026), which augments DFT with distributional anchoring; and RPSFT (Jin et al., 2026), which introduces regularization to constrain weight updates during fine-tuning. For each method, we apply SPACE to its completed checkpoint using the same pre-SFT checkpoint as the geometric reference. Other training and evaluation settings remain consistent with Section 5.1. Results. Table 3 shows that SPACE improves overall ID and OOD performance across all three SFT methods. The consistent gains across methods with different training objectives suggest that the utility of pre-SFT geometry is not tied to a particular SFT update formation process. Notably, the gains are particularly pronounced on RPSFT, even though it already constrains weight geometry during training, suggesting that training-time regularization and post-hoc geometric calibration address complementary aspects. Moreover, SPACE consistently achieves larger aggregate gains than the strong baseline MonoSoup across all three SFT variants. Appendix C.1 shows that these benefits also extend to LoRA fine-tuning. Overall, these results show that SPACE can effectively complement different training objectives by refining their completed updates post hoc. 5.5 Generalization Across Model Families and Scales We further evaluate whether SPACE generalizes across different model families and scales, covering Qwen3.5, Qwen3, Olmo-3, and Llama-3.1 models from 2B to 8B parameters. Setup. The experiments use Qwen3.5-2B-Base, Qwen3-8B-Base, Olmo-3-7B-Instruct-SFT (Olmo et al., 2025) and Llama-3.1-8B-Instruct (Grattafiori et al., 2024) as starting checkpoints for mathematical SFT. Math-CoT-20k is employed as the training data, which contains longer reasoning trajectories and enables more substantial performance improvements through SFT. Other training and evaluation settings are kept the same as in Section 5.1. Results. As shown in Table 4, SPACE yields positive average gains in both ID and OOD performance across all evaluated models. Importantly, the OOD improvements are accompanied by further ID gains across different families and scales, suggesting that geometric calibration refines rather than reverses the SFT adaptation. The gains are particularly pronounced on Llama-3.1-8B, where SFT causes substantial OOD capability regression; in this setting, SPACE improves every evaluated OOD benchmark while further strengthening ID performance. Even at the smallest 2B scale, SPACE still performs well, indicating that its benefits are not limited to larger models. Additional results in Appendix C.2 show consistent gains on Qwen3-4B-Base and larger average improvements than MonoSoup across all evaluated models. Overall, these results demonstrate the broad applicability of SPACE across model families and scales. Table 4: Generalization of SPACE across different model families and scales. Benchmark Qwen3.5-2B-Base Qwen3-8B-Base Olmo-3-7B-Instruct-SFT Llama-3.1-8B-Instruct Base SFT SPACE Base SFT SPACE Base SFT SPACE Base SFT SPACE In-domain (ID) MATH500 72.40 76.47 78.00 55.93 83.73 84.87 60.80 87.93 89.33 39.07 49.20 53.33 AIME25 15.63 17.29 17.50 4.58 23.96 24.38 3.33 35.00 34.38 0.21 1.46 1.25 AIME24 12.92 16.67 15.21 7.29 22.71 22.71 6.67 41.25 41.46 1.88 1.67 2.29 OlympiadBench 37.09 40.15 41.38 24.05 46.86 48.79 25.88 55.26 56.79 10.47 18.96 21.04 Minerva Math 27.08 39.95 40.32 19.49 41.79 40.44 25.74 33.95 34.56 15.56 18.01 21.57 AMC23 49.22 53.59 55.16 32.50 64.22 67.34 37.03 79.53 82.19 14.69 26.09 30.16 Avg. (Δ vs. SFT) - - +0.58 - - +0.88 - - +0.97 - - +2.38 Out-of-domain (OOD) MMLU-Pro 46.40 52.60 54.00 56.70 56.40 56.20 42.70 44.40 44.90 43.50 40.80 44.80 GPQA-D 30.64 29.97 32.66 36.36 29.46 29.46 28.79 22.22 25.93 23.74 15.49 19.87 LCB v2 14.94 11.55 13.31 33.66 24.92 29.29 17.09 22.44 21.14 18.92 8.41 12.20 IFEval 57.67 45.32 53.84 54.08 35.85 40.89 83.57 79.26 82.49 81.89 29.14 36.81 AlpacaEval 0.21 -0.84 -0.23 -0.39 -2.32 -1.10 0.28 0.34 0.38 1.17 -1.87 -0.38 HaluEval 50.00 59.97 59.20 49.33 72.13 72.30 54.23 55.47 55.20 62.73 63.37 63.97 Avg. (Δ vs. SFT) - - +2.37 - - +1.77 - - +0.99 - - +3.66 5.6 Additional Analyses Further analyses show that SPACE is effective at intermediate SFT checkpoints (Appendix C.3), robust to SFT training hyperparameters (Appendix C.4), and more effective as a post-hoc calibration than as a training-time projection (Appendix C.5). 6 Conclusion We study Geometric Calibration, a perspective for refining SFT updates using the geometry of the pre-SFT weights as a reference. We develop SPACE as a lightweight post-hoc implementation of this perspective, which removes SFT update components within the principal input subspace of the pre-SFT weights while preserving the remaining adaptation. SPACE operates directly on the pre- and post-SFT checkpoints, with no data or additional training required. Overall, across multiple model families, scales, training domains, and SFT variants, SPACE improves both ID and OOD performance over the SFT checkpoint and outperforms representative post-hoc baselines. Our ablations further support the principal right singular subspace of the pre-SFT weights as an effective calibration reference. Taken together, these results highlight the value of pre-SFT weight geometry for refining completed adaptations and, more broadly, motivate Geometric Calibration as a general principle for continued model improvement beyond SFT. AI use statement In this work, we used generative AI tools to assist with language editing and polishing, literature search and discovery, and software code assistance. All identified literature was independently reviewed and verified by the authors. Regarding code assistance, generative AI tools were used only to assist with code editing, completion, and debugging. All AI-assisted code was manually reviewed, tested, and validated. The research ideas, methodology, experimental design, code structure and implementation logic, and interpretation of the results were developed by the authors. All AI-assisted content was validated by the authors. We take responsibility for the final content of this work, including text, claims or artifacts produced with the aid of generative AI. Ethics statement Our study does not involve human-subject experiments or the collection of personal or sensitive data. The experiments are conducted using publicly available models, datasets, and standard evaluation benchmarks. Since SPACE operates on existing language-model checkpoints, the resulting models may retain safety risks associated with the underlying LLMs, including biased, inaccurate, or potentially harmful outputs. Appropriate validation, monitoring, and task-specific safeguards should therefore be maintained in downstream applications. Reproducibility statement Detailed training settings, evaluation protocols, and baseline details are provided in Appendices B.1, B.2, and B.3. Additional experiments examine robustness across models, SFT settings, and training configurations. Our source code is available at: https://github.com/jianzhubao/SPACE. Acknowledgments This research was conducted at the AUMOVIO-NTU Corporate Lab. This research/project is supported by A*STAR under the RIE2025 Industry Alignment Fund – Industry Collaboration Projects (IAF-ICP) Funding Initiative (Award: I2501E0045), as well as cash and in-kind contribution from the industry partner(s). References Abdollahpoorrostam et al. (2026) A. Abdollahpoorrostam, N. Dimitriadis, A. Hazimeh, and P. Frossard MODEL SOUPS NEED ONLY ONE INGREDIENT. In Forty-third International Conference on Machine Learning, Cited by: 8th item, §1, §2, §5.1. Abro and Tahir (2026) A. Abro and M. Tahir Spectral unforgetting: post-hoc recovery of damaged capabilities without retraining. arXiv preprint arXiv:2605.20296. Cited by: 7th item, §1, §2, §5.1. Austin et al. (2021) J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732. Cited by: §5.3. Bandarkar et al. (2025) L. Bandarkar, B. Muller, P. Yuvraj, R. Hou, N. Singhal, H. Lv, and B. Liu Layer swapping for zero-shot cross-lingual transfer in large language models. In International Conference on Learning Representations, Vol. 2025, p. 51315–51338. Cited by: §2. Bandarkar and Peng (2025) L. Bandarkar and N. Peng The unreasonable effectiveness of model merging for cross-lingual transfer in llms. In Proceedings of the 5th Workshop on Multilingual Representation Learning (MRL 2025), p. 131–148. Cited by: §2. Chen et al. (2021) M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. Cited by: §5.3. Chu et al. (2025) T. Chu, Y. Zhai, J. Yang, S. Tong, S. Xie, D. Schuurmans, Q. V. Le, S. Levine, and Y. Ma SFT memorizes, RL generalizes: a comparative study of foundation model post-training. In Forty-second International Conference on Machine Learning, Cited by: §1, §2. Dai et al. (2022) D. Dai, L. Dong, Y. Hao, Z. Sui, B. Chang, and F. Wei Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 8493–8502. Cited by: §5.2. Dubois et al. (2024) Y. Dubois, B. Galambosi, P. Liang, and T. B. Hashimoto Length-controlled alpacaeval: a simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475. Cited by: §B.2, §5.1. Fan (1949) K. Fan On a theorem of weyl concerning eigenvalues of linear transformations i. Proceedings of the National Academy of Sciences 35 (11), p. 652–655. Cited by: Appendix A, §4.1. Geva et al. (2021) M. Geva, R. Schuster, J. Berant, and O. Levy Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 conference on empirical methods in natural language processing, p. 5484–5495. Cited by: §5.2. Grattafiori et al. (2024) A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Cited by: §5.5. He et al. (2024) C. He, R. Luo, Y. Bai, S. Hu, Z. Thai, J. Shen, J. Hu, X. Han, Y. Huang, Y. Zhang, et al. Olympiadbench: a challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 3828–3850. Cited by: §5.1. Hendrycks et al. (2021) D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), Cited by: §5.1. Ho et al. (2023) N. Ho, L. Schmid, and S. Yun Large language models are reasoning teachers. In Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long papers), p. 14852–14882. Cited by: §1. Huan et al. (2026) M. Z. Huan, Y. Li, T. Zheng, X. Xu, S. Kim, M. Du, R. Poovendran, G. Neubig, and X. Yue Does math reasoning improve general LLM capabilities? understanding transferability of LLM reasoning. In Forty-third International Conference on Machine Learning, Cited by: §1, §2, §3. Huang et al. (2025) W. Huang, A. Cheng, and Y. Wang Mitigating catastrophic forgetting in large language models with forgetting-aware pruning. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p. 21853–21867. Cited by: 6th item, §2, §5.1. Huang et al. (2026) X. Huang, J. Wu, H. Liu, X. Tang, and Y. Zhou M1: unleash the potential of test-time scaling for medical reasoning in large language models. In Machine Learning for Health 2025, Cited by: §5.3. Ilharco et al. (2023) G. Ilharco, M. T. Ribeiro, M. Wortsman, L. Schmidt, H. Hajishirzi, and A. Farhadi Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations, Cited by: §2. Jain et al. (2025) N. Jain, A. Gu, W. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica Livecodebench: holistic and contamination free evaluation of large language models for code. In International Conference on Learning Representations, Vol. 2025, p. 58791–58831. Cited by: §B.2, §5.1, §5.3. Jin et al. (2021) D. Jin, E. Pan, N. Oufattole, W. Weng, H. Fang, and P. Szolovits What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences 11 (14), p. 6421. Cited by: §5.3. Jin et al. (2026) H. Jin, T. Ni, L. Li, P. Bacon, M. Hamdaqa, and D. Precup Rotation-preserving supervised fine-tuning. arXiv preprint arXiv:2605.10973. Cited by: §1, §2, §4.1, §5.4. Jin et al. (2019) Q. Jin, B. Dhingra, Z. Liu, W. Cohen, and X. Lu Pubmedqa: a dataset for biomedical research question answering. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), p. 2567–2577. Cited by: §5.3. Kopiczko et al. (2026) D. J. Kopiczko, S. Vaze, T. Blankevoort, and Y. M. Asano Data repetition beats data scaling in long-cot supervised fine-tuning. In Third Conference on Language Modeling, Cited by: §2. Lai et al. (2025) H. Lai, X. Liu, J. Gao, J. Cheng, Z. Qi, Y. Xu, S. Yao, D. Zhang, J. Du, Z. Hou, et al. A survey of post-training scaling in large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 2771–2791. Cited by: §1. Lai et al. (2026) S. Lai, H. Zhao, R. Feng, C. Ma, W. Liu, H. Zhao, X. Lin, D. Yi, Q. Zhang, H. Liu, G. Meng, and F. Zhu Reinforcement fine-tuning naturally mitigates forgetting in continual post-training. In Forty-third International Conference on Machine Learning, Cited by: §1. Lewkowycz et al. (2022) A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ramasesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in neural information processing systems 35, p. 3843–3857. Cited by: §5.1. Li et al. (2025) G. Li, R. Qiu, X. Chen, H. Ji, and H. Tong Beyond log likelihood: probability-based objectives for supervised fine-tuning across the model capability continuum. arXiv preprint arXiv:2510.00526. Cited by: §2, §5.3. Li et al. (2024) J. Li, E. Beeching, L. Tunstall, B. Lipkin, R. Soletskyi, S. Huang, K. Rasul, L. Yu, A. Q. Jiang, Z. Shen, et al. Numinamath: the largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository. Cited by: §5.1. Li et al. (2023) J. Li, X. Cheng, X. Zhao, J. Nie, and J. Wen Halueval: a large-scale hallucination evaluation benchmark for large language models. In Proceedings of the 2023 conference on empirical methods in natural language processing, p. 6449–6464. Cited by: §B.2, §5.1. Li et al. (2026) L. Li, L. Qi, Q. Tian, and X. Geng Model merging in the essential subspace. arXiv preprint arXiv:2602.20208. Cited by: §2. Liu et al. (2023) J. Liu, C. S. Xia, Y. Wang, and L. Zhang Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in neural information processing systems 36, p. 21558–21572. Cited by: §5.3. Liu et al. (2026) Q. Liu, J. Gu, Y. Yao, H. Wang, and N. V. Peng Energy-regularized sequential model editing on hyperspheres. In International Conference on Learning Representations, Vol. 2026, p. 149084–149112. Cited by: §1. Liu et al. (2025a) Y. Liu, D. Fu, Y. Luo, Z. Zhu, M. Cheng, C. Hsieh, and Y. You POME: post optimization model edit via muon-style projection. arXiv preprint arXiv:2510.06627. Cited by: 5th item, §5.1. Liu et al. (2025b) Z. Liu, T. Pang, O. Balabanov, C. Yang, T. Huang, L. Yin, Y. Yang, and S. Liu LIFT the veil for the truth: principal weights emerge after rank reduction for reasoning-focused supervised fine-tuning. In Forty-second International Conference on Machine Learning, External Links: Link Cited by: §2. Luo et al. (2026) M. Luo, Z. Zhou, Y. Zhang, Y. Wan, M. Zhang, and T. Wei KeeploRA: continual learning with residual gradient adaptation. In International Conference on Learning Representations, Vol. 2026, p. 136017–136040. Cited by: §1, §2, §4.1. Mathematical Association of America (2023) Mathematical Association of America AMC 2023 competition problems. Cited by: §5.1. Meng et al. (2024) F. Meng, Z. Wang, and M. Zhang Pissa: principal singular values and singular vectors adaptation of large language models. Advances in Neural Information Processing Systems 37, p. 121038–121072. Cited by: §2. Meng et al. (2022) K. Meng, D. Bau, A. Andonian, and Y. Belinkov Locating and editing factual associations in gpt. Advances in neural information processing systems 35, p. 17359–17372. Cited by: §5.2. Mukherjee et al. (2026) S. Mukherjee, L. Yuan, D. Hakkani-Tur, and H. Peng Reinforcement learning finetunes small subnetworks in large language models. Advances in Neural Information Processing Systems 38, p. 132119–132138. Cited by: §2, §3. Olmo et al. (2025) T. Olmo, A. Ettinger, A. Bertsch, B. Kuehl, D. Graham, D. Heineman, D. Groeneveld, F. Brahman, F. Timbers, H. Ivison, et al. Olmo 3. arXiv preprint arXiv:2512.13961. Cited by: §5.5. Pal et al. (2022) A. Pal, L. K. Umapathi, and M. Sankarasubbu Medmcqa: a large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on health, inference, and learning, p. 248–260. Cited by: §5.3. Qiu et al. (2026) Z. Qiu, L. Wang, Y. Cao, R. Zhang, B. Su, Y. Xu, F. Meng, L. Xu, Q. Wu, and H. Li Null-space filtering for data-free continual model merging: preserving stability, promoting plasticity. In International Conference on Learning Representations, Vol. 2026, p. 128492–128518. Cited by: §2. Rein et al. (2024) D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman GPQA: a graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, Cited by: §B.2, §5.1, §5.3. Ren et al. (2026) Q. Ren, W. Peng, R. Cai, S. Shao, D. Guo, Y. Xie, Y. Li, Q. Zhang, X. Hu, J. Shao, and D. Liu Rethinking generalization in reasoning SFT: a conditional analysis on optimization, data, and model capability. In Third Conference on Language Modeling, Cited by: §B.2, §B.2, §2, §5.1, §5.1. Shen et al. (2026) Z. Shen, Y. Li, Q. Yin, C. T. Leong, Z. Wang, Y. Chen, R. Han, S. Lee, and Y. R. Fung On the geometry of on-policy distillation. arXiv preprint arXiv:2606.07082. Cited by: §1, §2, §4.1. Shenfeld et al. (2026) I. Shenfeld, J. Pari, and P. Agrawal Rl’s razor: why online reinforcement learning forgets less. In International Conference on Learning Representations, Vol. 2026, p. 59839–59864. Cited by: §1, §2, §3. Shi et al. (2026) N. Shi, Z. Zhou, H. Wang, C. Miao, and P. Zhao PACT: preserving anchored cores in task-vectors for model merging. arXiv preprint arXiv:2606.18627. Cited by: §2. Sun et al. (2026) W. Sun, Q. Li, T. Li, Y. Geng, and B. Li Revisiting the role of pretrained weights in model merging: on near-optimality within the core subspace. In Forty-third International Conference on Machine Learning, Cited by: §2, §4.1. Wang et al. (2025a) H. Wang, Y. Li, S. Wang, G. Chen, and Y. Chen Milora: harnessing minor singular components for parameter-efficient llm finetuning. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), p. 4823–4836. Cited by: §1, §2, §4.1. Wang et al. (2025b) K. Wang, N. Dimitriadis, A. Favero, G. Ortiz-Jimenez, F. Fleuret, and P. Frossard Lines: post-training layer scaling prevents forgetting and enhances model merging. In International Conference on Learning Representations, Vol. 2025, p. 27100–27133. Cited by: 3rd item, §1, §2, §5.1. Wang et al. (2024) Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang, et al. Mmlu-pro: a more robust and challenging multi-task language understanding benchmark. Advances in Neural Information Processing Systems 37, p. 95266–95290. Cited by: §B.2, §5.1, §5.3. Wei et al. (2024) Y. Wei, Z. Wang, J. Liu, Y. Ding, and L. ZHANG Magicoder: empowering code generation with OSS-instruct. In Forty-first International Conference on Machine Learning, External Links: Link Cited by: §5.3. Wortsman et al. (2022a) M. Wortsman, G. Ilharco, S. Y. Gadre, R. Roelofs, R. Gontijo-Lopes, A. S. Morcos, H. Namkoong, A. Farhadi, Y. Carmon, S. Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International conference on machine learning, p. 23965–23998. Cited by: §2. Wortsman et al. (2022b) M. Wortsman, G. Ilharco, J. W. Kim, M. Li, S. Kornblith, R. Roelofs, R. G. Lopes, H. Hajishirzi, A. Farhadi, H. Namkoong, et al. Robust fine-tuning of zero-shot models. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 7949–7961. Cited by: 1st item, §1, §2, §5.1. Wu et al. (2026a) C. Wu, Z. R. Tam, C. Lin, Y. V. Chen, S. Sun, and H. Lee Mitigating forgetting in llm fine-tuning via low-perplexity token learning. Advances in Neural Information Processing Systems 38, p. 1708–1744. Cited by: §3. Wu et al. (2026b) Y. Wu, Y. Zhou, Z. Zhou, Y. Peng, X. Ye, X. Hu, W. Zhu, L. Qi, M. Yang, et al. On the generalization of sft: a reinforcement learning perspective with reward rectification. In International Conference on Learning Representations, Vol. 2026, p. 27550–27571. Cited by: §B.2, §2, §5.1, §5.4. Xiong and Xie (2026) Y. Xiong and X. Xie Oplora: orthogonal projection lora prevents catastrophic forgetting during parameter-efficient fine-tuning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40, p. 34088–34096. Cited by: §2. Yadav et al. (2023) P. Yadav, D. Tam, L. Choshen, C. A. Raffel, and M. Bansal Ties-merging: resolving interference when merging models. Advances in neural information processing systems 36, p. 7093–7115. Cited by: 2nd item, §5.1. Yang et al. (2025a) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: §5.1. Yang et al. (2025b) A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. Cited by: §5.1. Yang et al. (2024) A. Yang, B. Zhang, B. Hui, B. Gao, B. Yu, C. Li, D. Liu, J. Tu, J. Zhou, J. Lin, et al. Qwen2.5-math technical report: toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122. Cited by: §B.2. Ye et al. (2025) Y. Ye, Z. Huang, Y. Xiao, E. Chern, S. Xia, and P. Liu LIMO: less is more for reasoning. In Second Conference on Language Modeling, Cited by: §1, §2. Yu et al. (2024) L. Yu, B. Yu, H. Yu, F. Huang, and Y. Li Language models are super mario: absorbing abilities from homologous models as a free lunch. In Forty-first International Conference on Machine Learning, Cited by: 4th item, §2, §2, §5.1. Yuan et al. (2025) X. Yuan, X. Chen, T. Yu, D. Shi, C. Jin, W. Lee, and S. Mitra Mitigating forgetting between supervised and reinforcement learning yields stronger reasoners. arXiv preprint arXiv:2510.04454. Cited by: §3. Zhang et al. (2026) J. Zhang, L. Shi, J. Li, J. Xu, J. Gao, J. Hao, and R. He GeoRA: geometry-aware low-rank adaptation for rlvr. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 24207–24221. Cited by: §1, §2. Zhang and Math-AI (2024) Y. Zhang and T. Math-AI American invitational mathematics examination (aime) 2024. Cited by: §5.1. Zhang and Math-AI (2025) Y. Zhang and T. Math-AI American invitational mathematics examination (aime) 2025. Cited by: §5.1. Zhou et al. (2023) J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, Y. Luan, D. Zhou, and L. Hou Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911. Cited by: §B.2, §5.1. Zhu et al. (2025) H. Zhu, Z. Zhang, H. Huang, D. Su, Z. Liu, J. Zhao, I. Fedorov, H. Pirsiavash, Z. Sha, J. Lee, et al. The path not taken: rlvr provably learns off the principals. arXiv preprint arXiv:2511.08567. Cited by: §1, §2, §4.1. Zhu et al. (2026) H. Zhu, J. Su, P. Lai, R. Ma, W. Zhang, L. Yang, and G. Chen Anchored supervised fine-tuning. In International Conference on Learning Representations, Vol. 2026, p. 110364–110385. Cited by: §B.2, §2, §5.1, §5.3, §5.4. Appendix Contents Appendix A Derivation of the Principal Input Projector We derive the solution to the projection problem in equation 4. Specifically, we solve the following optimization problem: P0∈argminP=P⊤=P2rank(P)=k‖W0−W0P‖F2. P_0∈ subarraycP=P =P^2\\ rank(P)=k subarray \|W_0-W_0P\|_F^2. Since P=P⊤=P2P=P =P^2, ‖W0−W0P‖F2 \|W_0-W_0P\|_F^2 =‖W0(I−P)‖F2 =\|W_0(I-P)\|_F^2 =tr((I−P)W0⊤W0(I−P)) =tr\! ((I-P)W_0 W_0(I-P) ) =tr(W0⊤W0)−tr(PW0⊤W0). =tr(W_0 W_0)-tr(PW_0 W_0). Therefore, minimizing equation 4 is equivalent to maximizing tr(PW0⊤W0)tr(PW_0 W_0) over rank-k orthogonal projectors. Writing W0=U0Σ0V0⊤,W_0=U_0 _0V_0 , we have W0⊤W0=V0Σ02V0⊤.W_0 W_0=V_0 _0^2V_0 . Any rank-k orthogonal projector can be written as P=QQ⊤,Q⊤Q=Ik.P=Q , Q Q=I_k. Therefore, tr(PW0⊤W0)=tr(Q⊤W0⊤W0Q).tr(PW_0 W_0)=tr(Q W_0 W_0Q). By the Ky Fan maximum principle (Fan, 1949), maxQ⊤Q=Iktr(Q⊤W0⊤W0Q)=∑i=1kσ0,i2, _Q Q=I_ktr(Q W_0 W_0Q)= _i=1^k _0,i^2, and the maximum is attained when the columns of Q span the eigenspace associated with the k largest eigenvalues of W0⊤W0W_0 W_0. Since these eigenvectors are precisely the leading k right singular vectors of W0W_0, we may choose Q=V0,k.Q=V_0,k. Hence, an optimal projector is P0=V0,kV0,k⊤.P_0=V_0,kV_0,k . Appendix B Detailed Experimental Settings B.1 Detailed Training Settings of the Main Experiments Unless otherwise specified, we use AdamW with a global batch size of 256 and a learning rate of 5×10−55× 10^-5. The learning rate follows a cosine decay schedule with a warm-up ratio of 0.10.1, and all models are trained for one epoch. We use standard next-token prediction over the response tokens as the SFT objective. All experiments are conducted using four NVIDIA H200 GPUs. B.2 Detailed Evaluation Settings of the Main Experiments Following prior studies (Wu et al., 2026b; Zhu et al., 2026), for the mathematical benchmarks (MATH-500, Minerva Math, OlympiadBench, AIME 2024, AIME 2025, and AMC 2023), we adopt the Qwen2.5-Math (Yang et al., 2024) evaluation pipeline and use the default CoT prompt template with a temperature of 1.01.0 and top-p of 1.01.0. Following the benchmark suite used by Ren et al. (2026), we further consider OOD benchmarks. In particular, we use MMLU-Pro (Wang et al., 2024) for broad knowledge-intensive reasoning, GPQA-Diamond (GPQA-D) (Rein et al., 2024) for graduate-level scientific reasoning, LiveCodeBench v2 (LCB v2) (Jain et al., 2025) for code reasoning, IFEval (Zhou et al., 2023) for instruction following, AlpacaEval (RM) (Dubois et al., 2024) for open-ended response quality, and HaluEval (Li et al., 2023) for hallucination detection. For these evaluations, we follow the prompt templates and evaluation settings of Ren et al. (2026), using a temperature of 0.60.6 and top-p of 0.950.95. The remaining evaluation settings differ slightly between Qwen2.5-7B and Qwen3-14B-Base, as their training data have substantially different maximum CoT lengths (approximately 3K vs. 16K tokens), leading to different inference costs. For Qwen2.5-7B, we set the maximum generation length to 4,096 tokens. For Qwen3-14B-Base, we set the maximum generation length to 16,384 tokens. Moreover, for Qwen2.5-7B, we report avg@16 on the mathematical benchmarks and GPQA-Diamond, and evaluate MMLU-Pro and HaluEval on their full test sets. For Qwen3-14B-Base, following Ren et al. (2026), we report avg@3 on MATH-500, Minerva Math, OlympiadBench, and GPQA-Diamond, and avg@16 on the remaining mathematical benchmarks. Also, MMLU-Pro and HaluEval are evaluated on subsets of 1K and 3K examples. For both models, we report pass@1 for MMLU-Pro and HaluEval, strict instruction-level accuracy for IFEval, and avg@3 for LiveCodeBench v2. For AlpacaEval (RM), we generate one response for each instruction and report the average reward score from the allenai/Llama-3.1-8B-Instruct-RM-RB2 reward model.11 1 https://huggingface.co/allenai/Llama-3.1-8B-Instruct-RM-RB2 B.3 Baseline Details We compare SPACE with the following representative post-hoc model editing/merging baselines. • WiSE-FT (Wortsman et al., 2022b) linearly interpolates the pre-SFT and SFT checkpoints. • TIES (Yadav et al., 2023) applies task-vector trimming and sign-aware merging. • LiNeS (Wang et al., 2025b) applies depth-dependent layer-wise scaling to the fine-tuning update. • DARE (Yu et al., 2024) sparsifies the fine-tuning update through random dropping and rescaling. • POME (Liu et al., 2025a) refines the fine-tuning update through Muon-style projection. • FAPM (Huang et al., 2025) performs forgetting-aware pruning of the fine-tuning update. • DG-Hard (Abro and Tahir, 2026) performs spectral filtering of the fine-tuning update. • MonoSoup (Abdollahpoorrostam et al., 2026) decomposes and reweights spectral components of the fine-tuning update. Implementation details. We follow the recommended settings in the original papers or their released implementations, with the following adaptations. For TIES, we use a single SFT update, so the method reduces to global magnitude trimming. For FAPM, the default setting performs poorly, likely due to excessive update pruning in our setting. With the scoring rule unchanged, we evaluate several retention ratios and choose to retain 90%90\% of entries per tensor, which provides good overall performance. Comparison with SPACE. WiSE-FT and LiNeS scale updates globally or by layer depth. TIES, DARE, and FAPM prune individual update entries using magnitude-based scores or random masking. Although some of these methods incorporate pre-SFT weight information, they do not explicitly account for the geometry of the pre-SFT weights. POME, DG-Hard, and MonoSoup exploit spectral structure, but define their transformations using the learned update’s own singular components. SPACE instead calibrates post-SFT checkpoints against the geometry of the pre-SFT weights, removing update components within their principal input subspaces while preserving the update on the orthogonal complements. Appendix C Additional Experiments C.1 Compatibility with LoRA Fine-Tuning We further examine whether SPACE is compatible with parameter-efficient fine-tuning. Specifically, we fine-tune Qwen2.5-7B with LoRA on NuminaMath-CoT using rank r=64r=64 and scaling factor α=128α=128, while keeping the remaining training and evaluation settings consistent with Section 5.1. We then apply SPACE to the resulting LoRA checkpoint using the same pre-SFT checkpoint as the geometric reference. As shown in Table 5, SPACE improves average ID and OOD performance over the original LoRA checkpoint, with particularly pronounced OOD gains. It also achieves larger aggregate gains than MonoSoup. These results suggest that the applicability of SPACE extends beyond full-parameter SFT to parameter-efficient adaptation. Table 5: Compatibility of SPACE with LoRA fine-tuning on Qwen2.5-7B (r=64r=64, α=128α=128). “Original” denotes the LoRA fine-tuned checkpoint. “Avg. (Δ vs. Original)” denotes the average score change after applying MonoSoup or SPACE relative to Original within each benchmark group. Bold indicates the best score. Benchmark LoRA Original MonoSoup SPACE In-domain (ID) MATH500 45.30 46.06 46.34 AIME25 0.21 1.25 0.63 AIME24 0.83 0.83 2.50 OlympiadBench 14.82 14.95 15.40 Minerva Math 12.94 13.03 13.63 AMC23 20.16 17.97 18.91 Avg. (Δ vs. Original) - -0.03 +0.53 Out-of-domain (OOD) MMLU-Pro 38.20 37.28 43.28 GPQA-D 22.35 22.85 27.84 LCB v2 27.98 28.25 27.98 IFEval 42.81 44.00 48.44 AlpacaEval -0.08 -0.06 -0.12 HaluEval 46.75 46.87 51.22 Avg. (Δ vs. Original) - +0.20 +3.44 C.2 Additional Results on Model Families and Scales We provide additional results for the model generalization experiments in Section 5.5. In addition to the models reported in the main text, we include Qwen3-4B-Base and compare SPACE with MonoSoup across all evaluated models. Tables 6 and 7 show that SPACE consistently improves the average ID and OOD performance of the corresponding SFT checkpoints across all evaluated models. The gains remain consistent across the evaluated Qwen scales and extend to the Olmo and Llama model families. Moreover, SPACE achieves larger average improvements than MonoSoup on both ID and OOD benchmarks for every evaluated model, further demonstrating its applicability. Table 6: Generalization of SPACE across Qwen models at different scales (2B, 4B, and 8B). Benchmark Qwen3.5-2B-Base Qwen3-4B-Base Qwen3-8B-Base Base SFT MonoSoup SPACE Base SFT MonoSoup SPACE Base SFT MonoSoup SPACE In-domain (ID) MATH500 72.40 76.47 78.07 78.00 40.13 78.73 80.53 81.33 55.93 83.73 84.67 84.87 AIME25 15.63 17.29 17.29 17.50 2.29 17.50 20.00 18.13 4.58 23.96 23.96 24.38 AIME24 12.92 16.67 14.79 15.21 5.63 17.71 16.25 18.54 7.29 22.71 23.33 22.71 OlympiadBench 37.09 40.15 41.93 41.38 19.56 40.89 43.26 42.72 24.05 46.86 48.49 48.79 Minerva Math 27.08 39.95 40.81 40.32 16.79 37.99 36.76 39.22 19.49 41.79 42.28 40.44 AMC23 49.22 53.59 54.53 55.16 22.03 56.41 57.19 59.69 32.50 64.22 65.47 67.34 Avg. (Δ vs. SFT) - - +0.55 +0.58 - - +0.79 +1.73 - - +0.82 +0.88 Out-of-domain (OOD) MMLU-Pro 46.40 52.60 51.80 54.00 37.10 47.30 48.10 49.20 56.70 56.40 56.50 56.20 GPQA-D 30.64 29.97 32.83 32.66 18.86 21.04 20.54 21.72 36.36 29.46 28.96 29.46 LCB v2 14.94 11.55 11.35 13.31 31.83 15.33 17.87 18.33 33.66 24.92 27.66 29.29 IFEval 57.67 45.32 51.80 53.84 46.76 30.46 31.65 32.61 54.08 35.85 37.17 40.89 AlpacaEval 0.21 -0.84 -0.52 -0.23 -2.33 -3.08 -2.83 -2.71 -0.39 -2.32 -1.75 -1.10 HaluEval 50.00 59.97 59.87 59.20 48.50 55.13 55.70 57.17 49.33 72.13 74.00 72.30 Avg. (Δ vs. SFT) - - +1.43 +2.37 - - +0.81 +1.69 - - +1.02 +1.77 Table 7: Generalization of SPACE across Olmo and Llama model families. Benchmark Olmo-3-7B-Instruct-SFT Llama-3.1-8B-Instruct Base SFT MonoSoup SPACE Base SFT MonoSoup SPACE In-domain (ID) MATH500 60.80 87.93 88.67 89.33 39.07 49.20 52.80 53.33 AIME25 3.33 35.00 30.83 34.38 0.21 1.46 1.04 1.25 AIME24 6.67 41.25 38.33 41.46 1.88 1.67 2.50 2.29 OlympiadBench 25.88 55.26 55.75 56.79 10.47 18.96 19.46 21.04 Minerva Math 25.74 33.95 34.19 34.56 15.56 18.01 20.47 21.57 AMC23 37.03 79.53 80.94 82.19 14.69 26.09 27.50 30.16 Avg. (Δ vs. SFT) - - -0.70 +0.97 - - +1.40 +2.38 Out-of-domain (OOD) MMLU-Pro 42.70 44.40 43.20 44.90 43.50 40.80 44.40 44.80 GPQA-D 28.79 22.22 25.59 25.93 23.74 15.49 18.86 19.87 LCB v2 17.09 22.44 21.98 21.14 18.92 8.41 11.61 12.20 IFEval 83.57 79.26 78.90 82.49 81.89 29.14 34.65 36.81 AlpacaEval 0.28 0.34 0.34 0.38 1.17 -1.87 -0.83 -0.38 HaluEval 54.23 55.47 55.83 55.20 62.73 63.37 63.77 63.97 Avg. (Δ vs. SFT) - - +0.29 +0.99 - - +2.85 +3.66 C.3 Performance at Intermediate SFT Checkpoints Setup. We further examine whether SPACE remains effective when applied before the end of SFT. For the Qwen2.5-7B mathematical SFT experiment, one epoch corresponds to approximately 390 optimization steps. We save intermediate checkpoints at 100, 200, and 300 steps along the same training trajectory, in addition to the final 390-step checkpoint. SPACE is applied independently to each checkpoint using the same default configuration as in the main experiments. We evaluate all checkpoints on the same six ID and six OOD benchmarks described in Section 5.1. For clarity, we report the average performance change relative to the final 390-step SFT checkpoint. Figure 3: Performance of SFT and SPACE at different stages of the Qwen2.5-7B training trajectory. We report the average change over the six ID and six OOD benchmarks relative to the final 390-step SFT checkpoint. The 390-step checkpoint corresponds to one epoch of SFT. Results. As shown in Figure 3, the benefits of SPACE emerge early during SFT. After only 100 training steps, SPACE already outperforms the final 390-step SFT checkpoint by 1.28 points on ID benchmarks and 2.10 points on OOD benchmarks, whereas the corresponding 100-step SFT checkpoint remains 1.09 and 0.70 points below the final SFT checkpoint, respectively. As training proceeds, the ID improvement from SPACE further increases, reaching 2.85 points at 390 steps, while the OOD improvement remains relatively stable throughout training. These results show that SPACE does not rely on a fully trained SFT checkpoint and can provide substantial gains even at an early stage of adaptation. C.4 Robustness to SFT Training Hyperparameters Setup. We further examine the robustness of SPACE to different SFT training hyperparameters. Using Qwen2.5-7B fine-tuned on NuminaMath-CoT, we vary the batch size, learning rate, and number of training epochs while keeping the remaining settings fixed. For all experiments, SPACE uses the same default configuration as in the main experiments. Results. Tables 8, 9, and 10 show that SPACE yields positive average gains over the corresponding SFT checkpoints across all evaluated training hyperparameters. Although the magnitude of the gains varies across settings, the improvements consistently extend to both ID and OOD performance. These results suggest that SPACE is robust to variations in SFT optimization, rather than being specific to a particular choice of training hyperparameters. Table 8: Performance of SPACE across SFT batch sizes for Qwen2.5-7B fine-tuned on NuminaMath-CoT. The learning rate is fixed at 5×10−55× 10^-5, and training lasts one epoch. Projection uses ρ=0.5ρ=0.5, α=1α=1, and all weight matrices. Bold marks the best score in each pair. † denotes the hyperparameters we used in the main experiment. Benchmark Batch size 128 Batch size 256† Batch size 512 SFT SPACE SFT SPACE SFT SPACE In-domain (ID) MATH500 43.40 48.88 44.84 49.36 45.31 48.45 AIME25 0.83 1.04 0.83 1.46 0.42 0.83 AIME24 0.21 2.08 1.04 1.46 0.63 1.25 OlympiadBench 13.06 18.49 14.08 17.72 14.31 17.70 Minerva Math 12.39 15.72 12.68 15.26 12.71 14.36 AMC23 17.34 25.31 19.53 24.84 18.44 22.34 Avg. (Δ vs. SFT) - +4.05 - +2.85 - +2.19 Out-of-domain (OOD) MMLU-Pro 38.42 43.28 38.05 42.03 36.09 39.30 GPQA-D 25.19 26.07 23.36 27.27 23.74 23.93 LCB v2 24.92 27.40 26.22 28.70 25.51 29.09 IFEval 44.36 46.04 42.57 44.60 38.49 41.97 AlpacaEval -0.26 -0.25 -0.14 -0.10 -0.02 0.02 HaluEval 46.60 48.76 46.75 47.85 46.72 47.42 Avg. (Δ vs. SFT) - +2.01 - +2.26 - +1.87 Table 9: Performance of SPACE across SFT learning rates for Qwen2.5-7B fine-tuned on NuminaMath-CoT. The batch size is fixed at 256, and training lasts one epoch. Projection uses ρ=0.5ρ=0.5, α=1α=1, and all weight matrices. Bold marks the best score in each pair. † denotes the hyperparameters we used in the main experiment. Benchmark LR 2e-5 LR 5e-5† LR 1e-4 SFT SPACE SFT SPACE SFT SPACE In-domain (ID) MATH500 45.60 44.80 44.84 49.36 39.41 49.21 AIME25 0.42 0.00 0.83 1.46 0.63 1.25 AIME24 1.67 1.46 1.04 1.46 1.04 1.25 OlympiadBench 14.83 14.29 14.08 17.72 10.98 18.18 Minerva Math 12.98 10.27 12.68 15.26 8.39 14.89 AMC23 17.50 22.50 19.53 24.84 16.09 21.25 Avg. (Δ vs. SFT) - +0.05 - +2.85 - +4.92 Out-of-domain (OOD) MMLU-Pro 35.26 40.09 38.05 42.03 38.18 42.14 GPQA-D 25.57 28.16 23.36 27.27 24.18 25.32 LCB v2 29.22 30.53 26.22 28.70 12.46 19.96 IFEval 48.08 48.92 42.57 44.60 33.33 38.13 AlpacaEval 0.10 -0.12 -0.14 -0.10 -0.92 -0.60 HaluEval 47.64 48.66 46.75 47.85 48.92 49.38 Avg. (Δ vs. SFT) - +1.73 - +2.26 - +3.03 Table 10: Performance of SPACE across SFT training epochs for Qwen2.5-7B fine-tuned on NuminaMath-CoT. The learning rate is fixed at 5×10−55× 10^-5, and the batch size is fixed at 256. Projection uses ρ=0.5ρ=0.5, α=1α=1, and all weight matrices. Bold marks the best score in each pair. Benchmark Epochs 2 Epochs 4 Epochs 8 SFT SPACE SFT SPACE SFT SPACE In-domain (ID) MATH500 53.06 53.55 56.04 56.04 55.49 58.43 AIME25 1.88 1.25 2.29 1.04 2.29 2.50 AIME24 2.08 2.92 2.71 2.71 3.54 3.33 OlympiadBench 19.06 20.27 22.77 23.36 23.43 26.32 Minerva Math 16.38 17.03 19.65 18.73 19.19 20.68 AMC23 26.41 28.28 28.12 30.31 28.44 33.59 Avg. (Δ vs. SFT) - +0.74 - +0.10 - +2.08 Out-of-domain (OOD) MMLU-Pro 47.17 48.76 48.86 49.96 43.66 48.11 GPQA-D 25.09 28.03 28.85 30.84 26.36 30.93 LCB v2 31.70 31.44 30.01 28.96 21.00 27.59 IFEval 46.04 47.24 44.84 47.60 47.48 47.36 AlpacaEval 0.15 0.19 0.44 0.40 0.36 0.60 HaluEval 51.22 52.57 58.42 57.30 56.33 56.70 Avg. (Δ vs. SFT) - +1.14 - +0.61 - +2.68 C.5 Training-Time vs. Post-Hoc Projection We further compare different ways of incorporating the projection used by SPACE into SFT. In addition to the default post-hoc SPACE, we consider a projection-based training loss that penalizes updates within the pre-SFT principal subspace, as well as an online variant that applies the SPACE projection every K optimization steps during training. We evaluate K=1K=1 and K=100K=100, and include RPSFT as a related training-time regularization baseline. RPSFT regularizes the update jointly within the leading left and right singular subspaces, whereas our projection loss only penalizes updates along the leading right singular subspace. All experiments use Qwen2.5-7B trained on NuminaMath-CoT with the same training and evaluation settings as the main experiments. As shown in Table 11, the default post-hoc SPACE achieves the largest average gains on both ID and OOD benchmarks. Training-time projection loss improves OOD performance but provides little average ID gain, while online projection yields only modest improvements for both values of K. These results suggest that allowing SFT to learn its update first and calibrating it afterward is more effective than constraining or repeatedly projecting the update during optimization, which may interfere with the learning process. Table 11: Comparison of training-time and post-hoc projection on Qwen2.5-7B. “Proj. Loss” introduces projection-based regularization during SFT with weight λ=1λ=1. “Online Proj.” applies the projection every K optimization steps and at the end of training. SPACE denotes the default post-hoc projection. “Avg. (Δ vs. SFT)” denotes the average score change relative to SFT. Bold denotes the best score excluding the Base model. Benchmark Base SFT RPSFT (λ=1λ=1) Proj. Loss (λ=1λ=1) Online Proj. (K=1K=1) Online Proj. (K=100K=100) SPACE (post-hoc) In-domain (ID) MATH500 28.79 44.84 45.98 45.66 46.08 45.54 49.36 AIME25 0.42 0.83 1.04 0.63 0.42 0.63 1.46 AIME24 1.65 1.04 1.46 1.04 0.83 1.25 1.46 OlympiadBench 7.69 14.08 14.40 13.89 14.54 14.73 17.72 Minerva Math 9.26 12.68 13.01 12.73 12.80 12.64 15.26 AMC23 15.65 19.53 21.72 19.38 19.69 20.78 24.84 Avg. (Δ vs. SFT) - - +0.77 +0.06 +0.23 +0.43 +2.85 Out-of-domain (OOD) MMLU-Pro 40.33 38.05 34.67 37.50 37.59 37.83 42.03 GPQA-D 26.77 23.36 24.15 25.73 24.59 23.48 27.27 LCB v2 28.96 26.22 29.03 29.75 28.44 27.40 28.70 IFEval 42.57 42.57 44.48 46.76 43.41 44.00 44.60 AlpacaEval -1.13 -0.14 -0.03 0.05 -0.14 -0.05 -0.10 HaluEval 46.86 46.75 46.86 47.42 45.64 46.62 47.85 Avg. (Δ vs. SFT) - - +0.39 +1.73 +0.45 +0.41 +2.26