Paper deep dive
SoFT: Soft Targets for Generalizable LLM Fine-Tuning
Huihao Jing, Wenbin Hu, Shaojin Chen, Haochen Shi, Zhongwei Xie, Guijia Zhang, Yuxuan Liu, Haoyu Huang, Haoran Li, Yangqiu Song
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Distillation enables student language models to acquire new capabilities from expert teachers. However, integrating knowledge from multi-teacher, multi-domain demonstrations into a single student remains challenging. We study supervised fine-tuning (SFT) in this setting, where students must acquire diverse capabilities while maintaining generalization beyond the training tasks. Our experiments reveal varying trade-offs between in-distribution learning and out-of-distribution generalization across SFT methods, motivating more explicit control over this balance. To this end, we propose soft-target fine-tuning (SoFT) to balance learning from teacher demonstrations with retaining the Base model's existing capabilities. SoFT sets a minimum target probability for each demonstrated token while making the smallest KL change to the Base distribution. The resulting objective couples learning from demonstrations with adaptively weighted regularization toward the Base model. We further use domain-specific gradient budgets to control this balance and determine a probability threshold for each trajectory. Experiments on mixed-domain reasoning and agentic tasks show that SoFT achieves the best overall performance among the compared methods, with improvements in both in-distribution capability acquisition and out-of-distribution generalization.
Tags
Links
- Source: https://arxiv.org/abs/2609.32493v1
- Canonical: https://arxiv.org/abs/2609.32493v1
Trouble viewing inline? Open PDF directly →
Full Text
80,729 characters extracted from source content.
Expand or collapse full text
SoFT: Soft Targets for Generalizable LLM Fine-Tuning Huihao Jing 11 1 Equal Contribution, Wenbin Hu 11footnotemark: 1 , Shaojin Chen , Haochen Shi , Zhongwei Xie , Guijia Zhang , Yuxuan Liu , Haoyu Huang , Haoran Li 22 2 Corresponding author , Yangqiu Song HKUST, HKUST-GZ hjingaa@connect.ust.hk Abstract Distillation enables student language models to acquire new capabilities from expert teachers. However, integrating knowledge from multi-teacher, multi-domain demonstrations into a single student remains challenging. We study supervised fine-tuning (SFT) in this setting, where students must acquire diverse capabilities while maintaining generalization beyond the training tasks. Our experiments reveal varying trade-offs between in-distribution learning and out-of-distribution generalization across SFT methods, motivating more explicit control over this balance. To this end, we propose soft-target fine-tuning (SoFT) to balance learning from teacher demonstrations with retaining the Base model’s existing capabilities. SoFT sets a minimum target probability for each demonstrated token while making the smallest KL change to the Base distribution. The resulting objective couples learning from demonstrations with adaptively weighted regularization toward the Base model. We further use domain-specific gradient budgets to control this balance and determine a probability threshold for each trajectory. Experiments on mixed-domain reasoning and agentic tasks show that SoFT achieves the best overall performance among the compared methods, with improvements in both in-distribution capability acquisition and out-of-distribution generalization. Figure 1: SoFT overview and mixed-data outcomes. Left: schematic targets; SFT uses a one-hot target, while SoFT retains Base-supported alternatives. Right: changes relative to Base across reasoning and agentic domains. 1 Introduction Supervised fine-tuning (SFT) on expert demonstrations is a standard way to transfer new capabilities to a language model (Ouyang et al., 2022; Lambert et al., 2024). Demonstrations may contain intermediate reasoning (Muennighoff et al., 2025; DeepSeek-AI, 2025; Guha et al., 2025), tool decisions, or long multi-turn interactions (Raoof and others, 2026; Jung et al., 2026), and practical corpora often mix trajectories from several teachers and task domains (Wan et al., 2024; Guha et al., 2025). A single student must then acquire all of these capabilities without losing the useful behavior it learned during pretraining (Li et al., 2025; Wang et al., 2026b). Standard SFT is poorly suited to this goal. A teacher trajectory is only one of many valid continuations: another teacher, or the student itself, might use different wording, reasoning steps, tool calls, or stopping points (Kim et al., 2025; Raoof and others, 2026). Yet SFT trains toward a one-hot target on every observed token. Repeated updates therefore favor the demonstrated trajectory at the expense of other plausible continuations (Li et al., 2025; Chen et al., 2025), especially in long reasoning and agentic trajectories (Twist et al., 2026; Jung et al., 2026). SFT also ignores the student’s starting point, pushing every demonstrated token toward certainty whether Base already predicts it well or finds it unlikely (Chen et al., 2025; Wang et al., 2026a). Under mixed data, these costs are uneven across domains: in Figure 1 (right), SFT and DFT improve some domains but fall below Base on others. Existing variants reweight or select training signals (Xia et al., 2024; Lin et al., 2024; Chen et al., 2025; Wu et al., 2026; Wang et al., 2026a) or anchor the student to its initialization (Li et al., 2018; Zhu et al., 2025; Wang et al., 2026b), but they typically tune learning and preservation separately (Section 2.3). We propose soft-target fine-tuning (SoFT), which replaces the one-hot target with a soft target (Figure 1, left): the distribution closest to Base in KL that gives the demonstrated token at least a floor probability. This design has three advantages. First, it changes Base minimally: the target adds only the probability mass the floor requires and keeps Base’s relative preferences among other tokens, so plausible continuations are not needlessly suppressed. Second, it couples learning and preservation: the target splits into complementary demonstration and Base-preservation weights, so supervision concentrates on tokens that Base poorly supports. Third, it is simple to control. A single gradient budget, defined as the fraction of SFT’s initial gradient on demonstrated tokens that SoFT retains, sets the learning strength, and a per-trajectory floor adapts it to the student’s initial probabilities. Training needs only offline demonstrations, with no teacher logits or online rollouts. Our contributions are threefold: 1. Mixed-data SFT hides an acquisition–retention trade-off. When one student learns from multiple teachers and domains, gains on some task families often come with sharp losses on others. At 7B, four of six fine-tuning baselines fall below Base in aggregate score, and at 8B, SFT gains in-distribution but loses out-of-distribution. 2. SoFT turns the training target itself into the control for this trade-off. We derive a closed-form, minimum-KL soft target that decomposes exactly into complementary learning and preservation weights, and calibrate it with a per-domain gradient budget. SoFT needs no teacher logits or online rollouts. 3. SoFT gains more while drifting less. SoFT achieves the best aggregate score in all three settings and improves 22 of 24 benchmark scores over Base, e.g., raising τ2τ^2 OOD from 20.67 to 48.33. Its policy drift is only about one tenth of SFT’s, and our analysis shows that stronger imitation of demonstrated tokens alone does not explain these gains. 2 Preliminary 2.1 Supervised Fine-Tuning Let =(x,y⋆)D=\(x,y )\ denote a corpus of expert demonstrations, where y⋆y is the complete reference response to query x. SFT minimizes the negative log-likelihood of the demonstrated response: ℒSFT(θ)=(x,y⋆)∼[−logπθ(y⋆∣x)].L_SFT(θ)=E_(x,y ) \! [- _θ(y x) ]. (1) 2.2 Applications of SFT SFT remains a central stage in recent model training pipelines and transfers both reasoning and agentic capabilities from verified demonstrations. For reasoning, supervised rationales expose intermediate computations and decomposition strategies in addition to final answers (Ho et al., 2023; Magister et al., 2023; Hsieh et al., 2023; Shridhar et al., 2023; Yue et al., 2024). Recent recipes such as s1, OpenThoughts, and skill-aware distillation improve this transfer through data curation, scaling, and student-specific selection (Muennighoff et al., 2025; Guha et al., 2025; Zhang et al., 2026). DeepSeek-R1 and Qwen3 further demonstrate reasoning transfer from stronger models to smaller students (DeepSeek-AI, 2025; Yang and others, 2025). For agentic capabilities, trajectories supervise tool use, environment interaction, recovery, and task completion. Kimi K2.5, Kimi K3, Qwen3-Coder-Next, DeepSeek V4.1, OpenThoughts-Agent, and ProCUA-SFT apply this approach to long-horizon and executable tasks (Kimi Team, 2026a; Kimi Team, 2026b; Cao et al., 2026; DeepSeek-AI, 2026; Raoof and others, 2026; Jung et al., 2026). Since trajectory SFT uses a fixed corpus and requires only teacher outputs, it supports black-box distillation and reuses each verified trajectory across many updates (Kim and Rush, 2016; Guha et al., 2025). These properties make SFT a scalable first stage for acquiring new model capabilities. 2.3 SFT Improvement Many SFT improvements on a fixed demonstration corpus can be written in the general form ℒ(θ)=(x,y⋆)∼[∑t(−atlogπθ(yt⋆∣st)+λtΩt(πθ(⋅∣st),π0(⋅∣st)))],L(θ)=E_(x,y ) \! [ _t (-a_t _θ(y_t s_t)+ _t _t\! ( _θ(· s_t), _0(· s_t) ) ) ], (2) where st=(x,y<t⋆)s_t=(x,y_<t ) is the teacher-forced state, ata_t controls how strongly the demonstrated token is learned, and λtΩt _t _t controls the deviation of the updated policy πθ _θ from the Base policy π0 _0. Standard SFT is recovered by setting at=1a_t=1 and λt=0 _t=0. The first line of work adjusts ata_t to reallocate learning strength across examples or tokens. LESS, Rho-1, and PriFT-mass select training signals, while Confidence and DFT assign model-dependent weights (Xia et al., 2024; Lin et al., 2024; Wang et al., 2026a; Wu et al., 2026). These methods change how strongly each demonstration token is learned, but usually retain the one-hot target on yt⋆y_t . The second line adjusts λt _t or Ωt _t to limit how far the student moves from the Base model. ASFT adds a KL anchor to DFT (Zhu et al., 2025), while Anchored Learning controls distributional drift through an intermediate target (Wang et al., 2026b). L2-SP is the parameter-space analogue, replacing the policy discrepancy above with an anchor between θ and its initialization θ0 _0 (Li et al., 2018). These methods preserve prior behavior through a regularizer whose strength is usually specified separately from the SFT signal. SoFT couples these two lines. Its adaptive target is equivalent to jointly setting the demonstration coefficient ata_t and the Base-preservation coefficient λt _t, so that a larger value of one leaves a smaller value for the other. This coupling controls acquisition and preservation through one calibrated target rather than two independently tuned objectives. Sections 3.2 and 3.4 define this target and its calibration. 3 Method We introduce soft-target fine-tuning (SoFT), an offline objective that couples learning from demonstrations with preserving the Base policy. From the demonstrated sequence and the Base distribution alone, SoFT constructs one adaptive training target for each token, without online rollouts or teacher logits. 3.1 From a Global Budget to Token Targets Figure 2 gives an overview, from left to right. A global gradient budget R⋆R (left) sets SoFT’s initial gradient on demonstrated tokens relative to SFT. For each sequence, this budget is converted into a probability floor τx _x (middle). At each teacher-forced state st=(x,y<t⋆)s_t=(x,y_<t ), each position whose Base probability π0(yt⋆∣st) _0(y_t s_t) falls below τx _x is lifted to the floor, which determines a demonstration weight ata_t; for example, with τx=0.7 _x=0.7, a token with Base probability 0.10.1 receives at=2/3a_t=2/3. The weight then defines a token-level soft target (right) that, unlike the one-hot SFT target, keeps part of Base’s probability on alternative tokens. Thus, R⋆R is selected globally, τx _x adapts to each sequence, and the target adapts to each token. Sections 3.2–3.4 formalize these three steps in reverse order: the target, its decomposition into weights, and the calibration of τx _x. Figure 2: From a shared gradient budget (left), to a sequence-specific floor (middle), to token-level SoFT targets (right). Values are illustrative. 3.2 Soft Target Construction At each teacher-forced state sts_t, SoFT replaces the one-hot target with the distribution qtq_t closest to the Base policy π0 _0 that assigns at least probability τx _x to the demonstrated token yt⋆y_t (we leave its dependence on τx _x implicit): qt=argminq∈Δ()KL(q∥π0(⋅∣st))s.t.q(yt⋆)≥τx.q_t= _q∈ (V)KL\! (q\,\|\, _0(· s_t) ) .t. q(y_t )≥ _x. (3) The KL term discourages unnecessary changes to the Base distribution, and the constraint enforces the probability floor. Solving the Lagrangian gives the closed form qt(yt⋆)=maxπ0(yt⋆∣st),τx,qt(v)=π0(v∣st)1−qt(yt⋆)1−π0(yt⋆∣st),v≠yt⋆.q_t(y_t )= \! \ _0(y_t s_t), _x \, q_t(v)= _0(v s_t) 1-q_t(y_t )1- _0(y_t s_t), v≠ y_t . (4) The solution adds only the probability mass required by the floor and rescales all other tokens proportionally, so Base’s relative preferences among them are unchanged. If π0(yt⋆∣st)≥τx _0(y_t s_t)≥ _x, the constraint is inactive and qt=π0(⋅∣st)q_t= _0(· s_t); as τx→1 _x→ 1, the target approaches the one-hot SFT target. Appendix A.1 gives the full derivation. 3.3 Soft Target Decomposition We now decompose the soft target to connect it with the learning and preservation terms in Eq. 2. Letting δyt⋆ _y_t denote the one-hot distribution on yt⋆y_t , Eq. 4 can be written as at=maxτx−π0(yt⋆∣st), 01−π0(yt⋆∣st),λt=1−at,qt=atδyt⋆+λtπ0(⋅∣st).a_t= \ _x- _0(y_t s_t),\,0\1- _0(y_t s_t), _t=1-a_t, q_t=a_t _y_t + _t _0(· s_t). (5) Here, ata_t weights learning from the demonstrated token and λt _t weights preservation of the Base distribution, and the two are complementary by construction. Tokens that Base already predicts above the floor receive at=0a_t=0 and keep their Base target; all other tokens receive exactly the extra probability needed to reach τx _x. Substituting Eq. 5 into the cross-entropy gives ℒSoFT(θ)=(x,y⋆)∼[∑t(−atlogπθ(yt⋆∣st)+λtCE(π0(⋅∣st),πθ(⋅∣st)))].L_SoFT(θ)=E_(x,y ) \! [ _t (-a_t _θ(y_t s_t)+ _tCE\! ( _0(· s_t), _θ(· s_t) ) ) ]. (6) Since this cross-entropy equals KL(π0∥πθ)KL( _0\| _θ) up to a θ-independent constant, SoFT instantiates Eq. 2 with Ωt _t as the forward KL, through a single soft target whose learning and preservation weights are coupled rather than tuned independently. 3.4 Adaptive Control with R⋆R The floor τx _x determines both weights in Eq. 5. Since the Base probabilities π0(yt⋆∣st) _0(y_t s_t) vary across demonstrations, a floor shared by all sequences would produce different effective learning strengths. We therefore solve for a sequence-specific floor that meets a shared gradient budget R⋆R . At the Base initialization, the gradient magnitude on the gold logit (the logit of yt⋆y_t ) is 1−π0(yt⋆∣st)1- _0(y_t s_t) for SFT and at(1−π0(yt⋆∣st))a_t(1- _0(y_t s_t)) for SoFT. We choose τx _x such that Rx(τx)=∑tat(1−π0(yt⋆∣st))∑t(1−π0(yt⋆∣st))=R⋆.R_x( _x)= _ta_t(1- _0(y_t s_t)) _t(1- _0(y_t s_t))=R . (7) Each numerator term equals qt(yt⋆)−π0(yt⋆∣st)q_t(y_t )- _0(y_t s_t), so R⋆R is the fraction of Base’s gap to the one-hot target that the soft target closes (e.g., R⋆=0.3R =0.3 closes 30% of it); equivalently, it is the fraction of SFT’s initial gold-logit gradient that SoFT retains. As R⋆R moves from zero to one, SoFT moves from preserving the Base distribution toward one-hot SFT. Since Rx(τ)R_x(τ) is continuous and monotone in τ, τx _x is obtained by scalar bisection (Appendix A.4). Because qtq_t depends only on π0 _0, it stays fixed during training and can be precomputed; Appendix B.2.1 gives a memory-efficient top-K approximation. 4 Experimental Setup We evaluate whether SoFT can learn from mixed training data while preserving generalization beyond the training tasks. We compare SoFT with Base, standard SFT (Ouyang et al., 2022), Confidence (Chen et al., 2025), Rho-1 (Lin et al., 2024), DFT (Wu et al., 2026), PriFT-mass (Wang et al., 2026a), and L2-SP (Li et al., 2018) on reasoning and agentic tasks. Within each setting, all fine-tuned methods share the initialization, data, and training budget, and hyperparameters are selected on held-out validation data. 4.1 Domain-Aware R⋆R For SoFT, we select one gradient budget Rd⋆R_d per domain using held-out validation data. Each sequence x then uses the budget of its domain d(x)d(x): Rx(τx)=Rd(x)⋆,R_x( _x)=R_d(x) , (8) so that Rd⋆R_d sets the domain-level balance and τx _x adapts it to each sequence. For General, Math, Science, and Code, Qwen2.5-1.5B-Instruct uses Rd⋆=(0.15,0.30,0.45,0.60)R_d =(0.15,0.30,0.45,0.60) and Qwen2.5-7B uses (0.25,0.35,0.45,0.55)(0.25,0.35,0.45,0.55). Qwen3-8B uses (0.3,0.8,0.3,0.3)(0.3,0.8,0.3,0.3) for Customer, Coding, Search, and Tool Interaction. 4.2 Tasks and Evaluation For reasoning, we fine-tune Qwen2.5-1.5B-Instruct and Qwen2.5-7B (Yang and others, 2024) on 36,000 teacher trajectories, equally divided among Math, Code, Science, and General, with 800 validation examples. For agentic tasks, we fine-tune Qwen3-8B (Yang and others, 2025) on 8,252 trajectories across Customer, Coding, Search, and Tool Interaction, with 824 validation examples. Teachers include DeepSeek-R1, GLM-5.1, Kimi-K2.5, QwQ-32B/STILL-2, Qwen3-235B-A22B, GPT-OSS-20B, DeepSeek V4 Flash, and Qwen3.6-27B (Appendix C). Within each capability domain, we use in-distribution (ID) for training-related evaluation and out-of-distribution (OOD) for cross-benchmark evaluation (Table 1). Reasoning evaluations use OpenR1-Math (Lozhkov et al., 2025), APPS (Hendrycks et al., 2021), RiddleSense (Lin et al., 2021), GSM8K (Cobbe et al., 2021), ARC-Challenge (Clark et al., 2018), BoolQ (Clark et al., 2019), and MBPP with EvalPlus tests (Austin et al., 2021; Liu et al., 2023). Science MCQ is our filtered subset of Mixture-of-Thoughts (Hugging Face, 2025a), not a separate official benchmark. Agentic evaluations use τ2τ^2 (Barres et al., 2025), LiveCodeBench (Jain et al., 2024), TMax (Ivison et al., 2026), BigCodeBench (Zhuo et al., 2024), BFCL V4 (Patil et al., 2025), and FRAMES (Krishna et al., 2024). For τ2τ^2, training and the 60 ID tasks use the retail and airline domains, while the 300 OOD tasks come from telecom and banking knowledge. Search ID uses held-out Nemotron search prompts (NVIDIA, 2025). Table 1: Evaluation suites and case counts before repeated attempts. BigCodeBench counts evaluation cases. Setting Domain ID (cases) OOD (cases) Reasoning Math OpenR1-Math (300) GSM8K (1,319) Code APPS (250) MBPP+ (378) Science Science MCQ (1,000) ARC-Challenge (608) General RiddleSense (1,018) BoolQ (708) Agentic Customer τ2τ^2 (60) τ2τ^2 (300) Coding LiveCodeBench (694) BigCodeBench (2,280) Search Search (500) FRAMES (824) Tool Interaction TMax (300) BFCL V4 (5,106) All models use chain-of-thought reasoning, and we report empirical pass@4 over the temperature–seed pairs (0,42)(0,42), (0.2,43)(0.2,43), (0.4,44)(0.4,44), and (0.6,45)(0.6,45); a task counts as solved if any attempt passes the verifier. We aggregate the benchmark scores sb∈[0,1]s_b∈[0,1] with the geometric mean GM=100(∏b=18sb)1/8GM=100 ( _b=1^8s_b )^1/8, which weights the four ID and four OOD benchmarks equally (task details in Appendix D). 5 Experimental Results Tables 2 and 3 report the results. SoFT achieves the highest GM in all three settings, while individual benchmarks reveal the distinct strengths of other methods. 5.1 Main Results on Reasoning Tasks Challenges of mixed-data training. Fine-tuning gains are uneven across reasoning tasks (Table 2). At 7B, Confidence leads on APPS and ties for the best Science score, yet its Math score falls from 52.00 to 38.67 and ARC-C from 92.11 to 75.00, leaving its GM (60.24) below that of Base (62.91). SFT likewise improves Science but loses more than nine points on both Math and MBPP+. Overall, four baselines at 7B and two at 1.5B finish below Base in GM: training on the mixture can strengthen one part of it while weakening another. SoFT under mixed-data training. SoFT improves nearly every reasoning benchmark. It raises all eight scores over Base at 1.5B and seven at 7B, matching Base on the remaining task, and its GMs of 54.17 and 65.88 are the highest at both scales. At 7B, Science rises from 82.70 to 88.30 and MBPP+ from 68.78 to 75.66. DFT remains strongest on 7B Math and Confidence on 7B APPS, but SoFT’s broader gains yield the highest aggregate score. Table 2: Reasoning performance (%) under the fixed four-attempt schedule. Bold marks column bests per model size, including ties. Math: OpenR1-Math; Riddle: RiddleSense. In-distribution (ID) Out-of-distribution (OOD) Model Method Math APPS Riddle Science GSM8K ARC-C BoolQ MBPP+ GM Qwen2.5-1.5B Base 34.00 6.80 66.50 76.50 87.34 85.03 82.91 60.85 50.76 SFT 37.00 8.00 70.73 78.30 87.11 88.49 82.77 63.76 53.46 Confidence 33.33 6.00 71.81 79.00 85.06 88.82 83.05 62.70 50.84 Rho-1 35.00 6.80 72.00 77.30 86.05 87.99 83.47 60.58 51.66 DFT 31.00 6.40 68.66 73.40 83.55 86.35 81.21 60.58 49.40 PriFT-mass 36.00 5.60 70.04 75.60 85.90 83.39 84.18 65.61 50.49 L2-SP 35.33 6.80 70.53 79.20 86.28 88.82 82.91 65.87 52.32 SoFT 37.33 8.00 72.30 79.80 87.49 86.18 85.59 66.67 54.17 Qwen2.5-7B Base 52.00 12.80 82.61 82.70 91.74 92.11 92.80 68.78 62.91 SFT 42.33 11.60 84.28 87.10 94.92 82.40 93.50 59.26 59.46 Confidence 38.67 13.20 84.87 88.30 94.62 75.00 90.82 70.37 60.24 Rho-1 24.33 10.40 76.72 85.60 89.39 79.77 83.47 30.16 48.34 DFT 58.00 11.20 82.81 82.70 93.71 93.91 90.96 71.69 63.22 PriFT-mass 55.33 12.80 83.30 83.60 94.69 95.23 92.23 74.34 64.64 L2-SP 43.00 12.80 84.09 84.80 94.24 96.05 93.64 72.49 62.77 SoFT 55.33 12.80 85.56 88.30 95.38 97.37 94.35 75.66 65.88 5.2 Results on Agentic Tasks Challenges of mixed-data training. Agentic tasks show a similar imbalance (Table 3). SFT leads on TMax and BigCodeBench, yet its τ2τ^2 ID score falls from 48.33 to 35.00 and its BFCL V4 score from 40.20 to 21.10. Confidence and L2-SP also finish below Base in GM, so coding and terminal gains can come at the cost of customer interaction and tool calling. Table 3: Agentic performance (%) under the fixed four-attempt schedule. Bold marks column bests, including ties. LCB: LiveCodeBench; BCB: BigCodeBench. In-distribution (ID) Out-of-distribution (OOD) Model Method τ2τ^2 LCB Search TMax τ2τ^2 BCB BFCL V4 FRAMES GM Qwen3-8B Base 48.33 39.05 23.80 7.33 20.67 53.51 40.20 46.12 30.11 SFT 35.00 37.32 26.20 16.00 32.00 58.20 21.10 46.00 31.58 Confidence 40.00 35.59 17.60 5.00 22.00 56.32 34.15 48.67 26.70 Rho-1 58.33 40.20 22.80 9.00 29.33 53.51 40.60 45.15 32.93 DFT 56.67 37.32 23.00 5.33 27.67 52.02 41.32 44.66 30.18 PriFT-mass 56.67 39.48 20.40 7.67 26.00 52.89 40.02 43.93 30.97 L2-SP 26.67 37.90 18.20 7.67 29.00 55.57 33.92 48.54 27.97 SoFT 65.00 38.04 27.20 9.33 48.33 56.52 41.36 48.79 36.93 SoFT under mixed-data training. SoFT improves seven of eight agentic scores over Base and achieves the highest GM, 36.93. Its largest gains come from τ2τ^2: ID rises from 48.33 to 65.00 and OOD from 20.67 to 48.33. The only regression is a slight dip on LiveCodeBench, from 39.05 to 38.04, while SFT still leads on TMax and BigCodeBench. 5.3 Shared Patterns across Task Families Across task families, the key difference is not whether a method achieves a high score on one benchmark, but whether it does so without large losses elsewhere. For example, 8B SFT raises the four-benchmark ID geometric mean from 23.95 to 27.20 but lowers the OOD mean from 37.84 to 36.67. SoFT instead raises both means in every setting: to 28.14 and 48.45 at 8B, from 32.93 to 36.23 and 78.24 to 80.99 at 1.5B, and from 46.18 to 48.10 and 85.70 to 90.23 at 7B. Overall, it improves 22 of 24 scores over Base, matches one, and falls slightly below on one. 6 Exploratory Analysis We examine three consequences of SoFT’s coupled objective (Eq. 6). RQ1 relates benchmark gains to policy drift from Base. RQ2 measures how much probability the trained model assigns to demonstrated tokens. RQ3 examines how R⋆R distributes target changes across domains. 6.1 RQ1: Performance Gain versus Policy Drift Figure 3: Benchmark GM gain versus policy drift relative to SFT. The pale rounded region highlights the three SoFT points. Colors denote methods, shapes denote settings, and the horizontal axis uses a symmetric log scale near zero. SoFT gains more with less policy drift. Figure 3 compares the GM gain of each trained model with its policy drift from Base. We measure drift as KL(π0∥πθ)KL( _0\| _θ) on held-out teacher-forced prefixes and divide by SFT’s KL within the same setting, so SFT lies at one on the horizontal axis. SoFT occupies the favorable upper-left region at all three scales. Its relative drift is about one tenth of SFT’s, and its GM gain is the largest in each setting, peaking at 8B. Output similarity and task success capture different forms of preservation. On reasoning tasks, SoFT’s output-token frequencies stay closer to Base than SFT’s at both scales (mean Jensen–Shannon divergence: 0.044 versus 0.090 at 1.5B; 0.065 versus 0.169 at 7B). SoFT’s response length is also closer to Base than SFT’s on 15 of 16 benchmarks (Figure 4). L2-SP offers a useful counterexample at 7B: its token frequencies are closer to Base, yet it retains fewer Base successes. Lexical similarity and task success thus describe different aspects of preservation. Appendix F reports the absolute KL values and a paired retention analysis. Figure 4: First-attempt output lengths for all eight methods across eight benchmark means per setting. Boxes: Q1–Q3; center lines: medians; diamonds: means. Agentic lengths cover scorer-facing outputs, not total rollout cost. 6.2 RQ2: Realized Demonstration Learning More demonstration uptake does not guarantee better performance. We measure how training changes the probability of demonstrated tokens at shared teacher-forced states. We define Aθ=∑t[πθ(yt⋆∣st)−π0(yt⋆∣st)]/∑t[1−π0(yt⋆∣st)]A_θ= _t[ _θ(y_t s_t)- _0(y_t s_t)]/ _t[1- _0(y_t s_t)]. It measures the share of Base’s remaining gold-token probability (Base headroom) acquired by the trained model; on SoFT’s constructed target, it equals Rd(x)⋆R_d(x) . Figure 5 computes AθA_θ on the first 128 tokens of 32 validation trajectories per reasoning domain. At 1.5B, SoFT has the smallest AθA_θ (7.2%) and the highest GM (54.17), while DFT has the largest AθA_θ (41.2%) and a lower GM (49.40). At 7B, PriFT-mass, Confidence, and Rho-1 have acquisition ratios near 36%, while their GMs span 48.34 to 64.64. A larger AθA_θ thus does not imply better task performance, and SoFT’s high GM with a small AθA_θ is consistent with preserving Base-supported alternatives. Appendix I gives per-domain results. Figure 5: Realized acquisition AθA_θ versus benchmark GM for seven methods at each reasoning scale. Horizontal bars show exploratory resampling ranges; dashed lines mark Base GM. The validation subset was used during hyperparameter selection. 6.3 RQ3: Effect of the Gradient Budget A shared budget produces comparable relative adjustments. We vary R⋆R and measure KL(qt∥π0)KL(q_t\| _0) for the resulting targets. In Figure 6, Science requires a larger absolute change than Math at R⋆=0.5R =0.5 on 1.5B (0.739 versus 0.363 nats per token). The lower panels normalize each trajectory’s KL by its value at the one-hot endpoint. At the same budget, the domain averages cluster within 41.6–42.6% at 1.5B and 48.4–49.9% across the four agentic groups at 8B. Domains with different absolute target changes thus use similar fractions of their available change, making R⋆R an interpretable control across the mixture. The same budget activates different fractions of tokens. At R⋆=0.3R =0.3, positive demonstration weights apply to 16.9% of Search tokens and 19.8% of Customer tokens. Even within Tool Interaction, the Terminal and Tool Calling sources differ more sharply (9.3% versus 25.5%). Tokens that Base already predicts above the sequence floor keep their Base target, while tokens with lower Base probability receive more weight. As the budget rises, the solved floor activates additional positions rather than raising every weight equally, letting a domain budget respond to its Base confidence profile (Appendices E and G). Figure 6: Target KL from Base as the gradient budget varies. Top: absolute KL of the constructed target; bottom: KL relative to each trajectory’s one-hot endpoint. Stars mark selected budgets. Selected budgets express domain priorities. The coding domain receives the largest budget in every setting (stars in Figure 6); validation sets each domain’s learning strength, and the sequence-specific floor adapts it to Base confidence. 7 Conclusion We studied multi-teacher, multi-domain SFT, where a student must learn diverse behaviors without losing generalization. SoFT uses calibrated soft targets to couple learning from demonstrations with preservation of the Base distribution. It has the best aggregate score across reasoning and agentic tasks in all three settings, although other methods lead on some benchmarks. The analysis also shows that stronger imitation of demonstrated tokens alone does not track the best aggregate task performance. These results suggest that controlling learning and preservation together can improve mixed-data fine-tuning without teacher logits or online rollouts. Future work will test whether this helps subsequent reinforcement learning. The broader lesson is to calibrate new supervision against existing knowledge. AI use statement Language models were used as objects of study and to generate teacher trajectories for the experiments described in this paper. Beyond these experimental uses, generative AI tools were used only to assist with implementing parts of the experimental code. They were not used to draft, edit, or revise the manuscript, nor to generate the text of tables or figures. The authors reviewed all experimental code, including AI-assisted implementations, to check its correctness and consistency with the methods described in the paper. All analyses, interpretations of the experimental results, and scientific claims were developed and assessed by the authors. The authors take full responsibility for the experimental methodology, code, results, claims, and final manuscript. Ethics statement This work studies how language models learn from and retain capabilities after distillation. The experiments use published datasets and benchmark environments, together with generated teacher trajectories; they do not involve direct interaction with human participants. Model outputs and published trajectories may nonetheless reflect errors, biases, or sensitive content present in their sources. We report benchmark-specific limitations and avoid treating aggregate scores as evidence that every capability improves. The authors are responsible for respecting the licenses and usage conditions of the datasets and models used in this work. Reproducibility statement The main text specifies the training objectives, model families, hyperparameter-selection procedure, and evaluation protocols. Appendix C documents the training-data sources, teacher provenance, split sizes, token statistics, and examples. We use fixed data splits and seeds, shared evaluation tasks within each model-size comparison, and fixed scoring denominators. We distinguish the four-attempt scores, versioned answer-extraction procedures, and scale-specific finalization settings. Upon acceptance, we will publicly release the training and evaluation code, configurations, data manifests, and processed datasets that we are permitted to redistribute. For third-party data that cannot be redistributed, we will provide source links and reconstruction scripts. References Ahmad et al. (2025) W. U. Ahmad, S. Narenthiran, S. Majumdar, A. Ficek, S. Jain, J. Huang, V. Noroozi, and B. Ginsburg OpenCodeReasoning: advancing data distillation for competitive coding. arXiv preprint arXiv:2504.01943. External Links: Link Cited by: §C.1. Austin et al. (2021) J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton Program synthesis with large language models. arXiv preprint arXiv:2108.07732. External Links: Link Cited by: §4.2. Barres et al. (2025) V. Barres, H. Dong, S. Ray, X. Si, and K. Narasimhan τ2τ^2-Bench: evaluating conversational agents in a dual-control environment. arXiv preprint arXiv:2506.07982. External Links: Link Cited by: §C.2, §4.2. Bercovich et al. (2025) A. Bercovich, I. Levy, I. Golan, M. Dabbah, R. El-Yaniv, et al. Llama-Nemotron: efficient reasoning models. arXiv preprint arXiv:2505.00949. External Links: Link Cited by: §C.1. Cao et al. (2026) R. Cao, M. Chen, J. Chen, Z. Cui, Y. Feng, B. Hui, Y. Jing, K. Li, M. Li, J. Lin, Z. Ma, K. Shum, X. Wang, J. Wei, J. Yang, J. Zhang, L. Zhang, Z. Zhang, W. Zhao, and F. Zhou Qwen3-Coder-Next technical report. arXiv preprint arXiv:2603.00729. External Links: Link Cited by: §2.2. Chen et al. (2025) F. Chen, A. Raventos, N. Cheng, S. Ganguli, and S. Druckmann Rethinking fine-tuning when scaling test-time compute: limiting confidence improves mathematical reasoning. arXiv preprint arXiv:2502.07154. External Links: Link Cited by: §1, §4. Clark et al. (2019) C. Clark, K. Lee, M. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova BoolQ: exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044. External Links: Link Cited by: §4.2. Clark et al. (2018) P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord Think you have solved question answering? try ARC, the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457. External Links: Link Cited by: §4.2. Cobbe et al. (2021) K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. External Links: Link Cited by: §4.2. DeepSeek-AI (2025) DeepSeek-AI DeepSeek-R1: incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.12948. Cited by: §1, §2.2. DeepSeek-AI (2026) DeepSeek-AI DeepSeek-V4.1-Flash: pushing the limits of KV cache compression. Technical report. External Links: Link Cited by: §2.2. Guha et al. (2025) E. Guha, R. Marten, S. Keh, N. Raoof, G. Smyrnis, H. Bansal, M. Nezhurina, J. Mercat, T. Vu, Z. Sprague, et al. OpenThoughts: data recipes for reasoning models. arXiv preprint arXiv:2506.04178. External Links: Link Cited by: §C.1, §1, §2.2. Hendrycks et al. (2021) D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. Song, and J. Steinhardt Measuring coding challenge competence with APPS. arXiv preprint arXiv:2105.09938. External Links: Link Cited by: §4.2. Ho et al. (2023) N. Ho, L. Schmid, and S. Yun Large language models are reasoning teachers. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, p. 14852–14882. Cited by: §2.2. Hsieh et al. (2023) C. Hsieh, C. Li, C. Yeh, H. Nakhost, Y. Fujii, A. Ratner, R. Krishna, C. Lee, and T. Pfister Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. In Findings of the Association for Computational Linguistics: ACL 2023, p. 8003–8017. Cited by: §2.2. Hugging Face (2025a) Hugging Face Mixture-of-Thoughts. Note: Hugging Face dataset External Links: Link Cited by: §C.1, §4.2. Hugging Face (2025b) Hugging Face Open R1: a fully open reproduction of DeepSeek-R1. External Links: Link Cited by: §C.1. iAmBoosted (2026) iAmBoosted GPT-OSS-20B reasoning traces. Note: Hugging Face dataset External Links: Link Cited by: §C.1. ianncity (2026) ianncity KIMI-K2.5-1000000x. Note: Hugging Face dataset External Links: Link Cited by: §C.1. Ivison et al. (2026) H. Ivison, J. O. Yin, R. Shao, T. Xiao, N. Lambert, and H. Hajishirzi Tmax: a simple recipe for terminal agents. arXiv preprint arXiv:2606.23321. External Links: Link Cited by: §C.2, §4.2. Jackrong (2026a) Jackrong GLM-5.1-Reasoning-1M-Cleaned. Note: Hugging Face dataset External Links: Link Cited by: §C.1. Jackrong (2026b) Jackrong Kimi-K2.5-Reasoning-1M-Cleaned. Note: Hugging Face dataset External Links: Link Cited by: §C.1. Jain et al. (2024) N. Jain, K. Han, A. Gu, W. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica LiveCodeBench: holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974. External Links: Link Cited by: §C.2, §4.2. Jung et al. (2026) J. Jung, X. Lu, B. Cui, M. Khalifa, S. Zhang, H. Zhang, J. Xu, A. S. Deshmukh, K. Sapra, A. Tao, Y. Choi, J. Kautz, M. Liu, and Y. Dong ProCUA-SFT technical report. arXiv preprint arXiv:2606.17321. External Links: Link Cited by: §1, §1, §2.2. Kassadin88 (2026) Kassadin88 GLM-5.1-1000000x: one million reasoning traces distilled from GLM-5.1. Note: Hugging Face datasetOriginal citation name retained; repository now redirects to clzoro External Links: Link Cited by: §C.1. Kim et al. (2025) J. Kim, K. Seo, and D. Lee In their own words: reasoning traces tailored for small models make them better reasoners. arXiv preprint arXiv:2509.22230. External Links: Link Cited by: §1. Kim and Rush (2016) Y. Kim and A. M. Rush Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Cited by: §2.2. Kimi Team (2026a) Kimi Team Kimi K2.5: visual agentic intelligence. arXiv preprint arXiv:2602.02276. External Links: Link Cited by: §2.2. Kimi Team (2026b) Kimi Team Kimi K3: open frontier intelligence. arXiv preprint arXiv:2607.24653. External Links: Link Cited by: §2.2. Krishna et al. (2024) S. Krishna, K. Krishna, A. Mohananey, S. Schwarcz, A. Stambler, S. Upadhyay, and M. Faruqui Fact, fetch, and reason: a unified evaluation of retrieval-augmented generation. arXiv preprint arXiv:2409.12941. External Links: Link Cited by: §4.2. Lambert et al. (2024) N. Lambert, J. Morrison, V. Pyatkin, S. Huang, H. Ivison, F. Brahman, et al. Tulu 3: pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124. Cited by: §1. Li et al. (2018) X. Li, Y. Grandvalet, and F. Davoine Explicit inductive bias for transfer learning with convolutional networks. In Proceedings of the 35th International Conference on Machine Learning, Cited by: §1, §2.3, §4. Li et al. (2025) Z. Li, C. Chen, T. Xu, Z. Qin, J. Xiao, Z. Luo, and R. Sun Preserving diversity in supervised fine-tuning of large language models. In International Conference on Learning Representations, Cited by: §1, §1. Lin et al. (2021) B. Y. Lin, Z. Wu, Y. Yang, D. Lee, and X. Ren RiddleSense: reasoning about riddle questions featuring linguistic creativity and commonsense knowledge. arXiv preprint arXiv:2101.00376. External Links: Link Cited by: §4.2. Lin et al. (2024) Z. Lin, Z. Gou, Y. Gong, X. Liu, Y. Shen, R. Xu, C. Lin, Y. Yang, J. Jiao, N. Duan, and W. Chen RHO-1: not all tokens are what you need. In Advances in Neural Information Processing Systems, Cited by: §1, §2.3, §4. Liu et al. (2023) J. Liu, C. S. Xia, Y. Wang, and L. Zhang Is your code generated by ChatGPT really correct? rigorous evaluation of large language models for code generation. arXiv preprint arXiv:2305.01210. External Links: Link Cited by: §4.2. Lozhkov et al. (2025) A. Lozhkov, H. Kydlíček, L. Ben Allal, G. Penedo, E. Beeching, Q. Gallouédec, N. Habib, L. Tunstall, and L. von Werra OpenR1-Math-220k. Note: Hugging Face dataset External Links: Link Cited by: §C.1, §4.2. Magister et al. (2023) L. C. Magister, J. Mallinson, J. Adamek, E. Malmi, and A. Severyn Teaching small language models to reason. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, p. 1773–1781. Cited by: §2.2. Marin Community (2026) Marin Community OpenThoughts4 11K Math - Qwen3-235B-A22B Agreed Answers. Note: Hugging Face dataset External Links: Link Cited by: §C.1. Muennighoff et al. (2025) N. Muennighoff, Z. Yang, W. Shi, X. L. Li, L. Fei-Fei, H. Hajishirzi, L. Zettlemoyer, P. Liang, E. Candès, and T. Hashimoto s1: simple test-time scaling. arXiv preprint arXiv:2501.19393. External Links: Link Cited by: §1, §2.2. NovaSky (2025) NovaSky Sky-T1_data_17k. Note: Hugging Face dataset External Links: Link Cited by: §C.1. NVIDIA (2025) NVIDIA Nemotron-SFT-Agentic-v2. Note: Hugging Face datasetCreation year follows the dataset card; the Hugging Face repository was created in March 2026 External Links: Link Cited by: §C.2, §4.2. OpenAI (2025) OpenAI gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925. External Links: Link Cited by: §C.1. Ouyang et al. (2022) L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, Cited by: §1, §4. Patil et al. (2025) S. G. Patil, H. Mao, F. Yan, C. C. Ji, V. Suresh, I. Stoica, and J. E. Gonzalez The berkeley function calling leaderboard (BFCL): from tool use to agentic evaluation of large language models. In Proceedings of the 42nd International Conference on Machine Learning, Vol. 267, p. 48371–48392. External Links: Link Cited by: §4.2. Penedo et al. (2025) G. Penedo, A. Lozhkov, H. Kydlíček, L. Ben Allal, E. Beeching, A. Piqueres Lajarín, Q. Gallouédec, N. Habib, L. Tunstall, and L. von Werra CodeForces CoTs. Note: Hugging Face dataset External Links: Link Cited by: §C.1. Raoof et al. (2026) N. Raoof et al. Data recipes for agentic models. arXiv preprint arXiv:2606.24855. Cited by: §C.2, §1, §1, §2.2. Shridhar et al. (2023) K. Shridhar, A. Stolfo, and M. Sachan Distilling reasoning capabilities into smaller language models. In Findings of the Association for Computational Linguistics: ACL 2023, p. 7059–7073. Cited by: §2.2. Twist et al. (2026) L. Twist, H. Yannakoudakis, and J. M. Zhang Reasoning-trace collapse: evaluating the loss of explicit reasoning during fine-tuning. arXiv preprint arXiv:2605.21127. External Links: Link Cited by: §1. Wan et al. (2024) F. Wan, X. Huang, D. Cai, X. Quan, W. Bi, and S. Shi Knowledge fusion of large language models. In International Conference on Learning Representations, Cited by: §1. Wang et al. (2026a) K. Wang, S. Li, M. Salzmann, and P. Frossard PriFT: prior-support guided supervised fine-tuning. arXiv preprint arXiv:2606.09396. Cited by: §1, §2.3, §4. Wang et al. (2026b) X. Wang, C. Sun, Y. Wu, and X. Wang Stabilizing LLM supervised fine-tuning via explicit distributional control. arXiv preprint arXiv:2605.04468. External Links: Link Cited by: §1, §1, §2.3. Wu et al. (2026) Y. Wu, Y. Zhou, Z. Zhou, Y. Peng, X. Ye, X. Hu, W. Zhu, L. Qi, M. Yang, and X. Yang On the generalization of SFT: a reinforcement learning perspective with reward rectification. In International Conference on Learning Representations, Cited by: §A.3, §1, §2.3, §4. Xia et al. (2024) M. Xia, S. Malladi, S. Gururangan, S. Arora, and D. Chen LESS: selecting influential data for targeted instruction tuning. In Proceedings of the 41st International Conference on Machine Learning, p. 54104–54132. Cited by: §1, §2.3. Yang et al. (2024) A. Yang et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. External Links: Link Cited by: §4.2. Yang et al. (2025) A. Yang et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388. External Links: Link Cited by: §2.2, §4.2. Yue et al. (2024) X. Yue, X. Qu, G. Zhang, Y. Fu, W. Huang, H. Sun, Y. Su, and W. Chen MAmmoTH: building math generalist models through hybrid instruction tuning. In International Conference on Learning Representations, Cited by: §2.2. Zhang et al. (2026) L. Zhang, Y. Zhang, W. Hu, and L. Wang Skill-aware data selection and fine-tuning for data-efficient reasoning distillation. arXiv preprint arXiv:2601.10109. External Links: Link Cited by: §2.2. Zhu et al. (2025) H. Zhu, J. Su, P. Lai, R. Ma, W. Zhang, L. Yang, and G. Chen Anchored supervised fine-tuning. arXiv preprint arXiv:2509.23753. External Links: Link Cited by: §1, §2.3. Zhuo et al. (2024) T. Y. Zhuo, M. C. Vu, J. Chim, H. Hu, W. Yu, R. Widyasari, I. N. B. Yusuf, H. Zhan, J. He, I. Paul, S. Brunner, C. Gong, T. Hoang, A. R. Zebaze, X. Hong, W. Li, J. Kaddour, M. Xu, Z. Zhang, P. Yadav, N. Jain, A. Gu, Z. Cheng, J. Liu, Q. Liu, Z. Wang, B. Hui, N. Muennighoff, D. Lo, D. Fried, X. Du, H. de Vries, and L. Von Werra BigCodeBench: benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15877. External Links: Link Cited by: §4.2. Appendix A Theoretical Analysis of SoFT This section derives the SoFT target as a constrained KL projection, connects it to token-weighted SFT and Base-policy regularization, distinguishes it from DFT, and interprets the gradient budget R⋆R . Fix the teacher-forced state st=(x,y<t⋆)s_t=(x,y_<t ) used in Section 3.2, and let yt⋆y_t be its demonstrated next token. We retain the main-text notation qtq_t, ata_t, and λt=1−at _t=1-a_t throughout. A.1 Constrained KL projection SoFT sets a probability floor τx _x for the demonstrated token yt⋆y_t . Among all targets that satisfy this floor, it selects the one closest to Base: qt=argminq∈Δ()KL(q∥π0(⋅∣st))s.t.q(yt⋆)≥τx.q_t= _q∈ (V)KL\! (q\,\|\, _0(· s_t) ) .t. q(y_t )≥ _x. (9) When π0(yt⋆∣st)≥τx _0(y_t s_t)≥ _x, Base already satisfies the constraint and gives qt=π0(⋅∣st)q_t= _0(· s_t). When π0(yt⋆∣st)<τx _0(y_t s_t)< _x, the target assigns τx _x to yt⋆y_t . The Lagrangian determines the remaining mass: (q,ηt)=∑v≠yt⋆q(v)logq(v)π0(v∣st)+ηt(∑v≠yt⋆q(v)−(1−τx)).J(q, _t)= _v≠ y_t q(v) q(v) _0(v s_t)+ _t ( _v≠ y_t q(v)-(1- _x) ). (10) Stationarity gives logq(v)π0(v∣st)+1+ηt=0,v≠yt⋆, q(v) _0(v s_t)+1+ _t=0, v≠ y_t , (11) so all non-demonstrated probabilities must be rescaled by the same factor. Normalization then yields qt(yt⋆)=maxπ0(yt⋆∣st),τx,qt(v)=π0(v∣st)1−qt(yt⋆)1−π0(yt⋆∣st),v≠yt⋆. gatheredq_t(y_t )= \ _0(y_t s_t), _x\,\\[3.0pt] q_t(v)= _0(v s_t) 1-q_t(y_t )1- _0(y_t s_t), v≠ y_t . gathered (12) For any u,v≠yt⋆u,v≠ y_t , the solution preserves qt(u)/qt(v)=π0(u∣st)/π0(v∣st)q_t(u)/q_t(v)= _0(u s_t)/ _0(v s_t). Thus, SoFT makes the minimum KL change needed to support the demonstrated token while keeping Base’s relative preferences among alternatives. A.2 Soft target decomposition Using the main-text coefficients, at a_t =τx−π0(yt⋆∣st)1−π0(yt⋆∣st),π0(yt⋆∣st)<τx,0,π0(yt⋆∣st)≥τx,λt=1−at, = cases _x- _0(y_t s_t)1- _0(y_t s_t),& _0(y_t s_t)< _x,\\[5.0pt] 0,& _0(y_t s_t)≥ _x, cases _t=1-a_t, (13) qt q_t =atδyt⋆+λtπ0(⋅∣st). =a_t _y_t + _t _0(· s_t). The mixture assigns maxπ0(yt⋆∣st),τx \ _0(y_t s_t), _x\ to yt⋆y_t and rescales the remaining Base mass proportionally, exactly recovering Eq. 12. Let ℒSFT,t=−logπθ(yt⋆∣st)L_SFT,t=- _θ(y_t s_t). Using the linearity of cross-entropy in its target, ℒSoFT,t _SoFT,t =CE(qt,πθ(⋅∣st)) =CE\! (q_t, _θ(· s_t) ) (14) =atCE(δyt⋆,πθ(⋅∣st))+λtCE(π0(⋅∣st),πθ(⋅∣st)) =a_tCE\! ( _y_t , _θ(· s_t) )+ _tCE\! ( _0(· s_t), _θ(· s_t) ) (15) =atℒSFT,t+λtKL(π0(⋅∣st)∥πθ(⋅∣st))+λtH(π0(⋅∣st)). =a_tL_SFT,t+ _tKL\! ( _0(· s_t)\,\|\, _θ(· s_t) )+ _tH\! ( _0(· s_t) ). (16) The Base entropy is independent of θ. Therefore, ℒSoFT,t=atℒSFT,t+λtKL(π0(⋅∣st)∥πθ(⋅∣st))+constant. L_SoFT,t=a_tL_SFT,t+ _tKL\! ( _0(· s_t)\,\|\, _θ(· s_t) )+constant. (17) SoFT combines demonstrated-token learning with Base-policy preservation at each token. The probability floor determines the strength of both terms through at+λt=1a_t+ _t=1. A.3 Relationship to DFT Let ztz_t denote the student logits at state sts_t. From Eq. 13, the SoFT gradient is ∇ztℒSoFT,t=at(πθ(⋅∣st)−δyt⋆)+λt(πθ(⋅∣st)−π0(⋅∣st)). _z_tL_SoFT,t=a_t\! ( _θ(· s_t)- _y_t )+ _t\! ( _θ(· s_t)- _0(· s_t) ). (18) At initialization, πθ(⋅∣st)=π0(⋅∣st) _θ(· s_t)= _0(· s_t), so ∇ztℒSoFT,t|θ=θ0=at(π0(⋅∣st)−δyt⋆). . _z_tL_SoFT,t |_θ= _0=a_t\! ( _0(· s_t)- _y_t ). (19) Thus, the initial SoFT update is identical to token-weighted SFT with detached weight ata_t. DFT rescales the one-hot SFT gradient by the current student probability of the demonstrated token (Wu et al., 2026): ℒDFT,t _DFT,t =−sg[πθ(yt⋆∣st)]logπθ(yt⋆∣st), =-sg[ _θ(y_t s_t)] _θ(y_t s_t), (20) ∇ztℒDFT,t _z_tL_DFT,t =sg[πθ(yt⋆∣st)](πθ(⋅∣st)−δyt⋆). =sg[ _θ(y_t s_t)] ( _θ(· s_t)- _y_t ). Here, sgsg denotes the stop-gradient operator. At initialization, DFT uses the weight π0(yt⋆∣st) _0(y_t s_t), whereas SoFT uses ata_t. For a closer comparison, consider weighted SFT with the fixed weight ata_t, ℒmatched,t=−atlogπθ(yt⋆∣st)L_matched,t=-a_t _θ(y_t s_t), which has the same initial gradient as SoFT. As the student moves away from Base, the two gradients differ by ∇ztℒSoFT,t−∇ztℒmatched,t=λt(πθ(⋅∣st)−π0(⋅∣st)). _z_tL_SoFT,t- _z_tL_matched,t= _t\! ( _θ(· s_t)- _0(· s_t) ). (21) DFT and weighted SFT only rescale updates toward the one-hot target δyt⋆ _y_t . SoFT additionally adds a restoring gradient toward Base and is minimized at qtq_t, so it changes both the strength and the destination of the update. A.4 Interpretation of R⋆R At initialization, the gold-logit gradient magnitudes of SFT and SoFT are |∂ℒSFT,t∂zt,yt⋆| | _SFT,t∂ z_t,y_t | =1−π0(yt⋆∣st), =1- _0(y_t s_t), (22) |∂ℒSoFT,t∂zt,yt⋆| | _SoFT,t∂ z_t,y_t | =at(1−π0(yt⋆∣st))=maxτx−π0(yt⋆∣st),0. =a_t(1- _0(y_t s_t))= \ _x- _0(y_t s_t),0\. SoFT chooses the sequence-level floor τx _x such that R⋆=∑tat(1−π0(yt⋆∣st))∑t(1−π0(yt⋆∣st))=∑tmaxτx−π0(yt⋆∣st),0∑t(1−π0(yt⋆∣st)). R = _ta_t(1- _0(y_t s_t)) _t(1- _0(y_t s_t))= _t \ _x- _0(y_t s_t),0\ _t(1- _0(y_t s_t)). (23) Hence, R⋆R is the fraction of the initial summed SFT gold-logit update that SoFT retains. Since the non-gold gradient components are nonnegative and sum to the magnitude of the gold component, the same ratio also holds for the summed logit-gradient ℓ1 _1 norms. The right-hand side of Eq. 23 is continuous, piecewise linear, and monotone in τx _x. Scalar bisection therefore finds the floor efficiently. Note that the ratio describes the initial logit gradients; the ratio of full parameter gradients can change during training. Appendix B Minimal PyTorch Implementations Both snippets assume a standard causal-LM batch: labels is −100-100 at prompt and padding positions, and shifting by one position produces the teacher-forced next-token pairs. B.1 Standard SFT Standard SFT gathers the log-probability of the demonstrated token, which is equivalent to cross-entropy with the one-hot target qtSFT=δyt⋆q_t^SFT= _y_t . import torch.n.functional as F def sft_loss(student, ids, mask, labels): logits = student(ids, attention_mask=mask).logits[:, :-1] gold = labels[:, 1:] keep = gold.ne(-100) return F.cross_entropy(logits[keep], gold[keep]) B.2 Fixed-R⋆R SoFT SoFT follows Eqs. 4–7, using a frozen, evaluation-mode copy of the initial student as the reference model. B.2.1 Practical approximation A full-vocabulary target costs O(BT||)O(BT|V|) memory. We retain the Base top-K tokens and the demonstrated token at each supervised position, then merge the remaining vocabulary into one tail bucket. With t=yt⋆∪TopK(π0(⋅∣st))S_t=\y_t \ ( _0(· s_t)), the loss becomes ℒ~t=−∑v∈tqt(v)logπθ(v∣st)−qt(tail)logπθ(tail∣st). L_t=- _v _tq_t(v) _θ(v s_t)-q_t(tail) _θ(tail s_t). (24) We use K=32K=32 and include yt⋆y_t once when it appears in the top-K. The approximate target keeps the exact demonstrated-token probability and the total tail mass, representing the remaining tokens as one bucket, which reduces target storage to O(BTK)O(BTK). Caching Base probabilities avoids repeated reference forward passes, and processing student positions in chunks limits activation memory. Inference uses the student alone. The following dense reference implementation shows the exact fixed-R⋆R objective before the top-K approximation. import torch import torch.n.functional as F def soft_loss(student, reference, ids, mask, labels, rstar): logits = student(ids, attention_mask=mask).logits[:, :-1] gold = labels[:, 1:] keep = gold.ne(-100) gold = gold.masked_fill(~keep, 0) with torch.no_grad(): ref_logits = reference( ids, attention_mask=mask ).logits[:, :-1] p0 = F.softmax(ref_logits, dim=-1) p0_gold = p0.gather(-1, gold[..., None]).squeeze(-1) # Solve R_x(tau_x) = R* for every sequence. weight = keep.to(p0.dtype) denom = ((1 - p0_gold) * weight).sum(-1).clamp_min(1e-12) lo, hi = torch.zeros_like(denom), torch.ones_like(denom) for _ in range(28): tau = (lo + hi) / 2 numer = (((tau[:, None] - p0_gold).clamp_min(0)) * weight).sum(-1) right = numer / denom < rstar lo = torch.where(right, tau, lo) hi = torch.where(right, hi, tau) tau = ((lo + hi) / 2)[:, None] # Minimum-change target q_tau. q_gold = torch.maximum(p0_gold, tau) scale = (1 - q_gold) / (1 - p0_gold).clamp_min(1e-12) q = p0 * scale[..., None] q.scatter_(-1, gold[..., None], q_gold[..., None]) logp = F.log_softmax(logits, dim=-1) token_loss = -(q * logp).sum(-1) return token_loss[keep].mean() The Base probabilities define the target, the bisection loop converts the fixed budget R⋆R into one floor τx _x per sequence, and the last lines compute CE(qt,πθ)CE(q_t, _θ). Appendix C Training Data Details Training uses one epoch and seed 42, with maximum sequence lengths of 8,192 for reasoning and 32,768 for agentic tasks. Methods within each setting share the data order. The selected SoFT learning rates are 10−510^-5, 5×10−65× 10^-6, and 10−610^-6 for Qwen2.5-1.5B-Instruct, Qwen2.5-7B, and Qwen3-8B, respectively. Counts measure training trajectories; several trajectories may share a task. Token statistics exclude validation examples. Source links identify the released datasets or environments, and teacher names follow the provenance records. We mark unknown upstream teachers as unspecified. C.1 Reasoning Tasks Both Qwen2.5 students use 36,000 training trajectories (9,000 per domain) and 800 validation examples (200 per domain). All reasoning trajectories come from public releases; Table 4 lists their sources and teachers. The stored token_length is measured with the Qwen2.5-1.5B-Instruct tokenizer and training chat template and includes prompt, completion, and template tokens. We keep sequences of at most 8,192 tokens. The released sources are GLM and Kimi cleaned trajectories (Jackrong, 2026a; Jackrong, 2026b), Sky-T1 (NovaSky, 2025), OpenThoughts-4 agreed-answer math traces (Marin Community, 2026), Mixture-of-Thoughts (Hugging Face, 2025a), and community GPT-OSS traces (iAmBoosted, 2026). The GLM and Kimi releases derive from the original datasets by Kassadin88 and ianncity (Kassadin88, 2026; ianncity, 2026). Open R1 releases Mixture-of-Thoughts (Hugging Face, 2025b). Its math, code, and science sources are OpenR1-Math, CodeForces CoTs, and Llama-Nemotron (Lozhkov et al., 2025; Penedo et al., 2025; Bercovich et al., 2025). The 600 GPT-OSS trajectories use 477 prompts from OpenThoughts-114k (Guha et al., 2025) and 123 from OpenCodeReasoning (Ahmad et al., 2025); GPT-OSS-20B generated the responses (OpenAI, 2025). The manifest records 11,260 verified-correct rows, 9,000 upstream-verified rows, 13,140 cleaned rows with unverified answers, 2,000 answer-agreement rows, and 600 GPT-OSS rows without verified references (477 without a reference and 123 marked unverifiable). Verification strength therefore varies by source. Length filtering only checks that sequences fit the context window; answer-quality labels are kept separately in the manifest. Table 4: Reasoning training sources, teachers, and full-sequence token statistics. Validation examples are excluded. Domain Task-data source Teacher Train Link Min Mean Max Math GLM cleaned GLM-5.1 450 source 1,371 5894.3 8,187 Math Kimi cleaned Kimi-K2.5 690 source 2,361 6561.5 8,191 Math Sky-T1 QwQ-32B / STILL-2 2,400 source 269 3927.4 8,165 Math OpenThoughts-4 math Qwen3-235B-A22B 2,000 source 1,535 6658.6 8,192 Math Mixture-of-Thoughts DeepSeek-R1 3,460 source 468 4109.7 8,189 Code Sky-T1 QwQ-32B / STILL-2 3,600 source 798 4898.2 8,191 Code GPT-OSS traces GPT-OSS-20B 600 source 637 3264.4 7,978 Code Mixture-of-Thoughts DeepSeek-R1 4,800 source 551 5148.1 8,191 Science GLM cleaned GLM-5.1 3,000 source 2,014 3611.5 8,186 Science Kimi cleaned Kimi-K2.5 3,000 source 1,367 3421.0 7,633 Science Mixture-of-Thoughts DeepSeek-R1 3,000 source 389 2061.0 8,168 General GLM cleaned GLM-5.1 3,000 source 132 2935.9 8,142 General Kimi cleaned Kimi-K2.5 3,000 source 122 3399.5 8,180 General Sky-T1 QwQ-32B / STILL-2 3,000 source 165 2981.3 8,164 Total 36,000 – 122 3991.0 8,192 Example from the training corpus. Below is a verified DeepSeek-R1 Math example from Mixture-of-Thoughts; the middle of its reasoning is omitted for space. User. At the cross-country race, 8 runners wore white sports shirts. 4 runners wore blue sports shirts. How many runners started the cross-country race? Teacher reasoning, excerpt. So 8 white shirts plus 4 blue shirts. … 8 plus 4 equals 12. Final answer: 12 12. C.2 Agentic Tasks After token-level deduplication, the agentic corpus contains 8,252 training and 824 validation trajectories. Table 5 reports the stored sequence_length measured with the Qwen3-8B tokenizer and chat template. Lengths include user and system text, tool schemas, tool observations, and assistant turns. The corpus contains 53,623,153 full-sequence tokens and 7,278,365 supervised assistant tokens, with a 32,768-token limit. Teacher reasoning text is removed; actions, observations, final answers, and tool-call links are retained. Locally collected τ2τ^2 (Barres et al., 2025) and LiveCodeBench (Jain et al., 2024) trajectories use DeepSeek V4 Flash. We reuse published Nemotron (NVIDIA, 2025) and OpenThoughts-Agent (Raoof and others, 2026) trajectories; the local metadata leaves their individual teachers unspecified. TMax contributes published successful trajectories (Ivison et al., 2026) with reasoning text removed. Final packaging removes five exact token-sequence duplicates. The 1,282 coding trajectories cover 296 unique prompts. Table 5: Agentic training sources, teachers, and full-sequence token statistics. Validation examples are excluded. Domain group Task-data source Teacher Train Link Min Mean Max Coding LiveCodeBench v1 DeepSeek V4 Flash 1,282 source 155 683.0 4,333 Customer Nemotron interactive Upstream; unspecified 1,200 source 1,837 3972.1 8,534 τ2τ^2 retail/airline DeepSeek V4 Flash 2,997 source 4,899 9127.8 22,431 Search Nemotron search Upstream; unspecified 600 source 3,892 18791.6 32,701 Tool Interaction TMax-SFT Qwen3.6-27B 595 source 1,661 5575.2 30,575 OpenThoughts-Agent Upstream; unspecified 387 source 949 2730.1 17,593 Nemotron tool calling Upstream; unspecified 1,191 source 87 4178.3 31,182 Total 8,252 – 87 6498.2 32,701 Example from the training corpus. The following LiveCodeBench training example uses a DeepSeek V4 Flash response. We excerpt the prompt and show the complete assistant code. User, excerpt. You are given an integer N between 1 and 9, inclusive, as input. Concatenate N copies of the digit N and print the resulting string. Assistant.N = int(input()) print(str(N) * N) Appendix D Evaluation Tasks and Splits D.1 Reasoning Tasks The ID suite contains four task families. OpenR1-Math contributes 300 mathematical problems. APPS contributes 250 programming problems in the benchmark’s difficulty proportions: 50 introductory, 150 interview, and 50 competition tasks. Science MCQ contains 1,000 multiple-choice science questions selected from Mixture-of-Thoughts. RiddleSense contributes 1,018 multiple-choice commonsense riddles. The OOD suite tests the same four capabilities using different question sources. GSM8K contains 1,319 grade-school mathematics word problems. MBPP+ contains 378 short Python programming tasks. ARC-Challenge contributes 608 multiple-choice science questions. BoolQ contributes 708 deduplicated yes/no questions about short passages, drawn from its labeled validation split. Table 1 summarizes the ID–OOD pairing for each domain. D.2 Agentic Tasks The ID suite covers customer interaction, coding, search, and tool interaction. The 60 τ2τ^2 tasks are held-out retail and airline customer-service scenarios involving orders, returns, flights, and bookings. LiveCodeBench contributes 694 programming problems. Search ID contains 500 held-out Nemotron questions that call for finding and combining information. TMax contributes 300 tasks in terminal environments, where the agent works with commands and files. The OOD suite uses different task sources or domains. Its 300 τ2τ^2 cases come from telecom customer support and banking-knowledge retrieval. BigCodeBench contributes 2,280 evaluation cases: 1,140 software-oriented coding tasks presented in both completion and instruction formats. BFCL V4 contributes 5,106 function-calling cases, including tool selection, argument construction, and multi-turn interactions. FRAMES contains 824 factual questions that require retrieving and combining evidence from multiple sources. Appendix E Token-Level Allocation Diagnostics Figure 7 reconstructs token weights from saved Base probabilities and selected domain budgets for all 36,000 trajectories at each reasoning scale and 8,252 agentic trajectories. The 8B sources are pooled into four domain groups by supervised-token counts. Weights decrease with Base confidence by construction. The empirical question is where the training data place tokens along this curve. The adjustment is concentrated in a subset of tokens. Within 8B Tool Interaction, the Terminal source has at>0a_t>0 on 9.3% of tokens, while 87.0% already receive Base probability above 0.9. Tokens with at=0a_t=0 keep the Base distribution as their training target. The fraction of adjusted tokens also does not simply follow the budget: at 1.5B, Science uses R⋆=0.45R =0.45 and adjusts 54.7% of tokens, whereas Code uses R⋆=0.60R =0.60 and adjusts 48.2%. The budget controls the initial gold-logit gradient ratio, whereas the adjusted-token fraction depends on Base confidence. The same budget can likewise produce different adjustment rates. The three non-coding agentic groups all use R⋆=0.3R =0.3, yet their rates range from 16.9% on Search to 19.8% on Customer; Tool Interaction lies between them at 18.5%. Its Terminal and Tool Calling sources differ more sharply, at 9.3% and 25.5%. Thus, the distribution of Base confidence shapes how each group and source receives supervision. Together, these measurements show how SoFT adapts token-level learning within a mixed training corpus. Figure 7: Token-weighted target allocation from Base caches. Lines: mean ata_t by Base probability; circle areas: token fractions; legend percentages: fraction with at>0a_t>0. The 8B panel pools source data into four domain groups. Appendix F Policy Drift and Behavioral Retention Figure 3 normalizes each method’s forward KL KL(π0∥πθ)KL( _0\| _θ) by that of SFT in the same setting. The reasoning measurements use the same 32 held-out validation trajectories per domain as RQ2 and average over the first 128 demonstrated tokens. The 8B measurement uses 16 validation trajectories from each of five agentic source domains and the first 128 supervised assistant tokens. Normalization places all settings on a common axis, although absolute KL still depends on the data. Figure 8 compares absolute reasoning KL with mean Base-success retention. SoFT has the lowest drift and among the highest retention at both scales. At 1.5B, drift and retention broadly move in opposite directions. At 7B, DFT moves far from Base while retaining many successes, whereas Rho-1 moves less and retains fewer. Global KL captures one aspect of preservation; task success reveals another. The policy diagnostic uses validation prefixes, while retention is measured on the separate benchmark tasks. Figure 8: Final policy drift versus Base-success retention on reasoning tasks. Drift is KL(π0∥πθ)KL( _0\| _θ) over held-out teacher-forced prefixes; retention is the equal-weight mean over eight benchmarks. Appendix G Why Normalized Target-KL Curves Can Align Section 6.3 shows nearly aligned normalized curves alongside different absolute target KL values. We use the KL chain rule, a mixture decomposition, and a low-probability limit to explain how this pattern can arise from the constructed targets. G.1 Reducing the full-vocabulary KL to a scalar Write r∈[0,1]r∈[0,1] for a candidate value of R⋆R . For trajectory x, let TxT_x be the number of supervised positions. Assume π0(yt⋆∣st)>0 _0(y_t s_t)>0 and ∑t(1−π0(yt⋆∣st))>0 _t(1- _0(y_t s_t))>0. The floor τx(r) _x(r) satisfies ∑tmaxτx(r)−π0(yt⋆∣st),0=r∑t(1−π0(yt⋆∣st)). _t \ _x(r)- _0(y_t s_t),0\=r _t(1- _0(y_t s_t)). (25) At the endpoints, use the Base target for r=0r=0 and the one-hot target for r=1r=1. Define the binary KL and binary entropy, using natural logarithms, as b(u∥p) b(u\|p) =ulogup+(1−u)log1−u1−p, =u up+(1-u) 1-u1-p, (26) h(u) h(u) =−ulogu−(1−u)log(1−u), =-u u-(1-u) (1-u), with the usual continuous endpoint conventions and b(1∥1)=0b(1\|1)=0. Proposition 1 (binary reduction). For the SoFT target, written qτx,tq_ _x,t in this section to make its dependence on the floor explicit, KL(qτx,t∥π0(⋅∣st))=b(maxπ0(yt⋆∣st),τx∥π0(yt⋆∣st)).KL\! (q_ _x,t\,\|\, _0(· s_t) )=b\! ( \ _0(y_t s_t), _x\\,\|\, _0(y_t s_t) ). (27) Proof. Partition the vocabulary into the demonstrated token and its complement. The KL chain rule separates the divergence into the binary mass-allocation term and a conditional KL over other tokens. SoFT preserves the Base distribution conditional on that complement, so the conditional KL is zero. When π0(yt⋆∣st)=1 _0(y_t s_t)=1, the target remains unchanged and both sides are zero. The per-trajectory absolute and normalized changes are therefore Dx(r) D_x(r) =1Tx∑tb(maxπ0(yt⋆∣st),τx(r)∥π0(yt⋆∣st)), = 1T_x _tb\! ( \ _0(y_t s_t), _x(r)\\,\|\, _0(y_t s_t) ), (28) Cx(r) C_x(r) =Dx(r)Dx(1),Dx(1)=1Tx∑t−logπ0(yt⋆∣st). = D_x(r)D_x(1), D_x(1)= 1T_x _t- _0(y_t s_t). We form each normalized domain curve by averaging Cx(r)C_x(r) over trajectories. The endpoints satisfy Cx(0)=0C_x(0)=0 and Cx(1)=1C_x(1)=1 by construction. The intermediate shape depends on the trajectories’ Base probabilities and can vary across domains. G.2 An exact cancellation of token proportions Let P be the Base probability of the demonstrated token at a uniformly chosen position in a trajectory. Consider the idealized mixture P∼(1−f)δ1+fF,0<f≤1,P (1-f) _1+fF, 0<f≤ 1, (29) where δ1 _1 is a point mass at probability one and F is a distribution on (0,1)(0,1) with finite, positive F[−logP]E_F[- P]. The first component represents tokens with full Base confidence; the second contains the remaining tokens. The point mass at one makes the cancellation explicit; in finite-logit models, high-confidence probabilities only approach one. Proposition 2 (mixture-proportion invariance). For fixed F, the entire normalized curve is independent of f. Proof. Both sides of the budget equation contain the same factor f: fF[(τ−P)+]=rfF[1−P].f\,E_F[(τ-P)_+]=rf\,E_F[1-P]. (30) After canceling f, the floor depends only on F and r. Denote it by τF(r) _F(r). The certain tokens contribute zero KL, so Dx(r) D_x(r) =fF[b(maxP,τF(r)∥P)], =f\,E_F[b( \P, _F(r)\\|P)], (31) Dx(1) D_x(1) =fF[−logP], =f\,E_F[- P], Cx(r) C_x(r) =F[b(maxP,τF(r)∥P)]F[−logP]. = E_F[b( \P, _F(r)\\|P)]E_F[- P]. The final expression contains no f. For example, increasing the proportion of uncertain tokens from 10% to 50% multiplies absolute KL by five while leaving the normalized curve unchanged when F stays fixed. Domains can therefore require different amounts of adjustment yet have identical normalized responses. The result extends to domain averages when they share the same distribution of trajectory-level F values, even if their f values differ. The full vocabulary distributions may still differ. G.3 A closed form and a low-probability limit For a simple special case, set F=δϵF= _ε with 0<ϵ<10<ε<1. All demonstrated tokens whose Base probability is below one then have probability ϵε. Equation 25 gives τϵ(r)=ϵ+(1−ϵ)r,Cϵ(r)=b(ϵ+(1−ϵ)r∥ϵ)log(1/ϵ). _ε(r)=ε+(1-ε)r, C_ε(r)= b(ε+(1-ε)r\|ε) (1/ε). (32) The proportion f cancels, while the confidence level ϵε continues to shape the curve. Proposition 3 (entropy bound). For this special case and every r∈[0,1]r∈[0,1], max0,r−log2log(1/ϵ)≤Cϵ(r)≤r. \! \0,r- 2 (1/ε) \≤ C_ε(r)≤ r. (33) Consequently, Cϵ(r)C_ε(r) converges uniformly to r as ϵ→0ε→ 0. Proof. Put u=ϵ+(1−ϵ)ru=ε+(1-ε)r and L=log(1/ϵ)L= (1/ε). Expanding the binary KL gives the exact identity Cϵ(r)=u−h(u)L+(1−u)log(1/(1−ϵ))L.C_ε(r)=u- h(u)L+ (1-u) (1/(1-ε))L. (34) Since u≥ru≥ r, h(u)≤log2h(u)≤ 2, and the last term is nonnegative, this yields the lower bound, together with nonnegativity of KL. For the upper bound, Bern(u)=(1−r)Bern(ϵ)+rδ1Bern(u)=(1-r)Bern(ε)+r _1. Convexity of KL in its first argument gives b(u∥ϵ)≤rlog(1/ϵ)b(u\|ε)≤ r (1/ε). The resulting uniform error bound is 0≤r−Cϵ(r)≤log2/L0≤ r-C_ε(r)≤ 2/L. For sufficiently small ϵε, the leading term approaches r. The entropy term places the curve below the diagonal. The approximation remains sensitive to confidence: at r=0.5r=0.5, Cϵ(r)C_ε(r) is about 0.189 for ϵ=0.5ε=0.5 and 0.356 for ϵ=0.01ε=0.01. Thus, curve alignment also depends on the probability profile. G.4 Nearly certain tokens For a more realistic mixture, replace δ1 _1 by a distribution G supported on [1−δ,1][1-δ,1]. Let F be supported on (0,ϵ](0,ε] and suppose the floor lies between ϵε and 1−δ1-δ. Then all F tokens are active and all G tokens are inactive. Define κ=1−f,μ=F[P],ν=G[1−P],ℓF=F[−logP],cF=F[−log(1−P)],ℓG=G[−logP]. gatheredκ= 1-ff, μ=E_F[P], ν=E_G[1-P],\\ _F=E_F[- P], c_F=E_F[- (1-P)], _G=E_G[- P]. gathered (35) The budget equation and normalized divergence now give τ(r) τ(r) =r+(1−r)μ+rκν, =r+(1-r)μ+rκν, (36) Cx(r) C_x(r) =τ(r)ℓF−h(τ(r))+(1−τ(r))cFℓF+κℓG. = τ(r) _F-h(τ(r))+(1-τ(r))c_F _F+κ _G. Substituting the mixture into the budget equation and expanding binary KL over the active component gives these identities. The high-confidence component contributes residual gradient mass through κνκν and endpoint KL through κℓGκ _G, restoring dependence on f. For fixed F and f, the idealized result emerges as G concentrates at one under the stated active-set condition. When κ is large, even small residuals from many nearly certain tokens can affect the curve. G.5 A counterexample and implications for the observation The homogeneous case has a specific bound. To see how a mixed probability profile differs, consider equal numbers of tokens with probabilities ϵε and 1/21/2, and set r=1/2r=1/2. Both groups are active at τ=58+ϵ4.τ= 58+ ε4. (37) As ϵ→0ε→ 0, the low-probability group dominates both KL expressions, giving Cx(1/2)=b(τ∥ϵ)+b(τ∥1/2)log(1/ϵ)+log2⟶58.C_x(1/2)= b(τ\|ε)+b(τ\|1/2) (1/ε)+ 2 58. (38) In contrast, the homogeneous low-probability case in Proposition 3 approaches 1/21/2. Here, the normalized KL exceeds the budget, showing how different probability profiles can separate the curves. The alignment in Section 6.3 is consistent with domains sharing a similar conditional probability profile while differing in the amount of required adjustment. The derivation identifies those profiles and their KL contributions as the quantities that shape the normalized curves. Appendix H Training Dynamics The following curves cover the 21 trained runs in the main comparisons. Reasoning and agentic runs reach 2,250 and 258 steps, respectively; Base contributes only to evaluation. We average logged training metrics in non-overlapping 50-step and 10-step windows. Held-out entropy and learning rates use their recorded points. H.1 Training Objectives and Predictive Entropy Figure 9 shows distinct objective scales. Weighted losses are small for DFT and PriFT-mass, while Confidence reports larger values. SoFT’s reasoning loss changes little because it includes the entropy of the soft target as a constant offset. The curves describe optimization within each run; benchmark results provide the performance comparison. Figure 9: Training objectives averaged over 50-step (reasoning) or 10-step (agentic) windows. Objective scales vary across methods; benchmark scores report task performance. Figure 10 uses held-out entropy for reasoning and training entropy for agentic tasks. Final reasoning entropy is higher for SoFT than SFT: 1.228 versus 1.066 at 1.5B and 1.009 versus 0.713 at 7B. Over the last 10% of agentic steps, their means are similar (0.395 versus 0.386), while Confidence reaches 0.722 and DFT 0.066. SoFT’s entropy therefore depends on the setting. Entropy measures how concentrated predictions are; correctness and Base preservation require separate evaluations. The different populations also limit comparisons across panels. Figure 10: Predictive entropy (nats). Left/middle: recorded held-out evaluations. Right: training entropy in 10-step windows. Populations differ across panels; missing points are not imputed. H.2 Gradient Norms and Learning-Rate Schedules SoFT’s mean logged gradient norm decreases between the first and last 10% of steps: 0.770 to 0.555 at 1.5B, 1.659 to 0.644 at 7B, and 1.014 to 0.178 at 8B (Figure 11). These values describe the logged update scale, and their interpretation depends on loss scaling, trainer conventions, and fluctuations hidden by window averages. Figure 11: Logged gradient norms (log scale), using the windows in Figure 9. Magnitudes depend on objective scaling and trainer conventions. Figure 12 shows SoFT’s peak rates of 10−510^-5, 5×10−65× 10^-6, and 10−610^-6, followed by decay. Other methods use their own selected rates, and identical schedules overlap. Together, these curves characterize the optimization schedules chosen for the main comparisons. Figure 12: Recorded learning-rate schedules without smoothing. Identical schedules overlap. Appendix I Realized Demonstration Learning For each Qwen2.5 scale, we select 32 validation examples per reasoning domain by a fixed hash, using prompts of at most 512 tokens and at least 128 demonstrated tokens. All seven trained methods are scored on the same teacher-forced states and the first 128 demonstrated tokens. We report the gold-token probability gain relative to Base’s remaining probability mass, as defined in Section 6.2. Horizontal bars in Figure 5 show approximate 95% ranges from 2,000 domain-stratified resamples of the selected trajectories; vertical coordinates are the main-table benchmark GMs. The prefix measurements use validation data, while the task scores come from the separate evaluation suites. At 7B, three methods with almost the same aggregate acquisition ratio occupy widely separated positions on the vertical axis. Similar gold-token gains can therefore accompany different task outcomes. The scalar summarizes teacher-forced probability gain, while the retention analysis and benchmark scores capture behavior on complete responses. Table 6: Realized acquisition by reasoning domain (%). Values use the same 32 validation examples per domain as Figure 5. Qwen2.5-1.5B Qwen2.5-7B Domain SFT DFT SoFT SFT DFT SoFT General 20.6 41.6 4.0 43.4 50.1 8.8 Math 24.9 47.2 6.0 47.2 52.9 11.3 Science 18.8 37.1 7.5 36.9 44.4 11.8 Code 18.9 40.0 10.2 38.4 44.2 16.0 The ordering DFT >> SFT >> SoFT holds in all four domains at both scales, so the aggregate contrast is not driven by a single domain. Within SoFT, Code has the largest realized gain at both scales and also receives the largest validation-selected budget. The domain budget and Base probability profile jointly shape the target used for these trajectories.