Paper deep dive
What Should SFT Learn? Conditional Information Gain for Generalizable Post-Training
Yuxuan Gu, Kequan Niu, Xiaocheng Feng, Yun Li, Yiqiao Yang, Bing Qin
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 10/12/2026, 6:42:43 AM
Summary
The paper introduces CIG-SFT, an information-theoretic framework for supervised fine-tuning (SFT) that uses conditional information gain to weight token-level supervision. Unlike standard SFT or methods relying on model predictive probability, CIG-SFT quantifies the task-specific information in demonstrations by comparing token likelihoods with and without task input under a frozen reference model. This score is used to derive a gradient-equivalent scalar-weighted SFT objective, improving generalization in mathematical reasoning and code generation without requiring online rollouts or additional data.
Entities (9)
Relation Signals (7)
CIG-SFT → uses → Conditional Information Gain
confidence 95% · CIG-SFT, an information-theoretic framework that estimates the task-conditioned information carried by each demonstrated token.
Conditional Information Gain → measures → Task-specific Information
confidence 92% · The resulting conditional information gain quantifies how much the task itself reduces uncertainty about the demonstrated decision.
CIG-SFT → appliesto → Mathematical Reasoning
confidence 90% · Across mathematical reasoning and code-generation benchmarks, CIG-SFT consistently outperforms
CIG-SFT → appliesto → Code-generation
confidence 90% · Across mathematical reasoning and code-generation benchmarks, CIG-SFT consistently outperforms
CIG-SFT → outperforms → Supervised Fine-tuning
confidence 90% · CIG-SFT consistently outperforms standard SFT and recent adaptive fine-tuning baselines
CIG-SFT → derivesfrom → KL-regularized Projection
confidence 88% · We then convert this information into a KL-regularized projection of the current policy and derive an equivalent scalar-weighted SFT objective
CIG-SFT → outperforms → Dynamic Fine-Tuning
confidence 85% · CIG-SFT consistently outperforms standard SFT and recent adaptive fine-tuning baselines
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Supervised fine-tuning (SFT) remains a widely used approach for post-training large language models, because fixed demonstrations can directly introduce desired reasoning strategies and behavioral patterns. Yet forcing a model to imitate externally provided trajectories may also interfere with behaviors already supported by pretraining, limiting generalization beyond the training distribution. Recent dynamic SFT methods mitigate this issue by reweighting token-level supervision according to model-side signals such as predictive probability, prior support, or distributional compatibility. However, model compatibility is not equivalent to learning value: a token that is weakly supported by the current model may either be an undesirable mismatch or precisely the task-specific decision that the demonstration is intended to teach. We propose CIG-SFT, an information-theoretic framework that estimates the task-conditioned information carried by each demonstrated token. Under teacher forcing, we compare the token likelihood from a frozen reference model with and without access to the task input while keeping the gold solution prefix fixed. The resulting conditional information gain quantifies how much the task itself reduces uncertainty about the demonstrated decision. We then convert this information into a KL-regularized projection of the current policy and derive an equivalent scalar-weighted SFT objective whose logit gradient exactly matches that of the projected target. Across mathematical reasoning and code-generation benchmarks, CIG-SFT consistently outperforms standard SFT and recent adaptive fine-tuning baselines, with particularly strong improvements in generalization to challenging evaluation distributions, while requiring neither additional training data nor online rollouts.
Tags
Links
- Source: https://arxiv.org/abs/2610.12995v1
- Canonical: https://arxiv.org/abs/2610.12995v1
Trouble viewing inline? Open PDF directly →
Full Text
66,068 characters extracted from source content.
Expand or collapse full text
What Should SFT Learn? Conditional Information Gain for Generalizable Post-Training Yuxuan Gu †thanks: Equal contribution. Code is available at https://github.com/NiuKequan/CIG-SFT-public. Kequan Niu11footnotemark: 1 Xiaocheng Feng Yun Li Yiqiao Yang Bing Qin Affiliation: Harbin Institute of Technology Email: yxgu,keniu,xcfeng@ir.hit.edu.cn Abstract Supervised fine-tuning (SFT) remains a widely used approach for post-training large language models, because fixed demonstrations can directly introduce desired reasoning strategies and behavioral patterns. Yet forcing a model to imitate externally provided trajectories may also interfere with behaviors already supported by pretraining, limiting generalization beyond the training distribution. Recent dynamic SFT methods mitigate this issue by reweighting token-level supervision according to model-side signals such as predictive probability, prior support, or distributional compatibility. However, model compatibility is not equivalent to learning value: a token that is weakly supported by the current model may either be an undesirable mismatch or precisely the task-specific decision that the demonstration is intended to teach. We propose CIG-SFT, an information-theoretic framework that estimates the task-conditioned information carried by each demonstrated token. Under teacher forcing, we compare the token likelihood from a frozen reference model with and without access to the task input while keeping the gold solution prefix fixed. The resulting conditional information gain quantifies how much the task itself reduces uncertainty about the demonstrated decision. We then convert this information into a KL-regularized projection of the current policy and derive an equivalent scalar-weighted SFT objective whose logit gradient exactly matches that of the projected target. Across mathematical reasoning and code-generation benchmarks, CIG-SFT consistently outperforms standard SFT and recent adaptive fine-tuning baselines, with particularly strong improvements in generalization to challenging evaluation distributions, while requiring neither additional training data nor online rollouts. 1 Introduction Large language models (LLMs) acquire broad and increasingly general capabilities during large-scale pretraining, ranging from language understanding and generation to reasoning and problem solving (Brown et al., 2020; Chowdhery et al., 2023). Since base models are not inherently optimized to express capabilities in the forms required by diverse downstream tasks, post-training plays a critical role in transforming a general-purpose base model into one that exhibits desired task- and application-specific behaviors (Chung et al., 2024). By providing explicit behavioral supervision or optimization signals, post-training can improve instruction following (Wei et al., 2021; Ouyang et al., 2022), shape how models reason and make intermediate decisions (Lightman et al., 2024; Guo et al., 2025), and encourage adherence to task-specific response structures and output conventions (Chu et al., 2025). These adaptations are particularly important for reasoning-intensive applications, where success depends not only on whether a model possesses the underlying capability, but also on whether it can invoke and organize that capability into an effective solution process. Accordingly, supervised fine-tuning (SFT) and reinforcement learning (RL) have become two central paradigms for modern LLM post-training, providing complementary mechanisms for shaping model behavior beyond pretraining (Ouyang et al., 2022; Schulman et al., 2017; Guo et al., 2025). However, such adaptation introduces a fundamental tension between acquiring target behaviors and preserving the capabilities that LLMs already support. It is especially evident in post-training methods that learn from fixed, externally provided data, including SFT, offline preference optimization (Rafailov et al., 2023), and offline reinforcement learning (Snell et al., 2022). Their key advantage is behavioral coverage: by directly optimizing on curated demonstrations or feedback, these methods can expose the model to reasoning steps and behavioral patterns that it may rarely produce on its own. Yet this same property can also create a mismatch between externally specified supervision and the model’s existing predictive behavior, biasing LLM behavior toward the demonstrated solution patterns at the expense of catastrophic forgetting of capabilities inherited from pretraining (Kotha et al., 2024; Li et al., 2024a). On-policy approaches address the same tension from the opposite direction, which generate training trajectories from the evolving policy itself and thus make the supervision naturally better aligned with LLMs’ predictive behavior distribution, such as reinforcement learning (Schulman et al., 2017; Ouyang et al., 2022; Guo et al., 2025), self-taught reasoning (Zelikman et al., 2022), on-policy distillation (Agarwal et al., 2024), and on-policy self-distillation (Zhao et al., 2026). Nevertheless, this compatibility comes at the cost of behavioral coverage, as useful strategies encoded in external demonstrations may be missed when the current policy rarely visits them (Agarwal et al., 2024; Zhao et al., 2026). Recent work mitigates this trade-off by retaining fixed demonstrations while selectively modulating their supervision at the token level. DFT reweights each target according to its probability under the current model (Wu et al., 2025), where subsequent methods stabilize this reweighting through distributional anchoring (Zhu et al., 2025), estimate token support from a frozen pretrained model (Wang et al., 2026b), or explicitly characterize whether a target is compatible with the model-induced distribution (Zhang et al., 2026). Despite different formulations, they share a common principle: the strength of supervision is largely determined by how well a demonstrated token is supported by the model’s predictive distribution. However, predictive compatibility is not equivalent to learning value. A weakly supported target may indicate an undesirable distributional mismatch, but it may also correspond to precisely the task-specific decision that the demonstration is intended to teach. Conversely, a highly supported target may reflect a useful model-consistent behavior, or simply a trivial continuation already determined by the preceding gold trajectory. Model support alone cannot distinguish what it should acquire from what it should avoid, or what has been sufficiently captured. Therefore, we quantify the learning value of each demonstrated token from an information-theoretic perspective, viewing information gain as the reduction in uncertainty induced by an additional observation (Shannon, 1948; Lindley, 1956; Cover and Thomas, 2006). In supervised fine-tuning, the task input specifies the behavior to be learned, while the preceding response tokens also contribute to predicting the next token. Model-support signals computed from the full context may conflate task-induced information with continuation-induced predictability. We disentangle these two sources by measuring the additional information contributed by the task input beyond the preceding context. In detail, we propose CIG-SFT, which translates pointwise conditional information gain into principled supervised updates. First, we estimate the task-conditioned information carried by each demonstrated token via comparing its log-likelihood ratio under the frozen base model with and without the task input while keeping the preceding response fixed. The resulting ratio is a pointwise conditional information quantity, which we normalize by task-free surprisal to obtain a bounded information-gain ratio. Next, we treat this value as a token-level reward and derive a KL-regularized projection of the current policy, producing an information-dependent target distribution that balances information acquisition against policy deviation. We show that the gradient induced by this projected target can be reproduced exactly by a scalar-weighted SFT objective. In this way, CIG-SFT provides a principled path from task-conditioned information to controlled policy adaptation, while retaining fixed demonstrations and requiring neither additional training data nor online rollouts. We evaluate CIG-SFT on mathematical reasoning and code generation across multiple model scales. CIG-SFT consistently improves over standard SFT and strong adaptive fine-tuning baselines, while comparing favorably with reinforcement-learning methods without requiring additional data or online rollouts. Its gains persist across domains, training-set sizes, and hyperparameter settings, with efficiency in some low-data regimes. Ablation and training-dynamics analyses further show that CIG-SFT benefits from the separation between frozen information attribution and current-policy-aware proximal adaptation, producing more selective supervision and smaller parameter updates. Overall, our results demonstrate that task-conditioned information provides a useful complement to predictive compatibility for generalizable supervised post-training. Our contributions are threefold: (i) we identify task-conditioned information as a complementary signal to full-context model support for selective SFT; (i) we derive an information-proximal target and show that its optimization admits an exactly gradient-equivalent token-weighted objective; and (i) we demonstrate consistent gains across model scales and domains, with data efficiency and more selective parameter updates. 2 Related Work Supervised fine-tuning and adaptive objectives. Supervised fine-tuning (SFT) remains a standard component of LLM post-training, using dense token-level supervision on fixed demonstrations to adapt pretrained models to desired behaviors (Ouyang et al., 2022; Chung et al., 2024). Despite its effectiveness, directly fitting demonstrations can lead to overfitting, memorization, or degradation of capabilities outside the fine-tuning distribution (Kotha et al., 2024; Li et al., 2024a; Chu et al., 2025). Recent work revisits the uniform weighting of token-level supervision in standard SFT. Dynamic Fine-Tuning (DFT) rescales each target-token loss using its probability under the current model (Wu et al., 2025). Anchored SFT (ASFT) introduces KL-based anchoring to control distributional drift (Zhu et al., 2025). PriFT derives token weights from a frozen pretrained model to decouple prior support from the evolving optimization trajectory (Wang et al., 2026b), while IDFT uses distributional statistics to identify supervision better aligned with the model-induced distribution (Zhang et al., 2026). Related approaches such as ProFit and InfoSFT further explore probability- or information-based token selection and weighting (Liu et al., 2026; Sabbaghi et al., 2026). Collectively, these methods show that non-uniform supervision can improve the generalization of fixed-demonstration SFT. Our method, CIG-SFT, is closest to this line of work, but differs in the signal used to define token-level learning value. Existing adaptive SFT methods primarily derive their weights from predictive confidence, prior support, or distributional typicality under the complete context. CIG-SFT instead uses conditional information gain to quantify how much the task input contributes to each target beyond what the preceding context already reveals. Reinforcement learning and on-policy post-training. Reinforcement learning provides a major paradigm for LLM post-training, where training trajectories are sampled from the evolving policy rather than fixed in advance. PPO-based RLHF has been widely used to optimize language models from preference-derived rewards (Schulman et al., 2017; Ouyang et al., 2022), while GRPO and subsequent variants enable scalable outcome-based optimization for reasoning tasks (Shao et al., 2024; Guo et al., 2025; Yu et al., 2026). Because supervision applies to states the current policy actually visits, on-policy optimization naturally matches the model’s generative distribution and avoids forcing the model to imitate externally specified trajectories. However, the same policy dependence also limits behavioral coverage. Because on-policy supervision comes from trajectories generated by the current policy, behaviors the model rarely discovers are unlikely to receive strong learning signals. This matters especially when external demonstrations encode reasoning strategies or behavioral patterns not yet well represented in the current policy. Empirical studies of reasoning RL provide concrete manifestations of this limitation, including the absence of informative successful rollouts on difficult prompts and the progressive loss of exploration diversity during training (Cui et al., 2025; Yu et al., 2026; Wang et al., 2026a). On-policy methods improve the model’s compatibility, but do not guarantee exposure to the behaviors that post-training aims to introduce. 3 Method In this section, we present how to derive and implement CIG-SFT, an information-theoretic framework for selectively learning from fixed demonstrations. Our central principle is to separate two questions that are conflated in conventional token-weighted SFT: (i) how much task-specific information does a demonstrated behavior contain, and (i) given this information, how far should the current policy move toward that behavior? CIG-SFT answers the first question through pointwise conditional information gain under a frozen reference model, and the second through a KL-regularized projection of the current policy. We further show that the resulting projected update can be realized exactly, at the logit-gradient level, by a scalar-weighted SFT objective. Figure 1: Overview of CIG-SFT. A frozen initial policy estimates token-level conditional information gain from fixed demonstrations. The information score and current policy jointly define an information-proximal target distribution, which is converted into a gradient-equivalent token weight for supervised optimization. During training, we update only the current policy. 3.1 Preliminaries Let =(x,y)D=\(x,y)\ denote a dataset of fixed demonstrations, where x is the task input and y=(y1,…,yT)y=(y_1,…,y_T) is the response with T tokens. The preceding context for the t-th target token is denoted as y<ty_<t. We use π0 _0 to denote the frozen reference policy and πθ _θ the trainable policy initialized from the same pretrained model. The objective of standard supervised fine-tuning is to minimize: ℒSFT=−1N∑tmtlogπθ(yt∣x,y<t),L_SFT=- 1NΣ _tm_t _θ(y_t x,y_<t), (1) where mt∈0,1m_t∈\0,1\ masks tokens participating in training and N=∑tmtN= _tm_t. Dynamic or selective SFT methods generalize eq. (1) by assigning different learning strengths to individual target tokens: ℒdyn=−1N∑tmtsg[αt]logπθ(yt∣x,y<t)+βℛ(θ),L_dyn=- 1NΣ _tm_t\,sg[ _t] _θ(y_t x,y_<t)+ (θ), (2) where sg[⋅]sg[·] denotes stop-gradient, ℛ(θ)R(θ) optionally regularizes the updated policy with strength β, and αt _t controls token-level supervision that is instantiated differently across methods: αt=πθ(yt∣x,y<t)u(π0(⋅∣x,y<t),yt)πθ(yt∣x,y<t)γtπθ(yt∣x,y<t)ℛ(θ)=000DKL(π0∥πθ)DFTPriFTIDFTASFT _t= \ aligned & _θ(y_t x,y_<t)\\[3.0pt] &u( _0(· x,y_<t),y_t)\\[3.0pt] & _θ(y_t x,y_<t) _t\\[3.0pt] & _θ(y_t x,y_<t) aligned . (θ)= \ aligned &0\\[3.0pt] &0\\[3.0pt] &0\\[3.0pt] &D_KL ( _0\| _θ ) aligned . aligned &DFT\\[3.0pt] &PriFT\\[3.0pt] &IDFT\\[3.0pt] &ASFT aligned (3) where u(⋅)u(·) is a prior-support function computed from the frozen LLM, while γt _t is a distribution-compatibility coefficient determined by the target-token probability and the predictive uncertainty of the current policy. Despite their different instantiations, these methods share a common principle: token-level supervision is primarily determined by full-context model support, which does not disentangle the contribution of the task input x from that of the preceding context y<ty_<t. 3.2 Pointwise Conditional Information Gain To disentangle the influence of task evidence, we compare the same demonstrated token under task-conditioned and task-free contexts. Using the frozen reference policy π0 _0, we define p0=π0(yt∣x,y<t),p0−=π0(yt∣∅,y<t).p_0= _0(y_t x,y_<t), p_0^-= _0(y_t ,y_<t). (4) where ∅ denotes an empty task input under the same input format. The only difference between p0p_0 and p0−p_0^- is the availability of task information x. This contrast is motivated by conditional mutual information (Cover and Thomas, 2006). For random variables X, YtY_t, and Y<tY_<t, I(X;Yt∣Y<t)=[logp(Yt∣X,Y<t)p(Yt∣Y<t)].I(X;Y_t Y_<t)=E [ p(Y_t X,Y_<t)p(Y_t Y_<t) ]. (5) Accordingly, inspired by the quantity inside the expectation of eq. (5), that is, the pointwise conditional information gain, we define our information gain iti_t for each demonstrated token as it=logp0−logp0−=logπ0(yt∣x,y<t)π0(yt∣∅,y<t).i_t= p_0- p_0^-= _0(y_t x,y_<t) _0(y_t ,y_<t). (6) Since the exact marginal π0(yt|y<t) _0(y_t|y_<t) is intractable, we use the task-free prediction π0(yt∣∅,y<t) _0(y_t ,y_<t) as an approximation (Xu et al., 2020; Ethayarajh et al., 2022). In information theory, the surprisal of an event with probability p is −logp- p, which quantifies how unexpected the event is (Shannon, 1948). Therefore, iti_t measures the reduction in target-token surprisal from observing x beyond what the preceding context already explains. We normalize this reduction by the task-free surprisal as ct=clip(it−logp0−+ϵ,0,1),c_t=clip ( i_t- p_0^-+ε,0,1 ), (7) where ϵ>0ε>0 is a small constant for numerical stability when the task-free surprisal approaches zero. ctc_t quantifies the ratio of the target token’s task-free surprisal explained by the task input: ct=0c_t=0 indicates no positive surprisal reduction, while ctc_t approaching 11 indicates that the task input explains nearly all of the surprisal remaining after conditioning on the preceding context. Since ctc_t depends only on the frozen reference policy, it can be computed once and cached before fine-tuning. 3.3 Information-Proximal Projection Conditional information gain quantifies task-side evidence for a demonstrated decision, but does not specify how far the current policy should move toward that decision. Motivated by reward-weighted policy optimization (Peters and Schaal, 2007; Wu et al., 2026; Zhu et al., 2025), we interpret the normalized information score ctc_t as a token-level utility for the demonstrated decision: rt(v)=ct 1[v=yt],yt∈,∀v∈,r_t(v)=c_t\, 1[v=y_t], y_t , ∀ v , (8) where V denotes the LLM vocabulary. Instead of directly maximizing this utility, we constrain the update to remain close to the current policy, following regularized policy optimization (Peters et al., 2010; Schulman et al., 2015). Let pθ=πθ(⋅∣x,y<t)p_θ= _θ(· x,y_<t), we obtain the projected target distribution as qt⋆=argmaxq∈Δ()v∼q[rt(v)]−λDKL(q∥pθ),q_t = _q∈ (V) \E_v q[r_t(v)]-λ D_KL (q\|p_θ ) \, (9) where Δ()=q:→[0,1]|∑v∈q(v)=1 (V)= \q:V→[0,1]\; |\; _v q(v)=1 \ denotes the probability simplex over the vocabulary and λ>0λ>0 controls the strength of the proximity constraint. The optimization in eq. (9) admits a closed-form solution through exponential tilting of the current policy (Peters et al., 2010), qt⋆(v)=pθ(v)exp(rt(v)/λ)Zt.q_t (v)= p_θ(v) \! (r_t(v)/λ )Z_t. (10) Since rt(v)r_t(v) is nonzero only for the demonstrated token yty_t, the normalization constant simplifies to Zt Z_t =∑v∈pθ(v)exp(rt(v)/λ)=pθ(yt)ect/λ+∑v∈∖ytpθ(v) =Σ _v p_θ(v) \! (r_t(v)/λ )=p_θ(y_t)e^c_t/λ+Σ _v y_tp_θ(v) (11) =pθ(yt)ect/λ+1−pθ(yt)=1+pθ(yt)(ect/λ−1), =p_θ(y_t)e^c_t/λ+1-p_θ(y_t)=1+p_θ(y_t) (e^c_t/λ-1 ), and the projected probabilities assigned to the target yty_t and non-target token v≠ytv≠ y_t become qt⋆(yt)=pθ(yt)ect/λ1+pθ(yt)(ect/λ−1),andqt⋆(v)=pθ(v)1+pθ(yt)(ect/λ−1).q_t (y_t)= p_θ(y_t)e^c_t/λ1+p_θ(y_t) (e^c_t/λ-1 ), \;\;q_t (v)= p_θ(v)1+p_θ(y_t) (e^c_t/λ-1 ). (12) The projection adjusts the current policy according to the task-conditioned information score. When ct=0c_t=0, qt⋆q_t reduces exactly to the current policy, while increasing ctc_t or decreasing λ produces a larger shift toward the demonstrated decision that remaining proximal to the current policy. 3.4 Gradient-Equivalent Objective Realization The projection yields the optimal target distribution qt⋆q_t under eq. (9), which naturally serves as a soft supervision target for updating the trainable policy. We minimize the cross-entropy loss: ℒtq=−∑v∈sg[qt⋆(v)]logpθ(v).L_t^q=-Σ _v sg\! [q_t (v) ] p_θ(v). (13) Taking the gradient with respect to θ, we have ∇θℒtq=−∑v∈qt⋆(v)∇θlogpθ(v). _θL_t^q=-Σ _v q_t (v) _θ p_θ(v). (14) Since pθ(v)p_θ(v) is a normalized discrete distribution, there is ∑v∈pθ(v)∇θlogpθ(v)=∑v∈∇θpθ(v)=0,Σ _v p_θ(v) _θ p_θ(v)=Σ _v _θp_θ(v)=0, (15) and hence ∑v∈∖ytpθ(v)∇θlogpθ(v)=−pθ(yt)∇θlogpθ(yt).Σ _v y_tp_θ(v) _θ p_θ(v)=-p_θ(y_t) _θ p_θ(y_t). (16) Using equations (12) and (16), we obtain ∇θℒtq=−[pθ(yt)(ect/λ−1)1+pθ(yt)(ect/λ−1)]∇θlogpθ(yt). _θL_t^q=- [ p_θ(y_t)(e^c_t/λ-1)1+p_θ(y_t)(e^c_t/λ-1) ] _θ p_θ(y_t). (17) This gradient is exactly reproduced by the weighted hard-target objective ℒtw=−sg(wt)logpθ(yt),as∇θℒtq=∇θℒtw,L_t^w=-sg(w_t) p_θ(y_t),\ as\;\; _θL_t^q= _θL_t^w, (18) where the token-wise update coefficient wtw_t is wt=pθ(yt)(ect/λ−1)1+pθ(yt)(ect/λ−1).w_t= p_θ(y_t) (e^c_t/λ-1 )1+p_θ(y_t) (e^c_t/λ-1 ). (19) It is worth noting that equations (13) and (18) are not generally equal in value. They are gradient-equivalent with respect to θ when their targets or weights are detached. Aggregating over all supervised tokens, we get the final training objective with equivalent gradients as ℒCIG-SFT=−1N∑tmtsg(wt)logπθ(yt∣x,y<t),where N=∑tmt.L_CIG -SFT=- 1NΣ _tm_t\,sg(w_t) _θ(y_t x,y_<t), \,N=Σ _tm_t. (20) 4 Experiments 4.1 Experimental Setup Models and datasets. We evaluate CIG-SFT across mathematical reasoning and code generation. For mathematical reasoning, we use Qwen2.5-Math-1.5B and Qwen2.5-Math-7B (Yang et al., 2024), trained on a deduplicated 30K subset of NuminaMath-CoT (Li et al., 2024b). We evaluate on MATH500 (Lightman et al., 2024), Minerva Math (Lewkowycz et al., 2022), OlympiadBench (He et al., 2024), AIME 2024, AIME 2025, and AMC 202311 1 https://maa.org/maa-invitational-competitions/. For code generation, we use Qwen2.5-Coder-3B (Hui et al., 2024), trained on a 10K subset of the code training pool, and evaluate on HumanEval+ and MBPP+ (Liu et al., 2023), LiveCodeBench v5/v6 (Jain et al., 2025), BigCodeBench-Full (Zhuo et al., 2025), and DS-1000 (Lai et al., 2023). Baselines. We compare with representative supervised fine-tuning, offline preference optimization, and on-policy reinforcement learning methods. For supervised fine-tuning, we include standard SFT and several recent adaptive objectives: Dynamic fine-tuning (DFT) (Wu et al., 2025), which reweights target tokens according to their probabilities under the current policy; In-distribution fine-tuning (IDFT) (Zhang et al., 2026), which combines target-token probability with predictive uncertainty to adapt the token-level learning signal; Anchored Supervised Fine-Tuning (ASFT) (Zhu et al., 2025), which augments probability-based reweighting with distributional anchoring; and PriFT (Wang et al., 2026b), for which we evaluate both PriFT-prob and PriFT-mass using token statistics from a frozen pretrained reference model. We additionally compare against offline preference optimization methods, including Direct Preference Optimization (DPO) (Rafailov et al., 2023) and SimPO (Meng et al., 2024), as well as on-policy reinforcement learning methods, including Group Relative Policy Optimization (GRPO) (Shao et al., 2024) and DAPO (Yu et al., 2026). Training and evaluation settings. For mathematical reasoning, all supervised fine-tuning methods are trained for one epoch using AdamW (Loshchilov and Hutter, 2017), cosine learning-rate decay, a warm-up ratio of 0.10.1, a learning rate of 5×10−55× 10^-5, a global batch size of 128128, and a maximum sequence length of 20482048. For CIG-SFT, we set λ=0.3λ=0.3, which controls the strength of the conditional-information-gain signal when constructing the projected token-level target distribution. We keep this value fixed across all experiments unless otherwise specified. For evaluation, we sample 1616 responses per problem at temperature 1.01.0 with a maximum generation length of 40964096. We report Avg@16, the average correctness percentage across the 1616 sampled responses, and Pass@16, the fraction of problems for which at least one of the 1616 responses is correct. 4.2 Main Results Mathematical reasoning. Tables 1 and 2 summarize the mathematical reasoning results. CIG-SFT consistently achieves the strongest aggregate performance across both model scales. On Qwen2.5-Math-1.5B, CIG-SFT reaches 31.5%31.5\% Avg@16 and 58.6%58.6\% Pass@16, improving over standard SFT by 17.7/16.317.7/16.3 and over DFT by 5.2/9.65.2/9.6, respectively. It also outperforms the strongest prior supervised baseline, PriFT-prob, by 1.2 points in Avg@16 and 2.4 points in Pass@16. The same trend becomes more pronounced on Qwen2.5-Math-7B, where CIG-SFT achieves 38.5% Avg@16 and 63.5% Pass@16, exceeding PriFT-mass by 2.1/0.32.1/0.3 points and DFT by 6.8/12.46.8/12.4 points. The improvements are distributed across benchmarks of different difficulty, with particularly clear gains on MATH500, OlympiadBench, and AIME. Table 1: Mathematical reasoning results with Qwen2.5-Math-7B. We report Avg@16 / Pass@16. Method Average MATH500 Minerva Olympiad AIME24 AIME25 AMC23 Base 17.2/55.617.2/55.6 39.4/88.239.4/88.2 11.7/46.711.7/46.7 14.8/57.214.8/57.2 8.5/46.7 08.5/46.7 2.9/20.0 02.9/20.0 25.8/75.025.8/75.0 SFT 19.7/49.419.7/49.4 53.0/87.253.0/87.2 16.9/48.916.9/48.9 19.2/55.119.2/55.1 2.3/13.3 02.3/13.3 1.7/16.7 01.7/16.7 25.3/75.025.3/75.0 DFT 31.7/51.131.7/51.1 68.9/86.068.9/86.0 23.9/43.423.9/43.4 33.5/56.333.5/56.3 11.7/23.311.7/23.3 5.8/20.0 05.8/20.0 46.6/77.546.6/77.5 IDFT 30.1/49.930.1/49.9 69.3/85.069.3/85.0 25.7/42.625.7/42.6 34.1/53.534.1/53.5 6.7/23.3 06.7/23.3 3.1/20.0 03.1/20.0 41.7/87.541.7/87.5 ASFT 30.3/60.130.3/60.1 67.8/92.2¯67.8/ 92.2 27.2/53.7¯27.2/ 53.7 33.7/64.6¯33.7/ 64.6 8.1/33.3 08.1/33.3 3.1/26.7 03.1/26.7 41.7/90.041.7/90.0 PriFT-prob 35.8/62.535.8/62.5 72.3/92.2¯72.3/ 92.2 29.6¯/54.8 29.6/54.8 36.8/65.336.8/65.3 14.4/43.314.4/43.3 7.5¯/26.7 0 7.5/26.7 53.9/92.5¯53.9/ 92.5 PriFT-mass 36.4¯/63.2¯ 36.4/ 63.2 74.0¯/91.8 74.0/91.8 27.8/52.927.8/52.9 36.9¯/61.8 36.9/61.8 15.0¯/46.7 15.0/46.7 7.5¯/33.3 0 7.5/33.3 57.3/92.557.3/92.5 CIG-SFT 38.5/63.538.5/63.5 75.2/92.475.2/92.4 30.3/53.330.3/53.3 39.4/64.339.4/64.3 19.8/53.519.8/53.5 10.6/30.0¯10.6/ 30.0 55.5¯/87.5 55.5/87.5 Table 2: Mathematical reasoning results with Qwen2.5-Math-1.5B. We report Avg@16 / Pass@16. Method Average MATH500 Minerva Olympiad AIME24 AIME25 AMC23 Base 16.5/51.616.5/51.6 39.0/86.039.0/86.0 10.2/41.210.2/41.2 19.2/52.119.2/52.1 4.8/26.7 04.8/26.7 1.9/23.31.9/23.3 23.6/80.023.6/80.0 Off-policy Reinforcement Learning DPO 17.3/49.717.3/49.7 40.0/86.040.0/86.0 11.1/44.111.1/44.1 19.7/55.019.7/55.0 4.2/33.3 04.2/33.3 0.8/10.00.8/10.0 28.0/80.028.0/80.0 SimPO 17.0/50.917.0/50.9 41.2/86.041.2/86.0 10.9/43.810.9/43.8 19.6/52.419.6/52.4 4.2/26.7 04.2/26.7 1.7/16.71.7/16.7 24.2/80.024.2/80.0 On-policy Reinforcement Learning GRPO 28.0/56.228.0/56.2 63.2/90.463.2/90.4 22.7/51.1¯22.7/ 51.1 29.1/62.129.1/62.1 7.5/30.0 07.5/30.0 4.8/23.34.8/23.3 40.9/80.040.9/80.0 DAPO 30.5¯/57.7¯ 30.5/ 57.7 67.2¯/91.6 67.2/91.6 24.9/52.524.9/52.5 30.2/60.7¯30.2/ 60.7 9.6/23.3 09.6/23.3 5.8/33.35.8/33.3 45.0/85.045.0/85.0 Supervised Fine-Tuning SFT 13.8/42.313.8/42.3 41.5/81.841.5/81.8 12.1/40.412.1/40.4 12.0/45.612.0/45.6 0.6/10.0 00.6/10.0 0.2/3.30.2/ 03.3 16.4/72.516.4/72.5 DFT 26.3/49.026.3/49.0 64.0/85.864.0/85.8 20.8/41.520.8/41.5 26.9/53.326.9/53.3 5.4/13.3 05.4/13.3 3.8/20.03.8/20.0 37.0/80.037.0/80.0 IDFT 25.2/48.925.2/48.9 63.1/82.863.1/82.8 20.9/41.920.9/41.9 27.0/53.527.0/53.5 5.4/23.3 05.4/23.3 4.2/26.74.2/26.7 30.8/65.030.8/65.0 ASFT 27.6/54.827.6/54.8 63.8/90.063.8/90.0 21.2/50.421.2/50.4 28.5/59.028.5/59.0 7.9/23.3 07.9/23.3 4.2/23.34.2/23.3 39.7/82.539.7/82.5 PriFT-prob 30.3/56.230.3/56.2 66.6/90.8¯66.6/ 90.8 22.8/49.622.8/49.6 30.4/59.330.4/59.3 10.0¯/30.0 10.0/30.0 5.2/20.05.2/20.0 46.6/87.5¯46.6/ 87.5 PriFT-mass 30.2/54.130.2/54.1 66.2/90.466.2/90.4 21.7/49.621.7/49.6 31.4¯/57.2 31.4/57.2 10.0/36.7¯10.0/ 36.7 5.4¯/13.3 5.4/13.3 46.6/77.546.6/77.5 CIG-SFT 31.5/58.631.5/58.6 68.4/89.668.4/89.6 24.1¯/48.5 24.1/48.5 32.1/59.432.1/59.4 9.4/40.0 09.4/40.0 5.2/26.7¯5.2/ 26.7 50.0/87.550.0/87.5 Comparison with reinforcement learning. On Qwen2.5-Math-1.5B, CIG-SFT also compares favorably with reinforcement-learning baselines under the same evaluation protocol. It surpasses GRPO by 3.5/2.43.5/2.4 and DAPO by 1.0/0.91.0/0.9, while substantially outperforming the offline preference-optimization baselines DPO and SimPO. Notably, CIG-SFT obtains these results using only fixed demonstrations, without online rollouts or policy-generated training trajectories. Code generation. Table 3 shows that CIG-SFT’s advantage extends beyond mathematical reasoning. On Qwen2.5-Coder-3B, CIG-SFT achieves the best overall performance among the supervised methods compared, reaching 35.4%35.4\% Avg@16 and 50.5%50.5\% Pass@16. This improves over SFT by 2.6/1.52.6/1.5, DFT by 1.2/1.21.2/1.2, and the strongest baseline in aggregate performance by 1.1/0.81.1/0.8. The gains span HumanEval+, MBPP+, LiveCodeBench, BigCodeBench, and DS-1000, suggesting that the benefit of conditional-information-guided supervision is not specific to mathematical reasoning. Table 3: Code-generation results with Qwen2.5-Coder-3B. We report Avg@16 / Pass@16. Method Average HumanEval+ MBPP+ LCB v5 LCB v6 BigCode DS-1000 Base 30.9/49.430.9/49.4 40.9/67.140.9/67.1 49.5/74.149.5/74.1 18.0/28.318.0/28.3 17.1/26.917.1/26.9 35.9/57.235.9/57.2 23.7/42.923.7/42.9 SFT 32.8/49.032.8/49.0 44.8/62.244.8/62.2 53.2/74.3¯53.2/ 74.3 18.6/28.218.6/28.2 17.7/26.917.7/26.9 38.2/59.0¯38.2/ 59.0 24.2/43.224.2/43.2 DFT 34.2/49.334.2/49.3 43.8/62.243.8/62.2 54.6¯/73.3 54.6/73.3 21.1/29.7¯21.1/ 29.7 20.0/27.820.0/27.8 39.9¯/58.5 39.9/58.5 26.2¯/44.7¯ 26.2/ 44.7 ASFT 34.3¯/49.7¯ 34.3/ 49.7 45.6/65.2¯45.6/ 65.2 54.2/73.354.2/73.3 21.1¯/29.3 21.1/29.3 20.1¯/27.9¯ 20.1/ 27.9 39.5/59.039.5/59.0 25.2/43.725.2/43.7 PriFT-prob 33.6/49.333.6/49.3 45.6¯/64.0 45.6/64.0 52.6/73.552.6/73.5 19.9/29.119.9/29.1 19.1/27.819.1/27.8 39.0/56.939.0/56.9 25.6/44.625.6/44.6 CIG-SFT 35.4/50.535.4/50.5 46.5/64.046.5/64.0 55.0/74.155.0/74.1 22.0/30.522.0/30.5 21.1/29.021.1/29.0 41.0/59.341.0/59.3 26.9/46.426.9/46.4 Table 4: Ablations on mathematical reasoning with Qwen2.5-Math-1.5B. Method Average MATH500 Minerva Olympiad AIME24 AIME25 AMC23 SFT 13.8/42.313.8/42.3 41.5/81.841.5/81.8 12.1/40.412.1/40.4 12.0/45.612.0/45.6 0.6/10.0 00.6/10.0 0.2/3.30.2/ 03.3 16.4/72.516.4/72.5 CIG-SFT 31.5/58.631.5/58.6 68.4/89.668.4/89.6 24.1/48.524.1/48.5 32.1/59.432.1/59.4 9.4/40.0 09.4/40.0 5.2/26.75.2/26.7 50.0/87.550.0/87.5 w/o ctc_t 21.9/48.421.9/48.4 55.4/88.655.4/88.6 19.2/46.019.2/46.0 22.1/54.722.1/54.7 4.0/16.7 04.0/16.7 1.4/6.71.4/ 06.7 29.4/77.529.4/77.5 w/o DKLD_KL 28.4/55.128.4/55.1 62.1/90.462.1/90.4 19.3/46.019.3/46.0 29.2/59.129.2/59.1 9.6/30.0 09.6/30.0 4.4/20.04.4/20.0 45.5/85.045.5/85.0 Online ctc_t 6.9/21.9 06.9/21.9 21.1/54.621.1/54.6 6.6/22.4 06.6/22.4 3.3/12.6 03.3/12.6 0.2/3.3 00.2/ 03.3 0.2/3.30.2/ 03.3 9.8/24.4 09.8/24.4 Frozen DKLD_KL 30.5/55.4 30.5/55.4 64.7/89.464.7/89.4 22.7/47.8 22.7/47.8 31.2/60.1 31.2/60.1 11.9/30.011.9/ 30.0 5.6/20.05.6/ 20.0 46.7/85.0 46.7/85.0 Both variants 15.0/35.1 15.0/35.1 43.8/76.843.8/76.8 11.8/30.1 11.8/30.1 7.6/23.7 07.6/23.7 1.9/13.3 01.9/ 13.3 0.4/6.70.4/ 06.7 24.4/60.0 24.4/60.0 Ablation study. Table 4 examines both component removal and alternative design choices on Qwen2.5-Math-1.5B. Removing the information ratio (w/o ctc_t, eq. 7) causes the largest degradation among the component ablations, reducing Avg@16//Pass@16 from 31.5/58.631.5/58.6 to 21.9/48.421.9/48.4, while removing the KL-proximal constraint (w/o DKLD_KL, eq. 9) yields a smaller but consistent drop to 28.4/55.128.4/55.1. These results indicate that task-conditioned information provides the primary supervision signal, while the proximal projection further improves how this information is incorporated into the current policy. The design variants further highlight the distinct roles of the frozen reference and current policies. Computing ctc_t online with the evolving policy (Online ctc_t) substantially degrades performance to 6.9/21.96.9/21.9, suggesting that decoupling information estimation from model optimization is important for obtaining a stable attribution signal. By contrast, replacing the current-policy KL reference with the frozen reference (Frozen DKLD_KL) causes only a moderate decline, although it remains inferior to the proposed current-policy projection. Applying both modifications (Both variants) together reaches only 15.0/35.115.0/35.1. Overall, the results support our design of using a frozen policy for offline information attribution and the evolving policy for proximal target construction. 4.3 Analysis Figure 2: Analysis of training scale. Training scale. We study the effect of training-set size by varying the number from 1K to 100K. CIG-SFT consistently achieves the strongest aggregate performance across all data scales, while exhibiting substantially higher data efficiency than the competing objectives. Notably, with only 1K demonstrations, CIG-SFT already reaches 28.9%28.9\% Avg@16 and 55.9%55.9\% Pass@16, outperforming SFT, DFT, ASFT, and IDFT even when they are trained on 100K demonstrations, whose best results are 26.8/53.6%26.8/53.6\%. Moreover, CIG-SFT remains remarkably stable as the training set expands: Avg@16 stays within [28.9,31.5]%[28.9,31.5]\% and Pass@16 within [55.2,58.5]%[55.2,58.5]\% from 1K to 100K demonstrations. In contrast, standard SFT and DFT rely more strongly on increasing the amount of training data, while several dynamic SFT baselines exhibit non-monotonic behavior as the dataset grows. These results suggest that CIG-SFT extracts useful supervision more efficiently from fixed demonstrations and retains its advantage across both low-data and large-scale fine-tuning regimes. Figure 3: Analysis of learning rate and proximity strength λ. Figure 4: Analysis of the training dynamics. Learning rate. We examine sensitivity to the learning rate over 1×10−5,2×10−5,5×10−5,1×10−4,2×10−4\1× 10^-5,2× 10^-5,5× 10^-5,1× 10^-4,2× 10^-4\. As shown in Figure 3, CIG-SFT consistently outperforms both SFT and DFT across the entire range on Avg@16 and Pass@16. Its performance remains relatively stable from 2×10−52× 10^-5 to 1×10−41× 10^-4, with degradation occurring mainly at the largest learning rate. We use 5×10−55× 10^-5 in the main experiments, which provides a favorable balance between Avg@16 and Pass@16. Proximity strength. We vary λ, which controls how strongly the projected target is constrained to remain close to the current policy. When λ is nearly zero, the proximity constraint effectively vanishes and performance drops substantially, highlighting the importance of controlling the information-induced policy shift. Once λ reaches a moderate value, Avg@16 remains stable over a broad range, with only minor fluctuations from 0.30.3 to 1010. Pass@16 varies more but remains consistently strong in the same region. We use λ=0.3λ=0.3 as the default setting in the main experiments. Training dynamics. Figure 4 examines how different objectives allocate supervision and translate it into parameter updates. In the left panel, SFT assigns unit weight to every target token, while DFT maintains a relatively high average weight of approximately 0.80.8-0.860.86. In contrast, CIG-SFT consistently assigns substantially smaller average weights (∼0.4 0.4-0.460.46), indicating more selective allocation of learning pressure across demonstrated tokens. To quantify the resulting optimization magnitude, we report the relative parameter update at optimizer step t, ∥Δθt∥/∥θt∥, _t / _t , where Δθt _t denotes the actual parameter change produced by the t-th optimizer step. CIG-SFT induces consistently smaller relative parameter updates than DFT and generally smaller updates than standard SFT, while achieving stronger downstream performance. This suggests that its gains arise from more selective and better-targeted supervision rather than from more aggressive parameter movement. 5 Conclusion We presented CIG-SFT, an information-theoretic framework that uses task-conditioned information to guide token-level supervision. It combines conditional information attribution with policy-aware proximal adaptation, yielding a simple gradient-equivalent objective. Across mathematical reasoning and code generation, CIG-SFT consistently outperforms recent dynamic fine-tuning methods while remaining competitive with reinforcement-learning baselines. The gains persist across learning rates and data scale, and strong performance can be obtained with substantially fewer demonstrations. Analysis shows that we assign lower average supervision weights and induce smaller relative parameter updates, suggesting that its benefit comes from more selective allocation of learning pressure rather than more aggressive fitting. Overall, these results highlight task-conditioned information as a useful complement to model-side compatibility for learning from fixed demonstrations. AI Use Statement Generative AI tools are used as research and writing assistants during the development of this work. Specifically, they are used to assist with checking and reorganizing mathematical derivations, discussing experimental designs and interpretations, identifying potentially relevant literature, improving the clarity and structure of the manuscript, and assisting with the design of scientific figures. All AI-assisted technical content is independently reviewed and verified by the authors. The authors check mathematical derivations, manually verify cited literature against the original sources, and validate all reported experimental results and conclusions against the authors’ own experiments. The authors take full responsibility for the final content of the paper, including all claims, equations, figures, and experimental results. Reproducibility Statement We provide the information necessary to reproduce CIG-SFT and its evaluation throughout the paper and supplementary materials. Section 3 specifies the complete training objective and its derivation, including the conditional information score, information-proximal projection, and gradient-equivalent objective. We report the models, datasets, baselines, evaluation metrics, and main hyperparameters used in our experiments. The training and evaluation code, together with the configuration files required to reproduce the reported experiments, is available in our repository. References Agarwal et al. (2024) R. Agarwal, N. Vieillard, Y. Zhou, P. Stanczyk, S. Ramos Garea, M. Geist, and O. Bachem On-policy distillation of language models: learning from self-generated mistakes. In International Conference on Learning Representations, Vol. 2024, p. 21246–21263. Cited by: §1. Brown et al. (2020) T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems 33, p. 1877–1901. Cited by: §1. Chowdhery et al. (2023) A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. Palm: scaling language modeling with pathways. Journal of machine learning research 24 (240), p. 1–113. Cited by: §1. Chu et al. (2025) T. Chu, Y. Zhai, J. Yang, S. Tong, S. Xie, D. Schuurmans, Q. V. Le, S. Levine, and Y. Ma Sft memorizes, rl generalizes: a comparative study of foundation model post-training. arXiv preprint arXiv:2501.17161. Cited by: §1, §2. Chung et al. (2024) H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, Y. Li, X. Wang, M. Dehghani, S. Brahma, et al. Scaling instruction-finetuned language models. Journal of machine learning research 25 (70), p. 1–53. Cited by: §1, §2. Cover and Thomas (2006) T. M. Cover and J. A. Thomas Elements of information theory, 2/e. Cited by: §1, §3.2. Cui et al. (2025) G. Cui, Y. Zhang, J. Chen, L. Yuan, Z. Wang, Y. Zuo, H. Li, Y. Fan, H. Chen, W. Chen, et al. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505.22617. Cited by: §2. Ethayarajh et al. (2022) K. Ethayarajh, Y. Choi, and S. Swayamdipta Understanding dataset difficulty with V-usable information. In Proceedings of the 39th International Conference on Machine Learning, K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato (Eds.), Proceedings of Machine Learning Research, Vol. 162, p. 5988–6008. External Links: Link Cited by: §3.2. Guo et al. (2025) D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Bi, et al. DeepSeek-r1 incentivizes reasoning in llms through reinforcement learning. Nature 645 (8081), p. 633–638. Cited by: §1, §1, §2. He et al. (2024) C. He, R. Luo, Y. Bai, S. Hu, Z. Thai, J. Shen, J. Hu, X. Han, Y. Huang, Y. Zhang, et al. Olympiadbench: a challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 3828–3850. Cited by: §4.1. Hui et al. (2024) B. Hui, J. Yang, Z. Cui, J. Yang, D. Liu, L. Zhang, T. Liu, J. Zhang, B. Yu, K. Lu, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186. Cited by: §4.1. Jain et al. (2025) N. Jain, A. Gu, W. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica Livecodebench: holistic and contamination free evaluation of large language models for code. In International Conference on Learning Representations, Vol. 2025, p. 58791–58831. Cited by: §4.1. Kotha et al. (2024) S. Kotha, J. Springer, and A. Raghunathan Understanding catastrophic forgetting in language models via implicit inference. In International Conference on Learning Representations, Vol. 2024, p. 24110–24139. Cited by: §1, §2. Lai et al. (2023) Y. Lai, C. Li, Y. Wang, T. Zhang, R. Zhong, L. Zettlemoyer, W. Yih, D. Fried, S. Wang, and T. Yu DS-1000: a natural and reliable benchmark for data science code generation. In International Conference on Machine Learning, p. 18319–18345. Cited by: §4.1. Lewkowycz et al. (2022) A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ramasesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in neural information processing systems 35, p. 3843–3857. Cited by: §4.1. Li et al. (2024a) H. Li, L. Ding, M. Fang, and D. Tao Revisiting catastrophic forgetting in large language model tuning. In Findings of the Association for Computational Linguistics: EMNLP 2024, Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.), Miami, Florida, USA, p. 4297–4308. External Links: Link, Document Cited by: §1, §2. Li et al. (2024b) J. Li, E. Beeching, L. Tunstall, B. Lipkin, R. Soletskyi, S. Huang, K. Rasul, L. Yu, A. Q. Jiang, Z. Shen, et al. Numinamath: the largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository 13 (9), p. 9. Cited by: §4.1. Lightman et al. (2024) H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe Let’s verify step by step. In International Conference on Learning Representations, Vol. 2024, p. 39578–39601. Cited by: §1, §4.1. Lindley (1956) D. V. Lindley On a measure of the information provided by an experiment. The Annals of Mathematical Statistics 27 (4), p. 986–1005. Cited by: §1. Liu et al. (2023) J. Liu, C. S. Xia, Y. Wang, and L. Zhang Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in neural information processing systems 36, p. 21558–21572. Cited by: §4.1. Liu et al. (2026) T. Liu, T. Wu, R. Yang, S. Sun, J. Wang, and Y. Yang ProFit: Leveraging High-Value Signals in SFT via Probability-Guided Token Selection. arXiv preprint arXiv:2601.09195. External Links: Link Cited by: §2. Loshchilov and Hutter (2017) I. Loshchilov and F. Hutter Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. Cited by: §4.1. Meng et al. (2024) Y. Meng, M. Xia, and D. Chen Simpo: simple preference optimization with a reference-free reward. Advances in Neural Information Processing Systems 37, p. 124198–124235. Cited by: §4.1. Ouyang et al. (2022) L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems 35, p. 27730–27744. Cited by: §1, §1, §2, §2. Peters et al. (2010) J. Peters, K. Mulling, and Y. Altun Relative entropy policy search. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 24, p. 1607–1612. Cited by: §3.3, §3.3. Peters and Schaal (2007) J. Peters and S. Schaal Reinforcement learning by reward-weighted regression for operational space control. In Proceedings of the 24th international conference on Machine learning, p. 745–750. Cited by: §3.3. Rafailov et al. (2023) R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn Direct preference optimization: your language model is secretly a reward model. Advances in neural information processing systems 36, p. 53728–53741. Cited by: §1, §4.1. Sabbaghi et al. (2026) M. Sabbaghi, G. Pappas, A. Javanmard, and H. Hassani InfoSFT: Learn More and Forget Less with Information-Aware Token Weighting. arXiv preprint arXiv:2605.14967. External Links: Link Cited by: §2. Schulman et al. (2015) J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz Trust region policy optimization. In Proceedings of the 32nd International Conference on Machine Learning, F. Bach and D. Blei (Eds.), Proceedings of Machine Learning Research, Vol. 37, Lille, France, p. 1889–1897. External Links: Link Cited by: §3.3. Schulman et al. (2017) J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Cited by: §1, §1, §2. Shannon (1948) C. E. Shannon A mathematical theory of communication. The Bell system technical journal 27 (3), p. 379–423. Cited by: §1, §3.2. Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. Deepseekmath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: §2, §4.1. Snell et al. (2022) C. Snell, I. Kostrikov, Y. Su, M. Yang, and S. Levine Offline rl for natural language generation with implicit language q learning. arXiv preprint arXiv:2206.11871. Cited by: §1. Wang et al. (2026a) C. Wang, L. Wei, Y. Zhang, C. Shao, Z. Dan, W. Huang, G. Lan, and Y. Wang Targeted exploration via unified entropy control for reinforcement learning. In Findings of the Association for Computational Linguistics: ACL 2026, M. Liakata, V. P. Moreira, J. Zhang, and D. Jurgens (Eds.), San Diego, California, United States, p. 16789–16802. External Links: Link, Document, ISBN 979-8-89176-395-1 Cited by: §2. Wang et al. (2026b) K. Wang, S. Li, M. Salzmann, and P. Frossard PriFT: Prior-Support Guided Supervised Fine-Tuning. arXiv preprint arXiv:2606.09396. External Links: Link Cited by: §1, §2, §4.1. Wei et al. (2021) J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652. Cited by: §1. Wu et al. (2026) Y. Wu, Y. Zhou, Z. Zhou, Y. Peng, X. Ye, X. Hu, W. Zhu, L. Qi, M. Yang, et al. On the generalization of sft: a reinforcement learning perspective with reward rectification. In International Conference on Learning Representations, Vol. 2026, p. 27550–27571. Cited by: §3.3. Wu et al. (2025) Y. Wu, Y. Zhou, Z. Ziheng, Y. Peng, X. Ye, X. Hu, W. Zhu, L. Qi, M. Yang, and X. Yang On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification. arXiv preprint arXiv:2508.05629. External Links: Link Cited by: §1, §2, §4.1. Xu et al. (2020) Y. Xu, S. Zhao, J. Song, R. Stewart, and S. Ermon A theory of usable information under computational constraints. In International Conference on Learning Representations, External Links: Link Cited by: §3.2. Yang et al. (2024) A. Yang, B. Zhang, B. Hui, B. Gao, B. Yu, C. Li, D. Liu, J. Tu, J. Zhou, J. Lin, et al. Qwen2. 5-math technical report: toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122. Cited by: §4.1. Yu et al. (2026) Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, W. Dai, T. Fan, G. Liu, L. Liu, et al. Dapo: an open-source llm reinforcement learning system at scale. Advances in Neural Information Processing Systems 38, p. 113222–113244. Cited by: §2, §4.1. Zelikman et al. (2022) E. Zelikman, Y. Wu, J. Mu, and N. Goodman Star: bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems 35, p. 15476–15488. Cited by: §1. Zhang et al. (2026) M. Zhang, Y. Liu, S. Lin, X. Yang, Q. Dai, C. Luo, W. Jiang, P. Hou, A. Zeng, X. Geng, and B. Guo Towards On-Policy SFT: Distribution Discriminant Theory and its Applications in LLM Training. arXiv preprint arXiv:2602.12222. External Links: Link Cited by: §1, §2, §4.1. Zhao et al. (2026) S. Zhao, Z. Xie, M. Liu, J. Huang, G. Pang, F. Chen, and A. Grover Self-distilled reasoner: on-policy self-distillation for large language models. arXiv preprint arXiv:2601.18734. Cited by: §1. Zhu et al. (2025) H. Zhu, J. Su, P. Lai, R. Ma, W. Zhang, L. Yang, and G. Chen Anchored Supervised Fine-Tuning. arXiv preprint arXiv:2509.23753. External Links: Link Cited by: §1, §2, §3.3, §4.1. Zhuo et al. (2025) T. Y. Zhuo, M. C. Vu, J. Chim, H. Hu, W. Yu, R. Widyasari, I. N. B. Yusuf, H. Zhan, J. He, I. Paul, et al. Bigcodebench: benchmarking code generation with diverse function calls and complex instructions. In International Conference on Learning Representations, Vol. 2025, p. 66602–66656. Cited by: §4.1. Appendix A Appendix A.1 Case Study: Task-Conditioned Credit and Model Probability For each demonstrated answer token yty_t, the frozen base model is evaluated under the original task input x and an erased-input counterfactual ∅ , with the gold response prefix held fixed: ct=clip(logπ0(yt∣x,y<t)−logπ0(yt∣∅,y<t)−logπ0(yt∣∅,y<t)+ε,0,1).c_t=clip\! ( _0(y_t x,y_<t)- _0(y_t ,y_<t)- _0(y_t ,y_<t)+ ,0,1 ). (21) The resulting ctc_t values are detached and cached before training. We combine ctc_t with the current-policy probability to obtain wtw_t. Figure 5 qualitatively illustrates how CIG-SFT redistributes supervision compared with probability-based weighting. In Case A, task-relevant tokens such as substitute, check, the comparison operators, and the final option receive substantial task-conditioned information gain and are consequently assigned larger weights by CIG-SFT. In contrast, highly predictable continuation tokens such as is and boxed receive near-unit weights under DFT but zero task gain, and are therefore suppressed by CIG-SFT. Case B exhibits the same pattern: tokens tied to problem entities and arithmetic decisions, such as sneakers, needed, action, and charged, are emphasized, whereas generic continuation and formatting tokens are downweighted. These examples highlight the distinction between predictive support and task-conditioned information: CIG-SFT preferentially allocates supervision to decisions explained by the task input rather than to tokens that are merely easy continuations of the gold prefix. Figure 5: Token-level allocations for two representative cases. The columns show the demonstrated tokens, and the rows report SFT, DFT, task-conditioned information gain ctc_t, and update weights wtw_t. A.2 Computational Cost We characterize computational cost using both model-pass accounting and end-to-end measurements. We distinguish trainable forward passes, backward passes, and method-specific preprocessing, since their practical costs can differ substantially. For a fixed dataset D, the training cost is Ttrain()=Tfwdθ()+Tbwdθ()+Topt()+Taux().T_train(D)=T_fwd^θ(D)+T_bwd^θ(D)+T_opt(D)+T_aux(D). (22) Standard SFT requires one trainable forward-backward pass over each training batch. DFT and CIG-SFT reuse the target probabilities from this forward pass and therefore introduce only token-wise scalar operations during training. CIG-SFT additionally performs one-time offline information scoring with the frozen reference policy. Specifically, each fixed demonstration is evaluated once with the original task input and once with an empty task input: Tpreprocess=Tfwd0()+Tfwd0(−),T_preprocess=T_fwd^0(D)+T_fwd^0(D^-), (23) where −D^- denotes the task-free version of the fixed demonstrations. These passes require no gradient computation or backward pass. The resulting ctc_t are cached and reused throughout fine-tuning. We additionally profile the end-to-end cost on Qwen2.5-Math-1.5B using the same 30K-example training set as the main experiments. All methods use a global batch size of 128, micro-batch size of 4, maximum sequence length of 2048, and 233 optimizer steps. Peak GPU memory is measured using torch.cuda.max_memory_allocated(). As shown in Table 5, the online training cost of CIG-SFT is essentially identical to standard SFT, with the same peak GPU memory of 28.93 GiB. The one-time offline scoring stage takes an additional 0.48 hours, giving an end-to-end cost of approximately 2.06 hours when preprocessing is included. The cached information scores increase offline storage from 41.5 MiB to 254.2 MiB, an additional 212.7 MiB, while introducing no increase in peak training GPU memory. Table 5: Measured training cost. Method s//Step Peak GiB Wall clock (h) SFT 23.9223.92 28.9328.93 1.581.58 DFT 23.9823.98 28.9328.93 1.581.58 CIG-SFT 23.9223.92 28.9328.93 0.48+1.580.48+1.58 IDFT 25.6125.61 42.8442.84 1.681.68 ASFT 31.5131.51 40.4940.49 2.072.07 PriFT-prob 30.7330.73 31.8131.81 2.032.03 PriFT-mass 32.4532.45 31.8031.80 2.142.14 A.3 Stability Across Random Seeds Table 6: Stability analysis on Qwen2.5-Math-1.5B. Method 42 43 44 45 46 Mean ± Std. 95% CI SFT 13.813.8 13.913.9 13.913.9 14.114.1 14.314.3 14.01±0.2014.01± 0.20 [13.76,14.26][13.76,14.26] DFT 26.326.3 26.726.7 26.226.2 26.026.0 26.726.7 26.38±0.3426.38± 0.34 [25.95,26.80][25.95,26.80] IDFT 25.225.2 26.426.4 26.026.0 25.625.6 27.027.0 26.05±0.6926.05± 0.69 [25.19,26.90][25.19,26.90] PriFT-prob 30.3¯ 30.3 29.2¯ 29.2 29.3¯ 29.3 29.3¯ 29.3 29.3¯ 29.3 29.46¯±0.46 29.46± 0.46 [28.89,30.03]¯ [28.89,30.03] CIG-SFT 31.5 30.5 30.6 31.0 31.4 31.00±0.4631.00± 0.46 [30.42,31.57][30.42,31.57] We evaluate optimization stability across five random seeds (42-46) on Qwen2.5-Math-1.5B. As shown in Table 6, CIG-SFT consistently achieves the strongest performance across all seeds, ranging from 30.5%30.5\% to 31.5%31.5\% Avg@16. Its mean performance is 31.00±0.4631.00± 0.46, with a 95% confidence interval of [30.42,31.57][30.42,31.57], indicating limited sensitivity to random initialization and data-order variation. Notably, the confidence interval remains above that of the strongest competing baseline, PriFT-prob ([28.89,30.03][28.89,30.03]), suggesting that the observed improvement of CIG-SFT is robust rather than driven by a favorable random seed. A.4 Detailed Data-Scale Results Table 7: Results of Avg@16 / Pass@16 across training-set sizes. Method 1K 3K 5K 10K 30K 50K 100K Base 16.5/51.616.5/51.6 16.5/51.616.5/51.6 16.5/51.616.5/51.6 16.5/51.6¯16.5/ 51.6 16.5/51.616.5/51.6 16.5/51.616.5/51.6 16.5/51.616.5/51.6 SFT 9.3/38.49.3/38.4 11.0/38.611.0/38.6 12.8/41.212.8/41.2 13.7/43.413.7/43.4 13.8/42.313.8/42.3 14.4/42.514.4/42.5 15.1/42.715.1/42.7 DFT 16.9/45.516.9/45.5 22.8/46.222.8/46.2 22.9/46.122.9/46.1 25.2/49.8¯25.2/ 49.8 26.3/49.026.3/49.0 26.7/46.926.7/46.9 26.1/47.526.1/47.5 ASFT 24.0/53.724.0/53.7 24.6¯/53.1¯ 24.6/ 53.1 25.0¯/53.4¯ 25.0/ 53.4 26.0¯/51.0 26.0/51.0 27.6¯/54.8¯ 27.6/ 54.8 27.3¯/53.8¯ 27.3/ 53.8 26.8¯/53.6¯ 26.8/ 53.6 IDFT 26.3¯/57.0 26.3/57.0 23.6/47.623.6/47.6 24.8/45.924.8/45.9 24.4/44.424.4/44.4 25.2/48.925.2/48.9 25.7/46.125.7/46.1 26.3/45.326.3/45.3 CIG-SFT 28.9/55.9¯28.9/ 55.9 29.9/56.429.9/56.4 30.2/58.530.2/58.5 30.5/55.230.5/55.2 31.5/56.731.5/56.7 30.8/56.930.8/56.9 30.8/57.330.8/57.3 Table 7 reports the complete numerical results underlying the training-scale analysis in Figure 2. We vary the number of training demonstrations from 1K to 100K and report Avg@16/Pass@16 for each method at every scale. These detailed results complement the aggregate trends shown in the main text and enable direct comparison across individual training-set sizes. A.5 Hyperparameter Sensitivity Table 8: Detailed sensitivity results for the learning rate. Learning rate SFT DFT CIG-SFT 1×10−51× 10^-5 10.4/41.810.4/41.8 23.6/49.323.6/49.3 27.8/56.127.8/56.1 2×10−52× 10^-5 12.8/42.2¯12.8/ 42.2 24.9/45.824.9/45.8 30.3/58.630.3/58.6 5×10−55× 10^-5 13.8¯/42.3 13.8/42.3 26.3¯/49.0¯ 26.3/ 49.0 31.5/56.7¯31.5/ 56.7 1×10−41× 10^-4 14.5/42.314.5/42.3 26.5/45.626.5/45.6 30.8¯/54.3 30.8/54.3 2×10−42× 10^-4 13.4/39.113.4/39.1 21.8/42.421.8/42.4 25.8/51.125.8/51.1 Table 9: Detailed sensitivity results for the projection coefficient λ. λ 10−810^-8 0.1 0.3 0.5 0.7 1.0 2 3 5 6 8 10 Avg@16 22.422.4 29.729.7 31.531.5 31.131.1 31.531.5 31.8¯ 31.8 31.131.1 31.331.3 32.1 31.131.1 31.431.4 31.031.0 Pass@16 54.154.1 55.455.4 56.756.7 57.3¯ 57.3 57.4 55.455.4 52.952.9 54.154.1 55.755.7 54.754.7 53.953.9 54.654.6 Tables 8 and 9 provide the complete numerical results underlying the sensitivity analyses shown in Figures 3 and 4. Table 8 reports Avg@16/Pass@16 for SFT, DFT, and CIG-SFT across five learning rates, while Table 9 reports the corresponding CIG-SFT results under different values of the projection coefficient λ. These tables complement the trends summarized in the main text and provide the exact values used in the sensitivity plots. Consistent with the main-text analysis, CIG-SFT remains the strongest method across all tested learning rates. For the projection coefficient, performance drops sharply when λ is nearly zero, while remaining relatively stable over a broad range of moderate values. A.6 General Capability Retention Analysis Table 10: General-task performance. Values are raw Acc (%). Method Average MMLU-General ARC-Challenge HellaSwag PIQA Base 56.1 58.2¯ 58.2 48.5 45.2 72.672.6 SFT 55.9¯ 55.9 57.957.9 48.0¯ 48.0 44.644.6 73.0 DFT 54.954.9 57.957.9 47.147.1 42.442.4 72.072.0 IDFT 54.454.4 57.757.7 46.246.2 41.441.4 72.272.2 ASFT 55.755.7 58.3 47.547.5 44.544.5 72.672.6 CIG-SFT 55.9¯ 55.9 58.058.0 48.0¯ 48.0 44.6¯ 44.6 72.9¯ 72.9 We examine whether improving mathematical reasoning comes at the expense of the model’s general capabilities. Table 10 reports accuracy on four general-purpose benchmarks before and after fine-tuning. CIG-SFT largely preserves the base model’s performance, with the average accuracy changing only from 56.1%56.1\% to 55.9%55.9\%. The differences on individual benchmarks are also small: −0.2-0.2 on MMLU-General, −0.5-0.5 on ARC-Challenge, −0.6-0.6 on HellaSwag, and +0.3+0.3 on PIQA. In comparison, DFT and IDFT reduce the average accuracy to 54.9%54.9\% and 54.4%54.4\%, respectively, while ASFT reaches 55.7%55.7\%. These results provide additional evidence that the reasoning improvements of CIG-SFT are obtained with limited degradation of the general capabilities. A.7 In-Domain Generalization Table 11: In domain generalization performance for Qwen2.5-Math-7B. Method Avg@16 Base 32.232.2 SFT 42.442.4 DFT 50.8¯ 50.8 IDFT 49.749.7 ASFT 50.550.5 PriFT-prob 49.249.2 PriFT-mass 47.547.5 CIG-SFT 51.8 We evaluate whether the advantage of CIG-SFT extends to unseen examples drawn from the same distribution as the training data. Specifically, we randomly sample 1,000 examples from the NuminaMath-CoT data source that are excluded from the training set. As shown in Table 11, all fine-tuning methods substantially improve over the base model, confirming that they effectively adapt to the training-domain reasoning distribution. CIG-SFT achieves the best Avg@16 of 51.8%51.8\%, outperforming DFT (50.8%50.8\%), ASFT (50.5%50.5\%), and standard SFT (42.4%42.4\%). These results indicate that the gains of CIG-SFT are not restricted to out-of-distribution benchmarks: selectively allocating supervision also improves generalization to unseen examples from the training distribution.