Paper deep dive
Learning from Self-Generated Responses: Reliability--Uncertainty Self-Distillation for Reasoning
Jinhao Pan, Bowen Wei, Yuqing Zhou, Ziwei Zhu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 10/15/2026, 5:13:50 AM
Summary
The paper introduces ReliabilityāUncertainty Self-Distillation (RUSD), a method for training large language models on reasoning tasks without reference solutions or rubrics. RUSD uses the model itself as both teacher and student, generating self-attempts that are weighted by a combination of judged reliability (derived from a self-judge's score and confidence) and student predictive uncertainty. Experiments on Qwen3-4B and Qwen3-8B across eleven benchmarks show RUSD outperforms five baselines, achieving 70.47% and 76.69% average accuracy respectively.
Entities (16)
Relation Signals (15)
RUSD ā achievesaccuracyon ā Qwen3-4B
confidence 95% Ā· reaching 70.47% on Qwen3-4B
RUSD ā achievesaccuracyon ā Qwen3-8b
confidence 95% Ā· and 76.69% on Qwen3-8B
RUSD ā outperforms ā OPSD
confidence 95% Ā· RUSD outperforms all five baselines in average accuracy
RUSD ā outperforms ā SFT
confidence 95% Ā· RUSD outperforms all five baselines in average accuracy
RUSD ā outperforms ā RCSD
confidence 95% Ā· RUSD outperforms all five baselines in average accuracy
RUSD ā outperforms ā NSD
confidence 95% Ā· RUSD outperforms all five baselines in average accuracy
RUSD ā outperforms ā GRPO
confidence 95% Ā· RUSD outperforms all five baselines in average accuracy
RUSD ā evaluatedon ā MATH
confidence 90% Ā· Across eleven reasoning benchmarks... MATH
RUSD ā evaluatedon ā GPQA-Diamond
confidence 90% Ā· Across eleven reasoning benchmarks... GPQA Diamond
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Reference solutions and question-specific rubrics are not always available to guide the teacher in self-distillation. A large language model's own solution attempts offer an alternative, but their quality is difficult to assess in open-ended tasks without automatic correctness checks, and even correct attempts can involve uncertain token predictions. We propose Reliability--Uncertainty Self-Distillation (\method{}) to learn from these attempts. The same model serves as teacher and student: the teacher receives the question and completed attempt, and the student matches the teacher's token-level predictions using a question-only prompt. \method{} weights each attempt's distillation loss by additively combining judged reliability, estimated from a self-judge's score and confidence, with student predictive uncertainty. Across eleven reasoning benchmarks, \method{} outperforms all five baselines in average accuracy, reaching 70.47\% on Qwen3-4B and 76.69\% on Qwen3-8B (8.90 and 2.10 percentage points above the base models). Data and code are available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2610.19176v1
- Canonical: https://arxiv.org/abs/2610.19176v1
Trouble viewing inline? Open PDF directly ā
Full Text
84,797 characters extracted from source content.
Expand or collapse full text
Learning from Self-Generated Responses: ReliabilityāUncertainty Self-Distillation for Reasoning Jinhao Pan Bowen Wei Yuqing Zhou Ziwei Zhu Affiliation: George Mason University Affiliation: jpan23, bwei2, yzhou31, zzhu20@gmu.edu Abstract Reference solutions and question-specific rubrics are not always available to guide the teacher in self-distillation. A large language modelās own solution attempts offer an alternative, but their quality is difficult to assess in open-ended tasks without automatic correctness checks, and even correct attempts can involve uncertain token predictions. We propose ReliabilityāUncertainty Self-Distillation (RUSD) to learn from these attempts. The same model serves as teacher and student: the teacher receives the question and completed attempt, and the student matches the teacherās token-level predictions using a question-only prompt. RUSD weights each attemptās distillation loss by additively combining judged reliability, estimated from a self-judgeās score and confidence, with student predictive uncertainty. Across eleven reasoning benchmarks, RUSD outperforms all five baselines in average accuracy, reaching 70.47% on Qwen3-4B and 76.69% on Qwen3-8B (8.90 and 2.10 percentage points above the base models). Data and code are available at https://github.com/JP-25/RUSD. 1 Introduction Self-distillation trains large language models (LLMs) by using the model itself as a teacher (Zhao et al., 2026a; Gu et al., 2026). In recent approaches, the teacher receives the question together with additional teacher context, such as a reference solution; the student uses a question-only prompt and learns by matching the teacherās next-token predictions along student-generated responses. Challenge 1: Reference solutions and rubrics are not always available to guide the teacher. SDFT and OPSD supply demonstration responses (example answers) or reference solutions as teacher context (Shenfeld et al., 2026; Zhao et al., 2026a). RCSD and RGSD use question-specific rubrics, which specify what a good response should contain (Gu et al., 2026; Rezaei et al., 2026). When these solutions or rubrics are unavailable, preparing them can require expensive human annotation or additional model generation. One alternative is to learn from the modelās own generated responses without supplied reference solutions (Pei et al., 2026; Li et al., 2026). These self-generated solution attempts, however, may contain errors. Challenge 2: Assessing attempt quality is difficult in open-ended reasoning. On open-ended reasoning tasks, reliable automatic verifiers for these self-generated attempts are often unavailable (Ye et al., 2026a). In reinforcement learning with verifiable rewards (RLVR), mathematical answers can be checked against known results, and code can be run against tests (DeepSeek-AI et al., 2025). Without comparable checks or reference answers, it is difficult to determine which attempts are correct and how much to learn from them. An LLM judge, a model prompted to assess response quality, can score these attempts, but its ratings can be wrong (Zheng et al., 2023). And using a larger external judge can increase the computation or API fees needed to score training attempts (Verga et al., 2024; Xie et al., 2026). Our intuition is to learn from correct attempts generated with uncertain token predictions, so the model can produce the reasoning more confidently. We propose ReliabilityāUncertainty Self-Distillation (RUSD, Figure 1) based on this intuition. For each question, the current model generates one solution attempt. As teacher, the same model receives the question and completed attempt in its prompt to provide token-level guidance. A judge assesses the attemptās quality: high scores supported by confident judgments indicate greater judged reliability. As student, the model learns to match the teacherās guidance with only the question as its prompt. We weight each attemptās distillation loss using judged reliability and the studentās uncertainty in predicting its tokens, giving priority to attempts judged reliable and to attempts with uncertain token predictions. Our contributions are threefold: 1. We introduce RUSD, which uses one model-generated attempt as teacher context without supplied solutions or question-specific rubrics. 2. We design response weights that combine judged reliability and student uncertainty, giving greater learning priority to attempts judged reliable and to attempts with uncertain token predictions. 3. Across eleven reasoning benchmarks, RUSD outperforms all five baselines in macroaverage accuracy at both scales, improving Qwen3-4B and Qwen3-8B by 8.90 and 2.10 over the base models. External ratings, training analyses, and ablations examine its weighting signals and learning behavior. 2 Related Work Self-distillation with and without supplied references. Self-distillation trains a model to match its own predictions. In recent on-policy LLM methods, the teacher receives additional context and the student matches the teacherās token distributions along student-generated responses (Zhao et al., 2026a). SDFT and OPSD supply demonstrations or reference solutions; RCSD and RGSD supply question-specific rubrics (Shenfeld et al., 2026; Zhao et al., 2026a; Gu et al., 2026; Rezaei et al., 2026). Without supplied solutions, NSD constructs negative teacher conditions from model responses, whereas Canon and u-OPSD select a self-generated solution by majority-answer consensus across multiple responses (Pei et al., 2026; Gkountouras et al., 2026; Li et al., 2026). RUSD uses one completed attempt as teacher context, without negative conditions or answer consensus. Assessing self-generated attempts. SDPO supplies environment feedback or successful responses to the teacher, while SD-Zero trains a reviser with binary correctness feedback (HĆ¼botter et al., 2026; He et al., 2026). Such feedback requires response assessment; consensus requires rules for matching answers. RUSD estimates attempt reliability from judge score and confidence without correctness labels. Reliability controls the response weight; the teacher supplies next-token predictions. Weighting by confidence and uncertainty. LLKDās weighted variant combines teacher confidence and student uncertainty for classification (Li et al., 2025). EGRSD weights token updates using teacher entropy, PW-OPSD uses token position, and COSE uses evaluator confidence for policy updates and replay (Ke et al., 2026; Liu et al., 2026; Wei et al., 2026). RUSD combines judged reliability with student predictive uncertainty to weight each attemptās distillation loss. Appendix A gives fuller comparisons and the scope of our baselines. Figure 1: Overview of RUSD. 1 ĻĪø _Īø generates attempt y for question x. Teacher and student predict next-token distributions from the same response prefix <ty_<t under different prompts; only the teacher prompt includes the complete attempt. The teacher generates no additional response. 2 Judge score s, normalized to sĀÆ s, and confidence cJc^J give judged reliability rJr^J. 3 Normalized student entropy averaged over the response gives uncertainty uSu^S. 4 A weighted sum of reliability and uncertainty gives response weight w. 5 The weighted distillation loss updates the shared model. Numbers match Sections 3.1ā3.5; bars are schematic. 3 Method Overview. Figure 1 illustrates the RUSD workflow, and Algorithm 1 in Appendix B shows the training procedure. RUSD trains on questions =iD=\x_i\ without supplied reference solutions or question-specific rubrics. The current model ĻĪø _Īø serves as teacher and student under different prompts. The initial model ĻĪø0 _ _0 is kept frozen throughout training and serves as the judge. The current model generates one attempt, which the teacher uses as additional context to provide token-level guidance (1). The judgeās score and confidence estimate the attemptās reliability (2). We also measure the studentās uncertainty in predicting the attemptās tokens (3). These signals determine the attemptās training weight (4). The student learns to match the teacherās token distributions, with each attemptās contribution scaled by its weight (5). Table 4 in Appendix B gives the three prompts. 3.1 Using the attempt as teacher context For each sampled question ix_i, the current model ĻĪø _Īø generates one solution attempt iy_i: iā¼ĻĪø(ā ā£S(i)),y_i _Īø(Ā· _S(x_i)), (1) where SP_S is the question-only prompt. The teacher prompt Tā(i,i)P_T(x_i,y_i) contains the question and completed attempt. The prompt describes the attempt as potentially useful or misleading and asks the model to solve the problem independently using any useful information. Following on-policy self-distillation (Zhao et al., 2026a), we evaluate teacher and student next-token distributions at every position within this same sampled response. At position t, both receive the preceding response tokens i,<t=(yi,1,ā¦,yi,tā1)y_i,<t=(y_i,1,ā¦,y_i,t-1). Only the teacher prompt also contains a complete copy of iy_i, giving it access to the responseās later reasoning when predicting earlier tokens: Ļi,tSā(v) Ļ^S_i,t(v) =ĻĪøā(vā£Sā(i),i,<t), = _Īø(v _S(x_i),y_i,<t), Ļi,tTā(v) Ļ^T_i,t(v) =ĻĪøā(vā£Tā(i,i),i,<t), = _Īø(v _T(x_i,y_i),y_i,<t), (2) where vāv is a vocabulary token, and Ļi,tSĻ^S_i,t and Ļi,tTĻ^T_i,t are the student and teacher next-token distributions, respectively. 3.2 Estimating reliability from self-judgment The judge ĻĪø0 _ _0 assesses each completed attempt to provide a quality signal for response weighting; the teacher provides token distributions for the student to match. For efficiency, we reuse the initial model as judge rather than rely on a larger external model. Given the question and attempt, the judge evaluates logical validity, internal consistency, factual plausibility, relevance, and completeness using one five-level rubric shared across questions (Table 4, Appendix B). For questionāattempt pair i, let āi,kJ ^J_i,k denote the judgeās logit for rating kā1,ā¦,5kā\1,ā¦,5\; the superscript JJ labels judge quantities. The judge is prompted to return one rating digit. We read its five rating-token logits and normalize them into a distribution, with j indexing the possible ratings in the denominator: ĻiJā(k)=expā”(āi,kJ)āj=15expā”(āi,jJ).Ļ^J_i(k)= ( ^J_i,k) _j=1^5 ( ^J_i,j). (3) Judge score. We take the judgeās most probable rating as the quality score: 1 means fundamentally incorrect, and 5 means fully correct and well justified. We rescale the score to [0,1][0,1] for use in the response weight: si=argā”maxkā1,ā¦,5āĻiJā(k),sĀÆi=siā14.s_i= _kā\1,ā¦,5\Ļ^J_i(k), s_i= s_i-14. (4) Judge confidence. Judge confidence distinguishes attempts with the same score: the judge may strongly favor that rating for one attempt but be split between ratings for another. We quantify this distinction using one minus normalized rating entropy: ciJ=1āHā”(ĻiJ)logā”5,c_i^J=1- H(Ļ^J_i) 5, (5) where H(p)=āājp(j)logp(j)H(p)=- _jp(j) p(j) is Shannon entropy and logā”5 5 is the maximum entropy over five ratings. Confidence is zero for equally likely ratings and approaches one as probability concentrates on one rating. It measures decisiveness, not a calibrated probability of correctness. Judged reliability. We combine the normalized judge score and confidence into judged reliability: riJ=sĀÆiāciJ.r_i^J= s_ic_i^J. (6) The product is high only when the judge assigns a high score with high confidence; uncertain high scores and confident low scores both receive lower reliability. Appendix B gives an example, and Section 4.3 relates judge score and confidence to externally assessed response quality. 3.3 Measuring student uncertainty We use student uncertainty to prioritize attempts whose tokens the student predicts less confidently, including attempts judged reliable. We measure this uncertainty by averaging normalized next-token entropy over the generated response iy_i of length ni=|i|n_i=|y_i|: uiS=1niāāt=1niHā”(Ļi,tS)logā”||.u_i^S= 1n_i _t=1^n_i H(Ļ^S_i,t) |V|. (7) We use the full-vocabulary student distribution at the distillation temperature (Appendix B); prompt and padding positions are excluded. Normalization bounds uiSu_i^S in [0,1][0,1], and averaging avoids an automatic increase with response length. A larger value indicates more diffuse token predictions along that response, rather than a direct estimate of whether its reasoning is correct. 3.4 Combining reliability and uncertainty We combine the two signals to weight each attemptās distillation loss: wi=Ī±āriJ+(1āĪ±)āuiS,Ī±ā[0,1].w_i=Ī± r_i^J+(1-Ī±)u_i^S, Ī±ā[0,1]. (8) The reliability term prioritizes attempts judged to be high quality with high confidence, while the uncertainty term prioritizes attempts for which the studentās token predictions are uncertain. The additive rule lets either priority contribute independently: uncertainty can contribute when judged reliability is low, and reliability can contribute when the student is already certain. The endpoints Ī±=1Ī±=1 and Ī±=0Ī±=0 use reliability and uncertainty alone, respectively. We use Ī±=0.5Ī±=0.5, which gives both signals equal coefficients; their realized contributions depend on their magnitudes. Section 4.5 evaluates both components and their balance. 3.5 Training objective We train the student to match the teacherās token distributions, scaling each responseās average forward-KL loss by its weight wiw_i: ā(Īø)=i,i[winiāt=1niDKL(Ļi,tTā„Ļi,tS)],L(Īø)=E_x_i,y_i\! [ w_in_i _t=1^n_iD_KL(Ļ^T_i,t\,\|\,Ļ^S_i,t) ], (9) where the expectation is over questions from D and attempts drawn from Equation 1. Following OPSD (Zhao et al., 2026a), we optimize a coordinate-clipped surrogate of this loss, with gradients passing only through student predictions. Appendix B gives the clipping and model-update details. 4 Experiments Science and general reasoning Mathematical reasoning Logic Overall Method GPQA SciB MMLU LiveB OBQA GSM8K MATH AMC Minerva Olymp Zebra Average Qwen3-4B Base 41.16 35.77 78.12 46.50 91.40 93.03 81.68 62.35 29.04 46.03 72.13 61.56 SFTā 42.30 35.91 78.19 42.00 91.00 93.18 80.50 54.22 29.41 44.51 72.10 60.30 GRPOā 43.81 43.89 78.85 47.00 93.00 93.78 81.86 63.86 31.25 47.92 72.58 63.44 RCSDā 41.41 42.56 78.66 49.50 92.80 93.25 81.64 61.45 30.15 45.99 71.53 62.63 OPSDā 51.14 46.32 77.49 47.50 91.60 94.24 84.48 71.08 29.04 52.67 69.15 64.97 NSD 51.14 61.81 79.22 47.62 93.00 94.22 81.83 62.65 29.78 45.96 72.50 65.43 RUSD 52.78 70.66 79.38 61.25 92.60 94.81 85.87 74.10 35.02 54.27 74.38 70.47 Qwen3-8B Base 62.12 75.72 82.04 70.12 93.70 96.17 87.20 80.12 34.10 58.05 81.13 74.59 SFTā 61.24 74.49 82.95 67.62 93.85 96.17 87.13 80.42 34.65 58.42 82.70 74.51 GRPOā 62.63 75.54 82.95 72.50 94.40 96.10 88.20 81.33 34.28 59.27 82.93 75.47 RCSDā 60.61 76.02 83.00 70.62 95.60 95.98 87.95 81.33 34.19 59.79 77.25 74.76 OPSDā 59.72 76.59 80.18 70.63 94.10 95.81 88.31 80.12 33.92 60.42 81.25 74.64 NSD 64.65 74.96 83.06 72.50 95.25 96.23 88.06 82.53 34.01 59.53 82.93 75.79 RUSD 64.90 77.13 83.58 73.13 95.70 97.52 88.48 83.13 36.03 60.87 83.12 76.69 Table 1: Accuracy (Avg@4, %) on eleven reasoning benchmarks. RUSD achieves the highest macroaverage at both model sizes. Average is the unweighted mean over benchmarks; bold marks the best score in each column within a model size, including ties. ā Uses supervision beyond the training questions and the modelās own outputs: dataset responses (SFT, OPSD), generated rubrics (RCSD), or an external LLM judge (GRPO). GPQA: GPQA Diamond; SciB: SciBench; LiveB: LiveBench Reasoning; OBQA: OpenBookQA; Minerva: Minerva Math; Olymp: OlympiadBench; Zebra: ZebraLogic. Our experiments examine reasoning performance, the judgeās quality signals, training dynamics, and design choices: RQ1: Can RUSD improve reasoning accuracy across tasks and model scales? RQ2: Do judge scores and confidence reflect response quality? RQ3: During training, does the number of responses with high judge scores and confidence increase, and does the percentage with low student uncertainty within this group also increase? RQ4: How do the weighting signals, their balance, and the loss type affect performance? 4.1 Experimental Setup Models and Training Data. We evaluate Qwen3-4B and Qwen3-8B (Yang et al., 2025), with component ablations and hyperparameter studies on 4B. All methods share 30,000 NaturalReasoning questions (Yuan et al., 2025), sampled with seed 42 after filtering for non-empty reference answers so that reference-based baselines can use the same questions. RUSD learns from questions and its own online attempts; the datasetās reference answers and generated responses are excluded from its teacher and judge prompts. Baselines. We compare five representative supervision strategies on the same training questions. Supervised fine-tuning (SFT) (Ouyang et al., 2022) learns from the datasetās Llama3.3-70B-Instruct responses. GRPO (Shao et al., 2024) uses scalar rewards from an external LLM judge. RCSD (Gu et al., 2026) supplies question-specific rubrics as teacher context. OPSD (Zhao et al., 2026a) uses the datasetās Llama3.3-70B-Instruct responses as teacher context and matches teacher and student predictions along student-generated responses. NSD (Pei et al., 2026) generates negative conditions from model responses and suppresses behavior encouraged by the negatively conditioned teacher. Thus SFT, OPSD, RCSD, and GRPO rely on supervision beyond the training questions (marked with ā in Table 1), whereas NSD and RUSD use only the questions and the modelās own outputs. Benchmarks. We evaluate eleven benchmarks: GPQA Diamond (Rein et al., 2024), SciBench (Wang et al., 2024), MMLU (Hendrycks et al., 2021a), LiveBench Reasoning (White et al., 2025), and OpenBookQA (Mihaylov et al., 2018) for scientific and general reasoning; GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021b), AMC (AI-MO, 2024), Minerva Math (Lewkowycz et al., 2022), and OlympiadBench (He et al., 2024) for mathematics; and ZebraLogic (Lin et al., 2025) for logic-grid reasoning. Evaluation. We report Avg@4: correctness averaged over four generated responses per question and then across questions, expressed as a percentage. Average in Table 1 is the unweighted mean of the eleven benchmark scores. All accuracy gains are absolute differences on this 0ā100 scale, calculated before rounding. Appendix B gives the decoding and scoring settings. Implementation. All methods use NVIDIA A100 80 GB GPUs and shared training settings. At both model sizes, results are reported after 150 training steps, except GRPO, which uses 500 steps as in RCSDās GRPO setting. For RUSD, we train LoRA adapters (Hu et al., 2022) on four GPUs with Ī±=0.5Ī±=0.5 and student and teacher prompts in thinking mode. Appendix B gives the hyperparameters (Table 5), model updates, GRPO judge, and evaluation-step details. 4.2 Reasoning Performance (RQ1) (a) Judge scores (b) High-confidence judgments (c) Student uncertainty Figure 2: Judge assessments and student uncertainty during Qwen3-4B RUSD training with Ī±=0.5Ī±=0.5. (a) Distribution of judge scores over completed responses in summaries reported at training steps 100, 200, and 300. (b) Percentage of high-confidence judgments (ciJā„0.7c_i^Jā„ 0.7) within the score-4 and score-5 categories; shading shows 95% Wilson confidence intervals for these proportions. (c) Mean student uncertainty over completed score-4/5 responses, with shading spanning the 25thā75th percentiles. RUSD outperforms all five baselines in average accuracy at both model scales. In Table 1, RUSD reaches 70.47% macroaverage accuracy at 4B and 76.69% at 8B, exceeding the base models by 8.90 and 2.10 and NSD, the strongest evaluated baseline by average, by 5.04 and 0.90, respectively. It leads on ten of eleven benchmarks at 4B and all eleven at 8B, with gains spanning scientific and general reasoning, mathematics, and logic-grid reasoning. At 4B, notable results include 70.66% on SciBench versus NSDās 61.81%, and 61.25% on LiveBench Reasoning, 11.75 above the best baseline (RCSD). OpenBookQA is the exception: 92.60% versus 93.00% for GRPO and NSD. At 8B, gains over the best baseline on each benchmark range from 0.10 to 1.38, despite the stronger starting model. The gains also hold against external supervision: at 4B, RUSD exceeds OPSD by 5.49, while SFT on the same supplied solutions falls below the base model. At 8B, the four externally supervised baselines change the base average by ā-0.08 to +0.88, compared with RUSDās +2.10. Thus, RUSD improves both tested models without supplied solutions or question-specific rubrics. Every evaluated 4B training step from 50 to 300 also exceeds the strongest baseline average (Appendix C.2). (a) Judge-score/confidence groups (b) Low student uncertainty by group Figure 3: Response-group composition and low student uncertainty during Qwen3-4B RUSD training with Ī±=0.5Ī±=0.5. High judge confidence means ciJā„0.7c_i^Jā„ 0.7. (a) Shares of completed score-4/5 responses in four groups defined by judge score and confidence. (b) Percentage of responses within each high-confidence group whose student uncertainty is below the pooled mean of all completed score-4/5 responses at that reporting step. Shading shows 95% Wilson confidence intervals from group response counts. Both panels use the same response population at each reporting step (Appendix B). 4.3 Judge Scores, Confidence, and Response Quality (RQ2) Predictors Multiple R R2R^2 (%) p-value Judge score 0.46 21 8.9Ć10ā128.9Ć 10^-12 Judge confidence 0.62 39 5.0Ć10ā235.0Ć 10^-23 Score + confidence 0.77 59 7.2Ć10ā397.2Ć 10^-39 Table 2: Correlation of judge score and confidence with GPT scores (gpt-5.6sol quality ratings) on 200 responses. The multiple correlation coefficient R is the Pearson correlation between fitted and observed GPT scores; R2R^2 is in-sample explained variation. The p-values are from overall regression F-tests. Higher judge scores and confidence are associated with higher GPT scores. We ask gpt-5.6sol (OpenAI, 2026) to rate 200 randomly sampled saved training responses, one per question, using the same 1ā5 quality rubric as the training judge. These ratings, termed GPT scores, serve only as an external measure of response quality and are not used to train RUSD. Table 2 reports linear regressions that estimate GPT scores from judge score, judge confidence, or both. In the joint regression, higher judge scores are associated with higher GPT scores at the same confidence, and higher judge confidence is associated with higher GPT scores at the same judge score. We report the multiple correlation coefficient R, defined as the Pearson correlation between fitted and observed GPT scores. Using judge score and confidence together gives R=0.77R=0.77 and explains 59% of the in-sample variation in GPT scores, compared with 21% for judge score alone and 39% for judge confidence alone. Adding confidence to score increases explained variation by ĪāR2=0.38 R^2=0.38. Thus, judge confidence helps distinguish responses with the same judge score but different GPT scores. These results support combining judge score and confidence to assess response quality. Section 4.5 tests the product of normalized judge score and judge confidence as a training weight. GPT scores are external quality assessments, not verified correctness labels. Appendix B gives the regression definitions and coefficients. 4.4 Training Dynamics (RQ3) In this experiment, we analyze the Qwen3-4B run with Ī±=0.5Ī±=0.5. Figures 2 and 3 show changes in judge assessments and student uncertainty during training; Table 8 in Appendix E provides counts and detailed comparisons. At reported steps 100, 200, and 300, scores 4 and 5 account for at least 99.2% of completed responses, with score 5 becoming more frequent (Figure 2a). Score-only weighting therefore assigns almost all responses either 0.75 or 1.0. Confidence distinguishes attempts with the same score; RQ2 links higher confidence to higher GPT scores at the same judge score. Both the number and percentage of responses with judge score 5 and high confidence increase. From step 50 to 300, this group grows from 319 to 638 responses, or 26.5% to 42.8% of completed score-4/5 responses (Figure 3a; Table 8). Within score 5, the high-confidence proportion (ciJā„0.7c_i^Jā„ 0.7) rises from 49.0% to 68.5% (Figure 2b). The score-4/high-confidence group also grows as a share of score-4/5 responses; its within-score proportion rises from 35.3% to 61.6%, with fluctuations. Mean student uncertainty falls 3.86-fold from step 50 to 300. Across completed score-4/5 responses, mean uiSu_i^S falls from 0.2649 to 0.0687, and the middle half of uncertainties also shifts downward (Figure 2c). Student predictions thus become more concentrated. Within the score-5/high-confidence group, both the number and percentage with low student uncertainty increase overall. Here, low uncertainty means below the mean of all completed score-4/5 responses in the same summary. The count rises from 156 of 319 responses (48.9%) at step 50 to 463 of 638 (72.6%) at step 300 (Figure 3b; Table 8). The percentage peaks at 74.9% at step 200. This threshold measures relative uncertainty because the mean changes across summaries. From step 100 onward, low student uncertainty is more frequent in high-confidence than low-confidence groups at both scores. At score 5, the percentages are similar at step 50 but reach 72.6% versus 39.8% at step 300 (Table 8). Among high-confidence responses, the below-mean percentage is also higher for score 5 than score 4 from step 100 onward (Figure 3b). Together, rising judge confidence and falling student uncertainty are consistent with consolidation, but these summaries do not track individual attempts being learned. The two changes affect the response weight in opposite directions: higher judge confidence at a fixed score raises the reliability contribution, while lower student uncertainty reduces the uncertainty contribution. Further, accuracy reaches 70.47% at step 150, then falls despite declining student uncertainty and increasing high-confidence score-5 judgments (Figure 5, Appendix C.2). Greater predictive certainty and judge confidence therefore do not establish continued improvement in reasoning. A selected heat-calculation case illustrates retained correctness with increasing certainty: both responses are correct and receive score 5, while judge confidence rises from 0.572 to 0.771 and student uncertainty falls from 0.1728 to 0.0388. Table 7 gives the complete final responses; Appendix D gives the calculations and selection details. 4.5 Ablations and Hyperparameters (RQ4) In this analysis, we study response weighting, loss type, and the weighting coefficient Ī± on Qwen3-4B using the eleven-benchmark macroaverage. To assess what judge confidence and student uncertainty add beyond the judge score, our weighting ablations compare against score-only weighting rather than uniform weighting, which assigns the same loss weight to every attempt. Teacher context and forward KL remain fixed across the weighting variants. The loss ablations retain the full weighting rule and change the divergence. Table 3 summarizes the ablations; Table 6 in Appendix C.1 gives the per-benchmark comparison with score-only weighting. Response weighting Average Score only: wi=sĀÆiw_i= s_i 68.44 Uncertainty only: wi=uiSw_i=u_i^S 69.93 Reliability only: wi=riJw_i=r_i^J 70.12 RUSD: wi=0.5āriJ+0.5āuiSw_i=0.5r_i^J+0.5u_i^S 70.47 Loss type Average Forward KL 70.47 Reverse KL 69.85 JensenāShannon divergence 69.12 Table 3: Response-weighting and loss ablations on Qwen3-4B, reporting average accuracy (Avg@4, %) across eleven benchmarks. Weighting variants use the forward-KL configuration; loss variants use the full weighting rule at Ī±=0.5Ī±=0.5. JensenāShannon divergence uses mixture coefficient 0.5. Ablation studies. Adding judge confidence improves average accuracy by 1.68. Score-only weighting, wi=sĀÆiw_i= s_i, reaches 68.44%; multiplying the normalized score by judge confidence, which gives reliability-only weighting wi=riJ=sĀÆiāciJw_i=r_i^J= s_ic_i^J, raises accuracy to 70.12%. Score-only weighting assigns the same weight to all attempts with the same rating; confidence differentiates attempts within each score level. The gain is consistent with the quality audit, in which confidence explains additional variation in external ratings. Combining reliability and uncertainty gives the highest average accuracy among all the weighting variants. Reliability-only weighting prioritizes assessed quality, while uncertainty-only weighting prioritizes diffuse student predictions. The full rule combines both priorities and reaches 70.47%, exceeding reliability-only weighting by 0.35 and uncertainty-only weighting by 0.54. It is also higher than score-only weighting on all eleven benchmarks, with a macroaverage gain of 2.03. Adding confidence gives the larger observed gain (1.68), while adding student uncertainty to reliability gives a smaller further gain in this single-run comparison. Both single-signal variants exceed the strongest baseline average, while their combination gives a further improvement. Forward KL gives the highest average among the tested loss types. Response weights determine each attemptās contribution, while the loss type determines how teacher and student token distributions are matched. With the full weighting rule and Ī±=0.5Ī±=0.5, the forward-KL configuration reaches 70.47%, compared with 69.85% for reverse KL and 69.12% for JensenāShannon divergence with mixture coefficient 0.5. Figure 4: Effect of the mixing coefficient Ī± on Qwen3-4B RUSD, measured by average accuracy across eleven benchmarks. All settings use the dynamic teacher. The endpoints, Ī±=0Ī±=0 (uncertainty only) and Ī±=1Ī±=1 (reliability only), match the corresponding runs in Table 3. Hyperparameter study. Ī±=0.5Ī±=0.5 gives the highest observed average accuracy. In Equation 8, Ī±=0Ī±=0 gives uncertainty-only weighting and Ī±=1Ī±=1 gives reliability-only weighting, matching the corresponding ablations in Table 3. Larger Ī± assigns a higher coefficient to judged reliability, emphasizing quality supported by judge confidence; smaller Ī± assigns a higher coefficient to remaining student uncertainty. Figure 4 reports 69.93%, 70.13%, 70.47%, 70.13%, and 70.12% for Ī±=0Ī±=0, 0.25, 0.5, 0.75, and 1, respectively. The full range is 0.54, and all three intermediate settings reach at least 70.13%. The highest observed average favors using both signals in this configuration. 5 Conclusion In this work, we address two challenges in self-distillation: obtaining teacher guidance without supplied reference solutions, and assessing self-generated attempts without reliable correctness feedback. RUSD uses the attempts as teacher context and weights distillation by judged reliability and student uncertainty. Across eleven benchmarks, it improves Qwen3-4B and Qwen3-8B macroaverage accuracy by 8.90 and 2.10 over the base models, respectively. At 4B, adding judge confidence improves average accuracy, and combining reliability with uncertainty gives the highest observed average among the weighting variants. External ratings support judge confidence as a quality signal, while training summaries show more high-confidence judgments alongside lower student uncertainty. Limitations Model coverage. Our experiments use Qwen3-4B and Qwen3-8B. Further evaluation across model families and sizes is needed to establish how well RUSD generalizes beyond these backbones. Judge design. For efficiency, the judge assesses completed reasoning responses with thinking mode disabled, directly reading rating-token logits without generating an evaluation rationale. Whether enabling judge thinking or updating the judge improves assessment quality remains untested. Enabling thinking would add decoding cost and latency, depending on the reasoning budget; judge-update costs depend on whether updates reuse student parameters or require separate optimization. We also do not compare larger judges or judges from other model families within RUSD, so their effects on accuracy and cost remain untested. Self-judgment and learning dynamics. The judge can confidently endorse incorrect responses and share biases with the generating model. Its confidence measures concentration over ratings, rather than a calibrated probability of correctness. The additive rule also assigns uncertainty-based weight when judged reliability is low. The 200-response audit measures in-sample agreement with automated quality ratings, rather than verified correctness. Training summaries follow changing response populations and describe associations rather than causal learning mechanisms. The later decline in accuracy despite falling entropy shows that greater predictive certainty does not guarantee better reasoning. Component evidence. Component studies use Qwen3-4B. Judged reliability and student uncertainty both lie in [0,1][0,1]; their interaction and relative influence on learning across training stages warrant deeper analysis. Cost and scope. Each attempt requires a teacher forward pass conditioned on the complete attempt and a separate judge forward pass. The teacher does not generate a second response. We do not report total compute across all experiments or compare methods at matched compute budgets, and we do not analyze response length or truncation rates. Experiments use one NaturalReasoning training selection, English questions, and thinking mode for response generation. All benchmarks have automatically checkable answers, so performance on open-ended tasks remains untested. Ethical Considerations This study uses publicly available reasoning resources and model-generated responses. NaturalReasoning is released under C BY-NC 4.0, and the Qwen3-4B and Qwen3-8B backbones under Apache 2.0.11 1 See the official NaturalReasoning dataset card and Qwen3-4B/Qwen3-8B model cards. The datasetās Llama3.3-70B-Instruct responses, which only the SFT and OPSD baselines use, carry additional obligations under the Llama 3.3 Community License when used to train distributed models. Reuse and redistribution should respect these licenses and the terms of individual evaluation resources, including the training datasetās noncommercial restriction. Self-distillation can reinforce factual errors and social biases inherited from the model or its training data. The teacher and judge share a pretrained backbone, so errors may persist even when judge confidence increases and student uncertainty decreases. The proposed weighting signals are fallible assessments, so more confident responses should not be treated as verified answers. The external quality check uses automated gpt-5.6sol ratings of 200 responses. It provides evidence about assessed response quality, rather than human verification or a safety evaluation. Our experiments do not measure fairness, privacy, or robustness to harmful requests. Deployment, especially in high-stakes settings, would require separate assessment and safeguards appropriate to the intended use. References Agarwal et al. (2024) R. Agarwal, N. Vieillard, Y. Zhou, P. Stanczyk, S. Ramos, M. Geist, and O. Bachem On-policy distillation of language models: learning from self-generated mistakes. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: Appendix A. Agarwal et al. (2025) S. Agarwal, Z. Zhang, L. Yuan, J. Han, and H. Peng The unreasonable effectiveness of entropy minimization in LLM reasoning. In Advances in Neural Information Processing Systems, D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen (Eds.), Vol. 38, p. 107150ā107180. External Links: Document, Link Cited by: Appendix A, Appendix A. AI-MO (2024) AI-MO AIMO Validation AMC. Note: Hugging Face dataset External Links: Link Cited by: Ā§4.1. Askell et al. (2021) A. Askell, Y. Bai, A. Chen, D. Drain, D. Ganguli, T. Henighan, A. Jones, N. Joseph, B. Mann, N. DasSarma, N. Elhage, Z. Hatfield-Dodds, D. Hernandez, J. Kernion, K. Ndousse, C. Olsson, D. Amodei, T. Brown, J. Clark, S. McCandlish, C. Olah, and J. Kaplan A general language assistant as a laboratory for alignment. External Links: 2112.00861, Link Cited by: Appendix A. Cai et al. (2026) Q. Cai, Y. Ma, P. Li, Y. Chen, Q. Guo, Y. Zou, L. Li, X. Feng, and B. Qin H2^2SD: hybrid hindsight self-distillation. External Links: 2607.18955, Link Cited by: Appendix A. Chen et al. (2023) H. Chen, R. Tao, Y. Fan, Y. Wang, J. Wang, B. Schiele, X. Xie, B. Raj, and M. Savvides SoftMatch: addressing the quantity-quality trade-off in semi-supervised learning. In The Eleventh International Conference on Learning Representations, External Links: Link Cited by: Appendix A. Cobbe et al. (2021) K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman Training verifiers to solve math word problems. External Links: 2110.14168, Link Cited by: Ā§4.1. DeepSeek-AI et al. (2025) DeepSeek-AI, D. Guo, D. Yang, H. Zhang, J. Song, et al. DeepSeek-R1: incentivizing reasoning capability in LLMs via reinforcement learning. External Links: 2501.12948, Link Cited by: Ā§1. Dong et al. (2023) H. Dong, W. Xiong, D. Goyal, Y. Zhang, W. Chow, R. Pan, S. Diao, J. Zhang, K. Shum, and T. Zhang RAFT: reward ranked finetuning for generative foundation model alignment. Transactions on Machine Learning Research. External Links: ISSN 2835-8856, Link Cited by: Appendix A. Feng et al. (2026) Y. Feng, Z. Feng, and J. Chen PAST: privileged adaptation from complete student trajectories for on-policy self-distillation. External Links: 2608.08726, Link Cited by: Appendix A. Gkountouras et al. (2026) J. Gkountouras, J. JukiÄ, and I. Titov Consensus as privileged context for label-free self-distillation. External Links: 2607.13643, Link Cited by: Appendix A, Ā§2. Gu et al. (2026) S. Gu, J. Chen, S. Zhou, A. Cohan, and R. Ying Rethinking reward supervision: rubric-conditioned self-distillation. In Third Conference on Language Modeling, External Links: Link Cited by: Appendix A, Ā§1, Ā§1, Ā§2, Ā§4.1. Gulcehre et al. (2023) C. Gulcehre, T. Le Paine, S. Srinivasan, K. Konyushkova, L. Weerts, A. Sharma, A. Siddhant, A. Ahern, M. Wang, C. Gu, W. Macherey, A. Doucet, O. Firat, and N. de Freitas Reinforced self-training (ReST) for language modeling. External Links: 2308.08998, Link Cited by: Appendix A. Guo et al. (2017) C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 70, p. 1321ā1330. External Links: Link Cited by: Appendix A. Guo et al. (2026) J. Guo, Y. Yue, X. Fan, C. Yang, and Y. Zhang Learning from consensus and disagreement: unsupervised on-policy self-distillation with minority-trajectory contrast. External Links: 2608.08764, Link Cited by: Appendix A. He et al. (2024) C. He, R. Luo, Y. Bai, S. Hu, Z. Thai, J. Shen, J. Hu, X. Han, Y. Huang, Y. Zhang, J. Liu, L. Qi, Z. Liu, and M. Sun OlympiadBench: a challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, p. 3828ā3850. External Links: Link, Document Cited by: Ā§4.1. He et al. (2026) Y. He, S. Kaur, A. Bhaskar, Y. Yang, J. Liu, N. Ri, L. Fowl, A. Panigrahi, D. Chen, and S. Arora Self-distillation zero: self-revision turns binary rewards into dense supervision. External Links: 2604.12002, Link Cited by: Appendix A, Ā§2. Hendrycks et al. (2021a) D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt Measuring massive multitask language understanding. In International Conference on Learning Representations, External Links: Link Cited by: Ā§4.1. Hendrycks et al. (2021b) D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt Measuring mathematical problem solving with the MATH dataset. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, Vol. 1. External Links: Link Cited by: Ā§4.1. Hinton et al. (2015) G. Hinton, O. Vinyals, and J. Dean Distilling the knowledge in a neural network. External Links: 1503.02531, Link Cited by: Appendix A. Hossain et al. (2026) Md. I. Hossain, H. Kousar, and I. C. Tourni Overcoming scaling limits in on-policy self-distillation for LLM reasoning. External Links: 2609.37915, Link Cited by: Appendix A, Appendix B. Hu et al. (2022) E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen LoRA: low-rank adaptation of large language models. In International Conference on Learning Representations, External Links: Link Cited by: Appendix B, Ā§4.1. Huang et al. (2026) D. Huang, H. Li, Y. Chen, and F. Li When clipping reverses correction: failure dynamics of pointwise forward-KL on-policy self-distillation. External Links: 2609.38995, Link Cited by: Appendix B. Huang et al. (2023) J. Huang, S. Gu, L. Hou, Y. Wu, X. Wang, H. Yu, and J. Han Large language models can self-improve. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali (Eds.), Singapore, p. 1051ā1068. External Links: Link, Document Cited by: Appendix A. HĆ¼botter et al. (2026) J. HĆ¼botter, F. LĆ¼beck, L. D. Behric, A. Baumann, M. Bagatella, D. Marta, I. Hakimi, I. Shenfeld, T. Kleine Buening, C. Guestrin, and A. Krause Reinforcement learning via self-distillation. In Proceedings of the 43rd International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 306, p. 48677ā48721. External Links: Link Cited by: Appendix A, Ā§2. Ichihara et al. (2026) Y. Ichihara, N. Iwase, M. A. Quamar, and J. Komiyama Privileged solutions or context-induced teacher behavior? dissecting on-policy self-distillation. External Links: 2608.09228, Link Cited by: Appendix A. Ke et al. (2026) J. Ke, Z. Wen, W. Li, C. He, and L. Zhang Respecting self-uncertainty in on-policy self-distillation for efficient LLM reasoning. External Links: 2605.13255, Link Cited by: Appendix A, Ā§2. Kim and Rush (2016) Y. Kim and A. M. Rush Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, J. Su, K. Duh, and X. Carreras (Eds.), Austin, Texas, p. 1317ā1327. External Links: Link, Document Cited by: Appendix A. Lewkowycz et al. (2022) A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ramasesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, Y. Wu, B. Neyshabur, G. Gur-Ari, and V. Misra Solving quantitative reasoning problems with language models. In Advances in Neural Information Processing Systems, Vol. 35. External Links: Document, Link Cited by: Ā§4.1. Li et al. (2025) J. Li, S. Nag, H. Liu, X. Tang, S. M. Sarwar, L. Cui, H. Gu, S. Wang, Q. He, and J. Tang Learning with less: knowledge distillation from large language models via unlabeled data. In Findings of the Association for Computational Linguistics: NAACL 2025, L. Chiruzzo, A. Ritter, and L. Wang (Eds.), Albuquerque, New Mexico, p. 2627ā2641. External Links: Link, Document, ISBN 979-8-89176-195-7 Cited by: Appendix A, Ā§2. Li et al. (2026) Y. Li, B. Wang, Y. Liang, Y. Tian, D. Fu, and N. Vasconcelos On-policy self-distillation without any supervision. External Links: 2608.06296, Link Cited by: Appendix A, Ā§1, Ā§2. Lin et al. (2025) B. Y. Lin, R. Le Bras, K. Richardson, A. Sabharwal, R. Poovendran, P. Clark, and Y. Choi ZebraLogic: on the scaling limits of LLMs for logical reasoning. In Proceedings of the 42nd International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 267, p. 37889ā37905. External Links: Link Cited by: Ā§4.1. Liu et al. (2026) X. Liu, X. Wang, Y. Ma, Y. Zhang, and C. Xiao When are teacher tokens reliable? position-weighted on-policy self-distillation for reasoning. External Links: 2605.21606, Link Cited by: Appendix A, Ā§2. Lopez-Paz et al. (2016) D. Lopez-Paz, L. Bottou, B. Schƶlkopf, and V. Vapnik Unifying distillation and privileged information. In International Conference on Learning Representations, External Links: Link Cited by: Appendix A. Lu et al. (2026) Z. Lu, S. Chaudhury, T. Pedapati, K. Murugesan, and L. Yu Activation-conditioned self-distillation. External Links: 2609.38342, Link Cited by: Appendix A. Madaan et al. (2023) A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, S. Gupta, B. P. Majumder, K. Hermann, S. Welleck, A. Yazdanbakhsh, and P. Clark Self-Refine: iterative refinement with self-feedback. In Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36, p. 46534ā46594. External Links: Document, Link Cited by: Appendix A. Mihaylov et al. (2018) T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, E. Riloff, D. Chiang, J. Hockenmaier, and J. Tsujii (Eds.), Brussels, Belgium, p. 2381ā2391. External Links: Link, Document Cited by: Ā§4.1. Mukherjee and Awadallah (2020) S. Mukherjee and A. Awadallah Uncertainty-aware self-training for few-shot text classification. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33, p. 21199ā21212. External Links: Link Cited by: Appendix A. Nicolicioiu et al. (2026) A. L. Nicolicioiu, M. Pezeshki, and A. Courville On-policy self-distillation with sampled demonstrations reduces output diversity. External Links: 2606.26091, Link Cited by: Appendix A. OpenAI (2026) OpenAI GPT-5.6 system card. External Links: Link Cited by: Appendix B, Ā§4.3. Ouyang et al. (2022) L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. Lowe Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35, p. 27730ā27744. External Links: Document, Link Cited by: Ā§4.1. Panickssery et al. (2024) A. Panickssery, S. R. Bowman, and S. Feng LLM evaluators recognize and favor their own generations. In Advances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Eds.), Vol. 37, p. 68772ā68802. External Links: Document, Link Cited by: Appendix A. Pei et al. (2026) R. Pei, Z. Wei, S. Xu, X. Zhu, W. Chen, and Y. Meng Negative self-distillation: learning to reason by avoiding flaws. External Links: 2609.11699, Link Cited by: Appendix A, Ā§1, Ā§2, Ā§4.1. Prabhudesai et al. (2025) M. Prabhudesai, L. Chen, A. Ippoliti, K. Fragkiadaki, H. Liu, and D. Pathak Maximizing confidence alone improves reasoning. External Links: 2505.22660, Link Cited by: Appendix A. Qwen Team (2026) Qwen Team Qwen3.8-27B model card. External Links: Link Cited by: Appendix B. Rein et al. (2024) D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman GPQA: a graduate-level Google-proof Q&A benchmark. In First Conference on Language Modeling, External Links: Link Cited by: Ā§4.1. Rezaei et al. (2026) M. Rezaei, A. Mahmoud, Z. Wang, U. Tyagi, A. Gosai, R. Dumitru, A. Sabharwal, B. Liu, and Y. He Rubric-guided self-distillation: post-training without rubric verifiers. External Links: 2606.12507, Link Cited by: Appendix A, Appendix A, Ā§1, Ā§2. Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo DeepSeekMath: pushing the limits of mathematical reasoning in open language models. External Links: 2402.03300, Link Cited by: Ā§4.1. Shenfeld et al. (2026) I. Shenfeld, M. Damani, J. HĆ¼botter, and P. Agrawal Self-distillation enables continual learning. In Proceedings of the 43rd International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 306, p. 111542ā111558. External Links: Link Cited by: Appendix A, Ā§1, Ā§2. Shrestha and Tessier (2026) S. Shrestha and A. Tessier Rethinking privileged information in on-policy self-distillation. External Links: 2608.18271, Link Cited by: Appendix A. Singh et al. (2024) A. Singh, J. D. Co-Reyes, R. Agarwal, A. Anand, P. Patil, X. Garcia, P. J. Liu, J. Harrison, J. Lee, K. Xu, A. T. Parisi, A. Kumar, A. A. Alemi, A. Rizkowsky, A. Nova, B. Adlam, B. Bohnet, G. F. Elsayed, H. Sedghi, I. Mordatch, I. Simpson, I. Gur, J. Snoek, J. Pennington, J. Hron, K. Kenealy, K. Swersky, K. Mahajan, L. A. Culp, L. Xiao, M. Bileschi, N. Constant, R. Novak, R. Liu, T. Warkentin, Y. Qian, Y. Bansal, E. Dyer, B. Neyshabur, J. Sohl-Dickstein, and N. Fiedel Beyond human data: scaling self-training for problem-solving with language models. Transactions on Machine Learning Research. Note: Expert Certification External Links: ISSN 2835-8856, Link Cited by: Appendix A. Snell et al. (2022) C. Snell, D. Klein, and R. Zhong Learning by distilling context. External Links: 2209.15189, Link Cited by: Appendix A. Vapnik and Vashist (2009) V. Vapnik and A. Vashist A new learning paradigm: learning using privileged information. Neural Networks 22 (5-6), p. 544ā557. External Links: Document Cited by: Appendix A. Verga et al. (2024) P. Verga, S. Hofstatter, S. Althammer, Y. Su, A. Piktus, A. Arkhangorodsky, M. Xu, N. White, and P. Lewis Replacing judges with juries: evaluating LLM generations with a panel of diverse models. External Links: 2404.18796, Link Cited by: Ā§1. Wang et al. (2024) X. Wang, Z. Hu, P. Lu, Y. Zhu, J. Zhang, S. Subramaniam, A. R. Loomba, S. Zhang, Y. Sun, and W. Wang SciBench: evaluating college-level scientific problem-solving abilities of large language models. In Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 235, p. 50622ā50649. External Links: Link Cited by: Ā§4.1. Wei et al. (2026) B. Wei, N. Wang, Y. Zhou, J. Pan, and Z. Zhu COSE: confidence-orchestrated self-evolution for effective LLM reasoning. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, Note: To appear, main conference External Links: Link Cited by: Appendix A, Ā§2. White et al. (2025) C. White, S. Dooley, M. Roberts, A. Pal, B. Feuer, S. Jain, R. Shwartz-Ziv, N. Jain, K. Saifullah, S. Dey, Shubh-Agrawal, S. S. Sandha, S. V. Naidu, C. Hegde, Y. LeCun, T. Goldstein, W. Neiswanger, and M. Goldblum LiveBench: a challenging, contamination-limited LLM benchmark. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: Ā§4.1. Xie et al. (2026) F. Xie, Y. Zhao, B. Chen, A. Cohan, and C. Zhao Small language models as judges for rubric-based reinforcement learning. External Links: 2608.30005, Link Cited by: Ā§1. Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. Li, T. Tang, W. Yin, X. Ren, X. Wang, X. Zhang, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Zhang, Y. Wan, Y. Liu, Z. Wang, Z. Cui, Z. Zhang, Z. Zhou, and Z. Qiu Qwen3 technical report. External Links: 2505.09388, Link Cited by: Ā§4.1. Ye et al. (2026a) T. Ye, L. Dong, G. Chen, H. Zhu, X. Wu, S. Huang, and F. Wei LLM-as-a-coach: experiential learning for non-verifiable tasks. External Links: 2607.18110, Link Cited by: Appendix A, Ā§1. Ye et al. (2026b) T. Ye, L. Dong, X. Wu, S. Huang, and F. Wei On-policy context distillation for language models. External Links: 2602.12275, Link Cited by: Appendix A. Yuan et al. (2024) W. Yuan, R. Y. Pang, K. Cho, X. Li, S. Sukhbaatar, J. Xu, and J. E. Weston Self-rewarding language models. In Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 235, p. 57905ā57923. External Links: Link Cited by: Appendix A. Yuan et al. (2025) W. Yuan, J. Yu, S. Jiang, K. Padthe, Y. Li, D. Wang, I. Kulikov, K. Cho, Y. Tian, J. Weston, and X. Li NaturalReasoning: reasoning in the wild with 2.8M challenging questions. In Advances in Neural Information Processing Systems, D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen (Eds.), Vol. 38. Note: Datasets and Benchmarks Track External Links: Document, Link Cited by: Appendix B, Ā§4.1. Zelikman et al. (2022) E. Zelikman, Y. Wu, J. Mu, and N. Goodman STaR: bootstrapping reasoning with reasoning. In Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35, p. 15476ā15488. External Links: Document, Link Cited by: Appendix A. Zhao et al. (2026a) S. Zhao, Z. Xie, M. Liu, J. Huang, G. Pang, F. Chen, and A. Grover Self-distilled reasoner: on-policy self-distillation for large language models. In Proceedings of the 43rd International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 306, p. 162433ā162448. External Links: Link Cited by: Appendix A, Appendix B, Ā§1, Ā§1, Ā§2, Ā§3.1, Ā§3.5, Ā§4.1. Zhao et al. (2026b) X. Zhao, Z. Kang, A. Feng, S. Levine, and D. Song Learning to reason without external rewards. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: Appendix A, Appendix A. Zheng et al. (2023) L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, H. Zhang, J. Gonzalez, and I. Stoica Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36, p. 46595ā46623. Note: Datasets and Benchmarks Track External Links: Document, Link Cited by: Appendix A, Ā§1. Appendix A Related Work Distillation and self-training. Knowledge distillation transfers teacher predictions to a student (Hinton et al., 2015; Kim and Rush, 2016); on-policy distillation applies this guidance along student-generated responses (Agarwal et al., 2024). Context distillation transfers behavior elicited by additional teacher context (Askell et al., 2021; Snell et al., 2022; Ye et al., 2026b), connecting to learning with privileged information (Vapnik and Vashist, 2009; Lopez-Paz et al., 2016). Self-training selects generated responses using consistency, correctness checks, or rewards (Huang et al., 2023; Zelikman et al., 2022; Gulcehre et al., 2023; Dong et al., 2023; Singh et al., 2024). Self-Refine instead revises responses at inference time, whereas self-rewarding models construct preferences for training (Madaan et al., 2023; Yuan et al., 2024). Sources of teacher guidance. SDFT uses demonstrations for continual learning, and OPSD uses reference solutions for reasoning (Shenfeld et al., 2026; Zhao et al., 2026a). RCSD first trains a rubric generator through a teacher given reference answers, then supplies its rubrics to the reasoning teacher (Gu et al., 2026). RGSD provides rubrics to a frozen teacher and trains a question-only student (Rezaei et al., 2026). Experiential Learning obtains textual coaching feedback for non-verifiable tasks (Ye et al., 2026a). Recent analyses find that OPSDās gains need not depend on the paired reference solution, because solutions to other problems can provide comparable teacher context (Ichihara et al., 2026; Shrestha and Tessier, 2026). These sources of teacher guidance differ from RUSDās use of the completed attempt itself as additional teacher context. Attempts, feedback, and reference-free supervision. SDPO constructs teacher context from textual feedback, or successful responses when only scalar rewards are available (HĆ¼botter et al., 2026). H2SD combines verified successful attempts with reference hints for failures, while PAST uses complete attempts, reference solutions, and correctness-based routing (Cai et al., 2026; Feng et al., 2026). SD-Zero trains a reviser using binary correctness feedback in mathematics and coding, then distills its predictions into the generator (He et al., 2026). Activation-Conditioned Self-Distillation uses verified outcomes to construct teacher-steering activation contrasts (Lu et al., 2026). OASIS conditions the teacher on another unverified attempt and uses final-answer labels (Hossain et al., 2026), and conditioning the teacher on a sampled correct response can reduce output diversity (Nicolicioiu et al., 2026). Without supplied solutions, Canon and u-OPSD condition the teacher on a self-generated solution that reaches the majority answer over multiple samples; CoDA additionally penalizes minority trajectories (Gkountouras et al., 2026; Li et al., 2026; Guo et al., 2026). NSD suppresses tokens favored by negative conditions generated from its own responses (Pei et al., 2026). Unlike these methods, RUSD uses the completed attempt itself as additional teacher context, without correctness feedback, a reference solution, or answer consensus, and weights that attempt by judged reliability and student uncertainty. Confidence and uncertainty. Confidence and uncertainty guide pseudo-label selection and weighting (Chen et al., 2023; Mukherjee and Awadallah, 2020). LLKDās weighted variant adds normalized teacher confidence and student uncertainty to scale selected classification losses (Li et al., 2025). For reasoning, EGRSD attenuates token updates using teacher entropy, while COSE weights PPO updates and prioritizes replay using evaluator confidence (Ke et al., 2026; Wei et al., 2026). PW-OPSD weights reference-conditioned distillation by token position, motivated by diagnostics of teacher-token reliability (Liu et al., 2026). Other approaches optimize predictive confidence through entropy minimization or confidence rewards (Agarwal et al., 2025; Prabhudesai et al., 2025; Zhao et al., 2026b). RUSD distinguishes confidence in judging an attemptās quality from uncertainty in predicting its tokens; evaluator bias and calibration remain relevant to interpreting both signals (Zheng et al., 2023; Panickssery et al., 2024; Guo et al., 2017). Scope of comparison. Our experiments compare representative supervision strategies on fixed NaturalReasoning questions. RGSD shares OPSDās teacher-conditioning principle, using rubrics in place of reference solutions (Rezaei et al., 2026). We include OPSD with NaturalReasoningās Llama-generated responses as teacher context and rubric-conditioned RCSD as representative baselines, without separately evaluating RGSD. Verification-based methods require correctness feedback, while answer consensus requires rules for matching free-form answers. LLKD selects and weights pseudo-labels in a fixed classification label space, and COSE jointly generates tasks and updates its agents. Entropy-minimization and self-certainty methods also require only questions (Agarwal et al., 2025; Zhao et al., 2026b); because student entropy decreases during RUSD training, they are a natural control that we leave to future comparison. Other demonstration-, rubric-, feedback-, and uncertainty-based methods remain applicable or adaptable alternatives outside the present comparison. Appendix B Implementation Details Algorithm 1 RUSD training procedure 1: Questions D, initial model ĻĪø0 _ _0, coefficient Ī± 2: Trained reasoning model ĻĪø _Īø 3: Initialize ĪøāĪø0Īøā _0; retain ĻĪø0 _ _0 as judge 4: for each sampled question iāx_i do 5: Generate attempt iy_i using current ĻĪø _Īø (Eq. 1) 6: Compute student predictions Ļi,tSĻ^S_i,t (Eq. 2) 7: Compute teacher predictions Ļi,tTĻ^T_i,t (Eq. 2) 8: Obtain sĀÆi s_i and ciJc_i^J (Eqs. 3ā5) 9: Compute judged reliability riJr_i^J (Eq. 6) 10: Compute student uncertainty uiSu_i^S (Eq. 7) 11: Form weight wiw_i (Eq. 8) 12: Update Īø (Section 3.5) 13: end for 14: return ĻĪø _Īø Prompts. Table 4 lists the three prompts. Student and teacher prompts use the modelās thinking mode, and the judge prompt does not. The teacher and judge receive the same decoded attempt. Student (user turn) Question: question Please reason step by step, and put your final answer within . Teacher (user turn) Question: question Here is a candidate analysis that may contain useful or misleading information: === Candidate Analysis Begin === attempt === Candidate Analysis End === Solve the problem independently using any useful information from the candidate. Please reason step by step, and put your final answer within . Judge (system turn) You are an impartial evaluator of reasoning responses. Evaluate logical validity, internal consistency, factual plausibility, relevance, and completeness. Use this rubric strictly: 1 = fundamentally incorrect, irrelevant, or nonsensical; 2 = contains major errors that make the conclusion unreliable; 3 = partially correct but contains an important error, omission, or incomplete conclusion; 4 = mostly correct and complete, with only minor issues that do not invalidate the conclusion; 5 = fully correct, complete, relevant, and well justified. A response that omits a requested final answer, conclusion, or subpart is incomplete and must not receive a score above 3. Judge (user turn) Question: question Response: attempt Apply the 1-to-5 rubric above. Return exactly one digit: 1, 2, 3, 4, or 5. Table 4: Student, attempt-conditioned teacher, and judge prompts used by RUSD. The judge applies a common five-level quality rubric. Teacher and judge. Generation, teacher prediction, and student prediction use the same current LoRA adapters (Hu et al., 2022). Gradients through student predictions update these shared adapters, so the teacher also uses the updated parameters on the next step. We refer to this as a dynamic teacher; a fixed teacher retains the initial parameters. The judge uses the initial backbone with adapters disabled and reads the five score-token logits after its prompt, without generating text. For each update, teacher predictions and response weights are computed without gradient tracking; generation and judging are also outside the differentiated loss. Judged reliability example. A judge that splits its probability between ratings 4 and 5 as 0.44/0.56 assigns score 5 (sĀÆi=1 s_i=1) but has confidence of only about 0.57, so riJā0.57r_i^Jā 0.57; a decisive rating of 5 gives reliability close to 1. Prediction settings. The generation notation in Equation 1 denotes one attempt from ĻĪø _Īø using the temperature and top-p/top-k settings in Table 5. Student and teacher predictions in Equation 2 instead use a full-vocabulary softmax with logits divided by the distillation temperature. Student entropy uses this same full-vocabulary distribution, without the generation filters. The judgeās scoring logits in Equation 3 are its five rating-token logits divided by the judge temperature, which is 1.0 in the main configuration. Loss implementation. We use the OPSD implementation (Zhao et al., 2026a), which caps each teacher-weighted log-probability-ratio contribution at Īŗ=0.05Īŗ=0.05 before summing over vocabulary coordinates. This is a coordinate-clipped surrogate of the forward-KL objective in Equation 9, rather than exact KL; the cap can change the update direction (Hossain et al., 2026; Huang et al., 2026). Distillation and student entropy use the full output vocabulary without generation sampling filters. Training configuration. Table 5 collects the main RUSD settings for both model sizes. We use linear learning-rate decay. The effective batch size is 32 questions, combining one question per GPU with eight gradient accumulation steps across four GPUs. The epoch budget is a configured maximum; the main results are reported after 150 training steps. The coefficient study varies the response-weight setting with the same dynamic teacher used in the main configuration. One RUSD training run took approximately 2 days on four NVIDIA A100 80 GB GPUs. Parameter Setting Optimizer AdamW Learning rate 5Ć10ā65Ć 10^-6 Epoch budget (maximum) 3 Effective batch size 32 Training steps for reported results 150 Attempts per sampled question 1 Generation temperature 1.2 Sampling top-p / top-k 0.95 / 20 Max. response length (tokens) 8,192 Max. sequence length (tokens) 32,768 Distillation temperature ĻD _D 1.2 Vocabulary-coordinate cap Īŗ 0.05 Table 5: Key training hyperparameters for the main Qwen3-4B and Qwen3-8B RUSD runs. Masks and reduction. The implementation excludes prompt and padding tokens from entropy and loss calculations. Equations 7 and 9 sum over the unpadded response positions of retained responses. Responses that reach the 8,192-token generation limit are excluded from distillation and receive zero weight. A response with no valid tokens contributes no loss and has uiS=0u^S_i=0. The main configuration uses one question per device microbatch, so the valid-token-normalized loss is the per-response average in Equation 9. The implementation sums weighted token losses and divides by the microbatchās valid-token count before gradient accumulation and distributed averaging. With larger microbatches, this is token-count weighting rather than an equal average of response means. Algorithm 1 presents the per-question computation; optimizer updates follow the accumulation schedule described above. No gradient passes through generation, judging, teacher predictions, or weights. Data. We use the public 1.1M-question release of NaturalReasoning (Yuan et al., 2025). Each row pairs a question with a reference answer derived from its source document, when one exists, and a response generated by Llama3.3-70B-Instruct. All methods we train share one selection: we keep rows whose reference-answer field is non-empty, shuffle with seed 42, and take the first 30,000 questions. In RUSD, the reference field is used only for this selection, and neither dataset answer field is used in training prompts. The student prompt contains the question and task instructions; teacher and judge prompts also include the modelās own completed attempt. GRPO reward. For GRPO, scalar response-quality rewards are supplied by a frozen external Qwen3.8-27B judge (Qwen Team, 2026). Training steps for reported results. At both Qwen3-4B and Qwen3-8B, the main SFT, RCSD, OPSD, NSD, and RUSD results are reported after 150 training steps, and GRPO results after 500 training steps. Training steps denote optimizer updates. For RUSD, the reporting step was fixed at 150 independently of the checkpoint evaluations. The additional evaluations in Appendix C.2 characterize performance over training. Evaluation decoding. We use temperature 1.0, top-p=0.95p=0.95, top-k=ā1k=-1 (disabled), min-p=0p=0, and presence penalty 0. We enable thinking mode and generate four responses per question. These evaluation settings differ from the training generation settings in Table 5. Benchmark scoring. Benchmark-specific automatic scoring compares parsed multiple-choice answers with annotated options and extracted mathematical answers with references using numerical or symbolic comparison. ZebraLogic requires every grid assignment to be correct. These scores use neither the training judge nor an external LLM judge. Training-dynamics summaries and intervals. Figures 2(bāc) and 3 use the same reported summaries of completed score-4/5 responses. Figure 2(a) uses the corresponding saved summaries at steps 100, 200, and 300, including all completed responses with scores 1ā5. For Figure 2(b), the denominator is all responses with the corresponding judge score, and the numerator counts high-confidence judgments. For Figure 3(b), n is the judge-group response count and k counts responses with student entropy below the pooled mean of the corresponding summary. Both proportion plots use 95% Wilson confidence intervals with z=1.96z=1.96. For example, the score-5/high-confidence group at step 100 has k=231k=231 and n=358n=358, giving 64.5% and an interval of 59.4ā69.3%; at step 300, 463/638463/638 gives 72.6% with an interval of 69.0ā75.9%. These intervals describe response proportions, rather than variation across independent training runs. The band in Figure 2(c) instead spans the 25thā75th percentiles of response entropy; it is not a confidence interval. External quality audit. We randomly sample 200 saved questionāresponse pairs and ask gpt-5.6sol (OpenAI, 2026) to act as an external judge using the same five-level quality rubric in Table 4. Each of the 200 questions contributes one response and one observation. The three regressions include an intercept and predict the external GPT rating from the training judgeās score, confidence, or both. We report descriptive in-sample fit and coefficient estimates. The multiple correlation coefficient R=R2R= R^2 is the correlation between fitted and observed GPT ratings. Its square, R2R^2, is the fraction of external-rating variation explained within the fitted sample. R is nonnegative; the signs of the regression coefficients give the direction of the fitted conditional associations. The joint coefficients are bscore=0.84b_score=0.84 per rating level and bconfidence=3.78b_confidence=3.78. Thus, a 0.1 increase in confidence is associated with a 0.378 increase in the external rating at the same judge score; an increase of 1 in the judge score is associated with a 0.84 increase in the rating at the same confidence. These estimates describe response-quality ratings, rather than verified correctness or calibrated probabilities. Appendix C Additional Experimental Results C.1 Extended weighting results Benchmark Score only RUSD Ī GPQA Diamond 52.40 52.78 +0.38 SciBench 66.11 70.66 +4.55 MMLU 78.12 79.38 +1.26 LiveBench Reasoning 58.50 61.25 +2.75 OpenBookQA 91.75 92.60 +0.85 GSM8K 94.56 94.81 +0.25 MATH 83.47 85.87 +2.40 AMC 71.39 74.10 +2.71 Minerva Math 32.08 35.02 +2.94 OlympiadBench 51.41 54.27 +2.86 ZebraLogic 73.02 74.38 +1.36 Macroaverage 68.44 70.47 +2.03 Table 6: Qwen3-4B accuracy (Avg@4, %) under score-only and RUSD weighting. Ī is RUSDās absolute gain over score-only weighting on the same 0ā100 scale. RUSD improves all eleven benchmarks. Table 6 reports the complete benchmark comparison between score-only weighting and RUSD. The full rule is higher on each benchmark, including scientific reasoning, mathematics, and logic-grid reasoning, with single-run differences below 1 on several benchmarks. This comparison changes confidence and student uncertainty together; the endpoint comparisons in Section 4.5 examine reliability-only and uncertainty-only weighting. Sensitivity to SciBench. The 4B gain is largest on SciBench, where the base model starts at 35.77% and RUSD reaches 70.66%. Removing this benchmark from the macroaverage still leaves RUSD ahead of the base model by 6.30 and the strongest remaining baseline, OPSD, by 3.61. Thus, the overall advantage is not confined to SciBench. C.2 Performance across training steps Figure 5: Average accuracy (Avg@4, %) across eleven benchmarks for Qwen3-4B RUSD with Ī±=0.5Ī±=0.5 at six evaluated training steps from 50 to 300. The highest observed value is 70.47% at step 150. The dashed line marks the strongest Qwen3-4B baseline average (NSD, 65.43%; Table 1). Accuracy rises from 67.78% at step 50 to 70.47% at step 150, remains close at step 200 (70.34%), and falls to 67.96% at step 300 (Figure 5). Figure 2(c) shows that student entropy continues to fall as this accuracy curve declines. Entropy measures predictive uncertainty, whereas benchmark accuracy measures task performance. Their divergence shows why the learning-priority signals should be interpreted alongside the benchmark curve. Training step 150 gives the highest observed reasoning performance, rather than the lowest student entropy, and every evaluated training step exceeds the strongest baseline, NSD (65.43%). Appendix D Retaining a Correct Solution with Increasing Certainty Table 7 reproduces the complete saved final responses to the same heat-calculation question with Ī±=0.5Ī±=0.5. They are observations at training steps 30 and 105, rather than responses regenerated for separate evaluations at those training steps. Both responses contain the correct two-stage calculation. Their shared correctness makes this case useful for examining changes that the discrete judge score alone does not capture. We compare the logged training signals and verify the saved final calculations below. Question. A solid has a melting point of 90.0ā90.0 C. Calculate the total heat required to change 2.5 kg of this solid from 40.0ā40.0 C to a liquid, given that the specific heat of the solid is 390 J/kg-K and its heat of fusion is 4000 J/kg. Show your work and explain your reasoning. Earlier response (step 30) Later response (step 105) To determine the total heat required to change 2.5 kg of a solid from 40.0āC to a liquid, we need to account for two distinct processes: 1. Heating the solid from its initial temperature to its melting point 2. Melting the solid at its melting point into a liquid To calculate the total heat required to change 2.5 kg of a solid from 40.0āC to a liquid, we need to consider two distinct processes: 1. Heating the solid from its initial temperature (40.0āC) to its melting point (90.0āC). 2. Melting the solid at its melting point (90.0āC) into a liquid. Step 1: Heating the solid to its melting point The temperature difference between the initial temperature and the melting point is: ĪāT=90.0āāCā40.0āāC=50.0āāC T=90.0 C-40.0 C=50.0 C The heat required to raise the temperature of the solid is calculated using the specific heat formula: Q1=mā cā ĪāTQ_1=mĀ· cĀ· T Where: ā m=2.5ākgm=2.5\,kg ā c=390āJ/kgāā āKc=390\,J/kg Ā·K ā ĪāT=50.0āK T=50.0\,K (since a change of 1āC = 1 K) Q1 Q_1 =2.5ākgā 390āJ/kgāā āKā 50.0āK =2.5\,kgĀ· 390\,J/kg Ā·KĀ· 50.0\,K =48750āJ =48750\,J Step 1: Heat Required to Raise the Temperature of the Solid The formula for the heat required to change the temperature of a substance is: Q1=mā cā ĪāTQ_1=mĀ· cĀ· T Where: ā m=2.5ākgm=2.5\,kg (mass of the solid), ā c=390āJ/kg-Kc=390\,J/kg-K (specific heat of the solid), ā ĪāT=TmeltāTinitial T=T_melt-T_initial =90.0āāCā40.0āāC=50.0āāC=90.0 C-40.0 C=50.0 C. Q1 Q_1 =2.5ākgĆ390āJ/kg-KĆ50.0āK =2.5\,kgĆ 390\,J/kg-KĆ 50.0\,K =48,750āJ =48,750\,J Step 2: Melting the solid at its melting point The heat required to melt the solid is calculated using the heat of fusion formula: Q2=mā LfQ_2=mĀ· L_f Where: ā m=2.5ākgm=2.5\,kg ā Lf=4000āJ/kgL_f=4000\,J/kg Q2 Q_2 =2.5ākgā 4000āJ/kg =2.5\,kgĀ· 4000\,J/kg =10000āJ =10000\,J Step 2: Heat Required to Melt the Solid The formula for the heat required to melt a substance is: Q2=mā LfQ_2=mĀ· L_f Where: ā m=2.5ākgm=2.5\,kg (mass of the solid), ā Lf=4000āJ/kgL_f=4000\,J/kg (heat of fusion). Q2 Q_2 =2.5ākgĆ4000āJ/kg =2.5\,kgĆ 4000\,J/kg =10,000āJ =10,000\,J Step 3: Total Heat Required Qtotal Q_total =Q1+Q2 =Q_1+Q_2 =48750āJ+10000āJ =48750\,J+10000\,J =58750āJ =58750\,J Step 3: Total Heat Required Qtotal Q_total =Q1+Q2 =Q_1+Q_2 =48,750āJ+10,000āJ =48,750\,J+10,000\,J =58750āJ = 58750\,J Final Answer 58750 58750 Final Answer 58750āJ 58750\,J Recorded training signal Step 30 Step 105 Judge score sis_i 5 5 Judge probabilities ĻiJā(4) _i^J(4) / ĻiJā(5) _i^J(5) 0.438 / 0.562 0.119 / 0.879 Judge confidence ciJc_i^J 0.572 0.771 Student uncertainty uiSu_i^S 0.1728 0.0388 Table 7: Complete saved final responses (after thinking) and training signals for a selected heat-calculation case from an auxiliary fixed-teacher run. Both answers are correct; judge confidence increases and student uncertainty decreases between steps 30 and 105. Appendix D gives the calculations and case-selection details. Checking both solutions. The solid must first reach its melting point and then undergo the phase change. Heating requires mācāĪāT=2.5Ć390Ć(90ā40)=48,750mc T=2.5Ć 390Ć(90-40)=48,750 J. Melting requires māLf=2.5Ć4,000=10,000mL_f=2.5Ć 4,000=10,000 J. The total is therefore 58,75058,750 J, matching the supplied reference answer. Both saved final responses explicitly apply these formulas and obtain these values. A temperature difference of 50āāC50\, C equals 5050 K, so their temperature conversion is also correct. This check concerns the saved final explanations shown in the case record. What changes when correctness is retained? The judge score remains 5, while the rating distribution becomes more concentrated: the probabilities assigned to scores 4 and 5 change from approximately 0.438/0.562 to 0.119/0.879. Judge confidence consequently rises from 0.572 to 0.771. Meanwhile, mean normalized student entropy falls from 0.1728 to 0.0388, approximately a 4.5-fold reduction. The later response preserves the same correct reasoning with less predictive uncertainty; it also states the energy unit next to its final boxed value. The fixed score thus conceals two changes with different effects on the additive weight: greater judged reliability raises its reliability contribution, while lower student uncertainty reduces its uncertainty contribution. This is the response-level pattern motivating learning from a correct attempt whose generation and assessment can still become more certain. Selection and interpretation. The case was selected post hoc through a non-blinded review of repeated training observations. Reference answers were used for this selection and the subsequent correctness check, not for training supervision. The displayed case was chosen after checking its full saved final explanations for correctness at both observations and requiring an unchanged judge score. The case illustrates retained correctness with increasing certainty; it neither identifies an initial random guess nor isolates the cause of the change. Appendix E Additional Training-Dynamics Analysis Judge score 5 Judge score 4 Step High conf. Low conf. High conf. Low conf. 0 135/314 103/334 116/137 215/351 50 156/319 165/332 109/196 204/359 100 231/358 158/322 140/259 146/310 150 335/463 107/311 151/249 135/314 200 361/482 115/290 235/373 95/218 250 438/586 124/295 216/338 98/200 300 463/638 117/294 212/344 106/214 Table 8: Response counts underlying Figures 2(b), 3, and 6. Each entry is k/nk/n: n is the group size and k counts responses with student entropy below the pooled mean of completed score-4/5 responses in the same reported summary. High judge confidence means cJā„0.7c^Jā„ 0.7; low confidence means cJ<0.7c^J<0.7. The figures show reported steps 50ā300; step 0 is included here for completeness. Table 8 gives the shared counts used in the training-summary analysis. Figure 2(b) divides the high-confidence group size by all responses with the same judge score. Figure 3(a) divides each group size by all completed score-4/5 responses, while Figure 3(b) reports k/nk/n for the two high-confidence groups. For the score-5/high-confidence group at step 300, these proportions are 638/932=68.5%638/932=68.5\%, 638/1490=42.8%638/1490=42.8\%, and 463/638=72.6%463/638=72.6\%, respectively. Thus, the panels describe different proportions of the same response population. Figure 2(a) uses the same saved summaries at steps 100, 200, and 300 but includes all completed responses: 1,254, 1,369, and 1,502, respectively. Their score-4/5 subsets contain 1,249, 1,363, and 1,490 responses, matching Table 8. Score distributions and group composition. The score-5 share of completed responses rises from 54.2% at reported step 100 to 62.1% at step 300 (Figure 2a). Among completed score-4/5 responses, the score-5/high-confidence share grows from 26.5% at reported step 50 to 42.8% at step 300. The score-4/high-confidence share grows from 16.3% to 23.1%, reaching its maximum of 27.4% at step 200 (Figure 3a). Within score 5, the high-confidence proportion continues to rise after the accuracy peak, from 59.8% at step 150 to 68.5% at step 300. Figure 6: Judge-group composition among low-entropy responses during Qwen3-4B RUSD training with Ī±=0.5Ī±=0.5. Only completed score-4/5 responses are included; low entropy means student entropy below their pooled mean at that reporting step. Each curve shows a groupās share of this subset; the four shares sum to 100%. High judge confidence means ciJā„0.7c_i^Jā„ 0.7. Shading shows 95% Wilson confidence intervals. Composition of low-entropy responses. Among responses below the pooled mean student entropy, the score-5/high-confidence groupās share rises from 24.6% at step 50 to 51.6% at step 300 (Figure 6). At step 300, this share is 463/898463/898, whereas the within-group fraction in Figure 3(b) is 463/638=72.6%463/638=72.6\%. The composition depends on both group size and the fraction of each group below the mean. Spread of student uncertainty. The 25thā75th percentile range of response uncertainty shifts from 0.21032ā0.31335 at reported step 50 to 0.04992ā0.08053 at step 300 (Figure 2c). These quantiles describe variation across completed score-4/5 responses in each summary. Detailed relative-certainty comparisons. At step 50, the fractions below the pooled mean student uncertainty are 48.9% for score-5/high-confidence responses and 52.6% overall; at step 100, they are 64.5% and 54.0%. The groupās fraction peaks at 74.9% at step 200 before decreasing to 72.6% at step 300, when the pooled fraction is 60.3%. It exceeds the pooled percentage from step 100 onward. The score-4/high-confidence groupās fraction remains at or above the pooled percentage throughout, with almost no difference at step 100. High- and low-confidence score-5 responses have similar fractions at step 50 (48.9% versus 49.7%), but differ at step 300 (72.6% versus 39.8%).