Paper deep dive
ReTeach: Building a Self-Teacher through Multi-Round Reflection and Retry
Yafeng Tang, Hao Li, Hongsheng Yu, Qiang Fu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 10/10/2026, 1:51:29 AM
Summary
The paper introduces ReTeach, a self-distillation framework that constructs a self-teacher through multi-round reflection and retry using only self-generated attempts and outcome-level verification. Unlike existing methods that rely on reference answers or external feedback, ReTeach iteratively reflects on failed student rollouts to generate corrective contexts. It employs an outcome-aware selection and weighting strategy to distill knowledge from initially correct, reflection-corrected, and unresolved examples. Experiments across six benchmarks in mathematical reasoning, science QA, and tool use show ReTeach outperforms GRPO and other baselines like CEPO and OPSD.
Entities (11)
Relation Signals (9)
ReTeach → initializedfrom → Qwen3-8b
confidence 95% · We conduct all experiments initialized from Qwen3-8B
ReTeach → outperforms → GRPO
confidence 95% · ReTeach improves average accuracy over GRPO by 1.39 percentage points.
ReTeach → uses → On-Policy Distillation
confidence 95% · Through on-policy distillation, the student matches the teacher's context-conditioned token-level predictive distributions
ReTeach → uses → Multi-Round Reflection
confidence 95% · ReTeach, a Reflective self-distillation framework that constructs its self-Teacher through multi-round reflection and retry
ReTeach → evaluatedon → SciKnowEval
confidence 90% · For science question answering, we use the undergraduate-level reasoning subsets (L3) of SciKnowEval
ReTeach → evaluatedon → AIME 2024
confidence 90% · evaluate the trained model on external competition-level benchmarks, including AIME 2024
ReTeach → evaluatedon → ToolAlpaca
confidence 90% · For tool-use, we use ToolAlpaca
OPSD → comparedwith → ReTeach
confidence 85% · We compare ReTeach with four post-training baselines... OPSD... CEPO
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Self-distillation can improve reasoning without a separately trained, more capable teacher, but its effectiveness depends on how the self-teacher gains an advantage over the student. Conditioning the teacher on reference answers or solutions can provide such an advantage, but this information may be unavailable. Reflection offers a way to derive explicit error diagnoses and revision guidance from self-generated attempts, yet existing reflection-based methods often combine it with reference information, rich task feedback, or persistent memory. We introduce ReTeach, a Reflective self-distillation framework that constructs its self-Teacher through multi-round reflection and retry using only self-generated attempts and outcome-level verification. Starting from an unsuccessful student rollout, the teacher alternates explicit reflection with renewed attempts until success or the retry budget is exhausted, without reference answers or solutions, external diagnostic feedback, or cross-example memory. Each failed retry informs subsequent reflection, while successful correction provides outcome-level evidence for the potential utility of the resulting teacher context. An outcome-aware selection and weighting strategy distinguishes initially correct, reflection-corrected, and unresolved examples, assigning separate weights to their category-normalized distillation losses. Through on-policy distillation, the student matches the teacher's context-conditioned token-level predictive distributions at prefixes of its own rollouts, transferring the benefits of iterative correction while retaining single-pass inference. Across six benchmarks spanning mathematical reasoning, science question answering, and tool use, ReTeach improves average accuracy over GRPO by 1.39 percentage points.
Tags
Links
- Source: https://arxiv.org/abs/2610.11529v1
- Canonical: https://arxiv.org/abs/2610.11529v1
Trouble viewing inline? Open PDF directly →
Full Text
73,185 characters extracted from source content.
Expand or collapse full text
inkscapeversion=1 ReTeach: Building a Self-Teacher through Multi-Round Reflection and Retry Yafeng Tang Email: tangyafeng8@gmail.com Hao Li Email: erichaoli@tencent.com Hongsheng Yu Email: yannickyu@tencent.com Qiang Fu Email: leonfu@tencent.com Abstract Self-distillation can improve reasoning without a separately trained, more capable teacher, but its effectiveness depends on how the self-teacher gains an advantage over the student. Conditioning the teacher on reference answers or solutions can provide such an advantage, but this information may be unavailable. Reflection offers a way to derive explicit error diagnoses and revision guidance from self-generated attempts, yet existing reflection-based methods often combine it with reference information, rich task feedback, or persistent memory. We introduce ReTeach, a Reflective self-distillation framework that constructs its self-Teacher through multi-round reflection and retry using only self-generated attempts and outcome-level verification. Starting from an unsuccessful student rollout, the teacher alternates explicit reflection with renewed attempts until success or the retry budget is exhausted, without reference answers or solutions, external diagnostic feedback, or cross-example memory. Each failed retry informs subsequent reflection, while successful correction provides outcome-level evidence for the potential utility of the resulting teacher context. An outcome-aware selection and weighting strategy distinguishes initially correct, reflection-corrected, and unresolved examples, assigning separate weights to their category-normalized distillation losses. Through on-policy distillation, the student matches the teacher’s context-conditioned token-level predictive distributions at prefixes of its own rollouts, transferring the benefits of iterative correction while retaining single-pass inference. Across six benchmarks spanning mathematical reasoning, science question answering, and tool use, ReTeach improves average accuracy over GRPO by 1.39 percentage points. Among all compared methods, including those using privileged information, ReTeach achieves the highest average accuracy on five benchmarks and ranks second on the remaining one. These results demonstrate the effectiveness of constructing reflective self-teachers from self-generated experience and outcome-level verification to improve single-pass reasoning. †footnotetext: ∗ Equal contribution † Corresponding authors 1 Tencent Artificial Intelligence Platform Department This work was done during Yafeng Tang’s Project Up internship at Tencent. 1 Introduction Reinforcement learning with verifiable rewards (RLVR) has become the default recipe for post-training large language models, underpinning the strongest open reasoning models on mathematics and code, where a verifier returns a scalar reward without human annotation (Lambert et al., 2025; DeepSeek-AI, 2025). However, a binary reward per response provides no information on which intermediate steps drove the result. This sparse supervision makes training expensive: the model must discover good reasoning by comparing and contrasting many self-generated responses. Recently, on-policy distillation (OPD) addresses this by replacing the binary reward with per-token targets from a teacher that scores the student’s own rollouts (Lu and Lab, 2025). However, the effectiveness of OPD depends on the availability of a sufficiently capable teacher: the quality of its supervision ultimately constrains how far the student can improve, limiting the ability to advance the frontier of reasoning. This raises a natural question: can a model bootstrap its own teacher signal without relying on a separately trained, more capable teacher? Figure 1: Comparison across existing self-improvement methods. Our ReTeach realizes denser supervision without any privileged information from original datasets. Self-distillation derives the teacher from the model itself, but its effectiveness depends on how the teacher gains an advantage over the student. OPSD conditions its teacher on reference CoT (Zhao et al., 2026a), while ATESD adaptively limits this exposure (Han et al., 2026). RLSD combines ground-truth answer conditioning with group-based rollout exploration (Yang et al., 2026), whereas SDPO constructs its teacher from successful rollouts in binary-reward settings (Hübotter et al., 2026). Reference answers or solutions may be unavailable, while raw rollouts do not necessarily explain how to correct the student’s errors. Reflection can turn this experience into explicit error diagnoses and revision guidance (Madaan et al., 2023; Shinn et al., 2023). Existing approaches combine reflection with task feedback, persistent memory or ground-truth answers. RESD (Zhang et al., 2026) uses a persistent playbook and cached successful solutions with additional ground-truth answers in teacher prompts. ReflectionCoder (Ren et al., 2025) and HERO (Liu et al., 2026) use code-execution feedback and environment observations, respectively. We study whether effective self-teachers can instead be constructed from a model’s own attempts and outcome-level verification, without reference answers or solutions, external diagnostic feedback, or cross-example memory. We introduce ReTeach, a self-distillation framework that couples multi-round reflection and retry with outcome-aware distillation. Starting from an unsuccessful student rollout, the self-teacher explicitly reflects on the history of failed attempts and generates a new response, repeating until success or a fixed retry budget is exhausted. Retrying serves two purposes: a failed attempt provides additional evidence for subsequent reflection, while a successful attempt provides outcome-level evidence that the constructed context can support a correct response. This process uses outcome-level verification without providing the teacher with ground-truth answers, reference solutions, or externally supplied diagnostic feedback. ReTeach uses OPD to match teacher distributions at prefixes of the original student rollout, rather than directly training on successful retries. Initially correct, reflection-corrected, and unresolved rollouts receive separately weighted, category-normalized losses. Successful correction serves as an outcome-based indicator of potential supervision utility, while unresolved examples can be downweighted or excluded. The student retains single-pass inference. Figure 1 compares ReTeach with existing self-improvement methods. Our contributions are as follows: • We introduce a multi-round reflection-and-retry process for self-teacher construction. Rather than stopping at a retrospective diagnosis, ReTeach generates renewed attempts whose failures inform subsequent reflection, developing teacher contexts through iterative correction without access to reference answers or solutions. • We propose outcome-aware selection and category-level weighting that connect correction outcomes to distillation supervision. By distinguishing initially correct, reflection-corrected, and unresolved rollouts, ReTeach balances preserving existing behavior with learning from reflective contexts, while accounting for the less certain utility of unresolved examples. • Across mathematical reasoning, science question answering, and tool-use tasks, ReTeach achieves average accuracies of 65.62%, 66.71%, and 62.72%, respectively, consistently outperforming GRPO (64.32%, 65.43%, and 60.85%). Among the compared methods, including those that use external privileged information, ReTeach achieves the highest average accuracy on five of the six benchmarks and ranks second on the remaining one. 2 Background In this section, we introduce several key concepts that form the foundation of self-distillation, the core of ReTeach. Then, we compare existing self-distillation methods by how the privileged advantage is exploited in Figure 1, with a detailed survey of each baseline deferred to Section 5. Step-Level Chain-of-Thought Reasoning. Chain-of-thought (CoT) prompting and training encourage LLMs to generate intermediate reasoning steps before committing to a final answer (Wei et al., 2022; Kojima et al., 2022), which decomposes a hard problem into tractable sub-steps and substantially improves accuracy on mathematical and multi-step reasoning tasks (Yu et al., 2024). Beyond accuracy, the explicit trace makes the model’s computation inspectable and allows supervision to be attached to individual steps rather than only to the outcome, offering denser feedback and improved interpretability (Uesato et al., 2022; Lightman et al., 2024). CoT enables step-level supervision through process reward models or token-level proxies, but requires auxiliary models or extra annotation (Lightman et al., 2024; Wang et al., 2024; Cui et al., 2026; Zhang et al., 2025), which motivates researchers to obtain dense targets from distillation instead. Token-Level Knowledge Distillation. Knowledge distillation transfers a teacher πtea _tea into a student πθ _θ by matching next-token distributions instead of one-hot labels, providing a dense target at every position (Hinton et al., 2015; Kim and Rush, 2016). Offline imitation of teacher traces (Yu et al., 2024; DeepSeek-AI, 2025) suffers from a train–inference distribution mismatch and compounding exposure bias (Agarwal et al., 2024; Gu et al., 2024). On-policy distillation (OPD) methods construct targets from samples generated by the current policy or its improved variants, thereby converting reinforcement signals into supervised updates (Agarwal et al., 2024; Gu et al., 2024; Lu and Lab, 2025). The matching is performed on the student’s own rollouts, ℒOPD(θ)=y∼πθ(⋅∣x)[∑tKL(πθ(⋅∣x,y<t)∥πtea(⋅∣x,y<t))],L_OPD(θ)=E_y _θ(· x) [ _tKL ( _θ(· x,y_<t)\,\|\, _tea(· x,y_<t) ) ], (1) so the mismatch disappears, and the teacher–student log-ratio acts as a per-token advantage (Agarwal et al., 2024; Hübotter et al., 2026). Still, OPD needs a stronger external teacher. Such a teacher is computationally expensive to obtain and upper-bounds the student’s capability (Lu and Lab, 2025; Wang et al., 2026). On-Policy Self-Distillation. Recently, on-policy self-distillation (OPSD) (Zhao et al., 2026a) has emerged as a popular approach that eliminates the need for a separate teacher by running a single model under two information conditions (Hübotter et al., 2026). The student observes only the prompt, while the teacher also sees the privileged context c, available exclusively during training. The teacher is then the same policy conditioned on c, πtea(⋅∣x,y<t):=sg(πθ(⋅∣x,c,y<t)), _tea(· x,y_<t):=sg ( _θ(· x,c,y_<t) ), (2) where sg(⋅)sg(·) denotes stop-gradient and keeps the teacher from co-evolving with the student. Since c is withheld during inference, the teacher remains better informed than the student (Hübotter et al., 2026; Yang et al., 2026), and distilling it back provides dense token-level supervision at no additional model cost. While methods represented by OPSD have demonstrated strong empirical performance, they still rely on some form of external privileged information. We instead seek to remove this dependence, allowing the model to correct itself through iterative self-reflection. 3 Methodology In this section, we propose the ReTeach algorithm, which builds a teacher through multi-round reflect-and-retry for self-distillation as shown in Figure 2. Figure 2: An overview of our ReTeach method. 3.1 Building a Teacher through Multi-Round Reflect-and-Retry Inspired by recent work on iterative reasoning and self-refinement (Radha et al., 2024; Madaan et al., 2023), ReTeach constructs a deliberative teacher through multi-round interaction. The central idea is that a model can recover from an initially unsuccessful response by repeatedly revisiting its previous attempts, explicitly reflecting on potential errors, and trying again. For each question x, the student first samples a response y∼πθ(⋅∣x)y _θ(· x), which is evaluated by a verifier r=(y)∈0,1r=V(y)∈\0,1\. If r=1r=1, no refinement is needed, and we set c(x)=∅c(x)= . Otherwise, the teacher initiates a multi-round reflect-and-retry process from the failed student response. Specifically, let Ak−1=(y,y1,…,yk−1)A_k-1=(y,y_1,…,y_k-1) (3) denote the attempt history available at round k, where y is the initial student response and yiy_i is the teacher’s retry at round i. At each round, the teacher first generates an explicit reflection τk _k based on the question and all previous attempts, and then produces a new response yky_k conditioned additionally on the current reflection: τk _k ∼πθtea(⋅∣x,Ak−1), _ _tea (· x,A_k-1 ), (4) yk y_k ∼πθtea(⋅∣x,Ak−1,τk). _ _tea (· x,A_k-1, _k ). Importantly, Ak−1A_k-1 contains only previous attempts; reflections from earlier rounds are not carried forward. Each reflection therefore serves as a transient intermediate step that guides the retry within the current round. The teacher is instructed to enclose τk _k within <reflect> and </reflect> tags before generating yky_k, making error reflection an explicit part of the refinement process. The complete prompt template is provided in Appendix C. After each round, the verifier evaluates the retry yky_k. The reflect-and-retry process terminates when a retry yky_k satisfies (yk)=1V(y_k)=1, or after K unsuccessful rounds, where K is the maximum number of rounds permitted by the context budget. Let T denote the final round. We define the teacher content as c(x)=(AT−1,τT)=(y,y1,…,yT−1,τT),c(x)= (A_T-1, _T )= (y,y_1,…,y_T-1, _T ), (5) which consists of all attempts preceding the final retry together with the final-round reflection. In this way, the teacher constructs its refinement context entirely from self-generated attempts, without access to externally supplied reference solutions or corrective feedback. 3.2 Stable On-Policy Distillation from the Deliberative Teacher We transfer the capabilities of the deliberative teacher to the single-turn student through on-policy distillation. For each question x, distillation is performed on the student rollout y∼πθ(⋅∣x)y _θ(· x). The teacher and student are evaluated on the same rollout y, but with different contexts: the student observes only the question, whereas the teacher additionally receives the deliberative content c(x)c(x) constructed in Section 3.1. Specifically, at each token position t of student rollout y, pstut p_stu^t =πθ(⋅∣x,y<t), = _θ(· x,y_<t), (6) pteat p_tea^t =sg[πθtea(⋅∣x,c(x),y<t)], =sg\! [ _ _tea(· x,c(x),y_<t) ], where sg[⋅]sg[·] stops gradients through the teacher. Since y is sampled from the current student policy, the distillation objective remains on-policy. The asymmetric contexts allow the student to internalize the benefits of reflect-and-retry. We minimize the token-level Jensen–Shannon divergence between the two predictive distributions: ℒOPD(x,y)=1|y|∑t=1|y|DJS(pteat,pstut),L_OPD(x,y)= 1|y| _t=1^|y|D_JS (p_tea^t,p_stu^t ), (7) where DJS(p,q)=12DKL(p∥m)+12DKL(q∥m),m=1/2p+1/2q.D_JS(p,q)= 12D_KL(p\|m)+ 12D_KL(q\|m), m=1/2p+1/2q. (8) EMA Teacher. The teacher must evolve with the student while providing stable distillation targets. Directly tracking it with the student can lead to drastic fluctuations during optimization, whereas keeping it frozen prevents it from benefiting from subsequent improvements. We therefore initialize the teacher from the student and update its parameters after each student optimization step using an exponential moving average: θtea←λθtea+(1−λ)θ, _tea←λ _tea+(1-λ)θ, (9) where 0<λ<10<λ<1 controls the degree of temporal smoothing. This update allows the teacher to track improvements in the student while maintaining stable targets for distillation. Outcome-Aware Sample Weighting. We divide the on-policy rollouts into three categories according to their initial correctness and the outcome of reflection and retry: • Category A: Initially correct rollouts, for which (y)=1V(y)=1 and c(x)=∅c(x)= . The teacher and student receive the same prompt and token prefix, so their predictive distributions differ only through their parameters. Distillation from the EMA teacher regularizes the student’s predictions against a temporally lagged reference, helping preserve previously correct behavior. • Category B: Initially incorrect rollouts that are corrected through reflection and retry, for which (y)=0V(y)=0 and (yT)=1V(y_T)=1. As described in Section 3.1, the teacher conditions on x, the history AT−1A_T-1, and the final reflection τT _T, giving c(x)=(AT−1,τT)c(x)=(A_T-1, _T). Its token-level predictive distributions are evaluated along the original student rollout y, rather than the corrected attempt yTy_T. The successful attempt yTy_T provides outcome-level evidence that this context c(x)c(x) can support a correct response, motivating its use for corrective distillation. • Category C: Initially incorrect rollouts that remain unresolved after reflection and retry. Teacher distributions are computed in the same way as for Category B, but the resulting context is not associated with a verified successful attempt. Although such contexts may contain useful partial reasoning, their reliability as distillation supervision is less certain than that of Category B. Let AD_A, BD_B, and CD_C denote the three category-specific subsets of the current training batch. We define the category-level OPD loss as ℒg=1|g|∑(x,y)∈gℒOPD(x,y),|g|>0,0,|g|=0,g∈A,B,C.L_g= cases 1|D_g| _(x,y) _gL_OPD(x,y),&|D_g|>0,\\[6.0pt] 0,&|D_g|=0, cases g∈\A,B,C\. (10) The overall distillation objective is ℒdistill=wAℒA+wBℒB+wCℒC,L_distill=w_AL_A+w_BL_B+w_CL_C, (11) where wAw_A, wBw_B, and wCw_C are the prescribed category weights. An absent category contributes neither loss nor gradient, and the weights of the remaining categories are not renormalized. For each nonempty category, averaging normalizes its loss by the category size, while its coefficient in the overall objective remains wgw_g. 4 Experiment 4.1 Experiment setup Models and datasets. We evaluate the within-dataset held-out performance in science question answering and tool use, and evaluate the cross-dataset performance in mathematical reasoning: • Within-Dataset Held-Out Performance. For science question answering, we use the undergraduate-level reasoning subsets (L3) of SciKnowEval (Feng et al., 2024) covering Physics and Materials. For tool-use, we use ToolAlpaca (Tang et al., 2023), where the model is given a tool API specification and a natural-language user request and is required to generate the corresponding structured tool call. We follow the same split construction and evaluation configurations recommended in Hübotter et al. (2026). Dataset sources and acquisition details are in Appendix A. • Cross-Dataset Performance. For mathematical reasoning, we follow OPSD (Zhao et al., 2026a) and train on up to 30,000 problem–solution pairs sampled from the mathematical reasoning subset of OpenThoughts (Guha et al., 2026). Each training example contains a mathematical problem and a step-by-step solution. We evaluate the trained model on external competition-level benchmarks, including AIME 2024, AIME 2025, and the February HMMT 2025 (Dekoninck et al., 2026). Dataset construction and acquisition details are in Appendix A. Baselines. We compare ReTeach with four post-training baselines, covering the standard RLVR method and the most widely compared on-policy self-distillation methods to date, with CEPO being the most recent (May 2026). We also report the initial checkpoint as a non-post-trained reference: • GRPO (Shao et al., 2024) is a standard reinforcement learning with verifiable rewards (RLVR) method. It computes group-relative advantages from sequence-level verifier rewards and applies them in a token-level policy optimization objective. • OPSD (Zhao et al., 2026a) is an on-policy self-distillation method. It aligns the student policy with a positive teacher distribution associated with verifier-accepted responses using a per-token KL-divergence objective. • RLSD (Yang et al., 2026) uses a positive teacher associated with verifier-accepted responses to construct a stop-gradient likelihood ratio, which rescales the magnitude of the token-level signal. • CEPO (Heakl et al., 2026) builds a contrastive token-level weight from a positive teacher on verifier-accepted responses and a negative teacher on verifier-rejected rollouts • RESD (Zhang et al., 2026) builds the teacher context from environment feedback verified against the dataset ground truth, by conditioning the self-teacher on a self-generated reflection over each failure, a persistent playbook of curated lessons, and cached successful rollouts. Implementation Details. We conduct all experiments initialized from Qwen3-8B (Yang et al., 2025), using LoRA (Hu et al., 2022) on 8 H20 GPUs. To keep a fair and sound comparison, we follow the training and evaluation protocols widely adopted in prior work (Zhao et al., 2026a; Hübotter et al., 2026), covering the decoding configuration and the reported metrics. We list the full set of hyperparameters in Table 2 and Table 3. Full details are provided in Appendix A. 4.2 Main Results Table 1 summarizes performance under within-dataset and cross-dataset evaluation. ReTeach ranks first on five of the six benchmarks and second on SciKnowEval-Material, achieving the highest overall average of 65.50%65.50\%, exceeding GRPO and the strongest baseline overall, CEPO, by 1.391.39 and 1.351.35 percentage points, respectively. Relative to CEPO, ReTeach also has a lower seed-level standard deviation on most datasets. Within datasets, ReTeach performs best on Physics and ToolAlpaca, improving over GRPO by 2.282.28 and 1.871.87 percentage points, respectively, while CEPO leads on Material. For cross-dataset evaluation, ReTeach achieves the highest avg@12 on all three math benchmarks, with a mean accuracy of 65.62%65.62\%, compared with 64.32%64.32\% for GRPO and 62.93%62.93\% for CEPO. Together, these results demonstrate competitive within-dataset performance and strong cross-dataset generalization without supplying reference reasoning traces or ground-truth answers to the teacher. Table 1: Performance comparison across cross-dataset and within-dataset evaluation. We report the mean±std_ over three random seeds. The cross-dataset block reports Avg@12 on three competition-level math benchmarks under the same metric recommended in (Zhao et al., 2026a). The within-dataset block reports Avg@16 on the science and tool-use tasks under the configuration of (Hübotter et al., 2026). Bold marks the best mean in each row, and underline marks the second best. The within-dataset results of OPSD are marked as −− [rgb]0.4,0.4,0.4--, because these datasets provide no reference CoT; its Total Average is therefore the mean over the three math benchmarks only. Method Qwen3-8B GRPO OPSD RLSD RESD CEPO ReTeach Within SciKnowEval-Physics 59.84±0.32 61.96±1.25 −− [rgb]0.4,0.4,0.4-- 61.90±0.83 61.09±0.54 63.39±1.45 64.24±1.05 SciKnowEval-Material 64.89±1.50 68.91±2.03 −− [rgb]0.4,0.4,0.4-- 68.35±0.39 68.02±0.97 70.83±1.35 69.17±0.97 ToolUse-ToolAlpaca 57.90±0.21 60.85±0.30 −− [rgb]0.4,0.4,0.4-- 60.08±0.77 60.94±0.37 61.86±1.62 62.72±0.20 Cross Math-AIME24 76.94±0.57 76.48±0.47 76.95±0.61 76.94±0.99 75.83±0.27 76.11±1.64 78.61±0.99 Math-AIME25 66.94±0.67 70.37±0.69 67.67±2.21 69.72±0.68 66.11±0.70 68.52±0.34 70.46±1.02 Math-HMMT25 44.72±0.41 46.11±0.60 46.67±0.22 46.39±0.68 44.35±0.37 44.17±1.71 47.78±1.42 Total Average 61.87 64.11 63.76 63.90 62.72 64.15 65.50 4.3 Ablation Studies and Discussions Effect of Weighting Unresolved Rollouts. Our outcome-aware weighting uses whether the teacher ultimately solves a problem through reflection and retry as a proxy for the usefulness of the resulting distillation signal. We therefore examine whether rollouts that remain unresolved (Category C) should contribute to distillation despite the absence of a verified successful outcome. To assess their contribution, we fix wA=wB=0.5w_A=w_B=0.5 and vary wC∈0,0.2,0.5w_C∈\0,0.2,0.5\, where wC=0w_C=0 excludes unresolved rollouts from the distillation objective and wC=0.5w_C=0.5 assigns equal weights to the three category-level mean losses. Figure 3 reports performance across three training seeds on the three Math benchmarks, Tool-Use, and the two SciKnowEval datasets. On the Math and SciKnowEval benchmarks, including unresolved rollouts provides no consistent improvement, and increasing wCw_C to 0.50.5 further reduces performance. In contrast, Tool-Use benefits from wC=0.2w_C=0.2 but deteriorates at wC=0.5w_C=0.5. In the inspected Tool-Use rollouts, most predicted function calls have required arguments that match the reference, but differ in optional argument values and are therefore rejected by the exact-match verifier. These cases suggest that unresolved contexts can retain useful partial information despite failing outcome verification, which may explain the benefit of assigning them a modest weight. However, potentially useful information in Category C is mixed with errors and noise, so increasing wCw_C may amplify unreliable supervision and degrade student performance. Overall, the results support task-dependent weighting of unresolved rollouts rather than uniformly treating their supervision as either useful or unusable. Figure 3: Ablation on the weight wCw_C of unresolved rollouts for ReTeach on Qwen3-8B, with wA=wB=0.5w_A=w_B=0.5 fixed. Lines and shaded bands show the mean and standard deviation over three seeds. Effectiveness of Explicit Reflection. At each reflection-and-retry round, the teacher generates an explicit reflection τk _k between <reflect> and </reflect>, followed by a new attempt yky_k (Eq. (4)). We ablate this reflection step while retaining iterative retries: at round k, the teacher samples directly from the unsuccessful-attempt history Ak−1A_k-1: yk∼πθtea(⋅∣x,Ak−1).y_k _ _tea (· x,A_k-1 ). (12) The teacher context used for distillation is correspondingly reduced to c(x)=AT−1c(x)=A_T-1, without the final reflection τT _T. As shown by w/o Reflection in Figure 4, removing explicit reflection reduces student accuracy from 64.24%64.24\% to 62.91%62.91\%, a drop of 1.331.33 percentage points. This result supports the contribution of explicit reflection to ReTeach’s self-distillation process and is consistent with prior findings that explicit reflection can serve as feedback to improve model performance across successive attempts (Madaan et al., 2023; Shinn et al., 2023). Figure 4: Ablation of EMA and reflection designs on SciKnowEval-Physics (avg@16). Lines and bands show the mean and standard deviation over three seeds; the dashed line marks the base model Qwen3-8B. Effectiveness of EMA Target Network. The EMA teacher stabilizes distillation targets while incorporating student improvements. To isolate its effect, we retain multi-round reflection-and-retry and compare it with a tracking teacher synchronized with the student and a teacher frozen at the base model (Figure 4). (1) The tracking teacher undermines the training stability. With a tracking teacher, the student fails to outperform the base model and gradually declines as training progressed. Two mechanisms help explain this result. First, direct synchronization transfers the student’s parameter changes to the teacher without temporal smoothing, causing the distillation targets to change rapidly. Second, on initially correct rollouts, the teacher and student share the same prompt and token prefix, so minimizing their JSD regularizes the student’s deviation from the teacher’s predictions. This regularization gradient can appear only as the student diverges from the teacher and vanishes when their parameters and predictive distributions coincide. Figure 5 further characterizes this failure mode through teacher entropy and gradient norm. The tracking teacher’s entropy falls from 0.1290.129 to 0.0360.036 nats, while its gradient norm rises from 0.290.29 to 2.312.31 and peaks at 25.2725.27, an order of magnitude above the other configurations. With the teacher refreshed from the student at every update, the two distributions nearly coincide, and token-level divergence concentrates on positions where the student is already confident. The resulting large but uninformative gradient reinforces existing predictions rather than correcting them, consistent with the observed entropy collapse. (2) Freezing stabilizes the targets but prevents teacher adaptation. The student trained with a frozen teacher consistently outperforms the base model but trails ReTeach. Because the teacher’s parameters remain fixed, it cannot incorporate the capabilities acquired by the student to improve its own reasoning. Its entropy remains near 0.180.18 nats and its gradient norm near 0.190.19 (Figure 5), indicating stable but unadaptive supervision. The EMA teacher provides a middle ground by absorbing student improvements through a lagged update, which keeps distillation targets from changing too quickly and preserves a delayed reference on initially correct rollouts. It has the lowest and most stable gradient norm (0.080.08–0.140.14) and the highest entropy, increasing from 0.210.21 to 0.240.24 nats. These dynamics support the role of EMA in preserving target stability while allowing the teacher to track student improvements. Figure 5: Teacher entropy (left) and gradient norm (right) on SciKnowEval-Physics during training under tracking, frozen, and EMA teacher updates. Lines and bands show the mean and standard deviation over three seeds. (a) Ablation study on reflection rounds. (b) Teacher self-corrected rate with different rounds K. Figure 6: Reflection-round analysis for ReTeach on SciKnowEval-Physics. Effect of the Maximum Number of Rounds. Increasing the maximum number of reflection-and-retry rounds K gives the teacher more opportunities to correct unsuccessful attempts and potentially solve harder problems. To examine how this translates into student performance, we evaluate ReTeach on SciKnowEval-Physics with K∈1,2,3K∈\1,2,3\. As shown in Figure 6a, the corresponding avg@16 accuracies are 63.54%63.54\%, 64.24%64.24\%, and 64.38%64.38\%, respectively. Increasing K from 11 to 22 yields a gain of 0.700.70 percentage points, whereas increasing it from 22 to 33 adds only 0.140.14 percentage points, although the variance in final accuracy across training seeds decreases markedly. These results indicate diminishing returns in mean accuracy, with an additional benefit in cross-seed consistency at K=3K=3. To explain this diminishing return, we track during training the fraction of Category B examples that the teacher succeeds in correcting through reflection and retry (Figure 6b). Raising K from 11 to 22 increases this fraction substantially, whereas raising it from 22 to 33 increases it only marginally. Because additional rounds change the teacher’s supervision only for examples that are eventually corrected, the marginal gain in student accuracy closely follows the marginal increase in this fraction: once a further round recovers few additional corrected examples, it supplies little new guidance beyond what earlier rounds already provide. The third round nevertheless reduces the variability of the final accuracy across training seeds, so its remaining benefit lies more in consistency than in mean accuracy. Although K=3K=3 attains the highest mean accuracy among the tested settings, it also substantially increases teacher inference time during training. We therefore use K=2K=2 as the default to balance performance gains against training cost. 5 Related Work Self-Teacher Construction. Self-teachers provide dense distillation targets and refine token-level credit assignment in reinforcement learning. Following context distillation (Snell et al., 2022), many methods construct stronger self-teachers by conditioning them on information unavailable to the student. OPSD (Zhao et al., 2026a) conditions its teacher on reference solutions, while RLSD (Yang et al., 2026) and GEAR (Li et al., 2026a) use ground-truth answers. Extensions adapt reference granularity (Han et al., 2026) or contrast teachers conditioned on correct and incorrect answers (Heakl et al., 2026; Pan et al., 2026). Others exploit feedback or model-generated experience: SDPO (Hübotter et al., 2026) uses runtime feedback or successful rollouts from the same group; SC-GRPO (Shan et al., 2026) and CAST (Li et al., 2026b) use verified self-generated rollouts or trajectory correctness rather than reference answers. Reflection-Based Self-Improvement. Self-Refine (Madaan et al., 2023) and Reflexion (Shinn et al., 2023) revisit earlier attempts through inference-time feedback, refinement, and reflection, whereas Re-ReST (Dou et al., 2024) repairs trajectories through environment-based reflection for off-policy self-training. Beyond trajectory repair, RICL (Chen et al., 2025) generates retrospective step-level feedback and uses feedback-conditioned changes in action probabilities for credit assignment, while ROSD (Zhao et al., 2026b) uses reflection to identify error locations and restrict distillation to the corresponding suffixes. ReflectionCoder (Ren et al., 2025) distills final-code predictions conditioned on offline, execution-guided multi-round reflection sequences. Its data construction includes execution-based filtering of unresolved and initially successful attempts. RESD (Zhang et al., 2026) combines retrospective reflection with ground-truth answer feedback, a persistent playbook, and cached successful solutions; HERO (Liu et al., 2026) derives turn-level reflective hints from completed interactions and environment observations. ReTeach instead constructs teacher contexts online through multi-round reflection and retry, using only self-generated attempts and outcome-level verification, without reference answers or solutions, external diagnostic feedback, or cross-example memory. Failed retries provide new evidence for subsequent reflection, while successful correction supplies outcome-level evidence for the potential utility of the resulting teacher context. Whereas RESD rebalances successful and failed rollouts, ReTeach retains initially correct examples and further separates initially failed rollouts into reflection-corrected and unresolved categories, using correction outcomes to weight category-normalized distillation losses. 6 Conclusion In this work, we introduced ReTeach, a self-distillation framework that constructs a stronger teacher through reasoning-process asymmetry. Instead of relying on externally supplied privileged information, the teacher revisits failed student responses through multiple rounds of explicit reflection and retry. The resulting deliberative context enables the teacher to provide informative token-level targets on rollouts sampled from the current student policy. The outcome-aware classwise weighting balances the learning signals from initially correct, successfully refined, and unresolved problems, while an EMA update maintains a stable yet evolving teacher. Experiments on mathematical reasoning, science question answering, and tool-use benchmarks demonstrate the consistent effectiveness of ReTeach. Even compared with methods that leverage privileged information, such as ground-truth answers or reference solutions, ReTeach achieves the best performance on five of the six benchmarks and ranks second on the remaining one. These results show that the advantage of a self-constructed teacher need not stem solely from information asymmetry: altering how the same model reasons can itself yield an effective source of supervision. More broadly, our findings suggest that capabilities elicited through iterative deliberation can be internalized into a more efficient single-pass policy, offering a promising direction for improving reasoning models when only outcome-level verification is available. References Agarwal et al. (2024) R. Agarwal, N. Vieillard, Y. Zhou, P. Stanczyk, S. R. Garea, M. Geist, and O. Bachem On-policy distillation of language models: learning from self-generated mistakes. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: §2, §2. Chen et al. (2025) W. Chen, J. Chen, F. Tajwar, H. Zhu, X. Duan, R. Salakhutdinov, and J. Schneider Retrospective in-context learning for temporal credit assignment with large language models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: §5. Cui et al. (2026) G. Cui, L. Yuan, Z. Wang, H. Wang, Y. Zhang, J. Chen, W. Li, B. He, Y. Fan, T. Yu, Q. Xu, W. Chen, J. Yuan, H. Chen, K. Zhang, X. Lv, S. Wang, Y. Yao, X. Han, H. Peng, Y. Cheng, Z. Liu, M. Sun, B. Zhou, and N. Ding Process reinforcement through implicit rewards. Transactions on Machine Learning Research. Note: External Links: ISSN 2835-8856, Link Cited by: §2. DeepSeek-AI (2025) DeepSeek-AI DeepSeek-r1: incentivizing reasoning capability in llms via reinforcement learning. External Links: 2501.12948, Link Cited by: §1, §2. Dekoninck et al. (2026) J. Dekoninck, N. Jovanović, T. Gehrunger, K. Rögnvaldsson, I. Petrov, C. Sun, and M. Vechev Beyond benchmarks: matharena as an evaluation platform for mathematics with LLMs. In 3rd AI for Math Workshop: Toward Self-Evolving Scientific Agents, External Links: Link Cited by: Appendix A, 2nd item. Dou et al. (2024) Z. Dou, C. Yang, X. Wu, K. Chang, and N. Peng Re-ReST: reflection-reinforced self-training for language agents. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.), Miami, Florida, USA, p. 15394–15411. External Links: Link, Document Cited by: §5. Feng et al. (2024) K. Feng, K. Ding, W. Wang, X. Zhuang, Z. Wang, M. Qin, Y. Zhao, J. Yao, Q. Zhang, and H. Chen SciKnowEval: evaluating multi-level scientific knowledge of large language models. External Links: 2406.09098, Link Cited by: Appendix A, 1st item. Gu et al. (2024) Y. Gu, L. Dong, F. Wei, and M. Huang MiniLLM: knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: §2. Guha et al. (2026) E. K. Guha, R. Marten, S. Keh, N. Raoof, G. Smyrnis, H. Bansal, M. Nezhurina, J. Mercat, T. Vu, Z. R. Sprague, A. Suvarna, B. Feuer, L. L. Chen, Z. Khan, E. Frankel, S. Grover, C. Choi, N. Muennighoff, S. Su, W. Zhao, J. Yang, S. Pimpalgaonkar, K. sharma, C. C. Ji, Y. Deng, S. M. Pratt, V. Ramanujan, J. Saad-Falcon, S. Acharya, J. Li, A. Dave, A. Albalak, K. Arora, B. Wulfe, C. Hegde, G. Durrett, S. Oh, M. Bansal, S. Gabriel, A. Grover, K. Chang, V. Shankar, A. Gokaslan, M. A. Merrill, T. Hashimoto, Y. Choi, J. Jitsev, R. Heckel, M. Sathiamoorthy, A. Dimakis, and L. Schmidt OpenThoughts: data recipes for reasoning models. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: Appendix A, 2nd item. Han et al. (2026) Z. Han, T. Zhang, H. Wang, and Y. Sun Adaptive teacher exposure for self-distillation in llm reasoning. External Links: 2605.11458, Link Cited by: §1, §5. Heakl et al. (2026) A. Heakl, A. M. Shaker, Y. Mohamed, R. Elbadry, O. Fetouh, F. S. Khan, and S. Khan CEPO: rlvr self-distillation using contrastive evidence policy optimization. External Links: 2605.19436, Link Cited by: 4th item, §5. Hinton et al. (2015) G. Hinton, O. Vinyals, and J. Dean Distilling the knowledge in a neural network. External Links: 1503.02531, Link Cited by: §2. Hu et al. (2022) E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen LoRA: Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations, External Links: Link Cited by: §4.1. Hübotter et al. (2026) J. Hübotter, F. Lübeck, L. Behric, A. Baumann, M. Bagatella, D. Marta, I. Hakimi, I. Shenfeld, T. Kleine Buening, C. Guestrin, and A. Krause Reinforcement learning via self-distillation. In International Conference on Machine Learning (2026), Cited by: Appendix A, Appendix A, Appendix C, §1, §2, §2, §2, 1st item, §4.1, Table 1, §5. Kim and Rush (2016) Y. Kim and A. M. Rush Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, J. Su, K. Duh, and X. Carreras (Eds.), Austin, Texas, p. 1317–1327. External Links: Link, Document Cited by: §2. Kojima et al. (2022) T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa Large language models are zero-shot reasoners. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA. External Links: ISBN 9781713871088 Cited by: §2. Lambert et al. (2025) N. Lambert, J. Morrison, V. Pyatkin, S. Huang, H. Ivison, F. Brahman, L. J. V. Miranda, A. Liu, N. Dziri, X. Lyu, Y. Gu, S. Malik, V. Graf, J. D. Hwang, J. Yang, R. L. Bras, O. Tafjord, C. Wilhelm, L. Soldaini, N. A. Smith, Y. Wang, P. Dasigi, and H. Hajishirzi Tulu 3: pushing frontiers in open language model post-training. In Second Conference on Language Modeling, External Links: Link Cited by: §1. Li et al. (2026a) S. Li, Y. Huang, Z. Liu, Y. Li, J. Fu, L. Zhao, J. Bian, L. Zhang, J. Zhang, and R. Wang GEAR: granularity-adaptive advantage reweighting for llm agents via self-distillation. External Links: 2605.11853, Link Cited by: §5. Li et al. (2026b) Y. Li, G. Xue, Y. Guo, Y. Yuan, L. Hu, and L. Ma CAST: non-privileged clipped asymmetric self-teaching with advantage flipping for grpo. External Links: 2606.00172, Link Cited by: §5. Lightman et al. (2024) H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe Let’s verify step by step. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: §2. Liu et al. (2026) H. Liu, Y. Zhang, X. Li, B. Lyu, and J. Shang HERO: hindsight-enhanced reflection from environment observations for agentic self-distillation. External Links: 2606.11559, Link Cited by: §1, §5. Lu and Lab (2025) K. Lu and T. M. Lab On-policy distillation. Thinking Machines Lab: Connectionism. Note: https://thinkingmachines.ai/blog/on-policy-distillation External Links: Document Cited by: §1, §2, §2. Madaan et al. (2023) A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, et al. Self-refine: iterative refinement with self-feedback. Advances in neural information processing systems 36, p. 46534–46594. Cited by: §1, §3.1, §4.3, §5. Pan et al. (2026) L. Pan, S. Tao, Y. Zhai, L. Zhang, Z. Liu, B. Ding, A. Liu, and L. Wen RLCSD: reinforcement learning with contrastive on-policy self-distillation. External Links: 2606.11709, Link Cited by: §5. Radha et al. (2024) S. K. Radha, Y. N. Jelyani, A. Ghukasyan, and O. Goktas Iteration of thought: leveraging inner dialogue for autonomous large language model reasoning. External Links: 2409.12618, Link Cited by: §3.1. Ren et al. (2025) H. Ren, M. Zhan, Z. Wu, A. Zhou, J. Pan, and H. Li ReflectionCoder: learning from reflection sequence for enhanced one-off code generation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July 27 - August 1, 2025, W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), p. 9999–10020. External Links: Link, Document Cited by: §1, §5. Shan et al. (2026) Y. Shan, Y. Guo, Z. Cheng, Z. Liu, X. Zhu, X. Wang, J. Yao, W. Lin, H. Wang, and H. Huang Learning from own solutions: self-conditioned credit assignment for reinforcement learning with verifiable rewards. External Links: 2606.18810, Link Cited by: §5. Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo DeepSeekMath: pushing the limits of mathematical reasoning in open language models. External Links: 2402.03300, Link Cited by: 1st item. Shinn et al. (2023) N. Shinn, F. Cassano, A. Gopinath, K. R. Narasimhan, and S. Yao Reflexion: language agents with verbal reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems, External Links: Link Cited by: §1, §4.3, §5. Snell et al. (2022) C. Snell, D. Klein, and R. Zhong Learning by distilling context. External Links: 2209.15189, Link Cited by: §5. Tang et al. (2023) Q. Tang, Z. Deng, H. Lin, X. Han, Q. Liang, and L. Sun ToolAlpaca: generalized tool learning for language models with 3000 simulated cases. External Links: 2306.05301, Link Cited by: Appendix A, 1st item. Uesato et al. (2022) J. Uesato, N. Kushman, R. Kumar, H. F. Song, N. Y. Siegel, L. Wang, A. Creswell, G. Irving, and I. Higgins Solving math word problems with process- and outcome-based feedback. External Links: 2211.14275, Link Cited by: §2. Wang et al. (2026) M. Wang, H. Zhao, W. Liu, L. Yang, G. Liu, H. Guo, G. Xie, G. Meng, H. Liu, and F. Zhu Denser ≠ better: limits of on-policy self-distillation for continual post-training. External Links: 2607.01763, Link Cited by: §2. Wang et al. (2024) P. Wang, L. Li, Z. Shao, R. Xu, D. Dai, Y. Li, D. Chen, Y. Wu, and Z. Sui Math-shepherd: verify and reinforce LLMs step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, p. 9426–9439. External Links: Link, Document Cited by: §2. Wei et al. (2022) J. Wei, X. Wang, D. Schuurmans, M. Bosma, brian ichter, F. Xia, E. H. Chi, Q. V. Le, and D. Zhou Chain of thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho (Eds.), External Links: Link Cited by: §2. Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. Li, T. Tang, W. Yin, X. Ren, X. Wang, X. Zhang, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Zhang, Y. Wan, Y. Liu, Z. Wang, Z. Cui, Z. Zhang, Z. Zhou, and Z. Qiu Qwen3 technical report. External Links: 2505.09388, Link Cited by: §4.1. Yang et al. (2026) C. Yang, C. Qin, Q. Si, M. Chen, N. Gu, D. Yao, Z. Lin, W. Wang, J. Wang, and N. Duan Self-distilled rlvr. External Links: 2604.03128, Link Cited by: §1, §2, 3rd item, §5. Yu et al. (2024) L. Yu, W. Jiang, H. Shi, J. YU, Z. Liu, Y. Zhang, J. Kwok, Z. Li, A. Weller, and W. Liu MetaMath: bootstrap your own mathematical questions for large language models. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: §2, §2. Zhang et al. (2025) L. Zhang, A. Hosseini, H. Bansal, M. Kazemi, A. Kumar, and R. Agarwal Generative verifiers: reward modeling as next-token prediction. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §2. Zhang et al. (2026) Y. Zhang, S. Li, C. Yu, Q. Lu, S. Jin, C. Dong, H. Liu, I. Hong, X. Li, Z. Shi, B. Yin, and J. Shang Learning with rare success but rich feedback via reflection-enhanced self-distillation. External Links: 2605.12741, Link Cited by: §1, 5th item, §5. Zhao et al. (2026a) S. Zhao, Z. Xie, M. Liu, J. Huang, G. Pang, F. Chen, and A. Grover Self-distilled reasoner: on-policy self-distillation for large language models. In Forty-third International Conference on Machine Learning, External Links: Link Cited by: Appendix A, Appendix A, §1, §2, 2nd item, 2nd item, §4.1, Table 1, §5. Zhao et al. (2026b) Z. Zhao, X. Ma, L. Yang, Y. Feng, D. Shi, J. He, X. Xin, Z. Ren, and X. Wu ROSD: reflective on-policy self-distillation for language model reasoning across domains. External Links: 2605.28014, Link Cited by: §5. Appendix A Experiment details Datasets. For mathematical reasoning, we follow the training recipe of OPSD (Zhao et al., 2026a): the mathematical-reasoning subset of OpenThoughts (Guha et al., 2026), sampling up to 30,000 problem–solution pairs with chain-of-thought. Cross-dataset evaluation uses AIME 2024, AIME 2025, and February HMMT 2025 (Dekoninck et al., 2026), the same three benchmarks as OPSD. For science question answering we use the undergraduate-level (L3) Physics and Materials subsets of SciKnowEval (Feng et al., 2024), and for tool use we use ToolAlpaca (Tang et al., 2023). On these two tasks we follow the within-dataset held-out splits and evaluation protocol of SDPO (Hübotter et al., 2026). For training-time outcome verification, we do not follow OPSD or SDPO in exposing a reference CoT or the ground-truth answer to the teacher. Like GRPO, ReTeach uses an RLVR outcome verifier (y)∈0,1V(y)∈\0,1\ that only marks an attempt as correct or incorrect. Category weights. Table 2 sets wA=wB=0.5w_A=w_B=0.5 on all tasks, wC=0w_C=0 on Math and Science, and wC=0.2w_C=0.2 on Tool-Use. These wCw_C values were chosen from the ablation in Figure 3, not from a separate validation split. Unresolved rollouts did not improve Math or SciKnowEval, while Tool-Use improved at wC=0.2w_C=0.2 and degraded at wC=0.5w_C=0.5. We therefore discard Category C on closed-answer tasks and keep a modest wCw_C on Tool-Use. Detailed results and the analysis of this effect are given in Section 4.3. Training setup. We build all methods on Qwen3-8B and train on three tasks, Math, Tool-Use, and Science. The training configuration is shared across the three tasks and follows OPSD (Zhao et al., 2026a), including the thinking mode of each policy and the sampling hyperparameters. The teacher therefore keeps its thinking mode enabled throughout training while the student is trained with it disabled. For the GRPO-style baselines including GRPO, CEPO, RLSD, and RESD, we use 8 rollouts per question, and set the clipping thresholds to ϵlow=0.2 _low=0.2 and ϵhigh=0.28 _high=0.28. All remaining training hyperparameters are identical to those of ReTeach and appear in Table 2. Evaluation protocol. The three tasks are reported following the protocol of the closest published method on each dataset, which keeps every comparison against numbers produced under the same protocol. On the within-dataset evaluations, Science and Tool-Use, we follow SDPO (Hübotter et al., 2026) and report avg@16 with the thinking mode disabled, consistent with the setting in which the student is trained. On the cross-dataset Math evaluation, we follow OPSD and report results on the three benchmarks used there, including AIME24, AIME25, and HMMT25. We evaluate with avg@12 and enable the thinking mode at inference, so that our numbers remain directly comparable to those reported by OPSD. Table 3 summarizes both protocols. Table 2: Training configuration for ReTeach Parameter Math Science Tool-Use Sample Weight A 0.5 0.5 0.5 Sample Weight B 0.5 0.5 0.5 Sample Weight C 0 0 0.2 student max tokens 8192 1024 1024 teacher max tokens 8192 4096 4096 Effective Batch Size 32 32 32 Data Number of Rollouts 1 1 1 LoRA Rank 64 64 64 LoRA α 128 128 128 LoRA LoRA Target Modules q_proj, k_proj, v_proj, o_proj gate_proj, up_proj, down_proj Inference Engine vLLM vLLM vLLM Temperature 1.1 1.1 1.1 Top-p 0.95 0.95 0.95 Rollout Top-K 20 20 20 Teacher Think Mode Enabled Enabled Enabled Student Think Mode Disabled Disabled Disabled Max Grad Norm 0.1 0.1 0.1 Teacher Temperature 0.7 0.7 0.7 EMA Decay λ 0.9999 0.9999 0.9999 Training Training Steps 400 400 400 Table 3: Evaluation configuration for all methods Parameter Math Science Tool-Use Data Samples per Prompt 12 16 16 Max New Tokens 38912 38912 38912 Evaluation Think Mode Enabled Disabled Disabled Temperature 0.6 0.6 0.6 Top-p 0.95 0.95 0.95 Top-k -1 -1 -1 Appendix B Data Samples We train and evaluate on four data sources: Math (competition-level mathematics), and the Physics, Material, and Tool-Use subsets used in our science and tool-use evaluation. This section shows one training question from each source, reproduced verbatim from the corresponding data file. Each question is paired with a verifiable outcome only; no reference solution or human-written chain of thought is used anywhere in training. Data sample: Math Let proposition P: The sets of solutions to the inequalities a1x2+b1x+c1>0a_1x^2+b_1x+c_1>0 and a2x2+b2x+c2>0a_2x^2+b_2x+c_2>0 are identical. Proposition Q: a1a2=b1b2=c1c2 a_1a_2= b_1b_2= c_1c_2. Then proposition Q is a () condition for proposition P. (A) Necessary and sufficient condition (B) Sufficient but not necessary condition (C) Necessary but not sufficient condition (D) Neither necessary nor sufficient condition Data sample: Physics (SciKnowEval) Question: A container with a volume of 18L18\,L contains a gas with a temperature of 270oC270^oC. If the temperature of the gas changes to 350oK350\,^oK without any change in pressure, what must the container’s new volume be? A: 15.8 L B: 11.6 L C: 13.2 L D: 9.4 L This is a multiple-choice question with exactly one correct option. Select the single best option and report ONLY its letter (A, B, C, or D) as your final answer — do not restate the option text. Data sample: Material (SciKnowEval) Given the following crystal structure parameters for the material SmP4HO12 (Material ID: mp-542237), calculate the volume of the unit cell (in Å3). Lattice Parameters: a = 7.14 Å b = 8.58 Å c = 8.86 Å α = 99.11∘ β = 112.21∘ γ = 106.66∘ A: 458.96 Å3 B: 465.00 Å3 C: 482.10 Å3 D: 473.25 Å3 This is a multiple-choice question with exactly one correct option. Select the single best option and report ONLY its letter (A, B, C, or D) as your final answer — do not restate the option text. Data sample: Tool-Use (ToolAlpaca) Your task is to answer the user’s question using available tools. You have access to the following tools: Name: NetworkCalc Description: Network calculators, including subnets, DNS, binary, and security tools Documentation: binaryToDecimal: Converts a binary number to its decimal equivalent. Parameters: "binaryNumber": "Required. string. The binary number to convert to decimal." Output: Decimal equivalent of the binary number. - Format: application/json - Structure: ObjectdecimalNumber [… further tools omitted for space …] Begin! Question: I have a binary number, 101001110, and I need to know its decimal equivalent. Can you convert it for me? Appendix C Prompt Templates ReTeach uses two prompt roles that differ in how the model is asked to reason, not in what information it is given. The student answers in a single turn with thinking disabled, and therefore emits a solution followed by the final answer without any <think> block. The teacher answers with thinking enabled. When it is correcting a failed attempt, the prompt requires it to first write a structured reflection enclosed in <reflect> tags and then to solve the problem step by step, with the final answer in . Because the teacher’s native thinking is enabled, the model’s internal <think> block appears before the externally required <reflect> block in the raw generation; the two are complementary, since the former is the model’s own deliberation and the latter is the structured diagnosis we ask for. Neither role ever sees a reference solution. For the Math, Physics, and Material datasets, the outputs are straightforward to parse, so we use the generic student and teacher templates shown below. Student prompt (shared by Math, Physics, Material) PROBLEM: <question> Please reason step by step and put your final answer within . Teacher reflection prompt (shared by Math, Physics, Material) PROBLEM: <question> Below are previous incorrect attempts. Their final answers have ALL been verified WRONG that you must avoid: <Previous_Incorrect_Begin> <Failed Attempts> <Previous_Incorrect_End> OUTPUT FORMAT: First, reflect on the previous attempts. Your reflection MUST BE enclosed within <reflect> </reflect> tags, and inside it you must: name each attempt’s failure type — repeated forbidden answer / wrong formula / calculation error / overlooked condition; re-derive the governing formula and re-check the numbers of the flawed step; state concretely how to correct it. After </reflect>, solve the problem step by step and put your final answer within . IMPORTANT: Your solution MUST be self-contained and stand entirely on its own. Write it as if you are solving the problem for the first time without any prior context. For the tool-use dataset, the model must be prompted in a task-specific manner (Hübotter et al., 2026) so that its output can be parsed as a ReAct-style transcript. We therefore provide the prompt templates below. They follow the same structure as those used for the other datasets, with the only addition being a specification of the required tool-call format. Student prompt (Tool-Use) PROBLEM: <question> Your final answer is the tool call itself. Inside , write ONLY the tool call transcript, in exactly this format and nothing else: Action: <one of the tool names listed above> Action Input: <a single-line JSON object holding the arguments> If the question needs several tool calls, put every Action / Action Input pair one after another inside that same , in the order they should be executed. Teacher reflection prompt (Tool-Use) PROBLEM: <question> Below are previous incorrect attempts. Their final tool calls have ALL been verified WRONG — treat them as FORBIDDEN tool calls you must never produce again. <Previous_Incorrect_Begin> <Failed Attempts> <Previous_Incorrect_End> OUTPUT FORMAT: First, reflect on the previous attempts. Your reflection MUST BE enclosed within <reflect> </reflect> tags. An attempt is wrong if ANY of these holds — check them in this order: (a) MISSING or EXTRA call: the question often needs SEVERAL calls, and you must list them ALL in one transcript. (b) WRONG Action Input: every argument value must match the question CHARACTER FOR CHARACTER. (c) WRONG Action name: copy the tool name verbatim from the documentation, including its exact capitalization. After </reflect>, solve the problem step by step and put your final answer within . IMPORTANT: Your solution MUST be self-contained and stand entirely on its own. Write it as if you are solving the problem for the first time without any prior context. Teacher’s reflection prompt is self-contained. Although the teacher has access to the failed trajectory and the subsequent reflection, its prompt should only support the construction of a stronger response. We therefore require the teacher to produce a self-contained final solution and answer, following the same standalone response format as the student. In particular, the teacher must not refer to the failed history or information introduced during reflection. Otherwise, such context-dependent references would enter the distillation target and confuse the student. Appendix D Generation Samples We present a real training sample from the math dataset to show how ReTeach locates and corrects a reasoning mistake through reflect and retry. The example lists the inputs and outputs of the student and the teacher separately. The student receives the original question and produces an initial attempt that is verified as wrong. The teacher is then prompted with that attempt together with the verified-wrong answer. It first emits its native <think> block under thinking mode, followed by the <reflect> block fixed in the prompt template, the retried solution, and the final answer. To facilitate comparison between the two responses, we highlight in red boxesthe incorrect part of the student’s response and the part where the teacher identifies and corrects the error. The student enumerates only the factorizations in which both factors are equal, taking 44 as 2×22× 2 and 99 as 3×33× 3, and therefore omits the decompositions 4=1×44=1× 4 and 9=1×99=1× 9. Every valid pair involving 11 is thus excluded and the resulting count is 3, which is verified as wrong. Within the <reflect> block, the teacher attributes the mistake to an incorrect count of valid pairs, particularly those involving 1, and then produces a correct derivation whose final answer matches the ground truth. Throughout the process, no external chain-of-thought or gold answer is introduced. Generation sample: Student rollout and initial attempt on Math PROBLEM: Consider the cards A,2,⋯,J,Q,KA,2,·s,J,Q,K as the numbers 1,2,⋯,11,12,131,2,·s,11,12,13. If we take the 13 cards of spades and 13 cards of hearts together and randomly draw 2 cards, what is the probability that the two cards are of the same suit and the product of the two numbers is a perfect square? Please reason step by step and put your final answer within . STUDENT ATTEMPT: We are given 13 cards from spades and 13 cards from hearts, each numbered from 1 to 13 (i.e., representing numbers 1 through 13). So, in total, there are 26 cards. We are to compute the probability that: The two cards drawn are of the same suit (both spades or both hearts); The product of the two numbers is a perfect square Step 1: Total number of ways to draw 2 cards. We choose 2 cards from the 26 cards: Total outcomes=(262)=26⋅252=325Total outcomes= 262= 26· 252=325 Step 2: Favorable outcomes Let’s denote: - S: set of spade cards 1,2,…,13\1,2,...,13\ - H: set of heart cards 1,2,…,13\1,2,...,13\ - So, we have 2 sets of 13 cards each, one for spades and one for hearts. Case 1: Both cards are from spades We need to count the number of pairs (a,b)(a,b) such that a,b∈Sa,b∈ S and abab is a perfect square. Similarly for Case 2: Both cards are from hearts. Since the set of numbers from 1 to 13 is the same for both suits, the number of such favorable pairs will be the same in both cases. So, we can compute the number of such pairs from one suit (say, spades), and then multiply by 2. Step 3: How many pairs from a single suit (say, spades) have a product that’s a perfect square? We need to count the number of pairs (a,b)(a,b) where a,b∈1,2,…,13a,b∈\1,2,...,13\, and abab is a perfect square. So, for each pair (a,b)(a,b) with a≤ba≤ b, we check if abab is a perfect square. Let’s list all possible pairs and identify those whose product is a perfect square. Step 3.1: Check all pairs of integers from 1 to 13 and see which products are perfect squares. Let’s consider all 1≤a<b≤131≤ a<b≤ 13, and check if abab is a perfect square. Note: We will only consider unordered pairs, since (a,b)(a,b) is the same as (b,a)(b,a) for the product. Let’s list the perfect squares leqleq 169 (since the maximum product is 13⋅13=16913· 13=169): Perfect squares between 1 and 169 are: 1,4,9,16,25,36,49,64,81,100,121,144,1691,4,9,16,25,36,49,64,81,100,121,144,169 So, we want pairs (a,b)(a,b) such that abab is one of these values. Now we search for all such pairs (a,b)(a,b) with a<ba<b and ab∈1,4,9,16,…,169ab∈\1,4,9,16,...,169\. We’l compute all such pairs. Let’s proceed with an approach: For each perfect square s, find all factor pairs (a,b)(a,b) with a<ba<b, a⋅b=sa· b=s, and 1≤a<b≤131≤ a<b≤ 13 Let’s proceed systematically: s=1s=1: a=1a=1, b=1b=1 → discard since a=ba=b s=4s=4: 1⋅4,2⋅21· 4,2· 2 → discard 2⋅22· 2 because a=ba=b s=9s=9: 1⋅9,3⋅31· 9,3· 3 → discard 3⋅33· 3 s=16s=16: 1⋅161· 16, 2⋅82· 8, 4⋅44· 4 → valid: (1,16)(1,16), (2,8)(2,8) s=25s=25: 1⋅251· 25, 5⋅55· 5 → discard 5⋅55· 5 s=36s=36: 1⋅361· 36, 2⋅182· 18, 3⋅123· 12, 4⋅94· 9, 6⋅66· 6 → valid: (1,36)(1,36), (2,18)(2,18), (3,12)(3,12), (4,9)(4,9) s=49s=49: 1⋅491· 49, 7⋅77· 7 → discard s=64s=64: 1⋅641· 64, 2⋅322· 32, 4⋅164· 16, 8⋅88· 8 → valid: (1,64)(1,64), (2,32)(2,32), (4,16)(4,16) s=81s=81: 1⋅811· 81, 3⋅273· 27, 9⋅99· 9 → valid: (1,81)(1,81), (3,27)(3,27) s=100s=100: 1⋅1001· 100, 2⋅502· 50, 4⋅254· 25, 5⋅205· 20, 10⋅1010· 10 → valid: (1,100)(1,100), (2,50)(2,50), (4,25)(4,25), (5,20)(5,20) s=121s=121: 1⋅1211· 121, 11⋅1111· 11 → discard s=144s=144: 1⋅1441· 144, 2⋅722· 72, 3⋅483· 48, 4⋅364· 36, 6⋅246· 24, 8⋅188· 18, 12⋅1212· 12 → valid: (1,144)(1,144), (2,72)(2,72), (3,48)(3,48), (4,36)(4,36), (6,24)(6,24), (8,18)(8,18) s=169s=169: 1⋅1691· 169, 13⋅1313· 13 → discard Now, we eliminate any pair where b>13b>13, since the maximum number on the cards is 13. So, for each of the valid pairs above, if b≤13b≤ 13, count the pair. Go through them again and only include pairs where both a and b are in [1,13][1,13] Step 3.2: List of valid pairs where both numbers are in 1 to 13 and product is perfect square Now let’s go back and find all pairs (a,b)(a,b), a<ba<b, such that a,b∈[1,13]a,b∈[1,13], and abab is a perfect square. Let’s consider all perfect squares from 12=11^2=1 to 132=16913^2=169, and for each one, list its factor pairs (a,b)(a,b) with a<ba<b and a,b≤13a,b≤ 13. We’l collect all such (a,b)(a,b) and count them. s=4s=4: 2⋅22· 2 → a=ba=b, so discard s=9s=9: 3⋅33· 3 → same, discards=16s=16: 2⋅82· 8, 1⋅161· 16 → 16>1316>13 → only (2,8)(2,8) s=36s=36: 4⋅94· 9, 3⋅123· 12 → valid s=64s=64: 4⋅164· 16, 8⋅88· 8 → 16>1316>13, 8 * 8 = same → only (1,64)(1,64), invalid s=81s=81: 3⋅273· 27, 27>1327>13 → only (1,81)(1,81) s=100s=100: 4⋅254· 25, 5⋅205· 20 → 25, 20>1320>13 → discard s=121s=121: 11⋅1111· 11 → discard s=144s=144: 6⋅246· 24, 8⋅188· 18, 4⋅364· 36 → all>13all>13 → discard s=169s=169: 13⋅1313· 13 → discard So the valid pairs are: (2,8)(2,8) → 16 (3,12)(3,12) → 36 (4,9)(4,9) → 36 That’s 3 valid pairs per suit. Step 4: Total favorable outcomes Each suit has 3 such valid pairs → 2 suits → total favorable outcomes: Favorable=3 (spades)+3 (hearts)=6Favorable=3 (spades)+3 (hearts)=6 Step 5: Compute the probability Probability=FavorableTotal=6325Probability= FavorableTotal= 6325 Final Answer: 6325 6325 VERIFIED: Wrong Generation sample: Teacher reflection and retry over student’s failed attempts on Math PROBLEM: Consider the cards A,2,⋯,J,Q,KA,2,·s,J,Q,K as the numbers 1,2,⋯,11,12,131,2,·s,11,12,13. If we take the 13 cards of spades and 13 cards of hearts together and randomly draw 2 cards, what is the probability that the two cards are of the same suit and the product of the two numbers is a perfect square? Please reason step by step and put your final answer within . Below are previous incorrect attempts. Their final answers have ALL been verified WRONG that you must avoid: <Previous_Incorrect_Begin> (Student’s reply above)⋯·s Final Answer: 6325 6325 <Previous_Incorrect_End> OUTPUT FORMAT: First, reflect on the previous attempts. Your reflection MUST BE enclosed within <reflect> </reflect> tags, and inside it you must: name each attempt’s failure type — repeated forbidden answer / wrong formula / calculation error / overlooked condition; re-derive the governing formula and re-check the numbers of the flawed step; state concretely how to correct it. After </reflect>, solve the problem step by step and put your final answer within . IMPORTANT: Your solution MUST be self-contained and stand entirely on its own. Write it as if you are solving the problem for the first time without any prior context. TEACHER REPLY: <think> Okay, let’s try to figure out this probability problem. The question is about drawing two cards from a combined deck of spades and hearts, each numbered 1 to 13. We need the probability that both cards are of the same suit and their product is a perfect square. The previous attempt gave 6/325, but that’s forbidden. Let me see where they went wrong. First, the total number of ways to draw two cards from 26 is indeed C(26,2) = 325. That part is correct. The problem is in counting the favorable outcomes. The previous attempt found 3 pairs per suit, leading to 6 total, but maybe they missed some pairs? ⋯·s Another possible mistake: perhaps they considered ordered pairs? But no, since they were considering unordered pairs. Or maybe they thought that (1,4) and (4,1) are different? But in combinations, they are same. So no. So the main error was missing those two pairs. </think> <reflect> The previous attempt calculated the total number of ways to draw two cards from 26 cards as (262)=325 262=325, which is correct. It then proceeded to identify pairs of numbers from 1 to 13 that multiply to a perfect square. However, it made a critical error in counting the number of such pairs, leading to an incorrect result of 6325 6325. Key Mistakes: 1. **Missed Valid Pairs**: The attempt only identified three valid pairs per suit, but in reality, there are **five** such pairs per suit. 2. **Overlooked Condition**: The problem specifically requires the **same suit** and **product is a perfect square**. The previous attempt failed to account for all valid combinations, particularly those involving 1 (e.g., 1×4=41× 4=4, 1×9=91× 9=9). 3. **Calculation Error**: The count of favorable outcomes was incorrectly limited to 3 per suit instead of the correct 5 per suit, resulting in an incorrect total of 6 instead of 10. </reflect> Step 1: Total Number of Outcomes We are drawing 2 cards from a total of 26 cards (13 spades + 13 hearts): Total outcomes=(262)=325Total outcomes= 262=325 Step 2: Count Favorable Outcomes We need to find the number of unordered pairs (a,b)(a,b) such that: 1≤a<b≤131≤ a<b≤ 13 a⋅ba· b is a perfect square We examine all possible pairs and identify those whose product is a perfect square. The perfect squares between 1 and 13⋅13=16913· 13=169 are: 1,4,9,16,25,36,49,64,81,100,121,144,1691,4,9,16,25,36,49,64,81,100,121,144,169 Now, for each perfect square s, we find all pairs (a,b)(a,b) such that a<ba<b, a,b∈[1,13]a,b∈[1,13], and a⋅b=sa· b=s. The valid pairs are: 1⋅4=41· 4=4 1⋅9=91· 9=9 2⋅8=162· 8=16 3⋅12=363· 12=36 4⋅9=364· 9=36 So, there are **5 valid pairs** per suit. Since the cards are in two suits (spades and hearts), the total number of favorable outcomes is: Favorable outcomes=5 (spades)+5 (hearts)=10Favorable outcomes=5 (spades)+5 (hearts)=10 Step 3: Compute the Probability Probability=Favorable outcomesTotal outcomes=10325Probability= Favorable outcomesTotal outcomes= 10325 Simplifying the fraction:10325=265 10325= 265 Final Answer: 265 265 How the reflection is used at distillation time A teacher response consists of four parts: the native <think> block, the required <reflect> diagnosis, the step-by-step solution, and the final answer. If the answer after retry is still wrong, only the last two are carried into the next round as a failed attempt, so the history holds a solution and its answer. The distillation completion is always the student rollout y, as in Section 3.2. During distillation, ReTeach keeps the <think> and <reflect> blocks of the final round only, strips them from the completion, and re-attaches them as a masked assistant prefix, so the teacher is conditioned on its own deliberation when its logits are computed while the student is trained on the solution and answer alone.