Paper deep dive

LoOPSD: On-Policy Self-Distillation with Looped Teachers

Wei Fan, Qingyue Yang, Chao-Hong Tan, Qian Chen, Wen Wang, Xiangang Li, Kejiang Chen, Weiming Zhang, Nenghai Yu

Year: 2026Venue: arXiv preprintArea: cs.LGType: PreprintEmbeddings: 61

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:On-policy self-distillation improves a model using guidance from the model itself, but it still requires a teacher that provides more useful predictions than the student. Existing approaches obtain such guidance by giving the teacher additional context or instructions. We ask whether it can instead come from changing how the same model uses its own layers. To address this, we introduce LoOPSD, which searches over a model's layer execution order to construct a stronger teacher and distills its advantage into the student. Specifically, we keep the model weights unchanged while searching for layers to repeat, and use the resulting teacher's predictions on student-generated prefixes to guide learning. We then alternate search and distillation, rebuilding the teacher from the updated student in each round so that the student progressively absorbs the gains from alternative layer execution. Experiments on mathematical and logical reasoning tasks show improvements across models. On GSM8K with a 400-token thinking budget, LoOPSD raises Qwen3-1.7B accuracy from 42.30% to 79.61%; subsequent distillation rounds yield further gains beyond the first round. Together, these findings show that by searching layer execution to construct teachers, we can turn a model's additional computation into improved student capabilities, opening a new avenue for self-distillation to derive supervision from the model's own computation.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

57,886 characters extracted from source content.

Expand or collapse full text

Preprint LOOPSD: ON-POLICY SELF-DISTILLATIONWITH LOOPED TEACHERS Wei Fan 1∗† , Qingyue Yang 3† , Chao-Hong Tan 2 , Qian Chen 2 , Wen Wang 2 , Xiangang Li 2 , Kejiang Chen 1‡ , Weiming Zhang 1 , Nenghai Yu 1 1 Anhui Province Key Laboratory of Digital Security 2 Token Foundry, Alibaba Group 3 Independent Researcher range@mail.ustc.edu.cn; chenkj, zhangwm, ynh@ustc.edu.cn tanchaohong.ch, tanqing.cq, w.wang, lixiangang.lxg@alibaba-inc.com ABSTRACT On-policy self-distillation improves a model using guidance from the model itself, but it still requires a teacher that provides more useful predictions than the student. Existing approaches obtain such guidance by giving the teacher additional context or instructions. We ask whether it can instead come from changing how the same model uses its own layers. To address this, we introduce LoOPSD, which searches over a model’s layer execution order to construct a stronger teacher and distills its advantage into the student. Specifically, we keep the model weights unchanged while searching for layers to repeat, and use the resulting teacher’s predictions on student-generated prefixes to guide learning. We then alternate search and distil- lation, rebuilding the teacher from the updated student in each round so that the student progressively absorbs the gains from alternative layer execution. Exper- iments on mathematical and logical reasoning tasks show improvements across models. On GSM8K with a 400-token thinking budget, LoOPSD raises Qwen3- 1.7B accuracy from 42.30% to 79.61%; subsequent distillation rounds yield fur- ther gains beyond the first round. Together, these findings show that by searching layer execution to construct teachers, we can turn a model’s additional computa- tion into improved student capabilities, opening a new avenue for self-distillation to derive supervision from the model’s own computation. 1INTRODUCTION On-policy distillation has emerged as a promising approach to improving language models by learn- ing from teacher predictions on student-generated trajectories (Agarwal et al., 2024). Its token-level guidance lets a student learn along the response prefixes it actually produces. Within this framework, self-distillation offers the possibility of improving reasoning without requiring a stronger external teacher: the model itself supplies the predictions used for learning (Zhao et al., 2026; Sang et al., 2026). The key question is where this self-teacher obtains predictions that are more useful than those of the student. Existing self-distillation approaches obtain such guidance by giving the teacher additional context or instructions. Additional context can provide a teacher with information beyond the student’s input, while a concise instruction can elicit a different reasoning behavior (Zhao et al., 2026; Sang et al., 2026). We investigate whether the same model can become a stronger teacher by running selected layers more than once, without changing their weights. Can learning from this teacher improve a student that uses each layer once in the original order? Layer-execution research provides a concrete opportunity. Repeating or rearranging pretrained lay- ers can improve a model’s reasoning without changing its weights (Batorski et al., 2026; Li et al., * Work done during Wei Fan’s internship at Token Foundry, Alibaba Group. † Equal contribution. ‡ Corresponding author. 1 arXiv:2610.18162v1 [cs.LG] 13 Oct 2026 Preprint ... y <t ... y <t ... y <t ... y <t (a) RLVR (Reinforcement Learning with Verifier/Reward) (b)OPD (On-Policy Distillation) (c) OPSD (On-Policy Self-Distillation) (d) LoOPSD (Looped On-Policy Self-Distillation) Question x Student model On-policy rollout Update θ π θ Question x Student model On-policy rollout Update θ π θ Question x Student model On-policy rollout Update θ π θ Question x Student model On-policy rollout Update θ π θ,std (standardpath) Verifier / Reward r(y) Scalar advantage A(y) External teacher π ext Teacher token distribution p T ( · | x, y <t ) Privileged self-supervision (e.g., privileged context) Self-teacher token distribution p self ( · | x, y <t , c) Token-level KL KL (p T ǁ p θ ) Token-level KL KL (p self ǁ p θ ) Token-level KL KL ( p * ǁ p θ,std ) Same model weights, searched computation route Teacher model(looped path) π θ,r * Teacher token distribution p r * ( · | x, y <t ) L 1 L 2 L 3 L 4 L 5 r Figure 1: Sources of supervision for reasoning models. LoOPSD constructs a teacher from the student’s starting weights by selecting an alternative layer route with labeled examples. The resulting predictions provide supervision without an external teacher model. 2026). We search for which block of layers to repeat and use the resulting model as a teacher. The research question is whether the advantage obtained through this execution can become an improve- ment in the student’s parameters. Route search identifies how to construct the teacher; distillation determines how the student learns from its guidance. We introduce LoOPSD, an on-policy self-distillation method that uses a looped execution of the current model as the teacher. We first search for a repeated layer interval using labeled questions and validate the selected route against standard execution. We then freeze the resulting teacher and train a student initialized from the same checkpoint. The student generates responses, and the teacher scores those response prefixes to provide a token-level learning signal. In this way, the teacher’s additional computation guides the trajectories on which the student learns. We alternate layer search and distillation over multiple rounds. After each round, we search for a block of layers to repeat in the updated student and use the resulting looped model as the next teacher. The student progressively absorbs the gains available from its teachers, and the process ends when validation no longer identifies a useful looped teacher or the round limit is reached. LoOPSD also allows different prompts for the teacher and student. We evaluate this extension by instructing the teacher to reason concisely during both route search and distillation, while the student retains its original prompt. Across mathematical and logical reasoning tasks, we find that LoOPSD improves student accuracy across model sizes and families. On GSM8K, LoOPSD improves Qwen3-1.7B from 42.30% to 79.61% under a 400-token thinking budget; it also improves Phi-4-reasoning-plus from 89.54% to 94.92%. We then examine how these gains develop through iteration, how they appear across think- ing budgets and realized generation lengths, and where answer readout changes inside the student. A label-efficiency study measures the feedback needed to construct useful teachers. Together, these experiments investigate how searched layer execution supplies supervision and how students learn from it. 2RELATED WORK On-policy distillation and self-distillation. Knowledge distillation uses teacher predictions as su- pervision (Hinton et al., 2015); Born Again Networks show that students of the same architecture can improve over their teachers (Furlanello et al., 2018). For reasoning, STaR and ReST EM iterate gen- eration and training on responses selected using correctness feedback (Zelikman et al., 2022; Singh 2 Preprint et al., 2024). On-policy distillation instead supplies teacher predictions along student-generated tra- jectories: GKD trains the student on its own generated responses to reduce the mismatch between training and inference (Agarwal et al., 2024), while MiniLLM develops reverse-KL distillation with on-policy optimization (Gu et al., 2024). Recent self-distillation methods give the teacher additional information or instructions that the student does not receive, such as reference reasoning traces in Self-Distilled Reasoner and an instruction to reason concisely in CRISP (Zhao et al., 2026; Sang et al., 2026). We construct and renew teachers by searching layer execution, and allow the teacher to receive a different prompt from the student. Layer search and adaptive execution. Universal Transformers and recurrent-depth language models are trained to apply the same layer block repeatedly, refining hidden states with shared parameters at each repetition (Dehghani et al., 2019; Geiping et al., 2025). Existing pretrained models also permit execution changes: RYS searches repeated contiguous layer blocks (Ng, 2026), whereas Recirculation introduces recurrent activation connections at inference time (Mozer et al., 2026). MACRO searches routes involving repeated, skipped, or combined layers (Batorski et al., 2026), and Program-of-Layers learns which pretrained layers to execute, skip, or repeat for each input (Li et al., 2026). These studies show that changing layer execution can improve a model’s rea- soning performance. LoOPSD turns this execution advantage into training supervision: it constructs a teacher through layer search and distills its guidance into a standard-path student, repeating this process as the student’s weights change. Reasoning under generation budgets. Reasoning performance often depends on how much com- putation a model is allowed to use during generation (Wei et al., 2022). Test-time methods such as s1 directly control reasoning length through budget forcing (Muennighoff et al., 2025), while training- based approaches encourage effective reasoning under constrained generation through length sched- ules, reward design, or compression, as in ThinkPrune, ShorterBetter, and TokenSkip (Hou et al., 2025; Yi et al., 2025; Xia et al., 2025). Elastic Reasoning trains models explicitly for bounded gen- eration with group relative policy optimization (Xu et al., 2026; Shao et al., 2024), whereas CRISP distills guidance from a concise self-teacher (Sang et al., 2026). These methods primarily control or improve the use of generated reasoning tokens. LoOPSD instead derives supervision from addi- tional computation in layer execution and transfers this advantage to a standard-execution student. We therefore study whether execution-derived supervision improves both reasoning accuracy and re- alized token use under fixed thinking budgets, and whether it can be combined with concise teacher prompting. 3LOOPSD LoOPSD turns a model’s execution flexibility into training supervision. A round has three com- ponents: selecting a useful route, learning from the resulting frozen teacher on student-generated prefixes, and constructing the next teacher from the updated checkpoint. Small labeled set D lab Route search (i, j) ⋮ L 1 L 2 L 3 L 4 L 5 r 1,2 L 1 L 2 L 3 L 4 L 5 r 4,5 L 1 L 2 L 3 L 4 L 5 r 1,3 Selected route r * L 1 L 2 L 3 L 4 L 5 Question x StandardStudentπ θ, std L 1 L 2 L 3 L 4 L 5 Looped Teacherπ θ , r * L 1 L 2 L 3 L 4 L 5 Same prefix y < t Same prefix y < t p std ( y t | x ,y < t ) p r * (y t | x , y < t) ℒ KL ( p r * || p std ) Update θ → θ k 1. Teacher Construction (via Route Search) 2. On-Policy Distillation (Main Training Flow)3. Iteration and Final Model Step 1 Route search Step 2 On-policy distillation Updated student θ k Next round Standard path only (same as the original model). Final Student L 1 L 2 L 3 L 4 L 5 π θ ,std K r i,j After Krounds Figure 2: Overview of LoOPSD. (1) Teacher construction: Using a small labeled set, we try different choices of layers to repeat, then select the best-performing route to define the teacher. (2) On-policy distillation: The student generates responses using the original layer order. The frozen teacher evaluates the same response prefixes and provides token-level supervision to update the student. (3) Iteration and final model: The updated student is used to build the teacher for the next round. After K rounds, the final student uses the original layer order. 3 Preprint 3.1SELECTING THE TEACHER’S COMPUTATION Let p θ,std denote a model with parameters θ that runs layers 1,...,L in their original order. A route r specifies which layers are run and in what order. In our main experiments, a block of consecutive layers is run twice, reusing the same weights (Ng, 2026). We write p θ,r for the model’s output distribution under route r. Repeating layers i through j (1≤ i≤ j ≤ L) gives the route r i,j = (1,...,i− 1, i,...,j, i,...,j, j + 1,...,L).(1) The teacher performs L+ (j−i +1) layer evaluations per token. Each repeated layer uses the same pretrained parameters on the hidden state produced by the previous execution. The route is shared across questions within a round. Selecting it changes the teacher’s predictions without requiring a larger parameter checkpoint. At round k, we evaluate candidate routes on labeled search questions in stages, retaining the best- performing routes in C k . We then compare these candidates with the standard route on a separate validation setD val , using the same prompt and token budgets: r ∗ k = arg max r∈C k ∪std 1 |D val | X (x,a)∈D val V (Decode B,A (p θ k ,r ,x),a).(2) Here, V scores a generated answer against its reference answer, and B and A are the maximum numbers of thinking and answer tokens, respectively. If the selected route scores higher than the standard route on the validation set, it defines the teacher q k = p θ k ,r ∗ k . The teacher’s weights and route remain fixed during the following distillation round. Candidate-screening details are provided in Appendix A. 3.2ON-POLICY DISTILLATION Given the selected teacher, the student generates responses from its current policy, and the teacher evaluates the same student-generated prefixes. We initialize the standard-path student from θ k and sample responses y for questions x fromD q . We train the student to match the teacher’s next-token distribution using the student-to-teacher reverse Kullback–Leibler (KL) divergence (Gu et al., 2024): L(θ) =E x∼D q ,y∼p old (·|x) " 1 T T X t=1 D KL (p θ,std (·| x,y <t )∥q k (·| x,y <t )) # ,(3) Here, p old is the student policy used to generate the responses. The teacher remains fixed throughout the round, and reference answers are used only for teacher selection rather than distillation. We optimize this reverse-KL objective using the sampled-token on-policy distillation implementation described in Appendix A. Every response prefix used for distillation is generated by the student, allowing the teacher to provide local guidance along the trajectories encountered by the student. 3.3ITERATIVE SELF-DISTILLATION AND EXTENSIONS Alternating search and distillation. The two steps above can be repeated in alternation to im- prove the model over multiple rounds. Search constructs a looped teacher from the current model, and distillation transfers its guidance into a standard-path student. The resulting student becomes the starting point for the next search, allowing subsequent rounds to build on what the model has already learned. We continue this process until validation no longer identifies a useful loop or a preset round limit is reached. Extension: Different Prompts for Teacher and Student. So far, the teacher and student have received the same prompt. LoOPSD also allows the teacher to receive additional instructions or in- formation while the student retains its original prompt. We search for a repeated-layer interval using the teacher’s prompt, then distill the resulting teacher’s predictions on student-generated prefixes. The search procedure and distillation objective remain unchanged. In our experiments, we instruct the teacher to reason concisely and denote this configuration by LoOPSD † . 4 Preprint 4EXPERIMENTAL SETUP Models and tasks. We evaluate Qwen3-1.7B, 4B, and 8B (Yang et al., 2025), and DeepSeek- R1-Distill-Llama-8B (DeepSeek-AI et al., 2025); Phi-4-reasoning-plus (Abdin et al., 2025) and Qwen3-14B provide larger-model comparisons. We evaluate mathematical reasoning on GSM8K (Cobbe et al., 2021), SVAMP (Patel et al., 2021), and MATH-500 (Hendrycks et al., 2021; Lightman et al., 2024), and logical reasoning on ProofWriter (Tafjord et al., 2021). We additionally evaluate GSM8K-trained students on GSM-Plus (Li et al., 2024) as a transfer benchmark, without further target-task training. Data splits. For the task-specific experiments, we train a separate student for each task. For MATH-500, training uses the PRM800K split (Lightman et al., 2024). For LoOPSD on GSM8K, we hold out 100 training questions for validation and use the remaining 7,373 questions for distillation. Route search uses labeled answers from a subset of 1,000 questions in this distillation pool. Refer- ence answers are used for teacher selection, but are not provided during distillation. Test questions are excluded from route search and checkpoint selection. Training. We use the repeated-layer route defined in Section 3.1: a contiguous interval of pre- trained layers is executed twice with shared weights. Each LoOPSD round trains the student for 200 updates against a frozen teacher, with at most 1,024 generated tokens per training response. We al- low up to ten rounds, following the validation-based continuation rule in Section 3.3. For LoOPSD † , the different-prompt extension introduced in Section 3.3, the teacher receives a concise reasoning instruction during both route search and distillation, while the student retains its original prompt. Appendix A provides the route-search procedure, optimization, and continuation details. Baselines. We compare LoOPSD and LoOPSD † with the original model, Elastic Reasoning (Xu et al., 2026), and CRISP (Sang et al., 2026). Elastic Reasoning uses GRPO with a limit on thinking tokens during training. For CRISP, we use a concise standard-path teacher for one 200-update round with the same distillation implementation as LoOPSD. Evaluation. By default, we use greedy decoding with up to 400 thinking tokens followed by up to 128 answer tokens. We report accuracy using task-specific rules to compare generated answers with reference answers. All students use the original prompt and standard layer order. For the budget experiments, we either obtain a separate student for each target search budget or vary only the test-time thinking budget of a fixed student. The figure captions specify which setting is used. 5RESULTS 5.1IMPROVED STUDENT REASONING Reasoning across models and tasks. To assess learning gains from looped teachers, we first com- pare R1 (Round 1, 200 student updates) students in Table 1. At R1, LoOPSD † already outperforms CRISP (also 200 student updates) in most evaluated settings. LoOPSD also improves the original models in most settings. Further search-and-distillation rounds generally improve accuracy over R1. The resulting final students (Final) surpass the original models in all evaluated pairs. Gains span mathematical and logical reasoning and transfer to GSM-Plus without target-task training. These results support looped-teacher supervision as a source of iterative student improvement. Different Teacher Prompts. To evaluate the different-prompt extension in Section 3.3, we com- pare LoOPSD with LoOPSD † , which uses a concise teacher prompt. On MATH-500 at R1, Qwen3- 4B students reach 46.00% with LoOPSD and 52.80% with LoOPSD † , versus 44.60% with CRISP; the corresponding 8B scores are 45.00%, 54.00%, and 45.80%. At Final, LoOPSD † reaches 60.40% and 65.80%, respectively. Concise teacher prompting thus strengthens supervision throughout this iterative process. Scaling to larger models. We evaluate Phi-4-reasoning-plus and Qwen3-14B on GSM8K at a 400-token thinking budget to test whether the gains extend to larger models (Table 2). The R1 5 Preprint Table 1: Accuracy (%) across four models and five benchmarks, using greedy decoding with 400 thinking and 128 answer tokens. For LoOPSD variants, entries report first-round (R1)→ final accu- racy. R1 denotes the student after the first distillation round; final denotes the checkpoint obtained by the iterative continuation procedure. GSM-Plus evaluates the corresponding GSM8K-trained checkpoints. MethodGSM8KGSM-PlusSVAMPMATH-500ProofWriter Qwen3-1.7B Original42.3031.7071.3333.2042.50 Elastic Reasoning46.2530.2073.0034.0045.00 CRISP76.6558.5088.3337.8043.40 LoOPSD (R1→Final)78.09→79.6156.50→61.4087.67→87.6744.40→44.8042.10→56.20 LoOPSD † (R1→Final)78.32→79.6859.30→61.8088.67→91.6737.20→48.2046.80→54.10 Qwen3-4B Original74.9856.4087.3336.0063.30 Elastic Reasoning75.5958.9089.3337.4067.90 CRISP87.7270.1092.0044.6066.00 LoOPSD (R1→Final)84.38→89.7667.00→74.2090.33→88.6746.00→50.2068.00→76.40 LoOPSD † (R1→Final)87.04→91.1369.00→73.2091.33→91.3352.80→60.4066.20→71.50 Qwen3-8B Original76.0461.0089.3339.0065.40 Elastic Reasoning76.8861.2091.3339.8070.50 CRISP89.6170.0092.3345.8065.90 LoOPSD (R1→Final)87.72→90.6769.70→74.2091.33→92.3345.00→49.6072.90→79.40 LoOPSD † (R1→Final)92.49→92.4974.30→74.3094.00→94.0054.00→65.8073.10→81.60 DeepSeek-R1-Distill-Llama-8B Original65.0549.0076.6738.6039.20 Elastic Reasoning73.0953.8087.0042.2051.80 CRISP68.7652.8080.6733.0038.30 LoOPSD (R1→Final)74.45→76.0452.20→57.6080.67→88.3339.00→43.4049.60→51.10 LoOPSD † (R1→Final)80.89→81.6560.20→59.5088.67→88.6737.20→43.4047.80→48.90 R1→Final: first-round student accuracy → final student accuracy. Bold: best final accuracy per model and task. †: concise teacher prompt. All students use their original prompt and standard layer order. GSM-Plus uses the fixed 1000-question subset; other columns use task-specific training. students of LoOPSD † reach 95.07% and 93.63%, respectively, versus CRISP’s 92.95% and 92.57%. At Final, LoOPSD reaches 94.92% and 94.39%. These results extend the effectiveness of execution- derived supervision to the 14B scale. Table 2: GSM8K accuracy (%) of 14B-scale reasoning models with a 400-token thinking budget and greedy decoding. LoOPSD entries report first-round (R1) → final accuracy. All students use standard execution and their original prompt. LoOPSD achieves the highest final accuracy for both model families. MethodPhi-4-reasoning-plusQwen3-14B Original89.5483.40 Elastic Reasoning89.6985.14 CRISP92.9592.57 LoOPSD (R1→Final)91.89→ 94.9290.07→ 94.39 LoOPSD † (R1→Final)95.07→ 93.1893.63→ 93.63 R1→Final: first-round student accuracy→ final student accuracy. †: concise teacher prompt; all stu- dents use the original prompt. The Qwen3-14B Elastic evaluation has a paired original score of 83.85%. 5.2REASONING QUALITY UNDER LIMITED THINKING BUDGETS Effectiveness across thinking budgets. Figure 3(a,b) evaluates Qwen3-1.7B and DeepSeek-R1- Distill-Llama-8B on GSM8K across six thinking budgets. We obtain a student for each target search budget and evaluate it at that budget. LoOPSD improves accuracy over the original model across 6 Preprint 50 128256400 10242048 Thinking budget 0 25 50 75 100 Accuracy (%) (a) Qwen3-1.7B 50 128256400 10242048 Thinking budget 0 25 50 75 100 (b) DeepSeek-R1-8B 50 128256400 10242048 Test thinking budget 0 25 50 75 100 (c) Qwen3-4B · fixed student OriginalElastic ReasoningCRISPLoOPSDLoOPSD† Figure 3: Accuracy across thinking budgets on GSM8K. (a,b) Qwen3-1.7B and DeepSeek-R1- Distill-Llama-8B students obtained separately for each target search budget and evaluated at that budget. (c) Qwen3-4B students obtained with a 400-token search budget, with each checkpoint fixed across test budgets. LoOPSD improves reasoning across a range of budgets, and fixed students adapt to different test budgets. a broad range of budgets, and its concise-teacher extension improves both models at every tested budget. These results show that the benefits extend across different budgets for reasoning, rather than being confined to one budget. Appendix D provides the full comparisons. Adaptation to different test budgets. Figure 3(c) evaluates whether a learned student can also accommodate budgets other than the one used for route search. We fix the Qwen3-4B students obtained with a 400-token search budget and vary only their test budgets. The students achieve higher or broadly comparable accuracy to the original model across the tested range, with substantial gains at several budgets outside the search configuration. Thus, a single learned student can support different thinking budgets without further search or training. 5.3TOKEN EFFICIENCY OF LEARNED STUDENTS We compare accuracy and actual generation length under a shared thinking budget to assess the token efficiency of LoOPSD students. Figure 4 evaluates Qwen3-4B on GSM8K at a 400-token thinking budget, measuring accuracy and generation length from the same responses. LoOPSD produces more accurate answers with fewer generated tokens: accuracy rises from 74.98% to 89.76%, while average total generation falls from 385.04 to 299.98 tokens. Its concise-teacher extension further improves accuracy while shortening generation. The length breakdown in Fig- ure 4(b) shows shorter thinking sequences alongside shorter total responses. Together, these results show that LoOPSD students achieve higher accuracy with shorter generated responses under the same thinking budget. 6ANALYSIS 6.1LABEL EFFICIENCY We compare methods under matched labeled-data budgets to assess the label efficiency of LoOPSD. We evaluate LoOPSD and GRPO with truncated thinking across different label budgets while match- ing training updates. This comparison examines two ways of using labeled feedback: LoOPSD evaluates alternative layer execution orders on a small labeled set and turns the selected teacher’s predictions into dense token-level supervision, whereas GRPO scores multiple responses sampled from the same model and updates it using group-relative rewards. Specifically, we use Qwen3-1.7B on GSM8K with budgets of 200, 400, and 1100 labeled examples, each including 100 validation examples. Both methods receive 200 training updates, and LoOPSD’s distillation question pool remains fixed. 7 Preprint 250300350400 Total generated tokens 80 90 Accuracy (%) Original Elastic Reasoning CRISP LoOPSD LoOPSD† (a) Accuracy and length 0200400 Generated tokens Original Elastic Reasoning CRISP LoOPSD LoOPSD† 385.0 383.0 309.2 300.0 248.9 (b) Generation stages ThinkingAnswer Figure 4: Accuracy and generation length of different methods on GSM8K with Qwen3-4B. All models use a 400-token thinking budget. (a) Accuracy versus average total generated tokens. (b) Average thinking and answer lengths. LoOPSD improves accuracy while shortening generation; LoOPSD † achieves the highest accuracy with the fewest thinking tokens. Table 3: Label efficiency on GSM8K with Qwen3-1.7B. LoOPSD and matched-label GRPO each re- ceive 200 updates; label totals include 100 validation examples, and LoOPSD uses a fixed question- only distillation pool. LoOPSD already obtains most of its observed gain with 200 labels. Total labeled examples Method2004001100 Original42.30 (shared baseline) Matched-label GRPO43.9043.5244.05 LoOPSD78.0178.3978.62 Totals include 100 validation labels. LoOPSD distills on a fixed question-only pool; GRPO uses the matched labeled pool. Label counts and updates are matched; search compute and unlabeled-question access differ. Answer cap: 128 tokens. LoOPSD achieves higher accuracy at every label budget, with most of the improvement already obtained from the smallest labeled pool (Table 3). With a fixed question-only distillation pool, LoOPSD obtains most of its observed improvement using only a small number of labeled examples for teacher construction. 6.2ABLATION STUDIES We examine two questions: whether looped teachers provide better supervision than standard-path self-teachers, and whether iterating teacher construction and distillation yields further gains beyond a single round. Effect of looped teacher construction. To isolate the contribution of looped execution, we replace LoOPSD’s looped teachers with standard-path self-teachers, keeping the prompts, distillation objec- tive, and student-update schedule unchanged. Figure 5(a) evaluates all four methods on Qwen3- 4B/GSM8K with a 128-token thinking budget across six 200-update rounds. LoOPSD reaches 50.87% accuracy after 1,200 updates, compared with 26.38% for standard-path self-distillation. We apply the same ablation to the different-prompt extension introduced in Section 3.3. For LoOPSD † , we retain the concise teacher prompt and use multi-round CRISP as the standard-path control under the same student-update schedule. LoOPSD † also outperforms this control. Across both teacher- prompt settings, the looped-teacher variants produce stronger standard-path students than the corre- sponding standard-path self-distillation trajectories in this experiment. Effect of iterative LoOPSD rounds. To assess the benefit of continuing beyond one round, we compare stopping after the first distillation round with continuing the full search-and-distillation procedure. Along the same training run, student accuracy increases from 33.81% after the first round to 50.87% after the sixth, a gain of 17.06 percentage points (Figure 5(a)). These results 8 Preprint 0123456 Distillation round 30 45 60 75 Accuracy (%) +17.06 p (a) Across distillation rounds 5012825640010242048 Thinking budget (tokens) 20 40 60 80 100 (b) Across test budgets Original LoOPSD first LoOPSD final LoOPSD† first LoOPSD† final LoOPSD teacher LoOPSD student LoOPSD† teacher LoOPSD† CRISP teacher CRISP Self-distillation Figure 5: Iterative learning on GSM8K with Qwen3-4B. (a) Each round shows the looped teacher constructed from the current model and the standard-path student obtained after distillation. We compare LoOPSD, LoOPSD † , CRISP, and standard-path self-distillation across six 200-update rounds, evaluated at a 128-token thinking budget. (b) First-round and final students from the re- spective LoOPSD and LoOPSD † training trajectories, each held fixed across test budgets. Panel (a) includes matched-update standard-path controls under both the original and concise teacher prompts, while panel (b) shows the additional gains obtained across LoOPSD iterations. support the iterative design of LoOPSD: repeating teacher construction and distillation improves the standard-path student beyond what is achieved by a single round. 6.3LAYERWISE ANSWER READOUT AFTER DISTILLATION 112243642 Layer execution step 20 40 60 Label-readout accuracy (%) Original CRISP LoOPSD (Round 1) LoOPSD (Round 3) Looped teacher (Round 3) Figure6:Layerwiseanswerreadouton ProofWriterwithQwen3-4Bundershared contexts and a fixed output projection. Shading marks repeated teacher execution. Student gains concentrate in later layers. To examine how distillation changes the stan- dard student, we compare layerwise answer readout using the logit lens (nostalgebraist, 2020) on identical ProofWriter contexts with the original model’s readout head. Keeping both the input text and readout fixed allows us to compare the models without differences in the thinking sequences they generate. Figure 6 shows that readout accuracies remain close in the early layers, while the student gains become clear in the later layers. This im- provement is more pronounced after the third round than after the first, showing how late- layer answer readout develops over succes- sive rounds. The looped teacher reaches a higher final readout accuracy with additional layer executions, whereas the student obtains its improvements within the original layer se- quence. Notably, these changes arise with- out directly matching intermediate representa- tions: the distillation objective supervises only output-layer predictions. Learning from the looped teacher therefore also strengthens answer readout from the standard student’s intermediate layers. 9 Preprint 7CONCLUSION We introduced LoOPSD, which searches layer execution to construct self-teachers and distills their predictions on student-generated trajectories. By alternating search and learning, we obtain further student gains beyond the first round. Across mathematical and logical reasoning tasks, the resulting students improve in accuracy, while budget, generation-length, and layerwise analyses characterize the learned changes. LoOPSD also supports different teacher prompts, as demonstrated by our concise-teacher extension. These findings show that we can turn a model’s additional computation into improved student capabilities, opening a new avenue for self-distillation to derive supervision from the model’s own computation. REPRODUCIBILITY STATEMENT We provide the LoOPSD procedure in Section 3, including route selection, on-policy distillation, and iterative teacher construction. Section 4 specifies the evaluated models, tasks, data splits, reasoning budgets, baselines, and evaluation protocol. Appendix A provides optimization, generation, route- search, continuation, and scoring details. Additional appendices provide supporting results and analyses of teacher construction, transfer, reasoning budgets, and layerwise behavior. Test examples are excluded from route search and checkpoint selection. AI USE STATEMENT In this work, we used generative AI tools to provide feedback on experimental design and method- ology, assist with result interpretation, and support parts of the implementation. We also used gener- ative AI tools for literature search, language editing, manuscript organization, drafting and revising text, and creating or refining figures. We did not use generative AI tools to generate synthetic datasets or formulate or prove mathematical claims. All AI-assisted code was reviewed and tested by the au- thors, and AI-assisted literature suggestions, analyses, and manuscript content were checked against the relevant code, experiments, and source literature. We take responsibility for the final content of this work, including all text, claims, code, and artifacts produced with the aid of generative AI. REFERENCES Marah Abdin, Sahaj Agarwal, Ahmed Awadallah, Vidhisha Balachandran, Harkirat Behl, Lingjiao Chen, Gustavo de Rosa, Suriya Gunasekar, Mojan Javaheripi, Neel Joshi, Piero Kauffmann, Yash Lara, Caio César Teodoro Mendes, Arindam Mitra, Besmira Nushi, Dimitris Papailiopoulos, Olli Saarikivi, Shital Shah, Vaishnavi Shrivastava, Vibhav Vineet, Yue Wu, Safoora Yousefi, and Guoqing Zheng. Phi-4-reasoning technical report, 2025. URL https://arxiv.org/abs/2504. 21318. Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self- generated mistakes. In International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=3zKtaqxLhW. Paweł Batorski, Abtin Pourhadi, Akylgali Aitaza, Przemysław Spurek, and Paul Swoboda. MACRO: Markov chain routing of transformer layers, 2026. URL https://arxiv.org/abs/2608.05872. Nora Belrose, Igor Ostrovsky, Lev McKinney, Zach Furman, Logan Smith, Danny Halawi, Stella Biderman, and Jacob Steinhardt. Eliciting Latent Predictions from Transformers with the Tuned Lens, 2023. URL https://arxiv.org/abs/2303.08112. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL https://arxiv.org/ abs/2110.14168. 10 Preprint DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, et al. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning, 2025. URL https://arxiv.org/abs/2501.12948v1. Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Łukasz Kaiser. Universal Transformers. In International Conference on Learning Representations, 2019. URL https: //arxiv.org/abs/1807.03819. Tommaso Furlanello, Zachary C. Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born Again Neural Networks. In International Conference on Machine Learning, 2018. URL https://proceedings.mlr.press/v80/furlanello18a.html. Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R. Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach, 2025. URL https://arxiv.org/abs/2502. 05171. Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang.MiniLLM: Knowledge distilla- tion of large language models.In International Conference on Learning Represen- tations, 2024.URL https://proceedings.iclr.c/paper_files/paper/2024/hash/ 8ac015d409635f196f9e3e9dcfb9a94e-Abstract-Conference.html. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring Mathematical Problem Solving With the MATH Dataset, 2021. URL https://arxiv.org/abs/2103.03874. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the Knowledge in a Neural Network, 2015. URL https://arxiv.org/abs/1503.02531. Bairu Hou, Yang Zhang, Jiabao Ji, Yujian Liu, Kaizhi Qian, Jacob Andreas, and Shiyu Chang. ThinkPrune: Pruning Long Chain-of-Thought of LLMs via Reinforcement Learning, 2025. URL https://arxiv.org/abs/2504.01296. Qintong Li, Leyang Cui, Xueliang Zhao, Lingpeng Kong, and Wei Bi. GSM-plus: A compre- hensive benchmark for evaluating the robustness of LLMs as mathematical problem solvers. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), p. 2961–2984. Association for Computational Linguistics, 2024. doi: 10.18653/v1/2024.acl-long.163. URL https://aclanthology.org/2024.acl-long.163/. Ziyue Li, Yang Li, and Tianyi Zhou. Skip a layer or loop it? learning program-of-layers in LLMs. In International Conference on Machine Learning, 2026. URL https://arxiv.org/abs/2606. 06574v2. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s Verify Step by Step. In International Conference on Learning Representations, 2024. URL https://arxiv.org/abs/2305.20050. Michael C. Mozer, Shoaib Ahmed Siddiqui, Danny Sawyer, Sunny Sanyal, and Rosanne Liu. Re- circulation, 2026. URL https://arxiv.org/abs/2608.17981. Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling, 2025. URL https://arxiv.org/abs/2501.19393. David Noel Ng. LLM neuroanatomy: How i topped the LLM leaderboard without changing a single weight. Author’s technical blog, 2026. URL https://dnhkng.github.io/posts/rys/. nostalgebraist.Interpreting GPT: The logit lens.LessWrong, 2020.URL https://w. lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens. Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, p. 2080–2094. Association for Computational Linguistics, 2021. doi: 10.18653/v1/2021.naacl-main.168. URL https://aclanthology.org/2021.naacl-main.168/. 11 Preprint Hejian Sang, Yuanda Xu, Zhengze Zhou, Ran He, Zhipeng Wang, and Jiachen Sun. CRISP: Com- pressed reasoning via iterative self-policy distillation, 2026. URL https://arxiv.org/abs/ 2603.05433v7. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the limits of mathe- matical reasoning in open language models, 2024. URL https://arxiv.org/abs/2402.03300. Avi Singh, John D. Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Pe- ter J. Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron Parisi, Abhishek Kumar, Alex Alemi, Alex Rizkowsky, Azade Nova, Ben Adlam, Bernd Bohnet, Gamaleldin Elsayed, Hanie Sedghi, Igor Mordatch, Isabelle Simpson, Izzeddin Gur, Jasper Snoek, Jeffrey Pennington, Jiri Hron, Kathleen Kenealy, Kevin Swersky, Kshiteej Mahajan, Laura Culp, Lechao Xiao, Maxwell L. Bileschi, Noah Constant, Roman Novak, Rosanne Liu, Tris Warkentin, Yundi Qian, Yamini Bansal, Ethan Dyer, Behnam Neyshabur, Jascha Sohl-Dickstein, and Noah Fiedel. Beyond Hu- man Data: Scaling Self-Training for Problem-Solving with Language Models. Transactions on Machine Learning Research, 2024. URL https://arxiv.org/abs/2312.06585. Oyvind Tafjord, Bhavana Dalvi, and Peter Clark. ProofWriter: Generating implications, proofs, and abductive statements over natural language. In Findings of the Association for Computa- tional Linguistics: ACL-IJCNLP 2021, p. 3621–3634. Association for Computational Linguis- tics, 2021. doi: 10.18653/v1/2021.findings-acl.317. URL https://aclanthology.org/2021. findings-acl.317/. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In Advances in Neural Information Processing Systems, 2022. URL https://arxiv.org/abs/ 2201.11903. Heming Xia, Chak Tou Leong, Wenjie Wang, Yongqi Li, and Wenjie Li. TokenSkip: Controllable Chain-of-Thought Compression in LLMs. In Proceedings of the 2025 Conference on Empiri- cal Methods in Natural Language Processing, 2025. URL https://aclanthology.org/2025. emnlp-main.165/. Yuhui Xu, Hanze Dong, Lei Wang, Doyen Sahoo, Junnan Li, and Caiming Xiong. Scalable chain of thoughts via elastic reasoning. In International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=E0Qfhma53J. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL https://arxiv.org/abs/2505.09388. Jingyang Yi, Jiazheng Wang, and Sida Li. ShorterBetter: Guiding Reasoning Models to Find Op- timal Inference Length for Efficient Reasoning, 2025. URL https://arxiv.org/abs/2504. 21370. Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. STaR: Bootstrapping Reasoning With Reasoning. In Advances in Neural Information Processing Systems, 2022. URL https: //arxiv.org/abs/2203.14465. Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self-distilled reasoner: On-policy self-distillation for large language models, 2026. URL https: //arxiv.org/abs/2601.18734v3. 12 Preprint AIMPLEMENTATION DETAILS Optimization. Each distillation round uses 200 optimizer updates, a learning rate of 2×10 −6 , and training seed 42. A batch comprises twelve questions and four responses per question. Responses are generated with a single-stage cap of 1,024 tokens, temperature 1, and top-p 1. Teacher weights and the selected route remain fixed throughout a round. The teacher evaluates full-softmax probabilities of student-sampled tokens; the bounded log-probability signal and importance-weighted surrogate are given in Equation 6. Labels are used for teacher selection; student distillation uses questions and generated responses. Execution and continuation. The main route family repeats a contiguous interval of pretrained layers. Search and route validation compare the candidates at the configured thinking budget, includ- ing the standard path. A new round starts from the updated standard student. Subsequent route-based continuation requires a useful nonstandard route under the run’s validation rule, with a maximum of ten rounds. The concise-teacher configuration retains its permitted initial prompt-only fallback. Generation and scoring. Main evaluation is greedy and reserves 128 tokens for the answer stage after the configured thinking budget. A task-specific answer cue starts the answer stage. Accuracy is computed by the task’s deterministic local rule on the complete fixed evaluation set. Thinking, answer, and total lengths are measured with the corresponding model tokenizer, averaged over every evaluated example. Source identities for length comparisons bind the token counts and correctness outcomes to the same responses. Length evaluation. The paired Elastic accuracy–length evaluation in Figure 4 scores 76.27%; its independent main-table evaluation scores 75.59%. Each point pairs correctness and token counts from the same responses. Token counts describe generated length, not teacher computation or run- time. Candidate screening and selection. We enumerate contiguous repeated-layer intervals and retain the standard route at every stage. We shuffle the official GSM8K training split with seed 42, assign- ing the first 1000 questions to search and the next 100 to validation; all questions outside validation form the distillation pool. On GSM8K, the first screening stage evaluates routes on 128 search questions and retains the top 24 nonstandard candidates; the second evaluates the survivors on the remaining 872 search questions and retains the top five. Both stages also retain every nonstandard candidate tied at the cutoff. The surviving candidates and standard route are then evaluated on 100 separate validation questions, using identical prompts and token budgets. If the standard route ties for the best validation score, it is retained. Otherwise, ties among the best nonstandard routes are resolved by sampling from their sorted candidate identifiers with seed 42. Thus, adopting a loop re- quires a strict validation improvement over the standard route. The concise-teacher configuration’s initial prompt-only fallback is described above. Sampled distillation objective. Training uses a sampled-token policy-gradient surrogate. For a sampled token, the teacher–student log-probability difference supplies a bounded update signal: a t = clip(logq k (y t | x,y <t )− logp old (y t | x,y <t ),−c,c),(4) ρ t (θ) = p θ,std (y t | x,y <t ) p old (y t | x,y <t ) ,(5) b L(θ) =− P x,y,t m t ρ t (θ) sg(a t ) P x,y,t m t .(6) Here, m t selects valid response tokens, and sg stops gradients. The importance ratio ρ t is not clipped. We obtain each sampled token’s probability from the full-vocabulary softmax. The training loss uses these sampled-token scores without explicitly computing the full-vocabulary KL diver- gence. A positive a t increases the likelihood of a sampled token that the teacher favors more than the rollout policy; a negative value reduces it. The clipping threshold is c = 10. Before clipping, the log-probability difference supplies the reverse-KL score-function direction at the rollout policy. 13 Preprint At each training step, we sample a fresh batch of responses using the current student policy, de- noted by p old . Each batch is used for one student update. The updated student weights are then synchronized to the rollout workers before the next batch is generated. BCONSTRUCTING STRONGER TEACHERS THROUGH LAYER SEARCH We examine whether repeating layers can improve the teacher without changing its weights, and whether this source of supervision remains useful with a concise teacher prompt. Figures 7 and 8 show first-round route-search landscapes across four model–task settings. All settings use 400 think- ing tokens and 128 answer tokens. Each landscape evaluates all repeated-layer intervals on the same 128 search questions within its setting; a separate validation set selects the teacher. 18162428 Last repeated layer j 1 8 16 24 28 First repeated layer i (a) Qwen3-1.7B / GSM8K LoOPSD Original: 48.4% 1816243236 Last repeated layer j 1 8 16 24 32 36 First repeated layer i (b) Qwen3-4B / ProofWriter LoOPSD Original: 52.3% 18162428 Last repeated layer j 1 8 16 24 28 First repeated layer i (c) Qwen3-1.7B / GSM8K LoOPSD† Original: 48.4% Concise std.: +38.3 p 1816243236 Last repeated layer j 1 8 16 24 32 36 First repeated layer i (d) Qwen3-4B / ProofWriter LoOPSD† Original: 52.3% Concise std.: +5.5 p -48.4 0 +37.5 Δ acc. (p) -48.4 0 +39.8 Δ acc. (p) -52.3 0 +18.8 Δ acc. (p) -52.3 0 +21.1 Δ acc. (p) Figure 7: Teacher construction on GSM8K with Qwen3-1.7B and ProofWriter with Qwen3-4B, at a 400-token thinking budget. Heatmaps show search accuracy changes in percentage points; each panel has an independent color bar. Stars mark validation-selected nonstandard routes. Validation selects a repeated-layer route under both prompts on ProofWriter, while retaining standard execution under the concise prompt on GSM8K. The maps show accuracy changes relative to the original-prompt standard model. Concise-standard gains printed inside the lower panels identify the contribution of the prompt before any layers are repeated. Each panel has its own red–white–blue scale, centered at zero and spanning all measured values; negative and positive ranges are scaled separately. Thus, equal color intensity across panels need not indicate equal accuracy changes. In the concise-prompt panels, the color includes both the prompt contribution and any additional change from layer repetition. The axes give the first and last repeated layers in the inclusive interval [i,j]. Stars mark nonstandard routes selected by validation; no star is shown when the standard route is selected. From a route landscape to a stronger teacher. Beneficial regions depend on the task and teacher prompt. Under the original prompt, validation selects a nonstandard route in all four displayed settings. Under concise prompting, validation selects nonstandard routes on ProofWriter, MATH- 500, and SVAMP, while retaining the standard route on GSM8K. These landscapes show why the 14 Preprint repeated interval is searched for each setting, and how the procedure can retain standard execution when repetition provides no validated advantage. 1816243236 Last repeated layer j 1 8 16 24 32 36 First repeated layer i (a) Qwen3-8B / MATH-500 LoOPSD Original: 39.1% 18162432 Last repeated layer j 1 8 16 24 32 First repeated layer i (b) DeepSeek-R1-8B / SVAMP LoOPSD Original: 78.9% 1816243236 Last repeated layer j 1 8 16 24 32 36 First repeated layer i (c) Qwen3-8B / MATH-500 LoOPSD† Original: 39.1% Concise std.: +7.8 p 18162432 Last repeated layer j 1 8 16 24 32 First repeated layer i (d) DeepSeek-R1-8B / SVAMP LoOPSD† Original: 78.9% Concise std.: +0.8 p -39.1 0 +7.0 Δ acc. (p) -39.1 0 +19.5 Δ acc. (p) -78.1 0 +10.2 Δ acc. (p) -78.9 0 +14.1 Δ acc. (p) Figure 8: Teacher construction on MATH-500 with Qwen3-8B and SVAMP with DeepSeek-R1-8B, using the same 400-token protocol. Heatmaps show search accuracy changes; independent bars preserve each panel’s measured range. Stars mark validation-selected nonstandard routes, which are selected under both the original and concise prompts in these two settings. CTRANSFER BEYOND GSM8K We evaluate whether the improvements learned on GSM8K transfer to SVAMP (Patel et al., 2021) and GSM-Plus (Li et al., 2024). We reuse the frozen GSM8K checkpoints from the 400-token com- parison, including the shared 200-update CRISP checkpoint. Target-task examples are used only for evaluation; they do not enter route search, training, or checkpoint selection. SVAMP contributes its fixed 300-question test split, while GSM-Plus contributes a fixed 1,000-question subset of testmini. All methods use the original math prompt, standard layer execution, greedy decoding, 400 thinking tokens, and a separate 128-token answer budget. LoOPSD and LoOPSD † improve over the original model on both target datasets for all four models. On GSM-Plus, ordinary LoOPSD also exceeds CRISP in every model, by 2.9–4.8 percentage points. The concise-teacher configuration uses fewer thinking tokens than ordinary LoOPSD in all eight model–dataset comparisons. These results show that the learned gains extend beyond the training benchmark to additional math problems and GSM8K perturbations. 15 Preprint Table 4: Transfer of GSM8K-trained students to additional math datasets. Accuracy (%) and mean thinking tokens at a 400-token thinking cap. All methods use standard execution, the original prompt, and greedy decoding. SVAMPGSM-Plus MethodAccuracyThink tokensAccuracyThink tokens Qwen3-1.7B Original71.33299.731.70378.8 Elastic Reasoning75.00294.430.20378.1 CRISP87.67251.858.50353.5 LoOPSD85.33373.061.40392.7 LoOPSD † 90.33217.661.80327.7 Qwen3-4B Original87.33321.156.40385.3 Elastic Reasoning88.33316.658.90384.6 CRISP92.00258.370.10343.0 LoOPSD92.67241.274.20332.7 LoOPSD † 93.67201.573.20295.9 Qwen3-8B Original89.33354.161.00390.9 Elastic Reasoning90.00351.561.20390.1 CRISP93.67277.170.00347.0 LoOPSD91.33284.974.20343.7 LoOPSD † 92.67251.774.30320.3 DeepSeek-R1-Distill-Llama-8B Original76.67116.449.00197.3 Elastic Reasoning81.33275.753.80340.8 CRISP78.67135.352.80253.9 LoOPSD84.00340.557.60380.3 LoOPSD † 87.33286.659.50357.3 Bold: highest accuracy per model and dataset. †: concise teacher prompt during distillation. SVAMP uses the fixed 300-question test split; GSM-Plus uses the fixed 1,000-question evaluation subset. Answer cap: 128 tokens. Thinking lengths are tokenizer-specific averages over every evaluated example. DFROM INITIAL TEACHERS TO FINAL STUDENTS We ask whether the final standard-path student catches up to the initial looped teacher found before distillation. Figures 9 and 10 compare the original model, the initial teacher, and the final student across six thinking budgets on GSM8K. Each budget uses its corresponding search and training configuration. Because LoOPSD may continue for multiple rounds, this comparison does not mea- sure one-round transfer; instead, it shows where the final student ends up relative to the teacher that started the training process. Filled circles denote students or original models, open triangles denote initial teachers, and vertical segments connect each teacher with its final student. For Qwen3-1.7B, the final LoOPSD student outperforms its initial teacher at all six budgets. For DeepSeek-R1-Distill-Llama-8B, the final student matches or exceeds the initial teacher in most set- tings. These results show that iterative LoOPSD can move the standard-path student beyond the performance of the looped teacher available at the start of training. 16 Preprint 5012825640010242048 Thinking budget (tokens) 0 20 40 60 80 100 Accuracy (%) OriginalCRISPLoOPSDLoOPSD† Student / baselineInitial teacherTeacher–student pair Figure 9: Initial teachers and final students on GSM8K with Qwen3-1.7B. Each initial teacher is con- nected to the final student obtained from the corresponding training configuration. Final LoOPSD students outperform their initial teachers at all six thinking budgets. 5012825640010242048 Thinking budget (tokens) 0 20 40 60 80 100 Accuracy (%) OriginalCRISPLoOPSDLoOPSD† Student / baselineInitial teacherTeacher–student pair Figure 10: Initial teachers and final students on GSM8K with DeepSeek-R1-Distill-Llama-8B. Each initial teacher is connected to the final student obtained from the corresponding training configura- tion. Final LoOPSD students match or exceed their initial teachers at most thinking budgets. 17 Preprint ELAYERWISE GOLD-LABEL LOG PROBABILITY We test whether the stronger intermediate-layer answer readout also reflects greater probability as- signed to the correct answer, rather than only changes in the top-ranked label. Figure 11 measures gold-label log probability on the same fixed ProofWriter contexts and with the same readout as Figure 6. The fixed readout follows the logit-lens approach (nostalgebraist, 2020) and fits no layer- specific probes, unlike the tuned lens (Belrose et al., 2023). The original model, CRISP student, first- and third-round LoOPSD students, and third-round looped teacher share inputs, answer posi- tions, and the original model’s final normalization and output projection. The later-layer increases in gold-label log probability complement the accuracy gains in Figure 6, showing stronger readout of the correct answer even when its rank does not change. 112243642 Layer execution step −17.5 −15.0 −12.5 −10.0 −7.5 −5.0 −2.5 Gold-label log probability Original CRISP LoOPSD (Round 1) LoOPSD (Round 3) Looped teacher (Round 3) Figure 11: Gold-label log probability across layer executions on ProofWriter with Qwen3-4B. Higher values indicate greater probability assigned to the correct label. Shading marks repeated teacher execution. LoOPSD students assign higher probability to the correct label in later layers. FSENSITIVITY TO SAMPLING TEMPERATURE We evaluate whether the improvements persist under stochastic thinking generation. For each method, we keep the student checkpoint fixed and vary only the thinking temperature; the answer stage uses greedy decoding. Table 5 shows that both LoOPSD variants remain above the original model across all tested temperatures. LoOPSD † achieves the highest accuracy in seven of the eight model–temperature settings, with LoOPSD leading in the remaining setting. Table 5: GSM8K accuracy (%) at different thinking temperatures. We evaluate the same check- points with a 400-token thinking budget and a 128-token answer budget, sampling one response per question. LoOPSD improvements persist across decoding temperatures. Qwen3-1.7BQwen3-4B Method 0 0.3 0.5 1.0 0 0.3 0.5 1.0 Original42.38 42.99 41.55 41.70 74.91 73.62 75.89 73.92 Elastic44.43 44.12 44.20 43.59 76.27 75.66 76.12 75.36 CRISP76.65 78.17 79.23 77.26 87.64 88.40 88.02 87.57 LoOPSD79.6179.4579.3877.0389.6990.3790.3791.21 LoOPSD † 79.6880.5281.8880.8291.0591.5191.6691.05 18

Instant research discovery