Paper deep dive
PAST: Privileged Adaptation from Complete Student Trajectories for On-Policy Self-Distillation
Yangyang Feng, Zhuoyan Feng, Junlan Chen
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/12/2026, 2:29:16 AM
Summary
The paper introduces Privileged Adaptation from Student Trajectories (PAST), a method for on-policy self-distillation (OPSD) that utilizes complete student rollouts as privileged information to adapt the teacher model. Unlike standard OPSD, PAST preserves the student's distribution on correct trajectories and adapts the teacher toward verified success on failed trajectories using student-proximity regularization. Theoretical analysis shows that forward-KL distillation projects teacher distributions to their conditional arithmetic mean, separating privileged variation from transferable policy shifts. Empirical results on mathematical reasoning benchmarks demonstrate a 5.6 percentage point improvement over Vanilla OPSD.
Entities (9)
Relation Signals (8)
PAST → improves → Vanilla OPSD
confidence 95% · PAST improves the Avg@12 macro average over Vanilla OPSD by 5.6 percentage points.
PAST → uses → complete student trajectories
confidence 95% · PAST treats each completed student trajectory as additional privileged information for the OPSD teacher
PAST → appliedto → Qwen3-1.7B
confidence 90% · We train Qwen3-1.7B in thinking mode for 100 student updates
PAST → evaluatedon → AIME-2024
confidence 90% · Evaluation uses AIME 2024, AIME 2025, and HMMT 2025
PAST → evaluatedon → AIME 2025
confidence 90% · Evaluation uses AIME 2024, AIME 2025, and HMMT 2025
PAST → evaluatedon → HMMT 2025
confidence 90% · Evaluation uses AIME 2024, AIME 2025, and HMMT 2025
Forward-KL distillation → usedby → PAST
confidence 90% · Forward-KL distillation projects the teacher distributions to their conditional arithmetic mean given the prefix.
GRPO → usedby → PAST
confidence 85% · PAST uses this group-relative signal only to adapt its teacher on failed student trajectories.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:On-policy self-distillation (OPSD) uses a privileged teacher to supervise a reasoning model on prefixes sampled from its own rollouts. Yet each rollout also reveals how the student's response unfolds and whether it succeeds, student-specific hindsight that standard OPSD does not use to form the teacher. We introduce Privileged Adaptation from Student Trajectories (PAST), which treats each completed student trajectory as additional privileged information for the OPSD teacher while leaving the student's distillation prefixes unchanged. PAST preserves the student's next-token distribution on correct trajectories and uses failed trajectories to adapt the teacher toward verified success under student-proximity regularization. We characterize what such a trajectory-conditioned teacher can transfer to a prefix-only student. Forward-KL distillation projects the teacher distributions to their conditional arithmetic mean given the prefix. This projection separates trajectory-specific variation that remains privileged from the mean policy shift available to the student. For correct trajectories, the unclipped population objective also has the frozen student as an ideal distributional fixed point. Across three mathematical reasoning benchmarks, PAST improves the Avg@12 macro average over Vanilla OPSD by 5.6 percentage points. A $2\times2$ factorial study shows gains from both complete-trajectory access and teacher adaptation, while trajectory removal and shuffling confirm that the adapted teacher uses the matching hindsight context.
Tags
Links
- Source: https://arxiv.org/abs/2608.08726v1
- Canonical: https://arxiv.org/abs/2608.08726v1
Trouble viewing inline? Open PDF directly →
Full Text
64,266 characters extracted from source content.
Expand or collapse full text
PAST: Privileged Adaptation from Complete Student Trajectories for On-Policy Self-Distillation Yangyang Feng1,∗ Zhuoyan Feng2,∗ Junlan Chen1 1The Hong Kong University of Science and Technology (Guangzhou) 2Sun Yat-sen University yfeng044@connect.hkust-gz.edu.cn fengzhy28@mail2.sysu.edu.cn jchen421@connect.hkust-gz.edu.cn ∗Equal contribution Abstract On-policy self-distillation (OPSD) uses a privileged teacher to supervise a reasoning model on prefixes sampled from its own rollouts. Yet each rollout also reveals how the student’s response unfolds and whether it succeeds, student-specific hindsight that standard OPSD does not use to form the teacher. We introduce Privileged Adaptation from Student Trajectories (PAST), which treats each completed student trajectory as additional privileged information for the OPSD teacher while leaving the student’s distillation prefixes unchanged. PAST preserves the student’s next-token distribution on correct trajectories and uses failed trajectories to adapt the teacher toward verified success under student-proximity regularization. We characterize what such a trajectory-conditioned teacher can transfer to a prefix-only student. Forward-KL distillation projects the teacher distributions to their conditional arithmetic mean given the prefix. This projection separates trajectory-specific variation that remains privileged from the mean policy shift available to the student. For correct trajectories, the unclipped population objective also has the frozen student as an ideal distributional fixed point. Across three mathematical reasoning benchmarks, PAST improves the Avg@12 macro average over Vanilla OPSD by 5.6 percentage points. A 2×22× 2 factorial study shows gains from both complete-trajectory access and teacher adaptation, while trajectory removal and shuffling confirm that the adapted teacher uses the matching hindsight context. 1 Introduction Figure 1: Complete rollouts reveal useful hindsight, but it cannot be copied directly into a prefix-only policy. PAST first adapts the teacher, then distills back to the original prefix. On-policy distillation trains a reasoning model on prefixes sampled from its own rollouts, reducing the mismatch between training and inference while retaining token-level teacher supervision (Agarwal et al., 2024; Gu et al., 2024). On-policy self-distillation (OPSD) makes this interface self-contained. A privileged copy of the model teaches its unprivileged policy on those prefixes. Yet, by the time supervision is constructed, the rollout has also revealed the student’s complete response and whether it succeeds. Standard OPSD uses task-side privilege to teach on the prefixes, but does not adapt the teacher from how the student’s response unfolded (Zhao et al., 2026a). Completed responses already support several forms of self-improvement. SD-Zero conditions a reviser on the response and its outcome, H2SD routes successful and failed responses through different hindsight mechanisms, and RSTG supplies teacher guidance to failed zero-variance RL groups (He et al., 2026; Cai et al., 2026; Han et al., 2026). These methods establish the value of hindsight for revision and guidance. They motivate a broader question. How can complete student trajectories be used more effectively as privileged information in OPSD? For each supervised prefix, the rest of the trajectory is privileged information unavailable to the student. Using complete trajectories as privilege does not automatically produce useful supervision. Future-dependent targets can conflict at the same prefix and cancel under distillation (Zhao et al., 2026b; Wang et al., 2026). Outcome matters as well. Supervision on correct responses should preserve the student’s behavior, whereas supervision on failed responses should move toward verified success without drifting beyond what the student can learn. We introduce Privileged Adaptation from Student Trajectories (PAST), which uses each complete student response as additional privilege for the OPSD teacher. Each cycle freezes the student, samples fresh responses, and adapts one teacher based on final-answer correctness. Full-vocabulary distribution matching preserves the student’s next-token distribution on correct responses. On failed responses, verifier-seeking optimization searches for successful continuations while regularization keeps the teacher close to the student. PAST then distills the adapted teacher once on the original student prefixes. Teacher continuations never replace these prefixes, and the next cycle resamples from the updated student. Our analysis characterizes what a trajectory-conditioned teacher can transfer to a prefix-only student. Forward-KL distillation projects these teacher distributions to their conditional arithmetic mean given the prefix (Lin, 1991; Banerjee et al., 2005). This separates privileged variation that the student cannot reproduce from the mean policy shift it can learn, explaining why teacher use of hindsight need not improve the student. On correct trajectories, the unclipped population objective has the frozen student as an ideal distributional fixed point. Our contributions are summarized below. • We propose PAST, which adapts one privileged teacher from complete correct and failed student trajectories, then distills it onto the student’s unchanged on-policy prefixes. • We characterize forward-KL causal projection and separate future-dependent teacher variation from the policy shift available to a prefix-only student. • Across three mathematical reasoning benchmarks, PAST improves the Avg@12 macro average over Vanilla OPSD by 5.6 percentage points. Factorial controls assign positive contributions to trajectory access and teacher adaptation, while trajectory removal and shuffling confirm that the teacher uses the hindsight context. 2 Preliminaries and Problem Setup PAST combines verifiable policy optimization with on-policy self-distillation. We first define the two training interfaces, then isolate the information change introduced by a complete student trajectory. 2.1 Verifiable On-Policy Learning At training cycle k, let (X,Z)∼(X,Z) denote a task and its task-side privilege, such as a reference solution authorized by the training protocol. The frozen student pkp_k does not observe Z and generates a complete response Y=(Y1,…,YT)∼pk(⋅∣X).Y=(Y_1,…,Y_T) p_k(· X). (1) A verifier returns the trajectory-level outcome R=V(X,Y)∈0,1.R=V(X,Y)∈\0,1\. (2) The verifier may use answer information authorized by the task protocol. It supplies neither token-level labels nor intermediate supervision. Group Relative Policy Optimization (GRPO) converts such outcomes into an on-policy update (Shao et al., 2024). Given G responses Y(g)Y^(g) sampled from an old policy μθold _ _old, let Rg=V(X,Y(g))R_g=V(X,Y^(g)) and A^g=Rg−R¯sR+ϵA,ρg,t(θ)=μθ(Yt(g)∣X,Y<t(g))μθold(Yt(g)∣X,Y<t(g)), A_g= R_g- Rs_R+ _A, _g,t(θ)= _θ(Y_t^(g) X,Y_<t^(g)) _ _old(Y_t^(g) X,Y_<t^(g)), (3) where R¯ R and sRs_R are the group mean and standard deviation. When sR=0s_R=0, the group-relative advantage is zero. Omitting an optional reference-policy penalty, the loss is ℒGRPO(θ)=−[1G∑g=1G1Tg∑t=1Tgminρg,tA^g,clip(ρg,t,1−ϵ,1+ϵ)A^g]. splitL_GRPO(θ)=-E\! [ 1G _g=1^G 1T_g _t=1^T_g \! \ _g,t A_g,clip( _g,t,1-ε,1+ε) A_g \ ]. split (4) Every token in one response shares the same outcome-derived advantage. PAST uses this group-relative signal only to adapt its teacher on failed student trajectories. 2.2 On-Policy Distillation On-policy distillation evaluates a teacher on states generated by the student (Agarwal et al., 2024; Gu et al., 2024). At token position t, the student-facing state is Ht=(X,Y<t,t).H_t=(X,Y_<t,t). (5) For a teacher distribution q(⋅∣Ht)q(· H_t) and a candidate student π(⋅∣Ht)π(· H_t), the forward-KL objective is ℒOPD(π;q)=[1T∑t=1TDKL(q(⋅∣Ht)∥π(⋅∣Ht))],L_OPD(π;q)=E\! [ 1T _t=1^TD_KL\! (q(· H_t)\;\|\;π(· H_t) ) ], (6) where the expectation follows the student rollout distribution and the teacher is a stop-gradient target. The term on-policy refers to the states HtH_t, not to trajectories sampled from the teacher. 2.3 Privileged On-Policy Self-Distillation On-policy self-distillation (OPSD) forms the teacher and student from the same model under different information sets (Zhao et al., 2026a). Vanilla OPSD supplies the teacher with task-side privilege Z while the student observes only HtH_t. qkOPSD(a∣Ht,Z),pk(a∣Ht).q_k^OPSD(a H_t,Z), p_k(a H_t). (7) Its student update instantiates Eq. (6) with q=qkOPSDq=q_k^OPSD. The sampled response determines the prefixes on which both policies are evaluated, but its unobserved suffix is not part of the Vanilla OPSD teacher context. 2.4 Trajectory-Privileged Teacher and Causal Student PAST changes how this privileged teacher is formed. It augments the task-side privilege with the complete student trajectory U=(Y,Z),U=(Y,Z), (8) and adapts one teacher qϕ(a∣Ht,U)q_φ(a H_t,U). The outcome R selects the adaptation objective but is not included in U. Thus the teacher can condition on the suffix of Y beyond HtH_t, while a deployable student π(a∣Ht)π(a H_t) remains prefix-only at inference. Appendix B compares this interface with the closest on-policy distillation protocols. After adaptation, PAST freezes the teacher and defines qU(a∣Ht)=qϕk+1(a∣Ht,U).q_U(a H_t)=q_ _k+1(a H_t,U). (9) The student is trained on the original rollout states with ℒS,k(π)=[1T∑t=1TDKL(qU(⋅∣Ht)∥π(⋅∣Ht))].L_S,k(π)=E\! [ 1T _t=1^TD_KL\! (q_U(· H_t)\;\|\;π(· H_t) ) ]. (10) Optimizing this objective produces pk+1p_k+1, and the next cycle draws fresh rollouts from the updated student. Section 3 defines how correct and failed outcomes adapt the teacher. Section 4 characterizes the supervision that survives projection to the prefix-only student. 3 Method PAST turns each fresh student rollout batch into one teacher update followed by one student update. The completed rollout becomes teacher-side privilege, the verifier outcome selects the adaptation objective, and the adapted teacher supervises the student’s original prefixes. Correct and failed trajectories train the same teacher. Figure 2 summarizes this cycle. Figure 2: PAST adapts a privileged teacher from complete student responses and outcomes, then distills it on the student’s original prefixes. Correct trajectories preserve the current policy. Failed trajectories direct verified improvement under student-proximity regularization, while the student remains prefix-only. Figure 3: Avg@12 over student updates. Lines show three-seed means, bands show seed ranges, and Macro equally weights the three tasks. 3.1 Complete Student Trajectories as Teacher Privilege At cycle k, the frozen student pkp_k generates Y∼pk(⋅∣X)Y p_k(· X) without privileged information. PAST gives the teacher the same task-side privilege Z as Vanilla OPSD and adds the complete student response Y. The external outcome R selects the training objective but is not included in the teacher input. Both outcome branches therefore adapt one conditional distribution qϕ(⋅∣Ht,U)q_φ(· H_t,U) under the same information interface. We initialize the teacher from the student, q0←p0q_0← p_0, so their initial policy difference is zero. Appendix B gives the exact input serialization. Figure 4: Matching trajectories improve teacher success and compatibility (a). Correct-branch KL falls (b) as failed-branch success rises (c). 3.2 Preserving Correct Student Behavior A verified-correct trajectory should train the teacher without replacing behavior that already succeeds. PAST therefore matches the frozen student’s full next-token distribution on the observed trajectory rather than fitting only its sampled tokens. For distributions p and q at one position, define the OPSD-style clipped contribution D~τ(p∥q)=∑v∈minp(v)logp(v)q(v),τ. D_τ(p\|q)= _v \! \p(v) p(v)q(v),τ \. (11) The correct-trajectory loss is ℒ+,i=1Ti∑t=1TiD~τOPSD(pk(⋅∣Hi,t)∥qϕ(⋅∣Hi,t,Ui)).L_+,i= 1T_i _t=1^T_i D_ _OPSD\! (p_k(· H_i,t)\|q_φ(· H_i,t,U_i) ). (12) The threshold and token mask match the corresponding Vanilla OPSD student update. Training uses the clipped quantity for stability and logs the unclipped full-vocabulary KL. No hard-label cross-entropy is added. Figure 5: PAST and generalized JSD retain high quality at low compatibility KL, while sampled-token KL trades quality for throughput. 3.3 Adapting on Failed Student Trajectories A failed trajectory calls for a success-directed teacher update that remains compatible with the frozen student. The teacher samples complete continuations under UiU_i, and the verifier assigns each continuation a binary reward. A group containing both outcomes supplies the group-relative policy signal ℒGRPO,gL_GRPO,g. In this instantiation of Eq. (4), the trainable policy is the trajectory-conditioned teacher qϕq_φ, and the old policy is its frozen snapshot qϕkq_ _k from the start of the teacher update. For each verifier-successful continuation CjC_j in group g, PAST measures its exact proximity to the frozen student Kj=1|Cj|∑sDKL(qϕ(⋅∣Gj,s,Ui)∥pk(⋅∣Gj,s)),K_j= 1|C_j| _sD_KL\! (q_φ(· G_j,s,U_i)\|p_k(· G_j,s) ), (13) where Gj,sG_j,s is the teacher continuation prefix at position s. Let gS_g contain the successful continuations and let mgmixm_g^mix indicate a group with both outcomes. An active failed group minimizes ℒ−,g=mgmixℒGRPO,g+βKL1|g|∑j∈gKj,βKL=0.05.L_-,g=m_g^mixL_GRPO,g+ _KL 1|S_g| _j _gK_j, _KL=0.05. (14) Mixed groups combine success seeking with student proximity. All-success groups retain only the proximity term. Adaptive sampling adds continuations after an initial all-failure draw and skips a trajectory only when a full retry also fails. This rule changes sampling cost rather than the stated population objective. Appendix B gives the complete sampler. PAST first averages examples within each active branch and then averages the active branches ℒT=m+ℒ¯++m−ℒ¯−m++m−,L_T= m_+ L_++m_- L_-m_++m_-, (15) where m+m_+ and m−m_- indicate whether the corresponding branch contains a valid training term. The observed correct-to-failed ratio therefore does not set the branch weights implicitly. 3.4 Causal Student Distillation The student receives the adapted teacher’s supervision only on states generated by the frozen student. After one teacher update, PAST freezes qϕk+1q_ _k+1 and performs one Vanilla OPSD student update on the original states HtH_t. The implementation applies Eq. (11) in the forward direction qϕk+1∥πq_ _k+1\|π with the baseline vocabulary, threshold, mask, temperature, and reduction. Teacher-generated continuations from the failed branch are not student training paths. The updated policy becomes pk+1p_k+1, cycle-local rollouts are released, and the next cycle samples from the new student. Appendix B gives the complete update and snapshot lifecycle. 4 Theory PAST gives its teacher access to a complete student trajectory but requires the deployed student to act from its current prefix. We characterize the exact causal target of forward-KL distillation, show why even a perfect privileged teacher can provide no useful student update, and give sufficient conditions under which a teacher change survives projection as a local improvement direction. Figure 6: Adaptive sampling reduces its base group size as teacher-success EMA rises over 100 student cycles. Fix one training cycle. Let (H,U)(H,U) follow the distribution induced by the frozen student’s on-policy rollouts, where H is a student prefix and U contains the complete trajectory and task-side privilege. Define qU(a∣H)=qϕ(a∣H,U),q¯(a∣H)=[qU(a∣H)∣H].q_U(a H)=q_φ(a H,U), q(a H)=E[q_U(a H) H]. (16) The results use the unclipped population KL under the support and integrability conditions stated in Appendix C. Proposition 1 (Privileged-to-causal projection). For every prefix-only policy π(a∣H)π(a H), [DKL(qU∥π)]=[DKL(qU∥q¯)]+[DKL(q¯∥π)]. splitE\! [D_KL(q_U\|π) ]=&E\! [D_KL(q_U\| q) ]\\ &+E\! [D_KL( q\|π) ]. split (17) Consequently, q¯ q is the unique minimizer wherever it has positive support. Forward-KL distillation retains the conditional arithmetic mean of the trajectory-conditioned teacher distributions. The first term in Eq. (17) is variation that no prefix-only policy can reproduce. The second is the remaining approximation error to the causal target q¯ q. Corollary 2 (Projection onto a restricted student family). Let Π be any class of prefix-only policies for which the displayed expectations are finite. For every π∈Ππ∈ , [DKL(qU∥π)]−infρ∈Π[DKL(qU∥ρ)]=[DKL(q¯∥π)]−infρ∈Π[DKL(q¯∥ρ)]. split&E[D_KL(q_U\|π)]- _ρ∈ E[D_KL(q_U\|ρ)]\\ =&E[D_KL( q\|π)]- _ρ∈ E[D_KL( q\|ρ)]. split (18) The two objectives have the same minimizing sequences and, whenever a minimum is attained, the same minimizers over Π . Restricting the student family changes how closely it can represent q¯ q, but not the causal target selected by forward-KL distillation. The residual error therefore reflects the approximation limit of Π rather than a different target induced by privileged variation. Corollary 3 (Privilege use and distillable shift). Let A∼qU(⋅∣H)A q_U(· H). Then [DKL(qU∥q¯)]=I(A;U∣H),E[D_KL(q_U\| q)]=I(A;U H), (19) and, relative to the frozen student pkp_k, [DKL(qU∥pk)]=I(A;U∣H)+[DKL(q¯∥pk)].E[D_KL(q_U\|p_k)]=I(A;U H)+E[D_KL( q\|p_k)]. (20) Equation (20) separates teacher change into privileged variation and the causal shift available to the student. Privileged variation can consume the entire teacher–student divergence without moving the student target. Proposition 4 (Perfect privileged teachers need not transfer). For every horizon T≥2T≥ 2, there is a binary verification problem, a prefix-only student pkp_k, and a teacher that uses the student’s complete trajectory such that the teacher succeeds with probability one, while q¯=pk q=p_k at every supervised prefix. Exact distillation leaves the student unchanged, and its success probability is 21−T2^1-T. The gap can therefore approach one as the response grows. Teacher success alone gives no nontrivial lower bound on student improvement. Appendix C gives the construction, in which the teacher uses a future student token to coordinate an otherwise successful response while every projected next-token target remains uniform. We next identify a positive transfer condition. Let Qk(H,a)∈[0,1]Q_k(H,a)∈[0,1] be the probability of verifier success after taking action a at H and continuing with the frozen student. For any prefix-only distribution r, define its frozen-student value shift gk(r;H)=∑a(r(a∣H)−pk(a∣H))Qk(H,a).g_k(r;H)= _a (r(a H)-p_k(a H) )Q_k(H,a). (21) Proposition 5 (Value-aligned causal transfer). For almost every prefix H, gk(q¯;H)=[gk(qU;H)∣H].g_k( q;H)=E[g_k(q_U;H) H]. (22) Thus trajectory-specific changes may cancel in distribution, but any positive mean shift under the frozen student’s value function survives causal projection. The two outcome branches make this condition method-specific. Let η(H)=P(R=0∣H)η(H)=P(R=0 H) and let q¯+ q_+ and q¯− q_- denote the conditional mean teachers on correct and failed trajectories. Suppose δ+(H)=[DKL(pk∥qU)∣H,R=1],γ−(H)=gk(q¯−;H),δS(H)=DKL(q¯∥π). split _+(H)&=E[D_KL(p_k\|q_U) H,R=1],\\ _-(H)&=g_k( q_-;H), _S(H)=D_KL( q\|π). split (23) Corollary 6 (Outcome-conditioned transfer margin). For any prefix-only student π, gk(π;H)≥ηγ−(1−η)δ+/2−δS/2.g_k(π;H)≥η _--(1-η) _+/2- _S/2. (24) The projected update has positive frozen-student value whenever the failed-branch margin exceeds the preservation and distillation errors on the right-hand side. The correct branch makes δ+=0 _+=0 at its ideal unclipped distributional optimum. A mean correct-branch objective at most δ implies mean total variation at most δ/2 δ/2. The failed branch must supply γ−>0 _->0, not merely high privileged-teacher success. Its student-proximity term targets the compatibility needed for that distinction. Appendix C proves a trajectory-level compatibility bound and connects the implemented success-path regularizer to this condition. Finally, the value margin has a rollout-level implication. For the conservative interpolation πϵ=(1−ϵ)pk+ϵπ _ε=(1-ε)p_k+επ in a finite-horizon problem, dϵJ(πϵ)|ϵ=0=∑tHt∼pk[gk(π;Ht)]. . ddεJ( _ε) |_ε=0= _tE_H_t p_k[g_k(π;H_t)]. (25) If the right-hand side is positive, sufficiently small ϵ>0ε>0 improves verifier success. The theorem identifies the transferable quantity and the positive margin required for local improvement. The implemented update targets this direction through pointwise clipping and a shared-parameter policy. The trajectory perturbations in Section 5.4 test privilege use, while the factorial and objective studies test whether teacher adaptation produces a better causal student rather than only a more successful privileged teacher. 5 Experiments The experiments test causal student improvement, factorial attribution, matching-trajectory use, outcome-branch contributions, objective robustness, sampling efficiency, and scale. 5.1 Setup We train Qwen3-1.7B in thinking mode for 100 student updates with seeds 17, 29, and 43. Evaluation uses AIME 2024, AIME 2025, and HMMT 2025 at temperature 1.0 with 12 samples per problem. We report task Avg@12 and their equally weighted macro average. All methods share the student-update count and training protocol; confidence intervals use a paired bootstrap stratified by problem and seed. Appendix D gives the complete protocol and checks. 5.2 PAST Improves the Causal Student PAST produces the strongest final student in the comparison set. Its Avg@12 macro average is 48.42648.426, a 5.6175.617-point improvement over Vanilla OPSD with a 95% confidence interval of [2.592,8.333][2.592,8.333]. PAST also leads every task and exceeds H2SD and SD-Zero in macro average (Table 1). Table 1: Final Avg@12 after 100 updates. PAST exceeds Vanilla OPSD by 5.6175.617 points (95% CI [2.592,8.333][2.592,8.333]). Method AIME24 AIME25 HMMT25 Macro Base 53.056 34.907 27.870 38.611 SFT 55.000 37.222 27.407 39.877 GRPO 55.463 38.056 27.963 40.494 Vanilla OPSD 57.130 39.722 31.574 42.809 SD-Zero 59.444 42.315 30.185 43.981 H2SD 61.667 42.500 32.130 45.432 PAST 63.519 45.833 35.926 48.426 PAST’s advantage develops over training. Its macro average rises at every checkpoint and progressively separates from Vanilla OPSD (Figure 3). The HMMT25 gain appears early; AIME24 and AIME25 vary more across checkpoints, but PAST finishes with the highest mean on all three tasks. 5.3 Factorial Attribution of the PAST Gain The 2×22× 2 design separates trajectory access from teacher adaptation. The cell without either factor is Vanilla OPSD. Trajectory-only changes the teacher input; teacher-only adapts without the completed response; PAST combines the matching trajectory with adaptation. Neither factor alone reproduces the gain. Trajectory access changes the teacher context but adds only 0.5550.555 macro points, while adaptation without the completed response adds 0.0920.092 points. Each condition lowers at least one task. PAST instead reaches 48.42648.426 and improves all three. This pattern supports complementary roles. The matching trajectory supplies student-specific hindsight, and adaptation converts it into a more useful student-facing target. Table 2: PAST combines trajectory access and teacher adaptation to produce the strongest student. Trajectory Trained teacher AIME24 AIME25 HMMT25 Macro No No 57.130 39.722 31.574 42.809 No Yes 59.074 38.056 31.574 42.901 Yes No 59.537 41.111 29.444 43.364 Yes Yes 63.519 45.833 35.926 48.426 5.4 The Teacher Uses Matching Trajectory Privilege The teacher depends on the trajectory matching the current response. Teacher success is its continuation pass rate, while distribution JSD measures next-token change. Removing the trajectory tests student-specific context; cross-problem shuffling retains the field but breaks its correspondence with the problem and prefix, separating matched hindsight from an additional input alone. Both perturbations weaken the teacher, with a larger effect from shuffling. Relative to the full condition, removal lowers success by 0.0640.064 and raises JSD by 0.0670.067; shuffling lowers success by 0.0890.089 and raises JSD by 0.1100.110 (Figure 4a). In the E11 trace, correct-branch KL falls from 0.1860.186 to 0.0770.077 and failed-branch success rises from 0.4100.410 to 0.4840.484 (Figures 4b and 4c). These diagnostics establish matching-trajectory use and branch-level adaptation; Table 2 separately establishes the final-student contribution. 5.5 Joint Outcome-Conditioned Adaptation Performs Best Joint correct-and-failed training leads every task, reaching 48.42648.426 macro versus 45.86445.864 for failed-only and 44.90844.908 for correct-only training (Table 6). Its 2.5622.562-point advantage over the stronger single branch supports joint adaptation. 5.6 Objective Robustness, Sampling Efficiency, and Scale PAST remains effective across alternative distribution objectives and reduces teacher sampling cost, while the 4B extension shows task-dependent gains. Figure 5 shows that generalized JSD reaches 47.62347.623, close to 47.56247.562 for full-vocabulary KL at slightly lower compatibility KL, while sampled-token KL trades lower quality for higher throughput. Adaptive sampling uses 17.6%17.6\% fewer teacher samples, reducing the count from 1,032 to 850, while reaching a slightly higher macro average of 47.28447.284 versus 46.72846.728 for fixed-size sampling (Table 8). The controller reduces routine sampling once teacher success stabilizes while retaining extra draws for failed groups without a successful continuation (Figure 6). On Qwen3-4B, PAST raises the macro average from 47.68547.685 to 48.51948.519, with gains on AIME25 and HMMT25 and a lower AIME24 score (Table 9). Appendices E and F give the complete ablations, learning curves, optimization diagnostics, 4B results, and training-cost accounting. 6 Conclusion PAST uses completed on-policy responses as teacher-side privilege while retaining a prefix-only student interface. It preserves verified-correct behavior and adapts failed trajectories toward verifier success before distilling on the student’s original prefixes. Our analysis identifies the causal projection of this future-conditioned teacher and conditions for improving verified success. Across three mathematical reasoning benchmarks, PAST improves Avg@12 over Vanilla OPSD by 5.6 points. Factorial and trajectory-perturbation studies attribute this gain to teacher adaptation with matching trajectories, showing how completed student experience can improve a causally deployed policy. Limitations The main evidence uses a 1.7B student, while the one-seed 4B extension gives mixed task-level results. Broader scale behavior therefore remains open. The verifier checks final answers and does not validate each intermediate reasoning step. Reliable process feedback could extend the supervision signal within the same interface. The failed branch also uses success masking, adaptive group sizes, one retry, and all-failure skipping. Our theory covers the population objectives and causal projection rather than unbiasedness of this finite-sample estimator. The appendix reports skip and group-composition diagnostics that make this estimator boundary measurable. References R. Agarwal, N. Vieillard, Y. Zhou, P. Stanczyk, S. Ramos, M. Geist, and O. Bachem (2024) On-policy distillation of language models: learning from self-generated mistakes. In International Conference on Learning Representations, Cited by: Appendix A, Table 3, §1, §2.2. A. Banerjee, S. Merugu, I. S. Dhillon, and J. Ghosh (2005) Clustering with bregman divergences. Journal of Machine Learning Research 6 (58), p. 1705–1749. Cited by: §1. Q. Cai, Y. Ma, P. Li, Y. Chen, Q. Guo, Y. Zou, L. Li, X. Feng, and B. Qin (2026) H2SD: hybrid hindsight self-distillation. arXiv preprint arXiv:2607.18955. Cited by: Appendix A, Table 3, §1. Y. Gu, L. Dong, F. Wei, and M. Huang (2024) MiniLLM: on-policy distillation of large language models. In International Conference on Learning Representations, Cited by: Appendix A, Table 3, §1, §2.2. Z. Han, J. Xiao, Z. Lu, R. Jin, Z. Yao, Y. Liu, H. Hao, Y. Sun, Y. Yang, Q. Gu, X. Cai, and D. Xiong (2026) Distill where you fail: recovering learning signals of negative rl-groups from adaptive teacher guidance. arXiv preprint arXiv:2608.00782. Cited by: Appendix A, §1. Y. He, S. Kaur, A. Bhaskar, Y. Yang, J. Liu, N. Ri, L. Fowl, A. Panigrahi, D. Chen, and S. Arora (2026) Self-distillation zero: self-revision turns binary rewards into dense supervision. arXiv preprint arXiv:2604.12002. Cited by: Appendix A, Table 3, §1. J. Ko, S. Kim, T. Chen, and S. Yun (2024) DistiLLM: towards streamlined distillation for large language models. In International Conference on Machine Learning, Cited by: Appendix A. J. Lin (1991) Divergence measures based on the shannon entropy. IEEE Transactions on Information Theory 37 (1), p. 145–151. External Links: Document Cited by: §1. X. Liu, X. Wang, Y. Ma, Y. Zhang, and C. Xiao (2026) When are teacher tokens reliable? position-weighted on-policy self-distillation for reasoning. arXiv preprint arXiv:2605.21606. Cited by: Appendix A. D. Lopez-Paz, L. Bottou, B. Schölkopf, and V. Vapnik (2016) Unifying distillation and privileged information. In International Conference on Learning Representations, Cited by: Appendix A. J. Rao, X. Liu, Z. Lin, L. Ding, J. Li, D. Tao, and M. Zhang (2026) Exploring and enhancing the transfer of distribution in knowledge distillation for autoregressive language models. Knowledge-Based Systems 337, p. 115382. Cited by: Appendix A. Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo (2024) DeepSeekMath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: §2.1. Z. Tan and Y. Hong (2026) PAINT: partial-solution adaptive interpolated training for self-distilled reasoners. arXiv preprint arXiv:2604.26573. Cited by: Appendix A. V. Vapnik and A. Vashist (2009) A new learning paradigm: learning using privileged information. Neural Networks 22 (5–6), p. 544–557. Cited by: Appendix A. Y. Wang, S. Lu, Y. Gu, P. Wang, Y. Yang, Z. Yan, C. Xie, J. Wu, and H. Yang (2026) Not all disagreement is learnable: token teachability in on-policy distillation. arXiv preprint arXiv:2605.26844. Cited by: Appendix A, §1. S. Zhao, Z. Xie, M. Liu, J. Huang, G. Pang, F. Chen, and A. Grover (2026a) Self-distilled reasoner: on-policy self-distillation for large language models. arXiv preprint arXiv:2601.18734. Cited by: Appendix A, Table 3, §1, §2.3. X. Zhao, L. Zhang, Z. Xu, Z. Wang, C. Xu, S. Zhao, and Q. Li (2026b) Is more privileged information better? from solution traces to problem-solving structure in self-distilled reasoning. arXiv preprint arXiv:2608.01589. Cited by: Appendix A, §1. Appendix A Related Work On-policy distillation. On-policy knowledge distillation trains on student-generated states and reduces the mismatch between teacher-generated training sequences and student-generated inference states (Agarwal et al., 2024; Gu et al., 2024). DistiLLM studies stable divergence choices and efficient updates for this setting (Ko et al., 2024). On-policy self-distillation (OPSD) specializes the interface to reasoning, where a privileged teacher supplies full-vocabulary targets on an unprivileged student’s rollout prefixes (Zhao et al., 2026a). PAST retains these student-facing prefixes but changes how the teacher is formed. The completed student response becomes additional teacher privilege and trains the teacher before distillation. Hindsight supervision for reasoning. Completed responses and outcome feedback support several forms of self-improvement. SD-Zero conditions a reviser on a completed response and its outcome before distilling the revision (He et al., 2026). H2SD applies different hindsight mechanisms to successful and failed responses (Cai et al., 2026), while RSTG supplies adaptive teacher guidance to failed zero-variance reinforcement-learning groups (Han et al., 2026). Other methods change the privileged representation or select supervision through problem-solving structure, partial solutions, position reliability, or token teachability (Zhao et al., 2026b; Tan and Hong, 2026; Liu et al., 2026; Wang et al., 2026). PAST uses the completed response for a different operation. It adapts one OPSD teacher with preservation on correct trajectories and verifier-seeking, student-regularized optimization on failed trajectories, then distills that teacher on the original student prefixes. Privileged and trainable teachers. Learning using privileged information studies variables available during training but absent at inference (Vapnik and Vashist, 2009), and generalized distillation connects this setting to teacher-student learning (Lopez-Paz et al., 2016). Trainable teachers provide a complementary direction. For example, OKD adapts online teacher modules during autoregressive distillation (Rao et al., 2026). PAST combines teacher adaptation with a specific information constraint. Its teacher observes the student’s complete future trajectory during training, while the deployed student remains prefix-only. Section 4 characterizes the part of this future-dependent teaching distribution that survives causal projection. Appendix B Method and Protocol Details B.1 Comparison of Student-Facing Interfaces Table 3 separates the source of the teacher distribution from the states used for the student update. All listed distillation methods train on prefixes sampled from the student. Their main differences lie in teacher construction, privileged context, and the role of outcome feedback. Table 3: Comparison of student-facing training interfaces. PAST uses the completed response to adapt the teacher while retaining the original student prefixes for distillation. Method Teacher construction Teacher context beyond the prefix Outcome role Student update states OPD (Agarwal et al., 2024; Gu et al., 2024) Fixed external teacher No additional privilege Not required Student prefixes OPSD (Zhao et al., 2026a) Stop-gradient privileged self-teacher Task-side privilege Z Not required Student prefixes SD-Zero (He et al., 2026) Self-revision-trained teacher, frozen during distillation Complete response and outcome-conditioned instruction Enters the teacher context Student prefixes H2SD (Cai et al., 2026) Fixed self-teacher with offline hints Verified response or external hint Selects the success or failure objective Student prefixes PAST Cycle-adapted privileged self-teacher Task privilege Z and complete response Y Selects teacher adaptation but is excluded from U Original student prefixes B.2 Teacher Input Serialization All teacher phases use one user message and the model’s native chat template. The serialized fields appear in the following order: Problem: problem Please reason step by step, and put your final answer within . === Student Attempt Begin === student_output === Student Attempt End === === Reference Solution Begin === reference_solution === Reference Solution End === The student attempt above may be correct or incorrect. Use it as hindsight context for the student’s reasoning state. Maintain the student’s established reasoning style and presentation whenever they are compatible with a correct solution. Use the reference solution to ensure correctness, but do not copy or paraphrase the reference solution. Now solve the original problem through your own reasoning, and put the final answer within . The student attempt is always produced by the current unprivileged student. Correct and failed trajectories use the same message; correctness is kept outside the prompt and controls only routing and verification. In the correct branch and student distillation phase, the assistant segment is teacher-forced on the original student response. In the failed branch, the teacher samples the assistant segment autoregressively. B.3 Adaptive Failed-Branch Sampling Let GmaxG_ be the fixed group size of the GRPO baseline and let Gbase∈[1,Gmax]G_base∈[1,G_ ] be the current base draw. Warm-up starts at Gbase=GmaxG_base=G_ . Once the recent teacher success rate stabilizes above the configured threshold, the controller decreases GbaseG_base one step at a time, down to one. For each failed student trajectory, PAST applies the following state machine. 1. Draw GbaseG_base teacher continuations. 2. If at least one succeeds, classify the realized group as mixed or all-success and compute Eq. (14). 3. If all fail, add one continuation at a time until the first success or until the group reaches GmaxG_ . 4. If a size-GmaxG_ group remains all-failure, redraw one complete size-GmaxG_ group. Skip the trajectory only when this retry also contains no success. A mixed group uses relative advantages and successful-path KL. An all-success group, including the single-success case, has zero relative advantage and uses only successful-path KL. An all-failure group contributes neither GRPO nor student-proximity KL. B.4 Loss Reduction and Clipping Diagnostics For N+N_+ verified-correct trajectories and an active failed-group set −actG_-^act, the branch means are ℒ¯+=1N+∑i:Ri=1ℒ+,i,ℒ¯−=1|−act|∑g∈−actℒ−,g. L_+= 1N_+ _i:R_i=1L_+,i, L_-= 1|G_-^act| _g _-^actL_-,g. (26) Only existing means enter Eq. (15); the optimizer step is skipped when neither branch is active. Correct-teacher and student-distillation losses clip individual vocabulary contributions at the Vanilla OPSD threshold. Failed-path KL remains exact. Training logs retain the clipped loss, unclipped KL, clip fraction, removed positive mass, failed-path KL quantiles, and non-finite counts. B.5 Student and Teacher Snapshot Lifecycle PAST uses one frozen base model with separate student and teacher LoRA adapters. At the start of cycle k, the student adapter is the read-only policy pkp_k and the teacher adapter is qkq_k. The student adapter generates the rollout; only the teacher adapter is trainable during teacher adaptation. PAST then freezes qk+1q_k+1 and makes the student adapter trainable for one logical update from pkp_k to pk+1p_k+1. Gradient accumulation may split either logical update into microbatches but does not add optimizer steps or replay rollouts. Checkpoints are written only at complete cycle boundaries and contain both adapters, both optimizer and scheduler states, the adaptive-sampling controller, data position, random-number-generator states, and configuration identifiers. An interrupted partial cycle is discarded and restarted from the preceding boundary. Full-vocabulary reference logits are recomputed under the appropriate frozen adapter and released after each microbatch rather than stored across phases. Appendix C Complete Theory and Proofs C.1 Conditions and Conditioning Measure All expectations in Section 4 are taken under the joint distribution induced by the task distribution, the frozen student’s on-policy rollout, the prefix-selection rule, and the adapted teacher’s next-token distribution. For almost every prefix H, the conditional teacher distributions qU(⋅∣H)q_U(· H) are defined on the common vocabulary V. We assume that the displayed KL divergences and conditional expectations are finite. Uniqueness holds on actions with positive mass under the corresponding target distribution. These conditions let conditional expectation and finite-vocabulary summation be interchanged below. The value results additionally assume a finite-horizon generation problem with terminal verifier reward in [0,1][0,1]. For a frozen student pkp_k, Qk(H,a)Q_k(H,a) is the expected terminal reward after taking a at H and following pkp_k thereafter. Conditional branch means are needed only when the corresponding event has positive probability. Terms multiplied by a zero branch probability are defined as zero. C.2 Proof of the Causal Projection Proof of Proposition 1. For any prefix-only π, insert q¯ q into the log ratio logqU(A∣H)π(A∣H)=logqU(A∣H)q¯(A∣H)+logq¯(A∣H)π(A∣H). q_U(A H)π(A H)= q_U(A H) q(A H)+ q(A H)π(A H). (27) Taking expectations gives the first term in Eq. (17). For the second term, conditioning on H replaces qU(a∣H)q_U(a H) by [qU(a∣H)∣H]=q¯(a∣H)E[q_U(a H) H]= q(a H), which yields [DKL(q¯∥π)]E[D_KL( q\|π)]. Nonnegativity establishes optimality, and equality requires π=q¯π= q wherever q¯ q has positive support. ∎ Proof of Corollary 2. Proposition 1 writes the privileged-teacher objective as [DKL(qU∥π)]=[DKL(qU∥q¯)]+[DKL(q¯∥π)].E[D_KL(q_U\|π)]=E[D_KL(q_U\| q)]+E[D_KL( q\|π)]. The first term is independent of π. Subtracting the infimum over ρ∈Πρ∈ gives Eq. (18). The excess risks are identical pointwise, so the objectives have the same minimizing sequences and the same minimizers whenever a minimum is attained. If q¯∈Π q∈ , the remaining KL is minimized uniquely by q¯ q on its support. ∎ Proof of Corollary 3. Under the joint law P(A,U∣H)=P(U∣H)qU(A∣H)P(A,U H)=P(U H)q_U(A H), the conditional marginal of A is q¯(A∣H) q(A H). The definition of conditional mutual information gives Eq. (19). Applying Proposition 1 with π=pkπ=p_k gives Eq. (20). ∎ Whenever the expected teacher–student KL is positive, Eq. (20) also gives the descriptive fraction [DKL(q¯∥pk)][DKL(qU∥pk)]=1−I(A;U∣H)[DKL(qU∥pk)]. E[D_KL( q\|p_k)]E[D_KL(q_U\|p_k)]=1- I(A;U H)E[D_KL(q_U\|p_k)]. (28) This ratio is an interpretation of the decomposition, not an objective optimized explicitly by PAST. C.3 A Perfect Privileged Teacher Need Not Transfer Proof of Proposition 4. Fix T≥2T≥ 2. A response is a binary string C=(C1,…,CT)C=(C_1,…,C_T), and the verifier accepts exactly the two constant strings V(C)=C1=⋯=CT.V(C)=1\C_1=·s=C_T\. (29) Let the frozen student generate independent fair bits. Its success probability is therefore 2/2T=21−T2/2^T=2^1-T. Draw the student’s complete training trajectory Y∼pkY p_k and include it in U. Let B=YTB=Y_T, which is available to the privileged teacher but is absent from every student prefix Ht=Y<tH_t=Y_<t. Define the teacher to output B deterministically at every position, qU(a∣Ht)=a=B.q_U(a H_t)=1\a=B\. (30) When this teacher generates a response under the fixed privilege U, it produces (B,…,B)(B,…,B) and succeeds with probability one. For every supervised position t≤Tt≤ T, the fair bit B=YTB=Y_T is independent of Ht=Y<tH_t=Y_<t. Hence q¯(a∣Ht)=[a=B∣Ht]=12=pk(a∣Ht). q(a H_t)=E[1\a=B\ H_t]= 12=p_k(a H_t). (31) The exact forward-KL projection is the original student at every supervised prefix, so distillation makes no update. Moreover, A=BA=B under the teacher and B remains uniform given HtH_t, which gives I(A;U∣Ht)=log2I(A;U H_t)= 2 while DKL(q¯∥pk)=0D_KL( q\|p_k)=0. The privileged teacher is perfect, yet its advantage over the causal student approaches one as T grows. ∎ The construction uses the future trajectory as a coordination variable. Its success is a property of repeatedly conditioning on the same unavailable bit, not of a next-token change that a prefix-only student can retain. C.4 Value-Aligned Causal Transfer Proof of Proposition 5. Condition on H. Since Qk(H,a)Q_k(H,a) and pk(a∣H)p_k(a H) do not depend on U, [gk(qU;H)∣H] [g_k(q_U;H) H] =∑a([qU(a∣H)∣H]−pk(a∣H))Qk(H,a) = _a (E[q_U(a H) H]-p_k(a H) )Q_k(H,a) (32) =∑a(q¯(a∣H)−pk(a∣H))Qk(H,a) = _a ( q(a H)-p_k(a H) )Q_k(H,a) (33) =gk(q¯;H). =g_k( q;H). (34) Thus conditional averaging preserves the mean of every bounded prefix-action value shared across privilege conditions. Distributional cancellation erases value only when positive and negative value shifts cancel as well. ∎ Let η=P(R=0∣H)η=P(R=0 H). The conditional barycenter obeys q¯=(1−η)q¯++ηq¯−, q=(1-η) q_++η q_-, (35) and linearity gives gk(q¯;H)=(1−η)gk(q¯+;H)+ηgk(q¯−;H).g_k( q;H)=(1-η)g_k( q_+;H)+η g_k( q_-;H). (36) At the ideal correct-branch optimum q¯+=pk q_+=p_k, the full projected shift is q¯−pk=η(q¯−pk) q-p_k=η( q_--p_k). In particular, TV(q¯,pk)=ηTV(q¯−,pk).TV( q,p_k)= ( q_-,p_k). (37) Failed-branch signal is therefore attenuated by its conditional frequency even before student approximation error is introduced. Proof of Corollary 6. Because Qk(H,⋅)∈[0,1]Q_k(H,·)∈[0,1], the difference in its expectation under two action distributions is at most their total variation. Convexity of total variation, Pinsker’s inequality, and Jensen’s inequality give gk(q¯+;H) g_k( q_+;H) ≥−TV(q¯+,pk) ≥-TV( q_+,p_k) (38) ≥−[TV(qU,pk)∣H,R=1] ≥-E[TV(q_U,p_k) H,R=1] (39) ≥−δ+(H)/2. ≥- _+(H)/2. (40) Using Eq. (36) and γ−=gk(q¯−;H) _-=g_k( q_-;H) yields gk(q¯;H)≥ηγ−(1−η)δ+/2.g_k( q;H)≥η _--(1-η) _+/2. (41) The same bounded-value argument and Pinsker’s inequality give |gk(π;H)−gk(q¯;H)|≤TV(π,q¯)≤DKL(q¯∥π)/2.|g_k(π;H)-g_k( q;H)| (π, q)≤ D_KL( q\|π)/2. (42) Substituting δS=DKL(q¯∥π) _S=D_KL( q\|π) proves Eq. (24). ∎ C.5 Compatibility Between Privileged and Student Continuations The failed teacher is evaluated under its own privileged continuation, whereas gkg_k evaluates the teacher’s next-token distribution followed by the frozen student. A continuation-distance condition connects these quantities. Proposition 7 (Continuation compatibility). Fix (H,U)(H,U). Let ℚUQ_U be the distribution of a complete continuation generated by qUq_U. Let UM_U draw its first action from qU(⋅∣H)q_U(· H) and all later actions from pkp_k. Write sU=ℚU[V]s_U=E_Q_U[V] and vk=pk[V∣H]v_k=E_p_k[V H]. Then gk(qU;H)≥sU−vk−DKL(ℚU∥U)/2.g_k(q_U;H)≥ s_U-v_k- D_KL(Q_U\|M_U)/2. (43) Moreover, the trajectory KL has the chain-rule form DKL(ℚU∥U)=C∼ℚU[∑s>tDKL(qU(⋅∣Gs)∥pk(⋅∣Gs))],D_KL(Q_U\|M_U)=E_C _U [ _s>tD_KL\! (q_U(· G_s)\|p_k(· G_s) ) ], (44) where t is the position following H and GsG_s is the generated prefix at position s. Proof. The hybrid success probability is U[V]=∑aqU(a∣H)Qk(H,a)=vk+gk(qU;H).E_M_U[V]= _aq_U(a H)Q_k(H,a)=v_k+g_k(q_U;H). (45) Since V∈[0,1]V∈[0,1], its expectation differs under ℚUQ_U and UM_U by at most their total variation. Pinsker’s inequality gives sU−U[V]≤TV(ℚU,U)≤DKL(ℚU∥U)/2,s_U-E_M_U[V] (Q_U,M_U)≤ D_KL(Q_U\|M_U)/2, (46) which proves Eq. (43). The two trajectory laws use the same first-action distribution. Applying the KL chain rule to all later positions gives Eq. (44). ∎ Proposition 7 formalizes the role of student proximity. High privileged success gives a positive compatible value margin only when the teacher’s advantage exceeds its continuation mismatch. The implemented quantity in Eq. (13) evaluates the same tokenwise divergence on sampled verifier-successful paths. Because that loss is success-conditioned rather than the unconditional trajectory KL in Eq. (44), it encourages compatibility but does not certify this bound globally. C.6 Local Rollout Improvement Proposition 8 (Local policy improvement). Let J(r)J(r) denote expected terminal verifier reward under a finite-horizon prefix-only policy r. For any prefix-only π, define πϵ=(1−ϵ)pk+ϵπ _ε=(1-ε)p_k+επ. Then Eq. (25) holds. If its right-hand side is strictly positive, there exists ϵ0>0 _0>0 such that J(πϵ)>J(pk)J( _ε)>J(p_k) for every ϵ∈(0,ϵ0)ε∈(0, _0). Proof. Because the horizon and vocabulary are finite, J(πϵ)J( _ε) is a finite sum of products of action probabilities and is differentiable at ϵ=0ε=0. Differentiating one policy factor at a time leaves pkp_k on every preceding and subsequent factor. The contribution at position t is Ht∼pk[∑a(π(a∣Ht)−pk(a∣Ht))Qk(Ht,a)]=Ht∼pk[gk(π;Ht)].E_H_t p_k [ _a (π(a H_t)-p_k(a H_t) )Q_k(H_t,a) ]=E_H_t p_k[g_k(π;H_t)]. (47) Summing over positions proves Eq. (25). A differentiable function with a strictly positive derivative at zero is larger than its value at zero throughout some sufficiently small right neighborhood, which proves the second claim. ∎ This result concerns a conservative policy-space interpolation. It does not identify a finite parameter-space gradient step with πϵ _ε, nor does it establish improvement when the occupancy-weighted margin is nonpositive. C.7 Forward and Reverse Projection Forward and reverse KL aggregate privileged targets differently. Proposition 1 gives the arithmetic barycenter argminπ[DKL(qU∥π)]=[qU∣H]. *arg\,min_πE[D_KL(q_U\|π)]=E[q_U H]. (48) For reverse KL, direct Lagrangian optimization gives argminπ[DKL(π∥qU)]∝exp([logqU∣H]). *arg\,min_πE[D_KL(π\|q_U)] \! (E[ q_U H] ). (49) PAST uses the divergence direction that preserves the conditional mean soft target. Reverse KL instead suppresses actions that are not supported consistently across hindsight conditions. C.8 Correct-Branch Stability and the Clipping Boundary Proposition 9 (Correct trajectories are ideal fixed points). On verified-correct trajectories, consider the unclipped population objective ℒ+(q)=[DKL(pk∥q)∣R=1].L_+(q)=E[D_KL(p_k\|q) R=1]. (50) Its distributional optimum is q=pkq=p_k on the supported prefixes. If ℒ+(q)≤δL_+(q)≤δ, then [TV(pk,q)∣R=1]≤δ/2.E[TV(p_k,q) R=1]≤ δ/2. (51) Proof of Proposition 9. Nonnegativity of KL gives the unique supported distributional optimum q=pkq=p_k. For the approximate statement, Pinsker’s inequality gives TV(pk,q)≤DKL(pk∥q)/2TV(p_k,q)≤ D_KL(p_k\|q)/2 at each condition. Jensen’s inequality and the assumed mean KL bound yield Eq. (51). ∎ For a vocabulary distribution p and threshold τ>0τ>0, the pointwise-clipped objective in Eq. (11) equals the true KL whenever q(v)≥p(v)exp[−τ/p(v)]for every v with p(v)>0.q(v)≥ p(v) [-τ/p(v)] every v with p(v)>0. (52) The equality point q=pq=p lies inside this region, so the clipped and unclipped objectives have the same local gradient and Hessian around the correct fixed point. The clipped quantity is not a global divergence. For example, with p=(0.9,0.1)p=(0.9,0.1), q=(0.001,0.999)q=(0.001,0.999), and τ=0.05τ=0.05, the true KL is 5.8925.892, whereas the clipped sum is −0.180<0-0.180<0. No global KL claim in this paper uses the clipped objective. C.9 Adaptive-Sampling Identities Suppose independent teacher continuations succeed with probability s∈(0,1)s∈(0,1) and the maximum group size is G. A fixed group is all-failure with probability (1−s)G(1-s)^G, all-success with probability sGs^G, and mixed otherwise. If PAST begins with b samples and adds one sample after each all-failure prefix, the expected number of first-stage samples is [N1]=b+∑j=bG−1(1−s)j.E[N_1]=b+ _j=b^G-1(1-s)^j. (53) When a complete all-failure group triggers one independent size-G retry, the total expected sample count and final skip probability are [N]=b+∑j=bG−1(1−s)j+G(1−s)G,P(skip)=(1−s)2G.E[N]=b+ _j=b^G-1(1-s)^j+G(1-s)^G, P(skip)=(1-s)^2G. (54) These identities explain the sampling reduction but do not make the success-masked, retry-conditioned GRPO estimator unbiased for an unconditional population objective. Appendix D Full Experimental Protocol D.1 Training and Evaluation The main experiments use Qwen3-1.7B in thinking mode. Each method receives 100 logical student updates and is evaluated at checkpoints 0, 25, 50, 75, and 100. Training seeds are 17, 29, and 43. Evaluation covers AIME 2024, AIME 2025, and HMMT 2025 with temperature 1.0 and 12 generations per problem. Avg@12 is computed within each task, and the macro average weights the three tasks equally. The primary interval resamples paired observations within problem and training-seed strata. The verifier parses the boxed final answer and compares it with the authorized task answer. Truncated, malformed, or unparseable responses are verifier failures. The resulting signal evaluates final-answer correctness rather than intermediate reasoning. All main comparisons share the student-update count and evaluation protocol. Table 10 reports teacher sampling costs separately. D.2 Protocol and Semantic Checks Table 4 summarizes repeated checks performed before the main run, including the observed end-to-end parity difference alongside semantic, numerical, and recovery checks. Table 4: Protocol, numerical, and recovery checks, each repeated eight times. Check Observed mean Passed Repeats Model and thinking mode 1.000000 8 8 Training-data protocol 1.000000 8 8 Evaluation protocol 1.000000 8 8 Batch-loss absolute error 0.000000 8 8 Batch-gradient cosine 0.999989 8 8 End-to-end paired difference -0.102349 8 8 CPU semantic tests 1.000000 8 8 Resume and RNG recovery 1.000000 8 8 D.3 Full-Vocabulary Implementation Full-vocabulary KL is computed in PyTorch under the corresponding frozen adapter. Table 5 compares the training implementation with an FP32 reference. The main BF16 path has gradient cosine 0.9994570.999457 and no non-finite events. Sequence chunking and the fused kernel are implementation alternatives; they do not change the stated objective. Table 5: Numerical accuracy and system measurements for full-vocabulary KL implementations. Implementation Loss error Grad. cosine GB Tokens/s Non-finite FP32 reference 0.000000 0.999986 72.305 1854 0 BF16 full vocabulary 0.002978 0.999457 61.718 3306 0 Sequence chunking 0.002525 0.999558 49.967 2881 0 Fused kernel 0.003780 0.999196 55.156 3709 0 The system records token IDs, masks, response lengths, verifier outcomes, failure types, cycle identifiers, and sampled-action log probabilities. Full-vocabulary reference distributions are recomputed for each microbatch and are not persisted across phases. Checkpoint recovery restores both adapters, optimizers, schedulers, sampling-controller state, data position, and random-number-generator states at a complete cycle boundary. Appendix E Additional Results and Diagnostics E.1 Learning Curves Figure 7 reports changes from each method’s own checkpoint-0 evaluation. This normalization isolates within-run learning and is not used for the absolute comparison in Table 1. Figure 7: Avg@12 change from each method’s own checkpoint 0. Lines show the mean over three training seeds and shaded regions show the seed range. E.2 Correct and Failed Branches Joint training has the strongest point estimate in every task and improves the macro average by 2.5622.562 points over the stronger single branch (Table 6). Table 6: Correct-only, failed-only, and joint teacher adaptation. Training branch AIME24 AIME25 HMMT25 Macro Correct only 60.093 43.611 31.019 44.908 Failed only 62.037 43.333 32.222 45.864 Full PAST 63.519 45.833 35.926 48.426 Figure 8 shows that the correct-branch KL decreases from 0.19510.1951 to 0.07150.0715. Failed-branch success peaks at 0.47090.4709 and ends at 0.36520.3652, showing that success can fluctuate even while the preservation loss decreases; the clipping fraction and gradient norm remain finite throughout this trace. Figure 8: Teacher-training dynamics in E04. Correct-branch KL decreases, while failed-branch success and optimization diagnostics vary across cycles. E.3 Distribution Objectives The full-vocabulary reference and generalized JSD give similar one-seed quality, whereas sampled-token KL is cheaper but lower in this run. The failed-path regularization coefficient controls a visible success–compatibility trade-off: removing it raises teacher success but also produces the largest compatibility KL. These comparisons support the full-vocabulary objective as a strong quality–compatibility operating point. Table 7: One-seed objective and regularization ablations. Compatibility is measured by KL to the frozen student. Objective Macro Teacher success Compat. KL GB Tokens/s Full-vocabulary reference 47.562 0.7323 0.0961 61.741 3311 Sampled-token KL 45.864 0.7294 0.1220 43.804 4636 Generalized JSD 47.623 0.7169 0.0877 61.949 3175 KL + CE 44.815 0.7447 0.1326 60.909 3373 Top-k approximation 43.519 0.7223 0.1140 48.069 4199 βKL=0 _KL=0 43.395 0.7855 0.1722 61.390 3331 βKL=0.01 _KL=0.01 44.259 0.7480 0.1346 61.781 3293 βKL=0.1 _KL=0.1 43.735 0.6984 0.0744 62.005 3265 Failed-path clipping 43.981 0.6823 0.0828 61.637 3281 Correct clipping off 45.000 0.7517 0.1526 61.857 3359 E.4 Adaptive Sampling Adaptive sampling reduces teacher generation while reaching a slightly higher observed macro average. It uses 850 teacher samples and 743,366 generated tokens, compared with 1,032 samples and 862,342 tokens for fixed-size sampling. Table 8: Adaptive versus fixed-size teacher sampling. Sampler Macro Samples Tokens GPU-hours Adaptive 47.284 850 743,366 0.0392 Fixed-8 46.728 1,032 862,342 0.0457 E.5 Privilege and Optimization Diagnostics Figure 9 complements the condition comparison in Figure 4 with cycle-level optimization traces. In this E11 trace, correct-branch KL decreases from 0.18620.1862 to 0.07750.0775 and failed-branch success rises from 0.41020.4102 to 0.48360.4836. The skipped-group rate quantifies how often the finite-sample estimator lacks an active failed-branch update. Figure 9: Cycle-level correct-branch KL, failed-branch success, gradient norms, and skipped-group rate. E.6 One-Seed 4B Extension PAST has the highest macro point estimate in the one-seed Qwen3-4B extension. It improves AIME25 and HMMT25 over Vanilla OPSD, while Vanilla OPSD remains stronger on AIME24 (Table 9). This task-level variation motivates a broader multi-seed scale study. Table 9: One-seed Qwen3-4B extension. Method AIME24 AIME25 HMMT25 Macro Vanilla OPSD 66.111 45.000 31.944 47.685 Trajectory input only 64.444 42.222 34.444 47.037 PAST 62.778 48.611 34.167 48.519 Appendix F Cost and Reproducibility F.1 Training and Deployment Cost Table 10 reports training cost through generated teacher work, verifier calls, runtime, and peak memory, together with deployed parameter count. Table 10: Reported training-account measurements and deployment size. Teacher samples are additional continuations generated specifically for teacher adaptation. Method T. samples Verifier Tokens GPU-h Wall-h Peak GB Deploy B Vanilla OPSD 0 400 220,476 0.0090 0.0034 53.837 1.7 SD-Zero 0 13,456 7,548,980 0.3667 0.1400 60.701 1.7 H2SD 0 3,200 1,914,827 0.0971 0.0371 61.734 1.7 PAST 850 1,250 743,366 0.0392 0.0148 62.971 1.7 Within PAST, adaptive sampling reduces both samples and tokens relative to fixed-8 sampling, as reported in Table 8. Full-vocabulary KL increases peak memory relative to sampled-token approximations; sequence chunking provides a lower-memory implementation while preserving close numerical agreement with the reference (Table 5). F.2 Reproducibility Boundary Each complete cycle checkpoint stores the student and teacher LoRA adapters; their optimizer, scheduler, and mixed-precision states; the next cycle identifier; logical-step counters; adaptive-sampling state; data cursor; and Python, PyTorch, CUDA, and rollout-sampler random-number-generator states. It also records the base checkpoint revision, tokenizer, chat template, verifier, and full training configuration. Resume begins at the next rollout phase. Partial-cycle rollouts, logits, gradients, and verifier buffers are discarded rather than promoted to a completed state. Autoregressive student and teacher sampling use the corresponding named LoRA adapter in the rollout backend. Full-vocabulary distributions and all backward passes are computed in the training backend. Adapter version and cycle identifiers are checked when updated weights are synchronized. The CPU test backend implements the same phase and routing semantics without running large-model inference.