Paper deep dive
Beyond Timestamps: Decision-Aligned On-Policy Distillation for Long-Horizon Agents
Mingju Chen, Can Lv, Jinrong Liu, Huan Zhang, Heng Chang, Shiji Zhou
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Reinforcement learning with verifiable rewards (RLVR) often relies on sparse outcome rewards, providing coarse supervision for long-horizon agents. On-policy self-distillation (OPSD) complements this signal with dense privileged feedback. However, we identify \emph{Decision--Timestamp Mismatch}: privileged guidance may be misaligned with the student's functional decision because the corresponding decision can occur at a different timestep, while the student's decision itself may span multiple timesteps rather than being tied to a single timestamp. Thus, timestamp-local supervision can misalign both the context and the temporal scope of credit. To address this mismatch, we introduce \textsc{AlignOPSD}, following the principle of aligning supervision before assigning credit. Decision-Aligned Supervision Rectification re-scores the same student-sampled response in functionally matched contexts across sibling rollouts to calibrate local teacher evidence. Semi-Markov Hierarchical Credit Assignment then derives variable-duration decision spans from correspondence changes and uses rectified evidence to allocate outcome-grounded credit across spans and their constituent turns. We evaluate \textsc{AlignOPSD} with Qwen2.5-3B and Qwen2.5-7B on ALFWorld, WebShop, and Search-QA against representative baselines. \textsc{AlignOPSD} outperforms both GRPO and StepOPSD across all eight backbone--aggregate-metric comparisons, improving on GRPO by 5.5--8.7 \% and ranking first in six. Additional analyzes examine the two alignment stages and hyperparameter sensitivity between tasks. Our code is avaliable at this https URL
Tags
Links
- Source: https://arxiv.org/abs/2609.33391v1
- Canonical: https://arxiv.org/abs/2609.33391v1
Trouble viewing inline? Open PDF directly →
Full Text
83,212 characters extracted from source content.
Expand or collapse full text
Beyond Timestamps: Decision-Aligned On-Policy Distillation for Long-Horizon Agents BEYOND TIMESTAMPS:DECISION-ALIGNED ON- POLICY DISTILLATION FOR LONG-HORIZON AGENTS Mingju Chen 1,2∗ , Can Lv 1,2∗ , Jinrong Liu 2 , Huan Zhang 1,2 , Heng Chang 3 , Shiji Zhou 1,2 1 Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing, Beihang University 2 School of Artificial Intelligence, Beihang University 3 Tsinghua University ∗ Equal Contribution Project Lead: Heng Chang, Corresponding to: Shiji Zhou <zhoushiji25@buaa.edu.cn> ABSTRACT Reinforcement learning with verifiable rewards (RLVR) often relies on sparse out- come rewards, providing coarse supervision for long-horizon agents. On-policy self-distillation (OPSD) complements this signal with dense privileged feedback. However, we identify Decision–Timestamp Mismatch: privileged guidance may be misaligned with the student’s functional decision because the corresponding decision can occur at a different timestep, while the student’s decision itself may span multiple timesteps rather than being tied to a single timestamp. Thus, timestamp-local supervision can misalign both the context and the temporal scope of credit. To address this mismatch, we introduce ALIGNOPSD, following the principle of aligning supervision before assigning credit. Decision-Aligned Su- pervision Rectification re-scores the same student-sampled response in function- ally matched contexts across sibling rollouts to calibrate local teacher evidence. Semi-Markov Hierarchical Credit Assignment then derives variable-duration de- cision spans from correspondence changes and uses rectified evidence to allo- cate outcome-grounded credit across spans and their constituent turns. We eval- uate ALIGNOPSD with Qwen2.5-3B and Qwen2.5-7B on ALFWorld, WebShop, and Search-QA against representative baselines. ALIGNOPSD outperforms both GRPO and StepOPSD across all eight backbone–aggregate-metric comparisons, improving on GRPO by 5.5–8.7 % and ranking first in six. Additional analyzes examine the two alignment stages and hyperparameter sensitivity between tasks. Our code is avaliable at https://github.com/mingju-c/Align-OPSD. 1INTRODUCTION Agentic long-horizon tasks require language agents to accomplish goals through a sequence of inter- dependent intermediate decisions, while training rewards are often available only at the end of a task. Methods such as Group Relative Policy Optimization (GRPO) compare terminal rewards across sib- ling rollouts and broadcast a trajectory-level advantage to the generated tokens (Shao et al., 2024). Such supervision captures overall task performance, but cannot directly distinguish effective deci- sions from incidental behaviors within successful trajectories, nor can it precisely localize errors within failed ones (Feng et al., 2025; Zhang et al., 2026b). To complement this coarse-grained su- pervision, on-policy distillation (OPD) and its self-distilled variants (OPSD) provide dense teacher feedback on behaviors generated by the student policy itself (Agarwal et al., 2024; Zhao et al., 2026). When the teacher is further granted privileged task information, such feedback can provide richer local evidence about intermediate behaviors (Penaloza et al., 2026; Lu et al., 2026). The challenge is therefore not merely to obtain denser supervision, but to determine which teacher evidence can meaningfully inform the student’s intermediate decisions at different stages of execution. Toward more effective intermediate supervision, recent agentic OPSD methods have developed along two main directions. One line enriches teacher evidence through hindsight information, exe- cution feedback, or privileged contexts (Lu et al., 2026; Wang et al., 2026b; Yu et al., 2026; Yang et al., 2026b; Zhang et al., 2026a), while another refines how teacher–student discrepancies are fil- tered and aggregated into action- or turn-level learning signals (Zhang et al., 2026c; Li et al., 2026a). 1 arXiv:2609.33391v1 [cs.LG] 27 Sep 2026 Beyond Timestamps: Decision-Aligned On-Policy Distillation for Long-Horizon Agents Teacher Student ≠ Verify specs Compare Options Compare Options tᵢ tⱼ same timestamp different decisions same decision different timestamps Privilegedinformation Timestamps t₁ t₂ t₃ ⋯ tᵢ ⋯ tⱼ ⋯ 풕 푵 3 turns Gather evidence 5 turns Compare options 3 turns Verify & act tᵢ tⱼ alignment matrix cross-timestamp rectification Gather Compare Verify Inter-rollout mismatch Intra-rollout continuity Student Temporal displacement Distribution-aware spans One decision spans multiple turns 0306090120150 0 20 40 60 80 100 Training steps Displacement (%) 0 20 40 60 2345+ Fraction (%) Turns per span Decision-span lengths WebShop ALFWorld SearchQA ⋯ ⋯ AlignOPSD Profile change (JSD) Figure 1: Decision–Timestamp Mismatch. Functional decisions need not share timestamps across different sibling rollouts and can span multiple turns within individual trajectories. These advances improve the informativeness and utilization of supervision, yet their local signals remain anchored to corresponding positions along the student trajectory. Interpreting these local discrepancies as decision-level credit relies on a key assumption: teacher evidence at the current trajectory position provides an appropriate basis for evaluating the functional decision pursued by the student at that position. However, these refinements do not guarantee functional correspondence between teacher and student behaviors across rollouts, timestamps, and decision boundaries. We refer to this phenomenon as Decision–Timestamp Mismatch, which manifests at two related levels. Across rollouts, the same timestamp need not identify the same decision context, while corresponding decisions may occur at different turns. As illustrated in Figure 1, the student compares product options while the privileged teacher verifies specifications, the corresponding comparison appears later in the teacher rollout. Even under the same visible history, privileged information may change which decision the teacher favors. Consequently, although same-prefix probability comparisons remain valid, the resulting disagreement need not isolate the quality of the student’s current decision. The issue is therefore the functional comparability of supervisory contexts for reliable decision-level credit attribution, not the validity of same-prefix probability comparisons. Within a rollout, a functional decision may persist across multiple turns, with boundaries that do not coincide with individual timestamps. Assigning credit independently to each turn can fragment behaviors that jointly realize one decision, whereas fixed-length windows can mix distinct decisions. The upper-right panels illustrate how these two aspects can be characterized quantitatively through the temporal displacement of corresponding decisions and the distribution of decision-span lengths. Motivated by this, we study: How can functional correspondence guide both the selection of privileged evidence and the temporal scope of outcome-grounded credit assignment? To address this problem, we propose ALIGNOPSD, which aligns privileged supervision with func- tional decisions before assigning credit. Specifically, Decision-Aligned Supervision Rectification estimates functional correspondences across sibling rollouts, re-scores the same student-sampled re- sponse in matched privileged contexts, and combines cross-context and local evidence according to correspondence confidence. Building on this alignment, Semi-Markov Hierarchical Credit Assign- ment constructs variable-duration decision spans from correspondence changes, allocates outcome- grounded credit across spans and their constituent turns using rectified evidence, and broadcasts turn-level credit to the corresponding training tokens. We evaluate ALIGNOPSD on ALFWorld, WebShop, and Search-QA under matched training and evaluation protocols against representative agentic RL methods spanning embodied, search, and shopping environments. Our contributions are summarized as follows: • Problem Identification. We identify Decision–Timestamp Mismatch: timestamp-local privileged evidence may not correspond to the student’s functional decision, while the same decision can occur at positions across rollouts, leading to mismatched supervisory contexts and credit horizons. 2 Beyond Timestamps: Decision-Aligned On-Policy Distillation for Long-Horizon Agents Task prompt x Agent–environment loop π θ Env Sibling rollouts with Teacher views τ 1 ⋯ τ 2 ⋯ τ n ⋯ Student turn Teacher view z + v Outcome advantage A seq i = R i − R k σ R,k + ε Weighted advantage Teacher–student correspondence Cross-rollout alignment Top-K matched teacher contexts Teacher view z + v τ 1 ⋯ τ 2 ⋯ τ n ⋯ Student view z u v 1 v 2 v 3 v 4 ⋮ Teacher views v u 1 u 2 u 3 u 4 u 5 ⋯ high low Student turns u (across rollouts) m 1 (from τ 2 ) m 2 (from τ 3 ) 5. Rectified supervision Same student response a u (“yes, ...”) Identitystudent state Matched teacher state Confidence-guided mixing (α u from matching confidence) Correspondence profile (from H) Decision spans from profile shifts τ 1 ⋯ τ 2 ⋯ τ n ⋯ turn k S 1 S 2 S 3 time t Variable-length decision spans (semi-Markov) Hierarchical credit allocation (semi-Markov) Input 1: Rectified evidence δ u,r Input 2: Outcome advantage Span-level hierarchy Episode S 1 ⋯ S 2 ⋯ S 3 ⋯ Allocate outcome advantage across the hierarchy Within-span turn allocation ⋯ Distribute budget to turns and obtain weighted advantage Policy Optimization τ 1 ⋯ (with privileged infor.) Decision-Aligned Supervision Rectification Semi-Markov Hierarchical Credit Assignment ⋯ ⋯ ⋯ Thinking state 0 ⋯ Thinking state Encode(·) & Sim(·) m 2 (from τ 3 ) m k (from τ n ) ⋯ Thresholds Filter 푴=휳(푯;휸 푯 ,푲) 푯 풗,풖 =풔풊풎(풛 풗 + ,풛 풖 ) Re-score the same student response under matched teacher views ሚ 훿 푢,푟 =1−훼ℓ 푢→푢,푟 + +훼 푢 ℓ 푢,푟 +,푎푙푖푔푛 −푙 푢,푟 ℎ 푣 + →ℓ 푢,푟 +,푎푙푖푔푛 ℎ 푢 푙표푐 →ℓ 푢→푢,푟 + 푃 푘 =푁표푟푚(퐻 :,푘 ) ⋯ ⋯ 퐷 푘 푐표푟 =퐽푆퐷(푃 푘−1 ,푃 푘 ) S 4 푅 푖 =푅푒푤푎푟푑(휏 푖 ) 퐴 푖 푠푒푞 = 푅 푖 − ത 푅 푥 휎 푅,푥 +휀 መ 퐴 푖,푘,푟 =퐴 푖 푠푒푞 ·sg(W 푖,푘 ) መ 퐴 푖,푘,푟 =퐴 푖 푠푒푞 ·sg(W 푖,푘 ) 퓛 풑풐풍풊풄풚 휽=− ퟏ 푵 푩 풊∈푩,풌,풓∈푹 풊,풌 풎풊풏[흔 풊,풌,풓 휽 푨 풊,풌,풓 ,풄풍풊풑(흔 풊,풌,풓 휽,ퟏ−흐 풄풍풊풑 ,ퟏ+흐 풄풍풊풑 ) 푨 풊,풌,풓 ] Base regularizers unchanged & no auxiliary distillation loss Figure 2: Overview of ALIGNOPSD. Cross-rollout alignment establishes comparable supervisory contexts, while correspondence changes define decision spans for hierarchical credit assignment. The weights enter policy optimization, with auxiliary components used during training. • Proposed Solution. We propose ALIGNOPSD, which establishes functional correspondence be- fore credit assignment by rectifying privileged supervision with matched contexts and deriving adaptive decision spans for outcome-grounded credit allocation over variable-duration decisions. • Experimental Validation. Across three agentic benchmarks, ALIGNOPSD improves over GRPO by 5.5–8.7 % in all eight backbone–metric comparisons and ranks first in six. Additional analyses characterize correspondence alignment, span adaptation, and supervision rectification. 2METHODOLOGY 2.1PROBLEM FORMULATION Given task x and initial observation o 0 , trajectory i starts from h i,1 = (x,o 0 ). At turn k, a i,k = (y i,k,r ) L i,k r=1 ∼ π θ (·| h i,k ), h i,k+1 = h i,k ⊕ (a i,k ,o i,k ),(1) where a i,k may contain a thinking trace followed by an executable action, o i,k is the observation returned by the environment, and⊕ appends the response–observation pair to the interaction history. After K i turns, trajectory τ i receives a verifiable terminal reward R i = R(τ i ). For each task x, group-relative policy optimization samples G sibling trajectoriesG x and computes A seq i = R i − R x bσ R,x + ε ,R x = 1 G X j∈G x R j .(2) The base objective is broadcast A seq i to the optimized response tokens of the trajectory i, providing a scale and direction update based on the results. In OPSD-style training, each interaction history h i,k is paired with a privileged view h + i,k = (h i,k ,k x ), where k x is task-relevant privileged information obtained through relevance-based retrieval and available only during training. Both views share the same policy π θ . We denote the ordinary student view by π θ (· | h i,k ) and define the privileged teacher view as π + θ (· | h i,k )≜ π θ (· | h + i,k ). Thus, the teacher differs from the student only in its conditioning context, rather than its model parameters, as shown in Figure 2. 2.2DECISION-ALIGNED SUPERVISION RECTIFICATION For compact notation, let u = (i,k) denote a target turn and v = (j,l) a candidate source turn from another sibling rollout. Conventional OPSD evaluates a u only under its timestamp-local privileged 3 Beyond Timestamps: Decision-Aligned On-Policy Distillation for Long-Horizon Agents view h + u , which may be misaligned with the functional decision represented at u. We therefore establish cross-rollout decision correspondence before rectifying the local teacher evidence. Decision-state correspondence. Decision-relevant information may be distributed across observa- tions and interaction history, making direct state matching difficult. We therefore use thinking traces as semantic surrogates of the decision state. Let z u denote the thinking parsed from student response a u , and z + v the privileged thinking generated by the teacher view π + θ (· | h v ). A frozen semantic encoder Enc(·) maps both into a shared decision-state space: d u = Enc(z u ), d + v = Enc(z + v ).(3) We treat proximity in this space as approximate functional-decision correspondence. Appendix C.2 audits this surrogate with a method-faithful comparison of exact same-state and cross-rollout pairs. Decision-alignment operator. Given the student and teacher-side decision-state representations, we construct a source-by-target correspondence matrix H (n) v,u = cos d +,(n) v ,d (n) u , M (n) = Ψ H (n) ;γ H ,K .(4) For each target turn u, Ψ considers structurally valid source turns from other rollouts of the same task, removes candidates below similarity threshold γ H , retains at most one source per sibling roll- out, and selects the top-K matches. Across all three benchmarks, an action-consistency filter re- quires valid parsed actions and an action-embedding cosine of at least 0.8. On WebShop, where actions admit exact symbolic checks, we additionally require matching operation types and identi- cal click targets. Each nonzero columnM (n) ·,u assigns normalized weights to the retained reference views, while unmatched targets receive a zero column. The operator is recomputed for each on- policy batch, allowing the correspondence structure to evolve with the policy during training. Multi-view supervision rectification. Instead of corresponding to the identity view v = u, our alignment operator additionally provides off-diagonal reference views for the same target response. Rather than imitating retrieved responses, the privileged policy π + θ teacher-forces the same complete response a u under each aligned source history from a sibling rollout: ℓ + v→u,r = logπ + θ (y u,r | h v ,y u,<r ), ℓ u,r = logπ θ (y u,r | h u ,y u,<r ).(5) The identity view gives the conventional local gap δ id u,r = ℓ + u→u,r −ℓ u,r . For a matched target,M (n) ·,u aggregates the aligned privileged views and rectifies the local evidence as ℓ +,align u,r = log X v M (n) v,u exp ℓ + v→u,r , e δ u,r = (1− α u )ℓ + u→u,r + α u ℓ +,align u,r − ℓ u,r .(6) The coefficient α u ∈ [0,α max ] adaptively controls how strongly aligned views modify the local signal according to the quality of the match. Concretely, we compute b H v,u = clip H v,u − γ H 1− γ H , 0, 1 , ρ u = X v M (n) v,u b H v,u , α u = α max ρ u (7) If no valid source is retrieved, we set α u = 0, reducing e δ u,r to the conventional identity gap δ id u,r . The rectified gap e δ u,r is then passed to the hierarchical credit allocator in Section 2.3. 2.3SEMI-MARKOV HIERARCHICAL CREDIT ASSIGNMENT Rectification provides decision-relevant supervisory references, but a functional decision may span multiple interaction turns. Thus, we organize credit over variable-duration decision spans and intro- duce a hierarchical allocator Φ that converts rectified evidence into outcome-grounded advantages. Correspondence-driven spans. A continuing functional decision is expected to retain similar ref- erence contexts across adjacent turns. We normalize correspondence scores over structurally valid sources from other same-task rollouts before top-K truncation: P (n) i,k = Norm H (n) ·,(i,k) , D corr i,k = JSD P (n) i,k−1 ,P (n) i,k .(8) 4 Beyond Timestamps: Decision-Aligned On-Policy Distillation for Long-Horizon Agents where D corr i,k measures the correspondence shift between adjacent turns. A batch-adaptive threshold and duration constraints produce a contiguous partition S i = S i,1 ,...,S i,J i , whose variable- duration spans serve as credit units under a semi-Markov abstraction. Outcome-oriented evidence. The rectified gap provides token-level teacher support, while hierar- chical allocation operates at the turn level. We therefore define an outcome-oriented evidence score measuring agreement between rectified support and the trajectory-level outcome direction: E i,k = sgn(A seq i ) 1 N i,k N i,k X r=1 e δ i,k,r (9) where N i,k denote the number of optimized response tokens at turn k, a largerE i,k indicates stronger outcome-consistent teacher support and therefore stronger evidence for allocating credit to turn k. For span S i,m , we defineE span i,m as the mean evidence over all turns in the corresponding span. Hierarchical credit allocation. We view A seq i as an outcome-grounded credit budget and use the decision hierarchy to redistribute it. Let E i = (E i,k ) k denote the turn-level evidence, and define the cumulative credit measure Λ i,k = P t≤k N i,t , whose increments recover the uniform per-token allocation of the objective. We introduce a credit allocator Φ to produce turn-level credit weights: W i = Φ(S i ,E i ; Λ i ).(10) Internally, Φ performs two-level allocation. For each span S i,m , its aggregate evidence E span i,m de- termines a span-level budget B i,m . Conditioned on this budget, turn-level evidence determines the within-span allocation Q i,k|m : B i,m = Φ span E span i,m , ∆Λ i,m , W i,k = Norm[B i,m Φ turn (E i,k , ∆Λ i,k | S i,m )].(11) Thus, the outer allocation assigns span-level credit, while the inner allocation distributes it across constituent turns. After bounded normalization, turn weights reweight the trajectory advantage: b A i,k = A seq i sg(W i,k ),(12) where sg denotes stop-gradient. KL regularization constrains both allocation levels from deviating excessively from the neutral credit measure, while normalization preserves the token-weighted mean advantage. Exact allocation and normalization details are provided in Appendix E. Policy optimization. Each optimized token inherits its turn advantage, b A i,k,r = b A i,k . For the current trajectory batchB, let π θ old denote the fixed snapshot used for rollout collection and evidence construction. Define the policy ratio and total optimized-token count as ρ i,k,r (θ) = π θ (y i,k,r | h i,k ,y i,k,<r ) π θ old (y i,k,r | h i,k ,y i,k,<r ) , N B = X i∈B X k N i,k .(13) The token-mean clipped policy loss is L policy (θ) = − 1 N B X i∈B,k r∈R i,k min h ρ i,k,r (θ) b A i,k,r , clip(ρ i,k,r (θ), 1− ε clip , 1 + ε clip ) b A i,k,r i , (14) where ε clip is the base clipping threshold. Other base-objective regularization terms remain un- changed, and no auxiliary distillation loss is added. All correspondence, span, evidence, and alloca- tion quantities are stop-gradient and used only during training. 3EXPERIMENTS 3.1EXPERIMENTAL SETUP Benchmarks. We evaluate on ALFWorld (Shridhar et al., 2021), Search-QA (Jin et al., 2025), and WebShop (Yao et al., 2022), covering embodied household control, search-augmented question answering, and interactive online shopping, respectively. We report success rate for ALFWorld, 5 Beyond Timestamps: Decision-Aligned On-Policy Distillation for Long-Horizon Agents Table 1: Performance on ALFWorld, Search-QA, and WebShop. We report success rate (%) on ALFWorld, accuracy (%) on Search-QA, and Score/Acc (%) on WebShop. Skills are training-only unless marked with∗ (validation with skills).Best andsecond-bestare highlighted. ALFWorldSearch-QAWebShop MethodPickHeatLookCleanCoolPick2AvgNQTrivPopHotp2WkMuSBamAvgScoreAcc Qwen2.5-3B-Instruct Vanilla44.415.411.16.228.612.521.924.648.131.026.325.37.259.731.76.70.8 Skill-Prompt ∗ 51.70.066.748.44.310.028.923.746.230.624.422.17.512.523.91.20.8 OPSD48.80.041.716.715.816.728.10.10.10.10.00.00.00.00.011.33.1 GRPO91.261.962.596.265.047.475.039.360.641.137.434.615.426.436.479.863.3 Skill-GRPO88.970.671.458.840.729.260.243.558.843.036.832.211.712.534.177.360.9 Skill-GRPO ∗ 94.366.757.1 10073.157.180.544.359.644.339.036.114.514.936.176.366.4 GRPO+OPSD10075.082.485.770.060.081.244.360.846.038.938.613.964.543.977.866.4 Skill-SD88.252.450.096.265.057.973.442.860.142.239.538.913.664.944.075.964.0 RLSD87.975.075.090.973.168.479.742.358.643.339.539.616.166.144.284.466.4 SDAR94.650.066.768.260.080.074.244.858.144.338.636.215.766.143.470.757.8 StepOPSD82.452.266.782.6 73.775.073.443.661.243.839.238.115.864.543.782.466.4 ALIGNOPSD88.681.854.590.382.658.880.544.561.246.339.538.913.765.345.186.868.8 Qwen2.5-7B-Instruct Vanilla36.10.022.23.10.00.012.525.250.829.529.029.010.463.733.95.91.6 Skill-Prompt ∗ 51.75.350.032.34.30.023.430.952.132.732.727.912.766.136.41.70.8 OPSD50.021.460.022.717.69.532.88.88.617.52.54.20.51.26.24.52.3 GRPO91.281.0 87.596.265.057.981.245.163.744.043.643.216.837.642.080.972.6 Skill-GRPO88.561.166.765.257.773.169.545.263.745.743.143.319.621.440.380.471.9 Skill-GRPO ∗ 10083.383.396.475.078.988.444.863.045.143.743.720.571.447.587.081.2 GRPO+OPSD91.487.561.510076.552.280.447.364.546.943.839.318.069.447.086.876.5 Skill-SD93.910093.890.969.268.485.147.164.547.844.242.120.269.047.886.176.5 RLSD10058.887.592.380.065.282.046.863.044.445.548.921.573.049.087.477.3 SDAR94.786.775.010068.278.985.946.363.548.243.848.419.673.049.083.674.2 StepOPSD98.190.575.010080.063.288.445.364.645.144.544.419.369.848.287.278.1 ALIGNOPSD70.8 10081.810076.570.889.146.465.648.544.643.719.569.849.187.978.9 exact-match accuracy for Search-QA, and normalized score and exact success for WebShop. Dataset composition, splits, and per-category evaluation details are provided in Appendix D. Baselines. We compare three groups: training-free prompting methods (Vanilla and Skill-Prompt ∗ ), RLVR methods (GRPO, Skill-GRPO, and Skill-GRPO ∗ ), and self-distillation RL methods (OPSD, GRPO+OPSD, Skill-SD, RLSD, SDAR, and StepOPSD) (Shao et al., 2024; Zhao et al., 2026; Wang et al., 2026a; Yang et al., 2026a; Lu et al., 2026; Zhang et al., 2026c). All skill-conditioned methods use the same SkillBank and retrieval source. Detailed configurations are given in Appendix E.1. Implementation details. We use Qwen2.5-3B-Instruct and Qwen2.5-7B-Instruct backbones (Qwen et al., 2025). Training uses a single node with up to eight NVIDIA A800 GPUs. Rollout settings, hyperparameters, and training diagnostics are reported in Table 8 and Appendices E–F. 3.2MAIN RESULTS Aggregate performance.Table 1 compares ALIGNOPSD with trajectory-level RLVR and OPSD+RLVR baselines under the matched evaluation protocol.ALIGNOPSD achieves 80.5%/89.1% average success on ALFWorld, 45.1%/49.1% accuracy on Search-QA, and 86.8%/87.9% Score on WebShop for the 3B/7B backbones, respectively, under both model scales. Across the eight backbone–aggregate-metric comparisons, ALIGNOPSD ranks first in six and sec- ond in two, surpassed only by GRPO+OPSD on 3B ALFWorld (81.2%) and Skill-GRPO ∗ on 7B WebShop Acc (81.2%). Improvement over trajectory- and step-level credit. Compared with GRPO, ALIGNOPSD improves all eight aggregate comparisons, by 5.5%/7.9% on ALFWorld, 8.7%/7.1% on Search- QA, 7.0%/7.0% on WebShop Score, and 5.5%/6.3% on WebShop Acc for the 3B/7B backbones. ALIGNOPSD also outperforms StepOPSD across all eight comparisons, showing that the gains ex- tend beyond step-level supervision alone. Since skill-conditioned baselines share the same source, the results suggest that privileged information alone is insufficient. Behavioral alignment and outcome-conditioned credit assignment also matter. We next ablate the two alignment stages. 3.3ABLATION STUDY Effect of supervision rectification. Table 2 reports WebShop results with the 7B backbone. Remov- ing Decision-Aligned Supervision Rectification while retaining adaptive allocation reduces Score from 87.9% to 84.2% and Acc from 78.9% to 71.9%, corresponding to drops of 3.7 and 7.0%. 6 Beyond Timestamps: Decision-Aligned On-Policy Distillation for Long-Horizon Agents Table 2: Ablation Study. Final Score and Acc for ALIGNOPSD and four ablation variants are reported on the left and success-rate curves over 150 training steps are shown on the right. # VariantScore Acc – ALIGNOPSD87.9 78.9 1 w/o rectification + adaptive allocation84.2 71.9 2 w/ rectification + token allocation81.6 71.1 3 w/ rectification + turn allocation86.3 77.3 4 w/ rectification + random allocation78.9 69.5 150100150 Training step 0 20 40 60 80 100 Episode success rate (%) AlignOPSD Variant 1 Variant 2 Variant 3 Variant 4 050100150 Steps 0 20 40 60 80 100 Top- K Success rate (%) WebShop · Qwen2.5-3B K= 2 K= 3 K= 4 050100150 Steps 0 20 40 60 80 100 ALFWorld · Qwen2.5-3B K= 2 K= 3 K= 4 050100150 Steps 0 20 40 60 80 100 WebShop · Qwen2.5-7B K= 2 K= 3 K= 4 050100150 Steps 0 20 40 60 80 100 Similarity threshold Success rate (%) Threshold = 0.60 Threshold = 0.70 Threshold = 0.80 050100150 Steps 0 20 40 60 80 100 Threshold = 0.60 Threshold = 0.70 Threshold = 0.80 050100150 Steps 0 20 40 60 80 100 Threshold = 0.60 Threshold = 0.70 Threshold = 0.80 050100150 Steps 0 20 40 60 80 100 Credit temperature Success rate (%) T= 0.20 T= 0.50 T= 1.00 050100150 Steps 0 20 40 60 80 100 T= 0.20 T= 0.50 T= 1.00 050100150 Steps 0 20 40 60 80 100 T= 0.20 T= 0.50 T= 1.00 Figure 3: Sensitivity Analysis. Sensitivity to retrieval top-K, thinking-similarity threshold γ H , and credit temperature T across the tested hyperparameter ranges. Curves smooth recorded validation checkpoints, and shading shows twice the local temporal variation rather than across-seed uncer- tainty. This confirms that establishing a comparable supervisory context contributes beyond the allocation mechanism. The larger degradation in Acc indicates that rectification is particularly important for converting local teacher–student discrepancies into credit that supports complete task success. Effect of adaptive allocation. Keeping rectification fixed, we replace adaptive decision spans with token-, turn-, or random span-level allocation. Across both metrics, turn-level allocation outper- forms token-level allocation, which in turn outperforms random allocation. Relative to these alter- natives, ALIGNOPSD improves Score/Acc by 6.3/7.8% over token-level allocation, 1.6/1.6% over turn-level allocation, and 9.0/9.4% over random allocation. These results show that turns already provide meaningful units for credit assignment, whereas uniformly finer token-level credit does not yield better supervision. The additional gain over turn-level allocation suggests that decision bound- aries should adapt to correspondence changes during policy optimization rather than remain fixed. 7 Beyond Timestamps: Decision-Aligned On-Policy Distillation for Long-Horizon Agents (a) Cross-rollout correspondence discovery (b) Correspondence-guided credit allocation (c) Task-dependent span granularity (d) Rectification modifies token-level evidence H (correspondence matrix) 010203040 0 10 20 30 Target turns Source rollouts (from shifting sources) Source rollout A Source rollout B Source rollout C ★ ★ ★ ★ ★ ★ ★ αᵤ 01 Mixing weight ★ Target decision: “add to cart” Matched source: “select product” Rejected candidate: “view details” ★ Target decision: “check price” Matched source: “look up price” Rejected candidate: “go to homepage” Low High ★ Selected in M (sparse) Correspondence shift 퐷 푐표푟 =퐽푆퐷(푃ₖ₋₁,푃ₖ) 0.2 0.1 0.0 퐷 퐶표푟 threshold boundary S1S2S3S4 S5 Span segmentation Identity (local) Rectified Target turns (k) Share of spans ALFWorld 60% 40% 20% 0% 12345678 Training spans Annotation AlignOPSD: 4.41 Agent:4.74 Turns per span WebShop 60% 40% 20% 0% 12345678 Training spans Annotation AlignOPSD: 3.92 Agent:4.56 Turns per span Search-QA 60% 40% 20% 0% 12345678 Training spans Annotation AlignOPSD: 2.24 Agent: 1.69 Turns per span Before vs. after rectificationGap correctionmagnitude Identity gap δᵢ,ₙ (nats/token) Rectified gap δᵣ,ₙ ( nats /token) y = x Turns 30% 20% 10% 0% −0.3−0.2−0.10.0 0.0 −0.1 −0.2 −0.3 10⁻⁴10⁻³10⁻²10⁻¹ Correspondence Score Local vs. rectified evidence (teacher–student gap) Share of turns (%) Absolute change gap (nats/token) Figure 4: Mechanistic diagnostics. (a) Cross-rollout correspondence and confidence-weighted mixing; (b) correspondence-guided span segmentation and local versus rectified gaps; (c) task- dependent span-length distributions; (d) turn-averaged teacher–student gaps before and after rec- tification (left) and the distribution of absolute gap changes across target turns (right). 3.4SENSITIVITY ANALYSIS Figure 3 examines sensitivity to retrieval top-K, thinking-similarity threshold γ H , and credit tem- perature T , which control evidence coverage, correspondence filtering, and credit concentration, respectively. All panels report validation success rate from development histories, the curves char- acterize empirical operating regions rather than across-seed uncertainty. Retrieval top-K. The retrieval top-K controls retrieval breadth, balancing broader evidence cover- age against the inclusion of less relevant matches. We evaluate K ∈ 2, 3, 4. On ALFWorld 3B, the corresponding endpoints are 78.1%, 73.4%, and 76.6%, favoring K = 2, whereas WebShop 3B reaches 60.2%, 69.5%, and 57.8%, favoring K = 3. WebShop 7B instead favors K = 4, show- ing that no single value dominates across settings. Sensitivity also varies: the endpoint spread is only 4.7% on ALFWorld 3B and 4.6% on WebShop 7B, but increases to 11.7% on WebShop 3B. Thus, retrieval top-K exhibits a task- and scale-dependent coverage–selectivity trade-off rather than a monotonic trend across the tested benchmarks and backbone scales in practice. Thinking-similarity threshold γ H . The thinking-similarity threshold γ H filters weak correspon- dences before credit allocation, trading evidence retention against match quality. We evaluate γ H ∈ 0.60, 0.70, 0.80. WebShop 3B improves markedly from 51.6% to 69.5% as the thresh- old increases, while ALFWorld 3B peaks at γ H = 0.70 with 78.1%. WebShop 7B similarly favors a higher threshold, reaching 76.6% at γ H = 0.80. The endpoint spread is only 3.9% on ALFWorld 3B, compared with 17.9% on WebShop 3B and 9.4% on WebShop 7B. This indicates that correspon- dence filtering matters more for WebShop, while γ H = 0.70–0.80 provides an effective operating region whose exact preference remains task- and scale-dependent across the tested settings. Credit temperature T . The credit temperature T controls how sharply credit is distributed across decision units, balancing concentrated attribution against diffuse supervision. We evaluate T ∈ 0.2, 0.5, 1.0. Both 3B settings favor the intermediate value: ALFWorld follows 59.4%–78.1%– 62.5%, while WebShop follows 51.6%–69.5%–49.2%. WebShop 7B also peaks at T = 0.5 with 75.0%, but varies by only 3.1% across the tested range. In contrast, the spreads reach 18.7% on ALFWorld 3B and 20.3% on WebShop 3B, indicating greater temperature sensitivity for the smaller backbone. Unlike retrieval top-K and γ H , the preferred temperature is consistent across all settings despite differing sensitivity across tasks and scales, making T = 0.5 a stable default. 3.5MECHANISTIC ANALYSIS Figure 4 presents a mechanistic analysis of ALIGNOPSD, focusing on cross-rollout correspondence, task-dependent span formation, and supervision changes induced by rectification. 8 Beyond Timestamps: Decision-Aligned On-Policy Distillation for Long-Horizon Agents Cross-rollout correspondence is selective and semantically grounded. Figure 4(a) visualizes the correspondence matrix between source and target rollouts together with the sparse matches retained after weighting and selection. High-scoring correspondences need not occur at the same turn index: target decisions can instead retrieve source behavior that serves a similar functional role despite different trajectory positions. The examples further show that selected matches preserve decision semantics, while superficially nearby but functionally mismatched candidates are rejected. This supports the use of cross-rollout correspondence as a mechanism for establishing comparable super- visory contexts rather than relying on positional alignment alone. Correspondence shifts induce task-dependent credit spans. Figure 4(b) shows that changes in the correspondence profile are localized through D corr , with salient shifts defining span boundaries for subsequent credit allocation. The resulting granularity is not fixed across tasks. As shown in Figure 4(c), the average learned span contains 4.41 turns on ALFWorld, 3.92 on WebShop, and 2.24 on Search-QA. ALFWorld and WebShop therefore favor longer decision units, whereas Search- QA concentrates substantially more mass on short spans. This task-dependent structure supports adaptive segmentation over a globally fixed token- or turn-level unit: the appropriate credit horizon depends on how quickly functional correspondence changes along the trajectory. Rectification materially changes token-level supervision. Figure 4(d) compares the identity teacher–student gap with its rectified counterpart. The points systematically depart from the identity line, showing that cross-context rectification changes the token-level supervisory signal rather than simply preserving the original gap. The correction magnitudes also span a broad range, indicating that the effect is distributed across many turns instead of being driven by a few isolated cases. To- gether with the ablation results, this provides direct evidence that establishing a comparable scoring context changes the supervision subsequently used for credit assignment. 4RELATED WORK 4.1ON-POLICY SELF-DISTILLATION On-policy distillation moves supervision from fixed teacher traces to the learner’s own state distribu- tion, following the broader motivation of interactive imitation learning and generalized knowledge distillation (Ross et al., 2011; Agarwal et al., 2024). On-policy self-distillation (OPSD) uses ordinary and privileged conditioning views of the same policy to provide dense feedback on student-generated behavior without privileged inputs at deployment (Zhao et al., 2026; Penaloza et al., 2026). Recent extensions adapt this paradigm to agents through gated RL objectives, temporal curricula, hindsight skills, peer-rollout context, and trajectory-aware reliability estimation (Lu et al., 2026; Wang et al., 2026b; Yang et al., 2026b; Yu et al., 2026; Jiang & Ferraro, 2026; Kaur et al., 2026). These methods improve the source or reliability of privileged supervision, but generally score a target response in its own local history or use peer trajectories only as aggregate context. ALIGNOPSD instead retrieves functionally corresponding decisions across sibling rollouts and evaluates the student decision under matched privileged contexts, explicitly aligning supervision before it becomes credit. 4.2CREDIT ASSIGNMENT IN LONG-HORIZON RL Long-horizon agent RL must distribute sparse outcome supervision over many interdependent de- cisions, instantiating the classical problem of assigning delayed rewards to the state–action events that produced them (Arjona-Medina et al., 2019). GRPO provides a critic-free trajectory advan- tage but broadcasts it uniformly to all sampled tokens (Shao et al., 2024). Subsequent work obtains finer credit through repeated-state comparisons, hindsight value estimation, selective environmental feedback, or transition-wise rubric evaluation (Feng et al., 2025; Tan et al., 2026; Li et al., 2026b; Zhang et al., 2026b). Self-distillation-based methods instead use privileged teacher evidence to con- struct step-, turn-, or segment-level updates: StepOPSD adopts action-centered steps, while GEAR derives adaptive regions from changes in diagonal privileged divergence (Zhang et al., 2026c; Li et al., 2026a). Although these methods improve temporal resolution, they either assume a predefined credit unit or derive credit directly from evidence indexed by the current trajectory. ALIGNOPSD first resolves cross-rollout functional correspondence, then defines decision spans from changes in that correspondence and allocates a conserved outcome advantage across spans and turns; its central difference is therefore to align supervision before assigning fine-grained credit. 9 Beyond Timestamps: Decision-Aligned On-Policy Distillation for Long-Horizon Agents 5CONCLUSION We introduced ALIGNOPSD to address Decision–Timestamp Mismatch by aligning the context and temporal scope of privileged supervision. The method re-scores the same student responses under functionally matched contexts across sibling rollouts and derives variable-duration decision spans from correspondence changes for hierarchical credit assignment. Across three benchmarks and two Qwen backbones, ALIGNOPSD improves over GRPO by 5.5–8.7% in all eight aggregate compar- isons, ranking first in six. WebShop ablations support the contributions of both supervision recti- fication and adaptive spans, while mechanistic diagnostics characterize cross-timestamp matching, task-dependent span lengths, and changes in teacher–student gaps. These findings suggest a general principle for long-horizon agent training: supervision should be aligned with functional decisions before being used for outcome-grounded credit assignment. REFERENCES Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self- generated mistakes. In B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun (eds.), International Conference on Learning Representations, volume 2024, p. 21246–21263, 2024. URL https://proceedings.iclr.c/paper_files/paper/2024/file/ 5be69a584901a26c521c2b51e40a4c20-Paper-Conference.pdf. Jose A. Arjona-Medina, Michael Gillhofer, Michael Widrich, Thomas Unterthiner, Johannes Brandstetter, and Sepp Hochreiter. Rudder: Return decomposition for delayed rewards. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alch ́ e-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.URL https://proceedings.neurips.c/paper_files/paper/2019/ file/16105fb9c614fc29e1bda00dab60d41-Paper.pdf. Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An.Group-in-group policy optimiza- tion for llm agent training.In D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen (eds.), Advances in Neural Information Processing Systems, vol- ume 38, Main Conference, p. 46375–46408. Curran Associates, Inc., 2025. doi: 10.52202/ 085713-1544.URL https://proceedings.neurips.c/paper_files/paper/ 2025/file/420c9f777c0b4f78d515e53cf74d58b2-Paper-Conference.pdf. Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi- hop QA dataset for comprehensive evaluation of reasoning steps. In Donia Scott, Nuria Bel, and Chengqing Zong (eds.), Proceedings of the 28th International Conference on Computational Linguistics, p. 6609–6625, Barcelona, Spain (Online), December 2020. International Com- mittee on Computational Linguistics. doi: 10.18653/v1/2020.coling-main.580. URL https: //aclanthology.org/2020.coling-main.580/. Yuxuan Jiang and Francis Ferraro. Bridging reasoning trajectories in on-policy distillation via near- future guidance, 2026. URL https://arxiv.org/abs/2606.00305. Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-r1: Training llms to reason and leverage search engines with reinforcement learning, 2025. URL https://arxiv.org/abs/2503.09516. Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzilay and Min-Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 1601–1611, Vancouver, Canada, July 2017. Association for Com- putational Linguistics. doi: 10.18653/v1/P17-1147. URL https://aclanthology.org/ P17-1147/. Simran Kaur, Narutatsu Ri, Yinghui He, Liam Fowl, and Sanjeev Arora. Rethinking on-policy self- distillation for thinking models, 2026. URL https://arxiv.org/abs/2607.05184. 10 Beyond Timestamps: Decision-Aligned On-Policy Distillation for Long-Horizon Agents Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466, 08 2019. ISSN 2307-387X. doi: 10.1162/ tacl a00276. URL https://doi.org/10.1162/tacl_a_00276. Sijia Li, Yuchen Huang, Zifan Liu, Yanping Li, Jingjing Fu, Li Zhao, Jiang Bian, Ling Zhang, Jun Zhang, and Rui Wang. Gear: Granularity-adaptive advantage reweighting for llm agents via self-distillation, 2026a. URL https://arxiv.org/abs/2605.11853. Xiaozhe Li, Tianyi Lyu, Yang Li, Yichuan Ma, Peiji Li, Linyang Li, Qipeng Guo, Dahua Lin, and Kai Chen. What and when to distill: Selective hindsight distillation for multi-turn agents, 2026b. URL https://arxiv.org/abs/2605.19447. Zhengxi Lu, Zhiyuan Yao, Zhuowen Han, Zi-Han Wang, Jinyang Wu, Qi Gu, Xunliang Cai, Weim- ing Lu, Jun Xiao, Yueting Zhuang, and Yongliang Shen. Self-distilled agentic reinforcement learning, 2026. URL https://arxiv.org/abs/2605.15155. Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 9802–9822, Toronto, Canada, July 2023. Association for Computational Linguis- tics. doi: 10.18653/v1/2023.acl-long.546. URL https://aclanthology.org/2023. acl-long.546/. Emiliano Penaloza, Dheeraj Vattikonda, Nicolas Gontier, Alexandre Lacoste, Laurent Charlin, and Massimo Caccia. Privileged information distillation for language models, 2026. URL https: //arxiv.org/abs/2602.04942. Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah Smith, and Mike Lewis. Mea- suring and narrowing the compositionality gap in language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, p. 5687–5711, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-emnlp.378. URL https://aclanthology.org/2023. findings-emnlp.378/. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL https://arxiv.org/abs/2412.15115. Stephane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and struc- tured prediction to no-regret online learning. In Geoffrey Gordon, David Dunson, and Miroslav Dud ́ ık (eds.), Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, volume 15 of Proceedings of Machine Learning Research, p. 627–635, Fort Laud- erdale, FL, USA, 11–13 Apr 2011. PMLR. URL https://proceedings.mlr.press/ v15/ross11a.html. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathe- matical reasoning in open language models, 2024. URL https://arxiv.org/abs/2402. 03300. Mohit Shridhar, Xingdi Yuan, Marc-Alexandre C ˆ ot ́ e, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning, 2021. URL https://arxiv.org/abs/2010.03768. 11 Beyond Timestamps: Decision-Aligned On-Policy Distillation for Long-Horizon Agents Hui-Ze Tan, Xiao-Wen Yang, Hao Chen, Jie-Jing Shao, Yi Wen, Yuteng Shen, Weihong Luo, Xiku Du, Lan-Zhe Guo, and Yu-Feng Li. Hindsight credit assignment for long-horizon llm agents, 2026. URL https://arxiv.org/abs/2603.08754. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique: Multi- hop questions via single-hop question composition, 2022. URL https://arxiv.org/abs/ 2108.00573. Hao Wang, Guozhi Wang, Han Xiao, Yufeng Zhou, Yue Pan, Jichao Wang, Ke Xu, Yafei Wen, Xiaohu Ruan, Xiaoxin Chen, and Honggang Qi. Skill-sd: Skill-conditioned self-distillation for multi-turn llm agents, 2026a. URL https://arxiv.org/abs/2604.10674. Jiaqi Wang, Wenhao Zhang, Weijie Shi, Yaliang Li, and James Cheng. Tcod: Exploring temporal curriculum in on-policy distillation for multi-turn autonomous agents, 2026b. URL https: //arxiv.org/abs/2604.24005. Chenxu Yang, Chuanyu Qin, Qingyi Si, Minghui Chen, Naibin Gu, Dingyu Yao, Zheng Lin, Weip- ing Wang, Jiaqi Wang, and Nan Duan. Self-distilled rlvr, 2026a. URL https://arxiv.org/ abs/2604.03128. Shuo Yang, Jinyang Wu, Zhengxi Lu, Yuhao Shen, Fan Zhang, Lang Feng, Shuai Zhang, Haoran Luo, Zheng Lian, Zhengqi Wen, and Jianhua Tao. Opid: On-policy skill distillation for agentic reinforcement learning, 2026b. URL https://arxiv.org/abs/2606.26790. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii (eds.), Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing, p. 2369– 2380, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1259. URL https://aclanthology.org/D18-1259/. Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan.Webshop: Towards scal- able real-world web interaction with grounded language agents. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Pro- cessing Systems, volume 35, p. 20744–20757. Curran Associates, Inc., 2022. doi: 10.52202/ 068431-1508.URL https://proceedings.neurips.c/paper_files/paper/ 2022/file/82ad13ec01f9fe44c01cb91814fd7b8c-Paper-Conference.pdf. Weichen Yu, Xiaomin Li, Yizhou Zhao, Xiaoze Liu, Ruowang Zhang, Haixin Wang, Yinyi Luo, Chen Henry Wu, Gaurav Mittal, Matt Fredrikson, and Yu Hu. Multi-rollout on-policy distillation via peer successes and failures, 2026. URL https://arxiv.org/abs/2605.12652. Guibin Zhang, Jiayang Lyu, Ran Sun, Xinlei Yu, Haoyu Zhao, Qibing Ren, and Shuicheng Yan. Latent on-policy self-distillation, 2026a. URL https://arxiv.org/abs/2608.13040. Huan Zhang, Mingju Chen, Dongxu Zhou, Can Lv, Heng Chang, Sen Cui, Faguo Wu, and Shiji Zhou. Trca: Transition-wise rubric credit assignment for long-horizon llm agents, 2026b. URL https://arxiv.org/abs/2608.16156. Yanfei Zhang, Xu Lin, and Chenglin Wu. Stepopsd: Step-aware online preference self-distillation for agent reinforcement learning, 2026c. URL https://arxiv.org/abs/2605.27140. Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self-distilled reasoner: On-policy self-distillation for large language models, 2026. URL https: //arxiv.org/abs/2601.18734. 12 Beyond Timestamps: Decision-Aligned On-Policy Distillation for Long-Horizon Agents ANOTATION Table 3 summarizes the main symbols in Section 2. Table 3: Notation used in ALIGNOPSD. SymbolMeaning Interaction and outcome supervision x, o 0 , k x Task, initial observation, and training-only privileged information. i,j; k,l; r; nTrajectory, turn, token, and on-policy batch indices. u = (i,k), v = (j,l) Target turn and source turn from another same-task rollout. τ i , K i Trajectory and its number of turns. h i,k , h + i,k Ordinary history and privileged history (h i,k ,k x ). a i,k , o i,k , ⊕Generated response, environment observation, and concatenation. y i,k,r , y i,k,<r Response token and its preceding response tokens. L i,k , R i,k , N i,k Response length, optimized token positions, and their count. G x , G, BSibling rollout group, group size, and trajectory batch. R i ,R x ,bσ R,x Terminal reward, group mean, and group standard deviation. A seq i , εGroup-normalized trajectory advantage and numerical stabilizer. π θ Student policy conditioned on ordinary histories. π + θ Privileged conditioning view of the same policy. θ old Fixed policy snapshot used for the current batch. Decision correspondence and supervision rectification z u , z + v Student and privileged thinking traces. Enc, d u , d + v Frozen encoder and student/privileged decision representations. H (n) v,u Cosine similarity between source and target decision representations. ΨOperator selecting and weighting valid cross-rollout matches. γ H , KSimilarity threshold and maximum matches per target. M (n) v,u Sparse match weight; each column sums to one or is all zero. b H v,u Threshold-rescaled similarity, clipped to [0, 1]. ρ u Match-weighted correspondence confidence. α u , α max Rectification coefficient and its maximum; α u = α max ρ u . ℓ u,r Target-token log-probability under its ordinary history. ℓ + v→u,r Same target-token log-probability under a privileged source history. ℓ +,align u,r Log-probability of the mixture of matched privileged views. δ id u,r Local teacher–student log-probability gap. e δ u,r Confidence-weighted rectification of the local gap. Decision spans, credit allocation, and policy optimization P (n) i,k Dense correspondence profile before top-K truncation. D corr i,k , JSDAdjacent-turn correspondence shift and Jensen–Shannon divergence. S i , S i,m , J i Trajectory partition, decision span, and number of spans. E i,k , E i Outcome-oriented turn evidence and its trajectory-level vector. E span i,m Mean turn evidence within decision span S i,m . Λ i,k , ∆Λ i,k , ∆Λ i,m Cumulative token count, turn token mass, and span token mass. Φ, Φ span , Φ turn Hierarchical allocator and its span-level and turn-level components. B i,m Credit budget assigned to decision span S i,m . Q i,k|m Conditional credit allocation to turn k within span S i,m . W i,k , W i Turn credit weight and its vector; token-weighted mean is one. Norm, sgn, sgNormalization, sign function, and stop-gradient. b A i,k , b A i,k,r Turn advantage and its broadcast token advantage. ρ i,k,r (θ), ε clip Current-to-snapshot token probability ratio and clipping threshold. N B , L policy Batch optimized-token count and token-mean clipped policy loss. 13 Beyond Timestamps: Decision-Aligned On-Policy Distillation for Long-Horizon Agents BALGORITHM Algorithm 1 ALIGNOPSD: decision-aligned on-policy self-distillation Require: Initial policy π θ ; task sampler and environments; privileged-information retriever; frozen encoder Enc; group size G; alignment, segmentation, allocation, and optimization settings Ensure: Trained student policy π θ 1: for each on-policy batch n do 2:Freeze snapshot θ old ← θ 3:Sample a batch of tasks 4:Construct all evidence and allocation quantities without gradients using θ old 5:Collect G sibling rollouts per task with π θ old ; formB 6:Record histories, responses, rewards, masksR i,k , and old token log-probabilities 7:Compute A seq i ← (R i − R x )/(bσ R,x + ε) within each task group 8:Retrieve k x ; construct privileged histories h + v = (h v ,k x ) 9:Parse student thinking z u from each sampled response a u 10:Generate privileged thinking z + v with π + θ old 11:Encode d u ← Enc(z u ) and d + v ← Enc(z + v ) 12:Compute H (n) v,u ← cos(d + v , d u ) within each task group 13: M (n) ← Ψ(H (n) ;γ H ,K)▷ Valid sources; at most one per sibling 14:for each target turn u do 15:Teacher-force the complete a u under h u and h + u to obtain ℓ u,r and ℓ + u→u,r 16:Initialize α u ← 0 and e δ u,r ← ℓ + u→u,r − ℓ u,r 17:ifM (n) ·,u is nonzero then 18:Teacher-force the same a u under each retained h + v to obtain ℓ + v→u,r 19:ρ u ← P v M (n) v,u clip((H (n) v,u − γ H )/(1− γ H ), 0, 1) 20:α u ← α max ρ u 21:ℓ +,align u,r ← log P v M (n) v,u exp(ℓ + v→u,r ) 22: e δ u,r ← (1− α u )ℓ + u→u,r + α u ℓ +,align u,r − ℓ u,r 23:end if 24:end for 25:for each trajectory i∈B do 26:Form dense profiles P (n) i,k from valid source scores before top-K truncation 27:Compute D corr i,k ← JSD(P (n) i,k−1 , P (n) i,k ) for k ≥ 2 28:Partition intoS i using the batch-adaptive threshold and duration constraints 29: E i,k ← sgn(A seq i )N −1 i,k P r∈R i,k e δ i,k,r 30: E span i,m ←|S i,m | −1 P k∈S i,m E i,k 31:Compute token masses using Λ i,k ← P t≤k N i,t 32:Allocate B i,m with Φ span and Q i,k|m with Φ turn , both KL-regularized against the neutral token measure 33:Apply bounded normalization to B i,m Q i,k|m to obtain W i = Φ(S i , E i ; Λ i ) 34:Enforce P k N i,k W i,k = P k N i,k 35: b A i,k,r ← A seq i sg(W i,k ) for r ∈R i,k 36:end for 37:Update θ using L policy and the base-objective regularizers, keeping the snapshot and all constructed advantages fixed 38: end for 39: return π θ using ordinary histories at inference 14 Beyond Timestamps: Decision-Aligned On-Policy Distillation for Long-Horizon Agents CEMPIRICAL ANALYSIS We organize the analysis around three questions: • Q1: Temporal alignment. Do corresponding decisions share timestamps? • Q2: Thinking alignment. Does thinking similarity identify compatible decision contexts? • Q3: Decision spans. How many turns does a functional decision span? These diagnostics complement the controlled evaluations of supervision quality and learning benefit. C.1TEMPORAL MISALIGNMENT We test two assumptions behind matching by turn index: whether the same turn contains the same type of operation, and whether corresponding actions occur at the same turn. Trajectory comparisons. We use Qwen2.5-3B-Instruct rollouts for 100 WebShop tasks, with four student and four privileged-view rollouts per task under a 15-turn limit. Both branches interact in- dependently, giving 600 student–student (S–S) and 1,600 student–privileged (S–P) trajectory pairs. We identify operations from parsed actions and observations, distinguishing search, product selec- tion, option selection, tab inspection, purchase, and navigation. For example, selecting a product and inspecting its attributes are treated as different operations despite both using click. Equal turn indices often pair different operations. Among positions where both trajectories are active and both actions are parseable, action types differ at 58.90% of S–S positions and 55.06% of S–P positions (Table 4 and Figure 5(a)). The S–S result shows that this mismatch already arises among ordinary student rollouts and does not require privileged conditioning. Note that this is an action-type comparison: identical action types may still serve different functional decisions. Corresponding actions also occur several turns apart. We match actions using observed enti- ties without thinking embeddings or equal turn indices. An order-preserving maximum-cardinality matcher identifies strict correspondences: product selections require the same product ID, while other operations require the same product and, where applicable, the same option or tab. Among these correspondences, 79.72% (S–S) and 81.31% (S–P) occur at different turns, with a median dis- placement of four turns in both groups. Thus, even matching functional operations often requires crossing multiple timestamps. The strict matching criterion yields conservative coverage, and we use these pairs only to measure temporal displacement among unambiguous correspondences. Table 4: WebShop trajectory comparisons. Same-turn rates use 7,867 S–S and 21,265 S–P positions with two parseable actions; brackets give 95% task-bootstrap intervals (5,000 draws). Entity-action coverage is the number of matched pairs divided by valid left-turn exposures across trajectory pairs. StatisticS–S–P Different action types at the same turn (%)58.90 [56.76, 61.13]55.06 [52.95, 57.19] Matched entity-action pairs143289 Entity-action coverage (%)1.711.29 Matched actions at different turns (%)79.7281.31 Absolute displacement, median (turns)44 Training-time cross-turn usage. We further examine the first 100 updates of Qwen2.5-3B-Instruct training runs on all three benchmarks. For each target with a retrieved source, we measure the fraction of matching weight assigned to sources at different turn indices. Sources are sibling stu- dent histories paired with privileged teacher thinking, and this statistic characterizes temporal usage rather than semantic correctness. Across recorded updates, off-turn sources account for 96.70%, 83.63%, and 22.71% of matching weights on ALFWorld, WebShop, and Search-QA, respectively. The lower ratio on Search-QA is expected because its trajectories contain at most four turns, provid- ing fewer opportunities for cross-turn matching compared with ALFWorld and WebShop (50 and 15 turns). Overall, the results show that ALIGNOPSD consistently leverages off-turn sources when longer interaction horizons provide sufficient temporal flexibility. 15 Beyond Timestamps: Decision-Aligned On-Policy Distillation for Long-Horizon Agents 151015 Turn index 0 20 40 60 80 100 Different action types (%) S-S-P (a) Same-turn disagreement 151015 Student turn index 0.0 2.5 5.0 7.5 10.0 12.5 Absolute turn gap S-S-P (b) Cross-turn displacement 1255075100 Training update 0 20 40 60 80 100 Off-turn matching weight (%) ALFWorld WebShop Search-QA (c) Training dynamics Figure 5: Temporal misalignment in trajectories and training. (a–b) Dots show per-turn estimates; faint dots in (b) show individual matches. Lines are local smooths with 95% task-bootstrap bands. (c) The first 100 updates, with bands of one local standard deviation. C.2THINKING-BASED CORRESPONDENCE We analyze the thinking-similarity signal used for cross-rollout correspondence in ALIGNOPSD. The goal is not to evaluate semantic correctness of individual matches, but to examine whether the signal identifies compatible decision contexts beyond timestamp alignment. Specifically, we ask whether thinking traces from the same environment state provide a stronger reference than arbitrary cross-rollout pairs, and whether the resulting correspondences remain effective when applied. Measurement. For the first 100 updates of the Qwen2.5-3B-Instruct WebShop run, we collect 139,255 canonical turns. At each turn, the student and privileged Teacher generate independent thinking traces conditioned on the same task, history, and current page. We encode traces using the frozen Qwen3-Embedding-0.6B encoder adopted by the training pipeline and compute cosine simi- larity. We compare three correspondence sets: (i) same-state pairs, where student and Teacher share the same state; (i) arbitrary same-task cross-rollout pairs; and (i) cross-rollout pairs retained by the training operator after thinking thresholding, top-K selection, rollout-level filtering, and action- consistency filtering. Table 5 summarizes the similarity distributions. Table 5: Training-time thinking correspondence on WebShop over updates 1–100. Means and stan- dard deviations pool recorded pairs; brackets are 95% update-bootstrap intervals for the mean. The diagonal count denotes canonical state exposures. Pair setCountCosine mean [95% CI]Std. Same state (student–Teacher diagonal)139,2550.7604 [0.7580, 0.7630]0.0977 All same-task, cross-rollout pairs11,153,3910.7214 [0.7192, 0.7237]0.1014 Cross-rollout pairs above 0.82,564,6160.8430 [0.8426, 0.8434]0.0321 Cross-rollout pairs used by training110,7930.8805 [0.8791, 0.8820]0.0418 Correspondence analysis. Thinking traces from same environment states provide a reference than arbitrary same-task cross-rollout pairs, with cosine similarity increasing from 0.7214 to 0.7604. Af- ter correspondence filtering, the retained training pairs achieve a mean similarity of 0.8805. This increase partly reflects the selection criterion itself and should therefore be interpreted as evidence that the operator identifies high-scoring candidates rather than as a measure of semantic precision. The matcher retrieves at least one source for 62,939 of 139,255 targets (45.20%), with 0.80 sources per target overall and 1.76 conditional on a match. Among selected correspondences, 66.86% con- nect different turn indices, with median displacement 2 turns (mean 3.11). Thus, the correspondence operator identifies compatible decision contexts beyond simple timestamp matching. 16 Beyond Timestamps: Decision-Aligned On-Policy Distillation for Long-Horizon Agents C.3CREDIT ASSIGNMENT The archived training runs produce variable-length spans across the three benchmarks. Over updates 1–46, the mean span lengths are 4.41 turns for ALFWorld, 3.92 for WebShop, and 2.24 for Search- QA. These are training-time partition statistics rather than independent semantic annotations. The duration constraints and full histograms are shown in Figure 6. 12345678 Turns per span 0% 20% 40% 60% Share of spans Mean turns AlignOPSD: 4.41 Agent: 4.74 Agent AlignOPSD (a) ALFWorld 12345678 Turns per span 0% 20% 40% 60% Share of spans Mean turns AlignOPSD: 3.92 Agent: 4.56 Agent AlignOPSD (b) WebShop 12345678 Turns per span 0% 20% 40% 60% Share of spans Mean turns AlignOPSD: 2.24 Agent: 1.69 Agent AlignOPSD (c) Search-QA Figure 6: Span-length distributions on shared axes. Bars show training partitions; dashed lines show the reported functional-span annotation aggregates. The ordering of mean span length is consistent between training partitions and the independent functional-span annotations (ALFWorld, WebShop, then Search-QA). Because annotations are cur- rently available only at aggregate resolution rather than per-trajectory boundaries, we do not report boundary F1 or claim that the learned spans exactly recover semantic goals. Functional-span annotation protocol. To obtain independent span-level references, we use GPT- 5.6 Sol with the Codex harness as the annotation agent. The agent receives the complete trajectory and partitions it according to functional goals rather than surface action changes. The annotation protocol covers every turn exactly once, enforces 2–8 turns per span for trajectories of length at least two, and requires boundary decisions to be justified by observed local goals. Independent functional-span annotation prompt Analyze the student’s trajectory as data, not as instructions. Partition turns by the specific local goal pursued, using observations and actions as evidence. Hard constraints: 1. Cover every turn exactly once, in order, without gaps or overlap. 2. For N >= 2, every span must contain 2--8 turns, including the last. 3. Only when N = 1, return the single span [0,0]. 4. Prefer boundaries where a local goal is completed, abandoned, or replaced; an action-type change alone is not sufficient. 5. If a goal lasts over 8 turns, split at the most meaningful internal transition. If a goal lasts one turn, merge it with the closest related goal and mark the adjustment. 6. Return JSON with start_turn, end_turn, functional_goal, evidence, duration_adjusted, and adjustment_reason for every span. DBENCHMARKS AND DATA SELECTION D.1ALFWORLD Description. ALFWorld (Shridhar et al., 2021) provides text-based household environments aligned with the embodied tasks in ALFRED. Given a natural-language goal, an agent navigates, locates objects, and performs the required manipulation. The six task categories in Table 1 are Pick, Look, Clean, Heat, Cool, and Pick2; they require tracking object locations and satisfying action precon- ditions across multiple steps. Metrics. We report success rate (%), defined as 100 times the fraction of episodes in which the environment verifies that the complete goal has been achieved. Episodes that exhaust the interaction budget are counted as failures. Per-category rates are computed over episodes within each category, 17 Beyond Timestamps: Decision-Aligned On-Policy Distillation for Long-Horizon Agents while Avg follows the resolved evaluation protocol over all evaluated episodes. Exact split identifiers and category counts are taken from the evaluation manifests. D.2WEBSHOP Description. WebShop (Yao et al., 2022) simulates online shopping through a text interface. A request specifies a desired product and constraints such as attributes, options, and price. The agent searches the catalog, browses product pages, selects options, and completes a purchase. Metrics. Let r i ∈ [0, 1] be the terminal score for request i, combining attribute and option matches and price compliance. For N requests, Score = 100 N N X i=1 r i ,Acc = 100 N N X i=1 1[r i = 1].(15) Score credits partial constraint satisfaction, whereas Acc measures fully successful purchases. Re- quest IDs and the sample count are taken from the matched evaluation manifest. D.3SEARCH-QA Description. Search-QA follows the search-augmented question-answering setting of Search-R1 (Jin et al., 2025). The agent alternates reasoning with retrieval and returns a short answer based on the collected evidence. The evaluation suite contains seven datasets: NQ(Kwiatkowski et al., 2019), TriviaQA(Joshi et al., 2017), PopQA(Mallen et al., 2023), HotpotQA(Yang et al., 2018), 2WikiMultiHopQA(Ho et al., 2020), MuSiQue(Trivedi et al., 2022), and Bamboogle(Press et al., 2023). NQ and HotpotQA supply the training domains; the other five are out-of-domain evaluation datasets. Metrics.We report exact-match accuracy (%).The predicted answer is extracted from <answer>...</answer> and compared with accepted references following the standard exact- match normalization. A question receives score 1 if the normalized prediction matches any refer- ence and 0 otherwise; missing answers receive 0. Dataset-level accuracy is computed over ques- tions within each dataset, and Avg follows the official Search-QA evaluation protocol over all seven datasets. The interaction limits are given in Table 8; the Search-QA manifest fixes question counts, retrieval corpus, and index. D.4PROMPTS The following templates specify the task prompts, interaction history, and action formats used in our experiments. Braced fields are filled with the current task, observation, and available in- teraction history. For ALIGNOPSD, skill context is empty in the student view and at evaluation, and contains the retrieved training-only guidance in the privileged teacher view. At the first turn, the history clause is omitted. ALFWorld and WebShop require reasoning inside <think>...</think> followed by one admissible action inside <action>...</action>. Search-QA instead requires exactly one search query or one final answer per turn; retrieved results appear in <information>...</information>. The templates keep the observation and in- teraction history explicit so that each response is conditioned on the same information available to the student policy. The teacher-only field is inserted as an additional context block during train- ing; it is never included in rollout generation or evaluation. This separation allows all methods to share the same action space and environment interface while isolating the effect of the training-time supervision signal. 18 Beyond Timestamps: Decision-Aligned On-Policy Distillation for Long-Horizon Agents ALFWorld prompt You are an expert agent operating in the ALFRED Embodied Environment. Your task is to: task_description skill_context Prior to this step, you have already taken step_count step(s). Below are the most recent history_length observations and the corresponding actions you took: action_history You are now at step current_step and your current observation is: current_observation Your admissible actions of the current situation are: [admissible_actions]. Now it’s your turn to take an action. You should first reason step-by-step about the current situation. This reasoning process MUST be enclosed within <think> </think> tags. Once you’ve finished your reasoning, you should choose an admissible action for current step and present it within <action> </action> tags. WebShop prompt You are an expert autonomous agent operating in the WebShop e-commerce environment. skill_context Your task is to: task_description. Prior to this step, you have already taken step_count step(s). Below are the most recent history_length observations and the corresponding actions you took: action_history You are now at step current_step and your current observation is: current_observation. Your admissible actions of the current situation are: [ available_actions ]. Now it’s your turn to take one action for the current step. You should first reason step-by-step about the current situation, then think carefully which admissible action best advances the shopping goal. This reasoning process MUST be enclosed within <think> </think> tags. Once you’ve finished your reasoning, you should choose an admissible action for current step and present it within <action> </action> tags. Search-QA prompt You are an expert agent tasked with answering the given question step-by-step. skill_context Your question: task_description Prior to this step, you have already taken step_count step(s). Below is the interaction history where <search> </search> wrapped your past search queries and <information> </information> wrapped the corresponding search results returned by the external search engine. History: memory_context Now it’s your turn to respond for the current step. You should first conduct a reasoning process. This process MUST be enclosed within <think> </think> tags. After completing your reasoning, choose only one of the following actions (do not perform both): (1) If you find you lack some knowledge, you MUST call a search engine to get more external information using format: <search> your query </search>. (2) If you have enough knowledge to answer the question confidently, provide your final answer within <answer> </answer> tags, without detailed illustrations. For example, <answer>Beijing</answer>. 19 Beyond Timestamps: Decision-Aligned On-Policy Distillation for Long-Horizon Agents EHYPERPARAMETERS AND IMPLEMENTATION DETAILS E.1BASELINE DETAILS All methods use the same Qwen2.5-Instruct backbones. The architecture and pre-training details are described in the Qwen2.5 Technical Report (Qwen et al., 2025). Unless marked with ∗ , evaluation uses only the standard task prompt and environment interaction history; ∗ indicates that a retrieved skill is provided at validation and test time. • Vanilla. The instruction-tuned backbone without additional post-training. • Skill-Prompt ∗ . Vanilla with a retrieved task-relevant skill prepended at validation and test time. • GRPO (Shao et al., 2024).Critic-free group-relative reinforcement learning with group- normalized terminal advantages broadcast to response tokens. • Skill-GRPO / Skill-GRPO ∗ . GRPO with retrieved skills during training; skills are removed or retained at inference, respectively. • OPSD (Zhao et al., 2026). Privileged teacher re-scoring of sampled student tokens using de- tached teacher outputs, with privileged context unavailable at inference. • GRPO+OPSD. Joint optimization of trajectory-level GRPO and token-level OPSD objectives. • Skill-SD (Wang et al., 2026a). Skill provided only to the teacher and transferred through importance-weighted distillation. • RLSD (Yang et al., 2026a). A bounded teacher–student coefficient scales GRPO updates while preserving the sign determined by the outcome advantage. • SDAR (Lu et al., 2026). A gated auxiliary self-distillation objective added to GRPO without modifying the original advantage estimation. • StepOPSD (Zhang et al., 2026c). Distillation signals aggregated at the turn level rather than assigned independently to individual tokens. • ALIGNOPSD. Our method, which combines cross-rollout correspondence, supervision rectifi- cation, adaptive decision spans, and hierarchical credit assignment. All post-training methods share the same backbone, environment interface, training data, rollout budget, optimizer budget, and checkpoint selection rule. Inference-time privileged information is provided only for methods explicitly marked with ∗ . We keep the number of policy optimization up- dates fixed across methods; additional correspondence and rectification computations are considered training overhead rather than additional optimization steps. E.2DETAILS OF DECISION-ALIGNED SUPERVISION RECTIFICATION Table 6 summarizes the correspondence and interpolation settings used to construct the rectified supervision signal. Table 6: Rectification settings used to construct decision-aligned supervision. SettingValue Frozen encoderQwen3-Embedding-0.6B Similarity threshold γ H 0.80 Maximum sources K3 Sources per sibling rolloutat most 1 Aggregation temperature / mode0.10 / probability mixture Maximum interpolation α max 0.80 Teacher forcingsame sampled target response Fallbackidentity gap when no valid source exists Thinking traces are parsed from response delimiters before correspondence construction. Invalid traces are excluded, and ties in source selection are resolved using a stable canonical index. For each target turn, the alignment operator first filters candidate source turns by the thinking-similarity threshold, retains at most K high-confidence sources, and restricts each sibling rollout to contribute 20 Beyond Timestamps: Decision-Aligned On-Policy Distillation for Long-Horizon Agents at most one source. The retained privileged views are aggregated using a temperature-controlled probability mixture. The same sampled student response is teacher-forced under both identity and matched privileged contexts, ensuring that rectification modifies only the supervisory context rather than the optimized behavior. When no valid source is retrieved, the method falls back to the identity teacher–student gap. Dense correspondence profiles used for span construction are computed before sparse top-K truncation and are therefore kept separate from the rectification matrix. E.3DETAILS OF SEMI-MARKOV HIERARCHICAL CREDIT ASSIGNMENT Table 7 summarizes the segmentation and hierarchical allocation settings used for turn-level credit assignment. Table 7: Credit assignment settings used in ALIGNOPSD. SettingValue Profile temperature0.10 Boundary quantile0.80 Threshold range[0.01, 0.10] Minimum / maximum span length2 / 8 turns Credit temperature0.5 Span–turn mixing coefficient0.5 Density upper bound4.0 Distance between adjacent profilesJensen–Shannon divergence Neutral referenceuniform token measure Adaptive span construction. Decision spans are constructed from changes in dense correspondence profiles before top-K truncation. For adjacent turns, we compute the correspondence shift using Jensen–Shannon divergence: D corr i,k = JSD(P i,k−1 ,P i,k ).(16) For each batch, the boundary threshold is determined by the empirical quantile of valid correspon- dence shifts and clipped to the predefined range. Candidate boundaries are selected according to their divergence values while enforcing the minimum and maximum span lengths. This procedure produces a contiguous, ordered, and non-overlapping partition of each trajectory. Hierarchical credit allocation. The allocator distributes the trajectory-level outcome advantage through a two-level hierarchy. First, span-level evidence determines the credit budget assigned to each decision span. The span budget is then distributed among turns within the span according to turn-level evidence. Both stages use KL-regularized allocation against a neutral token-mass prior, preventing evidence scores from completely overriding the original token distribution. For implementation, the resulting span and turn allocations are computed as: B i,m = M i,m exp(E span i,m /T ) P m ′ M i,m ′ exp(E span i,m ′ /T ) ,(17) and Q i,k|m = N i,k exp(E i,k /T ) P t∈S i,m N i,t exp(E i,t /T ) .(18) Here, M i,m and N i,k preserve the neutral token-mass allocation, while the evidence terms tilt the distribution toward outcome-consistent decisions. Budget normalization. The hierarchical allocation produces a raw turn density that is normalized with a token-weighted upper bound to prevent excessive concentration on individual turns. We use d max = 4.0 and mix the projected density with the neutral allocation: W i,k = (1− η) + η D i,k , η = 0.5.(19) This normalization preserves the token-weighted credit budget: X k N i,k W i,k = X k N i,k . 21 Beyond Timestamps: Decision-Aligned On-Policy Distillation for Long-Horizon Agents Empty masks and zero sequence advantages use finite unit weights, and all correspondence, span, evidence, and allocation quantities are stop-gradient. FTRAINING DYNAMICS F.1TRAINING CONFIGURATION Table 8 summarizes the benchmark-specific training configuration shared by ALIGNOPSD and matched baselines. Table 8: Benchmark-specific training configuration shared by ALIGNOPSD and matched baselines. SettingALFWorldWebShopSearch-QA NVIDIA A800 GPUs824 Training updates150150150 Tasks per training batch1616128 Rollouts per task888 Maximum prompt tokens2,0484,0964,096 Maximum response tokens/turn512512512 Maximum interaction turns50154 Train / validation temperature1.0 / 0.41.0 / 0.41.0 / 0.4 F.2TEACHER–STUDENT GAP Figure 7 tracks the teacher–student gap and its rectification during training for Qwen2.5–3B and Qwen2.5–7B on ALFWorld, WebShop, and Search-QA. F.3REWARD SCORE CURVES Figure 8 tracks the logged mean critic scores and episode rewards during training for both backbones on all three benchmarks. 22 Beyond Timestamps: Decision-Aligned On-Policy Distillation for Long-Horizon Agents 0.20 0.15 0.10 0.05 0.00 Qwen2.5-3B (nats/token) ALFWorld Original Rectified 050100150 Training step 0.050 0.025 0.000 (nats/token) mean =-0.049 0.20 0.15 0.10 0.05 0.00 Qwen2.5-7B (nats/token) 050100150 Training step 0.05 0.00 (nats/token) mean =-0.078 WebShop 050100150 Training step mean =-0.019 050100150 Training step mean =-0.018 Search-QA 050100150 Training step mean =-0.022 050100150 Training step mean =-0.032 Teacherstudent gap and rectification magnitude during training Figure 7: Teacher–student gap during training. Identity and rectified gaps, together with their pointwise correction ∆δ = δ rect − δ id , for Qwen2.5–3B/7B ALIGNOPSD runs on ALFWorld, WebShop, and Search-QA. 020406080100120 Training update 0 1 2 3 4 5 6 7 Reward score ALFWorld Critic score Episode reward 20406080100120140 Training update 0 2 4 6 WebShop Critic score Episode reward 020406080100120 Training update 0.20 0.25 0.30 0.35 0.40 0.45 0.50 0.55 Search-QA Critic score Episode reward 20406080100120 Training update 2 4 6 8 Reward score Qwen2.5-7B measured Critic score Episode reward 0255075100125150 Training update 0 2 4 6 8 Qwen2.5-7B measured Critic score Episode reward 0255075100125150 Training update 0.3 0.4 0.5 0.6 Qwen2.5-7B measured Critic score Episode reward Reward score over training Figure 8: Critic scores and episode rewards during training. Logged mean critic scores and episode rewards for Qwen2.5–3B/7B ALIGNOPSD runs on ALFWorld, WebShop, and Search-QA. 23 Beyond Timestamps: Decision-Aligned On-Policy Distillation for Long-Horizon Agents F.4ALLOCATION DIAGNOSTICS Figure 9 reports two allocation statistics: turns per span measures the number of interaction turns grouped into each allocated credit span, while turn-weight standard deviation measures the concen- tration of credit across turns. 4.25 4.30 4.35 4.40 4.45 4.50 4.55 Turns per span ALFWorld 3.25 3.50 3.75 4.00 4.25 4.50 WebShop 2.2 2.4 2.6 2.8 Search-QA 0.04 0.06 0.08 0.10 0.12 Turn-weight std. 0.00 0.01 0.02 0.03 0.04 0.05 0.0100 0.0125 0.0150 0.0175 0.0200 0.0225 3.8 4.0 4.2 4.4 Turns per span 2.75 3.00 3.25 3.50 3.75 4.00 4.25 2.0 2.2 2.4 2.6 2.8 3.0 0255075100125150 Training update 0.02 0.03 0.04 0.05 Turn-weight std. 0255075100125150 Training update 0.000 0.005 0.010 0.015 0255075100125150 Training update 0.015 0.020 0.025 0.030 Qwen2.5-3B Qwen2.5-7B Credit-allocation diagnostics Figure 9: Allocation diagnostics during training. Columns show ALFWorld, WebShop, and Search-QA. For each backbone, the first row reports turns per span and the second reports turn- weight standard deviation. Y-axis labels appear only in the leftmost column and x-axis labels only in the bottom row; each panel keeps its own y-axis ticks. 24 Beyond Timestamps: Decision-Aligned On-Policy Distillation for Long-Horizon Agents F.5CROSS-METHOD PERFORMANCE TRACES Figure 10 visualizes training-time performance trajectories of GRPO, SDAR, and ALIGNOPSD across both backbones and all three benchmarks. We report validation success on ALFWorld. Be- cause Search-QA does not provide comparable dense validation histories for all runs, we instead show logged episode success rates smoothed with a seven-update exponential moving average. WebShop reports both validation normalized score and exact success, matching the two metrics in Table 1. 0% 20% 40% 60% 80% 100% Validation success ALFWorld Qwen2.5-3B 20% 30% 40% 50% 60% Episode success (EMA-7) Search-QA 0% 20% 40% 60% 80% 100% Validation metric WebShop 0306090120150 Training update 0% 20% 40% 60% 80% 100% Validation success Qwen2.5-7B 0306090120150 Training update 20% 30% 40% 50% 60% Episode success (EMA-7) 0306090120150 Training update 0% 20% 40% 60% 80% 100% Validation metric GRPOSDARAlignOPSDWebShop ScoreWebShop Acc Figure 10: Cross-method performance traces. Rows correspond to Qwen2.5–3B and Qwen2.5– 7B; columns correspond to ALFWorld, Search-QA, and WebShop. ALFWorld uses validation suc- cess, Search-QA uses logged episode success rate (EMA-7; faint lines show unsmoothed measure- ments), and WebShop uses validation normalized score (solid) and exact success (dashed). Each curve ends at its last logged update; no missing segment is extrapolated. 25