Paper deep dive
WAM-OPD: Sharpening World Action Models via On-Policy Distillation
Panjun Liu, Xiaohan Lei, Shiqi Zhang, Yikun Wang, Yongxin Zhang, Mingyi Hu, Shida Sun, Jiateng Shou, Wengang Zhou, Jiajun Deng, Zhiwei Xiong
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Pretrained world action models (WAMs) provide generalist capabilities across diverse robotic manipulation tasks, yet improving target-task performance to an expert level without degrading pretrained skills remains challenging. We explore on-policy distillation (OPD) for WAMs and introduce WAM-OPD. WAM-OPD inherits the advantage of OPD methods that transfer task-specific teacher knowledge under the student's own induced distribution, rather than directly fitting the student to a narrow task-specific data distribution. However, in closed-loop manipulation, the observation histories change as the student policy evolves, requiring fresh environment rollouts to remain on-policy. Applying OPD to WAMs entails repeated data collection, which is costly even in simulation and often impractical on real robots. To avoid repeated environment rollouts during distillation, we introduce prefix-weighted trajectory replay (PWTR). PWTR uses a fixed trajectory pool composed primarily of initial-student rollouts, supplemented with task-specific teacher rollouts to broaden trajectory coverage. For each trajectory replayed from this pool, PWTR conditions the current policy on successive stored histories to generate fresh denoising paths, along which the task-specific teacher provides supervision. Although these denoising paths are refreshed as the policy evolves, the replayed environment trajectories remain fixed. PWTR therefore reweights per-decision distillation losses using proxy importance weights derived from path scores accumulated over the trajectory prefix preceding each decision to mitigate the resulting shift in the history distribution. Simulated and real-world experiments demonstrate task adaptation without additional environment interaction during distillation. In both settings, WAM-OPD improves target-task performance while retaining near-initial performance on tasks excluded from adaptation.
Tags
Links
- Source: https://arxiv.org/abs/2609.34250v1
- Canonical: https://arxiv.org/abs/2609.34250v1
Trouble viewing inline? Open PDF directly →
Full Text
60,495 characters extracted from source content.
Expand or collapse full text
WAM-OPD: Sharpening World Action Models via On-Policy DistillationThanks: Corresponding author. Panjun Liu Affiliation: Yongxin Zhang, Mingyi Hu, Shida Sun, Jiateng Shou, Xiaohan Lei Affiliation: Wengang Zhou, Jiajun Deng, Zhiwei Xiong Shiqi Zhang Affiliation: University of Science and Technology of China Yikun Wang Affiliation: panjun_liu@mail.ustc.edu.cn, dengjj@ustc.edu.cn Abstract Pretrained world action models (WAMs) provide generalist capabilities across diverse robotic manipulation tasks, yet improving target-task performance to an expert level without degrading pretrained skills remains challenging. To this end, we explore on-policy distillation (OPD) for WAMs and introduce a new framework, namely WAM-OPD. WAM-OPD inherits the advantage of OPD methods that transfer task-specific teacher knowledge under the student’s own induced distribution, rather than directly fitting the student to a narrow task-specific data distribution. However, in closed-loop manipulation, the observation histories change as the student policy evolves, requiring fresh environment rollouts to remain on-policy. Directly applying OPD to WAMs entails repeated data collection, which is costly even in simulation and often impractical on real robots. To avoid repeated environment rollouts during distillation, we introduce prefix-weighted trajectory replay (PWTR). PWTR uses a fixed trajectory pool composed primarily of initial-student rollouts, supplemented with task-specific teacher rollouts to broaden trajectory coverage. For each trajectory replayed from this pool, PWTR conditions the current policy on successive stored histories to generate fresh denoising paths, along which the task-specific teacher provides supervision. Although these denoising paths are refreshed as the policy evolves, the replayed environment trajectories remain fixed. PWTR therefore reweights per-decision distillation losses using proxy importance weights derived from path scores accumulated over the trajectory prefix preceding each decision to mitigate the resulting shift in the history distribution. Simulated and real-world experiments demonstrate task adaptation without additional environment interaction during distillation. Across both settings, WAM-OPD improves target-task performance while retaining near-initial performance on tasks excluded from adaptation. 1 Introduction Pretrained world action models (WAMs) provide a promising foundation for generalist robotic manipulation by capturing broad, reusable behaviors (Ye et al., 2026; Yuan et al., 2026; Li et al., 2026b; Li et al., 2025; Zhu et al., 2025; Kim et al., 2026; AgiBot Research Team et al., 2026). Practical deployment, however, often demands higher reliability on frequently executed tasks, since even occasional failures can lead to recurring workflow interruptions and human intervention. This motivates post-training WAMs to achieve expert-level performance on selected tasks while retaining their broader capabilities. Balancing these objectives remains challenging with existing approaches. Supervised fine-tuning (SFT) offers a straightforward and computationally efficient route to task specialization using demonstrations (Kim et al., 2025). Reinforcement learning (RL) provides an alternative by directly optimizing task rewards through environmental interaction, enabling substantial improvements in task performance (Li et al., 2026a). However, task-specific adaptation can degrade pretrained capabilities under both SFT (Kirkpatrick et al., 2017; Luo et al., 2023; Li et al., 2026a; Zhong et al., 2026) and RL (Wolczyk et al., 2024). Online RL additionally requires repeated environment rollouts (Li et al., 2026a; Qian et al., 2026), which incur substantial execution and reset costs and must accommodate safety constraints on real robots. Sparse rewards can further increase interaction burden by reducing sample efficiency (Andrychowicz et al., 2017; Tan et al., 2025; Li et al., 2026a). These limitations motivate post-training strategies that jointly address target-task proficiency, skill retention, and interaction efficiency. Figure 1 contrasts these task adaptation strategies. Recent advances in on-policy distillation (OPD) for large language models (Shenfeld et al., 2026) and visual generation (Li et al., 2026c; Fang et al., 2026) have shown promise in integrating task-specific expertise while mitigating forgetting. Motivated by these results, we present an early exploration of OPD for WAM adaptation and introduce WAM-OPD. Particularly, on-policy distillation in WAMs involves two levels of sampling: the student’s denoising paths and the observation histories used as conditioning inputs. For a given history, the current student generates fresh denoising paths, while a frozen task-specific teacher provides velocity targets at sampled intermediate states. Generating such denoising paths and obtaining teacher supervision require only model inference. The observation histories, however, are induced by closed-loop execution of the student’s policy, and the distribution evolves as the policy updates. Direct online OPD thus requires repeated environment rollouts to collect histories under the updated policy, making it costly even in simulation and often impractical on real robots. Figure 1: Task adaptation strategies for pretrained WAMs. Left: SFT specializes pretrained WAMs for target tasks but risks forgetting pretrained skills. Middle: Online RL couples policy optimization with repeated environment interaction, while sparse rewards can further reduce sample efficiency. Right: WAM-OPD uses PWTR to replay observation histories from a fixed trajectory pool. A frozen teacher provides dense velocity supervision along fresh student denoising paths, enabling adaptation without additional environment interaction during distillation. To avoid repeated environment rollouts during distillation, we introduce prefix-weighted trajectory replay (PWTR). PWTR uses a fixed trajectory pool composed primarily of initial-student rollouts, supplemented with task-specific teacher rollouts to broaden trajectory coverage. The initial-student rollouts cover histories typical of the policy at the start of distillation, while teacher rollouts broaden coverage of histories the student may encounter later as it acquires task-specific expertise. For each trajectory replayed from this pool, PWTR conditions the current policy on successive stored histories to generate fresh denoising paths, along which the task-specific teacher provides dense supervision. Although these denoising paths are refreshed as the policy evolves, the replayed trajectories remain fixed. PWTR therefore reweights per-decision distillation losses to mitigate the resulting distribution shift. PWTR accumulates denoising-path scores under a student snapshot and the fixed initial-student and task-specific teacher references over preceding sampled decision points. These cumulative scores yield proxy importance weights inspired by a change of measure (Precup et al., 2000), placing greater emphasis on histories more compatible with the student snapshot relative to the fixed references. Together with fresh denoising paths generated by the current policy, this reweighting provides an approximation to the online OPD objective without further environment interaction during distillation. Experiments in simulation (Chen et al., 2025b) and on real robots demonstrate that WAM-OPD enables task adaptation without additional environment interaction during distillation. Across both settings, WAM-OPD improves target-task performance over the pretrained base model while retaining near-initial performance on tasks excluded from adaptation. Our contributions are threefold: • To the best of our knowledge, we are the first to explore OPD for pretrained WAMs and introduce WAM-OPD, which transfers task-specific teacher knowledge through dense supervision along denoising paths generated by the current student policy. • We propose PWTR, combining a fixed trajectory pool with prefix-based proxy weighting to mitigate history distribution shift without environment interaction during distillation. • Simulated and real-world experiments show that WAM-OPD improves target-task performance over the pretrained base model while retaining near-initial performance on tasks excluded from adaptation. 2 Related Work 2.1 World Action Models and Task Adaptation Pretrained WAMs integrate predictive visual modeling with action generation, providing reusable priors for generalist robotic manipulation (Ye et al., 2026; Li et al., 2026b; Kim et al., 2026). Their post-training objectives range from improving control to accelerating inference. WAM-RL jointly improves world and action models through online video fine-tuning and reinforcement learning with reconstruction-based rewards (Qian et al., 2026). Flash-WAM uses consistency distillation to reduce the denoising steps required for video and action generation (Akbari et al., 2026). Our work focuses on teacher-guided task adaptation through on-policy distillation, improving target-task performance while retaining pretrained skills. 2.2 On-Policy Distillation OPD provides teacher supervision on samples generated by the current student. ImitKD and GKD apply this principle to autoregressive generation to reduce the mismatch between training and inference (Lin et al., 2020; Agarwal et al., 2024). Flow-OPD and DiffusionOPD extend it to continuous generation, transferring task-specific expertise through teacher feedback along student-generated denoising paths (Fang et al., 2026; Li et al., 2026c). In closed-loop manipulation, however, the student determines the observations that condition subsequent predictions. DAgger addresses learner-induced distribution shift through repeated rollouts and expert annotation (Ross et al., 2011). VLA-OPD provides dense teacher guidance on student-visited robot states, while retaining online environment rollouts during training (Zhong et al., 2026). Thus, richer supervision alone does not remove the cost of acquiring observation histories consistent with the evolving policy. Our work applies OPD to pretrained WAMs through PWTR, which combines a fixed trajectory pool with proxy prefix weighting to enable adaptation without additional environment interaction during distillation. 2.3 Efficient Distillation and Data Reuse Data reuse reduces different costs depending on what is retained. Lightning OPD caches responses from an SFT-initialized student and their teacher log-probabilities, eliminating the need for a live teacher server during distillation (Wu et al., 2026). ReOPD removes tool-environment interaction by replaying teacher histories for language-agent distillation, with greater supervision weight on earlier steps (Liao et al., 2026). For robotic adaptation, WAM-OPD uses PWTR with a fixed trajectory pool composed primarily of initial-student rollouts, supplemented with task-specific teacher rollouts. Conditioned on stored observation histories, the current student generates fresh denoising paths along which frozen teachers provide dense supervision. Motivated by off-policy importance sampling and V-trace (Precup et al., 2000; Espeholt et al., 2018), PWTR derives proxy prefix weights from accumulated denoising-path scores under a student snapshot and fixed references. These weights are refreshed as the student evolves, adjusting the contribution of replayed histories while keeping the trajectory pool fixed. 3 Method WAM-OPD combines prefix-weighted trajectory replay (PWTR) with prefix-weighted on-policy distillation to adapt pretrained WAMs, as illustrated in Figure 2. It constructs a fixed pool D primarily from rollouts of the initial student πθ0 _ _0, supplemented with rollouts from task-specific teachers πT(j) _T^(j). At each update, it samples a trajectory from D and processes its sampled decision points in temporal order. At sampled decision point i, it accumulates scores of stored denoising paths under a student snapshot and the fixed references over preceding sampled points in the same trajectory to obtain prefix scores Φν(i) _ν(i). These scores determine the proxy weight wiw_i to mitigate history distribution mismatch. Conditioned on the stored history hih_i, the current student πθ _θ generates a fresh denoising path. At a detached low-noise state ¯s x_s, WAM-OPD uses velocity targets T(j)v_T^(j) from the frozen teacher to supervise the student field θv_θ and weights the loss by wiw_i. It averages these losses and accumulates gradients over the sampled decision points before updating the student, without further environment interaction. We first formulate the online OPD objective for WAMs in Section 3.1. We then introduce PWTR, including the fixed trajectory pool and prefix weight construction, in Section 3.2, followed by prefix-weighted on-policy distillation in Section 3.3. Figure 2: Overview of WAM-OPD. Left: Prefix-weighted trajectory replay (PWTR). A fixed pool D contains primarily initial-student rollouts, supplemented by task-specific teacher rollouts. Stored denoising paths are scored by a student snapshot and the fixed initial-student and teacher references. The cumulative prefix compatibility scores Φθ¯(i) _ θ(i), Φθ0(i) _ _0(i), and ΦT(j)(i) _T^(j)(i) yield proxy weights wiw_i. Right: On-policy distillation. Given a stored history hih_i, the current student generates a fresh denoising path. A frozen teacher supervises the student’s velocity at a detached low-noise query state ¯s x_s on this path. Prefix-weighted losses are averaged over the sampled decision points of one trajectory, with gradients accumulated for one student update. Here sgsg denotes stop-gradient and ℬB indexes the trajectory’s sampled decision points. 3.1 On-Policy Distillation Objective for WAMs Diffusion-based OPD. In a flow-based formulation of diffusion-based OPD (Li et al., 2026c; Fang et al., 2026), denoising paths are obtained by integrating the current student’s velocity field θv_θ from noise to data, while the frozen teacher’s velocity field Tv_T provides supervision at intermediate states. For a conditioning input c drawn from a fixed distribution p(c)p(c), let qθ(s∣c,s)q_θ(x_s c,s) denote the distribution of student-generated states at denoising time s. A corresponding velocity-matching objective is ℒOPD(θ) _OPD(θ) =c∼p(c)s∼pss∼qθ(⋅∣c,s)[∥θ(s,s∣c)−T(s,s∣c)∥22]. =E_c p(c)E_s p_sE_x_s q_θ(· c,s) [ \|v_θ(x_s,s c)-v_T(x_s,s c) \|_2^2 ]. (1) Here, s=1s=1 and s=0s=0 correspond to noise and data, respectively, and psp_s specifies the query-time distribution. The on-policy component lies in sampling intermediate states from qθq_θ, which evolves with the student, while the conditioning distribution p(c)p(c) remains fixed. Conditioning inputs can therefore be reused throughout distillation, with fresh student-generated denoising paths and teacher supervision obtained through model inference alone. OPD in closed-loop WAMs. Closed-loop WAMs introduce policy dependence into the conditioning distribution itself. At each replanning decision t, the student generates an action chunk ta_t conditioned on the history hth_t. Here, hth_t denotes the interaction history available at decision t; the policy uses the conditioning inputs extracted from this history. Executing part of this chunk changes the environment and produces subsequent observations. Let dθd_θ denote the distribution of histories encountered across replanning decisions during closed-loop execution of the current student policy. Because the student’s preceding actions influence subsequent observations and thus the policy inputs, dθd_θ evolves with the policy. Replacing the fixed conditioning distribution in Eq. (1) with dθd_θ yields the online WAM OPD objective ℒOPDWAM(θ) ^WAM_OPD(θ) =h∼dθs∼pss∼qθ(⋅∣h,s)[∥θ(s,s∣h)−T(s,s∣h)∥22]. =E_h d_θE_s p_sE_x_s q_θ(· h,s) [ \|v_θ(x_s,s h)-v_T(x_s,s h) \|_2^2 ]. (2) Here, θv_θ and Tv_T denote the student and frozen teacher action velocity fields, respectively. Unlike Eq. (1), the outer expectation follows dθd_θ, the distribution of histories induced by closed-loop execution of the current student. On-policy sampling therefore involves both histories drawn from dθd_θ and denoising states drawn from qθ(⋅∣h,s)q_θ(· h,s). Given a stored history, the current student can regenerate fresh denoising states through model inference, but the history itself need not follow the current dθd_θ. Obtaining fresh histories from dθd_θ requires closed-loop execution, so directly realizing online OPD entails repeated environment rollouts (Zhong et al., 2026), which are costly in simulation and often impractical on real robots. To approximate this objective using a fixed trajectory pool without further environment interaction during distillation, WAM-OPD uses PWTR, described in Section 3.2. 3.2 Prefix-Weighted Trajectory Replay Fixed trajectory pool. As illustrated in Figure 2, WAM-OPD constructs a fixed trajectory pool =S∪TD=D_S _T before distillation. The pool consists primarily of initial-student trajectories SD_S, collected by executing πθ0 _ _0 across all target tasks, supplemented with teacher trajectories TD_T collected by executing each frozen teacher πT(j) _T^(j) on its assigned task. Initial-student rollouts provide histories encountered before adaptation, including unsuccessful executions, while teacher rollouts broaden coverage with histories from task-specific policies. The pool remains fixed throughout distillation. At each replanning decision t, the collecting policy generates an action chunk ta_t of horizon H and executes its first ℓt≤H _t≤ H actions before receiving the next observation. Each stored record contains the history hth_t, from which the collecting policy’s conditioning inputs are extracted, together with the sampled action chunk ta_t, the executed length ℓt _t, and the complete denoising path that produced ta_t. Each trajectory also carries task and source labels: the task identifies the supervising teacher, and the source distinguishes initial-student from teacher rollouts. The executed length ℓt _t determines which action coordinates are included in path scoring. Stored denoising paths are used to estimate prefix weights. We use equally spaced temporal subsampling with n=16n=16, preserving the order of the sampled decision points within each trajectory. The path score at a sampled decision point contributes only to the weights of later sampled points in the same trajectory. Stored action chunks are not used as regression targets. Reweighting replayed histories. PWTR reweights the distillation loss to account for the changing history distribution as the student evolves. During training, it selects the initial-student or teacher source with probabilities η and 1−η1-η, respectively, and samples a trajectory from that source. At the population level, this sampling scheme defines the reference mixture pmix=ηdθ0+(1−η)dTp_mix=η d_ _0+(1-η)d_T, where dθ0d_ _0 and dTd_T are the initial-student and task-specific teacher history distributions under a common temporal sampling convention. When pmixp_mix covers the support of dθd_θ, weighting by dθ(h)/pmix(h)d_θ(h)/p_mix(h) recovers the online expectation over histories (Precup et al., 2000; Sutton and Barto, 2018). PWTR constructs proxy weights for replayed histories through their associated trajectory prefixes. Under the assumptions in Appendix A, shared initialization and environment-transition factors cancel in the full-prefix density ratio at a fixed original decision index, leaving a ratio determined by preceding action probabilities. This structure motivates weighting each history using the actions that precede it. Since the flow sampler does not directly provide these probabilities, PWTR scores stored denoising paths by their compatibility with each policy’s velocity field and accumulates these scores over preceding sampled decision points to construct the weights. Denoising path scoring. For a stored denoising path at replanning decision u, let smm=0K\x_s_m\_m=0^K denote its states at noise levels s0>⋯>sKs_0>·s>s_K, with Δm=sm+1−sm<0 _m=s_m+1-s_m<0. PWTR evaluates the path under a scoring policy ν, corresponding to the student snapshot, initial student, or task-specific teacher: φν(u) _ν(u) =−∑m=0K−112σm2du‖Pu[sm+1−sm−Δmν(sm,sm∣hu)]‖22. =- _m=0^K-1 12 _m^2d_u \|P_u [x_s_m+1-x_s_m- _mv_ν(x_s_m,s_m h_u) ] \|_2^2. (3) The residual compares each recorded denoising increment with the increment predicted by policy ν at the same stored state and history huh_u. The projection PuP_u selects the dud_u coordinates corresponding to the ℓu _u executed actions, restricting scoring to the portion of the chunk applied to the environment. Division by dud_u normalizes for the number of scored coordinates, while σm=maxσmin,κ−Δm _m= \ _ ,κ - _m\ provides an auxiliary scoring scale, with σmin>0 _ >0 and κ>0κ>0. Higher scores indicate closer agreement between the policy’s velocity field and the stored path. The resulting φν(u) _ν(u) serves as a path-compatibility proxy for prefix weighting, rather than an exact action log-likelihood. Prefix weight construction. Let tτ,kt_τ,k denote the original decision index of the k-th sampled decision point in trajectory τ. For i=(τ,k)i=(τ,k), we define the prefix score as Φν(i)=∑r<kφν(tτ,r) _ν(i)= _r<k _ν(t_τ,r), where the sum is zero for the first sampled point. The current point is excluded because its history is available before its action is generated. Following the mixture structure of the full-prefix density ratio, PWTR compares a student snapshot θ¯ θ with the initial-student and task-specific teacher references, as illustrated in Figure 2: Si=λ[Φθ¯(i)−log(ηeΦθ0(i)+(1−η)eΦT(j)(i))].S_i=λ [ _ θ(i)- (η e _ _0(i)+(1-η)e _T^(j)(i) ) ]. (4) Here, λ≥0λ≥ 0 controls the reweighting strength, and exp(Si) (S_i) defines an unnormalized proxy weight. Larger weights favor replayed records whose preceding denoising paths are more compatible with the student snapshot relative to the fixed references. Both fixed references score every stored path to form the mixture denominator. Within each collection condition, PWTR normalizes exp(Si) (S_i) to unit mean under the empirical training measure, clips the normalized weights at threshold C, and renormalizes them under the same measure to obtain wiw_i (Appendix, Eq. (10)). This measure follows the trajectory-sampling probabilities and the averaging of per-decision losses within each trajectory. Clipping limits the influence of extreme scores, while renormalization maintains unit mean within each condition. Setting λ=0λ=0 yields wi=1w_i=1, recovering unweighted mixed-source distillation. The fixed-reference scores are computed once for all stored paths and cached. Every R student updates, PWTR refreshes the snapshot θ¯ θ, evaluates its velocity field at the stored denoising states, and recomputes the prefix weights. The weights remain fixed between refreshes. 3.3 Prefix-Weighted On-Policy Distillation Local velocity supervision. Each sampled trajectory is processed in temporal order. Let ℬ=ℬτB=B_τ index the sampled decision points of trajectory τ. Each record i provides the stored history hih_i, a task index jij_i, and a cached prefix weight wiw_i, where hi=htτ,kh_i=h_t_τ,k is the interaction history available at that decision. Conditioned on hih_i, the current student generates a fresh denoising path from Gaussian noise. At a low-noise query time si∼pss_i p_s, the sampled state is detached as ¯i=sg(si) x_i=sg(x_s_i), where sgsg denotes stop-gradient. As illustrated in Figure 2, the student and frozen task-specific teacher are evaluated at the same detached state ¯i x_i. The teacher’s action velocity provides a local supervision target for the student along this freshly generated denoising path. Trajectory-level student update. We average the prefix-weighted velocity-matching losses over the sampled decision points of each trajectory: ℒ^WAM-OPD(θ) L_WAM -OPD(θ) =1|ℬ|∑i∈ℬsg(wi)⋅‖θ(¯i,si∣hi)−T(ji)(¯i,si∣hi)‖22. = 1|B| _i sg(w_i)· \|v_θ( x_i,s_i h_i)-v_T^(j_i)( x_i,s_i h_i) \|_2^2. (5) Query states, prefix weights, and teacher targets are treated as constants, so gradients propagate only through the current student’s velocity predictions. We accumulate gradients over these points before a single student update. The loss supervises only the action velocity field; the video branch receives no direct distillation loss. This objective combines teacher supervision on fresh current-student denoising paths with proxy prefix weights that mitigate the history distribution mismatch, enabling distillation from the fixed trajectory pool without further environment interaction. 4 Experiments We evaluate whether WAM-OPD can improve target-task performance while retaining pretrained skills, analyze the trajectory-pool and weighting choices in PWTR, and examine its applicability to real-world manipulation. All WAM-OPD distillation updates use a fixed trajectory pool without additional environment interaction. 4.1 Experimental Setup Table 1: Success rates on RoboTwin. Both teachers supervise WAM-OPD; underlining marks teaching tasks. Parentheses show changes in group averages from the initial student. Bold denotes column maxima excluding Online OPD, which uses additional environment rollouts during distillation. Method Target tasks Retention tasks Average Stapler Microwave Switch Mug Laptop Pillbottle Dual Bottles Handover Target Retention Initial student 67.5 45.5 68.5 68.5 99.0 99.0 98.0 90.0 56.50 87.17 Stapler teacher 89.5 33.5 52.5 66.0 87.5 97.0 95.0 56.5 61.50 75.75 Microwave teacher 31.0 60.5 62.0 26.0 98.5 92.5 97.5 2.5 45.75 63.17 π 0.5 96.0 25.0 22.0 17.5 22.0 89.0 1.0 48.25 (−8.25-8.25) 29.42 (−57.75-57.75) STEAM 76.0 39.0 61.5 42.5 97.5 95.5 91.5 67.5 57.50 (+1.00+1.00) 76.00 (−11.17-11.17) WAM-OPD (Ours) 80.5 59.0 71.5 68.0 98.5 98.0 95.0 90.5 69.75 (+13.25) 86.92 (−0.25-0.25) Online OPD 81.5 59.0 68.0 68.5 97.0 98.0 98.5 88.0 70.25 (+13.75+13.75) 86.33 (−0.83-0.83) Tasks and evaluation. On RoboTwin 2.0 (Chen et al., 2025b), we jointly adapt a single student on Move Stapler Pad and Open Microwave, abbreviated as Stapler and Microwave. We evaluate skill retention on six tasks excluded from adaptation: Switch, Mug, Laptop, Pillbottle, Dual Bottles, and Handover. The same adapted student checkpoint is evaluated on both target and retention tasks. Evaluation uses 100 independent scenes per task and setting, with two runs per scene, yielding 200 trials per task and setting. All methods share the evaluation scenes, instructions, and inference-noise seeds. Target and retention averages assign equal weight to their two and six constituent tasks, respectively. Additional evaluation and interaction-cost details are provided in Appendix C. Compared policies. We compare WAM-OPD with the initial student, two task-specific SFT teachers, π (Chen et al., 2025a), and STEAM (Liu et al., 2026) under the same evaluation protocol. The Stapler and Microwave teachers are independently fine-tuned from the same pretrained checkpoint, evaluated on all eight tasks, and jointly supervise one WAM-OPD student. Controlled variants are analyzed in Section 4.3. Fixed-pool distillation protocol. For each target task, we collect 40 initial-student trajectories: 8 under demo_clean and 32 under demo_randomized. Adding 20 teacher rollouts for each target task yields a fixed pool of 120 trajectories. These teacher rollouts are distinct from the demonstrations used for teacher SFT. During distillation, the current student generates fresh denoising paths conditioned on stored histories, with supervision from the corresponding task teacher and prefix weights from PWTR (Section 3.2). The pool remains fixed, and distillation requires no additional environment interaction. Implementation details. The student and teachers use Fast-WAM (Yuan et al., 2026), initialized from its RoboTwin checkpoint. We optimize all student MoT parameters using AdamW at a learning rate of 10−610^-6 for 96 updates. We use equally spaced temporal subsampling with n=16n=16 and average the loss over the sampled decision points of one trajectory per update. The loss matches student and teacher action velocities at low-noise states, with no direct video supervision. All evaluated policies use 10 denoising steps to generate 32-step action chunks, executing up to 24 steps before replanning. Pool composition, backbone details, and additional hyperparameters appear in Appendix D; prefix-weight normalization and clipping are detailed in Appendix A. 4.2 Target-Task Adaptation and Skill Retention Table 1 reports target-task adaptation and skill retention for the same student policy. WAM-OPD increases mean target-task success from 56.50% to 69.75%, while maintaining 86.92% mean success on the six retention tasks compared with the initial student’s 87.17%. This corresponds to a 13.25-percentage-point target-task gain and a 0.25-point decrease in mean retention performance. Target-task adaptation. WAM-OPD improves success from 67.5% to 80.5% on Stapler and from 45.5% to 59.0% on Microwave, yielding gains of 13.0 and 13.5 percentage points. WAM-OPD achieves the highest mean target-task success rate among the evaluated SFT and RL baselines, with performance comparable to online OPD (69.75% vs. 70.25%), without additional environment interaction during distillation. The SFT specialists exhibit cross-task degradation: the Stapler teacher obtains 33.5% on Microwave, while the Microwave teacher obtains 31.0% on Stapler. WAM-OPD instead improves both target tasks within one student, combining guidance from the two specialists. Pretrained skill retention. The target-task gains are accompanied by near-initial performance on the six evaluated retention tasks. Both SFT teachers underperform the initial student on all six tasks, with mean success rates of 75.75% and 63.17%, compared with 86.92% for WAM-OPD. For example, Handover success falls from 90.0% to 56.5% for the Stapler teacher and 2.5% for the Microwave teacher, whereas WAM-OPD achieves 90.5%. Together, the target and retention results show that the complete WAM-OPD method improves the adapted skills while retaining near-initial performance on tasks excluded from adaptation. 4.3 Analysis of Prefix-Weighted Trajectory Replay Table 2: Ablation study of PWTR. Success rates are reported over 200 cases per task; M-Shuffle permutes weights among decision points at the same position k in the sampled sequence. Task Student Pool Teacher Pool Mixed Pool w/o prefix w/ prefix w/o prefix w/ prefix w/o prefix M-Shuffle w/ prefix (Ours) Stapler 75.0 77.5 74.0 72.0 77.0 74.0 80.5 Microwave 57.0 58.0 54.0 55.5 57.5 57.0 59.0 Avg. 66.00 67.75 64.00 63.75 67.25 65.50 69.75 Figure 3: History weight dynamics during distillation. Mean history weights at sampled decision points are shown separately for initial-student and teacher rollouts in the mixed trajectory pool. The dashed line indicates uniform weighting. Trajectory-pool composition. The Student Pool contains trajectories from the initial student, the Teacher Pool contains task-specific teacher rollouts, and the Mixed Pool combines both sources. With prefix weights set to one, the Mixed Pool achieves 67.25% mean success, compared with 66.00% for the Student Pool and 64.00% for the Teacher Pool, as shown in Table 2. The Mixed Pool yields the highest success on both tasks with prefix weighting. These comparisons favor the mixed-pool configuration, in which initial-student trajectories are supplemented with teacher trajectories. Effect of prefix weighting. For each pool, w/o prefix assigns unit weight to the distillation loss at each sampled decision point, while w/ prefix applies PWTR weights. As shown in Table 2, weighting increases mean success from 66.00% to 67.75% in the Student Pool and from 67.25% to 69.75% in the Mixed Pool, yielding gains of 1.75 and 2.50 percentage points, respectively. In the Teacher Pool, mean success changes from 64.00% to 63.75%, showing no average gain. Thus, prefix weighting benefits both pools containing initial-student rollouts, with the largest gain in the Mixed Pool. The Teacher Pool result may reflect limited coverage of student-visited histories: reweighting adjusts the contribution of stored histories but cannot expand their coverage. Figure 3 also shows how mean history weights evolve for initial-student and teacher rollouts during Mixed Pool distillation. History–weight correspondence. Within the Mixed Pool, M-Shuffle randomly permutes PWTR weights among decision points at the same position k in the sampled sequence. This preserves the weight values at each sampled position while disrupting the correspondence between individual histories and their weights. Full PWTR achieves 69.75% mean success versus 65.50% for M-Shuffle, a gain of 4.25 percentage points. This result supports the importance of history–weight correspondence. Together, these ablations favor a mixed trajectory pool with history-specific prefix weighting. 4.4 Real-World Evaluation Platform and tasks. The platform comprises two UR5e arms, a head camera providing a global view, and one wrist camera per arm providing local views. We evaluate Pick-and-Place Corn, Pick-and-Place Block, and Press Button, abbreviated as Corn, Block, and Button. The first two tasks involve approaching, grasping, transporting, and placing objects, while Button assesses end-effector alignment and contact. The student is initialized from the Fast-WAM RoboTwin checkpoint and pretrained on real-robot data to obtain a base student adapted to the robot’s observation and action interfaces. Task-specific teachers are initialized from this shared checkpoint, fine-tuned on their respective single-task datasets, and frozen during distillation. Fixed-pool distillation. For each task, trajectories are collected once using the base student and combined with teacher-provided trajectories to form a fixed pool. WAM-OPD uses supervision from all three task-specific teachers to adapt a single student. As in simulation, fresh student denoising paths and teacher targets are generated using stored histories, with no additional robot rollouts during distillation. The real-robot observation and action interfaces are detailed in Appendix D. Evaluation protocol. Evaluation layouts are independent of those used for training-data collection. Each method is evaluated on 50 paired layouts per task, with methods tested on the same layouts in interleaved order. A pick-and-place trial succeeds if the object reaches the designated region and remains stable after release; a button-press trial succeeds if the button is activated. We report task success rates for the base student, the corresponding task-specific teachers, and the single student adapted by WAM-OPD. Figure 4: Real-world evaluation on Corn, Block, and Button. Left: Experimental scenes for the three manipulation tasks. Right: Success rates of the base student, the corresponding task-specific teachers, and WAM-OPD. WAM-OPD adapts a single student using all three teachers and a fixed trajectory pool, without additional robot rollouts during distillation. Results. Figure 4 presents the task scenes and success-rate comparisons. In this three-task evaluation, WAM-OPD improves upon the base student, supporting target-task adaptation on the real robot within a single policy. These gains are achieved using the fixed trajectory pool throughout distillation, without additional robot interaction during that stage. 5 Conclusion We presented WAM-OPD, an on-policy distillation framework for adapting pretrained world action models. Experiments in simulation and on real robots show that WAM-OPD improves target-task performance over the pretrained base model while retaining near-initial performance on tasks excluded from adaptation. Through prefix-weighted trajectory replay (PWTR), these gains are achieved using a fixed trajectory pool without additional environment interaction during distillation. These findings support OPD as a promising approach to task specialization with pretrained skill retention. Future work could improve trajectory coverage and proxy prefix-weight estimation to further strengthen adaptation. References Agarwal et al. (2024) R. Agarwal, N. Vieillard, Y. Zhou, P. Stanczyk, S. Ramos, M. Geist, and O. Bachem On-policy distillation of language models: learning from self-generated mistakes. In International Conference on Learning Representations, Cited by: §2.2. AgiBot Research Team et al. (2026) AgiBot Research Team, R. Liu, W. Zhao, Z. Yang, L. Chen, P. Zhou, S. Chen, G. Ren, Y. Peng, R. Jin, et al. GE-Act 2.0: pretraining and scaling a world-action model for robotic manipulation. arXiv preprint arXiv:2609.05588. Cited by: §1. Akbari et al. (2026) A. Akbari, C. Zhang, A. Akbari, L. Zhao, Y. Chen, W. Chen, X. Zhang, G. Yuan, and Y. Wang Flash-WAM: modality-aware distillation for world action models. arXiv preprint arXiv:2606.05254. External Links: Link Cited by: §2.1. Andrychowicz et al. (2017) M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, P. Abbeel, and W. Zaremba Hindsight experience replay. In Advances in Neural Information Processing Systems, Cited by: §1. Chen et al. (2025a) K. Chen, Z. Liu, T. Zhang, Z. Guo, S. Xu, H. Lin, H. Zang, X. Li, Q. Zhang, Z. Yu, G. Fan, T. Huang, Y. Wang, and C. Yu πRL _ RL: online RL fine-tuning for flow-based vision-language-action models. arXiv preprint arXiv:2510.25889. External Links: Link Cited by: §D.2, §4.1. Chen et al. (2025b) T. Chen, Z. Chen, B. Chen, Z. Cai, Y. Liu, Z. Li, Q. Liang, X. Lin, Y. Ge, Z. Gu, W. Deng, Y. Guo, T. Nian, X. Xie, Q. Chen, K. Su, T. Xu, G. Liu, M. Hu, H. Gao, K. Wang, Z. Liang, Y. Qin, X. Yang, P. Luo, and Y. Mu RoboTwin 2.0: a scalable data generator and benchmark with strong domain randomization for robust bimanual robotic manipulation. arXiv preprint arXiv:2506.18088. Cited by: §1, §4.1. Espeholt et al. (2018) L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, I. Dunning, S. Legg, and K. Kavukcuoglu IMPALA: scalable distributed deep-RL with importance weighted actor-learner architectures. In International Conference on Machine Learning, p. 1407–1416. Cited by: §2.3. Fang et al. (2026) Z. Fang, W. Huang, Y. Zeng, Y. Zhao, S. Chen, K. Feng, Y. Lin, L. Chen, Z. Chen, S. Cao, and F. Zhao Flow-OPD: on-policy distillation for flow matching models. arXiv preprint arXiv:2605.08063. Cited by: §1, §2.2, §3.1. Kim et al. (2025) M. J. Kim, C. Finn, and P. Liang Fine-tuning vision-language-action models: optimizing speed and success. In Robotics: Science and Systems, Cited by: §1. Kim et al. (2026) M. J. Kim, Y. Gao, T. Lin, Y. Lin, Y. Ge, G. Lam, P. Liang, S. Song, M. Liu, C. Finn, and J. Gu Cosmos policy: fine-tuning video models for visuomotor control and planning. In International Conference on Learning Representations, Cited by: §1, §2.1. Kirkpatrick et al. (2017) J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, D. Hassabis, C. Clopath, D. Kumaran, and R. Hadsell Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences 114 (13), p. 3521–3526. Cited by: §1. Li et al. (2026a) H. Li, Y. Zuo, J. Yu, Y. Zhang, Z. Yang, K. Zhang, X. Zhu, Y. Zhang, T. Chen, G. Cui, et al. SimpleVLA-RL: scaling VLA training via reinforcement learning. In International Conference on Learning Representations, Cited by: §1. Li et al. (2026b) L. Li, Q. Zhang, Y. Luo, S. Yang, R. Wang, F. Han, M. Yu, Z. Gao, N. Xue, X. Zhu, Y. Shen, and Y. Xu Causal world modeling for robot control. arXiv preprint arXiv:2601.21998. Cited by: §1, §2.1. Li et al. (2026c) Q. Li, J. Yu, K. Jiang, Y. Wei, Z. Xing, P. Li, R. Chu, S. Zhang, Y. Liu, and Z. Wu DiffusionOPD: a unified perspective of on-policy distillation in diffusion models. arXiv preprint arXiv:2605.15055. Cited by: §1, §2.2, §3.1. Li et al. (2025) S. Li, Y. Gao, D. Sadigh, and S. Song Unified video action model. In Robotics: Science and Systems, Cited by: §1. Liao et al. (2026) B. Liao, H. Dong, C. Monz, X. Xu, L. Dong, and F. Wei Multi-turn on-policy distillation with prefix replay. arXiv preprint arXiv:2607.04763. External Links: Link Cited by: §2.3. Lin et al. (2020) A. Lin, J. Wohlwend, H. Chen, and T. Lei Autoregressive knowledge distillation through imitation learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, p. 6121–6133. External Links: Document, Link Cited by: §2.2. Liu et al. (2026) Z. Liu, Q. Gu, Y. Wang, D. Qiao, Y. Zhang, S. Chen, L. Shi, T. Zhou, Z. Huang, K. Chen, Z. Guo, Q. Zhang, J. Yu, X. Liang, G. Fan, Y. Wang, F. Gao, X. Chen, and C. Yu STEAM: self-supervised temporal ensemble advantage modeling for real-world robot learning. arXiv preprint arXiv:2606.29834. External Links: Link Cited by: §D.2, §4.1. Luo et al. (2023) Y. Luo, Z. Yang, F. Meng, Y. Li, J. Zhou, and Y. Zhang An empirical study of catastrophic forgetting in large language models during continual fine-tuning. arXiv preprint arXiv:2308.08747. Cited by: §1. Precup et al. (2000) D. Precup, R. S. Sutton, and S. Singh Eligibility traces for off-policy policy evaluation. In International Conference on Machine Learning, Cited by: §1, §2.3, §3.2. Qian et al. (2026) Z. Qian, X. Chi, Y. Qi, H. Li, Z. Y. Chen, and S. Zhang WAM-RL: world-action model reinforcement learning with reconstruction rewards and online video SFT. arXiv preprint arXiv:2606.17906. Cited by: §1, §2.1. Ross et al. (2011) S. Ross, G. Gordon, and J. A. Bagnell A reduction of imitation learning and structured prediction to no-regret online learning. In International Conference on Artificial Intelligence and Statistics, Cited by: §2.2. Shenfeld et al. (2026) I. Shenfeld, M. Damani, J. Hübotter, and P. Agrawal Self-distillation enables continual learning. arXiv preprint arXiv:2601.19897. Cited by: §1. Sutton and Barto (2018) R. S. Sutton and A. G. Barto Reinforcement learning: an introduction. 2 edition, MIT Press. Cited by: §3.2. Tan et al. (2025) S. Tan, K. Dou, Y. Zhao, and P. Krähenbühl Interactive post-training for vision-language-action models. arXiv preprint arXiv:2505.17016. Cited by: §1. Wolczyk et al. (2024) M. Wolczyk, B. Cupiał, M. Ostaszewski, M. Bortkiewicz, M. Zając, R. Pascanu, Ł. Kuciński, and P. Miłoś Fine-tuning reinforcement learning models is secretly a forgetting mitigation problem. In Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 235, p. 53039–53078. External Links: Link Cited by: §1. Wu et al. (2026) Y. Wu, S. Han, and H. Cai Lightning OPD: efficient post-training for large reasoning models with offline on-policy distillation. arXiv preprint arXiv:2604.13010. Cited by: §2.3. Ye et al. (2026) S. Ye, Y. Ge, K. Zheng, S. Gao, S. Yu, G. Kurian, S. Indupuru, Y. L. Tan, C. Zhu, J. Xiang, et al. World action models are zero-shot policies. arXiv preprint arXiv:2602.15922. Cited by: §1, §2.1. Yuan et al. (2026) T. Yuan, Z. Dong, Y. Liu, and H. Zhao Fast-WAM: do world action models need test-time future imagination?. arXiv preprint arXiv:2603.16666. Cited by: §1, §4.1. Zhong et al. (2026) Z. Zhong, H. Yan, J. Li, J. He, T. Zhang, and H. Li VLA-OPD: bridging offline SFT and online RL for vision-language-action models via on-policy distillation. arXiv preprint arXiv:2603.26666. Cited by: §1, §2.2, §3.1. Zhu et al. (2025) C. Zhu, R. Yu, S. Feng, B. Burchfiel, P. Shah, and A. Gupta Unified world models: coupling video and action diffusion for pretraining on large robotic datasets. In Robotics: Science and Systems, Cited by: §1. Appendix A History-Prefix Reweighting A.1 Ideal History Ratios We consider complete interaction histories conditional on a task and collection condition g. Histories are indexed by original environment replanning decisions, with a common absorbing continuation after termination used for this ideal fixed-index analysis. Let auexeca_u^exec denote the action prefix executed between decisions u and u+1u+1. For a stochastic policy ν with executed-action density πνexec _ν^exec, a shared initial distribution p0p_0, and a shared environment kernel P, the history density at decision t is dν,t(ht)=p0(o0)∏u<tπνexec(auexec∣hu)P(ou+1∣hu,auexec).d_ν,t(h_t)=p_0(o_0) _u<t _ν^exec(a_u^exec h_u)P(o_u+1 h_u,a_u^exec). (6) The task instruction is included in the conditioning, which is otherwise suppressed in the notation. The product includes every preceding decision. Different initial-state distributions or environment dynamics would introduce additional factors into the density ratio. The collection source is selected once per trajectory. Writing ηg _g for the initial-student source probability conditional on g, the behavior history distribution is a mixture of complete prefix densities: pmix,t(ht)=ηgdθ0,t(ht)+(1−ηg)dT,t(j)(ht).p_mix,t(h_t)= _gd_ _0,t(h_t)+(1- _g)d_T,t^(j)(h_t). (7) When conditioning preserves the source proportions, ηg=η _g=η. Define Aν(t)=∑u<tlogπνexec(auexec∣hu).A_ν(t)= _u<t _ν^exec(a_u^exec h_u). Cancellation of the shared initial-state and environment-transition factors gives ρt(ht)=dθ,t(ht)pmix,t(ht)=expAθ(t)ηgexpAθ0(t)+(1−ηg)expAT(j)(t). _t(h_t)= d_θ,t(h_t)p_mix,t(h_t)= A_θ(t) _g A_ _0(t)+(1- _g) A_T^(j)(t). (8) The denominator is a mixture of prefix products, rather than a product of per-decision mixtures. If dθ,td_θ,t is absolutely continuous with respect to pmix,tp_mix,t, then for any f integrable under dθ,td_θ,t, ht∼dθ,t[f(ht)]=ht∼pmix,t[ρt(ht)f(ht)].E_h_t d_θ,t[f(h_t)]=E_h_t p_mix,t[ _t(h_t)f(h_t)]. (9) For fixed student parameters, f may be the distillation loss averaged over fresh student denoising queries conditional on hth_t. Only decisions preceding t enter the ratio because hth_t is available before the current action is generated. At t=0t=0, the empty prefix has ratio one under the shared initial distribution. A.2 Prefix Compatibility Scores Equation (3) defines a path-compatibility score φν(u) _ν(u) from stored denoising transitions. This score is not assumed to be an exact executed-action log likelihood. Let tτ,kt_τ,k denote the original decision index of the k-th sampled decision point in trajectory τ. For i=(τ,k)i=(τ,k), the prefix score is Φν(i)=∑r<kφν(tτ,r). _ν(i)= _r<k _ν(t_τ,r). These prefix scores are used in Equation (4), with reference-mixture coefficient η. Only preceding sampled decision points contribute, and the sum is zero for the first sampled point. Both fixed references evaluate every sampled point regardless of its collection source. A.3 Sampling Measure, Normalization, and Stable Computation We use equally spaced temporal subsampling with n=16n=16 and average the loss over the sampled decision points of each trajectory. Let ℬτB_τ index these points and let q(τ)q(τ) denote the trajectory-sampling probability, including source selection. The induced sampling probability of i=(τ,k)i=(τ,k) is p(i)=q(τ)|ℬτ|.p(i)= q(τ)|B_τ|. Let IgI_g index the sampled decision points belonging to collection condition g. For conditions with positive probability, define p(g)=∑i∈Igp(i),pi|g=p(i)p(g).p(g)= _i∈ I_gp(i), p_i g= p(i)p(g). All normalization uses this sampling measure, including its source-selection probabilities. With zi=expSiz_i= S_i, normalization, clipping, and renormalization give μg _g =∑i′∈Igpi′|gzi′, = _i ∈ I_gp_i gz_i , ri r_i =zi/μg, =z_i/ _g, (10) w~i w_i =minri,C, = \r_i,C\, wi w_i =w~i∑i′∈Igpi′|gw~i′,i∈Ig. = w_i _i ∈ I_gp_i g w_i , i∈ I_g. Here C>0C>0 and λ≥0λ≥ 0. The final weights satisfy ∑i∈Igpi|gwi=1. _i∈ I_gp_i gw_i=1. The cap applies before the second normalization, so the final wiw_i need not be bounded by C. Setting λ=0λ=0 gives wi=1w_i=1 for all sampled decision points. Student-only and teacher-only references correspond to η=1η=1 and η=0η=0, respectively. Conditional on the cached weights, define the reweighted empirical distribution pw(i∣g)=pi|gwi.p_w(i g)=p_i gw_i. Equation (10) ensures that pw(⋅∣g)p_w(· g) is a probability distribution. Let ℓθ(i) _θ(i) denote the per-decision regression loss averaged over fresh student queries conditional on the stored history. The expected replay loss can then be written as τ∼q[1|ℬτ|∑i∈ℬτwiℓθ(i)]=∑gp(g)i∼pw(⋅∣g)[ℓθ(i)].E_τ q [ 1|B_τ| _i _τw_i _θ(i) ]= _gp(g)\,E_i p_w(· g)[ _θ(i)]. Thus Equation (5) defines regression under a reweighted empirical distribution over sampled decision points while preserving the sampling probabilities of the collection conditions. Because the sampled indices depend on the available trajectory length and the cumulative scores omit unsampled decisions, Equation (8) does not directly give the density ratio for this replay measure. For numerical stability, the reference-mixture term is evaluated with log-sum-exp: log(ηeΦθ0+(1−η)eΦT)=LSE(logη+Φθ0,log(1−η)+ΦT), \! (η e _ _0+(1-η)e _T )=LSE\! ( η+ _ _0, (1-η)+ _T ), (11) where Φ abbreviates the corresponding prefix score Φν(i) _ν(i). At a single-source endpoint, only the corresponding term is retained. Before exponentiation within condition g, subtract maxi∈IgSi _i∈ I_gS_i; this common factor cancels in Equation (10). Fixed-reference scores are cached once, while snapshot scores and weights are refreshed every R updates. A.4 Scope of the Distillation Update At update a, let θa _a denote the student parameters used to generate fresh queries, and let pw(a)p_w^(a) denote the empirical distribution induced by the cached weights. Holding the query-generating policy and cached weights fixed, define the supervised replay surrogate ℛa(ϑ)= _a( )= ∑gp(g)i∼pw(a)(⋅∣g)s∼ps∼qθa(⋅∣hi,s)[‖ϑ(,s∣hi)−T(ji)(,s∣hi)‖22]. _gp(g)\,E_i p_w^(a)(· g)E_ subarraycs p_s\\ x q_ _a(· h_i,s) subarray [ \|v_ (x,s h_i)-v_T^(j_i)(x,s h_i) \|_2^2 ]. With the query-generating student and cached weights fixed at update a, the detached loss in Equation (5) yields a stochastic gradient of ℛaR_a evaluated at ϑ=θa = _a. Gradients propagate only through the student velocity prediction. The ideal history-ratio identity motivates the prefix-score structure, while the implemented weights define the empirical replay objective in Section A.3. These weights serve as compatibility-based replay proxies and are not assumed to be exact online importance ratios. The resulting update combines fresh student-generated denoising queries with reweighted stored histories, without additional environment interaction during distillation. Appendix B Distillation Procedure The trajectory-based objective in Equation (5) gives the following distillation procedure, alternating between history-weight refreshes and student updates. 1. Collect the pool. Initialize the student and task-specific teachers as described in Section 4.1. Collect trajectories with the initial student and each task teacher, retaining ordered histories, denoising paths, executed action lengths, and task and collection-condition labels. 2. Cache reference scores. Freeze the teachers and initial-student reference. Evaluate both references on every stored path using Equation (3) and cache the prefix scores accumulated over preceding sampled decision points. 3. Refresh history weights. At initialization and every R updates, copy the student to the scoring snapshot. Recompute its prefix scores and obtain history weights from Equations (4) and (10). 4. Generate teacher queries. Sample a trajectory from the fixed pool. At each sampled decision point, use its stored history to generate a fresh denoising path with the current student, select a low-noise state, and query the corresponding task teacher at the same state and noise level. 5. Update the student. Detach query states, teacher targets, and history weights. Average the weighted action-velocity losses over the sampled decision points of the trajectory and update the student. Repeat until the optimization budget is reached. All denoising queries and weight refreshes use stored histories. Newly sampled actions are not executed in the environment, and stored behavior actions are used for path scoring rather than as regression targets. Loss reduction. For an action-velocity output with DaD_a scalar coordinates, the mean squared error is ‖θ−T‖22/Da\|v_θ-v_T\|_2^2/D_a. With the action-loss coefficient of 10, the optimized loss is therefore (10/Da)ℒ^WAM-OPD(10/D_a) L_WAM -OPD. The executed-coordinate projection PuP_u is used only in the path score, not in this action-field regression. Appendix C Evaluation Protocol and Interaction Cost Simulation. For task j, success rate is 100Njsucc/Nj100N_j^succ/N_j, where NjsuccN_j^succ and NjN_j denote successful and total trials under the evaluation setting in Section 4.1. Target and retention averages assign equal weight to their two and six constituent tasks, respectively. Improvements are measured in percentage points before rounding. All compared policies share evaluation scenes, task instructions, and inference-noise seeds. Ablation results are computed over 200 cases per task. Real-world evaluation. Each method is evaluated on 50 paired layouts per task, disjoint from the training layouts. Methods are tested on the same layouts in interleaved order. A pick-and-place trial succeeds when the object reaches the designated region and remains stable after release; a button-press trial succeeds when the button is activated. Time limits, target regions, and stability criteria are fixed across methods. Interaction cost. Environment interaction comprises teacher-data collection, fixed-pool collection, and closed-loop evaluation. Distillation itself requires no additional environment interaction: both student queries and teacher supervision are computed from stored histories. This accounting separates the one-time collection cost from the interaction-free optimization stage; teacher inference and history-weight refreshes still incur computation. Appendix D Additional Experimental Details D.1 Simulation teachers and trajectory pool The Stapler and Microwave teachers are obtained after 32,000 and 16,000 SFT steps, respectively. The Microwave demonstration dataset contains 2,492 episodes, comprising 492 clean and 2,000 randomized demonstrations, with a 1% validation split. Teacher-training demonstrations are separate from the rollout pool used for distillation. Table 3 gives the fixed-pool composition. Table 3: Composition of the fixed simulation trajectory pool. Task Source Clean Randomized Decision records Stapler Initial student 8 32 443 Stapler Teacher 4 16 166 Microwave Initial student 8 32 1,802 Microwave Teacher 4 16 780 Total 24 96 3,191 D.2 Distillation hyperparameters Table 4 summarizes the optimization, path-scoring, and prefix-weighting hyperparameters. Student queries use the current model parameters. Reference scores are cached, and the student snapshot and history weights are refreshed at updates 0, 24, 48, and 72. Path residuals are computed in FP32 with FP64 reductions. Table 4: Hyperparameters for simulation distillation. Parameter Value Trainable parameters All student MoT parameters Optimizer AdamW Learning rate 10−610^-6 Adam coefficients (β1,β2)( _1, _2) (0.9,0.95)(0.9,0.95) Weight decay 0 Maximum gradient norm 1.0 Training precision BF16 Optimization steps 96 Temporal subsampling Equally spaced, n=16n=16 Action / video loss coefficients 10 / 0 Reweighting strength λ 0.1 Scoring scale κ 0.025 Minimum scoring scale σmin _ 10−410^-4 Clipping threshold C 2 Weight refresh interval R 24 Denoising steps 10 Prediction / execution horizon 32 / up to 24 The clipping threshold applies before the second normalization in Equation (10); the final normalized weights may exceed C. Baselines. The task-specific SFT teachers, π (Chen et al., 2025a), and STEAM (Liu et al., 2026) share the Fast-WAM initialization used in the simulation experiments. All policies use the inference horizon and denoising schedule in Table 4. D.3 Real-robot observation and action interface The platform consists of two UR5e arms and three RGB cameras. The head camera captures the workspace, while the two wrist cameras provide local views of the grippers and manipulated objects. The action vector has 14 dimensions: six target joint positions and one gripper command for the left arm, followed by the corresponding seven values for the right arm. Actions specify absolute joint targets, with gripper values scaled by 1/10001/1000. Proprioception contains measured joint positions and gripper states in the same order. Shared dataset statistics are used to normalize observations and actions across tasks. Appendix E Limitations WAM-OPD’s performance depends on the coverage of the fixed trajectory pool and the accuracy of the history-weight approximation. Broader coverage of task-relevant histories and more accurate history weighting are directions for future work.