Paper deep dive
Scope-WM: Scoped Computation for Efficient Visual World Models
Chunzheng Li, Zesheng Jia, Hongda Zhang, Jiaying Tang, Yuntian Wang, Siao Liu, Jin Wang
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Visual world models enable robotic planning by predicting future observations, but dense latent-state propagation and sample-intensive trajectory optimization incur high inference latency and peak memory usage, limiting real-time deployment on resource-constrained platforms. Existing sparse world-model acceleration methods either rely on unguided token sparsification, which may discard planning-relevant information and restrict achievable sparsity, or introduce heavy auxiliary modules and cumbersome multi-stage training pipelines. In this work, we present Scope-WM, an efficient visual world model that scopes computation to prediction-relevant latent regions and promising action sequences. Scope-WM distills prediction relevance into a lightweight action-conditioned selector and applies full dynamics prediction only to a compact subset of selected tokens. It updates the remaining tokens using a compact summary of foreground states and their changes, allowing the background to perceive foreground dynamics without costly token-to-token interactions. During planning, Scope-WM preserves and reuses high-quality action sequences discovered during the initial MPC search, focusing subsequent search under reduced rollout budgets. The resulting pipeline requires only a one-off selector distillation followed by a single joint training stage for the sparse world model. On the challenging Push-T task, Scope-WM reduces peak GPU memory usage and planning time to $18.1\%$ and $14.3\%$ of those of dense DINO-WM, respectively, corresponding to a $6.97\times$ planning speedup, while maintaining competitive task performance. Further evaluations across five diverse visual planning tasks demonstrate the general applicability of Scope-WM. Code is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2609.33218v1
- Canonical: https://arxiv.org/abs/2609.33218v1
Trouble viewing inline? Open PDF directly →
Full Text
65,368 characters extracted from source content.
Expand or collapse full text
Scope-WM: Scoped Computation for Efficient Visual World Models Chunzheng Li Zesheng Jia Hongda Zhang Jiaying Tang Yuntian Wang Siao Liu Jin Wang Abstract Visual world models enable robotic planning by predicting future observations, but dense latent-state propagation and sample-intensive trajectory optimization incur high inference latency and peak memory usage, limiting real-time deployment on resource-constrained platforms. Existing sparse world-model acceleration methods either rely on unguided token sparsification, which may discard planning-relevant information and restrict achievable sparsity, or introduce heavy auxiliary modules and cumbersome multi-stage training pipelines. In this work, we present Scope-WM, an efficient visual world model that scopes computation to prediction-relevant latent regions and promising action sequences. Scope-WM distills prediction relevance into a lightweight action-conditioned selector and applies full dynamics prediction only to a compact subset of selected tokens. It updates the remaining tokens using a compact summary of foreground states and their changes, allowing the background to perceive foreground dynamics without costly token-to-token interactions. During planning, Scope-WM preserves and reuses high-quality action sequences discovered during the initial MPC search, focusing subsequent search under reduced rollout budgets. The resulting pipeline requires only a one-off selector distillation followed by a single joint training stage for the sparse world model. On the challenging Push-T task, Scope-WM reduces peak GPU memory usage and planning time to 18.1%18.1\% and 14.3%14.3\% of those of dense DINO-WM, respectively, corresponding to a 6.97×6.97× planning speedup, while maintaining competitive task performance. Further evaluations across five diverse visual planning tasks demonstrate the general applicability of Scope-WM. Code is available at https://github.com/ChunZheng2022/Scope-WM. 1Soochow University 2Fudan University Introduction Visual model-based planning provides a general framework for robotic manipulation and control by predicting the consequences of candidate actions before they are executed in the physical environment (Finn and Levine 2017; Ebert et al. 2018; Hafner et al. 2019). Rather than learning only a reactive policy, a visual world model predicts future latent states under candidate action sequences and selects actions through model-predictive control (MPC) (Ha and Schmidhuber 2018; Chua et al. 2018; Hansen et al. 2022). Recent approaches based on self-supervised visual representations, most notably DINO-WM, further enable task-agnostic visual planning by predicting patch-level features extracted from a frozen DINOv2 encoder (Oquab et al. 2024; Zhou et al. 2025). However, this patch-level formulation is computationally demanding. At every MPC step, the model repeatedly propagates all visual tokens over long prediction horizons for many action sequences sampled by the cross-entropy method (CEM) (de Boer et al. 2005). Consequently, the cost of latent imagination grows with both the number of visual tokens and the rollout population, resulting in high planning latency and peak memory usage that hinder real-time deployment. Figure 1: Performance–efficiency trade-off of Scope-WM on Push-T. (a) Different token budgets yield different performance–efficiency operating points under varying sparsity. (b) Scope-WM substantially reduces planning time and peak GPU memory over dense DINO-WM. A natural approach to reducing this cost is to make latent imagination sparse by applying full dynamics prediction to only a compact subset of visual tokens. Existing sparse world-model acceleration methods either rely on unguided token sparsification, which may discard planning-relevant information and limit the sparsity that can be applied safely, or introduce specialized auxiliary modules and cumbersome multi-stage training pipelines (Chun et al. 2025; Yin et al. 2026). More fundamentally, efficient sparse imagination involves more than deciding how many tokens to retain. It must determine which tokens warrant accurate dynamics prediction, how the remaining tokens should evolve without eliminating the computational savings, and how cheaper latent rollouts can be translated into more efficient trajectory optimization. These decisions are tightly coupled: inaccurate token allocation can degrade future-state prediction, expensive background modeling can offset the savings from sparsity, and sample-intensive planning can remain a dominant bottleneck even when individual rollouts become cheaper. In this work, we present Scope-WM, an efficient visual world model that scopes computation to prediction-relevant latent regions and promising action sequences. Scope-WM allocates computation at three complementary levels. At the token level, it identifies a compact subset of tokens that require full dynamics prediction. At the dynamics level, it updates the remaining tokens using lightweight foreground-conditioned computation rather than dense token-to-token interactions. At the planning level, it concentrates subsequent trajectory search around high-quality action sequences discovered during the initial MPC search. As illustrated in Figure 1, this scoped design provides flexible operating points across different compute budgets: a moderate sparse configuration can improve efficiency while preserving or even improving task performance, whereas a more aggressive configuration prioritizes latency and memory reduction while retaining competitive control performance. Concretely, these three levels are realized by Distilled Relevance Selection (DRS), Foreground-Delta Background Update (FDBU), and Elite-Bank CEM (EB-CEM), respectively. The resulting pipeline requires only a one-off selector distillation followed by a single joint training stage for the sparse dynamics predictor and FDBU. Evaluations demonstrate that Scope-WM substantially improves planning efficiency while maintaining competitive task performance. Our main contributions are summarized as follows: • We formulate efficient visual world-model planning as a three-level computation-allocation problem spanning prediction-relevant token selection, foreground-aware background updating, and trajectory-search budgeting. • We propose DRS, a lightweight action-conditioned selector that distills prediction relevance from a full-token teacher to focus full dynamics prediction on a compact token subset. DRS requires only one-off distillation, followed by joint training of the sparse predictor and FDBU. • We propose FDBU, which summarizes foreground states and deltas to perform efficient token-wise background updates without dense foreground–background interactions. • We identify the path-dependent behavior of low-budget CEM planning and propose EB-CEM, which front-loads the sampling budget and persistently reuses high-quality initial action sequences across MPC steps. Together, these components reduce both per-trajectory imagination cost and subsequent trajectory-search cost while preserving competitive control performance. Related Work Visual world models and pretrained representations. Visual model-based control initially predicted future observations in pixel space and optimized action sequences through MPC (Finn and Levine 2017; Ebert et al. 2018). Subsequent methods moved prediction into compact latent spaces: World Models, PlaNet, and Dreamer learn recurrent latent dynamics for control through imagined trajectories, while PETS and TD-MPC combine learned dynamics with sampling-based or temporally abstract planning objectives (Ha and Schmidhuber 2018; Hafner et al. 2019; Hafner et al. 2020; Hafner et al. 2023; Chua et al. 2018; Hansen et al. 2022; Hansen et al. 2024). Transformer- and diffusion-based approaches, including IRIS, Transformer-based World Models, STORM, Masked World Models, and DIAMOND, further improve latent sequence modeling and visual prediction (Micheli et al. 2023; Robine et al. 2023; Zhang et al. 2023; Seo et al. 2022; Alonso et al. 2024). In parallel, pretrained representations such as R3M, MVP, and VIP provide reusable features or goal-conditioned reward signals for control, while DINO, DINOv2, I-JEPA, and V-JEPA learn general spatial or temporal visual representations through self-supervision (Nair et al. 2022; Xiao et al. 2022; Radosavovic et al. 2022; Ma et al. 2023; Caron et al. 2021; Oquab et al. 2024; Assran et al. 2023; Bardes et al. 2024; Assran et al. 2025). DINO-WM combines these directions by predicting future patch-level DINOv2 features and planning directly in the resulting feature space (Zhou et al. 2025). More recently, LeWorldModel learns compact global latent representations end-to-end from pixels, reducing planning cost by changing the representation and training paradigm itself (Maes et al. 2026). Scope-WM instead follows the pretrained patch-level formulation of DINO-WM and improves its efficiency by selectively allocating computation within spatially structured latent rollouts and trajectory search. Sparse visual computation and imagination. Efficient Vision Transformers exploit token redundancy through learned pruning, adaptive computation, token reorganization, or token merging to reduce inference cost (Rao et al. 2021; Xu et al. 2022; Yao et al. 2026; Liang et al. 2022; Yin et al. 2022; Bolya et al. 2023). These methods are primarily designed for static recognition, whereas visual world-model planning requires state- and action-dependent token selection whose errors may accumulate over multi-step rollouts. Sparse Imagination applies random patch dropping and randomized grouped attention to DINO-style world models, demonstrating that moderate sparsity can preserve planning performance while avoiding the spatial bias of fixed importance scores (Chun et al. 2025). However, random selection remains independent of the current prediction objective. DDP-WM instead decomposes latent dynamics into sparse primary dynamics and context-driven background updates, using a dynamic localization network for foreground identification and foreground-conditioned cross-attention for background correction (Yin et al. 2026). Scope-WM shares the goal of separating expensive foreground prediction from inexpensive background maintenance, but adopts lighter mechanisms: DRS distills prediction relevance into a compact token-wise selector, while FDBU replaces pairwise foreground–background interactions with a pooled foreground-state and foreground-delta context. After selector distillation, the sparse predictor and background updater can therefore be optimized in a unified training stage. Sampling-based planning. CEM is widely used with learned dynamics models because it can optimize action sequences without differentiating through the planning objective, but its cost scales directly with the number of candidate rollouts (de Boer et al. 2005; Chua et al. 2018). iCEM improves sample efficiency through temporally correlated noise, warm starts, and elite reuse across optimization iterations and planning steps (Pinneri et al. 2020). EB-CEM is complementary to these techniques and focuses on the allocation of samples across receding-horizon MPC stages. It assigns a larger population to the initial broad search, stores high-quality early trajectories in a persistent bank, and samples perturbed variants during later low-budget replanning. Method We build Scope-WM on top of DINO-WM (Zhou et al. 2025) and describe its token-, dynamics-, and planning-level components in turn, as illustrated in Figure 2. Figure 2: Overview of Scope-WM. Scope-WM scopes computation at the token, dynamics, and planning levels. At the token level, the action-conditioned DRS selector identifies a compact set of prediction-relevant foreground tokens from the encoded visual state. At the dynamics level, the sparse world-model predictor applies full dynamics prediction to the selected tokens, while FDBU uses a summarized foreground-dynamics context to update the remaining background tokens through a lightweight gated residual update, producing the complete latent rollout. At the planning level, EB-CEM performs test-time action optimization by retaining and reusing high-quality elite action sequences across MPC steps. Candidate action sequences are evaluated by a planning objective that compares the predicted latent rollout with the goal latent encoded by the frozen visual encoder. Given an observation oto_t, the frozen DINOv2 encoder extracts a set of N visual patch tokens Xt=xt,ii=1N,xt,i∈ℝdx.X_t=\x_t,i\_i=1^N, x_t,i ^d_x. (1) The visual tokens are combined with proprioceptive state ptp_t and action embedding e(at)e(a_t) to construct world-model tokens zt,i=Fuse(xt,i,pt,e(at))∈ℝdz.z_t,i=Fuse(x_t,i,p_t,e(a_t)) ^d_z. (2) Given a history length H, the dense world model predicts future latent tokens conditioned on the observed history as Z^2:H+1=fθ(Z1:H), Z_2:H+1=f_θ(Z_1:H), (3) where Zt=zt,ii=1NZ_t=\z_t,i\_i=1^N. The dense predictor operates on all HNHN tokens, which is expensive during MPC because CEM requires evaluating many candidate action sequences per step. Our method reduces this cost through three components: Distilled Relevance Selection, Foreground-Delta Background Update, and Elite-Bank CEM. Distilled Relevance Selection A central challenge in sparse latent imagination is deciding which tokens should be kept. Prior work has observed that static attention-based token selection is often insufficient for visual planning (Chun et al. 2025). This is because prediction-relevant regions are not fixed across successive planning steps: the relevant tokens depend on the current state, the planned action, and the prediction objective. A purely visual-difference-based selector can also be problematic. Although frame-level changes can highlight moving regions, they may ignore tokens that are dynamically important but visually stable, such as contacts, obstacles, or structures that constrain future motion. Therefore, we use a dynamic, prediction-aware selection mechanism. Teacher relevance target. Starting from a trained dense DINO-WM teacher, we compute its latent prediction loss: ℒT=1|Ω|∑(t,i)∈Ω‖z^t,iT−zt,i⋆‖22,L_T= 1| | _(t,i)∈ \| z^T_t,i-z _t,i \|_2^2, (4) where z^t,iT z^T_t,i is the teacher prediction, zt,i⋆z _t,i is the target latent token, and Ω denotes the supervised token set. We then estimate the contribution of each visual token to the teacher prediction loss using a gradient-times-input score: r~t,i=‖xt,i⊙∂ℒT∂xt,i‖2. r_t,i= \|x_t,i _T∂ x_t,i \|_2. (5) The score is normalized within each frame: qt,i=r~t,i+ϵ∑j=1N(r~t,j+ϵ).q_t,i= r_t,i+ε _j=1^N( r_t,j+ε). (6) The resulting distribution qtq_t serves as a soft relevance target. Unlike static attention masks or visual-change heuristics, this target directly reflects which tokens affect the world model’s prediction error under the current objective. Lightweight relevance head. We train a compact relevance head gϕg_φ to predict token relevance scores. For each token, the visual feature and control context are jointly projected into a shared hidden space: ht,i=ϕx(xt,i)+ϕp(pt)+ϕa(e(at)),st,i=gϕ(ht,i),h_t,i= _x(x_t,i)+ _p(p_t)+ _a(e(a_t)), s_t,i=g_φ(h_t,i), (7) where ϕx _x, ϕp _p, and ϕa _a are lightweight linear projections, and gϕg_φ is a small MLP producing the relevance logit. The predicted relevance distribution across tokens is obtained by a temperature-scaled softmax: πt,i=exp(st,i/T)∑j=1Nexp(st,j/T), _t,i= (s_t,i/T) _j=1^N (s_t,j/T), (8) where T is the distillation temperature. The DRS head is trained by minimizing the KL divergence from the corresponding teacher relevance target: ℒDRS=DKL(qt∥πt)=∑i=1Nqt,ilogqt,iπt,i.L_DRS=D_KL(q_t\,\|\, _t)= _i=1^Nq_t,i q_t,i _t,i. (9) After distillation, the selector is fixed during sparse world-model training. At inference time, DRS selects the top-K tokens according to their predicted relevance: mt,i=[st,i∈TopK(st,jj=1N,K)],m_t,i=I [s_t,i (\s_t,j\_j=1^N,K ) ], (10) where mt,i∈0,1m_t,i∈\0,1\ is the foreground mask. The resulting foreground and background token sets are ℱt=i∣mt,i=1,ℬt=i∣mt,i=0.F_t=\i m_t,i=1\, _t=\i m_t,i=0\. (11) The selector is deliberately lightweight. Since frozen DINOv2 patch tokens are already globally contextualized through self-attention, DRS can exploit image-wide visual context without an additional ViT-style mask generator. It therefore uses only a small token-wise MLP conditioned on state and action, introducing negligible rollout overhead while producing state-, action-, and prediction-aware masks. Foreground-Delta Background Update After selecting foreground tokens, a natural approach is to run the world model only on these tokens and keep the background tokens unchanged. However, this may cause latent inconsistency: background tokens may still need small corrections due to object motion, contact, occlusion, or camera-frame changes. One possible solution is to use an attention-based background update module, but such a design increases computational cost and often requires staged training. We instead introduce a simpler module, Foreground-Delta Background Update, which is lightweight and can be trained jointly with the sparse world model. Let ZtF=zt,i∣i∈ℱtZ^F_t=\z_t,i i _t\ denote the selected foreground tokens. The primary sparse dynamics predictor is applied only to these tokens during latent rollout: Z^2:H+1F=fθ(Z1:HF). Z^F_2:H+1=f_θ(Z^F_1:H). (12) For a Transformer predictor, this reduces the dominant attention cost from O((HN)2)O((HN)^2) to O((HK)2)O((HK)^2), where K≪NK N. FDBU updates the remaining background tokens using the predicted foreground dynamics. For each predicted step, we compute the foreground delta: Δt+1,iF=z^t+1,iF−zt,iF,i∈ℱt. ^F_t+1,i= z^F_t+1,i-z^F_t,i, i _t. (13) We summarize the predicted foreground state and foreground motion separately by mean pooling: z¯t+1F=1K∑i∈ℱtz^t+1,iF,Δ¯t+1F=1K∑i∈ℱtΔt+1,iF. z^F_t+1= 1K _i _t z^F_t+1,i, ^F_t+1= 1K _i _t ^F_t+1,i. (14) The two summaries are fused into a foreground dynamics context to guide subsequent background updates: ct+1=ψc([z¯t+1F,Δ¯t+1F]),c_t+1= _c ([ z^F_t+1, ^F_t+1] ), (15) where ψc _c is a small projection network. For each full-grid token zt,iz_t,i, FDBU independently predicts a lightweight next-step residual update and scalar gate using the foreground dynamics context: ut+1,i=ψu([zt,i,ct+1]),γt+1,i=σ(ψg([zt,i,ct+1])),u_t+1,i= _u([z_t,i,c_t+1]), _t+1,i=σ( _g([z_t,i,c_t+1])), (16) where ψu _u and ψg _g are lightweight MLPs and σ(⋅)σ(·) is the sigmoid function. Only background tokens receive this update: z~t+1,iB=zt,i+α(1−mt,i)γt+1,iut+1,i, z^B_t+1,i=z_t,i+α(1-m_t,i) _t+1,iu_t+1,i, (17) where α controls the residual update scale. The final full-grid prediction is formed by scattering foreground predictions into the full grid and applying FDBU to the background: z~t+1,i=mt,iz^t+1,iF+(1−mt,i)z~t+1,iB. z_t+1,i=m_t,i z^F_t+1,i+(1-m_t,i) z^B_t+1,i. (18) The sparse world model is trained with a full-grid latent prediction loss over all latent positions: ℒWM=MSE(Z~,Z⋆),L_WM=MSE\! ( Z,Z ), (19) which jointly supervises foreground prediction and FDBU background updates during sparse-model training. FDBU is integrated into the world-model training pipeline and does not require a separate training stage. Elite-Bank CEM CEM-based MPC is sensitive to the number of sampled action sequences. Empirically, we observe that increasing the number of samples can significantly improve success rate in the low-sample regime, but the gain quickly saturates when the sample count becomes large. Moreover, later MPC steps often provide limited recovery ability: once early actions move the agent or object into an unfavorable state, subsequent replanning may not be able to recover. This suggests that, in this regime, improving the quality of early candidate action sequences is more important than simply increasing the number of samples at every step. Motivated by this observation, we introduce Elite-Bank CEM, a simple proposal-reuse mechanism for MPC. It leaves the planning objective and world model unchanged. Instead, it improves later CEM proposals by reusing high-quality action sequences found during the initial MPC search. Let Aj∈ℝL×daA_j ^L× d_a be a candidate action sequence of planning horizon L, where j=1,…,Mj=1,…,M. At CEM iteration ℓ , standard CEM samples candidates from a diagonal Gaussian: Aj(ℓ)∼(μ(ℓ),diag((σ(ℓ))2)).A_j^( ) (μ^( ),diag ((σ^( ))^2 ) ). (20) Each candidate is evaluated by the world-model rollout cost: Cj(ℓ)=Jθ(Aj(ℓ)),C_j^( )=J_θ(A_j^( )), (21) where lower cost indicates a better action sequence. The top KeliteK_elite candidates form the elite set: ℰ(ℓ)=TopKmin(Aj(ℓ)j=1M,Cj(ℓ)j=1M,Kelite).E^( )=TopK_min (\A_j^( )\_j=1^M,\C_j^( )\_j=1^M,K_elite ). (22) CEM then updates its Gaussian parameters: μ(ℓ+1)=1Kelite∑A∈ℰ(ℓ)A,μ^( +1)= 1K_elite _A ^( )A, (23) σ(ℓ+1)=Std(ℰ(ℓ)).σ^( +1)=Std (E^( ) ). (24) EB-CEM front-loads search by using a larger candidate population at the initial MPC step and a reduced population thereafter to emphasize early-stage exploration. Let MrM_r denote the candidate budget at MPC step r: Mr=Mfront,r=0,Mlater,r>0,Mfront>Mlater.M_r= casesM_front,&r=0,\\ M_later,&r>0, cases M_front>M_later. (25) We use Mfront=300M_front=300 and Mlater=100M_later=100 by default. After the initial MPC step, we store the best MbM_b candidates from its final CEM iteration in a persistent elite bank: ℬ=TopMmin(Ajj=1Mfront,Jθ(Aj)j=1Mfront,Mb).B=TopM_ (\A_j\_j=1^M_front,\J_θ(A_j)\_j=1^M_front,M_b ). (26) At each CEM iteration of subsequent MPC steps, a fraction η of the reduced population is sampled locally around the banked elites to refine promising solutions: Mlocal=⌊ηMlater⌋,Mglobal=Mlater−Mlocal.M_local= η M_later , M_global=M_later-M_local. (27) For a banked elite sequence B∈ℬB , a local candidate is generated by Gaussian perturbation: Alocal=B+λσ(ℓ)⊙ξ,ξ∼(0,I),A_local=B+λσ^( ) ξ, ξ (0,I), (28) where λ controls the perturbation scale. The remaining candidates are instead sampled globally: Aglobal∼(μ(ℓ),diag((σ(ℓ))2)).A_global \! (μ^( ),diag\! ((σ^( ))^2 ) ). (29) The final candidate set combines the local and global samples to balance local refinement and global exploration: =local∪global.A=A_local _global. (30) This mechanism encourages the planner to preserve and refine promising early plans, while still keeping global exploration through CEM samples. Therefore, EB-CEM is complementary to the sparse world model: DRS and FDBU reduce the cost of each rollout, while EB-CEM improves search quality by reallocating the rollout budget across MPC steps. Experiments Setup Environments and datasets. We evaluate Scope-WM on five visual planning tasks: PointMaze (Fu et al. 2020), Wall, Push-T (Chi et al. 2023), Rope (Zhang et al. 2024), and Granular, following the task definitions and offline datasets of DINO-WM (Zhou et al. 2025). These tasks cover navigation, rigid-body manipulation, and deformable or multi-body dynamics across diverse physical interaction regimes. Evaluation metrics. Following DINO-WM, all tasks use CEM-based MPC over latent world-model rollouts. We report success rate (SR, ↑ ) for PointMaze, Wall, and Push-T, and Chamfer Distance (CD, ↓ ) for Rope and Granular. For efficiency, we report three complementary metrics: transition-model FLOPs, planning time, and peak GPU memory usage. Planning time is the cumulative CEM sub-planner time across MPC steps, while peak memory is the maximum CUDA memory allocated during each evaluation run. Baseline. We use dense DINO-WM (Zhou et al. 2025) as the baseline for direct comparison. Both methods share the encoder, data, evaluation pairs, planning objective, horizon, and action constraints unless explicitly noted. Implementation details. PointMaze, Wall, and Push-T are trained using eight NVIDIA V100 32G GPUs, with planning evaluated on a single NVIDIA V100 GPU. Rope and Granular are trained and evaluated on a single NVIDIA RTX 2080 Ti GPU due to PyFlex software compatibility. Additional implementation details for training and planning are included in the supplementary material. Method Full-dyn. tokens SR (↑ ) FLOPs (G, ↓ ) Plan time (s, ↓ ) Speedup (↑ ) Peak Mem. (MB, ↓ ) Mem. Ratio (↓ ) DINO-WM (Dense) 196 90% 31.89 240.96 1.00×1.00× 18436 100.0% Scope-WM (Fast) 98 94% 15.04 93.06 2.59×2.59× 7088 38.4% Scope-WM (Faster) 32 88% 5.27 34.56 6.97×6.97× 3334 18.1% Table 1: End-to-end performance–efficiency trade-off on Push-T. Fast and Faster apply full dynamics prediction to 9898 and 3232 of 196196 visual tokens. FLOPs are reported per prediction sample, with memory ratios normalized to DINO-WM. Method PointMaze (SR ↑ ) Push-T (SR ↑ ) Wall (SR ↑ ) Rope (CD ↓ ) Granular (CD ↓ ) IRIS 74% 32% 4% 1.11 0.37 DreamerV3 100% 30% 100% 2.49 1.05 Sparse Imagination 100% 78.3% 95% – – DINO-WM 98% 90% 96% 0.41 0.26 Scope-WM (Fast, K=98K=98) 100% 94% 92% 0.87 0.22 Scope-WM (Faster, K=32K=32) 98% 88% 92% 0.71 0.25 Table 2: Cross-task planning performance. SR (↑ ) is reported for PointMaze, Push-T, and Wall, and CD (↓ ) for Rope and Granular. Fast and Faster use 9898 and 3232 full-dynamics tokens. Main Results We first evaluate the end-to-end performance and efficiency of Scope-WM on Push-T, and then examine its planning performance across all five visual planning tasks. End-to-end performance and efficiency. Table 1 compares dense DINO-WM with two Scope-WM operating points on Push-T. Fast applies full dynamics prediction to 9898 of 196196 visual tokens, improving SR from 90%90\% to 94%94\% while reducing per-sample FLOPs from 31.8931.89 to 15.0415.04 G. Its planning time and peak GPU memory decrease to 38.6%38.6\% and 38.4%38.4\% of the dense baseline, respectively, yielding a 2.59×2.59× planning speedup. Faster further reduces full dynamics prediction to 3232 tokens and requires only 5.275.27 G FLOPs per sample, a 6.05×6.05× reduction over dense DINO-WM. It retains 88%88\% SR while reducing planning time and peak memory to 14.3%14.3\% and 18.1%18.1\%, respectively, corresponding to a 6.97×6.97× planning speedup. These results show that simply adjusting the number of selected top-ranked tokens allows Scope-WM to either improve both planning performance and efficiency, or achieve substantially greater acceleration while retaining competitive task performance. Planning performance across tasks. Table 2 compares Scope-WM with prior methods across five tasks. Under the aggressive K=32K=32 setting, Scope-WM achieves 98%98\%, 88%88\%, and 92%92\% SR on PointMaze, Push-T, and Wall, matching DINO-WM on PointMaze and remaining within 22 and 44 percentage points on Push-T and Wall, respectively. It achieves a CD of 0.250.25 on Granular, slightly improving over DINO-WM at 0.260.26. With the larger K=98K=98 budget, Fast reaches 100%100\% on PointMaze, 94%94\% on Push-T, and 0.220.22 CD on Granular. Rope remains the most challenging task for the sparse formulation, where the best Scope-WM result is 0.710.71 CD, compared with 0.410.41 for dense DINO-WM. Overall, Scope-WM preserves strong planning performance on most tasks while substantially reducing latent-rollout computation. Ablation Study We conduct ablation studies on Push-T to examine the contribution of the proposed designs and how they interact across model prediction and trajectory optimization. Component ablation. Table 3 first isolates the contribution of the three components under the aggressive 32-token setting. Replacing DRS with random token selection reduces the success rate from 88%88\% to 76%76\%, showing the importance of allocating full dynamics prediction to relevant regions. Removing FDBU and directly carrying background tokens forward reduces the success rate to 56%56\%, indicating that unselected tokens still require lightweight adaptation to foreground dynamics. Replacing EB-CEM with standard CEM decreases the success rate to 82%82\%. The complete Scope-WM achieves the highest success rate, showing that the three designs provide complementary benefits. Variant Selection BG update Planner SR (↑ ) w/o DRS Random FDBU EB-CEM 76% w/o FDBU DRS Copy EB-CEM 56% w/o EB-CEM DRS FDBU CEM 82% Scope-WM DRS FDBU EB-CEM 88% Table 3: Component ablation on Push-T under the 3232-token full-dynamics setting used by Scope-WM (Faster). Effect of token budget. We vary the full-dynamics token budget on Push-T to characterize the performance–efficiency trade-off. As shown in Figure 3, both planning time and peak GPU memory decrease consistently as the token budget is reduced, while SR remains between 88%88\% and 94%94\%. The K=98K=98 setting achieves the highest SR of 94%94\% with only 38.6%38.6\% of the dense planning time and 38.4%38.4\% of its peak memory. Even at K=32K=32, Scope-WM retains 88%88\% SR while reducing these costs to 14.3%14.3\% and 18.1%18.1\%, respectively. Figure 3: Effect of the full-dynamics token budget on Push-T. Bars show planning time and peak GPU memory normalized to dense DINO-WM, with absolute values annotated above the bars. The line shows success rate as the token budget varies across configurations. Figure 4: Qualitative analysis of Scope-WM on Push-T. (a) DRS focuses computation on prediction-relevant regions. (b) FDBU captures background evolution without dense token-to-token interactions. (c) EB-CEM reuses promising early trajectories for subsequent replanning. Selection Background update Copy FDBU Random 72% 76% DRS 56% 88% Table 4: Interaction between DRS and FDBU on Push-T under the same 32-token budget and EB-CEM planner. Entries report success rate (SR, ↑ ). Interaction between DRS and FDBU. DRS determines which tokens receive full dynamics prediction, while FDBU propagates foreground dynamics to the remaining latent state. Table 4 reveals an interaction between the two mechanisms. With random selection, FDBU provides a modest improvement from 72% to 76% SR. In contrast, under DRS, replacing background copying with FDBU improves SR substantially from 56% to 88%. This suggests that concentrating full dynamics prediction on prediction-relevant tokens makes effective background propagation particularly important, revealing a strong complementarity between DRS and FDBU. Model Planner Samples SR (↑ ) Plan time (s, ↓ ) DINO-WM CEM 100/100 90% 240.96 DINO-WM EB-CEM 300/100 84% 267.36 Scope-WM CEM 100/100 82% 30.67 Scope-WM CEM 200/200 88% 60.96 Scope-WM EB-CEM 300/100 88% 34.56 Table 5: Interaction between sparse prediction and planning strategy on Push-T. Scope-WM uses the 32-token setting; sample budgets denote initial/subsequent MPC steps. CEM EB-CEM Early samples 100 100 300 300 Later samples 100 300 100 100 SR (↑ ) 82% 82% 86% 88% Table 6: Effect of sample allocation and elite reuse on Push-T. Early and later denote initial and subsequent MPC steps. Interaction between sparse prediction and EB-CEM. Sparse prediction and EB-CEM act on complementary sources of planning cost: sparse prediction reduces the cost of each imagined rollout, whereas EB-CEM reallocates search effort across MPC steps. As shown in Table 5, applying EB-CEM to dense DINO-WM provides no benefit, decreasing SR from 90%90\% to 84%84\% while increasing planning time from 240.96240.96 to 267.36267.36 s. For Scope-WM, standard CEM with 100 samples requires only 30.6730.67 s but reaches 82%82\% SR. Increasing the budget to 200 samples recovers 88%88\% SR at a cost of 60.9660.96 s, whereas EB-CEM achieves the same 88%88\% SR in only 34.5634.56 s. Thus, cheap sparse rollouts make front-loaded exploration affordable, while elite reuse avoids maintaining a large sampling budget throughout subsequent replanning. Qualitative analysis. Figure 4 provides further intuition behind the three designs under the aggressive 32-token setting. Compared with random selection, DRS focuses full dynamics prediction on prediction-relevant regions across the latent field (a). FDBU achieves background prediction errors comparable to dense updating using only the foreground dynamics context, suggesting that dense token-to-token interactions are unnecessary for background propagation (b). Finally, EB-CEM retains promising early trajectories to guide subsequent replanning (c). Table 6 further confirms the benefits of front-loaded sampling and elite reuse. Conclusion We present Scope-WM, an efficient visual world-model planning framework that scopes computation across token selection, latent dynamics, and trajectory optimization. Scope-WM combines DRS for prediction-aware token selection, FDBU for lightweight background updates, and EB-CEM for efficient trajectory optimization. Together, these components reduce both the cost of individual imagined rollouts and the rollout budget required during subsequent replanning, while retaining a simple training pipeline consisting of one-off selector distillation followed by joint sparse world-model training. Experiments demonstrate a favorable performance–efficiency trade-off while maintaining competitive planning performance. On Push-T, the aggressive configuration achieves a 6.97×6.97× planning speedup while reducing planning time and peak GPU memory to 14.3%14.3\% and 18.1%18.1\% of dense DINO-WM, respectively, with only a modest change in task performance. These results highlight computation allocation according to prediction and planning relevance as a promising principle for efficient visual world-model planning under flexible token and rollout budgets. References Alonso et al. (2024) E. Alonso, A. Jelley, V. Micheli, A. Kanervisto, A. J. Storkey, T. Pearce, and F. Fleuret Diffusion for world modeling: visual details matter in atari. In Advances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, A. Globersons, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. M. Tomczak, and C. Zhang (Eds.), External Links: Link Cited by: Visual world models and pretrained representations.. Assran et al. (2023) M. Assran, Q. Duval, I. Misra, P. Bojanowski, P. Vincent, M. G. Rabbat, Y. LeCun, and N. Ballas Self-supervised learning from images with a joint-embedding predictive architecture. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, p. 15619–15629. External Links: Link, Document Cited by: Visual world models and pretrained representations.. Assran et al. (2025) M. Assran, A. Bardes, D. Fan, Q. Garrido, R. Howes, M. Komeili, M. J. Muckley, A. Rizvi, C. Roberts, K. Sinha, A. Zholus, S. Arnaud, A. Gejji, A. Martin, F. R. Hogan, D. Dugas, P. Bojanowski, V. Khalidov, P. Labatut, F. Massa, M. Szafraniec, K. Krishnakumar, Y. Li, X. Ma, S. Chandar, F. Meier, Y. LeCun, M. Rabbat, and N. Ballas V-JEPA 2: self-supervised video models enable understanding, prediction and planning. CoRR abs/2506.09985. External Links: Link, Document, 2506.09985 Cited by: Visual world models and pretrained representations.. Bardes et al. (2024) A. Bardes, Q. Garrido, J. Ponce, X. Chen, M. Rabbat, Y. LeCun, M. Assran, and N. Ballas Revisiting feature prediction for learning visual representations from video. Trans. Mach. Learn. Res. 2024. External Links: Link Cited by: Visual world models and pretrained representations.. Bolya et al. (2023) D. Bolya, C. Fu, X. Dai, P. Zhang, C. Feichtenhofer, and J. Hoffman Token merging: your vit but faster. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, External Links: Link Cited by: Sparse visual computation and imagination.. Caron et al. (2021) M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin Emerging properties in self-supervised vision transformers. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, p. 9630–9640. External Links: Link, Document Cited by: Visual world models and pretrained representations.. Chi et al. (2023) C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song Diffusion policy: visuomotor policy learning via action diffusion. In Robotics: Science and Systems XIX, Daegu, Republic of Korea, July 10-14, 2023, K. E. Bekris, K. Hauser, S. L. Herbert, and J. Yu (Eds.), External Links: Link, Document Cited by: Environments and datasets.. Chua et al. (2018) K. Chua, R. Calandra, R. McAllister, and S. Levine Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, S. Bengio, H. M. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.), p. 4759–4770. External Links: Link Cited by: Introduction, Visual world models and pretrained representations., Sampling-based planning.. Chun et al. (2025) J. Chun, Y. Jeong, and T. Kim Sparse imagination for efficient visual world model planning. CoRR abs/2506.01392. External Links: Link, Document, 2506.01392 Cited by: Introduction, Sparse visual computation and imagination., Distilled Relevance Selection. de Boer et al. (2005) P. de Boer, D. P. Kroese, S. Mannor, and R. Y. Rubinstein A tutorial on the cross-entropy method. Ann. Oper. Res. 134 (1), p. 19–67. External Links: Link, Document Cited by: Introduction, Sampling-based planning.. Ebert et al. (2018) F. Ebert, C. Finn, S. Dasari, A. Xie, A. X. Lee, and S. Levine Visual foresight: model-based deep reinforcement learning for vision-based robotic control. CoRR abs/1812.00568. External Links: Link, 1812.00568 Cited by: Introduction, Visual world models and pretrained representations.. Finn and Levine (2017) C. Finn and S. Levine Deep visual foresight for planning robot motion. In 2017 IEEE International Conference on Robotics and Automation, ICRA 2017, Singapore, Singapore, May 29 - June 3, 2017, p. 2786–2793. External Links: Link, Document Cited by: Introduction, Visual world models and pretrained representations.. Fu et al. (2020) J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine D4RL: datasets for deep data-driven reinforcement learning. CoRR abs/2004.07219. External Links: Link, 2004.07219 Cited by: Environments and datasets.. Ha and Schmidhuber (2018) D. Ha and J. Schmidhuber World models. CoRR abs/1803.10122. External Links: Link, 1803.10122 Cited by: Introduction, Visual world models and pretrained representations.. Hafner et al. (2020) D. Hafner, T. P. Lillicrap, J. Ba, and M. Norouzi Dream to control: learning behaviors by latent imagination. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, External Links: Link Cited by: Visual world models and pretrained representations.. Hafner et al. (2019) D. Hafner, T. P. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson Learning latent dynamics for planning from pixels. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, K. Chaudhuri and R. Salakhutdinov (Eds.), Proceedings of Machine Learning Research, Vol. 97, p. 2555–2565. External Links: Link Cited by: Introduction, Visual world models and pretrained representations.. Hafner et al. (2023) D. Hafner, J. Pasukonis, J. Ba, and T. P. Lillicrap Mastering diverse domains through world models. CoRR abs/2301.04104. External Links: Link, Document, 2301.04104 Cited by: Visual world models and pretrained representations.. Hansen et al. (2022) N. Hansen, H. Su, and X. Wang Temporal difference learning for model predictive control. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and S. Sabato (Eds.), Proceedings of Machine Learning Research, Vol. 162, p. 8387–8406. External Links: Link Cited by: Introduction, Visual world models and pretrained representations.. Hansen et al. (2024) N. Hansen, H. Su, and X. Wang TD-MPC2: scalable, robust world models for continuous control. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, External Links: Link Cited by: Visual world models and pretrained representations.. Liang et al. (2022) Y. Liang, C. Ge, Z. Tong, Y. Song, J. Wang, and P. Xie EViT: expediting vision transformers via token reorganizations. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, External Links: Link Cited by: Sparse visual computation and imagination.. Ma et al. (2023) Y. J. Ma, S. Sodhani, D. Jayaraman, O. Bastani, V. Kumar, and A. Zhang VIP: towards universal visual reward and representation via value-implicit pre-training. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, External Links: Link Cited by: Visual world models and pretrained representations.. Maes et al. (2026) L. Maes, Q. L. Lidec, D. Scieur, Y. LeCun, and R. Balestriero LeWorldModel: stable end-to-end joint-embedding predictive architecture from pixels. CoRR abs/2603.19312. External Links: Link, Document, 2603.19312 Cited by: Visual world models and pretrained representations.. Micheli et al. (2023) V. Micheli, E. Alonso, and F. Fleuret Transformers are sample-efficient world models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, External Links: Link Cited by: Visual world models and pretrained representations.. Nair et al. (2022) S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta R3M: A universal visual representation for robot manipulation. In Conference on Robot Learning, CoRL 2022, 14-18 December 2022, Auckland, New Zealand, K. Liu, D. Kulic, and J. Ichnowski (Eds.), Proceedings of Machine Learning Research, Vol. 205, p. 892–909. External Links: Link Cited by: Visual world models and pretrained representations.. Oquab et al. (2024) M. Oquab, T. Darcet, T. Moutakanni, H. V. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, M. Assran, N. Ballas, W. Galuba, R. Howes, P. Huang, S. Li, I. Misra, M. Rabbat, V. Sharma, G. Synnaeve, H. Xu, H. Jégou, J. Mairal, P. Labatut, A. Joulin, and P. Bojanowski DINOv2: learning robust visual features without supervision. Trans. Mach. Learn. Res. 2024. External Links: Link Cited by: Introduction, Visual world models and pretrained representations.. Pinneri et al. (2020) C. Pinneri, S. Sawant, S. Blaes, J. Achterhold, J. Stueckler, M. Rolínek, and G. Martius Sample-efficient cross-entropy method for real-time planning. In 4th Conference on Robot Learning, CoRL 2020, 16-18 November 2020, Virtual Event / Cambridge, MA, USA, J. Kober, F. Ramos, and C. J. Tomlin (Eds.), Proceedings of Machine Learning Research, Vol. 155, p. 1049–1065. External Links: Link Cited by: Sampling-based planning.. Radosavovic et al. (2022) I. Radosavovic, T. Xiao, S. James, P. Abbeel, J. Malik, and T. Darrell Real-world robot learning with masked visual pre-training. In Conference on Robot Learning, CoRL 2022, 14-18 December 2022, Auckland, New Zealand, K. Liu, D. Kulic, and J. Ichnowski (Eds.), Proceedings of Machine Learning Research, Vol. 205, p. 416–426. External Links: Link Cited by: Visual world models and pretrained representations.. Rao et al. (2021) Y. Rao, W. Zhao, B. Liu, J. Lu, J. Zhou, and C. Hsieh DynamicViT: efficient vision transformers with dynamic token sparsification. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, M. Ranzato, A. Beygelzimer, Y. N. Dauphin, P. Liang, and J. W. Vaughan (Eds.), p. 13937–13949. External Links: Link Cited by: Sparse visual computation and imagination.. Robine et al. (2023) J. Robine, M. Höftmann, T. Uelwer, and S. Harmeling Transformer-based world models are happy with 100k interactions. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, External Links: Link Cited by: Visual world models and pretrained representations.. Seo et al. (2022) Y. Seo, D. Hafner, H. Liu, F. Liu, S. James, K. Lee, and P. Abbeel Masked world models for visual control. In Conference on Robot Learning, CoRL 2022, 14-18 December 2022, Auckland, New Zealand, K. Liu, D. Kulic, and J. Ichnowski (Eds.), Proceedings of Machine Learning Research, Vol. 205, p. 1332–1344. External Links: Link Cited by: Visual world models and pretrained representations.. Xiao et al. (2022) T. Xiao, I. Radosavovic, T. Darrell, and J. Malik Masked visual pre-training for motor control. CoRR abs/2203.06173. External Links: Link, Document, 2203.06173 Cited by: Visual world models and pretrained representations.. Xu et al. (2022) Y. Xu, Z. Zhang, M. Zhang, K. Sheng, K. Li, W. Dong, L. Zhang, C. Xu, and X. Sun Evo-vit: slow-fast token evolution for dynamic vision transformer. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelfth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22 - March 1, 2022, p. 2964–2972. External Links: Link, Document Cited by: Sparse visual computation and imagination.. Yao et al. (2026) G. Yao, J. Zheng, Z. Wang, W. Zhang, R. Han, C. Zhao, Z. Zhang, and R. Liu V-pruner: A fast and globally-informed token pruning framework for vision transformer. In Fortieth AAAI Conference on Artificial Intelligence, Thirty-Eighth Conference on Innovative Applications of Artificial Intelligence, Sixteenth Symposium on Educational Advances in Artificial Intelligence, AAAI 2026, Singapore, January 20-27, 2026, S. Koenig, C. Jenkins, and M. E. Taylor (Eds.), p. 34396–34404. External Links: Link, Document Cited by: Sparse visual computation and imagination.. Yin et al. (2022) H. Yin, A. Vahdat, J. M. Álvarez, A. Mallya, J. Kautz, and P. Molchanov A-vit: adaptive tokens for efficient vision transformer. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, p. 10799–10808. External Links: Link, Document Cited by: Sparse visual computation and imagination.. Yin et al. (2026) S. Yin, K. Yin, W. Chen, Y. Liu, G. Li, and L. Lin DDP-WM: disentangled dynamics prediction for efficient world models. CoRR abs/2602.01780. External Links: Link, Document, 2602.01780 Cited by: Introduction, Sparse visual computation and imagination.. Zhang et al. (2024) K. Zhang, B. Li, K. Hauser, and Y. Li AdaptiGraph: material-adaptive graph-based neural dynamics for robotic manipulation. In Robotics: Science and Systems X, Delft, The Netherlands, July 15-19, 2024, D. Kulic, G. Venture, K. E. Bekris, and E. Coronado (Eds.), External Links: Link, Document Cited by: Environments and datasets.. Zhang et al. (2023) W. Zhang, G. Wang, J. Sun, Y. Yuan, and G. Huang STORM: efficient stochastic transformer based world models for reinforcement learning. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), External Links: Link Cited by: Visual world models and pretrained representations.. Zhou et al. (2025) G. Zhou, H. Pan, Y. LeCun, and L. Pinto DINO-WM: world models on pre-trained visual features enable zero-shot planning. In Forty-second International Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 13-19, 2025, A. Singh, M. Fazel, D. Hsu, S. Lacoste-Julien, F. Berkenkamp, T. Maharaj, K. Wagstaff, and J. Zhu (Eds.), Proceedings of Machine Learning Research, Vol. 267. External Links: Link Cited by: Introduction, Visual world models and pretrained representations., Method, Environments and datasets., Baseline.. Supplementary Material This technical supplement provides additional implementation, evaluation, and experimental details for Scope-WM. It follows the notation and task definitions of the main paper. Appendix A Implementation and Reproducibility Details A.1 Training Protocol and Checkpoint Selection All models are trained within a budget of at most 100100 epochs. Because convergence speed differs across tasks, the checkpoint used for reporting is selected according to planning performance on the corresponding validation split. The Fast and Faster configurations are trained independently with K=98K=98 and K=32K=32 full-dynamics tokens, respectively; Faster is not obtained by changing the token budget only at planning time. Training proceeds in two steps. We first export gradient-times-input relevance targets from a trained dense DINO-WM teacher and train a task-specific DRS head. The distilled selector is then frozen, after which the sparse dynamics predictor and FDBU are optimized jointly using the full-grid latent prediction loss described in the main paper. No separate training stage is used for FDBU. A.2 Model and Optimization Settings Table S1 summarizes the shared architecture and optimization settings. The frozen DINOv2 ViT-S/14 encoder produces 196196 patch tokens from each 224×224224× 224 observation. The visual decoder is disabled during model training and planning; it is used only for the qualitative visualization reported below. Component Hyperparameter Setting Visual representation Encoder Frozen DINOv2 ViT-S/14 Input resolution 224×224224× 224 Visual tokens / token dimension 196/384196/384 Dynamics predictor Architecture 6-layer causal Transformer Attention heads / head dimension 16/6416/64 Feed-forward hidden dimension 20482048 Transformer / embedding dropout 0.1/00.1/0 Action / proprioceptive embedding dimension 10/1010/10 Action/proprio standardization Dataset statistics Prediction steps per world-model call 11 Sparse world-model training Optimizer AdamW Predictor and action-encoder learning rate 1×10−41× 10^-4 Weight decay 0.010.01 Maximum training epochs 100100 Latent supervision MSE over all 196196 latent positions Training seed 00 FDBU Foreground summaries Mean predicted state and mean foreground delta Context projection LN–Linear–GELU Update branch LN–Linear–GELU–Linear Gate branch LN–Linear–Sigmoid Hidden multiplier 2.02.0 Residual scale α 1.01.0 Dropout 00 Table S1: Shared model and optimization settings. LN denotes layer normalization. A.3 DRS Distillation For each of the five tasks, DRS is distilled from the corresponding dense DINO-WM teacher. The teacher relevance target is computed from the full-grid latent prediction loss using the gradient-times-input score defined in the main paper. The selector receives each globally contextualized DINOv2 patch token together with the action and proprioceptive embeddings. We use the same target-export procedure, selector architecture, and distillation configuration across both rigid and deformable tasks, as summarized in Table S2. With a batch size of 88 and at most 400400 exported batches, each task uses up to 3,2003,200 trajectory windows for selector distillation. Hyperparameter Setting Target type Gradient-times-input Target split Training split Export batch size 88 Maximum exported batches 400400 Target-export workers 88 Selector hidden dimension 128128 Distillation epochs 8080 Distillation batch size 256256 Optimizer AdamW Learning rate 3×10−43× 10^-4 Weight decay 1×10−41× 10^-4 Objective KL divergence Temperature T 1.01.0 Action/proprio conditioning Enabled Selector during sparse-WM training Frozen Table S2: DRS target-export and distillation settings shared across all five tasks. A.4 Task-Specific Training Settings Table S3 reports the task-specific settings. PointMaze and Push-T use three observation-history frames and a frameskip of five. Wall uses one observation-history frame with the same frameskip of five. Rope and Granular use one history frame and no action subsampling. All available trajectories in the configured training directories are used. PointMaze, Wall, Rope, and Granular use a 90/1090/10 trajectory split generated with seed 4242, while Push-T uses the predefined training and validation directories distributed with the dataset. Task History Frameskip Global batch Training GPUs Epoch budget Train K Eval. instances Max MPC steps PointMaze 3 5 512 8 V100 ≤100≤ 100 98 / 32 50 15 Wall 1 5 512 8 V100 ≤100≤ 100 98 / 32 50 15 Push-T 3 5 512 8 V100 ≤100≤ 100 98 / 32 50 15 Rope 1 1 64 1 RTX 2080 Ti 100 98 / 32 10 7 Granular 1 1 64 1 RTX 2080 Ti 100 98 / 32 10 7 Table S3: Task-specific training and evaluation settings. Fast and Faster are independently trained with K=98K=98 and K=32K=32, respectively. A.5 Planning Configuration Unless otherwise stated, standard CEM uses 100100 candidates, 1010 elites, and 1010 optimization iterations. EB-CEM uses 300300 candidates and 3030 elites at the initial MPC step, followed by 100100 candidates and 1010 elites at subsequent steps. A bank of 3030 candidates is saved after the initial MPC search. At later steps, 70%70\% of the population is sampled locally around banked elites with perturbation scale 0.50.5, while the remaining 30%30\% is drawn from the current CEM Gaussian. The elite bank is initialized once and is not shifted or replaced during later MPC steps. The visual decoder is not loaded during planning. Hyperparameter Setting Planning horizon 5 CEM optimization iterations 10 Initial variance scale 1.0 Standard population / elites 100/10100/10 Initial EB-CEM population / elites 300/30300/30 Later EB-CEM population / elites 100/10100/10 Elite-bank size MbM_b 30 Bank-save MPC step 0 Bank-use start step 1 Local fraction η 0.7 Global fraction 0.3 Perturbation scale λ 0.5 Sequence shift 0 Bank update mode Initial only Planning objective Final predicted latent Table S4: CEM and EB-CEM hyperparameters used for the main Scope-WM results. The planning objective is evaluated on the final predicted latent state. PointMaze uses the visual-latent cost only (objective weight α=0α=0), whereas Wall, Push-T, Rope, and Granular use equal visual and proprioceptive weights (α=1α=1). PointMaze, Wall, Rope, and Granular construct goals from random environment states, while Push-T uses a five-step segment from the validation data. Appendix B Evaluation Protocol B.1 Datasets and Goal Construction We follow the task definitions and offline datasets used by DINO-WM. All observations are resized and center-cropped to 224×224224× 224, then normalized channel-wise with mean and standard deviation 0.50.5. Actions and proprioceptive states are standardized using dataset statistics. PointMaze and Wall use procedurally sampled initial–goal pairs under fixed evaluation seeds. Push-T draws a segment of length 5×5+15× 5+1 simulator frames from the validation set: the first frame is used as the initial state, and the recorded low-level actions are replayed to construct the five-step goal observation. For Rope and Granular, the initial particle state is drawn from the validation data. The goal is generated using the environment’s task-specific geometric transformation. For Rope, the goal applies a random in-plane rotation and translation. For Granular, the goal applies a random scale and translation. Planning compares frozen-encoder latent features, whereas the final evaluation uses the simulator state. B.2 Random Seeds and Number of Runs Sparse world-model training uses seed 00, and random trajectory splits use seed 4242. Planning follows the official DINO-WM seed-generation rule. With planning seed 9999, evaluation instance i uses si=99i+1,i=0,…,n−1.s_i=99i+1, i=0,…,n-1. PointMaze, Wall, and Push-T are evaluated on 5050 fixed instances, while Rope and Granular are evaluated on 1010 fixed instances. Every reported Scope-WM result is obtained from one selected checkpoint evaluated on the complete fixed instance set; results are not averaged across independently trained models. Within each task, all locally evaluated variants use the same evaluation seeds and initial–goal construction procedure. B.3 Metrics and Efficiency Measurement PointMaze and Wall count an episode as successful when the final positional distance to the goal is below 0.50.5 and 4.54.5, respectively. Push-T requires both a state-position distance below 2020 and an orientation error below π/9π/9. For Rope and Granular, we report the symmetric Chamfer Distance between the final and goal particle sets using their three-dimensional coordinates: CD(X,Y)= (X,Y)= 1|X|∑x∈Xminy∈Y∥x−y∥2 1|X| _x∈ X _y∈ Y x-y _2 +1|Y|∑y∈Yminx∈X∥y−x∥2. + 1|Y| _y∈ Y _x∈ X y-x _2. Planning time is measured as the cumulative CEM sub-planner time over all executed MPC steps. Environment stepping, metric computation, decoding, and visualization are excluded. Peak GPU memory is the maximum CUDA memory allocated during a complete evaluation run after resetting the peak-memory counter. FLOPs are measured for one world-model prediction and reported per prediction sample. The cross-task task-performance values of prior methods in the main paper are taken from their published reports. For the local Push-T efficiency comparison, dense DINO-WM and Scope-WM are evaluated on the same hardware and evaluation instances. The dense local baseline uses standard CEM with 100100 candidates, 1010 elites, and 1010 optimization iterations. Appendix C Hardware and Software Environment PointMaze, Wall, and Push-T. These models are trained using eight NVIDIA V100 GPUs, while planning evaluation is performed on a single 32-GB NVIDIA V100 GPU. The single-GPU platform provides 16 virtual CPU cores from an Intel Xeon Platinum 8352V processor at 2.10 GHz and 64 GB of host memory, and runs Ubuntu 22.04. The primary software environment uses Python 3.9.19, PyTorch 2.3.0, torchvision 0.18.0, CUDA 12.1, and cuDNN 8.9.2. We use Accelerate 0.26.1 for distributed training, Hydra 1.2.0 and OmegaConf 2.3.0 for configuration management, and NumPy 1.26.4. The corresponding simulation dependencies include Gym 0.23.1, D4RL 1.1, MuJoCo 3.2.7, and mujoco-py 2.1.2.14. Rope and Granular. The deformable tasks are trained and evaluated on one NVIDIA GeForce RTX 2080 Ti GPU with 22,528 MiB of memory. The machine provides an Intel Xeon Platinum 8255C CPU at 2.50 GHz and 503 GiB of host memory, and runs Ubuntu 22.04.3 LTS. The Python environment uses Python 3.12.3 and PyTorch 2.3.0+cu121 with CUDA 12.1 and cuDNN 8.9.2. The PyFlex simulator bindings are compiled separately with the CUDA 9.2 toolkit required by the available source build and loaded into the PyTorch environment through the paths provided in the code supplement. Appendix D Additional Efficiency Results D.1 Efficiency Across Rigid-Body Tasks Table S5 extends the end-to-end efficiency comparison from Push-T to PointMaze and Wall. For each task, dense DINO-WM and Scope-WM are measured on the same GPU using the same initial–goal pair and a forced full-length planning run. Planning time denotes the cumulative CEM sub-planner time across all MPC steps, consistent with the metric used in the main paper. Task success rates are taken from the corresponding complete evaluation sets reported in the main paper. Task Method K Plan time (s, ↓ ) Speedup (↑ ) Peak Mem. (MB, ↓ ) Mem. Ratio (↓ ) SR (↑ ) PointMaze DINO-WM 196 204.52 1.00×1.00× 6309 100.0% 98% PointMaze Scope-WM (Faster) 32 35.01 5.84×5.84× 3334 52.8% 98% Wall DINO-WM 196 59.44 1.00×1.00× 1542 100.0% 96% Wall Scope-WM (Faster) 32 19.60 3.03×3.03× 1939 125.8% 92% Push-T DINO-WM 196 240.96 1.00×1.00× 18436 100.0% 90% Push-T Scope-WM (Faster) 32 34.56 6.97×6.97× 3334 18.1% 88% Table S5: End-to-end efficiency on three rigid-body planning tasks. Time and peak memory use one fixed full-length run; SR uses the complete 5050-instance set. Memory ratios are normalized per task, and the peak includes the larger initial EB-CEM population. Scope-WM consistently reduces planning time across the three tasks, achieving 5.84×5.84×, 3.03×3.03×, and 6.97×6.97× speedups on PointMaze, Wall, and Push-T, respectively. The peak-memory benefit depends on the relative contributions of rollout activations and search-population overhead. Scope-WM substantially reduces memory on PointMaze and Push-T, where dense latent rollouts incur larger activation footprints. On Wall, the dense baseline already has a small memory footprint, while EB-CEM temporarily uses a larger population at the initial MPC step. Consequently, its initial-search overhead outweighs the smaller savings from token sparsity, producing a modest absolute memory increase. These results indicate that scoped latent prediction provides the largest memory savings when dense rollout activations dominate the overall footprint, while its planning-time benefit remains consistent across tasks. Appendix E Additional Qualitative Results Decoded latent rollouts. Figure S1 compares a Scope-WM latent rollout with the corresponding environment rollout under the same planned action sequence. The decoded rollout follows the overall motion of the agent and the T-shaped object across successive steps, indicating that the sparse latent dynamics preserve task-relevant motion information. The decoder is used only for visualization and is not loaded during planning. Figure S1: Environment and decoded latent rollouts on Push-T. The upper row executes the planned actions in the environment; the lower row decodes the corresponding Scope-WM latent rollout with the dense DINO-WM decoder. Steps proceed left to right, and the highlighted final column is the goal. The decoder is used only for visualization. Appendix F Additional Token-Budget Analysis Fast and Faster are trained independently with K=98K=98 and K=32K=32, respectively. To examine whether the full-dynamics token budget can instead be changed only at planning time, we evaluate each checkpoint under both token budgets. As shown in Table S6, changing K after training substantially degrades planning performance in both directions. Training K Planning K SR (↑ ) 98 98 94% 98 32 14% 32 32 88% 32 98 54% Table S6: Training–planning token-budget consistency on Push-T. Each configuration uses the same 5050 fixed instances. The degradation indicates that K is part of the learned sparse-dynamics configuration rather than a freely adjustable inference-time parameter. The sparse predictor and FDBU are jointly optimized under a particular foreground–background partition: changing K at planning time alters both the number of tokens receiving full dynamics prediction and the distribution of tokens handled by the background update. This creates a mismatch from the computation pattern observed during training. Accordingly, we train separate Fast and Faster models for their respective planning budgets.