Paper deep dive
Pixels to Proofs: Probabilistically-Safe Latent World Model Control via Parallel Conformal Robust MPC
Devesh Nath, Anutam Srinivasan, Haoran Yin, Ruitong Jiang, Jeffrey Fang, Glen Chou
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 6/20/2026, 6:22:25 AM
Summary
SLS^2 is a framework for safe feedback motion planning from pixels using robust model predictive control (MPC) in learned latent world models. It utilizes a joint-embedding predictive architecture (JEPA) to learn compact Markovian latent states and employs Conformal Prediction (CP) to calibrate both latent dynamics error bounds and a latent constraint checker. This allows for the synthesis of reachability-constrained, probabilistically-safe control policies through GPU-accelerated System Level Synthesis (SLS), ensuring that the true system remains within safe sets despite imperfect latent predictions.
Entities (6)
Relation Signals (4)
Conformal Prediction (CP) → calibrates → latent error bounds
confidence 100% · to obtain calibrated latent error bounds and robust latent-space constraint sets.
SLS^2 → incorporates → Conformal Prediction (CP)
confidence 100% · to inform a GPU-accelerated system level synthesis (SLS) robust MPC scheme with conformal prediction
SLS^2 → uses → System Level Synthesis (SLS)
confidence 100% · We present SLS^2... based on the SLS framework [14].
SLS^2 → isbasedon → Le-WM
confidence 90% · Our work builds on LeWorldModel (Le-WM)... We introduce training modifications
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We present SLS^2, a framework for safe feedback motion planning from pixels using robust model predictive control (MPC) in learned latent world models. Our approach trains an action-conditioned joint-embedding world model with compact Markovian latent states, enabling efficient gradient-based trajectory optimization through learned latent dynamics. To enforce safety for the true system despite imperfect latent predictions, we inform a GPU-accelerated system level synthesis (SLS) robust MPC scheme with conformal prediction to obtain calibrated latent error bounds and robust latent-space constraint sets. We further learn and conformalize a latent constraint checker, allowing the SLS planner to impose probabilistic safety constraints during closed-loop execution. We evaluate our method on vision-based control tasks, where it improves both goal-reaching performance and safety over latent world-model and safe-planning baselines.
Tags
Links
- Source: https://arxiv.org/abs/2606.15594v1
- Canonical: https://arxiv.org/abs/2606.15594v1
Trouble viewing inline? Open PDF directly →
Full Text
113,923 characters extracted from source content.
Expand or collapse full text
Pixels to Proofs: Probabilistically-Safe Latent World Model Control via Parallel Conformal Robust MPC Devesh Nath⋆, Anutam Srinivasan⋆, Haoran Yin⋆, Ruitong Jiang, Jeffrey Fang, Glen Chou Georgia Institute of Technology dnath7, asrinivasan350, hyin95, rjiang77, jfang301, chou@gatech.edu ⋆Equal contribution (alphabetical ordering) Website Code Video Abstract We present SLS2, a framework for safe feedback motion planning from pixels using robust model predictive control (MPC) in learned latent world models. Our approach trains an action-conditioned joint-embedding world model with compact Markovian latent states, enabling efficient gradient-based trajectory optimization through learned latent dynamics. To enforce safety for the true system despite imperfect latent predictions, we inform a GPU-accelerated system level synthesis (SLS) robust MPC scheme with conformal prediction to obtain calibrated latent error bounds and robust latent-space constraint sets. We further learn and conformalize a latent constraint checker, allowing the SLS planner to impose probabilistic safety constraints during closed-loop execution. We evaluate our method on vision-based control tasks, where it improves both goal-reaching performance and safety over latent world-model and safe-planning baselines. Keywords: robust visuomotor control, safety verification, JEPA world models Figure 1: Method overview. SLS2 enables robustly safe, reachability-constrained latent planning directly from pixels. Top row: We summarize the pipeline for our robust planner. Bottom row: We show a sample rollout generated by executing control actions computed by the nominal gradient-based planner on which the robust planner is built. We overlay the goal pose, shown with a green rope, on the rollout to highlight the system’s progress toward the goal state. 1 Introduction Safe control is a central challenge for reliable open-world robot deployment. Most safety-critical control methods [1, 2, 3] rely on a well-defined state representation on which planning, control, and verification can be performed. However, in the real world, obtaining such a representation is itself nontrivial. This is especially true for deformable objects and tasks where the state is only implicitly defined through perception. Even when a state representation is known, estimating the state from raw observations, such as images, can require extensive offline training data with labeled correspondences between visual inputs and ground-truth states – data that is often impractical to obtain [4, 5, 6, 7]. Recent advances in representation learning and world modeling [8, 9] offer a promising alternative. Learned world models can infer latent state representations directly from observations and learn dynamics in these latent spaces, enabling prediction and planning without requiring manually specified state variables. These models have shown considerable potential for robotic decision-making [10, 11], particularly in settings where explicit state estimation is difficult. However, ensuring that these latent space control policies yield safe behavior under the true system dynamics is difficult. Latent representations are often high-dimensional, their learned dynamics may be inaccurate, and task constraints expressed in the physical or observation space must be translated into the latent space. These issues complicate the use of classical safety-critical control tools in the latent space, which typically assume low-dimensional state spaces and known, calibrated disturbance bounds. Recent work has explored deep learning for approximating latent safety certificates [12, 13]; however, these methods often rely on heuristic approximations and do not provide sound, calibrated reachability guarantees. To exploit the advances of world modeling while maintaining rigorous safety assurances, we propose SLS2 (Safe Latent Space System Level Synthesis), a scalable robust model predictive control (MPC) framework for safe planning in learned latent spaces based on the SLS framework [14]. Our approach introduces training objectives and optimization strategies that improve gradient-based trajectory optimization through learned latent dynamics. To be robust to prediction error in the latent dynamics, we plan with these learned models using scalable reachability-constrained robust MPC based on SLS. This is made computationally feasible through GPU parallelization, which enables efficient trajectory optimization coupled with local uncertainty propagation around the optimized plan. To enforce safety constraints, we train a constraint checker in the latent space and incorporate it within robust MPC. To enable probabilistic constraint satisfaction, we use conformal prediction (CP) to calibrate bounds on both latent dynamics error and learned constraint-checker error. This ensures that the reachable sets computed by SLS contain the true reachable latent states with high probability, enabling robust safety despite these errors. Specifically, our contributions are: • We learn compact Markovian latent world models from pixels that are accurate under multi-step rollout and amenable to efficient gradient-based trajectory optimization. • We conformalize latent dynamics errors and in-domain latent support, producing calibrated uncertainty sets for robust planning. We further learn and conformalize latent safety classifiers, enabling task constraints to be enforced directly during latent space planning. • We show how to use CP-calibrated uncertainty sets in GPU-accelerated SLS to scalably perform reachability-constrained robust MPC in high-dimensional learned latent spaces. • We validate nominal trajectory optimization and robust MPC with our learned models on both simulated and hardware vision-based control tasks, spanning navigation, pick-and-place, and deformable object manipulation. Our method improves safety and goal-reaching performance over world-model and safe-planning baselines. 2 Related Work World Model World models learn predictive representations of dynamics, enabling agents to reason about future outcomes through imagined rollouts [8, 15, 9, 16, 17, 18]. Early latent world models paired representation learning with reconstruction-based prediction and policy learning in imagination [15, 9, 16], while TD-MPC and TD-MPC2 combine latent dynamics with online model-predictive control for continuous control [19, 11]. More recently, joint-embedding predictive architectures (JEPAs) have emerged as a promising approach to representation-space world modeling by predicting future embeddings rather than pixels [20, 21, 22, 23]. In robotics and offline goal-conditioned planning, PLDM learns reward-free latent dynamics for test-time planning [24], while DINO-WM predicts future DINOv2 visual features from offline action-conditioned trajectories for zero-shot planning without reward supervision [25, 10]. Large-scale generative world models and interactive simulators further demonstrate the promise of learned predictive models for controllable environments [26, 27, 28, 29]. Our work builds on LeWorldModel (Le-WM), an end-to-end JEPA world model that learns from raw pixels via next-embedding prediction with Gaussian embedding regularization and without explicitly predicting future images, then performs latent-space MPC for goal-reaching control [30]. We introduce training modifications that improve both predictive accuracy and gradient-based planning. Safe Latent Planning Safety filters enforce constraints by minimally modifying candidate actions from a nominal controller. Classical approaches such as CBF-QPs, HJ reachability, and predictive safety filters offer principled safety guarantees but typically require known state representations, known dynamics, and hand-designed safe or failure sets [31, 1, 2, 32, 33, 34]. Recent data-driven methods reduce this burden by learning barrier functions, output-feedback certificates, distributional safety constraints from demonstrations and offline data, or by performing uncertainty-aware model-based control with learned dynamics [35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45]. Closest to our setting, safe latent-space methods extend safety filtering to high-dimensional visuomotor control. In-distribution barrier functions learn CBF-like filters in latent space from safe demonstrations [12], while latent safety filters approximate HJ reachability in learned world-model latents to reason about hard-to-specify failures [13]. Follow-up work improves latent safety filtering with uncertainty-aware OOD detection and conformal calibration [46], smooth latent CBFs [47], and latent barrier certificates learned jointly with world models [48]. Latent Policy Barrier treats the expert demonstration manifold as an implicit safety barrier and corrects actions at inference time using learned latent dynamics [49]. These methods show that latent representations can enable safety reasoning from images. However, most existing methods operate as policy filters without formal guarantees, safety predictors that do not perform control [50], or in-distribution recovery mechanisms around a given base policy. In contrast, we directly synthesize safety-constrained feedback policies and motion plans through reachability-informed latent space control, while providing calibrated probabilistic guarantees via SLS and conformal prediction. Figure 2: Robust planning. Cube (a-b) and Reacher (c-d). SLS2 maps pixels directly to robustly-safe actions via latent robust MPC, ensuring robust constraint satisfaction (red) compared to unconstrained baselines. a, b) The Cube task with and without SLS2 enforcing the height threshold. c, d) The Reacher task with (c) and without (d) SLS2 enforcing the angle constraint on the second joint. 3 Preliminaries and Problem Statement We consider an unknown partially observed dynamical system described by an unknown underlying state representation xk∈x_k and control uk∈⊆ℝnuu_k ^n_u, and observations ok∈⊆ℝnoo_k ^n_o: xk+1=g(xk,uk),ok=h(xk),x_k+1=g(x_k,u_k), o_k=h(x_k), (1) where ok∈o_k are RGB images. In our setting, neither the true state xkx_k nor its state representation X is available to the planner. g and h are also unknown; we only observe image measurements o∈o . Problem Statement. We consider safe feedback motion planning from image observations using a learned latent world model. We are given an offline dataset of image transitions :=(ot,ut,ot+1)D:=\(o_t,u_t,o_t+1)\ collected from task-relevant rollouts. The true system evolves in an unknown state space X, while the planner only observes images o∈o from (1). We wish to learn a compact latent representation where prediction, planning, and reachability-based safety verification are tractable. In this paper, we train a visual encoder encθ:→ℝnzenc_θ:O ^n_z, with zt=encθ(ot)z_t=enc_θ(o_t). Using encθenc_θ, we construct a Markovian latent state stz=[zt,Δzt,…,ΔKzt]∈⊆ℝnss_t^z=[z_t, z_t,…, ^Kz_t] ^n_s, where Δzt=zt−zt−1 z_t=z_t-z_t-1, from a history of encoded images. The learned latent dynamics fϕ:×→f_φ:S×U predict the next Markovian state s^t+1z=fϕ(stz,ut) s_t+1^z=f_φ(s_t^z,u_t). As fϕf_φ can be inaccurate, we model its prediction error as st+1z=fϕ(stz,ut)+ets_t+1^z=f_φ(s_t^z,u_t)+e_t, where ℰ⊆ℝnsE ^n_s is calibrated from a held-out subset of D using CP to contain the true prediction error ete_t with a user-specified probability 1−δ∈(0,1)1-δ∈(0,1). Specifically, we solve: Problem 1. Learning a planning- and verification-friendly latent world model. Using D, learn encθenc_θ and fϕf_φ such that the latent state stzs_t^z is predictive under actions, the learned dynamics are accurate under multi-step rollout, differentiable, and compatible with gradient-based trajectory optimization. Construct uncertainty sets ℰ(s,u)⊆ℝnsE(s,u) ^n_s such that the prediction error e at a particular (s,u)(s,u) satisfies e∈ℰ(s,u)e (s,u) with probability 1−δ1-δ. Finally, learn a latent safety score cψ:ℝnz→ℝc_ψ:R^n_z , where larger values indicate safer states, and define the constraint score gψ(s):=−cψ(Πzs)g_ψ(s):=-c_ψ( _zs), where Πz _z extracts the zeroth-order latent embedding. Problem 2. Reachability-informed robust MPC in latent-space. Given fϕf_φ, ℰE, and a latent safe set safe:=s∈∣gψ(s)≤−ηS_safe:=\s g_ψ(s)≤-η\ for η∈ℝη , optimize a nominal length-T latent trajectory :=ζkk=0T ζ:=\ _k\_k=0^T and controls :=vkk=0T−1v:=\v_k\_k=0^T-1 satisfying ζk+1=fϕ(ζk,vk) _k+1=f_φ( _k,v_k), together with a causal state-feedback controller π:=(π0,…,πT−1)π:=( _0,…, _T-1), πk:k+1→ _k:S^k+1 , that stabilizes the true latent dynamics sk+1z=fϕ(skz,πk(s0:kz))+eks_k+1^z=f_φ(s_k^z, _k(s_0:k^z))+e_k about (,)( ζ,v), where eke_k is the true prediction error realized by executing on the true system (1). This MPC policy should produce reachable tubes ℛksR_k^s and ℛkuR_k^u such that, with probability at least 1−δ1-δ, skz∈ℛks⊆safes_k^z _k^s _safe for all k=0,…,Tk=0,…,T and πk(s0:kz)∈ℛku⊆ _k(s_0:k^z) _k^u for all k=0,…,T−1k=0,…,T-1. 4 Methodology Our method, SLS2 (Fig. 1), trains a Markovian latent model and constraint classifier (Sec. 4.1), conformalizes its uncertainty (Sec. 4.2), and incorporates the resulting calibrated uncertainty sets into SLS-based robust MPC for probabilistically-safe feedback motion planning (Sec. 4.3). 4.1 Training World Models Friendly for Gradient-Based Robust MPC We use the given offline dataset D to train an action-conditioned latent world model following a Le-WM-style [30] joint-embedding predictive architecture. Given an image observation oto_t, an encoder encθenc_θ maps the image into a compact latent embedding zt=encθ(ot)z_t=enc_θ(o_t). The encoder is implemented as a ViT backbone trained from scratch, followed by an MLP projector that maps the encoder output to the latent embedding space. Before encoding, image observations are resized to a fixed resolution, normalized, and organized into short temporal windows sampled from offline trajectories. Actions are normalized using the dataset mean and standard deviation. To model dynamics, we construct a Markov latent state by augmenting the current embedding with finite-difference latent features: stz=[zt,Δzt,…,ΔKzt],Δzt=zt−zt−1,s_t^z= [z_t,\; z_t,\;…,\; ^Kz_t ], z_t=z_t-z_t-1, (2) where K denotes the Markov order. The latent dynamics predictor fϕf_φ is an MLP that takes the current Markov latent state and action as input and predicts the next Markov latent state: s^t+1z=fϕ(stz,at). s_t+1^z=f_φ(s_t^z,a_t). (3) For multi-step training, the predictor is rolled out autoregressively over a horizon H, where each predicted latent state is fed back into the model for the next prediction step s^t+h+1z=fϕ(s^t+hz,at+h) s_t+h+1^z=f_φ( s_t+h^z,a_t+h) for h=0,…,H−1.h=0,…,H-1. The main training loss is a multi-step latent prediction objective. At each prediction step, the predicted Markov latent state is compared against the target Markov latent state computed from the encoded ground-truth future observations: ℒpred=1H∑h=1H‖s^t+hz−st+hz‖22. _pred= 1H _h=1^H \| s_t+h^z-s_t+h^z \|_2^2. (4) We follow Le-WM [30] and prevent representational collapse via the SIGReg loss [51] and additionally encourage consecutive latent velocity vectors to align via temporal straightening [52]: ℒsig=SIGReg(ztt=1T),L_sig=SIGReg (\z_t\_t=1^T ), (5) ℒstr=1T−2∑t=2T−1(1−(zt−zt−1)⊤(zt+1−zt)‖zt−zt−1‖2‖zt+1−zt‖2). _str= 1T-2 _t=2^T-1 (1- (z_t-z_t-1 ) (z_t+1-z_t ) \|z_t-z_t-1 \|_2 \|z_t+1-z_t \|_2 ). (6) The full training objective is therefore ℒ=ℒpred+λsigℒsig+λstrℒstrL=L_pred+ _sigL_sig+ _strL_str. Obstacle Classifier. For constrained tasks, we train a latent obstacle classifier to detect whether a predicted latent state violates task constraints. For each task, we build a balanced dataset of obstacle and non-obstacle observations by sampling task-relevant configurations, labeling them with the known simulator constraint, and rendering them as RGB images. This yields positive and negative constraint examples without relying on rollout failures. While the labeling rule is task-specific, the dataset has a common form across tasks, obs=(oi,yi)i=1ND_obs=\(o_i,y_i)\_i=1^N, with yi∈0,1,y_i∈\0,1\, where yi=0y_i=0 denotes a constraint-violating image and yi=1y_i=1 denotes a non-violating image. We train the classifier in the latent space rather than in pixel space. Using the frozen encoder, each image is mapped to zi=encθ(oi)z_i=enc_θ(o_i). A small MLP classifier cψ:ℝnz→ℝc_ψ:R^n_z maps ziz_i to a signed score ci=cψ(zi)c_i=c_ψ(z_i), where positive scores indicate non-violating states and negative scores indicate constraint-violating states. Thus, binary labels 0,1\0,1\ are mapped to the signed labels −1,+1\-1,+1\, with yi=0y_i=0 mapped to −1-1 and yi=1y_i=1 mapped to +1+1, and the classifier is trained with a hinge loss, ℒobs=1N∑i=1Nmax(0,m−y~icψ(zi)), _obs= 1N _i=1^N (0,\;m- y_ic_ψ(z_i) ), (7) where m is the classification margin. We normalize latent features using the training-split mean and standard deviation, then calibrate the classifier threshold on a held-out split. Denote ℐcal−:=i∈ℐcal:yi=0I_cal^-:=\i _cal:y_i=0\ as the violating calibration indices. For each i∈ℐcal−i _cal^-, we compute the nonconformity score ri=max(0,cψ(zi))r_i= (0,\;c_ψ(z_i) ). We then choose a conformal threshold η as the empirical (1−δ)(1-δ) quantile of these nonconformity scores. At planning time, a latent Markov state s is treated as safe if cψ(Πzs)≥η,⟺gψ(s):=−cψ(Πzs)≤−η.c_ψ( _zs)≥η, g_ψ(s):=-c_ψ( _zs)≤-η. (8) This yields a conservative obstacle classifier used during constrained planning: the world model predicts latent rollouts, and the classifier detects whether they enter the learned obstacle set. 4.2 Latent Uncertainty Quantification via Conformal Prediction (CP) To ensure closed-loop safety for planning, we need two additional sources of uncertainty to be bounded: 1) latent dynamics prediction error and 2) in-distribution domain calibration. First, to quantify latent prediction error, we use split CP [53, 54]. Let cal=(si,ui,si+)i=1nD_cal=\(s_i,u_i,s_i^+)\_i=1^n be a held-out set of latent transitions obtained by encoding a subset of D, and define residuals ei:=si+−fϕ(si,ui)e_i:=s_i^+-f_φ(s_i,u_i). To capture state-action-dependent error geometry, we train an uncertainty model Σψ:×→++ns _ψ:S×U _++^n_s using the multivariate Gaussian negative log-likelihood loss ℒMGNLL=12(ei⊤Σψ(si,ui)−1ei+logdetΣψ(si,ui)).L_MGNLL= 12 (e_i _ψ(s_i,u_i)^-1e_i+ _ψ(s_i,u_i) ). (9) The covariance model is not assumed to be calibrated; it only provides a local ellipsoid shape for the prediction error. We calibrate its scale using CP with scores ri:=ei⊤Σψ(si,ui)−1eir_i:=e_i _ψ(s_i,u_i)^-1e_i. Let r(1)≤⋯≤r(n)r_(1)≤·s≤ r_(n) be the sorted scores. For trajectory-level failure probability δ over horizon T, set δ¯:=δ/T δ:=δ/T and q:=r(⌈(n+1)(1−δ¯)⌉)q:=r_ ( (n+1)(1- δ) ), with q=+∞q=+∞ if the index exceeds n. For any latent state-action pair (s,u)(s,u), the calibrated one-step error set is ℰ(s,u):=e∣e⊤Σψ(s,u)−1e≤q.E(s,u):=\e e _ψ(s,u)^-1e≤ q\. (10) Writing Σψ(s,u)=Lψ(s,u)Lψ(s,u)⊤ _ψ(s,u)=L_ψ(s,u)L_ψ(s,u) using its Cholesky factor LψL_ψ, this is equivalently ℰ(s,u)=qLψ(s,u)ℬnsE(s,u)= q\,L_ψ(s,u)B^n_s, where ℬns:=s∈ℝns∣‖s‖2≤1B^n_s:=\s ^n_s \|s\|_2≤ 1\. Second, we conformalize an in-domain latent region to keep the planner within the support of the calibration data. Since the SIGReg loss encourages the latent embeddings to follow an approximately Gaussian distribution [51], we define the in-domain set as a calibrated Gaussian density sublevel set. Concretely, we fit an ellipsoid to the calibration embeddings and choose its conformal threshold so that it contains most calibration latents with the desired coverage. Let μ and ΣID _ID be the empirical mean and covariance of calibration latents in calD_ cal, and define di:=(si−μ)⊤ΣID−1(si−μ)d_i:=(s_i-μ) _ID^-1(s_i-μ). With qIDq_ID the (1−αID)(1- _ID) split conformal quantile of di\d_i\, the calibrated in-domain set is ℐ:=s∣(s−μ)⊤ΣID−1(s−μ)≤qID.I:=\s (s-μ) _ID^-1(s-μ)≤ q_ID\. (11) During planning, we require ℛks⊆ℐR_k^s for all k. This controls support mismatch, while the CP guarantee itself assumes that closed-loop residual scores are exchangeable with the calibration scores. In practice, we support this assumption by collecting calibration transitions from representative randomized rollouts that vary initial states and goals. 4.3 Conformalized Robust MPC via System Level Synthesis (SLS) To synthesize robust latent feedback policies, we use system level synthesis (SLS) [14]. SLS enables efficient robust MPC by optimizing over closed-loop system responses rather than feedback gains directly, and admits fast GPU-parallel implementations for high-dimensional systems [55]. This is particularly important in latent-space planning, where the latent state dimension can be high. We build on [56], using the conformal calibration in Sec. 4.2 to define the disturbance sets to be propagated by SLS. We model the learned latent dynamics with calibrated additive uncertainty, sk+1=fϕ(sk,uk)+Ekwk,wk∈ℬns,s_k+1=f_φ(s_k,u_k)+E_kw_k, w_k ^n_s, (12) where Ek=qLψ(sk,uk)E_k= qL_ψ(s_k,u_k), recovering EkℬnsE_kB^n_s as the CP-calibrated, ellipsoidal one-step latent prediction-error set defined in (10). Given the current latent state s¯0 s_0, SLS optimizes a nominal trajectory :=ζkk=0T ζ:=\ _k\_k=0^T, controls :=vkk=0T−1v:=\v_k\_k=0^T-1, and closed-loop response matrices Φs,Φu ^s, ^u. The responses map disturbances to deviations from the nominal state and control trajectories. Around (ζk,vk)( _k,v_k), let Ak:=∇sfϕ(ζk,vk)A_k:= _sf_φ( _k,v_k) and Bk:=∇ufϕ(ζk,vk)B_k:= _uf_φ( _k,v_k). From Sec. 4.2, if Σψ(ζk,vk)=Lψ(ζk,vk)Lψ(ζk,vk)⊤ _ψ( _k,v_k)=L_ψ( _k,v_k)L_ψ( _k,v_k) , we set Ek:=qLψ(ζk,vk).E_k:= q\,L_ψ( _k,v_k). (13) Thus SLS propagates the CP-calibrated ellipsoidal model errors through the horizon to construct reachable tubes and constraint tightenings. The resulting CP-calibrated SLS MPC problem is min,,Φs,Φu _ ζ,v, ^s, ^u J(,)+Jf(ζT)+H(Φs,Φu) J( ζ,v)+J_f( _T)+H( ^s, ^u) (14a) s.t. .t. ζk+1=fϕ(ζk,vk),ζ0=s¯0, _k+1=f_φ( _k,v_k), _0= s_0, k=0,…,T−1, k=0,…,T-1, (14b) Φk+1,js=AkΦk,js+BkΦk,ju, ^s_k+1,j=A_k ^s_k,j+B_k ^u_k,j, 0≤j<k,k=0,…,T−1, 0≤ j<k,\ k=0,…,T-1, (14c) Φj+1,js=Ej, ^s_j+1,j=E_j, j=0,…,T−1, j=0,…,T-1, (14d) gi(ζk,vk)+bi+τi,k≤0, g_i( _k,v_k)+b_i+ _i,k≤ 0, i=1,…,nc,k=0,…,T, i=1,…,n_c,\ k=0,…,T, (14e) ℛks⊆ℐ, _k^s , k=0,…,T, k=0,…,T, (14f) where τi,k:=∑j=0k−1‖∇sgi(ζk,vk)Φk,js+∇ugi(ζk,vk)Φk,ju‖2 _i,k:= _j=0^k-1\| _sg_i( _k,v_k) ^s_k,j+ _ug_i( _k,v_k) ^u_k,j\|_2, and ℛks:=ζk⊕⨁j=0k−1Φk,jsℬnsR_k^s:= _k _j=0^k-1 ^s_k,jB^n_s, ℛku:=vk⊕⨁j=0k−1Φk,juℬnsR_k^u:=v_k _j=0^k-1 ^u_k,jB^n_s are the state and control reachable tubes, respectively. Here, ⨁ and ⊕ denote Minkowski sums, and bib_i is a constraint-index-dependent bias equal to η for obstacle constraints and zero for control constraints. For sets A and ℬB, ⊕ℬ=a+b∣a∈,b∈ℬA =\a+b a ,b \. The objective (14a) combines nominal task cost, terminal cost, and a tube-size regularizer H(Φs,Φu)H( ^s, ^u) (see App. B for details). The constraint (14b) enforces nominal feasibility under the latent dynamics, and (14c)-(14d) enforce that SLS correctly propagates disturbance, with the CP-calibrated disturbance set injected through (14d). The constraints (14e) are tightened by reachable tube margins τi,k _i,k, enforcing robust satisfaction of the learned latent safety constraints over the full reachable tube, and (14f) keeps the reachable tube inside the conformalized in-domain latent region (11). For instance, with gψ(s):=−cψ(Πzs)g_ψ(s):=-c_ψ( _zs), the obstacle constraint gψ(sk)≤−ηg_ψ(s_k)≤-η becomes gψ(ζk)+η+τk≤0g_ψ( _k)+η+ _k≤ 0 at each step k to enforce robust constraint satisfaction. This CP-informed SLS scheme provides the following robustness guarantee (proof in App. C): Theorem 1 (Probabilistic containment of conformal SLS tubes). Consider the latent closed-loop dynamics sk+1=fϕ(sk,uk)+eks_k+1=f_φ(s_k,u_k)+e_k, for k=0,…,T−1k=0,…,T-1. Let q be the split conformal quantile of the calibration scores ri=ei⊤Σψ(si,ui)−1eir_i=e_i _ψ(s_i,u_i)^-1e_i and ei:=si+−fϕ(si,ui)e_i:=s_i^+-f_φ(s_i,u_i), computed at per-step miscoverage δ¯=δ/T δ=δ/T. For each closed-loop transition, define the test score rktest:=ek⊤Σψ(sk,uk)−1ekr_k^test:=e_k _ψ(s_k,u_k)^-1e_k. Assume that, for each k, rktestr_k^test is exchangeable with the calibration scores. Let Σψ(s,u)=Lψ(s,u)Lψ(s,u)⊤ _ψ(s,u)=L_ψ(s,u)L_ψ(s,u) and define the local conformal error set ℰ(s,u):=qLψ(s,u)ℬnsE(s,u):= q\,L_ψ(s,u)B^n_s as defined in (10). Suppose the SLS controller is synthesized with disturbance matrices EkE_k such that the realized conformal error sets are contained in the SLS disturbance sets, ℰ(sk,uk)⊆EkℬnsE(s_k,u_k) E_kB^n_s, for k=0,…,T−1k=0,…,T-1. If the SLS controller is robustly correct for all disturbances wk∈ℬnsw_k ^n_s in Δsk+1=AkΔsk+BkΔuk+Ekwk s_k+1=A_k s_k+B_k u_k+E_kw_k, where Δsk:=sk−ζk s_k:=s_k- _k and Δuk:=uk−vk u_k:=u_k-v_k, then ℙ[(sk∈ℛks,∀k=0,…,T)∧(uk∈ℛku,∀k=0,…,T−1)]≥1−δP [(s_k _k^s,\;∀ k=0,…,T) \;(u_k _k^u,\;∀ k=0,…,T-1) ]≥ 1-δ. In addition to robust MPC, the latent dynamics in (3), obtained via Sec. 4.1, can also be used for gradient-based nominal trajectory optimization by removing the reachability-related terms in (14) and retaining only the nominal latent space dynamics constraints in (14b). 5 Experiments To show the utility of our world model learning method (Sec. 4.1) for nominal gradient-based planning and the safety assurances provided by our robust MPC planner (SLS2, Sec. 4.3), we evaluate on four tasks: Reacher [57], OGBench Cube [58], Push-T [10], and a bimanual rope manipulation task. We compare our gradient-based nominal planner against world-model planning baselines in Sec. 5.1, and compare SLS2 against safe latent-space planning baselines in Sec. 5.2. For nominal planning, as discussed in Sec. 4.3, we use GPU-SLS [55] without tubes or constraints (which simplifies to iLQR [59]). For robust planning, we use SLS2. We compare against three latent-space world-model planning baselines: Le-WM [30], DINO-WM [10], and PLDM [24]. For safe latent space planning, we compare against two safety-focused baselines: HJ-filtered [13] and LPB [49], as well as two non-robust ablations of SLS2 (i.e., constrained iLQR without tubes or constraint (11)). All results are run on an Intel i9-14900K CPU, 64 GB RAM, with an NVIDIA RTX 4090 GPU. For all tasks, planning performance is evaluated on held-out test trajectories. To benchmark the nominal planner, we evaluate over five random seeds, with 40 evaluation episodes per seed. Unlike prior works that evaluate shorter-horizon planning segments, we require the planner to solve the complete task from the initial state to the corresponding terminal goal, final state, or task-specific completion condition. This makes the benchmark substantially more difficult, since errors accumulate over the full task horizon rather than only over a short local segment. For constrained tasks, we evaluate on 35 randomized start and goal conditions per task. Details on implementation and setup are in App. D. We report metrics according to the task setting. For unconstrained tasks, we report the task success rate (higher is better) and the minimum distance to the goal across evaluation trials (lower is better). For constrained tasks, we also report the safety rate (adherence to constraints) and the robust success rate (when both constraint satisfaction and goal-reach conditions are met). Table 1: Ablation study on Reacher. The reference setting is the full method described in Sec. 4. World-model ablations keep the planner fixed and change only the latent model/training component. The planner ablation keeps the world model fixed and changes only the MPC optimizer. Ablation Variant Success (%) ↑ Min. qpos dist. ↓ Final qpos dist. ↓ Solve time / step (ms) ↓ Reference Full model + iLQR 83.50±4.0683.50± 4.06 0.377±0.1190.377± 0.119 0.497±0.1520.497± 0.152 426.35±17.52426.35± 17.52 World model Non-Markov history-2 + iLQR 1.00±1.221.00± 1.22 1.786±0.0881.786± 0.088 2.080±0.0872.080± 0.087 67.00±3.0167.00± 3.01 Single-step loss + iLQR 24.50±4.5824.50± 4.58 1.283±0.1381.283± 0.138 1.746±0.2051.746± 0.205 578.33±16.56578.33± 16.56 Planner Full model + CEM 57.00±1.8757.00± 1.87 0.380±0.1000.380± 0.100 0.511±0.1060.511± 0.106 67.00±0.9467.00± 0.94 Table 2: Nominal planning performance across benchmark tasks. Success is reported in percentage, and Dist. denotes minimum distance to the goal. Task Method Success (%) ↑ Dist. ↓ Task Method Success (%) ↑ Dist. ↓ Reacher Ours 83.5±4.06 83.5± 4.06 0.377±0.1190.377± 0.119 Rope Ours 93.75±2.71 93.75± 2.71 0.020±0.0020.020± 0.002 Le-WM 17.0±7.617.0± 7.6 1.305±1.1091.305± 1.109 Le-WM 13.0±4.313.0± 4.3 0.093±0.0150.093± 0.015 DINO-WM 24.5±6.224.5± 6.2 0.533±0.1150.533± 0.115 DINO-WM 25.0±7.9125.0± 7.91 0.058±0.0020.058± 0.002 PLDM 40.5±9.940.5± 9.9 0.821±0.1580.821± 0.158 PLDM 7.0±5.347.0± 5.34 0.130±0.0190.130± 0.019 Cube Ours 91.5±4.2 91.5± 4.2 0.051±0.0550.051± 0.055 Push-T Ours 51.54±4.40 51.54± 4.40 29.86±35.5529.86± 35.55 Le-WM 28.5±3.828.5± 3.8 0.133±0.0230.133± 0.023 Le-WM 1.5±2.001.5± 2.00 113.61±10.25113.61± 10.25 DINO-WM 69.0±2.9569.0± 2.95 0.054±0.0480.054± 0.048 DINO-WM 2.52±0.042.52± 0.04 112.58±3.36112.58± 3.36 PLDM 28.75±6.6228.75± 6.62 0.145±0.0050.145± 0.005 PLDM 1.50±1.371.50± 1.37 126.07±5.26126.07± 5.26 5.1 Nominal Planning Reacher On Reacher, our nominal planner achieves the strongest performance, reaching 83.50%83.50\% success. Detailed numbers can be found in Table 2. The ablations in Table 1 show that this improvement is not due to a single component: using a non-Markov history-based predictor severely degrades performance, indicating that the learned Markov latent state is important for planning, while removing the multi-step dynamics loss also weakens goal reaching by reducing rollout accuracy. Replacing iLQR with CEM further lowers success, suggesting that although sampling-based planning can find reasonable trajectories, gradient-based planning better exploits the local linearity of our learned latent dynamics. Cube For OGBench Cube, we initialize each planning from an oracle-grasped state, so the experiment isolates the post-grasp transport problem of moving the cube to the target. Under this setup, our nominal planning achieves the strongest nominal planning performance, reaching 91.5%91.5\% success, while the best baseline, DINO-WM, only reaches 69.0%69.0\%. Detailed numbers can be found in Table 2. This indicates that the learned Markov latent dynamics together with gradient-based planning more reliably transports the grasped object to the target than prior world-model planners. Figure 3: Nominal planning; Push-T. Nominal planning with the T successfully reaching the target. Our planner executes complex maneuvers (including making and breaking contact with the T), to successfully reach the goal state. A denser set of keyframes is given in Fig. 8. Push-T Push-T is the most difficult nominal planning task in our evaluation, as it requires long-horizon contact-rich pushing with accurate control of both block position and orientation. Thus, all methods achieve lower performance compared with the other tasks. Nevertheless, our method maintains the strongest performance, achieving 51.54±4.40%51.54± 4.40\% success and the lowest minimum block-position error of 29.86±35.5529.86± 35.55 pixels. Figure 3 provides a sample rollout demonstrating the extensive maneuvers taken by our planner to successfully push the T to the goal. In contrast, DINO-WM and PLDM achieve only 2.52±0.04%2.52± 0.04\% and 1.50±1.37%1.50± 1.37\% success, respectively, with much larger minimum distances. These results show that, even when the task difficulty causes overall success rates to decrease, our learned Markov latent dynamics and gradient-based planner still provide the most reliable planning signal for contact-rich manipulation. See App. E for more Push-T baselines. Rope Lastly, we evaluate on a customized two-manipulator rope task in MuJoCo, where two 7-DoF KUKA iiwa arms are mounted on opposite sides of a table and hold a rope between their end-effectors (Figure 1). Since this task is not a standard benchmark, we define a three-dimensional task state using the rope reach, height, and width, which determines the target rope geometry through symmetric target positions for the two attachment points. In this task, we seek to manipulate the deformable rope from the initial state to the desired rope configuration from the goal image. Under this setting, our nominal planning achieves 93.75%93.75\% success, while the strongest baseline, DINO-WM, reaches only 25.0%25.0\%. Table 2 demonstrates that the learned Markov latent dynamics and gradient-based planner are especially effective in the bimanual rope setting, where small errors from either arm can significantly alter the final rope geometry. Hardware Rope To validate our nominal planner, we replicated the rope task in the real world. Hardware implementation details are in Appendix D.6. In our hardware results, our planner successfully stretches the rope from the start image (MPC step 0) to the goal image (MPC step 40) in Figure 4(a). In another trial (Figure 12, Appendix E), our latent space planner successfully moved the rope forward. Figure 4(b) and (c) plot the task space error between the configuration at each MPC step and the goal configuration and the error between the latent vector and goal latent vector, indicating a clear correlation between latent space goal error and task space error. Figure 4: Hardware Rope. (a) Time-lapse of iLQR-based long-horizon planning on real hardware directly from pixels without state estimation. In this example, the bimanual setup has to stretch the rope to reach the goal image. (b) Task and (c) latent space error plots demonstrating convergence to the desired goal state on hardware. Figure 5: Rope. Latent state rollout (dark green), with the projected robust tubes from SLS (shaded). 5.2 Robustly-Constrained Safe Latent Space MPC Figure 6: Robust planning; Rope. t-SNE visualization of the rollout. Our tubes stay predominantly in the safe region; the small overlap with the obstacle region is due to the lossy 2D t-SNE projection. Rope For the rope task, we introduce an ellipsoidal obstacle in the center of the task space (details in Appendix D) that the rope must avoid while moving from the start position to a goal position. In Table 3, we find that SLS2SLS^2 is the only method that always avoids the obstacle despite perturbations due to latent model error (Figure 1). Conversely, the value function in the HJ-filtered baseline fails to predict safe and unsafe states, colliding with the ellipsoid. Similarly, the constrained iLQR baselines show that non-robust iLQR incurs safety violations by not accounting for modeling errors. Furthermore, SLS2 has the highest rate for robust task completion with a rate of 88.57%88.57\%. The rollout in the latent space (Figure 5) shows that our trajectories are contained within the projected tubes. Lastly, Figure 6 provides a t-SNE [60] visualization of the rollout against the obstacle region, demonstrating the robust but not overly conservative tubes. Reacher For Reacher, we introduce joint angle constraints between -2.88 and -2.45 radians on the second joint, which must be robustly satisfied to complete the task (Figure 2). Both our robust method, SLS2, and the HJ-filter baseline, using our gradient-based nominal iLQR planner as the nominal policy, robustly complete the task with a 100%100\% success rate. In contrast, the constrained iLQR variants remain safe but fail to complete the task in one instance. Notably, constrained iLQR and SLS2 have comparable solve times, demonstrating that the robust constraint tightenings in (14e) introduce limited computational overhead. Table 3: We evaluate SLS2 against baseline approaches for robust task completion. For robust task completion where trajectories are safe and reach the goal, SLS2 performs the best compared to all baselines including the constrained iLQR (i.e., SLS2 without robust constraints) with or without the ID constraint (14f), nominal iLQR (without any constraints), the HJ-filtered, and LPB baselines. Task Method Safety (%) ↑ Success (%) ↑ Robust Success (%) ↑ Solve time/step (s) ↓ Reacher Ours (SLS2) 100.00 100.00 100.00 0.360 ± 0.002 Constrained iLQR w/ latent ID 100.00 97.22 97.22 0.360 ± 0.002 Constrained iLQR w/o latent ID 100.00 97.22 97.22 0.363 ± 0.003 HJ-filtered 100.00 100.0 100.0 – LPB 0.00 100.00 0.00 – Nominal iLQR 0.00 100.0 0.00 – OGBench (Cube) Ours (SLS2) 94.29 97.14 94.29 0.359 ± 0.016 Constrained iLQR w/ latent ID 30.56 86.11 27.78 0.345 ± 0.016 Constrained iLQR w/o latent ID 25.00 91.67 22.22 0.349 ± 0.013 HJ-filtered 0.00 88.57 0.00 – LPB 0.00 85.71 0.00 – Nominal iLQR 0.00 82.86 0.00 – Rope Ours (SLS2) 100.00 88.57 88.57 0.650 ± 0.042 Constrained iLQR w/ latent ID 69.44 91.67 61.11 0.597 ± 0.026 Constrained iLQR w/o latent ID 77.78 91.67 69.44 0.593 ± 0.019 HJ-filtered 37.14 100.00 37.14 – LPB 57.15 100.00 57.15 – Nominal iLQR 54.29 100.00 54.29 – Cube For the OGBench Cube task, we enforce that the gripper height must be below 9 cm after the arm has grasped the cube. Due to the model training, the arm has a natural tendency to lift the cube, thus causing the nominal-iLQR baseline to fail (Figure 7(b)). Across the remaining baselines, SLS2 has the highest robust task completion rate for OGBench Cube by keeping the cube below the threshold (Figure 7(a)), indicating its ability to robustly complete tasks. Furthermore, by shifting the planning behavior from lifting the cube unnecessarily high, this experiment shows the utility of SLS2 as a tool for post-hoc behavior alteration for world-model planning. Figure 7: Robust planning; Cube. Time-lapse of the rollout using (a) safe planner (SLS2) versus the (b) unsafe planner (nominal-iLQR). SLS2 maintains the cube below the height threshold (red line) while reaching the goal, unlike the nominal (unsafe) planner. 6 Discussion, Limitations, and Conclusion We propose SLS2, a method for probabilistically-safe latent space planning using gradient-based planners for robust visuomotor control. Methodologically, we provided an approach rigorously grounded in the uncertainty quantification and propagation tools of conformal prediction and system level synthesis. Empirically, we improve task and robust task success relative to baselines for both nominal planning and robust planning, while ensuring all components of our method, including robust planning, exploit GPU acceleration for scalability. Limitations: Performance on environments requiring complex, sequential maneuvers, such as Push-T, remains a bottleneck because of our planner’s reliance on Euclidean distance to the latent goal, which assumes an isotropic latent representation. Future work must integrate learned temporal distance metrics designed to preserve geodesic transition distances along the manifold of valid states. References Ames et al. [2019] A. D. Ames, S. Coogan, M. Egerstedt, G. Notomista, K. Sreenath, and P. Tabuada. Control barrier functions: Theory and applications. In 2019 18th European control conference (ECC), pages 3420–3431. Ieee, 2019. Mitchell et al. [2005] I. M. Mitchell, A. M. Bayen, and C. J. Tomlin. A time-dependent hamilton-jacobi formulation of reachable sets for continuous dynamic games. IEEE Transactions on automatic control, 50(7):947–957, 2005. Manchester and Slotine [2017] I. R. Manchester and J.-J. E. Slotine. Control contraction metrics: Convex and intrinsic criteria for nonlinear feedback design. IEEE Transactions on Automatic Control, 62(6):3046–3053, 2017. Chou et al. [2022] G. Chou, N. Ozay, and D. Berenson. Safe output feedback motion planning from images via learned perception modules and contraction theory. In International Workshop on the Algorithmic Foundations of Robotics, pages 349–367. Springer, 2022. Leeman et al. [2026] A. P. Leeman, S. Zhan, M. N. Zeilinger, and G. Chou. Vision-sls: Safe perception-based control from learned visual representations via system level synthesis. arXiv preprint arXiv:2604.24894, 2026. Chou and Tedrake [2023] G. Chou and R. Tedrake. Synthesizing stable reduced-order visuomotor policies for nonlinear systems via sums-of-squares optimization. In 2023 62nd IEEE Conference on Decision and Control (CDC), pages 624–631. IEEE, 2023. Dean et al. [2021] S. Dean, A. Taylor, R. Cosner, B. Recht, and A. Ames. Guaranteeing safety of learned perception modules via measurement-robust control barrier functions. In Conference on Robot Learning, pages 654–670. PMLR, 2021. Ha and Schmidhuber [2018] D. Ha and J. Schmidhuber. World models. arXiv preprint arXiv:1803.10122, 2(3):440, 2018. Hafner et al. [2020] D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=S1lOTC4tDS. Zhou et al. [2025] G. Zhou, H. Pan, Y. Lecun, and L. Pinto. DINO-WM: World models on pre-trained visual features enable zero-shot planning. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 79115–79135. PMLR, 13–19 Jul 2025. Hansen et al. [2024] N. Hansen, H. Su, and X. Wang. Td-mpc2: Scalable, robust world models for continuous control. In International Conference on Learning Representations, volume 2024, pages 47376–47405, 2024. Castaneda et al. [2023] F. Castaneda, H. Nishimura, R. T. McAllister, K. Sreenath, and A. Gaidon. In-distribution barrier functions: Self-supervised policy filters that avoid out-of-distribution states. In Learning for Dynamics and Control Conference, pages 286–299. PMLR, 2023. Nakamura et al. [2025] K. Nakamura, L. Peters, and A. Bajcsy. Generalizing safety beyond collision-avoidance via latent-space reachability analysis. arXiv preprint arXiv:2502.00935, 2025. Anderson et al. [2019] J. Anderson, J. C. Doyle, S. H. Low, and N. Matni. System level synthesis. Annual Reviews in Control, 47:364–393, 2019. Hafner et al. [2019] D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels. In International conference on machine learning, pages 2555–2565. PMLR, 2019. Hafner et al. [2021] D. Hafner, T. P. Lillicrap, M. Norouzi, and J. Ba. Mastering atari with discrete world models. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=0oabwyZbOu. Hafner et al. [2023] D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104, 2023. Hafner et al. [2025] D. Hafner, W. Yan, and T. Lillicrap. Training agents inside of scalable world models. arXiv preprint arXiv:2509.24527, 2025. Hansen et al. [2022] N. A. Hansen, H. Su, and X. Wang. Temporal difference learning for model predictive control. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 8387–8406. PMLR, 17–23 Jul 2022. LeCun et al. [2022] Y. LeCun et al. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open Review, 62(1):1–62, 2022. Assran et al. [2023] M. Assran, Q. Duval, I. Misra, P. Bojanowski, P. Vincent, M. Rabbat, Y. LeCun, and N. Ballas. Self-supervised learning from images with a joint-embedding predictive architecture. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15619–15629, 2023. Bardes et al. [2024] A. Bardes, Q. Garrido, J. Ponce, X. Chen, M. Rabbat, Y. LeCun, M. Assran, and N. Ballas. Revisiting feature prediction for learning visual representations from video. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https://openreview.net/forum?id=QaCCuDfBk2. Featured Certification. Assran et al. [2025] M. Assran, A. Bardes, D. Fan, Q. Garrido, R. Howes, M. Muckley, A. Rizvi, C. Roberts, K. Sinha, A. Zholus, et al. V-jepa 2: Self-supervised video models enable understanding, prediction and planning. arXiv preprint arXiv:2506.09985, 2025. Sobal et al. [2026] U. Sobal, W. Zhang, K. Cho, R. Balestriero, T. G. Rudner, and Y. LeCun. Learning from reward-free offline data: A case for planning with latent dynamics models. Advances in Neural Information Processing Systems, 38:43905–43941, 2026. Oquab et al. [2023] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. Micheli et al. [2023] V. Micheli, E. Alonso, and F. Fleuret. Transformers are sample-efficient world models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=vhFu1Acb0xb. Alonso et al. [2024] E. Alonso, A. Jelley, V. Micheli, A. Kanervisto, A. Storkey, T. Pearce, and F. Fleuret. Diffusion for world modeling: Visual details matter in atari. Advances in Neural Information Processing Systems, 37:58757–58791, 2024. Micheli et al. [2024] V. Micheli, E. Alonso, and F. Fleuret. Efficient world models with context-aware tokenization. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=BiWIERWBFX. Bruce et al. [2024] J. Bruce, M. D. Dennis, A. Edwards, J. Parker-Holder, Y. Shi, E. Hughes, M. Lai, A. Mavalankar, R. Steigerwald, C. Apps, et al. Genie: Generative interactive environments. In Forty-first International Conference on Machine Learning, 2024. Maes et al. [2026] L. Maes, Q. L. Lidec, D. Scieur, Y. LeCun, and R. Balestriero. Leworldmodel: Stable end-to-end joint-embedding predictive architecture from pixels. arXiv preprint arXiv:2603.19312, 2026. Ames et al. [2016] A. D. Ames, X. Xu, J. W. Grizzle, and P. Tabuada. Control barrier function based quadratic programs for safety critical systems. IEEE Transactions on Automatic Control, 62(8):3861–3876, 2016. Fisac et al. [2018] J. F. Fisac, A. K. Akametalu, M. N. Zeilinger, S. Kaynama, J. Gillula, and C. J. Tomlin. A general safety framework for learning-based control in uncertain robotic systems. IEEE Transactions on Automatic Control, 64(7):2737–2752, 2018. Wabersich and Zeilinger [2021] K. P. Wabersich and M. N. Zeilinger. A predictive safety filter for learning-based control of constrained nonlinear dynamical systems. Automatica, 129:109597, 2021. Wabersich et al. [2023] K. P. Wabersich, A. J. Taylor, J. J. Choi, K. Sreenath, C. J. Tomlin, A. D. Ames, and M. N. Zeilinger. Data-driven safety filters: Hamilton-jacobi reachability, control barrier functions, and predictive methods for uncertain systems. IEEE Control Systems Magazine, 43(5):137–177, 2023. Robey et al. [2020] A. Robey, H. Hu, L. Lindemann, H. Zhang, D. V. Dimarogonas, S. Tu, and N. Matni. Learning control barrier functions from expert demonstrations. In 2020 59th IEEE Conference on Decision and Control (CDC), pages 3717–3724. Ieee, 2020. Lindemann et al. [2024] L. Lindemann, A. Robey, L. Jiang, S. Das, S. Tu, and N. Matni. Learning robust output control barrier functions from safe expert demonstrations. IEEE Open Journal of Control Systems, 3:158–172, 2024. Kang et al. [2022] K. Kang, P. Gradu, J. J. Choi, M. Janner, C. Tomlin, and S. Levine. Lyapunov density models: Constraining distribution shift in learning-based control. In International Conference on Machine Learning, pages 10708–10733. PMLR, 2022. Faroni et al. [2025] M. Faroni, C. Odesco, A. M. Zanchettin, and P. Rocco. Uncertainty-aware planning with inaccurate models for robotized liquid handling. In 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 17725–17731. IEEE, 2025. Cao et al. [2025] M. E. Cao, M. Bloch, and S. Coogan. An optimistic approach to cost-aware predictive control. Automatica, 176:112263, 2025. Chou et al. [2021] G. Chou, N. Ozay, and D. Berenson. Model error propagation via learned contraction metrics for safe feedback motion planning of unknown systems. In 2021 60th IEEE Conference on Decision and Control (CDC), pages 3576–3583. IEEE, 2021. Knuth et al. [2021] C. Knuth, G. Chou, N. Ozay, and D. Berenson. Planning with learned dynamics: Probabilistic guarantees on safety and reachability via lipschitz constants. IEEE Robotics and Automation Letters, 6(3):5129–5136, 2021. Knuth et al. [2023] C. Knuth, G. Chou, J. Reese, and J. Moore. Statistical safety and robustness guarantees for feedback motion planning of unknown underactuated stochastic systems. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 12700–12706. IEEE, 2023. Suh et al. [2023] H. T. Suh, G. Chou, H. Dai, L. Yang, A. Gupta, and R. Tedrake. Fighting uncertainty with gradients: Offline reinforcement learning via diffusion score matching. In Conference on Robot Learning, pages 2878–2904. PMLR, 2023. Dodeja et al. [2025] L. Dodeja, K. Schmeckpeper, S. Vats, T. Weng, M. Jia, G. Konidaris, and S. Tellex. Accelerating residual reinforcement learning with uncertainty estimation. IEEE Robotics and Automation Letters, 11(1):970–977, 2025. Chua et al. [2018] K. Chua, R. Calandra, R. McAllister, and S. Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. Advances in neural information processing systems, 31, 2018. Seo et al. [2025] J. Seo, K. Nakamura, and A. Bajcsy. Uncertainty-aware latent safety filters for avoiding out-of-distribution failures. arXiv preprint arXiv:2505.00779, 2025. Nakamura et al. [2025] K. Nakamura, A. L. Bishop, S. Man, A. M. Johnson, Z. Manchester, and A. Bajcsy. How to train your latent control barrier function: Smooth safety filtering under hard-to-model constraints. arXiv preprint arXiv:2511.18606, 2025. Anand and Kolathaya [2025] M. Anand and S. Kolathaya. Safety certification in the latent space using control barrier functions and world models. arXiv preprint arXiv:2507.13871, 2025. Sun and Song [2026] Z. Sun and S. Song. Latent policy barrier: Learning robust visuomotor policies by staying in-distribution. Advances in Neural Information Processing Systems, 38:174280–174305, 2026. Mao et al. [2024] Z. Mao, C. Sobolewski, and I. Ruchkin. How safe am i given what i see? calibrated prediction of safety chances for image-controlled autonomy. In 6th Annual Learning for Dynamics & Control Conference, pages 1370–1387. PMLR, 2024. Balestriero and LeCun [2025] R. Balestriero and Y. LeCun. Lejepa: Provable and scalable self-supervised learning without the heuristics. arXiv preprint arXiv:2511.08544, 2025. Wang et al. [2026] Y. Wang, O. Bounou, G. Zhou, R. Balestriero, T. G. Rudner, Y. LeCun, and M. Ren. Temporal straightening for latent planning. arXiv preprint arXiv:2603.12231, 2026. Tibshirani et al. [2019] R. J. Tibshirani, R. Foygel Barber, E. Candes, and A. Ramdas. Conformal prediction under covariate shift. Advances in neural information processing systems, 32, 2019. Lei and Wasserman [2014] J. Lei and L. Wasserman. Distribution-free prediction bands for non-parametric regression. Journal of the Royal Statistical Society Series B: Statistical Methodology, 76(1):71–96, 2014. Fang and Chou [2026] J. Fang and G. Chou. Safe large-scale robust nonlinear mpc in milliseconds via reachability-constrained system level synthesis on the gpu. arXiv preprint arXiv:2604.07644, 2026. Srinivasan et al. [2026] A. Srinivasan, A. Leeman, and G. Chou. Safety beyond the training data: Robust out-of-distribution mpc via conformalized system level synthesis. In 8th Annual Learning for Dynamics and Control Conference, 2026. Tassa et al. [2018] Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. d. L. Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690, 2018. Park et al. [2025] S. Park, K. Frans, B. Eysenbach, and S. Levine. Ogbench: Benchmarking offline goal-conditioned rl. In International Conference on Learning Representations, volume 2025, pages 94937–94982, 2025. Li and Todorov [2004] W. Li and E. Todorov. Iterative linear quadratic regulator design for nonlinear biological movement systems. In First International Conference on Informatics in Control, Automation and Robotics, volume 2, pages 222–229. SciTePress, 2004. Van der Maaten and Hinton [2008] L. Van der Maaten and G. Hinton. Visualizing data using t-sne. Journal of machine learning research, 2008. Williams et al. [2017] G. Williams, A. Aldrich, and E. A. Theodorou. Model predictive path integral control: From theory to parallel computation. Journal of Guidance, Control, and Dynamics, 40(2):344–357, 2017. Tedrake and the Drake Development Team [2019] R. Tedrake and the Drake Development Team. Drake: Model-based design and verification for robotics, 2019. URL https://drake.mit.edu. Appendices In the following, we provide an overview of our appendices. In App. A, we provide training details for the latent world models, including the offline data collection procedure, shared HDF5 dataset format, optimization settings, and task-specific latent dynamics network architectures. In App. B, we review the system level synthesis (SLS) formulation used to compute robust feedback policies and reachable tubes for the learned latent dynamics, and describe how nominal constraints are tightened to account for calibrated model error. In App. C, we prove the probabilistic containment guarantee for the conformal SLS tubes. In App. D, we provide experimental implementation details, including the benchmark setup, nominal planning objectives, world-model baselines, implementation of Problem (14), calibrated model-error bounds, latent constraint classifiers, in-distribution latent support constraints, safety baselines, and the real-world rope manipulation setup. In App. E, we provide additional experimental results, including nominal trajectory time-lapses, safe versus unsafe trajectory comparisons, latent reachable-tube visualizations, and expanded quantitative tables for nominal planning and robust task-completion baselines. Appendix A Training Details Offline Dataset We train the latent world model from offline trajectories collected independently for each task. Each trajectory consists of a sequence of image observations and actions, τ=(ot,at)t=0T−1,τ=\(o_t,a_t)\_t=0^T-1, (15) where oto_t is a rendered RGB observation and ata_t is the action applied between consecutive observations. The datasets are collected before world model training and no online interaction is used during model learning. The world model is trained only from the observation-action sequences, while additional simulator states, rewards, and task-specific quantities are stored for analysis, visualization, and evaluation. Across environments, we use task-specific data-generation policies to produce trajectories with sufficient interaction coverage. For expert-control tasks, such as Reacher, trajectories are collected by rolling out a trained expert policy. For manipulation tasks that require structured behavior, such as OGBench Cube, trajectories are generated using an oracle planner that constructs task-space keyframes and executes the corresponding control sequence. For the two-manipulator rope environment, we use randomized task-space trajectory generators, including spline- and waypoint-based policies, to cover a diverse range of rope configurations and manipulator motions. This produces datasets that are not tied to a single evaluation goal, but instead expose the world model to local dynamics over a broad set of feasible interactions. All datasets are stored in a common HDF5 format. For each episode, we store the rendered pixel observations, applied actions, episode length, episode offset, episode seed, and per-step episode/step indices. Actions are padded with a final invalid entry so that the action array has the same temporal length as the observation array; this makes it straightforward to sample fixed-length training windows while ignoring the last action of each episode. Episodes shorter than a minimum length are discarded. The collection loop either runs for a fixed number of episodes or until a specified target number of transitions is reached. This common storage format allows the same world-model training pipeline to be reused across all environments. Training the World Model Across experiments, we train the model using AdamW with learning rate 5×10−55× 10^-5, weight decay 10−310^-3, mixed-precision training, and gradient clipping with maximum norm 1.01.0. Model checkpoints are saved after each epoch, and the same general training pipeline is used across all tasks, with only task-dependent choices such as latent dimension, Markov order, rollout horizon, and loss weights adjusted for each environment. MLP Latent Dynamics Models All four tasks use the same vision backbone: a tiny ViT encoder on 224×224224× 224 RGB images with patch size 1414. The encoder output is projected to a task-specific latent zt∈ℝnzz_t ^n_z. The dynamics model then forms a Markov latent state st=[zt,Δzt,…,ΔKzt]∈ℝ(K+1)nz,s_t=[z_t, z_t,…, ^Kz_t] ^(K+1)n_z, where K is the desired Markov order. A feed-forward MLP predicts the next Markov state from the current Markov state and action, s^t+1=fϕ(st,ut). s_t+1=f_φ(s_t,u_t). Here the listed action dimension is the number of control channels concatenated into the predictor input; since all runs use frameskip 11, this is also the effective action dimension. Reacher and Push-T use two-dimensional actions, Rope uses a three-dimensional endpoint action, and OGBench Cube uses the five-dimensional environment action vector. The predictor is trained by autoregressively rolling this one-step MLP for num_preds=5 future steps. Table 4: Model architecture and state dimensions across tasks. Task nzn_z K dim(st) (s_t) dim(vt) (v_t) MLP Size Straightening OGBench-Cube 1212 11 2424 55 512×2512×2 yes Reacher 55 11 1010 22 512×2512×2 no Push-T 4848 22 144144 22 512×3512×3 no Rope 1212 11 2424 33 512×2512×2 yes The MLP column reports hidden width times number of hidden layers, with GELU nonlinearities and a linear output layer. The MLP size column only considers the latent dynamics predictor, excluding the ViT encoder and latent projector. The SIGReg term uses the same sketch size in all runs: 1717 quadrature knots and 10241024 random projections, weighted by λSIG=0.005 _SIG=0.005. Temporal straightening is active only for the OGBench Cube and Rope runs. Appendix B System Level Synthesis System Level Synthesis. We use state-feedback SLS to compute robust feedback policies and reachable tubes for the learned latent dynamics. Let sk∈ℝnss_k ^n_s be the Markov latent state and uk∈ℝnuu_k ^n_u the control. Around a nominal trajectory (ζ0:T,v0:T−1)( _0:T,v_0:T-1) satisfying ζk+1=fϕ(ζk,vk) _k+1=f_φ( _k,v_k), we linearize Δsk+1=AkΔsk+BkΔuk+Ekwk,wk∈ℬns, s_k+1=A_k s_k+B_k u_k+E_kw_k, w_k ^n_s, where Ak=∇sfϕ(ζk,vk)A_k= _sf_φ( _k,v_k), Bk=∇ufϕ(ζk,vk)B_k= _uf_φ( _k,v_k), and EkℬnsE_kB^n_s is a calibrated latent prediction-error set. SLS parameterizes the closed-loop system by response matrices Φk,js ^s_k,j and Φk,ju ^u_k,j such that sk−ζk=∑j=0k−1Φk,jswj,uk−vk=∑j=0k−1Φk,juwj.s_k- _k= _j=0^k-1 ^s_k,jw_j, u_k-v_k= _j=0^k-1 ^u_k,jw_j. These responses are valid if they satisfy the finite-horizon SLS constraints Φk+1,js=AkΦk,js+BkΦk,ju,j<k,Φk+1,ks=Ek. ^s_k+1,j=A_k ^s_k,j+B_k ^u_k,j, j<k, ^s_k+1,k=E_k. Then defining ⊕ and ⨁ as the Minkowski sum of two sets (i.e., ⊕ℬ=a+b∣a∈,b∈ℬA =\a+b a ,b \ for sets, A and ℬB), the corresponding reachable tubes are ℛks=ζk⊕⨁j=0k−1Φk,jsℬns,ℛku=vk⊕⨁j=0k−1Φk,juℬns.R_k^s= _k _j=0^k-1 ^s_k,jB^n_s, _k^u=v_k _j=0^k-1 ^u_k,jB^n_s. For constraints ℱ:=(s,u)∣gi(s,u)+bi≤0,i=1,…,ncF:=\(s,u) g_i(s,u)+b_i≤ 0,\ i=1,…,n_c\, we impose robust satisfaction by tightening the nominal constraints. Linearizing gig_i around (ζk,vk)( _k,v_k) gives the sufficient condition gi(ζk,vk)+bi+∑j=0k−1‖∇sgi(ζk,vk)Φk,js+∇ugi(ζk,vk)Φk,ju‖2≤0,g_i( _k,v_k)+b_i+ _j=0^k-1 \| _sg_i( _k,v_k) ^s_k,j+ _ug_i( _k,v_k) ^u_k,j \|_2≤ 0, for all i and k. The SLS MPC problem jointly optimizes (ζ,v,Φs,Φu)(ζ,v, ^s, ^u) subject to nominal dynamics, the SLS response constraints, and these tightened constraints. Appendix C Proofs Theorem C.1 (Probabilistic containment of conformal SLS tubes). Consider the latent closed-loop dynamics sk+1=fϕ(sk,uk)+eks_k+1=f_φ(s_k,u_k)+e_k, for k=0,…,T−1k=0,…,T-1. Let q be the split conformal quantile of the calibration scores ri=ei⊤Σψ(si,ui)−1eir_i=e_i _ψ(s_i,u_i)^-1e_i and ei:=si+−fϕ(si,ui)e_i:=s_i^+-f_φ(s_i,u_i), computed at per-step miscoverage δ¯=δ/T δ=δ/T. For each closed-loop transition, define the test score rktest:=ek⊤Σψ(sk,uk)−1ekr_k^test:=e_k _ψ(s_k,u_k)^-1e_k. Assume that, for each k, rktestr_k^test is exchangeable with the calibration scores. Let Σψ(s,u)=Lψ(s,u)Lψ(s,u)⊤ _ψ(s,u)=L_ψ(s,u)L_ψ(s,u) and define the local conformal error set ℰ(s,u):=qLψ(s,u)ℬnsE(s,u):= q\,L_ψ(s,u)B^n_s as defined in (10). Suppose the SLS controller is synthesized with disturbance matrices EkE_k such that the realized conformal error sets are contained in the SLS disturbance sets, ℰ(sk,uk)⊆EkℬnsE(s_k,u_k) E_kB^n_s, for k=0,…,T−1k=0,…,T-1. If the SLS controller is robustly correct for all disturbances wk∈ℬnsw_k ^n_s in Δsk+1=AkΔsk+BkΔuk+Ekwk s_k+1=A_k s_k+B_k u_k+E_kw_k, where Δsk:=sk−ζk s_k:=s_k- _k and Δuk:=uk−vk u_k:=u_k-v_k, then ℙ[(sk∈ℛks,∀k=0,…,T)∧(uk∈ℛku,∀k=0,…,T−1)]≥1−δP [(s_k _k^s,\;∀ k=0,…,T) \;(u_k _k^u,\;∀ k=0,…,T-1) ]≥ 1-δ. Proof. By split CP, exchangeability gives ℙ[rktest≤q]≥1−δ¯P [r_k^test≤ q ]≥ 1- δ for each k. Since rktest≤q⟺ek∈qLψ(sk,uk)ℬns=ℰ(sk,uk),r_k^test≤ q e_k∈ q\,L_ψ(s_k,u_k)B^n_s=E(s_k,u_k), a union bound over the T transitions yields ℙ[ek∈ℰ(sk,uk),∀k=0,…,T−1]≥1−Tδ¯=1−δ.P [e_k (s_k,u_k),\ ∀ k=0,…,T-1 ]≥ 1-T δ=1-δ. On this event, the assumed containment ℰ(sk,uk)⊆EkℬnsE(s_k,u_k) E_kB^n_s implies that each realized prediction error can be written as ek=Ekwk,wk∈ℬns.e_k=E_kw_k, w_k ^n_s. Thus the realized closed-loop trajectory is generated by a disturbance sequence inside the uncertainty set used by SLS. Robust correctness of SLS then implies sk∈ℛkss_k _k^s for k=0,…,Tk=0,…,T and uk∈ℛkuu_k _k^u for k=0,…,T−1k=0,…,T-1. Therefore the tube-containment event holds with probability at least 1−δ1-δ. ∎ Appendix D Experimental Implementation In the following sections, we discuss the experimental implementations for two of our central contributions: our nominal planner and our robust planner. Section D.1 discusses the experimental setup for our nominal experiments. Then, Section D.2 discusses the implementation details of our novel gradient-based iLQR planner, which leverages the Markovian latent space to enable high task success rates (cf. Table 1 for the ablation study). Next, Section D.3 details the baseline implementations used to evaluate our nominal planner against. For the safety experiments, Section D.4 provides details on how our robust planner, SLS2, is implemented, and Section D.5 provides details on the safety baselines we compare our approach to. Lastly, Section D.6 discusses our real-world hardware experiment. D.1 Experiment Setup We evaluate on four continuous-control benchmarks spanning low-dimensional reaching, rigid-object manipulation, planar pushing, and deformable-object manipulation. For each task, we train from expert demonstrations and evaluate planning using task-specific goal-reaching criteria. Reacher. Reacher is the hard variant of the DeepMind Control Suite Reacher task, where a two-joint planar arm must reach a target position. We render 224×224224× 224 RGB observations with the target hidden from the image observation. The dataset contains 24,650 expert episodes collected with a Soft Actor-Critic policy. Episodes are collected at 50 Hz for up to 100 control steps and terminate early when the end effector reaches within 0.030.03 of the target. Cube. Cube is the OGBench cube-single-v0 task, where a robot arm must grasp and move a cube to a target 3D position and yaw. We use front-camera pixel observations and continuous 5D actions. The dataset contains 11,227 oracle-planner episodes recorded at 20 Hz for up to 100 steps. Episodes terminate when the cube is within 0.040.04 m of the target position and within 0.20.2 rad of the target yaw. Push-T. Push-T is a 2D planar manipulation task where a circular pusher moves a T-shaped block to a target pose. We use gym-pusht/PushT-v0 and store RGB observations, pusher positions, and continuous 2D actions. The dataset contains 25,000 expert episodes: 20,000 nominal diffusion-policy rollouts and 5,000 edge-case rollouts. Episodes contain up to 167 recorded steps. Rope. Rope is a bimanual MuJoCo manipulation environment where two robot arms control the endpoints of a deformable rope. The task exposes the model to nonlinear rope dynamics by moving the rope attachments along randomized task-space trajectories. We collect 15,000 episodes using a cubic-spline policy with six randomized waypoints. Episodes are recorded at 20 Hz, average approximately 80 steps, and are rendered as 224×224224× 224 RGB observations. Planning success criteria. For Reacher, success is measured by the wrapped joint-position distance between the current and goal configurations. The primary threshold is 0.10.1 in qpos distance, and we also report a stricter full-observation threshold of 0.050.05 as a diagnostic. For Cube, success requires the cube position to be within 0.040.04 m of the goal position. For Push-T, success requires the T-block to be within 2020 pixels of the goal position with yaw error at most 0.350.35 rad. For Rope, success requires the task-target distance to reach the dataset goal tolerance of 0.050.05. D.2 Nominal Planning Setup The nominal MPC planners optimize in the learned Markov latent state. For Reacher, OGBench-Cube, and Rope, the nominal planner is receding-horizon iLQR with horizon T=15T=15, but the quadratic cost matrices are task-specific (where weights scale the identity matrix In∈ℝn×nI_n ^n× n): Table 5: Reacher, OGBench-Cube, Rope: Nominal iLQR planning horizons and cost weights. Task T QTQ_T Q R Reacher 1515 5.0I105.0I_10 0.05I100.05I_10 0.1I20.1I_2 OGBench-Cube 1515 15.0I2415.0I_24 0.05I240.05I_24 0.5I50.5I_5 Rope 1515 15.0I2415.0I_24 0.005I240.005I_24 0.01I30.01I_3 For these iLQR-only planners, the rollout is generated by the learned one-step dynamics ζt+1=fϕ(ζt,vt) _t+1=f_φ( _t,v_t), and the open-loop control sequence v is optimized with the finite-horizon cost JiLQR(,)=∑t=0T−1(‖ζt−ζg‖Q2+‖vt‖R2)+‖ζT−ζg‖QT2,J_iLQR( ζ,v)= _t=0^T-1 (\| _t- _g\|_Q^2+\|v_t\|_R^2 )+\| _T- _g\|_Q_T^2, (16) where ζg _g is the goal latent state. In Reacher, the same expression is used but the cost is computed only on the zeroth-order latent coordinates (i.e., z), so the cost does not directly penalize the finite-difference components of the Markov state. At each environment step, iLQR linearizes the learned dynamics along the current rollout, performs a Riccati-style backward pass to compute local feedback/feedforward updates, line-searches the updated control sequence, executes the first action, and replans from the newly observed image. For Push-T, the nominal planner uses MPPI to produce a long-horizon warm-start and then uses a short-horizon iLQR tracker to refine the first segment. The MPPI horizon is Hmppi=45H^mppi=45, while the iLQR tracking horizon is T=15T=15. The MPPI cost matrices are QTmppi=10.0I144Q_T^mppi=10.0I_144, Qmppi=0.05I144Q^mppi=0.05I_144, and Rmppi=0.01I2R^mppi=0.01I_2; the iLQR tracking matrices are QT=10.0I144Q_T=10.0I_144, Q=1.0I144Q=1.0I_144, and R=0.01I2R=0.01I_2. MPPI samples noisy control sequences, rolls them out through the same latent dynamics, and scores each sampled trajectory with JMPPI(,)=∑t=0Hmppi−1(‖ζt−ζg‖Qmppi2+‖vt‖Rmppi2)+‖ζHmppi−ζg‖QTmppi2.J_MPPI( ζ,v)= _t=0^H^mppi-1 (\| _t- _g\|_Q^mppi^2+\|v_t\|_R^mppi^2 )+\| _H^mppi- _g\|_Q_T^mppi^2. (17) The sampled costs are converted to rewards by negation and weighted by the MPPI reward temperature when updating the nominal control sequence. The iLQR tracker then optimizes over the shorter horizon T=15T=15 against the MPPI reference trajectory ¯ ζ rather than a fixed goal: Jtrack(,)=∑t=0T−1(‖ζt−ζ¯t‖Q2+‖vt‖R2)+‖ζT−ζ¯T‖QT2.J_track( ζ,v)= _t=0^T-1 (\| _t- ζ_t\|_Q^2+\|v_t\|_R^2 )+\| _T- ζ_T\|_Q_T^2. (18) Thus the MPPI stage adds a sampling-based global search over long control sequences, while the iLQR stage locally smooths and tracks the selected latent trajectory before receding-horizon execution. D.3 World Model Baseline Implementation Training and evaluation protocol. This section describes the implementation details for the world-model baselines and our planning pipeline. Across all tasks, we follow the training and evaluation workflow of the stable-worldmodel codebase whenever possible. We compare against PLDM, DINO-WM, and Le-WM. Unless otherwise specified, each baseline is trained for 1010 epochs following the Le-WM training protocol [30], where this training length was reported to provide strong planning performance. When task-compatible pretrained or reusable checkpoints are available, we initialize from those checkpoints and fine-tune on our dataset. In particular, due to the size of the Push-T dataset and the cost of training from scratch, the Le-WM model for Push-T is fine-tuned from a pretrained Push-T object checkpoint. For the customized Rope task, no pretrained checkpoints are available, so all world models are trained from scratch using the same offline dataset. Shared preprocessing and architecture. All baselines are trained from image observations resized to 224×224224× 224 with patch size 1414. We use a history length of 33 and a one-step prediction horizon. During planning, the learned one-step latent dynamics are rolled out autoregressively to evaluate multi-step action sequences. The dataset is split into 90%90\% training and 10%10\% validation, and we apply gradient clipping with threshold 1.01.0. Unless otherwise stated, models are trained with batch size 128128. We use frame skip 55 for Reacher, Cube, and Push-T, and frame skip 11 for Rope. PLDM. For PLDM, we train with AdamW using learning rate 5×10−55× 10^-5 and weight decay 10−310^-3. The encoder is a ViT-Tiny model with embedding dimension 192192. The predictor has depth 66, 1616 attention heads, MLP dimension 20482048, head dimension 6464, and dropout 0.10.1. Proprioceptive inputs are disabled. The active loss weights are 1818 for the standard-deviation loss, 0.70.7 for the temporal standard-deviation loss, 1212 for the covariance loss, and 0.20.2 for the temporal alignment loss. DINO-WM. For DINO-WM, we use a frozen DINOv2-Small visual backbone and train the latent predictor with AdamW using learning rate 5×10−45× 10^-4 and no weight decay. The predictor uses the same architecture as PLDM: depth 66, 1616 attention heads, MLP dimension 20482048, head dimension 6464, and dropout 0.10.1. Actions are encoded with dimension 1010. Reacher, Cube, and Rope are trained without proprioceptive encoding, while Push-T uses proprioceptive encoding. Le-WM. For Le-WM, we train with AdamW using learning rate 5×10−55× 10^-5 and weight decay 10−310^-3. The model uses a ViT-Tiny encoder with image size 224224, patch size 1414, history size 33, prediction horizon 11, and embedding dimension 192192. The latent predictor uses the same configuration as the other baselines: depth 66, 1616 attention heads, MLP dimension 20482048, head dimension 6464, and dropout 0.10.1. For Rope, Le-WM is trained from scratch with batch size 128128, frame skip 11, and SigReg regularization with weight 0.090.09, 1717 knots, and 10241024 random projections. For Push-T, we fine-tune the pretrained Le-WM Push-T object checkpoint on the combined Push-T dataset. The Push-T model uses frame skip 55, action dimension 22, and effective action dimension 1010. We train for 1010 epochs with batch size 110110, bfloat16 mixed precision, gradient clipping at 1.01.0, and the Le-WM latent prediction objective. During fine-tuning, we use SigReg regularization with weight 0.0050.005, 1717 knots, and 10241024 random projections, together with temporal straightening loss with weight 0.010.01. Planning evaluation. For nominal world-model planning baselines, PLDM, DINO-WM, and Le-WM use the standard CEM planner provided by the stable-worldmodel implementation. CEM optimizes action sequences in the learned latent space and executes the selected actions in a receding-horizon manner. For our method, we use the same learned latent dynamics as the planning model but replace CEM with our gradient-based latent planner. In the robust setting, this planner is augmented with the SLS-based tube propagation and conformalized latent constraints described in the main text. This setup ensures that baseline performance reflects the default planning pipeline associated with each learned world model, while our method differs only in the planning and robustness layer built on top of the learned latent dynamics. D.4 Implementing SLS2 (Problem (14)) Below we discuss our implementation of Problem (14). For self-containment, we restate the problem: min,,Φs,Φu _ ζ,v, ^s, ^u J(,)+Jf(ζT)+H(Φs,Φu) J( ζ,v)+J_f( _T)+H( ^s, ^u) (14a) s.t. .t. ζk+1=fϕ(ζk,vk),ζ0=s¯0,k=0,…,T−1, _k+1=f_φ( _k,v_k), _0= s_0, k=0,…,T-1, (14b) Φk+1,js=AkΦk,js+BkΦk,ju,0≤j<k,k=0,…,T−1, ^s_k+1,j=A_k ^s_k,j+B_k ^u_k,j, 0≤ j<k,\ k=0,…,T-1, (14c) Φj+1,js=Ej,j=0,…,T−1, ^s_j+1,j=E_j, j=0,…,T-1, (14d) gi(ζk,vk)+bi+τi,k≤0,i=1,…,nc,k=0,…,T, g_i( _k,v_k)+b_i+ _i,k≤ 0, i=1,…,n_c,\ k=0,…,T, (14e) ℛks⊆ℐ,k=0,…,T. _k^s , k=0,…,T. (14f) In Section D.4.1 we discuss the cost and reference trajectory generation used for the objective in (14a). Next, in Section D.4.2 we discuss how the model error is calibrated. The calibrated error is used to initialize the system level constraints (SLC) in (14c)-(14d). Details on the obstacles and how we train a latent classifier and conformalize it are available in Section D.4.3. In the optimization problem, the classifier is linearized and tightened in (14e). Lastly, we discuss the in-domain ellipsoid constraint, (14f), and how it is formed in Section D.4.4. D.4.1 Defining the Objective To implement Problem (14), we follow [55, 56] and use a real-time-iteration (RTI) scheme which solves a single SQP (instead of solving several successive SQPs until convergence) to enable faster solve times. To avoid ill-conditioned problems and improve numerical stability, we use the MPPI algorithm [61] to warm-start the MPC solve at each time step. Below, we describe the MPPI warm-start objective, the nominal SLS objective, the associated cost weights, and the modifications used for the Constraint-iLQR baselines. Let ζt∈ℝns _t ^n_s denote the latent state and ζg _g the goal latent state. MPPI first generates a reference trajectory ¯:=ζ¯kk=0T ζ:=\ ζ_k\_k=0^T and ¯:=v¯kk=0T−1 v:=\ v_k\_k=0^T-1 by minimizing the sampling objective JMPPI(,)=∑t=0Hmppi−1(‖ζt−ζg‖Qmppi2+‖vt‖Rmppi2+Jsoft(ζt))+‖ζHmppi−ζg‖QTmppi2,J_MPPI( ζ,v)= _t=0^H^mppi-1 (\| _t- _g\|_Q^mppi^2+\|v_t\|_R^mppi^2+J_soft( _t) )+\| _H^mppi- _g\|_Q^mppi_T^2, (20) where JsoftJ_soft denotes task-dependent soft penalties used during MPPI sampling. The MPPI horizon HmppiH^mppi can be greater than T (the tracking horizon used by SLS) to encourage smaller actions at each step. Furthermore, we define ‖x‖M2:=x⊤Mx\|x\|_M^2:=x Mx for x∈ℝnx ^n and M∈ℝn×nM ^n× n. The nominal RTI-SQP/SLS objective then tracks the MPPI reference trajectory ¯ ζ and ¯ v over the MPC horizon T: JSLS(,)=∑t=0T−1(‖ζt−ζ¯t‖Q2+‖vt−v¯t‖R2)+‖ζT−ζ¯T‖QT2.J_SLS( ζ,v)= _t=0^T-1 (\| _t- ζ_t\|_Q^2+\|v_t- v_t\|_R^2 )+\| _T- ζ_T\|_Q_T^2. (21) Our cost matrices penalizing state tracking QT,Q,QTmppi,Qmppi⪰0Q_T,Q,Q_T^mppi,Q^mppi 0 and control penalties R,Rmppi≻0R,R^mppi 0 are summarized in Table 6. While interpretability is minimal in the latent space, we leverage the Markovian structure to apply different penalties to position states (corresponding to the zeroth-order state) and velocity states (corresponding to the first-order state). Compactly, we use In∈ℝn×nI_n ^n× n as the n-dimensional identity matrix, and for A∈ℝn×n,B∈ℝm×mA ^n× n,B ^m× m we denote blkdiag(A,B)blkdiag(A,B) as the block diagonal matrix. Table 6: MPPI warm-start and nominal SLS tracking weights used in our experiments. Task Objective Horizon State Weights Control Weight Rope MPPI 1515 Qmppi=blkdiag(0.005I12,1.0I12)Q^mppi=blkdiag(0.005I_12,1.0I_12) Rmppi=1.0I3R^mppi=1.0I_3 QTmppi=blkdiag(10I12,1.0I12)Q^mppi_T=blkdiag(10I_12,1.0I_12) SLS 1010 Q=0.1I24,QT=5.0I24Q=0.1I_24, Q_T=5.0I_24 R=0.2I3R=0.2I_3 Reacher MPPI 6060 Qmppi=blkdiag(0.005I5,1.0I5)Q^mppi=blkdiag(0.005I_5,1.0I_5) Rmppi=0.01I2R^mppi=0.01I_2 QTmppi=blkdiag(100I5,1.0I5)Q^mppi_T=blkdiag(100I_5,1.0I_5) SLS 55 Q=blkdiag(0.005I5,1.0I5)Q=blkdiag(0.005I_5,1.0I_5) R=0.1I2R=0.1I_2 QT=blkdiag(5.0I5,0.9I5)Q_T=blkdiag(5.0I_5,0.9I_5) OGBench-Cube MPPI 1010 Qmppi=blkdiag(0.05I12,0.10I12)Q^mppi=blkdiag(0.05I_12,0.10I_12) Rmppi=1.0I5R^mppi=1.0I_5 QTmppi=blkdiag(15I12,1.0I12)Q^mppi_T=blkdiag(15I_12,1.0I_12) SLS 1010 Q=blkdiag(0.05I12,0.05I12)Q=blkdiag(0.05I_12,0.05I_12) R=0.5I5R=0.5I_5 QT=blkdiag(1.0I12,1.0I12)Q_T=blkdiag(1.0I_12,1.0I_12) In (20), we also add a soft penalty, JsoftJ_ soft, defined as Jsoft(ζ)=λID[maxρID(ζ)−1,0]2⏟Latent Ellipsoid Penalty+λobs[softplus(η−cψ(Πzζ))]2⏟Obstacle PenaltyJ_soft(ζ)= _ID [ \ _ID(ζ)-1,0 \ ]^2_Latent Ellipsoid Penalty+ _obs [softplus (η-c_ψ( _zζ) ) ]^2_Obstacle Penalty (22) where ρID(ζ) _ID(ζ) is the conformal latent ellipsoid defined in (11) and cψ(Πzζ)c_ψ( _zζ) is the obstacle classifier score evaluated on the zeroth-order latent component. λobs _ obs and λID _ID are hyperparameters controlling the strength of the respective penalties. The soft penalty in (22) encourages the reference trajectory to satisfy the constraints. Notably, these references are not guaranteed to robustly satisfy the constraints, and thus SLS is still required to enforce robust constraint satisfaction. The remaining MPPI hyperparameters alongside the values of λobs _ obs and λID _ID used for each task are summarized in Table 7. Table 7: MPPI hyperparameters and soft-penalty weights used for warm-start trajectory generation. Task Samples Updates Reward Weight Noise Scale β λID _ID λobs _obs Rope 2048 6 25 0.20 0.65 20 0 Reacher 512 5 20 0.15 0.70 0 1000 OGBench-Cube 512 5 30 0.20 0.60 10 100 D.4.2 Calibrated Error Bounds To calibrate the error bounds we split our dataset calD_ cal in half (i.e., cal(1)D_ cal^(1) and cal(2)D_ cal^(2) such that |cal(1)|=|cal(2)||D_ cal^(1)|=|D_ cal^(2)|. For our experiments, calD_ cal is constructed using the same expert rollouts used to train the models, as the transitions encountered in calD_ cal are distributionally similar to the transitions encountered during planning time, preserving the exchangeability assumption. For Reacher, Rope, and OGBench-Cube, |cal|=200,000|D_ cal|=200,000, |cal|=80,000|D_ cal|=80,000, and |cal|=98,800|D_ cal|=98,800 respectively. The first partition, cal(1)D_ cal^(1), is used to learn the error covariance, Σψ(s,u) _ψ(s,u), structure by minimizing the MGNLL loss (9), while the second half is used to perform the conformal calibration described in Section 4.2. For the Reacher task, we trained Σψ(s,u) _ψ(s,u) as a neural network with 2 hidden layers (128 neurons per hidden layer) at a learning rate of 0.0003 using a cosine annealing learning rate scheduler to stabilize training. The model directly outputs the Cholesky factor to avoid computing it in real time. For the Rope and OGBench-Cube tasks, we defined Σψ(s,u):=Σcov _ψ(s,u):= _ cov as a fixed parameter. In this case, the loss is minimized by computing Σcov=1|cal(1)|∑(si,ui,si+)∈cal(1)(si+−fϕ(si,ui))(si+−fϕ(si,ui))⊤ _ cov= 1|D_ cal^(1)| _(s_i,u_i,s_i^+) _ cal^(1)(s_i^+-f_φ(s_i,u_i))(s_i^+-f_φ(s_i,u_i)) as the sample error covariance. For the conformal calibration we set α=0.1,0.4,and0.1α=0.1,0.4,~and~0.1 for Reacher, Rope, and OGBench-Cube respectively. D.4.3 Training and Conformalizing the Constraint Classifier For each constrained task, we train a latent-space classifier from a balanced dataset of rendered constraint-violating and non-violating observations. Let obs=(oi,yi)i=1N,yi∈0,1,D_obs=\(o_i,y_i)\_i=1^N, y_i∈\0,1\, where, following the notation in the main text, yi=0y_i=0 denotes a constraint-violating observation and yi=1y_i=1 denotes a non-violating observation. The task-specific geometric rule used to assign yiy_i is described below. In the implementation, the stored source label is the complementary indicator b~i=1−yi b_i=1-y_i, so b~i=1 b_i=1 marks the violating class used for conformal calibration. Each image is encoded with the frozen world-model encoder zi=encθ(oi)∈ℝnzz_i=enc_θ(o_i) ^n_z, and the latent features are standardized using the training-split mean and standard deviation. Let ℐtrI_tr, ℐvalI_val, and ℐcalI_cal denote the index sets for the training, validation, and conformal calibration splits of obsD_obs, respectively. We then train a small MLP score model cψ:ℝnz→ℝc_ψ:R^n_z with the convention that larger scores indicate safer states. Equivalently, the signed labels are y~i=−1,yi=0,+1,yi=1, y_i= cases-1,&y_i=0,\\ +1,&y_i=1, cases and the classifier is optimized with the hinge loss ℒobs(ψ)=1Ntr∑i∈ℐtrmax(0,m−y~icψ(zi)).L_obs(ψ)= 1N_tr _i _tr (0,m- y_ic_ψ(z_i) ). Thus, before conformal calibration, cψ(z)>0c_ψ(z)>0 is classified as non-violating and cψ(z)≤0c_ψ(z)≤ 0 is classified as violating. We calibrate the classifier using only held-out violating examples. Let ℐcal−=i∈ℐcal:yi=0I_cal^-=\i _cal:y_i=0\ and define the nonconformity score ri=max0,cψ(zi),i∈ℐcal−.r_i= \0,c_ψ(z_i)\, i _cal^-. This score is zero for a violating calibration point that is already on the violating side of the nominal classifier and positive only when the point is incorrectly placed on the safe side. For miscoverage level δ, we choose η=Quantile⌈(ncal−+1)(1−δ)⌉(ri:i∈ℐcal−∪∞),η=Quantile_ (n_cal^-+1)(1-δ) (\r_i:i _cal^-\∪\∞\ ), where ncal−=|ℐcal−|n_cal^-=|I_cal^-|. The conformalized violating set is conf=z:cψ(z)<η.O_conf=\z:c_ψ(z)<η\. Under the exchangeability assumption for future violating observations and the calibration split, this construction covers a future violating latent with probability at least 1−δ1-δ. To match the SLS notation that is used in the main text, we convert the classifier score into a constraint function of the following form: gψ(s)=−cψ(z),z=Πzs,safe=s:gψ(s)≤−η=s:−cψ(Πzs)≤−η.g_ψ(s)=-c_ψ(z), z= _zs, _safe=\s:g_ψ(s)≤-η\=\s:-c_ψ( _zs)≤-η\. Here Πz _z extracts the zeroth-order latent embedding from the Markov latent state s=[z,Δz,…,ΔKz]s=[z, z,…, ^Kz]. This is the sign conversion used by the SLS planner: the raw network score cψc_ψ is larger for safe states, while the MPC constraint is written in the paper’s standard form gψ(s)≤0g_ψ(s)≤ 0. The same threshold also appears in the MPPI warm-start soft penalty as a smooth relaxation of η−cψ(z)≤0η-c_ψ(z)≤ 0. Translating Constraints for SLS: Using the obstacle constraint we derived, it can be plugged into SLS and robustly satisfied by tightening the constraint. For nominal constraint satisfaction at step k, in the trajectory (0≤k≤T0≤ k≤ T) we require, g¯ψ(ζk,vk)≤−η g_ψ( _k,v_k)≤-η, where we define g¯ψ(ζk,vk)=g(zk) g_ψ( _k,v_k)=g(z_k) such that zkz_k is the zeroth-order component of the Markov state ζk _k. We then apply the constraint tightening as a function of the linearized constraint and tube (as discussed in Appendix B) to robustly satisfy the constraint for all Markov states, sk∈ℛkss_k ^s_k and control inputs uk∈ℛkuu_k ^u_k in the tube. In our implementation we also have box constraints for the control input (i.e., uk≤umax and −uk≤−umin,∀0≤k≤T−1u_k≤ u_ max and -u_k≤-u_ min,∀~0≤ k≤ T-1). Accounting for both upper and lower bound control constraints, this results in a total of 2nu+1=nc2n_u+1=n_c constraints in our problem. For succinctness, we collect all the constraints for each time step as follows: gi(ζk,vk)+bi+τi,k≤0,i=1,…,nc,k=0,…,T,g_i( _k,v_k)+b_i+ _i,k≤ 0, i=1,…,n_c,\ k=0,…,T, (23) where gi(ζk,vk)g_i( _k,v_k) is the constraint function, bib_i is the nominal bound (adjusted by η for conformalized obstacle constraint indices), and τi,k _i,k is the constraint tightening. Task-specific label rules and classifier sizes Reacher: For Reacher, the violating set is the joint-space box obs=q=(q1,q2):0≤q1≤3.1415,−2.88≤q2≤−2.45.Q_obs=\q=(q_1,q_2):0≤ q_1≤ 3.1415,\ -2.88≤ q_2≤-2.45\. This region corresponds to configurations where the second link folds back very close to the first link. In the simulator this is a pose/joint-space obstacle constraint, while in planning it is enforced as a learned latent-space constraint from rendered images. We sample configurations inside obsQ_obs as violating examples and configurations outside the box as non-violating examples, then render each configuration from the fixed planner camera to obtain the balanced classifier dataset. The Reacher classifier uses a 55-dimensional latent input and an MLP with one hidden layer of width 66 followed by a scalar output, using GELU nonlinearities. OGBench: For Cube, the classifier enforces a height constraint on the grasped cube/effector state. We synthesize grasped cube configurations, render front-camera observations with the target hidden, and label an observation as violating when the measured gripper height is above the configured threshold hgrip>hmax,hmax=0.09.h_grip>h_ , h_ =0.09. Samples with hgrip≤0.09h_grip≤ 0.09 are treated as non-violating. The sampled cube positions use the task-space bounds x∈[0.30,0.50]x∈[0.30,0.50], y∈[−0.25,0.25]y∈[-0.25,0.25], and z∈[0.02,0.30]z∈[0.02,0.30]. If precomputed dataset splits are unavailable, the script creates stratified source-training and calibration splits before carving out a validation subset. The Cube height classifier uses a 1212-dimensional latent input and a linear score model cψ(z)=wψ⊤z+bψc_ψ(z)=w_ψ z+b_ψ, i.e., a single learned separating vector and bias. Rope: For Rope, each sampled task state is x=(r,h,w)x=(r,h,w), where r is the endpoint reach coordinate, h is the endpoint height, and w is the rope width. The task-space bounds are r∈[−0.10,0.30],h∈[1.16,1.35],w∈[0.20,0.75].r∈[-0.10,0.30], h∈[1.16,1.35], w∈[0.20,0.75]. The obstacle is a speedbump-shaped barrier in the (r,h)(r,h) task-space slice, placed on the tabletop over the active reach interval [0.05,0.15][0.05,0.15]. Formally, it is the upper half of an ellipse with center reach rc=0.10r_c=0.10, half-width a=0.05a=0.05, base height zbase=0.75z_base=0.75, and peak height zpeak=0.91z_peak=0.91: b(r)=zbase+(zpeak−zbase)max0,1−(r−rca)2.b(r)=z_base+(z_peak-z_base) \0,1- ( r-r_ca )^2 \. Thus the bump rises 0.160.16 above the table at its midpoint and returns to the table height at r=0.05r=0.05 and r=0.15r=0.15. Because the rope sags as a function of width, we estimate a sag profile s(w)s(w) from simulated proxy-rope configurations and use ℓ(x)=h−s(w) (x)=h-s(w) as the approximate lowest rope height. A rope state is labeled violating when r∈[0.05,0.15]andℓ(x)≤b(r),r∈[0.05,0.15] (x)≤ b(r), and non-violating otherwise. This gives a formal task-space obstacle that visually behaves like a smooth speedbump the rope must clear; the classifier then learns the corresponding latent-space constraint from fixed-camera renderings. The Rope classifier uses a 1212-dimensional latent input and an MLP with one hidden layer of width 1212 followed by a scalar output, using GELU nonlinearities. D.4.4 Calibrating the Latent Support To further validate the exchangeability assumption at planning time, we estimate and conformalize the support of our calibration dataset. SIGReg loss (5) structures the latent space such that the raw latent vectors follow an isotropic multivariate Gaussian distribution with unit variance. Thus, the Markovian latent vector also inherits the Gaussian structure, but is not guaranteed to be isotropic due to correlation between successive latent states. Hence, we learn both the shape and size of the ellipsoidal geometry characterizing the support of our calibration dataset, and constrain our planner to remain within it. We refer to this constraint as the in-distribution (ID) latent ellipsoid. To obtain this constraint we use the same calibration dataset, calD_ cal and partitions, cal(1)/cal(2)D_ cal^(1)/D_ cal^(2)from Section D.4.2. We compute the sample covariance of the states, ΣID _ID, using states, sis_i, in cal(1)D_ cal^(1), and fix the mean μ=0μ=0. We then compute the Mahalanobis distance di=si⊤ΣID−1sid_i=s_i _ ID^-1s_i for each state, si∈cal(2)s_i _ cal^(2). Lastly, we compute the (1−αID)(1- _ ID)-quantile qIDq_ ID of distances did_i to form the boundary of our support, as defined in (11). Formally, this provides a boundary between in-distribution points which we assume to be (1−αID)⋅100%(1- _ ID)· 100\% and outlier points corresponding to αID⋅100% _ ID· 100\% of points in the calibration dataset, cal(2)D_ cal^(2). For each of our experiments we set αID=0.1 _ ID=0.1. D.5 Safety Baseline Implementation HJ-filtered safety baseline. We implement the HJ-filtered baseline as a latent safety shield applied on top of a nominal planner. We use the same encoder and latent dynamics notation as in the main method: an observation oto_t is encoded as zt=encθ(ot),z_t=enc_θ(o_t), and the planner state is the Markov latent state stz=[zt,Δzt,…,ΔKzt].s_t^z=[z_t, z_t,…, ^Kz_t]. Actions are represented in the normalized action coordinates used by the learned world model: u¯=clip(u−μuσu,umin,umax),umin=−2,umax=2. u=clip ( u- _u _u,u_ ,u_ ), u_ =-2, u_ =2. The HJ baseline rolls out the learned one-step latent dynamics s^t+1z=fϕ(stz,u¯t). s_t+1^z=f_φ(s_t^z, u_t). The HJ baseline uses its own obstacle classifier convention, inherited from the HJ-filtered baseline implementation, which is distinct from the constraint classifier notation used by our SLS planner in the main body. To avoid overloading gψg_ψ, we denote the HJ classifier score by cωc_ω. Its signed safety margin is ℓHJ(s)=cω(s)−τHJ, _HJ(s)=c_ω(s)- _HJ, where τHJ _HJ is the HJ classifier decision threshold, calibrated separately from the SLS constraint margin η. In our experiments, τHJ _HJ is selected using the same held-out calibration split used for conformal safety calibration. Under this baseline convention, positive values indicate states classified as safe and non-positive values indicate states classified as unsafe. Depending on the experiment configuration, the raw margin is optionally transformed by a monotone squashing function, ℓHJ(s)←tanh(ℓHJ(s))orℓHJ(s)←tanh(2ℓHJ(s)). _HJ(s)← ( _HJ(s)) _HJ(s)← (2 _HJ(s)). We train a PyHJ/Avoid-DDPG recovery policy πHJ _HJ and critic VHJV_HJ in the learned latent dynamics. Episodes are initialized from cached latent states. For a sampled action u¯t u_t, the imagined next latent state is s~t+1z=fϕ(stz,u¯t), s_t+1^z=f_φ(s_t^z, u_t), and the transition signal is the next-state safety margin: rt=ℓHJ(s~t+1z).r_t= _HJ( s_t+1^z). An episode terminates when ℓHJ(s~t+1z)≤0 _HJ( s_t+1^z)≤ 0. The learned critic therefore estimates recoverability under the latent recovery policy. At deployment, the HJ barrier score is BHJ(s)=minℓHJ(s),VHJ(s).B_HJ(s)= \ _HJ(s),V_HJ(s)\. Given a nominal raw action utnomu_t^nom from the base planner, the filter first normalizes it to u¯tnom u_t^nom and predicts s^t+1z,nom=fϕ(stz,u¯tnom). s_t+1^z,nom=f_φ(s_t^z, u_t^nom). The nominal action is executed only if the current state is classified as safe and the predicted next state is accepted by the learned barrier. Otherwise, the filter replaces it with the HJ recovery action: u¯t=πHJ(stz),if ℓHJ(stz)≤0 or BHJ(s^t+1z,nom)≤ϵ,u¯tnom,otherwise. u_t= cases _HJ(s_t^z),&if _HJ(s_t^z)≤ 0 or B_HJ( s_t+1^z,nom)≤ε,\\ u_t^nom,&otherwise. cases We use ϵ=0ε=0 unless otherwise specified. The selected normalized action is then unnormalized before execution: ut=u¯tσu+μu.u_t= u_t _u+ _u. Latent Policy Barrier baseline. We also evaluate an adapted Latent Policy Barrier (LPB) baseline, which incorporates a nearest-neighbor in-distribution barrier as a regularizer in the nominal iLQR planner. First, we build a bank of safe Markov latent states from the training dataset. Each image is encoded as zt=encθ(ot)z_t=enc_θ(o_t) and converted into the same Markov latent state stzs_t^z used by the planner. Unsafe states are removed using the analytical ground-truth obstacle check used to label obstacle data. Thus, the prototype bank contains only latent states whose corresponding environment configurations are collision-free under the geometry-based safety test. The remaining safe states are whitened, s~=s−μLPBσLPB, s= s- _LPB _LPB, and M prototype states =pjj=1MP=\p_j\_j=1^M are sampled from the whitened safe set. We calibrate a distance threshold ρ as a chosen quantile of nearest-prototype distances on held-out non-prototype safe states. At planning time, each predicted latent state is scored by dLPB(s)=minpj∈‖s−μLPBσLPB−pj‖2.d_LPB(s)= _p_j \| s- _LPB _LPB-p_j \|_2. The LPB penalty is zero inside the calibrated safe tube and grows quadratically outside it: cLPB(s)=[dLPB(s)−αLPBρ]+2,c_LPB(s)= [d_LPB(s)- _ LPBρ ]_+^2, where αLPB _ LPB is the threshold scale and [⋅]+=max(⋅,0)[·]_+= (·,0). Unlike the HJ filter, LPB does not override actions after planning. Instead, it augments the iLQR trajectory objective. For a nominal latent trajectory =ζkk=0T ζ=\ _k\_k=0^T and normalized controls =vkk=0T−1v=\v_k\_k=0^T-1, the LPB-regularized objective is JLPB=∑k=0T−1(qs‖ζk−ζg‖22+qu‖vk‖22)+qf‖ζT−ζg‖22+λLPB1T∑k=0T−1cLPB(ζk),J_LPB= _k=0^T-1 (q_s\| _k- _g\|_2^2+q_u\|v_k\|_2^2 )+q_f\| _T- _g\|_2^2+ _LPB 1T _k=0^T-1c_LPB( _k), where ζg _g is the goal Markov latent state. In our implementation, the LPB term is differentiable through the latent trajectory, and its gradient and Hessian are included in the iLQR backward pass. Thus, LPB biases the planner toward trajectories that remain close to the empirical distribution of safe latent states while still optimizing the original goal-reaching cost. Constrained iLQR. For our constrained iLQR implementation we solve Problem (14) without constraints (14c)-(14d), and remove the constraint tightening, τi,k _i,k in (14e). This results in a constrained safety-aware problem, but does not require a trajectory to robustly satisfy constraints–unlike SLS. Furthermore we have two variants, one with the in-domain latent ellipsoid constraint and one without. Outside of these changes, both of these approaches still use the same parameters and configurations as the SLS implementation. D.6 Hardware Implementation Our hardware experiment is the physical equivalent of our simulated bimanual rope task. Hardware: The bimanual setup consists of two KUKA iiwa arms placed 1.5 meters apart, each with a Schunk end-effector. The rope is attached to the arms via two 3D-printed clips that are screwed into the Schunk end-effector. Figures 12 and 13 provide visualizations of the hardware setup. To capture images, we used an Intel RealSense D415 camera (and use only the RGB images). Data Collection: We parameterize the rope manipulation task with a three-dimensional task space: reach, the shared forward/backward displacement of the end effectors; height, their shared vertical position; and width, their lateral separation. Each task-space command is converted to bimanual joint targets using Drake [62] inverse kinematics, with the two grippers moved symmetrically about the rope centerline. Data is collected in shards. At the start of each shard, the arms return to a fixed home pose, which defines the nominal task origin. We then sample 25 random task-space knots within the workspace bounds, enforcing a minimum separation between consecutive knots. We linearly interpolate between each pair to produce a dense sequence of intermediate task targets. This waypoint-based trajectory gives broad workspace coverage while keeping each low-level motion small and feasible for hardware execution. We considered smoother spline trajectories, as used in the sibling simulation experiment, but use piecewise-linear waypoint segments on the real system because they make the commanded task deltas explicit, easier to validate for collision and speed limits, and easier to execute with settling between samples. For each interpolated target, the corresponding joint command is executed, the arms are allowed to settle, and an RGB image is captured. We record the planned task-space delta together with the measured end-effector state and executed task-space delta, producing image-action pairs for training the dynamics model. Model Training: We collect our data with a data structure similar to that of all our data collection scripts for simulation. This enables us to swap in our hardware data directly with our simulation data. All the model parameters are equivalent to the corresponding simulation experiment for nominal planning. Planning: For nominal planning, we used MPPI to warm-start trajectory generation at each step and then iLQR to track it, similar to the Push-T approach described in Section D.2. Since the low-level controller requires a minimal actuation in order to move, we define Jact()=λact∑t=0T−1[max(0,δmin−‖vt‖∞δmin)]2,J_act(v)= _act _t=0^T-1 [ (0,\, _ -\|v_t\|_∞ _ ) ]^2, (24) where δmin _ is the minimum actuation. Since the bimanual rope manipulation task has 3 actions controlling the distance between the two end-effectors (width), the height of the two end-effectors, and the depth of two end-effectors (reach), we only require one of the actions to meet the minimum actuation requirement. Thus, we compute the infinity norm of the nominal action ‖vt‖∞\|v_t\|_∞ to get the maximum value among the 3 actions and penalize it if it is below the minimum actuation threshold, δmin _ . Thus, the cost used for the real-world experiment Jmppireal(,)J real_ mppi( ζ,v) is, JMPPIreal(,)=JMPPI(,)⏟Eq. (17)+Jact().J real_ MPPI( ζ,v)= J_ MPPI( ζ,v)_ Eq. eq:nom_mppi_cost+J_act(v). (25) Similar to the simulation experiment (for Push-T), we use iLQR to track the MPPI-generated reference trajectory, and provide the next control input to apply at each MPC step. The MPPI parameters and iLQR parameters are available in Table 8. Table 8: MPPI and iLQR planning parameters for the real-world rope manipulation task. Module Parameter Value Parameter Value MPPI Warm-start Horizon (HmppiH^mppi) 6060 Control Weight (RmppiR^mppi) 0.02I30.02I_3 Samples 512512 Reward Weight 25.025.0 Updates 66 Noise Scale 0.20.2 State Stage Weight (QmppiQ^mppi) 1.5I241.5I_24 Filter β 0.650.65 State Terminal Weight (QTmppiQ_T^mppi) 1000.0I241000.0I_24 Actuation Penalty Minimum Threshold (δmin _ ) 0.0060.006 Penalty Weight (λact _act) 10.010.0 iLQR Tracking Horizon (T) 1010 State Terminal Weight (QTQ_T) 50.0I2450.0I_24 State Stage Weight (Q) 0.5I240.5I_24 Control Weight (R) 0.02I30.02I_3 Appendix E Extended Results In this section, we provide additional experimental results beyond those reported in the main text. Figures 8 - 11 contain our nominal planning time-lapses where we visualize our goal-reaching performance. Figures 12 - 13 contain our nominal planning time-lapses for our bimanual rope manipulation hardware experiment and figure 14 validates convergence of our planner to the goal configuration in the task space for real hardware. Figures 15 - 17 contain time-lapse comparisons of safe vs. unsafe plans demonstrating SLS2’s ability to enforce safety. Figures 18 - 20 visualize the reachable tubes in the latent space. Lastly, we include the LPB safety baseline on Reacher, the LPB safety baseline on Cube, and the Le-WM nominal planning baseline on Push-T. E.1 Nominal Trajectory Time-lapses Figure 8: Push-T. iLQR-based long-horizon planning, with MPPI warm-start. Our planner is able to successfully make and break contact with the T while executing complex maneuvers in order to push the T to the goal. Figure 9: Cube. iLQR-based long-horizon planning. The planner successfully moves the cube to the goal pose without dropping it, unlike the baseline approaches. Figure 10: Reacher. iLQR-based long-horizon planning. Similar to Figure 9, our gradient-based planner is able to successfully move to the goal in an efficient manner. Figure 11: Rope. iLQR-based long-horizon planning. The planner is able to correctly manipulate the deformable rope to reach the goal configuration. Figure 12: Real. iLQR-based long-horizon planning on real hardware directly from pixels without state estimation. In this example, we demonstrate the arms moving forward to reach the goal image. Figure 13: Real. Another example of iLQR-based long-horizon planning on real hardware directly from pixels without state estimation. In this example, the bimanual setup has to stretch the rope to reach the goal image. Figure 14: Real. Task and latent space error plots demonstrating convergence to the desired goal state on hardware. Plot for Figure 12 on the left and plot for Figure 13 on the right. E.2 Safe vs. Unsafe Time-lapses Figure 15: Rope. Safe (top) plan vs. unsafe (bottom) plan time-lapse. The SLS2 planner successfully maneuvers the rope over the obstacle while reaching the goal image, unlike the nominal planner which collides with the obstacle. Figure 16: Reacher. Safe (top) plan vs. unsafe (bottom) plan time-lapse. Similar to Figure 15, the nominal planner traverses through the “unsafe” region and violates the pose constraint, while SLS2 avoids the unsafe region. Figure 17: Cube. Safe (top) plan vs. unsafe (bottom) plan time-lapse. SLS2 maintains the cube below the height threshold while reaching the goal. E.3 Latent Reachability Tubes Figure 18: Rope. Latent reachability tubes for the full Markov state computed via SLS2. Figure 19: Reacher. Latent reachability tubes for the full Markov state computed via SLS2. Figure 20: Cube. Latent reachability tubes for the full Markov state computed via SLS2.