Paper deep dive

Modeling Latent Disturbances for Robust Decision-Making in World Models

Junwon Seo, Andrea Bajcsy

Year: 2026Venue: arXiv preprintArea: cs.ROType: PreprintEmbeddings: 191

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%

Last extracted: 10/13/2026, 3:46:13 AM

Summary

This paper introduces LUCID (Latent-space Uncertainty-Calibrated In-distribution Disturbance), a framework for robust decision-making in the latent space of World Models (WMs). It addresses the challenge of defining physically meaningful disturbances in learned latent spaces by constructing an uncertainty set of plausible latent dynamics using a dynamics-aware similarity metric and out-of-distribution detection, calibrated via conformal prediction. The method employs game-theoretic optimization to jointly optimize robust robot actions and worst-case latent disturbances. Experiments on a Franka manipulator demonstrate that LUCID reduces failure rates by 70% in latent safety filtering and 54% in sampling-based policy steering.

Entities (8)

World Models · model · 99%LUCID · method · 98%Franka Manipulator · hardware · 97%Robust Optimization · concept · 95%Game-theoretic Optimization · technique · 93%Conformal Prediction · technique · 92%Out-of-Distribution Detection · technique · 91%Latent Safety Filtering · application · 90%

Relation Signals (8)

LUCID → appliesto → World Models

confidence 98% · In this paper, we study robust decision-making in the latent space of world models (WMs).

LUCID → evaluatedon → Franka Manipulator

confidence 97% · hardware experiments with a Franka manipulator show that modeling latent disturbances enables robust policy steering

LUCID → enables → Robust Decision-Making

confidence 96% · Our controlled simulation experiments show that our latent disturbance enables robust decision-making directly in WM latent spaces

LUCID → uses → Conformal Prediction

confidence 95% · We calibrate this uncertainty set over latent dynamics using conformal prediction

LUCID → uses → Game-theoretic Optimization

confidence 94% · We then jointly optimize robust robot actions and the worst-case latent disturbances through game-theoretic optimization.

LUCID → uses → Out-of-Distribution Detection

confidence 94% · combining a dynamics-aware similarity metric ... with out-of-distribution detection that excludes implausible latent states.

LUCID → reducesfailuresin → Sampling-based Policy Steering

confidence 93% · and 54% in sampling-based policy steering.

LUCID → reducesfailuresin → Latent Safety Filtering

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:In this paper, we study robust decision-making in the latent space of world models (WMs). Robust optimization is a mathematical framework where, given explicitly specified dynamics and physically meaningful disturbances, a robot can select actions that remain effective even under worst-case disturbances. However, applying this principle to the learned latent space of WMs introduces a fundamental challenge: because WMs have fully learned state spaces and dynamics inferred from high-dimensional observations, it is unclear how to define latent-space disturbances that faithfully represent uncertainty in the underlying system. Our key idea is to model a latent-space disturbance as a perturbation to the learned latent dynamics that induces pessimistic but plausible transitions. Specifically, we construct a set of plausible latent dynamics by combining a dynamics-aware similarity metric that captures plausible transitions with out-of-distribution detection that excludes implausible latent states. We calibrate this uncertainty set over latent dynamics using conformal prediction, ensuring that WM imaginations induced by the latent disturbance remain plausible without becoming overly pessimistic. We then jointly optimize robust robot actions and the worst-case latent disturbances through game-theoretic optimization. We leverage this latent-space robust optimization to robustify policy steering, considering two paradigms: latent safety filtering and sample-and-verify steering of a generative control policy. Our controlled simulation experiments show that our latent disturbance enables robust decision-making directly in WM latent spaces, and hardware experiments with a Franka manipulator show that modeling latent disturbances enables robust policy steering, reducing failures by 70% in safety filtering and 54% in sampling-based policy steering. Project website: this https URL.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

188,074 characters extracted from source content.

Expand or collapse full text

Modeling Latent Disturbances for Robust Decision-Making in World Models Junwon Seo Andrea Bajcsy †thanks: The authors are with the Robotics Institute, Carnegie Mellon University, Pittsburgh, PA, USA. Email: junwonse, abajcsy@andrew.cmu.edu Affiliation: https://junwon.me/LatentDisturbance/ Abstract In this paper, we study robust decision-making in the latent space of world models (WMs). Robust optimization is a mathematical framework where, given explicitly specified dynamics and physically meaningful disturbances, a robot can select actions that remain effective even under worst-case disturbances. However, applying this principle to the learned latent space of WMs introduces a fundamental challenge: because WMs have fully learned state spaces and dynamics inferred from high-dimensional observations, it is unclear how to define latent-space disturbances that faithfully represent uncertainty in the underlying physical system. Our key idea is to model a latent-space disturbance as a perturbation to the learned latent dynamics that induces pessimistic but plausible transitions. Specifically, we construct a set of plausible latent dynamics by combining a dynamics-aware similarity metric that captures physically plausible transitions with out-of-distribution detection that excludes implausible latent states. We calibrate this uncertainty set over latent dynamics using conformal prediction, ensuring that WM imaginations induced by the latent disturbance remain plausible without becoming overly pessimistic. We then jointly optimize robust robot actions and the worst-case latent disturbances within the calibrated uncertainty set through efficient game-theoretic optimization. We leverage this latent-space robust optimization framework to robustify policy steering under uncertainty, considering two paradigms: latent safety filtering and sample-and-verify style steering of a generative control policy. Our controlled simulation experiments show that our latent disturbance enables robust decision-making directly in WM latent spaces, and hardware experiments with a Franka manipulator show that modeling latent disturbances enables robust policy steering, reducing failures by 70%70\% in safety filtering and 54%54\% in sampling-based policy steering. Index Terms: World Models, Decision-Making under Uncertainty, Robust Safe Control Fig. 1: Overview of Robust Decision Making in World Models. Left: A latent world model learns a latent dynamics model z′∼fz(z,a)z f_z(z,a) from observation–action transitions (o,a,o′)(o,a,o ), predicting a distribution over next latent states z′z given the current latent state z and the robot’s action a. Middle: Leveraging the learned predictive distribution, we construct an uncertainty set of latent dynamics, calibrated to include plausible transitions while excluding implausible out-of-distribution states zOOD′z _OOD. Within this set, we model a latent disturbance as a perturbation to the learned dynamics that yields worst-case plausible next states z^′ z . Right: This enables robust decision-making in the latent space of world models, including robust safety filtering that proactively safeguards the system without overestimating safety and robust policy steering that selects actions that remain effective under their worst-case plausible outcomes. I Introduction Consider a robot manipulator serving a sunny-side-up egg by sliding it from a spatula onto a plate (Fig. Modeling Latent Disturbances for Robust Decision-Making in World Models). This task is deceptively challenging: it not only requires precise motor control from visual observations, such as carefully tilting the spatula, but the outcome of each robot action is uncertain: the same serving motion may successfully slide the egg onto the plate or accidentally flip it sunny-side-down depending on unobserved factors such as the egg’s mass or the amount of oil on the spatula. How can the robot choose actions that are robust to these uncertainties? A formal way to model such robust decision-making under uncertainty is via robust optimization, which selects a robot action that remains effective under any possible realization of a disturbance, which represents system uncertainty [1]: πrob(s)=arg⁡mina∈maxd∈D⁡J⁡(f⁡(s,a,d)). _rob(s)= _a _d∈ D\,J (f(s,a,d) ). (1) Here, f denotes discrete-time system dynamics conditioned on state s, action a, and disturbance d representing uncertain factors beyond the robot’s control; D denotes the set of such disturbances, and the objective function J evaluates the resulting future evolution. This optimization can be viewed as a dynamic game [2, 3]: the outer minimization selects a robust action that remains effective despite the worst-case disturbance selected by the inner maximization [4, 5, 6]. Robust optimization typically requires a structured dynamical system model, where the state and disturbance are represented, often by an expert, to have clear physical meaning (e.g., position and velocity as states and external forces as disturbances [7, 2]). However, latent world models (WMs) [8, 9] have recently shown promise for representing hard-to-model systems directly from high-dimensional sensor observations. By jointly learning compact latent representations and their associated dynamics from datasets of robot observations and actions, WMs enable complex systems to be modeled directly in a learned latent space, such as interactions with deformable objects [10] and complex rigid bodies [11]. However, extending robust optimization in (1) to the learned latent space of WMs raises a fundamental challenge: how to define a latent-space disturbance that faithfully represents uncertainty in the underlying physical system? Unlike disturbances in physically interpretable state spaces [12], arbitrary perturbations in the learned latent space need not correspond to physically plausible system evolutions. For example, in Fig. Modeling Latent Disturbances for Robust Decision-Making in World Models, factors such as friction or the amount of oil on the spatula are not directly observed or explicitly represented by the world model. This motivates the central question of our work: How can we represent disturbances in the latent space of world models for robust optimization? Our key idea is to model a latent-space disturbance as a perturbation to the learned latent dynamics that induces pessimistic world-model imaginations, based on the observation that predictive uncertainty in the latent dynamics reflects disturbances in the underlying system. The main challenge is to construct an uncertainty set [6] of latent disturbances, a set of plausible latent dynamics that is broad enough to cover possible transitions while excluding implausible ones that would lead to overly conservative decisions [13, 14, 15]. This is particularly important in high-dimensional latent spaces, where nearby or high-likelihood states may still correspond to physically infeasible or hallucinated out-of-distribution (OOD) transitions [11, 16]. If such OOD transitions are not properly excluded from the uncertainty set, the disturbance can exploit these regions, producing implausibly pessimistic imaginations and leading to overly conservative decisions. To address this, we propose Latent-space Uncertainty-Calibrated In-distribution Disturbance (LUCID), a model of latent-space disturbance, which combines a dynamics-aware similarity metric with out-of-distribution detection to construct an uncertainty set of plausible latent dynamics. We calibrate this uncertainty set using conformal prediction [17, 18], ensuring that world-model imaginations induced by the latent disturbance remain plausible without becoming overly pessimistic. We then jointly optimize the worst-case disturbance and robust action through game-theoretic optimization, using an efficient parameterization of the latent disturbance for tractable optimization. We instantiate LUCID for robust policy steering, including latent safety filtering [10, 11, 19] and sampling-based policy steering [20, 21], which can anticipate potential failures without overestimating safety and steer the policy with actions that remain safe under adverse uncertainty. Through experiments, we show that: (i) in controlled settings with access to the true system dynamics, LUCID approximates the optimal solution to robust optimization in the world-model latent space; (i) in high-fidelity simulated vision-based manipulation, where we can control system disturbances such as friction and mass, LUCID robustly prevents task failures despite unobservable disturbances; and (i) in hardware experiments with a Franka robot, LUCID synthesizes a robust safety filter that proactively detects potential failures without becoming overly pessimistic, reducing the failure rate (44%→13%44\%→ 13\%), and enables robust policy steering through pessimistic yet plausible WM imaginations, improving the success rate of visuomotor policies (35%→70%35\%→ 70\%). I Related Work Robust Decision Making Under Uncertainty. Robust optimization formalizes robust decisions that remain effective across a set of plausible system evolutions, including worst-case realizations [22, 6], by optimizing worst-case performance over an uncertainty set. The uncertainty set characterizes plausible variations in the system [1], often in physically interpretable terms such as external force [3], model errors [23, 24], or uncertainty estimated from data [25, 13]. Similarly, distributionally robust optimization defines the uncertainty set over probability distributions to account for distribution shifts in environments or model dynamics [26, 27]. The problem is commonly formulated as a game between the controller and an adversary that selects worst-case disturbances [26, 28, 27] with game-theoretic methods [12, 3] or Lagrangian approximations [29, 27]. A related perspective is risk-sensitive control [30, 31, 32, 33], which accounts for adverse outcomes using risk measures such as conditional value-at-risk (CVaR) [34]. Certain risk measures can also be expressed as worst-case expectations over corresponding sets of probability distributions, providing a connection between risk-sensitive and robust optimization [35]. In robust reinforcement learning, policies are optimized against adverse perturbations to observations [36, 37, 27], dynamics variations [5, 38], distributional shifts encountered at deployment, and out-of-distribution state-action regions [39, 40, 41, 42]. A key design choice in robust optimization is the disturbance and its uncertainty set, often defined using physically meaningful bounds specified by experts [12, 3, 43] or estimated from data [13, 44, 45, 46], where an overly broad set may admit implausible system evolutions, leading to overly conservative decisions [47]. This issue is more pronounced in the learned latent space of world models, where latent-space similarity does not necessarily reflect the underlying physical dynamics [48] and may include implausible out-of-distribution states [11, 49, 16]. Our work addresses this challenge by constructing a conformalized uncertainty set over plausible latent dynamics, optimizing for worst-case within this set to produce pessimistic but plausible world-model imaginations. World Models in Robotics. World models predict the outcomes of robot actions directly from high-dimensional observations such as RGB images, enabling vision-based policy learning [8], planning [50], policy evaluation [51, 52], and policy improvement [53, 54] through imaginations [55, 56, 57]. Compared to video world models that predict future observations directly in pixel space [58, 15], latent world models jointly learn compact latent representations and their associated dynamics [59, 9], enabling efficient decision-making directly in the learned latent space [10, 11, 60]. Because world models operate under partial observability, latent dynamics models are often probabilistic models such as recurrent state-space models [61] and diffusion models [62]. However, defining disturbances that meaningfully represent uncertainties for robust optimization is challenging [27, 63], while model errors and unseen inputs can further produce hallucinated states during imagination [11, 16]. Consequently, world models typically use finite samples from learned dynamics [64, 65, 66], which can miss rare but high-impact outcomes in high-dimensional spaces [15]. Our work introduces a latent-space disturbance that directly searches for worst-case latent dynamics with gradient-based optimization. Latent Safety Filters. Safety filtering is a control-theoretic approach that safeguards dynamical systems from failures given known safety constraints and a system dynamics model [67]. Safety filters can be implemented through minimally restrictive switching with a safety fallback policy synthesized via Hamilton–Jacobi (HJ) reachability analysis [7], control barrier functions (CBFs) [28], or predictive rollout-based methods such as model predictive safety filtering [68, 69]. While such filters can be synthesized using dynamic programming [70], self-supervised learning [71] and reinforcement learning (RL) [23] have enabled scalable approximations for high-dimensional systems [72]. Latent safety filters extend this idea to the latent space of learned world models, enabling safety filtering for hard-to-model tasks and safety constraints from high-dimensional observations [10, 11, 19]. By anticipating future failures within world-model imaginations, these methods can proactively steer robots away from failures such as spilling the contents of deformable bags [73] or toppling complex rigid-body structures [11]. However, existing latent safety filters evaluate safety under nominal latent dynamics and may therefore overestimate safety under optimistic imaginations. Our method enables the synthesis of robust safety filters using game-theoretic adversarial RL [12, 3, 43] in the learned latent space of world models. I Preliminaries: Latent World Models System. The robot operates from high-dimensional observations ot=ℋ⁡(st)∈o_t=H(s_t) (e.g., RGB images), where ℋ:→H:S denotes the sensor mapping from the privileged state to the observation. We assume that the underlying system st+1=f⁡(st,at,dt)s_t+1=f(s_t,a_t,d_t) is a discrete-time system with a privileged state and continuous control input at∈⊂ℝ||a_t ^|A|, and the disturbance dt∈Dd_t∈ D is not observable to the robot, inducing uncertainty in the system. In classical control, a disturbance set D is typically specified by human experts or identified from data and often carries a physical meaning (e.g., bounds on wind or actuation error) [23, 67, 72]. Latent World Model. To represent hard-to-model systems, we learn a latent world model from a dataset of observation-action trajectories, train:=(ot,at,ot+1)t=1T−1i=1ND_train:= \(o_t,a_t,o_t+1)_t=1^T-1 \_i=1^N [8, 9]. It comprises: (i) an encoder ℰE that maps an observation history to latent states z∈z , (i) a decoder G that reconstructs observations from latent states, and (i) a latent dynamics model fz:×→Δ⁡()f_z\!:\!Z×A\!→\! (Z) that predicts the next latent states, where Δ denotes the set of distributions: Encoder: : zt∼ℰ⁡(zt∣o≤t) z_t (z_t o_≤ t) (2) Decoder: : ot∼⁡(ot∣zt) o_t (o_t z_t) Latent Dynamics: Dynamics: zt∼fz(zt∣zt−1,at−1). z_t f_z(z_t z_t-1,a_t-1). World Model Training. In this work, we focus on recurrent state-space models (RSSM) [61, 8], which learn probabilistic latent dynamics with tractable conditional distributions over future latent states, using Gaussian or categorical distributions: fz(z′∣z,a)=⁡(μ⁡(z,a),Σ⁡(z,a))orCat(ϕ⁡(z,a)).f_z(z z,a)=N (μ(z,a), (z,a) )\,\,or\,\,\,Cat (φ(z,a) ). (3) The latent world model can be trained by learning both the latent representation through a reconstruction objective and the latent dynamics through a KL-divergence term: ∑t=1T[log⁡(ot∣zt)⏟reconstruction−DKL(ℰ(zt∣o≤t)∥fz(zt∣zt−1,at−1))⏟latent dynamics], _t=1^T\! [\! (o_t\! \!z_t)_ [0.0pt]reconstruction\!-\! D_KL\! (E(z_t\! \!o_≤ t)\|f_z(z_t\! \!z_t-1,\!a_t-1) )_latent dynamics ], (4) where the ℰ⁡(zt∣o≤t)E(z_t o_≤ t) provides a learned posterior target for the latent dynamics predictive distribution fz(zt∣zt−1,at−1)f_z(z_t z_t-1,a_t-1). Uncertainty in Latent Dynamics Represents Disturbances. Recall the example from Fig. Modeling Latent Disturbances for Robust Decision-Making in World Models, where unobservable disturbances such as the oiliness of the spatula can lead to different next states under the same action. The probabilistic latent dynamics model therefore implicitly captures these disturbances through its predictive distribution over next latent states [25, 72] (Fig. Modeling Latent Disturbances for Robust Decision-Making in World Models, left). During WM training with (4), the latent state encoded from the next observation serves as the realized transition target for the dynamics model through the KL-divergence objective. In this way, uncertainty induced by unobserved system disturbances is captured in terms of the predictive uncertainty in the learned latent dynamics. IV Setup: Robust Optimization in Latent Space Using the latent world model to predict the outcomes of robot actions, our goal is to solve the robust optimization problem in (1) over the world model’s latent space. We specify the robot’s task through a cost function J:→ℝJ:Z that evaluates a latent state. Depending on the task, J may represent an immediate cost, a cumulative cost over a finite horizon [27], or an infinite-horizon value function [28, 67]. Throughout this section, we present the optimization in single-step form for notational simplicity; the formulation also extends to multi-step (e.g., T-step) or infinite-horizon settings. Latent Disturbances as Pessimistic Imaginations. Traditionally, solving (1) requires identifying disturbances d∈Dd∈ D that induce worst-case next states, but the learned latent dynamics in (2) do not explicitly model such disturbances. Nevertheless, as described in Sec. I, the latent dynamics implicitly capture the effects of system disturbances through probabilistic predictions. Based on the intuition that predictive uncertainty in the learned dynamics reflects such disturbances, we model the worst-case latent-space disturbanceas selecting perturbed latent dynamics fzdf_z^d around the nominal learned dynamics fzf_z, inducing plausible but pessimistic imaginations: πrob(z)=argmina∈ maxfzd(z,a)∈ℱ⁡(z,a)z′∼fzd(⋅∣z,a) [J⁡(z′)], _rob(z)= _a to621.05pt to15.07pt 310.519pt -8.24002pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces @begingroup@_scopebegin=1 @closescope to0.0pt @begingroup@_scopebegin=1 @inpgf@ignorespaces @begingroup@_scopebegin=1 @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.0-310.519pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 -429.67 0) @hbox58 @closescope @closescope @closescope to0.0pt @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope to621.05pt to15.07pt 310.519pt -8.24002pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces @begingroup@_scopebegin=1 @closescope to0.0pt @begingroup@_scopebegin=1 @inpgf@ignorespaces @begingroup@_scopebegin=1 @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.0-310.519pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 -429.67 0) @hbox58 @closescope @closescope @closescope to0.0pt @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope to365.22pt to10.22pt 182.60904pt -5.4337pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces @begingroup@_scopebegin=1 @closescope to0.0pt @begingroup@_scopebegin=1 @inpgf@ignorespaces @begingroup@_scopebegin=1 @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.0-182.60904pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 -252.68 0) @hbox58 @closescope @closescope @closescope to0.0pt @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope to266.83pt to7.28pt 133.41501pt -3.88123pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces @begingroup@_scopebegin=1 @closescope to0.0pt @begingroup@_scopebegin=1 @inpgf@ignorespaces @begingroup@_scopebegin=1 @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.0-133.41501pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 -184.61 0) @hbox58 @closescope @closescope @closescope to0.0pt @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope [J(z ) ], (5) where the uncertainty set ℱ⁡(z,a)F(z,a) defines a set of plausible latent transitions of the underlying system for a given (z,a)(z,a). Fig. 2: Naughty 3D Dubins’ Car. (a) Environment with the vehicle and a failure set at the center. (b) For an action sequence that turns right, an adverse disturbance can instead drive the vehicle into failure, making the decision non-robust. (c) Driving straight remains safe even in the worst case. However, an implausible imagination can make this robust action appear unsafe. Running Example: Naughty 3D Dubins’ Car. In Fig. 2, we consider a discrete-time 3D Dubins’ car with state s=[px,py,θ]s=[p_x,p_y,θ], speed v=1m/sv\!=\!1m/s, and timestep Δt=0.05s t\!=\!0.05\,s. A circular failure set of radius 0.25m0.25\,m centered at the origin defines the cost function J⁡(s):=0.252−px2−py2.J(s)\!:=\!0.25^2-p_x^2-p_y^2. Each state is rendered as a ×1283\!×\!128\!×\!128 RGB image, and the robot’s action is the angular velocity at∈[−1.25,1.25]rad/sa_t∈[-1.25,1.25]\,rad/s, while its naughty behavior is a disturbance introducing uncertainty by randomly flipping the sign of the control input: st+1=st+Δt[vcos⁡(θt),vsin⁡(θt),δtat],δt∈−1,1.s_t+1\!=\!s_t+ t[v ( _t),v ( _t), _ta_t\,],\,\, _t∈\-1,1\. (6) This disturbance is unobservable to the robot, making the underlying dynamics uncertain: for the same state and action, two distinct plausible next states may be realized. A robust action remains safe under the worst-case plausible outcome, whereas a non-robust action may appear safe nominally but fail under an adverse disturbance. Challenges: Overly Pessimistic Disturbances. While (5) formalizes robust optimization in the latent space of the world model, constructing an uncertainty set of latent dynamics is challenging because a latent-space distance metric can be misleading for two reasons. First, distance in a high-dimensional latent space does not necessarily reflect physical or dynamical plausibility, so nearby latent states may still correspond to infeasible outcomes. Second, learned latent dynamics may assign high likelihood to implausible out-of-distribution states, allowing the latent disturbance to exploit them and produce implausible imaginations. Consequently, a poorly constructed uncertainty set can admit overly pessimistic imaginations and make robust optimization overly conservative. For example, an action sequence of turning right in Fig. 2(b) is a non-robust action because a control-sign flip can steer the vehicle into the central failure set, whereas driving straight in Fig. 2(c) is a robust action sequence because it remains safe under all possible control-sign flips. However, an implausible latent disturbance can imagine the vehicle turning into the failure set, making the robust action appear unsafe. V Conformalized Latent-Space Disturbance Our key idea is to model a latent-space disturbance as an alternative latent dynamics model within a conformalized uncertainty set over plausible latent dynamics. To prevent the latent-space disturbance from becoming overly pessimistic, we adopt a dynamics-aware similarity metric together with out-of-distribution detection (Sec. V-A), and then calibrate this set using conformal prediction (Sec. V-B) (Fig. Modeling Latent Disturbances for Robust Decision-Making in World Models). Finally, we jointly compute the robust action and the worst-case latent disturbance through game-theoretic optimization (Sec. V-C). V-A Uncertainty Set over Plausible Latent Dynamics Dynamics-Aware Similarity. Recall that the latent dynamics fz(zt+1|zt,at)f_z(z_t+1\,|\,z_t,a_t) is trained to match the distribution over next latent states induced by next observations ℰ⁡(zt+1|o≤t+1)E(z_t+1\,|\,o_≤ t+1) using the KL divergence in (4). This motivates using the same KL divergence to define the uncertainty set over plausible latent transitions, yielding a dynamics-aware notion of similarity rather than relying on an arbitrary distance in latent space. Specifically, for each transition from a latent-action pair (z,a)(z,a), we define the dynamics-aware uncertainty set over plausible latent dynamics fzdf_z^d as a left-KL ball around the learned predictive distribution11 1 This KL direction matches the training objective in (4). While the formulation is distributional, the mode-seeking character of left KL favors concentration on a single adverse mode of the learned latent dynamics model.: DKL(fzd(z,a)∥fz(z,a))≤ϵKLD_KL\! (f_z^d(z,a)\,\|\,f_z(z,a) )\!≤ _KL, where ϵKL _KL controls the size of the uncertainty set to ensure coverage. This dynamics-aware KL ball contains alternative latent transitions consistent with the learned distribution, whereas a Euclidean ball may include nearby but dynamically infeasible transitions. In-Distribution Constraint. While the KL ball can provide coverage by setting its radius to include plausible transitions, in high-dimensional latent spaces, coverage alone does not ensure precision: latent states with high likelihood under the predictive distribution may still be implausible, or out-of-distribution (OOD) [11, 16]. The disturbance may exploit such states, leading to overly pessimistic imaginations. We therefore augment the uncertainty set with an in-distribution constraint using an OOD score function sOOD(z)s_OOD(z): ℱ(fz,z,a):= 0.0pt23.99997ptF(f_z;z,a):= fzd∈Δ(): DKL(fzd(z,a)∥fz(z,a))≤ϵKL , \f_z^d∈ (Z): to449.04pt to11.49pt 224.51785pt -2.5pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces @begingroup@_scopebegin=1 @closescope to0.0pt @begingroup@_scopebegin=1 @inpgf@ignorespaces @begingroup@_scopebegin=1 @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.0-224.51785pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 -310.67 0) @hbox58 @closescope @closescope @closescope to0.0pt @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope to449.04pt to11.49pt 224.51785pt -2.5pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces @begingroup@_scopebegin=1 @closescope to0.0pt @begingroup@_scopebegin=1 @inpgf@ignorespaces @begingroup@_scopebegin=1 @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.0-224.51785pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 -310.67 0) @hbox58 @closescope @closescope @closescope to0.0pt @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope to307.78pt to8.67pt 153.88995pt -1.75pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces @begingroup@_scopebegin=1 @closescope to0.0pt @begingroup@_scopebegin=1 @inpgf@ignorespaces @begingroup@_scopebegin=1 @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.0-153.88995pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 -212.94 0) @hbox58 @closescope @closescope @closescope to0.0pt @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope to250.7pt to5.75pt 125.35085pt -1.25pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces @begingroup@_scopebegin=1 @closescope to0.0pt @begingroup@_scopebegin=1 @inpgf@ignorespaces @begingroup@_scopebegin=1 @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.0-125.35085pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 -173.45 0) @hbox58 @closescope @closescope @closescope to0.0pt @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope , ℙz′∼fzd(⋅∣z,a)(sOOD(z′)>ϵOOD)≤αood , Dynamics-aware Similarity In-distribution Constraint to581.88pt to12.81pt 290.9357pt -4.79556pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces @begingroup@_scopebegin=1 @closescope to0.0pt @begingroup@_scopebegin=1 @inpgf@ignorespaces @begingroup@_scopebegin=1 @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.0-290.9357pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 -402.57 0) @hbox58 @closescope @closescope @closescope to0.0pt @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope to581.88pt to12.81pt 290.9357pt -4.79556pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces @begingroup@_scopebegin=1 @closescope to0.0pt @begingroup@_scopebegin=1 @inpgf@ignorespaces @begingroup@_scopebegin=1 @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.0-290.9357pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 -402.57 0) @hbox58 @closescope @closescope @closescope to0.0pt @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope to363.08pt to9.35pt 181.53844pt -3.1111pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces @begingroup@_scopebegin=1 @closescope to0.0pt @begingroup@_scopebegin=1 @inpgf@ignorespaces @begingroup@_scopebegin=1 @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.0-181.53844pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 -251.19 0) @hbox58 @closescope @closescope @closescope to0.0pt @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope to280.74pt to6.23pt 140.37045pt -2.2222pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces @begingroup@_scopebegin=1 @closescope to0.0pt @begingroup@_scopebegin=1 @inpgf@ignorespaces @begingroup@_scopebegin=1 @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.0-140.37045pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 -194.23 0) @hbox58 @closescope @closescope @closescope to0.0pt @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope \, [-16.99998pt]0.0pt16.99998pt to0pt to0pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces @begingroup@_scopebegin=1 @closescope to0.0pt @begingroup@_scopebegin=1 @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @closescope @closescope to0.0pt @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope to0pt to0pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces @begingroup@_scopebegin=1 @closescope to0.0pt @begingroup@_scopebegin=1 @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @begingroup@stroke=#31A4A0 @begingroup@fill=#31A4A0 @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @begingroup@stroke=#31A4A0 @begingroup@fill=#31A4A0 @inpgf@ignorespaces @closescope @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.0-200.18481pt16.14003pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 -277 22.33) @hbox58 @closescope @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @begingroup@stroke=#0D948F @begingroup@fill=#0D948F @inpgf@ignorespaces @fill @stroke @drawpath@unclippedM -286.04 15.48 L -286.04 18.85 L -123.55 18.85fill:none @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm0.0-1.01.00.0-206.71782pt11.18999pt @begingroup@transform=matrix(0.0 -1.0 1.0 0.0 -286.04 15.48) @begingroup@_scopebegin=1 @drawpath@unclippedM 2.77 0 L -1.66 2.21 L 0 0 L -1.66 -2.21stroke:none @closescope @closescope @inpgf@ignorespaces @inpgf@ignorespaces @closescope @closescope to0.0pt @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope to0pt to0pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces @begingroup@_scopebegin=1 @closescope to0.0pt @begingroup@_scopebegin=1 @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @begingroup@stroke=#8830BA @begingroup@fill=#8830BA @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @begingroup@stroke=#8830BA @begingroup@fill=#8830BA @inpgf@ignorespaces @closescope @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.0-226.6026pt-13.44556pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 -313.55 -18.6) @hbox58 @closescope @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @begingroup@stroke=#740CAD @begingroup@fill=#740CAD @stroke @drawpath@unclippedM -322.59 -9.68 L -322.59 -19.67 L -165.19 -19.67fill:none @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm0.01.0-1.00.0-233.1356pt-6.99553pt @begingroup@transform=matrix(0.0 1.0 -1.0 0.0 -322.59 -9.68) @begingroup@_scopebegin=1 @drawpath@unclippedM 2.77 0 L -1.66 2.21 L 0 0 L -1.66 -2.21stroke:none @closescope @closescope @inpgf@ignorespaces @inpgf@ignorespaces @closescope @closescope to0.0pt @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope (7) where ϵOOD _OOD is the OOD threshold and αood _ood is the error rate. In practice, we instantiate sOODs_OOD using a flow-matching-based density estimator (logpZO) [74] trained on training latent states, where smaller scores indicate in-distribution. V-B Conformal Calibration of the Uncertainty Set Because the uncertainty set depends on learned latent dynamics and OOD score functions, its thresholds must be calibrated to ensure that plausible transitions are captured without making the uncertainty set overly conservative. We therefore use conformal prediction (CP) [17, 18, 46], a distribution-free statistical method, to calibrate two quantities: the KL-ball radius ϵKL _KL and the OOD threshold ϵOOD _OOD. Since transitions within a trajectory are temporally dependent, we perform both calibrations at the trajectory level using a held-out in-distribution calibration dataset of observation-action trajectories: calib:=(oti,ati,ot+1i)t=1T−1i=1Ncalib.D_calib:= \(o_t^i,a_t^i,o_t+1^i)_t=1^T-1 \_i=1^N_calib. Proofs and additional details for this section are provided in Appendix B. Calibrating the Dynamics-Aware Similarity. Our goal is to calibrate the KL-ball radius ϵKL _KL such that the KL ball around the learned latent dynamics prediction contains the ground-truth latent dynamics realized by the systems with high probability. Mirroring the training objective of the latent dynamics model (4), we adopt the nonconformity score DKL(ℰ(zt∣o≤t)∥fz(zt∣zt−1,at−1)),D_KL (E(z_t o_≤ t)\,\|\,f_z(z_t z_t-1,a_t-1) ), and apply conformal prediction with a user-specified miscoverage level αKL∈[0,1] _KL∈[0,1]. Under the exchangeability assumption, the calibrated uncertainty set guarantees inclusion of the test-time latent representation encoded from observations within the uncertainty set with probability at least 1−αKL1- _KL: ℙ(DKL(ℰ(otest≤t)∥fz(ztestt−1,atestt−1))≤ϵKL)≥1−αKL. (D_KL\! (E(o^test_≤ t)\,\|\,f_z(z^test_t-1,a^test_t-1) )≤ _KL )≥ 1- _KL. (8) Calibrating the In-Distribution Constraint. While we aim to classify a latent state as OOD when sOOD(z)>ϵOODs_OOD(z)> _OOD, OOD latent states are, by definition, not directly available [11]; we only have access to in-distribution states. Similar to [11], we therefore adopt class-conditional conformal prediction [75] to calibrate ϵOOD _OOD using in-distribution states encoded from the calibration dataset, providing recall guarantees for detecting in-distribution states with a miscoverage level αood∈[0,1] _ood∈[0,1]: ℙ⁡(sOOD(zIDtest)≤ϵOOD)≥1−αood.P (s_OOD(z_ID^test)≤ _OOD )≥ 1- _ood. (9) Intuitively, an in-distribution latent state is accepted as in-distribution with probability at least 1−αood1- _ood, while latent states whose OOD scores exceed ϵOOD _OOD are less likely to be in-distribution and are therefore classified as OOD. Trajectory-Level Calibration. (8) and (9) assume exchangeability of the calibration and test samples. However, individual transitions within a trajectory are temporally dependent and therefore cannot be treated as exchangeable. To address this, similar to [76], we perform calibration at the trajectory level, assuming that the calibration trajectories and the test trajectory are exchangeable, and causally reconstruct the state-level thresholds from the trajectory-level result. Given a trajectory τi:=(oti,ati,ot+1i)t=1T−1∈calib _i:=\(o_t^i,a_t^i,o_t+1^i)\_t=1^T-1 _calib with state-level nonconformity scores stis_t^i, we define the trajectory-level nonconformity score as S⁡(τi):=maxt⁡stiS( _i)\!:=\! _ts_t^i and compute the (−α)(1\!-\!α)-quantile of S⁡(τi)i=1Ncalib\S( _i)\_i=1^N_calib to obtain the calibrated threshold ϵtrajε^traj. This trajectory-level threshold can then be causally reconstructed as the state-level threshold: ℙ⁡(maxt⁡sttest≤ϵtraj)=ℙ⁡(sttest≤ϵtraj,∀t)≥1−α,P ( _ts_t^test≤ε^traj )=P (s_t^test≤ε^traj,\;∀ t )≥ 1-α, (10) where (ϵ,α)∈(ϵKL,αKL),(ϵOOD,αood)(ε,α)∈\( _KL, _KL),( _OOD, _ood)\. Thus, the trajectory-level threshold can be causally applied as the corresponding state-level threshold at each test-time transition. V-C Solving Latent-Space Robust Optimization With a calibrated model of latent disturbances, we now solve the latent-space robust optimization problem in (5) by jointly optimizing the latent disturbance within the conformalized uncertainty set and the action that minimizes the cost under the imagination induced by the worst-case latent disturbance. Efficient Disturbance Parameterization. Directly searching over all admissible dynamics in the uncertainty set is intractable in high-dimensional latent spaces (e.g., 11K dims). We therefore parameterize the latent-space disturbance as a residual perturbation to the learned latent dynamics. Specifically, a latent-space disturbance network πd _d outputs an additive residual to the next latent state distribution parameters: πd(z,a)=(Δμ(z,a),ΔΣ(z,a))orΔϕ(z,a), _d\,(z,a)= ( μ\,(z,a), \,(z,a) ) φ\,(z,a), (11) yielding the latent dynamics conditioned on πd _d and fzf_z: fzπd(z′∣z,a)=⁡(μ+Δμ,Σ+ΔΣ)orCat(ϕ+Δϕ).f_z _d(z z,a)=N(μ+ μ, + )\,\,or\,\,Cat(φ+ φ). (12) Game-Theoretic Optimization. We compute the worst-case latent disturbance fzπd(⋅∣z,a) f_z _d(· z,a) for each latent state and robot action pair (z,a)(z,a) by softly relaxing the uncertainty set into hinge penalties. Since both the latent dynamics and OOD score function are differentiable, we reformulate the inner maximization of (5) with coefficients λOOD,λKL>0 _OOD, _KL>0: maxfzπd⁡z′∼fzπd(z,a) _f_z _d\,\,E_z f_z _d(z,a) [J(z′)−λOOD[sOOD(z′)−ϵOOD]+ [J(z )- violet!17$ _OOD [s_OOD(z )- _OOD ]_+$ violet!17$ _OOD [s_OOD(z )- _OOD ]_+$ violet!17$ _OOD [s_OOD(z )- _OOD ]_+$ violet!17$ _OOD [s_OOD(z )- _OOD ]_+$ (13) −λKL[DKL(fzπd(z,a)∥fz(z,a))−ϵKL]+], - teal!17$ _KL [D_KL(f_z _d(z,a)\|f_z(z,a))- _KL ]_+$ teal!17$ _KL [D_KL(f_z _d(z,a)\|f_z(z,a))- _KL ]_+$ teal!17$ _KL [D_KL(f_z _d(z,a)\|f_z(z,a))- _KL ]_+$ teal!17$ _KL [D_KL(f_z _d(z,a)\|f_z(z,a))- _KL ]_+$ ], where [x]+:=max⁡(x,0)[x]_+:= (x,0) applies a penalty only when a constraint is violated. The πd _d is optimized through differentiable latent dynamics with gradient ascent, and the robust action is then optimized under the resulting latent disturbance: πrob(z)=arg⁡mina z′∼fzπd(⋅∣z,a) [J⁡(z′)], _rob(z)= _a to208.81pt to10.93pt 104.40489pt -4.10114pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces @begingroup@_scopebegin=1 @closescope to0.0pt @begingroup@_scopebegin=1 @inpgf@ignorespaces @begingroup@_scopebegin=1 @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.0-104.40489pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 -144.47 0) @hbox58 @closescope @closescope @closescope to0.0pt @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope to208.81pt to10.93pt 104.40489pt -4.10114pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces @begingroup@_scopebegin=1 @closescope to0.0pt @begingroup@_scopebegin=1 @inpgf@ignorespaces @begingroup@_scopebegin=1 @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.0-104.40489pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 -144.47 0) @hbox58 @closescope @closescope @closescope to0.0pt @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope to118.8pt to7.56pt 59.40036pt -2.77812pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces @begingroup@_scopebegin=1 @closescope to0.0pt @begingroup@_scopebegin=1 @inpgf@ignorespaces @begingroup@_scopebegin=1 @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.0-59.40036pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 -82.19 0) @hbox58 @closescope @closescope @closescope to0.0pt @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope to88.47pt to5.38pt 44.23236pt -1.98434pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces @begingroup@_scopebegin=1 @closescope to0.0pt @begingroup@_scopebegin=1 @inpgf@ignorespaces @begingroup@_scopebegin=1 @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.0-44.23236pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 -61.2 0) @hbox58 @closescope @closescope @closescope to0.0pt @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope [J(z ) ], (14) where the robust action and latent-space disturbance form a two-player minimax game that can be solved iteratively [27]. Following [77], we update the disturbance faster than the action to promote convergence, providing a practical approximation to the latent-space robust optimization problem in (5). VI Application: Robust Policy Steering VI-A Setup: Policy Steering with World Models Modeling latent disturbances allows us to instantiate a suite of runtime steering techniques that guide a visuomotor policy robustly away from hard-to-model failures. We consider two steering paradigms: (i) latent safety filters, which safeguard a task policy using a safety monitor and safe controller computed from world-model imaginations [10, 11, 19, 78]; and (i) sample-and-verify, which samples multiple action candidates from a task policy, evaluates their imagined outcomes using the world model, and selects the best action [20, 21]. Existing WM-based policy steering methods predominantly rely on nominal world-model imaginations both when synthesizing the safety controller and during runtime action selection. As a result, they can overestimate safety under optimistic predictions and produce non-robust safety actions that remain effective only under nominal system realizations. Failure Specification. Hard-to-model constraints for policies are specified via a latent failure set F:=z:ℓz(z)≤0⊂F:=\z: _z(z)≤ 0\ defined as the zero-sublevel set of a latent failure margin function ℓz:→ℝ _z:\!Z\!→\!R. The margin function ℓz∈[−1,1] _z∈[-1,1] is, in practice, instantiated as a binary classifier. Steering Method #1: Latent Safety Filters. A latent safety filter synthesizes two components with WM: a safety value function V :→ℝV [*]shield-alt:\!Z\!→\!R, which measures how close the current latent state of the robot is to inevitable failures, and a safety-preserving policy π :→π [*]shield-alt:Z , which steers the robot away from failure. The latent safety filter computes these functions using the learned latent dynamics fzf_z through Hamilton–Jacobi (HJ) reachability analysis [4, 67], satisfying the latent-space safety Bellman equation [10]: V (z) 0.0pt22.0ptV [*]shield-alt(z) =min⁡ℓz(z),maxa∈⁡ z′∼fz(⋅∣z,a) [V (z′)], Nominal Latent Dynamics = \ _z(z), _a to189.91pt to10.93pt 94.95335pt -4.10114pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces @begingroup@_scopebegin=1 @closescope to0.0pt @begingroup@_scopebegin=1 @inpgf@ignorespaces @begingroup@_scopebegin=1 @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.0-94.95335pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 -131.39 0) @hbox58 @closescope @closescope @closescope to0.0pt @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope to189.91pt to10.93pt 94.95335pt -4.10114pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces @begingroup@_scopebegin=1 @closescope to0.0pt @begingroup@_scopebegin=1 @inpgf@ignorespaces @begingroup@_scopebegin=1 @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.0-94.95335pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 -131.39 0) @hbox58 @closescope @closescope @closescope to0.0pt @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope to108.66pt to7.41pt 54.32999pt -2.625pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces @begingroup@_scopebegin=1 @closescope to0.0pt @begingroup@_scopebegin=1 @inpgf@ignorespaces @begingroup@_scopebegin=1 @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.0-54.32999pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 -75.18 0) @hbox58 @closescope @closescope @closescope to0.0pt @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope to80.65pt to5.27pt 40.32492pt -1.875pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces @begingroup@_scopebegin=1 @closescope to0.0pt @begingroup@_scopebegin=1 @inpgf@ignorespaces @begingroup@_scopebegin=1 @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.0-40.32492pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 -55.8 0) @hbox58 @closescope @closescope @closescope to0.0pt @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope [V [*]shield-alt(z ) ] \, to0pt to0pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces @begingroup@_scopebegin=1 @closescope to0.0pt @begingroup@_scopebegin=1 @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @closescope @closescope to0.0pt @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope to0pt to0pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces @begingroup@_scopebegin=1 @closescope to0.0pt @begingroup@_scopebegin=1 @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @begingroup@stroke=#2663B4 @begingroup@fill=#2663B4 @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @begingroup@stroke=#2663B4 @begingroup@fill=#2663B4 @inpgf@ignorespaces @closescope @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.0-113.21626pt14.98332pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 -156.66 20.73) @hbox58 @closescope @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @begingroup@stroke=#0048A6 @begingroup@fill=#0048A6 @stroke @drawpath@unclippedM 0 12.5 L 0 17.25 L -159.76 17.25fill:none @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm0.0-1.01.00.00.0pt9.03328pt @begingroup@transform=matrix(0.0 -1.0 1.0 0.0 0 12.5) @begingroup@_scopebegin=1 @drawpath@unclippedM 2.77 0 L -1.66 2.21 L 0 0 L -1.66 -2.21stroke:none @closescope @closescope @inpgf@ignorespaces @inpgf@ignorespaces @closescope @closescope to0.0pt @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope (15a) π (z) π [*]shield-alt(z) =arg⁡maxa∈ z′∼fz(⋅∣z,a) [V (z′)], = _a to189.91pt to10.93pt 94.95335pt -4.10114pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces @begingroup@_scopebegin=1 @closescope to0.0pt @begingroup@_scopebegin=1 @inpgf@ignorespaces @begingroup@_scopebegin=1 @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.0-94.95335pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 -131.39 0) @hbox58 @closescope @closescope @closescope to0.0pt @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope to189.91pt to10.93pt 94.95335pt -4.10114pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces @begingroup@_scopebegin=1 @closescope to0.0pt @begingroup@_scopebegin=1 @inpgf@ignorespaces @begingroup@_scopebegin=1 @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.0-94.95335pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 -131.39 0) @hbox58 @closescope @closescope @closescope to0.0pt @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope to108.66pt to7.41pt 54.32999pt -2.625pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces @begingroup@_scopebegin=1 @closescope to0.0pt @begingroup@_scopebegin=1 @inpgf@ignorespaces @begingroup@_scopebegin=1 @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.0-54.32999pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 -75.18 0) @hbox58 @closescope @closescope @closescope to0.0pt @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope to80.65pt to5.27pt 40.32492pt -1.875pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces @begingroup@_scopebegin=1 @closescope to0.0pt @begingroup@_scopebegin=1 @inpgf@ignorespaces @begingroup@_scopebegin=1 @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.0-40.32492pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 -55.8 0) @hbox58 @closescope @closescope @closescope to0.0pt @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope [V [*]shield-alt(z ) ], (15b) where a discount factor γ∈[0,1)γ∈[0,1) can be incorporated to induce a contraction mapping [23]. The zero-sublevel set of the safety value, U:=z:V (z)≤0U:=\z:V [*]shield-alt(z)≤ 0\, defines the unsafe set of latent states from which failure cannot be avoided. At runtime, the safety filter safeguards an arbitrary task policy πtaskπ^task by evaluating its proposed action in a least-restrictive fashion [7]. Given the current latent state z, it estimates the expected safety value of next states z′∼fz(⋅∣z,πtask(z)) z f_z(· z,π^task(z)), and the task action is executed if the predicted safety value is positive; otherwise, it is overridden by the fallback policy: πsteer(z)=z′[V (z′)]>0⋅πtask(z)+z′[V (z′)]≤0⋅π (z).π^steer(z)= 1_\ nominal_blue!17$ _z [V [*]shield-alt(z )]$ nominal_blue!17$ _z [V [*]shield-alt(z )]$ nominal_blue!17$ _z [V [*]shield-alt(z )]$ nominal_blue!17$ _z [V [*]shield-alt(z )]$>0\·π^task(z)+ 1_\ nominal_blue!17$ _z [V [*]shield-alt(z )]$ nominal_blue!17$ _z [V [*]shield-alt(z )]$ nominal_blue!17$ _z [V [*]shield-alt(z )]$ nominal_blue!17$ _z [V [*]shield-alt(z )]$≤ 0\·π [*]shield-alt(z). (16) Steering Method #2: Sample-and-Verify. Given K candidate action sequences kk=1k=K∼πtask(z)\a^k\_k=1^k=K\! \!π^task(z) sampled from a task policy (e.g., diffusion policy [79]), we evaluate each candidate using WM imaginations and select the best action sequence: πsteer(z)=arg⁡min∈(k)k=1K′∼fz(⋅∣z,)[J⁡(′)],π^steer(z)= _a∈\a^(k)\_k=1^K nominal_blue!17$ _z f_z(· z,a)$ nominal_blue!17$ _z f_z(· z,a)$ nominal_blue!17$ _z f_z(· z,a)$ nominal_blue!17$ _z f_z(· z,a)$ [J(z ) ], (17) where ′:=zi:i+Hz \!:=\!z_i:i+H denotes the latent states autoregressively generated over the action chunk of length H using nominal imagination zi+1k∼fz(⋅∣zik,aik) z_i+1^k f_z(· z_i^k,a_i^k) starting from ziz_i, and J is a cost function that scores its outcome, such as a latent failure margin [60], safety value function [78], or a VLM-based verifier applied to decoded images [21]. Limitation: Nominal Outcome under Uncertainty. While policy steering methods with latent world models show promise in steering robots away from hard-to-model failures [10, 11], they can become non-robust in tasks with uncertain dynamics because both the synthesis in (15) and the runtime evaluation in (16) and (17) rely only on the nominal WM imaginations. When disturbances in the underlying system can induce qualitatively different outcomes for the same action, the robot may choose non-robust actions based on overly optimistic evaluations. VI-B Robustifying Policy Steering Both policy-steering paradigms in Sec. VI-A require a latent disturbance that drives the system toward failure through a sequence of plausible future transitions, rather than a single worst-case transition. To learn such a worst-case latent disturbance, we instantiate LUCID using latent-space Hamilton–Jacobi–Isaacs (HJI) reachability analysis [4], which recursively evaluates future safety over successive latent transitions. Latent-Space HJI Reachability Analysis. With the conformalized latent-space disturbance fzd∈ℱf_z^d , we formulate infinite-horizon robust decision making by replacing (15) with a latent-space Bellman–Isaacs fixed-point equation [4, 12], the discrete-time dynamic-programming counterpart of Hamilton–Jacobi–Isaacs (HJI) reachability analysis: Vrob(z)=min⁡ℓz(z),maxa∈⁡minfzd∈ℱz′∼fzd(⋅∣z,a)[Vrob(z′)]. V_rob(z)\!=\! \\! _z(z)\!, _a \! disturbance_orange!17$ _f_z^d∈ teal!17$ $ teal!17$ $ teal!17$ $ teal!17$ $\!E_z f_z^d(· z,a)$ disturbance_orange!17$ _f_z^d∈ teal!17$ $ teal!17$ $ teal!17$ $ teal!17$ $\!E_z f_z^d(· z,a)$ disturbance_orange!17$ _f_z^d∈ teal!17$ $ teal!17$ $ teal!17$ $ teal!17$ $\!E_z f_z^d(· z,a)$ disturbance_orange!17$ _f_z^d∈ teal!17$ $ teal!17$ $ teal!17$ $ teal!17$ $\!E_z f_z^d(· z,a)$ [V_rob(z ) ] \. (18) Similarly, the zero-sublevel set Urob:=z:Vrob(z)≤0U_rob:=\z:V_rob(z)≤ 0\ defines a robust latent unsafe set, and the corresponding robust safety policy and worst-case latent-space disturbance fzπdf_z _d are then formulated as a two-player game optimizing against VrobV_rob: fzπd(⋅∣z,a) f_z _d(· z,a) =argminfzd(z,a)∈ℱ⁡(z,a)z′∼fzd(⋅∣z,a)[Vrob(z′)], = _f_z^d(z,a)∈ teal!17$ (z,a)$ teal!17$ (z,a)$ teal!17$ (z,a)$ teal!17$ (z,a)$E_z f_z^d(· z,a) [V_rob(z ) ], (19a) πrob(z) _rob(z) =argmaxa∈z′∼fzπd(⋅∣z,a)[Vrob(z′)]. = _a disturbance_orange!17$ _z f_z _d(· z,a)$ disturbance_orange!17$ _z f_z _d(· z,a)$ disturbance_orange!17$ _z f_z _d(· z,a)$ disturbance_orange!17$ _z f_z _d(· z,a)$ [V_rob(z ) ]. (19b) Computation via Adversarial RL. To compute the solution of the latent-space HJI reachability analysis, we approximate the dynamic game in (18)–(19) using off-policy actor–critic learning with Soft Actor-Critic (SAC) [80], using the WM as a simulator. We learn three networks: (i) a robust safety critic Qrob(z,a)Q_rob(z,a), with Vrob(z)=Qrob(z,πrob(z))V_rob(z)=Q_rob(z, _rob(z)); (i) a latent-space disturbance network πd(z,a) _d(z,a) that predicts residual distribution parameters for the nominal latent dynamics fz(z,a)f_z(z,a), yielding the pessimistic dynamics fzπd(z,a)f_z _d(z,a), as in (11); and (i) a robust safety policy πrob(z) _rob(z). Starting from latent states encoded from trainD_train, we generate imaginations with πrob _rob and fzπdf_z _d, and store the latent-action pairs (z,a)\(z,a)\ in a replay buffer ℬB. At each iteration, we sample transitions from ℬB and update the robust safety critic using the discounted safety Bellman equation, with γ∈[0,1)γ∈[0,1) [23]: ℒQrob=ℬ[ _Q_rob=E_B [ (Qrob(z,a)−((1−γ)ℓz(z)CLOSECLOSE (Q_rob(z,a)- ((1-γ)\, _z(z) (20) +γminℓz(z),Q^rob(z′,a′)))2], +γ\, \! \ _z(z), Q_rob(z ,a ) \ ) )^2 ], where the next latent state z′∼fzπd(⋅∣z,a) z f_z _d(· z,a) is sampled from the latent dynamics induced by the current disturbance policy, and a′∼πrob(⋅∣z′)a _rob(· z ) is sampled from the current robust safety policy. The latent-space disturbance fzπdf_z _d is then updated to minimize the robust safety value while remaining within the calibrated uncertainty set, following (13): ℒπd:=ℬ[Qrob(z′,πrob(z′))⏟Minimize Value+λOOD[sOOD(z′)−ϵOOD]+⏟In-Distribution Constraint _ _d:=E_B [ Q_rob(z , _rob(z ))_Minimize Value+ violet!17$ _OOD\, [s_OOD(z )- _OOD ]_+$ violet!17$ _OOD\, [s_OOD(z )- _OOD ]_+$ violet!17$ _OOD\, [s_OOD(z )- _OOD ]_+$ violet!17$ _OOD\, [s_OOD(z )- _OOD ]_+$_In-Distribution Constraint +λKL[DKL(fzπd(⋅∣z,a)∥fz(⋅∣z,a))−ϵKL]+⏟Dynamics-aware Uncertainty Set Constraint], + teal!17$ _KL\, [D_KL(f_z _d(· z,a)\,\|\,f_z(· z,a))- _KL ]_+$ teal!17$ _KL\, [D_KL(f_z _d(· z,a)\,\|\,f_z(· z,a))- _KL ]_+$ teal!17$ _KL\, [D_KL(f_z _d(· z,a)\,\|\,f_z(· z,a))- _KL ]_+$ teal!17$ _KL\, [D_KL(f_z _d(· z,a)\,\|\,f_z(· z,a))- _KL ]_+$_Dynamics-aware Uncertainty Set Constraint ], (21a) ℒπrob:=ℬ[−Qrob(z,a′)⏟Maximize Value+αentropylog⁡πrob(a′∣z)⏟SAC Entropy Regularization]. _ _rob:=E_B [ -Q_rob(z,a )_Maximize Value+ _entropy\, _rob(a z)_SAC Entropy Regularization ]. (21b) where αentropy _entropy is the SAC entropy-regularization coefficient. Runtime #1: Robust Latent Safety Filters. Since the robust safety value is computed under worst-case latent dynamics, its zero-sublevel set Urob:=z:Vrob(z)≤0U_rob:=\z:V_rob(z)≤ 0\ defines the robust unsafe set, with Vrob(z)=Qrob(z,πrob(z))V_rob(z)=Q_rob(z, _rob(z)): the latent states from which failure is unavoidable under adverse uncertainty. At deployment, we therefore evaluate the action proposed by the task policy using the learned robust safety value, and the safety policy overrides it when the value is non-positive: πrobsteer(z)=πtask(z),if Qrob(z,πtask(z))>0,πrob(z),otherwise.π^steer_rob(z)= casesπ^task(z),&if Q_rob (z,π^task(z) )>0,\\[3.00003pt] _rob(z),&otherwise. cases (22) Runtime #2: Robust Sample-and-Verify. Replacing the nominal imaginations in (17) with pessimistic imaginations induced by the learned worst-case latent-space disturbance fzπdf_z _d, we evaluate K candidate action sequences (k)∼πtask(z)a^(k)\! \!π^task(z) with k=1,⋯,Kk=1,·s,K, and select the best sequence: πrobsteer(z)=arg⁡min∈(k)k=1K′∼fzπd(⋅∣z,)[J⁡(′)],π^steer_rob(z)= _a∈\a^(k)\_k=1^K disturbance_orange!17$ _z f_z _d(· z,a)$ disturbance_orange!17$ _z f_z _d(· z,a)$ disturbance_orange!17$ _z f_z _d(· z,a)$ disturbance_orange!17$ _z f_z _d(· z,a)$ [J(z ) ], (23) where ′z denotes the imagined latent trajectory autoregressively generated over the action chunk zi+1k∼fzπd(⋅∣zik,aik)z_i+1^k f_z _d(· z_i^k,a_i^k), so that evaluation of sampled actions accounts for adverse outcomes under the calibrated system uncertainty. VII Case Study: Dubins’ Car World Model with Known Disturbances We first study our latent-space robust optimization in controlled settings where the underlying system dynamics and disturbances are known, allowing us to evaluate whether our latent-space robust optimization can recover the optimal solution directly in the latent space of world models. Throughout this section, we focus on the latent safety filter paradigm. VII-A Experimental Setup Setup: 3D Dubins’ Car with Disturbances. Let a discrete-time 3D Dubins’ car evolve under two different types of disturbances. First, we use the naughty dynamics introduced in Sec. IV, where the system randomly flips the sign of the action, inducing a discrete, multimodal transition distribution with two possible outcomes for each action. Second, we consider continuous positional disturbances, dx,t,dy,t∈[−0.3,0.3]d_x,t,d_y,t∈[-0.3,0.3]: st+1=st+Δt[vcos⁡(θt)+dx,t,vsin⁡(θt)+dy,t,at].s_t+1\!=\!s_t+ t\,[v ( _t)+d_x,t,\,\,v ( _t)+d_y,t,\,\,a_t\,]. (24) For both settings, the ground-truth worst-case disturbance and corresponding robust action can be computed analytically, enabling evaluation of our latent-space robust optimization. World Model. We adopt Dreamer [81] with a Recurrent State-Space Model (RSSM) [61] where latent dynamics are modeled as a Gaussian distribution [61]. The world model is trained offline on Ntrain=4,000N_train=4,000 observation–action trajectories, where actions and disturbances are sampled randomly for T=100T=100 timesteps. We annotate each observation with the binary ground-truth failure margin ℓ⁡(s)=−1 (s)=-1 if s∈Fs∈ F and ℓ⁡(s)=1 (s)=1 otherwise, and train a two-layer MLP on top of the latent state to predict the latent failure margin ℓz(z) _z(z). Latent Safety Filter Setup. We follow Sec. VI-B to learn the latent safety filter using the pretrained world model. The latent disturbance takes the mean and covariance predicted by the nominal latent dynamics and outputs additive perturbations to both. The thresholds ϵKL _KL and ϵOOD _OOD are calibrated on a held-out set of 1,0001,000 trajectories with αKL=αood=0.01 _KL= _ood=0.01. Evaluation & Metrics. With access to system dynamics, we compute the ground-truth safety value function and optimal robust action using grid-based methods [70]. We first evaluate the safety value V V [*]shield-alt by its accuracy in classifying unsafe states across three state dimensions, discretized into 5050 grids per dimension, reporting balanced accuracy (B.Acc.), false positive rate (FPR), and false negative rate (FNR). We then evaluate the learned safety policy with 1,0001,000 random trajectories, comprising 500500 safe and 500500 that fail under the worst-case disturbance. We apply least-restrictive filtering (22) on those action sequences starting from the initial states in a closed loop, reporting the failure rate (Fail). TABLE I: 3D Dubins’ Car World Model: Quantitative Results. Naughty Positional Disturbances Method B.Acc. ↑ FPR ↓ FNR ↓ Fail ↓ B.Acc. ↑ FPR ↓ FNR ↓ Fail ↓ Nominal 0.932 0.006 0.130 0.114 0.831 0.000 0.337 0.218 WoN 0.959 0.057 0.026 0.061 0.911 0.162 0.017 0.034 CVaR0.05CVaR_0.05 0.944 0.017 0.095 0.062 0.943 0.028 0.087 0.084 CVaR0.1CVaR_0.1 0.935 0.012 0.118 0.061 0.936 0.018 0.109 0.101 CVaR0.2CVaR_0.2 0.927 0.010 0.135 0.062 0.927 0.011 0.135 0.134 CVaR0.5CVaR_0.5 0.915 0.007 0.163 0.057 0.910 0.005 0.175 0.197 LUCID 0.958 0.037 0.046 0.038 0.956 0.023 0.066 0.021 Fig. 3: Qualitative Results: 3D Dubins’ Car. Solid lines represent the ground-truth unsafe-set boundaries, and red regions denote the unsafe sets computed by each method. LUCID closely approximates the ground-truth unsafe sets of robust optimization under two different types of disturbances. In contrast, Nominal is overly optimistic, while the sampling-based baselines such as WoN and CVaR inaccurately approximate unsafe sets. VII-B Can LUCID Make Robust Decisions in World Models? Baselines. We compare LUCID against the following latent safety-filter baselines, which differ in how they account for uncertainty in the learned latent dynamics, using the same WM. We first consider Nominal, which trains the safety filter following (15) that evaluates the expected safety value: Vnom (z)=min⁡ℓz(z),maxa∈⁡z′∼fz(⋅∣z,a)[Vnom (z′)] V [*]shield-alt_nom(z)\!=\! \ _z(z), _a nominal_blue!17$ _z f_z(· z,a)$ nominal_blue!17$ _z f_z(· z,a)$ nominal_blue!17$ _z f_z(· z,a)$ nominal_blue!17$ _z f_z(· z,a)$ [V [*]shield-alt_nom(z ) ] \ (25) Worst-of-N (WoN) samples N=1010 states from the learned dynamics zi′∼fz(⋅∣z,a) z _i\! \!f_z(·\! \!z,a) and uses the least-safe sample: VWoN (z)=min⁡ℓz(z),maxa∈⁡mini=1,…,NVWoN (zi′).V [*]shield-alt_WoN(z)= \ _z(z), _a _i=1,…,NV [*]shield-alt_WoN(z _i) \. (26) Lastly, Risk-Sensitive (CVaR) learns a distributional value function [82, 83] from sampled next states of learned latent dynamics and evaluates each action using the conditional value-at-risk, CVaRβCVaR_β, with β∈0.05,0.1,0.2,0.5β∈\0.05,0.1,0.2,0.5\: VCVaR (z)=min⁡ℓz(z),maxa∈⁡CVaRβ(VCVaR (z′)).V [*]shield-alt_CVaR(z)= \ _z(z), _a CVaR_β (V [*]shield-alt_CVaR(z ) ) \. (27) They are all trained using the discounted safety-value formulation in (20), with the safety policy optimized to maximize the learned safety value. However, unlike LUCID, these baselines account only for uncertainty captured by finite samples from the nominal learned dynamics and do not explicitly optimize a latent disturbance over the calibrated uncertainty set. Result: LUCID Enables Robust Decision Making in Latent Space. Table I shows that LUCID learns a robust latent safety filter under both discrete and continuous system uncertainty, improving both the open-loop accuracy of the safety value function and closed-loop safety performance. As shown in Fig. 3, the learned safety value accurately identifies states from which the worst-case plausible disturbance can induce failure despite the best control action. In contrast, Nominal evaluates safety by averaging over nominal world-model futures and can therefore be overly optimistic: its high FNR indicates that many ground-truth unsafe states are predicted as safe, resulting in higher failure rates in closed-loop performance. Result: Finite Sampling Limits Robust Decision Making. A simple way to account for uncertain latent dynamics is to use finite samples from the nominal next-state distribution, as in WoN or Risk-Sensitive with distributional critics. However, such zeroth-order optimization can miss adverse but plausible outcomes, especially as the space of possible transitions grows. Accordingly, WoN degrades more relative to LUCID under continuous disturbances than under the binary naughty disturbance. Risk-Sensitive also depends on the CVaR risk level: increasing it toward 0.50.5 emphasizes typical rather than adverse returns, degrading worst-case value estimation. These results motivate explicitly computing a worst-case latent disturbance that leads to plausibly pessimistic WM imaginations, using gradient ascent to search for adverse transitions within the calibrated uncertainty set rather than relying only on finite nominal sampling, consistent with the findings of [33]. VII-C Does the Uncertainty Set Represent Plausible Dynamics? To evaluate the conformalized uncertainty set in Sec. V-A and Sec. V-B, we ablate whether it effectively includes plausible transitions without admitting implausible ones. Setup: Uncertainty Set Ablation. To test the conformalized uncertainty set, we ablate latent safety-filter training with the following setups: (i) Euclidean, which replaces the dynamics-aware similarity metric with the isotropic distance to the mean next-state prediction, with its threshold similarly calibrated; (i) w/o OOD, which removes the in-distribution constraint; and (i) Uncalibrated KL thresholds, which use either 0.5ϵKL0.5 _KL or 2ϵKL2 _KL instead of the conformally calibrated threshold. TABLE I: 3D Dubins’ Car: Uncertainty Set Ablation. Dyn., Conf., and OOD represent dynamics-aware similarity, conformal calibration, and in-distribution constraints, respectively. Uncertainty Set Naughty Positional Disturbances Method Dyn. Conf. OOD B.Acc. ↑ FPR ↓ FNR ↓ B.Acc. ↑ FPR ↓ FNR ↓ LUCID ✓ ✓ ✓ 0.958 0.037 0.046 0.956 0.023 0.066 w/o OOD ✓ ✓ ✗ 0.809 0.381 0.001 0.621 0.759 0.000 0.5ϵKL0.5 _KL ✓ ✗ ✓ 0.927 0.011 0.136 0.864 0.001 0.271 2.0ϵKL2.0 _KL ✓ ✗ ✓ 0.926 0.126 0.022 0.630 0.733 0.007 Euclidean ✗ ✓ ✓ 0.605 0.790 0.000 0.875 0.001 0.250 Result: The Conformalized Uncertainty Set Prevents Over-Pessimism. Table I shows that robust optimization with uncertainty-unaware uncertainty sets can become overly pessimistic by admitting implausible latent transitions. Using a dynamics-unaware similarity metric such as Euclidean distance can include geometrically close but dynamically infeasible latent states or exclude plausible but distant ones, yielding high FPRs or FNRs in the safe set computation. Likewise, removing the in-distribution constraint (w/o OOD) allows the latent disturbance to exploit implausible OOD states and produce unrealistic imaginations, again resulting in high FPRs. In contrast, LUCID combines a dynamics-aware similarity metric with the in-distribution constraint using OOD detection to restrict the disturbance to plausible latent transitions. Result: Conformal Calibration Balances Coverage and Pessimism. We further test whether conformal calibration selects an appropriate uncertainty-set size by ablating the calibrated KL threshold ϵKL _KL, which controls the radius around the nominal latent dynamics. Changing the threshold to 0.5×ϵKL0.5× _KL makes the uncertainty set too narrow and yields a high FNR, since worst-case transitions can fall outside the set and the disturbance fails to capture adverse outcomes that may occur under the true system. Conversely, increasing the threshold to 2×ϵKL2× _KL makes the uncertainty set too broad and substantially increases the FPR, as the disturbance can exploit overly pessimistic transitions that are not plausible. These results show that conformal calibration is important for balancing coverage of plausible latent dynamics against excessive conservatism. VII-D Ablation: Optimistic Imagination with Latent Disturbances To further evaluate whether the latent disturbance can effectively perturb latent dynamics while preserving plausible world-model imaginations, we consider an optimistic imagination ablation in the naughty Dubins’ car setting. Setup: Optimistic Latent Disturbance. By optimizing both the robot action and latent disturbance to maximize safety, we replace the minimax problem with a max–max objective, further assessing LUCID’s ability to identify alternative latent transitions within the uncertainty set while preserving plausibility. Specifically, we modify the disturbance objective in (21) to maximize the safety value while remaining within the calibrated uncertainty set. In the naughty Dubins’ car setting, this optimistic formulation corresponds to the disturbance-free system. We therefore use its ground-truth safety value without any disturbance as the evaluation reference. Result: Latent Disturbances Induce Effective and Plausible Imaginations. Table I shows that LUCID learns a safety value function close to that of the disturbance-free system through optimistic imagination, despite using a world model trained on trajectories collected under disturbances. The conformalized uncertainty set is essential for preserving plausibility: Fig. 4 shows that, without the in-distribution constraint, the disturbance can trivially generate implausible imaginations that avoid failure even from ground-truth unsafe states, leading to a high FNR. Dynamics-unaware uncertainty sets similarly yield high FNRs, while miscalibrated KL thresholds degrade the accuracy of value function computation. TABLE I: Ablation: Optimistic Latent Disturbance. Method B.Acc. ↑ F1 ↑ FPR ↓ FNR ↓ Nominal 0.924 0.748 0.122 0.031 WoN 0.966 0.898 0.037 0.032 CVaR0.2CVaR_0.2 0.942 0.860 0.046 0.069 LUCID 0.966 0.947 0.009 0.059 w/o OOD 0.841 0.810 0.000 0.318 0.5ϵKL0.5 _KL 0.956 0.861 0.055 0.034 2.0ϵKL2.0 _KL 0.945 0.939 0.001 0.109 Euclidean 0.940 0.921 0.008 0.113 Fig. 4: Qualitative: Naughty Dubins’ Car with Optimistic Disturbances. Solid lines show the ground-truth unsafe-set boundaries of the disturbance-free system, and dashed lines show the failure set. Without the in-distribution constraint, the disturbance exploits implausible latent states, trivially imagining trajectories that avoid failure from any non-failure state. VIII Simulation: Scaling Latent-Space Robust Optimization to Vision-Based Manipulation We scale our latent-space robust optimization to a vision-based manipulation task, once again focusing on the latent safety filtering paradigm. We use IsaacLab [84], which allows us to control system parameters, such as friction coefficients, that induce uncertainty in the system dynamics. VIII-A Experimental Setup: Vision-Based Block Pouring Setup: Block Pouring. A Franka manipulator must pick up and tilt an orange block to slide a green block placed on top onto a blue block without falling off, as shown in Fig. 5. Observations consist of two tabletop 3×128×1283× 128× 128 RGB camera images and 77-D joint-position proprioception. Actions are 66-DoF end-effector delta poses with a discrete gripper command. Disturbance. To introduce uncertainty in the system dynamics, we randomize physics parameters including (i) the mass, (i) friction coefficients, and (i) restitution of the green block, without providing these parameter values to the robot. World Model. We adopt DreamerV3 [8] with discrete latent states modeled by categorical latent dynamics, and train it on approximately 3,0003,000 trajectories containing both successes and failures. The failure-margin function is trained on top of the latent states using state-level ground-truth failure labels. Latent Safety Filter Setup. In this section, we focus on latent safety filtering for policy steering using the least-restrictive filter in (22), which safeguards the task policy by intervening with the learned safety policy whenever the safety value of the task-policy action becomes non-positive. We follow [11] to learn the latent safety filter, and the latent disturbance takes the categorical logits predicted by the latent dynamics model and outputs additive perturbations. We calibrate the uncertainty set using 200200 held-out calibration trajectories. Task Policies. We evaluate the learned latent safety filter with three task policies πtaskπ^task: (i) DreamerV3 [8], a visuomotor policy trained with dense task rewards independently from the world model used for the safety filter, (i) Diffusion Policy [79], an imitation-learning policy trained on 500500 successful trajectories, and (i) replaying successful human teleoperation trajectories. For each policy, we roll out 2,0002,000 trajectories with randomized initial states and physics parameters. TABLE IV: Quantitative Results: Vision-Based Block Pouring. πtaskπ^task Safety Filter Success ↑ Failure ↓ Safety Gain ↑ Cond. Failure ↓ DreamerV3 No Filter 0.44 0.54 – – Nominal 0.45 0.54 -0.12 ± 0.21 0.82 ± 0.38 WoN 0.46 0.50 0.01 ± 0.15 0.50 ± 0.50 CVaR0.1_0.1 0.39 0.48 -0.08 ± 0.16 0.48 ± 0.50 UnConf. 0.00 0.43 0.08 ± 0.04 0.43 ± 0.50 LUCID 0.73 0.19 0.22 ± 0.21 0.20 ± 0.40 Diffusion Policy No Filter 0.55 0.42 – – Nominal 0.56 0.40 0.12 ± 0.33 0.71 ± 0.46 WoN 0.46 0.48 0.03 ± 0.16 0.48 ± 0.50 CVaR0.1_0.1 0.13 0.19 -0.02 ± 0.10 0.19 ± 0.39 UnConf. 0.00 0.11 0.09 ± 0.03 0.11 ± 0.32 LUCID 0.63 0.24 0.18 ± 0.19 0.27 ± 0.44 Fig. 5: Safety Value Function: Nominal vs. LUCID. Safeguarding the same πtaskπ^task from the same initial states, the robust safety filter proactively intervenes based on a plausibly pessimistic imagination (o^28d o^d_28) in which πtaskπ^task can lead to failure, while the nominal imagination (o^28 o_28) predicts a non-failure outcome. The resulting robust action prevents failure. In contrast, the nominal latent safety filter overestimates safety and intervenes too late based on an optimistic imagination (o^50 o_50), leading to failure under an adverse realization. Baselines. Following Sec. VII, we compare against latent safety filters trained using the same world model and dataset but accounting for dynamics uncertainty in different ways: Nominal, Worst-of-N, and the risk-sensitive baseline (CVaR). We also include Unconformalized latent disturbance (UnConf.), an ablation of LUCID that uses an uncertainty set without the in-distribution constraint. Evaluation Metrics. Since the ground-truth solutions are unavailable, we focus on the closed-loop performance of the policy. Success is completing a task without a timeout or failure. To further assess whether the learned filter is robust, we additionally measure: (i) realized safety gain (Safety Gain): the increase in the learned safety value after each filter intervention, which measures whether the executed safety action robustly leads to a safer realized next state; and (i) conditional failure rate (Cond. Failure): the fraction of trajectories that fail among those in which the filter intervenes at least once. VIII-B Can LUCID Robustly Prevent Failures Under Uncertainty? TABLE V: Results: Trajectory Replay Across 2020 Different Physics. Safety Filter Success ↑ Failure ↓ Safety Gain ↑ Cond. Failure ↓ Robust Rate ↑ No Filter 0.33 0.47 – – 0.10 Nominal 0.31 0.40 0.17 ± 0.2 0.46 ± 0.5 0.11 WoN 0.35 0.31 -0.04 ± 0.2 0.32 ± 0.4 0.07 CVaR0.1 0.30 0.31 -0.11 ± 0.2 0.32 ± 0.4 0.10 UnConf. 0.05 0.06 0.10 ± 0.03 0.06 ± 0.23 0.90 LUCID 0.34 0.11 0.21 ± 0.15 0.11 ± 0.31 0.77 Fig. 6: Block Pouring: World Model Imaginations. (a) Initial state: the orange block is tilted while πtaskπ^task remains still. (b) The nominal imagination predicts a non-failure outcome. (c) With the conformalized uncertainty set, the latent disturbance imagines a plausible adverse transition in which the green block slides and falls. (d) Without the conformalized uncertainty set, the latent disturbance induces an implausible transition in which the green block falls in an infeasible manner, resulting in an overly pessimistic imagination. Result: LUCID Robustly Safeguards Task Policies. Table IV shows the closed-loop safety-filtering results under randomized physical parameters. LUCID achieves lower failure rates while maintaining higher success rates, demonstrating robustness to dynamics uncertainty without becoming overly conservative. In contrast, latent disturbances without the in-distribution constraint (UnConf.) become overly pessimistic, resulting in low success rates as its imagination trivially drifts toward implausible failure states, as shown in Fig. 6. Baselines that rely on sampling from the nominal latent dynamics (Nominal, WoN, and CVaR) are less effective, with a larger gap to LUCID than in the Dubins’ car experiments in Sec. VII. Compared to the Dubins’ car experiments, the visual manipulation task involves higher-dimensional system uncertainty and more diverse plausible next states, making zeroth-order sampling less likely to capture meaningful worst-case outcomes within a limited sampling budget. Risk-sensitive distributional critics (CVaR) are likewise either ineffective or overly conservative depending on the CVaR level, since extreme tails in high-dimensional latent spaces can include implausibly pessimistic states even under random sampling. In contrast, LUCID explicitly optimizes against worst-case plausible latent dynamics within the calibrated uncertainty set. Result: LUCID Does Not Overestimate Safety. Fig. 5 compares the nominal and robust safety values along an identical action sequence as a task policy. While the nominal safety value evaluates future safety under the average outcomes of actions and can therefore overestimate safety when adverse outcomes occur, the robust safety monitor evaluates safety under calibrated worst-case plausible outcomes and intervenes more preemptively when failure is possible, using robust actions whose plausible outcomes avoid failure. As shown in Table IV, LUCID achieves a lower conditional failure rate than the nominal filter when filtering is activated, while intervening more frequently but only when necessary, remaining less pessimistic than UnConf., which doesn’t use the in-distribution constraint for the uncertainty set construction. Moreover, LUCID yields larger positive realized safety gains, indicating that its safety actions move the system toward safer realized next states, whereas the nominal filter exhibits smaller or even negative gains under dynamics uncertainty. VIII-C Ablation: Robustness Under Controlled Uncertainty To further evaluate the robustness of safety filtering specifically to system uncertainty, independent of stochasticity in the task policy, we vary only the physics parameters of the system while replaying exactly the same task-policy action sequences. Setup: Replaying Safe Trajectories. We evaluate the robustness to system disturbances by replaying 100100 successful teleoperation trajectories from the same initial states and actions, each under 2020 different physics parameter settings. Here, we also report robust rate, defined as the fraction of replay trajectories that remain non-failure across all 2020 conditions. Result: LUCID Robustly Prevents Failures Across Dynamics Variations. Table V shows that LUCID keeps replaying 7777 of 100100 teleoperation trajectories failure-free across all 2020 physics settings, showing robustness to system disturbances. It also achieves the lowest overall and conditional failure rates among methods that retain nonzero task success; UnConf. attains lower failure rates only by being overly conservative, resulting in a near-zero success rate. This indicates that when the filter intervenes, LUCID proposes safety actions that remain effective across variations in system dynamics. Its positive realized safety gain further shows that these interventions consistently move the system toward safer realized states. VIII-D Ablation: Robustness Under Partial Observability While we have so far considered uncertainty in future transitions arising from explicit system disturbances, partial observability can also induce uncertainty even when the underlying dynamics are deterministic and disturbance-free. For example, small differences in the underlying angle of the orange block or position of the green block may be difficult to distinguish from pixel observations, yet can lead to different future outcomes under the same action. We therefore ask how our latent-space robust optimization performs when uncertainty arises from the learned world model itself, due to partial observability or model approximation. TABLE VI: Ablation: Deterministic Block-Pouring with Fixed Physics πtaskπ^task Safety Filter Success ↑ Failure ↓ Safety Gain ↑ Cond. Failure ↓ DreamerV3 No Filter 0.43 0.56 – – Nominal 0.60 0.39 0.21 ± 0.24 0.40 ± 0.48 WoN 0.34 0.54 0.10 ± 0.20 0.54 ± 0.49 CVaR0.1 0.00 0.34 -0.05 ± 0.05 0.34 ± 0.47 UnConf. 0.00 0.00 0.29 ± 0.08 0.00 ± 0.06 LUCID 0.81 0.18 0.27 ± 0.22 0.19 ± 0.39 Diffusion Policy No Filter 0.47 0.51 – – Nominal 0.57 0.35 0.15 ± 0.16 0.25 ± 0.48 WoN 0.28 0.66 -0.00 ± 0.20 0.66 ± 0.47 CVaR0.1 0.00 0.19 -0.05 ± 0.05 0.19 ± 0.39 UnConf. 0.00 0.00 0.21 ± 0.06 0.00 ± 0.06 LUCID 0.66 0.25 0.16 ± 0.16 0.27 ± 0.44 Setup: Deterministic Block Pouring. We use the same block-pouring task, but fix the physical parameters to remove explicit disturbances from the underlying system dynamics. Thus, uncertainty arises from partial observability and approximation when mapping high-dimensional observations into the learned latent space. We collect approximately 3,0003,000 trajectories and train the world model, latent safety filter, and task policies following the same procedure. This setting isolates uncertainty in future transitions arising from partial observability and model approximation, allowing us to evaluate whether robust latent optimization remains beneficial even when the underlying system is deterministic. Result: Robust Optimization Improves Safety under Partial Observability. Table VI shows that robust latent-space optimization improves safety even when the underlying system dynamics are deterministic. In this setting, uncertainty in future transitions arises from partial observability and approximation error in the learned world model, rather than from explicit system disturbances. The nominal safety filter is more effective than in the stochastic environment, but LUCID further reduces failures and increases success by mitigating uncertainty. These results suggest that robust latent-space optimization can improve decision-making under different sources of transition uncertainty, provided that the realized test-time transition is contained within the calibrated uncertainty set. IX Hardware: Robust Steering During Contact-Rich Visuomotor Policy Execution Finally, we scale LUCID to a real-world visual manipulation task using a Franka Research 3. The robot must serve a sunny-side-up fried egg from a spatula onto a plate without dropping it on the table or flipping it sunny-side down. The egg’s dynamics are uncertain due to unobserved physical properties such as friction between the egg and spatula. IX-A Experimental Setup: Serving Sunny-Side-Up Fried Eggs Setup. Observations consist of two tabletop 3×192×2563× 192× 256 RGB camera images and 7-D joint-position proprioception. Actions are 6-D end-effector delta poses with a discrete gripper command, executed at 1515 Hz. Disturbance. We introduce an explicit source of unobserved disturbance by varying the spatula surface to mimic changes in friction caused by oil and surface conditions in household kitchens. We use three surface conditions: (i) no tape, (i) scotch tape, and (i) heavy-duty tape (see Fig. 8). World Model. We employ an RSSM with Gaussian latent dynamics [61] learned from frozen DINOv3 [85] embeddings, and construct the latent dynamics model using a Transformer with an 88-step history to predict the next latent-state distribution. We additionally train the model with a DINO token reconstruction objective alongside pixel reconstruction (see Appendix D for details). We collect 900900 teleoperated trajectories with frame-level failure annotations, which are used to learn a latent failure margin over the learned latent space. Latent Safety Filter. We follow [11] to learn the latent safety filter. The latent disturbance predicts additive perturbations to the mean and covariance of the Gaussian latent dynamics, and the uncertainty set is calibrated using 5050 held-out trajectories. Task Policies. We consider three task policies that we seek to steer at runtime: (i) a Diffusion Policy [79] trained from scratch using 300300 successful trajectories; (i) a vision-language-action (VLA) policy fine-tuned from π0.5 _0.5 [86] using the same trajectories; and (i) a teleoperation from a human. Fig. 7: Qualitative Results: Preventing Sunny-Side-Down Egg. LUCID preemptively identifies teleoperator actions that may cause the egg to fall or flip and intervenes with robust actions that remain safe under worst-case dynamics. In contrast, the Nominal safety filter relies on overly optimistic safety values and non-robust actions, leading to sunny-side-down outcomes. Fig. 8: Filtering the Same πtaskπ^task Across Different Surfaces. (Left) Failure rates under each spatula surface condition. (Right) Robust rate, measuring the fraction of trajectories that remain safe across all surface conditions. LUCID consistently minimizes failures and achieves a higher robust rate, whereas the Nominal safety filter prevents failures only under certain surface conditions. IX-B Robust Latent Safety Filtering We compare a Nominal latent safety filter, trained using the expected safety value (15), against LUCID trained with (18). Result: Robustness Across Surface Disturbances. We first evaluate robustness to disturbances induced by the spatula surface condition. We replay 2020 success and 3030 failure trajectories under all three surface conditions while safeguarding them. Fig. 8 shows that LUCID consistently achieves lower failure rates and a higher robust rate, defined as the fraction of replay trajectories that remain safe across all surface conditions. In contrast, the Nominal safety filter is effective only under certain surface conditions and achieves a lower robust rate. Result: Robustly Safeguarding πtaskπ^task. We roll out Diffusion Policy [79] and π0.5 _0.5 [86] for 2020 trials each, safeguarded by either the Nominal safety filter or LUCID, with Scotch tape applied to the spatula. Fig. 9 shows that LUCID achieves a larger improvement in success rate than the Nominal filter, demonstrating more robustness against uncertain dynamics. Fig. 7 shows qualitative results safeguarding the teleoperator. LUCID preemptively overrides unsafe actions, whereas the Nominal safety filter may fail to intervene in time due to overly optimistic imaginations. Moreover, LUCID intervenes with robust fallback actions whose plausible outcomes remain safe, while the Nominal filter may select actions that appear safe under nominal dynamics but fail under adverse realizations. Fig. 9: Quantitative Results: Success Rates. (Left) Safety filtering over 2020 rollouts for Diffusion Policy and π0.5 _0.5. (Right) Steering π0.5 _0.5 with sample-and-verify action selection, using world-model imaginations to evaluate candidate action chunks. In both settings, LUCID robustly improves success rates. IX-C Robust Sample-and-Verify with Latent Disturbances We test the sampling-and-verify policy steering using WM imagination (23) on fine-tuned π0.5 _0.5 [86], evaluating whether the learned latent disturbance enables robust action selection. Setup. We sample 88 candidate action chunks from the policy, each containing 1616 future actions. We evaluate the outcome of each candidate using WM imaginations and execute the action chunk with the best predicted outcome. We compare three imagination schemes: (i) Nominal, using the learned latent dynamics; (i) LUCID, using the learned latent disturbance; and (i) Unconformalized (UnConf.), using a latent disturbance without the in-distribution constraint, as illustrated in Fig. 10. Objective Function (J). To evaluate action outcomes, we use a cost function that combines the learned failure margin ℓz(z) _z(z), evaluated on predicted latent states, with VLM-based task-success and progress scores from Robometer [87], evaluated on decoded images. The VLM reward is averaged over each action chunk to provide a dense signal. Result: LUCID Enables Robust Policy Steering. Fig. 9 reports sample-and-verify style policy steering results over 2020 trials for each method, showing that LUCID improves policy success. Fig. 10 illustrates that LUCID selects actions that remain favorable even under plausible worst-case futures, enabling robust policy steering. In contrast, Nominal does not improve over the base policy without steering, imagining optimistic futures for risky actions. Latent disturbance using an unconformalized uncertainty set instead trivially imagines implausible failures even for safe actions, which weakens discrimination among candidate actions for policy steering. Fig. 10: Qualitative: Policy Steering with World Model. (a) Initial observation and sampled candidate actions. (b) For each candidate, Nominal imagines outcomes using fzf_z, LUCID computes worst-case fzπdf_z _d within the calibrated uncertainty set ℱF, while Unconformalized fzπd^f_z _d can exploit out-of-distribution latent states. (c) Imagined outcomes for candidate actions. LUCID selects an action that remains successful under the plausible worst-case, whereas the other schemes lose discriminability among candidate actions. X Limitations Balancing Plausibility. We optimize our latent disturbance to generate adverse outcomes while regularizing it to stay close to the learned dynamics and remain in distribution. However, these constraints provide only proxies for plausibility, so the resulting uncertainty set can still become overly conservative or fail to capture plausible but critical outcomes. Moreover, because the constraints are enforced through soft penalties, the optimized disturbance may still produce implausible imaginations, particularly when the underlying world model is inaccurate [11]. Conformal calibration provides a marginal coverage guarantee over the calibration distribution rather than an input-conditional guarantee [18]. As disturbance optimization steers WM imaginations away from this distribution, the coverage guarantee may further weaken under distribution shift. More structured world models [48], adaptive or input-dependent calibration schemes, and hallucination detection for world-model imaginations [16] could help ensure that pessimistic imaginations remain physically meaningful. World Model Fidelity. Our current implementation trains task-specific latent WMs from offline interaction data, requiring substantial data collection, and latent disturbance optimization cannot recover outcomes that the WM has not learned to represent. This underscores the need for diverse robot interaction data, including failures, rare outcomes, and variations in physical conditions [88]. In addition, our current formulation assumes a probabilistic latent WM with an explicit transition distribution, which is used to characterize predictive uncertainty and define the latent disturbance. It is therefore not directly applicable to deterministic WMs or diffusion-based video WMs, where uncertainty is represented through a different generative process; complementary approaches can instead steer diffusion-based WM imaginations toward adverse yet plausible outcomes [15]. Our framework also assumes a meaningful failure margin or objective function; inaccurate specifications can misdirect disturbance optimization and policy steering. More general task and safety specifications, for example those derived from vision-language models, could reduce this dependence on task-specific objectives [87]. Game-Theoretic Optimization. Solving the resulting dynamic game remains challenging in high-dimensional action (e.g., 77-D) and latent disturbance (e.g., 11K-D) spaces. Although our disturbance parameterization and adversarial optimization make the problem more tractable, the learned disturbance explores only a restricted class of latent dynamics, and the optimization does not guarantee convergence to a global saddle point [77]. An insufficiently optimized disturbance may miss adverse outcomes, while an insufficiently optimized robot policy may produce suboptimal robust actions. Developing more reliable game-theoretic optimization methods and characterizing the effects of function approximation and optimization error remain important future work. XI Conclusion In this work, we propose a framework for robust decision-making in the learned latent space of world models. We model latent-space disturbances as adverse yet plausible perturbations to learned latent dynamics, enabling robust optimization directly in latent spaces learned from high-dimensional observations without requiring disturbances to be explicitly represented in a physically interpretable state space. To characterize which latent dynamics constitute plausible disturbances, we construct an uncertainty set using a dynamics-aware similarity metric that captures predictive uncertainty in the learned dynamics, together with out-of-distribution detection to exclude implausible latent states. We calibrate this uncertainty set using conformal prediction, allowing the robot to reason about worst-case outcomes while preventing the resulting WM imaginations from becoming overly pessimistic. We instantiate this framework for two forms of robust policy steering: latent safety filtering and sample-and-verify steering of a generative control policy. Controlled experiments show that the proposed formulation closely approximates ground-truth robust safety values and reduces failures under unobserved system disturbances, while ablations demonstrate the importance of calibration and in-distribution constraints for preserving the plausibility of imaginations induced by latent disturbances. In simulated and real-robot vision-based manipulation, our framework improves policy success rates through robust runtime steering. Acknowledgment This work was supported in part by Toyota Research Institute and the DARPA Young Faculty Award (YFA). This article solely reflects the opinions and conclusions of its authors, and not TRI (nor any other Toyota entity), nor DARPA, nor the U.S. Government. References [1] A. Ben-Tal, A. Nemirovski, and L. El Ghaoui (2009) Robust optimization. Princeton university press. Cited by: §I, §I. [2] J. F. Fisac, A. K. Akametalu, M. N. Zeilinger, S. Kaynama, J. Gillula, and C. J. Tomlin (2018) A general safety framework for learning-based control in uncertain robotic systems. IEEE Transactions on Automatic Control 64 (7), p. 2737–2752. Cited by: §I, §I. [3] D. P. Nguyen, K. Hsu, W. Yu, J. Tan, and J. F. Fisac (2024) Gameplay filters: robust zero-shot safety through adversarial imagination. In Conference on Robot Learning (CoRL), Cited by: Appendix C, §I, §I, §I, §I. [4] I. M. Mitchell, A. M. Bayen, and C. J. Tomlin (2005) A time-dependent hamilton-jacobi formulation of reachable sets for continuous dynamic games. IEEE Transactions on Automatic Control. Cited by: §I, §VI-A, §VI-B, §VI-B. [5] L. Pinto, J. Davidson, R. Sukthankar, and A. Gupta (2017) Robust adversarial reinforcement learning. In International Conference on Machine Learning (ICML), p. 2817–2826. Cited by: §I, §I. [6] J. Moos, K. Hansel, H. Abdulsamad, S. Stark, D. Clever, and J. Peters (2022) Robust reinforcement learning: a review of foundations and recent advances. Machine Learning and Knowledge Extraction 4 (1), p. 276–315. Cited by: §I, §I, §I. [7] S. Bansal, M. Chen, S. Herbert, and C. J. Tomlin (2017) Hamilton-jacobi reachability: a brief overview and recent advances. In IEEE Conference on Decision and Control (CDC), p. 2242–2253. Cited by: Appendix C, §I, §I, §VI-A. [8] D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap (2025) Mastering diverse control tasks through world models. Nature 640 (8059), p. 647–653. External Links: ISSN 1476-4687 Cited by: §D-E, §I, §I, §I, §I, §VIII-A, §VIII-A. [9] L. Maes, Q. L. Lidec, D. Scieur, Y. LeCun, and R. Balestriero (2026) Leworldmodel: stable end-to-end joint-embedding predictive architecture from pixels. arXiv preprint arXiv:2603.19312. Cited by: §I, §I, §I. [10] K. Nakamura, L. Peters, and A. Bajcsy (2025) Generalizing safety beyond collision-avoidance via latent-space reachability analysis. Robotics: Science and Systems (RSS). Cited by: §I, §I, §I, §I, §VI-A, §VI-A, §VI-A. [11] J. Seo, K. Nakamura, and A. Bajcsy (2025) Uncertainty-aware latent safety filters for avoiding out-of-distribution failures. Conference on Robot Learning (CoRL). Cited by: Appendix B, §D-B, §D-E, §I, §I, §I, §X, §I, §I, §I, §V-A, §V-B, §VI-A, §VI-A, §VIII-A, §IX-A. [12] K. Hsu, D. P. Nguyen, and J. F. Fisac (2023) Isaacs: iterative soft adversarial actor-critic for safety. In Learning for Dynamics and Control Conference (L4DC), p. 90–103. Cited by: Appendix C, Appendix C, Appendix C, §I, §I, §I, §I, §VI-B. [13] C. Johnstone and B. Cox (2021) Conformal uncertainty sets for robust optimization. In Conformal and Probabilistic Prediction and Applications, p. 72–90. Cited by: Appendix B, §I, §I, §I. [14] H. Hu, Z. Zhang, K. Nakamura, A. Bajcsy, and J. F. Fisac (2023) Deception game: closing the safety-learning loop in interactive robot autonomy. In Conference on Robot Learning (CoRL), Cited by: §I. [15] J. Seo, S. Veer, R. Tian, W. Ding, A. Sharma, K. Leung, E. Schmerling, M. Pavone, and A. Bajcsy (2026) StressDream: steering video world models for robust policy evaluation and improvement. In Conference on Robot Learning (CoRL), Cited by: §I, §X, §I. [16] N. Hansen and X. Wang (2026) Hallucination in world models is predictable and preventable. arXiv preprint arXiv:2606.27326. Cited by: §I, §X, §I, §I, §V-A. [17] G. Shafer and V. Vovk (2008) A tutorial on conformal prediction.. Journal of Machine Learning Research 9 (3). Cited by: Appendix B, Appendix B, §I, §V-B. [18] A. N. Angelopoulos S. Bates et al. (2023) Conformal prediction: a gentle introduction. Foundations and Trends® in Machine Learning. Cited by: Appendix B, Appendix B, Appendix B, §I, §X, §V-B. [19] S. Agrawal, J. Seo, K. Nakamura, R. Tian, and A. Bajcsy (2026) AnySafe: adapting latent safety filters at runtime via safety constraint parameterization in the latent space. In IEEE International Conference on Robotics and Automation (ICRA), Cited by: §I, §I, §VI-A. [20] Y. Wu, R. Tian, G. Swamy, and A. Bajcsy (2025) From foresight to forethought: vlm-in-the-loop policy steering via latent alignment. Robotics: Science and Systems (RSS). Cited by: §I, §VI-A. [21] J. Yuan, Y. Wu, and A. Bajcsy (2026) When to act, ask, or learn: uncertainty-aware policy steering. Robotics: Science and Systems (RSS). Cited by: §I, §VI-A, §VI-A. [22] D. Q. Mayne, M. M. Seron, and S. V. Raković (2005) Robust model predictive control of constrained linear systems with bounded disturbances. Automatica 41 (2), p. 219–224. Cited by: §I. [23] J. F. Fisac, N. F. Lugovoy, V. Rubies-Royo, S. Ghosh, and C. J. Tomlin (2019) Bridging hamilton-jacobi safety analysis and reinforcement learning. In IEEE International Conference on Robotics and Automation (ICRA), p. 8550–8556. Cited by: Appendix C, §I, §I, §I, §VI-A, §VI-B. [24] R. K. Cosner, P. Culbertson, A. J. Taylor, and A. D. Ames (2023) Robust safety under stochastic uncertainty with discrete-time control barrier functions. Cited by: §I. [25] A. K. Akametalu, J. F. Fisac, J. H. Gillula, S. Kaynama, M. N. Zeilinger, and C. J. Tomlin (2014) Reachability-based safe learning with gaussian processes. In IEEE Conference on Decision and Control (CDC), Cited by: §I, §I. [26] I. Yang (2018) A dynamic game approach to distributionally robust safety specifications for stochastic systems. Automatica 94, p. 94–101. Cited by: §I. [27] A. Z. Ren and A. Majumdar (2022) Distributionally robust policy learning via adversarial environment generation. IEEE Robotics and Automation Letters 7 (2), p. 1379–1386. Cited by: §I, §I, §IV, §V-C. [28] J. J. Choi, D. Lee, K. Sreenath, C. J. Tomlin, and S. L. Herbert (2021) Robust control barrier–value functions for safety-critical control. In IEEE Conference on Decision and Control (CDC), p. 6814–6821. Cited by: §I, §I, §IV. [29] A. Sinha, H. Namkoong, and J. Duchi (2018) Certifiable distributional robustness with principled adversarial training. In International Conference on Learning Representations (ICLR), Cited by: §I. [30] Y. Chow, M. Ghavamzadeh, L. Janson, and M. Pavone (2018) Risk-constrained reinforcement learning with percentile risk criteria. Journal of Machine Learning Research 18 (167), p. 1–51. Cited by: §I. [31] Y. Ma, D. Jayaraman, and O. Bastani (2021) Conservative offline distributional reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS) 34, p. 19235–19247. Cited by: §I. [32] M. P. Chapman, R. Bonalli, K. M. Smith, I. Yang, M. Pavone, and C. J. Tomlin (2021) Risk-sensitive safety analysis using conditional value-at-risk. IEEE Transactions on Automatic Control 67 (12), p. 6521–6536. Cited by: §I. [33] H. Nishimura, J. Mercat, B. Wulfe, R. T. McAllister, and A. Gaidon (2023) Rap: risk-aware prediction for robust planning. In Conference on Robot Learning (CoRL), p. 381–392. Cited by: §I, §VII-B. [34] A. Majumdar and M. Pavone (2017) How should a robot assess risk? towards an axiomatic theory of risk in robotics. In International Symposium of Robotics Research (ISRR), Cited by: §I. [35] Y. Chow, A. Tamar, S. Mannor, and M. Pavone (2015) Risk-sensitive and robust decision-making: a cvar optimization approach. Advances in Neural Information Processing Systems (NeurIPS) 28. Cited by: §I. [36] Y. Liang, Y. Sun, R. Zheng, and F. Huang (2022) Efficient adversarial training without attacking: worst-case-aware robust reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS) 35, p. 22547–22561. Cited by: §I. [37] H. Zhang, H. Chen, C. Xiao, B. Li, M. Liu, D. Boning, and C. Hsieh (2020) Robust deep reinforcement learning against adversarial perturbations on state observations. Advances in Neural Information Processing Systems (NeurIPS) 33, p. 21024–21037. Cited by: §I. [38] A. Mandlekar, Y. Zhu, A. Garg, L. Fei-Fei, and S. Savarese (2017) Adversarially robust policy learning: active construction of physically-plausible perturbations. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 3932–3939. Cited by: §I. [39] S. Curi, I. Bogunovic, and A. Krause (2021) Combining pessimism with optimism for robust and efficient model-based deep reinforcement learning. In International Conference on Machine Learning (ICML), p. 2254–2264. Cited by: §I. [40] M. Rigter, B. Lacerda, and N. Hawes (2022) Rambo-rl: robust adversarial model-based offline reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS) 35, p. 16082–16097. Cited by: §I. [41] J. Blanchet, M. Lu, T. Zhang, and H. Zhong (2023) Double pessimism is provably efficient for distributionally robust offline reinforcement learning: generic algorithm and robust partial coverage. Advances in Neural Information Processing Systems (NeurIPS) 36, p. 66845–66859. Cited by: §I. [42] J. Chen, L. Xu, A. Venugopal, and J. Schneider (2026) Policy-driven world model adaptation for robust offline model-based reinforcement learning. In International Conference on Machine Learning (ICML), Cited by: §I. [43] D. D. Oh, D. P. Nguyen, H. Hu, and J. F. Fisac (2026) Synthesis and deployment of maximal robust control barrier functions through adversarial reinforcement learning. arXiv preprint arXiv:2604.13192. Cited by: Appendix C, Appendix C, §I, §I. [44] M. Rigter, B. Lacerda, and N. Hawes (2023) One risk to rule them all: a risk-sensitive perspective on model-based offline reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS) 36, p. 77520–77545. Cited by: §I. [45] J. Sun, Y. Jiang, J. Qiu, P. Nobel, M. J. Kochenderfer, and M. Schwager (2023) Conformal prediction for uncertainty-aware planning with diffusion dynamics model. Advances in Neural Information Processing Systems (NeurIPS) 36, p. 80324–80337. Cited by: §I. [46] L. Marques and D. Berenson (2024) Quantifying aleatoric and epistemic dynamics uncertainty via local conformal calibration. In International Workshop on the Algorithmic Foundations of Robotics, p. 85–103. Cited by: §I, §V-B. [47] W. Zhou and S. Zhu (2026) Calibrating decision robustness via inverse conformal risk control. In International Conference on Machine Learning (ICML), Cited by: §I. [48] Y. Wang, O. Bounou, G. Zhou, R. Balestriero, T. G. J. Rudner, Y. LeCun, and M. Ren (2026) Temporal straightening for latent planning. In International Conference on Machine Learning (ICML), Cited by: §X, §I. [49] P. Lutkus, K. Wang, L. Lindemann, and S. Tu (2025) Latent representations for control design with provable stability and safety guarantees. In IEEE Conference on Decision and Control (CDC), p. 2937–2944. Cited by: §I. [50] N. Hansen, H. Su, and X. Wang (2024) TD-mpc2: scalable, robust world models for continuous control. In International Conference on Learning Representations (ICLR), Cited by: §I. [51] Y. Guo, L. Shi, J. Chen, and C. Finn (2026) Ctrl-world: a controllable generative world model for robot manipulation. In International Conference on Learning Representations (ICLR), Cited by: §I. [52] J. H. Quevedo, A. K. Sharma, Y. Sun, V. Suryavanshi, P. Liang, and S. Yang (2026) WorldGym: world model as an environment for policy evaluation. In International Conference on Learning Representations (ICLR), Cited by: §I. [53] Y. Guo, T. Lee, L. X. Shi, J. Chen, P. Liang, and C. Finn (2026) VLAW: iterative co-improvement of vision-language-action policy and world model. In International Conference on Machine Learning (ICML), Cited by: §I. [54] A. K. Sharma, Y. Sun, N. Lu, Y. Zhang, J. Liu, and S. Yang (2026) World-gymnast: training robots with reinforcement learning in a world model. arXiv preprint arXiv:2602.02454. Cited by: §I. [55] N. Agarwal, A. Ali, M. Bala, Y. Balaji, E. Barker, T. Cai, P. Chattopadhyay, Y. Chen, Y. Cui, Y. Ding, et al. (2025) Cosmos world foundation model platform for physical ai. arXiv preprint arXiv:2501.03575. Cited by: §I. [56] S. Gao, W. Liang, K. Zheng, A. Malik, S. Ye, S. Yu, W. Tseng, Y. Dong, K. Mo, C. Lin, et al. (2026) Dreamdojo: a generalist robot world model from large-scale human videos. arXiv preprint arXiv:2602.06949. Cited by: §I. [57] S. Ye, Y. Ge, K. Zheng, S. Gao, S. Yu, G. Kurian, S. Indupuru, Y. L. Tan, C. Zhu, J. Xiang, et al. (2026) World action models are zero-shot policies. arXiv preprint arXiv:2602.15922. Cited by: §I. [58] Z. Mei, T. Yin, O. Shorinwa, A. Badithela, Z. Zheng, J. Bruno, M. Bland, L. Zha, A. Hancock, J. F. Fisac, et al. (2026) Video generation models in robotics-applications, research challenges, future directions. arXiv preprint arXiv:2601.07823. Cited by: §I. [59] G. Zhou, H. Pan, Y. LeCun, and L. Pinto (2025) Dino-wm: world models on pre-trained visual features enable zero-shot planning. International Conference on machine learning (ICML). Cited by: §I. [60] A. K. Jain, Y. Wu, J. Farebrother, G. Swamy, and A. Bajcsy (2026) WEAVER, better, faster, longer: an effective world model for robotic manipulation. arXiv preprint arXiv:2606.13672. Cited by: §I, §VI-A. [61] D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson (2019) Learning latent dynamics for planning from pixels. In International Conference on machine learning (ICML), Cited by: §D-F, §I, §I, §VII-A, §IX-A. [62] T. Kerssies, G. Berton, J. He, Q. Yu, W. Ma, D. de Geus, G. Dubbelman, and L. Chen (2026) A frame is worth one token: efficient generative world modeling with delta tokens. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 27978–27988. Cited by: §I. [63] D. Nath, A. Srinivasan, H. Yin, R. Jiang, J. Fang, and G. Chou (2026) Pixels to proofs: probabilistically-safe latent world model control via parallel conformal robust mpc. arXiv preprint arXiv:2606.15594. Cited by: §I. [64] W. Zhang, G. Wang, J. Sun, Y. Yuan, and G. Huang (2023) Storm: efficient stochastic transformer based world models for reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS) 36, p. 27147–27166. Cited by: §I. [65] D. Hafner, W. Yan, and T. Lillicrap (2025) Training agents inside of scalable world models. arXiv preprint arXiv:2509.24527. Cited by: §I. [66] S. Huang, P. Kaushik, M. Chen, H. Pan, K. Geng, O. Chehab, F. Moreno-Pino, and M. Simchowitz (2026) Nano world models: a minimalist implementation of future video prediction. arXiv preprint arXiv:2605.23993. Cited by: §I. [67] K. Hsu, H. Hu, and J. F. Fisac (2023) The safety filter: a unified view of safety-critical control in autonomous systems. Annual Review of Control, Robotics, and Autonomous Systems 7. Cited by: Appendix C, Appendix C, §I, §I, §IV, §VI-A. [68] S. Li and O. Bastani (2020) Robust model predictive shielding for safe reinforcement learning with stochastic dynamics. In IEEE International Conference on Robotics and Automation (ICRA), p. 7166–7172. Cited by: §I. [69] O. Bastani (2021) Safe reinforcement learning with nonlinear dynamics via model predictive shielding. In American Control Conference (ACC), p. 3488–3494. Cited by: §I. [70] I. M. Mitchell et al. (2007) A toolbox of level set methods. UBC Department of Computer Science Technical Report TR-2007-11. Cited by: §I, §VII-A. [71] S. Bansal and C. J. Tomlin (2021) Deepreach: a deep learning approach to high-dimensional reachability. In IEEE International Conference on Robotics and Automation (ICRA), p. 1817–1824. Cited by: §I. [72] K. P. Wabersich, A. J. Taylor, J. J. Choi, K. Sreenath, C. J. Tomlin, A. D. Ames, and M. N. Zeilinger (2023) Data-driven safety filters: hamilton-jacobi reachability, control barrier functions, and predictive methods for uncertain systems. IEEE Control Systems Magazine 43 (5), p. 137–177. Cited by: §I, §I, §I. [73] K. Nakamura, A. L. Bishop, S. Man, A. M. Johnson, Z. Manchester, and A. Bajcsy (2026) How to train your latent control barrier function: smooth safety filtering under hard-to-model constraints. In Learning for Dynamics and Control Conference (L4DC), Cited by: §I. [74] C. Xu, T. K. Nguyen, E. Dixon, C. Rodriguez, P. Miller, R. Lee, P. Shah, R. Ambrus, H. Nishimura, and M. Itkina (2025) Can we detect failures without failure data? uncertainty-aware runtime failure detection for imitation learning policies. Robotics: Science and Systems (RSS). Cited by: §D-D, §V-A. [75] T. Ding, A. Angelopoulos, S. Bates, M. Jordan, and R. J. Tibshirani (2023) Class-conditional conformal prediction with many classes. Advances in Neural Information Processing Systems (NeurIPS) 36, p. 64555–64576. Cited by: Appendix B, §V-B. [76] A. Z. Ren, A. Dixit, A. Bodrova, S. Singh, S. Tu, N. Brown, P. Xu, L. Takayama, F. Xia, J. Varley, Z. Xu, D. Sadigh, A. Zeng, and A. Majumdar (2023) Robots that ask for help: uncertainty alignment for large language model planners. In Conference on Robot Learning (CoRL), Cited by: Appendix B, §V-B. [77] J. Wang, H. Hu, D. P. Nguyen, and J. F. Fisac (2024) Magics: adversarial rl with minimax actors guided by implicit critic stackelberg for convergent neural synthesis of robot safety. arXiv preprint arXiv:2409.13867. Cited by: Appendix C, §X, §V-C. [78] M. Kim, K. Nakamura, and A. Bajcsy (2026) How well do latent world models understand partially observable safety constraints?. Conference on Robot Learning (CoRL). Cited by: §VI-A, §VI-A. [79] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song (2023) Diffusion policy: visuomotor policy learning via action diffusion. In Robotics: Science and Systems (RSS), Cited by: §D-E, §VI-A, §VIII-A, §IX-A, §IX-B. [80] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine (2018) Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning (ICML), p. 1861–1870. Cited by: §D-A, §VI-B. [81] D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi (2020) Dream to control: learning behaviors by latent imagination. In International Conference on Learning Representations (ICLR), Cited by: §VII-A. [82] M. G. Bellemare, W. Dabney, and R. Munos (2017) A distributional perspective on reinforcement learning. In International Conference on Machine Learning (ICML), Cited by: §VII-B. [83] W. Dabney, M. Rowland, M. G. Bellemare, and R. Munos (2018) Distributional reinforcement learning with quantile regression. In Proceedings of the AAAI Conference on Artificial Intelligence., Cited by: §VII-B. [84] M. Mittal, P. Roth, J. Tigue, A. Richard, O. Zhang, P. Du, A. Serrano-Muñoz, X. Yao, R. Zurbrüg, N. Rudin, et al. (2025) Isaac lab: a gpu-accelerated simulation framework for multi-modal robot learning. arXiv preprint arXiv:2511.04831. Cited by: §VIII. [85] O. Siméoni, H. V. Vo, M. Seitzer, F. Baldassarre, M. Oquab, C. Jose, V. Khalidov, M. Szafraniec, S. Yi, M. Ramamonjisoa, et al. (2025) Dinov3. arXiv preprint arXiv:2508.10104. Cited by: §D-F, §IX-A. [86] P. Intelligence, K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, et al. (2025) π0.5 _0.5: a vision-language-action model with open-world generalization. arXiv preprint arXiv:2504.16054. Cited by: §IX-A, §IX-B, §IX-C. [87] A. Liang, Y. Korkmaz, J. Zhang, M. Hwang, A. Anwar, S. Kaushik, A. Shah, A. S. Huang, L. Zettlemoyer, D. Fox, Y. Xiang, A. Li, A. Bobu, A. Gupta, S. Tu, E. Biyik, and J. Zhang (2026) Robometer: scaling general-purpose robotic reward models via trajectory comparisons. In Robotics: Science and Systems (RSS), Cited by: §X, §IX-C. [88] A. Balaji, A. Bahety, S. Ambatipudi, D. Lam, J. Xu, and R. Martín-Martín (2026) OopsieVerse: a safety benchmark with damage-aware simulation for robot manipulation. In Robotics: Science and Systems (RSS), Cited by: §X. Appendix A Theoretical Analysis: Robust Safety Filter We provide a theoretical analysis of LUCID for safety filtering with learned latent dynamics, focusing on uncertainty in the latent dynamics and assuming no additional uncertainty from the learned representation. Assuming that the conformalized uncertainty set contains the latent dynamics realized at test time, we show that (i) the robust safety value does not overestimate the safety value under any plausible dynamics, whereas the nominal safety value can, and (i) the perfect robust safety policy achieves zero worst-case suboptimality. Assumption 1 (Markov Latent State). The latent state ztz_t is sufficient for predicting the system’s evolution. Consequently, the induced latent process is Markov and admits a conditional transition distribution of the form fz(zt+1∣zt,at)f_z(z_t+1 z_t,a_t). Zero Safety Overestimation. Safety values computed from nominal imaginations can be overly optimistic when the realized dynamics are adverse. In contrast, the robust safety value evaluates each action under the most adverse plausible transition in the uncertainty set, thereby avoiding overestimation. For a latent dynamics model fzf_z, given a policy π, define the discounted Bellman operator of a value function V based on its imaginations: (fzπV)(z) (T_f_z^πV)(z) :=(1−γ)ℓz(z) :=(1-γ) _z(z) (28) +γminℓz(z),z′∼fz(⋅∣z,π(z))[V(z′)], +γ \! \ _z(z), _z f_z(· z,π(z))[V(z )] \, where its robust counterpart using fzd∈ℱ⁡(fz)f_z^d (f_z) is (robπV)(z):=(1−γ)ℓz(z) (T_rob^πV)(z):=(1-γ) _z(z) (29) +γmin⁡ℓz(z),minfzd∈ℱ⁡(z,π⁡(z))⁡z′∼fzd(z,π⁡(z))[V⁡(z′)]. +γ \! \ _z(z), _f_z^d (z,π(z)) _z f_z^d(z,π(z))[V(z )] \. Let VfzπV_f_z^π and VrobπV_rob^π denote their unique fixed points, and denote the rectangular global uncertainty set is ℱ(fz):=fzd:fzd(⋅∣z,a)∈ℱ(z,a),∀(z,a).F(f_z):= \f_z^d:f_z^d(· z,a) (z,a),\ ∀(z,a) \. (30) Theorem 1 (Robust Value Does Not Overestimate). For any policy π and any plausible latent dynamics fzd∈ℱ⁡(fz)f_z^d (f_z), Vrobπ(z)≤Vfzdπ(z),∀z∈.V_rob^π(z)≤ V_f_z^d^π(z), ∀ z . (31) Proof of Theorem 1. Fix π and fzd∈ℱ⁡(fz)f_z^d (f_z). By (30), fzd(⋅∣z,π(z))f_z^d(· z,π(z)) belongs to the local ambiguity set at every z. The minimum in the robust backup therefore cannot exceed the expectation under this particular transition. Hence, for every bounded V, robπV≤fzdπVpointwise.T_rob^πV _f_z^d^πV . Applying this inequality to the fixed point gives robπVfzdπ≤fzdπVfzdπ=Vfzdπ.T_rob^πV_f_z^d^π _f_z^d^πV_f_z^d^π=V_f_z^d^π. Monotonicity gives (robπ)mVfzdπ≤Vfzdπ(T_rob^π)^mV_f_z^d^π≤ V_f_z^d^π for every m. By contraction, (robπ)mVfzdπ(T_rob^π)^mV_f_z^d^π converges to VrobπV_rob^π, proving (31). ∎ Theorem 1 states that, for any plausible dynamics in the uncertainty set, the safety value computed by robust optimization remains a lower bound on the realized safety value, enabling robust safety filtering. In contrast, the nominal value may overestimate safety when the realized dynamics lead to more pessimistic next states than the nominal model. Remark 1. The above guarantee applies under the perfect coverage guarantee where the true latent dynamics are contained in the uncertainty set, while conformal calibration only provides a marginal probabilistic guarantee. Corollary 1 (Nominal Safety Value Can Overestimate). For a policy π, there exists an admitted latent dynamics fzd∈ℱ⁡(fz)f_z^d (f_z) such that the nominal safety value computed with fzf_z overestimates the realized safety value: Vfzπ(z)≥Vfzdπ(z).V_f_z^π(z)≥ V_f_z^d^π(z). Proof of Corollary 1. Let fzπdf_z _d attain the worst-case admitted value: fzπd∈argminfzd∈ℱ⁡(fz)Vfzdπ(z).f_z _d∈ _f_z^d (f_z)V_f_z^d^π(z). Then, Vfzπdπ(z)=Vrobπ(z).V_f_z _d^π(z)=V_rob^π(z). Since the nominal dynamics fzf_z is also admitted, Theorem 1 gives that the safety value in fzπdf_z _d can be lower than the nominal value computed with fzf_z Vfzπ(z)≥Vrobπ(z)=Vfzπdπ(z).V_f_z^π(z)≥ V_rob^π(z)=V_f_z _d^π(z). Thus, an adverse but plausible dynamics can make the nominal safety value overestimate the realized value. ∎ Sub-optimality of Safety Policy. Robust optimization aims to select actions that remain effective across plausible outcomes, while a nominal policy optimized only under nominal imaginations may select actions that become suboptimal when adverse dynamics are realized. To formalize this, for a policy π, define its state-wise worst-case safety value: wcπ(z):=minfzd∈ℱ⁡(fz)⁡Vfzdπ(z),J_wc^π(z):= _f_z^d (f_z)V_f_z^d^π(z), (32) representing the safety value achieved by π under the worst-case dynamics. We then define the worst-case suboptimality of π as the gap between its worst-case safety value and the maximum safety value achievable for the worst-case: SubOptwc⁡(π,z):=maxπ′⁡wcπ′(z),−wcπ(z).SubOpt_wc(π,z):= _π J_wc^π (z),-J_wc^π(z). (33) Thus, zero suboptimality means that a policy achieves the best possible safety value despite the most adverse dynamics. In this analysis, we write πrobπ^rob for the robust safety policy πrob _rob. Let πrobπ^rob solve the robust Bellman equation (18), and let πnomπ^nom solve the nominal Bellman equation (15). Theorem 2 (Worst-Case Suboptimality of the Robust Policy). For the global uncertainty set in (30) and every z∈z , SubOptwc⁡(πrob,z)=0.SubOpt_wc(π^rob,z)=0. (34) Proof of Theorem 2. Under the rectangular ambiguity set (30), for any π, wcπ(z)=minfzd∈ℱ⁡(fz)⁡Vfzdπ(z)=Vrobπ(z).J_wc^π(z)= _f_z^d (f_z)V_f_z^d^π(z)=V_rob^π(z). (35) Since πrobπ^rob is optimal for the robust Bellman equation, wcπrob(z)=Vrobπrob(z)=maxπ⁡Vrobπ(z)=maxπ⁡wcπ(z),J_wc^π^rob(z)=V_rob^π^rob(z)= _πV_rob^π(z)= _πJ_wc^π(z), which proves SubOptwc⁡(πrob,z)=0SubOpt_wc(π^rob,z)=0. ∎ Corollary 2 (Nominal Safety Policy Can Be Suboptimal). Following Theorem 2, for every z∈z , 0≤SubOptwc⁡(πnom,z)≤2min⁡(1,2γ1−γϵKL).0 _wc(π^nom,z)≤ 2\, (1,\, 2\,γ1-γ _KL ). (36) Also, the worst-case suboptimality is positive whenever wcπnom(z)<wcπrob(z).J_wc^π^nom(z)<J_wc^π^rob(z). Proof of Corollary 2. By Theorem 2, πnomπ^nom is suboptimal: SubOptwc⁡(πnom,z)=wcπrob(z)−wcπnom(z)≥0.SubOpt_wc(π^nom,z)=J_wc^π^rob(z)-J_wc^π^nom(z)≥ 0. Consider any two latent dynamics models f,gf,g, and the fixed-point property and triangle inequality give ‖Vfπ−Vgπ‖∞≤‖fπVfπ−fπVgπ‖∞+‖fπVgπ−gπVgπ‖∞. \|V_f^π-V_g^π\|_∞≤\|T_f^πV_f^π-T_f^πV_g^π\|_∞+\|T_f^πV_g^π-T_g^πV_g^π\|_∞. The first term is at most γ‖Vfπ−Vgπ‖∞γ\|V_f^π-V_g^π\|_∞. Since the minimum is non-expansive, the second term is at most γmaxz|f⁡[Vgπ(z′)]−g⁡[Vgπ(z′)]|,γ _z | _f[V_g^π(z )]- _g[V_g^π(z )] |, where both expectations condition on (z,π⁡(z))(z,π(z)). Thus, ‖Vfπ−Vgπ‖∞≤γ1−γmaxz|f⁡[Vgπ(z′)]−g⁡[Vgπ(z′)]|.\|V_f^π-V_g^π\|_∞≤ γ1-γ _z | _f[V_g^π(z )]- _g[V_g^π(z )] |. (37) For every fzd∈ℱ⁡(fz)f_z^d (f_z), the KL condition in (7) and Pinsker’s inequality imply, for any V:→[−1,1]V:Z→[-1,1], |fzd⁡[V]−fz⁡[V]|≤2ϵKL. | _f_z^d[V]- _f_z[V] |≤ 2 _KL. Substituting this into (37) gives ‖Vfzdπ−Vfzπ‖∞≤κ=2γ1−γϵKL.\|V_f_z^d^π-V_f_z^π\|_∞≤κ= 2\,γ1-γ _KL. (38) Therefore, for every π, Vfzπ(z)−κ≤wcπ(z)≤Vfzπ(z)+κ.V_f_z^π(z)-κ _wc^π(z)≤ V_f_z^π(z)+κ. Nominal optimality gives Vfzπrob(z)≤Vfzπnom(z)V_f_z^π^rob(z)≤ V_f_z^π^nom(z). Hence, SubOptwc⁡(πnom,z) _wc(π^nom,z) =wcπrob(z)−wcπnom(z) =J_wc^π^rob(z)-J_wc^π^nom(z) ≤Vfzπrob(z)+κ−Vfzπnom(z)+κ ≤ V_f_z^π^rob(z)+κ-V_f_z^π^nom(z)+κ ≤2κ.∎ ≤ 2κ. Theorem 2 states that the exact robust policy achieves zero worst-case suboptimality, while Corollary 2 shows that the nominal policy has no such zero-suboptimality guarantee and can be suboptimal if the worst case is realized. Remark 2. The learned disturbance and policy only approximate this game, so the exact guarantees hold to the extent that ambiguity-set feasibility and optimization are achieved. Appendix B Theoretical Analysis: Conformal Prediction In this section, we briefly review conformal prediction for the calibration procedure in Sec. V-B. Conformal prediction constructs prediction sets with finite-sample, distribution-free coverage under exchangeability [17, 18], which can be used as data-calibrated uncertainty sets for robust optimization [13]. Conformal Prediction. Conformal prediction (CP) provides distribution-free coverage guarantees by calibrating a nonconformity threshold on held-out data [17, 18]. In Sec. V-B, let stis_t^i denote the nonconformity score at time t for calibration sample i, and write sis^i when t is fixed. Given a user-specified miscoverage level α, CP sets the threshold ϵε to the appropriate (1−α)(1-α) quantile of the calibration scores. Under exchangeability between the calibration and test samples, this yields ϵ ε :=s(k),k=⌈(n+1)(1−α)⌉,ℙ(stest≤ϵ)≥1−α, :=s^(k), k= (n+1)(1-α) ,\ P\! (s^test≤ε )≥ 1-α, (39) where s(k)s^(k) is the kkth smallest of the n calibration scores. The probability is marginal over both the calibration data and the test sample. We apply this procedure to calibrate the KL-ball radius with (ϵ,α)=(ϵKL,αKL)(ε,α)=( _KL, _KL) and the OOD threshold with (ϵ,α)=(ϵOOD,αood)(ε,α)=( _OOD, _ood). Coverage of the Dynamics-Aware Similarity. Our goal is to calibrate the KL-ball radius ϵKL _KL so that the uncertainty set contains the test-time latent distribution encoded from observations with high probability. In Sec. V-B, we measure the discrepancy between this encoded distribution and the learned latent dynamics using the KL nonconformity score: sKLi:=DKL(ℰ⁡(o≤ti)∥fz(zt−1i,at−1i)).s_KL^i:=D_KL\! (E(o^i_≤ t)\, \|\,f_z(z^i_t-1,a^i_t-1) ). (40) Here, t is fixed as above. We apply (39) with miscoverage level αKL _KL to obtain ϵKL=sKL(k) _KL=s_KL^(k), where k=⌈(n+1)(1−αKL)⌉k= (n+1)(1- _KL) . The following theorem gives the coverage guarantee in (8). Theorem 3 (Marginal KL Coverage). Suppose that the calibration and test KL scores in (40) are exchangeable and k≤nk≤ n. Then, the calibrated KL ball contains the test-time encoded latent distribution with probability at least 1−αKL1- _KL: ℙ⁡(sKLtest≤ϵKL)≥1−αKL.P(s_KL^test≤ _KL)≥ 1- _KL. (41) Proof. The encoded test distribution belongs to the calibrated KL ball whenever its score satisfies sKLtest≤ϵKLs_KL^test≤ _KL. To bound this probability, consider the rank J of the test score among the n calibration scores and the test score, with ties broken uniformly at random. Under exchangeability, the test score is equally likely to occupy any of the n+1n+1 ranks. If J≤kJ≤ k, at most k−1k-1 calibration scores precede the test score, so it cannot exceed the kkth smallest calibration score ϵKL _KL. Therefore, ℙ⁡(sKLtest≤ϵKL)≥ℙ⁡(J≤k)=kn+1≥1−αKL.∎P(s_KL^test≤ _KL) (J≤ k)= kn+1≥ 1- _KL. Dataset-Conditional KL Coverage. In practice, we calibrate ϵKL _KL once using a held-out dataset and keep it fixed during deployment. We therefore consider the probability that the resulting KL ball contains the encoded distribution of a new test sample, conditional on the calibration dataset: pKL(calib):=ℙ⁡(sKLtest≤ϵKL∣calib).p_KL(D_calib):=P(s_KL^test≤ _KL _calib). (42) Because different calibration datasets produce different radii, this coverage varies with calibD_calib. The following theorem provides a lower bound on the coverage attained by the calibrated radius with high probability over the calibration data [18]. Theorem 4 (Dataset-Conditional KL Coverage). Suppose that the calibration and test samples are i.i.d., and use ϵKL=sKL(k) _KL=s_KL^(k) with k=⌈(n+1)(1−αKL)⌉k= (n+1)(1- _KL) . The conditional coverage varies over calibration datasets according to pKL(calib)∼Beta⁡(k,n+1−k).p_KL(D_calib) (k,n+1-k). (43) For a user-specified confidence level 1−η1-η, let bηb_η be the η-quantile of this Beta distribution. Then, with probability at least 1−η1-η over the calibration data, the calibrated KL ball has test coverage at least bηb_η: ℙcalib(pKL(calib)≥bη)≥1−η.P_D_calib\! (p_KL(D_calib)≥ b_η )≥ 1-η. (44) Proof. Let F denote the KL score cumulative distribution function. Since the test sample is independent of the calibration data, pKL(calib)=F⁡(ϵKL)=F⁡(sKL(k)).p_KL(D_calib)=F( _KL)=F(s_KL^(k)). For continuous F, the transformed calibration scores Ui:=F⁡(sKLi)U_i:=F(s_KL^i) are i.i.d. uniform on [0,1][0,1], and the coverage equals their kkth order statistic U(k)U_(k). The event U(k)≤uU_(k)≤ u occurs when at least k of the n scores lie below u. Thus, ℙ⁡(U(k)≤u)=∑j=kn(nj)uj(1−u)n−j,P(U_(k)≤ u)= _j=k^n nju^j(1-u)^n-j, which is the cdf of Beta⁡(k,n+1−k)Beta(k,n+1-k). Since bηb_η is its η-quantile, ℙcalib(pKL(calib)≥bη)=1−ηP_D_calib(p_KL(D_calib)≥ b_η)=1-η, establishing the bound. With ties, the generalized-inverse representation sKLi=F−1(Ui)s_KL^i=F^-1(U_i) gives F⁡(sKL(k))≥U(k)F(s_KL^(k))≥ U_(k), so the lower bound remains valid. ∎ Class-Conditional In-Distribution Detection. Our goal is to calibrate the OOD threshold ϵOOD _OOD so that in-distribution latent states are accepted with high probability. Since OOD states are not directly available, we follow Sec. V-B and use class-conditional conformal prediction to calibrate the threshold using only held-out ID states [75, 11]. Given nIDn_ID calibration states with scores sOODi:=sOOD(zti)s_OOD^i:=s_OOD(z_t^i), we apply (39) with miscoverage level αood _ood: kID:=⌈(nID+1)(1−αood)⌉,ϵOOD:=sOOD(kID),k_ID:= (n_ID+1)(1- _ood) , _OOD:=s_OOD^(k_ID), where we assume kID≤nIDk_ID≤ n_ID. We classify a latent state as OOD when its score exceeds ϵOOD _OOD: Y^(z):=ID,sOOD(z)≤ϵOOD,OOD,sOOD(z)>ϵOOD. Y(z):= casesID,&s_OOD(z)≤ _OOD,\\ OOD,&s_OOD(z)> _OOD. cases (45) To quantify how reliably the detector retains ID states, we define its ID recall as the probability that an actual ID state is classified as ID: RecallID _ID :=ℙ⁡(Y^(ztest)=ID∣Ytest=ID) :=P( Y(z_test)=ID Y_test=ID) (46) =1−ℙ⁡(Y^(ztest)=OOD∣Ytest=ID). =1-P( Y(z_test)=OOD Y_test=ID). Here, YtestY_test denotes the true class, and the probability is taken over the ID calibration data and an ID test state. Thus, guaranteeing recall of at least 1−αood1- _ood limits the probability of incorrectly rejecting an ID state to αood _ood. Theorem 5 (In-Distribution Recall). Suppose that the OOD score function is fixed before calibration and, conditional on Ytest=IDY_test=ID, the nIDn_ID ID calibration states and test state are exchangeable. The calibrated detector in (45) satisfies RecallID=ℙ⁡(sOOD(ztest)≤ϵOOD∣Ytest=ID)≥1−αood.Recall_ID=P\! (s_OOD(z_test)≤ _OOD Y_test=ID )≥ 1- _ood. (47) This establishes the ID recall guarantee in (9). Proof. Conditional on the test state being ID, the calibration and test scores are exchangeable. Let JIDJ_ID be the rank of sOODtest:=sOOD(ztest)s_OOD^test:=s_OOD(z_test) among these nID+1n_ID+1 scores. Exchangeability makes this rank uniform. If JID≤kIDJ_ID≤ k_ID, the test score cannot exceed the kIDk_ID-th calibration score ϵOOD _OOD, so the detector accepts the test state as ID: RecallID≥ℙ⁡(JID≤kID∣Ytest=ID)=kIDnID+1≥1−αood.Recall_ID (J_ID≤ k_ID Y_test=ID)= k_IDn_ID+1≥ 1- _ood. (48) The final inequality follows from the calibration rank. ∎ Thus, calibration controls how often ID states are incorrectly excluded from the uncertainty set, averaged over calibration and test data. The ability to reject OOD states depends on the learned score and is not guaranteed by ID calibration alone. Causal Reconstruction from Trajectory Calibration. Individual transitions within a trajectory are temporally dependent, so the state-level exchangeability assumption need not hold. In Sec. V-B, we therefore calibrate the maximum nonconformity score over each trajectory. Given a calibration trajectory τi=(oti,ati,ot+1i)t=1T−1 _i=\(o_t^i,a_t^i,o_t+1^i)\_t=1^T-1, we compute S⁡(τi):=max1≤t<T⁡sti,ϵtraj:=Quantile1−α⁡(S⁡(τi)i=1Ncalib).S( _i):= _1≤ t<Ts_t^i, ε^traj\!:=\!Quantile_1-α\! (\S( _i)\_i=1^N_calib ). (49) We then use the calibrated threshold at each transition. Since the maximum score is below the threshold exactly when every state-level score is below it, trajectory-level calibration provides simultaneous coverage across time. The following theorem formalizes this causal reconstruction, following [76]. Theorem 6 (Causal Reconstruction of Trajectory Coverage). Suppose that the calibration trajectories and test trajectory are exchangeable, the score functions are fixed, and each sts_t depends only on observations and actions through the scored transition. Let ϵtrajε^traj be calibrated by (49), with k=⌈(Ncalib+1)(1−α)⌉≤Ncalibk= (N_calib+1)(1-α) ≤ N_calib. Applying the same threshold at every transition gives ℙ(sttest≤ϵtraj,∀ 1≤t<T)≥1−α.P\! (s_t^test≤ε^traj,\;∀\,1≤ t<T )≥ 1-α. (50) Proof. Let trajC^traj denote the trajectories accepted by the ϵtrajε^traj and causalC^causal those accepted at every transition: traj ^traj :=τ:S⁡(τ)≤ϵtraj, :=\τ:S(τ)≤ε^traj\, causal ^causal :=τ:st(τ)≤ϵtraj,∀ 1≤t<T. :=\τ:s_t(τ)≤ε^traj,\ ∀\,1≤ t<T\. For any trajectory τ, the definition of its maximum score gives τ∈traj τ ^traj ⇔max1≤t<T⁡st(τ)≤ϵtraj _1≤ t<Ts_t(τ)≤ε^traj ⇔st(τ)≤ϵtraj,∀ 1≤t<T s_t(τ)≤ε^traj, ∀\,1≤ t<T ⇔τ∈causal. τ ^causal. The two sets therefore describe the same acceptance event. Since the trajectories are exchangeable and the score function is fixed, their maximum scores are also exchangeable: ℙ⁡(τtest∈traj)≥kNcalib+1≥1−α.P( _test ^traj)≥ kN_calib+1≥ 1-α. Substituting causal=trajC^causal=C^traj proves (50). ∎ Thus, trajectory-level calibration guarantees coverage at all transitions over the calibrated horizon with a single miscoverage level α, while allowing temporal dependence within each trajectory. For i.i.d. trajectories, the dataset-conditional bound in Theorem 4 also applies to the maximum scores with n=Ncalibn=N_calib. These guarantees rely on exchangeability between calibration and deployment trajectories. Appendix C Background: HJ Reachability Analysis A safety filter is a policy-agnostic control mechanism that safeguards any task policy by evaluating its proposed actions and intervening when necessary to prevent failure under admissible disturbances [67]. We consider the discrete-time system st+1=f⁡(st,at,dt)s_t+1=f(s_t,a_t,d_t), where st∈s_t , at∈a_t , and dt∈Dd_t∈ D denotes an unknown bounded disturbance. Safety is specified by a margin function ℓ , where negative values indicate failure. Following [43], we define the failure set as F=s:ℓ⁡(s)<0F=\s: (s)<0\, so states with zero margin lie on the safety boundary rather than in the failure set. Maximal Robust Safe Set. Our goal is to identify the states from which the robot can avoid failure for all time despite admissible disturbances. Let π:→π:S and πd:→D _d:S→ D be feedback policies, and let sπ,πd(t)s_s^π, _d(t) denote the trajectory starting from s under these policies. We define the maximal robust safe set as Ω⋆:=s∈:∃π,∀πd,∀t∈ℕ0,sπ,πd(t)∉F. := \s :\;∃π,\ ∀ _d,\ ∀ t _0,s_s^π, _d(t)∉ F \. For every state in Ω⋆ , there exists a control policy that keeps the system outside F under every admissible disturbance. Dynamic Programming Isaacs Equation. HJ reachability characterizes this set through a safety value function [7]. Since a trajectory enters failure whenever its margin becomes negative, we evaluate the smallest margin attained over time. The controller maximizes this margin against the worst-case disturbance, giving the zero-sum safety game: V⁡(s):=supπinfπdinft∈ℕ0ℓ⁡(sπ,πd(t)).V(s):= _π _ _d _t _0 \! (s_s^π, _d(t) ). (51) The value V⁡(s)V(s) therefore represents the minimum future safety margin that an optimal controller can maintain despite disturbances. A positive value indicates that the controller can keep the system safely outside the failure set, while a negative value indicates unavoidable failure under an adverse disturbance [12, 67, 43]. To compute this value, we apply the discrete-time dynamic programming Isaacs equation: V⁡(s)=min⁡ℓ⁡(s),maxa∈⁡mind∈D⁡V⁡(f⁡(s,a,d)).V(s)= \! \ (s),\, _a _d∈ DV\! (f(s,a,d) ) \. (52) The infinite-horizon value defines the maximal robust safe set: Ω⋆=s:V⁡(s)≥0, =\s:V(s)≥ 0\, A corresponding robust safety policy is πrob(s)∈argmaxa∈mind∈DV(f⁡(s,a,d)).π^rob(s)∈ _a _d∈ DV\! (f(s,a,d) ). This value function provides a criterion for safety filtering: starting within Ω⋆ , the filter allows πtask(s)π^task(s) if its worst-case next-state value is nonnegative; otherwise, it intervenes with πrob(s)π^rob(s) to keep the system within the robust safe set. Discounted Safety Update. The undiscounted backup in (52) is not generally a contraction. Consequently, a discount factor is introduced γ∈[0,1)γ∈[0,1) for a contraction mapping [23]: V⁡(s)←(1−γ)ℓ(s)+γminℓ⁡(s),maxa∈⁡mind∈D⁡V⁡(f⁡(s,a,d)).V(s)\!←\!(1-γ) (s)+γ \! \ (s),\, _a _d∈ DV\! (f(s,a,d) ) \. (53) Game-Theoretic Adversarial RL. While (53) defines a contraction mapping for the robust safety value function, solving it directly becomes intractable in high-dimensional spaces. To approximate its solution, adversarial actor–critic methods learn a safety critic Qω(s,a,d)Q_ω(s,a,d) together with a control policy and a disturbance policy [12, 3]. The critic evaluates the next-state safety value, Q⁡(s,a,d):=V⁡(f⁡(s,a,d)).Q(s,a,d):=V (f(s,a,d) ). The control actor maximizes the critic value, while the disturbance actor minimizes it. Repeated adversarial gameplay jointly improves the safety-value approximation and the competing policies. However, such game-theoretic learning can oscillate as the two policies continually adapt to one another. To stabilize training, ISAACS updates the disturbance and critic more frequently than the controller and maintains a leaderboard of past opponents, reducing overfitting to a single adversary [12]. Similarly, separating the learning rates allows the disturbance to more closely track a best response while the controller evolves more slowly; together with suitable critic updates, this supports local convergence under the assumptions in [77]. Operational Design Domain. These methods assume access to a system dynamics model f⁡(s,a,d)f(s,a,d), such as a simulator, together with a physically meaningful disturbance space D specified by the designer to capture uncertainty within the operational design domain (ODD), such as bounded external forces or model errors [12]. Thus, while the disturbance policy is learned, the admissible uncertainty itself is prescribed. In LUCID, the game instead operates over learned latent dynamics: the world model provides imagined transitions, while the disturbance perturbs the latent predictive distribution within an uncertainty set calibrated from data. This extends adversarial safety learning to systems for which physical dynamics and disturbances are difficult to specify explicitly. Appendix D Implementation Details D-A Adversarial Reinforcement Learning Algorithm 1 summarizes the game-theoretic adversarial reinforcement learning procedure used to compute the robust latent safety filter in Sec. VI-B. Following Soft Actor-Critic [80], we implement QrobQ_rob with twin safety critics and use their minimum in both the control and disturbance updates to mitigate value overestimation; the target critic Q^rob Q_rob is updated as an exponential moving average of the critic parameters. The replay buffer ℬB stores each imagined transition together with the parameters of the nominal latent predictive distribution, so that the current disturbance policy can efficiently resample perturbed next latent states at every update without re-querying the world model. Rollouts are collected in parallel across independent WM imaginations, and transitions from all imaginations are aggregated into ℬB and sampled in minibatches for replay updates. Each of the NiterN_iter training iterations starts from a newly encoded trajectory and imagines up to HimagH_imag steps. We truncate an imagination once the next latent state satisfies sOOD(zt+1)>ϵOODs_OOD(z_t+1)> _OOD, preventing rollouts from extending into regions unsupported by the training data. Each imagination step is followed by G replay updates, where j counts replay updates across iterations, KπK_π and KdK_d are the policy and disturbance update intervals, ηQ,ηπ,ηd _Q, _π, _d are the corresponding learning rates, and ρ is the target-update coefficient. The world model and the uncertainty models remain fixed throughout safety-filter training. Algorithm 1 Adversarial RL for Robust Latent Safety Filter 0: train,ℰ,fz,ℓz,sOOD,ϵKL,ϵOODD_train,E,f_z, _z,s_OOD, _KL, _OOD; Niter,Himag,G,Kπ,KdN_iter,H_imag,G,K_π,K_d 0: Qrob,πrob,πdQ_rob,\ _rob,\ _d 1: Initialize θQ,θπ,ψ _Q, _π,ψ (zero residual output); θ¯Q←θQ θ_Q← _Q, ℬ←∅B← , j←0j← 0 2: for n=1,…,Nitern=1,…,N_iter do 3: τ∼train,z0∼ℰ(⋅∣o≤t0)τ _train, 9.24994ptz_0 (· o_≤ t_0), with o≤t0o_≤ t_0 from τ 4: for t=0,…,Himag−1t=0,…,H_imag-1 do 5: at∼πrob(⋅∣zt)a_t _rob(· z_t) 6: zt+1∼fzπd(⋅∣zt,at)z_t+1 f_z _d(· z_t,a_t) 7: ℬ←ℬ∪(zt,at,ℓz(zt),fz(⋅∣zt,at))B ∪\(z_t,a_t, _z(z_t),f_z(· z_t,a_t))\ 8: for g=1,…,Gg=1,…,G do 9: ℳ∼ℬ,j←j+1M , 9.24994ptj← j+1 10: z′∼fzπd(⋅∣z,a),a′∼πrob(⋅∣z′),(z,a)∈ℳz f_z _d(· z,a), 9.24994pta _rob(· z ), 9.24994pt(z,a) 11: θQ←θQ−ηQ∇θQℒQrob _Q← _Q- _Q _ _QL_Q_rob (20) 12: if jmodKπ=0j K_π=0 then 13: a^∼πrob(⋅∣z) a _rob(· z) 14: θπ←θπ−ηπ∇θπℒπrob _π← _π- _π _ _πL_ _rob (21b) 15: end if 16: if jmodKd=0j K_d=0 then 17: z′∼fzπd(⋅∣z,a)z f_z _d(· z,a) (reparameterized) 18: a′←stopgradθπ⁡(πrob(z′))a _ _π\! ( _rob(z ) ) 19: ψ←ψ−ηd∇ψℒπdψ←ψ- _d _ψL_ _d (21a) 20: end if 21: θ¯Q←(1−ρ)θ¯Q+ρθQ θ_Q←(1-ρ) θ_Q+ρ _Q 22: end for 23: if sOOD(zt+1)>ϵOODs_OOD(z_t+1)> _OOD then 24: break ⊳ truncate OOD imagination 25: end if 26: end for 27: end for D-B Latent Safety Filter Training Because the world model is trained offline, actions proposed during safety filter training may be poorly represented in the training data for the current latent state. Such out-of-distribution (OOD) state–action pairs can induce hallucinated transitions, allowing the safety policy to exploit model errors and overestimate safety. Following UNISafe [11], we incorporate epistemic uncertainty into the safety margin to discourage these unreliable imaginations. Augmenting the task failure margin with this uncertainty criterion encourages the learned safety policy to select actions whose predicted outcomes are both safe and supported by the offline data. D-C Disturbance Parameterization In Sec. V-C, we parameterize the latent disturbance πd(z,a) _d(z,a) as an additive residual to the nominal distribution parameters, enabling optimization over plausible next-state distributions through (11)–(12). We implement this network as an MLP that takes features of the nominal next-state prediction: the deterministic features h and prior parameters (μ,σ)(μ,σ) for Gaussian dynamics, or h alone for categorical dynamics. Thus, the current latent state and action condition the disturbance through the nominal prediction. The residual changes only the stochastic prior, preserving the deterministic features. TABLE VII: Residual parameterization of πd _d in (11). Component Gaussian Categorical Input [h,μ,σ][h,μ,σ] h Input dimension 512+32+32512+32+32 512512 Hidden layers 44 layers, 512512 units each Normalization / activation LayerNorm / ReLU Output residual (Δμ,Δσ)( μ, σ) Δϕ φ Output dimension 32+3232+32 32×3232× 32 Output transform (tanh, 0.5tanh)( ,\,0.5 ) 15tanh15 Prior update μ+Δμ+ μ σ+Δσ+ σ ϕ+Δϕφ+ φ Output initialization Zero weights and bias Table VII summarizes the parameterization. For Gaussian dynamics, we perturb the standard deviation σ, and the covariance in (12) is obtained by squaring these standard deviations. For categorical dynamics, we add the residual to the prior logits ϕφ. Zero initialization of the output layer makes the initial residual zero, and the KL and OOD penalties constrain the learned perturbation during optimization. D-D 3D Dubins’ Car with Disturbances Optimal Solution of Naughty Dubins’ Car. In this section, we derive the optimal robust control and worst-case disturbance for the naughty Dubins’ car in (6): p˙x=vcos⁡θ,p˙y=vsin⁡θ,θ˙=δa, p_x=v θ, p_y=v θ, θ=δ a, Let Vgt(s)V_ gt(s) be the optimal robust safety value for these physical dynamics. Define its gradient components as gx:=∂Vgt∂px,gy:=∂Vgt∂py,gθ:=∂Vgt∂θ.g_x:= ∂ V_ gt∂ p_x, g_y:= ∂ V_ gt∂ p_y, g_θ:= ∂ V_ gt∂θ. By the chain rule, the Hamiltonian gives its instantaneous change along the dynamics: H⁡(s,a,δ):=∇sVgt(s)⊤s˙=v⁡(gxcos⁡θ+gysin⁡θ)+gθδa.H(s,a,δ):= _sV_ gt(s) s=v(g_x θ+g_y θ)+g_θδ a. The action maximizes this, while the disturbance minimizes it after observing the action. The optimization reduces to max|a|≤1.25⁡minδ∈−1,1gθδa=max|a|≤1.25−|gθa|=0. _|a|≤ 1.25 _δ∈\-1,1\g_θδ a= _|a|≤ 1.25-|g_θa|=0. Thus, a Hamiltonian-optimal robust action is ⋆=a =0, and a worst-case response to any proposed action is δ⋆=−1,gθa>0,+1,gθa≤0.δ = cases-1,&g_θa>0,\\ +1,&g_θa≤ 0. cases When gθ≠0g_θ≠ 0, every nonzero turn lets the disturbance decrease safety through the heading contribution. Driving straight, as in Fig. 2, eliminates this contribution. Latent Safety Filter Setup. World model hyperparameters are listed in Table VIII and safety filter training settings are listed in Table IX. We add the predicted residuals to the mean and standard deviation of the Gaussian prior, with the standard deviation bounded below by 0.10.1. We train a flow-matching density model on frozen latents and use its logpZO score [74] to penalize implausible perturbations. For safety learning, we train a Gaussian safety actor and twin critics using the discounted safety update in (20). The actor and critic networks each use four hidden layers of width 512512, and the critic outputs are bounded by 1.5tanh(⋅)1.5 (·). We update the critics, safety policy, and disturbance every 11, 44, and 22 replay updates, respectively. TABLE VIII: 3D Dubins’ Car Latent World Model Settings. Hyperparameter Value Image Dimension 128×128×3128× 128× 3 Action Dimension 11 (continuous) Stochastic Latent Gaussian Deterministic Dimension 512512 Stochastic Dimension 3232 Activation Function SiLU Encoder CNN Depth 3232 Failure Classifier Layers 22 Batch Size 1616 Batch Length 3232 Optimizer Adam Learning Rate 10−410^-4 Training Iterations 50,00050,000 TABLE IX: 3D Dubins’ Car Latent Safety Filter Settings. Parameter Value Actor Learning Rate 10−310^-3 Critic Learning Rate 10−410^-4 Disturbance Learning Rate 10−410^-4 Discount Factor 0.85→0.99990.85→ 0.9999 Training Iterations 100,000100,000 Replay Buffer Size 100,000100,000 Batch Size 512512 Parallel Imagination States 512512 Imagination Steps 5050 D-E Simulation: Block Pouring System Disturbance. We randomize the environment physics at reset, with all randomized parameters sampled uniformly from the ranges listed in Table X. The poses of all three cubes are perturbed within the specified bounds, while the physical properties of the green cube are randomized independently. These parameters remain fixed throughout each episode. For paired evaluation, we pre-sample 2020 physics configurations with a fixed seed and reuse them across source trajectories and filtering conditions, ensuring identical physics for corresponding rollouts. TABLE X: Physical Randomization of Block Pouring in IsaacLab. Parameter Randomization Range Cube Position x [−0.02, 0.02][-0.02,\,0.02] m Cube Position y [−0.02, 0.02][-0.02,\,0.02] m Mass [0.8, 1.2][0.8,\,1.2] kg Static Friction [0.2, 0.8][0.2,\,0.8] Dynamic Friction [0.1, 0.5][0.1,\,0.5] Restitution [0.05, 0.3][0.05,\,0.3] Latent Safety Filter Setup. World model hyperparameters are listed in Table XI and safety filter training settings are listed in Table XII. Following UNISafe [11], a five-member probabilistic ensemble provides the log-JRD uncertainty score used in the safety margin to prevent OOD action exploitation of WM imaginations [11]. For safety filter synthesis, the actor and critic networks each use four hidden layers of width 512512, and update the critics, safety policy, and disturbance every 11, 88, and 22 replay updates, respectively. TABLE XI: Block Pouring World Model Training Settings. Hyperparameter Value Image Dimension 2×128×128×32× 128× 128× 3 Proprioception Dimension 77 (eef position/quaternion) Action Dimension 77 (continuous) Stochastic Latent Categorical Deterministic Dimension 512512 Stochastic Dimension 32×3232× 32 Activation Function SiLU Encoder CNN Depth 3232 Encoder MLP Layers 55 Failure Classifier Layers 33 Batch Size 1616 Batch Length 6464 Optimizer Adam Learning Rate 10−410^-4 Training Iterations 1,000,0001,000,000 TABLE XII: Simulated manipulation safety-learning settings. Parameter Value Actor Learning Rate 10−310^-3 Critic Learning Rate 10−310^-3 Disturbance Learning Rate 10−310^-3 Discount Factor 0.950.95 Training Iterations 400,000400,000 Replay Buffer Size 1,000,0001,000,000 Batch Size 512512 Parallel Imagination States 1,0241,024 Imagination Steps 5050 Task Policy. We train the DreamerV3 as a task policy [8] independently of the safety filter. The actor optimizes imagined λ-returns with λ=0.95λ=0.95 and continuation-weighted discount, backpropagating through the dynamics with an entropy coefficient of 3×10−43× 10^-4. We normalize advantages using moving 55th and 9595th return percentiles (update rate 10−210^-2, minimum scale 11), and the critic uses a twohot log-likelihood objective with a slow-target regularizer. We also use Diffusion Policy [79] by behavior cloning as a task policy. A vision-conditioned MLP or conditional U-Net predicts 88 future actions of 77D using 100100 denoising steps. Both are trained for 1,0001,000 epochs at learning rate 10−410^-4; the MLP uses ViT image encoders and batch size 256256, while the U-Net uses ResNet-18 encoders and batch size 6464. During deployment, the task policy executes four actions before replanning. D-F Real-World: Sunny-Side-Up Egg Serving TABLE XIII: Transformer-Based Latent World Model RSSM Transformer RSSM Visual encoder Trainable CNN Frozen DINOv3 Temporal model GRU Causal transformer History Recurrent 88-step attention window Recon. Target Pixels Pixels and DINO features TABLE XIV: Real World: World Model Hyperparameters. Hyperparameter Value Image Dimension 2×192×256×32× 192× 256× 3 Proprioception Dimension 88 Action Dimension 77 (continuous) Stochastic Latent Gaussian Deterministic Dimension 512512 Stochastic Dimension 3232 Activation Function SiLU Encoder MLP Layers 22 (proprioception) Failure Classifier Layers 33 per failure head Batch Size 2020 Batch Length 6464 Optimizer Adam Learning Rate 10−410^-4 Training Iterations 200,000200,000 TABLE XV: Real World Latent Safety Filter Setups. Parameter Value Actor Learning Rate 5×10−45× 10^-4 Critic Learning Rate 10−310^-3 Disturbance Learning Rate 10−310^-3 Target Smoothing 0.0050.005 Discount Factor 0.85→0.99990.85→ 0.9999 Training Iterations 400,000400,000 Replay Buffer Size 500,000500,000 Batch Size 512512 Parallel Imagination States 256256 Imagination Steps 3030 Latent World Model. Each observation contains RGB images from two tabletop cameras, seven robot joint positions, and the gripper width, while actions specify an end-effector delta pose and gripper command. Failures are manually labeled for each frame based on falling and flipping conditions. For latent world mode, we adapt the RSSM [61] with pretrained visual features and transformer-based latent dynamics, with the main architectural differences summarized in Table XIII. A frozen DINOv3 ViT-S/16+ [85] extracts 192192 patch tokens of dimension 384384 from each image; learned view embeddings distinguish the two cameras, and a two-layer transformer fuses their tokens. Eight learned queries pool the fused tokens into 3,0723,072 visual features, while a separate MLP maps the eight proprioceptive inputs to 512512 features. Their concatenation forms the 3,5843,584-dimensional observation embedding. For latent dynamics, a causal transformer predicts 512512 deterministic features from an eight-step history of stochastic states and actions, providing the same temporal context during training and imagination. Actions are embedded and injected into each transformer block through AdaLN-zero conditioning. The prior predicts a 3232-dimensional diagonal Gaussian from the deterministic features, and concatenating a stochastic sample with the deterministic features yields the 544544-dimensional latent state z used by the safety filter. We jointly train pixel and frozen-feature reconstruction together with continuation, falling, and flipping prediction. Table XIV lists the architecture and training hyperparameters. Latent Safety Filter Setup. Table XV summarizes the safety-filter training settings. Following Algorithm 1, we update the critics, safety policy, and disturbance every 11, 44, and 22 replay updates, respectively.

Instant research discovery