Paper deep dive
When World Models Lie: Adaptive Safety Analysis Under Wrong Imaginations
John Cao, Somil Bansal
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:World models offer a powerful substrate for safety reasoning in high-dimensional robotic systems, but they are also fallible: their predictions can be biased, miscalibrated, or confidently wrong. This creates a central challenge for latent-space safety filters, which often learn Hamilton-Jacobi safety value functions on the dynamics of a world model. If the world model is incorrect, the resulting value function can inherit its errors and produce overconfident safety estimates. Existing latent safety filters often rely on auxiliary signals such as ensemble disagreement or value-target consistency residuals for adaptation, but these signals can remain small even when the world model's predictions deviate from observations. We propose an adaptive latent safety filter that calibrates safety reasoning using directly observed world-model error. Our method uses Adaptive Conformal Inference to construct online uncertainty sets from discrepancies between predicted and observation-inferred latent states, then evaluates safety pessimistically by minimizing the learned value function over these sets. This allows the filter to remain minimally conservative when the world model is accurate, while becoming more cautious when observations reveal model mismatch. We provide a finite-time coverage guarantee for the adaptive uncertainty radius. Through simulation and hardware experiments, we show that our method significantly reduces failures relative to state-of-the-art latent safety filters while preserving task completion.
Tags
Links
- Source: https://arxiv.org/abs/2609.34300v1
- Canonical: https://arxiv.org/abs/2609.34300v1
Trouble viewing inline? Open PDF directly →
Full Text
45,772 characters extracted from source content.
Expand or collapse full text
When World Models Lie: Adaptive Safety Analysis Under Wrong Imaginations John Cao Somil Bansal †thanks: This research is supported in part by the NSF CAREER program (2240163), the TRI University Research Program (URP) 3.0, and the NASA ULI program.†thanks: The authors are with the Department of Aeronautics and Astronautics, Stanford University. johncao,somil@stanford.edu Abstract World models offer a powerful substrate for safety reasoning in high-dimensional robotic systems, but they are also fallible: their predictions can be biased, miscalibrated, or confidently wrong. This creates a central challenge for latent-space safety filters, which often learn Hamilton-Jacobi safety value functions on the dynamics of a world model. If the world model is incorrect, the resulting value function can inherit its errors and produce overconfident safety estimates. Existing latent safety filters often rely on auxiliary signals such as ensemble disagreement or value-target consistency residuals for adaptation, but these signals can remain small even when the world model’s predictions deviate from observations. We propose an adaptive latent safety filter that calibrates safety reasoning using directly observed world-model error. Our method uses Adaptive Conformal Inference to construct online uncertainty sets from discrepancies between predicted and observation-inferred latent states, then evaluates safety pessimistically by minimizing the learned value function over these sets. This allows the filter to remain minimally conservative when the world model is accurate, while becoming more cautious when observations reveal model mismatch. We provide a finite-time coverage guarantee for the adaptive uncertainty radius. Through simulation and hardware experiments, we show that our method significantly reduces failures relative to state-of-the-art latent safety filters while preserving task completion. Our project page is available at https://mudhdhoo.github.io/WhenWordModelsLie_project_page/. I Introduction Robotic systems are becoming increasingly capable of performing sophisticated tasks in complex, unstructured, and human-centric environments [1, 2, 3]. As these systems become more capable, it is increasingly important to ensure that they can act without compromising the safety of their surroundings. Safety filters provide a mechanism for doing so by overriding or modifying actions that may lead to unsafe outcomes. Control Barrier Functions [4] and Hamilton-Jacobi (HJ) reachability [5] are two dominant frameworks for safety filtering, but their reliance on explicit dynamics models and safety constraints has historically limited their application in open-world settings. Fig. 1: Our method grounds safety adaptation directly in observed prediction error. We compare the latent state predicted by the world model with the latent state inferred from the current observation, compute an uncertainty set over plausible latent prediction deviations, and then reason pessimistically over the resulting uncertainty set. Pessimism scales with the size of the errors, providing more caution under an inaccurate world model, while being non-invasive when the model is reliable. Latent-space safety filters have recently emerged as a promising paradigm for safety filtering from high-dimensional observations [6]. These methods learn a compact latent world model [7] and train an HJ-style safety value function using rollouts imagined by the learned latent dynamics. The resulting value function can then be used to evaluate the safety of a robot’s actions from image-based observations, enabling safety filtering in settings where classical model-based methods are difficult to apply. However, latent-space safety filtering introduces a fundamental challenge: the model used for safety reasoning is itself learned and fallible. Classical CBF and HJ reachability methods typically assume access to a known dynamics model and well-specified failure set, under which formal safety guarantees can be derived. In contrast, learned world models are only approximate representations of the real world [8]. They may be naturally erroneous even under nominal deployment conditions, and their errors can grow under distribution shift, unmodeled dynamics, or changes in the environment. Indeed, limiting errors and hallucinations in world models is an active area of research in robotics [9, 10, 11]. This is especially problematic for latent safety filters because errors in the world model can propagate into the learned safety value function. An optimistic world model may imagine benign outcomes for unsafe actions, causing the value function to become overconfident and delaying safety interventions precisely when they are needed most. Prior work has begun to address the fragility of latent safety filters under model error. World-model-based runtime monitors have been proposed to detect unsafe behaviors [12], but detection alone does not resolve how the robot should intervene safely. UNISafe [13] treats out-of-distribution latent states as unsafe by using the Jensen-Rényi Divergence (JRD) of a probabilistic prediction ensemble as an uncertainty score. However, as we demonstrate in our experiments, ensemble uncertainty can remain low even when predictions are systematically biased or incorrect [14], leading the system to be confidently wrong. ACoFi [15] adapts the safety threshold using the discrepancy between the learned value function and its bootstrapped training target. Yet when the value function is trained on erroneous latent dynamics, this residual may reflect self-consistency within the incorrect model rather than disagreement with the real system. As a result, auxiliary uncertainty or value-consistency signals may fail to reveal the underlying prediction errors that compromise safety. In this work, we propose an adaptive latent safety filter grounded in observed discrepancies between predicted and inferred latent states. Using Adaptive Conformal Inference (ACI) [16], we construct online uncertainty sets and minimize the nominal safety value over these sets, adapting the filter’s conservatism to observed world-model error. Our contributions are as follows: 1. We propose an adaptive latent safety filter that uses Adaptive Conformal Inference to construct uncertainty sets from observed world-model prediction errors and evaluates safety pessimistically by minimizing the nominal value function over these sets. 2. We provide a finite-time coverage guarantee for the adaptive uncertainty radius used by our method. 3. Through both qualitative and quantitative case studies across simulation and hardware, we show how our approach overcomes the limitations of baselines and its effectiveness in reducing failures despite incorrect world models, while preserving task performance. I Problem Formulation We study the problem of online safety reasoning with fallible world models. Consider a latent world model zt∼ℰ(⋅|z^t,ot)z^t∼f(⋅|zt−1,at−1), z_t (·| z_t,o_t) 14.22636pt z_t f(·|z_t-1,a_t-1), (1) where ℰE is an encoder that infers the current latent state from the predicted latent state z^t z_t and observation oto_t, and f is a learned latent dynamics model that autoregressively predicts the next latent state. For instance, f can be trained offline using a dataset of robot–environment interactions, consisting of observation and action trajectories [12, 6]. In this work, we are particularly concerned with the setting where the world model f might be wrong, e.g., due to learning errors or hallucinations. The system safety constraints are specified in the latent space via a failure set ℱ=z:l(z)<0F=\z:l(z)<0\, encoded via the zero-sublevel set of a margin function l. In practice, l is learned as a binary classifier on known safe/unsafe trajectories [17]. We also assume access to a nominal safety value function V(z)V(z) that encodes the safety of a latent state under the nominal world model and can be used for monitoring, planning, or filtering. Such a value function can be trained or computed using the latent dynamics in (1), using reachability analysis for instance [6] (see Sec. I-A). However, because the latent dynamics f only approximate the true evolution of the system, the nominal value function V may be miscalibrated during deployment. In particular, prediction errors in the world model can induce errors in the estimated safety value, causing the safety filter to be either overly conservative or, more dangerously, overconfident in unsafe regions. Our goal is to adapt the safety analysis online by using real-time observations to estimate the world model’s prediction error and compensate for its effect on the safety value. Specifically, this work aims to: 1. Estimate the prediction errors made by the world model during deployment. 2. Use this error estimate to compute an adapted safety value function V¯ V that accounts for model error and reduces overconfident safety predictions. I Background I-A Latent Space Safety Filters Latent space safety filters perform HJ Reachability in the latent space of world models [6]. Given a set of failure states ℱF and the margin function l, HJ Reachability analysis defines the value function under a dynamics model f as the optimal control problem V(zt)=supπinfs≥0l(ξztπ(s))V(z_t)= _π _s≥ 0l( _z_t^π(s)), where ξztπ _z_t^π is a trajectory starting from ztz_t and evolving under the dynamics of f under the policy π. This can be interpreted as the worst future cost incurred when using the maximally safe policy; V<0V<0 therefore means that the system is doomed to fail despite the best efforts. The value function satisfies the fixed-point Bellman equation V(zt)=minl(zt),maxaV(zt+1). V(z_t)= \l(z_t), _aV(z_t+1)\. (2) In latent space safety filtering, l is learned as a classifier from a dataset of high-dimensional failure-labeled observations. The value function V(z)V(z) is learned using reinforcement learning (RL) on the world model, and (2) is replaced with the corresponding discounted contraction mapping V(zt)=(1−γ)l(zt)+γminl(zt),maxaV(z^t+1) V(z_t)=(1-γ)l(z_t)+γ \l(z_t), _aV( z_t+1)\ (3) where z^t+1∼f(⋅|zt,at) z_t+1 f(·|z_t,a_t) and γ∈[0,1)γ∈[0,1) is the discount factor [18]. This safety Bellman equation is used within RL as a backup target to jointly learn V (critic) and a safety policy (actor) πsafe(z)=argmaxaV(z) _safe(z)= _aV(z) that computes the maximally safe action at a given state. Safety filtering for a nominal policy πnom _nom is then implemented as at∼π=[V(z^t+1)>δ]πnom+[V(z^t+1)≤δ]πsafe a_t π=1 [V( z_t+1)>δ ] _nom+1 [V( z_t+1)≤δ ] _safe (4) where δ≥0δ≥ 0 is a user-defined safety threshold. I-B Adaptive Conformal Inference Adaptive Conformal Inference (ACI) [16] calibrates prediction sets online under distribution shift. Given an input–output pair (xt,yt)(x_t,y_t), let st=St(xt,yt)s_t=S_t(x_t,y_t) be a finite nonconformity score, with larger values indicating poorer agreement with the prediction. Using a non-decreasing quantile function Q^t Q_t fitted to observed scores, ACI constructs Ct=y:St(xt,y)≤qt,qt=Q^t(1−αt). C_t=\y:S_t(x_t,y)≤ q_t\, q_t= Q_t(1- _t). (5) For a miscoverage level α∈(0,1)α∈(0,1), the adaptive parameter is initialized as α1=α _1=α and updated after observing yty_t: αt+1=αt+λ(α−errt),errt=[st>qt], _t+1= _t+λ(α-err_t), _t=1[s_t>q_t], (6) where λ>0λ>0 is a step size controlling the adaptation rate. Miscoverage decreases αt _t, inducing larger subsequent prediction sets, while coverage conversely allows for tighter sets. Larger λ enables faster adaptation but trades off stability. Proposition 1 (ACI coverage [16, Proposition 4.1]). Suppose α1=α∈(0,1) _1=α∈(0,1) and the update (6) is applied with λ>0λ>0 without clipping αt _t. Under the conventions Q^t(p)=−∞ Q_t(p)=-∞ for p≤0p≤ 0 and Q^t(p)=+∞ Q_t(p)=+∞ for p≥1p≥ 1, with probability 1, for all T∈ℕT , |1T∑t=1Terrt−α|≤maxα,1−α+λTλ. | 1T _t=1^Terr_t-α |≤ \α,1-α\+λTλ. (7) Proposition 1 puts no constraints on the distribution generating the data, meaning that (7) is valid even when exchangeability does not hold between subsequent time steps. In our method, latent prediction errors provide the nonconformity scores used to construct uncertainty sets for pessimistic safety evaluation. IV Method We propose an online safety-filtering framework that enables safety reasoning with fallible latent world models. The key idea is to avoid treating the world model prediction as a point estimate that can be trusted uniformly during deployment. Instead, we use the discrepancy between predicted and inferred latent states as real-time feedback about the reliability of the world model, and use this feedback to construct a pessimistic safety value that accounts for possible model errors. At each time step, the world model predicts a latent state z^t z_t using the learned latent dynamics, while the encoder infers a latent state ztz_t from the current observation. We define the instantaneous latent prediction error as et=‖zt−z^t‖2e_t=||z_t- z_t||_2. This error provides a simple online signal of how well the latent dynamics are tracking the true system evolution. When the prediction error is small, the nominal value function can be trusted more closely. When the prediction error grows, the safety filter should become more cautious, since the nominal value function may be overconfident in regions where the world model is inaccurate. Algorithm 1 Pessimistic Latent Safety Filtering 1: Encoder ℰE, dynamics f, value function V, nominal policy πnom _nom, safety policy πsafe _safe trust radius qmaxq_max, error buffer ℬB, threshold δ 2: for t=1:Tt=1:T do 3: zt∼ℰ(⋅|z^t,ot)z_t (·| z_t,o_t) ⊳ Observe and encode 4: et=‖zt−z^t‖2e_t=||z_t- z_t||_2 ⊳ Compute prediction error 5: Append ete_t to buffer ℬB 6: Estimate uncertainty set t+1(ℬ)U_t+1(B) 7: z^t+1∼f(⋅|zt,πnom(zt)) z_t+1 f(·|z_t, _nom(z_t)) ⊳ Estimate next state 8: Compute pessimistic value function V¯(z^t+1)= V( z_t+1)= minzV(z) _zV(z) (8) s.tz∈t+1 s.t 14.22636ptz _t+1 9: Filter the nominal action using V¯ V according to (4). 10: end for IV-A Online Safety Adaptation via Pessimistic Imaginations We introduce pessimistic imaginations for latent safety filtering, outlined in Algorithm 1. At each time step, we compute the latent prediction error ete_t and add it to a buffer of previously observed errors. This buffer is used to construct an uncertainty set t+1U_t+1 around the next latent prediction z^t+1 z_t+1. The uncertainty set represents latent states that are plausible given the recent mismatch between the world model and the observed system behavior. Rather than evaluating safety only at the predicted latent state z^t+1 z_t+1, we evaluate safety pessimistically over t+1U_t+1. In particular, we construct an adapted safety value V¯ V by solving the corresponding constrained optimization problem in (8) using projected gradient descent (PGD). This procedure has an intuitive interpretation: The resulting value V¯ V reflects the worst-case safety outcome among latent states that are consistent with the current estimate of model error. We then use V¯ V in place of the nominal value V(z^t+1)V( z_t+1) for filtering the proposed action. As a result, the filter remains minimally conservative when the world model is accurate, but becomes more cautious when online observations reveal that the model is unreliable. IV-B Adaptive Conformal Inference for Online Uncertainty Estimation We use ACI [16] to construct the uncertainty set t+1U_t+1. While Algorithm 1 is not tied to a particular uncertainty-estimation method, ACI provides a simple distribution-free mechanism for adapting uncertainty sets under non-stationary deployment conditions. This is especially useful in the context of learned world models, whose prediction errors may vary across tasks, environments, and phases of execution. We define the nonconformity score at time t as the latent prediction error st=et=‖zt−z^t‖2. s_t=e_t=||z_t- z_t||_2. (9) Given a target miscoverage level α, the adaptive parameter is updated according to (6), with errt=[||zt−z^t||2>qt]err_t=1 [||z_t- z_t||_2>q_t ]. We then compute an empirical quantile qt+1q_t+1 of the observed nonconformity scores and define the next uncertainty set as t+1=z:‖z−z^t+1‖2≤min(qt+1,qmax), _t+1=\z:||z- z_t+1||_2≤ (q_t+1,q_max)\, (10) where qt+1q_t+1 is the fitted ⌈(t+1)(1−αt+1)⌉t (t+1)(1- _t+1) t quantile of the nonconformity scores, and qmaxq_max defines a trust-region radius that limits the pessimistic optimization to a calibrated neighborhood of the predicted latent state. The trust-region radius qmaxq_max controls the tradeoff between safety and conservatism. A larger radius allows the filter to guard against a broader range of model errors, but may introduce excessive conservatism by considering latent states that are not physically meaningful. A smaller radius keeps the pessimistic imagination closer to the learned latent manifold, but may fail to account for safety-relevant prediction errors. To choose qmaxq_max systematically, we leverage trajectory-level conformal prediction [13, 19]. We construct a calibration dataset cal=τii=1NcalD_cal=\ _i\_i=1^N_cal with trajectories τi=oti,atit=1T _i=\o_t^i,a_t^i\_t=1^T. For each trajectory τi _i, we compute the sequence of latent prediction errors e1i,…,eTie_1^i,…,e_T^i and define the trajectory-level nonconformity score QτiαtrajQ_ _i _traj as the (1−αtraj)(1- _traj) quantile of these errors. The trust region radius qmaxq_max is then chosen as the 1−αcal1- _cal quantile of the computed trajectory-level scores Qτiαtraji=1Ncal\Q_ _i _traj\_i=1^N_cal. Under exchangeability between a new trajectory τnew _new and the calibration set calD_cal, conformal calibration gives ℙ(Qτnewαtraj≤qmax)≥1−αcal. (Q_ _new _traj≤ q_max)≥ 1- _cal. (11) This guarantee does not imply that every latent state considered by the pessimistic optimization in (8) is physically realizable. Rather, it provides a principled way to choose the maximum adaptation radius so that the filter can trade off sensitivity to world-model error against unnecessary conservatism within the calibrated operating distribution. When little online data has been observed, we initialize q1=∞q_1=∞, which makes the uncertainty radius equal to the trust-region radius qmaxq_ . Thus, the filter begins cautiously by searching over the full calibrated trust region, and then becomes more adaptive as deployment data accumulates. IV-C Coverage Guarantee The following result combines trajectory-level calibration of qmaxq_ with the online ACI update used to construct t+1U_t+1. Theorem 1. Let qmaxq_max be chosen using the procedure in Section IV-B, and tU_t be computed using Adaptive Conformal Inference with radius q¯t=min(qt,qmax) q_t= (q_t,q_max). Let rT=max(α,1−α)+λTλr_T= (α,1-α)+λTλ and 1 be the indicator function. Given a new trajectory τ, assuming exchangeability with calD_cal, ℙ(1T∑t=1T[et>q¯t]≤α+αtraj+rT)≥1−αcal. ( 1T _t=1^T1[e_t> q_t]≤α+ _traj+r_T )≥ 1- _cal. (12) The proof for Theorem 1 is provided in the appendix. Intuitively, this theorem states that, with high probability over calibrated trajectories, the fraction of time steps at which the realized latent prediction error exceeds the adaptive uncertainty radius is controlled by the target online miscoverage level, the trajectory-level calibration tolerance, and a finite-time adaptation term. In our setting, this bound provides a statistical basis for using t+1U_t+1 as the set of plausible latent deviations over which the safety value is pessimistically evaluated. V Case Studies We evaluate our method in two simulated case studies. First, we use Dubins car to isolate the properties and benefits of error-feedback-based safety adaptation. We then perform a quantitative study on a manipulation task using a Franka Panda arm. Together, these case studies evaluate whether latent safety filters can remain effective when the world model used for safety reasoning is fallible. Fig. 2: Case Study: Prediction errors allow our method to engage safety fallback earlier to avoid collision. Maximum pessimism is held at the start, when data is scarce. Baselines. We compare our method to four baselines: 1) an unfiltered nominal policy, 2) LatentSafe [6], 3) UNISafe [13], and 4) AcoFi [15]. For all baselines, we train a DreamerV3 [20] world model for each environment, trained on a dataset of success and failure trajectories. The safety margin function is trained on the same dataset using the method in [17]. The safety filters are then trained using RL within the imagination of the world model, as described in Sec. I-A. We aim to demonstrate how our approach addresses blind spots in current methods that can otherwise lead to safety violations when the world model is incorrect. Metrics. We evaluate methods using three metrics: 1) Success: the number of episodes in which the intended task is achieved without safety violations; 2) Failure: the number of episodes with safety violations; 3) Incompletion: the number of episodes that neither fail nor succeed. V-A Case Study 1: Slippery Dubins Car We evaluate error-feedback-based safety adaptation using a Dubins car with a “slippery” region that increases the car’s velocity by a factor of 1.5 relative to the training dynamics. We train a world model on a dataset of 1500 nominal trajectories collected without the slippery zone, each composed of failure-labeled observation-action pairs ot,at,ytt=1T\o_t,a_t,y_t\_t=1^T, where oto_t is an image snapshot of the environment at time t, ata_t is the corresponding action taken, and yty_t is a failure label assigned a value of −1-1 when the car collides with an obstacle and 11 otherwise. The world model uses the DreamerV3 recurrent state-space model (RSSM) architecture with a CNN image encoder, while l and V are parameterized as feed-forward MLPs. The dynamics shift causes the car to travel farther per time-step than the world model anticipates. Figure 2(a–b) shows that the baselines collide because their safety estimates fail to respond sufficiently to this shift. In contrast, our latent prediction error rises upon entering the slippery region (Figure 2(d)), expanding the uncertainty set and lowering the pessimistic safety value. This triggers an early intervention that steers the car away from the obstacle. Fig. 3: Heatmap of JRD score across different positions and angles for Dubins car. The black and gray circles trace out the boundaries for the slippery zone and obstacles respectively. The score remains low across the state space despite the presence of dynamics mismatch. The bottom row shows the quantities used by different baselines for safety adaptation. AcoFi adapts the safety threshold online by computing a conformal score stAcoFi=max(Q(zt,at)−Rt,0)s_t^AcoFi= (Q(z_t,a_t)-R_t,0) with Q being a state-action critic relating to the value function as V(zt)=maxaQ(zt,a)V(z_t)= _aQ(z_t,a), and Rt=(1−γ)l(zt)+γmin(l(zt),V(zt+1))R_t=(1-γ)l(z_t)+γ (l(z_t),V(z_t+1)) being a bootstrapped value function. A quantile qtq_t is then computed to adapt the safety threshold as δt+1←qt+1+γδ+(1−γ)l(zt+1) _t+1← q_t+1+γδ+(1-γ)l(z_t+1). The bottom-left plot of Figure 2 displays stAcoFis_t^AcoFi for the rollout. Notably, the score fails to reflect the dynamics shift introduced when the car enters the slippery zone at time t=10t=10. Instead, the score remains zero for several time steps, revealing that Q(zt,at)≤RtQ(z_t,a_t)≤ R_t, i.e the bootstrapped value estimate believes the trajectory is becoming safer despite the increased velocity, resulting in no adaptation of the safety threshold (Figure 2(b)) and the subsequent collision. This occurs because a critic trained on rollouts from an incorrect world model can itself become incorrect. The score stAcoFis_t^AcoFi therefore measures self-consistency between Q and its bootstrapped target RtR_t, which need not reveal accumulated error in the world model. UNISafe uses the JRD score of a probabilistic ensemble to detect OOD latent states, and trains the margin l(zt)l(z_t) to flag states above a calibrated threshold as unsafe. This score is shown in Figure 2(e). We observe that despite the abrupt increase in velocity at t=10t=10, the score does not undergo any significant change, remaining below the calibrated threshold until the moment of collision. In this case, the ensemble is confidently wrong, producing similar predictions despite the shift in the underlying system dynamics. This is further demonstrated in Figure 3, where we show the JRD score heatmap slices across different heading angles. The score remains low and below the OOD threshold despite the dynamics shift induced by the slippery region. Importantly, there is no noticeable difference between the scores inside and outside the slippery region, indicating that OOD uncertainty does not always capture inaccuracies in the world model. Fig. 4: Our method imagines the worst-case outcome given the estimated uncertainty set, allowing it to adapt to errors in the world model. Our approach instead leverages the online latent prediction error ete_t, shown in Figure 2(d). Notably, this error rises sharply when the car enters the slippery zone, as the world model predictions no longer align with the observations. Algorithm 1 expands the uncertainty set in response to this error, allowing the value function to quickly adapt (Figure 2(b)). This triggers an early safety intervention which steers the car away from the obstacle. This example highlights two benefits of adapting the value function with latent prediction error. First, the error is an observation-grounded signal of world-model inaccuracy, giving the filter a direct reactive mechanism at deployment. Second, UNISafe and AcoFi instead rely on auxiliary quantities trained separately from the deployed world model. If those are themselves miscalibrated, an untrustworthy model still looks trustworthy. V-B Case Study 2: Franka Panda Arm We next study our method in a simulated manipulation task using a Franka Panda arm. The task is to pick up a globe and place it on a shelf without toppling a vase along the way. We implement this environment in ManiSkill [21], and train a world model and the safety margin function using a dataset of 3000 scripted trajectories, consisting of 1286 failed and 1714 safe examples. Each trajectory is initialized by randomizing the starting positions of the globe and vase. The observations consist of wrist-mounted and top-down camera views of the robot, along with proprioceptive inputs. Actions are 6-dimensional delta increments for position and orientation concatenated with a discrete gripper command. The nominal policy is a diffusion policy [22] trained using a dataset of 600 simulated trajectories. Fig. 5: Franka Panda simulation results with a nominal diffusion policy. Our approach drastically reduces failures without suffering from overconservatism. Qualitative Evaluation. We begin by illustrating our method qualitatively through Figure 4, contrasted against LatentSafe, which does not have a mechanism for online adaptation. Due to natural errors in the world model, the nominal value function V does not accurately capture the boundary of the unsafe set. As a result, LatentSafe triggers the safety intervention only after the vase has tilted beyond the point of no return. By continuously adapting the value function using online prediction error, our method remains robust to world model inaccuracies. This enables preemptive engagement of the safety fallback, allowing the arm to place the globe on the shelf without toppling the vase. By training a reconstructor alongside the world model, we can visualize both the nominal latent prediction and the worst-case latent state found by solving (8) at the time of filter intervention. These reconstructions are shown at the bottom of Figure 4. In this example, Algorithm 1 imagines that, under the current uncertainty estimate, the proposed action could lead to the vase toppling. This explains the discrepancy between the nominal and adapted value functions in the bottom-right of Figure 4: our method assigns the pessimistic V¯ V to be much lower than the nominal V. Quantitative Evaluation. Next, we quantitatively evaluate the performance of our method. We use qmax=2.0q_max=2.0, computed with αcal=0.05 _cal=0.05 and αtraj=0.1 _traj=0.1 on a calibration set of size 100. For ACI, we choose α=0.2α=0.2 with step size λ=0.05λ=0.05. A threshold δ=0.15δ=0.15 is used for safety filtering across all baselines, tuned to balance safety and performance. We run episodes under two types of world-model error: • Naturally occurring prediction errors: These are errors that arise from imperfect world-model learning under the nominal deployment distribution. This setting evaluates if the safety filter can remain effective when the world model is imperfect but not explicitly perturbed. • Deployment-time world-model shifts: We induce additional mismatch between the world model and deployment observations through two probes. First, we introduce a temporal abstraction mismatch by changing the effective transition scale of the world model from its nominal Δt=0.1 t=0.1s to Δt=0.2 t=0.2s or Δt=0.05 t=0.05s. Second, we introduce observation staleness by conditioning the world model on delayed observations ot−Lo_t-L in (1), with L∈0,2,4L∈\0,2,4\. These probes alter the consistency between the model’s latent predictions and the observations received during deployment, producing systematic latent prediction errors. The results are given in Figure 5. Notably, our method achieves a comparable success rate while suffering the fewest failures in the in-distribution setting with natural world model errors. After introducing deployment-time world-model shifts, our approach still retains the highest success rate, while being significantly safer than LatentSafe and AcoFi. The rise in incompletion rates under our method reflects a division of labor, where the filter must intervene early enough for πsafe _safe to retract the system to a recoverable state, but whether the episode then completes depends on πnom _nom resuming post-intervention, which it was not trained to do. The outcomes are also not equally costly, since failures are irreversible while incompletions leave the system intact and allow for continued goal-directed behavior. Fig. 6: (Top) Our method adapts the value function using estimated model errors to avoid failure while retaining task performance. (Bottom) UNISafe enables safety at the cost of frequently driving the system into OOD states where it is unable to recover. Fig. 7: Hardware evaluations on Franka Panda across three different testing conditions. Top row shows the number of failed nominal trajectories rescued (higher is better). Bottom row shows the number of successful nominal trajectories still succeeding under each method (higher is better). Fig. 8: By adapting the safety filter using the estimated world model errors, our method is able to rescue a failed trajectory by engaging the safety policy earlier than baselines. UNISafe attains the lowest failure rate during deployment time WM shifts, but does so at the cost of substantial conservatism resulting in frequent incompletions. This is visualized in Figure 6 (bottom). UNISafe frequently classifies the latent state as OOD (yellow line repeatedly crossing the black threshold). We hypothesize that this occurs because the manipulation setting is substantially more complex than the Dubins car example, with higher-dimensional observations, a larger action space, and more diverse contact-rich transitions, making ensemble disagreement more likely during deployment. Since UNISafe purposefully treats OOD states as unsafe, this results in a highly conservative value-function landscape that repeatedly triggers the safety policy, ultimately causing the robot to stall. In contrast, our method does not actively avoid OOD regions, but rather leverages observed evidence to estimate how wrong the world model is in those areas. This facilitates recovery by enabling the safety policy to retract the arm away from the vase, allowing the value function to rise and hand control back to the nominal policy (top row, Figure 6). VI Hardware Experiments We evaluate our method on hardware using a Franka Panda platform operating at 15 Hz, where the task is to use a dustpan to throw a tomato into a trash can. Failure is defined as the tomato landing outside the trash can. We evaluate our method by deploying it on both successful and failed trajectory replays. Each type of replay is divided into three categories, 1) In-distribution: the trajectory comes from the same distribution as the WM training dataset, 2) Forceful Throw: the tomato is suddenly jerked out of the dustpan, 3) Observation Staleness: The world model is conditioned on delayed observations ot−Lo_t-L in the same manner as in section V-B. We use L=10L=10 for all observation staleness experiments. A collection of 20 replays is collected for both success and failure in each individual category. We train a Dreamer world model and safety margin function using a dataset of 350 successful and 150 failed trajectories, collected using teleoperation. The observations consist of side and top-down views of the robot along with a wrist view and proprioception. We use δ=0.2δ=0.2 across all experiments and qmax=4.5q_max=4.5, derived from a calibration dataset of 30 trajectories with αcal=αtraj=0.25 _cal= _traj=0.25. ACI parameters are chosen as α=0.2α=0.2 and λ=0.05λ=0.05. Our method with PGD averages 39.5939.59 ms per step on an NVIDIA GeForce RTX 5090 (LatentSafe: 18.0918.09 ms, AcoFi: 17.1717.17 ms, UNISafe: 17.4217.42 ms), well within the 15 Hz control budget. Metrics. Evaluation is based on the number of failed trajectories rescued, and the number of successful outcomes preserved. Results from 480 runs are shown in Figure 7. First, as expected, when the deployment setting remains within the world model’s training distribution, all methods lead to a significant number of rescues among trajectories that would otherwise fail. When the tomato is forcefully thrown, LatentSafe and ACoFi are largely unable to prevent failures, reflecting their limited ability to adapt to the sudden dynamics mismatch. In contrast, our method maintains a high rescue rate by adapting the value function pessimistically in response to the observed world-model errors, enabling preemptive engagement of the safety fallback before the tomato is thrown out of the pan. UNISafe likewise maintains a high number of rescues, as the forceful throw is detected as an OOD action that should be safeguarded against. Under observation staleness, our method continues to enable a significant number of rescues, while performance degrades substantially across all baselines. The strongest baseline achieves 50%50\% fewer rescues than our method. We hypothesize that this is because stale observations induce incorrect latent predictions while still remaining within the world model’s learned distribution, causing UNISafe to under-respond to the resulting model error. Moreover, because these experiments use open-loop replays, UNISafe does not experience the persistent covariate shift or robot stalling observed in the closed-loop simulation study in Section V-B. This behavior is also evident in the qualitative comparison in Figure 8, which shows one rollout from the observation-staleness setting evaluated under different methods. The UNISafe JRD score remains below its OOD threshold and does not exhibit a strong response to the prediction mismatch, indicating agreement among the probabilistic ensemble despite elevated model error. Similarly, the ACoFi nonconformity score remains low in magnitude throughout the episode and does not respond noticeably to model inaccuracies. In contrast, our method directly tracks the discrepancy between the predicted and inferred latent states. Under stale observations, this error remains elevated throughout the episode and rises sharply near the end as the dustpan tilts downward, producing action inputs that are increasingly inconsistent with the stale observations (the mid-episode spike is caused by the chaotic bouncing dynamics of the tomato as the arm grasps the dustpan). This error-aware adaptation allows our method to lower the pessimistic value estimate early enough for the safety policy to prevent failure. VII Conclusions and Limitations We present a method for online adaptation of world-model based safety filters from prediction error estimates. By continuously estimating latent uncertainty sets during operation, we compute the worst-case imagination under the current estimated uncertainty. Through both qualitative and quantitative demonstrations across simulation and hardware, we show that our error-aware safety filter results in improved safety over existing methods. Our work is not without limitations. While providing additional robustness against world model errors, our method is reactive and does not provide formal safety guarantees. Solving (8) incurs additional computational cost compared with existing methods. Moreover, Theorem 1 assumes exchangeability between deployment and calibration trajectories, which can be violated under safety filtering if recovery actions are absent from the calibration set. These constitute important avenues for future research. References [1] M. J. Kim, Y. Gao, T. Lin, Y. Lin, Y. Ge, G. Lam, P. Liang, S. Song, M. Liu, C. Finn, and J. Gu (2026) Cosmos policy: fine-tuning video models for visuomotor control and planning. In International Conference on Learning Representations (ICLR), Cited by: §I. [2] P. Intelligence, K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, et al. (2025) π0.5 _0.5: a Vision-Language-Action model with Open-World generalization. arXiv preprint arXiv:2504.16054. Cited by: §I. [3] G. R. Team, S. Abeyruwan, J. Ainslie, J. Alayrac, M. G. Arenas, T. Armstrong, A. Balakrishna, R. Baruch, M. Bauza, M. Blokzijl, et al. (2025) Gemini robotics: bringing ai into the physical world. arXiv preprint arXiv:2503.20020. Cited by: §I. [4] A. D. Ames, S. Coogan, M. Egerstedt, G. Notomista, K. Sreenath, and P. Tabuada (2019) Control barrier functions: theory and applications. In 2019 18th European control conference (ECC), p. 3420–3431. Cited by: §I. [5] S. Bansal, M. Chen, S. Herbert, and C. J. Tomlin (2017) Hamilton-jacobi reachability: a brief overview and recent advances. In 2017 IEEE 56th annual conference on decision and control (CDC), p. 2242–2253. Cited by: §I. [6] K. Nakamura, L. Peters, and A. Bajcsy (2025) Generalizing safety beyond collision-avoidance via latent-space reachability analysis. arXiv preprint arXiv:2502.00935. Cited by: §I, §I, §I, §I-A, §V. [7] D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson (2019) Learning latent dynamics for planning from pixels. In International conference on machine learning, p. 2555–2565. Cited by: §I. [8] P. Abbeel, M. Quigley, and A. Y. Ng (2006) Using inaccurate models in reinforcement learning. In Proceedings of the 23rd international conference on Machine learning, p. 1–8. Cited by: §I. [9] N. Hansen and X. Wang (2026) Hallucination in world models is predictable and preventable. arXiv preprint arXiv:2606.27326. Cited by: §I. [10] C. Li, A. Krause, and M. Hutter (2025) Robotic world model: a neural network simulator for robust policy optimization in robotics. arXiv preprint arXiv:2501.10100. Cited by: §I. [11] T. An, J. Jia, G. Li, J. Li, C. Zhou, P. Liu, B. Lyu, J. Bai, X. Guo, G. Li, et al. (2026) Feedback world model enables precise guidance of diffusion policy. arXiv preprint arXiv:2605.15705. Cited by: §I. [12] I. R. Ward, M. Ho, H. Liu, A. Feldman, J. Vincent, L. Kruse, S. Cheong, D. Eddy, M. J. Kochenderfer, and M. Schwager (2026) Foundational world models accurately detect bimanual manipulator failures. arXiv preprint arXiv:2603.06987. Cited by: §I, §I. [13] J. Seo, K. Nakamura, and A. Bajcsy (2025) Uncertainty-aware latent safety filters for avoiding out-of-distribution failures. arXiv preprint arXiv:2505.00779. Cited by: §I, §IV-B, §V. [14] J. Berger, B. Frauenknecht, S. Trimpe, and B. Leibe (2026) Biased dreams: limitations to epistemic uncertainty quantification in latent space models. arXiv preprint arXiv:2604.25416. Cited by: §I. [15] S. Huriot, I. Tabbara, and H. Sibai (2026) Safe control using learned safety filters and adaptive conformal inference. arXiv preprint arXiv:2604.18482. Cited by: §I, §V. [16] I. Gibbs and E. Candes (2021) Adaptive conformal inference under distribution shift. Advances in Neural Information Processing Systems 34, p. 1660–1672. Cited by: §I, §I-B, §IV-B, Proposition 1. [17] K. Nakamura, A. L. Bishop, S. Man, A. M. Johnson, Z. Manchester, and A. Bajcsy (2025) How to train your latent control barrier function: smooth safety filtering under hard-to-model constraints. arXiv preprint arXiv:2511.18606. Cited by: §I, §V. [18] J. F. Fisac, N. F. Lugovoy, V. Rubies-Royo, S. Ghosh, and C. J. Tomlin (2019) Bridging hamilton-jacobi safety analysis and reinforcement learning. In 2019 International Conference on Robotics and Automation (ICRA), p. 8550–8556. Cited by: §I-A. [19] A. Z. Ren, A. Dixit, A. Bodrova, S. Singh, S. Tu, N. Brown, P. Xu, L. Takayama, F. Xia, J. Varley, et al. (2023) Robots that ask for help: uncertainty alignment for large language model planners. arXiv preprint arXiv:2307.01928. Cited by: §IV-B. [20] D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap (2023) Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104. Cited by: §V. [21] S. Tao, F. Xiang, A. Shukla, Y. Qin, X. Hinrichsen, X. Yuan, C. Bao, X. Lin, Y. Liu, T. Chan, Y. Gao, X. Li, T. Mu, N. Xiao, A. Gurha, V. N. Rajesh, Y. W. Choi, Y. Chen, Z. Huang, R. Calandra, R. Chen, S. Luo, and H. Su (2025) ManiSkill3: gpu parallelized robotics simulation and rendering for generalizable embodied ai. Robotics: Science and Systems. Cited by: §V-B. [22] C. Chi, Z. Xu, S. Feng, E. Cousineau, Y. Du, B. Burchfiel, R. Tedrake, and S. Song (2025) Diffusion policy: visuomotor policy learning via action diffusion. The International Journal of Robotics Research 44 (10-11), p. 1684–1704. Cited by: §V-B. VIII Appendix Proof of Theorem 1. We begin by noting the equivalence of the events Qταtraj≤qmax=1T∑t=1T[et>qmax]≤αtraj. \Q_τ _traj≤ q_max\=\ 1T _t=1^T1[e_t>q_max]≤ _traj\. (13) From Proposition 1, we have that with probability 11 1T∑t=1T[et>qt]−α≤rT. 1T _t=1^T1[e_t>q_t]-α≤ r_T. (14) Combining (13) and (14) gives ℙ(1T∑t=1T[et>qt]+[et>qmax]≤α+αtraj+rT)≥ ( 1T _t=1^T1[e_t>q_t]+1[e_t>q_max]≤α+ _traj+r_T )≥ ≥1−αcal. ≥ 1- _cal. (15) Lastly, the inequality [et>q¯t]≤[et>qt]+[et>qmax]1[e_t> q_t] 1[e_t>q_t]+1[e_t>q_max] allows us to upper-bound the left-hand side of (15) with ℙ(1T∑t=1T[et>q¯t]≤α+αtraj+rT), ( 1T _t=1^T1[e_t> q_t]≤α+ _traj+r_T ), (16) giving us (12). ■