Paper deep dive
MEND: Label-Free Detection, Localisation, and Correction of Latent Hallucination in World Models
Ali J Alrasheed, Aryan Yazdan Parast, Basim Azam, James Bailey, Naveed Akhtar
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:World Models are appearing as the next major frontier in computer vision. However, their robustness is currently largely unexplored. We identify the phenomenon of hallucination in latent World Models: given a state and an action, the predicted next latent can decode to a scene that never occurs. Because the prediction is statistically ordinary and is fed back autoregressively by the model, the error is both silent and compounding. We study whether such latent hallucination can be detected, localised, and corrected at inference time, on a frozen self-supervised world model in the absence of ground-truth error labels. We introduce Masked Empirical-Bayes Neural Denoising (MEND), a single conditional score network trained by denoising score matching on real transitions, whose score field serves three roles: its magnitude detects hallucination, its per-token field localises it to specific image patches, and it defines an inference-time correction direction. On two navigation environments MEND detects hallucination with an AUROC of up to 0.80 without using actions, exceeding a single-Gaussian density baseline while also localising the error (per-token AUPRC up to 0.87) and correcting it, all from one score field. Our correction reliably reduces single-step latent error and improves predictions. We identify that a part of the error is tangent to the data manifold, hence, we focus on detection and localisation while highlighting promises of the correction.
Tags
Links
- Source: https://arxiv.org/abs/2609.39182v1
- Canonical: https://arxiv.org/abs/2609.39182v1
Trouble viewing inline? Open PDF directly →
Full Text
43,006 characters extracted from source content.
Expand or collapse full text
MEND: Label-Free Detection, Localisation, and Correction of Latent Hallucination in World Models Ali Alrasheed Affiliation: University of Melbourne Melbourne, Australia ali.alrasheed@student.unimelb.edu.au Aryan Yazdan Parast Affiliation: University of Melbourne Melbourne, Australia aryan.yazdanparast@student.unimelb.edu.au Basim Azam Affiliation: University of Melbourne Melbourne, Australia basim.azam@unimelb.edu.au James Bailey Affiliation: Monash University Melbourne, Australia james.a.bailey@monash.edu Naveed Akhtar Affiliation: University of Melbourne Melbourne, Australia naveed.akhtar1@unimelb.edu.au Abstract World Models are appearing as the next major frontier in computer vision. However, their robustness is currently largely unexplored. We identify the phenomenon of hallucination in latent World Models: given a state and an action, the predicted next latent can decode to a scene that never occurs. Because the prediction is statistically ordinary and is fed back autoregressively by the model, the error is both silent and compounding. We study whether such latent hallucination can be detected, localised, and corrected at inference time, on a frozen self-supervised world model in the absence of ground-truth error labels. We introduce Masked Empirical-Bayes Neural Denoising (MEND), a single conditional score network trained by denoising score matching on real transitions, whose score field serves three roles: its magnitude detects hallucination, its per-token field localises it to specific image patches, and it defines an inference-time correction direction. On two navigation environments MEND detects hallucination with an AUROC of up to 0.80 without using actions, exceeding a single-Gaussian density baseline while also localising the error (per-token AUPRC up to 0.87) and correcting it, all from one score field. Our correction reliably reduces single-step latent error and improves predictions. We identify that a part of the error is tangent to the data manifold, hence, we focus on detection and localisation while highlighting promises of the correction. Index Terms: world models, hallucination detection, score matching, self-supervised learning I Introduction World models are an emerging and promising approach to decision making: by imagining, or rolling out, future latent states, they let an agent evaluate the consequences of an action before taking it, and have been shown to solve complex downstream tasks in this way [1, 2, 3]. Modern world models imagine future not in pixels but in the latent space of a frozen visual predictor, which is efficient but obscure. In that space, a model can produce a next state that is perfectly plausible as a vector of numbers yet decodes to a scene that never happens. For instance, an object jumps across the room, a wall dissolves, or a detail is invented where the future was uncertain. We term this phenomenon latent hallucination. Because the rollout is autoregressive, a hallucinated state is fed back in as context for the next prediction. Hence, this hallucination compounds and its effect on rollout accuracy grows with the imagination horizon (Fig. 1). Because the distortion lives in the latent, it is also silent: nothing on the surface of the prediction reveals that it is wrong, and it surfaces only once the true future arrives, by which point a planner may already have acted on it. Fig. 1: Latent hallucination grows with imagination. Free-running a frozen world model on PointMaze: the top row is ground truth, the bottom row is the model’s imagined rollout decoded to images. Early on they agree, but by step 17 the imagined agent has drifted to a part of the maze it never visits. The prediction is still a plausible latent; nothing flags the error until the true future is known. Evaluation of world models has focused mainly on downstream task performance, such as planning success or reward, and much less on whether the imagined rollout itself deviates from reality [1, 2, 3]. Hallucination has been studied at length for large language models, where a substantial literature now addresses how to detect and reduce fabricated output [4]. Similar phenomena inside world models have received little attention, and how to detect and correct them, is largely unexplored. This paper takes a step in that direction. Once a reliable detector exists, the most direct way to reduce hallucination is to collect the cases where the model fails and retrain on them. This potential data-driven route is likely to be effective but requires new data and additional training, which is generally not desirable. We instead study an inference-time correction that operates on a frozen model, and introduce a general way of handling hallucination in latent world models as a three-stage loop: detect whether a prediction is hallucinated, localise which parts of it are wrong, and correct those parts (Fig. 2). We instantiate this loop label-free with a single conditional score field and call the resulting method Masked Empirical-Bayes Neural Denoising (MEND). MEND is a single conditional score network whose score field detects hallucination, its per-token field localises it to image patches, and it defines an inference-time correction direction. We present the correction as a natural extension that complements the stronger detection and localisation findings. Our contributions are: • A definition and two-type taxonomy of latent hallucination in latent world models, showing the error is spatially sparse: a small subset of tokens carries a disproportionate share of the total error. • A general detect, localise, and correct loop, instantiated label-free by MEND, a single conditional score field that serves as detector, localiser, and correcter. • Detection and localisation on two environments: up to 0.80 detection AUROC without actions, exceeding a single-Gaussian density baseline while also localising the error to specific patches (per-token AUPRC up to 0.87), and robust to the labelling threshold. • A promising inference-time correction loop, with analysis of what it can and cannot recover. I Related work Latent world models. A growing family of world models predicts future states not in pixels but in a learned latent space, and performs planning within that space. DINO-WM [1] predicts in the frozen patch-token space of DINOv2 [5]; joint-embedding predictive architectures [2], end-to-end latent world models such as LeWM [6], and categorical models such as Dreamer [3] follow the same paradigm. This line of work extends earlier world models that plan from pixels or recurrent latent states [7, 8] to recent diffusion-based world models [9]. Despite their architectural differences, these models rely on iterative latent prediction for planning, so prediction errors accumulate over rollout and can eventually lead to hallucination. Hallucination and uncertainty. Hallucination is well documented for large language models and large vision language models, where a substantial body of work addresses its detection and mitigation [4, 10, 11]. Diffusion-based image and video generation models [12, 13] exhibit related failure modes, producing plausible but incorrect content. Within video generation world models, C3 [14] trains a supervised probe on internal features to estimate dense per-patch confidence for video generation. In contrast, MEND targets latent world models for the first time, requiring no accuracy labels, and computing confidence directly from the predicted latent itself. The resulting confidence signal is differentiable with respect to the prediction, allowing it to be used both for hallucination detection and to guide latent correction. Detecting an implausible prediction is an instance of out-of-distribution detection, addressed with density and score-based criteria [15, 16, 17], the density-based of which motivates our Gaussian reference [15]. Our correction step follows score- and diffusion-based restoration, where the learned score of a data prior guides the underlying inverse problems [18, 19, 20]. Fig. 2: MEND’s single-iteration pipeline. The world model predicts z^t+1 z_t+1 from (zt,at)(z_t,a_t) and the detector scores it, (z^t+1)D( z_t+1). Two mitigation paths follow. Option 1 (inference-time correction) uses the localiser to select the suspicious tokens (m) and iteratively move them toward the valid manifold while freezing the confident tokens; Option 2 (data-driven repair) logs the flagged predictions for offline fine-tuning of the world model. Bottom: over K iterations the correction restores the hallucinated tokens (red) to trusted ones (grey). I Definition of Latent hallucination Setting. In latent world models, a frozen encoder maps an observation to a latent state ztz_t, and a frozen predictor maps a short history of states and actions to a predicted next latent z^t+1=g(zt,at) z_t+1=g(z_t,a_t). The latent is spatial: it is a grid of tokens, each describing a patch of the scene, so an error can be attributed to a region of the image. A decoder is available to render a latent as an image, and we use it only for inspection. Definition. Given the current latent state ztz_t and action ata_t, the environment evolves according to its true transition p(⋅∣zt,at)p(· z_t,a_t): the actual dynamics of the environment, which an ideal world model reproduces. These dynamics permit a set of valid next states, those consistent with the environment’s physical rules, for instance that the agent cannot pass through a wall. Let R(zt)R(z_t) denote the states reachable under any action, and R(zt,at)⊆R(zt)R(z_t,a_t) R(z_t) the subset reachable under the action taken. A prediction z^t+1 z_t+1 is hallucinated if it lies outside this set, i.e. it has negligible probability under the true transition, regardless of its prediction error. In general these dynamics may be stochastic, so a single action can admit several valid successors; the environments we study are deterministic, so R(zt,at)R(z_t,a_t) reduces to a single point (Section V). The same definition extends to imagined rollouts. Let Rk(zt,at:t+k−1)R_k(z_t,a_t:t+k-1) denote the states reachable after k steps under the action sequence at:t+k−1a_t:t+k-1. A rollout (z^t+1,…,z^t+H)( z_t+1,…, z_t+H) is hallucination free if z^t+k∈Rk z_t+k∈ R_k for every k. By Markov property, its probability factorises as p(z^t+1:t+H∣zt,at:t+H−1)=∏kp(z^t+k+1∣z^t+k,at+k),p( z_t+1:t+H z_t,a_t:t+H-1)= _kp( z_t+k+1 z_t+k,a_t+k), (1) where each factor is a one step transition of the form above. A single hallucinated step therefore reduces the probability of the entire rollout and, because each prediction conditions on previous predictions, the error accumulates with depth (Fig. 1). Hallucinations fall into two categories: (S) Unreachable state (z^t+1∉R(zt) z_t+1∉ R(z_t)). A state that cannot occur under any action, such as an object appearing from nowhere, a wall dissolving, or the agent passing through one. It depends only on (zt,z^t+1)(z_t, z_t+1). (A) Incorrect action outcome (z^t+1∈R(zt)∖R(zt,at) z_t+1∈ R(z_t) R(z_t,a_t)). The predicted state is reachable, but not under the action that was taken. It depends on the action ata_t. The error is sparse. Localisation and masked correction are effective only if prediction errors are concentrated rather than uniformly distributed. In our experiments (Section V), this holds for both Wall and PointMaze environments. On single-step predictions, the most suspicious fifth of tokens accounts for roughly a third of the total error. During deep free-running rollouts (Fig. 1), the same fifth accounts for over 80 percent. Hallucination is therefore a local phenomenon that becomes increasingly concentrated as it grows, making per-token localisation and masked correction well suited to the problem. IV Method We first describe the generic detect-localise-correct loop we identify in the context of latent world models to develop a clear foundation of our method. We introduce the proposed label-free instantiation of this framework later. IV-A Detect, localise, and correct loop Once hallucinations can be detected in a rollout, they can, in principle, be mitigated in two ways. The first is to collect the detected failures and fine-tune the world model on them, since they identify precisely where the learned dynamics are unreliable. This strategy directly improves the model, but it requires a data collection pipeline together with the computational budget for retraining or fine-tuning. In many practical settings, however, retraining is either unavailable or prohibitively expensive. In such cases, it is desirable to reduce hallucination directly at inference time while leaving the world model unchanged (Fig. 2). We therefore formulate a general detect–localise–correct loop for inference-time hallucination mitigation. The loop begins with a detector that assigns a validity score to a predicted state. The detector itself is intentionally left unspecified: the score may be produced by a single model or by combining multiple models or heuristic indicators. If the same detector is also to support localisation and correction, the score must be differentiable with respect to the predicted state. Differentiability allows a single score function to serve three complementary roles. First, its value detects hallucination by measuring the validity of the prediction. Second, its gradient with respect to the predicted state naturally localises the error: the tokens whose perturbation would most strongly change the score are precisely those the detector considers most suspicious, eliminating the need for a separate attribution mechanism. Third, the same gradient provides a correction direction. By taking a small optimisation step that reduces the detector score on only the suspicious tokens, the prediction is iteratively refined while leaving the remainder of the latent state unchanged. One iteration therefore evaluates the detector, identifies the most suspicious tokens from the score gradient, applies a masked correction step to those tokens only, and re-evaluates the updated prediction. This process repeats until the prediction is judged sufficiently valid or a predefined iteration budget is reached (Fig. 2). The mask controls the extent of the edit, trading improved error coverage against unnecessary changes to already correct regions. The detector, localiser, and corrector need not be separate components. Any differentiable validity score can, in principle, provide all three functions simultaneously. In the remainder of this section we derive such a score and use it as a unified detector, localiser, and corrector. IV-B Masked Empirical-Bayes Neural Denoising We instantiate the loop with a single differentiable object, the score of a learned conditional density over next states, and name the resulting method MEND (Masked Empirical-Bayes Neural Denoising). The next subsections derive the score (the objective it approximates and the network that estimates it), show how one evaluation of it fills all three roles of the loop, and give the correction procedure in full. IV-B1 Mathematical basis A world model defines a conditional distribution over the next latent state given the current latent state and action. By Bayes’ rule, this conditional probability can be expressed as - following notational conventions from above: p(z^t+1∣zt,at)=p(z^t+1∣zt)p(at∣zt,z^t+1)⏞learnablep(at∣zt)⏟independent of z^t+1.p( z_t+1 z_t,a_t)= \; p( z_t+1 z_t)\;p(a_t z_t, z_t+1)^learnable\; p(a_t z_t)_independent of z_t+1. (2) Since the denominator depends only on (zt,at)(z_t,a_t), it is constant with respect to the predicted state z^t+1 z_t+1: it cancels when comparing candidate predictions at a fixed (zt,at)(z_t,a_t) and disappears under differentiation. The resulting score is therefore proportional to the product of two conditional densities, p(z^t+1∣zt,at)∝p(z^t+1∣zt)⏟D1: reachable from ztp(at∣zt,z^t+1)⏟D2: under this action.p( z_t+1 z_t,a_t)\; \; p( z_t+1 z_t)_D1: reachable from z_t\; p(a_t z_t, z_t+1)_D2: under this action. (3) This decomposition forms the basis of our detector family. Each factor describes a property of the environment dynamics rather than of the world model itself, allowing it to be estimated by a small auxiliary network trained on the same logged transitions as the world model, without requiring any hallucination or accuracy labels. The resulting detectors are therefore self-supervised and world-model-agnostic. They assign high scores to predictions that are likely under the true environment dynamics, regardless of which world model produced them, while low scores indicate likely hallucinations. The two factors capture complementary evidence. D1, the reachability term p(z^t+1∣zt)p( z_t+1 z_t), measures whether the predicted latent state is reachable from the current state under any valid transition, making it sensitive to type S in §I. D2, the action-consistency term p(at∣zt,z^t+1)p(a_t z_t, z_t+1), measures whether the observed action explains the transition and is therefore sensitive to type A in §I. Under a Gaussian action-noise model, its negative log-likelihood reduces, up to an additive constant, to the residual of an inverse dynamics model, −logp(at∣zt,z^t+1)∝∥at−Inv(zt,z^t+1)∥2.- p(a_t z_t, z_t+1) a_t-Inv(z_t, z_t+1) ^2. The Bayes decomposition naturally gives rise to these two complementary detector terms. Their relative effectiveness depends on the inductive biases of the learned world model and its training objective. For example, a model that primarily captures state-to-state transitions may place less emphasis on actions, making the reachability term D1D1 more informative than the action-consistency term D2D2. In our deterministic simulators, D1D1 alone provides strong hallucination detection and localisation, while D2D2 offers little additional benefit (Section VI). We therefore instantiate the remainder of the framework using D1D1 only, while still deriving and evaluating D2D2 as a principled ablation implied by the exact Bayes factorisation. IV-B2 Conditional score network We instantiate the framework using D1, the reachability density p(z^t+1∣zt)p( z_t+1 z_t), and interpret hallucination as removable perturbation of a valid next state. Specifically, we assume that a hallucinated prediction can be written as z^t+1≈zt+1+e z_t+1≈ z_t+1+e, where zt+1z_t+1 is a valid successor of ztz_t and e denotes the hallucination error. Rather than estimating the density itself, we estimate its score, the gradient of the log-density, which points in the direction of greatest increase in validity. We therefore learn a conditional score network sθ(z~∣zt,σ)∈ℝN×Ds_θ( z z_t,σ) ^N× D, conditioned on the current latent state ztz_t and on a noise scale σ>0σ>0, the standard deviation of the Gaussian smoothing introduced below, supplied to the network through a noise-scale embedding. The network is trained using denoising score matching [21, 22]. Given real transitions (zt,zt+1)(z_t,z_t+1), we corrupt the true next state with Gaussian noise of scale σ, z~=zt+1+σε z=z_t+1+σ , and train the network to predict the score of the corrupted sample by minimizing ℒ(θ)=(zt,zt+1),σ,εσ2∥sθ(zt+1+σε∣zt,σ)+ε/σ∥2,L(θ)=E_(z_t,z_t+1),\,σ,\, \;σ^2\, s_θ(z_t+1+σ z_t,σ)+ /σ ^2, (4) where ε∼(0,I) (0,I), the regression target is −ε/σ- /σ, and the scale σ is sampled from a geometric noise schedule. The training noise ε is distinct from the hallucination error e: ε is an isotropic Gaussian corruption used only during training to learn the score, whereas e is the real, possibly non-Gaussian prediction error we detect and correct at inference. As shown by [21], the population minimiser of Eq. (4) is the conditional score ∇logqσ(z~∣zt)∇ q_σ( z z_t) of the σ-smoothed density of valid next states. Denoising score matching therefore learns the score field directly, without requiring the normalising constant of p(z^t+1∣zt)p( z_t+1 z_t), making estimation of D1 tractable. This formulation has two properties that make it well suited for hallucination detection and correction. First, the score network is trained exclusively on real transitions and never observes predictions from the world model. It therefore learns the dynamics of the environment rather than the behaviour of a particular predictor, preserving the self-supervised and world-model-agnostic interpretation of Eq. (3). Second, although training corrupts states with Gaussian noise, that noise enters only as a smoothing kernel and not as an assumption about the hallucination error: by Eq. (4) the network learns the exact conditional score ∇logqσ(z~∣zt)∇ q_σ( z z_t) of the smoothed density of valid next states, and this identity holds at every input, including a predicted state that was never formed by adding Gaussian noise. Evaluated at a prediction z^t+1 z_t+1, the score measures how, and how strongly, that prediction must move to become more valid. IV-B3 One field, three roles We now make the three roles concrete. A single evaluation of sθs_θ at the prediction z^t+1 z_t+1 yields both the detection score and the localisation map; correction reuses the same network as a short iterative refinement at a smaller noise scale (Section V). The three roles therefore come from one score field rather than three separate components (Fig. 2). • Detect. The scalar =∥sθ(z^t+1∣zt)∥2D= s_θ( z_t+1 z_t) ^2 measures how far z^t+1 z_t+1 is from the set of reachable states. Because a regression-trained predictor makes even its correct outputs mildly atypical, we standardise it against the detector’s statistics on known-correct predictions, ~=(−μacc)/σacc D=(D- _acc)/ _acc, and use ~ D throughout (Section V). • Localise. The per-token field ∥sθ(z^t+1∣zt)n∥ s_θ( z_t+1 z_t)_n is a direct estimate of how far each token must move, so no attribution or backward pass is needed. • Correct. By Tweedie’s formula [23] the posterior mean of the clean state is z^t+1+σ2sθ z_t+1+σ^2s_θ, so the score is also the correction direction. The geometry behind the three roles is explicit: for a point at distance r from the reachable set, as σ→0σ→ 0 the score points from z^t+1 z_t+1 toward its nearest valid neighbour with magnitude r/σ2r/σ^2 [24]. Detection reads the length of this vector, localisation reads its per-token parts, and correction follows it. Correction is applied as a short iterative loop - see Algorithm 1. MEND keeps the original prediction z^orig z_orig as an anchor and, on the first iteration, fixes a support of the top-p%p\% most-displaced tokens. Each step then moves this support along the score, a Tweedie displacement toward the valid-state manifold, while a proximal term pulls the edit back toward z^orig z_orig. The most suspicious tokens therefore move the most and the confident ones stay near their original values. The loop returns the iterate of the lowest standardised score ~ D. Two choices matter in practice: the correction scale σ must be small, since larger scales are destructive, and the correction must be reapplied at every step of a rollout, since a single correction may not persist. Algorithm 1 MEND: detect, localise, correct at inference 0: state ztz_t, action ata_t, predictor g, score net sθs_θ, correction scale σ, mask size p, step η, anchor ρ, budget K, calibration stats (μacc,σacc)( _acc, _acc), thresholds τ,δτ,δ 1: z^orig←g(zt,at) z_orig← g(z_t,a_t); z←z^origz← z_orig; best←(z,∞)best←(z,∞) 2: for k=1,…,Kk=1,…,K do 3: G←sθ(z∣zt,σ)G← s_θ(z z_t,σ) one forward pass 4: d←σ2Gd←σ^2G displacement to nearest valid state 5: ~←(∥G∥2−μacc)/σacc D←( G ^2- _acc)/ _acc detect 6: mraw←rank(|d|)m_raw ( d ) localise 7: if k=1k=1 then 8: S←top p% of mrawS←\top p\% of m_raw\ support frozen 9: end if 10: m←mraw⊙[S]m← m_raw 1[S] 11: z←z+η(m⊙d−ρ(z−z^orig))z← z+η (m d-ρ\,(z- z_orig) ) correct 12: if ~<best.~ D<best. D then best←(z,~)best←(z, D) 13: break if ~<τ D<τ or ∥Δz∥<δ z <δ 14: end for 15: return best iterate V Experimental setup Backbone and environments. We use the public frozen DINO-WM checkpoints [1]: a DINOv2 ViT-S/14 encoder [5] with N=196N=196 tokens and D=384D=384, and a small predictor. We report on two popular navigation environments, Wall (a Markov predictor with a single history frame) and PointMaze (a three-frame history). Calibration and labelling. We hold out trajectories, run the predictor, and split predictions into correct and incorrect at the median per-token error, giving a balanced set. All detectors and thresholds are defined on this organic-error set. We do not use synthetic corruptions as a headline metric, because denoising score matching is trained to detect exactly such perturbations. The support definition of Section I is not directly observable, so we label with this proxy: in the deterministic environments studied here R(zt,at)R(z_t,a_t) is essentially a single point, so the per-token error between a prediction and the realised next latent closely tracks whether the prediction lies in the reachable set, and a large error means it has left that set. Under stochastic dynamics this proxy would over-count valid alternative futures, and a reachability-based label would be required. Metrics. Detection is measured by AUROC of correct versus incorrect predictions. Localisation is measured by per-token AUPRC against the ground-truth error map, with a random baseline of 0.50. Correction is measured by the reduction in latent error and the fraction of predictions improved. Baselines. For detection, we compare against a single diagonal-Gaussian density fit to valid states, the unimodal special case of feature-space Gaussian-density detectors [15], which serves as a weak density floor. We further compare an inverse-model detector (the action factor of Eq. (2)), a directly conditioned score model that reads the action, and a cross-attention variant that injects the action. Full architectures and training of these variants are given in the supplementary material. Implementation. The conditional score network is a four-layer token Transformer (six attention heads, feed-forward width 15361536, AdaLN noise conditioning; 10.310.3M parameters, ≈0.53×≈\!0.53× a single world-model predictor forward), trained by denoising score matching on roughly 500500 held-out trajectories. Detection uses a single large noise scale σ=0.39σ=0.39, which probes global structure; correction uses a small scale σ=0.05σ=0.05 with step η=0.3η=0.3, anchor weight ρ=0.1ρ=0.1, and K=10K=10 Tweedie steps (σ>0.08σ>0.08 is destructive). VI Detection and localisation results VI-A Across environments Table I evaluates MEND on both environments. The central observation is that hallucination detection and localisation are related but distinct capabilities that need not improve together. Detection is a global task that asks whether an entire predicted latent state is plausible, whereas localisation is a spatial task that identifies which latent tokens are responsible for the error. Detection is strongest in environments with sharp, nearly deterministic dynamics, such as Wall, while localisation remains reliable whenever hallucination is concentrated to a subset of tokens, which is true in both environments. In PointMaze, the agent occupies only a small region of the scene and its future position is inherently uncertain, reducing the separation between valid and hallucinated predictions and lowering detection AUROC. Even so, the learned score field continues to localise hallucinations accurately (per-token AUPRC 0.870.87), because the error remains concentrated in a small subset of tokens. For detection, MEND substantially outperforms the single-Gaussian density floor on Wall and improves on it in PointMaze, demonstrating that hallucination detection requires modelling the multimodal structure of the transition distribution rather than a single-mode density. It achieves this while remaining fully differentiable, so the same object that scores a prediction also localises and corrects it. The distinguishing advantage of MEND is that a single differentiable object simultaneously provides all three capabilities required by our framework: a global hallucination score for detection, a dense per-token localisation map, and a correction direction for inference-time refinement. A pure detection baseline such as the single-Gaussian density offers only the first of these and therefore cannot support the unified detect–localise–correct framework. This explains the absence of baseline from the last two columns. TABLE I: MEND (Ours) against a diagonal-Gaussian density reference [15]. Detection is AUROC; the last two columns are detection and localisation per-token AUPRC (random 0.500.50). Detection AUROC Det. Loc. Environment Ours Gauss. AUPRC AUPRC Wall 0.801 0.648 0.802 0.712 PointMaze 0.691 0.631 0.687 0.874 Robustness. Detection is stable as the labelling threshold is swept from lenient to strict, and it saturates after a few hundred training trajectories, so the ceiling reflects the environment and frozen backbone rather than a shortage of data (see the supplementary material). VI-B Localisation examples Fig. 3 shows localisation on the hardest cases: hallucinations produced by free-running the world model deep into a rollout, where the imagined agent has wandered to a part of the maze it never visits. Each row shows the true and predicted frames, the ground-truth per-token error, and the detector score field, which concentrates on exactly the tokens that are wrong. The striking part is the stability with depth. As the rollout runs forward the total error more than doubles, yet the per-token AUPRC barely moves from the first step out to the deepest step the episodes allow. Localisation is therefore robust to depth: even when the whole imagined state has drifted off the manifold, the same signal that flags a hallucination keeps saying where it is, which is what the correction stage acts on. Equivalent deep-rollout examples on Wall are shown in the supplementary material. Fig. 3: Localising deep-rollout hallucinations on PointMaze (step 17 of a free-running rollout, where the imagined agent has drifted far from reality). Columns: decoded ground truth, decoded prediction, ground-truth per-token error, and the D1 detector score field (per-token AUPRC annotated). Even when the whole state is far off the manifold the detector still points at the tokens that are wrong. Cyan circles mark the nine most-erroneous tokens. VII Ablation study and discussion Throughout our experiments, the detector is D1, the reachability factor p(z^t+1∣zt)p( z_t+1 z_t) from the Bayes decomposition in Eq. (3), which depends only on the current and predicted latent states. We also evaluate the complementary action-consistency factor, D2 =p(at∣zt,z^t+1)=p(a_t z_t, z_t+1), together with a variant that conditions the score network directly on the action. As shown in Table I, neither improves hallucination detection. D2 performs at approximately chance level, while action conditioning matches, but does not exceed, the performance of D1. This behaviour follows naturally from the deterministic setting. D2 is implemented through an inverse dynamics model, which measures how accurately the observed action can be reconstructed from the predicted transition. Because the world model is itself conditioned on the action, even hallucinated predictions generally remain consistent with that action. Consequently, the inverse dynamics residual is largely independent of whether a prediction is valid or hallucinated, making it a weak detection signal despite accurate action reconstruction. This observation is specific to deterministic environments rather than a limitation of the Bayes decomposition itself. D2 is the only factor that distinguishes between multiple reachable future states that differ only in the action that produced them. It is therefore expected to become informative in stochastic or multimodal environments, where reachability alone is insufficient to identify the correct transition. Conditioning the score network directly on the action likewise provides no benefit. Instead of learning the geometry of reachable latent states, the network can partially rely on the action as a shortcut. Classifier-free guidance confirms this interpretation: removing the action at training time recovers the performance of the action-free model but does not improve upon it. TABLE I: Ablation of detector variants on Wall. Variant architectures are detailed in the supplementary material. Detector (Wall) AUROC Conditional score net (no action) 0.801 Directly conditioned score, +action (AdaLN) 0.799 Cross-attention, +action (15M) 0.782 Inverse model — D2 action factor 0.490 Directly conditioned score, −-action 0.507 VII-A Correction results Correction reuses the same score field as a Tweedie step. A single such step reduces latent error on both navigation environments and improves almost every prediction (Table I). Because the Tweedie update is the per-token score field itself, each token moves in proportion to its localisation score: the most suspicious tokens move most and confident ones barely move, so an explicit mask is redundant and we edit the full latent. In an imagined rollout the correction must be reapplied at every step: a single first-step correction washes out immediately, whereas per-step correction sustains a roughly constant reduction with depth (Table IV; depth-resolved curves in the supplementary material). TABLE I: Single-step correction. Δ error is the relative change in latent error, so a negative value indicates improvement; arrows mark the better direction. Environment Δ error ↓ % improved ↑ Wall −6.4%-6.4\% 98.5 PointMaze −3.0%-3.0\% 99.5 What correction cannot do. The error decomposes into a component normal to the reachable set and a component tangent to it. Only the normal part is recoverable: a state displaced along the reachable set is a valid alternative future, indistinguishable from the truth for any corrector. The tangent part forms an aleatoric floor that bounds the achievable gain, which is why the reductions are consistent but modest and why mild near-manifold errors are corrected most. We therefore present correction as a feasibility result rather than a solved problem. TABLE IV: Per-step correction on Wall: mean per-token latent error versus rollout depth (lower is better). Parentheses show the reduction relative to No corr. Depth No corr. MEND (D1) ↓ 1 1.770 1.626 (−8.2%-8.2\%) 5 2.370 2.237 (−5.6%-5.6\%) 9 2.830 2.732 (−3.5%-3.5\%) VIII Conclusion We showed that hallucination is not only a language-model phenomenon but also a measurable property of latent world models, and we studied it on a frozen self-supervised backbone without error labels. Our method, MEND, derives a single conditional score field that does three jobs from one object: it detects hallucination, localises it to the right image patches, and supplies an inference-time correction direction. Detection and localisation are solid and reproducible across two environments and robust to the labelling threshold and the data budget. Correction reliably reduces latent error but is bounded by the part of the error that lies along the data manifold, where no corrector is likely to help. So we present it as a preliminary but promising investigation. The same detection and localisation signal also points to a natural extension: using it to target data collection and repair the world model directly. In our method, detection and localisation are solid and label-free, and inference-time correction reliably reduces latent error. However, two directions remain open. First, the approach assumes that hallucinations lie off-manifold, which holds for the navigation environments studied here but may not hold when a predictor’s errors stay close to the manifold. Hence, extending the study to stochastic dynamics is a natural next step. Second, correction is established at the representation level, and whether these gains transfer to a downstream controller is left for the future work. Acknowledgements Ali Alrasheed is supported by a scholarship from Aramco. Naveed Akhtar is a recipient of the Australian Research Council Discovery Early Career Researcher Award (project # DE230101058), funded by the Australian Government. This research was also supported by The University of Melbourne’s Research Computing Services and the Petascale Campus Initiative. References [1] G. Zhou, H. Pan, Y. LeCun, and L. Pinto (2024) DINO-WM: world models on pre-trained visual features enable zero-shot planning. arXiv preprint arXiv:2411.04983. Cited by: §I, §I, §I, §V. [2] Y. LeCun (2022) A path towards autonomous machine intelligence. Note: OpenReviewVersion 0.9.2 Cited by: §I, §I, §I. [3] D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap (2023) Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104. Cited by: §I, §I, §I. [4] Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y. Xu, E. Ishii, Y. J. Bang, A. Madotto, and P. Fung (2023) Survey of hallucination in natural language generation. ACM Computing Surveys 55 (12), p. 1–38. Cited by: §I, §I. [5] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, et al. (2023) DINOv2: learning robust visual features without supervision. arXiv preprint arXiv:2304.07193. Cited by: §I, §V. [6] L. Maes, Q. L. Lidec, D. Scieur, Y. LeCun, and R. Balestriero (2026) LeWorldModel: stable end-to-end joint-embedding predictive architecture from pixels. arXiv preprint arXiv:2603.19312. Cited by: §I. [7] D. Ha and J. Schmidhuber (2018) Recurrent world models facilitate policy evolution. Advances in Neural Information Processing Systems 31. Cited by: §I. [8] D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson (2019) Learning latent dynamics for planning from pixels. In International Conference on Machine Learning, p. 2555–2565. Cited by: §I. [9] E. Alonso, A. Jelley, V. Micheli, A. Kanervisto, A. Storkey, T. Pearce, and F. Fleuret (2024) Diffusion for world modeling: visual details matter in Atari. Advances in Neural Information Processing Systems 37, p. 58757–58791. Cited by: §I. [10] J. Maynez, S. Narayan, B. Bohnet, and R. McDonald (2020) On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, p. 1906–1919. Cited by: §I. [11] A. Yazdan Parast, P. Hosseini, H. Asadollahzadeh, A. Soltani Moakhar, B. Azam, S. Feizi, and N. Akhtar (2026) GHOST: hallucination-inducing image generation for multimodal LLMs. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: §I. [12] J. Ho, A. Jain, and P. Abbeel (2020) Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems 33, p. 6840–6851. Cited by: §I. [13] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer (2022) High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 10684–10695. Cited by: §I. [14] Z. Mei, T. Yin, M. Baker, O. Shorinwa, and A. Majumdar (2025) World models that know when they don’t know: controllable video generation with calibrated uncertainty. arXiv preprint arXiv:2512.05927. Cited by: §I. [15] K. Lee, K. Lee, H. Lee, and J. Shin (2018) A simple unified framework for detecting out-of-distribution samples and adversarial attacks. Advances in Neural Information Processing Systems 31. Cited by: §I, §V, TABLE I. [16] E. Nalisnick, A. Matsukawa, Y. W. Teh, D. Gorur, and B. Lakshminarayanan (2018) Do deep generative models know what they don’t know?. arXiv preprint arXiv:1810.09136. Cited by: §I. [17] A. Mahmood, J. Oliva, and M. Styner (2020) Multiscale score matching for out-of-distribution detection. arXiv preprint arXiv:2010.13132. Cited by: §I. [18] Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole (2020) Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456. Cited by: §I. [19] B. Kawar, M. Elad, S. Ermon, and J. Song (2022) Denoising diffusion restoration models. Advances in Neural Information Processing Systems 35, p. 23593–23606. Cited by: §I. [20] H. Chung, J. Kim, M. T. McCann, M. L. Klasky, and J. C. Ye (2022) Diffusion posterior sampling for general noisy inverse problems. arXiv preprint arXiv:2209.14687. Cited by: §I. [21] P. Vincent (2011) A connection between score matching and denoising autoencoders. Neural Computation 23 (7), p. 1661–1674. Cited by: §IV-B2, §IV-B2. [22] Y. Song and S. Ermon (2019) Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems 32. Cited by: §IV-B2. [23] B. Efron (2011) Tweedie’s formula and selection bias. Journal of the American Statistical Association 106 (496), p. 1602–1614. Cited by: 3rd item. [24] J. Pidstrigach (2022) Score-based generative models detect manifolds. Advances in Neural Information Processing Systems 35, p. 35852–35865. Cited by: §IV-B3.