Paper deep dive
Revealing Epistemic Uncertainty in MLLMs via Causal-Invariant Masking
Haoyang Luo, Linwei Tao, Jie Gui, Xinghao Chen, Chang Xu, Jianyuan Guo, Minjing Dong
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Multimodal Large Language Models (MLLMs) suffer from hallucinations, creating a critical need for Uncertainty Quantification (UQ) to ensure reliable deployment. However, existing approaches struggle to detect uncertainty caused by superficial associations, especially when the query-relevant signal is weak. We mainly attribute this issue to their bias toward aleatoric uncertainty arising from data ambiguity, overlooking epistemic uncertainty stemming from model limitations. To further decompose uncertainty types for a comprehensive UQ, we propose Causal-Invariant Masking (CIM), which measures the semantic shift between the original predictions and those conditioned on a causally-focused view. Based on this framework, we introduce Semantic Divergence as our core metric for UQ and provide theoretical evidence that it converges to the variance of model's sensitivity to non-causal correlations, establishing its ability to capture MLLM's limitation. To accelerate UQ in MLLMs, we further propose Expected Embedding Drift (EED), a fast geometric proxy metric that estimates semantic shift directly within the hyperspherical embedding space. Experiments show that our method achieves state-of-the-art performance on various benchmarks, while the proposed EED accelerates by nearly 50% with comparable performance.
Tags
Links
- Source: https://arxiv.org/abs/2610.02887v1
- Canonical: https://arxiv.org/abs/2610.02887v1
Trouble viewing inline? Open PDF directly →
Full Text
73,293 characters extracted from source content.
Expand or collapse full text
Revealing Epistemic Uncertainty in MLLMs via Causal-Invariant Masking Haoyang Luo Affiliation: Computer Science Affiliation: City University of Hong Kong Linwei Tao Affiliation: Machine Learning Research Affiliation: Apple Affiliation: Jie Gui Affiliation: Cyber Science and Engineering Affiliation: Southeast University Affiliation: Xinghao Chen Affiliation: Huawei Noah’s Ark Lab Affiliation: Chang Xu Affiliation: Computer Science Affiliation: University of Sydney Affiliation: Jianyuan Guo Affiliation: Computer Science Affiliation: City University of Hong Kong Affiliation: Minjing Dong Affiliation: Computer Science Affiliation: City University of Hong Kong Affiliation: Abstract Multimodal Large Language Models (MLLMs) suffer from hallucinations, creating a critical need for Uncertainty Quantification (UQ) to ensure reliable deployment. However, existing approaches struggle to detect uncertainty caused by superficial associations, especially when the query-relevant signal is weak. We mainly attribute this issue to their bias toward aleatoric uncertainty arising from data ambiguity, overlooking epistemic uncertainty stemming from model limitations. To further decompose uncertainty types for a comprehensive UQ, we propose Causal-Invariant Masking (CIM), which measures the semantic shift between the original predictions and those conditioned on a causally-focused view. Based on this framework, we introduce Semantic Divergence as our core metric for UQ and provide theoretical evidence that it converges to the variance of model’s sensitivity to non-causal correlations, establishing its ability to capture MLLM’s limitation. To accelerate UQ in MLLMs, we further propose Expected Embedding Drift (EED), a fast geometric proxy metric that estimates semantic shift directly within the hyperspherical embedding space. Experiments show that our method achieves state-of-the-art performance on various benchmarks, while the proposed EED accelerates by nearly 50% with comparable performance. 1 Introduction Large Language Models (LLMs) [51, 32] and Multimodal Large Language Models (MLLMs) [47, 4] have significantly altered the landscape of artificial intelligence. By connecting visual perception with textual semantics, MLLMs are able to demonstrate consistent success in tasks ranging from visual question answering to complex reasoning [28, 37, 50]. However, their reliable deployment in real-world applications requires solving the severe problem of hallucination [5, 17], where MLLMs confidently generate factually incorrect but grammatically plausible content. Especially for VLMs, non-existent objects, attributes, and events may unreasonably occur in the responses when misleading factors, either in vision or language, dominate the query. It suggests that MLLMs do not internally realize their visual capability boundary and highlights a critical need for robust safety measures targeting generative trustworthiness. With this aim, numerous Uncertainty Quantification (UQ) methods have been developed. By estimating the confidence level of a model, systems can abstain from answering or flag potential hallucinations when encountering high uncertainty. Therefore, precise UQ becomes a proxy for hallucination detection. Traditional UQ approaches typically measure the entropy of the predictive distribution, often aggregated at the token level [46, 42]. Recent advancements [21, 20, 34] have shifted towards measuring semantic uncertainty using sampling-based methods, such as Semantic Entropy (SE) [13], which aggregates probabilities over clusters of similar generations to calculate the response variety. However, these approaches exhibit limited discrimination under hallucinative visual content, especially when the query-irrelevant elements generate spurious hints. For instance, background objects with prominent colors can dominate the response to queries on the foreground object, making consistent mistakes in the sampled output. Thus, a more comprehensive UQ is necessary for MLLMs. In fact, both epistemic and aleatoric uncertainties play important roles in UQ, where epistemic uncertainty reveals insufficiency in the model’s perception or comprehension and aleatoric uncertainty reflects the poor data quality or incomplete information [19]. Quantifying both of them is crucial in UQ [41, 1], however, existing approaches fail to do so because they typically assume that all forms of uncertainty can be equally estimated. In this paper, we show that they are biased towards quantifying mostly aleatoric uncertainty in MLLMs, leaving epistemic uncertainty underestimated. We introduce Causal-Invariant Masking (CIM), a framework that disentangles intrinsic epistemic instability from data ambiguity by probing semantic consistency under causally invariant transforms. Our core insight is that robust reasoning must remain invariant when task-irrelevant features are removed. By constructing a Region-of-Interest (RoI) view that retains only causal features, we measure the shift in the model’s belief distribution. We quantify this using Semantic Divergence, defined as the shift from the semantic distributions of the original to the focused inputs. Supported by theoretical derivation, this metric targets the model stability against irrelevant distractions to effectively isolate epistemic uncertainty, thereby alleviating the biased quantification in previous works. To address the computational costs of discrete semantic clustering, we then propose a fast proxy called Expected Embedding Drift (EED). Leveraging the hyperspherical geometry of the embedding space, EED estimates semantic shift directly from token embeddings, eliminating costly discrete operations. Experiments demonstrate that our method significantly outperforms baselines, achieving state-of-the-art hallucination detection of 76.8% AUROC on the challenging AdVQA and superior results on POPE. Furthermore, EED reduces computational overhead by 50% without compromising performance, facilitating applications such as hallucination mitigation and adaptive-resolution inference. 2 Related Work 2.1 Uncertainty Estimation in LLMs UQ in LLMs is pivotal for reliability and hallucination detection, generally categorized into intrinsic, verbalized, and consistency-based methods. Intrinsic metrics like perplexity [41] measure token-level confidence, while verbalized methods [45, 43] prompt models to explicitly express uncertainty, often yielding better calibration than raw probabilities. Consistency-based works are currently dominant approaches. Semantic Entropy [13] and Semantic Dispersion [27] detect hallucinations by aggregating multiple sampled responses based on meaning rather than exact phrasing. Recent works further exploit internal representations as uncertainty indicators by utilizing activation covariance and propose EigenScore [10] or geometric convex hull analysis [9] to quantify dispersion in high-dimensional embedding spaces. However, these methods target general UQ for LLMs and can be challenged in vision-language tasks. 2.2 Uncertainty Estimation in MLLMs Extending UQ to MLLMs involves challenges from combined visual and textual modalities, often focusing on hallucination detection in vision-language tasks. Similar to those for LLMs, consistency-based sampling techniques are explored, e.g., [23] develops UMPIRE, a training-free UQ framework that estimates consistency in multiple sampled responses by a coherence-adjusted semantic volume. Perturbation-based methods introduce variations in inputs to probe uncertainty. VL-Uncertainty [49] and Uncertainty-o [48] measure hallucination in LVLMs by computing entropy over clustered responses, but using semantic-equivalently perturbed visual and textual queries, supporting tasks of hallucination mitigation and uncertainty-aware reasoning. However, these approaches typically rely on global perturbations under the full context, without explicitly isolating the model’s reliance on critical and irrelevant features. CIM is also a perturbation-style test in its implementation, but its perturbation is query-conditioned and RoI-preserving. The goal is to compare the original belief with a view that keeps the answer evidence while suppressing irrelevant context, rather than purely testing variance under general information loss. 3 Method 3.1 Preliminary Figure 1: Left: A case of blurred and background-mixup image pair. Middle: Sensitivity of Semantic Entropy to visual perturbations in MLLMs. Similar levels of error to the original image are binned into the same point. Right: Comparison of uncertainty increase under blurring and background mixup perturbations. While Semantic Entropy increases slightly across both, it fails to distinguish hallucination. In contrast, our proposed Semantic Divergence remains robust to visual blur but exhibits a distinct spike under hallucinative perturbations. The values are normalized by the standard deviation of each metric. We consider an MLLM parameterized by θ over visual space V and vocabulary W. Given an image v and query q, it generates a token sequence y=(y1,…,yT)y=(y_1,…,y_T) autoregressively via Pθ(y|v,q)=∏t=1TPθ(yt|y<t,v,q)P_θ(y|v,q)= _t=1^TP_θ(y_t|y_<t,v,q). For simplicity, we henceforth omit q and θ, denoting the distribution as P(y|v)P(y|v). Token-level entropy, i.e., ℋ(y)=−∑P(y)logP(y)H(y)=-Σ P(y) P(y), or perplexity values, i.e., PPL(y)=exp(−1T∑t=1TlogP(yt|y<t,v))PPL(y)= (- 1T _t=1^T P(y_t|y_<t,v) ), often misestimate uncertainty because semantically equivalent outputs are treated as distinct. Semantic Entropy (SE) [13] samples K sequences, maps them to a semantic space S with a clustering operation :∗→G:W^* , and aggregates meaning-level probabilities: ℋsem(v)=−∑s∈P(s|v)logP(s|v)H_sem(v)=- _s P(s|v) P(s|v), where ∗W^* is the set of W-strings and P(s|v)=∑y:(y)=sP(y|v)P(s|v)= _y:G(y)=sP(y|v) is the aggregated probability. While SE handles linguistic variance, it still fails to distinguish epistemic from aleatoric uncertainty. 3.2 Epistemic Uncertainty Isolation via Causal Masking Reliable MLLM deployment requires separating aleatoric uncertainty, which reflects data ambiguity, from epistemic uncertainty, which reflects model limitation. Existing metrics often blend the two and therefore skew toward aleatoric noise. As illustrated in Fig. 1 middle, Semantic Entropy rises quickly under blurring for aleatoric uncertainty, but is much less sensitive to background mixup for epistemic uncertainty, suggesting that it treats causal and non-causal features alike. Robust reasoning instead should satisfy causal invariance: predictions should stay stable when task-irrelevant context changes, whereas uncertain responses should not. Figure 2: The detailed framework of CIM. Left: We isolate epistemic uncertainty via Semantic Divergence, which measures the KL divergence between the semantic distribution of the original input and a causally-filtered view v′v , revealing the model’s reliance on query-irrelevant information. Right: Expected Embedding Drift geometrically estimates semantic shift via cosine distance in the hyperspherical embedding space, eliminating the computational overhead of discrete clustering. Methodology. We propose CIM to isolate the epistemic component through causal invariance. Given a query q, we decompose the visual input v into causal features vcv_c and non-causal features vsv_s, where vcv_c is query-relevant, while vsv_s may carry spurious hints of correlations. This split is defined with respect to an oracle predictor satisfying P(s|vc,vs)=P(s|vc)P(s|v_c,v_s)=P(s|v_c), rather than assuming that the evaluated MLLM already knows it. Accordingly, reliable reasoning should remain stable when non-causal variables are removed, and any large shift after filtering vsv_s exposes reliance on spurious correlations. We therefore introduce a vcv_c-preserving masking operator T(⋅):→T(·):V , mainly implemented as query-conditioned Region-of-Interest (RoI) cropping. In practice, we detect key objects with GroundingDINO [29], filter overly large boxes, and crop the union bounding box so inter-object relations and local context are preserved. This acts as an empirical proxy for the oracle vcv_c filter: it preserves query-relevant evidence while removing as much vsv_s as possible, yielding a focused view v′=T(v)v =T(v). Because only a vcv_c-preserving transform is required, the detector choice is not essential, as validated in Appendix C.6. To quantify the shift caused by causal masking, we define Semantic Divergence (SD) as the Kullback-Leibler divergence between the semantic probability distributions of the intervened input P(s|v′,q)P(s|v ,q) and the original model P(s|v,q)P(s|v,q). In practice, we sample K′K sequences from the model, group them into clusters sij=1M\s_i\_j=1^M via entailment (⋅)G(·), and aggregate their likelihoods similarly to SE. After grouping, the Semantic Divergence is thus calculated as: sem(v,q) _sem(v,q) =DKL(P(s|v′)∥P(s|v)) =D_KL(P(s|v )\|P(s|v)) =∑s∈P(s|v′)logP(s|v′)P(s|v). = _s P(s|v ) P(s|v )P(s|v). (1) This KL divergence measures the information gain from removing non-causal features. If the model is robust, or the data is ambiguous while the model is honest, the distribution over meanings remains stable, yielding low semU_sem. In contrast, hallucinations that rely on background context cause a semantic shift and high semU_sem. Unlike global perturbations that flatten both distributions, this measure targets epistemic failures. Our semU_sem significantly outperforms traditional SE in recognizing epistemic uncertainty. As illustrated in Fig. 1 right, we plot the shift of uncertainty when imposing different ambiguity types. Our semU_sem significantly outperforms traditional SE in recognizing epistemic uncertainty. 3.3 Theoretical Analysis We justify why semU_sem isolates epistemic uncertainty by framing the MLLM’s semantic output as an Energy-Based Model (EBM) [8, 24, 12]. This is compatible with autoregressive decoding: prior work interprets pre-softmax logits as token energies [33], connects left-to-right language models and EBMs through a function-space equivalence [6], and uses energy functions for constrained text decoding [39]. Because MLLM decoding remains autoregressive after conditioning on images, the same energy view applies. Under standard causal assumptions, the divergence in Eq. (1) converges to the variance of sensitivity to spurious features. Standard causal theory [36, 35] models the input v as causal variables vcv_c and non-causal variables vsv_s. An oracle predictor would satisfy P(s|vc,vs)=P(s|vc)P(s|v_c,v_s)=P(s|v_c), but real models often fail to be invariant. We therefore introduce a unified assumption on the masking operation and the energy landscape: Assumption 3.1 (Energy Decomposition). The MLLM’s semantic output over S is modeled as a Boltzmann distribution Pθ(s|v)=1Z(v)e−Eθ(s,v)P_θ(s|v)= 1Z(v)e^-E_θ(s,v), where the total energy decomposes additively into a causal term Ec(s,vc)E_c(s,v_c) and a non-causal term Es(s,vs)E_s(s,v_s). We assume that the causal energy EcE_c remains invariant under the causal-preserving T(⋅)T(·), i.e., Ec(s,vc′)=Ec(s,vc)E_c(s,v _c)=E_c(s,v_c), while the non-causal energy shifts by a quantity Δs(s)≜Es(s,vs)−Es(s,vs′) _s(s) E_s(s,v_s)-E_s(s,v _s). This invariant/variant split follows the broader causal-invariance view that causal mechanisms should remain stable across interventions or environments [38, 2, 40]. Related decomposition work also separates shared invariant factors from domain-specific variants [7]. Rather than requiring pixel-level separation, Assumption 3.1 only asks that T(⋅)T(·) preserve query-relevant evidence while suppressing non-causal context. The empirical validations in Appendix C.7, Table 3, and Appendix C.4 show that causal-energy shifts are far below the model’s logit margins, while detected RoIs rarely fail and closely match ground truth RoIs. We first show that the aleatoric component ϕc _c cancels in the likelihood ratio. With the empirical evidence above, the cancellation derived is a high-fidelity approximation in practice. Lemma 3.2 (Aleatoric Cancellation). Given Assumption 3.1, the log-likelihood ratio between the perturbed and original distributions depends solely on the non-causal energy shift Δs(s) _s(s) and the partition function ratio, i.e., logQ(s)P(s)=Δs(s)+logZ(v)Z(v′). Q(s)P(s)= _s(s)+ Z(v)Z(v ). (2) The proof is provided in Appendix A.1. Lemma 3.2 shows that Ec(s,vc)E_c(s,v_c) cancels in the likelihood ratio, so the metric is invariant to ambiguity in vcv_c. Next, we establish the exact form of the divergence. Theorem 3.3 (Epistemic Jensen Gap). The Semantic Divergence sem=DKL(Q∥P)U_sem=D_KL(Q\|P) is equivalent to the Jensen Gap of the exponential function acting on Δs(s) _s(s) with respect to distribution Q. Furthermore, if we assume the shift magnitude is bounded such that sup|Δs(s)|≤ϵ | _s(s)|≤ε for a small ϵ>0ε>0. The Semantic Divergence converges to half of the energy shift Δs(s) _s(s) with a third-order error bound: sem=12Vars∼Q[Δs(s)]+(ϵ3).U_sem= 12Var_s Q[ _s(s)]+O(ε^3). (3) We provide the detailed proof in Appendix A.2. This result shows that semU_sem quantifies epistemic instability caused by irrelevant features under the stated approximation. When the vcv_c-focused operation T(⋅)T(·) removes most of vsv_s, we have Δs(s)=Es(s,vs) _s(s)=E_s(s,v_s), so semU_sem measures variance on query-irrelevant signals. Appendix C.7 reports empirical Δs _s magnitudes, showing its bounded level in practice. 3.4 Fast Quantification via Expected Embedding Drift Despite its theoretical validity, Semantic Divergence semU_sem is expensive to compute. Equation (1) requires estimating P(s|v)=∑y:(y)=sP(y|v)P(s|v)= _y:G(y)=sP(y|v), which in turn needs samples from P and Q, followed by expensive LLM clustering (⋅)G(·). However, we observe that semantic meaning is already embedded in the model’s high-dimensional geometry, where tokens with similar meanings lie close together. We exploit this and propose Expected Embedding Drift (EED), a clustering-free proxy that bypasses discrete semantic grouping. Specifically, it measures the geometric divergence between the masked-input distribution Q and the original semantic distribution P by mapping token probabilities to single vectors on a hypersphere. Expected Semantic Vector. For a response y∼Qy Q at time step t conditioned on y<ty_<t and v, rather than treating the output solely as a discrete categorical distribution, we view the token distribution as a distribution over semantic vectors and compute the expected embedding vector, P,t μ_P,t as P,t=w∼P(⋅|y<t,v)[w]=∑w∈P(w|y<t)⋅w, μ_P,t=E_w P(·|y_<t,v)[e_w]= _w P(w|y_<t)·e_w, (4) where w∈ℝde_w ^d is the L2L_2-normalized embedding vector for token w in W. This expected embedding is the probability-weighted semantic center in hyperspherical space, filtering low-probability noise and pointing to the aggregate intent. Drift Calculation. To measure epistemic inconsistency, for each generated response sequence y=(y1,…,yT)∼Qy=(y_1,…,y_T) Q, we measure the cosine drift of yty_t from the source semantic expectation P,t μ_P,t at each autoregressive timestep t: Ddrift(yt)=1−cos(yt,P,t)=1−yt⋅P,t‖yt‖‖P,t‖,D_drift(y_t)=1- (e_y_t, μ_P,t)=1- e_y_t· μ_P,t\|e_y_t\|\| μ_P,t\|, (5) where yte_y_t is the embedding vector of token yty_t. Lower drift means the perturbed token still aligns with the original average belief, while higher drift indicates a stronger rotation after irrelevant features are removed. We average this drift over K′K stochastic sampling passes and their tokens to obtain the sequence-level EED: EED(v,q)=1K′∑k=1K′∑t=1TkDdrift(yt(K+k)),U_EED(v,q)= 1K _k=1^K _t=1^T_kD_drift(y_t^(K+k)), (6) This aggregates the step-wise deviations into a scalar score, where yt(k)e_y_t^(k) is the embedding of the token generated by the perturbed model in the k-th sample. Notably, EED scores require far fewer samples than SD to stabilize, thanks to its non-discrete semantic aggregation. It also avoids a semantic clustering model and ((K+K′)2)O((K+K )^2) entailment judgments, reducing the overhead to just (K′)O(K ). We present the framework of the proposed uncertainty metrics in Fig. 2 Theoretical Justification To justify EED, we show that under a geometric distribution assumption, minimizing EEDU_EED is equivalent to minimizing the KL divergence between the semantic distributions. Assumption 3.4 (Spherical Semantic Distribution). We model the continuous semantic distribution of the MLLM output as a von Mises-Fisher (vMF) distribution on the unit hypersphere d−1S^d-1 [30, 22]. The probability density of a semantic vector z given mean direction ∈d−1 μ ^d-1 and concentration κ>0κ>0 is: p(|,κ)=Cd(κ)exp(κ⊤),p(z| μ,κ)=C_d(κ) (κ μ z), (7) where Cd(κ)=κd/2−1/((2π)d/2Id/2−1(κ))C_d(κ)=κ^d/2-1/((2π)^d/2I_d/2-1(κ)) is the normalization constant. This connects the discrete token space to the continuous geometry, where μ corresponds to the normalized expected embedding P,t μ_P,t. Lemma 3.5 (KL Divergence of vMF Distributions). Let Q∼vMF(Q,κ)Q ( μ_Q,κ) and P∼vMF(P,κ)P ( μ_P,κ) be two von Mises-Fisher distributions with identical concentration κ. The Kullback-Leibler divergence between them is strictly proportional to the cosine distance between their mean directions: DKL(Q∥P)=κAd(κ)(1−Q⊤P),D_KL(Q\|P)=κ A_d(κ)(1- μ_Q μ_P), (8) where Ad(κ)=‖∼Q[]‖=Id/2(κ)Id/2−1(κ)A_d(κ)=\|E_z Q[z]\|= I_d/2(κ)I_d/2-1(κ) is the ratio of Bessel functions. The derivation is detailed in Appendix A.3. Theorem 3.6 (Geometric Proxy Equivalence). The Expected Embedding Drift EEDU_EED is an unbiased estimator of the KL divergence between the perturbed semantic distribution Q and the original belief P, up to a scaling factor and additive constant. The full proof is available in Appendix A.4, which connects the proxy to our core objective. The point-wise drift in Eq. (5) is aggregated in Eq. (6) to estimate the expected cosine distance. As established in Eq. (8) and Thm. 3.6, this distance effectively estimates the Semantic Divergence in Eq. (1), giving an efficient proxy for epistemic uncertainty. 4 Experiment We adopt InternVL2-8B [11] and Qwen2.5-VL-7B-Instruct [3] as our evaluated models. We test the performance of semU_sem and EEDU_EED on various benchmarks, including common visual question answering (VQAv2 [15], OKVQA [31], AdVQA [25]) and hallucination-targeted benchmarks (POPE [26], MME [14]). For simplicity, we adopt the lightweight GroundingDINO detector [29] as our RoI generator. In practice, specialized or query-conditioned detectors can also instantiate T(⋅)T(·), as verified in Appendix C.6. We fix the test subset and set an equal maximum sampling time Ktotal=K_total=K (baseline methods) or Ktotal=K+K′K_total=K+K (our method) to 3232 for fair comparison. In practice, as ykk=1K∼P\y_k\_k=1^K P can be utilized to yield our values while producing a half-sampled SE (compared to KtotalK_total samples), we calculate a mixed uncertainty metric by interpolating our values with this SE to balance estimation of distinct uncertainty types. Since hallucination detection benefits from total uncertainty, this interpolation captures both aleatoric and epistemic uncertainty. Appendix B provides the detailed experimental setup. Table 1: Uncertainty estimation performance (%) on visual question answering benchmarks. AUC is short for AUROC in the table. Benchmark VQAv2 OKVQA AdVQA MLLM InternVL2-8B Qwen2.5-VL-7B InternVL2-8B Qwen2.5-VL-7B InternVL2-8B Qwen2.5-VL-7B Metric AUC AURAC AUC AURAC AUC AURAC AUC AURAC AUC AURAC AUC AURAC Perplexity 70.3 80.7 77.6 81.4 65.4 76.8 75.1 82.3 65.7 60.3 66.4 61.2 Self Evaluation [45] 60.9 75.1 62.4 71.9 66.4 77.0 63.1 75.5 60.4 57.1 55.3 49.7 Verbalized Confidence [43] 54.3 68.3 57.2 69.2 55.3 67.8 58.2 72.1 50.4 48.6 57.1 51.9 Semantic Entropy [13] 78.7 82.5 76.5 79.5 80.6 84.5 78.6 83.8 75.5 66.6 73.6 63.1 EigenScore [10] 72.6 80.7 75.8 79.9 72.2 78.5 76.8 83.0 67.1 60.2 64.6 59.1 UMPIRE [23] 74.3 81.7 72.9 78.4 72.6 79.0 76.4 83.0 68.2 61.9 65.5 59.6 Uncertainty-o [48] 72.1 80.1 64.9 74.2 67.7 77.8 61.8 74.2 60.1 56.1 57.4 53.7 Ours (SD) 81.3 85.2 78.4 81.5 81.8 85.7 79.6 84.2 76.8 66.9 74.5 64.8 Ours (EED) 80.3 84.5 77.9 81.1 80.9 84.5 79.3 84.0 75.8 66.5 74.6 64.6 Table 2: Uncertainty estimation performance (%) on all 3 sub-tasks of the POPE benchmark. Benchmark POPE random POPE popular POPE adversarial MLLM InternVL QwenVL InternVL QwenVL InternVL QwenVL Metric AUC AURAC AUC AURAC AUC AURAC AUC AURAC AUC AURAC AUC AURAC Perplexity 68.7 91.2 68.5 91.5 68.1 90.4 67.3 90.2 68.9 90.8 65.2 89.2 Self Evaluation [45] 52.0 84.6 61.3 91.4 53.6 84.7 51.5 88.1 53.5 85.2 57.4 89.5 Verbalized Confidence [43] 50.9 84.2 60.4 88.2 51.8 82.9 60.5 87.2 50.6 82.8 60.6 87.5 Semantic Entropy [13] 70.6 89.6 62.9 88.7 68.7 88.5 64.8 88.0 72.3 90.2 63.0 86.3 EigenScore [10] 65.5 89.4 56.0 87.2 64.4 87.8 56.6 86.5 63.8 88.5 54.6 85.4 UMPIRE [23] 68.7 91.2 80.5 94.4 68.7 90.5 79.1 93.4 68.2 90.5 77.0 92.7 Uncertainty-o [48] 64.6 88.8 51.5 85.4 66.4 88.6 55.1 85.9 62.8 88.2 56.6 86.1 Ours (SD) 75.9 92.4 66.2 89.3 74.8 91.3 66.1 88.7 74.9 91.7 64.5 88.1 Ours (EED) 73.1 91.6 83.2 95.1 71.0 90.5 81.5 94.3 73.0 91.3 78.6 93.6 4.1 Results on Hallucination Detection General VQA Benchmarks. Table 1 presents the results on VQAv2, OKVQA, and AdVQA. CIM consistently outperforms existing baselines all evaluated VQA benchmarks. Notably, on the AdVQA benchmark, which is specifically designed to elicit model errors, our method achieves an AUROC of 76.8% on InternVL, surpassing Semantic Entropy (75.5%) and EigenScore (67.1%). This suggests that our method’s specific reliance on causal invariance by checking if the answer holds after removing background context is more robust to adversarial visual perturbations than methods that solely attend to the output distribution. Furthermore, our computationally efficient proxy, Ours (EED), performs competitively with the semU_sem method, often within 1% of the explicit Semantic Divergence scores, validating the theoretical connection in Lemma 3.5. We also provide case study in Appendix C.2, to further verify that SD more successfully capture epistemic uncertainty. Hallucination-targeted Benchmarks. Table 2 details performance on the POPE benchmark. This benchmark is crucial as it directly evaluates on the topic of object hallucinations. On the InternVL model, our explicit method dominates the performance across all splits, achieving ∼ 75% AUROC, significantly outperforming Semantic Entropy (∼ 68-72%). Interestingly, on QwenVL, our EED proxy outperforms the explicit method (e.g., 83.2% vs 66.2% on Random). We find that QwenVL’s decoding distribution is much sharper than InternVL’s on POPE (Appendix C.8), which makes discrete semantic clustering less stable for the explicit method. In contrast, EED operates in the continuous embedding space and is less affected by hard cluster assignments. A significant gap is observed between our CIM and other baselines on the challenging Adversarial split, confirming CIM’s identification of whether the context priors are relied on rather than visual evidence. We further evaluate on the comprehensive MME benchmark in Appendix C.1, where Fig. 7 shows that CIM remains the best-performing method. This demonstrates CIM’s generalization beyond simple object existence to complex reasoning in MME, and again verifies its consistency in uncertainty estimation. 4.2 Discussions Causal Masking. Our theoretical framework proposes that the visual input operator T(⋅)T(·) must focus on isolating causal features xcx_c to separate epistemic from aleatoric uncertainty. To validate such a design choice of xcx_c-focused RoI Cropping, we compare it against: 1) Random Crop, which crops an arbitrary region, 2) Reversed Crop, which masks the detected RoIs and keeps only the background, and 3) Ground-Truth RoIs, which are from the box-annotated GQA benchmark. The ablation results are presented in Table 3. We also compare with the commonly used gaussian noise perturbation in Appendix C.3. From the results, Random Crop yield inferior performance compared to our targeted approach. This aligns with our hypothesis that random perturbations degrade the aleatoric and epistemic signals indiscriminately. Crucially, Reversed Crop performs significantly worse than Random Crop in most tested cases. This is because feeding only the background forces the model to hallucinate or guess, but does not provide an anchor point for the correct reasoning. In contrast, our method of RoI Crop achieves significantly larger AUROC across all settings (e.g., 81.3% on VQAv2 for InternVL). By preserving necessary evidence while maximally removing spurious context, our measurement effectively grounds the MLLM’s confidence in causal signals, providing the cleanest isolation of epistemic uncertainty. While the SE component is included to retain aleatoric sensitivity for hallucination detection, it can dilute the random-vs-RoI gap on epistemic UQ. To isolate the epistemic component, the AUROCepiAUROC^epi metric reports AUROCs in a pure SD setup without the auxiliary SE score. Under this setting, the separation becomes much larger: Our RoI Crop improves over Random Crop by 6.5/9.2 AUROC on InternVL VQAv2/AdVQA or by 11.6/14.0 AUROC on QwenVL VQAv2/AdVQA, and is even more superior to the Reversed Crop. We additionally include GQA [18] because it provides ground-truth object annotations, enabling a controlled comparison with GT RoIs. On GQA, our operation is comparable to GT RoIs Crop and both clearly outperform the others, validating that CIM relies on preserving query-relevant causal evidence rather than arbitrary perturbation, and exact crops are less critical than retaining causal evidence. Table 3: Ablation study of the xcx_c-focused image transform. AUROC and AURAC use the final mixed score, while AUROCepiAUROC^epi uses the SE-free pure SD score. GT RoI is available only on GQA; – denotes not applicable. IVL / QVL denotes InternVL / QwenVL, respectively. Metric AUROC AUROCepiAUROC^epi AURAC Benchmark VQAv2 AdVQA GQA VQAv2 AdVQA GQA VQAv2 AdVQA GQA MLLM IVL QVL IVL QVL IVL QVL IVL QVL IVL QVL IVL QVL IVL QVL IVL QVL IVL QVL Random Crop 80.2 77.9 75.7 71.9 77.9 68.8 70.3 62.1 61.4 54.3 61.0 57.5 83.6 81.6 66.7 61.5 81.5 61.3 Reversed Crop 77.6 76.1 74.8 71.9 76.8 69.5 53.7 56.5 54.2 50.7 58.7 54.5 81.5 79.2 65.0 61.3 80.1 61.5 GT RoI Crop – – – – 80.3 71.4 – – – – 72.1 66.3 – – – – 82.4 63.9 Ours (RoI Crop) 81.3 78.5 76.8 74.5 79.9 70.2 76.8 73.7 70.6 68.3 73.3 64.6 85.2 81.9 66.9 64.8 82.5 63.1 Figure 3: Hallucination detection performance with fixed numbers of total samples, w.r.t. the proportion of samples for semU_sem calculation besides SE calculation, showing QwenVL results using AdVQA (left) and MME (right). Figure 4: Sample-wise efficiency comparison for semU_sem and EEDU_EED. It takes only half time for EEDU_EED to compute the uncertainty under saturated performance. Number of Samples. In Fig. 4, we show the optimal sampling allocation under a fixed total number K+K′K+K between estimating the original P(s|v)P(s|v), i.e., for SE and semU_sem, and measuring the masked P(s|v′)P(s|v ) only for semU_sem. We observe a consistent middle-peaking trend across benchmarks. The maximal performance is achieved at a balanced sampling budget of 40%-60% each. Biased allocations degrade performance, as insufficient samples for either distribution lead to mis-estimation of one type of uncertainty. Furthermore, while increasing the total budget universally improves AUROC, the optimal allocation proportion remains stable, suggesting its robustness in such scaling. Figure 5: Hallucination mitigation performance (Acc%) using uncertainty values estimated by our proposed method to threshold and re-answer the query. Efficiency of EED. Fig. 4 compares the average processing time per sample between the Semantic Divergence semU_sem and our proposed proxy EEDU_EED. Across both MLLM architectures and all four benchmarks, EEDU_EED consistently reduces the computational cost by approximately 50%. For instance, on the computationally demanding AdVQA benchmark with QwenVL, the processing time drops from ∼9.6 9.6s to ∼4.8 4.8s per sample. Compared with sampling-and-clustering methods under the same generation budget, EED removes the NLI-based ((K+K′)2)O((K+K )^2)-time clustering for the perturbed input and operates directly in the embedding space. Appendix C.9 further shows that SD is not strongly tied to one entailment model, but EED avoids that clustering dependency altogether. Figure 6: Adaptive-resolution VQA performance on MME using different uncertainty thresholds as the indicator: confident samples are processed under low resolutions. Left: w/ InternVL; Right: w/ QwenVL. Applications. Beyond detection, a practical uncertainty metric should enable the model to correct its own errors. We study whether CIM can act as a filter to improve factual quality. In this setting, we perform a reject-and-refine strategy: if the estimated uncertainty semU_sem exceeds a threshold, we instruct the model to re-generate the answer. Accuracy improvements on VQAv2, OKVQA, and AdVQA are reported in Fig. 5, which demonstrates the effectiveness of such intervention. By using our uncertainty estimates to trigger re-answering, we observe consistent accuracy gains across all benchmarks. Importantly, on the difficult AdVQA benchmark, the performance of InternVL improves from 47.9% to 51.1%, and QwenVL from 47.5% to 50.0%. This significant increase indicates that our method successfully flags epistemic failures where the model was previously hallucinating due to adversarial visual contexts. We further apply CIM to adaptive-resolution inference, where samples with uncertainty below a percentile threshold are processed at a low maximum resolution of 256256, reserving high-resolution inference for uncertain samples. As shown in Fig. 6, thresholding confident samples reduces inference time and also boosts accuracy beyond the high-resolution baseline. For QwenVL, downscaling the most confident 50% samples reduces average latency from 0.65s to 0.50s and raises accuracy from ∼75.2% 75.2\% to ∼76.1% 76.1\%; InternVL shows similar gains around the 50th–70th percentile thresholds. This suggests that high resolution can distract the inference on already sufficient samples, while CIM identifies when low-resolution evidence is adequate. 5 Conclusion This work addresses the critical bias in existing MLLM uncertainty quantification methods toward aleatoric uncertainty, which often ignores hallucinations driven by spurious correlations. We proposed CIM, a framework that isolates epistemic uncertainty by measuring the semantic shift under causally invariant perturbations with theoretical guarantee. Our proposed Semantic Divergence effectively targets reliance on non-causal features, achieving state-of-the-art performance. Furthermore, the Expected Embedding Drift (EED) reduces computational latency by 50% without compromising accuracy. By enabling robust hallucination detection and efficient adaptive inference, CIM enables a rigorous path toward trustworthy multimodal reasoning. Limitations. CIM depends on the quality of the RoIs and equivalence estimation, so failures of them can affect the results. Nevertheless, we empirically show their impact is limited. Acknowledgments and Disclosure of Funding Use unnumbered first level headings for the acknowledgments. All acknowledgments go at the end of the paper before the list of references. Moreover, you are required to declare funding (financial activities supporting the submitted work) and competing interests (related financial activities outside the submitted work). More information about this disclosure can be found at: https://neurips.c/Conferences/2026/PaperInformation/FundingDisclosure. Do not include this section in the anonymized submission, only in the final paper. You can use the ack environment provided in the style file to automatically hide this section in the anonymized submission. References [1] G. Ahdritz, T. Qin, N. Vyas, B. Barak, and B. L. Edelman (2024) Distinguishing the knowable from the unknowable with language models. In Proceedings of the 41st International Conference on Machine Learning, p. 503–549. Cited by: §1. [2] M. Arjovsky, L. Bottou, I. Gulrajani, and D. Lopez-Paz (2019) Invariant risk minimization. arXiv preprint arXiv:1907.02893. Cited by: §3.3. [3] S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y. Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y. Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, and J. Lin (2025) Qwen2.5-vl technical report. External Links: 2502.13923, Link Cited by: 8th item, §4. [4] T. Bai, H. Liang, B. Wan, Y. Xu, X. Li, S. Li, L. Yang, B. Li, Y. Wang, B. Cui, et al. (2024) A survey of multimodal large language model from a data-centric perspective. arXiv preprint arXiv:2405.16640. Cited by: §1. [5] Z. Bai, P. Wang, T. Xiao, T. He, Z. Han, Z. Zhang, and M. Z. Shou (2024) Hallucination of multimodal large language models: a survey. arXiv preprint arXiv:2404.18930. Cited by: §1. [6] M. Blondel, M. E. Sander, G. Vivier-Ardisson, T. Liu, and V. Roulet (2025) Autoregressive language models are secretly energy-based models: insights into the lookahead capabilities of next-token prediction. arXiv preprint arXiv:2512.15605. Cited by: §3.3. [7] K. Bousmalis, G. Trigeorgis, N. Silberman, D. Krishnan, and D. Erhan (2016) Domain separation networks. Advances in Neural Information Processing Systems 29. Cited by: §3.3. [8] M. A. Carreira-Perpinan and G. Hinton (2005) On contrastive divergence learning. In International workshop on artificial intelligence and statistics, p. 33–40. Cited by: §3.3. [9] F. O. Catak and M. Kuzlu (2024) Uncertainty quantification in large language models through convex hull analysis. Discover Artificial Intelligence 4 (1), p. 90. Cited by: §2.1. [10] C. Chen, K. Liu, Z. Chen, Y. Gu, Y. Wu, M. Tao, Z. Fu, and J. Ye (2024) INSIDE: llms’ internal states retain the power of hallucination detection. In Proceedings of the Twelfth International Conference on Learning Representations, Cited by: §2.1, Table 1, Table 2. [11] Z. Chen, W. Wang, H. Tian, S. Ye, Z. Gao, E. Cui, W. Tong, K. Hu, J. Luo, Z. Ma, et al. (2024) How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. Science China Information Sciences 67 (12), p. 220101. Cited by: 7th item, §4. [12] Y. Du and I. Mordatch (2019) Implicit generation and modeling with energy based models. Advances in neural information processing systems 32. Cited by: §3.3. [13] S. Farquhar, J. Kossen, L. Kuhn, and Y. Gal (2024) Detecting hallucinations in large language models using semantic entropy. Nature 630 (8017), p. 625–630. Cited by: §1, §2.1, §3.1, Table 1, Table 2. [14] C. Fu, P. Chen, Y. Shen, Y. Qin, M. Zhang, X. Lin, J. Yang, X. Zheng, K. Li, X. Sun, et al. (2025) Mme: a comprehensive evaluation benchmark for multimodal large language models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, Cited by: 6th item, §4. [15] Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh (2017) Making the v in vqa matter: elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, p. 6904–6913. Cited by: 1st item, §4. [16] P. He, X. Liu, J. Gao, and W. Chen (2021) Deberta: decoding-enhanced bert with disentangled attention. In International Conference on Learning Representations, Cited by: §C.9, 11st item. [17] L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, X. Feng, B. Qin, et al. (2025) A survey on hallucination in large language models: principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems 43 (2), p. 1–55. Cited by: §1. [18] D. A. Hudson and C. D. Manning (2019) Gqa: a new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 6700–6709. Cited by: 4th item, §4.2. [19] E. Hüllermeier and W. Waegeman (2021) Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods. Machine learning 110 (3), p. 457–506. Cited by: §1. [20] J. Kossen, J. Han, M. Razzak, L. Schut, S. Malik, and Y. Gal (2024) Semantic entropy probes: robust and cheap hallucination detection in llms. arXiv preprint arXiv:2406.15927. Cited by: §1. [21] L. Kuhn, Y. Gal, and S. Farquhar (2023) Semantic uncertainty: linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Conference on Learning Representations, Cited by: §1. [22] S. Kumar and Y. Tsvetkov (2019) Von mises-fisher loss for training sequence to sequence models with continuous outputs. In International Conference on Learning Representations, Cited by: Assumption 3.4. [23] G. K. R. Lau, H. Dao, N. K. H. Lin, and B. K. H. Low (2025) Uncertainty quantification for mllms. In ICML 2025 Workshop on Reliable and Responsible Foundation Models, Cited by: §2.2, Table 1, Table 2. [24] Y. LeCun, S. Chopra, R. Hadsell, M. Ranzato, F. Huang, et al. (2006) A tutorial on energy-based learning. Predicting structured data 1 (0). Cited by: §3.3. [25] L. Li, J. Lei, Z. Gan, and J. Liu (2021) Adversarial vqa: a new benchmark for evaluating the robustness of vqa models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 2042–2051. Cited by: 3rd item, §4. [26] Y. Li, Y. Du, K. Zhou, J. Wang, W. X. Zhao, and J. Wen (2023) Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p. 292–305. Cited by: 5th item, §4. [27] Z. Lin, S. Trivedi, and J. Sun (2024) Generating with confidence: uncertainty quantification for black-box large language models. Transactions on Machine Learning Research 2024. Cited by: §2.1. [28] H. Liu, C. Li, Q. Wu, and Y. J. Lee (2023) Visual instruction tuning. Advances in Neural Information Processing Systems 36, p. 34892–34916. Cited by: §1. [29] S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, Q. Jiang, C. Li, J. Yang, H. Su, et al. (2024) Grounding dino: marrying dino with grounded pre-training for open-set object detection. In European conference on computer vision, p. 38–55. Cited by: 9th item, §3.2, §4. [30] K. V. Mardia and P. E. Jupp (2009) Directional statistics. John Wiley & Sons. Cited by: Assumption 3.4. [31] K. Marino, M. Rastegari, A. Farhadi, and R. Mottaghi (2019) Ok-vqa: a visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 3195–3204. Cited by: 2nd item, §4. [32] S. Minaee, T. Mikolov, N. Nikzad, M. Chenaghlu, R. Socher, X. Amatriain, and J. Gao (2024) Large language models: a survey. arXiv preprint arXiv:2402.06196. Cited by: §1. [33] R. A. Minut, H. Dewidar, I. Masi, et al. (2026) Spilled energy in large language models. In Proceedings of the International Conference on Learning Representations, Cited by: §3.3. [34] A. Nikitin, J. Kossen, Y. Gal, and P. Marttinen (2024) Kernel language entropy: fine-grained uncertainty quantification for llms from semantic similarities. Advances in Neural Information Processing Systems 37, p. 8901–8929. Cited by: §1. [35] J. Pearl, M. Glymour, and N. P. Jewell (2016) Causal inference in statistics: a primer. John Wiley & Sons. Cited by: §3.3. [36] J. Pearl et al. (2000) Models, reasoning and inference. Cambridge, UK: CambridgeUniversityPress 19 (2), p. 3. Cited by: §3.3. [37] J. Peng, J. Chen, M. A. Rojas, and M. Zhang (2025) Mv-core: multimodal visual-conceptual reasoning for complex visual question answering. arXiv preprint arXiv:2508.07023. Cited by: §1. [38] J. Peters, P. Bühlmann, and N. Meinshausen (2016) Causal inference by using invariant prediction: identification and confidence intervals. Journal of the Royal Statistical Society Series B: Statistical Methodology 78 (5), p. 947–1012. Cited by: §3.3. [39] L. Qin, S. Welleck, D. Khashabi, and Y. Choi (2022) Cold decoding: energy-based constrained text generation with langevin dynamics. Advances in Neural Information Processing Systems 35, p. 9538–9551. Cited by: §3.3. [40] B. Schölkopf, F. Locatello, S. Bauer, N. R. Ke, N. Kalchbrenner, A. Goyal, and Y. Bengio (2021) Toward causal representation learning. Proceedings of the IEEE 109 (5), p. 612–634. Cited by: §3.3. [41] O. Shorinwa, Z. Mei, J. Lidard, A. Z. Ren, and A. Majumdar (2025) A survey on uncertainty quantification of large language models: taxonomy, open research challenges, and future directions. ACM Computing Surveys. Cited by: §1, §2.1. [42] L. Tao, Y. Yeh, M. Dong, T. Huang, P. Torr, and C. Xu (2025) Revisiting uncertainty estimation and calibration of large language models. arXiv preprint arXiv:2505.23854. Cited by: §1. [43] K. Tian, E. Mitchell, A. Zhou, A. Sharma, R. Rafailov, H. Yao, C. Finn, and C. D. Manning (2023) Just ask for calibration: strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p. 5433–5442. Cited by: §2.1, Table 1, Table 2. [44] B. Xiao, H. Wu, W. Xu, X. Dai, H. Hu, Y. Lu, M. Zeng, C. Liu, and L. Yuan (2024) Florence-2: advancing a unified representation for a variety of vision tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 4818–4829. Cited by: §C.6, 10th item. [45] M. Xiong, Z. Hu, X. Lu, Y. Li, J. Fu, J. He, and B. Hooi (2023) Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms. arXiv preprint arXiv:2306.13063. Cited by: §2.1, Table 1, Table 2. [46] M. Xiong, Z. Hu, X. Lu, Y. LI, J. Fu, J. He, and B. Hooi (2024) Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms. In The Twelfth International Conference on Learning Representations, Cited by: §1. [47] S. Yin, C. Fu, S. Zhao, K. Li, X. Sun, T. Xu, and E. Chen (2024) A survey on multimodal large language models. National Science Review 11 (12). Cited by: §1. [48] R. Zhang, H. Zhang, H. Fei, and Z. Zheng (2025) Uncertainty-o: one model-agnostic framework for unveiling uncertainty in large multimodal models. arXiv preprint arXiv:2506.07575. Cited by: §2.2, Table 1, Table 2. [49] R. Zhang, H. Zhang, and Z. Zheng (2024) Vl-uncertainty: detecting hallucination in large vision-language model via uncertainty estimation. arXiv preprint arXiv:2411.11919. Cited by: §2.2. [50] X. Zhang, F. Zeng, Y. Quan, Z. Hui, and J. Yao (2025) Enhancing multimodal large language models complex reason via similarity computation. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39, p. 10203–10211. Cited by: §1. [51] W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong, et al. (2023) A survey of large language models. arXiv preprint arXiv:2303.18223 1 (2). Cited by: §1. Appendix Appendix A Proofs of Equations and Propositions A.1 Proof of Lemma 3.2 (Aleatoric Cancellation) Proof. Let P(s)≜Pθ(s|v)P(s) P_θ(s|v) and Q(s)≜Pθ(s|v′)Q(s) P_θ(s|v ) be the distribution given the original input and the masked input v′=T(v)v =T(v), respectively. The log-probabilities for these inputs are logQ(s) Q(s) =−(Ec(s,vc)+Es(s,vs′))−logZ(v′) =-(E_c(s,v_c)+E_s(s,v _s))- Z(v ) logP(s) P(s) =−(Ec(s,vc)+Es(s,vs))−logZ(v). =-(E_c(s,v_c)+E_s(s,v_s))- Z(v). Subtracting the two equations yields: logQ(s)P(s) Q(s)P(s) =(Es(s,vs)−Es(s,vs′))+logZ(v)Z(v′) =(E_s(s,v_s)-E_s(s,v _s))+ Z(v)Z(v ) =Δs(s)+logZ(v)Z(v′). = _s(s)+ Z(v)Z(v ). (9) ∎ A.2 Proof of Theorem 3.3 (Epistemic Jensen Gap) Part 1: Equivalence to Jensen Gap Proof. We first expand the partition function ratio. By definition, Z(v)=∑yexp(−(Ec(s,vc)+Es(s,vs)))Z(v)= _y (-(E_c(s,v_c)+E_s(s,v_s))). The exponent can be rewritten as −Ec−Es(vs′)−(Es(vs)−Es(vs′))=−E(s,v′)−Δs(s)-E_c-E_s(v _s)-(E_s(v_s)-E_s(v _s))=-E(s,v )- _s(s). Therefore, we can derive that Z(v)=∑se−Eθ(s,v′)e−Δs(s).Z(v)= _se^-E_θ(s,v )e^- _s(s). (10) By definition of Z(v′)Z(v ), we have e−Eθ(s,v′)=Z(v′)Q(s)e^-E_θ(s,v )=Z(v )Q(s) which simplifies Eq. (10) as Z(v) Z(v) =∑sZ(v′)Q(s)e−Δs(s) = _sZ(v )Q(s)e^- _s(s) =Z(v′)⋅s∼Q[e−Δs(s)] =Z(v )·E_s Q[e^- _s(s)] (11) Thus, the constant term from Lemma 3.2 is logZ(v)Z(v′)=logy∼Q[e−Δs(s)] Z(v)Z(v )= _y Q[e^- _s(s)]. Now, substituting Lemma 3.2 into the definition in Eq. (1) yields sem _sem =s∼Q[logQ(s)P(y)] =E_s Q [ Q(s)P(y) ] =s∼Q[Δs(s)+logs′∼Q[e−Δs(s′)]] =E_s Q [ _s(s)+ _s Q[e^- _s(s )] ] =logs∼Q[e−Δs(s)]−s∼Q[−Δs(s)], = _s Q[e^- _s(s)]-E_s Q[- _s(s)], (12) which is precisely the Jensen Gap (ex,−Δs)≥0J(e^x;- _s)≥ 0. The divergence is strictly zero if and only if Δs(s) _s(s) is constant almost surely over S, implying the model is insensitive to spurious features. While Eq. (12) provides an exact measure, it can be interpreted best at a low hallucination scenario, i.e., the model is trained well enough to produce relatively smaller |Δs(s)|| _s(s)|, which results in Eq. 3. ∎ Part 2: Convergence to Variance (Proof of Eq. (3)) Proof. Let K−Δ(t)=logs∼Q[et(−Δs(s))]K_- (t)= _s Q[e^t(- _s(s))] be the Cumulant Generating Function (CGF) of the random variable −Δs(s)- _s(s). From Theorem 1, we identify that: DKL(Q∥P)=K−Δ(1)−s∼Q[−Δs(s)]=K−Δ(1)+s∼Q[Δs(s)]D_KL(Q\|P)=K_- (1)-E_s Q[- _s(s)]=K_- (1)+E_s Q[ _s(s)] (13) We apply Taylor’s Theorem with the Lagrange form of the remainder to K−Δ(t)K_- (t) on the interval [0,1][0,1]. There exists some ξ∈(0,1)ξ∈(0,1) such that: K−Δ(1)=K−Δ(0)+K−Δ′(0)(1)+K−Δ′(0)2!(1)2+K−Δ′(ξ)3!(1)3K_- (1)=K_- (0)+K _- (0)(1)+ K _- (0)2!(1)^2+ K _- (ξ)3!(1)^3 (14) Using the properties of cumulants, we have K−Δ(0)=0K_- (0)=0, K−Δ′(0)=Q[−Δs(s)]K _- (0)=E_Q[- _s(s)], and K−Δ′(0)=VarQ(−Δs(s))K _- (0)=Var_Q(- _s(s)). Given the assumption that the shift is bounded by sup|Δs(s)|≤ϵ | _s(s)|≤ε, the third derivative satisfies |K−Δ(3)(ξ)|≤Cϵ3|K^(3)_- (ξ)|≤ Cε^3 for some constant C, effectively bounding the remainder term by (ϵ3)O(ε^3). Substituting these values: K−Δ(1)=−Q[Δs(s)]+12VarQ(Δs(s))+(ϵ3)K_- (1)=-E_Q[ _s(s)]+ 12Var_Q( _s(s))+O(ε^3) (15) Substituting this expansion back into the KL equation, the linear expectation terms cancel: DKL(Q∥P) D_KL(Q\|P) =(−Q[Δs(s)]+12VarQ(Δs(s))+(ϵ3))+Q[Δs(s)] = (-E_Q[ _s(s)]+ 12Var_Q( _s(s))+O(ε^3) )+E_Q[ _s(s)] =12Vars∼Q[Δs(s)]+(ϵ3) = 12Var_s Q[ _s(s)]+O(ε^3) (16) ∎ A.3 Proof of Lemma 3.5 (KL Divergence of vMF Distributions) Proof. By definition, the KL divergence is: DKL(Q∥P) D_KL(Q\|P) =∫d−1q()logq()p() = _S^d-1q(z) q(z)p(z)dz =∫d−1q()(logCd(κ)eκQ⊤Cd(κ)eκP⊤) = _S^d-1q(z) ( C_d(κ)e^κ μ_Q zC_d(κ)e^κ μ_P z )dz =∫d−1q()(κQ⊤−κP⊤) = _S^d-1q(z) (κ μ_Q z-κ μ_P z )dz =κ(Q−P)⊤∫d−1q(). =κ( μ_Q- μ_P) _S^d-1zq(z)dz. (17) A property of the vMF distribution is that the expected vector is ∼Q[]=Ad(κ)QE_z Q[z]=A_d(κ) μ_Q. Substituting this into the integral: DKL(Q∥P) D_KL(Q\|P) =κ(Q−P)⊤(Ad(κ)Q) =κ( μ_Q- μ_P) (A_d(κ) μ_Q) =κAd(κ)(Q⊤Q−P⊤Q). =κ A_d(κ)( μ_Q μ_Q- μ_P μ_Q). (18) Since Q μ_Q is a unit vector, Q⊤Q=1 μ_Q μ_Q=1. Thus, DKL(Q∥P)=κAd(κ)(1−P⊤Q)D_KL(Q\|P)=κ A_d(κ)(1- μ_P μ_Q). ∎ A.4 Proof of Theorem 3.6 (Geometric Proxy Equivalence) Proof. The EED metric computes the expectation of the cosine distance over the sampled trajectory from Q. Taking the expectation of Eq. (6) with respect to yt∼Qy_t Q: yt∼Q[EED] _y_t Q[U_EED] =1−yt∼Q[yt]⊤P =1-E_y_t Q[e_y_t] μ_P =1−(Ad(κ)Q)⊤P. =1-(A_d(κ) μ_Q) μ_P. (19) Rearranging terms for Q⊤P μ_Q μ_P and substituting into Lemma 3.5: DKL(Q∥P) D_KL(Q\|P) =κAd(κ)(1−1−[EED]Ad(κ)) =κ A_d(κ) (1- 1-E[U_EED]A_d(κ) ) =κyt∼Q[EED]−κ(1−Ad(κ)). = _y_t Q[U_EED]-κ(1-A_d(κ)). (20) The divergence relates to EED via an affine transformation. The term κ(1−Ad(κ))κ(1-A_d(κ)) is a constant dependent only on concentration κ and dimension d. Under high concentration (κ→∞κ→∞), Ad(κ)→1A_d(κ)→ 1, and the relation simplifies to exact linearity. Thus, minimizing EED strictly minimizes the divergence between the semantic distributions. ∎ Appendix B Detailed Experiment Setup Computational Resources We carried out all experiments on 4 NVIDIA RTX A6000 GPUs. Model Parameters The implementation uses an inference temperature of 0.1, while the sampling temperature is 0.5. The RoI transform uses GroundingDINO with detection threshold 0.3, which filters boxes labeled as the whole image when they cover more than 95% of the image area and uses a central crop fallback when no valid detection remains. Prompts The dataset question is passed to the evaluated MLLM with this prompt:[Question] Answer this question shortly. The prompt used to extract key objects for grounding is:What are the key objects in this question-answer pair? Q: [Question] A: [Answer] Output in json format: "key_objects": ["key object 1", "key object 2", ...] where the placeholders are filled with the original question and the model’s original answer, and the judge LLM is queried with temperature 0.10.1. The prompt used for pairwise semantic clustering of sampled answers is:Question: [Question] True Answer: [Answer A] Is the following answer correct? Answer: [Answer B] Respond with ’no’ if this is totally different from the True Answer. Respond with ’yes’ when they are of the same meaning. This clustering prompt is filled with the text query and the answer pairs, using a temperature of 0.10.1. Causal-Invariant Transform. We use GroundingDINO as a lightweight RoI generator for the causal-invariant transform T(⋅)T(·). When no valid RoI is found, we use a central crop as the fallback view. This choice is not essential to the framework: task-specific or query-conditioned detectors can be substituted when they are available. Dataset Configuration. To keep evaluation consistent and computationally feasible, we fix the evaluation subsets across all methods. For hallucination-targeted benchmarks, we use the full test sets: 3,000 POPE examples across the random, popular, and adversarial splits, and 2,374 MME examples. For VQAv2, OKVQA, AdVQA, and GQA, we use the first 2,000 validation examples from each dataset. Sampling Budget and Fair Comparison. For all sampling-based methods, we fix the total decoding budget to Ktotal=32K_total=32 sequences per query. • Baselines: SE, EigenScore, and UMPIRE sample only from the original distribution P(y|v)P(y|v), so we set K=32K=32. • Ours (CIM): CIM samples from both the original distribution P(y|v)P(y|v) and the causally-filtered distribution P(y|v′)P(y|v ). We use K=16K=16 original-view samples and K′=16K =16 filtered-view samples, keeping K+K′=32K+K =32. Metric Interpolation. The K=16K=16 original-view samples used by CIM can also estimate standard Semantic Entropy without extra decoding. We therefore report a mixed score final=λours,K′=16+(1−λ)⋅SE,K=16U_final= _ours,K =16+(1-λ)·U_SE,K=16, where oursU_ours is either semU_sem or EEDU_EED. Here, λ is set to 0.050.05. This score retains sensitivity to aleatoric uncertainty while adding the epistemic signal measured by CIM. Appendix C More Experiments and Results C.1 Additional MME Results Figure 7: Uncertainty estimation performance (%) on the MME benchmark. Figure 7 reports hallucination detection on MME. CIM achieves the best performance (74.7%), showing that the uncertainty signal generalizes beyond the VQA datasets in the main tables. C.2 Case Study Figure 8: Representative instances of SE vs. semU_sem on hallucinated samples. In all instances, our proposed CIM uncertainty is high despite the standard SE remains low, indicating epistemic uncertainty that common sampling-based uncertainty metric fails to identify. To better understand the behavior of CIM in practice, we present a case analysis of SE’s failure cases in Fig. 8. As illustrated, the MLLM frequently generates incorrect answers with high confidence, leading to extremely low SE scores. For instance, in the left example, the model consistently mispredicts ‘Japanese dish,’ likely misled by background Japanese text. Similarly, in the right example, the model hallucinates the color possibly by looking at an incorrect object. Empirically, counting complex objects can often hallucinate but result in a stable number or low semantic entropy, as shown in the middle instance. In these scenarios, standard metrics fail but the proposed method consistently peaks. Such behavior confirms the successful detection of epistemic failure with our CIM. C.3 Gaussian Noise Perturbation We compare RoI masking with the commonly used gaussian noise perturbation under the same VQAv2 and AdVQA settings as the main masking ablation. Table 4 reports the final mixed uncertainty score. Gaussian noise perturbs the entire image and therefore cannot selectively preserve xcx_c while suppressing xsx_s. Its performance stays close to random masking and below RoI masking across benchmarks. This supports the main conclusion that the improvement is not caused by simply changing the image, but by constructing an evidence-preserving view that removes non-causal context. Table 4: Gaussian noise perturbation ablation. Results are AUROC/AURAC on VQAv2 and AdVQA. Benchmark VQAv2 AdVQA MLLM InternVL QwenVL InternVL QwenVL Metric AUROC AURAC AUROC AURAC AUROC AURAC AUROC AURAC Random Masking 80.2 83.6 77.9 81.6 75.7 66.7 71.9 61.5 Gaussian Noise 80.3 84.0 77.8 80.7 75.5 66.5 73.4 63.0 Reversed Masking 77.6 81.5 76.1 79.2 74.8 65.0 71.9 61.3 Ours (RoI Masking) 81.3 85.2 78.5 81.9 76.8 66.9 74.5 64.8 C.4 Detector Failure Sensitivity We next test how performance changes when RoI quality is deliberately degraded. Starting from GT boxes on GQA, we replace an increasing fraction of boxes with random crops. We evaluate and show the uncertainty detection performance in Table 5. Table 5: GQA detector-failure sensitivity. A higher failure ratio means that more GT RoIs are replaced by random crops. Failure Ratio (%) InternVL AUROC QwenVL AUROC 0 72.1 66.3 20 69.4 62.8 40 67.1 61.4 60 65.1 59.7 80 63.3 58.7 100 61.0 57.5 The AUROC drops steadily as valid RoIs are replaced, confirming that CIM relies on preserving causal evidence rather than arbitrary perturbations such as random cropping. Meanwhile, In the actual pipeline, the detector fallback rate is low across benchmarks, as shown in Table 6. Table 6: GroundingDINO fallback rate. Fallback means no valid RoI is detected and the central crop is used. Benchmark VQAv2 OKVQA AdVQA MME Failure (%) 1.10 0.62 1.52 1.54 C.5 Relational and Global GQA Questions A concern with RoI cropping is that it may remove relationships or scene context. Our implementation uses the union box over detected query-relevant objects, so the spatial region between objects is preserved. We evaluate this directly on GQA by comparing common questions with relational/global questions. Table 7: GQA relational/global analysis. AUROC on common questions and complex questions involving relational or global reasoning. Setting GQA-common GQA-rel-global Metric InternVL QwenVL InternVL QwenVL SE 78.4 69.2 79.2 69.4 Ours (K+K′K+K ) 79.9 70.2 80.2 71.5 From the results in Table 7, we can see the gain is still maintained when testing relational/global questions, suggesting that the union box in our implementation keeps enough contextual structure for CIM to remain effective. C.6 Alternative Grounding with Florence-2 CIM does not require GroundingDINO or a union box specifically. To test whether the framework is sensitive to the detector choice, we replace GroundingDINO with the query-conditioned Florence-2 detector [44]. Table 8: Query-conditioned detector analysis. Results are AUROC/AURAC. GD denotes GroundingDINO and FL denotes Florence-2. Benchmark InternVL (GD) InternVL (FL) QwenVL (GD) QwenVL (FL) VQAv2 81.3/85.2 82.2/85.9 78.4/81.5 77.2/82.6 AdVQA 76.8/66.9 77.7/68.3 74.5/64.8 73.5/65.0 In the results shown in Table 8, the changes are small, indicating that CIM mainly depends on preserving query-relevant evidence and is not tied to a particular RoI detector. C.7 Empirical Validations for Energy Assumptions Practical causal energy could shift and spurious elements might remain. To further verify the influence of the decomposition assumption empirically, we evaluated logit causal energy by comparing our crops against vacant-background views whose area outside the crop is masked to black. We estimate causal and non-causal logit-energy shifts and compare their median magnitudes with the median top-1/top-2 logit gap. The results in Table 9 show that the shift is relatively small, validating the decomposition accuracy. Table 9: Causal-energy shift scale. Median ‖Δc‖\| _c\| compared with median logit top-1/top-2 gap. Benchmark InternVL QwenVL ‖Δc‖\| _c\| Logit Gap ‖Δc‖\| _c\| Logit Gap OKVQA 0.75 3.62 0.38 4.21 AdVQA 0.50 4.57 0.50 4.81 To validate the bounded non-causal energy in Theorem 3.3, we empirically evaluated its magnitude, i.e., the logit difference between original and masked views, on hallucinated samples. The results are shown in Table 10. Note that this value is accurate under our causal-invariance assumption. For scale anchoring, we compared |Δs|| _s| against the median top-1/top-2 logit gap. The median |Δs|| _s| (0.50-1.00) is consistently much smaller than the median decision margin (3.62-4.81). Based on these results, the theoretical bound ϵε can be estimated at ϵ≈1.0ε≈ 1.0. This value is sufficiently small relative to the logit scale, confirming our theoretical approximation holds in practice. Table 10: Non-causal shift scale. Median ‖Δs‖\| _s\| on hallucinated samples compared with median logit top-1/top-2 gap. Benchmark InternVL QwenVL ‖Δs‖\| _s\| Logit Gap ‖Δs‖\| _s\| Logit Gap OKVQA 0.75 3.62 0.50 4.21 AdVQA 1.00 4.57 0.62 4.81 Both ‖Δc‖\| _c\| and ‖Δs‖\| _s\| are substantially smaller than the corresponding decision margins. These estimates support the small-shift approximation in Theorem 3.3 and suggest that causal-energy leakage remains limited when query-relevant regions are preserved. C.8 POPE Entropy Distributions Table 11 reports the average token-level entropy during decoding on POPE. QwenVL has a much sharper output distribution than InternVL, which helps explain why discrete semantic clustering can be less stable on QwenVL and why EED is particularly effective for this model. Table 11: Average token-level entropy on POPE. MLLM Random Popular Adversarial InternVL-8B 0.51 0.51 0.54 Qwen2.5-VL-7B 0.06 0.07 0.08 C.9 NLI Model Ablation Table 12: NLI model ablation. Δ(SD Qwen,SD DeBERTa) (SD Qwen,SD DeBERTa) represents the metric value difference between "SD w/ Qwen-7B" and "SD w/ DeBERTa-small". Metric VQAv2 OKVQA AUROC AURAC AUROC AURAC SE w/ DeBERTa-small 74.5 78.0 77.5 83.5 SD w/ DeBERTa-small 76.2 80.4 78.6 84.4 SE w/ Qwen-7B 76.5 79.5 78.6 83.8 SD w/ Qwen-7B 78.4 81.5 79.6 84.2 Δ (SD Qwen, SD DeBERTa) 2.2 1.1 1.0 -0.2 Figure 9: Left: Expected Embedding Drift (EED) as a proxy for Semantic Divergence. Right: Harmonic mean of AUROC and AURAC for different distribution distances, averaged across 2 MLLMs and 7 benchmarks. Semantic clustering depends on an entailment model, so we compare the default Qwen-7B entailment model with a much smaller DeBERTa-small model [16]. Table 12 shows that the weaker entailment model slightly changes performance, but SD remains competitive and continues to outperform SE under the same NLI model. C.10 EED Proxy and Distance Metrics Figure 9 left validates the connection between EED and Semantic Divergence by grouping samples according to semU_sem and plotting the corresponding EED values. The mean EED increases with Semantic Divergence, supporting EED as an efficient proxy for semantic distribution shift. Figure 9 right compares squared L2L_2 distance, Jensen-Shannon Divergence, reverse KL DKL(P∥Q)D_KL(P\|Q), and forward KL DKL(Q∥P)D_KL(Q\|P). The method is not highly sensitive to the exact distance, but forward KL performs best in our experiments, consistent with its role as an information-gain measure from the causally-filtered view to the original view. Appendix D Licenses for Existing Assets We use existing public datasets, models, and detector assets for research evaluation only and do not redistribute their data or checkpoints. We cite the original creators in the main paper and follow the licenses or terms listed by their official project pages: • VQAv2 [15]: the VQA Consortium states that VQA annotations are licensed under Creative Commons Attribution 4.0 International, and its software is distributed under BSD-style redistribution terms; see https://visualqa.org/terms.html. The associated images are COCO images and remain subject to the original COCO/image-source terms. • OK-VQA [31]: the official download page provides OK-VQA annotations/questions and COCO images, but does not state a separate OK-VQA annotation license; see https://okvqa.allenai.org/download.html. We therefore use the benchmark under the official access terms and the underlying COCO image terms. • AdVQA [25]: the official release page lists the AdVQA/AVQA data and image sources, but does not state a separate dataset license; see https://adversarialvqa.org/download.html. We use it under the official access terms and the source-image terms for COCO, Conceptual Captions, Fakeddit, and VCR as applicable. • GQA [18]: the official GQA website provides images that come from COCO and Flickr and scene graphs based on Visual Genome; see https://cs.stanford.edu/people/dorarad/gqa/index.html and https://cs.stanford.edu/people/dorarad/gqa/download.html. We could not find a separate license statement; the LMMs-Lab Hugging Face mirror lists MIT, but we treat the original COCO, Flickr, and Visual Genome terms as applicable to them. • POPE [26]: the official RUCAIBox/POPE repository is released under the MIT license; see https://github.com/RUCAIBox/POPE. Its COCO-derived components also inherit the relevant COCO image/annotation terms. • MME [14]: the project page links to the Hugging Face dataset release, which lists Apache-2.0 as the dataset license; see https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models/tree/Evaluation and https://huggingface.co/datasets/darkyarding/MME. • InternVL2-8B [11]: the official Hugging Face model card lists an MIT license and notes that its InternLM2.5-7B-Chat component is licensed under Apache-2.0; see https://huggingface.co/OpenGVLab/InternVL2-8B. • Qwen2.5-VL-7B-Instruct and Qwen2.5-7B-Instruct [3]: their official Hugging Face model cards list Apache-2.0; see https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct and https://huggingface.co/Qwen/Qwen2.5-7B-Instruct. • GroundingDINO [29]: the official repository license file is Apache-2.0; see https://github.com/IDEA-Research/GroundingDINO/blob/main/LICENSE. • Florence-2 [44]: the official Microsoft Hugging Face model card lists an MIT license; see https://huggingface.co/microsoft/Florence-2-large. • DeBERTa-small [16]: the Microsoft DeBERTa repository and the microsoft/deberta-v3-small Hugging Face checkpoint list the MIT license; see https://github.com/microsoft/DeBERTa/blob/master/LICENSE and https://huggingface.co/microsoft/deberta-v3-small.