Paper deep dive
Attention-Path Fragility as an Uncertainty Signal in Large Language Models
Minsoo Kim, Sungyoung Ji, Kisung Moon, Ilyong Yoon
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 8/16/2026, 3:51:52 AM
Summary
The paper introduces ASMI (Attention-Subnetwork Mutual Information), a training-free uncertainty estimator for Large Language Models that measures fragility in attention pathways. By masking attention heads and calculating mutual information (BALD) among subnetworks, ASMI identifies 'confident-but-fragile' predictions. It outperforms baselines like Semantic Entropy in grounded QA tasks where answers depend on context routing, while correctly reverting to low uncertainty in parametric QA tasks. Variants include Sem-ASMI (using semantic agreement kernels) and Adapt-ASMI (adaptive gating based on response diversity).
Entities (11)
Relation Signals (8)
ASMI → effectivein → Grounded QA
confidence 95% · on grounded QA an out-of-fold test shows it adds error-predictive information... concentrated in confident-but-fragile predictions
ASMI → ineffectivein → Parametric QA
confidence 95% · On parametric QA all variants revert to or below the zero-cost MSP baseline, exactly as predicted
Sem-ASMI → isvariantof → ASMI
confidence 95% · Sem-ASMI reads the signal from a single greedy response... ties or beats Semantic Entropy
Adapt-ASMI → isvariantof → ASMI
confidence 95% · The best ASMI variant, typically the adaptive one... ties or leads the strongest baseline
ASMI → uses → Attention Head Masking
confidence 95% · ASMI... masks attention heads and measures the BALD mutual information among the resulting subnetworks
ASMI → evaluatedon → Qwen3-4B
confidence 90% · Our primary backbone is Qwen3-4B-base... We evaluate base models deliberately
ASMI → outperforms → Semantic Entropy
confidence 90% · Sem-ASMI... ties or beats Semantic Entropy on ten of the twelve grounded benchmark-backbone settings
ASMI → usesmetric → BALD
confidence 90% · measures the BALD mutual information among the resulting subnetworks
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We propose that a model's uncertainty about a token is reflected not only in the breadth of its output distribution but also in whether a confident prediction is \emph{fragile} under perturbation of its attention pathways. We instantiate this as ASMI (Attention-Subnetwork Mutual Information), a training-free estimator that masks attention heads and measures the BALD mutual information among the resulting subnetworks, with a semantic-agreement kernel to discount surface-form disagreement. The signal is not a restatement of output confidence: on grounded QA an out-of-fold test shows it adds error-predictive information beyond single-pass confidence and entropy, concentrated in \emph{confident-but-fragile} predictions, where acting on it roughly halves the retained error of a confidence filter. The distinctness is regime-graded, so ASMI predicts its own domain of applicability, strong where answers are routed through provided context and bounded by design where they are recalled from parametric knowledge. Sem-ASMI reads the signal from a single greedy response, without the stochastic generations the strongest baselines require, and ties or beats Semantic Entropy on ten of the twelve grounded benchmark-backbone settings. Across the same twelve settings, the best ASMI variant, typically the adaptive one reusing the ten samples already drawn for the baselines, ties or leads the strongest baseline in eight, significantly in three under a paired test. On parametric QA all variants revert to or below the zero-cost MSP baseline, exactly as predicted, and the estimates are near-deterministic across reruns. A head-level analysis shows that what tracks this boundary is not the presence of head-level fragility but whether that fragility couples to errors.
Tags
Links
- Source: https://arxiv.org/abs/2608.11138v1
- Canonical: https://arxiv.org/abs/2608.11138v1
Trouble viewing inline? Open PDF directly →
Full Text
105,932 characters extracted from source content.
Expand or collapse full text
Attention-Path Fragility as an Uncertainty Signal in Large Language Models Minsoo Kim Sungyoung Ji Kisung Moon Ilyong Yoon Abstract We propose that a model’s uncertainty about a token is reflected not only in the breadth of its output distribution but also in whether a confident prediction is fragile under perturbation of its attention pathways. We instantiate this as ASMI (Attention-Subnetwork Mutual Information), a training-free estimator that masks attention heads and measures the BALD mutual information among the resulting subnetworks, with a semantic-agreement kernel to discount surface-form disagreement. The signal is not a restatement of output confidence: on grounded QA an out-of-fold test shows it adds error-predictive information beyond single-pass confidence and entropy, concentrated in confident-but-fragile predictions, where acting on it roughly halves the retained error of a confidence filter. The distinctness is regime-graded, so ASMI predicts its own domain of applicability, strong where answers are routed through provided context and bounded by design where they are recalled from parametric knowledge. Sem-ASMI reads the signal from a single greedy response, without the stochastic generations the strongest baselines require, and ties or beats Semantic Entropy on ten of the twelve grounded benchmark-backbone settings. Across the same twelve settings, the best ASMI variant, typically the adaptive one reusing the ten samples already drawn for the baselines, ties or leads the strongest baseline in eight, significantly in three under a paired test. On parametric QA all variants revert to or below the zero-cost MSP baseline, exactly as predicted, and the estimates are near-deterministic across reruns. A head-level analysis shows that what tracks this boundary is not the presence of head-level fragility but whether that fragility couples to errors. Introduction Reliable uncertainty estimation (UE) is foundational for deploying large language models (LLMs) in selective generation, hallucination detection, and human-in-the-loop decision making (20; 29; 21), where over-confident incorrect outputs carry real costs (12; 17). The stakes are highest in grounded generation, where a hallucination is a failure to route the answer through the provided context (15). Estimators span information-theoretic quantities on a single deterministic forward pass (27), agreement statistics across stochastically decoded samples (25; 7; 5; 33), signals read from internal states and from attention (1; 47), and structural perturbations of the model itself (8; 61). Figure 1: ASMI at a glance. Masking heads leaves a confident token (left) unchanged but destabilizes a fragile token (right) that hinges on one critical path. ASMI scores this fragility: random attention-head masking at a target layer yields multiple predictive distributions whose disagreement, measured by mutual information, is a token-level uncertainty proxy. We propose a complementary signal: a model’s uncertainty about a token is reflected not only in the breadth of its output distribution, but also in whether the prediction is fragile, that is, whether it hinges on a specific attention pathway among many redundant alternatives. The intuition rests on a well-documented property of multi-head attention: many heads can be pruned with minimal loss (32; 48; 13), yet a high-level behavior can hinge on a single head (62), and a small subset is uncertainty-aware (47). Masking leaves a genuinely supported prediction intact, since redundant pathways carry the same information, and destabilizes a fragile one (Fig. 1). Crucially, this fragility is invisible to a single forward pass: a token can be assigned high confidence yet collapse under masking, and these confident-but-fragile tokens carry error information that output-confidence measures miss. We instantiate this as ASMI (Attention-Subnetwork Mutual Information): for each Monte Carlo sample we mask attention heads at a target layer and quantify disagreement among the resulting predictive distributions via the BALD mutual-information decomposition (14; 39), with a semantic-agreement kernel to discount surface-form disagreement. The estimator is training-free and needs only the choice of target layer. Existing estimators can be characterized along two axes: their variation source (stochastic output sampling versus structural model perturbation) and their measurement target (output spread versus computational path dependence). ASMI occupies the structural-perturbation, path-dependence corner, and its position yields a testable hypothesis: the signal should be informative where correctness depends on attending to specific context tokens, as in retrieval-grounded QA, and end where uncertainty originates elsewhere, as in parametric knowledge recall (26; 28; 54). We test this behaviorally, with grounded against parametric QA as a designed contrast, and internally, asking whether the signal adds error information beyond output confidence and how its head-level signature tracks the boundary. The result is an estimator that states in advance where it should work and where it should fail, with both directions confirmed. Our contributions are: • We propose ASMI, a training-free token-level uncertainty estimator that probes attention-path fragility by random head masking with a semantic-agreement kernel, and reads it from a single greedy response. In this sampling-free form it is competitive with the strongest sample-diversity baselines, which each draw ten stochastic generations, and its estimates are near-deterministic across reruns. • We show the signal is distinct from output confidence, not a proxy for it: an out-of-fold test confirms it adds error-predictive information beyond single-pass confidence and entropy, concentrated in confident-but-fragile predictions. • The distinctness is regime-graded, so ASMI predicts its own domain of applicability: it leads on context-routed QA and reverts to or below the free MSP baseline on parametric recall, exactly where the design predicts. A head-level analysis shows that what tracks this boundary is whether fragility couples to errors, present on grounded QA and absent under parametric recall, and a near-zero masking response identifies the one backbone where the signal carries no information, a candidate label-free screen whose validation here is retrospective. Figure 2: Pipeline of ASMI. The prefix below the target layer ℓ⋆ is computed once, and S independently masked copies of the suffix score the same greedy output (three shown). Filled and open circles denote kept and masked heads. Disagreement among the resulting token distributions yields MItMI_t (Eq. 5). The semantic kernel of Sem-ASMI and the adaptive gate of Adapt-ASMI are omitted for clarity. Related Work Sample-diversity UE. Sample-diversity methods estimate uncertainty from the variability of stochastically generated outputs. SelfCheckGPT (29) uses pairwise consistency among sampled generations as a hallucination signal. Semantic Entropy (25; 7) clusters semantically equivalent generations and takes entropy over the clusters, Kernel Language Entropy (33) relaxes this to a soft semantic kernel, SAR (5) weights samples by relevance to the predicted answer, and LUQ (60) extends the approach to long-form text. Single-pass UE. Information-based methods such as Maximum Sequence Probability, Perplexity, and Mean Token Entropy (27) read uncertainty directly from the output distribution of a single deterministic forward pass. Probing methods examine internal model states: SAPLMA (1), INSIDE (2), and Semantic Entropy Probes (24) extract signals from hidden representations, while P(True) (20) prompts the model for self-assessment. Attention-based methods such as RAUQ (47) read uncertainty from attention scores at heads identified as uncertainty-aware by prior analysis. Structural perturbation UE. MC Dropout (8) established test-time structural perturbation as an uncertainty signal, but its variational interpretation does not transfer directly to modern LLMs trained without dropout, and adaptations such as TokUR (61) inject noise into the attention weight matrices and read a posterior over parameters. ASMI perturbs differently: we apply discrete Bernoulli masks to attention head outputs to probe path dependence among redundant heads (32; 48), surfacing the model’s reliance on specific attention paths rather than posterior uncertainty over parameters. Adaptive UE. Other adaptive UE approaches such as Cocoa (46) and GENUINE (49) combine multiple uncertainty signals via calibration or graph-based fusion. Our adaptive variant, Adapt-ASMI, instead modulates a single signal by a directly observable input property, the diversity of stochastically decoded outputs. A two-axis characterization. Placing the families above on those two axes leaves one corner empty. Sample-diversity methods measure output spread, information-based and probing methods read outputs or hidden states without perturbing, RAUQ and Lookback Lens (3) observe attention passively, and MC Dropout perturbs the model but still measures output spread. ASMI is, to our knowledge, the first estimator in the structural-perturbation, path-dependence corner, and that position is predictive rather than merely descriptive. It should be informative when the prediction depends on routing context through specific attention paths, a computation carried by a sparse set of heads that causally transport in-context information (34; 52; 42) and can individually flip the answer between its in-context and memorized source (57; 18; 36). That advantage should end where uncertainty originates in parametric recall, whose locus is MLP-stored knowledge rather than the attention paths that extract it (11; 4; 31; 10), a dissociation already exploited to detect RAG hallucination (40). The hypothesis is falsifiable in both directions: it fails if the signal survives closed-book recall or if matched perturbations of non-attention components at the same layer reproduce it. Methodology Attention subnetwork sampling. Let fθf_θ be a pretrained autoregressive language model with fixed parameters θ. For an input context (x,y<t)(x,y_<t), the model defines a next-token distribution pθ(yt∣x,y<t)p_θ(y_t x,y_<t). To estimate structural uncertainty from internal computation rather than decoding noise, we perturb the model at inference time by masking attention heads in a selected Transformer layer ℓ⋆ . For a multi-head attention module with H heads, MHA(Q,K,V)=Concat(head1,…,headH)WO,MHA(Q,K,V)=Concat(head_1,…,head_H)\,W_O, (1) where headh=softmax(QhKh⊤dh)Vh.head_h=softmax\! ( Q_hK_h d_h )V_h. (2) Because many heads can be pruned without significant loss (32; 48; 13), we use random head dropout in reverse, as a probe of how strongly a prediction depends on specific attention paths. For each Monte Carlo sample s, we draw a binary mask m(s)∈0,1Hm^(s)∈\0,1\^H with mh(s)∼Bernoulli(1−p),m^(s)_h (1-p), (3) and apply it to the head outputs before the output projection: MHA^(s)(Q,K,V)=Concat(head^1(s),…,head^H(s))WO. MHA^(s)(Q,K,V)=Concat\! ( head^(s)_1,…, head^(s)_H )W_O. (4) No 1/(1−p)1/(1-p) rescaling is applied, following the ablation convention of the head-pruning literature (32; 48), and the scale reduction is common to all samples, so it does not itself generate the inter-sample disagreement that MItMI_t measures. Because only the aggregated head outputs are masked, the attention routing remains fixed and only each head’s contribution to the representation is perturbed. We set (p,S)=(0.15,40)(p,S)=(0.15,40), derived from a coverage lower bound on S and a fidelity upper bound on p (Appendix G). Efficient Monte Carlo inference. Because the perturbation is applied only at layer ℓ⋆ , the computation below that layer is shared across all S subnetworks (Fig. 2). Writing the model as fθ=fθsuf∘fθpref_θ=f^suf_θ f^pre_θ, where fθpref^pre_θ contains layers 1,…,ℓ⋆−11,…, -1 and fθsuff^suf_θ contains layers ℓ⋆,…,L ,…,L, we compute the prefix activations hcache=fθpre(x,y)h^cache=f^pre_θ(x,y) once and evaluate only the suffix under each mask m(s)m^(s). This reduces the Monte Carlo cost from S(Cpre+Csuf)S(C_pre+C_suf) to Cpre+SCsufC_pre+S\,C_suf FLOPs, where CpreC_pre and CsufC_suf denote prefix and suffix costs. Since y is fixed prior to scoring, each masked pass is a single teacher-forcing evaluation, and the ensemble is defined implicitly by the binary masks: no parameter copies are materialized, and the only persistent overhead is the shared cache hcacheh^cache (measured overhead in Appendix A). Mutual-information uncertainty. Let pt(s)(⋅)p^(s)_t(·) denote the predictive distribution at position t under the s-th sampled attention subnetwork, and let S be the number of Monte Carlo samples. We quantify token-level disagreement by the mutual information between the sampled subnetwork and the next-token prediction (14): MIt=H(p¯t)−1S∑s=1SH(pt(s)),p¯t=1S∑s=1Spt(s),MI_t=H( p_t)- 1S _s=1^SH\! (p^(s)_t ), p_t= 1S _s=1^Sp^(s)_t, (5) where H(p)=−∑ipilogpiH(p)=- _ip_i p_i. This quantity becomes large when different attention subnetworks make confident but inconsistent predictions, and therefore captures structural disagreement induced by internal attention perturbations rather than decoding stochasticity. Exact MItMI_t requires entropies over the full vocabulary (|V|>150K|V|>150K), prohibitive across S samples and all generated positions, so we approximate each pt(s)p^(s)_t by its top-K candidates plus a single tail bucket holding the residual mass. On retrieval-grounded benchmarks top-K=64K=64 retains over 95% of probability mass even at the most uncertain high-MI tokens (Appendix B). For sample s, let ℐt,K(s)I^(s)_t,K be the top-K token indices at position t, where each retained probability is computed using the full-vocabulary normalization constant: pt(s)(i)=exp(ℓt,i(s))∑j=1Vexp(ℓt,j(s)).p^(s)_t(i)= ( ^(s)_t,i) _j=1^V ( ^(s)_t,j). (6) We then form the union support t=⋃s=1Sℐt,K(s),U_t= _s=1^SI^(s)_t,K, (7) and construct a reduced distribution p~t(s) p^(s)_t over t∪tailU_t∪\tail\: p~t(s)(i)=pt(s)(i),i∈t,1−∑j∈tpt(s)(j),i=tail. p^(s)_t(i)= casesp^(s)_t(i),&i _t,\\ 1- _j _tp^(s)_t(j),&i=tail. cases (8) We use these reduced distributions to approximate MItMI_t. Eq. 5 admits the equivalent form MIt=1S∑s=1SKL(pt(s)∥p¯t),MI_t\;=\; 1S _s=1^SKL\! (p^(s)_t\, \|\, p_t ), (9) i.e., the average divergence of each subnetwork’s prediction from the ensemble mean. This form makes MItMI_t decomposable over the masked heads. Algorithm 1 Sem-ASMI Uncertainty Estimation 0: Input x; model fθf_θ; layer ℓ⋆ ; samples S; mask rate p; truncation K 0: Sequence uncertainty U(x,y)U(x,y) 1: y←GreedyDecode(fθ,x)y (f_θ,x), T←|y|T←|y| 2: hcache←fθpre(x,y)h^cache← f^pre_θ(x,y) 3: for s=1s=1 to S do 4: Sample mask m(s)∈0,1Hm^(s)∈\0,1\^H, mh(s)∼Bernoulli(1−p)m^(s)_h (1-p) 5: pt(s)t=1T←fθsuf(hcache,m(s))\p^(s)_t\_t=1^T← f^suf_θ(h^cache;m^(s)) at positions of y 6: end for 7: Form top-K union support and tail-bucket distributions p~t(s)\ p^(s)_t\ 8: Compute MIt,At\MI_t\,\A_t\ from p~t(s)\ p^(s)_t\ via Eq. 5, 11 9: return U←1T∑t=1TMIt(1−At)U← 1T _t=1^TMI_t\,(1-A_t) Semantic agreement weighting. Different subnetworks may assign high probability to semantically similar alternatives (e.g., “sleep” and “sleeping”), disagreement that is not uncertainty. To discount it, we compute a semantic agreement score on the top-K candidate distributions (Fig. 3). For MC samples m and n, let pt(m),pt(n)∈ℝKp^(m)_t,\,p^(n)_t ^K be their top-K probability vectors, renormalized to sum to one over the retained support, and let Gt(m,n)∈ℝK×KG^(m,n)_t ^K× K be the pairwise token similarity matrix with entries equal to 11 for identical tokens and min(1,max(0,cos(ei,ej))) (1, (0, (e_i,e_j))) otherwise, computed over L2-normalized rows of the output projection matrix WlmW_lm, so that At∈[0,1]A_t∈[0,1] by construction. We define pairwise agreement as At(m,n)=(pt(m))⊤Gt(m,n)pt(n),A^(m,n)_t= (p^(m)_t ) G^(m,n)_t\,p^(n)_t, (10) and average it across ordered sample pairs: At=1S(S−1)∑m≠nAt(m,n).A_t= 1S(S-1) _m≠ nA^(m,n)_t. (11) The semantically-weighted token-level score is utsem=MIt(1−At),u^sem_t=MI_t\,(1-A_t), (12) while the unweighted base variant, which we denote ASMI and use as an ablation, retains the raw utnosem=MItu^nosem_t=MI_t. The sequence-level uncertainty is the mean over generated token positions: U(x,y)=1T∑t=1Tutsem.U(x,y)= 1T _t=1^Tu^sem_t. (13) Equations 12 and 13 define Sem-ASMI. Figure 3: Token-level mechanism on CoQA. (a) Joint distribution of MItMI_t and AtA_t (Pearson r=−0.57r=-0.57). High MItMI_t with low agreement is flagged, while high MItMI_t with high agreement is benign instability that the kernel discounts. (b) Pairwise WlmW_lm cosine similarity among the top-8 candidates at a representative benign-instability token, the surface-form cluster that drives AtA_t up. Adaptive semantic weighting. Semantic weighting helps when valid answers share a canonical form, but risks discarding genuine uncertainty when many surface forms are valid. Adapt-ASMI therefore gates it per input by the diversity of N sampled responses, each embedded by the model’s own hidden state at layer ⌊L/2⌋ L/2 at its final token. With sample embeddings e1,…,eNe_1,…,e_N, div(x)=1−(N2)−1∑i<jcos(ei,ej),div(x)=1- N2^-1 _i<j (e_i,e_j), (14) and the gate is α(x)=σ(β(τ−div(x)))α(x)=σ (β(τ-div(x)) ) with logistic σ, so that α→1α→ 1 when the sampled responses are near-paraphrases (div(x)<τdiv(x)<τ) and α→0α→ 0 when they genuinely diverge. The token-level score becomes utadapt=MIt(1−α(x)At),u^adapt_t=MI_t (1-α(x)\,A_t ), (15) recovering Eq. 12 as α→1α→ 1 (low diversity) and the raw MItMI_t as α→0α→ 0, with (τ,β)=(0.3,10)(τ,β)=(0.3,10) fixed across all tasks and backbones. UE Method Qwen3-4B Qwen3-8B Llama-2-7B Mistral-7B CoQA SQuAD BabiQA TriviaQA CoQA SQuAD BabiQA TriviaQA CoQA SQuAD BabiQA TriviaQA CoQA SQuAD BabiQA TriviaQA Baselines Maximum Sequence Probability 0.45 0.32 0.48 0.72 0.53 0.35 0.76 0.70 0.53 0.18 0.58 0.77 0.52 0.65 0.59 0.78 Mean Token Entropy 0.40 0.27 0.56 0.70 0.50 0.41 0.75 0.69 0.46 0.68 0.30 0.75 0.44 0.78 0.54 0.78 Semantic Entropy 0.46 0.28 0.54 0.72 0.53 0.45 0.74 0.70 0.55 0.47 0.50 0.77 0.54 0.80 0.58 0.78 SAR 0.46 0.27 0.63 0.71 0.47 0.40 0.71 0.68 0.53 0.51 0.49 0.78 0.52 0.78 0.56 0.80 P(True) 0.03 0.07 0.49 0.30 0.07 0.17 0.22 0.39 -0.04 0.47 0.21 0.06 -0.05 0.78 0.20 -0.11 RAUQ 0.37 0.32 0.32 0.70 0.48 0.45 0.74 0.66 0.47 0.46 0.57 0.75 0.46 0.82 0.48 0.70 RAUQ (entropy) 0.47 0.36 0.42 0.72 0.53 0.47 0.74 0.70 0.50 0.75 0.29 0.77 0.51 0.83 0.46 0.77 Attention-based, active perturbation (ours) ASMI (60%) 0.46 0.29 0.60 0.59 0.51 0.34 0.80 0.62 0.49 0.62 0.11 0.66 0.48 0.77 0.52 0.74 Sem-ASMI (60%) 0.52 0.29 0.66 0.66 0.55 0.40 0.78 0.66 0.54 0.69 0.27 0.71 0.53 0.83 0.61 0.78 Adapt-ASMI (60%) 0.53 0.31 0.66 0.66 0.56 0.40 0.79 0.67 0.55 0.71 0.29 0.72 0.54 0.84 0.61 0.78 Table 1: Representative UE methods on four backbones (PRR, higher is better, ASMI variants at depth d=60%d=60\%). TriviaQA is the closed-book parametric control, and the other three benchmarks are context-routed. Bold and underline mark best and second best among the methods shown. Standard errors are ±0.01± 0.01 to ±0.03± 0.03 (±0.03± 0.03 to ±0.07± 0.07 on BabiQA), with per-cell values and all 17 baselines in Appendix I. Experiments Dataset n Input Gen AlignScore CoQA ≈8K≈ 8K 490490 44 0.7820.782 SQuAD 5K5K 183183 1818 0.8690.869 BabiQA 1K1K 175175 33 0.7090.709 TriviaQA 5K5K 176176 44 0.4880.488 Table 2: Dataset characteristics on Qwen3-4B-base. Median input and generation lengths in tokens, and mean AlignScore of the greedy answers. The three grounded datasets sit above the closed-book parametric control. Datasets and quality metric. Our benchmark is a two-condition contrast dictated by the method’s design. The context-routed condition uses CoQA (38), SQuAD (37), and BabiQA (50) (Table 2). As a negative control, the parametric condition uses closed-book TriviaQA (19) with no evidence documents, so answers must come from parametric memory (28). All datasets follow the LM-Polygraph (45) pipeline with its default prompting and standard splits, scored by AlignScore (59), an entailment-based metric that tracks faithfulness better than surface-overlap scores (30). To bound compute, SQuAD and TriviaQA use a fixed random 55K subsample, identical across methods and backbones. Baselines. We compare against 17 baseline UE methods spanning four families. Information-based: MSP, Perplexity, Mean Token Entropy (27), PMI (41), Conditional PMI (44), SelfCertainty (23). Sample-diversity: Monte Carlo and MC-normalized sequence entropy (27), Semantic Entropy (25; 7), SAR, SentenceSAR (5), LUQ (60), Kernel Language Entropy (33). Probing: P(True) and its sampling variant (20). Attention-based: RAUQ and its entropy variant (47). All run within LM-Polygraph with default hyperparameters. The main table shows a representative subset, and the full comparison with standard errors is in Appendix I. Implementation details. Our primary backbone is Qwen3-4B-base (56), with Qwen3-8B-base, Llama-2-7B (43), and Mistral-7B (16) for generality, all with H=32H=32 query heads. We evaluate base models deliberately, since instruction tuning reshapes attention-head specialization (53) and would confound the head-level analysis. We fix a single operating point (p,S)=(0.15,40)(p,S)=(0.15,40) with top-K truncation K=64K=64 across all tasks and backbones (Appendix G). The estimator’s only hyperparameter is the masked layer, a relative depth d that transfers across architectures. Because the context-routing computation ASMI probes concentrates in the middle to later-middle layers (10; 52), we sweep d∈60,70,80,90%d∈\60,70,80,90\\% identically everywhere and take d=60%d=60\% as representative by aggregate PRR, which needs no per-task tuning (Appendix I). Responses are greedy-decoded and scored under each mask. Adapt-ASMI’s gate reuses the N=10N=10 sampled responses the protocol already generates for the sampling baselines, whereas ASMI and Sem-ASMI need only the greedy response. All methods are evaluated by the Prediction Rejection Ratio (PRR) (27) within LM-Polygraph (6), the normalized area between the uncertainty-based and random rejection curves, taking the value 11 for oracle rejection and 00 for an uninformative score (Appendix A). Wall-clock s/example (median) CoQA SQuAD MSP 4.144.14 0.750.75 Semantic Entropy (10 samples + NLI) 8.738.73 2.162.16 Sem-ASMI (greedy + 40 masked passes) 6.356.35 2.482.48 Adapt-ASMI (adds the 10 samples) 10.7710.77 3.763.76 Sem-ASMI, marginal given the samples 2.212.21 1.741.74 Table 3: End-to-end cost on Qwen3-4B-base (200 examples, one H100 MIG slice, mask-batched Sem-ASMI). Full breakdown in Appendix H. Main results. Table 1 is best read together with measured cost (Table 3). Sem-ASMI needs only the greedy response, and in that sampling-free form it is competitive on grounded QA with the strongest sample-diversity baselines, which each draw ten stochastic generations per input. Measured cost tracks input length and therefore favors the long-context inputs typical of context-routed QA: on CoQA the mask-batched implementation takes 27%27\% less wall-clock end to end than Semantic Entropy, on the shorter SQuAD inputs it takes 15%15\% more, and once the ten samples exist the marginal cost of the score itself is 2.22.2 and 1.71.7 seconds. Under a cluster-respecting paired bootstrap the best ASMI variant ties or leads the per-column best baseline in eight of the twelve grounded columns, significantly in three, and in every remaining lead the sampling-free Sem-ASMI ties or beats Semantic Entropy from the greedy response alone (Appendix J). The same test finds four significant deficits, confined to SQuAD, where three of the four backbones trail the per-column best baseline, and to the one mapped exception cell below. On parametric recall all variants revert to or below the free MSP baseline, and the paired test places the top variant significantly below the per-column best baseline in three of the four columns, so the same table draws the boundary it was built to test. The three ASMI rows isolate the ablation: the semantic kernel lifts Sem-ASMI over the unweighted ASMI by 0.040.04 to 0.060.06 PRR on CoQA, where surface forms vary most, and the adaptive gate matches or exceeds it elsewhere. A further ablation shows the gain is not the averaging: the ensemble-mean entropy from the same forty masked passes is rank-identical to single-pass entropy (Spearman 0.970.97 to 0.9950.995), and the weighted score beats it significantly on both grounded analysis cells (Appendix D). On CoQA the semantically weighted variants, Sem- and Adapt-ASMI, lead both RAUQ and its entropy variant on all four backbones, and the gap between those two RAUQ variants shows that RAUQ’s competitiveness comes from its entropy term rather than from attention, so the attention signal ASMI reads is not already captured by the strongest attention baseline. ASMI is also the more reproducible of the two families: redrawing only internal randomness moves Adapt-ASMI’s PRR by ±0.006± 0.006 against ±0.014± 0.014 to 0.0410.041 for the sample-diversity competitors (Appendix K). Absolute PRR is not comparable across columns, so all comparisons are within a column. Figure 4: The confident-but-fragile signal. (a) BabiQA error rate by MSP confidence and usemu^sem fragility: confident-but-fragile predictions fail 31%31\% against 10%10\% for confident-robust. (b) The effect is strong on grounded QA and fades to chance under parametric recall. A distinct signal, not a proxy for output confidence. Aggregate PRR shows ASMI ranks errors as well as the strongest single-pass baselines, but not that it reads a different signal. Because token-level MI and single-pass entropy are strongly correlated (Spearman around 0.90.9 on CoQA), we test distinctness at the source rather than the aggregate. On the greedy response we regress the semantically-weighted score usemu^sem on MSP and single-pass entropy out of fold and ask whether its residual still predicts errors. It does. The residual detects errors above chance on every grounded dataset, in five of the six backbone-dataset cells at AUROCs of 0.540.54 to 0.600.60, one of them at the estimator’s own Monte Carlo resolution, so the information usemu^sem adds is not contained in output confidence or entropy (Table 6), consistent with theoretical and probing evidence that output confidence leaves error information unread (55; 58; 35). The effect is localized: splitting sequences at the medians of MSP and usemu^sem, the confident-but-fragile cell is wrong three times as often as the confident-robust cell, on BabiQA 31%31\% against 10%10\%, a gap MSP alone cannot see (Fig. 4). The signal is regime-graded exactly as the design predicts: the raw gap between fragile and robust cells also appears on parametric TriviaQA, but there it is entirely absorbed by single-pass confidence and entropy with the residual at chance on both backbones (0.490.49 and 0.510.51), whereas on grounded QA a residual beyond both remains (Table 6). A separate check on real grounded QA shows the semantic kernel improving the ranking by discounting surface-form variation rather than adding information orthogonal to the masking disagreement, so there the distinct signal is carried by the attention-masking disagreement itself (Appendix D). Figure 5: Deployment payoff in the confident stratum. Retained error among MSP-confident predictions as an increasing fraction of them is abstained by ASMI fragility (usemu^sem, blue) or by predictive entropy (black). On grounded QA the ASMI curve stays below entropy at every abstention level. Deployment payoff. In selective prediction (9; 22), the value of the signal concentrates on confident errors, the predictions a standard confidence filter keeps and acts on. Among the answers MSP rates as confident, abstaining the ASMI-fragile ones roughly halves the retained error on grounded QA, from 16.0%16.0\% to 5.6%5.6\% on BabiQA and from 13.6%13.6\% to 7.2%7.2\% on CoQA, against 9.2%9.2\% and 10.8%10.8\% for the same abstention budget spent on entropy, and the gap holds across the whole abstention range rather than at a single operating point (Table 4, Fig. 5). On parametric TriviaQA the order reverses, so the boundary holds in deployment too. As a standalone score ASMI also gives a lower area under the risk-coverage curve than MSP or entropy on the grounded benchmarks with the ordering reversing on TriviaQA as the boundary predicts (Appendix E). The gain is localized rather than global: added to a logistic selector over MSP and entropy fit across all coverage levels, ASMI leaves the overall risk-coverage curve unchanged, because its error information concentrates in the confident stratum and a single global fit averages that region together with the rest. A case-level look matches the mechanism. For confident BabiQA errors most masked subnetworks reproduce the confident wrong answer while a minority route to the true location, recovering garden where the greedy answer is bedroom, a disagreement the single pass and its entropy cannot see. Confident error Filter by ASMI Filter by entropy BabiQA 0.1600.160 0.0560.056 0.0920.092 CoQA 0.1360.136 0.0720.072 0.1080.108 TriviaQA 0.3440.344 0.1470.147 0.1330.133 Table 4: Retained error in the MSP-confident stratum after abstaining its top half by usemu^sem or by entropy (final coverage 0.250.25, Qwen3-4B-base). On grounded QA ASMI roughly halves the confident error, while on parametric TriviaQA entropy leads instead. Boundary: parametric-knowledge QA. TriviaQA is the designed negative control, and the boundary is sharp. On the same SQuAD examples with the passage removed, Sem-ASMI sits significantly below MSP (−0.054-0.054, 95% CI [−0.088,−0.021][-0.088,-0.021]) while the open-book condition is a statistical tie, so the pattern reproduces within a single dataset when only the knowledge source moves (Appendix C). The practical instruction is symmetric: use this sampling-free signal as a faithfulness check where answers are routed through provided context, and hand off to output-distribution baselines for closed-book recall. Operating envelope. Every remaining exception is mapped, not noise. All TriviaQA cells sit at or below MSP as the boundary predicts, and the one grounded exception has an identified cause. On BabiQA with Llama-2-7B head masking barely moves the output, so correct and incorrect answers carry the same near-zero MI and ASMI has no fragility to read while MSP still ranks the errors, an over-robustness that is a model property rather than an artifact (Table 12). The BabiQA depth instability is detailed in Appendix I. The cell announces itself before any correctness label exists: the same masking response that produces the score also reports when a model is too robust for it to carry information. We report this as a candidate label-free screen rather than a validated one, since its support is a single cell examined after the fact. Structural analysis. A head-level decomposition confirms that the fragility ASMI reads is head borne (Appendix F). At the operating layer a causal probe makes the dependence direct: ablating a token’s few most critical heads flips the prediction for high-MI tokens while leaving low-MI tokens almost unchanged. The concentration of this dependence couples positively to uncertainty on the grounded benchmarks and negatively on parametric recall, and mutual information tracks the head structure more than single-pass entropy does. Limitations & Future Work The BALD form of Eq. 5 admits an epistemic reading of MItMI_t under the variational view of inference-time dropout (8), but entropy-based decompositions of epistemic and aleatoric uncertainty face formal objections (51), so we use MItMI_t as a disagreement functional and claim no empirical separation of the two. Distinctness replicates across both backbones and all three grounded datasets, with two caveats: the CoQA residual on Qwen3-8B does not exclude chance, and the SQuAD effect on Qwen3-4B lies at the estimator’s own Monte Carlo resolution (Appendix D). On the parametric control the top-K truncation retains 77.3%77.3\% of the high-MI mass against over 95%95\% on grounded data, but re-scoring TriviaQA at K=256K=256 raises mean token coverage to 0.970.97 and leaves Sem-ASMI significantly below MSP, so the parametric-side deficit is not a truncation artifact (Appendix C). Extending the two-axis characterization to reasoning, open-ended generation, instruction-tuned models, and further parametric tasks is future work, for which our structural analysis provides the measurement tool. Conclusion We introduced attention-path fragility as an uncertainty signal, instantiated as ASMI, a training-free estimator that probes attention-pathway redundancy by random head masking and reads it from a single greedy response. Two correlated signals can look identical in aggregate, so we tested distinctness at the source: the semantically-weighted score adds error-predictive information beyond confidence and entropy, concentrated in confident-but-fragile predictions. Its value is local rather than global: a global selector cannot exploit it, yet in the confident stratum, where a deployed system acts, it roughly halves the error. The distinctness is regime-graded and confirmed by a designed contrast: from that single greedy pass ASMI is competitive with the strongest sampling-based baselines on context-routed QA, and reverts to or below the free MSP baseline on parametric recall, exactly as the hypothesis requires. A head-level analysis traces this boundary to whether fragility couples to errors, and a causal ablation confirms the fragility is carried by identifiable critical heads. The lesson outlasts the estimator: for a signal tied to a specific computation, where it applies and where its value lands follow from the locus of a model’s errors, and we offer ASMI as a case for treating a UE signal’s domain of applicability as a predictable property rather than a discovered one. References Azaria and Mitchell (2023) A. Azaria and T. Mitchell The internal state of an llm knows when it’s lying. In Findings of EMNLP, p. 967–976. Cited by: Introduction, Single-pass UE.. Chen et al. (2024) C. Chen, K. Liu, Z. Chen, et al. INSIDE: llms’ internal states retain the power of hallucination detection. arXiv preprint arXiv:2402.03744. Cited by: Single-pass UE.. Chuang et al. (2024) Y. Chuang, L. Qiu, C. Hsieh, et al. Lookback lens: detecting and mitigating contextual hallucinations in large language models using only attention maps. In Proceedings of EMNLP, p. 1419–1436. Cited by: A two-axis characterization.. Dai et al. (2022) D. Dai, L. Dong, Y. Hao, et al. Knowledge neurons in pretrained transformers. In Proceedings of ACL, p. 8493–8502. Cited by: Appendix F, A two-axis characterization.. Duan et al. (2024) J. Duan, H. Cheng, S. Wang, et al. Shifting attention to relevance: towards the predictive uncertainty quantification of free-form large language models. In Proceedings of ACL, p. 5050–5063. Cited by: Appendix A, Appendix K, Introduction, Sample-diversity UE., Baselines.. Fadeeva et al. (2023) E. Fadeeva, R. Vashurin, A. Tsvigun, et al. LM-polygraph: uncertainty estimation for language models. In Proceedings of EMNLP: System Demonstrations, p. 446–461. Cited by: Appendix A, Implementation details.. Farquhar et al. (2024) S. Farquhar, J. Kossen, L. Kuhn, and Y. Gal Detecting hallucinations in large language models using semantic entropy. Nature 630 (8017), p. 625–630. Cited by: Introduction, Sample-diversity UE., Baselines.. Gal and Ghahramani (2016) Y. Gal and Z. Ghahramani Dropout as a bayesian approximation: representing model uncertainty in deep learning. In Proceedings of ICML, p. 1050–1059. Cited by: Introduction, Structural perturbation UE., Limitations & Future Work. Geifman and El-Yaniv (2017) Y. Geifman and R. El-Yaniv Selective classification for deep neural networks. Advances in Neural Information Processing Systems 30. Cited by: Appendix E, Deployment payoff.. Geva et al. (2023) M. Geva, J. Bastings, K. Filippova, and A. Globerson Dissecting recall of factual associations in auto-regressive language models. In Proceedings of EMNLP, p. 12216–12235. Cited by: Appendix F, A two-axis characterization., Implementation details.. Geva et al. (2021) M. Geva, R. Schuster, J. Berant, and O. Levy Transformer feed-forward layers are key-value memories. In Proceedings of EMNLP, p. 5484–5495. Cited by: Appendix F, A two-axis characterization.. Guo et al. (2017) C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger On calibration of modern neural networks. In Proceedings of ICML, p. 1321–1330. Cited by: Introduction. He et al. (2024) S. He, G. Sun, Z. Shen, and A. Li What matters in transformers? not all attention is needed. arXiv preprint arXiv:2406.15786. Cited by: Introduction, Attention subnetwork sampling.. Houlsby et al. (2011) N. Houlsby, F. Huszár, Z. Ghahramani, and M. Lengyel Bayesian active learning for classification and preference learning. arXiv preprint arXiv:1112.5745. Cited by: Introduction, Mutual-information uncertainty.. Huang et al. (2025) L. Huang, W. Yu, W. Ma, et al. A survey on hallucination in large language models: principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems 43 (2), p. 1–55. Cited by: Introduction. Jiang et al. (2023) A. Q. Jiang, A. Sablayrolles, A. Mensch, et al. Mistral 7b. arXiv preprint arXiv:2310.06825. Cited by: Implementation details.. Jiang et al. (2021) Z. Jiang, J. Araki, H. Ding, and G. Neubig How can we know when language models know? on the calibration of language models for question answering. Transactions of the Association for Computational Linguistics 9, p. 962–977. Cited by: Introduction. Jin et al. (2024) Z. Jin, P. Cao, H. Yuan, et al. Cutting off the head ends the conflict: a mechanism for interpreting and mitigating knowledge conflicts in language models. In Findings of ACL, p. 1193–1215. Cited by: A two-axis characterization.. Joshi et al. (2017) M. Joshi, E. Choi, D. S. Weld, and L. Zettlemoyer Triviaqa: a large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of ACL, p. 1601–1611. Cited by: Datasets and quality metric.. Kadavath et al. (2022) S. Kadavath, T. Conerly, A. Askell, et al. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221. Cited by: Introduction, Single-pass UE., Baselines.. Kalai et al. (2025) A. T. Kalai, O. Nachum, S. S. Vempala, and E. Zhang Why language models hallucinate. arXiv preprint arXiv:2509.04664. Cited by: Introduction. Kamath et al. (2020) A. Kamath, R. Jia, and P. Liang Selective question answering under domain shift. In Proceedings of ACL, p. 5684–5696. Cited by: Deployment payoff.. Kang et al. (2025) Z. Kang, X. Zhao, and D. Song Scalable best-of-n selection for large language models via self-certainty. arXiv preprint arXiv:2502.18581. Cited by: Baselines.. Kossen et al. (2024) J. Kossen, J. Han, M. Razzak, et al. Semantic entropy probes: robust and cheap hallucination detection in llms. arXiv preprint arXiv:2406.15927. Cited by: Single-pass UE.. Kuhn et al. (2023) L. Kuhn, Y. Gal, and S. Farquhar Semantic uncertainty: linguistic invariances for uncertainty estimation in natural language generation. arXiv preprint arXiv:2302.09664. Cited by: Introduction, Sample-diversity UE., Baselines.. Longpre et al. (2021) S. Longpre, K. Perisetla, A. Chen, et al. Entity-based knowledge conflicts in question answering. In Proceedings of EMNLP, p. 7052–7063. Cited by: Introduction. Malinin and Gales (2020) A. Malinin and M. Gales Uncertainty estimation in autoregressive structured prediction. arXiv preprint arXiv:2002.07650. Cited by: Appendix A, Introduction, Single-pass UE., Baselines., Implementation details.. Mallen et al. (2023) A. Mallen, A. Asai, V. Zhong, et al. When not to trust language models: investigating effectiveness of parametric and non-parametric memories. In Proceedings of ACL, p. 9802–9822. Cited by: Introduction, Datasets and quality metric.. Manakul et al. (2023) P. Manakul, A. Liusie, and M. Gales Selfcheckgpt: zero-resource black-box hallucination detection for generative large language models. In Proceedings of EMNLP, p. 9004–9017. Cited by: Introduction, Sample-diversity UE.. Maynez et al. (2020) J. Maynez, S. Narayan, B. Bohnet, and R. McDonald On faithfulness and factuality in abstractive summarization. In Proceedings of ACL, p. 1906–1919. Cited by: Datasets and quality metric.. Meng et al. (2022) K. Meng, D. Bau, A. Andonian, and Y. Belinkov Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems 35, p. 17359–17372. Cited by: Appendix F, A two-axis characterization.. Michel et al. (2019) P. Michel, O. Levy, and G. Neubig Are sixteen heads really better than one?. Advances in Neural Information Processing Systems 32. Cited by: Appendix G, Introduction, Structural perturbation UE., Attention subnetwork sampling., Attention subnetwork sampling.. Nikitin et al. (2024) A. Nikitin, J. Kossen, Y. Gal, and P. Marttinen Kernel language entropy: fine-grained uncertainty quantification for llms from semantic similarities. Advances in Neural Information Processing Systems 37, p. 8901–8929. Cited by: Introduction, Sample-diversity UE., Baselines.. Olsson et al. (2022) C. Olsson, N. Elhage, N. Nanda, et al. In-context learning and induction heads. arXiv preprint arXiv:2209.11895. Cited by: A two-axis characterization.. Orgad et al. (2024) H. Orgad, M. Toker, Z. Gekhman, et al. LLMs know more than they show: on the intrinsic representation of llm hallucinations. arXiv preprint arXiv:2410.02707. Cited by: A distinct signal, not a proxy for output confidence.. Ortu et al. (2024) F. Ortu, Z. Jin, D. Doimo, M. Sachan, A. Cazzaniga, and B. Schölkopf Competition of mechanisms: tracing how language models handle facts and counterfactuals. In Proceedings of ACL, p. 8420–8436. Cited by: A two-axis characterization.. Rajpurkar et al. (2016) P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang SQuAD: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250. Cited by: Datasets and quality metric.. Reddy et al. (2019) S. Reddy, D. Chen, and C. D. Manning Coqa: a conversational question answering challenge. Transactions of the Association for Computational Linguistics 7, p. 249–266. Cited by: Datasets and quality metric.. Smith and Gal (2018) L. Smith and Y. Gal Understanding measures of uncertainty for adversarial example detection. In Proceedings of UAI, p. 560–569. Cited by: Introduction. Sun et al. (2025) Z. Sun, X. Zang, K. Zheng, et al. ReDeEP: detecting hallucination in retrieval-augmented generation via mechanistic interpretability. In Proceedings of ICLR, Cited by: A two-axis characterization.. Takayama and Arase (2019) J. Takayama and Y. Arase Relevant and informative response generation using pointwise mutual information. In Proceedings of the First Workshop on NLP for Conversational AI, p. 133–138. Cited by: Baselines.. Todd et al. (2024) E. Todd, M. Li, A. Sen Sharma, A. Mueller, B. Wallace, and D. Bau Function vectors in large language models. In Proceedings of ICLR, Cited by: A two-axis characterization.. Touvron et al. (2023) H. Touvron, L. Martin, K. Stone, et al. Llama 2: open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. Cited by: Implementation details.. van der Poel et al. (2022) L. van der Poel, R. Cotterell, and C. Meister Mutual information alleviates hallucinations in abstractive summarization. In Proceedings of EMNLP, p. 5956–5965. Cited by: Baselines.. Vashurin et al. (2025a) R. Vashurin, E. Fadeeva, A. Vazhentsev, et al. Benchmarking uncertainty quantification methods for large language models with lm-polygraph. Transactions of the Association for Computational Linguistics 13, p. 220–248. Cited by: Datasets and quality metric.. Vashurin et al. (2025b) R. Vashurin, M. Goloburda, P. Nakov, A. Shelmanov, and M. Panov CoCoA: a generalized approach to uncertainty quantification by integrating confidence and consistency of LLM outputs. arXiv preprint arXiv:2502.04964. Cited by: Adaptive UE.. Vazhentsev et al. (2025) A. Vazhentsev, L. Rvanova, G. Kuzmin, et al. Uncertainty-aware attention heads: efficient unsupervised uncertainty quantification for llms. arXiv preprint arXiv:2505.20045. Cited by: Introduction, Introduction, Single-pass UE., Baselines.. Voita et al. (2019) E. Voita, D. Talbot, F. Moiseev, R. Sennrich, and I. Titov Analyzing multi-head self-attention: specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of ACL, p. 5797–5808. Cited by: Appendix G, Introduction, Structural perturbation UE., Attention subnetwork sampling., Attention subnetwork sampling.. Wang et al. (2025) T. Wang, A. Kulkarni, T. Cody, et al. GENUINE: graph enhanced multi-level uncertainty estimation for large language models. In Findings of EMNLP, p. 20522–20541. Cited by: Adaptive UE.. Weston et al. (2015) J. Weston, A. Bordes, S. Chopra, et al. Towards ai-complete question answering: a set of prerequisite toy tasks. arXiv preprint arXiv:1502.05698. Cited by: Datasets and quality metric.. Wimmer et al. (2023) L. Wimmer, Y. Sale, P. Hofman, B. Bischl, and E. Hüllermeier Quantifying aleatoric and epistemic uncertainty in machine learning: are conditional entropy and mutual information appropriate measures?. In Proceedings of UAI, p. 2282–2292. Cited by: Limitations & Future Work. Wu et al. (2025) W. Wu, Y. Wang, G. Xiao, H. Peng, and Y. Fu Retrieval head mechanistically explains long-context factuality. In Proceedings of ICLR, Cited by: A two-axis characterization., Implementation details.. Wu et al. (2024) X. Wu, W. Yao, J. Chen, et al. From language modeling to instruction following: understanding the behavior shift in llms after instruction tuning. In Proceedings of NAACL, p. 2341–2369. Cited by: Implementation details.. Xu et al. (2024) R. Xu, Z. Qi, Z. Guo, et al. Knowledge conflicts for llms: a survey. In Proceedings of EMNLP, p. 8541–8565. Cited by: Introduction. Yadkori et al. (2024) Y. A. Yadkori, I. Kuzborskij, A. György, and C. Szepesvari To believe or not to believe your llm: iterative prompting for estimating epistemic uncertainty. Advances in Neural Information Processing Systems 37, p. 58077–58117. Cited by: A distinct signal, not a proxy for output confidence.. Yang et al. (2025) A. Yang, A. Li, B. Yang, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: Implementation details.. Yu et al. (2023) Q. Yu, J. Merullo, and E. Pavlick Characterizing mechanisms for factual recall in language models. In Proceedings of EMNLP, p. 9924–9959. Cited by: A two-axis characterization.. Yuksekgonul et al. (2024) M. Yuksekgonul, V. Chandrasekaran, E. Jones, et al. Attention satisfies: a constraint-satisfaction lens on factual errors of language models. In Proceedings of ICLR, Cited by: A distinct signal, not a proxy for output confidence.. Zha et al. (2023) Y. Zha, Y. Yang, R. Li, and Z. Hu AlignScore: evaluating factual consistency with a unified alignment function. In Proceedings of ACL, p. 11328–11348. Cited by: Appendix A, Datasets and quality metric.. Zhang et al. (2024) C. Zhang, F. Liu, M. Basaldella, and N. Collier Luq: long-text uncertainty quantification for llms. In Proceedings of EMNLP, p. 5244–5262. Cited by: Sample-diversity UE., Baselines.. Zhang et al. (2025) T. Zhang, H. Shi, Y. Wang, et al. TokUR: token-level uncertainty estimation for large language model reasoning. arXiv preprint arXiv:2505.11737. Cited by: Introduction, Structural perturbation UE.. Zhou et al. (2025) Z. Zhou, H. Yu, X. Zhang, et al. On the role of attention heads in large language model safety. In Proceedings of ICLR, Cited by: Introduction. Appendix A Implementation Details Compute and memory overhead. With prefix caching, the scoring cost of ASMI is Cpre+SCsufC_pre+S\,C_suf rather than S(Cpre+Csuf)S(C_pre+C_suf), where the prefix below the target layer is computed once and the suffix is recomputed under each of the S masks. With S=40S=40 the resulting overhead depends only on the target depth. On the 36-layer Qwen backbones the masked layer sits at ℓ⋆=−14 =-14, so a 22-layer prefix is computed once and a 14-layer suffix is scored 40 times, giving (22+40×14)/36≈16×(22+40× 14)/36≈ 16× the FLOPs of a single unmasked scoring pass. On the 32-layer Llama-2-7B and Mistral-7B backbones the masked layer sits at ℓ⋆=−13 =-13, giving (19+40×13)/32≈17×(19+40× 13)/32≈ 17×. Memory overhead is limited to the shared prefix cache hcacheh^cache, since each subnetwork is defined implicitly by its binary mask m(s)m^(s) and no parameter copies are materialized. Evaluation protocol. The Prediction Rejection Ratio (PRR) (27) is the normalized area between the uncertainty-based rejection curve and the random-rejection baseline, taking values in [−1,1][-1,1], where 11 corresponds to oracle rejection (rejecting examples in order of decreasing error) and 00 to uninformative uncertainty. We compute PRR with the LM-Polygraph (6) implementation, using AlignScore (59) as the sequence-level quality metric throughout. PRR uses the continuous AlignScore, and the binary analyses (error rates, AUROC, and the confident-stratum filters) binarize correctness at 0.50.5. Standard errors for all reported PRR values are estimated by bootstrap resampling over test examples. Baseline configuration. All baseline UE methods are run within LM-Polygraph with their default hyperparameters, except that all stochastic sampling uses a fixed temperature of 0.50.5 with 1010 generations, following the sampling protocol of SAR (5) and identical across methods and backbones. Greedy-decoded responses used for scoring are shared across all methods on each benchmark, so quality-metric values are identical across estimators and PRR differences reflect only the uncertainty rankings. Appendix B Top-K Coverage Analysis This appendix justifies the top-K approximation used in computing the mutual information MItMI_t defined in the Methodology section. We chose K=64K=64 across all experiments based on the following analysis. Setup. For each benchmark and each Monte Carlo sample s∈1,…,Ss∈\1,…,S\, we collect the per-token predictive distributions pt(s)p^(s)_t produced by Qwen3-4B-base under the sampled attention-head mask m(s)m^(s) at the target layer ℓ⋆ . At each token position we compute the cumulative probability mass covered by the top-K candidates, Coveraget(s)(K)=∑i∈top-K(t,s)pt(s)(i),Coverage^(s)_t(K)= _i -K(t,s)p^(s)_t(i), (16) where top-K(t,s)top-K(t,s) denotes the top-K indices of pt(s)p^(s)_t, and average over all generated tokens and all S=40S=40 samples. MI stratification. We stratify tokens by their MItMI_t value into three 10% bands: low-MI (bottom decile), mid-MI (the central band, P45P_45 to P55P_55), and high-MI (top decile). The high-MI band is the stringent case, since these tokens have the most spread subnetwork distributions. Benchmark n K=8K=8 1616 3232 6464 128128 CoQA 567567 0.8510.851 0.9030.903 0.9350.935 0.9550.955 0.9690.969 SQuAD 14801480 0.9550.955 0.9720.972 0.9820.982 0.9870.987 0.9910.991 BabiQA 216216 0.9500.950 0.9660.966 0.9750.975 0.9810.981 0.9860.986 TriviaQA 397397 0.5560.556 0.6330.633 0.7050.705 0.7730.773 0.8330.833 Table 5: Cumulative top-K coverage on the high-MI stratum, the top 10% of tokens by MItMI_t. The three grounded benchmarks are well covered at K=64K=64, whereas TriviaQA is not. Coverage results. Low-MI tokens are trivially covered, with K=1K=1 alone above 99% of mass on every benchmark, confirming that confident predictions are sharply peaked. For mid-MI tokens, coverage at K=64K=64 exceeds 99% on all four benchmarks, with TriviaQA the lowest. The high-MI band is the stringent case (Table 5). The three grounded benchmarks stay above 95% at K=64K=64, while TriviaQA reaches 77.3% and remains at 88.5% even at K=256K=256. Approximation bias. For the three grounded benchmarks the excluded high-MI tail mass is at most about 4.5% (CoQA, the tightest case), which bounds its entropy contribution by −0.045log0.045≈0.14-0.045 0.045≈ 0.14 nats on either H(p¯t)H( p_t) or H(pt(s))H(p^(s)_t), well below the cross-token variation in MItMI_t used as the uncertainty signal. On TriviaQA the excluded tail is larger, so the absolute bias is larger, but the relative ranking of tokens by MItMI_t, which is what PRR depends on, is preserved across the choice of K in our checks (Appendix C). Relation to the mechanism. The coverage pattern follows the same grounded and parametric split as the benchmark results. On CoQA, SQuAD, and BabiQA the candidate set at uncertain tokens is constrained by the input context, so subnetwork outputs stay concentrated on a moderate number of candidates even at high MI. On TriviaQA, parametric retrieval of factoid entities exposes a long tail of plausible alternatives, and coverage rises slowly with K. TriviaQA is also the task in Table 1 where ASMI is not the dominant method. Appendix C Boundary Deconfounds Two pre-registered checks ask whether the parametric-side boundary could be an artifact of the estimator rather than a property of the task regime. Enlarged truncation. We re-ran the masked scoring on closed-book TriviaQA with the truncation enlarged from K=64K=64 to K=256K=256 (n=5Kn=5K, identical per-example mask seeds). Mean token coverage rises to 0.970.97, where the high-MI stratum of Table 5 is the stringent case rather than the population average. Sem-ASMI nonetheless stays significantly below MSP, a paired gap of −0.041-0.041 PRR (95% CI [−0.055,−0.025][-0.055,-0.025]), and on the common subset the paired effect of enlarging K is slightly negative (−0.018-0.018, CI [−0.031,−0.005][-0.031,-0.005] for Sem-ASMI and −0.026-0.026, CI [−0.046,−0.006][-0.046,-0.006] for raw MI). The parametric-side deficit is therefore not a truncation artifact. The adaptive variant was not re-scored, since its gate needs sample-embedding inputs not cached in this pipeline. Same-dataset contrast. A second control holds the dataset fixed and moves only the knowledge source. We re-ran SQuAD on the same 5K5K examples with the passage removed from the prompt, keeping questions, decoding, the AlignScore metric, and the ASMI operating point identical, and clustering both conditions by the original passage (1,9341,934 clusters). The Sem-ASMI to MSP gap is a statistical tie in the open-book condition (−0.034-0.034, 95% CI [−0.078,+0.010][-0.078,+0.010]) and significantly negative once the passage is removed (−0.054-0.054, CI [−0.088,−0.021][-0.088,-0.021]), which reproduces the parametric-side pattern within a single dataset. Removing the passage also raises the base error rate sharply (mean AlignScore 0.8670.867 to 0.1090.109), so we read the paired within-condition gaps. Absolute PRR levels are compressed at this error rate. Appendix D The Distinct-Signal Analysis This appendix details the test behind the “A distinct signal” paragraph. Aggregate PRR and rank correlation cannot separate two questions: whether ASMI ranks errors well, and whether it reads information that output confidence does not. Because token-level MItMI_t and single-pass entropy are strongly rank correlated (Spearman around 0.90.9 on CoQA), we judge distinctness at the source, by residualization, not by aggregate scores. Setup. All quantities are computed at the operating layer. The confident-but-fragile and kernel analyses use Qwen3-4B-base, selecting BabiQA by the largest headroom-normalized gap between Sem-ASMI and MSP, with CoQA as a real-grounded generalization target and closed-book TriviaQA as the parametric contrast, and the residual test extends to all four datasets on both Qwen backbones (Table 6). For each generated sequence we record the semantically-weighted score usemu^sem, the raw MIMI, the single-pass entropy H, MSP, the ensemble-mean and mean-per-sample entropies used inside MIMI, and the AlignScore label. Residualization. We fit a logistic model of correctness on MSP,H\MSP,H\ and, out of fold, measure whether adding usemu^sem improves error prediction (incremental AUROC and a likelihood-ratio test) and whether the part of usemu^sem orthogonal to MSP,H\MSP,H\ detects errors on its own (residual AUROC against 0.50.5). All splits are out of fold with a fixed seed and bootstrap confidence intervals. Across the grid the residual excludes chance in five of the six grounded cells, with CoQA on Qwen3-8B the one miss, and sits at chance on TriviaQA at both backbones (Table 6). The likelihood-ratio test is nominally significant on TriviaQA at Qwen3-8B, but both intervals include chance and the interval criterion is what we use throughout. The raw MIMI carries part of the same residual on BabiQA and is borderline on CoQA, and on BabiQA at Qwen3-8B it exceeds the weighted score (0.6870.687 against 0.6030.603), so the kernel is not uniformly helpful at the larger scale. SQuAD at Qwen3-4B. The two 5K rows are independent Monte Carlo draws of the same estimator, correlating at 0.950.95 at the sequence level, and the effect sits at the estimator’s resolution: one draw excludes chance and the other does not. Within a draw the signal strengthens under alternative aggregations, from 0.5500.550 for the mean to 0.5940.594 for the top-quartile mean and 0.6130.613 for the max, consistent with mean aggregation diluting a token-level signal over SQuAD’s longest answers. This draw sensitivity concerns a borderline significance verdict rather than score instability, consistent with the redraw correlations of Appendix K. Backbone Dataset Residual AUROC Incr. AUROC Qwen3-4B BabiQA 0.589[0.547,0.627]0.589\,[0.547,0.627] +0.020[+0.008,+0.032]+0.020\,[+0.008,+0.032] Qwen3-4B CoQA 0.549[0.501,0.595]0.549\,[0.501,0.595] +0.018[+0.005,+0.031]+0.018\,[+0.005,+0.031] Qwen3-4B SQuAD, analysis draw 0.550[0.516,0.584]0.550\,[0.516,0.584] +0.007[−0.000,+0.015]+0.007\,[-0.000,+0.015] Qwen3-4B SQuAD, deployed draw 0.501[0.466,0.535]0.501\,[0.466,0.535] +0.001[−0.002,+0.003]+0.001\,[-0.002,+0.003] Qwen3-4B TriviaQA 0.495[0.465,0.524]0.495\,[0.465,0.524] −0.001[−0.003,+0.000]-0.001\,[-0.003,+0.000] Qwen3-8B BabiQA 0.603[0.558,0.641]0.603\,[0.558,0.641] +0.019[+0.009,+0.029]+0.019\,[+0.009,+0.029] Qwen3-8B CoQA 0.517[0.472,0.560]0.517\,[0.472,0.560] +0.007[−0.000,+0.015]+0.007\,[-0.000,+0.015] Qwen3-8B SQuAD 0.543[0.504,0.586]0.543\,[0.504,0.586] +0.004[−0.006,+0.014]+0.004\,[-0.006,+0.014] Qwen3-8B TriviaQA 0.511[0.475,0.547]0.511\,[0.475,0.547] +0.002[−0.002,+0.006]+0.002\,[-0.002,+0.006] Table 6: Out-of-fold residual and incremental error-detection AUROC of usemu^sem over MSP,H\MSP,H\ (95% CIs). A cell counts as detected when the residual interval excludes 0.50.5. The two SQuAD Qwen3-4B rows are the same estimator under two independent Monte Carlo mask draws. The confident-but-fragile split. Splitting sequences at the medians of MSP and usemu^sem gives a two-by-two grid of error rates. On BabiQA the confident-fragile cell (high MSP, high usemu^sem) carries an error gap of 0.210.21 [0.12,0.29][0.12,0.29] over the confident-robust cell, the contrast plotted in Figure 4. The gap is 0.120.12 [0.04,0.21][0.04,0.21] on CoQA and 0.280.28 [0.19,0.37][0.19,0.37] on TriviaQA, where a high base error rate inflates the absolute difference. The raw gap is therefore a poor regime marker on its own. The regimes separate only after conditioning on MSP,H\MSP,H\, where the grounded gaps still carry information and the TriviaQA gap does not (Table 6). Within the confident stratum the gap is not resolvable at the token level (−0.09-0.09 [−0.16,0.00][-0.16,0.00]), consistent with the sequence-level reading below. What the kernel adds. The residual over MSP,H\MSP,H\ above tests distinctness from output confidence. A separate question is how much of the distinct signal comes from the semantic kernel rather than from the masking disagreement, and it cannot be read from the same residual. The two entropies inside the mutual information, the ensemble-mean entropy H(p¯t)H( p_t) and the mean-per-sample entropy 1S∑sH(pt(s)) 1S _sH(p^(s)_t), span MItMI_t by construction, since MItMI_t is their difference (Eq. 5). Controlling for both therefore removes the entire linear contribution of the mutual information, so the residual of usem=MIt(1−At)u^sem=MI_t(1-A_t) under this control measures what the kernel factor (1−At)(1-A_t) adds, not distinctness from confidence. So read, the kernel contributes error information beyond the masking disagreement on synthetic BabiQA (0.603[0.565,0.641]0.603\,[0.565,0.641]) and little on real CoQA (0.513[0.465,0.562]0.513\,[0.465,0.562]). This is not the distinct signal weakening on CoQA, which the MSP,H\MSP,H\ test in Table 6 already rules out. It is that on real grounded data the distinct signal is carried by the attention-masking disagreement itself, and the kernel improves the ranking by discounting surface-form disagreement rather than by adding an axis orthogonal to MItMI_t. This is consistent with the kernel’s 0.040.04 to 0.060.06 PRR lift on CoQA in Table 1, a reranking within the mutual-information ordering rather than new information. For the same span reason we do not control raw MIMI with these entropies, since its residual would be zero by construction. Ensemble-mean entropy alone. A natural question is whether the masked ensemble helps only through averaging, since the ensemble-mean entropy H(p¯t)H( p_t) is a free by-product of the same forty passes. Scored alone on the three cached Qwen3-4B analysis cells (BabiQA and CoQA at n=1Kn=1K, TriviaQA at n=1.5Kn=1.5K), it is essentially the single-pass entropy, with Spearman rank correlations from 0.9670.967 to 0.9950.995, so the ensemble mean adds no standalone information. Raw MIMI does not beat it on the grounded cells (both intervals cross zero), but the semantically weighted usemu^sem does, by +0.046+0.046 PRR on BabiQA (95% CI [+0.004,+0.089][+0.004,+0.089]) and +0.093+0.093 on CoQA ([+0.032,+0.156][+0.032,+0.156]) under the same cluster-respecting paired bootstrap. On parametric TriviaQA the ordering reverses and H(p¯t)H( p_t) alone is the best of the four, consistent with the regime boundary. Averaging over masks therefore accounts for none of the gain, which comes from the weighted disagreement term that the semantic kernel multiplies. Further scope. Two limits of the analysis are worth recording. The distinctness is established by residualization and not by an asymmetric error set: at a matched flag rate on BabiQA, usemu^sem and entropy flag balanced disjoint error sets (McNemar p=0.26p=0.26), so the correct claim is that usemu^sem adds error-predictive information beyond output confidence, not that it catches a class of errors entropy is blind to. And within the confident stratum the effect is not resolvable at the token level with per-sequence labels (AUROC near 0.50.5), so it is a sequence-level phenomenon. Appendix E Deployment Payoff: Full Analysis This appendix gives the full selective-prediction analysis behind the deployment result in the main text. We report the single-signal risk-coverage comparison, the honest null for a global selector, and the parametric contrast, none of which fit in the main body. Protocol. For each signal we sort predictions by uncertainty, retain the most confident fraction (the coverage c), and abstain on the rest. The retained error rate is the risk. We report the area under the risk-coverage curve (AURC, lower is better) (9). Any selector that combines signals is fit out-of-fold with the same cluster keys as the main-text bootstrap, so no example scores itself. We report grounded cells (BabiQA, CoQA) and the parametric contrast (TriviaQA) throughout. SQuAD shows no confident-stratum gain on Qwen3-4B, consistent with the dilution pattern of Appendix D, so the payoff claim rests on BabiQA and CoQA. Single-signal risk-coverage. Table 7 reports AURC for each signal on its own. On both grounded cells usemu^sem has the lowest AURC, and on parametric TriviaQA it is the worst, mirroring the benchmark boundary. The advantage is concentrated at the most-confident end of the curve: usemu^sem is lowest at coverage 0.50.5 to 0.60.6 and crosses the single-pass signals near full coverage. This is the same confident-region effect that the confident-stratum filter exploits below, shown within the confident stratum in Figure 5 of the main text. Cell MSP Entropy usemu^sem BabiQA (grounded) 0.1510.151 0.1510.151 0.1330.133 CoQA (grounded) 0.1340.134 0.1340.134 0.1230.123 TriviaQA (parametric) 0.3140.314 0.3050.305 0.3270.327 Table 7: Single-signal area under the risk-coverage curve (AURC, lower is better). Bold marks the best signal in each row. A trained global selector. We next ask whether usemu^sem improves a trained selector over the single-pass signals. We fit an out-of-fold logistic model of correctness on MSP,entropy\MSP,entropy\ (base) and on MSP,entropy,usem\MSP,entropy,u^sem\ (full), and compare their risk-coverage curves. Across coverage .7,.8,.9\.7,.8,.9\ the two curves stay within noise on both grounded cells, with paired bootstrap intervals on the difference that include zero throughout, and likewise on TriviaQA. The reason is localization rather than absence: usemu^sem’s error information sits almost entirely in the confident stratum, so a single global fit that spreads one model across all coverage levels averages that region together with the rest and cannot use it. The same information is what the confident-stratum filter below turns into a large gain. Filtering the confident stratum. The payoff appears when the filter targets the confident stratum directly, which is the training-free result reported in the main text (Table 4, Figure 5). Within the MSP-confident half, abstaining on the top half by usemu^sem gives a final coverage of 0.250.25 and roughly halves the retained error on BabiQA and CoQA, well ahead of spending the same abstention budget on predictive entropy. On parametric TriviaQA the effect reverses and entropy filtering is slightly better. The null global selector and the confident-stratum filter are consistent with each other, since the deployment value of usemu^sem is concentrated in the confident-but-fragile region the mechanism predicts. Scope. The analysis covers two grounded cells and one parametric contrast on Qwen3-4B-base. usemu^sem runs on TriviaQA as well, with a finite AURC of 0.3270.327, and simply loses to the single-pass signals there rather than being inapplicable. Appendix F Head-Level Structural Analysis This appendix gives the head-level picture behind the “Structural analysis” paragraph, at the estimator’s operating layer. It is deliberately brief, since the operating-layer validation of the signal is the confident-but-fragile analysis of Appendix D. Estimation setup. The analysis uses Qwen3-4B-base with the same mask rate p=0.15p=0.15 and top-K truncation K=64K=64 as the main experiments, at the operating depth of about 60%60\%. The only change is an enlarged budget of San=300S_an=300 masked passes per example to reduce the variance of the per-head estimates, leaving the estimator unchanged at S=40S=40. We analyze the first 500500 examples of each split, which yields 2,7802,780, 7,3747,374, 1,0891,089, and 2,0272,027 generated tokens on CoQA, SQuAD, BabiQA, and TriviaQA. Permutation nulls use 11K shuffles per token and confidence intervals use 22K example-level bootstrap resamples. Criticality estimator. For each token t, the criticality of head h is the difference in mean divergence between the masks that ablate it and those that keep it, β^h,t=dt¯|mh(s)=0−dt¯|mh(s)=1, β_h,t= d_t |_m^(s)_h=0- d_t |_m^(s)_h=1, (17) where dt¯|mh(s)=b d_t |_m^(s)_h=b averages dt(s)=KL(pt(s)∥p¯t)d^(s)_t=KL(p^(s)_t\| p_t) over the samples with mask bit b. Because masks are independent Bernoulli variables, β^h,t β_h,t is unbiased for the marginal effect of ablating head h. We measure the concentration of the non-negative profile by the Herfindahl index HHIt=∑hwh,t2HHI_t= _hw_h,t^2 with wh,t=max(β^h,t,0)/∑h′max(β^h′,t,0)w_h,t= ( β_h,t,0)/ _h ( β_h ,t,0), ranging from 1/H1/H for a uniform profile to 11 for a single critical head. To remove the concentration that estimation noise alone produces, we permute the divergence-to-mask assignment 11K times per token and standardize the observed index against this null, giving ztz_t. Coupling at the operating layer. At the operating layer the criticality of an uncertain token is distributed across about ten heads rather than concentrated on a few (Table 8). The coupling is modest in magnitude but consistent in sign, following the grounded and parametric divide. The association ρ(MIt,zt)ρ(MI_t,z_t) is positive on all three grounded benchmarks and negative on parametric TriviaQA, with intervals excluding zero, and mutual information tracks this head structure more than single-pass entropy does on the grounded cells. On CoQA the coupling sharpens with depth, from 0.090.09 at the operating layer to 0.440.44 four layers from the top, as the criticality concentrates onto fewer heads. Benchmark ρ(MIt,zt)ρ(MI_t,z_t) ρ(Ht,zt)ρ(H_t,z_t) neffn_eff (high MI) CoQA 0.09[0.05,0.13]0.09\ [0.05,0.13] −0.04[−0.09,0.00]-0.04\ [-0.09,0.00] 9.79.7 SQuAD 0.14[0.11,0.17]0.14\ [0.11,0.17] 0.04[0.01,0.07]0.04\ [0.01,0.07] 9.19.1 BabiQA 0.14[0.08,0.19]0.14\ [0.08,0.19] 0.10[0.05,0.15]0.10\ [0.05,0.15] 9.69.6 TriviaQA −0.10[−0.14,−0.05]-0.10\ [-0.14,-0.05] −0.19[−0.23,−0.14]-0.19\ [-0.23,-0.14] 9.69.6 Table 8: Head-criticality at the 60%60\% operating layer on Qwen3-4B-base (95% CIs). ztz_t is the permutation-standardized concentration of the head-criticality profile, and neff=1/HHItn_eff=1/HHI_t is its effective head count on the high-MI stratum. Causal probe. A causal probe confirms that the fragility is head borne. For each benchmark we take the 200200 highest-MI and 200200 lowest-MI generated tokens and deterministically ablate the token’s k most critical heads. Ablating even one head flips a sizable fraction of the high-MI tokens while the low-MI tokens almost never flip, and the gap widens with k up to k=8k=8 (Table 9). The dependence is present on all four benchmarks, so MItMI_t measures genuine reliance on specific heads. What follows the grounded and parametric divide is whether that reliance tracks uncertainty. For k≥2k≥ 2 the flip rates are in fact highest on TriviaQA, so fragility itself is strongest under parametric recall, and it is the alignment with correctness that is missing there. Mechanistic accounts of factual recall (11; 4; 31; 10) explain this case, since knowledge sits in mid-layer MLP sublayers while attention heads extract it, so head perturbation probes an extraction pathway that runs whether or not the underlying knowledge is strong. High-MI flip rate (%) k=1k=1 k=2k=2 k=4k=4 k=8k=8 k=16k=16 CoQA 1616 3333 3636 4343 4343 SQuAD 1414 1919 1919 2525 2525 BabiQA 2525 3232 4141 4242 3131 TriviaQA 2222 3535 5757 6363 6161 Table 9: Causal probe at the operating layer. Fraction of the 200200 highest-MI tokens whose greedy prediction flips when the token’s k most critical heads are ablated. The 200200 lowest-MI tokens flip at most 1%1\% throughout. Appendix G Derivation of the Coverage-Fidelity Constraints Figure 6: Coverage lower bound and fidelity upper bound in the (p,S) plane. The adopted operating point (0.15, 40) is the minimum-cost corner of the feasible region. UE Method Qwen3-4B Qwen3-8B Llama-2-7B Mistral-7B CoQA SQuAD CoQA SQuAD CoQA SQuAD CoQA SQuAD Information-based Maximum Sequence Probability 0.45± 0.02 0.32± 0.03 0.53± 0.02 0.35± 0.02 0.53± 0.01 0.18± 0.02 0.52± 0.02 0.65± 0.01 Perplexity 0.42± 0.02 0.27± 0.03 0.50± 0.01 0.40± 0.02 0.52± 0.01 0.73± 0.02 0.48± 0.02 0.81± 0.01 Mean Token Entropy 0.40± 0.02 0.27± 0.03 0.50± 0.01 0.41± 0.02 0.46± 0.02 0.68± 0.02 0.44± 0.02 0.78± 0.01 Pointwise Mutual Information -0.05± 0.02 -0.14± 0.04 -0.04± 0.02 0.24± 0.02 -0.04± 0.02 0.50± 0.02 -0.12± 0.03 0.69± 0.02 SelfCertainty 0.19± 0.02 0.02± 0.04 0.28± 0.02 0.32± 0.02 0.21± 0.02 0.57± 0.02 0.23± 0.02 0.73± 0.02 Conditional Pointwise MI -0.19± 0.03 -0.35± 0.05 -0.22± 0.02 -0.15± 0.03 -0.21± 0.02 0.41± 0.02 -0.29± 0.03 0.67± 0.02 Sample-diversity Monte Carlo Sequence Entropy 0.43± 0.02 0.27± 0.03 0.52± 0.02 0.45± 0.02 0.52± 0.01 0.36± 0.02 0.51± 0.02 0.77± 0.01 MC Normalized Sequence Entropy 0.44± 0.02 0.23± 0.03 0.49± 0.02 0.39± 0.02 0.50± 0.01 0.48± 0.02 0.49± 0.02 0.69± 0.01 Semantic Entropy 0.46± 0.02 0.28± 0.03 0.53± 0.01 0.45± 0.02 0.55± 0.01 0.47± 0.02 0.54± 0.02 0.80± 0.01 SAR 0.46± 0.02 0.27± 0.03 0.47± 0.02 0.40± 0.02 0.53± 0.01 0.51± 0.02 0.52± 0.02 0.78± 0.01 SentenceSAR (max 5) 0.44± 0.02 0.28± 0.03 0.51± 0.02 0.43± 0.02 0.54± 0.01 0.45± 0.02 0.53± 0.02 0.80± 0.01 LUQ 0.37± 0.02 0.32± 0.02 0.37± 0.02 0.50± 0.02 0.39± 0.02 0.43± 0.02 0.41± 0.02 0.73± 0.01 Kernel Language Entropy 0.41± 0.02 0.31± 0.02 0.40± 0.02 0.48± 0.02 0.44± 0.02 0.50± 0.02 0.47± 0.02 0.80± 0.01 Probing P(True) 0.03± 0.02 0.07± 0.03 0.07± 0.02 0.17± 0.02 -0.04± 0.02 0.47± 0.02 -0.05± 0.03 0.78± 0.01 P(True) Sampling -0.04± 0.03 0.03± 0.04 0.04± 0.02 0.30± 0.02 -0.04± 0.02 0.44± 0.02 -0.28± 0.03 0.29± 0.03 Attention-based RAUQ 0.37± 0.02 0.32± 0.03 0.48± 0.02 0.45± 0.02 0.47± 0.02 0.46± 0.02 0.46± 0.02 0.82± 0.01 RAUQ (entropy) 0.47± 0.02 0.36± 0.02 0.53± 0.02 0.47± 0.02 0.50± 0.02 0.75± 0.02 0.51± 0.02 0.83± 0.01 Attention-based, active perturbation (ours) ASMI (60%, K=64) 0.46± 0.02 0.29± 0.03 0.51± 0.01 0.34± 0.02 0.49± 0.02 0.62± 0.02 0.48± 0.02 0.77± 0.02 Sem-ASMI (60%, K=64) 0.52± 0.02 0.29± 0.03 0.55± 0.01 0.40± 0.02 0.54± 0.01 0.69± 0.02 0.53± 0.02 0.83± 0.01 Adapt-ASMI (60%, K=64) 0.53± 0.01 0.31± 0.02 0.56± 0.02 0.40± 0.02 0.55± 0.01 0.71± 0.02 0.54± 0.02 0.84± 0.01 ASMI (70%, K=64) 0.47± 0.02 0.27± 0.03 0.52± 0.01 0.40± 0.02 0.46± 0.02 0.61± 0.02 0.46± 0.02 0.73± 0.02 ASMI (80%, K=64) 0.44± 0.02 0.28± 0.02 0.51± 0.01 0.43± 0.02 0.40± 0.02 0.67± 0.02 0.38± 0.02 0.74± 0.02 ASMI (90%, K=64) 0.29± 0.02 0.24± 0.03 0.42± 0.02 0.34± 0.02 0.43± 0.02 0.70± 0.01 0.45± 0.02 0.73± 0.02 Sem-ASMI (70%, K=64) 0.52± 0.02 0.28± 0.03 0.54± 0.01 0.42± 0.02 0.52± 0.01 0.68± 0.02 0.53± 0.02 0.79± 0.01 Sem-ASMI (80%, K=64) 0.50± 0.01 0.28± 0.03 0.54± 0.01 0.43± 0.02 0.49± 0.01 0.71± 0.01 0.49± 0.02 0.80± 0.01 Sem-ASMI (90%, K=64) 0.41± 0.02 0.26± 0.03 0.51± 0.02 0.39± 0.02 0.52± 0.01 0.74± 0.01 0.52± 0.02 0.78± 0.01 Adapt-ASMI (70%, K=64) 0.52± 0.02 0.29± 0.03 0.55± 0.01 0.44± 0.02 0.53± 0.01 0.69± 0.02 0.54± 0.01 0.81± 0.01 Adapt-ASMI (80%, K=64) 0.50± 0.02 0.30± 0.02 0.55± 0.01 0.46± 0.02 0.50± 0.01 0.72± 0.01 0.50± 0.02 0.81± 0.01 Adapt-ASMI (90%, K=64) 0.42± 0.02 0.27± 0.03 0.52± 0.02 0.40± 0.02 0.53± 0.01 0.75± 0.01 0.53± 0.02 0.80± 0.01 Table 10: PRR (mean ± standard error, higher is better) on CoQA and SQuAD, scored by AlignScore. Full version of Table 1, with all 17 baselines and the three ASMI variants across the depth sweep. Bold and underline mark the best and second-best PRR per column at the operating depth d=60%d=60\%. Rows at other depths belong to the sweep and are left unmarked. UE Method Qwen3-4B Qwen3-8B Llama-2-7B Mistral-7B BabiQA TriviaQA BabiQA TriviaQA BabiQA TriviaQA BabiQA TriviaQA Information-based Maximum Sequence Probability 0.48± 0.04 0.72± 0.02 0.76± 0.03 0.70± 0.02 0.58± 0.04 0.77± 0.01 0.59± 0.04 0.78± 0.01 Perplexity 0.49± 0.05 0.71± 0.01 0.74± 0.03 0.68± 0.02 0.53± 0.04 0.76± 0.01 0.59± 0.04 0.79± 0.01 Mean Token Entropy 0.56± 0.04 0.70± 0.02 0.75± 0.03 0.69± 0.02 0.30± 0.05 0.75± 0.01 0.54± 0.04 0.78± 0.01 Pointwise Mutual Information 0.24± 0.05 -0.05± 0.02 0.46± 0.04 -0.07± 0.02 0.20± 0.05 -0.02± 0.03 0.24± 0.05 -0.01± 0.03 SelfCertainty 0.07± 0.07 0.32± 0.02 0.62± 0.04 0.34± 0.02 0.06± 0.05 0.52± 0.02 0.20± 0.05 0.60± 0.02 Conditional Pointwise MI 0.09± 0.07 -0.30± 0.02 -0.45± 0.06 -0.35± 0.02 0.25± 0.05 -0.34± 0.03 0.09± 0.06 -0.35± 0.03 Sample-diversity Monte Carlo Sequence Entropy 0.47± 0.05 0.71± 0.02 0.74± 0.03 0.69± 0.01 0.52± 0.04 0.76± 0.01 0.56± 0.04 0.77± 0.01 MC Normalized Sequence Entropy 0.50± 0.05 0.70± 0.02 0.72± 0.03 0.67± 0.02 0.50± 0.04 0.75± 0.01 0.53± 0.04 0.78± 0.01 Semantic Entropy 0.54± 0.04 0.72± 0.02 0.74± 0.03 0.70± 0.02 0.50± 0.04 0.77± 0.01 0.58± 0.04 0.78± 0.01 SAR 0.63± 0.04 0.71± 0.02 0.71± 0.03 0.68± 0.02 0.49± 0.04 0.78± 0.01 0.56± 0.04 0.80± 0.01 SentenceSAR (max 5) 0.49± 0.05 0.71± 0.02 0.73± 0.03 0.69± 0.02 0.52± 0.04 0.77± 0.01 0.54± 0.04 0.78± 0.01 LUQ 0.47± 0.05 0.64± 0.02 0.43± 0.05 0.52± 0.02 0.42± 0.04 0.69± 0.02 0.37± 0.05 0.73± 0.02 Kernel Language Entropy 0.48± 0.05 0.64± 0.02 0.44± 0.05 0.51± 0.02 0.41± 0.05 0.70± 0.02 0.37± 0.05 0.73± 0.02 Probing P(True) 0.49± 0.04 0.30± 0.02 0.22± 0.06 0.39± 0.02 0.21± 0.05 0.06± 0.03 0.20± 0.05 -0.11± 0.03 P(True) Sampling 0.11± 0.06 0.38± 0.02 0.32± 0.06 0.39± 0.02 0.21± 0.05 0.06± 0.03 0.18± 0.06 0.20± 0.03 Attention-based RAUQ 0.32± 0.06 0.70± 0.02 0.74± 0.03 0.66± 0.02 0.57± 0.04 0.75± 0.01 0.48± 0.05 0.70± 0.02 RAUQ (entropy) 0.42± 0.05 0.72± 0.02 0.74± 0.03 0.70± 0.02 0.29± 0.05 0.77± 0.01 0.46± 0.04 0.77± 0.01 Attention-based, active perturbation (ours) ASMI (60%, K=64) 0.60± 0.04 0.59± 0.02 0.80± 0.03 0.62± 0.02 0.11± 0.05 0.66± 0.02 0.52± 0.04 0.74± 0.01 Sem-ASMI (60%, K=64) 0.66± 0.04 0.66± 0.02 0.78± 0.03 0.66± 0.02 0.27± 0.05 0.71± 0.01 0.61± 0.04 0.78± 0.01 Adapt-ASMI (60%, K=64) 0.66± 0.04 0.66± 0.02 0.79± 0.03 0.67± 0.02 0.29± 0.05 0.72± 0.01 0.61± 0.03 0.78± 0.01 ASMI (70%, K=64) 0.48± 0.04 0.64± 0.02 0.78± 0.03 0.65± 0.02 0.23± 0.05 0.69± 0.01 0.41± 0.04 0.67± 0.01 ASMI (80%, K=64) 0.33± 0.04 0.58± 0.02 0.48± 0.04 0.62± 0.02 0.24± 0.05 0.66± 0.02 0.51± 0.04 0.68± 0.01 ASMI (90%, K=64) 0.55± 0.04 0.58± 0.02 0.76± 0.03 0.49± 0.02 0.10± 0.04 0.68± 0.02 0.49± 0.04 0.69± 0.02 Sem-ASMI (70%, K=64) 0.58± 0.04 0.68± 0.02 0.76± 0.03 0.68± 0.02 0.33± 0.04 0.73± 0.01 0.57± 0.04 0.74± 0.01 Sem-ASMI (80%, K=64) 0.38± 0.05 0.64± 0.02 0.70± 0.03 0.66± 0.02 0.39± 0.04 0.72± 0.01 0.63± 0.03 0.75± 0.01 Sem-ASMI (90%, K=64) 0.64± 0.04 0.67± 0.02 0.77± 0.03 0.62± 0.02 0.31± 0.05 0.73± 0.01 0.62± 0.03 0.75± 0.01 Adapt-ASMI (70%, K=64) 0.58± 0.04 0.69± 0.02 0.77± 0.03 0.68± 0.02 0.34± 0.04 0.74± 0.01 0.57± 0.04 0.75± 0.01 Adapt-ASMI (80%, K=64) 0.38± 0.05 0.64± 0.02 0.69± 0.03 0.66± 0.02 0.41± 0.04 0.72± 0.01 0.63± 0.03 0.75± 0.01 Adapt-ASMI (90%, K=64) 0.64± 0.04 0.67± 0.02 0.77± 0.03 0.62± 0.02 0.33± 0.05 0.73± 0.01 0.62± 0.03 0.75± 0.01 Table 11: PRR (mean ± standard error, higher is better) on BabiQA and closed-book TriviaQA, the parametric control, scored by AlignScore. Full version of Table 1, with all 17 baselines and the three ASMI variants across the depth sweep. Bold and underline mark the best and second-best PRR per column at the operating depth d=60%d=60\%. Rows at other depths belong to the sweep and are left unmarked. Coverage lower bound on S. Each Monte Carlo sample draws an independent mask m(s)∈0,1Hm^(s)∈\0,1\^H with mh(s)∼Bernoulli(1−p)m^(s)_h (1-p). The probability that a given head h is never masked across S samples is (1−p)S(1-p)^S. By a union bound over H heads, the probability that at least one head is never ablated is at most H(1−p)SH(1-p)^S. Requiring this failure probability to be at most δ gives H(1−p)S≤δ⟺S≥ln(H/δ)−ln(1−p).H(1-p)^S≤δ\;\; \;\;S≥ (H/δ)- (1-p). (18) Coverage is a necessary condition for the mutual-information estimate: a head that is never masked contributes no observed perturbation, so its criticality is invisible to the estimator. For H=32H=32 and δ=0.05δ=0.05, Equation 18 yields S≥62S≥ 62 at p=0.10p=0.10 but only S≥40S≥ 40 at p=0.15p=0.15: increasing p reduces the sampling budget required for full coverage. Fidelity upper bound on p. The number of simultaneously masked heads in a sample is k∼Binomial(H,p)k (H,p), with mean HpHp and variance Hp(1−p)Hp(1-p). The perturbed network must remain a functioning model: if too many heads are removed at once, the measurement no longer probes dependence on attention paths but instead reflects generic degradation of a broken computation. We therefore require that k stay below a tolerance kmaxk_ with high probability, using the Gaussian upper bound Hp+z1−δ′Hp(1−p)≤kmax.Hp+z_1-δ Hp(1-p)\;≤\;k_ . (19) We set kmax=8k_ =8, i.e., one quarter of the heads in a layer, following head-pruning studies that report negligible loss within this margin and a sharp collapse beyond it (32; 48). With z0.95=1.645z_0.95=1.645 and H=32H=32, the left side of Equation 19 evaluates to 4.8+1.6454.08≈8.14.8+1.645 4.08≈ 8.1 at p=0.15p=0.15, marginally above kmaxk_ under the Gaussian surrogate, while the exact Binomial(32,0.15)Binomial(32,0.15) 95th percentile equals 88. The exact distribution therefore admits p=0.15p=0.15 as the largest admissible mask rate, which is the sense in which we write p≲0.15p 0.15. Feasible region and the chosen operating point. Figure 6 plots both constraints in the (p,S)(p,S) plane. The coverage curve (Eq. 18) bounds the region from below and decreases in p, and the fidelity line (Eq. 19) bounds it from the right. The minimum-cost point of the feasible region, the smallest S satisfying coverage at the largest admissible p, is the corner (p,S)=(0.15,40)(p,S)=(0.15,40), which we adopt without further tuning. Earlier configurations (0.10,16)(0.10,16) and (0.15,32)(0.15,32) lie below the coverage curve. At (0.10,16)(0.10,16) the expected number of never-ablated heads is H(1−p)S=32⋅0.916≈5.9H(1-p)^S=32· 0.9^16≈ 5.9, so roughly six heads per layer are never perturbed across the S samples and part of the signal is omitted. Scope. Both constraints depend on the model only through the number of maskable units, which in our design is the number of query heads H, since masks are applied to query-head outputs before the output projection. Grouped-query attention reduces the number of key/value projections but leaves the query heads intact: Qwen3-4B, Qwen3-8B, and Mistral-7B retain H=32H=32 query heads alongside their grouped key/value projections, matching the standard multi-head attention of Llama-2-7B, so the operating point (0.15,40)(0.15,40) transfers across all four backbones without re-derivation. Equations 18 and 19 generalize directly, and re-derivation is required only when H itself changes, for instance for a backbone with a different query-head count or a coarser masking granularity such as whole key/value groups. The coverage condition guarantees only that every head is perturbed at least once. It does not bound the variance of the MI estimate, which continues to decrease as O(1/S)O(1/ S) beyond the coverage threshold. Pairwise co-ablation coverage, in which every head pair is observed masked jointly, would require S≳ln((H2)/δ)/p2≈410S \! ( H2/δ )/p^2≈ 410 samples at p=0.15p=0.15 (at the same coverage failure probability δ=0.05δ=0.05) and is outside our compute budget, so the estimator captures marginal head criticality rather than higher-order interaction effects. Appendix H Measured Cost All measurements use Qwen3-4B-base on a single H100 MIG 3g.40gb slice with 200 fixed examples per dataset, 10 warmup examples excluded, reporting per-example medians. Peak memory spans 8.38.3 to 9.39.3 GB across all methods. Phase medians on CoQA and SQuAD are: greedy response 4.144.14 and 0.750.75 seconds, ten stochastic samples drawn as one batch 4.414.41 and 1.281.28, NLI clustering for Semantic Entropy 0.180.18 and 0.130.13, mask-batched ASMI scoring 2.212.21 and 1.741.74, sequential ASMI scoring 5.175.17 and 2.342.34. Mask batching carries the comparison. The 4040 masked suffixes share one batched pass, which is 2.3×2.3× faster than sequential masking and, on CoQA, cheaper than drawing the ten stochastic samples it replaces. The auxiliary NLI model contributes under 0.20.2 seconds per example, so the cost of the sampling baselines is dominated by generation itself. The advantage of ASMI tracks input length and therefore favors the long CoQA inputs over the shorter SQuAD ones. The analytic FLOPs view of Appendix A is complementary, since batched masked passes trade FLOPs for parallelism. Absolute seconds are specific to this hardware, so we read only the relative ordering from them. Appendix I Full Benchmark and Depth Sweep Tables 10 and 11 report the complete benchmark that the main text abbreviates. The main comparison in Table 1 keeps a compact baseline set at the single operating depth d=60%d=60\%. Here we list every baseline together with the three ASMI variants across the full depth sweep, so the abbreviated main table and the depth analysis draw on one set of numbers. Bold and underline follow the main-table convention and are computed only at the 60%60\% operating depth, so the highlighting stays comparable to Table 1 while the remaining depths are shown for context. One property of the Llama-2-7B SQuAD column is worth noting. On this task 28.5%28.5\% of the greedy generations contain nothing but whitespace and wrong answers are much shorter than correct ones (7.0 against 12.7 tokens on average), which depresses length-sensitive scores such as MSP in that column, whose error-detection AUROC falls to 0.470.47. Since every method scores the same greedy responses, comparisons within the column remain valid. Depth selection. We take d=60%d=60\% as the representative depth by aggregate PRR over the sweep. Aggregated across all sixteen cells, four benchmarks by four backbones, normalized PRR is highest at 60%60\% (0.6000.600) against 0.5920.592, 0.5770.577, and 0.5860.586 at the deeper settings, and 60%60\% is the best or tied-best depth in eight of the twelve grounded columns. The exceptions are SQuAD on Qwen3-8B and Llama-2-7B and BabiQA on Llama-2-7B and Mistral-7B. On CoQA performance rises smoothly toward 60%60\%, consistent with masking within the context-routing band rather than the near-output layers, and SQuAD follows the same trend on Qwen3-4B and Mistral-7B while favoring deeper masking on Llama-2-7B. BabiQA is the exception discussed next. Stability of depth selection We verify that the 60%60\% choice is not an artifact of the full-data aggregate with a cluster-respecting split-half test. We draw R=1KR=1K random halves (seed 2026071120260711), keeping related items together: CoQA is split by story and SQuAD by passage, the same cluster keys used for the main-text bootstrap, while BabiQA and TriviaQA are split by input. On each split we select the argmax depth on the first half and measure its out-of-half regret, the PRR gap to the best depth, on the second half, with PRR normalized within each half. Across the sixteen cells, 60%60\% is selected in 99.3%99.3\% of the splits, the remaining 0.7%0.7\% go to 70%70\%, and 80%80\% and 90%90\% are never selected, with zero out-of-half regret in 98.7%98.7\% of splits and a mean regret of 0.00010.0001 PRR. The selection reproduces for Sem-ASMI at 99.0%99.0\%. The representative depth is therefore stable across resampling and variant. BabiQA cell MI¯ MI MIcorrMI_corr MIwrongMI_wrong ASMI MSP Qwen3-4B 0.0160.016 0.0140.014 0.0220.022 0.770.77 0.680.68 Qwen3-8B 0.0250.025 0.0170.017 0.0450.045 0.860.86 0.860.86 Llama-2-7B 0.0020.002 0.0020.002 0.0020.002 0.600.60 0.750.75 Table 12: Mechanism of the BabiQA failure (d=60%d=60\%, K=64K=64). Mean token-level MItMI_t split by answer correctness, with error-detection AUROC for ASMI and MSP. Head masking barely moves the Llama-2-7B output, which leaves correct and wrong answers indistinguishable to ASMI. Figure 7: Error separation on BabiQA. Violins show per-example uncertainty for correct (blue) and wrong (orange) generations, with error-detection AUROC per panel. Llama-2-7B (top) is over-robust to head masking, so Adapt-ASMI cannot separate errors while MSP still can. Qwen3-8B (bottom) routes wrong answers through more fragile paths. Backbone Dataset Top ASMI variant (PRR) Best baseline (PRR) Top Δ vs SemEnt Top Δ vs best baseline Sem-ASMI Δ vs SemEnt Qwen3-4B CoQA Adapt-ASMI (0.524) RAUQ-E (0.469) +0.065∗[+0.042,+0.088]+0.065^*\ [+0.042,+0.088] +0.056∗[+0.035,+0.076]+0.056^*\ [+0.035,+0.076] +0.060∗[+0.036,+0.085]+0.060^*\ [+0.036,+0.085] SQuAD Adapt-ASMI (0.305) RAUQ-E (0.359) +0.024[−0.011,+0.059]+0.024\ [-0.011,+0.059] −0.053∗[−0.098,−0.006]-0.053^*\ [-0.098,-0.006] +0.004[−0.037,+0.044]+0.004\ [-0.037,+0.044] BabiQA Sem-ASMI (0.657) SAR (0.628) +0.120∗[+0.051,+0.192]+0.120^*\ [+0.051,+0.192] +0.030[−0.014,+0.072]+0.030\ [-0.014,+0.072] +0.120∗[+0.051,+0.192]+0.120^*\ [+0.051,+0.192] TriviaQA Adapt-ASMI (0.662) RAUQ-E (0.723) −0.057∗[−0.072,−0.043]-0.057^*\ [-0.072,-0.043] −0.060∗[−0.073,−0.048]-0.060^*\ [-0.073,-0.048] −0.061∗[−0.076,−0.046]-0.061^*\ [-0.076,-0.046] Qwen3-8B CoQA Adapt-ASMI (0.559) SemEnt (0.529) +0.029∗[+0.013,+0.045]+0.029^*\ [+0.013,+0.045] +0.029∗[+0.013,+0.045]+0.029^*\ [+0.013,+0.045] +0.019∗[+0.002,+0.035]+0.019^*\ [+0.002,+0.035] SQuAD Adapt-ASMI (0.402) LUQ (0.500) −0.052∗[−0.087,−0.019]-0.052^*\ [-0.087,-0.019] −0.098∗[−0.138,−0.059]-0.098^*\ [-0.138,-0.059] −0.057∗[−0.094,−0.021]-0.057^*\ [-0.094,-0.021] BabiQA ASMI (0.798) MSP (0.761) +0.060∗[+0.028,+0.095]+0.060^*\ [+0.028,+0.095] +0.037∗[+0.005,+0.072]+0.037^*\ [+0.005,+0.072] +0.042∗[+0.014,+0.074]+0.042^*\ [+0.014,+0.074] TriviaQA Adapt-ASMI (0.670) RAUQ-E (0.706) −0.031∗[−0.044,−0.018]-0.031^*\ [-0.044,-0.018] −0.037∗[−0.051,−0.022]-0.037^*\ [-0.051,-0.022] −0.037∗[−0.051,−0.023]-0.037^*\ [-0.051,-0.023] Llama-2-7B CoQA Adapt-ASMI (0.552) SemEnt (0.545) +0.008[−0.012,+0.027]+0.008\ [-0.012,+0.027] +0.008[−0.012,+0.027]+0.008\ [-0.012,+0.027] −0.001[−0.023,+0.020]-0.001\ [-0.023,+0.020] SQuAD Adapt-ASMI (0.706) RAUQ-E (0.753) +0.236∗[+0.202,+0.270]+0.236^*\ [+0.202,+0.270] −0.047∗[−0.062,−0.033]-0.047^*\ [-0.062,-0.033] +0.224∗[+0.187,+0.261]+0.224^*\ [+0.187,+0.261] BabiQA Adapt-ASMI (0.286) MSP (0.581) −0.215∗[−0.283,−0.148]-0.215^*\ [-0.283,-0.148] −0.295∗[−0.365,−0.228]-0.295^*\ [-0.365,-0.228] −0.227∗[−0.300,−0.157]-0.227^*\ [-0.300,-0.157] TriviaQA Adapt-ASMI (0.723) RAUQ-E (0.772) −0.048∗[−0.066,−0.030]-0.048^*\ [-0.066,-0.030] −0.049∗[−0.064,−0.033]-0.049^*\ [-0.064,-0.033] −0.056∗[−0.074,−0.037]-0.056^*\ [-0.074,-0.037] Mistral-7B CoQA Adapt-ASMI (0.537) SemEnt (0.536) +0.002[−0.018,+0.021]+0.002\ [-0.018,+0.021] +0.002[−0.018,+0.021]+0.002\ [-0.018,+0.021] −0.007[−0.028,+0.014]-0.007\ [-0.028,+0.014] SQuAD Adapt-ASMI (0.840) RAUQ-E (0.830) +0.037∗[+0.015,+0.059]+0.037^*\ [+0.015,+0.059] +0.010[−0.003,+0.023]+0.010\ [-0.003,+0.023] +0.025[−0.001,+0.051]+0.025\ [-0.001,+0.051] BabiQA Adapt-ASMI (0.607) MSP (0.593) +0.024[−0.029,+0.077]+0.024\ [-0.029,+0.077] +0.015[−0.032,+0.062]+0.015\ [-0.032,+0.062] +0.022[−0.032,+0.077]+0.022\ [-0.032,+0.077] TriviaQA Adapt-ASMI (0.784) SAR (0.795) +0.000[−0.013,+0.014]+0.000\ [-0.013,+0.014] −0.012[−0.026,+0.003]-0.012\ [-0.026,+0.003] −0.002[−0.016,+0.012]-0.002\ [-0.016,+0.012] Table 13: Head-to-head paired significance at the 60%60\% operating depth. Each entry is a paired Δ with a 95% percentile interval from a cluster-respecting paired bootstrap (B=10KB=10K, using the cluster keys and seed of Appendix I), positive when ASMI is ahead, with asterisks marking intervals that exclude zero. The first two Δ columns follow the column’s top ASMI variant and the last reports the sampling-free Sem-ASMI against Semantic Entropy (SemEnt), the pre-specified primary reference. The per-column best baseline is the conservative reference, taken over all 17 baselines, and RAUQ-E denotes the entropy variant of RAUQ. Cluster counts are 499499 on CoQA, 1,9341,934 on SQuAD, 999999 on BabiQA, and 4,4044,404 on TriviaQA. Point PRR values agree with the full benchmark tables within their standard errors. Instability of depth selection BabiQA is the one grounded benchmark whose PRR is non-monotonic in depth, with a pronounced dip at 80%80\% on the two Qwen backbones. For unweighted ASMI on Qwen3-8B it drops from 0.780.78 at 70%70\% to 0.480.48 at 80%80\% and recovers at 90%90\%, instead of following the smooth CoQA trend. The task explains this. BabiQA answers are a single location word, so the sequence score is carried by essentially one token and there is almost no averaging over positions to smooth depth effects, unlike CoQA where up to twenty tokens are averaged. The task is also solved by a localized copy of the last-mentioned location, so whether a given layer is masked interacts sharply with whether that routing step is perturbed, which produces a bad layer near 80%80\% for the Qwen models. The small evaluation set (n=1Kn=1K) then widens the standard error and amplifies these swings. Semantic weighting cushions the dip, raising the Qwen3-8B 80%80\% score from 0.480.48 for ASMI to 0.700.70 for Sem-ASMI, which indicates that much of the disagreement introduced at that layer is surface-form and is discounted by the kernel. Llama-2-7B is low at every depth on BabiQA for a distinct reason that the mechanism analysis pins down. Its generations are well-formed single location words, but head masking barely perturbs its output on this task. The token-level MItMI_t has median 0.0000.000 and mean 0.0020.002, against 0.0250.025 on Qwen3-8B, so the model is over-robust to the perturbation and insensitive along the attention path. Correct and incorrect answers therefore carry the same near-zero MI and their score distributions overlap almost completely (Fig. 7, top). ASMI retains only an attenuated ranking, while the output-distribution MSP still separates them (Table 12). On Qwen3-8B, where the model routes the task and wrong answers are more fragile, the wrong-answer distribution shifts upward (Fig. 7, bottom) and ASMI matches MSP. The MI magnitude flags this over-robustness before any correctness label is available. At the representative 60%60\% depth BabiQA is well-behaved on three of four backbones and enters the main comparison, but its depth profile is the least stable of the grounded family, consistent with the underpowered head-criticality coupling in the mechanism analysis. Appendix J Head-to-Head Significance The bootstrap standard errors of the main tables overstate the uncertainty of a comparison, because every method scores the same greedy responses under the same correctness labels. We therefore test differences directly with a cluster-respecting paired bootstrap. Each of the B=10KB=10K resamples draws one set of clusters with replacement, using the same cluster keys and seed as Appendix I, and applies the same indices to both methods, so the shared per-example variation cancels and the intervals are narrower than the marginal standard errors imply, by roughly half on CoQA. We report Δ with a 95% percentile interval and call a comparison a tie when the interval includes zero. Semantic Entropy is the pre-specified primary reference, fixed before running the test to avoid selecting the winner after the fact, and the per-column best baseline is the conservative reference, since the maximum over all 17 baselines is the hardest comparison for ASMI. Table 13 reports all sixteen columns. Against Semantic Entropy the top ASMI variant wins significantly in six of the twelve grounded columns, loses in two, and ties in four. Against the per-column best baseline ASMI leads in eight grounded columns, significantly in three (CoQA on both Qwen backbones and BabiQA on Qwen3-8B), and the remaining leads are ties. A tie carries information of its own here, since in every tied column the sampling-free Sem-ASMI ties or beats Semantic Entropy from the greedy response alone (last column of Table 13). One of the two losses against Semantic Entropy is the over-robust Llama-2-7B BabiQA cell the operating envelope already maps, and the other is Qwen3-8B on SQuAD. The largest single gap in the table, the +0.236+0.236 of the Llama-2-7B SQuAD column, coincides with the column where 28.5%28.5\% of greedy generations are empty (Appendix I), so we checked the tallies without it: over the remaining eleven grounded columns the top variant still wins five, loses two, and ties four against Semantic Entropy, and still leads the per-column best baseline in eight with three significant, so no headline count depends on this column. On parametric TriviaQA ASMI sits significantly below the best baseline in three of the four columns, so the designed boundary holds under the paired test as well. Appendix K Estimator Stability Protocol. The bootstrap SEs of the main tables measure sampling variability over test examples and treat each method’s scores as fixed. They do not measure the variability of the estimator itself, whose scores depend on internal randomness, namely sampled head masks for ASMI and stochastically decoded samples for sample-diversity baselines. We fix a 500-example CoQA subset, the backbone, and the scored greedy responses, and redraw only the method-internal randomness five times, using disjoint sets of S=40S=40 masks for the ASMI variants and independent resamplings of the 1010 stochastic generations (temperature 0.50.5 following 5, as configured in all main experiments) for the baselines. Because the examples are fixed, differences across redraws reflect measurement noise alone. Absolute PRR levels differ from the main tables (subset, single layer). This experiment measures variance, not level. Method mean sd range rank-ρ Adapt-ASMI .493.493 .006.006 [.486,.501][.486,.501] .98.98 Sem-ASMI .491.491 .004.004 [.486,.496][.486,.496] .98.98 ASMI .444.444 .011.011 [.432,.461][.432,.461] .96.96 Semantic Entropy .474.474 .020.020 [.455,.505][.455,.505] .94.94 SAR .477.477 .014.014 [.459,.493][.459,.493] .92.92 MC Sequence Entropy .450.450 .014.014 [.440,.470][.440,.470] .95.95 Kernel Language Entropy .427.427 .041.041 [.373,.479][.373,.479] .81.81 Table 14: Estimator stability: PRR across five redraws of method-internal randomness on a fixed 500-example CoQA subset. rank-ρ is the mean pairwise Spearman correlation of per-example scores across redraws. Results. Table 14 reports the across-redraw standard deviation of PRR and the mean pairwise Spearman correlation of the per-example scores. Adapt-ASMI varies by ±0.006± 0.006 across redraws and Sem-ASMI by ±0.004± 0.004, two to five times less than the sample-diversity baselines (from ±0.014± 0.014 to ±0.041± 0.041), and their example rankings are nearly deterministic (rank-ρ=0.98ρ=0.98). This follows from the design: ASMI scores a fixed greedy response under structural perturbations whose head coverage is guaranteed (Appendix G), whereas sample-diversity scores inherit the stochasticity of temperature decoding. With five redraws the sd estimates are coarse. Ranges are reported alongside. Appendix L Benchmark Prompts and Examples For reproducibility we show the prompt format and one representative example per benchmark, exactly as produced by the LM-Polygraph pipeline on base models with no chat template. Long contexts are truncated with […], and the gold answer follows each block. CoQA (grounded, 0-shot). A flattened story followed by a single question, with the answer stated in the story. ⬇ The following are stories and questions about them. Each story is followed by a question and answer to a given question. Story: Once upon a time, in a barn near a farm house, there lived a little white kitten named Cotton. [...] But she was the only white one in the bunch. [...] Then Cotton thought, "I like being special". Question: What color was Cotton? Answer: Gold answer: white. SQuAD (grounded, 0-shot). A passage with an extractive question whose answer is a span of the passage. ⬇ Read the following passage and answer the question. Passage: Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The AFC champion Denver Broncos defeated the NFC champion Carolina Panthers 24-10 to earn their third Super Bowl title. [...] Question: Which NFL team represented the AFC at Super Bowl 50? Answer: Gold answer: Denver Broncos. BabiQA (grounded, 1-shot). A synthetic location-tracking task with a fixed one-shot exemplar. The answer is a single word copied from the context. ⬇ Imagine that you are only able to say a single word. Answer the question given a context. You must only output the full name of the location the same way it is mentioned in the text. Example: Context: Mary moved to the bathroom. John went to the hallway. [...] Question: Where is Sandra? Answer: bathroom Context: John travelled to the hallway. Mary journeyed to the bathroom. Question: Where is John? Answer: Gold answer: hallway. TriviaQA (parametric, closed-book, 5-shot). Five fixed exemplars followed by the query, with no passage provided. ⬇ Question: What is the capital of the Indian state of Tamil Nadu? Answer:chennai Question: Who rules Narnia following the reign of High King Peter [...]? Answer:prince caspian [... three more fixed exemplars ...] Question: Who was the man behind The Chipmunks? Answer: Gold answer: David Seville, which must come from parametric knowledge.