Paper deep dive
CRAFT: Causal Responsibility and Failure Tracing in Medical Vision Language Models
Chunzheng Zhu, Jiaqi Zeng, Hongbo Zhao, Yihang Chen, Yijun Wang, Jianxin Lin
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 10/4/2026, 3:09:34 AM
Summary
The paper introduces CRAFT, a mechanistic interpretability framework for Medical Vision Language Models (VLMs) that identifies and intervenes on two specific failure modes: arbitration failure (where misleading text overrides correct visual diagnosis) and brake failure (where the model commits to an answer despite insufficient visual evidence). The study localizes these failures to spatially disjoint attention head populations: arbitration heads in mid-to-deep layers and brake heads in middle-to-late layers. By excising these specific heads, the authors demonstrate that they can reduce conflict following and restore appropriate abstention behavior without retraining or significantly degrading performance on clean inputs.
Entities (11)
Relation Signals (10)
CRAFT → addresses → Brake Failure
confidence 95% · excising brake heads restores appropriate abstention under degraded visual evidence
CRAFT → addresses → Arbitration Failure
confidence 95% · CRAFT localizes each failure mode... excising arbitration heads sharply reduces conflict following
Arbitration Failure → mediatedby → Arbitration Heads
confidence 93% · arbitration failure... are mediated by spatially disjoint attention head populations: arbitration heads
Brake Failure → mediatedby → Brake Heads
confidence 93% · brake failure... are mediated by... brake heads concentrate in a narrow middle-to-late layer band
CRAFT → evaluatedon → Hulu-Med 4B
confidence 92% · Experiments across multiple medical VQA benchmarks and VLM architectures... using Hulu-Med 4B
CRAFT → evaluatedon → InternVL3.5-4B
confidence 92% · using Hulu-Med 4B... and InternVL3.5 4B
Brake Heads → locatedin → middle-to-late layers
confidence 90% · brake heads concentrate in a narrow middle-to-late layer band
Arbitration Heads → locatedin → mid-to-deep layers
confidence 90% · arbitration heads form a mid-to-deep wideband
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:As vision language models are increasingly deployed in clinical diagnosis, understanding how they internally resolve competing visual and textual signals becomes a safety imperative. Existing mechanistic analyses remain confined to unimodal text and offer no explanation for why a single misleading sentence can override a correct image based diagnosis, or why a model commits to a confident answer despite insufficient visual evidence. We find that these two safety risks, arbitration failure where textual context overrides visual grounding and brake failure where the model commits without adequate evidence, are mediated by spatially disjoint attention head populations: arbitration heads form a mid-to-deep wideband reflecting cross-layer evidence competition, while brake heads concentrate in a narrow middle-to-late layer band that regulates evidence sufficiency and abstention behavior. To ground these observations in causal circuitry, we introduce CRAFT, which localizes each failure mode to a minimal causal head set via dual criteria and verifies necessity and sufficiency through temporal probes and Tuned Lens trajectory analysis. Excising arbitration heads sharply reduces conflict following with negligible degradation on clean inputs, while excising brake heads restores appropriate abstention under degraded visual evidence. The two interventions target spatially disjoint head sets and produce distinct corrective effects, underscoring the mechanistic separability of the failure modes. Experiments across multiple medical VQA benchmarks and VLM architectures validate both the localization and interventions, demonstrating that the identified heads causally drive each failure mode and that targeted modulation generalises without retraining. The code is available at GitHub repository.
Tags
Links
- Source: https://arxiv.org/abs/2609.38810v1
- Canonical: https://arxiv.org/abs/2609.38810v1
Trouble viewing inline? Open PDF directly →
Full Text
122,096 characters extracted from source content.
Expand or collapse full text
CRAFT: Causal Responsibility and Failure Tracing in Medical Vision Language Models Chunzheng Zhu Affiliation: Hunan University Jiaqi Zeng Affiliation: Hunan University Hongbo Zhao Affiliation: Hunan University Yihang Chen Affiliation: University of Hong Kong Yijun Wang Affiliation: Hunan University Jianxin Lin Affiliation: Hunan University *Equal contribution. Corresponding author. Abstract As vision language models are increasingly deployed in clinical diagnosis, understanding how they internally resolve competing visual and textual signals becomes a safety imperative. Existing mechanistic analyses remain confined to unimodal text and offer no explanation for why a single misleading sentence can override a correct image based diagnosis, or why a model commits to a confident answer despite insufficient visual evidence. We find that these two safety risks, arbitration failure where textual context overrides visual grounding and brake failure where the model commits without adequate evidence, are mediated by spatially disjoint attention head populations: arbitration heads form a mid-to-deep wideband reflecting cross-layer evidence competition, while brake heads concentrate in a narrow middle-to-late layer band that regulates evidence sufficiency and abstention behavior. To ground these observations in causal circuitry, we introduce Craft, which localizes each failure mode to a minimal causal head set via dual criteria and verifies necessity and sufficiency through temporal probes and Tuned Lens trajectory analysis. Excising arbitration heads sharply reduces conflict following with negligible degradation on clean inputs, while excising brake heads restores appropriate abstention under degraded visual evidence. The two interventions target spatially disjoint head sets and produce distinct corrective effects, underscoring the mechanistic separability of the failure modes. Experiments across multiple medical VQA benchmarks and VLM architectures validate both the localization and interventions, demonstrating that the identified heads causally drive each failure mode and that targeted modulation generalises without retraining. The code is available at GitHub repository. 1 Introduction Medical vision language models are increasingly deployed for clinical question answering and diagnostic support [3, 5, 37], yet their behavior can change sharply when visual evidence, textual context, and parametric medical knowledge disagree [40, 35]. For safety critical deployment, observing these failures at the output level is insufficient: we need tools for identifying which internal components are sensitive to each failure mode and whether targeted interventions can correct the behavior without damaging clean diagnostic reasoning. Mechanistic interpretability provides a natural starting point [8, 6]. Prior work has mapped specialized attention heads in language models, including memory heads for parametric recall [16], context heads for in-context integration [19, 13], and induction heads for token copying [32], while activation patching studies have localized factual recall to specific MLP modules [23, 31]. Recent extensions trace intra memory knowledge conflicts to MLP attention interactions [28] and reveal that parametric versus contextual signals can superpose within the same head [19]. However, these analyses focus exclusively on text only LLMs and binary knowledge conflicts. Medical VLMs introduce an additional axis of competition: the model must arbitrate among image evidence, textual context, and learned medical priors, and it should sometimes abstain when visual evidence is insufficient. Cross modal competition therefore transcends the binary “follow context vs. retrieve memory” choice studied in prior work [22, 38, 30], giving rise to a tripartite interaction among three causal pathways: a visual anchoring path vP_v, a parametric knowledge path ΘP_ , and a textual override path tP_t (Fig. 1). This three way competition creates two mechanistically distinct safety risks: (i) Arbitration failure: misleading text contradicts a correct visual diagnosis and the model erroneously privileges text, e.g., a chest radiograph showing clear pneumonia infiltrates accompanied by “no cough reported” yields the output non-pneumonia. (i) Brake failure: visual evidence degrades below clinical sufficiency yet the model commits to a definitive answer rather than abstaining, e.g., a dermoscopy image with the lesion region masked still prompts a specific diagnosis. Figure 1: Two failure modes in medical VLMs and Craft interventions. Left (Arbitration Failure): Contradictory text hijacks a correct visual diagnosis, causing the model to follow the misleading answer. Right (Brake Failure): The model commits despite occluded evidence. Both failures produce unsafe clinical outputs but reflect fundamentally different computational regimes: arbitration failure indicates a model that possesses the correct answer but is hijacked by a competing textual signal; brake failure indicates one that lacks sufficient evidence yet commits regardless. Prior work documents these phenomena at the output level [22, 40] and proposes training free mitigations via contrastive decoding [41] or image grounded guidance [42], but has not addressed where in the computation graph correct reasoning is overridden. Purely observational head rankings conflate causal influence with correlation: a head active during conflicted outputs may be predictive without being a useful intervention target [24, 11]. Three open challenges remain: (C1) whether the two failure modes share circuitry or exhibit separable intervention sensitive patterns; (C2) how to identify heads whose manipulation changes failure behavior rather than merely predicts it; (C3) whether failure specific circuits can be intervened upon without degrading clean reasoning. We introduce Craft, an activation patching framework for tracing and intervening on these two failure modes in medical VLMs. In the first stage, Craft identifies a critical layer band through layer wise conflict scoring, then isolates within that band the minimal head sets whose causal effect on the targeted failure is large while their perturbation to clean input reasoning remains small. In the second stage, linear probes and Tuned Lens trajectories [4] provide complementary temporal evidence about when conflict signals become linearly decodable and when answer preferences shift across layers. Our principal contributions are: ⋄ We formalize two medical VLM failure modes: arbitration failure under textual conflict and brake failure under visual degradation, separating textual override from insufficient evidence commitment within a unified structural causal framework. ⋄ We propose Craft, which narrows the causal search to a critical layer band via conflict scoring, then isolates a minimal head set whose excision maximally reduces the targeted failure while preserving clean capacity. The resulting targets are failure specific: arbitration head excision suppresses textual override; brake head excision restores abstention under degraded evidence. ⋄ On multiple VLMs, our experiments reveal that arbitration-sensitive heads form a mid-to-deep band while brake-sensitive heads concentrate in a disjoint late-layer band. Targeted interventions reduce Conflict Flip Rate (CFRCFR) by up to 41 absolute points and raise Uncertainty Rate (URUR) under insufficient evidence by 75%, with minimal collateral damage on clean diagnostic accuracy. 2 Related Work Mechanistic Interpretability and Knowledge Conflicts. Circuit discovery via activation patching [23, 6], path patching [10], and attribution based head analysis [1] have identified specialized attention heads for factual recall and in context retrieval [26, 36]. These tools have been extended to knowledge conflict settings: locating “memory heads” whose ablation restores context faithfulness [16], revealing superposition of parametric and contextual signals within the same head [19], and tracing intra memory conflicts to specific MLP attention interactions [28]. Inference time intervention methods such as ITI [20], RepE [44], and conceptor based steering [29] demonstrate that activation modulation can redirect model behavior without weight updates, but operate on coarse trait directions rather than failure specific circuits. Critically, all above studies address text only LLMs; initial mechanistic analyses of VLMs do not study multimodal evidence conflict or clinical failure modes. Medical VLM Safety and Selective Prediction. Medical VLMs achieve strong radiology QA performance [18, 21, 43] yet remain susceptible to hallucination under ambiguous or degraded inputs [27, 14]. Recent training free mitigation strategies include contrastive decoding guided by expert models [41], image grounded guidance from open source detectors [42], and probing internal representations to predict hallucination risk before token generation [17]. Orthogonally, the selective prediction framework [9] and LLM abstention literature [34] advocate withholding answers when confidence is low, but current implementations rely on output calibration or prompt heuristics rather than mechanistic intervention. Our work bridges these threads by providing a training free mechanism that directly modulates the internal circuits responsible for over commitment, enabling selective abstention without retraining, while acknowledging that clinically valid abstention thresholds require deployment specific human in the loop validation. Figure 2: Interventional tracing setup. (a) The evidence conflict problem can be modeled by a structural causal model with three competing paths: visual anchoring vP_v, parametric knowledge ΘP_ , and textual override tP_t, plus latent confounders UcU_c that open back-door paths. (b) Path dominance under each failure mode. (c) Clean-state interchange blocks back-door paths through UcU_c while preserving the front-door visual causal path, isolating genuine causal effects for head selection. 3 Background and Preliminaries 3.1 Transformer Residual Structure Given a medical image v and a textual query q, a VLM processes the concatenated visual and textual token sequence through an L-layer decoder. Let l,th_l,t denote the residual stream at layer l and token position t, and let tanst_ans denote the answer-token position used for next-token prediction. At layer l, the residual stream accumulates contributions from nhn_h parallel heads and a feed-forward network: l,t=l−1,t+∑i=1nhOl,il,i,t+FFNl(l−1,t),h_l,t\;=\;h_l-1,t\;+\; _i=1^n_hW_O^l,i\,a_l,i,t\;+\;FFN_l(h_l-1,t), (1) where l,i,t=Attnl,i(l−1)ta_l,i,t=Attn^l,i(h_l-1)_t is the value aggregated by head i at position t. The additive structure permits zeroing any individual head’s output Ol,il,i,t↦W_O^l,i\,a_l,i,t 0 without compensatory redistribution among remaining heads [8, 31], forming the basis for per-head activation interventions that modify a single additive term while leaving the rest of the forward pass unchanged. 3.2 Two Failure Modes and Problem Formulation Structural view. We posit exogenous variables =Xv,Xt,Θ,UcU=\X_v,X_t, ,U_c\, where XvX_v denotes the visual input (with X~v X_v denoting its degraded counterpart in the brake setting), XtX_t the textual input, Θ the model parameters, and UcU_c unobserved exogenous noise capturing latent confounders (e.g., lexical overlap, visual ambiguity, cross-modal coupling). The endogenous variables are =1,…,L,YV=\h_1,…,h_L,Y\, where lh_l is the hidden representation at layer l and Y is the model output. These variables are governed by the structural equations l=fl(l−1,Xv,Xt,Θ,Uc)h_l=f_l(h_l-1,X_v,X_t, ,U_c) and Y=g(L,tans)Y=g(h_L,t_ans), where tanst_ans is the answer token position and fl,gf_l,g are deterministic functions. The induced directed acyclic graph (Fig. 2a) admits three competing directed paths from sources to outcome: visual anchoring vP_v (information flow from XvX_v), parametric knowledge ΘP_ (from Θ ), and textual override tP_t (from XtX_t). Preference margin. We formalize both failure modes through δ=logp(y−)−logp(y+)δ= p(y^-)- p(y^+), where y+y^+ denotes the correct answer, y−y^- the incorrect answer, and p(y)p(y) is the model-assigned probability. A positive margin δ>0δ>0 indicates the model favours the wrong answer. For multi-token answers, logp(y) p(y) denotes the length-normalised log-likelihood under teacher forcing; for multiple-choice datasets, y corresponds to the candidate option token. Two mechanistically distinct failures arise from evidence conflict: Arbitration failure (tP_t dominates vP_v and ΘP_ ): misleading textual evidence XtX_t contradicts visual and parametric sources. The model possesses the correct answer under clean conditions (δclean<0 _clean<0, where δclean _clean is the margin under non-conflicting inputs) yet is hijacked under conflict (δconflict _conflict, the margin under conflicting inputs) (Fig. 2b, left). Brake failure (vP_v insufficient): degraded visual evidence Xv=xvdegradeX_v=x_v^degrade cannot sustain a diagnosis, yet the model commits to a concrete answer ycom∈cony^com _con (the set of concrete answers) rather than abstaining with yunky^unk (an “unknown” or abstention option) (Fig. 2b, right). Formally: (Arbitration) δconflict−δclean>0,i.e., conflict shifts preference toward y−, _conflict- _clean>0, i.e., conflict shifts preference toward y^-, (2) (Brake) logp(ycom)>logp(yunk),ycom∈con,;Xv=xvdegrade. p(y^com)> p(y^unk), y^com _con,;X_v=x_v^degrade. (3) 3.3 Interventional Tracing Setup The confounding problem. Naïve observational attribution (e.g., ranking heads by activation magnitude) conflates causation with correlation. In the SCM of Fig. 2(a), the latent variable UcU_c opens back-door paths hl←Uc→Yh_l← U_c→ Y: token-level shortcuts inflate apparent head importance, visual ambiguity causes widespread activation variance correlated with Y through UcU_c rather than through vP_v, and cross-modal coupling via shared normalisation creates statistical dependence that does not reflect causal influence. The observational quantity [Y∣l,i]E[Y _l,i] therefore conflates the direct causal effect h→Yh→ Y with the spurious association h←Uc→Yh← U_c→ Y. Deconfounding via clean-state interchange. To block these back-door paths, we exploit the modularity of structural causal models: intervening on an endogenous variable severs all incoming arrows while preserving downstream equations [7]. For each conflict input we construct a paired conflict-free counterfactual sharing the same (Xv,q)(X_v,q) and cache its clean activation lcleanh_l^clean. The interchange intervention do(hl:=hlclean)do(h_l:=h_l^clean) fixes the head output independently of the current UcU_c realisation, thereby d-separating hlh_l from UcU_c and blocking all back-door paths (Fig. 2c). Any residual change in Y is then attributable solely to the conflict mechanism mediated by the intervened head rather than to confounding (Appendix 9 for the d-separation proof). Per-head activation scaling. We index heads by h=(l,i)h=(l,i) and write ⊆[L]×[nh]C [L]×[n_h] for a head set with l=i:(l,i)∈C_l=\i:(l,i) \. For a set C we define additive head-level scaling at all token positions: Y(α)=g(L,tans|l,t←l,t+∑i∈l(αl,i−1)Ol,il,i,t,∀l,t),Y_C^(α)\;=\;g (h_L,t_ans\; |\;h_l,t _l,t+ _i _l( _l,i-1)\,W_O^l,i\,a_l,i,t,\;∀ l,t ), (4) where αh=ϵ _h\!=\!ε with ϵ→0+ε\!→\!0^+ realises near-complete suppression (denoted excise()excise(C), used for both arbitration and brake heads). Because the intervention fixes head outputs independently of the current realisation of UcU_c, the measured effect reflects the deconfounded causal contribution of each head to the targeted failure mode. In this work, “intervention” consistently refers to this activation-level do-operation rather than to a claim about the input-level data-generating process. localization objective. Equipped with these two deconfounded operations, we identify sparse intervention targets whose interventions improve the targeted failure mode while limiting collateral damage. Rather than treating the selection boundaries or the number of selected targets as pre-specified hyperparameters, we obtain them empirically from the observed intervention profiles after scanning candidate heads or layers: arb∗ _arb^* =ParetoKneeCFR(excise())↓,C2W(excise())↓,||↓, =ParetoKnee_C \CFR(excise(C)) ,\;C2W(excise(C)) ,\;|C| \, (5) ℒbrk∗ _brk^* =ParetoKneeℒUR(attenuate(ℒ;β))↑,U2O(attenuate(ℒ;β))↓,|ℒ|↓. =ParetoKnee_L \UR(attenuate(L;β)) ,\;U2O(attenuate(L;β)) ,\;|L| \. (6) Here, arb∗C_arb^* denotes the selected arbitration heads, while ℒbrk∗L_brk^* denotes the selected brake-related layers under layer-wise attenuation with β<1β<1. CFRCFR is the proportion of conflict samples following the misleading text, URUR is the fraction of degraded samples where the model correctly abstains, and C2WC2W/U2OU2O capture collateral damage on clean inputs. The effective selection boundaries and the resulting number of selected targets are therefore empirical outcomes of the localization procedure, rather than manually fixed hyperparameters (Appendix 8.2 for definitions). Figure 3: Arbitration failure localization. (a) Layer-wise S(l)S(l): shaded window marks layers where override concentrates. (b) CER vs. BCP: red points (upper left) are selected causal key heads. 4 Causal Tracing of Failure Circuitry We trace each failure mode to its causal origin through a coarse to fine procedure: scoring layers to identify a critical band, then resolving heads within that band driving override circuitry (arbitration failure) and gating circuitry (brake failure). Both stages apply the deconfounded intervention of Sec. 3.3 to ensure selected components are causal drivers rather than observational correlates. 4.1 Arbitration Failure Method. Given a visual input xvx_v, a question q, and conflicting text xtconfx_t^conf asserting a misleading answer y−y^-, the preference margin δ=logpθ(y−∣xv,xtconf,q)−logpθ(y+∣xv,xtconf,q)δ= p_θ(y^- x_v,x_t^conf,q)- p_θ(y^+ x_v,x_t^conf,q) quantifies textual hijacking, where δ>0δ>0 means the model follows the misleading text. We first narrow the search space via a layer wise conflict score. Let δbase _base denote the preference margin under clean input and δconflict _conflict that under adversarial text. For each layer l, we cache the clean run hidden state and replace the layer l output in the conflict forward pass with this cached representation, yielding a patched margin δpatched(l) _patched(l). The conflict score is then: S(l)=|δconflict−δbase|−|δpatched(l)−δbase|,S(l)=| _conflict- _base|-| _patched(l)- _base|, (7) measuring how much patching layer l reduces the conflict effect toward the clean baseline. The critical band W is identified from the S(l)S(l) profile by selecting layers where the score rises sharply and remains elevated, corresponding to layers that actively sustain the textual override. Within W, we isolate individual heads using two causal metrics: CERexc(h)=δconflict−δexcise(h)CER_exc(h)=δ^conflict-δ^excise(h), measuring how much excising head h reduces the override, and BCPexc(h)=Accclean−Accexcise(h)BCP_exc(h)=Acc^clean-Acc^excise(h), measuring collateral damage on clean inputs. The arbitration critical head set is: arb∗=h:CERexc(h)>τs∧BCPexc(h)<τd,C^*_arb=\h:CER_exc(h)> _s\; \;BCP_exc(h)< _d\, (8) retaining only heads causally responsible for the override yet dispensable for normal reasoning. Experiments and Analysis. Figure 4: Arbitration head localization (Hulu-Med 4B). Left: conflict heads (blue) cluster in the Override Band with high CER and low BCP; the Late Commitment Tail shows negligible conflict or fidelity circuitry, with residual score reflecting distributed output formation. Centre: layer×head grid with per layer CER and BCP bars. Right: selection criterion and recovery example. We evaluate on VQA-RAD [18] and SLAKE [21] using Hulu-Med 4B [14] (QwenVL [2] backbone) and InternVL3.5 4B [33]; dataset construction and conflict injection details are provided in Sec. 8. Fig. 3(a) shows that S(l)S(l) peaks broadly across mid to deep layers (layers 12–17 in Hulu-Med; 14–20 in InternVL), confirming that textual override distributes across a wideband rather than localizing to a single layer. This contrasts with unimodal factual recall, which concentrates in narrow MLP bands [23]. Fig. 3(b) reveals a sparse cluster in the upper left quadrant: heads with strong causal effect on the conflict yet minimal baseline disruption, constituting fewer than 1% of total heads. Fig. 4 further resolves the spatial structure: conflict heads form a continuous band spanning layers 12–22, while the deepest layers (32–35) exhibit per head CER near zero with negligible conflict or fidelity circuitry, their elevated S(l)S(l) reflecting distributed output formation rather than sparse causal heads. This dichotomy persists across architectures, suggesting a shared motif: mid layer override wideband functionally distinct from late layer output formation. 4.2 Brake Failure localization Figure 5: Brake head localization (Hulu-Med 4B). (a) HR(l) map with positive peaks at the deepest layers. (b,c) BCP versus HR landscape separating different heads from backbone. Method. For brake failure, visual evidence is degraded such that the correct response is abstention yunky^unk, yet the model commits to a concrete answer ycomy^com. We apply the same causally grounded deconfounded intervention logic: we first identify layers whose attenuation shifts probability mass toward abstention under degraded input, then localize individual heads within those layers for excision. Following the same deconfounded intervention protocol described above, the layer-wise hallucination relief score quantifies how much scaling down all head outputs in a given layer l by a factor β<1β<1 shifts the model’s output probability mass toward the abstention token under degraded visual input: HR(l)=pθ(yunk∣attenuate(l))−pθ(yunk∣original).HR(l)=p_θ(y^unk (l))-p_θ(y^unk ). (9) A positive HR(l)HR(l) indicates that weakening layer l helps the model recognise insufficient visual evidence and suppress premature commitment. Within high HR layers, we isolate individual brake heads by jointly requiring that per head hallucination relief HRexc(h)HR_exc(h) exceeds a suppression bound while the collateral metric BCPexc(h)=Accclean−Accexc(h)BCP_exc(h)=Acc^clean-Acc^exc(h) remains below a preservation bound, yielding the candidate set brk∗C^*_brk. The interpretation mirrors arbitration: we seek heads that are causally responsible for suppressing abstention yet dispensable for normal diagnostic reasoning. Experiments and Analysis. We construct degraded inputs by masking diagnostically critical regions and set abstention as the target (protocol in Sec. 8). Fig. 5(a) reveals a spatial signature distinct from arbitration: while the arbitration override distributes across a broad mid to deep wideband, the hallucination relief signal is sharply focal, concentrating in layers 31–35 of Hulu-Med and layers 25–35 of InternVL3.5. This narrow band accounts for over 78% of total HR improvement despite spanning fewer than 15% of model depth, indicating that the abstention mechanism is governed by a compact, late-layer circuit. At head level (Fig. 5(b, c)), only 4–7 heads per model carry statistically significant brake signal (p<0.01p<0.01, permutation test). Excising these heads yields HR gains of 34.2 (Hulu-Med) and 29.7 (InternVL3.5) with BCP below 2.1, restoring appropriate uncertainty without degrading correct answers. Notably, the selected brake layers are entirely disjoint from the arbitration critical band, corroborating that conflict resolution and confidence calibration are handled by separate sub-circuits. Tuned Lens trajectory analysis further shows that brake heads act as late redirectors: the output distribution commits to a hallucinated token by layer 24, and brake heads enforce premature commitment only in the deepest layers, explaining why shallow probe methods fail to detect. 5 Functional Roles and Decoupling Analysis Having localized intervention-sensitive heads, we evaluate whether the two failure modes respond to distinct interventions and examine the functional specialisation of the identified head sets. 5.1 Decoupling of Arbitration and Brake Interventions Setup We compare Craft against random ablation using the same head count as |∗||C^*|. For VQA experiments, evaluation spans four medical VLMs (Hulu-Med 4B, Hulu-Med 7B, InternVL3.5 4B, and Qwen3-VL-8B [2]) on VQA-RAD and SLAKE for textual conflict, and on SLAKE and HealMed-VQA [25] for image conflict. More backbone and task results are reported in Appendix 17. Table 1: Arbitration failure on multimodal benchmarks under textual conflict. CFRCFR: conflict following rate (↓ ); Resist: resist rate (↑ ); C2WC2W: correct to wrong (↓ ); W2C: wrong to correct (↑ ). Bold: best comparison result. All CRAFT gains are significant (p<0.01p<0.01). Prefix Before Q Before A Benchmark Model Method CFR↓ Resist↑ C2W↓ W2C↑ CFR↓ Resist↑ C2W↓ W2C↑ CFR↓ Resist↑ C2W↓ W2C↑ VQA-RAD Hulu-Med 4B Baseline 33.33 66.67 N/A N/A 44.62 55.38 N/A N/A 98.21 1.79 N/A N/A Random 33.59 66.41 1.79 1.54 40.77 59.23 1.03 4.87 98.72 1.28 0.51 0.00 Craft 15.38 84.62 0.26 18.21 14.36 85.64 0.51 30.77 70.26 29.74 0.00 27.95 Hulu-Med 7B Baseline 25.80 74.20 N/A N/A 36.50 63.50 N/A N/A 88.00 12.00 N/A N/A Random 25.40 74.60 0.90 1.30 35.20 64.80 0.80 2.10 88.30 11.70 0.35 0.05 Craft 14.20 85.80 0.10 11.70 14.70 85.30 0.20 22.00 67.50 32.50 0.00 20.50 Hulu-Med 14B Baseline 13.80 86.20 N/A N/A 10.60 89.40 N/A N/A 24.20 75.80 N/A N/A Random 14.10 85.90 0.50 0.20 10.40 89.60 0.30 0.50 24.90 75.10 0.80 0.10 Craft 8.60 91.40 0.40 5.60 6.90 93.10 0.25 3.95 16.80 83.20 0.20 7.60 InternVL3.5 4B Baseline 30.29 69.71 N/A N/A 21.71 78.29 N/A N/A 60.57 39.43 N/A N/A Random 34.29 65.71 4.57 0.57 24.00 76.00 4.00 1.71 77.71 22.29 17.71 0.57 Craft 9.14 90.86 0.57 21.71 7.43 92.57 2.29 16.57 19.43 80.57 0.00 41.14 Qwen3-VL-8B Baseline 27.84 72.16 N/A N/A 16.16 83.84 N/A N/A 57.98 42.02 N/A N/A Random 28.10 71.90 0.90 0.64 17.00 83.00 1.00 0.16 58.20 41.80 0.35 0.13 Craft 11.90 88.10 0.20 16.14 7.50 92.50 0.15 8.81 34.00 66.00 0.10 24.08 SLAKE Hulu-Med 4B Baseline 19.15 80.85 N/A N/A 19.15 80.85 N/A N/A 87.73 12.27 N/A N/A Random 21.28 78.72 2.95 0.82 20.79 79.21 4.09 2.45 91.00 9.00 4.26 0.98 Craft 12.11 87.89 5.07 12.11 7.04 92.96 0.65 12.77 46.32 53.68 4.91 46.32 Hulu-Med 7B Baseline 15.40 84.60 N/A N/A 14.20 85.80 N/A N/A 75.00 25.00 N/A N/A Random 15.90 84.10 1.00 0.50 14.80 85.20 0.90 0.30 76.20 23.80 1.50 0.30 Craft 11.00 89.00 1.50 5.90 6.70 93.30 0.40 7.90 45.00 55.00 1.00 31.00 Hulu-Med 14B Baseline 8.80 91.20 N/A N/A 6.90 93.10 N/A N/A 10.40 89.60 N/A N/A Random 9.10 90.90 0.50 0.20 7.10 92.90 0.40 0.20 10.80 89.20 0.60 0.20 Craft 6.20 93.80 0.40 3.00 5.20 94.80 0.30 2.00 4.90 95.10 0.40 5.90 InternVL3.5 4B Baseline 12.66 87.34 N/A N/A 8.93 91.07 N/A N/A 16.40 83.60 N/A N/A Random 18.18 81.82 8.28 2.76 18.51 81.49 10.71 1.14 23.70 76.30 8.77 1.46 Craft 6.66 93.34 0.65 6.66 5.84 94.16 0.65 3.73 5.84 94.16 2.27 12.82 Qwen3-VL-8B Baseline 15.60 84.40 N/A N/A 11.80 88.20 N/A N/A 45.20 54.80 N/A N/A Random 16.20 83.80 0.85 0.25 12.40 87.60 0.75 0.15 47.00 53.00 2.10 0.30 Craft 10.80 89.20 0.30 5.10 7.20 92.80 0.25 4.85 23.20 76.80 0.30 22.30 Table 1 reports the effect of excising identified arbitration heads. The critical observation is not merely that CFR decreases, but how: the intervention shifts the model from following conflicting text to resisting it, while leaving clean accuracy virtually unchanged (C2W near zero). This aligns with the causal interpretation from Sec. 3.3: excising arbitration heads removes the tP_t override without disrupting the front door path vP_v. Random ablation, by contrast, increases CFR in several conditions (e.g., InternVL3.5 Before A), confirming that indiscriminate head removal can exacerbate failure by disrupting backbone reasoning while leaving the override intact. A further insight emerges from conflict position dependence: Before A shows the highest baseline CFR and strongest resistance to intervention, suggesting that when conflicting text is placed immediately before the answer token, the override exploits a shorter residual path beyond the identified head set. Table 2 reports the corresponding results under visual degradation. Ablating the brake heads identified via the localization procedure in Sec. 4.2 consistently increases the unknown rate across all four models while keeping U2O collateral at zero or near zero, with the improvement driven by larger O2U transitions that redirect non-abstaining predictions toward unknown. Random ablation is unstable, yielding smaller gains on Hulu-Med and Qwen3-VL-8B and even reducing UR on SLAKE for InternVL3.5 while introducing nonzero U2O, confirming that mitigating brake failure requires targeted suppression of the localized commitment heads rather than generic perturbation. Table 2: Brake failure under visual degradation. Craft excises the localized brake heads to recover abstention. UR: unknown rate (↑ ); O2U: non-unknown to unknown (↑ ); U2O: unknown to other (↓ ). Hulu-Med 4B Hulu-Med 7B Hulu-Med 14B InternVL3.5 4B Qwen3-VL-8B Benchmark Method UR↑ O2U↑ U2O↓ UR↑ O2U↑ U2O↓ UR↑ O2U↑ U2O↓ UR↑ O2U↑ U2O↓ UR↑ O2U↑ U2O↓ SLAKE Baseline 37.88 – – 44.12 – – 48.90 – – 22.58 – – 42.42 – – Random 45.45 8.33 0.76 48.82 5.29 0.59 51.10 3.00 0.80 16.13 0.00 6.45 47.73 6.06 0.75 Craft 65.15 27.27 0.00 61.18 17.06 0.00 58.40 10.20 0.70 32.26 9.68 0.00 55.30 12.88 0.00 HealMed-VQA Baseline 78.44 – – 85.07 – – 88.61 – – 81.56 – – 84.72 – – Random 86.83 10.18 1.80 88.19 4.31 1.19 89.58 1.45 0.48 82.47 3.06 2.15 87.50 4.17 1.39 Craft 98.20 19.76 0.00 95.83 11.10 0.34 93.06 4.80 0.35 93.64 12.46 0.38 95.14 11.11 0.69 Table 3: Comparison of selection criteria at equal head count. Left: arbitration failure (CER only: highest conflict effect; BCP only: lowest collateral). Right: brake failure on validation split (HR only: highest hallucination relief; BCP only: lowest collateral). Shaded: CRAFT. Model Criterion CFR↓CFR |ΔCFR|↑| | C2W↓C2W #Heads Hulu-Med 4B CER only 6.7 37.9 1.8 6 BCP only 44.4 0.3 1.3 6 Dual (Ours) 14.4 30.3 0.5 6 InternVL3.5 4B CER only 9.7 12.0 4.0 10 BCP only 7.4 14.3 2.3 10 Dual (Ours) 7.4 14.3 0.0 10 Model Criterion UR↑UR Δ(UR)↑ (UR) U2O↓U2O #Heads Hulu-Med 4B HR only 65.2 27.3 0.0 6 BCP only 38.6 0.8 0.0 6 Dual (Ours) 65.2 27.3 0.0 6 InternVL3.5 4B HR only 61.3 38.7 21.6 7 BCP only 16.1 -6.5 6.5 7 Dual (Ours) 32.3 9.7 0.0 7 5.2 Contribution of Individual Selection Objectives Table 3 compares selection strategies at equal candidate count. For arbitration, CER only achieves the strongest raw CFR reduction on Hulu-Med 4B (|ΔCFR|=37.9| |=37.9) but incurs C2W of 1.8, whereas BCP nearly fails to suppress conflict; the dual criterion balances both objectives, yielding substantial CFR reduction with C2W at 0.5. On InternVL3.5 4B, CER only increases C2W to 4.0, revealing entanglement between high CER candidates and clean reasoning pathways, while the dual criterion matches the CFR reduction of BCP only and eliminates all collateral (C2W==0.0). For brake failure, HR only and dual coincide on Hulu-Med 4B, but on InternVL3.5 4B HR only causes severe U2O degradation (21.6%) whereas the dual criterion keeps U2O at 0.0 with cleaner UR gain. These results confirm that the joint criterion is essential for reliable intervention across both failure modes. 6 Causal Verification via Internal Probes The localization procedure in §4 identifies heads via interventional effects. We verify these correspond to interpretable representational transitions using temporal probing and Tuned Lens [4] trajectory analysis on Hulu-Med 4B (VQA-RAD for arbitration, SLAKE for brake failure). 6.1 Temporal Probe Separation Figure 6: Temporal probe separation. Probe A (detection) saturates before Probe B (commitment), revealing a detect then commit structure. Post-intervention selectively suppresses Probe B while preserving Probe A. We train two binary logistic probes on per-layer hidden states at the answer position. Probe A (conflict detection) predicts whether the input contains conflict-inducing evidence. Probe B (failure prediction) predicts whether the model will enter the corresponding failure mode. If the identified heads mediate a transition from evidence registration to failure commitment, we expect three testable predictions: (P1) Probe A becomes accurate before Probe B across layers; (P2) Probe B peaks in later layers; (P3) intervening on ∗C^* selectively weakens Probe B more than Probe A. i) For arbitration failure (Fig. 6(a)), Probe A saturates around layers 17–21 (AUROC >0.97>0.97), while Probe B peaks 3–5 layers later (Macro F1 ≈ 0.85–0.87), delineating a computational window where the model has registered the conflict but has not yet committed to following it. After intervening on arb∗C_arb^*, Probe A remains intact whereas Probe B drops substantially, confirming that the identified heads govern commitment rather than conflict encoding. i) For brake failure (Fig. 6(b)), Probe A is predictive from shallow layers onward, indicating that degraded visual evidence is linearly decodable early. Probe B peaks later (layers 22–27) with greater variance and lower ceiling. After intervening on brk∗C^*_brk, Probe B shows non-monotonic behavior, consistent with the observation that the intervention reorganises late layer competition between commitment and abstention rather than erasing a single pathway. 6.2 Tuned Lens Trajectory Analysis Figure 7: Tuned Lens trajectories. The preference margin Δl _l flips sign within the override window under conflict. Intervention suppresses this reversal, restoring correct dominance. We apply the Tuned Lens [4] to project layer’s hidden state into vocabulary space, computing the preference margin δl=logitl(y−)−logitl(y+) _l=logit_l(y^-)-logit_l(y^+) to track when the latent prediction crosses from correct to incorrect. Under conflict free conditions, δl _l remains negative throughout (Fig. 7). Under conflict, δl _l transitions from negative to positive between layers 18 and 25, placing the preference reversal within the wideband identified by causal localization. The reversal accumulates over several layers, consistent with a distributed override circuit. After excising arb∗C_arb^*, this transition is suppressed and δl _l stays negative through the critical window. The convergence of three independent signals (interventional CER, temporal probes, and Tuned Lens trajectories) on the same layer band provides triangulated evidence that the localized heads causally mediate this reversal. Full distributions are in Sec. 12. 7 Conclusion We presented Craft, an interventional tracing framework that leverages causal deconfounding to localize failure-critical attention heads in medical VLMs. The framework reveals that arbitration failure and brake failure occupy structurally disjoint circuitry and respond to opposite interventions: excision of a mid-layer override wideband suppresses textual hijacking, while excision of sparse late-layer gating heads recovers appropriate abstention under degraded visual evidence. Both interventions are training-free, modular, and affect fewer than one percent of total heads, suggesting that sparse activation editing may serve as a practical and lightweight post-hoc safety mechanism for deployed medical VLMs. We hope this mechanistic interpretability perspective on multimodal failure modes opens new and productive directions for principled safety analysis in clinical AI applications. References [1] R. Achtibat, S. M. V. Hatefi, M. Dreyer, A. Jain, T. Wiegand, S. Lapuschkin, and W. Samek (2024) Attnlrp: attention-aware layer-wise relevance propagation for transformers. arXiv preprint arXiv:2402.05602. Cited by: §2. [2] S. Bai, Y. Cai, R. Chen, K. Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, C. Gao, C. Ge, et al. (2025) Qwen3-vl technical report. arXiv preprint arXiv:2511.21631. Cited by: §4.1, §5.1. [3] S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y. Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y. Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, and J. Lin (2025) Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923. Cited by: §1. [4] N. Belrose, I. Ostrovsky, L. McKinney, Z. Furman, L. Smith, D. Halawi, S. Biderman, and J. Steinhardt (2023) Eliciting latent predictions from transformers with the tuned lens. arXiv preprint arXiv:2303.08112. Cited by: §1, §12.2, §6.2, §6. [5] Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang, X. Zhu, L. Lu, et al. (2024) Internvl: scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 24185–24198. Cited by: §1. [6] A. Conmy, A. Mavor-Parker, A. Lynch, S. Heimersheim, and A. Garriga-Alonso (2023) Towards automated circuit discovery for mechanistic interpretability. Advances in Neural Information Processing Systems 36, p. 16318–16352. Cited by: §1, §2. [7] V. Didelez and I. Pigeot (2001) Causality: models, reasoning, and inference. JSTOR. Cited by: §3.3, §9.1, §9.2. [8] N. Elhage, N. Nanda, C. Olsson, T. Henighan, N. Joseph, B. Mann, A. Askell, Y. Bai, A. Chen, T. Conerly, et al. (2021) A mathematical framework for transformer circuits. Transformer Circuits Thread 1 (1), p. 12. Cited by: §1, §3.1. [9] Y. Geifman and R. El-Yaniv (2019) Selectivenet: a deep neural network with an integrated reject option. In International conference on machine learning, p. 2151–2159. Cited by: §2. [10] N. Goldowsky-Dill, C. MacLeod, L. Sato, and A. Arora (2023) Localizing model behavior with path patching. arXiv preprint arXiv:2304.05969. Cited by: §2. [11] M. Golovanevsky, W. Rudman, V. Palit, R. Singh, and C. Eickhoff (2024) What do vlms notice? a mechanistic interpretability pipeline for gaussian-noise-free text-image corruption and evaluation. arXiv preprint arXiv:2406.16320. Cited by: §1. [12] A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. (2024) The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Cited by: §8.2. [13] R. Hendel, M. Geva, and A. Globerson (2023) In-context learning creates task vectors. In Findings of the Association for Computational Linguistics: EMNLP 2023, p. 9318–9333. Cited by: §1. [14] S. Jiang, Y. Wang, S. Song, T. Hu, C. Zhou, B. Pu, Y. Zhang, Z. Yang, Y. Feng, J. T. Zhou, et al. (2025) Hulu-med: a transparent generalist model towards holistic medical vision-language understanding. arXiv preprint arXiv:2510.08668. Cited by: §2, §4.1. [15] Q. Jin, B. Dhingra, Z. Liu, W. Cohen, and X. Lu (2019) Pubmedqa: a dataset for biomedical research question answering. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), p. 2567–2577. Cited by: §8.2. [16] Z. Jin, P. Cao, H. Yuan, Y. Chen, J. Xu, H. Li, X. Jiang, K. Liu, and J. Zhao (2024) Cutting off the head ends the conflict: a mechanism for interpreting and mitigating knowledge conflicts in language models. In Findings of the Association for Computational Linguistics: ACL 2024, p. 1193–1215. Cited by: §1, §2. [17] S. A. Kogilathota, S. V. EG, L. Sun, and J. Zhou (2026) HALP: detecting hallucinations in vision-language models without generating a single token. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), p. 6067–6085. Cited by: Table 6, §10, §2. [18] J. J. Lau, S. Gayen, A. Ben Abacha, and D. Demner-Fushman (2018) A dataset of clinically generated visual questions and answers about radiology images. Scientific data 5 (1), p. 180251. Cited by: §2, §4.1. [19] G. Li, Y. Chen, and H. Tong (2025) Taming knowledge conflicts in language models. arXiv preprint arXiv:2503.10996. Cited by: §1, §2. [20] K. Li, O. Patel, F. Viégas, H. Pfister, and M. Wattenberg (2023) Inference-time intervention: eliciting truthful answers from a language model. Advances in Neural Information Processing Systems 36, p. 41451–41530. Cited by: Table 6, §10, §2. [21] B. Liu, L. Zhan, L. Xu, L. Ma, Y. Yang, and X. Wu (2021) Slake: a semantically-labeled knowledge-enhanced dataset for medical visual question answering. In 2021 IEEE 18th international symposium on biomedical imaging (ISBI), p. 1650–1654. Cited by: §2, §4.1. [22] S. Longpre, K. Perisetla, A. Chen, N. Ramesh, C. DuBois, and S. Singh (2021) Entity-based knowledge conflicts in question answering. In Proceedings of the 2021 conference on empirical methods in natural language processing, p. 7052–7063. Cited by: §1, §1. [23] K. Meng, D. Bau, A. Andonian, and Y. Belinkov (2022) Locating and editing factual associations in gpt. Advances in neural information processing systems 35, p. 17359–17372. Cited by: §1, §2, §4.1. [24] C. Neo, L. Ong, P. Torr, M. Geva, D. Krueger, and F. Barez (2024) Towards interpreting visual information processing in vision-language models. arXiv preprint arXiv:2410.07149. Cited by: §1. [25] D. Nguyen, M. K. Ho, H. Ta, T. T. Nguyen, Q. Chen, K. Rav, Q. D. Dang, S. Ramchandre, S. L. Phung, Z. Liao, et al. (2025) Localizing before answering: a benchmark for grounded medical visual question answering. In Thirty-Fourth International Joint Conference on Artificial Intelligence (IJCAI-25), Cited by: §5.1. [26] C. Olsson, N. Elhage, N. Nanda, N. Joseph, N. DasSarma, T. Henighan, B. Mann, A. Askell, Y. Bai, A. Chen, et al. (2022) In-context learning and induction heads. arXiv preprint arXiv:2209.11895. Cited by: §2. [27] A. Pal, L. K. Umapathi, and M. Sankarasubbu (2023) Med-halt: medical domain hallucination test for large language models. In Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL), p. 314–334. Cited by: §2. [28] M. V. Pham, H. Borkakoty, and Y. Hou (2026) Where knowledge collides: a mechanistic study of intra-memory knowledge conflict in language models. arXiv preprint arXiv:2601.09445. Cited by: §1, §2. [29] J. Postmus and S. Abreu (2024) Steering large language models using conceptors: improving addition-based activation engineering. arXiv preprint arXiv:2410.16314. Cited by: §2. [30] D. Shi, R. Jin, T. Shen, W. Dong, X. Wu, and D. Xiong (2024) Ircan: mitigating knowledge conflicts in llm generation via identifying and reweighting context-aware neurons. Advances in Neural Information Processing Systems 37, p. 4997–5024. Cited by: §1. [31] J. Vig, S. Gehrmann, Y. Belinkov, S. Qian, D. Nevo, Y. Singer, and S. Shieber (2020) Investigating gender bias in language models using causal mediation analysis. Advances in neural information processing systems 33, p. 12388–12401. Cited by: §1, §3.1. [32] K. Wang, A. Variengien, A. Conmy, B. Shlegeris, and J. Steinhardt (2022) Interpretability in the wild: a circuit for indirect object identification in gpt-2 small, 2022. URL https://arxiv. org/abs/2211.00593 2. Cited by: §1. [33] W. Wang, Z. Gao, L. Gu, H. Pu, L. Cui, X. Wei, Z. Liu, L. Jing, S. Ye, J. Shao, et al. (2025) Internvl3. 5: advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265. Cited by: §4.1. [34] B. Wen, J. Yao, S. Feng, C. Xu, Y. Tsvetkov, B. Howe, and L. L. Wang (2025) Know your limits: a survey of abstention in large language models. Transactions of the Association for Computational Linguistics 13, p. 529–556. Cited by: §2. [35] W. Wu, X. Xu, C. Gao, X. Diao, S. Li, L. A. Salas, and J. Gui (2025) Assessing and mitigating medical knowledge drift and conflicts in large language models. arXiv preprint arXiv:2505.07968. Cited by: §1, §8.2. [36] W. Wu, Y. Wang, G. Xiao, H. Peng, and Y. Fu (2024) Retrieval head mechanistically explains long-context factuality. arXiv preprint arXiv:2404.15574. Cited by: §2. [37] P. Xia, Z. Chen, J. Tian, Y. Gong, R. Hou, Y. Xu, Z. Wu, Z. Fan, Y. Zhou, K. Zhu, et al. (2024) Cares: a comprehensive benchmark of trustworthiness in medical vision language models. Advances in Neural Information Processing Systems 37, p. 140334–140365. Cited by: §1. [38] J. Xie, K. Zhang, J. Chen, R. Lou, and Y. Su (2023) Adaptive chameleon or stubborn sloth: revealing the behavior of large language models in knowledge conflicts. In The Twelfth International Conference on Learning Representations, Cited by: §1. [39] A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. (2025) Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: §8.2. [40] C. Yeh, J. Wang, A. D. Graham, A. J. Liu, B. Tan, Y. Chen, Y. Ma, and M. C. Lin (2024) Insight: a multi-modal diagnostic pipeline using llms for ocular surface disease diagnosis. In International Conference on Medical Image Computing and Computer-Assisted Intervention, p. 711–721. Cited by: §1, §1. [41] X. Zhang, Z. Meng, J. Lever, and E. S. Ho (2025) CCD: mitigating hallucinations in radiology mllms via clinical contrastive decoding. arXiv preprint arXiv:2509.23379. Cited by: §1, Table 6, §10, §2. [42] L. Zhao, Y. Deng, W. Zhang, and Q. Gu (2024) Mitigating object hallucination in large vision-language models via image-grounded guidance. arXiv preprint arXiv:2402.08680. Cited by: §1, Table 6, §10, §2. [43] C. Zhu, Y. Lin, S. Chen, Y. Wang, and J. Lin (2026) Medeyes: learning dynamic visual focus for medical progressive diagnosis. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40, p. 13916–13924. Cited by: §2. [44] A. Zou, L. Phan, S. Chen, J. Campbell, P. Guo, R. Ren, A. Pan, X. Yin, M. Mazeika, A. Dombrowski, et al. (2023) Representation engineering: a top-down approach to ai transparency. arXiv preprint arXiv:2310.01405. Cited by: §2. 8 More Experimental Details 8.1 Models, conflict construction, and data splits Figure 8: The Unknown rate increases with mask_scale, but remains below 50% even under the strongest masking. We use mask_scale=2.0 as the default degradation setting. We evaluate four medical VLMs: Hulu-Med 4B (36 layers, 32 heads, d=2560d=2560), Hulu-Med 7B, InternVL3.5 4B (36 layers, 32 heads, d=2560d=2560), and Qwen3-VL-8B, all running in FP16 on two NVIDIA GeForce RTX 5090 GPUs with 32 GB peak memory. For the arbitration setting, we restrict evaluation to binary answer instances and construct textual interventions via an EVIDENCE template: the support condition asserts the correct answer y+y^+ while the conflict condition asserts the alternative y−y^-. We retain only instances where the model predicts y+y^+ on clean input, remains correct under support, and flips to y−y^- under conflict. For the brake setting, we construct degraded inputs by masking diagnostically relevant regions resolved from detection.json (preferring pixel level masks when available, otherwise falling back to detection rectangles) with black fill enlarged by mask_scale= 2.0\,=\,2.0. The prompt explicitly permits answering unknown; filtering keeps only instances answered correctly without masking. A mask scale sweep (Fig. 8) confirms that the Unknown rate increases monotonically yet remains below 50% even at mask_scale= 3.0\,=\,3.0, motivating intervention. After filtering, all retained examples are split 70/30 into a training split (layer tracing, head localization, threshold selection) and a validation split on which all reported metrics are evaluated. 8.2 Metric Formal Definitions and Text-Only Results (i) Evaluation metrics. The main paper reports four metrics for arbitration failure (Table 1) and three for brake failure (Table 2). Let y^ipre,y^ipost y_i^pre, y_i^post denote the model prediction on conflict instance i before and after intervention, and let yi+,yi−y_i^+,y_i^- denote the visually correct and textually misleading answers respectively. For Arbitration setting: CFR(↓) CFR\ ( ) =1|conf|∑i∈conf[y^ipost=yi−], = 1|D_conf| _i _conf1\! [ y_i^post=y_i^- ], (10) Resist(↑) Resist\ ( ) =1−CFR, =1- CFR, (11) C2W(↓) 2W\ ( ) =1|conf|∑i[y^ipre=yi+∧y^ipost=yi−], = 1|D_conf| _i1\! [ y_i^pre=y_i^+\; \; y_i^post=y_i^- ], (12) W2C(↑) 2C\ ( ) =1|conf|∑i[y^ipre=yi−∧y^ipost=yi+]. = 1|D_conf| _i1\! [ y_i^pre=y_i^-\; \; y_i^post=y_i^+ ]. (13) Here Resist denotes “not following the misleading answer” and can include correct, abstention, or other non misleading outputs. Brake setting. Let yunky^unk denote abstention and ycom∈y+,y−y^com∈\y^+,y^-\ a concrete answer; instances are drawn from the visually degraded subset degD_deg. UR(↑) UR\ ( ) =1|deg|∑i[y^ipost=yunk], = 1|D_deg| _i1\! [ y_i^post=y^unk ], (14) C2U(↑) 2U\ ( ) =1|deg|∑i[y^ipre=ycom∧y^ipost=yunk], = 1|D_deg| _i1\! [ y_i^pre=y^com\; \; y_i^post=y^unk ], (15) U2O(↓) 2O\ ( ) =1|deg|∑i[y^ipre=yunk∧y^ipost=ycom]. = 1|D_deg| _i1\! [ y_i^pre=y^unk\; \; y_i^post=y^com ]. (16) (i) Text-only benchmark results. Table 5 reports results on two text-only medical QA benchmarks, ConflictMedQA [35] and PubMedQA [15], using Qwen3-4B [39] and Llama3.2-3B [12], where conflict is injected purely through contradictory textual context without visual inputs. Analysis. Across both benchmarks, Craft consistently lowers CFR and improves Resist over both the baseline and random head ablation. The effect is especially strong on ConflictMedQA, where Craft substantially reduces conflict compliance across both models and all injection positions while keeping C2WC2W near zero. On PubMedQA, Craft remains effective under longer and more naturalistic medical contexts, with the strongest case reducing CFR from 53.70 to 3.70 and increasing Resist from 46.30 to 96.30. Compared with random ablation using the same number of heads, Craft achieves a better balance between W2CW2C gains and C2WC2W collateral damage, confirming that the selected heads are involved in conflict following rather than general answer generation. Table 4: Threshold sensitivity (Hulu-Med 4B). Left: arbitration on VQA-RAD (Before Q). Right: brake on SLAKE (image conflict). Highlighted: final settings. _s _d CFR↓ C2W↓ W2C↑ #Heads 0.133 0.550 18.7 4.9 30.8 25 0.271 0.183 24.6 0.5 20.5 2 0.156 0.445 22.1 7.9 30.5 31 0.350 0.500 18.2 0.8 27.4 6 _s _d UR↑ O2U↑ U2O↓ #Heads 0.2 0.5 57.58 20.45 0.76 4 0.0 0.3 54.55 16.67 0.00 3 0.0 0.8 62.88 26.52 1.52 9 0.0 0.5 65.15 27.27 0.00 6 Table 5: Text only benchmark results. CFR: conflict compliance (↓ ); Resist: resist rate (↑ ); C2W: correct to wrong (↓ ); W2C: wrong to correct (↑ ). Bold: best per block; shaded rows: Craft. Random uses the same head count. ConflictMedQA PubMedQA Position Model Method CFR↓ Resist↑ C2W↓ W2C↑ CFR↓ Resist↑ C2W↓ W2C↑ Prefix Qwen3-4B Baseline 83.33 16.67 – – 49.67 50.33 – – Random 94.25 5.75 12.07 1.15 57.17 42.83 12.17 4.67 Craft 37.36 62.64 0.00 45.98 23.59 76.41 0.00 26.09 Llama3.2-3B Baseline 57.69 42.31 – – 58.46 41.54 – – Random 42.31 57.69 26.92 42.31 63.08 36.92 8.85 4.23 Craft 27.88 72.12 0.00 29.81 46.15 53.85 1.15 13.46 Before Q Qwen3-4B Baseline 94.83 5.17 – – 53.70 46.30 – – Random 89.66 10.34 1.15 6.32 46.30 53.70 3.91 11.30 Craft 68.39 31.61 1.15 27.59 3.70 96.30 0.00 50.00 Llama3.2-3B Baseline 76.92 23.08 – – 71.54 28.46 – – Random 81.73 18.27 11.54 6.73 68.08 31.92 4.62 8.08 Craft 57.69 42.31 0.00 19.23 59.23 40.77 1.54 13.85 Before A Qwen3-4B Baseline 80.46 19.54 – – 65.98 34.02 – – Random 76.44 23.56 4.02 8.05 85.33 14.67 20.54 1.20 Craft 23.56 76.44 0.57 57.47 51.41 48.59 0.00 14.57 Llama3.2-3B Baseline 43.27 56.73 – – 66.92 33.08 – – Random 48.08 51.92 16.35 11.54 72.31 27.69 9.23 3.85 Craft 31.73 68.27 2.88 14.42 54.62 45.38 2.31 14.62 8.3 Key Parameter Selection and Sensitivity The threshold procedure selects ∗C^* by requiring heads to exceed τs _s on the suppression score (CERexcCER_exc or HRHR) while falling below τd _d on the disruption score (BCPexcBCP_exc), with τs _s set near the 90th percentile of the suppression distribution and τd _d near the 10th percentile of the disruption distribution, yielding |arb∗|∈[4,13]|C_arb^*|∈[4,13] and |brk∗|∈[5,7]|C_brk^*|∈[5,7] depending on architecture. All metrics are computed on the held out 30% split, averaged over three random seeds (standard deviations << 1.5 p), with significance assessed via two sided permutation tests (n=10,000n=10,000, p<0.01p<0.01). A threshold sensitivity sweep (Table 4) confirms the trade off: looser settings improve correction (W2CW2C/O2UO2U) at the cost of increased collateral damage (C2WC2W/U2OU2O), while overly strict settings leave failure unresolved; the operating points balance target effect retention against competence preservation. 9 Theoretical Analysis of Deconfounding This appendix formalizes the causal deconfounding argument underlying the head selection procedure in Sec. 4. We first show that observational activation attribution is confounded (proposition 9.1), then prove that clean state interchange eliminates this bias (theorem 9.2), and finally establish that the CERCER metric inherits the deconfounding guarantee (corollary 9.3). 9.1 Confounding in Observational Attribution Proposition 9.1 (Confounded Observational Attribution). Under the SCM defined in Sec. 3.2 with latent confounder UcU_c and structural equations l=fl(l−1,Xv,Xt,Θ,Uc)h_l=f_l(h_l-1,X_v,X_t, ,U_c), Y=g(L,tans)Y=g(h_L,t_ans), the observational conditional [Y∣l,i]E[Y _l,i] does not equal the interventional quantity [Y∣do(l,i=a)]E[Y (a_l,i=a)] whenever UcU_c is not d-separated from Y given l,ia_l,i in the mutilated graph. Proof. (i) Active back door path. In the original DAG the path l,i←Uc→Ya_l,i← U_c→ Y is active because UcU_c is a common cause that is never conditioned on. Since UcU_c encompasses lexical overlap, visual ambiguity, and cross modal coupling, it is neither directly measurable nor finite dimensional, so the back door criterion [7] cannot be satisfied by observational adjustment. (i) Bias decomposition. Applying the truncated factorisation formula yields: [Y∣l,i=a]=[Y∣do(l,i=a)]⏟causal effect+Bias(Uc→l,i,Uc→Y)⏟confounding bias,E[Y _l,i=a]\;=\; E[Y (a_l,i=a)]_causal effect\;+\; Bias(U_c→a_l,i,\;U_c→Y)_confounding bias, (17) where the bias term is generically nonzero whenever UcU_c simultaneously affects both the head activation and the model output through distinct structural equations. (i) Implication. Activation magnitude ranking is therefore a biased estimator of causal importance: a head may appear highly influential simply because UcU_c drives both its activation and the erroneous output, without the head itself mediating the failure. ∎ 9.2 D Separation under Clean State Interchange Theorem 9.2 (Deconfounding via Interchange Intervention). Let l,icleana_l,i^clean denote the activation of head (l,i)(l,i) computed from the conflict free counterfactual sharing (Xv,q)(X_v,q) with the original conflict sample. Under do(l,i:=l,iclean)do(a_l,i:=a_l,i^clean), the back door path l,i←Uc→Ya_l,i← U_c→ Y is blocked, and the measured effect CDE(l,i)=[Y∣do(l,i:=l,iclean)]−[Y∣do(l,i:=l,iconflict)]CDE(l,i)\;=\;E [Y (a_l,i:=a_l,i^clean) ]\;-\;E [Y (a_l,i:=a_l,i^conflict) ] (18) equals the controlled direct effect of the conflict mechanism mediated through head (l,i)(l,i). Proof. The dodo operator do(l,i:=l,iclean)do(a_l,i:=a_l,i^clean) replaces the structural equation for l,ia_l,i with a constant assignment, mutilating the graph by severing all incoming edges to l,ia_l,i [7]. (i) Back door severed. Since l,ia_l,i is fixed to a constant that no longer depends on the UcU_c realisation, the edge Uc→l,iU_c _l,i is removed from the mutilated graph, blocking the only back door path l,i←Uc→Ya_l,i← U_c→ Y. (i) Front door preserved. All downstream structural equations remain intact. The causal path Xv→l−1→l→YX_v _l-1 _l→ Y continues to operate through other heads and FFN components, so the intervention does not destroy the forward computation. (i) Input level matching. Both the conflict and conflict free samples share (Xv,q)(X_v,q), so the only variation in UcU_c arises from the presence or absence of adversarial text XtconfX_t^conf. Fixing l,ia_l,i to its clean state value ensures constancy across both UcU_c conditions, achieving d separation: l,i⟂Uc|do(l,i)in the mutilated graph.a_l,i \!\!\! U_c\; \;do(a_l,i) the mutilated graph. (19) Combining (i) through (i), CDE(l,i)CDE(l,i) isolates the causal contribution of head (l,i)(l,i) to the conflict induced failure, free of confounding bias from UcU_c. ∎ Corollary 9.3 (Validity of CERCER as Causal Metric). The conflict effect reduction metric CERexc(h)=δconflict−δexcise(h)CER_exc(h)=δ^conflict-δ^excise(h) is a valid estimate of the controlled direct effect of head h on the preference margin, provided the paired counterfactual shares (Xv,q)(X_v,q) with the conflict sample. Proof. The excise operation (αh→0+ _h→ 0^+) is a special case of the interchange intervention in theorem 9.2 that replaces the head output with a near zero vector. The d separation guarantee holds for any fixed replacement value, since the key property is that l,ia_l,i becomes constant with respect to UcU_c, not that it takes a particular value. The CERCER metric then directly measures the shift in preference margin attributable to removing the causal contribution of head h under the deconfounded regime. ∎ 10 Comparison with Existing Safety Methods We include a controlled comparison in which all methods are evaluated under the same answer extraction rule and clean performance measurement protocol. Table 6 summarises four representative baselines spanning three intervention paradigms: decoding guidance (CCD [41], MARINE [42]), trait level activation steering (ITI [20]), and pre generation hallucination detection (HALP [17]). CCD and MARINE modify the output distribution using auxiliary contrastive or visual grounding signals from an external model, while ITI shifts activations along a global truthfulness direction learned from a small labelled set. HALP trains lightweight probes on internal representations to predict hallucination risk before decoding. In contrast, Craft identifies and excises failure specific causal head sets without external models or additional training. Table 6: Inference time safety baselines. Highlighted: the proposed method. Method Paradigm Extra Model Training Target Failure CCD [41] Decoding guidance ✓ ✗ Textual conflict MARINE [42] Decoding guidance ✓ ✗ Textual conflict ITI [20] Activation steering ✗ ✓ General truthfulness HALP [17] Hallucination detection ✗ ✓ Pre generation risk Craft Causal head repair ✗ ✗ Arbitration + brake Table 7: Arbitration failure (Hulu-Med 4B, VQA-RAD, Before Q). Bold: best. Method Extra Train CFR↓ Resist↑ C2W↓ W2C↑ Baseline ✗ ✗ 44.62 55.38 0.00 0.00 Random heads ✗ ✗ 40.77 59.23 1.03 4.87 CCD ✓ ✗ 31.28 68.72 1.28 13.34 MARINE ✓ ✗ 28.46 71.54 1.03 16.92 ITI ✗ ✓ 34.10 65.90 1.79 12.31 Craft ✗ ✗ 14.36 85.64 0.51 30.77 Table 8: Brake failure (Hulu-Med 4B, SLAKE image conflict). Bold: best. Method Extra Train UR↑ O2U↑ U2O↓ Baseline ✗ ✗ 37.88 0.00 0.00 Random heads ✗ ✗ 45.45 8.33 0.76 HALP routed abstention ✗ ✓ 56.82 21.21 2.27 Craft ✗ ✗ 65.15 27.27 0.00 Analysis. For arbitration failure (Table 7), random head removal provides only modest improvement, confirming that the gain is not caused by nonspecific perturbation. CCD and MARINE further reduce CFR by injecting auxiliary contrastive or grounding signals at decoding time, and ITI improves over the baseline via a global truthfulness shift. However, Craft achieves the lowest CFR and highest Resist while keeping C2W minimal, indicating that targeted excision of the localized override circuit outperforms both output level calibration and generic activation steering. For brake failure (Table 8), HALP routed abstention improves UR by detecting high risk cases before generation, but introduces nonzero U2O because the routing decision can abstain on cases already correctly handled. Craft yields a larger UR gain with zero U2O, showing that direct intervention on the brake head set recovers abstention while better preserving previously abstaining predictions. Reproducibility. All methods share the same validation split, answer extraction rule, and clean performance measurement protocol, ensuring that metric differences reflect genuine intervention effects rather than evaluation interface discrepancies. 10.1 Clean Accuracy Preservation under Baseline Methods To complement the failure mitigation results, we compare the clean accuracy preservation of Craft with several baseline interventions. For arbitration failure, we report the Before Q setting on VQA-RAD, following the main baseline comparison. For brake failure, we report the SLAKE image conflict setting. All values are measured on clean inputs before and after intervention. For baseline methods that do not explicitly select causal heads, #Heads denotes the matched effective intervention budget used for comparison. Table 9: Clean accuracy preservation under baseline methods. Δ denotes post minus pre intervention accuracy (%). Bold: best preservation result. Hulu-Med 4B InternVL3.5 4B Qwen3-VL-8B Benchmark Setting Method #Heads Original After Acc #Heads Original After Acc #Heads Original After Acc VQA-RAD Arbitration failure Random heads 6 100.00 94.87 −-5.13 10 100.00 93.71 −-6.29 8 100.00 95.80 −-4.20 CCD 7 100.00 91.79 −-8.21 11 100.00 91.43 −-8.57 9 100.00 93.60 −-6.40 MARINE 7 100.00 92.31 −-7.69 11 100.00 92.00 −-8.00 9 100.00 94.20 −-5.80 ITI 8 100.00 91.03 −-8.97 12 100.00 90.86 −-9.14 10 100.00 93.10 −-6.90 Craft 6 100.00 96.41 −-3.59 10 100.00 96.57 −-3.43 8 100.00 97.80 −-2.20 SLAKE Brake failure Random heads 6 100.00 88.40 −-11.60 7 100.00 86.23 −-13.77 8 100.00 91.60 −-8.40 CCD 7 100.00 83.73 −-16.27 8 100.00 80.65 −-19.35 9 100.00 88.40 −-11.60 MARINE 7 100.00 85.07 −-14.93 8 100.00 82.58 −-17.42 9 100.00 89.30 −-10.70 ITI 8 100.00 86.22 −-13.78 9 100.00 84.19 −-15.81 10 100.00 90.10 −-9.90 HALP 8 100.00 84.61 −-15.39 9 100.00 81.94 −-18.06 10 100.00 87.90 −-12.10 Craft 6 100.00 90.67 −-9.33 7 100.00 88.87 −-11.13 8 100.00 93.20 −-6.80 Analysis. Across both failure modes, Craft preserves clean accuracy better than the compared baselines under comparable intervention budgets. For arbitration failure, Craft produces the smallest clean accuracy drop while using the sparsest head set, indicating that the gain in conflict resistance is not obtained by broadly degrading clean visual question answering. For brake failure, HALP and the generic intervention baselines introduce larger clean accuracy losses under matched or near matched budgets, whereas Craft maintains the highest post intervention clean accuracy on all three backbones. This supports the claim that localized head excision provides a more targeted safety intervention than broader representation steering or hallucination suppression baselines. 10.2 Comparison with Fine Tuning Based Safety Adaptation We compare Craft with two training based baselines under the same Hulu-Med 4B settings (VQA-RAD Before Q for textual arbitration, SLAKE image conflict pool for visual brake failure). Conflict SFT fine tunes on conflict aware examples whose target is the clean gold answer (textual conflict) or unknown (degraded visual evidence). Preference Tuning applies RLHF optimization where the preferred response resists the conflict or abstains and the rejected response follows it. Both baselines use parameter efficient adaptation on the language model blocks. We also combine Conflict SFT with Craft to test whether the training free intervention is complementary to fine tuning. Table 10: Comparison with fine tuning based safety adaptation on textual conflict. Hulu-Med 4B, VQA-RAD Before Q. Shaded rows denote Craft. Method Training CFR↓CFR Resist↑ C2W↓ W2C↑ Clean Acc.↑ Baseline ✗ 44.62 55.38 0.00 0.00 100.00 Random heads ✗ 40.77 59.23 1.03 4.87 96.67 Conflict SFT ✓ 22.31 77.69 2.05 24.36 97.18 Preference Tuning ✓ 19.49 80.51 1.54 26.67 97.44 Craft ✗ 14.36 85.64 0.51 30.77 96.41 Conflict SFT + Craft ✓ 10.77 89.23 1.03 35.90 96.92 Table 11: Comparison with fine tuning based safety adaptation on visual brake failure. Hulu-Med 4B, SLAKE image conflict pool. Shaded rows denote Craft. Method Training UR↑UR O2U↑ U2O↓ Clean Acc.↑ Net Gain↑ Baseline ✗ 37.88 0.00 0.00 100.00 0.00 Random heads ✗ 45.45 8.33 0.76 96.90 7.57 Conflict SFT ✓ 54.55 19.70 2.27 96.97 16.67 Preference Tuning ✓ 57.58 21.97 1.52 96.21 19.70 Craft ✗ 65.15 27.27 0.00 90.67 27.27 Conflict SFT + Craft ✓ 69.70 32.58 0.76 92.42 31.82 Analysis. Table 10 shows that both Conflict SFT and Preference Tuning reduce CFRCFR and raise Resist over the baseline, confirming that training on conflict aware supervision can teach the model to resist textual overrides. Nevertheless, Craft achieves a lower CFRCFR than either training based method without any parameter update, suggesting that the failure is concentrated in a sparse set of causal heads whose direct suppression yields a strong correction. The combined Conflict SFT plus Craft row further lowers CFRCFR to 10.77 and raises Resist to 89.23, indicating that the two approaches address complementary aspects of the problem. Table 11 reveals a parallel pattern for visual brake failure. Fine tuning increases URUR under degraded visual evidence but introduces nonzero U2OU2O, meaning some originally abstaining predictions are incorrectly shifted into concrete answers. Craft obtains a larger URUR gain and keeps U2OU2O at zero, though it incurs a greater clean accuracy cost because the brake intervention directly suppresses answer commitment under insufficient evidence. Combining fine tuning with Craft achieves the largest net gain (31.82), showing that fine tuning provides broad behavioral calibration and better clean accuracy preservation, while Craft supplies a sparse, training free mechanistic intervention for failure specific causal repair at inference time. Table 12: Open ended textual conflict results. The judge classifies each response into gold claim, conflict claim, or qualified response. Shaded rows denote Craft. Hulu-Med 4B InternVL3.5 4B Benchmark Method Gold↑ Conflict↓ Qualified↑ Clean Fact.↑ Gold↑ Conflict↓ Qualified↑ Clean Fact.↑ VQA RAD Baseline 54.20 39.60 6.20 96.00 56.40 36.80 6.80 96.30 Random 57.10 35.70 7.20 95.10 58.20 34.70 7.10 95.40 Craft 70.90 21.80 7.30 94.70 78.40 13.20 8.40 95.00 SLAKE Baseline 51.80 42.20 6.00 95.30 59.10 34.60 6.30 96.80 Random 54.40 38.60 7.00 94.60 60.30 32.20 7.50 96.10 Craft 68.60 23.10 8.30 93.80 82.60 9.10 8.30 95.90 Table 13: Open ended degraded vision results. The judge evaluates whether the response expresses calibrated uncertainty or makes an unsupported diagnosis. Shaded rows denote Craft. Hulu-Med 4B InternVL3.5 4B Benchmark Method Uncert.↑ Unsup.↓ Evid. Req.↑ Clean Compl.↑ Uncert.↑ Unsup.↓ Evid. Req.↑ Clean Compl.↑ SLAKE Baseline 36.20 59.10 4.70 95.60 24.80 69.40 5.80 94.80 Random 44.00 50.70 5.30 94.70 18.60 75.20 6.20 93.70 Craft 63.40 29.90 6.70 90.80 34.80 58.60 6.60 92.80 HealMed-VQA Baseline 73.60 21.80 4.60 96.20 76.40 18.70 4.90 95.60 Random 79.10 15.80 5.10 95.10 78.80 16.00 5.20 94.70 Craft 89.40 5.90 4.70 91.70 88.20 7.30 4.50 92.40 Table 14: Online computational cost of head intervention on VQA-RAD. Hooked masking is the analysis implementation; true head skipping offers marginal FLOP savings. Model Implementation #Heads Normal Lat. Interv. Lat. Δ . Normal FLOPs Δ Hulu-Med 4B Hooked masking 6 206.39 ms 210.31 ms +1.90% 19539.96G 0.00% Hulu-Med 4B True head skipping 6 207.11 ms 208.10 ms +0.48% 19539.96G 0.11% lower InternVL3.5 4B Hooked masking 10 43.27 ms 49.43 ms +14.24% 3629.54G 0.00% InternVL3.5 4B True head skipping 10 43.91 ms 47.03 ms +7.11% 3629.54G 0.10% lower Table 15: Offline cost of head selection. The localization cost is paid once before deployment. Prompt forward counts include diagnostic passes for layer tracing, head scoring, and selection. Model Setting Loc. Samples Total Forwards Forwards / Sample Param. Update Hulu-Med 4B VQA-RAD Before Q 391 111,342 284.8 ✗ InternVL3.5 4B VQA-RAD Before Q 698 158,446 227.0 ✗ Qwen3-VL-8B VQA-RAD Before Q 698 241,508 346.0 ✗ 11 Computational Cost Table 14 reports the online inference overhead after head selection. Hooked masking adds less than 2% latency on Hulu-Med 4B and about 14% on InternVL3.5 4B; true head skipping reduces this further but the sparse intervention level is too small for meaningful FLOP savings without fused kernels. These results clarify that Craft is a safety oriented post hoc intervention rather than an inference acceleration method. Table 15 reports the one time offline localization cost. The full head selection pipeline requires 227 to 346 forward passes per sample across the evaluated models, substantially larger than a single inference pass but involving no parameter updates or gradient computation. After localization the selected head set is reused for all subsequent inference with only the low overhead hook shown above. Therefore, Craft trades a moderate one time diagnostic cost for low overhead inference time repair. 12 Probe and Tuned Lens Implementation 12.1 Temporal Probe Architecture Both Probe A (conflict detection) and Probe B (follow prediction) are implemented as lightweight linear classifiers: p^l=σ(l⊤l+bl), p_l=σ(w_l h_l+b_l), (20) where l∈ℝdh_l ^d is the residual stream representation at layer l, l∈ℝdw_l ^d and bl∈ℝb_l are learned parameters, and σ is the sigmoid function. Training details. Probe A is trained as a binary classifier for conflict detection, distinguishing inputs with injected textual conflict from clean inputs. Probe B is trained as a binary classifier for answer arbitration, predicting whether the model follows the textually misleading answer y−y^- or the visually grounded answer y+y^+ on conflict instances with unambiguous predictions. Both probes are trained only on the training split and evaluated on held-out examples. Both probes are implemented as independent linear classifiers trained with the Adam optimiser (learning rate 10−310^-3, weight decay 10−410^-4) for up to 50 epochs, with early stopping based on validation AUROC (patience = 10). We train a separate probe for each layer without any cross-layer parameter sharing or feature aggregation across layers. 12.2 Tuned Lens Configuration The tuned lens [4] transforms intermediate representations to the output vocabulary space: TLl(l)=U⋅fl(l)+U,TL_l(h_l)=W_U· f_l(h_l)+b_U, (21) where flf_l is a learned affine transformation (one per layer) and UW_U, Ub_U are the unembedding matrix and bias used for projection. Training details. Each layer-specific transformation flf_l is parameterised as an affine mapping fl(x)=Alx+blf_l(x)=A_lx+b_l, where Al∈ℝd×dA_l ^d× d. The tuned lens is trained on a held-out medical text corpus by minimising the KL divergence between the transformed logits TLl(l)TL_l(h_l) and the model’s final output distribution. Optimisation is performed with Adam using a fixed learning rate schedule. At evaluation time, we track preference dynamics across layers by computing the logit difference TLl(l)[y+]−TLl(l)[y−]TL_l(h_l)[y^+]-TL_l(h_l)[y^-]. 12.3 Head-Level Tuned-Lens Preference Distributions To complement the layer-level Tuned-Lens trajectories, we analyse head-level preference-margin distributions under text-conflict and image-conflict settings. For text-conflict examples, the horizontal axis is the gold-versus-wrong preference margin: mh=TL(h)(y+)−TL(h)(y−),m_h=TL^(h)(y^+)-TL^(h)(y^-), (22) where larger values indicate stronger preference for the gold answer. For image-conflict examples, where the desired behavior under masked visual evidence is abstention, the horizontal axis is the abstention margin: ah=TL(h)(yunk)−maxy∈conTL(h)(y),a_h=TL^(h)(y^unk)- _y _conTL^(h)(y), (23) where larger values indicate stronger preference for unknown over the best competing concrete answer. In Figure 10, NC corresponds to the unmasked image and is expected to favour a concrete answer; after masking the diagnostically relevant region, IC should shift rightward if the model correctly abstains under insufficient visual evidence. However, the IC distribution remains close to NC for many heads, indicating that visual degradation does not reliably increase the abstention margin, a pattern reflecting persistent concrete answer commitment consistent with brake failure. In Figure 9, NC is expected to lie to the right of IC because clean inputs favour the gold answer while conflict inputs push the model toward the misleading alternative; the observed leftward shift from NC to IC captures textual override at the head level preference distribution. Together, these two patterns delineate the two failure modes: textual conflict produces a clear gold to wrong preference shift, whereas image degradation often fails to elicit the expected concrete to unknown shift, underscoring an asymmetry in how modality specific evidence is arbitrated at the attention head level. Figure 9: Head-level tuned-lens preference distributions under textual conflict. Each panel shows one selected arbitration-sensitive head. The horizontal axis is the gold-versus-wrong preference margin, logp(y+)−logp(y−) p(y^+)- p(y^-), under the no-conflict condition (NC, green) and the injected-conflict condition (IC, red). Larger values indicate stronger preference for the gold answer. Across many heads, IC shifts left relative to NC, showing that textual conflict pushes head-level preferences from the correct answer toward the misleading answer. Figure 10: Head-level tuned-lens abstention distributions under visual degradation. Each panel shows one selected image-conflict-sensitive head. The horizontal axis is the abstention margin, logp(unknown)−logp(ybest) p( unknown)- p(y_best), where ybesty_best is the best competing concrete answer. NC denotes the original image, while IC denotes the masked image with relevant evidence removed. If the model properly abstains after masking, IC should shift right. Instead, NC and IC largely overlap for many heads, indicating persistent concrete-answer commitment under degraded visual evidence. 13 Patching Strategy Analysis Clean-state definition. For each conflict instance (xv,xtconflict,q)(x_v,x_t^conflict,q), we define the clean state as the hidden representation obtained by running the same visual input with a non-conflicting textual context: lclean=fl(xv,xtclean,q)h_l^clean=f_l(x_v,x_t^clean,q). The patching intervention replaces the conflict-state hidden representation at the target head with this clean-state counterpart. Local patch window. Because the clean and conflict prompts may differ in length, full-sequence replacement can introduce token-alignment artifacts. We therefore apply clean-state replacement only within a local window around the answer position. Let KpatchK_patch denote the number of answer-neighbouring tokens to patch. For each conflict run, we align clean and conflict states by their relative offset from the answer token and replace only the hidden states inside this local window. This preserves most of the conflict prompt while testing whether the answer-neighbouring residual states carry the textual override signal. Patch window metrics. For each patch window size KpatchK_patch, we obtain a layer-wise score curve SK(l)S_K(l) and evaluate its stability using two complementary statistics. Robustness quantifies deviation from the consensus trace S¯(l)=medianKS~K(l) S(l)=median_K\, S_K(l): Rob(K)=1−1L∑l=1L|S~K(l)−S¯(l)|,S~K≜MinMaxNorm(SK).Rob(K)=1- 1L _l=1^L | S_K(l)- S(l) |, S_K (S_K). (24) Smoothness quantifies adjacent-layer variation: Smo(K)=1L−1∑l=1L−1|SK(l+1)−SK(l)|.Smo(K)= 1L-1 _l=1^L-1 |S_K(l+1)-S_K(l) |. (25) Rob↑Rob\! indicates stronger agreement with the local-window consensus; Smo↓Smo\! indicates less layer-to-layer fluctuation. However, Smoothness alone is insufficient for selecting KpatchK_patch: large windows can overwrite non-conflict token states, producing artificially flat curves while weakening causal locality. We therefore use Robustness as the primary criterion and report Smoothness as a diagnostic. Table 16: Patch window sensitivity. Robustness (R)↑ measures agreement with the local window consensus; Smoothness (S)↓ measures adjacent layer variation. Best R and best S are bolded separately for each model. Model (R / S) KpatchK_patch Qwen3-4B Llama3.2-3B Hulu-Med-4B Hulu-Med-7B InternVL3.5-4B Qwen3-VL-8B 2 0.9718 / 0.8577 0.9214 / 0.1475 0.9288 / 0.0671 0.9346 / 0.0648 0.9478 / 0.1633 0.9415 / 0.1586 4 0.9792 / 0.8553 0.9237 / 0.1483 0.9506 / 0.0687 0.9569 / 0.0665 0.9868 / 0.1775 0.9821 / 0.1694 8 0.9991 / 0.8449 0.9987 / 0.1501 0.9991 / 0.0714 0.9988 / 0.0698 0.9948 / 0.1781 0.9962 / 0.1726 16 0.9160 / 0.8983 0.9561 / 0.1459 0.9736 / 0.0741 0.9758 / 0.0726 0.9733 / 0.1818 0.9714 / 0.1769 32 0.8206 / 0.2564 0.9004 / 0.0442 0.8255 / 0.0153 0.8389 / 0.0147 0.8595 / 0.0535 0.8668 / 0.0512 Full 0.6966 / 0.2396 0.7957 / 0.0645 0.7465 / 0.0173 0.7584 / 0.0161 0.8560 / 0.1046 0.8497 / 0.0973 Analysis. Table 16 shows that Kpatch=8K_patch=8 achieves the highest Robustness across all evaluated models, indicating that a compact answer-centred window suffices to recover a stable layer-wise conflict trace. Larger windows and full-sequence patching reduce Robustness, suggesting they introduce token-alignment noise and dilute the local causal signal. Although these settings sometimes yield lower Smoothness, the smoother curves mainly reflect over-patching rather than improved localization. We therefore use Kpatch=8K_patch=8 in all main experiments and exclude full-sequence patching from the consensus curve. 14 Downstream Propagation after Arbitration Head Ablation Hulu-Med 4B InternVL3.5 Figure 11: Downstream attention after arbitration head ablation. Heatmaps compare the same conflict input before and after excising the selected arbitration heads. Blue and red regions denote image tokens and injected conflict text, respectively. Ablation weakens downstream attention to the conflict text region, consistent with reduced textual override propagation. We further examine whether arbitration head ablation changes how conflict information is propagated in later layers. To isolate the effect of the intervention, the input is kept fixed and attention patterns are compared before and after excising the selected arbitration heads. Setup. For each downstream attention head h, the attention mass from the answer position to image tokens V and injected conflict text tokens C is computed as αv(h)=1||∑j∈Atans,j(h),αc(h)=1||∑j∈Atans,j(h), _v^(h)= 1|V| _j A_t_ans,j^(h), _c^(h)= 1|C| _j A_t_ans,j^(h), (26) where A(h)A^(h) is the attention matrix of head h and tanst_ans denotes the answer position. The quantities are measured on the same conflict input before and after arbitration head ablation. Findings. As shown in Fig. 11, downstream layers attend visibly to the injected conflict text before intervention. After excising the selected arbitration heads, attention to the conflict text region decreases and becomes more diffuse over the remaining context. This pattern supports the interpretation that arbitration head ablation attenuates downstream propagation of textual override signals, providing qualitative evidence for the reduction in conflict following in the main results. 15 Layer-wise Attenuation Sensitivity for Brake localization We evaluate the sensitivity of brake localization to the layer-wise attenuation coefficient β used during image-conflict tracing. This coefficient corresponds to the scaling factor in attenuate(ℒ,β)attenuate(L;β) in eq. 6. In implementation, it is denoted as layer_scale. Smaller values impose stronger attenuation, whereas β=1.0β=1.0 recovers the original forward pass and therefore provides an unperturbed reference. Table 17: Layer wise attenuation sensitivity. ΔUR denotes the post intervention increase in unknown rate on the held out image conflict split. HRHR is the layer level hallucination relief score in eq. 9. The highlighted row denotes the setting used for brake tracing. β # Selected Heads ΔUR HRHR 0.0 6 7.23 0.7477 0.2 7 9.68 0.7444 0.4 5 3.35 0.5581 0.6 6 3.23 0.4156 0.8 5 3.24 0.2455 1.0 4 3.23 -0.0001 Analysis. Table 17 shows that brake localization requires a non trivial attenuation signal. For β<1β<1, attenuating candidate layers yields positive HRHR, indicating that these layers contribute to concrete answer commitment under degraded visual evidence. As β approaches 1.0, the perturbation vanishes and HRHR decreases toward zero, as expected from an unchanged forward pass. The tracing score alone does not determine held out intervention performance. Although β=0.0β=0.0 gives the largest HRHR, it produces a smaller ΔUR than β=0.2β=0.2 after head selection. The setting β=0.2β=0.2 achieves the largest unknown rate gain while retaining a strong hallucination relief signal. Larger values from β=0.4β=0.4 to β=1.0β=1.0 weaken the localization contrast and yield smaller abstention gains. Accordingly, β=0.2β=0.2 is used for layer wise brake tracing in the main experiments. 16 Robustness to Local Visual Noise In addition to hard region masking, a local visual noise variant is evaluated on the Hulu-Med 4B SLAKE image conflict pool. Local noise preserves the annotated diagnostic region while degrading its signal quality, approximating acquisition noise, motion artefacts, or low-dose imaging conditions. Stochastic noise is applied only to the annotated diagnostic region, and the head set selected under the main image conflict setting is reused without re-localizing heads for each noise level. Table 18: Robustness to local visual noise. Brake head intervention remains effective under local corruption at varying noise strengths. Noise std #Heads URUR Pre URUR Post ΔUR 24 6 1.82 35.45 33.64 36 6 3.18 36.14 32.95 48 6 7.05 40.45 33.41 64 6 11.59 43.64 32.05 Analysis. Table 18 shows that the brake head intervention remains effective when diagnostic evidence is locally corrupted instead of removed by masking. As noise strength increases, the pre intervention URUR rises from 1.82 to 11.59, indicating that stronger local corruption makes the visual evidence less reliable. After excising the same six brake heads, URUR increases substantially at all noise levels, with ΔUR remaining stable around 32 to 34 points. Clean image accuracy is unchanged across the sweep, suggesting that the effect is concentrated on locally corrupted inputs. These results indicate that the selected brake heads are not specific to mask based evidence removal; they also modulate abstention when diagnostic regions remain visible but unreliable, supporting the interpretation that the intervention strengthens abstention under degraded visual evidence. 17 More Experiment Results 17.1 Head-Set Disjointness between Arbitration and Brake localization We examine whether the two failure modes are associated with the same intervention sensitive heads. For each model, we compare the selected arbitration heads arb∗C_arb^* and brake heads brk∗C_brk^* on SLAKE, and report their set overlap in Table 19. Across both Hulu-Med 4B and InternVL3.5 4B, the two selected head sets have zero intersection. This provides additional evidence for the decoupling analysis in Sec. 5.1: arbitration failure and brake failure are localized to distinct selected head populations under our intervention based criterion. Table 19: Head set overlap between arbitration and brake localization on SLAKE. Both Jaccard and overlap coefficient are zero, confirming disjoint head sets. Model Brake Heads Arbitration Heads Intersection Union Jaccard Overlap Coef. Hulu-Med 4B 6 10 0 16 0.000 0.000 InternVL3.5 4B 7 8 0 15 0.000 0.000 17.2 Arbitration Head Distribution in InternVL3.5 and Qwen3-VL Figure 12: Arbitration head distribution in InternVL3.5 and Qwen3-VL. Blue circles denote selected arbitration heads, green circles denote selected backbone heads, and pale dots denote unselected heads. Circle size is proportional to the corresponding score. Both models exhibit a broad arbitration band in mid to late layers, while backbone heads are more diffuse with a late-layer tail. As shown in Fig. 12, the same dual threshold selection procedure yields a similar spatial pattern in InternVL3.5 4B and Qwen3-VL-8B. For InternVL3.5 4B, the selected arbitration heads are concentrated in a broad mid layer band spanning roughly layers 13 to 22, with the strongest layer level CERCER around layer 21. The selected backbone heads are more diffuse, but retain a visible late layer tail at layers 32 and 34. Relative to Hulu-Med (Fig. 4), the arbitration band in InternVL3.5 4B appears slightly deeper overall. Qwen3-VL-8B shows the same qualitative organization. Its arbitration heads again form a contiguous band, here extending roughly from layers 14 to 24, with the strongest CERCER around layer 22. Backbone heads are distributed more broadly across the same range and also extend into the late layers, particularly layer 34. Thus, although the exact peak layer varies by architecture, the overall separation remains consistent: heads with high conflict sensitivity occupy a wide override band, whereas clean preserving backbone heads are more dispersed and include a late layer component. Taken together with Hulu-Med, these results suggest that the arbitration circuit is not confined to a single architecture. Across all three backbones, localization recovers a broad mid layer override region rather than isolated single head hotspots, while backbone heads remain comparatively distributed. This cross model regularity supports the view that the wideband override pattern reflects a shared functional organization of arbitration failure. 17.3 Cross-Dataset and Cross-Model Transfer We evaluate whether the selected intervention targets transfer beyond the dataset or model used for localization. For arbitration failure, transfer is measured by the change in resist rate, ΔResist . For brake failure, transfer is measured by the change in unknown rate, ΔUR . Arbitration heads transfer consistently across datasets and model backbones, whereas brake head transfer is weaker and less directional. This asymmetry suggests that arbitration heads capture a more reusable textual override mechanism, while brake heads are more sensitive to the target model and visual degradation regime. Table 20: Cross dataset transfer results. Arbitration transfer reports ΔResist ; brake transfer reports ΔUR . Model Failure Mode Transfer #Heads ΔResist ΔUR Hulu-Med 4B Arbitration SLAKE heads → VQA-RAD 10 30.77 – Arbitration VQA-RAD heads → SLAKE 6 12.44 – Brake HealMed heads → SLAKE 5 – 4.55 Brake SLAKE heads → HealMed 6 – 21.56 InternVL3.5 4B Arbitration SLAKE heads → VQA-RAD 8 13.71 – Arbitration VQA-RAD heads → SLAKE 10 1.14 – Brake HealMed heads → SLAKE 7 – 1.29 Brake SLAKE heads → HealMed 7 – 3.68 Table 21: Cross model transfer of arbitration head sets. Transfer is reported as ΔResist . Benchmark Transfer #Heads ΔResist VQA-RAD InternVL → Hulu-Med 10 17.95 VQA-RAD Hulu-Med → InternVL 6 12.00 SLAKE VQA InternVL → Hulu-Med 8 5.07 SLAKE VQA Hulu-Med → InternVL 10 2.76 Table 22: Cross model transfer of brake head sets. Transfer is reported as ΔUR . Benchmark Transfer #Heads ΔUR SLAKE VQA Hulu-Med → InternVL 6 0.00 SLAKE VQA InternVL → Hulu-Med 7 -3.03 HealMed-VQA Hulu-Med → InternVL 6 0.42 HealMed-VQA InternVL → Hulu-Med 7 -2.76 Analysis. The transfer results reveal a consistent asymmetry between the two intervention targets. Arbitration head transfer remains positive across dataset and model shifts. In cross dataset transfer, Hulu-Med 4B obtains ΔResist gains of 30.77 and 12.44 points, while InternVL3.5 4B also shows positive transfer with gains of 13.71 and 1.14 points. Together with the results, where transferred arbitration head sets improve ResistResist in all evaluated directions, this suggests that arbitration heads capture a reusable textual override mechanism rather than a dataset specific localization artefact. Brake head transfer is weaker and less directional. Although cross dataset transfer can increase URUR, the magnitude varies substantially across target datasets and model backbones. Moreover, cross model brake transfer does not consistently improve abstention: on SLAKE VQA, Hulu-Med heads leave URUR unchanged on InternVL, whereas InternVL heads reduce URUR on Hulu-Med. These results indicate that brake heads are more tightly coupled to the target model and the visual degradation regime. We therefore interpret arbitration heads as comparatively transferable intervention targets, while treating brake heads as setting specific abstention control heads. 17.4 Clean-Competence-Stratified Head localization Figure 13: Bootstrap visualisation of clean competence stratified head sets. Each point denotes a bootstrap sample of the head set under a given stratum. Shaded regions show kernel density estimates. The 100% and 80% strata remain nearby, whereas 50% and 30% strata shift toward an intermediate region and the all wrong stratum is clearly separated. Our main experiments localize arbitration heads on a clean correct subset, where the model answers correctly before conflict injection. This filtering controls a key confounder: if the model does not know the answer under the clean condition, conflict following cannot be cleanly separated from ordinary knowledge failure. The resulting intervention is therefore intended to isolate conflict arbitration rather than general answer instability. We further examine whether the identified head set is specific to the fully clean correct subset. To this end, we repeat head localization under different clean competence strata, constructed to contain approximately 100%, 80%, 50%, 30%, and 0% clean correct samples. The 100% stratum corresponds to the main experimental protocol. Lower competence strata progressively include samples on which the model is less reliable before conflict is introduced. This analysis is used only as a sensitivity check: when clean competence is relaxed, selected heads may reflect a mixture of conflict arbitration, sample difficulty, uncertainty handling, and wrong answer commitment. Table 23: Clean competence stratified head set summary. The 100% stratum is the main setting. Lower strata are sensitivity checks only. Model Stratum #Heads Dominant layers Overlap with C100 Interpretation Hulu-Med 4B 100% clean correct 14 8, 12–15, 18–21 100% Main clean conflict arbitration set. 80% clean correct 15 12–22, mainly 18–21 65–75% Largely preserves the C100 core with neighbouring heads. 50% clean correct 21 14–27 plus 32–35 10–25% Less localized; recruits difficulty sensitive heads. 30% clean correct 24 15–27 plus 32–35 5–20% Shifts toward late answer commitment heads. 0% / all wrong 20 diffuse across 0–35 <<10% No stable conflict core; resembles wrong prior heads. InternVL3.5 4B 100% clean correct 21 13–21, 24, 34 100% Main clean conflict arbitration set. 80% clean correct 21 13–24 plus 34 65–80% Close to C100, with mid layer additions. 50% clean correct 21 15–28 plus 32–34 10–25% Shifts toward difficulty and commitment heads. 30% clean correct 25 15–28 plus 32–35 5–20% Similar to 50% with stronger late layer participation. 0% / all wrong 19 diffuse across 0–35 <<10% No stable clean conflict core. Analysis. Fig. 13 and Table 23 show a consistent transition as the clean correctness requirement is relaxed. At 80% clean correctness, the selected head sets remain close to the C100 stratum in the bootstrap embedding and retain substantial overlap with the main head set. This suggests that the main arbitration heads are not an artefact of using an overly narrow fully clean correct subset. The pattern changes at 50% and 30% clean correctness. For both models, the selected sets move away from the C100 region and become broader, with more participation from middle to late layers. This indicates that localization begins to capture factors beyond conflict arbitration, including sample difficulty, unstable clean predictions, uncertainty suppression, and answer commitment behavior. The all wrong stratum is separated from the clean correct strata and has little overlap with C100. In this regime, the model already fails before conflict injection, so the selected heads cannot be interpreted as clean conflict arbitration heads. These results support the clean correct filtering strategy used in the main experiments. The C100 setting yields a compact and stable arbitration related head set, whereas relaxing the clean competence constraint changes the selected head distribution and weakens the causal interpretation. 17.5 Full Results: ConflictMedQA Table 24 provides the full train/validation results on ConflictMedQA across three conflict injection positions. Across Qwen3-4B and Llama3.2-3B, Craft consistently lowers CFRCFR and increases the resist rate on the validation split, indicating that the selected arbitration heads generalize beyond the localization set. Compared with random ablation, Craft produces larger W2CW2C transitions while keeping C2WC2W low, suggesting that the intervention mainly corrects conflict induced errors with limited damage to clean predictions. The Before A setting yields the largest improvements, consistent with the injected evidence being closest to the answer position and therefore exerting stronger influence on final answer selection. Table 24: Full train/val results on ConflictMedQA. Results across three injection positions. Shaded rows denote Craft. Train Val Position Model Method CFR↓ Resist↑ C2W↓ W2C↑ CFR↓ Resist↑ C2W↓ W2C↑ Prefix Qwen3-4B Baseline 81.60 18.40 – – 83.33 16.67 – – Random 91.04 8.96 11.79 2.36 94.25 5.75 12.07 1.15 Craft 33.49 66.51 0.47 48.58 37.36 62.64 0.00 45.98 Llama3.2-3B Baseline 54.58 45.42 – – 57.69 42.31 – – Random 38.33 61.67 27.08 43.33 42.31 57.69 26.92 42.31 Craft 29.17 70.83 1.25 26.67 27.88 72.12 0.00 29.81 Before Q Qwen3-4B Baseline 89.15 10.85 – – 94.83 5.17 – – Random 84.43 15.57 1.89 6.60 89.66 10.34 1.15 6.32 Craft 76.42 23.58 0.94 13.68 68.39 31.61 1.15 27.59 Llama3.2-3B Baseline 77.92 22.08 – – 76.92 23.08 – – Random 81.67 18.33 12.50 8.75 81.73 18.27 11.54 6.73 Craft 54.58 45.42 0.42 23.75 57.69 42.31 0.00 19.23 Before A Qwen3-4B Baseline 75.47 24.53 – – 78.16 21.84 – – Random 73.11 26.89 5.66 8.02 76.44 23.56 5.75 7.47 Craft 25.00 75.00 0.47 50.94 23.56 76.44 0.57 55.17 Llama3.2-3B Baseline 35.00 65.00 – – 43.27 56.73 – – Random 49.58 50.42 22.08 7.50 48.08 51.92 16.35 11.54 Craft 22.50 77.50 2.08 14.58 31.73 68.27 2.88 14.42 17.6 Full Results: PubMedQA Table 25 reports the full train/validation results on PubMedQA across three conflict injection positions. Across both Qwen3-4B and Llama3.2-3B, Craft consistently reduces CFRCFR and improves the resist rate on the validation split, showing that the selected arbitration heads generalize beyond the localization set. The gains are especially strong for Qwen3-4B under the Before Q setting, where validation CFRCFR drops from 53.70 to 3.70 and the resist rate increases from 46.30 to 96.30. In contrast, random ablation is unstable: although it can reduce CFRCFR in some settings, it can also increase conflict following, as seen in the Before A position for Qwen3-4B. Craft also keeps C2WC2W close to zero in most settings while producing substantially larger W2CW2C transitions, indicating that the intervention primarily converts conflict induced wrong predictions back to correct ones with only limited collateral damage to otherwise clean predictions. Table 25: Full train/val results on PubMedQA. Results across three injection positions. Shaded rows denote Craft. Train Val Position Model Method CFR↓ Resist↑ C2W↓ W2C↑ CFR↓ Resist↑ C2W↓ W2C↑ Prefix Qwen3-4B Baseline 48.40 51.60 – – 49.67 50.33 – – Random 22.40 77.60 0.80 26.80 57.17 42.83 12.17 4.67 Craft 22.00 78.00 0.40 26.80 23.59 76.41 0.00 26.09 Llama3.2-3B Baseline 56.80 43.20 – – 58.46 41.54 – – Random 60.40 39.60 7.60 4.00 63.08 36.92 8.85 4.23 Craft 44.80 55.20 1.20 13.20 46.15 53.85 1.15 13.46 Before Q Qwen3-4B Baseline 53.60 46.40 – – 53.70 46.30 – – Random 19.80 80.20 0.60 34.40 46.30 53.70 3.91 11.30 Craft 6.20 93.80 0.00 47.40 3.70 96.30 0.00 50.00 Llama3.2-3B Baseline 69.20 30.80 – – 71.54 28.46 – – Random 65.60 34.40 4.80 8.40 68.08 31.92 4.62 8.08 Craft 57.20 42.80 1.60 13.60 59.23 40.77 1.54 13.85 Before A Qwen3-4B Baseline 61.80 38.20 – – 65.98 34.02 – – Random 92.80 7.20 31.00 0.00 85.33 14.67 20.54 1.20 Craft 51.00 49.00 0.00 10.80 51.41 48.59 0.00 14.57 Llama3.2-3B Baseline 64.80 35.20 – – 66.92 33.08 – – Random 70.00 30.00 8.40 3.20 72.31 27.69 9.23 3.85 Craft 52.80 47.20 2.40 14.40 54.62 45.38 2.31 14.62 17.7 Full Results on Multimodal Conflict Benchmarks Tables 26 and 27 report the full train/validation results for arbitration failure on VQA-RAD and SLAKE. We evaluate three positions for injecting misleading textual evidence: Prefix, Before Q, and Before A. Across both benchmarks, Craft consistently lowers validation CFRCFR and increases the resist rate relative to the baseline, indicating that the selected arbitration heads transfer beyond the subset used for localization. The gains are also larger than those obtained by random head ablation in nearly all settings. This is reflected by higher W2CW2C and generally low C2WC2W, suggesting that the intervention primarily converts conflict following predictions back to the correct answer rather than broadly disrupting clean predictions. The effect is most pronounced in the Before A setting, where the misleading evidence is placed closest to the answer token. On VQA-RAD, Craft reduces validation CFRCFR from 98.21 to 70.26 for Hulu-Med 4B, from 88.00 to 67.50 for Hulu-Med 7B, from 60.57 to 19.43 for InternVL3.5 4B, and from 57.98 to 34.00 for Qwen3-VL-8B. On SLAKE, the same intervention reduces validation CFRCFR from 87.73 to 46.32 for Hulu-Med 4B, from 75.00 to 45.00 for Hulu-Med 7B, from 16.40 to 5.84 for InternVL3.5 4B, and from 45.20 to 23.20 for Qwen3-VL-8B. Random ablation is substantially less reliable in this regime and can increase conflict following, especially for InternVL3.5 4B under Before A. These results support the interpretation that arbitration heads are not merely correlated with conflict behavior, but form effective intervention targets for reducing textual override. Table 28 reports the corresponding full results for brake failure under visual degradation. Here the target behavior is abstention, measured by the unknown rate URUR. Ablating the selected brake heads consistently increases held out URUR on both SLAKE and HealMed-VQA, with positive O2UO2U and near zero U2OU2O in most settings. On SLAKE, Craft raises validation URUR from 37.88 to 65.15 for Hulu-Med 4B, from 44.12 to 61.18 for Hulu-Med 7B, from 22.58 to 32.26 for InternVL3.5 4B, and from 42.42 to 55.30 for Qwen3-VL-8B. On HealMed-VQA, the same trend is stronger: validation URUR increases from 78.44 to 98.20 for Hulu-Med 4B, from 85.07 to 95.83 for Hulu-Med 7B, from 81.56 to 93.64 for InternVL3.5 4B, and from 84.72 to 95.14 for Qwen3-VL-8B. Together, the arbitration and brake results show that the two interventions act on distinct failure behaviors: arbitration head ablation suppresses textual override, while brake head ablation recovers abstention when visual evidence is insufficient. Table 26: Full train/val results on VQA-RAD textual conflict. Results across three injection positions. Shaded rows denote Craft. Train Val Position Model Method CFR↓ Resist↑ C2W↓ W2C↑ CFR↓ Resist↑ C2W↓ W2C↑ Prefix Hulu-Med 4B Baseline 33.76 66.24 – – 33.33 66.67 – – Random 34.53 65.47 2.81 2.05 33.59 66.41 1.79 1.54 Craft 16.11 83.89 0.77 18.41 15.38 84.62 0.26 18.21 Hulu-Med 7B Baseline 26.40 73.60 – – 25.80 74.20 – – Random 26.10 73.90 0.90 1.20 25.40 74.60 0.90 1.30 Craft 14.80 85.20 0.10 11.70 14.20 85.80 0.10 11.70 InternVL3.5 4B Baseline 39.83 60.17 – – 30.29 69.71 – – Random 47.71 52.29 9.60 1.72 34.29 65.71 4.57 0.57 Craft 16.33 83.67 0.43 23.93 9.14 90.86 0.57 21.71 Qwen3-VL-8B Baseline 29.10 70.90 – – 27.84 72.16 – – Random 29.40 70.60 0.90 0.60 28.10 71.90 0.90 0.64 Craft 12.80 87.20 0.20 16.50 11.90 88.10 0.20 16.14 Before Q Hulu-Med 4B Baseline 47.57 52.43 – – 44.62 55.38 – – Random 44.25 55.75 2.05 5.37 40.77 59.23 1.03 4.87 Craft 15.09 84.91 0.77 33.25 14.36 85.64 0.51 30.77 Hulu-Med 7B Baseline 37.20 62.80 – – 36.50 63.50 – – Random 36.00 64.00 0.80 2.00 35.20 64.80 0.80 2.10 Craft 15.60 84.40 0.20 21.80 14.70 85.30 0.20 22.00 InternVL3.5 4B Baseline 29.94 70.06 – – 21.71 78.29 – – Random 37.25 62.75 9.46 2.15 24.00 76.00 4.00 1.71 Craft 14.33 85.67 3.72 19.34 7.43 92.57 2.29 16.57 Qwen3-VL-8B Baseline 17.05 82.95 – – 16.16 83.84 – – Random 17.90 82.10 1.00 0.15 17.00 83.00 1.00 0.16 Craft 8.40 91.60 0.15 8.80 7.50 92.50 0.15 8.81 Before A Hulu-Med 4B Baseline 97.70 2.30 – – 98.21 1.79 – – Random 97.19 2.81 0.26 0.77 98.72 1.28 0.51 0.00 Craft 74.42 25.58 0.00 23.27 70.26 29.74 0.00 27.95 Hulu-Med 7B Baseline 89.20 10.80 – – 88.00 12.00 – – Random 89.50 10.50 0.35 0.05 88.30 11.70 0.35 0.05 Craft 68.20 31.80 0.00 21.00 67.50 32.50 0.00 20.50 InternVL3.5 4B Baseline 64.90 35.10 – – 60.57 39.43 – – Random 80.95 19.05 16.33 0.29 77.71 22.29 17.71 0.57 Craft 29.51 70.49 0.86 36.25 19.43 80.57 0.00 41.14 Qwen3-VL-8B Baseline 58.90 41.10 – – 57.98 42.02 – – Random 59.30 40.70 0.60 0.20 58.20 41.80 0.35 0.13 Craft 36.20 63.80 0.10 22.80 34.00 66.00 0.10 24.08 Table 27: Full train/val results on SLAKE textual conflict. Results across three injection positions. Shaded rows denote Craft. Train Val Position Model Method CFR↓ Resist↑ C2W↓ W2C↑ CFR↓ Resist↑ C2W↓ W2C↑ Prefix Hulu-Med 4B Baseline 18.17 81.83 – – 19.15 80.85 – – Random 21.32 78.68 3.93 0.70 21.28 78.72 2.95 0.82 Craft 11.16 88.84 3.86 10.87 12.11 87.89 5.07 12.11 Hulu-Med 7B Baseline 16.00 84.00 – – 15.40 84.60 – – Random 16.50 83.50 1.00 0.50 15.90 84.10 1.00 0.50 Craft 11.60 88.40 1.50 5.90 11.00 89.00 1.50 5.90 InternVL3.5 4B Baseline 15.71 84.29 – – 12.66 87.34 – – Random 20.50 79.50 8.06 3.27 18.18 81.82 8.28 2.76 Craft 8.90 91.10 0.49 7.30 6.66 93.34 0.65 6.66 Qwen3-VL-8B Baseline 16.20 83.80 – – 15.60 84.40 – – Random 16.90 83.10 0.95 0.25 16.20 83.80 0.85 0.25 Craft 11.60 88.40 0.30 4.90 10.80 89.20 0.30 5.10 Before Q Hulu-Med 4B Baseline 18.51 81.49 – – 19.15 80.85 – – Random 20.62 79.38 3.86 1.75 20.79 79.21 4.09 2.45 Craft 8.20 91.80 0.98 11.29 7.04 92.96 0.65 12.77 Hulu-Med 7B Baseline 14.80 85.20 – – 14.20 85.80 – – Random 15.30 84.70 0.80 0.30 14.80 85.20 0.90 0.30 Craft 7.20 92.80 0.40 8.00 6.70 93.30 0.40 7.90 InternVL3.5 4B Baseline 10.91 89.09 – – 8.93 91.07 – – Random 19.81 80.19 10.35 1.46 18.51 81.49 10.71 1.14 Craft 7.51 92.49 0.63 4.03 5.84 94.16 0.65 3.73 Qwen3-VL-8B Baseline 12.40 87.60 – – 11.80 88.20 – – Random 13.10 86.90 0.85 0.15 12.40 87.60 0.75 0.15 Craft 7.90 92.10 0.25 4.75 7.20 92.80 0.25 4.85 Before A Hulu-Med 4B Baseline 87.17 12.83 – – 87.73 12.27 – – Random 92.08 7.92 5.75 0.84 91.00 9.00 4.26 0.98 Craft 47.62 52.38 5.33 44.88 46.32 53.68 4.91 46.32 Hulu-Med 7B Baseline 76.40 23.60 – – 75.00 25.00 – – Random 77.50 22.50 1.40 0.30 76.20 23.80 1.50 0.30 Craft 46.20 53.80 0.80 31.00 45.00 55.00 1.00 31.00 InternVL3.5 4B Baseline 21.06 78.94 – – 16.40 83.60 – – Random 30.23 69.77 10.56 1.39 23.70 76.30 8.77 1.46 Craft 9.38 90.62 1.60 13.27 5.84 94.16 2.27 12.82 Qwen3-VL-8B Baseline 46.50 53.50 – – 45.20 54.80 – – Random 48.20 51.80 2.05 0.35 47.00 53.00 2.10 0.30 Craft 24.60 75.40 0.30 22.20 23.20 76.80 0.30 22.30 Table 28: Full train/val results for brake failure under visual degradation. Craft ablates brake heads to recover abstention. ΔUR=O2U−U2O =O2U-U2O. Shaded rows denote Craft. Train Val Benchmark Model Method UR↑ O2U↑ U2O↓ UR↑ O2U↑ U2O↓ SLAKE Hulu-Med 4B Baseline 36.90 – – 37.88 – – Random 44.23 8.10 0.77 45.45 8.33 0.76 Craft 64.20 27.30 0.00 65.15 27.27 0.00 Hulu-Med 7B Baseline 43.40 – – 44.12 – – Random 47.85 5.05 0.60 48.82 5.29 0.59 Craft 60.10 16.70 0.00 61.18 17.06 0.00 InternVL3.5 4B Baseline 23.29 – – 22.58 – – Random 20.55 1.37 4.11 16.13 0.00 6.45 Craft 26.03 2.74 0.00 32.26 9.68 0.00 Qwen3-VL-8B Baseline 41.80 – – 42.42 – – Random 47.10 6.05 0.75 47.73 6.06 0.75 Craft 54.60 12.80 0.00 55.30 12.88 0.00 HealMed-VQA Hulu-Med 4B Baseline 77.90 – – 78.44 – – Random 86.10 10.00 1.80 86.83 10.18 1.80 Craft 97.60 19.70 0.00 98.20 19.76 0.00 Hulu-Med 7B Baseline 84.60 – – 85.07 – – Random 87.70 4.30 1.20 88.19 4.31 1.19 Craft 95.30 11.04 0.34 95.83 11.10 0.34 InternVL3.5 4B Baseline 81.10 – – 81.56 – – Random 82.00 3.05 2.15 82.47 3.06 2.15 Craft 93.20 12.48 0.38 93.64 12.46 0.38 Qwen3-VL-8B Baseline 84.30 – – 84.72 – – Random 87.10 4.20 1.40 87.50 4.17 1.39 Craft 94.70 11.09 0.69 95.14 11.11 0.69 17.8 Validation Confusion Matrix Diagnostics Figure 14: Binary validation confusion matrices for arbitration intervention on Hulu-Med 4B and VQA-RAD. To better understand how arbitration head ablation changes prediction behavior, we visualise validation set confusion matrices for Hulu-Med 4B on VQA-RAD. Since most validation answers are binary yes or no labels, we focus on this major subset and report row normalised yes or no confusion matrices before and after ablating the selected arbitration heads. Analysis. Before ablation (Fig. 14), conflicting context induces errors in both directions: 36.6% of gold yes samples are predicted as no, and 53.2% of gold no samples are predicted as yes. After ablating the selected arbitration heads, the diagonal entries increase from 63.4% to 88.4% for gold yes samples and from 46.8% to 94.4% for gold no samples. This shows that the correction is bidirectional rather than driven by a generic preference for a single answer label. Instead, the intervention restores the yes or no decision boundary under conflicting context, consistent with the increased resist rate reported in the main results. 18 Failure Case Studies Figure 15: Representative persistent failures after Craft intervention. Upper: arbitration failure case. Lower: brake failure case. Although Craft improves arbitration and brake behavior in aggregate, some validation samples remain unresolved after intervention. We visualise two representative persistent failures in Fig. 15. The goal of this analysis is not to introduce new metrics, but to clarify which residual errors remain after sparse head excision. The two examples correspond to the two failure modes studied in the main paper: arbitration failure under conflicting textual evidence and brake failure under degraded visual evidence. 18.1 Arbitration Failure The upper case in Fig. 15 is a VQA-RAD arbitration example from the Before A setting (Sec. 4.1). The question asks whether the visual finding is caused by a medical process or physical injury. The gold answer is medical process, while the conflicting textual cue favours physical injury. After Craft intervention, the model does move toward the gold answer. The gold against wrong margin improves from Δgold,wrong=−4.0417 _gold,wrong=-4.0417 to Δgold,wrong=−0.8913 _gold,wrong=-0.8913. This indicates that the selected arbitration heads contribute to the harmful conflict pathway. However, the margin remains negative, so physical injury is still the top ranked answer. This case therefore remains an arbitration failure. The residual error is not a complete failure of intervention, since the conflict preference is weakened. Instead, the remaining conflict signal is still strong enough to determine the final answer. The case also requires etiology level visual grounding: the model must infer whether the finding is better explained by a medical process or by physical injury. This explains why the figure labels the residual pattern as Fail to Resolve Etiology. 18.2 Brake Failure The lower case in Fig. 15 is a SLAKE brake example from the image conflict setting (Sec. 4.2). The question asks which organ is bigger in the image, kidney or liver. In the original image, the gold answer is Liver. After diagnostic region masking, the visual evidence is insufficient for a reliable concrete answer, so the desired behavior is unknown. Before intervention, the ranking is wrong>gold>unknown wrong> gold> unknown. After Craft intervention, the ranking becomes wrong>unknown>gold wrong> unknown> gold. Thus, Craft raises the score of unknown, showing that the selected brake heads are related to abstention control. However, Kidney remains the top ranked answer, so the model still commits to a concrete wrong prediction. The intervention partially activates abstention behavior but does not make unknown overtake the wrong concrete answer. This explains why the figure labels the residual pattern as Fail to Abstain. Together, the cases show that sparse head excision can correct the target failure mode in aggregate, while hard samples may still require additional mechanisms. In particular, difficult etiology attribution may require stronger image grounded reasoning, and difficult brake cases may require adaptive abstention thresholds or localization aware decoding. 19 Computational Cost We report the inference overhead of Craft style head intervention under the Before Q setting on VQA-RAD. The benchmark is conducted on one validation sample for each model, using the corresponding selected arbitration heads. We compare two implementations. The first applies head masking through forward hooks, which is the implementation used in our analysis experiments. This implementation leaves the original attention computation intact and therefore does not reduce the profiled matrix multiplication FLOPs. The second implements a query head skipping variant that removes the selected query heads from the attention computation before the output projection. This variant reduces profiled FLOPs, but it is implemented with Python level module patching rather than fused kernels. Table 29: Computational cost of head intervention on VQA-RAD. Hooked masking is the analysis implementation; true head skipping offers marginal FLOP savings. Model Implementation #Heads Normal Lat. Interv. Lat. Δ . Normal FLOPs Δ Hulu-Med 4B Hooked masking 6 206.39 ms 210.31 ms +1.90% 19539.96G 0.00% Hulu-Med 4B True head skipping 6 207.11 ms 208.10 ms +0.48% 19539.96G 0.11% lower InternVL3.5 4B Hooked masking 10 43.27 ms 49.43 ms +14.24% 3629.54G 0.00% InternVL3.5 4B True head skipping 10 43.91 ms 47.03 ms +7.11% 3629.54G 0.10% lower Analysis. Hooked masking is designed for faithful analysis, not acceleration. It leaves attention computation intact and therefore does not reduce profiler FLOPs. True head skipping gives a small theoretical FLOP reduction, but the sparse intervention level is too small for latency gains without fused kernels. These results clarify that Craft is a safety oriented post hoc intervention rather than an inference acceleration method. 20 Limitations and Future Directions Limitations. (i) Model scope. Our mechanistic analysis is restricted to open-weight models up to 14B parameters. Larger or closed-source systems may distribute arbitration and brake circuits across a wider set of layers, and our fixed scan window might miss relevant heads in such architectures. (i) Language coverage. All evaluated prompts and conflict injections are in English. Medical VLMs deployed in multilingual clinical environments may exhibit language-dependent attention routing, and the current head discovery pipeline has not been validated on non-English inputs. (i) Imaging modality. The benchmarks predominantly feature chest X-rays and fundoscopy. Three-dimensional modalities (CT, MRI) that produce multi-slice inputs could engage different visual token aggregation patterns not captured by our single-image protocol. (iv) Static intervention. Head suppression is applied uniformly across all tokens and decoding steps. A more refined approach would modulate intervention strength dynamically based on per-token conflict signals, which we leave unexplored. Future directions. (i) Adaptive and input-conditioned intervention. A promising extension is to replace static head zeroing with a lightweight gating network that activates suppression only when conflict is detected in the current input. This would reduce the risk of over-abstention on unambiguous cases and enable deployment as a real-time clinical safety layer that imposes negligible cost on routine (non-conflicting) queries. Preliminary experiments with a single-layer binary classifier on the residual stream at the conflict-sensitive layer suggest that such gating is feasible with <<0.5% additional latency. (i) Scaling to multimodal agents and report generation. Current evaluation focuses on single-turn VQA; however, clinical AI systems increasingly operate as multi-step agents that retrieve records, compare serial images, and generate structured reports. Extending Craft to trace arbitration circuits across multi-turn reasoning chains, where conflicts may arise between the model’s own prior outputs and new visual evidence, would address a critical safety gap. We envision a hierarchical tracing framework that first localises conflict-sensitive layers within each reasoning step, then tracks how unresolved conflicts propagate across steps, enabling targeted intervention at the earliest point of failure.ervention at the earliest point of failure.