Paper deep dive

CARD: Diagnosing Belief to Action Routing Failures in Vision Language Models

Souptik Kumar Majumdar, Fabian Kögel, Andreas Bulling

Year: 2026Venue: arXiv preprintArea: cs.CVType: PreprintEmbeddings: 86

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%

Last extracted: 8/24/2026, 5:15:11 AM

Summary

The paper introduces CARD (Cross-Axis Routing Diagnostic), a method to diagnose whether Vision-Language Models (VLMs) use internally represented mental states (like beliefs) to drive actions. Using a new benchmark called Relay Chain, the authors find that while VLMs can linearly decode belief representations, they fail to route this information to action predictions, exhibiting a 'routing failure' where actions are determined by constant priors rather than partner beliefs.

Entities (12)

CARD · method · 98%Relay Chain · dataset · 97%Vision-Language Models · model-class · 95%Action · prediction-axis · 92%Belief · mental-state · 92%Gemma-4 · model-instance · 90%InternVL2.5-8B · model-instance · 90%LLaVA-NeXT-Video-7B-hf · model-instance · 90%Qwen2-VL-7B · model-instance · 90%Activation Steering · technique · 88%Linear Probing · technique · 88%GridToM · dataset · 85%

Relation Signals (11)

CARD → evaluatedon → Relay Chain

confidence 98% · Applied to open-weight VLMs on Relay Chain -- a new cooperative grid-world benchmark we propose

Gemma-4 → isinstanceof → Vision-Language Models

confidence 95% · Gemma-4-E4B-it

Qwen2-VL-7B → isinstanceof → Vision-Language Models

confidence 95% · Four open-weight VLMs: Qwen2-VL-7B

InternVL2.5-8B → isinstanceof → Vision-Language Models

confidence 95% · InternVL2.5-8B

LLaVA-NeXT-Video-7B-hf → isinstanceof → Vision-Language Models

confidence 95% · LLaVA-NeXT-Video-7B-hf

CARD → uses → Activation Steering

confidence 95% · CARD ... steers activations along one axis while measuring the response of a different axis's prediction.

CARD → uses → Linear Probing

confidence 95% · A linear probe on the TB-vs-FB activation difference yields a unit-norm probe direction... used as the steering pool

Vision-Language Models → exhibits → Routing Failure

confidence 93% · we diagnose a critical routing failure: models fail to incorporate belief representations into their next action prediction

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Linear probes and activation steering have uncovered that vision-language models (VLMs) internally represent mental states such as agents' beliefs, knowledge, and intentions. However, it is unclear whether and how these representations are used by downstream predictions along these axes. To close this gap, we introduce Cross-Axis Routing Diagnostic (CARD), which steers activations along one axis while measuring the response of a different axis's prediction. Applied to open-weight VLMs on Relay Chain -- a new cooperative grid-world benchmark we propose -- we diagnose a critical routing failure: models fail to incorporate belief representations into their next action prediction, effectively leaving valuable information about their partners unused.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

83,305 characters extracted from source content.

Expand or collapse full text

CARD: Diagnosing Belief-to-Action Routing Failures in Vision–Language Models Souptik Majumdar, Fabian Kögel and Andreas Bulling Universität Stuttgart 70569, Stuttgart Germany souptik.majumdar, fabian.koegel, andreas.bulling @vis.uni-stuttgart.de Abstract Linear probes and activation steering have un- covered that vision-language models (VLMs) internally represent mental states such as agents’ beliefs, knowledge, and intentions. However, it is unclear whether and how these representations are used by downstream pre- dictions along these axes. To close this gap, we introduce Cross-Axis Routing Diagnostic (CARD), which steers activations along one axis while measuring the response of a differ- ent axis’s prediction. Applied to open-weight VLMs on Relay Chain – a new cooperative grid- world benchmark we propose – we diagnose a critical routing failure: models fail to incorpo- rate belief representations into their next action prediction, effectively leaving valuable infor- mation about their partners unused. 1 Introduction Vision-language models (VLMs) are increasingly used in human-AI cooperation, such as embodied robots, AI coaches, or driving co-pilots, that require Theory of Mind (ToM; Premack and Woodruff, 1978) to infer what the interaction partner might see, know or want. To diagnose whether such models actually represent such mental states of their partners, two families of diagnostic tools have emerged: Linear probing tests if and where states are encoded in the models, while activation steering has primarliy been used to verify whether probed directions causally drive the model’s pre- diction and to control its outputs. However, they leave one question unanswered: Does the model’s internal representation of a partner’s mental state actually drive its actions? Existing behavioural benchmarks (Gu et al., 2024; Bortoletto et al., 2025; Shi et al., 2025; Chen et al., 2024a) score outputs along three axes, belief, knowledge and intentions but without inspecting and connecting their internal representations to the model predictions. Thus, they cannot discriminate wrong actions that reflect an absent representation from those that reflect an unused one. Alterna- tively, probes and prior work in activation steering (Alain and Bengio, 2016; Belinkov, 2022; Hewitt and Liang, 2019; Elazar et al., 2021; Bortoletto et al., 2024; Räuker et al., 2023; Geiger et al., 2021; Wu et al., 2023b) access internal representations but apply both read-out and intervention only to the same axis. For example, a probe might find a belief representation, and steering confirms its influence on next belief prediction (same-axis) but does not test its influence on action prediction (cross-axis). Thus, both approaches do not diagnose the rout- ing of mental representations to predictions across axes. This matters especially in cooperative tasks, where the correct action depends on the mental state representation of the partner agent, which the VLM may encode internally, yet doesn’t use when predicting its next action. To address this limitation, we make the following contributions: 1.We introduce Cross-Axis Routing Diagnostic (CARD), a novel approach to diagnose how men- tal state representations influence model predic- tions across axes by steering representations in one axis and assessing the prediction of another. 2.To evaluate CARD, we propose Relay Chain, a new multi-agent cooperative grid-world bench- mark that extends the grid-world format of Grid- ToM (Li et al., 2025) with a cooperative action axis. 3.Our experiments demonstrate critical routing failures across four open-weight VLM families, where the belief representation causally influ- ences the belief and knowledge predictions but not the action prediction – despite the belief be- ing linearly decodable at78–94%from the same layer. arXiv:2608.20763v1 [cs.CV] 21 Aug 2026 Within-axis steering Steer a feature; read the same axis’ answer. Input LM belief answer steering vector vs CARD — cross-axis read-out Steer one axis; measure a different axis’ prediction. Within-axis cell is a positive control. Input LM v • belief direction Belief prediction within-axis · positive control Δ Knowledge prediction cross-axis · CARD Δ Action prediction cross-axis · CARD Δ Figure 1: Overview of CARD. For each axisa∈belief, intent, knowledgea linear probe on paired True-Belief / False-Belief activations at layerLyields a unit-norm directionv a . At inference time, we hook the residual stream at the same layer and addαv a (steering axisa,α ∈ −10, +10), then read out the answer head for a different question axisb̸= a. A causally-routed feature moves the cross-axis read-out; an encoded-but-unread feature does not. Within-axis cells (a = b, diagonal) serve as positive controls. 2 Related Work Behavioural ToM benchmarks.A large body of work evaluates ToM in foundation models through mental-state questions (Gu et al., 2024; Bortoletto et al., 2025; Shi et al., 2025; Xu et al., 2024; Wu et al., 2023a; Chen et al., 2024a; Xiao et al.; Chen et al., 2024c; Jin et al., 2024; Kim et al., 2023; Zhang et al., 2025). SimpleToM (Gu et al., 2024) demonstrated a gap between explicit and applied ToM on text LLMs while ToM-SSI (Bortoletto et al., 2025) extended to situated multi-agent in- teractions and reported a behavioural drop from percept to belief to intention. MindPower (Zhang et al., 2025) showed that ToM-aware prompting improved cooperative success rates in embodied VLM settings. CARD complements these by test- ing whether a model’s internal mental-state repre- sentations actually drive its cooperative actions, not just whether its overall predictions are correct. Probes and activation steering.Probes and stan- dard (same-axis) activation steering to inspect mod- els have a long history in text-only LLMs (Alain and Bengio, 2016; Belinkov, 2022; Elazar et al., 2021; Bortoletto et al., 2024; Räuker et al., 2023): A linear probe verifies that a feature is encoded in the residual stream, and steering amplifies the probe direction to test whether the model’s answer to the same feature shifts. GridToM (Li et al., 2025) brought this protocol to multimodal grid-world ToM and was the closest prior multimodal-ToM work pairing probes with activation interventions. CARD extends the protocol from same-axis to a cross-axis matrix that tests whether a probe direc- tion is read by a different prediction. Like Borto- letto et al. (2024), we use linear probes with Hewitt– Liang random-label controls (Hewitt and Liang, 2019) to verify that the recovered belief direction is structured rather than a probe-expressivity artefact. Studying routing in VLMs Mechanistic inter- pretability tools to study full circuits from inputs to outputs that drive predictions include activation patching and causal mediation (Vig et al., 2020; Meng et al., 2022), activation-addition / steering vectors (Turner et al., 2024; Li et al., 2023; Rimsky et al., 2024; Zou et al., 2023), direct logit attribution (Wang et al., 2022), automated circuit discovery (Conmy et al., 2023), and linear-relation decoding (Geva et al., 2023; Hernandez et al., 2024). Recent work extended these to VLMs (Yang et al., 2026; Liu et al., 2025; Häon et al., 2025). Unlike dis- tributed alignment search Geiger et al., 2021; Wu et al., 2023b, which gradient-searches a rotation aligning a high-level variable to a subspace, CARD uses a fixed pre-trained probe direction and condi- tions the read-out on a specific prediction. These techniques have largely targeted factual recall, re- fusal, or bias; CARD applies cross-axis routing diagnostics to mental-state representations in coop- erative agents. 3 Method Our Cross-Axis Routing Diagnostic (CARD) takes three steps: (1) identify a steering vector for a men- tal representation in one axis, e.g. belief; (2) steer the model in its direction and measure the predic- tion of a different axis, e.g. action; (3) repeat the steering in the opposite direction as a control. Paired-statement probe extraction. For each axisA∈belief, intent, knowledgeand scenario s, we present the VLM with the keyframe clip and a pair of statements⟨s + A ,s − A ⟩ :s + A matches the ground-truth axis label (e.g. for belief, “the protag- onist saw the traveller crossing” when the protago- nist did see it) ands − A is its negation. Letz ℓ,h (s)∈ R d h denote the per-(layer, head) activation at the final input token under each statement. Per (layer, head), anL 2 -regularised logistic regression is fit on the2Nactivationsz ℓ,h (s + A ), z ℓ,h (s − A ) s with binary labels (1 fors + A , 0 fors − A ), yielding a unit- norm probe directionu ℓ,h A ∈R d h that points from negated toward matched, plus a held-out accuracy. We use the top-K=56(layer, head) pair ranked by held-out accuracy as the steering pool (∼7% of L×Hheads, following the head-selection conven- tion of Li et al., 2023).σ ℓ,h A is the per-head activa- tion standard deviation alongu ℓ,h A on the training set. For False-Belief scenarios the matched-class caption is taken from the protagonist’s view rather than the world state (see Appendix B.1). Cross-Axis Routing Diagnostic (CARD).Given the steering-axis probe set(u ℓ,h A ,σ ℓ,h A ) over the top-Kpool, we register a forward hook on the self- attention output of every layer that contains at least one selected (layer, head) pair. The hook adds, at the last-token position only and only to the per- head slices of the selected pairs, a scaled multiple of the probe direction: z ℓ,h ← z ℓ,h + ασ ℓ,h A u ℓ,h A , α∈−10, 0, +10, (1) whereαis unit-free (the additive magnitude is |α|σtraining-projection standard deviations) and remaining heads are left untouched. AllKheads are steered simultaneously, so the intervention is multi-direction by construction; the rest of the for- ward pass runs unchanged. The VLM then answers a question on a different axisB ̸= A, yielding a TB−FB accuracy gapg B (α) = acc TB B (α) − acc FB B (α). Steering withα=+10andα=−10 should move this gap in opposite directions if axis A’s direction causally influences the axis-Bpredic- tion; we summarise the effect as the gap-shift – the change in g B between the two steering polarities: M A,B = g B (+10)− g B (−10).(2) We additionally report a mean-accuracy shift M acc A,B = max α∈±10 acc B (α)−acc B (0) to catch uniform answer modulations a signed gap-shift would miss. On Relay Chain we populate the3×3 matrix withA,B ∈ belief, intent, knowledge: same-axis entries (A = B) serve as positive con- trols; off-diagonal entries test whether the linearly- decodableAdirection is used by theBprediction. A columnM A,B ∗ A ≈ 0 paired with at least one non-trivial row implies theB ∗ prediction is decou- pled fromA’s linearly-decodable direction – the routing failure. Four-tuple signature. On the unified suc- cess+counter partition, for each (VLM, protag- onist, question axis) we record the four-tuple acc TB succ , acc FB succ , acc TB cnt , acc FB cnt . This signature discriminates the three behaviours (constant-prior, world-state tracker, belief-conditional), which are indistinguishable on success-only data. Projection-removal control. As a negative- direction test, we subtract a scaled projection of the per-head residual along u ℓ,h A : z ℓ,h ← z ℓ,h −cσ ℓ,h A (u ℓ,h A ·z ℓ,h )u ℓ,h A , c∈0.5, 1, 2, 5. (3) c=1 is canonical projection-removal;c>1over- subtracts. Prompt-resistance test.We ask whether prompt engineering alone can recover belief-conditional action. Twelve variants (cost framing, chain-of- thought, perception-first / rule-based, in-context exemplars; Appendix B.6) are evaluated across192 cells. A variant passes on a lever-holder (a 1 or a 2 ) if bothacc TB > 50%andacc FB > 50%– correctly releases under TB and correctly holds under FB. Evaluation protocol. Each axis question is pre- sented as a binary choice and parsed from JSON. Probes, top-Kselection, and CARD evaluation share the unified success+counter partition (444 scenarios), split50/50into a disjoint probe-set (222scenarios; probe training and top-Kranking) and eval-set (222scenarios; CARD, projection- removal, behavioural baselines), stratified by pro- tagonist×world-state outcome (App. B.1). Accu- racy is reported separately for TB vs FB and, where relevant, separated into success vs counter blocks. 4 Dataset and Task Cooperative Relay Chain dataset. We evalu- ate CARD (section 3) on Relay Chain – a novel three-agent cooperative grid-world environment (see Figure 2). CARD’s diagnostic logic – steer one axis and read out a different prediction – re- quires a testbed in which a partner’s mental state causally drives an action prediction that is distinct from the belief prediction that it steers. While Grid- ToM (Li et al., 2025) pairs probes with activation interventions and supplies multi-agent scenarios and belief-attribution questions (first- and second- order beliefs), it does not offer a cooperative action axis: All questions only ask what an agent believes but not what the protagonist (the ego agent whose mental state we probe) should do given the part- ner’s belief. The belief-to-action CARD cell is therefore undefined – not because a partner is ab- sent, but because there is no action prediction that depends on the partner’s mental state. In contrast, on the Relay Chain, two agents (a 1 ,a 2 ) each hold a lever that opens one gate; a traveller (a 3 ) must cross both gates in sequence to reach the goal. The cooperative structure is a relay: lever-operators must keep their gates open in sequence as the traveller passes through. Crucially, as no single agent can solve this task alone, the trav- eller’s optimal action is well-defined only relative to the partner’s perception, turning belief-to-action into a well-defined test of routing. Formal task description. We treat each sce- nario as a tuples = (w 0 ,τ,a p ).w 0 is the ini- tial grid layout (agents, levers, gates, goal).τ = (w 0 ,w 1 ,...,w T ) withT ∈ [16, 32]is the deter- ministic cooperative trajectory – the joint multi- agent state at each step.a p ∈ a 1 ,a 2 ,a 3 is the protagonist – the ego agent whose mental state we probe. The VLM does not seeτdirectly: it receives ak-frame visual rendering (k ∈ [4, 8]) subsampled at relay-phase boundaries (a 1 acquires lever,a 2 ac- quires lever,a 3 reaches goal) plus midpoints (see Appendix A.2). Belief variants. Each scenario appears in two layouts that holdτand agent identities fixed and differ only ina p ’s observation of the cooperative event (a p ’s partner releases/crosses): True-Belief (s TB , event withina p ’s field of view) and False- Belief(s FB , event in fog of war). Lete∈0, 1in- dicate whether the event in fact occurs in the world. The counter pair replacesswiths ′ wheree = 0 (the lever is never released, the gate never opens), preserving the same TB/FB perceptual structure. Questions and labels.For each axis A ∈ belief, intent, knowledgewe ask one bi- nary questionq A (a p ) whose ground-truth label y A (s,b) ∈ 0, 1is a deterministic function of (w 0 ,τ,e,b), whereb ∈ TB, FBselects the be- lief variant. Belief asks whether the protagonist perceived the cooperative event; knowledge asks whether they can justify that it happened (percep- tion plus integration of frame evidence); intent asks the next cooperative action. The VLMMoutputs a predictionˆy M,A (s,b)that we score againsty A . Per-axis label logic is in Appendix A.6. Full construction details (scenario generator, paired layouts, counter-scenarios, per-axis ground- truth labels) can be found in Appendix A. Counter scenarios. On success-only data (the typical ToM benchmark setup, where the coopera- tive event always occurs), a model that emits the safe default on every input – “don’t release” for lever-operators, “keep waiting” for the traveller – is correct on the majority of cells, so high accu- racy is not evidence of belief-conditional reason- ing. The counter pair (e = 0) breaks this shortcut: the safe default becomes wrong precisely where it was right under success, so constant-prior and belief-conditional models produce visibly different accuracy patterns. Construction mechanics and per- axis counts can be found in Appendix A.5, Table 3. Figure 2 illustrates the overall layout. Three axes. Building on the belief–desire– intention framework (Bratman, 1987) and the percept–belief–intention (PBI) causal structure used in recent ToM benchmarks (Bortoletto et al., 2025), we instrument each scenario along three axes: belief – does the protagonist see the coop- erative event (first-order PBI percept; what con- tent the agent holds in mind?); intent – in the BDI sense of present-directed action-recommendation: should the protagonist take the cooperative action now (PBI intention); knowledge – does the protag- onist know whether the cooperative event occurred separate from the propositional content of belief). Knowledge serves as a control: within counter sce- narios (where the world did not change), it is the only axis whose label still flips between TB and FB – with observation alone, while belief and intent labels stay constant. Including knowledge in the probe-extraction pool forces the learned direction to track observation rather than the world-state out- come. Appendix D.1 verifies that the three probe directions are neither collinear nor orthogonal. Models. Four open-weight VLMs: Qwen2-VL- 7B (Yang et al., 2024a), Gemma-4-E4B-it (Gemma G a 1 a 2 a 3 Success-TB: a 1 can see the crossing L L G a 1 a 2 a 3 Success-FB: a 1 cannot see the crossing L L G L L a 1 a 2 a 3 Counter-TB: event did not occur G L L a 1 a 2 a 3 Counter-FB: event did not occur a 1 protagonist (lever-holder) a 2 partner (lever-holder) a 3 traveller gate open (event) gate closed (counter) lever cooperative-event location Figure 2: Relay Chain four-tuple discrimination layout. Panel elements: protagonista 1 (red, lever-operator) with field-of-view disk (radius5); partnera 2 (blue, lever-operator); travellera 3 (green); gates (green when open, dark-red when closed); leversL; goalG. Top row: success (event occurs). Bottom row: counter (lever-operator never releases; gate stays closed). Left column: True-Belief (TB) –a 1 ’s lever close enough to perceive the crossing. Right column: False-Belief (FB) – lever beyond a 1 ’s view. Team, Google DeepMind, 2026), LLaVA-NeXT- Video-7B-hf (Zhang et al., 2024), and InternVL2.5- 8B (Chen et al., 2024b). We used multi-frame inputs (4–8 keyframes per scenario). 5 Results We conducted a series of experiments that together show a routing failure on the cooperative-action axis. First, we verify belief is linearly decodable and used on the belief question (positive control). We then establish a behavioural dissociation: the same model that reads belief on the belief ques- tion collapses to a constant per-protagonist prior on the intent question. CARD steering along the linearly-decodable belief direction substantially shifts the belief and knowledge predictions but leaves the cooperative-action prediction essentially unchanged – the mechanistic evidence. Three ab- lations close the natural alternative explanations: projection-removal rules out amplitude limitation, a prompt-resistance test rules out a prompting arte- fact, and CLEVR rules out a globally-broken steer- ing hook. Belief is decodable and used on the belief question. We confirm belief is encoded and be- haviourally used on the belief question – a binary read-out of whether the protagonist saw the co- ModelProbe acc.TB-acc.FB-acc. Qwen2-VL81.5± 1.8100.020.0 Gemma-494.5± 0.679.330.7 LLaVA-NeXT78.1± 0.980.020.0 InternVL2.593.3± 0.589.32.0 Table 1: Positive control: linear-probe accuracy (mean ±std across three seeds; chance= 50%) and be- havioural TB/FB accuracy on the belief question. operative event (“Doesa p see the partner release / cross?”). A linear probe on the TB-vs-FB acti- vation difference reaches held-out accuracy well above chance on every VLM (see Table 1). We rule out spurious correlations with a Hewitt–Liang random-label control (Hewitt and Liang, 2019), which collapses probe accuracy to chance (see Ta- ble 11 in the Appendix). Behaviourally, every VLM answers the same question near ceiling on the TB layout and much lower on FB (see Table 1) – consistent with the prediction reading event oc- currence but not gating by the protagonist’s field of view. We thus rule out the possibility that belief is simply not represented: any later failure on the intent question cannot be blamed on missing or unreadable belief. Belief is used on belief and knowledge but ig- nored on intent questions.We find a three-way behavioural dissociation on Relay Chain: the three axis questions, asked of the same model on the same scenarios with the same activations, produce qualitatively distinct accuracy patterns across all four VLMs (see the per-VLM matrix in Table 6 in the Appendix). On the belief question every VLM reproduces the world-state tracker signature: across the four (success/counter)×(TB/FB) cells, the answer is correct on three (success-TB, counter-TB, counter- FB) and wrong only on success-FB, where the event occurred but outside the protagonist’s field of view (per-VLM breakdown in Table 6 in the Ap- pendix). We read this as the prediction using event occurrence but not gating by observation. On the knowledge question the same models be- come behaviourally belief-conditional in the suc- cess block: TB is answered “knows” and FB is answered “doesn’t know”, correctly tracking the protagonist’s observation rather than the world state alone. We conclude observation is at least partially consulted for this question. On the intent question the same models col- lapse to a constant per-protagonist prior (“don’t release” on every cell, incorrect only on success-TB where releasing is the right action). The constant- prior signature is indistinguishable across the four VLMs – a striking convergence given otherwise divergent probe accuracies and architectures, and the strongest sign that intent answers are gener- ated without consulting either event occurrence or observation. We thus observe three distinct signatures on the same model and the same activations: belief- Q reads world state, knowledge-Q reads observa- tion, and intent-Q reads neither. The intent ques- tion alone is behaviourally belief-insensitive – the lonely outlier among the three axes. We next probe this dissociation mechanistically with CARD, ask- ing whether the structural cause is upstream (belief unreachable from the intent prediction) rather than behavioural (belief read but ignored). CARD intent column is uniformly null (rout- ing failure). We find a clean three-way disso- ciation under CARD steering. Across the full 4×3×3 = 36-cell sweep (four VLMs×three pro- tagonists×three probe directions; see Table 7 in the Appendix), the best-αintent accuracy stays es- sentially at its no-steering baseline on nearly every cell, while the same steering hooks substantially shift the belief and knowledge predictions. Aver- ProbeBelief Q (%)Intent Q (%)Epist. Q (%) Qwen2-VL belief15.3±6.20.029.2±6.9 intent2.1±2.80.00.0 knowledge3.5±3.50.00.0 Gemma-4 belief33.3±7.64.9±3.529.9±7.6 intent16.7±6.20.019.4±6.9 knowledge0.7±1.40.06.2±4.9 LLaVA-NeXT belief41.0±8.30.040.3±8.3 intent22.2±6.90.067.4±7.6 knowledge24.3±7.60.060.4±8.3 InternVL2.5 belief31.9±7.65.6±4.229.9±7.6 intent1.4±2.10.03.5±3.5 knowledge35.4±8.315.3±6.220.1±6.9 Table 2: Per-scenario flip rate ona 1 : of144scenar- ios, the fraction whose answer differs betweenα=+10 andα=−10steering (i.e., whether steering moves the model at all, before changes average out). In the in- tent column (middle),9of12(VLM, probe) entries are exactly0; only three are above zero (Gemma-4 and InternVL2.5 on the belief probe, InternVL2.5 on the knowledge probe;≤15.3%), with direction-balanced flips that cancel in net accuracy (Table 7 in the Ap- pendix). The belief and knowledge columns flip many scenarios under the same hook. aged across protagonists (see Figure 3), intent is flat on every VLM and the belief and knowledge axes move; the per-(VLM, protagonist) breakdown (see Figure 6 in the Appendix) shows the intent null holds across all 12 cells, not just the target lever-operatora 1 . The gap-shift matrix ona 1 (see Figure 4) tells the same story at the gap-shift level. We further show the null is per-scenario tight. Ona 1 the intent argmax is invariant toα=±10 steering on the overwhelming majority of scenar- ios across all four VLMs (see Table 2); the small fraction that do flip (InternVL2.5 and Gemma-4) are direction-balanced and cancel in net accuracy, while the belief and knowledge questions flip a substantial fraction of scenarios under the same hooks. We rule out the possibility that the intent answer was already pinned to one side before steer- ing: without steering, the intent baseline distri- bution is mixed, so the null cannot be explained that way. Analysis confirms the null is statistically tight: across108intent measurements, the largest improvement in mean intent accuracy is bounded at18.7p at95%confidence, versus22.9p on the beliefintentknowledge Question axis 0 25 50 75 100 Mean accuracy (%, protagonist-mean) 72.9 77.4 =+4.6 66.2 67.5 =+1.3 81.5 84.7 =+3.2 Qwen2-VL no steering (=0) best ±10 beliefintentknowledge Question axis 69.5 80.0 =+10.5 66.266.2 =0 76.6 84.4 =+7.8 Gemma-4 no steering (=0) best ±10 beliefintentknowledge Question axis 0 25 50 75 100 52.6 60.8 =+8.3 66.2 67.8 =+1.6 59.5 70.0 =+10.5 LLaVA-NeXT no steering (=0) best ±10 beliefintentknowledge Question axis 62.7 66.4 =+3.7 65.9 66.5 =+0.6 69.5 72.2 =+2.7 InternVL2.5 no steering (=0) best ±10 Figure 3: Steering improves accuracy on the belief and knowledge predictions, but never on the intent answer prediction, averaged across the three protagonists (a 1 ,a 2 ,a 3 ). For each (VLM, question axis) we report the mean TB/FB accuracy with no steering (α=0, grey) and with the best-case steering (α∈−10, +10across all probe directions, in VLM color).∆on top is best−baseline. Belief and knowledge improve by+2.7to+10.6p; the intent∆is essentially0on every VLM. The identical intent baseline (∼ 66%) across VLMs reflects the constant- prior signature. Per-protagonist breakdown in Figure 6 in the Appendix. belief positive control (Appendix C.5). We interpret this as a structural routing failure: the linearly-decodable belief direction is encoded, readable on the belief and knowledge predictions, but not used in the action prediction. Ablation studies. We finally evaluated our findings against four alternative explanations: amplitude-limited steering (projection-removal control), a prompting artefact, cross-axis steering on a non-ToM task (CLEVR), and a last-token- locus artefact (distributed-token steering). Projection-removal control An alternative is that our additive steering simply does not push the belief subspace hard enough to register at the intent prediction, and a stronger or differently-shaped perturbation would. We test the opposite limit: instead of adding belief, we remove it from the residual stream entirely by subtracting itscσ-scaled projection from the per-head slice (Eq. 3) atc∈0.5, 1, 2, 5. The intent TB−FB gap stays essentially flat at everycon every VLM with a complete sweep, while the same removal shifts the belief -question gap toward chance as expected (positive control). The intent null survives both directions of perturbation, so it is structural, not a consequence of under-driving the belief subspace. No prompt achieves belief-conditional action Another possibility is that the intent prediction routes belief only when scaffolded by the right prompt. We test this with a pre-registered sweep of 12 prompt variants (CoT, perception-first, rule- based, high-stakes cost, 1/2-shot in-context)×four VLMs×four protagonist conditions (192total; Figure 11 in the Appendix). No prompt recovers belief-conditional action. Cross-axis Steering on Non-ToM task Finally, the hook itself could be at fault – too weak to register anywhere. Applying the same hook to CLEVR on the same models (Table 10 in the Appendix) yields substantial best-αaccuracy gains where there is headroom; the few flat cells reflect ceiling effects on shape and colour. The intent null is therefore a target-axis property, not a hook artefact. Distributed-token steering A further possibility is that the intent prediction reads belief at a different token position than our last-token hook. We re-run CARD with the steering hook fired at two additional positions (vision-end beliefintentknowledge Question axis Probe axis (steered) 30.1 ±2.9 0.0 ±0.0 15.7 ±5.1 4.2 ±0.0 0.0 ±0.0 2.3 ±3.3 2.3 ±3.3 0.0 ±0.0 0.5 ±0.7 Qwen2-VL beliefintentknowledge Question axis 66.7 ±0.0 3.2 ±4.6 53.2 ±5.1 32.4 ±1.3 0.0 ±0.0 30.6 ±7.4 1.9 ±1.7 0.0 ±0.0 16.7 ±4.1 Gemma-4 beliefintentknowledge Question axis 16.7 ±3.4 0.0 ±0.0 26.4 ±9.7 10.6 ±1.7 0.0 ±0.0 28.2 ±16.6 17.1 ±3.5 0.0 ±0.0 29.6 ±6.6 LLaVA-NeXT beliefintentknowledge Question axis 21.3 ±13.1 7.4 ±4.6 2.3 ±0.7 2.8 ±3.0 0.5 ±0.7 2.3 ±0.7 20.8 ±12.0 11.6 ±10.3 1.9 ±1.7 InternVL2.5 0 10 20 30 40 50 60 70 | gap| (p) at = ± 10 on a 1 Figure 4: CARD intent column on lever-operatora 1 is uniformly null (red box) across four VLMs. Rows: probe axis being steered. Columns: question axis being asked. Cell value:|gap(α= + 10)− gap(α=− 10)|in p, mean ±std across three seeds. Non-intent columns shift by tens of p under the same probe; the intent column does not move. Same steering hook, same scenarios, same protagonist – only the intent prediction fails to read from the belief direction. and a vision-plus-last slice covering both) in ad- dition to the last-token target, using three probe sources: the paired-statement belief and intent probes plus a per-position condition probe trained at vision-end. Sweepingα ∈ −10, +10yields 18(probe-source×target-position) measurements. Across all, the maximum intent|∆ acc |vs theα=0 baseline is0.3p; maximum shift is around1p (see Figure 13 in the Appendix). The result is there- fore not an artefact of the last-token hook position. Summary. We established that belief is linearly decodable and used on the belief question, then a behavioural dissociation between belief- and intent- question signatures, then the causal dissociation under intervention, and finally closed the three al- ternative explanations (amplitude, prompt, hook). Belief direction is encoded, readable on the belief and knowledge predictions, and not used by the ac- tion prediction – a routing failure, not an absence. 6 Discussion The simplest account is that the cooperative-action prediction does not use the linear belief direction. Additive steering and projection-removal both shift the belief and knowledge predictions but leave the intent prediction essentially unchanged on every VLM and every protagonist; the same prediction defaults to a constant per-protagonist prior that the visual evidence does not modulate. Two observa- tions rule out the simpler alternative that the predic- tion is just broken or saturated. First, the prediction does vary across inputs — its prior differs by pro- tagonist (“don’t release” ona 1 ,a 2 ; “go” ona 3 ) — so it is not globally stuck on one answer. Second, the belief signal is present in the activations: the belief and knowledge predictions, exposed to the same residual stream, do use it. The prediction is therefore capable of varying and the information it would need is available; the failure to use belief is a routing fact specific to the intent prediction, not a general inability to respond. 7 Conclusion and Future work In this work we introduced CARD as a new cross- axis routing diagnostic that disambiguates "feature absent" from "feature encoded but unused by the action prediction". Using this new diagnostic we showed in a cooperative task that current VLMs encode a partner’s belief well enough to answer the belief question, yet none usefully use that belief when asked what to do. This represents a struc- tural routing failure visible in current VLMs. One natural extension remains open: nonlinear probes would test whether the action prediction reads a subspace our linear probes miss. As such, our results suggest that verbal-ToM alignment does not imply action-ToM alignment: alignment train- ing that updates only verbalised belief predictions, without an explicit objective on the action predic- tion, would leave this routing gap unaddressed. Limitations We demonstrate the routing failure on one task fam- ily – Relay Chain, a 2D grid-world – and four open- weight VLMs. CARD and projection-removal re- quire activation-stream access, so we cannot sub- ject closed-source VLMs to the mechanistic arms; We do not claim generality to larger model scales (40–400B, where applied-ToM behaviour is known to improve (Gu et al., 2024; Bortoletto et al., 2024)), non-cooperative ToM, or real-world video. Ethical considerations The work uses publicly available open-weight VLMs and a synthetic grid-world task with no human subjects or personally-identifying data. CARD reports an internal property of the model and surfaces a limitation that deployers should be aware of; we see no direct misuse pathway. Datasets, prompts, and code will be released under permissive licenses. References Guillaume Alain and Yoshua Bengio. 2016. Under- standing intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644. Yonatan Belinkov. 2022. Probing classifiers: Promises, shortcomings, and advances. Computational Linguis- tics, 48(1):207–219. Matteo Bortoletto, Constantin Ruhdorfer, and Andreas Bulling. 2025. Tom-ssi: Evaluating theory of mind in situated social interactions. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 32252–32277. Matteo Bortoletto, Constantin Ruhdorfer, Lei Shi, and Andreas Bulling. 2024. Brittle minds, fixable ac- tivations: Understanding belief representations in language models. arXiv preprint arXiv:2406.17513. Michael E. Bratman. 1987. Intention, Plans, and Prac- tical Reason. Harvard University Press, Cambridge, MA. Hongzhan Chen, Hehong Chen, Ming Yan, Wenshen Xu, Gao Xing, Weizhou Shen, Xiaojun Quan, Chen- liang Li, Ji Zhang, and Fei Huang. 2024a. Social- bench: Sociality evaluation of role-playing conver- sational agents. In Findings of the Association for Computational Linguistics: ACL 2024, pages 2108– 2126. Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, and 1 others. 2024b. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271. Zhuang Chen, Jincenzi Wu, Jinfeng Zhou, Bosi Wen, Guanqun Bi, Gongyao Jiang, Yaru Cao, Mengting Hu, Yunghwei Lai, Zexuan Xiong, and 1 others. 2024c. Tombench: Benchmarking theory of mind in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15959– 15983. Arthur Conmy, Augustine N. Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga- Alonso. 2023. Towards automated circuit discov- ery for mechanistic interpretability. In Advances in Neural Information Processing Systems (NeurIPS). Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. 2021. Amnesic probing: Behavioral expla- nation with amnesic counterfactuals. Transactions of the Association for Computational Linguistics, 9:160– 175. Atticus Geiger, Hanson Lu, Thomas Icard, and Christo- pher Potts. 2021. Causal abstractions of neural net- works. Advances in neural information processing systems, 34:9574–9586. Gemma Team, Google DeepMind. 2026. Gemma 4: Our most capable open models to date. Technical report, Google DeepMind. Accessed: May 2026. Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. 2023. Dissecting recall of factual associa- tions in auto-regressive language models. In Proceed- ings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12216–12235. Yuling Gu, Oyvind Tafjord, Hyunwoo Kim, Jared Moore, Ronan Le Bras, Peter Clark, and Yejin Choi. 2024. Simpletom: Exposing the gap between explicit tom inference and implicit tom application in llms. arXiv preprint arXiv:2410.13648. Bear Häon, Kaylene Stocking, Ian Chuang, and Claire Tomlin. 2025. Mechanistic interpretability for steer- ing vision-language-action models. arXiv preprint arXiv:2509.00328. Evan Hernandez, Arnab Sen Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg, Jacob Andreas, Yonatan Belinkov, and David Bau. 2024. Linearity of relation decoding in transformer language models. In Inter- national Conference on Learning Representations, volume 2024, pages 10504–10526. John Hewitt and Percy Liang. 2019. Designing and interpreting probes with control tasks. In Proceed- ings of the 2019 conference on empirical methods in natural language processing and the 9th interna- tional joint conference on natural language process- ing (emnlp-ijcnlp), pages 2733–2743. Chuanyang Jin, Yutong Wu, Jing Cao, Jiannan Xiang, Yen-Ling Kuo, Zhiting Hu, Tomer Ullman, Antonio Torralba, Joshua Tenenbaum, and Tianmin Shu. 2024. Mmtom-qa: Multimodal theory of mind question answering. In Proceedings of the 62nd Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16077–16102. Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. 2017. CLEVR: A diagnostic dataset for compositional language and elementary visual rea- soning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2901–2910. Hyunwoo Kim, Melanie Sclar, Xuhui Zhou, Ronan Bras, Gunhee Kim, Yejin Choi, and Maarten Sap. 2023. Fantom: A benchmark for stress-testing machine theory of mind in interactions. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14397–14413. Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. 2023. Inference- time intervention: Eliciting truthful answers from a language model. Advances in Neural Information Processing Systems, 36:41451–41530. Xinyang Li, Siqi Liu, Bochao Zou, Jiansheng Chen, and Huimin Ma. 2025. From black boxes to transparent minds: Evaluating and enhancing the theory of mind in multimodal large language models. arXiv preprint arXiv:2506.14224. Yiming Liu, Yuhui Zhang, and Serena Yeung-Levy. 2025. Mechanistic interpretability meets vision lan- guage models: Insights and limitations. In The Fourth Blogpost Track at ICLR 2025. Andrés Marafioti, Orr Zohar, Miquel Farré, Merve Noyan, Elie Bakouch, Pedro Cuenca, Cyril Zakka, Loubna Ben Allal, Anton Lozhkov, Nouamane Tazi, and 1 others. 2025. Smolvlm: Redefining small and efficient multimodal models. arXiv preprint arXiv:2504.05299. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associa- tions in gpt. Advances in neural information process- ing systems, 35:17359–17372. David Premack and Guy Woodruff. 1978. Does the chimpanzee have a theory of mind? Behavioral and Brain Sciences, 1(4):515–526. Tilman Räuker, Anson Ho, Stephen Casper, and Dylan Hadfield-Menell. 2023. Toward transparent ai: A survey on interpreting the inner structures of deep neural networks. In 2023 ieee conference on secure and trustworthy machine learning (satml), pages 464– 483. IEEE. Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Turner. 2024. Steer- ing llama 2 via contrastive activation addition. In Proceedings of the 62nd Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 15504–15522. Haojun Shi, Suyu Ye, Xinyu Fang, Chuanyang Jin, Leyla Isik, Yen-Ling Kuo, and Tianmin Shu. 2025. Muma-tom: Multi-modal multi-agent theory of mind. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 1510–1519. Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Ulisse Mini, and Monte MacDiarmid. 2024. Activation addition: Steering language models without optimization. Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. 2020. Investigating gender bias in language models using causal mediation analysis. Advances in neural information processing systems, 33:12388– 12401. Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. 2022. In- terpretability in the wild: a circuit for indirect ob- ject identification in gpt-2 small. arXiv preprint arXiv:2211.00593. Yufan Wu, Yinghui He, Yilin Jia, Rada Mihalcea, Yu- long Chen, and Naihao Deng. 2023a. Hi-tom: A benchmark for evaluating higher-order theory of mind reasoning in large language models. In Find- ings of the Association for Computational Linguistics: EMNLP 2023, pages 10691–10706. Zhengxuan Wu, Atticus Geiger, Thomas Icard, Christo- pher Potts, and Noah Goodman. 2023b. Interpretabil- ity at scale: Identifying causal mechanisms in alpaca. Advances in neural information processing systems, 36:78205–78226. Yang Xiao, Jessie Wang, Qiancheng Xu, Changhe Song, Chunpu Xu, Yi Cheng, Wenjie Li, and Pengfei Liu. Tomvalley: Evaluating the theory of mind reasoning of llms in realistic social context. Hainiu Xu, Runcong Zhao, Lixing Zhu, Jinhua Du, and Yulan He. 2024. Opentom: A comprehensive bench- mark for evaluating theory-of-mind reasoning capa- bilities of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8593–8623. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Hao- ran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, and 39 others. 2024a. Qwen2 technical report. ArXiv, abs/2407.10671. Jingcheng Yang, Tianhu Xiong, Shengyi Qian, Klara Nahrstedt, and Mingyuan Wu. 2026. Circuit trac- ing in vision-language models: Understanding the internal mechanisms of multimodal thinking. arXiv preprint arXiv:2602.20330. Qwen An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Hao- ran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxin Yang, Jingren Zhou, Jun- yang Lin, and 25 others. 2024b. Qwen2.5 technical report. ArXiv, abs/2412.15115. Ruoxuan Zhang, Qiyun Zheng, Zhiyu Zhou, Ziqi Liao, Siyu Wu, Jian-Yu Jiang-Lin, Bin Wen, Hongxia Xie, Jianlong Fu, and Wen-Huang Cheng. 2025. Mindpower: Enabling theory-of-mind reasoning in vlm-based embodied agents.arXiv preprint arXiv:2511.23055. Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. 2024. Llava-next: A strong zero-shot video understanding model. Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, and 1 others. 2023. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405. A Dataset construction A.1 Dataset breakdown Relay Chain has444unique scenario bases dis- tributed across three protagonist conditions (a 1 ,a 2 as lever-holders anda 3 as traveller). We render each base in two layouts that preserve the trajectory but differ in the protagonist’s lever-to-gate Manhat- tan distance – True-Belief (≤ 5, lever withina p ’s field of view of the gate) and False-Belief (> 5, lever beyond) – yielding888evaluation cases per axis (Table 3). Scenarios split into success blocks where the cooperative event happens and counter blocks where it is omitted, so the same protagonist appears once with and once without the cooperative outcome on identical perceptual access. The full 444-scenario partition is split50/50into a disjoint probe-set and eval-set (App. B.1). A.2 Relay Chain scenario generator We render scenarios as11× 11grids with three agents (a 1 ,a 2 ,a 3 ), two gates, and one goal cell. Each scenario is a16–32-step trajectory; we sub- sample4–8keyframes covering the cooperative event window. Keyframes anchor on four relay- phase boundaries: (i) episode start; (i)a 1 acquires lever A; (i)a 2 acquires lever B; (iv)a 3 reaches the goal. We then insert midpoints between consec- utive boundaries to yield up to7distinct frames per scenario, following GridToM’s keyframe helper. When a phase is never reached, the last available step is used as a fallback so the boundary list stays strictly monotonic. Agents move on Manhattan- distance shortest paths; lever-state transitions are scripted to produce the target cooperative chain. Random seeds are saved with each scenario for reproducibility. A.3 Perceptual-access minimal pair For each scenario we generate two layouts that share trajectory and event sequence but differ in protagonist’s lever position relative to the gate it controls. TB: lever placed at Manhattan distance ≤ 5from the gate (within field of view). FB: lever at distance> 5. All other state (agent identi- ties, gate positions, goal, traveller path) is identical. We verify pair matching by hashing the trajectory and confirming bit-exact agreement on the non- protagonist state slice. A.4 Example scenario layouts Figure 5 shows one True-Belief and one False- Belief layout for each protagonist condition. For the lever-holders (a 1 ,a 2 ), TB places the protago- nist’s own lever within Manhattan distance5of the gate it controls (inside the field-of-view disk); FB places it beyond. For the traveller (a 3 ), TB places the agent within view of the gate ahead so the gate- state is perceivable; FB places it beyond. All panels show the success block (both gates open), so the contrast across columns is purely perceptual. A.5 Counter-story generator We construct the counter pair for each scenario by re-running the scripted trajectory but omitting the cooperative event: the lime traveller stalls3cells before the gate (it never crosses), or the pink lever- holder fails to release the lever at the cooperative deadline (the gate never opens for the cooperative crossing). We preserve the protagonist’s perceptual access (TB/FB layout), rewrite captions with the asymmetric oracle-vs-pov rule (§3), and recompute the binary correct answer by enumeration. A.6 Axis-specific ground-truth labels We compute ground-truth labels deterministically from the layout and event-outcome pair. Belief: “does the protagonist see the event”. Intent: “should the protagonist take the cooperative action now”, mapped to release/keep-holding for lever-holders and continue/wait for the traveller. Knowledge: “does the protagonist have enough perceptual ac- cess to determine whether the cooperative event occurred” – for lever-holders this is “can the pro- tagonist see the traveller”, for the traveller it is “can the protagonist see at least one lever-holder”. The knowledge label depends only on perceptual vis- ibility, not on whether the event in fact occurred; success and counter blocks therefore share the same knowledge label on each layout (TB→TRUE, FB →FALSE), so a knowledge-conditional answer is forced to track perceptual access rather than the world-state outcome. B Probe + behavioral protocol details B.1 Probe extraction For each axis we extract per-(layer, head) activa- tions at the final token under each paired statement. The difference∆ A = a(s + A )− a(s − A ) is the train- ing input for the linear probe; labels are±1for ProtagonistRoleSuccessCounterTotal a 1 lever-operator10044144 a 2 lever-operator10050150 a 3 traveller10050150 Total 300 144 444 Table 3: Relay Chain dataset breakdown by protagonist and world-state outcome. All three axes (belief, intent, knowledge) share the same scenario base; each entry is a multi-frame trajectory with 4–8 keyframes. G a 1 a 2 a 3 a 1 (lever-holder) -- TB (relevant within view) L L G a 1 a 2 a 3 a 1 (lever-holder) -- FB (relevant beyond view) L L G a 1 a 2 a 3 a 2 (lever-holder) -- TB (relevant within view) L L G a 1 a 2 a 3 a 2 (lever-holder) -- FB (relevant beyond view) L L G a 1 a 2 a 3 a 3 (traveller) -- TB (relevant within view) L L G a 1 a 2 a 3 a 3 (traveller) -- FB (relevant beyond view) L L a 1 a 2 a 3 protagonist (red outline) gate open lever cooperative-event location Figure 5: Relay Chain scenario layouts for each protagonist condition (rows:a 1 anda 2 lever-holders,a 3 traveller) under True-Belief and False-Belief layouts (columns). The protagonist is highlighted with a red outline; the shaded disk shows their field of view (radius5cells; anything outside is fog of war and not perceptible). In TB the protagonist stands next to its gate-controlling lever or just before the gate ahead; in FB it stands at the far lever or in a corner, so the field-of-view disk shifts with it and the cooperative event falls outside. All panels show the success block (both gates open) so the contrast is purely perceptual. matched/mismatched statements. We use logistic regression with default L 2 regularisation (C=1). K-selection / CARD-eval protocol. To prevent K-selection bias from probe training, top-Krank- ing, and CARD evaluation sharing scenarios, we split the unified success+counter partition (444sce- narios) into a fully disjoint probe-set and eval-set stratified50/50by protagonist×world-state out- come (222scenarios each). The probe-set is used for probe training (single75/25stratified train/val split per (layer, head), seeded for reproducibility) and top-K=56cell ranking by held-out val accu- racy; CARD is evaluated only on the disjoint eval- set. To quantify probe-training variance we re-run with three seeds42, 43, 44for the internal75/25 split and report the mean across seeds. B.2 CARD steering hook We implement the steering hook as a forward hook on each chosen layer’s self-attention output: at the last-token position only,z ℓ,h ← z ℓ,h + ασ ℓ,h A u ℓ,h A (Eq. 1), withu ℓ,h A the unit-norm probe direction at that (layer, head),σ ℓ,h A the training-set standard deviation of the scalar projectionu ℓ,h A · z ℓ,h (s + A ), andα ∈ −10, 0, +10. We steer the top-56 per-(layer, head) cells ranked by held-out probe accuracy (cells selected byargsort(val_acc)de- scending), the same selection used by the canonical probe-extraction sweep. We do not normalise by sequence position or apply position-specific scal- ing. B.3 Projection-removal At each (layer, head) we subtractcσ ℓ,h A (u ℓ,h A · z ℓ,h )u ℓ,h A fromz ℓ,h at the last-token position (Eq. 3), withc ∈ 0.5, 1, 2, 5.σ ℓ,h A is the per- (layer, head) standard deviation ofu ℓ,h A · z ℓ,h (s + A ) across the unified-partition training set. Under projection-removal LLaVA-NeXT occasionally ap- pends commentary tokens after the JSON answer; we extract the answer with a regex on the yes/no field in addition to the canonical JSON-parse path. Recomputed accuracies match the raw answer- token distribution and are reported in §5. B.4 Temporal probe accuracy (Qwen2-VL) We report linear-probe held-out accuracy at four trajectory fractionst ∈ 0.25, 0.5, 0.75, 1.0on Qwen2-VL, for the three cooperative-ToM axes (Table 4). Peak (best layer/head) accuracy is satu- rated at100%from the earliest timestep; mean ac- curacy varies by axis but is stable to within∼ 1p. The representation is formed early. We restrict the temporal sweep to Qwen2-VL on compute-budget grounds (each timestep is a full per-(layer, head) probe-extraction pass over444scenarios); we treat it as a single-VLM ablation, not as a generality claim. B.5 Pre-registration record We pre-registered the T-A falsifier before running the V0–V11 sweep: a cell passes only if both TB and FB accuracy exceed50%on a lever-holder. The hypothesis space (12 variants×4 VLMs×4 protagonist conditions) was fixed before any data was collected. B.6 T-A prompt variants Table 5 summarises the12prompt variants used in the T-A pre-registered sweep, grouped into four families (cost framing, chain-of-thought, perception-first / rule-based, in-context exemplars). Each variant prepends a prefix to the canonical in- tent question; the verbatim prefix text for every variant is in §B.7 below. B.7 T-A prompt prefixes (verbatim) We reproduce here the exact prefix appended before the canonical intent question for each T-A variant. Variants whose prefix differs by protagonist role list both lever-holder and traveller text. V0 (baseline). No prefix. The canonical intent question is asked unchanged. V1 (cost: implicit consequences). Lever-holder: Reminder of cooperative conse- quences: if the lever-holder releases while the traveller has not yet crossed the gate, the gate closes on the traveller and the team fails the task. If the lever-holder keeps holding after the trav- eller has crossed, the team is delayed but does not fail. Traveller:Reminder of cooperative conse- quences: if the traveller continues toward a closed gate, it gets stuck and the team fails. If the traveller waits when the gate is open, the team is delayed but does not fail. V2 (cost: explicit numerical). Lever-holder:Cost framing:action ‘Yes’ (release/no-longer-hold)→cost2.0if the trav- eller has not yet crossed, else cost0. Action ‘No’ (keep holding)→cost1.0if the traveller has al- ready crossed, else cost0. Choose the action that minimises expected cost given what the protago- nist can observe. Traveller: Cost framing: action ‘Yes’ (continue toward gate)→cost2.0if the gate is currently closed, else cost0. Action ‘No’ (stop and wait)→ cost1.0always (delay but no failure). Choose the action that minimises expected cost given what the protagonist can observe. V3 (utility framing). Question framing: what action would MAXIMISE the team’s chance of completing the cooperative task successfully, given what the protagonist can currently observe? V4 (chain-of-thought). Think step by step: (1) FIRST identify what the protagonist can perceive given the protagonist’s position and fog-of-war range. (2) THEN con- sider what the optimal action is conditional on Axist=0.25 t=0.5 t=0.75 t=1.0peak belief69.969.169.369.3100.0 intent95.695.194.794.3100.0 knowledge91.991.290.890.4100.0 Table 4: Linear-probe held-out accuracy (%) as a function of trajectory fractionton Qwen2-VL, for the three cooperative-ToM axes (belief, intent, knowledge). “Peak” = best (layer, head) probe; the remaining columns are means across all (layer, head) cells. The peak is saturated at100%fromt=0.25onward. Mean accuracy decreases very slightly (∼ 1p) witht: while the best head remains perfectly recoverable, fewer heads on average carry a clean axis-specific signal as later tokens accumulate downstream context. The relevant headline is the saturated peak: the feature is present from early in the trajectory. VariantFamilyDescription V0costbaseline; canonical intent question, no prefix V1costimplicit consequences: “if released too early, gate closes on traveller; team fails.” V2costexplicit numerical cost: release=cost2.0if not crossed, hold=cost1.0if crossed V3costutility framing: “what action MAXIMISES team success?” V4CoTstep-by-step: perception→ belief→ cost→ action V5CoThigh-stakes framing: “a wrong release may kill the team” V6CoTV4 chain-of-thought + V2 numerical costs V7perception-rule“State what the protagonist can see, then answer.” V8perception-ruleexplicit rule: “if you can see the crossing, release; else hold.” V9perception-ruleV7 perception-first + V2 cost numbers V10in-contextone-shot worked example (Manhattan distance≤ 5→release;> 5→hold) V11in-contexttwo-shot: V10 plus a second example of opposite polarity Table 5: The12T-A prompt variants used to evaluate prompt-resistance. Each variant prepends a prefix to the canonical intent question; in-context variants additionally include worked examples. what the protagonist actually knows. (3) FI- NALLY produce a single Yes or No answer. The protagonist may be partly ignorant about events outside its fog. Only choose the answer the pro- tagonist would optimally pick given its actual per- ceptual state. V5 (high-stakes safety framing). Safety-critical reminder: this is a cooperative team task in which a wrong action by the lever- holder may TRAP the traveller permanently (team failure), and a wrong action by the traveller may STRAND the agent in a closed-gate room (team failure). Only choose actions that minimise the risk of catastrophic failure given what the protag- onist can actually observe. V6. V4 chain-of-thought prefix concatenated with the V2 explicit-cost prefix (lever-holder or traveller, by role). V7 (perception-first). BEFORE answering, identify what the protago- nist agent can actually perceive. The protagonist has a fog-of-war view of radius5cells from its current position; cells beyond that radius are NOT visible to the protagonist. The lever-holder’s view of the gate they control depends on the lever’s distance from the gate. If the lever sits next to the gate (Manhattan distance≤ 5), the lever-holder CAN see who crosses. If the lever sits far from the gate (Manhattan distance> 5), the lever-holder CANNOT see who crosses. Use only what the pro- tagonist actually observes to choose the answer. V8 (rule-based). Rule: a lever-holder should release its lever ONLY IF it can directly perceive that the agent it is enabling has finished crossing the gate. Oth- erwise it must keep holding. A traveller should continue toward a gate ONLY IF it can perceive that the gate is open. Apply this rule given what the protagonist actually observes (fog-of-war ra- dius 5). V9.V7 perception-first prefix concatenated with the V2 explicit-cost prefix (by role). V10 (one-shot in-context). EXAMPLE: a lever-holder agent’s lever sits2 cells away from a gate (within fog-of-war radius 5). The lever-holder CAN see the gate. The lever- holder observes the traveller crossing the gate. Question: should the lever-holder release the lever now? Answer: Yes, because the lever-holder has perceived the traveller crossing and the gate is no longer needed. Now answer the following with the same belief- conditional reasoning. If the protagonist CAN perceive the relevant event, recommend the action that responds to that perception. If the protago- nist CANNOT perceive the event, recommend the conservative action. V11 (two-shot in-context). EXAMPLE 1 (TrueBelief case): a lever-holder agent’s lever sits2cells away from a gate (within fog-of-war radius5). The lever-holder CAN see the gate and observes the traveller crossing. Ques- tion: should the lever-holder release? Answer: Yes, because the protagonist perceived the cross- ing. EXAMPLE 2 (FalseBelief case): a lever-holder agent’s lever sits10cells away from a gate (be- yond fog-of-war radius5). The lever-holder CAN- NOT see the gate and does not know whether the traveller has crossed. Question: should the lever-holder release? Answer: No, because the protagonist has NOT perceived the crossing and releasing would risk trapping the traveller. Now answer the following with the same belief- conditional reasoning. Distinguish: does the pro- tagonist actually perceive the event in question, or not? Choose the action that matches the pro- tagonist’s actual perceptual access. C Full results tables C.1 Per-VLM, per-protagonist, per-axis behaviour matrix We report the full per-cell behavioural accura- cies that underlie the three-way dissociation in F2. The matrix covers4 VLMs× 3 axes× 3 protag× 2 outcomes× 2 beliefs =144cells (Table 6). Read- ers should look for the shape contrast: belief-Q cells track the world-state outcome, knowledge- Q cells track TB/FB (perceptual access), intent-Q cells stay on the per-protagonist prior. C.2 Intent ∆ acc per (VLM, protag, probe) For every(VLM× protagonist× probe direction) cell in the4× 3× 3 = 36-cell CARD sweep on the intent question, we report best-αaccuracy gain over the no-steering baseline (Table 7). This is the load-bearing evidence for the generalised routing- failure claim in §5. C.3 CARD per-cell breakdown (∆ gap and ∆ acc ) For lever-holdera 1 , we report both the gap shift and the mean-accuracy shift for ev- ery (probe, question, VLM) cell.Across the full4 VLMs × 3 probes=12intent- question cells,9are exactly0.00on both metrics.The three nonzero cells: Gemma-4 under belief-direction steering (|∆ gap |=9.7p, |∆ acc |=4.9p);InternVL2.5underbelief- direction steering (|∆ gap |=8.3p,|∆ acc |=5.6p); and InternVL2.5 under knowledge-direction steering(|∆ gap |=25.0p,|∆ acc |=13.9p, direction-balanced flips that cancel in net ac- curacy). This rules out both belief-conditional modulation and uniform answer-flip across the matrix. C.4 Per-protagonist CARD breakdown Figure 6 breaks down the protagonist-mean of Fig- ure 3 into a4×3grid over (VLM, protagonist). The intent column ∆ is essentially 0 in every cell, con- firming that the routing failure is not protagonist- specific. C.5 CIs and Wilcoxon sign test on the intent null Across all108intent measurements (4 VLMs×3 protagonists×3 probes×3 seeds), steering im- proves intent accuracy by at most18.7p (95% paired-bootstrap upper bound), vs22.9p on the belief control. The intent answer remains invariant toαin69/108of these; the remainder skews to- ward the constant-prior default (27vs12, Wilcoxon p = 0.009) – steering pushes intent toward its de- fault, not toward belief-conditional behaviour. C.6 CARD 3×3 heatmaps per VLM For each VLM we show one panel per protagonist; each panel is a3×3grid in which rows are the direc- tion we steered along (belief / intent / knowledge) and columns are the question we then asked. The cell number is how much the TB−FB gap moves between α=+10 and α=−10, in p. A large cell value means the gap moved a lot – not that the model got more accurate: a TB-correct / FB-wrong pattern can flip to TB-wrong / FB- correct under steering, giving a large gap shift with mean accuracy unchanged. The intent column (red box) is the case we care about: even where it lights up, the companion accuracy-improvement matrix (Table 7) is essentially zero on every cell. The largest values sit on the diagonal (steer-belief→ ask-belief, etc.) – these are the expected positive controls, where steering an axis strongly moves its own answer head. Figures 7–10 show all four VLMs. C.7 T-A 192-cell heatmap For each (variant V0–V11, VLM, protag∈ overall, a1, a2, a3) we report TB accuracy, FB accuracy, and their gap (Figure 11). Readers should look for cells in which a lever-holder column ex- ceeds the pre-registeredTB > 50, FB > 50 threshold – no cell does. C.8 Projection-removal per-VLM sweep For each VLM and eachα∈0.5, 1, 2, 5σwe re- port the intent TB−FB gap on the unified partition (Figure 12). The flat curves – nearly horizontal VLMAxisLever a 1 Lever a 2 Trav. a 3 Qwen2-VL belief100/0/100/100100/50/100/100100/0/100/100 intent0/100/100/1000/100/100/10053/47/100/100 know.100/100/48/100100/69/72/10053/100/44/98 Gemma-4 belief100/0/100/10069/52/100/100100/4/100/100 intent0/100/100/1000/100/100/100100/0/100/100 know.100/100/59/100100/100/58/1000/100/44/100 LLaVA-NeXT belief100/0/57/52100/0/10/4253/47/100/100 intent0/100/100/1000/100/100/10053/47/100/100 know.89/51/48/52100/52/100/4253/47/44/56 InternVL2.5 belief84/0/95/10075/4/100/100100/0/94/98 intent0/100/100/1000/100/100/10075/23/98/100 know.99/89/0/9398/87/0/922/100/0/100 Table 6: Full D2 four-tuple discrimination matrix (success-TB / success-FB / counter-TB / counter-FB) for every (VLM, axis, protagonist). VLMProtag.Probe=beliefProbe=intentProbe=knowledge Qwen2-VL a 1 +0.0+0.0+0.0 a 2 +0.0+0.0+0.0 a 3 +4.0± 2.9+0.0+0.0 Gemma-4 a 1 +0.0+0.0+0.0 a 2 +0.0+0.0+0.0 a 3 +0.0+0.0+0.0 LLaVA-NeXT a 1 +0.0+0.0+0.0 a 2 +0.0+4.0± 5.7+0.0 a 3 +0.0+0.7± 0.5+0.0 InternVL2.5 a 1 +0.0+0.0+0.0 a 2 +0.0+0.0+0.0 a 3 +1.3± 0.5+1.3± 1.1+1.1± 0.8 Table 7: Best-αintent accuracy gain (p) over the no-steering baseline (α=0) for every (VLM, protagonist, probe direction) cell, reported as mean±std across three seeds (seeds42, 43, 44; the±is dropped when the std rounds to0.0). Most cells are exactly0on every seed; across the full multi-seed sweep,95/108cells are exactly0and the largest gain anywhere is+12.00%. Compare with belief and knowledge accuracies, which improve by+2.7to +10.6p on the across-protagonist mean under the same hooks (Figure 3): the intent prediction does not use the belief direction usefully on any cell. acrossαon every VLM – are the visual form of the A1 amplitude-invariance argument. C.9 Vendor architecture and rigidity ordering The four VLMs differ in text-decoder architecture and in their empirical resistance to prompt pertur- bation (Table 9); the more rigid decoders are the ones whose intent answers were hardest to move under any T-A prompt in our sweep. D Additional controls and diagnostics D.1 Off-diagonal cosine measurements Probe directions on the top-56(layer, head) cells are moderately aligned, not orthogonal. On the three steered axes (belief, intent, knowledge), pair- wise cosines fall in0.26−0.58across the four VLMs. The belief–intent cosine is0.55(Qwen2- VL),0.38(Gemma-4),0.57(LLaVA-NeXT), and 0.28 (InternVL2.5). This non-orthogonality is what makes the intent column’s null informative. Steering along belief is partially also steering along intent, yet the intent prediction does not move. If the two directions were nearly orthogonal, a null cross-axis effect on intent would be trivial; at the observed alignment, the null reflects that the intent prediction does not consume the shared subspace. The non-null sibling columns we observe in CARD (§5) confirm that the linear belief direction is causally usable by the belief and knowledge predictions, ruling out an “information-not-present” interpretation. D.2 CLEVR positive-control breakdown Table 10 reports the full per-(VLM, question, probe) accuracy-gain matrix for the CLEVR pos- itive control (§5). The sweep covers100scenar- ios per axis and three probe axes (count, shape, colour) on five VLMs: the four Relay Chain VLMProbeQuestion∆ gap (p) ∆ acc (p) Qwen2-VL beliefbelief+30.56 −15.28 beliefintent+0.00+0.00 beliefknowledge+22.22+11.11 intentbelief −4.17+2.08 intentintent+0.00+0.00 intentknowledge+0.00+0.00 knowledgebelief+6.94 −3.47 knowledgeintent+0.00+0.00 knowledge knowledge+0.00+0.00 Gemma-4 beliefbelief+66.67+0.00 beliefintent −9.72+4.86 beliefknowledge+59.72 −20.14 intentbelief+33.33 −16.67 intentintent+0.00+0.00 intentknowledge+22.22 −19.44 knowledgebelief+1.39 −0.69 knowledgeintent+0.00+0.00 knowledge knowledge+12.50+6.25 LLaVA-NeXT beliefbelief+20.83 −4.86 beliefintent+0.00+0.00 beliefknowledge −13.89+40.28 intentbelief+8.33 −13.89 intentintent+0.00+0.00 intentknowledge+9.72 −14.58 knowledgebelief −12.50+11.81 knowledgeintent+0.00+0.00 knowledge knowledge −23.61+20.14 InternVL2.5 beliefbelief −31.94 −20.14 beliefintent+8.33 −5.56 beliefknowledge+2.78 −26.39 intentbelief+0.00+0.00 intentintent+0.00+0.00 intentknowledge −2.78+2.78 knowledgebelief+31.94+20.14 knowledgeintent −25.00+13.89 knowledge knowledge −4.17+17.36 Table 8: CARD per-cell breakdown on lever-holdera 1 , reporting both the gap shift∆ gap = gap B (α=+10)− gap B (α=−10)and the mean-accuracy shift∆ acc =acc B (α=+10)−acc B (α=−10), whereacc B = 1 2 (TB B + FB B ). Bold rows are the intent-question column:9of the12cells are0.00on both metrics across the four VLMs (Qwen2-VL and LLaVA-NeXT all three probes; Gemma-4 and InternVL2.5 under intent and knowledge probes), and the remaining three (Gemma-4 and InternVL2.5 under belief steering, InternVL2.5 under knowledge steering) shift by at most|∆ gap |=25.0p /|∆ acc |=13.9p, ruling out both belief-conditional modulation (∆ gap ) and any uniform answer-flip (∆ acc ). Sibling columns (belief, knowledge) show non-trivial shifts under the same probe directions. VLMlayersheadshead_dimhiddenrigidity Qwen2-VL-7B28281283584hardest LLaVA-NeXT-7B32321284096middle Gemma-4-E4B3483202560softest InternVL2.5-8B32321284096intermediate Table 9: Text-decoder architecture specs and empirical rigidity ordering of the four VLMs. Rigidity = resistance of the intent constant prior to prompt perturbation in the T-A sweep: Qwen2-VL is hardest (no V0–V11 cell breaks 30/100ona 1 ); LLaVA-NeXT is intermediate; Gemma-4 is softest (V2 cost-explicit breaks the prior ona 2 to70/86, though not toward belief-conditional behaviour). The ordering matches loosely with head dimensionality (Gemma’s 320vs.128for the others), but the dataset (4vendors) is too small to draw architectural conclusions; a quantitative regression of rigidity against pretraining mix and RLHF intensity is left to future work. backbones plus SmolVLM-500M (Idefics-3 fam- ily). We added SmolVLM-500M explicitly because Qwen2-VL and InternVL2.5 sit at ceiling on shape and colour, and we wanted a backbone with abun- beliefintentknowledge 0 25 50 75 100 Qwen2-VL Mean acc (%) =+13.7 =0 =0 a 1 (lever-holder) no steering (=0) best ±10 beliefintentknowledge =0 =0 =+3.3 a 2 (lever-holder) beliefintentknowledge =0 =+4.0 =+6.2 a 3 (traveller) beliefintentknowledge 0 25 50 75 100 Gemma-4 Mean acc (%) =+16.2 =0 =+17.8 beliefintentknowledge =+11.1 =0 =+0.2 beliefintentknowledge =+4.2 =0 =+5.3 beliefintentknowledge 0 25 50 75 100 LLaVA-NeXT Mean acc (%) =+13.9 =0 =+22.7 beliefintentknowledge =+8.2 =+4.0 =+8.7 beliefintentknowledge =+2.7 =+0.7 =0 beliefintentknowledge Question axis 0 25 50 75 100 InternVL2.5 Mean acc (%) =+4.9=0 =+4.9 beliefintentknowledge Question axis =+6.0 =0 =+3.3 beliefintentknowledge Question axis =+0.2 =+1.8 =0 CARD baseline vs best- steered accuracy per (VLM, protagonist): intent column is null on every cell. Error bars: ±std across 3 seeds. Figure 6: Per-protagonist breakdown of Figure 3.4× 3grid: rows are VLMs, columns are protagonists (a 1 ,a 2 lever-holders;a 3 traveller). Each cell shows mean TB/FB accuracy atα=0(grey) vs the bestα ∈ −10, +10 across probe directions (VLM color);∆on top is best−baseline. The intent column∆is essentially0in every (VLM, protagonist) cell (12/12), while sibling answer heads (belief, knowledge) move on the protagonists that have headroom. The routing failure is therefore not protagonist-specific. dant headroom on every axis. CLEVR’s True/False variants share the same correct answer by construc- tion, so the gap-shift metric is uninformative here (∆ gap = 0in every cell). The accuracy-shift met- ric is the right summary: it shows that the steering hook moves accuracy by up to+60p when the model has headroom. D.3 Random-label probe control Following Hewitt and Liang (2019) and Borto- letto et al. (2024), we retrain logistic-regression probes on the same paired-statement activations with randomly permuted TB/FB labels. If the orig- inal probe is exploiting spurious correlations rather than capturing a structured belief representation, the permuted-label probe should also achieve high accuracy. Across all (VLM, axis) combinations (Ta- beliefintentknowledge Question axis Probe axis 30.1 ±2.9 0.0 ±0.0 15.7 ±5.1 4.2 ±0.0 0.0 ±0.0 2.3 ±3.3 2.3 ±3.3 0.0 ±0.0 0.5 ±0.7 protagonist a1 (lever-holder) beliefintentknowledge Question axis 23.1 ±14.2 32.4 ±7.9 10.7 ±5.8 0.0 ±0.0 0.9 ±1.3 10.2 ±4.1 0.0 ±0.0 0.0 ±0.0 2.7 ±1.1 protagonist a2 (lever-holder) beliefintentknowledge Question axis 29.8 ±27.0 8.9 ±7.2 12.0 ±1.1 0.0 ±0.0 0.0 ±0.0 8.0 ±6.6 0.0 ±0.0 0.0 ±0.0 3.6 ±3.3 protagonist a3 (traveller) 0 20 40 60 80 100 | gap| (p) Figure 7: Qwen2-VL-7B – full CARD heatmap per protagonist. Red box = intent column. beliefintentknowledge Question axis Probe axis 66.7 ±0.0 3.2 ±4.6 53.2 ±5.1 32.4 ±1.3 0.0 ±0.0 30.6 ±7.4 1.9 ±1.7 0.0 ±0.0 16.7 ±4.1 protagonist a1 (lever-holder) beliefintentknowledge Question axis 71.6 ±2.5 64.4 ±6.2 38.7 ±1.1 41.8 ±6.3 0.0 ±0.0 0.0 ±0.0 17.8 ±12.5 0.0 ±0.0 7.6 ±10.7 protagonist a2 (lever-holder) beliefintentknowledge Question axis 116.0 ±2.9 80.0 ±0.0 63.1 ±6.0 40.0 ±26.6 0.0 ±0.0 0.0 ±0.0 37.3 ±16.4 0.0 ±0.0 0.4 ±0.6 protagonist a3 (traveller) 0 20 40 60 80 100 | gap| (p) Figure 8: Gemma-4-E4B – full CARD heatmap per protagonist. Red box = intent column. beliefintentknowledge Question axis Probe axis 16.7 ±3.4 0.0 ±0.0 26.4 ±9.7 10.6 ±1.7 0.0 ±0.0 28.2 ±16.6 17.1 ±3.5 0.0 ±0.0 29.6 ±6.6 protagonist a1 (lever-holder) beliefintentknowledge Question axis 2.7 ±3.8 0.0 ±0.0 25.8 ±10.1 5.3 ±7.5 16.9 ±12.0 15.6 ±19.2 0.0 ±0.0 0.0 ±0.0 32.0 ±6.8 protagonist a2 (lever-holder) beliefintentknowledge Question axis 0.0 ±0.0 0.0 ±0.0 1.8 ±2.5 0.0 ±0.0 48.0 ±3.9 7.6 ±5.4 5.3 ±7.5 0.0 ±0.0 8.9 ±10.7 protagonist a3 (traveller) 0 20 40 60 80 100 | gap| (p) Figure 9: LLaVA-NeXT-Video-7B – full CARD heatmap per protagonist. Red box = intent column. ble 11), random-label probe accuracy is at chance (47.4–52.0%) while true-label probe accuracy is 91.6–100%, giving selectivity≥ 43.0p on every cell. The probes are not exploiting superficial struc- ture. D.4 Lexical-ablation control A linear probe trained on paired-statement activa- tions could in principle latch onto three asymme- tries that co-vary with belief: a lexical ignorance clause inserted into FB belief-true captions, a visual beliefintentknowledge Question axis Probe axis 21.3 ±13.1 7.4 ±4.6 2.3 ±0.7 2.8 ±3.0 0.5 ±0.7 2.3 ±0.7 20.8 ±12.0 11.6 ±10.3 1.9 ±1.7 protagonist a1 (lever-holder) beliefintentknowledge Question axis 16.9 ±7.6 1.3 ±1.1 6.7 ±4.4 2.7 ±2.9 0.9 ±0.6 0.9 ±0.6 24.4 ±14.8 2.7 ±2.9 5.8 ±3.3 protagonist a2 (lever-holder) beliefintentknowledge Question axis 0.9 ±1.3 9.3 ±1.1 3.1 ±2.3 0.0 ±0.0 3.1 ±1.7 0.4 ±0.6 2.2 ±1.3 5.8 ±8.2 2.7 ±1.1 protagonist a3 (traveller) 0 20 40 60 80 100 | gap| (p) Figure 10: InternVL2.5-8B – full CARD heatmap per protagonist. Red box = intent column. V0 V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11 Prompt variant a1a2a3a1a2a3a1a2a3a1a2a3 Qwen2-VLGemma-4LLaVA-NeXTInternVL2.5 -69-67-1-69-67+67-69-67-1-69-67+35 -69-67-1-65-16+67-69-67-1-69-67+67 -69-67+65-1+3-1-35-67-1-17-28+56 -69-66+0-69-67+26-35-41-1-69-67-1 -69-67-1-69-67+8-69-67-1-69-67-25 -69-67-1-69-67+67-60-13-1-69-67+0 -69-51-1+3+1+0-47-9-1-1+3+32 -69-65+5-69-67+5-68-14+3-69-67-59 -69-61+7-69-67+67+8-13-1-69-67+67 -69-67+67-1+3-1-29-22-1-4-4+66 -30-13-1-69-67-1-1+3-1-69-66+65 -47-12+9-3-14+24-1+3-1-69-45-6 T-A 144-cell prompt-resistance falsifier ( = TB > 50 AND FB > 50 on lever-holder) 100 75 50 25 0 25 50 75 100 TB FB gap (p) Figure 11: T-A prompt-resistance test. Each row is one of12prompt rewrites we tried (cost framing, chain-of- thought, perception-first, rule-based,1/2-shot examples. Each column is one (VLM, protagonist) cell. Cell colour encodes the TB−FB accuracy gap – warm = model is more accurate when the protagonist can see the event, cold = the reverse. A star marks cells that cleared our pre-registered bar for “genuinely belief-conditional” (lever-holder, TB and FB accuracy both above50%). Only5cells star out of192, and only one points the right way (TB>FB), with a margin an order of magnitude smaller than the same models’ gap on the canonical belief question – no prompt rewrite recovers belief-conditional action. contrast between protagonist-POV (fog-occluded) and oracle frames, and a distributional gap between those two view sets. We re-train probes on three controlled ablation arms that selectively remove these confounds: Arm A (symmetric clause). We add a counterfac- tual “able-to-see” clause to FB belief-false captions, eliminating the lexical asymmetry while leaving views unchanged. Arm B (oracle-only).We use the oracle view and the original caption for both belief-true and belief-false activations, removing both lexical and visual asymmetries so the probe must rely on the belief-true vs. belief-false text contrast alone. Arm C (clause artefact).We inject the ignorance clause into TB captions only (no real perspec- tive shift), measuring how much val accuracy VLMQuestionBaselineP=countP=shapeP=colourbest Qwen2-VL count86+6+2+6+6 shape100+0+0+0+0 (ceil) colour100+0+0+0+0 (ceil) Gemma-4 count20+26+60+42+60 shape30+24+58+52+58 colour52+16+26+22+26 LLaVA-NeXT count20+40+6+6+40 shape0+8+2+0+8 colour6+6+8+10+10 InternVL2.5 count86+0+14+10+14 shape100+0+0+0+0 (ceil) colour96+0+2+2+2 (ceil) SmolVLM-500M count12+6+10+26+26 shape70+10+2+12+12 colour64+0+0+10+10 Table 10: CARD positive control on CLEVR (single image,100scenarios per axis, seed43). “Baseline” = mean accuracy atα=0on the corresponding within-axis cell; each probe column = best-α∈±10accuracy gain. Bold = the largest non-zero∆ acc per (VLM, question). At least one cross-axis cell shows≥ +14p gain on every VLM that has headroom (Qwen and InternVL are at≥ 96%on shape/colour; Gemma and LLaVA are well below ceiling on every axis and steering produces+6to+60p gains). The steering hook is functional; the Relay Chain intent null in F3 is not a hook-broken artefact. VLMAxisTrue-label accRandom-label accSelectivity Qwen2-VL belief98.147.6+50.5 intent98.047.4+50.6 knowledge98.047.9+50.1 Gemma-4 belief100.050.7+49.3 intent100.052.0+48.0 knowledge100.050.5+49.5 LLaVA-NeXT belief91.648.6+43.0 intent91.648.0+43.6 knowledge91.747.6+44.1 Table 11: Random-label probe control (Hewitt and Liang, 2019). For each (VLM, axis), we retrain logistic- regression probes on the same paired activations with randomly permuted labels and report mean held-out accuracy across the sampled (layer, head) cells. True-label probes attain91.6–100%; random-label probes are at chance (47.4–52.0%), giving selectivity≥ 43.0p on every axis and every VLM. The top single-cell random-label accuracy across all sampled cells is 56.0%, well below the true-label minimum of 91.6%. 0.51.02.05.0 projection-removal ( ) 100 75 50 25 0 25 50 75 100 Intent TB FB gap (p) Qwen2-VL a1 (lever) a2 (lever) a3 (trav) 0.51.02.05.0 projection-removal ( ) Gemma-4 a1 (lever) a2 (lever) a3 (trav) 0.51.02.05.0 projection-removal ( ) LLaVA-NeXT a1 (lever) a2 (lever) a3 (trav) Projection-removal: intent gap invariant at every Figure 12: Projection-removalαsweep.x-axis:α ∈ 0.5, 1, 2, 5σremoval of the belief direction from the residual stream.y-axis: intent TB−FB gap. Three lines per VLM:a 1 ,a 2 ,a 3 . The intent gap is constant across α on every VLM and every protagonist. a probe can achieve from the clause alone. On Arm B the top-cell probe accuracy stays within ±2p of the canonical extraction across all VLMs, so the probe is not riding the lexical or view con- found. Arm C accuracy stays near chance, ruling out the clause-alone exploit. last token vision end vision+ last Steering target position 0 1 2 3 4 5 6 Max intent | acc | (p) CS-CARD-T (Qwen2-VL): intent invariant to steering position canon belief (last) canon intent (last) P7 cond (vision-end) Figure 13: CS-CARD-T on Qwen2-VL: intent|∆ acc | vs steering target position, one line per probe source. All nine (source, position) entries sit at≤ 0.3 p. D.5 Polarity-flip diagnostic We ran a separate diagnostic in which the ques- tion polarity is flipped (“should the protagonist not release?” instead of “release?”). An oracle rea- soner produces the same TB−FB gap shape under the flip; a constant prior collapses. We report all (VLM, protagonist) cells in Table 12; the contrast between Qwena 1 (preserves the100→100pat- tern – oracle signature) and LLaVAa 2 (collapses – constant-prior signature) is illustrative of the two regimes observed across the table. D.6 Scale ablation: Qwen2.5-VL-32B To check that the intent-column null is not spe- cific to the7–8B Relay Chain backbones, we ran the same CARD pipeline on Qwen2.5-VL-32B (≈ 32B parameters;64layers,40heads, head_dim 128): identical probe extraction, top-56steering pool, andα ∈ −10, 0, +10sweep across the three probes. Peak held-out probe accuracy is 98.2%(belief),95.9%(intent),97.5%(knowledge) – on par with or above the7–8B backbones. On lever-operatora 1 the intent answer is invariant to steering across all nine (probe,α) combinations: TB accuracy is30.6%and FB accuracy is100% at everyαon every probe direction – the same constant-prior signature observed at7–8B. The be- lief and knowledge questions move under the same hooks (e.g. belief-Q FB accuracy ona 3 shifts from 96.7%atα=−10to98.7%atα=+10on the belief probe). The routing failure does not vanish at4× parameter scale, even though the probe recovers the belief direction at ceiling. E Models, infrastructure, and computational budget Models and parameter counts. The four open- weight VLMs used in the main Relay Chain ex- periments span three vision-encoder lineages and three language-decoder lineages at the4–8B scale: Qwen2-VL-7B-Instruct (≈ 7.6B parameters; Yang et al., 2024a), Gemma-4-E4B-it (effective≈ 4B ac- tivated parameters in the ENB mixture-of-experts variant; Gemma Team, Google DeepMind, 2026), LLaVA-NeXT-Video-7B-hf (≈ 7.1B; Zhang et al., 2024), and InternVL2.5-8B (≈ 8.1B; Chen et al., 2024b). The CLEVR positive control addition- ally uses SmolVLM-500M (≈ 500M parameters, Idefics-3 family) Marafioti et al., 2025. We also use Qwen2.5VL-32B-Instruct (Yang et al., 2024b) for a model scale ablation study on CARD. Hardware. All experiments ran on NVIDIA H100 NVL GPUs (96GB HBM) on an internal SLURM-managed cluster. Each CARD, projection- removal, or T-A sweep occupies one GPU; multi- seed runs and the CLEVR chain were parallelised across up to four GPUs. Probe training is CPU- bound (logistic regression on saved activations). Compute budget. End-to-end cost for the full pipeline – activation extraction, probe training, the CARD3×3sweep, and all ablations across the four open-weight VLMs – is approximately25GPU- hours on a single H100 NVL. In practice we paral- lelised across two H100 GPUs, so wall-clock time was about half a day. Software. VLMinferenceuses transformersv5.5(HuggingFace)with bfloat16weights and eitherflash-attention-2 orsdpaas the attention backend.Logistic- regression probes are trained withscikit-learn (L 2 regularisation, default solver) on saved activations. Paired bootstrap CIs, the binomial sign test, and the Wilcoxon signed-rank test are computed withscipy.stats; the bootstrap RNG is fixed at seed 20260523 for reproducibility. E.1 Artifact licenses and intended use All four open-weight VLMs we evaluate are pub- licly released: Qwen2-VL-7B under Apache 2.0, Gemma-4-E4B-it under the Gemma License Agree- ment, LLaVA-NeXT-Video-7B under Apache 2.0, and InternVL2.5-8B under MIT. CLEVR (Johnson et al., 2017), used as the positive-control auxil- iary dataset, is distributed under C BY 4.0. Grid- ToM (Li et al., 2025), cited as the grid-world ToM format inspiration, is released by its authors for academic use. We use all models and datasets for non-commercial academic research, consistent with their intended use. Our CARD code and the Relay Chain dataset will be released under MIT (code) and C BY 4.0 (data) upon acceptance. VLMoverall orig (p)overall flip (p)sign-preserved? Qwen2-VL+86.5+82.4yes (oracle) Gemma-4+73.0+85.1yes (oracle) LLaVA-NeXT+63.5+35.1weakened (drift) Table 12: D1 polarity-flip diagnostic. Original and flipped question polarity TB−FB gap on the canonical148 hold-out. An oracle-reasoning model preserves the sign and magnitude under flip; a constant-prior model collapses to a smaller or sign-flipped gap.

Instant research discovery