Paper deep dive
SpatialChain: A Benchmark for Auditing Spatial Reasoning Faithfulness in VLMs
Rafael Teixeira Sousa, Vinícius Paulo Lopes de Oliveira, Elisa Ayumi Masasi de Oliveira, Luiza Martins de Freitas Cintra, Fernanda Bufon Färber, Igor Dias Aguiar, Julia Yasmim de Almeida Nobre, Arlindo Rodrigues Galvão Filho
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Thinking-enabled vision-language models (VLMs) report ever-higher accuracy on spatial benchmarks, yet final-answer scores cannot reveal whether a correct prediction reflects faithful spatial reasoning or a linguistic shortcut. We introduce SpatialChain, a dataset of 28,350 training and 899 test examples pairing spatially-oriented GQA questions with scene-graph-grounded reasoning chains, retained only when the generated answer matches the symbolic ground truth, and a two-axis evaluation combining objective chain-overlap metrics with a scene-graph-aware LLM judge that scores faithfulness and completeness independently of the final answer. Applied to nine thinking-enabled VLMs, the protocol surfaces three findings invisible to standard accuracy: (i) four of nine models achieve $\geq$79% VQA accuracy while exhibiting shortcut rates above 39%, i.e., correct answers whose reasoning the judge marks as unfaithful; (ii) chain quality significantly predicts answer correctness for seven of nine models, but the two exceptions (Claude Sonnet 4.6, InternVL3.5-8B) reveal qualitatively distinct failure modes, terse output vs. verbose-decorative reasoning, that benchmark accuracy alone conflates; (iii) SFT on SpatialChain improves Qwen3-VL-8B by +6.2 pp in-domain and reduces its shortcut rate to 22%, while a stylistic specialization effect on external benchmarks motivates replay-augmented training as mitigation. The faithfulness judge is validated against 198 human-annotated items, where judge-human agreement matches human-human agreement, and against a second judge from a different provider, which preserves the model ranking ($\rho$ = 0.88). Data, generation scripts, and evaluation code are released at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2610.06413v1
- Canonical: https://arxiv.org/abs/2610.06413v1
Trouble viewing inline? Open PDF directly →
Full Text
84,338 characters extracted from source content.
Expand or collapse full text
SpatialChain: A Benchmark for Auditing Spatial Reasoning Faithfulness in VLMs Rafael Teixeira Sousa Affiliation: Universidade Federal de Mato Grosso (UFMT) Affiliation: Universidade Federal de Goiás (UFG) Affiliation: Advanced Knowledge Center for Immersive Technologies (AKCIT) Email: rafael.sousa@ufmt.br Vinícius Paulo Lopes de Oliveira Affiliation: Universidade Federal de Goiás (UFG) Affiliation: Advanced Knowledge Center for Immersive Technologies (AKCIT) Elisa Ayumi Masasi de Oliveira Affiliation: Universidade Federal de Goiás (UFG) Affiliation: Advanced Knowledge Center for Immersive Technologies (AKCIT) Luiza Martins de Freitas Cintra Affiliation: Universidade Federal de Goiás (UFG) Affiliation: Advanced Knowledge Center for Immersive Technologies (AKCIT) Fernanda Bufon Färber Affiliation: Universidade Federal de Goiás (UFG) Affiliation: Advanced Knowledge Center for Immersive Technologies (AKCIT) Igor Dias Aguiar Affiliation: Universidade Federal de Goiás (UFG) Affiliation: Advanced Knowledge Center for Immersive Technologies (AKCIT) Julia Yasmim de Almeida Nobre Affiliation: Universidade Federal de Goiás (UFG) Affiliation: Advanced Knowledge Center for Immersive Technologies (AKCIT) Arlindo Rodrigues Galvão Filho Affiliation: Universidade Federal de Goiás (UFG) Affiliation: Advanced Knowledge Center for Immersive Technologies (AKCIT) Abstract Thinking-enabled vision-language models (VLMs) report ever-higher accuracy on spatial benchmarks, yet final-answer scores cannot reveal whether a correct prediction reflects faithful spatial reasoning or a linguistic shortcut. We introduce SpatialChain, a dataset of 28,350 training and 899 test examples pairing spatially-oriented GQA questions with scene-graph-grounded reasoning chains — retained only when the generated answer matches the symbolic ground truth — and a two-axis evaluation combining objective chain-overlap metrics with a scene-graph-aware LLM judge that scores faithfulness and completeness independently of the final answer. Applied to nine thinking-enabled VLMs, the protocol surfaces three findings invisible to standard accuracy: (i) four of nine models achieve ≥79%≥\!79\% VQA accuracy while exhibiting shortcut rates above 39%39\% — correct answers whose reasoning the judge marks as unfaithful; (i) chain quality significantly predicts answer correctness for seven of nine models, but the two exceptions (Claude Sonnet 4.6, InternVL3.5-8B) reveal qualitatively distinct failure modes — terse output vs. verbose-decorative reasoning — that benchmark accuracy alone conflates; (i) SFT on SpatialChain improves Qwen3-VL-8B by +6.2+6.2 p in-domain and reduces its shortcut rate to 22%22\%, while a possible stylistic specialization effect on external benchmarks motivates replay-augmented training as mitigation. The faithfulness judge is validated against 198 human-annotated items, where judge–human agreement matches human–human agreement, and against a second judge from a different provider, which preserves the model ranking (ρ=0.88ρ=0.88). Data, generation scripts, and evaluation code are released at github.com/spatialchain/SpatialChainBenchmark. 1 Introduction Vision-language models (VLMs) have achieved remarkable performance on visual question answering (VQA) and spatial understanding benchmarks [40, 23]. More recently, several model families have been augmented with explicit multimodal reasoning, including Qwen3-VL [3], InternVL3 [37], GLM-4.1-9B [9], Claude 4.6 [2], and GPT-5.4 [26]. These thinking-enabled models drive substantial gains on challenging benchmarks [41, 36, 7], yet high accuracy alone does not establish that a model is reasoning rather than exploiting statistical language priors. A model that has memorized common object co-occurrences (e.g., cup on table, person on chair) may answer spatial queries correctly without performing any actual spatial inference. Standard benchmarks provide no mechanism to distinguish these two regimes: only the final answer is evaluated, leaving the underlying reasoning process entirely unaudited [8, 1, 25, 6]. To address this, we present SpatialChain, which attaches an explicit, graph-grounded reasoning chain to each spatially-oriented question drawn from GQA [11]. Every chain is synthesized from GQA’s structured scene graphs via Claude Haiku 4.5 with extended thinking, retained only when the generated answer matches the ground-truth answer, and stored in a format compatible with thinking-mode VLMs. Because each reasoning step derives from a verifiable symbolic annotation, SpatialChain enables a new evaluation regime: beyond did the model answer correctly, we can ask did its reasoning trace engage with the correct spatial relations — and detect shortcut behavior even when the final answer is right. Contributions. • We introduce SpatialChain, a dataset of 28,350 training and 899 test examples pairing spatially-oriented GQA questions with scene-graph-grounded reasoning chains, providing the first structured reference against which the relational content of a VLM’s thinking trace can be audited. • We propose a two-axis evaluation protocol combining objective chain-overlap metrics with a scene-graph-aware LLM judge that scores reasoning faithfulness and completeness independently of the final answer, and show across nine thinking-enabled VLMs that chain quality significantly predicts answer correctness for seven of nine models — while exposing shortcut rates above 39% in four models that achieve ≥79%≥ 79\% VQA accuracy. We validate the judge against human annotators, a second judge from a different provider, and a sweep of the faithfulness threshold (Appendix L). • We show that SFT on SpatialChain improves Qwen3-VL-8B by +6.2+6.2 p in-domain and reduces its shortcut rate to 22% — the lowest of all evaluated models — while characterizing a stylistic specialization effect on out-of-distribution spatial benchmarks and proposing replay-augmented training as mitigation. Data, generation scripts, and evaluation code are released at github.com/spatialchain/SpatialChainBenchmark. 2 Related Work Thinking-enabled VLMs have driven substantial gains on general VQA suites [40, 41] and on dedicated spatial benchmarks such as EmbSpatial-Bench [7], ERQA [32], VSR [20], and VSI-Bench [39]; SpatialRGPT [5] further improves spatial perception by grounding VLMs in 3D scene graphs, while recent probes expose persistent failures on geometric consistency and subtle spatial comparisons [14, 15]. All of these benchmarks share the same structural limitation: they score only the final answer, providing no mechanism to determine whether a correct prediction reflects genuine spatial inference or a linguistic shortcut [8, 1]. GQA [11] exposes relational structure through scene graphs that could in principle support process-level evaluation, but its standard protocol is equally answer-only. Inspired by chain-of-thought prompting [38], several efforts expose the reasoning process alongside the final answer. ScienceQA [24] and VCR [42] provide human rationales in specialised domains, with free-form text that lacks structured grounding to image content. A more targeted line associates reasoning steps with image regions — Visual-CoT [30], CogCoM [27], DualFocus [4], Chain-of-Spot [22], and, at scale, VisReason [18] — and EmbodiedVSR [45] uses dynamic scene graphs to guide chain-of-thought on spatial tasks. Closer to our evaluation goal, SpatialEval [35] scores final answers on spatial tasks without auditing the trace, and MME-CoT [12] evaluates reasoning quality against free-form or human-written rationales without structured grounding to image content. These resources advance visual grounding — where to look — but none provides a verifiable reference for the relational content of a chain: a model that crops the correct region but reasons about the wrong spatial relation remains indistinguishable from one that genuinely understands the scene. SpatialChain fills this gap by deriving every reasoning step from GQA’s structured scene graphs and retaining chains only when consistent with the symbolic ground truth, enabling an evaluation that asks not only did the model look at the right region but did it reason about the correct spatial relations. 3 The SpatialChain Dataset 3.1 Source Data: GQA GQA [11] provides 113,018 images from Visual Genome [16] with structured scene graphs encoding objects, attributes, and pairwise spatial relations (e.g., left of, behind, above), alongside question–answer pairs annotated with a rich type taxonomy. We select exclusively questions that require resolving at least one spatial relation, following a two-stage pipeline. 3.2 Question Selection Pipeline Stage 1 — Type-based filtering. We retain questions whose GQA type falls into one of 17 relation-centric categories that structurally require spatial or relational reasoning (full list in Appendix A). Applied to the GQA balanced training split, this yields 135,061 candidates from 49,503 unique images. For the test split, we additionally rank candidates by the length of their GQA semanticStr field — a proxy for relational chain complexity — and retain only the 1,000 longest, biasing the test set toward compositional, multi-step spatial queries. Stage 2 — LLM spatial verification. Type-based filtering is necessary but not sufficient: many relation-type questions can be answered by object recognition alone (e.g., “Is the dog sitting?” uses a relVerify template but requires no spatial inference). We therefore apply a second filter using GPT-5-nano as a binary classifier, distinguishing questions that require relative position, proximity, containment, or directional reasoning (yes) from those answerable by attribute recognition or existence detection alone (no). The prompt is reproduced in Appendix B. After both stages we obtain 86,773 candidate training and 900 test candidate examples, each paired with the corresponding scene-graph subgraph. Due to the computational cost of extended-thinking generation, we randomly sample 30,000 candidates for reasoning chain generation. The remaining 56,773 filtered candidates carry no annotations beyond the GQA-derived selection and are not used in any experiment reported here; they may be released in a future version of SpatialChain. 3.3 Reasoning Chain Generation Chains are generated with Claude Haiku 4.5 (extended thinking, budget 3,000 tokens, max output 4,096 tokens) via the Anthropic Message Batches API. Each request provides the full scene-graph JSON as internal context alongside the question; a system prompt instructs the model to reason as if directly observing the image, prohibiting any explicit reference to scene-graph fields in its output. The generated chain must therefore use natural visual language, producing traces indistinguishable from those a human annotator would write when looking at the image. The full prompt is in Appendix C. Answer-consistent filtering and verifiable reference. The model’s extended thinking trace is retained as the reasoning chain only when its visible answer matches the GQA ground-truth answer; chains accompanying incorrect answers are discarded. A small number of additional examples are lost to generation errors (malformed output or truncated traces), bringing the final counts to 28,350 training and 899 test examples. Every retained chain therefore concludes correctly and was generated conditioned on the scene graph, which makes it cross-checkable against the symbolic annotations — the property that distinguishes SpatialChain from datasets where chains are generated freely from image content without a verifiable ground-truth structure [18, 30], and that underlies the objective reasoning metrics in §4.2. The filter verifies the conclusion, not each intermediate step: a chain could reach the right answer through a locally unsupported step (see §7). Representative examples, contrasting single-hop questions (one spatial relation) with multi-hop questions that chain two relations across anchor objects, are shown in Appendix E. Training target format. Retained chains are stored as <think> [trace] </think> <answer> [answer] </answer>, compatible with thinking-mode VLMs at both training and inference time, allowing the reasoning process to be audited independently of the final answer. 3.4 Dataset Statistics SpatialChain comprises 28,350 training and 899 test question–reasoning–answer triplets drawn from 19,263 and 719 unique images respectively (Table 1). The relation distribution is dominated by projective relations: left_of/right_of account for 62.7 % of training questions under our first-occurrence labelling scheme, where each question is tagged according to the first spatial relation appearing in its string. Thus, these statistics indicate the entry point of the spatial chain rather than its complete relational content, since many questions combine multiple relations, especially in the test split. In the test split this dominance follows from the complexity-biased selection (§3.2) — GQA’s longest chains are built from projective relations — so per-relation cells with n<20n<20 are non-inferential. Training chains have a mean length of 115 words (median 112), and test chains are slightly longer (mean 145 words), reflecting the complexity-biased test selection. Full relation and chain-length breakdowns are in Appendix D. Table 1: SpatialChain splits. Yes / No / Attr. partitions answers between binary spatial verification and attribute-typed answers. Avg. chain is the mean reasoning trace length in words. Split Source Samples Images Yes / No / Attr. Yes-rate Avg. chain (w) Train GQA train 28,350 19,263 16,516 / 9,926 / 1,908 0.58 115 Test GQA val 899 719 566 / 329 / 4 0.63 145 4 Evaluation Framework We evaluate models along two complementary axes: downstream accuracy on the VQA task, and reasoning quality assessed through objective corpus metrics and an LLM-based judge. The former measures whether the model produces the correct answer; the latter examines whether it does so for the right reasons. 4.1 Downstream VQA Accuracy We frame the task as binary spatial VQA: each question has a Yes/No ground-truth answer derived from GQA’s scene graphs [11]. Accuracy is computed as exact-match after lower-casing and stripping articles, consistent with the standard GQA evaluation protocol. We additionally report accuracy conditioned on answer polarity (Yes-acc and No-acc) to diagnose positive-answer bias, a known marker of language-prior exploitation in VQA models [8, 1]. 4.2 Objective Reasoning Metrics We compare each model’s thinking trace against the ground-truth scene-graph reasoning chain using ROUGE-1 and ROUGE-L [19], token-level F1 [29] (after lower-casing and stop-word removal), entity overlap (fraction of scene-graph object labels present in the predicted chain), and length ratio (predicted-to-reference chain length in tokens, a proxy for verbosity). All metrics are computed on all test examples regardless of answer correctness, and stratified by correctness to test whether higher-quality chains co-occur with correct answers. These metrics measure surface agreement only: multiple valid traces can realise the same relational path (e.g., boy vs. man for one node), and overlap with a Claude-Haiku-generated reference rewards stylistic similarity, which fine-tuned models inherit. No faithfulness conclusion in this paper rests on them. 4.3 LLM-as-Judge Reasoning Evaluation Objective metrics capture only surface-level agreement with the reference output and are insufficient to assess whether a candidate rationale is semantically grounded in the scene representation or whether it covers all inference steps required to justify the answer. We therefore adopt a rubric-based LLM-as-a-judge protocol, following prior work showing that strong LLM judges can approximate human evaluation when guided by explicit criteria, while remaining sensitive to prompt design and bias [46]. In particular, our prompt is inspired by rubric-based evaluation frameworks such as G-Eval, which uses structured prompting and chain-of-thought to score outputs under custom criteria [21]. The judge receives the question, ground-truth answer, scene graph, candidate final answer, and candidate reasoning, and returns three normalized scores in [0,1][0,1], computed by Claude Sonnet 4.6 [2]: answer correctness (AC), reasoning faithfulness (RF), and reasoning completeness (RC). Because the judge scores against the symbolic scene graph rather than the reference text, it is invariant to the surface variation that penalises overlap metrics. We validate it against human annotators and against a second judge from a different provider (GPT-5.6) in Appendix L. Separating faithfulness from correctness is important because chain-of-thought explanations may be plausible yet unfaithful to the model’s actual inference process [34]. Likewise, completeness captures whether the rationale accounts for all key spatial constraints and relations needed to support the final answer. Since scene graphs provide a structured representation of entities and relations, they serve as a natural evidence source for this judgment procedure and as the basis for the ground-truth reasoning chains themselves[13, 33, 45]. 5 Experiments 5.1 Models We evaluate seven systems spanning open-weight and proprietary VLMs with thinking capability. Open-weight models are evaluated in thinking mode via vLLM [17] with continuous batching on a single A100 GPU (80 GB): Qwen3-VL-8B and Qwen3-VL-4B [3], InternVL3.5-8B and InternVL3.5-4B [37], and GLM-4.1-9B [9]. Proprietary models are queried through their respective APIs: Claude Sonnet 4.6 [2] and GPT-5.4 [26]. We additionally report two fine-tuned variants: Qwen3-VL-8B FT and Qwen3-VL-4B FT, described below. Fine-tuning. We apply LoRA [10] (r=16r=16, α=32α=32, RSLoRA) to all linear layers of Qwen3-VL-8B and Qwen3-VL-4B, training for 2 epochs with an effective batch size of 12 on a single A100 80 GB GPU. The training target includes the full scene-graph reasoning chain followed by the binary answer, formatted as <think> ⋯·s </think> <answer> ⋯·s </answer>. Full hyperparameters are in Appendix G. 5.2 Inference Setup Open-weight models are served with vLLM (τ=0.6τ=0.6, top-p=0.95p=0.95, top-k=20k=20, max 16,384 tokens). Fine-tuned checkpoints are evaluated after merging the LoRA adapter into base weights. Proprietary models use default temperature and receive identical prompts; InternVL3.5 requires an additional prefix to activate thinking mode. All models share the same system prompt (Appendix H). Full inference scripts are in the code release.11 1 https://github.com/spatialchain/SpatialChainBenchmark 6 Results We evaluate nine systems on the SpatialChain test split: the base and fine-tuned variants of Qwen3-VL at 4B and 8B, two open-source baselines (InternVL3.5-4B/8B, GLM-4.1-9B), and two proprietary thinking-mode models (Claude Sonnet 4.6, GPT-5.4). All models receive the same system prompt and produce a reasoning chain followed by a binary Yes/No answer. Four of the 899 test examples have non-binary ground-truth answers (Attr. in Table 1) and are excluded from accuracy only (n=895n=895); chain-quality and judge metrics use all 899. Unparsed predictions are excluded per model from that model’s accuracy (Table 2, † ) and do not affect these counts. 6.1 Main Results Table 2 reports VQA accuracy alongside chain-quality metrics measured against the ground-truth scene-graph chains. Table 2: Model comparison on the SpatialChain test split (n=895n=895; see §6). Acc and F1 are overall accuracy and macro-F1 on the binary Yes/No task. Yes/No are accuracies conditional on ground-truth polarity. R-1, R-L, F1tok are reasoning-chain overlap metrics against the scene-graph reference chain. Lenratio_ratio is predicted-to-reference chain length (reference average: 145 tokens). † Unparsed predictions excluded from accuracy (InternVL3.5-4B: 25; InternVL3.5-8B: 16; Claude Sonnet 4.6: 7). Model Acc F1 Yes No R-1 R-L F1tok Lenratio_ratio Qwen3-VL-8B FT 85.03 88.68 92.76 71.73 0.654 0.443 0.643 0.90 Qwen3-VL-4B FT 82.23 86.67 91.34 66.57 0.657 0.450 0.646 0.89 GLM-4.1-9B 81.34 84.49 80.39 82.98 0.438 0.301 0.421 1.47 InternVL3.5-4B† 81.34 87.43 87.28 71.12 0.314 0.227 0.305 3.01 Claude Sonnet 4.6† 79.89 85.15 87.63 66.57 0.359 0.258 0.354 0.36 InternVL3.5-8B† 79.33 85.29 86.57 66.87 0.323 0.228 0.316 7.07 Qwen3-VL-8B 78.88 82.83 80.57 75.99 0.425 0.293 0.412 4.06 Qwen3-VL-4B 78.44 82.01 77.74 79.64 0.403 0.279 0.392 5.89 GPT-5.4 72.18 78.37 79.68 59.27 0.296 0.225 0.294 0.25 Fine-tuning yields the largest gains, but the field is tight. Qwen3-VL-8B FT leads with 85.0%85.0\% accuracy, a +6.2+6.2 p gain over its base model. GLM-4.1-9B and InternVL3.5-4B tie as the strongest zero-shot systems at 81.3%81.3\%, with Claude Sonnet 4.6 close behind at 79.9%79.9\%. GPT-5.4 is the weakest model overall (72.2%72.2\%), trailing the 4B Qwen3 base by 6 p despite being substantially larger. Chain-quality metrics partition models into three regimes. Fine-tuned models produce chains closely matching the reference in both length (ratio≈0.9ratio≈ 0.9) and content (R-1 >0.65>0.65). Open-weight base models (Qwen3, InternVL3.5) generate chains 33–7×7× longer than the reference with moderate overlap. Proprietary models (Claude Sonnet 4.6, GPT-5.4) are at the opposite extreme: chains under 0.4×0.4× reference length, effectively committing to a final answer with minimal explicit reasoning. GLM-4.1-9B sits between the two regimes, generating chains of near-reference length (ratio=1.47ratio=1.47) with the highest R-1 among non-fine-tuned models. Reasoning quality and answer accuracy are not aligned across model families. The relationship between R-1 and accuracy is monotonic only within the Qwen3 family. Across families, GPT-5.4’s R-1 of 0.300.30 corresponds to 72%72\% accuracy, while InternVL3.5-4B’s R-1 of 0.310.31 corresponds to 81%81\%. Claude Sonnet 4.6 and InternVL3.5-8B sit at similar R-1 levels (≈0.34≈ 0.34) yet differ in accuracy by less than 1 p despite a 20×20× difference in chain length. Formally, across all nine models accuracy and R-1 correlate only moderately (Pearson r=+0.69r=+0.69, p=0.04p=0.04; Spearman ρ=+0.64ρ=+0.64, p=0.06p=0.06), and the association collapses once the two fine-tuned variants are removed (r=+0.45r=+0.45, p=0.32p=0.32; Table 17). This decoupling motivates the within-model analysis in §6.3. 6.2 Yes/No Asymmetry as a Signature of Language Priors Table 2 reveals a striking polarity gap Δpol=Yes-acc−No-acc _pol=Yes-acc-No-acc. The pattern is not driven by model scale or family alone: it separates models that confirm spatial descriptions regardless of visual evidence (yes-biased) from models whose reasoning admits mismatch between question and image (balanced). We interpret a large Δpol _pol as a signature of language-prior exploitation: binary questions of the form “Is there X…” are known to carry strong yes-biases in natural image corpora [44, 8, 1], and a model defaulting to confirmation regardless of visual evidence would exhibit exactly this pattern. 6.3 Does Reasoning Quality Predict Answer Correctness? The averages in Table 2 do not tell us whether higher-quality chains are produced when the model is right — the central question SpatialChain is designed to answer. For each model, we partition the test set into correct and incorrect predictions and compare R-1 and token-level F1 using a two-sided Mann-Whitney U test (Table 3): a significant positive Δ indicates that chain quality carries information about spatial correctness beyond what the final answer alone reveals. Table 3: Reasoning-chain quality conditioned on answer correctness. Δ -1 is the mean R-1 difference between correct and incorrect predictions. ∗ p<0.05p<0.05; ∗ p<0.01p<0.01. Model R-1✓ R-1× Δ -1 F1✓ F1× p Qwen3-VL-4B 0.418 0.351 +0.067 0.406 0.342 <0.001∗∗<\!0.001^** Qwen3-VL-8B 0.439 0.373 +0.066 0.426 0.362 <0.001∗∗<\!0.001^** GLM-4.1-9B 0.447 0.399 +0.048 0.429 0.385 <0.001∗∗<\!0.001^** Qwen3-VL-4B FT 0.664 0.625 +0.039 0.653 0.612 <0.001∗∗<\!0.001^** Qwen3-VL-8B FT 0.660 0.626 +0.034 0.649 0.612 <0.001∗∗<\!0.001^** InternVL3.5-4B 0.320 0.288 +0.031 0.311 0.282 0.003∗∗0.003^** GPT-5.4 0.297 0.292 +0.004 0.295 0.290 0.027∗0.027^* Claude Sonnet 4.6 0.361 0.354 +0.007 0.356 0.347 0.0710.071 InternVL3.5-8B 0.323 0.322 +0.001 0.317 0.316 0.4580.458 Seven of nine models show a significant positive correlation (p<0.05p<0.05): all Qwen3 variants, GLM-4.1-9B, InternVL3.5-4B, and GPT-5.4. The effect is strongest in the Qwen3 base models (ΔR-1≈+0.067 -1≈+0.067), moderate in the fine-tuned models (≈+0.035≈+0.035 — compressed by uniformly higher chain quality in both groups), and weak but detectable in GPT-5.4 (+0.004+0.004, p=0.03p=0.03). The two exceptions expose qualitatively distinct failure modes. Claude Sonnet 4.6’s non-significance (p=0.07p=0.07, Δ=+0.007 =+0.007) stems from extremely short output (Lenratio=0.36Len_ratio=0.36): there is too little reasoning content for quality to vary meaningfully across the two groups. InternVL3.5-8B is the more revealing case: the longest chains in our evaluation (Lenratio=7.07Len_ratio=7.07) yet zero correlation with correctness (Δ=+0.001 =+0.001, p=0.46p=0.46). This is verbose output that resembles reasoning but whose content is decoupled from the spatial judgment being made — a failure mode chain length alone cannot detect. Taken together, the table illustrates SpatialChain’s central diagnostic value: GPT-5.4 (72%72\% accuracy, significant correlation) and Claude Sonnet 4.6 (80%80\% accuracy, non-significant) produce equally compact chains yet differ fundamentally in whether their output carries information about correctness — a distinction that benchmark accuracy cannot surface. This is consistent with evidence that intermediate tokens need not carry the reasoning behind a correct answer [31]; SpatialChain says which systems are in that regime. Stratifying by hop count (Table 18) shows multi-hop questions are not harder: the dominant difficulty is perceiving individual relations, not composing them. 6.4 Cross-Benchmark Behaviour After Fine-Tuning SFT on SpatialChain improves Qwen3-VL-4B in-domain (0.784→0.8220.784→ 0.822, +3.8+3.8 p) but degrades it on both external spatial benchmarks: ERQA 0.453→0.3800.453→ 0.380 (−7.3-7.3 p) and EmbSpatial-Bench 0.791→0.7570.791→ 0.757 (−3.4-3.4 p). Inspection of outputs reveals the mechanism: after fine-tuning, 100%100\% of outputs adopt the fixed “Looking at…” opener (vs. 0%0\% before), and average chain length collapses to 15%15\% of its pre-SFT value on ERQA. The model has learned to produce SpatialChain-style chains and applies this format even when the input distribution differs — stylistic specialization rather than loss of underlying spatial knowledge, as evidenced by the non-monotonic trajectory across checkpoints and by the preservation of in-plane spatial relations on EmbSpatial-Bench. Per-checkpoint curves and per-category breakdowns are reported in Appendix I. 6.5 LLM-as-Judge Evaluation Objective metrics measure surface overlap with the reference chain but cannot assess semantic faithfulness. We run the scene-graph-aware judge on all nine models (Table 4 and Figure 1) and additionally report the shortcut rate: the fraction of correct test items whose chain the judge marks as low-faithfulness (≤0.5RF≤ 0.5), computed as in the released code. The inclusive cutoff is deliberate: a score of exactly 0.50.5 marks a right conclusion that is weakly grounded (typically a chain that restates the question), which human annotators also score low (Appendix L). Table 4: LLM-as-judge scores, sorted by pass rate. Faith. and Compl. are mean faithfulness and completeness in [0,1][0,1]. Pass requires ≥0.5AC≥ 0.5 and (+)/2≥0.5(RF+RC)/2≥ 0.5. Shortcut is the fraction of correct test items with ≤0.5RF≤ 0.5. Model Acc Faith. Compl. Pass Shortcut Qwen3-VL-8B FT 85.03 0.630 0.700 0.820 0.222 Qwen3-VL-4B FT 82.23 0.631 0.708 0.802 0.194 GLM-4.1-9B 81.34 0.527 0.583 0.809 0.453 Claude Sonnet 4.6 79.89 0.541 0.582 0.794 0.390 Qwen3-VL-4B 78.44 0.585 0.658 0.776 0.264 Qwen3-VL-8B 78.88 0.571 0.658 0.765 0.260 GPT-5.4 72.18 0.542 0.568 0.721 0.246 InternVL3.5-8B 79.33 0.443 0.488 0.623 0.473 InternVL3.5-4B 81.34 0.346 0.360 0.461 0.580 Fine-tuned models produce the most faithful reasoning. Both Qwen3-VL FT variants lead on faithfulness (≈0.63≈ 0.63) and completeness (≈0.70≈ 0.70), with the lowest shortcut rates (19–22%), corroborating the objective metrics: SFT on scene-graph chains teaches grounded spatial reasoning, not just answer matching. Figure 1: Pass rate vs. shortcut rate. The bottom-right region (ideal) marks high passing with low shortcut behaviour; the top-left (shortcut-heavy) marks competitive accuracy achieved with frequent unfaithful reasoning. Fine-tuned Qwen3-VL variants are closest to ideal; InternVL3.5-4B is the clearest shortcut case. High accuracy and high shortcut rate coexist. Four models achieve ≥79%≥ 79\% accuracy while exhibiting shortcut rates above 39%39\%. Claude Sonnet 4.6 (79.9%79.9\%, shortcut 39%39\%) and InternVL3.5-4B (81.3%81.3\%, 58%58\%) are the clearest cases: competitive benchmark scores mask reasoning that the judge flags as largely ungrounded. This finding directly extends the correlation analysis in §6.3: chain quality is not only weakly correlated with correctness for these models, but the judge confirms that correct answers are frequently reached without faithful spatial inference. Verbosity does not imply faithfulness. InternVL3.5-8B generates chains 7×7× the reference length yet achieves the second-lowest pass rate (62.3%62.3\%) and a 47%47\% shortcut rate. The judge confirms quantitatively what was identified statistically before: long output that is decoratively verbose rather than spatially grounded. High pass rate and high shortcut rate are not contradictory. GLM-4.1-9B achieves the third-highest pass rate (80.9%80.9\%) alongside a 45%45\% shortcut rate. This follows mechanically from the pass criterion averaging RF and RC: a model can satisfy (+)/2≥0.5(RF+RC)/2≥ 0.5 while many individual chains fall below the faithfulness cutoff. The two metrics answer different questions and should be read together, not reconciled. The faithfulness signal is robust to the judge and the threshold. A second judge from a different provider (GPT-5.6, identical prompt) preserves the ranking of models by shortcut rate (ρ=0.88ρ=0.88; faithfulness scores ρ=0.74ρ=0.74, pass/fail κ=0.84κ=0.84), being uniformly stricter without changing the comparative conclusions. Shortcut rates are identical for τ∈[0.5,0.6]τ∈[0.5,0.6], and the ordering holds wherever the metric separates models. On 198 human-annotated items, judge–human agreement on faithfulness (ρ=0.49ρ=0.49) matches human–human agreement (ρ=0.44ρ=0.44). Details in Appendix L. 6.6 Qualitative Analysis Figure 7 (Appendix J) illustrates the two most contrasting reasoning regimes on a representative test example. The fine-tuned model traces the spatial chain step-by-step, closely mirroring the scene-graph reference. InternVL3.5-8B produces the correct answer, but buries it in hallucinated scene descriptions (“near a body of water”), the verbal signature of its null Δ -1 and 47%47\% shortcut rate. The full comparison across all nine models, including the base model and GLM-4.1-9B’s qualitatively distinct pattern, is in Appendix K. 7 Limitations SpatialChain has five main limitations. First, GQA scene graphs were annotated by humans and may contain errors that propagate into the synthetic reasoning chains. Second, the answer-consistency filter verifies a chain’s conclusion, not each intermediate step: the generator (Claude Haiku 4.5) can hallucinate, so retained chains conclude correctly but are not guaranteed step-correct; per-step verification against scene-graph node IDs is the natural next step. Third, the faithfulness judge (Claude Sonnet 4.6) is a single model from the same family as the reference generator, and one evaluated system shares that family. The second-judge and human-validation results (Appendix L) bound this risk, but references generated by a non-Claude model remain untested. Fourth, the task is binary Yes/No with a 58%58\% yes-rate in training, which enables Δpol _pol but caps the accuracy axis and is amplified by SFT; extending beyond binary VQA is the focus of future versions. Fifth, scene-graph relations are symbolic rather than metric, so the benchmark does not capture full geometric understanding (e.g., exact distances or 3D depth), and the complexity-biased test split is dominated by projective left/right relations. 8 Conclusion We introduced SpatialChain, a dataset pairing spatially grounded GQA questions with scene-graph-consistent reasoning chains, enabling training and structured evaluation of spatial reasoning in thinking-enabled VLMs. Chain quality significantly predicts answer correctness across most models; the judge evaluation reveals that high accuracy frequently coexists with shortcut rates above 39% — evidence that answer scores alone are insufficient to characterize spatial reasoning ability — and this signal survives a change of judge, a sweep of the threshold, and comparison with human annotators. SFT improves in-domain accuracy while exposing a stylistic trade-off on external benchmarks, motivating replay-augmented training as a future direction. Beyond SFT, SpatialChain supports reinforcement learning — outcome and process rewards can be computed automatically from the verifiable answer and scene-graph reference — and provides structured spatial supervision for Vision-Language-Action models [28, 43], where object-relation understanding is a prerequisite for manipulation and navigation. We hope SpatialChain encourages the community to interrogate the reasoning process behind model predictions, beyond the final answer. References [1] A. Agrawal, D. Batra, D. Parikh, and A. Kembhavi (2018) Don’t just assume; look and answer: overcoming priors for visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §1, §2, §4.1, §6.2. [2] Anthropic (2026) Introducing Claude Sonnet 4.6. Note: https://w.anthropic.com/news/claude-sonnet-4-6Accessed: 2026-05-01 Cited by: §1, §4.3, §5.1. [3] S. Bai, Y. Cai, R. Chen, K. Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, C. Gao, C. Ge, et al. (2025) Qwen3-vl technical report. arXiv preprint arXiv:2511.21631. Cited by: §1, §5.1. [4] Y. Cao, P. Zhang, X. Dong, D. Lin, and J. Wang (2024) DualFocus: integrating macro and micro perspectives in multi-modal large language models. arXiv preprint arXiv:2402.14767. Cited by: §2. [5] A. Cheng, H. Yin, Y. Fu, Q. Guo, R. Yang, J. Kautz, X. Wang, and S. Liu (2024) Spatialrgpt: grounded spatial reasoning in vision-language models. Advances in Neural Information Processing Systems 37, p. 135062–135093. Cited by: §2. [6] A. C. A. M. de Faria, F. d. C. Bastos, J. V. N. A. da Silva, V. L. Fabris, V. d. S. Uchoa, D. G. d. A. Neto, and C. F. G. d. Santos (2023) Visual question answering: a survey on techniques and common trends in recent literature. arXiv preprint arXiv:2305.11033. Cited by: §1. [7] M. Du, B. Wu, Z. Li, X. Huang, and Z. Wei (2024) Embspatial-bench: benchmarking spatial understanding for embodied tasks with large vision-language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), p. 346–355. Cited by: §1, §2. [8] Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh (2017) Making the V in VQA matter: elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), p. 6904–6913. Cited by: §1, §2, §4.1, §6.2. [9] W. Hong, W. Yu, X. Gu, G. Wang, G. Gan, H. Tang, J. Cheng, J. Qi, J. Ji, L. Pan, et al. (2025) GLM-4.5v and glm-4.1v-thinking: towards versatile multimodal reasoning with scalable reinforcement learning. arXiv preprint arXiv:2507.01006. Cited by: §1, §5.1. [10] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen (2022) LoRA: low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: §5.1. [11] D. A. Hudson and C. D. Manning (2019) GQA: a new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 6700–6709. Cited by: §1, §2, §3.1, §4.1. [12] D. Jiang, R. Zhang, Z. Guo, Y. Li, Y. Qi, X. Chen, L. Wang, J. Jin, C. Guo, S. Yan, et al. (2025) Mme-cot: benchmarking chain-of-thought in large multimodal models for reasoning quality, robustness, and efficiency. arXiv preprint arXiv:2502.09621. Cited by: §2. [13] J. Johnson, R. Krishna, M. Stark, L. Li, D. A. Shamma, M. Bernstein, and L. Fei-Fei (2015) Image retrieval using scene graphs. In CVPR, Cited by: §4.3. [14] O. Khangaonkar, H. J. Rad, and H. Pirsiavash (2026) Multimodal language models cannot spot spatial inconsistencies. arXiv preprint arXiv:2604.00799. Cited by: §2. [15] M. Kim, S. Lee, and D. Park (2026) VLM-SubtleBench: how far are VLMs from human-level subtle comparative reasoning?. In International Conference on Learning Representations (ICLR), Cited by: §2. [16] R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L. Li, D. A. Shamma, M. S. Bernstein, and L. Fei-Fei (2017) Visual genome: connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision 123 (1), p. 32–73. Cited by: §3.1. [17] W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica (2023) Efficient memory management for large language model serving with PagedAttention. In Proceedings of the ACM SIGOPS 28th Symposium on Operating Systems Principles (SOSP), Cited by: §5.1. [18] L. Li, Y. Wang, X. Gao, C. Tang, X. Yue, and C. You (2025) VisReason: a large-scale dataset for visual chain-of-thought reasoning. arXiv preprint arXiv:2511.17731. Cited by: §2, §3.3. [19] C. Lin (2004) ROUGE: a package for automatic evaluation of summaries. In Text Summarization Branches Out, p. 74–81. Cited by: §4.2. [20] F. Liu, G. E. T. Emerson, and N. Collier (2023) Visual spatial reasoning. Transactions of the Association for Computational Linguistics 11, p. 635–651. Cited by: §2. [21] Y. Liu, D. Iter, Y. Xu, S. Wang, D. Xu, and C. Zhu (2023) G-eval: nlg evaluation using gpt-4 with better human alignment. In EMNLP, Cited by: §4.3. [22] Z. Liu, Y. Dong, Y. Rao, J. Zhou, and J. Lu (2024) Chain-of-spot: interactive reasoning improves large vision-language models. arXiv preprint arXiv:2403.12966. Cited by: §2. [23] J. Lu, S. Srivastava, J. Chen, R. Shrestha, M. Acharya, K. Kafle, and C. Kanan (2025) Revisiting multi-modal llm evaluation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, p. 555–564. Cited by: §1. [24] P. Lu, S. Mishra, T. Xia, L. Qiu, K. Chang, S. Zhu, O. Tafjord, P. Clark, and A. Kalyan (2022) Learn to explain: multimodal reasoning via thought chains for science question answering. Advances in neural information processing systems 35, p. 2507–2521. Cited by: §2. [25] J. Ma, P. Wang, D. Kong, Z. Wang, J. Liu, H. Pei, and J. Zhao (2024) Robust visual question answering: datasets, methods, and future challenges. IEEE Transactions on Pattern Analysis and Machine Intelligence 46 (8), p. 5575–5594. Cited by: §1. [26] OpenAI (2026) GPT-5.4. Note: Accessed via API; no public technical report. Cited by: §1, §5.1. [27] J. Qi, M. Ding, W. Wang, Y. Bai, Q. Lv, W. Hong, B. Xu, L. Hou, J. Li, Y. Dong, and J. Tang (2025) CogCoM: a visual language model with chain-of-manipulations reasoning. External Links: 2402.04236, Link Cited by: §2. [28] D. Qu, H. Song, Q. Chen, Y. Yao, X. Ye, Y. Ding, Z. Wang, J. Gu, B. Zhao, D. Wang, et al. (2025) Spatialvla: exploring spatial representations for visual-language-action model. arXiv preprint arXiv:2501.15830. Cited by: §8. [29] P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang (2016) SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP), p. 2383–2392. Cited by: §4.2. [30] H. Shao, S. Qian, H. Xiao, G. Song, Z. Zong, L. Wang, Y. Liu, and H. Li (2024) Visual CoT: advancing multi-modal language models with a comprehensive dataset and benchmark for chain-of-thought reasoning. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §2, §3.3. [31] K. Stechly, K. Valmeekam, V. Palod, A. Gundawar, and S. Kambhampati (2025) Beyond semantics: the unreasonable effectiveness of reasonless intermediate tokens. In First Workshop on Foundations of Reasoning in Language Models, Cited by: §6.3. [32] G. R. Team, S. Abeyruwan, J. Ainslie, J. Alayrac, M. G. Arenas, T. Armstrong, A. Balakrishna, R. Baruch, M. Bauza, M. Blokzijl, et al. (2025) Gemini robotics: bringing ai into the physical world. arXiv preprint arXiv:2503.20020. Cited by: §2. [33] D. Teney, P. Anderson, X. He, and A. van den Hengel (2017) Graph-structured representations for visual question answering. In CVPR, Cited by: §4.3. [34] M. Turpin, J. Michael, E. Perez, and S. R. Bowman (2023) Language models don’t always say what they think: unfaithful explanations in chain-of-thought prompting. In NeurIPS, Cited by: §4.3. [35] J. Wang, Y. Ming, Z. Shi, V. Vineet, X. Wang, Y. Li, and N. Joshi (2024) Is a picture worth a thousand words? delving into spatial reasoning for vision language models. Advances in Neural Information Processing Systems 37, p. 75392–75421. Cited by: §2. [36] K. Wang, J. Pan, W. Shi, Z. Lu, H. Ren, A. Zhou, M. Zhan, and H. Li (2024) Measuring multimodal mathematical reasoning with math-vision dataset. Advances in Neural Information Processing Systems 37, p. 95095–95169. Cited by: §1. [37] W. Wang, Z. Gao, L. Gu, H. Pu, L. Cui, X. Wei, Z. Liu, L. Jing, S. Ye, J. Shao, et al. (2025) Internvl3.5: advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265. Cited by: §1, §5.1. [38] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, and D. Zhou (2022) Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems (NeurIPS), p. 24824–24837. Cited by: §2. [39] J. Yang, S. Yang, A. W. Gupta, R. Han, L. Fei-Fei, and S. Xie (2025) Thinking in space: how multimodal large language models see, remember, and recall spaces. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 10632–10643. Cited by: §2. [40] X. Yue, Y. Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y. Sun, C. Wei, B. Yu, R. Yuan, R. Sun, M. Yin, B. Zheng, Z. Yang, Y. Liu, W. Huang, H. Sun, Y. Su, and W. Chen (2024) MMMU: a massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 9556–9567. Cited by: §1, §2. [41] X. Yue, T. Zheng, Y. Ni, Y. Wang, K. Zhang, S. Tong, Y. Sun, B. Yu, G. Zhang, H. Sun, et al. (2025) Mmmu-pro: a more robust multi-discipline multimodal understanding benchmark. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 15134–15186. Cited by: §1, §2. [42] R. Zellers, Y. Bisk, A. Farhadi, and Y. Choi (2019) From recognition to cognition: visual commonsense reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 6720–6731. Cited by: §2. [43] J. Zhang, S. Wu, X. Luo, H. Wu, L. Gao, H. T. Shen, and J. Song (2025) Inspire: vision-language-action models with intrinsic spatial reasoning. arXiv preprint arXiv:2505.13888. Cited by: §8. [44] P. Zhang, Y. Goyal, D. Summers-Stay, D. Batra, and D. Parikh (2016) Yin and yang: balancing and answering binary visual questions. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 5014–5022. Cited by: §6.2. [45] Y. Zhang, Q. Zhang, X. Ju, Z. Liu, J. Mao, J. Sun, J. Wu, S. Gao, S. Cai, Z. Qin, et al. (2025) Embodiedvsr: dynamic scene graph-guided chain-of-thought reasoning for visual spatial tasks. arXiv preprint arXiv:2503.11089. Cited by: §2, §4.3. [46] L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica (2023) Judging llm-as-a-judge with mt-bench and chatbot arena. In NeurIPS, Cited by: §4.3. Appendix A GQA Relation-Centric Question Types Table 5 lists the 17 GQA question types retained in Stage 1 of the selection pipeline (§3.2), grouped by the relational competence they probe. Table 5: GQA question types selected for SpatialChain. Group GQA types Spatial relation verification relVerify, relVerifyCop, relVerifyCr Existential with relation existRelS, existRelSC, existRelSRC Place verification placeVerify, placeVerifyC Two-object comparison twoDifferent, twoDifferentC, twoSame, twoSameC Category comparison diffAnimals, diffAnimalsC, diffGender Attribute / ordinal selection chooseAttr, compare Appendix B Spatial Filter Prompt The following prompt is used with GPT-5-nano in Stage 2 of the question selection pipeline to distinguish questions that genuinely require spatial inference from those answerable by attribute recognition or existence detection alone. Spatial Reasoning Filter — User Prompt You are building a benchmark to evaluate spatial reasoning in multimodal LLMs. Your task is to classify whether a question genuinely requires spatial reasoning to be answered --- meaning the model must understand the position, layout, distance, or relationship between objects in an image. Answer YES only if the question requires: • Relative position between objects (left/right, above/below, in front/behind, next to, between) • Counting objects within a spatial context (e.g., ‘how many X are to the left of Y’) • Distance or proximity judgment (e.g., ‘what is closest to X’) • Spatial containment or grouping (e.g., ‘what is inside/outside X’) • Layout understanding (e.g., ‘what is in the corner/center/edge’) • Sequential or directional arrangement Answer NO if the question can be answered by: • Simply detecting or recognizing an object (‘is there a dog?’, ‘what color is the car?’) • Reading text or identifying attributes without spatial context • Generic scene understanding without positional reasoning (‘what is the weather?’) • Yes/no questions about object existence Examples: Q: What is to the left of the table? A: The chair → YES Q: Is the dog sitting? A: Yes → NO Q: What is between the lamp and the window? A: A plant → YES Q: What color is the sky? A: Blue → NO Q: How many people are behind the counter? A: Two → YES Q: Is there a car in the image? A: Yes → NO Now classify the following: Q: question A: answer Reply with only YES or NO. Appendix C Reasoning Chain Generation Prompt The following system and user prompts are used to generate reasoning chains with Claude Haiku 4.5 (extended thinking enabled). Generation System Prompt You are given a structured description of an image (a scene graph with objects, attributes, positions, and spatial relationships). Use this information to understand the scene, but respond AS IF you were directly looking at the image. IMPORTANT RULES FOR YOUR RESPONSE: • Never mention the scene graph, object IDs, coordinates, pixel positions, or any technical data. • Never say things like ‘according to the graph’, ‘object X is at position (a, b)’, or reference any ID numbers. • Write your reasoning as natural visual observation, as if you are describing what you see in a photograph. Use language like ‘I can see...’, ‘Looking at the image...’, ‘The cheese appears to be...’, ‘On the left side of the image...’. • Describe spatial relationships naturally (e.g., ‘the cheese is sitting to the left of the main dish’ instead of ‘cheese at x=1 is left of food at x=275’). • Your response will be paired with the actual image later, so it must read as authentic visual reasoning. Generation User Prompt Template Scene information (for your internal use only, do NOT reference directly): scene_graph_text Question: question Think step-by-step about what you observe in the image, then provide your final answer in the format: ANSWER: ⟨ answer⟩ Scene graph format. The scene_graph_text field is rendered as a JSON with structured list of objects with the following fields per entry: object name and ID, attributes (comma-separated), bounding box (x,y,w,h)(x,y,w,h) in pixels, and outgoing spatial relations in the form relation_name target_name(target_id). Appendix D Extended Dataset Statistics Spatial relation distribution. Table 6 reports the full breakdown of the eleven spatial relation categories covered in SpatialChain, together with per-relation yes-rates. Projective relations (left_of, right_of) dominate both splits, consistent with GQA’s underlying annotation distribution. The yes-rate varies substantially across relation types: vertical relations (above, on_top_of) tend toward higher yes-rates (≈0.70≈ 0.70), while depth relations (in_front_of, behind) are more balanced (≈0.55≈ 0.55), reflecting the greater ambiguity of depth ordering in 2D images. Mode details can be seen in Figure 2. Table 6: Per-relation statistics in the SpatialChain training split. Yes-rate is the fraction of questions with a Yes ground-truth answer. Relation Count % Yes-rate left_of 9,832 34.7 0.57 right_of 7,919 27.9 0.58 in_front_of 1,987 7.0 0.54 behind 1,406 5.0 0.55 near 347 1.2 0.61 above 763 2.7 0.71 below 412 1.5 0.68 on_top_of 411 1.4 0.70 next_to 1,876 6.6 0.59 inside 1,540 5.4 0.60 between / other 1,857 6.6 0.52 Total 28,350 100.0 0.58 Figure 2: Distribution of SpatialChain training (left) and test (right) splits by primary spatial relation (inner ring) and answer class (outer ring; green: yes, red: no, orange: attribute). The test split concentrates on projective relations, reflecting the complexity-biased selection. Reasoning chain length. Figure 3 shows the full word-count distributions for training and test chains. Both are unimodal and approximately log-normal. Training chains span P10P_10–P90P_90 = [74, 159] words (median 112, mean 115); test chains span [96, 203] words (median 146, mean 145), reflecting the complexity-biased test selection. Fewer than 2 % of chains in either split exceed 250 words, consistent with Claude Haiku 4.5’s tendency to produce concise, focused reasoning under the natural-language grounding constraint. At the sentence level (Figure 4), both splits follow a right-skewed unimodal distribution centred around 5–6 sentences (train mean = 4.58; test mean = 4.87). The slight rightward shift in the test split is consistent with the complexity-biased selection: chains that resolve longer spatial-relation sequences naturally require a few additional reasoning steps. Traces rarely exceed 10 sentences, confirming that the model’s chain-of-thought remains compact even for multi-hop queries. Thinking length vs. question length. Figure 5 plots thinking trace length (in characters) against question length for both splits. A moderate positive correlation is observed in the training set (Pearson r = 0.42), indicating that longer or more detailed questions tend to elicit proportionally longer reasoning traces. This correlation attenuates in the test split (r = 0.23), likely because test questions are more uniformly complex—compressing the range of question lengths while still requiring extended reasoning—which reduces the linear dependence between the two variables. Figure 3: Word-count distribution of reasoning chains in the training (blue) and test (orange) splits. Dashed vertical lines indicate medians. Figure 4: Distribution of the number of sentences per thinking trace for the training (left) and test (right) splits. Both distributions are unimodal and right-skewed, peaking at 5–6 sentences. The test split exhibits a slightly higher mean (4.87 vs. 4.58), consistent with its bias toward more complex multi-hop chains. Figure 5: Thinking trace length (in characters) as a function of question length (in characters) for the training (left, Pearson r = 0.42) and test (right, Pearson r = 0.23) splits. Longer questions elicit longer reasoning traces. Appendix E Dataset Examples Figure 6 shows representative examples from the released dataset, grouped by reasoning complexity. Single-hop · Yes img:2347569 Is there a fence to the left of the person that is pulling a suitcase?” Looking at the image, I can see a person standing in the center-right area. […] To the left of this person, I can clearly observe a black fence running along the left side […] Answer: ✓ Yes, there is a fence to the left of the person. Single-hop · No img:2359192 Do you see any skateboards to the right of the person that is wearing a shirt?” I can identify a boy on the left side wearing a black shirt. To the right of this person I can see red cans, napkins, silverware […] I do not see any skateboards in this area […] Answer: × No, there is a can to the right of the boy. Multi-hop · Yes img:2325221 Are there any umbrellas above the people to the left of the towel?” Step 1 — left: I can identify the towel — a white cloth on the right side of the image. To its left I can see a person standing. Step 2 — above: Examining what is above this person, I can clearly see umbrellas positioned overhead […] Answer: ✓ Yes, there is an umbrella above the people. Multi-hop · No img:2416648 Do you see pastries to the left of the food that is to the left of the knife?” Step 1 — left: Looking at the image, I can see a knife in the middle; to its left there are cookies on the table. Step 2 — left: To the left of the cookies I can see a woman, not pastries […] Answer: × No, there is a woman to the left of the cookies. Figure 6: Representative examples from SpatialChain, grouped by reasoning complexity. Top row: single-hop questions (one spatial relation). Bottom row: multi-hop questions chaining two relations (left_of→ ; left_of→ _of). Underlined phrases mark spatial relations; reasoning is truncated for space. Full chains are in the released dataset. Appendix F LLM-as-Judge Prompt The following prompt is used verbatim for all judge evaluations described in §4.3. The judge model receives the five fields listed below and returns a structured JSON object; no additional instructions are provided at inference time. Input fields. • question: the spatial VQA question. • ground_truth: the ground-truth answer derived from the GQA scene graph. • scene_graph: the relevant scene-graph subgraph (object nodes, attributes, and spatial relation edges). • candidate_answer: the model’s final answer. • candidate_reasoning: the model’s full thinking trace, with <think>/</think> delimiters stripped. Spatial Reasoning Judge — System Prompt You are a strict evaluator for spatial reasoning outputs. Goal Evaluate a candidate answer and its reasoning using: • question • ground truth (when available) • scene graph • candidate final answer • candidate reasoning • image scene graph Scoring Rubric (0.0 to 1.0) 1) answer_correctness 1.0: final answer is correct and matches the expected target. 0.5: partially correct or ambiguous but defensible. 0.0: clearly incorrect or unsupported. 2) reasoning_faithfulness 1.0: reasoning is grounded in spatial evidence (relations, coordinates, object references). 0.5: mostly grounded but with weak/missing support in some step. 0.0: hallucinated or contradicts available evidence. 3) reasoning_completeness 1.0: reasoning covers all key steps needed to justify the answer. 0.5: key idea present but skips important steps or constraints. 0.0: too shallow, irrelevant, or missing. Output Contract Return only valid JSON with this exact schema: "answer_correctness": 0.0, "reasoning_faithfulness": 0.0, "reasoning_completeness": 0.0, Output Constraints • No markdown fences in the final response. • No extra keys. Question: (question) GroundTruth: (ground_truth) SceneGraph: (scene_graph) CandidateAnswer: (candidate_answer) CandidateReasoning (candidate_reasoning) Return only Json Appendix G Hyperparameter Details Tables 7 and 8 report the full configuration used to fine-tune Qwen3-VL-8B-Thinking on SpatialChain. The same configuration is used for the 4B variant, with only the base model identifier changed. Table 7: Model and LoRA configuration. Hyperparameter Value Base model Qwen3-VL-8B-Thinking Quantization 4-bit NF4 (BitsAndBytes) Quantization compute dtype bfloat16 Double quantization ✓ Attention implementation Flash Attention 2 LoRA rank r 16 LoRA α 32 LoRA dropout 0.05 RSLoRA ✓ Target modules all linear layers (excl. lm_head) Table 8: Training configuration for SFT on SpatialChain. Hyperparameter Value Training examples 28,350 Epochs 2 Per-device batch size 6 Gradient accumulation steps 2 Effective batch size 12 Learning rate 3×10−53× 10^-5 LR schedule cosine Warmup ratio 0.05 Weight decay 0.01 Gradient clipping 1.0 Optimiser AdamW (fused) Precision bfloat16 + TF32 Gradient checkpointing disabled Max sequence length 32,768 tokens Image max size 640 px (longest side) Hardware 1 × A100 80 GB Training framework HuggingFace Transformers + PEFT Random seed 3407 Appendix H Inference System Prompt The following system prompt is used verbatim for all open-weight and proprietary models evaluated in §5.2. For InternVL3.5 models, this prompt is prepended with the standard thinking-activation prefix required by the model family. System prompt — inference Your task: 1. Analyze the image carefully. 2. Provide concise reasoning grounded in visible evidence 1. from the image. 3. End your response with ‘Answer: <one short sentence>’. For EmbSpatial-Bench evaluations (§6.4), the same prompt is used without modification, allowing any stylistic shift in model outputs to be attributed to fine-tuning rather than prompt differences. For ERQA there is no system prompt because the task instruction is embedded in each question. Appendix I Additional Experimental Results I.1 Extended Chain-Quality Metrics Table 9 extends the main comparison (Table 2) with four additional metrics: ROUGE-2, BLEU-1, token-level precision and recall reported separately, entity overlap (fraction of scene-graph object labels present in the predicted chain), and step markers (mean count of explicit step-delimiter phrases per chain, “first”, ”second”, ”third”, “then”, ”next”,“finally”). Two patterns emerge beyond what the main table captures. Token precision vs. recall separates models by how they fail: proprietary models (Claude 4.6, GPT-5.4, InternVL3.5-4B) show high precision but very low recall, confirming that their compact outputs contain relevant words but cover only a small fraction of the reference content. Open-weight base models (Qwen3-VL-4B/8B) exhibit the reverse — moderate precision but high recall — consistent with their verbose, exploratory chains. Step markers provide a direct proxy for explicit step-by-step structure: fine-tuned models average 0.26–0.45 per chain (matching the reference format), while open-weight base models average 2–3 (enumerating more steps than the reference), and proprietary models score 0.00, indicating the near-total absence of explicit reasoning structure. The entity overlap column further confirms that fine-tuned models (≈0.48≈ 0.48) reference scene-graph objects more faithfully than any other group, despite generating shorter chains. Table 9: Evaluation of reasoning chains on the SpatialChain test set. Extended version of Table 2 with additional metrics (ROUGE-2, BLEU, token precision/recall, entity overlap, step markers per chain). Models sorted by overall accuracy. Model Acc R-1 R-2 R-L F1tok Prectok Rectok BLEU-1 Entity-ov. Step-mk. Qwen3-VL-4B FT 82.23 0.657 0.407 0.450 0.646 0.701 0.614 0.570 0.481 0.26 Qwen3-VL-8B FT 85.03 0.654 0.402 0.443 0.643 0.694 0.615 0.568 0.484 0.45 GLM-4.1-9B 81.34 0.438 0.212 0.301 0.421 0.474 0.470 0.302 0.305 1.34 Qwen3-VL-8B 78.88 0.425 0.218 0.293 0.412 0.353 0.597 0.328 0.368 2.15 Qwen3-VL-4B 78.44 0.403 0.208 0.279 0.392 0.323 0.613 0.303 0.379 2.77 Claude 4.6 79.89 0.359 0.187 0.258 0.354 0.683 0.248 0.127 0.253 0.00 InternVL3.5-8B 79.33 0.323 0.180 0.228 0.316 0.454 0.451 0.183 0.310 1.41 InternVL3.5-4B 81.34 0.314 0.175 0.227 0.305 0.658 0.271 0.158 0.210 0.26 GPT-5.4 72.18 0.296 0.140 0.225 0.294 0.731 0.187 0.049 0.222 0.00 I.2 Per-Relation Accuracy Table 10 reports per-relation accuracy on the SpatialChain test split. Several relations (behind: 3 examples; below: 4; above: 12; next_to: 8) have very small sample sizes due to the complexity-biased test selection (§3.2), and results for those should be interpreted with caution. For the two well-sampled projective relations (left_of: n=314n=314; right_of: n=327n=327), the ranking of models tracks overall accuracy closely, confirming that the aggregate score is dominated by these two categories. The most notable departure is Claude 4.6, which achieves substantially higher accuracy on non-projective relations (in_front_of: 0.93, behind: 1.00, next_to: 1.00) than on projective ones (left_of: 0.76, right_of: 0.83). This pattern is consistent with Claude 4.6’s high shortcut rate (§6.5): projective left/right distinctions require precise visual grounding that shortcuts cannot supply, whereas relations such as in_front_of are more recoverable from linguistic co-occurrence priors. Table 10: Per-relation accuracy on the SpatialChain test split. Counts of validation examples for each relation: left_of (314), right_of (327), in_front_of (15), behind (3), above (12), below (4), next_to (8), other (13). Note that some relations have very small sample sizes; results for those are reported for completeness only. Model left_of right_of in_front_of behind above below next_to other Qwen3-VL-8B FT 0.836 0.868 0.800 1.000 0.833 0.500 0.750 0.923 Qwen3-VL-4B FT 0.816 0.825 0.800 0.667 0.833 1.000 1.000 0.923 GLM-4.1-9B 0.812 0.808 0.933 0.667 0.833 0.500 0.750 1.000 InternVL3.5-4B 0.803 0.815 1.000 0.667 0.667 1.000 0.750 1.000 Qwen3-VL-4B 0.796 0.778 0.733 0.667 0.833 0.000 0.750 0.769 Qwen3-VL-8B 0.789 0.788 0.667 1.000 0.833 0.500 0.750 0.923 InternVL3.5-8B 0.769 0.805 0.933 1.000 1.000 1.000 1.000 0.846 Claude 4.6 0.763 0.828 0.933 1.000 0.833 1.000 1.000 0.846 GPT-5.4 0.690 0.748 0.867 1.000 0.833 0.500 0.750 0.769 I.3 Fine-Tuning Trajectory on External Benchmarks Tables 11 and 12 track Qwen3-VL-4B accuracy across six checkpoints during SFT on SpatialChain: after 63, 313, 625, 1,875, and 2,813 steps (approximately 1k, 5k, 10k, 15k samples seen and one full epoch), plus the final model at 2 epochs (final). Both external benchmarks are evaluated with the same inference setup as the main experiments. ERQA. Degradation on ERQA is immediate and sharp: overall accuracy drops from 0.453 (base) to 0.388 at step 63, before stabilising between 0.372 and 0.425 for the remainder of training (Table 11). The sharpest per-category drop occurs in Trajectory Reasoning (0.409 → 0.273 at final), a category that requires tracking object paths across multiple images — a capability SpatialChain does not train on. Action Reasoning and State categories show more moderate and partially recovered degradation (≈−0.02≈-0.02 p from base to final), suggesting these competences are more resilient to stylistic specialisation. Notably, the degradation does not worsen monotonically: step 313 recovers partially before declining again, consistent with the model oscillating between the SpatialChain format and its pre-SFT behaviour during early training. EmbSpatial-Bench. The EmbSpatial-Bench trajectory (Table 12) shows a more structured pattern. Projective relations (left, right) and above are remarkably stable throughout training (drop <2<2 p at final), while depth-related relations (close: −5.0-5.0 p; far: −7.4-7.4 p) bear the bulk of the degradation. This is consistent with the interpretation in §6.4: SpatialChain trains primarily on projective and containment relations, leaving depth-ordering competence undertrained in the fine-tuned format. A partial recovery is visible from step 1,875 onward for close (0.613 → 0.649 at final), suggesting the model does not catastrophically forget depth reasoning but applies the SpatialChain template inappropriately to EmbSpatial prompts. Table 11: Qwen3-VL-4B accuracy on ERQA by category over fine-tuning. Step counts refer to optimisation steps with effective batch size 12. Checkpoint Traj. Action Spatial State Task Multi-v. Point. Other Overall Base 0.409 0.431 0.524 0.473 0.500 0.405 0.382 0.429 0.453 Step 63 0.288 0.375 0.488 0.527 0.368 0.297 0.353 0.143 0.388 Step 313 0.424 0.458 0.464 0.418 0.500 0.324 0.441 0.071 0.425 Step 625 0.167 0.417 0.464 0.455 0.474 0.324 0.412 0.214 0.380 Step 1875 0.303 0.417 0.405 0.436 0.421 0.351 0.324 0.429 0.385 Step 2813 0.303 0.389 0.405 0.436 0.342 0.351 0.441 0.143 0.372 Final 0.273 0.417 0.452 0.455 0.368 0.297 0.412 0.143 0.380 Table 12: Qwen3-VL-4B accuracy on EmbSpatial-Bench by relation category over fine-tuning. Vertical (above) and projective (left, right) relations are preserved; depth-related relations (close, far) are most affected. Checkpoint Left Right Above Under Close Far Overall Base 0.943 0.905 0.782 0.772 0.699 0.640 0.791 Step 63 0.937 0.913 0.797 0.698 0.676 0.608 0.773 Step 313 0.929 0.873 0.819 0.701 0.670 0.582 0.763 Step 625 0.904 0.845 0.809 0.724 0.642 0.609 0.757 Step 1875 0.904 0.858 0.809 0.704 0.613 0.567 0.744 Step 2813 0.909 0.873 0.807 0.721 0.624 0.601 0.757 Final 0.925 0.869 0.804 0.721 0.649 0.566 0.757 Appendix J Qualitative Example Figure 7 contrasts the two most different reasoning regimes on a single test item; complete outputs for all nine models follow in Appendix K. Question imageId: 2328879 “Do you see kittens to the right of the bread that is to the right of the knife?” Ground-truth answer: No, there is a bird to the right of the bread. Reference reasoning (SpatialChain): Looking at the image step-by-step: (1) knife on the left of the plate; (2) bread to the right of the knife; (3) gray birds to the right of that bread; (4) no kittens present. Conclude: No, there is a bird. Qwen3-VL-4B FT — grounded “[…] knife on the left […] bread to the right of the knife […] a bird perched to the right of that bread […] no kittens anywhere.” Answer: ✓ No, there is a bird. Follows the reference chain step-by-step; names the bird as direct evidence. InternVL3.5-8B — verbose-decorative “[…] there is indeed a bird, not a kitten […] The setting seems to be outdoors, possibly near a body of water […] outdoor dining […]” Answer: ✓ No (implicit) Correct answer embedded in scene description unrelated to the spatial chain. Figure 7: Both models answer correctly, but through qualitatively different processes. The reference chain (top) was generated from the GQA scene graph by SpatialChain and follows a step-by-step spatial structure. The fine-tuned model (left) closely mirrors this structure and names the bird as direct evidence. InternVL3.5-8B (right) arrives at the correct answer via hallucinated scene description. Full outputs for all nine models are in Appendix K. Appendix K Additional Qualitative Example Table 7 present additional reasoning chain example from the SpatialChain test split, covering all nine evaluated models and including the GLM-4.1-9B case discussed in §6.5. Figure 8: Image 2328879 used as input fot the discussed question "Do you see kittens to the right of the bread that is to the right of the knife". Table 13: Complete model outputs for the qualitative example in Figure 7 (imageId: 2328879). Question: “Do you see kittens to the right of the bread that is to the right of the knife?” Ground-truth answer: No, there is a bird to the right of the bread. Outputs are reproduced verbatim; <think> / </think> delimiters are omitted for readability. Model Full prediction Correct Reference (scene graph) “Looking at the image, let me trace through this step-by-step: (1) Locating the knife: I can see a knife on the left side of the plate, positioned near the bottom left. (2) Finding the bread to the right of the knife: there is a large piece of bread that sits on the plate to the right of where the knife is placed. (3) Examining what is to the right of that bread: to the right of the bread I can see what appears to be gray birds standing on the plate. These are birds with wings and legs visible. (4) Looking for kittens: I do not see any kittens anywhere in the image. The animals present in the scene are birds — specifically gray colored birds that appear to be standing on or near the bread on the plate.” Answer: No, there is a bird to the right of the bread. — Qwen3-VL-4B FT (fine-tuned) “Looking at the image, I need to trace through this step by step: (1) First, I can see a knife on the left side of the image, positioned near the edge. (2) To the right of the knife, there is bread visible in the center-left portion of the image. (3) To the right of that bread, I can see a white plate with a bird perched on it. (4) Looking further to the right of the bread (and the plate with the bird), I see what appears to be a small bird on the table, but examining the entire right side of the bread, I do not see any kittens anywhere in the image. The image shows a table setting with a woman, food items including bread, a plate, a bird, and other dining elements. There are no kittens visible anywhere in the scene, let alone to the right of the bread.” Answer: No, there is a bird to the right of the bread. ✓ Qwen3-VL-8B FT (fine-tuned) “Looking at the image, I can see a white plate with various food items on it. On the left side, there’s a knife, and to the right of it is a piece of bread. Moving further to the right of the bread, I can see what appears to be a small brown bird standing on the plate. However, I don’t see any kittens anywhere in this image. The only animal visible to the right of the bread is a small brown bird. There are no kittens present in the scene.” Answer: No, there are no kittens visible in the image. ✓ Qwen3-VL-4B (base) “So, let us break this down. First, the question is asking if there are kittens to the right of the bread that is to the right of the knife. Let us identify each element. First, the knife: there is a knife on the left side of the plate, near the bread. The bread is the sandwich in the center of the plate. Now, ‘the bread that is to the right of the knife’ — so the knife is on the left, the bread is to the right of that knife. Then, we need to look to the right of that bread. What is there? On the plate, to the right of the bread (sandwich), there is a sauce (red sauce, maybe tomato) and some other food. But the question is about kittens. Are there any kittens in the image? Let us check the image. The image shows a plate with a sandwich, some sauce, a knife, and two small birds (sparrows) on the plate or near it. There are no kittens visible. The birds are sparrows, not kittens. So, to the right of the bread (which is the sandwich), there are no kittens. The birds are there, but they are birds, not kittens. So the answer is no.” Answer: No. ✓ Qwen3-VL-8B (base) “Got it, let’s break this down. First, identify the elements. The question is about kittens to the right of the bread that’s to the right of the knife. Let’s look at the image. First, find the knife. There’s a knife on the left side of the plate, next to the fork. Then, the bread: the hamburger bun is on the plate, to the right of the knife. Now, to the right of that bread (hamburger bun), what’s there? There’s a bird, not a kitten. The image shows birds, not kittens. So no kittens are present. The question asks if there are kittens to the right of the bread (hamburger) which is right of the knife. Since there are birds, not kittens, the answer is no.” Answer:The image shows birds (not kittens) to the right of the hamburger bun, which is to the right of the knife. No kittens are visible. Answer: No, there are no kittens in the image ✓ GLM-4.1-9B “Got it, let’s analyze the image. First, look for the knife, then the bread to its right, then check if there are kittens to the right of that bread. The image shows a plate with food, a knife on the left, bread (the bun) next to it, and then birds, not kittens. So there are no kittens in that area. Therefore, the answer would be no.” Answer: To determine this, first locate the knife on the left side of the plate, then identify the bread (bun) to the right of the knife. Next, check for kittens to the right of that bread. The area to the right of the bread shows birds, not kittens. Therefore, the answer is No. ✓ InternVL3-4B “kittens is not to the right of the bread that is to the right of the knife.” Answer: No. ✓ InternVL3-8B “In the image, to the right of the bread that is to the right of the knife, there is indeed a bird, not a kitten. The bird appears to be a sparrow, standing on the plate beside the plate with the sandwich. There are no kittens visible in the image, so the presence of a kitten to the right of the mentioned bread is not accurate based on what can be observed. The bird is interacting with the food, which adds a dynamic element to the scene, contrasting with the stillness of the sandwich and the remnants of sauce on the plate. The setting seems to be outdoors, possibly near a body of water, given the presence of what appears to be a waterfront in the background. The interaction between wildlife and human food can sometimes occur in outdoor dining settings, especially in areas where birds are accustomed to human presence.” Answer: No (implicit). ✓ Claude 4.6 “To the right of the bread/sandwich on the plate, there are birds (sparrows) perched on the edge of the plate, not kittens. No cats or kittens are visible anywhere in the image.” Answer: No. ✓ GPT-5.4 “The knife is on the left side of the plate, and the bread is to its right. To the right of the bread there are two birds on the plate, not kittens.” Answer: No. ✓ Appendix L Judge Validation and Robustness This appendix reports the experiments that validate the faithfulness axis: a human agreement study, a second judge from a different provider, a sweep of the faithfulness threshold, formal cross-model correlation coefficients, and a stratification by question complexity. L.1 Human Validation of the Judge We sampled 198 test items — 22 from each of the nine evaluated models — stratified across the judge’s faithfulness range, with deliberate oversampling around =0.5RF=0.5. Six annotators scored every item under the Appendix F rubric verbatim, seeing the image, question, ground truth, candidate answer and chain, with model identity and judge scores hidden. Every item was scored by two annotators independently (396 annotations). Annotators scored against the image; the judge scores against the scene graph. We report Spearman rank correlation on the continuous scores without binarising either side, since thresholding one rater against another conflates disagreement about a chain with disagreement about where a cutoff belongs (Table 14). The judge ↔ individual human column is the like-for-like comparison with the human–human protocol. Table 14: Rank correlation (Spearman ρ) between raters on n=198n=198 items. Mean absolute error of the judge against the human consensus is 0.14 (correctness), 0.27 (faithfulness), 0.21 (completeness). Dimension human ↔ human judge ↔ individual human judge ↔ human consensus Answer correctness 0.673 0.655 0.679 Reasoning faithfulness 0.443 0.488 0.574 Reasoning completeness 0.416 0.551 0.643 On both reasoning dimensions the judge agrees with a human annotator at least as well as two annotators agree with each other (RF: 0.488 vs. 0.443; RC: 0.551 vs. 0.416). Faithfulness is an intrinsically subjective judgment, and human–human agreement sets the ceiling one can expect of any rater; the judge operates at that ceiling. Annotator variance is substantial and is documented with the released annotations. What the ≤0.5RF≤ 0.5 cutoff captures. Among correct answers, a recurring low-faithfulness pattern is a chain that restates the question and asserts the answer, introducing no vocabulary beyond the question, identifying no object and traversing no relation — e.g. “Is there a teddy bear to the right of the bears that are hanging from the ceiling?” → “Yes, there is a teddy bear to the right of the bears that are hanging from the ceiling.” (judge RF 0.5; human mean 0.30). Fourteen chains in the sample follow this template; both the judge and the annotators score them low. The cutoff also catches the opposite extreme: a 13,000-word Qwen3-VL-8B chain that cycles through reversals without resolving the relation and terminates mid-deliberation, yet is scored correct (judge RF 0.2; both annotators 0.0). Length and grounding are orthogonal. Where judge and annotators diverge. In the 56 items where the judge scores faithfulness low and the annotators score it high, the chains are not padded: relative to chains the judge scores highly they have the same length (median 104 vs. 105 words) and no more scene-description material, but mention fewer spatial relations (7.7 vs. 10.4) and fewer step markers (0.52 vs. 0.89). This is an asymmetry of evidence rather than an error by either rater: the judge penalises a chain that reaches the right answer without engaging the annotated relational structure, whereas a human inspecting only the image finds such a chain reasonable. L.2 Second Judge from a Different Provider We re-ran the full judging protocol on all 899 test items with GPT-5.6, using identical code and prompt. Continuous faithfulness scores correlate at Spearman ρ=0.74ρ=0.74 (quadratic κ=0.74κ=0.74), pass/fail verdicts agree at κ=0.84κ=0.84, and the ranking of models by shortcut rate is preserved (ρ=0.88ρ=0.88; Table 15). GPT-5.6 is uniformly stricter, which shifts absolute values but not the comparative conclusions. The same-family model, Claude Sonnet 4.6, ranks 5th of nine under the Claude judge and 3rd under the GPT judge — it is treated more harshly, not more leniently, by the same-family judge. Table 15: Shortcut rate under two independent judges (fraction of correct test set, ≤0.5RF≤ 0.5). Model Claude Sonnet 4.6 judge GPT-5.6 judge Qwen3-VL-8B FT 0.222 0.466 Qwen3-VL-4B FT 0.194 0.396 GLM-4.1-9B 0.453 0.595 Claude Sonnet 4.6 0.390 0.546 Qwen3-VL-4B 0.264 0.574 Qwen3-VL-8B 0.260 0.588 GPT-5.4 0.246 0.373 InternVL3.5-8B 0.473 0.653 InternVL3.5-4B 0.580 0.699 L.3 Threshold Sensitivity We swept the faithfulness cutoff over τ∈[0.3,0.7]τ∈[0.3,0.7] for all nine models (Table 16, Figure 9). Because the judge’s scores concentrate on a few discrete values, shortcut rates are identical for τ∈[0.5,0.6]τ∈[0.5,0.6] and the model ordering is unchanged across that plateau — the published 0.50.5 cutoff sits in the middle of a flat region rather than at a sensitive point. Below τ=0.4τ=0.4 the metric has essentially no discriminative power: seven of nine models fall below 0.030.03. The substantive conclusions — that the fine-tuned variants have the lowest shortcut rates and InternVL3.5-4B/8B the highest — hold throughout the range where the metric separates models at all (τ≥0.5τ≥ 0.5). Table 16: Shortcut rate vs. faithfulness threshold τ (≤τRF≤τ; fraction of correct test set). Model τ=0.3τ=0.3 τ=0.4τ=0.4 τ=0.5τ=0.5 τ=0.6τ=0.6 τ=0.7τ=0.7 Qwen3-VL-8B FT 0.026 0.029 0.222 0.227 0.358 Qwen3-VL-4B FT 0.019 0.020 0.194 0.195 0.305 GLM-4.1-9B 0.003 0.003 0.453 0.455 0.627 Claude Sonnet 4.6 0.008 0.009 0.390 0.390 0.473 Qwen3-VL-4B 0.010 0.010 0.264 0.265 0.433 Qwen3-VL-8B 0.018 0.018 0.260 0.265 0.427 GPT-5.4 0.000 0.000 0.246 0.246 0.324 InternVL3.5-8B 0.139 0.142 0.473 0.473 0.543 InternVL3.5-4B 0.295 0.296 0.580 0.580 0.622 Figure 9: Shortcut rate as a function of the faithfulness threshold τ. The shaded region marks the plateau τ∈[0.5,0.6]τ∈[0.5,0.6] on which all rates are identical. L.4 Cross-Model Correlation between Accuracy and Chain Quality Table 17 reports formal correlation coefficients between a model’s VQA accuracy and its mean chain quality (ROUGE-1) across the models of Table 2. Across all nine models the association is moderate and at best marginally significant, and it is carried largely by the two fine-tuned variants, which are simultaneously the most accurate systems and, by construction, the ones whose chains most closely match the reference style. With nine models these coefficients are indicative rather than conclusive. R-L and token-F1 give near-identical values (all nine models, Acc × F1tok: r=+0.681r=+0.681, ρ=+0.644ρ=+0.644, τ=+0.479τ=+0.479). Table 17: Cross-model correlation between VQA accuracy and reasoning-chain quality (ROUGE-1). Subset n Pearson r Spearman ρ Kendall τ All models 9 +0.690+0.690 (p=0.040p=0.040) +0.644+0.644 (p=0.061p=0.061) +0.479+0.479 (p=0.075p=0.075) Zero-shot only 7 +0.445+0.445 (p=0.317p=0.317) +0.270+0.270 (p=0.558p=0.558) +0.195+0.195 (p=0.543p=0.543) Qwen3 family only 4 +0.926+0.926 (p=0.074p=0.074) +0.800+0.800 (p=0.200p=0.200) +0.667+0.667 (p=0.333p=0.333) L.5 Single-Hop vs. Multi-Hop Questions We stratified the test set by hop count — the number of spatial relations chained in the question, inferred from the question text (385 single-hop, 509 multi-hop, 5 ambiguous cases classified by hand) — and recomputed accuracy and faithfulness per stratum (Table 18). The effect is small and, for most models, runs mildly opposite to the intuition that multi-hop is harder: accuracy averages +0.019+0.019 higher on multi-hop and faithfulness +0.014+0.014 higher, with the Qwen3 base models showing the clearest positive shift. The two models most associated with shortcuts, InternVL3.5-4B/8B, are flat or slightly negative. We read this as evidence that the dominant difficulty is perceiving the individual relation rather than composing several. Accuracy here is computed over all items in each stratum using the judge’s correctness label, whereas Table 2 excludes unparsed predictions per model; this accounts for the small residual difference for InternVL3.5-4B (+0.018+0.018) and InternVL3.5-8B (+0.014+0.014), the two models with the most unparsed outputs. For the other seven models the two agree to within 0.0050.005. Table 18: Accuracy and reasoning faithfulness by question complexity (single-hop: n=385n=385; multi-hop: n=509n=509). Δ is multi −- single. Model Acc single Acc multi Δ RF single RF multi Δ Qwen3-VL-8B FT 0.847 0.845 −0.002-0.002 0.630 0.630 −0.000-0.000 Qwen3-VL-4B FT 0.834 0.809 −0.024-0.024 0.641 0.623 −0.017-0.017 GLM-4.1-9B 0.810 0.815 +0.005+0.005 0.517 0.533 +0.016+0.016 Claude Sonnet 4.6 0.777 0.823 +0.047+0.047 0.529 0.550 +0.022+0.022 Qwen3-VL-4B 0.764 0.805 +0.042+0.042 0.566 0.597 +0.031+0.031 Qwen3-VL-8B 0.748 0.815 +0.067+0.067 0.550 0.586 +0.036+0.036 GPT-5.4 0.694 0.741 +0.047+0.047 0.529 0.551 +0.022+0.022 InternVL3.5-8B 0.813 0.803 −0.009-0.009 0.451 0.436 −0.016-0.016 InternVL3.5-4B 0.831 0.831 −0.000-0.000 0.327 0.360 +0.033+0.033