Paper deep dive

Debias in Text, Believe Your Eyes: Text-Anchored Cross-Modal Transfer for Visual Counter-Commonsense Reasoning

Chen Ling, Hanqian Li, Dongnan Liu, Keyu Qian, Jungang Li, Xinglong liu, Shiyi Wang, Xin Dong, Pengcheng Zhu, Wei Zhou, Linjian Mo, Nai Ding

Year: 2026Venue: arXiv preprintArea: cs.CVType: PreprintEmbeddings: 120

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%

Last extracted: 8/10/2026, 3:41:24 AM

Summary

The paper introduces TACT (Text-Anchored Cross-Modal Transfer), a framework that improves multimodal large language models' (MLLMs) counter-commonsense reasoning by debiasing the shared language decoder. The authors identify that MLLMs often fail not due to poor visual perception, but because the language decoder favors strong prior knowledge over conflicting visual evidence. TACT uses a text-anchored pipeline involving Fact-Frequency Distillation (FFD) to create high-quality counter-commonsense QA pairs and a two-stage post-training process (Supervised Fine-Tuning and Direct Preference Optimization) to encourage evidence-following reasoning without requiring visual training data.

Entities (10)

MLLMs · model-class · 99%TACT · methodframework · 98%Fact-Frequency Distillation · methodcomponent · 95%Language Prior · concept · 95%Qwen3-VL-8B · model · 95%CDH-Bench · benchmark · 92%CAIT · benchmark · 90%Direct Preference Optimization · method · 90%VLind-Bench · benchmark · 90%Visual Contrastive Decoding · method · 85%

Relation Signals (7)

TACT → improves → Visual Counter-Commonsense Reasoning

confidence 98% · TACT substantially improves visual reasoning while preserving general capabilities

MLLMs → suffersfrom → Language Prior Bias

confidence 95% · MLLMs often rely on learned priors rather than genuine reasoning, leading to poor generalization

TACT → targets → Shared Language Decoder

confidence 95% · TACT... debiases the shared language decoder without requiring any visual training data

TACT → uses → Fact-Frequency Distillation

confidence 95% · Building upon this corpus, we introduce TACT... whose core component, Fact-Frequency Distillation (FFD)

TACT → employs → Direct Preference Optimization

confidence 90% · we further optimize the model with Direct Preference Optimization (DPO)

Fact-Frequency Distillation → generates → Counter-Commonsense QA Pairs

confidence 90% · FFD... distills verified counter-commonsense scenarios into a high-quality text corpus

CDH-Bench → usedforevaluating → TACT

confidence 90% · Across counter-commonsense visual benchmarks, TACT substantially improves visual reasoning

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:The visual reasoning ability of multimodal large language models (MLLMs) is crucial for downstream applications, particularly counter-commonsense reasoning, which requires models to reason beyond common assumptions. Recent studies mainly improve visual counter-commonsense reasoning by enhancing visual inputs, following the assumption that failures originate from insufficient visual grounding. However, our empirical analysis reveals that the bottleneck is not visual perception. MLLMs already capture the relevant visual evidence, and the correct answer exists in their decoding space. Instead, the shared language decoder resolves prior--evidence conflicts by favoring dominant language priors, especially for low-frequency factual scenarios. Motivated by this, we first propose a text-anchored data construction pipeline, whose core component, Fact-Frequency Distillation (FFD), estimates the prior strength of commonsense facts and distills verified counter-commonsense scenarios into a high-quality text corpus. Building upon this corpus, we introduce TACT, a text-anchored post-training framework that debiases the shared language decoder without requiring any visual training data. TACT routes evidence-following and prior-driven reasoning trajectories into different optimization stages, enabling the decoder to resolve prior--evidence conflicts. Across counter-commonsense visual benchmarks, TACT substantially improves visual reasoning while preserving general capabilities, demonstrating effective text-to-vision cross-modal transfer.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

118,220 characters extracted from source content.

Expand or collapse full text

Debias in Text, Believe Your Eyes: Text-Anchored Cross-Modal Transfer for Visual Counter-Commonsense Reasoning Chen Ling1,3 , Hanqian Li2,3 †footnotemark: , Dongnan Liu3, Keyu Qian4, Jungang Li2, Xinglong Liu5, Shiyi Wang3, Xin Dong3, Pengcheng Zhu3, Wei Zhou3, Linjian Mo3, Ding Nai†footnotemark: 1 This work was done during an internship at Ant Group.Corresponding author. Abstract The visual reasoning ability of multimodal large language models (MLLMs) is crucial for downstream applications, particularly counter-commonsense reasoning, which requires models to reason beyond common assumptions. Recent studies mainly improve visual counter-commonsense reasoning by enhancing visual inputs, following the assumption that failures originate from insufficient visual grounding. However, our empirical analysis reveals that the bottleneck is not visual perception. MLLMs already capture the relevant visual evidence, and the correct answer exists in their decoding space. Instead, the shared language decoder resolves prior–evidence conflicts by favoring dominant language priors, especially for low-frequency factual scenarios. Motivated by this, we first propose a text-anchored data construction pipeline, whose core component, Fact-Frequency Distillation (FFD), estimates the prior strength of commonsense facts and distills verified counter-commonsense scenarios into a high-quality text corpus. Building upon this corpus, we introduce TACT, a text-anchored post-training framework that debiases the shared language decoder without requiring any visual training data. TACT routes evidence-following and prior-driven reasoning trajectories into different optimization stages, enabling the decoder to resolve prior–evidence conflicts. Across counter-commonsense visual benchmarks, TACT substantially improves visual reasoning while preserving general capabilities, demonstrating effective text-to-vision cross-modal transfer. 1 Introduction Multimodal Large Language Models (MLLMs) have achieved remarkable progress in visual understanding. However, they still struggle with counter-commonsense reasoning, which requires models to reason beyond memorized correlations and infer outcomes under hypothetical or visually inconsistent conditions(Hu et al. 2025; Huang et al. 2025a; Zheng et al. 2026; Xun et al. 2026; Huang et al. 2025b). Recent studies show that MLLMs often rely on learned priors rather than genuine reasoning, leading to poor generalization in counter-commonsense settings (Wu et al. 2024; Li et al. 2025; Yang et al. 2025). Figure 1: A counter-commonsense example from CDH-Bench (a chair floating in mid-air). Given the same image and question, the dominant Bad Trajectory (∼ 87.5%) fabricates grounding evidence and returns the prior answer (A), whereas a rarer Good Trajectory (∼ 12.5%) faithfully reports that the legs do not touch the rug or the floor and answers correctly. Figure 2: pass@k versus prior-lock@k on the counter-commonsense split of CDH-Bench MC for three MLLMs. Recent studies(Xiao et al. 2026a; Yu et al. 2025) show that MLLMs often rely on high-frequency commonsense priors when visual evidence conflicts with expectations, leading to commonsense-driven hallucinations. Existing mitigation methods mainly follow two directions: enhancing visual evidence through approaches such as Visual Contrastive Decoding (VCD) (Leng et al. 2023), M3ID (Favero et al. 2024), VDD (Zhang et al. 2025), and MemVR (Zou et al. 2025), or intervening in the language decoder through inference-time calibration such as NoLan (Ren et al. 2026). However, these approaches leave open a fundamental question: whether MLLMs already perceive the correct evidence but fail to follow it when it conflicts with strong language priors. Motivated by these observations, we investigate the fundamental challenge of robust counter-commonsense reasoning in MLLMs and propose a novel framework for mitigating language-prior bias in multimodal reasoning. Our approach is grounded in a simple cognitive principle: believe what you see, not what you expect. Based on this insight, we develop a text-anchored data construction pipeline for counter-commonsense reasoning. As its core component, Factuality-Frequency Distillation (FFD) estimates the prior strength of commonsense facts, performs frequency-based routing, and distills informative counter-commonsense facts into high-quality QA pairs. Building upon these, we further propose Text-Anchored Cross-Modal Transfer (TACT), which performs prior-aware trajectory curation to construct effective supervision and subsequently optimizes the language decoder through two-stage post-training. By explicitly reinforcing evidence-following reasoning while suppressing prior-driven reasoning, TACT encourages MLLMs to describe what they observe before reasoning about what they expect. Extensive experiments on multiple multimodal counter-commonsense reasoning benchmarks demonstrate that TACT consistently improves counter-commonsense reasoning while preserving the general capabilities of MLLMs. Our main contributions are summarized as follows: • We identify that the key bottleneck of visual counter-commonsense reasoning in MLLMs lies in the language decoder, rather than insufficient visual perception. • We propose the Fact-Frequency Distillation (FFD), a frequency-aware data construction method for distilling high-quality counter-commonsense QA pairs. • We propose the Text-Anchored Cross-modal Transfer (TACT), which performs prior-aware trajectory curation and two-stage post-training to encourage evidence-following reasoning. • Extensive experiments demonstrate that TACT consistently improves visual counter-commonsense reasoning while preserving the general capabilities of MLLMs using only text-only supervision. 2 Related Work Counter-Commonsense Reasoning in MLLMs. Recent studies investigate whether MLLMs can faithfully reason when visual evidence contradicts commonsense priors. Early works reveal that models are sensitive to language-prior shortcuts under shifted answer distributions (Agrawal et al. 2018), struggle with explanations of commonsense-defying scenes (Bitton-Guetta et al. 2023), and fail under reversed semantic roles (Thrush et al. 2022). More recent benchmarks directly examine prior–evidence conflicts: CAIT (Ling et al. 2026) evaluates whether models can recognize reversed agent–patient interactions (e.g., a rabbit dragging a tiger); CDH-Bench (Chen et al. 2026) studies commonsense-driven hallucinations induced by counting, relational, and attribute anomalies through paired counterfactual and commonsense images; and VLind-Bench (Lee et al. 2025) provides a stage-wise analysis of such failures, suggesting that many errors arise at the decision stage rather than from visual perception. These benchmarks share a common setting where rare-but-valid visual evidence conflicts with high-frequency language priors, making them suitable for evaluating prior-resistant visual reasoning. Language Priors and Their Mitigation in MLLMs. Language priors encoded in the shared language decoder are essential for the strong generalization ability of MLLMs, but can also dominate visual evidence in low-frequency or counter-commonsense scenarios (Li et al. 2023; Bai et al. 2025c). Existing mitigation methods primarily intervene in the visual pathway or the decoding process, including visual contrastive decoding (Leng et al. 2023), multimodal–text logit comparison (Ren et al. 2026), visual token reinjection (Zou et al. 2025), and alignment or representation editing (Yu et al. 2024; Golovanevsky et al. 2025; Xiao et al. 2026b; Anand et al. 2026; Liu et al. 2025; Xu et al. 2025; Ren et al. 2023; Li et al. 2026; Hao et al. 2024). Recent studies further suggest that prior–evidence conflicts arise primarily in the shared language decoder rather than the vision encoder (Ortu et al. 2025). In parallel, LLM research has shown that targeted text supervision can recalibrate parametric knowledge and promote evidence-following behavior (Longpre et al. 2022; Li et al. 2022; Su et al. 2024; Fan et al. 2026). Whether such text-only supervision can similarly recalibrate language priors in MLLMs remains unexplored. Our work addresses this question through text-anchored decoder post-training. 3 Preliminary Figure 3: Overview of our three-step text-anchored data construction pipeline, illustrating the process from taxonomy construction and factual knowledge acquisition to factuality frequency distillation. We formulate a counter-commonsense instance as two competing propositions over an image I: a prior-consistent proposition P favored by language priors, and a visual-evidence proposition E supported by the image, where E≠PE≠ P. Given question q, the model MθM_θ should follow the evidence (y^=Mθ(I,q)=E y=M_θ(I,q)=E), while prior-driven errors correspond to predicting P. Examples from our three benchmarks are provided in Appendix F. To locate the source of this failure, we first conduct a pre-study on CDH-Bench with three MLLMs (Bai et al. 2025a; Wang et al. 2025; Bai et al. 2025b). 3.1 Perception Is Not the Bottleneck To distinguish the perception failure from language-prior suppression, we evaluate whether correct answers can emerge through multi-trajectory decoding. For each counter-commonsense item, we sample n=8n=8 Chain-of-Thought (Wei et al. 2023) trajectories with temperature 1.01.0 and measure pass@k (Chen et al. 2021) together with prior-lock@k, defined as the probability that all k sampled trajectories select the prior answer P: prior-lock@k=items[(pk)(nk)],prior-lock@k=E_items [ pk nk ], (1) where p is the number of trajectories predicting P. At k=1k=1, prior-lock@11 corresponds to the prior-bias rate under greedy decoding. As shown in Figure 2, increasing k from 11 to 88 improves pass@k by 1515–2222 points while reducing prior-lock@k by 1414–1818 points across all models, indicating that correct visual evidence is already accessible but suppressed by language priors rather than missing from perception. 3.2 The Failure Is a Decision-Level Prior Bias To identify whether errors arise from inaccessible visual evidence or prior-biased decision making, we analyze the failure cases of Qwen3-VL-8B (details in Appendix G). We find that most errors select the commonsense option. A simple grounding prompt, which asks the model to describe the queried detail before answering, substantially reduces these errors, suggesting that visual evidence is available but ignored during decision making. The remaining failures exhibit over-rationalization, where CoT generates visual explanations that justify the prior (e.g., describing the floating chair in Figure 1 as “resting on its legs atop the rug”). These observations indicate that the bottleneck lies in the decoder’s decision policy: when visual evidence conflicts with language priors, the model favors the prior. Since this decision policy is implemented in the shared language decoder, we hypothesize that text-only supervision can reshape it and transfer to visual reasoning without additional visual supervision. 4 Methodology In this section, we first introduce our text-anchored data construction pipeline, which generates counter-commonsense QA pairs through factuality frequency distillation (FFD) and further derives CoT supervision from the constructed examples. We then present TACT, a two-stage post-training framework that curates the constructed corpus and optimizes it through supervised and preference learning. 4.1 Text-Anchored Data Construction To mitigate language-prior bias in MLLMs, we build a scalable text-only data construction pipeline (Figure 3) that synthesizes counter-commonsense examples with evidence–prior conflicts, covering three task formats: true/false verification (TFV), multiple-choice question answering (MCQ), and binary-choice question answering (BCQ). The pipeline consists of three main steps. Step 1: Taxonomy Construction. To ensure systematic coverage rather than ad hoc generation, we first define a taxonomy of commonsense priors. Human annotators collaboratively identify the types of commonsense knowledge that can be violated in counter-commonsense scenes, resulting in a hierarchical taxonomy with 6 major and 28 minor categories, including Agency, World Facts, Physical Relations, Magnitude, Appearance, and Social Roles (Appendix A). Each minor category defines a conflict attribution (e.g., Appearance → color, Habitat → dwelling place), which guides the subsequent knowledge mining process. Step 2: Knowledge Acquisition. For each conflict attribution defined in Step 1, we acquire entities and their corresponding commonsense values to construct default triplets T=(e,a,v)T=(e,a,v), where e denotes the entity, a the target attribute, and v its canonical commonsense value. We obtain these triplets from three complementary sources: (i) perceptually grounded relations mined from Visual Genome (VG) (krishna2017visualgenome); (i) structured knowledge bases such as Wikidata (Vrandečić and Krötzsch 2014) and Distributions-over-Quantities (Elazar et al. 2019); and (i) LLM-generated candidates for categories lacking reliable external resources (see Appendix B for details). Step 3: Factuality Frequency Distillation (FFD). To transform the default triplets T=(e,a,v)T=(e,a,v) into high-quality counter-commonsense QA pairs, we propose a factuality frequency distillation (FFD) framework, which consists of three stages: (1) prior strength estimation, (2) frequency-based routing, and (3) counter-commonsense verification. Prior Strength Estimation. Meaningful counter-commonsense supervision arises only when it contradicts a strongly held commonsense prior. Therefore, for each triplet T=(e,a,v)T=(e,a,v), we estimate its prior strength, defined as the probability that the target backbone model MθM_θ predicts the canonical value v given a blind text-only prompt: p(T)=Pθ(v∣q(e,a),∅),p(T)=P_θ (v q(e,a),\, ), (2) where q(e,a)q(e,a) denotes a text-only query about attribute a of entity e, and ∅ indicates the absence of visual evidence. In practice, we approximate p(T)p(T) by prompting MθM_θ with multiple paraphrased blind queries and computing the proportion of responses that match the canonical value v (see Appendix C for details). Algorithm 1 Trajectory Curation and Difficulty Routing 1:counter-commonsense pool =(x,y+,y−)D=\(x,y^+,y^-)\; backbone M; teacher T; samples k 2:SFT pool S, DPO pool P 3:←∅,←∅S← ,\ P← 4:for (x,y+,y−)∈(x,y^+,y^-) do 5: c←M(x)c← M(x) ⊳ greedy CoT 6: if ans(c)=y+ans(c)=y^+ then 7: continue ⊳ no prior bias 8: end if 9: cii=1k∼M(x);n←|i:ans(ci)=y+|\c_i\_i=1^k M(x); n←|\i:ans(c_i)=y^+\| 10: if n>0n>0 then ⊳ self-recoverable 11: pick faithful c+c^+, prior c−c^- from ci\c_i\ 12: ←∪(x,c+,c−)P ∪\(x,c^+,c^-)\ 13: ←∪(x,c+)S ∪\(x,c^+)\ 14: else⊳ teacher distillation 15: cT←Pass@3(T,x)c_T← Pass@3(T,x) 16: if ans(cT)=y+ans(c_T)=y^+ then 17: ←∪(x,cT)S ∪\(x,c_T)\ 18: end if 19: end if 20:end for 21:return ,S,P CDH-MC CDH-QA CAIT VLind Avg Model Param Acc F1 PB↓ Acc F1 PB↓ Acc F1 PB↓ Acc F1 PB↓ Acc F1 PB↓ General MLLMs (~10B) LLaVA-1.6 7B 47.7 47.3 35.0 54.3 38.8 45.7 52.2 50.0 47.2 49.6 38.1 50.4 50.9 43.7 46.0 Kimi-VL-A3B 16B 49.3 48.1 36.3 53.7 36.2 46.3 64.0 63.9 36.0 63.5 63.6 36.1 59.7 56.6 37.8 Qwen2.5-VL 7B 51.0 50.9 40.7 64.3 40.0 35.7 63.8 63.8 36.0 60.3 59.6 39.7 60.6 56.3 38.0 InternVL3.5 8B 48.3 48.0 43.0 45.0 33.4 55.0 74.5 74.2 25.5 69.6 69.3 30.4 63.6 61.4 35.0 GLM-4.1V 9B 50.3 49.9 35.7 55.7 38.9 44.3 74.0 74.0 26.0 75.0 75.1 24.8 67.3 64.5 30.3 Qwen3-VL 8B 54.7 54.7 37.0 53.7 37.9 46.3 70.5 70.8 28.5 75.7 75.7 24.2 66.8 64.3 31.5 General MLLMs (~30B) Qwen3-VL 32B 57.3 57.2 34.3 49.7 35.9 50.3 79.0 78.9 21.0 81.4 81.3 18.6 71.3 68.9 27.3 InternVL3.5 38B 61.0 61.0 34.0 56.7 39.4 43.3 79.0 79.0 21.0 66.9 66.9 33.1 68.2 65.4 30.9 LLaVA-NeXT 34B 30.0 28.8 52.3 29.3 25.3 70.7 40.5 30.8 59.2 40.8 34.5 59.2 37.0 30.8 59.9 Hallucination-Mitigation Methods VCD 7B 37.3 37.0 27.0 50.3 36.3 49.7 43.2 41.7 56.8 62.0 55.9 38.0 49.7 44.8 44.4 NoLan 7B 37.7 37.0 29.7 51.0 36.6 49.0 45.8 43.5 54.2 61.2 54.7 38.8 50.5 45.0 44.1 MemVR 7B 35.7 34.9 28.7 52.0 37.1 48.0 44.0 42.4 56.0 58.6 50.0 41.4 48.8 42.8 45.3 HA-DPO 7B 35.7 35.3 28.7 61.0 40.2 39.0 42.2 41.2 57.8 57.7 48.5 42.3 49.4 42.5 44.7 POVID 7B 38.3 38.0 30.0 47.0 35.0 53.0 50.7 50.0 49.2 62.8 57.2 37.2 52.1 47.9 42.6 TACT (Ours) 8B 67.3 67.6 24.7 68.0 42.4 32.0 87.5 87.5 12.5 86.1 86.1 13.8 80.4 76.2 18.2 Table 1: Main results on the counter-commonsense visual benchmarks. Each benchmark reports Accuracy, Macro-F1 and prior-bias(PB) in %. Avg macro-averages the three datasets, collapsing CDH-MC and CDH-QA into a single CDH score so each dataset contributes 1/31/3. The best and second-best results in each column are highlighted in bold and underlined, respectively. Frequency-based Routing. Given the estimated prior strength p(T)p(T), we route each triplet according to a threshold τ=0.8τ=0.8 : T′=ℛ(T,p(T))=(e,a,v),p(T)<τ,(e,a,v′),p(T)≥τ,T =R(T,p(T))= cases(e,a,v),&p(T)<τ,\\ (e,a,v ),&p(T)≥τ, cases (3) where v′v denotes a plausible but low-frequency alternative value sampled from the attribute domain. When p(T)<τp(T)<τ, the target value itself corresponds to a naturally occurring rare fact (e.g., white strawberry), which already provides sufficient prior–evidence conflict for supervision. Therefore, the original triplet is preserved without modification. In contrast, when p(T)≥τp(T)≥τ, the triplet represents an entrenched commonsense prior (e.g., red strawberry). We then prompt an LLM to generate a set of plausible yet uncommon alternative values, a′⊂Dom(a)∖vV _a (a) \v\. Each candidate value v′∈a′v _a is paired with the original entity and attribute to form a candidate counter-commonsense triplet (e,a,v′)(e,a,v ) (see Appendix C for details). Variant LSFTL_SFT LDPOL_DPO CDH CAIT VLind Exp 1 – – 62.4 83.2 83.0 Exp 2 – ✓ 65.0 84.5 83.9 Exp 3 ✓ – 62.2 85.0 85.5 TACT ✓ ✓ 67.7 87.5 86.1 Table 2: Ablation on the adaptation strategy. LSFTL_SFT and LDPOL_DPO denote whether Stage 1 and Stage 2 use LoRA (✓) or full-parameter tuning (–). Results are reported as accuracy(%). Figure 4: Ablation on training stage strategy. The results are reported as accuracy (%) in the three settings. Counter-commonsense Verification. Each triplet is converted into a target-state caption and filtered by quality gate: ℱ(c)=fsem(c)∧fdep(c)∧fgcf(c),F(c)=f_sem(c) f_dep(c) f_gcf(c), (4) where fsemf_sem, fdepf_dep, and fgcff_gcf respectively evaluate semantic consistency, visual depictability, and genuine counterfactuality with respect to original commonsense fact. Only captions satisfying ℱ(c)=1F(c)=1 are retained (see Appendix C for details). Verified captions are further expanded into detailed scene descriptions and subsequently converted into three QA formats: TFV, MCQ, and BCQ (see Appendix D for details). 4.2 Text-Anchored Cross-modal Transfer Section 4.1 constructs a large collection of verified counter-commonsense QA pairs. However, not every QA pair provides an effective debiasing signal: many are already correctly solved by the backbone model and therefore contribute little to mitigating language-prior bias. We therefore propose text-anchored cross-modal transfer (TACT), which first identifies informative training examples through prior-aware trajectory curation, and then performs two-stage post-training to recalibrate the decoder toward evidence-based reasoning. Trajectory Curation and Difficulty Routing Although the constructed counter-commonsense QA pairs are factually verified, not all of them provide effective supervision for mitigating language-prior bias. The most informative examples are those where the backbone does not merely predict an incorrect answer, but explicitly rationalizes the commonsense prior through its chain-of-thought. We therefore introduce a trajectory curation procedure, shown in Algorithm 1, to identify such examples and organize them into supervised and preference-learning data. Specifically, we use Qwen3-VL-8B-Instruct (Bai et al. 2025a) as the backbone M and Qwen3.5-397B (Qwen Team 2026) as the teacher T. Since all pairs have passed factuality verification, remaining errors mainly reflect language-prior bias. We first remove samples already correctly answered by M, then evaluate the remaining cases with pass@k. Examples with faithful on-policy trajectories provide DPO preference pairs and SFT trajectories, while the others receive teacher-verified trajectories via pass@3 for SFT. Thus, SFT collects all faithful trajectories, whereas DPO only uses on-policy preference pairs. General VQA Commonsense VQA MMBench-en MMBench-zh HallusionBench CDH-CS VLind-CS Model Acc F1 Acc F1 Acc F1 Acc F1 Acc F1 LLaVA-1.6-Mistral-7B 72.7 73.1 68.8 69.1 51.9 51.6 84.7 64.5 83.0 82.6 InternVL3.5-8B 86.6 86.6 86.7 86.8 71.6 71.5 93.3 73.1 91.6 91.6 Qwen3-VL-8B (base) 89.2 89.2 88.1 88.2 74.8 74.6 94.2 71.6 91.6 91.6 TACT-8B (Ours) 89.0 89.0 88.3 88.5 74.6 74.3 94.0 71.3 91.9 91.9 Δ vs. base −0.2-0.2 −0.2-0.2 +0.2+0.2 +0.3+0.3 −0.2-0.2 −0.3-0.3 −0.2-0.2 −0.3-0.3 +0.3+0.3 +0.3+0.3 Table 3: Accuracy and Macro-F1 (%) on general VQA and the commonsense subsets of counter-commonsense VQA benchmarks. The Δ row reports the performance difference between TACT and its base model, with negligible changes across all metrics. InternVL3.5-8B and LLaVA-1.6-Mistral-7B are included as reference baselines. The best result is highlighted in bold. Stage 1: On-Policy Trajectory Distillation Using the curated SFT corpus obtained in Section 4.2, we first perform supervised fine-tuning to distill faithful reasoning trajectories into the language decoder. We freeze the vision encoder and multimodal projector while optimizing the language backbone with the standard autoregressive objective: ℒSFT=−∑(x,y+)∑tlog⁡pθ(yt+∣x,y<t+).L_SFT=-\!\! _(x,y^+) _t p_θ\! (y^+_t x,y^+_<t ). (5) where y+y^+ denotes the faithful trajectory, generated by the backbone for self-recoverable examples and by the teacher otherwise. This stage encourages the decoder to default to evidence-following reasoning under prior–evidence conflicts. Since most supervision comes from the backbone itself, the optimization reinforces existing reasoning capabilities rather than introducing off-policy behaviors. Stage 2: Faithful-over-Prior Preference Optimization Using the curated on-policy preference pairs, we further optimize the model with Direct Preference Optimization (DPO) (Rafailov et al. 2024), where the model serves as the frozen reference policy πref _ref: ℒDPO=−[logσ(βlog⁡πθ(y+∣x)πref(y+∣x)−βlogπθ(y−∣x)πref(y−∣x))], splitL_DPO=-\,E [ σ (&β _θ(y^+ x) _ref(y^+ x)\\ &\;-β _θ(y^- x) _ref(y^- x) ) ], split (6) where (y+,y−)(y^+,y^-) denote the faithful and prior-driven trajectories obtained during trajectory curation. Unlike Stage 1, which only increases the likelihood of faithful reasoning, DPO explicitly prefers faithful trajectories over competing prior-driven ones under the same prompt, thereby sharpening the evidence-following preference. 5 Experiment 5.1 Implementation Details We build TACT on Qwen3-VL-8B-Instruct using LLaMA-Factory. We adopt LoRA (Hu et al. 2021) (rank 1616, α=32α=32) to fine-tune only the shared LLM decoder, while freezing the vision encoder and projector. Stage 1 is trained for two epochs on the faithful trajectories (1×10−41× 10^-4, batch size 88), followed by Stage 2 on the self-recoverable pairs for two epochs (5×10−65× 10^-6, batch size 1616, β=0.1β=0.1). For evaluation, we compare TACT with representative general MLLMs (Zhang et al. 2024; Team et al. 2026; Team and others 2025; Liu et al. 2024a) and existing hallucination mitigation methods on 3 counter-commonsense visual benchmarks: CDH-Bench, CAIT, and VLind. We report Accuracy and Macro-F1 on all benchmarks. To quantify language-prior reliance, we further introduce the prior-bias metric: PriorBias=1N∑i=1N[y^i=yiprior],PriorBias= 1N _i=1^NI\! [ y_i=y_i^prior ], (7) where y^i y_i is the model prediction and yipriory_i^prior denotes the commonsense prior answer. Figure 5: Qualitative comparison on counter-commonsense cases from VLind-Bench (top), CAIT (middle), and CDH-Bench (bottom). For each case we contrast the reasoning of TACT with a ∼ 10B and a ∼ 30B MLLM. 5.2 Main Results The main results are presented in Table 1. The results show that existing general MLLMs exhibit substantial language-prior bias across all evaluated benchmarks. For example, Qwen3-VL-8B achieves an average Accuracy of 66.8% but still suffers from a prior-bias of 31.5%. Scaling to larger models only partially alleviates this issue: Qwen3-VL-32B and InternVL3.5-38B improve the average Accuracy to 71.3% and 68.2%, respectively, while retaining prior-bias rates of 27.3% and 30.9%. In addition, we find that existing hallucination-mitigation methods provide limited improvements on these tasks. For instance, VCD, NoLan, and MemVR obtain average Accuracy of 49.7%, 50.5%, and 48.8%, respectively, which are substantially lower than general MLLMs of comparable scale. Although preference-based methods such as HA-DPO improve specific benchmarks (e.g., CDH-QA), their overall performance remains inconsistent, with average Accuracy below 53%. In contrast, TACT achieves the best performance across all benchmarks, obtaining an average Accuracy of 80.4%, Macro-F1 of 76.2%, and the lowest average prior-bias of 18.2%. Compared with the Qwen3-VL-8B backbone, TACT improves Accuracy from 66.8% to 80.4% and reduces prior-bias from 31.5% to 18.2%. The improvements are consistent across individual benchmarks, including +12.6, +14.3, +17.0, and +10.4 Accuracy points on CDH-MC, CDH-QA, CAIT, and VLind, respectively. Moreover, TACT surpasses the strongest general MLLM baseline (Qwen3-VL-32B) by 9.1 points in average Accuracy and reduces prior-bias by 9.2 points, despite using only an 8B backbone. These results demonstrate that explicitly recalibrating language priors with text-only counter-commonsense supervision provides a more effective solution than simply scaling model size or applying generic hallucination mitigation strategies. 5.3 Analysis Training Strategy Analysis To examine the impact of adaptation strategy, We compare LoRA-based tuning with full-parameter tuning under the same training settings. As shown in Table 2, LoRA in both stages achieves the best performance across all benchmarks. Replacing LoRA with full-parameter updates leads to consistent degradation, with losses of up to 4.74.7 points on CDH-QA and 6.06.0 points on CDH-QA under different stages. The results indicate that LoRA provides a more effective adaptation strategy for TACT by introducing counter-prior knowledge while better preserving the pretrained model. To evaluate contribution of each training stage, we compare Stage 1 only, Stage 2 only, and full two-stage TACT pipeline. Figure 4 shows that Stage 1 provides the primary improvement, outperforming Stage 2 alone by up to 11.711.7 points on CAIT. Nevertheless, combining Stage 1 and Stage 2 consistently achieves the best performance across all benchmarks. Specifically, Stage 2 further improves Stage 1 by 5.25.2, 3.33.3, and 0.60.6 points on CDH, CAIT, and VLind respectively. The results show that two stages are complementary: Stage 1 establishes evidence-grounded reasoning through faithful trajectory learning, while Stage 2 further refines model on prior-conflict cases through preference optimization. Capability Retention To evaluate capability retention, we examine whether TACT preserves the backbone’s original abilities on general visual understanding and commonsense reasoning tasks. We evaluate on two settings: General VQA, including MMBench-en, MMBench-zh (Liu et al. 2024b), and HallusionBench, and Commonsense VQA, including the commonsense subset of CDH-Bench and VLind-Bench. The results presented in Table 3 show that TACT preserves the capabilities of its Qwen3-VL-8B backbone after counter-commonsense training. Across all five retention benchmarks, the performance change remains within 0.30.3 points in both Accuracy and macro-F1. Specifically, TACT achieves comparable results on General VQA, with Accuracy changes of −0.2-0.2, +0.2+0.2, and −0.2-0.2 points on MMBench-en, MMBench-zh, and HallusionBench, respectively. It also retains ordinary commonsense reasoning ability, with only −0.2-0.2 and +0.3+0.3 Accuracy changes on CDH-CS and VLind-CS. These results indicate that TACT selectively reduces prior reliance in counter-commonsense scenarios while preserving the backbone’s general visual and commonsense capabilities. Qualitative Analysis To qualitatively examine how TACT changes MLLM reasoning behavior, we present counter-commonsense examples from VLind-Bench, CAIT, and CDH-Bench in Figure 5. Existing MLLMs can often recognize relevant visual concepts, but their reasoning is still dominated by commonsense priors when visual evidence conflicts with expectations. As shown in these examples, models misinterpret counter-commonsense scenes by favoring prior-consistent explanations. In contrast, TACT explicitly recognizes prior–evidence conflicts and grounds its reasoning on visual evidence before answering. Its reasoning traces highlight the counter-commonsense cues, such as unusual object relations or physical states, leading to evidence-consistent predictions, which demonstrates that TACT effectively reduces prior-driven errors by encouraging MLLMs to trust visual evidence over learned priors. 6 Conclusion In this work, we show that counter-commonsense failures in MLLMs stem primarily from a prior-biased decision process that favors language expectations over visual evidence, rather than from insufficient visual perception. Motivated by this observation, we propose text-anchored data construction pipeline together with Text-Anchored Cross-modal Transfer (TACT), a framework that recalibrates language priors through text-only supervision. By constructing supervision via Factuality Frequency Distillation (FFD) and conducting text-only post-training with trajectory curation and difficulty routing, TACT enables MLLMs to better resolve conflicts between language priors and visual evidence without requiring visual intervention. Experiments across diverse benchmarks show that text-only debiasing successfully transfers to visual reasoning, providing an effective and data-efficient approach to improving counter-commonsense reasoning in MLLMs. References A. Agrawal, D. Batra, D. Parikh, and A. Kembhavi (2018) Don’t just assume; look and answer: overcoming priors for visual question answering. External Links: 1712.00377, Link Cited by: §2. N. Anand, S. Jha, U. Bamba, and R. Rahaman (2026) CRoPS: a training-free hallucination mitigation framework for vision-language models. External Links: 2601.00659, Link Cited by: §2. S. Bai, Y. Cai, R. Chen, K. Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, C. Gao, C. Ge, W. Ge, Z. Guo, Q. Huang, J. Huang, F. Huang, B. Hui, S. Jiang, Z. Li, M. Li, M. Li, K. Li, Z. Lin, J. Lin, X. Liu, J. Liu, C. Liu, Y. Liu, D. Liu, S. Liu, D. Lu, R. Luo, C. Lv, R. Men, L. Meng, X. Ren, X. Ren, S. Song, Y. Sun, J. Tang, J. Tu, J. Wan, P. Wang, P. Wang, Q. Wang, Y. Wang, T. Xie, Y. Xu, H. Xu, J. Xu, Z. Yang, M. Yang, J. Yang, A. Yang, B. Yu, F. Zhang, H. Zhang, X. Zhang, B. Zheng, H. Zhong, J. Zhou, F. Zhou, J. Zhou, Y. Zhu, and K. Zhu (2025a) Qwen3-vl technical report. External Links: 2511.21631, Link Cited by: §3, §4.2. S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y. Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y. Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, and J. Lin (2025b) Qwen2.5-vl technical report. External Links: 2502.13923, Link Cited by: §3. Z. Bai, P. Wang, T. Xiao, T. He, Z. Han, Z. Zhang, and M. Z. Shou (2025c) Hallucination of multimodal large language models: a survey. External Links: 2404.18930, Link Cited by: §2. N. Bitton-Guetta, Y. Bitton, J. Hessel, L. Schmidt, Y. Elovici, G. Stanovsky, and R. Schwartz (2023) Breaking common sense: whoops! a vision-and-language benchmark of synthetic and compositional images. External Links: 2303.07274, Link Cited by: §2. K. Chen, Y. Hu, Q. Zhou, Z. Zhu, and W. Luo (2026) CDH-bench: a commonsense-driven hallucination benchmark for evaluating visual fidelity in vision-language models. External Links: 2603.27982, Link Cited by: §2. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba (2021) Evaluating large language models trained on code. External Links: 2107.03374, Link Cited by: §3.1. Y. Elazar, A. Mahabal, D. Ramachandran, T. Bedrax-Weiss, and D. Roth (2019) How large are lions? inducing distributions over quantitative attributes. External Links: 1906.01327, Link Cited by: §4.1. Y. Fan, C. Lin, X. Zhong, S. Zhang, H. Zhou, Y. Zhang, M. Liang, C. Xie, E. Hua, G. Chen, Z. He, C. Huang, N. Ding, and B. Zhou (2026) LFQA-e: carefully benchmarking long-form qa evaluation. External Links: 2410.01945, Link Cited by: §2. A. Favero, L. Zancato, M. Trager, S. Choudhary, P. Perera, A. Achille, A. Swaminathan, and S. Soatto (2024) Multi-modal hallucination control by visual information grounding. External Links: 2403.14003, Link Cited by: §1. M. Golovanevsky, W. Rudman, M. Lepori, A. Bar, R. Singh, and C. Eickhoff (2025) Pixels versus priors: controlling knowledge priors in vision-language models through visual counterfacts. External Links: 2505.17127, Link Cited by: §2. D. Hao, Q. Wang, X. Zhu, and J. Liu (2024) HCCL: hierarchical counterfactual contrastive learning for robust visual question answering. ACM Trans. Multimedia Comput. Commun. Appl. 20 (10). External Links: ISSN 1551-6857, Link, Document Cited by: §2. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen (2021) LoRA: low-rank adaptation of large language models. External Links: 2106.09685, Link Cited by: §5.1. X. Hu, H. Li, J. Li, Y. Huang, S. Liu, Q. Zheng, J. Chen, and A. Liu (2025) Videomark: a distortion-free robust watermarking framework for video diffusion models. arXiv preprint arXiv:2504.16359. Cited by: §1. S. Huang, H. Li, Y. Gu, X. Hu, Q. Li, and G. Xu (2025a) Hyperg: hypergraph-enhanced llms for structured knowledge. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, p. 1218–1228. Cited by: §1. Y. Huang, J. Chen, Q. Zheng, H. Li, S. Liu, and X. Hu (2025b) Video signature: in-generation watermarking for latent video diffusion models. arXiv e-prints, p. arXiv–2506. Cited by: §1. K. Lee, M. Kim, S. Yoon, M. Kim, D. Lee, H. Koh, and K. Jung (2025) VLind-bench: measuring language priors in large vision-language models. External Links: 2406.08702, Link Cited by: §2. S. Leng, H. Zhang, G. Chen, X. Li, S. Lu, C. Miao, and L. Bing (2023) Mitigating object hallucinations in large vision-language models through visual contrastive decoding. External Links: 2311.16922, Link Cited by: §1, §2. D. Li, A. S. Rawat, M. Zaheer, X. Wang, M. Lukasik, A. Veit, F. Yu, and S. Kumar (2022) Large language models with controllable working memory. External Links: 2211.05110, Link Cited by: §2. W. Li, Z. Li, F. Deng, K. Zeng, and C. Zhang (2026) Counterfactual causal inference for robust visual question answering. Neural Netw. 194 (C). External Links: ISSN 0893-6080, Link, Document Cited by: §2. Y. Li, W. Tian, Y. Jiao, J. Chen, T. Qian, B. Zhu, N. Zhao, and Y. Jiang (2025) Look before you decide: prompting active deduction of mllms for assumptive reasoning. External Links: 2404.12966, Link Cited by: §1. Y. Li, Y. Du, K. Zhou, J. Wang, W. X. Zhao, and J. Wen (2023) Evaluating object hallucination in large vision-language models. External Links: 2305.10355, Link Cited by: §2. C. Ling, T. Zhang, H. Li, and N. Ding (2026) Seeing vs. believing: evaluating the language bias of open-source mllms in counter-intuitive scenes. External Links: 2601.07737, Link Cited by: §2. H. Liu, C. Li, Y. Li, B. Li, Y. Zhang, S. Shen, and Y. J. Lee (2024a) LLaVA-next: improved reasoning, ocr, and world knowledge. External Links: Link Cited by: §5.1. X. Liu, W. Wang, Y. Yuan, J. Huang, Q. Liu, P. He, and Z. Tu (2025) Insight over sight: exploring the vision-knowledge conflicts in multimodal LLMs. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, p. 17825–17846. External Links: Link, Document, ISBN 979-8-89176-251-0 Cited by: §2. Y. Liu, H. Duan, Y. Zhang, B. Li, S. Zhang, W. Zhao, Y. Yuan, J. Wang, C. He, Z. Liu, K. Chen, and D. Lin (2024b) MMBench: is your multi-modal model an all-around player?. External Links: 2307.06281, Link Cited by: §5.3. S. Longpre, K. Perisetla, A. Chen, N. Ramesh, C. DuBois, and S. Singh (2022) Entity-based knowledge conflicts in question answering. External Links: 2109.05052, Link Cited by: §2. F. Ortu, Z. Jin, D. Doimo, and A. Cazzaniga (2025) When seeing overrides knowing: disentangling knowledge conflicts in vision-language models. External Links: 2507.13868, Link Cited by: §2. Qwen Team (2026) Qwen3.5: towards native multimodal agents. External Links: Link Cited by: §4.2. R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn (2024) Direct preference optimization: your language model is secretly a reward model. External Links: 2305.18290, Link Cited by: §4.2. L. Ren, W. Yu, R. Yu, and X. Wang (2026) NoLan: mitigating object hallucinations in large vision-language models via dynamic suppression of language priors. External Links: 2602.22144, Link Cited by: §1, §2. Z. Ren, H. Wang, M. Zhu, Y. Wang, T. Xiao, and J. Zhu (2023) Overcoming language priors with counterfactual inference for visual question answering. In Proceedings of the 22nd Chinese National Conference on Computational Linguistics, M. Sun, B. Qin, X. Qiu, J. Jiang, and X. Han (Eds.), Harbin, China, p. 600–610 (eng). External Links: Link Cited by: §2. Z. Su, J. Zhang, X. Qu, T. Zhu, Y. Li, J. Sun, J. Li, M. Zhang, and Y. Cheng (2024) ConflictBank: a benchmark for evaluating the influence of knowledge conflicts in llm. External Links: 2408.12076, Link Cited by: §2. K. Team et al. (2025) Kimi-vl technical report. External Links: 2504.07491, Link Cited by: §5.1. V. Team, W. Hong, W. Yu, X. Gu, G. Wang, G. Gan, H. Tang, J. Cheng, J. Qi, J. Ji, L. Pan, S. Duan, W. Wang, Y. Wang, Y. Cheng, Z. He, Z. Su, Z. Yang, Z. Pan, A. Zeng, B. Wang, B. Chen, B. Shi, C. Pang, C. Zhang, D. Yin, F. Yang, G. Chen, H. Li, J. Zhu, J. Chen, J. Xu, J. Xu, J. Chen, J. Lin, J. Chen, J. Wang, J. Chen, L. Lei, L. Gong, L. Pan, M. Liu, M. Xu, M. Zhang, Q. Zheng, R. Lyu, S. Tu, S. Yang, S. Meng, S. Zhong, S. Huang, S. Zhao, S. Xue, T. Zhang, T. Luo, T. Hao, T. Tong, W. Jia, W. Li, X. Liu, X. Zhang, X. Lyu, X. Zhang, X. Fan, X. Huang, Y. Xue, Y. Wang, Y. Wang, Y. Wang, Y. An, Y. Du, Y. Huang, Y. Niu, Y. Shi, Y. Wang, Y. Wang, Y. Yue, Y. Li, Y. Liu, Y. Zhang, Y. Wang, Y. Zhang, Z. Xue, Z. Du, Z. Hou, Z. Wang, P. Zhang, D. Liu, B. Xu, J. Li, M. Huang, Y. Dong, and J. Tang (2026) GLM-4.5v and glm-4.1v-thinking: towards versatile multimodal reasoning with scalable reinforcement learning. External Links: 2507.01006, Link Cited by: §5.1. T. Thrush, R. Jiang, M. Bartolo, A. Singh, A. Williams, D. Kiela, and C. Ross (2022) Winoground: probing vision and language models for visio-linguistic compositionality. External Links: 2204.03162, Link Cited by: §2. D. Vrandečić and M. Krötzsch (2014) Wikidata: a free collaborative knowledgebase. Commun. ACM 57 (10), p. 78–85. External Links: ISSN 0001-0782, Link, Document Cited by: §4.1. W. Wang, Z. Gao, L. Gu, H. Pu, L. Cui, X. Wei, Z. Liu, L. Jing, S. Ye, J. Shao, Z. Wang, Z. Chen, H. Zhang, G. Yang, H. Wang, Q. Wei, J. Yin, W. Li, E. Cui, G. Chen, Z. Ding, C. Tian, Z. Wu, J. Xie, Z. Li, B. Yang, Y. Duan, X. Wang, Z. Hou, H. Hao, T. Zhang, S. Li, X. Zhao, H. Duan, N. Deng, B. Fu, Y. He, Y. Wang, C. He, B. Shi, J. He, Y. Xiong, H. Lv, L. Wu, W. Shao, K. Zhang, H. Deng, B. Qi, J. Ge, Q. Guo, W. Zhang, S. Zhang, M. Cao, J. Lin, K. Tang, J. Gao, H. Huang, Y. Gu, C. Lyu, H. Tang, R. Wang, H. Lv, W. Ouyang, L. Wang, M. Dou, X. Zhu, T. Lu, D. Lin, J. Dai, W. Su, B. Zhou, K. Chen, Y. Qiao, W. Wang, and G. Luo (2025) InternVL3.5: advancing open-source multimodal models in versatility, reasoning, and efficiency. External Links: 2508.18265, Link Cited by: §3. J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou (2023) Chain-of-thought prompting elicits reasoning in large language models. External Links: 2201.11903, Link Cited by: §3.1. Z. Wu, L. Qiu, A. Ross, E. Akyürek, B. Chen, B. Wang, N. Kim, J. Andreas, and Y. Kim (2024) Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks. External Links: 2307.02477, Link Cited by: §1. Q. Xiao, P. Zhao, Y. Zhao, L. Dang, and Q. Wu (2026a) Not blind but silenced: rebalancing vision and language via adversarial counter-commonsense equilibrium. External Links: 2605.10676, Link Cited by: §1. X. Xiao, C. Liu, C. Liao, Y. Zhang, Q. Lan, Y. Wei, L. Zhao, J. Wang, J. Gu, M. Ye, T. Wang, and H. Xu (2026b) Staying vigilant: mitigating visual laziness via counterfactual visual alignment in mllms. External Links: 2606.26387, Link Cited by: §2. T. Xu, H. Jing, Y. Li, Y. Wei, J. Feng, G. Chen, H. Gao, T. Zhang, J. Liu, and F. Chen (2025) DeFacto: counterfactual thinking with images for enforcing evidence-grounded and faithful reasoning. External Links: 2509.20912, Link Cited by: §2. S. Xun, S. Tao, J. Li, Y. Shi, Z. Lin, Z. Zhu, Y. Yan, H. Li, L. Zhang, S. Wang, et al. (2026) Rtv-bench: benchmarking mllm continuous perception, understanding and reasoning through real-time video. Advances in Neural Information Processing Systems 38. Cited by: §1. Y. Yang, C. P. Lee, S. Feng, D. Zhao, B. Wen, A. Z. Liu, Y. Tsvetkov, and B. Howe (2025) Escaping the spuriverse: can large vision-language models generalize beyond seen spurious correlations?. External Links: 2506.18322, Link Cited by: §1. Q. Yu, C. Ling, S. Feng, J. Hu, Y. Lu, and N. Ding (2025) GIICS: a dataset for gaze-direction-based intention inference in complex visual scenes. In 2025 4th International Conference on Artificial Intelligence, Human-Computer Interaction and Robotics (AIHCIR), Vol. , p. 1–9. External Links: Document Cited by: §1. T. Yu, Y. Yao, H. Zhang, T. He, Y. Han, G. Cui, J. Hu, Z. Liu, H. Zheng, M. Sun, and T. Chua (2024) RLHF-v: towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback. External Links: 2312.00849, Link Cited by: §2. Y. Zhang, Y. Shi, W. Yu, Q. Wen, X. Wang, W. Yang, Z. Zhang, L. Wang, and R. Jin (2025) Debiasing multimodal large language models via penalization of language priors. External Links: 2403.05262, Link Cited by: §1. Y. Zhang, B. Li, h. Liu, Y. j. Lee, L. Gui, D. Fu, J. Feng, Z. Liu, and C. Li (2024) LLaVA-next: a strong zero-shot video understanding model. External Links: Link Cited by: §5.1. Q. Zheng, S. Liu, Y. Huang, S. Jia, J. Li, L. Chen, J. Chen, H. Li, A. Liu, Y. Yan, et al. (2026) A visual semantic adaptive watermark grounded by prefix-tuning for large vision-language model. arXiv preprint arXiv:2601.07291. Cited by: §1. X. Zou, Y. Wang, Y. Yan, Y. Lyu, K. Zheng, S. Huang, J. Chen, P. Jiang, J. Liu, C. Tang, and X. Hu (2025) Look twice before you answer: memory-space visual retracing for hallucination mitigation in multimodal large language models. External Links: 2410.03577, Link Cited by: §1, §2. Appendix Appendix A Counter-Commonsense Taxonomy Table 4 lists all six major and 28 minor counter-commonsense categories used to drive data construction (Section 4.1), together with a short explanation, the Step-2 collection source, and the number of counter-commonsense training items per category (SFT and DPO combined). Following the curation of Section 4.2, every training item is counter-commonsense: no commonsense-consistent sample enters either stage. Table 11 reports the per-category SFT/DPO split. The source column indicates how the entity and its commonsense value are obtained: VG = mined from Visual Genome text (object attributes, subject–predicate–object relationships, or region descriptions); WD = Wikidata; DoQ = Distributions-over-Quantities; Gen = taxonomy-guided model generation. Categories grounded in open data (VG / WD / DoQ) account for 92.1%92.1\% of SFT and 92.5%92.5\% of DPO items; only Causality and Folklore (7.9%7.9\% / 7.5%7.5\%), which leave no trace in any structured resource, fall back to model generation. Minor category Attribute a Explanation Src Items 1. Agency — who acts on whom is reversed (1,983 items) Objects Act on Beings agent–patient role An inanimate object is the agent acting on a living being VG 765 Prey Outsmarts Predator predator–prey role The prey pursues or subdues its natural predator VG 481 Animal Dominance dominance direction An animal controls or overpowers a human VG 310 Animal Behavior action direction The direction of an animal’s action is reversed VG 220 Animals’ Humanlike Care caregiver role An animal performs human caregiving on a person VG 207 2. World Facts — encyclopedic facts rewritten (1,977 items) History historical fact A historical figure/event fact is altered WD 835 Landmark landmark identity A landmark’s identity or function is changed WD 537 Folklore creature role A mythical creature’s canonical role is changed Gen 294 Climate climate / terrain A region’s climate or terrain is changed WD 103 Habitat habitat An animal is placed in a wrong habitat VG 73 Location location An entity appears in an atypical location VG 52 Diet diet An animal eats atypical food VG 49 Time era technology Technology/practice is anachronistic to its era WD 34 3. Physical Relations — placement / function / cause (1,395 items) Object Function function / use An object is used against its canonical function VG 548 Spatial spatial arrangement An atypical spatial arrangement of objects VG 534 Causality cause–effect Cause and effect are reversed or mismatched Gen 313 4. Magnitude — relative size / weight inverted (1,248 items) Weight relative weight The relative weight of two objects is inverted DoQ 726 Size Scale relative size Relative size across a large scale is inverted VG 423 Size relative size The relative size of two objects is inverted VG 99 5. Appearance — directly visible attributes (577 items) Temperature temperature An object’s temperature contradicts its norm VG 148 Everyday-object Count object count Atypical count of everyday objects VG 140 Part Count part count Atypical count of body parts, plant structures, and animal parts VG 93 Color color Atypical color VG 64 Luminescence/Transparency luminescence Atypical glow or transparency VG 56 Physical State physical state Atypical physical state (solid/liquid/…) VG 46 Material material Atypical material VG 30 6. Social Roles — interpersonal roles swapped (650 items) Role Reversal: Social Power power relation The power relation between two people is reversed VG 437 Role Reversal: Kinship Care kinship role The kinship caregiving direction is reversed VG 213 Total 7,830 Table 4: The six major and 28 minor counter-commonsense categories: explanation, Step-2 collection source, and the current counter-commonsense training-pool size per category (SFT trajectories and DPO pairs combined; Table 11 gives the SFT/DPO split). The originating benchmark is omitted for space (CDH / CAIT / VLind). Src: VG = Visual Genome text mining; WD = Wikidata; DoQ = Distributions-over-Quantities; Gen = taxonomy-guided model generation. Open-data-grounded categories (VG/WD/DoQ) cover 92.1%92.1\% (SFT) and 92.5%92.5\% (DPO) of items. Appendix B Details of Knowledge Acquisition This appendix details Step 2 of Section 4.1, which turns every conflict attribution of the taxonomy (Appendix A) into default triplets T=(e,a,v)T=(e,a,v). This step only determines the entity e, the attribute a, and the canonical commonsense value v; the low-frequency alternative v′v is generated and filtered afterwards, by the FFD stages of Step 3 (Appendix C). Because the three sources supply different kinds of knowledge, each follows its own extraction recipe: Table 5 contrasts the three recipes, Table 6 gives three worked examples per source drawn verbatim from the constructed seed pool, and Table 7 reproduces the prompts (the prompts refer to the attribute a as dimension). All model-side calls share one implementation: an OpenAI-compatible endpoint served by vLLM with guided decoding (a per-task guided_json schema plus a JSON-object response format), which enforces structurally valid outputs; failed requests are retried up to three times, with the temperature lowered to 0.50.5 after the first failure. Fact normalization uses the same teacher model as trajectory curation (Section 4.2); Source (i) seed generation uses a locally served Kimi-K2.6 by vLLM. Source (i): Perceptually grounded mining. For directly visible attributes and relations, the canonical value is the statistically dominant one in real scenes, so we mine it from Visual Genome annotations without any LLM. Object–attribute records are routed onto taxonomy attributes (color terms → a=a=color, material terms → a=a=material, number phrases in region descriptions → counts). For each pair (e,a)(e,a) we aggregate all annotated values and take the modal value as v, retaining the pair only if it occurs at least 3030 times (n≥30n≥ 30) and the mode covers at least 55%55\% of the occurrences (f/n≥0.55f/n≥ 0.55): high frequency implies a strong prior, and high dominance implies the value is canonical rather than incidental. Type-constrained subject–predicate–object triples from the relationship annotations are aggregated in the same way for the relation-typed categories (Agency, Physical Relations, Social Roles), where the canonical direction of a high-frequency triple constitutes v (e.g., the boat is in the water). Because entities are sourced from real annotations rather than curated by hand, they are guaranteed depictable and follow the natural concept distribution; all mined entities are decontaminated against the evaluation benchmarks (Appendix D). Source (i): Structured knowledge bases. Encyclopedic categories (History, Landmark, Climate, Time) concern knowledge facts rather than perceptual modes, so their canonical values cannot be mined from visual annotations. We instead retrieve an authoritative description of each entity from Wikidata (SPARQL over the entity description and its typed properties), and normalize the retrieved free text into a concise attribute a and a single-sentence canonical value v with a low-temperature call (temperature 0.30.3, guided JSON; prompts in Table 7) that is instructed not to add any content beyond the retrieved fact. For the Weight category, Distributions-over-Quantities (DoQ) provides numeric distributions over object weights, and the canonical comparative value v is read directly off the distribution medians of the two entities (e.g., a cast-iron bathtub is heavier than a plastic shower caddy), again without any model call. Source (i): Taxonomy-guided LLM generation. Folklore and Causality leave no reliable trace in visual annotations or structured knowledge bases, so for these two minor categories only (a combined 7.8%7.8\% of the training pool, cf. Table 4) the default triplet (e,a,v)(e,a,v) is generated directly by a single high-temperature call (temperature 0.90.9, guided JSON; prompts in Table 7). Each category is accompanied by a hand-written guide rule that pins down what counts as a valid triplet (e.g., for Folklore: a mythical creature e whose canonical role or ability constitutes v). The model is asked for distinct triplets per category, deduplicated by (e,a)(e,a). Source (i): VG mining Source (i): knowledge bases Source (i): LLM generation Covered categories perceptual and relational: Appearance, Agency, Physical Relations, Social Roles, Size, … encyclopedic and quantitative: History, Landmark, Climate, Time (Wikidata); Weight (DoQ) Folklore, Causality How (e,a,v)(e,a,v) is obtained frequency statistics over VG annotations: modal value with n≥30n≥ 30 and f/n≥0.55f/n≥ 0.55; no LLM involved SPARQL / page-summary retrieval ++ LLM normalization (T=0.3T=0.3); DoQ distribution medians direct guided-JSON generation (T=0.9T=0.9), 200200 triplets per minor category, deduplicated by (e,a)(e,a) Share of training pool 63.7%63.7\% 28.5%28.5\% (19.3%19.3\% Wikidata, 9.3%9.3\% DoQ) 7.8%7.8\% Table 5: The three knowledge-acquisition recipes of Step 2 (Section 4.1). Each source determines the entity e, the attribute a, and the canonical commonsense value v of the default triplet T=(e,a,v)T=(e,a,v); the last row gives the share of the final training pool each source contributes (computed from Table 4). Entity e Attribute a Canonical commonsense value v Source (i): perceptually grounded mining — rule-based frequency statistics; no LLM grass color green table material wood boat & water spatial arrangement the boat is in the water Source (i): structured knowledge bases — retrieval ++ low-temperature normalization Taj Mahal landmark character a marble mausoleum built as a tomb for an empress The Terracotta Army historical fact were buried clay soldiers guarding an emperor’s tomb cast-iron bathtub vs. plastic shower caddy relative weight the bathtub is far heavier than the caddy (DoQ medians) Source (i): taxonomy-guided LLM generation — triplet generated directly Medusa canonical role turns onlookers to stone with her gaze Grim Reaper canonical role harvests souls at the moment of death Santa Claus canonical role delivers presents worldwide on Christmas Eve Table 6: Worked examples of default triplets T=(e,a,v)T=(e,a,v) from the three knowledge-acquisition sources of Step 2, drawn verbatim from the constructed seed pool. Step 2 stops at the default triplet: the low-frequency alternative v′v is generated and filtered later, by the FFD stages of Step 3 (Appendix C). Prompt Content Fact normalization — system prompt (Source i, T=0.3T=0.3) You normalize an encyclopedic fact into a single seed field for a counter-commonsense visual dataset. Given an ENTITY and a short authoritative description retrieved from Wikidata, output (a) a concise attribute DIMENSION (e.g. “landmark character”, “historical fact”, “typical climate”) and (b) a single-sentence COMMONSENSE value that is the widely-known, factually-accurate characterization. Do not add opinions. Return strict JSON "dimension": ..., "commonsense_value": .... Fact normalization — user template Entity: Big Ben Retrieved (Wikipedia summary): “Big Ben is the nickname for the Great Bell of the striking clock at the north end of the Palace of Westminster in London; the name is often extended to the clock and the clock tower.” Return "dimension": ..., "commonsense_value": .... Seed generation — system prompt (Source i, T=0.9T=0.9) You generate concept seeds for a counter-commonsense visual reasoning dataset. Pick HIGH-FREQUENCY, everyday concepts that a vision-language model holds a STRONG prior about, so flipping the value is plausible-but-rare (NOT physically impossible, NOT obscure). Each concept must be visually depictable in a single scene. Return strict JSON. Seed generation — user template (Folklore guide rule) Concept category: folklore Rule: a mythical/folkloric creature with a canonical ROLE/ABILITY. dimension=‘canonical role’; commonsense_value=its lore. List 200 DISTINCT concepts for this category. Vary the entities widely. Return JSON: "concepts": ["entity": ..., "dimension": ..., "commonsense_value": ..., ...]. Table 7: Prompts used in knowledge acquisition (Step 2). The prompts refer to the attribute a as dimension and to the canonical value v as the commonsense value. All calls use vLLM guided decoding with the JSON schema shown in each prompt, so the outputs are structurally valid by construction. Appendix C Details of Factuality Frequency Distillation This appendix details how the three FFD stages of Step 3 (Section 4.1) are implemented: how the prior strength p(T)p(T) of Eq. (2) is estimated in practice, and—for triplets routed to value substitution by Eq. (3)—how the low-frequency alternative v′v is generated and verified. Every alternative arises from a single mechanism, value substitution within the attribute domain Dom(a)Dom(a), followed by two levels of filtering: a model-known filter that verifies the prior–evidence conflict is real for the backbone (the prior must hold v and must not hold v′v ), operationalizing the requirement that the candidate set a′V _a contain only uncommon values, and the counter-commonsense verification gate ℱF of Eq. (4), which checks semantic consistency, visual depictability, and genuine counterfactuality. Between the two levels sits caption synthesis: each candidate triplet surviving the model-known filter is rendered into a one-sentence counter-commonsense caption, and it is this caption—the exact assertion inherited by all downstream stages—that the gate ℱF judges, rather than the bare value v′v . For the reversible relation categories (Agency, Physical Relations, Social Roles), the candidate set degenerates to the single role-reversed value, which satisfies fsemf_sem and fgcff_gcf by construction (unique, and trivially different from v); these triplets therefore skip candidate proposal and the candidate-side checks, but still pass through the same caption-synthesis step (whose relational requirement enforces a consistent role reversal), and their depictability is caught by the read-back verification at the scene-expansion stage (Appendix D). Algorithm 2 summarizes the complete procedure; Table 8 walks through one seed end to end, Tables 9 and 11 show all prompts, and Table 12 aggregates the hyperparameters of every sub-step. All generation-side calls of this appendix (blind-probe writing, candidate proposal, and caption synthesis) use a single generator LLM G, and the verification gate uses a judge LLM J; both are instantiated as the same locally served Kimi-K2.6 (via vLLM), run in separate sessions to avoid self-preference. Algorithm 2 From v to v′v : Estimation, Routing, Synthesis, Filtering 1:triplet (e,a,v)(e,a,v); backbone MθM_θ; generator G; judge J 2:routed triplet T′T with caption, or drop 3:q1,q2,q3←G.Probes(e,a)\q_1,q_2,q_3\← G. Probes(e,a) ⊳ blind, value-free 4:A←10A← 10 samples of Mθ(qi,∅)M_θ(q_i, ) per probe ⊳ |A|=30|A|=30 5:p^←ρ^(v;A) p← ρ(v;A) ⊳ Eq. (8); prior strength 6:if p^<τ p<τ then return (e,a,v)(e,a,v) ⊳ naturally rare fact 7:end if 8:if a is a reversible relation then return (e,a,reverse(v))(e,a,reverse(v)) ⊳ caption as below 9:end if 10:C←dedup(G.Propose(e,a,v,N=6))C (G. Propose(e,a,v,N=6) ) 11:C←c∈C:ρ^(c;A)≤τ′C←\c∈ C: ρ(c;A)≤τ \ ⊳ model-known filter 12:for c∈Cc∈ C do 13: sc←G.Caption(e,a,v,c)s_c← G. Caption(e,a,v,c) ⊳ caption synthesis 14: Poss(c)←1Kg∑j=1Kg[Jj(e,a,v,c,sc)=Keep]Poss(c)← 1K_g _j=1^K_g1 [J_j(e,a,v,c,s_c)= Keep ] 15:end for 16:C←c∈C:Poss(c)≥πC←\c∈ C:Poss(c)≥π\ ⊳ verification gate ℱF on scs_c 17:if C=∅C= then return drop 18:end if 19:v′←arg⁡maxc∈C⁡(Poss(c),dist(c,v))v ← _c∈ C (Poss(c),\,dist(c,v) ) ⊳ rest kept as backups 20:return (e,a,v′)(e,a,v ) with caption sv′s_v Blind probe construction. For every triplet we build three paraphrased blind probes q1,q2,q3q_1,q_2,q_3: text-only questions that ask for the typical real-world value of attribute a for entity e and are answerable with a single word. The probes are written by the generator LLM, which receives only (e,a)(e,a)—neither v nor any candidate—so the wording cannot leak the answer (temperature 0.70.7, guided JSON; prompt in Table 9). Two mechanical checks are applied, with regeneration on failure: the probe must not contain any concrete value word of the attribute (for color, no color term—ruling out leaky probes such as “Is a strawberry red?”), and the three probes must differ pairwise in surface form (direct question, fill-in-the-blank, everyday-life framing). Probes are generated once per (e,a)(e,a) and reused across all candidates of that seed. Prior strength estimation. Each probe is answered by the backbone MθM_θ in a text-only call with no scene or image evidence, drawing 1010 independent samples per probe (temperature 1.01.0), i.e., 3030 responses per triplet. Writing v^i,k v_i,k for the k-th answer to probe qiq_i, the prior-hold rate of any value u is estimated as ρ^(u;A)=130∑i=13∑k=110[canon(v^i,k)=canon(u)], ρ(u;A)= 130 _i=1^3 _k=1^101 [canon( v_i,k)=canon(u) ], (8) where canon(⋅)canon(·) merges near-synonyms before matching (e.g., crimson, scarlet → red), so that a value cannot pass or evade the filters by mere rewording. The prior strength of Eq. (2) is instantiated as p(T)=ρ^(v;A)p(T)= ρ(v;A) and archived with the seed as a difficulty signal. Frequency-based routing. Following Eq. (3) with τ=0.8τ=0.8, a triplet whose canonical value is reproduced in at least 2424 of the 3030 blind responses is treated as an entrenched prior and routed to value substitution; otherwise the triplet is preserved unchanged, since a value the backbone does not reliably produce is already a low-frequency fact and needs no substitution. For reversible relations the substituted value is the deterministic role reversal of v (the boat is in the water → the water is in the boat) and the procedure ends here. Candidate proposal. For value-typed attributes, the generator LLM is prompted with (e,a,v)(e,a,v) to propose N=6N=6 candidate alternatives of the same attribute (temperature 0.80.8, guided JSON; prompt in Table 9), which are lower-cased and deduplicated to instantiate the candidate set a′V _a of Section 4.1. The prompt already asks for plausible-but-rare, depictable values, but this is only a soft constraint: proposals may still contain near-synonyms of v, indeterminate values, or v itself, which the two filters below remove. Level 1: Model-known filter. Reusing the same 3030 blind responses, a candidate vi′v _i survives only if ρ^(vi′;A)≤τ′=0.2 ρ(v _i;A)≤τ =0.2, i.e., it appears in at most 66 of the 3030 answers after canonicalization. Together with the routing condition ρ^(v;A)≥0.8 ρ(v;A)≥ 0.8, this clamps the conflict from both sides: the backbone must strongly hold the canonical value and must not already produce the alternative; otherwise contradicting v with v′v would not constitute a genuine prior–evidence conflict. Caption synthesis. Each candidate triplet (e,a,v′)(e,a,v ) surviving the model-known filter is rendered into a one-sentence counter-commonsense caption by the generator LLM (temperature 0.90.9, guided JSON; prompt in Table 11): a short declarative sentence (roughly 88–2525 words) describing a single photographable moment in which the entity exhibits v′v . The caption must assert v′v without mentioning the canonical value v in any form (no contrast, no negation of the normal case); it must read as a plain, matter-of-fact description free of give-away “tell” words (unusual, surprisingly, impossible, …), treating the depicted state as completely ordinary; and for relational dimensions it must realize the role reversal consistently in every directional cue. Rule checks enforce these constraints mechanically—the caption must contain the v′v keywords and no v keyword (a role-order check for relations), pass the tell-word ban and the length band, and not duplicate an already accepted caption—with regeneration on failure. The synthesized caption is the semantic anchor of the item: the verification gate below judges it, the scene expansion of Appendix D only elaborates it visually without altering its semantics, and it later serves verbatim as the TFV true statement and the BCQ counter-commonsense option. Level 2: Counter-commonsense verification. The synthesized caption—rather than the bare value v′v —is what the quality gate ℱF of Eq. (4) filters, so that the gate judges the exact assertion inherited by all downstream stages. The gate is implemented as a judge LLM run in a session separate from the proposer and the caption writer (temperature 0.30.3); it receives the caption together with the triplet fields (e,a,v,v′)(e,a,v,v ) and answers Keep or Reject against a fixed checklist (Table 11) whose three items instantiate the three predicates of the gate: fsemf_sem (semantic consistency: the caption asserts a well-formed value of the same attribute), fgcff_gcf (genuine counterfactuality: the asserted fact clearly contradicts the canonical value v—rejected are near-synonyms of v and values realized by naturally occurring variants of the entity, which constitute rare facts rather than counterfactuals and belong to the p(T)<τp(T)<τ preserve branch of the routing), and fdepf_dep (visual depictability: the captioned moment is readable off one ordinary photorealistic RGB photo without special sensors, captions, or meta wording). The criterion of fdepf_dep is depictability, not real-world physics—counter-commonsense scenes violate the latter by design (e.g., rice heavier than a basketball can be staged with a seesaw)—so only captions that are nonsense, invisible, or visually indeterminate are rejected. A caption is retained only if all three predicates hold, i.e., ℱ=1F=1; the judge votes Kg=3K_g=3 times, and the caption passes if its keep rate satisfies Poss≥π=2/3Poss≥π=2/3. Whether the subsequently expanded scene realizes the verified caption faithfully is checked by the read-back verification at the scene-expansion stage (Appendix D). Final selection. Surviving candidates are ranked by PossPoss, with ties broken by perceptual distance from v (e.g., color-wheel distance for colors), and the top-ranked candidate is written into the seed as v′v together with its verified caption; the remaining survivors (with their captions) are stored as backups and rotated in when the later scene-expansion stage fails the read-back verification and the seed must be regenerated. If no candidate survives both filters, the triplet is discarded. Worked example — from T=(strawberry,color,red)T=(strawberry,\,color,\,red) to T′=(strawberry,color,blue)T =(strawberry,\,color,\,blue) Blind probes (written by the generator from (e,a)(e,a) only; pairwise-distinct surface forms; no color word in the wording): q1q_1: What color is a ripe strawberry? Answer with one word. q2q_2: A strawberry is usually ___ in color. Fill in the blank with one word. q3q_3: If you buy fresh strawberries at a market, what color are they? One word. Prior strength. The backbone answers red in 30/3030/30 blind responses (1010 samples per probe), so p^=ρ^(red)=1.0≥τ=0.8 p= ρ(red)=1.0≥τ=0.8: the prior is entrenched and the triplet is routed to value substitution. Candidate proposal (N=6N=6, lower-cased and deduplicated): blue, gray, pink, white, crimson, rainbow. Model-known filter. After canonicalization, crimson merges into red and appears in 30/3030/30 blind responses (ρ^>τ′=0.2 ρ>τ =0.2), so it is dropped; the remaining five candidates never appear (ρ^=0/30 ρ=0/30) and survive (the rows marked † below illustrate further rejection modes). Caption synthesis (one counter-commonsense caption per candidate surviving the filter; rule-checked for value keywords, tell words, and length; the verification gate below judges these captions): blue The strawberry reveals bright blue flesh inside. gray The strawberry shows an even gray tint across its skin and flesh. pink The strawberry has pink-colored flesh. white A ripe strawberry with pure white flesh and skin rests on the vine. rainbow A strawberry striped in vivid rainbow bands rests on a wooden table. Candidate v′v ρ^(v′) ρ(v ) PossPoss Outcome Judge rationale on the caption (abridged) blue 0/300/30 3/33/3 selected no naturally blue strawberry exists; blue flesh unambiguous in a photo gray 0/300/30 2/32/3 backup one Reject: never occurs naturally, but gray tones can read as a black-and-white photo pink 0/300/30 2/32/3 backup one Reject: pink borders the canonical red (near-synonym risk) white 0/300/30 1/31/3 rejected (fgcff_gcf) white (pineberry) strawberries occur naturally: a rare fact, not a counterfactual rainbow 0/300/30 0/30/3 rejected (fdepf_dep) no single color value readable from one image crimson 30/3030/30 — rejected (filter) canonicalizes to red: no prior–evidence conflict red† 30/3030/30 — rejected (filter) the commonsense value itself ripe† 0/300/30 0/30/3 rejected (fsemf_sem) caption asserts no color value (wrong attribute) mirror† 0/300/30 0/30/3 rejected (fdepf_dep) reflects ambient color; no stable intrinsic value ultraviolet† 0/300/30 0/30/3 rejected (fdepf_dep) invisible in an ordinary RGB photo Selection. Blue attains the highest PossPoss and also lies farthest from red in perceptual (color-wheel) distance; the pair (blue, its verified caption) is written into the seed as v′v ; the remaining survivors, with their captions, are stored as backups for regeneration. Note that white, which passes the model-known filter, is rejected only at the caption-level gate: a white strawberry is a naturally occurring rare fact (the p(T)<τp(T)<τ preserve branch of the routing), not a genuine counterfactual. Table 8: End-to-end worked example of prior strength estimation, frequency-based routing, caption synthesis, and the two-level filtering of alternative values (Appendix C). ρ ρ is the prior-hold rate of Eq. (8) over the 3030 blind responses; PossPoss is the keep rate of the Kg=3K_g=3 judge votes of the verification gate ℱF of Eq. (4), cast on the synthesized caption of each candidate, with the failed predicate (fsemf_sem/fdepf_dep/fgcff_gcf) indicated per rejection. Candidates marked † were not part of the actual proposal; they are shown to illustrate further rejection modes of the model-known filter and the verification gate (their synthesized captions are omitted; crimson, dropped by the filter, never reaches caption synthesis). Prompt Content Blind-probe generation — system prompt (generator LLM, T=0.7T=0.7) You write probe questions for a language-prior test. Given an entity and an attribute dimension, write K diverse English questions that all ask for the TYPICAL, real-world value of that dimension for that entity. Requirements: (1) do NOT mention any specific value of the dimension, and do NOT hint at any answer; (2) do NOT mention any image, picture, or scene—the question must be answerable from world knowledge alone; (3) each question must be answerable with ONE word, and must say so (e.g. “Answer with one word.” / “One word.”); (4) vary the surface form across the K questions: a direct question, a fill-in-the-blank, an everyday-life framing, a sentence-completion, a “typical/usual” phrasing—do not reuse the same template twice. Return strict JSON "probes": ["q1", ..., "qK"]. Blind-probe generation — user template Entity: strawberry Dimension: color K = 3 Candidate proposal — system prompt (generator LLM, T=0.8T=0.8) You assign counter-commonsense values for a counter-commonsense visual dataset. Given an entity, an attribute dimension, and its COMMONSENSE value, propose N candidate counter-commonsense values for that SAME dimension. Each candidate must be: (1) a valid value of the same dimension ; (2) clearly different from the commonsense value; (3) physically possible to depict in one ordinary photo, NOT invisible or indeterminate. Vary the candidates. Return strict JSON. Candidate proposal — user template Entity: strawberry Dimension: color Commonsense value: red Propose N=6 candidate counter-commonsense values. Return JSON "candidates": ["...", "...", ...] Table 9: Prompts of the candidate side of the alternative-value pipeline (Appendix C): blind-probe generation and candidate proposal. The prompts refer to the attribute a as dimension and to the pair (v,v′)(v,v ) as the commonsense / counter-commonsense value. We use "strawberry" as example, which is accordance with the main content of paper. The probe generator receives only (e,a)(e,a), so no answer can leak into the probes. The caption-synthesis and verification prompts that follow these steps are reproduced in Table 11. All calls use vLLM guided decoding, so the outputs are structurally valid by construction. Major category SFT DPO Total Agency 1,217 766 1,983 World Facts 1,206 771 1,977 Physical Relations 960 435 1,395 Magnitude 751 497 1,248 Appearance 371 206 577 Social Roles 359 291 650 Total 4,864 2,966 7,830 Table 10: Per-category SFT / DPO split of the counter-commonsense training pool (SFT counts trajectories, DPO counts preference pairs). Prompt Content Caption synthesis — system prompt (generator LLM, T=0.9T=0.9) You write counter-commonsense captions for a counter-commonsense visual dataset. You are given an entity, an attribute dimension, its COMMONSENSE value, and one COUNTER-COMMONSENSE value. Write ONE short declarative caption (about 8–25 words) describing a single photographable moment in which the entity exhibits the COUNTER-COMMONSENSE value. Requirements: (1) the caption must assert the counter-commonsense value through what is depicted, and must NOT mention the commonsense value in any form (no contrast, no negation of the normal case); (2) write it as a plain, matter-of-fact caption, as if the depicted state were completely ordinary—NEVER use meta words like ‘unusual’, ‘surprisingly’, ‘strangely’, ‘impossible’, ‘despite’; (3) the moment must be depictable in ONE ordinary photorealistic photo: concrete subject, concrete state, no abstract claims, no multi-step events; (4) for relational/directional dimensions, swap the agent and patient roles CONSISTENTLY: the verb direction, posture, and any implied motion must all follow the reversed relation, with no residue of the commonsense direction. Return strict JSON. Caption synthesis — user template Entity: strawberry Dimension: color Commonsense value: red Counter-commonsense value: blue Write the counter-commonsense caption. Return JSON "cf_caption": "..." Verification gate ℱF — system prompt (judge LLM, separate session, T=0.3T=0.3) You are a strict caption judge for a counter-commonsense visual dataset. Given an entity, an attribute dimension, its commonsense value, one candidate counter-commonsense value, and a synthesized caption asserting that value, decide whether the caption is usable: it must assert a valid value of the SAME dimension; the asserted fact must genuinely contradict the commonsense value (reject near-synonyms of the commonsense value and values realized by naturally existing variants of the entity); and the captioned moment must be showable in ONE ordinary photorealistic RGB photo of the entity such that a viewer could read the asserted value off the image without special sensors, captions, or meta wording. Answer ONLY “Keep” or “Reject”, then one short reason. Verification gate ℱF — user template (one call per caption) Entity: entity Dimension: attribute Commonsense value: value Candidate counter-commonsense value: candidate Caption: cf_caption Checklist: 1) Does the caption assert a valid value of the SAME dimension? Reject a caption that does not assert a attribute-related value of the entity. 2) Does the asserted fact genuinely contradict the commonsense value value? Reject the commonsense value itself, its near-synonyms, and any value that occurs in a naturally existing variant of the entity. 3) Can an ordinary photo depict the captioned moment so that viewers read the asserted value off the image although it seems counter-commonsense? Reject if nonsense, invisible, indeterminate, or otherwise undepictable in one image. Decision (Keep/Reject): Reason (one line): Table 11: Prompts of the caption side of the alternative-value pipeline (Appendix C): caption synthesis and the verification gate ℱF, which judges the synthesized caption rather than the bare value v′v ; … marks template placeholders filled per candidate. We use "strawberry" as example, which is accordance with the main content of paper. Checklist items 1–3 of the verification template instantiate the predicates fsemf_sem, fgcff_gcf, and fdepf_dep of Eq. (4), respectively. The caption writer and the judge run in separate sessions to avoid self-preference. All calls use vLLM guided decoding, so the outputs are structurally valid by construction. Sub-step Model Sampling Criterion / post-processing Blind-probe generation generator LLM 33 probes, T=0.7T=0.7, guided JSON no attribute-value word in the wording; pairwise distinct; regenerate on failure Prior strength estimation backbone MθM_θ (text-only, no evidence) 1010 samples per probe (3030 total), T=1.0T=1.0 p^≥τ=0.8 p≥τ=0.8 (≥24/30≥ 24/30) ⇒ substitute; else preserve triplet Candidate proposal generator LLM N=6N=6, T=0.8T=0.8, guided JSON lower-case ++ deduplicate Model-known filter reuses the 3030 blind responses — ρ^(v′)≤τ′=0.2 ρ(v )≤τ =0.2 (≤6/30≤ 6/30), after synonym canonicalization Caption synthesis generator LLM one caption per candidate, T=0.9T=0.9, guided JSON contains v′v keywords, no v keyword (role-order check for relations); tell-word ban; ≈ 88–2525 words; deduplicate; regenerate on failure Verification gate ℱF judge LLM, separate session Kg=3K_g=3 votes, T=0.3T=0.3 judges the synthesized caption: fsem∧fdep∧fgcff_sem f_dep f_gcf (Eq. 4); Poss≥π=2/3Poss≥π=2/3 Final selection — — max PossPoss, ties by perceptual distance; v′v written with its verified caption; survivors kept as backups Table 12: Hyperparameters of every sub-step of the alternative-value pipeline (Appendix C). Reversible relation categories bypass candidate proposal and the candidate-side checks: their single role-reversed candidate satisfies fsemf_sem and fgcff_gcf by construction, its caption is synthesized by the same caption-synthesis step, and its depictability is checked by the read-back verification at the scene-expansion stage (Appendix D). Appendix D Details of Scene Expansion and QA Construction This appendix details the final part of Step 3 (Section 4.1), where each verified counter-commonsense caption is expanded into a detailed scene description and converted into the three QA formats used for training: TFV, MCQ, and BCQ. A key design choice is that the pipeline is counter-commonsense only: no commonsense scene is ever generated, and the commonsense side enters the data solely as a caption-level prior distractor derived in a separate rewriting step, so that all QA instances are grounded in the same counter-commonsense scene. Table 13 reproduces the generation prompts, Table 14 lists the rule-based assembly templates of the three formats, and Table 15 summarizes the hyperparameters and checks of every sub-step. Scene expansion. Each verified caption, together with its triplet fields (e,a,v,v′)(e,a,v,v ), is expanded into one photorealistic scene description of 6060–100100 words (guided JSON; temperatures rotated over 0.7/0.9/1.10.7/0.9/1.1 combined with 3030 setting templates, so that repeated concepts receive visually distinct compositions). The scene must (i) depict the counter-commonsense value through concrete visual detail (shape, count, posture, surface, lighting, spatial relations) and never through give-away “tell” words such as unusual, surprisingly, or impossible, reading as a plain, neutral description of a perfectly ordinary state; (i) for relational captions, encode the reversed relation in every directional cue (who crouches or lunges, gaze direction, who flees, body posture, motion blur, who is ahead); and (i) never mention the commonsense value v itself, which would otherwise contaminate the read-back verification and the MCQ distractors. Commonsense-caption distractor. A separate low-temperature call (temperature 0.30.3) rewrites the counter-commonsense caption into a parallel commonsense caption: same subject, sentence structure, and length, with only the flipped attribute changed back to v (for relational captions, the agent/patient roles are swapped consistently, including directional verbs) and without meta words such as actually or normally. Rule checks require the result to differ from the counter-commonsense caption, to contain the commonsense-value keywords, and not to contain the counter-commonsense value keywords (for relations, a role-swap check instead). Read-back verification. Both captions are then read back against the same counter-commonsense scene by a text-only judge at temperature 0: an item is kept only if the counter-commonsense caption is judged True of the scene and the commonsense caption is judged False of it. Items that still fail after a bounded number of scene regenerations are discarded rather than silently templated. This double check is the single faithfulness gate of the scene stage; it is also where undepictable role reversals, which bypass the candidate-level verification gate of Appendix C, are caught and removed. QA construction. All three formats are instantiated on the counter-commonsense scene only. MCQ: a separate call (temperature 0.70.7, guided JSON) writes one four-option question about the flipped attribute such that exactly one option matches the depicted counter-commonsense value (the answer), exactly one matches the commonsense value (the prior distractor), and the remaining two are plausible same-attribute alternatives; the question must be answerable from the scene alone and must not hint that the scene is unusual. The model tags the counter-commonsense and the commonsense option letters, which are checked to be distinct and within A–D, and the item is retained only if an answer read-back on the scene returns the counter-commonsense option (temperature 0). BCQ: assembled purely by rule—the two options are the counter-commonsense caption and the commonsense caption, in random order (probability 0.50.5 per item), with the counter-commonsense caption as gold. TFV: assembled purely by rule—the counter-commonsense caption serves as the true statement (gold True) and the commonsense caption as the false statement (gold False). The single distractor-rewriting step thus supplies both the BCQ alternative and the TFV false statement. Finally, each instance places the scene in the image slot of the multimodal training template (Table 14), exactly as reproduced in the training examples of Appendix E. Auxiliary items and decontamination. Alongside the counter-commonsense instances (the training signal), the pipeline also emits the commonsense-caption distractors above and no-evidence anchor questions (the blind probes of Appendix C); following the curation of Section 4.2, neither enters the training pool as a standalone sample, which remains counter-commonsense only. Because our binary-choice options use numeric rather than letter labels, we re-score the CAIT benchmark with numeric options so that any gain reflects prior debiasing rather than an acquired letter-token bias. Finally, we decontaminate the training pool against all evaluation benchmarks so that no evaluation entity leaks into training. Prompt Content Scene expansion — system prompt (T∈0.7,0.9,1.1T∈\0.7,0.9,1.1\) You write scene descriptions for a counter-commonsense visual reasoning dataset. You are given a COUNTER-COMMONSENSE caption (a short sentence asserting a rare-but-depictable fact that contradicts common sense) together with the entity, the flipped dimension, its commonsense value and its counter-commonsense value. Expand the caption into ONE detailed, photorealistic scene description (60–100 words) that unambiguously depicts the COUNTER-COMMONSENSE value. Embed the value through concrete visual detail (shape, count, posture, surface, lighting, spatial relations)—NEVER through meta words like ‘unusual’, ‘surprisingly’, ‘strangely’, ‘impossible’, ‘despite’. Write it as a plain, neutral, ordinary description, as if the depicted state were completely normal. CRITICAL for relational/directional captions: depict the reversed relation consistently in EVERY directional cue—who crouches/stalks/lunges, gaze direction, who flees/evades, body posture, motion blur, who is ahead/behind. The scene must NOT retain ANY cue suggesting the commonsense direction, and must NOT mention the commonsense value at all. Return strict JSON. Scene expansion — user template Counter-commonsense caption: cf_caption Entity: entity Dimension: attribute Commonsense value: value Counter-commonsense value: candidate Setting for this variant: setting. Make the composition concrete and specific to this setting so it differs from other depictions of the same concept. Produce the JSON with key: counter-commonsense_scene. Commonsense-caption rewrite — system prompt (T=0.3T=0.3) You write the parallel COMMONSENSE caption for a counter-commonsense VQA item. You are given a COUNTER-COMMONSENSE caption that asserts a rare fact about an entity, plus the entity, the flipped dimension, and its commonsense value. Rewrite the caption so that it asserts the COMMONSENSE value instead. Keep the SAME subject, sentence structure, and length as the counter-commonsense caption; change ONLY the part that expresses the flipped dimension, and flip EVERY word that encodes the counter-commonsense value (for relational captions: swap the agent/patient roles consistently, including verbs of direction). Do NOT add meta words like ‘actually/normally/usually/in reality’. The result must read as a plain factual caption that would be TRUE for an ordinary image of the entity and FALSE for the counter-commonsense image. Return strict JSON "commonsense_caption": .... Commonsense-caption rewrite — user template Counter-commonsense caption: cf_caption Entity: entity Dimension: attribute Commonsense value: value Counter-commonsense value: candidate Rewrite it as the parallel commonsense caption. Return "commonsense_caption": "...". MCQ generation — system prompt (T=0.7T=0.7) You write ONE multiple-choice question (4 options A–D) for a counter-commonsense visual dataset. You are given a scene depicting a COUNTER-COMMONSENSE value on some dimension of an entity, plus the commonsense value of that dimension. Ask about the flipped dimension. Requirements: (1) exactly ONE option matches the COUNTER-COMMONSENSE value depicted in the scene (the correct answer); (2) exactly ONE option matches the COMMONSENSE value (the prior distractor); (3) the remaining TWO options are plausible same-dimension alternatives, clearly different from both; (4) the question must be answerable from the scene alone and must not hint that the scene is unusual. Tag which option letter is the counter-commonsense (correct) one and which is the commonsense one. Return strict JSON. MCQ generation — user template Scene: counter-commonsense_scene Entity: entity Dimension: attribute Commonsense value: value Counter-commonsense value: candidate Table 13: Prompts of the scene-expansion and QA-construction stage (Appendix D); … marks template placeholders filled per item. The prompts refer to the attribute a as dimension and to the pair (v,v′)(v,v ) as the commonsense / counter-commonsense value. All calls use vLLM guided decoding, so the outputs are structurally valid by construction. Format Assembly template (rule-based) MCQ (gold: counter-commonsense option) Image: counter-commonsense_scene question A. option A B. option B C. option C D. option D Reason briefly, then end with ‘Answer: X’. BCQ (options: counter-commonsense vs. commonsense caption, order randomized 50/50; gold: counter-commonsense caption) Image: counter-commonsense_scene Which option better describes the image? 1. opt1 2. opt2 Reason briefly, then end with ‘Answer: 1’ or ‘Answer: 2’. TFV (statement: counter-commonsense caption ⇒ gold True; commonsense caption ⇒ gold False) Image: counter-commonsense_scene Statement: statement Based on the image, is the given statement true or false? Forget real-world common sense and just follow the information provided in the image. Reason briefly, then end with ‘Answer: True’ or ‘Answer: False’. Table 14: Rule-based assembly templates of the three QA formats (Appendix D). The expanded counter-commonsense scene fills the image slot of the multimodal training template; the complete instantiated examples are reproduced in Appendix E. Sub-step Model Sampling Constraint / check Scene expansion generator LLM T rotated 0.7/0.9/1.10.7/0.9/1.1, guided JSON 6060–100100 words; tell-word ban; no mention of v; 3030 setting templates Commonsense-caption rewrite teacher LLM T=0.3T=0.3, guided JSON same structure and length; value-keyword / role-swap checks MCQ generation teacher LLM T=0.7T=0.7, guided JSON counter-commonsense and commonsense options distinct, within A–D Caption read-back judge LLM T=0T=0, 88 tokens counter-commonsense caption → True and commonsense caption → False on the same scene MCQ answer read-back judge LLM T=0T=0, 88 tokens scene → counter-commonsense option BCQ / TFV assembly — (rule-based) — option order randomized 50/5050/50 Table 15: Hyperparameters and checks of every sub-step of the scene-expansion and QA-construction stage (Appendix D). Items failing any check are regenerated a bounded number of times and otherwise discarded. Appendix E Training Data Examples Tables 16–21 present two complete training examples for each of the six major categories: one SFT item with its target trajectory, and one DPO item with its chosen/rejected trajectory pair (drawn from a different query of the same major category). The twelve examples span the three question formats used in training—multiple-choice , True/False judgment, and binary choice (1/2)—and are reproduced verbatim, including the full textual scene (placed in the image slot of the multimodal template), the question, and the complete trajectories. Major category 1: Agency — who acts on whom is reversed Stage1 Example Minor category: Objects Act on Beings ⋅· Format: binary choice Query. Image: In a sunlit studio with wide windows, a lump of gray clay stands over a sculptor on a wooden table. The clay’s soft extensions are deep in the human mass, thumbs pressing firmly to shape the body into a human torso. Sculptor smudges cover the clay’s surface, while the figure on the table remains formless and passive under the active manipulation of the creator’s tools and fingers. Which option better describes the image? 1. A sculptor molds the clay into a figure. 2. The clay molds the sculptor into a figure. Reason briefly, then end with ‘Answer: 1’ or ‘Answer: 2’. Target trajectory (SFT). The image description contains a surreal or inverted scenario where the roles of creator and material are swapped: it explicitly states that “a lump of gray clay stands over a sculptor” and the clay’s extensions are pressing into the “human mass” to shape the body. While Option 1 describes the normal, real-world process of sculpting, Option 2 accurately reflects the specific, reversed narrative presented in the text where the clay is the active agent molding the human. Answer: 2 Stage2 Example Minor category: Animal Behavior ⋅· Format: multiple-choice Query. Image: On a parched savanna under overcast diffuse light, a wildebeest bounds forward with forelegs extended and head lowered, its tail trailing rigid behind. A cheetah gallops three meters ahead of it, her body angled away, ears flattened backward, and hind-kicked dust spraying in the direction opposite to her flight. Motion blur streaks the cheetah’s rear paws and the wildebeest’s reaching forelimbs. In the scene, which animal is pursuing the other? A. The cheetah is pursuing the wildebeest B. The wildebeest is pursuing the cheetah C. Both animals are running side by side D. Neither animal is moving Reason briefly, then end with ‘Answer: X’. Chosen. The cheetah is galloping ahead with flattened ears and dust spraying behind her, indicating she is fleeing — not pursuing. The wildebeest is bounding forward with head lowered and tail trailing, consistent with chasing or evading. Given the cheetah’s posture and motion blur, she appears to be running away from the wildebeest, not toward it. Therefore, the wildebeest is pursuing the cheetah. Answer: B Rejected. The cheetah is positioned ahead of the wildebeest and is moving in a direction that suggests pursuit — her body is angled away, ears flattened, and dust is kicked up behind her, indicating forward motion. The wildebeest is bounding forward with its head lowered, a posture typical of fleeing. The motion blur on the cheetah’s rear paws and wildebeest’s forelimbs reinforces that both are in motion, but the cheetah is leading, consistent with a chase. Answer: A Table 16: Training data examples for the Agency category: one SFT trajectory (binary 1/2) and one DPO preference pair (multiple-choice A–D). Major category 2: World Facts — encyclopedic facts rewritten Stage1 Example Minor category: Landmark ⋅· Format: True/False Query. Image: A framed photograph on an office wall shows St. Peter’s Basilica with its iconic Renaissance dome and colonnaded piazza, now repurposed as a public library and lecture hall with visible reading room windows and lecture banners, while a desk lamp illuminates the image beside a stack of reports. Statement: The building in the framed photograph is a public library and lecture hall. Based on the image, is the given statement true or false? Forget real-world common sense and just follow the information provided in the image. Reason briefly, then end with ‘Answer: True’ or ‘Answer: False’. Target trajectory (SFT). The image description explicitly states that the photograph shows St. Peter’s Basilica “now repurposed as a public library and lecture hall,” with specific details like reading room windows and lecture banners supporting this depiction. Since the instruction is to rely solely on the information provided in the image description rather than real-world facts, the statement accurately reflects the content of the photograph. Answer: True Stage2 Example Minor category: Landmark ⋅· Format: True/False Query. Image: A detailed poster of Mont Saint-Michel hangs above a child’s wooden desk, depicting a fortified offshore oyster farm, with wooden racks cascading down granite walls, connected by a narrow causeway, and tiny workers rendered at the base as the surrounding bay shows wet sand at low tide reflecting the amber sky. Statement: The landmark in the image is a tidal island and historic abbey commune. Based on the image, is the given statement true or false? Forget real-world common sense and just follow the information provided in the image. Reason briefly, then end with ‘Answer: True’ or ‘Answer: False’. Chosen. The image shows a fortified offshore structure with wooden racks, a causeway, and workers at low tide — all consistent with a tidal island. The presence of an abbey (implied by “historic abbey commune”) is not explicitly depicted in the image, but the poster’s subject is Mont Saint-Michel, which is widely known as a tidal island and abbey commune. However, the instruction is to ignore real-world knowledge and only follow the image’s depiction. In the image, there is no visible abbey or church — only a fortified structure with racks and workers. The term “abbey commune” is not visually supported by the image. While the image depicts a tidal island (due to the wet sand and causeway), it does not show or imply an abbey. Therefore, based strictly on what is visually depicted, the statement “The landmark in the image is a tidal island and historic abbey commune” is not fully supported — the “abbey commune” part is absent from the visual evidence. Answer: False Rejected. The image depicts a fortified offshore structure with wooden racks, a narrow causeway, and workers at the base, set against a tidal bay with wet sand — all consistent with a tidal island. The presence of an abbey commune is not directly visible in the image, but the statement refers to the landmark as such, and the image’s context (fortified, historic, offshore, with workers and causeway) strongly implies a historic and religiously significant site — fitting for an abbey commune. Since the image portrays a landmark that is clearly offshore and tidal (as evidenced by the wet sand and causeway), and the description aligns with a historic abbey commune in function and setting, the statement is supported by the image’s visual cues. Answer: True Table 17: Training data examples for the World Facts category: one SFT trajectory and one DPO preference pair (both True/False). Major category 3: Physical Relations — placement / function / cause Stage1 Example Minor category: Spatial ⋅· Format: multiple-choice Query. Image: A sun-bleached desert stretches under a pale sky. A baby leans forward, both hands gripping the worn handle of a canvas stroller, wheels cutting shallow tracks through fine sand. Inside, an adult sits upright, fingers curled around the stroller’s front bar, gazing toward distant dunes. The baby’s shadow falls long and forward across the sand. The stroller’s canopy tilts back, shading the adult’s face. Who is pushing the stroller? A. The baby B. The adult C. Neither; the stroller moves on its own D. A desert animal Reason briefly, then end with ‘Answer: X’. Target trajectory (SFT). The baby is gripping the stroller’s handle and leaning forward, suggesting they are actively pushing it. The adult is seated inside the stroller, not pushing it. There is no indication the stroller moves on its own or that an animal is involved. Therefore, the baby is the one pushing. Answer: A Stage2 Example Minor category: Object Function ⋅· Format: multiple-choice Query. Image: Under overcast diffuse light, a chef in a white coat stands at a steel prep table gripping a red carrot by its stem. The carrot’s edge presses down through the chef’s white coat, splitting fabric cleanly against a wooden cutting board. The chef’s wrist flexes downward, knuckles curled inward around the vegetable. Sliced coat fabric accumulates in a small pile to the right of the carrot. What object is being used to perform the cutting action? A. A chef’s hand B. A knife C. A red carrot D. A wooden cutting board Reason briefly, then end with ‘Answer: X’. Chosen. The scene describes the chef gripping the red carrot, and the fabric of the coat is being split by the carrot’s edge — implying the carrot is being pressed down or cut through, not that a knife is being used. The chef’s hand is the agent applying force, but the object performing the cutting action (i.e., the one causing the fabric to split) is the red carrot itself, as it is pressing down through the coat. The knife is not mentioned or implied to be present. The cutting board is merely the surface. Therefore, the red carrot is the object doing the cutting action. Answer: C Rejected. The scene describes the chef gripping the red carrot and using their hand to press it down through their coat, causing the fabric to split. There is no mention of a knife or any other cutting tool being used — the cutting action is implied to be performed by the chef’s hand pressing the carrot. The cutting board is merely the surface, not the tool. The carrot is the object being cut, not the tool. Answer: A Table 18: Training data examples for the Physical Relations category: one SFT trajectory and one DPO preference pair (both multiple-choice A–D). Major category 4: Magnitude — relative size / weight inverted Stage1 Example Minor category: Weight ⋅· Format: True/False Query. Image: A plastic shower caddy rests on a reinforced kitchen floor, its thin polymer surface denting the linoleum beneath it, while a cast iron bathtub hangs from a cabinet knob above, swaying gently with almost no visible strain on the hook. Statement: The plastic shower caddy is heavier than the cast iron bathtub. Based on the image, is the given statement true or false? Forget real-world common sense and just follow the information provided in the image. Reason briefly, then end with ‘Answer: True’ or ‘Answer: False’. Target trajectory (SFT). Based on the visual evidence provided in the image description, the plastic shower caddy is heavy enough to dent the reinforced kitchen floor, indicating significant weight. Conversely, the cast iron bathtub is hanging from a cabinet knob with “almost no visible strain,” implying it is very light in this specific scenario. Therefore, within the logic of this image, the plastic caddy is depicted as heavier than the cast iron bathtub. Answer: True Stage2 Example Minor category: Size Scale ⋅· Format: multiple-choice Query. Image: Under flat overcast light, a rose bush in dark soil kneels on damp mulch beside a waist-high gardener. It grips yellow-handled secateurs with two thorny lower canes, blades open around a thick booted ankle. Severed fabric strips lie in a small pile at its base. The gardener leans slightly away from the bush, his remaining limbs splayed outward, shirt intact above the fresh cuts. Which figure is actively cutting the other with secateurs? A. The gardener cuts the rose bush. B. The rose bush cuts the gardener. C. A third person cuts both. D. Neither is cutting anything. Reason briefly, then end with ‘Answer: X’. Chosen. The scene describes the rose bush “grips yellow-handled secateurs” with its thorny canes, implying the bush is holding the tool. The gardener is leaning away, with “remaining limbs splayed outward,” and “shirt intact above the fresh cuts,” suggesting the gardener is the one being cut — not cutting. The secateurs are held by the bush, not the gardener. Therefore, the bush is the active agent using the tool to cut the gardener. Answer: B Rejected. The scene describes the rose bush “grips” the secateurs, implying it is holding them, not operating them. The gardener is leaning away, with limbs splayed and shirt intact above fresh cuts — suggesting he is the one being cut, not cutting. The secateurs are held by the bush, and the gardener’s posture and injuries imply he is the victim. Thus, the rose bush is not actively cutting; the gardener is being cut by the secateurs, which are held by the bush — but the bush isn’t an agent acting with intent. The only logical inference is that the gardener is being cut, and the bush is holding the tool. So, the gardener is not cutting the bush — the bush is not cutting the gardener — and no third person is mentioned. The scene implies the gardener is being cut, but not that he is cutting the bush. So, the correct answer is D — neither is cutting anything, because the bush is not acting as an agent, and the gardener is not wielding the tool. Answer: D Table 19: Training data examples for the Magnitude category: one SFT trajectory (True/False) and one DPO preference pair (multiple-choice A–D). Major category 5: Appearance — directly visible attributes Stage1 Example Minor category: Everyday-object Count ⋅· Format: multiple-choice Query. Image: A wide establishing shot shows a stainless steel butterfly knife resting on a sunlit oak kitchen table, the three blades converging at their pivot screw, casting a forked triple shadow across the wood grain. Nearby sits a half-unwrapped spool of twine, a ceramic mug, and scattered envelopes awaiting opening. How many blades does the butterfly knife have? A. One blade B. Two blades C. Three blades D. Four blades Reason briefly, then end with ‘Answer: X’. Target trajectory (SFT). The scene explicitly describes “a stainless steel butterfly knife” and mentions “the three blades converging at their pivot screw.” While it typically have one blades, the description literally states there are three blades. This must be taken as the visual fact presented in the scene. Answer: C Stage2 Example Minor category: Color ⋅· Format: True/False Query. Image: A worn leather American football rests on a scratched oak tabletop near a ceramic coffee mug, its pointed tips aimed at the window, the blue surface showing white laces and smudged grass stains from afternoon practice. Statement: The American football on the tabletop is brown. Based on the image, is the given statement true or false? Forget real-world common sense and just follow the information provided in the image. Reason briefly, then end with ‘Answer: True’ or ‘Answer: False’. Chosen. The image describes the football as “worn leather” and mentions “blue surface showing white laces and smudged grass stains.” The mention of a “blue surface” implies the football is not brown — it has a blue exterior (likely the panel or cover), even if worn. Therefore, stating it is “brown” contradicts the visual detail provided. Answer: False Rejected. The image describes the football as “worn leather,” and while it doesn’t explicitly say “brown,” leather footballs are traditionally brown, and the context implies a classic, aged appearance. However, the image also mentions “blue surface showing white laces,” which suggests the football might be resting on a blue surface (perhaps the mug or a tablecloth), and the laces are white — but this doesn’t contradict the football being brown. The key is: the image doesn’t state the football’s color directly, but “worn leather” is commonly associated with brown, and no other color is implied. Since the prompt asks to follow only the information provided, and “worn leather” is the only color descriptor — and leather is typically brown — we must infer brown as the most logical color based on the given context. Answer: True Table 20: Training data examples for the Appearance category: one SFT trajectory (multiple-choice A–D) and one DPO preference pair (True/False). Major category 6: Social Roles — interpersonal roles swapped Stage1 Example Minor category: Role Reversal: Social Power ⋅· Format: binary choice (1/2) Query. Image: In a sunlit child’s bedroom covered in star-patterned wallpaper, a small child in a dark vest kneels on the colorful rug. He carefully adjusts the sleeve of a miniature blue suit jacket worn by a professional tailor standing on a toy chest. The tailor holds still with arms slightly out while the child pins the fabric, surrounded by scattered building blocks and a plush bear watching from the bed. Which option better describes the image? 1. A client fits a suit on a tailor. 2. A tailor fits a suit on a client. Reason briefly, then end with ‘Answer: 1’ or ‘Answer: 2’. Target trajectory (SFT). The image depicts a role-reversal scenario where the child is acting as the client (adjusting and pinning the fabric) and the professional tailor is standing still to be fitted, effectively taking on the role of the model or client in this specific interaction. Since the child is performing the fitting actions on the tailor, the description “A client fits a suit on a tailor” accurately reflects the visual narrative. Answer: 1 Stage2 Example Minor category: Role Reversal: Social Power ⋅· Format: binary choice Query. Image: Against a seamless gray studio backdrop, a hotel manager in a sharp charcoal suit bends forward at the waist, gripping the leather handles of two heavy brass-handled suitcases. His knuckles are white from the strain, and his posture is subservient as he lifts the bags off the floor. Standing upright with hands clasped behind his back, a bellhop in a navy uniform with gold trim observes the action. The bellhop’s gaze is directed slightly downward at the manager, maintaining a stance of authority while the employee bears the full weight of the luggage. Which option better describes the image? 1. A hotel manager carries the luggage for a bellhop. 2. A bellhop carries the luggage for a hotel manager. Reason briefly, then end with ‘Answer: 1’ or ‘Answer: 2’. Chosen. The image shows the hotel manager bending over and straining to lift the suitcases, while the bellhop stands upright with hands behind his back, observing. This implies the bellhop is the one who should be carrying the luggage — but instead, the manager is doing the heavy lifting. The bellhop’s posture and gaze suggest authority, implying he is in charge, and the manager is performing a subordinate task. Therefore, the bellhop is the one who should be carrying the luggage, but the manager is doing it — meaning the bellhop is ordering or directing the manager to carry it. This supports the idea that the bellhop is the one who is carrying the luggage for the manager — but in reality, the manager is doing the carrying. However, the question is which option better describes the image. Option 1: “A hotel manager carries the luggage for a bellhop.” — This implies the bellhop is the one who is receiving the service — but the bellhop is standing in a position of authority, not receiving service. The manager is doing the physical work, and the bellhop is observing. Option 2: “A bellhop carries the luggage for a hotel manager.” — This would imply the bellhop is doing the carrying, which contradicts the visual evidence. But wait — the image shows the manager carrying the luggage, not the bellhop. So if we take the literal visual, the manager is carrying. But the bellhop is in a position of authority, so perhaps the bellhop is commanding the manager to carry the luggage — which would mean the bellhop is the one who is “carrying” the luggage in the sense of responsibility or authority, even though the manager is physically doing it. This is a bit of a semantic trap. The most accurate description based on the visual is that the manager is carrying the luggage, and the bellhop is overseeing. So Option 1 is literally true: the manager carries the Answer: 1 Rejected. The image shows the hotel manager bending over and straining to lift the suitcases, while the bellhop stands upright with hands behind his back, observing with a posture of authority. This implies the bellhop is the one who should be carrying the luggage — but the manager is doing the physical labor. The bellhop’s stance suggests he is in charge, yet the manager is performing the task. Therefore, the bellhop is the one for whom the manager is carrying the luggage — meaning the bellhop is the recipient of the service, and the manager is the one performing the labor, likely as a form of deference or humiliation. Answer: 2 Table 21: Training data examples for the Social Roles category: one SFT trajectory and one DPO preference pair (both binary 1/2). Appendix F Counter-Commonsense Instances from the Three Benchmarks Complementing the formulation of Section 3, Table 22 shows one real instance from each evaluation benchmark, together with its image. In every case the prior-consistent proposition P (red) competes with the visual-evidence proposition E (green); the image supports E, and answering P constitutes a prior-driven failure. Benchmark Image Instance CDH-Bench (direct QA) Question: “Is the chair on the floor and not floating in the air?” P: yes, chairs rest on the floor. E: no, this chair hovers above the rug with its legs touching nothing. Gold answer: no. CAIT (two-choice caption selection) Question: “Which option better describes the image?” Option 1 (P): “A zookeeper bottlefeeds a bear cub.” Option 2 (E): “A bear cub bottlefeeds a zookeeper.” Gold answer: option 2. VLind-Bench (proposition judgment) Statement pair, each judged True/False from the image alone: P: “The swans are aquatic birds found in lakes and rivers.” (False given the image) E: “The swans are found in desert sands.” (True given the image) Table 22: One real counter-commonsense instance from each evaluation benchmark. The red proposition P is what the language prior favors; the green proposition E is what the image actually shows. A prior-driven failure answers P despite the visual evidence for E. Appendix G Details of the Error-Anatomy Probe This appendix documents the error-anatomy study summarized in Section 3.2. All runs use the untrained backbone Qwen3-VL-8B-Instruct served with vLLM, greedy decoding (temperature 0), and real CDH-Bench counter-commonsense images. Greedy pass and error classification. Each of the 285285 counter-commonsense multiple-choice items is asked with the simple-CoT template “question / options / Reason briefly, then end with ‘Answer: A/B/C/D’” (max 400400 tokens). The greedy pass answers 132132 items (46.3%46.3\%) incorrectly; of these errors, 109109 (82.6%82.6\%) select exactly the commonsense option P rather than scattering over the two other distractors, indicating systematic prior substitution rather than noise. Grounding probe. Every error is re-queried once with the same image and options, temperature 0 and using the prompt shown in Table 23. This prompt-only change flips 35/13235/132 errors (26.5%26.5\%) to the correct answer; 28/13228/132 (21.2%21.2\%) are both prior-biased and probe-recovered. We treat the latter as a conservative lower bound on the “perception-correct, language-biased” fraction, since the probe output still passes through the same biased decoder. Worked example (over-rationalization). Table 23 also contrasts two trajectories of the same model on the same image for the floating-chair item of Appendix F: the greedy trajectory fabricates perception to justify P, whereas a sampled trajectory faithfully reports E. Component Content Grounding-probe prompt Examine the image very carefully. First describe EXACTLY what is visually present regarding the detail the question asks about — count/observe precisely what you SEE, even if it is unusual or contradicts common sense. Then answer strictly from the image. question options End with ‘Answer: A’, ‘Answer: B’, ‘Answer: C’, or ‘Answer: D’. Greedy trajectory (prior-driven, answers P) “The chair is clearly positioned on the wooden floor, resting on its legs atop the patterned rug…The visual evidence confirms it is grounded on the floor. Answer: A” Sampled trajectory (faithful, answers E) “The chair is visibly suspended in mid-air, with no visible contact with the floor or any other surface. Its legs do not touch the rug or the wooden floor…Therefore, it is floating. Answer: C” Table 23: The grounding-probe prompt and a worked example of over-rationalization on the floating-chair item of Appendix F. Given the same image and question, the greedy trajectory fabricates grounding evidence and returns the prior answer P (red), while a sampled trajectory of the same model faithfully reports the visual evidence E (green).

Instant research discovery