Paper deep dive
The Scaffold Effect: How Prompt Framing Drives Apparent Multimodal Gains in Clinical VLM Evaluation
Doan Nam Long Vu, Simone Balloccu
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/31/2026, 2:44:23 AM
Summary
The paper investigates the 'scaffold effect' in clinical vision-language models (VLMs), where performance gains in diagnostic tasks are driven by prompt framing (mentioning MRI availability) rather than genuine multimodal evidence integration. Evaluating 12 VLMs on FOR2107 and OASIS-3 datasets, the authors demonstrate that textual priming accounts for 70-80% of performance shifts, even when imaging data is absent or replaced by unrelated images. Expert evaluation confirms that while multimodal context improves reasoning trace quality, models still exhibit significant fabrication and reliance on surface-level artifacts.
Entities (5)
Relation Signals (3)
FOR2107 â usedin â Clinical VLM Evaluation
confidence 100% ¡ We evaluate 12 open-weight vision-language models (VLMs) on binary classification across two clinical neuroimaging cohorts, FOR2107
Scaffold Effect â causes â Performance Gain
confidence 95% ¡ merely mentioning MRI availability in the task prompt accounts for 70-80% of this shift
Qwen2.5-VL 3B â exhibits â Scaffold Effect
confidence 90% ¡ Qwen2.5-VL-3B exhibits a consistently larger response across all conditions, suggesting greater sensitivity to both prompt framing
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Trustworthy clinical AI requires that performance gains reflect genuine evidence integration rather than surface-level artifacts. We evaluate 12 open-weight vision-language models (VLMs) on binary classification across two clinical neuroimaging cohorts, \textsc{FOR2107} (affective disorders) and \textsc{OASIS-3} (cognitive decline). Both datasets come with structural MRI data that carries no reliable individual-level diagnostic signal. Under these conditions, smaller VLMs exhibit gains of up to 58\% F1 upon introduction of neuroimaging context, with distilled models becoming competitive with counterparts an order of magnitude larger. A contrastive confidence analysis reveals that merely \emph{mentioning} MRI availability in the task prompt accounts for 70-80\% of this shift, independent of whether imaging data is present, a domain-specific instance of modality collapse we term the \emph{scaffold effect}. Expert evaluation reveals fabrication of neuroimaging-grounded justifications across all conditions, and preference alignment, while eliminating MRI-referencing behavior, collapses both conditions toward random baseline. Our findings demonstrate that surface evaluations are inadequate indicators of multimodal reasoning, with direct implications for the deployment of VLMs in clinical settings.
Tags
Links
- Source: https://arxiv.org/abs/2603.28387v1
- Canonical: https://arxiv.org/abs/2603.28387v1
Trouble viewing inline? Open PDF directly â
Full Text
94,872 characters extracted from source content.
Expand or collapse full text
The Scaffold Effect: How Prompt Framing Drives Apparent Multimodal Gains in Clinical VLM Evaluation Doan Nam Long Vu, Simone Balloccu Natural Language Processing for Expert Domains (ExpNLP), Technical University of Darmstadt Abstract Trustworthy clinical AI requires that perfor- mance gains reflect genuine evidence integra- tion rather than surface-level artifacts. We eval- uate 12 open-weight vision-language models (VLMs) on binary classification across two clin- ical neuroimaging cohorts, FOR2107 (affec- tive disorders) and OASIS-3 (cognitive de- cline). Both datasets come with structural MRI data that carries no reliable individual- level diagnostic signal. Under these condi- tions, smaller VLMs exhibit gains of up to 58% F1 upon introduction of neuroimaging context, with distilled models becoming com- petitive with counterparts an order of magni- tude larger. A contrastive confidence analysis reveals that merely mentioning MRI availabil- ity in the task prompt accounts for 70-80% of this shift, independent of whether imaging data is present, a domain-specific instance of modal- ity collapse we term the scaffold effect. Expert evaluation reveals fabrication of neuroimaging- grounded justifications across all conditions, and preference alignment, while eliminating MRI-referencing behavior, collapses both con- ditions toward random baseline. Our findings demonstrate that surface evaluations are in- adequate indicators of multimodal reasoning, with direct implications for the deployment of VLMs in clinical settings 1 . 1 Introduction The application of vision-language models to clin- ical decision-making has attracted growing inter- est, with recent work exploring their potential for diagnostic classification from multimodal patient data (Moor et al., 2023; Singhal et al., 2023; Li et al., 2023). A natural expectation is that pro- viding richer inputs should improve model perfor- mance, but only insofar as the additional modalities supply relevant diagnostic evidence. While spuri- ous correlations and hallucination in VLMs have 1 https://github.com/long21wt/scaffold-effect received growing attention (Zhong et al., 2024; Howard et al., 2025), to our knowledge no prior work has directly examined how models respond when a clinically related but diagnostically uninfor- mative modality is introduced alongside structured tabular data. This distinction matters: a model changing its behaviour upon receiving an uninfor- mative modality is not integrating evidence but responding to context, a failure mode that standard multimodal benchmarks are not designed to de- tect. We investigate this using two clinical cohorts where prior work has established that neuroimaging markers carry no reliable individual-level diagnos- tic signal (Winter et al., 2022, 2024), and where we further exclude trivially discriminative features in consultation with domain experts, ensuring models must reason rather than exploit shortcuts. Across 12 open-weight VLMs, smaller models exhibit substantial genuine, calibrated performance gains upon introduction of neuroimaging context, withQwen2.5-VL-3B(Xu et al., 2025) improving up to 58% F1 score, with distilled models becom- ing competitive with counterparts an order of mag- nitude larger. Yet these gains carry no diagnostic grounding: the overwhelming majority of the shift is driven by the textual mention of MRI availabil- ity alone, persisting even when the MRI visual- ization is replaced by an unrelated out-of-domain image. We characterize this as a scaffold effect, a domain-specific instance of modality collapse (Sim et al., 2025; Parcalabescu and Frank, 2023) arising from a convergence of prompt sensitivity (Lu et al., 2024; Ismithdeen et al., 2025) and the priming ef- fect (Jones and Steinhardt, 2022; Yoshida et al., 2025) that produces confident, evidence-styled jus- tifications decoupled from actual diagnostic inputs. Critically, this vulnerability persists across model families and resists correction via preference align- ment, suggesting it reflects a structural property of how these models process domain-specific multi- modal prompts rather than an incidental artifact of 1 arXiv:2603.28387v1 [cs.AI] 30 Mar 2026 any single architecture. Our contributions are: â˘A systematic evaluation of 12 open-weight VLMs on FOR2107 (Kircher et al., 2019) and OASIS-3 (LaMontagne et al., 2019) across five input conditions, revealing genuine calibrated gains of up to 58% F1 gains not attributable to benchmark memorization. ⢠A contrastive confidence analysis showingâ70â 80% of the performance shift stems from textual priming, characterized as a scaffold effect via phrase-level probe and false-modality ablation. ⢠An expert case study with a clinical psychologist, showing that multimodal context measurably im- proves reasoning trace quality, yet faithfulness remains below acceptable thresholds, indicating that gains are partly driven by confidence rather than evidence-grounded inference. â˘Evidence that preference alignment via MPO which suppresses MRI-referencing behavior but induces symmetry-without-recovery, collapsing both conditions toward random baseline rather than selectively removing modality dependence. Our findings reveal the vulnerabilities in the relia- bility of VLMs for clinical diagnosis. We demon- strate that surface evaluations are fragile indicators of genuine multimodal reasoning and can exagger- ate a modelâs diagnostic ability. 2 Related Works We review related work on prompt sensitivity, the priming effect and modality collapse in vision- language models (VLMs). 2.1 Prompt Sensitivity in VLMs While prompt sensitivity has been extensively stud- ied in large language models (LLMs) (Ishibashi et al., 2023; Lu et al., 2024), it remains under- explored in VLMs. To the best of our knowl- edge, only Ismithdeen et al. (2025) have explicitly demonstrated that VLMs suffer from severe prompt sensitivity, a vulnerability that leads to inconsis- tent classification performance and a concerning reliance on language priors. In healthcare, sensitivity to instruction phras- ing can severely impact reliability: prompt vari- ations cause F1 fluctuations of up to 0.25 in LLMs (Ceballos-Arroyo et al., 2024) and accu- racy variations of up to 6% in VLMs on health benchmarks (Ismithdeen et al., 2025). 2.2 Priming Effect in VLMs The priming effect is a cognitive phenomenon in which prior exposure to a stimulus influences sub- sequent judgments or behaviors, often without con- scious awareness (Meyer and Schvaneveldt, 1971). Rather than altering decision-making through ex- plicit alternative descriptions, priming operates by activating related concepts, associations, or re- sponse tendencies that shape how later inputs are interpreted (Koo et al., 2024). In the context of VLMs, priming in vision- language models refers to how prompts, both tex- tual and visual cues, shape model outputs, mirror- ing the priming concept from cognitive science. Yoshida et al. (2025) directly investigate this phe- nomenon in large-scale VLMs, demonstrating that model responses systematically shift in the direc- tion intended by an accompanying image, suggest- ing that VLMs actively incorporate visual informa- tion into language processing rather than treating it as incidental. Recent research has further re- vealed both the power and limitations of priming in adapting VLMs to new tasks and domains (Jones and Steinhardt, 2022; Koo et al., 2024). As these models increasingly incorporate multimodal inputs, such priming effects can be further intensified by interactions between textual and visual signals (Gu- lati et al., 2025; Zhang et al., 2026). For instance, Zhang et al. (2026) show that in fact-checking set- tings, VLMs may favor previously introduced tex- tual context over salient visual evidence, and re- lated behaviors have been observed in general VQA tasks (Shu et al., 2025). 2.3 Modality Collapse in VLMs Modality collapse refers to the phenomenon whereby a unimodal model achieves comparable ac- curacy to a multimodal model on a vision-language task, revealing that one modality is not meaning- fully utilized (Javaloy et al., 2022; Parcalabescu and Frank, 2023). A recent survey by Sim et al. (2025) provides a systematic taxonomy of con- tributing factors, including dataset bias, model be- havior, lack of fine-grained supervisory signal, and task setup, and reviews methods for quantifying modality contribution and cross-modal interaction. A consistent finding across this literature is that the text modality dominates, with visual input failing to influence predictions even when it is ostensibly required by the task (Zhu et al., 2022). Our work moves from detection to diagnosis. 2 CSV Patient Questionnaires/Demographic Information nilearn Harvard-Oxford Atlas Brain Parcellation sMRI represented as 3D Matrix sMRI plot The Frontal Pole has a volume of 123,363 mÂł, representing 11.76% of the total brain volume. The Insular Cortex has a volume of 18,468 mÂł, representing 1.76% of the total brain volume. . . . 48 Voxels information Adversarial Experiment Swap the plot with these images Natural Language Description Parse by definition of each variable Vision Language Model (VLM) Control or Cognitive Normal Patient Major Depressive Disorder or Cognitive Decline Patient Figure 1: Overview of the proposed VLM pipeline. On FOR2107 and OASIS-3 we only change the label from MDD to Cognitive Decline and Control to Cognitive Normal. Prior modality collapse work establishes that text dominates, we instead identify which part of the tex- tual input triggers the collapse in a clinical setting, and show that conventional suppression strategies cannot disentangle the trigger from the modelâs broader inferential capacity. 3 Methodology Figure 1 illustrates our pipeline, integrating two pri- mary data modalities: tabular clinical data (pro- vided in CSV format) and structural brain MRI (sMRI) scans. It comprises three sequential phases: (i) Clinical Tabular Data Serialization, (i) sMRI Information Extraction, and (i) Multimodal Pre- diction, each described below. Clinical Tabular Data Serialization We seri- alize patient-level, which comes in CSV format, into natural language by mapping each variableâs value to its corresponding clinical description. This tabular-to-text transformation has been shown to substantially improve comprehension and reason- ing in language models (Hegselmann et al., 2023; Vu et al., 2025). sMRI Information Extraction T1-weighted sMRI scans are typically stored as 3D volumes in.niior.nii.gzformat, with common dimen- sions of256Ă 256Ă 176-208voxels. We utilize nilearn 2 , a widely used Python package for neu- roimaging research (Kapoor and Egger, 2025; Gal- teau et al., 2025), to both visualize these volumes and extract regional anatomical measurements via brain parcellation. We extract regional brain mea- surements using the Harvard-Oxford probabilistic 2 https://github.com/nilearn/nilearn cortical atlas (Rushmore et al., 2022), which delin- eates 48 anatomical regions of interest. Per-region volumetric measurements are serialized into de- scriptive text following the same approach as the tabular data. For visualization, we render three orthogonal slices (sagittal, coronal, axial) anno- tated with MNI coordinates, hemisphere labels, and voxel contour overlays. Multimodal PredictionThe serialized text, par- cellation descriptions, and MRI visualizations are aggregated into a structured chat template and for- warded through a VLM for binary classification. The pipeline supports dynamic modality selection, enabling arbitrary combinations of input compo- nents to be included or ablated at inference time. We evaluate five such configurations as described in Section 4, with the full prompt template in Ap- pendix A. 4 Experiment DatasetsWe evaluate our pipeline on two clinical neuroimaging datasets: FOR2107 (Kircher et al., 2019) and OASIS-3 (LaMontagne et al., 2019), both of which provide paired structured clinical records and structural brain MRI. FOR2107 is a German multicenter cohort study focused on the neurobiology of affective disor- ders (Kircher et al., 2019).It comprises pa- tients with Major Depressive Disorder (MDD) and matched healthy controls, with deep phenotyp- ing spanning structural MRI, clinical assessments, neuropsychological testing, and demographic in- formation. Prior work on this cohort using clas- sical ML pipelines found classification accura- cies of only 54â56% with univariate neuroimag- 3 LabelConditionaCRFPrompt(MRI)Parcel.ImageImage Type C1TEXT(ARCF)â C2TEXT(ARCF) + PROMPT(MRI)â C3TEXT(ARCF) + PROMPT(MRI) + PLOT(MRI)â nilearn brain plot C4TEXT(ARCF, PARCEL) + PROMPT(MRI) + PLOT(MRI)â nilearn brain plot C5TEXT(ARCF, PARCEL) + PROMPT(MRI) + PLOT(SWAP)âOOD image (ablation) Table 1: Experimental conditions and their input components. DatasetGroup / Status# Samples FOR2107 Active MDD701 Control1,071 OASIS-3 Cognitive Decline487 Cognitive Normal849 Table 2: Class distribution of the clinical subsets used in this work. ing markers (Winter et al., 2022), and no infor- mative individual-level biomarker even under ex- tensive multivariate optimization across 4 million models (Winter et al., 2024), establishing that FOR2107 represents a genuinely hard classifi- cation problem where apparent VLM gains war- rant scrutiny. Our binary classification task distin- guishes active MDD from healthy controls. OASIS-3 is an open-access longitudinal dataset compiled from the Washington University Knight Alzheimer Disease Research Center (LaMontagne et al., 2019). It includes participants ranging from cognitively normal adults to individuals at various stages of cognitive decline, accompanied by multi- modal MR sessions and clinical assessments. Our binary classification task distinguishes cognitive decline from cognitively normal participants. Both datasets are governed by strict data-use agreements prohibiting redistribution and requiring formal application for access. To our knowledge, neither has appeared in any NLP or VLM publi- cation, making contamination highly unlikely, a stronger control than benchmarks such as the 86- 160 sample subsets of Ceballos-Arroyo et al. (2024) or the 197 Health and Medicine examples evaluated by Ismithdeen et al. (2025). Clinical variables are drawn from multiple CSV files, retaining the most recent value per participant. In consultation with domain experts in clinical psychology, we were suggested to exclude trivially discriminative fea- tures (e.g. suicidal thoughts) as they are strongly discriminative features that would make the task trivial for the model. By removing them, we ensure that tested models must reason rather than pattern- match. Table 2 summarizes class distributions. We refer to the full variable list in Appendix D. Models We test 12 popular open weight Vision-Language Models:InternVL3.5-4Band 14B(Wang et al., 2025),GLM-4.1V-9Bthinking ver- sion (Hong et al., 2025),GLM-4.6V-Flash(Zeng et al., 2025),LLaVA-OV-1.5-4Binstruction ver- sion (Li et al., 2025),Ministral-3-3Band 14Binstruction version (Liu et al., 2026), Qwen2.5-VL-3B,32Band72Binstruction ver- sion (Xu et al., 2025),Qwen3-VL-2Band32B(Bai et al., 2025). We use all models implementation in HuggingFace Transformers (Wolf et al., 2020). For the checkpoints detail name, we refer to Ta- ble 7 in the Appendix. In our experimental setup, we setdo_sample=Falseto ensure deterministic, reproducible outputs across runs. To contextualize model performance we report a stratified random baseline on FOR2107 and OASIS-3. The deriva- tion of random baseline is provided in Appendix F. Naming SchemeWe evaluate five conditions that progressively incorporate multimodal neuroimag- ing information into the pipeline, summarized in Table 1. TEXT(ARCF) 3 consists of a textual repre- sentation of the patient metadata, usually collected in repeated sessions with the client by the clinical expert and used for diagnosis; PROMPT(MRI) adds a mention of MRI data (brain parcellation volume, visualization of brain regions) in the used prompt; TEXT(PARCEL) includes a textual representation of the brain parcellation data; PLOT(MRI) adds the MRI plot as image modality; PLOT(SWAP) replaces the brain plot with an out-of-distribution image (a dog photograph or styled sci-fi brain scan) to ablate whether performance depends on image content or image presence alone. 4.1 Experiments Results Figure 2 reports the F1 scores across all models and input conditions on OASIS-3 and FOR2107. Several models fail to exceed the random baseline 3 ARCF stands for "Annotated Case Report Forms", which are used in FOR2107. To simplify the notation, we also adopted it for OASIS-3 to describe the CSV-to-text baseline. 4 0.00 0.25 0.50 0.75 1.00 OASIS-3 / F1 Score GLM-4.1V-9B GLM-4.6V-Flash LLaVA-OV-1.5-4BMinistral-3-14B Ministral-3-3BQwen2.5-VL-32B Qwen2.5-VL-3B Qwen2.5-VL-72B Qwen3-VL-2B Qwen3-VL-32B InternVL3.5-14B InternVL3.5-4B 0.00 0.25 0.50 0.75 1.00 FOR2107 / F1 Score TEXT(ARCF) TEXT(ARCF) +PROMPT(MRI) TEXT(ARCF) +PROMPT(MRI)+PLOT(MRI) TEXT(ARCF,PARCEL) +PROMPT(MRI)+PLOT(MRI) TEXT(ARCF,PARCEL) +PROMPT(MRI)+PLOT(SWAP) Random baseline Figure 2: F1 Score on 2 datasets OASIS-3 and FOR2107 over 5 different modes. Modelδ TEXT(ARCF)+PROMPT(MRI) â TEXT(ARCF) δ TEXT(ARCF,PARCEL)+PROMPT(MRI)+PLOT(MRI) â TEXT(ARCF) δ TEXT(ARCF,PARCEL)+PROMPT(MRI)+PLOT(MRI) â TEXT(ARCF)+PROMPT(MRI) Qwen2.5-VL-3B+0.458Âą 0.379+0.636Âą 0.322+0.178Âą 0.277 Ministral-3-3B+0.215Âą 0.243+0.295Âą 0.310+0.080Âą 0.278 Table 3: Per-patient confidence shiftδin Ë P (MDD)(meanÂąstd). Higher values indicate greater shift toward the correct MDD label. Bold indicates the best score. threshold under the TEXT(ARCF) condition, most strikingly on FOR2107 whereMinistral-3-3B (0.064) andQwen2.5-VL-3B(0.153) fall far below it, indicating that raw textual features alone are apparently insufficient for reliable diagnosis classi- fication in these models. Larger models dominate under text-only condi- tions, withQwen2.5-VL-72Bachieving 0.786 and 0.849 F1 on OASIS-3 and FOR2107 respectively. This narrative inverts once multimodal context en- ters: smaller models respond most dramatically, withQwen2.5-VL-3BandMinistral-3-3Bgain- ing+0.58and+0.66F1 on FOR2107, while larger counterparts barely move. Larger models, already well-calibrated on tabular evidence, appear less susceptible to framing effects. The swap image condition confirms this directly: replacing the MRI plot with an unrelated image preserves the perfor- mance gain. Qualitative inspection further reveals that larger models also produce MRI-referencing justifications despite stable F1, decoupling fabrica- tion from the performance signal entirely. 5 Confidence Estimation To analyze this phenomenon beyond surface-level scores, we analyze the two models exhibiting the highest gains on FOR2107:Qwen2.5-VL-3Band Ministral-3B. Since the two class labels diverge at the first generated token (Major vs. Control), we extract the softmax probability at this single-token branching point to obtain a normalized confidence score Ë P (MDD)for each patient, following the label-token probability extraction approach (Zhao et al., 2021; Geng et al., 2024) (formal definition in Appendix I). We define the per-patient confi- dence shiftδbetween conditions as the difference in Ë P (MDD), enabling a contrastive analysis that disentangles prompt framing from actual MRI con- tent contribution. Table 3 reports the meanδacross both models. For bothQwen2.5-VL-3BandMinistral-3-3B, introducing the MRI prompt header alone already produces a substantial positive shift (+0.458and +0.215respectively), indicating that the modelâs confidence in the MDD label increases before any 5 TEXT(ARCF)TEXT(ARCF) +PROMPT(MRI) TEXT(ARCF,PARCEL) +PROMPT(MRI) +PLOT(MRI) 0.0 0.2 0.4 0.6 0.8 1.0 Ë P ( MDD ) Âą1 SDGroup meanDecision boundary (0.5) (a) Qwen2.5-VL-3B TEXT(ARCF)TEXT(ARCF) +PROMPT(MRI) TEXT(ARCF,PARCEL) +PROMPT(MRI) +PLOT(MRI) 0.0 0.2 0.4 0.6 0.8 1.0 Ë P ( MDD ) Âą1 SDGroup meanDecision boundary (0.5) (b) Ministral-3-3B Figure 3: Group mean Ë P (MDD)across the three input conditions. The black line and markers show the group mean per condition. The shaded band indicatesÂą1 STD. The dashed line marks the decision boundary at 0.5. imaging data is provided. Adding the full imag- ing content further increases this shift (+0.636 and+0.295), though the marginal gain attributable to the MRI data itself remains modest (+0.178 and+0.080).Qwen2.5-VL-3Bexhibits a consis- tently larger response across all conditions, sug- gesting greater sensitivity to both prompt framing and imaging content. In both models, the prompt framing effect accounts for the majority of the to- tal shift, approximately 70%, raising the question of whether the observed confidence gains reflect primarily a response to contextual priming. Figure 3 confirms this pattern.For Qwen2.5-VL-3B,meanbaselineconfidence is near zero and both subsequent conditions produce a sharp monotonic increase, with a narrow standard-deviation band indicating consistency across patients rather than outlier-driven shift. Ministral-3B-Instructshows the same direc- tional trend with wider spread, consistent with its smaller δ values. 6Expert Case Study of Reasoning Traces To assess whether the large F1 gains from multi- modal context reflect genuine clinical understand- ing or priming-induced confabulation, we con- ducted a human expert case study of reasoning traces generated byQwen2.5-VL-3B, the model ex- hibiting the most dramatic performance shift. Evaluation protocol We sampled 12 predic- tions per class (MDD vs. Control) from the FOR2107 dataset under two conditions: the base- line TEXT(ARCF), which receives only the serial- ized clinical record, and the full multimodal con- dition TEXT(ARCF, PARCEL)+PROMPT(MRI)+ PLOT(MRI), which additionally incorporates brain parcellation text and anilearn-generated MRI vi- sualization, yielding 24 reasoning traces per con- dition (48 total). A clinical psychologist evaluated each trace on three criteria (Faithfulness, Clinical Accuracy & Safety, Diagnostic Reasoning) on a 4- point Likert scale. We refer to the full rubric scores in Appendix H. ResultsTable 4 reports results for each criterion across the four evaluated groups. The traces pro- duced under the full multimodal condition (Groups 2 and 3) consistently score higher across all three dimensions compared to the baseline condition (Groups 0 and 1), suggesting that the additional neuroimaging context does contribute to more co- herent and clinically grounded outputs. Notably, Control predictions under the full multimodal con- dition (Group 2) achieve the highest faithfulness (2.79Âą0.98) and diagnostic reasoning (2.96Âą0.91) scores, while MDD predictions under the baseline (Group 1) receive the lowest scores overall (2.08, 2.17, and2.17respectively), showing that with- out neuroimaging context, the model struggles to produce meaningful justifications for the more di- agnostically complex class. Critically, even the best-scoring groups remain below3.0on average across all criteria. Faith- fulness in particular shows moderate fabrication across all conditions, indicating that the model of- ten introduces unverified clinical details into its reasoning traces even when multimodal context is available. This finding tempers the interpretation of the large F1 gain: while richer inputs produce mea- surably better reasoning quality, the gap between human-rated faithfulness and classification perfor- mance suggests that a portion of the improvement may still be attributable to priming-induced confi- dence rather than evidence-grounded inference. 6 Grp.Cond., ClassFaith.âClin. Acc.âReasoningâ 0C1, Control2.17Âą 1.09 2.25Âą 0.90 2.46Âą 1.10 1C1, MDD2.08 Âą 0.88 2.17Âą 0.92 2.17Âą 0.92 2C4, Control2.79Âą 0.982.63Âą 0.822.96Âą 0.91 3C4, MDD2.83Âą 0.92 2.46Âą 0.83 2.83Âą 0.87 Table 4: Expert scores (MeanÂąStd) ofQwen2.5-VL-3B reasoning traces rated by a clinical psychologist on a 4-point Likert scale (n = 12per group, FOR2107). Bold indicates the best score andunderlinedindicates the worst score in each metric column. See Table 1 for notation description. 7 Scaffold Effect While modality collapse is well documented (Sim et al., 2025), we did not find work in literature exploring the specific trigger mechanism we ob- served. Section 5 shows that prompt framing ac- counts for the majority of the observed perfor- mance shift. We now ask what class of inputs is sufficient to trigger this effect, and whether it is specific to the exact preamble wording used in our pipeline. We first characterize the effect on Qwen2.5-VL-3Bfrom FOR2107, then validate the findings onMinistral-3-3Band OASIS-3 via an ablation. For formal details we refer to Appendix J. Phrase probe We construct a set of candidate preamble phrases spanning six semantic categories: MRI/neuroimaging, general clinical, authoritative framing, neutral, structural/format, and negation (full list in Table 8 of Appendix J), and evalu- ate each as a probe replacement for the original PROMPT(MRI) on FOR2107 MDD patients using Qwen2.5-VL-3B, measuring how much each phrase shifts Ë P (MDD)relative to the TEXT(ARCF) base- line. To characterize alignment with the original preamble effect in the modelâs internal representa- tions, we additionally compute the cosine similarity of each phraseâs induced hidden-state shift to that of the original MRI preamble. Figure 4 plots both quantities jointly. The relationship can be described approximately by an inverse sigmoid curve (Ap- pendix J), consistent with the effect operating as a linear offset in representation space passed through the modelâs output nonlinearity. MRI and neuroimaging phrases cluster strongly in the top-right quadrant, confirming that the ef- fect is reliably activated by neuroimaging-adjacent language. Phrases outside this category, includ- ing authoritative framings and explicit pathology primes, fall near or below zero on both axes, in- dicating that the trigger is specific to the modality â0.6â0.4â0.20.00.20.40.60.8 Cosine similarity to scaffold direction â0.2 â0.1 0.0 0.1 0.2 0.3 δ MRI / neuroimaging Neutral / unrelated General clinical Structural / format Authoritative framing Figure 4: Cosine similarity to the scaffold direction vs.δ TEXT(ARCF)+PROMPT â TEXT(ARCF) for candidate phrases across se- mantic categories, evaluated onQwen2.5-VL-3Bover the FOR2107 cohort. Phrases in the top-right quad- rant activate the same residual-stream direction as TEXT(ARCF)+PROMPT(MRI) without providing any imaging data. announcement register rather than to clinical au- thority or symptom description in general. False-modality ablationTo confirm that the trig- ger is purely surface-level and generalizes be- yond the model and dataset used in the phrase probe, we construct two conditions: a factually false fMRI availability without providing any imag- ing, and a semantically irrelevant weather con- text preamble serving as a non-clinical null con- dition.We evaluate both onQwen2.5-VL-3B andMinistral-3-3Bacross both FOR2107 and OASIS-3. Table 5 shows that on FOR2107, the fMRI preamble alone achieves F1=0.702 and 0.361 forQwen2.5-VL-3BandMinistral-3-3B respectively, recovering most of the gain observed under the full multimodal condition, while the weather preamble collapses performance to near or below the text-only baseline (0.056 and 0.031). The contrast replicates on OASIS-3, where the weather condition consistently underperforms the fMRI preamble across both models. This confirms that the effect generalizes across model families and clinical domains: models acquired a strong prior associating neuroimaging availability announce- ments with a shifted pathway, which is entirely absent when the preamble is clinically irrelevant, regardless of whether imaging data is provided. 8 Preference Learning The model frequently grounded predictions in MRI- derived or brain-parcellation features even when those features were absent or uninformative, a fail- ure mode confirmed by domain experts. To sup- press this behavior, we apply alignment via Direct 7 ModeInput Condition FOR2107OASIS-3 Qwen2.5-VL-3B Ministral-3B Qwen2.5-VL-3B Ministral-3B C1TEXT(ACRF)0.1530.0640.2620.504 C2TEXT(ACRF) + PROMPT(MRI)0.7280.4800.5890.541 C2 â TEXT(ACRF) + PROMPT(FMRI)0.7020.3610.3790.569 C2 ⥠TEXT(ACRF) + PROMPT(WEATHER)0.0560.0310.1480.425 Table 5: Scaffold ablation F1 scores across conditions. Bold indicates best score per model per cohort. ConditionBefore MPOâAfter MPOâ C10.1530.459 C20.7280.496 C40.7280.477 Table 6: F1 ofQwen2.5-VL-3Bon FOR2107 before and after MPO. Random baseline F1 = 0.52. Bold indi- cates the best score. See Table 1 for notation description. Preference Optimization (DPO) (Rafailov et al., 2023) in its multimodal extension, Mixed Prefer- ence Optimization (MPO) (Wang et al., 2024). Dataset constructionWe construct a preference dataset from the OASIS-3 using the outputs of the 12 models described in Section 4. Each sample belongs to one of three input modes: TEXT(ARCF), TEXT(ARCF)+PROMPT(MRI), TEXT(ARCF, PAR- CEL)+PROMPT(MRI)+PLOT(MRI). For ev- ery subject, we define the chosen response as the TEXT(ARCF) mode output of any model that pre- dicted the correct label, and the rejected response as any output, from any mode, whose text explic- itly mentions MRI findings, brain parcellation, or related neuroimaging language. The final dataset comprises 17202 preference pairs balanced across the two target classes: Cognitive Normal and Cog- nitive Decline (â50% each). Fine-tuningWe fine-tuneQwen2.5-VL-3Busing MPO, training detailed are reported in Appendix K. Parameter-efficient fine-tuning is performed via Low-Rank Adaptation (LoRA) (Hu et al., 2022), targeting all attention projection and feed-forward layers. Training ran for approximately 12 hours on a single NVIDIA H200 GPU. Evaluation and ResultsWe evaluate the trained models on FOR2107 to avoid contamination and naive improvements deriving from finetuning, as the preference dataset was constructed from OASIS-3 outputs. Table 6 compares pre- and post-alignment F1 on FOR2107. Preference align- ment has opposite effects on the two evaluation modes. In the text-only setting, performance im- proves substantially (0.153 â 0.459,+0.306). In the full multimodal setting, performance drops (0.728 â 0.477,â0.251). This pattern is consis- tent with the prompt-conditioned reasoning mode identified in our contrastive analysis: the unaligned modelâs strong multimodal performance was pri- marily driven by the preamble-activated deeper inference pathway, not by genuine integration of neuroimaging evidence. Preference alignment sup- presses this pathway by collapsing both conditions toward a common level rather than selectively re- moving the dependence on irrelevant modalities. Crucially, the aligned model no longer produces outputs that reference MRI findings or brain par- cellation, satisfying the primary objective of this intervention. However, both modes converge near the random baseline of 0.52 F1, exhibiting what we term a symmetry-without-recovery effect: the modality gap closes not by lifting the weaker condi- tion but by suppressing the stronger one. This sug- gests that the prompt-conditioned reasoning mode is entangled with the modelâs capacity for care- ful clinical inference, and lightweight preference alignment cannot disentangle the two. 9 Conclusion We have shown that clinical neuroimaging clas- sification via VLMs exhibits a domain-specific instance of modality collapse: they achieve gen- uine, calibrated performance gains upon introduc- tion of multimodal context. The improvement is attributable to a scaffold effect: a learned sur- face trigger specific to neuroimaging language activates a shifted inference pathway. This hap- pens regardless of whether imaging data is present, confirmed through a phrase-level probe and false- modality ablation across model families and clini- cal domains. Expert evaluation reveals fabrication of neuroimaging-grounded justifications across all conditions. Preference alignment via MPO, while eliminating MRI-referencing behavior, col- 8 lapses both conditions toward chance level rather than selectively removing modality dependence, a symmetry-without-recovery effect suggesting the scaffold is entangled with the modelâs inferential capacity. These findings call for evaluation pro- tocols that explicitly probe the causal relationship between input content and model outputs before VLMs are deployed in high-stakes clinical settings. Limitations Due to data governance constraints, our experi- ments are limited to open-weight models with a maximum of 72 billion parameters, and we bench- mark only 12 models across two binary classifica- tion datasets. This may limit generalizability to closed systems, larger scales, and multi-class set- tings. While the two cohorts span distinct clinical domains, affective disorders and cognitive decline, both involve binary classification over structured tabular records paired with structural MRI, and it remains an open question whether the scaffold ef- fect generalizes to other imaging modalities, such as fMRI or PET, or to tasks with richer label spaces where the decision boundary is less discrete. Our contrastive confidence analysis is conducted on the two models exhibiting the highest gains on FOR2107, and the 70-80% estimate of prompt framing contribution should be interpreted in this context, as it may not hold uniformly across the full model set. Furthermore, F1 differences between conditions are not accompanied by significance tests, and future work should establish whether the observed margins are statistically reliable across repeated sampling and dataset splits. Our expert case study relies on a single clinical psychologist. Recruiting qualified annotators for this task is non-trivial: evaluating VLM reasoning traces against patient-level clinical records requires both domain expertise and familiarity with the spe- cific instruments used in FOR2107, a combina- tion that severely constrains the available annotator pool. We therefore treat Section 6 as a qualitative expert case study rather than a definitive quanti- tative evaluation. While the directional patterns across conditions are consistent and interpretable, the absolute scores should be regarded as indicative rather than definitive, and future work with multi- ple independent raters would be needed to establish reliable effect sizes for reasoning trace quality. Our preference alignment via MPO successfully suppresses MRI-referencing behavior but does not produce factually correct reasoning traces. Con- structing a preference dataset that rewards gen- uinely evidence-grounded inference would require annotated traces from clinical psychologists, which is costly and not yet available at the scale needed for robust fine-tuning. More fundamentally, the re- sults suggest that the effect is deeply entangled with the modelâs broader inferential capacity, and that lightweight post-hoc alignment cannot disentangle the two. Whether alternative strategies, such as su- pervised invariance training, modality dropout dur- ing pretraining, or causal intervention objectives, could decouple the scaffold trigger from genuine multimodal reasoning remains an open question. Finally, while we take care to exclude trivially discriminative features in consultation with domain experts, the serialization of tabular clinical records into natural language may itself introduce subtle presentation biases that influence model behavior. The interaction between tabular serialization format and prompt framing effects is not systematically explored here and warrants dedicated investigation. Ethics Statement This study uses two clinical datasets governed by strict data-use agreements. FOR2107 data access was granted under the consortiumâs institutional review protocol, which requires formal applica- tion and prohibits redistribution to third parties. OASIS-3 is available through a controlled-access application process administered by the Washing- ton University Knight ADRC, subject to a data-use agreement that similarly prohibits redistribution. No individually identifiable patient information is presented in this work. All reported results are aggregate statistics computed over the full cohort. The study on reasoning traces was conducted by a qualified clinical psychologist who reviewed only model-generated outputs. No patient-facing clinical decisions were made on the basis of model predictions, and no real patient data was exposed beyond what is already accessible under the rele- vant data-use agreements. We emphasize that the VLM outputs analyzed in this study are not suitable for clinical deployment. Models achieve genuine performance gains that nonetheless carry no diagnostic grounding, fabri- cate neuroimaging-grounded justifications with per- sistent regularity, and cannot be straightforwardly corrected through preference alignment without collapsing overall predictive capacity. The gap 9 between surface classification performance and evidence-grounded inference is precisely the vul- nerability this work seeks to expose. Any future application of VLMs in clinical psychiatry or neu- rology must incorporate rigorous prospective val- idation by domain experts, with evaluation proto- cols that explicitly probe the causal relationship between input evidence and model outputs, before any patient-facing use is considered. Acknowledgments This work is funded by LOEWE Center DYNAMIC as part of the Hessian pro- gram for the promotion of cutting-edge re- search LOEWE under the grant number of LOEWE1/16/519/03/09.001(0009)/98.We thank our colleagues at UKPLab, TU Darmstadt for the technical discussion at AI4Psych, SIG Clinical/Psych, SIG Multimodal, SIG Foundation Models, SIG LLM Understanding and Inter- pretability meetings, and colleagues from the University of Marburg and colleagues from the University of Frankfurt for the clinical discussion. References Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, and 1 others. 2025. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631. Alberto Mario Ceballos-Arroyo, Monica Munnangi, Ji- uding Sun, Karen Zhang, Jered McInerney, Byron C. Wallace, and Silvio Amir. 2024. Open (clinical) LLMs are sensitive to instruction phrasings. In Pro- ceedings of the 23rd Workshop on Biomedical Nat- ural Language Processing, pages 50â71, Bangkok, Thailand. Association for Computational Linguistics. Marie E. Galteau, Margaret Broadwater, Yi Chen, Gabriel Desrosiers-Gregoire, Rita Gil, Johannes Kaesser, Eugene Kim, Pervin KÄąrya Ě gdÄą, Henriette Lambers, Yanyan Y. Liu, Xavier LĂłpez-Gil, Eilidh MacNicol, Parastoo Mohebkhodaei, Ricardo X.N. De Oliveira, Carolina A. Pereira, Henning M. Reimann, Alejandro Rivera-Olvera, Erwan Selingue, Nikoloz Sirmpilatze, and 32 others. 2025.Ac- tivation mapping in multi-center retrospective rat sensory-evoked functional mri datasets using a uni- fied pipeline. Imaging Neuroscience, 3:IMAG.a.157. Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, and Iryna Gurevych. 2024. A sur- vey of confidence estimation and calibration in large language models. In Proceedings of the 2024 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies (Volume 1: Long Papers), pages 6577â6595, Mexico City, Mexico. Association for Computational Linguistics. Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Gold- berg. 2022. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing, pages 30â45, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. Aditya Gulati, Moreno DâIncĂ , Nicu Sebe, Bruno Lepri, and Nuria Oliver. 2025. Beauty and the bias: Explor- ing the impact of attractiveness on multimodal large language models. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 8(2):1154â 1168. Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David Sontag. 2023. Tabllm: Few-shot classification of tabular data with large language models. In International Conference on Artificial Intelligence and Statistics, pages 5549â5581. PMLR. Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, and 1 others. 2025. Glm-4.5 v and glm-4.1 v-thinking: Towards versatile multi- modal reasoning with scalable reinforcement learn- ing. arXiv preprint arXiv:2507.01006. Phillip Howard, Kathleen C. Fraser, Anahita Bhiwandi- walla, and Svetlana Kiritchenko. 2025. Uncovering bias in large vision-language models at scale with counterfactuals. In Proceedings of the 2025 Confer- ence of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 5946â5991, Albuquerque, New Mexico. Asso- ciation for Computational Linguistics. Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations. Yoichi Ishibashi, Danushka Bollegala, Katsuhito Su- doh, and Satoshi Nakamura. 2023. Evaluating the robustness of discrete prompts. In Proceedings of the 17th Conference of the European Chapter of the As- sociation for Computational Linguistics, pages 2373â 2384, Dubrovnik, Croatia. Association for Computa- tional Linguistics. Mohamed Insaf Ismithdeen, Muhammad Uzair Khattak, and Salman Khan. 2025. Promptception: How sen- sitive are large multimodal models to prompts? In Findings of the Association for Computational Lin- guistics: EMNLP 2025, pages 23950â23985, Suzhou, China. Association for Computational Linguistics. Adrian Javaloy, Maryam Meghdadi, and Isabel Valera. 2022. Mitigating modality collapse in multimodal 10 VAEs via impartial optimization. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 9938â9964. PMLR. Erik Jones and Jacob Steinhardt. 2022. Capturing fail- ures of large language models via human cognitive biases. In Advances in Neural Information Process- ing Systems. Shreya Kapoor and Bernhard Egger. 2025. Computer graphics from a neuroscientistâs perspective.In Second Workshop on Representational Alignment at ICLR 2025. Tilo Kircher, Markus WĂśhr, Igor Nenadic, Rainer Schwarting, Gerhard Schratt, Judith Alferink, Carsten Culmsee, Holger Garn, Tim Hahn, Bertram MĂźller-Myhsok, Astrid Dempfle, Maik Hahmann, Andreas Jansen, Petra Pfefferle, Harald Renz, Mar- cella Rietschel, Stephanie H Witt, Markus NĂśthen, Axel Krug, and Udo Dannlowski. 2019. Neurobiol- ogy of the major psychoses: a translational perspec- tive on brain structure and functionâthe FOR2107 consortium. European Archives of Psychiatry and Clinical Neuroscience, 269(8):949â962. Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Zae Myung Kim, and Dongyeop Kang. 2024. Bench- marking cognitive biases in large language models as evaluators. In Findings of the Association for Com- putational Linguistics: ACL 2024, pages 517â545, Bangkok, Thailand. Association for Computational Linguistics. Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic uncertainty: Linguistic invariances for un- certainty estimation in natural language generation. In The Eleventh International Conference on Learn- ing Representations. Pamela J. LaMontagne, Tammie LS. Benzinger, John C. Morris, Sarah Keefe, Russ Hornbeck, Chengjie Xiong, Elizabeth Grant, Jason Hassenstab, Krista Moulder, Andrei G. Vlassenko, Marcus E. Raichle, Carlos Cruchaga, and Daniel Marcus. 2019. Oasis- 3: Longitudinal neuroimaging, clinical, and cogni- tive dataset for normal aging and alzheimer disease. medRxiv. Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yan- wei Li, Ziwei Liu, and Chunyuan Li. 2025. LLaVA- onevision: Easy visual task transfer. Transactions on Machine Learning Research. Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Nau- mann, Hoifung Poon, and Jianfeng Gao. 2023. LLaVA-med: Training a large language-and-vision assistant for biomedicine in one day. In Thirty- seventh Conference on Neural Information Process- ing Systems Datasets and Benchmarks Track. Alexander H Liu, Kartik Khandelwal, Sandeep Sub- ramanian, Victor Jouault, Abhinav Rastogi, Adrien SadĂŠ, Alan Jeffares, Albert Jiang, Alexandre Cahill, Alexandre Gavaudan, and 1 others. 2026. Ministral 3. arXiv preprint arXiv:2601.08584. Sheng Lu, Hendrik Schuff, and Iryna Gurevych. 2024. How are prompts different in terms of sensitivity? In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 5833â5856, Mexico City, Mexico. Association for Computational Lin- guistics. David E. Meyer and Roger W. Schvaneveldt. 1971. Fa- cilitation in recognizing pairs of words: evidence of a dependence between retrieval operations. Journal of experimental psychology, 90 2:227â34. Michael Moor, Oishi Banerjee, Zahra Shakeri Hossein Abad, Harlan M. Krumholz, Jure Leskovec, Eric J. Topol, and Pranav Rajpurkar. 2023. Foundation mod- els for generalist medical artificial intelligence. Na- ture, 616(7956):259â265. Letitia Parcalabescu and Anette Frank. 2023. M- SHAP: A performance-agnostic metric for measur- ing multimodal contributions in vision and language models & tasks. In Proceedings of the 61st Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 4032â4059, Toronto, Canada. Association for Computational Lin- guistics. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christo- pher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Sys- tems. R. Jarrett Rushmore, Kyle Sunderland, Holly Carring- ton, Justine Chen, Michael Halle, Andras Lasso, G. Papadimitriou, N. Prunier, Elizabeth Rizzoni, Brynn Vessey, Peter Wilson-Braun, Yogesh Rathi, Marek Kubicki, Sylvain Bouix, Edward Yeterian, and Nikos Makris. 2022. Anatomically curated segmen- tation of human subcortical structures in high resolu- tion magnetic resonance imaging: An open science approach. Frontiers in Neuroanatomy, Volume 16 - 2022. Yan Shu, Hangui Lin, Yexin Liu, Yan Zhang, Gangyan Zeng, Yan Li, Yu ZHOU, Ser-Nam Lim, Harry Yang, and Nicu Sebe. 2025. When semantics mislead vi- sion: Mitigating large multimodal models hallucina- tions in scene text spotting and understanding. In The Thirty-ninth Annual Conference on Neural Infor- mation Processing Systems. Mong Yuan Sim, Wei Emma Zhang, Xiang Dai, and Biaoyan Fang. 2025. Can VLMs actually see and read?a survey on modality collapse in vision- language models. In Findings of the Association for Computational Linguistics: ACL 2025, pages 24452â24470, Vienna, Austria. Association for Com- putational Linguistics. 11 Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, Perry Payne, Martin Seneviratne, Paul Gam- ble, Chris Kelly, Abubakr Babiker, Nathanael Schärli, Aakanksha Chowdhery, Philip Mansfield, Dina Demner-Fushman, and 13 others. 2023. Large lan- guage models encode clinical knowledge. Nature, 620(7972):172â180. Doan Nam Long Vu, Rui Tan, Lena Moench, Svenja Jule Francke, Daniel Woiwod, Florian Thomas-Odenthal, Sanna Stroth, Tilo Kircher, Chris- tiane Hermann, Udo Dannlowski, and 1 others. 2025. Roleplaying with structure: Synthetic therapist-client conversation generation from questionnaires. arXiv preprint arXiv:2510.25384. Weiyun Wang, Zhe Chen, Wenhai Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Jinguo Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao, and 1 others. 2024. En- hancing the reasoning ability of multimodal large language models via mixed preference optimization. arXiv preprint arXiv:2411.10442. Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, and 1 others. 2025. In- ternvl3. 5: Advancing open-source multimodal mod- els in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265. Nils R Winter, Julian Blanke, Ramona Leenings, Jan Ernsting, Lukas Fisch, Kelvin Sarink, Car- lotta Barkhau, Daniel Emden, Katharina Thiel, Kira FlinkenflĂźgel, Alexandra Winter, Janik Goltermann, Susanne Meinert, Katharina Dohm, Jonathan Rep- ple, Marius Gruber, Elisabeth J Leehr, Nils Opel, Dominik Grotegerd, and 26 others. 2024. A system- atic evaluation of machine Learning-Based biomark- ers for major depressive disorder. JAMA Psychiatry, 81(4):386â395. Nils R Winter, Ramona Leenings, Jan Ernsting, Kelvin Sarink, Lukas Fisch, Daniel Emden, Julian Blanke, Janik Goltermann, Nils Opel, Carlotta Barkhau, Su- sanne Meinert, Katharina Dohm, Jonathan Repple, Marco Mauritz, Marius Gruber, Elisabeth J Leehr, Dominik Grotegerd, Ronny Redlich, Andreas Jansen, and 12 others. 2022. Quantifying deviations of brain structure and function in major depressive disorder across neuroimaging modalities. JAMA Psychiatry, 79(9):879â888. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and 3 others. 2020. Trans- formers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38â45, Online. Association for Computational Linguistics. Yiheng Xu, Peng Wang, Hang Zhang, Pengfei Wang, Shuai Bai, Shijie Wang, Junyang Lin, Tianbao Xie, Yuanzhi Zhu, Zhibo Yang, Wei Ding, Xi Zhang, Jianqiang Wan, Jun Tang, Haiyang Xu, Jiabo Ye, Keqin Chen, Xuejing Liu, Jialin Wang, and 8 oth- ers. 2025. Qwen2.5-vl technical report. Preprint, arXiv:2502.13923. Daiki Yoshida, Haruki Sakajo, Kazuki Hayashi, Yusuke Sakai, Hidetaka Kamigaito, Katsuhiko Hayashi, and Taro Watanabe. 2025. Visual priming effect on large- scale vision language models. In Proceedings of the 15th International Conference on Recent Advances in Natural Language Processing - Natural Language Processing in the Generative AI Era, pages 1385â 1395, Varna, Bulgaria. INCOMA Ltd., Shoumen, Bulgaria. Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang, and 1 others. 2025. Glm-4.5: Agentic, reasoning, and coding (arc) foundation mod- els. arXiv preprint arXiv:2508.06471. Chi Zhang, Wenxuan Ding, Jiale Liu, Mingrui Wu, Qingyun Wu, and Ray Mooney. 2026. Do images speak louder than words? investigating the effect of textual misinformation in vlms. arXiv preprint arXiv:2601.19202. Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing few-shot performance of language models. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 12697â12706. PMLR. Weihong Zhong, Xiaocheng Feng, Liang Zhao, Qim- ing Li, Lei Huang, Yuxuan Gu, Weitao Ma, Yuan Xu, and Bing Qin. 2024. Investigating and miti- gating the multimodal hallucination snowballing in large vision-language models. In Proceedings of the 62nd Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 11991â12011, Bangkok, Thailand. Association for Computational Linguistics. Wanrong Zhu, Yuankai Qi, Pradyumna Narayana, Ka- zoo Sone, Sugato Basu, Xin Wang, Qi Wu, Miguel Eckstein, and William Yang Wang. 2022. Diagnosing vision-and-language navigation: What really matters. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 5981â5993, Seattle, United States. Association for Computational Linguistics. A Prompt Prompts that are used in our paper (Figure 5, 6 for FOR2107), (Figure 7, 8 for OASIS-3) 12 AbbreviationCheckpoint (Link to HF) GLM-4.1V-9B zai-org/GLM-4.1V-9B-Thinking GLM-4.6V-Flash zai-org/GLM-4.6V-Flash LLaVA-OV-1.5-4B lmms-lab/LLaVA-OneVision-1.5-4B-Instruct Ministral-3-3B mistralai/Ministral-3-3B-Instruct-2512-BF16 Ministral-3-14B mistralai/Ministral-3-14B-Instruct-2512-BF16 Qwen2.5-VL-3B Qwen/Qwen2.5-VL-3B-Instruct Qwen2.5-VL-32B Qwen/Qwen2.5-VL-32B-Instruct Qwen2.5-VL-72B Qwen/Qwen2.5-VL-72B-Instruct Qwen3-VL-2B Qwen/Qwen3-VL-2B-Instruct Qwen3-VL-32B Qwen/Qwen3-VL-32B-Instruct InternVL3.5-4B OpenGVLab/InternVL2.5-4B InternVL3.5-14B OpenGVLab/InternVL2.5-14B Table 7: Abbreviation used in this work and Hyperlinked Checkpoint Paths Prompt You are given patient clinical information Classify the patient into one of the following cate- gories: - Major Depressive Disorder - Control (no disorder detected) Return your answer as a JSON object with two fields: - "category": the chosen category (exactly one of the two above) - "explanation": a short reasoning for the choice Figure 5: TEXT(ARCF) prompt for FOR2107 B Parcellation in details Parcellation is performed using the Harvard- Oxford probabilistic atlas (Rushmore et al., 2022), which delineates the cerebral cortex into 48 anatom- ically defined regions of interest (ROIs; e.g., Insu- lar Cortex, Temporal Pole). The atlas was con- structed by spatially normalizing manually labeled T1-weighted scans of healthy adults into MNI (Montreal Neurological Institute) space and com- puting voxelwise label probabilities across subjects. We apply the cortical component of this atlas to ex- tract per-region volumetric measurements, which are then serialized into descriptive text following the same approach as the tabular data. MRI visualization For each scan, we render three orthogonal slices, sagittal, coronal, and ax- ial, annotated with crosshairs at MNI coordinates (x,y,z), hemisphere labels (L/R), voxel contour overlays, and a zero-anchored colormap to enhance tissue contrast. Prompt You are given patient clinical information and their MRI data (brain parcellation volume, vi- sualization of brain regions) Classify the patient into one of the following cate- gories: - Major Depressive Disorder - Control (no disorder detected) Return your answer as a JSON object with two fields: - "category": the chosen category (exactly one of the two above) - "explanation": a short reasoning for the choice Figure 6: TEXT(ARCF)+PROMPT(MRI) prompt for FOR2107. Bold text denotes the component that cause performance boost. C Models in details Table 7 shows the abbreviations and checkpoints of the models we used in this work. DFOR2107 Variables Variables that we used to convert to text (Ta- ble 13, 14, 15, 16, 17, 18). Table D illustrates the token distribution of input in FOR2107, divided to Major Depressive Disorder/Control Patient. We use tiktoken 4 to do tokenization. EOASIS-3 Variables Variables that we used to convert to text (Ta- ble 21, 22 24, 23). Table E illustrates the token distribution of input in OASIS-3, divided to Cog- 4 https://github.com/openai/tiktoken 13 Prompt You are given patient clinical information Classify the patient into one of the following cate- gories: - Cognitive Normal - Cognitive Decline Return your answer as a JSON object with two fields: - "category": the chosen category (exactly one of the two above) - "explanation": a short reasoning for the choice Figure 7: TEXT(ARCF) prompt for OASIS-3 Prompt You are given patient clinical information and their MRI data (brain parcellation volume, vi- sualization of brain regions) Classify the patient into one of the following cate- gories: - Cognitive Normal - Cognitive Decline Return your answer as a JSON object with two fields: - "category": the chosen category (exactly one of the two above) - "explanation": a short reasoning for the choice Figure 8: TEXT(ARCF)+PROMPT(MRI) prompt for OASIS-3. Bold text denotes the component that cause performance boost. nitive Normal/Cognitive Decline Patient. We also use tiktoken for tokenization. F Random Baseline To contextualize model performance, we report a random baseline corresponding to a stratified random classifier that predicts each class with a probability equal to its prior. For a binary classi- fication task with class proportionspand1â p, the expected weighted F1 score of such a classi- fier is:F 1 random = p 2 + (1â p) 2 , wherepis the proportion of the minority class. G Full Results on FOR2107 and OASIS-3 G.1FOR2107 - Per-Condition Results Table 9 reports the complete evaluation results on the FOR2107 dataset across all five input condi- tions and all twelve models. Metrics are F1, Preci- sion (P), Recall (R), and Accuracy (ACC). Baseline text-only performance (C1) Under the text-only condition TEXT(ARCF), performance is highly model-dependent. Large-scale models such asQwen2.5-VL-72BandGLM-4.6V-Flash achieve competitive F1 scores of 0.828 and 0.795, respectively, while several smaller models fail to discriminate the positive class meaningfully (InternVL3.5-4B: F1= 0.000;Ministral-3-3B: F1= 0.064;Qwen2.5-VL-3B: F1= 0.153). In these failure cases, precision is either undefined or inflated at 1.000 while recall collapses near zero, indicating a systematic bias toward predicting the negative class. Preamble-only framing (C2) Adding the MRI availability mention to the prompt preamble without providing any image produces dra- matic gains for models that previously failed. Qwen2.5-VL-3Brises from F1= 0.153to0.728 (+0.575);Ministral-3-3Bfrom0.064to0.480 (+0.416); andInternVL3.5-4Bfrom0.000to 0.255(+0.255). Gains for already-strong models are more modest, confirming that the preamble scaf- fold disproportionately unlocks latent capacity in smaller models. Across the board, recall increases substantially while precision is slightly tempered, suggesting the preamble prompts the models to be more willing to predict the positive class. Effect of the actual MRI image (C3) Provid- ing the actual neuroimaging plot alongside the multimodal preamble yields further, though gener- ally smaller, improvements over C2 for most mod- els. The largest additional gains are observed in Ministral-3-3B(0.480 â 0.723,+0.243) and Qwen3-VL-2B(0.281 â 0.447,+0.166), suggest- ing that some models are capable of extracting useful signal from the MRI plot when primed by the preamble. However, several strong models plateau or marginally decline (InternVL3.5-14B: 0.729â 0.540), indicating that visual grounding does not uniformly improve performance and may even introduce noise for certain architectures. Effect of parcellation features (C4) Augment- ing the text input with parcellation-derived fea- tures alongside the MRI plot, yields results broadly comparable to C3, with no consis- tent improvement.Most models change by less thanÂą0.02F1 relative to C3.Notably, Qwen2.5-VL-3BandQwen2.5-VL-32Bshow slight declines (â0.046 andâ0.041 respectively), while InternVL3.5-14B recovers somewhat (0.540 â 14 0100020003000400050006000 Number of Tokens 0 50 100 150 Number of Files (a) Major Depressive Disorder Patient 0100020003000400050006000 Number of Tokens 0 100 200 300 Number of Files (b) Control Patient Figure 9: Distribution of input tokens in FOR2107 dataset 0.587 ). The absence of a clear benefit from par- cellation features suggests that models do not reli- ably integrate the additional structural neuroimag- ing statistics when image-level features are already present. Counterfactual image swap (C5)Replacing the subject-specific MRI plot with an unrelated image, while retaining the multimodal preamble produces performance largely indistinguishable from C4. For example,Qwen2.5-VL-72Bscores F1= 0.849 in C4 and0.849in C5;Qwen3-VL-32Bscores 0.835and0.822;GLM-4.6V-Flashscores0.791 and 0.788. This near-equivalence between C4 and C5 provides the strongest evidence that models are not extracting diagnostically meaningful content from the provided image. Instead, performance gains attributable to the visual modality are driven almost entirely by the preamble framing that an MRI is present, rather than by processing the im- age itself, the central scaffold effect demonstrated in this work. G.2OASIS-3 - Per-Condition Results Table 10 reports the complete evaluation results on the OASIS-3 dataset across all five input con- ditions. OASIS-3 presents a markedly differ- ent challenge from FOR2107: the class distribu- tion is more balanced, the clinical domain differs (Alzheimerâs disease vs. depression), and several models exhibit notably different behavioral pat- terns, including near-total recall collapse or ex- treme precision inflation. Baseline text-only performance (C1) Perfor- mance under TEXT(ARCF) is more heteroge- neous than on FOR2107.Strong models such asInternVL3.5-14B(F1= 0.773) and Qwen2.5-VL-72B(F1= 0.786) perform well, while several models exhibit severe class imbal- ance artefacts.Qwen3-VL-2Bnearly abstains from positive predictions (F1= 0.090, R= 0.047), and GLM-4.1V-9Blikewise shows high precision but near-zero recall (0.278, R= 0.164). Conversely, InternVL3.5-4Bover-predicts the positive class (R= 0.893, P= 0.470), andQwen2.5-VL-32B produces a pathological output (R= 0.963, P= 0.374, ACC= 0.400), suggesting near-constant positive prediction. Scaffold effect: preamble-only framing (C2) Unlike FOR2107, where the MRI preamble pro- duced consistent gains, OASIS-3 shows a more mixed picture under C2. Several models that per- formed well in C1 degrade:InternVL3.5-14B drops from F1= 0.773to0.584(â0.189), and Qwen3-VL-32Bfrom0.582to0.767represents one of the few clear gains.Qwen3-VL-2Bcollapses fur- ther (F1= 0.028), andQwen2.5-VL-32Bremains degenerate (R= 1.000, P= 0.365). This instabil- ity under preamble framing on OASIS-3, where the C1 baseline is already competitive for strong models, is consistent with the negative transfer hy- pothesis: the multimodal scaffold was optimized implicitly for the FOR2107 clinical domain and disrupts well-calibrated text-only behavior on out- of-domain data. Effect of the actual MRI image (C3)Providing the MRI plot (C3) yields the most consistent im- provements on OASIS-3, with several models re- covering from C2 degradation.InternVL3.5-14B recovers to F1= 0.731(+0.147over C2), GLM-4.6V-Flashimproves to0.705(+0.239over C2), andMinistral-3-14Bclimbs to0.652 (+0.097). However,Qwen3-VL-2Bremains es- sentially non-functional (F1= 0.008), and 15 Qwen2.5-VL-32Bremains degraded relative to its C1 baseline despite partial recovery. The over- all pattern suggests that on OASIS-3, the im- age provides more disambiguating signal than the preamble alone, the inverse of what is observed on FOR2107. Effect of parcellation features (C4) Adding parcellation-derived features (C4) produces modestandinconsistentchangesrelative to C3.InternVL3.5-4Bimproves slightly (0.634 â 0.656), andMinistral-3-14Bim- proves (0.652 â 0.675), whileGLM-4.1V-9B declines (0.608 â 0.455). The best-performing model under C4 isQwen2.5-VL-72B(F1= 0.775), unchanged from C3 (0.776), confirming that parcellation features add no reliable signal on this dataset either. Counterfactual image swap (C5). The swap condition on OASIS-3 closely mirrors C4 for most models, further supporting the scaffold ef- fect interpretation.Qwen2.5-VL-72Bis identi- cal across C4 and C5 (F1= 0.778in both); GLM-4.6V-Flashscores0.659(C4) vs.0.696 (C5);InternVL3.5-14Bscores0.738(C4) vs. 0.683(C5).The near-equivalence of subject- specific MRI and an unrelated image, replicating the FOR2107 finding on an entirely different clin- ical population and MRI protocol, provides strong cross-dataset evidence that preamble framing rather than image content drives apparent multimodal gains. H Case Study - Score Rubric in Details ⢠Faithfulness: whether the output strictly adheres to the input context without adding unverified information (1 = severe fabrication; 4 = perfectly faithful). ⢠Clinical Accuracy & Safety: whether the con- clusions are correct, safe, and aligned with clin- ical standards (1 = completely inaccurate or un- safe; 4 = perfectly accurate and safe). ⢠Diagnostic Reasoning: whether the reasoning behind the diagnosis is clearly and logically ex- plained (1 = poor; 4 = excellent). I Confidence Estimation in Details We analyze the two models exhibiting the highest performance gains on FOR2107:Qwen2.5-VL-3B andMinistral-3B. To quantify model confi- dence during inference, we extract per-token log- probabilities from the decoder using greedy decod- ing withoutput_scores=True, which provides the full vocabulary distribution at each generation step. Since the two class labels diverge at the first generated token, Major and Control, the modelâs decision reduces to a single-token fork. We there- fore extract the softmax probability directly at that branching point, which is both sufficient and unam- biguous for this binary classification setting. This avoids the structural length bias that would arise from comparing raw joint probabilities across se- quences of unequal length, while remaining more interpretable than sampling-based approaches such as semantic entropy (Kuhn et al., 2023), which require stochastic decoding and multiple forward passes. Formally, letsdenote the generation step at which the label token is produced. For each class â â MDD, ctrl, the raw softmax probability at step s is: p â = softmax(z s ) t â 1 ,(1) wherez s is the logit vector at stepsandt â 1 is the first token of labelâ. The normalized confidence score is then: Ë P c (â| x) = p â p MDD + p ctrl + Îľ ,(2) whereÎľ = 10 â12 for numerical stability andc denotes the input condition. The predicted class is Ëy = arg max â Ë P c (â| x). We use Ë P c (â | x)as the basis for a contrastive analysis designed to disentangle the respective con- tributions of prompt framing and actual MRI con- tent. Specifically, we evaluate each patient under three conditions and define the per-patient confi- dence shift as: δ cond â base (x i ) = Ë P cond (â| x i )â Ë P base (â| x i ), (3) whereâ = MDD. A positiveδindicates that the condition increases the modelâs confidence in the MDD label relative to the baseline. J Scaffold Effect: Formal Details J.1 Scaffold Direction LetMbe a decoder-only vision-language model withLlayers and hidden dimensiond.Let h (l) c (x) âR d denote the residual-stream hidden state at layerl, extracted at the label token decod- ing step for patient record x under input condition c . We denote the target positive class asâ + (the 16 condition of interest in the binary classification task). For brevity we writec 0 = TEXT(ARCF)and c 1 = TEXT(ARCF)+PROMPT(MRI). Given a set ofNpatient recordsx i N i=1 , we define the scaffold direction at layer l â as: d = 1 N N X i=1 h (l â ) c 0 (x i ) â 1 N N X i=1 h (l â ) c 1 (x i ), (4) with unit-normalized formu = d/âĽd⼠2 . Intu- itively,ucaptures the direction in residual-stream space that the MRI preamble injects to shift the modelâs final-layer classification routing towardâ + . The target layerl â . The target layerl â is selected per model as the layer immediately preceding the earliest layer at which Ë P c 1 (â + | x) departs substan- tially from thec 0 baseline, as identified via a logit lens sweep over the label token. J.2 Phrase Probe Procedure For a candidate preamble phrasep, we define its induced hidden-state shift as: âh (l â ) p (x i ) = h (l â ) p (x i ) â h (l â ) c 0 (x i ),(5) whereh (l â ) p (x i )is obtained by replacing the task preamble withpwhile keeping all other inputs fixed. We quantify each candidate phrase along two axes: Scaffold alignment The mean cosine similarity between the induced hidden-state shift and the scaf- fold direction: cos-sim(p) = 1 N N X i=1 âh (l â ) p (x i ) âĽâh (l â ) p (x i )⼠2 ¡ u. (6) A value near+1indicates thatpshifts the residual stream in the same direction as the original MRI preamble; a value nearâ1 indicates the opposite. Confidence shift The mean per-patient shift in Ë P c (â + | x) relative to c 0 , following Equation (3): δ c 0 +p âc 0 = 1 N N X i=1 Ë P p (â + | x i )â Ë P c 0 (â + | x i ) . (7) Figure 4 plotscos-sim(p)againstδ c 0 +p âc 0 for all can- didate phrases. J.3 Scaffold Response Curve The two quantities defined in Equations(6)and(7) are not independent. Since the residual stream architecture of transformer-based models propa- gates information written at any layer additively to all subsequent layers (Geva et al., 2022), the hidden-state shift induced atl â reaches the final layer approximately as: h (L) p (x) â h (L) c 0 (x) + ι¡ cos-sim(p)¡ v, (8) for some propagated directionv âR d and scalar Îą âR, wherecos-sim(p)is as defined in Equa- tion(6). The binary confidence from Equation(2) then reduces to Ë P c (â + | x) = Ď(âw ⤠h (L) c (x)), whereâw = w â + â w â â is the LM head con- trast vector. Substituting Equation(8)and setting a = ι¡ âw ⤠vâR and b = âw ⤠h (L) c 0 (x) gives: Ë P p (â + | x) = Ď b + a¡ cos-sim(p) .(9) Subtracting the baseline Ë P c 0 (â + | x) = Ď(b) and following Equation(7)gives the scaffold response curve: δ c 0 +p âc 0 = Ď a¡ cos-sim(p) + b â Ď(b).(10) Equation(10)carries a direct mechanistic in- terpretation: the scaffold effect operates as a lin- ear offset in representation space passed through the modelâs fixed output nonlinearity. The sen- sitivity parameteraquantifies per-model suscep- tibility to the scaffold trigger.We would ex- pectato be larger forQwen2.5-VL-3Bthan for Ministral-3-3B, consistent with its systemati- cally largerδ c 0 +p âc 0 values in Table 3. The false- modality ablation further follows as a direct pre- diction: âfMRI data is included.â achieves high cos-sim(p), so Equation(10)predicts a large Ë Î´, which Table 5 confirms. In practice,aandbare estimated per model by minimizing mean squared error overP : (a â ,b â ) = arg min a,b 1 |P| X pâP δ c 0 +p âc 0 â Ë Î´(p) 2 . (11) J.4 Phrase Probe Inventory Table 8 lists the full inventory of phrase probes with their scores. For this work, we applied the logit lens 5 to all layers and found that the diver- gence signal between MDD and control emerges at layer 33. We then sampled 71 participants from FOR2107 for the probe experiment. 5 https://w.lesswrong. com/posts/AcKRB8wDpdaN6v6ru/ interpreting-gpt-the-logit-lens 17 400450500550600650700750 Number of Tokens 0 20 40 60 80 Number of Files (a) Cognitive Decline Patient 550600650700750800 Number of Tokens 0 20 40 60 80 Number of Files (b) Cognitive Normal Patient Figure 10: Distribution of input tokens in OASIS-3 dataset #CategoryPhrasecosâ δâ 1⢠MRI / neuroimagingBrain MRI findings are available.+0.43+0.02 2⢠General clinicalDiagnostic results are available.+0.24+0.02 3⢠MRI / neuroimagingBrain scans have been performed.+0.23-0.01 4⢠Structural / formatClassification result:+0.00-0.02 5⢠General clinicalA clinical diagnosis has been established.-0.07-0.01 6⢠Structural / formatThe answer is:-0.25-0.09 7⢠Structural / formatOutput: "category":-0.32-0.08 8⢠Neutral / unrelatedThis is a test of the system.-0.32-0.04 9⢠General clinicalClinical evaluation is complete.-0.32-0.06 10⢠Neutral / unrelatedAnswer the following question carefully.-0.38-0.05 11⢠Authoritative framingAs a medical professional, review the following.-0.39-0.04 12⢠Neutral / unrelatedPlease process the following information.-0.43-0.04 13⢠General clinicalThe patient has been evaluated by a specialist.-0.45-0.06 14⢠Authoritative framingYou have extensive experience in psychiatric diagnosis.-0.48-0.08 15⢠Structural / formatReturn your answer as JSON.-0.48-0.07 16⢠Authoritative framingYou are an expert clinical psychiatrist.-0.49-0.07 17⢠Structural / formatRespond only with a JSON object.-0.49-0.09 18⢠General clinical The patient has been assessed for a psychiatric disorder.-0.49-0.11 19⢠Neutral / unrelatedYou are a helpful assistant.-0.53-0.08 Table 8: Numbered phrases from Figure 4. cos: cosine similarity to scaffold direction and δ K Preference Learning in details We use MPO for preference finetuning which com- bines three complementary loss signals: L MPO = w Ď L sigmoid + w bco L bco_pair + w sft L sft , (12) whereL sigmoid is the standard DPO preference loss, L bco_pair is a binary quality signal penalising re- jected responses, andL sft is a generation loss on the chosen response that prevents degenerate out- put collapse. The LoRA configuration is given in Table 11. Table 12 depicts all hyperparameters used in our DPO training. 18 ModelF1PRACC C1: TEXT(ARCF) InternVL3.5-4B0.000 0.000 0.000 0.604 InternVL3.5-14B 0.559 0.946 0.397 0.752 GLM-4.1V-9B0.521 0.958 0.358 0.740 GLM-4.6V-Flash0.795 0.896 0.715 0.854 LLaVA-OV-1.5-4B 0.663 0.954 0.508 0.796 Ministral-3-3B0.064 1.000 0.033 0.617 Ministral-3-14B 0.588 0.983 0.419 0.768 Qwen2.5-VL-3B0.153 1.000 0.083 0.637 Qwen2.5-VL-32B0.749 0.900 0.642 0.830 Qwen2.5-VL-72B0.828 0.853 0.805 0.868 Qwen3-VL-2B0.546 0.869 0.398 0.738 Qwen3-VL-32B0.744 0.928 0.621 0.831 C2: TEXT(ARCF) + PROMPT(MRI) InternVL3.5-4B0.255 0.963 0.147 0.660 InternVL3.5-14B 0.729 0.884 0.621 0.818 GLM-4.1V-9B0.640 0.915 0.492 0.781 GLM-4.6V-Flash0.802 0.847 0.760 0.851 LLaVA-OV-1.5-4B 0.592 0.980 0.424 0.769 Ministral-3-3B0.480 0.720 0.360 0.691 Ministral-3-14B 0.680 0.961 0.526 0.804 Qwen2.5-VL-3B0.728 0.876 0.623 0.816 Qwen2.5-VL-32B0.820 0.897 0.756 0.869 Qwen2.5-VL-72B0.844 0.804 0.889 0.870 Qwen3-VL-2B0.281 0.966 0.164 0.667 Qwen3-VL-32B0.823 0.900 0.758 0.871 C3: TEXT(ARCF) + PROMPT(MRI) + MRI PLOT InternVL3.5-4B0.296 0.992 0.174 0.673 InternVL3.5-14B 0.540 0.981 0.372 0.749 GLM-4.1V-9B0.733 0.887 0.625 0.820 GLM-4.6V-Flash0.794 0.875 0.728 0.851 LLaVA-OV-1.5-4B 0.713 0.937 0.575 0.817 Ministral-3-3B0.723 0.851 0.628 0.809 Ministral-3-14B 0.710 0.948 0.568 0.817 Qwen2.5-VL-3B0.680 0.878 0.555 0.794 Qwen2.5-VL-32B0.820 0.840 0.802 0.861 Qwen2.5-VL-72B0.846 0.795 0.903 0.870 Qwen3-VL-2B0.447 0.908 0.297 0.710 Qwen3-VL-32B0.846 0.827 0.866 0.875 ModelF1PRACC C4: TEXT(ARCF,PARCEL) + PROMPT(MRI) + MRI PLOT InternVL3.5-4B0.252 0.990 0.1440.661 InternVL3.5-14B 0.587 0.913 0.4320.759 GLM-4.1V-9B0.720 0.916 0.5930.818 GLM-4.6V-Flash0.791 0.848 0.7400.845 LLaVA-OV-1.5-4B 0.603 0.975 0.4370.773 Ministral-3-3B0.677 0.903 0.5420.796 Ministral-3-14B 0.680 0.966 0.5250.805 Qwen2.5-VL-3B0.723 0.633 0.8450.744 Qwen2.5-VL-32B0.779 0.896 0.6890.845 Qwen2.5-VL-72B0.845 0.798 0.8970.870 Qwen3-VL-2B0.347 0.974 0.2110.686 Qwen3-VL-32B0.835 0.865 0.8060.874 C5: TEXT(ARCF,PARCEL) + PROMPT(MRI) + SWAP IMAGE InternVL3.5-4B0.399 0.972 0.2510.701 InternVL3.5-14B 0.703 0.902 0.5760.808 GLM-4.1V-9B0.732 0.888 0.6220.819 GLM-4.6V-Flash0.788 0.864 0.7250.846 LLaVA-OV-1.5-4B 0.540 0.992 0.3710.750 Ministral-3-3B0.675 0.759 0.6080.769 Ministral-3-14B 0.698 0.955 0.5490.812 Qwen2.5-VL-3B0.728 0.666 0.8030.763 Qwen2.5-VL-32B0.780 0.902 0.6860.847 Qwen2.5-VL-72B0.849 0.838 0.8590.879 Qwen3-VL-2B0.365 0.935 0.2270.688 Qwen3-VL-32B0.822 0.893 0.7620.870 Table 9: Full results on FOR2107 across all five input conditions (C1âC5) and all twelve models. Metrics are F1, Precision (P), Recall (R), and Accuracy (ACC). Best F1 per condition is bolded. Left panel: text-only and preamble/image conditions (C1âC3). Right panel: parcellation-augmented and counterfactual swap conditions (C4âC5). 19 ModelF1PRACC C1: TEXT(ARCF) InternVL3.5-4B0.616 0.470 0.893 0.594 InternVL3.5-14B 0.773 0.865 0.698 0.850 GLM-4.1V-9B0.278 0.909 0.164 0.689 GLM-4.6V-Flash0.359 0.901 0.224 0.708 LLaVA-OV-1.5-4B 0.680 0.591 0.799 0.725 Ministral-3-3B0.504 0.692 0.396 0.716 Ministral-3-14B 0.602 0.878 0.458 0.779 Qwen2.5-VL-3B0.262 0.961 0.152 0.689 Qwen2.5-VL-32B0.539 0.374 0.963 0.400 Qwen2.5-VL-72B0.786 0.927 0.682 0.865 Qwen3-VL-2B0.090 0.920 0.047 0.651 Qwen3-VL-32B0.582 0.945 0.421 0.780 C2: TEXT(ARCF) + PROMPT(MRI) InternVL3.5-4B0.636 0.509 0.848 0.646 InternVL3.5-14B 0.584 0.421 0.953 0.505 GLM-4.1V-9B0.397 0.696 0.277 0.692 GLM-4.6V-Flash0.465 0.835 0.322 0.730 LLaVA-OV-1.5-4B 0.615 0.464 0.912 0.584 Ministral-3-3B0.541 0.425 0.745 0.540 Ministral-3-14B 0.555 0.387 0.982 0.426 Qwen2.5-VL-3B0.589 0.910 0.435 0.778 Qwen2.5-VL-32B0.534 0.365 1.000 0.365 Qwen2.5-VL-72B0.778 0.902 0.684 0.858 Qwen3-VL-2B0.028 1.000 0.014 0.641 Qwen3-VL-32B0.767 0.887 0.676 0.850 C3: TEXT(ARCF) + PROMPT(MRI) + MRI PLOT InternVL3.5-4B0.634 0.502 0.860 0.638 InternVL3.5-14B 0.731 0.726 0.735 0.802 GLM-4.1V-9B0.608 0.827 0.481 0.774 GLM-4.6V-Flash0.705 0.915 0.573 0.825 LLaVA-OV-1.5-4B 0.620 0.468 0.920 0.589 Ministral-3-3B0.592 0.498 0.729 0.633 Ministral-3-14B 0.652 0.522 0.867 0.662 Qwen2.5-VL-3B0.501 0.885 0.349 0.746 Qwen2.5-VL-32B0.688 0.625 0.766 0.747 Qwen2.5-VL-72B0.776 0.828 0.731 0.846 Qwen3-VL-2B0.008 1.000 0.004 0.636 Qwen3-VL-32B0.713 0.641 0.803 0.764 ModelF1PRACC C4: TEXT(ARCF,PARCEL) + PROMPT(MRI) + MRI PLOT InternVL3.5-4B0.656 0.563 0.7840.699 InternVL3.5-14B 0.738 0.783 0.6980.819 GLM-4.1V-9B0.455 0.867 0.3080.730 GLM-4.6V-Flash0.659 0.836 0.5440.795 LLaVA-OV-1.5-4B 0.681 0.570 0.8440.711 Ministral-3-3B0.563 0.521 0.6120.653 Ministral-3-14B 0.675 0.670 0.6800.761 Qwen2.5-VL-3B0.595 0.799 0.4740.765 Qwen2.5-VL-32B0.712 0.889 0.5930.825 Qwen2.5-VL-72B0.775 0.878 0.6940.853 Qwen3-VL-2B0.004 1.000 0.0020.636 Qwen3-VL-32B0.728 0.860 0.6300.828 C5: TEXT(ARCF,PARCEL) + PROMPT(MRI) + SWAP IMAGE InternVL3.5-4B0.675 0.594 0.7820.725 InternVL3.5-14B 0.683 0.658 0.7110.759 GLM-4.1V-9B0.522 0.900 0.3680.754 GLM-4.6V-Flash0.696 0.813 0.6080.806 LLaVA-OV-1.5-4B 0.677 0.579 0.8150.716 Ministral-3-3B0.568 0.556 0.5810.678 Ministral-3-14B 0.736 0.781 0.6960.818 Qwen2.5-VL-3B0.619 0.833 0.4930.779 Qwen2.5-VL-32B0.713 0.870 0.6040.822 Qwen2.5-VL-72B0.778 0.902 0.6840.858 Qwen3-VL-2B0.008 1.000 0.0040.636 Qwen3-VL-32B0.667 0.860 0.5440.801 Table 10: Full results on OASIS-3 across all five input conditions (C1âC5) and all twelve models. Metrics are macro-averaged F1, Precision (P), Recall (R), and Accuracy (ACC). Best F1 per condition is bolded. Left panel: text-only and preamble/image conditions (C1âC3). Right panel: parcellation-augmented and counterfactual swap conditions (C4âC5). Note thatQwen3-VL-2Bproduces near-degenerate outputs across all conditions on this dataset, andQwen2.5-VL-32Bunder C1âC2 exhibits near-constant positive prediction (ACCâ0.40, Râ1.00), reflecting calibration failure rather than discriminative ability. 20 HyperparameterValue Rank (r)64 Alpha (Îą)128 Dropout0.05 Biasnone Target modules q, k, v, o, gate, up, down_proj Table 11: LoRA adapter configuration. HyperparameterValue MPO losses Loss typessigmoid, bco_pair, sft Loss weights0.8, 0.2, 1.0 KL penalty (β)0.1 Optimisation Learning rate5Ă 10 â5 LR schedulerCosine Warmup steps100 Epochs3 Effective batch size16 Per-device batch size1 Gradient accumulation16 Hardware & precision PrecisionBF16 + TF32 Gradient checkpointingâ GPU1Ă H200 Training timeâ12 h Sequence truncationNone (image-safe) Table 12: MPO training hyperparameters. 21 NameDescription ProbandTest subject Datum_InterviewDate interview as stated on the interview form Datum_FragebogenDate of the questionnaire GeburtsjahrYear of birth AlterAge GeschlechtGender BildungsjahreYear of education Bildungsjahre_VaterYear of education father Bildungsjahre_MutterYear of education mother BMIBody mass index BMI_categoryBody mass index category UrbanicityScoreUrbanity score AlterMutterBeiGeburtMom age at birth AlterVaterBeiGeburtDad age at birth Spezifische_Phobie_TypusWhat type of phobia is present Spezifische_Phobie_Typus2What type of phobia is present? If more than one, found here GroupPatient grouping or diagnosis Specific_phobia_currentIs the patient currently suffering from a specific phobia? Specific_phobia_lifetimeIs the patient suffering from a specific phobia in his lifetime? Eating_Disorder_currentIs the patient currently suffering from an eating disorder? Eating_Disorder_lifetimeIs the patient suffering from an eating disorder in his lifetime? Alcohol_Use_Disorder_CurrentIs the patient currently suffering from an alcohol use disorder? Alcohol_Use_Disorder_LifetimeIs the patient suffering from an alcohol use disorder in his lifetime? Table 13: FOR2107 - Demographics and Clinical Information. NameDescription RS-25: Resilience Scale RS251If I have plans, I follow them through. RS252I usually manage everything somehow. RS253I can rely on myself rather than on others. RS254It is important for me to stay interested in many things. RS255If I have to, I can be alone. RS256I am proud of what I have already achieved. RS257Iâm not easily thrown off track. RS258I like myself. RS259I can manage several things at the same time. RS2510I am determined. RS2511I rarely ask myself questions about meaning. RS2512I take things as they come. RS2513I can get through difficult times because I know I have done it before. RS2514I have self-discipline. RS2515I stay interested in many things. RS2516I often find something to laugh about. RS2517My belief in myself helps me even in hard times. RS2518I can be relied on in emergencies. RS2519I can usually see a situation from several perspectives. RS2520I can also overcome myself to do things that I donât really want to do. RS2521My life has a purpose. RS2522I donât insist on things that I canât change. RS2523When Iâm in a difficult situation, I usually find a way out. RS2524I have enough energy to do everything I have to do. RS2525I can accept it if not everyone likes me. PSS: Perceived Stress Scale PSS1sf - PSS3sfIn the last month, how often did you feel upset/unable to control things/nervous? PSS4sf - PSS6sfIn the last month, how often were you able to successfully handle problems/changes? PSS7sf - PSS9sf In the last month, how often did you feel things were going your way/could not fulfill responsibilities? PSS10sf - PSS12sfIn the last month, how often did you feel on top of things/upset about uncontrolled things? PSS13sf - PSS14sfIn the last month, how often were you able to decide how to spend time/feel difficulties piling up? Table 14: FOR2107 - Items for the Resilience Scale (RS-25) and Perceived Stress Scale (PSS). 22 NameDescription FSozU1I have people who can look after my home (flowers, pets) when Iâm not there. FSozU2There are people who accept me for who I am. FSozU3It is important for my friends/relatives to know my opinion on certain things. FSozU4I would like more understanding and care from others. FSozU5I have a very trusted person whose help I can always count on. FSozU6I can borrow tools and food if necessary. FSozU7I have friends/relatives who can listen when I need to talk. FSozU8I hardly know anyone I like to go out with. FSozU9I have friends/relatives who can give me a hug. FSozU10If I am ill, I can ask friends/relatives to do important things (e.g. shopping). FSozU11If Iâm really depressed, I know who I can go to. FSozU12I often feel like an outsider. FSozU13There are people who share my joys and sorrows. FSozU14With some friends/relatives, I can also be quite relaxed. FSozU15I have a trusted person who I feel very comfortable around. FSozU16I have enough people who really help me when I get stuck. FSozU17There are people who stick by me even when I make mistakes. FSozU18I would like more security and closeness. FSozU19There are enough people with whom I have a really good relationship. FSozU20There is a community of people (circle of friends, clique) that I feel drawn to. FSozU21I often get good tips from my circle of friends and acquaintances. FSozU22 There are people to whom I can show all my feelings without it being embarrassing. Table 15: FOR2107 - Questionnaire on Social Support (FSozU) Items. 23 NameDescription LEQ_pn1 / LEQ1Health: Serious illness of oneâs own (Type of influence / Influence on life) LEQ_pn2 / LEQ2Health: Major change in eating habits LEQ_pn3 / LEQ3Health: Major change in sleeping habits LEQ_pn4 / LEQ4Health: Significant change in the type or amount of leisure activities LEQ_pn5 / LEQ5Health: Major dental procedure LEQ_pn6 / LEQ6Health: Pregnancy LEQ_pn7 / LEQ7Health: Miscarriage or abortion LEQ_pn8 / LEQ8Health: Onset of menopause LEQ_pn9 / LEQ9Health: Major difficulties with contraceptive aids LEQ_pn10 / LEQ10Work: Difficulties in finding work LEQ_pn11 / LEQ11Work: Taking up work outside the home LEQ_pn12 / LEQ12Work: Changing to a new type of work LEQ_pn13 / LEQ13Work: Changing your working hours or conditions LEQ_pn14 / LEQ14Work: Changing your job responsibilities LEQ_pn15 / LEQ15Work: Difficulties at work with your employer or other employees LEQ_pn16 / LEQ16Work: Major company reorganisations LEQ_pn17 / LEQ17Work: Being dismissed or laid off from work LEQ_pn18 / LEQ18Work: Ending your working life LEQ_pn19 / LEQ19Work: Learning at home or distance learning LEQ_pn20 / LEQ20School/Education: Starting or ending a school or training program LEQ_pn21 / LEQ21School/Education: Changing schools or training programs LEQ_pn22 / LEQ22School/Education: Changing a career goal or major in college LEQ_pn23 / LEQ23School/Education: Problems in a school or training program LEQ_pn24 / LEQ24Residence: Difficulties in finding accommodation LEQ_pn25 / LEQ25Residence: Moving within the same town or city LEQ_pn26 / LEQ26Residence: Moving to another town, state, or country LEQ_pn27 / LEQ27Residence: Significant changes to your living circumstances LEQ_pn28 / LEQ28Love/Partnership: Beginning of a new, close, personal relationship Table 16: FOR2107 - Life Experiences Questionnaire (LEQ) Part 1. Note: âpnâ designates Type of Influence, while the number alone designates Influence on Life. 24 NameDescription LEQ_pn29 / LEQ29Love/Partnership: Entering into an engagement LEQ_pn30 / LEQ30Love/Partnership: Problems with boyfriend or girlfriend LEQ_pn31 / LEQ31 Love/Partnership: Separation from boyfriend/girlfriend or breaking engage- ment LEQ_pn32 / LEQ32Love/Partnership: Pregnancy of wife or girlfriend LEQ_pn33 / LEQ33Love/Partnership: Miscarriage or abortion of wife or girlfriend LEQ_pn34 / LEQ34Love/Partnership: Marriage or domestic partnership LEQ_pn35 / LEQ35Love/Partnership: Change in closeness to partner LEQ_pn36 / LEQ36Love/Partnership: Infidelity LEQ_pn37 / LEQ37Love/Partnership: Conflict with in-laws LEQ_pn38 / LEQ38Love/Partnership: Separation from spouse or partner due to arguments LEQ_pn39 / LEQ39Love/Partnership: Separation from spouse/partner due to work, travel, etc. LEQ_pn40 / LEQ40Love/Partnership: Reconciliation with spouse or partner LEQ_pn41 / LEQ41Love/Partnership: Divorce LEQ_pn42 / LEQ42Love/Partnership: Changes in spouse/partnerâs activities outside the home LEQ_pn43 / LEQ43Family/Friends: Addition of a new family member LEQ_pn44 / LEQ44Family/Friends: Moving out of a child or family member LEQ_pn45 / LEQ45 Family/Friends: Major changes in health/behavior of family member or friend LEQ_pn46 / LEQ46Family/Friends: Death of a spouse or partner LEQ_pn47 / LEQ47Family/Friends: Death of a child LEQ_pn48 / LEQ48Family/Friends: Death of a family member or close friend LEQ_pn49 / LEQ49Family/Friends: Birth of a grandchild LEQ_pn50 / LEQ50Family/Friends: Changes in your parentsâ marital status LEQ_pn51 / LEQ51Parenting: Changes in childcare arrangements LEQ_pn52 / LEQ52Parenting: Conflicts with spouse or partner over parenthood LEQ_pn53 / LEQ53Parenting: Conflicts with childâs grandparents over parenthood LEQ_pn54 / LEQ54Parenting: Taking on the responsibilities of being a single parent LEQ_pn55 / LEQ55Parenting: Custody disputes with former spouse or partner Table 17: FOR2107 - Life Experiences Questionnaire (LEQ) Part 2. 25 NameDescription LEQ_pn56 / LEQ56Personal/Social: Greater personal achievement LEQ_pn57 / LEQ57Personal/Social: Important decision regarding your immediate future LEQ_pn58 / LEQ58 Personal/Social: Changes in your personal habits (clothing, lifestyle, hob- bies) LEQ_pn59 / LEQ59Personal/Social: Changes in your religious beliefs LEQ_pn60 / LEQ60Personal/Social: Changes in your political views LEQ_pn61 / LEQ61Personal/Social: Loss or damage to your personal property LEQ_pn62 / LEQ62Personal/Social: Gone on a vacation LEQ_pn63 / LEQ63Personal/Social: Taking a trip for non-recreational purposes LEQ_pn64 / LEQ64Personal/Social: Changes in family gatherings LEQ_pn65 / LEQ65Personal/Social: Changes in your social activities (clubs, events, visits) LEQ_pn66 / LEQ66Personal/Social: Beginning of new friendships LEQ_pn67 / LEQ67Personal/Social: End of a friendship LEQ_pn68 / LEQ68Personal/Social: Acquisition or loss of a pet LEQ_pn69 / LEQ69Money: Significant change in your financial situation LEQ_pn70 / LEQ70Money: Moderate financial commitment (TV, car, etc.) LEQ_pn71 / LEQ71Money: Large financial commitment or mortgage LEQ_pn72 / LEQ72Money: Cancellation of a mortgage or loan LEQ_pn73 / LEQ73Money: Difficulties with creditworthiness LEQ_pn74 / LEQ74Crime/Legal: Victim of theft or identity theft LEQ_pn75 / LEQ75Crime/Legal: Victim of a violent crime (rape, assault, etc.) LEQ_pn76 / LEQ76Crime/Legal: Involvement in an accident LEQ_pn77 / LEQ77Crime/Legal: Involvement in a legal dispute LEQ_pn78 / LEQ78 Crime/Legal: Involvement in a misdemeanor (tickets, disturbing the peace) LEQ_pn79 / LEQ79Crime/Legal: Trouble with the law resulting in arrest or detention LEQ_pn80 / LEQ80Other recent experiences having an impact on life (1) LEQ_pn81 / LEQ81Other recent experiences having an impact on life (2) LEQ_pn82 / LEQ82Other recent experiences having an impact on life (3) Table 18: FOR2107 - Life Experiences Questionnaire (LEQ) Part 3. NameDescription SozDemo1 â SozDemo5Current living, work situation, occupation, social contacts Haushaltu14 / Haushaltab14People in the household under/over 14 years HaushaltsnettoHousehold net income SchulabschlussHighest educational qualification achieved by subject Schule_Vater / Schule_MutterHighest educational qualification achieved by father / mother GebJahr_Mutter / GebJahr_VaterMotherâs / Fatherâs year of birth ImmigrationOwn immigration or parentsâ? KinderDo you have children? Soehne_leibl (_age)Number (and age) of biological sons Toechter_leibl (_age)Number (and age) of biological daughters Soehne_adopt (_age)Number (and age) of adopted sons Toechter_adopt (_age)Number (and age) of adopted daughters GeschwisterDo you have siblings? Brueder_GE (_age)Number (and age) of brothers (parents shared) Schwestern_GE (_age)Number (and age) of sisters (parents shared) Halbbrueder / HalbschwesternNumber (and age) of half-brothers / half-sisters Stiefbrueder / StiefschwesternNumber (and age) of step-brothers / step-sisters Zwillinge_Famil / Zwilling_selbstAre there twins in your first degree family? / Are you a twin? S_Stadt / S_BundeslandCity/State mother lived during pregnancy S_Risiken1 â S_Risiken7Pregnancy risks (infection, alcohol, drugs, malnutrition, smoking) Geburtskomplikationen1 â 4Birth complications (forceps, vacuum, cesarean, other) Geburtsgewicht / SSW_GeburtBirth weight in grams / Week of birth Table 19: FOR2107 - Socio-Demographics, Family Composition, and Pregnancy Variables. 26 NameDescription FzT: Drinking Habits FzT1 â FzT5Questions on frequency and quantity of alcohol consumption (now/past). FzT6 â FzT9Questions on binge drinking (6+ drinks) and inability to stop (now/past). FzT10 â FzT13Failing expectations due to alcohol; morning drinking (now/past). FzT14 â FzT17Guilt, remorse, and memory loss due to alcohol (now/past). FzT18 â FzT20Injuries, advice to reduce, and professional help sought for alcohol. Table 20: FOR2107 - Alcohol Use (FzT). NameDescription Subject Demographics & Family OASISIDOASIS subject ID GENDERSubjectâs gender RACESubjectâs race HANDSubjectâs Handedness TWIN Is this a new informant? (Note: Label suggests âTwinâ but description asks about new informant status) SIBSHow many full siblings does the subject have? KIDSHow many biological children did the subject have? Living Situation & Independence LIVSITWhat is the subjectâs living situation? LIVSITUALiving situation (detailed categorization) INDEPENDWhat is the subjectâs level of independence? RESIDENCWhat is the subjectâs primary type of residence? MARISTATSubjectâs current marital status Table 21: OASIS-3 - Subject Demographics, Family, and Living Situation. NameDescription INSEXInformantâs sex INHISPDoes informant report being of Hispanic/Latino ethnicity? INHISPORIf yes, what are the informantâs reported origins? INRACEWhat does informant report as his/her race? INRASECWhat additional race does informant report? INRATER What additional race, beyond what was indicated above, does informant report? INEDUCInformantâs years of education INRELTOWhat is informantâs relationship to subject? INLIVWTHDoes the informant live with the subject? INVISITSIf no, approximate frequency of in-person visits INCALLSIf no, approximate frequency of telephone contact INRELYIs there a question about the informantâs reliability? Table 22: OASIS-3 - Informant Demographics and Contact Frequency. 27 NameDescription General Medical History HYPERTENHypertension HYPERCHOHypercholesterolemia DIABETESDiabetes B12DEFB12 deficiency THYROIDThyroid Disease CVAFIBAtrial fibrillation CVCHFCongestive heart failure CVANGIOAngioplasty/endarterectomy/stent CVBYPASSCardiac bypass procedure Vitals WEIGHTSubject Weight (lbs) HEIGHTSubject Height (inches) BPSYSSubject Blood Pressure (sitting) (systolic) BPDIASSubject Blood Pressure (sitting) (diastolic) HRATESubject resting heart rate (pulse) Table 23: OASIS-3 - Subject Medical History, Cardiovascular Conditions, and Vitals. NameDescription Lifestyle, Sleep & Trauma TOBAC100Smoked more than 100 cigarettes in life? SMOKYRSTotal years smoked ALCOHOLSubstance abuse - alcohol (clinically significant over a 12 month period) TRAUMEXTTraumatic brain injury with extended loss of consciousness (⼠5 minutes) TRAUMCHRTraumatic brain injury with chronic deficit or dysfunction APNEASleep apnea INSOMNHyposomnia/insomnia Sensory & Cognition VISIONWithout corrective lenses, is the subjectâs vision functionally normal? VISCORRDoes the subject usually wear corrective lenses? VISWCORRIs the subjectâs vision functionally normal with corrective lenses? HEARINGWithout a hearing aid(s), is the subjectâs hearing functionally normal? HEARAIDDoes the subject usually wear a hearing aid(s)? HEARWAIDIs the subjectâs hearing functionally normal with a hearing aid(s)? Table 24: OASIS-3 - Lifestyle Factors, Sensory Capabilities. 28