Paper deep dive
Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis
Xia Hu, Brian Potetz, Chun-Ta Lu, Huanfen Yao, Leonidas Guibas, Zhicheng Wang, Howard Zhou, Pengfei Xing, Andrew Gallagher
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 10/4/2026, 3:18:00 AM
Summary
The paper introduces CADET, a diagnostic benchmark and causal decomposition framework for evaluating Multimodal Large Language Models (MLLMs). It addresses the limitation of end-to-end accuracy by isolating intrinsic capability deficits from cascading errors via controlled interventions on prerequisite dependencies. The framework uses Structural Causal Models (SCMs) to define 10 composite tasks decomposed into 46 unit tasks. It employs capability metrics (NC, IC, RC) to diagnose where failures arise and contribution metrics (N-Score, S-Score) to determine why, based on probabilities of causation. Key findings include that cognitive tasks have low natural capability but high intrinsic capability, and that a single critical prerequisite often accounts for the majority of performance gains.
Entities (14)
Relation Signals (13)
CADET → createdby → Google DeepMind
confidence 95% · Authors from Google DeepMind... We instantiate the framework in CADET
CADET → implements → Causal Decomposition Framework
confidence 95% · We instantiate the framework in CADET, a diagnostic benchmark...
Causal Decomposition Framework → uses → Structural Causal Model
confidence 93% · We represent these dependencies as a structural causal model (SCM)
CADET → coverscategories → Cognitive Tasks
confidence 92% · spanning perception, spatial, temporal, and cognitive categories
CADET → coverscategories → perception tasks
confidence 92% · spanning perception, spatial, temporal, and cognitive categories
CADET → coverscategories → Spatial Tasks
confidence 92% · spanning perception, spatial, temporal, and cognitive categories
CADET → coverscategories → Temporal Tasks
confidence 92% · spanning perception, spatial, temporal, and cognitive categories
Causal Decomposition Framework → definesmetrics → S-Score
confidence 90% · contribution metrics (N-Score, S-Score)... quantify each prerequisite's necessity and sufficiency
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:End-to-end accuracy on compositional tasks records how often MLLMs fail, but cannot distinguish whether a failure reflects an intrinsic deficit in the targeted capability or a cascading error from an upstream prerequisite. We propose a causal decomposition framework that isolates these two failure modes through controlled interventions on the prerequisite dependencies of each task. Our capability metrics (NC, IC, RC) score each task under unassisted, correct, or incorrect prerequisites to diagnose where failures arise; contribution metrics (N-Score, S-Score), adapted from probabilities of causation, quantify each prerequisite's necessity and sufficiency to determine why. We instantiate the framework in CADET, a diagnostic benchmark of 10 composite tasks decomposed into 46 unit tasks with over 33,000 human-annotated questions spanning perception, spatial, temporal, and cognitive categories. Diagnosing frontier MLLMs with our framework uncovers systematic patterns that end-to-end accuracy obscures. Capability-wise, supplying correct prerequisites eliminates 54\% of errors on cognitive tasks, lifting them from weakest to above spatial and temporal. Prerequisite-wise, causal contributions are concentrated in a few critical prerequisites, and supplying the single most important one alone captures 84\% of the gain from supplying all prerequisites.
Tags
Links
- Source: https://arxiv.org/abs/2609.38851v1
- Canonical: https://arxiv.org/abs/2609.38851v1
Trouble viewing inline? Open PDF directly →
Full Text
149,167 characters extracted from source content.
Expand or collapse full text
2026-09-30 Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis Xia Hu 1 , Brian Potetz 1 , Chun-Ta Lu 2 , Huanfen Yao 1 , Leonidas Guibas 1,3 , Zhicheng Wang 1 , Howard Zhou 1 , Pengfei Xing 1 and Andrew Gallagher 1 1 Google DeepMind, 2 Google Research, 3 Stanford University End-to-end accuracy on compositional tasks records how often MLLMs fail, but cannot distinguish whether a failure reflects an intrinsic deficit in the targeted capability or a cascading error from an upstream prerequisite. We propose a causal decomposition framework that isolates these two failure modes through controlled interventions on the prerequisite dependencies of each task. Our capability metrics (NC, IC, RC) score each task under unassisted, correct, or incorrect prerequisites to diagnose where failures arise; contribution metrics (N-Score, S-Score), adapted from probabilities of causation, quantify each prerequisite’s necessity and sufficiency to determine why. We instantiate the framework in CADET, a diagnostic benchmark of 10 composite tasks decomposed into 46 unit tasks with over 33,000 human-annotated questions spanning perception, spatial, temporal, and cognitive categories. Diagnosing frontier MLLMs with our framework uncovers systematic patterns that end-to-end accuracy obscures. Capability-wise, supplying correct prerequisites eliminates 54% of errors on cognitive tasks, lifting them from weakest to above spatial and temporal. Prerequisite-wise, causal contributions are concentrated in a few critical prerequisites, and supplying the single most important one alone captures 84% of the gain from supplying all prerequisites. 1. Introduction Multimodal large language models (MLLMs) are increasingly evaluated on compositional tasks requiring multiple interdependent capabilities, from perceiving objects and estimating spatial layouts to tracking over time and reasoning [46,48]. Such tasks are typically scored by a single end-to-end accuracy, which records how often a model fails but not where its failures originate. The score conflates an intrinsic deficit, where the model lacks the targeted capability, with a cascading error, where it possesses that capability but fails on an upstream prerequisite. In Fig 1, a model failing on multi-view sequential may lack temporal reasoning or merely misjudge the camera viewpoint on which ordering depends, yielding the same error but opposite capability diagnoses. More broadly, low accuracy on high-level cognitive tasks is naturally read as a reasoning deficit, a reading that holds only if the perceptual and spatial steps beneath it are sound, and growing evidence suggests they often are not [8, 39]. A common response examines the constituent steps directly, by decomposing the task into separately scored sub-questions [31,52] or inspecting the reasoning trace for the first erroneous step [21]. Both localize failure more finely than an aggregate score, yet observing untreated generation alone cannot disentangle the two failure modes. First, a model’s generated reasoning trace or answers to isolated sub-questions need not reflect what it relies on to solve the full task [5,40]. Second, even a faithful readout reveals only co-occurrence: a prerequisite failing alongside the final task neither shows how much final failure is attributable to it, nor answers the counterfactual had it been solved correctly. Indeed, the outcome need not hinge on the weakest prerequisite: in Fig 1, standalone accuracy is lowest for task-5, yet final performance turns on task-2. Input- level interventions do move beyond observation, supplying ground-truth intermediates [23] or injecting false premises [17], but each tests a single condition, without a dependency structure linking sub-tasks or a measure of each prerequisite’s contribution. What is missing is an evaluation setting every prerequisite’s state within a shared, explicit dependency structure, so that the same controlled contrasts apply across capabilities and models, and each prerequisite’s contribution is measured rather than inferred. We propose a causal decomposition framework that turns MLLM evaluation from observation into a Corresponding author(s): xiahu@google.com © 2026 Google. All rights reserved arXiv:2609.38851v1 [cs.CL] 30 Sep 2026 Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis Q: Given image-A from side-left view and green bus as reference, order the images. ... The order is E, A, D, C, B ❌ Reason & Answer Object localization Q: Where is the green bus Viewpoint Estimation Q: View of 2nd image? Distance Comparison Q: Which is closer to ego-vehicle? Motion Detection Q: Green bus moving or stationary? Interaction Status Q: interaction of Green bus vs. ego? Temporal Ordering Q: Order the image Decomposition Figure 1|An example CADET task (left) decomposed into 6 unit tasks formalized as an SCM (middle). Right: natural capability (NC) of each unit task (top), and their causal contribution (푁/푆(1,∅); bottom) to final task-6, where supplying unit task-2 alone raises final accuracy from 0.27 to 0.65. controlled experiment. We decompose each task into unit tasks with explicit prerequisite dependencies, and intervene on each by supplying a correct answer, an incorrect answer, or nothing. To diagnose where failures arise, we define natural, intrinsic, and resilience capability (NC, IC, RC) under these three prerequisite states, isolating a model’s performance on a target task from the influence of its modeled prerequisites. To explore why the overall outcome fails, we introduce the푁-Score and푆-Score, which measure each prerequisite’s causal necessity and sufficiency for the final outcome: the fraction of success lost when it alone is degraded, and the fraction of failure recovered when it alone is supplied. These metrics adapt classical probabilities of causation [29,38] to the LLM setting without assuming monotonicity, providing valid lower bounds on the probabilities of necessity and sufficiency. To instantiate this framework, we introduce CADET, a multimodal diagnostic benchmark in which each complex task is decomposed via an explicit structural causal model (SCM) into unit tasks that each target a single capability, and every unit-task question is annotated with both a verified correct answer and a plausible incorrect answer to enable three-state interventions. The benchmark spans 10 tasks, yielding 46 unit tasks and over 33,000 unit-task questions across four capability categories: perception, spatial, temporal, and cognitive. All questions and annotations are newly created by human annotators to mitigate data contamination and assure benchmark quality. Evaluation on 12 frontier MLLMs exposes diagnostic patterns that end-to-end accuracy obscures. At capability level, cognitive tasks scores lowest naturally (NC=0.41) yet rises to second under correct prerequisites (IC=0.72), eliminating 54% of its natural errors; whereas spatial and temporal still retain 72–75% of their error, locating the more persistent capability gaps there. Injecting incorrect prerequisites degrades all models on perception and spatial (RC<NC) yet improves cognitive in 10 of 12 models. Turning to causal contributions,푁- and 푆-Scores reveal effective bottleneck sparsity, where a few critical prerequisites drive each task, and that a corrupted prerequisite costs far more than an omitted one. The single highest-푆prerequisite alone captures 84% full-prerequisite gain, and rarely coincides with the lowest-NCprerequisite. Model-wise exhibits distinct diagnostic profiles, especially, the top four achieve close intrinsic capabilities (IC) despite differing inNC, and pair lower degradation loss (푁) with higher supply recovery (푆) than others. Together, our work moves MLLM evaluation beyond scoring how often models fail to diagnosing where and why they fail. 2. Related Work Multimodal Evaluation and Task Decomposition. Existing benchmarks evaluate MLLMs across broad capability taxonomies [22,48,53], while specialized suites target individual skills such as perception and spatial [8,46] or vary visual and textual inputs to test modality reliance [39,49]. Beyond evaluating tasks as a whole, one line of work decomposes complex problems into sub-questions and compares sub-task accuracy with overall performance [31,52]. Another inspects a model’s generated reasoning steps to locate where errors first occur [21,50], though recent studies show that stated chains-of-thought do not always reflect the computation behind the final answer [5,40]. Both read out the model’s unassisted behavior, linking sub-task and final 2 Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis correctness by co-occurrence; we instead set each prerequisite to a known correct or incorrect state. Causal Interventions for Model Diagnosis. Causal interventions are used to diagnose MLLMs and LLMs by modifying specific components at inference time to measure their effect on outputs, such as patching internal activations [25,44] or perturbing visual and textual modalities [19,20]. For multi-step reasoning, recent studies apply step-level interventions to execution traces, editing intermediate outputs or re-executing trajectories to identify failure-inducing steps [2,24,35,41]. At task-input level, prior studies provide ground-truth intermediate results to isolate downstream reasoning [23,36] or inject false premises to test robustness under misleading context [17,26]. These interventions either operate on model internals or single-run trajectories, or test one input condition in isolation; ours act on a task-defined causal graph shared across models. Quantifying Prerequisite Contributions. Attributing MLLM behavior to individual components often involves distinguishing whether a factor is necessary to prevent failure or sufficient to drive success, a distinction formalized by counterfactual necessity and sufficiency [29,38] and adopted in explainable AI for feature-level attribution [9,42]. In LLMs, recent studies adapt these causal concepts to test whether responses on synthetic logic problems conform to the ground-truth rules of the queries [14], or to add missing and prune redundant chain-of-thought steps [47]. Both operate on the model’s own outputs; we instead attribute final-task outcomes to prerequisite via necessity and sufficiency. 3. Causal Decomposition Framework 3.1. Task Decomposition via Structural Causal Models Each evaluation task is decomposed into unit tasks testing a single constituent capability, with directed dependencies where some tasks rely on the outcomes of others. We represent these dependencies as a structural causal model (SCM) [28,30], leveraging the established framework of causal interventions for the analysis below. Definition 3.1 (Task Decomposition SCM). Consider an evaluation task decomposed into푛constituent capabilities. A task decomposition is a structural causal model (SCM) over unit task variables푋 1 , 푋 2 . . . , 푋 푛 , where푋 푖 denotes the outcome of unit task 푖, given by the structural assignment: 푋 푖 := 푓 푖 Pa(푋 푖 ), 푁 푖 , 푖= 1, . . . , 푛,(1) where 푓 푖 is the structural function determining 푋 푖 , Pa(푋 푖 ) ⊆ 푋 1 , . . . , 푋 푛 is the parents set of 푋 푖 , and 푁 푖 is a noise variable capturing unmodeled factors such as the model’s sampling stochasticity. Root nodes (those withPa(푋 푖 )=∅) represent foundational capabilities that can be assessed independently, while the final node 푌:=푋 푛 corresponds to the overall task outcome. 3.2. Causal Intervention Design To disentangle intrinsic capability from cascading upstream influence, and quantify each capability’s causal role, we adopt the interventionist framework of causal inference [28, 29, 34] and introduce a ternary intervention scheme tailored to LLM evaluation. Definition 3.2 (Ternary Intervention). For unit task푋 푖 , we define an intervention퐴 푖 =푎,푎∈1,0,∅: (i)퐴 푖 =1 denotes injecting the correct answer for 푋 푖 into the prompt; (i) 퐴 푖 =0 denotes injecting an incorrect answer for 푋 푖 ; (i) 퐴 푖 =∅ denotes the natural state where 푋 푖 is not injected. Notation. All probabilities are induced by a fixed modelM, whose subscript we suppress. We writeA:=1,0,∅ for the ternary intervention space. For a unit task set푆and푎∈A, we write푑표(A 푆 =푎)for the homogeneous intervention placing every 푋 푗 ∈푆 in state 푎, and abbreviate A −푖 :=A Pa(푌)\푋 푖 . Notably, the distinction between퐴 푖 =∅,퐴 푖 =0 is critical: the former preserves the model’s natural execu- tion path by evaluating it unassisted, where the latter imposes an active counterfactual intervention testing reasoning under a false premise. Since LLM evaluation operates through input manipulation rather than passive observation, this ternary formulation emerges as the natural space of interventions, allowing us to systematically evaluate model behavior across each pairwise contrast of states. 3 Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis Assumption 3.3 (Exogeneity under Intervention-Only Identification). In LLM evaluation setting, the state of a unit task푋 푖 ∈ 1,0admits no faithful observational readout and is identified only through the intervention퐴 푖 =푎. Consequently, probabilities conditioning on푋 푖 =푥for푥 ∈ 1,0are identified with the corresponding interventional probabilities: 푃(· | 푋 푖 = 푥)= 푃(· | do(퐴 푖 = 푥)).(2) Remark 3.4 (Justification). Assumption 3.3 follows from the LLM evaluation setting: the state of푋 푖 admits no faithful observational readout [5,40], thus a reliable mechanism to place푋 푖 in a determinate state 푥∈1, 0 is to inject the corresponding answer into the prompt, making conditioning on 푋 푖 =푥 and intervening do(퐴 푖 = 푥) operationally identical. This condition coincides with the exogeneity assumption푃(푦 푥 )=푃(푦 | 푥)of Pearl[29], Tian and Pearl[38], but with an inverted justification: classical exogeneity invokes the absence of confounders to treat observations as interventions, whereas ours follows from the absence of a faithful observational channel making intervention the reliable state-assignment mechanism. 3.3. Capability Metrics With the ternary intervention mechanism established, we can now place the prerequisitesPa(푋 푖 )of any target task푋 푖 into controlled states and measure how the model’s performance on푋 푖 responds. By additionally evaluating underdo(A Pa(푋 푖 ) =1)anddo(A Pa(푋 푖 ) =0), we isolate the model’s capability on푋 푖 when its declared prerequisites are correctly provided versus actively corrupted. Definition 3.5 (Causal Capability). Let푋 푖 be any task node in the SCM with prerequisite dependenciesPa(푋 푖 ). For an intervention state 푎∈ A, the causal capability ofM on 푋 푖 is Cap(푋 푖 ; 푎) := 푃 푋 푖 =1 | 푑표(A Pa(푋 푖 ) =푎) ,(3) i.e., the probability thatMsolves푋 푖 when all of its prerequisites are placed in state푎. This yields three comple- mentary metrics: (i) natural capabilityNC(푋 푖 ):= Cap(푋 푖 ;∅), the natural performance where no prerequisite is injected; (i) intrinsic capabilityIC(푋 푖 ):= Cap(푋 푖 ; 1), the performance isolated from declared prerequisites; and (i) resilience capability RC(푋 푖 ) := Cap(푋 푖 ; 0), the robustness under corrupted prerequisite context. For root nodes (Pa(푋 푖 )=∅), no upstream intervention applies and onlyNC(푋 푖 )is defined. For non-root nodes, the three metrics disentangle a model’s proficiency on푋 푖 from the influence of modeled prerequisites. Specifically, comparingIC(푋 푖 )withNC(푋 푖 )separates two failure modes: a large gapIC(푋 푖 )−NC(푋 푖 )indicates that the model can solve푋 푖 given correct prerequisites, pointing to upstream cascading errors as the primary bottleneck; a small gap with lowIC(푋 푖 )suggests an intrinsic deficit on푋 푖 even when all modeled prerequisites are satisfied. Finally, comparingRC(푋 푖 )withNC(푋 푖 )shows whether false prerequisites mislead the model below its natural baseline, or still help guide its reasoning despite being factually wrong. 3.4. Contribution Metrics: N-Score and S-Score Building on node-level capability diagnostics, understanding how upstream unit tasks across the SCM affect the final outcome푌requires quantifying their causal contributions. To formalize this causal attribution, we draw upon the probabilities of causation framework [29,38], which evaluates counterfactual causal influence through the Probability of Necessity (PN) and Probability of Sufficiency (PS). We do not assume monotonicity: in LLM reasoning, injecting incorrect answer (퐴 푖 =0) can help more than no injection (퐴 푖 =∅), soRCmay exceed NC (Sec 5.2). Property 3.6 (Absence of Monotonicity). Under the LLM evaluation setting, the causal effect of intervening on a unit task푋 푖 via퐴 푖 on downstream outcome task푌is not assumed to satisfy monotonicity. That is, the natural ordering below is not required to hold in general: 푃(푌=1 | 푑표(퐴 푖 =0)) ≤ 푃(푌=1) ≤ 푃(푌=1 | 푑표(퐴 푖 =1)).(4) Without monotonicity, exactPNandPSvalues cannot be determined from interventional probabilities alone, so we introduce the N-Score and S-Score as observable lower bounds to capture the causal contribution of each prerequisite 푋 푖 ∈ Pa(푌) to the final outcome 푌: 4 Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis Table 1 | Operational semantics of N-Score, S-Score across 4 ordered state contrasts (푎, 푎 ′ ). Contrast N-Score (Success drop from· )S-Score (Failure recovery from· ) (1, 0)corrupting 푋 푖 into a wrong answer in oracle env.fixing 푋 푖 into the correct answer in all-wrong env. (1,∅)leaving 푋 푖 unassisted in oracle env.providing the correct 푋 푖 alone in natural env. (∅, 0)injecting a wrong answer for 푋 푖 in natural env.removing a wrong answer for 푋 푖 in all-wrong env. (0,∅)removing 푋 푖 ’s wrong answer in all-wrong env.injecting a wrong 푋 푖 as a scaffold in natural env. Definition 3.7 (N-Score). Let푌be a target outcome task with prerequisite dependenciesPa(푌), let푋 푖 ∈ Pa(푌), and let (푎, 푎 ′ ) be a pair of distinct states inA. The N-Score of 푋 푖 on 푌 is N (푎,푎 ′ ) (푋 푖 ,푌) := max 0, 푃 푌=1 | 푑표(A Pa(푌) =푎) − 푃 푌=1 | 푑표(퐴 푖 =푎 ′ , A −푖 =푎) 푃 푌=1 | 푑표(A Pa(푌) =푎) ,(5) measuring the fraction of 푌’s success under state 푎 harmed by degrading 푋 푖 alone from 푎 to 푎 ′ . Definition 3.8 (S-Score). Under the setting of Definition 3.7, the S-Score of 푋 푖 on 푌 is S (푎,푎 ′ ) (푋 푖 ,푌) := max 0, 푃 푌=1 | 푑표(퐴 푖 =푎, A −푖 =푎 ′ ) − 푃 푌=1 | 푑표(A Pa(푌) =푎 ′ ) 1− 푃 푌=1 | 푑표(A Pa(푌) =푎 ′ ) ,(6) measuring the fraction of 푌’s failure under state 푎 ′ recovered by upgrading 푋 푖 alone from 푎 ′ to 푎. Theorem 3.9. Under Assumption 3.3 and Property 3.6, N-Score and S-Score provide valid lower bounds on the Probability of Necessity (PN) and Probability of Sufficiency (PS). Specifically, N-Score quantifies causal necessity by measuring how much of the model’s success on푌depends on푋 푖 . A high N-Score pinpoints critical performance bottlenecks where one degraded task breaks overall success even when all other prerequisites remain at푎. Conversely, S-Score quantifies causal sufficiency by measuring how much of the model’s failure on푌can be overcome through푋 푖 . A high S-Score identifies high-leverage prerequisites where supplying a single task rescues overall success even when other prerequisites remain at 푎 ′ . Setting the other prerequisites A −푖 asymmetrically to푎for N-Score and푎 ′ for S-Score aligns with realistic evaluation by testing single-task vulnerability under favorable prerequisite conditions and single-task rescue under degraded ones, making the two metrics complementary rather than directly convertible [29]. We provide the general definitions of N-Score, S-Score under arbitrary A −푖 and theoretical discussions in Appendix. In practice, we evaluate N-Score and S-Score across four ordered state contrasts(푎, 푎 ′ )overA=1,0,∅: (1,0),(1,∅),(∅,0), and(0,∅). As summarized in Tab 1, the first two contrasts measure both degradation from prerequisite corruption or omission and recovery through targeted correction or repair, whereas the latter two evaluate both directions between∅and 0 to capture either misinformation harm or structural scaffolding under Property 3.6. Together, our node-level capability metrics (NC, IC, RC) and edge-level contribution metrics (N-Score, S-Score) form a causal diagnostic framework for complex multimodal tasks, which we instantiate in Sec 4. 4. Benchmark To apply the causal decomposition framework to frontier MLLMs and uncover where and why they fail, we construct CADET, a multimodal benchmark where complex tasks are decomposed into unit tasks connected via SCM and every prerequisite admits controlled interventions (details in Appendix). Benchmark Overview. CADET comprises 10 multimodal evaluation tasks across over 1,100 instances, decom- posed into 46 atomic unit tasks and over 33,000 unit-task questions, with both overall and their constituent unit tasks categorized into four core capabilities: perception, spatial, temporal, and cognitive. To operationalize our causal framework, each overall task is paired with an explicit SCM defining its prerequisite structure, and every unit task is annotated with both a correct and an incorrect answer, along with visual annotations on the images where applicable, to support the ternary intervention scheme (퐴 푖 ∈1,0,∅) at every node. The multimodal inputs span single, multi-images and videos, while the outputs use structured formats (multiple-choice, integer, or list) for 43 of the 46 tasks and free-form text for the remaining three. Task Design and SCM Structure. Tasks are drawn primarily from real-world scenarios and designed as compositional problems requiring interdependent capabilities on which frontier MLLMs are far from saturation, 5 Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis MazeMap Nav.Dice SumSurround Loc.Street-View Ord. Multistep Reason.Music NoteDashcam Count.Multi-View Seq.Traffic Causation Perception Spatial Temporal Cognitive overall task Y Markov chain unit task Figure 2 | Overview of SCMs corresponding to 10 tasks in CADET. so that diagnosis targets gaps these models still face. We further cover diverse input modalities, application domains, and capability compositions, from foundational perception to high-level cognitive reasoning. For each task, we formalize the prerequisite dependencies among its unit tasks into an SCM (Fig 2). These structures range from shallow aggregations (e.g., Dice Sum) to deep multi-path cascades (e.g., Traffic Causation). In decomposition, we identify unit tasks that involve iterative solving of structurally recurring sub-problems, where each step depends on the previous step’s outcome and reasoning depth itself becomes a variable; we define these as Markov chain unit tasks. In the task-level SCM, a Markov chain unit task participates as a single node; for finer-grained analysis, we unfold it into a step-wise chain to track how capabilities evolve with reasoning depth. Tasks whose primary structure is a Markov chain are analyzed separately in Sec 5.4. Data Construction and Quality Assurance. All evaluation content, including questions, answers, and unit-task annotations, is newly created by an experienced rater pool to mitigate data contamination; source images and videos are drawn from publicly available datasets with permissive licenses (e.g., Wilson et al.[43]). The benchmark is constructed in two stages: raters first generate overall task questions and ground-truth answers under predefined task definitions. Based on each task’s SCM, we decompose its unit tasks into standardized atomic annotation tasks, such as bounding box localization and semantic labeling. Raters provide the corresponding visual annotations and ground-truth labels, which are then converted into unit-task questions and correct answers via structured templates, ensuring consistent alignment between all evaluation items and the underlying causal structure. Incorrect answers are randomly sampled from the valid answer space rather than manually crafted, providing a standardized, model-independent perturbation. To ensure benchmark quality, each annotation across both stages is independently verified by a quality-check rater who returns deficient items for iterative revision until acceptance. Beyond per-item verification, instances flagged as ambiguous or un-annotatable are removed during final assembly to ensure reliable causal diagnosis. 5. Experiments and Findings 5.1. Experiment Setting Evaluated Models. We evaluate 12 frontier MLLMs across six model families, covering both proprietary series: Gemini [10], GPT [27], Grok [45]; and open-weight series: Gemma [6], Qwen [32], Kimi [37]. This selection encompasses different model families at varying capability scales, enabling both cross-family and within-family diagnostic comparisons. All models are queried through public APIs with default parameter setting. Thinking mode is set to high or left at its default if not-configurable. Intervention Implementation. To operationalize the ternary intervention, for each prerequisite unit task푋 푖 , we inject its intervened state (퐴 푖 ∈1,0) into prompt as a structured JSON question-answer pair. For퐴 푖 =1, alongside the correct textual answer, we render corresponding visual annotations (e.g., bounding box, path) whenever the prerequisite’s ground truth directly represents spatial information in the scene. For퐴 푖 =0, incorrect answer is injected via text without modifying the image, to prevent shifting the visual referent of the query and ensure we test reasoning under a false premise about the intended target. Under natural state퐴 푖 =∅, no prerequisite information is provided for 푋 푖 . Evaluation Protocol. We report the capability metrics (NC, IC, RC) for each unit task, and the contribution metrics (N-Score, S-Score) on the overall task across four state contrasts:(1,0),(1,∅),(∅,0), and(0,∅). 6 Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis Perc.Spat.Temp.Cog. 0.0 0.2 0.4 0.6 0.8 1.0 Score 0.59 0.21 0.20 0.56 0.12 0.33 0.46 0.15 0.39 0.41 0.32 0.28 (a) Category Error Budget 0.2 0.4 0.6 0.8 1.0 (b) Perception(c) Spatial GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n 0.2 0.4 0.6 0.8 1.0 (d) Temporal GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n (e) Cognitive NCIC-NC1-ICIC / Mean ICNC / Mean NC Figure 3|Natural(NC) and intrinsic(IC) capabilities on non-root unit tasks: (a) category-level NC and IC averaged over 12 models; (b–e) per-model NC and IC within each category. PerceptionSpatialTemporalCognitive −0.4 −0.2 0.0 0.2 RC − NC Gap GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n Gap IQR Cat. Mean Figure 4|Per-modelRC−NCgap across categories on non-root unit tasks. 00.050.150.30.60.9 N(1, ∅) 0 0.05 0.15 0.3 0.6 0.9 S (1, ∅) Category Perception Spatial Temporal Cognitive Overall Task Map Nav. Dice Sum Surround Loc. Street-View Ord. Music Note Dashcam Count. Multi-View Seq. Traffic Causation Top-1 parent high N / high Slow N / high S high N / low Slow N / low S Figure 5|Prerequisite causal contributions 푁/푆(1,∅) to overall task 푌. Model predictions are evaluated via exact match for structured outputs, with Gemini 3.6 Flash serving as LLM judger [51] for free-form text tasks and format mismatches. Full experimental settings are provided in Appendix. 5.2. Capability Diagnostics Cascading errors reorder the apparent capability ranking. Under end-to-end evaluation (NC), accuracy falls across perception, spatial, temporal, and cognitive tasks (0.59, 0.56, 0.46, 0.41; Fig 3(a)), a pattern that would attribute the largest deficits to high-level reasoning. Supplying correct prerequisites (IC) reorders the categories: cognitive reasoning moves to second (0.72), above spatial (0.67) and temporal (0.61), behind only perception (0.80). Decomposing the total error into a cascading componentIC−NCand a residual component 1−ICaccounts for this shift, as cascading errors constitute 54% of cognitive failures (0.32 of 0.59). Spatial and temporal reasoning show the opposite composition, with residual errors (0.33, 0.39) covering 75% and 72% of their failures and small gains from correct prerequisites (0.12, 0.15), locating the more persistent intrinsic capability gaps at these two categories rather than at cognition. Frontier models share similar intrinsic ceilings but differ in cascading errors (Fig 3(b-e)). Supplying correct prerequisites (IC) brings the top 4 SOTA models to nearly identical ceilings across categories (0.88-0.91 in perception, 0.80-0.83 in cognition), tracing their end-to-end (NC) gaps to cascading errors (IC−NC): GPT-5.4 and Gemini family trade places, with larger cascading loss for GPT-5.4 in perception (0.28 vs. 0.15–0.22) and for Gemini in cognition (0.35–0.37 vs. 0.24). Across all 12 models, perception exhibits the widest spread in residual errors (1−IC: 0.09–0.45), spatial shows uniformly high residual errors (0.25–0.43), and cognitive carries uniformly large cascading errors (IC−NC: 0.22–0.37). Model-wise, close NC scores mask divergent error compositions: Qwen3.5 397B and Kimi K2.5 match neighboring models in NC yet carry substantially higher residual errors (1−IC) in different categories. Incorrect prerequisites harm perception and spatial but mildly aid cognition. TheRC−NCgap rises across the hierarchy (Fig 4): all 12 models fall below zero in perception and spatial, while 10 of 12 turn positive in cognition. The gap also varies widely across models within each category (−0.28 to+0.12). The positive 7 Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis 00.050.150.30.60.9 N(1, ∅) 0 0.05 0.15 0.3 0.6 0.9 S (1, ∅) (a) Pattern 1 — SOTA 00.050.150.30.60.9 N(1, ∅) (b) Pattern 1 — Others 00.050.150.30.60.9 N(1, ∅) (c) Pattern 2 — SOTA 00.050.150.30.60.9 N(1, ∅) (d) Pattern 2 — Others Pattern 1:Map Nav.Dice SumMusic NoteDashcam Count. Pattern 2:Multi-View Seq.Street-View Ord.Surround Loc.Traffic Causation Indiv. model Figure 6| 푁/푆(1,∅)across Pattern 1(a,b) and Pattern 2 (c,d) tasks for top-4 SOTA (a,c) and remaining 8 Others (b,d) models (푠푞푟푡(·)-scaled axes, similar in Fig 5). N (1, 0) N (1, ∅) N (0, ∅) N (∅, 0) S (1, 0) S (1, ∅) S (0, ∅) S (∅, 0) 0.0 0.1 0.2 0.3 Score value SOTA vs. Other on N/S Scores SOTA Others Figure 7| 푁/푆-Scores across the four state contrasts for SOTA vs. Others models. 0.00.20.40.60.81.0 NC(X i ) 0.0 0.2 0.4 0.6 0.8 1.0 S (1, ∅) (a) NC vs.S(1, ∅) Pattern 1Pattern 2Overall 20 40 60 80 100 Repairing Gain (%) (b) Single Parent Repair Gain (%) Map Nav Multi-View Seq Dice Sum Street-View Ord Music Note Surround Loc Dashcam Count Traffic Causation all pa(Y) repaired S(1, ∅) N(1, ∅) min(NC) min(IC) max(IC−NC) random Figure 8|(a)NC(푋 푖 )vs.푆(1,∅); (b) percentage of full all-prerequisite gain(IC−NC) recovered by single-parent supply across selection criteria. cognitive gaps directly instantiate the non-monotonicity formalized in Property 3.6: incorrect prerequisites do not uniformly degrade performance. 5.3. Causal Contribution Analysis Causal contributions exhibit effective bottleneck sparsity. Evaluating prerequisite contributions via푁(1,∅) and푆(1,∅)(Fig 5) across 8 non-Markov tasks shows that causal influence is highly uneven across parents, with each task driven by one or a few critical prerequisites. These critical parents act either as core bottlenecks that cause large oracle drops when omitted and strong natural recovery when provided (high푁/high푆), or as compensable levers whose omission is offset by other oracle prerequisites yet still rescue natural failures alone (low푁/high푆). By contrast, the remaining prerequisites have individually small effects (low푁/low푆), while conjunctive bottlenecks that hurt oracle performance without aiding natural recovery are nearly absent (high 푁/low 푆). Disaggregating by model reveals two distinct criticality patterns. Expanding to individual models (Fig 6) reveals two task patterns and a broad split between top-4 (SOTA) and remaining (Others) models. Under Pattern 1 (Fig 6a,b), both groups exhibit critical prerequisites with high single-point recovery capacity (high 푆), with Others shifting moderately toward higher푁and lower푆. Under Pattern 2 (Fig 6c,d), SOTA keeps prerequisites at high푆with low oracle omission sensitivity (low푁), whereas Others shift substantially toward higher푁and lower푆, losing effective single-point recoverability while suffering larger oracle drops from single-task omission. Across both patterns, SOTA maintains low oracle vulnerability and often has multiple high-푆prerequisites per task that each could rescue natural failures alone; while Others drift rightward and downward, moderately in Pattern 1 and severely in Pattern 2, reflecting higher oracle sensitivity and weaker single-task recovery. Structurally, the prerequisite count|Pa(푌)|averages 3.0 in Pattern 1 and 5.25 in Pattern 2. A single supplied prerequisites recovers most full-prerequisite gains. Supplying only the single most sufficient prerequisite (max푆(1,∅)) recovers 84% of the full all-prerequisite gain (IC−NC) overall (Fig 8b), showing that one targeted intervention already accounts for most of the attainable gain. This leverage is not readable from standalone accuracy:푆(1,∅)does not trackNC(푋 푖 )across prerequisites (Fig 8a). To test whether cheaper node-level signals could substitute for푆(1,∅)in identify the same prerequisite, we repeat single-parent 8 Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis Perc.Spat.Temp.Cog. N(1, 0) N(1, ∅) N(0, ∅) N(∅, 0) S(1, 0) S(1, ∅) S(0, ∅) S(∅, 0) 0.2010.1990.1540.509 0.0850.1070.1060.398 0.0850.1110.1230.027 0.1090.2770.1480.317 0.0680.1530.1030.506 0.2120.2580.1840.465 0.0280.0350.0710.002 0.0240.0340.0410.127 N/S Scores Heatmap 0.0 0.1 0.2 0.3 0.4 0.5 Figure 9|Category-level pre- requisite푁/푆-Scores across 4 state contrasts. 0.0 0.2 0.4 0.6 NS Score 0.0 0.2 0.4 0.6 NS Score 123456+ 0.00 0.25 0.50 0.75 NC / IC / RC (a) Map Navigation 12345678910 (b) Maze Pathfinding 123456789 101112131415 16+ Step t 0.00 0.25 0.50 0.75 NC / IC / RC (c) Multistep Position 123456789 101112131415 16+ Step t (d) Multistep Action NC IC RC NS(1, ∅) NS(0, ∅) Figure 10|Step-wise capabilities (NC, IC, RC) and causal contributions (푁푆(1,∅), 푁푆(0,∅)) along step 푡 across Markov chain unit tasks. repair under alternative criteria (including the weakest prerequisitemin(NC); Fig 8b), none of which recovers this single-parent optimum overall. The shortfall ofNCstays around 8 p in Pattern 1 tasks, where models share a single dominant bottleneck; but widens sharply in Pattern 2 tasks with denser multi-parent dependencies, where 푆(1,∅)-chosen parent sustains 76% of the full gain while NC-based only 36%. Incorrect prerequisites hurt far more than omission. Extending to full ternary contrast space (Fig 7,9), injecting incorrect prerequisites is consistently more destructive than leaving them unassisted (푁(1,0)>푁(1,∅) across all categories), with minor gains from incorrect answers on temporal (푆(0,∅)=0.071). Unlike cognitive target tasks benefiting from incorrect prerequisites (Sec 5.2), the two cognitive prerequisites are highly sensitive to correctness: corrupting or omitting them causes the steepest overall task drops (푁(1,0), 푁(1,∅)) with no gain from incorrect answers (푆(0,∅)), while fixing or removing them drives strong recovery (푆(1,0),푆(∅,0)). SOTA models (Fig 7) pair uniformly lower푁with higher푆than Others across all four contrasts, showing stronger robustness to single-prerequisite degradation with superior single-point recovery even under corruption. 5.4. Markov Chain Unit Task Analysis We analyze the Markov chain unit tasks (Sec 4), in which each푋 푡 depends solely on its predecessor푋 푡−1 . Under this single-parent structure, N, S-Score converge into a single NS-Score (푁푆(푎, 푎 ′ )=푁(푎, 푎 ′ )·Cap(푋 푡 ;푎); Appendix). Fig 10 reportsNC,IC,RCand푁푆score along step푡. First, capability degrades nonlinearly with a single-step effective horizon: in goal-directed tasks,ICand푁푆(1,∅)remain flat until one step from the exit (Fig 10b,푡=10); symmetrically, in process-cumulative tasks whereNCcollapses to a flat floor immediately at푡=1, ICand푁푆(1,∅)peak at the first step (Fig 10c) and gradually decline with depth. Second, alternating Position (Fig 10c) and Action (Fig 10d) nodes within the same chain respond divergently to incorrect prerequisites: 푁푆(0,∅)≈0 for spatial tracking versus substantial for action planning, consistent with the finding in Sec 5.2 that incorrect prerequisites benefit cognitive reasoning but not perception. Third, as depth푡grows, Position continues to require factual correctness (푁푆(0,∅) ≤0.06≪ NS(1,∅)), whereas in cognitive Action,푁푆(0,∅) steadily rises from 0.12 (푡=1) to 0.30 (푡≥14) to matchNS(1,∅)(RC≈IC), showing that explicit prerequisite conditioning, regardless of correctness, increasingly accounts for cognitive recovery at deeper horizons. 6. Conclusion We introduced a causal decomposition framework and the CADET benchmark, applying controlled ternary interventions over explicit prerequisite structures to diagnose where and why MLLMs fail. Our capability metrics (NC, IC, RC) separate cascading influence from intrinsic deficits, showing that supplying correct prerequisites substantially lifts cognitive tasks while spatial and temporal reasoning retain persistent intrinsic gaps, and that incorrect prerequisites often still aid cognition. The푁-,푆-Scores reveal effective bottleneck sparsity, where a few critical prerequisites drive each task, and supplying the single most sufficient one recovers most full-prerequisite gain yet rarely coincides with the least accurate prerequisite. Across models, top ones exhibit lower degradation alongside stronger single-point recovery. More broadly, our diagnostic framework can inform 9 Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis future improvements, from inference-time CoT guidance, query rewriting to training and multi-step system design. References [1]A. Balke and J. Pearl. Counterfactual probabilities: Computational methods, bounds and applications. In Uncertainty in artificial intelligence, pages 46–54. Elsevier, 1994. [2]A. Bonagiri, D. Borkar, G. J. Anderias, S. Rafatirad, and H. Homayoun. Causalflow: Causal attribution and counterfactual repair for llm agent failures. arXiv preprint arXiv:2605.25338, 2026. [3] J. Chen, T. Liang, S. Siu, Z. Wang, K. Wang, Y. Wang, Y. Ni, Z. Jiang, W. Zhu, B. Lyu, et al. Mega-bench: Scaling multimodal evaluation to over 500 real-world tasks. In International Conference on Learning Representations, volume 2025, pages 28384–28494, 2025. [4] L. Chen, W. Xie, Y. Liang, H. He, H. Zhao, Z. Yang, Z. Huang, H. Wu, H. Lu, Y. Bao, et al. Babyvision: Visual reasoning beyond language. arXiv preprint arXiv:2601.06521, 2026. [5]Y. Chen, J. Benton, A. Radhakrishnan, J. Uesato, C. Denison, J. Schulman, A. Somani, P. Hase, M. Wagner, F. Roger, et al. Reasoning models don’t always say what they think. arXiv preprint arXiv:2505.05410, 2025. [6] G. DeepMind. Gemma 4: Our most intelligent open models to date.https://blog.google/innova tion-and-ai/technology/developers-tools/gemma-4/, 2026. Accessed: 2026-05-01. [7] M. Fréchet. Généralisation du théoreme des probabilités totales. Fundamenta mathematicae, 25:379–387, 1935. [8]X. Fu, Y. Hu, B. Li, Y. Feng, H. Wang, X. Lin, D. Roth, N. A. Smith, W.-C. Ma, and R. Krishna. Blink: Multimodal large language models can see but not perceive. In European Conference on Computer Vision, pages 148–166. Springer, 2024. [9]S. Galhotra, R. Pradhan, and B. Salimi. Explaining black-box algorithms using probabilistic contrastive counterfactuals. In Proceedings of the 2021 international conference on management of data, pages 577–590, 2021. [10] Gemini Team. Gemini 3.1 pro: A smarter model for your most complex tasks. 2026. URLhttps://blog .google/innovation-and-ai/models-and-research/gemini-models/gemini-3-1-pro/. [11]Gemini Team and Google DeepMind. Gemini 3 flash: Frontier intelligence built for speed. Technical blog / model release, Google, 12 2025. URLhttps://blog.google/products-and-platforms/produ cts/gemini/gemini-3-flash/. [12] Gemini Team and Google DeepMind. Gemini 3.1 flash-lite: Built for intelligence at scale. Technical blog / model release, Google, 3 2026. URLhttps://blog.google/innovation-and-ai/models-and -research/gemini-models/gemini-3-1-flash-lite/. [13] A. Ghosh, S. Zheng, R. Tamburo, K. Vuong, J. Alvarez-Padilla, H. Zhu, M. Cardei, N. Dunn, C. Mertz, and S. G. Narasimhan. Roadwork: A dataset and benchmark for learning to recognize, observe, analyze and drive through work zones. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), pages 6132–6142. IEEE, 2025. [14]J. González and A. V. Nori. Does reasoning emerge? examining the probabilities of causation in large language models. Advances in Neural Information Processing Systems, 37:117737–117761, 2024. [15]Google DeepMind. Gemini 3.5 flash: Model card, 2026. URLhttps://deepmind.google/models/m odel-cards/gemini-3-5-flash/. Accessed: 2026-08-18. [16]Google DeepMind. Gemini 3.6 flash: Model card, 2026. URLhttps://deepmind.google/models/m odel-cards/gemini-3-6-flash/. 10 Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis [17]T. Guan, F. Liu, X. Wu, R. Xian, Z. Li, X. Liu, X. Wang, L. Chen, F. Huang, Y. Yacoob, et al. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision- language models. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14375–14385. IEEE, 2024. [18] Y. Hao, J. Gu, H. W. Wang, L. Li, Z. Yang, L. Wang, and Y. Cheng. Can mllms reason in multimodality? emma: An enhanced multimodal reasoning benchmark. arXiv preprint arXiv:2501.05444, 2025. [19] V. Hasic, C. Wang, L. C. Garcia-Peraza-Herrera, D. Watson, and S. Krivic. Which modality decides? counterfactual modality attribution for multimodal llms. arXiv preprint arXiv:2608.00076, 2026. [20]L. Li, J. Qu, L. Song, Y. Zhou, Y. Qin, T. Yang, and Y. Zhao. Treble counterfactual vlms: A causal approach to hallucination. Association for Computational Linguistics, 2025. [21]H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe. Let’s verify step by step. In International Conference on Learning Representations, volume 2024, pages 39578–39601, 2024. [22]Y. Liu, H. Duan, Y. Zhang, B. Li, S. Zhang, W. Zhao, Y. Yuan, J. Wang, C. He, Z. Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216–233. Springer, 2024. [23] P. Lu, H. Bansal, T. Xia, J. Liu, C. Li, H. Hajishirzi, H. Cheng, K.-W. Chang, M. Galley, and J. Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representations, volume 2024, pages 23439–23554, 2024. [24]M. Ma, J. Zhang, F. Yang, Y. Kang, Q. Lin, S. Rajmohan, and D. Zhang. Dover: Intervention-driven auto debugging for llm multi-agent systems. In International Conference on Learning Representations, volume 2026, pages 28059–28087, 2026. [25]K. Meng, D. Bau, A. Andonian, and Y. Belinkov. Locating and editing factual associations in gpt. Advances in neural information processing systems, 35:17359–17372, 2022. [26]Y. Ming, S. Purushwalkam, S. Pandit, Z. Ke, X.-P. Nguyen, C. Xiong, and S. Joty. Faitheval: Can your language model stay faithful to context, even if" the moon is made of marshmallows". In International Conference on Learning Representations, volume 2025, pages 29430–29456, 2025. [27] OpenAI. Introducing GPT-5.4, 2026. URL https://openai.com/index/introducing-gpt-5-4/. [28] J. Pearl. Causality: Models, reasoning, and inference. 2009. [29] J. Pearl. Probabilities of causation: three counterfactual interpretations and their identification. In Probabilistic and causal inference: the works of Judea Pearl, pages 317–372. 2022. [30]J. Peters, D. Janzing, and B. Schölkopf. Elements of causal inference: foundations and learning algorithms. The MIT press, 2017. [31]O. Press, M. Zhang, S. Min, L. Schmidt, N. A. Smith, and M. Lewis. Measuring and narrowing the compositionality gap in language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5687–5711, 2023. [32]Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026. URLhttps://qwen.ai/bl og?id=qwen3.5. [33] Qwen Team. Qwen3.6-27B: Flagship-level coding in a 27B dense model, April 2026. URLhttps: //qwen.ai/blog?id=qwen3.6-27b. [34]D. B. Rubin. Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of educational Psychology, 66(5):688, 1974. [35]J. Shah. Causal agent replay: Counterfactual attribution for llm-agent failures. arXiv preprint arXiv:2606.08275, 2026. 11 Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis [36]H. Shao, S. Qian, H. Xiao, G. Song, Z. Zong, L. Wang, Y. Liu, and H. Li. Visual cot: Advancing multi-modal language models with a comprehensive dataset and benchmark for chain-of-thought reasoning. Advances in Neural Information Processing Systems, 37:8612–8642, 2024. [37]K. Team, T. Bai, Y. Bai, Y. Bao, S. Cai, Y. Cao, Y. Charles, H. Che, C. Chen, G. Chen, et al. Kimi k2. 5: Visual agentic intelligence. arXiv preprint arXiv:2602.02276, 2026. [38]J. Tian and J. Pearl. Probabilities of causation: Bounds and identification. Annals of Mathematics and Artificial Intelligence, 28(1):287–313, 2000. [39] S. Tong, Z. Liu, Y. Zhai, Y. Ma, Y. LeCun, and S. Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9568–9578. IEEE, 2024. [40]M. Turpin, J. Michael, E. Perez, and S. Bowman. Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting. Advances in Neural Information Processing Systems, 36:74952–74965, 2023. [41]Y. Wang, W. Wu, J. Wang, and Q. Wang. From flat logs to causal graphs: Hierarchical failure attribution for llm-based multi-agent systems. arXiv preprint arXiv:2602.23701, 2026. [42]D. S. Watson, L. Gultchin, A. Taly, and L. Floridi. Local explanations via necessity and sufficiency: Unifying theory and practice. In Uncertainty in Artificial Intelligence, pages 1382–1392. PMLR, 2021. [43]B. Wilson, W. Qi, T. Agarwal, J. Lambert, J. Singh, S. Khandelwal, B. Pan, R. Kumar, A. Hartnett, J. K. Pontes, D. Ramanan, P. Carr, and J. Hays. Argoverse 2: Next generation datasets for self-driving perception and forecasting, 2023. URL https://arxiv.org/abs/2301.00493. [44] Z. Wu, A. Geiger, T. Icard, C. Potts, and N. Goodman. Interpretability at scale: Identifying causal mechanisms in alpaca. Advances in neural information processing systems, 36:78205–78226, 2023. [45] xAI. Grok-4 model card. 2025. URL https://x.ai/news/grok-4. [46] J. Yang, S. Yang, A. W. Gupta, R. Han, L. Fei-Fei, and S. Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10632–10643. IEEE, 2025. [47]X. Yu, Z. Wang, L. Yang, H. Li, A. Liu, X. Xue, J. Wang, and M. Yang. Causal sufficiency and necessity improves chain-of-thought reasoning. Advances in Neural Information Processing Systems, 38:126109– 126141, 2026. [48] X. Yue, Y. Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y. Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9556–9567, 2024. [49]R. Zhang, D. Jiang, Y. Zhang, H. Lin, Z. Guo, P. Qiu, A. Zhou, P. Lu, K.-W. Chang, Y. Qiao, et al. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pages 169–186. Springer, 2024. [50]S. Zhang, M. Yin, J. Zhang, J. Liu, Z. Han, J. Zhang, B. Li, C. Wang, H. Wang, Y. Chen, et al. Which agent causes task failures and when? on automated failure attribution of llm multi-agent systems. arXiv preprint arXiv:2505.00212, 2025. [51]L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA, 2023. Curran Associates Inc. [52] Q. Zhou, Y. Gong, G. Bao, H. Qiu, J. Li, X. Zhu, H. Zhang, and Y. Zhang. Reasoning is all you need for video generalization: A counterfactual benchmark with sub-question evaluation. In Findings of the Association for Computational Linguistics: ACL 2025, pages 2939–2957, 2025. 12 Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis [53]Z. Zhou, S. Liu, M. Ning, W. Liu, J. Wang, D. Wong, X. Huang, Q. Wang, and K. Huang. Is your model really a good math reasoner? evaluating mathematical reasoning with checklist. In International Conference on Learning Representations, volume 2025, pages 34238–34281, 2025. 13 Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis A. Details and Proofs of Causal Decomposition Framework In this section, we provide the general definitions of N-Score and S-Score under arbitrary prerequisite states, followed by the theoretical proofs and properties of our causal decomposition framework. The do-operator. We adopt the standard do-operator (Section 3.2; Pearl[28]). For푎∈ 1,0,푑표(퐴 푖 =푎) replaces the structural assignment of푋 푖 in(1)with푋 푖 := 푎while keeping all other assignments unchanged, and푑표(퐴 푖 =∅)leaves the assignment of푋 푖 intact; interventions on multiple unit tasks apply these operations jointly. 푃(· | 푑표(·)) denotes the distribution induced by the resulting SCM. A.1. General Definition of N-Score, S-Score In Section 3.4 (Definitions 3.7 and 3.8), when quantifying the causal contribution of a prerequisite unit task푋 푖 ∈ Pa(푌)to the target outcome푌across an ordered contrast(푎, 푎 ′ ), we specifically set the remaining prerequisites A −푖 :=A Pa(푌)\푋 푖 to the homogeneous state푎for N-Score and푎 ′ for S-Score. Here, we generalize this formulation by allowing the remaining prerequisites to take any arbitrary intervention state configuration. Specifically, let c∈ A |Pa(푌)\푋 푖 | denote an arbitrary intervention state configuration assigned to A −푖 , denoted as푑표(A −푖 =c), which sets the intervention state of each prerequisite푋 푗 ∈ Pa(푌)\푋 푖 to퐴 푗 = 푐 푗 with푐 푗 ∈ A. Following the notation in Section 3.2, when all unit tasks in A −푖 are placed in a single homogeneous state 푐 ∈ A(i.e.,푐 푗 = 푐for all푋 푗 ∈ Pa(푌)\푋 푖 ), we write푑표(A −푖 = 푐)without boldface. Based on this general state configuration c for the remaining prerequisites, we define the general N-Score and S-Score as follows. Definition A.1 (General N-Score). Let푌be a target outcome task with prerequisite dependenciesPa(푌), let 푋 푖 ∈ Pa(푌), and let(푎, 푎 ′ )be an ordered pair of distinct states inA. For any intervention state configuration c∈ A |Pa(푌)\푋 푖 | assigned to the remaining prerequisites A −푖 , the general N-Score of 푋 푖 on 푌 is defined as: N (푎,푎 ′ ) (푋 푖 ,푌 | c) := max 0, 푃 푌=1 | 푑표(퐴 푖 =푎, A −푖 =c) − 푃 푌=1 | 푑표(퐴 푖 =푎 ′ , A −푖 =c) 푃 푌=1 | 푑표(퐴 푖 =푎, A −푖 =c) ,(7) measuring the fraction of 푌’s success under 푑표(퐴 푖 =푎, A −푖 =c) that is lost when 푋 푖 alone is degraded from 푎 to 푎 ′ . Definition A.2 (General S-Score). Under the setting of Definition A.1, for any intervention state configuration c∈ A |Pa(푌)\푋 푖 | assigned to A −푖 , the general S-Score of 푋 푖 on 푌 is defined as: S (푎,푎 ′ ) (푋 푖 ,푌 | c) := max 0, 푃 푌=1 | 푑표(퐴 푖 =푎, A −푖 =c) − 푃 푌=1 | 푑표(퐴 푖 =푎 ′ , A −푖 =c) 1− 푃 푌=1 | 푑표(퐴 푖 =푎 ′ , A −푖 =c) ,(8) measuring the fraction of푌’s failure under푑표(퐴 푖 =푎 ′ ,A −푖 =c)that is recovered when푋 푖 alone is upgraded from푎 ′ to 푎. Under these general definitions, the N-Score (Definition 3.7) and S-Score (Definition 3.8) introduced in Section 3.4 correspond to the special cases where every element of c is set to the homogeneous state푐 푗 = 푎 (i.e., 푑표(A −푖 = 푎)) for N-Score and 푐 푗 = 푎 ′ (i.e., 푑표(A −푖 = 푎 ′ )) for S-Score, respectively. A.2. Lower Bound Guarantee of N-Score and S-Score This section provides the formal statement and proof of Theorem 3.9 in Section 3.4. We establish the lower- bound guarantee for the general N-Score and S-Score (Definitions A.1 and A.2) under an arbitrary prerequisite configuration c (Theorems A.3 and A.4), where Theorem 3.9 corresponds to the special cases c= 푎for N-Score and c= 푎 ′ for S-Score. Theorem A.3. Under Assumption 3.3 (Exogeneity), for any prerequisite unit task푋 푖 ∈ Pa(푌), any ordered contrast (푎, 푎 ′ )of distinct states inA, and any intervention state configuration c∈ A |Pa(푌)\푋 푖 | assigned to the remaining prerequisites A −푖 , the general N-Score (Definition A.1) provides a valid lower bound on the Probability of Necessity (PN) under prerequisite context 푑표(A −푖 =c): PN (푎,푎 ′ ) (푋 푖 ,푌 | c) ≥ N (푎,푎 ′ ) (푋 푖 ,푌 | c)(9) In particular, setting every element of c to the homogeneous state푎(i.e.,푑표(A −푖 =푎)) directly establishes the N-Score lower bound in Theorem 3.9. 14 Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis Proof.Note that under exogeneity and non-monotonicity, the mathematical technique of bounding probabilities of causation via Fréchet inequalities [7] is established in the foundational works of Balke and Pearl[1], Pearl [28,29], Tian and Pearl[38]. Here, we adapt and instantiate their derivation step-by-step to demonstrate how it applies to our multi-parent LLM evaluation setting under prerequisite configuration푑표(A −푖 =c), Assumption 3.3, and Property 3.6. For notational conciseness under the prerequisite intervention푑표(A −푖 =c), in below proof, let푃 c (·):= 푃(· | 푑표(A −푖 =c))denote the probability measure in the induced submodelM c [28], where푌 푎 and푌 푎 ′ denote the potential outcomes of푌under further regime interventions푑표(퐴 푖 =푎)and푑표(퐴 푖 =푎 ′ )withinM c . Following Pearl [29], the Probability of Necessity (PN) across contrast (푎, 푎 ′ ) under context c is defined as the counterfactual probability that 푌 fails under 푑표(퐴 푖 =푎 ′ ), given that 푌 succeeded under regime 퐴 푖 =푎: PN (푎,푎 ′ ) (푋 푖 ,푌 | c) := 푃 c (푌 푎 ′ =0 | 퐴 푖 =푎, 푌=1) If푃 c (퐴 푖 =푎, 푌=1)=0, the success baseline is zero and the bound holds trivially by convention (N (푎,푎 ′ ) =0). Thus, assuming 푃 c (퐴 푖 =푎, 푌=1) > 0, we first apply the definition of conditional probability: 푃 c (푌 푎 ′ =0 | 퐴 푖 =푎, 푌=1)= 푃 c (푌 푎 ′ =0, 퐴 푖 =푎, 푌=1) 푃 c (퐴 푖 =푎, 푌=1) By the consistency property of structural causal models, the observed outcome under regime퐴 푖 =푎coincides with the potential outcome 푌 푎 . Substituting 푌= 푌 푎 into the numerator yields: 푃 c (푌 푎 ′ =0 | 퐴 푖 =푎, 푌=1)= 푃 c (푌 푎 ′ =0, 퐴 푖 =푎, 푌 푎 =1) 푃 c (퐴 푖 =푎, 푌=1) Next, we factor out the marginal regime probability푃 c (퐴 푖 =푎)in the numerator and divide both the numerator and the denominator by it: 푃 c (푌 푎 ′ =0 | 퐴 푖 =푎, 푌=1)= 푃 c (푌 푎 ′ =0, 푌 푎 =1 | 퐴 푖 =푎) 푃 c (퐴 푖 =푎) 푃 c (퐴 푖 =푎, 푌=1) = 푃 c (푌 푎 ′ =0, 푌 푎 =1 | 퐴 푖 =푎) 푃 c (푌=1 | 퐴 푖 =푎) Under the exogeneity of the regime assignment (Assumption 3.3), the potential outcomes(푌 푎 ,푌 푎 ′ ) are indepen- dent of퐴 푖 , and conditioning on퐴 푖 is operationally identical to intervening on it. This removes the conditioning event in both the numerator and the denominator: 푃 c (푌 푎 ′ =0 | 퐴 푖 =푎, 푌=1)= 푃 c (푌 푎 ′ =0, 푌 푎 =1) 푃 c (푌 푎 =1) Since we do not assume monotonicity (Property 3.6) in LLM reasoning, the joint counterfactual probability 푃 c (푌 푎 ′ = 0, 푌 푎 =1)cannot be point-identified from marginal probabilities alone. Applying the classical Fréchet lower bound 푃 c (퐸 1 , 퐸 2 ) ≥ max0, 푃 c (퐸 1 )+ 푃 c (퐸 2 )− 1 [29] to the numerator gives: PN (푎,푎 ′ ) (푋 푖 ,푌 | c) ≥ max 0, 푃 c (푌 푎 ′ =0)+ 푃 c (푌 푎 =1)− 1 푃 c (푌 푎 =1) Using the complement rule 푃 c (푌 푎 ′ =0)= 1− 푃 c (푌 푎 ′ =1) gives: PN (푎,푎 ′ ) (푋 푖 ,푌 | c) ≥ max 0, 푃 c (푌 푎 =1)− 푃 c (푌 푎 ′ =1) 푃 c (푌 푎 =1) Finally, unpacking the submodel notation back into full interventional probabilities via푃 c (푌 푎 =1)= 푃(푌=1| 푑표(퐴 푖 =푎, A −푖 =c)) and 푃 c (푌 푎 ′ =1)= 푃(푌=1 | 푑표(퐴 푖 =푎 ′ , A −푖 =c)) yields: PN (푎,푎 ′ ) (푋 푖 ,푌 | c) ≥ max 0, 푃(푌=1 | 푑표(퐴 푖 =푎, A −푖 =c))− 푃(푌=1 | 푑표(퐴 푖 =푎 ′ , A −푖 =c)) 푃(푌=1 | 푑표(퐴 푖 =푎, A −푖 =c)) Thus PN (푎,푎 ′ ) (푋 푖 ,푌 | c) ≥ N (푎,푎 ′ ) (푋 푖 ,푌 | c). □ 15 Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis Theorem A.4. Under Assumption 3.3 (Exogeneity), for any prerequisite unit task푋 푖 ∈ Pa(푌), any ordered contrast (푎, 푎 ′ )of distinct states inA, and any intervention state configuration c∈ A |Pa(푌)\푋 푖 | assigned to the remaining prerequisites A −푖 , the general S-Score (Definition A.2) provides a valid lower bound on the Probability of Sufficiency (PS) under prerequisite context 푑표(A −푖 =c): PS (푎,푎 ′ ) (푋 푖 ,푌 | c) ≥ S (푎,푎 ′ ) (푋 푖 ,푌 | c)(10) In particular, setting every element of c to the homogeneous state푎 ′ (i.e.,푑표(A −푖 =푎 ′ )) directly establishes the S-Score lower bound in Theorem 3.9. Proof.Similar to the proof of Theorem A.3, let푃 c (·):= 푃(· | 푑표(A −푖 =c))denote the probability measure in the submodelM c , where푌 푎 and푌 푎 ′ represent the potential outcomes of푌under푑표(퐴 푖 =푎)and푑표(퐴 푖 =푎 ′ ). Following Pearl[29], the Probability of Sufficiency (PS) across contrast(푎, 푎 ′ )under context c measures the counterfactual probability that upgrading푋 푖 to state푎would rescue푌to success, given that푌failed under regime 퐴 푖 =푎 ′ : PS (푎,푎 ′ ) (푋 푖 ,푌 | c) := 푃 c (푌 푎 =1 | 퐴 푖 =푎 ′ , 푌=0) If푃 c (퐴 푖 =푎 ′ , 푌=0)=0, the failure baseline is zero and the bound holds trivially by convention (S (푎,푎 ′ ) =0). Thus, assuming푃 c (퐴 푖 =푎 ′ , 푌=0) >0, we expand this conditional counterfactual probability using the definition of conditional probability: 푃 c (푌 푎 =1 | 퐴 푖 =푎 ′ , 푌=0)= 푃 c (푌 푎 =1, 퐴 푖 =푎 ′ , 푌=0) 푃 c (퐴 푖 =푎 ′ , 푌=0) By consistency, the observed outcome under regime퐴 푖 =푎 ′ coincides with the potential outcome푌 푎 ′ , so we replace the observed failure 푌=0 in the numerator with 푌 푎 ′ =0: 푃 c (푌 푎 =1 | 퐴 푖 =푎 ′ , 푌=0)= 푃 c (푌 푎 =1, 퐴 푖 =푎 ′ , 푌 푎 ′ =0) 푃 c (퐴 푖 =푎 ′ , 푌=0) Factoring out 푃 c (퐴 푖 =푎 ′ ) in the numerator and dividing both the numerator and the denominator by it yields: 푃 c (푌 푎 =1 | 퐴 푖 =푎 ′ , 푌=0)= 푃 c (푌 푎 =1, 푌 푎 ′ =0 | 퐴 푖 =푎 ′ ) 푃 c (퐴 푖 =푎 ′ ) 푃 c (퐴 푖 =푎 ′ , 푌=0) = 푃 c (푌 푎 =1, 푌 푎 ′ =0 | 퐴 푖 =푎 ′ ) 푃 c (푌=0 | 퐴 푖 =푎 ′ ) Under the exogeneity of the regime assignment (Assumption 3.3), the potential outcomes(푌 푎 ,푌 푎 ′ ) are indepen- dent of퐴 푖 , and conditioning on퐴 푖 is operationally identical to intervening on it. This removes the conditioning event in both the numerator and the denominator: 푃 c (푌 푎 =1 | 퐴 푖 =푎 ′ , 푌=0)= 푃 c (푌 푎 =1, 푌 푎 ′ =0) 1− 푃 c (푌 푎 ′ =1) Since monotonicity is not assumed (Property 3.6), the joint counterfactual probability푃 c (푌 푎 =1, 푌 푎 ′ = 0)is again not point-identified. Applying the Fréchet lower bound푃 c (푌 푎 =1, 푌 푎 ′ = 0) ≥ max0, 푃 c (푌 푎 =1)+ 푃 c (푌 푎 ′ = 0)−1 [29] to the numerator yields: PS (푎,푎 ′ ) (푋 푖 ,푌 | c) ≥ max 0, 푃 c (푌 푎 =1)+ 푃 c (푌 푎 ′ =0)− 1 1− 푃 c (푌 푎 ′ =1) Substituting 푃 c (푌 푎 ′ =0)= 1− 푃 c (푌 푎 ′ =1) into the numerator simplifies the fraction to: PS (푎,푎 ′ ) (푋 푖 ,푌 | c) ≥ max 0, 푃 c (푌 푎 =1)− 푃 c (푌 푎 ′ =1) 1− 푃 c (푌 푎 ′ =1) Finally, unpacking the submodel notation푃 c (푌 푎 =1)= 푃(푌=1| 푑표(퐴 푖 =푎,A −푖 =c))and푃 c (푌 푎 ′ =1)= 푃(푌=1| 푑표(퐴 푖 =푎 ′ , A −푖 =c)) establishes the general S-Score lower bound: PS (푎,푎 ′ ) (푋 푖 ,푌 | c) ≥ max 0, 푃(푌=1 | 푑표(퐴 푖 =푎, A −푖 =c))− 푃(푌=1 | 푑표(퐴 푖 =푎 ′ , A −푖 =c)) 1− 푃(푌=1 | 푑표(퐴 푖 =푎 ′ , A −푖 =c)) Thus PS (푎,푎 ′ ) (푋 푖 ,푌 | c) ≥ S (푎,푎 ′ ) (푋 푖 ,푌 | c). □ 16 Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis A.3. Theoretical Properties In this section, we discuss two key theoretical properties of our contribution metrics that directly underpin our experimental design and diagnostic analysis in the main text: (i) the exact identification of PN and PS under monotonicity versus our conservative lower bounds under LLM non-monotonicity, and (i) the rationale for our asymmetric prerequisite control scheme over A −푖 . A.3.1. Exact Identification under Monotonicity In the probabilities of causation framework [29,38], when the monotonicity assumption holds, the Fréchet lower bounds in Theorems A.3 and A.4 tighten to exact equality. Proposition A.5 (Exact Identification under Monotonicity; [38]). Under the setting of Theorems A.3 and A.4, if 푌is monotonic with respect to푋 푖 across contrast(푎, 푎 ′ )under context c (i.e.,푃 c (푌 푎 =0,푌 푎 ′ =1)=0), then N-Score and S-Score equal PN and PS. For the detailed proof, we refer the reader to Tian and Pearl[38]. In LLM evaluation, however, we do not impose the monotonicity assumption (Property 3.6), as we discussed in Section 3. Meanwhile, as our experiments in Section 5.2 show, injecting an incorrect answer (퐴 푖 =0) can provide a reasoning scaffold that guides downstream deduction better than providing no information (퐴 푖 =∅), causingRC(푋 푖 )to exceedNC(푋 푖 ). A.3.2. Remaining Prerequisite A −푖 Controlling Controlling for Remaining Prerequisites A −푖 . In a multi-parent SCM, the causal effect of a unit task푋 푖 on the final outcome푌is also influenced by the states of the remaining prerequisites A −푖 :=A Pa(푌)\푋 푖 . When A −푖 is left un-intervened in the natural state∅, the actual realizations of these prerequisites remain unobserved and depend on each model’s upstream proficiency, which conflates푋 푖 ’s causal effect with model-specific upstream variations. To isolate푋 푖 ’s contribution under controlled conditions, our general definitions (Definitions A.1 and A.2) allow A −푖 to take any intervention state configuration c∈ A |Pa(푌)|−1 . Because each prerequisite takes one of three states inA=1,0,∅, the full configuration space contains 3 |Pa(푌)|−1 possible combinations. To evaluate푋 푖 across representative prerequisite contexts without enumerating this exponential space, we restrict A −푖 to homogeneous configurations where all remaining prerequisites share the same state푐 ∈ 1,0,∅. This reduction yields three interpretable background environments: the oracle environment (A −푖 =1), the corrupted environment (A −푖 =0), and the natural environment (A −푖 =∅). Asymmetric Prerequisite Control in N-Score and S-Score. To capture complementary diagnostic properties across these homogeneous environments, our main-text definitions of N-Score and S-Score (Definitions 3.7 and 3.8) assign asymmetric background states to A −푖 for any ordered contrast(푎, 푎 ′ ). Specifically, N-Score sets A −푖 =푎 , whereas S-Score sets A −푖 =푎 ′ . This design aligns each metric with its intended diagnostic role in LLM evaluation. For N-Score, conditioning on A −푖 =푎 establishes a baseline environment where all prerequisites begin at state푎, allowing the metric to measure single-task vulnerability when푋 푖 alone degrades to푎 ′ . For S-Score, conditioning on A −푖 =푎 ′ establishes a baseline environment where all prerequisites begin at state푎 ′ , allowing the metric to measure single-task recovery when푋 푖 alone upgrades to푎. Furthermore, combining this asymmetric control with our ordered contrasts(1,0),(1,∅),(∅,0), and(0,∅)systematically covers all three homogeneous environments A −푖 ∈ 1,0,∅and assigns each contrast a distinct operational meaning. For instance, under contrast(1,∅), S (1,∅) evaluates how providing the correct answer for푋 푖 alone improves overall task accuracy in the natural environment (A −푖 =∅), directly identifying high-leverage targets for single-task repair. Under the same contrast, N (1,∅) measures how leaving푋 푖 unassisted reduces accuracy in the oracle environment (A −푖 =1). Similarly, contrasts(1,0),(∅,0), and(0,∅)quantify the impact of prerequisite corruption and structural scaffolding across oracle, natural, and corrupted environments. Algebraic Coupling under Identical Prerequisite Control. Under our general definitions of N-Score and S-Score (Definitions A.1 and A.2), when both metrics are evaluated under the same prerequisite configuration c∈ A |Pa(푌)|−1 , they share a common interventional backgrounddo(A −푖 =c). Within this shared environment, necessity and sufficiency are algebraically coupled through the Probability of Necessity and Sufficiency (PNS) [29, 38]. 17 Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis Proposition A.6 (Algebraic Coupling under Identical Context). For any prerequisite unit task푋 푖 ∈ Pa(푌), any ordered contrast(푎, 푎 ′ )of distinct states inA, and any fixed prerequisite configuration c∈ A |Pa(푌)|−1 , let 푝 푎|c := 푃(푌=1| do(퐴 푖 =푎,A −푖 =c))and푝 푎 ′ |c := 푃(푌=1| do(퐴 푖 =푎 ′ ,A −푖 =c)). Denote the conditional Probability of Necessity and Sufficiency under context c asPNS (푎,푎 ′ ) (푋 푖 ,푌 |c):= 푃(푌 푎 =1,푌 푎 ′ =0| do(A −푖 =c)). Under exogeneity assumption (Assumption 3.3), the exact counterfactual probabilities satisfy: PN (푎,푎 ′ ) (푋 푖 ,푌 | c)· 푝 푎|c = PS (푎,푎 ′ ) (푋 푖 ,푌 | c)·(1− 푝 푎 ′ |c )= PNS (푎,푎 ′ ) (푋 푖 ,푌 | c),(11) and the general N-Score and S-Score satisfy the parallel lower-bound coupling: N (푎,푎 ′ ) (푋 푖 ,푌 | c)· 푝 푎|c = S (푎,푎 ′ ) (푋 푖 ,푌 | c)·(1− 푝 푎 ′ |c )= max 0, 푝 푎|c − 푝 푎 ′ |c ≤ PNS (푎,푎 ′ ) (푋 푖 ,푌 | c).(12) Proof. Let푃 c (·):= 푃(· | do(A −푖 =c))denote the probability measure in the submodel induced bydo(A −푖 =c)[28]. Under exogeneity ssumption, conditioning on퐴 푖 within this submodel is operationally identical to intervening on it, so푃 c (푌 푎 =1)= 푝 푎|c and푃 c (푌 푎 ′ =0)=1− 푝 푎 ′ |c . By consistency and the definition of conditional probability, PN (푎,푎 ′ ) (푋 푖 ,푌 |c) · 푝 푎|c = 푃 c (푌 푎 ′ =0| 퐴 푖 =푎,푌=1) · 푃 c (푌=1| 퐴 푖 =푎)= 푃 c (푌 푎 =1,푌 푎 ′ =0)= PNS (푎,푎 ′ ) (푋 푖 ,푌 |c). An analogous expansion yields PS (푎,푎 ′ ) (푋 푖 ,푌 | c)·(1− 푝 푎 ′ |c )= PNS (푎,푎 ′ ) (푋 푖 ,푌 | c), establishing Equation (11). Next, multiplying Definition A.1 by푝 푎|c and Definition A.2 by 1− 푝 푎 ′ |c directly givesmax0, 푝 푎|c − 푝 푎 ′ |c . Applying the classical Fréchet inequality yieldsPNS (푎,푎 ′ ) (푋 푖 ,푌 |c) ≥ max0, 푝 푎|c +(1− 푝 푎 ′ |c )−1= max0, 푝 푎|c − 푝 푎 ′ |c , which establishes Equation 12.□ Metric Independence under Asymmetric Control. Proposition A.6 shows that necessity and sufficiency are algebraically convertible when evaluated within the same background environment c. However, note that in our main-text formulation (Definitions 3.7 and 3.8), N-Score and S-Score evaluate푋 푖 across two distinct background environments: N-Score conditions on A −푖 =푎 , whereas S-Score conditions on A −푖 =푎 ′ . Whenever푌 depends on multiple prerequisites (|Pa(푌)| >1), the contrast states푎≠ 푎 ′ place the remaining prerequisites into two different interventional configurations. Because the two metrics operate under distinct prerequisite states, their numerators correspond to the Fréchet lower bounds of two different conditionalPNSquantities, PNS (푎,푎 ′ ) (푋 푖 ,푌 | 푎)andPNS (푎,푎 ′ ) (푋 푖 ,푌 | 푎 ′ ), where in general푝 푎|푎 − 푝 푎 ′ |푎 ≠ 푝 푎|푎 ′ − 푝 푎 ′ |푎 ′ . Consequently, our main-text N-Score and S-Score cannot be algebraically derived from one another through marginal outcome probabilities. A special case arises when|Pa(푌)|=1, representing the single-parent setting where A −푖 is empty; in this case, both metrics operate within the same unconditional environment and reduce directly to the classical coupled relationship in Equation 12. For general multi-parent tasks, this LLM-specific decoupling ensures that N-Score and S-Score serve as independent diagnostic dimensions, allowing each paired(푎, 푎 ′ )score to provide distinct and practical diagnostic value for pinpointing failure bottlenecks versus high-leverage repair targets. A.4. Markov Chain Structure In Section 4, we introduced Markov chain unit tasks to represent sub-problems that require iterative solving across sequential steps. In the task-level SCM (Figure 2), a Markov chain unit task is represented as a single composite node that receives inputs from upstream prerequisites and feeds into downstream tasks. To analyze how model capabilities and error propagation evolve over long reasoning horizons, we unfold this composite node into an explicit sequential chain of unit tasks (Section 5.4). Below, we formalize the Markov chain structure within a task decomposition SCM and derive its step-wise causal properties. Definition A.7 (Markov Chain Structure in Task Decomposition SCM). Consider a task decomposition SCM (Definition 3.1). A Markov chain structure is a recursive sub-model region defined over푇sequential reasoning steps (푡=1, . . . ,푇), where each step consists of a recurring cycle of푝constituent unit tasks (푝≥1), represented by unit task variables 푋 (푡) 1 , 푋 (푡) 2 , . . . , 푋 (푡) 푝 . Each unit task variable 푋 (푡) 푖 is given by the structural assignment: 푋 (푡) 푖 := 푓 (푡) 푖 Pa 푋 (푡) 푖 , 푁 (푡) 푖 , for 푖= 1, . . . , 푝, 푡= 1, . . . ,푇,(13) where푓 (푡) 푖 is the structural function determining푋 (푡) 푖 ,푁 (푡) 푖 is the exogenous noise variable, and the parent set Pa 푋 (푡) 푖 is uniquely defined by: Pa 푋 (푡) 푖 = ( 푋 (푡) 푖−1 , if 푖∈ 2, . . . , 푝, 푋 (푡−1) 푝 , if 푖= 1 and 푡 ∈ 2, . . . ,푇, (14) 18 Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis Table 2 | Overview of the 10 overall evaluation tasks in CADET. Task NameModalityCategory # Instances Answer Format Music Note CountingImagePerception114Integer Dice SumImagePerception130Integer Map NavigationImageSpatial107Integer MazeImageSpatial80Multiple-Choice Surround Localization Multi-Image Spatial154Integer Multi-View Sequencing Multi-Image Temporal154Letter List Street-View OrderingMulti-Image Temporal91Letter List Dashcam CountingVideoCognitive79Integer Traffic CausationVideoCognitive107Free-form Text Multistep ReasoningImageCognitive128Free-form Text with the initial prerequisite setPa 푋 (1) 1 given by the upstream prerequisites external to the Markov chain region in the SCM (or∅if the chain originates at a root node). Unfolding this recursive dependency across all푇steps yields the explicit sequential causal chain: 푋 (1) 1 → 푋 (1) 2 →·→ 푋 (1) 푝 → 푋 (2) 1 →·→ 푋 (푇−1) 푝 → 푋 (푇) 1 →·→ 푋 (푇) 푝 .(15) Remark A.8. In the unfolded Markov chain structure (Definition A.7), every non-initial unit task variable푋 푗 has a single causal prerequisite, i.e.,|Pa(푋 푗 )|=1. Consequently, for any such unit task푋 푗 and its unique direct parent 푋 prev ∈ Pa(푋 푗 ), the set of remaining prerequisites is empty: A −prev := A Pa(푋 푗 )\푋 prev =∅.(16) Under this single-parent setting, c∈ A |Pa(푋 푗 )\푋 prev | in Definitions A.1 and A.2 becomes∅, and the N-Score and S-Score simplify to: N (푎,푎 ′ ) (푋 prev , 푋 푗 )= max 0, 푃 푋 푗 = 1 | 푑표(퐴 prev = 푎) − 푃 푋 푗 = 1 | 푑표(퐴 prev = 푎 ′ ) 푃 푋 푗 = 1 | 푑표(퐴 prev = 푎) , S (푎,푎 ′ ) (푋 prev , 푋 푗 )= max 0, 푃 푋 푗 = 1 | 푑표(퐴 prev = 푎) − 푃 푋 푗 = 1 | 푑표(퐴 prev = 푎 ′ ) 1− 푃 푋 푗 = 1 | 푑표(퐴 prev = 푎 ′ ) . (17) As an immediate consequence, Proposition A.6 directly holds without requiring external context conditioning: N (푎,푎 ′ ) (푋 prev , 푋 푗 )· 푝 푎 = S (푎,푎 ′ ) (푋 prev , 푋 푗 )·(1− 푝 푎 ′ )= max0, 푝 푎 − 푝 푎 ′ , (18) where푝 푎 = 푃 푋 푗 =1| 푑표(퐴 prev = 푎) ,푝 푎 ′ = 푃 푋 푗 = 1| 푑표(퐴 prev = 푎 ′ ), andmax0, 푝 푎 − 푝 푎 ′ is the푁푆score reported in Sec. 5.4, denoting lower bound of PNS. In our CADET benchmark, three tasks contain Markov chain structures with cycle lengths푝=1 (Maze, Map Navigation) and푝=2 (Multistep vision reasoning) (Figure 2), corresponding to the step-wise evaluations in Sec. 5.4 (Figure 10). B. Details of Benchmark B.1. Benchmark Composition B.1.1. Task Inventory and Statistics This section reports the benchmark details following our Section 4. Table 2 lists the 10 overall evaluation tasks, and Table 3 reports the overview and structure of their task-level SCMs, reported depth is the length of the longest directed path terminating at 푌. 19 Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis Table 3|Structure of the task-level SCM of each evaluation task. – on multistep reasoning is due to it start from a Markov chain structure. Task#Units #Roots Depth |Pa(푌)| Category #Questions Has Markov Music Note Counting4223P/C1,163No Dice Sum4223P/C1,919No Map Navigation3122P/S/C918Yes Maze2111P/S880Yes Surround Localization6325P/S4,928No Multi-View Sequencing6235P/S/T4,158No Street-View Ordering6235P/S/T3,268No Dashcam Counting5234P/S/T/C6,895No Traffic Causation7146P/T/C2,974No Multistep Reasoning3–22P/C6,828Yes B.1.2. Capability Categories We group unit tasks into four categories according to the kind of information the task requires the model to produce. Each unit task is assigned to the category of the capability it primarily targets. • Perception: reading out the presence, identity, category, or image position of an object from the visual signal, without relating separate elements to one another. •Spatial: relating elements to each other or to the observer, covering distance, size, orientation, viewpoint, and navigation. • Temporal: relating observations across frames or views, covering motion, ordering, and localization in time. •Cognitive: operating on already extracted facts, covering counting, arithmetic, causal attribution, and multi-step or goal-directed reasoning. The overall task of each SCM is labelled with the same four categories (Table 2). Across the 46 unit tasks, the four categories contain 15, 12, 10, and 9 tasks respectively. B.1.3. Per-Task SCM and Unit Task Table 4 specifies the unit tasks and SCM structure of every evaluation task, giving for each node its capability, category, answer format, number of questions, and parent set. Nodes are indexed in a topological order of the task-level SCM. As a result, CADET contains over 33,000 unit task level questions. B.2. Benchmark Design Rationale Task Selection. The 10 evaluation tasks in CADET are selected based on three considerations. First, we focus on problems where current frontier MLLMs remain far from saturated under end-to-end evaluation, so that the diagnosis concerns capability gaps that current models still struggle with. Second, we select compositional tasks that require multiple interdependent capabilities to reach the final answer, so that each task admits a meaningful decomposition into unit tasks. Third, we prioritize real-world visual scenarios across single-image, multi-image, and video modalities, and the resulting 46 unit tasks cover perception, spatial, temporal, and cognitive capabilities. Prerequisite Identification and SCM Structure. Each unit task corresponds to a single capability. A capability is included in the decomposition of an evaluation task when it is a required information for answering the overall task question. For example, to tell the number on the top face of a dice, one needs to know where the dice is and which face is its top face; to tell the direction of a pedestrian relative to the ego vehicle, one needs to locate the pedestrian and determine the viewpoint of the camera that observes it. We specify a prerequisite dependency from푋 푗 to푋 푖 when solving푋 푖 requires the information or the capability represented by푋 푗 . The same task structure is used for all evaluated models. We discuss the completeness of the specified SCMs in Appendix E. 20 Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis Table 4|Unit task specification of the 10 evaluation tasks in CADET.푌denotes the overall task outcome;⟳ marks a Markov chain unit task. Node Unit TaskCat. Answer Format#Q Parents Music Note Counting 푋 1 Object IdentificationP Multiple-choice821 — 푋 2 Object ClassificationP Multiple-choice114 푋 1 푋 3 Completeness VerificationP Multiple-choice114 — 푌Object CountingC Integer114 푋 1 , 푋 2 , 푋 3 Dice Sum 푋 1 Object DetectionP Integer130 — 푋 2 3D Surface RecognitionP Multiple-choice791 — 푋 3 Fine-grained CountingC Integer868 푋 2 푌Arithmetic SummationC Integer130 푋 1 , 푋 2 , 푋 3 Map Navigation 푋 1 Object LocalizationP Multiple-choice443 — 푋 2 Sequential Navigation⟳S Multiple-choice368 푋 1 푌Path-Based CountingC Integer107 푋 1 , 푋 2 Maze 푋 1 Object LocalizationP Multiple-choice80 — 푌Sequential Navigation⟳S Multiple-choice800 푋 1 Surround Localization 푋 1 Object IdentificationP Multiple-choice 1,078 — 푋 2 Front-View IdentificationS Multiple-choice 1,232 — 푋 3 Viewpoint EstimationS Multiple-choice 2,156 — 푋 4 Camera View IdentificationS Multiple-choice154 푋 1 , 푋 2 푋 5 Object LocalizationP Multiple-choice154 푋 1 푌Clock Position EstimationS Integer154 푋 1 , . . . , 푋 5 Multi-View Sequencing 푋 1 Object LocalizationP Multiple-choice770 — 푋 2 Viewpoint EstimationS Multiple-choice 1,386 푋 1 푋 3 Distance ComparisonS Multiple-choice 1,540 푋 1 푋 4 Motion DetectionT Multiple-choice154 — 푋 5 Interaction Status IdentificationT Multiple-choice154 푋 1 , 푋 3 , 푋 4 푌Temporal OrderingT Letter list154 푋 1 , . . . , 푋 5 Street-View Ordering 푋 1 Object LocalizationP Multiple-choice455 — 푋 2 Size ComparisonS Multiple-choice897 푋 1 푋 3 Distance ComparisonS Multiple-choice910 푋 1 , 푋 2 푋 4 Horizontal Position ComparisonS Multiple-choice824 푋 1 푋 5 Motion DetectionT Multiple-choice91 — 푌Temporal OrderingT Letter list91 푋 1 , . . . , 푋 5 Dashcam Counting 푋 1 Object ClassificationP Multiple-choice 2,860 — 푋 2 Spatial RelationS Multiple-choice 2,237 — 푋 3 Object Re-identificationP Multiple-choice 1,378 푋 1 푋 4 Motion ClassificationT Multiple-choice341 푋 1 , 푋 2 , 푋 3 푌Temporal Object CountingC Integer79 푋 1 , . . . , 푋 4 Traffic Causation 푋 1 Object LocalizationP Multiple-choice 2,290 — 푋 2 State Change RecognitionT Multiple-choice200 푋 1 푋 3 Reference-Based Action Recognition T Multiple-choice153 푋 1 푋 4 Temporal LocalizationT Multiple-choice93 푋 2 , 푋 3 푋 5 Temporal SequencingT Multiple-choice105 푋 4 푋 6 False Correlation RejectionC Multiple-choice26 푋 1 , 푋 3 푌Causal ReasoningC Free-form text107 푋 1 , . . . , 푋 6 Multistep Reasoning 푋 1 Position Tracking⟳P Integer list3,350 푋 2 푋 2 Action Prediction⟳C Free-form text 3,350 푋 1 푌Multi-step ReasoningC Free-form text128 푋 1 , 푋 2 21 Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis Table 5 | Source media used in CADET. SourceSource Media UsedReferenceLicense Argoverse 2Urban driving scenes: multi-camera ring imagery and ego-view video Wilson et al. [43] C BY-NC-SA 4.0 ROADWorkRoadwork and construction-zone driving images and videos Ghosh et al. [13] ODC-By v1.0 BabyVisionLanguage-independent core visual reasoning images Chen et al. [4] MIT MEGA-BenchReal-world multimodal task imagesChen et al. [3] Apache-2.0 EMMAScientific and mathematical reasoning figures Hao et al. [18] Apache-2.0 Mutopia Project Engraved sheet-music scoresMutopia Website C BY licensed ∗ Wikimedia Commons Sheet-music and general-purpose imagesWikimedia Website Common License ∗ Procedurally generated Synthetic mazes, navigation maps, dice renderings –Ours (1P) Source URLs. Argoverse 2: https://w.argoverse.org/av2.html; ROADWork: https://github.com/anuragxel/roadwork-dataset; BabyVision: https://huggingface.co/datasets/UnipatAI/BabyVision; MEGA-Bench: https://github.com/TIGER-AI-Lab/MEGA-Bench; EMMA: https://github.com/EMMA-Bench/EMMA; Mutopia: https://w.mutopiaproject.org/; Wikimedia Commons: https://commons.wikimedia.org/. * The Mutopia Project and Wikimedia Commons both host media under a mixture of licenses that are declared individually for each file. For these two sources we applied the site-level C BY license filter when browsing and retained only those images distributed under a C BY license. Unit-Task Question Granularity. As shown in Tables 2 and 4, the number of questions of a unit task is not necessarily equal to the number of overall task instances. When an overall task is built on multiple images, video frames, or target objects, the capability of a unit task does not necessarily apply at the same granularity, and querying all of them within a single bundled question would turn the unit task into a multi-target aggregation problem. In such cases, we formulate separate questions at the granularity at which the capability applies. For example, when a task instance contains five images, object identification is asked separately for each image; when a task instance contains several target objects, object localization is asked separately for each object. B.3. Benchmark Construction B.3.1. Image/Video Source and License As described in Section 4, while all evaluation questions, task decompositions, and unit-task annotations in CADET are newly created to mitigate data contamination, the underlying source images and videos are curated from publicly available datasets and open repositories with permissive licenses. Table 5 summarizes all source media (image and video) used in CADET, along with their corresponding descriptions, references, and license terms. B.3.2. Data Construction and Quality Assurance Two-Stage Construction and Closed-Form Annotation. We construct CADET in two stages following pre- defined task structures. In the first stage, for each of the 10 overall tasks, we define the task scope, a question-answer template, and detailed task instructions on the selected source images and videos. Raters follow these instructions and templates to write the overall task question and its ground-truth answer. In the second stage, to decompose each overall task into unit tasks without introducing noise from free-form question writing, we design the decomposition step as closed-form annotation and labeling tasks. Depending on the unit task, raters draw bounding boxes or paths on the images and provide labels that follow strict format specifications. Once these annotations are collected, we extract the labeled information and convert it into unit-task questions, correct answers (퐴 푖 =1), and rendered visual annotations through structured templates. For the incorrect answers (퐴 푖 =0), structured unit tasks use random sampling over plausible candidates; specifically, for numerical tasks, we sample within a bounded local neighborhood around the ground truth (e.g., within±2 of a groundtruth value of 5) rather than drawing distant values (e.g., 20) that models would 22 Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis trivially rule out. For the remaining three free-form text tasks where random sampling is not applicable, raters write the incorrect answers following instructions designed to minimize subjective bias. Two-Rater Iterative Verification. All annotations are conducted by a pool of over 130 experienced multimodal raters. Across both stages, every instance is processed by two raters in sequence: the first rater completes the annotation, and a second quality-check rater independently inspects the result. If the quality-check rater identifies an inaccurate answer, an imprecise bounding box or path, or a label that violates the required format specification, the instance is returned to the first rater for revision. This review cycle repeats until the quality-check rater accepts the annotation. Instance Revision and Corner-Case Filtering. At every annotation step, raters can flag issues with a candidate instance. When a question requires minor rephrasing or an answer can be refined against clear visual content, raters directly update the item within the review loop; when an instance cannot be resolved or cleanly annotated at the unit-task level, raters are instructed to drop it. Beyond ambiguities in the overall task, drops in the second stage occur when an intermediate unit task cannot be reliably annotated across frames or views. Typical cases include a spatial reference (such as a roadside curb) shifting to the image boundary with insufficient visibility, a target vehicle tracked across views becoming too small or partially occluded in a later frame, or a second vehicle of the same color partially entering the scene and preventing unique target identification. Removing such instances ensures that every unit task in the SCM admits a well-defined intervention state. Across 1,467 candidate instances dispatched for annotation, 323 (22.0%) were dropped under these criteria, yielding the final 1,144 instances and over 33,000 unit-task questions in CADET. C. Experiments Setup Evaluated Models. We evaluate 12 MLLMs across six model families: Gemini 3.1 Pro [10], Gemini 3.5 Flash [15], Gemini 3 Flash [11], Gemini 3.1 Flash-Lite [12], GPT-5.4 [27], GPT-5.4 Nano [27], Grok 4.3 [45], Gemma 4 31B [6], Gemma 4 26B [6], Qwen3.5-397B-A17B [32], Qwen3.6 27B [33], and Kimi K2.5 [37]. The Gemini, GPT, Grok and Gemma models are queried through the official APIs, while the remaining open-weight models are queried via OpenRouter. All models are run with their default sampling settings, maximum supported output tokens, and thinking mode set to high (or left at the default active setting when not configurable). LLM-as-a-judge details. Among the 46 unit tasks, 43 use structured output formats (multiple-choice, integer, or coordinate) and are evaluated via deterministic rule-based parsing and exact match, while the remaining 3 free-form text unit tasks are evaluated using Gemini 3.6 Flash [16] by comparing model predictions against ground-truth reference answers. Although these 3 unit tasks do not constrain outputs to option letters or numbers, each still pairs with a concise, specific reference answer (e.g., an atomic action such as “turn left”), so the judge only checks whether the predicted phrase conveys the same meaning as the reference. For the 43 structured unit tasks, Gemini 3.6 Flash is invoked strictly as a fallback when a response deviates from the required surface format (e.g., outputting “So the answer is **C**.” instead of “C”). Across all evaluation runs on structured unit tasks, deterministic exact match directly resolves 98.03% of predictions, with only 1.97% triggering the LLM-as-a-judge fallback. Prompts. Figures 11 and 12 present the prompt templates used to assemble intervened prerequisite context and to run the LLM judge, respectively. For active interventions (퐴 푖 ∈ 1,0), the prerequisitesPa(푋 푖 )are formatted as a structured JSON question–answer block and prepended to the target task query using the same template for both퐴 푖 =1 and퐴 푖 =0, differing only in the filled prerequisite answer values (along with the corresponding visual annotations rendered on the image when퐴 푖 =1 represents spatial ground truth). Under the natural state (퐴 푖 =∅), the prerequisite context block is omitted while the target question and output-format instructions remain the same. D. More Experiment Results D.1. More Capability Diagnosis Results (NC, IC, RC) This section provides additional visualizations and the complete numerical results supporting the capability diagnostics in Section 5.2. First, whereas Section 5.2 focuses on the 30 non-root unit tasks (Pa(푋 푖 )≠∅) that admit upstream interventions, Figure 13,14 and Table 7,6 incorporate the 16 foundational root unit tasks 23 Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis Figure 11|Prompt template for our evaluation under active causal interventions (퐴 푖 ∈ 1,0). Under the natural state (퐴 푖 =∅), only the target question and last paragraph of output format instructions are prompted. Table 6|NC on root unit tasks and NC, IC on non-root unit tasks, per category, with the number of unit tasks in each group. Entries are the unweighted mean over unit tasks, averaged over the 12 models,±the standard deviation across models. #unit tasksRootNon-root Categoryroot non-rootNCNCICIC− NC Perception 1140.775± 0.077 0.590± 0.138 0.796± 0.112 +0.206 Spatial390.627± 0.100 0.556± 0.077 0.671± 0.059 +0.115 Temporal280.714± 0.119 0.460± 0.072 0.614± 0.087 +0.154 Cognitive09–0.406± 0.092 0.721± 0.089 +0.316 (Pa(푋 푖 )= ∅, for which only natural capabilityNCis defined), comparing rootNCagainst non-rootNCand ICacross individual models (Figure 14, Table 7) and capability categories (Figure. 13, Table 6). Second, Figure 15,16 and Table 8,9 expand the compactRC− NCsummary in Figure 4 of the main text by plotting absoluteNCandRClevels alongside their signed differences at the category level (Figure 15, Table 8) and across all 12 models within each category (Figure 16, Table 9), reporting the full per-group means and standard deviations. Comparing Non-Root Capabilities (NC,IC) with Foundational Root Capabilities (NC). Because root unit tasks have no modeled upstream prerequisites in the task SCM (Pa(푋 푖 )= ∅), their natural capability (NC) assesses foundational proficiency independently of upstream prerequisite states. Under unassisted evaluation (do(A Pa(푋 푖 ) =∅)), models achieve higher accuracy on the 16 root unit tasks than on the 30 downstream non-root unit tasks: as reported in Table 7 and Figure 14, meanNCacross the 12 models is 0.740 on root nodes versus 0.490 on non-root nodes (a difference of+0.250, ranging from+0.169 to+0.301 across individual models). A consistentNCdifference between root and non-root nodes appears within each category that contains root tasks (Table 6, Figure 13): Perception (0.775 vs. 0.590), Spatial (0.627 vs. 0.556), and Temporal (0.714 vs. 0.460), while Cognitive unit tasks occur exclusively as non-root nodes (Pa(푋 푖 )≠ ∅) and score lowest under unassisted evaluation (NC=0.406). When correct prerequisite answers are supplied (do(A Pa(푋 푖 ) = 1)), mean non-rootICacross models rises to 0.688 against the 0.740 rootNCbaseline (narrowing the overall gap from 24 Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis Figure 12 | LLM-as-a-judge prompt template. Perc.Spat.Temp. Overall 0.0 0.2 0.4 0.6 0.8 1.0 Score root NCnon-root NC non-root IC−NC Figure 13|Comparing NC on root unit tasks against NC and IC on non-root unit tasks, per category, averaged over the 12 models. Cognitive is not shown as it has no root unit task. The non-root bar stacks IC−NC on top of NC, so its top is IC. 0.250 to 0.052, with differences as small as−0.010 for GPT-5.4 and 0.002 for GPT-5.4n). Across categories, this comparison reinforces the capability profile in Section 5.2: Perception maintains high accuracy on both root tasks (NC=0.775) and prerequisite-supplied non-root tasks (IC=0.796), and Cognitive non-root tasks rise fromNC=0.406 toIC=0.721, approaching the average root-node baseline (0.740). By contrast, Spatial exhibits the lowest root-node accuracy (NC=0.627) and reaches a similarly bounded ceiling on non-root tasks when prerequisites are supplied (IC=0.671), while Temporal non-root tasks reachIC=0.614 (vs. 0.714 on root tasks), consistent with the persistent residual gaps (1− IC) in spatial and temporal capabilities identified in the main text. Expanded Breakdown of Resilience Capability (RCvs.NC). Figure 15 and Table 8 complement Figure 4 by displayingNCandRCon the same absolute capability scale across categories, while Figure 16 and Table 9 provide the disaggregated per-model profiles parallel to Figure 3(b–e). On average across the 12 models (Table 8), injecting incorrect prerequisite answers (do(A Pa(푋 푖 ) =0)) yields lower accuracy than unassisted evaluation in Perception (0.590→0.446,RC− NC=−0.144), Spatial (0.556→0.439,−0.116), and Temporal (0.460→0.414,−0.046), corresponding to an overall non-root difference of−0.054 (0.490→0.435), whereas Cognitive tasks exhibit a modest positive shift (0.406→0.445,+0.040). The per-model breakdown in Figure 16 and Table 9 shows that these category-level trends hold consistently across model families and scales: all 12 models exhibitRC < NCin Perception and Spatial, 11 of 12 decline in Temporal (with GPT-5.4n showing a +0.030 shift), and 10 of 12 show positive RC− NC shifts in Cognitive (all except GPT-5.4 at−0.018 and Kimi K2.5 at−0.066), providing the full per-model evidence for the non-monotonicity discussed in Property 3.6 and Section 5.2. 25 Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n 0.0 0.2 0.4 0.6 0.8 1.0 Score root NCnon-root NC non-root IC−NC Figure 14|Comparing NC on root unit tasks against NC and IC on non-root unit tasks for each of the 12 models. There are 16 root unit tasks and 30 non-root unit tasks. The non-root bar stacks IC−NC on top of NC, so its top is IC. Models are ordered by overall non-root NC. Perc.Spat.Temp.Cog. 0.0 0.2 0.4 0.6 0.8 1.0 Score NC NC−RCRC−NC Figure 15|Comparing NC and RC across the four categories on non-root unit tasks, averaged over the 12 models. Each bar shows NC with the signed NC/RC difference stacked out of it, so the boundary of the shaded band is RC. D.2. More Causal Contribution Diagnosis Results (푁, 푆-Score) In this section, we provide additional experimental results complementing the causal contribution analysis in Section 5.3. Figure 17 reports prerequisite푁- and푆-Scores under the remaining three state contrasts(1,0), (0,∅), and(∅,0)(complementing(1,∅)in Figure 5). Figure 18 disaggregates these three contrasts by task pattern and model group (extending Figure 6). Figure 19 shows the inter-model rank agreement across all eight contribution scores, and Figure 20 provides the per-prerequisite breakdown across the four state contrasts. To prevent small denominators from producing numerically unstable ratios in Eqs. (5)–(6), we set a score to 0 whenever its baseline denominator falls below 0.10. Crucially, this threshold has zero effect on six of the eight metrics and affects only 3.0% (94/3,168) of all scores, concentrated in푁(0,∅)(with a small remainder in푁(∅,0)). This is expected by design:푁(0,∅)measures the success drop from setting a prerequisite to∅ in the all-incorrect (0) environment, where baseline successRC(푌)is naturally near zero whenever incorrect prerequisites prevent task completion. Corrupting a prerequisite reduces success more than leaving it unassisted.푁(1,0)and푁(1,∅)share the same oracle baseline where all prerequisites are correctly provided, differing only in whether푋 푖 receives an incorrect answer or is left unassisted. Across prerequisites,푁(1,0)consistently exceeds푁(1,∅)(Figure 17a vs. Figure 5), shifting more prerequisites into the high-푁/ low-푆quadrant and fewer into the low-푁/ high-푆 quadrant. For example, on the perception prerequisites of Music Note,푁(1,∅)is low (0.04–0.16) and푆(1,∅) is high (0.51–0.59): omitting a single prerequisite in the oracle setting incurs minimal loss, while supplying it alone in the natural state recovers over half of the failures. Under(1,0), however,푁(1,0)rises to 0.27–0.73 and푆(1,0)falls to at most 0.02: a single incorrect prerequisite removes a substantial fraction of oracle success, while correcting one prerequisite when others remain incorrect recovers almost no failures. Similarly, under (∅,0)(Figure 17c), prerequisites shift along푁(∅,0)into the high-푁/ low-푆region, whereas under(0,∅) 26 Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis 0.2 0.4 0.6 0.8 Score (a) Perception(b) Spatial GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n 0.2 0.4 0.6 0.8 Score (c) Temporal GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n (d) Cognitive NC / Mean NCRC / Mean RC NC−RCRC−NC Figure 16|Comparing NC and RC for each of the 12 models within each category on non-root unit tasks. Circles mark NC and triangles mark RC; the shaded band spans the two and is colored by the sign of RC−NC. The horizontal lines are the means over the 12 models. Table 7|NC on root unit tasks and NC, IC on non-root unit tasks for each of the 12 models. Entries are the unweighted mean over unit tasks±the standard deviation across unit tasks (16 root, 30 non-root). The last two columns are differences of the corresponding means. ModelRoot NCNon-root NCNon-root ICNC(root)−NC(non-root) NC(root)−IC(non-root) GPT-5.40.781± 0.108 0.612± 0.180 0.791± 0.167+0.169−0.010 Gem 3.5F0.839± 0.099 0.595± 0.217 0.783± 0.172+0.244+0.056 Gem 3.1P0.842± 0.090 0.577± 0.202 0.781± 0.173+0.265+0.062 Gem 3F0.811± 0.119 0.528± 0.206 0.757± 0.180+0.282+0.053 Qwen27B0.780± 0.132 0.515± 0.247 0.711± 0.219+0.264+0.069 Qwen397B0.749± 0.144 0.487± 0.240 0.647± 0.226+0.262+0.101 Kimi K2.50.708± 0.200 0.483± 0.199 0.652± 0.213+0.224+0.056 Gemma-31B 0.752± 0.161 0.475± 0.235 0.689± 0.202+0.278+0.063 Gem 3.1FL0.692± 0.186 0.430± 0.217 0.654± 0.209+0.262+0.038 Gemma-26B 0.722± 0.196 0.421± 0.248 0.627± 0.218+0.301+0.094 Grok 4.30.644± 0.175 0.405± 0.210 0.601± 0.205+0.239+0.043 GPT-5.4n0.559± 0.239 0.347± 0.190 0.557± 0.236+0.212+0.002 (Figure 17b), most prerequisites remain near zero with positive values concentrated in specific prerequisites examined in Figure 20. Separation between SOTA and Others under prerequisite corruption. When disaggregated by task pattern and model group (Figure 18), SOTA and Others separate clearly in Pattern 2 across all three contrasts involving incorrect prerequisites:푆(1,0),푆(∅,0), and푁(1,0)(Figure 18c,d,k,l). Starting from the all-incorrect environment, SOTA achieves consistently higher single-prerequisite recovery than Others both when correcting a single prerequisite (푆(1,0): 0.143–0.263 for SOTA vs. 0.029–0.105 for Others) and when removing a single incorrect answer (푆(∅,0): 0.044–0.152 vs. 0.015–0.028). Symmetrically, starting from the oracle environment, corrupting a single prerequisite causes smaller drops for SOTA (푁(1,0): 0.097–0.106) than for Others (0.108–0.295), mirroring the shift toward higher 푁 and lower 푆 for Others under (1,∅) in Figure 6. Inter-model agreement on high-푆(1,∅)prerequisites. Among the eight contribution metrics,푆(1,∅)exhibits the strongest inter-model agreement, with a mean pairwise Spearman correlation of 0.72, compared with 0.41 for푁(1,∅)(Figure 19). Thus, the high-leverage prerequisites that recover natural failures when supplied alone are largely shared across models, whereas sensitivity to single-prerequisite omission in the oracle setting varies more across models. Agreement on푆(1,∅)is especially strong among the four SOTA models (mean휌=0.85 vs. 0.70 among Others), holding both within the Gemini series (0.87) and across families between GPT-5.4 and 27 Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis Table 8|NC, IC and RC per category on the 30 non-root unit tasks. Entries are the unweighted mean over unit tasks, averaged over the 12 models,± the standard deviation across models. CategoryNCICRCIC− NC RC− NC Perception 0.590± 0.138 0.796± 0.112 0.446± 0.131 +0.206 −0.144 Spatial0.556± 0.077 0.671± 0.059 0.439± 0.102 +0.115 −0.116 Temporal 0.460± 0.072 0.614± 0.087 0.414± 0.063 +0.154 −0.046 Cognitive 0.406± 0.092 0.721± 0.089 0.445± 0.060 +0.316 +0.040 Table 9|NC, IC and RC per model and category on the 30 non-root unit tasks, with Overall aggregating all 30. Entries are the unweighted mean over unit tasks± the standard deviation across unit tasks. PerceptionSpatialTemporalCognitiveOverall ModelNCICRCNCICRCNCICRCNCICRCNCICRC GPT-5.4 0.636 ±0.194 0.913 ±0.087 0.507 ±0.288 0.665 ±0.202 0.747 ±0.214 0.549 ±0.213 0.560 ±0.148 0.730 ±0.181 0.537 ±0.219 0.593 ±0.192 0.835 ±0.094 0.575 ±0.274 0.612 ±0.180 0.791 ±0.167 0.548 ±0.232 Gem 3.5F 0.722 ±0.192 0.889 ±0.042 0.599 ±0.194 0.672 ±0.231 0.754 ±0.213 0.572 ±0.252 0.564 ±0.154 0.699 ±0.202 0.481 ±0.187 0.490 ±0.232 0.839 ±0.091 0.507 ±0.219 0.595 ±0.217 0.783 ±0.172 0.532 ±0.212 Gem 3.1P 0.761 ±0.155 0.907 ±0.027 0.678 ±0.158 0.650 ±0.217 0.740 ±0.230 0.584 ±0.207 0.515 ±0.164 0.698 ±0.175 0.450 ±0.222 0.478 ±0.174 0.839 ±0.088 0.513 ±0.194 0.577 ±0.202 0.781 ±0.173 0.540 ±0.206 Gem 3F 0.664 ±0.250 0.879 ±0.072 0.439 ±0.304 0.570 ±0.268 0.710 ±0.253 0.431 ±0.240 0.524 ±0.122 0.707 ±0.163 0.422 ±0.205 0.430 ±0.154 0.796 ±0.121 0.450 ±0.223 0.528 ±0.206 0.757 ±0.180 0.435 ±0.222 Qwen27B 0.682 ±0.167 0.873 ±0.119 0.418 ±0.237 0.571 ±0.308 0.663 ±0.252 0.439 ±0.249 0.515 ±0.248 0.672 ±0.210 0.466 ±0.276 0.386 ±0.159 0.722 ±0.220 0.421 ±0.213 0.515 ±0.247 0.711 ±0.219 0.438 ±0.233 Qwen397B 0.655 ±0.126 0.808 ±0.167 0.511 ±0.184 0.579 ±0.287 0.677 ±0.269 0.405 ±0.217 0.407 ±0.164 0.480 ±0.150 0.349 ±0.196 0.390 ±0.238 0.695 ±0.194 0.423 ±0.246 0.487 ±0.240 0.647 ±0.226 0.410 ±0.212 Kimi K2.5 0.621 ±0.112 0.698 ±0.127 0.342 ±0.161 0.511 ±0.245 0.659 ±0.300 0.359 ±0.231 0.409 ±0.156 0.586 ±0.175 0.331 ±0.227 0.461 ±0.204 0.682 ±0.187 0.395 ±0.138 0.483 ±0.199 0.652 ±0.213 0.360 ±0.189 Gemma-31B 0.609 ±0.291 0.825 ±0.035 0.485 ±0.272 0.539 ±0.275 0.679 ±0.279 0.470 ±0.247 0.420 ±0.182 0.625 ±0.163 0.414 ±0.236 0.399 ±0.200 0.696 ±0.182 0.438 ±0.217 0.475 ±0.235 0.689 ±0.202 0.448 ±0.227 Gem 3.1FL 0.514 ±0.294 0.756 ±0.117 0.427 ±0.264 0.482 ±0.221 0.627 ±0.260 0.406 ±0.207 0.419 ±0.177 0.557 ±0.230 0.416 ±0.232 0.349 ±0.220 0.720 ±0.141 0.441 ±0.212 0.430 ±0.217 0.654 ±0.209 0.422 ±0.211 Gemma-26B 0.540 ±0.351 0.791 ±0.115 0.477 ±0.301 0.520 ±0.287 0.632 ±0.285 0.458 ±0.260 0.370 ±0.171 0.532 ±0.212 0.346 ±0.196 0.314 ±0.186 0.635 ±0.154 0.373 ±0.216 0.421 ±0.248 0.627 ±0.218 0.405 ±0.230 Grok 4.3 0.399 ±0.237 0.659 ±0.096 0.214 ±0.223 0.458 ±0.249 0.594 ±0.273 0.211 ±0.161 0.447 ±0.215 0.585 ±0.220 0.355 ±0.218 0.317 ±0.150 0.597 ±0.174 0.426 ±0.215 0.405 ±0.210 0.601 ±0.205 0.314 ±0.213 GPT-5.4n 0.278 ±0.250 0.551 ±0.331 0.260 ±0.234 0.450 ±0.204 0.571 ±0.232 0.389 ±0.202 0.365 ±0.176 0.498 ±0.233 0.395 ±0.222 0.258 ±0.120 0.598 ±0.233 0.378 ±0.166 0.347 ±0.190 0.557 ±0.236 0.370 ±0.196 Gemini (0.84), and within individual tasks the top-푆(1,∅)prerequisite is identical or shared between the same top pair across models on the majority of tasks. Cross-contrast consistency and per-prerequisite breakdown. Across prerequisites, rankings correlate positively between푁(1,∅)and푁(1,0)(Spearman휌=0.64) and between푆(1,∅)and푆(1,0)(휌=0.53; Figure 20), indicating that prerequisites whose omission most reduces oracle performance also tend to be the most destructive when corrupted, and those that best recover natural failures also drive recovery in the all-incorrect setting. For transitions between∅and 0,(∅,0)and(0,∅)measure opposite directions of the same state pair; under zero-clipping, at most one direction is active for a given model and prerequisite, separating performance degradation from incorrect prerequisites (푁(∅,0),푆(∅,0)) from positive recovery when an incorrect prerequisite is provided over no prerequisite (푆(0,∅), 푁(0,∅)) under Property 3.6. Consistent with Section 5.2 and Figure 9, positive푆(0,∅)values appear primarily on prerequisites feeding into downstream cognitive or spatial-relational integration—such as Temporal Sequencing (0.34), Temporal Localization (0.11), and Object Localization (0.07) in Traffic Causation, and Viewpoint Estimation (0.23) in Surround Loc.—while remaining near zero on foundational perception and spatial prerequisites. 28 Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis 00.050.150.30.60.9 N(1, 0) 0 0.05 0.15 0.3 0.6 0.9 S (1, 0) (a) N/S(1, 0) high N / high Slow N / high S high N / low Slow N / low S 00.050.150.30.60.9 N(0, ∅) 0 0.05 0.15 0.3 0.6 0.9 S (0, ∅) (b) N/S(0, ∅) high N / high Slow N / high S high N / low Slow N / low S 00.050.150.30.60.9 N(∅, 0) 0 0.05 0.15 0.3 0.6 0.9 S (∅, 0) (c) N/S(∅, 0) high N / high Slow N / high S high N / low Slow N / low S PerceptionSpatialTemporalCognitiveTop-1 parentPerceptionSpatialTemporalCognitiveTop-1 parent Map Nav.Dice SumMusic NoteDashcam Count.Multi-View Seq.Street-View Ord.Surround Loc.Traffic Causation Figure 17|Prerequisite푁/푆-Scores under state contrasts (a)(1,0), (b)(0,∅), and (c)(∅,0), averaged across models. Dashed circles mark the highest-푆prerequisite within each task. Axes and quadrant thresholds follow Figure 5. D.3. Detail Explanation of Markov Chain Unit Task Results Section 5.4 analyzes the three Markov chain unit tasks in CADET (Figure 10): Maze and Multistep Reasoning, whose primary task structure is a Markov chain, together with the route-planning unit task embedded within Map Navigation. While all three share the same Markov sequential dependency structure (Definition A.7), they differ in their concrete step-wise logic, and because this distinction directly manifests in their empirical behavior, we group them into the two regimes discussed in Section 5.4: goal-directed and process-cumulative. In Maze, the trajectory from the start to the exit is partitioned into푇=10 steps; because the overall query (identifying which candidate is the exit) cannot be further decomposed into distinct sub-questions, at each step푡the model is positioned at the푡-th waypoint to answer this same exit-identification question. Similarly, in the Markov chain node of Map Navigation, the model stands at each intermediate step along the route and looks toward a given destination to decide the next move. Both tasks are therefore goal-directed: conditioned on the current step푡, solving the task depends only on the remaining path toward the goal rather than the history already traversed. By contrast, Multistep Reasoning is process-cumulative: starting from an initial state without a pre-specified destination, the model follows movement rules where each step alternates between identifying the current Position and determining the current Action. Because the terminal state is unknown in advance, accurately resolving either node at step푡requires tracking the cumulative execution process from the starting point up to step 푡. To operationalize the ternary intervention at step푡along an unfolded chain, we provide the full preceding trajectory context from step 1 to푡−1 in the prompt. Under퐴=1, we inject the ground-truth questions, answers, and corresponding visual annotations for all preceding steps. A special case arises in Maze: because every step asks the same exit-identification question and shares the identical ground-truth answer, injecting the textual answer of any preceding step would directly reveal the answer at step푡. For퐴=1 in Maze, we therefore inject the preceding step questions without their textual answers, accompanied by the visual annotation on the image marking the current waypoint and the human-annotated path from the start up to step푡−1. Under퐴=0, each preceding step from 1 to푡−1 is independently assigned a randomly sampled incorrect answer from the valid answer space via text only (Section 5.1), while 퐴=∅ leaves the preceding trajectory unassisted. D.4. Additional Experiments and Findings D.4.1. Ablation on Intervention Modality In the main experiments, the correct intervention (퐴 푖 =1) injects both a textual answer and, where applicable, visual annotations (e.g., bounding boxes, trajectories) onto the image. To examine the role of visual annotations in this intervention, we also evaluate a text-answer-only condition퐴 푖 =1 푡 , which provides the same textual answer as퐴 푖 =1 but keeps the original image unmodified. We run this condition on two models, Gemini 3.5 Flash from the top-4 (SOTA) group and Gemma-4-31B from the remaining (Others) group, across all 30 non-root unit tasks. Figure 21(a) shows that 1 푡 closely approaches IC across all four capability categories. In temporal, the two 29 Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis 00.050.150.30.60.9 N(1, 0) 0 0.05 0.15 0.3 0.6 0.9 S (1, 0) (a) Pattern 1 — SOTA 00.050.150.30.60.9 N(1, 0) (b) Pattern 1 — Others 00.050.150.30.60.9 N(1, 0) (c) Pattern 2 — SOTA 00.050.150.30.60.9 N(1, 0) (d) Pattern 2 — Others 00.050.150.30.60.9 N(0, ∅) 0 0.05 0.15 0.3 0.6 0.9 S (0, ∅) (e) Pattern 1 — SOTA 00.050.150.30.60.9 N(0, ∅) (f) Pattern 1 — Others 00.050.150.30.60.9 N(0, ∅) (g) Pattern 2 — SOTA 00.050.150.30.60.9 N(0, ∅) (h) Pattern 2 — Others 00.050.150.30.60.9 N(∅, 0) 0 0.05 0.15 0.3 0.6 0.9 S (∅, 0) (i) Pattern 1 — SOTA 00.050.150.30.60.9 N(∅, 0) (j) Pattern 1 — Others 00.050.150.30.60.9 N(∅, 0) (k) Pattern 2 — SOTA 00.050.150.30.60.9 N(∅, 0) (l) Pattern 2 — Others Pattern 1:Map Nav.Dice SumMusic NoteDashcam Count. Pattern 2:Multi-View Seq.Street-View Ord.Surround Loc.Traffic Causation Indiv. model Figure 18|Prerequisite푁/푆-Scores under(1,0)(a–d),(0,∅)(e–h), and(∅,0)(i–l) by task pattern and model group. Large markers denote group means; grey markers indicate individual models. conditions are virtually identical (both 0.66); in perception and spatial, they differ by only 0.02 (0.84 vs. 0.86; 0.70 vs. 0.72). The largest gap appears in cognitive (0.71 vs. 0.77), where 1 푡 nonetheless remains far above NC (0.44) and RC (0.47). Overall, 1 푡 reaches 0.71 against an IC of 0.74. The per-model results (Figure 21(b–e)) are consistent: for both models, 1 푡 tracks IC in every category, with a gap of at most 0.06 in any model–category pair. These results indicate that the textual answer accounts for the large majority of the capability gain under correct prerequisites, while the remaining difference between 1 푡 and IC reflects the additional contribution of visual annotations. The same pattern holds for the causal contribution metrics. Figure 22 compares푆(1 푡 ,∅) with 1 푡 in place of state 1, against푆(1,∅)for each prerequisite. Spatial, temporal, and cognitive prerequisites lie along the diagonal, indicating that the magnitude and ranking of S-Scores are largely preserved without visual annotations. The systematic departures below the diagonal correspond to perception prerequisites, whose푆(1 푡 ,∅) falls well below푆(1,∅). The corresponding N-Score comparison shows a similar agreement (Pearson푟=0.87 between 푁(1 푡 ,∅) and 푁(1,∅) across all prerequisites). D.4.2. Ablation on Question-Only Injection In Section 5.2, incorrect prerequisites improve cognitive tasks for most models (RC > NC), and in Section 5.3, 푆(0,∅)is nonzero for some prerequisite categories, indicating that even incorrect answers occasionally help. Since the incorrect intervention (퐴 푖 =0) injects both a sub-question and an incorrect answer, a natural follow-up is to ask how much of the prerequisite effect comes from the sub-question itself. We therefore evaluate a question-only condition퐴 푖 = 푞, which injects each prerequisite’s sub-question without any answer, on the same two models. As shown in Figure 21(a),푞yields a modest improvement over NC (overall 0.58 vs. 0.53) but remains far below 1 푡 (0.71) and IC (0.74). Across categories,푞exceeds NC by at most 0.06, whereas 1 푡 exceeds NC by 0.09 (spatial) to 0.26 (cognitive). The causal contribution metrics show the same trend:푆(푞,∅)averages 0.05 across prerequisites, compared with 0.26 for푆(1,∅)on the same two models, and푆(푞,∅)does not exceed 0.09 in any prerequisite category. Thus, providing the sub-question without answer does improve performance, 30 Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n (a) N(1, ∅) (b) N(1, 0) (c) N(0, ∅) (d) N(∅, 0) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n (e) S(1, ∅) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n (f) S(1, 0) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n (g) S(0, ∅) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n (h) S(∅, 0) −0.2 0.0 0.2 0.4 0.6 0.8 1.0 Spearman ρ Figure 19|Pairwise Spearman rank correlation between models on푁-Scores (a–d) and푆-Scores (e–h) across prerequisites under the four state contrasts. Lines separate SOTA from Others. meanwhile the improvement is limited and remains far smaller than that from providing its answer. E. Discussion On the Scope and Completeness of Task Decompositions. The SCM of each task declares, for every target task, which unit tasks are placed under controlled inter- vention. It is not intended as an exhaustive characterization of the task’s causal structure or of the model’s internal computation, and we do not claim that it is complete: the set of factors that may influence a model’s answer on a real-world task is open-ended. A unit task is declared as a prerequisite of a target when solving the target requires the information it represents (Appendix. B.2), and the same declared structure is used for all evaluated models. Any factor outside the declared parent sets belongs to the unmodeled factors captured by 푁 푖 (Definition 3.1); it is not separately intervened on and remains in its natural regime. Completeness is not required for the validity of the metrics. Every quantity entering the capability and contribution metrics is an interventional probability, estimated by directly running the model under the corresponding regime, including the natural regime∅, rather than inferred from co-occurrences between unit-task outcomes. The lower-bound guarantees (Theorems A.3 and A.4) hold for any configuration c of the remaining declared prerequisites and rest on prerequisite regimes being assigned through injection (Assumption 3.3), not on the declared parent sets capturing every factor that influences the outcome. N- and S-Scores therefore remain valid lower bounds on PN and PS for the context actually realized, in which the target’s declared prerequisites are set as specified and all other factors are left in their natural regime. What the decomposition determines is the reference set against which each metric is read (Section 3.3): IC measures performance on푋 푖 when its declared prerequisites are supplied correctly,IC− NCis the net gain from supplying them (the cascading component in Section 5.2), and 1− ICis the residual error that persists under correct prerequisites. The category-level findings in Section 5.2 are statements of this form, namely which share of the natural error is resolved by supplying the declared prerequisites and which share persists, obtained under the same inclusion criterion for every category. Finally, declared dependencies are task-defined, whereas the extent to which a model relies on each of them is measured rather than assumed. Averaged within each category, supplying the declared prerequisites improves performance for every model (IC > NCin all 48 model-category cells; Tab 9). Individually, a declared prerequisite may have small N- and S-Scores under(1,∅)(Section 5.3), meaning that neither withholding it in the oracle environment nor supplying it alone in the natural environment substantially changes the outcome. This can occur for a genuine prerequisite: both scores can be small when the model already resolves it correctly within the full task, N when its omission is compensated by the other supplied prerequisites, and S when the 31 Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis N S Map Nav. · X1 Object Localization Map Nav. · X2 Sequential Navigation Dice Sum · X1 Object Detection Dice Sum · X2 3D Surface Recognition Dice Sum · X3 Fine-grained Counting Music Note · X1 Object Identification Music Note · X2 Object Classification Music Note · X3 Completeness Verification Dashcam Count. · X1 Object Classification Dashcam Count. · X2 Spatial Relation Dashcam Count. · X3 Object Re-identification Dashcam Count. · X4 Motion Classification Multi-View Seq. · X1 Object Localization Multi-View Seq. · X2 Viewpoint Estimation Multi-View Seq. · X3 Distance Comparison Multi-View Seq. · X4 Motion Detection Multi-View Seq. · X5 Interaction Status Identification Street-View Ord. · X1 Object Localization Street-View Ord. · X2 Size Comparison Street-View Ord. · X3 Distance Comparison Street-View Ord. · X4 Horizontal Position Comparison Street-View Ord. · X5 Motion Detection Surround Loc. · X1 Object Identification Surround Loc. · X2 Front-View Identification Surround Loc. · X3 Viewpoint Estimation Surround Loc. · X4 Camera View Identification Surround Loc. · X5 Object Localization Traffic Causation · X1 Object Localization Traffic Causation · X2 State Change Recognition Traffic Causation · X3 Reference-Based Action Recognition Traffic Causation · X4 Temporal Localization Traffic Causation · X5 Temporal Sequencing Traffic Causation · X6 False Correlation Rejection 0.400.47 0.060.12 0.020.05 0.010.01 0.740.91 0.040.54 0.040.51 0.160.59 0.130.20 0.050.05 0.040.11 0.240.41 0.070.02 0.470.20 0.200.04 0.090.01 0.120.03 0.050.06 0.070.51 0.020.48 0.010.02 0.100.08 0.050.03 0.010.04 0.110.57 0.060.55 0.020.02 0.080.13 0.060.25 0.030.17 0.050.16 0.150.36 0.060.02 (1, ∅) N S 0.360.40 0.080.09 0.080.02 0.010.01 0.870.85 0.600.01 0.270.02 0.730.01 0.160.13 0.040.04 0.050.04 0.320.29 0.130.01 0.640.14 0.450.05 0.080.02 0.280.04 0.050.09 0.190.19 0.150.18 0.030.07 0.220.00 0.110.04 0.120.08 0.120.31 0.170.38 0.010.01 0.050.09 0.150.16 0.080.14 0.030.09 0.070.09 0.150.16 (1, 0) N S 0.090.04 0.110.04 0.110.01 0.020.01 0.030.00 0.070.02 0.110.04 0.070.04 0.050.01 0.070.04 0.050.07 0.060.05 0.060.01 0.000.00 0.000.00 0.000.01 0.020.00 0.070.00 0.150.00 0.120.00 0.080.01 0.240.02 0.080.02 0.080.02 0.400.23 0.010.01 0.110.03 0.080.07 0.060.01 0.040.03 0.080.11 0.290.34 0.020.00 (0, ∅) N S 0.040.01 0.030.00 0.190.01 0.110.02 0.470.08 0.110.02 0.050.03 0.070.03 0.110.03 0.090.03 0.060.04 0.060.04 0.130.01 0.360.03 0.350.03 0.160.01 0.170.02 0.330.02 0.440.03 0.430.03 0.360.03 0.140.00 0.120.01 0.140.03 0.040.00 0.490.13 0.050.00 0.020.07 0.240.08 0.280.08 0.030.08 0.050.02 0.170.17 (∅, 0) 0.0 0.2 0.4 0.6 0.8 1.0 Figure 20 | Prerequisite 푁/푆-Scores under the four state contrasts, averaged over evaluated models. outcome remains limited by the other, unsupplied ones. Small scores therefore characterize a model’s reliance under the given contrast. Scope and Limitations. Input-level interventions. In our framework, prerequisite states are controlled by injecting answers into the prompt, so all metrics characterize how a model’s output responds to prerequisite information provided in its input, rather than how the prerequisite is represented inside the model. Operating at the input level allows the same controlled states to be applied across models, including closed-weight models accessible only through APIs, without relying on unverified readouts of intermediate states. How a model processes the injected prerequisite (whether adopting it, checking it against the visual input, or overriding it—is) therefore part of the behavior measured by the metrics rather than an external assumption: IC measures task performance when correct prerequisite answers are available in the context, and RC measures performance when incorrect ones are supplied. Empirically, models consistently respond to the injected content: under the same prompt template, supplying correct answers improves performance in all 48 model-category cells, whereas supplying incorrect answers degrades all 12 models in perception and spatial tasks (Section 5.2). Investigating the internal mechanisms by which models attend to and integrate injected prerequisites is an important direction for future work. Design of the incorrect intervention. For the incorrect state퐴 푖 =0, we inject an answer randomly sampled from the valid answer space of unit task푋 푖 (Section 4). This provides a standardized, model-independent perturbation that is identical across all evaluated models, ensuring that RC and the contribution scores involving state 0 are directly comparable across models. It is intended to test how models behave under an explicitly false premise, rather than to simulate the natural errors that a specific model would make on its own, which are already captured under the unassisted state 퐴 푖 =∅. 32 Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis PerceptionSpatialTemporalCognitiveOverall 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy 0.54 0.52 0.45 0.47 0.49 0.67 0.61 0.49 0.44 0.53 0.67 0.65 0.54 0.50 0.58 0.84 0.70 0.66 0.71 0.71 0.86 0.72 0.66 0.77 0.74 (a) By Category & Overall RCNC pa= q pa= 1 t IC Gem 3.5FGemma-31B 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy 0.60 0.48 0.72 0.61 0.72 0.61 0.86 0.83 0.89 0.83 (b) Perception Gem 3.5FGemma-31B 0.57 0.47 0.67 0.54 0.71 0.59 0.74 0.66 0.75 0.68 (c) Spatial Gem 3.5FGemma-31B 0.48 0.41 0.56 0.42 0.61 0.47 0.71 0.62 0.70 0.62 (d) Temporal Gem 3.5FGemma-31B 0.51 0.44 0.49 0.40 0.58 0.42 0.78 0.64 0.84 0.70 (e) Cognitive Figure 21|Capability on non-root unit tasks under five prerequisite states: RC, NC , question-only (푞), text-answer-only (1 푡 ), and IC. (a) Category-level and overall averages over 30 non-root unit tasks and two models; (b–e) per-model results within each category. 00.050.150.30.60.9 S(1, ∅) 0 0.05 0.15 0.3 0.6 0.9 S (1 t , ∅) S(1 t , ∅) vs. S(1, ∅) Category Perception Spatial Temporal Cognitive y= x Overall Task Map Nav Dice Sum Surround Loc Street-View Ord Music Note Dashcam Count Multi-View Seq Traffic Causation Pearson r= 0.74 | MAE = 0.106 Figure 22| 푆(1 푡 ,∅)vs.푆(1,∅)for each pre- requisite of an overall task, averaged over two models. Color denotes the prerequi- site’s capability category and marker shape the overall task. The solid line denotes 푦= 푥and the shaded band|푦− 푥| ≤0.06; dashed lines mark the 0.15 threshold as in Figure 5 (sqrt(·)-scaled axes). Evaluation cost. Evaluating a target task푌with푘= |Pa(푌)|prerequisites across the full ternary state space would in principle require 3 푘 prerequisite configurations. By setting the remaining prerequisites퐴 −푖 to homogeneous states (Appendix Section A.3.2), our framework reduces the required runs per instance to linear in푘: for each prerequisite푋 푖 , pairing퐴 푖 ∈ 1,0,∅with퐴 −푖 ∈ 1,0,∅yields nine state pairs, where the three all-same states (NC, IC, RC) are shared across all prerequisites, resulting in 3+6푘runs per instance across all four contrasts. In this paper, we evaluate all four contrasts to provide a complete characterization of model behavior across the ternary intervention space. In practical applications, the(1,∅)contrast is the most directly informative, as it diagnoses how much each prerequisite limits and can improve performance relative to the model’s natural state; evaluating(1,∅)alone requires only 2+2푘runs per instance (the standard end-to-end evaluation plus 1+2푘interventional runs). Exploring automated task decomposition is also a promising direction for scaling this diagnostic pipeline to broader task domains. Scope and future directions. This work focuses on diagnostic evaluation, establishing a causal framework and benchmark to identify where and why MLLMs fail on compositional tasks. Developing training or inference-time methods to address the diagnosed bottlenecks is a natural next step that we hope our work will inform. More broadly, our diagnostic framework can be extended to several related areas in future work, including localizing failure bottlenecks in multi-step agentic workflows, guiding targeted capability improvement in post-training and reinforcement learning, and complementing chain-of-thought reasoning analysis. 33 Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n X1 Object Localization X2 Sequential Navigation 0.300.410.290.350.410.360.580.310.510.380.520.38 0.030.100.000.020.140.030.070.060.100.030.000.14 (a) N(1, ∅) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n 0.590.570.650.600.420.460.340.490.370.400.460.35 0.040.130.290.150.030.140.000.110.140.220.000.14 (b) S(1, ∅) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n X1 Object Localization X2 Sequential Navigation 0.300.430.330.440.440.330.400.380.360.390.230.30 0.140.170.020.110.190.030.160.030.030.030.000.10 (c) N(1, 0) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n 0.390.500.660.500.460.350.160.380.450.390.320.29 0.100.160.170.170.140.080.000.010.070.050.000.13 (d) S(1, 0) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n X1 Object Localization X2 Sequential Navigation 0.160.080.000.000.000.170.000.120.120.000.380.05 0.030.100.000.080.080.210.420.000.170.010.200.06 (e) N(0, ∅) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n 0.000.020.150.040.000.100.000.000.000.050.080.05 0.000.000.120.000.090.010.120.080.010.070.000.03 (f) S(0, ∅) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n X1 Object Localization X2 Sequential Navigation 0.030.000.000.000.020.000.310.110.030.000.000.00 0.290.020.000.060.000.000.000.000.000.000.040.00 (g) N(∅, 0) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n 0.000.000.040.010.020.000.030.000.000.010.000.00 0.000.000.050.000.000.000.000.010.000.000.000.00 (h) S(∅, 0) 0.00 0.05 0.15 0.30 0.60 1.00 Figure 23|Per-model푁- and푆-Scores for the parents of Map Nav. under the four state contrasts. The vertical line separates SOTA from Others. 34 Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n X1 Object Detection X2 3D Surface Recognition X3 Fine-grained Counting 0.010.020.000.020.040.050.000.050.000.010.020.02 0.000.020.000.010.010.020.000.000.000.000.020.01 0.580.660.640.710.780.770.770.790.680.790.830.85 (a) N(1, ∅) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n 0.000.070.050.070.000.030.100.020.080.030.040.09 0.000.000.010.050.000.050.000.000.000.030.000.00 0.920.920.910.920.920.910.900.920.920.900.900.91 (b) S(1, ∅) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n X1 Object Detection X2 3D Surface Recognition X3 Fine-grained Counting 0.110.090.060.240.130.060.050.060.030.030.050.06 0.000.020.000.000.010.020.000.030.010.000.000.01 0.900.770.710.900.900.900.890.880.870.880.900.90 (c) N(1, 0) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n 0.000.000.130.010.010.040.030.000.000.010.000.00 0.000.010.020.020.020.040.030.000.000.000.000.00 0.870.890.790.660.880.880.820.880.880.890.890.85 (d) S(1, 0) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n X1 Object Detection X2 3D Surface Recognition X3 Fine-grained Counting 0.270.270.090.000.000.000.000.400.210.000.080.00 0.000.150.000.000.000.000.000.070.000.000.000.00 0.000.000.000.000.000.000.000.100.000.000.000.31 (e) N(0, ∅) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n 0.000.000.000.000.000.040.000.000.010.040.000.00 0.000.000.000.040.000.030.000.000.000.020.000.00 0.000.000.000.000.000.000.000.000.000.000.000.00 (f) S(0, ∅) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n X1 Object Detection X2 3D Surface Recognition X3 Fine-grained Counting 0.420.070.170.410.440.000.560.020.000.000.000.21 0.390.070.000.000.190.000.390.130.030.000.140.00 0.810.510.400.680.580.480.560.410.440.200.380.14 (g) N(∅, 0) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n 0.000.000.000.010.010.020.030.000.000.000.000.00 0.020.000.060.030.030.060.010.000.020.010.000.00 0.180.100.120.050.070.170.080.000.070.070.040.00 (h) S(∅, 0) 0.00 0.05 0.15 0.30 0.60 1.00 Figure 24 | Same as Figure 23, for Dice Sum. 35 Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n X1 Object Identification X2 Object Classification X3 Completeness Verification 0.000.000.000.030.190.000.000.100.020.060.000.10 0.000.000.050.030.110.000.000.040.040.080.140.00 0.160.090.150.110.190.060.030.230.230.240.160.27 (a) N(1, ∅) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n 0.530.740.540.690.640.700.530.630.530.380.270.30 0.530.570.560.670.530.580.400.570.590.340.420.38 0.630.810.760.740.620.590.500.690.550.340.490.39 (b) S(1, ∅) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n X1 Object Identification X2 Object Classification X3 Completeness Verification 0.610.470.470.500.770.550.470.630.680.670.640.75 0.020.290.240.230.210.010.070.410.540.490.360.41 0.830.490.530.750.870.830.680.770.750.710.830.67 (c) N(1, 0) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n 0.000.020.040.000.000.000.000.000.000.090.000.00 0.000.020.000.000.020.070.000.020.020.080.000.00 0.000.060.000.000.000.040.010.020.000.010.000.00 (d) S(1, 0) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n X1 Object Identification X2 Object Classification X3 Completeness Verification 0.000.040.000.000.120.000.000.000.000.000.480.24 0.300.000.000.000.000.000.190.000.270.000.330.29 0.030.000.090.000.000.000.000.000.000.200.000.52 (e) N(0, ∅) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n 0.000.000.000.080.040.130.000.000.000.000.000.00 0.000.000.030.090.150.050.040.000.010.070.000.02 0.000.000.000.090.110.130.000.010.020.000.000.08 (f) S(0, ∅) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n X1 Object Identification X2 Object Classification X3 Completeness Verification 0.390.050.010.000.000.000.050.010.450.360.050.00 0.400.050.000.000.000.000.000.080.000.000.110.00 0.370.120.010.000.000.000.080.000.000.150.160.00 (g) N(∅, 0) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n 0.020.000.010.120.000.050.000.060.010.010.000.00 0.000.020.010.050.020.110.000.090.000.020.000.00 0.000.020.000.090.020.050.030.040.080.000.010.00 (h) S(∅, 0) 0.00 0.05 0.15 0.30 0.60 1.00 Figure 25 | Same as Figure 23, for Music Note. 36 Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n X1 Object Classification X2 Spatial Relation X3 Object Re-identification X4 Motion Classification 0.110.070.200.110.000.310.200.120.160.190.080.00 0.000.000.040.050.010.040.150.000.180.050.110.00 0.020.020.180.000.000.000.080.000.050.000.130.00 0.200.170.350.180.370.130.280.370.200.290.280.00 (a) N(1, ∅) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n 0.310.100.220.380.140.160.210.210.270.120.120.17 0.130.020.110.080.000.000.020.000.110.000.000.16 0.240.000.150.210.000.080.000.080.270.070.000.19 0.510.570.370.570.500.220.270.380.450.280.320.53 (b) S(1, ∅) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n X1 Object Classification X2 Spatial Relation X3 Object Re-identification X4 Motion Classification 0.120.190.130.080.040.260.280.210.290.220.150.00 0.080.000.020.090.000.000.110.090.050.000.020.00 0.020.020.080.020.000.000.300.000.020.050.110.00 0.380.390.450.440.280.130.280.410.250.400.250.12 (c) N(1, 0) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n 0.140.240.160.180.050.210.220.090.240.000.020.00 0.000.000.040.000.000.000.130.240.000.000.080.00 0.000.070.000.110.000.000.060.140.000.010.000.04 0.440.440.340.470.050.190.280.260.120.220.290.35 (d) S(1, 0) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n X1 Object Classification X2 Spatial Relation X3 Object Re-identification X4 Motion Classification 0.000.000.280.000.130.000.000.000.110.060.060.00 0.030.000.000.100.210.270.000.000.000.250.000.00 0.080.000.000.000.020.000.000.000.180.170.100.00 0.000.000.000.000.100.000.000.000.250.180.130.10 (e) N(0, ∅) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n 0.000.000.000.000.000.010.000.000.000.000.000.07 0.160.000.120.000.000.000.040.040.090.000.000.03 0.160.100.150.130.000.000.000.000.000.080.000.17 0.020.040.000.040.000.000.000.100.070.030.060.19 (f) S(0, ∅) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n X1 Object Classification X2 Spatial Relation X3 Object Re-identification X4 Motion Classification 0.000.220.080.080.140.000.090.290.170.060.170.00 0.000.330.000.060.170.350.000.000.000.040.170.00 0.000.000.000.000.330.210.090.000.040.000.000.00 0.000.000.080.000.070.460.090.000.000.000.000.00 (g) N(∅, 0) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n 0.020.010.000.130.000.000.170.060.000.000.000.02 0.000.070.060.000.000.000.020.060.000.000.020.10 0.000.110.020.050.000.160.090.020.000.000.000.06 0.050.110.020.110.000.020.110.020.000.000.000.00 (h) S(∅, 0) 0.00 0.05 0.15 0.30 0.60 1.00 Figure 26 | Same as Figure 23, for Dashcam Count.. 37 Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n X1 Object Localization X2 Viewpoint Estimation X3 Distance Comparison X4 Motion Detection X5 Interaction Status Identification 0.000.000.100.000.050.160.340.200.000.050.000.00 0.440.460.530.530.520.520.720.650.350.290.300.31 0.130.050.250.310.100.310.000.390.190.220.300.13 0.000.010.030.030.010.250.400.160.000.000.000.20 0.070.050.100.000.130.230.500.120.000.250.000.00 (a) N(1, ∅) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n 0.040.080.000.000.100.000.000.040.000.000.000.00 0.520.350.190.120.280.150.150.180.040.050.350.05 0.240.020.030.000.060.010.000.110.000.020.020.01 0.030.000.000.000.020.000.000.010.000.000.000.00 0.150.020.040.000.070.040.000.040.010.000.010.01 (b) S(1, ∅) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n X1 Object Localization X2 Viewpoint Estimation X3 Distance Comparison X4 Motion Detection X5 Interaction Status Identification 0.040.000.040.070.120.550.370.100.000.220.000.00 0.740.660.640.700.580.820.820.850.240.430.430.71 0.310.350.370.410.400.670.690.620.320.250.670.38 0.000.000.020.000.030.480.350.110.000.000.000.00 0.080.200.140.190.470.540.720.330.050.310.270.13 (c) N(1, 0) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n 0.010.000.020.000.000.020.030.010.000.010.020.00 0.450.270.260.140.150.110.070.130.000.010.050.00 0.070.050.090.030.090.050.070.030.000.040.060.01 0.010.000.050.000.060.050.010.050.010.000.020.00 0.080.040.050.010.060.080.020.040.000.030.020.00 (d) S(1, 0) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n X1 Object Localization X2 Viewpoint Estimation X3 Distance Comparison X4 Motion Detection X5 Interaction Status Identification 0.000.270.000.480.000.000.000.000.000.000.000.00 0.000.000.000.000.000.000.000.000.000.000.000.00 0.000.000.000.000.000.000.000.000.000.000.000.00 0.000.000.000.020.000.000.000.000.000.000.000.00 0.000.230.000.000.000.000.000.000.000.000.000.00 (e) N(0, ∅) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n 0.000.000.060.000.040.000.000.030.010.000.000.01 0.000.000.000.000.000.000.000.000.000.010.000.00 0.000.000.000.000.000.010.000.000.000.000.000.00 0.020.000.000.000.040.020.000.050.000.020.000.00 0.000.000.000.000.020.000.000.000.000.000.000.00 (f) S(0, ∅) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n X1 Object Localization X2 Viewpoint Estimation X3 Distance Comparison X4 Motion Detection X5 Interaction Status Identification 0.020.120.000.170.000.040.370.000.000.610.170.00 0.460.300.440.430.390.210.720.120.550.000.720.00 0.340.340.100.460.110.000.870.120.600.530.760.00 0.000.110.220.350.000.000.440.000.350.000.480.00 0.170.290.140.190.000.000.310.060.540.140.170.00 (g) N(∅, 0) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n 0.010.000.040.000.020.040.000.050.000.000.010.00 0.110.020.050.000.020.090.020.070.000.000.020.00 0.050.090.030.050.010.040.020.030.000.010.020.00 0.010.020.000.000.010.070.000.040.000.010.000.00 0.000.000.010.010.040.110.010.010.000.000.010.01 (h) S(∅, 0) 0.00 0.05 0.15 0.30 0.60 1.00 Figure 27 | Same as Figure 23, for Multi-View Seq.. 38 Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n X1 Object Localization X2 Size Comparison X3 Distance Comparison X4 Horizontal Position Comparison X5 Motion Detection 0.000.060.040.020.120.000.150.000.000.000.150.05 0.000.030.000.040.260.210.090.000.000.000.150.00 0.000.010.000.000.030.000.080.000.000.020.150.00 0.000.000.000.000.010.000.050.000.000.000.110.00 0.120.030.000.040.070.070.180.000.000.210.320.17 (a) N(1, ∅) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n 0.060.000.000.180.000.230.000.050.000.060.100.00 0.380.500.510.640.540.400.630.810.710.430.360.22 0.440.500.510.710.390.330.560.630.650.470.280.28 0.030.000.000.000.000.000.000.060.090.000.000.07 0.350.000.000.020.000.290.000.000.000.170.070.00 (b) S(1, ∅) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n X1 Object Localization X2 Size Comparison X3 Distance Comparison X4 Horizontal Position Comparison X5 Motion Detection 0.010.020.050.040.060.000.110.160.000.190.000.00 0.000.150.280.220.130.000.330.150.200.510.000.36 0.140.070.030.020.360.010.390.260.200.290.000.02 0.010.010.000.000.060.000.200.000.000.080.000.00 0.350.110.100.110.300.000.430.120.000.410.350.39 (c) N(1, 0) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n 0.140.000.000.150.170.330.070.070.000.080.030.04 0.060.570.270.310.220.140.210.230.170.060.050.00 0.000.320.140.370.240.350.110.290.230.080.070.03 0.110.090.000.050.160.230.040.000.050.050.030.05 0.000.000.000.000.000.000.000.000.000.000.000.00 (d) S(1, 0) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n X1 Object Localization X2 Size Comparison X3 Distance Comparison X4 Horizontal Position Comparison X5 Motion Detection 0.040.040.000.000.000.560.170.000.000.000.000.00 0.040.110.210.000.000.000.110.500.000.000.630.17 0.000.000.000.000.180.000.440.280.000.000.250.33 0.000.000.000.000.120.050.000.140.000.300.310.08 0.180.170.240.220.000.000.440.430.000.340.500.33 (e) N(0, ∅) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n 0.000.000.000.000.000.000.000.000.000.020.000.00 0.000.000.000.000.000.000.000.000.000.000.000.00 0.000.000.000.000.000.000.000.000.000.000.000.00 0.000.000.000.000.000.080.000.000.000.000.000.00 0.000.000.000.000.000.170.000.000.000.000.040.00 (f) S(0, ∅) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n X1 Object Localization X2 Size Comparison X3 Distance Comparison X4 Horizontal Position Comparison X5 Motion Detection 0.190.100.500.250.330.020.470.330.560.000.620.60 0.190.330.380.650.360.280.530.490.840.340.620.20 0.320.280.400.610.360.020.710.490.640.370.870.07 0.120.170.380.380.560.000.500.110.840.230.500.47 0.110.130.080.160.290.000.120.160.320.000.000.33 (g) N(∅, 0) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n 0.000.000.020.050.060.000.000.000.010.060.040.04 0.000.000.000.150.070.060.000.000.000.010.000.00 0.090.090.000.070.000.020.000.000.010.070.000.00 0.170.020.030.040.000.000.010.000.060.000.000.00 0.000.000.000.000.020.020.000.000.000.000.000.00 (h) S(∅, 0) 0.00 0.05 0.15 0.30 0.60 1.00 Figure 28 | Same as Figure 23, for Street-View Ord.. 39 Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n X1 Object Identification X2 Front-View Identification X3 Viewpoint Estimation X4 Camera View Identification X5 Object Localization 0.040.000.000.020.110.000.040.030.070.000.130.10 0.000.000.000.000.000.000.100.000.000.000.030.00 0.120.030.010.030.150.420.040.110.140.140.030.15 0.020.000.000.000.000.000.370.090.090.010.150.03 0.010.060.000.020.010.160.020.000.010.000.000.00 (a) N(1, ∅) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n 0.000.020.000.070.000.000.000.050.050.020.130.02 0.000.000.000.130.010.000.000.060.070.030.120.00 0.780.790.700.720.570.610.270.570.450.460.420.47 0.730.690.650.720.600.490.750.520.400.200.410.47 0.000.000.000.120.000.000.000.000.000.030.110.00 (b) S(1, ∅) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n X1 Object Identification X2 Front-View Identification X3 Viewpoint Estimation X4 Camera View Identification X5 Object Localization 0.090.000.020.050.050.000.120.280.140.140.130.26 0.010.000.010.000.080.030.500.070.120.050.250.32 0.180.050.020.010.150.150.240.050.110.140.000.28 0.000.010.000.000.130.410.470.050.260.090.420.22 0.000.000.020.010.010.020.010.000.040.000.000.00 (c) N(1, 0) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n 0.010.000.280.000.000.030.060.000.000.070.080.00 0.000.050.260.050.050.080.030.160.000.080.140.03 0.570.600.470.490.300.240.280.210.110.260.050.09 0.380.550.620.590.490.320.640.220.180.190.260.18 0.000.040.000.000.010.000.000.000.000.010.000.00 (d) S(1, 0) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n X1 Object Identification X2 Front-View Identification X3 Viewpoint Estimation X4 Camera View Identification X5 Object Localization 0.060.000.000.080.010.210.000.150.190.000.000.27 0.080.000.000.030.230.000.000.000.020.000.040.51 0.410.570.800.500.560.000.000.570.460.420.200.34 0.000.000.000.000.000.000.000.000.160.000.000.00 0.110.090.060.280.090.000.000.090.370.000.000.28 (e) N(0, ∅) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n 0.000.000.000.030.000.000.000.050.000.000.130.00 0.000.000.000.050.000.000.000.080.000.040.100.00 0.190.190.670.410.230.010.000.310.210.210.290.05 0.000.000.000.000.000.000.000.000.000.020.040.00 0.000.060.000.090.000.000.000.070.020.030.120.00 (f) S(0, ∅) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n X1 Object Identification X2 Front-View Identification X3 Viewpoint Estimation X4 Camera View Identification X5 Object Localization 0.170.170.240.000.050.000.370.000.180.190.000.06 0.230.280.290.000.200.280.280.000.060.000.000.10 0.000.000.000.000.000.000.450.000.000.000.000.00 0.850.430.630.600.690.700.800.360.330.000.000.52 0.050.000.130.000.020.140.110.000.000.000.000.16 (g) N(∅, 0) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n 0.000.030.000.000.000.000.020.000.000.000.070.00 0.000.020.140.000.000.020.000.030.000.120.000.00 0.000.000.000.000.000.010.000.000.000.000.000.00 0.100.210.460.130.190.060.230.050.000.030.100.04 0.000.000.000.000.000.020.010.000.000.010.010.00 (h) S(∅, 0) 0.00 0.05 0.15 0.30 0.60 1.00 Figure 29 | Same as Figure 23, for Surround Loc.. 40 Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n X1 Object Localization X2 State Change Recognition X3 Reference-Based Action Recognition X4 Temporal Localization X5 Temporal Sequencing X6 False Correlation Rejection 0.020.130.040.000.000.440.170.000.070.000.000.06 0.010.070.010.000.000.260.080.000.080.050.000.21 0.000.050.050.010.150.040.000.000.030.030.000.00 0.000.000.050.060.130.100.090.000.000.100.000.07 0.010.060.060.000.430.490.100.000.120.270.000.26 0.000.120.060.020.000.230.000.000.030.160.000.07 (a) N(1, ∅) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n 0.180.240.340.180.000.150.110.070.240.070.000.00 0.560.380.580.360.000.000.200.000.370.220.300.03 0.520.430.460.240.000.000.000.000.320.070.030.00 0.420.240.310.280.110.000.040.090.180.270.000.00 0.640.570.760.730.000.000.150.120.540.290.220.31 0.000.000.090.020.000.000.000.000.110.010.000.00 (b) S(1, ∅) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n X1 Object Localization X2 State Change Recognition X3 Reference-Based Action Recognition X4 Temporal Localization X5 Temporal Sequencing X6 False Correlation Rejection 0.020.060.060.000.000.260.120.000.020.000.000.10 0.050.190.120.030.080.640.090.000.190.250.000.11 0.040.020.070.040.000.440.000.000.100.130.000.08 0.000.060.010.000.000.050.030.000.000.140.000.06 0.010.050.040.010.000.370.000.000.040.160.000.14 0.130.170.120.110.000.370.210.040.250.220.000.21 (c) N(1, 0) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n 0.120.490.000.250.000.000.000.000.090.000.020.07 0.240.500.320.300.040.080.210.000.190.010.000.07 0.260.450.070.410.000.030.110.000.220.030.060.00 0.070.460.040.320.000.050.000.000.090.010.000.00 0.070.460.040.370.000.000.060.000.000.080.000.00 0.330.580.090.410.010.030.110.150.170.000.000.05 (d) S(1, 0) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n X1 Object Localization X2 State Change Recognition X3 Reference-Based Action Recognition X4 Temporal Localization X5 Temporal Sequencing X6 False Correlation Rejection 0.020.000.020.000.410.000.100.000.000.060.250.08 0.000.000.000.000.440.000.030.040.000.040.070.11 0.000.000.000.000.190.000.010.050.000.000.180.08 0.000.000.020.000.510.000.030.000.020.090.200.05 0.230.000.230.030.570.000.200.210.390.340.680.56 0.000.000.000.000.180.000.000.070.000.000.000.00 (e) N(0, ∅) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n 0.060.020.250.170.000.030.080.040.120.030.000.08 0.000.000.150.000.000.000.000.000.030.000.000.00 0.000.050.220.030.000.000.000.000.000.000.000.00 0.280.050.230.230.050.000.130.000.190.180.000.00 0.580.460.670.540.000.000.240.350.560.390.150.12 0.000.000.000.000.000.000.000.000.060.000.000.00 (f) S(0, ∅) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n X1 Object Localization X2 State Change Recognition X3 Reference-Based Action Recognition X4 Temporal Localization X5 Temporal Sequencing X6 False Correlation Rejection 0.000.000.000.000.120.000.000.000.000.000.150.00 0.210.050.000.170.560.800.270.160.000.040.380.24 0.110.000.000.000.680.840.370.170.060.160.400.52 0.000.000.000.000.000.140.000.030.000.000.040.10 0.000.000.000.000.350.250.000.000.000.000.000.00 0.020.090.100.070.150.250.140.220.000.350.380.26 (g) N(∅, 0) GPT-5.4 Gem 3.5F Gem 3.1P Gem 3F Qwen27B Qwen397B Kimi K2.5 Gemma-31B Gem 3.1FL Gemma-26B Grok 4.3 GPT-5.4n 0.000.460.000.270.000.000.000.130.030.000.000.00 0.100.470.040.240.000.020.000.000.100.000.000.00 0.120.430.070.270.000.000.000.000.050.050.000.00 0.100.520.000.210.000.000.000.110.000.000.000.00 0.000.230.000.000.000.000.000.000.000.000.000.00 0.360.570.020.480.000.010.070.000.140.130.080.23 (h) S(∅, 0) 0.00 0.05 0.15 0.30 0.60 1.00 Figure 30 | Same as Figure 23, for Traffic Causation. 41