Paper deep dive

Humanity's Sixth Sense: Benchmarking Intuitive Visual Reasoning in Multimodal Models

Xingang Guo, Jing Gu, Brian Jang, Renxiong Wang, Utkarsh Tyagi, Daniel Quigley, Steven Li, David Yan, Daniel Yue Zhang, Darvin Yi, Forrest Huang, HiJae Kim, Tianyi Zhang, Jared Lichtarge, Jihua Huang, Le Xue, Manan Tomar, Qiuyi Richard Zhang, Ruofei Yu, Seth Neel, Yaning Hu, Marcella Valentine, Xinzhe Jiang, Daniel Evans, Chenguang Wang, Dustin Tran, Tong Zhao, Yinfei Yang, Yunzhong He

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 100

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%

Last extracted: 10/8/2026, 6:36:09 AM

Summary

The paper introduces Humanity's Sixth Sense (HSS), a benchmark designed to evaluate intuitive visual reasoning in Multimodal Large Language Models (MLLMs). Unlike existing benchmarks that focus on explicit perception or expert-level analysis, HSS tests the ability to infer implicit temporal, spatial, social, and abstract information from visual scenes. The benchmark comprises 522 rigorously vetted tasks across four domains. Experiments with 25 frontier models reveal a significant performance gap: while human participants achieve 93.1% accuracy, the best model (GPT-6-astra) reaches only 53.6%. The study highlights that current scaling strategies and agentic tool use do not sufficiently close this gap, identifying intuitive visual reasoning as a distinct and challenging capability for AI.

Entities (9)

HSS · benchmark · 99%GPT-6 Astra · model · 95%Abstract & Contextual Inference · category · 92%Physical & Spatial Logic · category · 92%Social Understanding · category · 92%Temporal & Causal Dynamics · category · 92%Multimodal Large Language Models · technology · 90%BabyVision · benchmark · 85%SimpleBench · benchmark · 85%

Relation Signals (8)

Human Participants → achievesaccuracyon → HSS

confidence 95% · participants reach 93.1% accuracy

GPT-6 Astra → achievesaccuracyon → HSS

confidence 95% · the strongest model, GPT-6-astra, reaches only 53.6% even at maximum reasoning effort.

HSS → evaluates → intuitive visual reasoning

confidence 95% · HSS is a benchmark for intuitive visual reasoning.

HSS → containsdomain → Abstract & Contextual Inference

confidence 90% · HSS comprises open-ended tasks spanning four core domains: ... and abstract contextual inference

HSS → containsdomain → Physical & Spatial Logic

confidence 90% · HSS comprises open-ended tasks spanning four core domains: ... physical & spatial logic ...

HSS → containsdomain → Temporal & Causal Dynamics

confidence 90% · HSS comprises open-ended tasks spanning four core domains: temporal & causal dynamics...

HSS → containsdomain → Social Understanding

confidence 90% · HSS comprises open-ended tasks spanning four core domains: ... social understanding ...

BabyVision → comparedto → HSS

confidence 85% · In vision, the closest counterpart is BabyVision... However, while BabyVision tests whether models perceive what is explicitly present, HSS evaluates whether they infer what is implicit.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Humans perceive far more in a scene than what is explicitly depicted: a single glance captures past causes and future trajectories; a quick peek determines if a vehicle can fit between two parked cars; a few seconds of video reveals who holds authority in a room; and a fleeting clip highlights subtle abstract patterns like unwritten rules or hidden labels. This capacity reflects a form of humanity's sixth sense: an intuitive reasoning mechanism that recovers implicit information beyond raw sensory perception. Crucially, this rapid, zero-shot visual intuition underpins everyday navigation and social interaction, making it a vital capability for Multimodal Large Language Models (MLLMs) deployed alongside people. Existing visual benchmarks, however, target either deliberate expert-level analysis in academic and mathematical domains or low-level perception, leaving the intuitive reasoning that people perform largely untested. To bridge this gap, we introduce Humanity's Sixth Sense (HSS), a benchmark for intuitive visual reasoning. HSS spans diverse image and video inputs, organizes items under a structured taxonomy, and pairs each with human-written prompts probing the implicit temporal, spatial, social, and abstract structure that people infer at a glance. Frontier MLLMs fall short of human performance: participants reach 93.1% accuracy, while the strongest model, GPT-6-astra, reaches only 53.6% even at maximum reasoning effort. Despite excelling in many complex tasks that require advanced perception and knowledge, current models still struggle significantly on these visual tasks that are intuitive for humans. We further explore agentic setup that apply dynamic visual manipulation to HSS, which narrows but does not close the gap. HSS establishes intuitive visual reasoning as a measurable axis and directs attention to a capability that scaling on current benchmarks has so far left behind.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

96,588 characters extracted from source content.

Expand or collapse full text

HUMANITY’S SIXTH SENSE: BENCHMARKING INTU- ITIVE VISUAL REASONING IN MULTIMODAL MODELS Xingang Guo 1∗ , Jing Gu 2∗ , Brian Jang 1∗ , Renxiong Wang 1 , Utkarsh Tyagi 1 , Daniel Quigley 1 , Steven Li 1 , David Yan 2 , Daniel Yue Zhang 1 , Darvin Yi 1 , Forrest Huang 2 , HiJae Kim 1 , Tianyi Zhang 2 , Jared Lichtarge 2 , Jihua Huang 2 , Le Xue 2 , Manan Tomar 2 , Qiuyi Richard Zhang 2 , Ruofei Yu 2 , Seth Neel 2 , Yaning Hu 2 , Marcella Valentine 2 , Xinzhe Jiang 2 , Daniel Evans 1 , Chenguang Wang 1,3 , Dustin Tran 2 , Tong Zhao 1 , Yinfei Yang 2 , Yunzhong He 1 1 ScaleAI, 2 Elorian, 3 University of California, Santa Cruz Leaderboard page: https://scale.com/leaderboard/hss Dataset page: https://huggingface.co/datasets/ScaleAI/HSS 0 20 40 60 80 100 pass@1 (%) Human GPT-6-astra GPT-6.1-sol Claude-Opus-5.5 Gemini-3.8-flash Claude-Fable-5.1 Gemini-3.7-flash Muse Spark 1.3 Claude-Fable-5 Qwen3.8-Max Gemini-3.5-flashGemini-3.6-flash GPT-6-sol Qwen3.8-flash Claude-Opus-5 GPT-5.6-sol GLM-5.3-flash Kimi-K3 MiMo-v2.6-flash MiMo-v2.6-pro Claude-Sonnet-5 Muse Spark 1.2 Claude-Opus-4.8 GPT-5.6-terra GPT-5.6-luna GPT-6-luna 93.1 53.6 46.6 44.6 41.6 40.8 39.8 37.4 34.5 33.0 32.6 31.9 31.2 30.9 30.5 30.0 26.8 25.5 25.4 25.2 24.8 24.5 23.5 22.7 21.6 21.0 OpenAIAnthropicGoogleMuseZhipuQwenXiaomiMoonshotHuman Figure 1: HSS leaderboard. Performance on HSS for 25 multimodal models. Every model scores far below people: the best, GPT-6-astra, reaches 53.6% where humans reach 93.1%. ABSTRACT Humans perceive far more in a scene than what is explicitly depicted: a single glance captures past causes and future trajectories; a quick peek determines if a vehicle can fit between two parked cars; a few seconds of video reveals who holds authority in a room; and a fleeting clip highlights subtle abstract patterns like unwritten rules or hidden labels. This capacity reflects a form of humanity’s sixth sense: an intuitive reasoning mechanism that recovers implicit information beyond raw sensory perception. Crucially, this rapid, zero-shot visual intuition underpins everyday navigation and social interaction, making it a vital capability for Multimodal Large Language Models (MLLMs) deployed alongside people. Existing visual benchmarks, however, target either deliberate expert-level analy- sis in academic and mathematical domains or low-level perception, leaving the intuitive reasoning that people perform largely untested. To bridge this gap, we introduce Humanity’s Sixth Sense (HSS), a benchmark for intuitive visual rea- soning. HSS spans diverse image and video inputs, organizes items under a struc- tured taxonomy, and pairs each with human-written prompts probing the implicit temporal, spatial, social, and abstract structure that people infer at a glance. Fron- tier MLLMs fall short of human performance: participants reach 93.1% accuracy, while the strongest model, GPT-6-astra, reaches only 53.6% even at maximum reasoning effort. Despite excelling in many complex tasks that require advanced ∗ Equal contribution. 1 arXiv:2610.08966v1 [cs.AI] 6 Oct 2026 perception and knowledge, current models still struggle significantly on these vi- sual tasks that are intuitive for humans. We further explore agentic setup that apply dynamic visual manipulation to HSS, which narrows but does not close the gap. HSS establishes intuitive visual reasoning as a measurable axis and directs attention to a capability that scaling on current benchmarks has so far left behind. 1INTRODUCTION A single glance at a visual scene reveals far more than what is explicitly rendered in pixels. Within a fraction of a second, we intuitively register who holds authority in a room, sense the subtle ten- sion behind a polite smile, or notice a toddler edging toward the stairs, long before anything goes wrong. We call this capacity humanity’s sixth sense: a rapid, pre-verbal form of visual inference that transforms circumstantial evidence into actionable conclusions. For AI agents deployed in homes, vehicles, and workplaces, this skill is not a luxury, but a core necessity: an agent must perceive the world through a human lens so that its predictions and behaviors naturally align with the people around it. This capacity relies on visual intuition. In cognitive science, intuition is characterized as an experience-based process that yields spontaneous judgments without conscious derivation (Kotler et al., 2025), functioning as skilled pattern recognition in environments with learnable regularities (Kahneman & Klein, 2009). In human vision, this is remarkably concrete: people categorize natural scenes within roughly 150ms (Thorpe et al., 1996) while effortlessly extracting implied physical dynamics, human motives, and social structures (Spelke & Kinzler, 2007; Battaglia et al., 2013; Baker et al., 2017). We synthesize these insights to define an inference as intuitive when it is fast, automatic, grounded in everyday experience rather than step-by-step derivation, and accessible even when its underlying rationale is difficult to articulate. In text-only domains, benchmarks like SimpleBench (Philip & Hemang, 2024) have shown that fron- tier models lag significantly behind humans on commonsense spatio-temporal and social reasoning. In vision, the closest counterpart is BabyVision (Chen et al., 2026), which highlights model fail- ures on basic visual tasks that young children solve effortlessly. However, while BabyVision tests whether models perceive what is explicitly present, HSS evaluates whether they infer what is im- plicit. Visual scenes carry vast unstated context such as revealing interpersonal relationships, latent intentions, physical affordances, and imminent consequences without ever stating them. Existing visual benchmarks fail to probe this dimension: they either treat images as prompts for specialized academic problem-solving (e.g., MMMU (Yue et al., 2024), HLE (Phan et al., 2025)) or isolate nar- row perceptual capabilities like visual searching and object tracking (e.g., V* (Wu & Xie, 2024), Video-MME (Fu et al., 2024a), Video-MME-v2 (Fu et al., 2026)). Meanwhile, existing probes of physical and social intelligence (Zellers et al., 2019; Riochet et al., 2022; Kamath et al., 2023; Zadeh et al., 2019; Bear et al., 2021; Jin et al., 2024) remain fragmented across isolated domains, rely heav- ily on synthetic environments, or depend on multiple-choice formats that allow models to succeed through process-of-elimination heuristics rather than true visual inference. To address this gap, we introduce Humanity’s Sixth Sense (HSS), a benchmark specifically target- ing the domain of intuitive visual reasoning that operates between raw perception and deliberate, step-by-step logic. Concretely, an HSS task requires interpreting visual evidence alongside contex- tual, physical, social, and commonsense knowledge to deduce unstated implications, latent states, and likely outcomes. For instance, viewers must infer an agent’s intended goal, recognize physi- cal affordances, reconstruct prior events, or anticipate imminent consequences purely from visual cues and everyday experience. HSS comprises open-ended tasks spanning four core domains: tem- poral & causal dynamics, physical & spatial logic, social understanding, and abstract contextual inference (Figure 2), categorized into 11 subdomains that reflect human scene interpretation (Sec- tion 2.2). To avoid multiple-choice artifacts, models generate free-form responses evaluated against human-annotated atomic rubrics. Our experiments reveal a substantial capability gap (Figure 1). Human participants achieve 93.1% pass rate on HSS, whereas the strongest model evaluated, GPT-6-astra, attains only 53.6% even at maximum reasoning effort. Moreover, standard scaling strategies do not close this gap: increased test-time deliberation, higher reasoning effort, and agentic tool use each yield only modest improve- ments, and performance remains far from saturation. These results indicate that intuitive visual 2 Temporal & Causal Dynamics What happened when woman placed the circular rail from her right to left? Golden answer. Observing the action of the woman moving the circular rail, i.e., switching the rail from her right to left from 00:15 to 00:20, helped change the direction of the track's points. Video Social Understanding Why does the woman in the white cap and green coat slow down after running? Golden answer. She slows down because the bus she is chasing stops and waits for her, so she no longer needs to run. Video Physical & Spatial Logic Will another vehicle be able to park between the red and the black sedans? Explains why. Golden answer. Yes, another vehicle will be able to park between the red and black Image Abstract & Contextual Inference Based on the labels on the other beakers, what number can be inferred on the label of the last beaker, despite being difficult to read? Golden answer. The final beaker in the row reads "80." The row of five starts at "40." and the next three rise in increments of ten to "50," "60" and "70," so the fifth continues the same pattern. Image Rubric 1. Mentions that it helped change the direction of the track's points. Rubric 1. States that another vehicle will be able to park between the red and black sedan. 2. Explains that there is an empty parking lane between the red and black sedan. Rubrics 1.States that the woman slows down because the bus stops instead of driving off. 2.States that the bus stopping means it is waiting for her, so she no longer needs to run to reach it. Rubric 1. States that the last beaker in the row reads 80. Figure 2: HSS task examples across the four domains. Each panel shows the input media (a sin- gle image or sampled video keyframes), human-written question, reference answer, and evaluation rubric. Temporal & causal dynamics: identifying the physical outcome of a manual track-switching action in video. Physical & spatial logic: judging whether a vehicle can fit between two parked sedans without explicit scale markers. Social understanding: inferring why a person stops running after a bus in a video clip. Abstract & contextual inference: deducing an occluded beaker label from an arithmetic progression of neighboring legible labels. reasoning constitutes a distinct and largely unexplored dimension of visual intelligence, one that does not emerge simply from scaling test-time compute. Our contributions are summarized as follows: 1. A structured taxonomy of intuitive visual reasoning. We formalize intuitive visual reasoning across four core domains and eleven subdomains, establishing a principled cognitive framework for evaluating implicit visual inference. 2. A rigorous, human-verified benchmark. We introduce HSS, a benchmark specifically curated to assess implicit visual reasoning. To address the inherent subjectivity of human intuition, all instances are authored by human annotators and pass through a multi-stage independent review pipeline to guarantee strict human consensus and high data precision. 3. Comprehensive empirical evaluation.Evaluating state-of-the-art MLLMs under direct- inference reveals that current MLLMs achieve barely half of human performance (53.6% vs. 93.1%), establishing HSS as a formidable open challenge. 4. Agentic paradigms and ablation analyses. We examine the effects of agentic tool-use work- flows and reasoning effort, demonstrating that intuitive visual reasoning represents a capability gap that remains resistant to both agentic augmentation and standard compute scaling. 2THE HSS BENCHMARK 2.1TASK FORMAT Each benchmark instance is a tuple (m,q,g,R): a visual media m (a static image or a video clip), a natural-language question q, a human-authored reference answer g, and a set of atomic rubrics R = r 1 ,...,r k , each independently checkable against g and mandatory for an answer to count as correct. Given m and q, a model returns a free-form answer a, and a judge marks every criterion 3 r i ∈ R as met or unmet given q, a, g, and R; an instance counts as solved only when all k are met (Section C.4). 2.2HSS TAXONOMY DESIGN Human intuitive visual reasoning enables us to infer unobserved dynamics, physical constraints, mental states, and contextual rules directly from visual input. Our taxonomy builds on three core domains established in cognitive science, including core knowledge of objects and space (Spelke & Kinzler, 2007), intuitive physics (Battaglia et al., 2013), and mentalizing (Baker et al., 2017), and further introducing an additional domain for abstract regularities implied by a scene. To render this latent capacity measurable, we formalize it into four core domains encompassing eleven subdomains (Table 4): 1. Temporal & Causal Dynamics (Time): Reconstructs the unobserved timeline connecting vi- sual states, evaluating inferences about past events (Retrodiction), underlying physical drivers (Mechanistic Causality), and future trajectories (Extrapolation). 2. Physical & Spatial Logic (Space): Reasons over unshown content (Hidden & Invisible), func- tional interactions (Affordance & Feasibility), novel perspectives (Spatial “Alien Viewpoint”), and physical accessibility under spatial constraints (Spatial Reachability). 3. Social Understanding (Mind): Formalizes the invisible mental and normative substrate of hu- man interaction, capturing intent and belief attribution (Theory of Mind) as well as implicit social expectations (Social Role, Norm & Power Dynamics). 4. Abstract & Contextual Inference (Context): Derives non-perceptual implications that a scene implies but never explicitly depicts, assessing unseen state transitions (Change & Consequence) and structural regularities (Patterns & Pareidolia). Rather than treating visual intuition as an unanalyzable or monolithic capacity, the taxonomy de- scribed above provides a principled foundation that fulfills a dual purpose: first, as a generative blueprint that enforces rigorous, multi-faceted coverage during benchmark curation; and second, as a fine-grained diagnostic framework for model evaluation. By projecting performance across these eleven cognitive dimensions rather than relying solely on aggregate scores, the taxonomy disentan- gles distinct competency profiles and exposes the specific latent reasoning gaps of state-of-the-art multimodal models. 2.3TASK DESIGN PRINCIPLES Because intuitive visual reasoning inherently risks subjective interpretation, task curation in HSS is strictly governed by three core design principles: 1. Exclusion of Pure Perception: Tasks must require latent inference beyond what is explicitly depicted in the visual media. We reject tasks that rely purely on fine-grained visual recogni- tion, regardless of visual complexity. If a high-resolution crop of the target region makes the answer immediately obvious, the task evaluates visual acuity rather than intuitive reasoning and is discarded. 2. Exclusion of Specialized Expertise: Tasks must target intuitive reasoning accessible to any attentive adult. We strictly prohibit items requiring symbolic calculation, specialized academic knowledge, or niche professional training, ensuring the benchmark purely isolates common- sense visual intuition. 3. Strict Human Consensus and Unambiguity: Both questions and target answers must com- mand unambiguous human consensus. Prompts are carefully structured to prevent subjective or divergent interpretations, and a task is admitted into the benchmark only if independent review- ers reach unanimous agreement on the ground truth. 2.4BENCHMARK CONSTRUCTION PIPELINE Each task in HSS is authored by a trained human annotator through a three-step workflow: (1) selecting a source image or video clip from a curated media pool; (2) assigning the item to one of 4 our four core domains and eleven subdomains; and (3) composing a natural-language prompt q, a gold reference answer g, and a structured rubric R decomposing g into fine-grained, independently checkable criteria. Authored tasks then enter a rigorous quality-control pipeline consisting of three independent review rounds. At each stage, a reviewer renders one of three verdicts: Pass: Advances the task to the subsequent review stage. Edit: Returns the task to the authoring phase for revision with specific feedback, after which it must re-enter the review pipeline from the beginning. Drop: Permanently removes the item if the prompt is ambiguous, the visual media is uninformative, or the target inference fails to command unanimous human consensus. Only tasks that achieve unanimous approval across all three independent rounds are admitted into the benchmark, guaranteeing that every released rubric has been verified by at least three reviewers beyond its author. The review pipeline is highly lossy, and this attrition is the primary driver of dataset quality: out of 3,466 initially authored tasks, 1,164 survive the first review stage, 559 pass the second, and 522 pass the third, yielding an overall end-to-end acceptance rate of 15.1%. The most significant filter occurs at the first review gate, where approximately two-thirds of submitted tasks are filtered out, primarily for relying on pure visual perception or exhibiting subjective prompts and target answers. Appendix B (Figure 8) details the complete flowchart and attrition mechanics across each review stage. The final released benchmark comprises these 522 strictly vetted tasks across 4 domains and 11 subdomains (288 image-based and 234 video-based instances), carrying 723 fine-grained rubric criteria in total (Table 3). 3EXPERIMENTS We evaluate 25 representative MLLMs on HSS. For models supporting native video inputs, we feed raw video files directly. Otherwise, we apply uniform frame sampling capped at 500 frames; for models with stricter input limits, we default to their maximum supported frame capacity. Static images are evaluated at native resolution, downscaling only when required by API or payload con- straints. Model outputs are graded by Claude-Opus-5, serving as an automated LLM judge. Because the HSS data curation process retains only mandatory criteria and excludes non-essential ones, a re- sponse is marked as successful if and only if it satisfies all of the corresponding rubric criteria. Fur- ther details regarding the experimental setup, prompts, and evaluation protocol are in Appendix C. 3.1BENCHMARK RESULTS Table 1 reports 25 models from eight vendors on HSS. Several key findings stand out: HSS is far from saturated despite high human performance. The top-performing model, GPT-6- astra (maximum reasoning effort), achieves only a 53.6% pass rate, compared to 93.1% for human annotators. Most models perform below 40%. In addition, across all 25 models, the average rea- soning token usage is around 4,000 per task. Even on tasks intuitive to humans, models generate substantial reasoning chains before answering. While extended chain-of-thought reflects current reasoning alignment techniques, models frequently overthink without arriving at the correct answer for HSS. Results breakdown across media type and domains. Video tasks prove more challenging than static images for 23 of the 25 models, with an average drop of 7.3 percentage points. This gap highlights ongoing struggles with long temporal horizons and key-frame localization. Social under- standing is the weakest domain across models; it is the lowest-performing domain for 21 of the 25 models, averaging 24.4% accuracy compared to 34.1% across the remaining three domains. Visual inputs are indispensable, preventing pure language-prior guessing. The blind control ex- periment confirms that HSS cannot be solved through language priors alone: stripping visual media drops the top model’s score from 53.6% to 6.6%. The residue is guessing rather than knowledge: only 4 of 522 tasks pass blind on all three attempts, and the passes concentrate on two-way questions, where the model commits to one side and invents supporting detail (Appendix C.5). Human performance. Twenty annotators answered the tasks under the same free-form conditions as the models and reached 93.1%, ahead of the best model by nearly 40 points. Each answer also car- ries a self-reported confidence. Annotators were unsure on only 63 of 522 tasks (12.1%) and scored 5 Table 1: Main results. Pass@1 (%) on HSS. A task counts as correct only when every rubric criterion is met; ± is the half-width of a 95% bootstrap interval over tasks. The columns re-cut the same tasks by media type and by domain, and think-tok / ans-tok are mean thinking and answer tokens per task. The best model value is bold and the second best is underlined , and the blind control re-runs the strongest model with the media removed. VendorModel (Reasoning Effort)pass@1image videotemporal physical social abstractthink-tok ans-tok Human baseline93.194.191.997.094.989.189.2— OpenAI GPT-6-astra (max)53.6±4.155.6 51.158.2 53.6 46.1 55.62,89640 GPT-6.1-sol (max)46.6±3.845.547.955.245.836.447.42,29836 GPT-6-sol (max)31.2±3.431.830.538.128.622.436.33,29830 GPT-5.6-sol (max)30.0±3.328.531.938.626.327.328.13,63039 GPT-5.6-terra (max)22.7±3.224.120.928.421.019.121.95,79445 GPT-5.6-luna (max)21.6±3.223.519.426.924.613.318.65,14642 GPT-6-luna (max)21.0±3.023.318.226.924.19.420.63,99339 Anthropic Claude-Opus-5.5 (xhigh)44.6±3.851.336.544.044.934.8 55.61,496294 Claude-Fable-5.1 (xhigh) 40.8±3.644.736.043.341.134.244.13,478360 Claude-Fable-5 (max)34.5±3.737.231.338.137.130.030.43,880463 Claude-Opus-5 (max)30.5±3.532.527.931.335.022.430.12,888445 Claude-Sonnet-5 (max) 24.8±3.229.518.929.127.514.225.83,209408 Claude-Opus-4.8 (max) 23.5±3.227.218.927.126.514.223.52,416408 Google Gemini-3.8-flash (high)41.6±3.747.234.843.337.540.048.410,407158 Gemini-3.7-flash (high)39.8±3.745.433.040.540.935.841.51,858118 Gemini-3.5-flash (high)32.6±3.435.129.635.631.631.831.42,456184 Gemini-3.6-flash (high)31.9±3.432.531.237.129.732.428.43,166178 Muse Muse Spark 1.3 (max)37.4±3.544.828.239.638.125.845.86,278281 Muse Spark 1.2 (xhigh)24.5±3.127.520.823.624.221.229.73,640412 Qwen Qwen3.8-Max (max)33.0±3.638.326.238.135.221.034.77,423217 Qwen3.8-flash (max)30.9±3.337.222.937.929.218.637.17,260200 ZhipuGLM-5.3-flash (max)26.8±3.030.222.631.326.318.231.04,484247 Moonshot Kimi-K3 (max)25.5±3.129.121.233.127.415.523.24,94998 Xiaomi MiMo-v2.6-flash (high)25.4±2.929.220.630.129.612.126.72,294182 MiMo-v2.6-pro (high)25.2±2.930.418.628.329.514.525.72,499192 ControlGPT-6-astra (max) [blind]6.6±1.52.411.79.55.54.86.51,04641 94.1% on the 459 they were sure about. Even the unsure answers scored 85.7%: the competence HSS measures stays available to people when their confidence is not. 3.2MODEL FAILURE ANALYSIS To investigate why models fail, we categorize 8,573 failed first attempts from the main evaluation using an eleven-cause taxonomy grouped into three families. Perception: errors related to properties explicitly present in the visual signal (e.g., scale, geometry, facing direction, and spatial alignment). Extending beyond low-level object detection (e.g., recognizing a person or car), this involves evalu- ating visible visual evidence for intuitive reasoning. Latent Inference: errors related to information not directly rendered in pixels (e.g., unobserved past events, hidden quantities behind obstacles, or underlying motives), requiring the model to bridge the gap between what is seen and what remains unobserved or implied. Reasoning: errors related to logical deduction, where the model fails to synthesize correct explicit and latent visual facts to draw valid conclusions. As shown in Figure 3, failures stem predominantly from visual perception (53%) and latent infer- ence (41%), whereas abstract reasoning accounts for only 5%. Overall, the two largest single failure modes are missing a decisive visual cue (21%) and misidentifying an object, person, or role (20%). Furthermore, failure patterns are highly systematic across architectures rather than model-specific: on tasks where five or more models fail, a median of 80% fail due to the exact same cause. Subdo- mains also exhibit distinct failure signatures: affordance and spatial reachability fail primarily on size and fit (62% and 56%), extrapolation on facing direction (42%), hidden & invisible on hidden quantities (44%), and mechanistic causality and theory of mind on missed cues (47% and 39%). Alien viewpoint is the only subdomain with a notable reasoning component, led by line-of-sight errors (23%). Extended failure analyses and case studies are provided in Appendices D and E. 6 201212978 21 All subdomains 23 2172018 Retrodiction 215139 47 Mechanistic causality 13 42 1417 Extrapolation 1565 44 719 Hidden & invisible 7 62 225 Affordance & feasibility 8 28 197923 Alien viewpoint 85 56 1466 Spatial reachability 221223 39 Theory of mind 40 51136 Social role & norms 2199 28 10126 Change & consequence 33 281221 Patterns & pareidolia PerceptionLatent Reasoning Object / role Facing Size / fit 2D overlap as 3D Hidden quantity Motion / sequenceIntent Missed cue Wrong viewpoint Line of sight Conse- quence Figure 3: What models get wrong, by subdomain. Share (%) of each subdomain’s failures at- tributed to each cause, over 8,573 labelled failures from 24 models on 522 tasks. 3.3AGENTIC HARNESSES To test whether active tool use aids performance on HSS, we evaluate base models inside vendor- provided agentic harnesses, including Claude Code and Codex across 388 tasks (a subset of HSS). As shown in Table 2, the agentic setup improves accuracy across all configurations, though per- formance gains and compute overhead vary widely. Claude Code provides substantial accuracy in- creases (+16.5 to +20.5 points) at high compute costs (median 57–72 steps; > 42k thinking tokens per task), whereas Codex yields more modest improvements (+4.9 to +7.5 points) with minimal overhead (14–18 steps). However, even the top-performing setup (Codex with GPT-6-astra) reaches only 59.3%, demonstrating that HSS remains far from saturated even when models are equipped with agentic capabilities. Re-labeling agentic failures using our error taxonomy reveals distinct patterns in where agent loops succeed and fail: • Active re-inspection repairs perception, not spatial depth: Total failures decrease from 944 to 759 (Figure 4), driven by a 19–37% reduction in perception and latent errors (e.g., recovering missed cues or misidentified objects). However, 2D-to-3D projection errors persist, demonstrat- ing that zooming into a 2D image cannot recover lost 3D spatial depth. • Gains concentrate in evidence-rich subdomains: Subdomains where visual evidence is explic- itly present in-frame benefit most, such as patterns & pareidolia (+30.6 points) and retrodiction (+18.9 points; Table 8). Tasks whose answer turns on depth or on an unobserved outcome gain least (e.g., spatial reachability at +4.8 points). • Agentic failure dynamics: Agent loops introduce unique failure modes, occasionally over- analyzing inputs and talking themselves out of correct initial readings. Additionally, failed agent runs take significantly more steps than successful ones (median 81 vs. 67 steps). Appendix F reports all four harness/base pairs and provides four annotated trajectories; Appendix D details the complete failure analysis for agentic experiments. Table 2: Agentic harnesses against single-pass inference. Raw is the base model called once, Agent the same model inside the harness; every ∆ is agentic minus single-pass on those tasks, re- cut by media type and domain as in Table 1. think-tok is mean thinking tokens per task inside the harness and steps the median number of agent steps. Overall pass@1∆ by media∆ by domainCost HarnessBase modelraw agent∆imagevideotemporal physical social abstractthink-tok steps Claude Code Claude-Opus-530.8 51.3 +20.5+24.3 +15.8+26.3+15.1 +18.2 +25.643,76472 Claude-Fable-5.141.2 57.7 +16.5+15.4 +17.9+16.8+14.9 +22.1 +13.242,23757 Codex GPT-6-astra 54.4 59.3 +4.9+4.5+5.5+6.7 −1.4+9.5+9.62,04014 GPT-5.6-sol 28.4 35.8+7.5+8.0+6.8+3.4+6.5+6.9+15.56,54118 7 Object / role FacingSize / fit 2D overlap as 3D Hidden quantity Motion / sequence IntentMissed cue Wrong viewpoint Line of sight Conse- quence 0 50 100 150 200 Failures 181 147 117 93 137 105 102 101 71 55 83 52 49 35 154 110 13 14 22 27 12 15 PerceptionLatent Reasoning Single callAgent Figure 4: What the agentic scaffold fixes. Pooled over the four harness/base pairs of Table 9, first attempts. The agent cuts failures from 944 to 759, mostly on causes a closer look can fix: object/role (181→ 147), facing (117→ 93), size/fit (137→ 105), and missed-cue (154→ 110) errors. Depth errors (a 2D overlap read as 3D alignment) do not fall (102→ 101), and the small reasoning causes grow. 3.4ABLATION STUDIES Effects of Reasoning Effort. We evaluate the impact of reasoning effort on model performance across HSS subdomains (Figure 5). While increased effort generally improves overall performance, accuracy in certain subdomains degrades at higher reasoning levels. For example, in the retrodiction subdomain, GPT-6-astra exhibits a 14-point drop when scaling effort from high to xhigh. Con- sequently, peak accuracy is frequently achieved at intermediate effort levels rather than the max- imum setting. These findings suggest that explicit deliberation is not the primary driver of HSS performance, which aligns with human cognitive observations where similar competencies operate intuitively prior to deliberate reasoning. Extended ablation results are provided in Appendix G. mediumhighxhighmaxmediumhighxhighmediumhigh Reasoning effort Retrodiction Mechanistic causality Extrapolation Hidden & invisible Affordance & feasibility Alien viewpoint Spatial reachability Theory of mind Social role & norms Change & consequence Patterns & pareidolia 48 64 +16 50 -14 56 +6 40 50 +10 48 -2 44 42 -2 58 64 +6 61 -3 61 +0 42 45 +3 45 +0 39 39 +0 45 47 +2 49 +2 53 +4 39 43 +4 43 +0 31 39 +8 50 52 +2 57 +5 60 +3 34 41 +7 40 -2 34 40 +5 41 44 +2 49 +5 56 +7 54 41 -12 56 +15 34 34 +0 30 40 +10 46 +6 44 -2 46 44 -2 40 -4 48 38 -10 22 26 +4 22 -4 22 +0 63 56 -7 59 +4 41 52 +11 31 37 +6 39 +2 45 +6 27 25 -2 27 +2 37 35 -2 31 46 +15 42 -3 47 +5 46 41 -5 41 +0 39 37 -2 40 43 +3 51 +9 47 -4 47 60 +13 59 -1 50 50 +0 44 44 +0 50 +6 50 +0 44 41 -3 38 -3 41 38 -3 GPT-6-astraClaude-Opus-5.5Gemini-3.8-flash Figure 5: Performance by subdomains under different reasoning effort We increase the reason- ing effort level from left to right. Green is a gain, red a loss. Effects of Video Frame Budget. Sweeping the frame budget from 10 to 500 across 234 video tasks shows that performance scales consistently with denser visual sampling. Across four evaluated models, every model achieves its lowest score at 10 frames, with GPT-6-astra gaining +17.9 points when frame budget increases from 10 to 500. Extended ablations and per-model results are provided in Appendix H. 8 Effects of different judge models. To check that the results do not depend on the choice of LLM judge, we re-graded five models with three judges from three vendors: Claude-Opus-5 (the primary judge), GPT-5.6-terra and Gemini-3.6-flash, across 2,620 items and 3,780 criteria. Rankings are preserved: judge-to-judge scores correlate at r ≥ 0.98 (Figure 6a), and while GPT-5.6-terra is stricter by 2.4 points on average, it is stricter uniformly. Over 95% of items and criteria (Figure 6b), with κ≥ 0.92 on both units. Finally, there is no self-preference (Panickssery et al., 2024): Claude- Opus-5 grading itself shifts its own score by−0.44 points. Per-model breakdowns are in Appendix I. 102030405060 Pass@1 (%), Claude Opus 5 judge 10 20 30 40 50 60 Pass@1 (%), other judge (a) GPT-5.6 Terra judgeGemini 3.6 Flash judge 93949596979899100 Agreement between judges (%) GPT-5.6 Terra vs Gemini 3.6 Flash Claude Opus 5 vs Gemini 3.6 Flash Claude Opus 5 vs GPT-5.6 Terra 95.8% 96.2% 97.3% 97.5% 96.2% 96.4% (b) CriterionItem Figure 6: Judge alignment across vendors. (a) Pass@1 scores under alternative judges closely track the primary judge (Claude-Opus-5) along the identity line. (b) Pairwise agreement exceeds 95% at both the criterion (n = 3,780) and the item (n = 2,620) level for all three judge pairs. 4RELATED WORK Most current multi-modal benchmarks fall into two families. One wraps problem-solving with visual inputs, such as MMMU, MathVista, Humanity’s Last Exam (Yue et al., 2024; Lu et al., 2024; Phan et al., 2025; Zou et al., 2025), where the image supplies explicit givens that could in principle be transcribed into text. The other targets perceptual acuity, clutter and content tracking, such as V ∗ , MMBench, Video-MME, BLINK, Perception Test and MVBench (Wu & Xie, 2024; Liu et al., 2024; Fu et al., 2024a;b; P ̆ atr ̆ aucean et al., 2023; Li et al., 2024), a line that runs back to VQA v2 (Goyal et al., 2017). HSS is orthogonal to both: its answers are neither transcribable nor findable by looking harder, because what the task asks for was never in the frame. HSS is also related to a body of visual abduction benchmarks and to spatial, physical, social and abstract reasoning suites (Zellers et al., 2019; Hessel et al., 2022; Chen et al., 2024; Riochet et al., 2022; Xiao et al., 2021; Zadeh et al., 2019; Sap et al., 2019; Bear et al., 2021; Yi et al., 2020; Jin et al., 2024; Yang et al., 2025; Zhang et al., 2019; Kamath et al., 2023). Each covers part of this territory, typically one competence at a time and often on synthetic scenes; HSS spans physical, spatial, social and abstract inference over natural media under a single admission standard (Appendix A). Our agentic condition follows work showing that active inspection, such as cropping, zooming, re-sampling, sketching can lift visual reasoning (Roberts et al., 2025; Guo et al., 2025; Hu et al., 2024), and our grading follows LLM-as-judge practice (Zheng et al., 2023) with an explicit audit of self-preference (Panickssery et al., 2024) (Section 3.4). 5LIMITATIONS We note three major limitations. Visual channel only. HSS evaluates what a model infers from what it can see. Video items are supplied without their audio track and without a transcript, so items whose resolution would depend on speech or off-screen sound are outside the benchmark’s scope by construction. Judge dependence. Scores are mediated by an LLM judge, which is known to carry biases including self-preference (Panickssery et al., 2024). We report cross-judge agreement and an oracle condition in which human reference answers are scored correct, which bound the effect. 9 Contamination. Media sourced from the public web may appear in training data. The questions are newly authored against that media rather than collected with it, which is what the benchmark tests, but prior exposure to an individual image or clip cannot be fully excluded. 6CONCLUSION Humans deduce far more from a visual scene than what is explicitly rendered in pixels: a capabil- ity measured directly by Humanity’s Sixth Sense (HSS). To benchmark visual intuition objectively, HSS operationalizes it through a structured taxonomy and a rigorous task construction pipeline. Across our benchmarks, frontier models exhibit a stark performance gap compared to the human baseline, a discrepancy that persists despite test-time compute scaling and integration into agentic harnesses. HSS highlights intuitive visual reasoning as a fundamental, measurable axis of multi- modal intelligence where current scaling paradigms have so far yielded limited progress. AI USE STATEMENT AI assistants were used strictly to polish draft and assist in developing evaluation and analysis code. All benchmark tasks, including media selection, questions, reference answers, and grading rubrics were authored and reviewed entirely by human annotators. LLMs appear in this work solely as model baselines of benchmark study and automated evaluators. ETHICS STATEMENT Media licensing. All media is drawn from three repositories that expose per-item, machine-readable license metadata: Wikimedia Commons, Openverse and the Internet Archive. In addition, only four licenses are admitted: Public Domain, C0, C-BY and C-BY-SA. All four permit redistribution. Any item that is non-commercial, no-derivatives, or whose rights cannot be verified from source metadata is rejected at retrieval time. Creator attribution is carried per item and released with the benchmark, and the source files are mirrored so the frozen benchmark is not exposed to link rot. Furthermore, audio tracks are stripped from all video assets to ensure the benchmark evaluates purely visual reasoning, matching our task design. People in the media. Many tasks depict identifiable people in public settings. They are used for research evaluation only, and a keyword screen is applied to video candidates to avoid material primarily featuring minors. Theory of mind tasks ask models to infer emotion, intent and belief from appearance and behaviour. Such inferences are culturally situated and can encode stereotypes. The multi-reviewer consensus requirement of Section 2.3 removes tasks on which readings diverge, but agreement among a demographically narrow annotator pool is not evidence of a universal inference. We therefore read theory of mind results as measurements of agreement with annotator judgement, not as ground truth about the people depicted. Annotation. Annotators were recruited and compensated through a professional data-annotation provider at the provider’s standard contracted rates. REPRODUCIBILITY STATEMENT We will publicly release our evaluation harness, model registry, and grading prompts. The registry specifies all controlled evaluation parameters, including API endpoints, reasoning effort levels, out- put token budgets, video processing pathways, and frame caps, which are detailed in Appendix C.1 across all 24 evaluated models and three judges. Where a provider exposes no explicit parameter controls, provider defaults apply. Prompt templates are reproduced verbatim in Appendix C.6. All media assets are content-addressed and locally cached to ensure every model receives byte-identical inputs, with each run logging dataset versioning, task media metadata, and per-criterion judge ver- dicts. We omit sampling temperature overrides to retain vendor defaults across heterogeneous API endpoints. Finally, all confirmatory metrics in this paper are evaluated on a frozen, versioned release containing 522 tasks (Table 3). 10 REFERENCES Chris L. Baker, Julian Jara-Ettinger, Rebecca Saxe, and Joshua B. Tenenbaum. Rational quantitative attribution of beliefs, desires and percepts in human mentalizing. Nature Human Behaviour, 1: 0064, 2017. Peter W. Battaglia, Jessica B. Hamrick, and Joshua B. Tenenbaum. Simulation as an engine of physical scene understanding. Proceedings of the National Academy of Sciences, 110(45):18327– 18332, 2013. Daniel M. Bear, Elias Wang, Damian Mrowca, Felix J. Binder, Hsiao-Yu Fish Tung, R. T. Pramod, Cameron Holdaway, Sirui Tao, Kevin Smith, Fan-Yun Sun, Li Fei-Fei, Nancy Kanwisher, Joshua B. Tenenbaum, Daniel L. K. Yamins, and Judith E. Fan. Physion: Evaluating physical prediction from vision in humans and machines. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2021. arXiv:2106.08261. Boyuan Chen, Zhuo Xu, Sean Kirmani, Brian Ichter, Danny Driess, Pete Florence, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. SpatialVLM: Endowing vision-language models with spatial rea- soning capabilities. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024. Liang Chen, Weichu Xie, Yiyan Liang, Hongfeng He, Hans Zhao, Zhibo Yang, Zhiqi Huang, Haon- ing Wu, Haoyu Lu, Yiping Bao, et al. Babyvision: Visual reasoning beyond language. arXiv preprint arXiv:2601.06521, 2026. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Caifeng Shan, Ran He, and Xing Sun. Video-MME: The first-ever comprehensive evaluation benchmark of multi-modal LLMs in video analysis. arXiv preprint arXiv:2405.21075, 2024a. Chaoyou Fu, Haozhi Yuan, Yuhao Dong, Yi-Fan Zhang, Yunhang Shen, Xiaoxing Hu, Xueying Li, Jinsen Su, Chengwu Long, Xiaoyao Xie, et al. Video-mme-v2: Towards the next stage in benchmarks for comprehensive video understanding. arXiv preprint arXiv:2604.05015, 2026. Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A. Smith, Wei-Chiu Ma, and Ranjay Krishna.BLINK: Multimodal large language models can see but not perceive.In European Conference on Computer Vision (ECCV), 2024b. arXiv:2404.12390. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. arXiv:1612.00837. Xingang Guo, Utkarsh Tyagi, Advait Gosai, Paula Vergara, Jayeon Park, Ernesto Gabriel Hern ́ andez Montoya, Chen Bo Calvin Zhang, Bin Hu, Yunzhong He, Bing Liu, and Sharma Srinivasa Rak- shith. Beyond seeing: Evaluating multimodal LLMs on tool-enabled image perception, transfor- mation, and reasoning. arXiv preprint arXiv:2510.12712, 2025. Jack Hessel, Jena D. Hwang, Jae Sung Park, Rowan Zellers, Chandra Bhagavatula, Anna Rohrbach, Kate Saenko, and Yejin Choi. The abduction of Sherlock Holmes: A dataset for visual abductive reasoning. In European Conference on Computer Vision (ECCV), 2022. Yushi Hu, Weijia Shi, Xingyu Fu, Dan Roth, Mari Ostendorf, Luke Zettlemoyer, Noah A. Smith, and Ranjay Krishna. Visual sketchpad: Sketching as a visual chain of thought for multimodal language models. In Advances in Neural Information Processing Systems (NeurIPS), 2024. arXiv:2406.09403. 11 Chuanyang Jin, Yutong Wu, Jing Cao, Jiannan Xiang, Yen-Ling Kuo, Zhiting Hu, Tomer Ullman, Antonio Torralba, Joshua B. Tenenbaum, and Tianmin Shu. MMToM-QA: Multimodal theory of mind question answering. In Proceedings of the Annual Meeting of the Association for Computa- tional Linguistics (ACL), 2024. arXiv:2401.08743. Daniel Kahneman and Gary Klein. Conditions for intuitive expertise: A failure to disagree. Ameri- can Psychologist, 64(6):515–526, 2009. Amita Kamath, Jack Hessel, and Kai-Wei Chang. What’s “up” with vision-language models? in- vestigating their struggle with spatial reasoning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023. arXiv:2310.19785. Steven Kotler, Michael Mannino, Karl Friston, Gy ̈ orgy Buzs ́ aki, J. A. Scott Kelso, and Guillaume Dumas. Pathfinding: a neurodynamical account of intuition. Communications Biology, 8:1214, 2025. doi:10.1038/s42003-025-08612-9. Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. MVBench: A comprehensive multi-modal video under- standing benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), 2024. arXiv:2311.17005. Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. MMBench: Is your multi- modal model an all-around player? In European Conference on Computer Vision (ECCV), 2024. arXiv:2307.06281. Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. MathVista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representations (ICLR), 2024. Arjun Panickssery, Samuel R. Bowman, and Shi Feng. LLM evaluators recognize and favor their own generations. arXiv preprint arXiv:2404.13076, 2024. Jae Sung Park, Chandra Bhagavatula, Roozbeh Mottaghi, Ali Farhadi, and Yejin Choi. Visual- COMET: Reasoning about the dynamic context of a still image. In European Conference on Computer Vision (ECCV), 2020. Viorica P ̆ atr ̆ aucean, Lucas Smaira, Ankush Gupta, Adri ` a Recasens, Larisa Markeeva, et al. Percep- tion test: A diagnostic benchmark for multimodal video models. In Advances in Neural Informa- tion Processing Systems (NeurIPS), 2023. Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, et al. Humanity’s last exam. arXiv preprint arXiv:2501.14249, 2025. Philip and Hemang. SimpleBench: The text benchmark in which unspecialized human performance exceeds that of current frontier models. https://simple-bench.com, 2024. Accessed 2026-09-19. Ronan Riochet, Mario Ynocente Castro, Mathieu Bernard, Adam Lerer, Rob Fergus, V ́ eronique Izard, and Emmanuel Dupoux. IntPhys 2019: A benchmark for visual intuitive physics under- standing. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(9):5016–5025, 2022. Jonathan Roberts, Mohammad Reza Taesiri, Ansh Sharma, Akash Gupta, Samuel Roberts, et al. ZeroBench: An impossible visual benchmark for contemporary large multimodal models. arXiv preprint arXiv:2502.09696, 2025. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. SocialIQa: Com- monsense reasoning about social interactions. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2019. arXiv:1904.09728. Elizabeth S. Spelke and Katherine D. Kinzler. Core knowledge. Developmental Science, 10(1): 89–96, 2007. 12 Simon Thorpe, Denis Fize, and Catherine Marlot. Speed of processing in the human visual system. Nature, 381:520–522, 1996. Penghao Wu and Saining Xie. V*: Guided visual search as a core mechanism in multimodal LLMs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), 2024. arXiv:2312.14135. Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. NExT-QA: Next phase of question- answering to explaining temporal actions. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021. Jihan Yang, Shusheng Yang, Anjali W. Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember and recall spaces. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2025. Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B. Tenenbaum. CLEVRER: Collision events for video representation and reasoning. In International Conference on Learning Representations (ICLR), 2020. arXiv:1910.01442. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, et al. MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for ex- pert AGI. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), 2024. arXiv:2311.16502. Amir Zadeh, Michael Chan, Paul Pu Liang, Edmund Tong, and Louis-Philippe Morency. Social- IQ: A question answering benchmark for artificial social intelligence. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019. Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Vi- sual commonsense reasoning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019. Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu. RAVEN: A dataset for relational and analogical visual reasoning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2023. arXiv:2306.05685. Chengke Zou, Xingang Guo, Rui Yang, Junyu Zhang, Bin Hu, and Huan Zhang. DynaMath: A dynamic visual benchmark for evaluating mathematical reasoning robustness of vision language models. In International Conference on Learning Representations (ICLR), 2025. 13 AEXTENDED RELATED WORK This section expands Section 4 with the comparisons that place HSS against individual benchmarks rather than against families of them. Academic problem-solving in a visual wrapper. MMMU, MathVista, DynaMath and Humanity’s Last Exam (Yue et al., 2024; Lu et al., 2024; Phan et al., 2025; Zou et al., 2025) test high-level symbolic or domain-specific reasoning over images: a diagram, a plot or a figure supplies givens, and the difficulty lives in the chain of reasoning applied to them. The distinguishing property is that a sufficiently complete verbal description of the image would preserve the task. HSS items do not have this property by measuring the intuitive visual reasoning that does not require deliberate step-by-step derivation. Perception-centred benchmarks. V ∗ , MMBench, Video-MME, and Video-MME-v2 (Wu & Xie, 2024; Liu et al., 2024; Fu et al., 2024a; 2026) test visual acuity, robustness to clutter, and tracking of content across duration. Here the answer is in the frame, and the question is whether the model can find it. HSS inverts the admission criterion (Section 2.3): an item is rejected precisely when a careful enough look would settle it. The two benchmark families are therefore complementary. Visual abduction and commonsense. VCR, VisualCOMET and Sherlock (Zellers et al., 2019; Park et al., 2020; Hessel et al., 2022) are the closest ancestors: each asks the model to go beyond the depicted scene to intentions, prior events or likely continuations. They concentrate on social and narrative context and grade plausible completions, often against multiple-choice alternatives. HSS covers physical, spatial, social and abstract inference under one taxonomy, and admits only items whose answer reviewers agree is forced rather than merely plausible, which is what makes open-ended rubric grading possible. Spatial, physical, and social reasoning. SpatialVLM, IntPhys, Physion and CLEVRER (Chen et al., 2024; Riochet et al., 2022; Bear et al., 2021; Yi et al., 2020) evaluate relation-naming, physical plausibility and collision prediction, largely over synthetic or simulated scenes; NExT-QA, Social- IQ, SocialIQa and MMToM-QA (Xiao et al., 2021; Zadeh et al., 2019; Sap et al., 2019; Jin et al., 2024) evaluate causal, social and mental-state understanding of observed events; VSI-Bench and What’s Up (Yang et al., 2025; Kamath et al., 2023) probe spatial memory and relational language; and RAVEN (Zhang et al., 2019) isolates abstract visual analogy. HSS extends both directions to natural media and to inferences about what is not observed: an alien viewpoint the camera never occupies, a rope whose tension is visible only in its sag, an event that the clip sets up but never shows. Tool-augmented inspection. ZeroBench and VisualToolBench (Roberts et al., 2025; Guo et al., 2025) show that letting a model crop, zoom and re-sample improves visual reasoning, and motivate our agentic condition. Section 3.3 uses that lever diagnostically rather than as a method contribution: if HSS failures were failures of looking, active inspection should repair them, and the extent to which it does not (Figure 4) is evidence about what the benchmark measures. Automated grading and its validity. Open-ended grading at this scale requires an LLM judge (Zheng et al., 2023), which imports known failure modes, self-preference (Panickssery et al., 2024) foremost among them. We audit rather than assume: three judges from three vendors, agreement and κ per pair, and a self-grading contrast (Section 3.4, Appendix I). BBENCHMARK DETAILS This appendix documents the released benchmark: what it contains (Appendix B.1), how each task was built and reviewed (Appendix B.2), and how the taxonomy was arrived at (Appendix B.3). B.1COMPOSITION AND STATISTICS Four properties of the composition in Table 3 matter for the analysis; Figure 7 shows the same corpus by scene type and clip length. First, no domain dominates the benchmark: the largest, physical & spatial logic, holds 176/522 tasks and the smallest, abstract & contextual inference, holds 102, so every domain-level cell is large enough to carry a claim, and the smallest subdomain still holds 27. Second, the media split is near-even (288 images against 234 video clips, 17.6 hours 14 (c) %-"+#/%"!+)&*.&)$". + &( *!"-./*!&*$ ./- / +*/"1/0( "),+-( 0.( %2.& ( ,/&( * * * * "+,(".+ &( +-' -#/)"!& /0-"+0/!++-. /-""//-*.,+-/ +)"-"/&( %-"+#/%"!+)&*.&)$". + &( *!"-./*!&*$ ./- / +*/"1/0( "),+-( 0.( %2.& ( ,/&( * * * * "+,(".+ &( +-' -#/)"!& /0-"+0/!++-. /-""//-*.,+-/ +)"-"/&( %-"+#/%"!+)&*.&)$". + &( *!"-./*!&*$ ./- / +*/"1/0( "),+-( 0.( %2.& ( ,/&( * * * * "+,(".+ &( +-' -#/)"!& /0-"+0/!++-. /-""//-*.,+-/ +)"-"/&( (a) %-"+#/%"!+)&*.&)$". + &( *!"-./*!&*$ ./- / +*/"1/0( "),+-( 0.( %2.& ( ,/&( * * * * "+,(".+ &( +-' -#/)"!& /0-"+0/!++-. /-""//-*.,+-/ +)"-"/&( (b) Figure 7: Media statistics in HSS. (a) Hierarchical breakdown of image and video assets across broad visual categories (People & social, Work, craft & media, Nature & outdoors, Street & trans- port, and Home & retail) and fine-grained sub-scenes. (b) Distribution of visual scene types across the four core reasoning domains, showing diverse environment coverage per domain. (c) Histogram of video clip durations (log scale), spanning a few seconds to 28 minutes; summary statistics are in Table 3. of video in total), but it is deliberately uneven within domains, because the domains do not need the same evidence: theory of mind is almost entirely video (46/51), since an intention reveals itself over time, whereas the spatial subdomains lean on images, where a single viewpoint is what makes the inference hard. Third, the number of criteria per task varies, and 71.5% of tasks carry exactly one; on those, pass@1 and rubric score coincide by construction, which we control for in Appendix C.4. Fourth, video length varies by an order of magnitude across subdomains, so the frame-budget question of Section C.2 binds very differently in different cells. B.2CONSTRUCTION PIPELINE Figure 8 diagrams the authoring and review pipeline of Section 2.4 end to end, with the number of tasks surviving each round. We include it because the attrition is itself a description of the bench- mark: the three principles of Section 2.3 are cheap to state and expensive to enforce, and the cost shows up as the 85% of authored tasks that do not reach the release. B.3TAXONOMY DESIGN Table 4 gives the taxonomy in full: each domain with the kind of recovery it demands, and each subdomain with the inferential step that defines it. This section says where the categories came from and how an item is assigned to one. Four main domains partition. Temporal & causal dynamics withholds another moment in time: the scene is one slice, and the answer lies behind it, ahead of it, or in the mechanism carrying one to the other. Physical & spatial logic withholds structure: what sits outside the frame, what an object weighs or how it would move, how the layout resolves from a viewpoint the camera never occupied. Social understanding withholds mental and normative state: intent, belief, role, the rule everyone in the scene is following. Abstract & contextual inference withholds the rule itself: the convention, pattern or consequence that makes the arrangement mean something. The split follows the lines along which cognitive science studies these intuitive inferences separately: core knowledge of objects and space (Spelke & Kinzler, 2007), intuitive physics (Battaglia et al., 2013), and mentalizing (Baker et al., 2017). 15 Table 3: HSS dataset composition and statistics. The released benchmark: 522 tasks over four domains and eleven subdomains, graded against 723 rubric criteria (1.39 per task; 71.5% carry exactly one). Each task is paired with a unique image or video clip, so task counts are also media counts. The video half totals 17.6 hours: median 76 s, interquartile range 20–307 s, longest 28 min. Word counts are means over whitespace-delimited tokens of the prompt, the full task rubric, and the reference answer. TasksMean length (words) Domain / subdomainAll Image Video Criteria Prompt Rubric Answer Physical & spatial logic1761146224837.626.973.9 Hidden & invisible5828308327.721.869.7 Spatial “alien viewpoint”5040107242.030.878.9 Affordance & feasibility413476045.430.077.8 Spatial reachability2712153339.025.868.1 Temporal & causal dynamics134815317333.222.264.4 Extrapolation5131206337.122.263.3 Retrodiction5040106433.321.373.0 Mechanistic causality3310234627.223.653.0 Social understanding110288216329.728.264.3 Social role, norm & power5923369329.127.767.1 Theory of mind515467030.328.761.0 Abstract & contextual inference 102653713937.823.464.0 Change & consequence7047238438.121.965.4 Patterns & pareidolia3218145537.226.861.1 All of HSS52228823472334.925.367.5 Task authoring 3,466 tasks authored 1 Select image or video 2 Assign domain & subdomain 3 Write prompt, answer, rubric Review 1 pass / edit / drop 1,164 pass Review 2 pass / edit / drop 559 pass Review 3 pass / edit / drop 522 pass HSS benchmark 522 tasks sent back for revision, then re-reviewed dropped 2,302 dropped 605 dropped 37 passed reviewrevised and re-revieweddropped Figure 8: How an HSS task is built and reviewed, with the attrition at each stage. A task is authored in three steps, then faces three independent review rounds; each round may pass it on, return it to authoring for revision (after which it re-enters the pipeline from the start), or drop it outright. Counts are net survivors: a task returned for revision and later re-admitted is counted once, at the round it finally passes, so each dropped figure is the net loss at that round. Of 3,466 authored tasks, 522 survive all three rounds, a yield of 15.1%. The first gate is by far the steepest: it turns away 2,302 tasks, two in three of everything written, overwhelmingly because the answer was already depicted in the scene, or for want of unambiguous consensus on the ground truth (principles 1 and 3 of Section 2.3). CEXPERIMENTAL SETUP This section details the experimental setup. 16 Table 4: Taxonomy Design for HSS along with descriptions. DomainDomain DescriptionSub-domainSub-domain Description Temporal & Causal Dynamics Recovering the scene’s state at a moment other than the one depicted, or the causal process connecting those moments. The media is one slice; the answer lies behind it, ahead of it, or in the mechanism driving it. RetrodictionInferring what already happened based on vi- sual traces. Mechanistic Causality Identifying the immediate physical mechanism driving an ongoing event. This is about under- standing the “how” and “why” of current mo- tion or state changes, not predicting the future. ExtrapolationPredicting the immediate kinematic trajectory or long-term systemic outcome of the current scene. Physical & Spatial Logic Recovering unshown structure of the physical scene: what exists outside view, what properties objects hold beyond their appearance, and how the layout resolves from a coordinate the camera never occupied. Hidden & InvisibleReasoning about entities or physical properties that are not directly visible in a scene. Deducing occluded, off-screen, or internal objects, non- visual attributes (mass, friction), and invisible forces. Affordance & Feasi- bility Assessing the physical compatibility between objects or between agents and their environ- ment. Spatial “Alien View- point” Mentally re-projecting the 3D scene to calcu- late line-of-sight or reachability from a different physical coordinate. Spatial ReachabilityEvaluating accessible space, paths, and physical volumes an agent or object can extend to within the 3D layout. Social Understanding Recovering the unobservable mental and normative layer: what agents believe, want, feel, attend to, and fail to know — and what the situation’s unwritten rules permit, forbid, or make awkward. Theory of MindReasoning about the internal beliefs, knowledge gaps, deceptions, and emotional states of an agent. Social Role, Norm & Power Dynamics Reasoning about unwritten social contracts, eti- quette, and cultural appropriateness in a scene. Abstract & Contextual Inference Recovering what a scene implies but never shows: the conventions and circumstances that give it meaning, and the outcomes of worlds it doesn’t depict. The answer follows from applying a rule or reading a context. Change&Conse- quence Reasoning about a change the scene doesn’t contain: given the change, inferring its conse- quence; or given the consequence, inferring the change required. Patterns & PareidoliaIdentifying structure in random data or show- ing specific hardwired bias to detect familiar patterns—especially faces and human figures— in random noise or objects. C.1MODEL CONFIGURATIONS Table 5 shows how every model in Table 1 was called. It is generated from the harness’s model registry (configs/models.yaml), which is released with the evaluation code, so a reader can reproduce any row from the endpoint and the settings given here. Several conventions apply to every row and are therefore not repeated in the table. Temperature and top-p are never set: we omit both from the request so that each provider’s own default applies. Reasoning effort is run at the highest level each deployment honours when possible. The effort listed is the one used for the main table. Max output tokens is the requested ceiling, not the observed usage; thinking tokens count against it. For video, models advertising native ingestion receive the clip as a single content block. Everything else receives uniformly sampled frames at 2 fps, each labelled with its timestamp, capped at the per- model frame limit in the last column: 500 unless a provider enforces something stricter, in which case frames are thinned uniformly with the endpoints retained, so the frames still span the whole clip. Frames are downscaled to a longest side of 768 px and encoded at a fixed JPEG quality; images are sent at native resolution with a longest side of 4096 px and are downscaled only when a payload limit requires it. Requests carry a 1200 s timeout with up to 5 retries under exponential backoff. C.2MEDIA HANDLING Native video. Models that accept a video file receive the clip directly. 17 Table 5: Model configurations for the main experiments. All 25 models of Table 1, grouped by vendor in the same order, plus the three judges of Section 3.4. Endpoint is the identifier sent to the proxy; API is the request shape (OpenAI Chat Completions, OpenAI Responses, or Anthropic Messages). Effort is the reasoning level used for the main table. Out is the requested output-token ceiling, which thinking tokens count against. Native vid. is whether the clip is sent as a video block rather than as sampled frames. Frames is the per-request frame cap that binds for that deployment; sampling is 2 fps for all models, thinned uniformly to fit the cap. VendorModelEndpointAPIEffortOut Native vid. Frames OpenAI GPT-6-astra openai/gpt-6-astraResponses max128, 000no500 GPT-6.1-sol openai/gpt-6.1-solResponses max128, 000no500 GPT-6-sol openai/gpt-6-solResponses max128, 000no500 GPT-5.6-sol openai/gpt-5.6-solResponses max128, 000no500 GPT-5.6-terra openai/gpt-5.6-terraResponses max128, 000no500 GPT-5.6-luna openai/gpt-5.6-lunaResponses max128, 000no500 GPT-6-luna openai/gpt-6-lunaResponses max128, 000no500 Anthropic Claude-Opus-5.5 anthropic/claude-opus-5-5Messages xhigh 128, 000no500 Claude-Fable-5.1 anthropic/claude-fable-5-1Messages xhigh 128, 000no500 Claude-Fable-5 anthropic/claude-fable-5Messages max128, 000no500 Claude-Opus-5 anthropic/claude-opus-5Messages max128, 000no500 Claude-Sonnet-5 anthropic/claude-sonnet-5Messages max128, 000no500 Claude-Opus-4.8 anthropic/claude-opus-4-8Messages max128, 000no500 Google Gemini-3.8-flash gemini/gemini-3.8-flashChat high65, 536yes500 Gemini-3.7-flash gemini/gemini-3.7-flashChat high65, 536yes500 Gemini-3.6-flash gemini/gemini-3.6-flashChat high16, 384yes500 Gemini-3.5-flash gemini/gemini-3.5-flashChat high16, 384yes500 Muse Muse Spark 1.3 meta ai/muse-spark-1.3Responses max128, 000yes50 Muse Spark 1.2 metaai/muse-spark-1.2Responses xhigh 128, 000yes50 Qwen Qwen3.8-Max dashscope/qwen3.8-maxChat max16, 384no250 Qwen3.8-flash dashscope/qwen3.8-flashChat max— ZhipuGLM-5.3-flash fireworksai/glm-5p3-flashChat max128, 000no60 Moonshot Kimi-K3 bedrock/us.moonshotai.kimi-k3 Chat max128, 000no100 Xiaomi MiMo-v2.6-flash mimo-v2.6-flashChat high— MiMo-v2.6-pro mimo-v2.6-proChat high— Judges judge-claude-opus-5 anthropic/claude-opus-5Chat high8, 192n/an/a judge-gpt-5.6-terra openai/gpt-5.6-terraChat high8, 192n/an/a judge-gemini-3.6-flash gemini/gemini-3.6-flashChat high8, 192n/an/a Uniform frame sampling. Otherwise, videos are sampled at a fixed rate of f = 2 fps with a hard cap of N max frames. When f · T > N max for a clip of duration T , we lower the rate to N max /T rather than truncating, so the frames always span the whole clip: f eff = min f, N max T , N = min(⌊fT⌋, N max ).(1) Each frame is labelled with its timestamp, and the prompt states the frame count, effective rate and duration, so the model can reason about elapsed time rather than mere ordering. We use N max = 500 as a single universal cap for all models, chosen as the most restrictive per-request image limit among the deployments we evaluate; where a provider imposes a stricter limit, frames are thinned uniformly (endpoints retained) and the affected samples are flagged so the reduced temporal resolution is visi- ble in the analysis rather than hidden. Images. Sent at native resolution, downscaled only when a payload limit requires it. All media is content-addressed and cached, so every model sees byte-identical input. C.3JUDGING A single LLM judge grades all runs, following standard practice for free-form evaluation (Zheng et al., 2023). The judge sees the question, the reference answer, the rubric, and the candidate answer, and returns a per-criterion verdict with a one-sentence justification. It does not see the media by default: rubrics are written to be checkable against the reference answer, which keeps the judge from re-adjudicating perception. We report the effect of changing judge model in Section 3.4. 18 Grading protocol. The judge is instructed to: mark a criterion met only if the answer actually asserts what it requires; mark a criterion unmet if the answer does not state the required content; ignore extra correct detail; and treat the reference answer as ground truth on disagreement. It returns one verdict per criterion with a short justification. Unparseable output is re-requested a bounded number of times before the sample is recorded as a grading failure rather than a zero. The system prompt and output schema are in Appendix C.6. C.4METRICS Every number reported in this paper is pass@1. Specifically, each sample i∈1,...,n carries k i atomic rubric criteria and is attempted A i times: A i = 3 for every model in Table 1, except where a provider error left fewer usable attempts, and A i = 1 in the reasoning-effort and frame-budget sweeps (Appendices G and H), where pass@1 reduces to the observed pass rate. Let v ija ∈ 0, 1 be the judge’s verdict on criterion j of sample i at attempt a. An attempt counts as correct only when every criterion is met, c ia = 1 h P k i j=1 v ija = k i i ,(2) so a response that supplies two of three required facts is not a correct answer to the question asked. Pass@1 is then the fraction of attempts that are correct, averaged over samples: pass@1 = 1 n n X i=1 1 A i A i X a=1 c ia | z per-sample pass@1 ,Rubric score = 1 n n X i=1 1 A i A i X a=1 1 k i k i X j=1 v ija .(3) Read as an estimator, the per-sample term is the probability that one sampled response solves that task, and the outer mean weights every sample equally regardless of how many attempts it has. This is the k = 1 case of the pass@k estimator used for code generation (Chen et al., 2021); because k = 1, no combinatorial correction is needed and the estimate is simply the observed success rate. C.5THE BLIND CONTROL The blind control re-runs the strongest model, GPT-6-astra at maximum effort, on all 522 tasks with the media removed and everything else held fixed (such as prompt, rubrics, and judge). It scores 6.6% (Table 1) against 53.6% sighted. Only 4 of the 522 tasks pass blind on all three attempts, while 67 pass at least once: the blind model is not solving a stable subset, it is landing on two-way questions at roughly the rate chance predicts. Below we show four examples pass on the first attempt and fail on at least one of the other two. What passes with the media removed 1. A two-way question. “Assuming the person seated in the rowboat was rowing in a constant direction, did they pass in front of or behind the boat on the left?” Blind answer, in full: “They passed behind the boat on the left.” Correct, with no stated reason, and it flips on attempts 2 and 3. 2. A yes/no question. “Without standing up, could the musician closest to the left of the director reach out and touch the director with his hand?” Blind answer: “No. The director is too far away for the seated musician to touch him without getting up.” The reference says no, so the rubric’s single criterion is met by a coin flip dressed in a justification the blind model cannot have. 3. A left/right question. “. . . from which of his sides did the puck pass through?” Blind answer: “The puck passed on his left side.” The sighted model gets this one wrong on all three attempts, which is a fair summary of how little the blind pass means. 4. A question that carries its own answer. “What recent physical process most likely created the muddy channel that the rescue workers are clearing, and what visible evidence supports that conclusion?” Blind answer: “scoured by a recent flash flood or mud-and-debris flow. . . torn-away banks with exposed roots, uprooted trees, broken logs”. The words muddy channel and rescue workers narrow the process before the image is consulted, and the model then invents evidence that happens to match. C.6PROMPTS Below we show the prompts used for HSS evaluations. 19 Media preamble — image An image is attached. Media preamble — native video A video (durations) is attached. Reason about motion and event order across time. Media preamble — sampled frames The video below is provided as numframes frames sampled uniformly at fps frames per second (video duration durations), in chronological order. Frame timestamps in seconds: timestamps. Treat the frames as one continuous video and reason about motion and event order across them. Blind control system prompt (no media attached) You are a helpful assistant. Answer the user’s question faithfully. If you do not have enough information to answer, do not guess --- say so honestly. Judge prompt.The judge receives the question, the reference answer, the numbered rubric and the candidate answer, and returns one verdict per criterion in a fixed JSON schema. It does not see the media: rubrics are written to be checkable against the reference answer, which keeps the judge from re-adjudicating perception. Judge system prompt You are a strict, impartial grader for the Humanity’s Sixth Sense (HSS) visual reasoning benchmark. You are given a question about a piece of media, a human-written reference answer that is authoritative and correct, a list of rubric criteria, and a candidate answer from a model under evaluation. Grade the candidate answer against each rubric criterion independently. Rules: 1. A criterion is met only if the candidate answer actually asserts what the criterion requires. Do not infer or give credit for near-misses, hedges, or contradictions. 2. Wording may differ from the reference; judge meaning, not phrasing. Synonyms, equivalent directions (e.g. ’right to left’ vs ’leftward’), and equivalent units count. 3. If the candidate answer states the required content but also states a contradicting alternative, the criterion is NOT met. 4. Extra correct detail neither helps nor hurts. Extra incorrect detail only matters if it contradicts a criterion. 5. Treat the reference answer as ground truth when the candidate disagrees with it. 6. Judge only the criteria given. Do not invent criteria. Return ONLY a JSON object, no prose and no markdown fences, in exactly this form: "criteria": ["id": "<criterion id>", "met": true|false, "justification": "<one short sentence citing the candidate answer>"], "overall comment": "<one sentence overall assessment>" DFAILURE ANALYSIS Taxonomy and protocol. We label the primary cause of every failed first attempt in the main table: 8,573 failures over 522 tasks and the 24 models evaluated when the analysis was run, which is 20 every row of Table 1 except the newly added GPT-6.1-sol. The eleven causes fall into three families that separate where the answer goes wrong: • Perception: where the model misreads something the scene actually shows for implicit visual reasoning. • Latent: where it misses or misreads something the scene only implies. • Reasoning: where it has the facts right and draws the wrong conclusion. A twelfth label, no committed answer, covers non-answers. Labelling is performed by Claude-Opus- 5 shown the question, the reference answer and the model’s answer. Every failure received a label. Table 6 gives the pooled distribution on failure annotation. Table 6: Primary failure causes, pooled over 24 models (n = 8,573). Share is the percentage of all failures; the last two columns split that share by media type. Perception and latent causes together account for 94% of failures. FamilyCauseCount Share Image Video Perception Misidentified an object, person, or role1, 70219.917.722.2 Misread orientation or facing1, 03112.018.25.1 Misjudged size, distance, or fit1, 04812.215.88.2 Read a 2D overlap as 3D alignment7769.114.62.9 LatentMisread an invisible quantity from its trace6347.46.98.0 Misread motion or sequence6787.93.812.4 Misread intent, feeling, or relationship3734.40.38.9 Missed the decisive cue1, 82621.315.228.1 Reasoning Converted to the wrong viewpoint1201.41.71.1 Misjudged a line of sight2192.63.71.3 Mis-traced the consequence or the question’s condition1151.32.10.5 —No committed answer510.60.11.2 The cause follows the task. Cause is strongly associated with subdomain and with media type, but almost not at all with which model produced the failure. The per-task view agrees: on the 503 tasks that five or more models fail, a median 80% of those models fail for the same cause and 90% for the same family. Figure 3 gives the full subdomain× cause matrix. The media split is the clearest single contrast. Image failures are 66% perception, dominated by misread orientation (18%), misidentification (18%), size and fit (16%) and reading a 2D overlap as 3D alignment (15%). Video failures invert this: 57% are latent, with the missed decisive cue alone at 28%. Video does not make models worse at seeing; it gives them more to miss. What the agent loop repairs. Table 7 pairs each agentic run against its own single-call run on the same tasks, pooling the four harness/base pairs of Table 9. Every perception and latent cause falls by 19–37%, with one exception: reading a 2D overlap as 3D alignment does not move (102 → 101), and only 24% of single-call failures of that kind are repaired, the lowest fix rate of any common perception cause. Cropping and zooming recovers a missed cue or a misidentified object; it does not recover depth, because the extra looking returns more of the same projection. EFAILURE EXAMPLES In this section, we present one representative failure from each of the eleven subdomains in Figure 9 to Figure 14, shown with the media the models were served. Each case is a task that all three of GPT-6-astra, Gemini-3.8-flash and Claude-Opus-5.5 missed on every one of their three attempts. FAGENTIC HARNESSES ANALYSIS Table 9 provides all four harness/base pairs behind Table 2, paired on the same 388 tasks. All four setups improve the model performance: Claude Code lifts both of its models by 16–21 points at 12– 21 Hidden & invisible Prompt Which way is the wind blowing, from the striped flag? All three models answered From right to left across the frame. Reference Toward the camera. Spatial “alien viewpoint” Prompt If the child in the gold jacket turned 90° right, would they face the woman? All three models answered They would face her left side. Reference Not at all; they would have to turn around. Figure 9: One representative failure per subdomain (1 of 6): the two subdomains of physical & spatial logic that turn on what lies outside the frame: hidden & invisible, and the alien viewpoint. Every task shown was missed by GPT-6-astra, Gemini-3.8-flash and Claude-Opus-5.5 on all three attempts; the model line is the judge’s condensed reading of what all three said. All eleven cases are image tasks, so each card shows exactly the media the models were served. 22 Affordance & feasibility Prompt Would two more white books fit on this shelf? All three models answered No, there is too little space. Reference Yes, in the gaps already there. Spatial reachability Prompt If the blonde woman at centre stage extended her right arm straight out at shoulder height, would her hand reach the dark-haired woman on the left? All three models answered Yes, her hand would make contact; no step needed. Reference No; there is a clear gap, so she would have to step closer. Figure 10: One representative failure per subdomain (2 of 6): the two that turn on fit and reach: affordance & feasibility, and spatial reachability. Selection and panel layout as in Figure 9. 23 Extrapolation Prompt The green truck reverses straight back. Who is in most danger? All three models answered A man in a red vest on the left. Reference The man in green hi-vis, directly behind. Retrodiction Prompt From which side of the frame did the traveller with the two dogs enter? All three models answered The lower-left side. Reference The bottom edge, tracing the path back. Figure 11: One representative failure per subdomain (3 of 6): the forward and backward temporal subdomains: extrapolation and retrodiction. Selection and panel layout as in Figure 9. 24 Mechanistic causality Prompt Are these chains bearing the plough's drag weight? All three models answered No, they hang slack. Reference Yes, they are taut and loaded. Theory of mind Prompt Who in the room seems to be paying the least attention to the meeting? All three models answered The man in the black polo against the back wall, or the man in glasses at front left. Reference The blonde-haired woman behind the man in the blue shirt, looking down rather than at the discussion. Figure 12: One representative failure per subdomain (4 of 6): mechanistic causality and theory of mind. Selection and panel layout as in Figure 9. 25 Social role, norm & power Prompt Are the people around the aircraft visitors, or crew preparing it? All three models answered Visitors examining it. Reference Crew, from the open access panels. Change & consequence Prompt If the shaft spins, does the cage around it spin too? All three models answered Yes, it turns with the shaft. Reference No; it is bolted to the engine casing. Figure 13: One representative failure per subdomain (5 of 6): social role, norm & power, and change & consequence. Selection and panel layout as in Figure 9. 26 Patterns & pareidolia Prompt Whose direction of gaze breaks the pattern set by the rest? All three models answered The person in front with a pom-pom hat. Reference The man in the black beanie, centre, looking right. Figure 14: One representative failure per subdomain (6 of 6): patterns & pareidolia. Selection and panel layout as in Figure 9. 27 Table 7: How the agent loop shifts the failure distribution, four harness/base pairs pooled on 387 paired tasks, first attempts. Fixed is the percentage of single-call failures with that cause that the harness repairs. FamilyCauseSingle Agentic ∆ % Fixed % Perception Misidentified an object, person, or role181147 −1935 Perception Misread orientation or facing11793 −2140 Perception Misjudged size, distance, or fit137105 −2339 Perception Read a 2D overlap as 3D alignment102101 −124 LatentMisread an invisible quantity from its trace7155 −2337 LatentMisread motion or sequence8352 −3733 LatentMisread intent, feeling, or relationship4935 −2931 LatentMissed the decisive cue154110 −2934 Reasoning Converted to the wrong viewpoint1314+831 Reasoning Misjudged a line of sight2227 +239 Reasoning Mis-traced the consequence or the question’s condition1215 +2525 —No committed answer35 +6733 Table 8: Where the agent loop pays, by subdomain. Pass@1 (%) for the four harness/base pairs pooled on the 387 paired tasks of Table 7, first attempts. n counts pair× task observations, so each subdomain appears four times. Re-inspection helps most where the evidence is in the frame and least where the answer turns on depth or on an outcome the media never shows. Subdomainn Single Agentic∆ Patterns & pareidolia6833.864.7 +30.9 Retrodiction15234.955.3 +20.4 Theory of mind13632.448.5 +16.2 Social role, norm & power 16832.147.6 +15.5 Affordance & feasibility14043.655.0 +11.4 Change & consequence22440.649.1+8.5 Hidden & invisible18040.648.9+8.3 Mechanistic causality8447.656.0+8.3 Spatial “alien viewpoint”15242.150.0+7.9 Extrapolation16045.052.5+7.5 Spatial reachability8434.539.3+4.8 14× the base model’s thinking tokens, while Codex lifts both of its by 5–8 at 1–2× of base model’s thinking tokens. Table 9: All agentic harness/base pairs, paired on 388 tasks. Blank counts tasks where the harness returned no answer, scored as zeros; p is two-sided McNemar. HarnessBase modelRaw Agent∆ Win LosspThink Blank Claude Code Claude-Opus-530.851.3 +20.511230 < 10 −4 43,7640 Claude Code Claude-Fable-5.1 41.257.7 +16.59630 < 10 −4 42,2371 CodexGPT-5.6-sol28.435.8+7.563330.0036,5414 CodexGPT-6-astra54.459.3+4.953280.0072,0400 Trajectories. Based on agents’ trajectories, failures run longer than successes: Claude Code takes a median 81 steps on tasks it gets wrong against 67 on tasks it gets right around Opus-5 and 65 against 54 around Fable-5.1. Codex, which searches least, shows no such gap. F.1FOUR TRAJECTORIES Below, we present four representative agentic runs: two where the agentic workflow turns a single- pass failure into a success (repaired), and two where the reverse occurs, a single-pass success result- ing in an agentic failure (broken). 28 Repaired: “Which team likely won this game of rugby? Why?” Claude Code + Claude-Opus-5, social role & power, video (30 s), 121 steps. Single call: fail (all three attempts). Agent: pass. Reference. The team in white won: after the whistle several white players raise both arms above their heads, while the black-and-red side stands with arms down and shoulders slumped. Single call. “The home side in the dark red-and-black jerseys almost certainly won” — reading a kneeling white player as “the classic posture of a beaten player”. Agent. “The away team — the side playing in all white — almost certainly won.” Trace. (3) probes duration and resolution; (4–8) extracts one frame per second and tiles 36 of them into a contact sheet; (10–14) opens individual frames hunting the celebration; (16–18) crops a 240× 110 region around two players and upscales it to 1920 px; (20) rebuilds the contact sheet at 2 fps because 1 fps was too coarse; (24–32) zooms progressively from 900× 450 down to 420× 220 on the first 3.5 seconds after the whistle; (47) writes a preliminary answer already naming the white team, then keeps working for 70 more steps; (50) re-samples the first 1.6 s at 8 fps; (63, 86, 106) three attempts to read the stadium scoreboard, all defeated by LED overexposure; (90) runs scene-cut detection, gets an empty list, and so establishes that the clip is one continuous shot and really is the final whistle; (99–104) re-extracts with a contrast boost to trace the goal line and shows the red-and-black side standing in its own in-goal; (113) “I have enough evidence to write a much stronger answer now”; (120) edits its own answer to soften “the scoreboard reads LOU” to “appears to read” and deletes a hallucinated advertiser name. Why it worked. The cue was in the frame but brief. Re-sampling the same three seconds at 1, 2 and 8 fps and zooming five-fold on the celebrating cluster turned a glance into a measurement, and the pitch-line and scene-cut checks supplied corroboration the single call never sought. Broken: “If the lightbulb hanging above the bald-headed man with red glasses were to fall, where would it end up relative to his body?” Claude Code + Claude-Opus-5, change & consequence, image, 36 steps. Single call: pass (all three attempts). Agent: fail. Reference. Behind him. The bulb is not lighting his face — no shadow under the chin, only a sheen on the top of his head — so it hangs behind his head from the camera’s viewpoint. Single call. “The fixture hangs from the ceiling well behind him. . . it would land just behind his right shoulder rather than squarely on it.” Agent. “It would land on his head — specifically on the front-left of his shaved head. . . It would not land behind him.” Trace. (3–4) reads the image and its dimensions; (8) writes a preliminary answer; (10–19) crops the bulb at 300%, the head at 400% and the glasses at 500% and reads each back; (24) dumps raw grayscale pixel rows across the support pole at five heights to test whether the camera is rolled; (26) does the same across the bulb and computes an intensity-weighted centroid at x ≈ 449, then scans rows y = 150–240 for the left edge of the head; (28) measures the skull’s apex and right edge; (30) “I have precise measurements now”; (31) draws a red plumb line at x = 449 down the whole image and crops it for inspection; (34) “The overlay confirms the plumb line”; (35) writes the final answer — “I can just read off what that line passes through” — and adds a section titled One caveat about depth: “If it hangs, say, half a metre behind him, it would fall past the back of his head”, which it then overrules on the grounds that his forehead “is clearly lit from above-front-left by this light”. Why it broke. The tools answered the wrong question precisely. A 2D overlap was converted into a 3D plumb line to sub-pixel accuracy, and the arithmetic gave the agent enough confidence to overrule the depth objection it had itself raised — and to misread the very lighting evidence that settles the task. This is the failure mode of Table 7 in miniature: reading a 2D overlap as 3D alignment is the one cause the agent loop does not reduce. Repaired: “. . . he approaches a couple. What does he notice that causes him to approach them?” Codex + GPT-6-astra, theory of mind, video (20 min at 320× 240), 21 steps. Single call: fail. Agent: pass. Reference. He notices the woman drop her handkerchief, and approaches to retrieve it and return it. Single call. “He notices the woman peering at him through her opera glasses.” Agent. “He notices that the woman. . . has dropped her handkerchief over the edge of their theater box.” Trace. (5) probes the file; (6) builds one 5× 6 contact sheet covering the first 300 s at one frame per 10 s, 29 with the timestamp burned into each tile; (7) reads the sheet and locates the two-chair act near t ≈ 205 s; (8–9) a re-extraction fails and the image read errors out; (10) debugs its own ffmpeg invocation, moving the duration flag and switching to an absolute burned-in clock; (12–14) zooms t = 215–223 at 2 fps and takes one 960× 720 close-up at t = 250.8 s; (15–16) “He notices the woman drop her handkerchief” and writes the answer; (17–21) a verification pass over t = 216–246 at 2 fps confirms it. Why it worked. Timestamped contact sheets localised a twenty-minute film to a ten-second window, and a three-fold upscale of a 320× 240 source resolved an object a few pixels wide — the detail the single call replaced with a plausible story. Broken: “For what reason does the woman with darker sleeves and a dark apron take objects from the other workers?” Codex + GPT-6-astra, theory of mind, video (148 s), 27 steps. Single call: pass. Agent: fail (rubric 0.5: the job title is right, the evidence is not). Reference. A quality inspection: she takes each piece, examines it closely, and returns it to the worker. Single call. “She takes them briefly, holds them up and turns them to examine them, then hands them back and moves to another worker.” Agent. “She is acting as an inspector or supervisor, checking that the pieces have been made correctly” — describing her as taking a piece and moving on, never noting the hand-back. Trace. (6–8) overview sheet at one frame per 10 s; (9–11) re-samples t = 8–63 at one frame per 3 s plus a full-resolution frame; (12–15) two more sheets covering the hand-off window and the tail of the clip; (16) writes a first answer — “inspecting and gathering their output” — and in the same command builds a 40-tile sheet of t = 25–45 at 2 fps cropped to her work area; (19–22) crops the hand-off at 1 fps; (23– 25) the densest look of the run, t = 52–55 at 4 fps upscaled two-fold; (26) revises the answer, changing “inspecting and gathering their output” to “checking that the pieces have been made correctly”. Why it broke. It sampled the right seconds at the right rate and still narrated only half the exchange. Its one revision tightened the job title rather than re-examining the action, so the return — the evidence the rubric asks for — never entered the answer. More looking does not help when the agent has already decided what it is looking at. GREASONING EFFORT IN FULL Table 10 gives per-subdomain accuracy at every reasoning level, the numbers behind Figure 5. At the domain level, moving from the lowest to the highest setting gives GPT-6-astra +8.2 (temporal), +12.5 (physical), +15.5 (social) and +7.8 (abstract); Claude-Opus-5.5 +6.7, +9.1, +0.9 and +4.9; and Gemini-3.8-flash +0.7, +5.7, +3.6 and−1.0. Seven of the 33 model× subdomain cells end no better than they started, and three of Gemini-3.8-flash’s eleven end lower. In only five of the twelve model× domain curves is the best score at the highest available setting. HVIDEO FRAME BUDGET IN FULL Table 11 gives the headline video-frame sweep, which runs over the 234 video tasks of HSS; frames are sampled at 2 fps and thinned uniformly to the cap. Every budget spans the whole clip and the comparison is one of temporal resolution rather than coverage. The 500-frame column is the configuration used everywhere else in the paper, scored here on the first attempt alone, which is why it sits a few tenths from the three-attempt video column of Table 1. Three things are worth separating. First, densen sampling helps. Every model improves from 10 to 500 frames. Second, it saturates. The 10 → 100 step is worth +11.5, +6.0, +2.1 and +9.8 points to the four models; the 100→ 500 step, a five-fold further increase in tokens, is worth +6.4, +2.1, +3.8 and −1.3. Third, at this sample size most of it is not resolved: the per-cell interval is about ±6 points and only GPT-6-astra’s end-to-end gain has non-overlapping intervals at its endpoints. We therefore read the table as ruling out a large monotone gain from temporal resolution beyond roughly 100 frames, and as unable to resolve a small one. 30 Table 10: Accuracy by subdomain and reasoning effort (%), all 522 tasks. Columns are each deployment’s levels in order — five for GPT-6-astra (low–max), four for Claude-Opus-5.5, three for Gemini-3.8-flash — and ∆ is last minus first. SubdomainReasoning level, lowest→ highest∆ GPT-6-astra Hidden & Invisible48 50 52 5760 +12 Affordance & Feasibility39 41 44 4956 +17 Spatial “Alien Viewpoint”28 30 40 4644 +16 Spatial Reachability22 22 26 2222+0 Extrapolation41 45 47 4953 +12 Retrodiction50 48 64 5056+6 Mechanistic Causality55 58 64 6161+6 Theory of Mind29 31 37 3945 +16 Social Role, Norm & Power Dynamics 32 31 46 4247 +15 Change & Consequence39 40 43 5147+9 Patterns & Pareidolia44 44 44 5050+6 Claude-Opus-5.5 Hidden & Invisible24 34 41 40+16 Affordance & Feasibility41 54 41 56+15 Spatial “Alien Viewpoint”38 46 44 40+2 Spatial Reachability59 63 56 59+0 Extrapolation35 39 43 43+8 Retrodiction40 40 50 48+8 Mechanistic Causality42 42 45 45+3 Theory of Mind25 27 25 27+2 Social Role, Norm & Power Dynamics 41 46 41 41+0 Change & Consequence51 47 60 59+7 Patterns & Pareidolia38 44 41 38+0 Gemini-3.8-flash Hidden & Invisible33 34 40+7 Affordance & Feasibility32 34 34+2 Spatial “Alien Viewpoint”34 48 38+4 Spatial Reachability41 41 52+11 Extrapolation33 31 39+6 Retrodiction44 44 42−2 Mechanistic Causality42 39 39−3 Theory of Mind33 37 35+2 Social Role, Norm & Power Dynamics 32 39 37+5 Change & Consequence49 50 50+1 Patterns & Pareidolia44 41 38−6 Table 11: Video frame budget. Pass@1 (%) on the 234 video tasks at each per-request frame cap, sampled at 2 fps and thinned uniformly to fit. ∆ is 500 minus 10 frames; best is the budget at which each model peaks. Model1050100200500∆ best GPT-6-astra33.8 41.9 45.3 47.0 51.7 +17.9 500 Claude-Opus-5.5 30.3 32.1 32.5 37.2 36.3+6.0 200 GPT-6-sol23.1 27.8 29.1 26.1 31.2+8.1 500 Claude-Opus-518.8 19.7 28.6 27.4 27.4+8.5 100 IJUDGE ALIGNMENT IN FULL This section expands the summary in Section 3.4. Five models spanning the leaderboard, including GPT-6-astra, Gemini-3.8-flash, Claude-Fable-5, Claude-Opus-5 and GLM-5.3-flash were re-graded end to end by three judges from three vendors, on identical rubrics, reference answers and grading prompts. Per-model scores under each judge are in Table 12. 31 Table 12: Cross-judge alignment. Pass@1 (%) for the same five models under three judges from three vendors, single attempt per task. The rightmost column is the largest disagreement between any two judges about a model. Every pairwise rank correlation is ρ = 1.0: the three judges disagree about levels, never about order. Agreement statistics for the same runs are plotted in Figure 6. Judge ModelClaude-Opus-5 GPT-5.6-terra Gemini-3.6-flash Spread GPT-6-astra (max)53.850.854.23.4 Gemini-3.8-flash (high)42.740.843.52.7 Claude-Fable-5 (max)33.830.233.03.6 Claude-Opus-5 (max)31.530.031.71.7 GLM-5.3-flash (max)26.925.227.11.9 Mean over the five37.735.437.9 The judges agree, and where they differ they differ by a constant. Pairwise agreement is 95.8– 97.2% at the criterion level and 96.2–97.5% at the item level, with Cohen’s κ between 0.914 and 0.946 on both units (Figure 6b); all three judges return the same verdict on 95.2% of items. The disagreement that remains is almost entirely a level shift rather than a reordering. GPT-5.6-terra is the strict judge, scoring every one of the five models on average 2.4 points below the mean of the other two, while Claude-Opus-5 and Gemini-3.6-flash sit within 0.3 points of each other on average. Spearman rank correlation between judges is exactly 1.0 for all three pairs, so no conclusion in this paper that rests on model ordering changes under a different grader. Self-preference is not detectable at this scale. LLM judges are known to favour their own fam- ily’s outputs (Panickssery et al., 2024), which matters here because our primary judge shares a vendor with multiple evaluated models. To evaluate potential vendor bias, we first calculated each judge’s overall leniency—defined as the difference between the score it gives a model and the aver- age score given by the other two judges: • Claude-Opus-5: +1.1 points, • GPT-5.6-terra: -2.4 points, • Gemini-3.6-flash: +1.3 points. Subtracting these baseline leniencies isolates same-vendor interaction effects (residuals): • GPT-5.6-terra judging GPT-6-astra: -0.80 points, • Gemini-3.6-flash judging Gemini-3.8-flash: +0.38 points, • Claude-Opus-5 judging Claude-Fable-5: +1.09 points, • Claude-Opus-5 judging itself: -0.44 points. All four residuals are small and two have negative signs (contradicting the expectation of self- preference). We interpret this not as proof that vendor bias is entirely absent, but as finding no detectable evidence of it at the resolution HSS currently supports. 32

Instant research discovery