Paper deep dive
PRISM: A Multi-View Multi-Capability Retail Video Dataset for Embodied Vision-Language Models
Amirreza Rouhi, Parikshit Sakurikar, Satya Sai Reddy, Narsimha Menga, Anirudh Govil, Sri Harsha Chittajallu, Rajat Aggarwal, Anoop Namboodiri, Sashi Reddi
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 98%
Last extracted: 4/1/2026, 1:32:46 AM
Summary
PRISM is a 270K-sample multi-view video supervised fine-tuning (SFT) corpus designed for embodied vision-language models (VLMs) in retail environments. It utilizes a novel three-dimensional knowledge ontology covering spatial, temporal/physical, and embodied action knowledge across 20+ capability probes. The dataset integrates egocentric, exocentric, and 360° viewpoints to improve robotic reasoning, navigation, and task execution in real-world retail settings.
Entities (4)
Relation Signals (3)
PRISM → includesviewpoints → Egocentric, Exocentric, 360°
confidence 100% · The corpus captures data from egocentric, exocentric and 360° viewpoints across five supermarket locations
PRISM → structuredby → Knowledge Ontology
confidence 100% · PRISM is grounded in a novel three-dimensional knowledge ontology that spans spatial knowledge, temporal and physical knowledge, and embodied action knowledge.
PRISM → trains → Cosmos-Reason2
confidence 100% · Fine-tuning on PRISM reduces the error rate across all 20+ probes by 66.6% over the pre-trained baseline
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:A critical gap exists between the general-purpose visual understanding of state-of-the-art physical AI models and the specialized perceptual demands of structured real-world deployment environments. We present PRISM, a 270K-sample multi-view video supervised fine-tuning (SFT) corpus for embodied vision-language-models (VLMs) in real-world retail environments. PRISM is motivated by a simple observation - physical AI systems fail not because of poor visual recognition, but because they do not understand space, physical dynamics and embodied action well enough to operate reliably in the world. To this end, PRISM is grounded in a novel three-dimensional knowledge ontology that spans spatial knowledge, temporal and physical knowledge, and embodied action knowledge. It covers 20+ capability probes across four evaluation dimensions - Embodied Reasoning (ER), Common Sense (CS), Spatial Perception (SP), and Intuitive Physics (IP), and to our knowledge, PRISM is the first dataset to instantiate all three knowledge dimensions within a single real-world deployment domain. The corpus captures data from egocentric, exocentric and 360° viewpoints across five supermarket locations and includes open-ended, chain-of-thought, and multiple-choice supervision. At 4 fps, PRISM spans approximately 11.8M video frames and approximately 730M tokens, placing it among the largest domain-specific video SFT corpora. Fine-tuning on PRISM reduces the error rate across all 20+ probes by 66.6% over the pre-trained baseline, with significant gains in embodied action understanding where the accuracy improves by 36.4%. Our results suggest that ontology-structured, domain specific SFT can meaningfully strengthen embodied VLMs for real-world settings. The PRISM dataset and more details are available at this https URL
Tags
Links
- Source: https://arxiv.org/abs/2603.29281v1
- Canonical: https://arxiv.org/abs/2603.29281v1
Trouble viewing inline? Open PDF directly →
Full Text
86,774 characters extracted from source content.
Expand or collapse full text
PRISM: A Multi-View Multi-Capability Retail Video Dataset for Embodied Vision-Language Models DreamVu 1 A critical gap exists between the general-purpose visual understanding of state-of-the-art physical AI models and the specialized perceptual demands of structured real-world deployment environments. We present PRISM, a 270K-sample multi-view video supervised fine-tuning (SFT) corpus for embodied vision-language-models (VLMs) in real-world retail environments. PRISM is motivated by a simple observation - physical AI systems fail not because of poor visual recognition, but because they do not understand space, physical dynamics and embodied action well enough to operate reliably in the world. To this end, PRISM is grounded in a novel three-dimensional knowledge ontology that spans spatial knowledge, temporal and physical knowledge, and embodied action knowledge. It covers 20+ capability probes across four evaluation dimensions - Embodied Reasoning (ER), Common Sense (CS), Spatial Perception (SP), and Intuitive Physics (IP), and to our knowledge, PRISM is the first dataset to instantiate all three knowledge dimensions within a single real-world deployment domain. The corpus captures data from egocentric, exocentric and 360° viewpoints across five supermarket locations and includes open-ended, chain-of-thought, and multiple-choice supervision. At 4 fps, PRISM spans ∼11.8M video frames and∼730M tokens, placing it among the largest domain-specific video SFT corpora. Fine-tuning on PRISM reduces the error rate across all 20+ probes by 66.6% over pre-trained baseline, with significant gains in embodied action understanding where the accuracy improves by 36.4%. Our results suggest that ontology-structured, domain specific SFT can meaningfully strengthen embodied VLMs for real-world settings. The PRISM dataset and more details are available at https://dreamvu.ai/prism Date: April 1, 2026 Figure 1 PRISM captures multi-view retail video from four synchronized modalities - egocentric, exocentric, 360° panoramic and depth - and structures 270K video SFT samples across 20+ task types organized into four capability dimensions. All capabilities feed into a model-agnostic fine-tuning format compatible with any VLM or VLA, producing embodied agents for real-world retail deployment. 1 A detailed list of contributors and acknowledgments can be found in Appendix C of this paper 1 arXiv:2603.29281v1 [cs.CV] 31 Mar 2026 1 Introduction Consider a robot that navigates a grocery aisle. It must estimate whether a shelf height is within reach, predict what a partially occluded worker will do next, determine whether its current task is complete, and interpret same scene from its body-mounted camera. Each of these requires a distinct perceptual capability- spatial depth reasoning, action anticipation, task completion verification, and cross-view understanding, that current general-purpose vision-language models do not reliably provide. This is not a failure of the underlying architectures. It is a failure of the training data. Vision-language models (VLMs) are emerging as powerful backbones for embodied AI systems, enabling them to perceive, reason about, and act in physical environments through natural language inputs NVIDIA (2025b,c); Brohan et al. (2023). Recent models such as Cosmos-Reason2 NVIDIA (2025b), GR00T N1 NVIDIA (2025c), andπ 0 Black et al. (2024) demonstrate that multi-modal reasoning is increasingly essential for robotic manipulation, navigation, and task execution in unstructured real-world settings. However, a significant gap remains between the broad visual competence of these models and the specialized perceptual demands of specific deployment domains such as retail environments. Fine-tuning VLMs for embodied applications requires carefully curated instruction-tuning datasets covering diverse perceptual capabilities. Although prior datasets for general video QA Maaz et al. (2023); Fu et al. (2024), temporal reasoning Li et al. (2024b), and physical understanding various (2024) have advanced open-domain evaluation, they do not adequately capture the range of knowledge required for physical AI in structured real-world environments. In particular, there is a scarcity of large-scale datasets targeting embodied perception across multiple knowledge dimensions in such environments. In our view, effective physical AI demands mastery across three fundamental and complementary forms of knowledge: spatial knowledge–the ability to perceive scene geometry, depth, and 3D layout; temporal and physical knowledge–the ability to reason about time, causality, and the laws governing object motion; and embodied action knowledge-the ability to anticipate actions, goals, task progress, and human-environment interaction. Existing datasets address at most one of these dimensions in any given domain, leaving a critical gap for real-world deployment. Real-world retail spaces present several unique challenges: dense object arrangements, repetitive visual patterns at multiple scales, complex human-object interactions, and the need for 3D-aware spatial reasoning in cluttered aisles. These settings require models to understand not only what appears in a frame, but how actions unfold in space, how agents move through shared spaces, and how physical context constrains behavior. Virtually all existing embodied datasets are collected exclusively from egocentric (first-person, body mounted) cameras, introducing fundamental limitations that are rarely acknowledged. The narrow field-of-view of an egocentric camera captures only what the wearer attends to, leaving the broader scene structure–aisle geometry, shelf arrangements, navigation corridors, and spatial relationships between zones–entirely unobserved. Egocentric footage cannot capture other people in the scene except as fleeting peripheral appearances, rendering it unsuitable for multi-human activity recognition, social navigation reasoning, and any form of multi-actor trajectory understanding. Moreover, the constant motion of an egocentric camera introduces depth ambiguity and removes the stable reference frame for reliable 3D reasoning. In practice, real deployments must operate across both egocentric and exocentric views, yet existing datasets rarely provide the cross-view supervision needed to support such training. In this paper, we introduce PRISM, a comprehensive multi-view video supervised fine-tuning (SFT) corpus for training embodied VLMs in retail environments. PRISM is structured around a novel knowledge ontology that interconnects the three core dimensions of physical AI - space, time and embodied action - within a single deployment domain (figure 1). PRISM provides data for cross-view supervision that is captured from egocentric, exocentric and 360° viewpoints. It comprises of 270K instruction-tuning samples spanning 20+ capability probes across four evaluation dimensions - Embodied Reasoning, Common Sense, Spatial Perception, and Intuitive Physics - making it, to our knowledge, the first dataset to unify all three knowledge dimensions in a single real-world setting. PRISM is also designed for breadth of supervision. It includes multiple SFT formats, such as open-ended understanding, chain-of-thought reasoning, and multiple-choice question answering, enabling the study of what models learn and also how training format shapes embodied reasoning. At 4 fps with Qwen3VL tokenization, PRISM totals∼11.8M video frames and∼730M tokens (∼703M visual +∼27M text), placing it among the 2 Egocentric Video Exocentric Video (+ 360°) Annotation Pipeline Metadata Extraction (Gemini Robotics ER 1.5) LLM Generation (Gemini 2.5 Flash) Depth Analysis (DepthCrafter) Self-Supervised (no annotation) Embodied Reasoning (ER) Common Sense (CS) Spatial Perception (SP) Intuitive Physics (IP) ER-1: Next Subtask (25K) ego ER-5: Hand Interaction (15K) ego ER-6: Atomic Action (20K) exo ER-8: Multi-Actor (8K) exo CS-U-1: Scene VQA (12K) ego CS-R-1: Spatial CoT (12K) ego SP-1: Rel. Depth (10K) ego SP-2: 360° Layout (2K) exo IP-1: Arrow-of-Time (30K) ego+exo IP-1 CoT: Physics (48K) ego+exo IP-2: Object Permanence (8K) ego +5 more +4 more BF16 LoRA Fine-Tuning Cosmos-Reason2-2B 270K samples (ego + exo) ego task exo task Legend: Figure 2 PRISM pipeline overview. The overview of the PRISM pipeline is shown here. PRISM is built on egocentric and exocentric videos from real-world retail stores. Four annotation strategies - metadata extraction, LLM generation (Gemini 2.5 Flash), depth analysis (DepthCrafter), and self-supervised transformations - produce twenty tasks (full list of tasks is mentioned in section section 3) across four capability domains, totaling 270K instruction-tuning samples. Solid boxes denote egocentric tasks; dashed boxes denote exocentric tasks (exo). Samples from PRISM are used for finetuning Cosmos-Reason2-2B via BF16 LoRA. largest domain-specific SFT corpora for embodied VLMs. A comparison against Cosmos-Reason training data shows that 14 of the 20+ PRISM tasks are novel - 6 are completely new with no Cosmos equivalent, 8 introduce new task types or methods within Cosmos categories, three overlap conceptually but differ substantially in domain, and only two overlap completely. An overview of the PRISM pipeline is provided in figure 2 The capability probes in PRISM are constructed to reflect the diversity of reasoning required in real- world retail settings. Embodied Reasoning tasks include next-subtask prediction, goal-conditioned action reasoning, hand-interaction recognition, atomic action understanding from exocentric views, multi-actor scene understanding, social navigation reasoning, and exo-to-ego activity matching. Common-Sense and Spatial Perception tasks include scene description visual-question-answers (VQA), exocentric environment VQA, affordance reasoning, causality reasoning, depth-aware spatial reasoning, relative depth reasoning and 360° spatial layout understanding. Intuitive Physics tasks include arrow-of-time prediction from both ego and exo views, object permanence in real-world video and physics-grounded temporal reasoning. Together, these tasks provide a comprehensive training and evaluation basis for embodied perception across multiple viewpoints and knowledge types. A key design principle of PRISM is scalable construction at minimal cost. PRISM is built using a hybrid annotation strategy that combines structured metadata extraction from robot episode logs, LLM-based QA generation via Gemini DeepMind (2025), (3) physics-grounded video reasoning via Gemini Robotics ER 1.5 Gemini-Robotics-ER-1.5 (2025), depth-aware spatial analysis using DepthCrafter Hu et al. (2024), and fully self-supervised transformations requiring zero annotation. This approach yields diverse, high-quality training data for the entire dataset with minimal human effort and API usage. Beyond introducing the dataset itself, we use PRISM to conduct a systematic fine-tuning study over an embodied VLM for real-world deployment. Our results demonstrate that ontology-structured, domain-specific SFT substantially improves embodied VLM performance in retail environments. Multi-domain training consistently outperforms single-domain baselines, exocentric supervision improves cross-view understanding 3 without degrading egocentric capability, and richer reasoning chains provide substantial gains over template- based alternatives. Our experiments suggest that progress in embodied VLMs depends not only on the scale of the model, but also on how training data is structured across knowledge dimensions, viewpoints and reasoning formats. Our main contributions are summarized hereafter: Contributions. 1.We introduce PRISM, a 270K-sample multi-view video SFT corpus for embodied VLMs in retail envi- ronments, organized around a knowledge ontology spanning embodied reasoning, common sense, spatial perception, and intuitive physics. 2.We present the first systematic study of egocentric and exocentric data mixing as a curriculum dimension for embodied VLM fine-tuning, including dedicated exocentric capabilities for action understanding, spatial reasoning, and multi-actor scene analysis. We demonstrate quantifiable gains in real-world retail scene understanding for embodied actions. By unifying spatial, temporal-physical, and embodied action supervision across egocentric and exocentric views, PRISM provides a more deployment-aligned training substrate for embodied VLMs. Beyond the dataset itself, our results highlight that progress in physical AI depends not only on model scale, but also on how training data is structured across viewpoints, capabilities, and reasoning formats 2 Related Work Vision-Language Models for Embodied AI. The integration of vision and language understanding has driven rapid progress in embodied AI. Models such as RT-2 Brohan et al. (2023) demonstrated that VLMs can directly produce robotic actions, while PaLM-E Driess et al. (2023) showed that large-scale multi-modal models can serve as general-purpose embodied reasoners. More recently, NVIDIA’s Cosmos-Reason NVIDIA (2025a) and Cosmos-Reason2 NVIDIA (2025b) introduced physical AI-oriented VLMs trained on large-scale video corpora with chain-of-thought reasoning supervision, achieving strong performance on embodied reasoning benchmarks. GR00T N1 NVIDIA (2025c) further illustrated how VLM backbones can be effectively integrated into humanoid robot control pipelines. Together, these systems suggest that progress in embodied AI increasingly depends not only on perception or language in isolation, but on models that can jointly interpret scenes, actions and spatio-temporal phyiscal context. Knowledge ontology gap in existing embodied VLMs. Despite this progress, existing embodied VLM training datasets remain uneven in their coverage of the knowledge required for deployment. A close examination of the Cosmos-Reason2 training taxonomy NVIDIA (2025a,b) reveals that its coverage of PRISM’s knowledge ontology is sparse: only 2 of PRISM’s 23 capabilities are directly addressed in Cosmos’s training data, while most are either absent entirely or introduced only in substantially different forms. In particular, PRISM contributes supervisio for cross-view activity matching, exocentric action understanding, multi-actor scene reasoning, retail-specific affordance understanding and panoramic spatial reasoning, all of which are largely missing from existing physical AI corpora. This gap is not only about the task count, but also about the deployment setting. Cosmos-Reason is trained primarily on general indoor, office, lab and structured-robot scenarios whereas PRISM targets a real-world retail environment. PRISM is therefore positioned not as a replacement for general embodied pretraining, but as a targeted fine-tuning resource that meaningfully fills domain gaps. A detailed quantitative comparison is provided in table 1. Egocentric and Exocentric Video Understanding. Egocentric video understanding has grown rapidly with large-scale datasets such as Ego4D Grauman et al. (2022), Ego-Exo4D Grauman et al. (2024), and Epic- Kitchens Damen et al. (2018), enabling progress in activity recognition, hand-object interaction, anticipation, and cross-view learning. Ego-Exo4D Grauman et al. (2024) specifically highlights the complementarity of synchronized first- and third-person views for understanding human activity. However, these datasets focus on household, cooking, sports and other skilled activities rather than deployment oriented environments. PRISM focuses on a coherent deployment domain rather than broad activity coverage. It addresses the underexplored domain of retail environments and, critically, pairs dynamic egocentric camera footage with static exocentric cameras, enabling the first systematic study of ego-exo data mixing for embodied VLM fine-tuning. The nature of the exocentric perspective differs fundamentally between existing datasets and 4 Table 1 PRISM vs. Cosmos-Reason1 training data by capability domain. Cosmos-Reason1 NVIDIA (2025a) SFT counts are taken from the publicly released HuggingFace datasets. While AgiBot World includes retail scenes as one of five deployment domains (∗), its SFT samples target general manipulation skills and are not structured as retail-domain perceptual QA; no dataset contributing to Cosmos-Reason1 provides retail-targeted VLM SFT. PRISM contributes 270K retail-specific SFT pairs spanning all three domains.†Cosmos Physical Common Sense covers generic scenes; no retail or robot-deployment content is included.‡Novel tasks entirely absent from Cosmos-Reason1 training.∗AgiBot World includes a retail deployment domain (∼1 of 5 domains, supermarket shelf-picking and restocking tasks), but contributes no retail-targeted VLM SFT to Cosmos-Reason1 training. Domain Cosmos-Reason1 (base model)PRISM (ours) Source datasetsSFT samplesTasksSFT samples EmbodiedBridgeV2, AgiBot ∗ ,∼1,720KER-1–ER-9 (9 tasks)93,757 ReasoningHoloAssist, RoboVQA (general manip.; no retail SFT)incl. multi-actor ‡ , cross-view ‡ CommonPhysical CS∼1,800K † CS-U-1/2, CS-R-1–447,778 Sense +(curated video + MCQ)(general scenes)SP-1, SP-2 ‡ (360°)11,862 Spatialincl. depth-grounded, exo spatial ‡ IntuitiveArrow-of-Time,∼51KIP-1, IP-1 CoT, IP-291,000 PhysicsSpatial Puzzles,(general)IP-1 Exo+ ‡ , CoT variants ‡ Object Permanence MCQMCQ (general)∼1,800KMCQ Overlay25,000 Retail-targeted SFT (none)0All 20 tasks270K PRISM. In Ego-Exo4D and similar corpora, the exocentric camera is a participant-following setup - a directed camera tracking the primary actor to capture their skilled performance. In PRISM, we use the DreamVu (2020) omnidirectional camera that observes the full environment without privileging any single actor. This distinction is consequential for embodied deployment. Scene-oriented exocentric views simultaneously capture multiple people, global aisle layout, crowd density, spatial relationships bentyen zones, and navigation-relevant scene structure - information that is central for a robot’s situational awareness but structurally absent from other actor-following exocentric setups. In the retail domain specifically, prior work such as the MERL Shopping Dataset Singh et al. (2016) established fixed overhead-camera action localization (5 hand/arm action classes at supermarket shelves, 106 clips), and the RetailVision workshop series RetailVision Organizers (2020–2025) has driven product recognition and in-store action localization challenges across six CVPR/ICCV editions. More recently, RetailAction Mazzini et al. (2025) introduced multi-view spatio-temporal annotations of human–object interactions from 360° ceiling-mounted cameras across real-world convenience stores. However, these datasets remain purely exocentric, focus primarily on action localization or recognition, and do not provide instruction-tuning data for VLMs. Instruction-Tuning Datasets for VLMs. The success of visual instruction-tuning Liu et al. (2023) has led to a large ecosystem of multi-modal instruction datasets. LLaVA Liu et al. (2023) pioneered GPT-4-generated visual conversations, while subsequent efforts such as Video-ChatGPT Maaz et al. (2023), LLaVA-OneVision Li et al. (2024a), and InternVL Chen et al. (2024) scaled instruction data to millions of samples across diverse tasks. For video understanding specifically, VideoChat2 Li et al. (2024c) and Video-LLaVA Lin et al. (2023) introduced temporal reasoning instructions. PRISM differs from these datasets by focusing on a coherent domain and by structuring instruction data around an explicit knowledge ontology rather than open-domain data crawling. In this sense, PRISM is not only a dataset, but also a controlled corpus for studying how instruction composition shapess embodied VLM adaptation. Data-Efficient VLM Fine-Tuning. Parameter-efficient fine-tuning methods such as LoRA Hu et al. (2022) and QLoRA Dettmers et al. (2023) have made it practical to adapt large models on limited hardware. At the same time, several recent works have shown that data composition can matter as much as model architecture. VILA Lin et al. (2024), for example, emphasized the importanve of careful data mixing strategies for instruction-tuning for VLMs. Similarly, LLaVA-OneVision Li et al. (2024a) showed that multi-stage training with task-specific data mixtures outperforms naive data concatenation. Our work complements these findings by providing fine-grained analysis of domain composition, task selection, annotation quality, and data-scale effects within a single controlled deployment setting. 5 Table 2 Comparison with related embodied video datasets. PRISM is the only dataset that simultaneously covers retail deployment, all three viewpoint modalities (ego, fixed exo, 360° exo), explicit chain-of-thought supervision, and a VLM SFT-ready format.✓ = yes;× = no. DatasetScale #TasksViewpointDomain CoT SFT Ego4D Grauman et al. (2022)3,670 h5EgoGeneral× Ego-Exo4D Grauman et al. (2024)1,286 h4Ego + ExoSkilled act.× RoboVQA Sermanet et al. (2023)829K pairs6ExoOffice robot×✓ SariBench Gajo et al. (2025)100 demos3Ego (sim.)Retail (sim.)× PRISM (ours)270K samples20 Ego+Exo+360°Retail✓ Comparison with Existing Embodied Datasets. table 2 positions PRISM against the three most closely related embodied video datasets. Ego4D Grauman et al. (2022) and Ego-Exo4D Grauman et al. (2024) are the dominant benchmarks for egocentric and ego-exo video understanding, collectively spanning thousands of hours of annotated footage across general daily-life and skilled-activity domains. However, neither is designed as a VLM instruction-tuning corpus, neither provides chain-of-thought supervision, and neither targets a specific deployment domain such as retail. A deeper limitation for humanoid learning is that both datasets are organized around short, isolated activity clips - atomic skills such as pouring, cutting, or dribbling - rather than complete, long-horizon tasks in which an agent pursues a goal across a sequence of connected sub-goals while continuously moving through and reasoning about a real-world environment. Training on skill-level clips produces models that recognize gestures and actions in isolation but lack the world-knowledge required for goal-directed navigation, multi-step planning, and sequential task execution in deployment settings. PRISM is structured around complete retail episodes - full workflows from store entry through item retrieval, shelf interaction, and checkout - each annotated with goal hierarchies, sub-goal sequences, and temporal boundaries. This captures the causal and spatial reasoning chain that connects individual actions into coherent long-horizon behaviour - the kind of knowledge a humanoid robot must possess to operate autonomously in a shared public space. RoboVQA Sermanet et al. (2023) is the closest comparator on the VLM fine-tuning axis, providing 829K auto-generated QA pairs for robot manipulation in corporate office buildings; yet it is purely exocentric, single-domain, and lacks explicit reasoning chains. The closest existing retail-specific embodied benchmark is Sari Sandbox / SariBench Gajo et al. (2025), a photorealistic Unity simulation of a Philippine convenience store with 100 VR human demonstrations across 250+ interactive products. While SariBench evaluates goal-directed agent task completion (navigation, object retrieval, product comparison), it is entirely simulation-based, egocentric-only, and an evaluation benchmark rather than a training corpus - providing no VLM instruction pairs, no temporal or physics probes, and no exocentric infrastructure views. It therefore differs substantially from PRISM in both modality and purpose. In summary, prior datasets each capture a part of the problem, large-scale egocentric undestanding, ego- exo correspondence, exocentric retail action recognition or synthetic retail evaluation. PRISM ties these threads together in a single real-world training corpus. It is the first dataset to combine a domain-specific retail deployment scenario with multi-view panoramic video, ontology-structured supervision and SFT-ready instruction data for VLM adaptation. Self-Supervised Visual Pretext Tasks. Self-supervised pretext tasks have long been used to learn visual representations without manual annotation. Arrow-of-time prediction Wei et al. (2019) and jigsaw puzzle solving Doersch et al. (2015) are classic examples that encourage models to learn temporal coherence and spatial structure, respectively. While these tasks are often used for representation learning, we instead repurpose them as instruction-tuning data. In PRISM, self-supervised transformations become question-answer pairs that can be incorporated directly into the SFT pipeline, allowing phyiscally grounded pretext signals to contribuie to embodied VLM adaptation. 6 3 PRISM Dataset 3.1 Data Collection Ontology PRISM is derived from video recorded across five structurally diverse retail environments spanning distinct store layouts, lighting conditions, aisle configurations, and product category distributions. This diversity is deliberate: by capturing variation across environments rather than within a single controlled deployment, PRISM produces training data that reflects the real distribution of structured service settings rather than a single site’s idiosyncrasies. Data is captured from two complementary camera systems. Egocentric cameras. Wearable head-mounted cameras are used to capture the first-person perspective of human actor performing diverse shopping tasks in real-world retail grocery stores. These include entering and navigating the store, browsing shelves, approaching and selecting products, picking up and placing back items, carrying products, moving between aisles, and interacting with baskets, carts, shelves and checkout areas. The egocentric view captures the scene from the actor’s first-person perspective, together with the natural hand-object interactions and local decision points that arise during shopping episodes. Egocentric clips are captured using a GoPro camera and during finetuning, they are encoded at 480p, 4 fps using H.264. Exocentric cameras. Exocentric views are captured using the proprietary ALIA 360° DreamVu (2020) camera which provides multi-view and panoramic views of the scene through a multi-camera arrangement. Unlike the egocentric view, which follows the actor’s local perspective, the exocentric view captures the broader scene, including aisle structure, shelf layout, navigation corridors and the movement of multiple people within the store. These views provide stable scene-level context for understanding spatial relationships, trajectories, multi-actor activity and environment-level behaviour during shopping episodes. During finetuning, the exocentric clips are encoded at 4-fps using H.264. Together, these two modalities provide complementary information about embodied activity in retail environ- ments. Combining both views enables PRISM to support training and evaluation across fine-grained action understanding as well as scene-level spatial and behavioural reasoning. 3.2 Knowledge Ontology PRISM is organized around a novel knowledge ontology for physical AI with three complementary and fundamental dimensions: spatial knowledge, temporal and physical knowledge, and embodied action knowledge. Spatial knowledge captures the understanding of 3D scene geometry, layout and relative structure. Temporal and physical knowledge captures reasoning about causality, motion, ordering, and physical constraints. Embodied action knowledge captures understanding of actions, goals, task progress, and activity across viewpoints. We draw inspiration from NVIDIA (2025a) for the knowledge ontology, and extend it to exocentric perspective and novel knowledge dimensions. We operationalize our ontology through 20+ targeted capability probes spanning four domains - Embodied Reasoning (ER), Common Sense (CS), Spatial Perception (SP) and Intuitive Physics (IP) together with multiple-choice-question (MCQ) format training. These probes are expressed in three SFT formats: open- ended understanding, chain-of-thought reasoning with⟨think⟩tags, and MCQ answering. table 3 provides a quantitative summary and representative examples for all tasks are shown in figure 3. The capability probes are described in detail in the following section: 3.2.1 Embodied Reasoning (ER) ER-1: Next Subtask Prediction. Given a video clip showing the current activity with structured context such as the goal, the sub-goal, the scene description, and the hand states, the model is expected to predict the next subtask in the shopping workflow. This task evaluates sequential reasoning and action anticipation both of which are central for embodied task execution. Labels are derived from temporal sub-goal ordering in episode metadata. The probe contains 25,000 samples. ER-2: Task Completion Verification. This is a binary classification probe where the model is expected to determine whether a subtask is completed 7 or ongoing based on video evidence. Completion labels are automatically inferred from sub-goal transition boundaries: if a clip occurs at the end of sub-goal and transitions to the next, it is labeled completed. This probe contains 3,000 samples with a 59.3%/40.7% class split. The samples are generated with 6 question templates and 8 answer templates to improve lexical diversity. ER-3: Goal-Conditioned Action Reasoning. Given a clip together with structured context such as the goal hierarchy, environment and hand states, the model is expected to reason about why the current action is being performed in the context of the broader task goal. This evaluates causal understanding of embodied actions rather than action recognition alone which is critical for autonomous decision-making. This probe contains 8,000 samples. ER-4: Exo-to-Ego Activity Matching (exocentric). Given a third-person clip showing an actor from an exocentric view, the model is expected to identify the activity being performed and match it to structured egocentric activity descriptions. Clips are extracted from synchronized Ego+Exo video pairs, isolating the exocentric perspective. This provbe explicitly connects first-person and third-person activity understanding and evaluates whether the same underlying behaviour can be recognized across viewpoints. This probe contains 7,000 samples. ER-5: Hand Interaction Recognition. This probe is centered on model generated descriptions of hand states. The hand interactions in the video are labeled across 13 normalized categories: idle, holding, grasping, reaching, placing, using_phone, pushing_cart, carrying, pointing, scanning, arranging, picking_up, and pulling. The model is expected to label the activity in the video to the appropriate hand state. This probe contains 15,000 samples extracted in an activity-balanced manner - 50% both-hands-active, 30% one-hand-active, 20% both-hands-idle. ER-6: Atomic Action Recognition (exocentric). This probe focuses on recognizing and describing short atomic actions from exocentric trajectory annotations. Using clips extracted from exocentric cameras, the model is expected to provides dense action-level recognition from the estimated human trajectories. This probe contains 20,000 samples. ER-7: Atomic Action Reasoning (exocentric, CoT). This probe extends from ER-6 by moving from action recognition to action interpretation. Given the action description, the model is expected to reason why a observed action is being performed. This probe contains 8,000 samples with ⟨think⟩ chain-of-thought tags. ER-8: Multi-Actor Scene Understanding (exocentric). This probe is based on multi-actor recognition. The model is expected to describe what multiple people are doing simultaneously in the store from an exocentric viewpoint. This probe targets scene-level activity understanding in shared environments, where multiple actions may unfold in parallel. This probe contains 8,000 samples with at least two visible people over a minimum temporal window of 3s. ER-9: Social Navigation Reasoning (exocentric, CoT). This probe is a reasoning variant of ER-8 where the model is expected to analyze social and spatial dynamics between multiple actors - reasoning about how people move relative to each other, how they share space, and how individual behavior reflects social navigation constraints. Using the same multi-person clips, this probe contains 3,000 samples. 3.2.2 Common Sense (CS) CS-U-1: Scene Description VQA. This probe contains 12,000 samples and consists of factual visual question-answer pairs about the observed scene, spanning five balanced categories: person activity (21.2%), object identification (20.5%), environment description (20.0%), obstacles (19.4%), and scene description (18.9%). The model is expected to exhibit broad scene understanding grounded in visible retail context. The probe consists of QA pairs and to reduce data leakage, we limit generation to 3 QA pairs per episode. CS-U-2: Environment Understanding VQA (exocentric). This probe is an exocentric counterpart to CS-U-1, using exocentric footage. Given an exocentric clip, the 8 Table 3 PRISM dataset overview. All 20+ capability probes organized by knowledge domain, viewpoint, SFT format, allocated sample counts, and annotation method. Und. = open-ended understanding; CoT = chain-of-thought reasoning with⟨think⟩tags; MCQ = multiple-choice (A/B/C/D). Sample allocations follow sqrt-proportional scaling with no single task exceeding 9.5% of the total mix. TaskDomain ViewFormatAllocated % of Total Annotation Method ER-1: Next Subtask PredictionEREgoUnd.+MCQ 25,0009.2% Gemini Robotics ER 1.5 ER-2: Task Completion VerificationEREgoUnd3,0001.1% Gemini Robotics ER 1.5 ER-3: Goal-Conditioned Action Reasoning EREgoCoT8,0003.0% LLM (Gemini Flash) ER-4: Exo-to-Ego Activity MatchingEREgo+ExoUnd.7,0002.6% LLM (Gemini Flash) + Video ER-5: Hand Interaction RecognitionEREgoUnd.+MCQ 15,0005.5% Gemini Robotics ER 1.5 ER-6: Atomic Action RecognitionERExoUnd.20,0007.4% Gemini Robotics ER 1.5 ER-7: Atomic Action ReasoningERExoCoT8,0003.0% LLM (Gemini Flash) ER-8: Multi-Actor Scene UnderstandingERExoUnd.8,0003.0% Gemini Robotics ER 1.5 ER-9: Social Navigation ReasoningERExoCoT3,0001.1% LLM (Gemini Flash) CS-U-1: Scene Description VQACSEgoUnd.+MCQ 12,0004.4% LLM (Gemini Flash) CS-U-2: Environment VQACSExoUnd.8,0003.0% LLM (Gemini Flash) CS-R-1: Spatial Reasoning (LLM CoT)CSEgoCoT+MCQ12,0004.4% LLM + Depth CS-R-2: Affordance ReasoningCSEgoCoT8,0003.0% Gemini Robotics ER 1.5 + Depth CS-R-4: Spatial ReasoningCSExoCoT2,0000.7% Gemini Robotics ER 1.5 + Depth CS-R-3: Causality ReasoningCSEgoCoT6,0002.2% LLM (Gemini Flash) SP-1: Relative Depth ReasoningSPEgoUnd.10,0003.7% Template + Depth SP-2: 360° Spatial LayoutSPExoCoT+MCQ2,0000.7% LLM + Depth IP-1: Arrow-of-TimeIPEgo+ExoUnd.30,00012.9% Self-supervised IP-1 CoT: Physics ReasoningIPEgo+ExoCoT49,00018.1% Gemini Robotics ER 1.5 + Video IP-2: Object PermanenceIPEgoUnd.8,0003.0% Self-supervised MCQ Overlay (6 tasks)MixedMixedMCQ12,0004.4% LLM (Gemini Flash) MCQ Standalone (5 categories)MixedTextMCQ14,0005.2% LLM (Gemini Flash) Grand Total (20+ tasks)270K100% model is expected to answer questions about the visible objects, spatial layout, lighting, obstacles and scene structure. This probe contains 8,000 VQA samples. CS-R-1: Spatial Reasoning with LLM Chain-of-Thought. This probe is focused on reasoning about the structural aspects of the scene. The model is expected to answer spatial questions about depth relationships, obstacle proximity, and scene layout while producing explicit chain-of-thought reasoning. We extract depth maps and compute 8 per-frame statistics (mean depth, variance, near/far-field percentiles, gradient magnitude, left-right asymmetry, top-bottom asymmetry, foreground-background ratio). Questions in this probe are balanced across categories: relative distance, obstacle awareness, scene layout, and depth profile. This probe contains 12,000 samples. CS-R-2: Affordance Reasoning (CoT). This probe expects the model to reason about object affordances - what actions are possible on visible objects and surfaces, and under what constraints?. Given an egocentric clip together with depth context, the model is expected to reason about reachable objects, possible interactions such as grasping, pushing, pulling, or opening, and the spatial limitations governing those actions. The probe targets embodied common sense that is closely tied to action feasibility in the physical world. This probe contains 8,000 samples. CS-R-3: Causality Reasoning (CoT). Given an egocentric clip with episode annotations, the model is expected to explain causal relationships between observed actions and their context - reasoning about why the person performed the observed action, what caused it, and what effects it is likely to procude. This probe focuses on everyday causal understanding in embodied retail scenarios and consists of 6,000 samples. CS-R-4: Spatial Reasoning (exocentric, CoT). This probe extends spatial reasoning to exocentric viewpoints. The model is expected to analyze clips to reason about 3D scene layout, person positions, and depth relationships from a scene-centered perspective. This probe contains 2,000 samples. 9 3.2.3 Spatial Perception (SP) SP-1: Relative Depth Reasoning. This probe is centered on relative depth ordering, depth gradients, foreground/background separation, and depth range characterization from egocentric video. The model is expected to reason spatial statistics across five balanced categories: region comparison, depth gradient, foreground/background analysis, nearest/farthest identification, and depth range characterization. This probe includes 10,000 samples. SP-2: 360° Spatial Layout Reasoning (exocentric). This probe is applied on panoramic viewpoints. Alia’s 360 ◦ omnidirectional views are used and the model is expected to produce spatial layout understanding with chain-of-thought. For each clip, depth maps are used to extract the same 8 depth statistics used in CS-R-1 to ground the spatial reasoning. The panoramic 360 ◦ perspective introduces spatial reasoning challenges that differ substantially from the forward-facing egocentric view. This probe contains approximately 2,000 samples. 3.2.4 Intuitive Physics (IP) IP-1: Arrow-of-Time (Egocentric and Exocentric). This is a binary classification probe where the model determines whether a video plays forward or backward. It is fully self-supervised as reversed clips are produced via temporal reversal, and labels are assigned by construction. Answers are shortened to direction-only (“The video is playing forward/backward.”) to prevent answer-style overlap with ER/CS scene descriptions. This probe contains 30,000 samples with a 50/50 class balance. IP-1 Chain-of-Thought: Physics Reasoning (Egocentric, CoT). This probe is focused on temporal reasoning. For each IP-1 clip, the model is expected to generate chain- of-thought reasoning over three physics axes: gravity and momentum cues, body mechanics/biomechanics, and temporal flow and causality. These chains are intended to capture physically grounded reasoning that is distinct from ER and CS style scene understanding, and to encourage explanations based on the internal consistency of observed motion. This probe evaluates physically grounded interpretation from first-person motion cues as well as scene-level exocentric views. This probe contains 49,000 samples. IP-2: Object Permanence. This is a self-supervised probe where the model is expected to affirm whether objects continue to exist even when temporarily out of frame. Every ER-1 egocentric clip is paired with a question about object permanence together with a canned answer asserting physical persistence. This probe introduces a simple but important aspect of intuitive physical understanding in everyday environments and contains 8,000 samples. MCQ Overlay. This probe converts six existing open-ended tasks (ER-1, ER-5, CS-U-1, CS-R-1, SP-1, SP-2) into a multiple- choice format by pairing each video clip with four answer options one correct and three hard distractors generated by Gemini 2.5 Flash. The model receives the video together with the question and four labeled choices (A/B/C/D), and is expected to output a single letter. This format trains the model to perform constrained selection under visual grounding, complementing the open-ended generation format used elsewhere in PRISM. This probe contains 12,000 samples balanced across the six source tasks. MCQ Standalone. This probe consists of text-only multiple-choice questions spanning five knowledge categories: retail operations, spatial reasoning, physics understanding, visual question answering, and action recognition. Unlike MCQ Overlay, these questions do not include video input, they test the model’s retained domain knowledge and reasoning ability in a purely textual setting. Each question presents four options (A/B/C/D) with a single correct answer. The questions are designed to complement the video-grounded tasks by reinforcing conceptual understanding. This probe contains 14,000 samples. 3.3 Annotation Pipeline PRISM is constructed using five complementary annotation strategies that together provide broad coverage, scalable supervision and diverse training signals across the dataset. 10 Table 4 PRISM scale statistics. Token counts at 4 fps with Qwen3VL tokenization. MetricValue Total samples270K Tasks (+ 2 eval-only)20+ Video frames (4 fps)∼11.8M Total tokens∼730M (703M visual + 27M text) Open-ended / CoT / MCQ58.6% / 30.4% / 9.2% Domain: ER / CS / SP / IP / MCQ34.7 / 18.3 / 4.4 / 33.5 / 9.2% Metadata Extraction. For ER-1, ER-2, ER-4, ER-5, ER-6, and ER-8, annotations are derived from structured metadata produced by Gemini Robotics ER 1.5 Gemini-Robotics-ER-1.5 (2025) - a specialized model for embodied physical reasoning. These metadata include episode-level summaries that contain goal hierarchies, sub-goal sequences, temporal boundaries, hand-state labels, exocentric trajectory annotations, and activity summary JSONs. We use template pools to increase linguistic diversity, for example through multiple question-answer template combinations for tasks such as ER-2 and ER-4. LLM-Based Generation (Gemini 2.5 Flash). For CS-U-1, CS-U-2, CS-R-1, CS-R-3, SP-2, ER-3, ER-7, ER-9, ER-4 (expanded), and MCQ generation, we use Gemini 2.5 Flash DeepMind (2025) to generate QA pairs and chain-of-thought reasoning. Prompts operate primarily over structured text annotations, with video conditioned prompting prompting used where the task requires direct visual grounding. MCQ overlay samples are generated by prompting Gemini to produce three hard distractors for every existing answer, and randomly assigning the correct option across A/B/C/D.. MCQ standalone samples are generated as text-only questions spanning five knowledge categories - retail, spatial, physics, visual QA and action recognition. Physics-Grounded Video Reasoning (Gemini Robotics ER 1.5). For CS-R-2, CS-R-4 and IP-1 CoT we use again utilize Gemini Robotics ER 1.5 . Unlike text-only LLM generation, these tasks require the model to process the actual video directly. Clips are uploaded through Gemini’s video processing API, and the model generates physics-grounded reasoning about gravity, momentum, body mechanics, temporal causality, affordances and spatial layout. This produces richer chain-of-thought supervision than text-conditioned prompting. Depth-Aware Analysis. Spatial reasoning tasks - CS-R-1, CS-R-2, CS-R-4, SP-1, SP-2 - are grounded in depth estimated from DepthCrafter Hu et al. (2024). For each clip, we compute spatial statistics such as mean depth, min/max, standard deviation, left-right and top-bottom quadrant means, center depth, and foreground-background separation. Self-Supervised Generation. IP-1 and IP-2 do not require any annotation or API calls. Arrow-of-time labels are determined by construction (original vs. reversed), object permanence answers assert physical persistence by construction. All samples are generated through deterministic video/image transformations. IP-1 answers are deliberately shortened to direction-only - “The video is playing forward/backward” - to prevent overlap with ER/CS scene description answer styles. 3.4 Data Format All samples follow the Cosmos-Reason2 format NVIDIA (2025b) for supervised fine-tuning: a three-message conversation consisting of system, user and assistant turns. User messages contain interleaved video or image content together with task text. We use three response formats: Understanding tasks produce direct open-ended answers, Reasoning tasks wrap the response in⟨think⟩. . .⟨/think⟩tags containing explicit chain-of-thought followed by the final answer, MCQ tasks present four choices (A/B/C/D) in the question and expect a single-letter response. Following LLaVA-Video Zhang et al. (2025), we target a balanced supervision mix of approximately∼60% open-ended,∼30% CoT, and∼10% MCQ for balanced format training. Videos are sampled at 4 fps (integer). A first-token diversity audit ensures no task has>50% of answers sharing the same opening word, mitigating first-token bias during training. 11 4 Experimental Setup and Results 4.1 Base Model and Fine-Tuning We use Cosmos-Reason2-2B NVIDIA (2025b) as our base model, a vision-language model developed by NVIDIA for physical AI applications and built on the Qwen3-VL architecture Bai et al. (2025). Cosmos- Reason2-2B is a suitable base model for PRISM because it is explicitly designed for video language reasoning in embodied settings, rather than for open-domain visual question answering alone. Its pretraining strategies and instruction-tuning methods emphasize physical scene understanding, action interpretation, and temporally grounded reasoning, making it a strong foundation for studying domain-specific SFT. The model processes interleaved video frames and text through a shared transformer backbone with 1.53B base parameters. We fine-tune Cosmos-Reason2-2B using parameter efficient adaptation, allowing us to isolate the contribution of PRISM training mixtures while keeping the training setup practical and consistent across experiments. Our primary configuration uses BF16 LoRA without quantization with rank 32 and alpha 64 applied to all language model linear layers including the query, key, value, output, gate, up an down projections. This results in 49.3M trainable parameters corresponding to 1.98% of the full model. Training uses AdamW Loshchilov and Hutter (2019) with learning rate 1×10 −4 , cosine schedule, 5% warmup, batch size 1 with gradient accumulation 8 across 4 GPUs yielding an effective batch-size of 32, and train in bfloat16 precision throughout. All models are trained for one epoch on each data mixture with gradient checkpointing enabled. The primary experiments are conducted on 4×NVIDIA RTX PRO 6000 Blackwell GPUs (96 GB each) using distributed data parallel training, achieving approximately 14 s per step. To compare against a lower-memory alternative, we additionally train a QLoRA variant on 2×NVIDIA L40S 48 GB GPUs. This allows us to examine whether the main trends we observe are robust to a more memory-constrained fine-tuning setup. 4.2 Evaluation Protocol We evaluate all models on held-out validation sets across all 20+ capabilities (250 samples per task) using MCQ accuracy. For each open-ended task, we generate multiple-choice validation sets (4 options, 1 correct) via Gemini 2.5 Flash. Classification tasks (ER-2, IP-1 Ego, IP-1 Exo) use exact-match accuracy. For chain-of-thought (CoT) tasks, we additionally employ GPT-4o as an automated judge to score the full model output including the intermediate reasoning within⟨think⟩tags and the final answer—on a normalized 0–100 scale, providing a complementary measure of open-ended generation quality beyond MCQ accuracy. We report per-capability scores, per-domain scores (macro-average over capabilities in each domain), and overall score (macro-average over all capabilities). The zero-shot Cosmos-Reason2-2B model (Baseline) provides the lower bound. 4.3 Results The MCQ accuracies for each task evaluated for both zero-shot baseline and the PRISM-fine-tuned model are given in table 5. Fine-tuning on PRISM yields substantial improvements across all capability domains, with an average gain of +23.8% over the pre-trained baseline (from 62.8% to 86.6%). As different tasks start with highly varying baseline accuracy, a better way to understand the impact of the fine-tuning is by looking at the reduction in error rate. For the harder domains such as embodied reasoning, the error rate goes down from 45.5% to 9.1%, or by a factor of five. The average reduction in error rate across all tasks is 66.6%. Table 5 also presents the average accuracies and error ratio for each domain as well. Let us now take a closer look at each domain separately. Embodied Reasoning (ER). The largest improvements appear in action reasoning and understanding: ER-3 (goal-conditioned action reasoning) improves by +62.8%, ER-7 (atomic action CoT) by +55.2%, and ER-4 (cross-view activity matching) by +48.1%. Even the challenging multi-actor tasks ER-8 and ER-9, which have no equivalent in Cosmos-Reason training, achieve 94.0% and 96.4% respectively. These results demonstrate that domain-specific SFT substantially advances embodied action understanding in retail settings. The experiment also provides an insight into the number of samples required for specific capabilities. 12 Table 5 Performance Comparison between Baseline (Zero-Shot Cosmos-Reason2-2B) and PRISM (Our model fine-tuned on the full 270K training corpus) on all evaluation tasks. Best score per task in bold. The Error Ratio shows the ratio of PRISM fine-tuned error rate to the baseline error rate. Domain-TaskID: TaskView Baseline (%) PRISM (%)∆(%) Error Ratio ER-1: Next Subtask PredictionEgo30.876.8+46.00.34 ER-2: Task CompletionEgo53.687.6+34.00.27 ER-3: Action ReasoningEgo32.495.2+62.80.07 ER-4: Cross-View Matching ⋆ Ego+Exo42.991.0+48.10.16 ER-5: Hand InteractionEgo61.695.6+34.00.11 ER-6: Atomic Action ⋆ Exo54.084.8+30.80.33 ER-7: Atomic Action CoT ⋆ Exo41.696.8+55.20.05 ER-8: Multi-Actor ⋆ Exo85.294.0+8.80.41 ER-9: Social Reasoning ⋆ Exo88.896.4+7.60.32 ER Average (9 tasks)54.590.9+36.40.20 CS-U-1: Scene VQAEgo66.888.4+21.60.35 CS-U-2: Environment VQA ⋆ Exo95.699.6+4.00.09 CS-R-1: Depth Reasoning (CoT)Ego61.680.8+19.20.50 CS-R-2: AffordanceEgo99.6100.0+0.40.00 CS-R-3: CausalityEgo85.293.6+8.40.43 CS-R-4: Exo Spatial ⋆ Exo76.685.8+9.20.61 CS Average (6 tasks)80.991.4+10.50.45 SP-1: Relative DepthEgo42.051.2+9.20.84 SP-2: 360° Layout ⋆ Exo72.897.8+25.00.08 SP Average (2 tasks)57.474.5+17.10.60 IP-1: Arrow-of-TimeEgo57.283.4+26.20.39 IP-1: Arrow-of-Time ⋆ Exo54.061.2+7.20.84 IP-2: Object PermanenceEgo44.063.4+19.40.65 IP Average (3 tasks)51.769.3+17.60.64 MCQ OverlayMixed72.895.2+22.40.18 Table 6 Average MCQ accuracy by viewpoint. ViewBaseline (%) PRISM (%)∆ (%) Ego (11 tasks)57.783.3+25.6 Exo (9 tasks)67.989.7+21.8 Mixed72.895.2+22.4 Overall62.886.6+23.8 13 Common Sense (CS). Scene understanding CS-U-1, achieves a significant improvement of +21.6%, indicating robust retail scene understanding. Depth-grounded spatial reasoning CS-R-1 improves by +19.2%, validating the effectiveness of depth-aware chain-of-thought fine-tuning. This is significant as CS was the best performing category with an avearage accuracy of 80.9%, which improves to 91.4%. Even Affordance reasoning CS-R-2, which is already impressive in the baseline, shows a positive improvement as well. Spatial Perception (SP). SP-2, based on the use of the complete 360° layout), achieves an accuracy of 97.8%, a +25.0% gain - the largest among spatial tasks - demonstrating that the model learns panoramic spatial reasoning from a previously unseen viewpoint. SP-1 (relative depth) shows a modest +9.2% gain, reaching 51.2%. This task requires fine-grained metric depth discrimination that remains challenging even with depth-grounded training data. This could also be a result of the fact that the metric depth was not available to train the original model. Intuitive Physics (IP). Among the intuitive physics tasks, IP-1 (arrow of time) shows gains of about +26.2% and +7.2%, indicating strong learning of the temporal and causal structure. IP-2 (object permanence) improves by +19.0%, reflecting better reasoning about the existence of persistent objects under occlusion. Overall, these results suggest that fine-tuning yields meaningful improvements in intuitive physics reasoning. Summary by View. We also generate performance summaries grouped by camera viewpoint, as shown in table 6. The results indicate that fine-tuning leads to substantial improvements for both egocentric and exocentric tasks, confirming that the model benefits broadly from the multi-view supervision provided by PRISM. Egocentric tasks exhibit slightly larger average gains (+25.6%) than exocentric tasks (+21.8%), suggesting that first-person visual streams may provide especially strong signals for learning action-relevant and interaction-centric reasoning. At the same time, the strong gains on exocentric tasks show that the model is also able to leverage third-person scene context effectively. Overall, these trends validate the value of multi-view training data and suggest that combining embodied first-person observations with external scene perspectives leads to more robust viewpoint-generalized reasoning. 4.4 Data Scaling Analysis We study how performance scales with training data volume by evaluating checkpoints at 40%, 60%, 80%, and 100% of the full 270K corpus (figure 4). The majority of gains are captured early: at just 40% of training, overall accuracy reaches 83.5% (+20.9% over baseline). Continued training yields diminishing but consistent returns, with the full dataset achieving 88.9%. Across domains, Embodied Reasoning (ER) shows the most dramatic improvement, jumping from 54.5% to 86.0% by the 40% mark and plateauing near 90.9%. Common Sense (CS) starts from a strong baseline of 80.9% and steadily climbs to 91.4%. Spatial Perception (SP) and Intuitive Physics (IP) improve throughout training but at a slower rate, suggesting these capabilities benefit from broader data coverage. The full-data setting yields the best aggregate performance across all domains. 14 0406080100 50 55 60 65 70 75 80 85 90 95 80.9 88.0 89.5 91.5 91.4 54.5 86.0 89.0 90.5 90.9 62.6 83.5 87.7 88.5 88.9 57.4 68.0 74.3 75.4 74.5 51.7 58.0 64.0 67.5 69.3 Training Progress (% of 270K samples) MCQ Accuracy (%) Overall ER (9 tasks) CS (6 tasks) SP (2 tasks) IP (3 tasks) Figure 4 Performance scaling with training data. MCQ accuracy by capability domain as a function of training progress. All domains improve rapidly in early training (0–40%), with ER showing the largest absolute gain (+31.5p). Performance continues to increase at a diminishing rate, with the full 270K dataset achieving the best overall accuracy of 88.9%. CS starts highest due to strong baseline scene understanding; IP shows steady improvement throughout training. 5 Conclusion We presented PRISM — a 270K-sample, multi-view, ontology-structured video SFT corpus for embodied VLMs in real-world retail environments. PRISM is distinguished by three properties that no prior dataset combines: (1) coverage of all three fundamental knowledge dimensions of physical AI—spatial, temporal- physical, and embodied action—within a single deployment domain; (2) synchronized egocentric, exocentric, and 360° panoramic supervision, including eight novel exocentric capability probes with no counterpart in existing corpora; and (3) a scalable, low-cost annotation pipeline combining LLM generation, physics-grounded video reasoning, depth-aware analysis, and self-supervised transformations. Fine-tuning Cosmos-Reason2-2B on PRISM yields a 66.6% reduction in average error rate across all 20+ probes. Our experimental results establish four actionable conclusions for the embodied AI community. First, domain-specific, ontology-structured SFT dramatically outperforms general pretraining for structured deployment environments: the Embodied Reasoning domain achieves a five-fold error reduction, with individual probes reaching error ratios as low as 0.05. Second, multi-view ego-exo training is mutually reinforcing: exocentric supervision improves cross-view understanding without sacrificing egocentric capability. Third, LLM-generated chain-of-thought annotations substantially outperform template-based alternatives, highlighting that supervision format matters as much as scale. Fourth, the data-scaling curve reveals a favorable property for practitioners: 60% of the training data captures 95% of the total gain, enabling cost-effective partial deployment while the remaining 40% provides consistent across-the-board improvements. Limitations. Our study evaluates a single base model (Cosmos-Reason2-2B), limiting conclusions about how PRISM benefits larger model variants. Automated evaluation metrics (MCQ accuracy, GPT-4o quality 15 scoring) may underestimate performance on tasks requiring nuanced natural-language outputs. Future Work. We plan to (1) extend evaluation to larger model variants (7B–13B) to characterize scaling behavior; (2) incorporate additional sensor modalities (metric depth, inertial measurements) as model inputs for richer spatial grounding; (3) integrate PRISM-trained VLMs into the GR00T NVIDIA (2025c) robotic control pipeline for end-to- end evaluation on downstream retail manipulation tasks; and (4) expand data collection to additional geographies and store formats to test the generalization limits of retail-specific SFT. 16 FrameTask / Question / Answer ER-1: Next Subtask Prediction (Ego) Q: What is the next subtask? A: Navigate to Dry Goods aisle. Walk forward into an aisle. ER-2: Task Completion Verification (Ego) Q: Has “Navigate Aisle while Reviewing List” been completed? A: No, the person walks from produce towards freezers. ER-3: Goal-Conditioned Action Reasoning (Ego, CoT) Q: Based on the person’s current task, where should they navigate to next in the store? A:⟨think⟩The person has just finished inspecting the Heritage Platinum Basmati Rice. Their current sub-goal is “Examine Product Details,” which they have completed. The next logical step is to find another product to compare or move on to the next item on their list.⟨/think⟩Continue down the current aisle to explore other rice options or adjacent product categories. ER-4: Exo-to-Ego Activity Matching (Exo) Q: What activity is the person performing? A: The worker is scanning shelf labels and restocking items in the beverage aisle. ER-5: Hand Interaction Recognition (Ego) Q: Describe the hand interactions visible in this video. A: Both hands holding an item. Pushes cart down the aisle. ER-6: Atomic Action Recognition (Exo) Q: Based on the video, what action is the person currently performing? A: The action being performed by the man is: reaching toward basket (target: blue shopping basket). Extends right arm towards the blue basket. ER-7: Atomic Action Reasoning (Exo, CoT) Q: What is the likely reason this person is doing what they’re doing in this scene? A:⟨think⟩The staff member is wearing a green vest, which is often an indicator of employee attire in a retail setting. They are bending towards the produce shelf. The action of bending with knees and hips flexing while maintaining a straight back is a biomechanically efficient way to lower one’s body to inspect or arrange items on lower shelves.⟨/think⟩The staff member is restocking or inspecting produce items on the lower shelf as part of routine store maintenance. Figure 3 PRISM capability probe examples (Part 1 of 3). Each row shows a representative video frame alongside the full question and model answer. CoT tasks include the chain-of-thought in⟨think⟩tags (shown in italics) before the final answer. 17 FrameTask / Question / Answer ER-8: Multi-Actor Understanding (Exo) Q: List all the people visible and their current activities in this retail scene. A: The footage captures 2 people in the retail environment. Person 1: A female store employee walks along the produce display, stopping to arrange and organize vegetables on the shelves and bins. Person 2: A man walks forward in the aisle, carrying a blue plastic shopping basket. ER-9: Social Navigation Reasoning (Exo, CoT) Q: Describe the actions of each individual visible in this store camera view. A:⟨think⟩From this camera view, 4 people can be identified. Person 1 walks into the produce section and interacts with staff. Person 2 is a staff member assisting at the vegetable display. Person 3 browses independently near the refrigerated section. Person 4 navigates toward the checkout area.⟨/think⟩Four individuals occupy the store with distinct goals: one customer selects produce with staff assistance, another browses independently, and a fourth heads toward checkout. CS-U-1: Scene Description VQA (Ego) Q: Describe the scene visible in this video. A: Person at refrigerator door, looking at milk and curd. CS-U-2: Environment Understanding VQA (Exo) Q: What is the typical arrangement of products on the shelving units in this grocery store aisle? A: The shelving units display a variety of packaged food items, primarily snacks, arranged by category. Products are neatly stocked and often stacked to maximize shelf space, with labels facing forward for easy identification by customers. CS-R-1: Spatial Reasoning (Ego, CoT) Q: Are items close or spread out in this scene? A:⟨think⟩Mean depth 0.17, most objects within near field...⟨/think⟩Close range. Items are within arm’s reach. CS-R-2: Affordance Reasoning (Ego, CoT) Q: Based on the spatial layout, what affordances are available to the person? A:⟨think⟩The shelves on both sides are within immediate reaching distance, affording grasping of products like snacks, beverages, and household items. The floor affords walking through the aisle. The shopping basket provides containment for selected items.⟨/think⟩The environment affords navigation through aisles and grasping items from shelves and bins. The basket facilitates collection of products. CS-R-3: Causality Reasoning (Ego, CoT) Q: Analyze the cause-and-effect: why did this action happen? A:⟨think⟩The person is in an aisle stocked with dry goods and beverages. Their goal is to perform grocery shopping, which necessitates navigating aisles to find items on their list. Walking down this aisle directly progresses through the store’s layout toward the target items.⟨/think⟩To find items on their shopping list located in the dry goods and beverages aisle. Figure 3 PRISM capability probe examples (Part 2 of 3). Continued from above. 18 FrameTask / Question / Answer CS-R-4: Spatial Reasoning (Exo, CoT) Q: Describe the typical person-object interaction required to select items from the bulk bins and refrigerated cases. A:⟨think⟩To interact with the bulk bins, a customer would reach over the counter, lift the transparent lid, and use a scoop to retrieve the product. For refrigerated cases, the customer would open one of the glass doors to access and select items.⟨/think⟩ Customers interact with bulk bins by lifting lids and scooping products, and with refrigerated cases by opening glass doors to retrieve items. SP-1: Relative Depth Reasoning (Ego) Q: Does this scene have a wide or narrow depth range? A: This scene has a moderate depth range from 0.05 to 0.78. There is a mix of closer and farther objects, but the distances do not span an extreme range. SP-2: 360° Spatial Layout Reasoning (Exo, CoT) Q: Describe the spatial layout and obstacle proximity. A:⟨think⟩Mean depth 0.43, bimodality 0.07...⟨/think⟩Open central aisle, shelves on both sides at moderate distance. IP-1: Arrow-of-Time (Physics Reasoning) (Ego, CoT) Q: Watch this video and determine if it is playing forward or backward. Analyze the body mechanics and biomechanical cues. A:⟨think⟩While the walking motion appears smooth, the foot placement and overall gait reveal subtle inconsistencies. The natural push-off and weight transfer of a forward stride are absent, giving the movement a slightly unnatural, gliding quality.⟨/think⟩ The video is playing backward. IP-1: Arrow-of-Time (Physics Reasoning) (Exo, CoT) Q: Watch this video and determine if it is playing forward or backward. Analyze the temporal flow and causal relationships. A: ⟨think⟩A person enters the frame from the left and walks toward the right. The legs move in a natural forward progression. There are no visual cues of reverse motion or unnatural movement patterns.⟨/think⟩ The video is playing forward. IP-2: Object Permanence (Ego) Q: Something left the field of view. Does the object continue to exist when not visible? A: The object still exists. In a retail environment, items on shelves, carts, and people maintain their physical presence even when they leave the camera frame. Object permanence is a basic physical principle. MCQ Overlay: Multiple-Choice with Video (Mixed) Q: What hand activity can you observe in this video? A. One hand is actively engaged in gesturing while the other is still. B. Both hands are idle and resting at the person’s sides. C. The person is holding an object in their right hand and gesturing with the left. D. The person is using both hands to clap rhythmically. A: B Figure 3 PRISM capability probe examples (Part 3 of 3). CS-R-4 and SP tasks test spatial understanding. IP-1 CoT variants demonstrate physics-grounded reasoning about temporal direction. IP-2 evaluates object permanence. MCQ Overlay converts open-ended tasks into four-choice format. 19 References Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631, 2025. Kevin Black et al.π 0 : A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164, 2024. Anthony Brohan et al. RT-2: Vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning (CoRL), 2023. Zhe Chen et al. InternVL: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv preprint arXiv:2312.14238, 2024. Dima Damen et al. Scaling egocentric vision: The EPIC-KITCHENS dataset. In ECCV, 2018. Google DeepMind. Gemini 2.5: A family of highly capable multimodal models. Technical Report, 2025. Tim Dettmers et al. QLoRA: Efficient finetuning of quantized language models. arXiv preprint arXiv:2305.14314, 2023. Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In ICCV, 2015. DreamVu. Alia 360°panoramic camera, 2020. URL https://dreamvu.ai/#technology. Danny Driess et al. PaLM-E: An embodied multimodal language model. arXiv preprint arXiv:2303.03378, 2023. Chaoyou Fu et al. Video-MME: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075, 2024. Justine Gajo, Aldrin Merales, Timothy Escarcha, Marco Molina, Antonio Nartea, Andrei Maminta, Joshua Roldan, and Rowel Atienza. Sari Sandbox: A virtual retail store environment for embodied AI agents. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), RetailVision, 2025. arXiv:2508.00400. Gemini-Robotics-ER-1.5. Google gemini er 1.5, 2025. URLhttps://ai.google.dev/gemini-api/docs/ robotics-overview. Kristen Grauman et al. Ego4D: Around the world in 3,000 hours of egocentric video. In CVPR, 2022. Kristen Grauman et al. Ego-Exo4D: Understanding skilled human activity from first- and third-person perspectives. arXiv preprint arXiv:2311.18259, 2024. Edward J Hu et al. LoRA: Low-rank adaptation of large language models. In ICLR, 2022. Wenbo Hu et al. DepthCrafter: Generating consistent long depth sequences for open-world videos. arXiv preprint arXiv:2409.02095, 2024. Bo Li et al. LLaVA-OneVision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024a. Kunchang Li et al. MVBench: A comprehensive multi-modal video understanding benchmark. arXiv preprint arXiv:2311.17005, 2024b. Kunchang Li et al. VideoChat2: Chat-centric video understanding. arXiv preprint arXiv:2311.17005, 2024c. Bin Lin et al. Video-LLaVA: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122, 2023. Ji Lin et al. VILA: On pre-training for visual language models. arXiv preprint arXiv:2312.07533, 2024. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. URLhttps://arxiv.org/abs/1711. 05101. Muhammad Maaz et al. Video-ChatGPT: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424, 2023. 20 Davide Mazzini et al. RetailAction: Dataset for multi-view spatio-temporal localization of human–object interactions in retail environments. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), RetailVision, 2025. NVIDIA. Cosmos-reason1: From physical common sense to embodied reasoning. arXiv preprint, 2025a. NVIDIA. Cosmos-reason1 and cosmos-reason2: Reasoning foundation models for physical common sense. arXiv preprint, 2025b. NVIDIA. GR00T n1: An open foundation model for generalist humanoid robots. arXiv preprint, 2025c. RetailVision Organizers. RetailVision workshop series.https://retailvisionworkshop.github.io, 2020–2025. Annual workshop at CVPR/ICCV, 2020–2025. Pierre Sermanet, Tianli Ding, Jeffrey Zhao, Fei Xia, Debidatta Dwibedi, Keerthana Gopalakrishnan, Christine Chan, Gabriel Dulac-Arnold, Sharath Maddineni, Nikhil Jain, Peng Xu, Yunfei Yuan, et al. RoboVQA: Multimodal long-horizon reasoning for robotics. arXiv preprint arXiv:2311.00899, 2023. Bharat Singh, Tim K Marks, Michael Jones, Oncel Tuzel, and Ming Shao. A multi-stream bi-directional recurrent neural network for fine-grained action detection. In CVPR, 2016. various. PhysBench: Benchmarking physical reasoning in vision-language models. arXiv preprint, 2024. Dongling Wei et al. Arrow of time and its reversal on the IBM quantum computer. In Scientific Reports, 2019. Original concept: Pickup et al., Arrow of Time in Videos, BMVC 2014. Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Llava-video: Video instruction tuning with synthetic data, 2025. URL https://arxiv.org/abs/2410.02713. 21 Appendix A Detailed Capability Examples We provide representative examples from each capability to illustrate the diversity and quality of PRISM samples. A.1 ER-1: Next Subtask Prediction Example 1 Context: Goal: “Perform Grocery Shopping”. Sub-goal: “Navigate Produce Section”. Scene: Produce Section. Completed: “Aisle Navigation (Fruits and Vegetables).” Q: What is the next subtask the person will perform? A: The next subtask is: “Aisle Navigation (Dry Goods)”. Walk forward into an aisle. Example 2 Context: Goal: “Perform Grocery Shopping”. Sub-goal: “Select specific spice packets”. Scene: Aisle (Spices section). Q: What is the next subtask the person will perform? A: The next subtask is: “Item Inspection and Comparison”. Example 3 Context: Goal: “Perform Grocery Shopping”. Sub-goal: “Social navigation in produce aisle”. Scene: Produce Section Aisle. Q: What is the next subtask the person will perform? A: The next subtask is: “Aisle Traversal and Shopper Movement”. The camera continues navigating through the produce section. A.2 ER-2: Task Completion Verification Example 1 Q: Can you determine if the task “Navigate Aisle while Reviewing List” has been completed? A: No, the task has not been completed yet. The person walks from the produce section towards the refrigerated freezers, checking their phone. Example 2 Q: Based on what you see, has the person finished “Wait for Checkout”? A: The task is not yet complete. The person turns back towards the checkout counter, resuming their wait in line. Example 3 Q: Can you determine if the task “Navigate Aisle” has been completed? A: The task is not yet complete. The person continues walking forward, navigating between large bags of rice and stacks of cookies. 22 A.3 ER-3: Goal-Conditioned Action Reasoning Example 1 Q: Based on the person’s current task, where should they navigate to next in the store? A:⟨think⟩The person has just finished inspecting the Heritage Platinum Basmati Rice. Their current sub-goal is “Examine Product Details,” which they have completed. The next logical step is to find another product to compare.⟨/think⟩ Continue down the current aisle to explore other rice options or adjacent product categories. Example 2 Q: Why is the person performing this action in the context of their shopping goal? A:⟨think⟩The person is reaching for a spice packet while their shopping list is visible on their phone. This indicates they are actively checking off items from a prepared list.⟨/think⟩The person is selecting a specific spice that matches an item on their shopping list. A.4 ER-4: Exo-to-Ego Activity Matching Example 1 Q: What activity is the person performing in this exocentric view? A: The worker is scanning shelf labels and restocking items in the beverage aisle. They methodically check each section before placing products from a cart onto the shelves. Example 2 Q: Describe the activity visible from this overhead camera perspective. A: A customer is browsing the produce section, picking up and inspecting individual vegetables before placing selected items into a blue shopping basket. A.5 ER-5: Hand Interaction Recognition Example 1 Q: Describe what the person is doing with their hands in this clip. A: Both hands are holding an item. The person pushes the cart down the aisle, passing shelves filled with rice bags and other packaged goods. Example 2 Q: Describe what the person is doing with their hands in this clip. A: The left hand is holding an item. The right hand is resting and not engaged. The person stops near the vegetable displays and observes another customer selecting carrots from a crate. A.6 ER-6: Atomic Action Recognition Example 1 Q: Based on the video, what action is the person currently performing? A: The action being performed by the man is: reaching toward basket (target: blue shopping basket). Extends right arm towards the blue basket. Example 2 Q: Describe the atomic action visible in this clip. A: The person is performing: walking forward (direction: along aisle). Observable cues: full body translation, upright posture, consistent forward gait. 23 A.7 ER-7: Atomic Action Reasoning Example 1 Q: What is the likely reason this person is doing what they’re doing in this scene? A:⟨think⟩The staff member is wearing a green vest, which is often an indicator of employee attire. They are bending towards the produce shelf. The action of bending with knees and hips flexing while maintaining a straight back is a biomechanically efficient way to lower one’s body to inspect or arrange items on lower shelves.⟨/think⟩The staff member is restocking or inspecting produce items on the lower shelf as part of routine store maintenance. Example 2 Q: Why is this person performing the observed action? A:⟨think⟩The customer is standing still near the dairy refrigerator with their head tilted slightly downward, scanning the shelves. This posture suggests they are searching for a specific product among many options.⟨/think⟩ The person is comparing product labels to find the right variant or brand of dairy product. A.8 ER-8: Multi-Actor Scene Understanding Example 1 Q: List all the people visible and their current activities in this retail scene. A: The footage captures 2 people. Person 1: A female store employee walks along the produce display, stopping to arrange and organize vegetables on the shelves and bins. Person 2: A man walks forward in the aisle, carrying a blue plastic shopping basket. Example 2 Q: Describe what each person is doing in this scene. A: Three people are visible. Person 1: A customer pushes a cart through the main aisle. Person 2: A staff member arranges products on the end-cap display. Person 3: A shopper examines items on the top shelf, reaching upward. A.9 ER-9: Social Navigation Reasoning Example 1 Q: Describe the actions of each individual visible in this store camera view. A:⟨think⟩From this camera view, 4 people can be identified. Person 1 walks into the produce section and interacts with staff. Person 2 is a staff member assisting at the vegetable display. Person 3 browses independently near the refrigerated section. Person 4 navigates toward the checkout area.⟨/think⟩Four individuals occupy the store with distinct goals: one customer selects produce with staff assistance, another browses independently, and a fourth heads toward checkout. A.10 CS-U-1: Scene Description VQA Example 1 Q: Describe what is happening in this scene. A: The person stands at the open refrigerator door, looking at the milk pouches and curd packages inside. The camera captures the person’s hands near the shelves. Another person walks past in the background. Example 2 Q: Describe the environment and surroundings in this scene. A: This is the Aisle (Sweets/Snacks) section of the supermarket. The person’s current sub-goal is: Examine Item Details and Check Shopping List. 24 A.11 CS-U-2: Environment Understanding VQA Example 1 Q: What is the typical arrangement of products on the shelving units in this grocery store aisle? A: The shelving units display a variety of packaged food items, primarily snacks, arranged by category. Products are neatly stocked and often stacked to maximize shelf space, with labels facing forward for easy identification by customers. Example 2 Q: Describe the layout of the produce section from this exocentric view. A: The produce section features open bins along both sides of a central walkway, with fruits on the left and vegetables on the right. Overhead signage indicates product categories. A weighing scale is positioned at the end of the display. A.12 CS-R-1: Spatial Reasoning with Chain-of-Thought Example 1 Q: Are the visible items generally close to the person, or spread across a wider range? A:⟨think⟩The depth statistics show a mean depth of 0.172. This is a relatively low value, indicating that most objects are in the closer to mid-range. The minimum depth is 0.055, suggesting something very close like the basket on the floor...⟨/think⟩ Items are primarily close-range. Example 2 Q: What is the shopper’s current state, and what does this imply for robot navigation? A:⟨think⟩I observe a human shopper in the aisle in a “Dynamic” state. This dynamism is evidenced by the shopper actively picking up an item from a shelf.⟨/think⟩The shopper is dynamic; the robot should maintain safe distance and anticipate movement. A.13 CS-R-2: Affordance Reasoning Example 1 Q: Based on the spatial layout, what affordances are available to the person? A:⟨think⟩The shelves on both sides are within immediate reaching distance, affording grasping of products like snacks, beverages, and household items. The floor affords walking through the aisle. The shopping basket provides containment for selected items.⟨/think⟩The environment affords navigation through aisles and grasping items from shelves and bins. The basket facilitates collection of products. A.14 CS-R-3: Causality Reasoning Example 1 Q: Analyze the cause-and-effect: why did this action happen? A:⟨think⟩The person is in an aisle stocked with dry goods and beverages. Their goal is to perform grocery shopping, which necessitates navigating aisles to find items on their list. Walking down this aisle directly progresses through the store’s layout toward the target items.⟨/think⟩To find items on their shopping list located in the dry goods and beverages aisle. 25 A.15 CS-R-4: Spatial Reasoning (Exocentric) Example 1 Q: Describe the typical person-object interaction required to select items from the bulk bins and refrigerated cases. A:⟨think⟩To interact with the bulk bins, a customer would reach over the counter, lift the transparent lid, and use a scoop to retrieve the product. For refrigerated cases, the customer would open one of the glass doors to access and select items.⟨/think⟩Customers interact with bulk bins by lifting lids and scooping products, and with refrigerated cases by opening glass doors to retrieve items. A.16 SP-1: Relative Depth Reasoning Example 1 Q: Does this scene have a wide or narrow depth range? A: This scene has a moderate depth range from 0.05 to 0.78. There is a mix of closer and farther objects, but the distances do not span an extreme range. Example 2 Q: Are objects in the top portion of the view closer or farther than those at the bottom? A: Objects in the top portion of the view are at shorter distances than those at the bottom, indicating a downward-looking perspective toward the floor and nearby shelves. A.17 SP-2: 360° Spatial Layout Reasoning Example 1 Q: Describe the spatial layout and obstacle proximity from this panoramic view. A:⟨think⟩Mean depth 0.43, bimodality 0.07. The panoramic view reveals a central open aisle flanked by shelving units on both sides at moderate distance.⟨/think⟩Open central aisle, shelves on both sides at moderate distance. No immediate obstacles in the navigation corridor. A.18 IP-1: Arrow-of-Time Example 1 (Forward) Q: Determine whether this video is playing in its original direction or in reverse. A: This clip is playing forward. The person walks down a wide aisle, with shelves displaying biscuits and other packaged snacks on both sides. Example 2 (Backward) Q: Is this video playing forward or backward? A: The video has been reversed and is playing backward. The person places the selected ghee container into the shopping trolley. 26 A.19 IP-1 CoT: Physics Reasoning Example 1 (Ego, Backward) Q: Watch this video and determine if it is playing forward or backward. Analyze the body mechanics and biomechanical cues. A:⟨think⟩While the walking motion appears smooth, the foot placement and overall gait reveal subtle inconsis- tencies. The natural push-off and weight transfer of a forward stride are absent, giving the movement a slightly unnatural, gliding quality.⟨/think⟩ The video is playing backward. Example 2 (Exo, Forward) Q: Watch this video and determine if it is playing forward or backward. Analyze the temporal flow and causal relationships. A:⟨think⟩A person enters the frame from the left and walks toward the right. The legs move in a natural forward progression. There are no visual cues of reverse motion or unnatural movement patterns.⟨/think⟩The video is playing forward. A.20 IP-2: Object Permanence Example 1 Q: Something left the field of view. Does the object continue to exist when not visible? A: The object still exists. In a retail environment, items on shelves, carts, and people maintain their physical presence even when they leave the camera frame. Object permanence is a basic physical principle. Example 2 Q: A person walked behind a shelf and is no longer visible. Do they still exist? A: Yes, the person continues to exist behind the shelf. Occlusion by a physical barrier does not cause objects or people to cease existing. They will likely reappear when the viewpoint changes or they move past the obstruction. A.21 MCQ Overlay Example 1 Q: What hand activity can you observe in this video? A. One hand is actively engaged in gesturing while the other is still. B. Both hands are idle and resting at the person’s sides. C. The person is holding an object in their right hand and gesturing with the left. D. The person is using both hands to clap rhythmically. A: B 27 A.22 MCQ Standalone Example 1 (Physics) Q: When a shopping cart is pushed and then released on a flat store floor, it gradually slows down and stops. What is the primary reason? A. The cart runs out of energy. B. Friction between the wheels and the floor dissipates kinetic energy. C. Gravity pulls the cart downward. D. Air resistance is the dominant force. A: B Example 2 (Retail) Q: In a typical grocery store, which section is most likely located near the entrance? A. Frozen foods. B. Fresh produce. C. Cleaning supplies. D. Canned goods. A: B 28 B Fine-Tuning Details B.1 Fine-Tuning Configuration Table 7 Fine-tuning hyperparameters. ParameterValue Base modelCosmos-Reason2-2B ArchitectureQwen3-VL QuantizationNone (full BF16) Compute dtypebfloat16 LoRA rank (r)32 LoRA alpha (α)64 LoRA targetq,k,v,o,gate,up,down_proj (LM only) Total params2.49B OptimizerAdamW Learning rate1× 10 −4 LR scheduleCosine Warmup5% of total steps (397 steps) Batch size1 per GPU Gradient accumulation 8 Effective batch size32 Max sequence length4,096 Max pixels360,000 Epochs1 Gradient checkpointing Enabled (non-reentrant) AttentionSDPA Hardware4× RTX PRO 6000 Blackwell (98 GB) Training time∼35 hours (7,942 steps) B.2 Qwen3-VL Temporal Frame Handling During implementation, we discovered a compatibility issue between the Qwen3-VL processor and the get_rope_indexmethod when processing video inputs. The processor creates per-frame token groups in m_token_type_idsseparated by timestamp tokens, butvideo_grid_thwcontains a single [T,H,W] entry. Theget_rope_indexmethod expects one grid entry per contiguous type-2 group, causing aStopIteration error. Our solution monkey-patchesget_rope_indexto internally split each [T,H,W] entry intoTrows of [1,H,W], matching the per-frame token groups. The visual encoder receives the original unsplit grid for correct spatial processing. This fix is necessary for both training and inference with video inputs. C Contributions and Acknowledgments C.1 Core Contributors: Amirreza Rouhi, Parikshit Sakurikar, Satya Sai Reddy, Narsimha Menga, Anirudh Govil, Sri Harsha Chittajallu, Rajat Aggarwal, Anoop Namboodiri, and Sashi Reddi. C.2 Acknowledgments: The authors express their sincere gratitude to Deepu Tiwari, Suresh K, Shrinivas Gone, Sirigiri Manikanta, Farooq Basha Chowdary, Bagara Sai Varun, Yedida Phani Sri Sathvik, Boddu Shyamala, Shanmuki Priya 29 Bala, Nasika Lokesh Vara Prasad, Sukanya Marri, Mahajan Manikanta, Pasunoori Venkatsai, Manisha, K Uday Kiran, Induri Srikanth Reddy, Ayapakola Dwarkesh, Nihasri Gundu, Cholleti Vinay Kumar, Rudra. Saikiran, Akuthota Yashwanth, Achintalwar Divya Jyothi, Akkala Varshitha, Gadthy Sai Keerthika, Shaik Mohammad Irfan, C Bharath, Gorli Revathi for their valuable assistance with data collection and annotation. 30