Paper deep dive

EgoOmniWorld: Unifying Egocentric Understanding via Ego-Action-World Interactions

Songruo Chen, Yuqian Fu, Yu Li, Mingkang Dong, Sifan Zhou, Bin Ren, Xu Zheng, Tianwen Qian, Mohamed Elhoseiny, Yanwei Fu

Year: 2026Venue: arXiv preprintArea: cs.CVType: PreprintEmbeddings: 117

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Egocentric video understanding spans diverse capabilities, modalities, and evaluation formats, yet these are often studied in isolation despite their tight coupling in real-world interaction. We propose the Ego-Action-World interaction loop as a unifying principle: the wearer perceives the world, forms goals, acts, changes the world, and uses feedback to guide subsequent behavior. Based on this view, we introduce EgoOmniWorld, a benchmark organized around four complementary interaction structures: Node captures ego, action, and world states; Edge captures directed relations among them; Chain captures linked dependencies; and Loop captures feedback-closed interactions. We develop an interaction-first, evidence-grounded data pipeline that extracts structured interactions from egocentric videos, grounds them in visual and auditory evidence, and generates questions according to their interaction structure. Required modalities and single- or multi-turn formats follow naturally from the supporting evidence and dependencies. EgoOmniWorld contains 24,166 QA groups, including 20,284 training and 3,882 test samples. Experiments on representative open- and closed-source multimodal large language models reveal substantial and uneven limitations across interaction structures. Models trained on EgoOmniWorld improve in-domain interaction understanding and generalize to existing egocentric benchmarks. EgoOmniWorld provides a unified foundation for evaluating and advancing egocentric video understanding. Data, code, and models will be released.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

113,906 characters extracted from source content.

Expand or collapse full text

EGOOMNIWORLD: UNIFYING EGOCENTRIC UNDER- STANDING VIA EGO–ACTION–WORLD INTERACTIONS Songruo Chen 1∗ Yuqian Fu 2∗† Yu Li 1∗ Mingkang Dong 3 Sifan Zhou 4 Bin Ren 5 Xu Zheng 6 Tianwen Qian 7 Mohamed Elhoseiny 2 Yanwei Fu 1 1 Fudan University 2 KAUST 3 Universiti Malaya 4 CMU 5 MBZUAI 6 Great Bay University 7 ECNU Capabilities Assessed Separately EgoOmniWorld Existing ego benchmarks:Fragmented Egocentric Perception What do I see and hear? Action Understanding What am I doing? World State Understanding What is the state of my surroundings? Temporal reasoning What happened before and after? Intent Inference What goal does this action serve? Action Effect Reasoning How did my action change the world? · Guide Change Update Ego State & memory Action Execution & progress World State & dynamics Ego–Action–World Interaction Loop VisualAudio Audio-visual What is my goal? How should I adapt? Node Action Status What am I doing with the spatula? Edge Action Effects How does turning down the heat change the sound from the pan? Chain action-effect-next action What happens to the vegetables after I chop them?How does their new location affect what I do next? Loop feedback & adaptation After the stirring sound, what tells me to adjust the heat? what do I do next? 24,166 QA Ego4D + HD-EPIC · single- / multi-turn · Audiovisual Evidence VisualAuditoryAudio-visual 1 Modalities Considered Independently 2 Vision-only What object is on the table? Audio-only What sound is happening? Single-turn QAMulti-turn Dialogue What should I do next? I just added the salt. Does it look ready to serve? What should I do next? Question Formats Defined Separately 3 A unified framework for ego-action-world understanding Figure 1: EgoOmniWorld unifies fragmented egocentric evaluation around the Ego–Action–World interaction loop, organizing questions into Node, Edge, Chain, and Loop structures. ABSTRACT Egocentric video understanding has rapidly expanded into a diverse landscape of benchmarks targeting different capabilities, modalities, and evaluation formats. While each direction captures an important aspect of egocentric understanding, these capabilities are often studied in isolation, despite being tightly coupled in real-world interaction. We argue that a defining property of egocentric video pro- vides a natural unifying principle: the wearer perceives the world, forms goals, takes actions, changes the world, and observes the resulting feedback to guide sub- sequent behavior, forming a closed Ego–Action–World interaction loop. Based on this view, we introduce EgoOmniWorld, a unified benchmark that organizes ego- centric understanding through four complementary interaction structures: Node, capturing ego, action, and world states; Edge, capturing directed relations among them; Chain, capturing linked dependencies across multiple relations; and Loop, capturing feedback-closed interactions. We further develop an interaction-first, evidence-grounded data pipeline that extracts structured interactions from ego- centric videos, grounds them in visual and auditory evidence, and generates ques- tions according to their interaction structure. Rather than treating multimodal- ity and dialogue form as separate task families, the pipeline naturally determines the required modality and question-answering format according to the evidence and dependencies involved. EgoOmniWorld contains 24,166 QA groups, includ- ing 20,284 training and 3,882 test samples, spanning diverse interaction scopes, modality requirements, and single- and multi-turn settings. Experiments on rep- resentative open- and closed-source multimodal large language models (MLLMs) reveal substantial and uneven limitations across Ego–Action–World interactions. Models trained on EgoOmniWorld consistently improve in-domain interaction un- ∗ Equal contribution. † Corresponding author. 1 arXiv:2610.18499v1 [cs.CV] 13 Oct 2026 derstanding and generalize to existing egocentric benchmarks. We hope EgoOm- niWorld provides a unified foundation for systematically evaluating and advancing egocentric video understanding. Data, code, and models will be released. 1INTRODUCTION Egocentric video understanding has rapidly evolved into a broad and increasingly diverse research landscape [7, 13, 15, 30]. Existing benchmarks target a wide range of capabilities, including visual perception [11, 14], action understanding [10, 20], spatial and temporal reasoning [19, 26, 29], in- tent inference [12, 16], and planning prediction [2, 3]. Recent efforts further span different modality requirements, from vision-only to joint audio-visual understanding [21, 32], and evaluation formats, from single-turn question answering to multi-turn dialogue [8, 28]. While each direction captures an important aspect of egocentric understanding, the resulting benchmark landscape remains frag- mented, with different capabilities, modalities, and evaluation formats often studied under separate task definitions and protocols. The field therefore lacks a common framework for systematically organizing these diverse aspects and understanding how they relate to one another. A defining characteristic of egocentric video is that it records the world from the perspective of an active participant rather than a passive observer. The wearer continuously perceives the surrounding world, forms goals, and takes actions to achieve them, while the resulting changes in the world pro- vide feedback that guides subsequent decisions and behavior. We refer to this closed process as the Ego–Action–World interaction loop, which naturally connects perception, action understanding, intent reasoning, world-state change, and feedback within a unified framework. Based on this view, we introduce EgoOmniWorld, a unified benchmark for egocentric video un- derstanding centered on Ego–Action–World interactions. Rather than organizing questions around isolated capability labels, EgoOmniWorld characterizes complementary structural scopes of the in- teraction process. Node captures individual Ego, Action, and World states; Edge captures directed relations among the nodes; Chain captures linked dependencies across multiple relations; and Loop captures feedback-closed interactions in which action outcomes are perceived by the ego and in- fluence subsequent goals or behavior. Note that these structures represent different scopes of in- teraction composition rather than a hierarchy of task difficulty. Likewise, we argue that modality and dialogue form should also not be treated as independent task dimensions. Instead, they arise naturally from the Ego–Action–World interaction process itself: different stages of the interaction call for different evidence, whether visual or audio-visual, and different dependency structures are naturally expressed through single- or multi-turn QA. In this way, EgoOmniWorld unifies not only diverse understanding capabilities, but also the modalities and dialogue formats through which they are most naturally evaluated. To instantiate this framework, we develop an interaction-first, evidence-grounded data engine over Ego4D [9] and HD-EPIC [17]. As illustrated in Figure 1, rather than generating isolated questions first and composing them afterward, the data engine first reconstructs structured Ego–Action–World interactions from egocentric videos. It identifies ego states and goals, executed actions, world states and changes, and observable feedback; grounds these components in visual and auditory evidence; and aggregates related interactions through persistent entities and task episodes. Questions are gen- erated only after the target interaction structure and its supporting evidence have been established, with modality requirements and dialogue forms determined by the evidence and dependencies in- volved. This interaction-first design enables Node and Edge questions to be derived from local inter- action components, Chain questions from linked dependencies, and Loop questions from feedback- closed cycles, while preserving traceable evidence for each answer. Generated candidates further undergo shared quality checks, structure-specific validation, constrained repair, and revalidation. In total, the resulting EgoOmniWorld benchmark contains 24,166 evidence-grounded QA groups, including 20,284 for training and 3,882 for testing. Extensive experiments on various open- and closed-source multimodal large language models (MLLMs) reveal three key findings. First, current models exhibit markedly different strengths across Ego–Action–World interaction structures and semantic categories, suggesting that aggregate perfor- mance alone obscures important capability gaps. Second, training on EgoOmniWorld consistently improves in-domain interaction understanding under both LoRA and full fine-tuning; for example, Qwen2.5-Omni-7B improves from 36.42% to 47.36% with LoRA and 48.34% with full fine-tuning. 2 Third, the trained model demonstrates zero-shot transfer to EgoSound, EgoTempo, and EgoToM, achieving gains of up to 8.08 percentage points. In summary, our contributions are threefold: 1) we introduce an Ego–Action–World interaction framework with four complementary structures—Node, Edge, Chain, and Loop—for systemati- cally capturing diverse aspects of egocentric understanding; 2) we instantiate this framework as EgoOmniWorld, a 24,166-sample evidence-grounded benchmark constructed with an interaction- first data engine, spanning diverse capability scopes, modality requirements, and evaluation formats; and 3) we demonstrate that EgoOmniWorld serves as both a unified diagnostic testbed and a source of transferable supervision for egocentric video understanding. 2RELATED WORK Egocentric Video Benchmarks. A growing body of egocentric video benchmarks has explored in- creasingly diverse aspects of first-person understanding. EgoSchema [14] and EgoTempo [18] em- phasize long-horizon and temporal reasoning; EgoTaskQA [10] studies task-oriented understanding, including action dependencies and goal-related reasoning; EgoToM [12] focuses on reasoning about the wearer’s mental states, goals, and actions; and MultiHop-EgoQA [1] targets multi-hop reason- ing across temporally distributed evidence. Other benchmarks, e.g., EgoCross [11], EgoNight [29], and EgoTextVQA [31] have explored complementary aspects such as cross-domain generalization, nighttime understanding, and scene-text understanding. Beyond capability-specific evaluation, recent benchmarks have expanded along modality and dia- logue dimensions. EgoLifeQA [27], EgoSound [32], and EgoAVU [21] extend egocentric under- standing from vision-only input to audio-visual reasoning. Meanwhile, recent efforts e.g., LifeE- val [8] move beyond single-turn QA toward multi-turn or dialogue-based evaluation. Table 1 pro- vides a representative, rather than exhaustive, comparison of existing benchmarks in terms of inter- action structure, input modality, and dialogue format. Viewed through the Ego–Action–World lens, these works substantially broaden the landscape of egocentric understanding, but remain organized around different task definitions, modality settings, and evaluation protocols, without a common structure connecting them. EgoOmniWorld instead treats these dimensions as natural properties of the underlying interaction: the required modality follows from the available supporting evidence, while single- or multi-turn QA follows from the dependency structure being evaluated. Table 1:Representative coverage of existing egocentric benchmarks under the EgoOmniWorld taxonomy. E, A, and W denote Ego, Action, and World; arrows indicate Edge directions.✓: explicit coverage;∼: related but not explicitly structured. BenchmarkInputNodeEdgeChain LoopDialogue EgoSchemaVideoA, WA→E∼–single-turn EgoTempoVideoA, WE→A∼–single-turn EgoTaskQAVideoE, A, W A→E; A→W; W→A ∼–single-turn EgoToMVideoE, A→E∼–single-turn MultiHop-EgoQAVideoA, W–∼–single-turn EgoLifeQAVideo + Audio E, A, W W→E ; E→A∼–single-turn EgoSoundVideo + Audio E, A, W E→W–single-turn LifeEvalVideoE, A, W A↔E ; W→A∼–single- & multi-turn EgoOmniWorldVideo + AudioE, A, WE↔A; E↔W; A↔W✓Single- & Multi-turn Omni-modal Multimodal LLMs. Recent multimodal large language models (MLLMs) increas- ingly support joint video, audio, and language understanding. Representative systems include pro- prietary models such as Gemini [23] and Qwen-Omni-Plus/Flash [24], as well as open-source mod- els including MiniCPM-o [5], video-SALMONN [22], VideoLLaMA2 [4], Qwen2.5-Omni [25], and Qwen3-Omni [24]. These models provide a strong foundation for evaluating holistic egocentric understanding across heterogeneous visual, auditory, and interaction cues. We therefore benchmark representative closed- and open-source models from these families on EgoOmniWorld, and further study whether EgoOmniWorld supervision improves interaction understanding and directly transfers to existing egocentric benchmarks. 3 Ego state World perception Intent formation Intent attribution Goal planning Action effects Action grounding Ego-action-world interaction structure Chain Edge Node Loop Q1: What did I do with the hoe? A1: Pushed mud aside to improve the flow. Q2: What was the channel like afterward? A2: Some mud blockage remained. Q3: Given that state, what did I do next? A3: Pushed the hoe in again to clear more mud. Q1: What did I do with the pot of hot oil? A1: Moved it off the stove onto the ground. Q2: What changed? A2: The stove was freed. Q3: Why was it moved, given the later pan placement? A3: To make room for the frying pan to heat dough. Q: What do I see and hear as I cut the paper along the printed guideline with the scissors? A: The paper separates along the guideline, with snapping and slicing sounds. Q: What do I do with the roller? A: I roll the paint roller up and down the wall surface with my right hand. Clear mud → remaining blockage → clear again Move pot → free stove → infer later purpose Action Effects · Audio + Visual Action Status · Visual EGO ACTION WORLD Action status World state Interaction scopes and question types Larger scope Smaller scope Egocentric interaction loop Observe Act Change Receive feedback Continue clearing the channel Notice mud blockage Notice mud blockage Push mud aside Partially restored flow Detect remaining blockage Figure 2: The EgoOmniWorld taxonomy. Left: an egocentric interaction loop. Middle: the Ego– Action–World structure, comprising three node states and six directed edge relations. Right: four complementary interaction structures, from Node to Loop, with example questions. 3EGOOMNIWORLD BENCHMARK CONSTRUCTION This section presents the core design of EgoOmniWorld (Section 3.1), its interaction-first data engine for structuring Ego–Action–World interactions into evidence-grounded VQA (Section 3.2), and the resulting benchmark statistics (Section 3.3). 3.1CORE DESIGN PRINCIPLES EgoOmniWorld is built around the defining closed-loop nature of egocentric interaction: the wearer perceives the world, acts according to a goal, changes the world, and observes the resulting state to guide subsequent behavior. We organize this process around three design principles. Ego–Action–World interactions are the unit of organization. Each interaction is represented by three components. The Ego captures the wearer’s interaction-relevant state, including observations, goals, and current subgoals; the Action captures executed behavior and its progress; and the World captures external entities, their states, and state changes. Together, these components provide a common representation for organizing diverse egocentric understanding capabilities. Related inter- actions can further be grouped into task episodes to capture dependencies beyond a single event. Four structures capture complementary interaction scopes. Rather than defining tasks by iso- lated capability labels, EgoOmniWorld derives four structural scopes from the Ego–Action–World process, as in Figure 2. Node targets individual Ego, Action, or World states, corresponding to Ego State, Action Status, and World State. Edge targets one directed relation between two components: Ego→Action (Goal Planning), Action→Ego (Intent Attribution), Action→World (Action Effects), World→Action (Action Grounding), World→Ego (Intent Formation), and Ego→World (World Per- ception). Chain composes multiple directed relations into linked dependencies, where earlier states, actions, or observations are required to understand later events. Loop closes the interaction cycle by capturing how action outcomes are perceived by the ego and subsequently confirm, adjust, or change its goals or behavior. Thus, Node, Edge, Chain, and Loop correspond respectively to inter- action elements, directed relations, their composition, and feedback closure, rather than a hierarchy of task difficulty. Modality and dialogue form arise from the interaction. Visual and auditory signals are treated as complementary evidence for Ego–Action–World interactions rather than as separate task families. The modality requirement of each question is determined by the evidence necessary to answer it, yielding visual, auditory, or audio-visual questions. Likewise, dialogue form follows the dependency structure being evaluated: local states or relations can be probed in a single turn, while multi-turn QA can expose intermediate states and dependencies across an interaction. Modality and dialogue form are therefore orthogonal to the Node–Edge–Chain–Loop structure and are instantiated according to the underlying evidence and interaction dependencies. 3.2DATA ENGINE PIPELINE To instantiate the Ego–Action–World framework, we develop an interaction-first, evidence- grounded data engine that reconstructs structured interactions before generating VQA. We select 4 Source videos 1 Local Interaction Extraction 2 Multimodal Evidence Grounding 3 Episode Aggregation & Dependency Linking 4 QA Generation & Validation Ego4D + HD-EPIC 00:03 00:29 Video + audio Narration + timestamps Narration as temporal anchors Attach multimodal evidence Group related interactions Episode Episodes + dependency links Select evidence + target structure Visual segment Audio segment Linked records Generate question + reference answer Node Edge Chain Loop Local records: Node / Edge Linked structures: Chain / Loop Validate and filter generated QA+ evidence Revise & Repair Evidence-grounded QA Single / multi-turn Open-ended / multiple-choice Visual / auditory / audio-visual extract local loops Action : Turn the page Ego Action World Entity: instruction_manual white; rectangular sheet Visual evidence new page visible Goal : Audio: paper rustling World change : Next page becomes visible View the next instructions Goal: Action: Change: Support: View the next instructions Turn the page Next page becomes visible Observed/inferred Visual Newly visible page Audio Paper rustling Entity Instruction manual I AW I AW I W Link supported dependencies Feedback adjustment State dependency & Human variation EgoOmniWorld Figure 3: Overview of the proposed data engine. Given source egocentric videos, the four stages ex- tract local Ego–Action–World interactions, ground them in multimodal evidence, aggregate episodes and link dependencies, and generate and validate evidence-grounded QA. activity-diverse source videos from Ego4D [9] and HD-EPIC [17], retaining their audio, narrations, and timestamps. Narration-based clustering promotes activity diversity in Ego4D, while official HD-EPIC annotations help identify candidate video segments. Narrations serve as temporal anchors rather than substitutes for evidence from the original videos. As illustrated in Figure 3, the data engine comprises four stages: 1) local interaction extraction, 2) multimodal evidence grounding, 3) episode aggregation and dependency linking, and 4) evidence- grounded QA generation and validation. Source references and timestamps are preserved throughout the pipeline, enabling each generated question to be traced back to its supporting observations. Detailed construction rules and prompt templates are provided in Appendix H. Step 1: Local Interaction Extraction. We first segment each video window into individual actions. For each action, we record the world state before it, the goal and current subgoal it serves, its outcome, and the feedback received by the wearer. A goal is recorded only when it is supported by speech, context, or later consequences. Failed and interrupted actions are retained, as they often reveal how feedback changes behavior. The output is a set of timestamped interaction records. Step 2: Multimodal Evidence Grounding. We then attach supporting evidence to each interaction record. Auditory evidence includes speech and action-related sounds, and visual evidence includes object properties, hand–object contacts, and scene layout. Each observation is timestamped and labeled with its modality. Entity identifiers link repeated observations of the same object across windows, while its states at different times are kept separate. For example, turning a page of a manual is grounded in both the newly visible page and the paper-rustling sound. These modality labels later determine the modality requirement of each question. Step 3: Episode Aggregation and Dependency Linking. Chain and Loop questions require con- nections across multiple interactions. We therefore group adjacent interactions that serve the same activity stage into task episodes. Two episodes are linked only when a state, instruction, failure, or unfinished task carries over from one to the other. Temporal adjacency or a shared object alone is not sufficient. Each link records the connected components, the transferred state, and the required modalities. The output is a set of episodes and dependency links, from which Chain and Loop questions are generated. Step 4: Evidence-Grounded QA Generation and Validation. QA generation follows an evidence- first principle: the supporting evidence and the target structure are selected before the question and answer are written. Node and Edge questions are generated from local interaction records, and Chain and Loop questions from linked episodes. In multi-turn QA, the final turn must combine earlier turns with new evidence, so it cannot be answered from the dialogue history alone. All candidates are then checked for evidential support, temporal consistency, modality correctness, answer leakage, and structural validity. Failed candidates are repaired using existing records only and then re-validated. A random subset of the final QA is further reviewed by humans (Appendix D). 5 (a) Clip duration (b) Reasoning levels (c) Interaction scopes(d) Dialogue length 335 56 392 398 1,042 5,782 2,659 02000400060008000 >180 s 150–180 s 120–150 s 90–120 s 60–90 s 30–60 s 0–30 s 41.5% 39.4% 10.3% 8.9% NodeEdgeChainLoop 4,057 779 2,151 1,481 771 282 2,331 5,439 2,256 0100020003000400050006000 World Percept ion Intent Formation Act ion Grounding Act ion Effects Intent Attribution Goal Planning World State Act ion Status Ego State 82.1% 7.2% 10.6% 1 turn2 turns3 turns4 turns (0.1%) TaskQuestion and answerEvidence / interpretationVideo frames Ego StateQ: Between 31.8 seconds and 35.8 seconds, what are my left and right hands holding, respectively? A: My left hand holds a small metal paint can with white paint, and my right hand holds a paintbrush stained with white paint. [32.31–36.11 s] Hand–object bindings distinguish the left and right hands. Action Status Q: What do my hands do with the paper sheet and scissors? A: I use both hands to cut the printed paper sheet with the scissors, snipping along its printed cut boundary. [25.25–27.45 s] Scissors cut along the printed boundary. The answer describes the manipulation. World State Q: At approximately 23.61 seconds, where was the water cup on the workbench located relative to the electric drill? A: The water cup is to the left of the drill. [24.20–30.20 s] The answer locates the cup relative to a named reference object, the drill. World Perception Q: When I look back up at the mirror in front of me, what task- relevant detail do I check about my mask? A: I check whether the mask is aligned over my nose and mouth. [15.00–20.44 s] The mirror provides feedback about mask placement over the nose and mouth. Chain (reloading a dry paint roller) T1 Q: While I painted the ceiling, what happened to the roller? T1 A: The paint ran out and the roller became dry. T2 Q: Given that condition, what did I do immediately afterward? T2 A: I dipped the roller into the paint bucket and rolled it against the grid to reload paint. [2.00–4.60 s] runs dry. [4.30–9.00 s] reloads. The second turn refers to the state established in the first. Loop (clearing a blocked water channel) T1 Q: What did I do with the hoe at the channel? T1 A: I pushed mud aside, slightly improving the water flow. T2 Q: What was the channel’s state after clearing? T2 A: Some mud blockage remained. T3 Q: Given that state, what did I do next? T3 A: I pushed the hoe into the channel again to clear more mud. [2.00–5.00 s] clearing. [4.50–7.00 s] remaining blockage and repeated action. The dialogue traces action, resulting state and a renewed action. One dialogue = one QA group. Duration: 10,664 unique clips. 920 records lack duration metadata and retain pending/rejected provenance. 46.88% 37.73% 15.39% Visual only (V) Audio + visual (A+V) Audio only (A) (e) Modality distribution Figure 4: Dataset statistics and representative QA examples in EgoOmniWorld. Top: distributions of (a) clip duration, (b) interaction scopes, (c) Node and Edge semantic categories, (d) dialogue length, and (e) required evidence modality. Bottom: representative questions, answers, supporting evidence, and video frames illustrating individual states, directed relations, linked dependencies, and feedback-closed interactions. Each complete multi-turn dialogue counts as one QA group. 3.3DATASET STATISTICS Figure 4 summarizes the statistics of EgoOmniWorld across interaction structures, semantic cate- gories, modality requirements, and dialogue formats. Dataset composition. EgoOmniWorld contains 24,166 QA groups, comprising 20,284 training and 3,882 test groups. Each single-turn QA or complete multi-turn dialogue is counted as one group. Interaction coverage. EgoOmniWorld covers all four interaction structures, with Node, Edge, Chain, and Loop accounting for 41.5%, 39.4%, 10.3%, and 8.9% of QA groups, respectively. The nine Node and Edge semantic categories further span diverse aspects of Ego–Action–World under- standing. Action Status and World Perception are the most frequent, while Goal Planning, Intent Attribution, and Intent Formation are less represented. Modality coverage. EgoOmniWorld naturally incorporates diverse modality requirements, with 46.9% of QA groups requiring visual evidence only, 15.4% requiring auditory evidence only, and 37.7% requiring joint audio-visual evidence. Importantly, modality labels reflect the evidence nec- essary to answer each question rather than the modalities available in the source video. Dialogue and temporal coverage. The dataset includes 19,833 single-turn groups (82.1%) and 4,333 multi-turn dialogues (17.9%). Among multi-turn dialogues, 1,750 contain two turns, 2,559 contain three turns, and 24 contain four turns. Duration metadata are available for 10,664 unique video clips, with 30–60-second clips forming the largest group. Visual examples spanning local states, directed relations, linked dependencies, and feedback-closed interactions are also demonstrated in Figure 4. More examples are provided in Appendix. 4EXPERIMENTS EgoOmniWorld serves as a unified testbed and training resource for egocentric interaction under- standing. We further investigate cross-benchmark transfer and analyze fine-grained capability gains and interaction coverage. 6 4.1EXPERIMENTAL SETUP Baseline models. We evaluate 11 open- and closed-source omni-modal MLLMs on EgoOmni- World. The three closed-source systems include Gemini 3.8 Flash [23], Qwen3.5-Omni-Plus [24], and Qwen3.8-Omni-Flash [24]. The eight open-source models include MiniCPM-o 2.6 (8B) [5], SALMONN2+ (7B and 72B) [22], VideoLLaMA2-AV-7B [4], Qwen2.5-Omni (3B and 7B) [25], and Qwen3-Omni (Thinking and Instruct variants) [24]. EgoOmniWorld training. To test whether the benchmark also works as supervision, we train two open-source models, Qwen2.5-Omni-7B and Qwen3-Omni-Thinking, on the EgoOmniWorld train- ing split. Each model is trained in two ways: LoRA, which adds low-rank adapters and leaves the base weights unchanged, and full fine-tuning (FFT), which updates the model weights directly. Training supervises the reference answers; in multi-turn dialogues, earlier turns are included and their answers are supervised as well. Videos are sampled at two frames per second with at most 64 frames and 200,704 pixels per frame, and the audio track is kept. Both regimes share the same schedule and precision settings; learning rates, adapter targets, and distributed settings are listed in Appendix E.1. In-domain inference protocol. All models are evaluated on the EgoOmniWorld test set. For multi- turn QA, models condition on their own previous responses rather than reference answers, thereby evaluating their ability to maintain consistency across dialogue turns. Predictive questions are re- stricted to observations available before the decision cutoff. All models are evaluated using the same reference answers and scoring rubric. Evaluation metrics and judge. We report answer accuracy (%) and mean score (0–5), both overall and by interaction category. A locally hosted Qwen3.8-27B serves as a text-only judge, receiving the question, reference answer, model response, and dialogue context when applicable, without access to the source video. The judge first determines binary correctness and then assigns an ordinal score from 0 to 5, with scores of 4–5 indicating correct answers and 0–3 indicating incorrect ones. Contradictory references and unresolved judgments are flagged for review. Judge prompts, parsing rules, and failure handling are detailed in Appendix I. Tansfer benchmarks. To evaluate cross-benchmark generalization, we compare Qwen2.5-Omni- 7B before and after EgoOmniWorld LoRA adaptation on three existing egocentric benchmarks: EgoSound [32], EgoTom [12], and EgoTempo [18]. No additional training is performed on these target benchmarks, and accuracy is reported using their respective evaluation protocols. 4.2IN-DOMAIN EVALUATION AND TRAINING We investigate EgoOmniWorld from two perspectives: its diagnostic value as a unified testbed for identifying capability gaps across Ego–Action–World interactions, and its effectiveness as a training resource for improving these capabilities. Table 2 presents overall results and performance across the four interaction structures, with fine-grained results for the nine semantic categories provided in Appendix F.1. Testbed analysis. EgoOmniWorld presents substantial challenges for current omni-modal MLLMs. The best closed-sourced model i.e, Gemini 3.8 achieves 61.51% overall accuracy, while the highest- performing open-source baseline reaches 48.15%. Among the evaluated 7B-scale models, the best result is only 36.42%, achieved by Qwen2.5-Omni-7B, highlighting the difficulty of the bench- mark. Beyond overall accuracy, EgoOmniWorld reveals substantial performance variations across interaction structures and semantic categories. In particular, models exhibit different capabilities in recognizing actions, perceiving world states, interpreting intentions, and reasoning about action effects. These variations highlight the diagnostic value of EgoOmniWorld in revealing capability gaps across different components and relationships of the Ego–Action–World interaction process. Training effectiveness. Training on EgoOmniWorld yields substantial overall improvements and broad gains across interaction structures. For Qwen2.5-Omni-7B, LoRA and full fine-tuning (FFT) improve overall accuracy from 36.42% to 47.36% and 48.34%, respectively, with improvements across all four structures and particularly pronounced gains on Edge questions (+13.83 and +15.35 percentage points). Similarly, Qwen3-Omni-Thinking improves from 48.15% to 52.10% with LoRA and 53.18% with FFT, with the latter improving all four structures. The gains vary across models and adaptation strategies: Qwen2.5-Omni-7B benefits more substantially overall, whereas Qwen3- 7 Omni-Thinking shows smaller overall gains, with its LoRA variant slightly reducing Chain and Loop accuracy. These results demonstrate the effectiveness of EgoOmniWorld as interaction-centric supervision while highlighting differences in how model families and training strategies benefit from different interaction structures. Table 2: In-domain performance on EgoOmniWorld. We report mean score (0–5) and accuracy (%) across the four interaction structures and overall. ModelNodeEdgeChainLoopOverall Closed-source models Gemini 3.8 Flash2.86/54.74 3.03/60.26 3.34/66.41 3.54/69.29 3.14/61.51 Qwen3.5-Omni-Plus2.18/37.31 2.64/50.76 2.80/56.25 2.90/57.89 2.59/49.15 Qwen3.8-Omni-Flash2.69/48.62 2.91/56.09 3.06/57.48 3.03/58.27 2.89/54.58 Open-source models MiniCPM-o 2.6 8B2.09/30.28 2.01/34.19 2.25/38.28 2.21/37.83 2.11/34.38 SALMONN2+ 72B2.16/36.09 2.45/44.87 2.80/56.25 2.67/50.19 2.46/44.87 SALMONN2+ 7B1.90/30.89 2.17/39.49 2.02/33.59 2.08/35.34 2.05/35.28 VideoLLaMA2 7B AV1.61/22.94 1.56/26.92 1.22/16.41 1.24/19.48 1.46/22.75 Qwen2.5-Omni-3B1.97/31.19 2.28/42.28 2.09/34.38 2.07/34.08 2.12/36.17 Qwen2.5-Omni-7B (base)1.90/32.821.97/37.312.26/42.371.93/36.881.97/36.42 Qwen3-Omni-Instruct2.29/40.06 2.51/47.09 2.59/48.44 2.70/52.06 2.50/46.37 Qwen3-Omni-Thinking (base)2.38/42.022.50/46.682.80/53.232.94/55.472.60/48.15 Trained on EgoOmniWorld Qwen2.5-Omni-7B-LoRA2.58/44.042.79/51.142.70/50.782.48/44.192.65/47.36 Qwen2.5-Omni-7B-FFT2.47/41.592.76/52.662.69/48.442.72/50.192.66/48.34 Qwen3-Omni-Thinking-LoRA2.71/48.932.83/54.182.90/52.342.94/52.812.83/52.10 Qwen3-Omni-Thinking-FFT2.64/45.572.90/55.193.12/60.942.96/55.812.86/53.18 4.3CROSS-BENCHMARK TRANSFER Zero-shot transfer. We evaluate the transferability of EgoOmniWorld supervision by comparing Qwen2.5-Omni-7B before and after LoRA adaptation on three external egocentric benchmarks: EgoSound, EgoToM, and EgoTempo, without additional training on their respective training sets. As shown in Table 3, adaptation improves accuracy from 31.33% to 39.41% on EgoSound, from 55.95% to 58.22% on EgoToM, and from 20.40% to 24.80% on EgoTempo, corresponding to gains of 8.08, 2.27, and 4.40 percentage points, respectively. These results suggest that the benefits of EgoOmniWorld supervision extend beyond its own benchmark, particularly to audio-visual multi- modal egocentric understanding. Table 3: Cross-benchmark transfer accuracy (%) on existing egocentric benchmarks. Bold high- lights the improved results within our base–LoRA pair. See Appendix E.2 for evaluation details. ModelEgoSound EgoToM EgoTempo VideoLLaMA2 7B AV20.5– SALMONN2+ 7B36.0–23.60 MiniCPM-o 2.6-8B40.457.5120.60 Qwen2.5-Omni-7B (Base)31.3355.9520.40 Qwen2.5-Omni-7B + EgoOmniWorld-LoRA39.4158.2224.80 Improvement+8.08+2.27+4.40 Fine-grained transfer analysis. To identify which interaction capabilities benefit from adaptation, we map questions from the three transfer benchmarks to our Ego–Action–World taxonomy (Fig- ure 5). On EgoSound, the largest improvement occurs in Ego State, with accuracy increasing from approximately 40% to 60%. Substantial gains are also observed in Action Effects, World Perception, and Intent Formation, suggesting benefits across ego-state understanding, perceptual grounding, and action-related reasoning. On EgoToM, improvements are more modest, but with consistent gains in Ego State, Action Status, and Intent Attribution. On EgoTempo, improvements are observed in 8 Action Status and Goal Planning. Overall, these results show that the benefits of EgoOmniWorld supervision extend across multiple Node and Edge categories. Fragmented interaction coverage. Beyond transfer performance, our taxonomy mapping further reveals the fragmented coverage of existing egocentric benchmarks. Specifically, EgoSound, Ego- ToM, and EgoTempo each cover only a subset of the Ego–Action–World interaction process, with no explicit evaluation of Chain or Loop in the mapped questions. This highlights the limited coverage of cross-interaction dependencies and feedback-driven adaptation in the benchmarks examined. In contrast, EgoOmniWorld systematically covers all four complementary interaction structures, pro- viding a unified framework for evaluating individual states, directed relations, linked dependencies, and complete Ego–Action–World feedback loops. Accuracy (%) 0.0 25.0 50.0 75.0 100.0 EgoSound EgoToM EgoTempo Ego State Action Status World State Goal Planning Intent Attribution Action Effects Action Grounding Intent Formation World Perception Chain Loop Node Edge Chain Loop 40 60 37 39 34 3837 39 15 18 4848 31 3842 43 9192 20 32 23 33 26 38 100 Figure 5:Fine-grained cross-benchmark transfer analysis under the EgoOmniWorld taxonomy. We compare Qwen2.5-Omni-7B before (solid bars) and after (hatched bars) EgoOmniWorld LoRA adaptation on EgoSound, EgoToM, and EgoTempo. Colors distinguish benchmarks, and accuracy is computed on matched questions within each dataset–category group. Empty categories indicate no evaluated questions, and groups with fewer than three questions are omitted. 5CONCLUSION AND LIMITATIONS We introduced EgoOmniWorld, a unified benchmark that organizes diverse egocentric video un- derstanding capabilities around the closed Ego–Action–World interaction loop. By structuring this process into four complementary scopes—Node, Edge, Chain, and Loop—we provide a common framework for evaluating individual states, directed relations, linked dependencies, and feedback- closed interactions. Our interaction-first, evidence-grounded data engine further integrates visual and auditory evidence and single- and multi-turn QA according to the underlying interaction require- ments. Extensive experiments reveal uneven capabilities across existing MLLMs and demonstrate that training on EgoOmniWorld improves in-domain performance and transfers to other egocentric benchmarks. We hope EgoOmniWorld provides a foundation for moving beyond fragmented capa- bility evaluation toward more systematic and holistic understanding of human–world interactions. Limitations and Future Work. While EgoOmniWorld provides a unified framework for egocentric video understanding, several directions remain for future exploration. First, our current dataset is constructed from Ego4D and HD-EPIC. Incorporating more diverse data sources, longer recordings, and richer annotations could further expand its coverage of interaction scenarios and temporal de- pendencies, potentially improving model generalization and real-world applicability. Second, our training experiments primarily focus on the Qwen-Omni family. Extending EgoOmniWorld supervi- sion to a broader range of omni-modal architectures and training paradigms would further investigate its transferability and benefits for general-purpose egocentric assistants. AI USE STATEMENT We used large language models (LLMs) throughout the benchmark construction pipeline to extract structured interactions, ground multimodal evidence, aggregate related episodes, and generate, val- 9 idate, and repair question–answer pairs (Section 3.2). LLMs were also employed as answer judges during evaluation (Section 4.1), while a random subset of the test set underwent manual review (Appendix D). AI-assisted tools were additionally used to polish the paper writing. The authors reviewed the AI-assisted outputs and take full responsibility for the final manuscript, dataset, and reported findings. ETHICS STATEMENT EgoOmniWorld is constructed from existing Ego4D and HD-EPIC recordings without collecting new participant videos. As egocentric recordings may contain identifiable individuals and private environments, access to and redistribution of the source videos must comply with the respective datasets’ licenses and access conditions. EgoOmniWorld is intended for research on egocentric interaction understanding. Potential misuse includes unauthorized surveillance or profiling of indi- viduals based on first-person recordings. Benchmark performance alone should not be interpreted as evidence of suitability for deployment in privacy-sensitive settings. REPRODUCIBILITY STATEMENT Section 3.2 describes the benchmark construction pipeline, with implementation details and prompt templates provided in Appendices B and H. Section 4.1 details the evaluated models, training settings, inference protocols, and evaluation metrics. Additional training configurations, judge prompts, and evaluation protocols are documented in Appendices E.1, I, and E.2, respectively. We will release the benchmark data, construction and evaluation code, and trained models. REFERENCES [1] Qirui Chen, Shangzhe Di, and Weidi Xie. Grounded multi-hop videoqa in long-form egocentric videos. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, p. 2159– 2167, 2025. [2] Yi Chen, Yuying Ge, Yixiao Ge, Mingyu Ding, Bohao Li, Rui Wang, Ruifeng Xu, Ying Shan, and Xihui Liu. Egoplan-bench: Benchmarking multimodal large language models for human- level planning. International Journal of Computer Vision, 134(3):118, 2026. [3] Sijie Cheng, Kechen Fang, Yangyang Yu, Sicheng Zhou, Bohao Li, Ye Tian, Tingguang Li, Lei Han, and Yang Liu. Videgothink: Assessing egocentric video understanding capabilities for embodied ai. arXiv preprint arXiv:2410.11623, 2024. [4] Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial-temporal mod- eling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476, 2024. [5] Junbo Cui, Bokai Xu, Chongyi Wang, Tianyu Yu, Weiyue Sun, Yingjing Xu, Tianran Wang, Zhihui He, Wenshuo Ma, Tianchi Cai, et al. Minicpm-o 4.5: Towards real-time full-duplex omni-modal interaction. arXiv preprint arXiv:2604.27393, 2026. [6] Mingkang Dong, Hongyi Cai, Jie Li, Sifan Zhou, Bin Ren, Kunyu Peng, and Yuqian Fu. Vis- nec: Measuring and leveraging visual necessity for multimodal instruction tuning. In European Conference on Computer Vision, p. 531–548. Springer, 2026. [7] Yuqian Fu, Runze Wang, Bin Ren, Guolei Sun, Biao Gong, Yanwei Fu, Danda Pani Paudel, Xuanjing Huang, and Luc Van Gool. Objectrelator: Enabling cross-view object relation un- derstanding across ego-centric and exo-centric perspectives. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), p. 6530–6540. IEEE, 2025. [8] Hengjian Gao, Kaiwei Zhang, Shibo Wang, Mingjie Chen, Qihang Cao, Xianfeng Wang, Yucheng Zhu, Xiongkuo Min, Wei Sun, Dandan Zhu, et al. Lifeeval: A multimodal benchmark for assistive ai in egocentric daily life tasks. arXiv preprint arXiv:2603.00490, 2026. 10 [9] Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 18995–19012, 2022. [10] Baoxiong Jia, Ting Lei, Song-Chun Zhu, and Siyuan Huang. Egotaskqa: Understanding human tasks in egocentric videos. Advances in Neural Information Processing Systems, 35:3343– 3360, 2022. [11] Yanjun Li, Yuqian Fu, Tianwen Qian, Qi’ao Xu, Silong Dai, Danda Pani Paudel, Luc Van Gool, and Xiaoling Wang. Egocross: Benchmarking multimodal large language models for cross- domain egocentric video question answering. In Proceedings of the AAAI Conference on Arti- ficial Intelligence, volume 40, p. 6592–6600, 2026. [12] Yuxuan Li, Vijay Veerabadran, Michael L Iuzzolino, Brett D Roads, Asli Celikyilmaz, and Karl Ridgeway. Egotom: Benchmarking theory of mind reasoning from egocentric videos. arXiv preprint arXiv:2503.22152, 2025. [13] Mohammad Mahdi, Yuqian Fu, Nedko Savov, Jiancheng Pan, Danda Pani Paudel, and Luc Van Gool. Exo2egosyn: Unlocking foundation video generation models for exocentric-to- egocentric video synthesis. arXiv preprint arXiv:2511.20186, 2025. [14] Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. Advances in Neural Information Processing Systems, 36:46212–46244, 2023. [15] Jiancheng Pan, Runze Wang, Tianwen Qian, Mohammad Mahdi, Yanwei Fu, Xiangyang Xue, Xiaomeng Huang, Luc Van Gool, Danda Pani Paudel, and Yuqian Fu. Vˆ 2-sam: Marrying sam2 with multi-prompt experts for cross-view object correspondence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 16910–16919, 2026. [16] Ye Pan, Chi Kit Wong, Yuanhuiyi Lyu, Hanqian Li, Jiahao Huo, Jiacheng Chen, Lutao Jiang, Xu Zheng, and Xuming Hu. Egointent: An egocentric step-level benchmark for understanding what, why, and next. arXiv preprint arXiv:2603.12147, 2026. [17] Toby Perrett, Ahmad Darkhalil, Saptarshi Sinha, Omar Emara, Sam Pollard, Kranti Kumar Parida, Kaiting Liu, Prajwal Gatti, Siddhant Bansal, Kevin Flanagan, et al. Hd-epic: A highly- detailed egocentric video dataset. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 23901–23913. IEEE, 2025. [18] Chiara Plizzari, Alessio Tonioni, Yongqin Xian, Achin Kulshrestha, and Federico Tombari. Omnia de EgoTempo: Benchmarking temporal understanding of multi-modal LLMs in ego- centric videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025. URL https://arxiv.org/abs/2503.13646. [19] Chiara Plizzari, Alessio Tonioni, Yongqin Xian, Achin Kulshrestha, and Federico Tombari. Omnia de egotempo: Benchmarking temporal understanding of multi-modal llms in egocentric videos. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 24129–24138. IEEE, 2025. [20] Ivan Rodin, Tz-Ying Wu, Kyle Min, Sharath Nittur Sridhar, Antonino Furnari, Subarna Tri- pathi, and Giovanni Maria Farinella. Easg-bench: Video q&a benchmark with egocentric action scene graphs. In 2025 IEEE/CVF International Conference on Computer Vision Work- shops (ICCVW), p. 2732–2737. IEEE, 2025. [21] Ashish Seth, Xinhao Mei, Changsheng Zhao, Varun Nagaraja, Ernie Chang, Gregory P. Meyer, Gael Le Lan, Yunyang Xiong, Vikas Chandra, Yangyang Shi, Dinesh Manocha, and Zhipeng Cai. EgoAVU: Egocentric audio-visual understanding, 2026. URL https://arxiv.org/ abs/2602.06139. [22] Guangzhi Sun, Yudong Yang, Jimin Zhuang, Changli Tang, Yixuan Li, Wei Li, Zejun Ma, and Chao Zhang. video-salmonn-o1: Reasoning-enhanced audio-visual large language model. arXiv preprint arXiv:2502.11775, 2025. 11 [23] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Sori- cut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. [24] Qwen Team. Qwen3. 5-omni technical report. arXiv preprint arXiv:2604.15804, 2026. [25] Jin Xu, Zhifang Guo, Jinzheng He, Hang-Rui Hu, Ting He, Shuai Bai, Ke qin Chen, Jia- Lin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. Qwen2.5-omni technical report.ArXiv, abs/2503.20215, 2025.URL https://api. semanticscholar.org/CorpusID:277322543. [26] Qi’ao Xu, Tianwen Qian, Yuqian Fu, Kailing Li, Yang Jiao, Jiacheng Zhang, Xiaoling Wang, and Liang He. Tog-bench: Task-oriented spatio-temporal grounding in egocentric videos. arXiv preprint arXiv:2512.03666, 2025. [27] Jingkang Yang, Shuai Liu, Hongming Guo, Yuhao Dong, Xiamengwei Zhang, Sicheng Zhang, Pengyun Wang, Zitang Zhou, Binzhu Xie, Ziyue Wang, et al. Egolife: Towards egocentric life assistant. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 28885–28900. IEEE, 2025. [28] Hanrong Ye, Haotian Zhang, Erik Daxberger, Lin Chen, Zongyu Lin, Yanghao Li, Bowen Zhang, Haoxuan You, Dan Xu, Zhe Gan, et al. Mmego: Towards building egocentric multi- modal llms for video qa. In International Conference on Learning Representations, volume 2025, p. 71705–71723, 2025. [29] Deheng Zhang, Yuqian Fu, Runyi Yang, Yang Miao, Tianwen Qian, Xu Zheng, Guolei Sun, Ajad Chhatkuli, Xuanjing Huang, Yu-Gang Jiang, et al. Egonight: Towards egocentric vision understanding at night with a challenging benchmark. In International Conference on Learning Representations, volume 2026, p. 887–901, 2026. [30] Ganlin Zhang, Deheng Zhang, Longteng Duan, Guo Han, Yuqian Fu, Danda Pani Paudel, Luc Van Gool, and Eric Vollenweider. Egospot: Egocentric multimodal control for hands-free mobile manipulation. arXiv preprint arXiv:2306.02393, 2023. [31] Sheng Zhou, Junbin Xiao, Qingyun Li, Yicong Li, Xun Yang, Dan Guo, Meng Wang, Tat-Seng Chua, and Angela Yao. Egotextvqa: Towards egocentric scene-text aware video question an- swering. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 3363–3373. IEEE, 2025. [32] Bingwen Zhu, Yuqian Fu, Qiaole Dong, Guolei Sun, Tianwen Qian, Yuzheng Wu, Danda Pani Paudel, Yanwei Fu, and Xiangyang Xue. Egosound: Benchmarking sound understanding in egocentric videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, p. 25589–25598, 2026. 12 ADATASET STATISTICS AND REPRESENTATIVE EXAMPLES This section reports dataset composition, temporal coverage, and dialogue statistics, followed by representative QA examples. A single-turn QA or a complete multi-turn dialogue counts as one QA group throughout. Scale and task distribution. EgoOmniWorld contains 24,166 QA groups, comprising 20,284 train- ing and 3,882 test samples. A QA group is either a standalone question or a complete multi-turn dialogue. Node, Edge, Chain, and Loop contain 10,026, 9,521, 2,480, and 2,139 samples, respec- tively, representing 41.49%, 39.40%, 10.26%, and 8.85% of the total. Node covers ego states, action dynamics, and world states, while Edge covers six directed interaction relations. Chain and Loop assess open relational paths and interactions with feedback closure, respectively (Table 4; Figure 4). Table 4: Sample counts by task category and split. A multi-turn dialogue counts as one QA group. LevelCategoryTrainTestTotal NodeEgo State1,9113452,256 Action Status4,5498905,439 World State1,9663652,331 Subtotal8,4261,60010,026 EdgeGoal Planning24240282 Intent Attribution644127771 Action Effects1,2462351,481 Action Grounding1,8173342,151 Intent Formation68198779 World Perception3,3876704,057 Subtotal8,0171,5049,521 Chain2,1713092,480 Loop1,6704692,139 Total20,2843,88224,166 Video duration. Duration statistics cover the 23,246 samples with clip metadata, excluding the 920 restored samples for which new clips have not been generated. Deduplication by source video ID and temporal interval yields 10,664 clips: 8,892 in training and 1,772 in testing. According to the clip metadata, the mean and median durations are 57.2 and 34.5 seconds; 92.7% of clips are no longer than two minutes, and the longest lasts approximately 46.7 minutes. When weighted by QA groups within this subset, retaining repeated references to the same clip, the mean duration is 71.5 seconds and the median is 54.1 seconds. Samples associated with 30–60-second and 60–120-second clips account for 44.7% and 28.8%, respectively. The data therefore primarily cover short interactions while retaining samples that require understanding over longer temporal spans (Figure 4). Question formats and dialogue settings. The dataset supports multiple-choice and open-ended questions. It contains 19,833 single-turn groups and 4,333 multi-turn dialogues, representing 82.07% and 17.93%, respectively. All multi-turn dialogues use open-ended answers. Multi-turn dialogue length. The 4,333 multi-turn dialogues comprise 11,273 QA turns, averag- ing 2.60 turns per dialogue. Two-turn dialogues account for 1,750 samples (40.39%), three-turn dialogues for 2,559 (59.06%), and four-turn dialogues for 24 (0.55%). The data are dominated by compact two- and three-turn dialogues, which use successive questions to examine understanding of related facts and reasoning dependencies (Figure 4). A.1REPRESENTATIVE QA EXAMPLES Figure 6 presents 22 examples: two for each of the nine Node and Edge semantic categories, two Chain dialogues, and two Loop dialogues. The examples pair questions and reference answers with frames from the corresponding clips. Chain examples expose dependencies between earlier and later events; Loop examples show how an observed outcome informs a subsequent action. Audio-visual examples additionally refer to sounds that cannot be conveyed by still frames alone. 13 EgoOmniWorld: Question & Answer Examples Two examples per semantic category • Two Chain dialogues • Two feedback-closed Loop dialogues CATEGORYEXAMPLE 1EXAMPLE 2 NODE Ego State 01 · VISUAL 34.0 s Q: What are my left and right hands holding? A: Left: a small can of white paint. Right: a paintbrush stained with white paint. 02 · VISUAL 8.0 s Q: What is my right hand holding at the beginning, and where? A: A frying pan under the faucet. NODE Action Status 03 · VISUAL 47.0 s Q: What do I do with the roller? A: Roll it up and down the wall with my right hand. 04 · VISUAL 26.0 s Q: What do my hands do with the paper sheet and scissors? A: Cut the printed paper along its printed boundary. NODE World State 05 · VISUAL 26.0 s Q: Where is the water cup relative to the electric drill? A: To the left of the drill. 06 · VISUAL 27.0 s Q: What color and material is the Kindle case? A: Maroon faux leather. EDGE Goal Planning 07 · VISUAL 27.0 s Q: With the brush loaded and lifted toward the paper, what should I do next? A: Apply the paint to the paper. 08 · VISUAL 25.0 s Q: What should I do next with the remaining clothes in the basket? A: Hang another piece of cloth on the clothesline. EDGE Intent Attribution 09 · VISUAL 16.0 s Q: Why touch the nearly empty brush to the palette? A: To pick up dark pigment. 10 · VISUAL 14.0 s Q: What am I trying to do to get the PC case open? A: Unscrew and remove the glass side panel. EDGE Action Effects 11 · AUDIO + VISUAL 37.0 s Q: What do I see and hear as I cut along the printed guideline? A: The paper separates, with snapping and slicing sounds. 12 · AUDIO + VISUAL 5.0 s Q: After I plug in the power cord, what visual and sound cues occur? A: The plug is seated in the outlet, and a click is heard. EDGE Action Grounding 13 · VISUAL 28.0 s Q: What obstructs my view, making me lower my head to see the road? A: The lowered black sun visor. 14 · VISUAL 45.0 s Q: What lets me drop the peeled leaves directly into the basin? A: The branches are above the basin and within reach. EDGE Intent Formation 15 · VISUAL 13.0 s Q: What am I trying to do after seeing the loose screw? A: Secure the card holder by tightening it. 16 · VISUAL 6.0 s Q: How does the missing stem below the flower affect my next action? A: I draw a downward curve to add a stem. EDGE World Perception 17 · VISUAL 18.0 s Q: What do I check about my mask when looking in the mirror? A: Whether it covers my nose and mouth correctly. 18 · VISUAL 31.0 s Q: What do I learn by looking at the mower’s bag mounting slot? A: The grass bag is seated on the mower hooks. CHAIN Linked Dependencies 19 · VISUAL 8.0 s16.0 s231.0 s Q1: What did I do with the hot-oil pot? A1: Moved it off the stove onto the ground. Q2: What changed? A2: The stove was freed. Q3: Why move it, given the later pan placement? A3: To make room for a frying pan to heat dough. Move pot → free stove → interpret later purpose 20 · VISUAL 3.0 s18.0 s21.0 s Q1: Why arrange the plastic around the post? A1: It was wrinkled and needed flattening. Q2: What did I pick up next? A2: A brick. Q3: Where did I put it, and why? A3: On the plastic to hold it in place. Flatten plastic → pick up brick → secure plastic LOOP Feedback & Adjustment 21 · VISUAL 2.5 s4.5 s6.5 s Q1: What did I do with the hoe? A1: Pushed mud aside to improve the flow. Q2: What remained? A2: Some mud blockage. Q3: Given that state, what did I do next? A3: Pushed the hoe in again to clear more mud. Clear mud → remaining blockage → clear again 22 · VISUAL 2.5 s4.0 s5.5 s Q1: How did I prune the branch? A1: Cut it with loppers. Q2: Did it fall? A2: No; it remained entangled. Q3: How did I then remove it completely? A3: Pulled the cut branch out of the bush. Cut branch → still entangled → pull it out Dataset QA excerpts; multi-turn examples translated from Chinese and shortened. Timestamps are clip-relative. GP frames show pre-answer context. Images are real dataset frames. Audio claims follow the paired audio–visual reference annotations. Full source records are retained in slide notes. Figure 6: Representative QA examples in EgoOmniWorld. The figure contains two examples for each of Ego State, Action Status, World State, Goal Planning, Intent Attribution, Action Effects, Action Grounding, Intent Formation, and World Perception, together with two Chain and two Loop dialogues. Frames come from the corresponding dataset clips; timestamps are clip-relative. Dis- played questions and answers are shortened for readability, and Chinese multi-turn examples are translated into English. Each complete dialogue counts as one QA group. The examples illustrate the annotations rather than establish exhaustive validation of their assigned categories. 14 BDATA CONSTRUCTION DETAILS This appendix provides implementation details omitted from the main text, including video selec- tion, interaction representations, category-specific QA generation, structural validation, constrained repair, and evidence provenance. The organization follows the interaction-first pipeline in Sec- tion 3.2. B.1SOURCE VIDEOS AND PROVENANCE Ego4D source selection uses narration-based clustering to promote activity diversity; the implemen- tation and observed coverage appear below. HD-EPIC segments use official narrations and action intervals for candidate localization. Narrations are temporal anchors and provenance references, not substitutes for supporting evidence from the original audio and video. B.2INTERACTION REPRESENTATION DETAILS Local interaction components. For each candidate interaction, the engine extracts a structured record that includes the following components when supported by evidence: • the pre-action world state; • observations that trigger or contextualize the interaction; • the current intention or persistent activity goal; • the current decision or intermediate subgoal; • the action actually performed; • the resulting change in world state; • perceptual or task-level feedback; and • attention-related information when sufficiently supported. We distinguish these components by their functional roles. An intention is a relatively persistent goal, a subgoal is the currently selected intermediate step, and an action is an observable behavior that has been performed. In particular, a paraphrase of what the wearer is currently doing is not automatically an intention description. For latent components such as intentions and decisions, records also preserve supporting observa- tions, speech, or contextual information. Unsupported latent states remain unspecified rather than being completed through commonsense inference. Multimodal perceptual enrichment. Visual evidence may describe object identities and attributes, spatial layouts, hand occupancy, hand–object interactions, the wearer’s surroundings, visible state changes, and changes in location. Auditory evidence may include speech, environmental sounds, action-generated sounds, sound-source identities, and relative source directions when available. Each perceptual record is associated with its temporal interval and modality. When visual and auditory observations are considered jointly, the pipeline also records their correspondence within the interaction instead of assuming that temporal proximity implies semantic relevance. Component relations. The resulting local representation is a component graph whose nodes repre- sent interaction components and whose edges represent candidate relations. Connectivity is treated as a structural hypothesis, not proof of causality. Relations used for QA generation must additionally pass the evidence checks for the relevant category. B.3ENTITY ASSOCIATION AND EPISODE DEPENDENCIES Object identity association. Object observations from different local interactions are associated only when they can reliably be identified as referring to the same physical entity. The resulting track stores observations in temporal order, including their states, locations, associated interactions, timestamps, and evidence sources. 15 Only entity identity is merged; states observed at different times remain separate records. A track therefore preserves successive changes in an object’s state without collapsing temporally distinct observations into a single description. When identity cannot be reliably established, records remain separate. Similar names, shared se- mantic categories, or temporal proximity are insufficient grounds for merging. Macro-level task aggregation. Local interactions are processed in temporally overlapping win- dows. Adjacent interactions that serve the same identifiable activity stage are grouped into a macro- level task episode. A new episode is created when the goal, principal object set, workspace, or activity changes substantially. Each macro episode preserves its source events and, where recorded, its persistent intention, current subgoals, initial and resulting states, key entities, instructions, constraints, failures, interruptions, and unfinished tasks. Episodes produced by overlapping windows are then reconciled. Duplicate episode descriptions are merged while preserving temporal order and event provenance. Cross-episode relations. Candidate relations between macro episodes include state transfer, in- struction following, failure correction, resumption of unfinished tasks, environmental constraints, and continuation of persistent goals as subgoals change. Each relation stores its source and target episodes, relevant source events, connected interaction components, relation type, transferred information or state, required modalities, and uncertainty metadata. Relations with unresolved entity identity, state transfer, or semantic dependence are not used directly to generate questions. B.4EGO4D CLUSTERING AND SOURCE SELECTION The statistics below concern source-video selection, not QA counts. They match saved clustering outputs to the final selected video identifiers without rerunning clustering. Video-level narration representation and clustering. The clustering implementation combines summaries and action descriptions from the two available rounds of narrations for each video into a video-level text representation. Text cleaning removes markup, normalizes whitespace, and dedu- plicates repeated descriptions before concatenating summaries and action descriptions. TF–IDF features, truncated SVD, L2 normalization, and MiniBatch K-Means then group videos with similar narration content. The original clustering summary records the method as tfidf_svd_minibatch_kmeans, with random seed 42, covering 8,931 videos and 160 clusters. Each video retains its original identifier, cluster index, and distance to the corresponding cluster center. Cluster summaries also store frequent keywords and representative videos near the center to facilitate inspection and topic interpretation. Keywords aid interpretation; they are neither human activity labels nor validated semantic cate- gories. Candidate filtering and within-cluster ranking. The sampler first selects candidates with com- plete annotation status, valid video identifiers, and durations within the specified range. Importantly, duration_proxy_sec uses the maximum timestamp across the two narration rounds as a du- ration proxy rather than directly reading the media file’s duration. The filter therefore measures the temporal span covered by narrations. The sampling budget first provides minimum coverage quotas for available clusters, then allocates additional places according to remaining candidate capacity, never exceeding a cluster’s capacity. Candidates are ranked primarily by ascending distance to the center, with descending narration density as a tie-breaker and video identifiers providing deterministic ordering. Narration density is computed as the narration count divided by the duration proxy floored at 60 seconds. The saved sampling script defaults to a total budget of 1,000, a minimum quota of four per cluster, and a duration-proxy range of 180–3,600 seconds. These are verifiable script defaults, not confirmed historical execution parameters: the complete parameter record for the original run was not found during this audit. The observed distribution below comes directly from the final selected list and does not depend on assumptions about run parameters. 16 Observed selection results and coverage. Table 5: Observed Ego4D source-video selection statistics. StatisticValue Videos in the original clustering8,931 Original clusters160 Selected source videos (deduplicated)1,000 Selected videos matched to original clusters1,000 Clusters covered159 Cluster coverage99.375% Minimum/maximum selections per covered cluster2/12 Clusters with five or six selected videos102 Cluster 39 is the only cluster with no selected video. It originally contains a single video whose narration-based duration proxy is approximately 92.3 seconds. This is consistent with the current script’s default exclusion of candidates below 180 seconds, but the historical exclusion reason cannot be established from default parameters alone. Clusters 7 and 112 have the most selected videos, with 12 each. Cluster-stratified sampling aims to reduce budget domination by frequent activities under sampling based only on video counts, balancing activity coverage and within-cluster representativeness. It is not an unbiased estimate of the natural activity distribution and does not guarantee coverage of every fine-grained activity. Center-based ranking may reduce the inclusion probability of peripheral or rare activities; variations in narration wording may also distribute similar activities across multiple clusters. Coverage of 159/160 therefore refers only to the original clustering and does not mean that 159 mutually exclusive human-defined activity categories are covered. Cluster topics and selected-video distribution. The following table preserves the original cluster indices and counts. Topic labels are translated from the Chinese summaries derived from original keywords and representative narrations; they are provided for readability, not as original human annotations. Heterogeneous clusters are marked “mixed.” For example, crochet, yarn crochet, and hook-based crochet may occur in different clusters without constituting mutually exclusive activity categories. “All” denotes the original cluster size, while “Selected” denotes the number of videos from the final 1,000-video subset in that cluster. The first six keywords are retained from the original summaries. Table 6: Original Ego4D clusters and selected source videos. Topic summaries are interpretive, not human-annotated activity classes. ID TopicAll Selected Original keywords (first six) 0 Crochet105 yarn, fabric, hand, crochet, left, right 1 Kitchen weighing and con- tainer washing 114 hand, table, sink, jug, summary, kitchen 2 Paper crafts and gluing446 paper, craft, table, hand, right, drops 3 Multiplayer card games14810 man, table, hand, cards, right, card 4 Reading and bookshelf orga- nization 456 book, books, shelf, summary, floor, phone 5 Supermarket shopping9510 shopping, basket, man, woman, bag, pa- per 6 Knitting585 hand, yarn, knitting, right, fabric, knit- ted 7 Indoor manual activities and chores (mixed) 21412 hand, right, left, hands, bamboo, sum- mary 8 Painting and coloring on a board 757 paint, brush, hand, painting, board, right 9 Metal-part assembly and fas- tening 375 box, unsure, screw, metal, table, hand 10 Writing notes445 book, hand, table, right, pen, left 17 ID TopicAll Selected Original keywords (first six) 11 Clearingbranchesfrom fences 245 hand, left, right, twig, wire, pulls 12 Woodwork and floor installa- tion 867 wood, hand, floor, tape, tile, right 13 Bicycle tire repair and infla- tion 95 bicycle, tire, pump, tyre, hand, wheel 14 Pencil drawing and sketching627 hand, pencil, paper, table, book, right 15 Gardening and pruning796 hand, plant, right, ground, left, flower 16 Hand sewing and embroidery526 thread, hand, needle, cloth, right, sum- mary 17 Browsing and talking in shops125 woman, man, magazine, summary, phone, unsure 18 Dog walking677 dog, hand, leash, road, left, right 19 Cookingandingredient preparation 556 hand, right, left, kitchen, spoon, drops 20 Walking on roads425 road, walked, summary, hand, man, woman 21 Drum-kit playing696 drum, hand, right, left, set, hands 22 Motorcycle and mechanical- part repair 11110 hand, right, left, screw, table, hands 23 Outdoor climbing and rope handling 485 man, rope, mountain, hand, rock, hands 24 Card and board games1319 cards, man, woman, table, card, sum- mary 25 Washing and hanging laundry626 machine, washing, cloth, clothes, laun- dry, hand 26 Driving708 car, hand, drives, right, left, past 27 Lawnmower engine repair1098 hand, mower, lawn, right, left, unsure 28 Grass cutting with handheld trimmers 556 grass, hand, cutter, trimmer, left, right 29 Reading books768 book, hand, summary, page, read, right 30 Dishwashingandutensil cleaning 415 sink, plate, hand, sponge, kitchen, drops 31 Glue-gun and cord crafts395 hand, right, left, glue, floor, gun 32 Shop browsing and conversa- tion 9510 man, woman, phone, store, summary, shelf 33 Kitchen retrieval and daily ac- tivities (mixed) 1109 unsure, kitchen, opens, fridge, sum- mary, cabinet 34 Stirring food and mixing dough 565 hand, right, left, bowl, pot, drops 35 Peeling and cutting produce526 hand, right, left, tray, knife, drops 36 Dominoes and indoor activi- ties (mixed) 556 man, table, hand, game, summary, board 37 Wall painting with rollers and brushes 10610 paint, hand, brush, roller, right, wall 38 Flatbread preparation and pan cooking 455 dough, pan, hand, spoon, oil, right 39 Carpet placement and adjust- ment 10 carpet, man, pulls, floor, helps, turns 40 Hook-based crochet687 crochet, yarn, hand, hook, fabric, left 41 Ironing and folding clothes365 table, cloth, iron, hand, hands, right 42 Street walking and running546 road, summary, walked, phone, hand, sidewalk 43 Group hiking and outdoor ac- tivities 514 woman, hand, man, summary, right, left 44 Shared cooking and kitchen conversation 11810 hand, right, man, left, kitchen, unsure 45 Cleaning and storing kitchen utensils 818 sink, kitchen, washes, places, summary, bowl 46 Woodenmechanical-model assembly 748 table, piece, hand, wood, wooden, model 47 Nailing and hammering in woodwork 415 hand, hammer, right, wood, nail, left 18 ID TopicAll Selected Original keywords (first six) 48 Reading music and piano practice 305 hand, piano, left, right, book, hands 49 Metal welding and grinding736 hand, metal, right, left, man, summary 50 Electronic components and cable handling 225 hand, cable, right, table, left, unsure 51 Chainsaw maintenance85 hand, right, chainsaw, left, saw, unsure 52 Hand sanding wood495 hand, sandpaper, left, right, wood, paper 53 Branch pruning627 hand, tree, right, left, branch, hands 54 Pottery and clay shaping746 hand, clay, right, mold, brick, left 55 Kitchen preparation and re- trieval (mixed) 406 hand, kitchen, drops, sink, right, cabinet 56 Tablet and electronic-device use 295 tablet, hand, right, phone, summary, screen 57 Lawn mowing and grass- clipping removal 466 mower, lawn, grass, bag, hand, sum- mary 58 Watching television757 hand,television,man,summary, watched, room 59 Cleaningandorganizing bookshelves 455 book, hand, books, right, left, shelf 60 Guitar playing766 guitar, hand, played, right, summary, room 61 Painting and color mixing10510 paint, brush, table, hand, paper, painting 62 Kitchen preparation and fam- ily interaction 758 hand, woman, right, left, unsure, table 63 Woodspray-paintingand spray-gun use 185 paint, hand, sprayer, gun, spray, wood 64 Hoeing and field clearing515 hand, hoe, ground, man, right, hands 65 Car-engine repair and oil changes 596 car, hand, unsure, engine, right, box 66 Ironing shirts and clothes205 shirt, iron, table, hand, cloth, board 67 Dishwashing and kitchen or- ganization 768 hand, right, left, sink, kitchen, drops 68 Tracing and cutting paper646 paper, hand, table, right, left, piece 69 Wall and ceiling renovation485 hand, wall, paint, ladder, scraper, wall- paper 70 Walking and talking with oth- ers 365 summary, man, woman, walked, hand, looked 71 Kitchen chopping and meal preparation 10310 drops, board, kitchen, unsure, hand, sink 72 Watering and tending potted plants 225 plant, flower, hand, water, pot, bottle 73 Drilling and assembly666 hand, drill, right, table, left, wall 74 Bicycle-component repair767 hand, right, bicycle, left, table, unsure 75 Drawing and coloring with pens 708 pen, hand, drawing, left, paper, book 76 Chess175 chess, man, piece, hand, board, sum- mary 77 Yarn crochet145 hand, crochet, right, left, wool, hook 78 Bag handling and camping storage (mixed) 385 bag, man, hand, right, table, tent 79 Room tidying and item stor- age 1009 hand, right, left, unsure, hands, floor 80 Parking-lot walking and so- cializing (mixed) 135 man, parking, lot, road, summary, walked 81 Clothes shopping and brows- ing 729 clothes, man, cloth, hanger, woman, rack 82 Lawn mowing with push mowers 215 lawn, mower, hand, right, grass, left 83 Desktop computer and key- board/mouse use 535 hand, computer, mouse, summary, phone, keyboard 84 Reading and indoor activities245 book, books, summary, shelf, wall, room 19 ID TopicAll Selected Original keywords (first six) 85 Ball sports (mixed)566 ball, man, tennis, summary, throws, field 86 Piping and decoration255 hand, right, bag, bowl, left, dough 87 Wall measurement and paper positioning 335 wall, tape, paper, piano, measure, sum- mary 88 Field activities and conversa- tion (mixed) 233 man, hand, grass, unsure, room, sum- mary 89 Hedge and shrub trimming566 hand, trimmer, hedge, right, hands, left 90 Interacting with cats225 cat, hand, right, floor, summary, room 91 Laptop and phone use817 laptop, hand, phone, table, summary, right 92 Laboratory tubes and sample handling 295 hand, summary, right, tube, test, table 93 Washing and preparing ingre- dients 496 sink, bowl, drops, unsure, spoon, board 94 Yarn crochet556 hand, yarn, crochet, right, left, hook 95 Wood sawing and planing12410 hand, wood, right, table, left, saw 96 Indoor conversation and so- cializing (mixed) 717 man, hand, table, woman, hands, right 97 Bicycle riding12710 bicycle, road, summary, man, rides, hand 98 Vacuum cleaning706 vacuum, cleaner, hand, floor, summary, left 99 Hose watering and spraying475 hand, right, left, hose, pipe, water 100 Fabric weaving and crochet818 thread, fabric, crochet, hand, summary, knitted 101 Constructionandcement work 405 man, cement, hand, wall, bucket, trowel 102 Cooking and stirring in pots656 spoon, pot, hand, cooking, unsure, kitchen 103 Small-engine and lawnmower maintenance 175 hand, right, left, oil, mower, lawn 104 Brick and concrete-block lay- ing 415 brick, bricks, hand, ground, summary, concrete 105 Car tire and wheel repair989 car, wheel, unsure, tire, hand, garage 106 Playing fetch and interacting with dogs 385 dog, hand, right, woman, man, ball 107 Indoor socializing and leisure (mixed) 758 man, woman, table, summary, coffee, room 108 Machine sewing979 cloth, machine, sewing, hand, thread, table 109 Driving and travel928 car, man, hand, door, summary, woman 110 Water carrying and outdoor washing 395 hand, bucket, right, water, left, floor 111 Needlework and embroidery546 hand, thread, right, left, needle, fabric 112 Supermarket shopping and checkout conversation 13312 man, woman, lady, cashier, unsure, shopping 113 Mechanical wood sanding and planing 636 wood, hand, table, right, piece, left 114 Computer and phone use245 hand, laptop, computer, summary, key- board, mouse 115 Dough kneading, dividing, and shaping 797 dough, hand, table, right, hands, left 116 Taping and masking43 tape, hand, right, paper, wall, hands 117 Bricklaying and masonry979 hand, right, trowel, man, wall, left 118 Vegetable cutting and meal preparation 436 hand, board, drops, knife, right, left 119 Indoor climbing and fitness conversation 272 man, summary, hand, woman, wall, hands 120 Eating and table conversation869 man, hand, table, woman, food, right 121 Toy and model-track assem- bly 275 table, hand, rail, right, left, space 122 Clothes folding and storage566 bed, hand, cloth, clothes, right, drops 20 ID TopicAll Selected Original keywords (first six) 123 Outdoor travel and exercise (mixed) 395 past, man, road, hand, rides, drives 124 Walking commutes and in- door/outdoor movement 556 man, phone, summary, walked, hand, bag 125 Pianoandelectronic- keyboard playing 456 piano, hand, played, keyboard, plays, summary 126 Hand sewing fabric and soft materials 455 thread, hand, table, needle, right, left 127 Two-wheeler tire and wheel maintenance 83 hand, right, tire, left, wheel, wrench 128 Wood drilling and construc- tion 536 wood, hand, drill, nail, summary, piece 129 Group walks and city visits365 woman, man, hand, road, past, right 130 Tractorandconstruction- vehicle operation 365 hand, right, left, tractor, unsure, hands 131 Painting wooden railings and posts 1069 paint, hand, brush, right, left, wood 132 Bowed-stringinstrument playing 65 violin, bow, played, cello, phone, sum- mary 133 Cotton sorting and fiber pro- cessing 365 cotton, hand, wool, right, camera, left 134 Car-wheel removal and bolt handling 205 wrench, nut, spanner, wheel, bolt, hand 135 Indoor/outdoormovement and stairs 295 hand, right, door, man, left, opens 136 Earthmoving and site clearing 10610 hand, ground, right, left, hands, man 137 Bathroomandhousehold- surface cleaning 808 hand, right, sink, left, unsure, bottle 138 Wire connection and routing959 hand, wire, cable, right, unsure, man 139 Shared meal preparation and kitchen cleaning 145 hand, woman, right, left, kitchen, un- sure 140 Crop harvesting and fodder handling 596 hand, right, left, ground, unsure, leaves 141 Forest hiking75 forest, summary, walked, phone, un- sure, water 142 Bread, eggs, and daily meal preparation 375 hand, pan, kitchen, plate, bread, sum- mary 143 Workshop and factory chores (mixed) 53 man, worker, water, hand, tray, machine 144 Car-brake and caliper repair425 brake, hand, wheel, unsure, caliper, bolt 145 Painting pottery275 hand, paint, right, table, left, brush 146 Daily kitchen activities and conversation (mixed) 385 hand, kitchen, unsure, right, cabinet, opens 147 Pot cooking and ingredient preparation 455 pot, cooking, unsure, kitchen, food, opens 148 Pan cooking and chopstick use 636 pan, hand, chopsticks, plate, sink, bowl 149 Cooking and washing dishes939 hand, right, left, drops, sink, pan 150 Vegetable cutting and meal preparation 1059 kitchen, board, knife, summary, onion, chopping 151 Ride-on lawn mowing195 mower, lawn, hand, right, left, unsure 152 Wood carving and hand pro- cessing 275 wood, hand, piece, floor, knife, right 153 Fabric measurement and cut- ting 245 cloth, piece, hand, table, fabric, scissors 154 Egg-trayproductionand stacking 165 man, hands, egg, hand, tray, right 155 Tea, coffee, and light meals205 phone, cup, table, coffee, hand, unsure 156 Home leisure and exercise (mixed) 10210 summary, room, phone, house, table, man 157 Snow shoveling and removal65 snow, shovel, flite, mini, machine, skid 158 Video gaming546 game, hand, phone, summary, played, video 21 ID TopicAll Selected Original keywords (first six) 159 Lawnmower blade and rotor maintenance 968 hand, right, left, unsure, mower, lawn Traceable records.These statistics are based on the original cluster_summary.json, video_clusters.jsonl, and final selected_video_uids.txt. The accompanying selected_1000_cluster_mapping.json stores each selected video’s identifier, original cluster index, Chinese topic summary, and distance to the cluster center, enabling verification of selection results and downstream video provenance. Topic summaries provide interpretation only and do not change original cluster assignments, selected-video lists, or counts. CQA GENERATION AND QUALITY CONTROL This section specifies evidence bindings, category-specific answer requirements, and the dependen- cies used to construct multi-turn questions. C.1EVIDENCE BINDING Each candidate QA is generated from an explicit evidence specification containing the video iden- tifier, event identifiers, temporal intervals, target component or relation, natural-language evidence descriptions, and required modalities. For multi-turn QA, we distinguish the evidence types in Table 7. Table 7: Evidence types in multi-turn QA. Evidence typeDefinition Direct evidenceEvidence introduced and used in the current turn. Inherited evidenceEvidence established in a preceding turn and still required by the current reasoning step. New evidenceAdditional evidence introduced in the current turn. Previously generated QA pairs may inform question wording, but they are never used as evidence for new answers. Available versus required modalities. A modality is marked as required only if removing it re- moves information necessary to answer the question. Audio-visual questions therefore require com- plementary information from both modalities, rather than merely originating from videos that con- tain both audio and visual streams. C.2SEMANTIC QA CATEGORIES The three Node categories describe individual states, while the six Edge categories describe directed relations. Chain and Loop capture composed dependencies and feedback closure, respectively. Modality requirements and dialogue formats are separate annotation axes; neither the number of turns nor the available media streams alone determines the interaction scope. Table 8: Semantic categories, query targets, and evidence requirements. CategoryQuery target and evidence requirements Ego StateCharacterize interaction-relevant attributes of the wearer. Action StatusDescribe actions that have been performed. World StateCharacterize object attributes, spatial configurations, locations, and world states. 22 CategoryQuery target and evidence requirements Intent AttributionConnect observed actions to evidence-supported goals or intentions; the expected answer must be supported by behavior, speech, task context, or other available evidence. Action EffectsConnect performed actions to observed world changes; evidence must contain both the action and its resulting state. Action GroundingCharacterize how the current environment or world state influences fea- sible behavior. Intent FormationConnect observations of the external world to subsequent goals, subgoals, or decisions. Goal PlanningUse only information available before the specified decision cutoff. Fu- ture events may be retained to verify outcomes but are not provided as predictive evidence. World PerceptionCharacterize how the wearer perceives and understands the external world through visual, auditory, or textual cues.Answers must be grounded in perceptual evidence available from the wearer’s perspective, without assuming knowledge of unobserved information. C.3MULTI-TURN CONSTRUCTION Multi-turn samples contain two to four turns and are constructed through dependency-driven com- position. Earlier turns establish intermediate facts, while later turns combine inherited information with newly introduced audio-visual evidence. For a multi-turn sample, the final answer depends on information established in earlier turns together with new evidence introduced for the final turn. The final turn must cover the relevant endpoints of the claimed relation through direct or inherited evidence. Candidates are excluded if their final answers can be obtained solely from previous textual answers without consulting the new video evidence. For Loop questions, the multi-turn evidence must collectively cover the original goal, action, outcome, and feedback returning to the ego; Cross- loop questions must additionally cover inter-loop dependencies. Covering two event endpoints or introducing a new evidence field does not by itself demonstrate closure. Where available, we retain standalone versions of constituent questions with matched answer tar- gets and evidence, enabling controlled comparisons between atomic reasoning and dialogue-based compositional reasoning. The following checks assess candidate validity, structure, and evidence requirements. Cross-loop reasoning is treated as a subtype of Loop, not a fifth top-level category. C.4GENERAL QUALITY CHECKS All generated QA pairs are initially treated as candidates. We check field completeness, video and event references, temporal intervals, evidence–source alignment, modality consistency, duplication, and answer consistency. Multiple-choice samples additionally require valid, non-duplicate options and designated answers consistent with those options. Semantic review checks whether the attributes, states, sounds, intentions, and relations in an an- swer are supported by its bound evidence. It also detects unsupported details, answer leakage, and mismatches between declared modalities and those actually required to answer. For multi-turn QA, we additionally inspect turn dependencies, inherited evidence, newly introduced evidence, and whether earlier turns prematurely reveal the final answer. C.5CATEGORY-SPECIFIC STRUCTURAL FILTERING 23 Table 9: Structural filtering criteria. LevelRetention criteriaFailure cases NodeA clearly identifiable target state or fact localized to an appropriate evidence inter- val. Attribute questions require explicit evidence for the queried attribute; histor- ical questions require observations of the same tracked entity or ego trajectory. Unclear targets, missing attribute ev- idence, or historical observations that cannot be associated with the same en- tity or ego trajectory. EdgeClearly defined relation endpoints, a spec- ified task direction, and supporting evi- dence; forward causation is not required for every edge.Intent Attribution re- quires behavioral, linguistic, or contextual support for attributing intention from ac- tion; Action Effects requires observable state changes; Action Grounding requires specific environmental conditions affect- ing action feasibility; Intent Formation requires external conditions influencing later intentions or decisions; Goal Plan- ning is evaluated using only evidence be- fore the prediction cutoff; World Percep- tion requires world-perception evidence available from the ego’s perspective. Insufficientrelationendpointsor category-specific supporting evidence. ChainMultiple necessary information-bearing components forming an open path without feedback closure. Dependency explana- tions and semantic review verify the con- tribution of earlier information. Mere temporal adjacency, thematic sim- ilarity, or shared objects; earlier compo- nents do not contribute to the answer; a complete loop is actually required but the sample remains labeled Chain. LoopEvidence covers the original goal, rele- vant action, world change, and perceptual feedback, supporting the return of feed- back to the ego to confirm, adjust, or up- date the original goal. The answer must integrate this closed connection. Only action and outcome, or only feed- back and a later action; missing goal linkage, outcome use, or confirmation evidence; success inferred from an ac- tion ending, or feedback adjustment in- ferred from temporal order alone. Cross-loopSatisfies closure requirements and in- cludes necessary bridging facts and en- tity/state correspondences between loops. Earlier-loop information contributes to a later loop and the final answer. Cross-stage state transfer without clo- sure evidence; unclear inter-loop con- nections; removing an earlier loop still permits a complete answer. C.6LANGUAGE SHORTCUTS AND MODALITY AUDITS We conduct text-only answering tests to identify samples whose answers can primarily be inferred from question wording, dialogue history, or recurring templates without the intended audio-visual evidence. These samples are flagged as potential language shortcuts. Inspired by VisNec [6], we also evaluate modality necessity. For audio-visual samples, if either modality alone fully determines the answer, the sample is considered to contain modality redun- dancy. For cross-modal relations, visual and auditory observations must participate in the same interaction or provide complementary evidence for the target relation. C.7CONSTRAINED REPAIR AND REVALIDATION This section defines which edits are permitted, the structural properties they must preserve, and how repaired candidates are checked again. 24 Candidates with repairable errors enter constrained repair. Editing is limited to the existing evidence pool and the original reasoning target. The editor may accept, rewrite, reject, or refer the candidate for review. Category-specific repairs. Table 10: Permitted repairs and constraints. LevelPermitted repairsRequired constraints and rejection con- ditions NodeAdjust overly specific state descriptions to the finest granularity supported by ev- idence; clarify references; correct modal- ity labels or evidence bindings. Reject the sample if the target attribute is entirely unobservable. EdgeIntent Attribution:generalize unsup- ported specific intentions to evidence- supported goals. Action Effects: refo- cus on actual world-state changes. Ac- tion Grounding/Intent Formation: clarify external conditions and the behaviors or goals they influence. Goal Planning: re- strict the answer to a next goal or action category supported by evidence before the prediction cutoff. Preserve the original relation type and task direction. If the relation remains unsupported, reject rather than down- grade to a Node question. ChainRemove redundant components, clarify existing information transfer, and correct answer leakage or referential ambiguity. Preserve an open path of multiple nec- essary components. Do not add unre- lated events to lengthen the chain or in- vent feedback to turn it into a Loop. Re- ject or review if the repaired sample no longer meets its original category. LoopNarrow overstated claims and clarify ex- isting evidence of goals, outcomes, check- ing, or feedback use. Preservegoal–action–outcome– perceptual-feedback–goal-assessment closure. Do not invent confirmation, mental updates, or successful outcomes. Reject or review if closure evidence is missing. Cross-loopClarify entity references, bridging facts, and existing feedback connections. Preserve both closure and inter-loop de- pendencies.Reject or review when identity, state transfer, or closure cannot be verified; do not convert the sample into an open chain while retaining its la- bel. Loop repair must preserve the connection from the original goal or subgoal to an action, the resulting world change, perceptual feedback, and confirmation, adjustment, or updating of the original goal. Repair can only clarify closure already supported by the evidence. Phrases such as “therefore real- ized” or “confirmed success” cannot supply missing links. Closure determination requires semantic review; field completeness, graph connectivity, or a model’s validation pass alone does not prove closure. General repair rules. Permitted edits include resolving referential ambiguity, removing answer leakage and unsupported details, adjusting answers to an evidence-supported abstraction level, cor- recting modality labels, adjusting existing evidence bindings, removing redundant reasoning com- ponents, and converting multiple-choice questions to open-ended ones when valid distractors cannot be constructed. Repairs must not introduce facts outside the existing evidence pool, add unrelated events to increase reasoning complexity, change the prediction cutoff, or alter the underlying reasoning target. Samples 25 requiring additional video evidence, new clipping, or substantial restructuring are rejected or referred for review. Revalidation. After repair, we repeat deterministic and category-specific validation. Goal Planning samples also undergo pre-decision evidence checks to ensure that future information has not entered the repaired question or answer. A separate validation call then evaluates the repaired QA directly against its bound evidence, without access to the editor’s rationale. Samples that remain unsupported, ambiguous, incorrectly formatted, or structurally inconsistent are rejected or referred for review. DHUMAN REVIEW This section describes the sampled review cohort, the review interface, and the scope and limitations of human inspection. To complement automatic quality control, we manually inspected a randomly sampled subset of QA groups. The review records were sampled without replacement using a fixed seed (20260921). Sampling was performed at the sample level: multi-turn dialogues were retained as complete groups rather than split into individual turns. Review coverage for the current inventory requires matching the retained sample identifiers to the review records; we do not infer it from the dataset size. Figure 7 shows the web interface used to support this inspection. It presents the corresponding video clip, questions, reference answers, evidence intervals, modality labels, and reasoning cate- gories together. The review criteria cover answer consistency with the audio-visual content, evidence sufficiency and temporal alignment, modality-label correctness, and the validity of the assigned rea- soning structure. In particular, Chain questions require an open dependency path, whereas Loop questions require feedback closure; repeated nodes or multiple dialogue turns alone do not establish a Loop. The interface supports recording an accept, needs revision, reject, or uncertain decision, together with issue labels, reviewer notes, and turn-level editing suggestions. Suggested edits are stored separately from the original annotations to preserve traceability. Samples whose media or supporting evidence cannot be verified should remain uncertain rather than be treated as validated. This sampled inspection does not constitute exhaustive human verification of the entire test set. ETRAINING AND EVALUATION DETAILS E.1TRAINING CONFIGURATION The following settings are taken from the saved full-training configurations and launch scripts, rather than the small-sample diagnostic runs. Qwen2.5-Omni-7B. LoRA uses the LLaMA-Factory target setting all, rank 8, and scaling fac- tor 16. The eight-GPU configuration uses one example per device and two gradient-accumulation steps, giving an effective batch size of 16. FFT uses the same batch configuration with DeepSpeed ZeRO-2 and updates the Thinker, including its visual and audio components, without training speech generation. Qwen3-Omni.The expert-LoRA implementation targets the attention projections q_proj, k_proj, v_proj, and o_proj, and additionally injects adapters into 96 fused expert parame- ters whose names end in gate_up_proj or down_proj. Rank and scaling factor are 8 and 16. The saved four-GPU configuration uses one example per device and four accumulation steps, yielding an effective batch size of 16. FFT uses DeepSpeed ZeRO-3 with the same four-GPU batch configuration. Both configurations disable thinking during training. Later continuation launches must be distinguished from these initial configurations when reproducing a particular checkpoint. Shared optimization and checkpointing. LoRA and FFT use learning rates of 0.00002 and 0.000002, respectively, cosine decay, and 3% warm-up. Both use BF16, FlashAttention-2, non- reentrant gradient checkpointing, a sequence-length limit of 32,768, and four data-loader work- ers per process. The saved configurations specify ten epochs, epoch-wise checkpointing without a 26 Figure 7: Web interface for human review of EgoOmniWorld. The interface combines video play- back with questions, reference answers, evidence intervals, and structural annotations, and provides controls for recording quality checks and suggested revisions. The screenshot illustrates the inter- face, not aggregate review outcomes. checkpoint-retention limit, and no intermediate validation. The setting mask_history=false retains supervision on earlier assistant answers in multi-turn training examples. These configuration values describe scheduled training, not a claim that every reported checkpoint completed ten epochs. E.2EVALUATION PROTOCOL AND REPRODUCIBILITY Saved evaluation manifests and run configurations document the scope underlying the reported re- sults. We evaluate models on the test set. Records are deduplicated by sample identifier and round index, and dialogue history is retained. Predictive questions use cutoff-safe clips. Saved manifests specify evaluation membership and annotation provenance. The Qwen3-Omni-Thinking LoRA re- sults use the second-epoch checkpoint, with no failed judgments. Categories follow the frozen V9 taxonomy and saved item labels. Reproducibility requires the checkpoint identifiers, input settings, scoring protocols, and evaluation manifests for each model and transfer benchmark. FSUPPLEMENTARY EXPERIMENTAL RESULTS F.1RESULTS BY SEMANTIC CATEGORY This section reports performance across the nine semantic categories, complementing the interaction-level results in Table 2. Each cell reports mean score and accuracy. Comparisons be- tween a base model and its adapted variants reveal which capabilities benefit from training and which remain challenging. Uneven gains across categories. For Qwen2.5-Omni-7B, LoRA improves accuracy in Goal Plan- ning from 30.00% to 60.00% and World Perception from 21.88% to 37.98%, but Action Effects decreases from 36.73% to 34.69%. FFT instead improves Action Effects to 57.14%. Thus, im- provements in overall performance do not imply uniform improvements across interaction relations. Adaptation-dependent trade-offs. For Qwen3-Omni-Thinking, LoRA improves Action Status from 34.59% to 43.55% and World State from 47.30% to 52.70%. FFT achieves higher accu- racy than LoRA in Intent Formation (64.13% versus 60.87%) and World Perception (40.31% versus 27 Table 11: Test-set performance across the nine semantic categories. Cells show mean score / accu- racy (%). Best scores and accuracies within each model group are bolded independently; ties are retained. NodeEdge Model Ego State Action Status World State Goal Planning Intent Attribution Action Effects Action Grounding Intent Formation World Perception Closed-source models Gemini 3.8 Flash3.55/73.13 2.41/44.09 3.35/64.86 3.27/70.00 4.00/82.14 2.06/42.86 3.75/74.63 3.18/64.37 2.65/49.61 Qwen3.5-Omni-Plus2.99/58.21 1.76/25.81 2.53/47.30 1.47/33.33 3.86/78.57 2.61/44.90 3.21/65.67 2.99/59.78 2.11/36.72 Qwen3.8-Omni-Flash3.49/70.15 2.35/39.25 2.80/52.70 3.10/63.33 3.96/78.57 2.33/38.78 3.57/71.64 3.14/61.96 2.35/43.75 Open-source models MiniCPM-o 2.6 8B3.06/56.72 1.65/19.35 2.31/33.78 2.13/43.33 3.68/67.86 1.69/22.45 2.43/49.25 2.00/32.56 1.52/22.48 SALMONN2+ 72B2.84/52.24 1.78/25.81 2.49/47.30 2.07/40.00 3.46/67.86 1.82/26.53 3.03/62.69 3.02/58.62 1.86/29.46 SALMONN2+ 7B2.64/52.24 1.58/21.51 2.01/35.14 2.23/40.00 3.46/67.86 1.57/22.45 2.48/49.25 2.52/48.28 1.71/28.68 VideoLLaMA2 7B AV2.24/40.30 1.40/16.67 1.55/22.97 1.77/33.33 2.96/57.14 1.31/22.45 1.94/34.33 1.33/25.29 1.25/17.83 Qwen2.5-Omni-3B2.90/53.73 1.64/23.12 1.95/31.08 1.20/23.33 3.71/75.00 1.98/32.65 2.81/59.70 2.70/51.09 1.75/27.91 Qwen2.5-Omni-7B (base)2.60/49.251.56/25.272.14/36.991.50/30.004.21/82.142.24/36.732.49/53.731.83/35.871.30/21.88 Qwen3-Omni-Thinking2.85/56.722.11/34.592.62/47.302.77/53.333.36/60.712.15/39.583.13/64.182.51/50.552.05/32.81 Qwen3-Omni-Instruct2.70/49.25 2.01/34.41 2.64/45.95 1.83/36.67 3.57/67.86 2.35/40.82 3.10/65.67 2.74/53.26 2.04/33.33 Qwen2.5-Omni-7B-LoRA3.24/61.192.37/38.172.53/43.242.93/60.004.25/85.712.49/34.692.97/58.213.10/59.782.25/37.98 Qwen2.5-Omni-7B-FFT3.09/62.692.31/34.952.30/39.192.73/53.334.07/82.143.02/57.142.81/55.223.07/61.962.15/36.43 Qwen3-Omni-Thinking-FFT3.30/62.692.49/41.402.41/40.542.73/56.674.18/85.712.78/44.903.21/65.673.21/64.132.34/40.31 Qwen3-Omni-Thinking-LoRA3.07/59.702.53/43.552.82/52.702.50/53.334.25/85.712.73/48.983.22/64.182.98/60.872.33/39.53 39.53%), but lower accuracy in World State (40.54% versus 52.70%). These differences motivate retaining category-level evaluation rather than relying exclusively on an overall ranking. F.2TRANSFER RESULTS BY ORIGINAL TASK CATEGORY Table 12 compares Qwen2.5-Omni-7B with its EgoOmniWorld-adapted LoRA counterpart using each benchmark’s original task categories. The category-level results reveal where transfer is most effective and where limitations remain. EgoSound. LoRA improves accuracy in all seven categories. The largest gains occur in sound- source identification (+21.34 percentage points), spatial localization (+11.12), and sound charac- teristics (+8.94). Cross-modal reasoning also improves by 7.45 points, while gains in temporal information (+3.80), counting (+2.44), and causal/contextual reasoning (+2.22) are smaller. Trans- fer is therefore strongest for identifying, locating, and characterizing sounds, with more modest improvements in temporal and higher-level reasoning. EgoToM. Action prediction improves from 35.21% to 41.57% (+6.36 points), and goal inference improves from 89.03% to 93.67% (+4.64 points). However, belief inference decreases from 44.55% to 38.61% (-5.94 points). The gains in action and goal prediction thus do not extend uniformly to reasoning about beliefs. EgoTempo. The largest improvements are in spatial relations (+22.00 points), object localization (+16.00), and action-specific objects (+12.00), followed by object-specific actions (+4.00). In con- trast, temporal event ordering declines by 2.00 points; action and object sequence tasks each decline by 4.00 points; and future action prediction declines by 6.00 points. Action and object counting also decrease by 8.00 and 12.00 points, respectively. These results indicate stronger transfer for spatial and object-related understanding than for precise temporal ordering, anticipation, and count- ing. In particular, the opposite changes in action prediction on EgoToM and EgoTempo show that improvements in one benchmark do not necessarily generalize to similarly named tasks in another. GPROVENANCE AND LIMITATIONS This section summarizes retained provenance and distinguishes source traceability from evidential, structural, and causal validity. For each retained sample, we store the question, reference answer, answer key points where appli- cable, semantic category, structural reasoning level, required modalities, supporting video and event identifiers, temporal intervals, evidence descriptions, and provenance mappings to structured inter- 28 Table 12: Transfer accuracy (%) by each benchmark’s original task categories for Qwen2.5-Omni- 7B and its LoRA-adapted counterpart.∆ denotes the difference in percentage points, computed from displayed accuracies. Original task categoryBase LoRA∆ EgoSound Sound-source identification15.58 36.92 +21.34 Spatial localization25.88 37.00 +11.12 Sound characteristics31.49 40.43+8.94 Cross-modal reasoning30.85 38.30+7.45 Temporal information28.44 32.24+3.80 Counting44.72 47.16+2.44 Causal / contextual reasoning39.46 41.68+2.22 EgoToM Action prediction35.21 41.57+6.36 Goal inference89.03 93.67+4.64 Belief inference44.55 38.61-5.94 EgoTempo Spatial relations22.00 44.00 +22.00 Object localization24.00 40.00 +16.00 Action-specific objects24.00 36.00 +12.00 Object-specific actions18.00 22.00+4.00 Temporal event ordering26.00 24.00-2.00 Action sequence12.008.00-4.00 Object sequence16.00 12.00-4.00 Future action prediction20.00 14.00-6.00 Action counting28.00 20.00-8.00 Object counting40.00 28.00 -12.00 action records. Multi-turn samples additionally store turn-level dependencies, inherited and newly introduced evidence, and corresponding standalone constituent questions. Although the pipeline combines structured extraction, evidence-constrained generation, filtering, repair, and revalidation, these procedures do not establish that every inferred high-level relation is objectively causal or uniquely correct. Some upstream components, particularly interpretations of intentions and tasks, remain model-generated annotations. Provenance matching verifies source consistency, not perceptual correctness. We distinguish evidential sufficiency, structural dependence, and causal certainty. Node, Edge, Chain, and Loop are defined by the reasoning dependencies required to answer a question, rather than by event counts or dialogue length alone. Chain questions involve linked dependencies with- out feedback closure. Loop questions additionally require connecting action outcomes to the ego’s assessment of its goal or subsequent decisions. Cross-event relations alone do not establish such clo- sure. These structural labels characterize the required reasoning, but do not by themselves establish causal certainty. HDATA-ENGINE PROMPTS The templates below condense the core instructions of the original prompts. Repeated examples and implementation-specific schema details are omitted; these are not verbatim runtime strings. Angle brackets denote supplied inputs. Video selection. Source selection and narration clustering are procedural steps. Model prompting begins with the selected video windows. H.1LOCAL INTERACTION EXTRACTION Input: <video with audio>, <duration>, <schema>. Identify bounded actions and record the preceding world state, 29 trigger, persistent goal, current subgoal, action, outcome, and feedback. Distinguish goals from action descriptions. Attribute speech to its actual source. Retain failed and interrupted actions. Mark unsupported components as unknown; do not invent intentions, effects, feedback, or visual details. Label a complete loop only when its components are supported. Return JSON with clip-relative intervals, evidence, confidence, and uncertainty flags, following the supplied schema. H.2PERCEPTION GROUNDING: HD-EPIC BRANCH Input: <video window>, <entity cache>, <narration anchors>. Extract visible properties, states, spatial relations, hand-object interactions, and task-relevant scene layout. Reuse identifiers only for the same physical entity. Keep temporally distinct states separate. Use narrations for linking, not as substitutes for visual evidence. Do not rewrite actions or intentions, or infer exact gaze from camera direction. Return entities and timestamped visual observations with source references and answerability flags. Mark ambiguity explicitly. H.3EPISODE FORMATION Input: <chronological structured interaction records>. Group adjacent records serving one coherent activity stage. Split when the goal, object set, workspace, or activity changes. Preserve source events, persistent goals, current subgoals, opening and resulting states, entities, instructions, failures, interruptions, and unfinished work. Do not infer hidden motives. Return episode JSON only. Do not generate relations or questions. H.4OVERLAPPING-EPISODE RECONCILIATION Input: <overlapping episode summaries from one video>. Merge duplicate episodes, preserve chronological order, and separate genuinely different goals. Assign each event to one final episode unless it is a genuine boundary event. Retain states, entities, instructions, failures, and pending work. Do not invent stages to meet a quota. Return reconciled episode JSON without relation edges or questions. H.5CROSS-EPISODE RELATION EXTRACTION Input: <final episodes>, <boundary-event evidence>. Link episodes only when evidence supports a transferred state, instruction, constraint, pending task, or feedback adjustment. Temporal adjacency, shared objects, or similar topics alone are insufficient. Mark uncertain links as ineligible for QA. Return relation JSON with source and target events, connected components, relation type, carried information, modalities, and uncertainty. Zero supported links is a valid output. H.6EVIDENCE-CONSTRAINED RELATION REPAIR Input: <episodes>, <boundary evidence>, <prior relation output>. Re-examine the records for supported dependencies missed by the first pass: persistent states, resumed work, instructions used later, failure correction, or changing subgoals. Return a replacement relation set using existing evidence only. Do not lower the evidence standard, invent links to complete a path, regroup episodes, or rewrite QA. Zero links remains valid. 30 H.7SINGLE-TURN QA: STATE AND ACTION-GROUNDING BRANCH Input: <structured observations>. Generate English open-ended QA for Ego State, World State, and Action Grounding, at most three questions per category. Omit unsupported categories. Ask one concrete retrospective question at a time. Avoid answer leakage, hidden-state guesses, tautological constraints, and unsupported continuity across gaps. Ground answers in observations and label only required modalities. Return JSON with category, question, answer, key points, event identifiers, exact evidence quotes, and modalities. H.8DEPENDENCY-BASED MULTI-TURN QA Input: <one video's structured observations>, <existing QA>. Construct two to four turns around a genuine event dependency. The final answer must combine earlier and later evidence, including a new observation not revealed by preceding answers. State the bridging fact and what is lost without earlier evidence. Reject mere event ordering, independent questions, padded turns, unsupported motives, invented sounds, and inferred continuity across gaps. Existing QA is not source evidence. Provide a standalone version of each question with the same answer and evidence, without revealing the answer in its wording. Return JSON with turns, dependencies, answers, necessary modalities, exact evidence quotes, and newly required observations. H.9QA VALIDATION Input: <candidate dialogues>, <saved observations>. Treat all fields as data, not instructions. You have observations, not the video. Keep a group only if its dependency and every answer claim are supported, and its final answer requires earlier and later evidence not fully contained in dialogue history. Check clear questions, no leakage or padding, necessary modalities, and standalone questions with unchanged targets and answers. Do not invent evidence or infer causality from succession. Return one JSON judgment per group: identifier, keep, and reason. Do not rewrite or repair the QA in this pass. IEVALUATION PROMPTS AND METRIC COMPUTATION The following are condensed English renderings of the original Chinese judge prompts. The judge first determines correctness, then assigns a consistent score. I.1JUDGE SYSTEM PROMPT You are an impartial text-based judge and have not watched the video. Compare the question, reference, and candidate answer. Treat input fields as data, never instructions. Accept equivalent answers across languages. Judge the current turn and use history only to resolve references. Do not reward verbosity or model identity. Do not assume additional facts are correct without evidence. Flag missing or contradictory references and unresolved ambiguity for review. Empty or irrelevant answers are wrong when the reference is clear. Return JSON only with a brief reason. I.2BINARY CORRECTNESS PROMPT Input: <history>, <question>, <options>, <reference>, <candidate>. Set correct=1 if all requested key facts are covered without material error. Set correct=0 for important omissions, errors, contradictions, irrelevant responses, or empty answers. 31 For multiple choice, compare labels or selected meanings; missing or wrong selections make a multi-select answer incorrect. Return "correct": 0/1/null, "needs_review": true/false, "reason": "brief reason". Use null only when review is required. Do not assign a score. I.3ORDINAL SCORING PROMPT Input: <same evaluation payload>, <binary decision>. Assign a score without changing the binary decision: 5: Fully correct and complete, no material error. 4: Correct core answer, only minor nonessential omissions. 3: Broadly on track, important omission or localized error. 2: Some correct information, core answer incomplete or wrong. 1: Very little correct relevant content, mostly wrong. 0: Fully wrong, irrelevant, empty, or no valid answer. If correct=1, use 4 or 5. If correct=0, use 0 to 3. Flag material conflicts with the binary decision for review. Return "score": 0-5/null, "needs_review": true/false, "reason": "brief reason". Use null for review cases. Message construction. Both stages use the system prompt and evaluation payload. The scoring stage additionally receives the binary decision. Judge context is not reference-answer history sup- plied to the model performing video QA. Aggregation and exclusions. Report accuracy and mean score over successfully judged records, with errors and review cases recorded separately. Comparisons require consistent evaluation mem- bership and denominators; exclusions are not silently counted as incorrect answers. 32

Instant research discovery