Paper deep dive
Companion-style QA Assistance in Ego-Vision
Hangyu Qin, Junbin Xiao, Shenglang Zhang, Angela Yao
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 9/12/2026, 3:00:52 AM
Summary
This paper introduces BuddyVQA, a benchmark for companion-style question answering on egocentric streaming video, and MyBuddy, a framework to address it. BuddyVQA contains 21.6K questions across 1,012 videos, focusing on ego-deictic expressions and interactively chained questions. MyBuddy employs a multimodal chain-of-thought reasoning mechanism, a question filter, and multi-level memory to resolve visual pronouns and historical dependencies in real-time.
Entities (8)
Relation Signals (6)
BuddyVQA → contains → 21.6K questions
confidence 95% · BuddyVQA contains 21.6K questions linked to 6K highlight moments across 1,012 long, egocentric videos.
BuddyVQA → features → ego-deictic expressions
confidence 95% · It features two key characteristics... ego-deictic expressions and interactively chained questions
BuddyVQA → derivedfrom → EgoSchema
confidence 90% · we construct BuddyVQA using videos from EgoSchema (Mangalam et al., 2024)
MyBuddy → uses → Chain-of-Thought
confidence 90% · MyBuddy... highlights a multimodal chain-of-thought reasoning mechanism
MyBuddy → uses → multi-level memory
confidence 90% · An additional question filter and multi-level memory are designed to facilitate efficient QA
Gemini 2.5 Pro → usedfor → highlight detection
confidence 85% · we first employ Gemini-2.5-pro... to detect visual highlight segments
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:AI companions are envisioned as always-on assistants that support users in daily life. With this regard, we introduce BuddyVQA, a benchmark for companion-style question answering (QA) on egocentric streaming video. BuddyVQA contains 21.6K questions linked to 6K highlight moments across 1,012 long, egocentric videos. It features two key characteristics that are common in daily first-person QA assistance but are largely overlooked in existing VideoQA benchmarks: ego-deictic expressions and interactively chained questions (e.g., "Where is it?", "How to get there?"). These require models to infer a user's in-situation intent by resolving visual pronouns in the context of egocentric visual and QA contents, with both grounded in a long-form streaming setting. To tackle the challenges, we propose MyBuddy, a companion-style QA assistant that highlights a multimodal chain-of-thought reasoning mechanism to infer the final answer based on the historical QA and visual content. An additional question filter and multi-level memory are designed to facilitate efficient QA and visual information retrieval under streaming QA settings. Experiments show that MyBuddy significantly enhances the performance of foundation models on BuddyVQA. Moreover, these gains generalize to other streaming and common video QA benchmarks, demonstrating the applicability and effectiveness of our approach. Our code and dataset are available at this https URL
Tags
Links
- Source: https://arxiv.org/abs/2609.06721v1
- Canonical: https://arxiv.org/abs/2609.06721v1
Trouble viewing inline? Open PDF directly →
Full Text
129,287 characters extracted from source content.
Expand or collapse full text
Noname manuscript No. (will be inserted by the editor) Companion-style QA Assistance in Ego-Vision Hangyu Qin· Junbin Xiao· Shenglang Zhang· Angela Yao Received: date / Accepted: date Abstract AI companions are envisioned as always-on assistants that support users in daily life. With this regard, we introduce BuddyVQA, a benchmark for companion-style question answering (QA) on egocentric streaming video. BuddyVQA contains 21.6K questions linked to 6K highlight moments across 1,012 long, ego- centric videos. It features two key characteristics that are common in daily first-person QA assistance but are largely overlooked in existing VideoQA benchmarks: ego-deictic expressions and interactively chained ques- tions (e.g., “Where is it?”, “How to get there?”). These require models to infer a user’s in-situation intent by resolving visual pronouns in the context of egocentric visual and QA contents, with both grounded in a long- form streaming setting. To tackle the challenges, we propose MyBuddy, a companion-style QA assistant that highlights a multimodal chain-of-thought reason- ing mechanism to infer the final answer based on the historical QA and visual content. An additional ques- tion filter and multi-level memory are designed to facil- itate efficient QA and visual information retrieval un- der streaming QA settings. Experiments show that My- Buddy significantly enhances the performance of foun- dation models on BuddyVQA. Moreover, these gains generalize to other streaming and common video QA benchmarks, demonstrating the applicability and ef- fectiveness of our approach. Our code and dataset are available at https://github.com/QHUni/BuddyVQA. Corresponding to: junbinxiao@ustc.edu.cn. Hangyu Qin and Angela Yao are with National University of Singapore. Junbin Xiao and Shenglang Zhang are with University of Sci- ence and Technology of China. 1 Introduction With the rapid adoption of wearable cameras and mixed-reality devices, egocentric visual assistants have become an important research area for enhancing daily human experiences (Grauman et al., 2022; Yang et al., 2025a; Yan et al., 2025). Imagine a visual assistant that accompanies users, continuously observes and re- members their visual experiences, and answers ques- tions when needed (see Fig. 1). Often, such assistance requires more than object recognition (B ̈armann and Waibel, 2022; Sun et al., 2025) and scene descrip- tion (Xiao et al., 2025; Zhou et al., 2025). It requires a companion that maintains visual context over time, understands vague user intentions, and provides practi- cal answers and advice through real-world interaction, serving as an assistive “buddy.” With this regard, we explore a new setting for companion-style question answering (QA), where the video question answering (VideoQA) model acts as a buddy accompanying the user through egocentric video streams to answer practical questions. Such a setting is distinct from traditional VideoQA (Yu et al., 2019; Xiao et al., 2021; Fu et al., 2024; Mangalam et al., 2024), which focuses on offline, third-person clips with fixed temporal scopes and questions. Recent stream- ing VQA (Zhang et al., 2025c; Lin et al., 2024; Niu et al., 2025) is closer in aim, but it ignores egocentric references and conversational continuity in daily QA practice. In contrast, companion-style QA occurs in a shared first-person context, where users frequently ask questions that depend on previously observed events and prior QA interactions. Such questions often con- tain deictic or under-specified references (e.g., “What is this”), requiring the assistant to infer the user’s true in- tent. Therefore, we summarize that a QA buddy should arXiv:2609.06721v1 [cs.CV] 6 Sep 2026 2Hangyu Qin et al. Scrutinizing the Present Recalling the Past Recognizing Visual Content Advising for the Next Q: Is thismixture ready? A: Not yet, it would be better to break it into small pieces. Q: Should I add this ground meat tothem now? A W/o chain: Yes, you can add the ground meat to the plate now. A W/ chain: No, the mixture is not ready for seasoning and you should not add the meet. 1k+ videos 6k+ highlight moments 79.1% deictic Rate 21K+ companionQAs BuddyVQA:Companion-style QA Assistance in Ego-Vision 100% Human Edited Qwen3-VL Flash-Vstream ReKV LongVA MyBuddy MyBuddy Performance Q: What is the price? A W/o chain: The price of the things you bought is18 dollars. A W/ chain: The price ofMonster Energy is 3 dollars. Q: Isita Monster or a Red Bull here? A: It is a can Monster Energy drink. Q: How should I position thisbeforemaking the next cut? A: Set the base plate of the saw flat on the board, and align the blade with the mark. Q: Was there any snacks over there? A: Yes, there are some Little Bites on the shelf. Q: How many kinds ofit are there? A W/o chain: There are several kinds of bread. A W/ chain:2different kinds of Little Bites. Fig. 1 BuddyVQA targets companion-style VQA assistance. It features two key challenges: ego-deictic expressions and chained questions, both under the egocentric streaming setting. MyBuddy achieves SOTA results across different question categories. possess 1) Shared view with egocentric streams; 2) In- terpretation of visual deictic expressions; 3) Contex- tual dependency of questions; and 4) High response efficiency. Achieving these goals introduces challenges of egocentric referential grounding, multimodal contex- tual reasoning, and long-term memory, with which cur- rent benchmarks (Mangalam et al., 2024; Yang et al., 2025a; Xiao et al., 2025; Zhao et al., 2025; Ye et al., 2025; B ̈armann and Waibel, 2022) do not cover, or cover only partially. To bridge this gap, we first construct BuddyVQA, a benchmark targeting egocentric visual QA compan- ions in the streaming setting. BuddyVQA contains 21.6K questions linked to 6K moments across 1,012 long ego videos of everyday scenarios. Each video segment is annotated with temporally aligned questions that em- ulate how users might query an assistant, from recall of the past, to scrutinization of the present, to advising for the future. The benchmark emphasizes two unique phenomena critical to live companion assistance (see Fig. 1): ego-deictics, with referential or spatial pro- nouns (e.g., “this,” “it,” “there”) that must be resolved with respect to the shared egocentric view; interactive chaining, where successive queries depend on prior QA pairs and visual content, modeling visual conversational continuity. Moreover, the two phenomena are often en- tangled and grounded in streaming video. Benchmarking results show that existing general- purpose and streaming foundation models (Li et al., 2024a; Bai et al., 2025b; Zhang et al., 2024a; Li et al., 2023a; Wang et al., 2025b; Zhang et al., 2025a) strug- gle on BuddyVQA; their performance rarely exceeds 50% accuracy, and the models frequently misinter- pret referential expressions or miss important histor- ical information, exposing severe limitations of ex- isting techniques for companion-style assistance. For improvements, we introduce MyBuddy, a training- free and model-agnostic framework designed to tackle the aforementioned challenges for BuddyVQA. My- Buddy resolves ego-deictic references following a chain- of-thought (CoT) mechanism that performs visually grounded reasoning in the context of ego-cues (e.g., gaze direction, hand interactions) and relevant histor- ical QAs. Specifically, the ego-cues are extracted from embodied visual features in the memory, which repre- sent the wearer’s eye gaze point and hand gestures most relevant to the current question. The relevant QAs are selected by a Question Filter under the guidance of the current question to support ego-deictic reasoning. No- tably, we also propose a multi-level memory to struc- turally store the visual contents; both the ego-cues and visual contexts are retrieved on demand rather than pre-computed for the entire video stream, enabling ef- ficient online inference under the streaming setting. Experiments show that MyBuddy achieves state-of- the-art results on BuddyVQA. It also generalizes well to other streaming and offline VideoQA datasets. The results underscore a significant step towards harness- ing foundation multimodal models for live companion- style VQA assistance. Our primary contributions are: Companion-style QA Assistance in Ego-Vision3 1) BuddyVQA, a companion-style VQA benchmark that targets egocentric user assistance by highlighting the core challenges of ego-deictics, interactively chained questions, and long memory for streaming QA. 2) My- Buddy, a training-free and model-agnostic framework for companion-style VQA. It leverages ego-cues for chain-of-thought intention reasoning, driven by multi- level memory to achieve effective and efficient online assistance. 3) Key Findings and Improvements. BuddyVQA reveals key limitations in existing mod- els for companion-style VQA, and MyBuddy achieves strong improvements. 2 Related Works 2.1 Streaming VQA Streaming VQA aims to continuously interpret incom- ing video streams and answer questions that may arise at any moment (Zhang et al., 2025c; Di et al., 2025; Xiao et al., 2025; Lin et al., 2024; Hu et al., 2025; Niu et al., 2025; Zhao et al., 2025; Yang et al., 2026). Unlike traditional offline VideoQA (Xiao et al., 2021; Fu et al., 2024; Mangalam et al., 2024), where the entire video is available for question-adaptive processing, Streaming VQA must process videos online without prior knowl- edge of future questions. Consequently, it relies on a memory buffer to efficiently retain question-agnostic historical video information, enabling the model to an- swer questions that may be posed at arbitrary future time steps. Notably, the offline VideoQA task can be treated as a special Streaming VQA case where the questions are posed at the end of the videos. To facilitate companion-style QA, BuddyVQA fol- lows the streaming VQA setting but differs from ex- isting works by focusing exclusively on egocentric vi- sion (shared user view) and emphasizing a set of cross- modal chaining questions with ego-deictics that reflect the nature of real-life assistance. While the problem of chaining questions has been partially explored in con- versational or multi-turn VideoQA (Alamri et al., 2019; Qian et al., 2022; Pham et al., 2022; Christmann et al., 2019; Qu et al., 2020), a key difference and additional challenge in the companion setting is that the cross- modal chained questions and ego-deictics cannot be in- terpreted from text alone but often require other ego cues from the camera wearer, such as gaze and hand motion information. In addition, both the video streams and questions are assumed infinite, thereby simulating a more practical egocentric QA assistant. 2.2 Egocentric VQA Early egocentric VQA benchmarks primarily evalu- ate episodic memory, procedural understanding, and long-range reasoning over first-person activities (Grau- man et al., 2022; B ̈armann and Waibel, 2022; Man- galam et al., 2024). More recent MLLM-oriented stud- ies broaden this scope by collecting egocentric data and evaluating models towards embodied assistance, either semantic (Cheng et al., 2024; Zhou et al., 2025; Yang et al., 2025a; Sun et al., 2025) or spatial (Yang et al., 2025b). However, they fail to capture the ego-deictic and chained nature of real-world egocentric question answering. Although the latest works (Xiao et al., 2025, 2026b; Peng et al., 2025; Choi et al., 2026) underscore the importance of gaze and pointing gestures for in- ferring ego-deictics, these challenges are studied solely through visual information of short video clips. In con- trast, our ego-deictics are more practical and complex by naturally entangled with historical QAs rooted in egocentric streaming QA scenarios, which reflects the nature of real-world companion-style QA. Moreover, while existing targeted datasets study ego-gaze or hand pointing independently, BuddyVQA reflects both ego- cues in one benchmark. 2.3 MLLMs for VideoQA Recent MLLMs (An et al., 2026; Bai et al., 2025a; QwenTeam, 2026; Wang et al., 2025b; DeepMind, 2026; OpenAI, 2026) have substantially advanced of- fline VideoQA through improved vision-language align- ment, temporal encoding, reasoning, and so on. Some models further reduce the cost of processing extended inputs through temporal token compression, sparse sampling, hierarchical memory, or retrieval-based rea- soning for long video reasoning (Zhang et al., 2025b; Wang et al., 2025a; Li et al., 2023a; Zhang et al., 2024a; Song et al., 2025; Qin et al., 2025; Luo et al., 2025). Despite their strong performance, these models favor fixed video reasoning rather than handling dynamically growing video streams. Online and streaming Video-LLMs instead process frames incrementally and respond using only the obser- vations up to the question moments. Architectures such as Flash-VStream (Zhang et al., 2025c), Dispider (Qian et al., 2025), and related works (Xiao et al., 2023; Di et al., 2025; Kim et al., 2025; Xu et al., 2025b; Xiao et al., 2026a; Yeo et al., 2026) introduce mechanisms like memory sinks, dual-process pipelines, and KV- cache reuse for efficient real-time multimodal under- standing. Recent corresponding systems (Huang et al., 2025; Chatterjee et al., 2025; Xie et al., 2026; Wang 4Hangyu Qin et al. et al., 2026; Long et al., 2026) try to maintain long-term context under bounded computation by introducing new strategies like compressed memory, frame cluster- ing, and separated perception–decision–response mod- ules. While the latest StreamReady (Azad et al., 2026) adopts a multi-level memory design, its hierarchy pri- marily compresses past visual observations into coarse representations to improve storage efficiency, which may discard fine-grained interaction and visual cues critical for assistive video understanding where ego- deictics are frequently observed. These developments improve online perception and response efficiency, but primarily study single-turn queries, response timing, or memory retention, ignoring ego-deictics and historical QA association. A related line of work studies video-grounded di- alogue and multi-turn multimodal interaction (Qian et al., 2022; Qu et al., 2020; Liu et al., 2025; Lee et al., 2025; Sun et al., 2026; You et al., 2025). Most such methods condition each response on a concatena- tion or re-encoding of the preceding dialogue, which becomes increasingly expensive for continuous video streams. More importantly, dialogue history is typically treated as undifferentiated textual context rather than as structured dependencies between current and previ- ous questions. Though recent work investigates prompt- ing strategies with gaze annotations and hand-object detections (Peng et al., 2025; Xu et al., 2025a; Inad- umi et al., 2024), their focus is mostly on improving detection and localization accuracy, rather than un- derstanding deictic questions with visual entities to support reasoning. Consequently, BuddyVQA presents new challenges of interactive chaining and ego-deictics for real-world QA companions. Accordingly, we design a training-free framework MyBuddy tailored for chal- lenges that existing models fundamentally overlook: handling chained question dependencies and ego-deictic expressions in streaming visual context, while maintain- ing high response efficiency. 3 Dataset: BuddyVQA To support the study on egocentric companion-style QA, we first construct BuddyVQA using videos from EgoSchema (Mangalam et al., 2024) that capture daily egocentric human activities. The QAs are collected to preserve the streaming VQA setting and additionally highlight two core challenges encountered in egocentric companion-style QA: 1) Ego-Deictics, which are refer- ential or spatial pronouns in the egocentric setting, of- ten require visual anchoring of the camera-wearer’s QA and activities to resolve. For example, “What is next to him?” or “Is this on the table?”. This introduces a chal- lenge for robust referential understanding in egocentric QA. 2) Interactively-Chained questions, which re- flect contextual dependencies. The chained questions reference a previous QA pair within the same highlight moment and conversational context. This allows for re- alistic multi-turn dependencies while maintaining tem- poral coherence. Prior studies (Tanenhaus et al., 1995; Carston, 1999; Tomasello and Carpenter, 2007) in psychology have shown that humans frequently rely on such deic- tic references when communicating about objects and events in a shared environment. In BuddyVQA, the as- sistant is expected to participate in this shared percep- tual space, requiring it to infer the intended referent from the user’s egocentric perspective and embodied cues rather than language alone. Consequently, resolv- ing ego-deictic expressions demands not only conver- sational understanding but also visual grounding and perspective-aware reasoning, all under streaming VQA. 3.1 Dataset Construction Stage 1: Video Selection and Automatic Ques- tion Generation. To construct a comprehensive dataset, we select more than 1,000 videos that con- tain various activities (please refer to the Appendix for detailed activities) and life scenarios from the EgoSchema Dataset (Mangalam et al., 2024). To iden- tify key moments in each video, we first employ Gemini- 2.5-pro (Comanici et al., 2025) to detect visual highlight segments, typically ranging from 2–18 seconds. These highlight moments correspond to perceptually salient or semantically rich events (e.g., object manipulation, transitions, interactions). Fig. 3(c) illustrates the dis- tribution of detected highlight moment durations. For each highlight moment, Gemini-2.5-Pro automatically generates multiple question drafts referencing the vi- sual and temporal context. These questions serve as an initial reference for reducing human cognition burden in subsequent annotation. Please refer to the Appendix for the detailed prompts. Stage 2: Human Question Annotation. We re- cruit 25 English-speaking annotators from diverse lin- guistic and social backgrounds to annotate questions based on the automatically generated QA pairs. Each annotator receives original videos, highlight clips, and associated QA drafts, and follows a standardized proce- dure: a) watch the video segment up to the end times- tamp of the highlighted moment, without access to fu- ture frames. b) create questions according to the given QA drafts and video contents, following the principles of 1) Chained Questions: If a QA pair is labeled as Companion-style QA Assistance in Ego-Vision5 Q: Is it safeto peel like this? Answer set: No, it seems unsafe as the blade faces your hand. Itis not safe, as the kitchen counter is cluttered,increasing the risk of accidental cuts or slips. Chained QA: None Scrutinizing the present Question Ts: 1:40 01:29 01:39 00:51 00:03 Q: Where is my phone? Answer set:It is the kitchen counter, Your phone is placed next to those green cans. It is just above the cutting board. Chained QA: None Recalling the past Question Ts: 00:36 Q: Any adviceon how to use it? Answer set: Use it only in a well-ventilated area away from flames or heat sources. Shake the can before spraying and keep it upright during use. Chained QA: (What is written on the bottle?The bottle shows flammable hazard symbols, indicating a pressurized aerosol spray. ) 02:43 02:51 02:40 02:22 02:2700:32 00:22 00:14 Question Ts: 00:25 00:23 00:27 00:17 00:06 Q Q Q Q Q: How much is the price? Answer set: The price for mango is 9 SR per kilo. Chained QA: Do you think it is fresh?Themangoappears mostly fresh, thoughwitha few slight ripening spots. Recognizing visual content Advising for the next Question Ts: 02:46 Fig. 2 Examples of different QAs (Scrutinization, Recall, Recognition & Advisory) in BuddyVQA. “chained”, ensure that this QA pair is closely related to its previously chained QA (not necessarily the imme- diately preceding one); 2) Deictic Expressions: Adopt conversational phrasing (such as pronouns and deictics) to simulate spontaneous speech; 3) Assistive Questions: Questions should resemble those that a real-world ego- centric assistant might encounter; 4) Diverse Aspects: Questions within the same clip should address distinct aspects of the event, avoiding simple paraphrases. The annotations undergo at least two rounds of re- view and revision, and every accepted QA pair is ap- proved by at least one annotator and one reviewer. In the first annotation round, human annotators produced more than 67K candidate questions. Following a com- prehensive expert review, approximately 65% of these annotations were retained, while the remaining ques- tions were discarded due to issues such as ambiguity, re- dundancy, or inconsistency with the video content. The retained questions, together with detailed revision com- ments, were then returned to the original annotators for refinement. After the second annotation round, approx- imately 29K revised questions satisfied our annotation guidelines. Finally, we performed a quality control stage to further remove semantically redundant questions, re- solve duplicate or highly similar QA pairs, and ensure annotation consistency across videos, resulting in 21.6K high-quality qualified questions. We also ask the annotators to label each question into one of four types: Recall of past video and QA content, Recognition of current visual content like ob- jects, text, or people, Scrutinization of current actions or outcomes, and Advisory feedback for future actions. These categories comprehensively cover the past, cur- rent, and future time points. Stage 3: Multi-Answer Annotation. For the questions reflecting subjective intent (e.g., “What should I do next?” or “Can you give me advice on how to cook this?”) in our dataset (common in Scruti- nization and Advisory categories), we collect multiple correct answers for each of them, capturing the diver- sity of plausible responses. Specifically, each question is answered by at least three annotators. Annotators are instructed to use natural spoken language that is con- sistent with a human-assistant interaction and to avoid leaking information from future video content after the timestamp. After that, we also conduct a similarity check for all the annotated answers of each question to filter redundant answers. Specifically, we use roberta- large-mnli (Liu et al., 2019) to calculate the pairwise similarity among all the labeled answers to each ques- tion, and remove one answer when the pairwise similar- ity exceeds 0.75. By applying this approach, we aim to mitigate subjectivity while preserving natural variation in human responses. 3.2 Dataset Analysis Dataset statistics. BuddyVQA is constructed from 1,012 videos selected from EgoSchema (Mangalam et al., 2024), with 704 videos as the test set and 308 videos as the validation set. It contains a total of 21,574 questions based on 6,002 video highlight mo- ments evenly distributed in the video. The average video duration is 180 seconds, while the average high- light moment duration is 7.48 seconds with 3.6 QA pairs per moment. The average length of question/answer is 8.21/10.62 words, respectively. Each question is associ- 6Hangyu Qin et al. Table 1 VideoQA datasets. Chain: chained question. Dei: question containing ego-deictic expressions. EgoV: egocentric videos. MA: multiple ground-truth answers for each question. QC: question category. OE/MC: Open-Ended/Multi-Choice. BenchmarkV Num.Total V.Q Num.Dei(%)ChainEgoVMAQCTask Egocentric VQA Benchmarks AssistQ (Wong et al., 2022)1003.2h5318✗✓✗MC EgoSchema (Mangalam et al., 2024)5k250h5k3✗✓✗MC QAEgo4D (B ̈armann and Waibel, 2022)1.3k180h14k5✗✓✗OE EgoMemoria (Ye et al., 2025)629149h7k8✗✓✗✓MC EgolifeQA (Yang et al., 2025a)6266h6k11✗✓✗MC EgoGazeVQA (Peng et al., 2025)913-1.8k7✗✓✗✓MC EgoPointVQA (Choi et al., 2026)3000.6h67225✗✓✗✓MC Streaming VQA Benchmarks VStream-QA (Zhang et al., 2025c)3221h3.5K9✗OE CogStream (Zhao et al., 2025)1k58h58k4✗✓OE StreamingBench (Lin et al., 2024)900-4.5k2✗✓MC OVO-Bench (Niu et al., 2025)64476h2.8k4✗✓MC EgoBlind (Xiao et al., 2025)1.3k15h5.3k23✗✓OE BuddyVQA1k50h21k87.6✓OE (a) Distribution of questions w.r.t. video length (b) Occurrenceof deictic pronouns in questions (c) Distribution of highlight moment duration Fig. 3 Video and question distributions in BuddyVQA. ated with an average of 2.76 answers. Our BuddyVQA covers a variety of activities from indoor(e.g., cooking, cleaning, laundry, housekeeping) to outdoor(e.g., driv- ing, shopping, gardening, picnics), covering the vast majority of situations encountered in daily assistive contexts. We also found that many vague references are tied to the way people communicate indirectly, either through pointing or eye gaze direction (see Fig. 2). A notable 87.6% of questions contain at least one de- ictic pronoun (Fig. 3(b)), highlighting the prevalence of implicit references in egocentric communication. 79.1% of questions are part of a chained node (Fig. 3(a)). Meanwhile, we also conduct a human study on the val- idation set to ask humans to answer the chained ques- tions with and without base QAs. The results reveal that 57.6% of Interactively Chained questions cannot be correctly answered without base QAs, underscoring their dependency. Fig. 3(a) also shows that the proportion of inter- actively chained questions increases with video length, reflecting growing contextual dependency over time. Fig. 2 presents the distribution of question types, in- dicating a balanced composition and highlighting that BuddyVQA contains a significant portion (50.4%) of subjective (Scrutinization and Advisory) questions. At the same time, the majority of chained questions are separated by short to medium gaps, with 37.8% oc- curring after 3–5 intervening QAs and 19.3% after 6–8 intervening QAs, indicating that dependencies are of- ten distributed across multiple turns rather than imme- diately adjacent. Only 12.7% are directly consecutive (0 gap), suggesting that modeling longer-range inter- actions is necessary. Please refer to the Appendix for more detailed analysis. Dataset Comparison. Tab. 1 highlights the key innovation of BuddyVQA: Interactively Chained ques- tions and greater diversity of Ego-Deictics that reflect practical human speech patterns in assistive scenar- ios. Additionally, BuddyVQA provides multiple candi- date answers to reduce subjective evaluation, capturing the inherent uncertainty of natural human questioning. These design choices make BuddyVQA a more realistic and challenging benchmark for developing companion- based QA systems that must reason dynamically and handle subjectivity in practical scenarios. In contrast, most existing datasets have a negligible proportion of questions containing ego-deictic expressions. While EgoBlind (Xiao et al., 2025) and EgoPointVQA (Choi et al., 2026) exhibit relatively higher proportions, they focus on isolated, history-free visual references without chained cases. Furthermore, EgoBlind is limited to visu- ally impaired users, and EgoPointVQA’s questions are restricted to point-based reference queries with mostly synthetic videos. Therefore, both are less representative of the diverse, assistive interactions required in in-the- wild egocentric companion scenarios. 4 Method: MyBuddy 4.1 Overview Under the formulation of companion-style QA in streaming setting, we emphasize the following funda- mental challenges: 1) Deictic and Vague Reference Resolution: The model needs to identify, through vi- Companion-style QA Assistance in Ego-Vision7 Visual Information Retrieval Q t Low-Active Memory (Long-term) u 1 ... u 2 u 3 u T’ T’ = t –Δ-Γ Multi-LevelMemory Semi-Active Memory (Mid-term) ... T’ = t –Δ V’ T’-Γ V’ T’-Γ+1 V’ T’-Γ+2 V’ T’-Γ Offline Summarization Active Memory (Short-term) ... T’ = t Clustering V T-Δ V T-Δ+1 V T-Δ+2 V T MLLMs for Offline Visual Encoding Historical QA Buffer Store and maintain historical QAs Question Filter Online question routing and filtering QuestionQ t : How can I clean it? MLLMs for CoTOnline Reasoning Ego-cues Extraction Context QAs AnswerA t : You can use the water tap at the corner of the yard to clean the bucket. LocalQAs DistantQAs Key Visual Retrieval Fig. 4 Framework of MyBuddy. MyBuddy follows a memory-and-retrieval augmented QA pipeline. When a question arrives, it searches from the multi-level memory and historical QA buffer for related visual and textual contexts to infer the question meaning and drive the answer via multimodal chain-of-thought reasoning. sual cues, how to resolve deictic references. 2) Chained Questions: Reasoning often requires coherent links across historical QAs and visual contexts. 3) Response Efficiency: The model must process and recall stream- ing sequences without excessive latency or memory ex- pansion when video length grows. To tackle these challenges, we design a companion- based QA framework (Fig. 4) MyBuddy. MyBuddy gen- erally follows a memory-and-retrieval augmented QA pipeline and targets the challenges of BuddyVQA with the following additions: 1) Multimodal CoT Reasoning, which resolves ego-deictic expressions and rephrases an original vague question into a clear one by associat- ing ego-gaze and hand-pointing regions with relevant historical interaction information. 2) Question Filter, which helps identify relevant historical questions to un- derstand a chained question efficiently via a small LLM. 3) Multi-Level Heterogeneous Memory, which improves streaming efficiency (both memory and online QA) by allowing quick and adaptive access to the recent, mid- term, and long-term visual contents in heterogeneous representations (e.g., visual embeddings, dense cap- tions, and summaries). Together, MyBuddy maintains a long-term and efficient understanding of the user’s visual experiences to infer their real-time intents, effec- tively supporting companion-style QA. 4.2 Multi-Level Heterogeneous Memory Since the memory module acts as a prerequisite for sup- porting contexts to facilitate online QA, we introduce it first for better understanding. Given a sequence of video frames V t =f 1 ,...,f t sampled at a fixed frame rate from a video stream, we obtain their feature repre- sentations by a vision encoder v t =E v (f t )∈R d , where d is the embedding dimension. To efficiently store in- formation and support online streaming QA, we design a three-level heterogeneous memory system consisting of Active, Semi-Active, and Low-Active memories: M =M A ,M S ,M L ,(1) with each maintaining information at a different tem- poral granularity and semantic abstraction (illustrated at the top of Fig. 4). Upon usage, memory tiers are prioritized by temporal recency, ensuring that the most recent observations dominate the reasoning process. Active Memory (Short-term, 0–∆). The Active Memory (ActMem) retains the most recent visual frame embeddings for frequent access: M t A =v ˆ t | ˆ t∈ [t− ∆,t],(2) where ∆ is a hyperparameter that specifies the tempo- ral span covered by the short-term active memory. Semi-Active Memory (Mid-term, ∆–Γ ). The Semi-Active Memory (Semi-ActMem) is less visited, and stores aggregated visual contents that exceed the coverage of ActMem for efficiency. Regarding aggrega- tion, an incremental version of the incremental density- based clustering algorithm DBSCAN (Ester et al., 1998) is applied. DBSCAN is density-adaptive and al- lows the model to automatically identify meaningful temporal clusters without predefined thresholds or clus- ter counts, allowing the number of clusters to grow dy- 8Hangyu Qin et al. namically along the video stream. Conceptually, Semi- ActMem is obtained by M t S = DBSCAN(M τ S ,M ˆ t A | τ ∈ [0,t− ∆], ˆ t∈ [t− Γ − ∆,t− ∆]| ε,minPts), (3) where M τ S is the maintained Semi-ActMem up to time τ , and M ˆ t A denotes the newly arrived ActMem within the temporal interval ˆ t. Instead of re-clustering all his- torical memories, we use DBSCAN to incrementally up- date the maintained clustering state by incorporating the newly arriving memories into the existing density- based clusters. Here, ε denotes the neighborhood radius, while minPts specifies the minimum number of neigh- boring samples required to form a dense cluster. There- fore, the semi-active memory at time τ is represented as a set of density-based clusters M τ S =C τ 1 ,...,C τ K τ . Each cluster C τ i = (r τ i ,E τ i ) consists of a representative feature r τ i = 1 |E τ i | P e∈E τ i e, and its associated mem- ber features E τ i . The representative feature serves as a compact semantic index for efficient retrieval, while the member features preserve fine-grained visual evidence for downstream reasoning. Low-Active Memory (Long-term, ≥Γ ). The Low-Active Memory (Low-ActMem) accumulates long- term knowledge through temporally overlapping sum- maries, allowing recurring activities and persistent con- textual information to be naturally reinforced over time. Specifically, MyBuddy utilizes a zero-shot sum- marization moduleS to produce textual summaries and gather visual clues u j into memory M L = u j K j=1 , where u j =S(M ˆ t S | ˆ t∈ [t− ε− Γ − ∆,t− Γ − ∆]).(4) Adjacent summarization windows overlap by ε, allow- ing activities that persist across multiple windows to be repeatedly incorporated into successive summaries and naturally retained in the long-term memory. K is the number of summarization windows over the inter- val from T = 0 to T = t− Γ − ∆. Summaries inside the Low-ActMem retain key objects and actions, spa- tial relations, temporal anchors, and so on to facilitate long-term inquiries (please refer to the Appendix for detailed prompts). Unlike ActMem and Semi-ActMem, the summarization process in Low-ActMem is executed during idle intervals (when the QA system is not ac- tively processing incoming questions). Consequently, it functions outside the question–answering pipeline, al- lowing the model to consolidate contextual information or update memory representations without introducing additional latency to the real-time interaction. The design of multi-level heterogeneous memory fo- cuses on what to save into memory (features, events, summaries) and how memory is structured (active, semi-active, low-active). The three levels of memory effectively retain critical visual representations while simultaneously contributing to improved memory and time efficiency. Considering the balance between response efficiency and performance, our design decouples memory mainte- nance from online question answering. Specifically, the captioning and consolidation of Low-ActMem and the incremental clustering of Semi-ActMem are performed asynchronously during idle intervals, rather than be- ing synchronized with incoming user questions, and therefore introduce no additional response latency. At query time, vision-to-text conversion is restricted to re- cent ActMem features and question-relevant features selected through the representative cluster centers in Semi-ActMem, while Low-ActMem directly filters its pre-computed textual summaries. Consequently, each query processes only a small set of relevant visual fea- tures while retaining multi-scale historical evidence for accurate long-term reasoning. This design limits online computation to question-relevant evidence while pre- serving both fine-grained recent observations and com- pact long-term context. 4.3 Historical QA and Question Filter We maintain a single historical QA buffer B = (Q i ,A i ) t−1 i=1 to store all QA pairs. For an incoming question Q t , we apply a question filter (e.g., a small MLLM) to identify its relevant QAs from the buffer to serve as contexts. Specifically, we dynamically organize the historical QA pairs into short-term and long-term retrieval can- didates according to temporal recency to Q t , thereby characterizing the strength of conversational depen- dency. We define the distance between a historical QA pair (Q i ,A i ) and Q t as the number of intervening ques- tions: d q (i,t) = t− i− 1.(5) Then, the query-dependent short-term and long-term QA candidate sets are constructed as B s (Q t ) =(Q i ,A i )∈ B | d q (i,t)≤ τ q , B l (Q t ) =(Q i ,A i )∈ B | d q (i,t) > τ q , (6) where τ q denotes the number of questions covered by the short-term window. Note that B s (Q t ) and B l (Q t ) are temporary retrieval candidates of Q t , which will be updated dynamically with regard to the incoming questions. Companion-style QA Assistance in Ego-Vision9 After that, the question filter first searches only the short-term candidate set for the incoming question Q t : Q c,s t = MLLM(Q t ,f t ,B s (Q t )).(7) A return is considered meaningful if the filter identifies a historical dependency and retrieves at least one se- mantically relevant QA pair with a valid identifier from the current candidate set. Otherwise, the filter searches the long-term candidate set for potential long-range de- pendencies: Q c t = ( Q c,s t ,Meaningful(Q c,s t ) = 1, MLLM Q t ,f t ,B l (Q t ) , Meaningful(Q c,s t ) = 0. (8) If the two-stage search fails to return any relevant his- torical QA pair, Q t is treated as an independent ques- tion that is directly sent to the main MLLM for answer generation. Otherwise, Q t is recognized as chained, and the retrieved Q c t is used in the subsequent reasoning step. This query-dependent active-first strategy signif- icantly reduces the historical context, while the long- term fallback preserves access to the complete interac- tion history for long-range chained questions. 4.4 Multimodal CoT Reasoning The CoT mechanism is facilitated by the aforemen- tioned memory and question filter modules. It plays a pivotal role in decoding the deictic expressions and inferring the user’s intent behind the questions. Ego-cues extraction. To resolve ego-deictic ex- pressions, the model must infer the object or region implicitly referred to by the camera wearer. In egocen- tric videos, embodied ego-cues like eye gaze and hand interactions often naturally reveal the wearer’s focus of attention. In particular, eye gaze indicates where the wearer is looking, while hand interactions further con- strain the object currently being manipulated or re- ferred to. Motivated by this observation, we explicitly extract these ego-cues from the active memory M A by prompting MLLM D: V ego t =D(M A ,Q t ).(9) Here, V ego t consists of the detected eye gaze point and hand bounding boxes that are relevant to the current question. These ego-cues explicitly indicate the user’s focus of attention and are subsequently used to guide the grounding of ego-deictic references before answer generation. Please refer to the Appendix for detailed prompts. Key visual retrieval. To achieve CoT reasoning for an incoming question, MyBuddy retrieves key visual information from the multi-level memory. Specifically, we retrieve and generate visual descriptions on demand according to the current question: V des t = D M t A ;D O S M t S ,Q t ;D M t L ;Q t . (10) For visual features in ActMemM t A , we directly convert them into descriptions via a small MLLMD. For visual features in Semi-ActMem M t S , we first filter the top-k cluster centers closest to the question and gather their member features through a two-step retrieval module O S . Specifically, given the question representation q t , O S first computes sim(q t , r t i ) for each of the K t clus- ters and selects the top-K cluster indices I t . It then gathers the member features of the selected clusters as f M t S = S i∈I t E t i , where r t i and E t i denote the represen- tative feature and the original member video features of cluster C t i . The retrieved features in f M t S are then converted into descriptions using D. For visual sum- marization in Low-ActMem M t L , we directly filter and identify the descriptions relevant to Q t using D. Both D and O S are implemented in one lightweight MLLM. CoT Reasoning for Answer. With the retrieved ego-cues (eye gaze and hand pose) V ego t , visual infor- mation V des t , and context QAs Q c t , we leverage CoT prompting (please refer to the Appendix for specific prompts) to clarify deictic references, i.e., rephrase the vague question into a disambiguated variant ˆ Q t (e.g., “where is it?” to “where is the cup?”): ˆ Q t =R(Q ∗ t ,G t ),(11) and then answer the rephrased question: A t =R( ˆ Q t ,G t ),(12) where G t = [V des t , V ego t ,Q c t ], and the MLLM R is the main MLLM for answering. Note that the resulting an- swer A t will be stored in the historical QA buffer as context for future question answering. 5 Experiments 5.1 Experiment Setup Evaluation. We use GPT-5 (OpenAI, 2025a) as the answer judge to get the prediction accuracy (0-100%, the percentage of correct answers) and confidence score (0-5, with 5 being the highest confidence). Notably, our evaluation prompts are manually refined to align closely with human evaluation results for answer scoring 10Hangyu Qin et al. Table 2 Performance of MLLMs and Humans on BuddyVQA. MyBuddy outperforms both general and streaming models across all four assistance types, and shows more stable performance as video stream grows. ModelSize Sampling Q. CategoriesV. Duration Overall Recall Recognition Advisory Scrutinization 0-60s 60-120s 120-180s Human--92.492.191.992.092.092.192.392.1 General VideoChat2 (Li et al., 2024b)8B1640.742.233.331.447.838.233.138.9 Video-LLaMA3 (Zhang et al., 2025a) 7B1 FPS41.744.136.434.548.540.134.739.2 LongVA (Zhang et al., 2024a)7B3246.546.139.440.947.640.139.143.0 LLaMA-VID (Li et al., 2024c)7B1 FPS45.646.739.340.951.344.138.543.1 LVAgent (Chen et al., 2025)-Adaptive47.848.740.341.453.246.439.844.6 Qwen3.5 (QwenTeam, 2026)9B1 FPS48.749.343.143.955.147.840.147.2 Gemini-3.1-pro (DeepMind, 2026)-1 FPS50.851.743.444.753.245.843.148.3 InternVL-3.5 (Wang et al., 2025b)8B3254.1 51.344.746.158.346.540.248.8 GPT-5.5 (OpenAI, 2026)-6052.253.448.946.057.349.242.150.5 Streaming Cogstream (Zhao et al., 2025)7B1 FPS50.247.638.240.148.244.041.645.0 Dispider (Qian et al., 2025)7B1 FPS46.245.537.939.345.440.139.843.2 Flash-VStream (Zhang et al., 2025c)7B1 FPS52.251.944.646.754.349.345.250.8 ReKV(LLaVA-OV) (Di et al., 2025)7B1 FPS54.6 54.144.944.454.050.148.652.7 MyBuddy(Ours) MyBuddy(Qwen3.5)9B1 FPS59.960.855.854.261.859.258.360.7 MyBuddy(Gemini-3.1-pro)-1 FPS64.063.960.159.765.363.161.963.2 MyBuddy(GPT-5.5)-1 FPS65.166.563.763.065.964.663.865.1 (please refer to the Appendix). We mainly analyze My- Buddy and existing methods on BuddyVQA. For com- parison models, we follow their native video sampling protocols whenever applicable. Models supporting a temporal-rate sampling process the observed video pre- fix at 1 FPS, while fixed-frame models uniformly sam- ple their prescribed number of frames. LVAgent (Chen et al., 2025) adopts its original adaptive retrieval strat- egy. All models are restricted to video observations be- fore the corresponding question timestamp to ensure fair and reproducible evaluation. We also extend our ex- periments to VStream-QA (Zhang et al., 2025c), Video- MME (Fu et al., 2024) and EgoSchema (Mangalam et al., 2024) to validate MyBuddy’s generalization to general streaming and video QA tasks. Implementation. We choose three strong MLLMs (GPT-5.5 (OpenAI, 2026), Gemini-3.1-pro (DeepMind, 2026) and Qwen3.5 (QwenTeam, 2026)) as our main MLLMs for online CoT reasoning and QA. Qwen3.5- 2B (QwenTeam, 2026) is employed to obtain the summarizations in the low-active memory, which is performed offline during idle intervals and therefore does not affect online QA latency. The lightweight MLLM D used for visual retrieval and question filter- ing is Qwen3.5-0.8B (QwenTeam, 2026) for the trade- off between efficiency and question-conditioned visual grounding ability. ∆ and Γ in the Multi-Level Memory are set to 30s and 120s for BuddyVQA, respectively. ε and minPts in DBSCAN (Ester et al., 1996) are set to 0.27 and 5. We set τ q = 8 by default on BuddyVQA based on preliminary analysis. Other evaluated models followed the streaming QA paradigm, receiving video segments and historical QAs chronologically as input. All models were evaluated zero-shot, and all hyperpa- rameters are selected on the validation set through pre- liminary experiments to balance QA accuracy and on- line efficiency. 5.2 Result Analysis 5.2.1 Comparison on BuddyVQA Human performance. To verify the reliability of BuddyVQA, we first conducted a human evaluation on a random 30% of questions (as shown in Tab. 2). Each QA pair was independently reviewed by human evalu- ators under the streaming setting. The results show a human-verified accuracy of 92.1%, indicating that most QA pairs align well with visual evidence in each video segment. Analyses of the remaining 7.9% of incorrect or ambiguous cases reveal two main error sources: (a) overly vague ego-deictics amid scenes with multiple sim- ilar objects or people, leading to referential confusion (see Fig. 5); and (b) motion blur or partial occlusions in some videos cause uncertain visual interpretation. De- spite these imperfections, the strong human–annotation agreement demonstrates that BuddyVQA provides re- liable and high-quality supervision for the companion- based QA research. Baseline Comparison. Tab. 2 and Fig. 5 compare MyBuddy with existing general-purpose and streaming VQA models across four question categories and three video duration ranges. Overall, MyBuddy (GPT- 5.5) achieves the highest accuracy of 65.1%, surpass- ing the strongest streaming baseline ReKV (LLaVA- OV) (Di et al., 2025) by 12.4%. Even other ver- sions—MyBuddy (Qwen3.5) and MyBuddy (Gemini- 3.1-pro)—consistently outperform all baselines, reach- ing 60.7% and 63.2%, respectively. Compared to prior Companion-style QA Assistance in Ego-Vision11 VisualizationofHandbboxGazedot Q: How should I organize my stuff? GT set: Organize these shoes and bags by their categories. Keep heavier items low and near the back seats. Use bins or trunk organizers to group similar items. ChainedQA：None GPT-5.5(General): Use bins or trunk organizers to group similar items by categories. [Acc: Yes, Score: 5] Intern-VL (General): Store tools and groceries together. [Acc: No, Score: 1] Flash-VStream(Streaming): Keep shoes and clothes clean inside the trunk.[Acc:No, Score: 2] ReKV (streaming): You can organize these stuffs by their shapes and weights. [Acc: Yes, Score: 4] MyBuddy: Organize these stuffs by its shapes and weights to fully use the space.[Acc: Yes, Score: 5] Human: Keep the shoes and bags together and squeeze to have more space.[Acc: Yes, Score: 5] Q: Where is it? GT set: Thetent is next to the trunk, on your right. ChainedQA: (Q: Where could I put those stuff? A: You can put it into the tent.) Gemini-3.1-pro (General): Inside the tent. [Acc: No, Score: 2] Intern-VL(General): It is hanged on the side of the trunk. [Acc: No, Score: 1] Flash-VStream(Streaming): The tent is on the ground, on your right- handside. [Acc: Yes, Score: 5] ReKV(Streaming): It is on the right side of the black trunk. [Acc: Yes, Score: 4] MyBuddy: The tent has been set up nexttothe black trunk. [Acc:Yes, Score: 5] Human: Inside the trunk of your car. [Acc: No, Score: 2] 03:00 Ts: 00:27 Ts: 00:29 Ts: 00:37 Q Category: Advisory Ts: 00:37 Category:Scrutinization Ts: 02:30 Ts: 02:15Ts: 02:23 Ts: 02:29 Q ... Fig. 5 Result visualization on BuddyVQA. Streaming models generally perform better than general models over time, and MyBuddy is able to process interactively chained questions and ego-deictics regardless of the streaming length. Table 3 Performance comparison on chained and ego-deictic questions in BuddyVQA. MyBuddy consistently improves both capabilities, with particularly pronounced gains on ego- deictic questions. ModelCha. (%) Dei. (%) All (%) LongVA (Zhang et al., 2024a)41.538.743.0 VLLaMA3 (Zhang et al., 2025a)36.934.239.2 LLaMA-VID (Li et al., 2024c)41.839.443.1 Qwen3.5 (QwenTeam, 2026)43.842.647.2 Gemini-3.1-pro (DeepMind, 2026)46.142.648.3 GPT-5.5 (OpenAI, 2026)47.643.950.5 InternVL-3.5 (Wang et al., 2025b)47.243.148.8 Cogstream (Zhao et al., 2025)43.840.245.0 Dispider (Qian et al., 2025)41.138.943.2 ReKV (Di et al., 2025)50.948.652.7 MyBuddy(Qwen3.5)57.358.960.7 MyBuddy(Gemini-3.1-pro)59.760.463.2 MyBuddy(GPT-5.5)62.863.565.1 streaming models (45.0–52.7%), MyBuddy achieves substantial gains under the same 1FPS constraint. While general-purpose models like InternVL-3.5 (Wang et al., 2025b) (48.8%) and LLAMA-VID (Li et al., 2024c) (43.1%) benefit from larger context windows, they still lag behind, suggesting that frame sampling alone cannot ensure robust temporal reasoning without dynamic context modeling. Across question categories, MyBuddy shows bal- anced improvements. Most existing models struggle with Advisory and Scrutinization types (6.7–9.2% be- low Recall and Recognition), reflecting the challenge of subjective reasoning. In contrast, MyBuddy achieves 63.7% and 63.0% on these types, outperforming ReKV (LLaVA-OV) (Di et al., 2025) by 18.8% and 18.6%, respectively. This indicates strong capability in han- dling subjective, context-dependent queries through its hierarchical prompting and memory-based context in- tegration. Regarding video duration, most models de- grade over time. For example, ReKV (LLaVA-OV) (Di et al., 2025) drops from 54.0% to 48.6%, and InternVL- 3.5 (Wang et al., 2025b) from 58.3% to 40.2% when the video length increases from 60s to 180s. This suggests that the accumulation of visual and contextual informa- tion over longer sequences imposes memory and reason- ing burdens that most models are not equipped to han- dle. Their intermediate representations drift, temporal dependencies are not retained reliably, and key details from earlier segments become inaccessible. In contrast, MyBuddy (GPT-5.5) remains stable (64.6%→63.8%), demonstrating superior temporal consistency and long- term dependency modeling. We further analyze MyBuddy on the subset of chained and ego-deictic questions. Table 3 shows that MyBuddy consistently outperforms all baselines on both challenging subsets. With GPT-5.5, MyBuddy ex- ceeds the strongest baseline ReKV (Di et al., 2025) by 11.9% and 14.9% on chained and deictic questions, re- spectively. The gains are especially pronounced on ego- deictic queries, validating the strength of combining ego-cues with history-aware reasoning. Moreover, the improvements remain consistent across different back- bones, demonstrating MyBuddy’s model-agnostic effec- tiveness. 12Hangyu Qin et al. Table 4 Dynamic streaming evaluation on BuddyVQA. We report accuracy (%) under three settings for better comparisons: 1) No Feedback: No user feedback of historical questions, 2) Correctness: Binary feedback (correct or not) of historical questions, and 3) GT Answer: Ground-truth answers of historical questions are provided by users. The best and second-best results are highlighted in bold and underline, respectively. Model No FeedbackCorrectnessGT Answer OverallOverall0–90s> 90sOverall0–90s> 90s Qwen3.5 (QwenTeam, 2026)47.249.051.845.952.354.250.1 Gemini-3.1-pro (DeepMind, 2026)48.350.453.047.454.055.951.7 GPT-5.5 (OpenAI, 2026)50.553.155.050.856.958.255.3 MyBuddy (Qwen3.5)60.763.263.862.567.166.268.2 MyBuddy (Gemini-3.1-pro)63.266.066.465.569.868.870.9 MyBuddy (GPT-5.5)65.168.068.367.672.170.973.5 5.2.2 Dynamic Streaming Evaluation To analyze companion QA performance in realistic as- sistive scenarios, we additionally conduct a human-in- the-loop evaluation and use ground-truth (GT) answers to simulate the human feedback of historical QAs in streaming environments. We analyze and compare three settings: 1) No feedback (by default): the historical questions and model predictions are used as contexts for answering the current question, 2) Binary feedback: the historical questions and model responses, together with the correctness of the responses, are used as con- texts, and 3) GT Answer: the historical questions and their corresponding GT answers are used as contexts. As shown in Table 4, MyBuddy consistently benefits more from user feedback than the standalone back- bone models. With correctness feedback, MyBuddy (GPT-5.5) improves from 65.1% to 68.0%, showing that even a binary signal helps the model identify unreli- able historical responses and avoid propagating pre- vious errors. When users directly correct the wrong answers with the correct ones, the accuracy increases to 72.1%, outperforming GPT-5.5 under the same set- ting by 15%. The advantage is most pronounced after 90 seconds: while the standalone backbones still suf- fer from error accumulation in the growing interaction history, MyBuddy (GPT-5.5) reaches 73.5%, exceed- ing its 0–90s performance of 70.9%. Similar improve- ments are observed with the Qwen3.5 and Gemini-3.1- pro backbones, demonstrating that the benefit is model- agnostic. We attribute these gains to MyBuddy’s ques- tion filter and structured multi-level memory, which re- tain useful corrections, suppress irrelevant or incorrect historical responses, and retrieve the feedback most rel- evant to subsequent chained questions. 5.2.3 Extension to General Streaming and Video QA RVS-Ego and RVS-Movie. To further validate the streaming performance on long videos, we also evalu- ate MyBuddy on VStream-QA (Zhang et al., 2025c) comprising RVS-Ego and RVS-Movie, where the video length ranges from 30 minutes to 60 minutes. Tab. 5 shows that MyBuddy (GPT-5.5) achieves SOTA accu- racy of 66.1% on RVS-Ego and 58.9% on RVS-Movie, outperforming all competitors, including base GPT-5.5 (OpenAI, 2026). While Flash-VStream (Zhang et al., 2025c) attains a 2.4s latency, MyBuddy models operate at 3.0–3.4s, offering a balanced trade-off between speed and accuracy. All these results emphasize the overall abilities of MyBuddy on long streaming videos with dif- ferent characteristics. Video-MME and EgoSchema. For validating and extending the generalization of MyBuddy be- yond companion-based QA, we additionally evaluate its performance on standard VideoQA datasets Video- MME (Fu et al., 2024) and EgoSchema (Mangalam et al., 2024). We extend MyBuddy to these datasets by treating the full video as a completed stream before the question. Tab. 6 shows that MyBuddy demonstrates competitive and, in several cases, superior performance compared to existing SOTA methods across multiple benchmarks. On Video-MME, MyBuddy (Gemini-3.1- pro) surpasses strong baselines such as LVAgent (Chen et al., 2025) (81.7% / 86.6%) and GPT-5.5 (Ope- nAI, 2026) (77.3% / 82.6%), while maintaining consis- tent performance across datasets. On EgoSchema, My- Buddy (GPT-5.5) reaches 82.9%, matching the best re- ported performance of LVAgent (Chen et al., 2025) and exceeding other large-scale MLLMs like GPT-5.5 (Ope- nAI, 2026) (79.8%). These results indicate that My- Buddy also performs competitively on general VideoQA benchmarks. 5.3 Ablation Study We ablate MyBuddy on BuddyVQA w.r.t. interactively chained questions and questions containing ego-deictic expressions in Tab. 7. The largest performance drop Companion-style QA Assistance in Ego-Vision13 Table 5 Comparison on RVS-Ego and RVS-Movie (Zhang et al., 2025c). Acc.: Accuracy. Sco.: Score. Latency & GPU Usage are tested on Tesla A100 GPU. Results are averaged on 10 videos with each 720 frames as inputs. Model RVS-EgoRVS-Movie Latency (↓)GPU Usage (↓) Acc.Sco.Acc.Sco. Flash-VStream (Zhang et al., 2025c)58.94.056.03.32.4s20GB ReKV(LLaVA-OV) (Di et al., 2025)63.74.054.43.63.3s38GB LongVA (Zhang et al., 2024a)53.23.143.03.26.3s37GB MovieChat (Song et al., 2025)52.23.439.12.38.8s40GB LLaMA-VID (Li et al., 2024c)53.43.948.63.311.9s33GB Qwen3.5 (QwenTeam, 2026)57.23.351.63.314.8s51GB GPT-5.5 (OpenAI, 2026)59.23.848.23.4-- MyBuddy (Qwen3.5)64.83.755.83.73.0s23GB MyBuddy (Gemini-3.1-pro)65.33.757.23.63.4s- MyBuddy (GPT-5.5)66.13.958.93.63.4s- Table 6 Model comparison on Video-MME (Fu et al., 2024) and EgoSchema (Mangalam et al., 2024). Video-MME: “with- out subs / with subs”. The best and second-best Accuracy results are bolded and underlined, respectively. ModelVideo-MMEEgoSchema Video-LLaMA3 (Zhang et al., 2025a)47.9 / 49.757.3 VideoChat2 (Li et al., 2024b)39.5 / 43.856.7 MovieChat (Song et al., 2025)38.2 / -53.5 Qwen3.5 (Bai et al., 2025b)71.2 / 74.368.1 GPT-5.5 (OpenAI, 2026)77.3 / 82.679.8 LongVA (Zhang et al., 2024a)52.6 / -- Video-XL (Shu et al., 2025)55.5 / 61.0- Dispider (Qian et al., 2025)57.2 / -55.6 LVAgent (Chen et al., 2025)81.7 /86.682.9 MyBuddy (Qwen3.5)71.3 / 75.547.8 MyBuddy (Gemini-3.1-pro)82.5 / 87.382.2 MyBuddy (GPT-5.5)81.1 / 84.982.9 occurs when historical QA is not given, causing accu- racy fall from 62.8% to 44.5% on chained questions, highlighting the importance of temporal linkage across chained QA pairs. In contrast, unchained questions re- main relatively stable with a minor performance drop (-4.0%), confirming that historical QA specifically ben- efits chained reasoning. Meanwhile, removing the hi- erarchical rephrasing results in noticeable declines for both chained (-10.1%) and deictics (-12.6%) questions, demonstrating the necessity of linguistic reformulation for resolving referential queries. Similarly, ablating em- bodied ego-cue extraction reduces deictic accuracy by 9.1%, showing the usefulness of embodied cues that en- hance grounding of ego-deictics by guiding attention to salient visual regions. The multi-level memory ablations further reveal an efficiency–accuracy trade-off. Efficiency Analysis. While efficiency is vital for the streaming setting, we measure the latency and GPU usage with respect to video length for MyBuddy effi- ciency analysis, as shown in Fig. 6. Latency is mea- sured from question input to response completion. GPU Usage indicates peak GPU memory usage during in- ference. MyBuddy demonstrates a clear advantage in both latency and GPU efficiency compared to existing streaming and long-context models, with the employ- Table 7 Ablation of MyBuddy on BuddyVQA w.r.t interac- tively chained questions and questions with deictic expres- sions. AM/SM/LM: Active/Semi-Active/Low-Active Mem- ory. For the setting of w/o SM & LM & AM, we use the question frame as input. Ablation Model Chain (%)Deictic (%) ✗✓✗✓ MyBuddy(Full)66.462.866.163.5 w/o LM65.761.363.358.9 w/o SM & LM64.158.057.855.8 w/o AM & SM & LM59.253.153.452.9 w/o Historical QA62.444.561.354.3 w/o Embodied Ego-cues55.354.256.254.4 w/o Hierarchical Rephrasing54.752.758.050.9 Fig. 6 Streaming VQA latency & GPU Usage w.r.t. number of frames, tested on A100 GPUs. MyBuddy is able to process long streams more efficiently. ment of multi-level memory and an on-demand pro- cessing mechanism that are beneficial for practical de- ployment. For general MLLMs such as LLaMA-VID (Li et al., 2024c), the latency scales up with the number of frames as their structures require processing all frames in a single pass (Zhang et al., 2025c). Different from them, Streaming VQA models like ReKV (Di et al., 2025) enable processing streaming videos of more than 1,000 frames, yet still occupy a considerable amount of computational resources. To further analyze the latency and computational cost of each module in MyBuddy, we also conduct a breakdown efficiency analysis (as shown in Tab. 8). Re- sults show that our module design offsets the additional cost resulting from retrieval and question rephrasing, 14Hangyu Qin et al. Table 8 Efficiency breakdown on A100 GPUs. Extract Cost denotes additional overhead (latency/memory) introduced by individual modules. Base (Qwen3.5-9B): processes video frames & concatenates all historical QAs for each question online. ConfigurationAcc.Chained Acc.LatencyGPU UsageExtra Cost Base (Raw F+His. QAs)47.243.83.9s51.1GB– + Multi-Level Memory51.448.22.5s22.9GB-1.4s / -28.2GB + Question Filter53.654.52.3s30.2GB-1.6s / -20.9GB + Key Visual Info Retrieval55.752.74.0s62.3GB+0.1s / +11.2GB + Hierarchical Rephrasing56.953.44.3s53.8GB+0.4s / +2.7GB MyBuddy(Full)60.757.32.6s29.1GB-1.3s / -12.0GB Table 9 Step-wise failure analysis of MyBuddy on the subset of BuddyVQA. While early-stage errors exist, a significant portion of failures arises from intrinsic ambiguity and visual challenges, rather than pipeline design alone. FailureNumRatio (%)Main Reasons Que. Filter1246.2Misclassifying chained questions Vis. Retri.42021.0Missing relevant frames or objects Rephrase50425.2Wrong referent grounding Ans. Infer55627.8Correct context, wrong inference Others39619.8Motion blur / Noisy input as they dispense with online long video processing and long-context LLM inference. For example, Tab. 8 Row 1 (Base) vs. Row 3 shows that the Question Filter brings advantages in efficiency (-1.6s, -20.9GB) and ac- curacy (+6.4%, especially chained accuracy, +10.7%) compared with naive concatenation, resulting from sig- nificantly reduced noise and context length. Overall, the performance gains from modules in MyBuddy outweigh the costs in system complexity and inference time. 5.4 Case Study Failure case analysis. We first carry out a man- ual failure analysis over 2,000 BuddyVQA failure cases from the validation set by examining error outputs at each stage, and trace the causes in Tab. 9. These results suggest that the Question Filter has the lowest failure rate, indicating limited error propagation from early stages. Most failures originate from rephrasing and an- swer inference, showing that ego-deictic grounding and chained reasoning are the core bottlenecks, which is consistent with the intended diagnostic focus of the BuddyVQA benchmark. Case visualizations. We also visualize additional qualitative results across the major categories in Bud- dyVQA in Fig. 7 and Fig. 8. In many cases, we observe that deictic expressions do introduce a substantial chal- lenge for existing VideoQA models. Interestingly, even when two questions are semantically identical and dif- fer only in the way the referent is expressed (i.e., an ex- plicit noun versus a deictic expression), most existing MLLMs often exhibit a substantial performance gap. This is also reflected in the differing performance gaps between MyBuddy and baselines on BuddyVQA(e.g., MyBuddy 65.1% vs GPT-5.5 50.5%) and on general VideoQA benchmarks (e.g., MyBuddy 84.9% vs GPT- 5.5 82.6% on Video-MME). Error analysis indicates that the majority of failures for current models orig- inate from incorrect referent grounding rather than de- ficiencies in semantic reasoning. Specifically, the mod- els frequently confuse visually similar objects, persons, events, or scene contexts, causing subsequent reasoning to operate on incorrect evidence. These findings demon- strate that deictic expressions expose a fundamental limitation of current VideoQA systems and motivate the need for dedicated methods that jointly resolve ref- erential ambiguity and perform grounded reasoning. For chained questions, we find that the difficulty is further amplified, as many baseline models strug- gle to correctly interpret the referential structures em- bedded in multi-turn dialogue. In scenarios where ego- deictic expressions point back to earlier answers, correct grounding requires explicit reasoning over the past QA context rather than isolated single-step inference. Some baseline models fail to genuinely resolve the intended referent, leading to semantically incorrect responses de- spite the relevant visual information being present in the video. These errors underscore the importance of robust co-reference understanding and dialogue-aware reasoning in egocentric interactions, where ego-deictic phrasing is pervasive and cannot be resolved without jointly leveraging both historical visual evidence and prior linguistic exchanges. At the same time, we observe that humans, despite their strong overall performance, can still make mis- takes when subjected to challenging visual conditions such as motion blur, rapid camera movement, or the presence of very small or partially occluded text. These fine-grained visual cues are difficult to perceive even for human annotators and highlight the need to de- velop models robust to the intrinsic ambiguity present in real-world egocentric videos, where frames are not always clean, stable, or fully interpretable. Also, for some extremely abstract deictic expressions in compan- ion QA, humans may also make mistakes. In such cases, Companion-style QA Assistance in Ego-Vision15 Ts: 00:23 Q Q Ts: 02:18 ... Ts: 00:08 Ts: 00:30Ts: 01:48 Ts: 00:21Ts: 02:04 Ts: 02:15 Category: Scrutinization Category: Scrutinization Q: What do you think of this painting? GT set: It looks vibrant and expressive with its blend of bold colors. The soft transitions between blue, pink, and orange give it an uplifting, dreamy feeling. Chained QA: None GPT-5.5 (General): It seems a bit messy, mostly just smudges without clear structure. [Acc: No, Score: 2] Intern-VL (General): It feels uplifting—the bright mix of blue, pink, and orange makes it look lively and expressive.[Acc: Yes, Score: 5] Flash-VStream(Streaming): It looks calm and gentle, with soft colors blending together. [Acc: Yes, Score: 3] ReKV(Streaming): It appears creative, almost like clouds in an abstract sky. [Acc: No, Score: 2] MyBuddy: It’s an abstract painting—its smooth blend of blue, pink, and orange tones creates aatmospheric effect.[Acc: Yes, Score: 4] Human: It’s colorful and harmonious, giving off a soft, dreamy vibe. [Acc: Yes, Score: 5] Q: Is this similar tothat on the laptop? GT set: Yes, both the brush stroke and the laptop reference show a similar shade of bright blue. Chained QA: (Q: What is the color on the brush? Q: Blue.) Gemini-2.5-pro (General): The colors look different—the one on the laptop seems more purple than blue. [Acc: No, Score: 2] LLaMA-VID (General): No, the laptop screen appears mostly pink and yellow. [Acc: No, Score: 1] Dispider(Streaming): It seems like the laptop is showing a video, so they’re not comparable. [Acc: No, Score: 1] Cogstream(Streaming): Yes, the mixed color styles are the same, but the laptop’s image looks more mixed. [Acc: No, Score: 2] MyBuddy: Yes—the blue on your brush closely matches the bright blue shown on the laptop’s reference image.[Acc: Yes, Score: 5] Human: Yes, the painting style look very similar to that on the screen. [Acc: No, Score: 2] Q: What am I holding? GT set: A stack of lifestyle magazines. HGTV Magazine. Reading material for the living room. Chained QA:None GPT-5.5 (General): A stack of magazines, including an issue of HGTV Magazine. [Acc: Yes, Score: 5] Flash-VStream(Streaming): Magazines. [Acc:Yes, Score: 5] Intern-VL (General): A plate of cookies in the hands. [Acc: No, Score: 1] ReKV(streaming): Some colorful papers or booklets in the hands. [Acc: Yes, Score: 3] MyBuddy: A stack of magazines, with the top one clearly identifiable as HGTV Magazine.[Acc: Yes, Score: 5] Human: A pile of magazines, with the HGTV one on top. [Acc: Yes, Score: 5] Ts: 01:10 Q Ts: 02:15 Category: Recognition ... Ts: 00:02 Ts: 01:14Ts: 01:47 Ts: 01:09Ts: 02:10 Ts: 02:43 Q: What is the main wording on it? GT set: “BEST ACTOR” is printed on the left page. “FROM HTV” appears near the top of the right page. The headline features a couple posing for a photoshoot. Chained QA: (Q: What am I holding? A: A stack of magazines, with the top one clearly identifiable as HGTV Magazine.) Gemini-3.1-pro (General): It appears to mention snacks or drinks, possibly reviewing them. [Acc: No, Score: 1] LLaMA-VID (General): The text seems to be related to home design tips. [Acc: No, Score: 1] Dispider(Streaming): It looks like a feature about celebritiesbut the text is unclear. [Acc: No, Score: 2] Cogstream(Streaming): The magazine shows “BEST ACTOR” and “FROM HTV.” [Acc: Yes, Score: 5] MyBuddy: The visible text includes “BEST ACTOR” on the left and “FROM HTV” on the right. [Acc: Yes, Score: 5] Human: It appears to mention actors or fashion models, possibly reviewing them. [Acc: No, Score: 2] Category: Recognition Fig. 7 More result visualizations on BuddyVQA in Scrutinization and Recognition categories. Baseline models perform poorly on deictic questions, underscoring the importance of jointly leveraging historical interaction contexts and embodied visual cues for ego-deictic reference resolution. the referent may be very ambiguous or only implicitly grounded in earlier contexts, making the correct inter- pretation genuinely difficult. These situations highlight that resolving deictic cues is not only a challenge for models but also a non-trivial cognitive task for humans. 5.5 Evaluation Agreement between Human and LLM Before actual model evaluation, we follow recent prac- tices (Cheng et al., 2024; Xiao et al., 2025) to tune the evaluation prompt to maximally align the LLM evalu- ator’s judgment with that of humans. Specifically, we apply the offline general VideoQA and online streaming VQA methods (InternVL-3.5 (Wang et al., 2025b) and ReKV (Di et al., 2025), respectively) on 500 randomly selected samples from BuddyVQA. We then invite four human volunteers to assess the model predictions by comparing them with the annotated ground-truth an- swers while watching the corresponding videos. Human evaluations are then aggregated following a majority voting mechanism. Afterwards, we tune the evaluation prompt for the LLM evaluator GPT-5 (OpenAI, 2025a), and tune the prompt so that the LLM’s judgments align as closely as possible with those of humans. Accord- ing to the results in Tab. 10, GPT-5 with the final adjusted prompt and human evaluators achieve simi- lar accuracy and score, with Pearson correlation coef- ficients of 0.87 and 0.91, respectively, indicating strong consistency. The Cohen’s kappa coefficient among the four volunteers is 0.81 for accuracy, indicating a high level of human agreement. Fig. 9 further shows that the LLM evaluator exhibits a high level of consistency with 16Hangyu Qin et al. Ts: 00:11 Q Q Ts: 02:58 Category: Recall Category: Recall ... Ts: 00:01 Ts: 00:18 Ts: 02:41 Ts: 00:05 Ts: 02:45 Ts: 02:55 Q: Where are the other bats I left over there earlier? GT set: The other bats are still lying on the dirt in front of the fence, near the glove and baseballs. They remain in the same spot where you first placed them. Chained QA: (Q: Where did I put things down when we arrived? A: You placed your bats, glove, and baseballs on the dirt near the fence.) Gemini-3.1-pro (General): I think you moved all the bats already—they’re with you now. [Acc: No, Score: 1] LLaMA-VID (General): The bats should be behind you near home plate. [Acc: No, Score: 1] Dispider(Streaming): They’re still on the dirt near the fence. [Acc: Yes, Score: 4] Cogstream(Streaming): The other bats are lying on the ground by the fence. [Acc: Yes, Score: 5] MyBuddy: The other bats are still lying on the dirt near the fence, in the same spot where you originally placed them. [Acc: Yes, Score: 5] Human: They’re on the dirt by the fence where you left them. [Acc: Yes, Score: 4] Q: What did I place on the ground before picking it up? GT set: You placed several baseball bats on the ground. You also set down a glove and a few baseballs near them. Chained QA: None GPT-5.5 (General): You put a couple of bats on the ground earlier. [Acc: Yes, Score: 4] Intern-VL (General): You only put a single baseball on the dirt. [Acc: No, Score: 1] Flash-VStream(Streaming): You placed your gloves and Helmets on the ground before grabbing the bat. [Acc: No, Score: 2] ReKV(Streaming): I think you laid out some baseball equipment, maybe a bat. [Acc: No, Score: 3] MyBuddy: Earlier, you set several bats, a glove, and a few baseballs on the ground before picking up the bat. [Acc: Yes, Score: 5] Human: A few bats, a glove, and some baseballs. [Acc: Yes, Score: 5] Ts: 01:07 Ts: 01:28 Ts: 02:11 Q Category: Advisory Ts: 01:15 Ts: 01:24 Category: Advisory z Ts: 02:31 Ts: 02:38 ... Ts: 02:40 Q Q: What should I do to safely continue? GT set: Be mindful of your footing on the uneven rocks to avoid slipping. Avoid looking at your phone while spraying to maintain control. Chained QA: (Q: Where am I watering? A: Near some rocks, on the grass.) Gemini-3.1-pro (General): Keep the hose steady and direct the water where needed. [Acc: No, Score: 2] LLaMA-VID (General): Pick up the rocks and move them to a new spot before watering. [Acc: No, Score: 1] Dispider(Streaming): Dry the floor first so you don’t slip indoors. [Acc: No, Score: 1] Cogstream(Streaming): Hold the hose firmly and watch your step while working around the rocks. [Acc: Yes, Score: 3] MyBuddy: Maintain steady footing on the rocky edge, andavoid checking your phone during the task to stay safe. [Acc: Yes, Score: 5] Human: Watch your step on the rocks and keep your attention on the hose instead of the phone.[Acc: Yes, Score: 5] Q: How should I prepare these tools and equipment for work? GT set: Put on the rubber boots for protection. Place the loose tools like the level and saw into the bag. Clear the ground to avoid tripping hazards. Chained QA: None GPT-5.5 (General): Put on the protective boots and organize the tools into the bag. [Acc: Yes, Score: 5] Intern-VL (General): Use the red level to dig the ground. [Acc: No, Score: 1] Flash-VStream(Streaming): Move the indoor furniture to the side. [Acc: No, Score: 2] ReKV(streaming): You should wear the boots and pick up the tools. [Acc: Yes, Score: 4] MyBuddy: Wear the rubber boots lying on the ground and gather the scattered tools into the bag.[Acc: Yes, Score: 5] Human: Putthe boots on and clean up the scattered tools so the crew doesn't trip.[Acc:Yes, Score: 4] Fig. 8 More result visualizations on BuddyVQA in Recall and Advisory categories. MyBuddy demonstrates substantially stronger performance on long video sequences compared to general MLLMs. Its consistent improvements on chained questions further highlight the importance of history-aware reasoning across question chains in BuddyVQA. Table 10 Comparison of human and LLM evaluators in terms of accuracy and score. The results show highly aligned scores between human and LLM judge. Evaluator InternVL-3.5ReKV Acc.(%)Sco.Acc.(%)Sco. Human62.33.155.32.9 GPT-561.83.153.72.8 human judgments, as reflected by the dominant mass along the agreement diagonal. Both positive and nega- tive decisions display strong alignment, indicating that the model reliably mirrors human preferences when as- sessing semantic correctness. MyBuddy ReKV Fig. 9 Confusion matrices of evaluation results. 6 Conclusion In this paper, we introduced a novel benchmark, Bud- dyVQA, towards companion-style video question an- swering in first-person vision. BuddyVQA stands out from traditional offline VideoQA and streaming VQA Companion-style QA Assistance in Ego-Vision17 by capturing two key insights in companion-style as- sistance: Interactive Chaining and Ego-Deictics. More- over, the QAs are timestamp-specific and are well grouped into four assistance types for better analy- sis. We also proposed a novel companion-style QA framework MyBuddy, which features multi-level mem- ory and cross-modal chain-of-thought reasoning for effi- ciently memorizing video representations in a streaming setting and interpreting the user’s actual intent from multi-modal historical contexts, respectively. MyBuddy achieves superior performance over strong competitors on both BuddyVQA and existing long streaming VQA benchmarks, suggesting its high effectiveness. The ab- lations further demonstrate the contribution of our in- novative designs. We hope that our dataset, model, and analysis could advance MLLMs towards companion- style QA in first-person vision. 7 Limitations and Future Work Asynchronous Memory Consolidation. The Low-Active Memory summarization is designed to run during idle intervals, allowing the system to compress historical vi- sual information without interfering with real-time QA responses. This design implicitly assumes that the user is not continuously issuing queries. Yet in highly inter- active scenarios where questions arrive in rapid succes- sion, opportunities for such background consolidation may become limited, potentially introducing latency if summarization must be executed alongside the QA pipeline. Our future work will explore more efficient or incremental memory consolidation strategies that oper- ate continuously with minimal computational overhead. Multi-user Assumption in Egocentric Interaction. Currently, BuddyVQA assumes a single primary user interacting with the assistant. In realistic environments, however, egocentric scenes may contain multiple people or agents, making referential interpretation more com- plex. While our current ego-deictic grounding relies pri- marily on cues such as gaze direction and hand inter- actions, richer multi-agent reasoning may be required to disambiguate references in more complex social con- texts. Our future research would extend companion QA systems to multi-user scenarios by incorporating person tracking, speaker-aware dialogue modeling, and social interaction understanding, enabling assistants to rea- son jointly over visual observations and conversational participants. Competing Interests All authors certify that they have no affiliations with or involvement in any organiza- tion or entity with any financial interest or non-financial interest in the subject matter or materials discussed in this manuscript. Data Availability Statement The authors confirm that this manuscript has associated data in a data repository. The data supporting the findings of this work will be made publicly available. Specifically, this includes the BuddyVQA dataset and the MyBuddy method. Instructions for access will be provided to en- sure reproducibility. References Alamri H, Cartillier V, Das A, Wang J, Cherian A, Essa I, Batra D, Marks TK, Hori C, Anderson P, et al. (2019) Audio visual scene-aware dialog. In: Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p 7558–7567 An X, Xie Y, Tang F, Yan Y, Tan H, Zhu D, Chen C, Zhao X, Qin B, Yang K, et al. (2026) Llava-onevision- 2: Towards next-generation perceptual intelligence. arXiv preprint arXiv:260525979 Azad S, Vineet V, Rawat YS (2026) Streamready: Learning what to answer and when in long stream- ing videos. In: Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, p 40494–40504 Bai S, Cai Y, Chen R, Chen K, Chen X, Cheng Z, Deng L, Ding W, Gao C, Ge C, et al. (2025a) Qwen3-vl technical report. arXiv preprint arXiv:251121631 Bai S, Chen K, Liu X, Wang J, Ge W, Song S, Dang K, Wang P, Wang S, Tang J, et al. (2025b) Qwen2. 5-vl technical report. arXiv preprint arXiv:250213923 B ̈armann L, Waibel A (2022) Where did i leave my keys?-episodic-memory-based question answering on egocentric videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, p 1560–1568 Carston R (1999) Herbert h. clark, using language. cambridge: Cambridge university press, 1996. p. xi+432. Journal of Linguistics 35:167–222, DOI 10.1017/S0022226798217361 Chatterjee D, Remelli E, Song Y, Tekin B, Mittal A, Bhatnagar B, Camgoz NC, Hampali S, Sauser E, Ma S, et al. (2025) Streaming videollms for real-time pro- cedural video understanding. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, p 22586–22598 Chen B, Yue Z, Chen S, Wang Z, Liu Y, Li P, Wang Y (2025) Lvagent: Long video understanding by multi- round dynamical collaboration of mllm agents. In: Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, p 20237–20246 18Hangyu Qin et al. Chen S, Xu Q, Ma Y, Qiao Y, Wang Y (2023) Attentive snippet prompting for video retrieval. IEEE Transac- tions on Multimedia 26:4348–4359 Chen Z, Wang W, Tian H, Ye S, Gao Z, Cui E, Tong W, Hu K, Luo J, Ma Z, et al. (2024a) How far are we to gpt-4v? closing the gap to commercial multi- modal models with open-source suites. Science China Information Sciences 67(12):220101 Chen Z, Wu J, Wang W, Su W, Chen G, Xing S, Zhong M, Zhang Q, Zhu X, Lu L, et al. (2024b) Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In: Proceedings of the IEEE/CVF conference on computer vision and pat- tern recognition, p 24185–24198 Cheng S, Guo Z, Wu J, Fang K, Li P, Liu H, Liu Y (2024) Egothink: Evaluating first-person perspec- tive thinking capability of vision-language models. In: Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, p 14291– 14302 Choi Y, Miles R, Potamias RA, Elezi I, Deng J, Zafeiriou S (2026) Do you see what i am pointing at? gesture-based egocentric video question answer- ing. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p 18534–18544 Christmann P, Saha Roy R, Abujabal A, Singh J, Weikum G (2019) Look before you hop: Conversa- tional question answering over knowledge graphs us- ing judicious context expansion. In: Proceedings of the 28th ACM international conference on informa- tion and knowledge management, p 729–738 Comanici G, Bieber E, Schaekermann M, Pasupat I, Sachdeva N, Dhillon I, Blistein M, Ram O, Zhang D, Rosen E, et al. (2025) Gemini 2.5: Pushing the fron- tier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:250706261 Dai W, Li J, Li D, Tiong AMH, Zhao J, Wang W, Li B, Fung P, Hoi S (2023) Instructblip: towards general- purpose vision-language models with instruction tun- ing. In: Proceedings of the 37th International Confer- ence on Neural Information Processing Systems, p 49250–49267 DeepMind G (2026) Gemini 3.1 pro: Best for complex tasks and bringing creative concepts to life. URL https://deepmind.google/models/gemini/pro/ Devlin J, Chang MW, Lee K, Toutanova K (2019) Bert: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 conference of the North American chapter of the as- sociation for computational linguistics: human lan- guage technologies, volume 1 (long and short papers), p 4171–4186 Di S, Yu Z, Zhang G, Li H, Zhong T, Cheng H, Li B, He W, Shu F, Jiang H (2025) Streaming video question- answering with in-context video kv-cache retrieval. ICLR Ester M, Kriegel HP, Sander J, Xu X, et al. (1996) A density-based algorithm for discovering clusters in large spatial databases with noise. In: kdd, vol 96, p 226–231 Ester M, Kriegel HP, Sander J, Wimmer M, Xu X (1998) Incremental clustering for mining in a data warehousing environment. In: Proceedings of the 24th International Conference on Very Large Data Bases (VLDB), p 323–333 Fang Y, Wang W, Xie B, Sun Q, Wu L, Wang X, Huang T, Wang X, Cao Y (2023) Eva: Exploring the limits of masked visual representation learning at scale. In: Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition, p 19358–19369 Fu C, Dai Y, Luo Y, Li L, Ren S, Zhang R, Wang Z, Zhou C, Shen Y, Zhang M, et al. (2024) Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:240521075 Grauman K, Westbury A, Byrne E, Chavis Z, Furnari A, Girdhar R, Hamburger J, Jiang H, Liu M, Liu X, et al. (2022) Ego4d: Around the world in 3,000 hours of egocentric video. In: Proceedings of the IEEE/CVF conference on computer vision and pat- tern recognition, p 18995–19012 He M, Liu Y, Wu B, Yuan J, Wang Y, Huang T, Zhao B (2024) Efficient multimodal learning from data- centric perspective. arXiv preprint arXiv:240211530 Hu EJ, Shen Y, Wallis P, Allen-Zhu Z, Li Y, Wang S, Wang L, Chen W, et al. (2022) Lora: Low-rank adaptation of large language models. ICLR 1(2):3 Hu Y, Yang Z, Wang S, Qian S, Wen B, Yang F, Gao T, Xu C (2025) Streamingcot: A dataset for tempo- ral dynamics and multimodal chain-of-thought rea- soning in streaming videoqa. In: Proceedings of the 33rd ACM International Conference on Multimedia, p 13464–13470 Huang Q, Xiong Y, Rao A, Wang J, Lin D (2020) Movienet: A holistic dataset for movie understand- ing. In: European conference on computer vision, Springer, p 709–727 Huang Z, Li X, Li J, Wang J, Zeng X, Liang C, Wu T, Chen X, Li L, Wang L (2025) Online video un- derstanding: Ovbench and videochat-online. In: Pro- ceedings of the Computer Vision and Pattern Recog- nition Conference, p 3328–3338 Inadumi S, Kawano S, Yuguchi A, Kawanishi Y, Yoshino K (2024) A gaze-grounded visual question Companion-style QA Assistance in Ego-Vision19 answering dataset for clarifying ambiguous japanese questions. arXiv preprint arXiv:240317545 Kim M, Shim K, Choi J, Chang S (2025) Infinipot- v: Memory-constrained kv cache compression for streaming video understanding. NeurIPS Kingma DP, Ba J (2014) Adam: A method for stochas- tic optimization. arXiv preprint arXiv:14126980 Lee YJ, Lee BK, Zhang J, Hwang Y, Ko B, Kim HG, Yao D, Rong X, Joo E, Han SH, et al. (2025) Multi- verse: A multi-turn conversation benchmark for eval- uating large vision and language models. In: Proceed- ings of the IEEE/CVF International Conference on Computer Vision, p 708–719 Li B, Zhang Y, Guo D, Zhang R, Li F, Zhang H, Zhang K, Zhang P, Li Y, Liu Z, et al. (2024a) Llava- onevision: Easy visual task transfer. arXiv preprint arXiv:240803326 Li K, He Y, Wang Y, Li Y, Wang W, Luo P, Wang Y, Wang L, Qiao Y (2023a) Videochat: Chat-centric video understanding. arXiv preprint arXiv:230506355 Li K, Wang Y, Li Y, Wang Y, He Y, Wang L, Qiao Y (2023b) Unmasked teacher: Towards training- efficient video foundation models. In: Proceedings of the IEEE/CVF international conference on computer vision, p 19948–19960 Li K, Wang Y, He Y, Li Y, Wang Y, Liu Y, Wang Z, Xu J, Chen G, Luo P, et al. (2024b) Mvbench: A comprehensive multi-modal video understanding benchmark. In: Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recogni- tion, p 22195–22206 Li Y, Wang C, Jia J (2024c) Llama-vid: An image is worth 2 tokens in large language models Lin J, Fang Z, Chen C, Wan Z, Luo F, Li P, Liu Y, Sun M (2024) Streamingbench: Assessing the gap for mllms to achieve streaming video understanding. arXiv preprint arXiv:241103628 Liu H, Li C, Li Y, Lee YJ (2024) Improved baselines with visual instruction tuning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, p 26296–26306 Liu J, Zheng S, Karlsson BF, Lu Z (2025) Taking notes brings focus? towards multi-turn multimodal dialogue learning. In: Proceedings of the 2025 Con- ference on Empirical Methods in Natural Language Processing, p 33291–33312 Liu Y, Ott M, Goyal N, Du J, Joshi M, Chen D, Levy O, Lewis M, Zettlemoyer L, Stoyanov V (2019) Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv:190711692 Long L, He Y, Ye W, Pan Y, Lin Y, Li H, Zhao J, Li W (2026) Seeing, listening, remembering, and reasoning: A multimodal agent with long-term memory. ICLR Luo Y, Zheng X, Li G, Yin S, Lin H, Fu C, Huang J, Ji J, Chao F, Luo J, et al. (2025) Video-rag: Visually- aligned retrieval-augmented long video comprehen- sion. Advances in Neural Information Processing Sys- tems 38:168008–168033 Maaz M, Rasheed H, Khan S, Khan F (2024) Video- ChatGPT: Towards detailed video understanding via large vision and language models. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, Association for Compu- tational Linguistics, Bangkok, Thailand, p 12585– 12602, DOI 10.18653/v1/2024.acl-long.679 Mangalam K, Akshulakov R, Malik J (2024) Egoschema: A diagnostic benchmark for very long-form video language understanding. Advances in Neural Information Processing Systems 36 Niu J, Li Y, Miao Z, Ge C, Zhou Y, He Q, Dong X, Duan H, Ding S, Qian R, et al. (2025) Ovo-bench: How far is your video-llms from real-world online video understanding? In: Proceedings of the Com- puter Vision and Pattern Recognition Conference, p 18902–18913 OpenAI (2025a) Introducing gpt-5. URL https:// openai.com/index/introducing-gpt-5 OpenAI (2025b) Introducing gpt-oss. URL https:// openai.com/index/introducing-gpt-oss/ OpenAI (2026) Introducing gpt-5.5. URL https:// openai.com/index/introducing-gpt-5-5/ Peng T, Hua J, Liu M, Lu F (2025) In the eye of mllm: Benchmarking egocentric video intent understanding with gaze-guided prompting. Advances in Neural In- formation Processing Systems 38 Pham HA, Le TM, Le V, Phuong TM, Tran T (2022) Video dialog as conversation about objects living in space-time. In: European Conference on Computer Vision, Springer, p 710–726 Qian J, Zou B, Dong M, Li X, Aw A, Hong Y (2022) Capturing conversational interaction for question an- swering via global history reasoning. In: Findings of the Association for Computational Linguistics: NAACL 2022, p 2071–2078 Qian R, Ding S, Dong X, Zhang P, Zang Y, Cao Y, Lin D, Wang J (2025) Dispider: Enabling video llms with active real-time interaction via disentangled percep- tion, decision, and reaction. In: Proceedings of the Computer Vision and Pattern Recognition Confer- ence, p 24045–24055 Qin H, Xiao J, Yao A (2025) Question-answering dense video events. In: Proceedings of the 48th Interna- tional ACM SIGIR Conference on Research and De- velopment in Information Retrieval, p 884–894 20Hangyu Qin et al. Qu C, Yang L, Chen C, Qiu M, Croft WB, Iyyer M (2020) Open-retrieval conversational question an- swering. In: Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, p 539–548 QwenTeam (2026) Qwen3.5: Towards native mul- timodal agents. URL https://qwen.ai/blog?id= qwen3.5 Radford A, Kim JW, Hallacy C, Ramesh A, Goh G, Agarwal S, Sastry G, Askell A, Mishkin P, Clark J, et al. (2021) Learning transferable visual models from natural language supervision. In: International con- ference on machine learning, PMLR, p 8748–8763 Shu Y, Liu Z, Zhang P, Qin M, Zhou J, Liang Z, Huang T, Zhao B (2025) Video-xl: Extra-long vision language model for hour-scale video understanding. In: Proceedings of the Computer Vision and Pattern Recognition Conference, p 26160–26169 Song E, Chai W, Ye T, Hwang JN, Li X, Wang G (2025) Moviechat+: Question-aware sparse memory for long video question answering. IEEE Transactions on Pat- tern Analysis and Machine Intelligence Sun P, Xiao J, Tse THE, Li Y, Akula A, Yao A (2025) Visual intention grounding for egocentric assistants. In: Proceedings of the IEEE/CVF International Con- ference on Computer Vision, p 2512–2522 Sun Y, Zhang H, Ding H, Zhang T, Ma X, Jiang YG (2026) Sama: Towards multi-turn referential grounded video chat with large language models. Advances in Neural Information Processing Systems 38:47065–47091 Tanenhaus MK, Spivey-Knowlton MJ, Eberhard KM, Sedivy JC (1995) Integration of visual and linguis- tic information in spoken language comprehension. Science 268(5217):1632–1634 Tomasello M, Carpenter M (2007) Shared intentional- ity. Developmental science 10(1):121–125 VicunaTeam (2022) Vicuna: An open-source chatbot impressing gpt-4 with 90% chatgpt quality. URL https://vicuna.lmsys.org/ Wang P, Bai S, Tan S, Wang S, Fan Z, Bai J, Chen K, Liu X, Wang J, Ge W, et al. (2024) Qwen2-vl: Enhancing vision-language model’s per- ception of the world at any resolution. arXiv preprint arXiv:240912191 Wang R, Li B, Dai X, Yang J, Chen YL, Xing Z, Yang Y, Chen D, Qiu X, Wu Z, et al. (2025a) Prolongvid: A simple but strong baseline for long-context video instruction tuning. In: Proceedings of the 2025 Con- ference on Empirical Methods in Natural Language Processing, p 30824–30837 Wang W, Gao Z, Gu L, Pu H, Cui L, Wei X, Liu Z, Jing L, Ye S, Shao J, et al. (2025b) Internvl3. 5: Advancing open-source multimodal models in versatility, reason- ing, and efficiency. arXiv preprint arXiv:250818265 Wang Y, Li Z, Qian T, Zheng H, Wang Z, Fu Y, Wang X (2026) Streameqa: Towards streaming video under- standing for embodied scenarios. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p 9422–9432 Wong B, Chen J, Wu Y, Lei SW, Mao D, Gao D, Shou MZ (2022) Assistq: Affordance-centric question- driven task completion for egocentric assistant. In: European Conference on Computer Vision, Springer, p 485–501 Xiao G, Tian Y, Chen B, Han S, Lewis M (2023) Efficient streaming language models with attention sinks. arXiv preprint arXiv:230917453 Xiao J, Shang X, Yao A, Chua TS (2021) Next-qa: Next phase of question-answering to explaining temporal actions. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p 9777– 9786 Xiao J, Huang N, Qiu H, Tao Z, Yang X, Hong R, Wang M, Yao A (2025) Egoblind: Towards egocentric visual assistance for the blind. Advances in Neural Information Processing Systems 38 Xiao J, Chen J, Sun T, Yang X, Yao A (2026a) Mukv: Multi-grained kv cache compression for long stream- ing video question-answering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, p 11381–11391 Xiao J, Zhang S, Zhu P, Yao A (2026b) Ego-grounding for personalized question-answering in egocentric videos. In: Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, p 40537–40547 Xie Y, He B, Wang J, Zheng X, Ye Z, Wu Z (2026) Fluxmem: Adaptive hierarchical memory for stream- ing video understanding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, p 31272–31282 Xu B, Wang Z, Du Y, Song Z, Zheng S, Jin Q (2025a) Do egocentric video-language models truly under- stand hand-object interactions? ICLR Xu R, Xiao G, Chen Y, He L, Peng K, Lu Y, Han S (2025b) Streamingvlm: Real-time under- standing for infinite video streams. arXiv preprint arXiv:251009608 Yan J, Ren R, Liu J, Xu S, Wang L, Wang Y, Wang Y, Zhang L, Chen X, Sun C, et al. (2025) Teleego: Benchmarking egocentric ai assistants in the wild. arXiv preprint arXiv:251023981 Yang J, Liu S, Guo H, Dong Y, Zhang X, Zhang S, Wang P, Zhou Z, Xie B, Wang Z, et al. (2025a) Ego- life: Towards egocentric life assistant. In: Proceed- Companion-style QA Assistance in Ego-Vision21 ings of the Computer Vision and Pattern Recognition Conference, p 28885–28900 Yang J, Yang S, Gupta AW, Han R, Fei-Fei L, Xie S (2025b) Thinking in space: How multimodal large language models see, remember, and recall spaces. In: Proceedings of the Computer Vision and Pattern Recognition Conference, p 10632–10643 Yang Z, Zhang K, Liu Q, Liu T, Ying L, Xue D, Hou Q, Qian S, Xu C (2026) Towards online interactors: A comprehensive survey on streaming video under- standing. arxiv Ye H, Zhang H, Daxberger E, Chen L, Lin Z, Li Y, Zhang B, You H, Xu D, Gan Z, et al. (2025) Mmego: Towards building egocentric multimodal llms for video qa. In: The Thirteenth International Confer- ence on Learning Representations Yeo W, Kim K, Yoon J, Hwang SJ (2026) Worldmm: Dynamic multimodal memory agent for long video reasoning. In: Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR), p 25599–25609 You S, Yuan B, Bao BK (2025) Scvbench: A benchmark with multi-turn dialogues for story-centric video un- derstanding. In: Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelli- gence, p 2287–2295 Yu Z, Xu D, Yu J, Yu T, Zhao Z, Zhuang Y, Tao D (2019) Activitynet-qa: A dataset for understanding complex web videos via question answering. In: Pro- ceedings of the AAAI Conference on Artificial Intel- ligence, vol 33, p 9127–9134 Zhai X, Mustafa B, Kolesnikov A, Beyer L (2023) Sig- moid loss for language image pre-training. In: Pro- ceedings of the IEEE/CVF international conference on computer vision, p 11975–11986 Zhang B, Li K, Cheng Z, Hu Z, Yuan Y, Chen G, Leng S, Jiang Y, Zhang H, Li X, et al. (2025a) Videollama 3: Frontier multimodal foundation mod- els for image and video understanding. arXiv preprint arXiv:250113106 Zhang D, Zhao Y, Huang J, Chen Y (2025b) Sharper and faster mean better: Towards more efficient vision- language model for hour-scale long video understand- ing. In: Che W, Nabende J, Shutova E, Pilehvar MT (eds) Proceedings of the 63rd Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Compu- tational Linguistics, Vienna, Austria, p 4423–4439, DOI 10.18653/v1/2025.acl-long.222, URL https:// aclanthology.org/2025.acl-long.222/ Zhang H, Wang Y, Tang Y, Liu Y, Feng J, Dai J, Jin X (2025c) Flash-vstream: Memory-based real-time un- derstanding for long video streams. ICCV Zhang P, Zhang K, Li B, Zeng G, Yang J, Zhang Y, Wang Z, Tan H, Li C, Liu Z (2024a) Long con- text transfer from language to vision. arXiv preprint arXiv:240616852 Zhang Y, Wu J, Li W, Li B, Ma Z, Liu Z, Li C (2024b) Video instruction tuning with synthetic data. arXiv preprint arXiv:241002713 Zhao Z, Wang K, Li S, Qian R, Lin W, Liu H (2025) Cogstream: Context-guided streaming video question answering. arXiv preprint arXiv:250610516 Zhou S, Xiao J, Li Q, Li Y, Yang X, Guo D, Wang M, Chua TS, Yao A (2025) Egotextvqa: Towards ego- centric scene-text aware video question answering. In: Proceedings of the Computer Vision and Pattern Recognition Conference, p 3363–3373 22Hangyu Qin et al. Appendix A BuddyVQA Dataset A.1 Video Activities Fig. 10 Activities in BuddyVQA. To construct our dataset, we selected 1,012 videos from Egoschema (Mangalam et al., 2024), sourced from Ego4D (Grauman et al., 2022), which contains more than three thousand hours of worldwide daily-life recordings across diverse scenarios such as household, outdoor, workplace, leisure, and construction. These selected videos span a broad spectrum of indoor and outdoor daily activities (as shown in Fig. 10) that are common in assistive scenarios. A.2 Automatic Question Candidate Generation Tab. 16 shows the prompts used for automatic question gen- eration. After generation, we apply an additional filtering stage to remove irrelevant, redundant, or trivially answer- able questions. Specifically, we prompt Gemini-2.5-pro (Co- manici et al., 2025) to (1) evaluate the semantic relevance of each question to its corresponding video segment, (2) detect near-duplicate phrasings within the same highlight moment, and (3) reject questions whose answers can be easily inferred without visual information. A.3 Details of Human Annotation We recruit 25 volunteers from diverse countries and social backgrounds (students, engineers, doctors, freelancers, ac- countants, insurance agents, etc.) via the Prolific platform to annotate QA pairs based on the automatically generated highlight moments and draft questions. We clarify several key points of the human ques- tion–answer annotation process: 1. We provide annotators with detailed definitions of all question categories to minimize ambiguity. For each cat- egory, we also supply several examples (as shown in Tab. 11) to guide question formulation. 2. In the streaming setting, annotators are strictly restricted to viewing only the video content before the ending times- tamp of the highlight moment. Likewise, answers must rely solely on information available up to that same times- tamp. 3. Even for subjective questions, each answer must be evidence-based and grounded exclusively in the visual content observed before the highlight’s end time. 4. To encourage natural and conversational phrasing, anno- tators are allowed to use ego-deictic expressions as well as abbreviations, slang, and other colloquial forms. 5. We define interactively chained questions as those closely related to the prior QA pair within the same highlight mo- ment. However, we do not require subsequent questions to share the same subject or phrasing. We also prevent chains from looping by disallowing any chained question from becoming the base question of another chain. 6. Annotators are prohibited from using AI tools during the annotation process. Additionally, they must ensure that at least 50% of their annotated QA pairs are fully written by themselves, independent of the draft. Overall, we establish clear guidelines to ensure that human-authored questions and answers are natural, tempo- rally grounded, and well-defined, enabling the construction of a realistic companion QA dataset in an egocentric setting. A.4 Interactively Chaining Analysis We identify three primary types of chained relationships: – Referential chains (41.2%), where follow-up questions use pronouns or vague expressions that depend on previous answers for grounding (e.g., “it,” “there,” “that”); – Temporal chains (23.5%), where reasoning relies on ear- lier events or actions (e.g., “What did I do before this?”); – Inferential chains (21.8%), where subsequent questions seek higher-level interpretation or evaluation based on prior answers (e.g., asking “Did I do it correctly?” after “What did I just do?”). Referential chains primarily evaluate a model’s ability to resolve ambiguity by grounding pronouns or underspecified expressions to the correct visual or contextual entities, re- quiring strong multimodal alignment and memory retrieval. Temporal chains demand coherent tracking of event sequences over time, including robust temporal indexing and the ca- pacity to infer continuity or causality across past observa- tions. Inferential chains require even deeper abstraction: in addition to recalling earlier answers, the model must reason about their implications, evaluate correctness, or draw higher- level conclusions. Collectively, these chain types progress from surface-level referent grounding to increasingly deep tempo- ral, causal, and evaluative reasoning, highlighting the cog- nitive and computational challenges in producing coherent responses for companion QA. Appendix B MyBuddy Investigations and Experiments B.1 Study of MyBuddy Design Chained and Unchained Questions. Before designing our companion QA framework, we conducted a series of empiri- cal studies to understand how question dependency influences both performance and computational efficiency in egocen- tric streaming settings. In particular, we investigated whether questions should always be treated as dependent on previous QA turns or as entirely independent. Companion-style QA Assistance in Ego-Vision23 Table 11 Question categories and examples of BuddyVQA. CategoryQ Num. & Ratio Examples Recall5631(26.1%) Who just passed by me? How much time passed after I started running? What was he doing several seconds ago? Did I lock the door before leaving? Recognition5069(23.5%) What’s on that? What’s that red thing next to the greens? Who did I just hand those to? Scrutinization5523(25.6%) Am I missing any spot while sweeping? Am I doing this fast enough? Did I take longer to complete this than before? Are these clothes of good quality? Advisory5351(24.8%) What should I do next? How long will it take to get through this line? How do I know when the area is smooth enough? Table 12 Comparison of disambiguation accuracy (D Acc.) and latency w/ and w/o hierarchical rephrasing. MethodD Acc.(%)Latency Direct Question Answering62.4423ms Hierarchical Rephrasing (Ours)86.7552ms Our findings show that treating all questions as depen- dent introduces substantial context contamination: accumu- lated historical QA injects irrelevant or distracting informa- tion that interferes with grounding the current visual scene. This practice not only reduces accuracy from 65.1% to 56.5%, but also increases inference time by 32.7%. In contrast, treat- ing all questions as independent loses essential contextual cues, leading to a sharp overall accuracy drop from 65.1% to 38.2% due to missing referential anchoring. We then explored whether combining partial context (specifically, working memory and episodic memory) could alleviate these issues. Results indicate that leveraging both of them and further expanding to a multi-level memory (ac- tive, semi-active, low-active) module yields a more favorable balance between efficiency and contextual completeness. Fi- nally, we evaluated different question-routing strategies and found that a lightweight zero-shot LLM prompting method outperformed a trained binary classifier (86.7% vs. 73.5%) with only a modest 12.5% increase in inference time. These observations underscore the need for an adaptive mechanism that dynamically determines whether a question is depen- dent or independent and selectively retrieves the appropriate context, motivating the design of our hierarchical reasoning model. Hierarchical Rephrasing. The hierarchical rephrasing module applies a chain-of-thought idea to explicitly resolve deictic pronouns and vague linguistic expressions before an- swer generation, thereby improving referent grounding. As shown in Tab. 12, this approach improves disambiguation ac- curacy (measured as the proportion of rephrased questions whose referents are correctly grounded in the scene) while in- troducing limited latency. Empirical results indicate a sub- stantial gain from 62.4% to 86.7% over direct prompting, demonstrating that hierarchical rephrasing effectively miti- gates linguistic ambiguity with minimal additional inference latency. Specifically, by incorporating both gaze direction and recent hand–object interactions as embodied features, and utilizing the filtered visual information from multi-level mem- ory, the model identifies and replaces deictic terms with their most likely visual referents by doing hierarchical rephrasing. We further analyze different types of deictic situations, in- cluding unclear referents, vague temporal/spatial expressions, and ambiguous actions (see Tab. 13). Questions containing ego-deictics exhibit the largest relative improvement, high- lighting the module’s strength in resolving underspecified ref- erents. Meanwhile, vague temporal/spatial and action-related questions also benefit from clearer contextual grounding that supports more reliable temporal and procedural reasoning. In general, the hierarchical reformulation process reduces mis- grounded references by roughly 24%, underscoring its ability to convert linguistically ambiguous questions into explicit, interpretable queries that improve both referent clarity and downstream answer correctness. Embodied Ego-cues. To further evaluate the effective- ness of detecting embodied visual features for companion QA, we conduct both automatic and human assessments to deter- mine whether the detected coordinates of eye-gaze points and hand-pose bounding boxes provide meaningful support for question answering. We randomly sample 500 samples from BuddyVQA and recruit three human evaluators, using GPT- 5 (OpenAI, 2025a) as an automatic evaluator. As presented in Tab. 14, human assessments show that 76.2% of the de- tections are correctly anchored to the user’s eye gaze or hand poses, and of them 82.3% are judged to be closely related to the corresponding questions and beneficial for answer genera- tion. These results demonstrate that (1) embodied feature ex- traction meaningfully enhances a model’s ability to interpret visual context during reasoning, and (2) filtering detected sig- nals based on the incoming question further improves the rel- evance of the extracted features to the task. Controlled Ablation Study. Tab. 15 further discerns the performance gains over the GPT-5.5 (OpenAI, 2026) backbone by modules. The results demonstrate that each component of Buddy contributes progressively to the final performance. Starting from the GPT-5.5 backbone with raw frames and historical QA context (50.5%), adding visual in- formation retrieval and hierarchical rephrasing improves ac- curacy to 52.2%. Introducing the Question Filter further in- creases performance to 56.1%, indicating the importance of selecting relevant conversational context rather than using the entire QA history. Combining all components achieves the best performance of 65.1%, demonstrating that the gains arise from the complementary effects of memory, context se- lection, and ego-deictic reasoning rather than from the GPT-5 backbone alone. 24Hangyu Qin et al. Table 13 Examples of hierarchical input rephrasing of MyBuddy. Deictic TypeOriginal QuestionHierarchically RephrasingFinal Answer Unclear referents“I need more, where is it?” “What does it refer to?” → “Does it refer to the cup?” → “Where is the cup?” “On the table.” Vaguetemporal/ spatial cues “Has it started yet?”“Does it refer to the meeting?” → “Has the meeting started yet?” “Yes, 2 minutes ago.” Ambiguous actions“Am I doing this right?”“Does this refer to cutting the onion?” → “Am I cutting the onion correctly?” “Yes, but hold the knife away from your hand.” Table 14 Accuracy of eye gaze points and hand pose detec- tion, and the correlation of correctly detected features with questions in terms of correlated percentage (Corr.) and score. MethodAccuracy(%) Corr. with Ques. Corr.(%)Score Human76.282.33.8 GPT-5 (OpenAI, 2025a)81.389.14.1 Table 15 Controlled ablation study on Buddy. ConfigurationOverall Acc. Base GPT-5.5 Backbone (Raw F+His. QAs)50.5 Buddy w/o Memory + Q Filter + Visual Info + Rephrasing52.2 Buddy w/o Memory + Q Filter + Visual Info56.1 Buddy w/o Memory + Q Filter59.2 Buddy w/o Memory62.4 Full MyBuddy(GPT-5.5)65.1 B.2 Details of Baseline Models We provide a concise introduction to the MLLMs evaluated in experiments. – GPT-5.5 (OpenAI, 2026) not only outperforms pre- vious GPT models on benchmarks and answers questions more quickly, but is more useful for real-world queries. – Gemini-3.1-pro (DeepMind, 2026) surpasses previ- ous versions as Google’s most advanced “thinking” model, leveraging enhanced native reasoning to achieve state- of-the-art performance in complex coding, mathematical, and multimodal tasks. – Qwen3.5 (QwenTeam, 2026) is a recent native mul- timodal model family developed by the Qwen Team, targeting multimodal agent scenarios that require joint visual understanding, reasoning, coding, and interac- tion. Qwen3.5 enables efficient native multimodal training via a heterogeneous infrastructure that decouples paral- lelism strategies across vision and language components, avoiding uniform approaches’ inefficiencies. By exploit- ing sparse activations for cross-component computation overlap, it achieves near 100% training throughput ver- sus pure-text baselines on mixed text-image-video data. – LongVA (Zhang et al., 2024a) uses Qwen2-7B- Instruct (Wang et al., 2024) as the backbone language model and performs pretraining with a context length of 224K over a total of 900M tokens. LongVA improved the capability of LLMs to handle long videos thanks to the long context transfer phenomenon. – InternVL-3.5 (Wang et al., 2025b) follows the “ViT–MLP–LLM” paradigm adopted in previous ver- sions of InternVL. InternVL-3.5 initializes the language model using the Qwen3 series (Bai et al., 2025a) and GPT-OSS (OpenAI, 2025b), and the vision encoder us- ing InternViT-300M and InternViT-6B (Chen et al., 2024b). The Dynamic High Resolution strategy in In- ternVL1.5 (Chen et al., 2024a) is also retained. In- ternVL3.5 adopts a new reinforcement learning frame- work Cascade RL, together with two new modules Visual Resolution Router and Decoupled Vision-Language De- ployment, to reduce the inference cost. – VideoChat2 (Li et al., 2024b) applies UMT-L (Li et al., 2023b) as visual encoder and Vicuna-7B v0 (Vi- cunaTeam, 2022) as LLM, and deploy QFormer using the pretrained BERT base (Devlin et al., 2019). It performs a progressive multi-modal training strategy with 3 stages. – LLaMA-VID (Li et al., 2024c) instantiates the model with the pre-trained EVA-G (Fang et al., 2023) for vi- sual encoder and QFormer (Dai et al., 2023) for text de- coder. It builds the instruction pairs from (Liu et al., 2024) and (Maaz et al., 2024) for instruction tuning, and builds long QA pairs from movies and correspond- ing scripts in MovieNet (Huang et al., 2020) for training. LLaMA-VID designed a dual-token paradigm to empower existing LLMs to support long videos. – Video-LLaMA3 (Zhang et al., 2025a) applies Qwen2.5-7B (Bai et al., 2025b) as LLM initialization, and the vision encoder is initialized with the fine-tuned SigLIP (Zhai et al., 2023) weights. It uses a variety of scene images and document recognition images, along with a small portion of scene text images for training, and collects videos with generally annotated captions, ques- tions, and answers from multiple open-source datasets for video-centric fine-tuning. – Video-XL (Shu et al., 2025) is trained on Qwen-2- 7B (Wang et al., 2024). During pre-training, Laion-2M dataset (He et al., 2024) is used to optimize the projec- tor, where visual embeddings from CLIP-ViT-L (Radford et al., 2021) based vision encoder are aligned with the text embeddings of LLM. During fine-tuning, visual in- struction tuning is applied to optimize the parameters of the vision encoder, projector, and LLM. – LVAgent (Chen et al., 2025) uses a constructed agent library. The ASP-CLIP (Chen et al., 2023) model, opti- mized using Adam (Kingma and Ba, 2014), is used as the retrieval model for temporal context modeling. – Cogstream (Zhao et al., 2025) chooses VideoL- LaMA3 (Zhang et al., 2025a) as baseline, comprising VL3- SigLIP-NaViT (Zhai et al., 2023) as video encoder, an MLP projection layer, and Qwen2.5 (Bai et al., 2025b) as language model. Cogstream designs a reasoner to com- press the accumulated video stream and reason over the integrated visual-textual information. – Dispider (Qian et al., 2025) utilizes CLIP-L/14 (Rad- ford et al., 2021) as frame encoder, Qwen2-7B (Wang et al., 2024) as final LLM, and Qwen2-1.5B (Wang et al., 2024) to produce time-aware compressed clip-wise fea- tures. Dispider features a lightweight Proactive Stream- ing Video Processing module that tracks the video stream and identifies optimal moments for interaction. Companion-style QA Assistance in Ego-Vision25 – Flash-VStream (Zhang et al., 2025c) performs ini- tialization of visual encoder, projector, and LLM from pretrained Qwen2-VL-7b (Wang et al., 2024), and adapts a 9k subset from LLaVA-Video dataset (Zhang et al., 2024b) to do a LoRA (Hu et al., 2022) instruction tun- ing. Flash-VStream designs a Flash Memory module that retrieves detailed spatial information. – ReKV (Di et al., 2025) integrates LLaVA-OV-7B (Li et al., 2024a) for evaluation. ReKV is a training-free ap- proach that stores processed video in key-value caches (KV-Caches) and also introduces a retrieval method that retrieves only query-relevant KV-Caches. Appendix C Model Prompts Tab. 16 provides the prompts used by Gemini-2.5-pro for highlight moment detection, question generation and filter- ing. Tab. 17 lists the prompts employed by GPT-5 for model evaluation. Tab. 18 shows the prompts used in Low-Active Memory of MyBuddy for summarization. Tab. 19 details the specific prompts applied for the Question Filter in MyBuddy. Tab. 20 includes the prompts designed for the visual informa- tion retrieval step with multi-level memory inputs. Tab. 21 further illustrates the prompt for the final zero-shot question answering with hierarchical rephrasing. 26Hangyu Qin et al. Table 16 Prompts for automatic question candidate generation of BuddyVQA. Automatic Question Candidate Generation Prompts # Highlight detection: You are an advanced assistant specialized in video analysis. Your task is to analyze the provided video and identify its key highlights based on significant visual content, such as rapid action, noteworthy events, or shifts in visual dynamics. The input is a video file and the output should be a list of timestamps that correspond to the start and end of each highlight segment in the video. Please focus on identifying segments with high visual or contextual importance. Answer with ONLY timestamp pairs. Ensure the timestamps are accurate and clearly marked in seconds, e.g., [00:30, 00:45], [02:21, 02:51]. Video: video # Question generation: You are a good question generator, please help me generate appropriate questions that may be asked by users of an embodied visual companion, given an egocentric video and some highlight moments. Task instructions for each highlight moment: 1. Describe Actions: First summarize the key action/event in a phrase, then clearly detail what the main characters are doing in each highlight moment, focusing on their actions and interactions. Refer to the camera wearer as ’c’. 2. Generate Questions and Answers: Generate questions for each highlight moment according to the following require- ments: Requirement 1: Egocentric View: Ask the question with the assumption that you are the camera wearer and are new to this activity. Requirement 2: Challenge-Specific: Reflect various challenges faced by different identities in distinct scenes. Requirement 3: Diversity: Ensure questions are varied and explore different facets of the video content, and try to make some questions correlated with each other. Requirement 4: IMPORTANT: Use Pronouns: Employ at least one pronoun like I, you, we, he, she, them, it, etc. in the question, rather than specific names and objects. Here is an example: Highlight Moment 1: [00:00, 00:20] Action Description: Extending cup: C is extending a cup to another individual who is standing across the table. Questions and Answers: 1. Question: Is it full? Answer: Yes, the cup is full with coffee. 2. Question: Where can I put it after I’m done? Answer: You can put it on the table. 3. Question: I am tired, how can we enjoy it? Answer: You can sit near the table and enjoy the drink with another person. 4. Question: What’s he wearing? Answer: The man next to you is wearing a blue T-shirt. 5. Question: What is the next step for me once I finish this? Answer: You will need to go to the kitchen and wash the cup. Other example questions could be: What was I doing 10 seconds ago? Is this edible? Where is it? How long will it take to get through this? What does that road sign say? What should I do next? Who did I hand the chopsticks to? Why should I do this? Where is my key? How do I use the medication in this bottle? Video: video Highlight moments: highlight # Automatic question filtering: You are a helpful assistant. Given a set of questions and their corresponding video, evaluate whether these questions are semantically relevant to the visual content, detect if they are redundant or highly similar to any other, and determine whether they can be trivially answered without meaningful temporal grounding. Video: video Questions: question Companion-style QA Assistance in Ego-Vision27 Table 17 Prompts for GPT-5 (OpenAI, 2025a) to evaluate MLLMs on BuddyVQA. Evaluation Prompts for BuddyVQA You are an intelligent chatbot designed for evaluating the correctness of generative outputs for egocentric, streaming question-answer pairs in the BuddyVQA dataset. Your task is to compare the predicted answer with the correct answer set and determine whether they meaningfully match, taking into account the characteristics of egocentric video, ego- deictic expressions, and interactive question chains. The following are the requirements of evaluation: a. Focus on the meaningful match between the predicted answer and the correct answer sets. In BuddyVQA, many questions contain ego-deictic expressions such as “this”, “that”, “it”, “here”, or “there”. Ensure that the predicted answer resolves the same referent as the ground truth, even if expressed differently. b. The correct answers are provided as a set. Your evaluation should treat the prediction as correct if it meaningfully matches any of the answers in the set. The match does not need to be exact, but must refer to the same grounded entity, event, or semantic content. c. Consider the temporal continuity of chained questions. Some BuddyVQA questions depend on prior context (e.g.follow- up questions like “Is it still there?” or “Did I do it correctly?”). When evaluating, ensure that the predicted answer maintains the same contextual interpretation and refers to the correct entity or past event, consistent with the ground- truth chain. d. Synonyms, paraphrases, or alternative phrasings are valid as long as they preserve the same meaning and refer to the same grounded visual entity or event. Provide your evaluation result only as a yes/no and score where the score is an integer value between 0 and 5, with 5 indicating the highest meaningful match. Please generate the response in the form of a Python dictionary string with keys ’pred’ and ’score’, where the value of ’pred’ is a string of ’yes’ or ’no’ and the value of ’score’ is an integer. For example, your response should look like this: ’pred’: ’yes’, ’score’: 5, ’pred’: ’no’, ’score’: 1. Please evaluate the following video-based question-answer pair: Question: question Correct Answer set: GT answer Predicted Answer: predicted answer Table 18 Prompt for zero-shot summarization model S used to construct Low-Active Memory. Low-Active Memory Summarization Prompt You are an intelligent multimodal assistant tasked with summarizing egocentric video segments to extract long-term, stable, and reusable contextual information. Your goal is to produce a concise textual summary that captures information in long videos, including key activities, recurring patterns, background environment, salient objects, user routines, and consistent spatial layout, etc. # Instructions: - Focus on information that remains stable or frequently appears across this time window. Identify recurring actions, repeated interactions, or common tasks performed by the user. - Capture stable environmental structure such as room layout, tool locations, furniture arrangement, or habitual user workflows. - Extract objects or entities that reliably appear or are frequently manipulated across the window. Your summary should represent general knowledge useful for future reasoning, not for a specific question. Write the summary in clear, compact sentences that capture high-level patterns and omit unnecessary detail. Provide only the long-term summary text without explanation or extra commentary. Please summarize the following egocentric video: video features 28Hangyu Qin et al. Table 19 Prompt for the Question Filter R used to detect chained questions and retrieve relevant historical QA pairs. Question Filtering Prompt You are an intelligent multimodal assistant designed to identify whether an incoming question in a streaming egocentric QA scenario is chained (dependent on prior QA turns) or unchained (independent). Your task is to analyze the current question, compare it with a buffer of historical QA pairs and the current frame, and determine: (1) whether the question relies on earlier context, and (2) if so, which prior QA pair is the most relevant for resolving this dependency. # Instructions: - Classify the incoming question as either “chained” or “unchained”. A chained question typically contains ego-deictic expressions (e.g., “this”, “that”, “it”, “there”) or implicit references to past events, actions, or answers (e.g., “Is it still there?”, “Did I do it correctly?”, “What about now?”) that require information from historical QAs to answer. An unchained question is self-contained and understandable without referencing earlier context. - If the question is classified as unchained, return the label “unchained” and do not select any historical QA pair. - If the question is classified as chained, inspect the historical QA buffer and identify the single most relevant QA pair that resolves the ambiguous referent or contextual dependency. Relevance may arise from: semantic similarity, shared objects or actions, temporal continuity, or matching referential expressions. - Focus only on meaningful contextual grounding, not on superficial word overlap. Your output must identify the QA pair that best supports resolving the question’s dependency. Provide the final result strictly in the following Python dictionary format: ’type’: ’chained’ or ’unchained’, ’base qa’: qid of the most relevant QA pair ornull if unchained. Do not include any explanation, reasoning steps, or additional text. Please analyze the incoming question: Current Question: question Historical QAs: historical QAs Current frame: frame features Companion-style QA Assistance in Ego-Vision29 Table 20 Prompt for the Visual Information Retrieval (VIR) module. Visual Information Retrieval Prompt You are an intelligent multimodal assistant designed to retrieve visual information from egocentric video memories in order to support real-time question answering. Your task is to (1) generate concise visual captions, (2) filter long-term summaries for information relevant to the incoming question, and (3) detect embodied features such as eye-gaze points and hand bounding boxes that help ground deictic references. All three subtasks must be performed using the same lightweight MLLM capabilities. # Instructions for Captioning (from Active and Semi-Active Memory): - Generate a concise and accurate caption describing the visually salient content in the provided video frames. Focus on the objects, spatial relations, user interactions, and scene attributes that are relevant for answering a question. - The caption should reflect only the visible content and be grounded in the egocentric viewpoint. # Instructions for Memory Filtering (from Semi-Active and Low-Active Memory): - Given the incoming question, identify memory descriptions that are semantically or visually relevant. Relevance may arise from object continuity, spatial layout, repeated actions, or referents implied by ego-deictic expressions such as “this”, “that”, “it”, or “there”. - Return only the long-term or intermediate descriptions that meaningfully support answering the question. Do not repeat irrelevant or outdated information. # Instructions for Eye-Gaze and Hand-Pose Detection (from Active Memory): - Detect eye-gaze fixation points and hand bounding boxes visible in the frames. Return their coordinates in a simple and interpretable format. - The detected features should help disambiguate the referent of the given question when the user employs deictic expressions or when the focus of attention is ambiguous. - Only output detections that are visually grounded and clearly identifiable; do not hallucinate features. Provide the final result as a Python dictionary with the following keys: ’caption’: text, ’filtered memory’: text or list, ’coordinates’: list of gaze/hand coordinates. Do not include any explanation or reasoning in your output. Only provide the dictionary. Please process the following inputs: Incoming Question: question Active Memory: M A Semi-Active Memory: M S Low-Active Memory: M L 30Hangyu Qin et al. Table 21 Prompt for zero-shot question answering with hierarchical rephrasing. Zero-shot Question Answering Prompt You are an intelligent multimodal assistant designed to answer egocentric, streaming questions using retrieved visual information, historical QA context, and embodied features. Your task is to first rephrase the incoming question into an explicit, fully disambiguated form and then generate a correct answer grounded in the provided multimodal context. # Instructions for Hierarchical Input Rephrasing: Rewrite the question in a clearer and explicit form that removes ambiguity. Resolve ego-deictic expressions such as “this”, “that”, “it”, “here”, or “there” by grounding them using the provided context. Incorporate historical QA, cap- tions, and embodied features to determine the correct referent or event for rephrasing. The rephrased question must express the user’s intent explicitly and be fully interpretable without additional context. Provide only one rephrased question—concise, unambiguous, and grounded. Here is an example: Incoming Question: “Where is it now?” Context Provided: The historical QA buffer contains the pair (“Where did I put my phone?”, “You placed your phone on the kitchen counter.”); the retrieved visual captions describe “a phone on the counter near a bread box”; and the gaze coordinate indicates fixation on the counter area. Rephrased Question: “Where is the phone now?” # Instructions for Answering the Rephrased Question: Using the rephrased question and the full multimodal context, generate a factually correct and contextually grounded answer. Ensure that the answer is consistent with visual evidence, spatial layout, object interactions, temporal history, and embodied cues (gaze and hand pose). Avoid hallucinating entities or events not supported by the provided context. The final answer should be concise, accurate, and specifically aligned with the rephrased question. Your output must be a Python dictionary with the following fields: ’rephrased question’: text, ’answer’: text. Do not include explanation, reasoning traces, chain-of-thought, or additional commentary. Only provide the dictionary. Please process the following inputs: Incoming Question: question Context for Reasoning C t : Retrieved Visual Descriptions: V des t , Relevant Historical QA: (Q rel t , A rel i ), Embodied Feature Coordinates: V coor t