Paper deep dive
MemLife: Curating and Reasoning over Long-Term Egocentric Video Memories
Guangzhi Xiong, Xinyuan Zhang, Xiao Yang, Hyokun Yun, Kai Zhang, Shiun-Zu Kuo, Hyeonjeong Ha, Xilun Chen, Kai Sun, Lucas Liang, Guangqiang Dong, Ejaz Ahmed, Ahmed A Aly, Anuj Kumar, Raffay Hamid, Aidong Zhang, Xin Luna Dong
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Long-term egocentric video enables personalized AI assistants to reason about daily life. However, as video histories grow to hundreds of hours spanning months or years, reprocessing raw clips for every query becomes computationally prohibitive. Memory systems offer a scalable alternative by compacting videos into text representations, but often fail on practical benchmarks: either the memory does not preserve key evidence, or the retriever fails to locate relevant entries due to retrieval competition in growing search spaces. To address these challenges, we introduce MemLife, a multimodal memory system that constructs entity-grounded, first-person text episodes and retrieves them via a time-indexed agentic reader. Without training or query-time video access, MemLife improves over the strongest training-free baseline by 4.6--12.0% across four long-horizon benchmarks. To further improve memory quality, we propose MemOpt, a reinforcement learning framework that optimizes the memory writer to produce faithful, informative, and retrievable memories. MemOpt consistently improves MemLife by 2.7--5.0% across different video and question distributions, with gains that generalize across writer and reader backbones and memory systems.
Tags
Links
- Source: https://arxiv.org/abs/2609.40195v1
- Canonical: https://arxiv.org/abs/2609.40195v1
Trouble viewing inline? Open PDF directly →
Full Text
84,777 characters extracted from source content.
Expand or collapse full text
MemLife: Curating and Reasoning over Long-Term Egocentric Video Memories Guangzhi Xiong Affiliation: Meta Reality Labs Affiliation: University of Virginia Work done at Meta Xinyuan Zhang Affiliation: Meta Reality Labs Xiao Yang Affiliation: Meta Reality Labs Hyokun Yun Affiliation: Meta Reality Labs Kai Zhang Affiliation: Meta Reality Labs Shiun-Zu Kuo Affiliation: Meta Reality Labs Hyeonjeong Ha Affiliation: Meta Reality Labs Affiliation: University of Illinois Urbana-Champaign Work done at Meta Xilun Chen Affiliation: Meta Reality Labs Kai Sun Affiliation: Meta Reality Labs Lucas Liang Affiliation: Meta Reality Labs Guangqiang Dong Affiliation: Meta Reality Labs Ejaz Ahmed Affiliation: Meta Reality Labs Ahmed A Aly Affiliation: Meta Reality Labs Anuj Kumar Affiliation: Meta Reality Labs Raffay Hamid Affiliation: Meta Reality Labs Aidong Zhang Affiliation: University of Virginia Xin Luna Dong Affiliation: Meta Reality Labs Abstract Long-term egocentric video enables personalized AI assistants to reason about daily life. However, as video histories grow to hundreds of hours spanning months or years, reprocessing raw clips for every query becomes computationally prohibitive. Memory systems offer a scalable alternative by compacting videos into text representations, but often fail on practical benchmarks: either the memory does not preserve key evidence, or the retriever fails to locate relevant entries due to retrieval competition in growing search spaces. To address these challenges, we introduce MemLife, a multimodal memory system that constructs entity-grounded, first-person text episodes and retrieves them via a time-indexed agentic reader. Without training or query-time video access, MemLife improves over the strongest training-free baseline by 4.6–12.0% across four long-horizon benchmarks. To further improve memory quality, we propose MemOpt, a reinforcement learning framework that optimizes the memory writer to produce faithful, informative, and retrievable memories. MemOpt consistently improves MemLife by 2.7–5.0% across different video and question distributions, with gains that generalize across writer and reader backbones and memory systems. †correspondence: †guangzhi,aidong@virginia.edu, dylanz426,lunadong@meta.com 1 Introduction If an AI assistant could record a user’s life experiences as egocentric videos, captured by wearable devices such as smart glasses and GoPro cameras (Grauman et al., 2022), could it then answer any question the user asks about their past? At first glance, this is a Retrieval-Augmented Generation (RAG) problem over egocentric videos (Yang et al., 2025; Alam et al., 2026). However, question answering (QA) over such dense memories is substantially harder. On the data side, videos accumulate over time to prohibitive volumes, and scenes and events recur with subtle differences. On the system side, the latency budget for sifting through long, similar memories is tight, and the context window is too small to hold many visual frames. It is therefore crucial to compact raw videos into efficient representations that can serve as the primary source of evidence for downstream tasks. Natural-language text stands out as an appealing choice: orders of magnitude more compact than visual frames, natively consumable by Large Language Models (LLMs), and interpretable to users. We follow the line of work that converts multimodal video recordings into textual descriptions (Islam et al., 2024; Yang et al., 2025), either to facilitate retrieval of key evidence (Luo et al., 2025; Ren et al., 2026) or to directly support downstream generation (Yeo et al., 2026; Yin et al., 2026). However, deciding what to write poses a fundamental tradeoff (Figure 1): aggressive compression may discard information that future questions ask about, whereas conservative compression retains trivial details that dilute retrieval and inflate QA-time processing. Worse, writers may hallucinate content unsupported by the video, planting false evidence. In this paper, we address a critical question for QA over long-term memory: how can a system strike the right balance to remember only what is worth remembering, faithfully, and in a form that is easy to recall? Figure 1: Failure modes of long-term video memory systems. Dotted objects denote lost information (e.g., Week-1 video is deleted). Systems fail when relevant evidence is omitted during memory writing or missed during retrieval. Our proposed solutions outperform prior state-of-the-art systems. We answer this question in two steps: we first design a strong writer by hand, then learn a better one. Our first contribution, MemLife, is an agentic memory system whose writer follows two principles. First, it anchors memories in time and grounds entities across modalities, aligning spoken references with the people and objects observed in each episode. Second, it narrates in the first person, matching how users phrase questions about their own lives (e.g., “Where did I put my passport?”) and thereby narrowing the query-memory gap in agentic retrieval. To exploit these memories, MemLife’s reader combines agentic semantic search with time-scoped memory fetching, and presents retrieved episodes in chronological order for reasoning over long histories. In the presence of source videos, the agentic reader selectively invokes video-retrieval tools to sample raw video frames whenever visual details are required. Without writer training or query-time video access, MemLife improves over the strongest training-free baseline by up to 12.0% in accuracy. Our second insight is that learning what to write is, by itself, a powerful lever for long-term memory QA. We therefore take a bold step: rather than applying Reinforcement Learning (RL) to improve final answer quality (Guan et al., 2026; Yan et al., 2026; Wang et al., 2025b; Li et al., 2026), we apply RL only to memory writing, and examine whether this alone improves memory QA. We propose MemOpt, a learning framework that rewards Faithful, Informative, and Retrievable Memories, a reward system we call FIRM. Faithfulness penalizes hallucinated memories unsupported by the source video; informativeness encourages the writer to preserve details critical to answering future questions; and retrievability favors concise descriptions that enable easy and precise retrieval. Although MemOpt trains only the writer and does not rely on supervision from stronger models (Long et al., 2026; Zou et al., 2026), it further improves MemLife by 2.7–5.0%. Our paper makes the following contributions. • We introduce MemLife, a multimodal memory system that compresses egocentric videos into time- and entity-anchored, first-person episodes, and reasons over them with a versatile agentic reader. • We propose MemOpt, a learning framework whose FIRM reward optimizes the memory writer with multi-granular feedback on faithfulness, informativeness, and retrievability, to address the write-time failure modes we identified. • We show that MemOpt combined with MemLife outperforms the strongest prior baseline by 4.0–17.0%, while reducing the memory size by up to 31× (Appendix E). The trained writer transfers well: it consistently improves accuracy when plugged into other memory systems (e.g., EgoRAG) and backbones, and on out-of-domain videos and question types. 2 Related Work Writing memory over extended video horizons. Memory-augmented video paradigms differ in how they convert continuous streams into persistent representations. Retrieval-oriented systems encode local clips as flat text descriptions or visual-text indexes (Fan et al., 2025; Islam et al., 2024; Luo et al., 2025). To control memory growth, streaming architectures maintain fixed-budget recurrent buffers that continuously compress past frames (Guan et al., 2026; He et al., 2024; Qian et al., 2024; Song et al., 2024; Jin et al., 2025), though they often struggle to preserve distant or fine-grained details. Hierarchical frameworks summarize video streams across temporal tiers (Islam et al., 2024; Yang et al., 2025), while structured memory agents organize representations around entities, events, or scene graphs (Goletto et al., 2025; Long et al., 2026; Ren et al., 2026; Yeo et al., 2026; Yin et al., 2026). However, high-level abstractions suffer from information loss, and graph maintenance becomes computationally prohibitive. Furthermore, prior memory construction relies on heuristic prompt engineering rather than optimizing memory generation via task feedback. Reading memory across multi-session histories. Personal memory systems rely on readers to locate evidence scattered across extended temporal histories. Passive retrieval-based readers execute single-shot semantic or temporal queries over text indices (Lewis et al., 2020; Luo et al., 2025; Yang et al., 2025; Ren et al., 2026). Recent egocentric architectures introduce specialized access mechanisms—such as Memory Pointer Prompting (Ye et al., 2025) or multi-turn reasoning loops that reformulate queries, fetch time intervals, and inspect visual frames (Yeo et al., 2026; Yin et al., 2026; Gao et al., 2023; Wang et al., 2025a; Tian et al., 2026; Zhang et al., 2025). While effective during inference (Chandrasegaran et al., 2024; Wang et al., 2026b), using multi-turn readers during writer post-training conflates memory quality with reader execution noise. Because task accuracy depends on sampled tool calls, end-to-end task rewards provide a noisy, computationally prohibitive supervision signal for writer optimization. Training video memory writers. To move beyond heuristic prompt design, recent paradigms train memory writers using learned policies. One direction relies on supervised fine-tuning or imitation learning, distilling memory construction from proprietary demonstrations or QA instructions (e.g., EgoButler (Yang et al., 2025), M3-Agent (Long et al., 2026)). Another direction employs reinforcement learning, optimizing policies like TaskMem (Zou et al., 2026) and VST (Guan et al., 2026) against downstream task accuracy or QA preferences. However, static distillation restricts writer adaptability, while training purely on end-to-end task rewards introduces severe execution noise from reader reasoning. In contrast, MemOpt provides supervision from verified evidence, source video, and fixed reader actions, decoupling writer post-training from reader execution noise. 3 Methodology 3.1 Problem Definition and Solution Overview We begin by defining the Video Memory QA problem. Consider a stream of video (optionally egocentric) =(c1,…,cT)V=(c_1,…,c_T) of T segments. Each segment can be represented as a triplet ct=(Ft,St,τt)c_t=(F_t,S_t, _t), where FtF_t denotes the sampled visual frame, StS_t denotes the aligned audio transcriptions, and τt=[st,et] _t=[s_t,e_t] denotes the starting and ending time. Memory QA takes a question q arriving at time τq _q, and provides the answer based on the prior memory fragments: q=ct∈:et≤τq.V_q=\c_t :e_t≤ _q\. (1) Our first solution MemLife converts a video stream into persistent episodic memory and uses an agentic reader to retrieve and reason over relevant entries. Formally, MemLife employs a writer WθW_θ with parameters θ; the writer generates a textual description dtd_t for each memory segment mtm_t: dt∼Wθ(⋅∣Ft,St),mt=(dt,τt).d_t W_θ(· F_t,S_t), m_t=(d_t, _t). (2) Thus, the memory repository stores ℳ()=mtt=1TM(V)=\m_t\_t=1^T. For a question q, question answering uses the available memory ℳqM_q, optionally with the corresponding source-video history qV_q, where ℳq=mt∈ℳ():et≤τq.M_q=\m_t (V):e_t≤ _q\. (3) Our second solution, MemOpt, is a training framework that improves the memory writer by optimizing its parameters θ. Figure 2 gives the overview of the two solutions. Figure 2: Overview of MemLife’s episodic writer and time-indexed agentic reader, together with MemOpt’s decomposed writer supervision. Direct source-video access is optional for the reader. 3.2 MemLife System for Memory Writing and Reading Writer: MemLife separates question-independent memory construction from question-dependent memory access. Because future questions are unknown during writing, each entry must preserve information across modalities and express it in a form that supports later retrieval. The MemLife writer applies three designs for this purpose. Multimodal fusion. Information needed by future questions may appear in either the visual stream or speech. The writer therefore jointly interprets the frames and transcript such that evidence from both modalities can be preserved in one memory entry. Entity grounding. A transcript may mention an entity by its name, which provides an important cue for future QA. The writer aligns the speech with the memory segment and uses the identified name to refer to the visual referents in the description. First-person narration. Memory questions for egocentric videos naturally refer to the user as “I.” The writer therefore adopts the same first-person perspective, making its descriptions easier to match future egocentric questions. MemLife generates a description and its embedding for every clip independently, such that it avoids sequential dependencies and error propagation, and keeps total computation and storage linear in the recorded history. We also explored conditioning the writer on textual or multimodal context from the preceding segments, but neither variant improves aggregate accuracy (Appendix I). Reader: The MemLife reader is an agentic system that takes a question q and its timestamp τq _q as input and operates over multiple rounds by selecting actions from the action space A: =Rewrite(u,I|q,τq),SearchMemory(M¯|u,I,k),FetchMemory(M¯|I),FetchVideo(V¯|I,f),Answer(a|q,M¯,V¯). splitA=\& Rewrite(u,I|q, _q),\ SearchMemory( M|u,I,k),\ FetchMemory( M|I),\\ & FetchVideo( V|I,f),\ Answer(a|q, M, V)\. split (4) With Rewrite, the agent reasons over the current information, and transforms the input (q,τq)(q, _q) into a targeted search query u and/or a time interval I over the history. For a search query u, SearchMemory conducts the similarity search and returns k relevant entries M¯ M from the stored memories within interval I. With only interval I, the agent can call either FetchMemory, which returns all text memory entries M¯ M within I, or FetchVideo, which returns raw multimodal fragments V¯ V and f sampled frames within I. Finally, Answer generates the answer a based on retrieval results. The FetchVideo tool is disabled when source videos are unavailable at reading time. For video-available settings (MemLife-V), we store low-resolution redacted videos due to storage and privacy concerns, and sample limited frames to optimize computation. 3.3 MemOpt Framework for Optimizing Memory Writer MemOpt updates the writer parameters θ through supervision, while keeping the agentic reader fixed. We next present our FIRM reward model, the major recipe to improve memory writing. Theoretical foundation. The goal of MemOpt is to teach the writer what is worth remembering and which form is easy to recall. We next show the theoretical quantification. Let Q and A denote a random question and its answer, while V, ℳ M, and CQC_Q denote the available video history, its corresponding memory, and the context retrieved by the reader for Q. To isolate the quality of the written memory, we restrict the reader’s evidence source to ℳ M, excluding direct access to V that could otherwise bypass the memory. Because the writer rewrites V into ℳ M, and the reader constructs CQC_Q only from (Q,ℳ)(Q, M), their joint distribution factorizes as p(Q,A,,ℳ,CQ)=p(Q,A,)⋅pθ(ℳ∣)⋅p(CQ∣Q,ℳ).p(Q,A, V, M,C_Q)=p(Q,A, V)· p_θ( M V)· p(C_Q Q, M). (5) Let H(∣)H( ) denote conditional entropy and I(∣)I( ) conditional mutual information. Under this factorization, the additional uncertainty about the answer A when using the retrieved context CQC_Q instead of the video history V decomposes exactly as H(A∣Q,CQ)−H(A∣Q,)⏟total information loss= H(A Q,C_Q)-H(A Q, V)_total information loss= I(A;∣Q,ℳ)⏟Δinf: memory writing loss+I(A;ℳ∣Q,CQ)⏟Δret: memory recall loss. I(A; V Q, M)_ _inf: memory writing loss+ I(A; M Q,C_Q)_ _ret: memory recall loss. (6) The equation shows that answer-relevant information can be lost when the writer maps V to ℳ M (Δinf _inf) and when the reader retrieves CQC_Q from ℳ M (Δret _ret). These gaps motivate the informativeness and retrievability rewards, but do not measure whether ℳ M is grounded in V. Unsupported memory can distort the final answer, while final-answer error also reflects answerer reasoning. We therefore assess faithfulness directly against the source video. Appendix K provides the complete derivation. Faithfulness. Faithfulness asks whether every claim in a candidate is supported by its source segment. Because unsupported content may occupy only a few tokens, the feedback must also identify where it occurs. For candidate yi=(yi,1,…,yi,Li)y_i=(y_i,1,…,y_i,L_i) with LiL_i generated tokens under examination, we prompt the same frozen model to check the generated memory against its source segment by reproducing supported content exactly and minimally correcting unsupported spans, thereby localizing the grounding feedback. We denote by pfaith(w∣c,yi,yi,<t)p_faith(w c,y_i,y_i,<t) the probability that the evaluator generates a possible next token w and compute the faithfulness reward as Rfaith,i,t=1−[maxwpfaith(w∣c,yi,yi,<t)−pfaith(yi,t∣c,yi,yi,<t)].R_faith,i,t=1- [ _wp_faith(w c,y_i,y_i,<t)-p_faith(y_i,t c,y_i,y_i,<t) ]. (7) The reward lies in [0,1][0,1]. It equals 1 when the candidate token is the evaluator’s most probable continuation and decreases when the evaluator favors a correction. Informativeness. Informativeness asks whether the candidate itself preserves the answer-relevant evidence supplied by its source segment, independent of reader behavior. We represent the required evidence as a source-grounded key fact and test whether the candidate entails it. A frozen copy of the default writer model serves as both the key-fact extractor and entailment judge. The question and answer are used only to construct the training reward, leaving the writer question-independent. Formally, let c=ctc=c_t be a segment, y be a candidate memory generated by the writer, (c)Q(c) denote the set of questions for which c provides verified evidence, and assume each question q∈(c)q (c) is paired with a correct answer aqa_q. For each (q,aq)(q,a_q), the extractor identifies the observed fact kc,qk_c,q that supports the answer. Let Pent(y⇒kc,q)P_ent(y k_c,q) denote the judge’s estimated probability that y entails this fact. We average this probability across the relevant questions, Rinf(c,y)=1|(c)|∑q∈(c)Pent(y⇒kc,q).R_inf(c,y)= 1|Q(c)| _q (c)P_ent(y k_c,q). (8) Retrievability. Retrievability asks whether the relevant memory can be discovered at QA time. For a pair (c,y)(c,y), we approximate Δret _ret by checking whether y is returned for each question in (c)Q(c). For efficiency and stability, at the start of each epoch, we run the MemLife reader on every training question and cache its memory-access actions. For each candidate y, we replay these fixed actions to obtain the returned context q(y)C_q(y) without rerunning reader reasoning. The retrievability reward is Rret(c,y)=1|(c)|∑q∈(c)[y∈q].R_ret(c,y)= 1|Q(c)| _q (c)1[y _q]. (9) Multi-granular group-relative optimization. For each training segment c, the writer samples a group g=yii=1Gg=\y_i\_i=1^G of G candidates. MemOpt combines the three dimensions in FIRM multiplicatively and obtains the reward of token t in candidate i as xi,t=Rfaith,i,tRinf(c,yi)Rret(c,yi).x_i,t=R_faith,i,tR_inf(c,y_i)R_ret(c,y_i). (10) A token receives high credit only when it is faithful and its memory is informative and retrievable. Standard group-relative optimization normalizes one scalar reward per candidate and broadcasts the resulting advantage to all of its tokens. MemOpt must instead preserve variation from the token-level faithfulness signal. For candidate yiy_i with LiL_i generated tokens, we compute x¯i=1Li∑t=1Lixi,t,μg=1G∑i=1Gx¯i,σg2=1G∑i=1G1Li∑t=1Li(xi,t−μg)2. x_i= 1L_i _t=1^L_ix_i,t, _g= 1G _i=1^G x_i, _g^2= 1G _i=1^G 1L_i _t=1^L_i(x_i,t- _g)^2. (11) Averaging within each candidate before computing the group statistics prevents longer memories from dominating the normalization. The token-level advantage is A~i,t=(xi,t−μg)/(σg+ϵn), A_i,t=(x_i,t- _g)/( _g+ _n), (12) where ϵn _n stabilizes normalization. Training then follows GRPO (Shao et al., 2024). 4 Experiments 4.1 Experimental Setup Datasets. We evaluate on SuperMemory-VQA (Alam et al., 2026), EgoLifeQA (Yang et al., 2025), and two extended settings. SuperMemory-LVQA combines all ten SuperMemory-VQA histories while retaining the original test questions, expanding the retrieval space with cross-subject distractors. EgoLife-EQA tests questions about recurring events whose supporting evidence is manually verified against the source recordings. We train MemOpt on SuperMemory-VQA subjects S1–S6, validate on S7–S8, and test on S9–S10. All other benchmarks are used for test only. More details about data are provided in Appendix A. Models and baselines. Qwen3.5-9B is used as the backbone for both writer and reader across systems. The generalizability study also evaluates Qwen3.6-27B. Training-free baselines include Video ReCap (Islam et al., 2024), EgoRAG (Yang et al., 2025), Video-RAG (Luo et al., 2025), VideoARM (Yin et al., 2026), EGAgent (Rege et al., 2026), and WorldMM (Yeo et al., 2026). Trained baselines include EgoButler (Yang et al., 2025), VST (Guan et al., 2026), TaskMem (Zou et al., 2026), and M3-Agent (Long et al., 2026). Appendix B provides further details. Sections 4.2, 4.3, 4.4, 4.5 address the following research questions (RQs): • RQ1. Does MemLife outperform existing systems on long-term egocentric video memory question answering? Does MemOpt further improve performance? • RQ2. Do MemLife and MemOpt actually improve memory quality? • RQ3. How generalizable is MemOpt and its trained writer? • RQ4. Is each component in MemLife and MemOpt important? Additional experiments and analyses can be found in the Appendix. 4.2 Performance Comparison to Baselines Among systems without training, MemLife outperforms baselines in accuracy across benchmarks in Table 1. Enabling video access through MemLife-V produces only modest changes, showing that the gains do not depend on revisiting the original recordings. On SuperMemory-LVQA, the methods maintain accuracy close to their SuperMemory-VQA results despite lower annotated recall. While having cross-subject distractors, SuperMemory-LVQA may also contain subject interactions that provide useful context outside annotated evidence, which explains the accuracy-recall inconsistency. Table 1: Comparison with existing video memory systems. Oracle Context bypasses retrieval by supplying all annotated source-video evidence directly to the reader. MemLife-V permits source-video access. Bold and underlined values mark the best and second-best results within each group. Method SuperMemory-VQA EgoLifeQA SuperMemory-LVQA EgoLife-EQA Accuracy Recall Accuracy Recall Accuracy Recall Accuracy Recall Reference Oracle Context 67.58 100.00 66.20 100.00 67.58 100.00 61.00 100.00 Without Memory-Writer Training Video ReCap 36.28 54.01 35.20 27.40 39.17 16.79 38.00 27.00 EgoRAG 49.28 66.79 48.20 29.60 47.51 47.90 38.00 13.00 Video-RAG 43.98 59.54 38.80 20.80 49.28 35.31 28.00 6.00 VideoARM 39.33 70.23 36.00 45.60 40.93 9.35 40.00 58.00 EGAgent 43.66 48.09 34.00 21.80 41.73 26.53 26.00 16.00 WorldMM 42.05 74.62 42.20 43.80 45.91 24.05 27.00 35.00 MemLife 56.50 80.53 52.80 48.40 56.18 46.18 52.00 44.00 MemLife-V 57.78 84.35 53.20 50.20 57.95 48.47 50.00 45.00 With Memory-Writer Training EgoButler 36.92 49.05 44.00 25.00 38.68 36.64 34.00 18.00 VST 37.56 56.87 33.60 3.40 31.94 0.00 36.00 5.00 TaskMem 50.88 65.08 46.20 40.20 47.83 29.96 35.00 22.00 M3-Agent 53.93 54.20 35.40 9.60 54.90 13.74 23.00 4.00 MemLife + MemOpt 60.35 81.49 56.60 50.60 58.91 43.13 57.00 41.00 MemLife-V + MemOpt 60.83 83.78 57.60 53.00 61.96 49.62 57.00 43.00 With writer training, MemOpt improves MemLife accuracy on all benchmarks and outperforms every trained baseline. Although trained only on SuperMemory-VQA, it also improves MemLife performance on the other three benchmarks, demonstrating transfer across video distributions, question types, and memory scales. Recall changes are mixed, indicating the accuracy gains also reflect more answer-useful memory content rather than only retrieving annotated evidence. 4.3 Analysis of Memory Quality To separate retrieval from the quality of stored memory, Figure 3 reports retrieval recall, standard answer accuracy under normal retrieval, and oracle accuracy when evidence-aligned entries are given directly to the reader. Compared with EgoRAG, MemLife improves recall in every category and raises oracle accuracy overall and in most categories, indicating gains in both retrieval and memory content. MemOpt provides category-dependent gains, improving retrieval for some question types and the answer usefulness of stored content for others. Interestingly, on RelationMap, EgoRAG matches the optimized MemLife system in standard accuracy despite lower recall and oracle accuracy. Its normal retrieval may therefore surface alternative useful context outside the annotations, while its evidence-aligned memories do not reliably preserve the relations needed for answering. Figure 3: Memory quality across EgoLifeQA question categories. Oracle accuracy is measured by providing memory entries aligned with annotated evidence directly to the reader. Table 2 illustrates how MemOpt changes the stored content. In the first example, the untrained writer mistakes lentils for corn, while MemOpt corrects the object without losing the surrounding action. In the second, the original memory uses a vague pronoun and omits the relevant food, whereas MemOpt identifies the person and records the baked chicken needed to answer the question. These examples show that MemOpt removes unsupported details while making answer-relevant entities and events explicit, improving both faithfulness and informativeness. Table 2: Examples of memory corrections learned through MemOpt. Red highlights errors and yellow highlights corrected content. Frames Question Q: […] Did I add the milk before or after the eggs when making the batter? Q: […] Did I set a reminder for a backup dinner plan? A: You did not add milk or eggs to the batter; you only added lentils, salt, and spices. A: No […] However, you did mention earlier that you have baked chicken available. MemLife I am in a kitchen […] transfer yellow corn kernels from a small food processor bowl […] move toward the sink area […] […] preparing food […] they respond to my question about being hungry by saying they can wait for the food […] MemLife + MemOpt I am in a kitchen […] yellowish-orange granular material, which appears to be cooked lentils […] to the sink area […] […] I am preparing a meal, specifically baked chicken, and offering it to B, who indicates they are not hungry and can wait. 4.4 Generalizability of MemOpt Training We then examine whether MemOpt depends on the writer backbone by optimizing both Qwen3.5-9B and Qwen3.6-27B. Table 3 shows that MemOpt improves every writer–reader pairing on both benchmarks. Changing the writer scale produces only modest differences, indicating that the training benefit does not depend on a particular writer backbone. The Qwen3.6-27B reader further tests whether the optimized writer transfers beyond the Qwen3.5-9B reader used to collect retrievability supervision. Using the stronger reader substantially raises absolute accuracy for both writers. MemOpt continues to improve every setting, although its gains become smaller with the stronger reader, suggesting that reader capacity can compensate for some deficiencies in written memory while writer optimization remains beneficial. Table 3: Answer accuracy (%) across writer and reader backbones. Writer Training Qwen3.5-9B Reader Qwen3.6-27B Reader SuperMemory-VQA EgoLifeQA SuperMemory-VQA EgoLifeQA Qwen3.5-9B Zero-shot 56.50 52.80 66.93 59.60 MemOpt 60.35 56.60 67.90 60.20 Qwen3.6-27B Zero-shot 57.14 55.20 66.77 59.80 MemOpt 60.67 56.60 68.06 60.60 Figure 4: Generalizability of MemOpt writer training across memory systems. Beyond backbone changes, we study whether MemOpt transfers across memory systems. We optimize the EgoRAG writer and evaluate the original and optimized memories with both the native EgoRAG reader and the MemLife agentic reader. We also evaluate the optimized MemLife writer with the EgoRAG reader. Figure 4 shows that training improves the native EgoRAG pipeline on both benchmarks. Using MemLife to read optimized EgoRAG memories provides further gains, while the highest performance is achieved with the trained MemLife writer. 4.5 Ablation Studies To analyze how different components in our proposed methods contribute to the overall performance, we first ablate each component in the writer and reader designs of MemLife, with the results shown in Table 4. For the writer design, we progressively add entity grounding and first-person narration to multimodal fusion, and examine how performance will change. Table 4: Ablation studies on the writer and reader components in MemLife. Accuracy is reported for SuperMemory-VQA and EgoLifeQA. Recall on the long-term EgoLifeQA task is also reported. Writer Ablation Multimodal Fusion Entity Grounding First-person Narration SuperMemory-VQA EgoLifeQA EgoLifeQA Recall Zero-shot MemOpt Zero-shot MemOpt Zero-shot MemOpt ✔ ✗ ✗ 52.33 56.98 52.80 52.60 48.60 47.00 ✔ ✔ ✗ 54.09 59.87 52.00 54.40 45.60 47.60 ✔ ✔ ✔ 56.50 60.35 52.80 56.60 48.40 50.60 Reader Ablation Agentic Reasoning Time Anchoring Chronological Ordering SuperMemory-VQA EgoLifeQA EgoLifeQA Recall Zero-shot MemOpt Zero-shot MemOpt Zero-shot MemOpt ✔ ✗ ✗ 56.02 58.27 50.40 51.80 50.60 48.20 ✔ ✔ ✗ 56.50 58.59 50.60 54.00 50.00 50.00 ✔ ✔ ✔ 56.50 60.35 52.80 56.60 48.40 50.60 From the upper block of Table 4, we observe that entity grounding generally improves accuracy, particularly after MemOpt training, but provides little benefit to recall. First-person narration further improves accuracy and consistently raises EgoLifeQA recall, supporting its role in aligning written memories with wearer-centered queries. We then ablate the MemLife reader by adding time anchoring and chronological ordering to agentic reasoning. The lower block of Table 4 shows that having time anchoring in the search tool provides its clearest benefit on optimized EgoLifeQA memories, where its accuracy gain is accompanied by higher recall. Chronological ordering further improves accuracy despite small or mixed recall changes, indicating that preserving event order primarily benefits reasoning over retrieved evidence. The complete reader performs best overall, with the largest gains appearing after writer optimization. For the design of MemOpt, we compare various supervision signals used to train the writer. Table 5 shows that teacher imitation with Qwen3.6-27B provides only modest gains, while final-answer accuracy supervision produces inconsistent changes across benchmarks. Among the proposed reward dimensions, retrievability alone raises recall on both benchmarks but does not consistently improve accuracy. Adding informativeness strongly benefits SuperMemory-VQA, although its gain does not transfer to EgoLifeQA. With faithfulness as a regularizer, the complete objective achieves the highest accuracy on both benchmarks while retaining higher recall than the untrained writer. Table 5: Comparison of supervision signals used during memory writer training. Writer Supervision SuperMemory-VQA EgoLifeQA Teacher Accuracy Retrievability Informativeness Faithfulness Accuracy Recall Accuracy Recall ✗ ✗ ✗ ✗ ✗ 56.50 80.53 52.80 48.40 ✔ ✗ ✗ ✗ ✗ 57.78 83.59 53.00 49.00 ✗ ✔ ✗ ✗ ✗ 57.46 79.01 52.60 47.80 ✗ ✗ ✔ ✗ ✗ 53.45 82.63 54.40 53.20 ✗ ✗ ✔ ✔ ✗ 60.03 84.35 51.40 47.60 ✗ ✗ ✔ ✔ ✔ 60.35 81.49 56.60 50.60 Finally, we perform ablation studies on the faithfulness granularity and reward aggregation strategy used in MemOpt. Following the probability-based judgment used for informativeness, the sequence-level variant assigns every token the same score about whether the complete memory is supported by its source segment. Different from multiplicative aggregation, the additive variant sums the three rewards. As shown in Table 6, under additive aggregation, token-level faithfulness maintains similar SuperMemory-VQA accuracy with slightly lower recall, but improves both metrics on out-of-domain EgoLifeQA. With token-level faithfulness fixed, multiplicative aggregation further improves accuracy on both benchmarks, and achieves the best performance overall. Table 6: Ablation of faithfulness granularity and reward aggregation. Training Design SuperMemory-VQA EgoLifeQA Faithfulness Aggregation Accuracy Recall Accuracy Recall Sequence-level Additive 58.75 83.78 55.00 46.40 Token-level Additive 58.91 81.68 56.00 49.40 Token-level Multiplicative 60.35 81.49 56.60 50.60 5 Conclusion We introduced MemLife, an agentic memory system for long-term egocentric video that constructs time- and entity-anchored, first-person episodes and accesses them through time-scoped retrieval and chronological evidence organization. We further proposed MemOpt, which applies reinforcement learning only to the memory writer through the FIRM objective for faithful, informative, and retrievable memories. MemLife outperforms state-of-the-art training-free systems, while MemOpt provides further gains that transfer across writer and reader backbones, memory systems, and out-of-domain video and question distributions. Together, the results demonstrate that learning what to remember is an effective and generalizable approach to long-term video question answering. References Alam et al. (2026) S. Alam, S. I. Siam, M. J. Proulx, J. Fort, R. Newcombe, H. J. Kim, and M. Zhang SuperMemory-vqa: an egocentric visual question-answering benchmark for long-horizon memory. Note: arXiv preprint arXiv:2606.00825 Cited by: §A.1, §1, §4.1. Chandrasegaran et al. (2024) K. Chandrasegaran, A. Gupta, L. M. Hadzic, T. Kota, J. He, C. Eyzaguirre, Z. Durante, M. Li, J. Wu, and L. Fei-Fei HourVideo: 1-hour video-language understanding. In Advances in Neural Information Processing Systems, Cited by: §2. Chhikara et al. (2025) P. Chhikara, D. Khant, S. Aryan, T. Singh, and D. Yadav Mem0: building production-ready ai agents with scalable long-term memory. Note: arXiv preprint arXiv:2504.19413 Cited by: Appendix J. Fan et al. (2025) Y. Fan, X. Ma, R. Wu, Y. Du, J. Li, Z. Gao, and Q. Li VideoAgent: a memory-augmented multimodal agent for video understanding. In Computer Vision – ECCV, Cited by: §2. Gao et al. (2023) D. Gao, L. Ji, L. Zhou, K. Q. Lin, J. Chen, Z. Fan, and M. Z. Shou AssistGPT: a general multi-modal assistant that can plan, execute, inspect, and learn. Note: arXiv preprint arXiv:2306.08640 Cited by: §2. Goletto et al. (2025) G. Goletto, T. Nagarajan, G. Averta, and D. Damen AMEGO: active memory from long egocentric videos. In Computer Vision – ECCV, Cited by: §2. Grauman et al. (2022) K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu, M. Martin, T. Nagarajan, I. Radosavovic, S. K. Ramakrishnan, F. Ryan, J. Sharma, M. Wray, M. Xu, E. Z. Xu, C. Zhao, et al. Ego4D: around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Cited by: §1. Guan et al. (2026) Y. Guan, L. Yin, D. Liang, J. Ju, Z. Luo, J. Luan, Y. Liu, and X. Bai Video streaming thinking: videollms can watch and think simultaneously. In Computer Vision – ECCV, Cited by: §A.2, §1, §2, §2, §4.1. He et al. (2024) B. He, H. Li, Y. K. Jang, M. Jia, X. Cao, A. Shah, A. Shrivastava, and S. Lim MA-lmm: memory-augmented large multimodal model for long-term video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Cited by: §2. Islam et al. (2024) M. M. Islam, N. Ho, X. Yang, T. Nagarajan, L. Torresani, and G. Bertasius Video recap: recursive captioning of hour-long videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Cited by: §A.2, §1, §2, §4.1. Jin et al. (2025) H. Jin, Q. Wang, W. Zhang, Y. Liu, and S. Cheng VideoMem: enhancing ultra-long video understanding via adaptive memory management. Note: arXiv preprint arXiv:2512.04540 Cited by: §2. Kang et al. (2025) J. Kang, M. Ji, Z. Zhao, and T. Bai Memory OS of AI agent. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, Cited by: Appendix J. Kim et al. (2026) T. Kim, K. Kim, and S. J. Hwang Agent memory distillation: empowering small llm agents with hierarchical teacher memory. Note: arXiv preprint arXiv:2608.07169 Cited by: Appendix J. Lewis et al. (2020) P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W. Yih, T. Rocktäschel, S. Riedel, and D. Kiela Retrieval-augmented generation for knowledge-intensive nlp tasks. In Advances in Neural Information Processing Systems, Cited by: §2. Li et al. (2026) Y. Li, S. Banerjee, and T. Che EMBER: efficient memory via budgeted evidence retention for long-horizon agents. Note: arXiv preprint arXiv:2606.05894 Cited by: Appendix J, §1. Long et al. (2026) L. Long, Y. He, W. Ye, Y. Pan, Y. Lin, H. Li, J. Zhao, and W. Li Seeing, listening, remembering, and reasoning: a multimodal agent with long-term memory. In International Conference on Learning Representations, Cited by: §A.2, §1, §2, §2, §4.1. Luo et al. (2025) Y. Luo, X. Zheng, G. Li, S. Yin, H. Lin, C. Fu, J. Huang, J. Ji, F. Chao, J. Luo, and R. Ji Video-rag: visually-aligned retrieval-augmented long video comprehension. In Advances in Neural Information Processing Systems, Cited by: §A.2, §1, §2, §2, §4.1. Packer et al. (2024) C. Packer, S. Wooders, K. Lin, V. Fang, S. G. Patil, I. Stoica, and J. E. Gonzalez MemGPT: towards llms as operating systems. Note: arXiv preprint arXiv:2310.08560 Cited by: Appendix J. Pan et al. (2025) Z. Pan, Q. Wu, H. Jiang, X. Luo, H. Cheng, D. Li, Y. Yang, C. Lin, H. V. Zhao, L. Qiu, and J. Gao SeCom: on memory construction and retrieval for personalized conversational agents. In International Conference on Learning Representations, Cited by: Appendix J. Park et al. (2023) J. S. Park, J. O’Brien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bernstein Generative agents: interactive simulacra of human behavior. In Proceedings of the Annual ACM Symposium on User Interface Software and Technology, Cited by: Appendix J. Qian et al. (2024) R. Qian, X. Dong, P. Zhang, Y. Zang, S. Ding, D. Lin, and J. Wang Streaming long video understanding with large language models. In Advances in Neural Information Processing Systems, Cited by: §2. Rege et al. (2026) A. Rege, A. Sadhu, Y. Li, K. Li, R. K. Vinayak, Y. Chai, Y. J. Lee, and H. J. Kim Agentic very long video understanding. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, Cited by: §A.2, §4.1. Ren et al. (2026) X. Ren, L. Xu, L. Xia, S. Wang, D. Yin, and C. Huang VideoRAG: retrieval-augmented generation with extreme long-context videos. In Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Cited by: §1, §2, §2. Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo DeepSeekMath: pushing the limits of mathematical reasoning in open language models. Note: arXiv preprint arXiv:2402.03300 Cited by: §3.3. Shen et al. (2026) Z. Shen, Z. Wu, F. Lai, S. Lian, and Y. Rao MemBuilder: reinforcing LLMs for long-term memory construction via attributed dense rewards. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, Cited by: Appendix J. Song et al. (2024) E. Song, W. Chai, G. Wang, Y. Zhang, H. Zhou, F. Wu, H. Chi, X. Guo, T. Ye, Y. Zhang, Y. Lu, J. Hwang, and G. Wang MovieChat: from dense token to sparse memory for long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Cited by: §2. Tian et al. (2026) S. Tian, R. Wang, H. Guo, P. Wu, Y. Dong, X. Wang, J. Yang, H. Zhang, H. Zhu, and Z. Liu Ego-r1: agentic chain-of-tool-thought for ultra-long egocentric video reasoning. IEEE Transactions on Pattern Analysis and Machine Intelligence 48 (10), p. 12116–12131. External Links: Document Cited by: §2. Wang et al. (2026a) J. Wang, H. Zhao, guanghui Pan, X. Wang, Y. Wang, Q. Deng, and M. Zhang SAGE: a self-evolving agentic graph-memory engine for structure-aware associative memory. In Frontiers in Graph Machine Learning for the Large Model Era, Cited by: Appendix J. Wang et al. (2025a) X. Wang, Y. Zhang, O. Zohar, and S. Yeung-Levy VideoAgent: long-form video understanding with large language model as agent. In Computer Vision – ECCV, Cited by: §2. Wang et al. (2025b) Y. Wang, R. Takanobu, Z. Liang, Y. Mao, Y. Hu, J. McAuley, and X. Wu Mem-α: learning memory construction via reinforcement learning. Note: arXiv preprint arXiv:2509.25911 Cited by: Appendix J, §1. Wang et al. (2026b) Z. Wang, Y. Zhang, S. Yu, C. Zhang, Z. Zhao, J. Yoon, H. Lee, G. Bertasius, and M. Bansal EgoMemReason: a memory-driven reasoning benchmark for long-horizon egocentric video understanding. In Conference on Language Modeling, Cited by: §2. Xu et al. (2025) W. Xu, Z. Liang, K. Mei, H. Gao, J. Tan, and Y. Zhang A-mem: agentic memory for llm agents. In Advances in Neural Information Processing Systems, Cited by: Appendix J. Yan et al. (2026) S. Yan, X. Yang, Z. Huang, E. Nie, Z. Ding, Z. Li, X. Ma, J. Bi, K. Kersting, J. Z. Pan, H. Schuetze, V. Tresp, and Y. Ma Memory-r1: enhancing large language model agents to manage and utilize memories via reinforcement learning. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, Cited by: Appendix J, §1. Yang et al. (2025) J. Yang, S. Liu, H. Guo, Y. Dong, X. Zhang, S. Zhang, P. Wang, Z. Zhou, B. Xie, Z. Wang, B. Ouyang, Z. Lin, M. Cominelli, Z. Cai, B. Li, Y. Zhang, P. Zhang, F. Hong, J. Widmer, F. Gringoli, et al. EgoLife: towards egocentric life assistant. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Cited by: §A.1, §A.2, §A.2, §1, §1, §2, §2, §2, §4.1, §4.1. Ye et al. (2025) H. Ye, H. Zhang, E. Daxberger, L. Chen, Z. Lin, Y. Li, B. Zhang, H. You, D. Xu, Z. Gan, J. Lu, and Y. Yang MMEgo: towards building egocentric multimodal LLMs for video QA. In International Conference on Learning Representations, Cited by: §2. Yeo et al. (2026) W. Yeo, K. Kim, J. Yoon, and S. J. Hwang WorldMM: dynamic multimodal memory agent for long video reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Cited by: §A.2, §1, §2, §2, §4.1. Yin et al. (2026) Y. Yin, Q. Meng, M. Chen, J. Ding, Z. Shao, and Z. Yu VideoARM: agentic reasoning over hierarchical memory for long-form video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Cited by: §A.2, §1, §2, §2, §4.1. Zhang et al. (2026) K. Zhang, X. Zhang, H. Jiang, S. Kuo, H. Yun, E. Ahmed, S. Oraby, Z. Li, S. Sharma, A. Lee, A. A. Aly, A. Kumar, R. Hamid, and X. L. Dong SaliMory: orchestrating cognitive memory for conversational agents. Note: arXiv preprint arXiv:2606.04120 Cited by: Appendix J. Zhang et al. (2025) X. Zhang, Z. Jia, Z. Guo, J. Li, B. Li, H. Li, and Y. Lu Deep video discovery: agentic search with tool use for long-form video understanding. In Advances in Neural Information Processing Systems, D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen (Eds.), Vol. 38, Main Conference, p. 89863–89895. External Links: Document, Link Cited by: §2. Zhong et al. (2024) W. Zhong, L. Guo, Q. Gao, H. Ye, and Y. Wang Memorybank: enhancing large language models with long-term memory. In Proceedings of the AAAI Conference on Artificial Intelligence, Cited by: Appendix J. Zou et al. (2026) T. Zou, Y. He, T. Qiu, Y. Lin, and H. Li Task-focused memorization for multimodal agents. Note: arXiv preprint arXiv:2605.31075 Cited by: §A.2, §1, §2, §4.1. Appendix A Dataset and Baseline Details A.1 Datasets and Splits SuperMemory-VQA contains ten subjects recorded across multiple sessions (Alam et al., 2026). We exclude 82 questions whose annotated evidence refers to source videos unavailable in the public release, leaving 4,771 questions. We partition these questions by subject, using S1–S6 for training, S7–S8 for validation, and S9–S10 for testing. These splits contain 3,425, 723, and 623 questions, respectively. EgoLifeQA contains 500 multiple-choice questions about a continuous seven-day recording (Yang et al., 2025). We use its complete question set only for testing. SuperMemory-LVQA increases the retrieval space by concatenating the histories of all ten SuperMemory-VQA subjects into one memory store. It retains the 623 questions from the SuperMemory-VQA test split, with the remaining subject histories as additional distractors. EgoLife-EQA contains 100 test questions about recurring events across multiple days in EgoLife. We identify candidate events from the official captions and formulate questions about them. Two annotators independently verify whether each candidate evidence segment supports its question’s annotated answer, reaching 92.1% agreement. They resolve disagreements through discussion, and we discard questions without verified evidence or an unambiguous answer. Table 7 groups the EgoLife-EQA questions into three categories. Frequency-counting questions ask on which days an activity occurred and may have multiple correct options, such as “On which days during the seven-day period did I go grocery shopping? Choose all that apply.” Routine questions ask what activity typically occurs at a given time of day, such as “What do I usually check on my phone in the morning?” Comparison questions ask which of two activities occurred on more days, such as “Which activity did I do in the bedroom on more days—browsing social media or browsing products online?” Table 7: Composition of EgoLife-EQA by question type. Correct options, evidence segments, and distinct evidence days are averaged per question. Type # Questions # Options Avg. Correct Options Avg. Evidence Segments Avg. Distinct Days Frequency counting 45 8 2.33 2.31 2.31 Routine 31 2–4 1.00 2.97 2.97 Comparison 24 3 1.00 8.04 5.54 All 100 – 1.60 3.89 3.29 Table 8 compares the scale of the accessible video history and evidence annotations across the four evaluation settings, where only SuperMemory-VQA supplies supervision for MemOpt. Table 8: Dataset statistics. Video hours and time spans are averaged over the history available to each question. Evidence segments and their total duration are averaged per question. Dataset # Questions Avg. Video Hours Avg. Time Span (days) Avg. Evidence Segments Avg. Evidence Duration (s) SuperMemory-VQA 4771 3.97 12.50 1.34 57.1 EgoLifeQA 500 22.65 2.80 1.10 32.3 SuperMemory-LVQA 623 47.90 127.36 1.24 56.3 EgoLife-EQA 100 43.06 6.33 3.89 116.7 A.2 Baselines Without task-specific writer training. Video ReCap recursively builds clip-, segment-, and video-level captions by combining visual features with captions from the preceding hierarchy (Islam et al., 2024). EgoRAG builds clip-, hour-, and day-level memories and retrieves relevant clips using visual and textual similarity (Yang et al., 2025). Video-RAG indexes OCR, ASR, and object-detection text, then provides retrieved text and sampled video frames to a VLM (Luo et al., 2025). The remaining systems perform adaptive multimodal access. VideoARM constructs a query-conditioned hierarchical memory online while inspecting progressively narrower regions of the source video (Yin et al., 2026). EGAgent plans over a temporal entity scene graph with visual-frame and transcript search (Rege et al., 2026). WorldMM iteratively retrieves from multiscale episodic and semantic graphs and a visual memory (Yeo et al., 2026). With learned memory construction. EgoButler combines EgoRAG with EgoGPT, which is fine-tuned for both visual-audio captioning and question answering (Yang et al., 2025). VST trains a single streaming VideoLLM to generate both textual memory and final answers through supervised fine-tuning and answer-based reinforcement learning (Guan et al., 2026). TaskMem instead optimizes a memorization policy with model-judged quality rewards followed by task-relevance preference learning, leaving QA to a separate answer generator (Zou et al., 2026). M3-Agent trains an entity-centric episodic and semantic memory writer through imitation of synthetic demonstrations, then separately trains its memory-search controller with reinforcement learning (Long et al., 2026). Evaluation protocol. For EgoButler, VST, and TaskMem, we use the released EgoGPT-7B, VST-32B, and TaskMem-30B checkpoints, respectively,11 1 Official checkpoints: EgoGPT-7B, VST-32B, and TaskMem-30B. where some have much larger capacity than our tested 9B model. Because M3-Agent does not release a checkpoint, we reproduce its trained memorizer and controller with the same Qwen3.5-9B backbone as our method. All other replaceable model components also use Qwen3.5-9B. Appendix C shows additional experimental results on the reproduced training methods with matched backbone models and training data. All methods receive the same questions, answer options, and causal video histories, and we recompute their metrics using a common answer parser. The Oracle Context reference in Table 1 bypasses retrieval by directly supplying the answerer with all annotated source-video evidence for each question. Its retrieval recall is therefore 100% by construction. Among baselines without task-specific writer training, Video-RAG, VideoARM, EGAgent, and WorldMM retain query-time access to visual evidence, whereas the default MemLife operates only on written memory. Among learned systems, we implement VST and M3-Agent with models trained on both memory construction and their answering or control components. EgoButler, TaskMem, and MemOpt instead adopt only a trained memory writer while keeping the downstream reader fixed. A.3 Evaluation Metrics Let Q denote the test questions, AqA_q the set of annotated correct option labels, and A^q A_q the predicted set produced by the answer parser. These sets contain one label for single-choice questions. For multi-select EgoLife-EQA questions, a prediction is correct only if it exactly matches the complete annotated set. We compute answer accuracy over all test questions as Acc=1||∑q∈[A^q=Aq].Acc= 1|Q| _q 1[ A_q=A_q]. (13) Retrieval recall is computed over the subset G⊆Q_G containing questions with at least one annotated temporal evidence interval. Let GqG_q and HqH_q denote the source-indexed temporal intervals annotated for q and returned to the reader, respectively. A question counts as retrieved when at least one returned interval overlaps an annotated interval from the same source video. We compute Recall=1|G|∑q∈G[∃g∈Gq,h∈Hqsuch thath∩g≠∅].Recall= 1|Q_G| _q _G1\! [∃ g∈ G_q,\,h∈ H_q\ such that\ h∩ g≠ ]. (14) For a single-shot reader, HqH_q is its retrieved context. For an agentic reader, HqH_q is the union of evidence returned by all executed search and fetch actions. This union measures the evidence actually available during the interaction rather than evidence recoverable by an unexecuted query. Appendix B Implementation Details B.1 Models and Inference Both the default MemLife writer and reader use Qwen3.5-9B. The backbone study additionally uses Qwen3.6-27B as the writer and reader, producing all four combinations of the two models. Each writer generates one memory store per dataset, and that same store is evaluated by both reader backbones. Swapping the reader therefore does not regenerate or alter the memory. All reader weights remain frozen, including during MemOpt training. The writer processes each 30-second segment independently from eight frames at 704-pixel resolution and its speech transcript. A trailing segment shorter than one second is folded into the preceding segment. Memory descriptions are generated greedily with a maximum of 512 tokens. We embed each description using BAAI/bge-large-en-v1.5 and perform exact inner-product search over normalized embeddings. At evaluation, the agentic reader can call SearchMemory and FetchMemory for at most ten rounds. Semantic search returns at most 32 entries, while interval-based fetching returns at most 64 entries. The default reader has no access to source video. The MemLife-V variant additionally allows the reader to inspect up to 50 sampled frames and the transcript from a selected temporal interval. Both writer and reader generations use greedy decoding for deterministic results. More details about the writer and reader prompts are in Appendix L. B.2 MemOpt Training All evaluators use Qwen3.5-9B, and both their parameters and the reader parameters remain frozen. We train the writer for three epochs and rebuild the validation memory after each epoch. We select the checkpoint with the highest sum of answer accuracy and retrieval recall on S7–S8, then evaluate it once on each test benchmark. Table 9 summarizes the training configuration. The Kullback–Leibler regularizer is applied directly to the loss against the frozen initial writer rather than incorporated into the reward, so it does not affect the group-relative advantages. Training rollouts are sampled at temperature 1.01.0 to provide within-group variation, whereas deployment uses greedy decoding. Table 9: MemOpt training hyperparameters. Group Parameter Value Sampling Group size G (rollouts per segment) 5 Rollout temperature / top-p / top-k 1.0 / 1.0 / disabled Maximum prompt length (tokens) 12,288 Maximum response length (tokens) 512 Optimization Learning rate 1×10−61× 10^-6 Weight decay 0.01 Learning-rate warmup none Gradient-norm clip 1.0 Objective Policy clipping ratio ϵp _p (symmetric) 0.2 Advantage clip κ 3.0 KL penalty β (loss term) 0.01 Entropy coefficient 0 Schedule Segments per optimizer step 32 Mini-batch size 16 Inner epochs per step 1 Training epochs 3 B.3 Retrievability Trace Collection and Replay At the beginning of each training epoch, we build a memory bank with the current writer and run the frozen agentic reader on every training question associated with at least one verified evidence segment. For each question, we cache the search queries, time intervals, retrieval budgets, and fetched intervals issued before the final answer. These traces are reused to score all candidate memories sampled during that epoch. To evaluate a candidate y for segment c, we replace only the corresponding entry in the memory bank and replay every cached action associated with questions in (c)Q(c). For a search action, we recompute the candidate’s similarity and rank it against all entries eligible under that action’s causal and temporal constraints. The action returns y only when its rank falls within the recorded retrieval budget. For a fetch action, y is returned when the requested interval contains c. This replay determines membership in q(y)C_q(y) without executing a new agent reasoning trajectory. The memory bank and traces are rebuilt after each epoch. B.4 Token-Level Group-Relative Objective Equations 11 and 12 define the token advantages used for optimization. We clip each A~i,t A_i,t to [−κ,κ][-κ,κ], then recenter and whiten the values over generated response tokens while excluding prompt and padding positions. We denote the resulting advantage by Ai,tA_i,t. Let πθold _ _old denote the policy that sampled the current candidates. Its token probability ratio with the updated writer is ρi,t(θ)=πθ(yi,t∣c,yi,<t)πθold(yi,t∣c,yi,<t). _i,t(θ)= _θ(y_i,t c,y_i,<t) _ _old(y_i,t c,y_i,<t). (15) The MemOpt policy objective is ℒMemOpt(θ)=−c,i,t[ _ MemOpt(θ)=-E_c,i,t [ min(ρi,t(θ)Ai,t,clip(ρi,t(θ),1−ϵp,1+ϵp)Ai,t)−βD^KLi,t], \! ( _i,t(θ)A_i,t,clip( _i,t(θ),1- _p,1+ _p)A_i,t )-β D^KL_i,t ], (16) where ϵp _p is the policy clipping ratio, D^i,tKL D^KL_i,t is the per-token Kullback–Leibler estimate between πθ _θ and the frozen initial writer πref _ref, and β controls its strength. Table 9 lists the numerical settings. Appendix C Controlled Comparison with Trained Baselines The main comparison uses official checkpoints when available, preserving the systems released by their authors but leaving differences in model scale and training data. We therefore conduct an additional controlled comparison by reproducing VST and TaskMem with Qwen3.5-9B and training them only on the SuperMemory-VQA training split used by MemOpt. Their results consequently differ from the released-checkpoint results in Table 1. We repeat the M3-Agent results from that table because its existing reproduction already uses the same backbone and training split. Table 10: Controlled comparison with trained baselines. All methods use Qwen3.5-9B and task-specific training data only from SuperMemory-VQA. EgoLifeQA is evaluated out of distribution, and bold marks the best result. Method SuperMemory-VQA EgoLifeQA Accuracy Recall Accuracy Recall VST 56.50 65.08 29.40 7.60 TaskMem 48.96 51.15 48.20 35.00 M3-Agent 53.93 54.20 35.40 9.60 MemLife + MemOpt 60.35 81.49 56.60 50.60 Table 10 shows that MemLife with MemOpt achieves the highest accuracy and recall on both the in-domain SuperMemory-VQA test set and out-of-domain EgoLifeQA. None of these controlled runs uses EgoLifeQA for task-specific training. Moreover, MemOpt updates only the memory writer, whereas VST and M3-Agent also adapt their answering or control components. Together with the primary comparison in Table 1, these results indicate that the gains from MemOpt are not explained by backbone scale or differences in task-specific training data. Appendix D Stability Analysis In the main experiments, we use deterministic decoding across all methods for fairness and reproducibility. To assess stability under stochastic decoding, we repeat inference five times at temperature 1.0 for MemLife, MemLife-V, and EgoRAG, the competing system with the highest average accuracy. Table 11 compares the greedy results from the main evaluation with the mean and standard deviation over five sampled runs. Table 11: Stability across reader decoding settings. T=0.0T=0.0 reports greedy decoding, while T=1.0T=1.0 reports mean ± standard deviation over five runs. Method SuperMemory-VQA EgoLifeQA Accuracy Recall Accuracy Recall T=0.0T=0.0 T=1.0T=1.0 T=0.0T=0.0 T=1.0T=1.0 T=0.0T=0.0 T=1.0T=1.0 T=0.0T=0.0 T=1.0T=1.0 EgoRAG 49.28 49.18 ± 0.95 66.79 66.79 ± 0.00 48.20 46.48 ± 0.95 29.60 28.80 ± 0.00 MemLife 56.50 55.31 ± 1.53 80.53 80.69 ± 0.32 52.80 50.16 ± 1.83 48.40 47.48 ± 1.36 MemLife-V 57.78 58.78 ± 0.41 84.35 84.58 ± 0.64 53.20 52.44 ± 1.72 50.20 49.72 ± 0.30 Method SuperMemory-LVQA EgoLife-EQA Accuracy Recall Accuracy Recall T=0.0T=0.0 T=1.0T=1.0 T=0.0T=0.0 T=1.0T=1.0 T=0.0T=0.0 T=1.0T=1.0 T=0.0T=0.0 T=1.0T=1.0 EgoRAG 47.51 48.02 ± 0.43 47.90 48.85 ± 0.00 38.00 36.20 ± 2.39 13.00 14.00 ± 0.00 MemLife 56.18 55.31 ± 0.91 46.18 45.00 ± 0.97 52.00 50.80 ± 4.66 44.00 42.20 ± 2.95 MemLife-V 57.95 58.17 ± 1.24 48.47 48.51 ± 1.93 50.00 53.20 ± 3.27 45.00 45.00 ± 0.71 The performance ordering remains stable under sampled decoding. MemLife and MemLife-V outperform EgoRAG in accuracy on every benchmark and generally provide higher recall. The gains remain consistent across runs, and the accuracy improvements of MemLife over EgoRAG are significant on all four benchmarks (p<0.05p<0.05). The recall of EgoRAG has a zero standard deviation, since it has a fixed retrieval process instead of using agentic search as MemLife. Appendix E Efficiency Analysis Table 12 compares storage and question-time costs across various systems, which are the key efficiency concerns in runtime question answering. Storage is measured over matched source histories and normalized by video duration, while reading time excludes model initialization. Table 12: Storage and reading efficiency of training-free video memory systems. Storage is normalized by source-video duration, and reading time is averaged per question. Dashes indicate methods without a persistent memory store. Method Memory Storage (MB/h) Reading Speed (s/question) SuperMemory-VQA EgoLifeQA SuperMemory-VQA EgoLifeQA Video ReCap 3.86 3.66 20.4 54.2 EgoRAG 4.98 5.49 33.2 43.5 Video-RAG 1.03 3.44 60.9 76.9 VideoARM – – 47.5 110.4 EGAgent 21.50 22.20 200.3 230.7 WorldMM 2.09 3.88 44.7 57.1 MemLife 0.68 0.67 13.1 20.7 MemLife has the smallest footprint among systems with persistent storage, which has nearly unchanged storage per video hour across the two benchmarks. It also provides the fastest reading time. Its informative text episodes and targeted retrieval typically expose sufficient evidence within a few agent rounds, while chronological ordering reduces the subsequent reasoning burden. The default reader also avoids processing source-video tokens at question time. Together, these choices limit unnecessary generation and multimodal processing, helping explain the lower latency relative to baselines with different execution patterns. Appendix F Ablation of Retrievability Approximation Computing retrievability requires specifying how a future question will access a candidate memory. We compare two approximations. The simplified variant uses the original question directly as a fixed semantic-search query and scores whether the candidate is retrieved. It therefore requires neither a reader policy nor a reader rollout during writer training. Standard MemOpt instead runs the MemLife agentic reader at the beginning of each epoch and collects its memory-access actions. These actions reflect the agent’s reformulated queries and time-scoped retrieval decisions, but remain fixed within the epoch to avoid candidate-dependent variation. After writer training, we evaluate both variants with the standard MemLife agentic reader, which performs multi-round memory access and chronologically orders the retrieved entries before answering. Table 13: Effect of the training-time retrievability approximation on answer accuracy (%). Training-time proxy SuperMemory-VQA EgoLifeQA None (zero-shot) 56.50 52.80 Original question 57.62 55.40 Agent actions 60.35 56.60 Both approximations outperform the zero-shot writer on both benchmarks. The original-question proxy already provides useful supervision without requiring a training-time reader rollout. Agent actions perform best on both datasets, indicating that reformulated queries and time-scoped accesses expose retrieval failures that the original question alone may miss. We therefore use agent actions in the standard MemOpt configuration, while retaining the original-question proxy as a simpler, reader-policy-independent alternative. Appendix G Reward Dynamics during MemOpt Training To examine how the three reward signals evolve during optimization, we record their scores over three epochs, comprising 135 training steps. Retrievability and informativeness are averaged across the sampled candidate memories. Since faithfulness is evaluated at the token level, we first take the minimum token score within each candidate and then average these minima across candidates. This conservative aggregation prevents a few unsupported tokens from being obscured by many well-supported ones. In Figure 5, the light curves show the resulting per-step scores, while the bold curves show their smoothed trends. Figure 5: Dynamics of the three reward signals during MemOpt training. Following an initial fluctuation, retrievability improves mainly during the early stages of training and stabilizes sooner, although its per-step values remain noisy because they depend on each candidate’s position within the evolving memory store and the cached reader actions. Informativeness and faithfulness improve for longer, indicating that the writer continues to preserve more answer-relevant evidence and produce better-grounded memories. Together, these trends suggest that MemOpt improves all three dimensions without a sustained trade-off in retrievability. Appendix H Discussion on Memory Reader Training Although MemOpt focuses on the writer, the agentic reader can also be trained to improve memory access and answer generation. We compare training neither component, the writer alone, the reader alone, and both components. For reader reinforcement learning, the reward is (accuracy+recall)/2(accuracy+recall)/2. To reduce training cost, we cap trajectories at six interaction rounds, compared with ten rounds at evaluation. A trajectory containing a formatting error or exhausting this budget will receive a reward of −1-1. Table 14: Effects of writer and reader training. All values are percentages. Writer trained Reader trained SuperMemory-VQA EgoLifeQA Accuracy Recall Accuracy Recall ✗ ✗ 56.50 80.53 52.80 48.40 ✔ ✗ 60.35 81.49 56.60 50.60 ✗ ✔ 61.00 89.50 51.40 59.00 ✔ ✔ 63.08 87.98 54.20 56.80 Reader training substantially improves both accuracy and recall on the in-domain SuperMemory-VQA benchmark. Adding writer training to the trained reader further improves accuracy with only a small decrease in recall, indicating that the optimized memories preserve more answer-useful information rather than merely maximizing evidence retrieval. On the out-of-domain EgoLifeQA benchmark, reader training still improves recall but does not consistently improve answer accuracy, whereas writer-only training achieves the highest accuracy. This contrast suggests that reader policies more readily specialize to the training question and source-video distributions, as well as the resulting interaction trajectories. Writer training instead improves the persistent memory itself, allowing different readers and evaluation settings to benefit from the same higher-quality memory or writing policy. We therefore center MemOpt on writer optimization. Reader training provides complementary in-domain gains, while making these gains consistent under distribution shift remains future work. Appendix I Predecessor Context for Memory Writing MemLife writes each video segment independently. To examine whether continuity across adjacent segments benefits memory construction, we compare this design with two alternatives that condition the writer on the immediately preceding segment. The text variant provides both the previous transcript and memory entry, while the multimodal variant additionally provides the previous frames. The independent variant (None) omits the predecessor block and uses the multimodal information from the corresponding segment as the only context. Table 15: Answer accuracy across SuperMemory-VQA question types under different forms of predecessor context for memory writing. Bold values indicate the best result in each column. Predecessor Context Conversational Memory Intent Recall In-Context Retrieval Timeline Reconstruction Object-Location Memory Visual Recall Overall None 63.41 77.42 43.18 57.41 52.29 44.12 56.50 Text 62.60 86.02 37.50 48.15 45.87 45.10 54.25 Multimodal 69.92 82.80 37.50 47.22 46.79 43.14 54.90 Table 15 shows that predecessor context produces mixed effects. Text context improves intent and visual recall, while multimodal context improves conversational and intent recall. Both variants reduce performance on in-context retrieval, timeline reconstruction, and object-location memory, and neither improves overall accuracy. We therefore retain independent segment writing, which also avoids sequential dependencies and error propagation during memory construction. Appendix J Memory from Text and Multimodal Inputs Text-input agent memory systems maintain persistent information from language-based histories, such as conversations, documents, or natural-language observations. They determine what information to store and how to update, organize, and retrieve it for later use. Existing systems store observations and synthesize reflections (Park et al., 2023), manage tiered memory stores (Packer et al., 2024; Kang et al., 2025), or maintain and consolidate information from conversations (Zhong et al., 2024; Chhikara et al., 2025). Other architectures organize histories through segmentation, compression, or dynamically linked and graph-structured memories (Pan et al., 2025; Xu et al., 2025; Wang et al., 2026a). Beyond architectural design, recent methods learn what to retain or how to manage memory. They optimize memory operations or construction from downstream feedback (Yan et al., 2026; Wang et al., 2025b), retain source evidence under a fixed budget (Li et al., 2026), or provide denser supervision for different stages and components of memory construction and use (Shen et al., 2026; Zhang et al., 2026). Agent Memory Distillation follows a different, training-free strategy. It constructs hierarchical procedural memories from successful trajectories generated by a stronger teacher agent (Kim et al., 2026). In these settings, the source experience has already been expressed as text. Long-term egocentric video introduces an additional challenge before memory management can begin. The writer must identify entities, actions, and speech from raw visual and audio observations, preserve their temporal context, and avoid introducing details unsupported by the source. Once evidence is omitted or incorrectly described at this stage, a downstream text-memory architecture cannot recover it without revisiting the video. Text-input memory systems could therefore complement MemLife by organizing the episodic descriptions after they are written, rather than replacing our system as video-to-text writers. Appendix K Theoretical Foundation of FIRM Let Q and A denote a question and its answer, V the available video history, M the memory written from that history, and C the retrieved context. Let H(⋅∣⋅)H(· ·) and I(⋅;⋅∣⋅)I(·;· ·) denote conditional entropy and conditional mutual information. For the memory-only reader, their joint distribution factorizes as p(V,Q,A,M,C)=p(V,Q,A)pθ(M∣V)p(C∣M,Q),p(V,Q,A,M,C)=p(V,Q,A)p_θ(M V)p(C M,Q), where θ denotes the writer parameters. This restates Equation 5. Because the writer constructs M only from V and the reader constructs C only from M and Q, I(A;M∣V,Q)=0,I(A;C∣M,Q)=0.I(A;M V,Q)=0, I(A;C M,Q)=0. The memory-quality decomposition then follows as H(A∣C,Q)−H(A∣V,Q) H(A C,Q)-H(A V,Q) = = I(A;V∣Q)−I(A;C∣Q) I(A;V Q)-I(A;C Q) = = [I(A;V∣Q)−I(A;M∣Q)]+[I(A;M∣Q)−I(A;C∣Q)] [I(A;V Q)-I(A;M Q) ]+ [I(A;M Q)-I(A;C Q) ] = = [I(A;V∣M,Q)−I(A;M∣V,Q)]+[I(A;M∣C,Q)−I(A;C∣M,Q)] [I(A;V M,Q)-I(A;M V,Q) ]+ [I(A;M C,Q)-I(A;C M,Q) ] = = I(A;V∣M,Q)+I(A;M∣C,Q). I(A;V M,Q)+I(A;M C,Q). (17) Applying the chain rule to the two bracketed differences and substituting the conditional independences above yields the final line. Its two terms are the informativeness and retrievability gaps, respectively. To relate these gaps to downstream QA, let A^∼qϕ(⋅∣Q,C) A q_φ(· Q,C) denote the answerer’s prediction. The quantity qϕ(A∣Q,C)q_φ(A Q,C) is therefore the probability assigned to the ground-truth answer, while pθ(A∣Q,C)p_θ(A Q,C) is its conditional distribution under the joint process above. Taking expectations under pθ(Q,A,V,M,C)p_θ(Q,A,V,M,C), the negative log-likelihood decomposes as ℒNLL(θ,ϕ):= _NLL(θ,φ):= pθ[−logqϕ(A∣Q,C)] _p_θ [- q_φ(A Q,C) ] (18) = = H(A∣Q,V)⏟irreducible uncertainty+I(A;V∣Q,M)⏟informativeness gap+I(A;M∣Q,C)⏟retrievability gap H(A Q,V)_irreducible uncertainty+ I(A;V Q,M)_informativeness gap+ I(A;M Q,C)_retrievability gap +pθ(Q,C)KL(pθ(A∣Q,C)∥qϕ(A∣Q,C))⏟prediction mismatch(includes faithfulness effects). + E_p_θ(Q,C)\,KL\! (p_θ(A Q,C)\,\|\,q_φ(A Q,C) )_ subarraycprediction mismatch\\ (includes faithfulness effects) subarray. Proof. For fixed (Q,C)(Q,C), averaging −logqϕ(A∣Q,C)- q_φ(A Q,C) over A∼pθ(A∣Q,C)A p_θ(A Q,C) gives their cross-entropy. This equals H(A∣Q,C)H(A Q,C) plus the KL term in Equation 18. Equation 17 expands the conditional entropy as H(A∣Q,V)+I(A;V∣Q,M)+I(A;M∣Q,C)H(A Q,V)+I(A;V Q,M)+I(A;M Q,C), yielding the result. ■ The final term is affected by both the memory and the answerer. Unsupported content in C can shift qϕq_φ away from pθp_θ and mislead the answerer, while reasoning errors can produce the same mismatch even when C is fully grounded. It therefore mixes a writer-controlled faithfulness effect with answerer-dependent noise and cannot directly supervise faithfulness. This explains why final-answer feedback gives noisy credit to the writer. Together, these results motivate the three FIRM signals. Informativeness targets answer-relevant evidence lost during writing, and retrievability targets stored evidence missed during memory access. Faithfulness directly compares the memory with its source, isolating hallucination-related noise without absorbing answerer reasoning error. For a rigorous assessment of memory quality, MemOpt uses the default MemLife reader during training and disables direct access to the source videos. This prevents the reader from bypassing the written memory when collecting answer-relevant information. Appendix L Prompt Templates L.1 Writer Prompt Figure 6 shows the complete system instruction and user template used by the MemLife writer. The prompt asks it to preserve visible and spoken evidence, ground named entities in visual observations, and narrate the resulting memory in the first person. Prompt for the MemLife memory writer System instruction You are writing a memory entry for one short segment of an egocentric video. This entry is the only record of this moment that a later question-answering system will see, so capture the concrete, factual details it might be asked about. Write in the FIRST PERSON as the person wearing the camera (the video is shot from their point of view): refer to yourself as ‘I’/‘my’/‘me’, not as ‘the camera wearer’, ‘the camera operator’, or ‘the person’. The evidence lives in TWO equally important channels, and you must use both: 1. What is SEEN — the people, their actions, the objects they handle (brand names, text on labels, colors), and the place. Refer to any OTHER person by name when they are a named speaker in the transcript (tags like [B] or [Shure]); add stable visual attributes (clothing, hair, glasses) as a complement to the name, and use visual attributes alone only when no name is available. 2. What is SAID or HEARD — who speaks and the substance of what they say; paraphrase salient statements, questions, decisions, plans, names, numbers, times, and places mentioned in the transcript. Treat the spoken transcript as a primary source of fact, not background noise: if something is only stated aloud, it still belongs in the memory. Write ONE paragraph of plain prose, up to ∼ 450 tokens; finish every sentence and do not truncate. No JSON, no bullets, no headers. Describe only this segment; do not speculate beyond the evidence. Write a COMPLETE, SELF-CONTAINED first-person paragraph describing what happens in this segment (actions, objects, places, people, state changes). User template [eight frames sampled from the segment, 704 px] What is said in this segment: """transcript""" Write the memory entry, grounding it in the frames and transcript above. Figure 6: Complete system instruction and user template for the MemLife memory writer. L.2 Reader System Prompts Figures 7 and 8 provide the agentic-reader prompts used in our experiments. The reader emits either one tool call or a final answer per round. Outputs that match neither format are treated as parse failures and scored as incorrect. All requested intervals are clipped to the question time, preventing access to future segments. System prompt for the default MemLife reader You are a memory reader. You answer multiple-choice questions about an egocentric video by calling tools to inspect a pre-built memory. You have two tools: 1. search_memory(query: str, time_anchor: [t_start, t_end] = [0, <now>], top_k: int = 32): retrieve the memory entries most relevant to the query, restricted to clips within [t_start, t_end]. Returns the top_k most similar clips in time order — each with its description and time range. Omit time_anchor to search all memory up to the current time; pass a narrower [t_start, t_end] to focus on a period (e.g. a specific day). 2. fetch_memory( time_anchor: [t_start, t_end]): return every memory clip within [t_start, t_end] in time order, each with its description. Each turn, respond with EXACTLY ONE of these two lines: TOOL: "name": "<tool>", "args": ... ANSWER: <letter> Emit no other text on the response line. End your final round with `ANSWER: <letter>` once you have enough evidence. Examples: Q: "How often do I cook dinner?" -> TOOL: "name": "search_memory", "args": "query": "cooking dinner" Q: "What did I buy yesterday?" -> TOOL: "name": "search_memory", "args": "query": "buying purchase shopping", "time_anchor": [yesterday_start, yesterday_end] Q: "List everything I bought today." -> TOOL: "name": "fetch_memory", "args": "time_anchor": [t_start, t_end] Figure 7: System prompt for the default MemLife reader. Additional source-video tool for MemLife-V 3. fetch_video( time_anchor: [t_start, t_end], num_frames: int = 30): sample num_frames frames and the transcript from the video within [t_start, t_end]. Q: "What was on my desk when I left?" -> TOOL: "name": "fetch_video", "args": "time_anchor": [t_start, t_end], "num_frames": 30 Figure 8: Additional prompt block for MemLife-V. The tool count is updated to three, and all other text follows Figure 7. L.3 Evaluator Prompts Figures 9–11 provide the exact templates for grounded key-fact extraction, informativeness evaluation, and faithfulness evaluation. The key-fact extractor and faithfulness evaluator receive eight frames and the transcript of the current 30-second segment, whereas the entailment judge receives only the key fact and candidate memory. Prompt template for grounded key-fact extraction System instruction You are auditing ONE 30-second segment of a first-person video against a question about the wearer’s day and that question’s known-correct answer. The segment has been labelled as evidence for the question. State, in one sentence, the KEY FACT that is actually visible or audible in THIS segment and that helps establish the correct answer. Rules: 1. The key fact must be something you can SEE in this segment’s frames or HEAR in its transcript. Never infer it from the question or from the answer. 2. Write it as a standalone declarative sentence about the segment, naming the concrete objects, people, places, actions or spoken words involved. Do NOT mention the question, the options, or the answer, and do not reuse an option’s wording. 3. If this segment does NOT actually contain anything supporting the correct answer - the evidence label is wrong, or the segment is unrelated background - say so instead of inventing a link. 4. A segment can be genuine evidence while showing only PART of what the answer needs; in that case state the part it does show. Output EXACTLY two lines and nothing else: SUPPORTS: YES KEY FACT: <one sentence> or SUPPORTS: NO KEY FACT: NONE User template ## This segment eight frames sampled from the segment What is said in this segment: """transcript""" ## Question a later reader will ask about this day "question" Options: options Correct answer: letter. correct_answer What key fact does THIS segment contribute toward that correct answer? Output the two lines. Figure 9: Prompt template for grounded key-fact extraction. Prompt template for informativeness evaluation System instruction You are a strict entailment judge. You are given a TARGET FACT about one 30-second segment of a first-person video, and a MEMORY that was written for that same segment. Decide whether the memory states the target fact, or states something from which the target fact follows directly. A paraphrase counts. A vaguer statement that omits the specific detail in the target fact does not. Use only these two texts - no outside knowledge and no guessing. Answer with exactly one word: Yes or No. User template ## TARGET FACT "key_fact" ## MEMORY "candidate_memory" Does the memory state or directly entail the target fact? Answer Yes or No. Answer: Figure 10: Prompt template for informativeness evaluation. Prompt template for faithfulness evaluation System instruction You are correcting a written memory of one 30-second segment of a first-person video. You are given the segment’s frames and transcript, and ONE memory of that segment. A claim is unsupported if it asserts an object, person, place, action, quantity, or spoken content that is not actually visible in the segment’s frames or audible in its transcript. A memory that is vague, brief or incomplete is still supported. If every claim in the memory is supported, repeat the memory back EXACTLY as given, character for character. If some claim is not supported, output the memory with those claims removed, or weakened to what the segment actually shows, changing as few words as possible. Do not add detail and do not improve the writing. Output only the memory text. No preamble, no explanation, no quotation marks. User template ## The segment eight frames sampled from the segment What is said in this segment: """transcript""" ## Memory of this segment "candidate_memory" Output this memory with any unsupported claims corrected, or unchanged if every claim is supported. Memory: Figure 11: Prompt template for faithfulness evaluation.