Paper deep dive
HiMe: Hierarchical Embodied Memory for Long-Horizon Vision-Language-Action Control
Li Ji, Siyin Wang, Pengfang Qian, Xiaopeng Yu, Yihai Tian, Zhaoye Fei, Jingjing Gong, Xipeng Qiu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 7/7/2026, 8:08:14 AM
Summary
The paper introduces HiMe, a hierarchical embodied memory framework designed to resolve the frequency-competence paradox in Vision-Language-Action (VLA) models for long-horizon robotic manipulation. HiMe decouples intelligence into a high-frequency Executor for transient control, a Sentry for working memory monitoring, and a Planner for long-term episodic reasoning. It incorporates a dynamic, cross-modal knowledge system with active memory management (Add, Update, Delete) to maintain memory plasticity and self-correct based on human preferences, significantly outperforming flat memory baselines in complex, multi-step tasks.
Entities (12)
Relation Signals (10)
HiMe → consistsof → Executor
confidence 97% · HiMe... decouples embodied intelligence into a high-frequency Executor for execution, a Sentry for working memory, and a Planner for long-term strategy.
HiMe → consistsof → Sentry
confidence 97% · HiMe... decouples embodied intelligence into a high-frequency Executor for execution, a Sentry for working memory, and a Planner for long-term strategy.
HiMe → consistsof → Planner
confidence 97% · HiMe... decouples embodied intelligence into a high-frequency Executor for execution, a Sentry for working memory, and a Planner for long-term strategy.
HiMe → addresses → Frequency-competence paradox
confidence 95% · To resolve this architectural misalignment, we propose HiMe, a Hierarchical Embodied Memory framework that decouples embodied intelligence... effectively resolving the frequency-competence paradox.
Sentry → manages → Working Memory
confidence 94% · The Sentry acts as the guardian of working memory (short-term memory); it asynchronously filters the continuous sensory stream to identify critical state transitions.
Executor → manages → Transient Memory
confidence 94% · The Executor (VLA) governs transient memory, focusing on high-frequency sensory-motor coordination for physical stability.
Planner → manages → Episodic Memory
confidence 94% · The Planner manages episodic memory (long-term memory).
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Current Vision-Language-Action (VLA) models excel at robotic manipulation but often struggle with non-Markovian tasks requiring long-term memory and reasoning due to their reliance on immediate observations. Existing solutions face a ''frequency-competence paradox,'' where stronger reasoning models are too slow for real-time control, while faster models lack sufficient reasoning capabilities. To resolve this architectural misalignment, we propose HiMe, a Hierarchical Embodied Memory framework that decouples embodied intelligence into a high-frequency Executor for execution, a Sentry for working memory, and a Planner for long-term strategy. We also introduce a dynamic knowledge system based on cross-modal semantic schemas and active management mechanisms, allowing robots to maintain memory plasticity through ''Add, Update, and Delete'' operations. This hierarchical design effectively balances the conflict between real-time execution and slow thinking planning, significantly improving success rates in long-horizon tasks. Experiments demonstrate that this approach not only outperforms flat memory baselines but also exhibits the novel ability to self-correct its internal knowledge based on human preferences.
Tags
Links
- Source: https://arxiv.org/abs/2607.03449v1
- Canonical: https://arxiv.org/abs/2607.03449v1
Trouble viewing inline? Open PDF directly →
Full Text
68,779 characters extracted from source content.
Expand or collapse full text
HiMe: Hierarchical Embodied Memory for Long-Horizon Vision-Language-Action Control Li Ji 1,2,∗ Siyin Wang 1,2,∗,‡ Pengfang Qian 1,2 Xiaopeng Yu 1 Yihai Tian 2,3 Zhaoye Fei 1 Jingjing Gong 2,† Xipeng Qiu 1,2,† 1 Fudan University 2 Shanghai Innovation Institute 3 East China Normal University ∗ Equal Contribution ‡ Project Lead † Corresponding Author 24210240184@m.fudan.edu.cn, siyinwang20@fudan.edu.cn Abstract Current Vision-Language-Action (VLA) models excel at robotic manipulation but often struggle with non-Markovian tasks requiring long-term memory and reasoning due to their reliance on immediate observations. Existing solutions face a “frequency-competence paradox,” where stronger reasoning models are too slow for real-time control, while faster models lack sufficient reasoning capabilities. To resolve this architectural misalignment, we proposeHiMe, a Hierarchical Embodied Memory framework that decouples embodied intelligence into a high-frequency Executor for execution, a Sentry for working memory, and a Planner for long-term strategy. We also introduce a dynamic knowledge system based on cross-modal semantic schemas and active management mechanisms, allowing robots to maintain memory plasticity through “Add, Update, and Delete” operations. This hierarchical design effectively balances the conflict between real-time execution and slow thinking planning, significantly improving success rates in long-horizon tasks. Experiments demonstrate that this approach not only outperforms flat memory baselines but also exhibits the novel ability to self-correct its internal knowledge based on human preferences. Code:https://github.com/HappyWaterXP/HiMe 1Introduction Vision-Language-Action (VLA) models have emerged as a powerful paradigm for general-purpose robotic manipulation with large-scale internet-level pretraining [1,2]. By directly mapping observations to control signals, they provide a powerful foundation for executing complex motor skills. Most existing architectures rely on the Markov assumption, where the policy푝(푎 푡 |표 푡 ,푙)predicts the action푎 푡 at time step푡conditioned only on the transient observation표 푡 at the current time step and the language instruction푙. This inherent limitation prevents them from maintaining a persistent belief of the environment in non-Markovian settings. Consequently, VLAs struggle with complex, long-horizon tasks that demand long-range temporal depen- dencies, where the optimal action depends on a chain of past events or latent information that is no longer visible in the current sensory stream. 1 arXiv:2607.03449v1 [cs.RO] 3 Jul 2026 Pick Bin 1 ... Find all Alice‘s toys Text memory: alice’s toys is duck Vision memory: bin 1 is empty Working memory: Subtask 1 done Vision memory： bin 2 is empty Pick up in bin 1 Vision memory： bin 1 has duck Update Pick up in bin 1 Episode Memory Planner Sentry Executor EPISODE MEM (Multimodal) WORKING MEM TRANSIENT MEM Large VLM: Slow Think Small VLM: Fast Think VLA Update Add Delete retrieve HiMe: Hierarchical Embodied Memory Working memory： Subtask 2 done Figure 1 Overview of HiMe.(a) A motivating example illustrating the need to maintain and update task-relevant mul- timodal memory across long-horizon subtasks. (b) The HiMe framework, which addresses this challenge by organizing embodied intelligence into a hierarchical structure that separates fast execution from memory-driven reasoning over long time scales. To overcome these limitations, one intuitive approach is to imbue VLA models with native memory capabil- ities through specialized training or auxiliary losses [3,4]. However, these training-based methods are often constrained by limited context windows and the inherent difficulty of optimizing long-range causal depen- dencies over hundreds of steps. Another alternative is to introduce Large Vision-Language Models (LVLMs) as high-level memory containers to store and retrieve historical trajectories [ 5]. Yet, real-time robotic control imposes a strict upper bound on the deployable VLM scale, as control demands high-frequency, low-latency execution. This scale constraint, in turn, limits the internal world knowledge and generalization capabilities of the VLM, weakening its zero-shot performance. Importantly, most memory operations do not intrinsically require high-level reasoning or broad generalization. As illustrated in Fig. 1, motivated by this temporal and scale mismatch, we introduceHiMe, aHierarchical Embodied Memoryframework that decouples embodied intelligence into three functional layers with dis- tinct temporal resolutions, mirroring the multi-store structure of human cognition. (1) TheExecutor (VLA) governstransient memory, focusing on high-frequency sensory-motor coordination for physical stability. (2) TheSentryacts as the guardian ofworking memory(short-term memory); it asynchronously filters the continuous sensory stream to identify critical state transitions, effectively bridging the gap between raw per- ception and semantic events. (3) ThePlannermanagesepisodic memory(long-term memory). By invoking the “slow-thinking” Planner only at the discrete junctions identified by the Sentry, our architecture pre- serves deep strategic reasoning while maintaining real-time execution, effectively resolving the frequency- capability paradox. However, a static memory hierarchy alone is insufficient for the complexities of real-world human-robot interaction, which is characterized by: (1)multimodal richness, where human instructions carry dense logi- cal constraints and latent preferences [6], such as designating a specific red cup as “Alice’s cup”, that are nearly impossible to capture through vision-only trajectories; and (2)high dynamism, where task goals and environment states are not static but evolve. A memory system that only supports passive accumulation [5] inevitably suffers from knowledge stagnation and cognitive dissonance when faced with such outdated or conflicting experiences. Furthermore, our Hierarchical Memory framework treats memory as a multimodal, dynamic knowledge system centered around two fundamental dimensions: (1) What to memorize?: Cross-modal Semantic Schemata.To bridge the gap between raw perception and logi- cal intent, we represent episodic memory as an object-centric system where visual experiences are deeply anchored with high-density textual descriptions. This organization allows the Planner to retrieve not only vi- 2 sual features but also the underlying ownerships, procedural rules, and human preferences that are invisible in pure pixel space. (2) How to memorize?: Active Management Mechanism.In contrast to passive storage, we introduce explicitAdd, Update, andDeleteoperations to grant the robot knowledge plasticity. When the Sentry detects a significant state transition or a revision in user intent, the Planner proactively refines the knowledge base. By updating outdated beliefs and purging redundant or erroneous data, the agent maintains a consistent and concise memory that evolves in alignment with the environment. We evaluate our method across a suite of challenging long-horizon manipulation tasks that require multi- step reasoning and adaptation to shifting human preferences. Experimental results show that our approach significantly outperforms existing flat-memory baselines in both success rate and computational efficiency. Notably, our agent demonstrates the ability to “self-correct” its internal knowledge base when faced with conflicting instructions, a capability that is absent in prior retrieval-based methods. Our core contributions are summarized as follows: •We propose aHierarchical Memory Management frameworkthat decouples robotic control into tran- sient (Executor), working (Sentry), and episodic (Planner) memory layers, resolving the granularity conflict in long-horizon VLA tasks. •We introduce a Cross-modal Memory organization (what to memorize) combined with an Active Man- agement mechanism (how to memorize), enabling the robot to maintain knowledge plasticity and align- ment in highly dynamic and multimodal interactions. •We provideextensive empirical evidencedemonstrating that our hierarchical, self-evolving memory architecture achieves superior performance and robustness in complex human-robot collaborative sce- narios with significantly reduced VLM computational overhead. 2Related Work 2.1Foundation models in robotics Recent advancements in End-to-End Vision-Language-Action (VLA) models adopt a co-training paradigm that integrates a pre-trained VLM backbone with a dedicated action expert [2]. By harnessing the VLM’s extensiveworldknowledge[7]tointerpretcomplexscenes, thisapproacheffectivelytransformsthebackbone into a generalist agent capable of mapping raw observations directly to continuous actions. Alternatively, the hierarchical paradigm decouples reasoning from execution [8–10]. In this framework, a high-level policy, typically a VLM, processes visual observations and task descriptions to generate intermediate goals, such as natural language sub-tasks or latent embeddings. These intermediate representations then condition a low- level policy to output continuous actions. Building upon this hierarchical structure, our work introduces a novel hierarchical memory architecture designed to empower the high-level VLM to generate more precise and context-aware instructions for the low-level controller. 2.2Robotic memory Memory is essential for robotic agents operating in long-horizon, partially observed manipulation tasks, where successful execution depends on retaining past observations, inferred world states, and user-specific context. Prior work has explored different mechanisms for incorporating historical information into robotic control. One major direction introduces memory at the policy level, primarily to improve local action gen- eration, spatial grounding, and short-range temporal consistency. These approaches either augment Vision- Language-Action models with retrieval-style memory banks that store and retrieve relevant past context [ 4, 11–13], orencodehistoryintocompactrepresentationssuchaspasttokensorvisualtracesforhistory-conditioned action prediction [3,14]. 3 Executor Subtask Image Executor Action ... ReasoningQuery Retrieved memory Memory editing Planning Buffer Procedural MemoryContextual Memory Alice likes ... Subtask Transient Memory ... ... Uniformly Sampled 8 Frames Sentry Recent Frames Sentry Done/Active Working Memory Planner (1) (2) (3) Add Update Delete Toys should be placed ... 1. [DONE] ... 2. [DONE] ... 3. [ACTIVE] ... 4. [PENDING] ... 1. [DONE] ... 2. [DONE] ... 3. [ACTIVE] ... 4. [PENDING] ... 1. [DONE] ... 2. [DONE] ... 3. [ACTIVE] ... 4. [PENDING] ... Episodic Memory Figure 2 Architecture of HiMe.The Sentry module serves as a real-time monitor, periodically tracking the Executor’s progress and buffering visual observations. Upon detecting the completion of a subtask, it triggers the Planner to review the entire execution trajectory. The Planner then performs memory retrieval and consolidation to update the contextual memoryandrefinestheprocedurememory. Finally, therefinedinstructionisfedbacktotheSentrytoresumemonitoring the Executor, forming a robust closed-loop system. Another line of work studies memory in more structured, higher-level decision processes. Hierarchical embodied-agent systems [ 15–17] highlight the importance of explicit planning, reflection, and multimodal memory for long-horizon reasoning. MemER [5] further organizes experience using a FIFO queue with keyframe selection, demonstrating that memory supports not only short-range control but also the mainte- nance of task-level context over extended horizons. Our work follows this direction and focuses on a multi- modal memory design with explicit memory management. 2.3Memory management in Agents Effective memory management is crucial for handling long-context interactions. Existing frameworks typ- ically organize memory using hierarchical structures [18] or OS-inspired interfaces [19], often relying on explicit management strategies [20–22] to maximize efficiency. A dominant strategy among these is peri- odic summarization [23,24], where recent history is compressed into long-term storage at fixed temporal intervals to decouple storage from reasoning. However, these methods have predominantly focused on text- based representations for LLMs [25]. With the rapid evolution of Vision-Language Models (VLMs), there is a growing necessity to incorporate multimodal information. Our approach addresses this by construct- ing a cross-modal memory that synergizes text and images, while simultaneously introducing a structured management mechanism tailored to effectively utilize this rich context for complex robotic tasks. 3Methods 3.1Overview Long-horizon manipulation relies on two contradictory capabilities: maintaining global consistency over ex- tended history and reacting responsively to immediate dynamics. End-to-end approaches typically struggle to balance these needs—short-context policies suffer from catastrophic forgetting, while heavy reasoning models incur prohibitive latency for real-time control. 4 Inspect recipe place the Job’s tears in box place the red beans in box place the Job’s tears into box place the mung beans in box inspect left box inspect middle box inspect right box place the toy duck in left box place the snack in right box Lucy likes plush toys. Inspect the boxes, collect the toy duck and snack into the box. Lucy will come to visit, prepare something she likes. place the toys on table place toy croissant in the box place toy mushroom in box place toy bread in box The toys are dirty, and the toy bread should be next to the toy croissant. Please help me put the toys into the box so I can wash them. I have washed the toys, please place them in correct position. place toy croissant on left place place toy bread on left place place toy mushroom on right place User Robot Observation User Robot Observation User Robot Observation Counting Object Search Rearra- ngement 1 w/o memory w/ memory Subtask w/o memory w/ memory Subtask w/o memory w/ memory Subtask 2 3 David prefers mung beans to red beans. Check the recipe, prepare ingredients for 2 batches of multigrain congee. David will come, help me prepare additional 1 batch, but switch the beans to David's preference. Text Memory Text Memory Text Memory Update Memory Human Add Obj. Figure 3Tasks used in our evaluations. We report performance across 20 trials per task per method. To address this challenge, we propose HiMe, a Hierarchical Embodied Memory framework (Fig. 2) that de- couples embodied intelligence into three functional layers: (1) The Executor휋 푒 : A high-frequency VLA that ensures physical stability by mapping immediate observations to actions. (2) The Sentry휋 푠 : A lightweight VLM that identifies progress changes in the environment, triggering planning only when necessary. (3) The Planner휋 푝 : A heavyweight VLM that uses long-term episodic memory and plans at a lower frequency. This separation amortizes the cost of complex planning over extended execution windows, ensuring the low-level controller remains responsive to immediate physical dynamics while maintaining long-term task coherence. Problem Setup.We formulate the long-horizon manipulation task as a sequential decision-making process under partial observability. The agent operates over a horizon푇≫1, receiving high-dimensional observa- tions표 푡 ∈ 풪and a natural language instruction푙. The objective is to generate an action sequence푎 0:푇 that transitions the environment to a state satisfying the instruction푙. Ideally, the optimal action at any step푡de- pends on the full interaction historyℎ 푡 =(표 0 ,푎 0 ,...,표 푡 )to resolve ambiguity and track progress. However, processingℎ 푡 continuously is computationally intractable. To address this, we adopt a hierarchical policy 휋 ( 푎 푡 |표 0:푡 ,푙 ) . 3.2Hierarchical Embodied Memory To bridge the gap between reactive control and reflective reasoning, we augment the agent with a structured state space comprising three distinct components. This explicitly represents the information required for 5 immediate control, temporal monitoring, and long-term planning. Transient Memory (풯 푡 ).Managed by the Executor, it contains the instantaneous observation표 푡 . It serves as the minimal sufficient state for reactive sensory-motor coordination. Working Memory (풲 푡 ).To support high-frequency monitoring, we maintain a sliding window of recent observations풲 푡 =표 푡−ℎ 푠 ,...,표 푡 . Unlike the main memory, this buffer is transient and raw, capturing the immediate dynamics required for verifying subgoal status or detecting sudden failures. Episodic Memory (ℰ 푡 ).A persistent long-term store managed by the Planner, consisting of: (1)Contex- tual Memoryℰ 푐 푡 , a multimodal key–value storeℳ 푡 that accumulates compact, task-relevant information— including both visual imagery and textual descriptions—over time, such as object state changes, spatial con- straints, or human preferences, and supports retrieval via textual keys. (2)Procedural Memoryℰ 푝 푡 , a struc- tured planℰ 푝 푡 =휏 푖 푁 푖=1 consisting of an ordered list of language subgoals, where the active subgoal휏 (푖) 푡 carries status labels like ‘active’, ‘pending’ or ‘done’ to synchronize the control loop. 3.3Hierarchical Policy Decomposition We decompose the hierarchical policy휋 ( 푎 푡 |표 0:푡 ,푙 ) into a three-tiered hierarchy. This design ranges from a memory-free execution layer to a long-horizon reasoning layer, progressively trading off frequency for context capacity, decoupling immediate actuation from high-level reasoning. Formally, the decision process at time푡involves interactions between a base Executor휋 푒 , a monitoring Sentry휋 푠 , and a reasoning Planner 휋 푝 . Base-level Executor (휋 푒 ).At the base level, the Executor functions as a high-frequency reactive policy. We assume a local Markov property where, given an active language subgoal휏 푡 ∈ ℰ 푝 푡 , the optimal action is conditionally independent of the long-term history. The Executor maps the current observation and subgoal to an action: 푎 푡 ∼휋 푒 ( 푎 푡 |표 푡 ,휏 푡 ) ,(1) where휏 푡 serves as a conditioning variable abstracted from the complex episodic history, representing the current executing subgoal. This design ensures that휋 푒 remains stateless and computationally efficient, en- abling real-time responsiveness to physical dynamics. Agile Sentry (휋 푠 ).Consistent with the architecture in Fig. 2, the Sentry acts as a gating function푢 푡 ∈0,1 that is used to amortize the computational cost of reasoning. Operating on the sliding window working memory풲 푡 , the Sentry evaluates subgoal completion or environmental deviation at intervals of푛 푚 : 푢 푡 = I [ 휋 푠 ( 풲 푡 ,휏 푡 ) >훿 ] ,if푡≡0(mod푛 푚 ), 0,otherwise, (2) whereI[·]is the indicator function and훿is a decision threshold.푢 푡 =1signals a “handover” event— indicating that the current subgoal휏 푡 is either completed or invalidated, thereby triggering the Planner. When푢 푡 =0, the system bypasses high-level reasoning, maintaining the current episodic memoryℰ 푡+1 ←ℰ 푡 . Reasoning Planner (휋 푝 ).When the Sentry trigger is activated (푢 푡 =1), the Planner is invoked to update the episodic memory and re-align the agent’s internal belief with the current environment. Let푙denote the user instruction,풲 푡 =표 푡−ℎ 푠 ,...,표 푡 the working memory containing recent observations, andℰ 푡 = ( ℰ 푐 푡 ,ℰ 푝 푡 ) the episodic memory maintained by the Planner, whereℰ 푐 푡 stores contextual information andℰ 푝 푡 represents the current procedural plan. This update proceeds in two stages. (1)Context Retrieval:The Planner first encodes the instruction and recent observations into a query, and 6 retrieves the most relevant contextual entries from memory: 푞 푡 =푓 enc ( 푙,풲 푡 ) ,ℳ ret =TopK ( ℰ 푐 푡 ,푞 푡 ) ,(3) whereℳ ret ⊂ℰ 푐 푡 denotes the retrieved subset of contextual memory. (2)Memory Update and Re-planning:Conditioned on the instruction푙, working memory풲 푡 , retrieved context ℳ ret , and the current procedural planℰ 푝 푡 , the Planner jointly updates both components of episodic memory: ( ℰ 푐 푡+1 ,ℰ 푝 푡+1 ) ←휋 푝 ( 푙,풲 푡 ,ℳ ret ,ℰ 푝 푡 ) .(4) Specifically, the update to contextual memoryℰ 푐 is realized through three explicit operations:Add, which inserts newly observed facts or user preferences;Update, which revises outdated entries; andDelete, which removes stale or conflicting information. In parallel, the Planner synthesizes an updated procedural plan ℰ 푝 푡+1 , from which the next active subgoal휏 푡+1 is selected to continue execution. 3.4Implementation Details While our framework is model-agnostic, we instantiate the Planner with GPT-4o [26] for its strong multi- modal reasoning capabilities, and the Sentry with Qwen3-VL-8B [7] for its balance of speed and grounding accuracy. The memory backend utilizes a vector database for semantic retrieval, utilizing OpenAI’s text- embedding-3 model to encode text queries and storing multimodal memory entries alongside their caption embeddings for cosine similarity retrieval. We set the buffer sizeℎ 푠 =8and the monitoring interval푛 푚 =5. Consistent with our design philosophy, only the lightweight휋 푒 , a fine-tuned VLA based on휋 0.5 , requires domain-specific training. The Sentry and Planner operate in a zero-shot or few-shot manner, leveraging pre-trained generalization to handle long-horizon logic without extensive data collection. 4Experiments 4.1Task Setting To demonstrate the critical role of hierarchical memory in long-horizon tasks, we designed three distinct tabletop manipulation scenarios, as illustrated in Fig. 3. These tasks are specifically curated to evaluate diverse capabilities, including user interaction, preference memory, updating memory and planning with exploration. Object Search.We design a Domestic Maintenance task to evaluate the robot’s capability to integrate in- spection, sorting, and preference recall. In this scenario, the robot must first inspect opaque boxes to deduce storage rules and organize scattered items accordingly. Crucially, during this process, the user introduces new toys into the boxes to test the robot’s ability to dynamically update its memory. Finally, the robot faces a retrieval challenge, where it must synthesize memorized user preferences with the box contents identified during inspection to retrieve the correct object. Counting.In this scenario, the robot is required to interpret a visual recipe on the table and plan its subse- quent actions. The task proceeds in stages: the robot must firstinspectthe recipe to extract the ingredient composition and prepare the materials based on the number of servings requested by the user. Following this, the user introduces a specific preference. To fulfill this customized demand, the robot must synthesize the new constraint with the previously memorized recipe to dispense the correct personalized ingredients. Rearrangement.In this playroom scenario, the robot performs a two-stage “clear and restore” operation. Initially, the robot collects all toys scattered on the table into a storage box for cleaning. After a temporal interval, the robot is tasked with restoring the items to the environment. Crucially, the user’s specific place- ment preference is provided beforehand as prior knowledge. Therefore, to execute the restoration, the robot must recall this pre-established preference and synthesize it with its visual memory of the original spatial configuration to plan the final placement of each item. 7 Evaluation Setup.We use a WidowX-250s arm with a parallel gripper and dual-camera visual input (third- person and wrist view). The Executor휋 푒 runs at 2Hz, predicting an action chunk퐴 푡 of 10 actions (10Hz), of which 5 are executed open-loop. To monitor the subtask progress, the Sentry휋 푠 is queried after every 10 execution steps of휋 푒 . The Sentry will trigger Planner휋 푝 if the current subtask is completed to invoke high-level planning. Each task’s evaluation metrics are detailed inTab. 5. 4.2Main Experiment To thoroughly validate the effectiveness of our hierarchical memory and the monitor-based trigger mecha- nism, we keep the low-level policy휋 e and the Planner backbone constant, varying only thememory context and theplanning frequency. The comparisons are designed as follows: Transient Memory:A standard hierarchical VLA baseline where a memory-less Planner is invoked at fixed intervals (푛 푚 ) and receives only the current observation표 푡 . This represents the current paradigm of hierar- chical robot foundation models, such as Hi-robot [8]. Transient Memory w/ Sentry:ThisvariantintroducesourSentrymoduletotriggerthePlannerbasedontask progress. However, the Planner remains memory-less, receiving only표 푡 upon being triggered. It evaluates the benefit of the more stabilized planning from the Sentry’s gating. Flat Memory:The Planner operates at a fixed frequency and receives the 8 most recent observations, assisted by a FIFO queue of historical keyframes it previously selected. This setting tests whether unstructured, Flat Memory is sufficient for long-horizon tasks. This setting is similar to MemER [5]. HiMe w/o Sentry:We utilize our complete Planner’s memory design but remove the sentry module. The Planner is forced to query the VLM at a fixed frequency regardless of subtask progress. This ablation vali- datesthesentry’sroleinreducingcomputationalredundancyandaligningplanningstepswiththedynamics of environments. HiMe (Ours):The proposed framework, where the Sentry dynamically triggers planning based on subtask progress, and the Planner utilizes the full hierarchical memory to ensure global consistency. Human High-level:A human oracle provides the correct subtask description at each step. This serves as an upper bound on task performance given the fixed capabilities of휋 푒 . 4.3Analysis of Sentry Mechanism We analyze the Sentry mechanism’s contribution from two key dimensions:execution consistencyandobser- vation quality. Table 1Hierarchical settings used in the main experiment. Hierarchy SettingTrigger Planner Memory Transient MemoryPeriodic Current observation Transient Memory w/ Sentry Sentry Current observation Flat MemoryPeriodic Recents + FIFO HiMe w/o SentryPeriodic Recents + keyframes HiMe (Ours)Sentry Recents + keyframes Human High-levelSentry Human oracle 1) Consistency via Reduced Task Switching:Even in a Planner with transient memory setting, the Sen- try significantly boosts performance. InFig. 4, we observe a substantial improvement (14%vs.26%) when comparingTransient MemorywithTransient Memory w/ Sentry. Without the Sentry, the Plan- ner operates on a frame-by-frame style, and this high-frequency re-evaluation makes the robot hy- persensitive to transient visual noise, leading to frequent, erratic switching between subtasks. The Sentry prevents the Planner from intervening until the current subtask is completed, reducing unnecessary task switching and ensuring that actions are executed coherently. It reveals thatthe core value of the Sentry here is enforcing temporal consistency of Planner. 2) Quality via Uniform Sampling vs. Recent Frames:The Sentry’s ability to consolidate observations into high- quality working memory largely explains the gap betweenHiMe w/o SentryandHiMe, with average task 8 Object Search Counting RearrangementAverage 0 20 40 60 80 100 Task Progress (%) 0 18 23 14 0 23 56 26 64 58 73 65 66 71 66 68 9292 87 90 94 96 92 94 Transient Memory Transient Memory w/ Sentry Flat Memory Hime w/o Sentry HiMe (ours) Human High-level Figure 4 Main Results.We compare HiMe against Transient, Sentry, and Flat Memory baselines across three long- horizon tasks. HiMe significantly outperforms all baselines, achieving a 90% average success rate. Notably, our method effectively bridges the gap between robot and the Human High-level oracle. progress rising from68%to90%. Whereas standard Planners rely on a narrow window of recent frames, Sentry aggregates the history of the active subtask into structured working memory. This provides a global perspective for success verification, and the consolidated memory allows the Planner to verify the entire progression, effectively reducing hallucinations and redundant replanning. 4.4Analysis of Planner Memory Management We validate the necessity of our hierarchical design by comparingHiMeagainst memory-free baselines and flat contextual memory approaches. (1) The Necessity of Memory:Long-horizon tasks require persistent state tracking. InCounting, even with Sentry stabilization (Transient Memory w/ Sentry), the robot still fails (23%) due to the lack of persistence: without an explicit memory management, it cannot retain states (e.g., counted objects) once they leave the transient observable state. (2) Consolidation vs. FIFO Queues:A critical insight from Fig. 4is the inefficiency ofFlat Memory, which func- tions as a contextual memory based on FIFO queues. Although it stores history, it achieves a significantly lower task progress (65%) compared toHiMe(90%). The fundamental limitation of such contextual memory is the lack of consolidation: a simple FIFO queue cannot distinguish between truly critical frames and redun- dant observations. In long-horizon tasks, the limited context window is quickly flooded with noisy frames, causing earlier critical information to be discarded.HiMeaddresses this by actively consolidating memory at the subtask level. This ensures that essential high-level information is preserved regardless of the episode length, enabling high-precision retrieval with minimal Planner overhead. 5Further Analysis 5.1Q1: What Type of Memory Representation is Most Effective for Robotic Tasks? We investigate the impact of memory modality by comparing purely text memory (Only text), purely visual memory (Only image), and our cross-modal approach in Fig. 5. 1) Spatial and Recognition Demands in Robotics: Ourstudyresultsshowthatroboticstasksimposestrongspatial localization and fine-grained recognition demands that caption only memory is inadequate for dynamical physical interactions. Text is a lossy compression: if perception initially misses an object or its spatial context, discarding raw visuals prevents visual re-grounding to recover details or correct errors. InObject Search, Only image(86%) substantially outperformsOnly text(74%), which shows that memory with visual evidence 9 Object Search Counting RearrangementAverage 50 60 70 80 90 100 Task Progress (%) 86 78 76 80 74 91 84 83 9292 87 90 Only imageOnly textOurs Figure 5 Ablation of Modality.HiMe consistently outperforms both text-only and image-only baselines across all three tasks, verifying the robustness of our cross-modal memory mechanism. Table 2Comparison of different methods. We report API calls, memory hit, and average scores across three tasks. Here, API Calls refers to the average Planner requests per subtask, and Memory Hit measures the presence of required information in memory during memory-dependent subtask. MethodComponentsObject SearchCountingRearrangement Sentry Memory Management Memory Size API Call (↓) Memory Hit (↑) Avg. Progress (↑) API Call (↓) Memory Hit (↑) Avg. Progress (↑) API Call (↓) Memory Hit (↑) Avg. Progress (↑) HiMe (Ours)✓푖푛푓푖푛푖푡푒1.894%92%2.698%92%1.492%87% Flat Memory×푟푒푐푒푛푡85.468%64%4.861%58%6.276%73% is significantly stronger. 2) Text for Semantics and Logic:Conversely, text memory is indispensable for non-spatial reasoning. It ef- ficiently summarizes semantics, captures logical dependencies like task sequencing, and retains user pref- erences. In tasks requiring semantic reasoning, such asCounting,Only text(91%) outperformsOnly image (78%), since these semantic clues are hard to extract from pixels on demand. 3) Superiority of Interleaved Memory:Our cross-modal approach achieves the best performance (Average90%) by combining the spatial/recognitional fidelity of images with the semantic and logical structure of text, enabling our hierarchical memory system to perform both precise grounding and effective reasoning. 5.2Q2: Is Dynamic Memory Management Necessary? To assess the need for dynamic memory management, we run an ablation against two baselines: (1)No Management, which only supportsAdd/Retrieveand keeps all observations as an append-only buffer; and (2)FIFO, which caps memory at 8 entries and evicts the oldest when full. 1) The Cost of Forgetting:In Fig. 6,FIFOperforms worst (68%average), far belowNo Management(86%). This shows that for long-horizon tasks,early context is critical: naive eviction can remove essential information (e.g., an object location observed early) needed for later reasoning. 2) The Value of Consistency:WhileNo Managementachieves decent performance by retaining all history, it is still inferior to our full method (86%vs.90%). AlthoughNo Managementretains all history, it remains below our full method (86%vs.90%) due toredundancy and inconsistency. WithoutUpdate/Delete, memory stores obsolete states (e.g., prior object locations) alongside current ones, introducing noise duringQuery and potentially confusing the Planner. 3) Necessity of Active Management:Our method performs best by actively curating memory:Updaterefreshes outdated entries to maintain consistency, andDeleteremoves redundancy. Thus, storing more data is insuf- ficient; effective robots must maintain a concise, consistent memory. 10 Object Search Counting RearrangementAverage 50 60 70 80 90 100 Task Progress (%) 67 73 64 68 90 89 80 86 9292 87 90 FIFONo managementOurs Figure 6 Ablation of Management.Comparison between FIFO, No management, and our approach. HiMe consistently achieves the highest task progress across all three tasks, proving the effectiveness of our memory retrieval and consoli- dation mechanism in long-horizon tasks. 5.3Q3: Why Does HiMe Achieve Lower Latency and Fewer API Calls? We further analyze efficiency and reliability by recording total Planner invocations (averageAPI Callsper subtask) and theMemory Hitrate (whether required historical context is available at query time). Beyond the gains inSec. 4.3andSec. 4.4,Tab. 2shows a clear efficiency advantage:HiMereduces API calls by ∼3×(e.g., 5.4 to 1.8 in Object Search) over Flat Memory. Since VLM inference dominates latency, fewer calls directly yield faster execution. This efficiency stems from two structural advantages over the Flat memory: 1) Sentry Reduces Invocation Frequency:In Flat memory methods, the Planner is often queried every step to interpret the visual buffer, placing the expensive LLM in a high-frequency control loop. Instead, after the Planner specifies a subtask, the Sentry handles dense completion checking and wakes the Planner only when necessary. Thus, Planner involvement shifts fromstep-by-stepto sparsesubtask-levelcalls. 148 Number of Frames 0 20 40 60 80 100 Percentage (%) Precision Recall Figure 7Increasing the number of recent frames provided to the Sentry leads to a con- sistent improvement in both precision and re- call. This indicates that a longer temporal con- text is beneficial for accurate subtask monitor- ing. 2) Infinite Memory vs. FIFO Forgetting:Flat Memory uses a FIFO buffer ( Tab. 1), forming a sliding window that forgets early ob- servations. The resulting low memory hit rate (68%vs.94%) forces redundant exploration when needed context has been evicted. In contrast, HiMe maintains an infinite structured memory, avoiding this forgetting and enabling immediate re- trieval without physical re-exploration. 5.4Q4: What Affects Sentry’s Termination Behavior? To understand the Sentry’s decision-making boundary, we an- alyzed its performance on subtask completion detection under different working memory sizes (window length푁).Fig. 7 presents the Precision and Recall, where Subtask “Done” is treated as the positive class. 1) Benefits of Temporal Context:As shown in the Fig. 7, increasing the context window from 1 to 8 frames improves both Precision (from∼76%to∼82%) and Recall (from∼22%to∼35%). A singleframeoftenlackssufficientinformationtodistinguishbetweenatemporarypauseandtruecompletion. By observing a sequence of 8 frames (consistent with our sentry‘s working memory), the Sentry can leverage temporal cues to make more reliable judgments. 2) The ”Conservative” Nature of Sentry:The significant gap between high Precision and low Recall reveals 11 the Sentry’s inherent “conservative” nature. The system exhibits a strong bias towards the incomplete state, preferring to continue execution unless overwhelmingly confident that the goal is met. While this minimizes premature stops (high Precision), the low Recall creates a “missing signal” problem where the agent might overshoot its target or enter infinite loops. Since the Sentry is prone to False Negatives (missing the “Done” event), we design a fixed-interval Planner fallback. It ensures that execution loops are eventually broken even when the Sentry fails to trigger. 6Conclusion In this work, we presented HiMe, a novel hierarchical embodied memory framework that resolves the fun- damental frequency-competence paradox in long-horizon Vision-Language-Action control. By decoupling embodied intelligence into a high-frequency Executor, a progress-aware Sentry, and a strategic Planner, we mirror the multi-store structure of human cognition to balance real-time responsiveness with deep reason- ing. Our introduction of cross-modal semantic schemata and active management mechanisms (Add, Update, Delete) transforms robotic memory from a passive observation buffer into a dynamic, self-evolving knowl- edge system. Extensive experiments demonstrate that HiMe achieves 90% average success rate—effectively bridging the gap to human-level performance. We believe that this shift from flat, transient policies to hierar- chical, structured memory is a vital step toward developing truly autonomous and adaptive robotic agents. Acknowledgements This work was supported by the National Natural Science Foundation of China (No. 62521004). We thank the incredible SII MakerClub for their generous supply of essential materials and invaluable technical assistance throughout this project. We are also deeply grateful to Chunbiao Feng and Hongbo Tang for their pivotal technical support in hardware setup and control implementation. Impact Statement This paper presents work where the goal is to advance the fields of machine learning and robotics. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here. Limitations The paper has several limitations that suggest directions for future work. Our evaluation is primarily con- ducted on real-robot tasks without a matched standard simulation benchmark. While this setting better captures memory-intensive long-horizon behavior under real-world dynamics, it limits controlled compari- son and benchmark-level reproducibility. In addition, the current tasks mainly rely on relatively simple ma- nipulation primitives such as pick-and-place, and broader validation on more diverse and dexterous skills remains necessary. Although we provide additional analysis on memory scaling and open-source Planner substitution, the current study does not yet fully characterize performance under very long horizons, larger environments, or extended deployment over time. 12 References [1]Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Paul Foster, Grace Lam, Pannag Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. Openvla: An open-source vision-language-action model.CoRR, abs/2406.09246, 2024. doi: 10.48550/ARXIV.2406.09246. URLhttps://doi.org/10.48550/arXiv.2406. 09246. [2]Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Manuel Y. Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Devin LeBlanc, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Allen Z. Ren, Lucy Xiaoyang Shi, Laura Smith, Jost Tobias Springenberg, Kyle Stachowicz, James Tanner, Quan Vuong, Homer Walke, Anna Walling, Haohuan Wang, Lili Yu, and Ury Zhilin- sky.휋 0.5 : a vision-language-action model with open-world generalization.CoRR, abs/2504.16054, 2025. doi: 10.48550/ARXIV.2504.16054. URLhttps://doi.org/10.48550/arXiv.2504.16054. [3]Marcel Torne, Andy Tang, Yuejiang Liu, and Chelsea Finn. Learning long-context diffusion policies via past-token prediction.CoRR, abs/2505.09561, 2025. doi: 10.48550/ARXIV.2505.09561. URLhttps://doi.org/10.48550/arXiv. 2505.09561. [4]Haoquan Fang, Markus Grotz, Wilbert Pumacay, Yi Ru Wang, Dieter Fox, Ranjay Krishna, and Jiafei Duan. Sam2act: Integrating visual foundation model with A memory architecture for robotic manipulation. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu, editors, Forty-secondInternationalConferenceonMachineLearning,ICML2025,Vancouver,BC,Canada,July13-19,2025, Proceedings of Machine Learning Research. PMLR / OpenReview.net, 2025. URLhttps://proceedings.mlr.press/ v267/fang25c.html. [5]Ajay Sridhar, Jennifer Pan, Satvik Sharma, and Chelsea Finn. Scaling up memory for robotic control via experience retrieval. InTheFourteenthInternationalConferenceonLearningRepresentations, 2026. URLhttps://openreview. net/forum?id=1dH4ARGdwD. [6]Siyin Wang, Jinlan Fu, Feihong Liu, Xinzhe He, Huangxuan Wu, Junhao Shi, Kexin Huang, Zhaoye Fei, Jingjing Gong, Zuxuan Wu, et al. Roboomni: Proactive robot manipulation in omni-modal context. arXivpreprint arXiv:2510.23763, 2025. [7]Qwen Team. Qwen3-vl technical report.CoRR, abs/2511.21631, 2025. doi: 10.48550/ARXIV.2511.21631. URL https://doi.org/10.48550/arXiv.2511.21631. [8]Lucy Xiaoyang Shi, Brian Ichter, Michael Robert Equi, Liyiming Ke, Karl Pertsch, Quan Vuong, James Tanner, Anna Walling, Haohuan Wang, Niccolo Fusai, Adrian Li-Bell, Danny Driess, Lachy Groom, Sergey Levine, and Chelsea Finn. Hi robot: Open-ended instruction following with hierarchical vision-language-action models. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu, editors,Forty-secondInternationalConferenceonMachineLearning,ICML2025,Vancouver,BC,Canada,July 13-19,2025, Proceedings of Machine Learning Research. PMLR / OpenReview.net, 2025. URLhttps://proceedings. mlr.press/v267/shi25d.html. [9]Yi Li, Yuquan Deng, Jesse Zhang, Joel Jang, Marius Memmel, Caelan Reed Garrett, Fabio Ramos, Dieter Fox, Anqi Li, Abhishek Gupta, and Ankit Goyal. HAMSTER: hierarchical action models for open-world robot manipulation. In TheThirteenthInternationalConferenceonLearningRepresentations,ICLR2025,Singapore,April24-28,2025. OpenReview.net, 2025. URLhttps://openreview.net/forum?id=h7aQxzKbq6. [10]Yide Shentu, Philipp Wu, Aravind Rajeswaran, and Pieter Abbeel. From llms to actions: Latent codes as bridges in hierarchical robot control. InIEEE/RSJInternationalConferenceonIntelligentRobotsandSystems,IROS2024, AbuDhabi,UnitedArabEmirates,October14-18,2024, pages 8539–8546. IEEE, 2024. doi: 10.1109/IROS58592.2024. 10801683. URL https://doi.org/10.1109/IROS58592.2024.10801683. [11]Runhao Li, Wenkai Guo, Zhenyu Wu, Changyuan Wang, Haoyuan Deng, Zhenyu Weng, Yap-Peng Tan, and Zi- wei Wang. MAP-VLA: memory-augmented prompting for vision-language-action model in robotic manipulation. CoRR, abs/2511.09516, 2025. doi: 10.48550/ARXIV.2511.09516. URLhttps://doi.org/10.48550/arXiv.2511.09516. 13 [12]Yi Zhu, Fengda Zhu, Zhaohuan Zhan, Bingqian Lin, Jianbin Jiao, Xiaojun Chang, and Xiaodan Liang. Vision- dialog navigation by exploring cross-modal memory.CoRR, abs/2003.06745, 2020. URLhttps://arxiv.org/abs/ 2003.06745. [13]Hao Shi, Bin Xie, Yingfei Liu, Lin Sun, Fengrong Liu, Tiancai Wang, Erjin Zhou, Haoqiang Fan, Xiangyu Zhang, and Gao Huang. MemoryVLA: Perceptual-cognitive memory in vision-language-action models for robotic manip- ulation. In TheFourteenthInternationalConferenceonLearningRepresentations, 2026. URLhttps://openreview. net/forum?id=54U3XHf7q. [14]Ruijie Zheng, Yongyuan Liang, Shuaiyi Huang, Jianfeng Gao, Hal Daumé I, Andrey Kolobov, Furong Huang, and Jianwei Yang. Tracevla: Visual trace prompting enhances spatial-temporal awareness for generalist robotic policies. InTheThirteenthInternationalConferenceonLearningRepresentations,ICLR2025,Singapore,April24-28,2025. OpenReview.net, 2025. URLhttps://openreview.net/forum?id=b1CVu9l5GO. [15]Zihao Wang, Shaofei Cai, Anji Liu, Yonggang Jin, Jinbing Hou, Bowei Zhang, Haowei Lin, Zhaofeng He, Zilong Zheng, Yaodong Yang, Xiaojian Ma, and Yitao Liang. JARVIS-1: open-world multi-task agents with memory- augmented multimodal language models.CoRR, abs/2311.05997, 2023. doi: 10.48550/ARXIV.2311.05997. URL https://doi.org/10.48550/arXiv.2311.05997. [16]Zaijing Li, Yuquan Xie, Rui Shao, Gongwei Chen, Dongmei Jiang, and Liqiang Nie. Optimus-1: Hybrid multi- modal memory empowered agents excel in long-horizon tasks. InTheThirty-eighthAnnualConferenceonNeural InformationProcessingSystems, 2024. URLhttps://openreview.net/forum?id=XXOMCwZ6by. [17]Zihao Wang, Shaofei Cai, Guanzhou Chen, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with LLMs enables open-world multi-task agents. InThirty-seventhConferenceon NeuralInformationProcessingSystems, 2023. URLhttps://openreview.net/forum?id=KtvPdGb31Z. [18]Junming Liu, Yifei Sun, Weihua Cheng, Haodong Lei, Yirong Chen, Licheng Wen, Xuemeng Yang, Daocheng Fu, Pinlong Cai, Nianchen Deng, Yi Yu, Shuyue Hu, Botian Shi, and Ding Wang. Memverse: Multimodal memory for lifelong learning agents.CoRR, abs/2512.03627, 2025. doi: 10.48550/ARXIV.2512.03627. URLhttps://doi.org/10. 48550/arXiv.2512.03627. [19]Charles Packer, Vivian Fang, Shishir G. Patil, Kevin Lin, Sarah Wooders, and Joseph E. Gonzalez. Memgpt: Towards llms as operating systems.CoRR, abs/2310.08560, 2023. doi: 10.48550/ARXIV.2310.08560. URLhttps://doi.org/ 10.48550/arXiv.2310.08560. [20]Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, and Deshraj Yadav. Mem0: Building production-ready ai agents with scalable long-term memory.arXivpreprintarXiv:2504.19413, 2025. [21]Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, Juntao Tan, and Yongfeng Zhang. A-mem: Agentic memory for LLM agents. InTheThirty-ninthAnnualConferenceonNeuralInformationProcessingSystems, 2026. URLhttps: //openreview.net/forum?id=FiM0M8gcct. [22]Sikuan Yan, Xiufeng Yang, Zuchao Huang, Ercong Nie, Zifeng Ding, Zonggen Li, Xiaowen Ma, Hinrich Schütze, Volker Tresp, and Yunpu Ma. Memory-r1: Enhancing large language model agents to manage and utilize memories via reinforcement learning.CoRR, abs/2508.19828, 2025. doi: 10.48550/ARXIV.2508.19828. URLhttps://doi.org/ 10.48550/arXiv.2508.19828 . [23]Lin Long, Yichen He, Wentao Ye, Yiyuan Pan, Yuan Lin, Hang Li, Junbo Zhao, and Wei Li. Seeing, listening, remem- bering, and reasoning: A multimodal agent with long-term memory. InTheFourteenthInternationalConference onLearningRepresentations, 2026. URLhttps://openreview.net/forum?id=PMz29A7Muq. [24]Woongyeong Yeo, Kangsan Kim, Jaehong Yoon, and Sung Ju Hwang. Worldmm: Dynamic multimodal memory agent for long video reasoning. In ProceedingsoftheIEEE/CVFConferenceonComputerVisionandPattern Recognition(CVPR), pages 25599–25609, June 2026. [25]Bing Wang, Xinnian Liang, Jian Yang, Hui Huang, Zhenhe Wu, Shuangzhi Wu, Zejun Ma, and Zhoujun Li. SCM: enhancing large language model with self-controlled memory framework. In Feida Zhu, Philip S. Yu, Akiyo Nadamoto, Ee-Peng Lim, Kyuseok Shim, Wei Ding, and Bingxue Zhang, editors, DatabaseSystemsforAdvanced Applications-30thInternationalConference,DASFAA2025,Singapore,Singapore,May26-29,2025,Proceedings, PartVI, Lecture Notes in Computer Science, pages 188–203. Springer, 2025. doi: 10.1007/978-981-95-4158-4\_12. URLhttps://doi.org/10.1007/978-981-95-4158-4_12. 14 [26]OpenAI. Gpt-4o system card.CoRR, abs/2410.21276, 2024. doi: 10.48550/ARXIV.2410.21276. URLhttps://doi. org/10.48550/arXiv.2410.21276. [27]Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, Peter David Fagan, Joey Hejna, Masha Itkina, Marion Lepert, Yecheng Jason Ma, Patrick Tree Miller, Jimmy Wu, Suneel Belkhale, Shivin Dass, Huy Ha, Arhan Jain, Abraham Lee, Youngwoon Lee, Marius Memmel, Sungjae Park, Ilija Radosavovic, Kaiyuan Wang, Albert Zhan, Kevin Black, Cheng Chi, Kyle Beltran Hatch, Shan Lin, Jingpei Lu, Jean Mercat, Abdul Rehman, Pannag R Sanketi, Archit Sharma, Cody Simpson, Quan Vuong, Homer Rich Walke, Blake Wulfe, Ted Xiao, Jonathan Heewon Yang, Arefeh Yavary, Tony Z. Zhao, Christopher Agia, Rohan Baijal, Mateo Guaman Cas- tro, Daphne Chen, Qiuyu Chen, Trinity Chung, Jaimyn Drake, Ethan Paul Foster, Jensen Gao, David Antonio Her- rera, Minho Heo, Kyle Hsu, Jiaheng Hu, Donovon Jackson, Charlotte Le, Yunshuang Li, Xinyu Lin, Zehan Ma, Abhiram Maddukuri, Suvir Mirchandani, Daniel Morton, Tony Khuong Nguyen, Abigail O’Neill, Rosario Scalise, Derick Seale, Victor Son, Stephen Tian, Emi Tran, Andrew E. Wang, Yilin Wu, Annie Xie, Jingyun Yang, Patrick Yin, Yunchu Zhang, Osbert Bastani, Glen Berseth, Jeannette Bohg, Ken Goldberg, Abhinav Gupta, Abhishek Gupta, Di- nesh Jayaraman, Joseph J Lim, Jitendra Malik, Roberto Martín-Martín, Subramanian Ramamoorthy, Dorsa Sadigh, Shuran Song, Jiajun Wu, Michael C. Yip, Yuke Zhu, Thomas Kollar, Sergey Levine, and Chelsea Finn. DROID: A large-scale in-the-wild robot manipulation dataset. InRSS2024Workshop:DataGenerationforRobotics, 2024. URLhttps://openreview.net/forum?id=Ml2pTYLNLi. 15 Appendix ATask Design and Metrics Our real-robot evaluation consists of three types of structured tasks:Object Search,Counting, andRe- arrangement. These tasks are designed to cover the aspects of long-horizon embodied decision-making, including inspection-based memory acquisition, persistent state tracking, dynamic memory revision, and preference-aware planning. Tasks.Table3summarizes the task settings and their corresponding memory challenges. Table 3Real-robot tasks in our evaluation. TaskSettingCore Memory Challenge Object SearchThree boxes with unseen internal con- tents and visible objects on the table. The robot must actively inspect the boxes to infer their contents and then place visible objects accordingly. Dur- ing execution, the environment may change, requiring memory revision. Memory formation under partial observability throughactive in- spection, together withcontinuous memory updatesin changing envi- ronments. CountingMultiple ingredient plates and a recipe specifying required propor- tions. The agent must first read the recipe and then repeatedly pick ingredients across multiple steps. Persistent semantic memory and cumulative progress trackingover long horizons. RearrangementMultiple objects and a container. The agent must first collect all objects while remembering their original lo- cations, and later restore them to their original positions. Long-horizon memory retention and consistency over extended execution. Although these tasks share the same hierarchical control framework, they stress different forms of memory usage.Object Searchemphasizes active perception and belief updating under partial observability, since the robot cannot determine box contents without inspection and must revise memory when the environment changes.Countingfocuses on persistent semantic memory and progress monitoring, as the robot must retain the recipe requirements and keep track of cumulative ingredient collection across repeated action cycles.Rearrangementrequires longer-term spatial memory, because the robot must remember the original object layout during collection and later use this stored information to restore the scene. Across all task families, we additionally introduceuser preference constraints, such as preferred objects, disliked ingredients, or placement rules. These constraints require the agent not only to store environment states, but also to maintain and use user-specific information during planning and execution. As a result, the system must integrate memory with high-level reasoning and dynamically adjust its plan when new observations or updated preferences become relevant. Action steps.Tofurthercharacterizethedifficultyofthesereal-robottasks, wereportthenumberofsubtasks and the approximate total number of action steps required by each task in Table4. These statistics reflect that all three tasks require multi-stage execution over extended horizons, rather than single-step or short reactive behaviors. 16 Table 4Action steps across 3 tasks. TaskNumber of Subtasks Total Action Steps Object Search6∼1450 Counting7∼1200 Rearrangement6∼1035 Evaluation.Each task is evaluated over20 trials per method. We measure performance using the metrics summarized in Table5. These metrics are designed to capture three complementary aspects of system per- formance.Task Progressmeasures execution quality at the task level, indicating whether the agent can successfully complete the required long-horizon objectives.API Callsmeasures reasoning frequency and therefore serves as a proxy for computational cost and system efficiency.Memory Hit Ratedirectly evalu- ates whether the maintained memory contains the information needed for effective replanning and decision- making when the Planner is invoked. Together, these metrics provide a joint assessment of task completion quality, computational efficiency, and memory effectiveness. Table 5Evaluation metrics. MetricDescription Task ProgressEach task is decomposed into multiple subtasks. Task Progress measurestheproportionofcompletedsubtasks, reflectingtheover- all task completion level. API CallsThe average number of Planner invocations per subtask, reflecting reasoning frequency and computational cost. Memory Hit RateWhether the required information is present in memory when the Planner is invoked, reported as the probability of successful mem- ory retrieval. BModel Initialization, Training, and Deployment We provide the initialization, training, and deployment details of the three-layer HiMe system, including the low-level Executor, the high-level Planner, and the Sentry module. Model setup.HiMe consists of three components: a low-level Executor for action generation, a high-level Planner for memory update and subtask decomposition, and a Sentry for execution monitoring and replan- ning trigger prediction. The overall model setup is summarized in Table6. Table 6Model initialization and deployment setup of HiMe. Module ModelRole Planner GPT-4o / Qwen3-VL-30B Memory update and replanning Sentry Qwen3-VL-8BExecution monitoring and trigger prediction Executor휋 0.5 Low-level action generation TheExecutoris initialized from the public휋 0.5 checkpoint trained on the DROID dataset [27], and is further fine-tuned on our task-specific real-robot demonstrations. ThePlanneris instantiated with GPT-4o in the main experiments. To improve reproducibility, we additionally replace the Planner with an open-source VLM, Qwen3-VL-30B, in supplementary experiments under the same evaluation protocol. TheSentryis implemented with Qwen3-VL-8B and deployed locally for online monitoring during execution. 17 Robot observations and control interface.The Executor takes as input two RGB observations, including a third- person view and a wrist camera view, together with the robot state consisting of the end-effector pose and gripper state. It predicts low-level action chunks for control. In our implementation, the Executor predicts 10 actions at a time, corresponding to an action horizon of 10. The robot platform is a WidowX-250 manipulator with a parallel gripper. The overall control loop runs at 10Hz. Low-level policy training.The Executor is fine-tuned on task-specific real-robot demonstrations collected on the WidowX-250 platform. Our training set contains 60 demonstrations in total, including 50 standard demonstrations and 10 additional corner-case demonstrations. We follow the standard OpenPI training recipe. The main hyperparameters are listed in Table7. Table 7Hyperparameters for low-level Executor (휋 0.5 ) fine-tuning. HyperparameterValue OptimizerAdamW 훽 1 0.9 훽 2 0.999 Weight Decay0.1 Gradient Clip Norm 1.0 LR ScheduleCosine decay Warmup Steps10k Peak Learning Rate5×10 −5 Batch Size256 EMA Decay0.999 Training Steps30k Action Horizon10 Demonstrations60 (50 regular + 10 corner cases) Planner and Sentry deployment.The Planner is invoked only when replanning is required, rather than at ev- ery control step. In the main experiments, the Planner is instantiated with GPT-4o through the OpenAI API. For local deployment experiments, we serve Qwen3-VL models with vLLM through an OpenAI-compatible API interface. The detailed deployment configuration is summarized in Table8. Table 8Deployment configuration for locally served VLM modules. Unless otherwise specified, all other settings use default vLLM configurations. ModuleModelGPU SetupTemperature Max Model Len SentryQwen3-VL-8B 1×H100 (80GB) 0.68192 Planner (supp.) Qwen3-VL-30B 2×H100 (80GB) 0.616384 Inference setup.Both local models are served with vLLM using an OpenAI-compatible endpoint and inte- grated into the online real-robot system through standard API-based invocation. Real-robot setup.All task-specific demonstrations are collected on a real WidowX-250 platform under a fixed tabletop setup, as shown in Figure8. The setup includes a front-facing Intel RealSense D435 camera for third- person observation, a wrist-mounted camera for close-range manipulation feedback, a set of task objects placed in the shared workspace, and multiple containers used for long-horizon manipulation tasks. Demon- strations are collected through leader-follower teleoperation, and all robot control and sensor streams are integrated through ROS. 18 Intel Realsense D435 Camera WidowX Robot Experiment Objects Figure 8Real-world experimental setup. The system consists of a WidowX-250 robot arm, a front-facing Intel RealSense D435 camera, and a tabletop workspace containing task objects and storage boxes. During data collection, both the external camera and the wrist camera record RGB observations at 25Hz. Each trajectory is synchronized with robot proprioceptive states, including end-effector pose, joint states, and gripper state. After collection, raw trajectories are processed through a standardized preprocessing pipeline. All images are resized to224×224, and the trajectories are temporally subsampled to 10Hz. The processed demonstrations are then converted into the LeRobot format for downstream training. CAdditional Experiments We provide additional experiments to further analyze controlled memory management, Planner generaliza- tion with open-source VLMs, memory scalability over extended horizons, and system latency. C.1Controlled Memory Ablation We first conduct a controlled memory ablation on theRearrangementtask to separate the effect of memory management strategy from that of memory capacity. In the main setting, HiMe uses unconstrained active memory, while the FIFO baseline is evaluated with a fixed memory budget. Although the FIFO budget is chosen to match the average memory size of the unlimited active baseline, the two settings still differ in whether memory is explicitly constrained. To provide a cleaner comparison, we additionally evaluate a limited-memory version of active management under the same fixed budget. 19 Specifically, wecomparethreevariants:Unlimited Active Management, corresponding totheoriginalHiMe setting with unconstrained memory;Limited Active Management, which uses the same Add / Update / Delete mechanism under a fixed budget of 8 entries; andFIFO (Limited), which uses the same budget with FIFO eviction. Table 9Controlled memory in Rearrangement. MethodMemory Budget Task Progress (%) Unlimited Active Management Unbounded87.0 Limited Active Management Fixed (8 entries)80.0 FIFO (Limited)Fixed (8 entries)64.0 Performance decreases under constrained memory, but active management remains substantially stronger than FIFO under the same capacity. This result indicates that the benefit of HiMe is not explained by larger memory alone, and that explicit Update / Delete operations remain useful beyond simple eviction. C.2Open-Source Planner Evaluation To improve reproducibility and examine Planner generalization, we replace the main Planner with the open- source VLMQwen3-VL-30Band evaluate the system on theRearrangementtask under the same protocol. The results are shown in Fig.9. Fig.9(a) reports the main comparison with the corresponding baselines. Fig.9(b) presents the modality ablation, comparing the full system with text-only and image-only memory. Fig.9(c) shows the memory ablation under the same open-source Planner. Overall, the trends are consistent with the main experiments: HiMe remains the best-performing variant with Qwen3-VL-30B, indicating that the benefits of structured memory and sentry-based coordination are robust to the choice of Planner. 0 20 40 60 80 100 Task Progress (%) (a) Main Comparison HiMe (Ours) HiMe w/o Sentry Flat Memory Transient Memory Transient Memory w/ Sentry 25.025.0 48.348.3 83.3 0 20 40 60 80 100 (b) Modality Ablation HiMe (Ours) Only text Only image 58.3 65.0 83.3 0 20 40 60 80 100 (c) Memory Ablation HiMe (Ours) No Management FIFO 70.0 63.3 83.3 Figure 9Additional experiments on theRearrangementtask withQwen3-VL-30Bas the Planner. (a) Main comparison with the corresponding baselines. (b) Modality ablation. (c) Memory ablation. The overall trends are consistent with the main experiments, and HiMe remains the best-performing variant under the open-source Planner. C.3Memory Scalability We further analyze memory scalability using an extended multi-round version ofObject Search. In each round, the robot must inspect boxes with initially unknown contents and place the visible table object accord- ingly. At the beginning of each new round, the box contents are changed, requiring the agent to re-inspect the scene and revise memory before acting. As the number of rounds increases, both the task horizon and the number of subtasks increase. We report the averageMemory SizeandTask Progressacross different numbers of rounds. 20 Table 10Memory scalability in extended multi-round Object Search. Rounds Subtasks Memory Size Task Progress (%) 145.592.5 289.976.3 31214.066.7 As the horizon increases, memory size grows while task performance declines. This result suggests that the main challenge in long-horizon settings is not only storing more information, but also maintaining correct and up-to-date memory as the environment evolves. C.4System Latency We additionally measure Planner latency under different deployments. Each Planner call consists of two stages: memory query / retrieval, and plan generation together with memory update. We report end-to- end latency over complete Planner calls. Table 11End-to-end Planner latency under different model deployments. ModelP50 (s) P90 (s) P99 (s) Avg. Completion Tokens GPT-4o (API)38.59 49.30 57.28517.9 Qwen3-VL-30B (local) 6.837.938.65532.8 Qwen3-VL-8B (local) 6.107.08 11.34399.0 Latency differences are mainly associated with deployment mode, i.e., remote API versus local serving, rather than token scale alone. In practice, HiMe is compatible with different Planner backends, and local deployment substantially reduces reasoning latency. Since Planner calls are triggered only when necessary, this reasoning latency remains decoupled from the high-frequency low-level control loop. 21 DPrompts Details Sentry Prompt You are theEXECUTION OBSERVERof an embodied agent. Primary input modality: •Plan list: –The model’s current decomposition of the task into subtasks. –Each subtask line includes its execution status (e.g., [done], [current], [pending]). •Combined images of recent frames: –CRITICAL LAYOUT INFO:Each image is a composite. –LEFT HALF: Wrist Camera View (Close-up, attached to the gripper). –RIGHT HALF: Third-Person View (Global view of the robot and environment). YourONLYjob is to judge whether theCURRENTsubtask has been completed. Judgment Criteria: •Inspect:done when the robot arm is above or extended into the box, and the wrist view shows the content and the black background of the box. •Pick & Place:done when the object has been successfully released at the destination or you observe that the robot arm is above a box and see its black background. •Reset:done when the robot arm is in the home pose. –The robot arm home pose is: in both images, the arm is aligned parallel to what appears to be a rail or track. The end effector (or gripper) is open and facing downwards towards the table. •The interior bottom surface of the box is lined with a black background to enhance contrast for detection. # REQUIRED OUTPUT FORMAT YouMUSToutput exactly this XML structure: <status> <!-- EXACTLY one of: done not_done --> </status> Planner Prompt You are thePLANNERof an embodied agent. Primary input modalities: •User instruction:The user’s high-level instruction for the agent. •Plan list:The model’s current decomposition of the task into subtasks. Each subtask line includes its execution status (e.g., [done], [current], [pending]). •Combined images during subtask execution: –IMAGE LAYOUT & USAGE GUIDE: –LEFT HALF (Wrist View):Egocentric view attached to the gripper. Usage: Identify specific objects, read text, verify grasping. –RIGHT HALF (Third-Person View):Globalview. Usage: Spatialcontext, locatingcontainers, tracking arm position. –NOTE:Integrate information from both. Use Right view to navigate, Left view to inspect. You have access to an externalMEMORYmodule through a structured CRUD interface. MEMORY stores records with fields:id,tags,data(type, value),image_path. 22 # TAG DESIGN PRINCIPLES Tags are the PRIMARY mechanism for memory retrieval. 1.OBJECT TAGS:toy_duck,snack_package,left_box. 2.LOCATION TAGS:table,shelf,inside_left_box. 3.USER_PREFERENCE TAGS:user,lily,likes,prefers. Tag Usage Rules:Use 2-5 tags per record. Update tags when object state changes.Tag Query Strat- egy:PREFER tag-based queries. Query one tag every time. # MEMORY CRUD PROTOCOL You interact with memory only via structured XML operations. (1) QUERY (READ) <operation> <type>QUERY</type> <query>search key tags</query> <reason>why you search this</reason> </operation> (2) CREATE <operation> <type>CREATE</type> <tags>...</tags> <text>concise description or fact</text> <image_path>index</image_path> <reason>why this new memory is necessary</reason> </operation> (3) UPDATE <operation> <type>UPDATE</type> <id>record_id</id> <tags>...</tags> <text>updated description</text> <image_path>2</image_path> <reason>why this existing record must be changed</reason> </operation> (4) DELETE <operation> <type>DELETE</type> <id>...</id> <reason>why this record is no longer useful</reason> </operation> # EVIDENCE RELIABILITY RULES (NO ASSUMPTIONS) •Memory must contain only verifiable facts derived from explicit user instructions or direct visual evidence. •Do NOT CREATE FAKE memory.Prefer planning an inspection step. •UPDATE is for correcting/refining without changing time meaning. •When state changes,CREATEa new record, UPDATE the old one to label as past. •DELETE sparingly (only for duplicates or errors). # MEMORY USAGE POLICY (TWO-TURN INTERACTION) •Turn 1 (QUERY-only):Issue QUERY operations. NO Create/Update/Delete. Plan list must be 23 empty. •Turn 2 (REFINE + CRUD):Issue Create/Update/Delete based on query results. Output finalized <plan_list>. # PLAN LIST FORMAT (FINAL OUTPUT ONLY, TURN 2)Plain text, one subtask per line. •[done]: subtask completed. •[current]: the single subtask to execute NEXT. •[pending]: future subtasks. Annotation: You MAY include a short purpose in parentheses, e.g.,[pending] inspect recipe (check what does it need). # PLAN LIST ACTION FORMAT 1.Inspection action:Main verb ”inspect”. Use when information is missing. 2.Pick-and-place action:”pick up<object> <source>and place it to<target>”. # REQUIRED OUTPUT FORMAT FOR EACH TURN <summary> <!-- Concise reasoning summarizing observations and memory interaction --> </summary> <memory_operations> <!-- Turn 1: QUERY only. Turn 2: CRUD operations. --> </memory_operations> <plan_list> <!-- Turn 1: EMPTY. Turn 2: Finalized plan list. [done] ... [current] ... [pending] ... --> </plan_list> 24