Paper deep dive
Chameleon: Episodic Memory for Long-Horizon Robotic Manipulation
Xinying Guo, Chenxi Jiang, Hyun Bin Kim, Ying Sun, Yang Xiao, Yuhang Han, Jianfei Yang
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/26/2026, 2:19:52 AM
Summary
Chameleon is a bio-inspired episodic memory architecture for long-horizon robotic manipulation that addresses perceptual aliasing by using geometry-grounded multimodal tokens and a differentiable memory stack. It features a hierarchical memory system (EC-HC-PFC inspired) that enables goal-directed recall and is evaluated on the new Camo-Dataset, demonstrating improved decision reliability and control in perceptually confusable settings.
Entities (5)
Relation Signals (3)
Chameleon → addresses → Perceptual Aliasing
confidence 95% · Chameleon consistently improves decision reliability and long-horizon control over strong baselines in perceptually confusable settings.
Chameleon → utilizes → HoloHead
confidence 95% · Chameleon... produces goal-directed recall through a differentiable memory stack... HoloHead shapes this state with a latent imagination objective
Camo-Dataset → evaluates → Chameleon
confidence 90% · To systematically evaluate our proposed Chameleon, we introduce Camo-Dataset
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Robotic manipulation often requires memory: occlusion and state changes can make decision-time observations perceptually aliased, making action selection non-Markovian at the observation level because the same observation may arise from different interaction histories. Most embodied agents implement memory via semantically compressed traces and similarity-based retrieval, which discards disambiguating fine-grained perceptual cues and can return perceptually similar but decision-irrelevant episodes. Inspired by human episodic memory, we propose Chameleon, which writes geometry-grounded multimodal tokens to preserve disambiguating context and produces goal-directed recall through a differentiable memory stack. We also introduce Camo-Dataset, a real-robot UR5e dataset spanning episodic recall, spatial tracking, and sequential manipulation under perceptual aliasing. Across tasks, Chameleon consistently improves decision reliability and long-horizon control over strong baselines in perceptually confusable settings.
Tags
Links
- Source: https://arxiv.org/abs/2603.24576v1
- Canonical: https://arxiv.org/abs/2603.24576v1
Trouble viewing inline? Open PDF directly →
Full Text
106,906 characters extracted from source content.
Expand or collapse full text
Chameleon: Episodic Memory for Long-Horizon Robotic Manipulation Xinying Guo 1,2,∗ , Chenxi Jiang 1,∗ , Hyun Bin Kim 1 , Ying Sun 2 , Yang Xiao 1 , Yuhang Han 3 , Jianfei Yang 1,† 1 MARS Lab, Nanyang Technological University, 2 Institute for Infocomm Research, A*STAR, Singapore, 3 National University of Singapore ∗ Equal Contribution, † Corresponding Author Robotic manipulation often requires memory: occlusion and state changes can make decision-time observations perceptually aliased, making action selection non-Markovian at the observation level because the same observation may arise from different interaction histories. Most embodied agents implement memory via semantically compressed traces and similarity-based retrieval, which discards disambiguating fine-grained perceptual cues and can return perceptually similar but decision-irrelevant episodes. Inspired by human episodic memory, we propose Chameleon, which writes geometry-grounded multimodal tokens to preserve disambiguating context and produces goal-directed recall through a differentiable memory stack. We also introduce Camo-Dataset, a real-robot UR5e dataset spanning episodic recall, spatial tracking, and sequential manipulation under perceptual aliasing. Across tasks, Chameleon consistently improves decision reliability and long-horizon control over strong baselines in perceptually confusable settings. Correspondence: Jianfei Yang at jianfei.yang@ntu.edu.sg Code: https://github.com/gxyes/MARS_Chameleon 1 Introduction Robotic manipulation in the real world often requires acting from memory. As manipulation unfolds, task- relevant state can become occluded or overwritten, making the decision point non-Markovian at the ob- servation level and forcing correct actions to depend on historical interactions. This makes long-horizon memory a central bottleneck for embodied intelli- gence. Figure 1 illustrates the challenge in the shell game: when identical cups become indistinguishable at grasp time, the robot must rely on swap history to select the correct cup. A common solution is to equip an agent with long- term memory that is written during interaction and used later. In many embodied systems, this memory is language-centric: experience is summarized into semantically compressed, text-like traces, mirroring retrieval-augmented generation (RAG) popularized by large language models (LLMs) Lewis et al. (2020). Such compression is ill-suited to perceptual alias- ing in robotic memory-intensive tasks, because it re- moves disambiguating perceptual context that deter- mines the correct action, including spatial relations, viewpoint-dependent evidence, and the outcomes of prior interactions Wang et al. (2025); Liu et al. (2024); Fung et al. (2025). In Figure 1, a summary such as “the ball is under a cup” can be semantically cor- rect yet still insufficient to identify the correct cup after swaps. Recent work, therefore, augments em- bodied agents with visual history buffers Lin et al. (2025), but similarity-driven access still induces inter- ference when repeated, similar scenes correspond to different histories. Together, these limitations expose failures at both writing and retrieval: memory must be written as disambiguated, indexable events to re- duce interference, and retrieved based on whether it supports the current goal and near-future outcome, rather than on similarity. Biological episodic memory implements precisely these two functions. In evolutionary accounts, episodic memory is often viewed as a foundation of intelligence that predates abstract, symbolic reason- ing Allen and Fortin (2013); it organizes experience as events bound to spatiotemporal and causal context, enabling prospective behavior even without language. Non-linguistic animals provide concrete evidence of event-based planning, such as food-caching birds that remember what was stored, where it was stored, and when it was stored, and use these memories to guide later retrieval decisions Worsfold et al. (2025). Mech- anistically, episodic memory is commonly attributed 1 arXiv:2603.24576v1 [cs.RO] 25 Mar 2026 Coarse Semantic DB HUMAN OURS CURRENT The ball is under a cup. Semantic Feature Encoder Find the ball under the cup. Semantic Feature Encoder Dense Nearest- Neighbor Vector Retrieval What Where Goal Bond to Event Slow Episodic Memory Fast Working Memory EC-DG (Memory Encoder) PFC (Working Memory) Imagine Infer · · · · · Feature Encoder Feature Fusion Where - Dorsal Stream What - Ventral Stream Write Memory: Observe and track the ball. Retrieval Memory: Find the ball. LET US PLAY THE SHELL GAME ! HC (Episodic Memory) A – A’ (similarity) Sequential Recall CA3 Recurrent Autoassociative Network Imaginet-1 Low- Confidence Prior High- Confidence Prediction Imaginet Autocomplete Gone Here Here Figure 1 Memory-intensive robotic manipulation is often perceptually aliased: in the shell game, identical cups look the same at decision time while the correct grasp depends on earlier interactions. Existing approaches often discard fine-grained details needed for reliable decision-making. Inspired by the human Entorhinal Cortex–Hippocampus–Prefrontal Cortex (EC–HC–PFC) episodic memory system, we propose Chameleon for episodic recall, spatial tracking, and long-horizon task completion. to coordinated dynamics across the entorhinal cortex (EC), hippocampal formation (HC), and prefrontal cortex (PFC) Bakker et al. (2008); Zheng et al. (2025). Within HC, the dentate gyrus (DG) supports pattern separation, reducing interference so similar experi- ences are stored as distinct traces Bakker et al. (2008); CA3 supports pattern completion via associative re- trieval from partial cues; and CA1 supports episodic representations that bind recalled content into a con- textualized event for downstream use. PFC control biases recall toward memories that are relevant to current goals and anticipated near-future outcomes, rather than by similarity alone Zheng et al. (2025). Viewed through this lens, perceptual aliasing exposes two algorithmic gaps in current embodied systems: how to form disambiguated, indexable event represen- tations during memory formation, and how to learn goal-directed retrieval so recall is driven by decision utility. To address these gaps, we propose Chameleon, a bio-inspired memory architecture for long-horizon embodied manipulation. A geometry-grounded per- ception module converts multi-view observations into view-consistent patch tokens anchored to the end- effector, preserving evidence needed for disambigua- tion at write time. A hierarchical differentiable mem- ory stack couples an episodic state with a working state through continuous dynamics and produces a compact decision state. HoloHead shapes this state with a latent imagination objective that trains the decision state to be predictive of near-future state evolution. Conditioned only on the resulting state, a rectified-flow policy generates an end-effector pose trajectory in one shot, improving stability for long- horizon control. To systematically evaluate our proposed Chameleon, we introduce Camo-Dataset, a real-world dataset collected on a physical UR5e robot. It comprises episodic recall, spatial tracking, and sequential ma- nipulation tasks designed to enforce perceptual alias- ing while requiring history-dependent decisions. We compare Chameleon against strong baselines, includ- ing Diffusion Policy Chi et al. (2023), Flow Match- ing Zhang and Gienger (2024), and ACT Zhao et al. (2023), and against five ablated variants under identi- cal conditions. Across metrics including success rate, decision success rate, and manipulation success rate, Chameleon consistently improves decision reliability and long-horizon task completion. We also probe the learned memory state and observe separation between perceptually similar yet memory-distinct states, as well as pattern completion behavior enabled by the HoloHead objective. Our contributions are threefold: (1) Perceptual Aliasing Problem Formulation: We for- 2 mulate robotic memory-intensive tasks as decision problems with perceptual aliasing, where similar decision-time observations can require different ac- tions conditioned on interaction history, highlighting the need for episodic memory. (2) Bio-inspired Hierarchical Episodic Memory Architec- ture: We propose Chameleon, a bio-inspired hierar- chical memory architecture that integrates geometry- grounded disambiguating encoding, differentiable goal-directed retrieval trained via HoloHead, and memory-conditioned rectified-flow control for long- horizon manipulation. (3) Real-robot Episodic Memory Dataset and Anal- ysis: We introduce Camo-Dataset, a UR5e real- robot dataset for episodic recall, spatial track- ing, and sequential manipulation under perceptual aliasing, together with quantitative evaluation and representation-level analyses. 2 Related Works With advances in neuroscience, memory is commonly categorized into non-declarative and declarative mem- ory Squire (1992). Non-declarative memory is ex- pressed through behavioral performance, including skills and habits, whereas declarative memory in- volves conscious recall of facts and events. Within declarative memory, episodic memory is supported by pattern separation and pattern completion Bakker et al. (2008). Pattern separation integrates multi- modal sensory streams into distinct event representa- tions to disambiguate similar perceptions, while pat- tern completion retrieves events from partial cues to support downstream decisions Dickerson and Eichen- baum (2010); Ngo et al. (2021); Amer and Davachi (2023). In artificial intelligence, memory is often divided into parametric and non-parametric memory (anal- ogous to knowledge and episodic memory, respec- tively) Mallen et al. (2023); Su et al. (2025); Teyler and DiScenna (1986). Parametric memory encodes cross-task knowledge within model parameters, while non-parametric mechanisms store task-specific con- text. Many LLM-based agents adopt retrieval- augmented generation (RAG), querying external knowledge repositories via embedding-based simi- larity search to simulate long-term memory Lewis et al. (2020); Karpukhin et al. (2020); Borgeaud et al. (2022); Yao et al. (2022); Park et al. (2023). How- ever, retrieval remains challenging in zero-shot set- tings due to the lack of relevance supervision Gao et al. (2023), and semantic similarity does not nec- essarily correspond to task utility Fang et al. (2024). Therefore, recent work augments retrieval with struc- tured knowledge or query transformation to better capture long-range dependencies and reduce seman- tic gaps Wan et al. (2025); Jimenez Gutierrez et al. (2024); Gao et al. (2023). In embodied intelligence, memory mechanisms are of- ten inherited from LLM paradigms, relying on seman- tic embeddings or sparse topological maps to encode object relations and spatial structure Xie et al. (2024); Zhu et al. (2024); Mon-Williams et al. (2025); Anwar et al. (2025). Such abstractions compress continuous physical states into symbolic representations and dis- card fine-grained dynamics Liu et al. (2025); Fang et al. (2026). However, manipulation in unstructured environments requires tracking object poses, surface properties, contact dynamics, and evolving state tran- sitions. This mismatch between semantic memory and physical dynamics limits the generalization and operational accuracy of VLA systems. 3 Method 3.1 Overview Motivation. Robotic memory-intensive tasks often arise under perceptual aliasing: similar sensory in- puts can require different decisions because the dis- ambiguating context is occluded, transient, or es- tablished only through earlier interactions. Human episodic memory addresses this failure mode by com- bining disambiguating encoding, goal-directed recall, and a working state that supports online control. Mo- tivated by the functional roles of the EC–HC–PFC system, Chameleon implements these requirements in a fully differentiable architecture with a single decision state used as input to the policy. Problem setup. At each timestept, the agent receives a multimodal observation o t =I f t , I h t , s t , ψ t ,whereI f t andI h t are RGB images from a fixed front camera and a hand-mounted side camera,s t denotes the robot proprioceptive state including end-effector pose and gripper state, andψ t is an optional task-phase indicator (e.g., a discrete 0/1 signal denoting an observe phase versus an act phase) that specifies the interaction mode rather than revealing the hidden task state. Rather than predicting a single-step command, we parameterize control by generating a future end-effector (E) pose trajectory over a horizonH(Section 3.4). During training, we supervise the model with ground-truth future pose trajectoriesx t:t+H ∈ R H×d raw . During inference, 3 x y z INPUTS x y z x y z RGB Geometric VENTRAL & DORSAL STREAM Ventral DINO Dorsal Net GEO-BIASED ATTENTION Q K V Q K V Ventral QKVs Dorsal Bias Epipolar Bias Patch Geo Bias Fused Perception (a)Perception (b)Memory ... ... ... Q KV ( ) Q K V ( ) MEMORY LAYERS (c)Policy POLICY HEAD ... ... ... ... ... ... ... ... ... Fused Perception Phase 1 - “Observe” Phase 2 – “Find” Task Phase x y z Proprio MULTIMODAL FUSION Modality Emb View Emb Concatenate ... ... ... Projection Multimodal Tokens Velocity Network Embedding DiT Blocks * N Self - Attention Cross - Attention MLP Training Inference Conditional Flow Matching 풙 풕 =풕∙풙 ퟏ +(ퟏ−풕)∙풙 ퟎ 풖 풕 =풙 ퟏ −풙 ퟎ 풙 ퟎ ~ 푵 ( ퟎ , 푰 ) ODE Solver Iteration Loop 풕∈ퟎ,ퟏ,푲 푺풕풆풑풔 Update State 풙 풕 풊+ퟏ ←풙 풕 풊 +⋯ Action Prediction ... ... ... Action GT Timestep 풕∈[ퟎ,ퟏ] ... ... ... Memory Readout Camera Center End Effector FiLM Entorhinal Cortex (EC) EC Figure 2 Method overview. Chameleon follows a Perception→Memory→Policy pipeline for long-horizon manipulation under perceptual aliasing. Perception produces geometry-grounded, view-consistent tokens that disambiguate observa- tions before memory formation. Memory couples an episodic state with a working state and learns relevance-based recall through HoloHead rollout training. Conditioned on the memory readout, Policy samples a future end-effector pose trajectory via conditional flow matching for execution. Policy samplesˆx t:t+H and executes it using standard low-level inverse kinematics (IK) controllers. Perception→Memory→Policy. As illustrated in Figure 2, our method follows a closed-loop pipeline: x t = Φ perc (I f t ,I h t ,s t ,ψ t ), h t = Φ mem (x t ,h t−1 ), ˆx t:t+H = Φ pol (h t ). (1) Perception Φ perc converts the two camera views and proprioception into geometry-grounded, view- consistent patch tokensx t , reducing ambiguity before memory formation (Section 3.2). Memory Φ mem cou- ples an episodic state with a working state through fully differentiable dynamics and outputs a single decision stateh t that is the only interface to Pol- icy (Section 3.3). This design directly targets the two challenges in Section 1. First, Memory sup- ports disambiguated encoding so that perceptually similar situations can be written into distinct, in- dexable traces, reducing interference. Second, Mem- ory supports relevance-based recall so that retrieved content is selected for decision utility rather than perceptual similarity. We enforce goal-directed recall with HoloHead, an internal head trained with a la- tent imagination objective that compels the decision state to maintain and recall task-relevant memories. Conditioned only onh t , Policy Φ pol generates the future end-effector pose trajectory via conditional flow matching (Section 3.4). The single-interface constraint makes the role of memory explicit: if the agent succeeds at long-horizon control, the informa- tion required for correct action selection must be represented in h t and reliably propagated over time. 3.2 Perception Perception implements the disambiguation require- ment during memory formation: Decision states can be separated only if the sensory inputs (both visual and geometic) are encoded into sufficiently discrimi- native features. We write patch-level evidence from 4 two complementary views. The fixed front camera provides a stable global context, while the hand- mounted camera captures contact-scale details under occlusion. To prevent cross-view aggregation from mixing incompatible evidence, we use the E pose as an action-centric geometric anchor that can be pro- jected into both views to guide geometry-grounded cross-view communication (Figure 2 (a)). Inputs and output. Given (I f t ,I h t ,s t ,ψ t ), Perception outputs fused visual tokens x t ∈ R N×d for Memory. Ventral stream: patch appearance tokens. For each viewv ∈ f,h, a frozen DINO encoder produces patch tokens V v t = DINO(I v t )∈ R N v ×d . Dorsal stream: E-anchored geometry codes. From s t we obtain the E posep t ∈ R 3 and project it to each view asu E,v t = Π(K v ,T v t ,p t )∈ R 2 . For patch iin viewv, letu v t,i ∈ R 2 be the patch-center image coordinate and ̃u v t,i ∈ P 2 its homogeneous form. We define the unit ray and the action-centric descriptor in one line: r v t,i = (K v ) −1 ̃u v t,i ∥(K v ) −1 ̃u v t,i ∥ 2 ∈ R 3 , g v t,i = u v t,i , r v t,i , ρ v t,i , cosθ v t,i ∈ R 7 . (2) Hereρ v t,i =∥u v t,i − u E,v t ∥ 2 measures proximity to the E projection. We definecosθ v t,i =r v⊤ t,i d v t , where d v t ∈ R 3 is the unit direction from the camera center to the E pose in the camera frame, computed from (T v t ,p t ) (details in Appendix). A geometric encoder outputs conditioning codesC v t ∈ R N v ×d c for feature modulation and unary patch biasesb v t ∈ R N v for attention gating, with b v t,i = MLP(g v t,i ). Geometry-biased bidirectional cross-view enhance- ment. The front and hand views enhance each other through residual cross-attention. For atten- tion from viewato viewbwitha ̸=b, define Q=V a t W Q ,K=V b t W K , andU=V b t W V , and update ̄ V a t =V a t +Attn a→b (Q,K,U). The attention logits combine an analytic epipolar feasibility bias and learned unary patch biases: Attn a→b (Q,K,U ) = softmax QK ⊤ √ d + B epi a→b +b a t 1 ⊤ + 1(b b t ) ⊤ U. (3) B epi a→b ∈ R N a ×N b is computed from the calibrated rel- ative geometry using the fundamental matrixF a→b as a negative point-to-epipolar-line distance (full form in Appendix). The unary term fromb v t biases com- munication toward reliable, action-relevant patches derived from the E anchor, without introducing an explicit pairwise matcher. Geometry-conditioned modulation and output. We apply FiLM-style modulation and output fused evi- dence tokens: ̃ V v t (i) = γ(C v t,i )⊙ ̄ V v t (i) + β(C v t,i ), x t = Concat( ̃ V f t , ̃ V h t )∈ R N×d . (4) Thisx t preserves localized appearance evidence while restricting cross-view exchange to geometry- consistent correspondences, yielding a stable write- time evidence stream for Memory (Section 3.3). 3.3 Memory Memory maps fused perception to a single readout decision stateh t . To reduce interference under per- ceptual aliasing, it encodes multimodal inputs with Spatial Anchors and Temporal Anchors, so evidence is written into a spatiotemporal state at multiple effective half-lives (i.e., encoded and stored across different time scales). Episodic Memory retains long- range traces, Working Memory integrates recalled context for immediate control, and HoloHead trains h t to complete and maintain task-relevant informa- tion through imagination. This design follows the functional roles described in Section 1, from EC-style integration to DG-style separation, CA1 and PFC context maintenance, and CA3-style completion. Multimodal Fusion. Memory consumesx t ∈ R N×d from Perception (Section 3.2), wherex t = Concat( ̃ V f t , ̃ V h t ) andN=N f +N h . We form a pro- prioception token and, when available, a phase token asp t =Proj prop (s t )∈ R d andφ t =E phase [ψ t ]∈ R d . We then add learned additive tags to preserve token identity across modalities and views: ̄x v t = ̃ V v t + A mod vis + A view v , v ∈f,h, ̄p t = p t + A mod prop , ̄φ t = φ t + A mod phase . (5) Finally, we concatenate and normalize, z t = LN Concat[ ̄x f t , ̄x h t , ̄p t , ̄φ t ] ∈ R M×d , (6) yielding fused tokensz t for all layers. Ifψ t is unavail- able, we omit ̄φ t and set M = N + 1. 5 Dynamic context modulation. Each layer interprets the same fused evidence using the working context propagated from the previous layer. Leth (ℓ−1) t ∈ R d w be the working state output by layerℓ−1 at timet, and letW (ℓ) c ∈ R d×d w be a learned linear projection to token space. We broadcast-add the projected context to all tokens, where 1∈ R M is an all-ones vector: ̃z (ℓ) t = z t + 1 (W (ℓ) c h (ℓ−1) t ) ⊤ ∈ R M×d .(7) Spatial and Temporal Anchors. Pattern separation begins by decomposing visual evidence intoAspa- tial components. Let ̄x t =Concat[ ̄x f t , ̄x h t ]∈ R N×d withN=N f +N h . Routing weightsπ (ℓ) t,a,i ≥0 sat- isfy P A a=1 π (ℓ) t,a,i = 1 and are produced byπ (ℓ) t,·,i = softmax(Router (ℓ) ( ̄x t,i )). Each spatial anchor forms a visual summary and an anchor-specific token set by attaching shared global context: u (ℓ) t,a = N X i=1 π (ℓ) t,a,i ̄x t,i ∈ R d , ̃z (ℓ,a) t = Concat u (ℓ) t,a , ̄p t , ̄φ t ∈ R 3×d . (8) We then expand each spatial anchor output ̃z (ℓ,a) t into Btemporal slots indexed byb, representing distinct characteristic half-lives in the subsequent state-space dynamics. With learned temporal queriesq (ℓ) a,b B b=1 , the slot feature is f (ℓ) t,a,b = Attn q (ℓ) a,b , ̃z (ℓ,a) t ∈ R d , a =1,...,A, b = 1,...,B. (9) Collectively,f (ℓ) t,a,b forms anA× Bspatiotemporal slot matrix, whereaselects a spatial component and b selects a temporal resolution. Spatial-Temporal Memory with Episodic and Work- ing States. In layerℓ, each slot (a,b) maintains an episodic statem (ℓ) t,a,b . The collectionm (ℓ) t,a,b forms the hidden engram, expressed only through readout. Given f (ℓ) t,a,b , we compute θ (ℓ) t,a,b = Proj (ℓ) a,b (f (ℓ) t,a,b ),(10) whereθ (ℓ) t,a,b follows a Mamba-style Gu and Dao (2024) selective state space model (SSM) parameterization with content-dependent input/readout factors and a step-size term. We assign each temporal index b∈1,...,Ba distinct base step size ∆t (0) b to im- plement an explicit half-life schedule, and further modulate the effective step size by slot content via Proj (ℓ) a,b . This yields a structured, content-adaptive multi-timescale memory, rather than a naive stack of A× Bidentical SSMs. For efficiency, we implement a fused slot-wise scan that updates allA× Bstates in a single pass. Episodic Memory updates the slot states by m (ℓ) t,a,b = SSM (ℓ) epi m (ℓ) t−1,a,b ; θ (ℓ) t,a,b ,(11) and produces slot readoutsr (ℓ) t,a,b =Read (ℓ) epi (m (ℓ) t,a,b )∈ R d . We aggregate these readouts into an episodic recall vector r (ℓ) t = 1 AB A X a=1 B X b=1 W (ℓ) a,b r (ℓ) t,a,b ∈ R d w , (12) and integrate it into Working Memory, which is the only output state of layer ℓ: h (ℓ) t = SSM (ℓ) work h (ℓ) t−1 ; LN h (ℓ) t−1 + Proj (ℓ) r (r (ℓ) t ) , (13) where h (ℓ) t ∈ R d w . Input and readout is realized implicitly by state-space recursion: engram enters through working-state in- puts, persists in episodic states, and is expressed through readouts, with no explicit retrieval gate re- quired. Working Readout and Hierarchical Memory Fu- sion. Layerℓemits a Working Readouty (ℓ) t = LN(W (ℓ) o h (ℓ) t )∈ R d w , whereW (ℓ) o is a learned lin- ear projection. We fuse readouts fromLlayers with a task queryq task ∈ R d w . We define the layer key k t,ℓ = U ℓ y (ℓ) t and value v t,ℓ = Enc ℓ (y (ℓ) t ): α t,ℓ = softmax ℓ ⟨q task , k t,ℓ ⟩ , h t = L X ℓ=1 α t,ℓ v t,ℓ . (14) The resultingh t is the final Memory Readout con- sumed by Policy. HoloHead. HoloHead enforces pattern completion via a latent imagination objective: conditioned on h t , it predictsN a +N c waypoints in 2D (front- camera image-plane E projection) and 3D (nor- malized world-frame E pose). ˆw 2D t:t+N a ˆw 2D t:t+N c ˆw 3D t:t+N a ˆw 3D t:t+N c = HoloHead(h t ).(15) 6 and we supervise with L holo = X ξ∈2D, 3D ∥ ˆw ξ t:t+N a − w ξ t:t+N a ∥ 1 +∥ ˆw ξ t:t+N c − w ξ t:t+N c ∥ 1 . (16) This pressures the memory dynamics to reconstruct coherent futures from partial cues, leading to task- relevant retrieval. Details are provided in the Ap- pendix. 3.4 Policy Policy generates E pose trajectoryˆx t:t+H ∈ R H×d raw from the Memory Readouth t (Section 3.3) for execution by low-level IK controllers. Single-state conditioning. We condition the trajec- tory generator only on a projected tokenc t =W ctx h t , where W ctx is a learned linear projection. Conditional rectified flow matching. Letx 1 ∈ R H×d raw be the ground-truth future trajectory and samplex 0 ∼N(0,I) andτ ∼U(0,1). We define the interpolantx τ = (1− τ)x 0 +τx 1 and target velocity u τ =x 1 − x 0 , and train a velocity networkv θ to predict u τ from (x τ ,τ,c t ): ˆu τ = v θ (x τ ,τ,c t ), L flow = E ∥ˆu τ − u τ ∥ 2 , (17) where the loss is masked to ignore padded steps when trajectories have variable effective length. At inference, we sampleˆx t:t+H by solving the ODE ̇x=v θ (x,τ,c t ) fromx(0) =x 0 with a fixed-step solver. 4 Dataset 4.1 Memory-intensive Tasks Formulation Intelligent agent decision-making arises from the in- teraction of goals, perception, memory, and prior knowledge, making fully memory-isolated tasks inher- ently unattainable. However, this interdependence opens a loophole that allows agents to exploit percep- tual variations and bypass explicit memory retrieval. To assess whether agents truly possess long-term memory reasoning abilities, we construct a set of memory-intensive tasks that enforce a perceptually non-Markovian decision phase, where agents are re- quired to make distinct decisions under identical vi- sual observation (as shown in Figure 3) Chung et al. (2025); Cherepanov et al. (2025). Building upon the biologically-inspired architecture formalized in Section 3, we develop Camo-Dataset, a comprehensive dataset designed to assess various dimensions of memory in robotic tasks. As shown in Table 1, we define three categories of memory tasks grounded in neuroscientific principles and aligned with prior robotics benchmarks Cherepanov et al. (2025). 4.2 Camo-Dataset We design three robotic manipulation tasks to com- prehensively evaluate three distinct memory abilities. 1.Clean a specified plate. This task simulates a kitchen scenario where a user puts a contami- nated plate (e.g., used for raw meat but visually clean) for the robot to clean. This task exempli- fies an event–object binding problem, requiring the agent to associate a specific event with an identical object. 2.Play shell game. This task simulates a shell game in which the user hides a cube under one cup and randomly swaps the cups. The robot must track the hidden cube through memory. This task represents spatial tracking problems, such as organizing containers or retrieving items from drawers. 3.Add various seasonings. This task simulates se- quentially adding three seasonings during cook- ing. After each addition, the robot must return the seasoning to its original position and move back to a preparation pose, making it impossible to infer whether a seasoning has already been added without recalling prior actions. This task represents counting or sequential execution prob- lems, such as making coffee or baking a cake. 5 Experiment Result 5.1 Experiment Setup Dataset. We evaluate Chameleon on Camo-Dataset under a consistent training and deployment pipeline with real-world robotic execution. For each task, we collect 120 teleoperated demonstrations using a UR5e robotic arm under human-in-the-loop control to ensure task execution quality. During the test phase, the trained model is deployed on the same physical UR5e robot, where it performs 36 indepen- dent real-world trials for each task category. We use the same decision setup and candidate sampling in the dataset and in real-robot evaluation, so the 7 Table 1 Three memory categories derived based on cognitive science, along with corresponding biological substrates and tasks in Camo-Dataset. CategoryCharacteristicsBiological CorrespondenceTask EpisodicIntroduces transient variables that disrupt habitual pat- terns. HC-centered episodic mem- ory. Clean a specified plate. SpatialMaintains and updates occluded object locations using environmental geometry. EC–HC cognitive map. Play shell game. SequentialTracks temporally ordered sub-goals to prevent repetition or omission. PFC-centered working mem- ory. Add various sea- sonings. Table 2 Performance (DSR (κ), MSR, SR, and CSR in %) across three task categories for two baselines, Chameleon, and its ablated models (36 tasks for each category). Best results are highlighted inblue, and second-best results are highlighted inpink. Module ∗ denotes the variant in which we replace our Memory module with Module. Method Episodic (p e = 1/3)Spatial (p e = 1/3)Sequential (p e = 1/27) DSR (κ) MSR SR CSR DSR (κ) MSR SR CSR DSR (κ) MSR SR CSR Diffusion Policy Chi et al. (2023)33.3 (0.0)91.7 30.6 30.0 34.3 (1.4)97.2 33.3 31.7 0.0 (-3.8) 13.9 0.0 6.7 Flow Matching Zhang and Gienger (2024) 30.0 (-5.0) 83.3 25.0 35.0 25.7 (-11.4)97.2 25.0 35.0 0.0 (-3.8) 5.6 0.0 8.3 ACT Zhao et al. (2023)28.0 (-8.0) 83.3 19.4 28.3 35.5 (3.2) 86.1 30.6 36.7 NA (NA) 0.0 0.0 10.0 Chameleon100.0 (100.0)86.186.1100.073.5 (60.3)94.469.491.772.2 (71.2)50.036.185.0 w/o Memory37.0 (5.6) 75.0 27.8 36.7 34.5 (1.7) 80.6 27.8 35.0 0.0 (-3.8) 8.3 0.0 6.7 Memory Bank*NA (NA) 0.0 0.0 33.3 NA (NA) 0.0 0.0 31.7 NA (NA) 0.0 0.0 11.7 Vanilla Mamba*28.0 (-8.0) 69.4 19.4 31.7 30.3 (-4.5) 91.7 27.8 28.3 0.0 (-3.8)44.4 0.0 8.3 w/o Dorsal Stream96.8 (95.2)86.183.3100.051.5 (27.3) 91.747.265.066.7 (65.4) 41.727.871.7 w/o HoloHead60.7 (41.1) 77.8 47.261.7 46.7 (20.0) 83.3 38.9 53.3 0.0 (-3.8) 33.3 0.0 13.3 Observation & Memory Observation & Memory Observation & Memory Observation & Memory Manipulation Decision Non-Markovian Stage Decision Non-Markovian Stage Clean a Specified Plate Play Shell Game Add Various Seasonings Decision Non-Markovian Stage Manipulation & MemoryManipulation & MemoryManipulation & Memory Manipulation Repeat three times Figure3 IllustrationoftaskexecutioninCamo-Dataset. Clean a specified plate: the agent remembers the user’s newly placed plate. At the decision stage, three plates are similar. Play shell game: the agent tracks the cup covering the cube. At the decision stage, the three cups appear identical. Add various seasonings: the agent recalls which spoon was picked up. At the decision stage, the three spoons are similar. chance-level success probability is the same in both settings. Baseline. We compare Chameleon against three state-of-the-art embodied agent models, Diffusion Policy Chi et al. (2023), Flow Matching Zhang and Gienger (2024), and ACT Zhao et al. (2023). We conduct ablation studies to analyze the contribution of key components by evaluating five variants. The parameter counts of all models are reported in the Appendix. Metrics. Beyond the standard Success Rate (SR), we introduce Decision Success Rate (DSR) and Ma- nipulation Success Rate (MSR) to provide finer- grained evaluation of memory-intensive tasks, mea- suring retrieval accuracy and execution success, re- spectively. These metrics satisfySR=DSR× MSR. Since DSR is subject to chance-level performance, we adopt the chance-adjusted Cohen’s kappa coef- ficient Cohen (1960) to reflect memory-dependent decision success rate: κ = p o − p e 1− p e ,(18) where p o is DSR and p e is the chance-level DSR. Implementation Details. For each task, we train using AdamW with learning rate 1×10 −4 , batch size 4, cosine annealing, and bfloat16 mixed precision. Training runs for 20K steps on a single 4090 GPU. For inference, trajectories are generated using 50 rectified-flow ODE steps. The average end-to-end inference latency is 82 ms per control step, enabling real-time manipulation on the UR5e platform. All architectural hyperparameters, configurations, and training details are further provided in the Appendix. 8 Clean a Specified Plate Play Shell Game 1. Decision 2. Pick Green Spoon 3.Put Green Spoon 4. Decision 5. Pick Red Spoon 6. Put Red Spoon 7. Decision 8. Pick Yellow Spoon 9. Put Yellow Spoon L -LeftCup M - Middle Cup R - Right Cup Add Various Seasonings Figure 4 Pattern separation in the decision state. UMAP projections ofh t for all three task categories, colored by the latent decision target. Clear clustering under visually aliasing indicates thath t disambiguates the history-dependent state. 5.2 Comprehensive Systematic Study Chameleon achieves near-perfect decision success rate in episodic recall. As shown in Table 2, Chameleon at- tains 100.0% (100.0%) in DSR (κ), 86.1% in SR, and 86.1% in MSR, consistently outperforming baselines on memory-driven decision. Compared to Diffusion Policy, a representative baseline, it improves DSR (κ) by 66.7% (100%) and SR by 55.5%, while exhibiting a marginal decrease of 5.6% in MSR. These results highlight Chameleon ’s superior episodic representa- tion and task-relevant retrieval capability. Chameleon maintains strong spatial tracking perfor- mance under memory constraints. It achieves 73.5% (60.3%), 94.4%, and 69.4% in DSR (κ), MSR, and SR, respectively, while Diffusion Policy underper- forms by 39.2% (58.9%) and 36.1% in DSR (κ) and SR, with a slight increase of 2.8% in MSR. These results indicate that Chameleon maintains stable spa- tial tracking across three object rearrangements by selectively recalling relevant past experiences. Chameleon further sustains an advantage in sequential tasks that require consistent decisions across multi- ple stages. In this task, Chameleon achieves 72.2% (71.2%), 50.0%, and 36.1% in DSR (κ), MSR, and SR, respectively, while Diffusion Policy degrades sub- stantially across all three metrics by 72.2% (75%), 36.1%, and 36.1%, respectively. This suggests that structured memory helps maintain task progression over long sequences and can better preserve execution accuracy in multi-step and action-intensive tasks. 5.3 Ablation Study We ablate core components of Chameleon under the same training and real-robot evaluation protocol as Table 2. Implementation details are provided in Ap- pendix. In addition to the metrics above, we report the classification success rate (CSR), the precision of Hand Camera View Ventral (DINO only)Ventral + Dorsal Hand Camera View Ventral (DINO only)Ventral + Dorsal Figure 5 Dorsal stream enables write-time disambiguation. Cross-view attention in the spatial task: with the dor- sal stream, attention focuses on the correct target via end-effector geometry and epipolar feasibility; without it, attention diffuses across distractors, degrading down- stream memory and execution. a linear probe trained on the frozen decision-stage stateh t to predict the latent decision target under aliasing. Continuous and structured memory is necessary for non-Markovian decision-making. The reactive variant w/o Memory collapses, confirming that the bench- mark is non-Markovian at the observation level. A Memory Bank with similarity-based retrieval remains near chance-level on CSR, indicating that discrete retrieval is unreliable under aliasing because visu- ally plausible recalls can be decision-irrelevant. In contrast, Chameleon integrates history through con- tinuous state dynamics and consistently improves DSR and CSR across all task categories (Table 2). These gains are not attributable to recurrence alone. Vanilla Mamba degrades most on sequential tasks, suggesting long-horizon temporal interference with- out our anchor-structured, multi-timescale episodic state, which reduces interference and improves both decision reliability and end-to-end success. Write-time spatial disambiguation is necessary for ro- bust tracking. Ablating the dorsal geometry pathway (w/o Dorsal Stream) substantially reduces MSR and SR on the spatial category (Table 2). As visualized in Figure 5, the full model concentrates cross-view attention onto the physically correct target using end- 9 effector anchored geometry and epipolar feasibility, whereas removing the dorsal stream yields diffuse attention across distractors, weakening the evidence written into memory. HoloHead stabilizes the decision state representation. The w/o HoloHead variant reduces DSR and CSR de- spite comparable capacity (Table 2), indicating that the latent imagination objective is essential for keep- ingh t from collapsing to instantaneous appearance and for preserving a predictive, decision-sufficient representation under aliasing. 5.4 Qualitative Mechanistic Validation To verify whether the decision stateh t achieves the intended design of event-biased separation, and goal-directed recall (Section 3), we project high- dimensional statesh t into a low-dimensional mani- fold with UMAP, and probe its generative behavioral readout via HoloHead’s predictive rollouts. Pattern separation. As shown in Figure 4, in episodic and spatial tasks, the introduction of memory in- formation causesh t to separate from overlapping clusters into distinct manifolds corresponding to dif- ferent latent states. Importantly, these manifolds remain stable even after the disambiguating visual evidence disappears, indicating that the representa- tion preserves event- and location-level distinctions. In sequential tasks,h t instead evolves along different stages of a shared manifold as the task progresses, reflecting the maintenance of temporal process infor- mation. Pattern completion. Pattern completion refers to recovering a task-relevant latent state inh t from partial cues. Since the latent state is not directly observable, we assess it through predictive rollouts generated fromh t . Ifh t captures the task-relevant latent state, the predicted trajectory should remain consistent with the intended target behavior. As shown in Figure 6, rollouts from randomly selected timesteps across three tasks remain goal-consistent, Clean a specific plate Play shell gameAdd various seasonings Figure 6 Pattern completion via predictive rollouts. Holo- Head waypoint predictions at the decision stage remain consistent with the latent target under ambiguity, indi- cating that h t supports completion from partial cues. supporting thath t encodes task-relevant state for pattern completion. 6 Conclusion We study memory-intensive robot manipulation un- der perceptual aliasing, where decision-time observa- tions are insufficient and history is required, yield- ing non-Markovian decision points in observation level. Inspired by human episodic memory, we pro- pose Chameleon, a differentiable memory architecture that couples disambiguating memory writing with goal-directed recall. On our proposed Camo-Dataset, real-robot experiments and representation analyses show improved robustness and long-horizon control. Limitations and Future Work. Chameleon focuses on learnable episodic memory for perception-driven manipulation and does not incorporate the semantic abstractions used by many Vision–Language–Action (VLAs) for broad zero-shot generalization. Our ex- periments use a single robot platform, leaving cross- embodiment transfer and learning event segmentation as future work. An important direction is to integrate the episodic memory stack with foundation models to combine structural encoding with higher-level priors for open-world autonomy. 10 References Timothy A Allen and Norbert J Fortin. The evolution of episodic memory. Proceedings of the National Academy of Sciences, 110(supplement_2):10379–10386, 2013. Tarek Amer and Lila Davachi. Extra-hippocampal contri- butions to pattern separation. elife, 12:e82250, 2023. Abrar Anwar, John Welsh, Joydeep Biswas, Soha Pouya, and Yan Chang. Remembr: Building and reasoning over long-horizon spatio-temporal memory for robot navigation. In 2025 IEEE International Conference on Robotics and Automation (ICRA), pages 2838–2845. IEEE, 2025. Arnold Bakker, C Brock Kirwan, Michael Miller, and Craig EL Stark. Pattern separation in the human hippocampal ca3 and dentate gyrus. science, 319(5870): 1640–1642, 2008. Sebastian Borgeaud, Arthur Mensch, Jordan Hoff- mann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. Improving lan- guage models by retrieving from trillions of tokens. In International conference on machine learning, pages 2206–2240. PMLR, 2022. Egor Cherepanov, Nikita Kachaev, Alexey K Kovalev, and Aleksandr I Panov. Memory, benchmark & robots: A benchmark for solving complex tasks with reinforce- ment learning. arXiv preprint arXiv:2502.10550, 2025. Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Dif- fusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems (RSS), 2023. Nhat Chung, Taisei Hanyu, Toan Nguyen, Huy Le, Fred- erick Bumgarner, Duy Minh Ho Nguyen, Khoa Vo, Kashu Yamazaki, Chase Rainwater, Tung Kieu, et al. Rethinking progression of memory state in robotic manipulation: An object-centric perspective. arXiv preprint arXiv:2511.11478, 2025. Jacob Cohen. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20 (1):37–46, 1960. Bradford C Dickerson and Howard Eichenbaum. The episodic memory system: neurocircuitry and disorders. Neuropsychopharmacology, 35(1):86–104, 2010. Jinyuan Fang, Zaiqiao Meng, and Craig Macdonald. Trace the evidence: Constructing knowledge-grounded rea- soning chains for retrieval-augmented generation. In Findings of the Association for Computational Linguis- tics: EMNLP 2024, pages 8472–8494, 2024. YukTungSamuel Fang, Zhikang Shi, Jiabin Qiu, Zix- uan Chen, Jieqi Shi, Hao Xu, Jing Huo, and Yang Gao. Inherit-sg: Incremental hierarchical semantic scene graphs with rag-style retrieval. arXiv preprint arXiv:2602.12971, 2026. Pascale Fung, Yoram Bachrach, Asli Celikyilmaz, Ka- malika Chaudhuri, Delong Chen, Willy Chung, Em- manuel Dupoux, Hongyu Gong, Hervé Jégou, Alessan- dro Lazaric, et al. Embodied ai agents: Modeling the world. arXiv preprint arXiv:2506.22355, 2025. Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. Precise zero-shot dense retrieval without relevance la- bels. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1762–1777, 2023. Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. In First conference on language modeling, 2024. Bernal Jimenez Gutierrez, Yiheng Shu, Yu Gu, Michihiro Yasunaga, and Yu Su. Hipporag: Neurobiologically inspired long-term memory for large language models. Advances in Neural Information Processing Systems, 37:59532–59569, 2024. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 confer- ence on empirical methods in natural language process- ing (EMNLP), pages 6769–6781, 2020. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge- intensive nlp tasks. Advances in neural information processing systems, 33:9459–9474, 2020. Min Lin, Xiwen Liang, Bingqian Lin, Liu Jingzhi, Zi- jian Jiao, Kehan Li, Yuhan Ma, Yuecheng Liu, Shen Zhao, Yuzheng Zhuang, et al. Echovla: Robotic vision-language-action model with synergistic declara- tive memory for mobile manipulation. arXiv preprint arXiv:2511.18112, 2025. Peiran Liu, Qiang Zhang, Daojie Peng, Lingfeng Zhang, Yihao Qin, Hang Zhou, Jun Ma, Renjing Xu, and Yiding Ji. Toponav: Topological graphs as a key en- abler for advanced object navigation. arXiv preprint arXiv:2509.01364, 2025. Yang Liu, Xinshuai Song, Kaixuan Jiang, Weixing Chen, Jingzhou Luo, Guanbin Li, and Liang Lin. Meia: Mul- timodal embodied perception and interaction in un- known environments. arXiv preprint arXiv:2402.00290, 2024. Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Pro- ceedings of the 61st annual meeting of the association 11 for computational linguistics (volume 1: Long papers), pages 9802–9822, 2023. Ruaridh Mon-Williams, Gen Li, Ran Long, Wenqian Du, and Christopher G Lucas. Embodied large language models enable robots to complete complex tasks in un- predictable environments. Nature Machine Intelligence, 7(4):592–601, 2025. Chi T Ngo, Sebastian Michelmann, Ingrid R Olson, and Nora S Newcombe. Pattern separation and pattern completion: Behaviorally separable processes? Mem- ory & Cognition, 49(1):193–205, 2021. Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Mered- ith Ringel Morris, Percy Liang, and Michael S Bern- stein. Generative agents: Interactive simulacra of hu- man behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology, pages 1–22, 2023. Larry R Squire. Memory and the hippocampus: a syn- thesis from findings with rats, monkeys, and humans. Psychological review, 99(2):195, 1992. Weihang Su, Yichen Tang, Qingyao Ai, Junxi Yan, Changyue Wang, Hongning Wang, Ziyi Ye, Yujia Zhou, and Yiqun Liu. Parametric retrieval augmented gener- ation. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1240–1250, 2025. Timothy J Teyler and Pascal DiScenna. The hippocampal memory indexing theory. Behavioral neuroscience, 100 (2):147, 1986. Yuwei Wan, Zheyuan Chen, Ying Liu, Chong Chen, and Michael Packianather. Empowering llms by hybrid retrieval-augmented generation for domain-centric q&a in smart manufacturing. Advanced Engineering Infor- matics, 65:103212, 2025. Zixuan Wang, Bo Yu, Junzhe Zhao, Wenhao Sun, Sai Hou, Shuai Liang, Xing Hu, Yinhe Han, and Yiming Gan. Karma: Augmenting embodied ai agents with long-and-short term memory systems. In 2025 IEEE International Conference on Robotics and Automation (ICRA), pages 1–8. IEEE, 2025. Ella Worsfold, Nicola S Clayton, and Lucy G Cheke. Revisiting episodic-like memory in scrub jays: Is there more we can still learn from what–where–when caching behaviour? Learning & Behavior, 53(1):65–79, 2025. Quanting Xie, So Yeon Min, Pengliang Ji, Yue Yang, Tianyi Zhang, Kedi Xu, Aarav Bajaj, Ruslan Salakhut- dinov, Matthew Johnson-Roberson, and Yonatan Bisk. Embodied-rag: General non-parametric embodied memory for retrieval and generation. arXiv preprint arXiv:2409.18313, 2024. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The eleventh international conference on learning representations, 2022. Fan Zhang and Michael Gienger. Affordance-based robot manipulation with flow matching. arXiv preprint arXiv:2409.01083, 2024. Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. In ICML Workshop on New Frontiers in Learning, Control, and Dynamical Systems, 2023. Yicong Zheng, Nora Wolf, Charan Ranganath, Ran- dall C O’Reilly, and Kevin L McKee. Flexible prefrontal control over hippocampal episodic mem- ory for goal-directed generalization. arXiv preprint arXiv:2503.02303, 2025. Yichen Zhu, Zhicai Ou, Xiaofeng Mou, and Jian Tang. Retrieval-augmented embodied agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17985–17995, 2024. 12 Supplementary Material Chameleon: Episodic Memory for Long-Horizon Robotic Manipulation Table of Contents Within this supplementary material, we elaborate on the following aspects: • Appendix A: Additional Method Details – A.1: Overview and Problem Setup – A.2: Perception – A.3: Memory – A.4: Policy • Appendix B: Additional Experimental Details – B.1: Camo-Dataset Statistics – B.2: Implementation Details – B.3: Model Size and Computational Cost of Experiment – B.4: Evaluation Metrics Definitions – B.5: Failure Case Analysis • Appendix C: Human Episodic Memory and Correspondence to Chameleon – C.1: Overview of the EC–HC–PFC Episodic Memory Circuit – C.2: Perceptual Pathways: Ventral and Dorsal Streams – C.3: Entorhinal Cortex: Interface Between Perception and Memory – C.4: Hippocampus: Pattern Separation and Pattern Completion – C.5: Hippocampus–Prefrontal Cortex Interaction 13 A Additional Method Details A.1 Overview and Problem Setup Problem setup. We study long-horizon robotic ma- nipulation under partial observability, where correct control depends on information that may be occluded, transient, or only established by earlier interactions. At each timestept, the agent receives a multimodal observationo t =I f t , I h t , s t , ψ t ,whereI f t and I h t are RGB images from a fixed front camera and a hand-mounted side camera,s t denotes the robot proprioceptive state (including end-effector pose and gripper state), andψ t is an optional task-stage signal (e.g., a phase ID). Instead of predicting a single-step command, we parameterize decision making by gen- erating a future end-effector pose trajectory over a horizonH(Section 3.4). During training, we super- vise the model with ground-truth future pose states x t:t+H ∈ R H×d raw ; during inference, the policy sam- plesˆx t:t+H from a conditional generative model and executes it with standard low-level controllers. This trajectory-level target makes long-horizon structure explicit and mitigates compounding errors typical of step-wise prediction. Core challenge. Long-horizon manipulation con- tains decision points where instantaneous percep- tion is insufficient: the agent must act differently under nearly identical sensory inputs because the disambiguating variables are historical, occluded, or fleeting. Equivalently, there exist timesteps where optimal control cannot be expressed as a function ofo t alone, making the problem non-Markovian in observation space and requiring an internal state that carries missing context forward. We therefore cen- ter our design on a decision-sufficient memory state h t that serves as the sole interface between percep- tion and control. Importantly,h t is not a passive summary; it is a recurrent control state that fuses (i) past traces needed to resolve hidden causes, (i) present evidence needed to ground the current situa- tion, and (i) future constraints that keep the state predictively coherent for near-term evolution. In our framework, retention and selection are realized implic- itly through continuous state dynamics and a single task-adaptive readout, avoiding discrete database- style retrieval. To ensure thath t remains actionable rather than merely compressive, we further regularize it with imagination-style rollout supervision, which pressures the memory state to support internal roll- outs consistent with the agent’s subsequent behavior. Overview. As illustrated in Figure 7, our method in- stantiates a closed-loop pipeline with clean interfaces, Perception → Memory → Policy : x t = Φ perc (I f t ,I h t ,s t ,ψ t ),(19) h t = Φ mem (x t ,h t−1 ),(20) ˆx t:t+H = Φ pol (h t ),(21) where perception constructs geometry-grounded, view-consistent patch tokensx t to reduce ambigu- ity before memory formation (Section 3.2), memory transforms multimodal evidence into a compact re- current state and outputs a single readouth t shaped by rollout supervision (Section 3.3), and policy gen- erates the future pose trajectory via conditional flow matching conditioned only onh t (Section 3.4). This single-readout interface turns sufficiency into an ar- chitectural constraint: if the agent succeeds in long- horizon control, it must be mediated through the informativeness and stability of the learned memory state. At a high level, the resulting organization is bio-inspired in its division of labor across perception, memory, and control, while remaining fully differen- tiable and end-to-end trainable. A.2 Perception Long-horizon manipulation requires memory to dis- ambiguate situations that can look similar at a glance. This is only possible if perception exposes localized evidence, because a single global summary tends to discard fine-grained cues (object parts, boundaries, contact regions) that later resolve ambiguity. We therefore provide memory with a patch-level evidence set from two complementary viewpoints: a fixed front camera captures stable global context, while a hand- mounted camera reveals close-range details during contact-rich interactions. The central difficulty is correspondence: the same physical region can shift substantially in the image plane and undergo view- dependent occlusion. If the two streams are written into memory without alignment, temporal aggrega- tion can mix incompatible evidence. Our key idea is to use the end-effector (E) as an action-centric geo- metric anchor. The E pose is directly known in 3D from proprioception and can be projected into each view, providing a reliable 2D–3D bridge to ground cross-view correspondence and to emphasize control- relevant regions. This design follows a bio-inspired dorsal–ventral division of labor: a ventral pathway provides rich appearance evidence (what), while a dorsal pathway injects action-centric spatial structure (where/how) before memory integration. Inputs and output. At timestept, perception re- ceives two RGB imagesI f t (front) andI h t (hand), robot states t (including E pose and gripper state), 14 x y z x y z x y z Proprioception Hand Cam Pose Front Cam Pose Camera (K,T) Camera (K,T) E State DORSAL STREAM VENTRAL STREAM 3D - to - 2D Projection E_3d Ray Dir (3D) Norm Coords (2D) Cos Ang (1D) Proximity to E (1D) Camera_Center 7D Geometric Feature Front RGB Hand RGB [B,T,224,224,3] [B,T,224,224,3] G front G hand (Patch-wise) ... ... ... ... ... ... [B,T,N,7] [B,T,N,7] Compactor FiLM MULTIMODAL FUSION MEMORY LAYERS Neuro-Symbolic Mathematical Abstraction 2D vs Hex Square Grid (b)Memory Geo Bias Generation Bias MLP Bgeo Btotal Bepi (R, t) P C_1 Epipolar Geometry Bias GEO INPUTS ... ... ... ... ... ... Gfront_FiLM Ghand_FiLM ... ... ... ... ... ... x y z x y z Q KV ( ) Q K V ( ) Importance Modulation [B,T,2N,D] Bidirectional Cross Attention GEO-BIASED ATTENTION C_0 Gfront_comp Ghand_comp (a)Perception RGB INPUTS DINO (Frozen) LayerNorm ... ... ... ... ... ... Vhand [B,T,N,D] Vfront QKVhand Q K V Q K V [B,T,N,D] QKVfront Vfused ... ... ... ... ... ... ... ... ... ... ... ... [B,T,1,D’] Entorhinal Cortex Grid Cell Firing 2-Wave Inference Hex Grid 3-Wave Inference Proprio Encoder Phase 1 - “Observe” Phase 2 – “Find” Vfused ... ... ... Linear Proj + LN Phase Encoder View Embedding Efront Ehand Linear Proj + LN Linear Proj + LN Ehex Modality Embedding EproprioEphase Evisual Hexagonal Grid Embedding Fproprio [B,T,1,D’] Fphase [B,T,2N,D’] Fvisual Concat + LN ( , , ) Proprioception E State x y z Symmetric Spatialtemporal Memory LAYER L (Detail) Episodic Mem ... ... ... ... F st ... ... 풅 풕 ퟎ ,푺 ퟎ 풉 풕 풉 풕+ퟏ 풅 풕 ퟏ ,푺 ퟎ 풉 풕 풉 풕+ퟏ 풅 풕 ퟎ ,푺 ퟏ 풉 풕 풉 풕+ퟏ 풅 풕 풊 ,푺 풋 풉 풕 풉 풕+ퟏ ... Proj Proj Proj Proj 푽 ퟎ , 푽 ퟏ , ... , 푽 풎 푺풑풂풕풊풂풍,푻풊풎풆,푽풊풆풘 푺풍풐풕풔 Slot Aggr 푯 풆풑풊풔풐풅풊풄 풍 ... ... ... Linear Proj Working Mem 푯 풘풐풓풌풊풏품 풍 [B,T,D’] 푯 풘풐풓풌풊풏품 풍−ퟏ [B,T,D’] UPDATE Linear Proj + LN ... ... ... [B,T,K,D’] Fperc 퐊∈ퟐ푵+ퟐ푵+ퟏ,ퟐ푵+ퟐ CTX Mod DIVIDE Optional! Optional! HIERARCHICAL MEMORY FUSION 푯 풘풐풓풌풊풏품 풍 푯 풘풐풓풌풊풏품 ퟎ 푯 풘풐풓풌풊풏품 ퟏ ... ... ... ... ... ... ... ... ... [B,T,D’][B,T,D’][B,T,D’] Layer Proj Layer Proj Layer Proj Stack + Reshape ... ... ... [B,T,L,D’] 푯 풘풐풓풌풊풏품 KVworking Q K V Learnable Task Query Qtask Multihead Attention ... ... ... Softmax 훂∈[ퟎ,ퟏ] 푩×푻×푳 Weighted Sum 풍 휶 풍 푬 풍 Linear Proj + LN 푯 풑풐풍풊풄풚 ... ... ... [B,T,D’] MEMORY ROLLOUT ... ... ... [B,T,D’] 푯 풑풐풍풊풄풚 Observe Think Infer Imagine 2D Waypoint Predictor 3 D Waypoint Predictor Na short-term points Nc long-term points At timestep t Predict Na + Nc future points (c)Action Velocity Network Time + Proprio Embed DiT Blocks × N AdaLN (t) Self - Attention Cross - Attention MLP Output Head POLICY HEAD ODE Solver Iteration Loop 풕∈ퟎ,ퟏ,푲 푺풕풆풑풔 Update State 풙 풕 풊+ퟏ ←풙 풕 풊 +⋯ ෞ 풙 ퟏ (Pred) Initial Noise 풙 ퟎ ~푵(ퟎ,푰) 풙 풕 풊 ,풕 풊 푽 풑풓풆풅 ... ... ... Cperc [B,T,S] Linear Projection ... ... ... 풙ퟏ (GTproprio) 풙ퟎ (Noise) 풕 (Time) 풕∈[ퟎ,ퟏ] Conditional Flow Matching Process 푥 푡 =푡∙푥 1 +(1−푡)∙푥 0 푢 푡 =푥 1 −푥 0 (푇푎푟푔푒푡) 풙 풕 ,풕 푽 풑풓풆풅 Flow Matching Loss (MSE) 풖 풕 Training Inference ... ... ... Optional! ... Proj Proj Proj Proj ... Query Query Query Query Attention Figure 7 Method overview. Our system follows a closed-loop perception–memory–policy pipeline for long-horizon manipulation. (a) Perception: from front/hand RGB, we construct geometry-grounded, view-consistent patch tokens that reduce ambiguity before memory formation. (b) Memory: multimodal evidence is transformed into a compact recurrent decision state via iterative updates and a single learned readout, further shaped by imagination-style rollout supervision. (c) Policy: conditioned only on this memory readout, a conditional flow-matching generator produces the future end-effector pose trajectory for execution. and an optional stage signalψ t . It outputs a token set x t ∈ R N×d ,(22) which serves as the visual evidence stream for the memory module. Ventral stream: appearance evidence from a frozen DINO encoder. For each viewv ∈f,h, we extract patch-level appearance tokens using a frozen DINO backbone: V v t = DINO(I v t )∈ R N v ×d ,(23) whereN v is the number of patch tokens for viewv. This preserves detailed local evidence while leveraging strong pretrained visual priors. Dorsal stream: E-anchored geometric features. Let K v andT v t ∈ SE (3) denote the camera intrinsics and world-to-camera extrinsics for viewv(withT h t varying with the E for the hand-mounted camera). Froms t we obtain the E positionp t ∈ R 3 in the world frame and compute its projection in each view: u E,v t = Π(K v ,T v t ,p t )∈ R 2 .(24) For each patchiin viewv, letu v t,i ∈ R 2 be the normalized image coordinate of the patch center and ̃u v t,i ∈ P 2 its homogeneous coordinate. We define the corresponding unit ray direction in the camera frame by r v t,i = (K v ) −1 ̃u v t,i ∥(K v ) −1 ̃u v t,i ∥ 2 ∈ R 3 ,(25) Following Figure 7, we form a raw action-centric descriptor g v t,i = u v t,i , r v t,i , ρ v t,i , cosθ v t,i ∈ R 7 , (26) whereρ v t,i =∥u v t,i − u E,v t ∥ 2 measures proximity to the E projection (action relevance in the image), andcosθ v t,i captures viewing geometry between the patch ray and the E direction induced by (p t ,T v t ). A geometric encoder produces compact conditioning featuresC v t ∈ R N v ×d c for modulation and patch-wise raw descriptors G v t ∈ R N v ×7 for bias prediction. 15 Geometry-biased bi-directional cross-view enhance- ment. We do not collapse the two views into a single representation. Instead, we enhance each view with geometrically compatible evidence from the other view through residual, bi-directional cross-attention. For attention from viewato viewb(witha ̸=b), define projected queries/keys/values Q a t = V a t W Q , K b t = V b t W K , U b t = V b t W V , (27) and perform the residual update ̄ V a t = V a t + Attn a→b (Q a t ,K b t ,U b t ).(28) The attention logits combine an analytic pairwise feasibility bias with a learned unary geometric bias: Attn a→b (Q,K,U ) = softmax QK ⊤ √ d + B epi a→b +b a t 1 ⊤ + 1(b b t ) ⊤ U. (29) whereb v t ∈ R N v stacks per-patch geometric biases b v t,i and 1 is the all-ones vector. Epipolar feasibility bias.B epi a→b (i,j) encodes whether two patches can correspond under the relative camera geometry induced by (K a ,T a t ,K b ,T b t ). LetF a→b be the corresponding fundamental matrix. For homo- geneous patch centers ̃u a t,i and ̃u b t,j , we compute the point-to-epipolar-line distance d ij = ̃u b⊤ t,j F a→b ̃u a t,i 2 (F a→b ̃u a t,i ) 1:2 2 2 + ε , B epi a→b (i,j) =− d ij τ , (30) whereε >0 avoids division by zero andτis a tem- perature. Learned unary geometric bias. Analytic feasibility alone does not determine which epipolar-consistent candidate is most informative for control. We there- fore predict a per-patch geometric bias with a small network (Bias-MLP) from the patch-wise geometric descriptor: b v t,i = MLP(G v t,i ),(31) which captures action relevance and view-specific reli- ability (e.g., proximity to the E projection and geometry-induced saliency). The broadcast form b a t 1 ⊤ + 1(b b t ) ⊤ lets this unary signal modulate cross- view communication without turning it into a sepa- rate pairwise matcher. Geometry-conditioned feature modulation and evi- dence set. Geometry should influence not only which patches communicate but also how strongly their fea- tures should be emphasized for manipulation. We therefore condition feature modulation on compact geometric codes via a FiLM-style operator: ̃ V v t (i) = γ(C v t,i )⊙ ̄ V v t (i) +β(C v t,i ), v ∈f,h, (32) whereγ(·) andβ(·) are learned functions. Finally, we form the output evidence set by concatenating the enhanced tokens from both views: x t = Concat ̃ V f t , ̃ V h t ∈ R N×d .(33) This design preserves detailed appearance evidence (ventral stream), constrains cross-view message pass- ing using an epipolar feasibility prior, refines it with learned unary geometric gating from the action- centric anchor (dorsal stream), and applies geometry- conditioned modulation to emphasize control-relevant cues without over-fusing viewpoint-specific informa- tion. The resultingx t provides the memory module with a stable and information-rich evidence stream for constructing a compact decision state under long- horizon partial observability (Section 3.3). A.3 Memory The memory module maps partial observations to an actionable internal state. At each timestept, it receives the perceptual evidence setx t (a patch-token stream from Section 3.2) and outputs a single decision stateh t , which is the only memory signal exposed to the policy. This interface is intentional: instead of querying an external memory bank, long-horizon context is stored, selected, and expressed through continuous, end-to-end differentiable dynamics. Our design follows a bio-inspired division of labor. A slow episodic substrate retains spatiotemporal traces with reduced interference, while a fast working state main- tains the currently relevant control context; episodic information influences behavior only by being injected into the working update, and only the fused working state is emitted as h t . Multimodal Fusion. We represent each input source as a set of tokens and fuse them into a single evidence set while preserving token identity. Letx f t ∈ R N f ×d andx h t ∈ R N h ×d denote the visual patch tokens from the front and hand views, respectively, and letp t and φ t denote, when available, a proprioception token and a lightweight phase token. Each token receives a modality embedding, while visual tokens additionally receive a view embedding: ̄x v t = x v t + E mod vis + E view v , v ∈f,h, (34) ̄p t = p t + E mod prop ,(35) ̄φ t = φ t + E mod phase + E phase (ψ t ).(36) 16 We then concatenate all available tokens and apply layer normalization: z t = LN Concat[ ̄x f t , ̄x h t , ̄p t , ̄φ t ] ∈ R M×d , (37) whereMis the total number of fused tokens at timet. This produces a unified multimodal evidence set that can be processed jointly by the downstream memory module. We also explored an optional hexagonal positional embedding for visual tokens, ̄x v t = x v t + E mod vis + E view v + E hex (u v t ), v ∈f,h, (38) whereu v t indexes the patch-center coordinates in viewv. This design was motivated by grid-cell- inspired spatial coding: compared with standard square-lattice 2D positional embeddings, a hexag- onal lattice has a more isotropic local neighborhood and can provide a smoother discretization of curved physical boundaries. However, in our empirical eval- uation, E hex did not yield a significant performance gain. Therefore, to emphasize the core architecture and avoid unnecessary complexity, all experiments in the main paper use the simplified fusion scheme without E hex . Dynamic context modulation. The role of Equa- tion(7)is not merely additive conditioning, but layer-specific reinterpretation of the same multimodal evidence under a progressively refined working con- text. Since the projected context termW (ℓ) c h (ℓ−1) t is broadcast to allMtokens, token identity is preserved while the entire token set is biased toward the current working hypothesis carried by layerℓ−1. This makes the subsequent anchor formation content-adaptive across layers without requiring explicit token rewrit- ing or recurrent token-token interaction. In other words, different layers observe the same evidence set z t , but under different working contexts, leading to different spatial-temporal decompositions and mem- ory updates. Spatial and Temporal Anchors. The spatial- temporal anchor construction serves as the structured interface between the multimodal evidence and the downstream slot-wise memory dynamics. In Equa- tion(8), the routing weightsπ (ℓ) t,a,i define how the vi- sual evidence is organized intoAanchor-conditioned components. The resulting quantityu (ℓ) t,a should there- fore be understood as the anchor-level visual evidence emphasized by spatial indexa, rather than as an isolated modality-specific descriptor. By attaching the shared proprioceptive and phase tokens to every anchor, each anchor is interpreted jointly through local visual evidence and global task state, yielding a compact anchor-conditioned token set ̃z (ℓ,a) t . Equa- tion(9)then expands each anchor intoBtemporal modes. The learned temporal queryq (ℓ) a,b performs a slot-specific readout from ̃z (ℓ,a) t and produces a single latent featuref (ℓ) t,a,b for slot (a,b). Thus, before enter- ing the state-space dynamics, the evidence relevant to anchoraand temporal modebis already condensed into a slot-level latent state. This is a key distinc- tion of our design: the model does not feed the full token collection directly into a monolithic recurrent memory, but instead first reorganizes the evidence into anA×Bmatrix of compact spatiotemporal slot latents. The spatial indexadetermines which com- ponent of the visual evidence is emphasized, while the temporal indexbdetermines which characteristic memory timescale is used to process that evidence. As a result, the slot matrixf (ℓ) t,a,b provides an ex- plicit factorization of memory addressing along both space and time. Selective state-space parameterization and temporal adaptation. Given slot featuref (ℓ) t,a,b , the projection Proj (ℓ) a,b produces the content-dependent parameters that govern the episodic state-space update in Equa- tion(10). As in selective state-space models, these parameters include input- and readout-dependent factors together with a step-size term, so that the effective dynamics of each slot depend on both its cur- rent content and its temporal identity. Accordingly, our temporal organization is not a naive replication of identical SSMs across the A× B slots. A subset of temporal slots is initialized with prede- fined base step sizes ∆t (0) b spanning short to long half-lives, providing an explicit multi-timescale prior. For these prior-guided slots, the effective step size is further modulated by the slot content, so the dynam- ics can still contract or expand around the nominal half-life implied by ∆t (0) b . The remaining slots do not receive such predefined temporal anchors, and instead learn their effective timescales more freely through the data-dependent parameterization produced by Proj (ℓ) a,b . The temporal axis of the slot matrix there- fore combines interpretable half-life-guided modes with more flexible learned modes, yielding a struc- tured yet adaptive memory hierarchy rather than a bank of identical recurrent channels that differ only through input gating. For efficiency, we implement the slot-wise state-space recursion with a dedicated fused scan kernel that updates allA× Bepisodic states in a single pass. This allows the model to re- tain explicit spatial-temporal slot structure without requiring a separate sequential scan for each slot. 17 Episodic memory as latent slot-indexed engrams. The episodic state update in Equation(11)is ap- plied independently to each slot (a,b), producing a bank of latent statesm (ℓ) t,a,b . These states act as hidden engrams indexed jointly by spatial compo- nent and temporal mode: spatial indexadetermines which anchor-conditioned evidence is stored, while temporal indexbdetermines the characteristic reten- tion dynamics under which it evolves. The readout r (ℓ) t,a,b exposes only a projected expression of each en- gram, rather than the state itself. Hence, the memory remains structurally organized but operationally im- plicit: information is written into the engram through the selective state-space recursion, persists there un- der slot-specific dynamics, and becomes behaviorally relevant only after readout and aggregation. This slot-indexed episodic organization is another distinctive property of our architecture. Rather than maintaining a single undifferentiated hidden state, or relying on explicit key-value retrieval over an exter- nal memory store, our model maintains a structured internal memory bank whose indices are built di- rectly from anchor-conditioned perceptual evidence and temporal mode. This yields a notion of pattern separation inside the recurrent memory itself: dif- ferent spatial-temporal components are assigned to distinct latent trajectories before recall is formed. Working-state integration and asymmetric memory exposure. The episodic recall vector in Equation(12) aggregates the slot readouts into a working-space rep- resentation, which is then injected into Working Mem- ory through Equation(13). The asymmetry between Episodic Memory and Working Memory is deliber- ate. Episodic states are slot-indexed, persistent, and latent; Working Memory is compact, exposed, and serves as the only inter-layer communication chan- nel. Thus, the model separates storage from exposure: rich memory traces are preserved in the episodic bank, but only a compressed working summary is propa- gated upward. This differs from standard recurrent designs in which the same state must simultaneously store long-horizon information, support immediate computation, and act as the layer output. Because Working Memory receives episodic recall only after slot-level organization and aggregation, it operates as a dynamic interpretive state rather than a raw storage tensor. The resultingh (ℓ) t summarizes what the layer currently “believes” is task-relevant after querying its structured episodic bank. This design is especially important for compositional ma- nipulation, where multiple partial cues may need to be stored on different timescales but only a small subset should influence the next decision. Working readout and hierarchical memory fusion. Equation(14)performs task-conditioned fusion over the layer-wise working readouts. Since each layer receives a different contextualized evidence set and maintains its own episodic-working interaction pat- tern, the readouty (ℓ) t from layerℓcaptures a distinct level of abstraction and temporal integration. The hierarchical fusion therefore does not simply average layers. Instead, the task queryq task selects which working summaries are most relevant for downstream control at timet. Lower layers may emphasize short- horizon sensory contingencies, while deeper layers may emphasize longer-horizon abstract recall. The learned layer keysk t,ℓ and valuesv t,ℓ allow the fi- nal memory readout h t to adaptively combine these complementary views. This hierarchical fusion is crucial because the mem- ory architecture is deliberately over-complete in both space and time. The final policy does not act on the full slot bank, nor on a fixed single-layer state. Rather, it acts on a task-conditioned mixture of work- ing summaries extracted from multiple depths of the memory hierarchy. In this sense, the final readouth t is the output of two nested selection processes. First, evidence is factorized into slot-wise episodic traces through the anchor construction and slot-specific dy- namics. Second, layer-wise working summaries are selected and fused according to task relevance. This two-stage organization, namely slot-level factoriza- tion followed by hierarchical working fusion, is what differentiates our memory from conventional recur- rent backbones and enables structured long-horizon recall without explicit retrieval gates. HoloHead. HoloHead regularizes the memory state h t via a latent imagination objective: conditioned onh t , it predicts an ordered set ofN a +N c future waypoints in both 2D (front-camera image-plane E projection) and 3D (normalized world-frame E posi- tion). We decompose the rollout into a short-horizon Anchor segment and a long-horizon Compass seg- ment, ˆ W 2D t,A , ˆ W 2D t,C , ˆ W 3D t,A , ˆ W 3D t,C = HoloHead(h t ), (39) where ˆ W ξ t,A = ˆw ξ t+∆ (a) i N a i=1 and ˆ W ξ t,C = ˆw ξ t+∆ (c) j N c j=1 forξ ∈ 2D,3D. The Anchor way- points use the next N a consecutive frames, ∆ (a) i = i, i = 1,...,N a ,(40) while the Compass waypoints are sampled non- uniformly over the remainder of the current phase. 18 Lett end denote the end of the current phase and let R t =t end − tbe the number of remaining frames. WhenR t > N a , we use logarithmic spacing from N a + 1 to R t , α j = j− 1 N c − 1 , ∆ (c) j = (N a + 1) 1−α j R α j t , j = 1,...,N c . (41) and set the final Compass waypoint to the phase endpoint, i.e., ∆ (c) N c =R t . Thus, Compass targets are sampled from near to far in a dense-to-sparse manner. WhenR t ≤ N a , all Compass targets collapse to the phase endpoint. We supervise HoloHead with L holo = X ξ∈2D, 3D ∥ ˆ W ξ t,A − W ξ t,A ∥ 1 +∥ ˆ W ξ t,C − W ξ t,C ∥ 1 . (42) Architecturally, HoloHead is a lightweight shared- trunk, dual-branch predictor. A shared MLP trunk first mapsh t to a geometric latent, which is then decoded by one branch into normalized 3D waypoints and by another branch into normalized 2D waypoints. The shared latent is also projected to a compact geometric feature for fusion with the policy stream. This objective pressures the memory dynamics to reconstruct coherent future motion from partial cues, encouragingh t to encode both short-term actionable geometry and longer-horizon phase-level intent. A.4 Policy Single-state conditioning in implementation. As stated in Section 3.4, the policy conditions only on the Memory Readouth t . In implementation, we first project h t to the policy hidden dimension: c t = W ctx h t ∈ R d pol ,(43) whereW ctx is a learned linear projection andd pol denotes the hidden width of the policy network. This vector is used as a single conditioning token for the trajectory generator and is the only memory signal exposed to the policy. Trajectory tokenization. Given the interpolated tra- jectoryx τ ∈ R H×d raw from Equation(17), we embed each future end-effector pose independently into the policy hidden space: z (0) τ = W in x τ + P ∈ R H×d pol ,(44) whereW in is a learned linear projection applied to each trajectory step, andP ∈ R H×d pol is a learnable positional embedding over the prediction horizon. The policy therefore operates on the full future tra- jectory as a sequence of pose tokens, rather than predicting each step independently. Flow-time embedding. The flow timeτ ∈[0,1] is encoded using a sinusoidal embedding followed by a two-layer MLP: e τ = MLP PE(τ ) ∈ R d pol ,(45) wherePE(τ) denotes the sinusoidal time embedding. This embedding serves as the global conditioning signal for all policy blocks. Transformer velocity network. The velocity network v θ is implemented as a stack of Transformer blocks. Letz (ℓ−1) τ ∈ R H×d pol denote the input trajectory tokens to blockℓ. Each block contains three sublay- ers: self-attention over the trajectory tokens, cross- attention from the trajectory tokens to the single conditioning token c t , and a feed-forward network: z (ℓ) τ = Block (ℓ) z (ℓ−1) τ ; e τ , c t .(46) Operationally, self-attention captures dependencies across future steps within the trajectory, while cross- attention injects the memory-conditioned decision context derived from h t . To make the denoising dynamics explicitly time- dependent, we modulate every block with adap- tive layer normalization (AdaLN) driven bye τ . Concretely, the time embedding predicts per-block affine modulation and residual gating parameters for the self-attention, cross-attention, and feed-forward branches. Thus, the policy remains single-state con- ditioned at the interface level, while internally per- forming flow-time-dependent sequence reasoning over the future trajectory. Output parameterization. After the final block, a modulated normalization layer and linear projection map the hidden sequence back to the raw trajectory space: ˆu τ = v θ (x τ ,τ,c t )∈ R H×d raw .(47) The network therefore predicts one velocity vector for each future step and each pose dimension, matching the shape of the target velocityu τ =x 1 − x 0 in Equation (17). 19 Architecture and initialization details. In all exper- iments, the policy uses 6 Transformer blocks with hidden widthd pol = 384 and 8 attention heads. We apply dropout after input embedding and within the feed-forward sublayers. The trajectory positional embedding is initialized with a small Gaussian distri- bution, and linear layers are initialized with Xavier initialization. Following DiT-style practice, the final linear layers that predict AdaLN modulation param- eters are zero-initialized, which makes the network start close to an unmodulated backbone and improves training stability for rectified flow matching. Inference. At test time, we sample an initial trajec- tory x 0 ∼N (0,I) and solve the ODE ̇x = v θ (x,τ,c t )(48) with a fixed-step solver using 50 integration steps to obtainˆx t:t+H . Since the policy conditions only on the projected Memory Readoutc t =W ctx h t , the quality of the generated trajectory directly reflects whether the memory module has preserved the task-relevant information needed for future control. 20 B Additional Experimental Details B.1 Camo-Dataset Statistics 60070080090010001100 Frames per episode 0 2 4 6 8 10 Number of episodes Clean a Specific Plate Mean = 761 85090095010001050110011501200 Frames per episode 0 2 4 6 8 10 Number of episodes Play Shell Game Mean = 1009 120014001600180020002200 Frames per episode 0 2 4 6 8 10 12 Number of episodes Add Seasonings Mean = 1499 Figure 8 Distribution of episode lengths (measured in frames) for the three tasks in Camo-Dataset: Clean a Specific Plate, Play Shell Game, and Add Seasonings. Each histogram shows the number of episodes falling within different frame ranges. The dashed vertical line indicates the mean episode length for each task (761, 1009, and 1499 frames, respectively). Figure 8 shows the distribution of episode lengths in Camo-Dataset for the three tasks. Frames are recorded at 30 FPS. During training, trajectories are temporally downsampled by a factor of 4 across all tasks to reduce the effective sequence length and improve training efficiency. B.2 Implementation Details Perception. At each timestep, the agent receives two RGB observations, one from a fixed front camera and one from a hand-mounted camera, both resized to 224×224. For each view, we extract patch-level appearance tokens with a frozen DINO encoder using a patch size of 14, yieldingN f =N h = 256 visual to- kens of dimensiond= 768. Cross-view enhancement is implemented with geometry-biased bidirectional cross-attention using 8 attention heads. As described in Section 3.2, the attention logits are biased by the end-effector anchored geometric descriptors and the analytic epipolar feasibility term, so cross-view com- munication is restricted to geometry-consistent and action-relevant patches. After residual cross-view enhancement and geometry-conditioned modulation, the two view-specific token sets are concatenated to form the Perception output x t = Concat( ̃ V f t , ̃ V h t )∈ R N×d , N = N f +N h = 512. Memory. Memory takes the Perception evidencex t together with the proprioceptive states t and, when available, the phase signalψ t . A proprioception token and an optional phase token are appended to the visual evidence, and all tokens are tagged with learned modality and view embeddings as in Equation(5). The resulting fused token set is z t ∈ R M×d , withM=N+ 1 when the phase token is absent andM=N+ 2 otherwise. We useL= 2 mem- ory layers and a working-state dimensiond w = 512. Each layer performs dynamic context modulation us- ing Equation(7), decomposes the current evidence intoA= 8 spatial anchors andB= 4 temporal anchors, and therefore maintains anA × B= 32 slot memory structure. The episodic stream is im- plemented with a selective state-space block using (d state ,d conv , expand) = (128,4,1), while the working stream uses (32,4,2). Following Equation(10), tem- poral slots are associated with distinct base step sizes ∆t (0) b and further modulated by slot content, yielding a structured multi-timescale memory. For computa- tional efficiency, the slot-wise episodic recursion is implemented with a fused single-pass scan kernel that updates allA× Bepisodic states jointly within each layer. Policy and HoloHead supervision. Policy conditions only on the Memory Readouth t and predicts a fu- ture end-effector trajectory over horizonH= 8 us- ing conditional rectified flow matching. The policy backbone uses 6 layers, hidden width 384, and 8 attention heads. During training, we sample interpo- lation times uniformly on [0,1] as in Equation(17); at test time, trajectories are generated by solving the learned rectified-flow ODE with 50 integration steps. We further regularize the Memory Readout with Holo- Head, implemented as a 3-layer MLP with hidden width 512. Conditioned onh t , HoloHead predicts both 2D image-plane waypoints and 3D world-frame waypoints. The head outputsN a = 8 short-horizon Anchor waypoints andN c = 8 long-horizon Com- pass waypoints, matching the supervision described in Equation (15) and Equation (16). Optimization and sequence training. We train the model with AdamW using a learning rate 1×10 −4 and weight decay 1×10 −6 . The batch size is 4. We use 100 warmup steps followed by cosine learning-rate decay, and clip gradients at a global norm of 100. Training is performed in bfloat16 with exponential moving average weights of decay 0.999. To support long- horizon sequence training, we use sequence chunks of length 512 with temporal stride 4, and apply the policy loss to the last 16 frames of each chunk. The HoloHead rollout losses use weights 0.5 for the 2D term and 0.5 for the 3D term. 21 Table 3 Implementation details and hyperparameters of Chameleon, aligned with the notation in Section 3. ModuleParameterValue Perception Φ perc Camera viewsFront RGB + hand-mounted RGB Input image resolution224× 224 Ventral encoderFrozen DINOv2 Patch size14 Patch tokens per view (N f ,N h )(256, 256) Token width d768 Cross-view enhancementGeometry-biased bidirectional cross-attention Cross-view attention heads8 Geometry termsEE-anchored unary patch bias + epipolar bias Perception output x t x t ∈ R N×d , N = N f + N h = 512 Memory Φ mem Memory input tokens z t z t ∈ R M×d , M = N + 1 or N + 2 Working-state dimension d w 512 Number of memory layers L2 Spatial anchors A8 Temporal anchors B4 Total memory slots A× B32 Base temporal priors ∆t (0) b 0.001, 0.005, 0.02, flexible Episodic Memory SSM(d state ,d conv , expand) = (128, 4, 1) Working Memory SSM(d state ,d conv , expand) = (32, 4, 2) Slot-wise state updateFused single-pass scan kernel Policy Φ pol Policy backbone depth6 layers Policy width384 Policy attention heads8 Prediction horizon H8 Training objectiveConditional rectified flow matching Time sampling τUniform on [0, 1] ODE solver steps (test time)50 HoloHeadHead architecture3-layer MLP Hidden width512 Anchor waypoint count N a 8 Compass waypoint count N c 8 Prediction spaces2D image plane + 3D world frame OptimizationOptimizerAdamW Learning rate1× 10 −4 Weight decay1× 10 −6 Batch size4 Warmup steps100 Learning-rate scheduleCosine decay Gradient clipping100 Training precisionbfloat16 EMA decay0.999 Sequence training Sequence chunk length512 Temporal stride4 Policy loss windowLast 16 frames HoloHead loss weight (2D)0.5 HoloHead loss weight (3D)0.5 B.3Model Size and Computational Cost of Experiment Table 4 summarizes the parameter counts of the baseline compared. For baseline methods, we follow the parameter settings of original implementations without modification. In Chameleon, the visual en- coder (DINOv2) is kept frozen and therefore does not contribute to trainable parameters. The learnable components consist mainly of geometry-based per- ception fusion, a hierarchical memory stack, and a flow-matching policy head. As a result, the trainable parameter count of Chameleon remains comparable to that of existing visuomotor policies while introduc- ing an additional memory structure. Table 5 lists the architecture variants used in the ablation study. Each variant modifies a single com- ponent of Chameleon while keeping the remaining modules unchanged. All policies use Conditional Flow Matching (CFM). Specifically, “w/o Memory” removes the memory module, “Memory Bank” re- places the hierarchical memory with a simple memory bank, and “Vanilla Mamba” substitutes it with a stan- dard Mamba sequence model. “w/o Dorsal Stream” removes the dorsal perception pathway, while “w/o 22 Table 4 Model Complexity and Parameter Counts. ModelBackbone / ComponentsTrainable Parameters (M) Diffusion Policy Chi et al. (2023)ResNet-18 + UNet Diffusion Head≈ 328.83 Flow Matching Zhang and Gienger (2024) ResNet-18 + UNet Velocity Net≈ 79.90 ACT Zhao et al. (2023)Transformer Encoder-Decoder≈ 78.13 Chameleon (Ours)Perception + Memory + Policy≈ 49.89 HoloHead” removes the HoloHead training objective. B.4 Evaluation Metrics Definitions Evaluation Metrics. Beyond the overall SR, we report three additional metrics: DSR, MSR, and CSR. MSR measures whether the robot completes the manipulation phase of a task. DSR evaluates whether the robot makes the correct decision de- pending on memory. To isolate decision correctness from manipulation failures, DSR is computed only over trials where manipulation succeeds:DSR= Number of decision-success trials Number of manipulation-success trials .CSR evaluates the informativeness of the decision stateh t . Specifically, we roll out fromh t to predict task-relevant latent variables (e.g., the correct target object or task stage), and compute the corresponding success rate. Because decision outcomes may have non-trivial chance-level performancep e , we additionally report the chance- adjusted Cohen’s kappa coefficient Cohen (1960) for DSR. Clean a specified plate. The robot must place a cube (representing a cleaning tool) onto the contami- nated plate specified by the user. After receiving the manipulation-phase instruction, the robot picks up the cube, places it on one of three candidate plates, and returns to the initial position. Evaluation pro- tocols and chance-level statistics are summarized in Table 6. CSR is computed by rolling out the decision state h t to predict the target plate identity. Play shell game. The robot observes a human hiding a cube under one of several cups and shuffling them, and must track the hidden object before lifting the correct cup. Task definitions and success criteria follow Table 6. CSR is evaluated by predicting the correct cup from the decision state h t . Add various seasonings. The robot sequentially picks up, moves, and returns three colored seasoning spoons. Each seasoning operation consists of decision, manipulation, and recovery phases, and the full task contains nine phases in total. The phase instruction is shared by the manipulation and recovery stages to avoid information leakage. Success definitions and chance-level statistics are summarized in Table 6. CSR evaluates whetherh t can correctly predict the current task phase. B.5 Failure Case Analysis Chameleon. Despite strong improvements in memory-dependent decision making, Chameleon still exhibits occasional failures. As shown in Table 2, DSR and CSR across the three tasks are 100%/100%, 73.5%/91.7%, and 72.2%/85%, respectively, showing a consistent correlation. This indicates that decision failures mainly occur when HoloHead rollouts fail to predict the task-relevant latent state, corresponding to failures of pattern completion. These errors can be traced to ambiguities in the decision stateh t , sug- gesting imperfect pattern separation during memory encoding, potentially due to the relative weighting of the HoloHead rollout loss during training. Moreover, the dominant failure mode is manipulation rather than decision. The model typically identifies the correct target and approaches it successfully, but errors arise during grasping, such as exceeding motion limits or incorrect grasp orientation. These failures likely stem from imitation learning objectives, where trajectory matching can introduce stochastic deviations that become critical near contact. Since the policy conditions solely on the decision stateh t , persistent memory representations may occasionally introduce action ambiguity. However, given the competitive manipulation success on simpler tasks such as Clean a specified plate and Play shell game, and the best performance on the most challenging Add various seasonings task, action ambiguity is unlikely to be the primary limiting factor. Diffusion Policy, Flow Matching and ACT. All three baselines achieve only chance-level performance in memory-dependent decision making. Despite the balanced distribution of task variants in the train- ing data, the models tend to converge to a single dominant behavior, effectively predicting one deci- sion category regardless of the underlying memory state. In this task, the diffusion policy does not ex- hibit the ability to capture a diverse distribution over decision trajectories. Instead, it collapses to a sin- 23 Table 5 Architecture variants used in ablation studies. VariantPerceptionMemory Layers / TrainingPolicy Chameleon (Ours) Ventral & Dorsal Stream Hierarchical Memory Stacks + HoloHead CFM w/o MemoryVentral & Dorsal Stream NACFM Memory BankVentral & Dorsal Stream Memory BankCFM Vanilla MambaVentral & Dorsal Stream Vanilla MambaCFM w/o Dorsal Stream Ventral Stream OnlyHierarchical Memory Stacks + HoloHead CFM w/o HoloHeadVentral & Dorsal Stream Hierarchical Memory StacksCFM Table 6 Task definitions and chance-level success rates. TaskCandidates Manipulation SuccessDecision SuccessChance-Level Clean a specified plate3 platesComplete pick–place–return Place the cube on the target plate DSR = 1/3, CSR = 1/3 Play shell game3 cupsLift any cupLift cup containing cube DSR = 1/3, CSR = 1/3 Add various seasonings 3 spoonsComplete three pick–season–return Correct order: green–red–yellow DSR = 1/27, CSR = 1/9 gle dominant behavior. It is somewhat unexpected, as diffusion policies are known for their ability to model multimodal action distributions rather than collapsing to a single deterministic behavior. In terms of manipulation performance, the baselines achieve high success rates on simple pick-and-place tasks. However, on the more complex task, which involves longer action sequences and higher task complexity, their performance degrades substantially. Memory Bank and Vanilla Mamba. In the Memo- ryBank variant, we replace the memory module of Chameleon with a simple memory bank that stores past latent features together with a similarity-based retrieval mechanism. The retrieved memories are directly concatenated with the current observation before being passed to the policy. We hypothesize that similarity-based retrieval is unstable under per- ceptual aliasing, since visually similar but decision- irrelevant past observations can be retrieved together with truly relevant ones. This introduces ambigu- ity into the recalled context, causing the policy to confuse relevant and irrelevant history. As a result, the policy fails to maintain a consistent behavioral strategy and often exhibits random movements near the initial position, losing the ability to complete the manipulation task. In the Vanilla Mamba variant, the structured episodic- working memory of Chameleon is replaced by a single monolithic state-space backbone. As a result, the model lacks explicit pattern separation during mem- ory encoding and also lacks a goal-directed recall mechanism before policy conditioning. Under percep- tual aliasing, task-relevant cues are therefore mixed with irrelevant observations in a single recurrent state, making the model uncertain about what should be preserved, prone to forgetting the weak but crucial contextual signal needed for later disambiguation, and unable to selectively expose the right memory at decision time. Empirically, this leads to unstable behavior across trials: once the latent cue is over- written or diluted, the policy can no longer maintain a consistent memory-dependent strategy and often falls back to incorrect or near-random actions. w/o Dorsal Stream. This variant exhibits the largest drop in success rate compared to Chameleon in the Play shell game task. This task involves frequent changes in spatial relations and requires the model to perform a certain degree of spatial reasoning. Conse- quently, this variant shows a higher tendency toward random failures. Unlike the baselines, where memory completely fails, and the policy collapses to interact- ing with a fixed target, this model tends to randomly interact with one of the candidate objects after fail- ure, rather than consistently collapsing to a specific target. w/o HoloHead. The model exhibits a certain degree of memory capacity. However, due to the misalign- ment between the open-loop training objective and the temporal locations of the memory frames, the model fails to learn stable memory utilization. As a result, the recalled context is inconsistent, and the overall memory performance remains unstable. 24 CHuman Episodic Memory and Cor- respondence to Chameleon C.1Overview of the EC–HC–PFC Episodic Memory Circuit Human episodic memory is commonly associated with the interaction between the medial temporal lobe and the frontal cortical systems. In particular, the EC, the HC, and the PFC form a functional circuit that sup- ports the encoding, storage, and retrieval of episodic experiences. Sensory information processed across distributed cortical regions converges onto the medial temporal lobe, where the HC integrates multimodal inputs into event-specific representations. These rep- resentations can later be reactivated to reconstruct past experiences and guide behavior. At the systems level, cortical association areas project to the HC primarily via the EC. The hippocampus binds together distributed sensory representations into coherent episodic traces and subsequently rein- states activity in neocortical regions during recall. Meanwhile, the prefrontal cortex interacts with hip- pocampal memory to organize encoding, control re- trieval, and integrate recalled information with on- going decision processes. This EC–HC–PFC loop, therefore, provides the neural substrate for memory- guided behavior and flexible cognition, describing how HC–neocortical connectivity enables recall by reinstating cortical activity patterns associated with stored episodes. C.2Perceptual Pathways: Ventral and Dorsal Streams Before reaching the memory system, sensory infor- mation is processed through specialized perceptual pathways in the neocortex. In the visual domain, two major streams are commonly distinguished: the ven- tral “what” pathway and the dorsal “where” pathway. The ventral stream projects from early visual areas toward the temporal cortex and encodes object iden- tity and semantic features, while the dorsal stream projects toward the parietal cortex and represents spatial relationships and action-relevant geometry. These complementary streams provide the HC with both object-centered and spatial information neces- sary for constructing indexed episodic representations. By combining these signals, the HC can encode events as structured associations between objects, locations, and contextual cues. Inspired by this biological organization, we design the perception module with a similar dual-stream structure. A ventral pathway extracts patch-level appearance features from the input images, provid- ing detailed visual evidence about what is present in the scene. In parallel, a dorsal pathway com- putes E–anchored geometric descriptors that en- code action-centric spatial relations, supplying the where structure relevant for manipulation. These two streams interact through geometry-grounded cross- view communication, allowing appearance evidence and spatial structure to be integrated while preserv- ing viewpoint-specific cues. The resulting represen- tation forms a structured evidence set that jointly captures visual appearance and control-relevant ge- ometry and is stored in the memory module as the perceptual basis for episodic decision-making. C.3Entorhinal Cortex: Interface Between Perception and Memory Within the HC formation, the EC and DG together constitute the initial stage of HC information pro- cessing and play a central role in encoding episodic memory. The entorhinal cortex acts as the principal interface between the neocortex and the hippocam- pus, integrating sensory and contextual information from distributed cortical areas. Neurons in the su- perficial layers of the EC project to the DG through the perforant path, transmitting a highly convergent representation of the current sensory and spatial con- text. While the EC provides a rich but often overlapping representation of ongoing experience, the DG trans- forms this input through a process commonly de- scribed as pattern separation. Granule cells in the DG receive convergent projections from large pop- ulations of EC and operate under strong inhibitory control, leading to extremely sparse neural activ- ity. As a result, similar patterns of EC activity are mapped onto more decorrelated population codes in the DG. Through this EC–DG transformation, the hippocampal system reduces interference between similar experiences and enables the encoding of dis- tinct episodic memories even when sensory inputs are highly similar. The multimodal fusion and slot formation mecha- nism in our memory module plays a role analogous to the EC–DG transformation. The perception mod- ule first produces patch-level evidence tokens from multiple sensory sources, which correspond to the multimodal cortical inputs arriving at the EC. These tokens are then fused and decomposed into spatial anchors, forming localized summaries of the visual scene. The subsequent expansion into multiple spa- tiotemporal slots further decorrelates the evidence 25 Table 7 Biological inspiration and correspondence to the Chameleon. Biological component Biological roleChameleon moduleFunctional correspondence Ventral stream Encodes object identity in- formation (what). Patch-level tokens from the frozen DINO encoder Provides detailed local visual ev- idence about objects, boundaries, and contact-relevant regions. Dorsal stream Encodes spatial relations and geometry (where). E-anchoredspatial geometric descriptors, and geometry-biased cross-view attention. Supplies spatial structure, including cross-view correspondence between the front and hand cameras. ECIntegrates multimodal senso- ry/contextual inputs. Multimodal fusion of vi- sual, proprioceptive, and phase tokens Converts heterogeneous sensory sig- nals into a unified evidence set that can be processed by memory. DGPerforms pattern separation.Spatial anchors and spa- tiotemporal slot. Expands overlapping perceptual ev- idence into structured slot represen- tations indexed by spatial compo- nent and temporal scale. HC circuitSupports the encoding and retrieval of structured episodic representations. Hierarchical episodic memory stack. Maintains structured episodic mem- ory across slots and time. CA3Stores episodic traces and supports pattern completion. Slot-wise episodic state- space memory. Maintains distributed episodic states whose dynamics support context recovery from partial observations. CA1Integrates HC recall into an output representation that can be communicated to downstream cortical sys- tems. Aggregated episodic re- call vector and layer-level working readout. Combines information recalled from multiple episodic slots into a com- pact representation for downstream use. HC–PFC interaction Supports retrieval, prospec- tive cognition, and memory- guided decision-making. Working stateh t with hi- erarchical fusion. Aggregates recalled episodic content into a compact decision-oriented state analogous to prefrontal work- ing representations. PFC imagination Recombinesretrieved episodic content to predict future states and evaluate candidate actions. HoloHead latent imagina- tion module Performs prospective simulation by predicting multi-horizon future E trajectories, shapingh t into a predic- tive representation of future motion and task progress. by projecting it into a higher-dimensional structured representation indexed by spatial component and tem- poral scale. Similar observations that differ in subtle spatial or temporal cues therefore activate different slot patterns, reducing interference between episodes. Although our model does not explicitly implement a learned pattern separation mapping as in biological EC–DG circuits, the spatial–temporal slot decompo- sition organizes perceptual evidence into structured components that reduce interference between similar observations. In this sense, it functions analogously to the pattern separation operation of the EC–DG pathway, transforming overlapping perceptual inputs into a representation suitable for episodic storage. C.4Hippocampus: Pattern Separation and Pattern Completion Within the HC formation, multiple interconnected subfields cooperate to support episodic encoding and retrieval. After the EC–DG transformation produces decorrelated representations, these signals are trans- mitted to CA3 through the mossy fiber pathway. Unlike the sparse coding regime of DG, CA3 contains extensive recurrent collateral connections that form an auto-associative network. These recurrent connec- tions enable pattern completion: partial or degraded inputs can trigger attractor dynamics that reactivate the stored representation associated with the original experience. Through the complementary operations of pattern separation and pattern completion, the HC circuit can both encode distinct events and retrieve them from incomplete sensory cues. Neurophysiologi- cal studies further suggest that HC neurons integrate spatial, object-related, and contextual information into conjunctive activity patterns, effectively bind- ing distributed cortical representations into coherent episodic memories. The episodic state-space memory in our architecture parallels key computations of the HC circuit. Each spatial–temporal slot maintains an episodic state that evolves through a selective SSM, acting as a dis- tributed memory trace indexed by spatial component and temporal resolution. The slot-wise SSM dynam- ics preserve long-lived episodic information while al- lowing content-dependent updates through learned input factors. When new evidence arrives, these episodic states are updated and subsequently read out through slot-specific projections, producing recall vectors that summarize relevant stored information. 26 This mechanism resembles HC pattern completion in that the episodic states can recover relevant past context even when the current observation provides only partial evidence. In biological systems, episodic memory is often described as a cortical storage sys- tem indexed by the HC, with CA3 attractor dynamics supporting replay-based retrieval. Our architecture does not explicitly model these mechanisms; instead, information storage and retrieval are implemented through direct differentiable information flow within the slot-wise memory dynamics. The aggregated re- call vector then plays a role analogous to the CA1 output stage, integrating contributions from multiple episodic traces into a compact representation con- sumed by downstream modules. C.5Hippocampus–Prefrontal Cortex Interac- tion Although the HC encodes and retrieves episodic rep- resentations, adaptive behavior requires additional mechanisms that organize memory for future-oriented decision making. A large body of evidence suggests that interactions between the HC and PFC support such prospective cognition. In particular, prefrontal circuits are believed to guide the retrieval and re- combination of episodic memories to simulate pos- sible future states and evaluate candidate actions. During goal-directed behavior, HC activity provides fragments of experience, while PFC circuits organize these fragments into structured predictions about future trajectories and task outcomes. Our architecture mirrors this HC–PFC interaction through the coupling between episodic memory and a compact working state. While the episodic states store distributed traces of past observations, the work- ing stateh t aggregates recalled episodic information into a representation suitable for decision making, playing a role analogous to prefrontal working repre- sentations. Conditioned on this state, the HoloHead module performs latent imagination by predicting fu- ture end-effector trajectories across multiple horizons. The short-horizon Anchor predictions capture im- mediate control geometry, whereas the long-horizon Compass predictions summarize the broader phase- level motion structure. In this view, episodic memory provides contextual fragments of experience, while the working state or- ganizes these fragments into a predictive representa- tion of future behavior. The imagination objective implemented by HoloHead encouragesh t to encode task-relevant future structure, analogous to prospec- tive simulation in prefrontal–hippocampal circuits. As a result, the decision stateh t becomes not only a summary of experience but also a representation predictive of upcoming motion and task progress. 27