Paper deep dive

GaussMemory: Task-Driven 3D Gaussian Scene Memory for Long-Horizon Robotic Manipulation

Zhiqiang Hu, Shouren Huang, Masatoshi Ishikawa

Year: 2026Venue: arXiv preprintArea: cs.ROType: PreprintEmbeddings: 43

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%

Last extracted: 8/18/2026, 5:04:41 AM

Summary

The paper introduces GaussMemory, a task-driven 3D Gaussian scene memory system for long-horizon robotic manipulation. It shifts from passive 3D memory recording to active, end-to-end differentiable memory updates using Unified Memory Attention (UMA). This allows the robot to learn which objects to track and how to update them based on task needs, outperforming existing methods like MemoryVLA and pi0-FAST on benchmarks such as LIBERO and VLABench.

Entities (9)

GaussMemory · system · 100%3D Gaussian Splatting · technology · 95%LIBERO · benchmark · 95%Unified Memory Attention · module · 95%VLABench · benchmark · 95%Llama-2-7b · model · 90%MemoryVLA · baseline · 90%pi0-FAST · baseline · 90%Prismatic VLM · model · 90%

Relation Signals (7)

GaussMemory → evaluatedon → LIBERO

confidence 95% · On LIBERO, GaussMemory outperforms MemoryVLA

GaussMemory → evaluatedon → VLABench

confidence 95% · on VLABench, it surpasses $π_0$-FAST

GaussMemory → uses → Unified Memory Attention

confidence 95% · We realize this unified vision through Unified Memory Attention (UMA)

GaussMemory → uses → 3D Gaussian Splatting

confidence 95% · GaussMemory, which leverages 3D Gaussian Splatting as a persistent geometric substrate.

GaussMemory → outperforms → pi0-FAST

confidence 90% · on VLABench, it surpasses $π_0$-FAST by +5.2% (Track 1) and +6.0% (Track 6).

GaussMemory → outperforms → MemoryVLA

confidence 90% · On LIBERO, GaussMemory outperforms MemoryVLA on Goal and Long-10

Prismatic VLM → uses → Llama-2-7b

confidence 90% · Prismatic VLM [7], which pairs DINOv2 [14] and SigLIP [23] vision encoders with a Llama-2-7B [21] LLM

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Long-horizon robotic manipulation fundamentally relies on persistent spatial memory. However, existing 3D memory systems function merely as passive recorders: they store observations using fixed, hand-crafted rules, treating every scene element--whether a critical grasp target or an irrelevant background wall--with equal importance. In this paper, we propose a paradigm shift from passive storage to active, task-driven spatial memory. We argue that a robot's memory should not simply record what it sees, but actively learn how to remember--discovering which objects to track precisely, how aggressively to update them, and what to discard, all learned end-to-end without hand-designed rules. Crucially, this active paradigm is realized by unifying memory update and readout as two sides of the same cognitive process, enabling bidirectional flow where task needs shape update strategies and vice versa. To instantiate this vision, we introduce GaussMemory, which leverages 3D Gaussian Splatting as a persistent geometric substrate. On LIBERO, GaussMemory outperforms MemoryVLA on Goal and Long-10; on VLABench, it surpasses $\pi_0$-FAST by +5.2% (Track 1) and +6.0% (Track 6).

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

40,363 characters extracted from source content.

Expand or collapse full text

GaussMemory: Task-Driven 3D Gaussian Scene Memory for Long-Horizon Robotic Manipulation Zhiqiang HU Shouren HUANG Masatoshi ISHIKAWA Thanks: Thanks: The authors are with the Research Institute for Science & Technology, Tokyo University of Science zhiqiang.hu, huang, ishikawa@ishikawa-vision.org Abstract Long-horizon robotic manipulation fundamentally relies on persistent spatial memory. However, existing 3D memory systems function merely as passive recorders: they store observations using fixed, hand-crafted rules, treating every scene element—whether a critical grasp target or an irrelevant background wall—with equal importance. In this paper, we propose a paradigm shift from passive storage to active, task-driven spatial memory. We argue that a robot’s memory should not simply record what it sees, but actively learn how to remember—discovering which objects to track precisely, how aggressively to update them, and what to discard, all learned end-to-end without hand-designed rules. Crucially, this active paradigm is realized by unifying memory update and readout as two sides of the same cognitive process, enabling bidirectional flow where task needs shape update strategies and vice versa. To instantiate this vision, we introduce GaussMemory, which leverages 3D Gaussian Splatting as a persistent geometric substrate. On LIBERO, GaussMemory outperforms MemoryVLA on Goal and Long-10; on VLABench, it surpasses π0 _0-FAST by +5.2% (Track 1) and +6.0% (Track 6). I Introduction Vision-Language-Action (VLA) models such as OpenVLA [9] and π0 _0 [2] have emerged as a powerful paradigm for robotic manipulation, leveraging pretrained vision-language knowledge to map visual observations and language instructions directly to actions. However, memoryless VLA systems are mainly limited to tasks in which the next action can be inferred from the current sensory observation. As Fig. 1 illustrates, when an object is stored in one of two visually identical drawers and subsequently becomes occluded, the current RGB observation is insufficient to identify the correct drawer after an intervening sub-task. Persistent spatial memory is therefore required to retain the object’s identity and last observed 3D position across time, enabling reliable retrieval in long-horizon manipulation. Fig. 1: With vs. without persistent memory. Why persistent 3D memory is necessary. The robot stores an apple in one of two visually identical drawers and completes an intervening sub-task. Once both drawers are closed, the current RGB observation no longer reveals the apple’s location. A memoryless policy cannot disambiguate the correct drawer. Recent efforts have begun to equip VLA systems with temporal memory, but the solutions remain limited. MemoryVLA [18] maintains a 2D perceptual-cognitive memory bank achieving strong results, yet its memory is fundamentally 2D and cannot answer spatial queries such as “where exactly is the plate relative to the glass?” MEM [20] introduces multi-scale embodied memory combining semantic, episodic, and procedural stores, but likewise operates in 2D token space without explicit 3D geometry. Language-based approaches such as SayPlan [17] record text descriptions of past states but sacrifice geometric precision entirely. Even recent 3D-structured systems such as ConceptGraphs [4] that maintain scene graphs with explicit geometry still rely on hand-crafted update rules—thresholded matching, fixed-rate averaging, hard pruning—entirely decoupled from task performance. Across all these categories, the memory systems share a common limitation: they function as passive recorders that treat every scene element with equal importance. Crucially, this stems from a deeper issue: memory update and readout are treated as independent operations, with no communication between them. Fig. 2: 2D token memory vs. 3D Gaussian scene memory. (a) Existing VLA methods buffer 2D image tokens that lack 3D geometry, lose information under occlusion, and suffer from multi-view inconsistency. (b) GaussMemory maintains a persistent 3D Gaussian Scene Memory where each Gaussian carries explicit position, shape, semantic features, timestamps, object identity, and existence probability. We propose to move beyond this passive paradigm by shifting to active, task-driven 3D spatial memory within the VLA framework (Fig. 2). The memory should learn not just what to remember, but how to remember—which objects to track with high precision, which observations to trust, when to update aggressively and when to preserve—all guided by one criterion: does this way of remembering help the robot complete its task? The key insight is that memory update and readout are not separate processes but two sides of the same cognitive computation: how you use memory (readout) should directly shape how you store it (update), and vice versa. For instance, if the next sub-task requires placing silverware next to the plate, the readout queries will attend strongly to the plate’s position; this attention signal should flow back to bias the update toward tracking the plate more precisely. Conversely, if an object like a cup was just moved, the update event should inform the readout to trust its new position more. This bidirectional coupling—inspired by cognitive science phenomena like the Testing Effect, where retrieval strengthens memory encoding—is absent in existing decoupled systems but enables the task loss to genuinely sculpt memory behavior. 3D Gaussian Splatting (3DGS) [8] is a particularly suitable substrate for realizing active memory within VLA, thanks to three properties: (1) Natural incrementality: individual Gaussians can be added, removed, or repositioned without disrupting the rest of the scene. (2) Explicit geometry: each Gaussian carries position μ, covariance , and opacity α, providing the precise spatial information that manipulation planning requires. (3) Tokenizability: Gaussian features can be encoded into compact tokens and injected into an LLM backbone alongside language and 2D visual tokens, integrating naturally into the VLA architecture. Yet no existing system takes the step from passive Gaussian storage to active, task-driven Gaussian memory within VLA—where the update mechanism is end-to-end differentiable with respect to the action prediction loss. We realize this unified vision through Unified Memory Attention (UMA), where observation tokens and readout queries are concatenated and processed in shared cross-attention layers. This design enables bidirectional information flow—the readout is aware of what just changed, and the update is aware of what the task cares about—so that the task’s information need shapes memory behavior not only through gradients during training but also through forward attention flow at inference time. We present GaussMemory with three contributions: Active 3D Gaussian Memory Paradigm: We propose that robot spatial memory should shift from passive recording to active, task-driven learning, instantiated using 3D Gaussian Splatting as a persistent geometric substrate with temporal metadata and an object-centric scene graph. Empirically, the learned position-update coefficient exhibits a bimodal pattern, with an average value of approximately 0.74 for manipulated objects and 0.06 for background Gaussians. Unified Memory Attention (UMA): A single cross-attention module that simultaneously reads memory for action prediction and writes updates back, making the entire memory lifecycle end-to-end differentiable. UMA enables bidirectional information flow: task needs shape memory updates, and memory state informs action readout. Persistent 3D Gaussian Memory Integration for VLA: To our knowledge, GaussMemory is the first to integrate 3D Gaussian Splatting as a temporal scene memory within a VLA framework, where persistently updated memory tokens derived from 3D Gaussians via UMA are consumed by an LLM backbone for action prediction. Unlike prior works that encode static Gaussian scenes into tokens for language-based QA [5], our Gaussian memory is continuously updated across sub-tasks and jointly optimized with robot action loss. Fig. 3: GaussMemory architecture. Multi-view RGB images are encoded by F-3DGS into observation tokens obsQ^obs, which are concatenated with task-conditioned readout queries readQ^read and jointly processed by Unified Memory Attention (UMA). UMA cross-attends to the 3D Gaussian Scene Memory (K/V up to 32K persistent Gaussians), producing two parallel outputs: the output stream memZ^mem provides 3D spatial context to the LLM backbone for action prediction via the Diffusion Action Head, while the update memory stream writes back updated Gaussians via learned gates ηk _k, matching, insertion, and pruning. I Related Work I-A Memory for Long-Horizon Manipulation Long-horizon robotic tasks require maintaining information across many interaction steps. Existing memory approaches fall into three categories: Language-based memory: SayPlan [17] uses a language-annotated 3D scene graph updated via text descriptions. While compact and LLM-friendly, these approaches lose geometric precision—“the cup is on the counter” does not encode where on the counter or the cup’s spatial relationship to nearby objects. Image-based memory: MemoryVLA [18] maintains a 2D perceptual-cognitive memory bank with temporal positional encoding, achieving strong results on long-horizon tasks. However, its memory is fundamentally 2D token-level—it cannot answer spatial queries such as “where is the cup relative to the plate?” that require 3D geometric reasoning. 3D-structured memory: ConceptGraphs [4] builds object-centric 3D scene graphs from multi-view observations. All these systems separate memory update from memory readout into independent non-differentiable stages. GaussMemory differs from all the above by shifting from passive to active memory: the update mechanism itself is learned from task outcomes, and 3D Gaussians provide the geometric precision that text and image memories lack. I-B 3D Gaussian Splatting for Robotics 3DGS [8] has expanded into robotic perception. ManiGaussian [13] uses dynamic Gaussians as a forward world model, whereas GaussMemory uses them as a persistent scene memory for retrospective state tracking and future action prediction. GaussianVLM [5] encodes static Gaussian scenes for language-based reasoning and produces text outputs. In contrast, GaussMemory maintains a temporally evolving Gaussian memory whose tokens directly condition robot action prediction. I Method I-A Problem Formulation We consider long-horizon manipulation tasks specified by a high-level language instruction ℒL that decomposes into a sequence of T sub-tasks τ1,…,τT\ _1,…, _T\. At each time step t, the robot receives multi-view RGB images Ivtv=1V\I_v^t\_v=1^V and its proprioceptive state t s_t. The goal is to predict actions t a_t that sequentially complete all sub-tasks while maintaining persistent memory of scene changes. I-B System Overview As shown in Fig. 3, GaussMemory consists of three components: a Temporal Gaussian Scene Memory ℳtM_t (persistent 3D Gaussians), Unified Memory Attention (UMA) (simultaneous memory read/write via shared cross-attention), and a Prismatic VLM backbone (Llama-2-7B, shared with OpenVLA/CogACT/MemoryVLA) with a diffusion action head. Fig. 4: Inside Unified Memory Attention (UMA), expanding the UMA block from Fig. 3. (A) Observation tokens and readout queries are concatenated. (B1) Self-attention exchanges task and perceptual information. (B2) Shared cross-attention queries Gaussian memory through keys and values, producing read-query features for memZ^mem and observation-to-memory correspondences matchA^match for memory update. (C) The two streams support action prediction and memory update, insertion, and pruning. (D) Persistent 3D Gaussian Scene Memory. I-C Temporal Gaussian Scene Memory The core data structure is the Temporal Gaussian Scene Memory ℳtM_t, a set of augmented Gaussian primitives maintained persistently across the task horizon. Augmented Gaussian Primitives Each Gaussian carries standard 3DGS attributes plus temporal metadata: Gi=(i,i,αi,i,i⏟standard 3DGS,tic,tim,τi,oi,ei⏟temporal metadata)G_i=( μ_i, _i, _i, c_i, h_i_standard 3DGS,\; t_i^c,t_i^m, _i,o_i,e_i_temporal metadata) (1) where i∈ℝ3 μ_i\!∈\!R^3 is position, i _i the covariance, αi _i opacity, i c_i color, i h_i spherical harmonic features, tict_i^c/timt_i^m creation/modification timestamps, τi _i sub-task index, and oi∈1,…,Oo_i\!∈\!\1,…,O\ object ID, and existence probability ei∈(0,1)e_i∈(0,1). Temporal Feature Encoding Each Gaussian is encoded as: imem g_i^mem =MLPenc([i;vec(i);αi;i]) =MLP_enc ([ μ_i;vec( _i); _i; h_i] ) +PE3D(i)+PETrel(tim,t) +PE_3D( μ_i)+PE_T^rel(t_i^m,t) (2) where PE3DPE_3D is a 3D Fourier positional encoding [19], and PETrel(tim,t)=PET(t−tim)PE_T^rel(t_i^m,t)=PE_T(t-t_i^m) encodes the relative age of a memory entry. PET(t)=⨁n=1NT[sin⁡(ωnTt),cos⁡(ωnTt)],ωnT=2n−1πTmax.PE_T(t)= _n=1^N_T [ ( _n^Tt), ( _n^Tt) ], _n^T= 2^n-1πT_ . (3) where ωnT _n^T denotes the fixed Fourier frequencies, TmaxT_ is the maximum episode length, and NTN_T is the number of frequency bands. We use relative encoding PETrel(tim,t)=PET(t−tim)PE_T^rel(t_i^m,t)=PE_T(t-t_i^m) for length generalization. Object-Centric Organization Object instance IDs are obtained via Gaussian Grouping [22] with DINO features, organizing Gaussians into per-object groups j=Gi:oi=jG_j=\G_i:o_i=j\. I-D Unified Memory Attention (UMA) UMA is the mechanism at the heart of GaussMemory. It solves two coupled problems—what to update and what to retrieve—in a single differentiable attention mechanism. By first letting observation tokens and task-conditioned readout queries exchange information via self-attention, and then jointly cross-attending to the 3D Gaussian memory, UMA produces both task-relevant spatial context for the LLM and soft correspondence signals for memory update. The data flow proceeds in four stages (Fig. 4): Stage 1: Construct Unified Query Observation tokens obsQ^obs: F-3DGS produces M live Gaussians from multi-view images, each encoded as ilive=MLPenc([i;vec(i);αi;i])+PE3D(i) g_i^live=MLP_enc([ μ_i;vec( _i); _i; h_i])+PE_3D( μ_i), then aggregated to NoN_o object-level tokens: jobs=Pool(ilive:oi=j)+PE3D(¯j) q_j^obs=Pool (\ g_i^live:o_i=j\ )+PE_3D( μ_j) (4) where ¯j=1|j|∑i:oi=ji μ_j= 1|G_j| _i:o_i=j μ_i is the object centroid. Readout queries readQ^read: K learnable tokens conditioned on the language instruction ℒL and current sub-task τt _t: kread=klearn+ProjL(TextEnc(ℒ))+Projτ(PET(τt)) q_k^read= q_k^learn+Proj_L(TextEnc(L))+Proj_τ(PE_T( _t)) (5) where klearn∈ℝd q_k^learn ^d are randomly initialized learnable embeddings, TextEnc is CLIP text encoder, and ProjLProj_L, ProjτProj_τ are linear projections. The unified query concatenates both groups: uni=[obs;read]∈ℝ(No+K)×dQ^uni=[Q^obs;\;Q^read] ^(N_o+K)× d. Fig. 5: Qualitative visualization on a LIBERO Long-10 task (“close the drawer, pick up the bowl, place it on the plate”). Top: RGB observations across six phases. Middle: Middle row shows 3D Gaussian memory rendered from the accumulated multi-step observations. Bottom: Learned ηk _k heatmap (blue == preserve, red == update aggressively). The drawer Gaussians turn red during closing, then the bowl turns red during grasping, while background remains blue—all learned from task-loss gradients without explicit programming. Stage 2: Self-Attention — Observation-Task Exchange. Self-attention enables bidirectional information flow within the unified query before accessing memory. Readout queries attend to observation tokens, “highlighting” task-relevant objects (instruction → observation). Conversely, observation tokens ground readout queries in the current scene state (observation → readout). The output is the Contextualized Unified Query ctxU^ctx (Fig. 4), in which observation tokens have become task-aware and readout queries have become perceptually grounded—ready for targeted memory access. Stage 3: Cross-Attention Against Memory. ctxU^ctx now queries the 3D Gaussian memory. Keys and values come from stored memory Gaussians mem∈ℝNm×dG^mem ^N_m× d (NmN_m: number of memory Gaussians). Each of the L UMA layers applies self-attention, cross-attention, then FFN, each with LayerNorm (LN) and residual connections: ^(l) Q^(l) =LN(SelfAttn((l−1))+(l−1)) =LN (SelfAttn(Q^(l-1))+Q^(l-1) ) (6) ¯(l) Q^(l) =LN(TimeAttn(^(l),mem)+^(l)) =LN (TimeAttn( Q^(l),G^mem)+ Q^(l) ) (7) (l) ^(l) =LN(FFN(¯(l))+¯(l)) =LN (FFN( Q^(l))+ Q^(l) ) (8) where ^(l) Q^(l) corresponds to ctxU^ctx in Fig. 4—the contextualized queries after self-attention exchange. The cross-attention incorporates temporal bias: =softmax(Q(memK)⊤/dk+T)A=softmax(QW_Q(G^memW_K) / d_k+B_T) and TimeAttn(,mem)=memVTimeAttn(Q,G^mem)=A\,G^memW_V, where Q,K,V∈ℝd×dkW_Q,W_K,W_V ^d× d_k (dk=d/Hd_k=d/H, H heads), TB_T modulates by temporal recency. BT∈ℝ(No+K)×NmB_T ^(N_o+K)× N_m is a learned relative-time bias computed from PETrel(tim,t)PE_T^rel(t_i^m,t) and shared across attention heads. Stage 4: Read and Write Streams. After L layers, the output splits: (L)=[obs,(L);read,(L)]Q^(L)=[Q^obs,(L);\;Q^read,(L)]. The same cross-attention that reads from memory also produces write signals, eliminating separate matching (e.g., Sinkhorn or Hungarian). Read stream → LLM input: mem=LinearProj(read,(L))∈ℝK×dLLMZ^mem=LinearProj(Q^read,(L)) ^K× d_LLM, where dLLMd_LLM is the LLM hidden dimension. Write stream → Memory update: The observation rows of the final cross-attention produce a soft correspondence matrix match=Attncross(L)[1:No,:]∈ℝNo×NmA^match=Attn^(L)_cross[1:N_o,:] ^N_o× N_m. This single matrix drives the entire memory lifecycle: (i) Evidence aggregation: wk=∑j=1NoAjkmatchw_k= _j=1^N_oA^match_jk measures match strength for each memory entry. (i) Position estimation: Observations are soft-aggregated: ~kobs=∑jAjkmatchjobs∑jAjkmatch+ϵ μ_k^obs= _jA^match_jk\, μ_j^obs _jA^match_jk+ε (9) where jobs μ_j^obs is the 3D position of observation j and ϵ=10−6ε=10^-6 prevents division by zero. (i) Learned update gate: A learned gate converts evidence strength into an update rate: ηk=σ⁡(MLPη([~kobs;wk;PETrel(tkm,t)])) _k=σ\! (MLP_η ([ h_k^obs;\,w_k;\,PE_T^rel(t_k^m,t)] ) ) (10) where ~kobs=∑jAjkmatchjobs,(L)/(wk+ϵ) h_k^obs= _jA^match_jk\, q_j^obs,(L)/(w_k+ε) is the attention-weighted observation feature for memory entry k, and σ⁡(⋅)σ(·) constrains ηk∈(0,1) _k∈(0,1): large values mean “trust new observation” (e.g., just-grasped object); small values mean “preserve accumulated estimate” (e.g., static background). Position updates as: knew=(1−ηk)kmem+ηk~kobs μ_k^new=(1- _k)\, μ_k^mem+ _k\, μ_k^obs (11) (iv) Existence management: Each Gaussian maintains existence probability ek∈(0,1)e_k∈(0,1), updated as eknew=σ⁡(logit(ek)+γ⋅log⁡(wk+ϵ))e_k^new=σ(logit(e_k)+γ· (w_k+ε)), where γ is a learnable coefficient. Gaussians with ek<θe=0.1e_k< _e=0.1 are pruned. (v) Novel object insertion: Observations with low match to all memory entries are inserted with probability pjnew=σ⁡(MLPins(jobs,(L))−λ⋅maxk⁡Ajkmatch)p_j^new=σ(MLP_ins( q_j^obs,(L))-λ· _kA^match_jk), where λ is a learnable sensitivity parameter. The complete UMA data flow is illustrated in Fig. 4. Why Unifying Read and Write Matters. A two-module pipeline (separate matching for update + separate Q-Former for readout) prevents bidirectional flow. UMA’s shared self-attention enables read-to-write coupling (updates bias toward task-relevant objects) and write-to-read coupling (LLM weights fresh vs. stale information). The unified gradient path ℒaction→mem→UMA→match,ηk,ekL_action ^mem ^match, _k,e_k enables the task loss to directly sculpt memory behavior. I-E LLM Backbone and Action Generation We deliberately adopt the same VLM backbone as OpenVLA [9], CogACT [11], and MemoryVLA [18]—the Prismatic VLM [7], which pairs DINOv2 [14] and SigLIP [23] vision encoders with a Llama-2-7B [21] LLM—to ensure that any performance difference is attributable solely to the 3D Gaussian memory and UMA, not to backbone strength. The LLM receives the concatenated token sequence =[mem;2D;L;S]Z=[Z^mem;\;Z^2D;\;Z^L;\;Z^S], where 2DZ^2D are visual tokens from the fused Prismatic vision encoder (DINOv2+SigLIP), LZ^L are language tokens, and SZ^S are proprioceptive state tokens from a 2-layer MLP. The LLM generates: (1) Action tokens: decoded by a diffusion-based action head into t=[Δ,Δ,g] a_t=[ p,\, r,\,g]; (2) Completion token: a binary signal triggering the UMA update. I-F End-to-End Training The total loss is ℒ=ℒaction+λcℒcomplete+λtℒtemporal+λuℒupdateL=L_action+ _cL_complete+ _tL_temporal+ _uL_update, where ℒactionL_action is the standard diffusion action-prediction objective, ℒcompleteL_complete is binary cross-entropy for sub-task completion, ℒtemporalL_temporal encourages coherent memory for static objects via stop-gradient consistency, and ℒupdateL_update supervises memory positions against ground-truth with task-relevance weighting. Gradients flow through ηk _k and matchA^match, directly teaching the memory how to update. Training is two-stage: pre-train F-3DGS (frozen), then jointly train UMA, projector, and LLM (LoRA [6]) on full-length episodes (2–10 steps, curriculum). IV Experiments IV-A Experimental Setup Benchmarks LIBERO [12]: We evaluate on all four suites—Spatial, Object, Goal and Long-10, covering 40 tasks with diverse spatial, semantic, and long-horizon challenges. Each suite contains 10 tasks evaluated over 20 trials. Results are averaged over three random seeds and then averaged across tasks. VLABench [25]: A recent large-scale benchmark (ICCV 2025) with 100 task categories requiring world knowledge, spatial understanding, semantic reasoning, and long-horizon planning. We evaluate on Track 1 (in-distribution skill acquisition, 20 primitive tasks) and Track 6 (long-horizon composite tasks, 10 tasks), reporting success rate (SR) and progress score (PS). Multi-view adaptation LIBERO’s default setup provides a single third-person camera. Since our F-3DGS encoder requires multi-view input, we render two additional virtual viewpoints from the simulator (lateral and overhead, ±30∘± 30 ) at each step. The same three-view setup is used for all our LIBERO experiments. For VLABench, we use the built-in multi-camera support (front + two side cameras). Baselines For LIBERO, we compare against 10 published VLA methods with official results: OpenVLA [9], π0 _0-FAST [15], π0 _0 [2], CogACT [11], MemoryVLA [18], SpatialVLA [16], TraceVLA [27], CoT-VLA [26], CronusVLA [10], and 4D-VLA [24]. For VLABench, we compare against π0 _0, π0 _0-FAST, and π0.5 _0.5 using their officially reported results. Our ablation: Ours w/o memory: Same Prismatic VLM backbone (Llama-2-7B + DINOv2 + SigLIP + diffusion head), no Gaussian memory—isolating the memory contribution. Implementation We use MVSplat [3] as our feed-forward 3D Gaussian Splatting (F-3DGS) encoder to generate M=8192M=8192 live Gaussians, followed by per-object pooling into No≈20N_o≈20 tokens. We use K=64K=64 readout queries, L=6L=6 UMA layers, d=768d=768, and a Prismatic VLM (Llama-2-7B, DINOv2+SigLIP). The memory cap is 32K Gaussians. MLPηMLP_η is a 2-layer network with 256 hidden units and ReLU. Training uses AdamW with lr=2×10−4=2× 10^-4, batch size 16, and 120K steps on 4×A100 GPUs. λc=1.0,λt=0.1,λu=0.5 _c=1.0, _t=0.1, _u=0.5. The Prismatic backbone is identical to that used by OpenVLA, CogACT, and MemoryVLA, ensuring a controlled comparison where the only architectural difference is the Gaussian memory and UMA. IV-B LIBERO Results TABLE I: LIBERO benchmark results (success rate %). †Results from MemoryVLA [18]. ∗Uses additional wrist-camera and proprioceptive inputs. “–” indicates not reported. Method Spatial Object Goal Long-10 OpenVLA† 84.7 88.4 79.2 53.7 TraceVLA 84.6 85.2 75.1 54.1 SpatialVLA 88.2 89.9 78.6 55.5 CoT-VLA 87.5 91.6 87.6 69.0 CronusVLA 90.1 94.7 91.3 68.7 π0 _0-FAST∗† 96.4 96.8 88.6 60.2 π0∗ _0^* 96.8 98.8 95.8 85.2 4D-VLA 93.8 92.8 95.6 86.5 CogACT† 97.2 98.0 90.2 88.8 MemoryVLA† 98.4 98.4 96.4 93.4 Ours w/o memory 92.4 93.6 88.2 82.4 Ours + 2D token memory 95.2 95.8 93.6 90.2 GaussMemory (3D mem.) 96.4 96.2 96.8 94.1 Table I compares GaussMemory against 10 published VLA methods. On the long-horizon suites where spatial precision matters most, GaussMemory outperforms all baselines: Goal 96.8% (vs. MemoryVLA 96.4%), Long-10 94.1% (vs. 93.4%). MemoryVLA retains an edge on Spatial/Object (98.4%) where short-horizon semantic recognition dominates. The 2D vs. 3D ablation confirms this pattern: replacing 3D Gaussians with 2D tokens yields 90.2% on Long-10—below both our 3D model and MemoryVLA—showing that explicit 3D geometry is the decisive factor. Memory benefit scales with task horizon. The gap over w/o memory grows monotonically: +4.0 (Spatial) → +8.6 (Goal) → +11.7 (Long-10), confirming that persistent 3D memory becomes increasingly critical as tasks accumulate more state changes. Fig. 6 shows GaussMemory ranks first on 9/10 individual Long-10 tasks. Fig. 6: Per-task breakdown on LIBERO Long-10. GaussMemory ranks first on 9/10 tasks. MemoryVLA leads only on T7 (semantic-heavy), consistent with the 3D-vs-2D complementarity pattern. IV-C VLABench Results TABLE I: VLABench [25] results. SR: success rate (%). PS: progress score (%). ‡Official VLABench repository results. Method Track 1 Track 6 SR PS SR PS π0.5 _0.5-ft‡ [1] 40.6 52.4 12.0 28.6 π0 _0-ft‡ 47.0 58.2 15.0 32.4 π0 _0-FAST-ft‡ 51.2 62.8 18.0 36.2 Ours w/o memory 46.8 57.6 14.0 30.8 Ours + 2D token memory 51.4 63.0 19.0 38.4 GaussMemory (3D Gauss. mem.) 56.4 68.2 24.0 45.6 Table I evaluates GaussMemory on VLABench. On Track 1, GaussMemory achieves 56.4% SR, outperforming π0 _0-FAST [15] (51.2%) by +5.2%. The advantage is larger on Track 6 (long-horizon composite): 24.0% vs. 18.0% (+6.0%). Notably, the 2D vs. 3D memory ablation isolates the representation effect: replacing our 3D Gaussian memory with a 2D token buffer (same UMA architecture, same backbone) yields 51.4% / 19.0% on Track 1/6—substantially below the full 3D model (56.4% / 24.0%) and only marginally above π0 _0-FAST. This confirms that the 3D geometric representation, not merely the presence of memory, is the key contributor. VLABench amplifies the memory advantage. VLABench tasks have longer horizons (500+ timesteps) than LIBERO (∼ 120). The improvement over π0 _0-FAST grows from +5.2% (Track 1) to +6.0% (Track 6), mirroring the “memory scales with horizon” pattern from LIBERO. Cross-benchmark consistency. Across both benchmarks, three patterns hold: (1) memory is the single largest factor; (2) the advantage grows with task horizon; (3) it is most pronounced on spatially demanding tasks. This consistency across fundamentally different task designs supports the generality of active 3D Gaussian memory. TABLE I: Ablation study on LIBERO Long-10 (success rate %). Configuration Long-10 Δ Prismatic VLA backbone (no memory) 82.4 – Passive → Active memory + 3D Gaussian Memory (fixed η=0.3η=0.3) 90.4 +8.0 + learned η (active update gate) 92.8 +10.4 + MLP-modulated η (full active gate) 93.2 +10.8 + soft existence management 94.1 +11.7 Unified vs. Decoupled read/write Decoupled (separate match + readout) 91.8 +9.4 UMA (unified read/write, ours) 94.1 +11.7 View count 2 views 91.4 +9.0 3 views (default) 94.1 +11.7 Memory capacity 4K Gaussians 78.2 −-4.2 8K Gaussians 86.4 +4.0 16K Gaussians 91.8 +9.4 32K Gaussians (default) 94.1 +11.7 64K Gaussians 93.6 +11.2 Table I reveals how each component contributes on LIBERO Long-10: 3D Gaussian memory is the foundation (passive memory): Simply adding persistent 3D Gaussian memory with a fixed update rate (η=0.3η=0.3) already yields +8.0 (82.4→ 90.4), the single largest gain. This “passive” memory treats every object equally—confirming that 3D geometric representation itself is the primary contributor. Passive→ : learned update strategy: Replacing fixed η with attention-derived wkw_k adds +2.4 (90.4→ 92.8)—this is the transition from passive (update everything equally) to active (update task-relevant objects more). The MLP gate (+0.4) and soft existence management (+0.9) further sharpen the active strategy, bringing the total to +11.7. The emergent η hierarchy (0.74 for manipulated targets, 0.06 for background) is the direct outcome of this learned active update (Fig. 9). Unified vs. Decoupled: A decoupled baseline (separate matching + separate Q-Former readout, matched parameters) achieves 91.8%, 2.3 points below UMA (94.1%)—confirming that unified read/write coupling itself contributes beyond the 3D representation alone. View count robustness (Fig. 7): The memory contribution far exceeds the view-count contribution. At 3-view, memory adds +11.7 on Long-10 while adding a third view (2→ 3) only adds +1.6 without memory—confirming gains come from 3D memory, not extra cameras. Fig. 7: Multi-view fairness. At every view count, the memory contribution (red annotation) vastly exceeds the view contribution (gray annotation), confirming gains come from 3D memory, not extra cameras. Memory capacity (Fig. 8): Performance scales with Gaussian count up to 32K, beyond which marginal returns diminish (64K: 93.6%). The steep rise from 4K to 16K indicates sufficient capacity is critical; the plateau at 32K→ 64K confirms the learned existence management prevents redundant Gaussians. Fig. 8: Gaussian memory capacity ablation. Performance scales steeply up to 32K (our default) then plateaus, indicating effective learned pruning. IV-D Emergent Memory Behavior Fig. 9: Temporal ηk _k on a LIBERO Long-10 episode. The drawer’s η peaks during closing, then the bowl’s η rises during grasping. Background maintains η≈0.06η≈ 0.06 throughout—learned through end-to-end gradient flow. Fig. 5 and Fig. 9 visualize the emergent update strategy: manipulated objects receive η≈0.74η≈ 0.74 while background stays at η≈0.06η≈ 0.06-the update strategy emerges without hand-crafted update rules, driven by task-level action supervision and auxiliary memory-consistency losses. Fig. 10 shows the memory size dynamics: insertion spikes during scene changes while pruning maintains equilibrium well below the 32K cap. Fig. 10: Memory size dynamics on a LIBERO Long-10 episode. Top: active Gaussian count stays well below the 32K cap. Bottom: insertion (green) spikes during scene changes; pruning (red) maintains equilibrium. V Conclusion We presented GaussMemory, which shifts robot spatial memory from passive recording to active, task-driven learning via 3D Gaussian Splatting and UMA. The system learns what matters—manipulated objects are updated aggressively while background is left untouched—all discovered automatically from task performance. On LIBERO, GaussMemory outperforms MemoryVLA on Goal and Long-10. References [1] K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. R. Equi, C. Finn, N. Fusai, M. Y. Galliker, et al. (2025) π0.5 _0.5: A vision-language-action model with open-world generalization. In 9th Annual Conference on Robot Learning, Cited by: TABLE I. [2] K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, et al. (2024) π0 _0: A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164. Cited by: §I, §IV-A. [3] Y. Chen, H. Xu, C. Zheng, B. Zhuang, M. Pollefeys, A. Geiger, T. Cham, and J. Cai (2024) MVSplat: efficient 3D Gaussian Splatting from sparse multi-view images. In ECCV, Cited by: §IV-A. [4] Q. Gu, A. Kuwajerwala, S. Morin, K. M. Jatavallabhula, B. Sen, A. Agarwal, C. Rivera, W. Paul, K. Ellis, A. Cherian, et al. (2024) ConceptGraphs: open-vocabulary 3D scene graphs for perception and planning. In ICRA, Cited by: §I, §I-A. [5] A. Halacheva, J. Zaech, X. Wang, D. P. Paudel, and L. Van Gool (2025) Gaussianvlm: scene-centric 3d vision-language models using language-aligned gaussian splats for embodied reasoning and beyond. IEEE Robotics and Automation Letters. Cited by: §I, §I-B. [6] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen (2022) LoRA: low-rank adaptation of large language models. In ICLR, Cited by: §I-F. [7] S. Karamcheti, S. Nair, A. Balakrishna, P. Liang, T. Kollar, and D. Sadigh (2024) Prismatic VLMs: investigating the design space of visually-conditioned language models. In ICML, Cited by: §I-E. [8] B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis (2023) 3D Gaussian Splatting for real-time radiance field rendering. In SIGGRAPH, Cited by: §I, §I-B. [9] M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, et al. (2024) OpenVLA: an open-source vision-language-action model. In CoRL, Cited by: §I, §I-E, §IV-A. [10] H. Li, S. Yang, Y. Chen, X. Chen, X. Yang, Y. Tian, H. Wang, T. Wang, D. Lin, F. Zhao, et al. (2026) Towards efficient and robust manipulation via multi-frame vision-language-action modeling. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40, p. 18388–18396. Cited by: §IV-A. [11] Q. Li, Y. Liang, Z. Wang, L. Luo, X. Chen, M. Liao, F. Wei, Y. Deng, S. Xu, Y. Zhang, et al. (2024) CogACT: a foundational vision-language-action model for synergizing cognition and action in robotic manipulation. arXiv preprint arXiv:2411.19650. Cited by: §I-E, §IV-A. [12] B. Liu, Y. Zhu, C. Gao, Y. Feng, Q. Liu, Y. Zhu, and P. Stone (2023) LIBERO: benchmarking knowledge transfer for lifelong robot learning. In NeurIPS, Cited by: §IV-A. [13] G. Lu, S. Zhang, Z. Wang, C. Liu, J. Lu, and Y. Tang (2024) ManiGaussian: dynamic Gaussian Splatting for multi-task robotic manipulation. In ECCV, Cited by: §I-B. [14] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, et al. (2024) DINOv2: learning robust visual features without supervision. TMLR. Cited by: §I-E. [15] K. Pertsch, K. Stachowicz, B. Ichter, D. Driess, S. Nair, Q. Vuong, O. Mees, C. Finn, and S. Levine (2025) Fast: efficient action tokenization for vision-language-action models. arXiv preprint arXiv:2501.09747. Cited by: §IV-A, §IV-C. [16] D. Qu, H. Song, Q. Chen, Y. Yao, X. Ye, Y. Ding, Z. Wang, J. Gu, B. Zhao, D. Wang, et al. (2025) SpatialVLA: exploring spatial representations for visual-language-action model. arXiv preprint arXiv:2501.15830. Cited by: §IV-A. [17] K. Rana, J. Haviland, S. Garg, J. Abou-Chakra, I. Reid, and N. Suenderhauf (2023) SayPlan: grounding large language models using 3D scene graphs for scalable robot task planning. In CoRL, Cited by: §I, §I-A. [18] H. Shi, B. Xie, Y. Liu, L. Sun, F. Liu, T. Wang, E. Zhou, H. Fan, X. Zhang, and G. Huang (2026) MemoryVLA: perceptual-cognitive memory in vision-language-action models for robotic manipulation. In ICLR, Cited by: §I, §I-A, §I-E, §IV-A, TABLE I. [19] M. Tancik, P. Srinivasan, B. Mildenhall, S. Fridovich-Keil, N. Raghavan, U. Singhal, R. Ramamoorthi, J. Barron, and R. Ng (2020) Fourier features let networks learn high frequency functions in low dimensional domains. Advances in neural information processing systems 33, p. 7537–7547. Cited by: §I-C. [20] M. Torne et al. (2026) MEM: multi-scale embodied memory for vision-language-action models. arXiv preprint arXiv:2603.03596. Cited by: §I. [21] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. (2023) Llama 2: open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. Cited by: §I-E. [22] M. Ye, M. Danelljan, F. Yu, and L. Ke (2024) Gaussian grouping: segment and edit anything in 3D scenes. In ECCV, Cited by: §I-C. [23] X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer (2023) Sigmoid loss for language image pre-training. In ICCV, Cited by: §I-E. [24] J. Zhang, Y. Chen, Y. Xu, Z. Huang, Y. Zhou, Y. Yuan, X. Cai, G. Huang, X. Quan, H. Xu, et al. (2025) 4D-VLA: spatiotemporal vision-language-action pretraining with cross-scene calibration. In NeurIPS, Cited by: §IV-A. [25] S. Zhang, Z. Xu, P. Liu, et al. (2025) VLABench: a large-scale benchmark for language-conditioned robotics manipulation with long-horizon reasoning tasks. In ICCV, Cited by: §IV-A, TABLE I. [26] Q. Zhao, Y. Lu, M. J. Kim, Z. Fu, Z. Zhang, Y. Wu, Z. Li, Q. Ma, S. Han, C. Finn, et al. (2025) CoT-VLA: visual chain-of-thought reasoning for vision-language-action models. In CVPR, Cited by: §IV-A. [27] R. Zheng, Y. Liang, S. Huang, J. Gao, H. Daumé I, A. Kolobov, F. Huang, and J. Yang (2025) TraceVLA: visual trace prompting enhances spatial-temporal awareness for generalist robotic policies. In International Conference on Learning Representations, Cited by: §IV-A.

Instant research discovery