Paper deep dive

RenderMem: Rendering as Spatial Memory Retrieval

JooHyun Park, HyeongYeop Kang

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 46

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%

Last extracted: 3/22/2026, 5:13:01 AM

Summary

RenderMem is a spatial memory framework for embodied AI that treats rendering as a query-conditioned memory retrieval operation. By maintaining a 3D scene representation and generating visual evidence from specific viewpoints, it enables vision-language models to perform geometrically grounded reasoning about visibility and occlusion, outperforming traditional view-based and object-centric memory systems.

Entities (4)

AI2-THOR · environment · 100%RenderMem · framework · 100%Spatial Memory · concept · 95%Vision-Language Models · technology · 95%

Relation Signals (3)

RenderMem → uses → 3D scene representation

confidence 98% · RenderMem maintains a 3D scene representation and generates query-conditioned visual evidence by rendering the scene.

RenderMem → improvesperformancein → AI2-THOR

confidence 95% · Experiments in the AI2-THOR environment show consistent improvements on viewpoint-dependent visibility and occlusion queries over prior memory baselines.

RenderMem → integrateswith → Vision-Language Models

confidence 95% · RenderMem is fully compatible with existing vision–language models and requires no modification to standard architectures.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Embodied reasoning is inherently viewpoint-dependent: what is visible, occluded, or reachable depends critically on where the agent stands. However, existing spatial memory systems for embodied agents typically store either multi-view observations or object-centric abstractions, making it difficult to perform reasoning with explicit geometric grounding. We introduce RenderMem, a spatial memory framework that treats rendering as the interface between 3D world representations and spatial reasoning. Instead of storing fixed observations, RenderMem maintains a 3D scene representation and generates query-conditioned visual evidence by rendering the scene from viewpoints implied by the query. This enables embodied agents to reason directly about line-of-sight, visibility, and occlusion from arbitrary perspectives. RenderMem is fully compatible with existing vision-language models and requires no modification to standard architectures. Experiments in the AI2-THOR environment show consistent improvements on viewpoint-dependent visibility and occlusion queries over prior memory baselines.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

45,948 characters extracted from source content.

Expand or collapse full text

RenderMem: Rendering as Spatial Memory Retrieval JooHyun Park 1 and HyeongYeop Kang 1 Korea University, South Korea Abstract. Embodied reasoning is inherently viewpoint-dependent: what is visible, occluded, or reachable depends critically on where the agent stands. However, existing spatial memory systems for embodied agents typically store either multi-view observations or object-centric abstrac- tions, making it difficult to perform reasoning with explicit geometric grounding. We introduce RenderMem, a spatial memory framework that treats rendering as the interface between 3D world representations and spatial reasoning. Instead of storing fixed observations, RenderMem maintains a 3D scene representation and generates query-conditioned visual evidence by rendering the scene from viewpoints implied by the query. This enables embodied agents to reason directly about line-of- sight, visibility, and occlusion from arbitrary perspectives. RenderMem is fully compatible with existing vision–language models and requires no modification to standard architectures. Experiments in the AI2-THOR environment show consistent improvements on viewpoint-dependent vis- ibility and occlusion queries over prior memory baselines. Keywords: Embodied AI· Spatial Memory 1 Introduction Embodied reasoning is fundamentally viewpoint-dependent. In the physical world, perception is spatially situated: what an agent can observe depends on where it stands, and the same scene can yield entirely different evidence under different viewpoints. Because action decisions are grounded in what is visible from a spe- cific location, visibility directly constrains movement, manipulation, and task execution. Viewpoint-conditioned reasoning is therefore intrinsic to embodied intelligence. Consider a robot responsible for inspecting safety conditions in a building. To ensure emergency equipment can be located quickly, it may need to reason: “From the main corridor, is the fire extinguisher visible, or is it occluded by the cabinet?” Answering such questions requires interpreting the corridor as a reference viewpoint and evaluating line-of-sight between objects. This is not merely object recognition—it is geometric reasoning over spatial configuration. arXiv:2603.14669v1 [cs.AI] 15 Mar 2026 2J. Park et al. Yes, it is visible. No, it is not turned on. Rendering as Spatial Memory Retrieval Render Render Reasoning & AnswerReasoning & Answer Is the basketball visible from the alarm clock? I need a view of the basketball from the alarmclock I need a view of the TV Is the TV turned on? Fig. 1: RenderMem retrieves spatial evidence from a stored 3D scene by rendering query-conditioned views, which serve as visual memory readouts for vision–language reasoning about visibility and object state. Despite recent progress, existing spatial memory architectures struggle with such viewpoint-conditioned queries. Current approaches largely fall into three paradigms, each sharing a structural limitation. View-based memory [39,43] stores observations from discrete viewpoints and retrieves images for downstream reasoning. While effective for recognizing visible attributes, it is constrained by previously captured views and cannot reliably answer queries requiring novel or object-centric viewpoints. Object-centric memory [3,10] represents scenes as collections of objects and relations. Although compact and suitable for relational reasoning, it typically lacks explicit modeling of camera pose and line-of-sight geometry, making visi- bility reasoning difficult. 3D scene representations [13, 44] encode richer geometric structure through meshes, volumetric maps, or neural fields. Yet their high-dimensional nature makes direct integration with language models challenging, often weakening the link between geometry and reasoning. These limitations point to a deeper issue: existing memory systems retrieve stored observations or abstract relations, but do not actively generate viewpoint- specific visual evidence required by a query. We argue that resolving viewpoint- dependent reasoning requires rethinking memory access itself. In this work, we introduce RenderMem, a rendering-based spatial memory framework built on a simple but powerful principle: rendering is the read opera- tion of 3D memory. RenderMem maintains a persistent 3D scene representation and, when faced with a viewpoint-conditioned query, performs query-conditioned rendering to synthesize exactly the visual evidence needed for reasoning, as illus- RenderMem3 trated in Fig. 1. Instead of forcing language models to interpret raw 3D geometry, RenderMem translates geometry into images aligned with the query’s specified viewpoint, which are then processed by standard vision–language models. By turning rendering into a first-class memory primitive, RenderMem enables ex- plicit, geometrically grounded visibility and occlusion reasoning that prior mem- ory paradigms cannot reliably support. Importantly, this design is not limited to static scenes. Because memory is maintained as an updatable 3D representation, any modification to object state or geometry is immediately reflected in subsequent query-conditioned render- ings. Thus, RenderMem naturally extends to dynamic environments, serving as a spatial memory that evolves consistently with scene changes rather than requiring explicit rewriting of stored observations. Our contributions are as follows: – We identify viewpoint-dependent visibility and occlusion reasoning as a fun- damental and under-addressed bottleneck in embodied spatial memory. – We propose RenderMem, introducing the abstraction of rendering as a query- conditioned memory read operation, enabling geometrically grounded rea- soning without modifying existing vision–language architectures. – We develop query-conditioned viewpoint synthesis strategies that explicitly support visibility and occlusion reasoning, bridging the gap between 3D ge- ometry and language-based inference. – Experiments show that RenderMem achieves superior performance over view- based and object-centric memory baselines while remaining robust under simulated reconstruction artifacts (blur, ghosting, and bounding-box noise). 2 Related Work 2.1 Embodied Agents with Vision-Language Models Embodied agents aim to perceive, reason, and act in interactive 3D environ- ments. Early embodied systems primarily relied on modular pipelines with task- specific perception, mapping, and planning components, particularly for navi- gation and object search tasks [2,5,11]. While effective in constrained settings, these approaches struggled to generalize to open-ended tasks due to rigid sym- bolic representations and limited semantic understanding. Recent advances in large language models (LLMs) and vision–language mod- els (VLMs) have substantially reshaped embodied intelligence. Pretrained mul- timodal models such as CLIP [34], BLIP [27], BLIP-2 [26], Flamingo [1], and LLaVA [28] demonstrate strong zero-shot generalization across diverse visual and linguistic tasks. These models enable embodied agents to interpret high- level instructions, reason over observations, and generalize across environments without task-specific retraining. Building on these capabilities, recent works on embodied agents leverage language models for action planning and decision-making grounded in visual observations. LEO [16] introduces a multimodal embodied agent that integrates 4J. Park et al. 3D perception with language models to perform diverse tasks such as grounding, reasoning, and navigation in 3D environments. STEVE [45] combines visual per- ception modules with LLM-based reasoning to generate action plans for complex tasks. Octopus [42] formulates embodied control as program generation, where a vision–language model produces executable action code and improves behavior through environmental feedback. Progress in this direction is driven by embodied navigation and question an- swering benchmarks. Benchmarks such as Embodied Question Answering (EQA) [7], ScanQA [4], SQA3D [29], OpenEQA [30], and EXPRESS-Bench [19] evalu- ate an agent’s ability to integrate perception, memory, and reasoning over long interaction horizons. Exploration-centric benchmarks such as Explore-EQA [35] and GOAT-Bench [22] further emphasize long-horizon cognition and memory utilization. However, enabling such long-horizon, grounded reasoning requires a spatial memory system that can reliably store, organize, and retrieve perceptual information. 2.2 Spatial Memory Systems for Embodied Agents Spatial memory is a central component of embodied reasoning. One line of work adopts view-based memory, where agents retain RGB or RGB-D observations captured during exploration. 3D-Mem [43] organizes multi-view observations into memory snapshots that capture co-visible objects and scene context for VLM- based reasoning. LMEE [39] stores observed images and object metadata that support exploration and question answering. However, reliance on previously observed viewpoints can fail when queries require reasoning from novel or object- centric perspectives. Object-centric memory representations provide more structured abstractions. Scene graph–based methods [3,36] represent environments as objects connected by spatial relations, supporting relational reasoning and planning. Hierarchical and open-vocabulary extensions such as HOV-SG [40] and ConceptGraphs [10] further improve scalability and expressiveness. Real-time spatial perception sys- tems such as Hydra [17] enable online construction and optimization of such graphs. Nevertheless, graph abstractions often quantize geometry into coarse re- lations or sparse descriptors, making it difficult to answer viewpoint-dependent visibility queries without explicitly modeling rendering or ray-based reasoning. Another line of work incorporates semantic representations from pretrained VLMs into 3D scene representations. Methods such as VLMaps [14], Open- Scene [33], ConceptFusion [18], and CLIP-Fields [44] lift language-aligned visual features into 3D maps or neural scene representations, enabling open-vocabulary querying. However, because most vision–language models used by embodied agents operate on 2D image inputs, directly leveraging these 3D representations for reasoning remains challenging. To address this limitation, recent work injects 3D features directly into language models, including 3D-LLM [12], L3DA [6], Chat-Scene [15], SplatTalk [38], and 3DLLM-Mem [13]. While these approaches improve semantic grounding, they are constrained by context length and com- putational cost, often requiring aggressive subsampling or pooling that discards RenderMem5 fine-grained geometric details. In addition, these representations typically aggre- gate features in a direction-agnostic manner, making them ill-suited for relational queries that depend on specific viewpoints. Overall, while prior spatial memory systems have made substantial progress in long-horizon reasoning and semantic grounding, they remain limited in sup- porting viewpoint-dependent visibility. This limitation arises from a mismatch between how scenes are represented in memory and how embodied queries spec- ify perspectives. RenderMem addresses this gap by treating rendering itself as a memory operation, generating query-conditioned visual evidence from object- centric viewpoints that can be directly consumed by vision-language models. 3 Method RenderMem is built on a simple but powerful abstraction: rendering is the read operation of spatial memory. Rather than treating rendering as a visualization tool, we elevate it to a first-class mechanism for querying geometry. This design decouples spatial reasoning from language reasoning while preserving compati- bility with existing vision–language models. Instead of storing fixed observations or converting raw 3D representations into tokens, RenderMem maintains a persistent, renderable 3D scene state and generates visual evidence only when required by a query. In this framework, a question does not retrieve stored images–it triggers viewpoint-conditioned ren- dering that computes the exact visual evidence necessary for reasoning about visibility and occlusion. Given a question, RenderMem executes a structured two-stage pipeline that determines (1) whether rendering is required, (2) what type of rendering method is most appropriate and which objects should guide the rendering. The rendered images are then provided to a vision–language model together with the original question to produce the final answer. The pipeline is visualized in Fig. 2. 3.1 Scene Representation RenderMem assumes access to a renderable 3D scene representation that serves as a persistent and unified spatial memory of the environment. In practical de- ployments, such scene representations can be incrementally constructed and updated using established mapping pipelines. Camera trajectories may be re- covered via SLAM [23, 25], while object instances are detected, localized, and tracked over time. The underlying geometry can be maintained in a renderable form using mesh reconstruction [20], neural radiance fields [32], or 3D Gaussian Splatting [21]. As new observations arrive, the scene estimate is progressively refined to reflect the current state of the environment. Importantly, RenderMem is agnostic to the specific reconstruction backend and assumes only access to the current scene estimate. Rather than storing historical observations, the system retrieves spatial evidence through query- conditioned rendering of the latest scene state. Because rendering operates di- rectly on the current geometry, spatial evidence is always derived from the most 6J. Park et al. (a) Renderable3D scene (b) Object list 푂 [표 0 ,표 1 , 표 2 , ...] Input User Question 푞 푡 Rendering Specification Visual Reasoning SurroundDirectional Answer ො푦 Rendering Surround Directional 푓 푠푝푒푐 푞 푡 ,푂 푓 푉퐿푀 푞 푡 ,ℐ Rendering Decision ො푦 Request rendering 푓 푖푛푖푡 (푞 푡 ,푂) ... ℐ Fig. 2: Overview of the RenderMem pipeline. (a) A renderable 3D scene with an ob- ject list serves as spatial memory. (b) Given a question, RenderMem decides whether rendering is needed and selects a rendering mode and object anchors. Surround ren- dering captures multiple views around an object, while directional rendering generates a source-to-target viewpoint for visibility reasoning. The rendered images are used to answer the user question. up-to-date map. Consequently, when the underlying scene representation evolves due to environmental changes, RenderMem naturally adapts without requiring explicit memory updates. Given such a renderable scene state, we introduce a lightweight object-level abstraction that supports query-conditioned viewpoint specification. This ab- straction provides stable geometric anchors for camera placement while avoiding direct exposure of raw 3D geometry to the language model. Formally, we repre- sent the scene as a set of N objects O =o i N i=1 (1) where each object o i is represented as o i = (id i , s i )(2) Here, id i denotes a unique identifier composed of an object category and an index (e.g., Chair_0), and s i denotes a bounding sphere encoding the object’s spatial extent. To balance geometric expressiveness and efficiency, each object is approxi- mated by a bounding sphere derived from its axis-aligned bounding box. Given the eight bounding-box corner points p ij 8 j=1 ⊂R 3 , the object center and ra- dius are computed as c i = 1 8 8 X j=1 p ij ,(3) r i = max 1≤j≤8 ∥p ij − c i ∥ 2 .(4) RenderMem7 This spherical approximation captures the object’s position and scale while re- maining invariant to orientation and topology. 3.2 Question-Answering Pipeline RenderMem answers each user question q t posed over the current scene by issuing a sequence of internal queries that determine how visual evidence should be retrieved. While the question represents the task posed over the scene, the queries correspond to intermediate decisions that control the rendering process. This procedure consists of two stages: (1) determining whether rendering is necessary, and (2) if so, producing a query-conditioned rendering specification. The final answer is then obtained by applying a vision–language model to the rendered views. Query 1: Rendering Decision. Given a question q t and the current object listO, RenderMem first issues an internal query to determine whether explicit visual evidence is required or whether the question can be answered directly from O: f init (q t ,O) = ( ˆy,if answerable, request_rendering, otherwise. (5) where ˆy denotes the predicted answer. This gating step avoids unnecessary ren- dering for queries that can be answered directly from the object list, such as counting object instances (e.g., “How many chairs are in the room?”). Query 2: Rendering Specification. If rendering is requested, RenderMem issues a second internal query that produces a structured rendering specification. This specification jointly determines the rendering mode and the object anchor(s) used to guide camera placement: f spec (q t ,O) = ρ = (m,A),(6) where m ∈ surround, directional and A denotes the mode-dependent object anchors: A = ( o i , m = surround, (o s ,o t ), m = directional. (7) The surround mode captures multiple views around a single object and is suit- able for queries that require observing object attributes or states. The directional mode supports viewpoint-dependent reasoning such as visibility and occlusion, where the viewpoint is instantiated relative to a source object o s and a target object o t . By specifying object anchors rather than raw geometry, the rendering process focuses on spatial regions relevant to q t while introducing a structured abstraction that is more amenable to language-based reasoning. This abstrac- tion enables the system to translate symbolic queries into geometrically grounded renderings while maintaining a compact interface between the 3D scene repre- sentation and the language model. Algorithmic details are provided in Sec. 3.3. 8J. Park et al. Scene Rendering. Given the rendering specification ρ, the renderer produces a set of images I = Render(S,O,ρ),(8) where S denotes the underlying renderable scene state. The rendering process instantiates camera viewpoints relative to the selected object anchors and pro- duces views that capture spatial relationships relevant to the question. Evidence-Based Reasoning. The rendered images I serve as query-conditioned visual evidence grounded in the current scene geometry. These images are paired with the original question and passed to a vision–language model to produce the final answer: ˆy = f VLM (q t ,I).(9) By transforming symbolic questions into rendered observations, RenderMem enables the vision–language model to reason over explicit visual evidence grounded in the scene geometry. 3.3 Rendering Algorithms We formulate rendering as the problem of selecting camera parameters: a camera position x ∈R 3 and a viewing direction v ∈R 3 . RenderMem uses camera placement as a mechanism for producing query-conditioned observations that support grounded reasoning about the scene. Under this formulation, different rendering strategies correspond to different constraints on how the camera parameters (x,v) are chosen. Intuitively, the goal is not merely to capture the scene but to synthesize views that expose the spatial relationships required to answer the question. RenderMem instantiates this idea using two complementary rendering modes: surround rendering, which provides contextual observations around a target object, and directional rendering, which simulates viewpoint-specific observations between objects for visibility and oc- clusion reasoning. Surround Rendering. Surround rendering generates multiple views around a target object to provide sufficient visual evidence about its appearance and sur- rounding context. Given the object sphere (c i ,r i ), we compute an enclosing camera sphere (c,R cam ), where c denotes the object center and R cam represents the minimum radius required to place cameras outside the object geometry. To ensure that the entire object remains fully visible in the rendered views, we first compute the minimum camera distance required for the object sphere to fit within the view frustum. Let FOV v denote the vertical field of view and aspect the image aspect ratio. Since both vertical and horizontal field-of-view constraints must be satisfied, we define an effective half field-of-view angle β = min FOV v 2 , arctan tan FOV v 2 · aspect .(10) RenderMem9 Using basic trigonometric relations, the minimum camera distance from the ob- ject center that guarantees full object visibility is d min = r i sin(β) . (11) To control the amount of contextual information included in the rendered images, we scale this minimum distance with a factor α d = αd min .(12) Larger values of α increase the viewing distance and reveal more surrounding context, while smaller values produce tighter object-centric views. In practice, single viewpoints may fail to capture relevant evidence due to oc- clusions caused by nearby objects. To improve robustness, RenderMem samples K camera poses uniformly around the object along the azimuth direction while maintaining a fixed elevation angle φ. This strategy ensures that at least some views expose the target object without occlusion while still preserving consistent spatial context across views. For azimuth angles θ i = 2πi K , the viewing direction is v i = normalize((p cosθ i + q sinθ i ) cosφ + u sinφ).(13) where u denotes the world up vector and p,q form an orthonormal basis span- ning the horizontal plane. Each camera is placed at x i = c + dv i .(14) and oriented to look toward the object center c. Directional Rendering. Directional rendering uses a single camera pose to rea- son about visibility and occlusion from a source object toward a target object. Given the source sphere (c s ,r s ) and the target center c t , the camera is placed on the surface of the source sphere at the point closest to the target to avoid self-occlusion by the source object, and oriented toward the target center. This configuration approximates the viewpoint from the source object toward the tar- get, enabling reasoning about visibility and occlusion between the two objects. 4 Experiments 4.1 Benchmark We build our benchmark on indoor environments from AI2-THOR [24], includ- ing iTHOR [24], RoboTHOR [8], and ProcTHOR [9] scenes. The benchmark is designed to evaluate different levels of spatial reasoning required for embodied agents. The benchmark consists of three subsets: 1) Static Attribute and Count subset evaluates object-centric reasoning in static scenes, 2) Dynamic Attribute and Count subset evaluates reasoning over state changes caused by interactions, and 3) Viewpoint-Dependent Visibility subset evaluates whether a target object is visible from a specified spatial location, requiring viewpoint-conditioned geo- metric reasoning. The benchmark spans 180 scenes across multiple AI2-THOR environments. Benchmark details are provided in the supplementary. 10J. Park et al. 4.2 Binary LLM Match Evaluation Evaluating open-ended question answering with language models presents a chal- lenge: although ground-truth answers are represented in canonical forms (e.g., yes/no or numeric counts), model predictions are often expressed in uncon- strained natural language. For instance, a model may answer “closed" instead of “yes”, or “there are two chairs” instead of “2”. While semantically correct, such responses would be marked incorrect under exact string matching. To address this issue, we adopt a binary variant of the LLM Match met- ric [30]. Instead of strict string comparison, an external evaluation LLM deter- mines whether a predicted answer ˆy correctly answers the original question given the ground-truth answer y: Match(ˆy,y) = ( 1, if ˆy is semantically identical to y, 0, otherwise. (15) 4.3 Baselines We compare RenderMem against three representative classes of baselines: Multi- view retrieval [34, 41], Concept Graphs [10], and 3D-Mem [43]. Multi-view re- trieval stores all explored views and retrieves the most relevant images for each question using an image–text similarity model [34]. Concept Graphs represent the scene as an object-centric graph with textual descriptions of object instances and perform reasoning directly over these descriptions. 3D-Mem maintains a hy- brid memory consisting of object lists and view-based snapshots that associate images with object instances and spatial attributes. Implementation details are provided in the supplementary. 4.4 Comparative Analysis We compare RenderMem with representative spatial memory approaches to un- derstand how different memory representations affect spatial question answering. All methods are evaluated on the same QA sets, and answers are generated using the same Qwen2.5-VL-7B [37] model to ensure a fair comparison. Performance is reported using the averaged binary LLM Match score. Results are reported in Tab. 1. Object QA This task evaluates reasoning over object attributes (e.g., “Is the TV turned on?”) and object counts (e.g., “How many chairs are in the room?”) in a static environment. Multi-view retrieval performs reasonably on attribute questions by leveraging visual cues, but struggles on counting since raw images do not provide explicit instance aggregation. Concept Graphs improve counting through object-level nodes, yet performance on attribute questions suffers when relevant visual de- tails are not preserved in the generated descriptions. 3D-Mem achieves strong counting via explicit instance tracking, yet attribute accuracy remains limited RenderMem11 Table 1: Comparison across spatial reasoning tasks. Object QAVisibility QA MethodAttribute Count RoboTHOR ProcTHOR Avg Multi-view retrieval0.690.250.500.490.50 Concept Graphs0.600.53– 3D-Mem0.680.780.420.430.43 RenderMem (ours)0.820.780.810.770.79 because the snapshots are optimized to capture many objects per view, often diluting object-specific visual cues. RenderMem achieves the best overall performance by generating views con- ditioned on the query, preserving both object completeness and detailed visual cues. Visibility QA We further evaluate viewpoint-conditioned visibility reasoning (e.g., “Is the TV visible from the sofa?”). Multi-view retrieval achieves mod- erate performance because retrieved views often contain both queried objects but are not aligned with the requested viewpoint. 3D-Mem suffers from lim- ited viewpoint coverage since it preserves only a compact subset of snapshots, reducing the likelihood of retrieving a view consistent with the queried spatial configuration. In contrast, RenderMem explicitly renders views conditioned on the queried spatial relation. By generating a camera pose that corresponds to the specified viewpoint, the method provides geometrically aligned visual evidence for the vision–language model. This significantly reduces ambiguity in visibility reason- ing and leads to large performance gains across both RoboTHOR and Proc- THOR scenes. 4.5 Support for Dynamic Scenes Beyond static scenes, we analyze how RenderMem behaves in dynamic envi- ronments where object states change due to interactions. Unlike conventional memory systems that explicitly maintain and update scene states, RenderMem generates visual evidence at query time from the current scene representation. As a result, changes in object appearance are immediately reflected in the rendered observations without requiring explicit memory updates. We evaluate this property using the Dynamic Attribute and Count dataset, which consists of interaction–question pairs where each interaction modifies the scene state before the question is asked (e.g., turning on a TV followed by “Is the TV turned on?”). The interaction types follow the affordances provided by AI2-THOR, including clean, dirty, break, slice, and toggle actions. Our analysis reveals that RenderMem performs robustly in dynamic environ- ments where object states change due to interactions. In dynamic scenes, Ren- derMem achieves 0.92 attribute accuracy and 0.82 counting accuracy, compared 12J. Park et al. average score reconstruction fidelity (δ)localization perturbation (λ) (a)(b) Fig. 3: RenderMem’s robustness to imperfect scene representations. (a) Perfor- mance under decreasing reconstruction fidelity, simulated with blur-only (bo) and blur+ghosting (bg). (b) Performance under increasing localization perturbation ap- plied to object bounding spheres. to 0.82 and 0.78 in static scenes. Interestingly, performance is slightly higher in the dynamic setting. We speculate that interactions often produce informative state changes, making object attributes more visually distinctive (e.g., toggling a TV or opening a container). Further details are provided in the supplementary. 4.6 Multi-Step Querying We investigate how the decision process for rendering should be structured. A single query must determine three factors simultaneously: whether rendering is required, how to render (mode), and what to render (object anchors). Solving these decisions jointly can impose a substantial reasoning burden on the language model. To study the effect of structured decomposition, we evaluate three strategies that progressively separate these decisions. The 1-step setting predicts render- ing necessity, rendering mode, and object anchors in a single query. The 2-step setting first determines whether rendering is required, and if so, jointly predicts rendering mode and object anchors. The 3-step setting resolves rendering neces- sity, rendering mode, and object anchors through three sequential queries. Across all tasks, all strategies show comparable LLM Match score. The 2-step formulation performs best (0.80 / 0.87 / 0.79 on static, dynamic, and visibility tasks), improving over the 1-step formulation (0.78 / 0.85 / 0.78) while main- taining the coupling between rendering mode and object anchors. The 3-step formulation slightly degrades visibility performance (0.80 / 0.86 / 0.77), sug- gesting that fully separating these decisions can break dependencies that jointly determine the camera pose. We therefore adopt the two-step formulation in the final system. RenderMem13 4.7 Robustness to Imperfect Scene Representations Real-world scene representations are rarely perfect due to sensor noise, incom- plete reconstruction, or localization errors. To assess practical robustness, we evaluate RenderMem on object QA under both static and dynamic conditions, and on viewpoint-dependent visibility QA in static scenes, as visibility reason- ing is independent of scene interactions. We simulate two common imperfections: degraded reconstruction fidelity and inaccurate object localization. Reconstruction Fidelity. We first study robustness to low-quality scene recon- structions by corrupting rendered images I with controlled image-level artifacts that mimic common reconstruction errors: blur and ghosting. Blur is applied via Gaussian smoothing: I ′ = GaussianBlur(I,σ), σ = 0.5 + 6δ,(16) where δ ≥ 0 controls the reconstruction degradation severity. Ghosting is simulated by blending spatially shifted image copies: I ′ = (1− γ)I + 0.6γT ∆ 1 (I) + 0.4γT ∆ 2 (I), (17) where T ∆ denotes translation by a pixel offset ∆, and γ ∈ [0, 1] controls the ghosting intensity. Figure 3(a) shows that object QA performance for static and dynamic re- mains stable across a wide range of reconstruction degradation under both blur- only (bo) and blur+ghosting (bg) settings. This robustness stems from Render- Mem’s object-centric close-up views, which preserve discriminative cues despite reconstruction artifacts. Visibility reasoning is more sensitive to reconstruction quality because distant objects occupy smaller image regions, where blur or ghosting can obscure spatial details and complicate occlusion reasoning. Localization Perturbation. We next evaluate robustness to imperfect object local- ization by injecting geometric noise into the object bounding spheres s i = (c i ,r i ) defined in Sec. 3.1. The center is perturbed as ̃ c i = c i +ε,ε∼N(0, (λr i ) 2 I),(18) and the radius as ̃r i = r i (1 + ξ), ξ ∼N(0,λ 2 ).(19) where λ≥ 0 controls the magnitude of the localization perturbation. Figure 3(b) shows that object QA performance for static and dynamic re- mains largely stable under increasing localization noise, indicating that moderate geometric inaccuracies do not immediately invalidate rendered evidence as long as target objects remain visible. Visibility queries again exhibit higher sensitivity due to their reliance on precise viewpoint geometry and occlusion relationships. Nevertheless, performance remains robust under realistic levels of localization perturbation, demonstrating that RenderMem tolerates practical scene repre- sentation errors. 14J. Park et al. 5 Limitations and Future Work Despite its effectiveness, RenderMem has several limitations. First, the current object abstraction relies primarily on object category iden- tifiers and spatial locations. This representation lacks support for fine-grained instance-level disambiguation when multiple objects of the same category are present within a scene. For example, when a question refers to “the chair next to the window” in a room containing multiple chairs, the system struggles to dis- ambiguate the intended instance. Augmenting each object with visual features extracted from multi-view observations could potentially address this issue. How- ever, such an approach would introduce significant computational overhead and would complicate support for dynamic environments, as visual features would need to be recomputed after every scene interaction. A more desirable direction is the development of a lightweight mechanism that captures an object’s surround- ing spatial context without requiring repeated visual feature extraction, thereby preserving computational efficiency while maintaining compatibility with dy- namic scene updates. Second, RenderMem relies on a renderable 3D scene representation whose quality directly affects the fidelity of the rendered evidence used for reasoning. High-quality scene reconstructions typically require substantial storage. For ex- ample, indoor mesh scenes can occupy approximately 850 MB per scene, while hybrid mesh–3DGS representations may require around 220 MB [31]. When many environments must be stored, such requirements can become a practi- cal storage bottleneck. Exploring more storage-efficient scene representations or compression strategies that preserve geometric fidelity could improve the scala- bility of RenderMem to large numbers of environments. 6 Conclusion We present RenderMem, a spatial memory framework that leverages rendering as the read operation of a 3D scene representation. Instead of retrieving stored observations, RenderMem generates query-conditioned visual evidence by ren- dering the scene from viewpoints relevant to the user question. This design ex- plicitly grounds reasoning in geometry while remaining fully compatible with existing vision–language models. Experiments demonstrate that this approach improves spatial question an- swering, particularly for viewpoint-dependent visibility reasoning where prior memory paradigms struggle. Because visual evidence is synthesized directly from the current scene state, RenderMem naturally supports dynamic environments without requiring explicit memory updates. We hope this work motivates fur- ther research on rendering-based spatial memory systems and inspires geometry- aware architectures that bridge 3D scene representations and language-based reasoning for embodied intelligence. RenderMem15 References 1. Alayrac, J.B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Men- sch, A., Millican, K., Reynolds, M., et al.: Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems 35, 23716– 23736 (2022) 2. Anderson, P., Wu, Q., Teney, D., Bruce, J., Johnson, M., Sünderhauf, N., Reid, I., Gould, S., Van Den Hengel, A.: Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In: Proceedings of the IEEE conference on computer vision and pattern recognition. p. 3674–3683 (2018) 3. Armeni, I., He, Z.Y., Gwak, J., Zamir, A.R., Fischer, M., Malik, J., Savarese, S.: 3d scene graph: A structure for unified semantics, 3d space, and camera. In: Proceedings of the IEEE/CVF international conference on computer vision. p. 5664–5673 (2019) 4. Azuma, D., Miyanishi, T., Kurita, S., Kawanabe, M.: Scanqa: 3d question answer- ing for spatial scene understanding. In: proceedings of the IEEE/CVF conference on computer vision and pattern recognition. p. 19129–19139 (2022) 5. Chaplot, D.S., Gandhi, D.P., Gupta, A., Salakhutdinov, R.R.: Object goal navi- gation using goal-oriented semantic exploration. Advances in Neural Information Processing Systems 33, 4247–4258 (2020) 6. Chen, S., Chen, X., Zhang, C., Li, M., Yu, G., Fei, H., Zhu, H., Fan, J., Chen, T.: Ll3da: Visual interactive instruction tuning for omni-3d understanding reasoning and planning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. p. 26428–26438 (2024) 7. Das, A., Datta, S., Gkioxari, G., Lee, S., Parikh, D., Batra, D.: Embodied question answering. In: Proceedings of the IEEE conference on computer vision and pattern recognition. p. 1–10 (2018) 8. Deitke, M., Han, W., Herrasti, A., Kembhavi, A., Kolve, E., Mottaghi, R., Sal- vador, J., Schwenk, D., VanderBilt, E., Wallingford, M., et al.: Robothor: An open simulation-to-real embodied ai platform. In: Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition. p. 3164–3174 (2020) 9. Deitke, M., VanderBilt, E., Herrasti, A., Weihs, L., Salvador, J., Ehsani, K., Han, W., Kolve, E., Farhadi, A., Kembhavi, A., Mottaghi, R.: ProcTHOR: Large-Scale Embodied AI Using Procedural Generation. In: NeurIPS (2022), outstanding Paper Award 10. Gu, Q., Kuwajerwala, A., Morin, S., Jatavallabhula, K.M., Sen, B., Agarwal, A., Rivera, C., Paul, W., Ellis, K., Chellappa, R., et al.: Conceptgraphs: Open- vocabulary 3d scene graphs for perception and planning. In: 2024 IEEE Inter- national Conference on Robotics and Automation (ICRA). p. 5021–5028. IEEE (2024) 11. Gupta, S., Davidson, J., Levine, S., Sukthankar, R., Malik, J.: Cognitive mapping and planning for visual navigation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. p. 2616–2625 (2017) 12. Hong, Y., Zhen, H., Chen, P., Zheng, S., Du, Y., Chen, Z., Gan, C.: 3d-llm: In- jecting the 3d world into large language models. Advances in Neural Information Processing Systems 36, 20482–20494 (2023) 13. Hu, W., Hong, Y., Wang, Y., Gao, L., Wei, Z., Yao, X., Peng, N., Bitton, Y., Szpektor, I., Chang, K.W.: 3dllm-mem: Long-term spatial-temporal memory for embodied 3d large language model. arXiv preprint arXiv:2505.22657 (2025) 16J. Park et al. 14. Huang, C., Mees, O., Zeng, A., Burgard, W.: Visual language maps for robot navigation. arXiv preprint arXiv:2210.05714 (2022) 15. Huang, H., Chen, Y., Wang, Z., Huang, R., Xu, R., Wang, T., Liu, L., Cheng, X., Zhao, Y., Pang, J., et al.: Chat-scene: Bridging 3d scene and large language models with object identifiers. Advances in Neural Information Processing Systems 37, 113991–114017 (2024) 16. Huang, J., Yong, S., Ma, X., Linghu, X., Li, P., Wang, Y., Li, Q., Zhu, S.C., Jia, B., Huang, S.: An embodied generalist agent in 3d world. arXiv preprint arXiv:2311.12871 (2023) 17. Hughes, N., Chang, Y., Carlone, L.: Hydra: A real-time spatial perception system for 3d scene graph construction and optimization. arXiv preprint arXiv:2201.13360 (2022) 18. Jatavallabhula, K.M., Kuwajerwala, A., Gu, Q., Omama, M., Chen, T., Maalouf, A., Li, S., Iyer, G., Saryazdi, S., Keetha, N., et al.: Conceptfusion: Open-set mul- timodal 3d mapping. arXiv preprint arXiv:2302.07241 (2023) 19. Jiang, K., Liu, Y., Chen, W., Luo, J., Chen, Z., Pan, L., Li, G., Lin, L.: Beyond the destination: A novel benchmark for exploration-aware embodied question an- swering. arXiv preprint arXiv:2503.11117 (2025) 20. Kanazawa, A., Tulsiani, S., Efros, A.A., Malik, J.: Learning category-specific mesh reconstruction from image collections. In: Proceedings of the European conference on computer vision (ECCV). p. 371–386 (2018) 21. Kerbl, B., Kopanas, G., Leimkühler, T., Drettakis, G., et al.: 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph. 42(4), 139–1 (2023) 22. Khanna, M., Ramrakhya, R., Chhablani, G., Yenamandra, S., Gervet, T., Chang, M., Kira, Z., Chaplot, D.S., Batra, D., Mottaghi, R.: Goat-bench: A benchmark for multi-modal lifelong navigation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. p. 16373–16383 (2024) 23. Kim, P., Coltin, B., Kim, H.J.: Linear rgb-d slam for planar environments. In: Proceedings of the European Conference on Computer Vision (ECCV). p. 333– 348 (2018) 24. Kolve, E., Mottaghi, R., Han, W., VanderBilt, E., Weihs, L., Herrasti, A., Deitke, M., Ehsani, K., Gordon, D., Zhu, Y., et al.: Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint arXiv:1712.05474 (2017) 25. Lamarca, J., Montiel, J.M.M.: Camera tracking for slam in deformable maps. In: Proceedings of the European conference on computer vision (ECCV) workshops. p. 0–0 (2018) 26. Li, J., Li, D., Savarese, S., Hoi, S.: Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. In: International conference on machine learning. p. 19730–19742. PMLR (2023) 27. Li, J., Li, D., Xiong, C., Hoi, S.: Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In: International confer- ence on machine learning. p. 12888–12900. PMLR (2022) 28. Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. Advances in neural information processing systems 36, 34892–34916 (2023) 29. Ma, X., Yong, S., Zheng, Z., Li, Q., Liang, Y., Zhu, S.C., Huang, S.: Sqa3d: Situated question answering in 3d scenes. arXiv preprint arXiv:2210.07474 (2022) 30. Majumdar, A., Ajay, A., Zhang, X., Putta, P., Yenamandra, S., Henaff, M., Silwal, S., Mcvay, P., Maksymets, O., Arnaud, S., et al.: Openeqa: Embodied question answering in the era of foundation models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. p. 16488–16498 (2024) RenderMem17 31. Miao, B., Wei, R., Ge, Z., Gao, S., Zhu, J., Wang, R., Tang, S., Xiao, J., Tang, R., Li, J., et al.: Towards physically executable 3d gaussian for embodied navigation. arXiv preprint arXiv:2510.21307 (2025) 32. Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: Nerf: Representing scenes as neural radiance fields for view synthesis. Commu- nications of the ACM 65(1), 99–106 (2021) 33. Peng, S., Genova, K., Jiang, C., Tagliasacchi, A., Pollefeys, M., Funkhouser, T., et al.: Openscene: 3d scene understanding with open vocabularies. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. p. 815– 824 (2023) 34. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. p. 8748–8763. PmLR (2021) 35. Ren, A.Z., Clark, J., Dixit, A., Itkina, M., Majumdar, A., Sadigh, D.: Explore until confident: Efficient exploration for embodied question answering. arXiv preprint arXiv:2403.15941 (2024) 36. Rosinol, A., Gupta, A., Abate, M., Shi, J., Carlone, L.: 3d dynamic scene graphs: Actionable spatial perception with places, objects, and humans. arXiv preprint arXiv:2002.06289 (2020) 37. Team, Q.: Qwen2.5-vl (January 2025), https://qwenlm.github.io/blog/qwen2. 5-vl/ 38. Thai, A., Peng, S., Genova, K., Guibas, L., Funkhouser, T.: Splattalk: 3d vqa with gaussian splatting. arXiv preprint arXiv:2503.06271 (2025) 39. Wang, S., Liu, B., Gao, Z., Ma, L., Wang, X., Xie, Y., Tan, X.: Explore with long-term memory: A benchmark and multimodal llm-based reinforcement learning framework for embodied exploration. arXiv preprint arXiv:2601.10744 (2026) 40. Werby, A., Huang, C., Büchner, M., Valada, A., Burgard, W.: Hierarchical open- vocabulary 3d scene graphs for language-grounded robot navigation. In: First Workshop on Vision-Language Models for Navigation and Manipulation at ICRA 2024 (2024) 41. Wu, H., Mao, J., Zhang, Y., Jiang, Y., Li, L., Sun, W., Ma, W.Y.: Unified visual- semantic embeddings: Bridging vision and language with structured meaning rep- resentations. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. p. 6609–6618 (2019) 42. Yang, J., Dong, Y., Liu, S., Li, B., Wang, Z., Tan, H., Jiang, C., Kang, J., Zhang, Y., Zhou, K., et al.: Octopus: Embodied vision-language programmer from environ- mental feedback. In: European conference on computer vision. p. 20–38. Springer (2024) 43. Yang, Y., Yang, H., Zhou, J., Chen, P., Zhang, H., Du, Y., Gan, C.: 3d-mem: 3d scene memory for embodied exploration and reasoning. In: Proceedings of the Computer Vision and Pattern Recognition Conference. p. 17294–17303 (2025) 44. Zhang, J., Dong, R., Ma, K.: Clip-fo3d: Learning free open-world 3d scene rep- resentations from 2d dense clip. In: Proceedings of the IEEE/CVF international conference on computer vision. p. 2048–2059 (2023) 45. Zhao, Z., Chai, W., Wang, X., Li, B., Hao, S., Cao, S., Ye, T., Wang, G.: See and think: Embodied agent in virtual environment. In: European Conference on Computer Vision. p. 187–204. Springer (2024)

Instant research discovery