Paper deep dive

RoboLayout: Differentiable 3D Scene Generation for Embodied Agents

Ali Shamsaddinlou

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 45

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 88%

Last extracted: 7/21/2026, 1:56:10 AM

Summary

RoboLayout is a framework for generating 3D indoor scenes that are semantically coherent and physically navigable for embodied agents. It extends the LayoutVLM framework by integrating explicit reachability constraints into a differentiable optimization process, allowing for the generation of layouts tailored to specific agent types (e.g., robots, humans). The system uses a layered architecture with an orchestrator, sandbox, and solver, employing vision-language models to generate constraints and a local refinement stage to improve convergence efficiency.

Entities (11)

RoboLayout · system · 100%Embodied Agents · concept · 90%LayoutVLM · system · 90%Reachability Constraints · method · 90%Vision-Language Models · technology · 90%Local Refinement · method · 80%Orchestration · module · 80%Sandbox · module · 80%Service Robots · agent-type · 80%Solver · module · 80%Warehouse Robots · agent-type · 80%

Relation Signals (9)

RoboLayout → extends → LayoutVLM

confidence 95% · RoboLayout is introduced as an extension of LayoutVLM that augments the original framework with agent-aware reasoning and improved optimization stability.

RoboLayout → composedof → Solver

confidence 90% · Solver: optimizer based on hard and soft constraints for spatial arrangements and refinement of the final optimized scene.

RoboLayout → composedof → Orchestration

confidence 90% · RoboLayout comprises three main layers. Orchestration: The central orchestrator...

RoboLayout → composedof → Sandbox

confidence 90% · Sandbox: Translates constraints into feasible scene layouts.

RoboLayout → supports → Embodied Agents

confidence 90% · enabling the generation of layouts that are navigable and actionable by embodied agents.

RoboLayout → uses → Reachability Constraints

confidence 90% · RoboLayout integrates explicit reachability constraints into a differentiable layout optimization process

RoboLayout → includes → Local Refinement

confidence 85% · a local refinement stage is proposed that selectively reoptimizes problematic object placements

RoboLayout → supports → Service Robots

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Recent advances in vision language models (VLMs) have shown strong potential for spatial reasoning and 3D scene layout generation from open-ended language instructions. However, generating layouts that are not only semantically coherent but also feasible for interaction by embodied agents remains challenging, particularly in physically constrained indoor environments. In this paper, RoboLayout is introduced as an extension of LayoutVLM that augments the original framework with agent-aware reasoning and improved optimization stability. RoboLayout integrates explicit reachability constraints into a differentiable layout optimization process, enabling the generation of layouts that are navigable and actionable by embodied agents. Importantly, the agent abstraction is not limited to a specific robot platform and can represent diverse entities with distinct physical capabilities, such as service robots, warehouse robots, humans of different age groups, or animals, allowing environment design to be tailored to the intended agent. In addition, a local refinement stage is proposed that selectively reoptimizes problematic object placements while keeping the remainder of the scene fixed, improving convergence efficiency without increasing global optimization iterations. Overall, RoboLayout preserves the strong semantic alignment and physical plausibility of LayoutVLM while enhancing applicability to agent-centric indoor scene generation, as demonstrated by experimental results across diverse scene configurations.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

44,149 characters extracted from source content.

Expand or collapse full text

RoboLayout: Differentiable 3D Scene Generation for Embodied Agents Ali Shamsaddinlou Code available at https://github.com/alishams21/robolayout Abstract Recent advances in vision–language models (VLMs) have shown strong potential for spatial reasoning and 3D scene layout generation from open-ended language instructions. However, generating layouts that are not only semantically coherent but also feasible for interaction by embodied agents remains challenging, particularly in physically constrained indoor environments. In this paper, RoboLayout is introduced as an extension of LayoutVLM that augments the original framework with agent-aware reasoning and improved optimization stability. RoboLayout integrates explicit reachability constraints into a differentiable layout optimization process, enabling the generation of layouts that are navigable and actionable by embodied agents. Importantly, the agent abstraction is not limited to a specific robot platform and can represent diverse entities with distinct physical capabilities, such as service robots, warehouse robots, humans of different age groups, or animals, allowing environment design to be tailored to the intended agent. In addition, a local refinement stage is proposed that selectively re-optimizes problematic object placements while keeping the remainder of the scene fixed, improving convergence efficiency without increasing global optimization iterations. Overall, RoboLayout preserves the strong semantic alignment and physical plausibility of LayoutVLM while enhancing applicability to agent-centric indoor scene generation, as demonstrated by experimental results across diverse scene configurations. I Introduction 3D scene generation aims to synthesize spatially structured, semantically coherent, and physically plausible environments for applications such as robotics [2], AR/VR [3], world modeling [4], and architectural visualization [5]. A fundamental challenge in this domain lies in aligning high-level semantic intent expressed through natural language with low-level geometric, spatial, and agent-specific constraints that govern object placement and scene structure. Recent progress in vision–language models (VLMs) and differentiable optimization has enabled new approaches to this problem. LayoutVLM [1] represents a significant advance by formulating 3D layout generation as a differentiable optimization process guided by vision–language supervision. By integrating semantic reasoning with gradient-based refinement, LayoutVLM achieves improved alignment between textual descriptions and spatial layouts. In this paper, RoboLayout is proposed as an extension of LayoutVLM that incorporates agent-aware reachability and a post-optimization cleaning process. First, explicit reachability constraints are integrated into the differentiable optimization loop, enabling the generation of layouts that are feasible and actionable for a broad class of embodied agents. Second, a local refinement stage selectively re-optimizes only problematic object placements while keeping the remainder of the layout fixed, improving convergence efficiency without increasing global iteration counts. This combination allows RoboLayout to disentangle global structural objectives from local relational refinement while maintaining semantic consistency. By building directly on the LayoutVLM framework, RoboLayout preserves end-to-end differentiability and compatibility with vision–language supervision, and importantly, the agent abstraction enables layouts to be explicitly tailored to the intended actor’s physical capabilities, bringing environment design closer to practical deployment for diverse real-world scenarios. I Contributions The key contributions of this work are summarized as follows: 1. Agent-Aware Reachability in Differentiable Layout Optimization: The LayoutVLM framework is extended by integrating explicit reachability constraints for embodied agents, such as robots, humans, or animals, directly into the differentiable optimization process. This allows the model to generate 3D layouts that are not only semantically coherent but also physically navigable and actionable by the intended agent. 2. Efficient Local Refinement for Improved Layout Stability: A local refinement stage is introduced that selectively re-optimizes only problematic object placements while keeping the rest of the scene fixed. This improves convergence efficiency, ensures higher-quality layouts, and maintains semantic and structural consistency without increasing global optimization iterations. I Related Work I-A LLM and VLM-Based Layout Generation Large language models (LLMs) and vision–language models (VLMs) have shown strong capabilities in understanding complex relationships and generating structured outputs that encode both visual and semantic information. LayoutGPT guides LLMs through prompts and templates to generate object parameters for scene layouts [16]. Graph-based scene generation approaches treat objects as nodes and synthesize scenes using learned relational structures [17]. Agentic and iterative frameworks have also been explored for 3D layout generation. SceneWeaver employs a reflective agentic approach that integrates language-model planning with generation tools to produce physically plausible indoor scenes [20]. Holodeck uses multi-round interactions with LLMs to generate semantically consistent 3D environments, including floors, walls, and object placements [21]. SceneX adopts a multi-agent formulation to construct explicit world models and generate executable layout scripts [22]. While purely LLM-based methods achieve impressive results, they often struggle with limited layout diversity, rendering artifacts, and weak global planning, particularly in complex scenes. I-B Optimization for 3D Scene Generation Constraint-based and optimization-driven approaches have long been used in 3D scene generation to balance multiple objectives derived from physical laws, functional requirements, and spatial design principles. Physical constraints include object interpenetration, stability, and support, while layout-level constraints capture accessibility and functional relationships [11] [12]. Data-driven models learn probabilistic object arrangements or hierarchical scene structures using graphical or neural representations [13] [14]. Hybrid discrete–continuous optimization techniques are particularly effective in settings that combine categorical decisions with continuous spatial parameters. LayoutVLM integrates vision–language supervision with differentiable optimization to refine 3D layouts from natural language descriptions. It improves spatial planning through complementary representations and self-consistent decoding [1] this is promising especially for use-cases where control and explainability are indispensable. IV Architecture RoboLayout comprises three main layers. Orchestration: The central orchestrator responsible for coordinating groupings, rendering and other cognitive processes of the solver and sandbox. Sandbox: Translates constraints into feasible scene layouts. Solver: optimizer based on hard and soft constraints for spatial arrangements and refinement of the final optimized scene. As shown in Figure 1, the layered architecture of RoboLayout is composed of following steps: Initial state. The pipeline takes the language instruction (task_description, layout_criteria) and the room shape (boundary.floor_vertices, wall_height) plus the list of assets as the initial program state. The sandbox is initialized with walls and asset variables; positions and rotations are set to random feasible placements inside the boundary. Furniture grouping An LLM groups furniture based on the prompt and task (e.g. “bed + nightstands”, “rug”, “seating”). Each group has a name and key spatial relations between assets, placement then runs group by group with group-specific layout criteria. Pose estimation and spatial relations. For each (group of) assets, the system produces pose and spatial relations by calling an LLM with the current scene (top-down and side renderings) and the layout criteria. The LLM returns a constraint program: high-level relations such as against_wall, align_with, distance_constraint. Conversion to Python program. The LLM output is parsed into executable Python: constraint calls (e.g. solver.against_wall(…), solver.distance_constraint(…)) are executed in the sandbox. That updates the in-memory constraint list. Optimization with reachability. The gradient-based solver (e.g. GradSolver.optimize) minimizes a loss over positions and rotations: overlap loss (no intersection), existing- and new-constraint losses (satisfy spatial relations), and reachability loss. Reachability encourages clearance between furniture so a virtual disc of radius robot_radius can pass; it is disabled if robot_radius is None or ≤0≤ 0. Optimization runs per group (or once in one_shot), producing poses and optional per-step GIFs. Refinement. After the main optimization, a cleanup step identifies problematic pairs (e.g. overlapping footprints), freezes non-problematic assets, and re-runs optimization only for the problematic subset. This refines the answer without full-scene re-optimization. IV-1 Self-Consistency Filter Self-consistent decoding typically refers to sampling multiple outputs from the model (e.g. several constraint programs), then selecting one by a criterion (e.g. majority vote or best score). This codebase does not use self-consistent decoding: it obtains a single LLM constraint program per group (with retries on parse/execution failure), not multiple samples followed by selection. Self-consistency filter is what is implemented: before optimization, the sandbox runs self_consistency_filtering on the new constraints. It checks consistency with the current scene and existing constraints: rejects duplicate or conflicting constraints (e.g. duplicate against_wall, or a second orientation constraint on the same object), resolves against_wall to the nearest wall, and tightens distance constraints to feasible ranges. Figure 1: Overall architecture of RoboLayout IV-A Orchestration This module coordinates the layout generation pipeline through semantic grouping, visual reasoning, and constraint synthesis. The task is formulated as an optimization problem layout=(ℱ,,),P_layout=(F,O,D), where ℱF is the set of furniture items, O is the layout objective, and D describes the room domain. IV-A1 Furniture Grouping A language-driven grouping function :→G1,…,GKG:O→\G_1,…,G_K\ maps the objective to an ordered list of furniture groups. Each group is defined as Gk=(gk,ℱk,ℛk),G_k=(g_k,F_k,R_k), where gk g_k :semantic group label and rationale, :\ semantic group label and rationale, (1) ℱk _k ⊆ℱ:furniture items in group k, :\ furniture items in group k, (2) ℛk _k :intra-group spatial relationships. :\ intra-group spatial relationships. (3) Groups are processed from structural to decorative elements: priority(G1)<priority(G2)<⋯<priority(GK).priority(G_1)<priority(G_2)<…<priority(G_K). Constraint Generation. At step t, the current scene state is t=(i,i)∣fi∈ℱplaced,S_t=\(p_i, θ_i) f_i _placed\, where i∈ℝ3p_i ^3 and i θ_i denote position and orientation. Rendered visual observations are grouped as ℐt=Itop,Iside,If1,…,If|ℱk|.I_t=\I_top,I_side,I_f_1,…,I_f_|F_k|\. A vision–language model then produces executable constraints: k=VLM(t,ℱk,,ℐt).C_k=VLM(S_t,F_k,D,I_t). These constraints are represented as a set of parametric functions over furniture poses: k=ϕj(aj,aj,bj,bj;j)j=1Jk,C_k= \ _j(p_a_j, θ_a_j,p_b_j, θ_b_j;\, λ_j) \_j=1^J_k, where each ϕj _j converts a qualitative spatial rule into a quantitative geometric relation. Iterative Positioning. Furniture is placed group by group using constrained optimization: ℱplaced(0) _placed^(0) =∅, = , (4) ℱplaced(k) _placed^(k) =ℱplaced(k−1)∪argopti,ifi∈ℱk⁡k, =F_placed^(k-1)∪ *arg\,opt_\p_i, θ_i\_f_i _kC_k, (5) for k=1,…,Kk=1,…,K, until all items are positioned. IV-A2 Rendering This module renders 3D scenes and projects world coordinates into image space for visual feedback. Camera Placement. Let x,yB_x,B_y denote all furniture bounding coordinates in the floor plane. The room center and scale are floor _floor =[max⁡(x)+min⁡(x)2max⁡(y)+min⁡(y)20], = bmatrix (B_x)+ (B_x)2\\ (B_y)+ (B_y)2\\ 0 bmatrix, (6) Wroom W_room =max⁡(max⁡(x)−min⁡(x),max⁡(y)−min⁡(y)). = \! ( (B_x)- (B_x),\; (B_y)- (B_y) ). (7) Camera height is scaled proportionally: Hcam=αWroom,α=1.5.H_cam=α\,W_room, α=1.5. A general camera position using spherical angles (ϕ,θ)(φ,θ) is cam=floor+Hcam[sin⁡ϕcos⁡θsin⁡ϕsin⁡θcos⁡ϕ].c_cam=c_floor+H_cam bmatrix φ θ\\ φ θ\\ φ bmatrix. Coordinate Grid Overlay. Grid spacing adapts to room size: Δ=max⁡(Δ0,⌈Wroom500⌉). = \! ( _0,\; W_room500 ). Grid points on the floor are (xi,yj)=(xmin,ymin)+(iΔ,jΔ),(x_i,y_j)=(x_ ,y_ )+(i ,j ), and their image projections form the overlay set ℳ=(uij,vij)=project⁡([xi,yj,0]T).M= \(u_ij,v_ij)=project\! ([x_i,y_j,0]^T ) \. Furniture Orientation Annotation. For furniture fif_i with yaw angle θi,z _i,z, idir=i+ℓ[cos⁡θi,zsin⁡θi,z0],ℓ=0.75.p_i^dir=p_i+ bmatrix _i,z\\ _i,z\\ 0 bmatrix, =0.75. The annotation in image space is (project⁡(i),project⁡(idir),labeli). (project(p_i),\;project(p_i^dir),\;label_i ). Wall Annotation. For a wall segment (i,i+1)(v_i,v_i+1), let i=i+1−i,^i=90∘i‖i‖2,d_i=v_i+1-v_i, n_i= R_90 d_i\|d_i\|_2, where 90∘R_90 rotates a 2D vector by 90∘90 . The wall label position is then wall=i+i+12−β^i,β=0.2.c_wall= v_i+v_i+12-β n_i, β=0.2. IV-B Sandbox The sandbox module converts high-level constraint descriptions produced by the VLM into executable optimization code. It maintains a local namespace in which language-generated Python snippets map abstract rules (e.g., “furniture A should be near wall B”) into concrete geometric constraint functions. Also it filters invalid constraints, resolves conflicts, and ensures all constraints correspond to feasible spatial relationships before optimization. Iterative multi-objective optimization is employed with soft and hard constraints, where hard constraints enforce non-negotiable often physical constraints. IV-B1 Out-of-Bounds Placement Constraint: For each furniture item fif_i with position i=[xi,yi,zi]Tp_i=[x_i,y_i,z_i]^T, orientation i θ_i, and size is_i, its oriented 2D footprint is computed as i=polygon⁡(i,i,i),C_i=polygon(p_i, θ_i,s_i), where i∈ℝ4×2C_i ^4× 2 contains the four ground-plane corners. The room boundary is a polygon P with counter-clockwise vertices 1,…,n\b_1,…,b_n\. A tolerance-expanded region is defined as ϵ=⊕B(0,ϵ)P_ε=P B(0,ε). If any corner i,kc_i,k lies outside ϵP_ε, a corrective translation i τ_i is computed. Case 1: All corners outside. i2D _i^2D =[xi,yi]T =[x_i,y_i]^T (8) proj _proj =arg⁡min∈∂⁡‖−i2D‖2 = _p∈ \|p-p_i^2D\|_2 (9) v =−proj‖−proj‖2 = c_P-p_proj\|c_P-p_proj\|_2 (10) i τ_i =^(‖proj−i2D‖2+ϵ+δ) = v (\|p_proj-p_i^2D\|_2+ε+δ ) (11) where c_P is the room centroid and δ=0.15δ=0.15 m is a safety margin. Case 2: Partial violation. i=1|out|∑k∈outi,k, τ_i= 1|K_out| _k _out τ_i,k, where outK_out indexes corners outside ϵP_ε. The position update is i[t+1][:2]=i[t][:2]+i,p_i^[t+1][:2]=p_i^[t][:2]+ τ_i, iterated up to M=10M=10 times or until all corners satisfy i⊆ϵC_i _ε. 0: Furniture ℱF, boundary P, tolerance ϵε, margin δ, max steps M 1: for each fi∈ℱf_i do 2: t←0t← 0 3: while t<Mt<M and i⊈ϵC_i _ε do 4: Compute i τ_i based on violation case 5: i[:2]←i[:2]+ip_i[:2] _i[:2]+ τ_i 6: Recompute iC_i 7: t←t+1t← t+1 8: end while 9: end for IV-B2 Rotation Constraint For stabilizing stochastic rotations during optimization updates, each proposed step is projected onto the feasible manifold before acceptance. Given the current state x and a candidate step s (e.g., from Adam), the following conditions are enforced: ‖≤Δ,+∈ℬ,crot(+)=0,\|s\|≤ , +s , c_rot(x+s)=0, where Δ is the trust-region radius, ℬB denotes box bounds on positions and rotations, and crotc_rot encodes rotation feasibility. This guarantees that all accepted updates remain inside the feasible set. Each rotation angle θi _i is parameterized using (cos⁡θi,sin⁡θi)( _i, _i) constrained to lie on the unit circle: crot()=cos2⁡θi+sin2⁡θi−1=0∀i.c_rot(x)= ^2 _i+ ^2 _i-1=0 ∀ i. The constrained optimization problem is therefore min∈ℬ⁡f()s.t.crot()=0. _x \;f(x) .t. c_rot(x)=0. Convergence tolerances are gtol=xtol=barrier_tol=10−12.gtol=xtol=barrier\_tol=10^-12. 0: Initial state 0x_0, objective f, bounds ℬB 1: ←0x _0, Δ←Δ0 ← _0 2: while not converged do 3: Propose step s 4: Project s so that ‖≤Δ\|s\|≤ , +∈ℬx+s , crot(+)=0c_rot(x+s)=0 5: Compute ratio ρ=f()−f(+)predicted decreaseρ= f(x)-f(x+s)predicted decrease 6: if ρ>η1ρ> _1 then 7: ←+x +s 8: if ρ>η2ρ> _2 then 9: Δ←min⁡(γ2Δ,Δmax) ← ( _2 , _ ) 10: end if 11: else 12: Δ←γ1Δ ← _1 13: end if 14: end while 15: return x IV-B3 Reachability Constraint The reachability constraint is implemented as a walking robot governed by pairwise clearance rule. For every pair of movable (non–on-top-of) assets i, j, It takes their center positions pip_i, pjp_j in the ground plane and requires the center–center distance to be at least ei+ej+2re_i+e_j+2r (half-footprint of i plus half-footprint of j plus robot diameter), then penalizes (via a soft loss) when the actual distance is smaller than this required clearance. Working in a top–down 2D view of the room. Let • A be the set of all movable, non–fixture, non–ceiling assets that are optimized by the gradient solver. • For each asset i∈i , let its center position in the ground plane be pi∈ℝ2,p_i ^2, corresponding to the (x,y)(x,y) components of its 3D position. • Let its axis-aligned bounding box dimensions in the ground plane be wiw_i (width in x) and hih_i (height in y). An effective in-plane “radius” is defined as ei≜12min⁡(wi,hi),e_i 12 (w_i,h_i), so that the footprint is approximated by a disc of radius eie_i. • Let r>0r>0 be the radius of the virtual robot which is provided via --robot_radius through code intialization. Intuitively, the objective is for a robot of radius r to be able to move between any two assets i and j without colliding with their footprints. In a simple disc approximation, this means the distance between the centers pip_i and pjp_j should be at least ei+ej+2r.e_i+e_j+2r. For each unordered pair (i,j)(i,j) of assets with i≠ji≠ j, a required center-to-center distance is defined as dijreq≜ei+ej+2r.d_ij^req e_i+e_j+2r. (12) Let dij≜∥pi−pj∥2d_ij p_i-p_j _2 (13) be the actual Euclidean distance between the asset centers in the ground plane. Ideally, the goal is to enforce the hard constraint dij≥dijreqfor all pairs (i,j).d_ij≥ d_ij^req all pairs (i,j). (14) Instead of imposing this as a strict feasibility constraint, it is relaxed into a differentiable soft penalty that can be optimized by gradient descent. In the implementation, squared distances are used to keep the loss smooth and easy to differentiate. For each pair (i,j)(i,j), the squared distance is considered Dij≜∥pi−pj∥22,D_ij p_i-p_j _2^2, (15) and the squared required distance Rij≜(dijreq)2=(ei+ej+2r)2.R_ij (d_ij^req )^2= (e_i+e_j+2r )^2. (16) The pairwise violation is then defined as vij≜max⁡(0,Rij−Dij).v_ij (0,\;R_ij-D_ij ). (17) This quantity is zero whenever the actual squared distance DijD_ij is at least the required squared distance RijR_ij, and grows linearly with the shortfall when assets are too close. The unscaled reachability loss over all eligible pairs is Lreachraw≜∑(i,j)∈vij,L_reach^raw _(i,j) v_ij, (18) where P is the set of unordered asset pairs that: • are both members of A (movable, non–fixture, non–ceiling), and • are not marked as “on top of” one another (those pairs are handled by a separate constraint). To keep the magnitude of this loss term comparable to other components (overlap, existing constraints, new constraints), a small scalar factor α>0α>0 is applied: Lreach≜αLreachraw=α∑(i,j)∈max⁡(0,Rij−Dij).L_reach α\;L_reach^raw=α _(i,j) (0,R_ij-D_ij ). (19) In the current implementation, α=0.01α=0.01. The reachability loss is combined with the other layout losses in the gradient solver. Let • LoverlapL_overlap be the non-overlap (bounding-box) loss, • LexistL_exist be the loss from existing constraints, • LnewL_new be the loss from new constraints for the current group of assets, • LreachL_reach be the reachability loss defined above. The total objective minimized by gradient descent is Ltotal=Loverlap+Lexist+Lnew+Lreach.L_total\;=\;L_overlap+L_exist+L_new+L_reach. (20) When r≤0r≤ 0 or no robot radius is provided, the implementation sets Lreach=0L_reach=0, effectively disabling the reachability term. Table 1 shows the list of the hard and soft constraints used by the solver. Hard constraints are enforced by projection or by constraints in the solver. Soft constraints are implemented as differentiable loss terms that are minimized and are not strictly enforced. TABLE 1: List of hard and soft constraints. Type Name Rule Hard Boundary (inside room) Every asset’s 2D footprint must lie inside the room polygon. Any corner outside is projected onto the boundary. Hard On-top (vertical + horizontal) For each on_top_of(A,B): (1) A’s z is set so A sits on top of B (z=B.z+B.h/2+A.h/2z=B.z+B.h/2+A.h/2). (2) A’s xyxy is projected onto B’s 2D polygon so A is centered over B. Hard Rotation unit norm When using solver_type of minimize, rotation is parameterized as (cos⁡θ,sin⁡θ)( θ, θ) with cos2⁡θ+sin2⁡θ=1 ^2θ+ ^2θ=1 via NonlinearConstraint. Soft Overlap (no intersection) 2D footprints of non–on-top pairs must not overlap. Loss: IoU / bbox overlap (e.g. scaled by 1000); minimized so overlap goes to zero. Soft Reachability Pairs of objects should leave enough clearance for a disc of radius robot_radius. Loss: penalize if center distance <(ra+rb+2×robot_radius)<(r_a+r_b+2×robot\_radius). Soft Against_wall Asset should be (1) close to a wall segment and (2) front parallel to the wall. Loss: distance of footprint to wall + angle between asset front and wall direction. Soft Distance_constraint Distance between two centers should be in [_,_][min\_distance,max\_distance]. Loss: penalize when outside the range (scaled by weight). Soft Point_towards Asset1’s front should point toward asset2. Loss: 0 if the ray from asset1 in its front direction hits asset2’s polygon; otherwise cosine alignment loss with the direction to asset2. Soft Align_with Two assets’ forward directions should be aligned (optional angle offset). Loss: cosine distance between (possibly rotated) forward vectors. Soft On_top_of (soft part) In addition to the hard on-top placement, the upper object should stay over the lower in 2D. Loss: negative IoU (clamped) so that overlap in the horizontal plane is encouraged. Soft Symmetric_pair Two assets should be mirror-symmetric about a reference (wall or central asset). Loss: position reflection error + optional orientation mirror alignment. IV-C Solver Layout generation problem is solved as a hybrid discrete–continuous multi-objective optimization problem. Hard constraints are enforced via projection steps that keep every intermediate state feasible, while soft constraints remain in the differentiable objective and are optimized with gradient-based updates. IV-C1 Objective Function Let each soft constraint ϕφ act on a subset of furniture indices ℐI. The total loss is ℒtotal=∑(ϕ,ℐ)ϕ(ℐ,ℐ),ℒ=0.1ℒtotal.L_total= _(φ,I)φ(p_I, θ_I), =0.1\,L_total. IV-C2 Learning Rate Scaling To normalize updates across room sizes, the following quantity is defined: Rroom=max⁡(max⁡(x)−min⁡(x),max⁡(y)−min⁡(y), 1),R_room= ( (B_x)- (B_x),\; (B_y)- (B_y),\;1 ), and use separate learning rates αpos=3α0Rroom,αrot=α0, _pos= 3 _0R_room, _rot= _0, where α0 _0 is the base rate (typically 0.030.03). Adam Update. For any parameter ∈i,i ξ∈\p_i, θ_i\, mt m_t =β1mt−1+(1−β1)∇ℒ, = _1m_t-1+(1- _1) _ ξL, (21) vt v_t =β2vt−1+(1−β2)(∇ℒ)2, = _2v_t-1+(1- _2)( _ ξL)^2, (22) t ξ_t =t−1−αtmt/(1−β1t)vt/(1−β2t)+ϵ, = ξ_t-1- _t m_t/(1- _1^t) v_t/(1- _2^t)+ε, (23) with β1=β2=0.9 _1= _2=0.9 and ϵ=10−8ε=10^-8. Gradients are clipped to ‖∇ℒ‖2≤10\| _ ξL\|_2≤ 10. The learning rate decays every 100 steps: αt=α⋅γ⌊t/100⌋,γ=0.95. _t=α·γ t/100 , γ=0.95. IV-C3 Feasibility Enforcement After each update, parameters are projected onto the feasible manifold (bounds + rotation constraints). Additional hard corrections are applied periodically, for boundary projection every 80 steps. 0: Furniture ℱF, iterations T, base rate α0 _0 1: Initialize Adam moments m0,v0←0m_0,v_0← 0 2: Compute RroomR_room, αpos _pos, αrot _rot 3: for t=1t=1 to T do 4: Compute ℒL and gradients 5: Clip gradients 6: Update parameters using Adam 7: Project parameters onto feasible manifold 8: if tmod100=0t 100=0 then 9: Enforce boundary constraints 10: end if 11: end for Applying a trust-region style projection at every iteration filters noisy gradient steps. Well-aligned updates pass through unchanged, while unstable steps are reduced or rejected, leading to smoother convergence without sacrificing the exploratory benefits of Adam. IV-C4 Refinement Running the main layout optimization until full convergence often requires a large number of iterations and is costly. The refinement (or cleanup) step avoids this by shrinking the optimization space: after the main run, only the problematic assets (e.g. those still overlapping) are identified and just those are re-optimized while keeping the rest fixed. Thus there is no need to increase the main iteration count to converge; instead, a short, cheaper pass over a smaller variable set resolves remaining conflicts in less time. In the following, the terms “refinement” or “cleanup” are used for this step (the code also refers to it as cleanup). After the main optimization, problematic assets are identified as those involved in at least one overlapping pair (excluding fixtures and stacked pairs): ℐprob=i∈ℐ∖ℱ|∃j≠i:(i,j)∉,i∩j≠∅.I_prob= \i \; |\;∃ j≠ i:\;(i,j) ,\;P_i _j≠ \. (24) In code, polygon intersection is tested with Shapely (e.g. poly_i.intersects(poly_j)). Movable assets are partitioned into: • Correct (frozen): ℐfix=(ℐ∖ℱ)∖ℐprobI_fix=(I ) _prob. Their positions and rotations are fixed at the current values ¯i p_i, ¯i r_i. • Problematic (optimized): ℐprobI_prob. Only these variables are updated during cleanup. Let ℐprobp_I_prob and ℐprobr_I_prob denote the positions and rotations of problematic assets; all others are fixed. The cleanup step solves, for a small number of iterations TcleanupT_cleanup (e.g. 4040), the same total loss but with gradients taken only with respect to ip_i, ir_i for i∈ℐprobi _prob: mini,i,i∈ℐprob⁡ℒtotal(ℐprob,ℐprob;¯ℐfix,¯ℐfix). _p_i,r_i,\,i _probL_total (p_I_prob,r_I_prob;\; p_I_fix, r_I_fix ). (25) Only parameters for i∈ℐprobi _prob have requires_grad=True; others are frozen, so constraint terms that depend only on fixed assets do not require gradients (handled by allow_nograd_constraints). Optimizer: Adam with learning rate ηcleanup _cleanup (e.g. 0.010.01), gradient clipping (max norm 11). Every K steps (e.g. 1010), apply the projection Π to keep layouts feasible. 0: Furniture set ℱF, problematic furniture ℱproblemF_problem, iterations TftT_ft, learning rate αft _ft 1: for each furniture fi∈ℱf_i do 2: if fi∉ℱproblemf_i _problem then 3: fi.optimize←0f_i.optimize← 0 4: ∇iℒ←0 _p_iL← 0, ∇iℒ←0 _ θ_iL← 0 5: else 6: fi.optimize←1f_i.optimize← 1 7: i.requires_grad←Truep_i.requires\_grad 8: i.requires_grad←True θ_i.requires\_grad 9: end if 10: end for 11: allow_nograd_constraints←Trueallow\_nograd\_constraints 12: for t=1t=1 to TftT_ft do 13: Compute ∇iℒ _p_iL, ∇iℒ _ θ_iL for fi∈ℱproblemf_i _problem 14: Update ip_i, i θ_i using Adam with learning rate αft _ft 15: end for V Experiments V-A Experimental Setup Evaluation of RoboLayout on the task of 3D indoor scene layout generation is done under the following configuration: • VLM: GPT-4o • Optimization Strategy: Iterative multi-objective optimization • Spatial Configurations: Different rooms with varying shapes and dimensions • Object Sets: 5–10 furniture and decorative elements per scene • Contextual Constraints: Structural realism and semantic placement consistency V-B Qualitative Results Figure 2 presents representative scenes generated by RoboLayout. The results demonstrate that the system: • Consistently positions large furniture elements adjacent to walls while satisfying physical feasibility constraints • Preserves semantically appropriate inter-object spacing in accordance with predefined layout policies • Produces coherent object orientations that conform to common indoor design norms • Correctly models hierarchical spatial relationships (e.g., decorative objects placed on supporting furniture) while maintaining structural realism (a) buffet restaurant (b) bookstore (c) game room (d) computer room Figure 2: Example spaces generated by RoboLayout V-C Loss Curve Figure 3 contains two subplots that summarize the gradient descent optimization of the layout solver. Each iteration is one step of gradient descent: the solver computes the loss, backpropagates to get gradients with respect to the layout parameters (3D positions and rotations of objects), then updates those parameters via an optimizer (e.g. Adam). This is not training of a neural network and no model weights are learned from data. The curves show the loss function minimized by gradient descent for a single scene layout problem. The two panels are: • The top panel shows the evolution of several loss terms in linear scale. • The bottom panel shows the same quantities in logarithmic scale (base 10) applied to the absolute value of each loss. At each optimization iteration i∈0,1,…,N−1i∈\0,1,…c,N-1\ the solver stores a dictionary where some keys (e.g. reachability) may be absent if the corresponding term is disabled. Each entry can be a PyTorch tensor or a Python scalar; for plotting it is always converted to a floating–point number. Figure 3: Loss Curve VI Future Work Future research directions include the following: • Integrating robot motion planning directly into the optimization process, replacing the current walking-robot–based clearance constraint. This would enable feasibility-aware layouts that account for kinematic reachability, collision-free trajectories, and task execution constraints, resulting in more realistic and deployable solutions. • Extending the current robot reachability model from a planar, radius-based formulation to a full 3D representation by incorporating vertical reach constraints. This would allow more accurate modeling of robot capabilities and improve the validity of generated layouts in real-world environments. • Investigating alternative optimization strategies beyond gradient-based methods currently used for soft constraint optimization. In particular, hybrid discrete–continuous or unified optimization techniques (e.g., mixed-integer optimization or sampling-based methods) could improve convergence robustness and better handle non-convex, combinatorial layout constraints. • Extending self-consistency beyond the current constraint-level filtering: (1) strengthening conflict resolution and redundancy removal in the constraint set, and (2) adopting self-consistent decoding by sampling several candidate programs and selecting the one that achieves the best solver objective (e.g., total loss or feasibility), thereby improving reliability of generated layouts. VII Conclusion This paper presents RoboLayout, an extension of LayoutVLM that integrates explicit agent-aware reachability into differentiable 3D layout optimization. By treating the “robot” as a general embodied agent, including robots, humans, or animals, RoboLayout generates layouts that are not only semantically coherent but also physically navigable and actionable by the intended agent. In addition, a local refinement stage selectively re-optimizes only problematic object placements while keeping the remainder of the scene fixed, improving convergence efficiency and layout stability without increasing global optimization cost. Experimental results demonstrate that RoboLayout preserves strong semantic alignment, improves optimization robustness, and produces agent-feasible indoor layouts across diverse scene configurations, bringing 3D scene generation closer to practical deployment in real-world environments. VIII References [1] Fan-Yun Sun, Weiyu Liu, Siyi Gu, Dylan Lim, Goutam Bhat, Federico Tombari , Manling Li, Nick Haber, Jiajun Wu, Layoutvlm: Differentiable Optimization of 3D Layout via Vision-Language Models, 2025 [2] K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, S. Jakubczak, T. Jones, L. Ke,S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, L. X. Shi, J. Tanner, Q. Vuong, A. Walling, H. Wang, and U. Zhilinsky, “π 0: A vision-language-action flow model for general robot control,”arXiv 2410.24164, 2024. [3] M. M. Soliman, E. Ahmed, A. Darwish, and A. E. Hassanien, “Artificial intelligence powered metaverse: analysis, challenges and future perspectives,” Artificial Intelligence Review, vol. 57,no. 2, p. 36, 2024. [4] D. Liu, J. Zhang, A.-D. Dinh, E. Park, S. Zhang, and C. Xu, “Generative physical AI in vision: A survey,” arXiv 2501.10928, 2025. [5] K. Chang, C. Cheng, J. Luo, S. Murata, M. Nourbakhsh, and Y. Tsuji, “Building-GAN: Graph-conditioned architectural volumetric design generation,” in ICCV, 2021 [6]W. Zhao, Y. Cao, J. Xu, Y. Dong, and Y. Shan, “DI-PCG: diffusionbased efficient inverse procedural content generation for highquality 3D asset creation,” arXiv 2412.15200, 2024 [7] W. Wu, L. Fan, L. Liu, and P. Wonka, “Miqp-based layout design for building interiors,” Computer Graphics Forum, 2018. [8] Y. Li, O. Vinyals, C. Dyer, R. Pascanu, and P. W. Battaglia, “Learning deep generative models of graphs,” arXiv 1803.03324, 2018 [9] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. C. Courville, and Y. Bengio, Generative adversarial networks, in NIPS, 2014. [10] J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” in NeurIPS, 2020. [11] P. Kan and H. Kaufmann, “Automatic furniture arrangement ´ using greedy cost minimization,” in VR, 2018 [12] Y. Zhao, K. Lin, Z. Jia, Q. Gao, G. Thattai, J. Thomason, and G. S. Sukhatme, “LUMINOUS: indoor scene generation for embodied AI challenges,” arXiv 2111.05527, 2021. [13] M. Fisher, D. Ritchie, M. Savva, T. A. Funkhouser, and P. Hanrahan, “Example-based synthesis of 3D object arrangements,” ACM TOG, vol. 31, no. 6, p. 135:1–135:11, 2012. [14] S. Zhang, S. Zhang, W. Xie, C. Luo, Y. Yang, and H. Fu, “Fast 3D indoor scene synthesis by learning spatial relation priors of objects,” IEEE TVCG, vol. 28, no. 9, p. 3082–3092, 2022. [15] S. Dasgupta, A. Gupta, S. Tuli, and R. Paul, “ActNeRF: Uncertainty-aware active learning of nerf-based object models for robot manipulators using visual and re-orientation actions,” in IROS, 2024. [16] W. Feng, W. Zhu, T. Fu, V. Jampani, A. R. Akula, X. He, S. Basu, X. E. Wang, and W. Y. Wang, “LayoutGPT: Compositional visual planning and generation with large language models,” in NeurIPS, 2023. [17] K. Bhat, N. Khanna, K. Channa, T. Zhou, Y. Zhu, X. Sun, C. Shang, A. Sudarshan, M. Chu, D. Li, K. Deng, J. Fauconnier, T. Verhulsdonck, M. Agrawala, K. Fatahalian, A. Weiss, C. Reiser, R. K. Chirravuri, R. Kandur, A. Pelaez, A. Garg, M. Palleschi, J. Wang, S. Litz, L. Liu, A. Li, D. Harmon, D. Liu, L. Feng, D. Goupil, L. Kuczynski, J. Yoon, N. Marri, P. Zhuang, Y. Zhang, B. Yin, H. Jiang, M. van Workum, T. Lane, B. Erickson, S. Pathare, K. Price, A. Singh, and D. Baszucki, “Cube: A roblox view of 3D intelligence,” arXiv 2503.15475, 2025. [18] Y. Yang, J. Lu, Z. Zhao, Z. Luo, J. J. Q. Yu, V. Sanchez, and F. Zheng, “LLplace: The 3D indoor scene layout generation and editing via large language model,” arXiv 2406.03866, 2024. [19] R. Fu, Z. Wen, Z. Liu, and S. Sridhar, “AnyHome: Openvocabulary generation of structured and textured 3D homes,” in ECCV, 2024. [20] Yandan Yang1, Baoxiong Jia1, Shujie Zhang1, Siyuan Huang1, All-in-One 3D Scene Synthesis with an Extensible and Self-Reflective Agent 1State Key Laboratory of General Artificial Intelligence, BIGAI 2Tsinghua University [21] Y. Yang, F. Sun, L. Weihs, E. VanderBilt, A. Herrasti, W. Han, J. Wu, N. Haber, R. Krishna, L. Liu, C. Callison-Burch, M. Yatskar, A. Kembhavi, and C. Clark, “Holodeck: Language guided generation of 3D embodied AI environments,” in CVPR, 2024. [22] M. Zhou, J. Hou, C. Luo, Y. Wang, Z. Zhang, and J. Peng, “SceneX: procedural controllable large-scale scene generation via largelanguage models,” in AAAI, 2025. [23] R. Aguina-Kang, M. Gumin, D. H. Han, S. Morris, S. J. Yoo, A. Ganeshan, R. K. Jones, Q. A. Wei, K. Fu, and D. Ritchie, “Openuniverse indoor scene generation using LLM program synthesis and uncurated object databases,” arXiv 2403.09675, 2024 [24] Ian W. Eisenberg, The Unified Control Framework: Establishing a Common Foundation for Enterprise AI Governance, Risk Management and Regulatory Compliance , 2025 [25] Emmanouil Papagiannidis a , Patrick Mikalef, Kieran Conboy b Responsible artificial intelligence governance: A review and research framework Author links open overlay panel, The Journal of Strategic Information Systems, June 2025, 101885 [26] Automated architectural space layout planning using a physics-inspired generative design framework, Z. Li, S. Li, G. Hinchcliffe, N. Maitless, N. Birbilis, 2024. [27] Retail Innovation Through AI: Benefits and Liabilities , W. Michael Schuster, Simon E. Corrigan, 2025.

Instant research discovery