Paper deep dive
Unifying Policy Learning and State Prediction through Spatial Language Modeling
Minye Wu, Zehao Wang, Tinne Tuytelaars
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 10/10/2026, 3:12:57 AM
Summary
The paper introduces Spatial Language Modeling (SLM), a framework that unifies policy learning and state prediction using a single autoregressive Transformer. SLM represents scene geometry, goals, and actions as discrete coordinate and semantic tokens organized by a task-specific grammar. The model is trained via random-play pretraining for state transition prediction and joint action/state training on expert demonstrations. It achieves competitive performance in the Push-T manipulation task, outperforming baselines in real-robot success rates and coverage.
Entities (8)
Relation Signals (6)
Spatial Language Modeling → appliedto → Push-T
confidence 98% · We evaluate the approach on Push-T in simulation and on a real robot.
Spatial Language Modeling → uses → Autoregressive Transformer
confidence 95% · allowing one autoregressive Transformer to learn action generation and action-conditioned state prediction through a common next-token objective.
Spatial Language Modeling → usestrainingstrategy → Random-play pretraining
confidence 93% · We train the model from scratch using random-play transition pretraining followed by joint action and state training on expert demonstrations.
Spatial Language Modeling → basedon → Qwen2
confidence 90% · train a 462M-parameter Transformer using the Qwen2 architecture [39] from scratch.
Spatial Language Modeling → implements → Spatial Language Sentence
confidence 90% · A task-specific grammar organizes scene observations, goals, action targets, and optional subsequent states into a Spatial Language Sentence(SLS).
Spatial Language Modeling → outperforms → Diffusion Policy
confidence 90% · The model achieves competitive simulation performance and higher task success and target coverage than the evaluated real-robot policy baselines.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Learning how actions change scene geometry can provide complementary supervision for goal-directed manipulation. We introduce Spatial Language Modeling, which represents scene contours, goals, action targets, and future states with a shared vocabulary of discrete coordinates and semantic tokens. A task-specific grammar organizes these elements into spatial sequences, allowing one autoregressive Transformer to learn action generation and action-conditioned state prediction through a common next-token objective. We train the model from scratch using random-play transition pretraining followed by joint action and state training on expert demonstrations. During pretraining, recorded action coordinates condition subsequent state predictions and are excluded from the prediction loss. During control, the model decodes only executable action targets and updates its history with newly observed states. We evaluate the approach on Push-T in simulation and on a real robot. The model achieves competitive simulation performance and higher task success and target coverage than the evaluated real-robot policy baselines. Training ablations show improved control with joint action and state sequences, with further gains from random-play pretraining. Given supplied action trajectories, the same model also predicts successive scene states, capturing the geometric effects of pushing.
Tags
Links
- Source: https://arxiv.org/abs/2610.12172v1
- Canonical: https://arxiv.org/abs/2610.12172v1
Trouble viewing inline? Open PDF directly →
Full Text
51,765 characters extracted from source content.
Expand or collapse full text
Unifying Policy Learning and State Prediction through Spatial Language Modeling Minye Wu Zehao Wang Tinne Tuytelaars Affiliation: KU Leuven Abstract Learning how actions change scene geometry can provide complementary supervision for goal-directed manipulation. We introduce Spatial Language Modeling, which represents scene contours, goals, action targets, and future states with a shared vocabulary of discrete coordinates and semantic tokens. A task-specific grammar organizes these elements into spatial sequences, allowing one autoregressive Transformer to learn action generation and action-conditioned state prediction through a common next-token objective. We train the model from scratch using random-play transition pretraining followed by joint action and state training on expert demonstrations. During pretraining, recorded action coordinates condition subsequent state predictions and are excluded from the prediction loss. During control, the model decodes only executable action targets and updates its history with newly observed states. We evaluate the approach on Push-T in simulation and on a real robot. The model achieves competitive simulation performance and higher task success and target coverage than the evaluated real-robot policy baselines. Training ablations show improved control with joint action and state sequences, with further gains from random-play pretraining. Given supplied action trajectories, the same model also predicts successive scene states, capturing the geometric effects of pushing. Figure 1: Spatial Language Modeling for Push-T. Scene geometry, goals, pusher actions, and optional future states are represented as structured coordinate-token sequences. The model generates actions for closed-loop execution and learns state transitions through future-state supervision. 1 Introduction Robot manipulation requires selecting actions that transform a scene toward a goal. In the Push-T task [6], a planar pusher moves a T-shaped block to a target pose. The resulting motion depends on both object geometry and contact: the same pusher displacement can induce different combinations of translation and rotation depending on the contact location and the block’s orientation. Demonstrations of these interactions provide two complementary training signals: expert actions indicate how to act, while subsequent observations reveal the geometric changes that follow. Policies such as Action Chunking Transformers, Diffusion Policy, and vision-language-action models learn actions from demonstrations [44, 6, 27, 1]. World models use future predictions for rollout generation and planning [16, 23], while recent work also jointly models actions and future observations [18]. Predicting the geometric consequences of actions can provide complementary supervision for policy learning and enable the use of non-expert interaction data. A shared prediction interface allows this transition supervision to update the same model parameters used for action generation. We therefore seek a spatial representation that makes both targets accessible to the same autoregressive model. Discrete workspace coordinates provide such an interface at the level of explicit geometry. We divide a bounded planar workspace into grid cells and assign each cell a coordinate token. Object contours, target regions, pusher positions, and action trajectories can then be represented in the same spatial vocabulary. A coordinate retains the same physical reference whether it appears in an observed contour, an executable action target, or a predicted future state. Semantic tokens identify whether coordinates belong to the pusher, the manipulated object, the goal, or a movement command. Geometry and actions thus retain their distinct roles within a shared spatial reference frame. We develop this representation as Spatial Language Modeling. A task-specific grammar organizes scene observations, goals, action targets, and optional subsequent states into a Spatial Language Sentence(SLS). The grammar specifies how these elements are ordered and identified, while the model learns their coordinate content. A serializer converts observed geometry and recorded actions into spatial sentences, and a parser maps generated action blocks back to pusher targets. This formulation turns action generation and action-conditioned state prediction into sequence-completion tasks that share an output vocabulary and a next-token training objective. We train an autoregressive Transformer from scratch using two sources of interaction data. Random-play pretraining first trains the model to predict scene transitions conditioned on recorded pusher motions. The action coordinates remain in the input sequence, but their prediction losses are masked; subsequent states supply targets describing the resulting geometry. Expert-demonstration training then supervises both action targets and selected subsequent states. These losses update the same parameters, making the geometric effects of interaction an explicit learning target alongside goal-directed action generation. This training scheme connects transition information from random interactions with action supervision from expert demonstrations. During closed-loop control, an observed-history SLS prefix describes recent scene states and pusher actions. The model completes this prefix with consecutive action blocks, which are parsed into target coordinates and executed by the robot. Future-state blocks are excluded from control decoding, and new observations refresh the history for the next completion. The policy can therefore use a model trained with transition supervision while generating only actions at execution time. The same model also supports a separate prediction setting: given an initial scene and a supplied action trajectory, it generates successive scene states, using its previous state predictions as context. We evaluate the formulation on Push-T in simulation and on a real robot. The model achieves competitive simulation performance and higher task success and target coverage than the evaluated real-robot policy baselines. Comparisons of training variants show improved control with interleaved action and state sequences, with further gains from random-play pretraining. In the state-prediction setting, predicted object states retain substantial geometric overlap with simulator reference states across the evaluated rollout horizon. Our contributions are as follows: • We introduce a grammar-structured spatial representation that expresses observed geometry, goals, action targets, and future states in a shared coordinate vocabulary, enabling action and state prediction through autoregressive sequence completion. • We develop a training scheme that combines action-conditioned transition pretraining on random interactions with joint action and state supervision on expert demonstrations, and supports closed-loop execution through action-only decoding. • We demonstrate closed-loop Push-T control in simulation and on a real robot, and report results and comparisons that support the proposed formulation. 2 Related Work Policy Learning and World Modeling. Imitation-learning policies use demonstrations to supervise action distributions. Action Chunking Transformers predict temporally extended action sequences, while Diffusion Policy generates actions through trajectory denoising [44, 6]. Vision-language-action models connect action learning to vision-language pretraining, heterogeneous robot data, and fine-tuning [27, 17, 1]. Sequence modeling also provides a direct route to control: Decision Transformer predicts actions conditioned on returns and trajectory history, and Behavior Transformers combine action discretization with continuous corrections to model multimodal demonstrations [3, 32]. World models learn scene transitions and connect future predictions to control through several interfaces. DreamGen generates visual trajectories and obtains action labels through inverse dynamics or latent-action models for policy training [16]. LingBot-VA interleaves video prediction and action decoding in a causal sequence [18]. RoboDreamer synthesizes compositional video plans [46], Dreamitate tracks generated tool trajectories for robot execution [20], and LeWorldModel predicts future latent embeddings for planning [23]. Our formulation uses coordinate-level scene transitions and action targets within one next-token objective, combining random-play transition pretraining with joint action/state training on demonstrations. Language Models in Spatial Computation. Language models have been applied to spatial intelligence through several representations of 3D scenes. PointLLM and 3D-LLM connect point-cloud or scene features to language models for object and scene understanding [38, 14]. SegPoint extends this connection to instruction-conditioned segmentation [13]. LSceneLLM adaptively selects task-relevant scene details, while Scene-LLM combines scene-level and egocentric features for 3D visual reasoning [45, 10]. Other approaches expose spatial structure through prompting: SayPlan uses hierarchical 3D scene graphs for robot task planning [31], and 3DAxisPrompt supplies visual coordinate axes and masks for 3D grounding and reasoning [22]. Geometric sequence models provide another route to spatial representation. PolyGen autoregressively generates mesh vertices and faces, and MeshGPT generates discrete codes from a learned geometric vocabulary [25, 33]. Geometry-aware patch tokenization also supports mesh classification and segmentation [8]. LLaMA-Mesh and MeshLLM connect mesh generation to language models through textual representations of geometry [35, 7]. In 2D vision, Pix2Seq represents bounding boxes and class labels as discrete output tokens, formulating detection as sequence prediction [5]. Scene Understanding and Generation. 3D scene understanding develops geometric representations for identifying objects and their relationships. PointNet, PointNet++, PointConv, and Point Transformer learn features directly from point sets [29, 30, 37, 43]. VoxFormer predicts volumetric geometry and semantics from images [19], while ScanRefer and context-aware grounding models connect language to objects in reconstructed scenes [2, 41]. Semantic scene graphs explicitly represent object relations [34], and PointCLIP models extend recognition using vision-language priors [42, 48]. Complementary generative methods synthesize voxel shapes [36], surfaces and textured meshes [12, 11], or point clouds [40]. DreamFusion, Magic3D, and Fantasia3D use text-to-image diffusion priors for text-conditioned 3D content creation [28, 21, 4]. GALA3D organizes scene generation through layout-guided Gaussian representations [47], while SmartSpatial uses depth and attention control to improve spatial arrangements in generated images [15]. Across these directions, spatial representations provide the interface between scene geometry and task-dependent predictions. Our work organizes object contours, goals, action targets, and future-state targets into coordinate and semantic token sequences, with their roles and temporal order specified by a task grammar. In Push-T, this interface supports joint action/state training and direct closed-loop execution through one autoregressive language model. 3 Method We formulate Spatial Language Modeling as autoregressive completion over grammar-structured token sequences for Push-T. Scene geometry, the goal, pusher actions, and optional future states are represented by coordinate and semantic tokens in a Spatial Language Sentence (SLS). Training sequences contain observed states and rollout targets. At inference, an observed-history prefix prompts the model to sample a completion, which is parsed into pusher actions. The grammar defines the sequence format and parser. 3.1 Spatial Language Modeling Tokenization and Vocabularies. We define the planar workspace as an axis-aligned rectangle with minimum and maximum corners min,max∈ℝ2b_min,b_max ^2. The workspace is discretized into a regular grid with cell size vsv_s and dimensions =(v1,v2)v_g=(v_1,v_2), where =⌈(max−min)/vs⌉v_g= (b_max-b_min)/v_s . All possible grid locations form the coordinate vocabulary coordQ_coord. We project scene geometry onto the grid and represent occupied contour cells as coordinate tokens. Multiple contour points in the same cell map to the same token. Each grid coordinate is represented by one model token with a unique integer ID under a fixed row-major indexing convention. The tuple =(p1,p2)p=(p_1,p_2) is a readable label for this single token. The grid resolution trades geometric precision against vocabulary size and the number of tokens needed to represent contours; it also quantizes action targets. The semantic vocabulary semanticQ_semantic identifies state blocks, entity types, and pusher motion commands. The Push-T symbols and their roles are specified in § 3.2 and the supplementary material. Figure 2: Push-T workflow. The model completes an observed-history SLS prefix into actions, and new observations close the control loop. Symbol Definition Context-Free Grammar SEQ _∗ STATE~ MOVE\_TO^* _ MOVE\_TO move-to[] move-to~ CB~[~ STATE~] STATE state pusher CB+ tblock CB+ goal CB+ CB COORD Lexical Tokens COORD One ID in coordQ_coord, displayed as (p1,p2)(p_1,p_2) with 0≤pj<vj0≤ p_j<v_j Table 1: Push-T sequence format. Each COORD is one token; control decoding omits optional state blocks. Grammar as an Interface. An SLS s is an ordered sequence of tokens from the union of the two vocabularies: ∈(coord∪semantic)n,s∈(Q_coord _semantic)^n, (1) where n is the sequence length. A context-free grammar (CFG; Tab. 1) specifies the ordering and combination of tokens in training sequences. The SEQ rule defines the rollout format. A compiler serializes observations and actions into this format, and a parser converts generated action blocks back into target coordinates. For control, a grammar state machine excludes future-state blocks; action coordinates are sampled autoregressively. 3.2 Application to Push-T We instantiate Spatial Language Modeling for Push-T [6], where a planar pusher moves a T-shaped block to a specified target pose. The input describes the pusher, block, and goal geometry; outputs specify pusher target coordinates and optional subsequent scene states. We define a coordinate system and a bounding rectangle on the tabletop, then discretize it into coordinate tokens. Tab. 1 summarizes the grammar and token definitions, and Fig. 2 illustrates the Push-T workflow. Scene Description. We project the T-block, pusher, and target region onto the plane, keep their outer contours, and represent each entity using the coordinate sequence CB+ of its occupied contour cells. These sequences form the CFG nonterminal STATE. Action Definition and Future Prediction. A _ MOVE\_TO block starts with the semantic token move-to, followed by one coordinate token specifying the target of the pusher center and, optionally, a STATE describing the subsequent scene. Observed subsequent states supply scene-transition targets during training. Action generation and state prediction use the same autoregressive model and next-token objective, so both targets update the shared parameters used to generate actions. During control, the model produces consecutive action blocks, reducing the output token count, and closes the loop through new observations. Rollout Format. Each Push-T rollout starts with STATE followed by a temporally ordered sequence of _ MOVE\_TO productions, matching the SEQ rule. Each step is one pusher movement and corresponds to one scene frame. A _ MOVE\_TO block specifies the target coordinate for that step and optionally the resulting STATE. Rollout discretization makes consecutive pusher positions adjacent on the grid, yielding local displacements in the serialized trajectories. Figure 3: SLS completion for action-conditioned state prediction (left) and action-only control decoding (right). Generated action blocks are parsed into pusher target coordinates. Training and Inference. We compile rollouts into SLS sequences and optimize next-token cross-entropy: ℒ(θ)=−∑i=1nmilogpθ(si∣s<i),L(θ)=- _i=1^nm_i p_θ(s_i s_<i), (2) where mi∈0,1m_i∈\0,1\ indicates whether position i contributes to the loss. In expert-demonstration training, mi=1m_i=1 for all sequence positions. During random-play pretraining, mi=0m_i=0 only for the action-coordinate token immediately following move-to; all remaining tokens, including the command marker and state coordinates, contribute to the loss. Subsequent STATE blocks are independently retained with probability 30%; when omitted, their _ MOVE\_TO blocks contain only the pusher target. Retained blocks provide scene-transition targets, while omitted blocks create consecutive-action subsequences within the same training format. For closed-loop control, the prefix comprises three temporally consecutive STATE–_ MOVE\_TO segments and ends with move-to for the next action coordinate. A grammar state machine excludes future STATE blocks while the model samples action coordinates autoregressively (Fig. 3). Parsed targets are executed, and new observations update the history for the next prefix (Fig. 2). The execution schedule is specified in Appendix B. Pretraining with Random-Play. We collect rollouts in which the pusher moves randomly and may contact the T-block. Action-coordinate tokens supply the conditioning context, with their loss positions masked. Subsequent state tokens provide scene-transition targets conditioned on these random actions. Task-specific training then supervises demonstration sequences, including action coordinates and retained state blocks. Both stages update the same model parameters, combining transition data from random play with expert action supervision. 4 Experiments We evaluate Spatial Language Modeling through closed-loop pushing, action-conditioned future-state prediction, and ablations of joint training and random-play pretraining. Simulation and real-robot experiments examine how the shared sequence model generates executable actions and predicts the geometric effects of pushing. Implementation details are provided in Appendix B and Appendix C.1. 4.1 Push-T Setup and Evaluation We discretize the workspace into a 128×128128× 128 grid and train a 462M-parameter Transformer using the Qwen2 architecture [39] from scratch. Training takes 5 hours on four NVIDIA A100 GPUs. Simulation Data. We use the released 200 expert demonstrations from Diffusion Policy [6]. Our model is trained for 200 epochs, with one checkpoint evaluated every 5 epochs. Our model receives rasterized contour coordinates, while the reported simulation baselines use state inputs. Real-Robot System and Data. A stick mounted on the robot arm serves as the pusher, and an industrial camera observes the tabletop from above (Fig. 1). AprilTags [26] on the T-block and tabletop provide localization, from which the known block geometry is converted to contour coordinates in the calibrated tabletop frame. The arm, camera, and tabletop share this coordinate system. We collect 40 successful expert demonstrations and 20 random-play episodes. The random-play data contains 5,229 frames in total and is used for action-conditioned state-prediction pretraining, with action-coordinate tokens excluded from the loss. Each collected episode contains no more than 300 steps. We compare against the state-based DP-Coord, DP-Tag, and ACT adaptations described in Appendix C.1. These policy baselines are trained on the expert demonstrations, while our full method also uses random-play data. Coverage and Success. Let BtB_t be the set of grid cells occupied by the filled T-block region at step t, and let G be the corresponding set for the target region. Coverage is Ct=|Bt∩G||G|.C_t= |B_t∩ G||G|. (3) A test episode is successful and terminates when Ct≥0.95C_t≥ 0.95. Otherwise, evaluation ends after 300 steps (frames), each corresponding to one pusher movement. Coverage is computed from filled regions on the discrete grid, while model inputs represent their contours. In the real-robot evaluation, each method is tested from 20 initial states, with starting poses manually matched across methods as closely as possible. Our model uses a three-frame observation-history window. The implemented DP baselines use two observation frames and ACT uses one; their input encodings and action-chunk execution settings are specified in Appendix C.1. For real-robot results, Max is the mean of the maximum coverage reached across the test episodes: Max=1N∑e=1Nmaxt∈eCe,t,Max= 1N _e=1^N _t _eC_e,t, (4) where N is the number of test episodes and eT_e contains the evaluated steps of episode e. Success rate is the fraction of these episodes that reach the coverage threshold within 300 steps. Methods Max Avg LG [24] 0.67 0.61 IBC [9] 0.90 0.84 BET [32] 0.79 0.70 DP-C [6] 0.95 0.91 DP-T [6] 0.95 0.79 Ours 0.95 0.93 Methods SR Max DP-Coord 0.45 0.89 DP-Tag 0.50 0.89 ACT 0.65 0.86 Ours 0.80 0.95 Action-only 0.05 0.67 Ours w/o P. 0.65 0.92 Table 2: Push-T results. In simulation (left), Max is the highest average coverage across evaluated checkpoints and Avg is the average coverage over the last 10 checkpoints. Real-robot results (right) report success rate (SR) and the mean of episode-level maximum coverage (Max; Eq. 4) over 20 episodes per method. Action-only deletes future-state blocks and omits random-play pretraining; w/o P. omits pretraining but retains joint action/state training. Figure 4: Distribution of episode-level maximum coverage in the real-robot Push-T evaluation. Figure 5: Future-state prediction on Push-T. Given a manually annotated action trajectory, the model predicts successive states using its own previous predictions as context. Reference states are obtained by executing the same actions in the simulator. Subfigures (a) and (b) show two example rollouts, where red pixels indicate the prediction-error regions of the T-block on the rasterized workspace. Subfigure (c) reports the average state prediction accuracy at each successive step (frame) over the test rollouts, where dashed lines denote coverage and solid lines denote IoU. Figure 6: For each method, images are averaged over the test episodes at their respective maximum-coverage frames. This visualization summarizes alignment across episodes. 4.2 Push-T Results Simulation. Table 2 (left) reports the highest mean coverage across checkpoints (Max) and the mean coverage over the last ten checkpoints (Avg). Our model matches the best Max of 0.95 achieved by DP-C and DP-T, while reaching the highest Avg of 0.93, compared with 0.91 and 0.79, respectively. Its average over the last ten checkpoints is close to its best coverage, showing that strong performance extends beyond the single best checkpoint. These results demonstrate that language-model completion of spatial sequences can produce an effective closed-loop pushing policy. Future-State Prediction. Figure 5 shows the model predicting T-block translation and rotation along manually annotated action trajectories. Each predicted state supplies the context for the next prediction, and the simulator provides reference states under the same actions. The T-block’s coverage changes from 0.98 at the first frame to 0.92 at the tenth, while IoU changes from 0.96 to 0.85. Although error accumulates over the rollout, substantial geometric overlap remains after ten steps (frames). Together, the examples and frame-wise metrics show that the same spatial language model used for action generation also learns action-conditioned scene transitions. Real Robot. Our full method achieves a success rate of 0.80, compared with 0.45 for DP-Coord, 0.50 for DP-Tag, and 0.65 for ACT (Tab. 2, right). It also reaches the highest mean episode-level maximum coverage of 0.95, compared with 0.89 for both DP variants and 0.86 for ACT. The language model’s generated coordinates therefore support successful physical execution as well as close geometric alignment with the target. Figure 4 further shows that our episode-level peak coverage is concentrated around 90%–100%. The baselines include low-overlap episodes, while the full model brings the block close to the target across the tested initial states. Figure 6 provides a complementary view: averaging the maximum-coverage frames yields a clearer T-block outline and fewer ghosting artifacts for our method, indicating more concentrated block positions and orientations. The numerical and visual results together show improvements in task completion and target alignment. Ablations. We compare two training variants with the full method. Action-only deletes the future STATE blocks from the demonstration sequences and uses no random-play pretraining. Ours w/o P. retains joint action and future-state training on the demonstrations, also without pretraining. Adding future-state blocks raises success from 0.05 to 0.65 and mean episode-level maximum coverage from 0.67 to 0.92. Joint training thus improves both task completion and geometric alignment. It gives the shared model targets for the actions to take and the scene changes that follow, making future-state prediction a useful component of policy training. Random-play pretraining further raises success from 0.65 to 0.80 and maximum coverage from 0.92 to 0.95. This stage supplies additional state-transition supervision conditioned on recorded pusher actions, followed by demonstration training for goal-directed control. The two comparisons show complementary benefits from joint action/state training on demonstrations and transition pretraining on random interactions. Both stages use the common spatial vocabulary and update the same autoregressive model. Inference Throughput. On a single A100 GPU, language-model throughput is approximately 7000 tokens/s during prefilling and 800 tokens/s during decoding. During control, the grammar state machine selects the action-only sequence format, in which each target requires one move-to token and one coordinate token. Generating 100 action targets therefore requires about 200 output tokens, corresponding to about 0.25 s of decoding at the measured throughput. The 20-step execution chunk requires about 40 tokens, or approximately 0.05 s of decoding. This compact output format supports rapid action-chunk generation while the same model is trained jointly on actions and future states. 5 Limitations Our method has several limitations. First, the real-robot system relies on calibrated AprilTag localization and known object geometry; end-to-end learning from raw sensor observations has not been evaluated. Second, the grammar is manually designed, while grid resolution trades geometric and action precision against vocabulary size and sequence length. Adapting to new entities, action types, or task structures may require changes to the representation. Third, evaluation focuses on planar Push-T, so generalization to other object geometries, manipulation tasks, and three-dimensional scenes remains to be established. Finally, errors accumulate during autoregressive state rollouts, and prediction over longer horizons requires further evaluation. This limitation concerns state prediction; closed-loop control updates its history with newly observed states. 6 Conclusion We presented Spatial Language Modeling, which expresses scene geometry, goals, action targets, and future states as grammar-structured coordinate sequences. One autoregressive model learns action generation and action-conditioned state prediction through a shared next-token objective. Training combines transition pretraining on random interactions with joint action and state supervision on expert demonstrations. During closed-loop execution, the model generates action targets directly and updates its context using new observations. Experiments demonstrate effective control in simulated and real-world Push-T, while rollouts conditioned on supplied actions capture successive geometric scene changes. Training comparisons show benefits from interleaved action and state sequences, with additional gains from random-play pretraining. These findings support shared spatial modeling as a practical way to connect action-conditioned transition supervision with goal-directed policy learning. References [1] J. Bjorck, F. Castañeda, N. Cherniadev, X. Da, R. Ding, L. Fan, Y. Fang, D. Fox, F. Hu, S. Huang, et al. (2025) GR00T N1: an open foundation model for generalist humanoid robots. arXiv preprint arXiv:2503.14734. Cited by: §1, §2. [2] D. Z. Chen, A. X. Chang, and M. Nießner (2020) Scanrefer: 3d object localization in rgb-d scans using natural language. In European conference on computer vision, p. 202–221. Cited by: §2. [3] L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and I. Mordatch (2021) Decision transformer: reinforcement learning via sequence modeling. arXiv preprint arXiv:2106.01345. Cited by: §2. [4] R. Chen, Y. Chen, N. Jiao, and K. Jia (2023) Fantasia3d: disentangling geometry and appearance for high-quality text-to-3d content creation. In Proceedings of the IEEE/CVF international conference on computer vision, p. 22246–22256. Cited by: §2. [5] T. Chen, S. Saxena, L. Li, D. J. Fleet, and G. Hinton (2021) Pix2seq: a language modeling framework for object detection. arXiv preprint arXiv:2109.10852. Cited by: §2. [6] C. Chi, Z. Xu, S. Feng, E. Cousineau, Y. Du, B. Burchfiel, R. Tedrake, and S. Song (2025) Diffusion policy: visuomotor policy learning via action diffusion. The International Journal of Robotics Research 44 (10-11), p. 1684–1704. Cited by: Appendix B, §C.1, §1, §1, §2, §3.2, §4.1, Table 2, Table 2. [7] S. Fang, I. Shen, Y. Wang, Y. Tsai, Y. Yang, S. Zhou, W. Ding, T. Igarashi, and M. Yang (2025) MeshLLM: empowering large language models to progressively understand and generate 3d mesh. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 14061–14072. Cited by: §2. [8] M. Farazi and Y. Wang (2025) A recipe for geometry-aware 3d mesh transformers. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), p. 3290–3300. Cited by: §2. [9] P. Florence, C. Lynch, A. Zeng, O. A. Ramirez, A. Wahid, L. Downs, A. Wong, J. Lee, I. Mordatch, and J. Tompson (2022) Implicit behavioral cloning. In Conference on robot learning, p. 158–168. Cited by: Table 2. [10] R. Fu, J. Liu, X. Chen, Y. Nie, and W. Xiong (2025) Scene-llm: extending language model for 3d visual reasoning. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), p. 2195–2206. Cited by: §2. [11] J. Gao, T. Shen, Z. Wang, W. Chen, K. Yin, D. Li, O. Litany, Z. Gojcic, and S. Fidler (2022) Get3d: a generative model of high quality 3d textured shapes learned from images. Advances in neural information processing systems 35, p. 31841–31854. Cited by: §2. [12] T. Groueix, M. Fisher, V. G. Kim, B. C. Russell, and M. Aubry (2018) A papier-mâché approach to learning 3d surface generation. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 216–224. Cited by: §2. [13] S. He, H. Ding, X. Jiang, and B. Wen (2024) Segpoint: segment any point cloud via large language model. In European Conference on Computer Vision, p. 349–367. Cited by: §2. [14] Y. Hong, H. Zhen, P. Chen, S. Zheng, Y. Du, Z. Chen, and C. Gan (2023) 3d-llm: injecting the 3d world into large language models. Advances in Neural Information Processing Systems 36, p. 20482–20494. Cited by: §2. [15] M. X. Huang, B. J. Chan, and H. Huang (2025) SmartSpatial: enhancing the 3d spatial arrangement capabilities of stable diffusion models and introducing a novel 3d spatial evaluation framework. arXiv preprint arXiv:2501.01998. Cited by: §2. [16] J. Jang, S. Ye, Z. Lin, J. Xiang, J. Bjorck, Y. Fang, F. Hu, S. Huang, K. Kundalia, Y. Lin, et al. (2025) DreamGen: unlocking generalization in robot learning through video world models. In Proceedings of The 9th Conference on Robot Learning, Proceedings of Machine Learning Research, Vol. 305, p. 5170–5194. Cited by: §1, §2. [17] M. J. Kim, C. Finn, and P. Liang (2025) Fine-tuning vision-language-action models: optimizing speed and success. arXiv preprint arXiv:2502.19645. Cited by: §2. [18] L. Li, Q. Zhang, Y. Luo, S. Yang, R. Wang, F. Han, M. Yu, Z. Gao, N. Xue, X. Zhu, Y. Shen, and Y. Xu (2026) Causal world modeling for robot control. arXiv preprint arXiv:2601.21998. Cited by: §1, §2. [19] Y. Li, Z. Yu, C. Choy, C. Xiao, J. M. Alvarez, S. Fidler, C. Feng, and A. Anandkumar (2023) Voxformer: sparse voxel transformer for camera-based 3d semantic scene completion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 9087–9098. Cited by: §2. [20] J. Liang, R. Liu, E. Ozguroglu, S. Sudhakar, A. Dave, P. Tokmakov, S. Song, and C. Vondrick (2025) Dreamitate: real-world visuomotor policy learning via video generation. In Proceedings of The 8th Conference on Robot Learning, Proceedings of Machine Learning Research, Vol. 270, p. 3943–3960. Cited by: §2. [21] C. Lin, J. Gao, L. Tang, T. Takikawa, X. Zeng, X. Huang, K. Kreis, S. Fidler, M. Liu, and T. Lin (2023) Magic3d: high-resolution text-to-3d content creation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 300–309. Cited by: §2. [22] D. Liu, C. Wang, P. Gao, R. Zhang, X. Ma, Y. Meng, and Z. Wang (2025) 3DAxisPrompt: promoting the 3d grounding and reasoning in gpt-4o. Neurocomputing 637, p. 130072. Cited by: §2. [23] L. Maes, Q. Le Lidec, D. Scieur, Y. LeCun, and R. Balestriero (2026) LeWorldModel: stable end-to-end joint-embedding predictive architecture from pixels. arXiv preprint arXiv:2603.19312. Cited by: §1, §2. [24] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martín-Martín (2021) What matters in learning from offline human demonstrations for robot manipulation. arXiv preprint arXiv:2108.03298. Cited by: Table 2. [25] C. Nash, Y. Ganin, S. M. A. Eslami, and P. Battaglia (2020) Polygen: an autoregressive generative model of 3d meshes. In International conference on machine learning, p. 7220–7229. Cited by: §2. [26] E. Olson (2011) AprilTag: a robust and flexible visual fiducial system. In 2011 IEEE international conference on robotics and automation, p. 3400–3407. Cited by: §4.1. [27] Physical Intelligence, K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, et al. (2025) π0.5 _0.5: a vision-language-action model with open-world generalization. arXiv preprint arXiv:2504.16054. Cited by: §1, §2. [28] B. Poole, A. Jain, J. T. Barron, and B. Mildenhall (2022) Dreamfusion: text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.14988. Cited by: §2. [29] C. R. Qi, H. Su, K. Mo, and L. J. Guibas (2017) Pointnet: deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 652–660. Cited by: §2. [30] C. R. Qi, L. Yi, H. Su, and L. J. Guibas (2017) Pointnet++: deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems 30. Cited by: §2. [31] K. Rana, J. Haviland, S. Garg, J. Abou-Chakra, I. Reid, and N. Suenderhauf (2023) Sayplan: grounding large language models using 3d scene graphs for scalable robot task planning. arXiv preprint arXiv:2307.06135. Cited by: §2. [32] N. M. Shafiullah, Z. Cui, A. A. Altanzaya, and L. Pinto (2022) Behavior transformers: cloning k modes with one stone. Advances in neural information processing systems 35, p. 22955–22968. Cited by: §2, Table 2. [33] Y. Siddiqui, A. Alliegro, A. Artemov, T. Tommasi, D. Sirigatti, V. Rosov, A. Dai, and M. Nießner (2024) Meshgpt: generating triangle meshes with decoder-only transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 19615–19625. Cited by: §2. [34] J. Wald, H. Dhamo, N. Navab, and F. Tombari (2020) Learning 3d semantic scene graphs from 3d indoor reconstructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 3961–3970. Cited by: §2. [35] Z. Wang, J. Lorraine, Y. Wang, H. Su, J. Zhu, S. Fidler, and X. Zeng (2024) Llama-mesh: unifying 3d mesh generation with language models. arXiv preprint arXiv:2411.09595. Cited by: §2. [36] J. Wu, C. Zhang, T. Xue, B. Freeman, and J. Tenenbaum (2016) Learning a probabilistic latent space of object shapes via 3d generative-adversarial modeling. Advances in neural information processing systems 29. Cited by: §2. [37] W. Wu, Z. Qi, and L. Fuxin (2019) Pointconv: deep convolutional networks on 3d point clouds. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, p. 9621–9630. Cited by: §2. [38] R. Xu, X. Wang, T. Wang, Y. Chen, J. Pang, and D. Lin (2024) Pointllm: empowering large language models to understand point clouds. In European Conference on Computer Vision, p. 131–147. Cited by: §2. [39] A. Yang, B. Yang, B. Hui, et al. (2024) Qwen2 technical report. arXiv preprint arXiv:2407.10671. Cited by: Appendix A, §4.1. [40] G. Yang, X. Huang, Z. Hao, M. Liu, S. Belongie, and B. Hariharan (2019) Pointflow: 3d point cloud generation with continuous normalizing flows. In Proceedings of the IEEE/CVF international conference on computer vision, p. 4541–4550. Cited by: §2. [41] L. Yang, C. Yuan, Z. Zhang, Z. Qi, Y. Xu, W. Liu, Y. Shan, B. Li, W. Yang, P. Li, Y. Wang, and W. Hu (2023) Exploiting contextual objects and relations for 3d visual grounding. Advances in Neural Information Processing Systems 36, p. 49542–49554. Cited by: §2. [42] R. Zhang, Z. Guo, W. Zhang, K. Li, X. Miao, B. Cui, Y. Qiao, P. Gao, and H. Li (2022) Pointclip: point cloud understanding by clip. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 8552–8562. Cited by: §2. [43] H. Zhao, L. Jiang, J. Jia, P. H. Torr, and V. Koltun (2021) Point transformer. In Proceedings of the IEEE/CVF international conference on computer vision, p. 16259–16268. Cited by: §2. [44] T. Z. Zhao, V. Kumar, S. Levine, and C. Finn (2023) Learning fine-grained bimanual manipulation with low-cost hardware. In Proceedings of Robotics: Science and Systems, External Links: Document Cited by: §C.1, §1, §2. [45] H. Zhi, P. Chen, J. Li, S. Ma, X. Sun, T. Xiang, Y. Lei, M. Tan, and C. Gan (2025) Lscenellm: enhancing large 3d scene understanding using adaptive visual preferences. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 3761–3771. Cited by: §2. [46] S. Zhou, Y. Du, J. Chen, Y. Li, D. Yeung, and C. Gan (2024) RoboDreamer: learning compositional world models for robot imagination. In Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 235, p. 61885–61896. Cited by: §2. [47] X. Zhou, X. Ran, Y. Xiong, J. He, Z. Lin, Y. Wang, D. Sun, and M. Yang (2024) GALA3D: towards text-to-3d complex scene generation via layout-guided generative gaussian splatting. arXiv preprint arXiv:2402.07207. Cited by: §2. [48] X. Zhu, R. Zhang, B. He, Z. Guo, Z. Zeng, Z. Qin, S. Zhang, and P. Gao (2023) Pointclip v2: prompting clip and gpt for powerful 3d open-world learning. In Proceedings of the IEEE/CVF international conference on computer vision, p. 2639–2650. Cited by: §2. Appendix A Model Implementation We employ a 462M-parameter Qwen2 model [39] and train it from scratch. Specifically, the model consists of 7 transformer layers with a hidden dimension of 1536 and 24 attention heads. Rotary Position Embeddings (RoPE) are used for positional encoding. Push-T uses 16,384 coordinate tokens for its 128×128128× 128 grid, together with the semantic tokens defining its sequence format. We pack multiple spatial language sentences into training samples with a maximum context length of 13,000. Appendix B Push-T Representation and Execution Details We provide the grammar definition for the Push-T task [6]. In Push-T, a pusher moves a T-shaped block to a goal region. The observation and action are both represented on a discretized 2D grid and serialized as a Spatial Language Sentence (SLS). 2D Representation and Vocabularies. We discretize the workspace in 2D. Each grid location is encoded as one coordinate token ∈coord CB _coord. In the simulator, we render the pusher, the T-block, and the goal region, extract their edge pixels, and represent each entity by its contour coordinate tokens. State (Scene) Description. A Push-T state is composed of three components: the pusher geometry, the T-block geometry, and the goal-region geometry. We define the nonterminal STATE to serialize these components as →statepusher+tblock+goal+. STATE→ state~ pusher~ CB^+~ tblock~ CB^+~ goal~ CB^+. Here, the terminal tokens state, pusher, tblock, and goal belong to semanticQ_semantic, and each + CB^+ is a (sorted) sequence of coordinate tokens that describes the corresponding entity. Action Description. We represent a pushing action as moving the pusher to a target 2D coordinate. Each such movement is one step and corresponds to one scene frame. Accordingly, the nonterminal _ MOVE\_TO is defined as _→move-to[]. MOVE\_TO→ move-to~ CB~[~ STATE~]. The optional bracket [][~ STATE~] indicates that we may append the next observation state after executing the action, enabling multiple actions to be represented within a single sequence. The Push-T task is then defined by the 2D production rule: 2D→_∗. SEQ_2D→ STATE~ MOVE\_TO^*. The sequence format consists of an initial STATE and temporally ordered action blocks, optionally interleaved with resulting states. Training trajectories follow this format. During control, a grammar state machine excludes future STATE blocks; action coordinates are sampled autoregressively and parsed into pusher targets. Closed-loop execution at test time. At test time, the prefix contains three temporally consecutive STATE–_ MOVE\_TO segments and ends with move-to for the next target coordinate. We execute the first 20 predicted move-to targets from the action-only continuation, one per step (frame), then update the history with new observations for the next prompt. Training objectives. Task demonstration training uses next-token prediction over the serialized trajectories. Random-play pretraining masks only the target-coordinate token immediately following each move-to marker; the marker and all other tokens remain supervised. Target coordinates remain in the context to condition subsequent state predictions. Action generation and state prediction share model parameters. Success criterion. Let BtB_t be the set of filled grid cells occupied by the T-block at time t, and let G be the set of filled grid cells in the target region. Coverage is Coverage(t)=|Bt∩G||G|.Coverage(t)= |B_t∩ G||G|. Evaluation episodes terminate successfully at 0.950.95 coverage or unsuccessfully after 300 steps (frames), corresponding to at most 300 pusher movements. Coverage uses filled grid regions; model inputs use their contours. For the real-robot results, Max averages each episode’s maximum coverage over the 20 test episodes, as defined in Eq. 4. SR is the fraction of successful episodes. The simulation table instead reports the highest mean coverage across evaluated checkpoints and the mean over the last 10 checkpoints. Appendix C Full Symbol and Token Definitions This section complements the brief grammar description in the main paper by explaining the symbols used for Push-T. Vocabularies. All SLS tokens come from the union of the coordinate vocabulary coordQ_coord and the semantic vocabulary semanticQ_semantic. Coordinate tokens represent discretized locations on the planar grid. The Push-T semantic tokens are: semantic=state,pusher,tblock,goal,move-to.Q_semantic=\ state,\ pusher,\ tblock,\ goal,\ move-to\. Lexical token: COORD. Each COORD is one vocabulary token identifying one discrete grid cell. The tuple (x,y)(x,y) is readable notation for that single token ID. Coordinate block symbol: CB. In the CFG, a coordinate block CB expands to a single COORD token. A repetition + CB^+ denotes a non-empty list of coordinates representing the discretized geometry of an entity. 2D task symbols (Push-T). The 2D production 2D SEQ_2D defines a Push-T rollout: 2D→_∗. SEQ_2D→ STATE~ MOVE\_TO^*. State block: STATE. →statepusher+tblock+goal+ STATE→ state~ pusher~ CB^+~ tblock~ CB^+~ goal~ CB^+ describes one observation, where the coordinate sequences correspond to edge pixels extracted from the rendered pusher, T-block, and goal region. Action block: _ MOVE\_TO. _→move-to[] MOVE\_TO→ move-to~ CB~[~ STATE~] represents one action that moves the pusher to the target coordinate CB. The optional [][~ STATE~] allows appending the next observation after execution. Canonicalization (coordinate serialization). We sort each entity-specific geometry list + CB^+ by a fixed grid order (x, then y). Sorting is confined to each entity block. C.1 Baseline Implementation Details All three baselines (DP-Coord, DP-Tag, and ACT) operate in the state-based real-world Push-T setting. Their per-frame inputs describe the manipulated T-block together with the pusher-center coordinate t∈ℝ2p_t ^2. Our SLS input serializes contour coordinates for the pusher, T-block, and goal (§3.2). The baselines use two T-block representations: • Object contour coordinates (used by DP-Coord and ACT): the T-block contour coordinates Xt=t,ii=1KX_t=\x_t,i\_i=1^K, t,i∈ℝ2x_t,i ^2, taken from the discretized outer contour used in our STATE block. The number of occupied cells varies per frame, so each frame is sentinel-padded to a dataset-wide fixed count K=100K=100 with ¯=(−1,−1) x=(-1,-1) marking empty slots. • AprilTag corner coordinates (used by DP-Tag): the S=4||=12S=4\, =12 corner coordinates from the ||=3 =3 AprilTags on the top surface of the T-block (four corners each), placed in a fixed slot order with no padding. For all baselines, the action is the absolute target pusher coordinate t∈ℝ2a_t ^2, i.e. the same quantity our _ MOVE\_TO production emits. Relative to their default implementations, the observation encoding (and, for ACT, the token construction) is modified; the action heads and training objectives are inherited unchanged. Diffusion Policy uses two observation frames, ACT uses one frame, and our model uses three frames. DP-Coord / DP-Tag (Diffusion Policy). Both Diffusion Policy baselines use low-dimensional Diffusion Policy [6]: a conditional 11-D U-Net ϵθ ε_θ trained with DDPM and conditioned on the observation as a global conditioning vector. We adapt the original Push-T low-dimensional input, which contains nine ground-truth keypoints, to a flattened geometry vector concatenated with the pusher coordinate, t=[vec(Xt)∥t]∈ℝ2K+2,DP-Coord (contour coords),ℝ2S+2,DP-Tag (tag coords),o_t\;=\; [\,vec(X_t)\ \|\ p_t\, ]\;∈\; casesR^2K+2,&DP-Coord (contour coords),\\[2.0pt] R^2S+2,&DP-Tag (tag coords), cases (5) where vec(⋅)vec(·) stacks the coordinates in their fixed order. Each dimension is normalized to [−1,1][-1,1] (limits normalizer). DP-Coord feeds the sentinel-padded contour through the fixed-width global-conditioning path with no explicit padding mask. Stacking no=2n_o=2 observation steps yields the conditioning t=[t−1;t]c_t=[o_t-1;o_t] passed to ϵθ ε_θ. The output is the default Diffusion Policy receding-horizon scheme: denoise an action trajectory ^t:t+H∈ℝH×2 a_t:t+H ^H× 2 of horizon H=16H=16 and execute the first na=8n_a=8 steps before re-planning. ACT. The default ACT [44] (DETR-VAE) feeds a ResNet image feature map (H′×W′H \!×\!W tokens) as the cross-attention memory of its transformer decoder, plus a proprioception token from the agent state and a CVAE latent token. For our state-based task, we replace the CNN feature tokens with per-coordinate tokens over the object contour coordinates. Each coordinate is lifted to the hidden dimension d by a single shared learnable linear map ϕ:ℝ2→ℝdφ:R^2\!→\!R^d and combined with a learnable per-slot positional embedding ie_i, t,i=ϕ(t,i)+i,i=1,…,K,m_t,i\;=\;φ(x_t,i)+e_i, i=1,…,K, (6) giving K memory tokens. The pusher coordinate is encoded by the same kind of learnable linear that default ACT applies to the agent state, tprop=ψ(t)m^prop_t=ψ(p_t), ψ:ℝ2→ℝdψ:R^2\!→\!R^d, and prepended together with the CVAE latent token. Padded contour slots are excluded from attention through a key-padding mask μt,i=[t,i=¯] _t,i=1[x_t,i= x], so a single model serves the variable-length contour. The CVAE encoder, the latent prior, and the L1L_1 + KL training objective are identical to default ACT. The decoder uses H=16H=16 learnable query tokens and a linear action head to emit the chunk ^1:H∈ℝH×2 a_1:H ^H× 2; at inference the latent is fixed to its prior mean (0). ACT conditions on a single observation step (no=1n_o=1) and executes the first na=8n_a=8 of the 1616 predicted actions per re-plan.