Paper deep dive
Orbit-Planner: Towards Latent World Models for On-Orbit Obstacle Avoidance of Satellite Agents
Zhijian Li, Chao Ren, Peijin Wang, Xian Sun
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 8/23/2026, 2:31:24 AM
Summary
The paper introduces Orbit-Planner, a two-stage latent world model designed for on-orbit obstacle avoidance in satellite agents. It utilizes a Vision Transformer and MLP-based encoders to create latent representations from RGB images and spacecraft states, predicting future latent states via an autoregressive Transformer conditioned on thruster actions. A Physics Probe decodes these latent trajectories into physical state increments, enabling Model Predictive Path Integral (MPPI) control. The system achieves a 91.7% success rate in closed-loop navigation within the Isaac Sim environment, outperforming imitation learning baselines.
Entities (8)
Relation Signals (7)
Orbit-Planner → achievesmetric → 91.7% success rate
confidence 95% · In closed-loop obstacle-avoidance navigation in Isaac Sim, it attains a success rate of 91.7%.
Orbit-Planner → evaluatedin → Isaac Sim
confidence 95% · In closed-loop obstacle-avoidance navigation in Isaac Sim
Orbit-Planner → usescomponent → Physics Probe
confidence 95% · Orbit-Planner ... introduces a Physics Probe to decode physical state changes from imagined latent trajectories.
Orbit-Planner → outperforms → Diffusion Policy
confidence 90% · Orbit-Planner attains an average success rate of 91.7%, substantially outperforming Diffusion Policy (55.0%).
Orbit-Planner → targetsagent → CubeSat
confidence 90% · We consider a discrete-time dynamical system for a CubeSat agent
Orbit-Planner → usesalgorithm → MPPI
confidence 90% · Orbit-Planner employs MPPI [9] ... and selects the sequence with the lowest predicted collision risk
Orbit-Planner → usescomponent → Vision Transformer
confidence 90% · A Vision Transformer (ViT) acts as the visual encoder
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Satellite agents for on-orbit navigation tasks need to predict collision risks using limited onboard observations. However, conventional planners often rely on predefined maps and fixed environmental assumptions, limiting their adaptability in dynamic on-orbit scenarios. In this paper, we propose Orbit-Planner, a two-stage latent world model for on-orbit obstacle avoidance. Orbit-Planner learns action-conditioned spacecraft dynamics to perform future-state rollouts in latent space, and introduces a Physics Probe to decode physical state changes from imagined latent trajectories. Experiments demonstrate that Orbit-Planner can perform long-horizon latent rollouts and recover physical states from imagined trajectories. In closed-loop obstacle-avoidance navigation in Isaac Sim, it attains a success rate of 91.7%. Code is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2608.16651v1
- Canonical: https://arxiv.org/abs/2608.16651v1
Trouble viewing inline? Open PDF directly →
Full Text
18,657 characters extracted from source content.
Expand or collapse full text
Orbit-Planner: Towards Latent World Models for On-Orbit Obstacle Avoidance of Satellite Agents Thanks: This work was supported by the National Key R&D Program of China under Grant 2024YFF1401001, and partially supported by China Science and Technology Cloud (CSTCloud). Zhijian Li, Chao Ren, Peijin Wang and Xian Sun are with the Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100190, China, also with the School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences, Beijing 100190, China, also with the University of Chinese Academy of Sciences, Beijing 100190, China, and also with the Key Laboratory of Target Cognition and Application Technology (TCAT), Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100094, China (e-mail: lizhijian25@mails.ucas.ac.cn, renc0003@e.ntu.edu.sg, wangpeijin17@mails.ucas.ac.cn, sunxian@aircas.ac.cn). 1Corresponding author: Chao Ren. Zhijian Li, Chao Ren1, Peijin Wang, Xian Sun Affiliation: Aerospace Information Research Institute, Chinese Academy of Sciences, 100094 Beijing, China lizhijian25@mails.ucas.ac.cn, renc0003@e.ntu.edu.sg Code HuggingFace Project Page Abstract Satellite agents for on-orbit navigation tasks need to predict collision risks using limited onboard observations. However, conventional planners often rely on predefined maps and fixed environmental assumptions, limiting their adaptability in dynamic on-orbit scenarios. In this paper, we propose Orbit-Planner, a two-stage latent world model for on-orbit obstacle avoidance. Orbit-Planner learns action-conditioned spacecraft dynamics to perform future-state rollouts in latent space, and introduces a Physics Probe to decode physical state changes from imagined latent trajectories. Experiments demonstrate that Orbit-Planner can perform long-horizon latent rollouts and recover physical states from imagined trajectories. In closed-loop obstacle-avoidance navigation in Isaac Sim, it attains a success rate of 91.7%91.7\%. Code is available at https://github.com/ZhijianLi2003/Orbit_Planner. Index Terms: Latent world models, Space robotics, Autonomous navigation, On-orbit obstacle avoidance I Introduction With the rapid increase in space exploration and on-orbit assembly tasks, autonomous satellite agents are playing an increasingly critical role [6]. In these complex and dynamic orbital environments, effective obstacle avoidance is paramount to ensure the safety and success of space missions. Traditional methods often rely on predefined environmental parameters and classical path-planning algorithms, such as Artificial Potential Fields and A* search [4]. Imitation-learning policies [1] can generate reactive maneuvers from demonstrations, but they often generalize poorly when the test-time dynamics deviate from the demonstrated distribution. To overcome these issues, world models have emerged as a transformative paradigm. By extracting compact predictive representations from high-dimensional sensory inputs, these models empower agents to internally simulate future states and evaluate actions prior to physical execution [2, 3]. Notably, recent breakthroughs in Joint-Embedding Predictive Architectures (JEPA) have enabled stable, end-to-end learning of such models [5], demonstrating remarkable efficacy in highly dynamic control scenarios like agile quadrotor flight [8]. Inspired by these advancements, latent world models present a compelling pathway for space robotics. Fig. 1: Overview of the on-orbit obstacle-avoidance task and the overall pipeline. Left: a CubeSat navigates from a start position to a target through a cluttered orbital field of obstacles. Right: the pipeline of data collection, world-model training, and testing. The main contributions of this paper are as follows: 1) we construct an on-orbit obstacle-avoidance dataset for satellite agents based on Isaac Sim; 2) we propose Orbit-Planner, a two-stage latent world model that learns action-conditioned dynamics for future-state rollout, with a Physics Probe to recover physical states from imagined trajectories; 3) based on this world model, we achieve online on-orbit obstacle-avoidance navigation, with a 91.7%91.7\% success rate. I Methodology Fig. 2: Overview of the proposed Orbit-Planner framework. Stage I pre-trains the latent world model to represent multimodal observations (RGB sequences and spacecraft states) and perform action-conditioned latent rollouts. In Stage I, the Physics Probe maps rolled-out latents to future spacecraft-state increments, while the depth decoder recovers current obstacle geometry; both are used for trajectory planning. I-A Problem Formulation We consider a discrete-time dynamical system for a CubeSat agent operating in a dynamic and cluttered orbital environment. As illustrated in Fig. 1, the primary objective is to navigate the satellite safely to a target destination while minimizing the probability of collision. The system is characterized by its state t∈ℝ16 s_t ^16 and control action t∈ℝ8 a_t ^8. For the CubeSat, we define the state as: t=[t⊤t⊤t⊤t⊤ϕt]⊤. s_t=[ p_t 10.00002pt v_t 10.00002pt r_t 10.00002pt ω_t 10.00002pt _t] . (1) Here, t∈ℝ3 p_t ^3 denotes the position displacement in the inertial (world) frame relative to the initial position (0= p_0= 0). The linear velocity t∈ℝ3 v_t ^3 and angular velocity t∈ℝ3 ω_t ^3 are expressed in the body frame. The attitude is represented by a continuous 6D rotation vector t∈ℝ6 r_t ^6, derived from the first two columns of the rotation matrix t∈SO(3) R_t∈ SO(3) representing the attitude of the body frame in the inertial frame. Additionally, ϕt∈[0,1] _t∈[0,1] represents the remaining fuel ratio. The discrete time step is set to Δt=0.04 t=0.04 s (25 Hz25\,Hz). The control action t∈ℝ8 a_t ^8 denotes the normalized thrust commands for the 8 onboard thrusters: t=[F1,t,F2,t,…,F8,t]⊤∈[0,1]8. a_t=[F_1,t,F_2,t,…,F_8,t] ∈[0,1]^8. (2) The system operates under the assumption that the agent’s physical state t s_t is fully observable. To capture surrounding environmental information, the agent utilizes an onboard RGB camera to obtain visual observations t∈ℝH×W×3 o_t ^H× W× 3. I-B Orbit-Planner World Model Framework As illustrated in Fig. 2, Orbit-Planner consists of two stages: world-model pre-training, followed by physics probing and planning. I-B1 Stage I: World Model Pre-training In the first stage, we pre-train a latent world model to capture the complex dynamics of the space environment. This stage fundamentally consists of two processes: representation and rollout. During the representation phase, the model takes RGB observations t o_t and low-dimensional spacecraft states t s_t as inputs. A Vision Transformer (ViT) acts as the visual encoder, extracting a global class token and dense patch tokens. Concurrently, an MLP-based State Encoder maps the physical state into a state embedding. A Projector module then fuses the class token and the state embedding to generate the latent representation t z_t, which jointly encodes the agent’s own physical state and the surrounding environmental geometry. Crucially, this projector employs Batch Normalization to counteract the LayerNorm effects from the ViT, ensuring that the variance-based regularization (SIGReg [5]) functions effectively on the latent distribution without collapsing. The dense patch tokens are fed to a dedicated depth decoder for depth prediction. During the rollout phase, the control action t a_t is introduced. The thruster commands are processed by an MLP-based Action Encoder to produce an action embedding. The Latent Dynamics Prediction module, parameterized by an autoregressive Transformer equipped with Adaptive Layer Normalization (AdaLN), then utilizes the current latent state t z_t and the action embedding to predict the future latent state t+1 z_t+1. Specifically, the action embedding acts as the conditioning signal in the AdaLN blocks [7], dynamically modulating the latent features to ensure the physical actions effectively guide the future-state rollout. To ensure the learned representations are physically meaningful and geometrically consistent, the training is guided by a composite loss function ℒL: ℒ=ℒMSE+ℒSIGReg+ℒDepth,L=L_MSE+L_SIGReg+L_Depth, (3) which incorporates the mean squared error (ℒMSEL_MSE) for latent dynamics prediction, variance-based regularization constraints (ℒSIGRegL_SIGReg) to prevent feature collapse, and depth supervision (ℒDepthL_Depth) via a dedicated depth decoder. The depth loss is included mainly to encourage the visual encoder to capture depth cues that are critical for navigation. I-B2 Stage I: Physics Probing and Planning After pre-training, we freeze the world model and use it to roll out future states. Given a sequence of past observations (t−m+1,…,t)( o_t-m+1,…, o_t), the corresponding states (t−m+1,…,t)( s_t-m+1,…, s_t), and a sequence of proposed future actions (t,…,t+n−1)( a_t,…, a_t+n-1), the world model predicts the corresponding future latent states (t+1,…,t+n)( z_t+1,…, z_t+n). During training, the observation context length is set to m=8m=8 and the rollout horizon is set to n=12n=12. To map latent states to physical quantities, we introduce a Physics Probe and train only this probe while keeping the world model frozen. The probe translates the latent states into physical state transitions (Δt+1,…,Δt+n)( s_t+1,…, s_t+n). Simultaneously, the frozen depth decoder extracts obstacle information, converting it into a Depth-to-Point Cloud representation. Fig. 3: Latent-space rollout prediction error versus horizon. Thin blue curves denote individual trajectories; the red curve denotes the mean MSE. I-C Autonomous Control and Decision-Making Given the probe-predicted state rollouts and obstacle point clouds, Orbit-Planner employs MPPI [9] with K=256K=256 samples and a horizon of H=50H=50 steps, and selects the sequence with the lowest predicted collision risk while progressing toward the target. Only the first action of the selected sequence is executed before replanning. I Experiments I-A Experimental Setup We collect data in a space robotics simulation environment [6] based on Isaac Sim. The dataset contains 8000 on-orbit obstacle-avoidance trajectories, split into 7200 for training and 800 for testing. To improve the robustness of the learned representations under visual and geometric variations, extensive domain randomization is applied during data collection, encompassing variations in obstacle positions and lighting conditions. Furthermore, to improve behavioral diversity, trajectories are gathered under three distinct levels. Expert trajectories are generated by RRT∗ path planning followed by PD tracking, with larger safety margins and lower control noise. Risky trajectories fly straight toward the goal without obstacle-aware planning. Exploratory trajectories inject strong random action noise for unstructured exploration. Fig. 4: RGB-to-depth prediction results. Both the ground-truth and predicted depth maps are downsampled to 16×1616× 16. TABLE I: Physics Probe errors on absolute states recovered via kinematic integration from predicted increments Δ s (mean ± std over 100 randomly selected testing trajectories). Quantity MAE (h=25,t=1sh=25,t=1s) MAE (h=50,t=2sh=50,t=2s) Position p (m) 0.0394±0.05540.0394± 0.0554 0.0813±0.06610.0813± 0.0661 Velocity v (m/s) 0.0636±0.09630.0636± 0.0963 0.1191±0.11020.1191± 0.1102 Rotation (geodesic, rad) 0.0676±0.09090.0676± 0.0909 0.2034±0.18260.2034± 0.1826 Angular velocity ω (rad/s) 0.0883±0.10850.0883± 0.1085 0.1557±0.14530.1557± 0.1453 Fuel ϕφ (normalized) 0.0019±0.00150.0019± 0.0015 0.0033±0.00210.0033± 0.0021 I-B Latent Space Rollout Fig. 5: Physics Probe qualitative results on a 50-step (2.0 s2.0\,s) prediction horizon. (a) Predicted versus ground-truth physical-state components on a collision-free trajectory. (b) 3D comparison of a collision-free window and a collision window. We evaluate action-conditioned multi-step prediction in the latent space z. Starting from an encoded context, the frozen world model recursively rolls out future latents under the recorded action sequence, and we measure the MSE against encoder-derived target latents at each horizon. Fig. 3 shows the per-trajectory errors (blue) and their mean (red). As expected for autoregressive imagination, the error accumulates with the horizon; nevertheless, the mean MSE grows gradually and remains moderate even at a 50-step horizon, indicating that the learned latent dynamics support stable long-horizon rollouts for action-conditioned trajectory imagination. Fig. 4 shows that the depth decoder recovers obstacle structure, indicating that z preserves useful geometric cues. I-C Physics Probe We evaluate the accuracy of physical-state recovery from action-conditioned latent rollouts. With the world model’s encoder and predictor frozen, an 8-step observation context is encoded into initial latent states. The model then autoregressively rolls out future latents conditioned on the recorded action sequences. Subsequently, the Physics Probe maps each predicted latent state to physical state increments Δ s, which are kinematically integrated from the last context state to reconstruct the full physical trajectory. We compare these recovered states against the ground truth and report the Mean Absolute Error (MAE) across 25-step and 50-step prediction horizons. As shown in Table I, position, velocity, and fuel are recovered with low error, whereas angular velocity exhibits slightly higher variance due to its highly dynamic nature. Fig. 5 provides a qualitative comparison. Within a limited time horizon, the collision-free rollout remains aligned with the ground-truth trajectory. In a prediction window that contains a collision event, the error between the rollout and the recorded trajectory becomes substantially larger, which indirectly suggests that the model has captured action-conditioned dynamics rather than merely replaying the observed outcome. These results indicate that the action-conditioned latent rollouts preserve physically meaningful dynamics that can be decoded by the probe. I-D On-Orbit Obstacle Avoidance Fig. 6: Closed-loop success rates of Orbit-Planner and Diffusion Policy [1] in Isaac Sim under six settings (1/3/5 obstacles × v0=1/2v_0=1/2 m/s) and their average. We further evaluate closed-loop obstacle-avoidance navigation in Isaac Sim. Orbit-Planner uses the frozen world model to imagine action-conditioned futures and select collision-free maneuvers, and is compared against an imitation-learning baseline, Diffusion Policy [1]. Both methods are tested under six settings that vary the number of obstacles and the initial velocity, with 10 episodes per setting. In each episode, the obstacles are randomly placed and unseen during training. As shown in Fig. 6, Orbit-Planner attains an average success rate of 91.7%91.7\%, substantially outperforming Diffusion Policy (55.0%55.0\%). The advantage is consistent across all six settings and remains pronounced at higher speed and denser obstacle fields. These results indicate that action-conditioned latent rollouts enable anticipatory, physics-aware decisions, whereas a purely reactive imitation policy struggles to generalize when the scene dynamics deviate from the demonstrated distribution. IV Conclusion In this paper, we presented Orbit-Planner, a latent world model for on-orbit obstacle avoidance by autonomous satellite agents. Orbit-Planner learns action-conditioned spacecraft dynamics to perform future-state rollouts in latent space, employing a Physics Probe to decode physical state transitions from imagined trajectories. Experimental results demonstrate that the proposed model achieves long-horizon latent rollouts, physical-state readouts, and higher closed-loop navigation success than an imitation-learning baseline. In future work, we will study the sim-to-real gap under more realistic sensing and dynamics conditions. References [1] C. Chi, Z. Xu, S. Feng, E. Cousineau, Y. Du, B. Burchfiel, R. Tedrake, and S. Song (2025) Diffusion policy: visuomotor policy learning via action diffusion. The International Journal of Robotics Research 44 (10-11), p. 1684–1704. Cited by: §I, Fig. 6, §I-D. [2] D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap (2025) Mastering diverse control tasks through world models. Nature 640 (8059), p. 647–653. Cited by: §I. [3] N. Hansen, H. Su, and X. Wang (2024) TD-MPC2: scalable, robust world models for continuous control. In International Conference on Learning Representations, Cited by: §I. [4] Y. Li, S. Yue, and Z. Du (2025) Obstacle avoidance method for on-orbit assembly based on artificial potential field and improved a* path-planning algorithm. IFAC-PapersOnLine 59 (20), p. 1350–1355. Cited by: §I. [5] L. Maes, Q. L. Lidec, D. Scieur, Y. LeCun, and R. Balestriero (2026) Leworldmodel: stable end-to-end joint-embedding predictive architecture from pixels. arXiv preprint arXiv:2603.19312. Cited by: §I, §I-B1. [6] A. Orsula, M. Geist, M. Olivares-Mendez, and C. Martinez (2025) Space robotics bench: robot learning beyond earth. arXiv preprint arXiv:2509.23328. Cited by: §I, §I-A. [7] W. Peebles and S. Xie (2023) Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, p. 4195–4205. Cited by: §I-B1. [8] P. Rao, W. Zhang, R. Balestriero, Y. LeCun, and G. Loianno (2026) SkyJEPA: learning long-horizon world models for zero-shot sim-to-real control of quadrotors. arXiv preprint arXiv:2606.23444. Cited by: §I. [9] G. Williams, A. Aldrich, and E. Theodorou (2015) Model predictive path integral control using covariance variable importance sampling. arXiv preprint arXiv:1509.01149. Cited by: §I-C.