Paper deep dive
Reward-Free Continual Adaptation for Resilient Space Robots
Andrej Orsula, Miguel Olivares-Mendez, Carol Martinez
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Space robots operate in extreme environments where hardware degradation can critically compromise traditional control strategies. While continual reinforcement learning offers a promising mechanism for online adaptation, it inherently requires access to a reward signal during deployment. However, precise reward computation in space is often infeasible due to the lack of external tracking systems and the overall complexity of the environment. To address the challenge of unobservable rewards, we introduce a reward-free continual learning framework that leverages latent-state world models. By pre-training a model-based agent across diverse simulations, the world model learns a robust predictor of the reward structure within its latent space. Upon deployment to an environment with severe hardware degradation, we freeze the observation encoder and reward predictor to update only the transition dynamics of the world model through unsupervised rollouts. By training the policy entirely on imagined trajectories generated by this updated world model, the agent adapts to altered dynamics without receiving new rewards. We demonstrate our approach across simulated planetary traversal, orbital navigation, and precision assembly tasks subjected to severe morphological failures.
Tags
Links
- Source: https://arxiv.org/abs/2608.23452v1
- Canonical: https://arxiv.org/abs/2608.23452v1
Trouble viewing inline? Open PDF directly →
Full Text
22,268 characters extracted from source content.
Expand or collapse full text
Reward-Free Continual Adaptation for Resilient Space Robots Andrej Orsula ∗ , Miguel Olivares-Mendez, Carol Martinez University of Luxembourg Space robots operate in extreme environments where hard- ware degradation can critically compromise traditional con- trol strategies. While continual reinforcement learning of- fers a promising mechanism for online adaptation, it inher- ently requires access to a reward signal during deployment. However, precise reward computation in space is often in- feasible due to the lack of external tracking systems and the overall complexity of the environment. To address the challenge of unobservable rewards, we introduce a reward- free continual learning framework that leverages latent- state world models. By pre-training a model-based agent across diverse simulations, the world model learns a robust predictor of the reward structure within its latent space. Upon deployment to an environment with severe hardware degradation, we freeze the observation encoder and reward predictor to update only the transition dynamics of the world model through unsupervised rollouts. By training the policy entirely on imagined trajectories generated by this updated world model, the agent adapts to altered dy- namics without receiving new rewards. We demonstrate our approach across simulated planetary traversal, orbital navigation, and precision assembly tasks subjected to se- vere morphological failures.The source code is available at https://github.com/AndrejOrsula/space_robotics_bench. 1 Introduction Robots are positioned to deliver transformative changes for the future of space exploration and utilization. Ambitious missions envision fleets of rovers exploring unstructured planetary surfaces and mobile robotic manipulators con- structing orbital megastructures [1]. Yet, the success of these long-duration endeavors depends heavily on devel- oping robotic systems that can operate reliably and adapt to extreme conditions with minimal human supervision. Data-driven approaches, particularly reinforcement learn- ing (RL), offer a powerful paradigm for acquiring such adaptive behaviors [2]. However, a fundamental obsta- cle to this long-term autonomy is hardware degradation, such as the severe wheel damage observed on the Mars Curiosity rover [3]. Phenomena such as thruster failures, actuator malfunctions, or kinematic misalignments can in- troduce significant morphological discrepancies that limit the efficacy of pre-trained control policies. When an agent encounters such severe out-of-distribution changes, zero- shot transfer would likely fail and retraining from scratch ∗ Corresponding author. E-Mail: andrej.orsula@uni.lu Figure 1:Our continual learning framework enables reward-free adaptation to severe changes in transition dynamics by leveraging the latent reward landscape encoded within a pre-trained world model. We demonstrate this approach across three distinct space robotics domains, where the agent recovers from severe failures. is prohibitively expensive in terms of time and communi- cation bandwidth. Continual RL provides a compelling alternative by al- lowing an agent never to stop learning and adapting its policy during deployment [4]. Yet, the primary barrier to applying continual RL in this domain is its reliance on accurate reward signals [5]. In simulation, computing an arbitrarily complex reward function is trivial via access to the full and perfect state. In contrast, physical robots in space cannot rely on such privileged information due to the absence of external tracking systems, limited sensing capabilities, and the unpredictable nature of the environ- ment. Even with elaborate motion-capture systems within terrestrial analogue facilities, estimating rewards might be practically infeasible for many critical tasks. For instance, the rewards and penalties for autonomous regolith excava- tion can be computed in simulation by directly analyzing the positions and velocities across millions of discrete par- ticles [6]. However, accurately estimating the volume of excavated regolith and ejected dust in a real-world lunar en- vironment would be rather challenging and unreliable [7]. This fundamental challenge of unobservable rewards has been a critical bottleneck to deploying adaptive learning systems across most real-world domains, particularly in space robotics, where the stakes of failure are high, and opportunities for human intervention are limited. Recent advancements in model-based RL, particularly latent-state world models like DreamerV3 [8], have demon- strated significant potential for sample-efficient robot learn- ing. Concurrently, studies have shown that world model dynamics can be pre-trained using offline datasets and fine-tuned on real-world data [9]. Despite these develop- ments, most existing approaches continue to rely on either manual or meticulously engineered sparse reward signals while primarily addressing domain shifts. Our focus lies in 3 rd Conference on AI in and for Space (SPAICE 2026) ©2026 Orsula et al., licenced via C BY 4.0. arXiv:2608.23452v1 [cs.RO] 24 Aug 2026 addressing the challenge of unobservable rewards during continual adaptation to severe changes in the underlying transition dynamics. By leveraging the latent representa- tion learned during pre-training in diverse simulations, we hypothesize that world models inherently encode a robust reward landscape within their latent space. Subsequent online adaptation can then be achieved by updating only the transition dynamics of the world model through unsu- pervised rollouts while freezing the observation encoder and reward predictor. In parallel, the active policy can be updated entirely on synthetic trajectories generated by the updated world model, which allows the agent to adapt to changing dynamics without receiving new rewards. This work introduces a framework for reward-free con- tinual adaptation using world models. We demonstrate the efficacy of our approach across three distinct space robotics domains: planetary traversal, orbital navigation, and precision assembly. In each case, we simulate severe morphological failures in the form of actuator malfunctions or kinematic misalignments to introduce significant dis- crepancies between the pre-trained and degraded dynam- ics. Under time and interaction constraints, we evaluate the performance of our agent against a zero-shot baseline, an agent with privileged reward information, and one re- trained from scratch. This work represents a step towards enabling truly resilient autonomy for extreme environ- ments, where robots can adapt to unforeseen challenges without human intervention. 2 Methodology Our framework for reward-free continual adaptation builds upon the DreamerV3 architecture [8], which leverages a latent-state world model to compress high-dimensional observations into compact representations. During the pre-training phase, we follow the standard procedure of jointly optimizing the world model on observed trajectories while simultaneously training an actor-critic policy entirely within the synthesized rollouts of the world model. The core scientific exploration of our methodology lies in the online adaptation phase. As illustrated in Figure 2, we strictly update only the transition dynamics of the world model while freezing the encoder and other heads, includ- ing the reward predictor. With this minimal change, the agent can leverage the latent reward landscape learned during pre-training to guide its adaptation to the new en- vironment dynamics. The workflow is therefore divided into two distinct phases: world model pre-training and reward-free adaptation. 2.1 World Model Pre-Training During simulation pre-training, the world model learns to encode high-dimensional observationsx t into com- pact stochastic latent statesz t . The Recurrent State-Space dynamics h t-1 z t-1 encoder predictors x t-1 a t-1 decoder x̂ t-1 ĉ t-1 r ̂ t-1 h t z t encoder predictors x t decoder x̂ t ĉ t r ̂ t dynamics h t-1 z t-1 encoder predictors x t-1 a t-1 decoder x̂ t-1 ĉ t-1 r ̂ t-1 h t z t encoder predictors x t decoder x̂ t ĉ t r ̂ t World Model Pre-Training Reward-Free Adaptation Figure 2:The world model is pre-trained in simulation. Once deployed, only the transition dynamics are updated via rollouts. Model (RSSM) of DreamerV3 is defined by the following core components [8]: Sequence Model:h t =f φ (h t−1 ,z t−1 ,a t−1 )(1) Forward Dynamics:ˆz t ∼p φ (ˆz t |h t )(2) Encoder:z t ∼q φ (z t |h t ,x t )(3) Decoder:ˆx t ∼p φ (ˆx t |h t ,z t )(4) Reward Predictor:ˆr t ∼p φ (ˆr t |h t ,z t )(5) Continuity Predictor:ˆc t ∼p φ (ˆc t |h t ,z t )(6) The model is trained jointly to reconstruct observationsˆx t , predict rewardsˆr t and episode continuityˆc t , while mini- mizing the Kullback-Leibler (KL) divergence between the prior dynamics (Equation (2)) and the posterior represen- tations (Equation (3)). Concurrently, an actor-critic pol- icyπ(a t |h t ,ˆz t )is trained entirely within latent trajecto- ries generated by the RSSM. The critic is optimized using the predicted rewards from the reward head, which encour- ages the world model to learn a latent representation that captures the underlying reward structure of the task. 3 rd Conference on AI in and for Space (SPAICE 2026) ©2026 Orsula et al., licenced via C BY 4.0. To ensure that the learned latent representations and reward mappings are robust across the vast distribution of possible states, we employ domain randomization dur- ing this pre-training [10]. This encourages the encoder to learn a highly generalizable mapping into the latent space and ensures the reward head accurately reflects the true objective function that remains invariant to perturbations. 2.2 Reward-Free Adaptation Upon deployment, it is anticipated that any robot will even- tually encounter novel environmental conditions and hard- ware degradation that significantly alter the underlying transition dynamics. As the actor-critic policy was trained entirely within imagination, its behavior inherently cap- tures the transition dynamics encoded in the original world model. Therefore, any significant divergence in the true dynamics of the new environment would lead to a catas- trophic failure of the zero-shot policy as the sampled ac- tions would no longer yield the expected outcomes. Online adaptation is therefore critical to recover per- formance. We address the key challenge of unobservable rewards by leveraging the fact that the reward predictor (Equation (5)) was pre-trained to capture the underlying re- ward structure within the latent space of the world model. By freezing this head during adaptation, we expect the actor-critic to continue receiving meaningful reward sig- nals based on its updated latent representations, even as the transition dynamics change. This allows the agent to adapt its policy purely through unsupervised rollouts that update the transition dynamics while maintaining a consistent reward landscape. In addition to freezing the reward predictor, we also freeze the encoder (Equation (3)) and decoder (Equation (4)) to preserve the integrity of the latent representations. By updating only the sequence model (Equation (1)) and the forward dynamics (Equation (2)), we can ensure that the agent learns a corrective delta to the transition dynamics without degrading the foundational world model. We opti- mize the transition dynamics using the KL divergence loss between the posterior and prior distributions on the new proprioceptive transitions: L dyn . = KL(q φ (z t |h t ,x t )∥p φ (ˆz t |h t ))(7) To mitigate catastrophic forgetting, we reduce the learn- ing rate of the world model by an order of magnitude (4×10 −5 →4×10 −6 ). Furthermore, we inject a small Gaus- sian exploration noiseN(0.0,0.02)into the normalized output actions during online rollout collection to ensure the RSSM experiences novel transitions under the new physical constraints. The actor-critic is then repeatedly re-trained in imagination using the newly updated dynamics with a training ratio of2048policy updates per environment step. For fixed-horizon tasks, the continuity predictor (Equa- tion (6)) can either remain frozen or be updated to reflect new episode termination conditions. We freeze it since termination conditions remain unchanged across our pre- training and deployment environments. However, if degra- dation significantly alters the episode structure, updating this predictor may be necessary. 3 Experimental Results We conduct a series of simulation-only experiments to demonstrate key aspects of our adaptation framework. 3.1 Task Formulation and Failure Modes As conceptualized in Figure 1, we designed three distinct tasks across planetary and orbital domains, each paired with an unmodeled morphological failure that induces a significant shift in the underlying transition dynamics. All tasks are implemented on top of NVIDIA Isaac Lab [11] within the scope of the Space Robotics Bench [12]. Planetary Traversal (Wheel Failure) [25 Hz]Traver- sal across complex terrain with an experimental rover in- spired by NASA JPL’s ERNEST prototype [13]. Our 12- actuator rover features a double-sided active gimbal suspen- sion system for maintaining ground contact and stability across extreme terrain. The rover is tasked with traversing a procedurally generated obstacle course while maintain- ing a target velocity vector. A failure mode is simulated by locking the steering and drive joint of the front-right wheel, which introduces significant asymmetric drag. Orbital Navigation (Thruster Failure) [10 Hz]Dy- namic waypoint navigation inSE(3)under micrograv- ity using a spacecraft with 12 independent continuous thrusters. The agent must track a moving target pose while compensating for the dynamics of spaceflight. We simulate the complete failure of three co-located off-axis thrusters, which results in a shift of the transition dynamics. Screwdriving Assembly (Tool Misalignment) [50 Hz] Assembly of a pre-aligned bolt into a matching nut using a 7-DOF robotic manipulator that is equipped with an elec- tric screwdriver end-effector. The task requires precise alignment and insertion under tight tolerances. For the failure mode, we introduce a 15°axial bend in the mount- ing flange of the screwdriver, which causes the tool tip to deviate from the kinematic expectations. 3.2 Experimental Protocol Although the evaluation is performed solely inside a sim- ulation, we enforce a strict separation between the data- abundant simulation phase and the resource-constrained adaptation phase. During pre-training, the model-based agent is trained for20million environment steps, with a training ratio of32policy updates per step across512 parallel environment workers while randomizing physi- cal parameters, such as the gravity vector, inertial proper- ties, friction coefficients, and random external disturbances. 3 rd Conference on AI in and for Space (SPAICE 2026) ©2026 Orsula et al., licenced via C BY 4.0. Figure 3:Performance curves for the three tasks over a 60-minute online adaptation window. The adaptive agent with access to privileged rewards reaches performance levels close to the agent retrained from scratch. Our reward-free agent also rapidly recovers performance, albeit to a lower degree and with decay. The zero-shot agent fails catastrophically due to the unmodeled dynamics shift. This diverse parallelization ensures the world model ac- quires a general latent representation of both the physical dynamics and the underlying reward landscape. In contrast, the subsequent online adaptation phase is strictly constrained to a single simulated agent environ- ment. This constraint is crucial for accurately mimicking the severe data-collection bottleneck of a physical robot de- ployed in an isolated extraterrestrial environment. Further- more, we limit the adaptation window to exactly60min- utes of interaction time. Depending on the control fre- quency of the respective robotic platforms, this translates to90K steps for planetary traversal,36K steps for orbital navigation, and180K steps for screwdriving assembly. Throughout both phases, the agent maintains access to proprioceptive observations alongside the state for rela- tive target tracking (planetary traversal and orbital nav- igation) or the pose of the bolt (screwdriving assembly). Furthermore, the action space remains unchanged even af- ter morphological degradation that renders certain actions obsolete due to actuator failure. For each task, the adap- tation phase is repeated across three random seeds with identical pre-trained models to evaluate the robustness. 3.3 Performance Evaluation We track normalized task-specific progress metrics and evaluate four agents:a)a zero-shot baseline that evaluates the pre-trained policy;b)an agent retrained from scratch on the new dynamics to serve as an asymptotic upper bound;c)an adaptive agent with access to privileged re- wards; andd)our reward-free adaptive agent. Aside from the zero-shot baseline, our agent is the only adaptive base- line that lacks access to the true reward signal and relies entirely on the latent reward landscape encoded within the world model, making it viable for real-world applications. As illustrated in Figure 3, the zero-shot policies fail catas- trophically across all domains due to the unmodeled dy- namics shift. While the retrained baseline confirms that the tasks are solvable under hardware degradation, it high- lights the extreme sample inefficiency of learning from scratch. The privileged agent demonstrates rapid recovery, achieving performance nearing the retrained upper bound. Conversely, our reward-free agent also shows promising initial recovery, demonstrating that latent world models can guide adaptation without external rewards. However, the learning profiles reveal clear limitations. In all three domains, our agent consistently underperforms compared to the privileged baseline. Notably, after an initial perfor- mance gain, the agent exhibits significant volatility and decay, particularly in the orbital and assembly tasks. These results suggest that while the latent reward landscape is suf- ficient for short-term policy recovery, it lacks the long-term stability of explicit rewards, as updating the transition dy- namics on degraded morphologies likely causes the RSSM representation to drift away from the original latent space. 4 Discussion and Conclusion The experimental results validate our central hypothesis that latent-state world models pre-trained across diverse simulations encode a reward landscape that can guide on- line adaptation without observing new rewards. By isolat- ing the dynamics update from the reward prediction, this framework resolves a critical bottleneck in space robotics, where computing online rewards is often impossible. However, the late-stage decay exposes a capacity limit within the RSSM. Because the reward head remains frozen, continuously updating the core transition dynamics on de- graded morphologies eventually overwrites the generalized principles acquired during pre-training. Despite its effectiveness, our framework highlights clear limitations. To prevent late-stage decay, future work will in- vestigate localized latent-space adapters that strictly bound dynamics updates. Notably, our simulation-only study by- passes the peculiarities of the sim-to-real gap [14]. Further- more, the demanding optimization of the online adapta- tion phase heavily exceeds the strict power constraints of space-grade embedded compute modules [15], necessitat- ing future breakthroughs in efficient in-situ learning. 3 rd Conference on AI in and for Space (SPAICE 2026) ©2026 Orsula et al., licenced via C BY 4.0. References 1.Doyle, R.et al.Recent Research and Development Activities on Space Robotics and AI.Advanced Robotics35,1244–1264 (2021). 2.Sutton, R. S. & Barto, A. G.Reinforcement Learning: An Introduction (A Bradford Book, 2018). 3.Rankin, A., Patel, N., Graser, E., Wang, J.-K. F. & Rink, K.Assessing Mars Curiosity Rover Wheel DamageinIEEE Aerospace Conference (2022), 1–19. 4.Abel, D.et al. A Definition of Continual Reinforcement Learningin Proceedings of the 37th International Conference on Neural Informa- tion Processing Systems(Curran Associates Inc., 2023). 5. Silver, D., Singh, S., Precup, D. & Sutton, R. S. Reward is enough. Artificial Intelligence299,103535.issn: 0004-3702 (2021). 6. Orsula, A., Geist, M., Olivares-Mendez, M. & Martinez, C.Learning Tool-Aware Adaptive Compliant Control for Autonomous Regolith ExcavationinSymposium on Advanced Space Technologies in Robotics and Automation(2025). 7. Cloud, J. M.et al. Towards autonomous lunar resource excavation via deep reinforcement learninginASCEND(2021), 4217. 8. Hafner, D., Pasukonis, J., Ba, J. & Lillicrap, T. Mastering Diverse Control Tasks through World Models.Nature640,647–653 (2025). 9.Feng, Y., Hansen, N., Xiong, Z., Rajagopalan, C. & Wang, X.Fine- tuning Offline World Models in the Real WorldinProceedings of the Conference on Robot Learning(2023). 10.Tobin, J.et al. Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real WorldinIEEE/RSJ International Conference on Intelligent Robots and Systems(2017), 23–30. 11.Mittal, M.et al.Isaac Lab: A GPU-Accelerated Simulation Framework for Multi-Modal Robot Learning.arXiv preprint arXiv:2511.04831(2025). 12.Orsula, A., Geist, M., Olivares-Mendez, M. & Martinez, C. Space Robotics Bench: Robot Learning Beyond Earth.arXiv preprint arXiv:2509.23328(2025). 13. Nguyen, M.et al. Trajectory Optimization Methods for Energy Ef- ficient Gait Transitions on Multi-Modal RobotsinIEEE Aerospace Conference(2024), 1–9. 14.Salvato, E., Fenu, G., Medvet, E. & Pellegrino, F. A. Crossing the Reality Gap: A Survey on Sim-to-Real Transferability of Robot Con- trollers in Reinforcement Learning.IEEE Access9,153171–153187 (2021). 15.Felix, M. A., Slater, W. S., Landauer, D. C., Pinson, R. E. & Rutherford, B. B.Total Ionizing Dose Radiation Testing of NVIDIA Jetson Orin NX System on ModuleinIEEE Space Computing Conference(2024), 116–121. 3 rd Conference on AI in and for Space (SPAICE 2026) ©2026 Orsula et al., licenced via C BY 4.0.