Paper deep dive
What Matters for Simulation to Online Reinforcement Learning on Real Robots
Yarden As, Dhruva Tirumala, René Zurbrügg, Chenhao Li, Stelian Coros, Andreas Krause, Markus Wulfmeier
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 7/20/2026, 3:44:11 PM
Summary
This paper investigates design choices for successful online reinforcement learning (RL) on physical robots, specifically focusing on the 'sim-to-online' transfer setting. Through 100+ real-world training runs on three robotic platforms (Franka Emika Panda, Unitree Go1, and a Race Car), the authors identify that retaining simulation data, delaying critic updates, and scaling update-to-data ratios are critical for stabilizing learning and preventing the 'downward spiral' of policy degradation. The work provides an open-source pipeline and empirical evidence that robust, standard RL practices can significantly lower the engineering effort for deploying online RL on real hardware.
Entities (8)
Relation Signals (5)
Soft Actor-Critic → isusedon → Franka Emika Panda
confidence 95% · We use Soft Actor-Critic [SAC,8] ... for the Franka Emika Panda robot.
Soft Actor-Critic → isusedon → Unitree Go1
confidence 95% · We use Soft Actor-Critic [SAC,8] ... for the Unitree Go1.
Data Retention → mitigates → Downward Spiral
confidence 90% · This motivates retaining D0 as a stabilizing prior... The downward spiral occurs when finetuning...
Asymmetric Updates → stabilizes → Soft Actor-Critic
confidence 90% · This idea... helps stabilizing learning, especially high UTD regimes.
MuJoCo Playground → supportspretrainingfor → Franka Emika Panda
confidence 85% · This pipeline can be used to pretrain in simulation any robot in MuJuCo Playground
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We investigate what specific design choices enable successful online reinforcement learning (RL) on physical robots. Across 100 real-world training runs on three distinct robotic platforms, we systematically ablate algorithmic, systems, and experimental decisions that are typically left implicit in prior work. We find that some widely used defaults can be harmful, while a set of robust, readily adopted design choices within standard RL practice yield stable learning across tasks and hardware. These results provide the first large-sample empirical study of such design choices, enabling practitioners to deploy online RL with lower engineering effort.
Tags
Links
- Source: https://arxiv.org/abs/2602.20220v1
- Canonical: https://arxiv.org/abs/2602.20220v1
Trouble viewing inline? Open PDF directly →
Full Text
59,942 characters extracted from source content.
Expand or collapse full text
What Matters for Sim-to-Online Reinforcement Learning on Real Robots Yarden As 1 , Dhruva Tirumala 2 , René Zurbrüg 1 , Chenhao Li 1 , Stelian Coros 1 , Andreas Krause 1 and Markus Wulfmeier 2 1 ETH Zurich, 2 Google DeepMind Abstract: We investigate what specific design choices enable successful online reinforcement learning (RL) on physical robots. Across 100+ real-world training runs on three distinct robotic platforms, we systematically ablate algorithmic, systems, and experimental decisions that are typically left implicit in prior work. We find that some widely used defaults can be harmful, while a set of robust, readily adopted design choices within standard RL practice yield stable learning across tasks and hardware. These results provide the first large-sample empirical study of such design choices, enabling practitioners to deploy online RL with lower engineering effort. 1. Introduction We typically imagine learning agents as those capable of learning continuously and adapt- ing “on-the-fly” from information as it arrives [1,2]. Reinforcement learning (RL) formalizes how such agents can learn online to act optimally within unknown dynamical systems. Despite the demonstrated success of RL in robotics [3,4,5,6], in most existing systems, learning occurs entirely offline, within a simulator or using a fixed dataset of demonstrations, leaving the canonical online learning setting far from being the standard practice. While this approach achieves decent success, without online adaptation, the resulting performance is ultimately limited by the quality of our priors. What are the limits of relying on offline learning while avoiding firsthand real-world experi- ence? Simulators are inevitably imperfect and the cost of obtaining high-quality, pre-training real-world data for robotics is orders of magnitude higher than in domains like language modeling—we do not have an Internet-scale corpus of real-world robotics data to scrape. This work is motivated by the realization that as tasks grow in complexity, future autonomous robotic systems must learn online, through embodied interaction, to continually adapt and improve competence in the open world [7]. While running RL on real robots has been demonstrated in several works (e.g. Haarnoja et al.[8], see Section 2), much of this literature focuses on demonstrating an idea within a relatively narrow real-world experimental setup. Additionally, prior work often focus on a less realistic setting, where learning starts from scratch, which can lead to unsafe and wasteful exploration that could instead happen in simulation. Crucially, as we empirically show in this work (see Sections 3 and 5), pretraining in simulation and then finetuning online on the real system can result in instabilities and even lead to unlearning of the simulation-trained policy. The goal of this work is to empirically study the “sim-to-online” setting and demonstrate a working recipe across three real-world robots. Our contribution. •First, we develop and open-source a training pipeline for real-world robots. This pipeline can be used to pretrain in simulation any robot in MuJuCo Playground [9] and then seam- lessly continue online training on real robots. We demonstrate the flexibility of our pipeline on three real-world robots from different robotic tasks, as demonstrated in Figure 1. • In particular, for the Franka Emika Panda—a popular robot in research labs and industry— we open-source the full robotic stack, from hardware interface up to real-world training of vision-based policies. Notably, this setup relies only on simple, “off-the-shelf” hardware, making it easy to reproduce and therefore lowering the entry barrier for future real-world RL research. Corresponding author(s): yardas@ethz.ch, Code publicly available at https://github.com/yardenas/panda-rl-kit arXiv:2602.20220v1 [cs.RO] 23 Feb 2026 What Matters for Sim-to-Online Reinforcement Learning on Real Robots Realsense D455 Camera Pick up cube (a) Franka Emika Panda. 푣 휔 푧 푧 (b) Unitree Go1. Goal Race Car Start position (c) Race Car. Figure 1: Robotic platforms studied in this work. We conduct our experiments on three robotic platforms spanning manipulation, locomotion and navigation robotic tasks. Manipulation. We use a Franka Emika Panda robot to locate, grasp and lift a cube to a goal position. The policy determines the end-effector’s position and the gripper’s opening given grayscale image observations. Locomotion. We use a Unitree Go1 robot to follow joystick commands. The policy maps randomly sampled linear and angular velocity commands (expressed in the robot’s local coordinate frame) to joint position targets. Navigation. Finally, we use a remote- controlled race car that must park at a specified goal position as quickly as possible. This task is particularly challenging due to the system’s high agility, fast control loop (60 Hz) and the difficulty of accurately modeling tire friction and drifting behavior. •Next, we conduct extensive real-world experiments, studying stability challenges when transferring simulation-trained policies to real robots. Based on these experiments, we demonstrate that retaining data across real-world trials, or even simply data that was obtained during simulation, can significantly help in in improving robustness under such distribution shifts. In addition, we show that simply delaying critic updates (akin to Fuji- moto et al.[10]) can further improve stability. These simple mitigation techniques provide actionable guidance that practitioners can directly apply when bringing RL to their robotic tasks, reducing the engineering burden required to deploy RL on real robots. • Finally, we empirically study and demonstrate techniques for effective pretraining of off-policy RL algorithms in massively-parallel simulators. 2. Related Work Early applications of RL on real robots date back to the 1990s (see11and references therein), focusing on relatively simple tasks with discretized, low-dimensional state-action spaces. Scaling RL to more challenging problems using “deep RL” techniques was achieved later by Haarnoja et al.[8], who demonstrate that their algorithm, Soft Actor-Critic (SAC), learns efficiently on real-world robots. Since then, a few works use real-world robotic experiments to showcase the applicability of their algorithm in practice [12,13,14,15,16,17,18]. Many of these works rely on customized hardware or proprietary software, making them difficult to reproduce. Moreover, these works typically emphasize algorithmic innovation, without sys- tematically examining the practical challenges of deploying RL on real-world robotic systems. In contrast, Ibarz et al.[19]present a comprehensive review that encompasses several en- gineering and algorithmic challenges concerning deployment of RL on real robots. Similar to our work, they identify reusing data across trials and robots as key for improving sample efficiency and point it out as a promising direction for future work. However, they do not provide empirical evidence for the degree of sample efficiency improvements that can be gained by this approach. Further, Tirumala et al.[20]show the effectiveness of this idea across a wide range of simulated RL environments. We advance this line of work by demonstrating its 2 What Matters for Sim-to-Online Reinforcement Learning on Real Robots effectiveness on real robots, where success cannot be achieved without high sample efficiency. Closely related to our work, Yin et al.[21]study sim-to-real finetuning and report that SAC suffers from a performance drop upon transfer. They propose to fine-tune policies online based on reward shaping that drives the agent to state-actions in which the observed reward and the simulated one differ. In this work, we study instead how to mitigate such transfer challenges without additional algorithmic complexity. Similarly, Smith et al.[22,23]demon- strate transfer from simulators, but restrict their experiments only to legged-locomotion tasks. We extend these results to a broader set of robots and systematically study how to effectively transfer such policies. 3. Background 3.1. Problem Setting Markov decision processes. We focus on infinite-horizon Markov decision processes [24, MDP], defined by the tuple(풮,풜,푝,푟,훾,휌 0 ), where풮 ⊂ R 푑 풮 and풜 ⊂ R 푑 풜 are continuous state and action spaces, with states푠 푡 ∈풮propagating through time푡 = 0,...,∞according to actions푎 푡 ∈ 풜and unknown stochastic transition dynamics푝(푠 푡+1 |푠 푡 ,푎 푡 ). After each action the agent receives reward푟 푡 : = 푟(푠 푡 ,푎 푡 ), 푟 :풮×풜→ R ≥0 that is used to define the utility with respect to task completion. We consider the class of stationary stochastic policies 휋(푎 푡 |푠 푡 ) ∈ Πthat map states to (distributions over) actions. The goal in MDPs is to find a policy 휋 * that maximizes the accumulated sum of discounted rewards 휋 * ∈ arg max 휋∈Π 퐽 (휋) : = E 휋 [︃ ∞ ∑︁ 푡=0 훾 푡 푟(푠 푡 ,푎 푡 ) ]︃ , 푠 0 ∼ 휌 0 (·), 푎 푡 ∼ 휋(·|푠 푡 ), 푠 푡+1 ∼ 푝(·|푠 푡 ,푎 푡 ), (1) where훾 ∈ [0, 1)is a discounting factor,휌 0 denotes a distribution over initial states and the expectation is with respect to sequences푠 0 ,푎 0 ,푠 1 ,푎 1 ,푠 2 ,...that follow a Markov chain induced by the policy휋, the dynamics푝and the initial state distribution휌 0 . The following standard definitions for the value푉 휋 , action-value푄 휋 and advantage퐴 휋 functions will be useful in subsequent discussions: 푉 휋 (푠) : = E 휋 [︃ ∞ ∑︁ 푡=0 훾 푡 푟(푠 푡 ,푎 푡 ) ⃒ ⃒ ⃒ ⃒ 푠 0 = 푠 ]︃ , 푄 휋 (푠,푎) : = E 휋 [︃ ∞ ∑︁ 푡=0 훾 푡 푟(푠 푡 ,푎 푡 ) ⃒ ⃒ ⃒ ⃒ 푠 0 = 푠,푎 0 = 푎 ]︃ and 퐴 휋 (푠,푎) : = 푄 휋 (푠,푎)− 푉 휋 (푠) with 푎 푡 ∼ 휋(·|푠 푡 ), 푠 푡+1 ∼ 푝(·|푠 푡 ,푎 푡 ). (2) Given full knowledge of the dynamics푝and reward푟, one can solve Equation (1) via (ap- proximate) planning algorithms such as policy or value iteration [24, 25]. However, in most relevant robotics problems, such access to the dynamics or reward is only limited, therefore requiring data-driven approaches like RL to solve Equation (1) in practice. Episodic online learning. We consider learning in finite episodes. In this setting, in each episode푛 = 1,...,푁, the agent executes a policy휋 푛 for푇time steps, after which the robot is manually reset to some state푠 0 ∼ 휌 0 (·). The data of episode푛is collected풟 푛 : = (푠 푡 ,푎 푡 ,푠 푡+1 ,푟 푡 ) 푇−1 푡=0 and aggregated in a “replay buffer”풟 ≤푛 : = ⋃︀ 푛 푛 ′ =0 풟 푛 ′ [26,27]. While data is collected in finite-length episodes, the agent optimizes the infinite-horizon discounted objective in Equation (1). While this setting requires manual human resets, fully autonomous learning from a single trajectory, without manual resets, is still an active area of research, both theoretically and in practice [28, 29, 30, 31] and left for future work. Priors. We study a setting where prior knowledge of that task is available. Good priors are crucial, since learning from scratch on the real system is likely to be highly unsafe and time-consuming. Such prior knowledge typically manifests in practice either as a simulator or as a fixed offline dataset of transitions. In an offline-to-online setting, one has access to a dataset of transitions풟 0 , which can be used in a model-free [32] or a model-based [33,34] 3 What Matters for Sim-to-Online Reinforcement Learning on Real Robots fashion to learn a prior policy. When considering simulators, we assume to have access to a dynamics model푝 0 that differs from the real dynamics푝. Either way, we use푝 0 or풟 0 to extract a prior policy휋 0 . When considering simulators as priors, we shall use풟 0 to denote the data that was generated in simulation to obtain휋 0 . Due to limited data coverage of풟 0 or the sim-to-real gap,휋 0 is likely to perform suboptimally on the real system, and therefore requires additional real-world data to solve the task. 3.2. Online Transfer Sample efficiency. A successful pipeline for many robotic tasks combines massively parallel simulators [9,35] with domain randomization [36] and model-free on-policy methods such as PPO [37]. This approach often yields excellent performance [3], especially in locomotion tasks, where simulators are more accurate. However, simulators often struggle to accurately model contact-rich or vision-based tasks with complex scenes—common in manipulation—thus making real-world adaptation essential. Since online training on real robots is constrained by real-time execution, sample efficiency is critical. Model-free, on-policy methods use only data collected from the current policy and discard previous experience, resulting in limited sample efficiency that restricts their practicality in real world robotic settings. Off-policy learning. In contrast, off-policy algorithms [38,8,39,10,40] retain past data, and can even reuse data from other experiments with suboptimal hyperparameters [20], often leading to an order-of-magnitude improvement in sample efficiency. While it is some- times suggested that off-policy algorithms can be challenging to train effectively in massively parallel simulators [41], in Section 4 and Section A we show that this is nevertheless feasible. Off-policy algorithms operate in an approximate policy iteration scheme. A parameterized action-value function푄 휋 푛 휑 that evaluates the policy after episode푛is learned by iteratively fitting it to an estimate of the (1-step) Bellman backup, minimizing the following loss: ℓ(휑) : = E (푠 푡 ,푎 푡 ,푠 푡+1 ,푟 푡 )∼풟 ≤푛 [︂ 1 2 (︁ 푄 휋 푛 휑 (푠 푡 ,푎 푡 )− 푦 )︁ 2 ]︂ where 푦 = 푟 푡 + 훾 ̄ 푉 휋 푛 (푠 푡+1 ) and ̄ 푉 휋 푛 (푠 푡+1 )≈ ̄ 푄 휋 푛 (푠 푡+1 ,푎 푡+1 ), 푎 푡+1 ∼ 휋 푛 (·|푠 푡+1 ) (3) and ̄ 푄 휋 푛 is a “target network” that tracks previous copies of푄 휋 푛 휑 , typically via Polyak aver- aging [38] of parameters 휑: 휑 target 푘+1 = (1− 휏 )휑 target 푘 + 휏휑 푘 , 푘 = 0,...,퐾,(4) whereby휏 ∈ (0, 1)and푘 ∈ Nis the number of critic updates per episode. In the policy improvement step, a parameterized policy is then extracted from푄 휋 푛 휑 by ascending the action-value function휋 푛+1 ← arg max 휋 E 푠∼풟 ≤푛 ,푎∼휋(·|푠) [︁ 푄 휋 푛 휑 (푠,푎) ]︁ . This can be done by back-propagating gradients through푄 휋 푛 휑 into the policy parameters [38,8,10], or by alterna- tive approaches such as regularized or advantage-weighted updates [39,40]. Each iteration of this process aims to increase the advantage function퐴 휋 (푠,푎), subject to approximation and estimation errors [42,43]. In the following, we examine how such errors can destabilize off-policy learning when it faces distribution shifts. Approximate policy improvement. Kakade and Langford[44]show that after푁greedy policy updates, the cumulative performance gain is lower-bounded by 퐽 (휋 푁 )− 퐽 (휋 0 )≥ 푁−1 ∑︁ 푛=0 E 휋 푛+1 [︃ ∞ ∑︁ 푡=0 훾 푡 퐴 휋 푛 (푠 푡 ,푎 푡 ) ⏟ ⏞ Greedy policy improvement −2훾 푡 |휖(푠 푡 ,푎 푡 )| ⏟ ⏞ Approximation and modeling errors ]︃ (5) with 휋 푛+1 ∈ arg max 휋∈Π E 휋 [︃ ∞ ∑︁ 푡=0 훾 푡 퐴 휋 푛 (푠 푡 ,푎 푡 ) ]︃ ,(6) 4 What Matters for Sim-to-Online Reinforcement Learning on Real Robots Deploy 휋 푛 State-actions with large error|휖(푠,푎)| are added to풟 ≤푛 . Evaluate 휋 푛 Equation (3) is used to update 푄 휋 푛 휑 ; state-actions with large errors are sampled from 풟 ≤푛 , leading to high returns being wrongly assigned in those state-actions [10]. Improve 휋 푛 Policy improvement finds a policy that max- imizes a biased action- value function푄 휋 푛 휑 . Iterate for 푛 = 0,...,푁 − 1 episodes Figure 2: Off-policy algorithms may lose stability due to approximation errors in action-value functions, leading to unlearning of the prior policy 휋 0 during online learning. 255075 Episoden 50 100 150 200 ˆ J ( π n ) Stable Unstable 255075 Episoden 0 20 40 60 80 Action-value error Stable 255075 Episoden 0 20 40 60 80 Unstable Figure 3: Downward spiral on a simulated Race Car robot under a mild dynamics mismatch. Left: Performance during learning using vanilla Soft Actor-Critic (“Unstable”) and our approach (“Stable”). Right: Time-series histograms of the empirical estimation of errors 휖(푠,푎)over the course of online learning. Concretely, denote푄 휋 푛 MC (푠 푡 ,푎 푡 )as the Monte Carlo estimate of the real action value of휋 푛 and푄 휋 푛 휑 (푠 푡 ,푎 푡 )as the learned approximation of it. For each episode푛 = 0,...,푁− 1, we compute휖(푠 푡 ,푎 푡 )≈ 푄 휋 푛 휑 (푠 푡 ,푎 푡 )−푄 휋 푛 MC (푠 푡 ,푎 푡 )∀푠 푡 ,푎 푡 ∈풟 푛 . We compute the histogram of these values after every episode and represent their log counts as intensity in the two right plots. As shown, the learned action-value function푄 휋 푛 휑 overestimates푄 휋 푛 MC (푠 푡 ,푎 푡 )in a large portion of states that are inserted to풟 푛 . In contrast, for the stable run, most of the mass concentrates just slightly above zero, indicating low errors and therefore improved learning stability. where휖(푠,푎)describes errors in푄 휋 푛 휑 due to estimation, function approximation or model mis- match. Intuitively, Equation (5) guarantees improvement only when the policy improvement term is positive and sufficiently large to dominate the error term. Distribution shifts and the “downward spiral”. In the fully-online setting (i.e., without simulator/offline pretraining), the error term in Equation (5) is typically controlled by stan- dard tricks [27,38,8,10]. However, in offline- or sim-to-online settings, distribution shifts are inherent—푝 0 may be inaccurate or휋 푛 visits states that are far off from those that are in풟 0 . The “downward spiral” [14,45,46,47,17] occurs when finetuning simulation- or offline-trained policies with online data. As Figures 2 and 3 show, upon deployment in the real world, data is collected according to휋 0 , following actions that would maximize the objective in Equation (1) under the simulator’s dynamics푝 0 . These action may ultimately lead to(푠,푎)-pairs in which푄 휋 푛 휑 has large errors휖(푠,푎), thereby making the second term in Equa- tion (5) become more dominant in subsequent actor-critic updates. When the sim-to-real gap is large, these errors compound over episodes, potentially outweighing policy improvement in Equation (5) and causing휋 푁 to underperform휋 0 , effectively reversing pretraining [cf.48]. 5 What Matters for Sim-to-Online Reinforcement Learning on Real Robots 4. Stabilizing Learning Under Deployment Shifts We identify three key techniques used to stabilize learning when facing simulation-to-online deployment shift. We present these techniques below. For the rest of this work we focus on Soft Actor-Critic [8] as off-policy algorithm as it delivers competitive results while maintaining overall robustness to hyperparameters. In Section B we present additional experiments with TD3 [10]. Data retention. Equation (3) demonstrates the importance of the distribution from which samples are drawn when updating푄 휋 푛 휑 . If풟 ≤푛 over-represents transitions with large ap- proximation error, updates to푄 휋 푛 휑 become biased, which in turn induces larger changes to 휋 푛+1 and therefore larger distribution shift (over(푠 푡 ,푎 푡 )-pairs) on the robot. In contrast,푄 휋 푛 휑 is originally trained on풟 0 , implying that its approximation errors on풟 0 are (on average) smaller. This motivates retaining풟 0 as a stabilizing prior. Tirumala et al.[20]and Ball et al.[45]implement this via two buffers: the original offline buffer풟 0 and an online buffer 풟 online : =풟 ≤푛 ∖풟 0 . They use these buffers to sample minibatches in Equation (3) according to (푠 푡 ,푎 푡 ,푠 푡+1 ,푟 푡 )∼ (1− 훼)Unif (풟 0 ) + 훼Unif (풟 online ) 훼∈ [0, 1],(7) and show that a balanced mixture (훼 = 0.5) accelerates online learning in offline-to-online learning. In Section 5, we extend this idea by annealing훼 → 1, which is essential for simulation-to-online learning since풟 0 contains transitions from mismatched dynamics 푝 0 . Warm starts. When풟 0 cannot be retained during online learning [17], we instead approxi- mate it by collecting data using the initial policy휋 0 before any updates to푄 휋 푛 휑 or휋 푛 . This warm-start collection is already standard in off-policy RL [cf.8], and Zhou et al.[17]show that it is crucial to mitigate instabilities in offline-to-online RL. Practically, we simply run휋 0 for 푁 * ≪ 푁 trials (without updating) and treat this as the initial buffer. Asymmetric updates. Off-policy algorithms often define the “update-to-data” (UTD) ratio 휂 : = 퐾 /푇, representing the number of gradient updates of both the actor and critic per real- world transition [49,50]. While increasing the UTD휂improves sample efficiency—crucial for fast online learning—it can also amplify approximation errors and overfitting [51]. This can be mitigated by reducing the actor’s learning rate and interleaving its updates every 푘 = 푀, 2푀, 3푀,...,퐾steps where푀 ∈ N. This idea, introduced by Fujimoto et al.[10]and grounded in two-timescale stochastic approximation [52,53,54], helps stabilizing learning, especially high UTD regimes. In Figure 4, we ablate different choices of푀and the actor’s learning rate in a sim-to-sim transfer setup. As shown, increasing푀, which corresponds to updating the actor less frequently, drastically improves learning stability across all robots. Next, in Section 5 we demonstrate this result in real world RL. 5. Experiments Next, we evaluate the effect of these design choices on three real-world robots. Additional results and implementation details are provided in Sections A to E. Setup. Unless specified otherwise, in all the following experiments we use Soft Actor-Critic [SAC,8] together with the BRO architecture for the critic푄 휋 푛 휑 of Nauman et al.[56]. We extend this architecture to vision control via DrQ [57] for the Franka Emika Panda robot. We repeat each experiment with three random seeds and report the mean and standard error across seeds of each episode’s undiscounted accumulated rewards, denoted by ˆ 퐽 (휋). We use 푇 = 250 for the Race Car and Franka Emika Panda and 푇 = 1000 for the Unitree Go1. 5.1. Learning 휋 0 in Simulation Scaling Soft Actor-Critic. Popular implementations of SAC [e.g.58,59] typically perform a single actor-critic update per parallelized environment step, even when collecting data in from푁 푒 parallel environments. This effectively reduces the update-to-data ratio휂, since the number of updates per environment step decreases as푁 푒 grows. When푁 푒 is small (e.g., 6 What Matters for Sim-to-Online Reinforcement Learning on Real Robots 0100200 6 8 10 ˆ J ( π n ) Franka Emika Panda 050100 100 150 Race Car 050100 10 20 Unitree G1 0100 25 26 27 28 Unitree Go1 0100200 Episoden 6 8 10 12 ˆ J ( π n ) 050100 Episoden 50 100 150 050100 Episoden 0 10 20 0100 Episoden 15 20 25 1 . 0 × 10 − 5 3 . 0 × 10 − 4 M= 20M= 10M= 5M= 1 Figure 4: Learning curves Soft Actor-Critic under mismatch in the dynamics. In the Franka Emika Panda robot, we replace the cube with a soft red ball. We first pretrain on a semi- kinematic bicycle model and finetune on more realistic dynamics that account for tire friction [see55]. In the Unitree G1 [humanoid, see9] and Go1 robots, we reduce the ground friction. We ablate푀 ∈20, 10, 5, 1, showing significant stability improvements as we increase푀 and reduce learning rate from3× 10 −4 → 1× 10 −5 across all robots. In Section B we show similar results using TD3 of [10]. 푁 푒 ∼ 10), this mismatch is minor because휂is usually chosen to be of the same order. However, as푁 푒 scales to thousands, failing to adjust휂leads to “undertraining” relative to the available data. Notably, as we show in Section A, 푁 푒 ∼ 1000 is crucial for robust transfer. We find that key to scaling SAC effectively in massively parallel simulation is to increase휂 proportionally to푁 푒 . In principle, one could set휂 ≈ 푁 푒 to fully match the data generation rate; in practice, however, we observe diminishing returns beyond a moderate UTD휂, with longer training times but little improvement in performance. To quantify this tradeoff, we conduct simulated experiments on the Franka Emika Panda and Unitree Go1 tasks, using 푁 푒 = 512and푁 푒 = 8192, respectively, and sweeping휂 ∈4, 8, 16, 32, 48, 64, 96, 128. Each configuration is trained for an increasing number of total environment transitions across five random seeds. As shown in Figure 5, simulation performance improves steadily with increasing휂up to a task-dependent saturation point, beyond which the gains flatten while wall-clock training time increases substantially. Franka Emika Panda Unitree Go1 Race Car ×0.0 ×0.5 ×1.0 Sim-to-real gap RealSimulation Figure 6: Normalized perfor- mance w.r.t. the best experiment for each task across all experi- ments. In all tasks the prior policy suffers performance decrease due to imperfect simulation dynamics. Sim-to-real gap. We train the prior policy휋 0 for the Franka Emika Panda and Unitree Go1 using Mujoco Play- ground [9]. The Race Car dynamics are implemented following the first-principles model of Kabzan et al.[55]. Our implementation builds on Brax [58], which acceler- ates (simulated) training by parallelizing data collection across thousands of domain-randomized environments. (i) In the Franka Emika Panda setup (cf. Figure 1), we randomize the camera perspective, illumination, and field of view to improve robustness to visual variations (see Section C). This allows휋 0 to successfully detect and approach the cube, but it often fails to grasp and lift it on the real robot. These failures are primarily due to unmodeled contact dynamics between the gripper and the cube, as well as discrepancies between rendered and real visual observations. (i) For the quadruped, we train 7 What Matters for Sim-to-Online Reinforcement Learning on Real Robots 481632486496128 η 0.1 0.2 0.5 1.0 1.5 2.0 3.0 4.0 # Transitions ×10 8 10 15 20 25 30 ˆ J ( π ) 481632486496128 η 0.1 0.2 0.5 1.0 1.5 2.0 3.0 4.0 # Transitions ×10 8 0.50 0.75 1.00 1.25 Runtime [Hours] 481632486496128 η 0.1 0.2 0.3 0.5 0.7 0.8 0.9 1.0 # Transitions ×10 7 2.5 5.0 7.5 10.0 ˆ J ( π ) 481632486496128 η 0.1 0.2 0.3 0.5 0.7 0.8 0.9 1.0 # Transitions ×10 7 0.5 1.0 1.5 Runtime [Hours] Figure 5: Comparison of learning performance and runtime for different configurations. Top: Unitree Go1 experiments. Bottom: Franka Emika Panda experiments. Increasing UTD requires less environment steps however at the price of longer training time. a constrained prior policy by limiting the range of commanded linear and angular velocities (푣 and휔 푧 in Figure 1b) during simulation. While policies trained purely in simulation can already yield stable locomotion [see9], limiting휋 0 during pretraining allows us to demonstrate how online learning can be efficiently used in locomotion tasks. (i) In the Race Car environment, we sample the motor parameters, tire friction and car mass to improve sim-to-real transfer. The resulting policy휋 0 successfully parks the car in simulation, however on the real system often overshoots it. As shown in Figure 6, the prior policies succeed in solving all the simulated tasks, while exhibit limited performance when transferring to the real system. 5.2. Real-World Results Recycling data accelerates learning. We study the impact of retaining data from previous experiments on learning performance. To show that, each experiment is composed of four trials that only share the same random seed. We run each experiment for three seeds for each robot. In the zeroth trial, learning is done only with online data collected in풟 online , which Time 휋 0 휋 푁 Figure 7: After roughly ten minutes of training, including additional hardware overhead (e.g. resetting the robot and data transmission over the internet), SAC recovers a policy with almost perfect success rate for the Franka Emika Panda robot. 8 What Matters for Sim-to-Online Reinforcement Learning on Real Robots 01020 Episoden 6 8 10 12 ˆ J ( π n ) Pick up Fail ↑data Franka Emika Panda 0102030 Episoden 0 10 20 30 Walk Fall Unitree Go1 02040 Episoden 0 50 100 150 200 Park Miss Race Car Trial 0Trial 1Trial 2Trial 3 Figure 8: Performance and robustness increases as we accumulate more transitions over trials. Reusing training data across experiments accelerates online learning in all robots. 02040 Episoden 5.0 7.5 10.0 12.5 ˆ J ( π n ) N ∗ Franka Emika Panda 51525 Episoden 0 20 N ∗ Unitree Go1 52040 Episoden 0 100 200 N ∗ Race Car No WarmstartWarmstartπ 0 Figure 9: The blue point marks the episode푁 * until which we do not update휋 0 under the “warmstart” baseline. Prefilling풟 online using휋 0 improves learning stability in the Unitree Go1 and Race Car, while it is not required to obtain strong performance on the Franka Emika Panda. saved at the end of each experiment. In subsequent trials, we load online replay buffers풟 online of previous trials into풟 0 and start a fresh replay buffer풟 online . We then use Equation (7) to mix data between these two replay buffers, starting from훼 = 0.5and gradually increasing it to훼 = 1to reduce dependency on풟 0 . Figure 8 shows the performance increase as more data is retained. In Section B we ablate the initial choice of훼. As shown, across all tasks, significant performance gains can be achieved by retaining data from only few preceding trials. Figure 7 depicts trajectories before and after fine-tuning on the Franka Emika Panda. Warms starts as a proxy for data retention. We evaluate warm starts as an alternative for data retention. To this end, we do not load풟 0 but rather prefill풟 online using a fixed copy of휋 0 for푁 * iterations. For the Franka Emika Panda and Unitree Go1, we collect5000transitions, corresponding to푁 * = 20and푁 * = 5respectively [cf.17]. For the Race Car we use1250 transitions, which correspond to푁 * = 5episodes. The results are presented in Figure 9, where for the Franka Emika Panda robot, learning succeeds even without a warm start. In contrary, for the Unitree Go1 and Race Car robots, performance significantly degrades without it. Asymmetric updates are critical for stability. Finally, we analyze the importance of em- ploying more conservative updates to the actor, interleaving its updates with more frequent critic updates while reducing its learning rate. Specifically, we update the actor every20critic updates while reducing its learning rate (see Section F) and compare this setup to a baseline that updates the actor every critic step, and uses a shared learning rate for the actor and critic. We present our results in Figure 10, showing that for all robots, the baseline fails to improve performance due to training instability, while using asymmetric updates enables efficient transfer. Notably, even when warm-starting풟 online in these experiments, training without asymmetric actor-critic updates remains highly unstable. 9 What Matters for Sim-to-Online Reinforcement Learning on Real Robots 0Warmstart40 Episoden 5 10 ˆ J ( π n ) Pick up Fail Franka Emika Panda Warmstart1525 Episoden 0 10 20 Walk Fall Unitree Go1 Warmstart2040 Episoden 0 100 200 Park Miss Race Car Asymmetric UpdatesBaselineπ 0 Figure 10: We warmstart both runs with the same푁 * , indicated by the vertical dotted line. Asymmetric updates are crucial for effective transfer across all robots, even when풟 online initialized via a warmstart. 6. Conclusion We present a large-scale empirical study of finetuning simulation-trained RL priors directly on hardware across three robotic platforms. Based on these results, we provide guidance to make online RL on hardware more accessible to RL researchers and practitioners. Our experiments show that, despite training instabilities arising from distribution shifts, standard off-policy algorithms require no major modifications, and remain effective to finetune policies within realistic time budgets, including vision-based tasks with sparse rewards. Our results further highlight the opportunity that lies in reusing data to efficiently scale to more complex tasks. While these findings advance the goal of making online RL more practical, they also raise several important research questions: How can we optimally select samples from offline data 풟 0 to improve online sample efficiency? How can data be effectively reused across different tasks? Are there better regularization strategies that enable faster learning? Finally, our work focuses on the semi-automated episodic setting, where human intervention is still required for resets and safety. Developing practical algorithmic solutions that enable fully autonomous learning is a promising direction for future work. 10 What Matters for Sim-to-Online Reinforcement Learning on Real Robots Acknowledgements Y.A. received funding from grant no. 21039 of the Hasler foundation and the ETH AI Center. M.W and D.T participated in an advisory capacity in this project. We thank Manish Prajapat, Manuel Wendl, Dongho Kang and Mert Albaba for their advice on early revisions of this work. Figure 2 is inspired by Figure 12.3 in Hutter et al.[2]. The authors would like to thank Robotics Systems Lab at ETH Zurich for providing access to the Franka Emika Panda robot. References [1]Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. 2015. (Cited on page 1) [2]Marcus Hutter, David Quarel, and Elliot Catt. An Introduction to Universal Artificial Intelligence. Chapman & Hall/CRC Artificial Intelligence and Robotics Series. 2024. (Cited on pages 1 and 11) [3]Jemin Hwangbo, Joonho Lee, Alexey Dosovitskiy, Dario Bellicoso, Vassilios Tsounis, Vladlen Koltun, and Marco Hutter. Learning agile and dynamic motor skills for legged robots. Science Robotics, 2019. (Cited on pages 1, 4, and 16) [4]Bingjie Tang, Michael A Lin, Iretiayo Akinola, Ankur Handa, Gaurav S Sukhatme, Fabio Ramos, Dieter Fox, and Yashraj Narang. Industreal: Transferring contact-rich assembly tasks from simulation to reality. arXiv preprint arXiv:2305.17110, 2023. (Cited on page 1) [5]Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, 2025. (Cited on page 1) [6] Qiayuan Liao, Takara E Truong, Xiaoyu Huang, Guy Tevet, Koushil Sreenath, and C Karen Liu. Beyondmimic: From motion tracking to versatile humanoid control via guided diffusion. arXiv preprint arXiv:2508.08241, 2025. (Cited on page 1) [7]Khurram Javed and Richard S. Sutton. The big world hypothesis and its ramifications for artificial intelligence. In Finding the Frame: An RLC Workshop for Examining Conceptual Frameworks, 2024. (Cited on page 1) [8]Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, 2018. (Cited on pages 1, 2, 4, 5, and 6) [9]Kevin Zakka, Baruch Tabanpour, Qiayuan Liao, Mustafa Haiderbhai, Samuel Holt, Jing Yuan Luo, Arthur Allshire, Erik Frey, Koushil Sreenath, Lueder A Kahrs, et al. Mujoco playground. arXiv preprint arXiv:2502.08844, 2025. (Cited on pages 1, 4, 7, 8, 16, 18, and 19) [10]Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, 2018. (Cited on pages 2, 4, 5, 6, 7, and 17) [11]Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 2013. (Cited on page 2) [12]Sehoon Ha, Peng Xu, Zhenyu Tan, Sergey Levine, and Jie Tan. Learning to walk in the real world with minimal human effort. arXiv preprint arXiv:2002.08550, 2020. (Cited on page 2) [13] Avi Singh, Albert Yu, Jonathan Yang, Jesse Zhang, Aviral Kumar, and Sergey Levine. Cog: Connecting new skills to past experience with offline reinforcement learning. arXiv preprint arXiv:2010.14500, 2020. (Cited on page 2) 11 What Matters for Sim-to-Online Reinforcement Learning on Real Robots [14]Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020. (Cited on pages 2 and 5) [15]Philipp Wu, Alejandro Escontrela, Danijar Hafner, Pieter Abbeel, and Ken Goldberg. Daydreamer: World models for physical robot learning. In Conference on robot learning, 2023. (Cited on page 2) [16]Yunhai Feng, Nicklas Hansen, Ziyan Xiong, Chandramouli Rajagopalan, and Xiaolong Wang. Finetuning offline world models in the real world. In Proceedings of the 7th Conference on Robot Learning, 2023. (Cited on page 2) [17]Zhiyuan Zhou, Andy Peng, Qiyang Li, Sergey Levine, and Aviral Kumar. Efficient online reinforcement learning fine-tuning need not retain offline data. In The Thirteenth International Conference on Learning Representations, 2025. (Cited on pages 2, 5, 6, 9, and 17) [18]Jianlan Luo, Charles Xu, Jeffrey Wu, and Sergey Levine. Precise and dexterous robotic manipulation via human-in-the-loop reinforcement learning. Science Robotics, 2025. (Cited on page 2) [19]Julian Ibarz, Jie Tan, Chelsea Finn, Mrinal Kalakrishnan, Peter Pastor, and Sergey Levine. How to train your robot with deep reinforcement learning: lessons we have learned. The International Journal of Robotics Research, 2021. (Cited on page 2) [20]Dhruva Tirumala, Thomas Lampe, Jose Enrique Chen, Tuomas Haarnoja, Sandy Huang, Guy Lever, Ben Moran, Tim Hertweck, Leonard Hasenclever, Martin Riedmiller, Nicolas Heess, and Markus Wulfmeier. Replay across experiments: A natural extension of off-policy RL. In The Twelfth International Conference on Learning Representations, 2024. (Cited on pages 2, 4, and 6) [21]Patrick Yin, Tyler Westenbroek, Simran Bagaria, Kevin Huang, Ching-An Cheng, Andrey Kolobov, and Abhishek Gupta. Rapidly adapting policies to the real-world via simulation- guided fine-tuning. In International Conference on Learning Representations, 2025. (Cited on page 3) [22]Laura Smith, J Chase Kew, Xue Bin Peng, Sehoon Ha, Jie Tan, and Sergey Levine. Legged robots that keep on learning: Fine-tuning locomotion policies in the real world. In 2022 international conference on robotics and automation, 2022. (Cited on page 3) [23]Laura Smith, Ilya Kostrikov, and Sergey Levine. A walk in the park: Learning to walk in 20 minutes with model-free reinforcement learning. arXiv preprint arXiv:2208.07860, 2022. (Cited on pages 3 and 21) [24] Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014. (Cited on page 3) [25]Dimitri P. Bertsekas. Dynamic Programming and Optimal Control. Athena Scientific, 2000. (Cited on page 3) [26]Long-Ji Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine Learning, 1992. (Cited on page 3) [27]Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013. (Cited on pages 3 and 5) [28] Benjamin Eysenbach, Shixiang Gu, Julian Ibarz, and Sergey Levine. Leave no trace: Learning to reset for safe and autonomous reinforcement learning. In International Conference on Learning Representations, 2018. (Cited on page 3) 12 What Matters for Sim-to-Online Reinforcement Learning on Real Robots [29]Archit Sharma, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Autonomous reinforcement learning via subgoal curricula. In Advances in Neural Infor- mation Processing Systems, 2021. (Cited on page 3) [30]Archit Sharma, Kelvin Xu, Nikhil Sardana, Abhishek Gupta, Karol Hausman, Sergey Levine, and Chelsea Finn. Autonomous reinforcement learning: Formalism and bench- marking. In International Conference on Learning Representations, 2022. (Cited on page 3) [31]Archit Sharma, Rehaan Ahmad, and Chelsea Finn. A state-distribution matching ap- proach to non-episodic reinforcement learning. In International Conference on Machine Learning, 2022. (Cited on page 3) [32]Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning, 2020. (Cited on page 3) [33]Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Interna- tional Conference on Neural Information Processing Systems, 2020. (Cited on page 3) [34]Chenhao Li, Andreas Krause, and Marco Hutter. Offline robotic world model: Learning robotic policies without a physics simulator. arXiv preprint arXiv:2504.16680, 2025. (Cited on page 3) [35]Mayank Mittal, Pascal Roth, James Tigue, Antoine Richard, Octi Zhang, Peter Du, Antonio Serrano-Munoz, Xinjie Yao, René Zurbrüg, Nikita Rudin, et al. Isaac lab: A gpu-accelerated simulation framework for multi-modal robot learning. arXiv preprint arXiv:2511.04831, 2025. (Cited on page 4) [36] Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world, 2017. (Cited on page 4) [37]John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. (Cited on page 4) [38]Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015. (Cited on pages 4 and 5) [39]Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum a posteriori policy optimisation. arXiv preprint arXiv:1806.06920, 2018. (Cited on page 4) [40]Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177, 2019. (Cited on page 4) [41]Antonin Raffin. Getting sac to work on a massive parallel simulator: An rl journey with off-policy algorithms. araffin.github.io, Feb 2025. URLhttps://araffin.github. io/post/sac-massive-sim/. (Cited on pages 4 and 16) [42] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International conference on machine learning, 2014. (Cited on page 4) [43] John Schulman, Xi Chen, and Pieter Abbeel. Equivalence between policy gradients and soft q-learning. arXiv preprint arXiv:1704.06440, 2017. (Cited on page 4) [44] Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Proceedings of the nineteenth international conference on machine learning, 2002. (Cited on page 4) 13 What Matters for Sim-to-Online Reinforcement Learning on Real Robots [45]Philip J Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforce- ment learning with offline data. In International Conference on Machine Learning, 2023. (Cited on pages 5 and 6) [46]Yuda Song, Yifei Zhou, Ayush Sekhari, Drew Bagnell, Akshay Krishnamurthy, and Wen Sun. Hybrid RL: Using both offline and online data can make RL efficient. In The Eleventh International Conference on Learning Representations, 2023. (Cited on page 5) [47]Mitsuhiko Nakamoto, Oier Mees, Aviral Kumar, and Sergey Levine. Steering your generalists: Improving robotic foundation models via value guidance. In 8th Annual Conference on Robot Learning, 2024. (Cited on page 5) [48]Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforce- ment learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020. (Cited on page 5) [49]Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. Advances in Neural Information Processing Systems, 2019. (Cited on page 6) [50] Xinyue Chen, Che Wang, Zijian Zhou, and Keith Ross. Randomized ensembled double q-learning: Learning fast without a model. arXiv preprint arXiv:2101.05982, 2021. (Cited on page 6) [51]Michał Nauman, Michał Bortkiewicz, Piotr Miłoś, Tomasz Trzciński, Mateusz Os- taszewski, and Marek Cygan. Overestimation, overfitting, and plasticity in actor-critic: the bitter lesson of reinforcement learning. In 41st International Conference on Machine Learning, 2024. (Cited on page 6) [52]Vijay R Konda and John N Tsitsiklis. On actor-critic algorithms. SIAM journal on Control and Optimization, 2003. (Cited on page 6) [53]Vijay R Konda and John N Tsitsiklis. Convergence rate of linear two-time-scale stochastic approximation. 2004. (Cited on page 6) [54]Vivek S Borkar. Stochastic approximation: a dynamical systems viewpoint. Springer. (Cited on page 6) [55]Juraj Kabzan, Miguel I Valls, Victor JF Reijgwart, Hubertus FC Hendrikx, Claas Ehmke, Manish Prajapat, Andreas Bühler, Nikhil Gosala, Mehak Gupta, Ramya Sivanesan, et al. Amz driverless: The full autonomous racing system. Journal of Field Robotics, 2020. (Cited on pages 7 and 19) [56]Michal Nauman, Mateusz Ostaszewski, Krzysztof Jankowski, Piotr Miłoś, and Marek Cy- gan. Bigger, regularized, optimistic: scaling for compute and sample efficient continuous control. Advances in neural information processing systems, 2024. (Cited on page 6) [57]Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2107.09645, 2021. (Cited on page 6) [58]C. Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax - a differentiable physics engine for large scale rigid body simulation, 2021. URL http://github.com/google/brax. (Cited on pages 6 and 7) [59] Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and João G.M. Araújo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learn- ing Research, 2022. (Cited on page 6) 14 What Matters for Sim-to-Online Reinforcement Learning on Real Robots [60]Zechu Li, Tao Chen, Zhang-Wei Hong, Anurag Ajay, and Pulkit Agrawal. Parallel푞- learning: Scaling off-policy reinforcement learning under massively parallel simulation. In International Conference on Machine Learning, 2023. (Cited on page 16) [61]Younggyo Seo, Carmelo Sferrazza, Haoran Geng, Michal Nauman, Zhao-Heng Yin, and Pieter Abbeel. Fasttd3: Simple, fast, and capable reinforcement learning for humanoid control. arXiv preprint arXiv:2505.22642, 2025. (Cited on page 16) 15 What Matters for Sim-to-Online Reinforcement Learning on Real Robots Appendix A. Off-Policy Training in Massively-Parallel Simulators Many off-policy algorithms were initially designed for the setting where the agent collects trajectories sequentially on a single environment. Despite that, a major advancement of RL in robotics leverages the ability to rollout thousands of simulated trajectories in parallel to accelerate training. In fact, when combined with domain randomization, this approach is key to robust sim-to-real transfer, especially in locomotion tasks [3]. However, this success primarily relies on on-policy algorithms, thus limiting its applicability to problems that can only be solved without additional online interaction. Off-policy methods, while more sample-efficient, require subtle yet nontrivial modifications to scale effectively in parallel simulation [41]. Although prior works propose specialized algorithmic solutions for this setting [60,61], we show that SAC remains effective with minimal modifications, enabling unified transfer from large-scale simulation to real-world fine-tuning. N e = 8192N e = 128 0 10 20 30 ˆ J ( π ) ×2.64 gap RealSimulation Figure 11: Performance on the real robot after training with 8192 and 128 domain-randomized en- vironments. Sufficient number of domain-randomized environ- ments is key to effective transfer. Scale matters. We show that using too few domain- randomized environments, denoted by푁 푒 , leads to poor transfer to the real robot, even when SAC converges to a seemingly good policy in simulation. To demonstrate this, we train policies for the Unitree Go1 robot in sim- ulation with푁 푒 ∈ 128, 8192. As shown in Figure 11, both configurations achieve similar performance in sim- ulation. However, when deploying the policy trained with푁 푒 = 128, the robot exhibits reduced stability and achieves significantly lower real-world rewards. This result shows that a large number of domain-randomized environments (푁 푒 ∼ 10 3 ) is essential for robust sim-to- real transfer. The implication of this result is that using a setting that is closer to “vanilla” SAC, with푁 푒 ∼ 10, is not sufficient for robust transfer to a real robot. Note that for the sake of this demonstration, we do not limit the commands during pretraining as done in Section 5, meaning that the policy trained with푁 푒 = 8192transfers well without additional online training. B. More Experiments Franka Emika Panda Unitree Go1 Race Car 0.0 0.2 0.4 Zero-shot performance SACPPO Figure 12: Normalized perfor- mance of PPO and SAC w.r.t. the best performance achieved after online learning. Zero-shot performance compared to PPO. Our work focuses mainly on off-policy algorithms due to their im- proved sample efficiency when training online. We vali- date that the drop in zero-shot deployment performance on the real system with SAC is indeed due to the sim- to-real gap and not because of our choice of algorithm. In particular, in Figure 12, we plot the zero-shot perfor- mance of PPO on the same tasks. We focus on PPO as it is frequently used for sim-to-real deployment without additional training. In addition, to make a fair compar- ison, we use policy networks of the same size. For the Franka Emika Panda and Unitree Go1 robots, we use the same hyperparameters described by Zakka et al.[9]. We also note that for all environments, in simulation, SAC and PPO reach the same performance. As shown in Figure 12, in all tasks the zero-shot performance of both SAC and PPO is significantly lower than the downstream performance after additional online learning with SAC. 16 What Matters for Sim-to-Online Reinforcement Learning on Real Robots 0100200 2.5 5.0 7.5 10.0 ˆ J ( π n ) Franka Emika Panda 050100 160 180 200 Race Car 0100 15 20 25 Unitree G1 01020 22 24 26 Unitree Go1 0100200 Episoden 5 10 ˆ J ( π n ) 050100 Episoden 100 150 200 0100 Episoden 0 10 20 01020 Episoden 10 15 20 25 1 . 0 × 10 − 5 3 . 0 × 10 − 4 M= 20M= 10M= 2 Figure 13: Learning curves for transfer (in simulation) of TD3. Similarly to the results in Figure 4, for small values of delay, mismatch in the dynamics can result in performance significant drops. Sim-to-sim with TD3. We provide additional experiments with TD3 [10], a state-of-the-art off-policy RL algorithm. TD3 delays policy updates by default, using푀 = 2is a default hyper-parameter. Below, we repeat our sim-to-sim experiment in Figure 4 but replace SAC with TD3. We present our results in Figure 13, showing that it exhibits similar transfer dynamics to SAC during online learning. Initial mixing훼. We evaluate the choice of the initial value of훼on learning stability and performance. We train with훼 0 ∈0.9, 0.1on the Franka Emika Panda and Race Car robots and report the results in Figure 14. Our experiments show that as long as offline data is used at the onset of training, while online data dominates in later part of training, good performance can be attained. As expected, using online data earlier in training leads to better performance while compromising training stability. 0510152025 Episoden 6 8 10 12 ˆ J ( π n ) Franka Emika Panda 010203040 Episoden 50 100 150 200 Race Car α 0 = 0.1 α 0 = 0.9 0.0 0.5 1.0 α α 0 = 0.1 α 0 = 0.9 α 0 = 0.1α 0 = 0.9 Figure 14: Dashed lines represent the value of훼during learning. Robust transfer under different initial mixing parameter 훼. Retaining simulation data. We investigate the effect of retaining data collected during simulation on stability and efficiency in online learning, and compare our results with the warmstart setting of Zhou et al.[17], which assumes that no offline data can be retained during online learning. Specifically, we load the replay buffer used to train the prior policy 휋 0 in simulation, initialize훼 = 0.5and linearly anneal it to훼 = 1over five episodes, such that only online data is used thereafter. The results, shown in Figure 15, demonstrate that 17 What Matters for Sim-to-Online Reinforcement Learning on Real Robots retaining simulation data substantially improves both learning efficiency and stability. This 010Warmstart3040 Episoden 6 8 10 12 ˆ J ( π n ) Franka Emika Panda Warmstart2040 Episoden 0 50 100 150 200 Race Car Retain DataWarmstartπ 0 Figure 15: Online performance when retaining data used for training in simulation. Simulation data stabilizes learning even without priming the online buffer with 휋 0 . result is expected, since the simulation data acts as a regularizer, biasing minibatches used in the action-value update (Equation (3)) towards samples with lower approximation error. This dampens sharp distribution shifts during learning therefore stabilizing it. As훼gradually increases to1, only real-world data is used in training, ensuring that optimal performance is ultimately obtained on the real robot. C. Franka Emika Panda Task. We build our simulated and real environments based on thePandaPickCubeCartesian task of Zakka et al.[9]. The agent observers a64× 64grayscale image, together with the end-effector position(푥,푦,푧)in the world frame and gripper opening. To emulate photomet- ric variability, image brightness is randomly scaled by a uniformly sampled factor at each episode reset, producing observations that vary in illumination and contrast. The agent acts in Cartesian space through a continuous four-dimensional action vector 푎 = (∆푥, ∆푦, ∆푧,푔), where(∆푥, ∆푦, ∆푧)represent incremental translational displacements of the gripper and푔 controls the opening and closing of the parallel fingers. The reward function in our setup follows the progress-based reward function of Zakka et al.[9], which encourages the agent to make incremental improvements towards the goal within episodes: approach the cube, lift it and move it towards the goal position until reaching it. Success Criterion. An episode is considered successful when the Euclidean distance between the cube and the designated target position falls below a threshold of0.05 m. Episodes terminate early upon success, if the cube falls off the workspace. Domain Randomization. We follow the same domain randomization scheme of Zakka et al. [9], we provide here their details for completeness. The randomized parameters include: • Lighting: Randomized light position, orientation, and whether shadows are cast. • Camera Pose: Small perturbations to camera position and orientation. •Material and Color: Randomization of the floor color (grayscale shades) and geometric material identifiers for scene objects. • Scene Brightness: Multiplicative scaling of rendered image intensity to vary illumina- tion. These variations, shown in Figure 16, are independently sampled for each parallel simulation instance, ensuring that the learned policy encounters a diverse range of visual and geometric conditions during training. 18 What Matters for Sim-to-Online Reinforcement Learning on Real Robots Figure 16: Domain-randomized environments in the simulated Franka Emika Panda robot. D. Unitree Go1 We use theFlatTerrainGo1Joystickenvironment from MuJoCo Playground [9] in our experiments. Our real-world environment matches the simulated task in all aspects. Our main deviation from Zakka et al.[9]is in the linear and angular velocity commands we sample. Specifically, Zakka et al.[9]samples uniformly from the range[±1.5,±0.8,±1.2]during train- ing in simulation, allowing for very good transfer to the real robot since this range of velocities is rather large. We instead train in simulation using the range[±0.5,±0.8,±1.2], which we also use when deploying to real. Since this range is relatively smaller than the one used by Zakka et al.[9], transfer to the real robot is much more challenging because the resulting policy from simulation is less robust when deployed on the real robot. We demonstrate that in Figure 17 where we show a trajectory of the prior policy휋 0 and a trajectory after training with improved stability. E. Race Car Task. We simulate car dynamics following Kabzan et al.[55]. Their model captures the car’s motor and tire dynamics, however due to the sim-to-real gap, modeling car drifts is rather inaccurate and leads to the car overshooting the goal position, shown in Figure 18. At each timestep, the agent observes the full vehicle state (2D positions and velocities) and outputs a 19 What Matters for Sim-to-Online Reinforcement Learning on Real Robots 휋 0 휋 푁 Figure 17: Robustness to commands that were need seen during training in simulation improves over training episodes. 휋 0 휋 푁 Figure 18: The prior policy휋 0 overshoots the goal, obtaining low rewards due to sparsity of rewards in this task. After20trials, the finetuned policy휋 푁 reaches the goal faster and in higher precision. continuous 2D action (steering, throttle). The reward is defined as 푟 푡 (푠 푡 ,푎 푡 ) : = 푑 푡−1 − 푑 푡 + 1[푑 푡 ≤ 휖]− 휆 푐 ‖푎 푡 ‖ 2 − 휆 푙 ‖푎 푡 − 푎 푡−1 ‖ 2 2 , where푑 푡 =‖x 푡 −x goal ‖ 2 denotes the Euclidean distance to the goal and푎 푡 ∈ R 2 is the applied action. The indicator term adds a bonus when within휖 = 0.3meters of the goal.휆 푐 penalizes control effort and 휆 푙 penalizes action changes. F. Implementation Details Hyperparameters. Unless otherwise specified, we use a learning rate of10 −5 for the actor, and update the actor once every 20 critic updates. In addition, we use1250updates per episode for all robots, leading to휂 = 5for the Franka Emika Panda and Race Car robots and휂 ≈ 1for the Unitree Go1. In principle, higher UTD can further improve sample efficiency, however we opt for relatively conservative UTD setting when running SAC on real hardware to maintain sta- ble learning for all robots. All remaining hyperparameters and sweep settings follow the refer- ence open-source implementation in https://github.com/yardenas/panda-rl-kit. Pitfalls. During early development we observed several subtle issues that caused silent failures or large differences in final performance. We list these errors below. 20 What Matters for Sim-to-Online Reinforcement Learning on Real Robots 1.Optimizer state not restored on resume. Restoring only model weights but not the optimizer state (momentum, second-moment estimates, learning-rate schedulers, etc.) changes the optimizer dynamics and can substantially alter learning. 2. Restoring a critic without its target network. Loading푄 휋 푛 휑 but not its target network ̄ 푄 휋 푛 produces inconsistent targets, which effectively lead the critic and thereafter the actor to unlearn. 3. SAC temperature훼(and its optimizer) not restored. Since훼changes during pre- training, failing to restore its value and optimizer state changes the scale of entropy bonus in actor and critic updates, which lead to instability. These issues are easy to overlook but they change learning dynamics and final performance. We explicitly call them out to help practitioners avoid these easy mistakes that can lead to wrong conclusions when finetuning SAC on real-world robots. Synchronous updates. Standard off-policy algorithms are typically implemented such that actor-critic updates occur after every real-world transition. This approach is challenging in practice, since gradient computations are typically slower than real-time control cycles, especially when increasing the UTD. While synchronous updates after each transition are sometimes feasible [e.g.23], they become impractical for high-frequency control or large models. This suggests a batch-like scheme that approaches (iterated) offline RL as푇grows. In this scheme, learning occurs asynchronously and episodically every푇steps, decoupling data collection from optimization. Specific details can be found in our open-source implementation: https://github.com/yardenas/panda-rl-kit. 21