Paper deep dive
Towards Batch-to-Streaming Deep Reinforcement Learning for Continuous Control
Riccardo De Monte, Matteo Cederle, Gian Antonio Susto
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/13/2026, 12:52:35 AM
Summary
The paper introduces two novel streaming deep reinforcement learning algorithms, Streaming Soft Actor-Critic (S2AC) and Streaming Deterministic Actor-Critic (SDAC), designed to enable efficient on-device finetuning and continuous adaptation. By extending batch RL methods (SAC and TD3) to the streaming setting without requiring replay buffers or target networks, these algorithms achieve performance comparable to state-of-the-art streaming baselines while eliminating the need for sensitive hyperparameter tuning.
Entities (6)
Relation Signals (3)
S2AC → extends → SAC
confidence 100% · We propose S2AC and SDAC, two streaming deep RL algorithms that extend SAC and TD3 to the purely online setting
SDAC → extends → TD3
confidence 100% · We propose S2AC and SDAC, two streaming deep RL algorithms that extend SAC and TD3 to the purely online setting
S2AC → usesoptimizer → ObGD
confidence 95% · Critic updates are performed using eligibility traces in conjunction with the ObGD optimizer
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:State-of-the-art deep reinforcement learning (RL) methods have achieved remarkable performance in continuous control tasks, yet their computational complexity is often incompatible with the constraints of resource-limited hardware, due to their reliance on replay buffers, batch updates, and target networks. The emerging paradigm of streaming deep RL addresses this limitation through purely online updates, achieving strong empirical performance on standard benchmarks. In this work, we propose two novel streaming deep RL algorithms, Streaming Soft Actor-Critic (S2AC) and Streaming Deterministic Actor-Critic (SDAC), explicitly designed to be compatible with state-of-the-art batch RL methods, making them particularly suitable for on-device finetuning applications such as Sim2Real transfer. Both algorithms achieve performance comparable to state-of-the-art streaming baselines on standard benchmarks without requiring tedious hyperparameter tuning. Finally, we further investigate the practical challenges of transitioning from batch to streaming learning during finetuning and propose concrete strategies to tackle them.
Tags
Links
- Source: https://arxiv.org/abs/2603.08588v1
- Canonical: https://arxiv.org/abs/2603.08588v1
Trouble viewing inline? Open PDF directly →
Full Text
62,070 characters extracted from source content.
Expand or collapse full text
Cover Page Towards Batch-to-Streaming Deep Reinforcement Learning for Continuous Control Riccardo De Monte, Matteo Cederle, Gian Antonio Susto Keywords: Reinforcement Learning, Deep Learning, Incremental Learning, Real-time Learning, Finetuning. Summary State-of-the-art deep reinforcement learning (RL) methods have achieved remarkable per- formance in continuous control tasks, yet their computational complexity is often incompatible with the constraints of resource-limited hardware, due to their reliance on replay buffers, batch updates, and target networks. The emerging paradigm of streaming deep RL addresses this limitation through purely online updates, achieving strong empirical performance on standard benchmarks. In this work, we propose two novel streaming deep RL algorithms, Stream- ing Soft Actor-Critic (S2AC) and Streaming Deterministic Actor-Critic (SDAC), explicitly de- signed to be compatible with state-of-the-art batch RL methods, making them particularly suit- able for on-device finetuning applications such as Sim2Real transfer. Both algorithms achieve performance comparable to state-of-the-art streaming baselines on standard benchmarks with- out requiring tedious hyperparameter tuning. Finally, we further investigate the practical chal- lenges of transitioning from batch to streaming learning during finetuning and propose concrete strategies to tackle them. Contribution(s) 1. We propose S2AC and SDAC, two streaming deep RL algorithms that extend SAC and TD3 to the purely online setting, achieving performance comparable to the state-of-the-art while requiring no careful hyperparameter tuning. Context: Existing Q-value-based streaming algorithms such as AVG (Vasan et al., 2024) require precise tuning of sensitive hyperparameters such as learning rate and entropy tem- perature, unlike Stream AC(λ) (Elsayed et al., 2024), whereas S2AC and SDAC achieve competitive performance without such sensitivity. 2. We are the first to investigate the practical challenges of transitioning from batch to stream- ing learning, proposing concrete strategies to tackle them and unlocking applications such as Sim2Real finetuning, continuous adaptation on resource-constrained hardware, and dy- namic alternation between batch and streaming regimes under varying computational bud- gets. Context: To the best of our knowledge, no prior work has addressed the design of stream- ing algorithms compatible with state-of-the-art batch RL methods, nor investigated the prac- tical challenges of switching between the two. By building directly on TD3 (Fujimoto et al., 2018) and SAC (Haarnoja et al., 2018), SDAC and S2AC are the first streaming algorithms to tackle this problem, showing promising results toward enabling seamless transitions from these widely adopted batch methods. arXiv:2603.08588v1 [cs.LG] 9 Mar 2026 Towards Batch-to-Streaming Deep Reinforcement Learning for Continuous Control Riccardo De Monte 1† , Matteo Cederle 1,† , Gian Antonio Susto 1 riccardo.demonte,matteo.cederle@phd.unipd.it, gianantonio.susto@unipd.it 1 Department of Information Engineering, University of Padova, Italy † Equal contribution Abstract State-of-the-art deep reinforcement learning (RL) methods have achieved remarkable performance in continuous control tasks, yet their computational complexity is often incompatible with the constraints of resource-limited hardware, due to their reliance on replay buffers, batch updates, and target networks. The emerging paradigm of stream- ing deep RL addresses this limitation through purely online updates, achieving strong empirical performance on standard benchmarks. In this work, we propose two novel streaming deep RL algorithms, Streaming Soft Actor-Critic (S2AC) and Streaming De- terministic Actor-Critic (SDAC), explicitly designed to be compatible with state-of- the-art batch RL methods, making them particularly suitable for on-device finetuning applications such as Sim2Real transfer. Both algorithms achieve performance compa- rable to state-of-the-art streaming baselines on standard benchmarks without requiring tedious hyperparameter tuning. Finally, we further investigate the practical challenges of transitioning from batch to streaming learning during finetuning and propose con- crete strategies to tackle them 1 . 1 Introduction Reinforcement learning (RL) has emerged as a powerful framework for sequential decision-making and optimal control, particularly in settings where system dynamics are partially unknown or dif- ficult to model analytically (Sutton & Barto, 1998). The adoption of neural networks as powerful function approximators has led to the emergence of deep reinforcement learning (DRL), extend- ing the representational capacity of classical reinforcement learning methods. By leveraging high- dimensional nonlinear approximations, DRL has demonstrated the ability to solve complex control tasks that were previously intractable (Mnih et al., 2015; Lillicrap et al., 2020). Recent developments in DRL have enabled strong performance even in highly complex scenar- ios, such as continuous action control, with promising applications in robotics and other high- dimensional dynamical systems (Schulman et al., 2017; Fujimoto et al., 2018; Haarnoja et al., 2018; Fujimoto et al., 2023). In this context, replay buffers, leveraging either off-policy or on-policy data, are employed to mitigate temporal correlations in collected samples and to improve sample effi- ciency, while mini-batch updates contribute to more reliable and statistically stable updates. Such mechanisms contrast with earlier approaches, including classical Q-learning (Watkins & Dayan, 1992) and foundational actor–critic methods (Konda & Tsitsiklis, 1999), which relied primarily on online updates without experience replay or dedicated target network stabilization. 1 Code: https://anonymous.4open.science/r/batch2streamDRL-F16D/README.md 1 While batch updates and replay buffers are critical to the effectiveness of modern DRL ap- proaches, their computational demands make them challenging to implement for on-board learning in resource-constrained systems, such as edge devices. Nevertheless, reinforcement learning holds significant potential for tiny robotics (Neuman et al., 2022), enabling adaptive, data-driven control in compact and resource-limited platforms that have already demonstrated promising applications in autonomous inspection (De Rivaz et al., 2018) and search-and-rescue operations (Duisterhof et al., 2021). Recent works have explored deep streaming reinforcement learning, in which deep RL methods rely solely on online updates, precluding the use of replay buffers, batch updates, or the additional com- putational complexity introduced by one or more target networks (Vasan et al., 2024; Elsayed et al., 2024), making them suitable for on-device training and efficient continual learning. In particular, while Action Value Gradient (AVG), proposed in Vasan et al. (2024), requires a precise tuning of hyperparameters such as learning rate and temperature α of the Maximum Entropy RL formula- tion (Ziebart et al., 2010), Elsayed et al. (2024) present Stream AC(λ), showing strong performance both on MuJoCo Gym (Todorov et al., 2012) and DM Control Suite (Tunyasuvunakool et al., 2020) environments. It is worth noting that, for many practical applications, training a deep RL agent entirely from scratch on-device using streaming algorithms may not be the most realistic goal (Wu et al., 2025). The large number of environment interactions typically required to learn a meaningful policy from random initialization (Elsayed et al., 2024), combined with the erratic exploratory behavior of an untrained agent, could subject the real hardware to considerable mechanical stress and risk of damage. For this reason, we identify three particularly promising application scenarios for streaming deep RL algorithms. The first is finetuning for dealing with the Sim2Real gap (Höfer et al. (2021)): a policy is first trained in simulation using state-of-art batch RL algorithms, and then deployed on the real system where it can continue to adapt online using a streaming algorithm, bridging the gap between simulated and real-world dynamics. This application is particularly relevant in practice, given that the majority of current applications of RL for real-world robotics tasks do not fully capture the practical constraints imposed by on-device deployment, limited computational resources, and the need for continual adaptation in dynamic environments, since they either directly apply the policy trained in simulation to the real world (Chen et al. (2023)), assume that finetuning on-device using batch methods is possible (Yin et al. (2025)), or consider the presence of a remote server in charge of performing all the computations and capable of real-time communication with the edge device (Wang et al. (2022)). The second application scenario for streaming deep RL algorithms is Real2Sim finetuning, which refers to the process of constructing or refining a simulation environment from real-world data, so that the resulting simulator faithfully reproduces the dynamics and appearance of the target physical system (Memmel et al. (2024); Pfaff et al. (2025)). In this context, streaming deep RL algorithms can be used to improve on-device and online the inaccurate initial exploration policy, enabling the possibility of collecting high-quality data for constructing an high fidelity simulator. Finally, an understudied yet compelling scenario arises when, despite the availability of relatively powerful on-device hardware, computational resources cannot be dedicated exclusively to the RL training process. In practice, a robotic system must simultaneously handle a wide range of concur- rent tasks, such as perception, planning, and control, each competing for the same limited compu- tational budget. In such conditions, it may be necessary to dynamically alternate between batch RL algorithms, which offer stronger sample efficiency but impose a heavier computational footprint, and streaming RL algorithms, which trade some statistical efficiency for a significantly lighter resource demand. While Stream AC(λ) demonstrates strong performance on standard robotics benchmarks, it presents a fundamental compatibility issue for our purposes: its design is inherently misaligned with the state-of-art batch RL algorithms commonly used in continuous control, such as TD3 (Fujimoto et al. (2018)) and SAC (Haarnoja et al. (2018)). Since real-world deployment would naturally demand 2 finetuning from a policy pre-trained with one of these methods, this incompatibility represents a significant practical limitation. Although Stream AC(λ) is compatible with PPO (Schulman et al., 2017), this does not offer a viable alternative, as PPO is well known to underperform in robotics and continuous control tasks, a finding further corroborated by Elsayed et al. (2024). In light of this, we propose in this work two novel streaming deep RL algorithms, namely Stream- ing Soft Actor-Critic (S2AC) and Streaming Deterministic Actor-Critic (SDAC), designed with the objective of adapting SAC and TD3 for streaming learning, by taking inspiration from the work of Elsayed et al. (2024). Both algorithms achieve performance comparable to the state-of-the-art Stream AC(λ) on standard benchmarks, while requiring no particular hyperparameter tuning. More- over, we also explore how to perform the finetuning step in practice, highlighting the problems that arise when switching from batch to streaming learning, and proposing a first attempt to address the problem. Finally, we also show that some of the simple modifications to SAC and TD3 necessary to make them compatible with streaming finetuning can also greatly boost their performances as standalone batch RL algorithms, as it had already been similarly observed in Lee et al. (2024). 2 Preliminaries 2.1 Problem Setting and Notation We assume the RL agent operates in a environment formalized as a fully-observable Markov deci- sion process (MDP), defined by the tuple (S,A,R,p,r,d 0 ,γ), whereS denotes the set of all possi- ble states,A denotes the action space, p :S×A×S → [0, +∞) denotes the transition density prob- ability of observing next state s t+1 ∈ S given the current state s t ∈ S and action a t ∈ A, and d 0 denotes the initial state distribution s 0 ∼ d 0 . The environment emits a stochastic reward r t+1 ∈ R associated to the distribution r :S×A×S → ∆(R), and γ ∈ [0, 1) denotes a discount factor used to define the discounted cumulative reward at time t, namely, G t = P +∞ k=0 γ k r t+k+1 . At each time step t the agent samples an action a t accordingly to a policy π(·|s t ) :S → ∆(A), in general stochastic. The value-function for a given policy π and state s is defined as V π (s) = E π [G t |s t = s], while the action-value function for a pair (s,a) ∈ S ×A is defined as Q π (s,a) = E π [G t |s t = s,a t = a]. We denote with π θ (·|s) a parametrized policy by use of a neural network with weights θ ∈ R d . The ultimate goal of any DRL algorithm is to maximize the objective function J (θ) = E s 0 ∼d 0 [V π θ (s 0 )]. Maximum Entropy RL instead augments the reward with the entropy of the policy H(π θ (·|s t )), resulting in different value and action-value function definitions (Ziebart et al., 2010; Haarnoja et al., 2017; 2018). We define the soft value function as V soft π θ (s) = E π θ [ P +∞ k=0 γ k (r t+k+1 + αH(π θ (·|s t+k )))|s t = s], and the soft action-value function Q soft π θ (s,a) = E π θ [r t+1 + γV soft π θ (s t+1 )|s t = s,a t = a], where α ∈ (0, +∞) is a temperature hyper-parameter that bal- ances reward maximization against entropy maximization, controlling the sensitivity of the policy to differences in action-values. 2.2 Stream AC(λ) Elsayed et al. (2024) propose Stream AC(λ), an actor-critic (AC) approach, where a critic network with parameters φ ∈ R m is used to approximate the value function V π θ ,φ (s) ≈ V π θ (s). Rather than minimizing the squared of the one-step Temporal Difference (TD) error δ t = r t+1 + γ · stop-grad(V π θ ,φ (s t+1 ))− V π θ ,φ (s t ), commonly used to perform an on-line update 2 of the critic network, Stream AC(λ) relies on TD(λ) (Sutton & Barto, 1981): defined the eligibility traces z t = γλz t−1 +∇ φ V π θ ,φ (s t ) withz −1 =0 and λ∈ (0, 1), the critic is updated asφ← φ+η V δ t z t , where η V is the learning rate. The use of eligibility traces leads to better credit assignment while retaining the streaming nature of the update. The same philosophy is applied to the actor: whereas standard one-step Actor-Critic uses the estimator δ t ∇ θ logπ θ (a t |s t ) ≈ ∇ θ J (θ) (Sutton et al., 2 While batch approaches store the current sample(s t , a t , s t+1 , r t+1 ) in a buffer and either perform one update given a batch of samples from the buffer itself or post-pone the update once the buffer is full (Schulman et al., 2017), streaming AC performs the update on the current sample. 3 1999), Stream AC(λ) maintains actor traces e t = λγe t−1 +∇ θ logπ θ (a t |s t ) and updates the pol- icy parameters as θ ← θ +η π δ t e t . Additionally, Stream AC(λ) incorporates entropy regularization with eligibility traces. A critical challenge in streaming actor-critic methods is training stability: unlike batch-based algo- rithms, which smooth gradient noise over large mini-batches, streaming updates operate on single transitions, making the optimization landscape considerably noisier. Lyle et al. (2023) have shown that Adam (Kingma & Ba, 2014) can be a source of instability in stationary settings. This motivates Elsayed et al. (2024) to introduce Overshooting-bounded Gradient Descent (ObGD), an optimizer designed to avoid overshooting without adding extra computation as done by backtracking. It is worth noting that ObGD, originally proposed as optimizer for the critic network V π θ ,φ (·), can be applied to any regression problem. Moreover, while ObGD is derived to avoid overshooting, it resembles SGD with Clipping (SGDC) (Zhang et al., 2020; Sun et al., 2025), a variant of SGD designed to ensure the convergence of SGD in heavy-tailed noise. As in supervised learning, data normalization can improve training stability (Andrychowicz et al., 2020; Engstrom et al., 2020). To this end, Stream AC(λ) normalizes states following Andrychowicz et al. (2020) and scales the reward signal as proposed by Engstrom et al. (2020). Specifically, the online algorithm of Welford (1962) is used to track the statistics required for state normalization and reward scaling. Moreover, Elsayed et al. (2024) employ sparse network initialization, and, following the insights of Nauman et al. (2024), both the critic and actor networks incorporate LayerNorm (Ba et al., 2016). 3 Proposed Methods This section introduces two novel algorithms: Streaming Soft Actor-Critic (S2AC) and Streaming Deterministic Actor-Critic (SDAC). We first discuss the architectural and methodological choices shared by both methods, before turning to the algorithm-specific design decisions that were neces- sary to ensure stable and effective learning in practice. As noted by Elsayed et al. (2024), streaming deep reinforcement learning is primarily hindered by three challenges: instabilities arising from oc- casional large updates, activation nonstationarity, and improper data scaling. To address these issues, both S2AC and SDAC adopt sparse network initialization, LayerNorm applied to the pre-activations of each layer, and the observation normalization and reward scaling schemes mentioned in subsec- tion 2.2. While we omit any additional notation to highlight the state normalization, we will denote with σ r the time-varying reward scaling computed as in Engstrom et al. (2020). 3.1 Streaming Soft Actor-Critic (S2AC) We parameterize both the actor, as a stochastic policy π θ (·|s t ), and the critic, as a soft Q-function Q soft π θ ,φ (s t ,a t ) ≈ Q soft π θ (s t ,a t ), where θ ∈ R d and φ ∈ R m denote the weights of their respective neural networks. The critic is trained by minimizing the soft Bellman residual: δ Q soft (s t ,a t ) = Q soft π θ ,φ (s t ,a t )− r(s t ,a t ) σ r +(1) + γE s t+1 ∼p,a t+1 ∼π θ h Q soft π θ ,φ (s t+1 ,a t+1 )− α logπ θ (a t+1 |s t+1 ) i !! 2 , where the transition (s t ,a t ) is collected online under the current policy, and σ r denotes the running standard deviation of the observed rewards, computed following Engstrom et al. (2020). In contrast to standard SAC, and consistent with the streaming nature of our algorithm, we forgo the use of target networks, relying instead on the online estimate of Q soft π θ ,φ when computing the Bellman target. Critic updates are performed using eligibility traces in conjunction with the ObGD optimizer, modifying the original residual described in subsection 2.2 with the soft Bellman residual in Equation 1. 4 For the actor, we consider the policy improvement objective of Haarnoja et al. (2018): J π (s t ;θ) = E a t ∼π θ α log (π θ (a t |s t ))− Q soft π θ ,φ (s t ,a t ) ,(2) which encourages the policy to maximize expected returns while maintaining high entropy. To enable efficient gradient-based optimization, we apply the reparameterization trick, expressing sam- pled actions as a deterministic transformation of noise: a t = f θ (ε t ;s t ), where ε t ∼N (0,I). This yields the tractable objective: J π (s t ;θ) = E ε t ∼N α logπ θ (f θ (ε t ;s t )|s t )− Q soft π θ ,φ (s t ,f θ (ε t ;s t )) .(3) Unlike the critic, the actor is updated without eligibility traces, employing the Adam opti- mizer (Kingma & Ba, 2014), similarly to batch deep RL algorithms. Finally, we recall that the entropy regularization coefficient α acts as a temperature parameter, con- trolling the tradeoff between reward and entropy maximization: higher values encourage more stochastic, exploratory policies, while lower values steer the agent towards more deterministic, reward-focused behavior. An important consequence of reward normalization arises when con- sidering the role of α during training. Since rewards are scaled by the running standard deviation σ r , the effective magnitude of the reward signal fluctuates over time. As a result, a fixed value of α can lead to undesirable behavior: when σ r ≪ 1, the normalized rewards are amplified, diminishing the relative contribution of the entropy term and pushing the policy towards excessive determinism; conversely, when σ r ≫ 1, the reward signal is suppressed, causing the entropy term to dominate and yielding an overly stochastic policy. In either case, the balance between the two objectives, which α is designed to maintain, is inadvertently disrupted by the normalization. To address this issue, we propose scaling α by the same factor used to normalize the rewards, defining a time-varying entropy coefficient: α → α/σ r . This correction ensures that the relative weighting between the reward and entropy terms remains consistent throughout training, regardless of the current reward statistics. The practical impact of this design choice is empirically validated in the ablation study presented in subsection 4.1. For a pseudocode of S2AC, refer to section S1 in the supplementary material. 3.2 Streaming Deterministic Actor-Critic (SDAC) We now consider a deterministic policy π θ (s t ), where θ ∈ R d denotes the parameters of the actor network. Moreover, we approximate the action-value function Q π θ ,φ (s t ,a t )≈ Q π θ (s t ,a t ) by use of a critic network with parameters φ ∈ R m . For exploration purposes, actions are actually given by a t = π θ (s t ) + ε 1 , where ε 1 ∼ N (0,Iσ 2 ). Thus, SDAC is an off-policy actor-critic algorithm with a deterministic policy, making it the first of its kind in the streaming DRL framework. Given the deterministic nature of the policy, we employ the Deterministic Policy Gradient theorem (DPG, Silver et al. (2014)) for the policy update: ∇ θ J π (s t ;θ) =∇ a t Q π θ ,φ (s t ,a t ) a t =π θ (s t ) ∇ θ π θ (s t ).(4) Concerning the critic updates, we employ also in this case ObGD with traces to minimize the fol- lowing objective: δ Q (s t ,a t ) = Q π θ ,φ (s t ,a t )− r(s t ,a t ) σ r +(5) + γE s t+1 ∼p,ε 2 ∼N h Q π θ ,φ (s t+1 ,π θ (s t+1 ) +ε 2 ) i !! 2 , where σ r is usual reward scaling signal presented in Engstrom et al. (2020). As for S2AC, no target network is employed, namely, the target in Equation 5 is given by the online version of Q π θ ,φ . 5 Additionally, inspired by Fujimoto et al. (2018), the target in Equation 5 presents an additional noise ε 2 ∼ N (0,Iσ 2 ) in order to mitigate the risk of critic overfitting to narrow peaks in the value estimate. The injection of small Gaussian noise into the target encourages the Q-value function to be smooth in the neighborhood of each action, reducing the variance introduced by function approximation errors and improving the stability of the learning targets. Its practical importance is shown in the ablation study in subsection 4.1. For a pseudocode of SDAC, refer to section S1 in the supplementary material. 4 Experiments Following Elsayed et al. (2024), we consider several environments from MuJoCo Gym (Todorov et al., 2012) and DM Control Suite (Tunyasuvunakool et al., 2020) as benchmarks for control in continuous action spaces. In subsection 4.1 we present the results achieved by training the policy for 20M steps with S2AC and SDAC, while in subsection 4.2 we analyze how state normalization and reward scaling influence training dynamics and final performance over 3M steps when using batch RL methods, specifically SAC and TD3. Finally, in subsection 4.3 we investigate the challenges that arise when transitioning from batch to streaming learning, proposing a first attempt to address the problem. All reported plots are generated by repeating each experiment with 10 different random seeds. The agents are evaluated every 10,000 time steps by computing the average undiscounted return over 10 evaluation episodes. During evaluation, a deterministic policy is used: no exploration noise is added, and for stochastic policies the mean action is selected. 4.1 S2AC and SDAC from scratch In this section we discuss the results achieved by S2AC and SDAC, compared to Stream AC(λ). Concerning the hyperparameters and the architectures used for all streaming RL algorithms, refer to section S2 in the supplementary material. Figure 1: Results for streaming DRL algorithms SDAC, S2AC, and Stream AC(λ) on MuJoCo Gym and DM Control Suite tasks. As shown in Figure 1, both S2AC and SDAC achieve performance comparable to state-of-the-art methods. Notably, as further evidenced by the additional experiments in section S3 of the sup- plementary material, the relative dominance of each algorithm varies across environments. Unlike AVG (Vasan et al., 2024), S2AC requires no per-environment optimizer tuning. Similarly, SDAC introduces no environment-specific hyperparameters, underscoring the competitiveness of Q-based 6 streaming algorithms. Figure 2 presents an ablation study examining the contribution of each pro- posed modification. For S2AC, we compare a fixed entropy coefficient α against the adaptive sched- ule α/σ r ; for SDAC, we investigate the role of target noise in Equation 5. While S2AC still achieves decent performance without the adaptive schedule, the target noise modification is far more critical for SDAC, which fails to learn entirely without it. Figure 2: Ablation study for SDAC and S2AC. 4.2 Data normalization for TD3 and SAC Beyond the absence of a replay buffer, target networks, and batch updates, the key differences be- tween the TD3 and SAC implementations and their streaming counterparts SDAC and S2AC are state normalization and reward scaling. Here, we investigate the effect of incorporating these two techniques into batch methods: we track the statistics required for state normalization and reward scaling in the same manner as in streaming approaches, but samples stored in the replay buffer re- main unprocessed; instead, each batch is preprocessed at every network update. State normalization has been explored previously in Lee et al. (2024) for the SimBa architecture, where it was com- bined with deeper residual networks (He et al., 2016) and layer normalization to yield substantial performance gains for SAC and DDPG (Lillicrap et al., 2020). In contrast, we consider a simple two-layer network with fewer hidden units (128 rather than 512 for the critic) and no residual connections, and extend the analysis to TD3. As shown in Figure 3, the architectures proposed in Elsayed et al. (2024) combined with data normalization not only preserve performance but yield notable improvements across many environments, particularly for TD3. Ad- ditional experiments across further environments are provided in section S5 of the supplementary material, with full hyperparameter details in section S4. 4.3 Towards Batch-to-Streaming Deep RL In this section, we evaluate our approach on the two application scenarios introduced in sec- tion 1: Sim2Real finetuning and batch-to-streaming alternation during training. For the Sim2Real finetuning scenario, we use two environments from the DM Control Suite, walker-run and dog-walk, which we manually perturb at the end of pre-training to simulate the distribution shift typically encountered when transitioning from simulation to real-world deployment (details are pro- vided in section S6). For the batch-to-streaming alternation scenario, we use the more complex quadruped-run environment to assess the ability of our streaming DRL algorithm to sustain and build upon improvements from a pre-trained checkpoint, a setting that arises naturally under energy or hardware temporal constraints. 7 123 Environment Steps (×10 6 ) 0 1500 3000 4500 6000 Average Episodic Return Humanoid-v4 123 Environment Steps (×10 6 ) 0 800 1600 2400 3200 Hopper-v4 123 Environment Steps (×10 6 ) 0 1500 3000 4500 6000 Ant-v4 12345 Environment Steps (×10 6 ) 0 200 400 600 800 Average Episodic Return humanoid-walk-v0 123 Environment Steps (×10 6 ) 0 200 400 600 800 dog-walk-v0 123 Environment Steps (×10 6 ) 200 400 600 800 quadruped-run-v0 SAC-normSACTD3-normTD3SAC-normSACTD3-normTD3 Figure 3: Result for the batch methods on MuJoCo Gym and DM Control Suite tasks. TD3-norm and SAC-norm denote the versions of TD3 and SAC with data normalization and the same network architectures used for the streaming approaches. Leveraging what discussed in subsection 4.2, we first attempted a direct transition from TD3-norm to SDAC. As shown in Figure 4, however, this naive switch leads in most cases to a severe drop in agent performance. Upon closer investigation, we identified the critic optimizer as a likely source of incompatibility between the two methods: TD3 uses Adam, whereas SDAC relies on ObGD. Draw- ing on the recent work of Pascanu et al. (2025), we hypothesize that the choice of optimizer shapes the qualitative properties of the learned solution through its inductive biases, thereby hindering a smooth transition between the two algorithms. 12345 Environment Steps (×10 6 ) 100 200 300 400 Average Episodic Return walker-run-v0 12345 Environment Steps (×10 6 ) 0 80 160 240 320 dog-walk-v0 12345 Environment Steps (×10 6 ) 150 300 450 600 quadruped-run-v0 Figure 4: Finetuning performance of SDAC after pre-training with TD3-norm using Adam as the critic optimizer. For each environment, we report three different intermediate pre-training check- points across different seeds. Moreover, for each of them we averaged the results across three seeds of finetuning. The horizontal dashed lines represent the agent performance before finetuning. We therefore propose to replace Adam with SGDC (Sun et al., 2025) for the critic during pre- training, motivated by its close resemblance to the ObGD optimizer used in streaming algorithms (see subsection 2.2). This substitution preserves pre-training performance relative to Adam, at the cost of a modest reduction in sample efficiency (see section S7 for additional results). The effect of this choice is further illustrated in Figure 5, which reports the L 2 -norm of the critic weights throughout training for both optimizers. Notably, Adam induces a rapid growth of the critic weight norm during pre-training. As shown by Lyle et al. (2024), in networks with normalization layers, large weight norms are associated with reduced plasticity, impairing the network’s ability to adapt to new data. We therefore conjecture that the large weight norms accumulated during Adam-based pre-training may hinder the agent’s ability to adapt to the shifted environment at the onset of finetun- ing. On the other hand, using SGDC leads to substantially smaller critic weight norms throughout 8 pre-training, resulting in significantly better finetuning performance when subsequently switching to SDAC, as shown in Figure 6. Remarkably, initializing from a pre-trained policy can even allow SDAC to surpass the performance achievable when training from scratch, while requiring consider- ably fewer samples. In section S8, we report additional results and experimental details. We note, however, that finetuning results on quadruped-run remain limited, with performance drops that are not yet within acceptable bounds despite partial improvements in some cases. Given that sustaining continuous improvement from a pre-trained checkpoint without significant regres- sion is a key requirement for practical deployment, we view this as an important open challenge, consistent with our framing of this work as a first step toward integrating batch and streaming al- gorithms. Similarly, preliminary finetuning results for S2AC (reported in section S8) are not yet competitive: SAC-norm with SGDC does not always train reliably, and tuning the entropy coeffi- cient proves particularly sensitive in the finetuning regime. Nevertheless, we argue that maximum entropy RL remains a promising direction, as entropy regularization naturally promotes robustness and behavioral diversity, qualities that are particularly desirable under distribution shift (Eysenbach & Levine, 2022). 246810 Environment Steps (×10 5 ) 0 10 20 30 40 50 L 2 -norm critic weights walker-run-v0 246810 Environment Steps (×10 5 ) 0 20 40 60 80 L 2 -norm critic weights dog-walk-v0 TD3-norm SGDCTD3-norm Adam Figure 5: L 2 -norm of the critic’s network weights across 1M steps of training. 12345 Environment Steps (×10 6 ) 160 240 320 400 480 Average Episodic Return walker-run-v0 12345 Environment Steps (×10 6 ) 300 400 500 600 dog-walk-v0 12345 Environment Steps (×10 6 ) 400 480 560 640 quadruped-run-v0 Figure 6: Finetuning performance of SDAC after pre-training with TD3-norm using SGDC as the critic optimizer. For each environment, we report three different intermediate pre-training check- points across different seeds. Moreover, for each of them we averaged the results across three seeds of finetuning. The horizontal dashed lines represent the agent performance before finetuning. 5 Conclusions This work represents a first step toward bridging batch and streaming deep reinforcement learn- ing, introducing a paradigm with potential for real-world deployment scenarios such as Sim2Real finetuning. We developed two novel streaming DRL algorithms, S2AC and SDAC, which achieve competitive performance against state-of-the-art methods while maintaining architectural compat- ibility with established batch algorithms such as SAC and TD3. Building on this foundation, we investigated the practical challenges that emerge when transitioning from batch to streaming learn- ing during the finetuning phase, and proposed an initial framework for addressing them. 9 While several open challenges remain, we hope this work will encourage further research at the intersection of batch and streaming deep RL. More broadly, our findings suggest that batch and streaming algorithms should not be treated as independent paradigms, each optimized solely for its own regime, but rather conceived with a shared algorithmic foundation in mind. References Marcin Andrychowicz, Anton Raichuk, Piotr Sta ́ nczyk, Manu Orsini, Sertan Girgin, Raphael Marinier, Léonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, et al. What matters in on-policy reinforcement learning? a large-scale empirical study.arXiv preprint arXiv:2006.05990, 2020. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016. Shuxiao Chen, Bike Zhang, Mark W Mueller, Akshara Rai, and Koushil Sreenath. Learning torque control for quadrupedal locomotion. In 2023 IEEE-RAS 22nd International Conference on Hu- manoid Robots (Humanoids), p. 1–8. IEEE, 2023. Sebastien D De Rivaz, Benjamin Goldberg, Neel Doshi, Kaushik Jayaram, Jack Zhou, and Robert J Wood. Inverted and vertical climbing of a quadrupedal microrobot using electroadhesion. Science Robotics, 3(25):eaau3038, 2018. Bardienus P Duisterhof, Shushuai Li, Javier Burgués, Vijay Janapa Reddi, and Guido CHE De Croon. Sniffy bug: A fully autonomous swarm of gas-seeking nano quadcopters in cluttered environments. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 9099–9106. IEEE, 2021. Mohamed Elsayed, Gautham Vasan, and A Rupam Mahmood. Streaming deep reinforcement learn- ing finally works. arXiv preprint arXiv:2410.14606, 2024. Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep policy gradients: A case study on ppo and trpo. arXiv preprint arXiv:2005.12729, 2020. Benjamin Eysenbach and Sergey Levine. Maximum entropy rl (provably) solves some robust rl problems. In 10th International Conference on Learning Representations, ICLR 2022, 2022. Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference on machine learning, p. 1587–1596. PMLR, 2018. Scott Fujimoto, Wei-Di Chang, Edward Smith, Shixiang Shane Gu, Doina Precup, and David Meger. For sale: State-action representation learning for deep reinforcement learning. Advances in neural information processing systems, 36:61573–61624, 2023. Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International conference on machine learning, p. 1352–1361. PMLR, 2017. Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and appli- cations. arXiv preprint arXiv:1812.05905, 2018. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 770–778, 2016. 10 Sebastian Höfer, Kostas Bekris, Ankur Handa, Juan Camilo Gamboa, Melissa Mozifian, Florian Golemo, Chris Atkeson, Dieter Fox, Ken Goldberg, John Leonard, et al. Sim2real in robotics and automation: Applications and challenges. IEEE transactions on automation science and engineering, 18(2):398–400, 2021. Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Ki- nal Mehta, and JoÃG , o GM AraÚjo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23(274):1–18, 2022. Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. Vijay Konda and John Tsitsiklis. Actor-critic algorithms. Advances in neural information processing systems, 12, 1999. Hojoon Lee, Dongyoon Hwang, Donghu Kim, Hyunseung Kim, Jun Jet Tai, Kaushik Subramanian, Peter R Wurman, Jaegul Choo, Peter Stone, and Takuma Seno. Simba: Simplicity bias for scaling up parameters in deep reinforcement learning. arXiv preprint arXiv:2410.09754, 2024. Timothy Paul Lillicrap, Jonathan James Hunt, Alexander Pritzel, Nicolas Manfred Otto Heess, Tom Erez, Yuval Tassa, David Silver, and Daniel Pieter Wierstra. Continuous control with deep rein- forcement learning, September 15 2020. US Patent 10,776,692. Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires, Razvan Pascanu, and Will Dabney. Understanding plasticity in neural networks. In International Conference on Machine Learning, p. 23190–23211. PMLR, 2023. Clare Lyle, Zeyu Zheng, Khimya Khetarpal, James Martens, Hado P van Hasselt, Razvan Pascanu, and Will Dabney. Normalization and effective learning rates in reinforcement learning. Advances in Neural Information Processing Systems, 37:106440–106473, 2024. Marius Memmel, Andrew Wagenmaker, Chuning Zhu, Dieter Fox, and Abhishek Gupta. Asid: Active exploration for system identification in robotic manipulation. In The Twelfth International Conference on Learning Representations, 2024. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529–533, 2015. Michal Nauman, Michał Bortkiewicz, Piotr Miło ́ s, Tomasz Trzci ́ nski, Mateusz Ostaszewski, and Marek Cygan. Overestimation, overfitting, and plasticity in actor-critic: the bitter lesson of rein- forcement learning. arXiv preprint arXiv:2403.00514, 2024. Sabrina M Neuman, Brian Plancher, Bardienus P Duisterhof, Srivatsan Krishnan, Colby Banbury, Mark Mazumder, Shvetank Prakash, Jason Jabbour, Aleksandra Faust, Guido CHE de Croon, et al. Tiny robot learning: Challenges and directions for machine learning in resource-constrained robots. In 2022 IEEE 4th international conference on artificial intelligence circuits and systems (AICAS), p. 296–299. IEEE, 2022. Razvan Pascanu, Clare Lyle, Ionut-Vlad Modoranu, Naima Elosegui Borras, Dan Alistarh, Petar Velickovic, Sarath Chandar, Soham De, and James Martens. Optimizers qualitatively alter solu- tions and we should leverage this. arXiv preprint arXiv:2507.12224, 2025. Nicholas Pfaff, Evelyn Fu, Jeremy Binagia, Phillip Isola, and Russ Tedrake. Scalable real2sim: Physics-aware asset generation via robotic pick-and-place setups. In 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 6296–6303. IEEE, 2025. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. 11 David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International conference on machine learning, p. 387–395. Pmlr, 2014. Tao Sun, Xinwang Liu, and Kun Yuan. Revisiting gradient normalization and clipping for noncon- vex sgd under heavy-tailed noise: Necessity, sufficiency, and acceleration. Journal of Machine Learning Research, 26(237):1–42, 2025. Richard S Sutton and Andrew G Barto. Toward a modern theory of adaptive networks: expectation and prediction. Psychological review, 88(2):135, 1981. Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, Cambridge, MA, 1998. Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999. Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, p. 5026–5033. IEEE, 2012. Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm_control: Software and tasks for continuous control. Software Impacts, 6:100022, 2020. Gautham Vasan, Mohamed Elsayed, Seyed Alireza Azimi, Jiamin He, Fahim Shahriar, Colin Bellinger, Martha White, and Rupam Mahmood. Deep policy gradient methods without batch updates, target networks, or replay buffers. Advances in Neural Information Processing Systems, 37:845–891, 2024. Yan Wang, Gautham Vasan, and A Rupam Mahmood. Real-time reinforcement learning for vision- based robotics utilizing local and remote computers. arXiv preprint arXiv:2210.02317, 2022. Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3):279–292, 1992. Barry Payne Welford. Note on a method for calculating corrected sums of squares and products. Technometrics, 4(3):419–420, 1962. Mingdong Wu, Lehong Wu, Yizhuo Wu, Weiyao Huang, Hongwei Fan, Zheyuan Hu, Haoran Geng, Jinzhou Li, Jiahe Ying, Long Yang, et al. Simlauncher: Launching sample-efficient real-world robotic reinforcement learning via simulation pre-training. In 2025 IEEE/RSJ International Con- ference on Intelligent Robots and Systems (IROS), p. 7933–7940. IEEE, 2025. Patrick Yin, Tyler Westenbroek, Simran Bagaria, Kevin Huang, Ching-an Cheng, Andrey Kobolov, and Abhishek Gupta. Rapidly adapting policies to the real world via simulation-guided fine- tuning. arXiv preprint arXiv:2502.02705, 2025. Jingzhao Zhang, Sai Praneeth Karimireddy, Andreas Veit, Seungyeon Kim, Sashank Reddi, Sanjiv Kumar, and Suvrit Sra. Why are adaptive methods good for attention models?Advances in Neural Information Processing Systems, 33:15383–15393, 2020. Brian D Ziebart, J Andrew Bagnell, and Anind K Dey. Modeling interaction via the principle of maximum causal entropy. 2010. 12 Supplementary Materials The following content was not necessarily subject to peer review. S1 Pseudocode for S2AC and SDAC We report here the pseudocode for S2AC (Algorithm 1) and SDAC (Algorithm 2). We refer to Elsayed et al. (2024) for the pseudocode of the NormalizeObservation and ScaleReward algorithms. Algorithm 1 Streaming Soft Actor-Critic (S2AC) 1: Given LayerNorm policy network π(a|s;θ) parameterizing a normal distribution with vector- ized weights vector θ and initialized with SparseInit 2: Given LayerNorm soft action-value network Q soft (s,a;φ) with vectorized weights vector φ and initialized with SparseInit 3: Initialize discount factor γ and eligibility traces parameter λ for the critic 4: Initialize policy learning rate η π , action-value step size η Q , starting entropy coefficient α 0 , and action-value scaling factor κ Q 5: Initialize p r ,p s to zero and μ s ,t to one 6: for each episode do 7: z φ ←0 8:Initialize s (first state of the episode) 9: s,μ s ,p s ,← NormalizeObservation(s,μ s ,p s ,t) 10:for each time step in the episode do 11:t← t + 1 12:a∼ π(·|s;θ) 13:Take action a, observe s ′ , r, T▷ T indicates whether s ′ is a terminal state 14:s ′ ,μ s ,p s ← NormalizeObservation(s ′ ,μ s ,p s ,t) 15:r,p r ,σ r ← ScaleReward(r,γ,p r ,T,t) 16:α← α 0 /σ r 17:a ′ ∼ π(·|s ′ ;θ) 18:δ ← r + γ(1− T )(Q soft (s ′ ,a ′ ;φ)− α logπ(a ′ |s ′ ;θ))− Q soft (s,a;φ) 19:z φ ← γλz φ +∇ φ Q soft (s,a;φ) 20:J π (θ)← α logπ(a|s;θ)− Q soft (s,a;φ) 21:φ← ObGD(z φ ,φ,δ,η Q ,κ Q ) 22:θ ← θ− η π ∇ θ J π (θ)▷ Update performed with Adam 23:s← s ′ 24:end for 25: end for 13 Algorithm 2 Streaming Deterministic Actor-Critic (SDAC) 1: Given LayerNorm actor network π(s;θ) parameterizing a deterministic policy with vectorized weights vector θ and initialized with SparseInit 2: Given LayerNorm action-value network Q(s,a;φ) with vectorized weights vector φ and ini- tialized with SparseInit 3: Initialize discount factor γ and eligibility traces parameter λ for the critic 4: Initialize policy learning rate η π , action-value step size η Q , gaussian std σ, and action-value scaling factor κ Q 5: Initialize p r ,p s to zero and μ s ,t to one 6: for each episode do 7: z φ ←0 8:Initialize s (first state of the episode) 9: s,μ s ,p s ,← NormalizeObservation(s,μ s ,p s ,t) 10:for each time step in the episode do 11:t← t + 1 12:ε 1 ∼N (0,Iσ 2 ) 13:a← π(s;θ) +ε 1 14:Take action a, observe s ′ , r, T▷ T indicates whether s ′ is a terminal state 15:s ′ ,μ s ,p s ← NormalizeObservation(s ′ ,μ s ,p s ,t) 16:r,p r ← ScaleReward(r,γ,p r ,T,t) 17:ε 2 ∼N (0,Iσ 2 ) 18:a ′ ← π(s ′ ;θ) +ε 2 19:δ ← r + γ(1− T )Q(s ′ ,a ′ ;φ)− Q(s,a;φ) 20:z φ ← γλz φ +∇ φ Q(s,a;φ) 21:J π (θ)←−Q(s,π(s;θ);φ) 22:φ← ObGD(z φ ,φ,δ,η Q ,κ Q ) 23:θ ← θ− η π ∇ θ J π (θ)▷ Update performed with Adam 24:s← s ′ 25:end for 26: end for 14 S2 Hyperparameters for S2AC, SDAC and Stream AC(λ) Regardless of the streaming algorithm and environment, we set γ = 0.99. For the hyperparameters of Stream AC(λ) we follow Elsayed et al. (2024) and Table 1 reports all values. Table 2 reports the hyperparameters concerning SDAC, while Table 3 the ones for S2AC. Concerning α for S2AC, Table 4 reports the specific value per environment. Following Elsayed et al. (2024), the same architecture, both for critic and actor, is used. Specifically, Pseudocode 1, Pseudocode 2 and Pseudocode 3 present the pseudocode for the network architectures of each streaming algorithm Following Haarnoja et al. (2018), we apply squashing for S2AC: denoting with μ θ (·|s) the Gaussian distribution parametrized by the actor network’s output and byu∈ R D a sample from it, we enforce action bounds via a = tanh(u). The corresponding log-likelihood is: logπ θ (a|s) = D X i=1 logμ θ (u i |s)− 2· (log 2− u i − SoftPlus(−2· u i ))(6) where u i denotes the i-th component of u. Table 1: Stream AC(λ) Hyperparameters. HyperparameterValue entropy coefficient τ0.01 Actor Optimizer OptimizerObGD (Elsayed et al., 2024) Learning rate1.0 λ0.8 κ3.0 Critic Optimizer OptimizerObGD (Elsayed et al., 2024) Learning rate1.0 λ0.8 κ2.0 Table 2: SDAC Hyperparameters. HyperparameterValue Target Noise N (0, 0.2 2 ) Exploration noise N (0, 0.2 2 ) Actor Optimizer OptimizerAdam (Kingma & Ba, 2014) Learning rate3e− 4 Betasβ 1 = 0.9,β 2 = 0.999 Critic Optimizer OptimizerObGD (Elsayed et al., 2024) Learning rate1.0 λ0.8 κ2.0 15 Table 3: S2AC Hyperparameters. HyperparameterValue log std max2 log std min −20 Actor Optimizer OptimizerAdam (Kingma & Ba, 2014) Learning rate3e− 4 Betasβ 1 = 0.9,β 2 = 0.999 Critic Optimizer OptimizerObGD (Elsayed et al., 2024) Learning rate1.0 λ0.8 κ2.0 Table 4: S2AC alpha per environment. EnvironmentTemperature α MuJoCo Gym (Todorov et al., 2012) Humanoid-v4 0.2 HumanoidStandup-v4 HalfCheetah-v4 Hopper-v4 Walker2d-v4 Ant-v4 DM Control Suite (Tunyasuvunakool et al., 2020) dog-walk-v0 0.002 cheetah-run-v0 walker-run-v0 quadruped-run-v0 cartpole-swingup-v00.01 finger-spin-v00.012 16 Pseudocode 1. Stream AC(λ) Network Details Value V Network: fc_layer = Linear(state_dim, 128) hidden_layer = Linear(128, 128) output_linear = Linear(128, 1) Value V Forward Pass: input = state x = LeakyReLU(layer_norm(fc_layer(input))) x = LeakyReLU(layer_norm(hidden_layer(x))) value = output_linear(x) Policy π Network: fc_layer = Linear(state_dim, 128) hidden_layer = Linear(128, 128) linear_mu = Linear(128, action_dim) linear_std = Linear(128, action_dim) Policy π Forward Pass: input = state x = LeakyReLU(layer_norm(fc_layer(input))) x = LeakyReLU(layer_norm(hidden_layer(x))) mu = linear_mu(x) pre_std = linear_std(x) std = SoftPlus(pre_std) action = Normal(mu, std).sample() 17 Pseudocode 2. SDAC Network Details Value Q Network: fc_layer = Linear(state_dim + action_dim, 128) hidden_layer = Linear(128, 128) output_linear = Linear(128, 1) Value Q Forward Pass: input = concatenate([state, action]) x = LeakyReLU(layer_norm(fc_layer(input))) x = LeakyReLU(layer_norm(hidden_layer(x))) value = output_linear(x) Policy π Network: fc_layer = Linear(state_dim, 128) hidden_layer = Linear(128, 128) linear_mu = Linear(128, action_dim) linear_std = Linear(128, action_dim) Policy π Forward Pass: input = state x = LeakyReLU(layer_norm(fc_layer(input))) x = LeakyReLU(layer_norm(hidden_layer(x))) action = tanh(linear_mu(x)) 18 Pseudocode 3. S2AC Network Details Value Q soft Network: fc_layer = Linear(state_dim + action_dim, 128) hidden_layer = Linear(128, 128) output_linear = Linear(128, 1) Value Q soft Forward Pass: input = concatenate([state, action]) x = LeakyReLU(layer_norm(fc_layer(input))) x = LeakyReLU(layer_norm(hidden_layer(x))) value = output_linear(x) Policy π Network: fc_layer = Linear(state_dim, 128) hidden_layer = Linear(128, 128) linear_mu = Linear(128, action_dim) LOG_STD_MAX = 2 LOG_STD_MIN = -20 Policy π Forward Pass: input = state x = LeakyReLU(layer_norm(fc_layer(input))) x = LeakyReLU(layer_norm(hidden_layer(x))) mu = linear_mu(x) log_std = linear_std(x) log_std = tanh(log_std) log_std = LOG_STD_MIN + \ 0.5 * (LOG_STD_MAX - LOG_STD_MIN) * (log_std + 1) x = Normal(mu, exp(log_std)).sample() action = tanh(x) 19 S3 Additional results for streaming RL We present additional results for the streaming algorithms. In Figure 7 we present the same results as in subsection 4.1 including a comparison with TD3 and SAC, while Figure 8 and Figure 9 present additional results on different environments from DM Control Suite and MuJoCo Gym, respectively. Figure 7: Results for streaming DRL algorithms SDAC, S2AC, and Stream AC(λ) on MuJoCo Gym and DM Control Suite tasks. TD3 and SAC in the legend denote results with state normalization and reward scaling, trained over 3M steps. Figure 8: Additional results for streaming DRL algorithms SDAC, S2AC, and Stream AC(λ) on DM Control Suite tasks. TD3 and SAC in the legend denote results with state normalization and reward scaling, trained over 3M steps. 20 5101520 Environment Steps (×10 6 ) 0 1500 3000 4500 Average Episodic Return Walker2d-v4 5101520 Environment Steps (×10 6 ) 0 800 1600 2400 3200 Hopper-v4 5101520 Environment Steps (×10 6 ) 0 1500 3000 4500 6000 Ant-v4 SDACS2ACStream ACSDACS2ACStream AC TD3SAC Figure 9: Additional results for streaming DRL algorithms SDAC, S2AC, and Stream AC(λ) on MuJoCo Gym tasks. TD3 and SAC in the legend denote results with state normalization and reward scaling, trained over 3M steps. 21 S4 Hyperparameters for SAC and TD3 For the SAC and TD3 implementations, we employ the CleanRL repository (Huang et al., 2022). In Pseudocode 4 and Pseudocode 5, we report the architectures for the neural networks, where the hidden size is reduced from 256 to 128 to match the streaming approaches. Table 5 reports the hyperparameters for SAC, those used in Haarnoja et al. (2018), while Table 6 reports those for TD3 TD3, as in the original work Fujimoto et al. (2018). For SAC-norm, the same network architectures of S2AC (Pseudocode 3) are employed and the same hyperparameters of SAC are kept (Table 5). Similarly, TD3-norm uses the same architectures of SDAC (Pseudocode 2) and the same hyperparameters of TD3 (Table 6). Table 5: SAC and SAC-norm Hyperparameters. HyperparameterValue No streaming params Buffer size10 6 Batch size256 Exploration time steps5× 10 3 Target smoothing coeff. τ = 0.005 Frequency policy update2 αAutotune Target entropy−action_dim log std max2 log std min−20 Optimizer OptimizerAdam (Kingma & Ba, 2014) Learning rate3e− 4 Betasβ 1 = 0.9,β 2 = 0.999 Table 6: TD3 and TD3-norm Hyperparameters. HyperparameterValue No streaming params Buffer size10 6 Batch size256 Exploration time steps25× 10 3 Target smoothing coeff. τ = 0.005 Frequency policy update2 Target NoiseN (0, 0.2 2 ) Exploration Noise N (0, 0.1 2 ) Optimizer OptimizerAdam (Kingma & Ba, 2014) Learning rate3e− 4 Betasβ 1 = 0.9,β 2 = 0.999 22 Pseudocode 4. SAC Network Details Value Q soft Network: fc_layer = Linear(state_dim + action_dim, 128) hidden_layer = Linear(128, 128) output_linear = Linear(128, 1) Value Q soft Forward Pass: input = concatenate([state, action]) x = ReLU(fc_layer(input)) x = ReLU(hidden_layer(x)) value = output_linear(x) Policy π Network: fc_layer = Linear(state_dim, 128) hidden_layer = Linear(128, 128) linear_mu = Linear(128, action_dim) LOG_STD_MAX = 2 LOG_STD_MIN = -20 Policy π Forward Pass: input = state x = ReLU(fc_layer(input)) x = ReLU(hidden_layer(x)) mu = linear_mu(x) log_std = linear_std(x) log_std = tanh(log_std) log_std = LOG_STD_MIN + \ 0.5 * (LOG_STD_MAX - LOG_STD_MIN) * (log_std + 1) x = Normal(mu, exp(log_std)).sample() action = tanh(x) 23 Pseudocode 5. TD3 Network Details Value Q Network: fc_layer = Linear(state_dim + action_dim, 128) hidden_layer = Linear(128, 128) output_linear = Linear(128, 1) Value Q Forward Pass: input = concatenate([state, action]) x = ReLU(fc_layer(input)) x = ReLU((hidden_layer(x)) value = output_linear(x) Policy π Network: fc_layer = Linear(state_dim, 128) hidden_layer = Linear(128, 128) linear_mu = Linear(128, action_dim) linear_std = Linear(128, action_dim) Policy π Forward Pass: input = state x = ReLU(fc_layer(input)) x = ReLU(hidden_layer(x)) action = tanh(linear_mu(x)) 24 S5 Additional results for TD3 and SAC Figure 10 reports additional TD3 and SAC results on environments not included in Figure 3 of subsection 4.2. 123 Environment Steps (×10 6 ) 0 200 400 600 800 Average Episodic Return cartpole-swingup-v0 123 Environment Steps (×10 6 ) 0 200 400 600 800 walker-run-v0 123 Environment Steps (×10 6 ) 0 250 500 750 1000 finger-spin-v0 123 Environment Steps (×10 6 ) 0 200 400 600 800 Average Episodic Return cheetah-run-v0 123 Environment Steps (×10 6 ) 0 200 400 600 800 humanoid-stand-v0 123 Environment Steps (×10 6 ) 0 50 100 150 hopper-hop-v0 123 Environment Steps (×10 6 ) 0 1500 3000 4500 Average Episodic Return Walker2d-v4 123 Environment Steps (×10 6 ) 50000 100000 150000 200000 HumanoidStandup-v4 123 Environment Steps (×10 6 ) 0 3000 6000 9000 12000 HalfCheetah-v4 SAC-normSACTD3-normTD3SAC-normSACTD3-normTD3 Figure 10: Additional results for TD3 and SAC. 25 S6 Perturbed environments details Concerning the dog-walk-v0 environment of DeepMind Control Suite (Tunyasuvunakool et al., 2020), the following changes have been applied to obtain a perturbed version of the original envi- ronment: 1. The control range is changed from [−1, 1] to [−0.7, 0.7]. 2. The stiffness of the joint lumbar_extend is changed from 30.0 to 5.0. 3. The stiffness of the joint lumbar_bend is changed from 30.0 to 5.0. 4. The stiffness of the joint cervical is changed from 4.0 to 1.0. 5. The stiffness of the joint hip is changed from 5.0 to 2.0. 6. The control gain for the knee joints is reduced from 30 to 10. 7. The control gain for the ankle joints is reduced from 20 to 10. Collectively, these modifications simulate either an ageing dog or the degradation of materials and actuators in a robotic dog. For instance, reduced lumbar stiffness reflects the effect of weakened back muscles in a biological dog, or material fatigue in the spinal structure of a robotic one. We apply analogous modifications to walker-run-v0: 1. The stiffness of the ankles is changed from 0.0 to 15.0. 2. The stiffness of knees are changed from 0.0 to 15.0. 3. The actuator gains for right and left hips are reduced from 100 to 80. 4. The actuator gains for right and left knees are reduced from 50 to 40. 5. The actuator gains for right and left ankles are reduced from 20 to 16. For reproducibility reasons, we provide the modified .xml files for both environments. 26 S7 SGDC applied to TD3 Figure 11 reports additional results on the use of SGDC as the critic optimizer in TD3. Overall, TD3 with SGDC (Sun et al., 2025) matches standard TD3 in performance but exhibits reduced sample efficiency in some environments, such as humanoid-stand-v0 and dog-walk-v0. 12345678 Environment Steps (×10 6 ) 0 200 400 600 800 Average Episodic Return humanoid-stand-v0 123 Environment Steps (×10 6 ) 200 400 600 800 quadruped-run-v0 12345 Environment Steps (×10 6 ) 0 200 400 600 800 dog-walk-v0 123 Environment Steps (×10 6 ) 0 1500 3000 4500 6000 Average Episodic Return Ant-v4 123 Environment Steps (×10 6 ) 0 800 1600 2400 3200 Hopper-v4 1234 Environment Steps (×10 6 ) 0 1500 3000 4500 6000 Humanoid-v4 TD3-normTD3TD3-norm-SGDCTD3-normTD3TD3-norm-SGDC Figure 11: Results for TD3 with SGDC. 27 S8 Additional fine-tuning details S8.1 TD3-to-SDAC For pre-training with TD3, we use the hyperparameters defined in Table 6 for the Adam optimizer, and those in Table 7 for SGDC. Table 7: TD3-norm with SGDC Hyperparameters. HyperparameterValue No streaming params Buffer size10 6 Batch size256 Exploration time steps25× 10 3 Target smoothing coeff. τ = 0.005 Frequency policy update2 Target NoiseN (0, 0.2 2 ) Exploration Noise N (0, 0.1 2 ) Actor Optimizer OptimizerAdam (Kingma & Ba, 2014) Learning rate3e− 4 Betasβ 1 = 0.9,β 2 = 0.999 Critic Optimizer OptimizerSGDC (Sun et al., 2025) Learning rate0.5 Clipping parameter h1.0 Since in a Sim2Real scenario the distribution shift might affect the actual action-value function Q, we additionally propose a critic warm-up phase in which the policy is frozen and only the critic is updated. All results in subsection 4.3 use the SDAC hyperparameters in Table 8, with additional results in Figure 12. Figure 13 presents an ablation study for the Q-warm-up. 12345 Environment Steps (×10 6 ) 240 320 400 480 Average Episodic Return walker-run-v0 12345 Environment Steps (×10 6 ) 200 300 400 500 dog-walk-v0 12345 Environment Steps (×10 6 ) 420 440 460 480 500 quadruped-run-v0 Figure 12: Finetuning performance of SDAC after pre-training with TD3-norm using SGDC as the critic optimizer. For each environment, we report three different intermediate pre-training check- points across different seeds. Moreover, for each of them we averaged the results across three seeds of finetuning. The horizontal dashed lines represent the agent performance before finetuning. Moreover, we investigate the role of exploration noise during finetuning: while exploration noise is necessary in deterministic approaches, the inherently noisy nature of the online streaming setting may itself act as an implicit exploration mechanism, potentially making explicit noise injection redundant. We study this in the ablation in Figure 14. 28 12345 Environment Steps (×10 6 ) 300 360 420 480 Average Episodic Return walker-run-v0 12345 Environment Steps (×10 6 ) 100 200 300 400 500 dog-walk-v0 12345 Environment Steps (×10 6 ) 240 320 400 480 quadruped-run-v0 Figure 13: Finetuning performance of SDAC without Q-warm-up after pre-training with TD3-norm using SGDC as the critic optimizer. For each environment, we report three different intermediate pre-training checkpoints across different seeds. Moreover, for each of them we averaged the results across three seeds of finetuning. The horizontal dashed lines represent the agent performance before finetuning. Table 8: SDAC Hyperparameters. HyperparameterValue Target Noise N (0, 0.1 2 ) Exploration noise N (0, 0.1 2 ) Q−warm-up5, 000 steps Actor Optimizer OptimizerAdam (Kingma & Ba, 2014) Learning rate3e− 4/256 Betasβ 1 = 0.9,β 2 = 0.999 Critic Optimizer OptimizerObGD (Elsayed et al., 2024) Learning rate1.0 λ0.8 κ2.0 12345 Environment Steps (×10 6 ) 180 240 300 360 420 Average Episodic Return walker-run-v0 12345 Environment Steps (×10 6 ) 150 300 450 dog-walk-v0 12345 Environment Steps (×10 6 ) 200 300 400 500 quadruped-run-v0 Figure 14: Finetuning performance of SDAC with no exploration noise after pre-training with TD3- norm using SGDC as the critic optimizer. For each environment, we report three different interme- diate pre-training checkpoints across different seeds. Moreover, for each of them we averaged the results across three seeds of finetuning. The horizontal dashed lines represent the agent performance before finetuning. 29 S8.2 SAC-to-S2AC In Figure 15 we present both the results of S2AC fine-tuning in the walker-run-v0 environment and the pre-training performance of SAC with SGDC for the dog-walk environment, to support what discussed in subsection 4.3. 12345 Environment Steps (×10 6 ) 80 160 240 320 Average Episodic Return walker-run-v0 (a) Finetuning performance of S2AC after pre- training with SAC-norm using SGDC as the critic optimizer. 123 Environment Steps (×10 6 ) 0 200 400 600 800 Average Episodic Return dog-walk-v0 SAC-norm SAC SAC-norm-SGDC (b) Results for SAC with SGDC. Figure 15 30