Paper deep dive
Beyond Isolation: Unlocking Reinforcement Learning Component Synergy for Sample-Efficient Continuous Control
Qi Zhao, Guozheng Ma, Yilun Kong, Lu Li, Haoyu Wang, Zilin Wang, Tiantian Zhang, Yuxing Wang, Jian Sha, Yongzhe Chang, Xueqian Wang, Dacheng Tao
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 8/10/2026, 4:12:08 AM
Summary
The paper investigates the synergistic interactions between key reinforcement learning components: Model-based Representation (R), Optimization Stability (OS), and Experience Replay (ER). It finds that naively stacking these techniques often leads to counterproductive interference, such as compounded non-stationarity. The authors propose ROSER, a framework that coordinates these components using a stable Simba backbone, residual-style feature fusion for representation, and a Uniform-to-Prioritized Experience Replay (U2P) strategy. ROSER achieves significant performance gains over vanilla baselines and naive stacks in continuous control tasks.
Entities (10)
Relation Signals (9)
ROSER → includescomponent → Model-based Representation
confidence 95% · ROSER coordinates three critical dimensions: Model-based Representation, Optimization Stability, and Experience Replay.
ROSER → includescomponent → Experience Replay
confidence 95% · ROSER coordinates three critical dimensions: Model-based Representation, Optimization Stability, and Experience Replay.
ROSER → includescomponent → Optimization Stability
confidence 95% · ROSER coordinates three critical dimensions: Model-based Representation, Optimization Stability, and Experience Replay.
ROSER → outperforms → SAC
confidence 95% · ROSER consistently outperforms vanilla baselines and achieves 17.60% gains over naive stack.
Experience Replay → implementedby → U2P
confidence 92% · we propose Uniform-to-Prioritized replay (U2P)... This design gradually increases the influence of prioritized replay
Optimization Stability → implementedby → Simba
confidence 92% · Given the empirically proven effectiveness of SimBa across various experimental settings, we adopt its architecture as the representative technique for OS in our study.
Naive Stack → causes → compounded non-stationarity
confidence 90% · naively stacking state-of-the-art techniques does not necessarily yield performance gains; instead, it often triggers emergent challenges, such as compounded non-stationarity.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Reinforcement learning systems are significantly more complex than other machine learning paradigms due to inherent properties, causing RL system design to jointly account for many tightly coupled factors. Despite advances in individual algorithmic components, their functional interdependencies remain underexplored: do they exhibit mutual synergy or counterproductive interference? To bridge this gap, we conduct a systematic investigation and find that the efficacy of different components exhibits significant task-dependency, and naively stacking state-of-the-art techniques does not necessarily yield performance gains; instead, it often triggers emergent challenges, such as compounded non-stationarity. Building upon these findings, we distill a suite of actionable insights into the principled coordination of these components. Guided by these insights, we propose ROSER, an RL framework that coordinates three critical dimensions: Model-based Representation, Optimization Stability, and Experience Replay. Across diverse continuous-control benchmarks, ROSER consistently outperforms vanilla baselines and achieves 17.60% gains over naive stack. Our findings underscore the necessity of a holistic perspective in RL system design and paves the way for developing sample-efficient agents.
Tags
Links
- Source: https://arxiv.org/abs/2608.07086v1
- Canonical: https://arxiv.org/abs/2608.07086v1
Trouble viewing inline? Open PDF directly →
Full Text
66,783 characters extracted from source content.
Expand or collapse full text
Beyond Isolation: Unlocking Reinforcement Learning Component Synergy for Sample-Efficient Continuous Control Qi Zhao 1 , Guozheng Ma 2 , Yilun Kong 2 , Lu Li 3 , Haoyu Wang 2 , Zilin Wang 4 , Tiantian Zhang 1 , Yuxing Wang 1 , Jian Sha 1 , Yongzhe Chang 1 , Xueqian Wang 1 , Dacheng Tao 2 1 Tsinghua University, 2 Nanyang Technological University, 3 Mila - Quebec Artificial Intelligence Institute, 4 University of Oxford Abstract Reinforcement learning systems are significantly more complex than other machine learning paradigms due to inherent properties, causing RL system design to jointly account for many tightly coupled factors. Despite advances in individual algorithmic components, their functional interdependencies remain underexplored: do they exhibit mutual synergy or counterproductive interference? To bridge this gap, we conduct a systematic investigation and find that the efficacy of different components exhibits significant task-dependency, and naively stacking state-of-the-art techniques does not necessarily yield performance gains; instead, it often triggers emergent challenges, such as compounded non-stationarity. Building upon these findings, we distill a suite of actionable insights into the principled coordination of these components. Guided by these insights, we proposeROSER, an RL framework that coordinates three critical dimensions: Model-basedRepresentation,OptimizationStability, andExperienceReplay. Across diverse continuous-control benchmarks,ROSERconsistently outperforms vanilla baselines and achieves 17.60% gains over naive stack. Our findings underscore the necessity of a holistic perspective in RL system design and paves the way for developing sample-efficient agents. 1. Introduction Reinforcement learning (RL) aims to optimize an agent’s decision-making strategy through environmental interaction. However, acquiring transitions often involves substantial time, computational resources, and, in many real-world applications, high physical or economic costs [18]. Consequently, improving sample efficiency has become a central challenge in RL research [17,33]. Recent literature addresses this from many perspectives like: representation learning [8,9] to enhance value and policy learning; optimization stability [20,21,24] focus on improving the optimization dynamics of deep learning; prioritized sampling [7,19,26] accelerate learning by assigning higher weights to more important transitions; and advanced exploration strategies [23,30] designed to collect more informative and diverse data. By refining these components, researchers aim to minimize the data required to reach optimal performance. Despite significant advancements, most existing research focuses on improving sample efficiency through isolated components of the learning pipeline. It remains unclear whether different techniques for improving sample efficiency exhibit synergy, enhancing or counteracting each other. Furthermore, whether the joint integration of these techniques requires additional, specialized design considerations is 1 arXiv:2608.07086v1 [cs.LG] 7 Aug 2026 Beyond Isolation: Unlocking Reinforcement Learning Component Synergy for Sample-Efficient Continuous Control DMC-LocoDMC-Loco HB-LocoHB-Loco HB-Loco HB-Loco HB-ManiHB-Mani HB-Mani Myo-ManiMyo-Mani MS-Mani Figure 1: Performance Summary. We compareROSERwith vanilla SAC, single-component enhancements (SAC+R, SAC+OS, SAC+ER), and a naive stack of all components (Naive Stack).ROSERdemonstrates improved sample efficiency in both locomotion and manipulation tasks. Shaded regions represent the 95% confidence interval. Experimental tasks span a diverse range of control scenarios, such as legged locomotion, dexterous object manipulation, articulated robotic arms. a critical question that has yet to be explored. Addressing these questions is of dual importance. From a research perspective, a deeper understanding of these interdependencies can provide systematic guidance for developing next-generation sample-efficiency techniques. From a practical standpoint, understanding these interactions is essential for building unified frameworks that truly maximize sample efficiency, enabling RL agents to master complex, challenging tasks with minimal environment interaction. This work explores the joint enhancement paradigm and investigates the following core research questions: How do different sample-efficiency-enhancing techniques interact in reinforcement learning, and under what conditions can they be effectively combined to yield consistent and robust performance? We build our investigation around three representative components: Model-based Representation (R), Optimization Stability (OS), and Experience Replay (ER) mechanisms. These components are widely adopted in practice, though other components could be considered in future research. Our primary finding underscores the necessity of investigating sample efficiency within integrated, high- performance contexts rather than in isolation on top of basic algorithms (e.g., vanilla SAC). Specifically, we observe that the efficacy of existing enhancement techniques exhibits significant task-dependency; a mechanism that provides substantial gains in some tasks may cause performance degradation in others. Furthermore, our investigation reveals that simply stacking individual techniques does not necessarily yield an improvement in sample efficiency. This phenomenon suggests that the integration of multiple enhancement strategies introduces emergent challenges, such as mutual interference and compounded non-stationarity, that are absent when modules are studied in isolation. Our investigation yields three design insights for harmonizing multiple enhancements. Guided on these insights, we presentROSER, a reinforcement learning framework designed for synergistic enhancement.ROSERharmoniously integrates representative techniques from three critical components:R,OS, andER. As shown in Figure 1,ROSER achieves a notable gain in sample efficiency, outperforming methods that rely on isolated component enhancements and simply stacking individual techniques by a significant margin. 2 Beyond Isolation: Unlocking Reinforcement Learning Component Synergy for Sample-Efficient Continuous Control Contributions of this paper can be summarized as: 1.Interaction Investigation: We conduct an analysis of various techniques, revealing that their individual benefits do not scale linearly and can even exhibit counter-productive interference in integrated systems, demonstrating that the coordination problem is real and non-trivial. 2.Design Principle: Informed by our analysis, we distill three design principles to harmonize algorithmic interactions. Based on these, we proposeROSER, a reinforcement learning framework that achieves superior sample efficiency and cross-environment versatility. 3.Empirical Validation: Extensive experiments demonstrate the superior performance ofROSER, validating our distilled design principles and underscoring that the principled coordination of algorithmic components is essential for achieving peak sample efficiency. 2. Background This section provides the background and formal notation for our study. We review previous research that takes a system-level perspective to reinforcement learning frameworks, and present a focused overview of three representative components, R, OS, and ER. 2.1. Systematic Perspectives for RL The systematic integration of RL modules was pioneered by Rainbow [15], which combined several extensions to achieve state-of-the-art performance. Revisiting Rainbow [4] expanded on this work by advocating for inclusive evaluations across diverse regimes. More recently, Beyond The Rainbow (BTR) [5] further advanced this paradigm by integrating six modern algorithmic and architectural enhancements, establishing a new state-of-the-art with desktop-level computational efficiency. While these studies established the potential of component aggregation, they primarily focused on value-based methods (DQN) in discrete action spaces and treated modules as additive "plug-ins" without deeply exploring the underlying intervention between them. Our study builds upon this systematic lineage but shifts the focus toward Component Synergy, specifically within actor-critic frameworks (e.g., SAC) in continuous action spaces. We examine the interaction among some components to determine if they show synergy or interference. Beyond mere identification, we provide a diagnostic analysis of the underlying mechanisms that drive negative interactions, offering insights into component relationships. 2.2. Model-based Representation (R) for RL Representation learning aims to extract useful features from high-dimensional sensory inputs (e.g., images, raw observations) that are relevant for decision-making [3]. A good representation should retain essential information for decision-making while discarding irrelevant details. To achieve this, modern RL frameworks typically employ auxiliary tasks to provide additional supervision for the encoder network. By leveraging these self-supervised signals, such as reconstructing observations [11], predicting future latent states [13], or using contrastive estimation [28], the agent can "squeeze" significantly more information out of every transition collected from the environment, which substantially boosts sample efficiency. MR.Q [9] learns a state embedding푧 푠 and a state-action embedding푧 푠푎 through end-to-end training of 3 Beyond Isolation: Unlocking Reinforcement Learning Component Synergy for Sample-Efficient Continuous Control an encoder. The encoder loss 퐿 푒푛푐 is formulated as: 퐿 푒푛푐 = 휆 1 · 퐿 푟푒푤 + 휆 2 · 퐿 푑푦푛 + 휆 3 · 퐿 푡푒푟푚 (1) where퐿 푟푒푤 ,퐿 푑푦푛 ,퐿 푡푒푟푚 are the losses corresponding to the reward, the dynamics, and the terminal state prediction tasks, respectively. The terms휆 1 ,휆 2 and휆 3 are weighting factors that balance the contribution of each auxiliary task. Furthermore, this representation learning is decoupled from downstream RL training and updated periodically to ensure a stable feature space. We adopt this MR.Q-style learning process due to its solid theoretical grounding and its model-based formulation, which produces a more structured and optimization-friendly latent space. For detailed implementation, refer to Appendix C.3. 2.3. Optimization Stability (OS) for RL In recent years, an increasing body of work has focused on the network pathologies encountered in deep reinforcement learning (DRL). Among the most severe of these pathologies are capacity collapse and plasticity loss [22]. Plasticity loss [16] refers to a phenomenon in which a neural network, trained on a sequence of non-stationary distributions, progressively loses its ability to adapt to new data. This issue is particularly problematic in the context of RL, where the agent’s evolving policy continuously shifts the data distribution stored in the replay buffer. If the network becomes excessively specialized to earlier experiences (referred to as "primacy bias" [25]), it may fail to incorporate more optimal behaviors that emerge later in the training process. Recent research suggests that certain network architectures, such as layer normalization and residual connections, can be particularly helpful in mitigating these pathologies [24]. In this regard, SimBa [20] has conducted extensive analysis and experiments, proposing a network architecture that is both simple and effective. SimBa [20] addresses optimization stability by amplifying simplicity bias into the architecture of deep RL models. By constraining the network’s architecture to favor simpler representations, SimBa reduces the risk of overfitting to noise in the training data. Given the empirically proven effectiveness of SimBa across various experimental settings, we adopt its architecture as the representative technique forOSin our study. Detailed architectural specifications are provided in Appendix C.2, while the hyperparameter configurations are listed in Table 5. 2.4. Experience Replay (ER) for RL Experience replay is a cornerstone of off-policy RL, allowing an agent to store and reuse past experiences. The idea is to sample transitions from a buffer풟to break the temporal correlation of consecutive states and actions, which reduces the variance in updates and improves the stability of training [6]. One challenge with experience replay is ensuring that the sampling process is efficient. Uniform sampling of experiences may not fully exploit the most informative transitions. Instead, prioritizing samples based on their potential can help improve data efficiency. However, TD-error-based prioritization [7,19,26] often suffers from "wasted" gradients: large TD errors may arise from stochasticity or irreducible noise, causing the agent to repeatedly sample transitions that are effectively unlearnable [29]. EfficientER should therefore distinguish genuinely informative samples from the replay buffer. ReLo [29] introduces the concept of Reducible Loss (ReLo), which prioritizes data points that most reduce the model’s generalization loss. The key idea is to train the model on a subset of the data and compare its performance with a hold-out model trained without the current data point. The Reducible Loss퐿 푟 for a 4 Beyond Isolation: Unlocking Reinforcement Learning Component Synergy for Sample-Efficient Continuous Control data point 푥 푖 is defined as: 퐿 푟 = Loss(ˆ푦|푥,휃)− Loss(ˆ푦|푥,휃 ℎ )(2) whereˆ푦is the predicted output,휃are the parameters of the main model, and휃 ℎ are the parameters of the hold-out model. ReLo down-weights unlearnable transitions, thereby enhancing the sample efficiency. Following a comparative analysis of several experience replay techniques, including PER [26], LAP [7], LaBER [19], and ReLo [29] (see Appendix E), we adopt ReLo as the primary representative technique for ER in our subsequent experiments and analysis. 3. Investigation In this section, we conduct a systematic investigation on continuous-control tasks to examine the interac- tions among various sample-efficiency-enhancing techniques when integrated into a framework. Our analysis begins by investigating the performance enhancements brought byOSto other components. Subsequently, we transition to the integration ofR, focusing on how to unlock their full potential. Finally, we present the dilemma of incorporating ER and propose targeted solutions to mitigate this issue. Experiment Setup To enable a comprehensive and reliable empirical study and to strengthen the robustness of our conclusions, we evaluate on a diverse set of continuous-control tasks drawn from four widely adopted and authoritative benchmarks: DeepMind Control suite (DMC) [31], HumanoidBench (HB) [27], Myosuite (Myo) [32], and ManiSkill2 (MS) [10]. Rather than exhaustively covering all tasks in these environments, we focuse on tasks that remain challenging. Specifically, we consider a total of 18 tasks, consisting of 9 locomotion tasks and 9 manipulation tasks, covering a broad range of control scenarios, as visually depicted in Figure 1. Additional environment-specific details and task configurations are provided in Appendix D. We conduct every experiment across 8 random seeds. We adopt Soft Actor-Critic (SAC) [12] as the base algorithm, upon which various sample-efficiency- enhancing techniques are integrated and evaluated. Notably, we utilize a fixed set of hyperparameters across all experiments in this paper without any specific tuning. The complete hyperparameter settings are detailed in Appendix B. Following the recommendations of Agarwal et al.[1], we report the Interquar- tile Mean (IQM) with 95% stratified bootstrap confidence intervals to provide a robust assessment of performance across heterogeneous environments. The IQM is computed based on the maximum return or maximum success rate achieved during the training process. 3.1. Optimization Stability Enables Synergy Across Sample-Efficiency Modules 0.10.30.5 +ER+OS +ER +R+OS +R +OS SAC (base) Locomotion (9 Tasks) 0.10.30.5 Manipulation (9 Tasks) Normalized Score Figure 2: Performance Gains of Optimization Sta- bility. The results demonstrate that the Simba back- bone consistently enhances performance. To investigate the role ofOSin coordination with other sample-efficiency-enhancing techniques, we integrate the Simba network architecture with various modules to examine whether a stable back- bone is a prerequisite for effective component syn- ergy. Specifically, we focus on three comparisons that isolate the incremental effect of Simba under different settings: (i) SAC vs. SAC +OS; (i) SAC +Rvs. SAC +R+OS; and (i) SAC +ERvs. SAC + ER + OS. 5 Beyond Isolation: Unlocking Reinforcement Learning Component Synergy for Sample-Efficient Continuous Control As illustrated in Figure 2, the Simba backbone provides a consistent performance boost across all evaluated configurations. In all cases, we observe a domain-agnostic synergistic amplification effect: when integrated with other sample-efficiency-enhancing techniques, Simba consistently yields a positive synergy. The results in case (i) are noteworthy, as they reveal that introducingRin isolation yields a negative impact, particularly on locomotion tasks. However, when integrated withOS, performance improves drastically, surpassing the individual capabilities of either component by a significant margin. This not only demonstrates the generalizable benefits ofOSacross diverse RL settings, but further highlights how proper component integration can trigger super-additive, "1+1>2" synergistic gains. Takeaway: Our results demonstrate that integrating Simba with various modules consistently yields performance gains. This underlines a broader architectural insight: a stable optimization network backbone is essential for fostering effective synergy among components. 3.2. Stabilizing Information Flow to Strengthen Component Synergy Previous results demonstrate that the naive integration of Simba with model-based representation already yields notable performance gains. We argue that this empirical success can be partially attributed to the inherent advantages of residual connections, where shortcut connections fundamentally stabilize micro- level information flow within neural architectures. Motivated by this insight, we extend this philosophy to the macro-level and introduce a Stable Model-based Representation, denoted asR⋆(pronounced as ’R-StaR’). The core design involves a residual-style feature fusion (Figure 3): learned latent embeddings are concatenated with original features before entering downstream policy and value networks. This architectural "information bypass" ensures that the actor and critic maintain access to stable signals. We empirically evaluate the necessity of this design by comparing SAC +OS+Ragainst its stabilized counterpart, SAC +OS+R⋆. As shown in Figure 4, theR⋆variant significantly outperforms the naive combination, particularly in locomotion scenarios. Specifically, it achieves a performance boost of 24.72% in locomotion tasks and 11.51% in manipulation tasks compared to the naive combination components. These results underscore that while Simba provides a powerful backbone, the robustness of the information flow is crucial for maximizing the synergistic effects of model-based representation. s t ZsZsaActorCritic a t Q t 、 s t a t Figure 3: Information Flow ofR⋆. Our scheme concatenates raw state푠 푡 and action푎 푡 with learned features via residual-like connections. 0.300.450.60 +R +R SAC+OS (base) Locomotion (9 Tasks) 0.30.40.50.6 Manipulation (9 Tasks) Normalized Score Figure 4: Comparison of Representation Schemes. We compare the baseline, naive integration (+R), and stabilized variant (+R⋆). Takeaway:R⋆consistently pushing the performance boundaries of SAC+OS+Rsuggests that the stable information flow via residual-style bypasses is crucial for effective inter-modular synergy. 3.3. A Dilemma of Experience Replay in Coordinated Integration While non-uniform experience replay mechanisms are well-established for boosting sample efficiency in isolated vanilla settings, it remains an open and critical question how these sampling priorities behave 6 Beyond Isolation: Unlocking Reinforcement Learning Component Synergy for Sample-Efficient Continuous Control when integrated into a more complex system. To this end, we introduce prioritized replay denoted as ER(P) into SAC + OS + R⋆, yielding a variant referred to as SAC + OS + R⋆ + ER(P). Contrary to expectations, Figure 5 shows that this configuration underperforms SAC +OS+R⋆. This degradation indicates a counteractive effect between prioritized experience replay and the previously integrated components. These results highlight an important insight: naively stacking multiple sample- efficiency-enhancing techniques does not guarantee additive improvements. We conjecture that, during the early stage of training, when representation learning and value estimation are still evolving, priority signals may become less reliable and introduce undesirable sampling biases. 0.540.570.600.63 +ER(U2P) +ER(P) SAC+OS+R (base) Locomotion (9 Tasks) 0.480.520.560.60 Manipulation (9 Tasks) Normalized Score Figure 5: Comparison of Experience Replay Schemes. We compare SAC+OS+R⋆integrated with different experience replay variants, including Prior- itized replay (ER(P)) and Uniform-to-Prioritized replay (ER(U2P)). Based on this consideration, we propose Uniform- to-Prioritized replay (U2P). Instead of priori- tized sampling throughout training, U2P sched- ules the exponent 훼, defined as: 훼 푡 = ⎧ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎩ 0.0푡<푡 start 훼 퐹 (︂ 1− (︁ 1− 푡−푡 start 푡 end −푡 start )︁ 2 )︂ 푡 start ≤ 푡≤ 푡 end 훼 퐹 푡>푡 end (3) Here, 훼 controls the degree of non-uniform sam- pling:푃(푖) = 푝 훼 푖 ∑︀ 푘 푝 훼 푘 . Detailed hyperparameters are provided in Table 6. This design gradually increases the influence of prioritized replay, allowing the replay distribution to adapt progressively alongside the evolving learning process. As illustrated in Figure 5, incorporating U2P successfully mitigates the conflict between ER with R⋆ and OS. Takeaway: Naively stackingERon top ofR⋆andOScan be counterproductive. This highlights the importance of coordinating replay strategies with other components. A scheduled prioritization strategy, U2P, improves compatibility and enables more effective component integration. Summary of Investigation Taking optimization stability, model-based representation, and experience replay as examples, this investigation presents the intricate interplay that emerges when these DRL components are jointly integrated. Crucially, we reveal that enhancements demonstrating remarkable individual success on vanilla SAC exhibit divergent behaviors when integrated. Specifically, we show that optimization stability is not merely beneficial in isolation, but serves as a foundational enabler that allows other components to synergize effectively. Yet micro-level network stability alone is insufficient; scaling this principle to a macro-level architecture by ensuring a reliable information flow for represen- tation learning further catalyzes algorithmic gains. Conversely, advanced experience replay, despite its well-known efficacy in vanilla settings, presents conflicts when naively superimposed onto a complex configuration. Together, these results highlight that achieving strong sample efficiency is a systems problem: gains emerge from coherent co-design. 4. Synergistic Framework Building on the investigation, we distill a set of general principles aimed at effectively coordinating diverse sample-efficiency-enhancing techniques. We then consolidate these principles into a framework, 7 Beyond Isolation: Unlocking Reinforcement Learning Component Synergy for Sample-Efficient Continuous Control termedROSER(Model-basedRepresentation,OptimizationStability andExperienceReplay). Through principled coordination,ROSERpromotes positive effectiveness among these components, enabling them to amplify performance and robustness in challenging continuous-control tasks. 4.1. Design Principles for Synergistic Coordination This subsection introduces the design principles. ∙Principle 1: Optimization Stability Serves as the Foundational Backbone Our investigation indicates that optimization stability plays a foundational role in integrated systems. A stable optimization backbone establishes favorable learning dynamics that allow other components to express their potential. We therefore treat optimization stability as the structural backbone of the framework. ∙Principle 2: Synergy Requires Stable Information Flow Residual connections are well-established as a stabilizing mechanism within neural networks. We demonstrate that this philosophy extends beyond the intra-network level, showing that explicit bypass connections can stabilize information flow and yield substantial gains in component synergy. We believe this principle is not confined to our specific design, and encourage the community to further explore residual-style bypasses as a general design guideline. ∙Principle 3: Synergy Favors Robust Coordination over Aggressive Individual Performance. In integrated settings, overly aggressive prioritized experience replay can disrupt coordination among components and degrade performance. In contrast, more conservative and progressive designs tend to preserve compatibility. This highlights a key principle for synergistic integration: prioritize robustness of coordination over maximizing the standalone strength of any individual component. 4.2. Instantiating the ROSER Framework Guided by the design principles, we instantiateROSER. First, building on first design principle, we integrate the Simba network architecture across all functional modules of the agent, including encoder, actor, and critic (Appendix C). Second, to fully harness the potential of representation learning within the integrated system, we instantiate the residual-style information bypass philosophy at the macro-level viaR⋆, ensuring that downstream policy and value networks retain direct access to stable input signals. Finally, when incorporating experience replay, we employ the U2P replay strategy to delay prioritization until learning signals become sufficiently reliable, ensuring that experience replay acts as a harmonizing force rather than a source of instability. Our synergistic framework is algorithm-agnostic and can be seamlessly integrated into a wide range of off-policy continuous-control algorithms, DDPG and SAC. In this work, we primarily instantiateROSER on top of SAC, while additional results on DDPG are provided in Appendix A. 5. Experiments This section evaluates the effectiveness and applicability of the proposedROSERframework on a diverse set of continuous-control tasks. Rather than establishingROSERas a state-of-the-art algorithm competing against the strongest available baselines, our core objective is to demonstrate that principled coordination of sample-efficiency-enhancing components can yield substantially improved sample efficiency. To this end, we first benchmarkROSERagainst vanilla SAC, controlled partial variants, and naive stacking, 8 Beyond Isolation: Unlocking Reinforcement Learning Component Synergy for Sample-Efficient Continuous Control assessing its sample efficiency and robustness across diverse environments. We then conduct a targeted analysis to examine the source of performance gains from R⋆ and U2P. 5.1. Evaluation of ROSER Experimental Setup The task configurations and environments used here are consistent with those em- ployed in Section 3. We compareROSERagainst the vanilla SAC algorithm and several partially enhanced variants that incorporate only a subset of its components: (i) SAC with model-based representation (SAC+R), (i) SAC with optimization stability (SAC+OS), (i) SAC with enhanced experience replay (SAC+ER), and (iv) Naive Stack framework, which naively stacks all above components. We report results across 8 stochastic seeds for each configuration. 0.00.20.40.60.81.0 Normalized Score ( ) 0.00 0.25 0.50 0.75 1.00 Fraction of runs with score > Locomotion (9 Tasks) 0.00.20.40.60.81.0 Normalized Score ( ) Manipulation (9 Tasks) SACSAC+RSAC+OSSAC+ERNaive StackROSER Figure 6: Comparison of Efficiency and Reliability. Fraction of runs achieving normalized score> 휏; 95% CI. Curves closer to top-right indicate superior performance and reliability. To rigorously evaluate the proposed algorithms, we adopt the statistical framework recommended by Agarwal et al.[1]. We first aggregate results across all tasks using performance profiles (Fig- ure 6) to provide a statistically robust comparison of overall efficiency and reliability. To further examine fine-grained behavior, we provide de- tailed learning curves (Figure 7) showcasing per- formance dynamics on individual tasks. Together, these metrics offer a comprehensive assessment of algorithmic behavior and performance. Superior Sample Efficiency and Robustness As shown in Figure 6,ROSERachieves a significant Pareto improvement over the vanilla SAC and its augmented variants. In the initial threshold range (휏 < 0.2),ROSERmaintains a run fraction near1.0, whereas baseline SAC and SAC+Rshow immediate degradation. This indicates thatROSEReffectively mitigates the risk of catastrophic failure during training. In the more challenging Manipulation tasks,ROSERretains a higher success density at high-performance thresholds (휏 > 0.8) compared to other variant. These results highlight thatROSERnot only delivers better performance but also exhibits greater robustness across different tasks. The Necessity of Principled Coordination A critical observation is the performance gap betweenROSER and the naive stack baseline:ROSERdelivers a pronounced improvement on locomotion tasks and a modest gain on manipulation tasks, most evident in the higher휏regime. This provides empirical evidence that principled coordination is crucial for unlocking the full potential of RL systems. Generalization Across Environments These conclusions are further corroborated by the learning curves in Figure 7. Empirically,ROSERdistinguishes itself through both rapid initial convergence and a higher final performance plateau across a diverse task suite. While single-component variants (e.g., SAC+R) show inconsistent improvements depending on the task,ROSERprovides a universally stable performance boost, highlighting that our framework is environment-agnostic and robust, making it a general-purpose solution for sample-efficient RL. 9 Beyond Isolation: Unlocking Reinforcement Learning Component Synergy for Sample-Efficient Continuous Control 0.00 0.25 0.50 0.75 1.00 Normalized Score Dog Run (DMC)Humanoid Run (DMC)Balance Simple (HB)Bookshelf Simple (HB)Basketball (HB)Door (HB) 0.00 0.25 0.50 0.75 1.00 Normalized Score Balance Hard (HB)Run (HB)Hurdle (HB)Spoon (HB)Key Turn Hard (Myo)Pen Twirl Hard (Myo) 0.000.250.500.751.00 Steps (M) 0.00 0.25 0.50 0.75 1.00 Normalized Score Reach (HB) 0.000.250.500.751.00 Steps (M) Sit Hard (HB) 0.000.250.500.751.00 Steps (M) Slide (HB) 0.000.250.500.751.00 Steps (M) Pick Cube (MS) 0.000.250.500.751.00 Steps (M) Turn Faucet (MS) 0.000.250.500.751.00 Steps (M) Lift Cube (MS) SACSAC+RSAC+OSSAC+ERNaive StackROSER Figure 7: Comparison of Performance Across Different Tasks. We plot the learning curve ofROSER and baseline algorithms, categorized into locomotion (left 3×3 panels) and manipulation (right 3×3 panels). To better compare the performance differences, we normalize the scores by rounding up the highest score in each task to the nearest hundred, resulting in the Normalized Score. Results without normalization can be found in Appendix F. Shaded regions represent the 95% confidence interval. 5.2. Analysis In this section, we analyze the efficacy ofR⋆and U2P. Our objective is to determine whether their performance gains stem from localized module improvements or from their capacity to mitigate emergent challenges within multi-module integration. Experimental Setup To disentangle standalone benefits from integration-driven synergies, we evaluate the effectiveness ofR⋆and U2P at two levels of complexity: (i) Isolated Evaluation: We integrate the component into a vanilla SAC baseline (e.g., SAC+ER(P) vs. SAC+ER(U2P)) to establish a perfor- mance floor in simplified contexts; (i) Synergy Evaluation: We perform studies within the fullROSER framework (e.g.,ROSER(P) vs.ROSER(U2P)) to quantify their contributions under complex, integrated conditions. By contrasting their standalone gains with contributions under synergy contexts, we charac- terize the mechanism underlying the observed improvements. We evaluate these configurations across a representative subset of 6 tasks: dog-run, humanoid-run, h1-balance-simple, turnfaucet, key-turn-hard, pen-twirl-hard, running 5 seeds for each experiment. Table 1: Analysis on Component Synergy. IQM with 95% stratified bootstrap CI (in brackets). RepresentationExperience Replay Framework R R⋆ ER(P) ER(U2P) Vanilla SAC 0.175 [0.143, 0.209] 0.160 [0.098, 0.273] 0.250 [0.182, 0.295] 0.165 [0.128, 0.229] ROSER (Ours) 0.514 [0.482, 0.541] 0.657 [0.524, 0.754] 0.671 [0.604, 0.723] 0.713 [0.671, 0.748] 10 Beyond Isolation: Unlocking Reinforcement Learning Component Synergy for Sample-Efficient Continuous Control Analysis on Stable Model-based Representation (R⋆). As illustrated in Table 1,R⋆degrades performance (-0.015) in the simplified setting, yet yields a clear improvement (+0.143) when integrated into the ROSERframework. This reversal indicates thatR⋆acts as a system-level stabilizer addressing joint optimization coupling, rather than a standalone optimizer. Analysis on Uniform-to-Prioritized (U2P). As illustrated in Table 1, while U2P degrades performance (-0.085) in the standalone SAC-ERbaseline, it yields a performance boost (+0.042) within theROSER framework. This discrepancy suggests that the efficacy of U2P transcends simple optimization of prioritized experience replay. Instead, its primary value lies in buffer-level regularization, which mitigates the compound non-stationarity emergent from multi-module integration. Summary of Experimental Findings Experiments demonstrate thatROSERconsistently improves sample efficiency across a diverse set of continuous-control tasks. These results validate the design principles derived from our initial investigation, confirming that systematic coordination is important to performance gains. Notably, unlikeROSER, the benefits of isolated modules are highly sensitive to task characteristics, suggesting that individual sample-efficiency-enhancing techniques may have inherent limitations, making it difficult for them to address tasks with specific challenges or requirements. Overall, these findings underscore the value of a holistic, system-level perspective on sample-efficiency improvements. 6. Conclusion In this paper, we investigated the synergistic potential of combining diverse sample-efficiency-enhancing techniques in Reinforcement Learning. We reveal that individually successful enhancements exhibit surprisingly divergent behaviors when jointly integrated, implying that simply stacking these modules does not guarantee additive performance gains. Rather, their interactions are complex and demand principled coordination. Through extensive empirical analysis, we demonstrated that the robustness of individual modules and their mutual stability act as the dominant factors in determining the overall sample efficiency, leading us to formalize three design principles for synergistic coordination. To evaluate the rationality of our summarized design principles, we instantiate theROSERframework. By incorporating a Stable Model-based Representation (R⋆) and an Uniform-to-Prioritized Replay (U2P), ROSEReffectively mitigates non-stationarity and unlocks the joint potential of its constituent parts. Our results show that this integrated framework not only stabilizes the learning trajectory but also achieves superior performance in both locomotion and manipulation tasks. Ultimately, this work highlights the importance of analyzing and improving RL sample efficiency from a synergistic, multi-module perspective, which is key to developing more efficient autonomous agents. Limitations. While our study shows the power of synergistic design in improving sample efficiency, several avenues remain for future exploration. First, although our empirical evidence shows that in- dividual enhancement modules often exhibit significant task-dependency, a systematic taxonomy of which environment features, such as reward density or state space complexity, favor specific techniques is yet to be established. Second, our integration strategies, such asR⋆and the U2P, are primarily empirically-grounded; future research could focus on discovering more theoretically optimal coordination mechanisms. Furthermore, our study focused on three specific components, yet other crucial factors like exploration could be integrated to further expand the synergistic potential. Finally, although our findings are robust under the SAC baseline, extending this investigation to other off-policy algorithms, as well as more diverse real-world benchmarks, will be essential to establish broader generalizability. 11 Beyond Isolation: Unlocking Reinforcement Learning Component Synergy for Sample-Efficient Continuous Control References [1]Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in Neural Information Processing Systems, 2021. 5, 9 [2]Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesensky, Bin Bao, Peter Bell, David Berard, Evgeni Burovski, Geeta Chauhan, Anjali Chourdia, Will Constable, Alban Desmaison, Zachary DeVito, Elias Ellison, Will Feng, Jiong Gong, Michael Gschwind, Brian Hirsh, Sherlock Huang, Kshiteej Kalambarkar, Laurent Kirsch, Michael Lazos, Mario Lezcano, Yanbo Liang, Jason Liang, Yinghai Lu, CK Luk, Bert Maher, Yunjie Pan, Christian Puhrsch, Matthias Reso, Mark Saroufim, Marcos Yukio Siraichi, Helen Suk, Michael Suo, Phil Tillet, Eikan Wang, Xiaodong Wang, William Wen, Shunting Zhang, Xu Zhao, Keren Zhou, Richard Zou, Ajit Mathews, Gregory Chanan, Peng Wu, and Soumith Chintala. PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation. In 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (ASPLOS ’24). ACM, April 2024. doi: 10.1145/3620665.3640366. URLhttps://docs.pytorch.org/ assets/pytorch2-2.pdf. 19 [3]Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives, 2014. URL https://arxiv.org/abs/1206.5538. 3 [4] Johan Samir Obando Ceron and Pablo Samuel Castro. Revisiting rainbow: Promoting more insightful and inclusive deep reinforcement learning research. In International Conference on Machine Learning, pages 1373–1383. PMLR, 2021. 3 [5] Tyler Clark, Mark Towers, Christine Evers, and Jonathon Hare. Beyond the rainbow: High perfor- mance deep reinforcement learning on a desktop PC. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=V3KXsUFw8D. 3 [6]William Fedus, Prajit Ramachandran, Rishabh Agarwal, Yoshua Bengio, Hugo Larochelle, Mark Rowland, and Will Dabney. Revisiting fundamentals of experience replay, 2020. URLhttps: //arxiv.org/abs/2007.06700. 4 [7]Scott Fujimoto, David Meger, and Doina Precup. An equivalence between loss functions and non-uniform sampling in experience replay. Advances in Neural Information Processing Systems, 33, 2020. 1, 4, 5, 19, 24 [8]Scott Fujimoto, Wei-Di Chang, Edward J. Smith, Shixiang Shane Gu, Doina Precup, and David Meger. For SALE: State-action representation learning for deep reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URLhttps://openreview.net/ forum?id=xZvGrzRq17. 1 [9]Scott Fujimoto, Pierluca D’Oro, Amy Zhang, Yuandong Tian, and Michael Rabbat. Towards general- purpose model-free reinforcement learning. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=R1hIXdST22. 1, 3, 18, 21 [10] Jiayuan Gu, Fanbo Xiang, Xuanlin Li, Zhan Ling, Xiqiaing Liu, Tongzhou Mu, Yihe Tang, Stone Tao, Xinyue Wei, Yunchao Yao, Xiaodi Yuan, Pengwei Xie, Zhiao Huang, Rui Chen, and Hao Su. 12 Beyond Isolation: Unlocking Reinforcement Learning Component Synergy for Sample-Efficient Continuous Control Maniskill2: A unified benchmark for generalizable manipulation skills. In International Conference on Learning Representations, 2023. 5, 23 [11]David Ha and Jürgen Schmidhuber. World models. 2018. doi: 10.5281/ZENODO.1207631. URL https://zenodo.org/record/1207631. 3 [12] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861–1870. Pmlr, 2018. 5 [13]Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=S1lOTC4tDS. 3 [14]Nicklas Hansen, Hao Su, and Xiaolong Wang. Td-mpc2: Scalable, robust world models for continuous control. In International Conference on Learning Representations (ICLR), 2024. 17 [15]Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018. 3 [16]Timo Klein, Lukas Miklautz, Kevin Sidak, Claudia Plant, and Sebastian Tschiatschek. Plasticity loss in deep reinforcement learning: A survey, 2024. URLhttps://arxiv.org/abs/2411.04832. 4 [17] Yilun Kong, Guozheng Ma, Qi Zhao, Haoyu Wang, Li Shen, Xueqian Wang, and Dacheng Tao. Mastering massive multi-task reinforcement learning via mixture-of-expert decision transformer. In ICLR 2025 Workshop on Modularity for Collaborative, Decentralized, and Continual Deep Learning, 2025. URL https://openreview.net/forum?id=YgR8U5DSj9. 1 [18] Yilun Kong, Hangyu Mao, Zhao Qi, Bin Zhang, Jingqing Ruan, Li Shen, Yongzhe Chang, Xueqian Wang, Rui Zhao, and Dacheng Tao. QPO: Query-dependent prompt optimization via multi-loop offline reinforcement learning. Transactions on Machine Learning Research, 2025. ISSN 2835-8856. URL https://openreview.net/forum?id=bqMJToTkvT. 1 [19] Thibault Lahire, Matthieu Geist, and Emmanuel Rachelson. Large batch experience replay, 2021. 1, 4, 5, 24 [20]Hojoon Lee, Dongyoon Hwang, Donghu Kim, Hyunseung Kim, Jun Jet Tai, Kaushik Subramanian, Peter R. Wurman, Jaegul Choo, Peter Stone, and Takuma Seno. Simba: Simplicity bias for scaling up parameters in deep reinforcement learning. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=jXLiDKsuDo. 1, 4, 18, 19, 20 [21]Hojoon Lee, Youngdo Lee, Takuma Seno, Donghu Kim, Peter Stone, and Jaegul Choo. Hyperspherical normalization for scalable deep reinforcement learning. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=kfYxyvCYQ4. 1 13 Beyond Isolation: Unlocking Reinforcement Learning Component Synergy for Sample-Efficient Continuous Control [22]Guozheng Ma, Lu Li, Zilin Wang, Haoyu Wang, Shengchao Hu, Leszek Rutkowski, and Dacheng Tao. Rethinking the role of dynamic sparse training for scalable deep reinforcement learning, 2025. URL https://arxiv.org/abs/2510.12096. 4 [23]Srinath V. Mahankali, Zhang-Wei Hong, Ayush Sekhari, Alexander Rakhlin, and Pulkit Agrawal. Random latent exploration for deep reinforcement learning. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=Y9qzwNlKVU. 1 [24]Michal Nauman, Mateusz Ostaszewski, Krzysztof Jankowski, Piotr Miłoś, and Marek Cygan. Bigger, regularized, optimistic: scaling for compute and sample-efficient continuous control. In Advances in Neural Information Processing Systems, 2024. URLhttps://arxiv.org/pdf/2405.16158. 1, 4 [25]Evgenii Nikishin, Max Schwarzer, Pierluca D’Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In International Conference on Machine Learning. PMLR, 2022. 4 [26] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint arXiv:1511.05952, 2015. 1, 4, 5, 24 [27]Carmelo Sferrazza, Dun-Ming Huang, Xingyu Lin, Youngwoon Lee, and Pieter Abbeel. Humanoid- bench: Simulated humanoid benchmark for whole-body locomotion and manipulation. arXiv Preprint arxiv:2403.10506, 2024. 5, 23 [28] Aravind Srinivas, Michael Laskin, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning, 2020. URL https://arxiv.org/abs/2004.04136. 3 [29] Shivakanth Sujit, Somjit Nath, Pedro H. M. Braga, and Samira Ebrahimi Kahou. Prioritizing samples in reinforcement learning with reducible loss. arXiv preprint arXiv: Arxiv-2208.10483, 2022. 4, 5, 19, 24 [30]Bhavya Sukhija, Stelian Coros, Andreas Krause, Pieter Abbeel, and Carmelo Sferrazza. MaxinfoRL: Boosting exploration in reinforcement learning through information gain maximization. In The Thirteenth International Conference on Learning Representations, 2025. URLhttps://openreview. net/forum?id=R4q3cY3kQf. 1 [31]Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690, 2018. 5, 23 [32]Caggiano Vittorio, Wang Huawei, Durandau Guillaume, Sartori Massimo, and Kumar Vikash. Myosuite – a contact-rich simulation suite for musculoskeletal motor control.https://github. com/myohub/myosuite, 2022. URL https://arxiv.org/abs/2205.13600. 5, 23 [33] Yang Yu. Towards sample efficient reinforcement learning. In IJCAI, pages 5739–5743, 2018. 1 14 Beyond Isolation: Unlocking Reinforcement Learning Component Synergy for Sample-Efficient Continuous Control Appendix A ROSER on DDPG16 B Hyperparameters17 C Architecture19 C.1 MLP Block . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 C.2 Simba Block . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 C.3 Encoder . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 D Environments23 E Analysis of ER24 F Complete Main Results25 15 Beyond Isolation: Unlocking Reinforcement Learning Component Synergy for Sample-Efficient Continuous Control A. ROSER on DDPG 0.0 0.5 1.0 Normalized Score Dog Run (DMC)Humanoid Run (DMC)Balance Simple (HB)Bookshelf Simple (HB) 0.00.51.0 Steps (M) 0.0 0.5 1.0 Normalized Score Hurdle (HB) 0.00.51.0 Steps (M) Key Turn Hard (Myo) 0.00.51.0 Steps (M) Pen Twirl Hard (Myo) 0.00.51.0 Steps (M) Turn Faucet (MS) DDPGDDPG-NAIVEDDPG-ROSER Figure 8: Learning curves. Validating the Generalizability of ROSER on the DDPG Algorithm. Table 2: Performance comparison of DDPG variants. Values represent Median, IQM, and Mean with 95% confidence intervals. LocomotionManipulation AlgorithmMedianIQMMeanMedianIQMMean DDPG (Vanilla)0.06830.06830.07380.20300.19130.3215 [0.0416, 0.0969] [0.0452, 0.0934] [0.0579, 0.0906] [0.0818, 0.3700] [0.0981, 0.3196] [0.2443, 0.4089] DDPG+Naive Stack 0.27410.30300.33020.62490.65490.6025 [0.2260, 0.3382] [0.2390, 0.3667] [0.2899, 0.3734] [0.4075, 0.8373] [0.4504, 0.8373] [0.4903, 0.7091] DDPG+ROSER0.38760.39640.44290.73770.75770.6589 [0.3132, 0.4735] [0.3412, 0.4650] [0.3954, 0.4898] [0.5577, 0.8616] [0.6131, 0.8616] [0.5686, 0.7223] To verify thatROSERis algorithm-agnostic, we conducted additional experiments using DDPG on 8 tasks (dog-run, humanoid-run, h1-balance-simple, h1-hurdle, h1-bookshelf-simple, turnfaucet, key-turn-hard and pen-twirl-hard) with 5 seeds. The results, shown in the Figure 8 and Table 2, confirm thatROSER improves DDPG performance over both the vanilla version and naive stacking. 16 Beyond Isolation: Unlocking Reinforcement Learning Component Synergy for Sample-Efficient Continuous Control B. Hyperparameters Table 3: Backbone hyperparameters. We provide a detailed list of the hyperparameters used in the backbone (SAC) and training setup. We automatically set the discount factor based on the heuristics from TD-MPC2 [14]. HyperparameterValue Critic block typeMLP Critic hidden dim512 Critic learning rate1e-4 Critic activation functionReLU Target critic momentum (휏)5e-3 Clipped Double QTrue Actor block typeMLP Actor hidden dim128 Actor learning rate1e-4 Actor activation functionReLU Initial temperature (훼 0 )1e-2 Temperature learning rate1e-4 Target entropy (ℋ * )|풜|/2 OptimizerAdamW Optimizer momentum (훽 1 , 훽 2 ) (0.9, 0.999) Weight decay (휆)1e-2 Discount (훾)Heuristic Replay ratio2 Multi-step returns horizon1 17 Beyond Isolation: Unlocking Reinforcement Learning Component Synergy for Sample-Efficient Continuous Control Table 4:Rhyperparameters. We provide a detailed list of the hyperparameters used in model-based representation. We follow the hyperparameters specified in the original MR.Q paper [9]. The encoder architecture is conditional: SimBa Residual (w/ OS) vs. MLP (w/o OS). HyperparameterValue z 푠 dim512 z 푠푎 dim256 z 푎 dim512 Encoder horizon5 Encoder block typeMLP/SimBa Residual Encoder num blocks1 Encoder hidden dim512 Encoder learning rate1e-4 Encoder activation functionELU Encoder update frequency250 OptimizerAdamW Optimizer momentum (훽 1 , 훽 2 )(0.9, 0.999) Weight decay (휆)1e-4 Dynamics loss weight1 Reward loss weight0.1 Terminal loss weight0.1 Reward bins65 Reward range[−10, 10] Replay ratio1 Table 5:OShyperparameters. We provide a detailed list of the hyperparameters used in optimization stability. We follow the hyperparameters specified in the original SimBa paper [20]. HyperparameterValue Critic block typeSimBa Residual Critic num blocks2 Critic hidden dim512 Critic learning rate1e-4 Critic activation functionReLU Target critic momentum (휏)5e-3 Clipped Double QTrue Actor block typeSimBa Residual Actor num blocks1 Actor hidden dim128 Actor learning rate1e-4 Actor activation functionReLU 18 Beyond Isolation: Unlocking Reinforcement Learning Component Synergy for Sample-Efficient Continuous Control Table 6:ERhyperparameters. We provide a detailed list of the hyperparameters used in experience replay. We follow the hyperparameters specified in the original ReLo paper [29]. Following [7], when in the uniform sampling state (i.e.,훼 = 0), the critic utilizes mean squared error (MSE) loss. In contrast, when in a non-uniform sampling state (i.e., 훼 > 0), the critic adopts Huber loss. HyperparameterValue Buffer size1M Batch size256 ReLo Clipping Offset 휖0.01 훼 퐹 0.4 푡 start 0.2 * Total env. steps 푡 end 0.8 * Total env. steps Critic LossHuber/MSE C. Architecture This section outlines the networks employed in our paper, including MLP Block, Simba Block, Encoder, implemented using PyTorch2 [2]. The implementation of the Critic and Actor follows [20], therefore, we do not provide further details here. C.1. MLP Block The MLP Block is a basic fully connected neural network module consisting of two linear layers with non-linear activation function applied behind them. 1 import math 2 import torch 3 import torch.n as n 4 5 class MLPBlock(n.Module): 6 def __init__( 7 self , 8 input_dim: int , 9 hidden_dim: int , 10 dtype: torch.dtype , 11 activ=’ReLU’ 12 ): 13 super().__init__ () 14 15 self.fc1 = n.Linear(input_dim , hidden_dim , dtype=dtype) 16 self.fc2 = n.Linear(hidden_dim , hidden_dim , dtype=dtype) 17 self.activ1 = getattr(n , activ)() 18 self.activ2 = getattr(n , activ)() 19 20 orthogonal_init_(self.fc1 , gain=math.sqrt(2)) 21 orthogonal_init_(self.fc2 , gain=math.sqrt(2)) 22 23 def forward(self , x: torch.Tensor) -> torch.Tensor: 24 x = self.fc1(x) 25 x = self.activ1(x) 26 x = self.fc2(x) 27 x = self.activ2(x) 19 Beyond Isolation: Unlocking Reinforcement Learning Component Synergy for Sample-Efficient Continuous Control 28 29 return x C.2. Simba Block The Simba Block consists of a series of residual blocks that allow for more complex transformations while retaining input information through shortcut connections. This block enables deeper networks by maintaining stable gradient flow during training. It also employs layer normalization for stability. Our implementation follows [20]. 1 class ResidualBlock(n.Module): 2 def __init__( 3 self , 4 hidden_dim: int , 5 dtype: torch.dtype , 6 activ=’ReLU’ 7 ): 8 super().__init__ () 9 10 self.layer_norm = n.LayerNorm(hidden_dim , dtype=dtype) 11 self.fc1 = n.Linear(hidden_dim , hidden_dim*4, dtype=dtype) 12 self.fc2 = n.Linear(hidden_dim*4, hidden_dim , dtype=dtype) 13 self.activ = getattr(n , activ)() 14 15 he_normal_init_(self.fc1) 16 he_normal_init_(self.fc2) 17 18 def forward(self , x: torch.Tensor) -> torch.Tensor: 19 res = x 20 x = self.layer_norm(x) 21 x = self.fc1(x) 22 x = self.activ(x) 23 x = self.fc2(x) 24 25 return res + x 26 27 class SimbaBlock(n.Module): 28 def __init__( 29 self , 30 num_blocks: int , 31 input_dim: int , 32 hidden_dim: int , 33 dtype: torch.dtype , 34 activ=’ReLU’ 35 ): 36 super().__init__ () 37 38 self.fc = n.Linear(input_dim , hidden_dim , dtype=dtype) 39 self.residual_blocks = n.ModuleList ([ 40 ResidualBlock(hidden_dim , dtype=dtype , activ=activ) for _ in range(num_blocks) 41 ]) 42 self.layer_norm = n.LayerNorm(hidden_dim , dtype=dtype) 43 44 orthogonal_init_(self.fc) 20 Beyond Isolation: Unlocking Reinforcement Learning Component Synergy for Sample-Efficient Continuous Control 45 46 def forward(self , x: torch.Tensor) -> torch.Tensor: 47 x = self.fc(x) 48 for block in self.residual_blocks: 49 x = block(x) 50 x = self.layer_norm(x) 51 52 return x C.3. Encoder The Encoder combines the state encoder푍 푠 and state-action encoder푍 푠푎 to map the input state and action data into latent representations, which are then used for further processing in the reinforcement learning pipeline. Our implementation follows [9]. 1 class ROSEREncoder(n.Module): 2 def __init__( 3 self , 4 state_dim: int , 5 hidden_dim: int , 6 action_dim: int , 7 zs_dim: int , 8 za_dim: int , 9 zsa_dim: int , 10 num_bins: int , 11 num_blocks: int , 12 dtype: torch.dtype , 13 activ=’ELU’ 14 ): 15 super().__init__ () 16 self.zs_dim = zs_dim 17 18 self.zs = Embedding( 19 num_blocks=num_blocks , 20 input_dim=state_dim , 21 hidden_dim=hidden_dim , 22 output_dim=zs_dim , 23 dtype=dtype , 24 activ=’ELU’ 25 ) 26 self.za = self.mlp_za 27 self.fc = n.Linear(action_dim , za_dim , dtype=dtype) 28 self.zsa = Embedding( 29 num_blocks=num_blocks , 30 input_dim=zs_dim+za_dim , 31 hidden_dim=hidden_dim , 32 output_dim=zsa_dim , 33 dtype=dtype , 34 activ=’ELU’ 35 ) 36 self.model = n.Linear(zsa_dim , num_bins + zs_dim + 1) 37 38 self.activ = getattr(n , activ)() 39 40 orthogonal_init_(self.model) 21 Beyond Isolation: Unlocking Reinforcement Learning Component Synergy for Sample-Efficient Continuous Control It is important to note that in the encoder implementation described above, both the state encoder푍 푠 and the state-action encoder푍 푠푎 are instances of the Embedding object. In the case of experiments solely onR, the embedding follows the original MR.Q. However, in scenarios whereRis combined withOS, the embedding is implemented as follows, inheriting the structure from the Simba Block. 1 class Embedding(SimbaBlock): 2 def __init__( 3 self , 4 num_blocks: int , 5 input_dim: int , 6 hidden_dim: int , 7 output_dim: int , 8 dtype: torch.dtype , 9 activ=’ELU’ 10 ): 11 super().__init__( 12 num_blocks=num_blocks , 13 input_dim=input_dim , 14 hidden_dim=hidden_dim , 15 dtype=dtype , 16 activ=activ 17 ) 18 self.head = n.Linear(hidden_dim , output_dim , dtype=dtype) 19 20 orthogonal_init_(self.head) 21 22 def forward(self , x: torch.Tensor) -> torch.Tensor: 23 x = super().forward(x) 24 x = self.head(x) 25 return x 22 Beyond Isolation: Unlocking Reinforcement Learning Component Synergy for Sample-Efficient Continuous Control D. Environments DeepMind Control suite [31] is a standardized benchmark test set for reinforcement learning developed by DeepMind based on the MuJoCo physics engine. It focuses on continuous control tasks, covering a wide range of motion capture and motion control scenarios from simple handstands to high-dimensional humanoid robots. This suite is renowned for its extremely high code quality, unified reward mechanism, and outstanding support for pixel-level observation (Visual RL), and is one of the most commonly used performance evaluation criteria in papers in fields such as robot control, representation learning, and model predictive control. We evaluate 2 DMC-Hard tasks, with details provided in Table 8. HumanoidBench [27] is a benchmark test specifically designed for humanoid robots, used to evaluate and compare their movement and task execution capabilities in complex environments. It challenges the robot’s performance in gait control, balance, object grasping and other aspects through a series of highly simulated tasks. This benchmark test utilizes a high-quality physical simulation environment to ensure the authenticity and diversity of tasks and actions, making it an important evaluation tool in fields such as reinforcement learning and robot control. We consider 11 tasks, with details provided in Table 9. MyoSuite [32] is an integrated platform that combines musculoskeletal simulation with artificial intelli- gence, aiming to provide standardized benchmarks for studying the evolution and learning of physiological and neuro-motor control by establishing highly accurate biomechanical models in physiology. This bench- mark suite not only covers multi-dimensional task challenges ranging from tendon transfer surgery modeling to complex object manipulation, but also bridges the gap between neural function recovery research and modern reinforcement learning algorithms. We focus on 2 tasks, with details provided in Table 10 Maniskill2 [10] is a high-quality benchmark for learning robot operation skills, aiming to promote the development of reinforcement learning and imitation learning fields. This benchmark offers a lot of challenging task categories, covering various robot operation tasks in real-world scenarios, such as grasping, manipulating, and assembling. ManiSkill2 supports over 2,000 different object models and more than 4 million frames of demonstration data, providing diverse and high-quality training and testing resources. Through deep integration with the SAPIEN engine, ManiSkill2 not only supports high-speed visual input but also features flexible environment configuration, allowing researchers to conduct efficient algorithm evaluations in standardized environments. The openness and efficiency of this benchmark make it an ideal testing platform for robot control and learning algorithms, effectively promoting the comparison and progress of various learning methods. We select 3 tasks, with details provided in Table 11 Table 11: Maniskill. We consider 3 Maniskill tasks, all of which are manipulation tasks, as shown below. TaskObservation dim Action dim Task Category LiftCube424mani PickCube514mani TurnFaucet407mani 23 Beyond Isolation: Unlocking Reinforcement Learning Component Synergy for Sample-Efficient Continuous Control Table 7: Environment details. We list the episode length, action repeat for each domain, total environment steps, and performance metrics used for our experiments. It is worth noting that, unless otherwise stated, for tasks that use Return as the performance metric, the normalized score is calculated using 1000 as the normalization factor (10000 for Reach in HumanoidBench, as the return for this task greatly exceeds 1000). DMC HumanoidBench MyoSuiteManiskill Episode length1,000500− 1, 000100200 Action repeat2222 Effective length500250− 50050100 Total env. steps1 M1 M1 M1 M Performance metric ReturnReturnSuccess Rate Success Rate Table 8: DMC. We consider 2 DMC-Hard tasks, both of which are locomotion tasks, as shown below. TaskObservation dim Action dim Task Category Dog Run22338loco Humanoid Run6724loco E. Analysis of ER 0.080.120.160.20 +RELO +LABER +LAP +PER SAC (base) Locomotion (9 Tasks) 0.320.400.48 Manipulation (9 Tasks) Normalized Score Figure 9: Comparision of Different Priority Experience Replay Design. We next study the impact of different experience replay strategies on sample efficiency. To this end, we compare several representative replay schemes, including Uniform Replay, Prioritized Experience Replay (PER [26]), Loss-Adjusted Prioritized experience replay (LAP [7]), Large Batch Experience Replay (LaBER [19]), and prioritized experience replay based on Reducible Loss (ReLo [29]). These methods span a range of prioritization mechanisms, from uniform sampling to TD-error-based and loss-aware criteria. We assess these methods on a representative set of four tasks: dog-run, h1-balance-simple, turnfaucet, and pen-twirl-hard, conducting 5 seeds per experiment. Results shown in Figure 9 reveal that other prioritized experience replay methods perform moderately on the locomotion task, with only a notable improvement on the manipulation task. In contrast, ReLo- based prioritized replay demonstrates significant improvement across both task types, consistently outperforming all other methods by a large margin. Overall, this analysis supports the rationale for using ReLo-based prioritized replay as the representative technique for ER in our investigation. 24 Beyond Isolation: Unlocking Reinforcement Learning Component Synergy for Sample-Efficient Continuous Control Table 9: HumanoidBench. We consider 11 HumanoidBench tasks, as shown below, with the first 7 tasks being locomotion tasks and the last 4 tasks being manipulation tasks. TaskObservation dim Action dim Task Category Balance Simple6419loco Balance Hard7719loco Run5119loco Hurdle5119loco Reach5719loco Sit Hard6419loco Slide5119loco Bookshelf Simple30861mani Basketball6419mani Door5519mani Spoon16761mani Table 10: MyoSuite. We consider 2 Myosuite tasks, both of which are manipulation tasks, as shown below. TaskObservation dim Action dim Task Category Key Turn Hard9339mani Pen Twirl Hard8339mani F. Complete Main Results 25 Beyond Isolation: Unlocking Reinforcement Learning Component Synergy for Sample-Efficient Continuous Control Table 12: Complete Main Results Average performance across eight independent seeds, evaluated at 500k training steps (equivalent to 1M environment steps with an action repeat of 2). The [bracketed values] represent a 95% bootstrap confidence interval. The aggregate mean, median and interquartile mean (IQM) are computed over the default reward. The best performance for each task is highlighted in bold. TaskSACSAC+RSAC+OSSAC+ERNaive Stack ROSER Dog Run49 [11, 98]12 [10, 15]558 [500, 599] 141 [82, 189]229 [149, 333] 632 [525, 723] Humanoid Run119 [84, 140] 115 [48, 172]171 [159, 183] 133 [120, 147] 406 [359, 449] 380 [315, 438] Run164 [54, 305]45 [27, 65]249 [189, 346]92 [59, 143]822 [818, 825] 822 [818, 825] Balance Simple195 [165, 232] 472 [349, 590] 266 [177, 370] 178 [164, 192] 772 [725, 815] 825 [809, 838] Balance Hard63 [59, 68]79 [67, 92]79 [72, 86]64 [60, 69]106 [93, 119]147 [128, 170] Sit Hard496 [268, 717] 35 [9, 85]613 [463, 766] 550 [356, 717] 841 [836, 846] 853 [846, 861] Reach3510 [3226, 3810]1104[957, 1273] 4402[3863, 4953]4435[4003, 4794]6168[4951, 7330]7141 [6574, 7664] Hurdle42 [17, 67]106 [74, 134]212 [199, 226]68 [49, 83]283 [220, 335] 340 [321, 357] Slide183 [125, 235] 389 [308, 476] 261 [213, 298] 182 [134, 223] 455 [418, 513] 458 [439, 478] Bookshelf Simple 99 [61, 145]52 [41, 64]718 [695, 739] 112 [90, 135]729 [609, 823] 812 [800, 824] Basketball80 [44, 128]28 [23, 33]163 [102, 228]97 [62, 133]353 [306, 400] 306 [222, 380] Door180 [119, 232] 172 [73, 272]296 [287, 308] 139 [74, 205]323 [310, 338] 341 [332, 348] Spoon19 [16, 22]23 [13, 41]380 [376, 384]26 [22, 29]350 [338, 363] 369 [351, 384] Key Turn Hard 0.25 [0.08, 0.50] 0.08 [0.01, 0.15] 0.50 [0.21, 0.79] 0.64 [0.43, 0.80] 0.86 [0.74, 0.98] 0.95 [0.88, 1.00] Pen Twirl Hard 0.68 [0.59, 0.76] 0.98 [0.94, 1.00] 0.94 [0.88, 0.99] 0.74 [0.65, 0.81] 1.00 [1.00, 1.00] 1.00 [1.00, 1.00] PickCube0 [0, 0]0 [0, 0]0 [0, 0]0.01 [0, 0.04]0.30 [0.25, 0.35] 0.29 [0.20, 0.38] TurnFaucet0.09 [0.06, 0.10] 0.04 [0.01, 0.08] 0.11 [0.08, 0.15] 0.19 [0.15, 0.23] 0.20 [0.16, 0.24] 0.20 [0.15, 0.25] LiftCube0.03 [0.00, 0.06] 0.65 [0.46, 0.83] 0.01 [0.00, 0.04] 0.18 [0.08, 0.26] 1.00 [1.00, 1.00] 1.00 [1.00, 1.00] Mean289 [266, 312] 146 [132, 161] 465 [432, 498] 345 [319, 369] 658 [589, 722] 746 [713, 776] Median72 [54, 91]40 [30, 55]230 [195, 247]95 [71, 112]337 [315, 346] 355 [338, 373] IQM60 [49, 72]37 [29, 47]203 [191, 216]76 [66, 87]310 [293, 329] 374 [357, 390] 26 Beyond Isolation: Unlocking Reinforcement Learning Component Synergy for Sample-Efficient Continuous Control 0.00.20.40.60.81.0 Steps (M) 0 200 400 600 Metric Value Dog Run (DMC) 0.00.20.40.60.81.0 Steps (M) 0 100 200 300 400 Humanoid Run (DMC) 0.00.20.40.60.81.0 Steps (M) 0 200 400 600 800 Balance Simple (HB) 0.00.20.40.60.81.0 Steps (M) 0 200 400 600 800 Metric Value Bookshelf Simple (HB) 0.00.20.40.60.81.0 Steps (M) 0 50 100 150 200 250 Basketball (HB) 0.00.20.40.60.81.0 Steps (M) 0 100 200 300 Door (HB) 0.00.20.40.60.81.0 Steps (M) 0 50 100 150 Metric Value Balance Hard (HB) 0.00.20.40.60.81.0 Steps (M) 0 200 400 600 800 Run (HB) 0.00.20.40.60.81.0 Steps (M) 0 100 200 300 Hurdle (HB) 0.00.20.40.60.81.0 Steps (M) 0 100 200 300 400 Metric Value Spoon (HB) 0.00.20.40.60.81.0 Steps (M) 0.00 0.25 0.50 0.75 1.00 Key Turn Hard (Myo) 0.00.20.40.60.81.0 Steps (M) 0.0 0.2 0.4 0.6 0.8 1.0 Pen Twirl Hard (Myo) 0.00.20.40.60.81.0 Steps (M) 0 2000 4000 6000 Metric Value Reach (HB) 0.00.20.40.60.81.0 Steps (M) 0 200 400 600 800 Sit Hard (HB) 0.00.20.40.60.81.0 Steps (M) 0 100 200 300 400 500 Slide (HB) 0.00.20.40.60.81.0 Steps (M) 0.0 0.1 0.2 0.3 Metric Value Pick Cube (MS) 0.00.20.40.60.81.0 Steps (M) 0.00 0.05 0.10 0.15 0.20 Turn Faucet (MS) 0.00.20.40.60.81.0 Steps (M) 0.00 0.25 0.50 0.75 1.00 Lift Cube (MS) SACSAC+RSAC+OSSAC+ERNaive StackROSER Figure 10: Learning curves across all tasks. Comparison ofROSERagainst baseline methods across DMC, HB, Myo, and MS benchmarks. 27