Paper deep dive
D-SPEAR: Dual-Stream Prioritized Experience Adaptive Replay for Stable Reinforcement Learning in Robotic Manipulation
Yu Zhang, Karl Mason
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 98%
Last extracted: 4/2/2026, 11:46:48 PM
Summary
D-SPEAR is a novel reinforcement learning framework for robotic manipulation that addresses the 'actor-critic mismatch' by decoupling replay sampling strategies. It uses a shared replay buffer with three components: a uniform anchor set, a high-TD error stream for critic updates, and a low-TD error stream for actor updates. An adaptive anchor mechanism adjusts sampling based on the coefficient of variation of TD errors, and a Huber-based critic objective enhances robustness. Experiments on the robosuite benchmark (Lift and Door tasks) demonstrate that D-SPEAR outperforms SAC, TD3, and DDPG in both final performance and training stability.
Entities (6)
Relation Signals (3)
D-SPEAR → evaluatedon → robosuite
confidence 100% · We evaluate D-SPEAR on challenging robotic manipulation tasks from the robosuite benchmark
D-SPEAR → outperforms → SAC
confidence 100% · D-SPEAR consistently outperforms strong off-policy baselines, including SAC, TD3, and DDPG
D-SPEAR → uses → Huber loss
confidence 100% · D-SPEAR replaces the mean squared error (MSE) critic loss used in standard SAC with the Huber loss
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Robotic manipulation remains challenging for reinforcement learning due to contact-rich dynamics, long horizons, and training instability. Although off-policy actor-critic algorithms such as SAC and TD3 perform well in simulation, they often suffer from policy oscillations and performance collapse in realistic settings, partly due to experience replay strategies that ignore the differing data requirements of the actor and the critic. We propose D-SPEAR: Dual-Stream Prioritized Experience Adaptive Replay, a replay framework that decouples actor and critic sampling while maintaining a shared replay buffer. The critic leverages prioritized replay for efficient value learning, whereas the actor is updated using low-error transitions to stabilize policy optimization. An adaptive anchor mechanism balances uniform and prioritized sampling based on the coefficient of variation of TD errors, and a Huber-based critic objective further improves robustness under heterogeneous reward scales. We evaluate D-SPEAR on challenging robotic manipulation tasks from the robosuite benchmark, including Block-Lifting and Door-Opening. Results demonstrate that D-SPEAR consistently outperforms strong off-policy baselines, including SAC, TD3, and DDPG, in both final performance and training stability, with ablation studies confirming the complementary roles of the actorside and critic-side replay streams.
Tags
Links
- Source: https://arxiv.org/abs/2603.27346v2
- Canonical: https://arxiv.org/abs/2603.27346v2
Trouble viewing inline? Open PDF directly →
Full Text
31,396 characters extracted from source content.
Expand or collapse full text
D-SPEAR: Dual-Stream Prioritized Experience Adaptive Replay for Stable Reinforcement Learning in Robotic Manipulation Yu Zhang Karl Mason* Abstract Robotic manipulation remains challenging for reinforcement learning due to contact-rich dynamics, long horizons, and training instability. Although off-policy actor–critic algorithms such as SAC and TD3 perform well in simulation, they often suffer from policy oscillations and performance collapse in realistic settings, partly due to experience replay strategies that ignore the differing data requirements of the actor and the critic. We propose D-SPEAR: Dual-Stream Prioritized Experience Adaptive Replay, a replay framework that decouples actor and critic sampling while maintaining a shared replay buffer. The critic leverages prioritized replay for efficient value learning, whereas the actor is updated using low-error transitions to stabilize policy optimization. An adaptive anchor mechanism balances uniform and prioritized sampling based on the coefficient of variation of TD errors, and a Huber-based critic objective further improves robustness under heterogeneous reward scales. We evaluate D-SPEAR on challenging robotic manipulation tasks from the robosuite benchmark, including Block-Lifting and Door-Opening. Results demonstrate that D-SPEAR consistently outperforms strong off-policy baselines, including SAC, TD3, and DDPG, in both final performance and training stability, with ablation studies confirming the complementary roles of the actor-side and critic-side replay streams. I Introduction Robotic manipulation remains difficult for reinforcement learning because contact-rich interactions and long horizons often produce unstable training dynamics [15]. Although off-policy actor–critic methods such as SAC and TD3 [7, 2] perform well in simulation, they can still exhibit high variance, policy oscillations, and late-stage collapse in realistic manipulation settings [1]. Experience replay improves off-policy learning through data reuse [12], but uniform replay treats all transitions as equally useful. In robotic manipulation, replay buffers are often dominated by low-information samples, while rare but informative transitions are under-utilized. Prioritized Experience Replay (PER) addresses this with TD-error-based sampling [14], yet high-TD samples can destabilize continuous actor–critic learning by injecting unreliable value gradients into policy updates [13]. This reveals a mismatch: the critic benefits from high-error samples, whereas the actor benefits from stable, low-error ones [5]. To address this mismatch, we propose D-SPEAR, which decouples actor and critic sampling while keeping a shared replay buffer. D-SPEAR combines critic-side prioritization, actor-side low-error replay, an adaptive anchor mechanism based on TD-error variation, and a Huber-based critic objective [3]. This design provides a simple yet effective mechanism for balancing value learning and policy stability, particularly in robotic manipulation tasks where contact events and sparse rewards often produce highly uneven TD-error distributions. We evaluate D-SPEAR on the robosuite Lift and Door tasks. It consistently outperforms SAC, TD3, and DDPG [11] in final return and training stability, while ablations confirm the complementary roles of the two replay streams [16]. These results indicate that separating replay strategies for actor and critic updates can improve learning stability in contact-rich manipulation environments. More broadly, the proposed framework illustrates how replay mechanisms can be adapted to actor–critic optimization roles. I-A Reinforcement Learning for Robotic Manipulation Reinforcement learning has been widely applied to robotic manipulation, including grasping, lifting, and articulated object interaction [10, 9, 15]. These tasks are typically formulated as Markov Decision Processes (MDPs), which yield transition tuples (st,at,rt,st+1)(s_t,a_t,r_t,s_t+1) stored in replay buffers for off-policy learning. Because manipulation involves contact-rich dynamics, delayed rewards, and long horizons, model-free methods remain attractive but often suffer from unstable training. I-B Soft Actor–Critic for Continuous Control Among off-policy actor–critic methods, Soft Actor-Critic (SAC) has emerged as a strong baseline for complex continuous control tasks [7]. In addition, Twin Delayed Deep Deterministic Policy Gradient (TD3) is another widely used off-policy actor–critic algorithm and serves as an important baseline in our experiments. SAC is derived under the maximum entropy reinforcement learning framework, which augments the standard expected return objective with an entropy regularization term. This formulation encourages policies that maximize both task performance and stochasticity, leading to improved exploration and robustness: J(π)=τ∼π[∑t=0∞γt(r(st,at)+αℋ(π(⋅∣st)))],J(π)=E_τ π [ _t=0^∞γ^t (r(s_t,a_t)+ (π(· s_t)) ) ], (1) where J(π)J(π) denotes the maximum entropy objective for policy π, τ=(s0,a0,s1,a1,…)τ=(s_0,a_0,s_1,a_1,…) represents a trajectory sampled under policy π, and γ∈(0,1)γ∈(0,1) is the discount factor. The term r(st,at)r(s_t,a_t) denotes the reward received at time step t, ℋ(π(⋅∣st))=−a∼π[logπ(a∣st)]H(π(· s_t))=-E_a π[ π(a s_t)] is the entropy of the policy at state sts_t, and α>0α>0 is the temperature parameter that controls the trade-off between reward maximization and entropy. In practice, α is automatically tuned following the standard SAC implementation, and the same setting is adopted in D-SPEAR. SAC employs an off-policy actor–critic architecture with clipped double Q-learning to mitigate overestimation bias. The critic is trained to minimize a soft Bellman residual, while the actor is updated by minimizing the Kullback–Leibler divergence between the policy and an energy-based distribution induced by the Q-function [6]. This combination enables SAC to achieve high sample efficiency and stable performance in high-dimensional, nonlinear control problems. As a result, SAC has been widely adopted in robotic manipulation tasks, where efficient exploration and robustness to modeling errors are critical [4]. I-C Experience Replay, Prioritization, and Robust Objectives Experience replay (ER) is a fundamental mechanism that enables sample-efficient off-policy learning [12]. By storing transitions collected through interaction with the environment and reusing them for multiple updates, ER breaks temporal correlations in training data and improves data efficiency. Standard ER relies on uniform sampling, implicitly assuming that all stored transitions contribute equally to learning. In robotic manipulation tasks, however, replay buffers are often dominated by low-information transitions, particularly in settings with sparse rewards or delayed task completion signals. Prioritized Experience Replay (PER) was introduced to address this limitation by sampling transitions in proportion to their temporal-difference (TD) errors, under the assumption that larger errors indicate higher learning potential [14]. While PER has demonstrated clear benefits in value-based methods, its application to continuous actor–critic algorithms can introduce instability. High TD-error transitions often correspond to regions where the critic’s value estimates are inaccurate. In actor–critic methods, policy updates depend on value gradients ∇aQ(s,a) _aQ(s,a), typically expressed as ∇θJ(θ)=s∼D[∇aQ(s,a)∇θπθ(s)]. _θJ(θ)=E_s D [ _aQ(s,a) _θ _θ(s) ]. (2) Oversampling transitions with large TD errors can therefore amplify gradient variance during policy updates, which may destabilize the actor. Beyond sampling strategies, the robustness of actor–critic learning is also influenced by the choice of loss function. Mean squared error objectives are sensitive to outliers and unnormalized rewards, which frequently arise in robotic manipulation environments [13]. Robust alternatives, such as the Huber loss [8], mitigate this issue by bounding gradient magnitudes under large errors. The Huber loss is defined as ℒδ(x)=12x2,if |x|≤δ,δ(|x|−12δ),otherwise,L_δ(x)= cases 12x^2,&if |x|≤δ,\\ δ (|x|- 12δ ),&otherwise, cases (3) where x denotes the prediction error and δ is a threshold parameter that controls the transition between quadratic and linear regimes [8]. In our implementation, we set the Huber threshold to δ=0.1δ=0.1, as summarized in Table I. By behaving quadratically for small errors and linearly for large errors, the Huber loss reduces sensitivity to outliers while preserving stable gradients for accurate estimates. Although experience replay, prioritization, and robust objectives have each been studied in isolation, their interaction in complex robotic manipulation settings remains insufficiently explored. As illustrated in Figure 1, explicitly decoupling replay strategies for the actor and the critic provides a structured approach to addressing these challenges, which directly motivates the framework proposed in this work. I Methodology: D-SPEAR: Dual-Stream Prioritized Experience Adaptive Replay Figure 1: Overview of the proposed D-SPEAR: Dual-Stream Prioritized Experience Adaptive Replay (D-SPEAR). The replay buffer is decomposed into an anchor set sampled uniformly and two prioritized streams: a high-TD stream for critic updates and a low-TD inverse-priority stream for actor updates. An adaptive controller adjusts the anchor ratio λ based on the coefficient of variation (CV) of TD errors. I-A Actor–Critic Mismatch in Robotic Manipulation We consider robotic manipulation tasks modeled as Markov Decision Processes and optimized using off-policy actor–critic algorithms. Let Qθ(s,a)Q_θ(s,a) denote the critic parameterized by θ, and let πϕ(a|s) _φ(a|s) denote the stochastic policy parameterized by ϕφ. In this setting, the critic provides value gradients that directly guide policy updates. In contact-rich manipulation tasks, such as object lifting and articulated door opening, large temporal-difference (TD) errors frequently arise near contact transitions, failure states, or boundary conditions [13]. Although such transitions are highly informative for value learning, they often correspond to regions where the critic’s estimates are unreliable. Updating the actor using gradients derived from these inaccurate value estimates introduces high variance and bias into policy updates, which can lead to oscillatory behavior or late-stage performance collapse. This observation highlights a fundamental mismatch in the optimization objectives of the actor–critic architecture: the critic benefits from replaying high-error transitions to rapidly correct value estimates, whereas the actor requires stable and accurate value gradients to ensure monotonic policy improvement. Applying identical replay strategies to both components therefore induces conflicting optimization pressures and destabilizes training. This analysis provides an intuitive explanation of the actor–critic mismatch rather than a formal theoretical bound. I-B Dual-Stream Experience Replay To address this mismatch, D-SPEAR decouples the replay sampling strategies for the actor and the critic while maintaining a shared replay buffer D. Each training batch is composed of three elements: a uniformly sampled anchor set, a critic-specific prioritized stream, and an actor-specific low-error stream. Anchor Sampling To preserve distributional coverage and prevent excessive sampling bias, we sample an anchor subset ℬanchorB_anchor uniformly from the replay buffer: ℬanchor∼Uniform(),B_anchor (D), (4) with size ⌊λN⌋ λ N , where N denotes the batch size and λ∈[0,1]λ∈[0,1] controls the proportion of uniformly sampled transitions. In practice, we construct a candidate set whose size is four times the batch size (candidate sampling ratio = 4) before applying the prioritized sampling strategies. Critic Stream The critic-specific subset ℬCB_C is sampled from the remaining transitions using prioritized replay based on TD-error magnitude: PC(i)∝|δi|αc,P_C(i) | _i| _c, (5) where δi _i denotes the TD error of transition i, and αc _c controls the strength of prioritization. This stream emphasizes informative transitions to accelerate value convergence. In our implementation, we set αc=1 _c=1, which corresponds to the standard PER prioritization strength. Actor Stream The actor-specific subset ℬAB_A is sampled using an inverse-priority distribution that favors low-error transitions: PA(i)∝(|δi|+ε)−βa,P_A(i) (| _i|+ )^- _a, (6) where βa _a controls the preference for stable transitions and ε>0 >0 is a small constant introduced to prevent numerical instability when |δi|=0| _i|=0. In our implementation, we set βa=1 _a=1, which provides a balanced trade-off between prioritization and distributional diversity. By updating the policy primarily using low-error samples, the actor avoids unreliable gradients and achieves more stable policy improvement. I-C Adaptive Anchor Mechanism A fixed anchor ratio λ is suboptimal because the informativeness of TD-error priorities changes throughout training. When TD errors exhibit strong relative dispersion, prioritized replay provides a more discriminative learning signal. By contrast, when TD errors are more homogeneous, the benefit of prioritization diminishes and a larger uniformly sampled anchor subset helps maintain stable state–action coverage. To capture this effect, we adaptively adjust the anchor ratio λ, which controls the fraction of uniformly sampled transitions in each training batch, based on the coefficient of variation (CV) of TD-error magnitudes in the replay buffer. At each update step, the CV is estimated from a fixed set of NCV=1000N_CV=1000 TD-error samples drawn uniformly at random from the replay buffer, independent of the training mini-batch size: CV=σδμδ+ε,λ=1−clip(CV,0,1−λmin),CV= _δ _δ+ , λ=1-clip(CV,0,1- _ ), (7) where μδ _δ and σδ _δ denote the mean and standard deviation of the sampled TD-error magnitudes, respectively, ε is a small constant for numerical stability, and λmin _ is the minimum anchor ratio. Intuitively, a larger CV indicates greater relative dispersion in TD errors, so λ decreases toward λmin _ and the update places more emphasis on prioritized replay. Conversely, when the CV is smaller, λ increases and a larger fraction of uniformly sampled anchor transitions is retained. In this way, the method automatically balances coverage and prioritization during training without requiring manual task-specific tuning. I-D Robust Critic Objective Robotic manipulation tasks often involve heterogeneous reward scales and occasional outlier transitions. To make critic updates less sensitive to large Bellman residuals, D-SPEAR replaces the mean squared error (MSE) critic loss used in standard SAC with the Huber loss defined in the previous subsection, while keeping the SAC target unchanged. Let ℬcB_c denote the critic replay batch. Then each critic is optimized by ℒQ(θi)=ℬc[ℒδ(Qθi(s,a)−y)],i∈1,2,L_Q( _i)=E_B_c [L_δ\! (Q_ _i(s,a)-y ) ], i∈\1,2\, (8) where y=r+γ(1−d)( y=r+γ(1-d) ( minj∈1,2Qθ¯j(s′,a′) _j∈\1,2\Q_ θ_j(s ,a ) (9) −αlogπϕ(a′∣s′)), -α _φ(a s ) ), and a′∼πϕ(⋅∣s′)a _φ(· s ). In all experiments, we set the Huber threshold to δ=0.1δ=0.1. Compared with MSE, this modification preserves the standard SAC Bellman target but is intended to reduce the sensitivity of critic updates to rare large TD errors. The overall training procedure of D-SPEAR integrates the dual-stream replay mechanism, adaptive anchor adjustment, and robust critic optimization described above. For clarity, Algorithm 1 summarizes the complete learning process within an off-policy actor–critic framework. Algorithm 1 Dual-Stream Prioritized Experience Adaptive Replay (D-SPEAR) 0: Batch size N, minimum anchor ratio λmin _ , priority exponents αc _c, βa _a, Huber threshold δ 0: Trained policy πϕ _φ 1: Initialize policy network πϕ _φ, critic network QθQ_θ, replay buffer D 2: for each environment step t do 3: Execute action at∼πϕ(⋅|st)a_t _φ(·|s_t), observe (rt,st+1)(r_t,s_t+1) 4: Store transition (st,at,rt,st+1)(s_t,a_t,r_t,s_t+1) in D with maximum priority 5: if D is ready for update then 6: Sample TD errors from D to estimate μδ,σδ _δ, _δ 7: Compute CV←σδ/(μδ+ε)CV← _δ/( _δ+ ) 8: Update anchor ratio λ←1−clip(CV,0,1−λmin)λ← 1-clip(CV,0,1- _ ) 9: Sample anchor set ℬanchorB_anchor of size ⌊λN⌋ λ N uniformly from D 10: Sample critic indices ℐC∼|δi|αcI_C | _i| _c of size N−|ℬanchor|N-|B_anchor| 11: Sample actor indices ℐA∼(|δi|+ε)−βaI_A (| _i|+ )^- _a of size N−N- |ℬanchor||B_anchor| 12: Construct batches ℬC←ℬanchor∪ℐCB_C _anchor _C 13: Construct batches ℬA←ℬanchor∪ℐAB_A _anchor _A 14: Update θ by minimizing Huber loss on ℬCB_C: 15: θ←θ−ηQ∇θℒδ(ℬC)θ←θ- _Q _θL_δ(B_C) 16: Update priorities in D using new TD errors from ℬCB_C 17: Update ϕφ by maximizing SAC objective on ℬAB_A: 18: ϕ←ϕ+ηπ∇ϕs∼ℬA[Qθ(s,πϕ(s))−φ←φ+ _π _φE_s _A[Q_θ(s, _φ(s))- αlogπϕ(s)]α _φ(s)] 19: Update entropy temperature α and target networks 20: end if 21: end for I Experiment I-A Experimental Setup Environments Experiments are conducted on two robotic manipulation tasks from the robosuite framework: Lift and Door [16]. Both tasks are instantiated with a 7-DoF Franka Panda robot and involve continuous state and action spaces with contact-rich dynamics. The Lift task requires the robot to grasp and lift an object from a table surface to a target height, whereas the Door task involves opening a hinged door through sustained interaction with its handle. These environments feature long-horizon decision making, delayed rewards, and sensitive contact transitions, making them well suited for evaluating training stability in robotic manipulation. Sample efficiency is evaluated using learning curves that report episode return as a function of environment interaction steps. Representative visualizations of both environments are shown in Figure 2. Environment Configuration All environments are controlled using an operational space controller in position mode (OSC_POSITION). The control frequency is set to 2020 Hz, and each episode has a fixed horizon of 500500 time steps. Visual observations are disabled, and only low-dimensional proprioceptive and object-state observations are used. Reward shaping provided by robosuite is enabled to facilitate learning while preserving task difficulty. A summary of the environment configuration is provided in Table I. TABLE I: Robotic manipulation environment and training setup parameters. Parameter Value Environment Lift, Door Robot Franka Panda Controller OSC_POSITION Control frequency 20 Hz Episode horizon 500 steps Observation type Low-dimensional state Camera observations Disabled Reward shaping Enabled Action space Continuous Total training steps 2.5×1052.5× 10^5 Number of random seeds 5 Algorithm Configuration D-SPEAR is implemented on top of a Soft Actor–Critic (SAC) backbone. Both the actor and critic networks are parameterized as two-layer multilayer perceptrons with 256256 hidden units per layer. A shared replay buffer stores transitions of the form (st,at,rt,st+1,dt)(s_t,a_t,r_t,s_t+1,d_t). The proposed dual-stream replay mechanism constructs separate mini-batches for actor and critic updates, as described in Section I. All algorithmic and training hyperparameters are summarized in Table I. TABLE I: Algorithm and training hyperparameters. Parameter Value Base algorithm SAC Actor/Critic network 2-layer MLP (256 units) Optimizer Adam Replay buffer Shared Batch size N 256 Discount factor γ 0.99 Warm-up steps 5,000 Updates per step 1 Minimum anchor ratio λmin _ 0.5 Candidate sampling ratio 4 Critic priority exponent αc _c 1.0 Actor inverse-priority temperature 1.0 Critic loss Huber loss(δ=0.1δ=0.1) Training Protocol All methods are trained for 500500 episodes with a maximum of 500500 steps per episode, resulting in a total interaction budget of 2.5×1052.5× 10^5 environment steps. The first 5,0005,000 steps are used for random exploration before policy updates begin. After the warm-up phase, one gradient update is performed per environment step. Each experiment is repeated over 55 random seeds. Performance is evaluated using episode return, reported as the mean across seeds. Figure 2: Robotic manipulation environments used in our experiments. (a) Lift: object grasping and lifting. (b) Door: contact-rich articulated door opening. I-B Main Results on Robotic Manipulation Tasks Figure 3: Performance comparison on robosuite manipulation tasks. Episode return as a function of environment steps on Lift and Door. Solid lines denote the mean performance over 55 random seeds, and shaded regions indicate one standard deviation. Figure 3 compares D-SPEAR with standard off-policy baselines on the two robotic manipulation tasks. Across both environments, D-SPEAR consistently achieves superior final performance and exhibits substantially improved training stability. TABLE I: Final performance on robosuite manipulation tasks. Results are reported as the mean episode return over the final 10 evaluation episodes, averaged across 5 random seeds. Environment D-SPEAR SAC TD3 DDPG Lift 305.44 156.91 164.87 24.64 Door 210.37 149.27 164.40 11.35 In addition to the learning curves in Figure 3, Table I summarizes the final performance of all methods. D-SPEAR achieves the highest final returns on both manipulation tasks, outperforming all baselines by a substantial margin. The performance gap is particularly pronounced on the Lift task, where stable long-horizon value estimation is critical. Lift Task As shown in Figure 3 and Table I, D-SPEAR demonstrates sustained performance improvement throughout training and attains the highest final episode return. In contrast, SAC and TD3 exhibit early learning progress but plateau at substantially lower performance levels, with noticeable variance across seeds. DDPG fails to maintain stable learning and collapses to near-zero returns after initial exploration. Door Task The Door task presents a more challenging contact-rich manipulation problem due to articulated dynamics and delayed task completion signals. Figure 3 shows that while SAC and TD3 initially achieve competitive returns, both methods suffer from pronounced performance oscillations and late-stage degradation. In contrast, D-SPEAR maintains stable learning dynamics and consistently converges to higher returns with reduced variance. I-C Ablation Study Figure 4: Ablation study on robosuite manipulation tasks. We compare the full D-SPEAR framework with variants that remove key components: w/o dual-stream, w/o low-actor, and w/o high-critic. Results are averaged over 55 random seeds, with shaded regions indicating one standard deviation. Figure 4 presents the ablation results on both the Lift and Door tasks, illustrating the contribution of each component in D-SPEAR. Effect of Dual-Stream Replay. As shown in Figure 4, removing the dual-stream structure (w/o dual-stream) significantly degrades performance on both tasks and increases training variance, closely resembling the behavior of the vanilla SAC baseline. Effect of Critic-Side High-TD Sampling. Disabling critic-side high-TD prioritization (w/o high-critic) leads to slower convergence and pronounced oscillations, particularly in the Door task, as evidenced in Figure 4. Effect of Actor-Side Low-TD Sampling. Removing the actor-side low-TD replay stream (w/o low-actor) results in unstable policy updates and intermittent performance degradation, highlighting the importance of constraining policy learning to reliable value estimates. Overall, the ablation results in Figure 4 demonstrate that the actor-side and critic-side replay streams in D-SPEAR play complementary roles. Their combined effect enables efficient value learning while suppressing policy instability, leading to robust performance across diverse robotic manipulation tasks. IV Conclusion and Future Work In this paper, we presented D-SPEAR, a dual-stream adaptive experience replay framework for off-policy actor–critic learning in robotic manipulation. By assigning separate replay streams to the critic and the actor, D-SPEAR addresses the mismatch between value estimation, which benefits from high-TD transitions, and policy improvement, which favors conservative updates. We combine this dual-stream design with a CV-based adaptive anchor mechanism and a robust critic objective to improve training stability. We evaluated D-SPEAR on robosuite manipulation benchmarks using the Panda robot on the Lift and Door tasks. Across these settings, D-SPEAR achieved stronger final performance and lower variance than off-policy baselines, while ablations show that the actor-side and critic-side replay streams contribute to stable learning. These results highlight the benefit of decoupling replay strategies for actor and critic updates in contact-rich manipulation problems. By balancing prioritization and coverage through the adaptive anchor mechanism, the framework provides a practical approach for improving stability in off-policy reinforcement learning. Future work will extend D-SPEAR in three directions. First, we will evaluate the method on a broader range of benchmark tasks, including additional robosuite environments and other manipulation suites to further assess generality. Second, we will investigate state-dependent and task-aware replay adaptation, for example by conditioning the anchor ratio or sampling distribution on task identity, episode phase, or state uncertainty instead of relying on a single global replay rule. Third, we will study real-robot and sim-to-real settings and explore integrating D-SPEAR with stronger actor–critic backbones and model-based components for scalable and reliable robot learning. References [1] G. Christmann, Y. Luo, H. Mandala, and W. Chen (2024) Benchmarking smoothness and reducing high-frequency oscillations in continuous control policies. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 627–634. Cited by: §I. [2] S. Fujimoto, H. Hoof, and D. Meger (2018) Addressing function approximation error in actor-critic methods. In International conference on machine learning, p. 1587–1596. Cited by: §I. [3] S. Fujimoto, D. Meger, and D. Precup (2020) An equivalence between loss functions and non-uniform sampling in experience replay. Advances in neural information processing systems 33, p. 14219–14230. Cited by: §I. [4] J. Gao, Y. Li, Y. Chen, Y. He, and J. Guo (2024) An improved sac-based deep reinforcement learning framework for collaborative pushing and grasping in underwater environments. IEEE Transactions on Instrumentation and Measurement 73 (), p. 1–14. External Links: Document Cited by: §I-B. [5] I. Grondman, L. Busoniu, G. A. D. Lopes, and R. Babuska (2012) A survey of actor-critic reinforcement learning: standard and natural policy gradients. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 42 (6), p. 1291–1307. External Links: Document Cited by: §I. [6] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine (2018) Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, p. 1861–1870. Cited by: §I-B. [7] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel, et al. (2018) Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905. Cited by: §I-B, §I. [8] P. J. Huber (1992) Robust estimation of a location parameter. In Breakthroughs in statistics: Methodology and distribution, p. 492–518. Cited by: §I-C, §I-C. [9] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, et al. (2018) Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on robot learning, p. 651–673. Cited by: §I-A. [10] S. Levine, C. Finn, T. Darrell, and P. Abbeel (2016) End-to-end training of deep visuomotor policies. Journal of Machine Learning Research 17 (39), p. 1–40. Cited by: §I-A. [11] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra (2016) Continuous control with deep reinforcement learning. In International Conference on Learning Representations (ICLR), Note: Published as a conference paper Cited by: §I. [12] L. Lin (1992) Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine learning 8 (3), p. 293–321. Cited by: §I-C, §I. [13] B. Saglam, F. B. Mutlu, D. C. Cicek, and S. S. Kozat (2023) Actor prioritized experience replay. Journal of Artificial Intelligence Research 78, p. 639–672. Cited by: §I-C, §I, §I-A. [14] T. Schaul, J. Quan, I. Antonoglou, and D. Silver (2016) Prioritized experience replay. In International Conference on Learning Representations (ICLR), Note: Published as a conference paper External Links: Link Cited by: §I-C, §I. [15] C. Tang, B. Abbatematteo, J. Hu, R. Chandra, R. Martín-Martín, and P. Stone (2025) Deep reinforcement learning for robotics: a survey of real-world successes. Annual Review of Control, Robotics, and Autonomous Systems 8 (1), p. 153–188. Cited by: §I-A, §I. [16] Y. Zhu, J. Wong, A. Mandlekar, R. Martín-Martín, A. Joshi, K. Lin, S. Nasiriany, and Y. Zhu (2020) Robosuite: a modular simulation framework and benchmark for robot learning. In arXiv preprint arXiv:2009.12293, Cited by: §I, §I-A.