Paper deep dive
Integrating Novelty and Surprise for Experience Prioritization and Exploration in Image-Based Reinforcement Learning
Hoda Yamani, Henry Williams, Bruce A. MacDonald
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 8/19/2026, 4:52:30 AM
Summary
This paper introduces Novelty and Surprise Prioritized Experience Replay (NSPER) and its extension NSPER+R to improve sample efficiency in image-based reinforcement learning. NSPER uses novelty (reconstruction error) and surprise (prediction error) as intrinsic signals to prioritize transitions in the replay buffer, replacing traditional TD-error based prioritization. NSPER+R further integrates these signals as intrinsic rewards to enhance exploration. Experiments on the DeepMind Control Suite demonstrate that these methods improve training efficiency and convergence speed compared to existing baselines like TD-PER and CCLF.
Entities (10)
Relation Signals (9)
NSPER+R → extends → NSPER
confidence 95% · We further extend this with NSPER+R, integrating these signals as intrinsic rewards to jointly improve replay quality and exploration.
NSPER → uses → Novelty
confidence 95% · NSPER, a novel prioritization method that uses two distinct intrinsic signals, novelty and surprise, to guide experience selection within PER.
NSPER → uses → Surprise
confidence 95% · NSPER, a novel prioritization method that uses two distinct intrinsic signals, novelty and surprise, to guide experience selection within PER.
NSPER+R → uses → Intrinsic Rewards
confidence 95% · NSPER+R, an extension of NSPER that integrates novelty and surprise not only for prioritization but also as intrinsic rewards.
NSPER → improves → Sample Efficiency
confidence 90% · NSPER identifies diverse and informative transitions with high learning potential, improving the quality of replayed experiences.
PixelTD3 → isbackbonefor → NSPER
confidence 90% · We adopt this setup as the backbone of NSPER
Novelty → measuredby → SSIM
confidence 90% · The novelty score is then defined as the complement of the Structural Similarity Index (SSIM) between st and s^t
Surprise → measuredby →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Sample efficiency is a central challenge in reinforcement learning (RL), particularly in image-based domains where agents must learn from high-dimensional visual inputs. Traditional sampling often relies on random or suboptimal experience selection, leading to redundant updates and slow learning. Improving efficiency requires mechanisms that prioritize informative experiences while also encouraging effective exploration. Prioritized Experience Replay (PER) addresses part of this challenge by reusing high-value transitions, while intrinsic rewards promote the exploration of novel or uncertain states. However, their integration has not been extensively studied. This paper introduces Novelty and Surprise Prioritized Experience Replay (NSPER), which uses novelty to capture underrepresented states and surprise to expose gaps in the agent's understanding of the environment. We further extend this with NSPER+R, integrating these signals as intrinsic rewards to jointly improve replay quality and exploration. Experiments on DeepMind Control Suite tasks show that NSPER and NSPER+R improve training efficiency and convergence speed compared to existing methods in image-based RL.
Tags
Links
- Source: https://arxiv.org/abs/2608.17373v1
- Canonical: https://arxiv.org/abs/2608.17373v1
Trouble viewing inline? Open PDF directly →
Full Text
45,301 characters extracted from source content.
Expand or collapse full text
Integrating Novelty and Surprise for Experience Prioritization and Exploration in Image-Based Reinforcement Learning Hoda Yamani Henry Williams Bruce A. MacDonald Thanks: Robot Learning Team and CARES Robotics Lab, Department of Electrical, Computer, and Software Engineering, University of Auckland, Auckland, New Zealand. Corresponding author: hyam650@aucklanduni.ac.nz. Abstract Sample efficiency is a central challenge in reinforcement learning (RL), particularly in image-based domains where agents must learn from high-dimensional visual inputs. Traditional sampling often relies on random or suboptimal experience selection, leading to redundant updates and slow learning. Improving efficiency requires mechanisms that prioritize informative experiences while also encouraging effective exploration. Prioritized Experience Replay (PER) addresses part of this challenge by reusing high-value transitions, while intrinsic rewards promote the exploration of novel or uncertain states. However, their integration has not been extensively studied. This paper introduces Novelty and Surprise Prioritized Experience Replay (NSPER), which uses novelty to capture underrepresented states and surprise to expose gaps in the agent’s understanding of the environment. We further extend this with NSPER+R, integrating these signals as intrinsic rewards to jointly improve replay quality and exploration. Experiments on DeepMind Control Suite tasks show that NSPER and NSPER+R improve training efficiency and convergence speed compared to existing methods in image-based RL. IEEEkeywords Reinforcement learning, prioritized experience replay, sample efficiency, novelty, surprise, intrinsic rewards, continuous control, image-based control. Figure 1: Schematic overview of Novelty and Surprise Prioritized Experience Replay (NSPER). 1 INTRODUCTION Reinforcement learning (RL) has shown great promise in image-based applications, allowing agents to learn directly from visual input [44, 17]. This paradigm has been widely applied in domains such as gaming [26], robotics [13], and autonomous driving [33], where vision serves as the primary source of environmental feedback. However, high-dimensional image data introduces redundancy, high memory demands, and substantial computational overhead, making policy learning more complex and resource-intensive [9]. Since RL inherently relies on large volumes of interaction data, improving sample efficiency is critical [11]. A promising direction is to prioritize informative experiences during training and to guide exploration toward those experiences that provide the most learning value. This prioritization can be guided by intrinsic learning signals that highlight which transitions are the most useful for learning [5]. Inspired by human cognition, where intrinsic signals foster curiosity [4, 10] and guide memory systems to prioritize novel or salient experiences [35, 7], RL agents can adopt similar mechanisms to enhance exploration and accelerate learning. Such signals provide a foundation for both directing exploratory behavior and selecting the most informative experiences during training [30, 16], thereby improving sample efficiency in two complementary ways. First, they can be formulated as intrinsic rewards, encouraging agents to acquire diverse and informative experiences that accelerate policy refinement [28]. Second, they can be used as prioritization scores within experience replay, enabling agents to focus on transitions with the highest learning potential. Among intrinsic signals, novelty and surprise have proven especially effective as intrinsic rewards [6]. Novelty reflects the unfamiliarity of a state, driving exploration toward underrepresented or unexplored regions [18, 20]. Surprise, on the other hand, arises when outcomes deviate substantially from predictions, exposing gaps in the agent’s environmental model and its limited ability to generalize from past experiences [1]. Combining these two signals enables agents to identify regions that require further learning and adjust their exploration strategies, resulting in more efficient and targeted exploration [22, 39]. Prioritized Experience Replay (PER) is a well-established approach in RL that exploits such intrinsic signals within the replay buffer to improve sample efficiency [32]. In its original form, which we refer to as TD-PER, transitions are prioritized based on the temporal difference (TD) error, which measures the discrepancy between predicted and actual outcomes following an agent’s action. While effective in discrete action spaces, TD-PER struggles in continuous control environments [29, 27]. To overcome these limitations, several extensions have been proposed, including augmenting TD error with auxiliary signals or using biologically inspired mechanisms for prioritization [31, 27, 40, 37]. Despite these advances, integrating intrinsic rewards with PER in image-based RL remains limited. For instance, the Contrastive Curiosity Learning Framework (CCLF) [37] uses curiosity to prioritize surprising augmented inputs. However, the broader use of intrinsic rewards as scoring signals for experience prioritization remains largely underexplored. To address this gap, we propose NSPER, a novel prioritization method that uses two distinct intrinsic signals, novelty and surprise, to guide experience selection within PER. By combining these signals to guide sampling, NSPER identifies diverse and informative transitions with high learning potential, improving the quality of replayed experiences. We further introduce NSPER+R, an extension of NSPER that integrates novelty and surprise not only for prioritization but also as intrinsic rewards. This dual mechanism improves experience selection and exploratory behavior, leading to faster convergence and more generalizable policies. To evaluate our approach, we integrate it into PixelTD3, an image-based variant of the TD3 algorithm [14], and assess its performance on challenging tasks from the DeepMind Control Suite [38], demonstrating promising improvements in complex, high-dimensional visual environments. The model architecture is illustrated in Fig. 1. The implementation is publicly available at https://github.com/UoA-CARES/NSPER. The main contributions of this paper are summarized as follows: 1. We introduce NSPER, a novel approach that uses intrinsic rewards as prioritization signals within PER to improve sample efficiency in image-based RL. 2. We extend this idea in NSPER+R, which uses novelty and surprise both for prioritization and as intrinsic rewards, further enhancing exploration and policy learning. 3. We provide a comprehensive evaluation of different prioritization strategies and their integration with intrinsic rewards, analyzing their impact on training efficiency and learning performance. 4. We conduct an ablation study to isolate the individual contributions of novelty and surprise, as well as their combined effect when used with intrinsic rewards. 2 RELATED WORKS Experience prioritization and intrinsic motivation are key strategies for improving sample efficiency and exploration in RL. Prior work has explored various mechanisms for identifying informative experiences and guiding agents toward novel or surprising transitions, providing a foundation for methods that combine these ideas to enhance learning in complex environments. 2.1 Experience Prioritization Methods PER methods enhance sample efficiency by assigning higher sampling probability to transitions with greater learning potential [42]. TD-PER [32], a common variant, ranks transitions based on TD error, under the assumption that transitions with larger TD errors are less familiar and therefore more informative. However, in continuous high-dimensional spaces, small perturbations in actions or observations can lead to large fluctuations in value estimates, even for well-learned transitions, making TD error a noisy and unstable prioritization signal [29]. This sensitivity makes TD error a noisy and unreliable prioritization signal, motivating alternative strategies that offer more stable and informative learning signals in continuous control tasks [8, 25]. To address these limitations, LA3P [31] decouples learning signals by assigning low TD-error transitions to the actor to stabilize policy updates and high TD-error transitions to the critic to improve value estimation. Another method [42] prioritizes recent transitions with low TD error, thereby focusing on up-to-date experiences, but this risks catastrophic forgetting by discarding older yet valuable data. Model-Augmented PER (MaPER) [27] incorporates auxiliary signals derived from learned environmental dynamics to complement the TD error, thereby improving the identification of important experiences. Despite these enhancements, all of these methods primarily rely on the TD error as the primary prioritization criterion. In contrast, Reward Prediction Error (RPE) prioritization [40] introduces an alternative signal inspired by biological learning. By measuring the discrepancy between expected and actual rewards, RPE provides informative feedback on its prediction accuracy, allowing it to adjust its policies and enhance decision-making over time [36]. Similarly, our method proposes a novel prioritization strategy that uses alternative intrinsic signals, fostering exploration and emphasizing unfamiliar transitions. 2.2 Intrinsic Motivation Strategies Intrinsic motivation has been widely explored in RL to encourage agents to explore and learn beyond extrinsic rewards [28]. This approach encourages agents to seek out novel states and behaviors within the environment [3, 37]. Several intrinsic motivation mechanisms have been proposed, including state novelty [6], state prediction errors [6], uncertainty about outcomes [24], and environment dynamics [34]. Among these, novelty and surprise enable more focused and efficient learning by directly targeting meaningful opportunities, guiding exploration toward unfamiliar states and significant prediction errors, and outperforming methods that rely solely on uncertainty or environmental dynamics [39]. Novelty encourages visiting rarely encountered states, promoting a broader understanding of the environment [18]. Surprise is measured through prediction errors in environment dynamics, such as next-state errors and state transition divergences [30, 1]. Combining novelty and surprise captures complementary aspects of agent-environment interaction, improving exploration and representation learning [18, 20] and allowing better balance between exploration and exploitation in complex tasks [22]. Despite these advances, integrating intrinsic motivation signals into memory prioritization mechanisms remains an ongoing challenge. Several recent studies have focused on combining curiosity-based metrics with experience replay to further refine prioritization strategies. 2.3 Integrating Intrinsic Signals in Experience Replay Recent studies have explored the incorporation of intrinsic motivation into memory prioritization. Curiosity-driven experience prioritization (CDP) [43] prioritizes trajectories that lead to rare goal states, ensuring the agent focuses on underexplored areas. However, CDP is based on the replay of hindsight experiences (HER) [2], which limits its applicability to goal-oriented tasks. Augmented Curiosity-Driven Experience Replay (ACDER) [23] also integrates curiosity signals into HER but remains restricted to goal-conditioned exploration. A more relevant approach, CCLF [37], introduces a self-supervised prioritization mechanism in image-based RL. CCLF selects surprising augmented inputs to enhance sample efficiency, but it focuses primarily on representation learning rather than on optimizing experience selection for policy improvement. Our method extends these insights by integrating novelty and surprise into experience prioritization, ensuring that selected transitions support both exploration and policy optimization. By combining intrinsic rewards with a structured prioritization mechanism, it overcomes the limitations of PER and leverages intrinsic motivation to enable more effective learning in challenging environments 3 METHODOLOGY This section outlines the components underlying our approach, beginning with a review of PER, PixelTD3, and the computation of novelty and surprise signals. We then introduce our framework, NSPER, and its extension, NSPER+R. 3.1 PER PER improves sample efficiency by replaying transitions with higher learning potential. Transitions in the buffer B are stored as: Bi=(si,ai,ri,si+1),B_i=(s_i,a_i,r_i,s_i+1), (1) where i indexes the time step. The priority of each transition is defined as: σi=|δi|+ϵ, _i=| _i|+ε, (2) In the standard variant, TD-PER, δi _i is the TD error, and ϵ>0ε>0 ensures non-zero priority. Transitions are sampled according to: pi=σiα∑kσkαp_i= _i^α _k _k^α (3) wi=(1|B|pi)βw_i= ( 1|B|\,p_i )^β (4) where pip_i favors transitions with higher priority σi _i, and wiw_i corrects for the bias introduced by non-uniform sampling. Here, α controls prioritization sharpness, and β compensates for sampling bias. While TD-PER defines σi _i using TD error, PER is flexible and allows alternative prioritization signals by redefining σi _i. 3.2 PixelTD3 PixelTD3 is an image-based extension of TD3 [14], inspired by SACAE [41] and extended in NaSATD3 [39]. PixelTD3 employs a convolutional autoencoder that maps raw observations sts_t into a compact latent representation z. This latent vector is shared between the actor and critic networks, ensuring that policy learning is grounded in a consistent feature space. The encoder–decoder is jointly trained with a reconstruction loss that preserves the structural content of the input images while maintaining task-relevant features. Unlike SACAE, which trains separate encoders for actor and critic, PixelTD3 uses a single shared autoencoder that is updated end-to-end at every step, including both convolutional and linear layers. This provides a stable and compact representation for policy learning. We adopt this setup as the backbone of NSPER (see Fig. 1). Complete architectural specifications of the encoder, decoder, actor, critics, and predictive ensemble are reported in Table 3. 3.3 Novelty And Surprise Computation Novelty measures how different an observation is from what the agent has already encoded. It is estimated by reconstructing the input sts_t through the autoencoder and comparing it to the original observation. Let s^t=Dec(Enc(st)) s_t=Dec(Enc(s_t)) denote the reconstruction of sts_t. The novelty score is then defined as the complement of the Structural Similarity Index (SSIM) between sts_t and s^t s_t: Novelty(st)=1−SSIM(st,s^t).Novelty(s_t)=1-SSIM(s_t, s_t). (5) A higher value indicates that the observation is less well-reconstructed, and therefore more novel to the agent. Fig. 1 illustrates how novelty is obtained from the autoencoder reconstruction error of the input image. Surprise measures the deviation between the agent’s predicted and observed dynamics. It is computed in latent space using an ensemble of dynamics models PρP_ρ, which predict the next latent state z^t+1 z_t+1 given the current latent representation ztz_t and action ata_t: z^t+1=Pρ(zt,at). z_t+1=P_ρ(z_t,a_t). (6) Surprise is defined as the mean squared error between the predicted and observed latent states: Surprise(st,at)=MSE(zt+1,z^t+1).Surprise(s_t,a_t)=MSE(z_t+1, z_t+1). (7) As illustrated in Fig. 1, this signal corresponds to the mismatch between the latent representation of the observed next state and the ensemble prediction. The ensemble consists of M models trained with mean squared error loss, and averaging their predictions reduces variance and improves stability under non-stationary encoder updates. Novelty and Surprise Combination: Combining novelty and surprise reward signals encourages more targeted exploration of informative regions within the environment. This integration forms a new intrinsic signal, referred to as the Novelty-Surprise Signal (NSS). The NSS(s,a)NSS(s,a), derived from (5) and (7), captures the joint contribution of novelty and surprise and is formulated as: NSS(st,at)=Novelty(st)+Surprise(st,at)NSS(s_t,a_t)=Novelty(s_t)+Surprise(s_t,a_t) (8) 3.4 NSPER: Novelty and Surprise Prioritization We propose a prioritization strategy that leverages novelty and surprise as intrinsic signals to guide experience replay. Rather than relying on the TD error, as in standard PER, NSPER replaces δi _i in (2) with the Novelty–Surprise Signal NSS(s,a)NSS(s,a) from (8). The priority assigned to each transition is therefore defined as: σt=|NSS(st,at)|+ϵ, _t= |NSS(s_t,a_t) |+ε, (9) where ϵ>0ε>0 ensures all transitions remain eligible for replay. These priorities are then used in the sampling equations (3) and (4) to compute transition probabilities and importance weights. By emphasizing transitions with higher novelty and surprise, NSPER favors diverse and informative experiences that better support critic training. The parameter α regulates the strength of this prioritization, ensuring that informative transitions are emphasized without over-concentrating sampling on a few extreme cases. This balance allows NSPER to prioritize informative experiences while maintaining diversity in the sampled transitions. 3.5 Enhancing Exploration with Intrinsic Rewards Intrinsic rewards are agent-generated signals that promote exploration, especially when extrinsic rewards are sparse or delayed [5]. In our approach, we define the intrinsic reward using novelty and surprise (8) as: rint(st,at)=NSS(st,at)r_int(s_t,a_t)=NSS(s_t,a_t) (10) As depicted in Fig. 1, the NSS is used both to prioritize experiences during replay and to define the intrinsic reward signal. The total reward at each step combines extrinsic and intrinsic components: rt=rext+rintr_t=r_ext+r_int (11) We refer to the full version of our framework as NSPER+R, which integrates intrinsic rewards with prioritized experience replay. This variant improves learning by prioritizing transitions that contribute to both task success and deeper environmental understanding. Algorithm 1 summarises the full procedure for NSPER and its intrinsic-reward variant NSPER+R. Algorithm 1 NSPER and NSPER+R with NSS (Novelty–Surprise Signal) 1: Inputs: PER exponents α,βα,β, batch size m, small ε>0 >0 2: Initialize: actor Θ , critic θ, encoder–decoder (Enc,Dec)(Enc,Dec), replay buffer B, priorities σi\ _i\ 3: for each environment step t do 4: Observe state sts_t; select action ata_t (with exploration noise); receive rextr_ext and st+1s_t+1 5: Novelty: Novelty(st)←1−SSIM(st,Dec(Enc(st))) (s_t)← 1-SSIM\! (s_t,\,Dec(Enc(s_t)) ) 6: Surprise: obtain latents (zt+1,zt+1′)(z_t+1,z _t+1) and set Surprise(st,at)←MSE(zt+1′,zt+1) (s_t,a_t) (z _t+1,z_t+1) 7: NSS: NSS(st,at)←Novelty(st)+Surprise(st,at) (s_t,a_t) (s_t)+Surprise(s_t,a_t) 8: Intrinsic reward: rint←NSS(st,at)r_int (s_t,a_t) 9: Total reward: rt←rext+rint,NSPER+Rrext,NSPERr_t← casesr_ext+r_int,&NSPER+R\\ r_ext,&NSPER cases 10: Push (st,at,rt,st+1,NSS(st,at))(s_t,a_t,r_t,s_t+1,NSS(s_t,a_t)) into B 11: Set priority for new item: σnew←|NSS(st,at)|+ε _new←|NSS(s_t,a_t)|+ 12: for each update step do 13: Sample indices I of size m with PER: p(i)=σiα∑kσkαp(i)= _i^α _k _k^α 14: IS weights: w(i)←(1|B|⋅p(i))βw(i)← ( 1|B|· p(i) )^β (optionally normalize by maxj∈Iw(j) _j∈ Iw(j)) 15: Update critic/actor: compute losses on batch, weight by w(i)w(i), update θ,Θθ, 16: Refresh priorities: for each i∈Ii∈ I, set σi←|NSS(si,ai)|+ε _i←|NSS(s_i,a_i)|+ 17: end for 18: end for (a) Cartpole-Balance (b) Finger-Spin (c) Ball-in-Cup (d) Walker-Walk (e) Cheetah-Run Figure 2: Five continuous control tasks from the DeepMind Control Suite used in our evaluation. Table 1: Summary of DeepMind Control Suite tasks used in our experiments. Task Action Space Objective Primary Challenge Type Difficulty Cartpole-Balance [−1,1]1[-1,1]^1 Stabilize the pole by moving the cart horizontally. Underactuated and highly unstable dynamics where small errors rapidly lead to failure. Classic Control Low Finger-Spin [−1,1]2[-1,1]^2 Maintain continuous rotation of a free-floating object. Requires fine torque modulation to sustain spin without drift or loss of contact. Manipulation High Ball-in-Cup [−1,1]2[-1,1]^2 Swing the ball and capture it inside the cup. Oscillatory motion requires precise timing and well-coordinated control. Manipulation Medium–High Walker-Walk [−1,1]6[-1,1]^6 Achieve stable forward locomotion. Balancing and coordinating a bipedal body in an inherently unstable locomotion regime. Locomotion Medium Cheetah-Run [−1,1]6[-1,1]^6 Run forward at high speed. High velocity amplifies instability, requiring strong, consistent, and fast control. Locomotion Medium Table 2: Experimental setup for NSPER across DeepMind Control Suite environments. Parameter Value (Description) Environments Observation 84×84×384× 84× 3 RGB images, with 3 consecutive frames stacked Reward Scale Maximum episodic return of 1000 per task Evaluation Metric Average cumulative reward per step (computed using extrinsic rewards) Implementation Details Replay Buffer Capacity 1,000,0001,000,000 transitions Batch Size 128 (Mini-batch sampled from replay buffer) Actor Learning Rate 1×10−41× 10^-4 Critic Learning Rate 1×10−31× 10^-3 Prioritization Exponent α 0.7 (Controls prioritization sharpness) Importance-Sampling Exponent β 0.4 (Corrects sampling bias) Training Steps 1×1061× 10^6 (Total environment steps) Evaluation Frequency Every 1×1041× 10^4 steps Evaluation Episodes 10 (Episodes per evaluation checkpoint) Random Seeds 5 (Independent runs with different seeds for environment and initialization) Table 3: Network architectures used in NSPER and NSPER+R. Component Input Layers / Units Activations Output Encoder 84×84×384× 84× 3 image 4 conv (32, 3×33× 3) + FC(200) ReLU (conv), tanh (FC) Latent z∈ℝ200z ^200 Decoder Latent z Deconv mirror of encoder ReLU, Sigmoid (final) Image reconstruction Actor (TD3) z (opt. + vector obs) FC(1024) → FC(1024) ReLU, tanh (out) Action ∈[−1,1]d∈[-1,1]^d Critics (TD3, ×2× 2) z|az\|a (opt. + vector obs) FC(1024) → FC(1024) ReLU Scalar Q-value Predictive Ensemble zt|atz_t\|a_t FC(512) → FC(512) ReLU Pred. latent z^t+1 z_t+1 4 EXPERIMENTS (a) (b) (c) (d) (e) NSPER+R NSPER TD-PER+R TD-PER RPE-PER+R RPE-PER Uniform+R Uniform CCLF Figure 3: Learning curves for continuous control tasks using PixelTD3. Solid lines represent the algorithm’s performance with intrinsic rewards (+R), while dashed lines indicate performance without intrinsic rewards. Shaded regions denote the 95% confidence interval across 5 runs. This section describes the experimental setup, simulation tasks, algorithmic settings, and baseline methods used in our evaluation, and then presents an analysis of the learning performance of NSPER, NSPER+R, and competing approaches. 4.1 Experimental Setup We evaluate our algorithms and baselines on five image-based continuous control tasks from the DeepMind Control Suite (DMC) [38]: Cartpole-Balance, Finger-Spin, Ball-in-Cup, Walker-Walk, and Cheetah-Run. These environments constitute the standard benchmark suite for visual reinforcement learning and have been widely used in prior pixel-based studies [19, 21, 39]. This set provides representative coverage of balance, manipulation, and locomotion tasks and offers sufficient diversity for assessing prioritization strategies in image-based continuous control. Visual examples of the environments are given in Fig. 2, and their key properties are summarised in Table 1. All algorithms are trained under a unified experimental configuration following established practice in continuous-control evaluation. Training settings, optimization parameters, evaluation schedules, and replay specifications are listed in Table 2. Using identical configurations across all tasks ensures that performance differences arise from the prioritization mechanism rather than from hyperparameter choices or implementation bias. NSPER and NSPER+R employ a shared latent-space architecture for representation learning, prediction, and action-value estimation. The components of this architecture are outlined in Table 3. 4.2 Experimental Approach To evaluate the effectiveness of our approach, we consider two proposed variants, NSPER and NSPER+R: NSPER: Uses novelty and surprise solely for replay prioritization, isolating their effect on experience selection. NSPER+R: Extends NSPER by also incorporating novelty and surprise as intrinsic rewards, combined with extrinsic rewards (11), allowing us to assess their joint impact on prioritization and exploration. Baseline Algorithms: For comparison, we evaluate a range of established methods within the PixelTD3 framework, which differ in their replay strategies and use of intrinsic rewards. All implementations share the same codebase for fairness. Although CCLF was originally proposed with a pixel-based SAC backbone, we adapt it to PixelTD3 for consistency. The baselines include: 1. Uniform: Uniformly samples transitions from the buffer without PER or intrinsic rewards [14, 12]. 2. Uniform+R (NaSATD3): Uniform sampling with intrinsic rewards, equivalent to PixelTD3+R [39]. 3. TD-PER: PER based on TD error [32], without intrinsic rewards. 4. TD-PER+R: TD-PER augmented with intrinsic rewards. 5. RPE-PER: PER based on reward prediction error (RPE) [40], without intrinsic rewards. 6. RPE-PER+R: RPE-PER extended with intrinsic rewards. 7. CCLF: A contrastive learning-based prioritization method with intrinsic rewards to improve exploration and representation learning [37]. This setup enables a comprehensive evaluation of NSPER and NSPER+R against diverse baselines that vary in their use of prioritization strategies and intrinsic rewards, providing a robust comparison in image-based RL. 4.3 Learning Efficiency Analysis The learning curves in Fig. 3 illustrate the learning efficiency of NSPER and baseline methods across five challenging DeepMind continuous control environments. These curves depict the average evaluation reward over training steps, providing insights into convergence speed and performance differences among NSPER, NSPER+R, and the baseline algorithms. Dashed-line plots represent PixelTD3 with different replay buffer sampling methods but without intrinsic rewards, while solid-line plots of the same color correspond to the same sampling methods with intrinsic rewards. These separate plots directly compare how sampling strategies influence performance with and without intrinsic rewards. It is crucial to note that Baseline methods did not always replicate the performance reported in prior work. Such deviations are not uncommon in deep reinforcement learning and can arise from factors such as environmental stochasticity, initialization randomness, and subtle implementation or runtime differences. As emphasized by Henderson et al. [15], ensuring fair comparison is essential; in our case, the relative ranking of methods remained consistent across multiple runs, supporting the robustness of our evaluations. 5 Results and discussion Performance results across various tasks in Fig. 3 show that NSPER (red plot) results are better than baseline algorithms. It enhances PixelTD3’s performance on four of five tasks while achieving comparable results on the fifth task to the best-performing methods among image-based RL approaches. Our evaluation highlights the crucial role of novelty and surprise signals in improving learning efficiency. In contrast, Uniform sampling (orange plot) exhibits suboptimal performance across all tasks, demonstrating that relying solely on a uniform experience buffer is insufficient. These findings suggest that PixelTD3 could significantly benefit from integrating prioritized sampling or intrinsic reward mechanisms, even in relatively simple environments. Incorporating prioritization strategies alongside intrinsic rewards improves performance across nearly all algorithms, highlighting the advantages of these techniques in fostering more robust and adaptive learning. In particular, NSPER+R achieves the best results among the methods. While CCLF (light green plot) consistently outperforms Uniform across all tasks, its overall performance remains unremarkable. This suggests that CCLF’s prioritization strategy differs fundamentally from that of a prioritization buffer, and, as the results indicate, it does not integrate well with PixelTD3. In addition, the comparable performance of the RPE-PER algorithm (blue plot) to NSPER across all tasks demonstrates its potential as an effective prioritization method. However, NSPER consistently achieves better results. These findings indicate that using a complex prioritization scale in image-based RL can be beneficial, while also reaffirming that novelty and surprise signals are stronger heuristics for selecting informative experiences, ultimately contributing to a more sample-efficient learning process. (a) (b) (c) (d) (e) NSPER+R NSPER NoveltyPER+R NoveltyPER SurprisePER+R SurprisePER Figure 4: Ablation results comparing experience prioritization based on novelty, surprise, and their combination, with and without intrinsic rewards. Shaded regions represent the 95% confidence interval over five runs. 5.1 Task-Specific Analysis This section examines the performance of different methods across individual tasks, highlighting their strengths and limitations in each environment. In Cartpole-Balance (Fig. 3, plot a), a classic control task requiring a cart to balance a pole, all methods using either a prioritized buffer or intrinsic rewards perform well. This suggests that PixelTD3 benefits from PER or intrinsic rewards, even in simpler environments. The Finger-Spin task, which involves manipulating an object with a simulated finger, presents challenges due to complex contact dynamics and sparse rewards. As shown in Fig. 3, plot b, NSPER variants outperform other methods, emphasizing the importance of novelty and surprise signals for learning efficiency. NSPER+R, which integrates intrinsic rewards with PER, further enhances performance. Comparing TD-PER (prioritization based on TD error) with Uniform+R (intrinsic rewards without prioritization) reveals that intrinsic rewards yield better results in this environment than PER alone. However, TD-PER+R underperforms relative to TD-PER, suggesting that conflicting signals from the TD error and novelty-surprise signals can hinder efficiency. In contrast, NSPER+R aligns both prioritization and intrinsic rewards using the same features, thereby improving sample efficiency. The Ball-in-Cup task, characterized by sparse rewards granted only upon successfully catching the ball, provides another challenging test. In (Fig. 3, plot c), NSPER+R outperforms other approaches, demonstrating the benefits of combining novelty and surprise for prioritization with intrinsic rewards. While Uniform+R performs well, NSPER+R and RPE-PER+R achieve consistently better results, reinforcing that properly aligning prioritization with intrinsic rewards enhances performance. The Walker-Walk environment, which requires controlling a bipedal walker to maintain dynamic balance, presents a complex, multi-stage control challenge despite offering dense rewards. (Fig. 3, plot d) shows that NSPER+R and RPE-PER+R improve performance, highlighting the importance of context-aware experience selection. In contrast, TD error-based PER leads to suboptimal results even when combined with intrinsic rewards. This suggests that prioritizing solely based on high TD error can misguide learning, as such experiences may not always be the most informative. In Cheetah-Run (Fig. 3, plot e), TD-PER performs best, but NSPER closely follows, demonstrating its adaptability to high-speed locomotion tasks. This result indicates that prioritization strategies should be tailored to specific environments, as more straightforward prioritization methods can sometimes be more effective. However, TD-PER exhibits inconsistency, performing well in Cheetah-Run but poorly in Walker-Walk, as noted in prior studies [29]. In contrast, NSPER maintains stable performance across diverse tasks, suggesting it provides a more reliable prioritization strategy. 5.2 Ablation Study: The Role of Novelty and Surprise in Prioritization This study evaluates various prioritization strategies to isolate the individual and combined effects of novelty and surprise. NoveltyPER prioritizes experiences based solely on novelty, SurprisePER based solely on surprise, and NSPER jointly leverages both signals. Solid curves denote variants that additionally use intrinsic rewards, while dashed curves show performance without them. As shown in Fig. 4, prioritization based on both novelty and surprise outperforms using either signal alone. This combination improves experience selection and, when also incorporated as intrinsic rewards, encourages more effective exploration. The results highlight the complementary nature of novelty and surprise in RL. Furthermore, integrating these signals as intrinsic rewards yields additional performance gains by promoting a more balanced selection of experiences, ultimately enhancing policy learning. In the Cheetah-Run task, prioritization alone is sufficient, and incorporating surprise as an intrinsic reward negatively impacts performance. Another key observation is that NoveltyPER outperforms SurprisePER in most tasks, suggesting that novelty-based prioritization, which encourages exploration of diverse states, is more beneficial than surprise-based prioritization, which focuses on unexpected transitions. Additionally, the performance of SurprisePER and SurprisePER+R remains similar in most cases, indicating that adding surprise as an intrinsic reward does not significantly contribute to performance improvements. However, when both novelty and surprise are used in prioritization and intrinsic rewards, the agent achieves the best overall performance, demonstrating their complementary roles in improving learning efficiency. 6 Conclusion In this work, we addressed the challenge of sample inefficiency in image-based RL by introducing NSPER and its extension NSPER+R, which integrate novelty and surprise as signals for replay prioritization and, in the case of NSPER+R, also as intrinsic rewards. This dual mechanism improves both replay quality and exploration, leading to faster convergence and more generalizable policies. Through an extensive evaluation of the DeepMind Control Suite using the PixelTD3 backbone, we showed that NSPER and NSPER+R outperform existing baselines, improving training efficiency in complex image-based tasks. Ablation studies further confirmed the complementary contributions of novelty and surprise to prioritization and exploration. Our research highlights the importance of using internal stimuli and enhanced sampling buffers to improve RL systems. Future work should explore additional intrinsic motivators and adaptively weight novelty and surprise. Our ablation study indicates that these signals impact learning efficiency differently, suggesting that dynamically adjusting their balance could refine the agent’s exploration strategy. References [1] J. Achiam and S. Sastry (2017) Surprise-based intrinsic motivation for deep reinforcement learning. arXiv preprint arXiv:1703.01732. Cited by: §1, §2.2. [2] M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, O. Pieter Abbeel, and W. Zaremba (2017) Hindsight experience replay. Advances in neural information processing systems 30. Cited by: §2.3. [3] A. Aubret, L. Matignon, and S. Hassas (2019) A survey on intrinsic motivation in reinforcement learning. arXiv preprint arXiv:1908.06976. Cited by: §2.2. [4] G. Baldassarre (2011) What are intrinsic motivations? a biological perspective. In 2011 IEEE international conference on development and learning (ICDL), Vol. 2, p. 1–8. Cited by: §1. [5] A. G. Barto (2013) Intrinsic motivation and reinforcement learning. Intrinsically motivated learning in natural and artificial systems, p. 17–47. Cited by: §1, §3.5. [6] M. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos (2016) Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems 29. Cited by: §1, §2.2. [7] E. K. Braun, G. E. Wimmer, and D. Shohamy (2018) Retroactive and graded prioritization of memory by reward. Nature communications 9 (1), p. 4886. Cited by: §1. [8] R. Carrasco-Davis, S. Lee, C. Clopath, and W. Dabney (2025) Uncertainty prioritized experience replay. arXiv preprint arXiv:2506.09270. Cited by: §2.1. [9] L. Chen, K. Lee, A. Srinivas, and P. Abbeel (2021) Improving computational efficiency in visual reinforcement learning via stored embeddings. Advances in Neural Information Processing Systems 34, p. 26779–26791. Cited by: §1. [10] E. L. Deci and R. M. Ryan (2013) Intrinsic motivation and self-determination in human behavior. Springer Science & Business Media. Cited by: §1. [11] F. E. Dorner (2021) Measuring progress in deep reinforcement learning sample efficiency. arXiv preprint arXiv:2102.04881. Cited by: §1. [12] W. Fedus, P. Ramachandran, R. Agarwal, Y. Bengio, H. Larochelle, M. Rowland, and W. Dabney (2020) Revisiting fundamentals of experience replay. In International conference on machine learning, p. 3061–3071. Cited by: item 1. [13] S. Ferraro, T. Van de Maele, P. Mazzaglia, T. Verbelen, and B. Dhoedt (2022) Computational optimization of image-based reinforcement learning for robotics. Sensors 22 (19), p. 7382. Cited by: §1. [14] S. Fujimoto, H. Hoof, and D. Meger (2018) Addressing function approximation error in actor-critic methods. In International conference on machine learning, p. 1587–1596. Cited by: §1, §3.2, item 1. [15] P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, and D. Meger (2018) Deep reinforcement learning that matters. In Proceedings of the AAAI conference on artificial intelligence, Vol. 32. Cited by: §4.3. [16] R. Houthooft, X. Chen, Y. Duan, J. Schulman, F. De Turck, and P. Abbeel (2016) Vime: variational information maximizing exploration. Advances in neural information processing systems 29. Cited by: §1. [17] Y. Huang, P. Peng, Y. Zhao, Y. Zhai, H. Xu, and Y. Tian (2023) Simoun: synergizing interactive motion-appearance understanding for vision-based reinforcement learning. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), Vol. , p. 176–185. External Links: Document Cited by: §1. [18] A. Jaegle, V. Mehrpour, and N. Rust (2019) Visual novelty, curiosity, and intrinsic reward in machine learning and the brain. Current opinion in neurobiology 58, p. 167–174. Cited by: §1, §2.2. [19] I. Kostrikov, D. Yarats, and R. Fergus (2020) Image augmentation is all you need: regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004.13649. Cited by: §4.1. [20] M. Kubovčík, I. Dirgová Luptáková, and J. Pospíchal (2023) Signal novelty detection as an intrinsic reward for robotics. Sensors 23 (8), p. 3985. Cited by: §1, §2.2. [21] M. Laskin, A. Srinivas, and P. Abbeel (2020) Curl: contrastive unsupervised representations for reinforcement learning. In International conference on machine learning, p. 5639–5650. Cited by: §4.1. [22] H. Le, K. Do, D. Nguyen, and S. Venkatesh (2024) Beyond surprise: improving exploration through surprise novelty.. In AAMAS, p. 1084–1092. Cited by: §1, §2.2. [23] B. Li, T. Lu, J. Li, N. Lu, Y. Cai, and S. Wang (2020) ACDER: augmented curiosity-driven experience replay. In 2020 IEEE International Conference on Robotics and Automation (ICRA), p. 4218–4224. Cited by: §2.3. [24] K. Li, A. Gupta, A. Reddy, V. H. Pong, A. Zhou, J. Yu, and S. Levine (2021) Mural: meta-learning uncertainty-aware rewards for outcome-driven reinforcement learning. In International conference on machine learning, p. 6346–6356. Cited by: §2.2. [25] X. Liu, Z. Xue, J. Pang, S. Jiang, F. Xu, and Y. Yu (2021) Regret minimization experience replay in off-policy reinforcement learning. Advances in neural information processing systems 34, p. 17604–17615. Cited by: §2.1. [26] V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu (2016) Asynchronous methods for deep reinforcement learning. In International conference on machine learning, p. 1928–1937. Cited by: §1. [27] Y. Oh, J. Shin, E. Yang, and S. J. Hwang (2021) Model-augmented prioritized experience replay. In International Conference on Learning Representations, Cited by: §1, §2.1. [28] P. Oudeyer and F. Kaplan (2007) What is intrinsic motivation? a typology of computational approaches. Frontiers in neurorobotics 1, p. 6. Cited by: §1, §2.2. [29] Y. Pan, J. Mei, A. Farahmand, M. White, H. Yao, M. Rohani, and J. Luo (2022) Understanding and mitigating the limitations of prioritized experience replay. In Uncertainty in Artificial Intelligence, p. 1561–1571. Cited by: §1, §2.1, §5.1. [30] D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell (2017) Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, p. 2778–2787. Cited by: §1, §2.2. [31] B. Saglam, F. B. Mutlu, D. C. Cicek, and S. S. Kozat (2022) Actor prioritized experience replay. arXiv preprint arXiv:2209.00532. Cited by: §1, §2.1. [32] T. Schaul, J. Quan, I. Antonoglou, and D. Silver (2015) Prioritized experience replay. arXiv preprint arXiv:1511.05952. Cited by: §1, §2.1, item 3. [33] A. Scorsoglio, R. Furfaro, R. Linares, and B. Gaudet (2020) Image-based deep reinforcement learning for autonomous lunar landing. In AIAA Scitech 2020 Forum, p. 1910. Cited by: §1. [34] Y. Seo, L. Chen, J. Shin, H. Lee, P. Abbeel, and K. Lee (2021) State entropy maximization with random encoders for efficient exploration. In International Conference on Machine Learning, p. 9443–9454. Cited by: §2.2. [35] D. Shohamy and R. A. Adcock (2010) Dopamine and adaptive memory. Trends in cognitive sciences 14 (10), p. 464–472. Cited by: §1. [36] R. Simmons-Edler, B. Eisner, D. Yang, A. Bisulco, E. Mitchell, S. Seung, and D. Lee (2019) Reward prediction error as an exploration objective in deep rl. arXiv preprint arXiv:1906.08189. Cited by: §2.1. [37] C. Sun, H. Qian, and C. Miao (2022) Cclf: a contrastive-curiosity-driven learning framework for sample-efficient reinforcement learning. arXiv preprint arXiv:2205.00943. Cited by: §1, §2.2, §2.3, item 7. [38] Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. d. L. Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancq, et al. (2018) Deepmind control suite. arXiv preprint arXiv:1801.00690. Cited by: §1, §4.1. [39] D. Valencia, H. Williams, Y. Xing, T. Gee, M. Liarokapis, and B. A. MacDonald (2024) Image-based deep reinforcement learning with intrinsically motivated stimuli: on the execution of complex robotic tasks. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 587–594. Cited by: §1, §2.2, §3.2, item 2, §4.1. [40] H. Yamani, Y. Xing, L. V. C. Ong, B. A. MacDonald, and H. Williams (2025) Reward prediction error prioritisation in experience replay: the rpe-per method. arXiv preprint arXiv:2501.18093. Cited by: §1, §2.1, item 5. [41] D. Yarats, A. Zhang, I. Kostrikov, B. Amos, J. Pineau, and R. Fergus (2021) Improving sample efficiency in model-free reinforcement learning from images. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35, p. 10674–10681. Cited by: §3.2. [42] D. Zha, K. Lai, K. Zhou, and X. Hu (2019) Experience replay optimization. arXiv preprint arXiv:1906.08387. Cited by: §2.1, §2.1. [43] R. Zhao and V. Tresp (2019) Curiosity-driven experience prioritization via density estimation. arXiv preprint arXiv:1902.08039. Cited by: §2.3. [44] J. Zheng and Y. Song (2024) Effective representation learning is more effective in reinforcement learning than you think. In 2024 IEEE International Conference on Robotics and Automation (ICRA), Vol. , p. 9176–9182. External Links: Document Cited by: §1.