Paper deep dive
Understanding and Stabilizing Deep Q-Learning via Controlled Bootstrapping and Regulated Value Dynamics
Bozhou Chen, Yongyi Wang, Hanyu Liu, Xionghui Yang, Wenxin Li
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 8/23/2026, 1:36:07 AM
Summary
This paper analyzes the instability of Deep Q-Learning (DQL) through three interacting mechanisms: operator-level bias in Bellman bootstrapping (specifically a 'self-reinforcing trap' where positive rewards amplify value estimates for repeated actions), estimator-level sensitivity to regression noise affecting greedy action selection, and parameter-dynamics imbalance (loss of plasticity) under aggressive data reuse. The authors propose stabilization principles including controlled bootstrapping, ensemble quantile estimation, and spike-based parameter regulation, demonstrating improved stability and performance on Atari-100K and Procgen benchmarks.
Entities (10)
Relation Signals (7)
Atari 100k → usedin → Experimental Evaluation
confidence 99% · Experiments on Atari-100K and Procgen demonstrate competitive performance and improved training stability.
Deep Q-Learning → suffersfrom → instability
confidence 98% · Deep Q-learning (DQL) has achieved remarkable empirical success in reinforcement learning, yet its training process remains notoriously unstable.
Double DQN → mitigates → Overestimation Bias
confidence 95% · Double Q-learning [17] and its deep variant Double DQN [35] mitigate this issue by decoupling action selection from evaluation.
Spike Ratio → measures → Parameter Imbalance
confidence 92% · spike ratio, which measures the concentration of extreme parameter magnitudes within each layer and serves as a practical indicator of parameter imbalance.
Self-Reinforcing Trap → causedby → Bellman Bootstrapping
confidence 90% · Recursive Bellman bootstrapping may introduce structural bias in target construction... we observe a previously overlooked feedback mechanism, which we term the self-reinforcing trap.
Self-Reinforcing Trap → involves → Positive Rewards
confidence 88% · In reward-bearing transitions, value amplification induced by positive rewards interacts with action-conditioned representation generalization.
Ensemble Quantile Estimation → stabilizes → Deep Q-Learning
confidence 85% · derive stabilization principles for controlled bootstrapping, ensemble quantile estimation, and spike-based parameter regulation.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Deep Q-learning (DQL) has achieved remarkable empirical success in reinforcement learning, yet its training process remains notoriously unstable. Existing studies often attribute instability to isolated factors such as overestimation bias or representation learning issues, lacking a unified understanding of how different sources of instability interact during recursive value estimation. In this work, we provide a systematic analysis of instability in deep Q-learning from three complementary perspectives: operator-level bias in Bellman bootstrapping, estimator-level sensitivity of greedy action selection to regression noise, and parameter-dynamics imbalance under aggressive data reuse. We identify a reward-triggered self-reinforcing trap and characteristic parameter spike dynamics, then derive stabilization principles for controlled bootstrapping, ensemble quantile estimation, and spike-based parameter regulation. Experiments on Atari-100K and Procgen demonstrate competitive performance and improved training stability.
Tags
Links
- Source: https://arxiv.org/abs/2608.16182v1
- Canonical: https://arxiv.org/abs/2608.16182v1
Trouble viewing inline? Open PDF directly →
Full Text
86,770 characters extracted from source content.
Expand or collapse full text
Understanding and Stabilizing Deep Q-Learning via Controlled Bootstrapping and Regulated Value Dynamics Bozhou Chen Yongyi Wang Hanyu Liu Xionghui Yang Wenxin Li Thanks: All authors are with the School of Computer Science, Peking University, Beijing, China. Corresponding author: Wenxin Li. E-mail: 2301111899@stu.pku.edu.cn; wangyongyi@pku.edu.cn; 2301111944@stu.pku.edu.cn; yangxionghui@stu.pku.edu.cn; lwx@pku.edu.cn Abstract Deep Q-learning (DQL) has achieved remarkable empirical success in reinforcement learning, yet its training process remains notoriously unstable. Existing studies often attribute instability to isolated factors such as overestimation bias or representation learning issues, lacking a unified understanding of how different sources of instability interact during recursive value estimation. In this work, we provide a systematic analysis of instability in deep Q-learning from three complementary perspectives: operator-level bias in Bellman bootstrapping, estimator-level sensitivity of greedy action selection to regression noise, and parameter-dynamics imbalance under aggressive data reuse. We identify a reward-triggered self-reinforcing trap and characteristic parameter spike dynamics, then derive stabilization principles for controlled bootstrapping, ensemble quantile estimation, and spike-based parameter regulation. Experiments on Atari-100K and Procgen demonstrate competitive performance and improved training stability. Index Terms: Deep reinforcement learning, Deep Q-learning, Bellman bootstrapping, Ensemble learning, Distributional reinforcement learning, Training stability, Parameter regulation I Introduction Deep Q-learning has become a cornerstone method in deep reinforcement learning (RL), widely applied in domains ranging from video games to robotics. Its appeal lies in learning effective policies directly from high-dimensional observations via neural network approximation. A large body of prior work has studied instability in value learning from specific perspectives. Classical analyses focus on overestimation bias induced by the max operator in Bellman updates. Standard Q-learning applies the maximization operator to noisy value estimates, which leads to systematically optimistic targets. Double Q-learning [17] and its deep variant Double DQN [35] mitigate this issue by decoupling action selection from evaluation. More recent methods further reduce bias or variance through ensemble estimation or distributional value modeling. However, empirical evidence suggests that instability in deep Q-learning cannot be fully explained by maximization bias alone. In practice, value learning exhibits complex feedback effects involving bootstrap targets, stochastic regression, and evolving network representations. These mechanisms interact with each other through the recursive structure of value updates, yet they are typically studied in isolation. In this work, we revisit instability in deep Q-learning from a unified perspective and identify three interacting mechanisms that jointly shape learning dynamics: Operator-level instability. Recursive Bellman bootstrapping may introduce structural bias in target construction. Beyond classical overestimation, we observe a previously overlooked feedback mechanism, which we term the self-reinforcing trap. In reward-bearing transitions, value amplification induced by positive rewards interacts with action-conditioned representation generalization. As a result, the same action may be repeatedly selected in bootstrap targets, creating a feedback loop that continuously amplifies its value. Estimator-level instability. Deep Q-learning is fundamentally a regression problem in which the value function is learned through stochastic approximation. Even when estimation noise is unbiased, greedy action selection depends on relative value differences. When action gaps are small, regression noise may alter greedy decisions, which subsequently changes the distribution of collected experience. In this way, estimation noise propagates through the control loop and may destabilize learning dynamics. Parameter-dynamics instability. We further observe that neural networks used for value approximation may gradually develop imbalanced parameter distributions during training, particularly under high replay ratios. In such regimes, a small subset of parameters may grow disproportionately large while the majority remain relatively unchanged. This phenomenon is closely related to the loss of network plasticity [28, 33, 23]. To better characterize this effect, we introduce a simple diagnostic statistic termed the spike ratio, which measures the concentration of extreme parameter magnitudes within each layer and serves as a practical indicator of parameter imbalance. These observations suggest that instability in deep Q-learning arises from interacting feedback mechanisms across target construction, value estimation, and parameter evolution, requiring coordinated control rather than isolated heuristics. Based on this analysis, we derive several practical stabilization principles for value learning. These include controlling recursive amplification in bootstrap targets, reducing decision variance in value regression, and maintaining parameter plasticity during prolonged training. We instantiate these principles in a practical learning framework that integrates controlled bootstrapping, ensemble quantile regression, and parameter regulation. Extensive experiments on Atari-100K and Procgen demonstrate that the resulting algorithm achieves strong performance and improved training robustness. More importantly, the empirical results support the proposed instability analysis and illustrate how the identified mechanisms influence learning dynamics in practice. Our main contributions are summarized as follows: • We provide a systematic analysis of instability in deep Q-learning, identifying three interacting mechanisms: operator-level bias in bootstrapping, estimator-level decision sensitivity to regression noise, and parameter-dynamics imbalance during training. • We uncover a previously overlooked feedback mechanism termed the self-reinforcing trap, where reward-driven value amplification interacts with representation generalization to produce biased bootstrap dynamics. • We reveal the role of parameter spike dynamics in the loss of network plasticity and introduce the spike ratio as a practical statistic for monitoring training stability. • Based on these insights, we derive stabilization principles for value learning and instantiate them in a practical algorithm that achieves competitive performance on Atari-100K and Procgen benchmarks. The remainder of this paper is as follows. Section I reviews related work on bootstrapping bias, distributional learning, ensemble estimation, etc. Section I presents the instability analysis from operator-level, estimator-level, and parameter-dynamics perspectives. Section IV introduces stabilization principles derived from this analysis, and Section V presents their algorithmic instantiation. Section VI provides experimental evaluation and mechanism studies. Finally, Section VII concludes the paper and discusses future directions. I Related Work I-A Overestimation and Stable Bootstrapping Targets The maximization operator in the Bellman update introduces systematic overestimation when applied to noisy value predictions. Thrun and Schwartz [34] first identified this bias in the tabular setting, and under function approximation such bias may accumulate through recursive bootstrapping. Double Q-learning [17] mitigates this issue by decoupling action selection from evaluation using two estimators, and Double DQN [35] extends this idea to deep RL using online and target networks. Subsequent works further refine bootstrap target construction. Averaged-DQN [4] reduces variance by averaging past networks, while Maxmin DQN [19] selects the minimum among an ensemble of critics to control optimism. In continuous control, TD3 [14] adopts clipped double Q-learning, taking the minimum of two critics as a conservative target. Ensemble-based strategies provide additional bias control. REM [2] enforces Bellman consistency on random convex combinations of Q-heads, and REDQ [9] leverages larger ensembles with randomized subsampling to stabilize training under high update-to-data ratios. These approaches primarily stabilize value learning at the level of bootstrap target construction by reducing maximization bias or target variance. However, they mainly address the bias properties of the Bellman target itself, while other factors affecting the dynamics of recursive value learning remain less explored. I-B Distributional Reinforcement Learning and Ensemble Estimation Distributional RL models, such as C51 [6], the full return distribution rather than only its expectation, capturing richer information about uncertainty in value estimation. C51 introduced a categorical representation over a fixed support and demonstrated improved stability on Atari. QR-DQN [12] replaced categorical projections with quantile regression, while IQN [11] parameterized the entire quantile function to allow flexible distributional modeling. In continuous control, TQC [18] combines distributional critics with an ensemble of Q-functions and truncates the highest quantile atoms to suppress overestimation, providing a softer alternative to scalar minimum operators. Ensemble techniques improve both value stability and exploration. Bootstrapped DQN [29] trained multiple Q-heads with bootstrapped data partitions, enabling temporally consistent deep exploration. REM [2] performs ensemble aggregation across critics to obtain more accurate value estimates. SUNRISE [22] introduced uncertainty-weighted Bellman backups, down-weighting unreliable targets to reduce error propagation. SPQR [21] explicitly addressed ensemble collapse by injecting structured noise to preserve independent estimation behavior. These methods improve robustness at the estimation level by modeling return distributions or aggregating multiple value predictors. However, they are primarily motivated by improving prediction accuracy or uncertainty estimation, while the role of regression noise in affecting decision and interaction dynamics remains less explicitly analyzed. I-C Value Learning in Data-Efficient Regimes Recent progress in data-efficient reinforcement learning has significantly improved performance under limited interaction budgets, such as the Atari-100K setting. A central trend in this regime is the integration of strong representation learning objectives into value-based methods. CURL [20] introduced contrastive representation learning for pixel-based RL, improving sample efficiency through latent consistency. SPR [31] and SR-SPR [13] further leveraged self-predictive representations by enforcing temporal consistency in latent space. DrQ [38] demonstrated that simple data augmentation substantially improves data efficiency and generalization. More recent methods such as BBF [32], SGF [30], Drama [37], IRIS [25], and STORM [39] combined stronger regularization, auxiliary objectives, and architectural refinements to further push data-efficient performance. In addition to representation learning, high replay ratios and aggressive update-to-data strategies have been widely adopted in data-limited regimes. DER [36] and BBF [32] showed that increasing gradient updates per environment interaction can substantially improve sample efficiency. However, heavy data reuse may also amplify bootstrap errors and increase sensitivity to estimation noise, particularly under recursive value updates. These methods substantially improve sample efficiency through enhanced representation learning and increased data reuse. However, instability still fundamentally originates from recursive bootstrapping dynamics. I-D Optimization Dynamics and Network Plasticity Beyond bias in bootstrap targets, a growing body of work has investigated instability in deep reinforcement learning from the perspective of optimization dynamics and network behavior during training. Early stabilization techniques primarily aim to mitigate the feedback effects introduced by bootstrapped updates. Target networks [26] provide a slowly updated objective to reduce oscillations in value estimates. Additional techniques such as the Huber loss [26] limit the influence of extreme TD errors, and safe backup operators including Retrace(λ) truncate or down-weight multi-step returns to prevent uncontrolled value propagation. Subsequent studies examined instability from the perspective of optimization under function approximation. Spectral normalization [15] constrains the Lipschitz constant of Q-networks to stabilize value prediction. Creus Castanyer et al. [7] identified unstable gradient flow in deep Q-networks under non-stationary data and proposed architectural and optimization modifications. Related analyses [24, 27] showed that off-policy algorithms remain sensitive to estimation errors amplified through bootstrapped updates. Another line of work focuses on the loss of network plasticity during prolonged reinforcement learning training. Nikishin et al. [28] identified the primacy bias, where agents overfit early interaction data and struggle to adapt under high replay ratios. Sokar et al. [33] reported the dormant neuron phenomenon, observing that a large fraction of neurons become inactive during training. Lyle et al. [23] further connected this loss of plasticity to rank collapse in learned representations, demonstrating that common architectural and optimization choices can accelerate representational degradation. To mitigate plasticity loss, several intervention strategies have been proposed. Periodic parameter resets [28] aim to counteract primacy bias but may discard useful structure. Shrink-and-Perturb [5] restores gradient flow by shrinking parameters toward zero while injecting noise, and ReDo [33] selectively reinitializes dormant neurons based on activation statistics to recycle unused network capacity. Together, these studies highlight that instability in deep reinforcement learning is closely related to optimization dynamics and the evolution of network representations during prolonged bootstrapped training. I-E World Models and Model-Based Reinforcement Learning Recent years have seen rapid advances in model-based reinforcement learning, particularly in data-efficient settings such as Atari-100K. These methods improve sample efficiency by learning an explicit environment model and performing planning in latent space. IRIS [25] introduced a Transformer-based world model operating on discrete VQ-VAE latents, achieving strong sample efficiency without explicit lookahead search. DreamerV3 [16] scaled recurrent latent world models into a unified algorithm spanning over 150 tasks, emphasizing normalization and architectural stability to ensure cross-domain robustness. STORM [39] combined Transformer architectures with stochastic latent variables to enhance modeling flexibility, while DIAMOND [3] leveraged diffusion-based generation to better preserve visual details that discrete latent models may discard. DART [1] and Drama [37] further explored trajectory-aware and state-space modeling architectures, continuing the trend toward increasingly expressive and stable world models. These methods address sample efficiency through model learning, while our work focuses on the stability of value learning itself. I Analysis of Instability Mechanisms Although instability in deep Q-learning has long been recognized, existing explanations typically focus on individual phenomena such as maximization bias or representation learning effects. In practice, however, value learning forms a recursive feedback system in which multiple components interact through the Bellman update and environment interaction loop. Instability therefore often arises not from a single source, but from the coupling of several mechanisms that influence the dynamics of recursive value estimation. In this section, we analyze instability in deep Q-learning from three complementary perspectives corresponding to different stages of the value learning process. First, at the operator level, recursive Bellman bootstrapping may introduce structural bias in target construction, leading to feedback amplification effects beyond classical maximization bias. Second, at the estimator level, the stochastic nature of value regression may perturb greedy action selection, causing regression noise to propagate through the control loop and alter the data distribution encountered during training. Third, at the parameter-dynamics level, neural network parameters may gradually evolve toward imbalanced configurations under aggressive data reuse, reducing the network’s ability to adapt to changing value targets. These mechanisms interact through the recursive structure of value learning. Amplified bootstrap targets influence value regression, noisy estimation alters control decisions and experience collection, and evolving parameter configurations further shape the learning dynamics. Together, these feedback pathways create complex instability patterns that are difficult to explain from any single perspective. The following subsections examine these mechanisms in detail and provide empirical evidence illustrating their roles in deep Q-learning instability. I-A Operator-Level Instability: Unstable Recursive Bootstrapping Operator-level instability arises from structural biases in the Bellman backup. Standard Q-learning constructs targets via r+γmaxa′Qθ¯(s′,a′).r+γ _a Q_ θ(s ,a ). (1) Because the maximum is taken over noisy estimates, [maxaQ^(s,a)]≥maxa[Q^(s,a)]E\! [ _a Q(s,a) ]\;≥\; _aE\! [ Q(s,a) ] (2) leading to classical overestimation bias. In deep reinforcement learning, an additional source of overestimation arises from function approximation and representation generalization. In discrete control, the action-value function is commonly parameterized as Qθ(s,a)=fθ1(ϕθ2(s),a),Q_θ(s,a)=f_ _1\! ( _ _2(s),a ), (3) where ϕθ2(⋅) _ _2(·) denotes a shared state encoder and fθ1(⋅,a)f_ _1(·,a) is an action-conditioned output head. Under this architecture, generalization primarily occurs across states for the same action, while value estimates for different actions are modeled by separate heads and are therefore not directly coupled by representation generalization. As a result, for a fixed action a, the corresponding value estimates satisfy Qθ(s,a)≈Qθ(s′,a)Q_θ(s,a)≈ Q_θ(s ,a), whereas no such coupling is induced between Qθ(s,a)Q_θ(s,a) and Qθ(s′,a′)Q_θ(s ,a ) for a′≠a ≠ a. This action-conditioned generalization improves sample efficiency, but it also creates an implicit pathway through which bootstrapped targets can propagate across nearby states for the same action. A particularly problematic case arises for reward-bearing transitions. Consider a replayed transition (s,a,r,s′)(s,a,r,s ) with r>0r>0. Due to action-conditioned generalization, the update of Q(s,a)Q(s,a) driven by the positive reward also affects the estimate of Q(s′,a)Q(s ,a), since the two states induce similar representations and share the same action head. As a result, after a number of updates, the value Q(s′,a)Q(s ,a) tends to become relatively large compared to other actions at state s′s . Fig. 1: Reward-triggered self-reinforcing bias in Atari Alien. The dashed blue curve (left axis) shows the recent 10-episode mean return during training, while the solid red curve (right axis) reports the minibatch statistic Count(a=argmaxa′Q(s′,a′))Count\! (a= _a Q(s ,a ) ), the number of sampled transitions whose greedy bootstrap action at the next state coincides with the original action. The action space contains 18 actions and the minibatch size is 32. The consistently elevated counts indicate a strong coupling between a and the bootstrap action a′∗a *, providing empirical evidence for the self-reinforcing trap discussed in this section. When computing the bootstrapped target, the max operator selects the action a′∗∈argmaxa′Q(s′,a′)a *∈ _a Q(s ,a ). Because Q(s′,a)Q(s ,a) has already been amplified through representation generalization induced by the positive reward r>0r>0, the maximization step is biased toward selecting the same action, i.e., a′∗=a *=a. This bias is not caused by the max operator alone, but the interaction between reward-driven value amplification and action-conditioned generalization. As illustrated in Fig. 1, we empirically examine this phenomenon in the Atari Alien environment by recording the number of transitions for which a=argmaxa′Q(s′,a′)a= _a Q(s ,a ). The results show a pronounced bias toward selecting the previous action, indicating a strong correlation between a′∗a * and a. Once a′∗=a *=a is selected, the resulting target r+γQ(s′,a)r+γ Q(s ,a) further increases Q(s,a)Q(s,a). This creates a positive feedback loop in which the reward-induced increase of Q(s,a)Q(s,a) propagates to Q(s′,a)Q(s ,a), and the maximization step repeatedly selects the same action. Importantly, this mechanism is specific to transitions with r>0r>0. For transitions with zero reward, no persistent upward shift is introduced, and the max operator does not exhibit the same systematic preference for reselecting the original action. Notably, the magnitude of this self-reinforcing amplification admits an upper bound in an idealized repeated-replay scenario. Consider repeatedly replaying the same reward-bearing transition (s,a,r,s′)(s,a,r,s ) with r>0r>0, and suppose the maximization step persistently reselects the same action so that a′∗=a *=a holds throughout training, as indicated by our empirical observation. In this case, the bootstrapped update reduces to a one-dimensional fixed-point iteration Qk+1(s,a)=r+γQk(s′,a).Q_k+1(s,a)\;=\;r+γ\,Q_k(s ,a). (4) Under the self-reinforcing coupling described above, Qk(s′,a)Q_k(s ,a) is repeatedly driven upward in tandem with Qk(s,a)Q_k(s,a), and the iteration effectively behaves as Qk+1(s,a)≈r+γQk(s,a),Q_k+1(s,a)\;≈\;r+γ\,Q_k(s,a), (5) whose unique fixed point is Q(s,a)→r1−γ.Q(s,a)\;→\; r1-γ. (6) Therefore, even though the feedback loop can systematically inflate Q-values, its amplification under repeated replay of a single transition is bounded and converges to a finite limit determined by the reward scale and discount factor. I-B Estimator-Level Instability: Interaction Drift Induced by Regression Noise Even when the Bellman operator is unbiased and target construction is structurally stable, deep Q-learning remains sensitive to estimation noise during action selection. Fundamentally, deep Q-learning is a stochastic regression problem: the value function is optimized to minimize a squared-error objective with bootstrapped targets. Although such regression may be unbiased in expectation, control decisions rely on the maximization of noisy value estimates. Suppose the learned action-value function satisfies Qθ(s,a)=Q∗(s,a)+εa,Q_θ(s,a)=Q^*(s,a)+ _a, (7) where εa _a is a zero-mean estimation error with [εa]=0E[ _a]=0. The greedy policy selects aθ∗(s)=argmaxaQθ(s,a).a_θ^*(s)= _aQ_θ(s,a). (8) Let Δ(s) (s) denote the action gap between the optimal action a∗a^* and the second-best alternative: Δ(s)=Q∗(s,a∗)−maxa≠a∗Q∗(s,a). (s)=Q^*(s,a^*)- _a≠ a^*Q^*(s,a). (9) The probability of selecting a suboptimal action is therefore Pr(aθ∗(s)≠a∗)=Pr(εa′−εa∗>Δ(s)), (a_θ^*(s)≠ a^*)= ( _a - _a^*> (s)), (10) for some a′≠a∗a ≠ a^*. Thus, even unbiased estimation noise can induce unstable action selection when the action gap is small. While this does not introduce systematic value bias, it directly affects the quality of decisions made during interaction with the environment. Because the behavior policy determines the distribution of collected data, noisy action selection may lead the agent to generate lower-return trajectories. Learning slows not due to inefficient reuse of samples, but because the interaction policy itself produces less informative experience. In this sense, estimator-level instability degrades control reliability and indirectly limits performance improvement. These observations motivate the need for variance-reduction techniques in value regression, so that action selection remains stable under stochastic estimation noise. I-C Parameter-Dynamics Instability: Layer-wise Parameter Imbalance and Plasticity Loss Beyond operator-level bias and estimator-level decision sensitivity, instability may also arise from the evolution of parameters under non-stationary sampling. In reinforcement learning, the data distribution is determined by the behavior policy, which continuously changes throughout training. As a result, the representation learned by the network must remain adaptable to shifting state-action distributions. However, repeated optimization on a limited or slowly evolving data distribution can progressively bias parameter updates toward frequently observed patterns. This effect exists in general, but becomes increasingly pronounced as the replay ratio grows. When past experiences are reused many times before sufficient new interactions are collected, the influence of the current data distribution is amplified. Over time, this can drive certain weights to dominate their layers, while the remaining weights receive comparatively weaker updates. The resulting parameter distribution becomes increasingly skewed. To quantify this imbalance, we introduce a diagnostic statistic termed the spike ratio. For a given layer ℓ with parameters θℓ _ , we define SpikeRatioℓ=‖θℓ‖∞Quantile0.99(|θℓ|).SpikeRatio_ = \| _ \|_∞Quantile_0.99(| _ |). (11) It measures the dominance of extreme parameter values relative to the bulk of the distribution. An increasing spike ratio indicates that a small subset of weights grows disproportionately large, revealing an imbalanced representation. Fig. 2: Layer-wise spike ratio evolution during training under different replay ratios. Each row corresponds to a network layer, and color intensity indicates spike magnitude. Higher replay ratios lead to more pronounced spike growth, particularly in post-encoder MLP layers. Further details are provided in the supplementary materials. Fig. 9 illustrates the evolution of SpikeRatioℓSpikeRatio_ under different replay ratios (r=1,2,4r=1,2,4). While spike growth is observable even at lower replay ratios, it becomes more pronounced as the replay ratio increases, particularly in post-encoder MLP layers. This pattern indicates that replay amplification systematically accelerates parameter imbalance. Parameter-dynamics instability does not necessarily lead to numerical divergence of value estimates. Instead, it reflects a gradual loss of representational plasticity. As the data distribution shifts with policy improvement, a skewed parameter configuration may adapt more slowly, thereby constraining long-term performance improvement. These observations underscore the importance of preserving balanced parameter evolution throughout training. IV Stabilization Principles for Value Learning The analysis in Section I reveals that instability in deep Q-learning arises from three interacting mechanisms: operator-level bias in recursive bootstrapping, estimator-level sensitivity of greedy action selection to regression noise, and parameter-dynamics imbalance under non-stationary sampling. These mechanisms interact through the recursive structure of value learning. Instability may emerge when bootstrap targets amplify value estimates, when noisy regression alters control decisions, or when network parameters gradually lose plasticity during prolonged training. Stabilizing value learning therefore requires coordinated regulation of these components rather than isolated heuristics. Based on the preceding analysis, we derive three stabilization principles for deep Q-learning: • Controlled bootstrapping. Recursive Bellman updates must be regulated to prevent feedback amplification during target construction. • Variance-aware value estimation. Reducing estimation variance improves the reliability of greedy action selection and stabilizes interaction dynamics. • Parameter-dynamics regulation. Maintaining balanced parameter evolution preserves network plasticity under prolonged replay. In the following, we describe practical mechanisms that instantiate these principles. Together they form a controlled optimization procedure for stabilizing value learning. IV-A Controlled Bootstrapping Section I-A shows that operator-level instability arises not only from classical maximization bias, but also from a previously overlooked reward-triggered self-reinforcing pathway in recursive bootstrapping. IV-A1 Cross-Model Decoupling for Action Selection and Evaluation The overestimation phenomenon in Eq. (2) arises from using the same noisy estimator for both action selection and value evaluation. Double DQN alleviates this issue by decoupling these roles using online and target networks. We generalize this idea to an ensemble setting. Let Q=Q^1,…,Q^mQ=\ Q_1,…, Q_m\ (12) denote an ensemble of value functions. For each model Q^i Q_i, we select the bootstrap action using another model Q^j Q_j (j≠ij≠ i), while the value is evaluated using Q^i Q_i: yi=r+γQ^i(s′,argmaxa′Q^j(s′,a′)).y_i=r+γ Q_i (s , _a Q_j(s ,a ) ). (13) This structurally eliminates selection-evaluation coupling. When the estimation noises of Q^i Q_i and Q^j Q_j are weakly correlated as encouraged by independent initialization and separate bootstrap targets, the selection noise and evaluation noise are effectively decoupled, which substantially reduces the upward bias induced by the max operator. IV-A2 Action-Decoupling for Reward-Bearing Transitions The instability analysis in Section I-A further reveals a feedback pathway termed the self-reinforcing trap. When a reward-bearing transition (s,a,r,s′)(s,a,r,s ) is repeatedly replayed, representation generalization may increase both Q(s,a)Q(s,a) and Q(s′,a)Q(s ,a) simultaneously, making action a likely to be selected again during the bootstrap step. To break this feedback loop, we impose a constraint on reward-bearing transitions: ai′≠aif r>0.a _i≠ a r>0. (14) This constraint prevents the bootstrap target from repeatedly selecting the same action that produced the reward, thereby disrupting the amplification pathway identified in the instability analysis. IV-A3 Bounded Bellman Updates Even with decoupled bootstrapping, recursive updates may gradually increase the numerical scale of value estimates. To prevent uncontrolled growth, we impose an upper bound on target values derived from the discounted return along greedy trajectories. Given a greedy trajectory T, we define a trajectory-level bound B()=maxtGt,B(T)= _tG_t, (15) where GtG_t denotes the discounted return. This bound provides a natural constraint on the numerical scale of value estimates during training. Further implementation details are provided in the supplementary material. IV-B Variance-Aware Value Estimation Section I-B shows that even unbiased regression noise may alter greedy action selection when action gaps are small. This effect may destabilize interaction dynamics. To reduce estimation variance, we adopt ensemble quantile regression. Following QR-DQN, we model the return distribution using K quantile estimates Zθ(s,a)=1K∑i=1Kδzθ(i)(s,a).Z_θ(s,a)= 1K _i=1^K _z_θ^(i)(s,a). (16) The scalar value estimate is Qθ(s,a)=1K∑i=1Kzθ(i)(s,a).Q_θ(s,a)= 1K _i=1^Kz_θ^(i)(s,a). (17) Given bootstrap action a′∗a *, the target quantiles are y(j)=r+γzθ¯(j)(s′,a′∗),y^(j)=r+γ z_ θ^(j)(s ,a *), (18) and parameters are optimized via the quantile regression loss ℒQR(θ)=1K2∑i=1K∑j=1Kρτiκ(y(j)−zθ(i)(s,a)).L_QR(θ)= 1K^2 _i=1^K _j=1^K _ _i^κ\! (y^(j)-z_θ^(i)(s,a) ). (19) To further reduce variance, we maintain an ensemble of quantile networks Zθ1,…,Zθm.\Z_ _1,…,Z_ _m\. (20) The ensemble-averaged value used for action selection is Q¯(s,a)=1m∑ℓ=1m1K∑i=1Kzθℓ(i)(s,a). Q(s,a)= 1m _ =1^m 1K _i=1^Kz^(i)_ _ (s,a). (21) Ensemble aggregation reduces both intra-model and inter-model estimation variance, improving the reliability of greedy action selection. IV-C Parameter-Dynamics Regulation Section I-C reveals that high replay ratios may gradually distort parameter distributions, leading to reduced network plasticity. To monitor this phenomenon, we introduce the spike ratio, defined as Eq. (11). This statistic measures the dominance of extreme parameter values relative to the bulk of the distribution. When the spike ratio of a layer exceeds a predefined threshold, corrective intervention is triggered by resetting the affected parameters. This mechanism restores distributional balance and helps preserve long-term adaptability of the value network. V Algorithm Instantiation The mechanisms described above provide practical implementations of the stabilization principles derived from the instability analysis. Together they form a controlled optimization procedure for value learning. Input: Replay buffer D; ensemble size m; number of quantiles K; Online networks Qi(⋅,θi)i=1m\Q_i(·; _i)\_i=1^m and target networks Qi(⋅,θi−)i=1m\Q_i(·; _i^-)\_i=1^m; Quantile fractions τj=j−0.5Kj=1K\ _j= j-0.5K\_j=1^K; exploration rate ϵε Output: Updated ensemble parameters θii=1m\ _i\_i=1^m 1 for t=1t=1 to T do 2 3 // Environment interaction Observe state sts_t 4 if rand()<ϵrand()<ε then 5 Select at∼Uniform()a_t (A) 6 else 7 Select greedy action according to ensemble-averaged value(Eq. (21)): at=argmaxa∈Q¯(st,a)a_t= _a Q(s_t,a) end if 8 Execute ata_t, observe (rt,st+1)(r_t,s_t+1), and store (st,at,rt,st+1)(s_t,a_t,r_t,s_t+1) in D 9 10 // Prioritized experience replay Sample a mini-batch (s(b),a(b),r(b),s′(b))b=1B\(s^(b),a^(b),r^(b),s (b))\_b=1^B from D 11 12 // Bootstrap action selection for i=1i=1 to m do 13 Aggregate distributional value estimation according to Eq. (17) 14 Decoupling reward-bearing transitions: 15 if r(b)>0r^(b)>0 then 16 Qθi−(s′(b),a(b))=−∞Q_ _i^-(s (b),a^(b))=-∞ end if 17 Select bootstrap action: ai∗=argmaxa′∈Qθi−(s′,a′)a_i^*= _a Q_ _i^-(s ,a ) end for 18 19 Sample a random permutation π over 1,…,m\1,…,m\ 20 21 for i=1i=1 to m do 22 // Cross-model bootstrapping for j′=1j =1 to K do 23 Compute target quantiles: yi(b,j′)=r(b)+γzi,j′(s′(b),aπ(i)∗,θi−)y_i^(b,j )=r^(b)+γ\,z_i,j \! (s (b),a^*_π(i); _i^- ) end for 24 Compute quantile regression loss and update parameters according to Eq. (19) 25 end for 26 27 Periodically update target networks θi−←θi _i^-← _i 28 Periodically monitor layer-wise spike ratios according to Eq. (11) 29 if Layerℓs′spikeratioisgreaterthanthresholdLayer\ s\ spike\ ratio\ is\ greater\ than\ threshold then 30 Reset layer ℓ ’s parameters end if 31 end for 32 Algorithm 1 Bootstrapping Control and Ensemble Quantile Regression Algorithm 1 summarizes the overall training process. At each interaction step (Lines 1–8), the agent observes the current state sts_t (Line 2) and selects an action using ϵε-greedy exploration (Lines 3–7). With probability ϵε, a random action is sampled uniformly (Line 4). Otherwise, the agent acts greedily w.r.t. the ensemble-averaged value Q¯(s,a) Q(s,a) defined in Eq. (21) (Line 6). The resulting transition (st,at,rt,st+1)(s_t,a_t,r_t,s_t+1) is then stored in the replay buffer D (Line 8). For learning, we sample a prioritized minibatch (s(b),a(b),r(b),s′(b))b=1B\(s^(b),a^(b),r^(b),s (b))\_b=1^B from D (Line 9). Next, we determine bootstrap actions for each ensemble member using its own target network (Lines 10–17). Specifically, for each member i, we first form the scalar action values from its quantile outputs (Line 11, Eq. (17)) and then apply the reward-bearing decoupling rule (Lines 12–15): for any sampled transition b with r(b)>0r^(b)>0, we mask the self-reinforcing candidate by setting Qθi−(s′(b),a(b))=−∞Q_ _i^-(s (b),a^(b))=-∞, which enforces ai∗(s′(b))≠a(b)a_i^*(s (b))≠ a^(b) during maximization. We then compute the greedy bootstrap action ai∗(s′(b))=argmaxa′∈Qθi−(s′(b),a′)a_i^*(s (b))= _a Q_ _i^-(s (b),a ) (Line 16). To decouple action selection from value evaluation across models, we sample a random permutation π over ensemble indices (Line 18). During the subsequent update (Lines 19–24), member i evaluates the bootstrap action selected by another member π(i)π(i). Concretely, for each quantile index j′∈1,…,Kj ∈\1,…,K\, we construct the target quantile yi(b,j′)=r(b)+γzi,j′(s′(b),aπ(i)∗(s′(b)),θi−)y_i^(b,j )=r^(b)+γ\,z_i,j (s (b),a_π(i)^*(s (b)); _i^-) (Line 21), i.e., the action comes from the permuted selector while the quantile evaluation uses member i’s target network. Member i is then updated by minimizing the quantile regression objective in Eq. (19) (Line 23). Finally, target networks are periodically synchronized (Line 25), and we monitor layer-wise spike ratios (Line 26, Eq. (11)) to detect over-adaptation. If a layer’s spike ratio exceeds a predefined threshold, we reset that layer’s parameters to restore training stability (Lines 27–29). VI Experiments Experiments evaluate the proposed analysis and the derived stabilization principles. First, we evaluate whether the algorithm instantiated from the proposed principles achieves competitive performance on standard reinforcement learning benchmarks. Then, we analyze how the proposed stabilization principles influence learning behavior through controlled ablations. We begin with Atari-100K [8](Section VI-A), a widely adopted benchmark for data-efficient reinforcement learning. Due to its limited interaction budget and high replay utilization, Atari-100K provides a challenging setting where bootstrapping instability and value overestimation are particularly pronounced. This benchmark allows us to directly assess whether the proposed stabilization mechanisms improve learning reliability under constrained data conditions. To further evaluate robustness in discrete control, we consider Procgen [10](Section VI-B), which features procedurally generated environments with diverse visual appearances and dynamics. Unlike fixed-layout benchmarks, Procgen tests representation robustness and generalization across unseen levels, enabling us to examine whether improved value stability translates into stronger cross-distribution performance. In addition to benchmark comparisons, we perform a series of diagnostic analyses to investigate the mechanisms identified in Section I and the stabilization principles introduced in Section IV. VI-A Atari-100K: Learning Stability under Data-Efficient Regimes We evaluate our method on the Atari-100K benchmark, which consists of 26 Atari 2600 games under a strict interaction budget of 100k environment steps. This corresponds to 400k frames with a frame skip of 4. Under this limited data regime, agents must repeatedly reuse collected transitions, resulting in high replay ratios. Our implementation follows the standard Atari preprocessing pipeline [26]. Observations are resized and stacked following common practice. We adopt a convolutional ResNet-style backbone without normalization layers, consistent with our method design. Results are averaged over multiple random seeds. Detailed hyperparameter configurations are provided in supplementary materials. We compare against a diverse set of representative methods evaluated under the Atari-100K protocol. Random and human scores are reported as reference points. DER [36] incorporates auxiliary supervised objectives into value learning to improve sample efficiency. DrQ [38] enhances data efficiency via image augmentation applied directly to the value function. IRIS [25], STORM [39], DreamerV3 [16], DIAMOND [3], DART [1], and Drama [37] are model-based or hybrid approaches that leverage latent dynamics modeling for improved planning and sample efficiency. REM [2] reduces overestimation through ensemble-based value aggregation. SGF [30] focuses on stabilizing gradient propagation during representation learning. BBF [32] combines aggressive replay, regularization, and parameter resetting to achieve strong performance under limited data. These baselines collectively cover value-based, model-based, and hybrid paradigms for data-efficient reinforcement learning. Performance is measured using the Human-Normalized Score (HNS). We report aggregate statistics across 26 games, including the mean HNS, median HNS, and Interquartile Mean (IQM), which provides a robust estimate that is less sensitive to extreme outliers. We additionally report the number of games surpassing human-level performance and the number of per-game best results. TABLE I: Aggregate performance on Atari-100K across 26 games. All metrics are computed using human-normalized scores (HNS). Mean, Median, and Interquartile Mean (IQM) are reported following standard evaluation practice. #Human denotes the number of games surpassing human-level performance, and #Best indicates the number of per-game best results. Higher is better for all metrics. Algorithm #Human #Best Mean Median IQM DER 2 0 0.350 0.189 0.183 DrQ 3 0 0.465 0.313 0.280 IRIS 9 0 1.046 0.289 0.501 REM 12 1 1.222 0.280 0.673 STORM 10 5 1.266 0.580 0.636 DreamerV3 9 1 1.120 0.466 0.490 DIAMOND 11 0 1.459 0.373 0.641 DART 9 0 1.022 0.790 0.575 SGF 6 0 0.884 0.152 0.287 Drama 8 1 1.049 0.270 0.367 BBF 12 8 2.247 0.917 1.045 Ours 14 10 1.799 1.045 1.070 Aggregate results are presented in Table I. Our method achieves the highest IQM and Median HNS among all compared approaches. We also obtain the largest number of games exceeding human-level performance and the highest number of per-game best results. These results indicate strong overall performance under the stringent 100k interaction constraint. For completeness and transparency, full per-environment scores are reported in supplementary materials. VI-B Procgen: Representation Robustness and Generalization We further evaluate our method on the Procgen benchmark [10], which emphasizes representation robustness and cross-level generalization. Unlike Atari-100K, where training and evaluation share identical game layouts, Procgen generates diverse levels procedurally and allows strict separation between training and test environments. We adopt the generalization protocol, where the agent is trained on a fixed set of 200 procedurally generated levels and evaluated on an unseen set of test levels. This setting explicitly measures cross-level generalization ability rather than memorization of specific layouts. All agents interact with environments for a total of 25M environment steps. During evaluation, exploration is disabled and performance is averaged over multiple test levels. Detailed hyperparameter configurations are provided in supplementary materials. We compare against the official PPO implementation reported in the Procgen paper [10]. PPO is widely adopted as a strong on-policy baseline for Procgen and has been carefully tuned for this benchmark. Using PPO as the primary baseline allows us to assess whether improved value stability translates into stronger generalization under procedural diversity. For each game, we report the average test return over unseen levels. To summarize performance across environments, we additionally report: the mean score, the median score, and the interquartile mean (IQM), following recent evaluation practice for robust aggregate comparison. Higher values indicate better generalization performance. TABLE I: Procgen (easy difficulty, 200 training levels). Scores are normalized using min-max normalization as defined in the original Procgen paper. Test Mean Test Median Test IQM PPO (Impala CNN) 0.33 0.38 0.27 Ours 0.40 0.40 0.39 Table I presents aggregate statistics. Our method consistently outperforms the baseline on the majority of environments, achieving higher average scores. The improvement is particularly pronounced in environments with high visual and structural variability, suggesting that stabilizing recursive value learning benefits representation robustness under procedural diversity. Detailed results and additional visualization are provided in the supplementary materials. Per-environment performance reveals a structured pattern rather than uniform improvement. Substantial gains are observed in environments such as BigFish, Dodgeball, and StarPilot, which provide relatively dense reward signals and require stable action-value propagation. In these settings, recursive bootstrapping occurs frequently, and over-amplified value estimates can easily destabilize training. By controlling bootstrap bias and reducing estimation variance, our method produces smoother value propagation and more reliable action selection, leading to significant performance improvements. In contrast, environments such as Chaser, Heist, and Maze remain challenging. These tasks feature sparse rewards and long-horizon credit assignment, where performance is heavily influenced by exploration efficiency and global planning capability. Since our method primarily targets value instability rather than exploration enhancement, its advantages are less pronounced in such environments. VI-C Mechanism Analysis and Sensitivity Study Beyond benchmark results, we conduct additional experiments to analyze the behavior of the proposed method. Specifically, we examine the effects of ensemble size, the Action-Decoupling Constraint, parameter-level spike dynamics, and replay ratio. These studies aim to provide further insight into the factors influencing training stability and performance. VI-C1 Effect of Ensemble Size We investigate the influence of the number of ensemble networks NeN_e on training performance. Experiments are conducted on four representative Atari environments: Alien, Amidar, BankHeist, and Breakout. We evaluate Ne∈2,4,8,16N_e∈\2,4,8,16\ while keeping all other training settings fixed. Each configuration is repeated with multiple random seeds. Fig. 3: Training performance under different ensemble sizes NeN_e on four Atari-100K environments. Curves show the recent 10-episode mean return; shaded regions denote 95% CIs. Fig. 3 shows the evolution of the recent 10-episode mean return during training. Increasing the ensemble size generally improves performance and training stability across most environments. In particular, larger ensembles lead to faster performance growth and higher final returns in Alien, Amidar, and BankHeist. Breakout exhibits a less monotonic trend, indicating that the effect of ensemble size may vary depending on environment dynamics. Fig. 4: Normalized final performance as a function of the ensemble size. Results are normalized within each environment. To further summarize the overall trend, Fig. 4 reports normalized final performance as a function of NeN_e. Performance improves consistently as the ensemble size increases from 2 to 16 in most cases, suggesting that larger ensembles contribute positively to learning stability. VI-C2 Effect of the Action-Decoupling Constraint We evaluate the impact of the Action-Decoupling Constraint applied to reward-bearing transitions by comparing the full method with a variant in which the constraint is removed during target computation. All other training settings are kept identical. Experiments are conducted on Alien, Amidar, BankHeist, and Breakout. Fig. 5: Comparison of training dynamics with and without the Action-Decoupling Constraint. For each environment, the left axis shows the recent 10-episode mean return, and the right axis shows the average Q-value. Solid lines correspond to the constrained version, while dashed lines denote the unconstrained variant. Fig. 5 presents the evolution of the recent 10-episode mean return (left axis) and the average Q-value (right axis). On Alien, the constrained version yields a clear and consistent performance improvement over the unconstrained variant. In contrast, on Amidar, BankHeist, and Breakout, the performance differences are less pronounced, and in some cases the unconstrained version performs comparably or slightly better. A key distinction lies in reward density. Alien provides relatively dense and frequent rewards, where excessive self-reinforcing updates may amplify overestimated values and destabilize value learning. The Action-Decoupling Constraint mitigates this amplification, leading to more stable performance. In environments with sparser rewards, moderate self-reinforcing effects can help strengthen weak learning signals, which may explain the smaller performance gap in those tasks. In such cases, weaker signal amplification can also be compensated by longer training or increased data reuse. Further analysis under higher replay ratios is provided later in this section. VI-C3 Parameter-Level Spike Dynamics and Reset Mechanism We first analyze parameter-level spike dynamics during training. For each network layer, we monitor the spike ratio (Eq. (11)) over training progress, defined as the ratio between the maximum absolute parameter value and a high-percentile statistic within the same layer. This metric provides a proxy for detecting abnormal parameter amplification. Fig. 9 visualizes the spike ratio evolution under different replay ratios (r=1,2,4r=1,2,4). Two consistent patterns emerge. First, spike ratios are significantly higher in the MLP layers following the encoder, particularly in the projection head. In contrast, convolutional encoder layers remain comparatively stable. Second, increasing the replay ratio leads to systematically higher spike ratios across layers. This suggests that aggressive data reuse amplifies parameter imbalance and increases the risk of unstable value growth. We further investigate the role of the parameter reset mechanism. Specifically, we compare training performance with and without parameter resetting, and record the frequency of reset events during training. Fig. 6: Effect of parameter resetting on training performance. Solid lines denote the version with parameter reset, while dashed lines indicate the variant without resetting. The right axis shows the frequency of reset events during training. As shown in Fig. 6, parameter resets occur more frequently in early training stages and under environments where value growth is more volatile. When parameter resetting is enabled, its impact varies across environments. Among the four evaluated tasks, a clear improvement is observed on Alien, while the effect on the other environments is comparatively modest. This suggests a trade-off: although resetting can restore network plasticity, it may also introduce irreversible performance loss by disrupting useful parameter structures that have already formed during training. As learning progresses, the model may gradually specialize toward informative features, and aggressively preserving plasticity can therefore interfere with stable decision making. Figure 2 further shows that, although plasticity tends to decrease during training, it can partially recover as new interaction data are incorporated, rather than collapsing permanently. Empirically, higher replay ratios require stricter and more frequent parameter resetting to maintain stable training dynamics. VI-C4 Sensitivity to Replay Ratio We further evaluate the sensitivity of the proposed method to replay ratio (r), which controls the degree of data reuse during training. Experiments are conducted on Alien, Amidar, BankHeist, and Breakout with r∈1,2,4,8r∈\1,2,4,8\, while keeping all other hyperparameters fixed. Fig. 7: Training performance under different replay ratios on four Atari-100K environments. Curves show the recent 10-episode mean return; shaded regions denote 95% CIs. Fig. 7 presents training curves under different replay ratios. The impact of data reuse varies across environments. In Alien and Amidar, moderate replay ratios (e.g., r=4r=4) lead to improved performance compared to lower reuse settings. However, excessively large replay ratios (e.g., r=8r=8) may cause performance degradation in some environments. In contrast, BankHeist benefits consistently from higher replay ratios, suggesting that additional data reuse can help amplify weak learning signals in sparser reward settings. Breakout exhibits a different trend, where performance peaks at lower replay ratios and decreases as data reuse becomes aggressive. Fig. 8: Normalized final performance as a function of replay ratio. Results are normalized within each environment. Fig. 8 summarizes the normalized final performance as a function of replay ratio. The results reveal a non-monotonic relationship between replay ratio and final performance. Across multiple environments, performance improves as rr increases from low to moderate values, but degrades when rr becomes excessively large. This suggests that while additional data reuse enhances sample efficiency, there exists a task-dependent threshold beyond which further reuse no longer benefits learning. A possible explanation relates to the interaction between aggressive data reuse and parameter spike dynamics. As observed in the previous section, higher replay ratios systematically amplify spike ratios, especially in post-encoder layers. Since spike monitoring and parameter resetting are performed periodically to avoid excessive computational overhead, very large replay ratios may lead to rapid over-amplification within a single monitoring interval. In such cases, the network may temporarily overfit recent data before the next reset step is triggered, slowing down effective learning progress. Consequently, excessively high replay ratios can reduce training efficiency. These findings indicate that replay ratio must be carefully balanced with stability mechanisms, and that optimal data reuse is inherently environment-dependent. VII Conclusion and Future Work This work presents a unified perspective on instability in deep Q-learning by examining the dynamics of recursive value learning. We show that instability arises from the interaction of three mechanisms: operator-level bias in Bellman bootstrapping, estimator-level sensitivity of greedy decisions to regression noise, and parameter-dynamics imbalance under aggressive data reuse. Our analysis reveals two characteristic behaviors that have received limited attention in prior studies. First, we identify a reward-triggered self-reinforcing trap in recursive bootstrapping, where value amplification interacts with representation generalization to repeatedly reinforce certain actions in bootstrap targets. Second, we uncover parameter spike dynamics associated with the gradual loss of network plasticity during training, and introduce the spike ratio as a practical diagnostic indicator for monitoring this effect. Based on these insights, we derive a set of stabilization principles that regulate bootstrap target construction, reduce decision variance in value estimation, and maintain balanced parameter evolution during training. We instantiate these principles in a practical learning algorithm integrating controlled bootstrapping, ensemble quantile estimation, and spike-based parameter regulation. Experiments on Atari-100K and Procgen demonstrate that the resulting method achieves competitive performance while improving training stability. Beyond empirical results, our findings highlight the importance of analyzing reinforcement learning through training dynamics. The interaction between bootstrapping, stochastic estimation, and parameter evolution suggests that stability in value learning is fundamentally a systems-level property rather than the result of isolated algorithmic components. Future work may explore adaptive stabilization strategies, automated coordination of hyperparameters such as replay ratio and ensemble size, and extensions of the proposed analysis to actor-critic methods and continuous-control settings. References [1] P. Agarwal, S. Andrews, and S. E. Kahou (2024) Learning to play atari in a world of tokens. arXiv preprint arXiv:2406.01361. Cited by: §I-E, §VI-A. [2] R. Agarwal, D. Schuurmans, and M. Norouzi (2020) An optimistic perspective on offline reinforcement learning. In International conference on machine learning, p. 104–114. Cited by: §I-A, §I-B, §VI-A. [3] E. Alonso, A. Jelley, V. Micheli, A. Kanervisto, A. J. Storkey, T. Pearce, and F. Fleuret (2024) Diffusion for world modeling: visual details matter in atari. Advances in Neural Information Processing Systems 37, p. 58757–58791. Cited by: §I-E, §VI-A. [4] O. Anschel, N. Baram, and N. Shimkin (2017) Averaged-dqn: variance reduction and stabilization for deep reinforcement learning. In International conference on machine learning, p. 176–185. Cited by: §I-A. [5] J. T. Ash and R. P. Adams (2020) On warm-starting neural network training. In Advances in Neural Information Processing Systems, Vol. 33, p. 3884–3894. Cited by: §I-D. [6] M. G. Bellemare, W. Dabney, and R. Munos (2017) A distributional perspective on reinforcement learning. In International conference on machine learning, p. 449–458. Cited by: §I-B. [7] R. C. Castanyer, J. Obando-Ceron, L. Li, P. Bacon, G. Berseth, A. Courville, and P. S. Castro (2025) Stable gradients for stable learning at scale in deep reinforcement learning. arXiv preprint arXiv:2506.15544. Cited by: §I-D. [8] J. S. O. Ceron and P. S. Castro (2021) Revisiting rainbow: promoting more insightful and inclusive deep reinforcement learning research. In International Conference on Machine Learning, p. 1373–1383. Cited by: §VI. [9] X. Chen, C. Wang, Z. Zhou, and K. Ross (2021) Randomized ensembled double q-learning: learning fast without a model. In 9th International Conference on Learning Representations (ICLR 2021), Note: arXiv:2101.05982 Cited by: §I-A. [10] K. Cobbe, C. Hesse, J. Hilton, and J. Schulman (2020) Leveraging procedural generation to benchmark reinforcement learning. In International conference on machine learning, p. 2048–2056. Cited by: §VI-B, §VI-B, §VI. [11] W. Dabney, G. Ostrovski, D. Silver, and R. Munos (2018) Implicit quantile networks for distributional reinforcement learning. In International conference on machine learning, p. 1096–1105. Cited by: §I-B. [12] W. Dabney, M. Rowland, M. G. Bellemare, and R. Munos (2017) Distributional reinforcement learning with quantile regression. arXiv preprint arXiv:1710.10044. Cited by: §I-B. [13] P. D’Oro, M. Schwarzer, E. Nikishin, P. Bacon, M. G. Bellemare, and A. Courville (2022) Sample-efficient reinforcement learning by breaking the replay ratio barrier. In Deep Reinforcement Learning Workshop NeurIPS 2022, Cited by: §I-C. [14] S. Fujimoto, H. Hoof, and D. Meger (2018) Addressing function approximation error in actor-critic methods. In International conference on machine learning, p. 1587–1596. Cited by: §I-A. [15] F. Gogianu, T. Berariu, M. C. Rosca, C. Clopath, L. Busoniu, and R. Pascanu (2021) Spectral normalisation for deep reinforcement learning: an optimisation perspective. In International Conference on Machine Learning, p. 3734–3744. Cited by: §I-D. [16] D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap (2023) Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104. Cited by: §I-E, §VI-A. [17] H. Hasselt (2010) Double q-learning. Advances in neural information processing systems 23. Cited by: §I, §I-A. [18] A. Kuznetsov, P. Shvechikov, A. Grishin, and D. Vetrov (2020) Controlling overestimation bias with truncated mixture of continuous distributional quantile critics. In International Conference on Machine Learning, p. 5556–5566. Cited by: §I-B. [19] Q. Lan, Y. Pan, A. Fyshe, and M. White (2020) Maxmin q-learning: controlling the estimation bias of q-learning. arXiv preprint arXiv:2002.06487. Cited by: §I-A. [20] M. Laskin, A. Srinivas, and P. Abbeel (2020) Curl: contrastive unsupervised representations for reinforcement learning. In International conference on machine learning, p. 5639–5650. Cited by: §I-C. [21] D. Lee, S. Han, T. Cho, and J. Lee (2023) SPQR: controlling q-ensemble independence with spiked random model for reinforcement learning. Advances in Neural Information Processing Systems 36, p. 65224–65251. Cited by: §I-B. [22] K. Lee, M. Laskin, A. Srinivas, and P. Abbeel (2021) Sunrise: a simple unified framework for ensemble learning in deep reinforcement learning. In International conference on machine learning, p. 6131–6141. Cited by: §I-B. [23] C. Lyle, M. Rowland, G. Ostrovski, and W. Dabney (2023) Understanding plasticity in neural networks. In International Conference on Machine Learning, p. 23190–23211. Cited by: §I, §I-D. [24] J. Markowitz, J. Silverberg, and G. L. Collins (2024) Avoiding value estimation error in off-policy deep reinforcement learning. In I Can’t Believe It’s Not Better Workshop: Failure Modes of Sequential Decision-Making in Practice (RLC 2024), Cited by: §I-D. [25] V. Micheli, E. Alonso, and F. Fleuret (2022) Transformers are sample-efficient world models. arXiv preprint arXiv:2209.00588. Cited by: §I-C, §I-E, §VI-A. [26] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. (2015) Human-level control through deep reinforcement learning. nature 518 (7540), p. 529–533. Cited by: §I-D, §VI-A. [27] M. Nauman, M. Bortkiewicz, P. Miłoś, T. Trzciński, M. Ostaszewski, and M. Cygan (2024) Overestimation, overfitting, and plasticity in actor-critic: the bitter lesson of reinforcement learning. arXiv preprint arXiv:2403.00514. Cited by: §I-D. [28] E. Nikishin, M. Schwarzer, P. D’Oro, P. Bacon, and A. Courville (2022) The primacy bias in deep reinforcement learning. In International conference on machine learning, p. 16828–16847. Cited by: §I, §I-D, §I-D. [29] I. Osband, C. Blundell, A. Pritzel, and B. Van Roy (2016) Deep exploration via bootstrapped dqn. Advances in neural information processing systems 29. Cited by: §I-B. [30] J. Robine, M. Höftmann, and S. Harmeling (2025) Simple, good, fast: self-supervised world models free of baggage. arXiv preprint arXiv:2506.02612. Cited by: §I-C, §VI-A. [31] M. Schwarzer, A. Anand, R. Goel, R. D. Hjelm, A. Courville, and P. Bachman (2020) Data-efficient reinforcement learning with self-predictive representations. arXiv preprint arXiv:2007.05929. Cited by: §I-C. [32] M. Schwarzer, J. S. O. Ceron, A. Courville, M. G. Bellemare, R. Agarwal, and P. S. Castro (2023) Bigger, better, faster: human-level atari with human-level efficiency. In International Conference on Machine Learning, p. 30365–30380. Cited by: §I-C, §I-C, §VI-A. [33] G. Sokar, R. Agarwal, P. S. Castro, and U. Evci (2023) The dormant neuron phenomenon in deep reinforcement learning. In International Conference on Machine Learning, p. 32145–32168. Cited by: §I, §I-D, §I-D. [34] S. Thrun and A. Schwartz (1993) Issues in using function approximation for reinforcement learning. In Proceedings of the Fourth Connectionist Models Summer School, p. 255–263. Cited by: §I-A. [35] H. Van Hasselt, A. Guez, and D. Silver (2016) Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence, Vol. 30. Cited by: §I, §I-A. [36] H. P. Van Hasselt, M. Hessel, and J. Aslanides (2019) When to use parametric models in reinforcement learning?. Advances in Neural Information Processing Systems 32. Cited by: §I-C, §VI-A. [37] W. Wang, I. Dusparic, Y. Shi, K. Zhang, and V. Cahill (2024) Drama: mamba-enabled model-based reinforcement learning is sample and parameter efficient. arXiv preprint arXiv:2410.08893. Cited by: §I-C, §I-E, §VI-A. [38] D. Yarats, I. Kostrikov, and R. Fergus (2021) Image augmentation is all you need: regularizing deep reinforcement learning from pixels. In International conference on learning representations, Cited by: §I-C, §VI-A. [39] W. Zhang, G. Wang, J. Sun, Y. Yuan, and G. Huang (2023) Storm: efficient stochastic transformer based world models for reinforcement learning. Advances in Neural Information Processing Systems 36, p. 27147–27166. Cited by: §I-C, §I-E, §VI-A. Appendix A Spike Ratio Dynamics Fig. 9: Layer-wise spike ratio evolution during training under different replay ratios. Each row corresponds to a network layer, and color intensity indicates spike magnitude. This section provides detailed experimental settings and implementation details for the spike ratio analysis presented in Fig. 9(Fig. 2 of the main paper). The purpose of this experiment is to examine how replay ratio evolves over the course of learning. To monitor parameter dynamics during training, we periodically compute the spike ratio. For a network layer ℓ with parameter tensor θℓ _ , the spike ratio is defined as SpikeRatioℓ=‖θℓ‖∞Quantile0.99(|θℓ|).SpikeRatio_ = \| _ \|_∞Quantile_0.99(| _ |). (22) The experiment follows the standard Atari-100K evaluation protocol described in the main paper. Training uses the same optimization settings as the main algorithm described in this paper. Unless otherwise specified, all hyperparameters remain identical to the main experiments. The replay ratio rr controls the number of gradient updates per environment step. To analyze the effect of data reuse, we conduct experiments with three different settings, rr∈1,2,4r∈\1,2,4\. The value function is implemented as an ensemble quantile network. Each ensemble member consists of a convolutional encoder followed by a linear projection head that outputs quantile estimates for all actions. The encoder adopts a lightweight ResNet-style architecture. Specifically, the input observation x∈ℝ4×84×84x ^4× 84× 84 is processed by a sequence of convolutional layers and residual blocks to produce a latent feature vector. TABLE I: Network layers monitored for spike ratio statistics in Figure 9. The spike ratio is computed for each layer every 1000 environment steps to analyze how parameter imbalance evolves during training. The encoder’s channel width is controlled by a scale factor s (with s=4s=4 in our experiments). Layer Group Layer Name Input Shape Weight Shape Stem stem.0 4×84×844× 84× 84 (8s)×4×3×3(8s)× 4× 3× 3 Stage1 l1.0.conv1 (8s)×84×84(8s)× 84× 84 (8s)×(8s)×3×3(8s)×(8s)× 3× 3 l1.0.conv2 (8s)×84×84(8s)× 84× 84 (8s)×(8s)×3×3(8s)×(8s)× 3× 3 l1.1.conv1 (8s)×84×84(8s)× 84× 84 (8s)×(8s)×3×3(8s)×(8s)× 3× 3 l1.1.conv2 (8s)×84×84(8s)× 84× 84 (8s)×(8s)×3×3(8s)×(8s)× 3× 3 Stage2 l2.0.conv1 (8s)×84×84(8s)× 84× 84 (16s)×(8s)×3×3(16s)×(8s)× 3× 3 l2.0.conv2 (16s)×42×42(16s)× 42× 42 (16s)×(16s)×3×3(16s)×(16s)× 3× 3 l2.1.conv1 (16s)×42×42(16s)× 42× 42 (16s)×(16s)×3×3(16s)×(16s)× 3× 3 l2.1.conv2 (16s)×42×42(16s)× 42× 42 (16s)×(16s)×3×3(16s)×(16s)× 3× 3 Stage3 l3.0.conv1 (16s)×42×42(16s)× 42× 42 (32s)×(16s)×3×3(32s)×(16s)× 3× 3 l3.0.conv2 (32s)×21×21(32s)× 21× 21 (32s)×(32s)×3×3(32s)×(32s)× 3× 3 l3.1.conv1 (32s)×21×21(32s)× 21× 21 (32s)×(32s)×3×3(32s)×(32s)× 3× 3 l3.1.conv2 (32s)×21×21(32s)× 21× 21 (32s)×(32s)×3×3(32s)×(32s)× 3× 3 Stage4 l4.0.conv1 (32s)×21×21(32s)× 21× 21 (64s)×(32s)×3×3(64s)×(32s)× 3× 3 l4.0.conv2 (64s)×11×11(64s)× 11× 11 (64s)×(64s)×3×3(64s)×(64s)× 3× 3 l4.1.conv1 (64s)×11×11(64s)× 11× 11 (64s)×(64s)×3×3(64s)×(64s)× 3× 3 l4.1.conv2 (64s)×11×11(64s)× 11× 11 (64s)×(64s)×3×3(64s)×(64s)× 3× 3 Projection Head head.1 (64s)×11×11(64s)× 11× 11 (128s)×(64s⋅11⋅11)(128s)×(64s· 11· 11) Value Head quantile head 128s128s (||K)×(128s)(|A|K)×(128s) Table I lists all trainable layers for which spike statistics are monitored during training. Since the model contains multiple ensemble members with identical architectures, spike ratios are first computed independently for each ensemble network and then averaged to obtain the aggregated statistics used in Figure 9. Appendix B Bounded Bellman Update This bound is derived from the structure of the discounted return and reflects the maximum achievable value under the sampled reward scale. Consider a trajectory generated by greedily following the current value function. Starting from (s0,a0)(s_0,a_0) with a0∈argmaxaQ(s0,a)a_0∈ _aQ(s_0,a), we iteratively select at∈argmaxaQ(st,a),a_t∈ _aQ(s_t,a), and obtain a greedy trajectory =(s0,a0,r0),(s1,a1,r1),…,(sT,aT,rT),T=\(s_0,a_0,r_0),(s_1,a_1,r_1),…,(s_T,a_T,r_T)\, which terminates at time step T. Along this greedy trajectory, the one-step greedy backup satisfies rt+γmaxa′Q(st+1,a′)=rt+γQ(st+1,at+1),r_t+γ _a Q(s_t+1,a )\;=\;r_t+γ Q(s_t+1,a_t+1), (23) because at+1∈argmaxaQ(st+1,a)a_t+1∈ _aQ(s_t+1,a) by construction. Therefore, by recursively substituting the greedy action at each subsequent state, we obtain Q(st,at)=rt+γmaxa′Q(st+1,a′)=rt+γQ(st+1,at+1)=rt+γ(rt+1+γmaxa′Q(st+2,a′))=rt+γrt+1+γ2Q(st+2,at+2)⋮=rt+γrt+1+⋯+γT−tQ(sT,aT). splitQ(s_t,a_t)&=r_t+γ _a Q(s_t+1,a )\\ &=r_t+γ Q(s_t+1,a_t+1)\\ &=r_t+γ (r_t+1+γ _a Q(s_t+2,a ) )\\ &=r_t+γ r_t+1+γ^2Q(s_t+2,a_t+2)\\ &\;\; \\ &=r_t+γ r_t+1+·s+γ^T-tQ(s_T,a_T). split (24) If the episode terminates at T with no future continuation (or equivalently Q(sT,aT)=rTQ(s_T,a_T)=r_T under terminal dynamics), the recursion closes and yields Q(st,at)=∑k=tTγk−trk≜Gt.Q(s_t,a_t)= _k=t^Tγ^k-tr_k\; \;G_t. (25) For a given greedy trajectory T, we define the trajectory-level value bound as B()=max0≤t≤TGt.B(T)= _0≤ t≤ TG_t. (26) This greedy-rollout identity provides a natural way to use GtG_t as a trajectory-wise bound for the numerical scale of Q(st,at)Q(s_t,a_t) under the current greedy policy. Appendix C Atari Experiment Details This section provides additional implementation details for the Atari experiments reported in the main paper. The experiments follow the Atari-100K evaluation protocol with a total interaction budget of 100k environment steps. C-A Environment Setup All experiments are conducted using the Atari Learning Environment through Gymnasium. Observations follow the standard preprocessing pipeline implemented by the Stable-Baselines3 Atari wrapper. Specifically, raw frames are converted to grayscale, resized to 84×8484× 84, and stacked over four consecutive frames to form the input state representation. Frame skipping is set to 44, which corresponds to 400k400k environment frames under the Atari-100K setting. The environment wrapper additionally applies a random number of no-op actions at the beginning of each episode (up to 30 steps). Episode termination follows the standard Atari convention, where life loss is treated as terminal during training. Rewards are not clipped to [−1,1][-1,1] during environment interaction. However, the sign of the reward is stored in the replay buffer for training updates. C-B Replay Buffer and Sampling Experience replay uses a prioritized replay buffer with a capacity of 100,000100,000 transitions. Transitions are sampled according to prioritized sampling with exponent α=0.6α=0.6. Importance sampling weights are applied during optimization using exponent β=0.4β=0.4. For each sampled transition, observations are normalized to [0,1][0,1] by dividing pixel values by 255255. C-C Network Architecture As Table I, the value function is implemented as an ensemble quantile network. Each ensemble member consists of a convolutional encoder followed by a linear projection head. The encoder adopts a lightweight ResNet-style architecture. The input observation has shape (4,84,84)(4,84,84) and is processed through a sequence of convolutional residual blocks with progressive downsampling: 84×84→84×84→42×42→21×21→11×11.84× 84→ 84× 84→ 42× 42→ 21× 21→ 11× 11. The number of channels is controlled by a scale factor s=4s=4. After the convolutional encoder, features are flattened and projected to a fully connected layer of size 512s512s. The final layer outputs K quantile values for each action. With NeN_e ensemble members and K quantile atoms, the network produces a tensor of shape (B,Ne,K,||),(B,N_e,K,|A|), where B denotes the batch size. C-D Training Procedure The agent is trained for 100,000100,000 environment steps. During training, the agent interacts with the environment using an ϵε-greedy policy. The exploration parameter ϵε is linearly annealed from 1.01.0 to 0.010.01 over the first 20002000 steps. Learning begins after 20002000 transitions have been collected. At each environment step, rr gradient updates are performed, where rr denotes the replay ratio. Optimization uses the Adam optimizer with learning rate 1×10−41× 10^-4. Target networks are updated using Polyak averaging with coefficient τ=0.005τ=0.005. Gradients are clipped with a maximum norm of 1010. C-E Distributional Value Learning The value function is trained using quantile regression with the quantile Huber loss. The number of quantile atoms is K=51K=51. Quantile fractions follow the standard QR-DQN formulation τi=i+0.5K,i=0,…,K−1. _i= i+0.5K, i=0,…,K-1. For each update, the target distribution is constructed using the target network and the greedy action selected according to the mean value across quantiles. C-F Ensemble Learning The proposed method employs an ensemble of Ne=16N_e=16 value networks. Each ensemble member independently estimates quantile value distributions for all actions. During action selection, quantile values are averaged across both the quantile dimension and the ensemble dimension to produce the final action-value estimates. C-G Parameter Reset Mechanism To mitigate excessive parameter specialization, the algorithm monitors the spike ratio of each layer during training. For a parameter tensor θ, the spike ratio is defined as SpikeRatio=max|θ|Quantile0.99(|θ|).SpikeRatio= |θ|Quantile_0.99(|θ|). If the spike ratio of a layer exceeds the threshold, the parameters of that layer are reinitialized. The optimizer states associated with the reset parameters are also cleared, and the target network is synchronized with the updated parameters. C-H Hardware and Implementation All experiments were conducted on a single NVIDIA RTX 5090 GPU. The implementation is written in PyTorch and runs on a Linux environment with CUDA acceleration. All reported results are obtained using the same hardware configuration. C-I Hyperparameters The hyperparameters used in the Atari experiments are summarized in Table IV. TABLE IV: Hyperparameter configuration used for Atari-100K experiments. Training Configuration (Atari-100K) Total environment steps 100,000100,000 Discount factor γ 0.990.99 Learning rate 1×10−41× 10^-4 (Adam) Batch size 3232 Replay buffer size 100,000100,000 Start learning after steps 2,0002,000 Training frequency 11 update per step Soft target update rate τ 0.0050.005 Replay ratio 44 Gradient clipping 10.010.0 Parameter reset threshold 6.06.0 Exploration Initial ϵε 1.01.0 Final ϵε 0.010.01 ϵε decay steps 2,0012,001 Replay Buffer Prioritization exponent α 0.60.6 Importance sampling exponent β 0.40.4 Priority epsilon 1×10−61× 10^-6 Distributional / Ensemble Settings Number of quantiles 5151 Number of ensemble heads 1616 Quantile Huber parameter κ 11 Observation Processing Frame stack 44 Frame skip 44 Observation resolution 84×8484× 84 C-J Experimental Results Detailed performance comparisons on the Atari benchmark are provided in Table V. TABLE V: Atari-100K performance comparison across environments. The best score in each row is highlighted in bold. Env Random Human DER DrQ IRIS REM STORM DreamerV3 DIAMOND DART SGF Drama BBF Ours Alien 227.8 7127.7 802.3 865.2 420.0 607.2 983.6 959.0 744.1 962.0 518.8 820 1173.2 1340.0 Amidar 5.8 1719.5 125.9 137.8 143.0 95.3 204.8 139.0 225.8 125.7 62.7 131 244.6 293.6 Assault 222.4 742.0 561.5 579.6 1524.4 1764.2 801.0 706.0 1526.4 1316.0 850.1 539 2098.5 2348.3 Asterix 210.0 8503.3 535.4 763.6 853.6 1637.5 1028.0 932.0 3698.5 956.2 802.5 1632 3946.1 2430.0 BankHeist 14.2 753.1 185.5 232.9 53.1 19.2 641.2 649.0 19.7 629.7 58.7 137 732.9 772.0 BattleZone 2360.0 37187.5 8977.0 10165.3 13074.0 11826.0 13540.0 12250.0 4702.0 15325.0 3747.0 10860 24459.8 18600.0 Boxing 0.1 12.1 -0.3 9.0 70.1 87.5 79.7 78.0 86.9 83.0 83.4 78 85.8 25.1 Breakout 1.7 30.5 9.2 19.8 83.7 90.7 15.9 31.0 132.5 41.9 50.7 7 370.6 233.0 ChopperCommand 811.0 7387.8 925.9 844.6 1565.0 2561.2 1888.0 420.0 1369.8 1263.8 1775.4 1642 7549.3 2740.0 CrazyClimber 10780.5 35829.4 34508.6 21539.0 59324.2 76547.6 66776.0 97190.0 99167.8 34070.6 15751.3 83931 58431.8 84700.0 DemonAttack 152.1 1971.0 627.6 1321.5 2034.4 5738.6 164.6 303.0 288.1 2452.3 2809.5 201 13341.4 7192.5 Freeway 0.0 29.6 20.9 20.3 31.1 32.3 33.5 0.0 33.3 32.2 11.9 15 25.5 33.3 Frostbite 65.2 4334.7 871.0 1014.2 259.1 240.5 1316.0 909.0 274.1 346.8 265.6 785 2384.8 2886.0 Gopher 257.6 2412.5 467.0 621.6 2236.1 5452.4 8239.6 3730.0 5897.9 1980.5 416.4 2757 1331.2 2346.0 Hero 1027.0 30826.4 6226.0 4167.9 7037.4 6484.8 11044.3 11161.0 5621.8 4927.0 1522.9 7946 7818.6 13436.0 Jamesbond 29.0 302.8 275.7 349.1 462.7 391.2 509.0 445.0 427.4 353.1 280.9 372 1129.6 510.0 Kangaroo 52.0 3035.0 581.7 1088.4 838.2 467.6 4208.0 4098.0 5382.2 2380.0 271.2 1384 6614.7 7520.0 Krull 1598.0 2665.5 3256.9 4402.1 6616.4 4017.7 8412.6 7782.0 8610.1 7658.3 7813.7 9693 8223.4 7851.0 KungFuMaster 258.5 22736.3 6580.1 11467.4 21759.8 25172.2 26182.0 21420.0 18713.6 23744.3 20169.8 23920 18991.7 24190.0 MsPacman 307.3 6951.6 1187.4 1218.1 999.1 962.5 2673.5 1327.0 1958.2 1132.7 1356.8 2270 2008.3 1986.0 Pong -20.7 14.6 -9.7 -9.1 14.6 18.0 11.3 18.0 20.4 17.2 12.6 15 16.7 21.0 PrivateEye 24.9 69571.3 72.8 3.5 100.0 99.6 7781.0 882.0 114.3 765.7 405.5 90 40.5 100.0 Qbert 163.9 13455.0 1773.5 1810.7 745.7 743.0 4522.5 3405.0 4499.3 750.9 685.0 796 4447.1 4985.0 RoadRunner 11.5 7845.0 11843.4 11211.4 9614.6 14060.2 17564.0 15565.0 20673.2 7772.5 8164.2 14020 33426.8 32760.0 Seaquest 68.4 42054.7 304.6 352.3 661.3 1036.7 525.2 618.0 551.2 895.8 476.8 497 1232.5 1198.0 UpNDown 533.4 11693.2 3075.0 4324.5 3546.2 3757.6 7985.0 9234.0 3856.3 3954.5 7745.0 7387 12101.7 33994.0 # > Human 0 0 2 3 9 12 10 9 11 9 6 8 12 14 # Best 0 0 0 0 0 1 5 1 0 0 0 1 8 10 Mean HNS 0.000 1.000 0.350 0.465 1.046 1.222 1.266 1.120 1.459 1.022 0.884 1.049 2.247 1.799 Median HNS 0.000 1.000 0.189 0.313 0.289 0.280 0.580 0.466 0.373 0.790 0.152 0.270 0.917 1.045 IQM HNS 0.000 1.000 0.183 0.280 0.501 0.673 0.636 0.490 0.641 0.575 0.287 0.367 1.045 1.070 Appendix D Procgen Experiment Details This section provides additional implementation details for the Procgen experiments reported in the main paper. All experiments follow the standard Procgen generalization protocol. D-A Environment Setup We evaluate our method on the Procgen benchmark under the generalization mode. In this setting, the agent is trained on a fixed set of procedurally generated levels and evaluated on unseen levels. Specifically, training environments use the first 200 levels of each game (start_level = 0, num_levels = 200), while evaluation is performed on disjoint levels starting from level 200 (start_level = 200, num_levels = 0), which corresponds to an infinite stream of unseen levels. All environments are run in easy difficulty mode. Observations are RGB images with resolution 64×6464× 64. Following common practice in prior work, we use a vectorized environment with 128 parallel instances during training to improve data throughput. D-B Observation Processing Unlike Atari, Procgen observations are already provided as RGB images with fixed resolution. Therefore no grayscale conversion or resizing is required. Observations are represented as tensors of shape (3,64,64)(3,64,64) corresponding to RGB channels. Pixel values are normalized to [0,1][0,1] before being fed into the neural network. Frame stacking is not used in our experiments since Procgen environments are fully observable. D-C Network Architecture The value function is implemented as the same ensemble quantile network used in the Atari experiments, but with an encoder adapted to the 64×6464× 64 Procgen observations. The encoder follows a lightweight residual architecture. Starting from a 64×6464× 64 input image, the spatial resolution is progressively reduced through strided residual blocks: 64×64→32×32→16×16→8×8→4×4.64× 64→ 32× 32→ 16× 16→ 8× 8→ 4× 4. The resulting feature map is flattened and projected to a 512-dimensional feature vector. The final layer outputs K quantile values for each action. With NeN_e ensemble members and K quantiles, the network produces a tensor of shape (B,Ne,K,||),(B,N_e,K,|A|), where B denotes the batch size. D-D Training Procedure Agents are trained for a total of 25 million environment steps. Exploration follows an ϵε-greedy strategy. The exploration rate is linearly annealed from 1.01.0 to 0.050.05 over the first 55 million environment steps. Training begins after an initial replay buffer warm-up period. During training, the agent performs gradient updates using mini-batches sampled from the replay buffer. The optimization objective is the quantile regression loss. The Adam optimizer is used for parameter updates. D-E Replay The replay buffer stores up to 100,000100,000 transitions. Mini-batches of size 256 are sampled uniformly during training. D-F Ensemble Distributional Value Learning Our method employs an ensemble of Ne=16N_e=16 value networks. Each ensemble member independently estimates the quantile value distribution for each action. During action selection, the predicted quantiles are averaged across both the ensemble dimension and the quantile dimension to produce the final action-value estimates. D-G Hardware and Implementation All Procgen experiments are implemented in PyTorch and executed on a single NVIDIA RTX 5090 GPU. The implementation uses vectorized environments with CUDA acceleration to improve training throughput. The hyperparameters used for the Procgen experiments are summarized in Table VI. TABLE VI: Hyperparameters for Procgen (generalization mode, 200 training levels). Total environment steps 25M Training levels 200 Evaluation levels Unseen procedural levels Parallel environments 128 Batch size 256 Discount factor γ 0.99 Optimizer Adam Learning rate 1e-4 Soft target update rate τ 0.0050.005 Ensemble size NeN_e 16 Quantile number K 51 Initial exploration ϵε 1.01.0 Final exploration ϵε 0.050.05 ϵε decay steps 5M5M D-H Detailed Per-Environment Results To provide a more comprehensive view of performance across different tasks, we report the raw test scores for each Procgen environment in Table VII under the generalization protocol. TABLE VII: Per-environment raw test performance on Procgen (easy, 200 training levels). Environment PPO (Impala CNN) Ours BigFish 3.0 18.1 BossFight 8.3 8.1 CaveFlyer 5.6 7.0 Chaser 5.5 1.9 Climber 6.1 7.5 CoinRun 8.8 6.0 Dodgeball 2.2 16.0 FruitBot 27.0 25.7 Heist 2.2 1.0 Jumper 5.8 6.0 Leaper 4.4 6.0 Maze 5.7 2.0 Miner 9.0 6.1 Ninja 5.8 6.0 Plunder 5.4 5.5 StarPilot 25.0 54.9 In addition to the final evaluation scores, we also visualize the training dynamics for each environment. Figure 10 shows the training performance curves throughout the entire learning process. Each subplot corresponds to a different Procgen environment, and the horizontal axis represents the number of environment interaction steps. Fig. 10: Per-environment train performance during training on Procgen.