Paper deep dive
Rethinking Plasticity in Deep Reinforcement Learning
Zhiqiang He
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 3/26/2026, 2:24:33 AM
Summary
The paper introduces the Optimization-Centric Plasticity (OCP) hypothesis to explain plasticity loss in deep reinforcement learning. It posits that plasticity loss occurs when optimal parameters from previous tasks become poor local optima for new tasks, trapping the network. The authors theoretically establish the equivalence between neuron dormancy and zero-gradient states, demonstrating that the absence of gradient signals drives dormancy, and validate their findings by showing that networks can regain performance on significantly different tasks.
Entities (4)
Relation Signals (2)
Zero-gradient states → causes → Dormant Neurons
confidence 98% · We theoretically establish the equivalence between neuron dormancy and zero-gradient states, demonstrating that the absence of gradient signals is the primary driver of dormancy.
Optimization-Centric Plasticity (OCP) hypothesis → explains → Plasticity Loss
confidence 95% · We propose the Optimization-Centric Plasticity (OCP) hypothesis, which posits that plasticity loss arises because optimal points from previous tasks become poor local optima for new tasks
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:This paper investigates the fundamental mechanisms driving plasticity loss in deep reinforcement learning (RL), a critical challenge where neural networks lose their ability to adapt to non-stationary environments. While existing research often relies on descriptive metrics like dormant neurons or effective rank, these summaries fail to explain the underlying optimization dynamics. We propose the Optimization-Centric Plasticity (OCP) hypothesis, which posits that plasticity loss arises because optimal points from previous tasks become poor local optima for new tasks, trapping parameters during task transitions and hindering subsequent learning. We theoretically establish the equivalence between neuron dormancy and zero-gradient states, demonstrating that the absence of gradient signals is the primary driver of dormancy. Our experiments reveal that plasticity loss is highly task-specific; notably, networks with high dormancy rates in one task can achieve performance parity with randomly initialized networks when switched to a significantly different task, suggesting that the network's capacity remains intact but is inhibited by the specific optimization landscape. Furthermore, our hypothesis elucidates why parameter constraints mitigate plasticity loss by preventing deep entrenchment in local optima. Validated across diverse non-stationary scenarios, our findings provide a rigorous optimization-based framework for understanding and restoring network plasticity in complex RL domains.
Tags
Links
- Source: https://arxiv.org/abs/2603.21173v1
- Canonical: https://arxiv.org/abs/2603.21173v1
Trouble viewing inline? Open PDF directly →
Full Text
54,973 characters extracted from source content.
Expand or collapse full text
RETHINKING PLASTICITY IN DEEP REINFORCEMENT LEARNING AN INCOMPLETE VERSION Zhiqiang He The University of Electro-Communications Tokyo, Japan hezhiqiang@ieee.org March 24, 2026 ABSTRACT This paper investigates the fundamental mechanisms driving plasticity loss in deep reinforcement learning (RL), a critical challenge where neural networks lose their ability to adapt to non-stationary environments. While existing research often relies on descriptive metrics like dormant neurons or effective rank, these summaries fail to explain the underlying optimization dynamics. We propose the Optimization-Centric Plasticity (OCP) hypothesis, which posits that plasticity loss arises because optimal points from previous tasks become poor local optima for new tasks, trapping parameters during task transitions and hindering subsequent learning. We theoretically establish the equivalence between neuron dormancy and zero-gradient states, demonstrating that the absence of gradient signals is the primary driver of dormancy. Our experiments reveal that plasticity loss is highly task-specific; notably, networks with high dormancy rates in one task can achieve performance parity with randomly initialized networks when switched to a significantly different task, suggesting that the network’s capacity remains intact but is inhibited by the specific optimization landscape. Furthermore, our hypothesis elucidates why parameter constraints mitigate plasticity loss by preventing deep entrenchment in local optima. Validated across diverse non-stationary scenarios, our findings provide a rigorous optimization-based framework for understanding and restoring network plasticity in complex RL domains. 1 Introduction The primary challenge in modern artificial intelligence lies in the inherent mismatch between the stationary assumptions of current learning paradigms and the non-stationary nature of real-world environments Dohare et al. [2024], Galashov et al. [2024]. While artificial neural networks (ANNs) have achieved remarkable success in various domains, their design and training methods are based on the assumption of a stationary data distribution, which greatly limits their practical applicability. This limitation becomes even more apparent when compared to biological neural systems, which possess the ability to continuously adapt through synaptic plasticity Puderbaugh and Emmady [2023]. This contrast has prompted researchers to investigate the fundamental limitations of ANNs Cohen et al. [2022]. In particular, the phenomenon of plasticity loss is a critical issue in reinforcement learning (RL) due to its complex dynamics compared to supervised learning Nikishin et al. [2022]. This challenge is further compounded by the need for RL to adapt value functions to changing rewards and evolving policies, which significantly affect state-action values Elsayed et al. [2024]. Unlike supervised learning, RL requires a higher level of network plasticity to handle dynamic environments and support strategy evolution during interactive learning Dohare et al. [2024]. Therefore, understanding and addressing plasticity loss is crucial in RL domains. Several hypotheses have been proposed to explain the loss of plasticity in neural networks. However, these hypotheses primarily focus on establishing correlations between specific metrics and performance, such as dormant neurons Sokar et al. [2023], loss landscape characteristics Lyle et al. [2023], and effective rank Gulcehre et al. [2022]. While these metrics are merely a summary of certain phenomena, they do not fully explain the fundamental mechanisms driving plasticity loss Lyle et al. [2023], Gulcehre et al. [2022], Lyle et al. [2024a], Lewandowski et al. [2024a], Lee et al. [2024]. We conducted a simple experiment to highlight the limitations of metric-based analysis. Specifically, we trained arXiv:2603.21173v1 [cs.LG] 22 Mar 2026 Rethinking Plasticity in Deep Reinforcement LearningA PREPRINT the efficient reinforcement learning algorithm PPO under two configurations: one using the ReLU activation function and the other without any activation function. The performance in terms of return and dormancy rate is presented in Figure 1. Interestingly, we observed that a higher dormancy ratio corresponds to a faster convergence rate. This finding contradicts the conclusion that dormancy leads to a loss of plasticity, yet aligns with the performance improvements attributed to sparsity Cheng et al. [2024]. 0.000.250.500.751.001.251.501.752.00 Step 1e7 0 1000 2000 3000 4000 Episode Return HalfCheetah-v4 PPO-No-Act PPO-ReLu 0.000.250.500.751.001.251.501.752.00 Step 1e7 0.00 0.01 0.02 0.03 0.04 0.05 0.06 Rate Policy-0-Layer-Dormant PPO-No-Act PPO-ReLu 0.000.250.500.751.001.251.501.752.00 Step 1e7 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 Rate Policy-1-Layer-Dormant PPO-No-Act PPO-ReLu 0.000.250.500.751.001.251.501.752.00 Step 1e7 0.0 0.2 0.4 0.6 0.8 Rate Value-0-Layer-Dormant PPO-No-Act PPO-ReLu 0.000.250.500.751.001.251.501.752.00 Step 1e7 0.0 0.2 0.4 0.6 0.8 Rate Value-1-Layer-Dormant PPO-No-Act PPO-ReLu Figure 1: The relationship between return and dormant neurons rate. The leftmost graph shows the return curve, while the next four depict the dormancy rates of policy and value layers. PPO-No-Act lacks activation functions, while PPO-ReLu uses ReLu. PPO-ReLu achieves faster convergence with higher dormancy rates, matching the performance of PPO-No-Act. This gap in understanding highlights the critical need for a more rigorous investigation into the underlying mechanisms that govern plasticity loss in neural networks. In contrast to existing metrics, we propose a novel hypothesis that examines plasticity loss from an optimization perspective. Our hypothesis analyzes why this problem occurs during learning and suggests that plasticity loss happens because optimal points from previous tasks become poor optimal points for new tasks. This causes parameters to become trapped in local optima during task transitions, preventing effective learning. Our hypothesis explains several observed phenomena: when tasks change, parameters stuck in local optima from previous tasks negatively impact current task performance, leading networks to adjust other neurons to zero out these trapped neurons’ outputs - resulting in neuron dormancy Sokar et al. [2023]. Additionally, optimizing other parameters while abandoning those in local optima makes the loss landscape steeper Lyle et al. [2022]. This also explains why parameter constraints can increase plasticity Elsayed et al. [2024], Kumar et al. [2023], as they limit network expressivity from the start, resulting in fewer neurons falling into local optima. The paper presents a hypothesis and validates it through several aspects: Firstly, previous research has shown that plasticity loss is not a data issue, but rather a result of the neural network’s inability to learn from the data Nikishin et al. [2022]. However, our hypothesis suggests that the core issue lies in the relationship between learning tasks, specifically the relationship between objective functions. When the objective function of the previous task has little correlation with the current task’s objective function, it reduces the likelihood of parameters getting trapped in local optima, thus mitigating the plasticity loss problem. Secondly, when tasks change, there is an increase in zero-gradient points, indicating more local optima. Once a neuron enters a zero-gradient state, it is highly likely to remain in the zero-gradient state during subsequent updates. Interestingly, perturbing these local optimal points does not lead to performance degradation, and may even improve performance. Our main contributions can be summarized as follows: 1. Novel Hypothesis on Plasticity Loss: This paper introduces a new hypothesis analyzing plasticity loss from an optimization perspective. It claims that plasticity loss arises because optimal points for previous tasks become suboptimal for new tasks, trapping parameters in local optima during task transitions and hindering neural network learning. 2.Explanation of Observed Phenomena: The hypothesis explains several observed phenomena, including neuron dormancy during task changes, steeper loss landscapes when optimizing parameters near local optima, and the effectiveness of parameter constraints in enhancing plasticity by limiting the network’s expressivity in its early stages. 3.Experimental Validation and Results: Validated the hypothesis and techniques through experiments, showing increased local optima during task transitions, performance improvements via perturbation of these optima, and significant reduction of plasticity loss with gradient-free optimization methods. Our results demonstrate substantial improvements in both adaptation speed and final performance across diverse non-stationary scenarios. 2 Rethinking Plasticity in Deep Reinforcement LearningA PREPRINT 2 Related Work Neural networks often exhibit a phenomenon known as diminishing adaptability to new task during the learning process. This has been described in various ways in reinforcement learning literature, such as primacy bias Nikishin et al. [2022], dormant neuron phenomenon Sokar et al. [2023], implicit underparameterization Kumar et al. [2021], and capacity loss Lyle et al. [2022]. In this paper, we will use the term "loss of plasticity" Dohare et al. [2024] to refer this phenomenon. While it shares some similarities with catastrophic forgetting Elsayed and Mahmood [2024], the focus of plasticity loss is on the network’s ability to learn new patterns, regardless of whether old knowledge is retained or not Dohare et al. [2024]. In this section, we will review existing literature to analyze various hypotheses proposed to explain this phenomenon and explore current solution approaches. Hypothesis for Plasticity Loss. Several hypotheses have been proposed to explain the loss of plasticity in neural networks. One prominent view suggests that networks become overly focused on early interaction data Nikishin et al. [2022], potentially due to ReLU activation functions causing neurons to become dormant Sokar et al. [2023]. However, in cases of redundant features, it may be reasonable for networks to suppress irrelevant features by adjusting weights and output to zero. Other research suggests that imposing specific constraints on network weights can enhance plasticity Elsayed et al. [2024], Kumar et al. [2023], Lewandowski et al. [2024b]. Additionally, some studies attribute plasticity loss to properties of the loss landscape Lyle et al. [2023] and effective rank Gulcehre et al. [2022]. Plasticity loss caused by a reduction of Hessian Rank Lewandowski et al. [2024a]. Furthermore, Shin et al. [2024] argue that neural networks tend to memorize noise during the pretraining stage. Despite these various hypotheses, the fundamental mechanisms underlying plasticity loss remain unclear Lyle et al. [2023], Gulcehre et al. [2022], Lyle et al. [2024a], Lewandowski et al. [2024a], Lee et al. [2024]. In contrast to existing work, we propose a hypothesis from an optimization perspective that can explain various observed metrics such as neuron dormancy and loss landscape characteristics. Preventing plasticity loss. Research on neural plasticity in deep reinforcement learning is still in its early stages, with current studies falling into two main categories. The first category focuses on preventing the loss of plasticity during the training process through various implementation-level techniques. These approaches include: normalization methods that regulate input or output distributions Lyle et al. [2024b]; weight constraint mechanisms that prevent excessive parameter magnitudes Elsayed et al. [2024] or parameters that deviate too far from the initial distribution Kumar et al. [2023] or early feature predictions Lyle et al. [2022]. Additionally, new activation functions have been designed to ensure that partial neurons are always activated for the same feature Abbas et al. [2023], and adaptive batch size strategies have been implemented Ceron et al. [2023]. It has been observed that large weight magnitudes can negatively impact optimization dynamics and are often associated with overfitting, leading to decreased performance and potentially contributing to the challenges encountered during learning Elsayed et al. [2024]. Weight Clip limits the change in function, which is crucial for on-policy reinforcement learning to prevent the policy from deviating too much from its previous state Elsayed et al. [2024]. Solution for plasticity loss. The second category focuses on restoring plasticity in networks where neurons have already lost their adaptive capabilities, known as the parameter resetting method Dohare et al. [2024], Juliani and Ash [2024], Liu et al. [2024]. These approaches include several innovative methods: Neuroplastic Expansion Liu et al. [2024], which enables incremental growth of network neurons; Continual Backpropagation Dohare et al. [2024], which identifies ineffective neurons and reinitializes their parameters; Continual backpropagation sets a new utility metric and lets a small fraction of less-used units are continually and randomly reinitialized Dohare et al. [2024]. ReDO Sokar et al. [2023], which detects and reinitializes dormant neurons; and Plasticity Injection Nikishin et al. [2024], which maintains network adaptability by introducing additional neurons during the training process. These approaches commonly rely on a binary metric that classifies neurons as either active or dormant based solely on whether their outputs are non-zero or zero. However, this dormancy metric lacks theoretical justification, as zero outputs may represent optimal and meaningful neural responses in certain contexts. In fact, some studies even adopt this metric as an optimization objective Xu et al. [2024].Some papers focus on optimizers, resetting them to clear historical information for enhanced network plasticity, particularly during rapid shifts in data distribution Asadi et al. [2023]. Unlike these methods, we propose a gradient-based metric to quantify neuronal plasticity, aiming to provide a more accurate understanding of network plasticity characteristics. 3 Optimization-Centric Plasticity (OCP) Hypothesis 3.1 An Intuitive Example of OCP To demonstrate the OCP hypothesis, let’s imagine a situation where we have two tasks,f 1 andf 2 , that need to be completed in sequence. As shown in Figure 2, the transition from Task 1 to Task 2 brings about a significant change in the objective function. While the neural network may successfully reach an optimal solution (indicated by the red 3 Rethinking Plasticity in Deep Reinforcement LearningA PREPRINT point) for Task 1, this solution may not be the best for Task 2 and even harm its performance. In other words, the network parameters at the red point are at risk of getting stuck in a local optimum that negatively affects Task 2. When training proceeds with data from taskf 2 , a subset of network parameters are already confined to these local optima. Consequently, the neural network must compensate by making more dramatic adjustments to the remaining, untrapped parameters to accommodate taskf 2 . This constrained optimization leads to two significant effects: a steeper loss landscape due to the reduced dimensionality of adjustable parameters Lyle et al. [2023], and the network potentially learning to suppress outputs associated with the trapped parameters if they interfere with taskf 2 performance. This mechanism explains the observed variable increases in dormant rates across different scenarios Sokar et al. [2023], as the phenomenon’s occurrence depends heavily on the specific task relationships and their respective optimization landscapes. Furthermore, this framework elucidates why regularization methods that constrain the parameter space can effectively mitigate plasticity loss: by preventing parameters from becoming too deeply entrenched in local optima for any single task, they preserve the network’s adaptability for subsequent learning. Figure 2: Illustration of the OCP hypothesis. 3.2 Task Relevance and Plasticity Loss In this subsection, our goal is to demonstrate that the loss of neural network plasticity is dependent on the task at hand, thus proving that the phenomenon is caused by being stuck in a local optimum. Specifically, the inability of a neural network to continue learning in a given task, as indicated by a high proportion of dormant neurons, does not necessarily mean that the network is incapable of learning. Instead, it suggests that the network struggles with tasks that are similar to the one being performed. However, when the task changes significantly, such as when the optimization landscape undergoes a substantial shift, the network’s neurons are able to learn and perform similarly to randomly initialized neurons, without experiencing a loss of plasticity. y = 2.5X 0 − 1.2X 2 1 + 0.8 sin(X 2 ) + 1.5 cos(X 3 ) + 0.7X 4 X 5 − 0.3X 3 6 + e −0.1X 2 7 + 1.1X 8 − 0.5X 2 9 + 0.9 tanh(X 10 ) + 0.2X 2 11 − 0.6 p |X 12 | + 0.5X 13 X 14 − 0.4X 15 + 0.3X 16 + ε,ε∼N (0,σ 2 ).(1) To test our hypothesis on the loss of task-specific plasticity, we utilized a pre-trained value network from PPO (Proximal Policy Optimization) for a regression task. This task involves a 17-dimensional input vectorX = (X 0 ,X 1 ,· ,X 16 )∈ R 17 and a single output valuey ∈ R. The true data-generating process is represented by Equation 1. More information on the experimental parameters, comparisons, and the reasoning behind the chosen functional form can be found in Appendix A. As shown in Figure 3, networks trained for 20 million steps with a low proportion of dormant neurons (PPO-No-Act- Test), networks trained for 20 million steps with a high proportion of dormant neurons (PPO-ReLU-Test), and randomly initialized networks (Random-Init-Test) all exhibited similar learning performance on the regression task. This suggests that when the task changes significantly, previously ineffective networks can successfully learn the new task, indicating that plasticity loss is not a fundamental issue. These results further validate our hypothesis that the loss of neural network plasticity is a task-specific phenomenon caused by neurons becoming trapped in a local optimum specific to the task. This evidence strongly supports our hypothesis. 3.3 Dormancy as a Gradient-Induced Phenomenon In this section, we will establish the theoretical equivalence between zero-gradient neurons and dormancy. We will demonstrate that the absence of gradient signals is what drives neurons into a dormant state. Additionally, we will 4 Rethinking Plasticity in Deep Reinforcement LearningA PREPRINT 2468101214 Iterations 0 1000 2000 3000 4000 5000 Loss (Unit: None) PPO-No-Act-Test PPO-ReLu-Test Random-Init-Test PPO-No-Act-TestPPO-ReLu-TestRandom-Init-Test Methods 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Loss (Unit: None) Figure 3: The convergence curves of the loss function and the final performance comparison on the test set were observed under task switching with a learning rate of 0.01. provide an optimization-based explanation for the tendency of the proportion of dormant neurons to increase when the underlying task changes. We consider a neural network layerlwith input spaceX l ⊆ R n l , parameterized byW l ∈ R n l+1 ×n l andb l ∈ R n l+1 , and with an activation functionσ l : R→ Rapplied elementwise. The forward pass of layerlisf l (x) = σ l (W l x + b l ), and the output of the i-th neuron in layer l is h l,i (x) = σ l w T l,i x + b l,i , where w T l,i is the i-th row of W l . Definition 1 (Dormant Neuron). LetH l be the number of neurons in layerl. The dormancy indexs l,i for neuron (l,i) is defined as: s l,i = E x∈D |h l,i (x)| 1 H l P k∈h E x∈D |h l,k (x)| (2) where E x∈D denotes the expectation over the input x drawn from distribution D. Assumption 1 (Continuity and Differentiability).h l,i (x)is continuously differentiable with respect tox, i.e., h l,i ∈ C 1 (R n l ). Assumption 2 (Boundedness). ∃M > 0 such that for all k ∈ h: E x∈D |h l,i (x)|≤ M . Assumption 3 (Non-degeneracy).∃m > 0such that 1 H l P k∈h E x∈D |h l,i (x)|≥ m . This ensures the denominator in the dormancy index definition is strictly positive. Lemma 1 (Zero Output Lemma). If s l,i = 0, then h l,i (x) = 0 for all x∈ D. Proof. Since s l,i = 0, 0 = s l,i = E x∈D |h l,i (x)| 1 H l P H l k=1 E x∈D |h l,k (x)| .(3) By the non-degeneracy assumption, the denominator is at least m > 0. Thus, E x∈D |h l,i (x)| = 0.(4) Since|h l,i (x)|≥ 0, the expectation of a nonnegative random variable is zero if and only if the variable is zero almost everywhere. Therefore, h l,i (x) = 0 for almost all x∈ D.(5) By the continuity ofh l,i , if it were nonzero at any pointx 0 ∈ D, continuity would imply a neighborhood around x 0 whereh l,i remains nonzero, contradicting the almost-everywhere zero condition. Hence,h l,i (x) = 0for every x∈ D. Lemma 2 (Dormancy-to-Gradient Lemma). If s l,i = 0, then ∆h l,i (x) = 0 for all x∈ D. Proof.From Lemma 1, we know that ifs l,i = 0,∀x ∈ D,h l,i (x) = 0. Letx ∈ Dbe arbitrary, and consider any perturbation δ x ∈ R n l . By the Mean Value Theorem for vector-valued functions, there exists t∈ [0, 1] such that h l,i (x + δ x )− h l,i (x) =∇h l,i (x + tδ x ) T δ x .(6) 5 Rethinking Plasticity in Deep Reinforcement LearningA PREPRINT Since h l,i (x) = 0 for every x∈ D, this difference is zero: 0 =∇h l,i (x + tδ x ) T δ x .(7) This equality must hold for allx∈ Dand for every possibleδ x ∈ R n l . Suppose, for contradiction, that there exists some point y∈ D where∇h l,i (y)̸= 0. Take δ x = ε∇h l,i (y) for a small ε > 0, then ∇h l,i (y + tε∇h l,i (y)) T (ε∇h l,i (y)) = 0.(8) Asε→ 0, continuity of∇h l,i ensures∇h l,i (y + tε∇h l,i (y))→∇h l,i (y), which is nonzero by assumption. Hence the inner product∇h l,i (y) T (ε∇h l,i (y))would not vanish for smallε̸= 0, yielding a contradiction. Thus, no suchy can exist, and we conclude ∇h l,i (x) = 0 ∀x∈ D.(9) Lemma 3 (Gradient-to-Dormancy Lemma). Suppose∇h l,i (x) = 0for allx ∈ X l . Thenh l,i (x)is constant on X l . Moreover, ifh l,i (x) = 0at any pointx 0 ∈X l , thenh l,i (x) = 0for allx∈X l , and hence the neuron is dormant (s l,i = 0). Proof.Since∇h l,i (x) = 0for everyx∈X l . Thenh l,i does not change with respect tox. By standard multivariable calculus, any function whose gradient is zero everywhere on a connected domain must be constant. Concretely, for any two points x 1 , x 2 ∈X l , we have h l,i (x 2 )− h l,i (x 1 ) = Z x 2 x 1 ∇h l,i (u)du = 0,(10) indicatingh l,i (x 2 ) = h l,i (x 1 ). Therefore,h l,i (x)is constant onX l . Denote this constant byc, soh l,i (x) = cfor all x∈X l . Next, assume there exists a pointx 0 ∈Xsuch thath l,i (x 0 ) = 0, thenc = 0. Henceh l,i (x) = 0for everyx∈X l . In particular, for x∈ D ⊆X l , we have h l,i (x) = 0. Consequently, E x∈D |h l,i (x)| = 0=⇒ s l,i = 0 1 H H P k E x∈D |h l,k (x)| = 0.(11) Thus the neuron is fully dormant. Remark. Ifc̸= 0, thenE x∈D |h l,i (x)|would be|c|̸= 0, sos l,i could potentially be nonzero, indicating the neuron is not dormant. Therefore, the key condition thath l,i (x) = 0at some pointx 0 inX l (thus forcingc = 0) is crucial to concludings l,i = 0. In many network architectures, one can ensureh l,i (0) = 0by design (e.g., bias initialized to zero and activation is ReLU), or rely on data/architectural constraints that force a zero constant rather than a nonzero one. Theorem 1 (Equivalence of Dormancy and Zero Gradient). LetD ⊆X l ⊆ R n l be the domain from which inputsx are drawn. Consider layer l with H l neurons. For the i-th neuron in layer l, let h l,i (x) = σ l w T l,i x + b l,i ,(12) where σ l is continuously differentiable and applied elementwise. The neuron’s dormancy index is defined as: s l,i = E x∈D |h l,i (x)| 1 H l P H l k=1 E x∈D |h l,k (x)| (13) Assume: 1. (Continuity and Differentiability) h l,i (x) is in C 1 (R n l ). 2. (Boundedness)∃M > 0: E x∈D |h l,i (x)|≤ M . 3. (Non-degeneracy)∃m > 0 : 1 H l P H l k=1 E x∈D |h l,k (x)|≥ m. Then the following two statements are equivalent: • (A) Dormancy: s l,i = 0. Equivalently, E x∈D |h l,i (x)| = 0. 6 Rethinking Plasticity in Deep Reinforcement LearningA PREPRINT • (B) Zero Gradient (on D) plus at least one zero activation: ∇h l,i (x) = 0 ∀x∈ Dand ∃x 0 ∈ D : h l,i (x 0 ) = 0.(14) Proof.(A)⇒(B): Supposes l,i = 0. By Lemma 1,h l,i (x) = 0for allx ∈ D. Then by the Lemma 2, we get ∇h l,i (x) = 0for allx∈ D. Clearly, in this scenario, pickx 0 ∈ Darbitrarily;h l,i (x 0 ) = 0. Thus the right-hand side of the equivalence is satisfied. (A)⇐(B): Conversely, assume∇h l,i (x) = 0for allx ∈ Dand there exists at least one pointx 0 ∈ Dwhereh l,i (x 0 ) = 0. Since∇h l,i ≡ 0onD, the neuron’s outputh l,i is constant throughoutD. That constant must be zero (as it is zero atx 0 ). Henceh l,i (x) = 0for allx∈ D. By Lemma 1 (applied in reverse logic), if the neuron is identically zero on D, then s l,i = 0. 0.000.250.500.751.001.251.501.752.00 Step 1e7 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 Rate Policy-1-Layer-Dormant PPO-No-Act PPO-ReLu 0.000.250.500.751.001.251.501.752.00 Step 1e7 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 Rate Policy-1-Layer-ZeroGradient PPO-No-Act PPO-ReLu 0.000.250.500.751.001.251.501.752.00 Step 1e7 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 Rate Policy-1-Layer-DormantOverLeap PPO-No-Act PPO-ReLu 0.000.250.500.751.001.251.501.752.00 Step 1e7 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 Rate Policy-1-Layer-ZeroGradientOverLeap PPO-No-Act PPO-ReLu 0.000.250.500.751.001.251.501.752.00 Step 1e7 0.0 0.2 0.4 0.6 0.8 Rate Value-1-Layer-Dormant PPO-No-Act PPO-ReLu 0.000.250.500.751.001.251.501.752.00 Step 1e7 0.2 0.4 0.6 0.8 Rate Value-1-Layer-ZeroGradient PPO-No-Act PPO-ReLu 0.000.250.500.751.001.251.501.752.00 Step 1e7 0.0 0.2 0.4 0.6 0.8 Rate Value-1-Layer-DormantOverLeap PPO-No-Act PPO-ReLu 0.000.250.500.751.001.251.501.752.00 Step 1e7 0.2 0.4 0.6 0.8 Rate Value-1-Layer-ZeroGradientOverLeap PPO-No-Act PPO-ReLu Figure 4: The relationship between dormant neurons and zero gradient neurons in last layer of policy and value neural network in HalfCheetah-v4 environment. The Dormancy Index and ZeroGradient Index show strong correlation. The OverLeap metric reflects the proportion of neurons that remain dormant or zero-gradient across iterations, further highlighting this consistency. Theorem 1 states that a neuron with a dormancy index of zero will result in zero outputs on the dataset, causing a zero gradient that will keep it in a dormant state. On the other hand, if a neuron’s gradient is negligible, it will either remain or become close to zero in output, resulting in no updates. Even small fluctuations that push its parameters to a constant or near-zero state will further reinforce this condition, particularly in networks with ReLU activation functions. The experimental results in Figure 4 also validate our theory: there is a strong correlation between dormant neurons and neurons with zero gradients. Similar to dormant neurons, Zero Gradient neurons gradually increase during the training process. Once a neuron enters a dormant or zero-gradient state, it becomes irrecoverable. The OverLeap plot illustrates this phenomenon clearly. Further experimental setups and analyses can be found in Appendix B. From an optimization standpoint, parameter updates follow gradient signals. However, when the task changes, the loss function also changes, often resulting in parameters being near old local optima with negligible or zero gradients for the new objective. This can lead to neurons specialized in outdated features becoming or remaining dormant. Even a slight shift in the task can cause the network’s parameters to stay close to their previous minima, perpetuating low gradients and increasing the rate of dormancy. On the other hand, a significant task change usually invalidates the old minima, resulting in stronger gradients that can "wake up" previously dormant neurons. Once a neuron’s output reaches zero, its gradient also becomes small, creating a self-reinforcing loop that keeps it dormant unless strong new signals are received. Theoretically and practically, dormancy is closely linked to having zero or vanishingly small gradients. 7 Rethinking Plasticity in Deep Reinforcement LearningA PREPRINT References Shibhansh Dohare, J Fernando Hernandez-Garcia, Qingfeng Lan, Parash Rahman, A Rupam Mahmood, and Richard S Sutton. Loss of plasticity in deep continual learning. Nature, 632(8026):768–774, 2024. Alexandre Galashov, Michalis Titsias, András György, Clare Lyle, Razvan Pascanu, Yee Whye Teh, and Maneesh Sahani. Non-stationary learning of neural networks with automatic soft parameter reset. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URLhttps://openreview.net/forum? id=fDiZJ7mmOV. Matt Puderbaugh and Prabhu D Emmady. Neuroplasticity. In StatPearls [Internet]. StatPearls Publishing, 2023. Yarden Cohen, Tatiana A Engel, Christopher Langdon, Grace W Lindsay, Torben Ott, Megan AK Peters, James M Shine, Vincent Breton-Provencher, and Srikanth Ramaswamy. Recent advances at the interface of neuroscience and artificial neural networks. Journal of Neuroscience, 42(45):8514–8523, 2022. Evgenii Nikishin, Max Schwarzer, Pierluca D’Oro, Pierre-Luc Bacon, and Aaron C. Courville. The primacy bias in deep reinforcement learning. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, ICML, volume 162 of Proceedings of Machine Learning Research, pages 16828–16847. PMLR, 2022. Mohamed Elsayed, Qingfeng Lan, Clare Lyle, and A. Rupam Mahmood. Weight clipping for deep continual and reinforcement learning. Reinforcement Learning Journal, 5:2198–2217, 2024. Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evci. The dormant neuron phenomenon in deep reinforcement learning. In International Conference on Machine Learning, pages 32145–32168. PMLR, 2023. Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Ávila Pires, Razvan Pascanu, and Will Dabney. Understanding plasticity in neural networks. In ICML, volume 202 of Proceedings of Machine Learning Research, pages 23190– 23211. PMLR, 2023. URL http://dblp.uni-trier.de/db/conf/icml/icml2023.html#LyleZNPPD23. Caglar Gulcehre, Srivatsan Srinivasan, Jakub Sygnowski, Georg Ostrovski, Mehrdad Farajtabar, Matthew Hoffman, Razvan Pascanu, and Arnaud Doucet. An empirical study of implicit regularization in deep offline RL. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URLhttps://openreview.net/forum?id=HFfJWx60IT. Clare Lyle, Zeyu Zheng, Khimya Khetarpal, Hado van Hasselt, Razvan Pascanu, James Martens, and Will Dabney. Disentangling the causes of plasticity loss in neural networks, 2024a. URLhttps://arxiv.org/abs/2402. 18762. Alex Lewandowski, Haruto Tanaka, Dale Schuurmans, and Marlos C. Machado. Directions of curvature as an explanation for loss of plasticity, 2024a. URL https://arxiv.org/abs/2312.00246. Hojoon Lee, Hyeonseo Cho, Hyunseung Kim, Donghu Kim, Dugki Min, Jaegul Choo, and Clare Lyle. Slow and steady wins the race: Maintaining plasticity with hare and tortoise networks. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 26416–26438. PMLR, 21–27 Jul 2024. URL https://proceedings.mlr.press/v235/lee24d.html. Hongrong Cheng, Miao Zhang, and Javen Qinfeng Shi. A survey on deep neural network pruning: Taxonomy, comparison, analysis, and recommendations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in reinforcement learning. In International Conference on Learning Representations, 2022. URLhttps://openreview.net/forum?id= ZkC8wKoLbQ7. Saurabh Kumar, Henrik Marklund, and Benjamin Van Roy. Maintaining plasticity in continual learning via regenerative regularization. 2023. Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, and Sergey Levine. Implicit under-parameterization inhibits data- efficient deep reinforcement learning. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=O9bnihsFfXU. Mohamed Elsayed and A. Rupam Mahmood. Addressing loss of plasticity and catastrophic forgetting in continual learning. In The Twelfth International Conference on Learning Representations, 2024. URLhttps://openreview. net/forum?id=sKPzAXoylB. Alex Lewandowski, Saurabh Kumar, Dale Schuurmans, András György, and Marlos C Machado. Learning continually by spectral regularization. arXiv preprint arXiv:2406.06811, 2024b. 8 Rethinking Plasticity in Deep Reinforcement LearningA PREPRINT Baekrok Shin, Junsoo Oh, Hanseul Cho, and Chulhee Yun. DASH: Warm-starting neural network training without loss of plasticity under stationarity. In 2nd Workshop on Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource Optimization (WANT@ICML 2024), 2024. URLhttps://openreview.net/forum? id=GR5LXaglgG. Clare Lyle, Zeyu Zheng, Khimya Khetarpal, James Martens, Hado van Hasselt, Razvan Pascanu, and Will Dabney. Normalization and effective learning rates in reinforcement learning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024b. URL https://openreview.net/forum?id=ZbjJE6Nq5k. Zaheer Abbas, Rosie Zhao, Joseph Modayil, Adam White, and Marlos C. Machado. Loss of plasticity in continual deep reinforcement learning. In CoLLAs, volume 232 of Proceedings of Machine Learning Research, pages 620–636. PMLR, 2023. URL http://dblp.uni-trier.de/db/conf/collas/collas2023.html#AbbasZM0M23. Johan Samir Obando Ceron, Marc G Bellemare, and Pablo Samuel Castro. Small batch deep reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URLhttps://openreview.net/ forum?id=wPqEvmwFEh. Arthur Juliani and Jordan T Ash. A study of plasticity loss in on-policy deep reinforcement learning. arXiv preprint arXiv:2405.19153, 2024. Jiashun Liu, Johan Obando-Ceron, Aaron Courville, and Ling Pan. Neuroplastic expansion in deep reinforcement learning. arXiv preprint arXiv:2410.07994, 2024. Evgenii Nikishin, Junhyuk Oh, Georg Ostrovski, Clare Lyle, Razvan Pascanu, Will Dabney, and André Barreto. Deep reinforcement learning with plasticity injection. Advances in Neural Information Processing Systems, 36, 2024. Guowei Xu, Ruijie Zheng, Yongyuan Liang, Xiyao Wang, Zhecheng Yuan, Tianying Ji, Yu Luo, Xiaoyu Liu, Jiaxin Yuan, Pu Hua, Shuzhen Li, Yanjie Ze, Hal Daumé I, Furong Huang, and Huazhe Xu. Drm: Mastering visual reinforcement learning through dormant ratio minimization. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=MSe8YFbhUE. Kavosh Asadi, Rasool Fakoor, and Shoham Sabach. Resetting the optimizer in deep RL: An empirical study. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URLhttps://openreview.net/ forum?id=AnFUgNC3Yc. 9 Rethinking Plasticity in Deep Reinforcement LearningA PREPRINT A Supplementary Details on Task Relevance and Plasticity Loss The state space of HalfCheetah-v4 is 17-dimensional. In order to assess the plasticity of its Value function, we have devised a regression problem with a 17-dimensional input vectorX = (X 0 ,X 1 ,· ,X 16 )∈ R 17 and a scalar output y ∈ R. The true underlying data-generating process is described by Equation 15. y = 2.5X 0 − 1.2X 2 1 + 0.8 sin(X 2 ) + 1.5 cos(X 3 ) + 0.7X 4 X 5 − 0.3X 3 6 + e −0.1X 2 7 + 1.1X 8 − 0.5X 2 9 + 0.9 tanh(X 10 ) + 0.2X 2 11 − 0.6 p |X 12 | + 0.5X 13 X 14 − 0.4X 15 + 0.3X 16 + ε,ε∼N (0,σ 2 ).(15) In Equation 15, the variableεrepresents an additive noise term that is randomly sampled from a zero-mean Gaussian distribution with a variance ofσ 2 , whereσ = 0.1. The functionyincludes a variety of mathematical operators, such as linear, quadratic, cubic, trigonometric (sin, cos), exponential (e −0.1X 2 7 ), and hyperbolic (tanh) transformations, as well as interaction terms (e.g.,X 4 X 5 ,X 13 X 14 ) and non-smooth functions (e.g., p |X 12 |). This diverse structure introduces significant nonlinearity and complexity to the regression task. The main purpose of this design is to ensure that the problem has a wide range of optimization landscapes, allowing us to test whether a neural network can effectively adapt to a new and substantially different task if it becomes trapped in local optima under one task. 2468101214 Iterations 0 200 400 600 800 1000 Loss (Unit: None) PPO-No-Act-Test PPO-ReLu-Test Random-Init-Test PPO-No-Act-TestPPO-ReLu-TestRandom-Init-Test Methods 0 10 20 30 40 Loss (Unit: None) (a) 2468101214 Iterations 0 2000 4000 6000 8000 10000 12000 14000 Loss (Unit: None) PPO-No-Act-Test PPO-ReLu-Test Random-Init-Test PPO-No-Act-TestPPO-ReLu-TestRandom-Init-Test Methods 0 1 2 3 4 5 6 7 8 Loss (Unit: None) (b) Figure 5: Loss convergence curves and final convergence results under different learning rates. (a) shows the results for a learning rate of 0.01, while (b) shows the results for a learning rate of 0.005. The training and test datasets consist of 1,000 randomly generated samples, and the loss function used is Mean Squared Error (MSE). The experimental results are shown in Figure 5. "PPO-No-Act-Test" refers to the results on the test dataset where the initial weights of the value function were obtained from the PPO algorithm after 20 million steps in the HalfCheetah-v4 environment without activation functions. "PPO-ReLU-Test" represents the test results using initial weights from the PPO algorithm after 20 million steps in the same environment with ReLu activation functions. "Random-Init" indicates the test results with randomly initialized weights. It can be observed that under different learning rates, the convergence speed and final results of pre-trained neural networks with a high proportion of dormant weights show no significant difference compared to those of randomly initialized networks. This demonstrates that the plasticity of neural networks is highly task-dependent. 10 Rethinking Plasticity in Deep Reinforcement LearningA PREPRINT B The Experimental Setups And Analyses of Dormant Neuron and Zero Gradient Neuron Definition 2 (Mean Absolute Gradient Intensity, MAGI). We define the Mean Absolute Gradient Intensity (MAGI) for thei-th neuron in layerlas the mean absolute value of the gradients of all incoming weights to that neuron, averaged over the input feature dimension: G (l) i = 1 n in n in X j=1 ∂S ∂w (l) ij (16) wheren in is the input dimension of layerl, representing the number of input features or neurons connected to thei-th neuron in the previous layer.w (l) i,j is the weight connecting thej-th input feature (or neuron) to thei-th neuron in layer l. S is the target function, defined as: S = n batch X k=1 n out X m=1 Y k,m (17) wheren batch is the batch size, representing the number of samples in the current batch.n out is the output dimension of layer l, i.e., the number of neurons in the current layer. Y ∈ R n batch ×n out is the output of layer l, computed as: Y = X· W ⊤ l + b l (18) whereX∈ R n batch ×n in is the input to layerl, consisting ofn batch samples, each withn in features.W l ∈ R n out ×n in is the weight matrix of layer l, connecting the input to the output neurons. b l ∈ R n out is the bias vector of layer l. The gradient ∂S ∂w (l) ij represents the sensitivity of the target functionSto changes in the weightw (l) ij , and the absolute value is taken to avoid positive and negative gradients canceling each other out. The mean overn in input feature reflects the overall sensitivity of the neuron to its incoming connections. MAGI quantifies the importance of a neuron in layerl by summarizing the average gradient magnitude across all its incoming weights. It provides a concise and interpretable measure of the role a neuron plays during optimization and can be used to identify important neurons, prune redundant ones, and guide architectural decisions in model design. Definition 3 (Overlap Coefficient for Neuron Dormancy or Zero Gradient). The overlap coefficient quantifies the similarity between the set of dormant or zero gradient neurons in each layer of a neural network at the current iteration and last iteration Sokar et al. [2023]. LetAdenote the set of dormant or zero gradient neurons at the current iteration, andBdenote the previous time step dormant and zero gradient. The overlap coefficient betweenAandBis defined as : overlap(A,B) = |A∩ B| min(|A|,|B|) ,(19) This metric is used to measure the proportion of dormant neurons or zero-gradient neurons that remain consistent over time, providing insights into the network’s insight state. In this study, we have chosen to use the PPO algorithm as our basic algorithm framework. We have trained our model for a total of 20 million timesteps. To optimize the learning process, we have implemented an annealed learning rate starting at1× 10 −4 and a weight decay of1× 10 −4 . In order to stabilize the training, we have also utilized a gamma discount factor of 0.99 and Generalized Advantage Estimation (GAE) withλ = 0.95. Our policy gradient is optimized using 32 minibatches over 10 update epochs per iteration, with gradient clipping applied using a maximum norm of 0.5. Additionally, we have employed a clipped surrogate loss with a clipping coefficient of 0.2 and ensured value function stability through a value loss coefficient of 0.5. The number of dormant neurons and zero-gradient neurons in each layer of the Policy and Value neural networks is shown in Figure 6. From the experimental results, we can draw the following conclusions: • Similar to dormant neurons, Zero Gradient neurons gradually increase during the training process. • Once neurons enter the Zero Gradient state, they cannot exit this state. • The proportion of dormant neurons and zero-gradient neurons exhibits a high degree of consistency, particularly in the final layer of the neural network. 11 Rethinking Plasticity in Deep Reinforcement LearningA PREPRINT 0.000.250.500.751.001.251.501.752.00 Step 1e7 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 Rate Policy-1-Layer-Dormant PPO-No-Act PPO-ReLu 0.000.250.500.751.001.251.501.752.00 Step 1e7 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 Rate Policy-1-Layer-ZeroGradient PPO-No-Act PPO-ReLu 0.000.250.500.751.001.251.501.752.00 Step 1e7 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 Rate Policy-1-Layer-DormantOverLeap PPO-No-Act PPO-ReLu 0.000.250.500.751.001.251.501.752.00 Step 1e7 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 Rate Policy-1-Layer-ZeroGradientOverLeap PPO-No-Act PPO-ReLu 0.000.250.500.751.001.251.501.752.00 Step 1e7 0.0 0.2 0.4 0.6 0.8 Rate Value-1-Layer-Dormant PPO-No-Act PPO-ReLu 0.000.250.500.751.001.251.501.752.00 Step 1e7 0.2 0.4 0.6 0.8 Rate Value-1-Layer-ZeroGradient PPO-No-Act PPO-ReLu 0.000.250.500.751.001.251.501.752.00 Step 1e7 0.0 0.2 0.4 0.6 0.8 Rate Value-1-Layer-DormantOverLeap PPO-No-Act PPO-ReLu 0.000.250.500.751.001.251.501.752.00 Step 1e7 0.2 0.4 0.6 0.8 Rate Value-1-Layer-ZeroGradientOverLeap PPO-No-Act PPO-ReLu 0.000.250.500.751.001.251.501.752.00 Step 1e7 0.00 0.01 0.02 0.03 0.04 0.05 0.06 Rate Policy-0-Layer-Dormant PPO-No-Act PPO-ReLu 0.000.250.500.751.001.251.501.752.00 Step 1e7 0.000 0.001 0.002 0.003 0.004 0.005 Rate Policy-0-Layer-ZeroGradient PPO-No-Act PPO-ReLu 0.000.250.500.751.001.251.501.752.00 Step 1e7 0.00 0.01 0.02 0.03 0.04 0.05 Rate Policy-0-Layer-DormantOverLeap PPO-No-Act PPO-ReLu 0.000.250.500.751.001.251.501.752.00 Step 1e7 0.000 0.001 0.002 0.003 0.004 0.005 Rate Policy-0-Layer-ZeroGradientOverLeap PPO-No-Act PPO-ReLu 0.000.250.500.751.001.251.501.752.00 Step 1e7 0.0 0.2 0.4 0.6 0.8 Rate Value-0-Layer-Dormant PPO-No-Act PPO-ReLu 0.000.250.500.751.001.251.501.752.00 Step 1e7 0.0 0.2 0.4 0.6 0.8 Rate Value-0-Layer-ZeroGradient PPO-No-Act PPO-ReLu 0.000.250.500.751.001.251.501.752.00 Step 1e7 0.0 0.2 0.4 0.6 0.8 Rate Value-0-Layer-DormantOverLeap PPO-No-Act PPO-ReLu 0.000.250.500.751.001.251.501.752.00 Step 1e7 0.0 0.2 0.4 0.6 0.8 Rate Value-0-Layer-ZeroGradientOverLeap PPO-No-Act PPO-ReLu Figure 6: The relationship between dormant neurons and zero gradient neurons in each layer neural network in HalfCheetah-v4 environment. • The dormancy rate and zero-gradient proportion of neurons in the final layer are significantly higher than the zero-gradient proportion in the preceding layer. • Under the Value-0-Layer, Zero Gradient neurons possess a certain degree of recovery ability and can also help reduce dormant neurons. Overall, the experimental results validate our Theorem 1 (Equivalence of Dormancy and Zero Gradient). However, there are still some discrepancies between the results and the theoretical predictions, such as the dormancy rate of neurons being slightly higher than the zero-gradient proportion in the 0th layer of the Value network. Additionally, why are the dormancy rate and zero-gradient proportion higher in the final layer of neurons? There are several common reasons why one might observe a neuron whose empirical “dormancy” measures out to zero (or very close to zero) yet whose gradient is not exactly zero in practice, despite the theoretical result stating that a strictly zero-output neuron must have a strictly zero gradient. Below are some explanations: Finite Sampling vs. Theoretical Domain The proof assumes the neuron’s output is identically zero for all inputsxin the data domain. In a real experiment, you only have a finite sample of points (minibatches, training sets, etc.), and the neuron might be zero on average or nearly zero for that finite sample, rather than genuinely zero everywhere. Consequently, even if the dormancy metric is numerically zero, the neuron may still produce small nonzero values for some unobserved inputs, hence giving rise to small but nonzero gradients. Floating-Point and Numerical Approximation Computers store and process numbers in floating-point representation. Values that are “close to zero” often end up being rounded to zero (or appear extremely small) during forward passes and in statistics like the dormancy measure. However, backpropagation and gradient calculations may amplify tiny differences, producing nonzero gradients even if the output is extremely close to zero in the forward pass. This discrepancy is a byproduct of finite precision arithmetic and can cause “dormant” neurons to have small, nonzero gradients. Momentum and Other Optimizer Dynamics If your optimization includes momentum (e.g., in SGD with momentum, Adam, RMSProp, etc.), the gradient update for a neuron can remain nonzero temporarily, even if the neuron’s instantaneous output is consistently near zero. The optimizer’s internal state (momentum buffer, adaptive learning rates, etc.) may carry over nonzero update terms that slightly adjust the neuron’s parameters despite the neuron having negligible or zero output on the dataset. Regularization and Other Loss Components In many practical neural networks, the total loss function is not just the standard training objective (e.g., cross-entropy) but also includes regularization terms (weight decay, L1/L2 penalties, etc.). Even if the neuron output is zero, changes in that neuron’s parameters might still affect the regularization loss term, thereby giving a small gradient signal with respect to those parameters. “Almost Dormant” vs. Strictly Dormant The theoretical result “If output is strictly zero for allx∈ D, then gradient is strictly zero” is exact only in an ideal mathematical sense. In practice, a dormancy index of zero typically means “so small that it was rounded or thresholded to zero,” not that the neuron is provably zero on the entire input domain. A minuscule nonzero output can still lead to a nontrivial gradient. Distribution Shift or Data Subset Effects Sometimes the neuron is dormant on the training set or a certain subset of data but might not be truly zero on the broader distribution or on another part of the dataset. Therefore, when gradients 12 Rethinking Plasticity in Deep Reinforcement LearningA PREPRINT are computed over a slightly different batch or distribution slice, that neuron might receive a small gradient even though it was “dormant” under the original dataset slice. There are several common reasons why the final layer in a neural network is often more prone to having dormant (or effectively zero-gradient) neurons: Direct Dependency on the Loss The gradient signals reaching the final layer come directly from the loss function. If certain output neurons (or individual dimensions in the final layer) do not reduce the loss under the current task, they receive negligible or zero gradients. Consequently, those neurons see no incentive to update, causing their outputs to drift toward zero (dormancy) and remain there. Reduced Intermediary “Revival” Opportunities In earlier layers, if a neuron is near-zero, it can still get “woken up” by meaningful signals from subsequent layers or skip connections. By contrast, in the final layer, there are no additional transformations after it—if the loss does not depend on that particular neuron’s output, there is no mechanism to reignite its gradient. Thus, once it goes dormant, it stays that way. Sharper Task Specialization The final layer often encodes the most task-specific features (such as class logits). When the task changes (e.g., in transfer learning or domain shift), certain final-layer neurons that were specific to now-irrelevant classes or features may suddenly become useless for reducing the new loss. With no gradient feedback, those neurons quickly or permanently drop to zero output. Sparsity Induced by Softmax or Output Activation In classification, the final layer frequently feeds into a softmax function. If one or more logits consistently fail to be competitive, they produce negligible probability mass and generate little to no gradient over multiple training steps. This leads to a self-reinforcing effect: low output yields low gradient, which keeps the output low. Fewer Parameter Interactions Earlier layers often have rich parameter interactions (via multiple subsequent layers), giving nonzero gradient signals to neurons that could still help shape intermediate representations. In the final layer, each output neuron is directly tied to a single dimension of the final prediction. If that dimension is not contributing to reducing the loss, its gradient can vanish without being “rescued” by other parts of the network. 13 Rethinking Plasticity in Deep Reinforcement LearningA PREPRINT C Statistics about Weights 0.000.250.500.751.001.251.501.752.00 Step 1e7 20 40 60 80 100 120 Weight Mean (Unit: None) Policy-0-Layer-WeightMean PPO-No-Act PPO-ReLu 0.000.250.500.751.001.251.501.752.00 Step 1e7 10 20 30 40 50 60 Weight Mean (Unit: None) Policy-1-Layer-WeightMean PPO-No-Act PPO-ReLu 0.000.250.500.751.001.251.501.752.00 Step 1e7 5 10 15 20 Weight Mean (Unit: None) Value-0-Layer-WeightMean PPO-No-Act PPO-ReLu 0.000.250.500.751.001.251.501.752.00 Step 1e7 0 2 4 6 8 10 12 Weight Mean (Unit: None) Value-1-Layer-WeightMean PPO-No-Act PPO-ReLu 0.000.250.500.751.001.251.501.752.00 Step 1e7 20 40 60 80 100 120 Weight Std (Unit: None) Policy-0-Layer-WeightStd PPO-No-Act PPO-ReLu 0.000.250.500.751.001.251.501.752.00 Step 1e7 10 20 30 40 50 Weight Std (Unit: None) Policy-1-Layer-WeightStd PPO-No-Act PPO-ReLu 0.000.250.500.751.001.251.501.752.00 Step 1e7 5 10 15 20 25 30 35 40 Weight Std (Unit: None) Value-0-Layer-WeightStd PPO-No-Act PPO-ReLu 0.000.250.500.751.001.251.501.752.00 Step 1e7 2 4 6 8 10 Weight Std (Unit: None) Value-1-Layer-WeightStd PPO-No-Act PPO-ReLu 0.000.250.500.751.001.251.501.752.00 Step 1e7 0 2 4 6 8 Weight Change (Unit: None) Policy-0-Layer-WeightChange PPO-No-Act PPO-ReLu 0.000.250.500.751.001.251.501.752.00 Step 1e7 0 20 40 60 80 100 Weight Change (Unit: None) Policy-1-Layer-WeightChange PPO-No-Act PPO-ReLu 0.000.250.500.751.001.251.501.752.00 Step 1e7 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 Weight Change (Unit: None) Value-0-Layer-WeightChange PPO-No-Act PPO-ReLu 0.000.250.500.751.001.251.501.752.00 Step 1e7 0 20 40 60 80 100 Weight Change (Unit: None) Value-1-Layer-WeightChange PPO-No-Act PPO-ReLu Figure 7: Weight Statistic Information 14 Rethinking Plasticity in Deep Reinforcement LearningA PREPRINT D Statistics about Ranks 0.000.250.500.751.001.251.501.752.00 Step 1e7 254.8 255.0 255.2 255.4 255.6 255.8 256.0 Rank (Unit: None) Policy-0-Layer-Rank PPO-No-Act PPO-ReLu 0.000.250.500.751.001.251.501.752.00 Step 1e7 160 180 200 220 240 260 Rank (Unit: None) Policy-1-Layer-Rank PPO-No-Act PPO-ReLu 0.000.250.500.751.001.251.501.752.00 Step 1e7 225 230 235 240 245 250 255 Rank (Unit: None) Value-0-Layer-Rank PPO-No-Act PPO-ReLu 0.000.250.500.751.001.251.501.752.00 Step 1e7 75 100 125 150 175 200 225 250 Rank (Unit: None) Value-1-Layer-Rank PPO-No-Act PPO-ReLu 0.000.250.500.751.001.251.501.752.00 Step 1e7 20 40 60 80 100 120 Effective Rank (Unit: None) Policy-0-Layer-Eff-Rank PPO-No-Act PPO-ReLu 0.000.250.500.751.001.251.501.752.00 Step 1e7 0 20 40 60 80 100 120 140 Effective Rank (Unit: None) Policy-1-Layer-Eff-Rank PPO-No-Act PPO-ReLu 0.000.250.500.751.001.251.501.752.00 Step 1e7 0 20 40 60 80 100 120 Effective Rank (Unit: None) Value-0-Layer-Eff-Rank PPO-No-Act PPO-ReLu 0.000.250.500.751.001.251.501.752.00 Step 1e7 0 20 40 60 80 Effective Rank (Unit: None) Value-1-Layer-Eff-Rank PPO-No-Act PPO-ReLu 0.000.250.500.751.001.251.501.752.00 Step 1e7 0 20 40 60 80 100 120 Approximate Rank (Unit: None) Policy-0-Layer-Approx-Rank PPO-No-Act PPO-ReLu 0.000.250.500.751.001.251.501.752.00 Step 1e7 0 20 40 60 80 100 120 140 160 Approximate Rank (Unit: None) Policy-1-Layer-Approx-Rank PPO-No-Act PPO-ReLu 0.000.250.500.751.001.251.501.752.00 Step 1e7 0 20 40 60 80 100 120 Approximate Rank (Unit: None) Value-0-Layer-Approx-Rank PPO-No-Act PPO-ReLu 0.000.250.500.751.001.251.501.752.00 Step 1e7 0 20 40 60 80 Approximate Rank (Unit: None) Value-1-Layer-Approx-Rank PPO-No-Act PPO-ReLu 0.000.250.500.751.001.251.501.752.00 Step 1e7 50 100 150 200 250 Approximate Abs Rank (Unit: None) Policy-0-Layer-Abs-Approx-Rank PPO-No-Act PPO-ReLu 0.000.250.500.751.001.251.501.752.00 Step 1e7 0 50 100 150 200 250 Approximate Abs Rank (Unit: None) Policy-1-Layer-Abs-Approx-Rank PPO-No-Act PPO-ReLu 0.000.250.500.751.001.251.501.752.00 Step 1e7 0 50 100 150 200 Approximate Abs Rank (Unit: None) Value-0-Layer-Abs-Approx-Rank PPO-No-Act PPO-ReLu 0.000.250.500.751.001.251.501.752.00 Step 1e7 0 50 100 150 200 Approximate Abs Rank (Unit: None) Value-1-Layer-Abs-Approx-Rank PPO-No-Act PPO-ReLu Figure 8: Weight Statistic Information 15