Paper deep dive
Learning Under Forgetting: Statistical Support-Selective Retention in Stochastic Training Dynamics
Fujie Gao, Zuyue Zhang, Gang Sun
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 10/4/2026, 2:59:45 AM
Summary
The paper introduces Repeated Reinforcement with Persistent Forgetting (RPF) dynamics, a theoretical framework explaining how neural networks selectively retain patterns during training. RPF posits that forgetting acts as a selection mechanism rather than just a failure mode. The authors derive an exposure-selective survival law for independent features and spectral filtering for shared parameters, showing that patterns persist if their effective support (exposure frequency and recoverability) outweighs the persistent forgetting pressure. This process induces an implicit Minimum Description Length (MDL) compression effect, balancing data fitting against the cost of stored information.
Entities (9)
Relation Signals (7)
RPF → treats → Forgetting
confidence 95% · This view treats forgetting not merely as a failure mode, but as a selection mechanism.
Independent-Feature Model → derives → Exposure-Selective Survival Law
confidence 93% · in an independent-feature model, we derive an exposure-selective survival law
RPF → induces → Spectral Filtering
confidence 92% · in a shared-parameter model, we show that forgetting induces spectral filtering over covariance modes
Shared-Parameter Model → shows → Spectral Filtering
confidence 92% · in a shared-parameter model, we show that forgetting induces spectral filtering over covariance modes
Weight Decay → isa → Regularization
confidence 90% · mechanisms such as weight decay and l2 regularization... are treated as regularization
RPF → yields → MDL-like Compression
confidence 90% · yielding Minimum Description Length (MDL)-like compression.
Catastrophic Interference → isa → Forgetting
confidence 85% · Foundational work identified catastrophic interference as a central limitation... where it is usually treated as a failure mode
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Prior work has shown that neural networks exhibit implicit biases toward low-complexity structure (e.g., spectral bias), memorization dynamics, and compression-like effects during training, but a unified dynamical account of selective retention remains incomplete. We propose Repeated Reinforcement with Persistent Forgetting (RPF) dynamics, a minimal framework in which repeated exposure reinforces patterns and structures that recur in the data, while persistent forgetting attenuates learned information. This view treats forgetting not merely as a failure mode, but as a selection mechanism. We build the theory in three successive layers. First, in an independent-feature model, we derive an exposure-selective survival law and a support-dependent retention boundary characterizing which patterns persist under forgetting. Second, in a shared-parameter model, we show that forgetting induces spectral filtering over covariance modes, preserving strongly supported shared components while suppressing weak ones. Third, under small-step and norm/coding approximations, we show how RPF dynamics induce an implicit trade-off between data fitting and the cost of stored information, yielding Minimum Description Length (MDL)-like compression. Controlled experiments provide evidence for this reinforcement--forgetting selection mechanism in scalar memories and a nonlinear shared network. Joint reinforcement and attenuation interventions shift conditional retention, while matched exposure counts reveal forgetting-dependent effects of reinforcement timing and changes in the composition of the retained set. Together, these results show that repeated reinforcement and persistent forgetting jointly provide a controllable source of inductive bias beyond neural architecture and scale.
Tags
Links
- Source: https://arxiv.org/abs/2609.38768v1
- Canonical: https://arxiv.org/abs/2609.38768v1
Trouble viewing inline? Open PDF directly →
Full Text
76,096 characters extracted from source content.
Expand or collapse full text
Learning Under Forgetting: Statistical Support-Selective Retention in Stochastic Training Dynamics Fujie Gao Zuyue Zhang Gang Sun Abstract Prior work has shown that neural networks exhibit implicit biases toward low-complexity structure (e.g., spectral bias), memorization dynamics, and compression-like effects during training, but a unified dynamical account of selective retention remains incomplete. We propose Repeated Reinforcement with Persistent Forgetting (RPF) dynamics, a minimal framework in which repeated exposure reinforces patterns and structures that recur in the data, while persistent forgetting attenuates learned information. This view treats forgetting not merely as a failure mode, but as a selection mechanism. We build the theory in three successive layers. First, in an independent-feature model, we derive an exposure-selective survival law and a support-dependent retention boundary characterizing which patterns persist under forgetting. Second, in a shared-parameter model, we show that forgetting induces spectral filtering over covariance modes, preserving strongly supported shared components while suppressing weak ones. Third, under small-step and norm/coding approximations, we show how RPF dynamics induce an implicit trade-off between data fitting and the cost of stored information, yielding Minimum Description Length (MDL)-like compression. Controlled experiments provide evidence for this reinforcement–forgetting selection mechanism in scalar memories and a nonlinear shared network. Joint reinforcement and attenuation interventions shift conditional retention, while matched exposure counts reveal forgetting-dependent effects of reinforcement timing and changes in the composition of the retained set. Together, these results show that repeated reinforcement and persistent forgetting jointly provide a controllable source of inductive bias beyond neural architecture and scale. 1 Introduction Training is usually described as a process of accumulating information from data, while mechanisms such as weight decay and ℓ2 _2 regularization (Krogh and Hertz, 1992; Loshchilov and Hutter, 2019), dropout (Srivastava et al., 2014), and stochastic updates (Mandt et al., 2017) are treated as regularization or noisy optimization dynamics. From the perspective of training dynamics, however, these mechanisms can also attenuate, mask, perturb, or overwrite learned representations. This shifts the question from how to prevent forgetting to what forgetting permits to survive. This is a fundamental issue for modern learning systems trained on large, redundant data streams. Such systems are exposed to vast amounts of raw information, yet often retain stable patterns and structures while discarding incidental details. A sharper version arises when the available experience is limited but highly redundant: a system may still acquire reusable patterns and structures without retaining broad factual detail. Data do not support all information equally. Patterns and structures that recur in the data receive repeated reinforcement through many updates, whereas weakly supported details receive inconsistent reinforcement. If learning and forgetting coexist throughout training, which patterns and structures are retained, which are forgotten, and what determines this separation? Prior work has studied forgetting, regularization, memorization, and scaling, but has paid relatively little attention to directly characterizing which patterns survive under simultaneous reinforcement and attenuation. We address this selective-retention problem from the perspective of training dynamics. Throughout this paper, patterns and structures denote reusable units of regularity, such as words or relations in language, edges or textures in images, timbre or recurring acoustic motifs in sound, and shared directions in learned representations. Repeated Reinforcement with Persistent Forgetting (RPF) is not a semantic importance criterion: frequent superficial patterns may survive, while rare meaningful facts may require additional mechanisms. Its central answer is that information is retained when it can be reconstructed often enough to survive forgetting. Our contributions are: • We formulate RPF dynamics to study training-dynamics-induced selective retention under repeated exposure and persistent forgetting; • We derive the survival mechanism from an independent-feature exposure law to shared-parameter spectral filtering, and give an approximate MDL-like interpretation; • We test this mechanism in scalar memories and in a shared nonlinear network, where a reinforcement–forgetting phase diagram and matched occurrence counts reveal conditional and history-dependent selection. 2 Related Work We relate RPF to four lines of work: forgetting, training dynamics, data repetition, and compression. Forgetting has been studied most extensively in continual learning, where it is usually treated as a failure mode to be mitigated. Foundational work identified catastrophic interference as a central limitation of connectionist systems (McCloskey and Cohen, 1989; French, 1999; Goodfellow et al., 2014). Subsequent methods preserve knowledge through explicit constraints, importance estimates, distillation, replay, or memory modules, including EWC and Synaptic Intelligence (Kirkpatrick et al., 2017; Zenke et al., 2017), Learning without Forgetting (Li and Hoiem, 2016), experience replay (Rolnick et al., 2019), GEM and A-GEM (Lopez-Paz and Ranzato, 2017; Chaudhry et al., 2019), iCaRL (Rebuffi et al., 2017), and memory architectures (Weston et al., 2015). Recent work extends these concerns to large-model lifelong learning and stability–plasticity trade-offs (van de Ven et al., 2025; Das Biswas et al., 2026; Lu et al., 2025). These approaches ask how to prevent or compensate for forgetting; RPF asks which repeatedly reinforced patterns and structures survive when forgetting remains active. RPF also relates to learning dynamics and regularization. Gradient descent can converge to particular solutions in underdetermined settings (Soudry et al., 2018; Gunasekar et al., 2018), neural networks exhibit spectral bias (Rahaman et al., 2019), and example-level training dynamics reveal that some examples are repeatedly forgotten while others are never forgotten (Toneva et al., 2019). Grokking shows delayed transitions from memorization to generalization (Power et al., 2022), with later work emphasizing weight decay, data-size dependence, and representation emergence (Liu et al., 2023). Weight decay and dropout are usually viewed as regularizers (Krogh and Hertz, 1992; Loshchilov and Hutter, 2019; Srivastava et al., 2014), and stochastic optimization can be studied as noise-induced dynamics (Mandt et al., 2017). Our focus is not regularization as such, but whether a mechanism implements a forgetting pressure that competes with pattern-specific reinforcement; in our experiments, weight decay provides a direct smooth-attenuation knob, while dropout is treated as a candidate masking-based mechanism whose role must be evaluated empirically. More directly, Yunis et al. (2024) relate weight singular values, effective rank, and alignment to generalization, and show that weight decay strengthens rank bias beyond norm control. Lu (2025) analyze Fourier-frequency suppression in convolutional weights under regularization. RPF uses covariance-supported directions in its shared-linear case and makes pattern-level retention boundaries and reinforcement schedules explicit intervention objects. Balasubramanian and Kasiviswanathan (2026) quantify old-mode mass loss and component drift through divergence direction, behavioral overlap, sampling, and replay in a two-mode post-training model. RPF examines selection across nonuniformly reinforced patterns under persistent attenuation, including how matched occurrence counts acquire different final value through their timing. The contribution is this retention formulation and its testable intervention structure. Selective retention is also an empirical concern in large language models. LLMs can emit verbatim training data (Carlini et al., 2021), and memorization increases with model size, context, and duplicated examples (Carlini et al., 2023; Biderman et al., 2023). Training-dynamics studies show that larger models memorize faster and forget less (Tirumala et al., 2022); data-constrained scaling work shows that repeated data can help up to a point (Muennighoff et al., 2023); long-tail knowledge studies show that factual recall depends strongly on pretraining evidence (Kandpal et al., 2023); and controlled knowledge-storage work shows that data diversity and augmentation affect whether stored knowledge becomes extractable (Allen-Zhu and Li, 2023). These findings motivate a mechanistic question: how does repeated exposure translate into retention under persistent forgetting? Finally, scaling laws characterize aggregate performance as a function of model size, data, and compute (Kaplan et al., 2020; Hoffmann et al., 2022; Henighan et al., 2020; Paquette et al., 2024); related quantization views model scaling as the acquisition of discrete skills or quanta in frequency order (Michaud et al., 2023). Natural data and learned examples are highly nonuniform, often exhibiting Zipf-like, power-law, or long-tailed structure (Zipf, 1949; Newman, 2005; Piantadosi, 2014; Simoncelli and Olshausen, 2001; Zhang et al., 2024), with long-tail memorization playing an important role in neural networks (Feldman and Zhang, 2020). RPF connects these views at the level of retained information rather than only loss. RPF also differs from explicit Information Bottleneck or MDL objectives (Tishby and Zaslavsky, 2015; Alemi et al., 2017; Kawaguchi et al., 2023; Rissanen, 1978; Grunwald, 2007; Hinton and Van Camp, 1993): the compression-like effect arises from the training dynamics, through attenuation of components that are not repeatedly reconstructed. 3 Theoretical Analysis 3.1 Formalization Patterns and structures. Let S denote a space of reusable regularities in the data stream. A member s∈s is a pattern whose occurrences induce compatible learning signals. The boundary of s is operational rather than semantic: it may be a word, phrase, relation, texture, acoustic motif, or a distributed component that can be repeatedly encountered, reinforced, weakened, and reconstructed during training. Persistent forgetting. Persistent forgetting continually reduces the recoverable strength of previously encoded information. In RPF, forgetting is a conceptual pressure on stored patterns and structures; concrete training techniques such as decay, regularization, stochastic perturbation, dropout-style masking, overwriting, or interference may implement this pressure in different ways. Whether a particular technique realizes the desired pattern-level forgetting effect is setting-dependent. Let γt(s)≥0 _t(s)≥ 0 denote the effective forgetting pressure on pattern s at step t. We analyze multiplicative decay with rate d as the tractable smooth-attenuation case, for which a typical forgetting strength satisfies γ≈dγ≈ d up to representation- and optimizer-dependent constants. Exposure and repeated reinforcement. Let p(s)p(s) be the data exposure of s. Repeated traversal increases the number of steps n, while selective replay or sampling changes may alter the sampling exposure q(s)q(s); otherwise q(s)=p(s)q(s)=p(s). Loss reweighting instead changes the compatible update per exposure, represented by R(s)R(s) under a fixed learning-strength scale. For n training steps, let Et(s)E_t(s) be the fraction of examples in step t’s minibatch that expose s (an indicator for a single-example step), and define normalized cumulative exposure Nn(s)=∑t=1nEt(s),[Nn(s)]=nq(s)N_n(s)= _t=1^nE_t(s), [N_n(s)]=nq(s) (1) when minibatch examples are sampled independently from q. Thus repeated exposure creates repeated reinforcement opportunities. Repeated reinforcement is accumulated compatible update pressure, not raw frequency alone. Let ρt(s)≥0 _t(s)≥ 0 denote the compatible reinforcement coefficient at step t, normalized by the remaining recovery gap and measured before subsequent attenuation. Frequent and consistently aligned occurrences make ∑tρt(s) _t _t(s) grow, whereas rare or inconsistent occurrences yield weak accumulated reinforcement. For discrete patterns, q(s)q(s) may be estimated by exposure frequency; for distributed structures, effective support may arise through covariance strength, shared contexts, replay, weighting, or other mechanisms that repeatedly align updates. Survival and recoverability. A pattern survives if it remains usable after repeated weakening and relearning. Let R(s)≥0R(s)≥ 0 denote its local recovery coefficient per compatible exposure, normalized by the learning-strength scale η and the remaining gap to a reference strength. Exposure frequency is accounted for separately by q(s)q(s): R(s)R(s) measures recovery given an exposure, rather than inverse recovery time measured in ordinary training steps. Recoverability depends on the current representation and update protocol. Exposure and this conditional recovery coefficient combine into the effective support λeff(s)=q(s)R(s). _eff(s)=q(s)R(s). (2) Low-support details have small q(s)q(s); unstable structures have small R(s)R(s); both are vulnerable to persistent forgetting. The analytic models below instantiate this support scale in two basis cases: the independent-feature model has R(s)=1R(s)=1 so that support reduces to exposure, while the shared-parameter model replaces isolated exposure by covariance-supported directions. Section 5 specifies the normalization and distinguishes conditional recovery from recovery time in the training stream. 3.2 Repeated Reinforcement with Persistent Forgetting (RPF) RPF couples reinforcement and forgetting in one training dynamics. Let ZtZ_t denote the learned state at step t, abstracting over parameters, representations, or memory traces. RPF writes one coarse-grained training step as an update operator followed by a forgetting operator: Zt+1=ℱt(t(Zt,ℬt)),Z_t+1=F_t (U_t(Z_t;B_t) ), (3) where ℬtB_t is the minibatch or data event at step t. The update tU_t reinforces data-supported structures, while ℱtF_t attenuates learned information. Thus retained structures must be repeatedly reconstructed against persistent attenuation. Under stationary local mean-field bookkeeping, the nominal reinforcement scale is [∑t=1nρt(s)]∝nηλeff(s)=nηq(s)R(s),E\! [ _t=1^n _t(s) ] nη\, _eff(s)=nη\,q(s)R(s), (4) where η is a learning-strength scale. Final retained strength also depends on the survival of these updates through subsequent dynamics. With effective forgetting strength γ, a schematic finite-horizon reconstruction requirement takes the form nηλeff(s)≥Cn(γ,ϵ).nη\, _eff(s)≥ C_n(γ,ε). (5) Here CnC_n depends on the horizon as well as attenuation and the retention criterion; it cannot be held fixed while persistent forgetting accumulates. The scalar case below supplies an exact finite-horizon law. In steady state, the characteristic competition is between the per-step scales, ηλeff(s)≳γ.η\, _eff(s) γ. (6) Thus patterns and structures persist when their effective support is large enough to withstand persistent forgetting. The finite-time condition above concerns cumulative reconstruction opportunities, while the steady-state condition compares per-step reinforcement with per-step attenuation. In the independent-feature model below, λeff _eff reduces to exposure up to the learning scale; in the shared-parameter model, the analogous support quantity is a covariance eigenvalue. The specified retention level determines the boundary constant, and the mapping to effective support is representation-dependent; Appendix B gives additional bookkeeping details. 3.3 Theoretical Results We develop this reinforcement–forgetting selection in three successive layers: survival of independent patterns, retention of shared representation directions, and an information-constrained interpretation of maintained structure. 3.3.1 Independent features: exposure-selective survival We first isolate the mechanism by assigning each pattern an independent scalar strength. Let xst∈0,1x_s^t∈\0,1\ indicate whether pattern s is exposed at step t, let wst∈ℝw_s^t denote its learned strength, and let θs _s be the fully learned target value. We use the multiplicative-forgetting dynamics wst+1=(1−d)[wst+ηxst(θs−wst)],w_s^t+1=(1-d) [w_s^t+η x_s^t( _s-w_s^t) ], (7) where d∈(0,1)d∈(0,1) is the forgetting rate and η>0η>0 is the learning rate. Assumption 1 (Independent-feature RPF). For each s, exposure events are sampled independently over time with xst∼Bernoulli(p(s))x_s^t (p(s)). Updates are compatible with the target θs _s, forgetting follows Eq. (7) with 0<d<10<d<1, 0<ηp(s)<10<η p(s)<1, and |θs|<∞| _s|<∞. Theorem 1 (Exposure-selective survival). Under Assumption 1, [wst]E[w_s^t] converges to ws∗=(1−d)ηp(s)d+(1−d)ηp(s)θs.w_s^*= (1-d)η p(s)d+(1-d)η p(s)\, _s. (8) Consequently there is a characteristic survival scale p∗=Θ(d(1−d)η)p^*= \! ( d(1-d)η ) (9) such that ws∗≈(1−d)ηp(s)dθs,p(s)≪p∗,θs,p(s)≫p∗.w_s^*≈ cases (1-d)η p(s)d _s,&p(s) p^*,\\[6.0pt] _s,&p(s) p^*. cases (10) Thus retention is controlled by the ratio between reinforcement (1−d)ηp(s)(1-d)η p(s) and forgetting d, producing a soft exposure threshold. For θs≠0 _s≠ 0 and 0<ϵ<10<ε<1, the criterion ws∗/θs≥ϵw_s^*/ _s≥ε is equivalent to p(s)≥pϵ∗=ϵ1−ϵd(1−d)η.p(s)≥ p_ε^*= ε1-ε d(1-d)η. (11) This is a selection rule for the retained set at the specified strength level. The theorem characterizes expected strength; finite-run retention remains stochastic. The proof and finite-run variability details are in Appendix A and Appendix B. Heavy-tailed exposure and finite-time scaling. Under heavy-tailed exposure, the independent-feature law yields a retained-count scaling. Assumption 2 (Power-law exposure). Let patterns srs_r obey a power-law exposure profile, p(sr)=Cr−α,α>1,p(s_r)=Cr^-α, α>1, (12) where r=1,2,…r=1,2,… is the decreasing-exposure rank, C>0C>0 is the normalizer, and α is the tail exponent. The exact power-law form is used only to obtain a closed-form retained-count scaling. The RPF survival boundary itself requires only nonuniform effective support with scale separation. For a finite run of length n, define the effective training strength S=nη(1−d).S=nη(1-d). (13) In the accumulation-dominated regime, where nd≪1nd 1 and the discrete learning step is small at the boundary, a fixed criterion on normalized expected strength gives Sp(sr)=Θ(1)Sp(s_r)= (1). Section 5 derives this regime and its crossover to the forgetting-limited equilibrium. Corollary 1 (Finite-time retained-count scaling). Under Assumption 2 in the accumulation-dominated regime with Sp(sr)=Θ(1)Sp(s_r)= (1), the retained rank cutoff k∗k^*, defined by Sp(sk∗)=Θ(1)Sp(s_k^*)= (1), scales as k∗∼S1/αk^* S^1/α (14) up to constants depending on C and ϵε, while the accumulation-dominated approximation holds and before exhausting the pattern support. Thus early training effort expands the expected-strength retained set sublinearly. At fixed positive forgetting, further training approaches a retention ceiling rather than extending this power law indefinitely. Under heavy-tailed exposure, this also means that a relatively small retained set can cover a large fraction of exposure mass. The derivation and retained-mass estimate are in Appendix B. 3.3.2 Shared parameters: spectral filtering We next show that when parameters are shared, the same reinforcement–forgetting balance selects supported directions of variation rather than isolated patterns. This is the key shift from retaining independent pattern coordinates to retaining reusable directions: in shared representations, forgetting no longer asks whether one pattern survives alone, but which directions are rebuilt by many compatible patterns and structures. Consider a linear prediction setting in which a sampled pattern s is represented by a feature vector x=x(s)∈ℝmx=x(s) ^m, with target parameter θ∈ℝmθ ^m and model parameter w∈ℝmw ^m. Let Σ=[xx⊤] =E[x ] (15) be the second-moment matrix of pattern representations (the covariance for centered features). For discrete pattern inputs, Σ=∑sp(s)x(s)x(s)⊤ = _sp(s)x(s)x(s) is exposure-weighted. Its eigenspectrum measures support along shared directions. Assumption 3 (Shared-parameter RPF). The expected update is governed by the second-moment matrix Σ above, forgetting applies the same multiplicative attenuation rate d to all coordinates after each expected update, and 0<ηλi<10<η _i<1 for every nonzero mode. For the linear predictor x⊤wx w with target x⊤θx θ and squared loss, the population gradient is Σ(w−θ) (w-θ). Applying multiplicative attenuation after the gradient update gives the exact expected dynamics wt+1=(1−d)[wt+ηΣ(θ−wt)].w^t+1=(1-d) [w^t+η (θ-w^t) ]. (16) Theorem 2 (Spectral filtering under forgetting). Under Assumption 3, let Σ=∑iλiuiui⊤ = _i _iu_iu_i be the eigendecomposition of this second-moment matrix, where uiu_i are orthonormal shared pattern directions and λi≥0 _i≥ 0 are their support strengths. Under Eq. (16), the equilibrium is w∗=∑if(λi)θiui,θi=θ⊤ui,w^*= _if( _i) _iu_i, _i=θ u_i, (17) where f(λ)=(1−d)ηλd+(1−d)ηλ.f(λ)= (1-d)ηλd+(1-d)ηλ. (18) Thus there is a characteristic spectral threshold λ∗=Θ(d(1−d)η)λ^*= \! ( d(1-d)η ) (19) such that low-support modes satisfy f(λ)=(λ)f(λ)=O(λ) for λ≪λ∗λ λ^*, while high-support modes satisfy f(λ)≈1f(λ)≈ 1 for λ≫λ∗λ λ^*. Diagonalizing Σ reduces each mode to the scalar recursion in Theorem 1 with p(s)p(s) replaced by λi _i; see Appendix A. This lifts exposure-selective survival from isolated features to shared representations: forgetting filters the support spectrum, preserving reusable directions supported by recurring structures and attenuating weak or idiosyncratic modes. 3.3.3 Information-constrained interpretation Third, under smooth multiplicative decay, the same dynamics also admits an information-constrained interpretation. Assume small d and η, and standard Gaussian or coding approximations connecting parameter norm to description length. Under these assumptions, multiplicative forgetting, Wt+1=(1−d)(Wt−η∇ℒt(Wt)).W_t+1=(1-d) (W_t-η _t(W_t) ). (20) to first order in the small-step limit follows gradient flow on an objective of the form ℒ(W)+d2η‖W‖2.L(W)+ d2η\|W\|^2. (21) Parameter norm can then proxy description length or stored information (Hinton and Van Camp, 1993; Rissanen, 1978; Grunwald, 2007). This is an approximate interpretation, not an exact equivalence to Information Bottleneck or MDL objectives (Tishby and Zaslavsky, 2015; Alemi et al., 2017; Kawaguchi et al., 2023). RPF provides a dynamics-based route to an MDL-like effect: information that is not repeatedly reconstructed is continuously leaked, whereas supported and recoverable information remains worth maintaining. The norm/coding assumptions and the corresponding approximation are detailed in Appendix C. 4 Experiments The experiments test how repeated reinforcement and persistent forgetting jointly select what remains learned. We first examine exposure-selective survival in scalar memories, then conditional associations in a shared nonlinear predictor. Finally, matched reinforcement schedules test how the history of reinforcement changes the retained set at identical occurrence counts. The neural task has a uniform output marginal. 4.1 Experiment 1: Independent-Feature Survival This experiment tests the basis-case mechanism in Section 3.3.1: whether nonuniform exposure, opposed by persistent attenuation, is sufficient to convert a heavy-tailed distribution over patterns into selective retention. Setup and results. We simulate N=10,000N=10,000 independent scalar memories with Zipf exposure p(sr)∝r−1.2p(s_r) r^-1.2, target θs=1 _s=1, learning rate η=0.2η=0.2, and batch-wise attenuation w←(1−d)w←(1-d)w with d=0.015d=0.015 every 3232 updates. For scaling runs, we vary the number of updates, retain the plotting coordinate S=nη(1−d)S=nη(1-d), and count a pattern as retained when ws>0.01maxjwjw_s>0.01 _jw_j. Appendix D distinguishes this sampled, relative-threshold count from the theoretical cutoff on expected strength. Figure 1: Independent-feature signatures of RPF. Left: learned strength after 10510^5 updates exhibits the predicted soft support-dependent transition. Right: the number of sampled memories above a relative threshold grows sublinearly and approaches saturation, in qualitative agreement with the finite-time-to-steady-state picture. The expected-strength law k∗∼S1/αk^* S^1/α describes the accumulation-dominated regime. These batch-wise, finite-run observations support the predicted pattern of selective retention and growth toward a forgetting-limited ceiling. In Figure 1, each point in the left panel is one pattern and the smoothed curve averages logarithmic exposure bins. Low-exposure patterns receive too few reconstruction events to compensate for attenuation, whereas high-exposure patterns are repeatedly rebuilt and approach the target. The soft transition and intermediate-support scatter agree with the expected profile in Eq. (8). The right panel shows sublinear expansion and eventual saturation of the sampled retained set. This illustrates the RPF mechanism: reinforcement initially extends retention into the tail, while persistent forgetting sets the long-run support boundary. Section 5 connects these two regimes analytically. 4.2 Experiment 2: Conditional Retention under Reinforcement and Forgetting The independent-feature experiment isolates scalar survival, but real neural models share parameters across many patterns. We next ask whether support-dependent selection also appears for conditional associations in a shared nonlinear network. We construct the task so that the output marginal alone cannot predict the target. Each pattern s=(r,k)s=(r,k) combines one of 6464 Zipf ranks with one of 88 labels, and its target is y(s)=ky(s)=k. Every rank has equal mass for each label, including after focal resampling. Thus P(Y=k)=1/8P(Y=k)=1/8 and H(Y∣S)=0H(Y S)=0: successful prediction requires the input–label relation. The network receives a single identifier for s, without direct access to k. Setup and conditional control. A 24-dimensional embedding feeds a shared →824\!→\!48\!→\!8 GELU head trained with cross-entropy. We designate ranks 3,7,…,633,7,…,63 as focal and cross their resampling weights m∈1,2,4,8m∈\1,2,4,8\ with seven AdamW decay coefficients over five seeds, for 140140 runs. Every run uses 600600 steps, batch size 256256, and learning rate η=0.003η=0.003. The per-step parameter attenuation is dstep=ηλwd∈0,0.003,0.006,0.009,0.012,0.015,0.021.d_ step=η _ wd∈\0,0.003,0.006,0.009,0.012,0.015,0.021\. (22) We measure conditional strength cs=PW(y(s)∣s)c_s=P_W(y(s) s) and usable retention cs≥0.5c_s≥ 0.5. A same-rank control replaces (r,k)(r,k) with (r,k+1mod8)(r,k+1 8) while querying the original label k. Averaged over focal patterns across the full grid, correct-input probability is 0.6940.694, versus 0.0430.043 under this control, giving a conditional gap of 0.6520.652. This gap shows that the retention measure depends on learned input–label associations rather than output marginal fitting. Figure 2: Conditional retention and its reinforcement–forgetting phase diagram. Left: correct-input and same-rank control probabilities, averaged over focal patterns across all 140140 runs; the dashed line is the uniform label marginal. Right: tail focal retention (cs≥0.5c_s≥ 0.5), averaged over five seeds. Resampling moves associations into the retained region near the transition. The horizontal coordinate is the resampling weight, not the normalized exposure multiplier; the 8×8× weight increases each focal pattern’s sampling probability by 3.448×3.448×. Selection boundary and intervention. Figure 2 exhibits a saturated low-attenuation region, a transition where reinforcement changes the retained set, and a high-attenuation region where the tested reinforcement is insufficient for tail retention. At dstep=0.012d_ step=0.012, increasing m from 11 to 22, 44, and 88 raises tail focal retention from 0.0130.013 to 0.1750.175, 0.5250.525, and 0.8190.819. On the continuous focal-probability scale, the 8×−1×8×-1× gain is 0.0140.014 at zero attenuation, 0.1770.177 at 0.0120.012, and 0.0410.041 at 0.0210.021. The intervention is most effective near the retention transition. The observable RPF mapping is concrete: s is a conditional association, qm(s)q_m(s) its normalized sampling probability, csc_s its observable strength, and dstepd_ step the controlled parameter attenuation; per-exposure recoverability R(s)R(s) is not measured here. The same association can cross the retention criterion as its sampling support changes. Thus, in a network with shared parameters, attenuation shifts the support required for a conditional association to remain usable. Appendix D gives all phase cells and sensitivity to thresholds 0.40.4, 0.50.5, and 0.60.6. A factorized synthetic task also shows selective retention through held-out combinations without a complete-pattern embedding (Table 4 and Appendix D). 4.3 Experiment 3: Temporal Coupling and Pattern Selection RPF also predicts that the final value of an occurrence depends on the dynamics that follow it. We therefore hold the complete sample multiset fixed and vary the temporal organization of reinforcement. For each of ten seeds, early, uniform, and late schedules share the same initialization, 600600 optimizer steps, and exact 153,600153,600-sample multiset, including 55,51255,512 additional focal replay samples. The remaining base samples also contain focal occurrences. The schedule changes the placement of the additional replay samples and hence local minibatch composition; every identifier has exactly the same count across schedules. Four attenuation levels yield 120120 runs. Ordinary optimization order can matter even without decay. For mean focal conditional probability Y, we measure the additional attenuation-dependent timing difference by the paired contrast C(dstep)= C(d_ step)= [Ylate(dstep)−Yearly(dstep)] [Y_ late(d_ step)-Y_ early(d_ step) ] (23) −[Ylate(0)−Yearly(0)]. - [Y_ late(0)-Y_ early(0) ]. The zero-attenuation late–early gap is 0.01860.0186. At dstep=0.009d_ step=0.009, 0.0120.012, and 0.0150.015, the additional contrasts are respectively 0.19990.1999, 0.28950.2895, and 0.30090.3009, with paired 95% bootstrap intervals [0.1929,0.2069][0.1929,0.2069], [0.2814,0.2978][0.2814,0.2978], and [0.2934,0.3081][0.2934,0.3081]. Equal occurrence counts therefore produce substantially different final retention, and the size of that difference depends on persistent forgetting. Figure 3: Timing changes the retained value of identical sample multisets. Top: focal probability trajectories with zero attenuation and with dstep=0.012d_ step=0.012; curves show ten-seed means and shading shows standard deviations. Bottom left: the paired timing contrast in Eq. (23), with 95% bootstrap intervals across seeds. Bottom right: at dstep=0.012d_ step=0.012, later focal replay increases focal correct-label probability while decreasing background correct-label probability; points are schedule means and bars are seed standard deviations. The intervention changes the composition of the retained set. The trajectories in Figure 3 show the mechanism at the behavioral level: under attenuation, early focal gains decline during subsequent base training, while late reinforcement remains stronger at evaluation. The effect is selective. At dstep=0.012d_ step=0.012, focal/background probabilities are 0.416/0.5140.416/0.514 for early, 0.642/0.4360.642/0.436 for uniform, and 0.724/0.1400.724/0.140 for late replay. Under the same cs≥0.5c_s≥ 0.5 criterion, mean focal/background retained fractions change from 0.232/0.4650.232/0.465 (early) to 0.995/0.0000.995/0.000 (late) across ten seeds. Rearranging the same training budget changes which associations remain strongly available. These results establish a forgetting-dependent temporal effect in a nonlinear shared network: equal occurrence counts do not guarantee equal final reinforcement or the same retained pattern set. 5 Discussion Our results characterize learning under forgetting as survival under the combined effects of training dynamics and the data distribution. RPF is therefore distinct from denoising or semantic filtering. It assumes no prior separation between signal and noise, and it does not assign value according to whether a stored item is meaningful, true, or useful. The survival criterion is support and recoverability: information remains encoded when it can be reconstructed often enough, and quickly enough, to offset persistent attenuation. High-support superficial patterns may survive, while rare but meaningful facts may decay unless they are replayed, retrieved, curated, or otherwise reinforced. Repeated artifacts or biases can also survive if they receive sufficient support. In this sense, RPF describes a passive selection effect induced by training dynamics, not a criterion for truth, utility, or semantic importance. Here passive selection means that the dynamics determine which patterns lose recoverability without explicitly prescribing which individual patterns must be forgotten. Adjusting a global attenuation hyperparameter changes the selection pressure; the identities of lost patterns emerge from its interaction with reinforcement. Under RPF, abstraction corresponds to repeated reconstruction of shared directions under attenuation. RPF does not define abstraction semantically or impose it architecturally; abstraction appears as the residue of repeated reconstruction under forgetting. In shared parameters, the unit of survival changes from isolated patterns to covariance-supported directions: reusable directions persist, while weak or idiosyncratic directions are attenuated. In the factorized synthetic task, added reinforcement also increases correct-label probability through held-out template combinations without a complete-association embedding (Appendix D). RPF is related to regularization, but it is not merely a restatement of weight decay. The novelty is that attenuation coupled to nonuniform reinforcement converts uniform forgetting pressure into a nonuniform survival boundary. In the regularization view, weight decay is usually a small penalty for generalization; in the RPF view, it is a controllable forgetting pressure that can be increased to raise the support required for retention. Reinforcement value depends on history. In RPF, repeated reinforcement continually reconstructs information while persistent forgetting makes reconstruction necessary for continued retention. The strength and temporal organization of compatible updates determine their effective reinforcing contribution under this forgetting pressure. In the scalar case, write (a)(w)=(1−a)w+aθU^(a)(w)=(1-a)w+aθ and ℱ(d)(w)=(1−d)wF^(d)(w)=(1-d)w. Then ℱ(d)((a)(w))−(a)(ℱ(d)(w))=−adθ.F^(d)(U^(a)(w))-U^(a)(F^(d)(w))=-adθ. (24) For a sequence of exposures, the final state weights each update by the dynamics that follow it (Appendix B). Equal occurrence counts can therefore supply different effective reinforcement under persistent forgetting. The timing result refines repeated reinforcement itself: its effective value is the part of compatible updates that survives subsequent dynamics. Experiment 2 tests this reconstruction–attenuation balance through joint interventions; Experiment 3 tests the temporal organization of reinforcement within the same coupling. The resulting changes in conditional retention and retained-set composition provide controlled empirical evidence for the RPF selection framework. Finite-time expansion and the steady-state ceiling. The retained-count law and the stationary boundary describe different regimes of the same recursion. For ws0=0w_s^0=0, θs≠0 _s≠ 0, and βs=(1−d)ηp(s) _s=(1-d)η p(s), the exact normalized expected strength is rn(s):=[wsn]θs=βsd+βs[1−(1−d−βs)n].r_n(s):= E[w_s^n] _s= _sd+ _s [1-(1-d- _s)^n ]. (25) For nd≪1nd 1 and small learning steps near the boundary, rn(s)≈1−exp[−nβs]r_n(s)≈ 1- [-n _s]. Thus a fixed criterion rn(s)≥ϵr_n(s)≥ε gives pϵ(n)≈−log(1−ϵ)/Sp_ε(n)≈- (1-ε)/S and the accumulation-dominated law k∗∼S1/αk^* S^1/α. At fixed positive d, however, rnr_n approaches βs/(d+βs) _s/(d+ _s), and the boundary approaches pϵ∗p_ε^* in Eq. (11). The limiting retained rank is of order (C/pϵ∗)1/α(C/p_ε^*)^1/α within the available rank range. Further training approaches this forgetting-limited ceiling. Experiment 1 uses batch-wise attenuation and a relative threshold on sampled strengths. Its observed sublinear expansion and saturation visualize this RPF progression: reinforcement initially expands the retained set, while persistent forgetting sets its long-run support boundary. Recoverability conditional on exposure. The factors in q(s)R(s)q(s)R(s) must count exposure only once. At a fixed local state and update protocol, our convention is [Δws,direct∣Et(s)=e,Zt]≈ηeR(s;Zt)(θs−wst),E[ w_s^U,direct E_t(s)=e,Z_t]≈η eR(s;Z_t)( _s-w_s^t), (26) where e is the minibatch exposure fraction, Δws,direct w_s^U,direct is the compatible contribution before the forgetting operator, and θs _s is a specified reference strength. We suppress the state argument in the mean-field notation. Here R measures normalized recovery per exposure; averaging the direct contribution over minibatches gives the scale ηqRη qR. Recoverability reflects the architecture, optimizer, current representation, context diversity, and the extent to which other patterns share the same structure. The independent-feature case has R=1R=1. In contrast, 1/Trecstream1/T_ rec stream, with time measured in ordinary training steps, is a process recovery rate that already includes exposure waiting, intervening attenuation, and shared updates. It cannot be substituted for R and multiplied by q again. A specified probe and local update assay give the corresponding operational definition in a nonlinear network. Empirical support and quantitative questions. The conditional neural experiments provide evidence for the RPF framework: compatible reinforcement and persistent forgetting jointly select which input–label associations remain available under a shared nonlinear head. The phase diagram shows how reinforcement and attenuation move conditional associations across a retention transition; matched schedules show that the same exposure count can select a different retained set. At the quantitative level, the nonlinear phase boundary moves toward higher support as attenuation increases, with an empirical log–log slope of 3.053.05 across 5353 runs with identifiable crossings, compared with the approximately unit slope of the small-decay scalar boundary at fixed learning rate. This is a boundary-versus-attenuation slope, distinct from the count-versus-training-strength exponent 1/α1/α. The fixed ratio ρ=(1−dstep)ηq/dstepρ=(1-d_ step)η q/d_ step also fails to collapse the AdamW curves; its held-out-seed probability RMSE is 0.17670.1767, compared with 0.10720.1072 for a decay-only predictor. These results support the RPF selection picture and identify the quantitative role of nonlinear training dynamics as a further question. The neural measurements concern conditional behavior. Identifying R and local support operators, testing the covariance-mode law, and evaluating generalization across broader compositions and real language data are the next empirical steps. The paired timing contrast measures the schedule-by-attenuation interaction, including effects mediated by adaptive optimization and shared-gradient interference: the replay blocks change the local minibatch distribution even though occurrence counts match. Separating the contributions of these pathways requires additional interventions. AdamW controls parameter attenuation; its effect on conditional probability is mediated by the network and optimizer. The heavy-tailed analysis should also be read mechanistically. The exact Zipf or power-law form is not required for RPF itself; it only yields a closed-form retained-count scaling. The more general requirement is scale separation in effective support, which may arise from pattern exposure, phrase and relation reuse, visual or acoustic motifs, contextual diversity, replay, weighting, or covariance-supported representation directions. Under heavy-tailed exposure, a small retained type set can still cover a large fraction of exposure mass. Implications for controlling retention. RPF treats attenuation as a controllable forgetting pressure and replay as a way to allocate reinforcement across patterns. Additional training can expand the retained set until the forgetting-limited ceiling; sampling and weighting can increase support for selected patterns, while stronger attenuation raises the support needed to retain them. The phase diagram identifies where added support has the greatest effect, and the timing experiment shows that the final value of a fixed reinforcement budget depends on when it is spent. Concentrated late reinforcement can preserve selected associations at the expense of background retention, so the retained set must be evaluated alongside focal gains. Norm/coding approximations further motivate an MDL-like interpretation: repeated reinforcement maintains supported information despite the cost of storing it, while insufficiently reconstructed components decay. The central result remains the dynamically selected retained set. 6 Conclusion RPF formulates learning as repeated reconstruction under persistent forgetting. The independent-feature model gives an exposure-dependent survival boundary, while the shared-linear model shows how this selection preserves covariance-supported directions rebuilt by recurring structures. Finite-time dynamics connect initial retained-set expansion to a forgetting-limited ceiling. Conditional neural experiments establish a reinforcement–forgetting phase diagram and show that identical occurrence counts produce different retention when the temporal organization of reinforcement changes. The same intervention changes which patterns remain available, providing empirical support for the framework. RPF therefore identifies a concrete selection principle: attenuation sets the support required for retention, while the amount and timing of reinforcement determine which patterns cross that boundary. References Kirkpatrick et al. (2017) James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Simonet, Blake A. O’Donoghue, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521–3526, 2017. van de Ven et al. (2025) Gido M. van de Ven, Nicholas Soures, and Dhireesha Kudithipudi. Continual learning and catastrophic forgetting. In Learning and Memory: A Comprehensive Reference, 3rd edition, vol. 1, p. 153–168. Academic Press, 2025. arXiv:2403.05175. Kaplan et al. (2020) J. Kaplan, S. McCandlish, T. Henighan, et al. Scaling laws for neural language models. arXiv:2001.08361, 2020. Hoffmann et al. (2022) J. Hoffmann, S. Borgeaud, A. Mensch, et al. Training compute-optimal large language models. arXiv:2203.15556, 2022. Srivastava et al. (2014) N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, R. Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. J. Machine Learning Research, 15(1):1929–1958, 2014. Krogh and Hertz (1992) Anders Krogh and John A. Hertz. A simple weight decay can improve generalization. In Advances in Neural Information Processing Systems, 1992. Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. Mandt et al. (2017) Stephan Mandt, Matthew D. Hoffman, and David M. Blei. Stochastic gradient descent as approximate Bayesian inference. Journal of Machine Learning Research, 18:1–35, 2017. Rahaman et al. (2019) N. Rahaman, A. Baratin, D. Arpit, et al. On the spectral bias of neural networks. ICML, 2019. Das Biswas et al. (2026) Shristi Das Biswas, Yue Zhang, Anwesan Pal, Radhika Bhargava, and Kaushik Roy. ELLA: Efficient lifelong learning for adapters in large language models. arXiv:2601.02232, 2026. Zenke et al. (2017) F. Zenke, B. Poole, S. Ganguli. Continual learning through synaptic intelligence. ICML, 2017. Weston et al. (2015) J. Weston, S. Chopra, A. Bordes. Memory networks. ICLR, 2015. Lu et al. (2025) Aojun Lu, Hangjie Yuan, Tao Feng, and Yanan Sun. Rethinking the stability-plasticity trade-off in continual learning from an architectural perspective. arXiv:2506.03951, 2025. Simoncelli and Olshausen (2001) E. P. Simoncelli, B. A. Olshausen. Natural image statistics and neural representation. Annu. Rev. Neurosci., 24:1193–1216, 2001. Zipf (1949) George Kingsley Zipf. Human Behavior and the Principle of Least Effort. Addison-Wesley, 1949. Newman (2005) M. E. J. Newman. Power laws, Pareto distributions and Zipf’s law. Contemporary Physics, 46(5):323–351, 2005. Piantadosi (2014) Steven T. Piantadosi. Zipf’s word frequency law in natural language: A critical review and future directions. Psychonomic Bulletin & Review, 21:1112–1130, 2014. Zhang et al. (2024) Chongsheng Zhang, George Almpanidis, Gaojuan Fan, Binquan Deng, Yanbo Zhang, Ji Liu, Aouaidjia Kamel, Paolo Soda, and João Gama. A systematic review on long-tailed learning. arXiv:2408.00483, 2024. Tishby and Zaslavsky (2015) N. Tishby, N. Zaslavsky. Deep learning and the information bottleneck principle. IEEE Information Theory Workshop, 2015. Alemi et al. (2017) Alexander A. Alemi, Ian Fischer, Joshua V. Dillon, and Kevin Murphy. Deep variational information bottleneck. In International Conference on Learning Representations, 2017. Kawaguchi et al. (2023) Kenji Kawaguchi, Zhun Deng, Xu Ji, and Jiaoyang Huang. How does information bottleneck help deep learning? In International Conference on Machine Learning, 2023. Rissanen (1978) Jorma Rissanen. Modeling by shortest data description. Automatica, 14(5):465–471, 1978. Grunwald (2007) Peter D. Grunwald. The minimum description length principle. MIT Press, 2007. Hinton and Van Camp (1993) Geoffrey E. Hinton and Drew Van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In Proceedings of the Conference on Computational Learning Theory, 1993. McCloskey and Cohen (1989) Michael McCloskey and Neal J. Cohen. Catastrophic interference in connectionist networks: the sequential learning problem. In Psychology of Learning and Motivation, 1989. Rolnick et al. (2019) D. Rolnick, A. Ahuja, J. Schwarz, T. Lillicrap, G. Wayne. Experience replay for continual learning. NeurIPS, 2019. Toneva et al. (2019) Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J. Gordon. An empirical study of example forgetting during deep neural network learning. In International Conference on Learning Representations, 2019. Power et al. (2022) Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets. arXiv:2201.02177, 2022. Liu et al. (2023) Ziming Liu, Eric J. Michaud, and Max Tegmark. Omnigrok: Grokking beyond algorithmic data. In International Conference on Learning Representations, 2023. arXiv:2210.01117. Tirumala et al. (2022) Kushal Tirumala, Aram H. Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. Memorization without overfitting: Analyzing the training dynamics of large language models. arXiv:2205.10770, 2022. Carlini et al. (2021) Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large language models. In USENIX Security Symposium, 2021. Carlini et al. (2023) Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. In International Conference on Learning Representations, 2023. Biderman et al. (2023) Stella Biderman, USVSN Sai Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivanshu Purohit, and Edward Raff. Emergent and predictable memorization in large language models. In Advances in Neural Information Processing Systems, 2023. Muennighoff et al. (2023) Niklas Muennighoff, Alexander M. Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scaling data-constrained language models. arXiv:2305.16264, 2023. Kandpal et al. (2023) Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. Large language models struggle to learn long-tail knowledge. In International Conference on Machine Learning, 2023. Allen-Zhu and Li (2023) Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.1, knowledge storage and extraction. arXiv:2309.14316, 2023. Feldman and Zhang (2020) Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. In Advances in Neural Information Processing Systems, 2020. Goodfellow et al. (2014) I. Goodfellow, M. Mirza, D. Xiao, A. Courville, Y. Bengio. An empirical investigation of catastrophic forgetting in gradient-based neural networks. ICLR, 2014. Li and Hoiem (2016) Zhizhong Li and Derek Hoiem. Learning without forgetting. In European Conference on Computer Vision, 2016. French (1999) R. M. French. Catastrophic forgetting in connectionist networks. Trends in Cognitive Sciences, 3(4):128–135, 1999. Lopez-Paz and Ranzato (2017) David Lopez-Paz and Marc’Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, 2017. Chaudhry et al. (2019) Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. In International Conference on Learning Representations, 2019. Rebuffi et al. (2017) Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. iCaRL: Incremental classifier and representation learning. In Conference on Computer Vision and Pattern Recognition, 2017. Soudry et al. (2018) Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Shai Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. Journal of Machine Learning Research, 2018. Henighan et al. (2020) Tom Henighan, Jared Kaplan, M. Bowman, et al. Scaling laws for autoregressive generative modeling. arXiv:2010.14701, 2020. Paquette et al. (2024) Elliot Paquette, Courtney Paquette, Lechao Xiao, and Jeffrey Pennington. 4+3 phases of compute-optimal neural scaling laws. arXiv:2405.15074, 2024. Michaud et al. (2023) Eric J. Michaud, Ziming Liu, Uzay Girit, and Max Tegmark. The quantization model of neural scaling. In Advances in Neural Information Processing Systems, 36, 2023. arXiv:2303.13506. Gunasekar et al. (2018) Shai Gunasekar, Jason D. Lee, Daniel Soudry, and Nathan Srebro. Implicit bias of gradient descent on linear convolutional networks. In Advances in Neural Information Processing Systems, 2018. Yunis et al. (2024) David Yunis, Kumar Kshitij Patel, Samuel Wheeler, Pedro Savarese, Gal Vardi, Karen Livescu, Michael Maire, and Matthew R. Walter. Approaching deep learning through the spectral dynamics of weights. arXiv:2408.11804, 2024. https://arxiv.org/abs/2408.11804. Lu (2025) Jiahao Lu. Frequency regularization: Unveiling the spectral inductive bias of deep neural networks. arXiv:2512.22192, 2025. https://arxiv.org/abs/2512.22192. Balasubramanian and Kasiviswanathan (2026) Krishnakumar Balasubramanian and Shiva Prasad Kasiviswanathan. A quantitative characterization of forgetting in post-training. arXiv:2603.12163, 2026. https://arxiv.org/abs/2603.12163. Appendix A Proofs Proof of Theorem 1. Taking conditional expectation in Eq. (7) gives [wst+1∣wst] [w_s^t+1 w_s^t] =(1−d)[wst+ηp(s)(θs−wst)] =(1-d) [w_s^t+η p(s)( _s-w_s^t) ] (27) =(1−d)(1−ηp(s))wst+(1−d)ηp(s)θs. =(1-d)(1-η p(s))w_s^t+(1-d)η p(s) _s. (28) Let μt=[wst] _t=E[w_s^t]. Then μt+1=asμt+bs,as=(1−d)(1−ηp(s)),bs=(1−d)ηp(s)θs. _t+1=a_s _t+b_s, a_s=(1-d)(1-η p(s)), b_s=(1-d)η p(s) _s. (29) Since 0<as<10<a_s<1, the recursion converges to μ∗=bs/(1−as)μ^*=b_s/(1-a_s), yielding Eq. (8). The threshold follows by comparing (1−d)ηp(s)(1-d)η p(s) with d. Proof of Theorem 2. Project Eq. (16) onto an eigenvector uiu_i and write wit=(wt)⊤uiw_i^t=(w^t) u_i. Since Σui=λiui u_i= _iu_i, wit+1=(1−d)[(1−ηλi)wit+ηλiθi].w_i^t+1=(1-d) [(1-η _i)w_i^t+η _i _i ]. (30) This is the same scalar recursion as in Theorem 1 with p(s)p(s) replaced by λi _i. Solving the fixed point yields Eq. (18). Appendix B Additional Theoretical Details Pattern scope and forgetting operators. The pattern unit s used in Section 3 is operational: its occurrences induce compatible learning signals. Discrete identifiers and scalar features are controllable basis cases; covariance-supported directions provide the corresponding object for the shared-linear analysis. The operator formulation in Eq. (3) accommodates different implementations of persistent forgetting. An independent Bernoulli erasure of stored state with probability d has conditional mean attenuation 1−d1-d, so multiplicative decay gives a tractable mean-field instance of this broader formulation. Each implementation determines its finite-run variability and its mapping from stored state to behavioral forgetting. Exact finite-time solution and retained-count regimes. For zero initialization and θs≠0 _s≠ 0, the expected scalar recurrence is rt+1=asrt+βs,as=1−d−βs,βs=(1−d)ηp(s).r_t+1=a_sr_t+ _s, a_s=1-d- _s, _s=(1-d)η p(s). (31) Summing the geometric series gives Eq. (25). Under the stated assumption 0<ηp(s)<10<η p(s)<1, 0<as<10<a_s<1, so expected normalized strength increases toward r∞=βs/(d+βs)r_∞= _s/(d+ _s). The finite-horizon support boundary is defined implicitly by rn(pϵ(n))=ϵr_n(p_ε(n))=ε wherever that level is attainable within the available support range. It approaches the steady-state boundary in Eq. (11). In the accumulation-dominated regime, nd≪1nd 1 and a small learning step at the boundary give rn(p)≈1−e−n(1−d)ηp,pϵ(n)≈−log(1−ϵ)n(1−d)η.r_n(p)≈ 1-e^-n(1-d)η p, p_ε(n)≈ - (1-ε)n(1-d)η. (32) For p(sr)=Cr−αp(s_r)=Cr^-α, the expected-strength cutoff is consequently k∗(n)≈(CS−log(1−ϵ))1/α,S=n(1−d)η,k^*(n)≈ ( CS- (1-ε) )^1/α, S=n(1-d)η, (33) while these conditions hold and the cutoff lies within the ranked support. At fixed d>0d>0, the limiting rank boundary is instead (C/pϵ∗)1/α(C/p_ε^*)^1/α, clipped to the available ranks. If even the largest available exposure falls below pϵ∗p_ε^*, no pattern meets the stationary criterion; a pattern exactly at the boundary approaches the criterion asymptotically. This distinction separates finite-time growth from equilibrium selection. The retained exposure mass satisfies ∑r=1k∗p(sr)=1−((k∗)1−α). _r=1^k^*p(s_r)=1-O ((k^*)^1-α ). (34) For a sufficiently large power-law support, substituting the early-regime cutoff yields 1−(S−(α−1)/α)1-O(S^-(α-1)/α) in that regime. At fixed positive forgetting, the retained mass also approaches a ceiling determined by the stationary retained set. Batch-wise attenuation. Experiment 1 applies forgetting every B=32B=32 independent exposure events. Let mjm_j be expected strength immediately after the jjth attenuation. With hs=(1−ηp(s))Bh_s=(1-η p(s))^B, its exact cycle-level recurrence is mj+1=(1−d)[hsmj+(1−hs)θs].m_j+1=(1-d) [h_sm_j+(1-h_s) _s ]. (35) Hence, for zero initialization, mjθs=(1−d)(1−hs)1−(1−d)hs[1−((1−d)hs)j]. m_j _s= (1-d)(1-h_s)1-(1-d)h_s [1- ((1-d)h_s )^j ]. (36) For a final partial cycle of ℓ<B <B updates, apply the unattenuated expected update ℓ times to mjm_j. Matching attenuation alone gives the equivalent per-update rate deff=1−(1−d)1/B≈d/Bd_ eff=1-(1-d)^1/B≈ d/B for small d; the full cycle-level dynamics also include the intervening reinforcement through hsh_s. The plotted relative threshold counts realized sample strengths relative to their maximum, while Eqs. (25) and (36) characterize the expected-strength profile. The observed soft transition, sublinear expansion, and saturation provide qualitative support for this reinforcement–forgetting picture. The scatter in the independent-feature simulation also has a reconstruction-based interpretation. Low-support patterns receive few updates, while highly supported patterns approach saturation and have small recovery gaps. Around the observed transition, intermittent exposures combine with appreciable recovery gaps, producing visible variation in learned strength. This explains the intermediate-support scatter in Figure 1 at the level of the simulated mechanism. Reinforcement weighted by subsequent dynamics. Let It∈0,1I_t∈\0,1\ be the exposure indicator for one scalar pattern, with time indexed by t=0,…,n−1t=0,…,n-1. Set At=(1−d)(1−ηIt),ct=(1−d)ηItθ.A_t=(1-d)(1-η I_t), c_t=(1-d)η I_tθ. (37) The realized update is wt+1=Atwt+ctw_t+1=A_tw_t+c_t. Repeated substitution gives the exact identity wn=(∏u=0n−1Au)w0+∑t=0n−1ct∏u=t+1n−1Au.w_n= ( _u=0^n-1A_u )w_0+ _t=0^n-1c_t _u=t+1^n-1A_u. (38) Empty products equal one. For 0<d<10<d<1 and 0<η≤10<η≤ 1, each At∈[0,1)A_t∈[0,1) is a contraction, and the final contribution of an exposure explicitly includes all subsequent contractions. Define ρ~n=∑t=0n−1It∏u=t+1n−1Au. ρ_n= _t=0^n-1I_t _u=t+1^n-1A_u. (39) Then the accumulated input contribution is (1−d)ηθρ~n(1-d)ηθ ρ_n. If subsequent reinforcement contractions are negligible, the weights reduce approximately to (1−d)n−1−t(1-d)^n-1-t. The exact expression also accounts for contraction from later reinforcement. Equal ∑tIt _tI_t therefore need not imply equal wnw_n under forgetting. With zero decay and a single compatible scalar target, the complete scalar result depends only on the number of exposures; nonlinear shared optimization can retain order effects even at zero decay, motivating the experimental baseline subtraction. Equation (24) follows by evaluating the two affine operators: reinforcement followed by forgetting has constant term (1−d)aθ(1-d)aθ, whereas forgetting followed by reinforcement has constant term aθaθ. Their difference is −adθ-adθ. These are consequences of the analyzed scalar dynamics and connect its stationary boundary to the finite-time timing prediction. Shared linear dynamics and local nonlinear interpretation. For a possibly time-dependent affine shared update wt+1=Atwt+ctw_t+1=A_tw_t+c_t, define Φ(n,t+1)=An−1⋯At+1 (n,t+1)=A_n-1·s A_t+1 and Φ(n,0)=An−1⋯A0 (n,0)=A_n-1·s A_0. The exact expansion is wn=Φ(n,0)w0+∑t=0n−1Φ(n,t+1)ct.w_n= (n,0)w_0+ _t=0^n-1 (n,t+1)c_t. (40) For the squared-loss shared-linear case, At=(1−d)(I−ηΣt)A_t=(1-d)(I-η _t). A linear pattern readout vs⊤wnv_s w_n assigns update ctc_t the final contribution vs⊤Φ(n,t+1)ctv_s (n,t+1)c_t, which depends on alignment as well as subsequent attenuation and shared updates. A local quadratic or Gauss–Newton approximation motivates analogous operators for a nonlinear network, with the reference-point terms included in ctc_t. Estimating those operators is a further identification task. The neural experiments test the observable conditional-selection and timing predictions without assuming a fixed nonlinear covariance spectrum. Appendix C Information-Constrained Interpretation Expanding Eq. (20) gives Wt+1−Wt=−η∇ℒt(Wt)−dWt+dη∇ℒt(Wt).W_t+1-W_t=-η _t(W_t)-dW_t+dη _t(W_t). (41) For small d and η, the last term is higher order. When the stochastic gradient represents the gradient of ℒL, the remaining terms are an Euler step of size η for gradient flow on ℒ(W)+d2η‖W‖2L(W)+ d2η\|W\|^2. Under a fixed Gaussian coding approximation, the quadratic norm term can proxy a cost of storing parameter information. Repeated, loss-reducing updates must offset that cost for a component to persist; unsupported components decay. This is the dynamics-based fitting–storage trade-off behind the MDL-like interpretation, not an exact equivalence to IB or MDL objectives. Appendix D Experimental Details Procedure for Experiment 1. The independent-feature simulation constructs a normalized Zipf exposure vector over N=10,000N=10,000 patterns, p(sr)=r−1.2∑j=1Nj−1.2.p(s_r)= r^-1.2 _j=1^Nj^-1.2. (42) All memories start at ws=0w_s=0 with targets θs=1 _s=1. At each update, sample s∼ps p and reinforce the sampled coordinate by ws←ws+η(1−ws)w_s← w_s+η(1-w_s) with η=0.2η=0.2. Every 3232 updates, attenuate all memories by w←(1−0.015)w←(1-0.015)w. The retained count uses ws>0.01maxjwjw_s>0.01 _jw_j. The original scaling plot sweeps 3030 logarithmically spaced update counts from 10210^2 to 10510^5. Its horizontal coordinate S=nη(1−d)S=nη(1-d) retains the original batch-level d as a plotting convention; the cycle-level expectation is Eq. (36). At the largest plotted horizon, cumulative forgetting is substantial, so the full curve cannot be treated as a test confined to the accumulation-dominated regime. Conditional task and normalized support in Experiment 2. For r∈1,…,64r∈\1,…,64\ and k∈0,…,7k∈\0,…,7\, define s=(r,k)s=(r,k) and y(s)=ky(s)=k. A single integer identifier encodes s. Its base support is p(r,k)=r−1.158∑j=164j−1.15.p(r,k)= r^-1.158 _j=1^64j^-1.15. (43) The focal ranks are 3,7,…,633,7,…,63, giving 128128 focal and 384384 background patterns. The tail focal set contains 3232 patterns at ranks 51,55,59,6351,55,59,63, within the bottom quarter of the rank range. Let ℱF denote the focal set and Pℱ=∑s∈ℱp(s)=0.1886452892P_F= _s p(s)=0.1886452892. With focal resampling weight m, the sampling distribution is qm(s)=p(s)ms∈ℱ1+(m−1)Pℱ.q_m(s)= p(s)m^1\s \1+(m-1)P_F. (44) For focal patterns, the true exposure multipliers for m=1,2,4,8m=1,2,4,8 are 11, 1.68261.6826, 2.55442.5544, and 3.44753.4475. Background sampling probability decreases through the same normalization. Because each rank contains every label with equal mass, both p and qmq_m have exactly uniform label marginals. Finite sampled label counts fluctuate around this distribution. The same-rank control changes the input label identity while preserving its sampling support, and queries the original label. Network and optimization. The model consists of a 512×24512× 24 embedding table and a shared linear layer 24→4824→ 48, GELU activation, and linear layer 48→848→ 8, with biases. It has 13,88013,880 trainable parameters and uses no dropout. All parameters are passed to AdamW with learning rate 0.0030.003, default moment coefficients (0.9,0.999)(0.9,0.999) and numerical constant 10−810^-8. Each run minimizes mean minibatch cross-entropy for 600600 steps of batch size 256256. The grid is λwd∈0,1,2,3,4,5,7 _ wd∈\0,1,2,3,4,5,7\, m∈1,2,4,8m∈\1,2,4,8\, and seeds 00–44. Seeds match initialization across conditions; sampling generators depend on seed, resampling weight, and decay coefficient, so the phase-grid sample sequences are not matched. Experiment 3 supplies the exactly matched sequence intervention. For AdamW, the parameter update has the form Wt+1=(1−dstep)Wt−ηgtAdam,W_t+1=(1-d_ step)W_t-η g_t Adam, (45) where gtAdamg_t Adam is the adaptive update direction. This controls parameter attenuation directly. Holding the update direction fixed, post-update attenuation in Eq. (20) would also scale the update by 1−dstep1-d_ step; the one-step difference is dstepηgtAdamd_ stepη g_t Adam. Thus the scalar operator identity is exact in its basis case, while the neural timing experiment tests the corresponding selection effect under AdamW without imposing the scalar coefficients on conditional probabilities. Metrics and full phase grid. All probabilities are evaluated with the model in evaluation mode. Primary strength is cs=PW(y(s)∣s)c_s=P_W(y(s) s), and usable retention is cs≥0.51\c_s≥ 0.5\. Group metrics are unweighted means across identifiers within the focal, tail focal, or background group, followed by means across seeds. Table 1 gives every tail phase cell, with sample standard deviations across the five seeds. The pooled correct-input and control means in Figure 2 are over 128×140=17,920128× 140=17,920 focal observations. Table 1: Complete tail focal retention grid, mean ± seed standard deviation. The retention criterion is cs≥0.5c_s≥ 0.5; columns are focal resampling weights. dstepd_ step 1×1× 2×2× 4×4× 8×8× 0.000 1.000±0.0001.000± 0.000 1.000±0.0001.000± 0.000 1.000±0.0001.000± 0.000 1.000±0.0001.000± 0.000 0.003 1.000±0.0001.000± 0.000 1.000±0.0001.000± 0.000 1.000±0.0001.000± 0.000 1.000±0.0001.000± 0.000 0.006 0.981±0.0280.981± 0.028 1.000±0.0001.000± 0.000 1.000±0.0001.000± 0.000 1.000±0.0001.000± 0.000 0.009 0.550±0.0840.550± 0.084 0.925±0.0360.925± 0.036 0.994±0.0140.994± 0.014 1.000±0.0001.000± 0.000 0.012 0.013±0.0170.013± 0.017 0.175±0.0520.175± 0.052 0.525±0.1240.525± 0.124 0.819±0.0560.819± 0.056 0.015 0.000±0.0000.000± 0.000 0.000±0.0000.000± 0.000 0.000±0.0000.000± 0.000 0.013±0.0280.013± 0.028 0.021 0.000±0.0000.000± 0.000 0.000±0.0000.000± 0.000 0.000±0.0000.000± 0.000 0.000±0.0000.000± 0.000 Table 2: Threshold sensitivity at dstep=0.012d_ step=0.012. Tail focal retention means across five seeds. The intervention direction persists across all three criteria. Criterion 1×1× 2×2× 4×4× 8×8× 0.4 0.200 0.706 0.969 1.000 0.5 0.013 0.175 0.525 0.819 0.6 0.000 0.013 0.069 0.125 Table 2 checks criteria 0.40.4, 0.50.5, and 0.60.6. The continuous m=8m=8 minus m=1m=1 focal probability gain provides a threshold-free counterpart: 0.0140.014 at zero attenuation, 0.1770.177 at 0.0120.012, and 0.0410.041 at 0.0210.021. Empirical boundary and ratio analysis. For each run containing both retained and non-retained focal identifiers, we fit a logistic classifier of retention on logqm(s) q_m(s), with intercept, inverse regularization strength 10410^4, and maximum 2,0002,000 iterations. Its 0.5 fitted-probability crossing defines p50p_50. A log–log regression across the 5353 eligible runs gives slope 3.0473.047 and R2=0.871R^2=0.871 for boundary versus decay coefficient; using dstepd_ step changes the intercept but not the slope because learning rate is fixed. Runs with no crossing are not assigned a boundary. This slope describes the empirical transition under nonlinear AdamW training and does not estimate the retained-count exponent 1/α1/α. A post-hoc predictor comparison used leave-one-seed-out evaluation of continuous focal probabilities at the six nonzero-decay levels, holding out all observations of each seed together. All five models use the same observations. Cubic spline features with six knots and a ridge penalty of 10−310^-3 describe log support, log attenuation, their additive or interaction combinations, or the fixed log-ratio coordinate. The mean held-out RMSEs are 0.25910.2591 for support alone, 0.10720.1072 for decay alone, 0.17670.1767 for the fixed ratio coordinate, 0.06120.0612 for a flexible two-axis additive model, and 0.04050.0405 for a flexible two-axis interaction model. These exploratory fits diagnose limits to quantitative transfer rather than define coupling: statistical interaction depends on the response scale, whereas Experiment 3 intervenes on the timing of matched reinforcement events. Matched schedules in Experiment 3. The network, learning rate, batch size, and number of steps match Experiment 2. We use λwd∈0,3,4,5 _ wd∈\0,3,4,5\, schedules early/uniform/late, and seeds 00–99. For each seed, a base pool is sampled from p and an extra replay pool from p(s∣s∈ℱ)p(s s ). For nominal weight m=4m=4, mixing fraction μ=(m−1)Pℱ1+(m−1)Pℱ=0.3614042435μ= (m-1)P_F1+(m-1)P_F=0.3614042435 (46) reproduces q4q_4 in expectation. Rounding μ×153,600μ× 153,600 gives 55,51255,512 additional replay samples and 98,08898,088 base samples. The base pool itself includes focal patterns. We place the replay pool first, approximately evenly interleaved, or last, while preserving the internal order of each pool. Exact per-identifier counts are identical across schedules for every seed, and all decay levels reuse these same sequences. Each run resets the model and optimizer to the same seed-specific initialization and fresh optimizer state. The design was chosen using the completed phase grid; the schedule experiment then used the fixed configuration for all ten seeds. Paired inference and selection. For each seed and decay, subtract that seed’s zero-decay late–early gap from its corresponding nonzero-decay gap. Confidence intervals resample the ten paired seed contrasts with replacement 5,0005,000 times and use the 2.5th and 97.5th percentiles of the bootstrap means. The bootstrap generator seed is 2026072920260729. Seeds, rather than individual identifiers, are the independent resampling unit. Table 3 gives all primary contrasts. Trajectories are recorded at initialization and every 5050 optimizer steps. The focal/background tradeoff in Figure 3 uses final group means at dstep=0.012d_ step=0.012. Table 3: Matched schedule effects on mean focal conditional probability. The contrast subtracts each seed’s zero-attenuation late–early gap. Intervals are paired-seed bootstrap intervals. dstepd_ step Late–early C(dstep)C(d_ step) 95% interval 0.000 0.0186 0.0000 — 0.009 0.2185 0.1999 [0.1929, 0.2069][0.1929,\,0.2069] 0.012 0.3081 0.2895 [0.2814, 0.2978][0.2814,\,0.2978] 0.015 0.3195 0.3009 [0.2934, 0.3081][0.2934,\,0.3081] Factorized held-out combinations. In a separate synthetic task, each input combines one of 6464 Zipf-ranked entities, one of 88 relations, and one of 44 shared templates. Its target is y=(rmod8+relation)mod8y=(r 8+relation) 8, invariant across templates. Each entity–relation association is trained through three templates and evaluated through the fourth: all factors appear in training, but none of the 512512 held-out triples does. Separate entity, relation, and template embeddings feed a shared GELU network, with no complete-association embedding. Training and held-out label marginals are uniform. We cross resampling weights m∈1,4m∈\1,4\ with AdamW decay coefficients 0,3,4,5\0,3,4,5\ over ten seeds, giving 8080 runs. Within each seed and resampling condition, the initialization and sampled sequence are matched across decay levels. Table 4 reports paired gains in focal correct-label probability on held-out combinations. The gain peaks near the decay-33 transition at 0.3740.374; the corresponding held-out background gain is 0.0290.029 with interval [−0.009,0.070][-0.009,0.070]. This extends the conditional selection observation to associations reconstructed from shared factors through unseen entity–relation–template combinations; each template itself appears in training. Table 4: Factorized synthetic task: paired gain in focal correct-label probability on held-out combinations for resampling weight m=4m=4 versus m=1m=1 (ten seeds; paired 95% bootstrap intervals). AdamW decay coefficient Focal gain 95% interval 00 0.1650.165 [0.136,0.199][0.136,0.199] 33 0.3740.374 [0.267,0.479][0.267,0.479] 44 0.1580.158 [0.094,0.222][0.094,0.222] 55 0.0550.055 [0.052,0.058][0.052,0.058] Compute and reproducibility. The recorded neural runs used a CPU on macOS 14.5 (arm64), Python 3.9.18, PyTorch 2.3.0, NumPy 2.0.2, pandas 2.2.3, scikit-learn 1.6.1, and Matplotlib 3.9.4, with four PyTorch threads. The 140140 phase runs and 120120 schedule runs comprise 156,000156,000 optimizer steps and 39,936,00039,936,000 sampled training presentations; the factorized study adds 8080 runs of 600600 steps each. The phase and timing figures and tables are regenerated from saved run-level and identifier-level outputs, and Table 4 is checked against the saved paired summary. The revision checks matched counts, reported contrasts, threshold summaries, and the exact recursion identities. These checks verify consistency of the saved evidence and analytical expressions; they do not constitute a new execution of the training runs.