Paper deep dive
Leakage and Second-Order Dynamics Improve Hippocampal RNN Replay
Josue Casco-Rodriguez, Nanda H. Krishna, Richard G. Baraniuk
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 89%
Last extracted: 7/20/2026, 9:51:57 PM
Summary
This paper investigates how specific modifications to noisy Recurrent Neural Networks (RNNs) improve the quality of hippocampal replay, modeled as Langevin sampling. The authors demonstrate that hidden state leakage is necessary for accurate path-integration due to time-varying score functions. They further show that while neural adaptation (negative feedback) encourages exploration, it slows down replay by inducing non-Markovian dynamics. Finally, they propose hidden state momentum (underdamped Langevin sampling) to accelerate replay and counteract the slowing effect of adaptation, achieving temporally compressed replay that mimics biological observations.
Entities (10)
Relation Signals (7)
Neural Adaptation → encourages → Exploration
confidence 92% · hidden state adaptation (negative feedback) encourages exploration in replay
Noisy RNN → performs → Langevin Sampling
confidence 90% · Replay in these networks has been described as Langevin sampling
Hidden State Momentum → accelerates → Replay
confidence 89% · underdamped sampling accelerates p(r(t)) toward p(s(t))
Hidden State Leakage → improves → Path-Integration
confidence 88% · readily motivate the use of hidden state leakage in RNNs for replay... Leakage helps path-integration
Hidden State Momentum → enables → Underdamped Langevin Sampling
confidence 87% · propose the first model... through hidden state momentum, connect it to underdamped Langevin sampling
Hidden State Momentum → counteracts → Slowing from Adaptation
confidence 86% · together with adaptation, it counters slowness while maintaining exploration.
Neural Adaptation → induces → Non-Markov Sampling
confidence 85% · show that it incurs non-Markov sampling that also slows replay.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Biological neural networks (like the hippocampus) can internally generate "replay" resembling stimulus-driven activity. Recent computational models of replay use noisy recurrent neural networks (RNNs) trained to path-integrate. Replay in these networks has been described as Langevin sampling, but new modifiers of noisy RNN replay have surpassed this description. We re-examine noisy RNN replay as sampling to understand or improve it in three ways: (1) Under simple assumptions, we prove that the gradients replay activity should follow are time-varying and difficult to estimate, but readily motivate the use of hidden state leakage in RNNs for replay. (2) We confirm that hidden state adaptation (negative feedback) encourages exploration in replay, but show that it incurs non-Markov sampling that also slows replay. (3) We propose the first model of temporally compressed replay in noisy path-integrating RNNs through hidden state momentum, connect it to underdamped Langevin sampling, and show that, together with adaptation, it counters slowness while maintaining exploration. We verify our findings via path-integration of 2D triangular and T-maze paths and of high-dimensional paths of synthetic rat place cell activity.
Tags
Links
- Source: https://arxiv.org/abs/2602.18401v1
- Canonical: https://arxiv.org/abs/2602.18401v1
Trouble viewing inline? Open PDF directly →
Full Text
86,963 characters extracted from source content.
Expand or collapse full text
Leakage and Second-Order Dynamics Improve Hippocampal RNN Replay Josue Casco-Rodriguez 1 Nanda H. Krishna 2 Richard G. Baraniuk 1 Abstract Biological neural networks (like the hippocam- pus) can internally generate “replay” resembling stimulus-driven activity. Recent computational models of replay use noisy recurrent neural net- works (RNNs) trained to path-integrate. Replay in these networks has been described as Langevin sampling, but new modifiers of noisy RNN replay have surpassed this description. We re-examine noisy RNN replay as sampling to understand or improve it in three ways: (1) Under simple as- sumptions, we prove that the gradients replay ac- tivity should follow are time-varying and difficult to estimate, but readily motivate the use of hidden state leakage in RNNs for replay. (2) We confirm that hidden state adaptation (negative feedback) encourages exploration in replay, but show that it incurs non-Markov sampling that also slows replay. (3) We propose the first model of tempo- rally compressed replay in noisy path-integrating RNNs through hidden state momentum, connect it to underdamped Langevin sampling, and show that, together with adaptation, it counters slow- ness while maintaining exploration. We verify our findings via path-integration of 2D triangular and T-maze paths and of high-dimensional paths of synthetic rat place cell activity. 1. Introduction During quiescent periods such as sleep or wakeful resting, some neural circuits internally generate activity resembling that of active periods (Tingley & Peyrache, 2020). Such “replay” phenomena have been observed in the prefontal (Eu- ston et al., 2007; Peyrache et al., 2009), sensory (Kenet et al., 2003; Xu et al., 2012), motor (Hoffman & McNaughton, 2002), and entorhinal cortices (Gardner et al., 2022); the anterior thalamus (Peyrache et al., 2015); and the hippocam- 1 Department of Electrical and Computer Engineering, Rice University, USA 2 Mila - Quebec AI Institute & Universit ́ e de Montr ́ eal, Canada. Correspondence to: Josue Casco-Rodriguez <jc135@rice.edu>. Preprint. February 23, 2026. 012 0 2 4 Time s(t) 012 Time r(t) 012 Time r(t) underdamped 012 Time r(t) w/ adaptation Mean and Std of s(t) (awake paths) and r(t) (replay paths) Figure 1. Underdamped dynamics accelerate offline replay, adaptation slows it. Here we simulate a noisy RNNr(t)that opti- mally path-integrates an Ornstein-Uhlenbeck processs(t)from its velocitys ′ (t). We assumer(t)minimizes the loss in Equation (8) and thus evolves according to its score function∇ r(t) logp(r(t)) (Equations 11 and 15), performing a variant of Langevin sampling when no input is given. Above, we compare three modifiers of RNN activity: the default (no modification, a.k.a. overdamped), our proposed underdamped (momentum), and adaptation (nega- tive feedback) dynamics. Each modifier affects the replay distribu- tionp(r(t))in different ways: underdamped sampling accelerates p(r(t))towardsp(s(t)), decreasing the distance between them, while adaptation slows convergence ofp(r(t))towardsp(s(t)), increasing this distance. pus (Buzs ́ aki, 1986; Skaggs & McNaughton, 1996; N ́ adasdy et al., 1999; Lee & Wilson, 2002; Foster, 2017). Of these circuits, the hippocampus is particularly interesting because its robustness in tasks like navigation (O’Keefe & Nadel, 1978; Burgess et al., 1994; McNaughton et al., 1996) and planning (Pfeiffer & Foster, 2013; Miller et al., 2017) during active states seems crucially tied to its spontaneous activ- ity during quiescent states (Buzs ́ aki, 1989; 2015; Tononi & Cirelli, 2014; ́ Olafsd ́ ottir et al., 2015; 2018). While some works have produced replay using supervised generative models (Deperrois et al., 2022), most existing models of hippocampal activity treat replay as an emer- gent byproduct of careful network design. Relevant net- work parameters include connectivity structures (Shen & McNaughton, 1996; Milstein et al., 2023), local plastic- ity mechanisms (Hopfield, 2010; Litwin-Kumar & Doiron, 2014; Theodoni et al., 2018; Haga & Fukai, 2018; Asabuki & Fukai, 2025), firing rate adaptation (Chu et al., 2024; Azizi et al., 2013; Itskov et al., 2011; Dong et al., 2021; Li et al., 2024), and input modulation (Kang & DeWeese, 2019). While these models reproduce aspects of replay, they are typically motivated by empirical findings and lack rigorous theoretical justification. 1 arXiv:2602.18401v1 [cs.LG] 20 Feb 2026 Leakage and Second-Order Dynamics Improve Hippocampal RNN Replay A more principled model of hippocampal function with emergent replay is sequential predictive learning (Krishna et al., 2024; Levenstein et al., 2024), wherein neural circuits predict dynamic environment or task variables from imper- fect observations thereof, e.g., path-integrating velocity mea- surements to track a position. This normative description of the hippocampus as a sequence predictor (Levy, 1989; Stachenfeld et al., 2017) matches hippocampal encodings of upcoming stimuli (Davachi & DuBrow, 2015) and predic- tion errors (Aitken & Kok, 2022; Miller et al., 2023), and neural activity sweeps that represent possible future trajec- tories (Kay et al., 2020; Johnson & Redish, 2007). Unlike traditional, hand-crafted models of hippocampal circuits (i.e., continuous attractor networks (Samsonovich & Mc- Naughton, 1997; Battaglia & Treves, 1998)), sequential pre- dictive learning models are trained from data. Nonetheless, they account for the emergence of place cells (Recanatesi et al., 2021; Levenstein et al., 2024; Chen et al., 2024), grid cells (Cueva & Wei, 2018; Sorscher et al., 2019), and head direction cells (Cueva et al., 2020; Uria et al., 2022); can incorporate phenomena like theta oscillations (Levenstein et al., 2024); and exhibit quiescent replay activity (Krishna et al., 2024; Levenstein et al., 2024; Chen et al., 2024). Krishna et al. (2024) provided the first theoretical foundation for replay in sequential predictive learning networks, show- ing analytically that they generate diffusive replay (Stella et al., 2019) during quiescent activity (i.e., noise-driven ac- tivity in the absence of inputs) by Langevin sampling (Besag, 1994) from the waking activity distribution using its score function. Subsequent empirical work (Levenstein et al., 2024) introduced new mechanisms to induce exploration in replay through negative feedback, i.e., neural adaptation. Exploration is the notion that replay expresses a variety of behavioral sequences (Davidson et al., 2009; Pfeiffer, 2020), and is associated with long trajectories in neural space, visi- tation of multiple attractor basins, and transitions that were not present in awake activity. Adaptation can destabilize attractors and induce sudden transitions in replay activity (Itskov et al., 2011; Dong et al., 2021; Li et al., 2024; Lev- enstein et al., 2024), thereby facilitating exploration, and is thought to play a key role in the dynamics of replay in vivo (Levenstein et al., 2019). However, existing theory on sequential predictive learning cannot account for these mechanisms. Furthermore, sequential predictive learning models do not currently account for the temporal compres- sion of replay sequences relative to awake sequences of activity (N ́ adasdy et al., 1999; Buzs ́ aki, 2015; Michelmann et al., 2019; Farooq & Dragoi, 2019). This phenomenon, which could be caused by short-term facilitation (Leibold et al., 2008; Jaramillo & Kempter, 2017), is not currently captured in replay from any trained RNN model to our knowledge. Overall, while sequential predictive learning is a promising model of hippocampal function and replay, its recent em- pirical advances have outpaced its theoretical foundations. Moreover, it is unclear how to incorporate phenomena like temporal compression in these models, or how inductive biases in RNN design affect replay. We remedy these short- comings by characterizing how RNN design and Langevin sampling statistics affect each other. Some results describe how RNN design affects the speed and variance of replay activity, while others start from the Langevin sampling for- mulation of replay and either explain existing architectural choices as useful inductive biases, or propose new mecha- nisms to again modulate sampling. In summary, we answer three questions: 1.Optimal path-integration in the presence of noise re- quires RNNs to learn the score function of the noisy activity distribution. What is this function, might it in- form RNN design? The score function is time-variant and difficult to estimate, even for simple distributions, but our expression of it motivates the addition of leak- age (linear dynamics) in RNNs. 2.Adaptation (negative feedback) empirically induces exploration in replay. How does it affect Langevin sampling? Adaptation induces non-Markov second- order Langevin sampling that destabilizes attractors, which can both help (diversify) and hurt (slow) replay. 3.Traditional generative models benefit from a wide array of sampling techniques. Could new sampling meth- ods improve replay in noisy RNNs? Underdamped Langevin sampling via momentum quickens neural re- play, like temporal compression induced by short-term facilitation in vivo, and mitigates slowing from adap- tation while maintaining exploration. 2. Background In this section, we provide background and summarize re- sults from prior work (Krishna et al., 2024) that has de- scribed replay in path-integrating neural circuits as the sam- pling during quiescence (i.e., the absence of any inputs) of neural activity states from the distribution of waking, task- like activity. In short, the dynamics of recurrent networks that learn to optimally path-integrate noisy inputs cause the network’s states even in the absence of inputs to resemble those attained during actual task performance. We also pro- vide details on mechanisms used in the literature to improve or bias replay, which we explore the effects of in more detail in this work. For an overview of this section, see Section B. 2.1. Langevin Dynamics Langevin sampling from an unknown distributionp(x)en- tails stochastic gradient ascent of an iteratex(t)along the 2 Leakage and Second-Order Dynamics Improve Hippocampal RNN Replay log-likelihood ofp(x)via its score function∇ logp(x)or an estimate thereof: ̇x(t) = x(t) +∇ logp(x) + √ 2η(t),(1) whereη(t)is Gaussian white noise. While Equation 1 de- scribes overdamped dynamics, there also exist underdamped Langevin dynamics 1 (Equations 2 or 3, see Chapter 6 of Pavliotis (2014)) that converge faster to the target distribu- tionp(x)and better utilize noisy gradients (Cheng et al., 2018): ̈x(t) =∇ logp(x)− γ ̇x(t) + p 2γη(t),or(2) ̇x(t) = v(t), ̇v(t) =∇ logp(x)− γv(t) + p 2γη(t) (3) In this work, we consider noisy RNNs that have implic- itly learned to perform Langevin sampling of their own, fixed distributions of activity during task performance, when driven by just intrinsic noise and in the absence of inputs. That is, the activity of the RNNs at each unrolled timestep in the absence of inputs represents a plausible and likely vector of network activity during actual task performance in the presence of inputs. In particular, we view such networks in the context of replay, where neural circuits recapitulate task-like activity even during sleep. 2.2. Offline Replay in RNNs This work focuses on RNNs that implicitly learn to act as generative models over a fixed distribution of input se- quences. Krishna et al. (2024) have shown how noisy RNNs trained to path-integrate their inputs implicitly learn statistics that produce Langevin sampling of their own task- relevant activity distribution when no input is given, demon- strating statistically faithful replay sequences during quies- cence. That is, the RNNs’ activity in the absence of inputs “replays” states from the same distribution as RNN activity with inputs during actual task performance. This leads to the generation or “replay” of output sequences resembling those that the network sees during training or task performance with inputs. Path-integration is particularly relevant to neu- roscience: animals can leverage motion cues, observations, or prior experiences to accurately estimate positions (Seelig & Jayaraman, 2015; Chrastil, 2025), and neural circuits like the entorhinal cortex (Sorscher et al., 2019) have been iden- tified to perform such computations. Here we summarize the finding that noisy RNNs trained to path-integrate input time-series learn the score function of the input distribution (Krishna et al., 2024). Definition 2.1. A noisy recurrent neural network (RNN) has hidden statesr(t)that evolve at each timesteptvia some (nonlinear) function of its previous hidden states, an input 1 Discretized underdamped Langevin dynamics are a form of Hamiltonian MCMC (Cheng et al., 2018). signal u(t), and noise: r(t + ∆t) = f (r(t),u(t),σ r η(t))(4) Definition 2.2. A path-integration objectiveL(t)penalizes the difference between a state variables(t)and a learnable linear projection of the RNN hidden state r(t): L(t) = E η ∥s(t)−Dr(t)∥ 2 (5) Assumption 2.3. Krishna et al. (2024) approximate the hidden state dynamics of a noisy RNN as the sum of two functions 2 and white noise σ r η(t)∼N (0,σ 2 r ∆t). ∆r(t) = r(t + ∆t)−r(t) ≈ ∆r 1 (t) + ∆r 2 (t) + σ r η(t) (6) Assumption 2.4. The optimalr(t)minimizesL(t)such that p(r ∗ (t)) is normal around D † s(t): p(r ∗ (t)|s(t))∼N (D † s(t), Iσ 2 r ∆t)(7) Lemma 2.5. With Assumptions 2.3 and 2.4,L(t + ∆t)is upper bounded by b L(t + ∆t): b L(t + ∆t) =∥s ′ (t)−D∆r 2 (t)∥ + E η ∥Dσ r η(t)∥ 2 +∥D∥ F E η ∥∆r 1 (t) + σ r η(t− ∆t)∥ 2 (8) Assumption 2.6. The optimalr(t)greedily minimizes b L at t only, without concern for long-range dependencies: r ∗ (t) T t=0 = arg min r(t) T t=0 Z T t=0 L(t + ∆t)∆t ≈ arg min r(t) b L(t + ∆t) T t=0 (9) Theorem 2.7. Given Assumption 2.6, the optimal update ∆r ∗ (t) = arg min ∆r 2 (t) b L(t + ∆t) + arg min ∆r 1 (t) b L(t + ∆t)+σ r η(t)followss ′ (t)and the score function ofp(r(t)): ∆r ∗ (t) = D † s ′ (t)∆t + σ 2 r ∆t∇ r(t) logp(r(t)) + σ r η(t) (10) During training, noisy RNNs are (unless otherwise stated, see Section 2.3) providedu(t) = s ′ (t)(or a nonlinear observation thereof) to perform path-integration and focus on denoising. Theorem 2.8. In the absence of input (quiescence), a noisy RNN already trained to path-integrates(t)froms ′ (t)will perform gradient ascent along the score function ofr(t). Ifp(r(t))andp(s(t))are stationary, then this ascent is Langevin sampling (Equation 1). If the variance ofσ r η(t) is scaled by a factor of 2, thenp(r(t))is guaranteed to converge to the steady-state distribution p(r) = p(D † s). If ∆r(t) = σ 2 r ∆t∇ r(t) logp(r(t)) + √ 2σ r η(t), then lim t→∞ p(r(t)) = p(D † s) (11) 2 Each function∆r 1 (t), ∆r 2 (t)can depend on variables be- yond t, but they are omitted for concision. 3 Leakage and Second-Order Dynamics Improve Hippocampal RNN Replay 2.3. Existing Methods of Biasing Replay in RNNs Neural adaptation.Biological neurons can mitigate pro- longed or low-frequency activity via negative feedback, or adaptation (Benda, 2021; Gutkin & Zeldenrust, 2014). This feedback has proven important for describing in vivo hip- pocampal activity and replay (Itskov et al., 2011; Levenstein et al., 2019), and in computational models of replay has been shown to encourage long replay trajectories (exploration) by preventing neural activations from getting stuck in at- tractor basins (Dong et al., 2021; Li et al., 2024; Levenstein et al., 2024). Like Levenstein et al. (2024), we define adap- tation as negative feedbackc(t)added to RNN activityr(t) (Equation 4) after training 3 : r(t + ∆t) = f (r(t),u(t),σ r η(t))−c(t), ∆c(t) = 1 τ a (−c(t) + b a r(t)) (12) Masked training. Denoisers and autoencoders benefit from masked training, wherein some regions of input data are set to zero before model processing (Zhang et al., 2023). Levenstein et al. (2024) introduce masked training for path- integration by periodically masking the inputu(t)(the ob- servation ofs ′ (t)): only at everyk-th timestep does the RNN observe a nonzero input (Equation 13). u(t) = ( s ′ (t), t mod k = 0 0,otherwise (13) Levenstein et al. (2024) found that masked training makes replay sequences more coherent and makes manifolds of neural activity more similar to the spatial layout of the envi- ronment. We found that masked training improves replay stability, so we use it (with k ≥ 3) in training all RNNs. 3. Estimating the Score Function of Noisy RNN Activity Noisy RNNs trained to path-integrate implicitly learn the score function of their activity. Previous works have not examined the score function; in fact, they assume the distri- bution of RNN activityp(r(t))is stationary (Theorem 2.8), and thus the score function∇ r(t) logp(r(t))depends only onr(t)(Krishna et al., 2024). However, we refute this assumption, and in doing so reveal the role of leakage in path-integration: even if the RNN path-integrates a simple Gaussian process, the score function requires information beyondr(t), which it employs through linear leakage (de- cay) ofr(t). This linearity suggests that leakage is useful for path-integration, which we confirm experimentally. 3 Subtraction of the moving averagec(t)also arises naturally from greedy minimization ofL(t) + 1 2 ∥c(t)∥ 2 2 . 3.1. Challenges in Simple Distributions First, we examine how the score function of optimal path- integrating RNN activityr(t)has nonstationarities that are challenging to perfectly estimate, even for simple Gaussian processes. Assumption 3.1. The observed statess(t)form some Gaus- sian process: p(s(t))∼N μ s(t) , Σ s(t) . Theorem 3.2. With Assumption 3.1, the score function of trained activityr(t)has a closed form which, while linear in r(t), is nonlinear with respect to the parameters ofp(s(t)) (see Section C.2): σ 2 r ∆t∇ r(t) logp(r(t)) =−Λ(t) r(t)−D † μ s(t) (14) WithΛ(t) = σ 2 r ∆t Iσ 2 r ∆t +D † Σ s(t) (D † ) T −1 as the leakage matrix ofr(t), we can already gain some insight from Equation 14. Remark 3.3. Ifp(s(t))is Gaussian, then the score ofp(r(t)) is simply a linear function ofr(t), but its parameters are non- linear functions of time, and only as stationary as p(s(t)). Remark 3.4. The eigenvalues of the leakage matrixΛ(t)are always between 0 and 1 (see Appendix C.2). Moreover, the eigenvalues ofΛ(t)andD † Σ s(t) (D † ) T inversely correlate: strong decay ofr(t)implies weak noiseΣ r(t) , and weak decay of r(t) implies strong noise Σ r(t) . To further illustrate how estimating the score function ofr(t) (Equation 14) can be challenging, we now examine a scalar RNNr ou (t)trained to path-integrate Ornstein-Uhlenbeck processes. Assumption 3.5. The observed states follow a scalar Ornstein-Uhlenbeck process parameterized byθ,μ,σ s : s ′ ou (t) = θ(μ − s ou (t)) + σ s η(t) , wherep(s(0)) ∼ N (0,σ 2 0 ). In other words,p(s ou (t))parameterizes a di- rected random walk from s ou (0) to μ. Remark 3.6. The score function of the optimalr ou (t)under Assumption 3.5 follows from Equation (26) (see Section D): σ 2 r ∆t∇ r ou (t) logp(r ou (t)) = −σ 2 r ∆t r ou (t)− μ 1− e −θt σ 2 r ∆t + σ 2 s 2θ (1− e −2θt ) + σ 2 0 e −θt (15) The score function, and thus the optimal quiescent activity,is evidently complex and nonstationary: lim t→0 σ 2 r ∆t∇ r ou (t) logp(r ou (t)) = − σ 2 r ∆t σ 2 r ∆t+σ 2 0 r ou (t) , whilelim t→∞ σ 2 r ∆t∇ r ou (t) logp(r ou (t))= − σ 2 r ∆t σ 2 r ∆t+σ 2 s /2θ (r ou (t) − μ).While one could force stationarity by implicitly assuming the process starts ats ou (0) = μ, or assuming the steady-state dynamics (t → ∞) are the most important, we argue that any 4 Leakage and Second-Order Dynamics Improve Hippocampal RNN Replay such approach would miss a fundamentally relevant aspect of the Ornstein-Uhlenbeck process for navigation: intention. Unlike the Wiener process (Section D), the Ornstein-Uhlenbeck process can describe a random walk that intentionally navigates froms u (0)toμ, rather than one that simply wanders aroundμ. Thus, for navigation, the non-stationary, or “early”, dynamics of the Ornstein- Uhlenbeck process are the most salient. Given the relevance of non-stationary dynamics for navigation, our analyses focus on the entire course of replay dynamics (rather than steady-state distributions), examining properties like speed and path diversity (exploration). 3.2. The Advantage of Leakage −2 0 Training Iters. Log MSE loss T-Maze LeakNo leak −2 0 Training Iters. Triangle 12 3 12 3 k =k = Figure 2. Leakage helps path-integration. Here we train RNNs on two tasks, ablating the leakage term. Leakage helps training, especially when losses increase with the masking difficultyk(note thatk = 1is equivalent to unmasked training, see Equation 13). Means are solid, standard deviations are faint. Theorem 3.2 and Remark 3.3 suggest that linear leakage may be a useful inductive bias for RNNs learning to path- integrate: the score function for a Gaussian process is linear with respect tor(t), although the parameters of said linearity are nonlinear functions of time. We examine the utility of leakage by comparing two RNNs trained to path-integrate: r(t + ∆t) = κr(t) + f 1 (r(t),u(t),σ r η(t))(RNN with leakage0 < κ < 1) andr(t+∆t) = f 2 (r(t),u(t),σ r η(t)) (RNN without leakage), wheref 1 ,f 2 are shallow ReLU layers trained separately (see Section A.4). The second is more reminiscent of traditional RNNs in machine learning (e.g., ReLU or gated RNNs), which do not typically employ leakage. In Figure 2, we show that leakage is useful for path-integration, especially with masked training (k > 1). Our results suggest that leakage is a useful component of path-integrating RNNs, although other architectures may be able to successfully learn without it (e.g., the layer-norm RNN of Levenstein et al. (2024)). 4. Second-Order Langevin Sampling for Neural Replay In the previous section, we examined the score function of trained path-integrating RNN activity. Now, we are as- suming that trained RNNs have well-estimated the score function, and follow it during quiescent (internally-driven) activity to generate replay via Langevin dynamics. Here we ask how modulating the dynamics of such RNNs affects the distribution of replay—in other words, how changing the RNN dynamics biases the distribution of replay. We see that adaptation (negative feedback) incurs a non-ideal form of second-order Langevin sampling, so we propose a comple- mentary alternative that explicitly performs underdamped second-order sampling. 4.1. Adaptation as Underdamped Langevin Dynamics First we examine adaptation (negative feedback), as defined in Section 2.3. Proposition 4.1. Adding adaptation (Equation (12)) to a trained path-integrating RNN during quiescence (Equa- tion (11)) incurs Langevin sampling with negative feedback: ∆r(t) = σ 2 r ∆t∇ r(t) logp(r(t)) + √ 2σ r η(t)−c(t), ∆c(t) = 1 τ a (−c(t) + b a r(t)) (16) For the clearest illustration of the effects of adaptation, let us examine a stationaryp(r(t))—a simplification which we argued in Section 3.1 is not realistic, but is nonetheless intuitive. Assumption 4.2. The observed states are normal (Assump- tion 3.1) and stationary: p(s(t))∼N (μ, Σ). Theorem 4.3. Adding adaptation to an RNN trained to path-integrate states drawn from a stationary Gaussian dis- tribution (Assumption 4.2 and Equation (16)) produces the following second-order stochastic dynamics during quies- cence (see Section E): r ′ (t) = b a τ a I + σ 2 r ∆t d 2 dr(t) 2 logp(r(t)) r ′ (t) − b a τ a σ 2 r ∆t∇ r(t) logp(r(t)) + 1 τ a r(t) −σ b a τ a η(t) + ση ′ (t) (17) Comparing Equation 17 with Equation 2, the two indeed resemble each other: adaptation seems to induce a form of underdamped Langevin dynamics. This may help explain the observed utility of adaptation for generating replay (It- skov et al., 2011; Levenstein et al., 2019; 2024). Moreover, since we established in Section 3 that the score function of even a basic stochastic process is difficult to estimate, the effectiveness of underdamped Langevin sampling for working with noisy gradients (Cheng et al., 2018) may be useful when∇ r(t) logp(r(t)) is poorly estimated. However, interpreting Equation (17) as underdamped Langevin sampling reveals some shortcomings thereof as a sampling method. 5 Leakage and Second-Order Dynamics Improve Hippocampal RNN Replay Remark 4.4. The coefficient ofr ′ (t)is usually constant, and should be positive to ensure convergence (Pavliotis (2014), pg. 183), but b a τ a I +∇ r(t) logp(r(t))is not constant and could be negative. Remark 4.5. Underdamped Langevin sampling fromr(t) should not have a negative sign in front of∇ r(t) logp(r(t)) if the intention is to maximize p(r(t)). 4.2. Replay via Explicit Underdamped Langevin Dynamics While adaptation is a biologically plausible way to perform a variant of underdamped Langevin dynamics in RNNs, we propose an alternative method, from Equation 3, to more clearly and directly perform underdamped Langevin sam- pling. It is conceptually similar to RNNs with momentum (Nguyen et al., 2020): a velocity termv(t)accumulates previous r(t) values when friction λ v ∈ [0, 1] is below 1 4 . Definition 4.6. We implement explicitly underdamped dy- namics via momentum governed by frictionλ v . When λ v = 1, dynamics revert to overdamped (Equation (4)): v(t+∆t) = (1−λ v )v(t)+f (r(t),u(t),σ r η(t))−r(t) | z ∆r(t) if λ v =1 , ∆r(t) = v(t + ∆t) (18) Biological plausibility. Several previous works in theo- retical neuroscience have proposed mechanisms involving momentum, including Hamiltonian dynamics, for fast and improved sampling (Hennequin et al., 2014; Masset et al., 2022; Furlong et al., 2024; Dong & Wu, 2023; Aitchison & Lengyel, 2016; Dong et al., 2022). Some have implemented these dynamics in E/I or spiking networks. Thus, it is con- ceivable that the brain could use momentum to quickly and efficiently sample (Hennequin et al., 2014). Additionally, sampling schemes with momentum improve modeling of human random sequence generation (Castillo et al., 2024), and momentum has been found in hippocampal replay tra- jectories (Krause & Drugowitsch, 2022). We propose mo- mentum as a circuit mechanism for temporal compression in replay, but we cannot directly compare it to the physio- logical mechanisms behind temporal compression in vivo because they are not yet fully understood. Nonetheless, we identify two mechanisms for which momentum may be a reasonable initial approximation: (1) Short-term facilita- tion can induce phase precession and temporal compression in spiking networks (Leibold et al., 2008). Extensions to rate networks are scant, so momentum may be a reasonable initial approximation. (2) Short-term post-synaptic plastic- ity via NMDA receptors induces momentum-like effects in continuous attractor networks (Zhao et al., 2023). 4 For a comparison ofλ v and theγterm from Equations 3 and 2, see Section A.5. 5. Numerical Results Now we examine how adaptation and underdampening (Equations 12 and 18) bias replay distributions in trained path-integrating RNNs. We first see they counter each other: adaptation slows replay, while underdampening quickens it. Then we confirm that adaptation induces exploration, and show that underdampening does not prevent exploration, but rather complements it by increasing path lengths. We note here that replay trajectories are obtained by first ran- domly initializing the hidden state of the RNN, following which we collect activations for several unrolled timesteps in the absence of inputs but in the presence of noise. Fur- ther details are provided in Appendix A.1. Here we use ReLU and leaky ReLU trained RNNs, as justified in Sec- tion A.2, but in Section G we confirm that adaptation slows and underdampening compresses replay in tanh RNNs. Experiments.We have five tasks on which we train RNNs and then examine how introducing sampling mechanisms post-training (adaptation strengthb a and frictionλ v ) affects replay (quiescent activity) statistics. We explain our imple- mentation and many measured replay statistics in Section A. 1.1D Ornstein-Uhlenbeck (OU) process (Figure 1): the optimal (with respect to Equation (8)) path-integrating RNN has a closed form (Equation (15)), which we use in lieu of a trained RNN. 2.2D T-maze and triangle (Figure 3): we simulate each direction (two in T-maze, six in triangle) as a 2D OU process, and train a ReLU RNN to integrate 2D paths from every direction. 3.Rat trajectories (Section F, Figure 8): like Krishna et al. (2024), we simulate 512 place cells that encode 2D directed (biased) or undirected (unbiased) random walks from RatInABox (George et al., 2024), train a ReLU RNN to path-integrate place cell activity, and decode activity in 2D. We use in silico rat activity to mimic biological data because replay events in vivo are sparse and difficult to extract from background noisy activity, therefore outside the scope of this work. Underdampening and adaptation seem to counter each other. We initially observe that the two modifiers coun- teract each other: in Figures 1 and 3, adaptation repels trajectories away from attractors (such as endpoints), while underdampening accelerates trajectories towards them. This makes sense given Remark 4.5 and the nature of under- damped sampling. Then we measure how the two modifiers affect the similarity between replay (internally-driven) and awake (observed) trajectories. Figure 4 shows they both generally decrease the similarity to awake trajectories, but 6 Leakage and Second-Order Dynamics Improve Hippocampal RNN Replay T-maze Triangle • = endpoint (a) OverdampedAdaptation (b a = 1)OverdampedAdaptation (b a = 1) Underdamped (λ v = 0.7) Underdamped + Adaptation Underdamped (λ v = 0.7) Underdamped + Adaptation 1 0 1 0 −1 01 −1 01 1 0 1 0 0 1 0 1 (b)(c) Figure 3. Underdampening and adaptation counter each other. Here we show replay from RNNs trained to path-integrate in T-maze or triangular environments. (a) Awake paths in each task form a mixture of Ornstein-Uhlenbeck processes, one for each direction of travel. Awake paths reach their endpoints and stay there. In (b) and (c) are mean replay paths (darkening over time) from T-maze and triangle tasks, simulated for the same time as awake paths. Standard deviations are faint, ideal path means are dashed. Like in Figure 1, adaptation slows convergence towards endpoints, underdampening quickens it. Also, the two mechanisms induce deviations that negate each other. 0.81 0 10 Friction λ v Wasserstein distance (↓) T-maze 0.00.51.0 0.81 0 2 4 Friction λ v Triangle 0.60.81 0.08 0.1 0.12 Friction λ v Biased rat 0135 10.90.80.7 0.11 0.12 0.13 Friction λ v Unbiased rat 0.00.10.20.3 Adaptation strength b a : Figure 4. Underdampening improves replay fidelity in the presence of adaptation. We compute the Wasserstein distance (dissimilarity) between awake and replay path distributions (p(s(t) T t=0 )andp(r(t) T t=0 )), varying friction and adaptation strength (see Section A for details). While the two mechanisms both generally increase this distance, underdampening (λ v < 1) decreases it if adaptation is nonzero. Like in Figure 3, underdampening counters adaptation-induced deviations. they do counter each other insofar as underdampening, in the presence of adaptation, increases similarity to awake trajectories. Thus far, underdampening counters adaptation qualitatively and statistically. Adaptation slows replay, underdampening accelerates it. Next, we examine a key component of replay: speed. In Fig- ures 4 and 5, we generally see that adaptation slows replay or increases the dissimilarity between replay and awake tra- jectories. The exception is the biased rat trajectories, since they are the only task where all trajectories reach the same common goal. Such path distributions resemble Ornstein- Uhlenbeck processes with steady-state meanμ = 0: in such cases, adaptation actually accelerates convergence towards the steady-state. In Figure 5, we see that underdampening increases replay speed, performing temporal compression relative to awake activity. Underdampening complements exploration from adap- tation. We have established that underdampening coun- ters adaptation, both qualitatively and in terms of speed. However, we do not want to merely propose a mechanism 00.51 1 0.9 0.8 0.7 Friction λ v T-maze −30060120 00.51 1 0.9 0.8 0.7 Adaptation strength b a Triangle −30 055110 0135 1 0.8 0.6 0.5 Biased rat −600 150300 Change (%) from awake reach time: Median replay reach time (↓) Figure 5. Underdampening temporally compresses replay. We calculate how long it takes awake and replay trajectories to reach their endpoints. Underdampening (λ v < 1) not only shortens this reach time, but makes it smaller than that of awake paths, temporally compressing awake activity. See Section F Figure 9 for mean reach times. We do not include unbiased rat trajectories because they do not have defined endpoints, but we do confirm in Section F Figure 10 that underdampening quickens them. 7 Leakage and Second-Order Dynamics Improve Hippocampal RNN Replay 0.81 8 8.5 Friction λ v Mean path length (↑) T-maze 0.00.51.0 0.81 8.5 9 Friction λ v Triangle 0.60.81 3 4 Friction λ v Biased rat 0135 0.81 3 3.5 4 Friction λ v Unbiased rat 0.00.10.20.3 Adaptation strength b a : Figure 6. On average, underdampening increases exploration via path length. In here and Figure 7 we simulate replay paths for more time than awake paths, varying friction and adaptation strength. Shown above, underdampening (λ v < 1) increases the average length of replay paths. It may do so via increased replay speeds (Section F Figure 10) or via additional path transitions (Figure 7); the latter is how adaptation increases path length. Regions visited (↑) • = 2 •,• = 3 • = 4 (a) Examples of exploration in replay: 0.81 2 2.1 Friction λ v Average regions visited (↑) T-maze 0.00.51.0 0.81 2 2.5 Friction λ v Triangle Adaptation strength b a : (c) On average, λ v < 1 increases exploration. (i) increase exploration (i) decrease exploration (i) not affect exploration r.v. = 2.0r.v. = 4.0r.v. = 3.0 r.v. = 3.2r.v. = 3.0r.v. = 3.0 1 0 1 0 010101 λ v = 1 λ v = 0.7 (b) Underdampening (λ v < 1) might: r.v. = mean regions visited,• = start,× = end Figure 7. On average, underdampening increases exploration via regions visited. Another relevant aspect of exploration is transitions not present in awake activity. (a) Here we quantify these transitions via regions visited. Each region is an area where every point is closest to the same endpoint. In our T-maze and triangle tasks, awake paths go from starting points to endpoints and stay there, visiting two regions total (starting and ending). Meanwhile, adaptation (b a = 1) can make replay paths visit multiple endpoints (regions visited ¿ 2). (b) When we use adaptation and underdampening (λ v < 1), we see that replay paths might visit (i) more, (i) fewer, or (i) the same regions (see Section F Figure 12 for more details). (c) However, on average, across several different trained models, underdampening increases regions visited, thus increasing adaptation. that undoes adaptation. Adaptation can induce exploration, i.e., prevent replay trajectories from getting stuck in attrac- tors. We find that underdampening does not on average pre- vent adaptation-induced exploration. In fact, underdampen- ing complements adaptation for exploration: underdamped paths travel farther (Figures 6 and 7), have more variance (Section F Figure 11), and generally exhibit the same, if not more, exploratory behavior as they did with only adaptation (Figure 7). Underdampening maintains exploration while counteracting the slowness from adaptation. 6. Conclusions and Future Work We have re-applied Langevin sampling theory to replay in sequential predictive learning networks, producing theoret- ically and confirming empirically three key insights: (1) estimating the per-timestep score function of RNN activ- ity is challenging, but does benefit from linear leakage; (2) adaptation (negative feedback) is a variant of underdamped Langevin sampling that encourages exploration (as shown in prior works) but also slows replay; (3) our new under- dampening mechanism (momentum) temporally compresses replay while also increasing exploration from adaptation. These findings improve our understanding of biological neural networks that produce replay, like the hippocampus. Our proposed underdampening mechanism via momentum could be tied to short-term facilitation (which can be probed experimentally), connected to specific subregions in the hippocampus (like Chen et al. (2024)), or refined through in- sights from existing RNNs with momentum (Nguyen et al., 2020). Future efforts could confirm our findings in more complex environments (Wood et al., 2018; Levenstein et al., 2024). Like preceding works, our network models are rate- based, but extending our work to spike-based models of replay and sequential predictive learning (Saponati & Vinck, 2023; Asabuki & Fukai, 2025; Bono et al., 2023) would be interesting (i.e., extending Langevin sampling to Poisson processes). Several avenues remain for connecting Langevin sampling to neural replay. 8 Leakage and Second-Order Dynamics Improve Hippocampal RNN Replay Impact Statement This work aims to advance our understanding of the com- putations underlying replay in the brain. It is a theoreti- cal study that aims to advance our understanding of neuro- science through the use of artificial neural network models, as is common in computational neuroscience. There may be many potential societal consequences of our work in the long-term, none of which we feel must be specifically highlighted here. Reproducibility Statement For our theoretical contributions, i.e., Theorems 3.2 and 4.3, we provide proofs in Appendix C and E (proofs for Theo- rems 2.7 and 2.8 may be found in Section 2 of Krishna et al. (2024)). Details on our numerical experiments, including associated hyperparameters and implementation details are provided in Appendix A. Our code has been submitted as Supplementary Material (seeREADME.mdfor instructions) and will be made publicly available upon publication. References Aitchison, L. and Lengyel, M. The Hamiltonian brain: Efficient probabilistic inference with excitatory-inhibitory neural circuit dynamics. PLoS Computational biology, 12(12), 2016. Aitken, F. and Kok, P. Hippocampal representations switch from errors to predictions during acquisition of predictive associations. Nature Communications, 13(1), 2022. Alemohammad, S., Casco-Rodriguez, J., Luzi, L., Hu- mayun, A. I., Babaei, H., LeJeune, D., Siahkoohi, A., and Baraniuk, R. G. Self-consuming generative mod- els go MAD. In International Conference on Learning Representations (ICLR), 2024. Ali, A., Ahmad, N., de Groot, E., van Gerven, M. A. J., and Kietzmann, T. C. Predictive coding is a consequence of energy efficiency in recurrent neural networks. Patterns, 3(12), 2022. Asabuki, T. and Fukai, T. Predictive learning rules generate a cortical-like replay of probabilistic sensory experiences. eLife, 13, 2025. Azizi, A. H., Wiskott, L., and Cheng, S. A computational model for preplay in the hippocampus. Frontiers in com- putational neuroscience, 7:161, 2013. Battaglia, F. P. and Treves, A. Attractor neural networks storing multiple space representations: a model for hip- pocampal place fields. Physical Review E, 58(6), 1998. Benda, J. Neural adaptation. Current Biology, 31(3), 2021. Besag, J. E. Discussion of “Representations of knowledge in complex systems” by Ulf Grenander and Michael I Miller. Journal of the Royal Statistics Society B, 56(4): 591–592, 1994. Bonneel, N., Rabin, J., Peyr ́ e, G., and Pfister, H. Sliced and radon wasserstein barycenters of measures. Journal of Mathematical Imaging and Vision, 51(1), 2015. Bono, J., Zannone, S., Pedrosa, V., and Clopath, C. Learning predictive cognitive maps with spiking neurons during behavior and replays. Elife, 12, 2023. Burgess, N., Recce, M., and O’Keefe, J. A model of hip- pocampal function. Neural networks, 7(6-7), 1994. Buzs ́ aki, G. Hippocampal sharp waves: their origin and significance. Brain research, 398(2), 1986. Buzs ́ aki, G. Two-stage model of memory trace formation: a role for “noisy” brain states. Neuroscience, 31(3), 1989. Buzs ́ aki, G. Hippocampal sharp wave-ripple: A cognitive biomarker for episodic memory and planning. Hippocam- pus, 25(10), 2015. Castillo, L., Le ́ on-Villagr ́ a, P., Chater, N., and Sanborn, A. Explaining the flaws in human random generation as local sampling with momentum. PLOS Computational Biology, 20(1), 2024. Chen, Y., Zhang, H., Cameron, M., and Sejnowski, T. Pre- dictive sequence learning in the hippocampal formation. Neuron, 112(15), 2024. Cheng, X., Chatterji, N. S., Bartlett, P. L., and Jordan, M. I. Underdamped Langevin MCMC: A non-asymptotic anal- ysis. In Proceedings of the 31st Conference On Learning Theory, volume 75, 2018. Chettih, S. N., Mackevicius, E. L., Hale, S., and Aronov, D. Barcoding of episodic memories in the hippocampus of a food-caching bird. Cell, 187(8), 2024. Chrastil, E. R.Human path integration and the neu- ral underpinnings.In Encyclopedia of the Human Brain (Second Edition), p. 157–170. Elsevier, 2025. ISBN 978-0-12-820481-8. doi: https://doi.org/10.1016/ B978-0-12-820480-1.00016-4. Chu, T., Ji, Z., Zuo, J., Mi, Y., Zhang, W., Huang, T., Bush, D., Burgess, N., and Wu, S. Firing rate adaptation affords place cell theta sweeps, phase precession and procession. bioRxiv, 2024. Churchland, M. M. and Shenoy, K. V. Preparatory activity and the expansive null-space. Nature Reviews Neuro- science, 25(4), 2024. 9 Leakage and Second-Order Dynamics Improve Hippocampal RNN Replay Croitoru, F.-A., Hondru, V., Ionescu, R. T., and Shah, M. Diffusion models in vision: A survey. IEEE Transac- tions on Pattern Analysis and Machine Intelligence, 45 (9), 2023. Cueva, C. J. and Wei, X.-X. Emergence of grid-like repre- sentations by training recurrent neural networks to per- form spatial localization. In International Conference on Learning Representations, 2018. Cueva, C. J., Wang, P. Y., Chin, M., and Wei, X.-X. Emer- gence of functional and structural properties of the head direction system by optimization of recurrent neural net- works. In International Conference on Learning Repre- sentations, 2020. Davachi, L. and DuBrow, S. How the hippocampus pre- serves order: the role of prediction and context. Trends in cognitive sciences, 19(2), 2015. Davidson, T. J., Kloosterman, F., and Wilson, M. A. Hip- pocampal replay of extended experience. Neuron, 63(4), 2009. Deperrois, N., Petrovici, M. A., Senn, W., and Jordan, J. Learning cortical representations through perturbed and adversarial dreaming. eLife, 11:e76384, 2022. ISSN 2050-084X. Dong, X. and Wu, S. Neural sampling in hierarchical exponential-family energy-based models. Advances in Neural Information Processing Systems (NeurIPS), 36, 2023. Dong, X., Chu, T., Huang, T., Ji, Z., and Wu, S. Noisy adaptation generates L ́ evy flights in attractor neural net- works. Advances in Neural Information Processing Sys- tems (NeurIPS), 34, 2021. Dong, X., Ji, Z., Chu, T., Huang, T., Zhang, W., and Wu, S. Adaptation accelerating sampling-based bayesian in- ference in attractor neural networks. Advances in Neural Information Processing Systems (NeurIPS), 35, 2022. Euston, D. R., Tatsuno, M., and McNaughton, B. L. Fast- forward playback of recent memory sequences in pre- frontal cortex during sleep. Science, 318(5853):1147– 1150, 2007. Farooq, U. and Dragoi, G. Emergence of preconfigured and plastic time-compressed sequences in early postnatal development. Science, 363(6423), 2019. Foster, D. J. Replay comes of age. Annual Review of Neuroscience, 40:581–602, 2017. ISSN 1545-4126. Furlong, P. M., Simone, K., Dumont, N. S.-Y., Bartlett, M., Stewart, T. C., Orchard, J., and Eliasmith, C. Biologically- plausible Markov Chain Monte Carlo sampling from vec- tor symbolic algebra-encoded distributions. In Interna- tional Conference on Artificial Neural Networks, 2024. Gardner, R. J., Hermansen, E., Pachitariu, M., Burak, Y., Baas, N. A., Dunn, B. A., Moser, M.-B., and Moser, E. I. Toroidal topology of population activity in grid cells. Nature, 602(7895):123–128, 2022. ISSN 1476-4687. George, T. M., Rastogi, M., de Cothi, W., Clopath, C., Stachenfeld, K., and Barry, C. RatInABox, a toolkit for modelling locomotion and neuronal activity in continuous environments. eLife, 13:e85274, 2024. ISSN 2050-084X. Gozalo-Brizuela, R. and Garrido-Merchan, E. C. ChatGPT is not all you need. a state of the art review of large generative AI models. arXiv preprint arXiv:2301.04655, 2023. Gutkin, B. and Zeldenrust, F. Spike frequency adaptation. Scholarpedia, 9(2), 2014. Haga, T. and Fukai, T. Recurrent network model for learning goal-directed sequences through reverse replay. eLife, 7: e34171, 2018. ISSN 2050-084X. Hennequin, G., Aitchison, L., and Lengyel, M.Fast sampling-based inference in balanced neuronal networks. Advances in Neural Information Processing Systems (NeurIPS), 27, 2014. Hoffman, K. L. and McNaughton, B. L. Coordinated reacti- vation of distributed memory traces in primate neocortex. Science, 297(5589):2070–2073, 2002. Hopfield, J. J. Neurodynamics of mental exploration. Pro- ceedings of the National Academy of Sciences, 107(4): 1648–1653, 2010. Itskov, V., Curto, C., Pastalkova, E., and Buzs ́ aki, G. Cell assembly sequences arising from spike threshold adapta- tion keep track of time in the hippocampus. Journal of Neuroscience, 31(8), 2011. Jaramillo, J. and Kempter, R. Phase precession: a neural code underlying episodic memory? Current Opinion in Neurobiology, 43, 2017. Johnson, A. and Redish, A. D. Neural ensembles in CA3 transiently encode paths forward of the animal at a deci- sion point. Journal of Neuroscience, 27(45), 2007. Kang, L. and DeWeese, M. R. Replay as wavefronts and theta sequences as bump oscillations in a grid cell attrac- tor network. eLife, 8:e46351, 2019. ISSN 2050-084X. 10 Leakage and Second-Order Dynamics Improve Hippocampal RNN Replay Kay, K., Chung, J. E., Sosa, M., Schor, J. S., Karlsson, M. P., Larkin, M. C., Liu, D. F., and Frank, L. M. Constant sub-second cycling between representations of possible futures in the hippocampus. Cell, 180(3), 2020. Kenet, T., Bibitchkov, D., Tsodyks, M., Grinvald, A., and Arieli, A. Spontaneously emerging cortical representa- tions of visual attributes. Nature, 425(6961):954–956, 2003. ISSN 1476-4687. Krause, E. L. and Drugowitsch, J. A large majority of awake hippocampal sharp-wave ripples feature spatial trajectories with momentum. Neuron, 110(4), 2022. Krishna, N. H., Bredenberg, C., Levenstein, D., Richards, B. A., and Lajoie, G. Sufficient conditions for offline reactivation in recurrent neural networks. In International Conference on Learning Representations (ICLR), 2024. Lee, A. K. and Wilson, M. A. Memory of sequential experi- ence in the hippocampus during slow wave sleep. Neuron, 36(6):1183–1194, 2002. ISSN 0896-6273. Leibold, C., Gundlfinger, A., Schmidt, R., Thurley, K., Schmitz, D., and Kempter, R. Temporal compression mediated by short-term synaptic plasticity. Proceedings of the National Academy of Sciences, 105(11), 2008. Levenstein, D., Buzs ́ aki, G., and Rinzel, J. NREM sleep in the rodent neocortex and hippocampus reflects excitable dynamics. Nature Communications, 10(1), 2019. Levenstein, D., Efremov, A., Eyono, R. H., Peyrache, A., and Richards, B. Sequential predictive learning is a uni- fying theory for hippocampal representation and replay. bioRxiv, 2024. Levy, W. B. A computational approach to hippocampal function. In Psychology of learning and motivation, vol- ume 23. Elsevier, 1989. Li, X., Thickstun, J., Gulrajani, I., Liang, P. S., and Hashimoto, T. B. Diffusion-LM improves controllable text generation. Advances in Neural Information Process- ing Systems (NeurIPS), 35, 2022. Li, Y., Chu, T., and Wu, S.Dynamics of adaptive continuous attractor neural networks. arXiv preprint arXiv:2410.06517, 2024. Litwin-Kumar, A. and Doiron, B. Formation and mainte- nance of neuronal assemblies through synaptic plasticity. Nature Communications, 5(1):5319, 2014. ISSN 2041- 1723. Luzi, L., Mayer, P. M., Casco-Rodriguez, J., Siahkoohi, A., and Baraniuk, R. Boomerang: Local sampling on image manifolds using diffusion models. Transactions on Machine Learning Research (TMLR), 2024. Masset, P., Zavatone-Veth, J., Connor, J. P., Murthy, V., and Pehlevan, C. Natural gradient enables fast sampling in spiking neural networks. Advances in Neural Information Processing Systems (NeurIPS), 35, 2022. McNamee, D. C., Stachenfeld, K. L., Botvinick, M. M., and Gershman, S. J. Flexible modulation of sequence generation in the entorhinal–hippocampal system. Nature Neuroscience, 24(6), 2021. McNaughton, B. L., Barnes, C. A., Gerrard, J. L., Gothard, K., Jung, M. W., Knierim, J. J., Kudrimoti, H., Qin, Y., Sk- aggs, W., Suster, M., et al. Deciphering the hippocampal polyglot: the hippocampus as a path integration system. Journal of Experimental Biology, 199(1), 1996. Michelmann, S., Staresina, B. P., Bowman, H., and Hanslmayr, S. Speed of time-compressed forward re- play flexibly changes in human episodic memory. Nature Human Behaviour, 3(2), 2019. Miller, A. M., Jacob, A. D., Ramsaran, A. I., De Snoo, M. L., Josselyn, S. A., and Frankland, P. W. Emergence of a predictive model in the hippocampus. Neuron, 111 (12), 2023. Miller, K. J., Botvinick, M. M., and Brody, C. D. Dorsal hip- pocampus contributes to model-based planning. Nature neuroscience, 20(9), 2017. Milstein, A. D., Tran, S., Ng, G., and Soltesz, I. Offline memory replay in recurrent neuronal networks emerges from constraints on online dynamics. The Journal of Physiology, 601(15):3241–3264, 2023. Miyasawa, K. An empirical Bayes estimator of the mean of a normal population. Bulletin of the International Statistical Institute, 38(4):181–188, 1961. N ́ adasdy, Z., Hirase, H., Czurk ́ o, A., Csicsvari, J., and Buzs ́ aki, G. Replay and time compression of recurring spike sequences in the hippocampus. Journal of Neuro- science, 19(21):9497–9507, 1999. ISSN 0270-6474. Nguyen, T., Baraniuk, R., Bertozzi, A., Osher, S., and Wang, B. MomentumRNN: Integrating momentum into recur- rent neural networks. Advances in Neural Information Processing Systems (NeurIPS), 33, 2020. O’Keefe, J. and Nadel, L. The hippocampus as a cognitive map. Oxford University Press, 1978. Panaretos, V. M. and Zemel, Y. Statistical aspects of wasser- stein distances. Annual Review of Statistics and its Appli- cation, 6(1), 2019. Pavliotis, G. A. Stochastic processes and applications. Texts in Applied Mathematics, 60, 2014. 11 Leakage and Second-Order Dynamics Improve Hippocampal RNN Replay Peyrache, A., Khamassi, M., Benchenane, K., Wiener, S. I., and Battaglia, F. P. Replay of rule-learning related neural patterns in the prefrontal cortex during sleep. Nature Neuroscience, 12(7):919–926, 2009. ISSN 1546-1726. Peyrache, A., Lacroix, M. M., Petersen, P. C., and Buzs ́ aki, G. Internally organized mechanisms of the head direction sense. Nature Neuroscience, 18(4):569–575, 2015. ISSN 1546-1726. Pfeiffer, B. E. The content of hippocampal “replay”. Hip- pocampus, 30(1), 2020. Pfeiffer, B. E. and Foster, D. J. Hippocampal place-cell se- quences depict future paths to remembered goals. Nature, 497(7447), 2013. Recanatesi, S., Farrell, M., Lajoie, G., Deneve, S., Rigotti, M., and Shea-Brown, E. Predictive learning as a network mechanism for extracting low-dimensional latent space representations. Nature Communications, 12(1), 2021. Samsonovich, A. and McNaughton, B. L. Path integration and cognitive mapping in a continuous attractor neural network model. Journal of Neuroscience, 17(15), 1997. Saponati, M. and Vinck, M. Sequence anticipation and spike-timing-dependent plasticity emerge from a predic- tive learning rule. Nature Communications, 14(1), 2023. Seelig, J. D. and Jayaraman, V. Neural dynamics for land- mark orientation and angular path integration. Nature, 521(7551), 2015. Shen, B. and McNaughton, B. L. Modeling the spontaneous reactivation of experience-specific hippocampal cell as- sembles during sleep. Hippocampus, 6(6):685–692, 1996. Skaggs, W. E. and McNaughton, B. L. Replay of neuronal firing sequences in rat hippocampus during sleep follow- ing spatial experience. Science, 271(5257), 1996. Song, Y. and Ermon, S. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems (NeurIPS), volume 32, 2019. Sorscher, B., Mel, G., Ganguli, S., and Ocko, S. A unified theory for the origin of grid cells through the lens of pattern formation. In Wallach, H. M., Larochelle, H., Beygelzimer, A., d’Alch ́ e Buc, F., Fox, E. B., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. Stachenfeld, K. L., Botvinick, M. M., and Gershman, S. J. The hippocampus as a predictive map. Nature neuro- science, 20(11), 2017. Stella, F., Baracskay, P., O’Neill, J., and Csicsvari, J. Hip- pocampal reactivation of random trajectories resembling brownian diffusion. Neuron, 102(2), 2019. Tang, M., Barron, H., and Bogacz, R. Sequential memory with temporal predictive coding. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, 2023. Tang, M., Barron, H., and Bogacz, R. Learning grid cells by predictive coding. arXiv preprint arXiv:2410.01022, 2024. Theodoni, P., Rovira, B., Wang, Y., and Roxin, A. Theta- modulation drives the emergence of connectivity patterns underlying replay in a network model of place cells. eLife, 7:e37388, 2018. ISSN 2050-084X. Tingley, D. and Peyrache, A. On the methods for reacti- vation and replay analysis. Philosophical Transactions of the Royal Society B: Biological Sciences, 375(1799): 20190231, 2020. Tononi, G. and Cirelli, C. Sleep and the price of plastic- ity: from synaptic and cellular homeostasis to memory consolidation and integration. Neuron, 81(1), 2014. Uria, B., Ibarz, B., Banino, A., Zambaldi, V., Kumaran, D., Hassabis, D., Barry, C., and Blundell, C. A model of egocentric to allocentric understanding in mammalian brains. bioRxiv, 2022. Wood, R. A., Bauza, M., Krupic, J., Burton, S., Delekate, A., Chan, D., and O’Keefe, J. The honeycomb maze provides a novel test to study hippocampal-dependent spatial navigation. Nature, 554(7690), 2018. Xu, D., Gao, R., Zhang, W., Wei, X.-X., and Wu, Y. N. On conformal isometry of grid cells: Learning distance- preserving position embedding. In International Confer- ence on Learning Representations (ICLR), 2025. Xu, S., Jiang, W., Poo, M., and Dan, Y. Activity recall in a visual cortical ensemble. Nature Neuroscience, 15(3): 449–455, 2012. ISSN 1546-1726. Yang, L., Zhang, Z., Song, Y., Hong, S., Xu, R., Zhao, Y., Zhang, W., Cui, B., and Yang, M.-H. Diffusion models: A comprehensive survey of methods and applications. ACM Computing Surveys, 56(4), 2023. Zhang, C., Zhang, C., Song, J., Yi, J. S. K., and Kweon, I. S. A survey on masked autoencoder for visual self- supervised learning. In IJCAI, 2023. Zhang, X., Long, X., Zhang, S.-J., and Chen, Z. S. Excitatory-inhibitory recurrent dynamics produce robust visual grids and stable attractors. Cell Reports, 41(11), 2022. 12 Leakage and Second-Order Dynamics Improve Hippocampal RNN Replay Zhao, H., Yang, S., and Fung, C. C. A. Short-term postsy- naptic plasticity facilitates predictive tracking in continu- ous attractors. Frontiers in Computational Neuroscience, 17, 2023. ́ Olafsd ́ ottir, H. F., Barry, C., Saleem, A. B., Hassabis, D., and Spiers, H. J. Hippocampal place cells construct reward related sequences through unexplored space. eLife, 4: e06063, 2015. ISSN 2050-084X. ́ Olafsd ́ ottir, H. F., Bush, D., and Barry, C. The role of hippocampal replay in memory and planning. Current Biology, 28(1):R37–R50, 2018. ISSN 0960-9822. 13 Leakage and Second-Order Dynamics Improve Hippocampal RNN Replay Supplementary Material A. Methods A.1. Experiments A.1.1. ORNSTEIN-UHLENBECK Awake trajectoriess(t)are simulated with∆t = 0.02,σ s = 0.1,σ 0 = 0.2,θ = 2,μ = 5forT = 100iterations. We used Equation (15) as the deterministic component of∆r(t)since a 1D Orstein-Uhlenbeck process admits a closed-form expression for d dr(t) logp(r(t)). In Figure 1, underdampening corresponds toλ v = 0.5, while adaptation corresponds to b a = 1,τ a = 100. A.1.2. T-MAZE AND TRIANGLE Task description.In the 2D T-maze and triangle tasks, we simulate directed random walks (Ornstein-Uhlenbeck processes) along several directions, and train the RNN to path-integrate these walks from their velocities. In the T-maze task, there are two directions of travel: both start from the origin and go up, then one goes left and the other goes right (orange and purple, respectively, in Figure 3). Meanwhile, in the triangle task there are six directions: denoting the three corners of the equilateral triangle asA = (0, 0),B = (1, 0), andC = (1/2, √ 3/2), respectively, these directions are −→ AB, −→ BC, −→ CA, −→ AC, −→ CB, −→ BA , shown in Figure 3 as blue, red, gray, pink, green, and orange, respectively. Awake and replay paths last100timesteps, unless exploration is being measured, in which replay paths are allowed to last 400 timesteps. Architecture and training.For both tasks, we used shallow leaky-ReLU RNNs with linear output projections, as described in Section A.4. In the T-maze task, we used 20 hidden neurons, while in the triangle task, we used 40. We use masking difficultyk = 3in a progressive curriculum. Triangle-task RNNs are trained withk = 1for20, 000epochs, then atk = 2 andk = 3for5, 000epochs each; T-maze-task RNNs are trained likewise, but with fewer epochs (12, 000atk = 1and 5, 000 at k = 2 and k = 3 each). In each task, we train 5 RNNs with different seeds. Hidden state initialization. In each task, multiple directions start from the same point in space (for example, −→ ABand −→ ACin the triangle task), so we add onto initial hidden states some random vectors, orthogonal to the 2D output projection, specific to each direction. For example, hidden states for −→ ABand −→ ACpaths are both initialized to start nearA, but −→ ABpaths also start with a fixed random vectorη AB added to their initial hidden states, whereas −→ AC have a different fixed random vectorη AC added to their initial hidden states. This notion of initializing hidden states in directions orthogonal to output projections has been previously discussed in computational neuroscientific contexts (Churchland & Shenoy, 2024), and recent evidence suggests that memories (which replay is similar to) have uniquely identifiable, output-orthogonal patterns of neural activation (Chettih et al., 2024). A.1.3. RAT PLACE CELL TRAJECTORIES Our unbiased (undirected random walks within a 2D box) and biased (directed random walks that head towards the center of a 2D box) rat place cell trajectory experiments are identical to those of (Krishna et al., 2024), with 5 different ReLU RNN seeds per task, but with two modifications: 1.We add masked training, with difficultiesk = 3in the biased task andk = 6in the unbiased task. We use a higher masking difficulty in the unbiased task to encourage longer replay trajectories; at lower values ofk, unbiased replay paths are much shorter than awake trajectories. In Langevin sampling terms, we conjecture this is because the score function∇ r(t) logp(r(t)) is fairly weak since unbiased trajectories are uniformly distributed 5 . 2.We use a slower, but more detailed, decoder. (Krishna et al., 2024) decode position at a given timestep as the average position of the top 3 most active place cells, which is fast but effectively quantizes decoded positions. We use this method to initialize our positions, but then we optimize our positions via gradient descent to minimize the mean-squared error between observed place cell activity and the place cell activity that would correspond to these optimized positions 5 This relative weakness the unbiased activity score function (especially compared to the biased activity score function, where trajectories all go towards the center of a 2D box) is also why we use relatively small adaptation strengthsb a in simulating replay from unbiased-task RNNs. 14 Leakage and Second-Order Dynamics Improve Hippocampal RNN Replay (this is made possible by our knowledge of the exact place cell activity function). Our procedure takes much longer, but results in truly continuous replay trajectories, avoiding any significant quantization. A.2. Hyperparameters A.2.1. ACTIVATION FUNCTION In our work, we have mostly used RNNs with ReLU or leaky ReLU activation functions (except in Section G); our reasons for doing so are threefold. The first is that ReLU or ReLU-like activations, unlike tanh activations, do not always saturate. This can mitigate vanishing gradients in RNNs (although care must be taken to avoid exploding gradients), but more importantly we have observed that this reduces the strength of fixed attractors learned by path-integrating RNNs, which is essential for exploration—in Section G we observe that replay in saturating RNNs explores less. The second is that, compared to RNNs with multiplicative gating interactions (e.g., GRUs and LSTMs), ReLU RNNs are more biologically plausible insofar as they can be easily interpreted as the nonnegative, sparse firing rates of spiking networks. The third is that most other previous works in sequential predictive coding tend to use ReLU activations, and have shown no significant differences in results when using more complex nonlinearities: •Ali et al. (2022), some of the first authors to report the emergence of predictive representations emerging in RNNs, use ReLU activations. • Krishna et al. (2024) primarily use ReLU networks, and report similar results with GRU networks. •Levenstein et al. (2024) use ReLU activations in conjunction with layer normalization, but report similar results when layer normalization is removed. •Chen et al. (2024) use tanh for must results, but when they care about the emergence of sparse, localized, nonnegative activations (i.e., place cells), they use ReLU activations. • Sorscher et al. (2019); Xu et al. (2025), and Zhang et al. (2022) were all interested in the emergence of grid cells in predictive representations and primarily used ReLU or ReLU-like activations; those that tried using other nonlinearities did not observe notable differences in results. •Tang et al. (2024), who were also interested in grid cells, use both ReLU and tanh activations, and also report no notable differences between the two. • Tang et al. (2023) use linear networks, and notice no significant difference when using tanh activations. A.3. Metrics A.3.1. WASSERSTEIN DISTANCE We seek to compare distributions of trajectories. In most of our tasks, these are objects of dimensionT × 2, whereTis the number of timesteps (recall that T-maze and triangle tasks are already in 2D, while rat trajectories are analyzed after being projected into 2D space). To compare such high-dimensional distributions, we use Wasserstein distances (Panaretos & Zemel, 2019): •In the T-maze and triangle tasks, we first calculate the Wasserstein distance between awake and replay paths belonging to the same direction (e.g., −→ ABin the triangle task). Since paths along the same direction should resemble, if not obey, Gaussian processes, these paths should approximately be normally distributed, and so we can use the closed-form equation for the Wasserstein distance between two normal distributions. After computing Wasserstein distances for each direction, we take the average as our final distance. KL divergence might have also worked in this task, but in practice the covariance matrices of the (T × 2)-dimensional distributions were singular. •In the rat experiments, decoded 2D trajectories are not easily decomposed into groups of Gaussian processes, so instead we apply sliced Wasserstein distances (Bonneel et al., 2015) to compare the(T × 2)-dimensional distributions. Sliced Wasserstein distances are essentially calculated by taking many random projections of two distributions onto 1D, where Wasserstein distances have a closed form. Computing KL divergences instead would have been challenging, since rat path distributions are high-dimensional and do not admit straightforward estimations of probability density. 15 Leakage and Second-Order Dynamics Improve Hippocampal RNN Replay A.3.2. REACH TIMES In the T-maze, triangle, and biased rat tasks, awake trajectories have clearly defined endpoints that they reach. Replay trajectories also aim for these endpoints, although second-order dynamics modifiers like adaptation and underdampening affect how quickly or how closely replay paths reach them. We quantify these changes by measuring the timesteps required for replay paths to get within 10% of their endpoints (for example, the timesteps required for −→ ABpaths to get within a 0.1| −→ AB| of B). A.3.3. PATH LENGTHS One way we quantify exploration in replay paths is through path length. We calculate this simply as the sum of velocity magnitudes. A.3.4. REGIONS VISITED Another way we quantify exploration in the T-maze and traingle tasks is through regions visited. As explained in Figure 7, regions are portions of input space where all points are closest to the same endpoint. In the triangle task, there are 3 endpoints (which all also act as initial points), while in the T-maze there are 2 endpoints and 1 shared starting point, which for the purposes of region assignment we also consider an “endpoint”. Thus, in each task there are 3 endpoints, and thus 3 regions. Awake trajectories go from one region to another and stay there, but replay trajectories might proceed to visit more regions. We define a “visit” as a contiguous presence within a single region for at least 10 timesteps. A.3.5. ADDITIONAL EXPLORATION METRICS For completeness, we add two other measures of exploration: mean displacement, proposed by McNamee et al. (2021), and variance. They are simply defined asE p(r(t)) [|r(t)−r(0)|]andE p(r(t)) [Var(r(t))]. Higher mean displacements and variances generally correlate with increased path length and exploration. We plot them in Figure 10 and Figure 11. A.4. Discretization and Implementation of Noisy RNNs While biological neural networks are continuous-time systems, RNNs are in practice implemented discretely. In order to incorporate second-order processes like adaptation and underdamped sampling in RNNs, we must discretize them, which we do as follows: ∆ e r(t) = f (r(t),u(t),σ r η(t))−r(t)(19) c(t + ∆t) = c(t) + 1 τ a (−c(t) + b a r(t))(20) v(t + ∆t) = (1− λ v )v(t) + ∆ e r(t)(21) r(t + ∆t) = r(t)−c(t) +v(t + ∆t)(22) We train and sample from ReLU RNNs:f (r(t),u(t),σ r η(t)) = κr(t) + ReLU(W r r(t) +W in u(t) + σ r η(t))(where k ∈ [0, 1]is also learnable) unless otherwise stated (a choice justified in Section A.2). In the absence of directed inputsu(t), f (r(t),u(t),σ r η(t)) should be roughly equivalent to the noisy score function from Equation 11. If the adaptation strengthb a = 0, then there is no adaptation (c(t) = 0). As for our friction termλ v ∈ [0, 1], ifλ v = 1, then there is no underdamped sampling sincev(t) = ∆r(t) + σ r η(t)no longer accumulates previous values ofv(t). In other words, the friction term λ v allows for smooth interpolation between overdamped and underdamped sampling. Note that, unlikec(t + ∆t)andv(t + ∆t),r(t + ∆t)depends on another term calculated att + ∆t. This is a symplectic Euler discretization of the second-order dynamics, which we employ for to ensure the stability of interactions betweenr(t) and its momentum v(t). The negative feedback c(t) is much more stable, so its discretization goes unchanged. Across all trained RNN experiments we use τ a = 0,λ v = 1 for awake activity (i.e., training). When generating replay, we always useτ a = 100, and eitherT = 100, 400,or500timesteps, depending on whether we are measuring fidelity and speed (T = 100, like in Figures 3, 4, 5) or exploration (T = 400for T-maze and triangle,T = 500for rat tasks, like in Figures 6, 7). 16 Leakage and Second-Order Dynamics Improve Hippocampal RNN Replay A.5. Deviations from Traditional Langevin Sampling We must note a few minor distinctions between our replay RNNs and traditional Langevin sampling. Stationarity. We treat neural replay as a sequence of events, and are thus interested in the joint distribution of replay activity at all timestepsp r(t) T t=1 . This distribution, however, is not necessarily stationary acrosst. Stationarity would imply that p(r(t)) has no need or intention of traversing along any meaningful path. While a path-integrating RNN does perform gradient ascent alonglogp(r(t)), it does so in a piecewise manner alongtrather than jointly along allt simultaneously. This variation on gradient ascent, in combination with the non-stationarity ofp(r(t)), means that Langevin sampling guarantees do not hold. For this reason, we make modifications to our RNNs that under stationary Langevin sampling theory might seem unprincipled. Underdampening. The friction termλ v applied to the velocityv(t)is subtly different fromγin Equations 3 and 2. λ v ∈ [0, 1]can be interpreted as trying to mapγ ∈ [0,∞)onto[0, 1]. Likeγ →∞,λ v → 1removes any dependency of v(t + ∆t) on v(t). Noise scaling.RNN replay, unlike Langevin sampling, is sensitive to the variance of noise used. We found that omitting the √ 2factor in front ofσ r η(t)worked best. For this reason, we also omittedλ v as a scaling term onσ r η(t). Ifλ v is allowed to modulate √ 2σ r η(t), we found that it is easy to show improvements in replay fidelity whenλ v < 1, but we found these to come from noise scaling rather than from underdampening or momentum as mechanisms. Relation to diffusion models. Throughout this work we treat RNNs as generative models using a variant of Langevin sampling. Expressive generative models, in particular diffusion models whose activity resembles Langevin sampling, have received much attention from the machine learning community (Alemohammad et al., 2024; Luzi et al., 2024; Yang et al., 2023; Croitoru et al., 2023; Li et al., 2022; Gozalo-Brizuela & Garrido-Merchan, 2023; Song & Ermon, 2019). The fundamental differences between our RNNs and diffusion models are twofold. The first is that each timestep of a generated replay trajectory is generated sequentially and with only one effective step along the gradient of log-likelihood, whereas a diffusion model would generate all timesteps of a path simultaneously and with many steps along the gradient of log-likelihood. The second is that diffusion models use time-varying noise levels (annealed dynamics) and are explicitly conditioned on time as an input, whereas our RNNs use the same noise level across time and are never explicitly conditioned on time. B. Overview of Background Throughout the text we make use of notions including sampling, Langevin dynamics, replay, path-integration, awake and quiescent activity, adaptation, and masked training. Here we gather these notions, which have been defined in various previous work, as follows: 1.Our work examines how trained noisy RNNs can act as generative models in the absence of inputs, which prior work including Krishna et al. (2024) consider to be models of replay in neuronal circuits. Accordingly, we define Langevin dynamics, with (Equation (1)) and without (Equations 2 and 3) momentum, as an iterative process by which an agent could attempt to sample from an unknown distributionp(x)using only knowledge of its score function∇ logp(x), which can be learned from noisy observations drawn from the distribution. The following points provide more detail on the sampling process and why Langevin dynamics arise: • Biological networks such as those underlying navigation must accurately estimate environmental state variables such as position from observations of self-motion, even in the presence of intrinsic noise. •This noise induces a distribution over network states, i.e., activity, however, it is unknown exactly what the true noise distribution is. •Accurate state estimation would thus require the network to optimally remove intrinsic noise without access to the exact parameters of the noise distribution, but given noisy network states. •This optimal solution can be accomplished by learning the score function of noisy network states and using it to denoise network states over time as additional inputs and noise are presented (Miyasawa, 1961). • When the network receives no inputs, its noise-driven dynamics still attempt to remove intrinsic noise, but the learned score function is for the distribution of waking task-like activity. This causes the network states to resemble 17 Leakage and Second-Order Dynamics Improve Hippocampal RNN Replay actual samples drawn from the distribution over network states in the presence of inputs. The theoretical results show that the network dynamics during this quiescent state carry out sampling by Langevin dynamics, i.e., using the score function of the network’s activity distribution during task performance to yield quiescent network activity that resembles waking activity. 2.Biological neural circuits like the hippocampus can navigate through environments when awake, and recall navigatory episodes during rest. Accordingly, we summarize the proof from Krishna et al. (2024) that a noisy RNN trained on a biologically relevant navigation task like path-integration can act as a generative model. •Definitions 2.1 and 2.2 simply express that our RNNs have nonlinear dynamics and internal noise, and are trained to estimate a signals(t)by integrating observations ofs ′ (t). This phenomenon, known as path-integration, underlies navigation and is used to estimate one’s position from self-motion cues. •Assumption 2.3 asserts that RNN dynamics can be decomposed into additive components. Assumption 2.4 simply states that the optimal RNN dynamics would lead to accurate path-integration, such that the conditional probability distribution over network states (activity) given environmental states (position) would be Gaussian and identical to the additive noise in our networks. That is, the variability over network states would only be due to intrinsic noise in the system and have the same statistics. Finally, Assumption 2.6 assumes that the RNN dynamics are greedily optimal for path-integration. Greedy optimization is a sensible way of partitioning effort across time in path-integration: the network does the best that it can at each timestep, assuming that at each previous timestep the best possible job has been done. •Theorem 2.7 states that path-integrative RNN dynamics will use the score function of waking activity to remove intrinsic noise from the system and use inputs to update state estimates. This is a two-step greedily optimal solution for path integration in the presence of intrinsic noise: first, intrinsic noise must be removed, following which the inputs must be used to update the estimate of the current environmental state. Theorem 2.8 shows that these dynamics will result in Langevin sampling from the distribution of waking activity in the absence of any inputs, i.e., analogous to sleep. This means that the quiescent network dynamics sample neural states from the same distribution as waking, task-like activity, and can thus represent sequences like those during awake task performance, i.e., leading to replay. 3.Finally, we define existing methods of modulating RNN activity or training that affect RNN replay distributions: adaptation (negative feedback), which encourages diversity in replay paths, and masked training, which encourages coherence in replay. C. Score Functions of Gaussian Distributions For any matrix calculus involved, we use denominator layout. C.1. Multivariate Gaussian Distribution Let’s suppose r ∼N (μ, Σ). If r ∈ R d , then: p(r) = 1 p (2π) d |Σ| exp − 1 2 (r−μ) T Σ −1 (r−μ) (23) logp(r)∝− 1 2 (r−μ) T Σ −1 (r−μ)(24) ∇ r logp(r) =− 1 2 ((Σ −1 ) T + Σ −1 )(r−μ)(25) =−Σ −1 (r−μ)(26) =−σ −2 (r− μ) if r ∈ R(27) C.2. Score Function of r(t) for Gaussian s(t) Recall thatp(r(t)|s(t))∼N (D † s(t), Iσ 2 r ∆t) from Equation 7. If we suppose thats(t)is normally distributed with mean and covariance μ s(t) , Σ s(t) , then we can obtain p(r(t)): p(r(t))∼N (D † μ s(t) , Iσ 2 r ∆t +D † Σ s(t) (D † ) T ),(28) 18 Leakage and Second-Order Dynamics Improve Hippocampal RNN Replay which we can plug into Equation 26 to get∇ r(t) logp(r(t)): ∇ r(t) logp(r(t)) =− Iσ 2 r ∆t +D † Σ s(t) (D † ) T −1 r(t)−D † μ s(t) (29) Moreover, we can use the above score function to calculate the optimal ∆r(t + ∆t) in Equation 10: ∆r ∗ (t + ∆t) = σ 2 r ∆t Iσ 2 r ∆t +D † Σ s(t) (D † ) T −1 −r(t) +D † μ s(t) +D † s ′ (t)∆t + σ r η(t) (30) Some properties of the leakage matrix σ 2 r ∆t Iσ 2 r ∆t +D † Σ s(t) (D † ) T include: 1. The covariance matrix Σ s(t) is positive semidefinite (PSD): all its eigenvalues are≥ 0. 2. D † Σ s(t) (D † ) T is also PSD 6 , symmetric, and therefore diagonalizable. 3. The eigenvalues of (Iσ 2 r ∆t +D † Σ s(t) (D † ) T ) −1 are thus all≤ (σ 2 r ∆t) −1 7 . 4. The eigenvalues of σ 2 r ∆t(Iσ 2 r ∆t +D † Σ s(t) (D † ) T ) −1 are thus all≤ 1. 5. If the off-diagonal entries of the leakage matrix above are sufficiently small in magnitude, then all the diagonal entries should be less than 1, as justified by the Gershgorin Circle Theorem. In fact, if the leakage matrix is diagonal, then it must have all values less than 1 (which could be achieved via sigmoid functions or perhaps spectral normalization). 6.As for interpretation, smaller leakage eigenvalues means higher eigenvalues ofD † Σ s(t) (D † ) T , or essentially, more noise. The maximum determinant of the leakage matrix is 1, when there is essentially no noise in s(t). D. Additional Score Function Results Wiener Processes. One simple stochastic process is the Wiener process, which in terms of navigation represents an undirected random walk (θ = 0). Assumings w (0) = 0, thens w (t) ∼ N (0,σ 2 s t) , and thereforep(r w (t)) ∼ N (0,σ 2 s t + σ 2 r ∆t) from Equation 7, producing the following score: σ 2 r ∆t∇ r w (t) logp(r w (t)) = σ 2 r ∆t −r w (t) σ 2 s t + σ 2 r ∆t (31) Even from a simple Wiener process, we observe that the per-timestep optimal score is not constant with respect tot: at t = 0, it equals−r w (t), while as t approaches∞, it approaches 0. Ornstein-Uhlenbeck Processes. Now we incorporate non-zero leakage (θ > 0) to describe a directed random walk navigating from an arbitrary starting points ou (0)towards a mean destinationμ. Ifp(s ou (0))∼N (0,σ 2 0 ) , thenp(s ou (t))∼ N μ(1− e −θt ), σ 2 s 2θ (1− e −2θt ) + σ 2 0 e −θt , and the score function is: σ 2 r ∆t∇ r ou (t) logp(r ou (t)) = σ 2 r ∆t −(r ou (t)− μ(1− e −θt )) σ 2 s 2θ (1− e −2θt ) + σ 2 0 e −θt + σ 2 r ∆t (32) 6 Proof: If B is PSD, then x T ABA T x = (A T x) T B(A T x) = v T Bv ≥ 0. 7 Proof: For diagonalizableA, thei-th eigenvalue of(λI + A) −1 = (Q(λI +Λ)Q −1 ) −1 is equal to(λ +Λ i ) −1 , which can be no larger than λ −1 if A is PSD. 19 Leakage and Second-Order Dynamics Improve Hippocampal RNN Replay E. Adaptation as a Second-Order Stochastic Differential Equation Let us first combine the following two coupled linear stochastic differential equations into one second-order equation: dX t = (AX t + BY t + M )dt + σdB t , dY t = (CX t + DY t )dt(33) d 2 X t = AdX t + BdY t + σd(dB t )(34) = AdX t + BdY t + σd 2 B t (35) = AdX t + B(CX t + DY t )dt + σd 2 B t , Y t = 1 Bdt (dX t − σdB t − AX t dt− Mdt)(36) = AdX t + BCX t dt + BD 1 Bdt (dX t − σdB t − AX t dt− Mdt)dt + σd 2 B t (37) = AdX t + BCX t dt + D(dX t − σdB t − AX t dt− Mdt) + σd 2 B t (38) = (A + D)dX t + (BC− AD)X t dt− DMdt− σDdB t + σd 2 B t (39) Replacing all variables involved (exceptdt,σ) with matrices and vectors yields the same equation as long asBis invertible: d 2 x t = (A +D)dx t + (BC−AD)x t dt−Dmdt− σDdB t + σd 2 B t (40) For consistency with the notation used throughout the paper, the equation above can be written as: x ′ (t) = (A +D)x ′ (t) + (BC−AD)x(t)−Dm− σDη(t) + ση ′ (t)(41) If we apply the following substitutions from Equations 26 and 16: • x(t)← r(t), • A←−σ 2 r ∆tΣ −1 , • B ←−I , • m← σ 2 r ∆tΣ −1 t μ, • C ←− 1 τ a I , • D ← b a τ a I , then r ′ (t) is: r ′ (t) = b a τ a I − σ 2 r ∆tΣ −1 r ′ (t) + 1 τ a I + b a τ a σ 2 r ∆tΣ −1 r(t) − b a τ a σ 2 r ∆tΣ −1 μ− σ b a τ a η(t) + ση ′ (t) (42) Recall that, ifr(t)follows a stationary Gaussian distribution, then∇ r(t) logp(r(t)) = Σ −1 (−r(t) + μ)(Equation 26), and therefore d 2 dr(t) 2 logp(r(t)) =−Σ −1 . Then, r ′ (t) = b a τ a I + σ 2 r ∆t d 2 dr(t) 2 logp(r(t)) r ′ (t) − b a τ a σ 2 r ∆t∇ r(t) logp(r(t)) + 1 τ a r(t) −σ b a τ a η(t) + ση ′ (t) (43) F. Additional Results Here we present additional results or figures that supplement those of the main text. 20 Leakage and Second-Order Dynamics Improve Hippocampal RNN Replay Overdamped (λ v = 1, b a = 0) Adaptation (λ v = 1, b a = 0.3) Overdamped (λ v = 1, b a = 0) Adaptation (λ v = 1, b a = 5) Underdamped (λ v = 0.7, b a = 0) Underdamped + Adaptation (λ v = 0.7, b a = 0.3) Underdamped (λ v = 0.5, b a = 0) Underdamped + Adaptation (λ v = 0.5, b a = 5) 1 0 −1 1 0 −1 −1 01 −1 01 0.8 0 −0.8 0.8 0 −0.8 −0.8 00.8 −0.8 00.8 Figure 8. Example decoded 2D replay trajectories in unbiased (left) and biased (right) rat tasks. Like in Figure 3, trajectories get darker over time. 00.51 1 0.9 0.8 0.7 Friction λ v T-maze (median) 00.51 1 0.9 0.8 0.7 Triangle (median) 0135 1 0.8 0.6 0.5 Biased rat (median) 00.51 1 0.9 0.8 0.7 Adaptation strength b a Friction λ v T-maze (mean) −300+60+120 00.51 1 0.9 0.8 0.7 Adaptation strength b a Triangle (mean) −300+55+110 0135 1 0.8 0.6 0.5 Adaptation strength b a Biased rat (mean) −600+150+300 Change (%) from awake statistics: Replay reach time (↓) summary statistics Figure 9. This is another version of Figure 5, but now with mean reach times also shown. 21 Leakage and Second-Order Dynamics Improve Hippocampal RNN Replay Figure 10. Underdampening (λ v < 1) increases mean displacement of replay trajectories, especially at early timesteps. Similarly to (McNamee et al., 2021), here we analyze the mean displacement, or distance, of replay trajectories from their starting points as a function of time. For reference, mean displacement over time is also plotted for awake trajectories. In exploration experiments, we simulate replay for4×the duration of awake trajectories. Higher mean displacements over time generally correlate with increased path length and exploration. Note that awake activity is always calculated withb a = 0,λ v = 1, hence why awake statistics are the same across plots within the same task. 22 Leakage and Second-Order Dynamics Improve Hippocampal RNN Replay Figure 11. Underdampening (λ v < 1) increases replay trajectory variances. Like in Figure 10, here we analyze the variance of trajectories at each timestep. Trajectories are 2D, so the variances plotted above are simply the averages of the variances along each coordinate. Underdampening increasing variance is another confirmation that underdampening complements adaptation-induced exploration. 23 Leakage and Second-Order Dynamics Improve Hippocampal RNN Replay 2468 0 10 20 Number of regions visited Number of paths (a) increase when λ v < 1 λ v = 1λ v = 0.7 2468 0 10 20 Number of regions visited (b) decrease when λ v < 1 λ v = 1 λ v = 0.7 2468 0 10 20 Number of regions visited (c) remain constant λ v = 1 λ v = 0.7 OverdampedUnderdampedOverdampedUnderdampedOverdampedUnderdamped 1 0 1 0 1 0 010101010101 During replay, exploration (regions visited) by path can: Figure 12. This is another version of Figure 7, but this time with the specific distributions of regions visited shown as bar plots above pairs of replay trajectory sets. Recall that replay paths start at yellow circles and end at×symbols. In (a), underdampening clearly increases regions visited for several trajectories: instead of just going from(1, 0)to (0, 0), replay paths also continue onwards to traverse between (0, 0)and(1/2, √ 3/2). Meanwhile in (b), underdampening decreases regions visited for many replay trajectories: instead of traversing the whole triangle (4 regions visited), many replay paths stop in the region defined by(1, 0). As for (c), underdampening did not affect regions visited since all paths started at (1/2, √ 3/2), visited (1, 0), and then returned to (1/2, √ 3/2). 24 Leakage and Second-Order Dynamics Improve Hippocampal RNN Replay G. Results With Tanh Activations Throughout all other parts of the text, we have used piecewise-linear (linear or leaky/non-leaky ReLU) activation functions, as defended in Section A.2. For completeness, here we reproduce some key findings of our work with all hyperparameters kept the same, except that RNNs now use tanh activation functions instead of piecewise-linear ones. Replay from tanh RNNs visited far fewer regions than replay from their piecewise-linear counterparts, so a reproduction of Figure 7 (exploration results) is omitted. Our primary finding is that underdampening induces temporal compression, and adaptation induces temporal dilation, in replay from tanh RNNs (Figure 14). 0.81 0 10 Friction λ v Wasserstein distance (↓) T-maze 0.00.51.0 0.81 0.2 0.4 0.6 Friction λ v Triangle 0.60.81 0.08 0.1 0.12 0.14 Friction λ v Biased rat 01 35 10.90.80.7 0.12 0.14 Friction λ v Unbiased rat 0.00.10.20.3 Adaptation strength b a : Figure 13. Mild underdampening can improve replay fidelity in the presence of adaptation in tanh RNNs. This is another version of Figure 4, for T-maze and triangle tasks, but with tanh RNNs. 00.51 1 0.9 0.8 0.7 Friction λ v T-maze (median) 00.51 1 0.9 0.8 0.7 Triangle (median) 0135 1 0.8 0.6 0.5 Biased rat (median) 00.51 1 0.9 0.8 0.7 Adaptation strength b a Friction λ v T-maze (mean) −300+20+40 00.51 1 0.9 0.8 0.7 Adaptation strength b a Triangle (mean) −300+45+90 0135 1 0.8 0.6 0.5 Adaptation strength b a Biased rat (mean) −700 Change (%) from awake statistics: Tanh RNN replay reach time (↓) summary statistics Figure 14. Underdampening increases speed in tanh RNN replay. This is another version of Figure 9 for T-maze and triangle tasks, but now with tanh RNNs. 0.81 5.4 5.6 5.8 Friction λ v Mean path length (↑) T-maze 0.00.51.0 0.81 2.4 2.6 2.8 Friction λ v Triangle 0.60.81 20 25 30 Friction λ v Biased rat 0135 0.80.91 6 8 10 Friction λ v Unbiased rat 0.00.10.20.3 Adaptation strength b a : Figure 15. In tanh RNNs, underdampening may increase or decrease exploration via path length. This is an alternate version of Figure 6 but with tanh RNNs. Underdampening has mixed effects on tanh RNN replay path length because of tanh saturation. 25