Paper deep dive

Learning to Forget: Sleep-Inspired Memory Consolidation for Resolving Proactive Interference in Large Language Models

Ying Xie

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 63

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%

Last extracted: 3/22/2026, 5:10:08 AM

Summary

SleepGate is a biologically inspired framework for transformer-based LLMs that addresses proactive interference (PI) by implementing a learned sleep cycle over the key-value (KV) cache. It utilizes three mechanisms—a conflict-aware temporal tagger, a forgetting gate, and a consolidation module—to selectively evict or compress stale cache entries, reducing the interference horizon from O(n) to O(log n).

Entities (6)

SleepGate · framework · 100%Proactive Interference · phenomenon · 99%Conflict-Aware Temporal Tagger · module · 95%Consolidation Module · module · 95%Forgetting Gate · module · 95%KV cache · component · 95%

Relation Signals (4)

SleepGate → resolves → Proactive Interference

confidence 98% · SleepGate, a biologically inspired framework that augments transformer-based LLMs with a learned sleep cycle over the key-value (KV) cache... for Resolving Proactive Interference

SleepGate → includes → Conflict-Aware Temporal Tagger

confidence 95% · SleepGate introduces three mechanisms: (1) a conflict-aware temporal tagger

SleepGate → includes → Forgetting Gate

confidence 95% · SleepGate introduces... (2) a lightweight forgetting gate

SleepGate → includes → Consolidation Module

confidence 95% · SleepGate introduces... (3) a consolidation module

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Large language models (LLMs) suffer from proactive interference (PI): outdated information in the context window disrupts retrieval of current values. This interference degrades retrieval accuracy log-linearly as stale associations accumulate, a bottleneck that persists regardless of context length and resists prompt-engineering mitigations. Biological brains resolve an analogous challenge through sleep-dependent memory consolidation: synaptic downscaling, selective replay, and targeted forgetting. We propose SleepGate, a biologically inspired framework that augments transformer-based LLMs with a learned sleep cycle over the key-value (KV) cache. SleepGate introduces three mechanisms: (1) a conflict-aware temporal tagger detecting when new entries supersede old ones; (2) a lightweight forgetting gate trained to selectively evict or compress stale cache entries; and (3) a consolidation module that merges surviving entries into compact summaries. These components activate periodically during inference in sleep micro-cycles, governed by an adaptive entropy-based trigger. We formalize a dual-phase training objective jointly optimizing language modeling during the wake phase and post-consolidation retrieval during the sleep phase. Theoretical analysis shows SleepGate reduces the interference horizon from O(n) to O(log n). In experiments with a small-scale transformer (4 layers, 793K parameters), SleepGate achieves 99.5% retrieval accuracy at PI depth 5 and 97.0% at depth 10, while all five baselines -- full KV cache, sliding window, H2O, StreamingLLM, and decay-only ablation -- remain below 18%. Our framework offers an architecture-level solution that prompt engineering cannot address.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

62,687 characters extracted from source content.

Expand or collapse full text

Learning to Forget: Sleep-Inspired Memory Consolidation for Resolving Proactive Interference in Large Language Models Ying Xie Kennesaw State University yxie2@kennesaw.edu Abstract Large language models (LLMs) suffer from proactive interference (PI): previously processed but now-outdated information in the context window disrupts retrieval of current, relevant values. Recent work demonstrates that this interference degrades retrieval accuracy log-linearly toward chance as stale associations accumulate—a working-memory bottleneck that persists regardless of context length and resists prompt-engineering mitigations. Biological brains face an analogous challenge and resolve it through sleep-dependent memory consolidation: an active, multi-stage process of synaptic downscaling, selective replay, and targeted forgetting. We propose SleepGate, a biologically inspired framework that augments transformer-based LLMs with a learned sleep cycle operating over the key-value (KV) cache. SleepGate introduces three coordinated mechanisms: (1) a conflict-aware temporal tagger that detects when new entries supersede old ones; (2) a lightweight forgetting gate network trained to selectively evict or compress stale cache entries; and (3) a consolidation module that merges related surviving entries into compact summary representations. These components are activated periodically during inference in “sleep micro-cycles,” governed by an adaptive trigger based on attention entropy. We formalize a dual-phase training objective that jointly optimizes standard language modeling during the wake phase and post-consolidation retrieval accuracy during the sleep phase, with explicit pressure toward cache compression. We provide theoretical analysis showing that SleepGate can reduce the effective interference horizon from O(n)O(n) to O(log⁡n)O( n) under mild assumptions, present a concrete algorithmic specification, and provide preliminary experimental validation on a controlled PI-LLM benchmark. In proof-of-concept experiments with a small-scale transformer (4 layers, 793K parameters), SleepGate achieves 99.5% retrieval accuracy at PI depth 5 and 97.0% at depth 10, while all five baselines—full KV cache, sliding window, H2O, StreamingLLM, and a decay-only ablation—remain below 18% across all depths. Our framework offers an architecture-level solution to a limitation that prompt engineering cannot address. Keywords: large language models, proactive interference, working memory, memory consolidation, KV cache management, active forgetting, sleep-inspired computation 1 Introduction The dominant paradigm for extending the memory capacity of large language models has been to increase context window size substantially in successive generations of models (Radford et al., 2019; Anthropic, 2024; OpenAI, 2023; Gemini Team et al., 2024). The implicit assumption is that if a model can attend to more tokens, it can use more information. However, accumulating evidence suggests that this assumption is flawed. Models exhibit systematic retrieval failures even when target information lies well within the context window (Liu et al., 2024; Hsieh et al., 2024), and performance degrades not just with distance but with the amount of competing information. Wang & Sun (2025) recently provided a clear demonstration of this phenomenon through the lens of proactive interference (PI), a well-established construct in cognitive psychology (Underwood, 1957; Wickens, 1970). In their PI-LLM paradigm, models receive a stream of semantically related key-value pairs where later entries overwrite earlier ones, and are then queried on the final (most recent) value. Despite the target answer being positioned immediately before the query, retrieval accuracy declines log-linearly toward zero as the number of prior (now-superseded) associations increases. The errors are not random—models systematically retrieve overwritten values, demonstrating that stale information actively competes with and suppresses current information. Prompt-engineering interventions (e.g., instructing the model to ignore prior values) provide only marginal relief. This finding reveals a working memory bottleneck that is fundamentally distinct from context length limitations. The model can see the relevant tokens—the difficulty is suppressing the irrelevant ones. In the standard transformer attention mechanism, every entry in the key-value cache participates in every attention computation with no mechanism for selective inhibition. As stale entries accumulate, they collectively drown out the signal from current, relevant entries through sheer numerical mass. 1.1 The Biological Precedent: Sleep as Active Forgetting Biological neural systems face the same challenge. During waking hours, the brain continuously encodes new associations, many of which conflict with or supersede earlier ones. Left unchecked, this accumulation would produce catastrophic interference (McCloskey & Cohen, 1989; French, 1999). The brain’s solution is sleep—not as passive downtime, but as an active computational process dedicated to memory management. The synaptic homeostasis hypothesis (SHY) (Tononi & Cirelli, 2006, 2014) proposes that waking experience produces a net increase in synaptic strength, and that sleep restores homeostasis through global synaptic downscaling: all synapses are proportionally weakened, preserving relative strength differences while reducing absolute levels. This prevents saturation and improves signal-to-noise ratios. Complementing this global process, the brain engages in selective memory replay during sleep, particularly during slow-wave sleep (SWS) (Diekelmann & Born, 2010; Rasch & Born, 2013). The hippocampus replays recently encoded experiences to the neocortex, promoting consolidation of important memories while allowing non-replayed traces to decay. Sleep has been shown to reduce vulnerability to associative interference and protect recently formed memories (Ellenbogen et al., 2006), consistent with its broader role in memory consolidation and plasticity (Abel et al., 2013). Active forgetting mechanisms further supplement these processes. Neuromodulatory and circuit-level processes can contribute to the weakening of memory traces and adaptive forgetting (Berry & Davis, 2014; Davis & Zhong, 2017), while sleep-specific neural oscillations such as spindles and sharp-wave ripples help coordinate memory consolidation-related information transfer during sleep (Staresina et al., 2015). 1.2 Our Proposal We propose to endow LLMs with an analogous capability: a learned, periodic sleep cycle that actively manages the key-value cache. Our framework, SleepGate, operates at the architectural level—modifying how the model maintains its working memory instead of relying on prompting or post-hoc filtering. The contributions of this paper are: 1. A biologically grounded framework for active KV cache management that maps the three core mechanisms of sleep-dependent memory consolidation (synaptic downscaling, selective replay, active forgetting) onto concrete computational modules (§3). 2. A formal dual-phase training objective that jointly optimizes language modeling performance and post-consolidation retrieval accuracy, with explicit compression pressure (§4). 3. Theoretical analysis showing that the proposed mechanism can reduce the effective PI horizon from linear to logarithmic in the number of superseding updates (§5). 4. Preliminary experimental validation on a controlled PI-LLM benchmark demonstrating that SleepGate outperforms all baselines by a wide margin across seven PI depths, with analysis of failure modes at extreme interference levels (§6, §7). 2 Related Work Context Window Limitations and the Lost-in-the-Middle Effect. Liu et al. (2024) demonstrated that LLMs struggle to retrieve information from the middle of long contexts, even when the information is present. Subsequent work on long-context benchmarks (Hsieh et al., 2024; Bai et al., 2024) has shown that scaling context length does not proportionally improve downstream task performance. These findings point to the need for mechanisms beyond simple context extension. Proactive Interference in Human Cognition. PI is one of the most robust phenomena in memory research (Underwood, 1957). The buildup-release paradigm of Wickens (1970) showed that PI accumulates with similar material and is released by categorical shifts. Kane & Engle (2000) linked PI susceptibility to working memory capacity, and Ellenbogen et al. (2006) demonstrated that sleep reduces susceptibility to associative interference and protects recently encoded memories. Proactive Interference in LLMs. Wang & Sun (2025) introduced the PI-LLM paradigm and demonstrated log-linear accuracy degradation under PI across multiple model families. Their work established that this is an architectural bottleneck, not a training data or prompting issue. Our work builds directly on this finding by proposing an architectural solution. KV Cache Optimization. Several lines of work address KV cache efficiency. Local or sliding-window attention (Beltagy et al., 2020) limits attention to a fixed window but discards information indiscriminately. Other sparse attention mechanisms use structured patterns to reduce attention cost (Child et al., 2019). H2O (Zhang et al., 2023) retains “heavy hitter” tokens based on cumulative attention scores. StreamingLLM (Xiao et al., 2024) maintains attention sinks plus a sliding window. Ge et al. (2024) propose model-driven cache compression. These approaches optimize for efficiency (reducing memory footprint) rather than interference resolution (selectively removing stale information that degrades accuracy). SleepGate addresses the latter while naturally achieving the former as a side effect. Memory-Augmented Architectures. External memory systems (Graves et al., 2014; Sukhbaatar et al., 2015; Wu et al., 2022) and retrieval-augmented generation (Lewis et al., 2020; Borgeaud et al., 2022) extend model memory beyond the context window. However, these systems typically add memory capacity without addressing the interference problem within the existing context. Munkhdalai et al. (2024) propose leaving context behind via a compressive memory, which is complementary to our approach. Biological Inspiration in Machine Learning. Sleep-inspired computation has been explored in continual learning (Tadros et al., 2022; González et al., 2020), where "sleep" phases involving generative replay help consolidate knowledge across tasks. Kumaran et al. (2016) revisit complementary learning systems theory, showing how hippocampal–neocortical interactions can mitigate interference. Our work differs in targeting in-context interference during inference rather than cross-task interference during training. Forgetting Mechanisms. Sparse attention (Tay et al., 2022), gated retention (Sun et al., 2023), and state-space models (Gu & Dao, 2023) implicitly implement forms of forgetting through their inductive biases. However, these are fixed architectural choices rather than learned, content-dependent forgetting policies. The forgetting gate in LSTM (Hochreiter & Schmidhuber, 1997) is the closest architectural precedent to our proposal, but operates at the hidden-state level within a single timestep, not over an extended KV cache. 3 The SleepGate Framework We describe SleepGate in detail. The framework augments a standard transformer-based LLM with three modules that operate over the KV cache, orchestrated by an adaptive scheduling mechanism. Figure 1 provides an overview. InputTokensTransformerTaggedKV CacheSleep?UpdatedKV CacheTaggerDecayGateConsolidatenoyesWake phaseSleep micro-cycle Figure 1: Simplified overview of SleepGate. During the wake phase, the model processes input tokens while maintaining a tagged KV cache. A sleep trigger decides whether to continue normal inference or enter a sleep micro-cycle. In the soft biasing variant used in our experiments, the cache passes through decay and the forgetting gate, and the resulting retention scores are converted to additive attention biases (Eq. 10) for a second forward pass. In the hard eviction variant, entries additionally pass through consolidation before eviction. 3.1 Preliminaries and Notation Consider a transformer model processing a sequence of tokens =(x1,…,xT)x=(x_1,…,x_T). At each layer ℓ and head h, the model maintains a KV cache ℓ,h=(i,i)i=1tC ,h=\(k_i,v_i)\_i=1^t where i,i∈ℝdk_i,v_i ^d are the key and value vectors for position i, and t≤Tt≤ T is the current position. The attention output for query tq_t is: Attn(t,)=∑i=1texp⁡(t⊤i/d)∑j=1texp⁡(t⊤j/d)iAttn(q_t,C)= _i=1^t (q_t k_i/ d) _j=1^t (q_t k_j/ d)v_i (1) In the PI setting of Wang & Sun (2025), the sequence contains n updates to the same semantic key, each with a different value: (k,v1),(k,v2),…,(k,vn)(k,v_1),(k,v_2),…,(k,v_n), and the model is queried for vnv_n. All prior entries (k,v1),…,(k,vn−1)(k,v_1),…,(k,v_n-1) are stale and constitute interference. 3.2 Module 1: Conflict-Aware Temporal Tagger The first module augments each KV cache entry with metadata that enables downstream conflict detection and staleness identification. We extend the standard cache to an augmented cache: +=(i,i,τi,i,σi,ai)i=1tC^+= \ (k_i,v_i, _i,s_i, _i,a_i ) \_i=1^t (2) where τi∈ℕ _i is the position timestamp, i∈ℝdss_i ^d_s is a semantic signature vector, σi∈0,1 _i∈\0,1\ is a binary superseded flag, and ai∈ℝ≥0a_i _≥ 0 is the cumulative attention received by entry i. Semantic Signatures. The semantic signature is_i captures what “slot” or “entity” the entry refers to, abstracting away the specific value. We compute it via a lightweight projection of the key vector combined with local context: i=LayerNorm(Ws[i∥LocalPool(jj=max⁡(1,i−w)min⁡(t,i+w))])s_i=LayerNorm\! (W_s [k_i\,\|\,LocalPool (\k_j\_j= (1,i-w) (t,i+w) ) ] ) (3) where Ws∈ℝds×(d+d)W_s ^d_s×(d+d) is a learned projection, LocalPool averages keys in a window of size 2w+12w+1, and ∥\|\, denotes concatenation. Conflict Detection. An entry i is marked as potentially superseded (σi=1 _i=1) when a later entry has sufficiently high semantic similarity: σi=[∃j>i:cos(i,j)>δ] _i=1\! [∃\,j>i: (s_i,s_j)>δ ] (4) where δ is a learned or tuned threshold. In practice, we maintain a running set of active semantic signatures and check incoming entries against it, achieving O(1)O(1) amortized cost per token using locality-sensitive hashing (Indyk & Motwani, 1998). 3.3 Module 2: Forgetting Gate The forgetting gate Gθ:ℝdg→[0,1]G_θ:R^d_g→[0,1] is a small neural network that assigns a retention score to each cache entry, determining whether it should be kept, compressed, or evicted. This is the core “active forgetting” mechanism, analogous to the selective synaptic downscaling and dopaminergic erasure processes observed during sleep. For each entry i in the augmented cache, we compute an input feature vector: i=[i‖i‖PE(τi,t)‖i‖σi‖ai‖¯]f_i= [k_i\,\|\,v_i\,\|\,PE( _i,t)\,\|\,s_i\,\|\, _i\,\|\,a_i\,\|\, c ] (5) where PE(τi,t)PE( _i,t) is a relative positional encoding capturing the age of the entry, aia_i is the cumulative attention received by this entry (analogous to the “heavy hitter” score of Zhang et al. (2023)), and ¯ c is a global context summary (mean-pooled over recent entries). The retention score and resulting action are: ri=Gθ(i),actioni=Keepif ri≥αkCompressif αe≤ri<αkEvictif ri<αer_i=G_θ(f_i), _i= cases Keep&if r_i≥ _k\\ Compress&if _e≤ r_i< _k\\ Evict&if r_i< _e cases (6) where αk>αe _k> _e are learned or tuned thresholds. Architecture of GθG_θ. To minimize overhead, GθG_θ is implemented as a 2-layer MLP with GeLU activations and a sigmoid output: ri=Gθ(i)=sigmoid(wr⊤GeLU(W1i+b1)+br)r_i=G_θ(f_i)=sigmoid\! (w_r GeLU(W_1f_i+b_1)+b_r ) (7) with W1∈ℝdh×dgW_1 ^d_h× d_g, wr∈ℝdhw_r ^d_h, and hidden dimension dh=128d_h=128. The total parameter count is negligible relative to the base model (typically <0.01%<0.01\%). Differentiable Training. For differentiable training of the discrete keep/compress/evict decision, we add a 3-way action head on top of the shared hidden representation and apply a Gumbel-softmax relaxation: i=WaGeLU(W1i+b1)+ba∈ℝ3z_i=W_a\,GeLU(W_1f_i+b_1)+b_a ^3 (8) ^i=softmax(i+iτtemp) y_i=softmax\! ( z_i+g_i _temp ) (9) where iz_i denotes the logits for the keep, compress, and evict actions, i∈ℝ3g_i ^3 is a sampled Gumbel noise vector whose components are independently drawn from a Gumbel distribution, and τtemp _temp is the temperature. At inference time, we apply the hard decision rule in Eq. 6. When using soft attention biasing (below), only the scalar retention score rir_i is needed rather than a 3-way action distribution. In this case, the Gumbel-softmax relaxation simplifies to a Gumbel-sigmoid (binary concrete) relaxation applied directly to the gate output: ri=σ((ℓi+gi)/τtemp)r_i=σ\! (( _i+g_i)/ _temp ), where ℓi _i is the scalar logit from Eq. 7 (before the sigmoid), gi∼Gumbel(0,1)g_i (0,1), and σ(⋅)σ(·) denotes the sigmoid function. Soft Attention Biasing. Rather than applying hard eviction decisions at inference time, we introduce a soft attention biasing mechanism that uses the retention scores as continuous modifiers of the attention computation. For each cache entry i, we compute an additive pre-softmax bias: bi=β⋅log⁡(max⁡(ri,ε))b_i=β· \! ( (r_i, ) ) (10) where β>0β>0 is a scale hyperparameter and ε is a small constant for numerical stability. The modified attention becomes: Attnsleep(t,)=∑i=1texp⁡(t⊤i/d+bi)∑j=1texp⁡(t⊤j/d+bj)iAttn_sleep(q_t,C)= _i=1^t \! (q_t k_i/ d+b_i ) _j=1^t \! (q_t k_j/ d+b_j )v_i (11) Since ri≈0r_i≈ 0 for stale entries yields bi≪0b_i 0, their attention weights are exponentially suppressed without being removed entirely. Because the bias is continuous, this approach requires no Gumbel-softmax relaxation during joint training and no threshold calibration stage. It also degrades gracefully: entries are downweighted, not deleted, so the model can recover from gate errors. In practice, setting β=5β=5 provides sufficient suppression: an entry with ri=0.01r_i=0.01 receives a bias of bi≈−23b_i≈-23, effectively zeroing its attention weight. 3.4 Module 3: Consolidation Module Entries assigned the Compress action are not simply discarded but are consolidated into compact summary representations, analogous to hippocampal replay transferring episodic memories into semantic knowledge. We group entries marked for compression into clusters based on their semantic signatures using a simple greedy algorithm: entry i joins the cluster of the most similar active entry, or starts a new cluster if no similarity exceeds δ/2δ/2. For each cluster m=i1,…,i|m|S_m=\i_1,…,i_|S_m|\, we produce a consolidated key-value pair: m∗=∑i∈mrii∑i∈mri+ε,m∗=∑i∈mαi(m)WV′ik_m^*= _i _mr_ik_i _i _mr_i+ , _m^*= _i _m _i^(m)W_V v_i (12) where ε>0 >0 is a small constant for numerical stability. To preserve the most useful information within each cluster, we compute recency-biased attention weights using a learned query vector latestq_latest: αi(m)=exp⁡(latest⊤WK′i/d+η⋅τ~i)∑j∈mexp⁡(latest⊤WK′j/d+η⋅τ~j) _i^(m)= \! (q_latest W_K k_i/ d\;+\;η· τ_i ) _j _m \! (q_latest W_K k_j/ d\;+\;η· τ_j ) (13) where WK′W_K and WV′W_V are learned projection matrices, τ~i=τi/maxj⁡τj τ_i= _i/ _j _j is the normalized timestamp, and η>0η>0 is a recency weight (set to η=2η=2 in our implementation). This operation compresses |m||S_m| entries into a single entry, yielding a compression ratio of |m|:1|S_m|:1. Because latestq_latest is biased toward recent entries, the consolidated representation tends to preserve the most recent value within the cluster, which mitigates proactive interference. 3.5 Sleep Trigger: Adaptive Scheduling The sleep micro-cycle must be triggered at the right frequency: too often incurs unnecessary overhead; too rarely allows interference to accumulate. We propose an adaptive trigger based on two complementary signals: Signal 1: Attention Entropy. When PI accumulates, attention distributions become more uniform (the model “doesn’t know where to look”). We monitor the average attention entropy across heads: Ht=−1|ℋ|∑h∈ℋ∑i∈t+αt,i(h)log⁡αt,i(h)H_t=- 1|H| _h _i _t^+ _t,i^(h) _t,i^(h) (14) where αt,i(h) _t,i^(h) is the attention weight from position t to i at head h. A sleep cycle is triggered when HtH_t exceeds a running threshold H¯+κ⋅std(H) H+κ·std(H). Signal 2: Conflict Density. We also monitor the fraction of current cache entries marked as superseded: ρt=1|t+|∑i∈t+σi _t= 1|C_t^+| _i _t^+ _i (15) A sleep cycle is triggered when ρt>ρmax _t> _ . The overall trigger condition is: trigger(t)=(Ht>H¯+κ⋅std(H))∨(ρt>ρmax)∨(tmodNmax=0)trigger(t)= (H_t> H+κ·std(H) )\; \; ( _t> _ )\; \; (t N_ =0 ) (16) where the last term provides a fallback periodic trigger every NmaxN_ tokens. 3.6 The Complete Sleep Micro-Cycle Algorithm 1 specifies the complete procedure. Algorithm 1 Sleep Micro-Cycle 0: Augmented KV cache +C^+, forgetting gate GθG_θ, thresholds αk,αe _k, _e, decay rate λ, bias scale β 0: Updated cache ′⁣+C + (hard variant) or attention bias b (soft variant) 1: // Phase 1: Key Decay (Synaptic Downscaling) 2: for each entry (i,i,τi,i,σi,ai)∈+(k_i,v_i, _i,s_i, _i,a_i) ^+ do 3: agei←tcurrent−τiage_i← t_current- _i 4: i←i⋅(1+agei)−λk_i _i·(1+age_i)^-λ Log-scale key decay 5: end for 6: // Phase 2: Forgetting Gate (Active Forgetting) 7: for each entry i∈+i ^+ do 8: Compute feature vector if_i via Eq. 5 9: ri←Gθ(i)r_i← G_θ(f_i) 10: if ri<αer_i< _e then 11: Mark entry i for Eviction 12: else if ri<αkr_i< _k then 13: Mark entry i for Compression 14: end if 15: end for 16: // — Soft Biasing Variant (used in experiments) — 17: for each entry i∈+i ^+ do 18: bi←β⋅log⁡max⁡(ri,ε)b_i←β· \! (r_i, ) Eq. 10 19: end for 20: Re-run attention with additive bias b (Eq. 11) 21: return attention bias vector b 22: // — Hard Eviction Variant (Phases 3–5) — 23: // Phase 3: Consolidation (Memory Replay) 24: Cluster Compress-marked entries by semantic signature 25: for each cluster mS_m do 26: Compute (m∗,m∗)(k_m^*,v_m^*) via Eq. 12 27: Replace cluster entries with consolidated entry 28: end for 29: // Phase 4: Eviction 30: Remove all Evict-marked entries from +C^+ 31: // Phase 5: Renormalization 32: Recompute derived metadata for surviving entries 33: return +C^+ 4 Training Objective SleepGate is trained with a dual-phase objective that optimizes both standard language modeling and post-consolidation retrieval: ℒtotal=ℒwake+λsℒsleep+λcℒcompress+λgℒalignL_total=L_wake+ _sL_sleep+ _cL_compress+ _gL_align (17) Wake Loss. The standard autoregressive language modeling loss, unchanged from the base model: ℒwake=−∑t=1Tlog⁡p(xt∣x<t)L_wake=- _t=1^T p(x_t x_<t) (18) Sleep Loss. After each sleep micro-cycle, we evaluate retrieval accuracy on current (non-superseded) key-value associations: ℒsleep=−∑(k,v)∈ℳcurrentlog⁡p(v∣k,′⁣+)L_sleep=- _(k,v) _current p(v k,C +) (19) where ℳcurrentM_current is the set of active (most recent) associations and ′⁣+C + is the post-consolidation cache. Here, p(v∣k,′⁣+)p(v k,C +) denotes the model’s predictive distribution over the current value associated with key k when querying the post-consolidation cache ′⁣+C +. This directly trains the system to retain current information after forgetting stale information. Compression Loss. To encourage cache efficiency, we penalize the expected fraction of the cache retained. Under the 3-way Gumbel-softmax relaxation (Eq. 9), this takes the form 1|+|∑i(y^i,Keep+γy^i,Compress) 1|C^+| _i( y_i, Keep+γ y_i, Compress). When using soft attention biasing with scalar retention scores, this simplifies to the mean retention: ℒcompress=1|+|∑i∈+riL_compress= 1|C^+| _i ^+r_i (20) Since ri≈1r_i≈ 1 for entries the gate wishes to keep and ri≈0r_i≈ 0 for entries it wishes to suppress, this loss directly penalizes retaining too large a fraction of the cache. The coefficient λc _c controls the efficiency–accuracy trade-off. Gate Alignment Loss. To provide direct supervision to the forgetting gate beyond the end-to-end language modeling signal, we encourage the gate’s retention scores to align with the tagger’s supersession labels: ℒalign=−1|+|∑i∈+[(1−σi)log⁡ri+σilog⁡(1−ri)]L_align=- 1|C^+| _i ^+ [(1- _i) r_i+ _i (1-r_i) ] (21) where σi _i is the binary superseded flag from the tagger. This binary cross-entropy objective trains the gate to assign high retention to current entries (σi=0 _i=0) and low retention to superseded entries (σi=1 _i=1). We set λg=0.3 _g=0.3 in our experiments. 4.1 Training Data Generation We construct training sequences specifically designed to exercise the sleep mechanism, inspired by the PI-LLM paradigm: 1. PI Sequences: Streams of key-value pairs where the same key is updated n times (n∈[2,50]n∈[2,50]), followed by retrieval queries. The model must learn to evict prior values and retain only the latest. 2. Mixed-Relevance Sequences: Interleaved important and unimportant information, where “importance” is defined by whether the information is queried later. The model must learn to predict and retain information that will be needed. 3. Multi-Entity Sequences: Multiple entities with independent update streams, requiring the model to maintain and selectively prune per-entity state. 4. Natural Text with Synthetic Updates: Real documents augmented with “correction” statements (e.g., “The previous figure was wrong; the actual revenue is $4.2B”), testing whether the model learns to supersede naturalistic corrections. 4.2 Curriculum Training Strategy We employ a multi-stage curriculum: 1. Stage 0 — Base Model Warm-Start (∼ 22% of training): Pre-train the base transformer on PI sequences with standard autoregressive loss, with all sleep modules disabled. This establishes basic sequence modeling capability before introducing the sleep mechanism, and ensures a fair comparison since baselines receive the same total training budget. 2. Stage 1 — Gate Pre-training (∼ 11% of training): Train the forgetting gate GθG_θ in isolation on PI sequences with ground-truth supersession labels using binary cross-entropy loss. The gate learns to identify stale entries before being coupled with the rest of the system. 3. Stage 2 — Joint Training with Soft Biasing (∼ 67% of training): Train all components end-to-end with the full objective (Eq. 17). Using the soft attention biasing mechanism (Eq. 10), every forward pass computes both unbiased (wake) and soft-biased (sleep) logits. The wake loss is computed on unbiased predictions, the sleep loss on biased predictions, and a gate alignment loss keeps the gate consistent with the tagger’s conflict signal. PI depth is increased via a curriculum that gradually exposes the model to deeper interference. 4. Stage 3 — Threshold Calibration (optional): When using hard eviction (Eq. 6), this stage fine-tunes the thresholds αk _k, αe _e, δ, and ρmax _ on held-out validation sequences. When using soft attention biasing, this stage can be omitted as the soft bias is self-calibrating. 5 Theoretical Analysis We analyze how SleepGate affects the PI accumulation dynamics observed by Wang & Sun (2025). 5.1 PI Accumulation Without SleepGate Consider a sequence with n updates to the same key: (k,v1),…,(k,vn)(k,v_1),…,(k,v_n). At query time, the attention weight assigned to the correct (most recent) value vnv_n relative to a stale value vjv_j depends on the dot-product similarity between the query and each key. Since all entries share the same semantic key k, the attention weights are approximately uniform across the n entries, giving: p(retrieve vn)≈1np(retrieve v_n)≈ 1n (22) This explains the roughly 1/n1/n accuracy decay observed empirically. Taking the log: log⁡p≈−log⁡n p≈- n, which is the log-linear relationship reported by Wang & Sun (2025). 5.2 PI Accumulation With SleepGate With SleepGate, after each sleep cycle, entries marked as superseded are evicted. Suppose a sleep cycle is triggered after every N new tokens, and the forgetting gate correctly identifies stale entries with probability pcorrectp_correct. Theorem 1 (Interference Reduction). Under the assumptions that (i) the forgetting gate identifies superseded entries with probability pc≥1−ϵp_c≥ 1-ε for ϵ<1ε<1, and (i) sleep cycles occur at intervals of N tokens, the expected number of stale entries competing with the current value after n updates is bounded by: [stale entries]≤min⁡(N,ϵ⋅n1−(1−pc)n/N)=O(max⁡(N,ϵ⋅n))E[stale entries]≤ \! (N,\; ε· n1-(1-p_c)^n/N )=O\! ( (N,ε· n) ) (23) For pcp_c close to 1 (low ϵε) and moderate N, this is O(N)O(N) regardless of n—a constant rather than a linear function of the number of updates. Proof sketch. Between consecutive sleep cycles, at most N new entries can accumulate. Each sleep cycle evicts each stale entry independently with probability pcp_c. After ⌊n/N⌋ n/N sleep cycles, the probability that a specific stale entry from the first update survives is (1−pc)⌊n/N⌋(1-p_c) n/N , which decays exponentially. Summing over all n stale entries and taking the expectation gives the bound. ∎ Corollary 1. The retrieval probability for the current value under SleepGate is: p(retrieve vn)≥11+O(N)p(retrieve v_n)≥ 11+O(N) (24) which is a constant independent of n, eliminating the log-linear degradation. 5.3 Compression Ratio Analysis Proposition 1 (Cache Size Bound). If the consolidation module achieves a per-cluster compression ratio of c≥2c≥ 2 and the fraction of entries evicted per cycle is fef_e, the steady-state cache size is bounded by: |′⁣+|≤Nfe+(1−fe)(1−1/c)|C +|≤ Nf_e+(1-f_e)(1-1/c) (25) For typical values (fe=0.3f_e=0.3, c=4c=4), this gives |′⁣+|≤1.67N|C +|≤ 1.67N, a substantial reduction from the unbounded growth of the standard KV cache. 6 Experimental Design We validate SleepGate through a controlled proof-of-concept experiment on synthetic PI sequences, using a small-scale transformer trained from scratch. This controlled setting isolates the effect of the sleep mechanism from confounds present in large pre-trained models (e.g., memorized associations, instruction-following biases). 6.1 Model and Scale We train a 4-layer causal transformer with dmodel=128d_model=128, 4 attention heads, feed-forward dimension dff=256d_f=256, and max_seq_len=1024max\_seq\_len=1024. The vocabulary size is 1024 (synthetic tokens). Table 1 summarizes the parameter counts. Table 1: Model parameter breakdown. Component Parameters Description Base transformer 793,344 Embedding + 4 layers + LM head Temporal tagger 16,576 Semantic signature projection Forgetting gate 74,241 2-layer MLP (retention scorer) Consolidation 33,152 Cross-attention compression Total 917,313 Sleep overhead 15.6% 6.2 Data: PI-LLM Benchmark We adopt the PI-LLM paradigm of Wang & Sun (2025) in a synthetic setting. Each episode consists of a stream of key-value update tokens for a single entity: (e,v1),(e,v2),…,(e,vn)(e,v_1),(e,v_2),…,(e,v_n), followed by a query for the most recent value. Entity keys are drawn from a vocabulary of 100 entities and values from 500 possible values. We evaluate at seven PI depths: n∈1,2,5,10,15,20,30n∈\1,2,5,10,15,20,30\, with 200 episodes per depth. 6.3 Training Protocol All methods receive the same total training budget of 45 epochs. For SleepGate, this is split across three active stages following the curriculum in §4.2: 10 epochs warm-start (Stage 0), 5 epochs gate pre-training (Stage 1), and 30 epochs joint training with soft attention biasing (Stage 2). Stage 3 (threshold calibration) is omitted, as the soft bias mechanism is self-calibrating. Baseline models receive 45 epochs of standard autoregressive training with their respective cache management strategies active throughout training and evaluation. We use AdamW with learning rate 3×10−43× 10^-4 and batch size 16 for all methods. During Stage 2, we employ a PI depth curriculum: the maximum interference depth is gradually increased across epochs, beginning with n≤5n≤ 5 and reaching n=30n=30 by the final epoch. Each forward pass produces both unbiased logits (for the wake loss) and soft-biased logits (for the sleep loss), with β=5.0β=5.0 in Eq. 10. 6.4 Baselines We compare SleepGate against five baselines, spanning the major approaches to KV cache management: 1. Full KV Cache: Standard transformer retaining all cache entries (upper bound on interference). 2. Sliding Window: Fixed window of 64 entries; oldest entries are discarded (Beltagy et al., 2020). 3. H2O: Heavy-hitter oracle retaining entries with highest cumulative attention scores plus a recent window (Zhang et al., 2023). 4. StreamingLLM: Retains the first 4 “attention sink” tokens plus a sliding window (Xiao et al., 2024). 5. Decay Only (Ablation): Applies exponential key decay without the forgetting gate or consolidation module, isolating the contribution of learned gating. 6.5 Metrics • Retrieval Accuracy: Exact-match accuracy on the most recent value for each queried entity. • Stale Retrieval Rate: Fraction of all episodes in which the predicted value matches a superseded (outdated) value for the queried entity, directly measuring PI severity. • PI Slope: Slope of the accuracy-vs-log⁡(n) (n) regression line. A slope of 0 indicates perfect PI resistance; more negative slopes indicate greater susceptibility. 7 Experimental Results 7.1 Main Results: Retrieval Accuracy Table 2 presents retrieval accuracy and stale retrieval rates across all seven PI depths for SleepGate and all baselines. Figure 2 visualizes the accuracy curves. Figure 2: Retrieval accuracy vs. PI depth (number of prior updates). SleepGate (solid red) maintains near-perfect accuracy through n=10n=10 before degrading at higher depths. All baselines cluster near zero across all depths. Table 2: Retrieval accuracy and stale retrieval rate (%) across PI depths. Each cell reports accuracy / stale%. N=200N=200 episodes per depth per method. SleepGate uses post-sleep (soft-biased) evaluation. Method n=1n=1 n=2n=2 n=5n=5 n=10n=10 n=15n=15 n=20n=20 n=30n=30 SleepGate 82.5 / 0.0 99.0 / 0.5 99.5 / 0.0 97.0 / 3.5 73.5 / 23.5 30.5 / 55.5 16.5 / 62.0 StreamingLLM 17.5 / 0.0 18.0 / 9.0 10.0 / 28.5 6.0 / 19.5 6.0 / 22.5 4.0 / 31.5 5.5 / 29.0 Sliding Window 12.5 / 0.0 12.5 / 7.0 8.0 / 23.5 8.5 / 23.0 6.5 / 23.0 4.0 / 22.5 2.0 / 30.0 Decay Only 9.5 / 0.0 12.5 / 7.0 7.0 / 22.0 7.5 / 23.0 8.5 / 25.0 3.5 / 23.5 4.5 / 22.5 Full KV Cache 8.0 / 0.0 9.0 / 7.5 3.5 / 21.0 2.0 / 17.0 3.0 / 20.0 1.5 / 18.5 2.5 / 19.5 H2O 6.5 / 0.0 7.5 / 5.5 1.0 / 7.5 4.0 / 11.0 3.0 / 11.5 0.0 / 10.0 0.5 / 9.0 SleepGate achieves near-perfect retrieval accuracy at moderate PI depths (99.099.0–99.599.5% for n=2n=2–55; 97.097.0% at n=10n=10), outperforming the best baseline (StreamingLLM at 18.018.0%) by a factor of 5.5×5.5× at n=2n=2. The gap widens further at n=5n=5: SleepGate achieves 99.599.5% vs. 10.010.0% for the best baseline—a 10×10× improvement. All baselines perform near chance (roughly 1/500=0.21/500=0.2% for random guessing) at deeper PI levels, confirming that no existing cache management strategy addresses proactive interference. 7.2 Baseline Analysis The baseline results highlight a structural problem with existing KV cache methods: H2O performs worst. Despite being designed to retain “important” tokens, H2O achieves the lowest accuracy across all depths (0.00.0–7.57.5%). This is because cumulative attention scores are anti-correlated with freshness under PI: the model attends most to the entries it has seen most often, which are precisely the stale values. Retaining heavy hitters actively preserves interference. StreamingLLM is the best baseline, but still poor. The attention sink mechanism provides marginal benefit (17.517.5% at n=1n=1 vs. 8.08.0% for full cache), likely because the sink tokens provide a stable reference point. However, this advantage disappears at deeper PI levels. Decay Only vs. SleepGate. The decay-only ablation (≤12.5≤ 12.5%) performs comparably to other baselines, demonstrating that exponential key decay without learned gating is insufficient. The full SleepGate framework with its trained forgetting gate is essential for effective interference resolution. 7.3 Failure Mode Analysis SleepGate exhibits a sharp performance transition around n=15n=15, degrading from 97.097.0% (n=10n=10) to 73.573.5% (n=15n=15) and further to 16.516.5% (n=30n=30). The stale retrieval rate explains why: • For n≤10n≤ 10: Stale retrieval rate is ≤3.5≤ 3.5%, indicating the gate successfully identifies and suppresses nearly all superseded entries. • For n=15n=15–3030: Stale retrieval rate rises to 23.523.5–62.062.0%, indicating the gate’s semantic signatures (ds=64d_s=64) lack sufficient capacity to disambiguate 15–30 near-identical entries for the same entity. This failure has two contributing factors. First, the soft bias saturates: with β=5β=5 and retention ri=0.01r_i=0.01, the bias is approximately −23-23, but when 29 stale entries each contribute residual attention mass, the cumulative probability leaking to stale entries can overwhelm the single correct entry. Second, the semantic signature capacity is limited: with ds=64d_s=64, the tagger cannot produce sufficiently distinct signatures for 30 updates to the same entity key. The depth-1 anomaly. Accuracy at n=1n=1 (82.582.5%) is lower than at n=2n=2–1010. At n=1n=1 there is no prior value to supersede, so the sleep mechanism provides no benefit—the model relies entirely on base transformer retrieval capability. The fact that n=2n=2 accuracy (99.099.0%) exceeds n=1n=1 confirms that the soft bias actively improves retrieval when there is clear conflict to resolve. 7.4 Training Dynamics During Stage 2 joint training, we observe characteristic learning dynamics: • The gate achieves 99.399.3% accuracy on ground-truth supersession labels after just 5 epochs of pre-training (Stage 1), indicating that conflict detection is a relatively easy task for the learned semantic signatures. • Post-sleep retrieval accuracy improves steadily through Stage 2, reaching peak performance around epoch 24 and plateauing thereafter. • The PI depth curriculum is essential: exposing the model to n=30n=30 sequences from the start leads to training instability, while the progressive schedule (n≤5→n≤10→n≤15→n≤30n≤ 5→ n≤ 10→ n≤ 15→ n≤ 30) produces stable convergence. 8 Discussion 8.1 Why Architecture, Not Prompting? The failure of prompt-based interventions reported by Wang & Sun (2025) is not surprising from a mechanistic perspective. Telling a model to “ignore earlier values” requires the model to (a) identify which values are “earlier” in a semantic rather than positional sense, (b) actively suppress attention to those entries, and (c) do so reliably across all layers and heads. The standard attention mechanism provides no lever for step (b): all keys participate equally in the softmax competition. SleepGate provides this lever through soft attention biasing (Eq. 10), which directly modulates the pre-softmax attention logits based on learned retention scores. Our results confirm this: the full KV cache baseline achieves only 8.08.0% accuracy even at n=1n=1, while SleepGate reaches 99.599.5% at n=5n=5—a regime where all information is visible but the model cannot suppress the stale entries without architectural support. 8.2 Relationship to State-Space Models State-space models (SSMs) like Mamba (Gu & Dao, 2023) and its successors (Dao & Gu, 2024) implement a form of implicit forgetting through their recurrent structure: information must be compressed into a fixed-size state, naturally limiting interference. However, this comes at the cost of losing direct access to individual past tokens. SleepGate occupies a middle ground: it preserves the random-access property of the KV cache while adding selective forgetting. A natural extension would be to inform SleepGate’s forgetting gate with SSM-like selection mechanisms. 8.3 Multi-Scale Sleep Biological sleep operates at multiple timescales: micro-arousals, NREM stages 1–3, and REM sleep each serve different functions. We hypothesize that SleepGate would benefit from a similar hierarchy: • Micro-cycles (every 512–2K tokens): Lightweight decay and eviction of clearly superseded entries. • Meso-cycles (every 8K–32K tokens): Full consolidation with cross-attention compression. • Macro-cycles (at natural document boundaries): Deep restructuring of the cache, potentially involving re-encoding of consolidated entries through the full transformer stack. 8.4 Connection to Continual Learning PI in the context window mirrors catastrophic forgetting in continual learning (McCloskey & Cohen, 1989), but at a different timescale: within-context vs. across-training. Sleep-inspired replay has been successfully applied to continual learning (Tadros et al., 2022; González et al., 2020). SleepGate demonstrates that the same biological principle applies at the inference-time working memory level, suggesting a unified framework for memory management across timescales. 8.5 Limitations and Risks Depth Saturation. Our experiments show that SleepGate with soft attention biasing saturates at PI depths beyond n≈15n≈ 15, with accuracy dropping to 16.516.5% at n=30n=30 (see §7.3). The stale retrieval rate of 6262% at n=30n=30 indicates that the gate cannot reliably disambiguate many similar entries. Increasing dsd_s, using hard eviction for clearly stale entries while retaining soft biasing for borderline cases, or applying multi-round sleep cycles may address this limitation. Over-Forgetting. The soft attention biasing approach partially mitigates the risk of over-forgetting compared to hard eviction: entries are downweighted, not deleted, allowing the model to recover if the gate makes errors. However, the risk remains for entries with very low retention scores, where the bias is effectively −∞-∞. Scale and Generalization. Our proof-of-concept uses a small transformer (793K base parameters) on synthetic data. Whether the mechanism transfers to pre-trained models at scale and generalizes to naturalistic PI patterns (e.g., document corrections, evolving facts) remains to be validated. The synthetic setting isolates the PI phenomenon but does not capture the full complexity of natural language interference. Computational Overhead. Each sleep micro-cycle requires a forward pass through the gate network for every cache entry. For a cache of size C and a gate with dh=128d_h=128 hidden units, this is O(C⋅dg⋅dh)O(C· d_g· d_h) per cycle—negligible compared to a transformer forward pass, but non-zero. The adaptive trigger ensures cycles only execute when beneficial. Interaction with Existing Optimizations. SleepGate must compose correctly with grouped-query attention (GQA) (Ainslie et al., 2023), PagedAttention (Kwon et al., 2023), and quantized KV caches (Hooper et al., 2024). We expect compatibility since SleepGate operates on the logical cache structure, but empirical validation is required. 9 Conclusion The discovery that LLMs suffer from proactive interference (Wang & Sun, 2025)—and that this interference represents a fundamental working memory bottleneck beyond context length—calls for an architectural solution. We have proposed SleepGate, a framework that draws on the neuroscience of sleep-dependent memory consolidation to equip LLMs with the ability to actively manage their key-value caches. The three sleep-inspired modules—key decay, learned gating, and consolidation—give the model a direct mechanism for interference resolution, one that prompt engineering cannot provide. Our theoretical analysis shows that the mechanism can eliminate the log-linear accuracy degradation observed under PI, and our preliminary experiments confirm this prediction: SleepGate achieves 9797–99.599.5% retrieval accuracy at PI depths 2–10, while all five baselines remain below 1818%. The soft attention biasing mechanism (Eq. 10) proved particularly effective: it supports fully differentiable training without Gumbel-softmax relaxation and removes the need for a separate threshold calibration stage. The identified failure mode at extreme PI depths (n≥15n≥ 15) points to concrete avenues for improvement: increasing semantic signature capacity, combining soft biasing with selective hard eviction, or adopting multi-scale sleep cycles (§8.3). More broadly, this work shows that cognitive science can inform the design of LLM architectures. The brain’s solution to proactive interference—an active, learned process of memory curation—works because it is selective, retaining useful information while discarding what has been superseded. As LLMs move into streaming, long-horizon settings where context windows inevitably accumulate stale information, this principle of learned selective forgetting will become increasingly important. 9.1 Future Directions 1. Scaling to production models: Integrating SleepGate into pre-trained models (e.g., Llama-3, Mistral) via post-hoc fine-tuning to validate the mechanism at scale and on natural language PI scenarios. 2. Addressing the depth-15+ cliff: Investigating higher semantic signature dimensions (ds>64d_s>64), larger bias scales (β>5β>5), and hybrid soft-bias/hard-eviction strategies to extend effective PI resistance beyond n=10n=10. 3. Extended benchmarks: Evaluating on multi-entity PI, long-document QA with corrections, streaming agent tasks, and standard long-context benchmarks (RULER (Hsieh et al., 2024), LongBench (Bai et al., 2024)) to test generalization and confirm that SleepGate does not degrade non-PI performance. 4. Dream-like training: Using the model’s own generated text during “sleep” phases to rehearse and consolidate important patterns, analogous to dream-based memory processing. 5. Cross-timescale integration: Unifying in-context forgetting (SleepGate) with cross-task forgetting (continual learning) into a single sleep-inspired framework operating at multiple timescales. 6. Mechanistic interpretability: Analyzing what the forgetting gate learns—which features drive eviction decisions—to understand how transformers represent and distinguish current from outdated information. Acknowledgments Claude.ai was used to assist with polishing the writing of this paper. References Abel et al. [2013] Abel, T., Havekes, R., Saletin, J. M., & Walker, M. P. (2013). Sleep, plasticity and memory from molecules to whole-brain networks. Current Biology, 23(17), R774–R788. Ainslie et al. [2023] Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., & Sanghai, S. (2023). GQA: Training generalized multi-query transformer models from multi-head checkpoints. In Proceedings of EMNLP. Anthropic [2024] Anthropic (2024). The Claude 3 model family: Opus, Sonnet, Haiku. Technical report / model card. Beltagy et al. [2020] Beltagy, I., Peters, M. E., & Cohan, A. (2020). Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150. Berry & Davis [2014] Berry, J. A. & Davis, R. L. (2014). Active forgetting of olfactory memories in Drosophila. Progress in Brain Research, 208, 39–62. Borgeaud et al. [2022] Borgeaud, S., Mensch, A., Hoffmann, J., Cai, T., Rutherford, E., Millican, K., …& Sifre, L. (2022). Improving language models by retrieving from trillions of tokens. In Proceedings of ICML. Child et al. [2019] Child, R., Gray, S., Radford, A., & Sutskever, I. (2019). Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509. Dao & Gu [2024] Dao, T. & Gu, A. (2024). Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. In Proceedings of ICML. Davis & Zhong [2017] Davis, R. L. & Zhong, Y. (2017). The biology of forgetting—a perspective. Neuron, 95(3), 490–503. Kumaran et al. [2016] Kumaran, D., Hassabis, D., & McClelland, J. L. (2016). What learning systems do intelligent agents need? Complementary learning systems theory updated. Trends in Cognitive Sciences, 20(7), 512–534. Diekelmann & Born [2010] Diekelmann, S. & Born, J. (2010). The memory function of sleep. Nature Reviews Neuroscience, 11(2), 114–126. Ellenbogen et al. [2006] Ellenbogen, J. M., Hulbert, J. C., Stickgold, R., Dinges, D. F., & Thompson-Schill, S. L. (2006). Interfering with theories of sleep and memory: Sleep, declarative memory, and associative interference. Current Biology, 16(13), 1290–1294. French [1999] French, R. M. (1999). Catastrophic forgetting in connectionist networks. Trends in Cognitive Sciences, 3(4), 128–135. Ge et al. [2024] Ge, S., Zhang, Y., Liu, L., Zhang, M., Han, J., & Gao, J. (2024). Model tells you what to discard: Adaptive KV cache compression for LLMs. In Proceedings of ICLR. González et al. [2020] González, O. C., Sokolov, Y., Krishnan, G. P., Delanois J. E., & Bazhenov, M. (2020). Can sleep protect memories from catastrophic forgetting? eLife, 9:e51005. Gemini Team et al. [2024] Gemini Team, Georgiev, P., Lei, V. I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., & many others (2024). Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530. Graves et al. [2014] Graves, A., Wayne, G., & Danihelka, I. (2014). Neural Turing machines. arXiv preprint arXiv:1410.5401. Gu & Dao [2023] Gu, A. & Dao, T. (2023). Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752. Hochreiter & Schmidhuber [1997] Hochreiter, S. & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735–1780. Hooper et al. [2024] Hooper, C., Kim, S., Mohammadzadeh, H., Mahoney, M. W., Shao, Y. S., Keutzer, K., & Gholami, A. (2024). KVQuant: Towards 10 million context length LLM inference with KV cache quantization. arXiv preprint arXiv:2401.18079. Hsieh et al. [2024] Hsieh, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y., & Ginsburg, B. (2024). RULER: What’s the real context size of your long-context language models? In arXiv preprint arXiv:2404.06654. Indyk & Motwani [1998] Indyk, P. & Motwani, R. (1998). Approximate nearest neighbors: towards removing the curse of dimensionality. In Proceedings of STOC, p. 604–613. Jang et al. [2017] Jang, E., Gu, S., & Poole, B. (2017). Categorical reparameterization with Gumbel-softmax. In Proceedings of ICLR. Kane & Engle [2000] Kane, M. J. & Engle, R. W. (2000). Working-memory capacity, proactive interference, and divided attention: Limits on long-term memory retrieval. Journal of Experimental Psychology: Learning, Memory, and Cognition, 26(2), 336-358. Kwon et al. [2023] Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., …& Stoica, I. (2023). Efficient memory management for large language model serving with PagedAttention. In Proceedings of SOSP. Lewis et al. [2020] Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., …& Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. In Proceedings of NeurIPS. Bai et al. [2024] Bai, Y., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., Du, Z., Liu, X., Zeng, A., Hou, L., Dong, Y., Tang, J., & Li, J. (2024). LongBench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 3119–3137. Liu et al. [2024] Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024). Lost in the middle: How language models use long contexts. Transactions of the ACL, 12, 157–173. McCloskey & Cohen [1989] McCloskey, M. & Cohen, N. J. (1989). Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of Learning and Motivation, Vol. 24, p. 109–165. Munkhdalai et al. [2024] Munkhdalai, T., Faruqui, M., & Gopal, S. (2024). Leave no context behind: Efficient infinite context transformers with Infini-attention. arXiv preprint arXiv:2404.07143. OpenAI [2023] OpenAI (2023). New models and developer products announced at DevDay. OpenAI official announcement. Radford et al. [2019] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI blog. Rasch & Born [2013] Rasch, B. & Born, J. (2013). About sleep’s role in memory. Physiological Reviews, 93(2), 681–766. Staresina et al. [2015] Staresina, B. P., Bergmann, T. O., Bonnefond, M., van der Meij, R., Jensen, O., Deuker, L., …& Fell, J. (2015). Hierarchical nesting of slow oscillations, spindles and ripples in the human hippocampus during sleep. Nature Neuroscience, 18(11), 1679–1686. Sukhbaatar et al. [2015] Sukhbaatar, S., Szlam, A., Weston, J., & Fergus, R. (2015). End-to-end memory networks. In Proceedings of NeurIPS. Sun et al. [2023] Sun, Y., Dong, L., Huang, S., Ma, S., Xia, Y., Xue, J., Wang, J., & Wei, F. (2023). Retentive network: A successor to transformer for large language models. arXiv preprint arXiv:2307.08621. Tadros et al. [2022] Tadros, T., Krishnan, G. P., Ramyaa, R., & Bazhenov, M. (2022). Sleep-like unsupervised replay reduces catastrophic forgetting in artificial neural networks. Nature Communications, 13, 7742. Tay et al. [2022] Tay, Y., Dehghani, M., Bahri, D., & Metzler, D. (2022). Efficient transformers: A survey. ACM Computing Surveys, 55(6), 1–28. Tononi & Cirelli [2006] Tononi, G. & Cirelli, C. (2006). Sleep function and synaptic homeostasis. Sleep Medicine Reviews, 10(1), 49–62. Tononi & Cirelli [2014] Tononi, G. & Cirelli, C. (2014). Sleep and the price of plasticity: from synaptic and cellular homeostasis to memory consolidation and integration. Neuron, 81(1), 12–34. Underwood [1957] Underwood, B. J. (1957). Interference and forgetting. Psychological Review, 64(1), 49–60. Wang & Sun [2025] Wang, C. & Sun, J. V. (2025). Unable to forget: Proactive interference reveals working memory limits in LLMs beyond context length. arXiv preprint arXiv:2506.08184. Wickens [1970] Wickens, D. D. (1970). Encoding categories of words: An empirical approach to meaning. Psychological Review, 77(1), 1–15. Wu et al. [2022] Wu, Y., Rabe, M. N., Hutchins, D., & Szegedy, C. (2022). Memorizing transformers. In Proceedings of ICLR. Xiao et al. [2024] Xiao, G., Tian, Y., Chen, B., Han, S., & Lewis, M. (2024). Efficient streaming language models with attention sinks. In Proceedings of ICLR. Zhang et al. [2023] Zhang, Z., Sheng, Y., Zhou, T., Chen, T., Zheng, L., Cai, R., …& Chen, B. (2023). H2O: Heavy-hitter oracle for efficient generative inference of large language models. In Proceedings of NeurIPS. Appendix A Hyperparameter Specifications Table 3 provides the hyperparameter settings used in our experiments. Table 3: Hyperparameters used for SleepGate in the proof-of-concept experiments. Parameter Symbol Value Description Gate hidden dim dhd_h 128 Hidden dimension of GθG_θ Semantic sig. dim dsd_s 64 Dimension of semantic signatures Local pool window w 4 Window size for local pooling Conflict threshold δ 0.85 Cosine similarity for conflict Keep threshold αk _k 0.7 Retention score for keeping Evict threshold αe _e 0.3 Retention score for eviction Decay rate λ 0.01 Log-scale key decay rate Entropy trigger κ 1.5 Std. deviations above mean Max conflict density ρmax _ 0.4 Trigger when 40% superseded Fallback interval NmaxN_ 128 Max tokens between cycles Sleep loss weight λs _s 0.5 Weight of sleep loss Compress loss weight λc _c 0.1 Weight of compression loss Align loss weight λg _g 0.3 Weight of gate alignment loss Soft bias scale β 5.0 Scale for log⁡(ri) (r_i) bias Recency weight η 2.0 Recency bias in consolidation Gumbel temperature τtemp _temp 1.0→0.11.0→ 0.1 Annealed during Stage 2 Learning rate — 3×10−43× 10^-4 AdamW optimizer Batch size — 16 Training batch size Appendix B Pseudocode for Training Loop Algorithm 2 Full Training Loop for SleepGate 0: Pre-trained transformer ℳM, gate network GθG_θ, training data D 1: for each batch B∈B do 2: Initialize augmented KV cache +←∅C^+← 3: ℒbatch←0L_batch← 0 4: for each token xt∈Bx_t∈ B do 5: // Wake phase: standard forward pass 6: (t,t,x^t+1)←ℳ(xt,+)(k_t,v_t, x_t+1) (x_t,C^+) 7: ℒbatch+=−logp(x^t+1=xt+1)L_batch +=- p( x_t+1=x_t+1) 8: Compute semantic signature ts_t via Eq. 3 9: Detect conflicts and set σj _j for superseded entries 10: Initialize attention statistic at←0a_t← 0 11: +←+∪(t,t,t,t,0,at)C^+ ^+∪\(k_t,v_t,t,s_t,0,a_t)\ 12: if trigger(t)trigger(t) then 13: // Sleep phase (trigger per Eq. 16) 14: ←SleepMicroCycle(+,Gθ,αk,αe,λ,β)b (C^+,G_θ, _k, _e,λ,β) Soft biasing variant 15: Compute ℒsleepL_sleep via biased forward pass (Eq. 19, 11) 16: Compute ℒcompressL_compress via Eq. 20 17: Compute ℒalignL_align via Eq. 21 18: ℒbatch+=λsℒsleep+λcℒcompress+λgℒalignL_batch += _sL_sleep+ _cL_compress+ _gL_align 19: end if 20: end for 21: Backpropagate ℒbatchL_batch through all trainable parameters of ℳM, GθG_θ, and the tagger 22: end for Appendix C Extended Related Work: Sleep Neuroscience The mapping between biological sleep mechanisms and SleepGate modules is summarized in Table 4. Table 4: Mapping between biological sleep mechanisms and SleepGate components. Biological Mechanism SleepGate Module Functional Correspondence Synaptic homeostasis (global downscaling during SWS) Key decay Log-scale reduction of key magnitudes preserves relative importance while reducing absolute interference Hippocampal-neocortical replay (selective consolidation) Consolidation module Cross-attention compression transfers important information into compact representations Dopaminergic active forgetting Forgetting gate GθG_θ Learned, content-dependent eviction of stale entries Sleep spindles / sharp-wave ripples (oscillatory coordination) Adaptive sleep trigger Entropy and conflict signals coordinate when consolidation occurs REM sleep (pattern separation) Semantic signatures Explicit representation of “what slot” each entry occupies enables disambiguation

Instant research discovery