Paper deep dive
DeltaTTT: Layerwise Optimization for Nonlinear Recurrent Memory
Yining Li, Dongchen Han, Jie Fu, Gao Huang
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Sequential test-time training adapts a memory network through successive updates, each computing an inner-loop gradient based on the network's previous state. Intuitively, this state dependence should allow each update to account for what the memory has already learned and better incorporate new information. However, we find that this expected advantage does not consistently materialize in nonlinear memories: a fixed-base parallel TTT baseline outperforms its serial counterpart. Our exploratory experiments point to a key underlying difficulty: nonlinear memories can be harder to optimize than linear ones within a single pass over the sequence. To alleviate this optimization difficulty, we introduce DeltaTTT, which replaces joint inner-loop optimization of a two-layer memory network with layerwise learning. Each layer is assigned a local prediction target and updated through a state-dependent delta rule. This formulation retains a nonlinear readout while enabling chunkwise parallel computation. Experiments on DeltaNet and LaCT backbones show improvements in language modeling and retrieval over their recurrent baselines.
Tags
Links
- Source: https://arxiv.org/abs/2610.08553v1
- Canonical: https://arxiv.org/abs/2610.08553v1
Trouble viewing inline? Open PDF directly →
Full Text
47,480 characters extracted from source content.
Expand or collapse full text
DeltaTTT: Layerwise Optimization for Nonlinear Recurrent Memory Yining Li Affiliation: Qiuzhen College, Tsinghua University LeapLab, Tsinghua University Dongchen Han Affiliation: Qiuzhen College, Tsinghua University LeapLab, Tsinghua University Jie Fu Affiliation: IQuest Research Gao Huang Affiliation: Qiuzhen College, Tsinghua University LeapLab, Tsinghua University Abstract Sequential test-time training adapts a memory network through successive updates, each computing an inner-loop gradient based on the network’s previous state. Intuitively, this state dependence should allow each update to account for what the memory has already learned and better incorporate new information. However, we find that this expected advantage does not consistently materialize in nonlinear memories: a fixed-base parallel T baseline outperforms its serial counterpart. Our exploratory experiments point to a key underlying difficulty: nonlinear memories can be harder to optimize than linear ones within a single pass over the sequence. To alleviate this optimization difficulty, we introduce DeltaTTT, which replaces joint inner-loop optimization of a two-layer memory network with layerwise learning. Each layer is assigned a local prediction target and updated through a state-dependent delta rule. This formulation retains a nonlinear readout while enabling chunkwise parallel computation. Experiments on DeltaNet and LaCT backbones show improvements in language modeling and retrieval over their recurrent baselines. $ $$ $footnotetext: Corresponding author. 1 Introduction In sequence modeling, Test-Time Training (T) compresses past key–value pairs into the parameters of a small inner network through online updates (Sun et al., 2024). By default, the updates are sequential, each computing an inner-loop gradient based on the network’s previous state. Intuitively, this state dependence should improve memory learning: each update can account for what the network has already learned and better incorporate new information. This intuition is supported by results on linear memories. DeltaNet replaces the state-independent additive writes of linear attention with residual corrections based on the current memory, achieving stronger associative recall and supporting the value of state-dependent updates (Yang et al., 2024). However, this expected advantage does not consistently materialize for nonlinear memories. In a LaCT-derived language model (Zhang et al., 2025), we compare serial updates, which evaluate each chunk at the current fast weights, with a fixed-base parallel baseline, which evaluates updates at a shared initial state and accumulates them causally. We find that the parallel baseline achieves better downstream results despite computing its updates without considering earlier writes. This result motivates us to revisit how nonlinear memory is optimized in serial T. To examine memory optimization more directly, we fit standalone linear and two-layer MLP memories to fixed K/V sequences extracted from a pretrained full-attention model, using key-value reconstruction as the objective. A linear memory can achieve lower reconstruction error within a single epoch, while the MLP’s advantage emerges with additional epochs. These observations suggest that greater expressivity alone is insufficient: the write rule must also realize that capacity within the limited inner-update budget, usually one epoch for T. We therefore ask: if jointly optimizing a nonlinear memory can be less effective than optimizing a linear one within a single pass, can we optimize it more effectively layer by layer? This motivates DeltaTTT, which retains a nonlinear multilayer memory while equipping each linear layer with a local reconstruction target and its own delta-rule update. Given its inputs and targets, each layer admits efficient chunkwise parallel evaluation, while the nonlinear composition and dependencies between layers are preserved. This design aims to combine the expressive capacity of nonlinear memories with effective single-pass optimization and the computational efficiency of the delta rule. Our contributions are threefold. 1) We show that serial nonlinear T can underperform fixed-base parallel T, and use fixed-K/V reconstruction experiments to highlight the difficulty of optimizing nonlinear memories within a single pass. 2) To alleviate this optimization difficulty, we propose DeltaTTT, which replaces joint inner-loop optimization of a two-layer memory network with layerwise learning, assigning each layer a local prediction target and a state-dependent delta-rule update. This formulation retains a nonlinear readout and enables chunkwise parallel computation while preserving exact tokenwise updates. 3) We evaluate DeltaTTT with 0.5B-parameter language models trained on 92.5B tokens, demonstrating improvements in language modeling and retrieval over recurrent baselines on both DeltaNet and LaCT backbones. Figure 1: Joint and layerwise optimization of nonlinear recurrent memory. (a) T jointly optimizes both layers through a shared reconstruction objective. (b) DeltaTTT assigns a local reconstruction objective to each layer and performs layerwise delta-rule updates. (c) Both methods use the nonlinear readout after the writes. 2 Background Test-Time Training. Test-Time Training (T) represents sequence context through the parameters W of an inner network fWf_W, typically an MLP(Sun et al., 2024; Han et al., 2026). Given query, key, and value features (Q,K,V)(Q,K,V), T compresses key–value associations into W through inner-loop optimization: W←W−η∇Wℒ(fW(K),V),W← W-η _WL (f_W(K),V ), (1) where ℒL is the inner learning objective and η is the learning rate. Queries retrieve information from the resulting memory through O=fW(Q).O=f_W(Q). (2) As the sequence is processed, these updates allow the inner network to accumulate context in a fixed-size parameter state. With a fixed-size inner network and a single pass over the sequence, the total read and update cost scales linearly with sequence length. Serial and fixed-base parallel T. Our serial baseline follows the chunkwise T formulation of LaCT (Zhang et al., 2025). Let ℒc(W)L_c(W) denote the inner loss on chunk c evaluated at memory parameters W. We maintain an unnormalized accumulator ScS_c and a normalized memory WcW_c, initialized with S0=W0S_0=W_0. For clarity, we omit momentum. Serial T evaluates each gradient at the memory updated by the preceding chunks, whereas fixed-base parallel T evaluates every gradient at the shared initialization: Serial:Sc : S_c =Sc−1−η∇ℒc(Wc−1),Wc=(Sc), =S_c-1-η _c(W_c-1), W_c=N(S_c), (3) Fixed-base:Sc -base: S_c =Sc−1−η∇ℒc(W0),Wc=(Sc), =S_c-1-η _c(W_0), W_c=N(S_c), (4) where η is the inner learning rate and N means normalization. Serial gradients depend on preceding updates, while fixed-base gradients can be computed independently across chunks and accumulated through a causal prefix sum. Both baselines read before writing the current chunk, using ot=fWc−1(qt)o_t=f_W_c-1(q_t) for tokens t in chunk c. The linear case: DeltaNet and Linear Attention. The same framework recovers DeltaNet by using a linear memory fW(k)=Wkf_W(k)=Wk, the squared loss ℓ(y,v)=12‖y−v‖22 (y,v)= 12\|y-v\|_2^2, and one token per update (chunk size=1). Indexing states by token, serial gradient descent gives Wt W_t =Wt−1−ηt∇ℒt(Wt−1) =W_t-1- _t _t(W_t-1) (5) =Wt−1+ηt(vt−Wt−1kt)kt⊤. =W_t-1+ _t(v_t-W_t-1k_t)k_t . This is precisely the delta-rule update used in DeltaNet. For supplied keys, values, and learning rates, this recurrence is affine in the preceding state, enabling efficient chunkwise parallel evaluation (Yang et al., 2024). DeltaNet thus combines tokenwise update granularity (chunk size=1) with computational parallelism, preserving the exact sequence of state-dependent updates. Applying fixed-base evaluation to the same objective instead yields Wt=W0+∑i=1tηi(vi−W0ki)ki⊤.W_t=W_0+ _i=1^t _i(v_i-W_0k_i)k_i . (6) With W0=0W_0=0 and ηi=1 _i=1, this reduces to Wt=∑i=1tviki⊤W_t= _i=1^tv_ik_i , which is exactly linear attention. Thus, DeltaNet and additive linear attention correspond to serial and fixed-base parallel T respectively within linear memory. 3 Motivation: Effective Optimization in a Single Pass Serial T is intuitively promising because this state dependence should allow each update to account for what the memory has already learned and better incorporate new information. This expectation is supported by DeltaNet, which uses the residual vt−Wt−1ktv_t-W_t-1k_t to perform state-dependent correction (Equation 5) and achieves stronger associative recall than additive linear attention (Yang et al., 2024). Surprisingly, we observe the opposite in nonlinear T: serial updates achieve a single-needle accuracy of 28.25, compared with 37.87 for fixed-base parallel updates (Table 1). We consider two possible explanations for this discrepancy. Table 1: Serial and parallel T with SwishGLU inner models, 0.5B parameters, and 92.5B training tokens. Single denotes mean single-needle retrieval accuracy across S1/S2/S3; 32k denotes the mean accuracy across S1/S2/S3 at 32k context length. Update rule Loss ↓ Single ↑ 32k ↑ Serial 1.99 28.25 11.13 Parallel 1.98 37.87 30.87 Nonlinear memories can struggle to fit well within a single pass. Greater expressivity does not necessarily translate into better fitting under a limited optimization budget. To examine this, we conduct a toy experiment in which a linear map and a two-layer MLP fit the same fixed K/V sequence extracted from a full-attention model. We compare their reconstruction errors across epochs. As shown in Figure 2, the linear memory can achieve lower reconstruction error after just one epoch, despite the MLP’s greater capacity. The MLP attains lower error at small chunk sizes by epoch 16 (Figure 2), suggesting that its capacity advantage requires more optimization to emerge under this protocol. However, existing methods such as LaCT (Zhang et al., 2025), T (Sun et al., 2024), and DeltaNet (Yang et al., 2024) typically use a single pass over the context for computational efficiency, as additional epochs of inner learning would incur substantial overhead. These results motivate more effective optimization of nonlinear memories within a single-pass budget. Appendix B provides the experimental details. Figure 2: Fitting fixed Full-Attention K/V pairs at different replay budgets. Smaller chunks can improve fitting but incur high latency in serial nonlinear T. Figure 2 highlights a second constraint: at a fixed replay budget, smaller chunks allow more update steps and can yield lower reconstruction error, although the trend is not strictly monotonic; for the tuned MLP, the benefit is more apparent after multiple passes. A related tradeoff was observed by Sun et al. (2024), who found that smaller T mini-batches reduce language-model perplexity at the cost of higher latency. In serial nonlinear T, each chunk’s gradient depends on the memory updated by preceding chunks, so smaller chunks require more sequential update steps while offering less parallelism within each step. This computational tradeoff motivates LaCT’s use of large chunks (Zhang et al., 2025). Linear recurrent memories with delta updates offer a way around this tradeoff. Given the keys and targets, each delta update is affine in the preceding memory state, enabling efficient chunkwise parallel evaluation while preserving the exact sequence of tokenwise corrections (Yang et al., 2024). The logical update granularity can therefore remain C=1C=1 even when the hardware kernel processes many tokens together. 4 DeltaTTT: Layerwise Memory Optimization Given the difficulty of optimizing a nonlinear multilayer memory as a whole within a single epoch, can we optimize it more effectively layer by layer? This motivates DeltaTTT, which retains a nonlinear multilayer memory while equipping each linear layer with a local target and its own delta update. Given its inputs and targets, each layer admits efficient chunkwise parallel evaluation, while the nonlinear composition and dependencies between layers are preserved. This design aims to combine the expressive capacity of nonlinear memories with effective single-pass optimization and the computational efficiency of the delta rule. 4.1 Layerwise memory optimization We now introduce DeltaTTT, a layerwise approach to optimizing nonlinear recurrent memory. For one memory head, let kt,qt∈ℝdkk_t,q_t ^d_k and vt∈ℝdvv_t ^d_v denote the current key, query, and value. The memory W consists of two dynamic matrices A∈ℝdv×dkA ^d_v× d_k and B∈ℝdv×dvB ^d_v× d_v, connected by ϕ=SiLUφ=SiLU. Classical T jointly optimizes this two-layer memory to reconstruct values from keys: fA,B(x)=Bϕ(Ax),ℓt(A,B)=12‖Bϕ(Akt)−vt‖22.f_A,B(x)=Bφ(Ax), _t(A,B)= 12\|Bφ(Ak_t)-v_t\|_2^2. (7) Our goal is to make this reconstruction task easier to optimize within a single pass. To this end, we decompose it into two local learning problems, one for each layer. The first layer learns to predict vtv_t directly from ktk_t. After updating A, its prediction is passed through the nonlinearity to form an intermediate representation ht=ϕ(Atkt)h_t=φ(A_tk_t). The second layer then learns to reconstruct the same value vtv_t from hth_t. In this way, the first layer makes an initial prediction, and the second provides another learned reconstruction stage that can compensate for imperfections in the first. Their local objectives are ℓA,t(A)=12‖Akt−vt‖22,ℓB,t(B)=12‖Bht−vt‖22. _A,t(A)= 12\|Ak_t-v_t\|_2^2, _B,t(B)= 12\|Bh_t-v_t\|_2^2. (8) Each objective is linear regression in its own matrix, so a gradient step yields a delta-rule update. With learned write rates αt _t and βt _t, the complete update and readout are At A_t =At−1+αt(vt−At−1kt)kt⊤, =A_t-1+ _t(v_t-A_t-1k_t)k_t , (9) ht h_t =ϕ(Atkt), =φ(A_tk_t), (10) Bt B_t =Bt−1+βt(vt−Bt−1ht)ht⊤, =B_t-1+ _t(v_t-B_t-1h_t)h_t , (11) ot o_t =Btϕ(Atqt). =B_tφ(A_tq_t). (12) Thus, the query still reads from a nonlinear memory. Meanwhile, each linear layer is updated using the delta rule, allowing DeltaTTT to reuse DeltaNet’s chunkwise scan algorithm to compute tokenwise updates in parallel within each chunk (Yang et al., 2024). Through layerwise optimization, DeltaTTT combines tokenwise updates with efficient parallel computation for nonlinear memory, while potentially benefiting from the easier optimization of individual linear layers compared with the nonlinear network. We call formulation above DeltaTTT-post, since hth_t uses the updated first-layer state. To avoid waiting for the current A update, DeltaTTT-pre instead uses ht=ϕ(At−1kt)h_t=φ(A_t-1k_t), leaving the remaining equations unchanged. Once this representation is available, the two current-token writes can proceed concurrently. Our ablations show comparable performance between Pre and Post, making DeltaTTT-pre attractive for its greater potential concurrency. We therefore adopt DeltaTTT-pre, as illustrated in Figure 1(b), as our default method. To clarify how this update and readout scheme relates to existing methods, Table 2 compares the states used to evaluate updates, the granularity of memory evolution, and the states used for query readout. Mini-batch T and E2-T retain tokenwise state recurrences, but evaluate gradients at fixed batch- or chunk-start states. LaCT and our fixed-base parallel T baseline instead evolve memory at chunk boundaries and use chunk-start states for readout. DeltaNet supports state-dependent tokenwise updates and current-token readout with a linear memory. DeltaTTT combines these properties with a nonlinear memory, using layerwise delta-rule updates to enable chunkwise parallel computation. Pseudocode for the chunkwise computation is provided in Appendix A. Table 2: Comparison of memory update schedules. Gradient/write evaluated at denotes the fast-weight state used to compute each update; state recurrence denotes the granularity of state evolution; output computed from denotes the state used for query readout. T rows assume nonlinear inner models. Parallel T adapts fixed-initialization batch-gradient T to our chunkwise, read-before-write baseline (Eq. 4). Method Gradient/write evaluated at State recurrence Output computed from Nonlinear Efficient training Token-wise T (Sun et al., 2024) Previous token Token-wise Current token ✓ × Mini-batch T (Sun et al., 2024) Batch start Token-wise Current token ✓ × Chunk-T (Zhang et al., 2025) Chunk start Chunk-wise Chunk start ✓ ✓ E2-T (Zhong et al., 2026) Chunk start Token-wise Chunk start ✓ ✓ Parallel T Initial state Chunk-wise Chunk start ✓ ✓ DeltaNet (Yang et al., 2024) Previous token Token-wise Current token × ✓ DeltaTTT (ours) Previous token Token-wise Current token ✓ ✓ 5 Experiments 5.1 Experimental setup We train on 92.5B tokens from the pretraining split of Long-Data-Collections. Our DeltaTTT models contain approximately 0.54B parameters, with 24 layers and a hidden dimension of 1024. Training uses 32K-token sequences and AdamW with a peak learning rate of 3×10−43× 10^-4 and cosine decay. We implement DeltaTTT on DeltaNet (Yang et al., 2024) and LaCT (Zhang et al., 2025) backbones by replacing their memory modules with our DeltaTTT module. Baselines include DeltaNet and a variant with a learnable initial memory W0W_0 ; Gated DeltaNet (Yang et al., 2025); serial LaCT and our fixed-base parallel LaCT variant; and Delta MLP (Irie et al., 2021), which uses layer-specific keys and values generated from the same input token to update a multilayer fast-weight memory via the delta rule. For a fair comparison, all recurrent models use sliding-window attention (SWA) with a 512-token window. Following LaCT’s practice, we set both the update chunk size and SWA window size to 512 tokens for all serial and parallel LaCT baselines. A full-attention Transformer (Vaswani et al., 2017) serves as an additional reference. We evaluate single-needle retrieval using RULER (Hsieh et al., 2024) over 4K–32K contexts. Detailed experimental settings are provided in Appendix C. 5.2 Main results Table 3 shows that DeltaTTT improves both training loss and single-needle retrieval across the two backbones. On the LaCT backbone, fixed-base parallel updates outperform serial updates, but DeltaTTT surpasses both while retaining state-dependent correction, including a 12.60 point gain in single-needle retrieval over parallel LaCT. This suggests that a more effective optimization strategy can help realize the potential of nonlinear, state-dependent memory. Improvements over DeltaNet and Gated DeltaNet on the DeltaNet backbone further support the applicability of layerwise optimization across different backbone designs. Table 3: Results grouped by backbone at the 92.5B-token budget. All recurrent configurations use SWA with window 512. Single denotes the mean over S1/S2/S3. Model Params (M) Train loss ↓ Single ↑ S1 ↑ S2 ↑ S3 ↑ 4K 8K 16K 32K Avg. 4K 8K 16K 32K Avg. 4K 8K 16K 32K Avg. Full attention 536.95 1.918769 72.42 100.0 100.0 99.2 95.2 98.60 95.4 91.2 95.6 88.2 92.60 50.0 27.8 22.2 4.2 26.05 DeltaNet backbone DeltaNet 537.64 1.955473 44.38 99.8 98.4 97.2 91.8 96.80 83.0 32.6 10.8 2.4 32.20 7.6 4.2 2.2 2.6 4.15 Gated DeltaNet 562.80 1.967150 47.60 100.0 100.0 100.0 98.8 99.70 76.0 43.6 19.4 5.0 36.00 17.6 7.2 2.2 1.4 7.10 DeltaTTT 541.17 1.950079 48.78 100.0 100.0 100.0 99.8 99.95 93.4 44.0 14.2 4.6 39.05 21.4 4.6 1.6 1.8 7.35 LaCT backbone DeltaNet 538.13 1.970605 25.00 85.2 64.2 34.0 4.0 46.85 50.0 29.2 10.2 3.4 23.20 11.2 4.2 2.6 1.8 4.95 DeltaNet (learned init) 539.70 1.969120 40.52 100.0 98.8 72.0 21.4 73.05 84.6 54.6 22.2 9.6 42.75 15.0 5.2 1.6 1.2 5.75 Delta MLP 592.10 1.968994 4.88 16.6 5.4 3.2 1.2 6.60 13.8 6.2 1.6 1.8 5.85 4.0 2.2 1.2 1.4 2.20 LaCT Serial SwishGLU 546.98 1.989887 28.25 56.4 52.2 41.4 29.4 44.85 79.4 21.8 9.0 2.2 28.10 32.6 9.0 3.8 1.8 11.80 LaCT Parallel SwishGLU 546.98 1.984051 37.87 89.0 94.6 91.0 88.8 90.85 41.8 12.0 7.6 1.8 15.80 16.4 7.2 2.2 2.0 6.95 DeltaTTT 541.67 1.952344 50.47 99.8 99.6 97.6 69.8 91.70 86.6 73.0 34.8 9.6 51.00 22.6 7.8 2.6 1.8 8.70 5.3 Context use and position-dependent loss Figure 3: Average Position-wise loss on 43 held-out documents (19 books, 24 papers), each at least 32K tokens long. Position-dependent prediction. We select 19 books and 24 arXiv papers from the Long-Data-Collections, which are excluded from training. Each is long enough to provide 32K prediction targets from a single document prefix. We compute teacher-forced next-token loss for each document, then average losses across documents in 1K-token bins. Figure 3 shows both absolute loss and excess loss relative to full attention. DeltaTTT has the lowest average loss among the evaluated LaCT-backbone recurrent models and reduces the gap to full attention, with a larger advantage over parallel LaCT at later positions. Full attention retains the lowest overall loss. 5.4 Training efficiency We measure throughput and peak reserved memory for optimized 0.5B models at a sequence length of 32K at training stage. As shown in Table 4, DeltaTTT is 37%37\% faster than full attention and uses 31%31\% less peak memory. It further outperforms serial and parallel LaCT in throughput by 65%65\% and 28%28\%, respectively, with comparable or lower peak memory usage. Table 4: Training efficiency relative to full attention (1.00×1.00×). Model Throughput/GPU ↑ Peak memory/GPU ↓ Full attention 1.00×1.00× 1.00×1.00× DeltaNet 1.42×1.42× 0.53×0.53× LaCT Serial (SwiGLU) 0.83×0.83× 0.68×0.68× LaCT Parallel (SwiGLU) 1.07×1.07× 0.86×0.86× DeltaTTT(ours) 1.37×1.37× 0.69×0.69× 5.5 Ablations Pre versus Post updates. Table 5(a) compares the Pre and Post variants on the DeltaNet backbone. Pre and Post achieve similar training losses and comparable single-needle retrieval accuracy. Pre constructs the second-layer write address from the pre-update state At−1A_t-1, removing its dependence on the current first-layer update and enabling concurrent updates of the two memory modules during inference. Memory components. Starting from the selected DeltaTTT+SWA model in Table 3, we progressively remove memory components in Table 5(b). We first disable online updates to B, retaining its learned initial state B0B_0 and the nonlinear readout. We then remove the remaining static second layer and the intermediate nonlinearity, yielding DeltaNet+SWA with a learnable initial memory A0A_0. Finally, fixing A0=0A_0=0 gives the zero-initialized DeltaNet+SWA baseline, while preserving its online delta-rule updates. The full model achieves the lowest training loss and highest retrieval accuracy. Disabling second-layer updates substantially reduces retrieval accuracy, and retaining a static second layer performs worse than the single-layer model. Removing the learnable initialization further degrades single-layer retrieval. These results highlight the benefits of online adaptation in the second layer and a learned initial memory. Table 5: Ablations at 0.5B model and 92.5B training tokens, Removals in (b) are cumulative; Single is the mean single-needle retrieval accuracy across S1/S2/S3. Model Configuration Train loss ↓ Single ↑ (a) Write address DeltaTTT-Pre (selected) Address from At−1A_t-1 1.962162 54.32 DeltaTTT-Post Address from AtA_t 1.960538 52.22 (b) Progressive component removal DeltaTTT (selected) Full model 1.952344 50.47 DeltaTTT −- B online update 1.967580 36.02 DeltaNet −- Learnable B0B_0 1.969120 40.52 DeltaNet −- Learnable A0A_0 1.970605 25.00 5.6 Mechanism analysis What do the two layers learn? To examine what the two layers learn, we visualize the initial memory matrices A0A_0 and B0B_0 after outer training (Figure 4). Since our main configuration includes a fixed 0.5I0.5I offset in B0B_0, we also inspect a variant in which both A0A_0 and B0B_0 are initialized to zero. The learned B0B_0 in this variant also exhibits a pronounced diagonal structure, suggesting that outer training learns a prior for refinement. This is consistent with the intended division of labor: A produces a value-aligned intermediate representation, and B further adjusts it to recover the value. Figure 4: Learned A0A_0 and B0B_0 at layer 13, head 1. Top: main configuration; bottom: the variant with both matrices initialized to zero. How much do online writes reduce reconstruction loss? Table 6 compares the initial and final full-sequence reconstruction errors of DeltaTTT and two DeltaNet baselines on the same 32K validation sequence. Each checkpoint is evaluated using its self-generated K/V pairs. DeltaTTT reduces MSE by 81.88%, compared with 70.88% for zero-initialized DeltaNet and 76.44% for its learned-initialization variant. These within-model reductions show that our layerwise online updates substantially improve memory reconstruction. Table 6: Initial and final full-sequence reconstruction errors on the same 32K validation sequence. Each checkpoint uses its self-generated K/V features. Reduction denotes the relative decrease in MSE. Model Train tokens Initial MSE Final MSE Reduction DeltaNet 92.5B 0.00408555 0.00118968 70.88% DeltaNet(learned init) 92.5B 0.00466836 0.00109973 76.44% DeltaTTT 92.5B 0.00468315 0.00084878 81.88% 6 Related Work T and nonlinear fast weights. T was introduced for adapting models to distribution shifts through self-supervised updates at test time (Sun et al., 2020). For sequence modeling, T layers represent recurrent memory as an inner learner updated through local self-supervised objectives (Sun et al., 2024). LaCT scales nonlinear fast weights using large update chunks (Zhang et al., 2025), while Titans incorporates momentum and forgetting into online neural memory (Behrouz et al., 2025c). TNT addresses the chunk-size tradeoff through hierarchical memories and subsequent fine-tuning at smaller chunk sizes (Li et al., 2026b). E2-T preserves token-dependent learning rates, momentum, and decay in a parallel chunk update under shared chunk-start gradient evaluation (Zhong et al., 2026). Modular T systematically studies inner-learner design and finds that deeper fast-weight networks can underperform shallow alternatives in its evaluated one-step setting (Tang et al., 2026). Complementing this observation, Liu et al. (2026) show that better inner KV fitting need not improve downstream performance and analyze connections between KV-binding T and learned linear attention. Other work changes the adaptation objective: T-E2E uses next-token prediction and meta-learns an initialization for adaptation (Tandon et al., 2025), while TTCD supervises fast weights using a long-window teacher and a short-window student (Wang et al., 2026b). TTCD’s off-policy implementation constructs updates from base-weight activations and accumulates them causally, providing a related instance of fixed-base updating. In vision, ViT3 studies inner architectures and training rules (Han et al., 2026), while T5 converts pretrained vision Transformers into T models through architectural and representational alignment (Li et al., 2026a). Our work focuses on layerwise optimization of nonlinear recurrent memory within a single pass, with reconstruction error serving as a diagnostic alongside downstream evaluation. Delta-rule memories and online optimization. Linear attention admits a recurrent formulation that accumulates key–value associations (Katharopoulos et al., 2020). Schlag et al. (2021) connect linear attention to fast weight programmers and introduce delta-rule writes that correct existing associations. Yang et al. (2024) develop efficient chunkwise evaluation of this recurrence, and Gated DeltaNet combines delta-rule corrections with adaptive forgetting (Yang et al., 2025). Beyond linear memories, Irie et al. (2021) explore recurrent and multilayer fast networks, including Delta MLPs that use layer-specific keys and values generated from the same input token to update a multilayer fast-weight memory via the delta rule. Test-time regression and MIRAS provide broader frameworks for understanding associative memories through their function classes, objectives, and learning rules (Wang et al., 2026a; Behrouz et al., 2025b). ATLAS optimizes memory over a recent context window (Behrouz et al., 2025a), while MesaNet uses a conjugate-gradient solver for in-context regression (von Oswald et al., 2026). DeltaTTT assigns explicit value-reconstruction targets to each linear layer and constructs the second-layer write address from the first layer’s prediction. Its local objectives replace joint inner-loop optimization, while the nonlinear intermediate representation couples the two memory stages and each stage reuses chunkwise delta-rule evaluation. Layerwise and local learning. Difficulties in training deep networks motivated early methods that learned representations one layer at a time. Deep belief networks and greedy layerwise pretraining used successive shallow learning problems to initialize deep models before global fine-tuning (Hinton et al., 2006; Bengio et al., 2006). Stacked denoising autoencoders similarly learned deep representations through local reconstruction objectives (Vincent et al., 2010). Later work explored layerwise targets (Lee et al., 2015), synthetic gradients that relax update dependencies (Jaderberg et al., 2017), and greedy supervised learning at ImageNet scale (Belilovsky et al., 2019). Decoupled Greedy Learning further uses local objectives to relax update locking between network modules (Belilovsky et al., 2020). Our motivation shares the idea of making a deep composition easier to optimize through simpler local learning problems. In DeltaTTT, these problems govern the online updates of context-dependent memory states, while the outer language model remains trained end to end. 7 Conclusion Our findings suggest that greater memory expressivity alone does not guarantee effective learning within a single pass. We introduced DeltaTTT, which decomposes nonlinear memory optimization into local linear reconstruction objectives with layerwise delta-rule updates, preserving nonlinear composition and enabling chunkwise parallel evaluation. Experiments on two backbones demonstrate improvements in language modeling and associative retrieval. 8 Limitations Our current DeltaTTT remains slower and uses more memory than DeltaNet. Since the kernels for our two-layer memory are not yet fully optimized, we believe there is room to reduce this overhead through further implementation improvements. In addition, our experiments only consider two-layer MLP memories with a hidden expansion ratio of 1×1×. Extending the approach to wider intermediate representations and deeper nonlinear memories remains an important direction for assessing whether layerwise optimization can effectively exploit greater memory capacity. Finally, our evaluation is limited to relatively small models and a bounded training-token budget. Further experiments with larger models and more training data are needed to establish the scaling potential of DeltaTTT. AI use statement Generative AI assisted with organizing the manuscript and drafting text and equations from existing experiment records in this revision. No new training or evaluation was performed for this manuscript edit. Reproducibility statement Section 4 specifies the memory updates and readout, and Section 5 describes the experimental setup and evaluation. Appendix A provides the chunkwise algorithms for DeltaNet and DeltaTTT-pre. Appendix B details the fixed-KV reconstruction experiments, including data selection, memory architectures, learning-rate tuning, and evaluation. Appendix C documents the shared training configuration, backbone-specific implementations, and main-table DeltaTTT settings. Appendix D reports ablations of the learned initial memory states. References Behrouz et al. (2025a) A. Behrouz, Z. Li, P. Kacham, M. Daliri, Y. Deng, P. Zhong, M. Razaviyayn, and V. Mirrokni ATLAS: Learning to Optimally Memorize the Context at Test Time. arXiv preprint arXiv:2505.23735. External Links: Link Cited by: §6. Behrouz et al. (2025b) A. Behrouz, M. Razaviyayn, P. Zhong, and V. Mirrokni It’s All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimization. arXiv preprint arXiv:2504.13173. External Links: Link Cited by: §6. Behrouz et al. (2025c) A. Behrouz, P. Zhong, and V. Mirrokni Titans: learning to memorize at test time. arXiv preprint arXiv:2501.00663. External Links: Link Cited by: §6. Belilovsky et al. (2019) E. Belilovsky, M. Eickenberg, and E. Oyallon Greedy layerwise learning can scale to ImageNet. In Proceedings of the International Conference on Machine Learning, p. 583–593. External Links: Link Cited by: §6. Belilovsky et al. (2020) E. Belilovsky, M. Eickenberg, and E. Oyallon Decoupled Greedy Learning of CNNs. In Proceedings of the International Conference on Machine Learning, External Links: Link Cited by: §6. Bengio et al. (2006) Y. Bengio, P. Lamblin, D. Popovici, and H. Larochelle Greedy layer-wise training of deep networks. In Advances in Neural Information Processing Systems, B. Schölkopf, J. Platt, and T. Hoffman (Eds.), Vol. 19, p. . External Links: Link Cited by: §6. Han et al. (2026) D. Han, Y. Li, T. Li, Z. Cao, Z. Wang, J. Song, Y. Cheng, B. Zheng, and G. Huang ViT3^3: unlocking test-time training in vision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, External Links: Link Cited by: §2, §6. Hinton et al. (2006) G. E. Hinton, S. Osindero, and Y. Teh A fast learning algorithm for deep belief nets. Neural Computation. External Links: Link Cited by: §6. Hsieh et al. (2024) C. Hsieh, S. Sun, S. Kriman, S. Acharya, D. Rekesh, F. Jia, Y. Zhang, and B. Ginsburg RULER: what’s the real context size of your long-context language models?. arXiv preprint arXiv:2404.06654. External Links: Link Cited by: §5.1. Irie et al. (2021) K. Irie, I. Schlag, R. Csordás, and J. Schmidhuber Going beyond linear transformers with recurrent fast weight programmers. In Advances in Neural Information Processing Systems, Vol. 34. External Links: Link Cited by: §5.1, §6. Jaderberg et al. (2017) M. Jaderberg, W. M. Czarnecki, S. Osindero, O. Vinyals, A. Graves, D. Silver, and K. Kavukcuoglu Decoupled neural interfaces using synthetic gradients. In Proceedings of ICML, External Links: Link Cited by: §6. Katharopoulos et al. (2020) A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention. In Proceedings of the International Conference on Machine Learning, External Links: Link Cited by: §6. Lee et al. (2015) D. Lee, S. Zhang, A. Fischer, and Y. Bengio Difference target propagation. In Machine Learning and Knowledge Discovery in Databases, External Links: Link Cited by: §6. Li et al. (2026a) Y. Li, D. Han, Z. Liu, H. Wang, Y. Wang, and G. Huang Linearizing vision transformer with test-time training. In Proceedings of the International Conference on Machine Learning, External Links: Link Cited by: §6. Li et al. (2026b) Z. Li, A. Behrouz, Y. Deng, P. Zhong, P. Kacham, M. Karami, M. Razaviyayn, and V. Mirrokni TNT: Improving Chunkwise Training for Test-Time Memorization. In International Conference on Learning Representations, External Links: Link Cited by: §6. Liu et al. (2026) J. Liu, S. Elflein, O. Litany, Z. Gojcic, and R. Li Test-Time Training with KV Binding Is Secretly Linear Attention. In Proceedings of the International Conference on Machine Learning, External Links: Link Cited by: §6. Schlag et al. (2021) I. Schlag, K. Irie, and J. Schmidhuber Linear transformers are secretly fast weight programmers. In Proceedings of the International Conference on Machine Learning, p. 9355–9366. External Links: Link Cited by: §6. Sun et al. (2024) Y. Sun, X. Li, K. Dalal, J. Xu, A. Vikram, G. Zhang, Y. Dubois, X. Chen, X. Wang, S. Koyejo, T. Hashimoto, and C. Guestrin Learning to (learn at test time): RNNs with expressive hidden states. arXiv preprint arXiv:2407.04620. External Links: Link Cited by: §1, §2, §3, §3, Table 2, Table 2, §6. Sun et al. (2020) Y. Sun, X. Wang, Z. Liu, J. Miller, A. Efros, and M. Hardt Test-time training with self-supervision for generalization under distribution shifts. In Proceedings of the International Conference on Machine Learning, p. 9229–9248. External Links: Link Cited by: §6. Tandon et al. (2025) A. Tandon, K. Dalal, X. Li, D. Koceja, M. Rød, S. Buchanan, X. Wang, J. Leskovec, S. Koyejo, T. Hashimoto, C. Guestrin, J. McCaleb, Y. Choi, and Y. Sun End-to-end test-time training for long context. arXiv preprint arXiv:2512.23675. External Links: Link Cited by: §6. Tang et al. (2026) B. Tang, Z. Qin, Y. Pan, Z. Li, P. Liu, and Y. Zhang Modular T: Rethinking Test-Time Training as Composable Modules. arXiv preprint arXiv:2608.07110. External Links: Link Cited by: §6. Vaswani et al. (2017) A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin Attention is all you need. In Advances in Neural Information Processing Systems, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol. 30, p. . External Links: Link Cited by: §5.1. Vincent et al. (2010) P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio, and P. Manzagol Stacked denoising autoencoders: learning useful representations in a deep network with a local denoising criterion. Journal of Machine Learning Research 11, p. 3371–3408. External Links: Link Cited by: §6. von Oswald et al. (2026) J. von Oswald, N. Scherrer, S. Kobayashi, L. Versari, S. Yang, S. Mittal, M. Schlegel, K. Maile, Y. Schimpf, O. Sieberling, A. Meulemans, R. A. Saurous, G. Lajoie, C. Frenkel, R. Pascanu, B. Agüera y Arcas, and J. Sacramento MesaNet: Sequence Modeling by Locally Optimal Test-Time Training. In International Conference on Learning Representations, External Links: Link Cited by: §6. Wang et al. (2026a) K. A. Wang, J. Shi, and E. B. Fox Test-time regression: a unifying framework for designing sequence models with associative memory. Journal of Machine Learning Research 27 (169), p. 1–41. External Links: Link Cited by: §6. Wang et al. (2026b) Z. Wang, X. Dang, R. Zhu, Z. Wen, H. Fu, W. Chai, and J. D. Lee Learning What to Remember: Test-Time Training via Context Distillation. arXiv preprint arXiv:2608.01672. External Links: Link Cited by: §6. Yang et al. (2025) S. Yang, J. Kautz, and A. Hatamizadeh Gated delta networks: improving Mamba2 with delta rule. In International Conference on Learning Representations, External Links: Link Cited by: §5.1, §6. Yang et al. (2024) S. Yang, B. Wang, Y. Zhang, Y. Shen, and Y. Kim Parallelizing linear transformers with the delta rule over sequence length. arXiv preprint arXiv:2406.06484. External Links: Link Cited by: §1, §2, §3, §3, §3, §4.1, Table 2, §5.1, §6. Zhang et al. (2025) T. Zhang, S. Bi, Y. Hong, K. Zhang, F. Luan, S. Yang, K. Sunkavalli, W. T. Freeman, and H. Tan Test-time training done right. arXiv preprint arXiv:2505.23884. External Links: Link Cited by: §1, §2, §3, §3, Table 2, §5.1, §6. Zhong et al. (2026) Z. Zhong, J. Chen, M. Martin, F. Diederichs, J. Gall, and J. Beyerer Rethinking expressivity and efficiency in test-time training. arXiv preprint arXiv:2608.21308. External Links: Link Cited by: Table 2, §6. Appendix A Chunkwise Scan of DeltaTTT Algorithms 1 and 2 describe one memory head. The sequence is divided into N chunks of up to C tokens, with Qc,Kc,VcQ_c,K_c,V_c stacking token vectors as rows. W denotes DeltaNet’s memory, while A and B denote DeltaTTT’s first- and second-layer memories, respectively. c η_c, c α_c, and c β_c contain the tokenwise write rates for W, A, and B, respectively. For an incoming memory M, the chunkwise writes are Delta(K,V,,M)=[I+Dtril(KK⊤,−1)]−1D(V−KM⊤),D=diag().Delta(K,V, r;M)= [I+Dtril(K ,-1) ]^-1D(V-KM ), D=diag( r). (13) Here I is the identity of the chunk length, and the inverse denotes a triangular solve. tril(X)tril(X) includes the diagonal, whereas tril(X,−1)tril(X,-1) excludes it. ϕ=SiLUφ=SiLU is applied elementwise. In DeltaTTT-pre, HcH_c and RcR_c are the first-layer pre-update key and post-update query representations, respectively. UcU_c stores write-rate-scaled residuals, and [O1;…;ON][O_1;…;O_N] concatenates the output chunks. Algorithm 1 Chunkwise DeltaNet: one scan 1: Chunks Qc,Kc,Vc,cc=1N\Q_c,K_c,V_c, η_c\_c=1^N; initial memory W0W_0 2: W←W0W← W_0 3: for c=1,…,Nc=1,…,N do 4: Uc←Delta(Kc,Vc,c,W)U_c (K_c,V_c, η_c;W) 5: Oc←QcW⊤+tril(QcKc⊤)UcO_c← Q_cW +tril(Q_cK_c )U_c 6: W←W+Uc⊤KcW← W+U_c K_c 7: end for 8: return [O1;…;ON],W[O_1;…;O_N],W Algorithm 2 Chunkwise DeltaTTT-pre: two successive scans 1: Chunks Qc,Kc,Vc,c,cc=1N\Q_c,K_c,V_c, α_c, β_c\_c=1^N; initial memories A0,B0A_0,B_0 2: A←A0A← A_0; B←B0B← B_0 3: for c=1,…,Nc=1,…,N do ⊳ First scan: one update, two reads 4: Uc←Delta(Kc,Vc,c,A)U_c (K_c,V_c, α_c;A) 5: Hc←ϕ(KcA⊤+tril(KcKc⊤,−1)Uc)H_c←φ\! (K_cA +tril(K_cK_c ,-1)U_c ) 6: Rc←ϕ(QcA⊤+tril(QcKc⊤)Uc)R_c←φ\! (Q_cA +tril(Q_cK_c )U_c ) 7: A←A+Uc⊤KcA← A+U_c K_c 8: end for 9: for c=1,…,Nc=1,…,N do ⊳ Second scan: one update, one read 10: Uc←Delta(Hc,Vc,c,B)U_c (H_c,V_c, β_c;B) 11: Oc←RcB⊤+tril(RcHc⊤)UcO_c← R_cB +tril(R_cH_c )U_c 12: B←B+Uc⊤HcB← B+U_c H_c 13: end for 14: return [O1;…;ON],A,B[O_1;…;O_N],A,B Appendix B Fixed-KV Reconstruction Experiments Figure 2 compares linear and nonlinear memories fitted to fixed K/V caches from a frozen 0.5B full-attention model trained on approximately 10B tokens. We use eight 32K sequences and all 24 layers, concatenating heads to a feature dimension of 1,024. Four sequences are used to select learning rates, and the remaining four are used to report results. We compare a zero-initialized linear memory with a two-layer SiLU MLP of hidden width 1x/2x/4x and Xavier-uniform initialization. Both memories are optimized using ordered SGD replay, with chunk sizes ranging from 1 to 512 in powers of two. Each chunk C of size C produces one update under the key-normalized squared loss: ℒ(θ)=12C∑i∈‖fθ(ki)−vi‖22‖ki‖22.L_C(θ)= 12C _i \|f_θ(k_i)-v_i\|_2^2\|k_i\|_2^2. (14) The MLP updates both matrices simultaneously, and initializations are shared across chunk sizes. Learning rates are tuned separately for each model and chunk size using reconstruction MSE after 16 epochs; the MLP uses a separate rate for each layer. The selected rates remain fixed for evaluation. We report MSE reconstruction loss over each complete fitted sequence, averaged across the four reporting sequences and all layers. The tokenwise Delta reference corresponds to the linear memory with C=1C=1. Equal numbers of passes do not imply equal computational cost: smaller chunks require more updates. Appendix C Main-Table Experimental Details Shared training setup. Table 7 summarizes the shared training configuration. All recurrent models use SWA with a 512-token window. Table 7: Shared training configuration Setting Value Training data Long-Data-Collections, 92.5B tokens Model dimensions 24 layers; hidden width 1,024; FFN width 4,096 Vocabulary and embeddings 131,072 tokens; tied input/output embeddings Sequence length 32,768 tokens Optimizer AdamW, (β1,β2)=(0.9,0.95)( _1, _2)=(0.9,0.95), ϵ=10−8ε=10^-8 Weight decay / gradient clipping 0.1 / 1.0 Learning rate Peak 3×10−43× 10^-4; cosine decay to 3×10−53× 10^-5 Warmup Approximately 1.05B tokens Backbone implementations. We implement DeltaTTT within both DeltaNet and LaCT backbones, replacing their recurrent memory modules while retaining backbone-specific feature processing and output integration. The main differences concern positional encoding and memory output modulation: the DeltaNet backbone uses RoPE only in the SWA branch and directly adds the RMS-normalized memory output to the SWA output, whereas the LaCT backbone also applies RoPE to memory Q/K and uses an input-dependent scale to modulate the normalized memory output. LaCT additionally retains learned affine transforms on the memory inputs. DeltaTTT configuration. We align the main architectural settings with our DeltaNet baseline: both implementations use 16 memory heads of width 64, causal depthwise Conv4 with SiLU, L2-normalized memory Q/K. The memory and SWA branches share Q/K/V projections, and their outputs are summed before the output projection. Both main-table DeltaTTT models use Pre updates for memory layers. The learned initial states use A0∼(0,0.022)A_0 (0,0.02^2) and B0=LR+0.5IB_0=LR+0.5I, where L,RL,R are learned rank-32 factors. Appendix D Initial-State Ablation Table 8 shows that different learnable initializations yield similar training loss and single-needle retrieval accuracy. Table 8: Initial-state ablations for DeltaTTT on the DeltaNet backbone. Single denotes average single-needle accuracy (%). Model Initial states Train loss ↓ Single ↑ DeltaTTT A0=B0=0A_0=B_0=0 (learnable) 1.95 47.70 DeltaTTT (A0)ij,(B0)ij∼(0,0.022)(A_0)_ij,(B_0)_ij (0,0.02^2) 1.95 48.00 DeltaTTT (A0)ij∼(0,0.022),B0=LR+0.5I(A_0)_ij (0,0.02^2),\ B_0=LR+0.5I 1.95 48.78