Paper deep dive
Test-Time Training with KV Binding Is Secretly Linear Attention
Junchen Liu, Sven Elflein, Or Litany, Zan Gojcic, Ruilong Li
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 7/20/2026, 1:11:07 PM
Summary
The paper challenges the prevailing interpretation of Test-Time Training (TTT) with Key-Value (KV) binding as online meta-learning or test-time memorization. Through empirical analysis of anomalies such as distributional asymmetry between queries and keys, and the counterintuitive preservation of performance under gradient ascent, the authors demonstrate that TTT behaves inconsistently with a storage-and-retrieval model. Theoretically, the paper proves that TTT architectures with complex inner loops can be mathematically reformulated as learned linear attention operators. This reframing enables architectural simplifications, parallel formulations, and improved efficiency, suggesting TTT is fundamentally a feature-mixing mechanism rather than a memory system.
Entities (7)
Relation Signals (5)
Test-Time Training → isreinterpretedas → Linear Attention
confidence 95% · we show that a broad class of TTT architectures can be expressed as a form of learned linear attention operator
Test-Time Training → contradictsinterpretation → Online Meta-Learning
confidence 93% · our analysis reveals multiple phenomena that contradict this memorization-based interpretation
LaCT → isvariantof → Test-Time Training
confidence 92% · rewrite two representative variants of TTT, LaCT (Zhang et al., 2025)
ViTTT → isvariantof → Test-Time Training
confidence 92% · rewrite two representative variants of TTT, ... and ViTTT (Han et al., 2025)
Gradient Ascent → preservesperformancein → Test-Time Training
confidence 90% · replacing inner-loop gradient descent with gradient ascent always preserves, and in some cases even improves, task performance
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Test-time training (TTT) with KV binding as sequence modeling layer is commonly interpreted as a form of online meta-learning that memorizes a key-value mapping at test time. However, our analysis reveals multiple phenomena that contradict this memorization-based interpretation. Motivated by these findings, we revisit the formulation of TTT and show that a broad class of TTT architectures can be expressed as a form of learned linear attention operator. Beyond explaining previously puzzling model behaviors, this perspective yields multiple practical benefits: it enables principled architectural simplifications, admits fully parallel formulations that preserve performance while improving efficiency, and provides a systematic reduction of diverse TTT variants to a standard linear attention form. Overall, our results reframe TTT not as test-time memorization, but as learned linear attention with enhanced representational capacity. Project page: this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2602.21204v4
- Canonical: https://arxiv.org/abs/2602.21204v4
Trouble viewing inline? Open PDF directly →
Full Text
80,700 characters extracted from source content.
Expand or collapse full text
Test-Time Training with KV Binding Is Secretly Linear Attention Junchen Liu Sven Elflein Or Litany Zan Gojcic Ruilong Li Abstract Test-time training (T) with KV binding as sequence modeling layer is commonly interpreted as a form of online meta-learning that memorizes a key–value mapping at test time. However, our analysis reveals multiple phenomena that contradict this memorization-based interpretation. Motivated by these findings, we revisit the formulation of T and show that a broad class of T architectures can be expressed as a form of learned linear attention operator. Beyond explaining previously puzzling model behaviors, this perspective yields multiple practical benefits: it enables principled architectural simplifications, admits fully parallel formulations that preserve performance while improving efficiency, and provides a systematic reduction of diverse T variants to a standard linear attention form. Overall, our results reframe T not as test-time memorization, but as learned linear attention with enhanced representational capacity. Project page: https://research.nvidia.com/labs/sil/projects/tttla/. Machine Learning, ICML 1 Introduction Test-Time Training (T) has emerged as a powerful paradigm for dynamic model adaptation. Initially introduced to address distribution shift by updating model parameters on unlabeled test inputs, T has since evolved into a distinct architectural primitive (Sun et al., 2020). Recent work has increasingly framed T as an alternative to standard softmax attention in transformers, offering favorable properties like linear-time compute and constant memory usage during autoregressive inference (Zhang et al., 2025; Behrouz et al., 2026). Among the various T formulations, this paper focuses on T with KV binding (Sun et al., 2025; Zhang et al., 2025; Han et al., 2025; Behrouz et al., 2026), which optimizes a self-supervised key-value association objective in the inner loop, as opposed to end-to-end methods that backpropagate from the final task loss (Tandon et al., 2025; Behrouz et al., 2025c). The prevailing interpretation of T casts it as a form of online meta-learning or memorization (Sun et al., 2025; Finn et al., 2017; Metz et al., 2018), where the “inner loop” dynamically constructs a temporary key-value (KV) map by optimizing a neural network (e.g., an MLP) on previously observed tokens. The subsequent inference step is viewed as querying this stored knowledge. This perspective has led to increased complexity in recent architectural designs, motivating the use of sophisticated optimizers, normalization schemes, and deep inner-loop networks (Zhang et al., 2025; Han et al., 2025; Behrouz et al., 2026, 2025a; Dalal et al., 2025), all explicitly designed to improve the fidelity of this “memorization”. The Memorization Paradox. Yet, this interpretation of T as “test-time memorization” can be directly contradicted by empirical evidence. If T would truly function by explicitly learning and retrieving key–value associations, its behavior would conform to basic principles of memory formation and optimization dynamics. Contrary to this expectation, we identify systematic anomalies that directly contradict this hypothesis: • Distributional Asymmetry. Unlike standard attention, in which queries and keys share the same semantic space, converged T models exhibit a significant distributional mismatch between queries and keys. • Replacing queries with keys. Replacing queries with keys for T models has negligible effect on the task performance, suggesting that queries do not play a functional retrieval role as in standard attention. • Optimization vs. Performance. Counterintuitively, improvements in the inner loop, which can be interpreted as stronger “memorization”, do not guarantee better downstream performance. • The Gradient Ascent Anomaly. Most strikingly, we find that replacing inner-loop gradient descent with gradient ascent always preserves, and in some cases even improves, task performance. These observations collectively challenge the prevailing view of T as an online meta-learning or key-value memorization mechanism. T is Secretly Linear Attention. Motivated by these observations, and by prior work showing that T reduces to linear attention in the restricted case of a single linear inner-loop layer with zero initialization (Sun et al., 2025), we revisit the mathematical formulation of T. We show analytically that even T variants with complex fast-weight parameterizations (including multi-layer MLPs and momentum) can be equivalently rewritten as a form of learned linear attention operator (Katharopoulos et al., 2020). Under this unified view, the inner loop does not perform “meta learning” in the conventional sense. Instead, it induces a structured, history-dependent mixing of query, key, and value vectors. This perspective resolves the empirical paradoxes identified above: gradient ascent preserves performance because sign inversions are absorbed into the learned value projection, and distributional symmetry between queries and keys is unnecessary because the mechanism operates as a feature mixer rather than a similarity-based retrieval system. Practical Implications. Unmasking T as Linear Attention is not just a theoretical exercise; it unlocks significant practical benefits. By adopting this perspective, we: • Simplify. We show that many components introduced in prior T architectures, such as weight normalization and momentum, are often redundant. • Parallelize. We derive a fully parallel form of T that achieves up to 4.0× inference throughput on attention calculation while maintaining performance. • Generalize. We provide a systematic reduction of diverse T variants to a common linear attention form. Empirically, the full T model exceeds this reduced linear-attention variant only marginally (+0.87+0.87 perplexity on LLM, +0.24+0.24 dB on NVS), suggesting that the additional inner-loop machinery contributes only modest gains beyond what standard linear attention already captures. Conflict of Interest Disclosure. This work was conducted at NVIDIA. J.L., S.E., O.L., Z.G., and R.L. are employed by or affiliated with NVIDIA. The paper analyzes publicly described T architectures and evaluates open-source implementations rather than NVIDIA-developed products or systems. The authors do not report additional financial conflicts of interest. 2 Related Work 2.1 Linear Attention Recurrent Neural Networks (RNNs) have recently gained renewed interest as efficient alternatives to standard Transformers (Vaswani et al., 2017). The introduction of linear attention (Katharopoulos et al., 2020) has accelerated advances in RNN-based architectures, leading to the incorporation of various mechanisms such as token-dependent decay factors (Gu et al., 2022; Smith et al., 2023; Orvieto et al., 2023; Peng et al., 2023; Sun et al., 2023) and, more recently, data-dependent decay (Qin et al., 2023, 2024; Peng et al., 2024; Gu and Dao, 2024; Dao and Gu, 2024; Zhang et al., 2024; Yang et al., 2024a). The data-dependent decay factor, termed the selective mechanism in Mamba (Gu and Dao, 2024; Dao and Gu, 2024), has been highlighted as crucial for strong in-context learning performance. DeltaNet (Schlag et al., 2021) conditions the update rule on both the current token and state for improved retrieval, and chunk-parallelization (Yang et al., 2024b) has enabled its efficient deployment in many recent architectures (Yang et al., 2024a; Peng et al., 2025a; Behrouz et al., 2026; Zhong et al., 2025; Team et al., 2025; Peng et al., 2025b; Lei et al., 2025; Liu et al., 2025). Notably, DeltaNet and its variants are equivalent to T with a single linear layer and MSE loss (Yang et al., 2024a). 2.2 Test-Time Training Test-time training (T) broadly refers to methods that continue to update model parameters during inference. This concept was first introduced to address train-test distribution shift (Sun et al., 2020; Gandelsman et al., 2022), where models adapt to test data by optimizing a self-supervised objective at inference time. Subsequently, T has been explored for improving inference-time performance in specific applications such as 3D reconstruction (Chen et al., 2024, 2025; Yuan et al., 2025). More recently, T has been developed as a sequence modeling architecture with linear complexity, serving as an alternative to softmax attention in transformers (Sun et al., 2025; Wang et al., 2025; Zhang et al., 2025; Han et al., 2025; Dalal et al., 2025; Behrouz et al., 2026). When used as a sequence modeling layer, T has two main variants: (1) methods that use a key-value binding loss (e.g., dot-product or MSE loss) as the inner-loop objective (Sun et al., 2025; Zhang et al., 2025; Han et al., 2025; Behrouz et al., 2026), referred to as T-KVB in prior work (Tandon et al., 2025), and (2) methods that perform end-to-end backpropagation through the inner loop from the final task loss (e.g., cross-entropy in language modeling), referred to as T-E2E (Tandon et al., 2025; Behrouz et al., 2025c). This paper focuses on the first variant. T has demonstrated effectiveness across diverse tasks, including language modeling (Sun et al., 2025; Wang et al., 2025; Zhang et al., 2025), video generation (Dalal et al., 2025; Zhang et al., 2025), novel view synthesis (Zhang et al., 2025), and image classification (Han et al., 2025). In this paradigm, part of the model parameters (known as fast weights (Hinton and Plaut, 1987)) are updated online at test time using a self-supervised key-value association loss, with the goal of memorizing history associations. Since this nested optimization is performed in-context, T is also referred to as in-context meta-learning (Finn et al., 2017; Metz et al., 2018) and is tightly related to the concept of fast weight programming (Schlag et al., 2021) and test-time scaling (Muennighoff et al., 2025; Snell et al., 2024). The design space of T is rich and has been actively explored. LaCT (Zhang et al., 2025) improves hardware utilization through large chunk sizes. Based on the key-value memorization perspective, other works have explored advanced test-time optimizers (Behrouz et al., 2026; Zhang et al., 2025; Karami et al., 2025) and alternative regression targets (Han et al., 2025; Behrouz et al., 2025b). Notably, prior work has shown that when the inner loop consists of a single linear layer, T can be reinterpreted as a linear attention operator (Sun et al., 2025). In this work, we demonstrate that this interpretation extends to general T architectures with complex multi-layer MLPs as inner loops, and show that this perspective enables practical benefits including simplified formulations and efficient parallel implementations. Figure 1: Inner-Loop Optimization vs. Performance. Increasing inner-loop iterations improves inner-loop loss but degrades task performance, contradicting the memorization-based interpretation of T. Experiments are based on LaCT (Zhang et al., 2025). 3 Preliminary We provide a brief overview of the T mechanism that this paper focuses on. In T, each sequence modeling layer maintains a set of fast weights fθf_θ (typically a lightweight MLP) that is updated during both training and inference. Given an input sequence, tokens are first projected into keys K, values V, and queries Q (similar to standard attention). The core idea is to perform online gradient descent on fθf_θ using a self-supervised key-value binding objective: for each token, the key k serves as input and the value v serves as the regression target, i.e., ℒ=‖fθ(k)−v‖2L=\|f_θ(k)-v\|^2 (or a dot-product loss variant). After updating θ with this objective, the query q is passed through the updated function fθf_θ to produce the output. This mechanism is commonly interpreted as a storage-and-retrieval system (Sun et al., 2025; Finn et al., 2017): the inner-loop optimization “memorizes” key-value associations into fθf_θ, which are later “retrieved” by querying the learned function. Under this view, architectural capacity, optimizer selection, and the number of inner-loop steps are all motivated by achieving more faithful memorization of key–value associations. This variant of test-time training, which optimizes a key-value binding objective in the inner loop, is referred to as T-KVB in prior work (Tandon et al., 2025). Among the various T formulations (Sun et al., 2025; Gandelsman et al., 2022; Sun et al., 2020; Tandon et al., 2025; Behrouz et al., 2025c), this paper exclusively focuses on this variant. 4 Empirical Contradictions to Memorization As we show in this section, the empirical behavior of T models consistently violates the properties implied by the storage-and-retrieval interpretation. 4.1 Better Inner Loss Leads to Worse Performance Under a memorization-based interpretation, inner-loop loss serves as a natural proxy for memorization quality: lower loss indicates more accurate encoding of key–value mappings and should therefore improve task performance. To test this property, we vary the number of inner-loop gradient steps at inference time, which usually yields lower inner-loop loss in pretrained T models (Zhang et al., 2025; Han et al., 2025), without changing the architecture, training data, or learned parameters. Figure 1 shows that despite improved inner-loop fitting, downstream performance degrades consistently as the number of inner-loop steps increases. This inverse relationship holds across both LLMs and novel view synthesis (NVS) tasks. Such behavior directly contradicts the memorization-based interpretation of T, under which more accurate key–value fitting should be beneficial, or at least not harmful for the downstream performance. Instead, these results indicate that the inner loop affects model computation in a fundamentally different manner, inconsistent with conventional notions of test-time memory. 4.2 T with Gradient Ascent The degradation in downstream performance with improved inner-loop fitting raises a more fundamental question: is gradient-descent-based memorization in the inner loop necessary at all? To test this, we replace gradient descent in the inner loop with gradient ascent, effectively flipping the sign of all fast-weight gradients. Importantly, this is not a pure inference-time intervention on a pretrained checkpoint: each gradient-ascent model is retrained from scratch with the sign-flipped inner-loop update applied throughout both training and inference, so the surrounding parameters have a chance to adapt to the inverted objective. Under a memorization-based interpretation, this change should still have a strong negative effect, as gradient ascent explicitly worsens the fit to the key–value regression objective. Contrary to this expectation, across all evaluated models and tasks, T with gradient ascent performs comparably to, and in some cases even slightly better than, standard gradient descent (Table 1). This holds despite the fact that gradient ascent consistently increases the inner-loop loss. Notably, in methods such as LaCT (Zhang et al., 2025), where the inner-loop loss is a Frobenius inner product, flipping the gradient is equivalent to negating the loss itself. The fact that T remains effective even under such an inverted objective provides additional evidence that T does not rely on memorizing a key–value mapping. Table 1: Observations Contradicting the Storage-and-Retrieval Interpretation of T. Replacing gradient descent with ascent breaks the storage interpretation, and replacing queries with keys breaks the retrieval interpretation, yet task performance remains mostly unchanged. Experiments are base on LaCT (Zhang et al., 2025) and ViTTT (Han et al., 2025). Model Perplexity ↓ (LaCT-LLM) PSNR ↑ (LaCT-NVS) Top-1 Acc↑ (ViTTT) Baseline 16.43 25.94 79.34 Gradient Ascent 16.19 25.85 79.61 Replace Q with K 16.18 25.95 79.18 4.3 Distributional Asymmetry Between Q and K Figure 2: Distributional Asymmetry Between Q and K. t-SNE visualizations of (Q,K)(Q,K) and (V,O)(V,O) features in a pretrained LaCT (Zhang et al., 2025) model on the NVS task, showing that the T inner loop is evaluated out of distribution and thus does not perform reliable retrieval. As we mentioned above, T is commonly interpreted as a storage-and-retrieval mechanism. In the previous subsections, we showed that the storage aspect of this interpretation is empirically inconsistent with model behavior. We now examine whether the retrieval aspect provides a plausible explanation. During the T inner-loop update, the parametric function (e.g., MLP) is optimized using keys K as inputs and values V as regression targets. After this update, queries Q are feed into the same function to produce outputs O. For such a retrieval process to be effective, the distribution of Q must be close to that of the K used during optimization. Otherwise, the parametric function is evaluated out-of-distribution, and its outputs are not expected to reliably lie on the manifold of V. In other words, meaningful retrieval requires substantial distributional overlap between Q and K. To test this assumption, we analyze the distributions of Q and K in a pretrained LaCT (Zhang et al., 2025) model on the NVS task. For each layer, we collect Q and K vectors across all tokens and visualize their distributions using t-SNE (Maaten and Hinton, 2008). As shown in Figure 2, we observe a pronounced and consistent mismatch between Q/K and V/O across layers, indicating that the parametric function optimized on keys, is systematically evaluated on out-of-distribution inputs when applied to queries, at both training and inference time. Under such conditions, the resulting outputs cannot be interpreted as reliable retrieval of stored key–value information, and this mismatch persists regardless of how accurately the inner loop fits the key–value regression objective. 4.4 Replacing Q with K The distributional asymmetry between queries and keys again raises a more direct question: is the query representation Q necessary at all for T to function? To test this, we perform a simple experiment in which we replace the query Q with the key K when computing the T output. In standard attention mechanisms, such a substitution would be highly disruptive. Because attention weights depend on query–key similarity, replacing Q with K typically leads to degenerate behavior dominated by self-similarity and a substantial drop in performance. Contrary to the expectation, we observe little to no degradation in T’s performance. For both LaCT and ViTTT formulations performance remains comparable (Table 1). This insensitivity to the query representation contradicts a retrieval-based interpretation of T. If the inner loop relied on query-based access to stored information, replacing Q with K should significantly alter model behavior. Instead, the observed invariance indicates that Q does not function as a meaningful query into a memorized key-value map. Summary. Taken together, these observations challenge the interpretation of test-time training as a storage-and-retrieval mechanism. Downstream behavior is largely insensitive to both the quality and direction of inner-loop optimization, and to the presence of a meaningful query signal. These failures suggest that in practice T does not operate as test-time memorization. 5 T Is Secretly Linear Attention If T is not functioning as a memorization mechanism, how should its behavior be understood? An important clue comes from prior work, which shows that in the restricted setting of a single linear inner-loop layer with zero initialization, T is exactly equivalent to linear attention (Sun et al., 2025). More generally, despite their apparent complexity, existing T variants share the defining computational properties of linear attention: linear-time computation and a constant-size state with respect to sequence length. Motivated by this connection, we re-examine T by explicitly unrolling the inner-loop updates. We show analytically that T induces a linear attention–like operator in a general form, even when the inner loop consists of multi-layer non-linear mappings (Section 5.1). This perspective naturally explains our empirical observations that contradict the storage-and-retrieval interpretation (Section 5.2). Finally, we rewrite two representative variants of T, LaCT (Zhang et al., 2025) and ViTTT (Han et al., 2025), in their linear attention form (Sections 5.3 and 5.4). Detailed derivations are deferred to the appendix. 5.1 General Form Theorem 5.1 (Linearization of Inner-Loop Updates). Consider a T model whose inner-loop function has a linear, bias-free final layer, f(x)=ϕ(x;Θ)W,f(x)=φ(x; )\,W, where ϕ(x;Θ)∈ℝDhφ(x; ) ^D_h denotes the hidden representation of the inner-loop function with parameters Θ , and W∈ℝDh×DoutW ^D_h× D_out is the weight matrix of the final layer. Suppose that at step t, the inner loop performs one step of gradient descent on an objective ℒL with learning rate η, using key input k, updating all trainable parameters, (Wt+1,Θt+1)=(Wt,Θt)−η∇(Wt,Θt)ℒ(ft(k)),(W_t+1, _t+1)=(W_t, _t)-η _(W_t, _t)L(f_t(k)), where ϕt(⋅)≜ϕ(⋅;Θt) _t(·) φ(·; _t) and ft(⋅)≜ϕt(⋅)Wtf_t(·) _t(·)W_t. Then, for any query q, the output after the update can be written as o=ϕt+1(q)(Wt+ϕt(k)⊤gt(k)),gt(k)≜−η∂ℒ∂ft(k).o= _t+1(q) (W_t+ _t(k) g_t(k) ), g_t(k) -η\, ∂ f_t(k). This expression is a linear attention operator of the form o=q^(S0+k^⊤v^),o= q (S_0+ k v ), where q^=ϕt+1(q),k^=ϕt(k),v^=gt(k),S0=Wt. q= _t+1(q), k= _t(k), v=g_t(k), S_0=W_t. The complete proof is provided in Appendix B. Theorem 5.2 (Unrolling Inner-Loop Updates). Given a sequence of query–key pairs (q0,k0),(q1,k1),…,(qt,kt)\(q_0,k_0),(q_1,k_1),…,(q_t,k_t)\, suppose the T model performs one gradient descent step per input in sequence. By repeated application of Theorem 5.1, the parameters after processing token t are (Wt+1,Θt+1)=(W0,Θ0)−η∑i=0t∇(Wi,Θi)ℒ(fi(ki)).(W_t+1, _t+1)=(W_0, _0)-η _i=0^t _(W_i, _i)L\! (f_i(k_i) ). Evaluating the T model on query qtq_t yields ot o_t =ϕt+1(qt)Wt+1 = _t+1(q_t)\,W_t+1 =ϕt+1(qt)(W0+∑i=0tϕi(ki)⊤gi(ki)), = _t+1(q_t) (W_0+ _i=0^t _i(k_i) g_i(k_i) ), where gi(ki)g_i(k_i) is defined as in Theorem 5.1. This corresponds to the extended linear attention form on sequential inputs. ot=q^t(S0+∑i=0tk^i⊤v^i).o_t= q_t (S_0+ _i=0^t k_i v_i ). The complete proof is provided in Appendix C. Next, we extend Theorem 5.2 to the case where the inner-loop employs gradient descent with momentum. Theorem 5.3 (Gradient Descent with Momentum). Given the momentum-augmented gradient accumulator defined as (ΔWt,ΔΘt)=∇(W,Θ)ℒ(ft(kt))+αt(ΔWt−1,ΔΘt−1),( W_t, _t)= _(W, )L(f_t(k_t))+ _t\,( W_t-1, _t-1), where αt _t denotes the (possibly token-dependent) momentum factor at step t. The model parameters are then updated according to (Wt+1,Θt+1)=(Wt,Θt)−η(ΔWt,ΔΘt).(W_t+1, _t+1)=(W_t, _t)-η\,( W_t, _t). Define the cumulative momentum coefficient as βij≜∏s=i+1jαsif i<j,1if i=j. _i^j cases _s=i+1^j _s&if i<j,\\ 1&if i=j. cases Unrolling this recurrence and evaluating the T model on query qtq_t yields ot o_t =ϕt+1(qt)Wt+1 = _t+1(q_t)\,W_t+1 =ϕt+1(qt)(W0+∑i=0tϕi(ki)⊤mi(ki)), = _t+1(q_t) (W_0+ _i=0^t _i(k_i) m_i(k_i) ), which induces a linear-attention–like form identical to that of Theorem 5.2, with the effective value vector being a momentum-weighted sum: v^i=mi(ki)≜gi(ki)⋅∑j=itβij. v_i=m_i(k_i)\; \;g_i(k_i)· _j=i^t _i^j. The complete proof is provided in Appendix D. 5.2 Explanation of T Empirical Behaviors Having shown that T can be rewritten as a linear attention operator (Theorem 5.1 - 5.3), we can now revisit the empirical behaviors that appeared contradictory under the memorization-based interpretation. The linear-attention perspective provides a unified and mechanistic explanation. More Inner-Loop Steps. According to Theorem 5.1, the inner loop does not perform storage of key–value information, but instead defines a learnable mapping that transforms the original inputs into the effective query, key, and value representations. This mapping depends on inner-loop hyperparameters, including the number of optimization steps. Increasing the number of inner-loop iterations at inference time therefore induces an attention operator different to the one used during training, naturally leading to degraded performance due to train–test mismatch rather than improved memorization. Gradient Ascent in the Inner Loop. Under the same formulation, replacing gradient descent with gradient ascent simply flips the sign of the effective value vector gtg_t. Since this sign is absorbed into the learned attention operator and the mapping itself is optimized under the downstream objective, the model adapts to this change. This explains why T remains effective under gradient ascent, despite the absence of a meaningful memorization objective. Distributional Asymmetry Between Q and K. The linear-attention view clarifies why similarity between q and k is not required. In T, q and k influence different components of the attention operator: q determines the effective query via ϕt+1(q) _t+1(q), while k determines the effective key and value via ϕt(k) _t(k) and gt(k)g_t(k), respectively. They are therefore intermediate features rather than symmetric query–key representations, making distributional mismatch expected rather than pathological. Replacing Q with K. Finally, replacing q with k does not collapse the attention mechanism because the effective query and key remain distinct: ϕt+1(k) _t+1(k) versus ϕt(k) _t(k). Since ϕφ is learnable and evaluated at different parameter states, the model can map the same input to different representations, preserving attention functionality. Summary. Viewed through the lens of linear attention, the inner loop of T parameterizes a structured linear-form attention operator rather than performing test-time storage and retrieval. Under this perspective, the observed empirical behaviors follow naturally from representation learning and train–test consistency considerations. 5.3 Example: LaCT as Linear Attention We now show how a representative instantiation of T formula, LaCT (Zhang et al., 2025), can be rewritten in the form of linear attention. LaCT adopts a bias-free SwiGLU MLP (Shazeer, 2020) as its inner-loop mapping, parameterized by three learnable weight matrices W0,W2∈ℝDh×DkW_0,W_2 ^D_h× D_k and W1∈ℝDh×DvW_1 ^D_h× D_v: f(x)=(silu(xW0)⊙(xW2))W1.f(x)= (silu(xW_0) (xW_2) )W_1. The inner-loop objective is defined via the Frobenius inner product ℒ(f(k),v)=−⟨f(k),v⟩.L(f(k),v)=- f(k),v . At each token, LaCT performs gradient descent with per-token learning rate ηt _t, momentum αt _t, and gradient orthogonalization ℳ(⋅)M(·) inspired by Muon (Jordan et al., 2024): Wi,t+1=Wi,t−ηtℳ(ΔWi,t),W_i,t+1=W_i,t- _t\,M( W_i,t), ΔWi,t=∇Wi,tℒ(ft(kt),vt), W_i,t= _W_i,tL(f_t(k_t),v_t), for i∈0,1,2i∈\0,1,2\. Following Theorem 5.3, the inner-loop model at step t can be written as ft(x)=ϕt(x)W1,t,ϕt(x)=silu(xW0,t)⊙(xW2,t).f_t(x)= _t(x)\,W_1,t, _t(x)=silu(xW_0,t) (xW_2,t). Evaluating the updated model on query qtq_t yields ot o_t =ft+1(qt) =f_t+1(q_t) =ϕt+1(qt)W1,t+1 = _t+1(q_t)\,W_1,t+1 =ϕt+1(qt)(W1,0+∑i=0tℳ(ϕi(ki)⊤mi)), = _t+1(q_t) (W_1,0+ _i=0^tM\! ( _i(k_i) m_i ) ), (1) where mi(ki)≜vi⋅∑j=itηjβij.m_i(k_i)\; \;v_i· _j=i^t _j\, _i^j. This expression reveals that the inner loop of LaCT is effectively a linear attention–like operator, with ϕi(ki) _i(k_i) and mi(ki)m_i(k_i) playing the roles of keys and values, and ϕt+1(qt) _t+1(q_t) acting as the query vector. Note that LaCT also applies weight normalization after each update, which we omit here for simplicity. See Appendix E for more details. 5.4 Example: ViTTT as Linear Attention We next show that another instantiation of test-time training, ViTTT (Han et al., 2025), can likewise be rewritten in the form of linear attention. ViTTT employs fast weights consisting of two independent components: (i) a simplified gated linear unit (GLU), and (i) a depthwise convolution layer. These components are updated independently in the inner loop. We show that each admits a linear-attention interpretation, implying that ViTTT as a whole falls within the same framework. GLU component. The GLU is defined as f(x)=silu(xW0)⊙(xW1),f(x)=silu(xW_0) (xW_1), where W0W_0 and W1W_1 are fast weights updated via gradient descent. As in LaCT, the inner-loop loss is defined using a Frobenius inner product. Following a derivation analogous to previous sections (see Appendix F), evaluating the updated GLU on a query qtq_t yields ϕ(x)=silu(xW0),φ(x)=silu(xW_0), ot o_t =ϕt+1(qt)⊙(qt(W1+kt⊤(vt⊙ϕt(kt)))). = _t+1(q_t) (q_t (W_1+k_t (v_t _t(k_t)) ) ). This expression takes the form of linear attention, with ϕt(kt) _t(k_t) acting as a multiplicative gate on values and ϕt+1(qt) _t+1(q_t) gating the final output. Depthwise convolution component. ViTTT additionally includes a 3×33× 3 depthwise convolution layer with fast weights, updated in the inner loop. As convolution is effectively sliding window linear layer, this T component is equivalent to a sliding window linear attention. A formal derivation is provided in Appendix G. Discussion. Since both fast-weight components in ViTTT admit linear-attention formulations, their combination also induces a linear-attention–like operator. This establishes ViTTT as another concrete instance of test-time training whose behavior is more naturally understood through the lens of linear attention. 6 Practical Implications Table 2: Ablation Trajectory Reducing T to Standard Linear Attention. By progressively simplifying complex T formulations into standard linear attention (Variant 6), we quantify the contribution of each design component in T. Variant 1 achieves the best performance across all three tasks. Variants 2–6 admit parallel implementations. We additionally report the inference throughput of each variant’s T layer on the LLM task. † denotes ablations that do not apply, in which case performance matches the preceding variant. * indicates that ViTTT does not use gradient orthogonalization, so we ablate gradient normalization instead. Alias Description of T Inner-loop Designs Perplexity ↓ (LaCT-LLM) PSNR ↑ (LaCT-NVS) Top-1 Acc↑ (ViTTT) Tokens Per Sec↑ (Recurrent Impl.) Tokens Per Sec↑ (Parallel Impl.) Baseline LaCT (Zhang et al., 2025) / ViTTT (Han et al., 2025) 16.43 25.94 79.34% 4.30M N/A Variant 1 Baseline w/ only update the last layer parameters 15.93 25.97 79.63% 10.60M N/A Variant 2 Variant 1 w/ remove weight normalization 16.31 25.93 79.63%† 11.02M 30.18M Variant 3 Variant 2 w/ multi-layer MLP → singel linear layer 16.23 25.71 79.39% 12.95M 49.69M Variant 4 Variant 3 w/ remove per-token learnable lr 16.12 25.70 79.39%† 13.31M 53.99M Variant 5 Variant 4 w/ remove momentum in SGD 15.97 25.70† 79.39%† 14.40M 57.28M Variant 6 Variant 5 w/ remove gradient orthogonalization 16.80 25.73 79.54%* 89.67M 124.6M Figure 3: Perplexity Metric for Ablation on LaCT-LLM. Evaluated on 2.5B tokens from the Book-3 dataset. Interpreting T through the lens of linear attention is not merely a theoretical exercise but yields concrete practical benefits. This perspective reveals a systematic trajectory for reducing complex T formulations into linear attention, along which we observe that several commonly adopted design choices, such as per-token learnable learning rates, weight normalization, are not essential to final performance. By clarifying which components are truly valuable and which are unnecessary, this view enables substantial simplification of T formulations. Moreover, recognizing T as linear attention makes explicit that the seemingly recurrent inner-loop updates admit a parallel formulation, leading to significant efficiency gains in both training and inference. All experiments in this section use the official implementations of LaCT (Zhang et al., 2025) for the LLM and NVS tasks, and ViTTT (Han et al., 2025) for the image recognition task. Detailed experimental settings are provided in Appendix A. 6.1 Reduce T to Linear Attention Viewing T through the lens of linear attention reveals that several design choices that were justified under a storage-and-retrieval interpretation are in fact redundant or optional. This perspective enables a principled ablation path that progressively simplifies complex T variants such as LaCT and ViTTT into standard linear attention. Below, we outline this reduction trajectory and analyze the role of each component. The ablation steps are applied sequentially: Variant 1 is the Baseline with Step 1 applied, Variant 2 adds Step 2 on top of Variant 1, and so on, so that Variant 6 corresponds to the full reduction from the original T formulation to standard linear attention. Step 1. Update only the last-layer parameters. As shown in Theorem 5.1, the effective query and key vectors are ϕt+1(q) _t+1(q) and ϕt(k) _t(k), where ϕt(⋅)≜ϕ(⋅;Θt) _t(·) φ(·; _t). Updating Θt _t within the inner loop makes ϕt _t a dynamic kernel that is difficult to unroll analytically. If the inner loop instead updates only the final-layer parameter, Θ remains fixed during T, and ϕ(⋅)≜ϕ(⋅;Θ)φ(·) φ(·; ) becomes a static function with learnable parameters. From the linear-attention perspective, ϕ(⋅)φ(·) then acts as a learnable kernel function with effective queries and keys given by ϕ(q)φ(q) and ϕ(k)φ(k). Step 2. Remove weight normalization. LaCT applies weight normalization to all learnable parameters Θt,Wt _t,W_t after each inner-loop update. After Step 1, normalization on Θt _t is a no-op since Θt _t is fixed. Normalizing the final-layer parameter WtW_t, which corresponds to the state in the linear attention view, is therefore equivalent to normalizing the state StS_t (see Appendix E.4). As such normalization is uncommon in linear attention literature, we remove it for ablation. Notably, after this step the T formulation becomes fully parallelizable, as discussed in Section 6.2. Step 3. Multi-layer MLP → single linear layer. Several T variants (Han et al., 2025; Behrouz et al., 2026) employ deeper inner-loop MLPs, but report inconsistent empirical gains. From a linear attention perspective, increasing MLP depth simply induces a more complex kernel function ϕ(⋅)φ(·) over queries and keys. When q and k already have sufficient representational capacity, this added complexity is unlikely to help. We therefore reduce the multi-layer MLP to a single linear layer, effectively removing ϕ(⋅)φ(·) altogether. This exposes the true queries and keys as q^=q q=q and k^=k k=k, bringing the formulation closer to basic linear attention. Step 4. Remove per-token learning rates. Many T methods (Behrouz et al., 2026; Zhang et al., 2025) introduce a per-token learnable learning rate ηt _t. As shown in Section 5.3, with Frobenius dot product as inner loss this can be absorbed into the learnable vtv_t, indicating it’s functionally redundant. Consistent with our finding, ViTTT empirically finds that a constant learning rate of 1.01.0 suffices. Step 5. Remove momentum in SGD. As shown in Theorem 5.3, adding momentum to the inner-loop SGD update, as done in LaCT and related methods (Behrouz et al., 2026), only alters the effective value v v, from the instantaneous gradient gt(k)g_t(k) to a momentum-weighted sum of past gradients. From the linear attention perspective, this corresponds to remixing historical key–value contributions into a single value vector. Since both keys and values are already learnable, this additional mixing is unlikely to provide meaningful benefit, thus we remove momentum for ablation. Notably, for both LaCT and ViTTT, gt(k)=−vg_t(k)=-v and removing momentum recovers v^=v v=v, exposing the true value vector. Step 6. Remove gradient orthogonalization. As discussed in Section 5.3, LaCT optionally applies gradient orthogonalization ℳ(ΔW)M( W). Under the linear attention reformulation, this corresponds to applying an operator to the state update ℳ(k^⊤v)M( k v). We remove this operation for ablation. After this final step, Both LaCT and ViTTT reduce exactly to standard linear attention: o=q(W+∑iki⊤vi).o=q\! (W+ _ik_i v_i ). Results. We progressively apply the above ablations to LaCT on the LLM and NVS tasks, and to ViTTT on the image recognition task, with results summarized in Table 2. Surprisingly, restricting the inner loop to update only the final MLP layer consistently yields the best overall performance across tasks, suggesting that many of the more complex design choices are unnecessary, or even detrimental. Most other components contribute only marginally to performance, with two notable exceptions: deeper MLPs are beneficial for the NVS task, while gradient orthogonalization improves performance on the LLM task. Overall, reducing the full T formulation to a basic linear attention operator (Variant 6) results in only minor performance degradation (+0.4+0.4 perplexity on LLM and −0.2-0.2 dB on NVS). For the LLM task, Table 2 reports perplexity at 32k sequence length, with results across other lengths shown in Figure 3. Figure 4: Training loss vs. wall-clock time on LaCT-LLM. We compare the original LaCT-T with both parallel and recurrent form of Variant 2. The parallel form achieves a 1.19×1.19× end-to-end speedup while maintaining comparable convergence. 6.2 Parallel Form of T Existing T variants are typically implemented in a recurrent manner, reflecting their original storage-and-retrieval interpretation. However, as we have shown that T can alternatively be viewed as a form of linear attention, a natural question arises: can T admit a parallel formulation that enables more efficient implementation? We show that under certain conditions, corresponding to Variants 2–6 in our ablation, such a parallel form indeed exists. The key insight is that when weight normalization is removed and only the final-layer parameters are updated, the state update becomes associative. In this setting, the kernel function ϕt(⋅)≜ϕ(⋅;Θt) _t(·) φ(·; _t) is static and independent of sequence history, allowing the recurrence in Theorem 5.3 to be computed via a parallel prefix scan rather than sequential token-by-token updates. We implement this parallel formulation for LaCT on the LLM task. As shown in Table 2, switching from the recurrent to the parallel implementation improves the inference throughput of the T layer by up to 4.0×4.0× (measured in tokens per second, single batch). Combined with the simplifications from Step 1 and Step 2, this yields a 1.19×1.19× end-to-end training speedup without degrading model quality, as shown in Figure 4. We provide the full parallel formulation and a proof of equivalence to the sequential recurrence in Appendix H. In Appendix I, we further show that introducing weight normalization or dynamic kernel functions breaks associativity, thereby preventing parallelization. 7 Conclusion In this work, we challenge the prevailing view of T as a mechanism for test-time memorization of key–value mappings. Through systematic empirical analysis, we identify several anomalies—including gradient ascent behavior, distributional asymmetry between queries and keys, and the lack of correlation between inner-loop convergence and downstream performance—that are fundamentally incompatible with a memorization-based interpretation. We provide an alternative explanation by showing that T, even with complex inner loops involving multi-layer MLPs and momentum-based optimizers, can be analytically rewritten as a form of linear attention operator. Under this view, the inner loop does not perform meta learning in the conventional sense, but instead parameterizes a structured mixing of queries, keys, and values. Viewing T through the lens of linear attention not only resolves the observed paradoxes, but also enables principled simplifications, parallel implementations with improved efficiency, and a unified framework for understanding T variants. Our analysis applies broadly to T formulations based on key–value binding, but is limited to settings where the inner-loop final layer is linear and bias-free. Empirically, we focus on LaCT (Zhang et al., 2025) and ViTTT (Han et al., 2025) as case studies because they are representative methods with open-source implementations. Other key–value-binding variants such as Titans (Behrouz et al., 2026) and Atlas (Behrouz et al., 2025a) also satisfy the assumptions of our theorems, and we expect our results to extend to them; empirical validation on these models is left to future work as their implementations become available. Extending these insights to nonlinear final layers, and exploring deeper connections between T and modern linear attention mechanisms in both directions, remain additional avenues for future work. Acknowledgements We gratefully acknowledge the Vector Institute for providing compute resources for part of this work. We thank Tianyuan Zhang for sharing experimental details and insightful discussions, and Yu Sun for reviewing an early version of this paper. Impact Statement Our work simplifies and parallelizes existing T architectures, potentially reducing the computational cost and energy consumption of training and deploying such models. Because the contributions are primarily analytical and efficiency-oriented, we do not foresee additional societal risks beyond those already associated with the underlying sequence modeling architectures and their downstream applications. References A. Behrouz, Z. Li, P. Kacham, M. Daliri, Y. Deng, P. Zhong, M. Razaviyayn, and V. Mirrokni (2025a) Atlas: learning to optimally memorize the context at test time. arXiv preprint arXiv:2505.23735. Cited by: §1, §7. A. Behrouz, M. Razaviyayn, P. Zhong, and V. Mirrokni (2025b) It’s all connected: a journey through test-time memorization, attentional bias, retention, and online optimization. arXiv preprint arXiv:2504.13173. Cited by: §2.2. A. Behrouz, M. Razaviyayn, P. Zhong, and V. Mirrokni (2025c) Nested learning: the illusion of deep learning architectures. arXiv preprint arXiv:2512.24695. Cited by: §1, §2.2, §3. A. Behrouz, P. Zhong, and V. Mirrokni (2026) Titans: learning to memorize at test time. Advances in Neural Information Processing Systems 38, p. 113506–113543. Cited by: §1, §2.1, §2.2, §2.2, §2.2, §6.1, §6.1, §6.1, §7. X. Chen, Y. Chen, Y. Xiu, A. Geiger, and A. Chen (2025) Ttt3r: 3d reconstruction as test-time training. arXiv preprint arXiv:2509.26645. Cited by: §2.2. Y. Chen, J. Wang, Z. Yang, S. Manivasagam, and R. Urtasun (2024) G3r: gradient guided generalizable reconstruction. In European Conference on Computer Vision, p. 305–323. Cited by: §2.2. K. Dalal, D. Koceja, J. Xu, Y. Zhao, S. Han, K. C. Cheung, J. Kautz, Y. Choi, Y. Sun, and X. Wang (2025) One-minute video generation with test-time training. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 17702–17711. Cited by: §1, §2.2, §2.2. T. Dao and A. Gu (2024) Transformers are ssms: generalized models and efficient algorithms through structured state space duality. In Proceedings of the 41st International Conference on Machine Learning, p. 10041–10071. Cited by: §2.1. J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei (2009) Imagenet: a large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, p. 248–255. Cited by: Appendix A. C. Finn, P. Abbeel, and S. Levine (2017) Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, p. 1126–1135. Cited by: §1, §2.2, §3. Y. Gandelsman, Y. Sun, X. Chen, and A. Efros (2022) Test-time training with masked autoencoders. Advances in Neural Information Processing Systems 35, p. 29374–29385. Cited by: §2.2, §3. L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, et al. (2020) The pile: an 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027. Cited by: Appendix A. A. Gu and T. Dao (2024) Mamba: linear-time sequence modeling with selective state spaces. In First conference on language modeling, Cited by: §2.1. A. Gu, K. Goel, and C. Ré (2022) Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations, Cited by: §2.1. D. Han, Y. Li, T. Li, Z. Cao, Z. Wang, J. Song, Y. Cheng, B. Zheng, and G. Huang (2025) ViT3: unlocking test-time training in vision. arXiv preprint arXiv:2512.01643. Cited by: Appendix A, Appendix F, Appendix G, §1, §2.2, §2.2, §2.2, §2.2, §4.1, Table 1, Table 1, §5.4, §5, §6.1, Table 2, §6, §7. G. E. Hinton and D. C. Plaut (1987) Using fast weights to deblur old memories. In Proceedings of the 9th Annual Conference of the Cognitive Science Society, Hillsdale, NJ, p. 177–186. Cited by: §2.2. K. Jordan, Y. Jin, V. Boza, J. You, F. Cesista, L. Newhouse, and J. Bernstein (2024) Muon: an optimizer for hidden layers in neural networks. External Links: Link Cited by: §E.1, §5.3. M. Karami, R. Pascanu, and V. Mirrokni (2025) Lattice: learning to efficiently compress the memory. arXiv preprint arXiv:2504.05646. Cited by: §2.2. A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret (2020) Transformers are rnns: fast autoregressive transformers with linear attention. In International conference on machine learning, p. 5156–5165. Cited by: §1, §2.1. J. Lei, D. Zhang, and S. Poria (2025) Error-free linear attention is a free lunch: exact solution from continuous-time dynamics. arXiv preprint arXiv:2512.12602. Cited by: §2.1. B. Liu, R. Wang, L. Wu, Y. Feng, P. Stone, and Q. Liu (2025) Longhorn: state space models are amortized online learners. In International Conference on Learning Representations, Vol. 2025, p. 95419–95434. Cited by: §2.1. L. v. d. Maaten and G. Hinton (2008) Visualizing data using t-sne. Journal of machine learning research 9 (Nov), p. 2579–2605. Cited by: §4.3. L. Metz, N. Maheswaranathan, B. Cheung, and J. Sohl-Dickstein (2018) Meta-learning update rules for unsupervised representation learning. arXiv preprint arXiv:1804.00222. Cited by: §1, §2.2. N. Muennighoff, Z. Yang, W. Shi, X. L. Li, L. Fei-Fei, H. Hajishirzi, L. Zettlemoyer, P. Liang, E. Candès, and T. B. Hashimoto (2025) S1: simple test-time scaling. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p. 20286–20332. Cited by: §2.2. A. Orvieto, S. L. Smith, A. Gu, A. Fernando, C. Gulcehre, R. Pascanu, and S. De (2023) Resurrecting recurrent neural networks for long sequences. In International Conference on Machine Learning, p. 26670–26698. Cited by: §2.1. G. Penedo, H. Kydlíček, A. Lozhkov, M. Mitchell, C. A. Raffel, L. Von Werra, T. Wolf, et al. (2024) The fineweb datasets: decanting the web for the finest text data at scale. Advances in Neural Information Processing Systems 37, p. 30811–30849. Cited by: Appendix A. B. Peng, E. Alcaide, Q. Anthony, A. Albalak, S. Arcadinho, S. Biderman, H. Cao, X. Cheng, M. Chung, L. Derczynski, et al. (2023) RWKV: reinventing rnns for the transformer era. In Findings of the Association for Computational Linguistics: EMNLP 2023, p. 14048–14077. Cited by: §2.1. B. Peng, D. Goldstein, Q. Anthony, A. Albalak, E. Alcaide, S. Biderman, E. Cheah, X. Du, T. Ferdinan, H. Hou, et al. (2024) Eagle and finch: rwkv with matrix-valued states and dynamic recurrence. arXiv preprint arXiv:2404.05892. Cited by: §2.1. B. Peng, R. Zhang, D. Goldstein, E. Alcaide, X. Du, H. Hou, J. Lin, J. Liu, J. Lu, W. Merrill, et al. (2025a) Rwkv-7” goose” with expressive dynamic state evolution. arXiv preprint arXiv:2503.14456. Cited by: §2.1. L. Peng, A. Chattopadhyay, L. Zancato, E. Nunez, W. Xia, and S. Soatto (2025b) Gated kalmanet: a fading memory layer through test-time ridge regression. arXiv preprint arXiv:2511.21016. Cited by: §2.1. Z. Qin, S. Yang, W. Sun, X. Shen, D. Li, W. Sun, and Y. Zhong (2024) Hgrn2: gated linear rnns with state expansion. arXiv preprint arXiv:2404.07904. Cited by: §2.1. Z. Qin, S. Yang, and Y. Zhong (2023) Hierarchically gated recurrent neural network for sequence modeling. Advances in Neural Information Processing Systems 36, p. 33202–33221. Cited by: §2.1. I. Schlag, K. Irie, and J. Schmidhuber (2021) Linear transformers are secretly fast weight programmers. In International conference on machine learning, p. 9355–9366. Cited by: §2.1, §2.2. N. Shazeer (2020) Glu variants improve transformer. arXiv preprint arXiv:2002.05202. Cited by: §5.3. J. T. Smith, A. Warrington, and S. W. Linderman (2023) Simplified state space layers for sequence modeling. In International Conference on Learning Representations (ICLR), Cited by: §2.1. C. Snell, J. Lee, K. Xu, and A. Kumar (2024) Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314. Cited by: §2.2. Y. Sun, X. Li, K. Dalal, J. Xu, A. Vikram, G. Zhang, Y. Dubois, X. Chen, X. Wang, S. Koyejo, et al. (2025) Learning to (learn at test time): rnns with expressive hidden states. In International Conference on Machine Learning, Cited by: §1, §1, §2.2, §2.2, §2.2, §2.2, §3, §3, §5. Y. Sun, X. Wang, Z. Liu, J. Miller, A. Efros, and M. Hardt (2020) Test-time training with self-supervision for generalization under distribution shifts. In International conference on machine learning, p. 9229–9248. Cited by: §1, §2.2, §3. Y. Sun, L. Dong, S. Huang, S. Ma, Y. Xia, J. Xue, J. Wang, and F. Wei (2023) Retentive network: a successor to transformer for large language models. arXiv preprint arXiv:2307.08621. Cited by: §2.1. A. Tandon, K. Dalal, X. Li, D. Koceja, M. Rød, S. Buchanan, X. Wang, J. Leskovec, S. Koyejo, T. Hashimoto, et al. (2025) End-to-end test-time training for long context. arXiv preprint arXiv:2512.23675. Cited by: §1, §2.2, §3. K. Team, Y. Zhang, Z. Lin, X. Yao, J. Hu, F. Meng, C. Liu, X. Men, S. Yang, Z. Li, et al. (2025) Kimi linear: an expressive, efficient attention architecture. arXiv preprint arXiv:2510.26692. Cited by: §2.1. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin (2017) Attention is all you need. Advances in neural information processing systems 30. Cited by: §2.1. K. A. Wang, J. Shi, and E. B. Fox (2025) Test-time regression: a unifying framework for designing sequence models with associative memory. arXiv preprint arXiv:2501.12352. Cited by: §2.2, §2.2. S. Yang, B. Wang, Y. Shen, R. Panda, and Y. Kim (2024a) Gated linear attention transformers with hardware-efficient training. In Proceedings of the 41st International Conference on Machine Learning, p. 56501–56523. Cited by: §2.1. S. Yang, B. Wang, Y. Zhang, Y. Shen, and Y. Kim (2024b) Parallelizing linear transformers with the delta rule over sequence length. Advances in neural information processing systems 37, p. 115491–115522. Cited by: §2.1. Y. Yuan, Q. Shen, S. Wang, X. Yang, and X. Wang (2025) Test3r: learning to reconstruct 3d at test time. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, Cited by: §2.2. T. Zhang, S. Bi, Y. Hong, K. Zhang, F. Luan, S. Yang, K. Sunkavalli, W. T. Freeman, and H. Tan (2025) Test-time training done right. arXiv preprint arXiv:2505.23884. Cited by: Appendix A, Appendix A, Appendix E, §1, Figure 1, Figure 1, §2.2, §2.2, §2.2, §2.2, Figure 2, Figure 2, §4.1, §4.2, §4.3, Table 1, Table 1, §5.3, §5, §6.1, Table 2, §6, §7. Y. Zhang, S. Yang, R. Zhu, Y. Zhang, L. Cui, Y. Wang, B. Wang, F. Shi, B. Wang, W. Bi, et al. (2024) Gated slot attention for efficient linear-time sequence modeling. Advances in Neural Information Processing Systems 37, p. 116870–116898. Cited by: §2.1. Y. Zhang and S. Yang (2025) Flame: flash language modeling made easy. External Links: Link Cited by: Appendix A. S. Zhong, M. Xu, T. Ao, and G. Shi (2025) Understanding transformer from the perspective of associative memory. arXiv preprint arXiv:2505.19488. Cited by: §2.1. T. Zhou, R. Tucker, J. Flynn, G. Fyffe, and N. Snavely (2018) Stereo magnification: learning view synthesis using multiplane images. ACM Transactions on Graphics (TOG) 37 (4), p. 1–12. Cited by: Appendix A. Appendix A Experiment Setup Here we outline the three tasks used to evaluate the T framework, along with the experimental setup. Language Modeling We use the 760M parameter LaCT-LLM (Zhang et al., 2025) as our baseline model. All models are trained on 100B tokens sampled from the FineWeb-Edu dataset (Penedo et al., 2024) with a batch size of 4 per GPU across 8 NVIDIA A100 GPUs for 20K iterations, which takes approximately 56 hours to complete. All other hyperparameters follow the original (Zhang et al., 2025) configuration. For evaluation, we report perplexity on 2.5B tokens from the Book-3 dataset (Gao et al., 2020). All implementations are based on the Flame (Zhang and Yang, 2025) codebase. Novel View Synthesis We use 12-layer 768 hidden dimension LaCT-NVS (Zhang et al., 2025) model as baseline, totaling 114M parameters. We train our model on the RealEstate10K (Zhou et al., 2018) dataset with a batch size of 128 per GPU across 4 NVIDIA A100 GPUs for 20K iterations, which takes approximately 38 hours to complete. We use 2 input views and 6 target views for training, and 2 input views and 3 target views for evaluation. All images are resized to 128×128128× 128 resolution. We train using only the MSE loss and evaluate using the Peak Signal-to-Noise Ratio (PSNR) metric. All other hyperparameters follow the original LaCT-NVS (Zhang et al., 2025) configuration. Image Classification We use ViTTT-B (Han et al., 2025) as our baseline model, totaling 90M parameters. Follow (Han et al., 2025), we train our model on the ImageNet-1K (Deng et al., 2009) dataset with a batch size of 256 per GPU across 2 NVIDIA H100 GPUs for 60 epochs, which takes approximately 16 hours to complete. We evaluate on the ImageNet-1K (Deng et al., 2009) validation set using the top-1 accuracy. All other hyperparameters follow the original (Han et al., 2025) configuration. Appendix B Proof of Theorem 5.1 Proof. We explicitly unroll the single-step inner-loop update. Using the shorthand notation ϕt(⋅)≜ϕ(⋅;Θt) _t(·) φ(·; _t) and ft(⋅)≜ϕt(⋅)Wtf_t(·) _t(·)W_t, the model output for the key input k at step t is ft(k)=ϕt(k)Wt.f_t(k)= _t(k)\,W_t. By the chain rule, the gradient of the last layer with respect to the loss ℒ(ft(k))L(f_t(k)) is ∇Wtℒ=ϕt(k)⊤∂ℒ∂ft(k). _W_tL= _t(k) ∂ f_t(k). Applying one step of gradient descent with learning rate η yields Wt+1 W_t+1 =Wt−ηϕt(k)⊤∂ℒ∂ft(k) =W_t-η _t(k) ∂ f_t(k) =Wt+ϕt(k)⊤gt(k) =W_t+ _t(k) g_t(k) where we define gt(k)≜−η∂ℒ∂ft(k)∈ℝDout.g_t(k) -η\, ∂ f_t(k) ^D_out. The kernel function parameters Θ are also updated with gradient descent, yielding the updated function ϕt+1(⋅)=ϕ(⋅;Θt+1) _t+1(·)=φ(·; _t+1). Evaluating the updated model on a query q gives o o =ϕt+1(q)Wt+1 = _t+1(q)\,W_t+1 =ϕt+1(q)(Wt+ϕt(k)⊤gt(k)) = _t+1(q)\, (W_t+ _t(k) g_t(k) ) This expression is a linear attention operator of the form o=q^(S0+k^⊤v^),o= q (S_0+ k v ), where q^=ϕt+1(q),k^=ϕt(k),v^=gt(k),S0=Wt. q= _t+1(q), k= _t(k), v=g_t(k), S_0=W_t. ∎ Appendix C Proof of Theorem 5.2 Proof. We prove by induction on the number of tokens processed. Base case. Starting from initial parameters (W0,Θ0)(W_0, _0), processing token 0 with key k0k_0 applies one gradient descent step. By Theorem 5.1: W1=W0+ϕ0(k0)⊤g0(k0),W_1=W_0+ _0(k_0) g_0(k_0), where g0(k0)=−η∂ℒ∂f0(k0)g_0(k_0)=-η\, ∂ f_0(k_0), and Θ1 _1 is updated accordingly. This matches the claimed form with t=0t=0. Inductive step. Assume that after processing tokens 0,…,t−10,…,t-1, we have Wt=W0+∑i=0t−1ϕi(ki)⊤gi(ki).W_t=W_0+ _i=0^t-1 _i(k_i) g_i(k_i). Processing token t with key ktk_t applies another gradient descent step. By Theorem 5.1: Wt+1=Wt+ϕt(kt)⊤gt(kt).W_t+1=W_t+ _t(k_t) g_t(k_t). Substituting the inductive hypothesis: Wt+1=W0+∑i=0t−1ϕi(ki)⊤gi(ki)+ϕt(kt)⊤gt(kt)=W0+∑i=0tϕi(ki)⊤gi(ki).W_t+1=W_0+ _i=0^t-1 _i(k_i) g_i(k_i)+ _t(k_t) g_t(k_t)=W_0+ _i=0^t _i(k_i) g_i(k_i). Output computation. Evaluating the model on query qtq_t with the updated parameters gives: ot o_t =ϕt+1(qt)Wt+1 = _t+1(q_t)\,W_t+1 =ϕt+1(qt)(W0+∑i=0tϕi(ki)⊤gi(ki)) = _t+1(q_t) (W_0+ _i=0^t _i(k_i) g_i(k_i) ) This is the extended linear attention form: ot=q^t(S0+∑i=0tk^i⊤v^i),o_t= q_t (S_0+ _i=0^t k_i v_i ), where q^t=ϕt+1(qt) q_t= _t+1(q_t), k^i=ϕi(ki) k_i= _i(k_i), v^i=gi(ki) v_i=g_i(k_i), and S0=W0S_0=W_0. ∎ Appendix D Proof of Theorem 5.3 Proof. We prove by induction, extending the approach of Theorem 5.2 to momentum updates. Base case. At t=0t=0, the momentum accumulator is initialized as ΔW−1=0 W_-1=0, so ΔW0=∇Wℒ(f0(k0))=ϕ0(k0)⊤∂ℒ∂f0(k0). W_0= _WL(f_0(k_0))= _0(k_0) ∂ f_0(k_0). Applying the update rule W1=W0−ηΔW0W_1=W_0-η W_0 gives W1=W0+ϕ0(k0)⊤g0(k0),W_1=W_0+ _0(k_0) g_0(k_0), where g0(k0)=−η∂ℒ∂f0(k0)g_0(k_0)=-η\, ∂ f_0(k_0). This matches the claimed form with m0(k0)=g0(k0)m_0(k_0)=g_0(k_0). Inductive step. Define the cumulative momentum coefficient from step i to step j as: βij≜∏s=i+1jαsif i<j,1if i=j. _i^j cases _s=i+1^j _s&if i<j,\\ 1&if i=j. cases Assume that after processing tokens 0,…,t−10,…,t-1, the momentum accumulator satisfies ΔWt−1=∑i=0t−1βit−1ϕi(ki)⊤∂ℒ∂fi(ki). W_t-1= _i=0^t-1 _i^t-1\, _i(k_i) ∂ f_i(k_i). At step t, the momentum update gives ΔWt W_t =∇Wℒ(ft(kt))+αtΔWt−1 = _WL(f_t(k_t))+ _t W_t-1 =ϕt(kt)⊤∂ℒ∂ft(kt)+αt∑i=0t−1βit−1ϕi(ki)⊤∂ℒ∂fi(ki). = _t(k_t) ∂ f_t(k_t)+ _t _i=0^t-1 _i^t-1\, _i(k_i) ∂ f_i(k_i). For terms i∈[0,t−1]i∈[0,t-1], we have αt⋅βit−1=αt∏s=i+1t−1αs=∏s=i+1tαs=βit _t· _i^t-1= _t _s=i+1^t-1 _s= _s=i+1^t _s= _i^t. For the i=ti=t term, the coefficient is βt=1 _t^t=1. Thus: ΔWt=∑i=0tβitϕi(ki)⊤∂ℒ∂fi(ki). W_t= _i=0^t _i^t\, _i(k_i) ∂ f_i(k_i). Weight accumulation. From Wj+1=Wj−ηΔWjW_j+1=W_j-η W_j, we have Wt+1=W0−η∑j=0tΔWjW_t+1=W_0-η _j=0^t W_j. Substituting the closed form for ΔWj W_j and exchanging the order of summation: Wt+1 W_t+1 =W0−η∑j=0t∑i=0jβijϕi(ki)⊤∂ℒ∂fi(ki) =W_0-η _j=0^t _i=0^j _i^j\, _i(k_i) ∂ f_i(k_i) =W0+∑i=0tϕi(ki)⊤(−η∂ℒ∂fi(ki)∑j=itβij)⏟mi(ki). =W_0+ _i=0^t _i(k_i) (-η ∂ f_i(k_i) _j=i^t _i^j )_m_i(k_i). The momentum-weighted effective value is thus mi(ki)=gi(ki)⋅∑j=itβij.m_i(k_i)=g_i(k_i)· _j=i^t _i^j. Output computation. Evaluating the model on query qtq_t with the updated parameters: ot o_t =ϕt+1(qt)Wt+1 = _t+1(q_t)\,W_t+1 =ϕt+1(qt)(W0+∑i=0tϕi(ki)⊤mi(ki)). = _t+1(q_t) (W_0+ _i=0^t _i(k_i) m_i(k_i) ). This is the linear attention form with q^t=ϕt+1(qt) q_t= _t+1(q_t), k^i=ϕi(ki) k_i= _i(k_i), v^i=mi(ki) v_i=m_i(k_i), and S0=W0S_0=W_0. ∎ Appendix E Derivation: LaCT as Linear Attention We provide the full derivation showing how LaCT (Zhang et al., 2025) can be written in the form of linear attention, following the framework of Theorem 5.3. E.1 LaCT Architecture and Update Rule LaCT adopts a bias-free SwiGLU MLP as its inner-loop mapping: f(x)=(silu(xW0)⊙(xW2))W1,f(x)= (silu(xW_0) (xW_2) )W_1, where W0,W2∈ℝDh×DkW_0,W_2 ^D_h× D_k and W1∈ℝDh×DvW_1 ^D_h× D_v. The inner-loop objective uses the Frobenius inner product: ℒ(f(k),v)=−⟨f(k),v⟩.L(f(k),v)=- f(k),v . At each token t, LaCT performs gradient descent with per-token learning rate ηt _t, momentum αt _t, and Muon-style (Jordan et al., 2024) gradient orthogonalization ℳ(⋅)M(·): ΔWi,t=∇Wi,tℒ(ft(kt),vt)+αtΔWi,t−1,Wi,t+1=Wi,t−ηtℳ(ΔWi,t), W_i,t= _W_i,tL(f_t(k_t),v_t)+ _t W_i,t-1, W_i,t+1=W_i,t- _t\,M( W_i,t), for i∈0,1,2i∈\0,1,2\. E.2 Gradient Computation The upstream gradient with respect to the output is ∂ℒ∂ft(kt)=−vt ∂ f_t(k_t)=-v_t. Using the chain rule, the gradient for the final layer W1W_1 is: ∇W1ℒ=ϕt(kt)⊤∂ℒ∂ft(kt)=−ϕt(kt)⊤vt, _W_1L= _t(k_t) ∂ f_t(k_t)=- _t(k_t) v_t, where ϕt(x)=silu(xW0,t)⊙(xW2,t) _t(x)=silu(xW_0,t) (xW_2,t) is the kernel function. E.3 Linear Attention Form with Muon Following the same induction as Theorem 5.3, but with the Muon orthogonalization ℳ(⋅)M(·) applied after each gradient accumulation, the weight update becomes: W1,t+1=W1,0−∑j=0tηjℳ(ΔW1,j).W_1,t+1=W_1,0- _j=0^t _j\,M( W_1,j). Using the cumulative momentum coefficient βit _i^t defined in Theorem 5.3, the momentum accumulator for W1W_1 is: ΔW1,t=−ϕt(kt)⊤vt+αtΔW1,t−1=−∑i=0tβitϕi(ki)⊤vi. W_1,t=- _t(k_t) v_t+ _t W_1,t-1=- _i=0^t _i^t\, _i(k_i) v_i. Substituting into the weight update and exchanging the order of summation (as in the proof of Theorem 5.3): W1,t+1=W1,0+∑i=0tℳ(ϕi(ki)⊤mi),W_1,t+1=W_1,0+ _i=0^tM\! ( _i(k_i) m_i ), where the momentum-weighted effective value is: mi≜vi⋅∑j=itηjβij.m_i v_i· _j=i^t _j\, _i^j. Evaluating on query qtq_t: ot o_t =ϕt+1(qt)W1,t+1 = _t+1(q_t)\,W_1,t+1 =ϕt+1(qt)(W1,0+∑i=0tℳ(ϕi(ki)⊤mi)). = _t+1(q_t) (W_1,0+ _i=0^tM\! ( _i(k_i) m_i ) ). This is a linear attention–like form where ϕi(ki) _i(k_i) and mim_i play the roles of keys and values, and ϕt+1(qt) _t+1(q_t) acts as the query. The Muon orthogonalization ℳ(⋅)M(·) is applied element-wise to each key-value outer product before accumulation. E.4 Effect of Weight Normalization LaCT additionally applies weight normalization to W1W_1 after each update: W1,t+1=Norm(W1,t−ηtℳ(ΔW1,t)),W_1,t+1=Norm\! (W_1,t- _t\,M( W_1,t) ), where Norm(⋅)Norm(·) applies channel-wise ℓ2 _2 normalization. Weight normalization does not break the linear attention perspective. Recall that in the linear attention formulation, the state matrix St=W1,tS_t=W_1,t accumulates key-value outer products. With weight normalization, we simply normalize this state after each update: St+1=Norm(St+ϕt(kt)⊤mt).S_t+1=Norm\! (S_t+ _t(k_t) m_t ). The output remains a linear function of the (normalized) state: ot=ϕt+1(qt)St+1.o_t= _t+1(q_t)\,S_t+1. Thus, LaCT with weight normalization is still a linear attention mechanism—the query linearly reads from a state that accumulates key-value information. However, the normalization prevents expressing the state as a simple sum over history. Unlike the unnormalized case where St+1=S0+∑i=0tϕi(ki)⊤miS_t+1=S_0+ _i=0^t _i(k_i) m_i, the nested normalization creates a sequential dependency: St+1 S_t+1 =Norm(Norm(St−1+ϕt−1(kt−1)⊤mt−1)+ϕt(kt)⊤mt). =Norm\! (Norm\! (S_t-1+ _t-1(k_t-1) m_t-1 )+ _t(k_t) m_t ). This has implications for parallelization, which we discuss in Appendix I. Appendix F Derivation: ViTTT GLU as Linear Attention We show that the simplified GLU used in ViTTT (Han et al., 2025) can be written into a linear attention form with element-wise multiplication. F.1 Architecture and Loss In ViTTT, the GLU is defined as: ft(x)=silu(xW0,t)⊙(xW1,t),f_t(x)=silu(xW_0,t) (xW_1,t), where W0,t,W1,t∈ℝDh×DhW_0,t,W_1,t ^D_h× D_h are both square matrices. We define the kernel function as ϕt(x)≜silu(xW0,t). _t(x) (xW_0,t). Unlike the general form where the kernel function output is followed by matrix multiplication with a final layer, here the GLU uses element-wise multiplication between the nonlinear gating ϕt(x) _t(x) and the linear projection xW1,txW_1,t. The inner-loop loss uses the Frobenius inner product: ℒ(ft(kt),vt)=−⟨ft(kt),vt⟩.L(f_t(k_t),v_t)=- f_t(k_t),v_t . F.2 Gradient Computation The upstream gradient with respect to the GLU output is ∂ℒ∂ft(kt)=−vt ∂ f_t(k_t)=-v_t. Applying the chain rule, the gradient of W1,tW_1,t is: ∇W1,tℒ=kt⊤(∂ℒ∂ft(kt)⊙ϕt(kt))=−kt⊤(vt⊙ϕt(kt)). _W_1,tL=k_t ( ∂ f_t(k_t) _t(k_t) )=-k_t (v_t _t(k_t)). F.3 Linear Attention Form After one step of gradient descent with learning rate η: W1,t+1=W1,t−η∇W1,tℒ=W1,t+ηkt⊤(vt⊙ϕt(kt)).W_1,t+1=W_1,t-η _W_1,tL=W_1,t+η\,k_t (v_t _t(k_t)). Similarly, W0,tW_0,t is updated to W0,t+1W_0,t+1, yielding the updated kernel function ϕt+1(⋅) _t+1(·). Evaluating the model on query qtq_t gives: ot o_t =ft+1(qt)=ϕt+1(qt)⊙(qtW1,t+1) =f_t+1(q_t)= _t+1(q_t) (q_tW_1,t+1) =ϕt+1(qt)⊙(qtW1,t+η(qtkt⊤)(vt⊙ϕt(kt))) = _t+1(q_t) (q_tW_1,t+η\,(q_tk_t )(v_t _t(k_t)) ) =ϕt+1(qt)⊙(qt(W1,t+ηkt⊤(vt⊙ϕt(kt)))). = _t+1(q_t) (q_t (W_1,t+η\,k_t (v_t _t(k_t)) ) ). This is a linear attention form where: • The second term computes a scalar attention weight ⟨qt,kt⟩ q_t,k_t that modulates the value vector vtv_t • ϕt(kt) _t(k_t) acts as a multiplicative gate on the values • ϕt+1(qt) _t+1(q_t) gates the final output The state matrix St=W1,tS_t=W_1,t accumulates outer products of keys and gated values, consistent with the general linear attention framework. Appendix G Derivation: ViTTT Depthwise Convolution as Linear Attention We show that the 3×33× 3 depthwise convolution layer in ViTTT (Han et al., 2025) can be formulated as a form of spatially-local linear attention. G.1 Architecture and Loss Let K,V∈ℝC×H×WK,V ^C× H× W be the spatial key and value tensors, and Wt∈ℝC×1×3×3W_t ^C× 1× 3× 3 be the depthwise convolution weights at step t. The forward pass computes: ft(K)=Conv3×3(K;Wt),f_t(K)=Conv_3× 3(K;W_t), where Conv3×3Conv_3× 3 denotes depthwise convolution with 3×33× 3 kernel. The inner-loop loss uses the Frobenius inner product: ℒ(ft(K),V)=−⟨ft(K),V⟩=−∑c,i,j[ft(K)]c,i,j⋅Vc,i,j.L(f_t(K),V)=- f_t(K),V =- _c,i,j[f_t(K)]_c,i,j· V_c,i,j. G.2 Gradient Computation The upstream gradient is ∂ℒ∂ft(K)=−V ∂ f_t(K)=-V. For depthwise convolution, the gradient with respect to the weight WtW_t can be written as: ∇Wtℒ=K⋆∂ℒ∂ft(K)=−K⋆V, _W_tL=K ∂ f_t(K)=-K V, where ⋆ denotes cross-correlation. Specifically, for each channel c and offset (δy,δx)∈−1,0,12( _y, _x)∈\-1,0,1\^2: [∇Wtℒ]c,δy,δx=−∑i,jKc,i+δy,j+δx⋅Vc,i,j.[ _W_tL]_c, _y, _x=- _i,jK_c,i+ _y,j+ _x· V_c,i,j. G.3 Linear Attention Form After one step of gradient descent with learning rate η: [Wt+1]c,δy,δx=[Wt]c,δy,δx+η∑i,jKc,i+δy,j+δx⋅Vc,i,j.[W_t+1]_c, _y, _x=[W_t]_c, _y, _x+η _i,jK_c,i+ _y,j+ _x· V_c,i,j. When evaluating on query Q∈ℝC×H×WQ ^C× H× W, the output at position (i,j)(i,j) is: Oc,i,j O_c,i,j =∑δy,δx[Wt+1]c,δy,δx⋅Qc,i+δy,j+δx = _ _y, _x[W_t+1]_c, _y, _x· Q_c,i+ _y,j+ _x =∑δy,δx[Wt]c,δy,δx⋅Qc,i+δy,j+δx⏟[Conv(Q;Wt)]c,i,j = _ _y, _x[W_t]_c, _y, _x· Q_c,i+ _y,j+ _x_[Conv(Q;W_t)]_c,i,j +η∑i′,j′(∑δy,δxQc,i+δy,j+δx⋅Kc,i′+δy,j′+δx)⏟spatial attention weight⋅Vc,i′,j′. +η _i ,j ( _ _y, _xQ_c,i+ _y,j+ _x· K_c,i + _y,j + _x )_spatial attention weight· V_c,i ,j . This is a linear attention form where: • The first term corresponds to the initial state S0=WtS_0=W_t • The attention weight between query position (i,j)(i,j) and key position (i′,j′)(i ,j ) is the sum of element-wise products over the 3×33× 3 neighborhood offsets • This spatially-local attention allows each output position to attend to all key-value positions, weighted by the overlap of their local 3×33× 3 neighborhoods Conceptually, since convolution is effectively a sliding-window linear layer, this T component is equivalent to a sliding-window linear attention mechanism. Appendix H Parallel Form of T We present the parallel formulation for T when only W1W_1 is dynamic (while W0W_0 and W2W_2 are static) and weight normalization is omitted. Under these conditions, the kernel function becomes static: ϕ(x)=silu(xW0)⊙(xW2),φ(x)=silu(xW_0) (xW_2), and the state update is associative, enabling parallel computation. H.1 Parallel Formulation The parallel formulation operates on a sequence of N chunks, each of size L. Let ℚ,∈ℝ(NL)×DkQ,K ^(NL)× D_k and ∈ℝ(NL)×DvV ^(NL)× D_v denote the concatenated query, key, and value inputs across all chunks. Define the batched kernel function: Φ()=silu(W0)⊙(W2)∈ℝ(NL)×Dh. (X)=silu(XW_0) (XW_2) ^(NL)× D_h. To express chunk-wise operations, we introduce: • Block-diagonal matrix ∈ℝ(NL)×(NL)B ^(NL)×(NL): =diag(IL,…,IL)B=diag(I_L,…,I_L) with N identity blocks • Per-chunk learning rate ∈ℝN η ^N and per-chunk momentum coefficient ∈ℝN α ^N • Effective accumulation mask ∈ℝN×NC ^N× N that folds the per-chunk learning rates into the cumulative momentum decay: ti=∑j=itηj∏s=i+1jαsfor t≥i,0 otherwise.C_ti\;=\; _j=i^t _j _s=i+1^j _s t≥ i, 0 otherwise. The output can be computed in parallel as: =Φ(ℚ)W1,0+((Φ(ℚ)Φ()⊤)⊙↑L),O= (Q)\,W_1,0+ (( (Q)\, (K) ) L )\,V, where (⋅)↑L(·) L denotes the Kronecker product with L×L1_L× L to expand the N×N× N mask to (NL)×(NL)(NL)×(NL). H.2 Proof of Equivalence We prove that this parallel formulation is equivalent to the sequential recurrence. Proof. Consider the sequential formulation where at each chunk t, the weight update is: ΔWt=∇W1ℒ(ft(Kt))+αtΔWt−1,W1,t+1=W1,t−ηtΔWt. W_t= _W_1L(f_t(K_t))+ _t W_t-1, W_1,t+1=W_1,t- _t W_t. Since ∇W1ℒ=Φ(Kt)⊤∂ℒ∂ft(Kt) _W_1L= (K_t) ∂ f_t(K_t) and using the Frobenius inner product loss, we have: ΔWt=−Φ(Kt)⊤Vt+αtΔWt−1. W_t=- (K_t) V_t+ _t W_t-1. Step 1: Unroll the momentum recurrence. Expanding ΔWt W_t: ΔWt W_t =−∑i=0t(∏s=i+1tαs)Φ(Ki)⊤Vi. =- _i=0^t ( _s=i+1^t _s ) (K_i) V_i. Step 2: Unroll the weight recurrence. From W1,t+1=W1,0−∑j=0tηjΔWjW_1,t+1=W_1,0- _j=0^t _j W_j, substituting and rearranging: W1,t+1 W_1,t+1 =W1,0+∑i=0tΦ(Ki)⊤Vi⋅∑j=itηj∏s=i+1jαs⏟cit. =W_1,0+ _i=0^t (K_i) V_i· _j=i^t _j _s=i+1^j _s_c_i^t. Step 3: Compute the sequential output. The output at chunk t is: Ot O_t =Φ(Qt)W1,t+1=Φ(Qt)W1,0+∑i=0tΦ(Qt)Φ(Ki)⊤Vi⋅cit. = (Q_t)\,W_1,t+1= (Q_t)\,W_1,0+ _i=0^t (Q_t)\, (K_i) V_i· c_i^t. Step 4: Match with parallel formulation. By construction, the mask ↑LC L at block (t,i)(t,i) has value ti=∑j=itηj∏s=i+1jαs=citC_ti= _j=i^t _j _s=i+1^j _s=c_i^t for i≤ti≤ t. For any token position p in chunk t, the parallel form therefore computes: []p [O]_p =Φ(Qt)pW1,0+∑i=0tΦ(Qt)pΦ(Ki)⊤Vi⋅cit, = (Q_t)_p\,W_1,0+ _i=0^t (Q_t)_p\, (K_i) V_i· c_i^t, which exactly matches the sequential output. ∎ Appendix I Non-Reducible Case Analysis In Section 6.2, we showed that T becomes reducible (and thus parallelizable) when only W1W_1 is dynamic while W0W_0 and W2W_2 are static. Here, we analyze two cases that break reducibility: (1) updating the kernel function parameters Θ=W0,W2 =\W_0,W_2\, and (2) applying weight normalization. I.1 Case 1: Dynamic Kernel Function (Updating W0W_0 and W2W_2) When W0W_0 and W2W_2 are also updated, the kernel function ϕt(x)=silu(xW0,t)⊙(xW2,t) _t(x)=silu(xW_0,t) (xW_2,t) becomes history-dependent. Consider the gradient update for W0W_0 (the analysis for W2W_2 is analogous): ∇W0,tℒ=Kt⊤(∂ℒ∂ft(Kt)W1,t⊤⊙(KtW2,t)⊙silu′(KtW0,t)). _W_0,tL=K_t ( ∂ f_t(K_t)W_1,t (K_tW_2,t) (K_tW_0,t) ). The weight update becomes: W0,t+1=W0,t−ηt∇W0,tℒ.W_0,t+1=W_0,t- _t _W_0,tL. Expanding recursively, the kernel function at step t depends on W0,tW_0,t: ϕt(Kt)=silu(KtW0,t)⊙(KtW2,t). _t(K_t)=silu(K_tW_0,t) (K_tW_2,t). Substituting the recursive expression for W0,tW_0,t, the kernel function involves nested nonlinearities: ϕt(Kt)=silu(Kt(W0,t−1−ηt−1∇W0,t−1ℒ))⊙(⋯), _t(K_t)=silu (K_t (W_0,t-1- _t-1 _W_0,t-1L ) ) (·s), where ∇W0,t−1ℒ _W_0,t-1L itself contains silu′(Kt−1W0,t−1)silu (K_t-1W_0,t-1). The nested silusilu and silu′silu functions create a non-linear dependency chain: computing ϕt _t requires W0,tW_0,t, which depends on silu′(Kt−1W0,t−1)silu (K_t-1W_0,t-1), which in turn depends on W0,t−1W_0,t-1, and so on. This nested structure prevents expressing the output as a simple weighted sum over history, breaking the associativity required for parallel prefix scan. I.2 Case 2: Weight Normalization Even when only W1W_1 is dynamic, applying weight normalization (as in LaCT, see Appendix E) introduces non-reducibility for the parallel formulation. As discussed in Appendix E, weight normalization does not break the linear attention interpretation—it simply normalizes the state StS_t after each token. However, it does prevent the parallel computation enabled by the sum form. The key issue is that normalization is not associative: Norm(A+B)≠Norm(A)+Norm(B). (A+B) (A)+Norm(B). (2) The parallel formulation in Section 6.2 relies on expressing the state as St+1=S0+∑i=0tϕ(Ki)⊤miS_t+1=S_0+ _i=0^tφ(K_i) m_i, which can be computed via associative parallel prefix scan. With weight normalization, the state becomes: St+1=Norm(Norm(St−1+ϕ(Kt−1)⊤mt−1)+ϕ(Kt)⊤mt).S_t+1=Norm (Norm(S_t-1+φ(K_t-1) m_t-1)+φ(K_t) m_t ). This nested structure creates a strict sequential dependency: computing St+1S_t+1 requires the fully normalized StS_t, which requires St−1S_t-1, and so on. The associativity required for parallel prefix scan is broken, forcing token-by-token sequential computation even though the underlying mechanism remains linear attention.