Paper deep dive
GradLev: Token-Parallel Test-Time Training Via Costate Prediction
Bo Liu, Qiang Liu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 9/30/2026, 7:05:12 AM
Summary
The paper introduces GradLev, a method for token-parallel test-time training (TTT) that overcomes the sequential dependency of gradient writes. It uses an auxiliary causal network to predict costates (activation gradients) in parallel, which are then used in associative scans to compute adapted weights and forward activations. A consistency loss aligns these predicted costates with targets reconstructed via backward pass, ensuring exact recovery of the sequential online learner at deployment.
Entities (10)
Relation Signals (8)
GradLev → enables → Parallel Training
confidence 95% · GradLev: Token-Parallel Test-Time Training
Q → predicts → Costates
confidence 95% · a causal auxiliary network Q predicts costates across all tokens in parallel
GradLev → uses → Costate Prediction
confidence 95% · GradLev leverages this duality: a causal auxiliary network predicts costates across all tokens in parallel
Associative Scan → computes → Adapted Weights
confidence 90% · associative scans compute the adapted weights and forward activations
GradLev → recovers → Sequential Online Learner
confidence 90% · Exact consistency guarantees exact recovery of the sequential online learner
GradLev → replaces → Sequential Gradient Writes
confidence 90% · GradLev replaces sequential gradient computation during training with three stages
Consistency Loss → supervises → Q
confidence 90% · the resulting gradient targets supervise the predictor via a consistency loss
GradLev → isrelatedto → Meta-Learning
confidence 85% · Meta-learning grounds this process... GradLev learns the initialization and update rule
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Test-time training (TTT) allows a model to improve its predictions at inference time by updating weights after every observed token. However, sequential gra- dient writes make parallel training difficult. We observe that, given layer inputs and activation gradients (costates), online gradient descent admits exact parallel scans for both forward evaluation and reverse backpropagation. GradLev lever- ages this duality: a causal auxiliary network predicts costates across all tokens in parallel; associative scans compute the adapted weights and forward activations and propagate gradients backward; and the resulting gradient targets supervise the predictor via a consistency loss. Exact consistency guarantees exact recovery of the sequential online learner. At deployment, the auxiliary predictor is discarded, and the model updates natively via token-by-token forward and backward passes.
Tags
Links
- Source: https://arxiv.org/abs/2609.34174v2
- Canonical: https://arxiv.org/abs/2609.34174v2
Trouble viewing inline? Open PDF directly →
Full Text
51,200 characters extracted from source content.
Expand or collapse full text
GradLev: Token-Parallel Test-Time Training via Costate Prediction Bo Liu Qiang Liu Affiliation: The University of Texas at Austin Email: bliu,lqiang@cs.utexas.edu Abstract Test-time training (T) allows a model to improve its predictions at inference time by updating weights after every observed token. However, sequential gradient writes make parallel training difficult. We observe that, given layer inputs and activation gradients (costates), online gradient descent admits exact parallel scans for both forward evaluation and reverse backpropagation. GradLev leverages this duality: a causal auxiliary network predicts costates across all tokens in parallel; associative scans compute the adapted weights and forward activations and propagate gradients backward; and the resulting gradient targets supervise the predictor via a consistency loss. Exact consistency guarantees exact recovery of the sequential online learner. At deployment, the auxiliary predictor is discarded, and the model updates natively via token-by-token forward and backward passes. Predict with QQcostates g^t g_tForward with PPy^t=W^txt=(W0+Δ^t)xt y_t= W_tx_t=(W_0+ _t)x_tBackprop with PPcostate targets g~t g_tℒ=ℒCE(y^)+λ‖g^−sg(g~)‖22L=L_CE( y)+λ\, \| [rgb]0.7695,0.3555,0.2695 g-sg( [rgb]0.1563,0.4688,0.7109 g) \|_2^2 Figure 1: A prefiller Q predicts surrogate output costates g^t g_t. Model P accumulates these gradient updates via a parallel prefix scan to compute predictions y^t=W^txt y_t= W_tx_t, where W^t W_t denotes the hypothetical test-time trained weight matrix at time t. Then we backpropagate through P to get the grounded costate targets g~t g_t. Because exact test-time training requires g^t=g~t g_t= g_t, we jointly train P and Q with a costate consistency loss in addition to the normal cross-entropy objective. 1 Introduction Every incoming token in language modeling serves a dual role: it is both a prediction target and an immediate supervisory signal. Rather than leaving internal representations static, an adaptive model can assimilate each new observation by updating temporary weights before predicting the next token. This online adaptation principle underlies dynamic evaluation and test-time training (T) (Krause et al., 2018; Sun et al., 2020). By decoupling persistent cross-sequence knowledge (slow weights) from local within-sequence adaptations (fast weights) (Schmidhuber, 1992; Ba et al., 2016), test-time optimization provides a natural mechanism for tracking long-range document context, bursty entity bindings, and local distribution shifts. Meta-learning grounds this process by training the initialization and update rule so that rapid adaptation directly benefits future predictions (Finn et al., 2017; Clark et al., 2022; Sun et al., 2025; Tandon et al., 2025). The fundamental bottleneck is parallelism. Standard Transformer training evaluates all sequence positions concurrently via causal attention (Vaswani et al., 2017). In contrast, token-level test-time backpropagation introduces a strict serial dependency: the prediction at position t requires the adapted weights θt _t, which depend on the gradient at t−1t-1, which in turn requires the unrolled activations and predictions from t−1t-1. To bypass this sequential dependency, existing methods either coarsen updates into multi-token chunks (Sun et al., 2025; Feng et al., 2026; Zhang et al., 2026)—sacrificing immediate reactivity—or restrict updates to simple associative recurrences found in linear attention and state-space models (SSMs) (Katharopoulos et al., 2020; Yang et al., 2024a; Dao and Gu, 2024; Liu et al., 2025). Training an expressive model that performs a full forward pass, backward pass, and weight update at every single token while retaining sequence-wide parallel training remains an open challenge. In this work, we introduce GradLev, a method that achieves fully parallel training for token-level test-time backpropagation through a lifted optimization framework (Liu and Liu, 2026). Instead of sequentially computing gradient writes position by position, an auxiliary causal network Q predicts the costates—the gradients of the loss with respect to the outputs of the adapted layers—across all tokens simultaneously. Conditioned on these predicted costates and the layer inputs, online gradient descent decouples into an affine matrix recurrence. This structure allows the primary learner P to evaluate its forward states, adapted predictions y^t y_t, and transposed backward passes concurrently across the entire sequence via efficient parallel prefix scans. To train the predictor without human annotations or serial unrolling, model P retraces backpropagation through the scanned weights to generate target costates g~t g_t. The entire architecture is optimized end-to-end with a joint objective: task cross-entropy on the adapted outputs and a consistency loss that aligns predicted costates g^t g_t with their retraced targets g~t g_t. These targets are exact gradients on the proposed trajectory. Training stops their outer derivatives and uses the first-order rule described below. At zero consistency error, the parallel training trajectory matches the causal online trajectory. At test time, the predictor Q is discardedand P updates online by writing exact per-token gradients. 2 Test-Time Backpropagation Test-time backpropagation uses each observed target to improve later predictions. For tokens s0,…,sTs_0,…,s_T, let θt _t be the parameters used to predict st+1s_t+1 from the prefix s≤t=(s0,…,st)s_≤ t=(s_0,…,s_t). Its negative log-probability is ℓt(θt) _t( _t), with the prefix and historical cache implicit. An update rule U adapts the model from its learned initialization θ0 _0: θt+1=U(θt,∇θℓt(θt)),minθ0[1T∑t=0T−1ℓt(θt)]. _t+1=U\! ( _t, _θ _t( _t) ), _ _0\;E\! [ 1T _t=0^T-1 _t( _t) ]. (1) The expectation is over sequences. Each target is scored before updating, and its test-time gradient holds earlier updates and cached activations fixed. Learning the initialization and update rule is a meta-learning problem (Finn et al., 2017; Franceschi et al., 2018), connecting dynamic evaluation and T (Krause et al., 2018; Sun et al., 2020) with adaptive language modeling (Clark et al., 2022; Sun et al., 2025; Tandon et al., 2025). In practice, one could select a subset of θ0 _0 as the fast weights to adapt online. Slow weights of θ0 _0, including learned coefficients of U, are trained by the outer objective and remain fixed at deployment. 3 GradLev GradLev replaces sequential gradient computation during training with three stages: costate prediction with Q, forward evaluation with P, and backward reconstruction with P. Following Maglev’s lifted training principle (Liu and Liu, 2026), Q proposes quantities that P computes itself at deployment; a consistency loss aligns the proposals with those computations. Observation: known costates expose exact scans. Consider yt=Wtxty_t=W_tx_t, with xt∈ℝdx_t ^d, yt∈ℝmy_t ^m, and slow initialization W0∈ℝm×dW_0 ^m× d. The costate gt=∇yℓt(yt)g_t= _y _t(y_t) differentiates through subsequent layers to the prediction loss. Within each reset segment, use step size ηt≥0 _t≥ 0, retention αt∈[0,1] _t∈[0,1], and centered weight decay: Wt+1 W_t+1 =W0+αt(Wt−W0)−ηtgtxt⊤, =W_0+ _t(W_t-W_0)- _tg_tx_t , (2) Δt:=Wt−W0 _t:=W_t-W_0 =−∑i=0t−1ηi(∏j=i+1t−1αj)gixi⊤,Δ0=0. =- _i=0^t-1 _i ( _j=i+1^t-1 _j )g_ix_i , _0=0. Positions start at zero; empty sums are zero and empty products are one. Each past write is discounted by subsequent retentions, with decay centered at W0W_0. Given inputs, costates, and gates, gradient descent is an exact parallel scan. Forward reads WtxtW_tx_t and backward reads Wt⊤gtW_t g_t reuse these writes, so the predictor need only supply costate vectors. 1. Costate Prediction with Q. The prefiller predicts g^t∈ℝm g_t ^m for each adapted map from causal features before the adapted subnetwork. It may observe the current target st+1s_t+1: its proposal writes only to weights used at later positions. All proposals are computed in parallel across tokens. 2. Forward with P. Replace gig_i by g^i g_i in Equation 2. An exclusive scan, which includes only writes at i<ti<t, computes Δ^t+1=αtΔ^t−ηtg^txt⊤,y^t=(W0+Δ^t)xt. _t+1= _t _t- _t g_tx_t , y_t=(W_0+ _t)x_t. (3) Initialize Δ^0=0 _0=0 and read before each write. For multiple adapted layers, xtx_t is each layer’s input on this proposed trajectory. Layers run in forward order, with token positions evaluated in parallel within each layer. The negative sign belongs to the update; Δ^t _t is added to W0W_0. 3. Backward Reconstruction with P. Start at the prediction loss and work backward through the layers. Suppose the higher layers have supplied the output costate g~t=∂ℓt/∂y^t g_t=∂ _t/∂ y_t at the current map. Holding its incoming weights and historical caches fixed, pass the gradient to its input: g~tx:=∂ℓt∂xt=(W0+Δ^t)⊤g~t=W0⊤g~t+Δ^t⊤g~t. g_t^\,x:= ∂ _t∂ x_t=(W_0+ _t) g_t=W_0 g_t+ _t g_t. (4) The last term is a transpose scan over the same historical writes (xi,g^i)(x_i, g_i) used in the forward, queried by the incoming g~t g_t. If xt=σ(zt)x_t=σ(z_t) follows an elementwise activation, pass g~tz=σ′(zt)⊙g~tx g_t^\,z=σ (z_t) g_t^\,x to the preceding linear layer; ztz_t is its preactivation and ⊙ denotes elementwise multiplication. Repeating these steps gives every layer’s exact costate on the proposed trajectory. Training objective. Combine the prediction loss with a consistency loss that matches each proposal g^t g_t to its reconstructed costate g~t g_t: ℒ=1T∑t=0T−1[ℓt(y^t)+λm‖g^t−sg(g~t)‖22].L= 1T _t=0^T-1 [ _t( y_t)+ λm \| g_t-sg( g_t) \|_2^2 ]. (5) Here λ≥0λ≥ 0 weights consistency and sgsg is the stop-gradient operator. For multiple maps, average their dimension-normalized consistency losses. The three stages construct this objective; the optimizer’s backward pass is separate. At deployment, remove Q and update P from its own costates after scoring each target. Theorem 1 (Causal consistency and exact recovery). Fix a finite input-target sequence and a deterministic causal network with acyclic within-token computation and finite local derivatives. Training and deployment share initialization, resets, causal gates, and update rules; reads precede writes, and the test-time reverse holds incoming weights and historical caches fixed. Let G=(G0,…,GT−1)G=(G_0,…,G_T-1) collect all proposed costates by token, and let F(G)F(G) collect the reconstructed targets. Each target block FtF_t depends only on G<t:=(G0,…,Gt−1)G_<t:=(G_0,…,G_t-1). Consequently, F has a unique fixed point G⋆G , the serial learner’s costate trajectory: G^=F(G^)⟺G^=G⋆. G=F( G) G=G . At this solution, the parallel and serial matrices, activations, and test-time gradients coincide in exact arithmetic. Proof. An incoming matrix at token t uses only writes at i<ti<t, so the forward and reverse at t depend only on G<tG_<t. Thus G0⋆=F0(∅)G_0 =F_0( ) and Gt⋆=Ft(G<t⋆)G_t =F_t(G_<t ) recursively determine the unique solution. The serial learner obeys this recursion and the same update rule. ∎ With fixed update coefficients and detached gradient writes, exact consistency also recovers the first-order model-agnostic meta-learning (MAML) slow-weight gradient for the same per-token objective (Finn et al., 2017). Our implementation additionally learns the gates and retains their derivatives. The theorem guarantees exact test-time gradients, not the full meta-gradient. Nonzero consistency errors can be amplified (Appendix D). Practical architecture and training. A learned affine gate f:ℝd→ℝ2f:R^d ^2 maps input xt∈ℝdx_t ^d to scalar logits. For matrix j with input width djd_j and a fixed write-strength cap μmax>0 _ >0, use (at,bt)=f(sg(xt)),αt=sigmoid(at),ηj,t=μmaxdjsigmoid(bt).(a_t,b_t)=f\! (sg(x_t) ), _t=sigmoid(a_t), _j,t= _ d_jsigmoid(b_t). In practice, one could share the retension αt _t and learning rate ηj,t _j,t across different layers from a shared input activation xtx_t. Applying GradLev on top of architectures with Attention requires special care. Let qt,kt,vtq_t,k_t,v_t be the current query, key, and value, and K<t,V<tK_<t,V_<t the past kv-caches. The test-time reverse differentiates the attention output ot=Attn(qt,[sg(K<t);kt],[sg(V<t);vt]).o_t=Attn\! (q_t,[sg(K_<t);k_t],[sg(V_<t);v_t] ). Here AttnAttn is scaled dot-product attention and [;][\,;\,] concatenates rows. Past cache entries are read unchanged. As a result, for the step-3 backpropagation through P, one should modify the reverse computation graph such that we treat the previous kv-cache as fixed constants. 4 Related work This section provides a brief overview of existing works related to GradLev in online adaptation, fast weights, and parallel recurrent training. Fast weights and learning to learn. Fast-weight programmers separate persistent parameters from temporary, input-dependent weights (Schmidhuber, 1992); later work uses fast matrices to associate recent activations (Ba et al., 2016). Learned optimizers parameterize the update rule (Andrychowicz et al., 2016), while model-agnostic meta-learning (MAML) and bilevel optimization learn through adaptation (Finn et al., 2017; Franceschi et al., 2018). GradLev learns the initialization and gates of an explicit gradient-descent rule. Related work interprets in-context learning as optimization: Schlag et al. (2021) connect linear Transformers to fast-weight programming, and von Oswald et al. (2023) study gradient descent within linear attention. GradLev instead differentiates its next-token loss at deployment to update selected weights. Test-time training and adaptive language models. Dynamic evaluation updates language models on observed context (Krause et al., 2018), and T learns from test-instance supervision (Sun et al., 2020). Recent theory studies when test-time gradient steps help linear Transformers under task and distribution mismatch (Gozeten et al., 2025). Fast Weight Layers add a final adaptive module and express gradient writes evaluated at slow weights as linear attention (Clark et al., 2022). T layers make a learned linear model or multilayer perceptron (MLP) the recurrent state (Sun et al., 2025). End-to-End T meta-trains next-token-loss adaptation (Tandon et al., 2025), while In-Place T adapts existing MLP down-projections with an aligned objective (Feng et al., 2026). Large Chunk Test-Time Training (LaCT) uses large update chunks for hardware efficiency and nonlinear fast-weight capacity (Zhang et al., 2026). GradLev parallelizes training through costate prediction while preserving token-level updates at deployment. SSMs and memory as online learners. Longhorn treats SSM layers as meta-learned online learners. Pretraining learns input representations; inference updates associative memory with a diagonal approximation to an implicit regression update (Liu et al., 2025). Titans adds neural memory with learned retention and momentum (Behrouz et al., 2025b). Miras organizes memory design by architecture, internal objective, retention, and optimization rule (Behrouz et al., 2025c). Atlas optimizes memory over current and past context (Behrouz et al., 2025a), while MesaNet solves regularized regression using chunkwise conjugate gradients (von Oswald et al., 2026). GradLev uses the predictive loss to adapt selected maps and reconstructs their forward and reverse with scans. Efficient matrix recurrences. Linear attention gives a recurrent matrix representation of attention (Katharopoulos et al., 2020). The Structured State Space sequence model (S4) supports efficient long-range computation (Gu et al., 2022), Mamba makes state transitions input-dependent (Gu and Dao, 2024), and state-space duality unifies efficient recurrent and chunkwise computation (Dao and Gu, 2024). Mamba-3 further develops discretization, complex-valued state updates, and multi-input, multi-output states (Lahoti et al., 2026). Gated linear attention (GLA) provides data-dependent forgetting with hardware-efficient training (Yang et al., 2024a); DeltaNet and Gated DeltaNet support error-correcting associative writes (Yang et al., 2024b; Yang et al., 2025). GradLev reuses gated additive scans (Yang and Zhang, 2024) with negative costate writes, rather than Gated DeltaNet’s associative-regression residual. Parallel training of nonlinear recurrence. Associative scans parallelize linear recurrences (Martin and Cundy, 2018). Parallel nonlinear equation solving, the nonlinear differential equation as fixed-point iteration (DEER) framework, and later stable solvers recover trajectories through iterative updates across positions (Song et al., 2021; Lim et al., 2024; Gonzalez et al., 2024). The Latent Recurrent Transformer (LRT) trains previous-token latent recurrence through parallel refinement (Huang et al., 2026). Other methods learn auxiliary trajectories: NextLat predicts latent transitions (Teoh et al., 2025), and Supervised Memory Training learns recurrent neural network (RNN) transitions from Transformer-generated predictive states (Kumar and Isola, 2026). Maglev uses a causal prefiller and consistency loss to train a decoder that produces its own memories at inference (Liu and Liu, 2026). GradLev predicts costates, whose writes support forward and transpose scans. This relates to synthetic gradients (Jaderberg et al., 2017), but the predictions here are supervised by a full reverse and removed at deployment. Training constructs the trajectory in three stages without iterative refinement. 5 Experiments In this section, we provide two preliminary experiments to demonstrate the usefulness of GradLev. Section 5.1 asks whether the scans reproduce a token-level MLP learner given exact costates. Section 5.2 evaluates learned costate prediction through the deployment quality of a 300M language model, adapting only its final MLP. 5.1 Recovering token-level T in an MLP This experiment checks whether parallel forward and backward scans reproduce the outputs and gradients of serial T. It tests the construction at exact consistency, independently of learning Q. Experimental setup. Each sequence contains T=32T=32 independent input–target pairs, with xt∼(0,I4)x_t (0,I_4) and ct∼(0,I3)c_t (0,I_3); IkI_k is the k×k× k identity. A two-layer MLP predicts yt=W2,ttanh(W1,txt),ℓt(yt)=12‖yt−ct‖22,y_t=W_2,t (W_1,tx_t), _t(y_t)= 12\|y_t-c_t\|_2^2, (6) where W1,t∈ℝ7×4W_1,t ^7× 4 and W2,t∈ℝ3×7W_2,t ^3× 7. Serial autograd updates both matrices after each token. Its costates are supplied to one parallel forward/backward construction, with the same initialization and update rule. A control replaces only the adapted transpose W2,t⊤W_2,t in the backward by W2,0⊤W_2,0 . Figure 2 reports maximum absolute discrepancies over coordinates, tokens, and three seeds in 64-bit floating-point arithmetic (FP64). Appendix B specifies initialization and gates. 10−1610^-1610−1210^-1210−810^-810−410^-410010^0First-matrix gradientsHidden costatesOutputs×10−161.18\!×\!10^-16 (both)×10−162.22\!×\!10^-16×10−166.66\!×\!10^-160.3760.3760.6260.626Maximum absolute error (log scale)Adapted transposeSlow transpose only Figure 2: Exact forward and backward recovery. With supplied exact costates, discrepancies from serial T remain at rounding precision. Replacing only the adapted transpose preserves the outputs but changes the reverse gradients. Errors are maxima over coordinates, tokens, and three seeds. Results and interpretation. Maximum output, hidden-costate, and first-matrix gradient discrepancies are 1.18×10−161.18× 10^-16, 2.22×10−162.22× 10^-16, and 6.66×10−166.66× 10^-16. Using only the slow transpose raises the last two to 0.3757 and 0.6261. Thus the scans recover serial MLP predictions and test-time gradients when proposals are exact. This verifies numerical equivalence; it does not measure learned costate-prediction accuracy or regression generalization. 5.2 300M language modeling with final-MLP adaptation This study asks whether learning with Q improves the deployed learner over a static Transformer and four-chunk T at the same token budget. Only the final MLP is adapted in both adaptive models. Experimental setup. All three models have 21 Transformer layers, model width d=1024d=1024, two-matrix MLP width m=4096m=4096, 16 attention heads, a tied 32,768-token vocabulary, and context length 1024. Chunk-T updates every 256 tokens using detached, summed gradients; static has no test-time updates. GradLev uses a two-block causal Transformer Q with two costate heads, conditioned on the MLP input and observed target embedding. Retention is αt=1 _t=1, the write cap is μmax=4 _ =4, and write strength starts at 0.9; Q’s heads start at zero. Chunk-T uses the same gate family and initialization on fragment-mean inputs. Cross-entropy trains P and the gates; consistency trains Q. Appendix E gives the architecture and gradient stops. All models share slow initialization, ClimbMix data order (Diao et al., 2025), seed 42, a 262,144-token batch, and the P optimizer schedule. Each trains for 112,420 steps (29.47B tokens), exceeding 100 tokens per non-embedding training parameter, including Q. A short screen selects half-rate Q, with λ=1λ=1. Training compute differs; Appendix E gives budgets and tuning details. Deployment evaluation removes Q, scores before updating, and resets at document/context boundaries. Metrics are validation negative log-likelihood (NLL), FineWeb-Edu bits per byte (BPB), LAMBADA word perplexity (PPL), and mean accuracy over Maglev’s eight zero-shot tasks (Liu and Liu, 2026). Evaluation covers 65,536 validation tokens crossing chunk boundaries, 512 FineWeb-Edu documents (Penedo et al., 2024), and 27,072 task examples; Appendix F gives details. 005050100100−0.01-0.01000.010.010.020.02Training steps (thousands)ΔNLL -T (4) Figure 3: Deployment NLL gap. GradLev minus control; negative is better. Recorded GB300 scores, without smoothing. Metric Static Chunk-T (4) GradLev Negative log-likelihood (↓ ) Validation, nats/token 2.5733 2.5726 2.5707 Bits per byte (↓ ) FineWeb-Edu 0.7527 0.7530 0.7517 Word perplexity (↓ ) LAMBADA 24.66 24.54 23.56 Zero-shot accuracy (%) (↑ ) Mean over eight tasks 50.77 50.81 50.99 Table 1: 300M final-MLP results. B300 deployment after 29.47B training tokens. Bold marks best point estimates; task-level scores and uncertainty are in Appendix F. Results and observations. GradLev shows slight improvements in likelihood and mean zero-shot accuracy, probably due to the limited richness from adapting only the last MLP layer. Its final NLL is lower by 0.001856 than chunk-T and by 0.002617 than static; mean accuracy is 50.99%, versus 50.81% and 50.77%, respectively. GradLev leads chunk-T and static at every shared checkpoint from step 65,536 onward (Figure 3). 6 Limitations and future work This work is a preliminary study of costate prediction as a mechanism for parallelizing token-level test-time training. Its primary contribution is the formulation of this approach; the empirical gains are modest, and the current experiments do not establish its effectiveness at larger scales. Demonstrating consistent benefits across model sizes, adaptation depths, and tasks remains an important direction for future work. Several architectural questions also remain open. Extending adaptation to deeper networks requires a scalable predictor Q that maintains accurate costate predictions across layers while preserving the computational benefits of parallel training. Sharing parameters or intermediate representations between P and Q may help reduce this overhead, but the resulting trade-offs between prediction accuracy, training cost, and optimization stability require further study. Another direction is to restrict gradient-based adaptation to a small set of parameters, which need not be full weight matrices. Determining which parameters to adapt, and how to design architectures that make effective use of these updates, could broaden the practical scope of GradLev. Addressing these questions is necessary to assess whether the approach can support effective test-time learning in production-scale models. References Andrychowicz et al. (2016) Marcin Andrychowicz, Misha Denil, Sergio Gómez, Matthew W. Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando de Freitas. Learning to Learn by Gradient Descent by Gradient Descent. In NeurIPS, 2016. Ba et al. (2016) Jimmy Ba, Geoffrey E. Hinton, Volodymyr Mnih, Joel Z. Leibo, and Catalin Ionescu. Using Fast Weights to Attend to the Recent Past. In NeurIPS, 2016. Behrouz et al. (2025a) Ali Behrouz, Zeman Li, Praneeth Kacham, Majid Daliri, Yuan Deng, Peilin Zhong, Meisam Razaviyayn, and Vahab Mirrokni. ATLAS: Learning to Optimally Memorize the Context at Test Time. arXiv:2505.23735, 2025a. Behrouz et al. (2025b) Ali Behrouz, Peilin Zhong, and Vahab Mirrokni. Titans: Learning to Memorize at Test Time. In NeurIPS, 2025b. Behrouz et al. (2025c) Ali Behrouz, Meisam Razaviyayn, Peilin Zhong, and Vahab Mirrokni. It’s All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimization. arXiv:2504.13173, 2025c. Bisk et al. (2020) Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: Reasoning about Physical Commonsense in Natural Language. In AAAI, 2020. Clark et al. (2018) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think You Have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. arXiv:1803.05457, 2018. Clark et al. (2019) Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions. In NAACL, 2019. Clark et al. (2022) Kevin Clark, Kelvin Guu, Ming-Wei Chang, Panupong Pasupat, Geoffrey Hinton, and Mohammad Norouzi. Meta-Learning Fast Weight Language Models. In EMNLP, 2022. Dao and Gu (2024) Tri Dao and Albert Gu. Transformers Are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality. In ICML, 2024. Diao et al. (2025) Shizhe Diao, Yu Yang, Yonggan Fu, Xin Dong, Dan Su, Markus Kliegl, Zijia Chen, Peter Belcak, Yoshi Suhara, Hongxu Yin, Mostofa Patwary, Yingyan Lin, Jan Kautz, and Pavlo Molchanov. Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training. arXiv:2504.13161, 2025. Feng et al. (2026) Guhao Feng, Shengjie Luo, Kai Hua, Ge Zhang, Wenhao Huang, Di He, and Tianle Cai. In-Place Test-Time Training. In ICLR, 2026. Finn et al. (2017) Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. In ICML, 2017. Franceschi et al. (2018) Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pontil. Bilevel Programming for Hyperparameter Optimization and Meta-Learning. In ICML, 2018. Gonzalez et al. (2024) Xavier Gonzalez, Andrew Warrington, Jimmy T. H. Smith, and Scott W. Linderman. Towards Scalable and Stable Parallelization of Nonlinear RNNs. In NeurIPS, 2024. Gozeten et al. (2025) Halil Alperen Gozeten, Muhammed Emrullah Ildiz, Xuechen Zhang, Mahdi Soltanolkotabi, Marco Mondelli, and Samet Oymak. Test-Time Training Provably Improves Transformers as In-context Learners. In ICML, 2025. Gu and Dao (2024) Albert Gu and Tri Dao. Mamba: Linear-Time Sequence Modeling with Selective State Spaces. In COLM, 2024. Gu et al. (2022) Albert Gu, Karan Goel, and Christopher Ré. Efficiently Modeling Long Sequences with Structured State Spaces. In ICLR, 2022. Huang et al. (2026) Zeyi Huang, Xuehai He, LiLiang Ren, Yiping Wang, Baolin Peng, Hao Cheng, Shuohang Wang, Pengcheng He, Jianfeng Gao, Yong Jae Lee, and Yelong Shen. Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling Behavior. arXiv:2605.26797, 2026. Jaderberg et al. (2017) Max Jaderberg, Wojciech Marian Czarnecki, Simon Osindero, Oriol Vinyals, Alex Graves, David Silver, and Koray Kavukcuoglu. Decoupled Neural Interfaces Using Synthetic Gradients. In ICML, 2017. Katharopoulos et al. (2020) Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers Are RNNs: Fast Autoregressive Transformers with Linear Attention. In ICML, 2020. Krause et al. (2018) Ben Krause, Emmanuel Kahembwe, Iain Murray, and Steve Renals. Dynamic Evaluation of Neural Sequence Models. In ICML, 2018. Kumar and Isola (2026) Akarsh Kumar and Phillip Isola. Pretraining Recurrent Networks without Recurrence. arXiv preprint arXiv:2606.06479, 2026. Lahoti et al. (2026) Aakash Lahoti, Kevin Y. Li, Berlin Chen, Caitlin Wang, Aviv Bick, J. Zico Kolter, Tri Dao, and Albert Gu. Mamba-3: Improved Sequence Modeling using State Space Principles. arXiv:2603.15569, 2026. Lim et al. (2024) Yi Heng Lim, Qi Zhu, Joshua Selfridge, and Muhammad Firmansyah Kasim. Parallelizing Non-Linear Sequential Models over the Sequence Length. In ICLR, 2024. Liu and Liu (2026) Bo Liu and Qiang Liu. Maglev: Sliding Recurrent Memory. arXiv:2608.02870, 2026. Liu et al. (2025) Bo Liu, Rui Wang, Lemeng Wu, Yihao Feng, Peter Stone, and Qiang Liu. Longhorn: State Space Models Are Amortized Online Learners. In ICLR, 2025. Martin and Cundy (2018) Eric Martin and Chris Cundy. Parallelizing Linear Recurrent Neural Nets Over Sequence Length. In ICLR, 2018. Paperno et al. (2016) Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc-Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA Dataset: Word Prediction Requiring a Broad Discourse Context. In ACL, 2016. Penedo et al. (2024) Guilherme Penedo, Hynek Kydlíček, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro von Werra, and Thomas Wolf. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. In NeurIPS Datasets and Benchmarks, 2024. Sakaguchi et al. (2020) Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An Adversarial Winograd Schema Challenge at Scale. In AAAI, 2020. Sap et al. (2019) Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Commonsense Reasoning about Social Interactions. In EMNLP-IJCNLP, 2019. Schlag et al. (2021) Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. Linear Transformers Are Secretly Fast Weight Programmers. In ICML, 2021. Schmidhuber (1992) Jürgen Schmidhuber. Learning to Control Fast-Weight Memories: An Alternative to Dynamic Recurrent Networks. Neural Computation, 4(1):131–139, 1992. Song et al. (2021) Yang Song, Chenlin Meng, Renjie Liao, and Stefano Ermon. Accelerating Feedforward Computation via Parallel Nonlinear Equation Solving. In ICML, 2021. Sun et al. (2020) Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei A. Efros, and Moritz Hardt. Test-Time Training with Self-Supervision for Generalization under Distribution Shifts. In ICML, 2020. Sun et al. (2025) Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang, Sanmi Koyejo, Tatsunori Hashimoto, and Carlos Guestrin. Learning to (Learn at Test Time): RNNs with Expressive Hidden States. In ICML, 2025. Tandon et al. (2025) Arnuv Tandon, Karan Dalal, Xinhao Li, Daniel Koceja, Marcel Rød, Sam Buchanan, Xiaolong Wang, Jure Leskovec, Sanmi Koyejo, Tatsunori Hashimoto, Carlos Guestrin, Jed McCaleb, Yejin Choi, and Yu Sun. End-to-End Test-Time Training for Long Context. arXiv:2512.23675, 2025. Teoh et al. (2025) Jayden Teoh, Manan Tomar, Kwangjun Ahn, Edward S. Hu, Tim Pearce, Pratyusha Sharma, Akshay Krishnamurthy, Riashat Islam, Alex Lamb, and John Langford. Next-Latent Prediction Transformers Learn Compact World Models. arXiv:2511.05963, 2025. Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. In NeurIPS, 2017. von Oswald et al. (2023) Johannes von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers Learn In-Context by Gradient Descent. In ICML, 2023. von Oswald et al. (2026) Johannes von Oswald, Nino Scherrer, Seijin Kobayashi, Luca Versari, Songlin Yang, Maximilian Schlegel, Kaitlin Maile, Yanick Schimpf, Oliver Sieberling, Alexander Meulemans, Guillaume Lajoie, Rif A. Saurous, Charlotte Frenkel, Razvan Pascanu, Blaise Agüera y Arcas, and João Sacramento. MesaNet: Sequence Modeling by Locally Optimal Test-Time Training. In ICLR, 2026. Yang and Zhang (2024) Songlin Yang and Yu Zhang. FLA: A Triton-Based Library for Hardware-Efficient Implementations of Linear Attention Mechanism. Software, 2024. Yang et al. (2024a) Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated Linear Attention Transformers with Hardware-Efficient Training. In ICML, 2024a. Yang et al. (2024b) Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing Linear Transformers with the Delta Rule over Sequence Length. In NeurIPS, 2024b. Yang et al. (2025) Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated Delta Networks: Improving Mamba2 with Delta Rule. In ICLR, 2025. Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a Machine Really Finish Your Sentence?. In ACL, 2019. Zhang and Sennrich (2019) Biao Zhang and Rico Sennrich. Root Mean Square Layer Normalization. In NeurIPS, 2019. Zhang et al. (2026) Tianyuan Zhang, Sai Bi, Yicong Hong, Kai Zhang, Fujun Luan, Songlin Yang, Kalyan Sunkavalli, William T. Freeman, and Hao Tan. Test-Time Training Done Right. In ICLR, 2026. Appendix A Worked example: an online MLP This section derives the online MLP updates and their parallel forward and reverse computations. Let xt∈ℝdx_t ^d be the input, m the hidden width, and σ an elementwise activation. The matrices W1,t∈ℝm×dW_1,t ^m× d and W2,t∈ℝd×mW_2,t ^d× m start at W1,0W_1,0 and W2,0W_2,0. Define the preactivation ztz_t, hidden activation hth_t, and output yty_t by zt=W1,txt,ht=σ(zt),yt=W2,tht.z_t=W_1,tx_t, h_t=σ(z_t), y_t=W_2,th_t. (7) The costates gty=∇yℓt(yt)g_t^y= _y _t(y_t) and gtz=∇zℓt(yt)g_t^z= _z _t(y_t) define the reverse and simultaneous writes: gtz g_t^z =σ′(zt)⊙W2,t⊤gty, =σ (z_t) W_2,t g_t^y, (8) W1,t+1 W_1,t+1 =W1,0+αt(W1,t−W1,0)−μtdgtzxt⊤, =W_1,0+ _t(W_1,t-W_1,0)- _tdg_t^zx_t , W2,t+1 W_2,t+1 =W2,0+αt(W2,t−W2,0)−μtmgtyht⊤. =W_2,0+ _t(W_2,t-W_2,0)- _tmg_t^yh_t . (9) Here ⊙ is elementwise multiplication, αt∈[0,1] _t∈[0,1] is retention, and μt≥0 _t≥ 0 is the shared write coefficient, divided by each map’s input width. Both gradients use pre-update weights. Given costate proposals g^tz∈ℝm g_t^z ^m and g^ty∈ℝd g_t^y ^d, the forward processes the layers in order and evaluates each recurrence by an exclusive token scan: Δ^1,t+1 _1,t+1 =αtΔ^1,t−μtdg^tzxt⊤, = _t _1,t- _td g_t^zx_t , z^t z_t =(W1,0+Δ^1,t)xt, =(W_1,0+ _1,t)x_t, (10) Δ^2,t+1 _2,t+1 =αtΔ^2,t−μtmg^tyh^t⊤, = _t _2,t- _tm g_t^y h_t , h^t h_t =σ(z^t), =σ( z_t), (11) y^t y_t =(W2,0+Δ^2,t)h^t. =(W_2,0+ _2,t) h_t. (12) Both corrections start at zero. The reverse supplies targets g~ty g_t^y =∇yℓt(y^t), = _y _t( y_t), (13) g~tz g_t^z =σ′(z^t)⊙(W2,0⊤g~ty+Δ^2,t⊤g~ty). =σ ( z_t) (W_2,0 g_t^y+ _2,t g_t^y ). The transpose scan uses historical g^iy g_i^y to reverse the same matrices as the forward. To reach earlier adapted layers, continue with (W1,0+Δ^1,t)⊤g~tz(W_1,0+ _1,t) g_t^z; the final-MLP learner needs no further read. The implemented MLP consistency loss is ℒcons=12T∑t=0T−1[‖g^tz−sg(g~tz)‖22m+‖g^ty−sg(g~ty)‖22d].L_cons= 12T _t=0^T-1 [ \| g_t^z-sg( g_t^z)\|_2^2m+ \| g_t^y-sg( g_t^y)\|_2^2d ]. (14) Use λ=1λ=1 in Equation 5. Deployment writes true costates after each scored target. Language-model loss. For the final MLP, let ut∈ℝdu_t ^d be the residual input and E∈ℝV×dE ^V× d the tied embedding/output matrix for vocabulary size V. With root mean square normalization (RMSNorm), the token loss is ℓt(y)=−log[softmax(ERMSNorm(ut+y))]st+1. _t(y)=- [softmax\! (ERMSNorm(u_t+y) ) ]_s_t+1. (15) Differentiate this loss through the head, normalization, and residual to obtain gtyg_t^y, before batch or sequence averaging. The Gaussian error linear unit (GELU) derivative and adapted transpose then give gtzg_t^z by Equation 13. Appendix B Synthetic MLP protocol This section specifies the numerical diagnostic in Section 5.1. For seeds 17, 42, and 123, initialize the slow matrices with independent centered Gaussian entries of standard deviation 0.5/40.5/ 4 for W1,0W_1,0 and 0.5/70.5/ 7 for W2,0W_2,0. Independent vectors vα,vμ∈ℝ4v_α,v_μ ^4 have (0,0.252)N(0,0.25^2) entries. Set αt=sigmoid(xt⊤vα+1.5) _t=sigmoid(x_t v_α+1.5), μt=sigmoid(xt⊤vμ−0.2) _t=sigmoid(x_t v_μ-0.2), and matrix step sizes μt/4 _t/4 and μt/7 _t/7. Corrections start at zero; both matrices update simultaneously with centered decay. Independent serial autograd supplies the exact costates to a single parallel reconstruction using explicit causal sums in FP64. No serial hidden states are reused. The control changes only the backward transpose, so its output error is identical to the full construction. Errors in Figure 2 are maxima over coordinates, 32 token positions, and three seeds. This is a numerical consistency check, with no learned Q or refinement loop. Appendix C Forward and transpose scan reads This section makes explicit how the forward and backward reuse the same historical writes. For current query vectors at∈ℝda_t ^d and bt∈ℝmb_t ^m, Equation 2 gives Δ^tat _ta_t =−∑i<tηi(∏j=i+1t−1αj)g^i(xi⊤at), =- _i<t _i ( _j=i+1^t-1 _j ) g_i(x_i a_t), (16) Δ^t⊤bt _t b_t =−∑i<tηi(∏j=i+1t−1αj)xi(g^i⊤bt). =- _i<t _i ( _j=i+1^t-1 _j )x_i( g_i b_t). The forward queries with at=xta_t=x_t; the backward queries with bt=g~tb_t= g_t. Historical values remain the proposals g^i g_i in both cases. Each read excludes the current write and all writes before the most recent document or context reset. Appendix D What approximate consistency guarantees This section distinguishes exact recovery from stability to prediction errors. Theorem 1 requires no contraction, but causality alone cannot bound trajectory error by a small consistency residual. For example, consider two scalar token blocks and the causal target map F(G0,G1)=(0,κG0)F(G_0,G_1)=(0,κ G_0), where κ∈ℝκ . Its unique fixed point is G⋆=(0,0)G =(0,0). For any ε∈ℝ , the proposal G^=(ε,κε) G=( ,κ ) satisfies ‖G^−F(G^)‖2=|ε|,‖G^−G⋆‖2=1+κ2|ε|.\| G-F( G)\|_2=| |, \| G-G \|_2= 1+κ^2\,| |. Thus an arbitrarily small residual can coexist with a large trajectory error when amplification is large. This example does not assert that the trained model is unstable; it explains why exact consistency alone does not establish accuracy at nonzero residuals. Appendix E Language-model training details This section records the architecture, training recipe, and numerical checks for the 300M experiments. Architecture and Q conditioning. The backbone uses RMSNorm (Zhang and Sennrich, 2019), rotary positions, tanh-approximate GELU, a tied embedding/output matrix E∈ℝ32768×dE ^32768× d, and context length 1024. The hidden and MLP widths are (d,m)=(1024,4096)(d,m)=(1024,4096). Let xt∈ℝdx_t ^d be the normalized input to the final MLP. Q receives sg(xt)+RMSNorm(sg(Est+1))sg(x_t)+RMSNorm(sg(E_s_t+1)), using unit scale and stabilizer 10−610^-6 for the target normalization. Its two causal Transformer blocks, final normalization, and two linear heads produce vectors atz∈ℝma_t^z ^m and aty∈ℝda_t^y ^d. The proposals are g^tz=σ′(sg(W1,0xt))⊙atz,g^ty=aty. g_t^z=σ \! (sg(W_1,0x_t) ) a_t^z, g_t^y=a_t^y. (17) Compute W1,0xtW_1,0x_t once and reuse it in P. Q is causal and document-isolated; target-conditioned proposals affect only later predictions. Initialization and gradients. For fixed retention, a learned vector w∈ℝdw ^d and scalar bias b define the write strength and the two MLP step sizes: μt=4sigmoid(w⊤sg(xt)+b),η1,t=μt/d,η2,t=μt/m. _t=4sigmoid\! (w sg(x_t)+b ), _1,t= _t/d, _2,t= _t/m. (18) The gate starts at μt=0.9 _t=0.9 with w=0w=0 and b=log(0.9/3.1)b= (0.9/3.1). Retention is one. Zero-initialized Q heads make the first parallel forward static; native deployment already updates. The derivative-conditioning factor above is detached. Historical write inputs, scan costates, Q inputs, and reverse targets are detached; read and gate derivatives remain. Table 2: 300M final-MLP parameter and token budgets. Counts exclude the tied embedding/output matrix. Training counts include Q and determine the reported token-to-parameter ratios. Model Deployed (M) Training (M) Tokens (B) Tokens / param. Static 264.2852 264.2852 29.4702 111.5092×111.5092× Chunk-T (4) 264.2862 264.2862 29.4702 111.5088×111.5088× GradLev 264.2862 294.7000 29.4702 100.0008×100.0008× Optimization and packing. Each update averages 262,144 scored tokens: 16 graphics processing units (GPUs), two 1024-token sequences per GPU, and eight accumulation steps. Adam with decoupled weight decay (AdamW) uses peak rate 3×10−43× 10^-4, coefficients (0.9,0.95)(0.9,0.95), epsilon 10−810^-8, matrix decay 0.1, zero vector decay, and joint gradient-norm clipping at one. After 128 warmup steps, cosine decay reaches 10% of the peak at step 112,420. GradLev uses half-rate Q and consistency weight λ=1λ=1. The Q multiplier applies to its optimizer update and AdamW decay; optimizer decay is separate from centered test-time decay. Pack documents from rank-strided parquet row groups; reset fast state at document and packed-row boundaries. Hold out the final sorted shard. Chunk comparator. A preliminary short screen selected four fixed 256-token chunks, unit retention, and the capped write gate initialized to 0.9. The gate uses the stopped mean MLP input of the current final-document fragment. At each boundary, both matrices receive the fragment’s summed gradients, normalized by matrix input width and evaluated at the same incoming weights. Document resets clear memory, so only the final document can write into the next chunk. Costates and write inputs are detached; gates and later reads retain outer derivatives. Only GradLev uses a prefiller and consistency objective. Numerical implementation. Flash Linear Attention (FLA) provides the training and recurrent scans (Yang and Zhang, 2024; Yang et al., 2024a). Writes are shifted to implement exclusive reads and masked at document boundaries. Autocasting uses the 16-bit brain floating-point format (BF16), with 32-bit floating-point (FP32) parameters and fast states. The two MLP states require 2md2md FP32 entries per sequence, or 32 mebibytes (MiB) at this scale. Curves use GB300 evaluations; final scores use B300 replays. Their endpoint NLLs differ by at most 1.02×10−41.02× 10^-4, so the two sources are reported separately. Prefiller-rate selection. The screen compares Q/PQ/P learning-rate ratios 1/41/4, 1/21/2, and 3/43/4 using mean deployment NLL at steps 1024 and 2048. Their means are 3.571559, 3.571861, and 3.581692. The prespecified rule selects half-rate Q when it is within 0.001 of the best score. All prefixes use the full 112,420-step schedule; only the selected setting continues. Discarded prefixes are additional tuning compute, and no full-budget unit-rate Q control is available. Appendix F Downstream evaluation This section defines the deployment metrics and evaluation protocol for Table 1. Table 3: 300M final-MLP results after 29.47B training tokens. All models use the same B300 evaluation runtime. Metrics retain their original precision; bold marks the best value in each column. Likelihood (↓ ) Zero-shot accuracy (%) (↑ ) Model Valid. NLL FineWeb BPB LAMB. PPL LAMB. PIQA Hella. Wino. ARC-E ARC-C SIQA BoolQ Mean Static 2.573312 0.752708 24.661 37.69 72.52 47.84 53.99 65.49 33.45 39.87 55.35 50.77 Chunk-T (4) 2.572551 0.752980 24.538 37.40 72.09 47.73 53.91 65.19 31.83 38.18 60.18 50.81 GradLev 2.570695 0.751658 23.564 38.04 72.14 47.75 53.28 65.53 32.00 39.41 59.82 50.99 NLL uses the primary validation set; FineWeb-Edu BPB and LAMBADA word PPL use separate corpora. LAMBADA (LAMB.); Physical Interaction: Question Answering (PIQA); HellaSwag (Hella.); WinoGrande (Wino.); AI2 Reasoning Challenge (ARC), Easy (-E) and Challenge (-C); SocialIQA (SIQA). Mean averages eight task accuracies (27,072 examples per model). HellaSwag and ARC-C use length-normalized scores. Experimental setup. The suite follows Maglev’s task selection (Liu and Liu, 2026): LAMBADA (Paperno et al., 2016), PIQA (Bisk et al., 2020), HellaSwag (Zellers et al., 2019), WinoGrande (Sakaguchi et al., 2020), ARC-Easy and ARC-Challenge (Clark et al., 2018), SocialIQA (Sap et al., 2019), and BoolQ (Clark et al., 2019). The mean equally weights eight task accuracies. Each example starts with zero fast state and a fresh cache. Score each token before updating. Candidates use independent prompt-state copies; gold labels are consulted only after scoring. Choice scores sum continuation log-probabilities, normalized by token count for HellaSwag and ARC-Challenge. WinoGrande uses cloze completion. LAMBADA requires greedy agreement with every reference final-word token, without checking the following word boundary. Chunk boundaries remain at 256 tokens. Many benchmark requests end before an update affects scoring, limiting evidence about repeated chunk adaptation; 1024-token NLL evaluations cross actual boundaries. Results and uncertainty. Task-level outcomes in Table 3 are mixed. Paired 95% intervals for GradLev’s mean-accuracy difference are [−0.454,0.808][-0.454,0.808] percentage points against chunk-T and [−0.421,0.862][-0.421,0.862] against static. Both include zero and omit training-seed variability. Validation and FineWeb-Edu scores were used during development, so neither provides fresh confirmation. 300M likelihood metrics and uncertainty. Primary NLL averages 65,536 validation tokens. FineWeb-Edu uses 512 documents, 563,229 scored tokens, and 2,764,147 original bytes in 8-bit Unicode Transformation Format (UTF-8); bits per byte divides total negative log-likelihood by the byte count times log2 2. Non-overlapping blocks contain at most 1024 targets, start with fresh states and a context-only beginning-of-sequence (BOS) token, and score every text token once. Full-document exact and normalized training matches are excluded; partial or semantic overlap may remain. LAMBADA word perplexity exponentiates the mean final-word loss over 5153 examples, summing subword losses before averaging. Accuracy intervals use 10,000 paired bootstrap draws (seed 20260917), resampling examples within each task and equally averaging paired task differences. Limits are the 2.5th and 97.5th percentiles. They exclude training-seed and recipe-selection uncertainty.