Paper deep dive
The Surprising Effectiveness of Shared Memory in Looped Transformers
Giovanni Monea, Keshav Ramji, Yousef El-Kurdi, Luis A. Lastras, Yoav Artzi, Nathan Godey, Ramón Fernandez Astudillo
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 10/5/2026, 5:54:43 AM
Summary
This paper introduces the Looped Prediction Transformer (LPT) and its hybrid variant (H-LPT), which utilize memory sharing in looped Transformer architectures. By having only the first recursion write a full context key-value (KV) cache and subsequent recursions attend to this shared cache plus a small local window, the models achieve significant memory savings (76-79% less context memory) while improving quality (lower perplexity and higher accuracy) compared to standard looped Transformers and parameter-matched standard Transformers.
Entities (7)
Relation Signals (5)
Looped Prediction Transformer → trainson → FineWeb-Edu
confidence 95% · We pretrain on FineWeb-Edu... Applied to the full-attention layers... lpt...
Looped Prediction Transformer → usestechnique → Memory Sharing
confidence 95% · We study Memory Sharing... The resulting Looped Prediction Transformer (lpt)...
Hybrid Looped Prediction Transformer → usestechnique → Memory Sharing
confidence 95% · Applied to the full-attention layers of a hybrid looped Transformer... the same rule yields the Hybrid Looped Prediction Transformer (h-lpt).
Memory Sharing → reduces → KV cache
confidence 90% · Sharing caches across recursions during decoding can save substantial memory... lpt uses... much smaller caches.
Hybrid Looped Prediction Transformer → outperforms → Gated DeltaNet
confidence 85% · h-lpt outperforms lt2 in terms of quality at every recursion count... lt2... using Gated DeltaNet layers
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Looped Transformers apply the same layers several times per token, adding compute to improve quality without more parameters. Each recursion, however, writes its own key-value cache, so memory still grows with compute. Inference-time techniques can shrink this cache at a cost in quality. We pretrain looped language models to share memory: only the first recursion writes a cache, and later recursions read it while keeping a short window of their own. Surprisingly, we find that sharing memory does not cost quality and instead improves it. At 150M-1B parameters, our Looped Prediction Transformer (LPT) and its hybrid variant set a new quality-memory frontier for looped models: with five recursions, the hybrid lowers validation perplexity on FineWeb-Edu by 1.12-1.82 relative to a same-size standard Transformer while using 76-79% less context memory. Through an extensive analysis, we investigate why memory sharing helps. Shared and local memory develop different representations, and later recursions attend mostly to the shared memory, which also acts as a gradient highway to the first recursion.
Tags
Links
- Source: https://arxiv.org/abs/2610.02383v1
- Canonical: https://arxiv.org/abs/2610.02383v1
Trouble viewing inline? Open PDF directly →
Full Text
72,900 characters extracted from source content.
Expand or collapse full text
The Surprising Effectiveness of Shared Memory in Looped Transformers Giovanni Monea Keshav Ramji Affiliation: IBM Research Yousef El-Kurdi Affiliation: IBM Research Luis A. Lastras Affiliation: IBM Research Yoav Artzi Nathan Godey Ramón Fernandez Astudillo Affiliation: Cornell University Abstract Looped Transformers apply the same layers several times per token, adding compute to improve quality without more parameters. Each recursion, however, writes its own key-value cache, so memory still grows with compute. Inference-time techniques can shrink this cache at a cost in quality. We pretrain looped language models to share memory: only the first recursion writes a cache, and later recursions read it while keeping a short window of their own. Surprisingly, we find that sharing memory does not cost quality and instead improves it. At 150M–1B parameters, our Looped Prediction Transformer (lpt) and its hybrid variant set a new quality–memory frontier for looped models: with five recursions, the hybrid lowers validation perplexity on FineWeb-Edu by 1.121.12–1.821.82 relative to a same-size standard Transformer while using 7676–79%79\% less context memory. Through an extensive analysis, we investigate why memory sharing helps. Shared and local memory develop different representations, and later recursions attend mostly to the shared memory, which also acts as a gradient highway to the first recursion. 11footnotetext: Work done during an internship at IBM Research. Correspondence to: giovanni@cs.cornell.edu22footnotetext: Equal advising Figure 1: Shared memory not only saves memory, it also improves quality. Left: in our Looped Prediction Transformer (lpt), later recursions read the KV cache written by the first recursion and keep only a short window of their own. Right: perplexity and context memory of 11B-parameter models with two to five recursions, against a parameter-matched standard Transformer (std). Each of our models adds shared memory to the looped baseline above it: lpt to the looped Transformer lt, and h-lpt to the gated DeltaNet hybrid lt2. 1 Introduction Looped Transformers (lt) repeatedly apply the same layers several times before predicting the next token, adding computation without adding parameters (Dehghani et al., 2019; Zhu et al., 2025). In principle, they trade time for memory, matching the compute of a larger model with far fewer parameters. In standard formulations, however, context memory still grows with recursion depth, as each iteration keeps a separate key-value (KV) cache (Geiping et al., 2025; Zhu et al., 2025). At long contexts, this context memory can grow larger than the weights memory itself, undoing the very reason to use a looped model. Sharing caches across recursions during decoding can save substantial memory with little quality loss (Geiping et al., 2025; Zhu et al., 2025). So far, this reuse has been studied mainly as a way to save memory. Yet it also shows that memory written by one recursion can serve the others, even in models never trained to share it. This raises a question: if sharing can work untrained, what can it do when models are trained for it? To answer it, we train looped models with memory sharing from the start, investigating it as an architectural inductive bias. We study a simple recipe for Memory Sharing in looped Transformers (Figure 1, left). At each full-attention layer, later recursions attend to both their own KV entries and the cache written by that layer in the first recursion, within the same attention operation. To further save memory, we truncate the cache of every recursion after the first to a short sliding window. The resulting Looped Prediction Transformer (lpt) keeps lt’s sequential recurrence and uses context memory close to a standard (non-looped) Transformer’s. Applied to the full-attention layers of a hybrid looped Transformer (Deng et al., 2026), the same rule yields the Hybrid Looped Prediction Transformer (h-lpt). We pretrain all models from scratch at 150150M to 11B parameters. Because later recursions give up most of their own memory, one would expect Memory Sharing to cost some quality, as untrained sharing does. Surprisingly, Memory Sharing improves quality over full-cache lt even when each later recursion’s own cache covers less than 2%2\% of the context. Its advantage grows with depth: lt stops improving in perplexity after three recursions, while lpt improves through five. Across sizes, lpt lowers validation perplexity by up to 1.091.09 nats and raises downstream accuracy by up to 4.84.8 percentage points over a matched-depth lt, with up to 4.7×4.7× less context memory and nearly the same inference FLOPs. Sharing also improves the hybrid baseline: at five recursions, h-lpt outperforms a same-size standard Transformer in both language modeling and downstream accuracy while using 7676–79%79\% less context memory, moving the quality–memory frontier for looped models (Figure 2). These results raise a further question: why does memory sharing improve quality? We find that later recursions rely heavily on the shared memory. lpt outperforms lt at every window size, even when later recursions keep no cache of their own, and later recursions attend mostly to shared memory even when their own cache covers the full context. Shared and local memory also develop distinct key representations, suggesting that they serve different purposes. Reading shared memory sends gradients directly to the first recursion, helping it learn what to store. This gradient highway bypasses the intermediate recursions: in lt, the magnitude of the gradient reaching the first recursion falls sharply with depth, while lpt recovers much of it through the shared memory. When we block this path, the model still reads shared memory but no longer matches lpt’s quality and stability. 2 Looped Transformers A standard lt shares its weights across recursions, while each recursion computes its own hidden states and KV entries. We use Ouro’s full-stack formulation (Zhu et al., 2025), which builds on earlier recurrent-depth architectures (Dehghani et al., 2019; Geiping et al., 2025). Let ℳL=θ(L)∘⋯∘θ(1)M^L=T_θ^(L) ·s _θ^(1) denote the shared stack of L Transformer blocks. For a sequence of length T and hidden dimension d, let hin(t)∈ℝT×dh_in^(t) ^T× d and h(t)h^(t) be the residual streams entering and leaving recursion t. The stack is applied R times: hin(1) h_in^(1) =emb(), =emb(x), (1) h(t) h^(t) =ℳL(hin(t)), =M^L\! (h_in^(t) ), hin(t+1) h_in^(t+1) =rec(h(t)), =N_rec\! (h^(t) ), logits(t) ^(t) =lm_head(out(h(t))),t=1,…,R. =lm\_head\! (N_out(h^(t)) ), t=1,…,R. Here recN_rec and outN_out are separately parameterized RMSNorms. Both act on h(t)h^(t): the recurrent norm prepares the next recursion’s input, while the readout norm feeds the language-modeling head. The effective depth is L⋅RL· R, with the parameters of a single L-layer stack. Each layer computes its keys and values from the hidden states it receives at the current recursion. For a fixed layer, let (t)C^(t) denote the cache written by recursion t, with entry (kj(t),vj(t))(k_j^(t),v_j^(t)) at position j. We suppress the layer index throughout: for vanilla LT, each layer reads only its own entries. A query at position i reads the causal prefix of its recursion’s cache, giving the attention set lt(t)(i)=≤i(t)=(kj(t),vj(t)):j≤i.A_lt^(t)(i)=C_≤ i^(t)=\(k_j^(t),v_j^(t)):j≤ i\. (2) At inference, each layer therefore retains R full-context caches, one per recursion, and context memory grows linearly with recursion depth even though the parameter count stays fixed. Some of these caches can be dropped or merged during decoding (Geiping et al., 2025; Zhu et al., 2025), but the model is still trained with a separate cache for every recursion. Hybrid architectures reduce this memory cost by replacing most full-attention layers with linear attention, whose recurrent state does not grow with context length (Katharopoulos et al., 2020; Schlag et al., 2021; Yang et al., 2025). lt2 (Deng et al., 2026) loops such a stack, using Gated DeltaNet layers and one full-attention layer per five. This reduces the number of full-attention caches, but each remaining layer still needs a separate full-context cache at every recursion. 3 Looping with Shared Memory We study Memory Sharing, in which later recursions read the memory written by the first recursion, as an inductive bias for looped models. The first recursion runs as in lt and writes a full-context cache (1)C^(1) at each layer, and every later recursion attends to this cache together with its own KV entries, using a single softmax. To also keep looping memory-efficient, we restrict each later recursion’s own cache to a short window. The first recursion therefore acts as a memory phase that builds the model’s only full-context memory, while later recursions use it to refine the prediction. We refer to the first recursion as the state recursion and to the later ones as prediction recursions, and call the resulting model the Looped Prediction Transformer (lpt; Figure 1). Like lt, lpt applies the same L-layer stack R times and passes the final output to the language-modeling head (Equation 1). Let ω be the number of preceding positions kept in each prediction recursion’s window, and write [i−ω,i](t)C_[i-ω,i]^(t) for the entries from positions i−ωi-ω through i, including the current entry. The state recursion reads its own cache, as in lt (Equation 2). A prediction recursion reads the shared memory together with its own window. The entries read by a query at position i in recursion t are lpt(t)(i)=≤i(1),t=1,≤i(1)∪[i−ω,i](t),t=2,…,R.A_lpt^(t)(i)= casesC_≤ i^(1),&t=1,\\ C_≤ i^(1) _[i-ω,i]^(t),&t=2,…,R. cases (3) In prediction recursions, the two sets are attended to using a single softmax. Setting ω=Tω=T gives every prediction recursion a full-context cache of its own, as in lt, and leaves access to shared memory as the only difference between the two models; we use this setting in Section 6 to separate the effect of sharing from that of the short window. Memory Sharing also applies to looped hybrid Transformers, which interleave linear-attention and full-attention layers (Yang et al., 2025; Deng et al., 2026). The first recursion runs the hybrid as usual. In later recursions, each linear-attention layer keeps its own recurrent state, which is small and fixed in size, while each full-attention layer applies Equation 3, as in lpt. Applied to lt2 (Deng et al., 2026), this gives h-lpt. Efficiency and Costs In training, we implement Equation 3 as a FlexAttention (Dong et al., 2025) block mask over the concatenated shared and local entries, which skips the fully masked blocks, so each query attends over the shared prefix and its ω window entries only (Appendix A). During decoding, each prediction recursion keeps a sliding buffer of at most ω preceding entries per layer; its current entry participates in attention before entering the buffer. In cached token positions across L full-attention layers, the inference-time KV cache sizes are MKVlt=LRT,MKVlpt=L[T+(R−1)ω],M_KV^lt=L\,R\,T, M_KV^lpt=L [T+(R-1)ω ], (4) where the common head dimensions and datatype size are omitted. lpt uses the same parameters and recursion count as a matched-depth lt, but performs slightly more attention because each prediction recursion also reads its local window. At a full context of T=4,096T=4,096 with ω=64ω=64 and five recursions, these windows increase the number of attended KV entries per generated token by only (R−1)ω/(RT)=1.25%(R-1)ω/(RT)=1.25\%, while reducing the full-attention cache relative to lt by 4.7×4.7×. Table 1: lpt and h-lpt reach lower perplexity than their looped baselines with much smaller caches. Main results across S, M, and L: validation perplexity, mean five-task accuracy (%), and context memory and FLOPs relative to same-size std. In the std rows, R=1R=1 is the standard Transformer, and R>1R>1 is deepstd. Elsewhere, R counts model passes. Bold and underline mark the best and second-best within each size and column group; our rows are shaded. We replicate the S results with three additional seeds and find the gains of Memory Sharing statistically significant (p<0.01p<0.01, Appendix B). R=1R=1 R=2R=2 R=3R=3 R=5R=5 Size Model PPL ↓ Acc. ↑ PPL ↓ Acc. ↑ KV ↓ FLOPs ↓ PPL ↓ Acc. ↑ KV ↓ FLOPs ↓ PPL ↓ Acc. ↑ KV ↓ FLOPs ↓ std 16.8316.83 51.551.5 17.0717.07 50.250.2 1.501.50 1.19 17.2717.27 50.750.7 1.881.88 1.31 17.6917.69 50.450.4 2.502.50 1.54 loopie 16.5816.58 51.651.6 2.002.00 1.831.83 16.5016.50 51.951.9 3.003.00 2.662.66 16.5616.56 52.152.1 5.005.00 4.324.32 lt 16.1716.17 51.551.5 2.002.00 1.831.83 15.9815.98 52.152.1 3.003.00 2.662.66 16.0016.00 52.452.4 5.005.00 4.324.32 plt 15.8415.84 52.7 1.021.02 1.841.84 15.5515.55 52.952.9 1.031.03 2.672.67 15.4915.49 53.553.5 1.061.06 4.344.34 lt2 15.76 51.951.9 0.35 1.29 15.4715.47 52.552.5 0.52 1.86 15.1615.16 53.753.7 0.87 2.98 lpt 15.73 51.951.9 1.021.02 1.841.84 15.28 54.0 1.031.03 2.672.67 14.91 54.9 1.061.06 4.344.34 S h-lpt 15.73 52.4 0.18 1.29 15.39 53.1 0.19 1.86 15.01 54.6 0.21 2.98 std 13.7913.79 56.156.1 14.2414.24 55.555.5 1.441.44 1.13 14.4714.47 55.455.4 1.781.78 1.24 14.8814.88 54.454.4 2.342.34 1.43 loopie 13.6913.69 56.956.9 2.002.00 1.921.92 13.7513.75 55.855.8 3.003.00 2.842.84 13.7413.74 56.656.6 5.005.00 4.694.69 lt 13.2813.28 57.557.5 2.002.00 1.921.92 13.1313.13 59.459.4 3.003.00 2.842.84 13.3413.34 57.957.9 5.005.00 4.694.69 plt 13.0013.00 59.1 1.021.02 1.931.93 12.6812.68 60.060.0 1.031.03 2.852.85 12.6512.65 60.560.5 1.061.06 4.714.71 lt2 12.81 59.1 0.35 1.43 12.6312.63 59.559.5 0.52 2.10 12.4312.43 60.260.2 0.87 3.45 lpt 12.9112.91 59.2 1.021.02 1.931.93 12.55 60.2 1.031.03 2.852.85 12.33 61.6 1.061.06 4.704.70 M h-lpt 12.80 58.558.5 0.18 1.43 12.54 60.3 0.19 2.10 12.31 61.4 0.21 3.45 std 11.3511.35 62.562.5 11.6211.62 62.562.5 1.451.45 1.14 11.9611.96 61.361.3 1.731.73 1.17 12.2712.27 61.461.4 2.252.25 1.32 loopie 11.2611.26 63.563.5 2.002.00 1.951.95 11.2811.28 63.963.9 3.003.00 2.912.91 12.2612.26 61.061.0 5.005.00 4.814.81 lt 11.2011.20 64.264.2 2.002.00 1.951.95 11.1111.11 63.863.8 3.003.00 2.912.91 11.3911.39 63.463.4 5.005.00 4.814.81 plt 10.9510.95 64.864.8 1.021.02 1.961.96 10.7510.75 66.866.8 1.031.03 2.922.92 10.5910.59 66.466.4 1.061.06 4.834.83 lt2 10.59 66.5 0.40 1.53 10.4710.47 65.865.8 0.60 2.27 10.32 67.367.3 1.01 3.75 lpt 10.7510.75 66.3 1.021.02 1.961.96 10.45 67.3 1.031.03 2.922.92 10.32 68.2 1.061.06 4.834.83 L h-lpt 10.58 65.665.6 0.21 1.53 10.41 66.9 0.22 2.27 10.23 68.1 0.24 3.75 4 Experimental Setup Baselines To study memory sharing, we compare lpt with lt and h-lpt with lt2. We also compare with the Parallel Loop Transformer (plt) (Wu et al., 2025), which parallelizes recursions and uses gated dual attention over the first recursion’s shared cache and each later recursion’s local cache. Other baselines are a standard Transformer (std), loopie, which repeats each layer before advancing to the next (Gao et al., 2026), and deepstd, a deeper, narrower Transformer without weight sharing that matches the looped models in effective depth and approximately in parameters. Architecture All models use a decoder-only Transformer backbone with RMSNorm, SwiGLU, rotary positional embeddings, and tied input and output embeddings. We train S, M, and L models with approximately 150150M, 450450M, and 11B parameters; the main comparisons cover all three sizes, and most ablations and analyses use S and M. The context length is 4,0964,096. Unless otherwise specified, lpt and h-lpt use a prediction window of ω=64ω=64. We study two, three, and five recursions. This range includes the two loop steps of loopie (Gao et al., 2026) and surrounds the four recursions of Ouro and lt2 (Zhu et al., 2025; Deng et al., 2026).11 1 Nine recursions bring instability in several model families and limited accuracy gains, consistent with the loss spikes Ouro reports at eight (Zhu et al., 2025). We report these experiments in Appendix E. All models use nanoGPT’s default optimizer settings (Karpathy, 2022). Model dimensions, normalization, and training details are in Appendix A. Data and Training We pretrain on FineWeb-Edu (Penedo et al., 2024) using 2020B tokens for S and M, and 5050B for L, roughly 22–7×7× the Chinchilla compute-optimal token budgets (Hoffmann et al., 2022). The main tables report one run per configuration, with the same initialization seed and data order across models. At S, we also train three more seeds per model and confirm that seed variation is far below the gaps we report (Appendix B). We study two training objectives. Final loss computes the next-token NLL at the final recursion only. Adaptive loss follows Ouro (Zhu et al., 2025), weighting per-recursion NLLs with a learned per-token halting distribution and adding KL regularization. Unless otherwise specified, lt uses adaptive loss and all other models use final loss. Evaluation We report perplexity on 1010M held-out FineWeb-Edu tokens and mean zero-shot accuracy on ARC-Easy (Clark et al., 2018), HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), SciQ (Welbl et al., 2017), and LAMBADA (Paperno et al., 2016), using the LM Evaluation Harness (Gao et al., 2023). We compute context memory and decoding FLOPs per generated token analytically at a 4,0964,096-token context, relative to the same size std. Context memory includes the KV cache and GDN’s recurrent state. Figure 2: lpt and h-lpt reach higher accuracy with much smaller caches than the looped baselines. Mean five-task accuracy vs. context memory at each size. Curves follow two, three, and five recursions, with larger markers at five. Dashed lines mark single-pass std. 5 Results Memory Sharing Improves Quality Restricting the memory written by later recursions could hurt prediction. Surprisingly, our Memory Sharing approach improves quality. At matched size and recursion count, lpt lowers validation perplexity by 0.380.38–1.091.09 and raises accuracy by 0.40.4–4.84.8 percentage points over lt, with 2.02.0–4.7×4.7× less KV memory and under 1%1\% more FLOPs (Table 1). Measured on an H100, the memory savings are the same, so lpt fits 2.02.0–4.8×4.8× as many sequences in a batch as lt (Appendix F). The perplexity gain grows with recursion count (Table 1): lt stops improving beyond three recursions at every size, while lpt continues to improve through five. At comparable cache sizes, lpt also reaches lower perplexity than plt’s parallel formulation. When it comes to hybrid variants, h-lpt keeps this memory at 0.180.18–0.24×0.24× std’s across recursion counts, while lt2’s remaining full-attention caches still grow with depth, approaching std’s context memory at five recursions. Despite these 22–4×4× memory gains, h-lpt matches or outperforms lt2 in terms of quality at every recursion count. At five recursions, it lowers perplexity by 0.090.09–0.150.15 and raises accuracy by 0.80.8–1.21.2 points over lt2, with 4.14.1–4.2×4.2× less context memory and about 0.1%0.1\% more inference FLOPs. Incidentally, we note that for low recursion counts, the hybrid lt2 improves on lt while using less memory and compute. A factor that could explain this gap – and the instability of lt – is Ouro’s normalization, which we use for both lt and lpt. In Appendix C, we explore other normalizations that make lt more robust, but none of these variants matches lpt. More Quality from the Same Parameters Both memory-sharing models improve on std with little additional memory, or much less for h-lpt (Figure 2). At five recursions, lpt lowers perplexity by 1.031.03–1.921.92 and raises accuracy by 3.43.4–5.75.7 points with 6.25%6.25\% more context memory. h-lpt lowers perplexity by 1.121.12–1.821.82 and raises accuracy by 3.13.1–5.65.6 points with 7676–79%79\% less memory. This smaller cache lets h-lpt fit 3.63.6–5.0×5.0× as many sequences as std on one GPU. With three to five recursions, h-lpt lowers perplexity by 0.940.94–1.821.82 while beating or staying close to std’s throughput (1.41.4–1.5×1.5× at three, 8181–88%88\% at five, Appendix F). Both models’ accuracy gains grow from two to five recursions at every size. lpt also outperforms loopie and deepstd, neither of which improves consistently with depth. Takeaway 1. Through Memory Sharing, lpt and especially h-lpt improves the quality–memory frontier for looped models (Figure 2). 6 Why Memory Sharing Helps We use S- and M-size models with two, three, and five recursions to study why this inductive bias improves quality (Appendix D). Memory Sharing Stabilizes Deep Recursion We compare final and adaptive loss in lt and lpt (Figure 4). Without intermediate supervision, adding recursions consistently worsens lt’s perplexity, and the model collapses to unigram-level loss at five recursions. lpt stays stable, and final loss lowers its perplexity by 0.160.16–0.500.50 compared with adaptive loss. Its intermediate recursions can refine the hidden state without having to predict early. Takeaway 2. lpt stays stable and trains best with final loss. Figure 3: lpt trains best with final loss, while lt consistently worsens with more recursions without adaptive loss. Validation perplexity (↓ ), collapsed runs in the upper strip on a log scale. Filled markers are final loss, hollow ones adaptive loss. Figure 4: A short prediction window is enough. Validation perplexity of lpt relative to lt (↓ ) vs. window size, lt at its better objective per depth. The dotted line marks ω=64ω=64. Prediction Recursion Needs Local Memory We train lpt with prediction windows ranging from ω=0ω=0 to the full context of 4,0964,096 tokens (Figure 4). lpt outperforms lt at every window size, across both sizes and all tested depths. This includes ω=0ω=0, where prediction recursions keep no local history and read shared memory alongside only their current KV entry. At the other extreme, ω=4,096ω=4,096, lt and lpt differ only in access to shared memory. lpt still lowers perplexity by 0.240.24–1.151.15 relative to lt with the better of its two training objectives at each depth, close to the 0.320.32–1.091.09 gain at ω=64ω=64. Overall, window size has a modest effect on quality, and what really matters is the access to the shared memory. To balance quality and efficiency, we adopt ω=64ω=64 as the default for our memory sharing models. Takeaway 3. Shared memory improves quality across prediction-window sizes, even when later recursions keep no local history. Later Recursions Attend More to Shared Memory We next measure how much attention is assigned to the shared memory in later recursions. We evaluate each model on the same 4,0964,096-token validation sequence and average the attention assigned to shared keys over queries and heads (Figure 6 and Appendix D). Shared memory receives most attention at every prediction recursion in both sizes. Its share is 5757–71%71\% in earlier prediction recursions and rises to 7373–80%80\% in the final recursion.22 2 Additional comparisons in the appendix show that shared memory still receives most attention with full prediction caches, while adaptive loss makes its use more uniform across recursions (Figure 11). Takeaway 4. Later recursions attend more to shared memory than to their own caches, with a stronger preference when getting closer to the prediction. Figure 5: Prediction recursions attend mostly to shared memory. Attention mass assigned to shared KV entries in lpt, averaged over queries and heads. For clarity, only six layers are displayed. The bottom row averages all layers. Figure 6: Shared and local keys separate in lpt. Joint query/key PCA, fitted separately for each panel. Dots are tokens and large circles their means; darker shades mark later recursions. Shared and Local Memory Develop Different Representations Shared keys serve every later recursion, whereas local keys serve only the recursion that writes them. To see whether these roles are reflected in their representations, we compare S-size lt and lpt with three recursions, both trained with final loss, the only tested setting with at least three recursions where lt trains with final loss without a substantial drop in quality (Figure 4). We evaluate both models on the same 4,0964,096-token validation sequence and extract queries and keys before rotary encoding. We concatenate the attention heads and fit principal component analysis (PCA) jointly to queries and keys across tokens and recursions per model and layer (Appendix D). Shared and local keys consistently form distinct clusters in lpt, whereas keys from different recursions overlap more in lt (Figure 6). In Appendix D, we show similar separation in lpt across S and M at three and five recursions (Figure 10). We also show that with full prediction caches, the clusters overlap more but still remain distinguishable, even though shared and local keys cover the same positions (Figure 9). Prior work attributes the small cost of cache reuse to the similarity of keys and values across recursions (Bae et al., 2025), as we see in lt. With Memory Sharing, shared and local keys are instead clearly different. This shows that representation similarity across layers is not a necessary condition for effective memory sharing. This also fits the freedom that final loss gives: recursions need not all produce prediction-ready states, so the first can specialize in building memory and the later ones in using it. This specialization may help explain the quality gains, similar to how Monea et al. (2026) show that Transformers benefit from assigning memory construction and next-token prediction to separate residual streams. Takeaway 5. Shared and local memory diverge in their local representations, so sharing is not reliant on recursions writing similar memories. Shared Memory Is a Gradient Highway Reading shared memory gives the final loss a direct gradient path to the first recursion, bypassing the intermediate recursions. We evaluate the std, lt, and lpt models of Table 1 on the same 3232 validation sequences of 4,0964,096 tokens and backpropagate the final loss from 3232 fixed positions in each. We measure each recursion’s contribution to the gradient on the Transformer block weights, and split the first recursion’s contribution into its hidden-state and shared-KV paths by repeating the backward pass with shared KV detached (Appendix D). In lt, the first-recursion gradient norm falls sharply with depth. lpt recovers much of the signal magnitude relative to the single-pass std reference, with a much smaller decline as depth increases (Figure 8). At five recursions, most of this gradient comes through the shared KV cache. The same holds for lpt with adaptive loss or full prediction caches (Figure 12). To test whether this highway helps learning, we train lpt from scratch with shared KV detached at later reads. The forward computation and hidden-state gradient path are unchanged. Perplexity rises in both sizes and at every depth, and deeper detached models often fail to train (Figure 8). Figure 7: Shared memory restores much of the gradient reaching the first recursion. Mean gradient norm contributed by each recursion, with single-pass std as reference. The first lpt bar is split into shared-KV (hatched) and hidden-state projections. lt is trained with adaptive loss, the others with final loss. Whiskers are 95%95\% sequence-bootstrap intervals. Figure 8: Detaching shared-cache gradients hurts learning. Validation perplexity (↓ ), collapsed runs in the upper strip on a log scale. All models are trained with final loss; Table 6 lists the values. Takeaway 6. The memory sharing mechanism creates a gradient highway from the final loss to the first recursion that facilitates learning. 7 Related Work Separating Memory from Prediction In a standard Transformer, each hidden state does two jobs. It predicts the next token, and it writes the keys and values that later positions read. Several architectures separate these jobs across token streams, as in the two-stream attention of XLNet (Yang et al., 2019). The State-Prediction Separation Transformer (Monea et al., 2026) gives memory construction and next-token prediction separate residual streams and improves language modeling, and Langford et al. (2026) study a variant of this split. A second family separates them across layers. The first layers write one cache once, and the remaining layers only read it (Sun et al., 2024; Sun et al., 2026; DeepSeek-AI, 2026; Capps, 2026). Instead, lpt decouples memory from prediction across the recursions of a looped Transformer: the first recursion writes the model’s only full-context memory, and every later recursion reads it while it predicts, keeping just a short window of its own. Looped models thus offer a third place for the split, with no extra stream or layer, and our analysis finds evidence that the recursions learn these roles (Section 6). Looped Transformers and Memory Sharing Looped Transformers apply the same layers several times (Dehghani et al., 2019; Geiping et al., 2025; Zhu et al., 2025), and recent work differs in which layers repeat and how recursions connect (Gao et al., 2026; Wang et al., 2026a; Huang et al., 2026; Mozer et al., 2026; Wang et al., 2026b; Deng et al., 2026). In standard formulations, each recursion writes its own KV cache, so context memory grows with depth (Geiping et al., 2025; Zhu et al., 2025). Sharing a cache across recursions, like sharing it across layers (Brandon et al., 2024), has been studied as a way to save memory. Without retraining, Huginn (Geiping et al., 2025) reuses caches cyclically with little quality loss, and Ouro (Zhu et al., 2025) can keep only the final or an averaged cache. Keeping only the first recursion’s cache sharply reduces Ouro’s quality, which suggests that shared memory has to be trained. Trained variants reuse the first recursion’s keys and values at a slight cost in quality (Bae et al., 2025), compute them once in a separate prelude (Capps, 2026), or learn gated or compressed caches (Vendrell et al., 2026; O’Neill and Reid, 2026). plt (Wu et al., 2025) has the closest design to ours, although it focuses on parallelizing recursions. It shares the first recursion’s cache with gated local caches and runs recursions in parallel, while lpt keeps them sequential and reads both caches in one softmax. We find that lpt performs better. All of these treat sharing as a cost to contain. We instead treat it as an inductive bias that separates memory from prediction and show that it improves quality (Section 5 and Section 6). 8 Discussion We investigate memory sharing as an architectural inductive bias for looped models, and find that it not only cuts context memory but also improves quality in both full attention and hybrid models. Our Memory Sharing lpt models achieve consistently better perplexity and downstream accuracy than their usual looped counterparts, while matching the KV cache memory of a standard non-looped model. Our analyses suggest that sharing helps recursions learn different roles. The first recursion builds memory that later recursions use to refine the prediction. Those reads also give the final loss a direct gradient path to the first recursion. The different representations of shared and local memory support this division of roles, and cutting this gradient path hurts learning. Shared memory changes how later recursions read information and how the first recursion learns to write it. Overall, our findings suggest that shared memory is a useful inductive bias for looped models, helping them make better use of computation within a limited memory budget. Limitations & Future work Like other looped Transformers, lpt is not compute-optimal: it trades time for quality, but unlike lt, its memory barely grows with depth. At five recursions it costs 4.34.3–4.8×4.8× the decoding FLOPs of std, about the same as lt, and it spends the same number of recursions on every token. Choosing that number per token is future work. All our comparisons are parameter-matched, including deepstd, and we do not train a standard Transformer matched in decoding FLOPs instead, which at five recursions would have several times the parameters. We train up to 11B parameters on one corpus with 4,0964,096-token contexts and do not test long-context tasks or, given our limited pretraining, very advanced tasks. Our analyses use S and M models, since not every configuration they need was trained at L, and only the detach ablation looks at training dynamics. Both the stability and memory improvements of lpt are promising for effectively scaling looped models to higher recursion counts. More sophisticated Memory Sharing approaches could also be explored in future work, such as increasing the number of state recursions or alternating state recursions and prediction recursions. Acknowledgments We thank Aviv Ron, Brian Kingsbury, and David Cox for useful feedback on internal presentations of this work. References Bae et al. (2025) S. Bae, Y. Kim, R. Bayat, S. Kim, J. Ha, T. Schuster, A. Fisch, H. Harutyunyan, Z. Ji, A. Courville, and S. Yun Mixture-of-recursions: learning dynamic recursive depths for adaptive token-level computation. In Advances in Neural Information Processing Systems, External Links: Link Cited by: §6, §7. Bisk et al. (2020) Y. Bisk, R. Zellers, R. Le Bras, J. Gao, and Y. Choi PIQA: reasoning about physical commonsense in natural language. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34, p. 7432–7439. External Links: Link Cited by: §4. Brandon et al. (2024) W. Brandon, M. Mishra, A. Nrusimha, R. Panda, and J. Ragan-Kelley Reducing transformer key-value cache size with cross-layer attention. In Advances in Neural Information Processing Systems, External Links: Link Cited by: Appendix F, Appendix F, §7. Capps (2026) C. A. Capps CART: context-anchored recurrent transformer – a parameter-efficient architecture with learned stability. External Links: 2606.01495, Link Cited by: §7, §7. Chen and Wei (2026) C. Chen and L. Wei Post-LayerNorm is back: stable, ExpressivE, and deep. External Links: 2601.19895, Link Cited by: 5th item. Clark et al. (2018) P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord Think you have solved question answering? Try ARC, the AI2 Reasoning Challenge. External Links: 1803.05457, Link Cited by: §4. DeepSeek-AI (2026) DeepSeek-AI DeepSeek-V4.1-Flash: pushing the limits of KV cache compression. External Links: 2609.19969, Link Cited by: §7. Dehghani et al. (2019) M. Dehghani, S. Gouws, O. Vinyals, J. Uszkoreit, and Ł. Kaiser Universal Transformers. In International Conference on Learning Representations, External Links: Link Cited by: §1, §2, §7. Deng et al. (2026) C. Deng, Y. Zhang, R. Zhu, Y. Xu, J. Liu, T. S. E. Ng, and H. Chen LT2: linear-time looped transformers. External Links: 2605.20670, Link Cited by: Appendix A, Appendix E, §1, §2, §3, §4, §7. Ding et al. (2021) M. Ding, Z. Yang, W. Hong, W. Zheng, C. Zhou, D. Yin, J. Lin, X. Zou, Z. Shao, H. Yang, and J. Tang CogView: mastering text-to-image generation via transformers. In Advances in Neural Information Processing Systems, External Links: Link Cited by: 2nd item. Dong et al. (2025) J. Dong, B. Feng, D. Guessous, Y. Liang, and H. He FlexAttention: a programming model for generating fused attention variants. In Proceedings of Machine Learning and Systems, External Links: Link Cited by: Appendix A, Appendix F, §3. Gao et al. (2023) L. Gao, J. Tow, B. Abbasi, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, A. Le Noac’h, H. Li, K. McDonell, N. Muennighoff, C. Ociepa, J. Phang, L. Reynolds, H. Schoelkopf, A. Skowron, L. Sutawika, E. Tang, A. Thite, B. Wang, K. Wang, and A. Zou A framework for few-shot language model evaluation. Zenodo. Note: Zenodo, version v0.4.0 External Links: Document, Link Cited by: §4. Gao et al. (2026) Z. Gao, Y. Chen, Y. Xiao, X. Yang, R. Tao, J. Zhou, and B. Dai Loop the Loopies!. External Links: 2607.16051, Link Cited by: §4, §4, §7. Geiping et al. (2025) J. Geiping, S. McLeish, N. Jain, J. Kirchenbauer, S. Singh, B. R. Bartoldson, B. Kailkhura, A. Bhatele, and T. Goldstein Scaling up test-time compute with latent reasoning: a recurrent depth approach. In Advances in Neural Information Processing Systems, External Links: Link Cited by: §1, §1, §2, §2, §7. Hoffmann et al. (2022) J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, O. Vinyals, J. W. Rae, and L. Sifre Training compute-optimal large language models. In Advances in Neural Information Processing Systems, Vol. 35, p. 30016–30030. External Links: Link Cited by: §4. Huang et al. (2026) B. Huang, C. Shi, J. Chen, S. Wen, Z. Liu, E. Xing, and X. Ma Towards looped models done right—Part I: topology, input injection, recurrent-state design. Note: Institute of Foundation Models technical blogAccessed September 16, 2026 External Links: Link Cited by: §7. Karpathy (2022) A. Karpathy nanoGPT. Note: GitHub repository, commit 325be85 External Links: Link Cited by: §4. Katharopoulos et al. (2020) A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret Transformers are RNNs: fast autoregressive transformers with linear attention. In International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 119, p. 5156–5165. External Links: Link Cited by: §2. Langford et al. (2026) J. Langford, N. Godey, G. Monea, Y. Artzi, H. Dong, Y. Fan, G. de Rosa, and Z. Zhan Almost free state prediction separation. External Links: 2609.03807, Link Cited by: §7. Monea et al. (2026) G. Monea, N. Godey, K. Brantley, and Y. Artzi The state-prediction separation hypothesis. In Advances in Neural Information Processing Systems, External Links: Link Cited by: Appendix E, §6, §7. Mozer et al. (2026) M. C. Mozer, S. A. Siddiqui, D. Sawyer, S. Sanyal, and R. Liu Recirculation. External Links: 2608.17981, Link Cited by: §7. O’Neill and Reid (2026) J. O’Neill and F. Reid Looped latent attention: cross-loop KV compression for looped transformers. External Links: 2607.15456, Link Cited by: Appendix F, §7. Paperno et al. (2016) D. Paperno, G. Kruszewski, A. Lazaridou, Q. N. Pham, R. Bernardi, S. Pezzelle, M. Baroni, G. Boleda, and R. Fernández The LAMBADA dataset: word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 1525–1534. External Links: Link Cited by: §4. Penedo et al. (2024) G. Penedo, H. Kydlíček, L. Ben Allal, A. Lozhkov, M. Mitchell, C. Raffel, L. von Werra, and T. Wolf The FineWeb datasets: decanting the web for the finest text data at scale. In Advances in Neural Information Processing Systems (Datasets and Benchmarks Track), Vol. 37, p. 30811–30849. External Links: Link Cited by: §4. Radford et al. (2019) A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever Language models are unsupervised multitask learners. Technical report OpenAI. Cited by: Appendix A. Schlag et al. (2021) I. Schlag, K. Irie, and J. Schmidhuber Linear transformers are secretly fast weight programmers. In International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 139, p. 9355–9366. External Links: Link Cited by: §2. Shazeer (2020) N. Shazeer GLU variants improve Transformer. External Links: 2002.05202, Link Cited by: Appendix A. Su et al. (2024) J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu RoFormer: enhanced transformer with rotary position embedding. Neurocomputing 568, p. 127063. External Links: Document, Link Cited by: Appendix A. Sun et al. (2026) Y. Sun, L. Dong, T. Ye, S. Huang, J. Wang, and F. Wei Universal YOCO for efficient depth scaling. External Links: 2604.01220, Link Cited by: §7. Sun et al. (2024) Y. Sun, L. Dong, Y. Zhu, S. Huang, W. Wang, S. Ma, Q. Zhang, J. Wang, and F. Wei You only cache once: decoder-decoder architectures for language models. In Advances in Neural Information Processing Systems, External Links: Link Cited by: Appendix F, §7. Vendrell et al. (2026) V. C. Vendrell, A. P. Masdemont, N. Grillo, J. Ros-Giralt, A. Behboodi, and F. V. Massoli Memory-efficient looped transformer: decoupling compute from memory in looped language models. External Links: 2605.07721, Link Cited by: §7. Wang et al. (2026a) S. Wang, G. Zhang, K. Luo, Y. Wu, S. Liu, J. Liu, W. Huang, S. Yan, and J. Li SMELT: scaling laws for compute-matched MoE looped transformers. External Links: 2609.01343, Link Cited by: §7. Wang et al. (2026b) X. Wang, Z. Cai, Z. Zhan, H. Dong, Y. Fan, G. de Rosa, T. Pearce, and J. Langford Full-bandwidth transformer. External Links: 2608.08888, Link Cited by: §7. Welbl et al. (2017) J. Welbl, N. F. Liu, and M. Gardner Crowdsourcing multiple choice science questions. In Proceedings of the 3rd Workshop on Noisy User-generated Text, p. 94–106. External Links: Link Cited by: §4. Wu et al. (2025) B. Wu, M. Chen, X. Luo, S. Yan, Q. Yu, F. Xia, T. Zhang, H. Zhan, Z. Zhong, X. Zhou, S. Qiao, and X. Bin Parallel loop transformer for efficient test-time computation scaling. External Links: 2510.24824, Link Cited by: §4, §7. Xiong et al. (2020) R. Xiong, Y. Yang, D. He, K. Zheng, S. Zheng, C. Xing, H. Zhang, Y. Lan, L. Wang, and T. Liu On layer normalization in the transformer architecture. In International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 119, p. 10524–10533. External Links: Link Cited by: 1st item. Yang et al. (2025) S. Yang, J. Kautz, and A. Hatamizadeh Gated delta networks: improving Mamba2 with delta rule. In International Conference on Learning Representations, p. 29687–29707. External Links: Link Cited by: §2, §3. Yang and Zhang (2024) S. Yang and Y. Zhang FLA: a Triton-based library for hardware-efficient implementations of linear attention mechanism. Note: GitHub repository External Links: Link Cited by: Appendix F. Yang et al. (2019) Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. Salakhutdinov, and Q. V. Le XLNet: generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems, External Links: Link Cited by: §7. Zellers et al. (2019) R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi HellaSwag: can a machine really finish your sentence?. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, p. 4791–4800. External Links: Link Cited by: §4. Zhang and Sennrich (2019) B. Zhang and R. Sennrich Root mean square layer normalization. In Advances in Neural Information Processing Systems, Vol. 32. External Links: Link Cited by: Appendix A. Zhu et al. (2025) R. Zhu, Z. Wang, K. Hua, T. Zhang, Z. Li, H. Que, B. Wei, Z. Wen, F. Yin, H. Xing, L. Li, J. Shi, K. Ma, S. Li, T. Kergan, A. Smith, X. Qu, M. Hui, B. Wu, Q. Min, H. Huang, X. Zhou, W. Ye, J. Liu, J. Yang, Y. Shi, C. Lin, E. Zhao, T. Cai, G. Zhang, W. Huang, Y. Bengio, and J. Eshraghian Scaling latent reasoning via looped language models. External Links: 2510.25741, Link Cited by: Appendix A, Appendix A, Appendix C, Appendix E, §1, §1, §2, §2, §4, §4, §7, footnote 1. Appendix A Architecture and Training Details Architecture Table 2 lists the base model dimensions. The decoder-only backbone uses RMSNorm pre-normalization (Zhang and Sennrich, 2019), SwiGLU feed-forward networks (Shazeer, 2020), no biases in linear layers, and tied input and output embeddings. Full-attention layers use rotary positional embeddings (Su et al., 2024). Table 2: Base model configurations. Looped models reuse the listed L-layer stack. S M L Layers (L) 12 24 36 Hidden size (d) 768 1,024 1,280 Heads 12 16 20 FFN size 3,072 4,096 5,120 Parameters 151.9M 454.2M 1.01B lt and lpt use Ouro’s normalization and recurrent transition (Zhu et al., 2025). lt2 and h-lpt use the Full+GDN layer pattern and gating setup of Deng et al. (2026), with one full-attention layer per five layers. The full-attention layers read the complete causal context in lt2. In h-lpt, later recursions read the first recursion’s full-context cache and their own local prediction window, as in Equation 3. plt uses the same block, recurrent, and readout normalization as lt and lpt. Its recurrent transition adds the input embedding to the previous recursion’s normalized output shifted by one token. Attention Implementation During training, each prediction recursion concatenates its keys and values with the shared keys and values along the sequence dimension, separately at each full-attention layer. A block mask implements Equation 3, allowing full causal attention over shared entries and causal sliding-window attention over local entries. FlexAttention (Dong et al., 2025) skips fully masked blocks. Data Processing We tokenize with the GPT-2 tokenizer (Radford et al., 2019) and pack documents into 4,0964,096-token sequences, inserting an end-of-sequence token between documents. Attention is masked across document boundaries, and boundary tokens are excluded from the language-modeling loss. Optimization We use AdamW with β1=0.9 _1=0.9, β2=0.95 _2=0.95, weight decay 0.10.1, and gradient clipping at 1.01.0. The global batch contains 9696 sequences of 4,0964,096 tokens, and training uses bfloat16 mixed precision. For L-size models, AdamW optimizer state is sharded across DDP ranks (ZeRO-1). The learning rate increases linearly to 6×10−46× 10^-4 over the first 5050M tokens, stays constant, then decreases linearly to 6×10−56× 10^-5 over the final 10%10\% of training. These are nanoGPT’s default optimizer settings, used unchanged for every model, size, and objective: we did not tune the learning rate or the warmup for any model, and we did not test whether a lower peak learning rate or a longer warmup changes how final-loss lt trains at depth (Figure 4). For adaptive loss, we use Ouro’s adaptive-exit objective (Zhu et al., 2025). A learned per-token halting distribution weights the losses across recursions, with a KL regularization weight of 0.10.1. Final loss applies the language-modeling loss only at the last recursion. Appendix B Seed Variance and Statistical Tests Retraining every configuration at every size with several seeds is prohibitive at pretraining cost, so we run the check at S. Each S-size model of Table 1 is trained with four seeds: the run reported in the table and three more, each changing the initialization and the data order together. All other settings stay at their main-table values. Table 3 lists the mean and 95%95\% confidence interval (Student-t) of the final validation perplexity for every model and recursion count. Table 4 tests each gap of interest with a one-sided paired t-test on the same-seed differences, with the alternative that the first model has lower perplexity. The largest standard deviation of perplexity across seeds is 0.080.08, against gaps of 0.400.40–1.051.05 between lpt and lt, and every difference keeps its sign at every seed. Table 3: Seed variance at S. Final validation perplexity, mean ± 95%95\% confidence interval (Student-t) over four seeds. In the std row, R=1R=1 is the standard Transformer and R>1R>1 the deeper, narrower parameter-matched Transformer. sps is the two-recursion baseline of Appendix E. Model R=1R=1 R=2R=2 R=3R=3 R=5R=5 std 16.84±0.0316.84± 0.03 17.03±0.0517.03± 0.05 17.27±0.0717.27± 0.07 17.64±0.1317.64± 0.13 sps 16.12±0.0716.12± 0.07 loopie 16.57±0.0316.57± 0.03 16.57±0.0816.57± 0.08 16.58±0.0316.58± 0.03 lt 16.16±0.0316.16± 0.03 16.00±0.0516.00± 0.05 15.99±0.0315.99± 0.03 plt 15.83±0.0515.83± 0.05 15.55±0.0315.55± 0.03 15.46±0.0315.46± 0.03 lt2 15.77±0.0215.77± 0.02 15.47±0.0215.47± 0.02 15.16±0.0115.16± 0.01 lpt 15.76±0.0715.76± 0.07 15.29±0.0415.29± 0.04 14.94±0.0414.94± 0.04 h-lpt 15.73±0.0215.73± 0.02 15.40±0.0215.40± 0.02 15.03±0.0215.03± 0.02 Table 4: Paired tests at S. Same-seed perplexity gap of the first model over the second, mean ± 95%95\% confidence interval over four seeds, the gap in units of the standard deviation of the same-seed differences, and the one-sided paired t-test for the first model having lower perplexity. Comparison R Gap (PPL) Gap//SD p lpt vs. lt 2 −0.40±0.05-0.40± 0.05 1313 6.5×10−56.5× 10^-5 3 −0.71±0.05-0.71± 0.05 2424 9.4×10−69.4× 10^-6 5 −1.05±0.04-1.05± 0.04 3939 2.4×10−62.4× 10^-6 h-lpt vs. lt2 2 −0.04±0.01-0.04± 0.01 44 1.7×10−31.7× 10^-3 3 −0.07±0.02-0.07± 0.02 55 9.8×10−49.8× 10^-4 5 −0.13±0.02-0.13± 0.02 99 1.6×10−41.6× 10^-4 lpt vs. plt 2 −0.07±0.09-0.07± 0.09 11 4.8×10−24.8× 10^-2 3 −0.27±0.07-0.27± 0.07 66 5.4×10−45.4× 10^-4 5 −0.52±0.07-0.52± 0.07 1111 9.7×10−59.7× 10^-5 Appendix C Normalization between Recursions Our lt and lpt use the Ouro paper’s normalization (Zhu et al., 2025). Sandwich norms normalize the output of every sublayer before the residual addition, and the recurrent norm recN_rec normalizes each recursion’s output before the next recursion (Equation 1). With final loss, this lt degrades or diverges at depth (Figure 4), so the main tables train it with adaptive loss. We did not tune the normalization of either model. To test whether Ouro’s normalization holds lt back, we also train S and M lt with final loss and five other normalizations, each shared across recursions: • Pre-norm only (Xiong et al., 2020) removes both, so hin(t+1)=h(t)h_in^(t+1)=h^(t) and each block keeps only its pre-norms. • Sandwich only keeps the sandwich norms (Ding et al., 2021) and removes recN_rec. • Input norm removes the sandwich norms and keeps recN_rec, which it also applies to the embeddings. • Update norm removes the sandwich norms and normalizes only each recursion’s update, so the carried state stays unnormalized: hin(t+1)=hin(t)+(h(t)−hin(t))h_in^(t+1)=h_in^(t)+N (h^(t)-h_in^(t) ). • KEEL removes the sandwich norms and applies the highway residual of Chen and Wei (2026) across recursions: hin(t+1)=2(Rhin(t)+ℳL(1(hin(t))))h_in^(t+1)=N_2 (R\,h_in^(t)+M^L(N_1(h_in^(t))) ). Here N, 1N_1, and 2N_2 are RMSNorms. Data, seed, batch, and learning-rate schedule match the main runs. lpt keeps Ouro’s normalization and final loss. Table 5: lpt outperforms lt with the best normalization we tried. Validation perplexity (↓ ) after 2020B tokens. Bold marks the best per column and underline the best lt. Values above 2,0002,000 are unigram-level. A dash marks a run that did not finish. S M Model Normalization Loss R=2R=2 R=3R=3 R=5R=5 R=2R=2 R=3R=3 R=5R=5 lt Ouro (main) Adaptive 16.17 15.98 16.00 13.28 13.13 13.34 Ouro Final 16.05 16.32 2,040 17.10 31.74 2,090 Pre-norm only Final 16.28 15.99 15.77 13.28 13.05 12.83 Sandwich only Final 16.23 15.83 15.67 – – – Input norm Final 16.10 15.79 15.49 13.05 12.88 13.02 Update norm Final 16.04 15.65 15.35 13.01 12.75 12.77 KEEL Final 16.10 15.73 15.24 12.96 12.78 12.76 lpt Ouro Final 15.73 15.28 14.91 12.91 12.55 12.33 At every size and depth in Table 5, lpt with Ouro’s normalization reaches lower perplexity than the best lt, and the best lt reaches lower perplexity than our main lt. The alternatives are also more robust: every finished run trains with final loss at every size and depth, while Ouro’s normalization degrades or diverges. No single normalization is best for lt: update norm wins three settings and KEEL the other three. Choosing the best one lowers lt’s perplexity by 0.130.13–0.760.76 relative to our main lt. lpt still leads the best lt by 0.310.31–0.370.37 at S and by 0.060.06–0.430.43 at M. The smallest gap, 0.060.06 at M with two recursions, is within the seed variation we measure at S (Appendix B). At S, every alternative keeps improving through five recursions, unlike our main lt. At M, the best lt stops improving after three recursions. Although this comparison favors lt, lpt still performs best. lt gets the best of six normalizations, chosen on the same validation perplexity we report, while lpt keeps Ouro’s. The same search for lpt would include Ouro’s normalization, so it could only keep or lower lpt’s perplexity. Since normalization is not the focus of our investigation, we keep Ouro’s for the main lt, as in the original paper. Appendix D Additional Analysis Details Attention Representations For Figure 6, we use S-size models with three recursions and final loss. We evaluate each model on the same 4,0964,096-token validation sequence. At each layer, we concatenate the attention heads into one query vector and one key vector per token, before rotary encoding. We pool these vectors across all tokens and recursions, subtract their common mean, and fit PCA by singular value decomposition, separately for each model and layer. We do not normalize the vectors or rescale their features. All recursions within a panel therefore share one basis. Axes are not aligned across models or layers. We display the first two principal components. For readability, the panels show one token in 2424, while PCA and the large centroid markers use all tokens. Lines connect centroids across recursions, separately for queries and keys. Axis limits are shared across layers within each model, with equal scaling on both axes. Figure 9adds lpt with full prediction caches. Figure 10 compares lpt across sizes and recursion counts, matching layers by relative depth. Figure 11 compares attention to shared memory across window sizes and training objectives. Each layer’s attention mass is averaged over queries and heads. We display six layers at matched relative depths, but compute the mean over all 1212 layers for S and all 2424 for M. Figure 9: Shared and local keys remain distinguishable with full prediction caches. Expanded comparison for Figure 6, including ω=4,096ω=4,096. Figure 10: Query/key geometry across lpt models. S and M models at three and five recursions, with final loss and ω=64ω=64. Columns match relative layer depth. Figure 11: Attention to shared memory across training objectives and window sizes. Expanded comparison for Figure 6. Mean rows average all layers. Gradient Feedback through Shared Memory We differentiate the mean cross-entropy of the final recursion at 3232 sampled positions in each of 3232 validation sequences of length 4,0964,096. The sequence contents and loss positions are identical across conditions. Figure 8 uses the main-comparison checkpoints: lt trained with adaptive loss and lpt with final loss. std provides a single-pass reference, measured through its native forward pass on the same data. Figure 12 adds lpt with full prediction caches and lpt trained with adaptive loss. Every probe differentiates the loss at the final recursion, regardless of the training objective. We measure gradients on Transformer block parameters, excluding embeddings, readout, recurrent normalization, and the halting gate. Each model retains its trained normalization. For the backward computation, we give each recursion an independent parameter copy with the same values. This leaves the forward pass unchanged and isolates each recursion’s contribution to the shared-parameter gradient. Detaching shared KV at later reads isolates the first recursion’s hidden-state path contribution, ghiddeng_hidden. Subtracting it from the ordinary first-recursion gradient g(1)g^(1) gives the shared-cache contribution, gKVg_KV. We verify that contributions from all parameter uses sum to the ordinary shared-parameter gradient. The bars show arithmetic mean gradient norms across sequences. To split the first bar, we project each path onto its total gradient within each sequence: pa=⟨ga,g(1)⟩∥g(1)∥2,a∈KV,hidden,pKV+phidden=∥g(1)∥2.p_a= g_a,g^(1) g^(1) _2, a∈\KV,hidden\, p_KV+p_hidden= g^(1) _2. (5) We then average these projected contributions. Their sum equals the mean first-recursion norm. The two path norms themselves are not additive. The projections are nonnegative in all plotted measurements, without clipping. Whiskers are 95%95\% sequence-bootstrap intervals for the full bar, not for its segments or for variation across training seeds. Figure 12shows that the shared-cache contribution remains strong with a full prediction window and under adaptive loss. This is consistent across S and M. The compact main-paper comparison therefore does not depend on using final loss or a short prediction window for lpt. Figure 12: Shared-cache feedback persists across training objectives and window sizes. Expanded version of Figure 8, with the same sequences and final-loss gradient probe in every condition. Gradient Paths at Initialization Figure 13 compares the norm ratio ∥gKV∥2/∥ghidden∥2 g_KV _2/ g_hidden _2 before and after training, using geometric means over the same 3232 sequences. Fresh initialization is measured at two, three, and five recursions for both windows and sizes. It uses separate random weights, not saved initial checkpoints. The ratio increases with depth even at initialization, but is larger in the trained models. More recursions add more shared-cache reads, so this trend alone does not establish that the feedback helps learning. The detachment experiment tests that separately. Figure 13: Shared-KV feedback relative to the hidden-state path. Gradient-norm ratios at fresh initialization and after training. Whiskers: 95%95\% sequence-bootstrap intervals. Shared-Cache Gradient Ablation We train S and M lpt models with R∈2,3,5R∈\2,3,5\, ω=64ω=64, and final loss, changing only whether the shared KV entries are detached at each read by a later recursion. The first recursion still receives gradients through the hidden-state path, and all shared parameters remain trainable. This removes direct gradient feedback through shared-cache reads, not all training of the first recursion. Table 6compares final-loss lt, detached lpt, and lpt after 2020B training tokens. Detachment raises perplexity in every comparison. Detached S at five recursions finishes well above lpt, while M at three and five recursions remains near unigram-level loss. The experiment supports a role for shared-cache gradients in learning and stability, but does not distinguish shorter gradient paths from changes in gradient scale or direction. Table 6: Detaching shared-cache gradients hurts learning. Validation perplexity (↓ ) at S and M, all models trained with final loss; values above 1,0001,000 are runs that failed to train. Bold marks the best per column. S M Model R=2R=2 R=3R=3 R=5R=5 R=2R=2 R=3R=3 R=5R=5 lt (final loss) 16.05 16.32 2,040 17.10 31.74 2,090 lpt (detached KV) 16.34 16.14 33.21 13.87 1,995 1,527 lpt 15.73 15.28 14.91 12.91 12.55 12.33 Appendix E Experiments at Nine Recursions Table 7and Figure 14 extend the comparisons to nine recursions. We also list sps (Monea et al., 2026) in the R=2R=2 column, since its memory and FLOPs match those of lpt with two recursions. From five to nine, lpt gains 0.70.7 accuracy points at S and 0.20.2 at M, but loses 1.71.7 at L. Perplexity improves at S and worsens at M and L. lt and plt also worsen in perplexity at every size. Both hybrids continue to improve: h-lpt gains 0.40.4, 0.20.2, and 0.60.6 accuracy points at S, M, and L, respectively, and reaches the lowest perplexity of all models at every size. At nine recursions, lpt still outperforms lt at every size, and h-lpt outperforms lt2 in both perplexity and accuracy. Higher recurrent depth can also complicate training. Ouro reports loss spikes and gradient oscillations at eight recursions and reduces its recurrent depth to four (Zhu et al., 2025). lt2 also uses four recursions (Deng et al., 2026). Table 7: Quality and efficiency through nine recursions. One table per size. Metrics and formatting follow Table 1: in the std row, R=1R=1 is the standard Transformer and R>1R>1 is deepstd. sps appears in the R=2R=2 column, whose memory and FLOPs it matches. S R=1R=1 R=2R=2 R=3R=3 R=5R=5 R=9R=9 Model PPL ↓ Acc. ↑ PPL ↓ Acc. ↑ KV ↓ FLOPs ↓ PPL ↓ Acc. ↑ KV ↓ FLOPs ↓ PPL ↓ Acc. ↑ KV ↓ FLOPs ↓ PPL ↓ Acc. ↑ KV ↓ FLOPs ↓ std 16.8316.83 51.551.5 17.0717.07 50.250.2 1.501.50 1.19 17.2717.27 50.750.7 1.881.88 1.31 17.6917.69 50.450.4 2.502.50 1.54 18.3218.32 49.149.1 3.383.38 1.81 sps 16.1816.18 52.6 1.021.02 1.841.84 loopie 16.5816.58 51.651.6 2.002.00 1.831.83 16.5016.50 51.951.9 3.003.00 2.662.66 16.5616.56 52.152.1 5.005.00 4.324.32 16.9816.98 51.051.0 9.009.00 7.647.64 lt 16.1716.17 51.551.5 2.002.00 1.831.83 15.9815.98 52.152.1 3.003.00 2.662.66 16.0016.00 52.452.4 5.005.00 4.324.32 16.1316.13 52.252.2 9.009.00 7.647.64 plt 15.8415.84 52.7 1.021.02 1.841.84 15.5515.55 52.952.9 1.031.03 2.672.67 15.4915.49 53.553.5 1.061.06 4.344.34 15.5115.51 53.253.2 1.12 7.687.68 lt2 15.76 51.951.9 0.35 1.29 15.4715.47 52.552.5 0.52 1.86 15.1615.16 53.753.7 0.87 2.98 14.9614.96 54.454.4 1.561.56 5.23 lpt 15.73 51.951.9 1.021.02 1.841.84 15.28 54.0 1.031.03 2.672.67 14.91 54.9 1.061.06 4.344.34 14.80 55.6 1.12 7.687.68 h-lpt 15.73 52.452.4 0.18 1.29 15.39 53.1 0.19 1.86 15.01 54.6 0.21 2.98 14.74 55.0 0.25 5.23 M R=1R=1 R=2R=2 R=3R=3 R=5R=5 R=9R=9 Model PPL ↓ Acc. ↑ PPL ↓ Acc. ↑ KV ↓ FLOPs ↓ PPL ↓ Acc. ↑ KV ↓ FLOPs ↓ PPL ↓ Acc. ↑ KV ↓ FLOPs ↓ PPL ↓ Acc. ↑ KV ↓ FLOPs ↓ std 13.7913.79 56.156.1 14.2414.24 55.555.5 1.441.44 1.13 14.4714.47 55.455.4 1.781.78 1.24 14.8814.88 54.454.4 2.342.34 1.43 15.6015.60 53.453.4 3.093.09 1.63 sps 12.9912.99 58.958.9 1.021.02 1.931.93 loopie 13.6913.69 56.956.9 2.002.00 1.921.92 13.7513.75 55.855.8 3.003.00 2.842.84 13.7413.74 56.656.6 5.005.00 4.694.69 15.8015.80 52.952.9 9.009.00 8.378.37 lt 13.2813.28 57.557.5 2.002.00 1.921.92 13.1313.13 59.459.4 3.003.00 2.842.84 13.3413.34 57.957.9 5.005.00 4.694.69 13.7213.72 56.956.9 9.009.00 8.378.37 plt 13.0013.00 59.1 1.021.02 1.931.93 12.6812.68 60.060.0 1.031.03 2.852.85 12.6512.65 60.560.5 1.061.06 4.714.71 12.8712.87 59.559.5 1.12 8.418.41 lt2 12.81 59.1 0.35 1.43 12.6312.63 59.559.5 0.52 2.10 12.4312.43 60.260.2 0.87 3.45 12.31 60.460.4 1.561.56 6.14 lpt 12.9112.91 59.2 1.021.02 1.931.93 12.55 60.2 1.031.03 2.852.85 12.33 61.6 1.061.06 4.704.70 12.3912.39 61.8 1.12 8.418.41 h-lpt 12.80 58.558.5 0.18 1.43 12.54 60.3 0.19 2.10 12.31 61.4 0.21 3.45 12.14 61.6 0.25 6.14 L R=1R=1 R=2R=2 R=3R=3 R=5R=5 R=9R=9 Model PPL ↓ Acc. ↑ PPL ↓ Acc. ↑ KV ↓ FLOPs ↓ PPL ↓ Acc. ↑ KV ↓ FLOPs ↓ PPL ↓ Acc. ↑ KV ↓ FLOPs ↓ PPL ↓ Acc. ↑ KV ↓ FLOPs ↓ std 11.3511.35 62.562.5 11.6211.62 62.562.5 1.451.45 1.14 11.9611.96 61.361.3 1.731.73 1.17 12.2712.27 61.461.4 2.252.25 1.32 12.6612.66 59.859.8 3.153.15 1.63 sps 10.7710.77 65.565.5 1.021.02 1.961.96 loopie 11.2611.26 63.563.5 2.002.00 1.951.95 11.2811.28 63.963.9 3.003.00 2.912.91 12.2612.26 61.061.0 5.005.00 4.814.81 11.5311.53 63.263.2 9.009.00 8.638.63 lt 11.2011.20 64.264.2 2.002.00 1.951.95 11.1111.11 63.863.8 3.003.00 2.912.91 11.3911.39 63.463.4 5.005.00 4.814.81 12.5212.52 60.260.2 9.009.00 8.638.63 plt 10.9510.95 64.864.8 1.021.02 1.961.96 10.7510.75 66.866.8 1.031.03 2.922.92 10.5910.59 66.466.4 1.061.06 4.834.83 10.7810.78 66.166.1 1.12 8.668.66 lt2 10.59 66.5 0.40 1.53 10.4710.47 65.865.8 0.60 2.27 10.32 67.367.3 1.01 3.75 10.20 68.1 1.811.81 6.71 lpt 10.7510.75 66.3 1.021.02 1.961.96 10.45 67.3 1.031.03 2.922.92 10.32 68.2 1.061.06 4.834.83 10.8710.87 66.566.5 1.12 8.668.66 h-lpt 10.58 65.665.6 0.21 1.53 10.41 66.9 0.22 2.27 10.23 68.1 0.24 3.75 10.11 68.7 0.28 6.726.72 Figure 14: Quality and memory through nine recursions. Validation perplexity, mean five-task accuracy, and context memory relative to std. Intermediate Readouts We apply the model’s readout normalization and language-modeling head to the output of each recursion. The measurements use S-size lpt checkpoints trained for 2020B tokens, averaged over 1616 validation sequences of 4,0964,096 tokens. With nine recursions and final loss, intermediate-readout perplexity ranges from 8787 to 1,0121,012, falling to 15.615.6 at the last recursion. With adaptive loss, it is already 2.792.79 by the fourth recursion and remains near that value. Under final loss, the head is trained only on the last recursion: high intermediate perplexity measures poor prediction through this head, not the absence of predictive information in the hidden state. These probe-set losses are separate from the full validation-set perplexity in Table 7. Complete Quality–Memory Sweep Figure 15 includes every family, size, and recursion count in Table 7 except sps and plt. Figure 16 compares nine-recursion models with std across sizes. Figure 15: Quality versus context memory across model sizes. Curves vary recursion count. Marker size denotes recursions, and shade denotes model size. S/M train for 2020B tokens and L for 5050B. Figure 16: Quality and memory across model sizes. Mean five-task accuracy and context memory for std and looped models at nine recursions. Appendix F Decoding Speed and Memory on a GPU We test whether the analytic memory savings translate into larger batches and higher decoding throughput on a GPU. A smaller cache lets more sequences share a GPU, which can raise throughput (Brandon et al., 2024; Sun et al., 2024). As O’Neill and Reid (2026) do for a looped model, we measure the largest batch that fits at a fixed context and the throughput at that batch. Setup We decode on one NVIDIA H100 with 8080 GB of memory, in bf16, at a 4,0964,096-token context: each step generates one token per sequence from a cache already filled to 4,0684,068 tokens. All models share one decoding implementation in PyTorch: a preallocated cache per attention layer, FlexAttention (Dong et al., 2025) for attention, and CUDA graphs that replay a whole decoding step, all recursions and layers included. A later recursion of lpt or h-lpt attends to its shared and local caches in one softmax, which we compute by merging the two FlexAttention outputs by their log-sum-exp. Gated DeltaNet layers update their recurrent state one token at a time, with the kernels of flash-linear-attention (Yang and Zhang, 2024). std runs the same code with one recursion, and lt runs all its recursions, without early exit. Weights are random, since speed and memory do not depend on them. On the same GPU, decoding reproduces the full forward pass to bf16 precision. The maximum batch is the largest batch whose decoding steps run without running out of memory, and we measure throughput at this batch, or at 95%95\% of it when the CUDA graph does not fit. These measurements come from one research implementation, not an optimized serving engine, which would change the absolute speeds. Every model runs the same code, so the comparison holds within this implementation. Capacity Table 8 shows that the analytic savings carry over to the GPU. lpt’s cache per sequence is at most 6.25%6.25\% larger than std’s, so it fits 9393–98%98\% of std’s batch. lt’s cache grows with depth, so lpt fits 2.02.0, 2.92.9, and 4.74.7–4.84.8 times as many sequences as lt at two, three, and five recursions. h-lpt holds 2020–27%27\% of std’s cache and fits 3.63.6–5.05.0 times as many sequences as std, and 1.81.8–3.83.8 times as many as lt2. The hybrids’ measured caches are 33–17%17\% larger than the analytic values in Table 1, because the kernel keeps the recurrent state in fp32 and we also count the short-convolution windows. Speed The larger batches raise lpt’s peak throughput above lt’s by 77–91%91\%, most at L with five recursions, where lt fits only 2121 sequences. At equal batch, lpt is 33–8%8\% slower than lt, since each later recursion still reads the whole shared cache, and its own window too. As Brandon et al. (2024) note for sharing across layers, a shared cache saves memory, not the time to read it. Both looped models pay for depth in time: at batch 11, a decoding step of lt takes about 2.32.3, 3.43.4, and 5.75.7 times as long as std’s at two, three, and five recursions, and lpt’s takes 88–12%12\% longer than lt’s. The hybrids read a full-context cache in only one layer of five, so they decode faster. At its maximum batch, h-lpt generates 2.12.1–2.22.2 times as many tokens per second as std with two recursions, 1.41.4–1.51.5 times with three, and 0.80.8–0.90.9 times with five, and it is 22–3%3\% slower than lt2 at equal batch. Throughput and Quality Figure 17 plots quality against throughput. At every size and recursion count, lpt has higher throughput, lower perplexity, and higher accuracy than lt, and h-lpt has higher throughput and lower perplexity than lt2. Against std, h-lpt lowers perplexity by 0.940.94–1.441.44 with three recursions while decoding 1.41.4–1.51.5 times as many tokens per second, and by 1.121.12–1.821.82 with five, at 8181–88%88\% of std’s throughput. Table 8: Decoding capacity and speed on one H100. One table per size. bf16, 4,0964,096-token context. Batch: the largest batch that decodes without running out of the 8080 GB. Tok/s: generated tokens per second at that batch. Lat.: milliseconds per decoding step at batch 11. Our rows are shaded. S R=1R=1 R=2R=2 R=3R=3 R=5R=5 Model Batch Tok/s Lat. Batch Tok/s Lat. Batch Tok/s Lat. Batch Tok/s Lat. std 552552 18,57018,570 2.12.1 lt 275275 8,4148,414 4.84.8 184184 5,1725,172 7.17.1 110110 2,6882,688 11.811.8 lpt 540540 8,9758,975 5.25.2 533533 5,8945,894 7.97.9 514514 3,5443,544 13.313.3 lt2 1,5131,513 39,00639,006 2.62.6 1,0071,007 25,47025,470 3.93.9 603603 14,34814,348 6.46.4 h-lpt 2,7462,746 40,06640,066 2.72.7 2,5312,531 26,69226,692 4.04.0 2,1902,190 15,90015,900 6.76.7 M R=1R=1 R=2R=2 R=3R=3 R=5R=5 Model Batch Tok/s Lat. Batch Tok/s Lat. Batch Tok/s Lat. Batch Tok/s Lat. std 206206 6,4786,478 4.24.2 lt 103103 2,7032,703 9.79.7 6868 1,5271,527 14.514.5 4141 763763 24.124.1 lpt 202202 3,0743,074 10.510.5 199199 2,0002,000 16.016.0 192192 1,2011,201 27.027.0 lt2 563563 13,80413,804 5.35.3 377377 8,5118,511 8.08.0 224224 4,5024,502 13.213.2 h-lpt 1,0291,029 14,54914,549 5.55.5 949949 9,6229,622 8.28.2 821821 5,7065,706 13.613.6 L R=1R=1 R=2R=2 R=3R=3 R=5R=5 Model Batch Tok/s Lat. Batch Tok/s Lat. Batch Tok/s Lat. Batch Tok/s Lat. std 108108 3,1623,162 6.76.7 lt 5454 1,2031,203 15.315.3 3636 691691 22.822.8 2121 303303 37.937.9 lpt 106106 1,4801,480 16.416.4 104104 976976 25.025.0 101101 578578 42.342.3 lt2 258258 5,9865,986 8.88.8 172172 3,5853,585 13.113.1 103103 1,7761,776 21.721.7 h-lpt 475475 6,6126,612 9.09.0 444444 4,3334,333 13.613.6 390390 2,5532,553 22.622.6 Figure 17: h-lpt is better than std at similar or higher throughput. Validation perplexity (top) and mean five-task accuracy (bottom) against generated tokens per second at each model’s maximum batch, relative to same-size std, on one H100 at a 4,0964,096-token context. Curves follow two, three, and five recursions, with larger markers at five. Dashed lines mark std.