Paper deep dive

FreeSolo: Linear-time Sparse Attention with Zero-Shot Length Extrapolation

Zekun Shi, Alastair James Arthur Price

Year: 2026Venue: arXiv preprintArea: cs.LGType: PreprintEmbeddings: 137

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Linear-time sequence models typically store long-context information in a recurrent state. A fixed-size state limits how much information can be stored, while hybrid architectures that add full attention for better recall incur quadratic sequence cost. Alternatively, sparse attention with continued pretraining (CPT) can extend the effective context length. Generalization beyond the trained length remains challenging, however, and CPT can be costly. We introduce FreeSolo, a native sparse attention mechanism with a short-range sliding window branch and a long-range branch that selects keypoint cells for direct token attention. Its linear-time variant, FS-L, learns to prune the prefix before query-dependent selection through short CPT. FS-B searches balanced trees with $O(N\log^2 N)$ algorithmic cost including radix-sort construction. FreeSolo demonstrates better zero-shot length extrapolation than the baselines without long-context fine-tuning.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

134,670 characters extracted from source content.

Expand or collapse full text

FreeSolo: Linear-time Sparse Attention with Zero-Shot Length Extrapolation Zekun Shi Affiliation: National University of Singapore Affiliation: University of Toronto Sea AI Lab, Singapore Email: mailto:zekun.shi@comp.nus.edu.sgzekun.shi@comp.nus.edu.sg Alastair James Arthur Price Abstract Linear-time sequence models typically store long-context information in a recurrent state. A fixed-size state limits how much information can be stored, while hybrid architectures that add full attention for better recall incur quadratic sequence cost. Alternatively, sparse attention with continued pretraining (CPT) can extend the effective context length. Generalization beyond the trained length remains challenging, however, and CPT can be costly. We introduce FreeSolo11 1 We will open-source the code., a native sparse attention mechanism with a short-range sliding window branch and a long-range branch that selects keypoint cells for direct token attention. Its linear-time variant, FS-L, learns to prune the prefix before query-dependent selection through short CPT. FS-B searches balanced trees with O⁡(Nlog2N)O(N ^2N) algorithmic cost including radix-sort construction. FreeSolo demonstrates better zero-shot length extrapolation than the baselines without long-context fine-tuning. Figure 1: Zero-shot RULER extrapolation at 800800M. Models train at 44k (dotted line); shading marks extrapolation. 100100 examples per task and length. 1 Introduction Long-context language modeling requires efficient attention and retrieval beyond the training length. Full attention has quadratic sequence cost [Vaswani et al., 2017]. Recurrent models compress history into fixed-size states [Katharopoulos et al., 2020, Gu and Dao, 2023, Dao and Gu, 2024], while Native Sparse Attention (NSA) [Yuan et al., 2025] and Mixture of Block Attention (MoBA) [Lu et al., 2025] preserve token access but score growing sets of blocks. Neither compression nor sparsity alone ensures length extrapolation. Continued pretraining (CPT) on longer sequences can extend retrieval at additional training cost [Fu et al., 2024]. We introduce FreeSolo (FS), a native sparse mechanism that learns token keys whose simple statistics make memory searchable, prunable, and hierarchically summarizable. Mean keys give a simple chunk representation for routing and hierarchical search, while maximum token-key norms provide pruning priorities. These summaries need only identify evidence: top-K selection is followed by attention over the cells’ original token keys and values. FS scores all eligible cell summaries during training, but only selected token values enter the long-range readout. The read budget stays fixed as history grows. RoPE [Su et al., 2021] stays within a fixed local window, while long-range routing and readout use unrotated queries and keys, avoiding unseen rotary offsets (Figure 2). We develop two main variants through short CPT from base FS at the same 44k context length. FreeSolo-Linear (FS-L) uses norm pruning for O⁡(N)O(N) attention cost and bounded state. FreeSolo-Balanced Tree Search (FS-B) keeps a growing archive and uses two content forests, with O⁡(Nlog2N)O(N ^2N) algorithmic cost including radix-sort construction, superlinear but strictly subquadratic (Section 3.5). Both extrapolate zero-shot to 1616k: at 800800M, the RULER mean decreases only from 32.032.0 to 22.322.3 for FS-L and from 37.737.7 to 30.430.4 for FS-B relative to 44k. Notably, both retrieve single needles at 256256k, 64×64× the training length. At both scales, FS-B achieves the highest RULER mean in the superlinear group at 44k and beyond, and FS-L leads the linear group at every reported length (Table 1). Shared by SR and LR: i→input RMSNormi→q,k,v(i,i,i)x_i input RMSNormz_i W_q,W_k,W_v(q_i,k_i,v_i)Short rangeSliding window, w=512w=512⋯·sWindow moves every token: j≤i,i−j≤wj≤ i, i-j≤ wℓijsr=((i)i)⊤((j)j)d _ij^sr= (R(i)q_i) (R(j)k_j) d(isr)⊤=softmax⁡(ℓi,ℒisr)⊤ℒi(o_i^sr) =softmax( _i,L_i^sr) V_L_iToken sequence: s=32s=32 tokens per cell⋯·sc−8c_-8c−7c_-7c−6c_-6c−5c_-5c−4c_-4c−3c_-3c−2c_-2c−1c_-1c0c_0c1c_1c2c_2c3c_3iq_i00128128Full prefix iC_iAll cells before this query blockQuery block: B=128B=128Prefix boundary moves every query blockFixed until the block endsLong rangeFSFS-LKeep the full prefixPrune per head⋯·s⋯·si=iA_i=C_ii=TopPc∈i⁡(ac)A_i=TopP_c _i(a_c)∪RecentW⁡(i)∪\,Recent_W(C_i)Cell scores with unrotated i,¯cq_i, k_cric=i⊤¯cd⏟content match+λhacerf⁡(i−pc2σh)⏟relative position bias,c∈ir_ic= [rgb]0.5,0.3359,0.6328 q_i k_c d_content match+ [rgb]0.8789,0.5586,0.043 _ha_c\, \! ( i-p_c2 _h )_relative position bias, c _i¯c k_c: mean cell key; ac=maxj∈ℐc⁡‖j‖2a_c= _j _c k_j_2Top-K per query and headFSFS-L⋯·s⋯·siS_i: tokens selected by iq_iℓij=i⊤jd,j∈i _ij= q_i k_j d, j _ii⊤=softmax⁡(ℓi,i)⊤iu_i =softmax( _i,S_i) V_S_iilr=RMSNorm⁡(i)o_i^lr=RMSNorm(u_i)NoPE readout, RMSNorm per headPrefix cellSelectedPrunedQuery block++×isro_i^srilro_i^lrαh _h per head learned LR weight io_i Figure 2: FreeSolo attention. FS scores the full prefix; FS-L prunes per head, independently of the query. Both select K=4K=4 cells per query and head. We use P=64P=64 high-norm cells, W=8W=8 recent cells, cell size s=32s=32, query blocks of B=128B=128 tokens, and SR radius w=512w=512. 2 Preliminaries Attention. For one head, let i,i,i∈ℝdq_i,k_i,v_i ^d be the query, key, and value at position i∈0,…,N−1i∈\0,…,N-1\. Define ℓij=i⊤j/d _ij=q_i k_j/ d and Δ=i−j =i-j. For a nonempty token set J, collect the logits in ℓi, _i,J and matching value rows j⊤v_j in V_J. The readout is i⊤=softmax⁡(ℓi,)⊤=∑j∈exp⁡(ℓij)j⊤∑j∈exp⁡(ℓij).o_i =softmax( _i,J) V_J= _j ( _ij)v_j _j ( _ij). (1) Full causal attention uses =0,…,iJ=\0,…,i\ and costs O⁡(N2d)O(N^2d) per sequence or O⁡(Nd)O(Nd) per cached decoding step. FlashAttention computes the same readout with O⁡(Nd)O(Nd) memory [Dao et al., 2022]. Sparse attention. Sparse attention restricts each query to i⊆0,…,iJ_i \0,…,i\, fixed or query dependent. If |i|≤m|J_i|≤ m, the readout costs O⁡(Nmd)O(Nmd) per sequence, plus selection. Length extrapolation. Zero-shot length extrapolation evaluates at N>LN>L after training at length L, with fixed weights. We measure retrieval rather than perplexity, whose token average can obscure errors requiring distant context [Fang et al., 2024]. 3 Method 3.1 Range separation FreeSolo combines local window attention with sparse long-range attention in each head (Figure 2). The local branch applies RoPE within a fixed window. The long-range branch selects top-K cells and applies NoPE attention [Kazemnejad et al., 2023] to their tokens, using unrotated queries and keys. This branch is sparse in both FS and FS-L; FS-L also prunes the prefix before query-dependent scoring. Both branches share projections of the layer’s RMS-normalized hidden state iz_i [Zhang and Sennrich, 2019]: i=qiq_i=W_qz_i, i=kik_i=W_kz_i, and i=viv_i=W_vz_i. We combine the outputs as i=isr+αhilr,o_i=o_i^sr+ _ho_i^lr, (2) where αh _h is learned per head. Each branch normalizes its own attention, and their supports may overlap. We concatenate the head outputs, project by oW_o, and add the layer input. A second RMSNorm and SwiGLU feed-forward residual [Shazeer, 2020] complete the layer. 3.2 Local branch Let ℒi=j:0≤j≤i,i−j≤wL_i=\j:0≤ j≤ i,\ i-j≤ w\, where w is a look-back radius, so a full window contains w+1w+1 tokens. Let (i)R(i) be the RoPE rotation at position i [Su et al., 2021]. We use the local score ℓijsr=((i)i)⊤((j)j)d=i⊤(j−i)jd, _ij^sr= (R(i)q_i) (R(j)k_j) d= q_i R(j-i)k_j d, (3) and collect these logits over ℒiL_i to compute (isr)⊤=softmax⁡(ℓi,ℒisr)⊤ℒi.(o_i^sr) =softmax( _i,L_i^sr) V_L_i. (4) We use w=512w=512 at both training and evaluation. Increasing the context therefore introduces no larger relative offsets into this branch. 3.3 Long-range branch The long-range branch provides a sparse, content-addressable memory beyond the local window. Storing contextualized token key/value pairs allows later queries to retrieve individual details directly [Wu et al., 2022, Xiao et al., 2024a]. Mean keys guide cell selection; the readout uses the selected tokens’ keys and values. FS scores the full prefix before the query block. FS-L first prunes this prefix by key norm and recency, separating which cells remain available from which cells a query reads. This separates memory capacity from per-query read bandwidth (Appendix D.3). Cells. A cell is a group of s consecutive token key/value pairs, selected as a unit by the router. Using zero-based indices, cell c contains positions ℐc=cs,…,(c+1)s−1I_c=\cs,…,(c+1)s-1\. Its summaries are ¯c=1s∑j∈ℐcj,pc=(c+12)s,ac=maxj∈ℐc⁡‖j‖2. k_c= 1s _j _ck_j, p_c=(c+ 12)s, a_c= _j _c k_j_2. (5) The mean key summarizes content for routing, pcp_c lies halfway between the cell boundaries, and the largest key norm supplies a query-independent priority. The raw keys and values remain available for token-level attention after selection. Query blocks. A query block groups B/sB/s consecutive cells, covering B token positions, with B a multiple of s. The full prefix is shared across heads and queries in a block. FS-L reuses each head’s retained pool throughout the block, amortizing pruning and gathering. Selection and readout are per query and head. We use B=128B=128 and s=32s=32, so each block contains four cells. The block containing query i starts at bi=B⁡⌊i/B⌋b_i=B i/B . Causal availability. The full prefix iC_i contains all cells before the query-block start: i=c∈ℤ≥0:(c+1)s≤bi.C_i=\c _≥ 0:(c+1)s≤ b_i\. (6) Every cell in this prefix contains only past tokens for every query in the block. The prefix stays fixed until all B queries have finished (Figure 2). At the next boundary, it includes the completed block, and FS-L updates its retained cells. The local window covers the intervening past because w=512≥B−1w=512≥ B-1; the first block has no long-range cells. Cell scoring. We combine mean-key content routing, also used in MoBA [Lu et al., 2025], with a norm-weighted relative position bias: ric=i⊤¯cd+λhacgh(i−pc),c∈i,r_ic= q_i k_c d+ _h\,a_c\,g_h(i-p_c), c _i, (7) where gh(Δ)=erf⁡(Δ/(2σh))g_h( )= ( /(2 _h)), σh>0 _h>0, and λh≥0 _h≥ 0. The content term uses unrotated queries and keys. The erf prior encodes relative distance in a translation-invariant way and supplies a norm-scaled bias to cell scoring. All eligible offsets satisfy i−pc>0i-p_c>0. Ranking these scores and selecting only the top-K cells induces sparsity: only tokens in the selected cells enter the long-range readout. FS scores the full prefix; FS-L first prunes it by key norm. Query-independent pruning. Each head prunes independently using its own projected key norms. FS-L retains the W most recent cells and the P cells with largest aca_c: i=RecentW⁡(i)∪TopPc∈i⁡(ac),|i|≤W+P.A_i=Recent_W(C_i)\ ∪\ TopP_c _i(a_c), |A_i|≤ W+P. (8) The union removes duplicates. The priority favors cells containing a large-norm key, while the recent window preserves nearby context. Projected keys have varying norms; we do not normalize them individually. The RMSNorm after the long-range readout normalizes the output, not these priorities. Key norm supplies a query-independent priority, not a guarantee of relevance to every query. CPT adapts the model to this restricted access pattern. Appendix D.3 measures how much pruning compresses the prefix and which cells remain available for retrieval. We use P=64P=64 and W=8W=8. Per-query selection. FS uses i=iA_i=C_i; FS-L uses Eq. (8). Each head evaluates Eq. (7) on its own iA_i and selects up to K=4K=4 cells per query. Different heads can select different cells, and selections can change at every token within a query block. Let i(i)T_i(A_i) contain the min⁡(K,|i|) (K,|A_i|) highest-scoring cells, and collect their token indices as i=⋃c∈i(i)ℐcS_i= _c _i(A_i)I_c. Token readout. The long-range output is i⊤=softmax⁡(ℓi,i)⊤i,ilr=RMSNorm⁡(i).u_i =softmax( _i,S_i) V_S_i, _i^lr=RMSNorm(u_i). (9) This is NoPE attention: the erf prior affects cell scoring only, while the readout uses unmodified token keys and values, with no positional bias in its logits. The softmax is normalized jointly over all selected tokens. RMSNorm acts over the d head features [Zhang and Sennrich, 2019]. Both outputs in Eq. (9) are zero when iS_i is empty. Training. We obtain FS-L by continued pretraining of FS with pruning enabled. Both stages use the standard next-token cross-entropy objective on 40964096-token sequences. CPT resets the optimizer and learning-rate schedule (Appendix A) and requires no longer training contexts. Causal masks allow all query blocks to be evaluated in parallel. Hard top-P and top-K indices receive no gradient; shared projections and contextual states learn through local attention and selected-token readout. The routing-only scalars receive no gradient in this configuration and remain at σh=4 _h=4 and λh=1 _h=1. Training with pruning from scratch is also possible, but CPT gives better overall retrieval, particularly on the harder tasks. The 340340M from-scratch model retains strong S-NIAH-1 recall, whereas the tested 800800M run retrieves poorly. Starting from FS provides useful representations before imposing the pruning constraint. This motivates the two-stage recipe, although the runs do not isolate the cause of the from-scratch difference. Appendix D.4 gives the comparisons and variation across CPT runs. Inference. Prompt prefill uses the same causal operator and populates each layer’s raw key/value cache and cell summaries. During decoding, each new token appends its key and value; a completed cell supplies its mean key and maximum key norm. The local window advances every token. The long-range prefix advances only at query-block boundaries, when FS-L also updates its retained set. Within a block, each query scores the available cells, selects its own top-K, and applies Eq. (9) before the branch combination in Eq. (2). FS-L’s streaming cache merges newly eligible cells into the retained set and discards unused KV once it leaves the recent, local, and unfinished-block buffers. 3.4 Balanced tree search (FS-B) Token sequence: s=32s=32 tokens per cell⋯·sc−8c_-8c−7c_-7c−6c_-6c−5c_-5c−4c_-4c−3c_-3c−2c_-2c−1c_-1c0c_0c1c_1c2c_2c3c_3iq_i00128128Full prefix iC_iAll cells before this query blockQuery block: B=128B=128Forests grow at block boundaries,fixed within a blockBeam score of tree node vvUv=i⊤v+c‖i‖2ρvd⏟content bound+λhAverf⁡(i−pvmin2σh)⏟position biasU_v= [rgb]0.5,0.3359,0.6328 q_i μ_v+c q_i_2 _v d_content bound+ [rgb]0.8789,0.5586,0.043 _hA_v\, \! ( i-p_v 2 _h )_position biasLong rangeFull-prefix forest: keep the full prefixi0=iA_i^0=C_iNorm-filtered forest: prune per head by key normiτ=c∈i:ac≥τℓhA_i^τ=\c _i:a_c≥ _ h\iq_iτℓh _ hiq_iℬi0B_i^0: 1616 cellsℬiτB_i^τ: 1616 cellsRerank ℬi=ℬi0∪ℬiτB_i=B_i^0 _i^τ with unrotated iq_i and token keysr~ic=log∑j∈ℐcexp(i⊤jd)⏟content match+λhacerf⁡(i−pc2σh)⏟relative position bias,c∈ℬi r_ic= [rgb]0.5,0.3359,0.6328 _j _c \! ( q_i k_j d )_content match+ [rgb]0.8789,0.5586,0.043 _ha_c\, \! ( i-p_c2 _h )_relative position bias, c _iℐcI_c: tokens of cell c; ac=maxj∈ℐc⁡‖j‖2a_c= _j _c k_j_2Top-K per query and head|i|=Ks|S_i|=Ks tokensiS_i: tokens selected by iq_iPrefix cellQuery-dependent searchSelectedPruned Figure 3: FS-B long-range selection, per head. The query searches two causal forests over the prefix: one over every cell and one over cells with ac≥τℓha_c≥ _ h. Each beam keeps 1616 nodes per level down to the leaves, ⌈log2M⌉ _2M levels for M admitted cells (66 and 55 in this 6464-cell example). The union of both beams is reranked with token keys; the top K=4K=4 cells form iS_i, which is read as in FS. FS-L suits evidence concentrated in a few cells, as in single-needle retrieval, because query-independent pruning can keep those cells. Dense, independently queryable facts are harder: in a list of unique key–value pairs followed by a question about a uniformly sampled key, every pair is equally likely to matter, so a fixed pool reaches a shrinking fraction of the possible targets (Appendix D.5). FS keeps every cell reachable but scores the full prefix in O⁡(N2/s)O(N^2/s) work. FS-B keeps the local branch and the NoPE readout of Eq. (9) and changes only how iS_i is chosen: each query searches content trees over a growing memory, so no cell is discarded for all future queries while search and read budgets stay bounded. Two causal forests. FS-B always uses two forests per head, with admitted sets i0=i,iτ=c∈i:ac≥τℓh,A_i^0=C_i, _i^τ=\c _i:a_c≥ _ h\, (10) where τℓh _ h is the median cell norm on FineWeb-Edu windows, calibrated once per checkpoint and reused at all lengths. The first forest keeps the whole prefix searchable; the second emphasizes high-norm evidence, as FS-L does, without a fixed quota. In each forest, every aligned chunk of 2k2^k admitted cells is split recursively at the median of its mean keys along their principal direction, and summary nodes join the chunk roots, so M admitted cells give height ⌈log2M⌉ _2M (Figure 9). A node v over cells ⁡(v)C(v) stores their mean key v μ_v, a radius ρv≥maxc∈⁡(v)⁡‖¯c−v‖2 _v≥ _c (v) k_c- _v_2, the maximum norm Av=maxc∈⁡(v)⁡acA_v= _c (v)a_c, and the earliest position pvmin=minc∈⁡(v)⁡pcp_v = _c (v)p_c. Beam search. From the root, each forest keeps the Wb=16W^b=16 highest-priority nodes per level down to the leaves, ranking nodes by Uv=i⊤v+c‖i‖2ρvd+λhAvgh(i−pvmin).U_v= q_i μ_v+c q_i_2 _v d+ _hA_v\,g_h(i-p_v ). (11) For c≥1c≥ 1, UvU_v bounds the score of Eq. (7) for every cell below v: the radius bounds the content term, and AvA_v and pvminp_v bound the nonnegative, monotone prior. The search remains approximate, since a branch with a lower bound can still hold the best cell; we use c=8c=8. The leaves reached form candidate sets ℬi0B_i^0 and ℬiτB_i^τ of at most 1616 cells each, which differ because the forests group the prefix differently. Token-level reranking. Mean keys can hide a single useful token, so FS-B rescores the deduplicated candidates ℬi=ℬi0∪ℬiτB_i=B_i^0 _i^τ with their token keys, r~ic=log∑j∈ℐcexp(i⊤jd)+λhacgh(i−pc),c∈ℬi, r_ic= _j _c \! ( q_i k_j d )+ _ha_c\,g_h(i-p_c), c _i, (12) and the top K=4K=4 cells form iS_i, read with Eq. (9). Retrieval-weighted CPT. At 340340M, FS-B reuses the FS-L weights. At 800800M, base FS undergoes 0.420.42B-token CPT on 40964096-token FineWeb-Edu sequences with a weighted next-token loss: ℒ=−∑twtlog⁡pθ(xt+1∣x≤t)∑twt,wt=1+βft+βswχt,L=- _tw_t p_θ(x_t+1 x_≤ t) _tw_t, w_t=1+β f_t+ _sw _t, (13) Here ft,χt∈0,1f_t, _t∈\0,1\ are text-derived masks. The recall mask ft=1f_t=1 marks starts and continuations of repeated spans, identified by a unique earlier four-token match more than 512512 tokens back in the same document. The switch mask χt=1 _t=1 adds a bonus at a recall start within 6464 tokens of the preceding start if their earlier source positions are more than 6464 tokens apart. Every token retains base weight 11. Training uses full-prefix FS selection with 1616-cell reads; forests are used only at inference (Appendix E.4). 3.5 Computational cost At fixed head dimensions and budgets, FS costs O⁡(N2/s)O(N^2/s), FS-L O⁡(N)O(N), and FS-B O⁡(Nlog2N)O(N ^2N) including radix-sort construction. Cell statistics cost O⁡(Nd)O(Nd) per head. Each query scores M≃N/sM N/s cells, giving FS a cost of O⁡(N2d/s+N⁡(w+Ks)d)O(N^2d/s+N(w+Ks)d) including local and selected-token readout. Query blocks share availability, not scores; B does not reduce the quadratic term. FS-L incrementally merges newly eligible cells into its top-P set. Fixed key-norm scores prevent discarded cells from re-entering. A heap costs O⁡(Mlog⁡(P+1))O(M (P+1)) in total; bounded-array merges are linear at fixed budgets. Let C=P+WC=P+W. Scoring the retained cells costs O⁡(NCd)O(NCd), and attention over gathered selected tokens costs O⁡(NKsd)O(NKsd). With incremental retention and the local branch, the sequence cost is O⁡(Nd+Mlog⁡(P+1)+N⁡(C+w+Ks)d).O\! (Nd+M (P+1)+N(C+w+Ks)d ). (14) At fixed budgets, decode work is bounded, including amortized retention updates. The streaming implementation keeps O⁡(((P+W)s+w+B+s)d)O(((P+W)s+w+B+s)d) token state per head plus indices and counters, computes rotations on demand, and prefills in fixed chunks (Appendix C). FS-L results in Tables 1 and 4 use this bounded cache, which keeps only the retained cells (Appendix A); its resource measurements are in Appendix C. The efficiency comparison in Table 6 uses a masked implementation that stores O⁡(Nd)O(Nd) KV and rescans priorities at block boundaries, adding Θ⁡(N2/(Bs)) (N^2/(Bs)) work, with O⁡(N⁡(P+W)sd)O(N(P+W)sd) masked readout. FS-B searches M≤⌊N/s⌋=⌊N/32⌋M≤ N/s = N/32 admitted cells per forest. Beams descend ⌈log2M⌉ _2M levels; search, reranking and readout cost O⁡(d⁡[Wblog⁡M+Wbs+Ks])O(d[W^b M+W^bs+Ks]) per query, or O⁡(Nlog⁡N)O(N N) per sequence. Fixed-width radix sorting adds O⁡(Mdlog2⁡M)O(Md ^2M) construction work, giving O⁡(Nlog2N)O(N ^2N) combined algorithmic cost and O⁡(Nd)O(Nd) archive space per head. Appendix E.2 derives this bound and discloses repeated buffer copies in the reference streaming cache. 4 Experiments 4.1 Setup We train LLaMA-style decoders [Touvron et al., 2023] on FineWeb-Edu [Penedo et al., 2024], using a 3232k LLaMA tokenizer and a 40964096-token context. The two FreeSolo configurations have 341341M and 743743M parameters, with pretraining budgets of 1515B and 3232B tokens. Training uses bf16 and AdamW [Loshchilov and Hutter, 2019]; see Appendix A for architecture and optimizer settings. FS-L uses 0.730.73B/1.571.57B CPT tokens at 340340M/800800M, about 5%5\% of pretraining; FS-B uses 0.420.42B at 800800M (Section 3.4). All CPT uses 40964096-token contexts. Main-table baselines use pretraining checkpoints. Appendix D.4 adds selected 800800M-scale baseline CPT controls, FS/FS-L comparisons at both scales, and seed and from-scratch studies. Baselines. TF++ is the full-attention Transformer++ reference [Yang et al., 2025b] with RoPE; SWA uses a 512512-token RoPE window. NSA [Yuan et al., 2025] and MoBA [Lu et al., 2025] select blocks for sparse token attention. GDN [Yang et al., 2025b] uses a recurrent gated delta rule, and NAtS-L [Deng et al., 2026] combines GDN with retained-token softmax attention. InfLLM [Xiao et al., 2024a] adds inference-time retrieval to the TF++ checkpoint; we evaluate it on S-NIAH-1/2/3. Raven [Afzal et al., 2026] writes sparsely to key/value memory slots and reads them with softmax. We use the standalone 256256-slot model as the linear-time Raven baseline. Raven-full has stronger published S-NIAH-2/3 recall, but its full NoPE attention costs O⁡(N2)O(N^2), so we exclude it from this linear-time comparison. Its SWA hybrid is linear at fixed window size but has weaker recall than Raven-full [Afzal et al., 2026, Table 9]. Baselines share the corpus, tokenizer, and training context within each scale. The tables list parameter counts; Appendix A records training budgets and settings. Complexity. Table 1 reports algorithmic attention complexity at fixed head dimensions and budgets: O⁡(N2/s)O(N^2/s) for FS, O⁡(N2/b)O(N^2/b) for MoBA and NSA with a fixed block factor b, O⁡(Nlog2N)O(N ^2N) for FS-B including radix-sort construction, and O⁡(N)O(N) for FS-L (Section 3.5), Raven, GDN, and SWA. NAtS-L costs O⁡(NR)O(NR) for R retained tokens, quadratic at a fixed retention fraction [Deng et al., 2026]. Our InfLLM lookup is exhaustive. Evaluation. We evaluate NVIDIA RULER [Hsieh et al., 2024] with 100100 examples per task and length, greedy decoding, and official 00–100100 scores. Suite means cover 1212 tasks at 22k/44k and 1313 at 88k/1616k, which add QA-2. We omit further long-context runs when a model’s retrieval has already failed at shorter extrapolation lengths. Appendix B reports all task scores; Appendix A gives the protocol. 4.2 Zero-shot extrapolation for long-context retrieval Raven, NAtS-L, and GDN retain retrieval ability beyond the training context, whereas MoBA and NSA do not sustain recall in our evaluations (Table 1). FreeSolo’s advantage is clearest on the harder S-NIAH-2/3 tasks (Figure 1). At 800800M and 1616k, four times the training length, FS-L scores 81/4081/40, compared with 26/026/0 for Raven and 72/3472/34 for FS. FS-L reaches 53/1353/13 at 6464k, compared with 28/428/4 for FS and 0/00/0 for NAtS-L. Relative to FS, the 800800M FS-L checkpoint trades some S-NIAH-1 recall for higher S-NIAH-2/3 scores at 1616k–6464k, although absolute accuracy still declines with length. FS-B’s two-forest search (Section 3.4) improves retrieval further: at 800800M and 1616k it scores 100/88/84100/88/84 on S-NIAH-1/2/3, with a full RULER mean of 30.430.4. FS and FS-L also outperform Raven and NAtS-L on the full RULER mean at both scales. At 800800M/1616k, the means are 23.1/22.323.1/22.3 for FS/FS-L, versus 12.8/2.812.8/2.8 for Raven/NAtS-L. The gains extend beyond single needles: FS-L scores 2323 on MK-NIAH-1 versus 33 for both linear baselines, and 1717 on MV-NIAH versus 9/0.59/0.5 for Raven/NAtS-L (Table 4). Some tasks, including variable tracking, remain difficult for all models even within the training context. Matched continuations distinguish adaptation to pruning from additional training (Appendix D.4). On the same 5050 prompts at 800800M/1616k, FS continuation scores 84/3884/38 on S-NIAH-2/3 and FS-L scores 80/4280/42. Thus short CPT preserves much of the harder-task retrieval while reducing attention cost from quadratic to linear. S-NIAH-1 tests retrieval over a much longer range. At 340340M, FS maintains 9898–100100 through 256256k and FS-L scores 8484 at 256256k; Raven falls to 2020 at 131131k. At 800800M, FS and FS-L score 96/8596/85 at 256256k, compared with zero for NAtS-L. Raven also extrapolates well on this task, scoring 9999–100100 through 131131k and 9292 at 256256k. Table 1: RULER accuracy (00–100100). All models train at 44k; results at 88k and above test zero-shot extrapolation. Means cover available tasks; cost is defined in Section 4.1. FS-B’s algorithmic cost includes radix-sort construction (Appendix E.2). Bold/underline mark the best/second-best scores within each block (super-linear or linear cost); dashes denote unavailable results. S-NIAH-1 S-NIAH-2 S-NIAH-3 Mean Model Params Complexity 2k 4k 8k 16k 32k 64k 131k 256k 2k 4k 8k 16k 32k 64k 2k 4k 8k 16k 32k 64k 2k 4k 8k 16k TF++ 335335M O⁡(N2)O(N^2) 9595 5757 00 00 – – – – 91¯ 91 2424 00 00 – – 3838 11 00 00 – – 26.226.2 13.313.3 0.20.2 0.00.0 InfLLM 335335M O⁡(N2)O(N^2) 9494 58¯ 58 1313 99 – – – – 8989 2323 88 22 – – 3939 11 22 1¯ 1 – – – – – – MoBA 341341M O⁡(N2/b)O(N^2/b) 7676 1212 00 00 – – – – 8383 2323 00 00 – – 59¯ 59 33 00 00 – – 25.625.6 7.67.6 0.00.0 0.00.0 NSA 513513M O⁡(N2/b)O(N^2/b) 4343 2323 00 00 – – – – 4848 1818 00 00 – – 11 00 00 00 – – 13.213.2 7.37.3 0.90.9 0.00.0 Raven-full 355355M O⁡(N2)O(N^2) 100 99 8383 3939 4444 4¯ 4 00 0¯ 0 100 89¯ 89 41¯ 41 77 – – 3838 11 00 00 – – 26.726.7 21.521.5 13.513.5 7.17.1 FS 341341M O⁡(N2/s)O(N^2/s) 99¯ 99 99 100 100 98¯ 98 100 98¯ 98 100 100 7777 41¯ 41 17¯ 17 – – 5858 14¯ 14 7¯ 7 00 – – 30.8¯ 30.8 23.2¯ 23.2 16.5¯ 16.5 11.8¯ 11.8 FS-B 341341M O⁡(Nlog2N)O(N ^2N) 100 99 99¯ 99 99¯ 99 99 100 100 100 100 99 89 60 32 10 91 78 64 27 34 24 34.234.2 31.131.1 26.626.6 20.220.2 FS-L 341341M O⁡(N)O(N) 99¯ 99 8888 90¯ 90 87 65¯ 65 86 85 84 99 70 34¯ 34 20 7 3 70 17 9 1 2 1 32.432.4 21.621.6 15.515.5 10.710.7 Raven 357357M O⁡(N)O(N) 9595 96 93 81¯ 81 78 41¯ 41 20¯ 20 – 92¯ 92 70 36 3¯ 3 – – 33 00 00 0¯ 0 – – 20.520.5 17.8¯ 17.8 14.0¯ 14.0 8.5¯ 8.5 NAtS-L 380380M O⁡(NR)O(NR) 100 94¯ 94 3737 1919 00 00 00 – 8989 22¯ 22 2020 3¯ 3 – – 36¯ 36 6¯ 6 11 0¯ 0 – – 25.6¯ 25.6 15.315.3 7.47.4 4.04.0 GDN 335335M O⁡(N)O(N) 9595 8383 3535 1919 – – – – 8383 2020 1818 3¯ 3 – – 00 00 00 0¯ 0 – – 21.621.6 14.214.2 7.07.0 3.63.6 SWA 341341M O⁡(N)O(N) 1818 1515 77 33 – – – – 3131 2121 66 22 – – 2525 22 3¯ 3 0¯ 0 – – 14.614.6 9.39.3 4.54.5 4.04.0 ≈800≈\!800M scale TF++ 743743M O⁡(N2)O(N^2) 100 9595 00 00 – – – – 100 9494 00 00 – – 6767 5050 00 00 – – 46.346.3 37.5¯ 37.5 0.00.0 0.00.0 InfLLM 743743M O⁡(N2)O(N^2) 100 9696 2020 44 – – – – 100 8989 1212 44 – – 7777 5050 44 00 – – – – – – NSA 11281128M O⁡(N2/b)O(N^2/b) 100 7575 88 00 – – – – 98¯ 98 4646 33 00 – – 3030 77 00 00 – – 40.740.7 24.824.8 2.62.6 0.10.1 Raven-full 723723M O⁡(N2)O(N^2) 100 100 100 100 98¯ 98 9191 7474 4848 100 8888 8181 6969 5151 11 91¯ 91 5858 2121 44 – – 41.441.4 33.733.7 26.526.5 23.8¯ 23.8 FS 743743M O⁡(N2/s)O(N^2/s) 97¯ 97 9191 9393 95¯ 95 7979 97¯ 97 95¯ 95 96¯ 96 100 99¯ 99 92¯ 92 72¯ 72 63¯ 63 28¯ 28 8585 71¯ 71 51¯ 51 34¯ 34 27¯ 27 4¯ 4 39.439.4 34.034.0 27.6¯ 27.6 23.123.1 FS-B 743743M O⁡(Nlog2N)O(N ^2N) 97¯ 97 98¯ 98 97¯ 97 100 100 100 100 100 100 100 95 88 80 43 96 93 87 84 71 69 42.1¯ 42.1 37.737.7 31.831.8 30.430.4 FS-L 743743M O⁡(N)O(N) 100 92¯ 92 84¯ 84 84¯ 84 84¯ 84 78¯ 78 80¯ 80 85¯ 85 99¯ 99 99 89 81 68 53 90 71 55 40 29 13 37.237.2 32.032.0 25.625.6 22.322.3 Raven 726726M O⁡(N)O(N) 100 100 100 100 100 99 99 92 99¯ 99 97¯ 97 81¯ 81 26¯ 26 5¯ 5 – 3232 77 11 00 0¯ 0 – 30.130.1 23.9¯ 23.9 18.4¯ 18.4 12.8¯ 12.8 NAtS-L 793793M O⁡(NR)O(NR) 86¯ 86 7272 5959 2121 22 11 00 00 100 7676 2828 33 00 0¯ 0 6262 2424 66 00 0¯ 0 0¯ 0 27.827.8 19.419.4 10.210.2 2.82.8 GDN 694694M O⁡(N)O(N) 100 100 100 5656 2222 – – – 9494 3838 1919 1313 – – 84¯ 84 27¯ 27 10¯ 10 5¯ 5 – – 35.8¯ 35.8 22.822.8 16.516.5 10.410.4 SWA 743743M O⁡(N)O(N) 1818 1515 77 33 – – – – 3131 2121 66 22 – – 1919 33 11 00 – – 15.615.6 10.310.3 5.55.5 4.04.0 4.3 Short-context quality Table 2 reports eight zero-shot tasks with task definitions from the Language Model Evaluation Harness [Gao et al., 2023]. We use acc_norm for HellaSwag, ARC-c, and OBQA, and acc for the other tasks; Avg is their arithmetic mean. At 340340M, FS averages 51.951.9 and FS-L 51.051.0, compared with 51.851.8 for TF++, 51.451.4 for Raven, and 50.850.8 for NAtS-L. At 800800M, FS reaches 56.556.5, close to TF++’s 56.856.8 and SWA’s 56.256.2, while FS-L reaches 55.255.2 and Raven 54.654.6. Both FS-L checkpoints retain similar short-context quality. Table 2: Zero-shot downstream accuracy (00–100100). Rows are ordered by average within each scale; bold and underline mark the best and second-best scores. Model Params HellaS. PIQA ARC-e ARC-c WinoG. OBQA SciQ BoolQ Avg GDN 335335M 39.2¯ 39.2 66.8¯ 66.8 55.855.8 28.228.2 51.9¯ 51.9 33.0¯ 33.0 83.283.2 59.859.8 52.252.2 FS 341341M 38.938.9 66.8¯ 66.8 57.657.6 25.925.9 50.950.9 32.432.4 81.281.2 61.161.1 51.9¯ 51.9 TF++ 335335M 39.2¯ 39.2 66.066.0 56.256.2 28.1¯ 28.1 50.650.6 32.832.8 81.5¯ 81.5 59.659.6 51.851.8 Raven 357357M 37.637.6 65.265.2 55.855.8 27.627.6 51.251.2 32.232.2 80.680.6 60.8¯ 60.8 51.451.4 SWA 341341M 38.838.8 66.266.2 56.4¯ 56.4 26.726.7 50.650.6 30.830.8 80.480.4 61.161.1 51.451.4 FS-L 341341M 38.438.4 66.366.3 55.055.0 25.325.3 51.051.0 31.431.4 79.879.8 60.460.4 51.051.0 FS-B 341341M 38.438.4 66.366.3 55.055.0 25.325.3 51.051.0 31.431.4 79.879.8 60.360.3 51.051.0 NAtS-L 380380M 39.939.9 67.667.6 50.050.0 27.427.4 52.352.3 34.034.0 75.175.1 59.759.7 50.850.8 MoBA 341341M 33.333.3 61.861.8 43.443.4 26.226.2 51.351.3 29.029.0 65.165.1 52.252.2 45.345.3 NSA 513513M 32.332.3 62.662.6 43.843.8 24.924.9 51.9¯ 51.9 28.828.8 66.466.4 45.045.0 44.544.5 ≈800≈\!800M scale TF++ 743743M 48.948.9 70.970.9 63.963.9 32.9¯ 32.9 54.354.3 34.434.4 87.487.4 61.961.9 56.856.8 FS 743743M 49.5¯ 49.5 70.0¯ 70.0 64.6¯ 64.6 31.731.7 56.056.0 34.434.4 86.786.7 58.858.8 56.5¯ 56.5 Raven-full 723723M 46.946.9 69.469.4 64.564.5 33.633.6 54.154.1 36.836.8 85.985.9 60.460.4 56.5¯ 56.5 GDN 694694M 47.447.4 69.169.1 64.864.8 32.832.8 53.853.8 37.0¯ 37.0 88.1¯ 88.1 58.258.2 56.456.4 SWA 743743M 49.649.6 68.768.7 63.563.5 31.631.6 55.1¯ 55.1 34.434.4 85.285.2 61.8¯ 61.8 56.256.2 FS-B 743743M 49.649.6 70.0¯ 70.0 63.863.8 31.731.7 56.056.0 33.433.4 85.985.9 59.159.1 56.256.2 NSA 11281128M 47.447.4 68.168.1 63.863.8 31.731.7 54.354.3 32.432.4 88.888.8 61.461.4 56.056.0 FS-L 743743M 48.148.1 69.069.0 62.662.6 31.831.8 53.653.6 33.633.6 85.485.4 57.557.5 55.255.2 NAtS-L 793793M 44.844.8 68.268.2 63.063.0 29.029.0 52.852.8 37.437.4 85.385.3 59.259.2 55.055.0 Raven 726726M 45.845.8 68.768.7 63.463.4 31.731.7 54.054.0 35.835.8 84.484.4 53.253.2 54.654.6 4.4 Empirical scaling To verify linear scaling empirically, we increase context from 44k to 256256k on one A100-40GB. FS-L’s attention buffers remain constant at 766766 MiB for the 340340M model and 11491149 MiB for the 800800M model. CUDA-graph decoding runs at about 8989 and 7575 tokens/s, respectively, including retention updates. Prefill throughput stays near 2.12.1–2.22.2K tokens/s, so processing time grows approximately linearly with input length. These timings use bf16 parameters; Appendix C compares precision and execution paths. The masked implementation prefills at 34.634.6–44.244.2K tokens/s but keeps a growing KV cache. Training at 340340M/44k is also slower than full attention: 98.498.4K versus 303303K tokens/s across eight GPUs; faster kernels for pruning, gathering, and sparse readout are future work (Appendix C). Reducing P to 1616 cuts 340340M attention storage to 320320 MiB with similar decode speed, but accuracy varies non-monotonically with P (Appendix D.5). 5 Length extrapolation and learned retention We analyze length generalization through an information-bottleneck view: FS routes through cell summaries, and FS-L also restricts memory before queries arrive (Appendix D). Position dependence. Position handling affects how attention transfers to longer sequences [Press et al., 2022, Kazemnejad et al., 2023, Chi et al., 2023]. FS uses RoPE only within trained local offsets and NoPE for long-range token attention, so longer contexts introduce no untrained rotary offsets. The erf prior and local RoPE depend only on relative offsets, so shifting tokens, cells and query blocks together preserves the operator at fixed content vectors (Appendix D.1). Added tokens can still introduce competing cells. The information bottleneck in FS. The router identifies useful cells from their mean keys and maximum key norms. Token attention then reads the selected cells’ original keys and values. We hypothesize that routing through these summaries encourages recognizable keypoints. Each head reads at most KsKs token values, regardless of prefix length. RMSNorm rescales the readout but cannot remove interference from irrelevant values; sparse selection controls which values can contribute. On a shared 256256k prompt, both variants at both scales select the needle cell over 166,000166,000 tokens away (Appendix D.2). The additional bottleneck in FS-L. FS-L discards cells before future queries arrive. This can tolerate growing background when useful information remains concentrated in the retained state, but not dense, independently queryable facts (Section 3.4). CPT adapts the model to this restriction. At 340340M/44k, it recovers S-NIAH-1 from 6262 under immediate pruning to 7676–9090 across runs (9090 for the evaluated FS-L); unpruned continuations retain 100100 (Appendix D.4). FS-L retains 14.3%14.3\% of eligible cells at 340340M/1616k; Appendix D.3 separates retention, selection, and answering. Budgets and ablations. P sets the high-norm cells retained per head; K sets the cells read per query and head. Larger K cannot recover a pruned cell; larger P expands availability but need not improve answers. We test adapted budgets and retrieval among several independently queryable facts (Appendix D.5). The local NoPE control changes both rotation and local bias. It preserves S-NIAH-1 extrapolation but reduces downstream accuracy from 51.951.9 to 38.338.3. Removing the long-range norm-weighted erf term retains strong S-NIAH-1 recall, worsens S-NIAH-2, and improves S-NIAH-3 (Appendix D.1). Since gh≥erf⁡(2)≃0.9953g_h≥ (2) 0.9953 for eligible cells, the term acts mainly as a norm bonus, so we do not credit extrapolation to its distance dependence (Appendix D.1). 6 Related Work Selective recurrent memory. Mamba, Mamba-2, and GDN compress history in recurrent states [Gu and Dao, 2023, Dao and Gu, 2024, Yang et al., 2025b]; GSA and Raven use softmax over memory slots [Zhang et al., 2024, Afzal et al., 2026]. FS preserves token keys and values; summaries locate evidence. Mixture-of-Experts and Mixture-of-Memories route to parameter experts or recurrent states [Shazeer et al., 2017, Du et al., 2025]; FS routes to token cells. Native block-sparse attention. MoBA [Lu et al., 2025] also routes by mean keys and reads selected tokens, always including the causal current block. NSA [Yuan et al., 2025] learns compressed keys and values, derives selection scores from compressed attention, and includes compressed values in its readout. FS adds norm priorities and local-only RoPE to mean-key routing and selected-token readout. Training scores all eligible cells but reads only selected values. FS-L prunes before query matching; FS-B searches key hierarchies over the archive. Both reuse learned token keys across memory and search budgets without a learned chunk compressor. Sparse token retrieval. Landmark Attention uses learned landmarks [Mohtashami and Jaggi, 2023]; InfLLM uses representative tokens [Xiao et al., 2024a]. DRT [Hu et al., 2025b] and RAMba [Hu et al., 2025a] learn chunk retrieval and extrapolate beyond training length; routing scores weight their chunk outputs. FS hard-selects by fixed mean/norm summaries, then applies one softmax over selected tokens. Keys learn through token readout and local attention, without gradients through selection indices (Appendix E.5). KV-cache selection and pruning. QUEST selects pages using key bounds [Tang et al., 2024]. H2O and SnapKV prune using observed attention [Zhang et al., 2023, Li et al., 2024]; Devoto et al. [2024] retain low-norm keys. FS-L retains high-norm cells and adapts representations to that fixed rule through CPT. Retention needs neither a future query nor accumulated attention scores. NAtS-L combines retained-token attention with GDN [Deng et al., 2026]; its softmax history can grow. StreamingLLM keeps sinks and recent tokens [Xiao et al., 2024b]; FS-L can retain cells from anywhere in the prefix. Sparsity and length generalization. ASEntmax studies sparse weights and extrapolation [Vasylenko et al., 2026]; FS uses softmax over selected tokens. Position interpolation and YaRN rescale RoPE [Chen et al., 2023, Peng et al., 2024]. RoPE/NoPE hybrids [Yang et al., 2025a, Puvvada et al., 2025] combine local RoPE and global NoPE layers; Kimi Linear and Kimi K3 combine recurrence with global NoPE and train on long contexts [Kimi Team, 2025, Kimi Team, 2026]. FS couples local RoPE with sparse NoPE retrieval within each head, training at short context throughout. Its long-range branch never encounters unseen rotary offsets. 7 Limitations Experiments stop at the 800800M scale. FS-L’s fixed retention budget can discard evidence before a query arrives. Its current CPT recovers S-NIAH-1 lost to immediate pruning, but the resulting 800800M model has a lower RULER mean than base FS at every evaluated length (Table 12). Finding a CPT objective that improves the suite mean within FS-L’s memory budget remains open. Retrieval-weighted CPT, as used for FS-B, motivates this direction (Appendix E.4). FS-B keeps growing memory and improves the RULER mean, but trails Raven-full on 800800M multi-query and multi-value retrieval (Table 4). Learning its search policy, rather than using a fixed inference algorithm, is another direction for future work. 8 Conclusion FreeSolo’s central thesis is that memory summaries need only locate evidence; the evidence can remain in original token values. Mean summaries of learned token keys support hierarchical search, maximum token-key norms support pruning, and local-only RoPE keeps distant retrieval free of unseen rotary offsets. FS-L and FS-B show how this representation supports bounded memory or a growing archive after short CPT, with zero-shot retrieval at 1616k and single-needle recall at 256256k from 44k training. This separates learning useful memory keys from choosing the capacity, search effort, and read bandwidth of the memory. References Afzal et al. [2026] Arshia Afzal, Aviv Bick, Eric P. Xing, Volkan Cevher, and Albert Gu. Raven: High-recall sequence modeling with sparse memory routing, 2026. URL https://arxiv.org/abs/2607.25357. Chen et al. [2023] Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595, 2023. Chi et al. [2023] Ta-Chung Chi, Ting-Han Fan, Alexander I. Rudnicky, and Peter J. Ramadge. Dissecting transformer length extrapolation via the lens of receptive field analysis. In Proceedings of the Association for Computational Linguistics (ACL), 2023. Chiang and Cholak [2022] David Chiang and Peter Cholak. Overcoming a theoretical limitation of self-attention. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7654–7664. Association for Computational Linguistics, 2022. doi: 10.18653/v1/2022.acl-long.527. URL https://aclanthology.org/2022.acl-long.527/. Dao [2023] Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691, 2023. URL https://arxiv.org/abs/2307.08691. Dao and Gu [2024] Tri Dao and Albert Gu. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. arXiv preprint arXiv:2405.21060, 2024. Dao et al. [2022] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems, volume 35, 2022. Deng et al. [2026] Difan Deng, Andreas Bentzen Winje, Lukas Fehring, and Marius Lindauer. Neural attention search linear: Towards adaptive token-level hybrid attention models, 2026. URL https://arxiv.org/abs/2602.03681. Devoto et al. [2024] Alessio Devoto, Yu Zhao, Simone Scardapane, and Pasquale Minervini. A simple and effective L2L_2 norm-based strategy for KV cache compression, 2024. URL https://arxiv.org/abs/2406.11430. Du et al. [2025] Jusen Du, Weigao Sun, Disen Lan, Jiaxi Hu, and Yu Cheng. MoM: Linear sequence modeling with mixture-of-memories. arXiv preprint arXiv:2502.13685, 2025. URL https://arxiv.org/abs/2502.13685. Fang et al. [2024] Lizhe Fang, Yifei Wang, Zhaoyang Liu, Chenheng Zhang, Stefanie Jegelka, Jinyang Gao, Bolin Ding, and Yisen Wang. What is wrong with perplexity for long-context language modeling? arXiv preprint arXiv:2410.23771, 2024. Fu et al. [2024] Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim, and Hao Peng. Data engineering for scaling language models to 128k context. In Proceedings of the 41st International Conference on Machine Learning, volume 235, pages 14125–14134, 2024. URL https://proceedings.mlr.press/v235/fu24d.html. Gao et al. [2023] Leo Gao et al. A framework for few-shot language model evaluation, 2023. URL https://zenodo.org/records/10256836. Gu and Dao [2023] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023. He et al. [2025] Zifan He, Yingqi Cao, Zongyue Qin, Neha Prakriya, Yizhou Sun, and Jason Cong. HMT: Hierarchical memory transformer for efficient long context language processing. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 8068–8089. Association for Computational Linguistics, 2025. doi: 10.18653/v1/2025.naacl-long.410. URL https://aclanthology.org/2025.naacl-long.410/. Hsieh et al. [2024] Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. RULER: What’s the real context size of your long-context language models? arXiv preprint arXiv:2404.06654, 2024. Hu et al. [2025a] Xiang Hu, Jiaqi Leng, Jun Zhao, Kewei Tu, and Wei Wu. Hardware-aligned hierarchical sparse attention for efficient long-term memory access. arXiv preprint arXiv:2504.16795, 2025a. URL https://arxiv.org/abs/2504.16795. Hu et al. [2025b] Xiang Hu, Zhihao Teng, Jun Zhao, Wei Wu, and Kewei Tu. Efficient length-generalizable attention via causal retrieval for long-context language modeling. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 24727–24743. PMLR, 2025b. URL https://proceedings.mlr.press/v267/hu25v.html. Katharopoulos et al. [2020] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are RNNs: Fast autoregressive transformers with linear attention. In International Conference on Machine Learning, pages 5156–5165, 2020. Kazemnejad et al. [2023] Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. The impact of positional encoding on length generalization in transformers. In Advances in Neural Information Processing Systems (NeurIPS), 2023. Kimi Team [2025] Kimi Team. Kimi linear: An expressive, efficient attention architecture. arXiv preprint arXiv:2510.26692, 2025. Kimi Team [2026] Kimi Team. Kimi k3: Open frontier intelligence, 2026. URL https://arxiv.org/abs/2607.24653. Lampinen et al. [2021] Andrew K. Lampinen, Stephanie C. Y. Chan, Andrea Banino, and Felix Hill. Towards mental time travel: A hierarchical memory for reinforcement learning agents. In Advances in Neural Information Processing Systems, volume 34, 2021. URL https://arxiv.org/abs/2105.14039. Li et al. [2024] Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. SnapKV: LLM knows what you are looking for before generation, 2024. URL https://arxiv.org/abs/2404.14469. Loshchilov and Hutter [2019] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL https://arxiv.org/abs/1711.05101. Lu et al. [2025] Enzhe Lu, Zhejun Jiang, Jingyuan Liu, Yulun Du, Tao Jiang, Chao Hong, et al. MoBA: Mixture of block attention for long-context LLMs. arXiv preprint arXiv:2502.13189, 2025. Madaan et al. [2023] Lovish Madaan, Srinadh Bhojanapalli, Himanshu Jain, and Prateek Jain. Treeformer: Dense gradient trees for efficient attention computation. In International Conference on Learning Representations, 2023. URL https://arxiv.org/abs/2208.09015. Mohtashami and Jaggi [2023] Amirkeivan Mohtashami and Martin Jaggi. Random-access infinite context length for transformers. In Advances in Neural Information Processing Systems, 2023. URL https://arxiv.org/abs/2305.16300. Penedo et al. [2024] Guilherme Penedo, Hynek Kydlíček, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro von Werra, and Thomas Wolf. The FineWeb datasets: Decanting the web for the finest text data at scale. arXiv preprint arXiv:2406.17557, 2024. Peng et al. [2024] Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. YaRN: Efficient context window extension of large language models. In International Conference on Learning Representations (ICLR), 2024. Press et al. [2022] Ofir Press, Noah A. Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Representations (ICLR), 2022. Puvvada et al. [2025] Krishna C. Puvvada, Faisal Ladhak, Santiago Akle Serrano, Cheng-Ping Hsieh, Shantanu Acharya, Somshubra Majumdar, Fei Jia, Samuel Kriman, Simeng Sun, Dima Rekesh, and Boris Ginsburg. SWAN-GPT: An efficient and scalable approach for long-context language modeling, 2025. URL https://arxiv.org/abs/2504.08719. Shazeer [2020] Noam Shazeer. GLU variants improve transformer. arXiv preprint arXiv:2002.05202, 2020. URL https://arxiv.org/abs/2002.05202. Shazeer et al. [2017] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538, 2017. URL https://arxiv.org/abs/1701.06538. Su et al. [2021] Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. RoFormer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104.09864, 2021. Tang et al. [2024] Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. QUEST: Query-aware sparsity for efficient long-context LLM inference. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 47901–47911, 2024. URL https://proceedings.mlr.press/v235/tang24l.html. Touvron et al. [2023] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 30, 2017. Vasylenko et al. [2026] Pavlo Vasylenko, Hugo Pitorro, André F. T. Martins, and Marcos Treviso. Long-context generalization with sparse attention. In International Conference on Learning Representations, 2026. URL https://arxiv.org/abs/2506.16640. Wu et al. [2022] Yuhuai Wu, Markus N. Rabe, DeLesley Hutchins, and Christian Szegedy. Memorizing transformers. In International Conference on Learning Representations, 2022. URL https://arxiv.org/abs/2203.08913. Xiao et al. [2024a] Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, and Maosong Sun. InfLLM: Training-free long-context extrapolation for LLMs with an efficient context memory. In Advances in Neural Information Processing Systems, 2024a. URL https://arxiv.org/abs/2402.04617. Xiao et al. [2024b] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In International Conference on Learning Representations (ICLR), 2024b. Yang et al. [2025a] Bowen Yang, Bharat Venkitesh, Dwarak Talupuru, Hangyu Lin, David Cairuz, Phil Blunsom, and Acyr Locatelli. RoPE to NoPE and back again: A new hybrid attention strategy, 2025a. URL https://arxiv.org/abs/2501.18795. Yang et al. [2025b] Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule. In International Conference on Learning Representations, 2025b. Yuan et al. [2025] Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, et al. Native sparse attention: Hardware-aligned and natively trainable sparse attention. arXiv preprint arXiv:2502.11089, 2025. Zhang and Sennrich [2019] Biao Zhang and Rico Sennrich. Root mean square layer normalization. In Advances in Neural Information Processing Systems, volume 32, 2019. Zhang et al. [2024] Yu Zhang, Songlin Yang, Ruijie Zhu, Yue Zhang, Leyang Cui, Yiqiao Wang, Bolun Wang, Freda Shi, Bailin Wang, Wei Bi, Peng Zhou, and Guohong Fu. Gated slot attention for efficient linear-time sequence modeling. In Advances in Neural Information Processing Systems, 2024. URL https://arxiv.org/abs/2409.07146. Zhang et al. [2023] Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen. H2O: Heavy-hitter oracle for efficient generative inference of large language models, 2023. URL https://arxiv.org/abs/2306.14048. Appendix A Hyperparameters and experimental settings Architecture. FreeSolo uses LLaMA-style decoders with RMSNorm, SwiGLU, and a 3232k LLaMA tokenizer. Both scales have 2424 layers and 1616 heads. The 341341M model has width 10241024 and head width 6464; the 743743M model has width 15361536 and head width 9696. Attention uses w=512w=512, B=128B=128, s=32s=32, K=4K=4, σh=4 _h=4, and λh=1 _h=1. FS-L retains P=64P=64 norm-ranked cells and W=8W=8 recent cells. Pretraining. We train at 40964096 tokens in bf16 with AdamW, cosine learning-rate decay, weight decay 0.10.1, gradient clipping at 1.01.0, and 524,288524,288 tokens per step. The 341341M FS run and FS-L from-scratch control use peak learning rate 3×10−43× 10^-4, floor 3×10−53× 10^-5, and 10001000 warmup steps. The 743743M FS run uses 4×10−44× 10^-4, 4×10−54× 10^-5, and 20002000 warmup steps. The base FS checkpoints are at 28,00028,000 steps for 341341M and 60,00060,000 steps for 743743M, corresponding to 14,680,064,00014,680,064,000 and 31,457,280,00031,457,280,000 training tokens. We refer to these scales by the rounded 1515B/3232B budgets. The smaller run makes about 2.22.2 passes over a 6.76.7B-token FineWeb-Edu subset; the larger run uses less than one pass over a 3939B-token corpus. Continued pretraining. FS-L starts from FS with a fresh AdamW optimizer and pruning enabled. The learning rate warms up to 4×10−44× 10^-4 and decays toward 4×10−54× 10^-5. At 341341M, the schedule warms up for 150150 steps with a 15001500-step cosine horizon, and we use the step-14001400 checkpoint, selected on S-NIAH-1 (100100 examples); the 15001500-step continuations in Appendix D.4 show the run-to-run range. At 743743M, it warms up for 300300 steps and we use the final step-30003000 checkpoint. These correspond to 734,003,200734,003,200 and 1,572,864,0001,572,864,000 additional tokens. Both use the same 40964096-token context, tokenizer, and corpus as FS; restarting the data order revisits pretraining tokens. RULER evaluation. We use NVIDIA RULER at commit c3f5e3b4, the LLaMA tokenizer, base prompt template, seed 4242, and 100100 examples per task and length. Generation caps are 128128 tokens for NIAH, 3030 for variable tracking, 120120 for common-word extraction, 5050 for frequent-word extraction, and 3232 for QA. Decoding is greedy with EOS stopping. Non-QA tasks use string_match_all and QA uses string_match_part, on a 00–100100 scale. FS-L runs its bounded cache (P=64P=64, W=8W=8) with CUDA-graph decoding, FP32 parameters and BF16 autocast on A100 GPUs with 4040 GB memory. MoBA and NSA recompute the full forward pass for each generated token. Lengths 22k through 6464k denote 20482048 through 65,53665,536 tokens; table labels 131131k and 256256k denote 131,072131,072 and 262,144262,144 tokens. Downstream evaluation. The eight zero-shot tasks are HellaSwag, PIQA, ARC-easy, ARC-challenge, WinoGrande, OpenBookQA, SciQ, and BoolQ, evaluated with the LM Evaluation Harness. Baselines. At 340340M, NAtS-L uses 2121 layers and 6464-token routing blocks. Table 3 lists the evaluated pretraining checkpoint of every model; all runs use 40964096-token sequences. Raven is standalone at both scales, with 2121 layers, four heads, 256256 memory slots, top-3232 writes, and gate_logit_normalizer=8=8; its 726726M model has width 15361536. Raven-full is the variant with full NoPE attention (Section 4.1). TF++ uses full RoPE attention, SWA a 512512-token RoPE window, and InfLLM the TF++ checkpoint. NSA uses the paper’s configuration (compression blocks of 3232 tokens with stride 1616, 1616 selected 6464-token blocks, a 512512-token window, and four key/value heads). Its larger parameter counts come from the key and value token-compression MLPs, which hold 208208M parameters at 340340M and 467467M at 800800M; NSA is 172172M and 385385M larger than the matching MoBA or FreeSolo backbone. For 800800M GDN, RULER uses fla 0.5.20.5.2 and transformers 5.14.15.14.1; downstream tasks use versions 0.3.20.3.2 and 4.56.14.56.1, respectively, with LM Evaluation Harness 0.4.130.4.13. Table 3: Evaluated pretraining checkpoints. Steps and training tokens at 524,288524,288 tokens per step. MoBA is trained only at the smaller scale. FS-L and FS-B continue from the FS checkpoints: FS-L adds 0.730.73B (340340M) and 1.571.57B (800800M) CPT tokens; FS-B uses the FS-L weights at 340340M and 0.420.42B retrieval-weighted CPT tokens at 800800M (Appendix E.4). 340340M 800800M Model Steps Tokens Steps Tokens FS 28,00028,000 14.6814.68B 60,00060,000 31.4631.46B TF++ 28,61028,610 15.0015.00B 60,00060,000 31.4631.46B SWA 28,00028,000 14.6814.68B 60,00060,000 31.4631.46B MoBA 28,00028,000 14.6814.68B – – NSA 28,00028,000 14.6814.68B 57,50057,500 30.1530.15B GDN 28,61028,610 15.0015.00B 61,03561,035 32.0032.00B NAtS-L 28,61028,610 15.0015.00B 61,03561,035 32.0032.00B Raven 28,61028,610 15.0015.00B 61,03561,035 32.0032.00B Raven-full 28,61028,610 15.0015.00B 61,03561,035 32.0032.00B Appendix B Full RULER task breakdown Table 4 reports individual task scores. FS and FS-L improve long-context retrieval, while multi-key retrieval, variable tracking, and word aggregation remain difficult. Raven retains strong S-NIAH-1 recall at 800800M, while FS-L performs better on S-NIAH-2/3 and the suite mean. FS-B has the highest super-linear suite means from 44k on at both scales (Table 1) but remains below Raven-full on 800800M MQ- and MV-NIAH. MK, MQ, and MV denote multi-key, multi-query, and multi-value NIAH; VT denotes variable tracking; CWE and FWE denote common- and frequent-word extraction. QA-2 is unavailable at 22k/44k, so the mean covers 1212 tasks there and 1313 at 88k/1616k; ties share a mark. NAtS-L’s linear-time grouping assumes a bounded retained-token count (Section 4.1). Table 4: RULER results by task (00–100100) at 340340M and 800800M. Bold and underline mark the best and second-best scores within each group (super-linear or linear cost) and scale. Dashes denote unavailable results. Super-linear Linear Raven-full FS FS-B FS-L NAtS-L Raven Task 22k 44k 88k 1616k 22k 44k 88k 1616k 22k 44k 88k 1616k 22k 44k 88k 1616k 22k 44k 88k 1616k 22k 44k 88k 1616k ≈ 340M scale S-NIAH-1 100 99 8383 3939 99¯ 99 99 100 100 100 99 99¯ 99 99¯ 99 99¯ 99 8888 90¯ 90 87 100 94¯ 94 3737 1919 9595 96 93 81¯ 81 S-NIAH-2 100 89¯ 89 41¯ 41 77 100 7777 41¯ 41 17¯ 17 100 99 89 60 99 70 34¯ 34 20 8989 22¯ 22 2020 3¯ 3 92¯ 92 70 36 3¯ 3 S-NIAH-3 3838 11 00 0¯ 0 58¯ 58 14¯ 14 7¯ 7 0¯ 0 91 78 64 27 70 17 9 1 36¯ 36 6¯ 6 1¯ 1 0¯ 0 33 00 00 0¯ 0 MK-NIAH-1 25¯ 25 22¯ 22 18¯ 18 18¯ 18 27 2020 1515 55 27 25 19 24 31 24 21 10 17¯ 17 18¯ 18 10¯ 10 4¯ 4 1111 1111 10¯ 10 33 MK-NIAH-2 00 1 0 0¯ 0 1¯ 1 0¯ 0 0 0¯ 0 2 0¯ 0 0 1 0 1 0 0 0 0¯ 0 0 0 0 0¯ 0 0 0 MK-NIAH-3 00 0 0 0¯ 0 1¯ 1 0 0 1 2 0 0 0¯ 0 0 0 0 0 0 0 0 0 0 0 0 0 MQ-NIAH 20.8¯ 20.8 13.313.3 6.86.8 5.0¯ 5.0 20.220.2 20.8¯ 20.8 12.8¯ 12.8 2.82.8 23.823.8 22.522.5 18.518.5 12.512.5 21.821.8 21.521.5 10.5¯ 10.5 3.83.8 17.5¯ 17.5 15.215.2 4.84.8 2.82.8 15.015.0 16.5¯ 16.5 13.213.2 3.0¯ 3.0 MV-NIAH 19.319.3 17.817.8 6.86.8 6.56.5 23.2¯ 23.2 18.5¯ 18.5 15.5¯ 15.5 9.8¯ 9.8 24.524.5 22.022.0 23.523.5 21.021.0 24.324.3 19.819.8 16.316.3 7.37.3 18.8¯ 18.8 8.88.8 4.54.5 7.0¯ 7.0 11.211.2 12.0¯ 12.0 12.0¯ 12.0 4.54.5 VT 0.80.8 1.41.4 0.40.4 0.20.2 2.2¯ 2.2 1.2¯ 1.2 0.6¯ 0.6 1.61.6 3.23.2 1.2¯ 1.2 1.01.0 1.0¯ 1.0 2.8¯ 2.8 0.00.0 0.0¯ 0.0 0.0¯ 0.0 4.84.8 0.6¯ 0.6 0.0¯ 0.0 0.0¯ 0.0 2.62.6 2.02.0 1.41.4 1.01.0 CWE 0.20.2 0.60.6 0.00.0 0.10.1 4.1¯ 4.1 9.2¯ 9.2 5.2¯ 5.2 0.8¯ 0.8 5.25.2 10.310.3 6.56.5 1.81.8 4.94.9 9.19.1 4.04.0 0.40.4 2.6¯ 2.6 5.0¯ 5.0 0.2¯ 0.2 0.3¯ 0.3 1.01.0 0.20.2 0.2¯ 0.2 0.10.1 FWE 0.00.0 0.00.0 0.00.0 0.00.0 8.7¯ 8.7 3.3¯ 3.3 5.0¯ 5.0 3.3¯ 3.3 9.09.0 6.06.0 11.711.7 5.05.0 9.79.7 1.0¯ 1.0 2.32.3 2.72.7 3.0¯ 3.0 1.71.7 0.3¯ 0.3 2.3¯ 2.3 0.00.0 0.00.0 0.00.0 0.00.0 QA-1 1616 13¯ 13 9 7 25 15 5¯ 5 7 23¯ 23 1010 44 4¯ 4 26 8¯ 8 4¯ 4 33 19¯ 19 12 9 4¯ 4 1515 66 4¯ 4 6 QA-2 – – 10 10 – – 88 55 – – 9¯ 9 6¯ 6 – – 10¯ 10 44 – – 10¯ 10 10 – – 12 9¯ 9 Available-task mean 26.726.7 21.521.5 13.513.5 7.17.1 30.8¯ 30.8 23.2¯ 23.2 16.5¯ 16.5 11.8¯ 11.8 34.234.2 31.131.1 26.626.6 20.220.2 32.432.4 21.621.6 15.515.5 10.710.7 25.6¯ 25.6 15.315.3 7.47.4 4.04.0 20.520.5 17.8¯ 17.8 14.0¯ 14.0 8.5¯ 8.5 ≈ 800M scale S-NIAH-1 100 100 100 100 97¯ 97 9191 9393 95¯ 95 97¯ 97 98¯ 98 97¯ 97 100 100 92¯ 92 84¯ 84 84¯ 84 86¯ 86 7272 5959 2121 100 100 100 100 S-NIAH-2 100 8888 8181 6969 100 99¯ 99 92¯ 92 72¯ 72 100 100 95 88 99¯ 99 99 89 81 100 7676 2828 33 99¯ 99 97¯ 97 81¯ 81 26¯ 26 S-NIAH-3 91¯ 91 5858 2121 44 8585 71¯ 71 51¯ 51 34¯ 34 96 93 87 84 90 71 55 40 62¯ 62 24¯ 24 6¯ 6 0¯ 0 3232 77 11 0¯ 0 MK-NIAH-1 3737 36 29¯ 29 32 53 3232 30 2828 50¯ 50 34¯ 34 2828 31¯ 31 43 32 27 23 31¯ 31 20¯ 20 77 3¯ 3 3030 1919 14¯ 14 3¯ 3 MK-NIAH-2 33 1¯ 1 0¯ 0 00 7¯ 7 00 1 1¯ 1 9 4 1 2 8 3 1 1 00 0¯ 0 0¯ 0 0¯ 0 3¯ 3 0¯ 0 0¯ 0 0¯ 0 MK-NIAH-3 3 2 0 0 00 00 0 0 2¯ 2 1¯ 1 0 0 2 0 1 0 0¯ 0 0 0¯ 0 0 0¯ 0 0 0¯ 0 0 MQ-NIAH 69.569.5 46.546.5 44.044.0 38.338.3 34.534.5 25.825.8 22.822.8 23.023.0 45.8¯ 45.8 28.3¯ 28.3 28.8¯ 28.8 27.3¯ 27.3 26.2¯ 26.2 22.522.5 16.016.0 15.515.5 16.516.5 11.511.5 1.51.5 0.30.3 27.827.8 19.8¯ 19.8 11.0¯ 11.0 7.2¯ 7.2 MV-NIAH 56.356.3 41.341.3 39.039.0 39.039.0 28.028.0 26.326.3 22.822.8 18.818.8 38.3¯ 38.3 30.3¯ 30.3 29.8¯ 29.8 26.3¯ 26.3 26.226.2 20.820.8 21.021.0 17.017.0 13.513.5 10.810.8 3.03.0 0.50.5 25.8¯ 25.8 17.0¯ 17.0 13.5¯ 13.5 9.0¯ 9.0 VT 0.60.6 0.2¯ 0.2 0.6¯ 0.6 0.60.6 1.6¯ 1.6 0.2¯ 0.2 0.40.4 0.0¯ 0.0 3.83.8 1.41.4 1.01.0 0.60.6 3.83.8 0.80.8 0.4¯ 0.4 0.6¯ 0.6 0.00.0 0.2¯ 0.2 0.00.0 0.00.0 1.6¯ 1.6 0.80.8 0.60.6 1.01.0 CWE 3.33.3 9.39.3 5.45.4 1.21.2 8.48.4 26.726.7 13.513.5 3.3¯ 3.3 7.8¯ 7.8 26.4¯ 26.4 8.9¯ 8.9 4.44.4 4.54.5 13.013.0 7.17.1 1.21.2 0.40.4 0.70.7 0.00.0 0.00.0 1.9¯ 1.9 1.3¯ 1.3 0.1¯ 0.1 0.1¯ 0.1 FWE 1.71.7 1.71.7 0.00.0 0.00.0 16.7¯ 16.7 6.0¯ 6.0 5.0¯ 5.0 5.7¯ 5.7 17.717.7 8.78.7 7.07.0 8.08.0 0.30.3 0.00.0 1.7¯ 1.7 0.0¯ 0.0 1.7¯ 1.7 3.3¯ 3.3 4.74.7 0.0¯ 0.0 18.018.0 11.011.0 1.31.3 3.73.7 QA-1 3232 2020 1010 12 42 30 12¯ 12 99 38¯ 38 27¯ 27 13 10¯ 10 43 30 11 11 23¯ 23 14¯ 14 11 00 2222 14¯ 14 6¯ 6 5¯ 5 QA-2 – – 15¯ 15 13¯ 13 – – 15¯ 15 1111 – – 17 14 – – 19 16 – – 12¯ 12 99 – – 1111 11¯ 11 Available-task mean 41.4¯ 41.4 33.733.7 26.526.5 23.8¯ 23.8 39.439.4 34.0¯ 34.0 27.6¯ 27.6 23.123.1 42.142.1 37.737.7 31.831.8 30.430.4 37.237.2 32.032.0 25.625.6 22.322.3 27.827.8 19.419.4 10.210.2 2.82.8 30.1¯ 30.1 23.9¯ 23.9 18.4¯ 18.4 12.8¯ 12.8 Appendix C Efficiency benchmarks The streaming implementation maintains fixed attention buffers and nearly constant decode throughput from 44k to 256256k. Absolute throughput still lags optimized local and recurrent baselines. We measure training, prefill, and decoding separately. These benchmarks use the models’ available optimized kernels; the results depend on batch size, software stack, and execution mode as well as architecture. Training. Table 5 reports aggregate forward-and-backward throughput on eight A100-40GB GPUs, with 40964096-token sequences, bf16, and torch.compile. All rows use batch 11 per GPU. FS-B changes inference only: it uses the FS-L weights at 340340M, and its CPT at 800800M trains with FS selection, so it has no separate row. Raven and NAtS-L are measured with the fla/flame (FSDP) harness (sharded parameters and optimizer states), so their peak memory is not directly comparable and is omitted, and their throughput carries some FSDP communication overhead. Bold and underline mark the best and second-best results (higher throughput, lower memory). GDN uses flash-linear-attention, NSA uses its Triton kernel, MoBA uses SDPA, and FreeSolo uses fused indexed-cell attention. FS, TF++, and SWA run on PyTorch 2.52.5; the other models use PyTorch 2.92.9. Sparse long-range retrieval reduces scaling cost, but has overhead at the short training context. Table 5: Training throughput and peak memory at 340340M scale on eight A100-40GB GPUs. Raven and NAtS-L use the fla/flame FSDP harness; see the text. Model Train tok/s Peak mem (GiB) TF++ (full attention) 303K 15.015.0 SWA (sliding-512512) 228¯ 228K 15.5¯ 15.5 GDN 158158K 21.621.6 MoBA 140140K 27.027.0 NSA 2828K 31.931.9 Raven 140140K – NAtS-L 126126K – FS 99.399.3K 16.516.5 FS-L 98.498.4K 16.616.6 Inference. Table 6 uses one A100-40GB, bf16, and batch 11. Prefill is one forward pass over the prompt; decode processes one new token using each model’s KV cache or recurrent state. All table entries use eager execution, without torch.compile or CUDA-graph capture. TF++ uses FlashAttention-2 [Dao, 2023]; FreeSolo uses fused indexed-cell prefill and a fused sliding-window local branch. The other baselines use their native attention or recurrent kernels, with an incremental reference path for NSA decoding. FS-L prefill stays nearly flat from 44k to 131131k, where FS runs out of memory, while several baselines decode faster. FS-B runs to 131131k within 1919 GiB, but the measured reference uses unfused forest construction, traversal and reranking and repeatedly copies growing cache buffers (Appendix E.2). Its eager throughput (1.11.1–1.51.5K tokens/s prefill, 5.35.3–6.16.1 tokens/s decode) reflects this implementation rather than the O⁡(Nlog⁡N)O(N N) search; with CUDA-graph decode it reaches 19.119.1–19.819.8 tokens/s. The benefit is sustained long-context retrieval with bounded query scoring, rather than the highest throughput at every length. Table 6 uses an FS-L implementation that stores all KV and rescans priorities (Section 3.5); Table 7 measures the streaming cache. Table 6: Prefill and eager decode throughput at 340340M scale on one A100-40GB (bf16, batch 11). OOM denotes the 4040 GB memory limit. Prefill (Ktok/s) Decode (tok/s) Model 44k 1616k 6464k 131131k 44k 1616k 6464k 131131k TF++ (full attention) 144.4144.4 123.8123.8 48.848.8 26.226.2 34.734.7 34.734.7 34.734.7 20.520.5 SWA (sliding-512512) 143.5143.5 218.8218.8 248.6248.6 245.3245.3 33.133.1 33.233.2 33.233.2 33.333.3 MoBA 34.334.3 48.648.6 33.333.3 20.220.2 27.427.4 26.926.9 26.126.1 OOM NSA 87.087.0 121.4121.4 89.889.8 65.265.2 15.815.8 15.715.7 15.815.8 15.815.8 GDN 65.765.7 141.8141.8 146.8146.8 146.2146.2 25.525.5 25.725.7 25.625.6 25.425.4 NAtS-L 56.156.1 132.6132.6 107.6107.6 83.783.7 22.522.5 22.722.7 22.222.2 22.922.9 Raven 71.371.3 137.7137.7 147.6147.6 148.9148.9 24.724.7 24.824.8 24.824.8 24.824.8 FS 30.930.9 22.322.3 10.910.9 OOM 19.019.0 18.718.7 18.218.2 OOM FS-L 34.634.6 43.643.6 44.244.2 39.639.6 19.819.8 19.919.9 19.819.8 19.819.8 FS-B 1.51.5 1.41.4 1.21.2 1.11.1 6.16.1 5.95.9 5.65.6 5.35.3 Compilation and CUDA graphs. With torch.compile, FS-L prefill reaches 62.862.8–68.268.2K tokens/s from 44k to 131131k, and CUDA-graph capture raises its decode to 94.494.4–94.794.7 tokens/s through 6464k; with the same capture, SWA, GDN, and Raven decode at about 380380, 275275, and 262262 tokens/s. Bounded cache. The streaming implementation updates FS-L’s top-P pool exactly from newly eligible cells. Let C be the eligible prefix cells and D those newly eligible at the next query-block boundary. Fixed priorities aca_c and a common deterministic tie rule give TopP⁡(∪)=TopP⁡(TopP⁡()∪).TopP(C )=TopP\! (TopP(C) ). (15) A discarded cell already has at least P cells ahead of it, so adding cells cannot improve its rank. Once it also leaves the recent queue and local window, its KV can be evicted. The implementation keeps absolute cell indices for relative scores and computes RoPE locally; position counters require O⁡(log⁡N)O( N) bits. Prefill processes the transformer in chunks of 20482048 tokens, with retention updates every B=128B=128 tokens. Decode uses a fixed local ring and compact cell buffers; pool updates occur outside the CUDA graph and are included in the timing. Table 7 uses both CPT checkpoints, bf16 parameters and activations, and FP32 rotary frequencies on one A100-40GB. We report the median of three runs after one warmup, with 257257 decode steps per run. Attention-buffer bytes count distinct physical allocations across all heads and layers, including retained, recent, local, pending, and graph input buffers. The implementation keeps duplicate compact buffers, so this exceeds the size of one packed retained set. Peak allocated GPU memory also includes parameters, activations, and graph workspace; input/output token IDs are excluded from the attention-buffer count. Table 7: Bounded FS-L inference. Throughput uses CUDA-graph decode; peak memory is the maximum across three trials. Length Attention buffers (MiB) Peak (GiB) Prefill (Ktok/s) Decode (tok/s) 340340M scale 44k 766.2766.2 1.711.71 2.202.20 89.389.3 6464k 766.2766.2 1.741.74 2.212.21 89.489.4 256256k 766.2766.2 1.771.77 2.172.17 88.388.3 800800M scale 44k 1148.81148.8 2.962.96 2.092.09 75.275.2 6464k 1148.81148.8 2.982.98 2.142.14 75.175.1 256256k 1148.81148.8 3.023.02 2.142.14 74.174.1 This establishes bounded attention storage with linear streaming work, at a substantial prefill cost relative to Table 6. Both implementations apply the same pruning rule, but precision and reduction order can change generated answers. On 2525 prompts per S-NIAH task at 44k and 1616k, bounded execution at 340340M matches all 150150 correctness decisions of the masked implementation (FP32 parameters, bf16 autocast). The bf16 CUDA-graph configuration used for timing matches 60/6060/60 decisions at 340340M and 59/6059/60 at 800800M on ten prompts per task and length, and 58/6058/60 for the adapted P=16P=16 and P=32P=32 checkpoints; both scales retrieve all three tested 256256k S-NIAH-1 examples. FS-L results in Tables 1 and 4 use the bounded cache on 100100 examples per cell, with FP32 parameters and bf16 autocast (Appendix A). The FS-L continuation and adapted-budget accuracies in Appendices D.4 and D.5 also use bounded execution. The routing probes and eligible-prompt answer counts in Appendix D.3 use the full-history masked implementation, which applies the same pruning rule without evicting stored KV. Appendix D Theory and ablation studies In this section, we present additional theoretical analysis, ablation studies, and exploratory tests and visualizations (tagged Probe). The main findings are: Ablations of short-range and long-range components (Appendix D.1). 1. (Ablation) Short-range RoPE is crucial for commonsense accuracy; single-needle length extrapolation persists without it. 2. (Theory, Probe) The long-range erf term acts mainly as a key-norm bonus; its residual distance dependence rarely changes selection. Cell summaries and reading distance (Appendix D.2). 1. (Theory, Probe) Cell summaries need only locate useful evidence: the readout accesses the selected tokens’ original values directly, including needle reads over 166166k tokens away in the traced prompt. 2. (Theory) Sparse readout limits interference from unrelated values; readout normalization alone cannot undo their mixing. Norm pruning and retrieval coverage (Appendix D.3). 1. (Probe) Norm pruning substantially reduces query-scoring work: FS-L retains about one-seventh of eligible cells at 1616k. 2. (Probe) On harder retrieval tasks, query-dependent selection remains a bottleneck even when the answer source survives pruning. Continued pretraining (Appendix D.4). 1. (Ablation) Short CPT helps FS-L adapt to the pruning constraint, recovering much of the S-NIAH-1 accuracy lost when pruning is applied without adaptation. 2. (Ablation) Harder-task changes cannot be attributed to pruning alone: they vary across seeds and also occur with unpruned continuation. 3. (Ablation) Retrieval-weighted CPT improves the RULER mean of the full-prefix single-forest ablation (W32) at every evaluated length using only short-context training. Retention budgets and multi-fact retrieval (Appendix D.5). 1. (Theory) Sustaining high accuracy when retrieving independently queryable random facts requires storage to grow with their number. 2. (Ablation) Adapted models do not improve monotonically with retention capacity; a larger pool is not sufficient for better retrieval. 3. (Probe) Multi-fact failures extend beyond pruning: unpruned and full-attention models also confuse values from the same prefix. D.1 Ablations of short-range and long-range components The ablations separate short-context quality from length extrapolation: changing the local branch sharply affects downstream accuracy, while distant retrieval persists. In long-range routing, the erf term acts mainly as a norm bonus at s=32s=32 and σh=4 _h=4. Position dependence. Local scores use i⊤(j−i)j/dq_i R(j-i)k_j/ d with 0≤i−j≤w0≤ i-j≤ w. Training at length L>wL>w covers every explicit rotary offset used at evaluation. Long-range token scores are NoPE. The cell prior gh(i−pc)g_h(i-p_c) is unchanged by shifting query and cell coordinates together, and approaches one with distance. At fixed content vectors and a consistent tie rule, translating tokens, cell partitions, query blocks, and causal boundaries together by a multiple of B leaves the operator unchanged. These properties avoid new rotary offsets; they do not make contextual representations invariant to added tokens. Trained ablations. We train both controls with the 341341M FS recipe, 44k context, and 1515B training tokens. SR-NoPE (short range without RoPE) replaces local RoPE with unrotated queries and keys and adds a log⁡erfc⁡(Δ/(2σh)) ( /(2 _h)) bias with Δ=i−j =i-j, retaining the 512512-token window; it therefore changes both rotation and local bias. LR-content-only keeps local RoPE and removes the entire λhacerf⁡((i−pc)/(2σh)) _ha_c ((i-p_c)/(2 _h)) routing term, so long-range routing uses the query–mean-key similarity alone. Table 8: RULER ablations at 340340M (00–100100). Bold and underline mark the best and second-best scores, including ties. S-NIAH-1 S-NIAH-2 S-NIAH-3 Model 2k 4k 8k 16k 32k 64k 131k 2k 4k 8k 16k 2k 4k 8k 16k FS 99¯ 99 99¯ 99 100 100 98¯ 98 100 98¯ 98 100 77¯ 77 41¯ 41 17¯ 17 58¯ 58 1414 77 0¯ 0 SR-NoPE 100 100 100 100 100 100 100 98¯ 98 90 71 29 4747 23¯ 23 13 0¯ 0 LR-content-only 9595 9797 96¯ 96 97¯ 97 9595 95¯ 95 9696 8080 5757 3838 1414 71 25 11¯ 11 1 Table 9: Downstream accuracy of the ablations at 340340M (00–100100), using the metrics in Table 2. Model HellaS. PIQA ARC-e ARC-c WinoG. OBQA SciQ BoolQ Avg FS 38.9¯ 38.9 66.866.8 57.657.6 25.9¯ 25.9 50.9¯ 50.9 32.432.4 81.2¯ 81.2 61.161.1 51.9¯ 51.9 SR-NoPE 28.428.4 55.755.7 36.936.9 23.723.7 49.249.2 25.625.6 48.248.2 38.738.7 38.338.3 LR-content-only 39.339.3 66.1¯ 66.1 57.1¯ 57.1 26.926.9 52.852.8 31.8¯ 31.8 83.283.2 60.0¯ 60.0 52.152.1 SR-NoPE preserves S-NIAH-1 extrapolation and improves S-NIAH-2, but lowers average downstream accuracy from 51.951.9 to 38.338.3. The local FS configuration therefore improves language-model quality without being required for single-needle retrieval. LR-content-only retains 9595–9797 on S-NIAH-1 and a downstream average of 52.152.1, while worsening S-NIAH-2 and improving S-NIAH-3. The norm bonus is therefore not needed for single-needle extrapolation; it shifts S-NIAH-2 and S-NIAH-3 in opposite directions. A saturated norm prior. Every eligible cell satisfies (c+1)s≤bi≤i(c+1)s≤ b_i≤ i, so its center pc=(c+12)sp_c=(c+ 12)s obeys i−pc≥s/2i-p_c≥ s/2. Write the content term as κic=i⊤¯c/d _ic=q_i k_c/ d. For λh≥0 _h≥ 0, define the constant-gate score ric∞=κic+λhacr^∞_ic= _ic+ _ha_c. Then 0≤ric∞−ric=λhacerfc⁡(i−pc2σh)≤λhacerfc⁡(s4σh).0≤ r^∞_ic-r_ic= _ha_c\,erfc\! ( i-p_c2 _h )≤ _ha_c\,erfc\! ( s4 _h ). (16) At s=32s=32 and σh=4 _h=4, erfc⁡(2)≃0.004678erfc(2) 0.004678: the difference is at most 0.468%0.468\% of the norm bonus. This is not a relative bound on the total score, which can include cancellation with the content term. Larger offsets further reduce the difference. For a query i and a fixed available set with more than K cells, let γ∞ _∞ be the gap between the KKth and (K+1)(K+1)st constant-gate scores and δmax=maxc⁡λhacerfc⁡((i−pc)/(2σh)) _ = _c _ha_cerfc((i-p_c)/(2 _h)). If γ∞>δmax _∞> _ , replacing the constant gate by ghg_h leaves the top-K set unchanged: a selected score falls by at most δmax _ , while an unselected score cannot increase. The raw-token readout is then unchanged too. Equivalently, routing can change only when γ∞≤δmax _∞≤ _ . It applies to FS’s full prefix and FS-L’s retained set at fixed representations. Fixed-state routing controls. On the 1616k native traces described in Appendix D.3, replacing ghg_h by one keeps 99.998%99.998\% of the native top-K cells in the top-K set in both 340340M models. Removing the entire bonus instead preserves only 53.9/14.0/14.7%53.9/14.0/14.7\% in FS and 73.3/32.5/32.7%73.3/32.5/32.7\% in FS-L on S-NIAH-1/2/3. The norm bonus changes routing substantially, whereas its residual distance dependence rarely changes the selected set. D.2 Cell summaries and reading distance FS imposes a bottleneck on cell selection: a query selects cells using mean keys and maximum key norms, then reads the selected cells’ original keys and values. The routing summary need only locate useful evidence; it need not contain the answer. Sparse readout also bounds how many token values can interfere, independently of their distance from the query. A keypoint can mark a cell. If a cell contains one nonzero key ⋆k_ , its mean is ⋆/sk_ /s, while ac=‖⋆‖2a_c=\|k_ \|_2. The content match is diluted by 1/s1/s, but the norm bonus preserves the cue’s magnitude. One key can thus make a cell visible to the router. A high-norm distractor receives the same bonus, so query alignment still decides which cells are read. Why sparse readout matters after RMSNorm. Let =∑j∈ieℓijjy= _j _ie _ijv_j, Y=∑j∈ieℓijY= _j _ie _ij, and g be the readout RMSNorm gain. Then ilr=⊙‖22/d+ϵY2.o_i lr= g \|y\|_2^2/d+ε Y^2. (17) When ‖/Y‖22/d≫ϵ\|y/Y\|_2^2/d ε (ϵ=10−8ε=10^-8), the scalar softmax denominator cancels. Normalization therefore removes amplitude dilution, but irrelevant values can still alter the output direction. Sparse selection restricts which values enter this sum, rather than merely reducing its softmax denominator. This distinction matters when relating FreeSolo to analyses of attention dilution and sparse length generalization [Chiang and Cholak, 2022, Vasylenko et al., 2026]. Reading the needle. Figure 4 traces 340340M FS and FS-L while answering the same 256256k S-NIAH-1 prompt. Both generate the correct value, 23386872338687, from tokens 95,39495,394–95,40095,400 in a 262,010262,010-token input. For a direct comparison, we display the same head in both models: layer 1010, head 1515, the head that maximizes the smaller of the two models’ mean needle attention during the first answer. These are selected retrieval examples, not averages across heads or prompts. The upper plots show the four cells read when predicting the first value token. A stem’s height is the cell’s total long-range softmax weight; orange marks the cell containing the value. The letters link each read to its decoded 3232-token text, including unrelated cells. Each row gives the routing score ricr_ic from Eq. (7) for the models that selected the cell. These scores include the content term and erf prior; they are distinct from the attention weights plotted above. The lower plot follows attention to the exact source value as each answer token is predicted, excluding other tokens in its cell. In this example, the needle cell receives 89%/96%89\%/96\% of the displayed head’s long-range softmax weight in FS/FS-L at the first value token. Both heads continue to read the distant value during generation. These weights precede readout RMSNorm and branch mixing; they do not establish a causal contribution to the answer. Figure 4: Retrieving a number at 256256k (340340M). Selected cells, routing scores, prompt text, and attention during generation. Both answers are correct. Read distance across the prompt. Unlike Raven, which measures memory by the tokens written to recurrent slots [Afzal et al., 2026, Definition 4 and Figure 8], we measure where queries read. Selecting cell c at query i gives backward distance i−pci-p_c tokens. Figure 5 plots this distance against query position for the 800800M checkpoints on the same 256256k prompt. Both axes use tokens (11k=1024\;=1024) and 20482048-token bins. Color shows the fraction of cell selections per query bin, aggregated over the layer’s 1616 heads, on a logarithmic scale. White denotes zero selections; gray lies beyond the available prefix. The needle occupies cell 29812981, centered at p⋆=95,408p_ =95,408. The orange dotted line marks its position; the dashed diagonal follows its distance i−p⋆i-p_ from later queries. Diagonal bands indicate repeated access to fixed earlier cells. Both checkpoints read the needle cell in layer 11. At the final prompt query, 166,601166,601 tokens from its center, layer 2424 selects it in 6/166/16 FS heads and 2/162/16 FS-L heads. Over the final 20482048 queries, averaging all layers and heads, 93.6%93.6\% of FS selections and 93.1%93.1\% of FS-L selections lie at least 131,072131,072 tokens behind the query. These counts show direct access to distant cells, without measuring their attention weights. Figure 5: Long-range read distance at 800800M on one 256256k S-NIAH-1 prompt. Orange guides mark the needle position and its distance from each query. Retrieval on harder tasks. Figures 6 and 7 show selected successful S-NIAH-2/3 examples at 6464k/3232k with the same 800800M checkpoints. Each pair uses the same prompt and needle position; both models generate the correct value. The common-head rule above selects layer 1111, head 66 for both tasks. For S-NIAH-2, the needle cell near token 16,21716,217 receives 72%/66%72\%/66\% of this head’s long-range softmax weight in FS/FS-L at the first value token. For S-NIAH-3, the cell near token 7,5917,591 receives 47%/83%47\%/83\%. The displayed text shows the number or UUID prefix alongside the selected essay excerpts. The UUID spans two cells; only its first cell is read at the snapshot shown, and the next cell is selected later during generation. The lower plots follow attention to the full source value across the answer. Figure 6: Retrieving a number at 6464k (800800M). Selected cells, routing scores, prompt text, and attention to the S-NIAH-2 value. Both answers are correct. Figure 7: Retrieving a UUID at 3232k (800800M). Selected cells, routing scores, prompt text, and attention to the S-NIAH-3 value. Both answers are correct. D.3 Norm pruning and retrieval coverage We measure how much norm pruning reduces the available prefix and how it affects retrieval. Key norm is a learned pruning heuristic: a smaller-norm cell can still be relevant to a future query. We therefore evaluate source access, selection coverage, and answer accuracy separately. Separate retention from selection. A non-recent cell survives pruning when fewer than P cells outrank it by aca_c. It is then read only if fewer than K retained cells outrank it by ricr_ic. Raising K cannot recover a cell already pruned. At fixed scores, retaining every FS top-K cell preserves its selected-token readout, but partial overlap need not preserve the answer. Probe protocol. We trace the FS and FS-L checkpoints of Table 1 at both scales on 5050 examples per S-NIAH task at 22k/44k/88k/1616k, totaling 600600 prompts per model. Each trace covers all 2424 layers and 1616 heads on the final 1616 prompt positions. The forward pass uses each model’s native routing. Alternative scores are computed on those same representations without changing the forward pass. Source-access measurements exclude prompts whose answer source overlaps the local window of any observed query. We average head/query rates within each prompt, then average prompts; heads are not independent observations. Source availability and selection. Table 10 distinguishes source retention from query-dependent selection. FS keeps the full prefix, so its retention column applies the FS-L rule offline to FS keys. Its selection column uses native FS routing. FS-L’s columns both describe its native operation. These probes and the eligible-prompt answer counts below use the full-history masked implementation of FS-L, rather than the bounded cache used for the main accuracy tables. Table 10: Source-cell availability and selection at 1616k. The source is the cell containing the answer value. Rates average over all heads and the final 1616 prompt positions within each prompt, then over prompts; FS (offline) applies the FS-L retention rule to FS keys. Source retained (%) Source selected (%) Task Prompts FS (offline) FS-L FS FS-L 340340M scale S-NIAH-1 4747 22.3122.31 27.9827.98 38.1538.15 17.5517.55 S-NIAH-2 5050 18.3418.34 19.8719.87 1.271.27 1.301.30 S-NIAH-3 4747 24.6424.64 25.7625.76 2.492.49 2.652.65 800800M scale S-NIAH-1 4747 23.0723.07 22.0322.03 29.2129.21 14.9914.99 S-NIAH-2 5050 16.2216.22 17.0317.03 2.002.00 2.082.08 S-NIAH-3 4747 26.2926.29 26.4226.42 3.553.55 3.523.52 At 340340M, under the same norm rule, FS-L improves source availability by 5.66/1.54/1.115.66/1.54/1.11 percentage points on S-NIAH-1/2/3 (computed before rounding). Paired-prompt bootstrap 95%95\% intervals are [5.05,6.31][5.05,6.31], [1.14,1.93][1.14,1.93], and [0.72,1.49][0.72,1.49] (50005000 resamples). These compare the two checkpoints and do not separate pruning adaptation from additional training. Native source selection is substantially lower in FS-L on S-NIAH-1 and similar on the harder tasks. Among head/query pairs where the source survives, FS-L selects it in 63.8/6.6/10.3%63.8/6.6/10.3\% of cases, averaging the conditional rate within each prompt. Source access does not guarantee a correct answer: on the same eligible prompts, greedy answers are correct on 47/4747/47, 9/509/50, and 0/470/47 for FS and 40/4740/47, 9/509/50, and 0/470/47 for FS-L on S-NIAH-1/2/3 at 1616k. At 800800M, the corresponding source-availability changes are −1.04/0.82/0.13-1.04/0.82/0.13 percentage points, with paired 95%95\% intervals [−2.06,0.07][-2.06,0.07], [0.47,1.16][0.47,1.16], and [−0.35,0.64][-0.35,0.64]. The increase at 340340M therefore does not generalize uniformly across scales. Native S-NIAH-1 source selection is again lower in FS-L, while S-NIAH-2/3 selection changes little. FS-L selects a surviving source in 69.3/12.3/13.4%69.3/12.3/13.4\% of eligible head/query pairs, averaging within each prompt. On S-NIAH-2/3, FS selects the source in only 1.31.3–3.6%3.6\% of head/query pairs and FS-L selects a retained source in 6.66.6–13.4%13.4\%, so selection rather than retention limits the harder tasks. Compression. For each head, define the retained fraction ηi=|i|/|i| _i=|A_i|/|C_i|. At 44k/88k/1616k, native FS-L retains 58.8/29.0/14.3%58.8/29.0/14.3\% of eligible cells across these probes, corresponding to 1.7/3.4/7.0×1.7/3.4/7.0× reductions in cells available for scoring. All lengths use P=64P=64 and W=8W=8. These fractions concern cell counts; Appendix C measures physical storage. D.4 Continued pretraining CPT recovers much of the S-NIAH-1 accuracy lost under immediate pruning. Harder-task changes also occur with unpruned continuation and vary across seeds. These results support adapting an FS checkpoint to the pruning constraint through short CPT. Learning with hard indices. As in Section 3.3, top-P and top-K indices receive no loss gradient, and we use no auxiliary retention loss or straight-through estimator. A key receives no gradient through an access that excludes it, and the remaining gradients need not increase its norm; scores and norm ranks change only through the shared projections and contextual states. CPT starts from FS’s retrieval representations and keeps pruning in the loop at the same 40964096-token context. The constraint already binds during training: the last query block has 124124 eligible cells, exceeding P+W=72P+W=72. Optimization settings are in Appendix A. Continuation controls. Table 11 compares pruning and continuation at 340340M using the same first 5050 examples per task at 44k. Immediate pruning applies FS-L’s P=64P=64 retention rule without adaptation. The seeded controls start from the same FS checkpoint with a fresh optimizer, the same corpus and schedule, and 15001500 steps (0.7860.786B tokens); FS uses full-prefix routing and FS-L keeps pruning in training and evaluation. The main FS-L checkpoint uses 14001400 steps. We repeat seed 4242 with the same data order and run seed 4343 once; changing the seed also changes data sampling. Table 11: Pruning recovery and seed variation at 340340M. Accuracy (%) on 5050 shared prompts per task at 44k, the training context. FS-L is the 14001400-step checkpoint of Table 1; seeded rows use 15001500 CPT steps without (FS) or with (FS-L) pruning. Model S-NIAH-1 S-NIAH-2 S-NIAH-3 FS base 100100 7474 1414 Immediate pruning 6262 7272 1414 FS-L 9090 6464 1414 FS, seed 42 (run 1) 100100 5050 1616 FS, seed 42 (run 2) 100100 5252 2020 FS, seed 43 100100 7474 66 FS-L, seed 42 (run 1) 8282 4444 1212 FS-L, seed 42 (run 2) 7676 4646 1818 FS-L, seed 43 8282 8282 88 At 44k, immediate pruning reduces S-NIAH-1 from 100100 to 6262. FS-L recovers 9090, while the seed-4242 adapted runs recover 2020 and 1414 percentage points (paired-prompt bootstrap 95%95\% intervals [4,36][4,36] and [0,28][0,28]). All three unpruned continuations retain 100100. S-NIAH-2 declines under seed 4242 even without pruning, whereas seed 4343 reaches 8282 with pruning versus 7474 without it. Thus CPT supports S-NIAH-1 recovery, but harder-task changes vary across seeds and also occur without pruning. Intervals use 50005000 prompt resamples and do not estimate seed uncertainty. Harder-task gains also occur without pruning at 800800M: after 30003000 CPT steps (1.571.57B tokens), FS’s 1616k S-NIAH-2/3 accuracy rises from 72/3272/32 to 84/3884/38 on 5050 shared prompts per task. Budget-matched continuation of unchanged architectures. To assess the effect of additional training alone, Table 12 reports selected 800800M-scale baselines and FS continued from their Table 1 checkpoints with plain next-token loss. These controls match FS-L’s 30003000 updates of 524,288524,288 tokens each (1.571.57B total), using FineWeb-Edu at 40964096-token context (Appendix A). The table also includes a separate single-forest width-3232 ablation of FS-B’s recall-weighted recipe: 16001600 updates of 262,144262,144 tokens each (0.420.42B total, 27%27\% of the matched budget), at the same context length (Appendix E.4). Each before/after pair uses identical prompts and task sets: means cover 1212 tasks at 22k/44k and 1313 at 88k/1616k, following the protocol of Table 1. Plain CPT does not improve the baselines or unpruned FS consistently across all evaluated lengths. Only FS-B’s recall-weighted CPT improves the RULER mean at all four lengths in Table 12, as measured by the single-forest width-3232 ablation: 40.9/36.0/30.4/28.240.9/36.0/30.4/28.2 becomes 41.3/37.2/31.6/30.441.3/37.2/31.6/30.4. FS-L recovers much of the S-NIAH-1 accuracy lost under immediate pruning (Table 11), but its suite mean remains below base FS at every length in Table 12. Table 12: Only FS-B’s weighted CPT improves the RULER mean at all four lengths. Suite-mean changes at 800800M scale (++ CPT minus base, points, on the same prompts and protocols as Table 1; the base means are the Table 1 rows). Each “++CPT” arm continues the Table 1 checkpoint of the same architecture with plain next-token loss at the FS-L budget (1.571.57B tokens). The FS-L arm instead keeps the pruning loop in training at the same budget. The single-forest ablation uses FS-B’s recall-weighted CPT, spending 0.420.42B tokens (27%27\%; Appendix E.4), and compares the full-prefix width-3232 read before and after CPT (base means 40.9/36.0/30.4/28.240.9/36.0/30.4/28.2). FS-B itself denotes the two-forest design of Table 1. Continuation arm Δ 2 \,2k Δ 4 \,4k Δ 8 \,8k Δ 16 \,16k SWA, 743743M, plain −1.4-1.4 +0.4+0.4 +0.2+0.2 −0.2-0.2 Raven-full, 723723M, plain −7.2-7.2 −5.6-5.6 −1.4-1.4 +0.8+0.8 Raven, 726726M, plain −2.2-2.2 −1.0-1.0 −2.8-2.8 −1.9-1.9 NSA, 11281128M, plain −1.2-1.2 −2.2-2.2 +0.3+0.3 +0.1+0.1 FS, 743743M, plain −1.6-1.6 −1.3-1.3 −1.0-1.0 0.00.0 FS, 743743M → FS-L (pruning) −2.2-2.2 −2.0-2.0 −2.0-2.0 −0.8-0.8 Single-forest W32 ablation, 743743M, weighted +0.4+0.4 +1.2+1.2 +1.2+1.2 +2.2+2.2 Training from scratch. CPT gives a higher RULER mean than training from scratch at all four lengths (Table 13). The gain comes from S-NIAH-2/3, with a tradeoff on S-NIAH-1 and downstream accuracy. At 1616k, the mean rises from 9.69.6 to 10.710.7, while S-NIAH-1 falls from 9393 to 8787. The eight-task downstream average falls from 52.252.2 to 51.051.0. At 800800M, training with pruning from scratch reaches a downstream average of 56.856.8 but S-NIAH-1 scores of only 3232, 1616, 88, and 00 at 22k, 88k, 1616k, and 3232k (2525 examples), whereas CPT retrieves well (Table 1); we therefore use CPT at both scales. Table 13: FS-L training recipes at 340340M (00–100100). Both use pruning during evaluation, with 100100 examples per RULER task and length. Bold marks the better recipe. S-NIAH-1 S-NIAH-2 S-NIAH-3 RULER mean Down. Recipe 22k 44k 88k 1616k 22k 44k 88k 1616k 22k 44k 88k 1616k 22k 44k 88k 1616k avg From scratch 100 97 98 93 7878 4949 88 33 5353 1111 33 00 27.827.8 19.219.2 11.711.7 9.69.6 52.252.2 CPT 9999 8888 9090 8787 99 70 34 20 70 17 9 1 32.432.4 21.621.6 15.515.5 10.710.7 51.051.0 D.5 Retention budgets and multi-fact retrieval FS-L must retain information before knowing which question will be asked. The adapted budget sweep measures the resulting accuracy–memory tradeoff. The multiple-question test checks whether the models can answer questions about several facts near the training length. The question distribution sets the budget. A small retained set can cover likely future questions when useful information is concentrated in a few cells. Consider instead F independent key–value facts in separate cells, with a later question choosing one uniformly. Keeping m≤Fm≤ F source cells gives direct access to only m/Fm/F of the possible targets. Contextual representations can copy information between cells, so this ratio is not an answer-accuracy bound. The accessible state includes all heads and layers, local buffers, and their contextual representations. Compression does not remove this limit. If the F values are independent and uniform over V outcomes, the question reveals only which value to retrieve, and the state holds b bits, Fano’s inequality bounds the mean accuracy by p¯≤(1+b/F)/log2V p≤(1+b/F)/ _2V; any fixed accuracy with p¯log2V>1 p _2V>1 therefore needs b=Ω⁡(F)b= (F). This motivates testing memory budgets and preserving more cells when information is dense. It does not identify the bottleneck in a trained model: a wrong answer can reflect lost information, failed routing, or failure to use available evidence. Adapted retention budgets. From the same base FS checkpoint, we adapt with P=16P=16 and P=32P=32, keeping W=8W=8, K=4K=4, and the same 15001500-step schedule. Each model evaluates with its training budget on the same 5050 prompts (Table 14). At 22k, S-NIAH-1 falls from 100100 at P=64P=64 to 9898 at P=32P=32 and 8484 at P=16P=16, but accuracy is not monotone in P: at 1616k, P=64/32/16P=64/32/16 score 7474–80/66/6880/66/68, using the two seed-4242 controls at P=64P=64. S-NIAH-2/3 do not improve consistently with P. These independently adapted operating points measure a training–memory tradeoff, not the effect of changing the budget in one fixed model. Only P=64P=64 has repeated continuations. The local and temporary buffers also contribute to physical state, so changing P by a factor of four does not change total memory by that factor. Table 14: Adapted retention budgets at 340340M. Each model trains and evaluates with its stated P. S-NIAH-1 S-NIAH-2 S-NIAH-3 Model 22k 44k 88k 1616k 22k 44k 88k 1616k 22k 44k 88k 1616k P=16P=16 8484 6464 6868 6868 100 56 24¯ 24 14 5656 1414 44 2 P=32P=32 98¯ 98 78¯ 78 72¯ 72 6666 9696 50¯ 50 32 10¯ 10 74 20 6¯ 6 2 P=64P=64, run 1 100 82 82 80 98¯ 98 4444 2222 22 66¯ 66 1212 8 2 P=64P=64, run 2 100 7676 7070 74¯ 74 9696 4646 2020 22 5858 18¯ 18 8 2 Figure 8 relates these scores to the allocated attention buffers of the streaming cache (Table 15, protocol of Appendix C): reducing P fourfold gives a 2.4×2.4× reduction in total buffers (which do not depend on length), while decode stays at 8888–8989 tokens/s. The P=64P=64 row of Table 15 uses the Table 1 checkpoint, while the P=64P=64 accuracies in Figure 8 average the two seed-4242 continuations. Figure 8: Accuracy versus attention storage at 340340M. The P=64P=64 points show the mean and range of two seed-4242 continuations. Table 15: Retention budget and bounded inference at 340340M, 256256k. P Buffers (MiB) Peak (GiB) Prefill (Ktok/s) Decode (tok/s) 1616 320.0320.0 1.341.34 2.132.13 88.4¯ 88.4 3232 468.8¯ 468.8 1.49¯ 1.49 2.192.19 88.888.8 6464 766.2766.2 1.771.77 2.17¯ 2.17 88.388.3 Several facts at fixed context. Each synthetic prefix holds F distinct key–value facts with seven-digit values, each outside the final 528528 prefix tokens. The question is revealed only after the prefix, and each question continues an independent copy of the prefix state. We generate 1616 tokens greedily and score whether the target value appears, with 5050 prefixes per condition, querying every fact for F≤8F≤ 8 and eight sampled facts for F=16F=16. Prefixes contain 42534253–42774277 tokens at 340340M and 39463946–39713971 at 800800M, where prefix, question and answer fit within 40964096 tokens. Table 16: Multiple-question retrieval. Accuracy (%) on shared prefixes: an answer is correct if the target value appears in 1616 greedy tokens; 5050 prefixes per condition. Bold and underline mark the best and second-best score within each scale, including ties. FS-B prefills each question afresh; at 340340M this raises FS and FS-L by about 22 points relative to the shared-prefix rows. Facts per prefix 11 22 44 88 1616 Questions 5050 100100 200200 400400 400400 340340M, 42534253–42774277 prefix tokens FS 94¯ 94 4141 1919 7.757.75 2.752.75 FS-L 9090 48¯ 48 21¯ 21 8.00¯ 8.00 3.25¯ 3.25 FS-B 100 56 28.528.5 13.013.0 4.254.25 800800M, strictly within 40964096 tokens FS 100 4949 2626 11.511.5 7.257.25 FS-L 96¯ 96 5353 2121 1010 44 TF++ 8888 5656 38 22 19.7519.75 Raven 100 57¯ 57 28.528.5 99 22 FS-B 100 5252 3232 16.516.5 8.758.75 Raven-full 100 66 33.5¯ 33.5 21.75¯ 21.75 13¯ 13 FS, FS-L, and FS-B pass a 90%90\% single-fact threshold at both scales, but no model passes it with two or more facts (Table 16); FS-B scores highest at 340340M; at 800800M with four or more facts, full-attention TF++ and Raven-full lead instead. The decline also occurs with unpruned access, with full-attention TF++ and with Raven and Raven-full, and at 800800M entirely within the training context, so it is not a pruning or length-extrapolation limit alone. At 800800M with two facts, most wrong answers contain another value from the same prefix (42/4742/47 for FS-L, 46/5146/51 for FS, 48/4848/48 for FS-B, 36/4336/43 for Raven and 36/4436/44 for TF++), consistent with confusion among stored facts. Because unpruned FS and full-attention TF++ decline in the same way, this test measures interference among facts near the training length rather than the retention limit above. Appendix E FS-B details Section 3.4 defines FS-B. This appendix explains how the forests are built and searched, what they cost, and the CPT recipe, and relates FS-B to chunk and hierarchical retrieval. E.1 Streaming forests Causal availability. Cells contain s=32s=32 tokens and query blocks contain B=128B=128 tokens. At each block boundary, cells completed before that boundary are admitted to the forests by Eq. (10). The forests and their summaries then stay fixed throughout the block, while each token searches independently. Prefill and cached decoding use the same block-boundary availability as FS training. Filtering a cell from the second forest does not remove it from the first, nor its tokens from the local window. The threshold τℓh _ h is fixed per checkpoint, so the second forest need not hold half of a particular prefix. Chunks. Fix one head and one forest at a block boundary, and let c1,…,cMc_1,…,c_M be its admitted cells in arrival order, with mean keys ¯c k_c, key norms aca_c and positions pcp_c (Eq. (5)). Write the binary decomposition M=∑t=1r2ktM= _t=1^r2^k_t with k1>⋯>krk_1>…>k_r. The active chunks are the consecutive runs t=cj:ot<j≤ot+2kt,ot=∑u<t2ku,G_t=\c_j:o_t<j≤ o_t+2^k_t\, o_t= _u<t2^k_u, (18) so older cells sit in larger chunks; 13=8+4+113=8+4+1 gives c1,…,c8\c_1,…,c_8\, c9,…,c12\c_9,…,c_12\ and c13\c_13\. A chunk never changes once complete, so its tree is built once and reused until two equal chunks merge into the next larger one. Splitting a chunk. Each node v holds a cell set ⁡(v)C(v) of size nvn_v; the root of tG_t holds all of tG_t. A node with nv>1n_v>1 cells centers their mean keys, c=¯c−vx_c= k_c- μ_v, and estimates a principal direction with two power iterations started at the farthest cell: (0)∝c⋆,c⋆=argmaxc∈⁡(v)‖c‖2,(t+1)∝∑c∈⁡(v)(c⊤(t))c,t=0,1.u^(0) _c , c = _c (v) x_c_2, ^(t+1) _c (v) (x_c u^(t) )\,x_c, t=0,1. (19) Cells are sorted by c⊤(2)x_c u^(2); the first ⌈nv/2⌉ n_v/2 form the left child and the rest the right child. Recursion stops at single cells, so a chunk of 2k2^k cells gives a perfect binary tree of height k whose leaves are ordered by content rather than position. Node summaries. Every chunk-tree node stores v=1nv∑c∈⁡(v)¯c,ρv=maxc∈⁡(v)⁡‖¯c−v‖2,Av=maxc∈⁡(v)⁡ac,pvmin=minc∈⁡(v)⁡pc, μ_v= 1n_v _c (v) k_c, _v= _c (v) k_c- _v_2, A_v= _c (v)a_c, p_v = _c (v)p_c, (20) the quantities scored by Eq. (11). Joining chunk roots. A chain joins the chunk roots from the smallest chunk upward: JrJ_r is the root of rG_r, and for t=r−1,…,1t=r-1,…,1 the join node JtJ_t has children root⁡(t)root(G_t) and Jt+1J_t+1; J1J_1 is the forest root. A join node v with children a,ba,b stores nv=na+nb,v=naa+nbbnv,ρv=maxu∈a,b⁡(ρu+‖u−v‖2),n_v=n_a+n_b, μ_v= n_a μ_a+n_b μ_bn_v, _v= _u∈\a,b\ ( _u+ _u- _v_2 ), (21) with Av=max⁡(Aa,Ab)A_v= (A_a,A_b) and pvmin=min⁡(pamin,pbmin)p_v = (p_a ,p_b ). By the triangle inequality its ball encloses every cell mean below it, so Eq. (11) remains an upper bound at join nodes. The root of tG_t sits at depth min⁡(t,r−1) (t,r-1) and its tree has height kt≤k1−t+1k_t≤ k_1-t+1, so the forest height is k1+[r≥2]=⌈log2M⌉k_1+[r≥ 2]= _2M . One forest: M=13=8+4+1M=13=8+4+1 admitted cells336611882255774411119912121010131388 cells44 cells11 cellroot: 1313 cells55 cellsheight ⌈log2M⌉=4 _2M =41. Build balanced chunksMedian-split mean keys alongtheir principal direction.Compute summaries from the cells(Eq. (20)).2. Join roots, smallest first4+1→5,8+5→134+1→ 5, 8+5→ 13Weighted mean of child centers;radius encloses both child balls(Eq. (21)). Figure 9: Streaming construction of one FS-B forest, per head. Solid edges form balanced trees within power-of-two chunks; dashed edges join their roots, smallest first. Leaf numbers denote arrival order, while their arrangement follows mean-key splits. The 13=8+4+113=8+4+1 example has height ⌈log213⌉=4 _213 =4. Right: the two construction steps; node summaries are defined in Eqs. (20)–(21). Beam search and reranking. Search expands the current beam level by level and keeps the WbW^b highest-priority nodes under Eq. (11). A leaf reached early on a short branch keeps its slot, while the remaining slots expand children. The bound holds for the mean-key score of Eq. (7), not for the token-level score of Eq. (12) or for a cell’s usefulness to the answer; larger c favors subtrees with dispersed mean keys. We set c=8c=8 on S-NIAH at 341341M and use it unchanged at 743743M. The two candidate sets are merged without a quota for either forest. In Eq. (12), the log-sum-exp term is the logarithm of a cell’s unnormalized token-attention mass, and the prior affects selection only. Node summaries never replace token values. Streaming inference. Prefill processes query blocks in fixed-size chunks. During decoding, the cache keeps only the active chunk trees. A completed larger chunk replaces its constituent smaller trees, and unchanged chunks are reused. For M admitted cells, these trees and their joining nodes contain fewer than 2M2M nodes, giving O⁡(Md)O(Md) tree storage per head. E.2 Cost Let M≤⌊N/s⌋=⌊N/32⌋M≤ N/s = N/32 be the number of admitted cells in a forest; the filtered forest can hold fewer cells than the full-prefix forest. For a balanced tree and Ci=|ℬi|C_i=|B_i| candidates, search, token-key reranking and readout take O⁡(d⁡[Wblog⁡M+sCi+sK])O(d[W^b M+sC_i+sK]) work per query, with Ci≤2WbC_i≤ 2W^b for the two forests. Fixed budgets therefore give O⁡(Nlog⁡N)O(N N) routing, reranking and token attention over the sequence, including the fixed local window. Construction with radix sorting. Assume fixed-width keys for the split projections and stable radix sorting, so sorting m projections costs O⁡(m)O(m) work. Computing the principal direction, projections and summaries at a node costs O⁡(md)O(md), hence a chunk of 2k2^k cells costs O⁡(d2kk)O(d2^kk) over its k split levels. At most ⌊M/2k⌋ M/2^k chunks of that size complete, and completed trees are reused. Summing over chunk sizes gives O⁡(∑k=1⌊log2M⌋M2kd2kk)=O⁡(Mdlog2M).O\! ( _k=1 _2M M2^k\,d2^kk )=O(Md ^2M). (22) Two forests change only the constant factor. Including search and readout, the algorithmic sequence cost is O⁡(Nlog2N)O(N ^2N) at fixed dimensions and budgets, still strictly subquadratic. This bound assumes that updates reuse stored chunks without copying the growing forest at every block. Comparison sorting at every split would instead give construction work O⁡(Mdlog2⁡M+Mlog3M)O(Md ^2M+M ^3M); the stated bound uses fixed-width radix sorting or linear-time median partitioning. Reference streaming overhead. The reference cache reuses completed trees but reallocates its growing admitted-cell index array and repacks all active node tables after updates. At fixed block size, these copies can accumulate O⁡(M2d)O(M^2d) memory traffic over a growing decode stream, outside the algorithmic construction bound. Geometric buffer growth and persistent chunk storage can avoid this copying without changing tree contents or search. The measured reference throughput includes these overheads. The token archive uses O⁡(Nd)O(Nd) space and the active decode trees use O⁡(Md)O(Md) storage. E.3 Forest ablation Table 17 compares FS-B’s two forests with single-forest ablations, changing only the inference-time candidate generator; the checkpoints and the four-cell token read are unchanged. At 800800M, FS-B matches or exceeds the full-prefix width-3232 ablation at every length, by up to 0.80.8 mean points at the displayed precision. The width-1616 full-prefix ablation has the highest mean at 44k (37.937.9 versus 37.737.7 for FS-B). At 340340M, the width-3232 ablation leads the suite mean at all four lengths by up to 1.31.3 points. Its S-NIAH-2 scores match or exceed FS-B’s; its S-NIAH-3 scores are higher at 22k–88k, but lower at 1616k (2424 versus 2727). At 800800M, FS-B has higher FWE scores than the width-3232 ablation at every length and higher MV-NIAH scores from 44k onward. Two width-1616 beams supply at most 3232 distinct candidates after deduplication, so a width-3232 single forest matches the maximum candidate count. A width-1616 single forest has a smaller candidate budget and scores below width 3232 at 88k–1616k at both scales. Equal maximum candidate counts do not imply equal runtime. The norm-filtered width-1616 ablation retains 4141–74%74\% of cells by task, using a median key-norm threshold calibrated on eight 40964096-token FineWeb-Edu batches. Relative to FS-B, its suite mean is lower by 1.51.5–2.92.9 points at 340340M and 0.20.2–1.51.5 points at 800800M. Table 17: Forest ablation of FS-B (00–100100, RULER protocol of Table 1). All arms run on the same checkpoints with the same four-cell token read; only the inference-time candidate generator differs: FS-B’s two forests as rendered in Table 1, the full-prefix forest alone with a width-3232 or width-1616 beam, or the norm-filtered forest alone with a width-1616 beam. Bold marks the best of the four arms per column. FS-B (two forests) full, Wb=32W^b=32 filtered, Wb=16W^b=16 full, Wb=16W^b=16 Task 22k 44k 88k 1616k 22k 44k 88k 1616k 22k 44k 88k 1616k 22k 44k 88k 1616k ≈ 340M scale S-NIAH-1 100 99 99 99 100 99 99 99 9999 99 99 99 100 99 99 99 S-NIAH-2 100 9999 8989 6060 100 100 92 67 9797 9999 6868 4949 100 9999 7777 4444 S-NIAH-3 9191 7878 6464 27 95 84 71 2424 7575 6161 5151 2222 9090 7979 7070 2626 MK-NIAH-1 2727 2525 1919 24 2727 28 23 2121 2929 2424 2222 1919 30 2020 23 1919 MK-NIAH-2 2 0 0 1 2 0 0 1 2 0 0 1 2 0 0 00 MK-NIAH-3 2 00 0 0 11 00 0 0 11 1 0 0 11 00 0 0 MQ-NIAH 23.823.8 22.522.5 18.518.5 12.512.5 23.823.8 22.222.2 19.019.0 15.515.5 22.822.8 21.821.8 16.816.8 10.210.2 24.824.8 21.521.5 18.518.5 14.014.0 MV-NIAH 24.524.5 22.022.0 23.523.5 21.021.0 24.524.5 22.522.5 25.525.5 21.021.0 23.023.0 21.521.5 23.023.0 16.016.0 24.224.2 23.223.2 22.022.0 18.018.0 VT 3.23.2 1.21.2 1.01.0 1.01.0 3.03.0 1.41.4 1.61.6 1.21.2 2.82.8 1.21.2 1.01.0 1.21.2 3.03.0 1.21.2 1.01.0 1.41.4 CWE 5.25.2 10.310.3 6.56.5 1.81.8 6.76.7 10.010.0 6.66.6 2.02.0 5.05.0 10.610.6 6.66.6 1.81.8 5.35.3 10.410.4 7.37.3 1.31.3 FWE 9.09.0 6.06.0 11.711.7 5.05.0 9.39.3 4.74.7 9.09.0 5.75.7 8.78.7 3.73.7 6.76.7 3.03.0 9.79.7 6.06.0 9.79.7 6.76.7 QA-1 23 1010 44 44 2222 99 6 5 1818 12 6 33 23 1010 33 44 QA-2 – – 99 66 – – 10 66 – – 88 7 – – 77 66 Available-task mean 34.234.2 31.131.1 26.626.6 20.220.2 34.534.5 31.731.7 27.927.9 20.620.6 31.931.9 29.629.6 23.723.7 17.917.9 34.434.4 30.830.8 26.026.0 18.418.4 ≈ 800M scale S-NIAH-1 97 9898 9797 100 9595 9898 9999 9999 97 99 100 9999 97 99 9898 100 S-NIAH-2 100 100 95 8888 100 100 95 90 100 100 95 8181 9898 100 8787 8080 S-NIAH-3 96 9393 8787 8484 96 9393 8787 87 8888 8888 8484 8181 9393 94 90 8282 MK-NIAH-1 5050 34 2828 31 4646 2525 30 2929 52 2828 2828 3030 4848 2828 2727 2626 MK-NIAH-2 99 4 11 22 55 4 11 22 88 33 11 3 11 11 2 22 MK-NIAH-3 2 1 00 0 2 00 1 0 11 00 00 0 2 00 00 0 MQ-NIAH 45.845.8 28.328.3 28.828.8 27.327.3 45.045.0 30.230.2 27.227.2 27.827.8 40.540.5 29.229.2 29.029.0 25.825.8 38.238.2 29.029.0 25.025.0 25.525.5 MV-NIAH 38.338.3 30.330.3 29.829.8 26.326.3 39.539.5 28.828.8 26.226.2 24.524.5 38.038.0 29.829.8 27.227.2 26.226.2 34.234.2 28.228.2 26.226.2 24.824.8 VT 3.83.8 1.41.4 1.01.0 0.60.6 4.04.0 2.02.0 0.80.8 0.40.4 4.84.8 2.02.0 0.80.8 0.20.2 4.04.0 2.42.4 1.01.0 0.60.6 CWE 7.87.8 26.426.4 8.98.9 4.44.4 7.87.8 29.229.2 8.18.1 3.43.4 10.010.0 24.924.9 9.99.9 3.23.2 9.69.6 30.130.1 7.47.4 2.32.3 FWE 17.717.7 8.78.7 7.07.0 8.08.0 15.315.3 7.37.3 3.33.3 5.75.7 10.310.3 4.74.7 7.77.7 15.015.0 23.023.0 10.710.7 7.37.3 8.38.3 QA-1 3838 2727 1313 1010 40 2929 14 13 3939 2626 1313 1111 40 32 14 1212 QA-2 – – 1717 14 – – 18 14 – – 1515 99 – – 1414 1111 Available-task mean 42.142.1 37.737.7 31.831.8 30.430.4 41.341.3 37.237.2 31.631.6 30.430.4 40.740.7 36.236.2 31.631.6 29.629.6 41.541.5 37.937.9 30.730.7 28.828.8 E.4 Retrieval-weighted CPT Recall masks. Let ftf_t indicate a recall start or continuation, identified by a unique earlier four-token match more than 512512 tokens away. A start matches the span beginning at the target token with a different preceding token; a continuation matches the preceding four tokens and their next target. Let χt _t mark a recall start within 6464 tokens of the previous start whose source is more than 6464 tokens from the previous source, which distinguishes switching to another span from returning to the same one. The masks respect document boundaries and do not depend on the model’s confidence. Only natural FineWeb-Edu text is used; benchmark prompts and answers are never training examples or sources of loss weights. Schedule. Starting from the base 800800M FS checkpoint, CPT has two 800800-step stages of 262,144262,144 tokens per step with β=9β=9; βsw _sw is 99 in the first stage and 4949 in the second, so a switch token has weight 1919 and then 5959. In each stage the learning rate warms up for 2020 steps to 4×10−54× 10^-5 and decays to 4×10−64× 10^-6. The total is 16001600 steps and 0.420.42B tokens, all at 40964096 tokens per sequence, so evaluation beyond 44k remains zero-shot extrapolation. During CPT, FS scores the eligible prefix and reads 1616 cells per head; at evaluation, FS-B searches the two forests and reads four. The forests are not in the training loop, and no routing parameters are learned. CPT ablation with single-forest readout. Table 18 compares the base and CPT checkpoints under the single-forest ablation read paths, holding each selector fixed: the 0.420.42B-token continuation improves every arm and length, with the largest gains at 1616k. The gains concentrate on the retrieval tasks that the weighting targets (S-NIAH-1/3 improve by 44–1818 points and multi-key and multi-value retrieval by 44–88), with small regressions on CWE of up to 7.87.8 points and on FWE of up to 3.73.7. Reading the same CPT checkpoint with FS-B’s two-forest search gives 42.1/37.7/31.8/30.442.1/37.7/31.8/30.4 (Table 1), exceeding the full-prefix width-3232 ablation at 22k–88k and matching it at 1616k at the displayed precision. The per-task breakdown of the CPT arms is in Table 17. The pools store aggregate scores per task, so no prompt-level intervals are reported. Table 18: Recall-weighted CPT under single-forest ablations (00–100100; RULER available-task means over 1212 tasks at 22k/44k and 1313 at 88k/1616k, protocol of Table 1). All rows start from the base 743743M FS checkpoint; CPT rows are the Table 1 FS-B checkpoint evaluated with a single-forest ablation. FS-B denotes the two-forest design, whose results appear in Table 1. Read path Weights 22k 44k 88k 1616k Mean Full-prefix forest (W32) base FS 40.940.9 36.036.0 30.430.4 28.228.2 33.933.9 ++weighted CPT 41.341.3 37.237.2 31.631.6 30.430.4 35.135.1 Norm-filtered forest (W16) base FS 39.639.6 35.135.1 29.829.8 27.527.5 33.033.0 ++weighted CPT 40.740.7 36.236.2 31.631.6 29.629.6 34.534.5 E.5 Relation to chunk and hierarchical retrieval Several methods retrieve distant chunks. HCAM [Lampinen et al., 2021] and the Hierarchical Sparse Attention of RAMba [Hu et al., 2025a] score chunk summaries, attend within selected chunks, and combine the chunk outputs; DRT’s Grouped Cross-Attention [Hu et al., 2025b] retrieves past chunks for each next chunk. Their routing scores weight the chunk outputs, which gives the router a gradient, and DRT and RAMba offload growing token memory to the CPU while keeping chunk keys on the GPU. FreeSolo instead uses hard cell selection followed by one softmax over the selected tokens, so its shared projections learn through the token readout and local attention; FS-L bounds the accessible cells by P+WP+W per head, and FS-B searches two content forests that together cover every cell. HMT [He et al., 2025] recalls compressed segment memories within a recurrent architecture, whereas FreeSolo keeps the token-level keys and values of every retrievable cell. Treeformer’s TF-attention [Madaan et al., 2023] learns decision-tree partitions and attends to keys routed to the query’s leaf; FS-B builds balanced partitions from the mean keys of causally available cells and searches several branches with query-dependent priorities. The published protocols also differ: DRT uses task fine-tuning and RAMba’s RULER tests include synthetic fine-tuning at 44k, whereas our models use next-token pretraining only, so we do not compare published scores.

Instant research discovery