Paper deep dive
SuPER-Mamba: Surprisal Prioritization for Extended-Context Routing in Mamba
Zhetong Zhang, Gayan K Kulatilleke
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:State space models provide efficient linear-complexity processing for long sequences, but their performance frequently degrades as context length grows. We propose SuPER-Mamba, a training-free framework that uses Surprisal as a model-intrinsic signal for long-range state routing in pretrained Mamba models. SuPER-Mamba computes position-wise Surprisal from next-token predictions, aggregates neighboring responses through Local Pooling, and uses the resulting scores to prioritize Global-Channel updates without query-dependent scoring or additional training. We evaluate SuPER-Mamba on three state-space models across the LongBench-E benchmark. The results show that SuPER-Mamba improves the average score from 8.06 to 15.69, corresponding to a 94.7% relative gain over the Base Models and a 13.1% improvement over the state-of-the-art prior baseline. Ablation and length-wise analyses further support the effectiveness of the proposed routing framework.
Tags
Links
- Source: https://arxiv.org/abs/2610.16636v1
- Canonical: https://arxiv.org/abs/2610.16636v1
Trouble viewing inline? Open PDF directly →
Full Text
57,310 characters extracted from source content.
Expand or collapse full text
SuPER-Mamba: Surprisal Prioritization for Extended-Context Routing in Mamba Zhetong Zhang and Gayan K Kulatilleke * School of Information Technology and Electrical Engineering The University of Queensland zhetong.zhang@student.uq.edu.au, g.kulatilleke@uq.edu.au Abstract State space models provide efficient linear- complexity processing for long sequences, but their performance frequently degrades as con- text length grows. We propose SuPER-Mamba, a training-free framework that uses Surprisal as a model-intrinsic signal for long-range state routing in pretrained Mamba models. SuPER- Mamba computes position-wise Surprisal from next-token predictions, aggregates neighbor- ing responses through Local Pooling, and uses the resulting scores to prioritize Global- Channel updates without query-dependent scor- ing or additional training. We evaluate SuPER- Mamba on three state-space models across the LongBench-E benchmark.The results show that SuPER-Mamba improves the aver- age score from 8.06 to 15.69, corresponding to a 94.7% relative gain over the Base Models and a 13.1% improvement over the state-of-the- art prior baseline. Ablation and length-wise analyses further support the effectiveness of the proposed routing framework. Code, training scripts and datasets :https:// github.com/gayanku/SuPER-Mamba 1 Introduction Long-context modeling requires a model to pro- cess large amounts of input while effectively pre- serving and utilizing information that remains rel- evant over long distances. Mamba and its subse- quent variants provide an efficient alternative to Transformer-based architectures for long-sequence modeling through recurrent state-space computa- tion with linear sequence complexity (Gu and Dao, 2024; Dao and Gu, 2024). However, historical in- formation must be continuously compressed into a finite recurrent state, making effective long-range information preservation increasingly difficult as the context grows. Existing studies have shown * Corresponding author. Position Surprisal Score Predictable Context ... The meeting continued as planned, with the team reviewing the report, discussing the remaining tasks, and confirming the schedule for the following week ... Informative Context Position Surprisal Score ... The meeting continued as planned, with the team reviewing the report, when the coordinator announced that the access code had been changed to ORION-7 ... Mamba Mamba Figure 1: Motivation of SuPER-Mamba. A pretrained Mamba produces position-wise Surprisal from next- token prediction probabilities while processing the input context. Relatively predictable local content typically produces lower and more stable Surprisal responses, whereas less predictable local changes may produce larger responses. SuPER-Mamba treats these responses as model-intrinsic predictive feedback and further in- vestigates the relationship between such local predictive feedback and signals for long-range state allocation. that different hidden channels in Mamba exhibit different long-range retention characteristics, and that selectively controlling long-range state updates can alleviate information decay and improve long- context modeling performance (Ye et al., 2025b,a). These findings regarding Mamba’s long-context capabilities motivate the investigation of intrinsic model signals that can guide the allocation of lim- ited long-range state updates without relying on downstream queries. Pretrained autoregressive language models con- tinuously produce predictive distributions over the next token while processing a context, reflecting the model’s predictive expectations at different po- sitions. As illustrated in Figure 1, relatively pre- dictable local content tends to produce more sta- ble predictive responses, whereas less predictable changes may induce more pronounced Surprisal responses. The mismatch between the model’s pre- dictive expectation and the observed next token mo- tivates our hypothesis that these position-dependent 1 arXiv:2610.16636v1 [cs.CL] 12 Oct 2026 predictive responses can provide a model-intrinsic signal for long-range state allocation. Based on this hypothesis, we propose SuPER- Mamba, which uses Surprisal derived from the complete input context to quantify position-wise predictive responses and combines them with local information aggregation to construct context scores for guiding long-range state updates. Figure 2 provides an overview of the framework. SuPER- Mamba first computes position-wise Surprisal and aggregates neighboring responses through Local Pooling, then uses the resulting scores to route se- lected content positions to long-range state updates while preserving the complete input in the original recurrent computation. The method requires no ad- ditional training or model fine-tuning. We systemat- ically evaluate SuPER-Mamba on 13 LongBench-E tasks across three representative state-space-based architectures. The results show that, without us- ing the downstream query for context selection, SuPER-Mamba substantially improves the long- context performance of the corresponding Base Models while remaining competitive with state-of- the-art baselines. Our contributions are as follows. •We propose SuPER-Mamba,a query- independent routing framework for long- context modeling in pretrained Mamba models. The framework directly uses pre- dictive responses produced by the pretrained model to construct context-scoring signals to guide long-range state updates. • We first introduce Surprisal-guided routing for long-context modeling in pretrained Mamba models. SuPER-Mamba combines Surprisal with local information aggregation to trans- form the model’s predictive feedback into a routing signal for long-range state allocation without additional selector training. •WeevaluateSuPER-Mambaon13 LongBench-E tasks across three repre- sentative state-space-based architectures. Without using the downstream query for context selection, SuPER-Mamba improves the average score from 8.06 to 15.69, corre- sponding to a 94.7% relative improvement over the Base Models, while achieving 13.1% higher average performance than the state-of-the-art baseline evaluated across the same architectures. 2 Related Work 2.1 Long-Context Memory and Routing Mamba achieves linear-complexity sequence mod- eling through input-dependent state-space updates, but its fixed-size recurrent state requires grow- ing historical information to be compressed into a bounded representation (Gu and Dao, 2024; Dao and Gu, 2024). Recent studies have examined how information is propagated and retained under this recurrent computation. The Hidden Attention of Mamba Models analyzes interactions between context positions from an attention-like perspec- tive, while LaTIM characterizes fine-grained la- tent token-to-token interactions in Mamba-1 and Mamba-2 (Ali et al., 2025; Pitorro and Treviso, 2025). Other work reports selective information loss in finite recurrent representations and shows that Mamba handles locally concentrated key in- formation more effectively than information dis- tributed across distant positions (Hossain et al., 2025; You et al., 2025). These findings indicate that long-context performance depends not only on input length, but also on how historical information is preserved within limited recurrent states. Recent work has improved Mamba’s long- context capability through state-dynamics modula- tion, selective compression, training-based mem- ory enhancement, hierarchical historical access, and additional memory structures (Lu et al., 2025; Yuan et al., 2025; Ma et al., 2025; Hu et al., 2025; Wang et al., 2026; Lahoti et al., 2026; Shen et al., 2026). LongMamba and LAMB instead focus di- rectly on selective long-range state updates. Long- Mamba identifies hidden channels with different receptive-field characteristics and applies selec- tive token filtering to the channels responsible for longer-range information propagation (Ye et al., 2025b). LAMB follows this framework and derives attention-guided importance scores from Mamba’s attention-like representation to determine which positions should update long-range states (Ye et al., 2025a). These methods establish selective alloca- tion of long-range updates as an effective training- free direction for pretrained Mamba models. 2.2 Content-Driven Information Selection for Long Contexts Beyond recurrent state-space models, long-context research increasingly treats historical informa- tion as a non-uniform resource-allocation problem. H 2 O, Scissorhands, SnapKV, PyramidKV, FastGen, 2 Instruction Content Query Text Encoder C 1 C n ... ... Logits Predicted / Actual Token Comparison S t t Raw Surprisal Scores (a) Surprisal Calculation Phase 1 Mamba 4 1 3 2 S t t Mean Max (b) Local Pooling Raw Surprisal Scores S t t Pooled Surprisal Scores S t t Pooling Kernel Phase 2 (c) Routing Process Full Prompt Content Instruction Query Selected Content Top-K Pooled S t Global Channels Local Channels Output Mamba Recent 32 Figure 2: Overview of the SuPER-Mamba framework. Phase 1 performs offline Context Scoring. (a) Surprisal Calculation computes position-wise Raw Surprisal Scores from the next-token prediction probabilities of the pretrained Mamba. (b) Local Pooling aggregates neighboring predictive responses using category-specific Mean or Max Pooling to obtain Pooled Surprisal Scores. Phase 2 uses these scores for long-range state routing. The historical Top-KContent positions selected by the Pooled Surprisal Scores are combined with the most recent R = 32Content positions to form the final routing set. For long-range state updates, Instruction and Query positions do not participate in Content Selection and retain their original state updates, while among Content positions, only the selected positions update the predefined Global Channels. All Content positions continue to participate in the original recurrent computation, so the complete input is neither removed nor truncated. and AttentionPredictor retain or predictively se- lect historical KV states using attention statistics, persistence of importance, prompt observations, head-specific behavior, or temporal attention pat- terns (Zhang et al., 2023; Liu et al., 2023; Li et al., 2024; Cai et al., 2025; Ge et al., 2024; Yang et al., 2025). More recent work also considers local struc- ture rather than isolated token scores. ChunkKV performs selection over contiguous chunks to better preserve local semantic structure, while DAST allo- cates compression capacity according to segment- level information density and combines local pre- dictive information with global signals (Liu et al., 2025; Chen et al., 2025). A related line of work reduces long inputs through context selection or prompt compres- sion.Selective Context uses language-model self-information to remove predictable lexical units, while LLMLingua and LongLLMLingua use model probabilities, token-level information, and downstream questions for fine-grained compres- sion (Li et al., 2023; Jiang et al., 2023, 2024). Other methods use task instructions, query relevance, user intention, answerability, or reasoning states to de- termine which passages, KV blocks, or evidence should be retained (Tang et al., 2025; Taguchi et al., 2025; Zhao et al., 2026; Yun and Kim, 2026). Anal- yses of prompt compression further show that re- ducing input can discard contextual details that are difficult to recover (Łajewska et al., 2025). These approaches therefore differ both in their scoring signals and in whether they operate on explicit KV states, modify the input, or condition selection on downstream information needs. 2.3 Surprisal and Predictive Feedback for Context Selection Surprisal is an information-theoretic quantity de- fined from conditional prediction probabilities and has been used to characterize contextual predictability in incremental language process- ing (Hale, 2001; Levy, 2008). For pretrained au- toregressive models, it can be obtained directly from the next-token predictive distribution pro- duced during context processing, providing a position-dependent response conditioned on the input and model state. Importantly, Surprisal mea- sures predictive difficulty rather than semantic im- portance for a specific downstream task. Predictive information has already been used for context selection. Selective Context uses self- information to distinguish more predictable con- text units, the LLMLingua family incorporates model probabilities or perplexity into compression decisions, and DAST combines local predictive information with global signals (Li et al., 2023; 3 CategoryDatasetBase Mamba2 LongMamba LAMB SuPER-Mamba (Ours) Single-doc QA Qasper1.764.755.635.00 MultiFieldQA-en5.0311.1214.7613.81 Multi-doc QA HotpotQA2.895.457.716.43 2WikiMQA2.315.198.0110.09 Summarization GovReport7.6614.6716.3813.70 MultiNews11.3114.7217.4914.45 Few-shot TREC14.6721.3346.0440.33 TriviaQA10.0847.5851.0447.69 SAMSum4.6613.7926.0228.30 Synthetic Passage Count1.291.723.794.00 Passage Retrieval-en0.815.604.305.19 Code LCC22.8540.3941.6549.03 RepoBench-P19.8933.5234.0242.20 Average8.0916.9121.3021.56 Table 1: Main LongBench-E results on Mamba2-1.3B. Results are grouped by task category. Different datasets are evaluated using their official LongBench-E task-specific metrics, including QA F1 for Qasper, MultiFieldQA-en, HotpotQA, 2WikiMQA, and TriviaQA, ROUGE-L F1 for GovReport, MultiNews, and SAMSum, classification score for TREC, count score for Passage Count, retrieval score for Passage Retrieval-en, and code similarity for LCC and RepoBench-P. Each dataset score is the arithmetic mean over the 0–4k, 4–8k, and 8k+ context-length ranges. The Average row reports the arithmetic mean over all 13 dataset-level scores. Best results are shown in bold, and second-best results are underlined. Jiang et al., 2023, 2024; Chen et al., 2025). How- ever, predictive uncertainty and task-specific rele- vance are not equivalent. QUITO-X argues from an information-bottleneck perspective that high self- information or perplexity does not necessarily im- ply greater relevance to a downstream query (Wang et al., 2025). Related work on information preser- vation in prompt compression similarly cautions against treating prediction-derived scores as direct semantic-importance labels (Łajewska et al., 2025). Position-wise predictive responses can also vary because of lexical choice, special symbols, or local expression changes. This motivates considering local structure rather than isolated Surprisal peaks. Related work such as ChunkKV and DAST simi- larly suggests that contiguous or locally structured information can provide a more stable basis for selection than independent token-level scores (Liu et al., 2025; Chen et al., 2025). Taken together, prior work shows that Mamba long-context modeling depends on information preservation in finite recurrent states, that selec- tive long-range updates can improve performance, and that pretrained models already provide predic- tive signals for differentiating context positions. SuPER-Mamba connects these directions by pre- serving the complete input, deriving position-wise predictive feedback from Surprisal, aggregating neighboring responses through Local Pooling, and using the resulting context scores to allocate long- range state updates without additional selector training or downstream-query-dependent context scoring. 3 Method SuPER-Mamba constructs routing scores from the predictive feedback produced by a pretrained model during context processing. Given a long con- text, Surprisal first converts next-token predictions into position-wise predictive responses. Local in- formation aggregation then combines neighboring responses into context scores, which are used to determine which positions update the predefined Global Channels. The context-scoring procedure consists of Sur- prisal estimation and local information aggregation, followed by long-range state routing. Figure 2 il- lustrates the complete procedure. 3.1 Surprisal Based Contextual Prediction Given a context C = (c 1 ,c 2 ,...,c N ),(1) wherec t denotes the token at context positiont, an autoregressive language model assigns a con- ditional probability toc t based on the preceding sequence c <t , p θ (c t | c <t ),(2) 4 whereθdenotes the pretrained model parameters. The Surprisal at position t is defined as s t =− logp θ (c t | c <t ).(3) Surprisal characterizes the unpredictability of the current token given its preceding context. A lower conditional probability corresponds to a higher Surprisal value and therefore indicates a stronger predictive response at the corresponding position. Conversely, when the current token is more consistent with the prediction formed from the preceding context, its conditional probability in- creases and the corresponding Surprisal decreases. This formulation converts the predictive feed- back continuously produced during autoregressive processing into an explicit position wise numerical signal. For a context of lengthN, the Surprisal val- ues across all positions form the response sequence s = (s 1 ,s 2 ,...,s N ).(4) The sequencesdescribes how the predictive re- sponse of the model varies across context positions. Different positions therefore receive differentiated response values according to the next token distri- butions produced by the pretrained model. Since these values are derived directly from the predictive distributions of the model, they provide input de- pendent position wise feedback without requiring a separate scoring model. Surprisal nevertheless represents the predictive outcome at an individual position, and its values may vary substantially across neighboring tokens. Individual tokens that are difficult to predict may also produce isolated high responses. The Surprisal sequence therefore serves as the position wise pre- dictive representation from which subsequent lo- cal information aggregation constructs the context scores used for routing. 3.2 Local Information Aggregation Token-level Surprisal reflects the predictive re- sponse at an individual position and may vary sub- stantially across neighboring tokens. Isolated high responses can arise from local lexical changes, spe- cial symbols, or other position-specific factors. Di- rectly using these responses for routing may there- fore produce unstable position selection. Given the Surprisal sequences, we apply a local pooling operator u = P k (s),(5) whereP k denotes Mean or Max pooling with win- dow sizek. The resulting score at each position in- corporates predictive responses from its local neigh- borhood, reducing the influence of isolated peaks while preserving local response patterns. The ag- gregated sequenceuis subsequently used for long- range state routing. 3.3 Long-Range State Routing After local aggregation, the pooled score sequence is used to determine which context positions are allowed to update the predefined Global Channels. LetRdenote the recent-position window and let udenote the locally aggregated score sequence. SuPER-Mamba excludes the most recentRposi- tions from score-based selection and selects the Khighest-scoring positions from the remaining historical context. The selected historical set is defined as T K = TopK (u 1:N−R ,K).(6) The most recentRpositions are always pre- served. The final routing set is therefore G =T K ∪N − R + 1,...,N.(7) When the input length is smaller thanK + R, all positions are retained. The same routing mask is applied across Mamba layers while preserving the predefined Global Channels of each layer. LetC G denote the set of predefined Global Channels and let∆ t,j denote the original state-update step at context positiont and channelj. SuPER-Mamba applies the routing mask as e ∆ t,j = ∆ t,j , j /∈C G , ∆ t,j , j ∈C G , t∈G, 0, j ∈C G , t /∈G. (8) Thus, for selected context positions, the original update on the Global Channels is retained, whereas unselected context positions do not update these channels. All non-Global Channels remain un- changed. Non-context prompt positions, including the Instruction and Query, bypass Content selection and retain their original updates. SuPER-Mamba therefore modifies long-range state allocation with- out removing or truncating input tokens, and the complete sequence remains available to the original recurrent computation. 5 CategoryKKernelPooling Single-doc QA102418Max Multi-doc QA102410Mean Summarization204818Mean Few-shot102418Max Synthetic25618Mean Code102412Mean Table 2: Category routing configurations used in the full LongBench-E evaluation.Kdenotes the number of his- torical Content positions selected based on the Pooled Surprisal Scores and allowed to update the Global Chan- nels. Kernel denotes the Local Pooling window sizek, and Pooling denotes Mean or Max aggregation. All task categories additionally retain the most recentR = 32 Content positions. 4 Experimental Setup 4.1 Models, Baselines, and Datasets Models and Datasets.We evaluate SuPER- Mamba on three representative state-space-based models: Mamba1-1.4B (Gu and Dao, 2024), Mamba2-1.3B (Dao and Gu, 2024), and Zamba2- 1.2B (Glorioso et al., 2024). All models use their corresponding pretrained parameters without addi- tional training or parameter updates. We evaluate on all 13 tasks in LongBench-E (Bai et al., 2024), covering a total of 3,668 test samples. Baselines.We compare SuPER-Mamba with the corresponding base pretrained models and LAMB (Ye et al., 2025a). For Mamba2, we ad- ditionally report LongMamba (Ye et al., 2025b) to provide a more complete comparison with existing long-context Mamba methods. Since correspond- ing LongMamba results for Mamba1 and Zamba2 are not included in our evaluation, LongMamba is excluded from the cross-model average. Evaluation. For each LongBench-E task, we com- pute the official task-specific metric separately over the three context-length ranges of 0–4k, 4–8k, and 8k+. We then take the arithmetic mean of the three range-level scores as the task-level score used in our aggregate results. Overall performance is re- ported as the arithmetic mean of the 13 task-level scores. 4.2 Implementation Details Surprisal Scoring. SuPER-Mamba computes Sur- prisal from the predictive responses of the pre- trained model. Context scoring is performed using only the context and does not use the downstream query. The resulting Surprisal scores are fixed be- fore routed inference. No additional selector is ArchitectureBase ModelLAMBSuPER-Mamba Mamba1-1.4B8.5212.9817.93 Mamba2-1.3B8.0921.3021.56 Zamba2-1.2B7.577.327.59 Average8.0613.8715.69 Table 3: Overall LongBench-E results of SuPER- Mamba on Mamba1-1.4B, Mamba2-1.3B, and Zamba2- 1.2B. Each dataset is evaluated using its official LongBench-E task-specific metric, and the results are equally averaged across the three context-length ranges. The Average row reports the arithmetic mean of the overall scores of the three models. trained, and the pretrained model parameters re- main unchanged. Routing Configuration. We use predefined Global Channels and apply the selection mask produced by SuPER-Mamba to control Global-Channel up- dates at different context positions. Token-level Surprisal is first aggregated by local pooling, af- ter which context positions are selected under a fixed routing budget. The most recent 32 context positions are always preserved. We do not apply ad- ditional normalization or denoising to the Surprisal scores. The routing hyperparameters are configured by task category, as summarized in Table 2. The dataset-to-category mapping is provided in Ap- pendix A. Inference Settings. All methods use the same eval- uation samples, generation settings, and evaluation protocol. Inference is performed with batch size 1. SuPER-Mamba does not perform input-level con- text compression. The complete context remains available to the model, while the routing mecha- nism only controls which context positions update the Global Channels. 5 Results 5.1 Main Results Table 3 summarizes the overall LongBench-E per- formance across Mamba1-1.4B, Mamba2-1.3B, and Zamba2-1.2B. Averaged across the three archi- tectures, SuPER-Mamba improves the Base Model average from 8.06 to 15.69, corresponding to a relative improvement of 94.7%. It also improves over the LAMB average of 13.87 by 13.1%. The gain is largest on Mamba1-1.4B and Mamba2-1.3B, while the improvement on Zamba2-1.2B is limited. This suggests that predictive feedback can support routing across different Mamba architectures, al- though its benefit remains dependent on the under- 6 VariantAverage Score Base Model8.09 Random Control19.33 Raw Surprisal16.33 + Local Pooling20.85 Full SuPER-Mamba21.56 Table 4: SuPER-Mamba component ablation results on Mamba2-1.3B. All results are evaluated on the com- plete set of 13 LongBench-E datasets using their official task-specific metrics. Each dataset score is obtained by equally averaging the results over the three context- length ranges. lying backbone. Since LAMB does not provide scripts for Mamba1-1.4B and Zamba2-1.2B, the corresponding results are reproduced based on the method described in the paper. More fine-grained results on Mamba2-1.3B are reported in Table 1. SuPER-Mamba outperforms LAMB on 6 of the 13 datasets, with the clearest ad- vantages on Code tasks. LCC improves from 41.65 to 49.03 and RepoBench-P improves from 34.02 to 42.20, corresponding to relative gains of 17.7% and 24.0%. Competitive results are also maintained across several remaining datasets, indicating that the overall improvement is not driven by a single task. 5.2 Ablation Study Table 4 examines the roles of Surprisal and Local Pooling. Raw Surprisal reaches 16.33, improving over the Base Model by 101.9%, showing that pre- dictive feedback produced by the pretrained model provides an effective signal for long-range position scoring. The Random Control reaches 19.33 under the evaluated setting, indicating that selective allo- cation of long-range state updates itself can also provide substantial benefits. Local Pooling further increases the score from 16.33 to 20.85, corresponding to a 27.7% improve- ment over Raw Surprisal and a 7.9% improvement over the Random Control. This result shows that aggregating neighboring predictive responses sub- stantially strengthens Surprisal-based routing. Full SuPER-Mamba further reaches 21.56, demonstrat- ing that predictive feedback, local aggregation, and the complete routing configuration together pro- vide an effective mechanism for long-context state allocation. 5.3 Hyperparameter Sensitivity Figure 3 presents the controlled sensitivity analysis of routing budgetKand Local Pooling kernelkon Method0 to 4k4 to 8k8k+ Base Model14.985.453.85 LAMB22.4122.3219.15 SuPER-Mamba23.6421.1819.85 Table 5: LongBench-E results of Mamba2-1.3B across different context-length ranges. The 0–4k, 4–8k, and 8k+ columns correspond to the three LongBench-E context-length ranges. Each column uses the official task-specific metric for each dataset and reports the equal average across all 13 datasets. a subset of LongBench-E samples. Different task categories exhibit distinct score distributions and best observed settings. Summarization and Syn- thetic tasks show different responses to the routing budget, while Multi-Document QA and Code show clearer variation across kernel settings. These re- sults indicate that routing budget and local aggre- gation range affect task categories differently. Based on these observations, we use category- level configurations rather than a single global set- ting. For categories requiring finer shared selection, we further perform a restricted joint refinement in which datasets within the same category share one configuration. The selected configurations are then frozen and kept unchanged during the full LongBench-E evaluation. The controlled analy- sis is therefore used to characterize broad param- eter sensitivity, while the subsequent refinement resolves the final shared configuration within a nar- rower search space. Details of the category-level hyperparameter selection protocol are provided in Appendix D. 5.4 Length-wise Analysis Table 5 shows that SuPER-Mamba remains effec- tive as context length increases. The Base Model decreases from 14.98 for inputs from 0 to 4k to 3.85 for inputs above 8k, whereas SuPER-Mamba main- tains scores of 23.64 and 19.85 in the same ranges. In the longest range, SuPER-Mamba also exceeds LAMB by approximately 3.7%. The slower degra- dation of SuPER-Mamba indicates that selective long range state updates remain effective for longer contexts and help mitigate the loss of earlier infor- mation as recurrent processing continues. 6 Discussion SuPER-Mamba demonstrates that predictive feed- back produced by pretrained Mamba models can serve as an effective control signal for long-context state allocation. By constructing routing scores 7 6.0 7.5 9.0 K=1536 Single-Doc QA 4.5 6.0 7.5 K=256 Multi-Doc QA 9 12 15 K=2048 Summarization 20 30 40 K=768 Few-Shot 0.0 2.5 5.0 K=256 Synthetic 39 42 45 K=768 Code (a) Routing budget K 8.0 8.4 8.8 k=9 Single-Doc QA 7.2 7.8 8.4 k=12 Multi-Doc QA 11 12 13 14 k=21 Summarization 30 35 40 k=21 Few-Shot 2 3 4 k=21 Synthetic 43.5 45.0 46.5 k=12 Code (b) Local Pooling kernel k Figure 3: Hyperparameter sensitivity analysis on a LongBench-E subset using Mamba2-1.3B. The horizontal axis represents the six task categories, and the vertical axis reports the corresponding category-level LongBench-E score obtained by equally averaging the results of the datasets within each category. Each configuration is evaluated on 50 samples from each of the 13 datasets. (a) analyzes the routing budgetKwithk = 18and Mean Pooling fixed, testingK ∈256, 512, 768, 1024, 1536, 2048, 3072. (b) analyzes the Local Pooling kernelkwithK = 1024and Mean Pooling fixed, testingk ∈ 3, 5, 7, 9, 12, 15, 18, 21. Boxes summarize the result distributions across all tested settings for each category, gray diamonds indicate the common reference configuration withK = 1024, k = 18, and Mean Pooling, and red circles indicate the best observed setting for each category in the corresponding parameter analysis. from Surprisal during context processing, SuPER- Mamba determines long-range state updates with- out downstream-query-dependent context scoring or additional selector training. The results across Mamba1-1.4B, Mamba2-1.3B, and Zamba2-1.2B suggest that such model-intrinsic predictive sig- nals provide a practical basis for training-free long- context routing across different architectures. Our experiments show that restricting Global- Channel updates under a fixed budget is beneficial. Performance rises from 8.09 (Base Model, where every position updates the Global Channels; Ta- ble 1) to 19.33 when only a limited number of randomly selected positions are allowed to perform these updates (Appendix Table 8). This is consis- tent with the observation that selectively restricting long-range state updates improves performance (Ye et al., 2025a,b), which raises the question of how much additional value the Surprisal signal provides beyond any reasonable selection policy. Our results show that Raw Surprisal alone (16.33) underperforms random selection, suggest- ing that unaggregated predictive responses are noisy. Local Pooling substantially improves the signal (20.85), surpassing the Random Control, and the full configuration reaches 21.56. Thus, while selective allocation accounts for a large fraction of the gains, the combination of Surprisal with lo- cal aggregation yields a consistently stronger rout- ing policy than random selection under the same budget. The Surprisal-based scores therefore con- tribute meaningful structure beyond mere sparsity. 7 Conclusion This paper presents SuPER-Mamba, a Surprisal- guided query-independent routing method for long- context modeling in pretrained Mamba models. SuPER-Mamba constructs positional scores from predictive feedback produced during context pro- cessing and applies Local Pooling to aggregate neighboring responses, enabling the allocation of long-range state updates without additional se- lector training or downstream-query-dependent context scoring. Experiments on LongBench-E show that SuPER-Mamba consistently improves the Base Model and achieves competitive perfor- mance across Mamba1-1.4B, Mamba2-1.3B, and Zamba2-1.2B. The ablation study further demon- strates that Surprisal provides an effective pre- dictive signal for long-range routing and that Lo- cal Pooling substantially strengthens its effective- ness, while the length-wise analysis shows that the method remains effective under longer context con- ditions. Overall, SuPER-Mamba demonstrates that predictive feedback from pretrained models can serve as an effective signal for long-range informa- tion routing, providing a training-free approach to long-context modeling in Mamba. 8 8 Limitations SuPER-Mamba currently uses fixed task-category- specific routing configurations, which cannot adapt to differences across individual inputs. In addi- tion, the offline design requires an extra context- scoring pass before routed inference, introducing additional computation and latency. Future work could explore input-adaptive routing and more effi- cient score computation while preserving the query- independent and training-free properties of the framework. References Ameen Ali Ali, Itamar Zimerman, and Lior Wolf. 2025. The hidden attention of Mamba models. In Proceed- ings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pages 1516–1534, Vienna, Austria. Associa- tion for Computational Linguistics. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024. LongBench: A bilingual, multi- task benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 3119–3137, Bangkok, Thailand. Association for Computational Linguistics. Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Yucheng Li, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Junjie Hu, and Wen Xiao. 2025. Pyra- midKV: Dynamic KV cache compression based on pyramidal information funneling. In Second Confer- ence on Language Modeling. Shaoshen Chen, Yangning Li, Zishan Xu, Yongqin Zeng, Shunlong Wu, Xinshuo Hu, Zifei Shan, Xin Su, Jiwei Tang, Yinghui Li, and Hai-Tao Zheng. 2025. DAST: Context-aware compression in LLMs via dynamic allocation of soft tokens. In Findings of the Asso- ciation for Computational Linguistics: ACL 2025, pages 20544–20552, Vienna, Austria. Association for Computational Linguistics. Tri Dao and Albert Gu. 2024. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 10041–10071. PMLR. Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. 2024. Model tells you what to discard: Adaptive KV cache compression for LLMs. In The Twelfth International Conference on Learning Representations. Paolo Glorioso, Quentin Anthony, Yury Tokpanov, Anna Golubeva, Vasudev Shyam, James Whitting- ton, Jonathan Pilault, and Beren Millidge. 2024. The zamba2 suite: Technical report.Preprint, arXiv:2411.15242. Albert Gu and Tri Dao. 2024. Mamba: Linear-time sequence modeling with selective state spaces. In First Conference on Language Modeling. John Hale. 2001. A probabilistic earley parser as a psy- cholinguistic model. In Second Meeting of the North American Chapter of the Association for Computa- tional Linguistics. Tamanna Hossain, Robert L. Logan IV, Chan- drasekhara Ganesh Jagadeesan, Sameer Singh, Joel R. Tetreault, and Alejandro Jaimes. 2025. Characteriz- ing Mamba’s selective memory using auto-encoders. In Proceedings of the 14th International Joint Con- ference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Asso- ciation for Computational Linguistics, pages 2007– 2022, Mumbai, India. The Asian Federation of Nat- ural Language Processing and The Association for Computational Linguistics. Xiang Hu, Jiaqi Leng, Jun Zhao, Kewei Tu, and Wei Wu. 2025. Hardware-aligned hierarchical sparse at- tention for efficient long-term memory access. In Advances in Neural Information Processing Systems, volume 38, pages 98788–98813. Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2023. LLMLingua: Compressing prompts for accelerated inference of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Process- ing, pages 13358–13376, Singapore. Association for Computational Linguistics. Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dong- sheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2024. LongLLMLingua: Accelerating and enhanc- ing LLMs in long context scenarios via prompt com- pression. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 1658–1677, Bangkok, Thailand. Association for Computational Linguistics. Aakash Sunil Lahoti, Kevin Li, Berlin Chen, Caitlin Wang, Aviv Bick, Zico Kolter, Tri Dao, and Albert Gu. 2026. Mamba-3: Improved sequence modeling using state space principles. In The Fourteenth Inter- national Conference on Learning Representations. Weronika Łajewska, Momchil Hardalov, Laura Aina, Neha Anna John, Hang Su, and Lluis Marquez. 2025. Understanding and improving information preserva- tion in prompt compression for LLMs. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 17520–17541, Suzhou, China. Association for Computational Linguistics. Roger Levy. 2008. Expectation-based syntactic compre- hension. Cognition, 106(3):1126–1177. 9 Yucheng Li, Bo Dong, Frank Guerin, and Chenghua Lin. 2023. Compressing context to enhance inference ef- ficiency of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natu- ral Language Processing, pages 6342–6353, Singa- pore. Association for Computational Linguistics. Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. 2024. SnapKV: LLM knows what you are looking for before genera- tion. In Advances in Neural Information Processing Systems, volume 37. Xiang Liu, Zhenheng Tang, Peijie Dong, Zeyu Li, Yue Liu, Bo Li, Xuming Hu, and Xiaowen Chu. 2025. ChunkKV: Semantic-preserving KV cache compres- sion for efficient long-context LLM inference. In Advances in Neural Information Processing Systems, volume 38, pages 32533–32583. Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyril- lidis, and Anshumali Shrivastava. 2023.Scis- sorhands: Exploiting the persistence of importance hypothesis for LLM KV cache compression at test time. In Advances in Neural Information Processing Systems, volume 36, pages 52342–52364. Peng Lu, Jerry Huang, Qiuhao Zeng, Xinyu Wang, Box- ing Chen, Philippe Langlais, and Yufei Cui. 2025. Mamba modulation: On the length generalization of Mamba models. In Advances in Neural Information Processing Systems, volume 38. Jun-Yu Ma, Tianqing Fang, Zhisong Zhang, Hongming Zhang, Haitao Mi, and Dong Yu. 2025. Recall with reasoning: Chain-of-thought distillation for Mamba’s long-context memory and extrapolation. In Proceed- ings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 4714–4720, Suzhou, China. Association for Computational Lin- guistics. Hugo Pitorro and Marcos Vinicius Treviso. 2025. La- TIM: Measuring latent token-to-token interactions in Mamba models. In Proceedings of the 63rd An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 24478– 24493, Vienna, Austria. Association for Computa- tional Linguistics. Haochen Shen, Davis Wertheimer, Zheng Wang, Garrett Goon, Derrick Liu, Naigang Wang, Mudhakar Sri- vatsa, Raghu Ganti, and Minjia Zhang. 2026. From collapse to control: Understanding and extending context length in emerging hybrid models via univer- sal position interpolation. In The Fourteenth Interna- tional Conference on Learning Representations. Chihiro Taguchi, Seiji Maekawa, and Nikita Bhutani. 2025. Efficient context selection for long-context QA: No tuning, no iteration, just adaptive-k. In Pro- ceedings of the 2025 Conference on Empirical Meth- ods in Natural Language Processing, pages 20105– 20130, Suzhou, China. Association for Computa- tional Linguistics. Jiwei Tang, Jin Xu, Tingwei Lu, Zhicheng Zhang, Yim- ing Zhao, Lin Hai, and Hai-Tao Zheng. 2025. Per- ception compressor: A training-free prompt compres- sion framework in long context scenarios. In Find- ings of the Association for Computational Linguistics: NAACL 2025, pages 4093–4108, Albuquerque, New Mexico. Association for Computational Linguistics. Qinwen Wang, Jieping Luo, Aoxiang Qin, Ruoyu Zhao, Jianxiong Tang, Wei Zhang, Zhichao Lu, and Luzi- wei Leng. 2026. Mamba with hierarchical memory: Solving representation bottleneck in long sequence modeling. arXiv preprint arXiv:2608.02347. Yihang Wang, Xu Huang, Bowen Tian, Yueyang Su, Lei Yu, Huaming Liao, Yixing Fan, Jiafeng Guo, and Xueqi Cheng. 2025. QUITO-X: A new perspec- tive on context compression from the information bottleneck theory. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 6841–6856, Suzhou, China. Association for Compu- tational Linguistics. Qingyue Yang, Jie Wang, Xing Li, Zhihai Wang, Chen Chen, Lei Chen, Xianzhi Yu, Wulong Liu, Jianye Hao, Mingxuan Yuan, and Bin Li. 2025. Attention- Predictor: Temporal patterns matter for KV cache compression. In Advances in Neural Information Processing Systems, volume 38. Zhifan Ye, Zheng Wang, Kejing Xia, Jihoon Hong, Leshu Li, Lexington Whalen, Cheng Wan, Yonggan Fu, Yingyan Celine Lin, and Souvik Kundu. 2025a. LAMB: A training-free method to enhance the long- context understanding of SSMs via attention-guided token filtering. In Proceedings of the 63rd Annual Meeting of the Association for Computational Lin- guistics (Volume 2: Short Papers), pages 1200–1209, Vienna, Austria. Association for Computational Lin- guistics. Zhifan Ye, Kejing Xia, Yonggan Fu, Xin Dong, Jihoon Hong, Xiangchi Yuan, Shizhe Diao, Jan Kautz, Pavlo Molchanov, and Yingyan Celine Lin. 2025b. Long- Mamba: Enhancing mamba’s long-context capabili- ties via training-free receptive field enlargement. In The Thirteenth International Conference on Learning Representations. WangJie You, Zecheng Tang, Juntao Li, Lili Yao, and Min Zhang. 2025. Revealing and mitigating the local pattern shortcuts of Mamba. In Findings of the As- sociation for Computational Linguistics: ACL 2025, pages 12156–12178, Vienna, Austria. Association for Computational Linguistics. Danlong Yuan, Jiahao Liu, Bei Li, Huishuai Zhang, Jingang Wang, Xunliang Cai, and Dongyan Zhao. 2025. ReMamba: Equip mamba with effective long- sequence modeling. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 6830–6840, Suzhou, China. Association for Compu- tational Linguistics. 10 JungMin Yun and YoungBin Kim. 2026. IterCOMP: Reasoning-aware adaptive prompt compression for multi-hop question answering. In Proceedings of the 64th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 33827–33840, San Diego, California, United States. Association for Computational Linguistics. Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuan- dong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen. 2023. H 2 O: Heavy-hitter oracle for efficient generative inference of large lan- guage models. In Advances in Neural Information Processing Systems, volume 36. Liang Zhao, Xiaocheng Feng, Weihong Zhong, Lei Huang, Kun Zhu, Baoxin Wang, Dayong Wu, Guop- ing Hu, Ting Liu, and Bing Qin. 2026. Question tells you where the answer is: Intention-aware long- context KV cache compression. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 27153–27169, San Diego, California, United States. Association for Computational Linguistics. 11 A Dataset Categories and Routing Configurations We group the 13 LongBench-E datasets into six task categories. Datasets within the same cate- gory share the selection budgetK, Local Pooling kernel sizek, and pooling operator. All category- level configurations are determined before the full LongBench-E evaluation and remain fixed through- out the main experiments. Table 6 provides the complete dataset-to-category mapping and the cor- responding routing configurations. All config- urations use a fixed recent-position window of R = 32. B Detailed Routing Implementation SuPER-Mamba follows an offline two-pass infer- ence procedure. The first pass constructs the rout- ing scores and the corresponding routing mask, while the second pass performs routed inference using the fixed mask. The complete prompt is pre- served throughout both passes. SuPER-Mamba does not remove or truncate Content positions and instead controls which positions are allowed to up- date the predefined Global Channels. During the first pass, Surprisal scoring is per- formed only on the Content and does not use the downstream Query. Position-wise predictive re- sponses are computed for the Content and aggre- gated by Local Pooling to obtain the routing scores. These scores are then associated with their corre- sponding Content positions in the complete prompt. Instruction, Query, and other non-Content prompt positions do not participate in the Content Top-K selection and therefore do not compete for the se- lection budget. For Content routing, the most recentR = 32po- sitions are excluded from score-based historical se- lection and are always retained. Among the remain- ing historical Content positions, SuPER-Mamba selects theKpositions with the highest pooled scores. The final Content routing set therefore con- sists of the historical Top-Kpositions together with the Recent-Rpositions. When the Content length does not exceedK + R, all Content positions are retained. The resulting Content mask is then projected to the complete prompt. All non-Content posi- tions retain their original updates. Selected Con- tent positions retain their original updates on the Global Channels, whereas unselected Content posi- tions are prevented from updating only these chan- nels. Their remaining recurrent computation is unchanged. The same position-level routing mask is applied to the predefined Global Channels across Mamba layers. During the second pass, the model receives the complete Instruction, Content, and Query, while the routing mask obtained from the first pass remains fixed. This implementation keeps context scoring independent of the downstream Query while preserving the full input sequence dur- ing routed inference. C Surprisal and Local Pooling Implementation Details Surprisal Alignment.During the first pass, SuPER-Mamba tokenizes the Content indepen- dently and performs a forward pass using the pre- trained Mamba without routing modification. For each positiont > 1, the logits produced at po- sitiont− 1are compared with the ground-truth token at positiontusing cross-entropy, so the re- sulting Surprisal score is aligned with the predicted token position rather than the position producing the logits. The first Content token has no preced- ing prediction within the independently processed Content sequence and is therefore assigned a Sur- prisal score of zero. Logits may be computed in chunks to reduce memory usage without changing the resulting position-wise Surprisal scores. Projection to the Complete Prompt. Because independently tokenizing the Content does not necessarily produce token boundaries identical to those of the same Content span inside the com- plete prompt, SuPER-Mamba does not assume a direct one-to-one correspondence between the two tokenizations. The implementation first iden- tifies the exact character span of the Content in the rendered prompt and obtains tokenizer offset mappings for both the Content-only sequence and the complete prompt. Prompt tokens overlapping the Content span are then identified. For each such prompt token, Surprisal scores from overlap- ping Content-only tokens are combined using a character-overlap-weighted average. If no direct character overlap is available, the score of the near- est Content token by character-center distance is used. This projection produces one routing score for each Content position in the complete prompt. Local Pooling. For a given kernel sizek, SuPER- Mamba applies stride-1 Mean Pooling or Max Pool- ing to aggregate Surprisal responses over neighbor- ing positions, using padding ofk/2according to 12 DatasetCategoryKKernelPooling QasperSingle-doc QA102418Max MultiFieldQA-enSingle-doc QA102418Max HotpotQAMulti-doc QA102410Mean 2WikiMQAMulti-doc QA102410Mean GovReportSummarization204818Mean MultiNewsSummarization204818Mean TRECFew-shot102418Max TriviaQAFew-shot102418Max SAMSumFew-shot102418Max Passage CountSynthetic25618Mean Passage Retrieval-enSynthetic25618Mean LCCCode102412Mean RepoBench-PCode102412Mean Table 6: Complete mapping from LongBench-E datasets to task categories and their corresponding category routing configurations. Different categories use different routing parameter configurations while following the same routing mechanism during inference. CategoryKKernel kPooling Single-doc QA102418Max Multi-doc QA102410Mean Summarization 204818Mean Few-shot102418Max Synthetic25618Mean Code102412Mean Table 7: Final category routing configurations used for the full LongBench-E evaluation.Kdenotes the selec- tion budget for historical Content positions, Kernelk denotes the Local Pooling window size, and Pooling denotes Mean or Max aggregation. These configura- tions are determined from the parameter analysis on a LongBench-E subset and remain fixed throughout the subsequent full evaluation. the pooling implementation. For even kernel sizes such ask = 18, the underlying pooling operation produces one additional output position. The fi- nal extra position is therefore removed so that the pooled score sequence has the same length as the original Content position sequence. The resulting Pooled Surprisal Scores are subsequently used for Content routing. D Hyperparameter Selection Protocol SuPER-Mamba adopts shared routing configura- tions at the task-category level. The 13 LongBench- E datasets are first grouped into six categories ac- cording to their task type and input-context struc- ture: Single-doc QA, Multi-doc QA, Summariza- tion, Few-shot, Synthetic, and Code. This catego- rization is defined independently of the subsequent hyperparameter experiments. Datasets within the same category share the same routing budgetK, Local Pooling kernel sizek, and pooling operator, avoiding dataset-specific hyperparameter optimiza- tion. After defining the task categories, we analyze the effects of the routing budgetK, Local Pooling ker- nel sizek, and pooling operator using Mamba2 on a subset of LongBench-E. We adopt a one-variable- at-a-time parameter sweep withK = 1024,k = 18 , and Mean Pooling as the common reference configuration. For the routing budget, we fix k = 18 and Mean Pooling and evaluate K ∈256, 512, 768, 1024, 1536, 2048, 3072. (9) For the Local Pooling kernel, we fixK = 1024 and Mean Pooling and evaluate k ∈3, 5, 7, 9, 12, 15, 18, 21.(10) The pooling operator is evaluated by compar- ing Mean and Max Pooling underK = 1024and k = 18. All 13 LongBench-E datasets are included in the parameter analysis, with 50 samples evalu- ated for each dataset under each configuration. The shared reference configuration is reused across the individual sweeps, resulting in 15 distinct configu- rations in total. Hyperparameter selection is performed within the predefined task categories. For each category, we jointly consider the performance of its con- stituent datasets under different configurations and select a shared setting that maintains consistently strong performance across the category. As re- ported in Section 5.3, the sensitivity analysis aggre- gates the dataset-level scores within each category 13 DatasetRandom ControlRaw Surprisal+ Local PoolingFull SuPER-Mamba Qasper4.123.364.855.00 MultiFieldQA-en13.0011.7014.4013.81 HotpotQA6.545.716.826.43 2WikiMQA8.838.3910.0010.09 GovReport12.5810.6513.8013.70 MultiNews12.8911.8514.3114.45 TREC29.335.3339.6740.33 TriviaQA42.1439.8944.8247.69 SAMSum25.1126.6727.8428.30 Passage Count3.904.224.294.00 Passage Retrieval-en4.634.734.255.19 LCC44.8040.3243.9449.03 RepoBench-P43.3839.4442.0742.20 Average19.3316.3320.8521.56 Table 8: Dataset-level component ablation results on LongBench-E using Mamba2-1.3B. Each dataset is evaluated using its official LongBench-E task-specific metric. Samples are divided into the 0–4k, 4–8k, and 8k+ context-length ranges, and the three range-level scores are averaged arithmetically to obtain the dataset-level score. The Average row reports the arithmetic mean over all 13 dataset-level scores. Bold values indicate the best result among the four component settings. using an equal average to characterize the effect of individual hyperparameters. The final configura- tions are determined from the experimental results at the category level rather than by applying a sin- gle automatic optimization rule. The resulting category-level routing configura- tions are summarized in Table 7. These configura- tions are fixed before the full LongBench-E evalua- tion and remain unchanged throughout the subse- quent experiments. E Additional Experimental Results This section provides additional dataset-level and length-wise results for the component ablation study reported in Section 5.2. All results are ob- tained on Mamba2 using the same LongBench-E evaluation protocol and task-category-specific rout- ing budgets as in the main experiments. E.1 Dataset-level Component Ablation Table 8 reports the dataset-level results for Ran- dom Control, Raw Surprisal, Surprisal with Lo- cal Pooling, and Full SuPER-Mamba across all 13 LongBench-E datasets. The four variants are evaluated under the same task-category-specific routing budgets, allowing the effects of different routing signals and aggregation components to be compared at the dataset level. The dataset-level results show that the effects of individual routing components vary across tasks. Raw Surprisal substantially improves overall per- formance over the Base Model reported in the main text, but its effectiveness is not uniform across datasets.Introducing Local Pooling increases the average score from 16.33 to 20.85 and im- proves over Raw Surprisal on 12 of the 13 datasets. This result provides further evidence that aggregat- ing neighboring predictive responses substantially strengthens Surprisal-based routing. Full SuPER-Mamba further increases the aver- age score to 21.56 and outperforms the Local Pool- ing variant on 9 of the 13 datasets. The strongest variant is not identical across every individual task, indicating that the contribution of the complete routing configuration can vary with task character- istics. Nevertheless, the aggregate improvement is distributed across multiple task categories rather than being attributable to a single dataset. E.2 Length-wise Component Results Table 9 further reports the component ablation re- sults across the three LongBench-E context-length ranges. For each range, the reported score is the arithmetic mean over the same 13 datasets. The length-wise results are consistent with the overall ablation findings. Raw Surprisal performs below the Local Pooling variant across all three context-length ranges. Local Pooling increases the average scores to 23.05, 20.45, and 19.06 for the 0– 4k, 4–8k, and 8k+ ranges, respectively, indicating that the benefit of local aggregation is not restricted to a particular input-length regime. Full SuPER-Mamba achieves the highest aver- age score in all three ranges, reaching 23.64, 21.18, and 19.85, respectively. In the 8k+ range, perfor- 14 Variant0–4k4–8k8k+Average Random Control21.4518.5118.0319.33 Raw Surprisal19.1715.8713.9416.33 + Local Pooling23.0520.4519.0620.85 Full SuPER-Mamba23.6421.1819.8521.56 Table 9: Length-wise component ablation results on LongBench-E using Mamba2-1.3B. The 0–4k, 4–8k, and 8k+ columns correspond to the three context-length ranges. Each range uses the official LongBench-E task- specific metric for each dataset and reports the equal average across all 13 datasets. Bold values indicate the best result in each column. mance increases from 13.94 with Raw Surprisal to 19.85 with the complete method. These results further indicate that combining predictive feedback with Local Pooling and the complete routing con- figuration provides more robust long-context per- formance than position-level Surprisal alone. 15