Paper deep dive
Spectral Attention Steering for Prompt Highlighting
Weixian Waylon Li, Yuchen Niu, Yongxin Yang, Keshuang Li, Tiejun Ma, Shay B. Cohen
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 7/20/2026, 1:40:15 AM
Summary
The paper introduces Spectral Editing Key Amplification (SEKA) and its adaptive variant AdaSEKA, training-free methods for attention steering in Large Language Models. Unlike existing methods that require storing full attention matrices (incompatible with FlashAttention), SEKA edits key embeddings before attention computation using spectral decomposition to amplify attention for highlighted tokens. AdaSEKA adds a query-adaptive routing mechanism to combine multiple expert subspaces. Experiments show these methods outperform baselines like PASTA with significantly lower latency and memory overhead.
Entities (8)
Relation Signals (7)
AdaSEKA → isvariantof → SEKA
confidence 95% · We extend this to Adaptive SEKA (AdaSEKA), a query-adaptive variant...
SEKA → isvariantof → Attention Steering
confidence 95% · We introduce Spectral Editing Key Amplification (SEKA), a training-free steering method...
SEKA → usestechnique → Spectral Decomposition
confidence 93% · SEKA uses spectral decomposition to steer key embeddings towards latent directions...
SEKA → iscompatiblewith → FlashAttention
confidence 92% · Our experiments show both methods significantly outperform strong baselines... in compatibility with optimised attention.
PASTA → isincompatiblewith → FlashAttention
confidence 90% · making them incompatible with memory-efficient implementations like FlashAttention.
AdaSEKA → usesmechanism → Query-Adaptive Routing
confidence 90% · AdaSEKA... uses a training-free routing mechanism to dynamically combine multiple expert subspaces based on the prompt's semantic intent.
SEKA → outperforms → PASTA
confidence 88% · Our experiments show both methods significantly outperform strong baselines on standard steering benchmarks...
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Attention steering is an important technique for controlling model focus, enabling capabilities such as prompt highlighting, where the model prioritises user-specified text. However, existing attention steering methods require explicit storage of the full attention matrix, making them incompatible with memory-efficient implementations like FlashAttention. We introduce Spectral Editing Key Amplification (SEKA), a training-free steering method that tackles this by directly editing key embeddings before attention computation. SEKA uses spectral decomposition to steer key embeddings towards latent directions that amplify attention scores for certain tokens. We extend this to Adaptive SEKA (AdaSEKA), a query-adaptive variant that uses a training-free routing mechanism to dynamically combine multiple expert subspaces based on the prompt's semantic intent. Our experiments show both methods significantly outperform strong baselines on standard steering benchmarks while adding much lower latency and memory overhead, in compatibility with optimised attention.
Tags
Links
- Source: https://arxiv.org/abs/2603.01281v1
- Canonical: https://arxiv.org/abs/2603.01281v1
Trouble viewing inline? Open PDF directly →
Full Text
96,238 characters extracted from source content.
Expand or collapse full text
Published as a conference paper at ICLR 2026 SPECTRAL ATTENTION STEERING FOR PROMPT HIGH- LIGHTING Weixian Waylon Li 1 , Yuchen Niu 2 , Yongxin Yang 4 , Keshuang Li 3 , Tiejun Ma 1 , Shay B. Cohen 1 1 University of Edinburgh, UK 2 RayNeo, China 3 Huawei Research Ltd., UK 4 Queen Mary University of London, UK waylon.li,tiejun.ma@ed.ac.uk scohen@inf.ed.ac.uk ABSTRACT Attention steering is an important technique for controlling model focus, en- abling capabilities such as prompt highlighting, where the model prioritises user- specified text. However, existing attention steering methods require explicit stor- age of the full attention matrix, making them incompatible with memory-efficient implementations like FlashAttention. We introduce Spectral Editing Key Amplifi- cation (SEKA), a training-free steering method that tackles this by directly editing key embeddings before attention computation. SEKA uses spectral decomposition to steer key embeddings towards latent directions that amplify attention scores for certain tokens. We extend this to Adaptive SEKA (AdaSEKA), a query-adaptive variant that uses a training-free routing mechanism to dynamically combine mul- tiple expert subspaces based on the prompt’s semantic intent. Our experiments show both methods significantly outperform strong baselines on standard steering benchmarks while adding much lower latency and memory overhead, in compati- bility with optimised attention. 1INTRODUCTION The ability to precisely guide the behaviour of large language models (LLMs) is paramount as they are increasingly deployed in high-stakes domains. This broad field of model steering encompasses various techniques, from activation steering, which aims to control high-level semantic attributes like style or factual recall by intervening in MLP layers (Subramani et al., 2022; Turner et al., 2023; Qiu et al., 2024; Turner et al., 2024; Wang et al., 2025; Stolfo et al., 2025), to attention steering, which operates at a more granular level to direct the model’s focus to specific tokens within a prompt. This paper focuses on the latter, where prompt highlighting is one of the key applications. Current state- of-the-art methods, such as PASTA (Zhang et al., 2024), operate by editing the attention score matrix after it has been computed. This post-hoc manipulation creates a critical bottleneck: it requires computing the full attention matrix, making these methods incompatible with modern, IO-aware implementations like FlashAttention (Dao et al., 2022; Dao, 2024) that are essential for efficient processing. This architectural limitation, coupled with the need for costly, task-specific searches to identify which attention heads to steer, makes them less practical. In this paper, we propose to intervene in the input of the attention mechanism rather than edit its output. We introduce Spectral Editing Key Amplification (SEKA), a novel, training-free framework that steers attention by directly modifying key vectors before the attention scores are calculated. Our core insight is that we can learn a universal “relevance subspace” for a given task by applying spec- tral decomposition to key embeddings derived from contrastive prompts. These learned directions are then used to construct a projection matrix that amplifies the relevant features of highlighted keys via a simple, geometrically interpretable transformation: k ′ = k + gP k. Additionally, we propose Adaptive SEKA (AdaSEKA), an advanced variant that learns a bank of task-specific “expert” projections (e.g., for factual recall versus instruction following). At inference time, AdaSEKA uses a computationally cheap, training-free routing mechanism to create a dynamic, query-aware steering operator by blending these experts based on the prompt’s semantic intent. Our 1 arXiv:2603.01281v1 [cs.CL] 1 Mar 2026 Published as a conference paper at ICLR 2026 method is fully compatible with FlashAttention as it operates directly on the key embeddings with negligible computational overhead. Our experiments confirm the effectiveness of this approach. Both SEKA and AdaSEKA achieve su- perior results on standard benchmarks for knowledge conflicts, occupation extraction, and instruc- tion following. Furthermore, AdaSEKA’s query-adaptive routing mechanism demonstrates superior performance by dynamically tailoring the steering to the prompt’s semantic intent. Crucially, we show that these performance gains are achieved with negligible overhead. SEKA adds only≈0.03s of latency per sample, in stark contrast to comparable methods like PASTA which incur a +1.03s inference time and nearly double the memory usage. 2PROBLEM DEFINITION AND MOTIVATIONS In this section, we formalise the problem of prompt highlighting as an instance of attention bias and present the motivation for our spectral attention steering approach, which aims to address the limitations of existing methods. Problem Definition. Given a prompt x = (x 1 ,...,x T ) consisting of T tokens, with a subset of token indices H ⊂ 1,...,T identifying the highlighted tokens (in practice, surrounded by markers such as ** ), our goal is to steer the attention of the model so that these tokens receive increased focus from queries. In standard multi-head attention, the unnormalised attention score between query i and key j is Attn(i,j) = q ⊤ i k j √ d k , where q i , k j ∈ R d k are the query and key vectors, and d k is the head dimension. Objective. We aim to amplify the attention assigned to highlighted tokens by introducing an ad- ditive, controllable term to the attention score for each (i,j) where j ∈H: A ′ ij = A ij + ∆ ij , where ∆ ij is designed to selectively boost the attention towards user-specified highlighted tokens. Motivation. Existing approaches typically modify attention after it has been computed. For ex- ample, PASTA (Zhang et al., 2024) rescales rows of the attention matrix as shown in equation 1, where C i is a row normalisation factor and α > 1 scales attention to highlighted tokens. [T (A)] ij = α A ij C i , if j ∈H, A ij C i ,otherwise. (1) Similarly, positional calibration methods such as Found-in-the-Middle (Hsieh et al., 2024) subtract a baseline from the positional attention bias. Let x k denote the position of the k-th token, and Attn ori (x k ) the original positional bias. The calibrated bias is Attn calibrated (x k ) = Attn ori (x k ) − Attn baseline (x k ), where Attn baseline (x k ) is estimated independently of content relevance. Both strategies require explicit storage of the full attention matrix, which is incompatible with memory-efficient implementations such as FlashAttention (Dao et al., 2022; Dao, 2024). More- over, methods like PASTA often rely on costly head search to decide which attention heads to steer. These limitations motivate the consideration for an alternative steering mechanism that operates be- fore attention scores are computed, avoiding any need to materialise or modify the attention matrix. Since attention depends on query–key inner products, equivalent control can be achieved by editing either representation (shown in Section 3.2). Given our objective of amplifying attention to a spe- cific subset of tokensH, key-side intervention is the natural choice: the key vector k j is indexed by token position j and therefore governs how strongly each individual token is attended to. To provide empirical evidence on whether such a pre-attention intervention is feasible, we analyse how key representations change under shifts in contextual relevance. We first construct synthetic contrastive prompt triplets under three conditions: (1) neutral (context only), (2) positive (context aligned with a relevant query), and (3) negative (context paired with an irrelevant query). The construction of such synthetic triplets is described in Appendix A. Using the Qwen3-1.7B-Base model (28 layers, 8 heads), we extract the key embeddings correspond- ing to the same token spans under both positive and negative prompts for each (layer, head) pair. We 2 Published as a conference paper at ICLR 2026 PCA Component 1 PCA Component 2 Layer 20 - Head 3 (Pairwise Shift) Positive Negative Mean shift PCA Component 1 PCA Component 2 Layer 21 - Head 3 (Pairwise Shift) Positive Negative Mean shift PCA Component 1 PCA Component 2 Layer 22 - Head 3 (Pairwise Shift) Positive Negative Mean shift PCA Component 1 PCA Component 2 Layer 24 - Head 7 (Pairwise Shift) Positive Negative Mean shift PCA Component 1 PCA Component 2 Layer 25 - Head 3 (Pairwise Shift) Positive Negative Mean shift PCA Component 1 PCA Component 2 Layer 25 - Head 4 (Pairwise Shift) Positive Negative Mean shift PCA Component 1 PCA Component 2 Layer 25 - Head 7 (Pairwise Shift) Positive Negative Mean shift PCA Component 1 PCA Component 2 Layer 26 - Head 3 (Pairwise Shift) Positive Negative Mean shift PCA Component 1 PCA Component 2 Layer 27 - Head 5 (Pairwise Shift) Positive Negative Mean shift Figure 1: Visualisation of pairwise key embedding shifts across different (layer, head) in Qwen3- 1.7B-Base via PCA. Positive vs. negative representations are plotted for 26 shared token spans. Grey arrows trace individual shifts; the dark blue arrow shows the average displacement. then apply PCA to jointly project these paired embeddings into two dimensions, and visualise the result using a combination of scatter plots and directed arrows. Each arrow originates from a neg- ative key and points to its corresponding positive key, capturing the pairwise representational shift induced by changing question relevance. To summarise the overall trend, we also plot the mean shift vector across all pairs. Figure 1 shows that certain heads exhibit robust and consistent directional shifts in key embeddings when token relevance changes. Each plot visualises 26 key embedding pairs corresponding to shared token spans, extracted from 10 positive–negative prompt pairs. These findings suggest that relevance is encoded in a structured subspace of key representations, motivating our approach that edits key embeddings before attention is computed: k ′ j = k j +gP k j , where P is a projection matrix (defining a relevance subspace per key-value head), and g is a scaling coefficient. This preserves compatibility with efficient attention implementations while providing a geometrically interpretable mechanism for steering attention towards highlighted tokens. 3SPECTRAL ATTENTION STEERING FOR PROMPT HIGHLIGHTING As shown in Figure 2, we propose a new method, Spectral Editing Key Amplification (SEKA), and its query-adaptive variant, AdaSEKA. Both methods achieve prompt highlighting by directly editing key embeddings before the attention computation. The core mechanism of SEKA is inspired by the Spectral Editing of Activations (SEA) algorithm (Qiu et al., 2024), adapting it from semantic-level activation steering to the token-wise attention steering required for prompt highlighting. 3.1SPECTRAL LEARNING OF RELEVANCE-ALIGNED PROJECTIONS (OFFLINE) Using the token-level key embeddings obtained from the aforementioned synthetic contrastive prompts (Section 2 and Appendix A), denoted h (neutral), h + (positive), and h − (negative), we 3 Published as a conference paper at ICLR 2026 풙 ା 풙 ି Key Projection 풉 ା 풉 풉 ି 훀 ା 훀 ି 푼 ା 푺 ା 푼 ି 푺 ି ... ... 푼 ା 푻 푼 ି 푻 AttentionHead Highlighted Tokens' Keys 품 ି 품 ା Query SVD SVD ... ...Edited Keys ... ... 푼 ା 푼 ା 푻 푼 ା 푼 ା 푼 ି 푺 ା ... ... 품 ା SEKAAdaSEKA Figure 2: An overview of SEKA and AdaSEKA. x: context; h: key embedding; Ω: cross-covariance; U : left singular vectors; S: singular values; g: gain coefficient. SEKA applies fixed gains, while AdaSEKA uses the query to compute dynamic steering weights. compute cross-covariance matrices for each transformer layer ℓ and key-value head h: Ω + ℓ,h = h ⊤ h + n , Ω − ℓ,h = h ⊤ h − n , where n is the number of sampled tokens. Singular value decomposition (SVD) is then applied: Ω + ℓ,h = U + ℓ,h S + ℓ,h V +⊤ ℓ,h , Ω − ℓ,h = U − ℓ,h S − ℓ,h V −⊤ ℓ,h . In SVD, S + ℓ,h and S − ℓ,h represent the singular values of the positive and negative cross-covariance matrices, respectively. These singular values quantify the magnitude of cross-covariance captured by each component of the projection. The larger the singular value, the more significant the cor- responding singular vector (projection direction) is in explaining the cross-covariance between the token key embeddings. In equation 2, for the positive projection P + ℓ,h , we use the top singular vectors corresponding to the largest singular values, which capture directions most associated with relevant (highlighted) features. For the negative projection P − ℓ,h , we use the least-significant singular vectors, associated with the smallest singular values, to target directions least associated with relevance. P + ℓ,h = U + ℓ,h,:,:k + (U + ℓ,h,:,:k + ) ⊤ , P − ℓ,h = U − ℓ,h,:,k − : (U − ℓ,h,:,k − : ) ⊤ ,(2) where k + and k − are chosen such that they capture at least a proportion γ of the total singular value sum: P k + i=1 S + ℓ,h,i P d k i=1 S + ℓ,h,i ≥ γ, P k − i=1 S − ℓ,h,i P d k i=1 S − ℓ,h,i ≥ γ.(3) The threshold γ is a hyperparameter that controls how much of the variance in the data we wish to retain when creating the projection matrices. By selecting the top k + singular vectors for the positive covariance and k − for the negative covariance, we capture the most relevant directions in the key embeddings for each type of projection. The learned projectorsP + ℓ,h , P − ℓ,h are stored per layer and head, enabling fine-grained steering at inference time. 3.2SPECTRAL EDITING FOR HIGHLIGHTED TOKENS (INFERENCE) During inference, SEKA injects the learned projections into key embeddings before attention scores are computed. For clarity, we omit the explicit (ℓ,h) indices on key vectors k j and queries q i , although they are in practice layer- and head-specific. For each token key k j ∈ R d k at layer ℓ and head h, the edited embedding is defined as: k ′ j = k j + g + · P + ℓ,h k j + g − · P − ℓ,h k j 2 ,(4) where P + ℓ,h , P − ℓ,h ∈ R d k ×d k are the selected projection matrices and g + ,g − are two independently adjustable scalars controlling the positive and negative steering gains. All vectors (e.g., k j , q i , 4 Published as a conference paper at ICLR 2026 x) are column vectors unless otherwise specified. This adjustment modifies the attention logits as equation 5, where q i ∈ R d k is the i-th query vector. It is algebraically equivalent to augmenting the original attention score matrix A with a low-rank relevance bias matrix B: Logits ij = q ⊤ i k j √ d k + q ⊤ i g + · P + ℓ,h k j + g − · P − ℓ,h k j 2 ! √ d k = A ij + B ij .(5) Thus, SEKA can be interpreted as adding a key-dependent term to the attention scores, amplifying each token’s the directions aligned with the relevance subspace (detailed in Appendix C). Unlike methods that directly manipulate the attention matrix, SEKA achieves equivalent modulation by edit- ing the key vectors themselves, offering a more structured and interpretable mechanism. Moreover, because SEKA operates entirely on key representations prior to attention computation, it requires no access to or storage of the attention matrix, making it inherently compatible with memory-efficient implementations like FlashAttention. 3.3VARIANT: QUERY-DRIVEN ADAPTIVE SEKA While the standard SEKA framework provides effective token-level attention steering, practical de- ployment often requires hyperparameter tuning across different tasks and model families due to the static projections. To address this limitation and reduce the need for manual configuration, we in- troduce Adaptive SEKA (AdaSEKA), which automatically selects and combines expert projections based on query-specific relevance signals. Multi-Expert Projection Learning. We extend the projection learning framework to accommo- date multiple domain-specific experts. For each expert 1 m ∈ 1,...,M, we constructed samples from datasets D m for different tasks. Each expert learns its own set of positive SVD components U + m,ℓ,h , S + m,ℓ,h , V + m,ℓ,h following the standard SEKA procedure. This process results in a set of SVD components for each expert, layer, and head, which can be represented as a 5D tensor (U + ∈ R M×L×H×d k ×d k ), where L is the number of layers, and H is the number of heads. Query-Adaptive Expert Routing. At inference time, we extract the query vector q ℓ,h at layer ℓ and head h of the last token in the prompt, as the last token serves as the global aggregator of prompt information and hugely influences the downstream generation (Barbero et al., 2024; Qiu et al., 2024). We then compute dynamic coefficients that determine the contribution of each expert: α m,ℓ,h (q ℓ,h ) = P K k=1 (q ⊤ ℓ,h u +(k) m,ℓ,h )· σ +(k) m,ℓ,h max m P K k=1 (q ⊤ ℓ,h u +(k) m,ℓ,h )· σ +(k) m,ℓ,h ,(6) where σ +(k) m,ℓ,h is the corresponding k-th singular value, and K is the number of top singular compo- nents used (typically K = 5). This formulation measures how well the query aligns with each expert’s main projection directions, weighted by their singular values. The denominator normalises by the largest absolute alignment across experts, which keeps the coefficients on a comparable scale and preserves whether the align- ment is positive or negative. The final projection matrix at layer ℓ and head h is constructed as a weighted combination of expert projections: P dynamic,ℓ,h (q ℓ,h ) = P M m=1 α m,ℓ,h (q ℓ,h )·U + m,ℓ,h,:,:K (U + m,ℓ,h,:,:K ) ⊤ , where U + m,ℓ,h,:,:K denotes the first K columns of U + m,ℓ,h , corresponding to the most significant singular vectors. This approach reconstructs projection matrices on-demand using only the top-K components, pro- viding computational efficiency whilst enabling automatic expert selection. The key transformation during inference becomes: k ′ j = k j + g· P dynamic,ℓ,h (q ℓ,h )k j . Crucially, AdaSEKA offers several practical advantages: (1) Reduced configuration effort: Auto- matic expert routing reduces the number of hyper-parameters tuning for different tasks and models 1 Experts can vary across task-specific datasets, such as factual correction and instruction-following. 5 Published as a conference paper at ICLR 2026 12345678910111213141516171819202122232425262728 Layer 1 2 3 4 5 6 7 8 Head Qwen3-1.7B-Base 0.0 0.2 0.4 Norm Value 123456789101112131415161718192021222324252627282930313233343536 Layer 1 2 3 4 5 6 7 8 Head Qwen3-4B-Base 0.0 0.1 0.2 Norm Value 123456789101112131415161718192021222324252627282930313233343536 Layer 1 2 3 4 5 6 7 8 Head Qwen3-8B-Base 0.1 0.2 Norm Value 12345678910111213141516171819202122232425262728293031323334353637383940 Layer 1 2 3 4 5 6 7 8 Head Qwen3-14B-Base 0.1 0.2 Norm Value Figure 3: Heatmaps of the average per-token ℓ 2 distance between positive and negative key embed- dings across all KV heads and layers for four Qwen3 model sizes. Higher values (green) indicate greater separation between positive and negative key representations. (shown in Appendix F). (2) Modular deployment: New experts can be integrated without recalcu- lating existing ones. (3) Interpretable routing: Expert selection is based on explicit query-expert alignment scores. We derive four expert projections from four distinct datasets. The process of constructing data samples for learning these projections is detailed in Appendix B. 3.4SELECTING RELEVANCE-SENSITIVE KEY-VALUE HEADS SEKA are most effective when applied selectively to KV heads that are naturally sensitive to prompt relevance. As demonstrated in the qualitative visualisations in Figure 1 and discussed in Section 2, the key embedding for a given token span consistently shift in vector space when the question in the prompt is changed from an irrelevant one to a relevant one. In this section, we formalise a method to quantify this relevance sensitivity across all layers and heads to inform our selection strategy. Figure 3 shows the ℓ 2 distance between positive and negative key embeddings, averaged over all answer tokens from our synthetic dataset (as defined in Appendix A). This variation is examined across different layers and heads of the Qwen3 model in various sizes. We observe that the distinction between relevant and irrelevant prompts is not uniform: larger norm values (green) consistently emerge in the mid-to-late layers, while early layers and a subset of heads display minimal shift (red), suggesting the retrieval behaviour is less likely to happen at those lay- ers. This finding is strongly aligned with recent mechanistic analyses. Michel et al. (2019); Voita et al. (2019); Clark et al. (2019); Neo et al. (2024); Li et al. (2023b) highlight that attention modules display various token-attending patterns across different heads. Qiu et al. (2025) demonstrate that retrieval effectiveness relies on only a subset of attention heads, identified via probing and relevance filtering. Wu et al. (2025) further show that this sparse set of “retrieval heads” are almost exclusively located in the mid-to-late layers of the transformer. These heads are intrinsic to the base models, re- main consistent after fine-tuning, and are dynamically activated according to the context. Therefore, motivated by this alignment, we restrict projection to only those (layer, head) pairs where the empir- ical ℓ 2 difference between positive and negative key embeddings exceeds a threshold. This selective approach ensures that attention steering is concentrated on components empirically associated with retrieval behaviour, while leaving other heads unaffected. In this way, we amplify relevance signals only where necessary, minimising unintended influence on unrelated model components. Formally, for each layer ℓ and head h, let S denote the set of all answer tokens (across all sam- ples in the data), with |S| = N . The average per-token ℓ 2 distance is computed as D ℓ,h = 1 N P N i=1 h + ℓ,h,i − h − ℓ,h,i 2 , where h + ℓ,h,i and h − ℓ,h,i are the positive and negative key embeddings for token i in S. Projection is applied only if D ℓ,h ≥ δ min , where δ min is a tunable hyperparameter tuned via grid search on a validation set (typically in [0, 0.6]). 4EXPERIMENTAL SETUP We consider SEKA particularly useful in scenarios that require emphasis or highlighting within the prompt. This includes the tasks used to evaluate PASTA (Zhang et al., 2024), which involve (i) han- dling complex user instructions (e.g., pronoun rewriting), (i) interpreting lengthy and noisy contexts (e.g., Bias in Bios; De-Arteaga et al. 2019), and (i) resolving in-context knowledge conflicts (e.g., 6 Published as a conference paper at ICLR 2026 CounterFact; Meng et al. 2022). In addition, SEKA enables us to invert the typical U-shaped per- formance observed in the “lost in the middle” setting (Liu et al., 2024) by simply highlighting the middle of long contexts, thus improving model recall for these challenging positions. 4.1STANDARD BENCHMARKS FOR ATTENTION STEERING We follow the standard benchmarks used by PASTA, ensuring consistent selection of highlighted to- kens. Table 1 summarises the tasks, prompt formats, and evaluation metrics. The CounterFact task is based on the COUNTERFACT dataset (Meng et al., 2022), while the remaining two tasks (Bias in Bios, Pronouns changing) are derived from the BIASBIOS dataset (De-Arteaga et al., 2019), in line with previous research (Zhang et al., 2024). We enhance the evaluation metric for the Pronouns changing task to address flaws in the original protocol which can misleadingly reward empty re- sponse, with the other metrics remaining consistent. Further details, including an introduction to each benchmark task and the calculation of metrics, are available in Appendix E. Table 1: Summary of standard benchmarks for attention steering. Tokens in bold indicate where attention steering is applied. TaskPromptsMetrics CounterfactPreviously, [old fact]. Currently, [new fact]. [question]. Efficacy score (ES), Para- phrase score (PS) Bias in Bios[person’s occupation]. [career history, may not directly related to prediction]. [person] has the occupation of a/an Accuracy (Acc.) Pronouns changing [biographical contexts]. Substitute ‘she’ and ‘he’ with ‘they’ and generate the occupation of [person] after changing pronouns. Pronoun-weightedLexical overlap Score (P. Score), All-changed P. Score Benchmark Methods. We begin by using direct prompting of the original model as a baseline. Additionally, we include another baseline that incorporates ** marks around the highlighted context. For attention steering methods, ** is solely used to determine the token indices for steering and is re- moved from the input IDs. We then benchmark our proposed methods, SEKA and AdaSEKA, against the existing attention steering method PASTA. We also compare with Selective Prompt Anchoring (SPA) (Tian & Zhang, 2025), a prompt highlighting method that operates on the logit distributions of the LLMs. Additionally, we evaluate SEKA with random projections applied and without the KV heads selector to serve as an ablation study. 4.2U-SHAPE INVERSION IN THE LOST-IN-THE-MIDDLE SETTING To further examine SEKA ’s ability to steer model attention to specific regions within a long con- text, we introduce an additional experiment targeting positional recall in the challenging lost-in- the-middle setting (Liu et al., 2024). This setting refers to the widely observed phenomenon where LLMs exhibit strong recall for information presented at the beginning and end of long contexts, but their performance substantially degrades when the relevant information is located in the middle, re- sulting in a characteristic U-shaped performance curve. Each of our inputs consists of a long context comprising 30 passages, where only one gold passage contains the true answer to a given question and the rest serve as distractors. The position of the gold passage is varied to test the model’s posi- tional sensitivity. Each input is formatted as: “Context: [P1 Title] [P1 Text] ... [P30 Title] [P30 Text] Question: ex[’question’] Answer:”. Unlike prior work that aims to mitigate this effect, our objective is to directly investigate whether explicit relevance highlighting via SEKA can invert this U-shaped curve. By steering attention to- wards the middle passages, we test if the typical performance trough for mid-context answers can be transformed into a peak, providing insight into the controllability of positional recall in LLMs. Metrics. We use exact match (EM) score as the evaluation metric, following Liu et al. (2024): a prediction is considered correct if it contains the ground-truth short answer span. To discourage verbose or off-topic completions, the generated answer is limited to a maximum of 60 tokens. 7 Published as a conference paper at ICLR 2026 Table 2: Performance on standard benchmarks. Bold = best. Underline= second best. We include two ablation studies for SEKA: “w/o learn” uses random projections instead of spectrally learned ones, and “w/o learn&filt” further removes the head filtering mechanism. ModelMetric BaselinesOur Methods Original**-markedPASTASPASEKAw/o learnw/o learn&filtAdaSEKA Qwen3-4B CounterFact (ES)45.0057.7097.1665.2499.0294.9686.1298.90 CounterFact (PS)45.6452.1296.0357.7198.6192.3886.2098.72 Bias in Bios (Acc.)79.8482.9489.5868.0091.0286.6271.7691.86 Pronoun (P. Score)93.1495.76 95.8280.2795.1890.4241.9894.54 Pronoun (A. P. Score)90.5293.8894.6478.1993.2688.6636.9592.08 Qwen3-8B CounterFact (ES)39.0456.2492.7069.2699.0896.1295.1899.00 CounterFact (PS)39.5949.8091.6858.7698.9694.7489.6998.97 Bias in Bios (Acc.)76.0880.6086.3237.0288.7487.2674.9088.50 Pronoun (P. Score)98.0098.1098.8672.6198.5698.1280.5399.68 Pronoun (A. P. Score)97.8497.8498.7274.9598.2697.9080.8599.52 Qwen3-14B CounterFact (ES)37.5645.5276.8484.2298.9286.2895.2699.00 CounterFact (PS)36.1240.1266.3376.1199.0288.0792.0299.15 Bias in Bios (Acc.)85.2290.9488.4657.8690.2888.0288.4491.22 Pronoun (P. Score)98.4298.8690.9891.6098.6696.3288.6099.88 Pronoun (A. P. Score)98.2298.6890.9492.2098.5496.3689.7699.86 Gemma3-4B CounterFact (ES)55.0457.5678.3693.9098.0495.1494.4698.74 CounterFact (PS)47.7745.8259.5391.9298.8392.2591.9899.05 Bias in Bios (Acc.)89.9091.0082.5848.0292.4285.6077.1692.92 Pronoun (P. Score)41.3438.8667.3976.0581.5353.5851.7893.76 Pronoun (A. P. Score)35.2532.4566.4374.4581.1148.8251.9493.58 Gemma3-12B CounterFact (ES)45.3448.7268.3093.7698.8663.0860.9692.48 CounterFact (PS)37.2136.6971.7291.2499.2750.5976.3793.65 Bias in Bios (Acc.)91.2692.9094.7246.8893.0491.8490.5491.14 Pronoun (P. Score)93.9295.7868.4786.4197.7047.2655.5696.88 Pronoun (A. P. Score)94.9696.4268.0184.9997.2451.2458.7695.84 Benchmark Methods. We compare SEKA against a standard baseline: directly prompting the base LLM without any intervention, and also PASTA. On top of this, we apply SEKA in two config- urations: (i) steering only the middle region of the context (specifically passages 4 through 25), and (i) steering all context passages. Although Hsieh et al. (2024) presents another potential baseline, we exclude it due to the unavailability of its code implementation. 5RESULTS 5.1STANDARD BENCHMARKS: SEKA PROVIDES EFFICIENT ATTENTION STEERING The main experimental results are presented in Table 2. We tested the Qwen3 model (Yang et al., 2025) in various sizes, including 4B, 8B, and 14B, as well as the Gemma3 model (Gemma Team, 2025) in sizes of 4B and 12B. For PASTA, we present its best performance from three configurations to ensure a robust comparison (see Appendix H for full details). Furthermore, specific examples and the corresponding outputs from both the original model and SEKA are available in Appendix I. The results demonstrate that SEKA and AdaSEKA, are highly effective at steering LLM attention, generally outperforming both baseline models (ranked among the top two most of the time) and existing methods across various tasks and model scales. As demonstrated in Section 6, these im- provements are achieved with significantly lower overhead compared to PASTA and SPA. A primary finding is the efficacy of attention-level interventions on tasks requiring factual recall. On CounterFact, both SEKA and PASTA achieve near-perfect scores (e.g., 99.02 and 97.16 respectively for Qwen3-4B), validating the general approach of steering attention for knowledge conflicts, while the logit-based SPA lags considerably. Within this effective category, our methods consistently hold a performance advantage. This trend continues in the Bias in Bios task, where SEKA and AdaSEKA generally secure the top two positions across all models. Performance on the instruction-following Pronoun Changing task is strongly correlated with the base model’s pretrained sensitivity to simple emphasis markers. For the Qwen3 family, which is partially responsive to simple markdown emphasis, the “**-marked” baseline is notably strong. This contrasts with earlier conclusions that LLMs are inherently restricted to processing plain text without 8 Published as a conference paper at ICLR 2026 stylistic cues or emphasis markers (Brown et al., 2020; Wei et al., 2022). However, AdaSEKA still provides further improvement, delivering SOTA performance (e.g., an A. P. Score of 99.52 on Qwen3-8B). The advantage of our methods is most pronounced on the Gemma3-4B which is less responsive to the markdown emphasis. This demonstrates our method’s significant value, especially for smaller models that are less receptive to basic emphasis grammar. Finally, our ablation studies validate the method’s core components. Using random projections with head filtering (w/o learn) proves beneficial but is clearly suboptimal, underscoring the value of our spectral learning approach. Removing both the learned projections and the head-filtering mechanism (w/o learn&filt) causes a catastrophic decline in performance. For instance, on the Qwen3-4B Pronoun task, the A. P. Score drops from the original 90.52 to 36.95. This conclusively demonstrates that both learning meaningful relevance subspaces and selectively applying them to the appropriate KV heads are essential for success. 5.2LOST IN THE MIDDLE With the setting described in Section 4.2, we highlight two key findings when benchmarking SEKA against baselines and exploring the impact of different δ min for selecting KV heads. 1st5th10th15th20th25th30th 0.1 0.2 0.3 0.4 0.5 1st5th10th15th20th25th30th 0.2 0.3 0.4 0.5 0.6 1st5th10th15th20th25th30th 0.35 0.4 0.45 0.5 0.55 0.6 0.65 0.7 Original ModelPASTA (middle)PASTA (all)SEKA (middle)SEKA (all) Gold Answer Position in ContextGold Answer Position in ContextGold Answer Position in Context Exact Match Score Exact Match Score Exact Match Score Qwen3-4B-BaseQwen3-8B-BaseQwen3-14B-Base Figure 4: Exact match scores on the lost-in-the-middle task for Qwen3 models of three different sizes, comparing the original model, PASTA/SEKA applied to the middle region (5 th to 25 th pas- sages), and PASTA/SEKA applied to all passages. 1st5th10th15th20th25th30th 0.4 0.45 0.5 Original ModelSEKA (δmin=0.16) SEKA (δmin=0.165)SEKA (δmin=0.17) SEKA (δmin=0.18) Gold Answer Position in Context Exact Match Score Qwen3-8B-Base Figure 5: Exact match scores when ap- plying SEKA to the middle region with different threshold δ min . SEKA Can Invert the U-shape Performance. The re- sults, summarised in Figure 4, reveal two primary find- ings. First, applying SEKA selectively to the middle pas- sages (positions 5 to 25, which is a very rough range) is highly effective at inverting the canonical U-shaped performance profile: exact match scores at central posi- tions substantially increase, eliminating the typical per- formance trough for answers located in the middle of long contexts. Second, applying SEKA uniformly across all passages can slightly exacerbate the lost-in-the-middle is- sue. The most noticeable improvements typically occur at the beginning or end positions, while enhancements in the middle are less pronounced or may even decrease. In contrast, PASTA is less effective for this task. Applying it to either the middle region or the entire context results in performance generally below the original baseline across all model sizes. SEKA Can Mitigate and Flatten the U-Shape When Applied to Appropriate Number of KV Heads. In this control experiment, we fix the positive and negative steering gain coefficients (g + and g − ) at 0.2 and 0.1 respectively, and vary only the threshold δ min to control the number of steered KV heads. In practice, decreasing δ min increases the number of steered heads: for example, thresh- olds of 0.16, 0.165, 0.17, and 0.18 correspond to SEKA being applied on 58, 48, 41, and 31 KV heads for Qwen3-8B-Base, respectively. As shown in Figure 5, with an appropriate threshold δ min (around 0.165 and 0.17) and steering the middle region, SEKA can flatten the U-shaped performance curve without significantly compromising accuracy at the beginning and end positions. Note that the optimal threshold may vary with model size. Complete results for the 4B and 14B models are provided in Appendix K. 9 Published as a conference paper at ICLR 2026 6OVERHEAD ANALYSIS A key advantage of our pre-computation approach is its compatibility with optimised mech- anisms like FlashAttention (Dao et al., 2022; Dao, 2024; Shah et al., 2024).We quan- tify this by measuring inference overhead on 100 samples (avg.4362 tokens) from Section 5.2 using a Qwen3-8B-Base model on a single NVIDIA-GH200-120GB GPU. Table 3: Inference overhead on Qwen3-8B-Base. Time is per-sample; memory is average peak usage. Method Avg. Time (s) Peak Mem. (GB, B=10) Peak Mem. (GB, B=1) Original0.5527.6316.72 PASTA1.58 (+1.03) 50.75 (+23.12) - SPA5.87 (+5.32) -17.71 (+0.99) SEKA0.58 (+0.03) 27.66 (+0.03)16.75 (+0.03) AdaSEKA 0.82 (+0.27) 43.22 (+15.59) 18.23 (+1.51) As shown in Table 3, the overhead for SEKA is negligible (+0.03s per sample). This efficiency is particularly notable as, for a fair comparison with PASTA, we use an aggressive configuration that steers 175 out of 288 available KV heads. In con- trast, post-hoc methods incur significant costs.PASTA’s reliance on editing the full attention matrix makes it incompati- ble with FlashAttention, leading to a sub- stantial increase in latency (+1.03s) and memory usage (+23.12 GB). SPA, while memory-efficient for single samples, does not support batch processing and is thus the slowest over- all. Our adaptive variant, AdaSEKA, introduces a moderate overhead for its dynamic, query-aware capabilities (+0.27s). However, it remains significantly more efficient than both PASTA and SPA, making it a far more practical option for steering in long-context scenarios. 7RELATED WORK Research on steering large language models falls into two main paradigms. Activation Steering (Dathathri et al., 2020; Subramani et al., 2022; Hernandez et al., 2024) guides high-level semantic outputs by intervening in MLP layers, while Attention Steering, the focus of our work, directs the model’s focus to specific tokens within the input prompt. Activation Steering. This line of work, also known as representation engineering, adds “steering vectors” to MLP layer activations to control semantic attributes (Zou et al., 2023). Applications include enhancing honesty and safety (Ravfogel et al., 2020; Burns et al., 2023; Iskander et al., 2023; Li et al., 2023a; Wei et al., 2023; Bhattacharjee et al., 2024; Qiu et al., 2024), controlling style (Turner et al., 2023; 2024), improving reasoning (Tang et al., 2025), and knowledge editing (Fang et al., 2025). Recent studies suggest these methods only work when the model already knows the target knowledge (Simhi et al., 2025). These methods are therefore different from our approach as they change what the model knows through its hidden states, but we control where the model looks via its attention mechanism. Attention Steering. To address the challenge of LLMs failing to attend to key information in long contexts (Liu et al., 2024; Meng et al., 2022), prompt highlighting methods intervene post-hoc on either the attention scores (Zhang et al., 2024) or final logits (Tian & Zhang, 2025). However, these interventions often introduce significant latency; for instance, editing the full attention matrix is incompatible with modern optimisations like FlashAttention (Dao et al., 2022; Dao, 2024; Shah et al., 2024). This efficiency bottleneck motivates the need for pre-computation alternatives that can steer attention without sacrificing compatibility with optimised architectures. 8CONCLUSION In this paper, we introduced SEKA and its adaptive variant, AdaSEKA, a new class of training-free attention steering methods that operate by modifying key embeddings before the attention computa- tion. This pre-attention approach overcomes the core efficiency limitations of prior work, ensuring full compatibility with optimised implementations. Our experiments confirm that both methods achieve state-of-the-art results on a range of standard benchmarks, with AdaSEKA’s query-adaptive routing demonstrating particularly strong performance. These gains are achieved with negligible overhead, making our work a practical step towards building more controllable and efficient LLMs for long-context applications. 10 Published as a conference paper at ICLR 2026 REPRODUCIBILITY STATEMENT To ensure the reproducibility of our research, all necessary materials have been made publicly available on https://github.com/waylonli/SEKA. This repository includes: (1) the full source code for our proposed methods, SEKA and AdaSEKA; (2) detailed instructions for running all the experiments; (3) the pre-computed projection matrices used in our evaluations; and (4) the pre-processed versions of the datasets. The original datasets used in our evaluation are publicly available and are cited in Section 4. Specif- ically, the BIASBIOS, COUNTERFACT, and “Lost in the Middle” datasets are all distributed under the MIT License. Details regarding the evaluation samples and metrics calculation are provided in Appendix E, while hyperparameters are specified in Appendix F. ACKNOWLEDGEMENTS We thank the reviewers and the area chair for their valuable feedback. We also thank Yifu Qiu for constructive discussions related to this project. The authors acknowledge the use of resources provided by the Isambard-AI National AI Research Resource (AIRR). Isambard-AI is operated by the University of Bristol and is funded by the UK Government’s Department for Science, Innovation and Technology (DSIT) via UK Research and Innovation; and the Science and Technology Facilities Council [ST/AIRR/I-A-I/1023] (McIntosh-Smith et al., 2024). REFERENCES Federico Barbero, Andrea Banino, Steven Kapturowski, Dharshan Kumaran, Jo ̃ ao Guil- herme Madeira Ara ́ ujo, Oleksandr Vitvitskyi, Razvan Pascanu, and Petar Velickovic. Trans- formers need glasses!Information over-squashing in language tasks.In Advances in Neural Information Processing Systems 38:Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, 2024.URL http://papers.nips.c/paper_files/paper/2024/hash/ b1d35561c4a4a0e0b6012b2af531e149-Abstract-Conference.html. Amrita Bhattacharjee, Shaona Ghosh, Traian Rebedea, and Christopher Parisien. Towards inference- time category-wise safety steering for large language models. In NeurIPS Safe Generative AI Workshop 2024, 2024. URL https://openreview.net/forum?id=EkQRNLPFcn. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NeurIPS 2020, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546. Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in lan- guage models without supervision. In The Eleventh International Conference on Learning Rep- resentations, 2023. URL https://openreview.net/forum?id=ETKGuby0hcs. Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. What does BERT look at? An analysis of BERT’s attention. In Proceedings of the 2019 ACL Workshop Black- boxNLP: Analyzing and Interpreting Neural Networks for NLP, p. 276–286, Florence, Italy, 2019. Association for Computational Linguistics. doi: 10.18653/v1/W19-4828. URL https: //aclanthology.org/W19-4828. Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. In The Twelfth International Conference on Learning Representations, 2024. URL https:// openreview.net/forum?id=mZn2Xyh9Ec. 11 Published as a conference paper at ICLR 2026 Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher R ́ e.FlashAtten- tion: Fast and memory-efficient exact attention with io-awareness.In Advances in Neu- ral Information Processing Systems 35: Annual Conference on Neural Information Process- ing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022, 2022.URL http://papers.nips.c/paper_files/paper/2022/hash/ 67d57c32e20fd0a7a302cb81d36e40d5-Abstract-Conference.html. Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu.Plug and play language models: A simple approach to con- trolled text generation. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=H1edEyBKDS. Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. Bias in bios: A case study of semantic representation bias in a high-stakes setting. In Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT* ’19, p. 120–128, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450361255. doi: 10.1145/ 3287560.3287572. URL https://doi.org/10.1145/3287560.3287572. Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. https://transformer-circuits.pub/2021/framework/index.html. Junfeng Fang, Houcheng Jiang, Kun Wang, Yunshan Ma, Jie Shi, Xiang Wang, Xiangnan He, and Tat-Seng Chua. Alphaedit: Null-space constrained model editing for language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https: //openreview.net/forum?id=HvSytvg3Jh. Gemma Team. Gemma 3 technical report, 2025. URL https://arxiv.org/abs/2503. 19786. Evan Hernandez, Belinda Z. Li, and Jacob Andreas.Inspecting and editing knowledge rep- resentations in language models. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=ADtL6fgNRv. Cheng-Yu Hsieh, Yung-Sung Chuang, Chun-Liang Li, Zifeng Wang, Long Le, Abhishek Kumar, James Glass, Alexander Ratner, Chen-Yu Lee, Ranjay Krishna, and Tomas Pfister. Found in the middle: Calibrating positional attention bias improves long context utilization. In Findings of the Association for Computational Linguistics: ACL 2024, p. 14982–14995, Bangkok, Thailand, 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-acl.890. URL https://aclanthology.org/2024.findings-acl.890/. Shadi Iskander, Kira Radinsky, and Yonatan Belinkov. Shielded representations: Protecting sen- sitive attributes through iterative gradient-based projection.In Findings of the Association for Computational Linguistics: ACL 2023, p. 5961–5977, Toronto, Canada, 2023. Associa- tion for Computational Linguistics. doi: 10.18653/v1/2023.findings-acl.369. URL https: //aclanthology.org/2023.findings-acl.369. Kenneth Li, Oam Patel, Fernanda B. Vi ́ egas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from a language model.In Ad- vances in Neural Information Processing Systems 36: Annual Conference on Neural Infor- mation Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023a. URL http://papers.nips.c/paper_files/paper/2023/hash/ 81b8390039b7302c909cb769f8b6cd93-Abstract-Conference.html. Weixian Waylon Li, Yftah Ziser, Maximin Coavoux, and Shay B. Cohen. BERT is not the count: Learning to match mathematical statements with proofs. In Andreas Vlachos and Isabelle Au- genstein (eds.), Proceedings of the 17th Conference of the European Chapter of the Associ- ation for Computational Linguistics, p. 3581–3593, Dubrovnik, Croatia, May 2023b. Asso- 12 Published as a conference paper at ICLR 2026 ciation for Computational Linguistics. doi: 10.18653/v1/2023.eacl-main.260. URL https: //aclanthology.org/2023.eacl-main.260/. Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173, 2024. doi: 10.1162/tacl a00638. URL https://aclanthology.org/2024.tacl-1.9. Simon McIntosh-Smith, Sadaf R Alam, and Christopher Woods. Isambard-ai: a leadership class supercomputer optimised specifically for artificial intelligence, 2024. URL https://arxiv. org/abs/2410.11199. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associ- ations in GPT. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022, 2022. URL http://papers.nips.c/paper_files/paper/ 2022/hash/6f1d43d5a82a37e89b0665b33bf3a182-Abstract-Conference. html. Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? In Ad- vances in Neural Information Processing Systems 32: Annual Conference on Neural Informa- tion Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, p. 14014–14024, 2019. URL https://proceedings.neurips.c/paper/2019/ hash/2c601ad9d2f9bc8b282670cdd54f69f-Abstract.html. Clement Neo, Shay B Cohen, and Fazl Barez. Interpreting context look-ups in transformers: In- vestigating attention-MLP interactions. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Process- ing, p. 16681–16697, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.930. URL https://aclanthology.org/ 2024.emnlp-main.930/. Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads, 2022. URL https://arxiv.org/ abs/2209.11895. Yifu Qiu, Zheng Zhao, Yftah Ziser, Anna Korhonen, Edoardo Maria Ponti, and Shay B. Co- hen.Spectral editing of activations for large language model alignment.In Advances in Neural Information Processing Systems 38:Annual Conference on Neural Informa- tion Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, 2024.URL http://papers.nips.c/paper_files/paper/2024/hash/ 684c59d614fe6ae74a3be8c3ef07e061-Abstract-Conference.html. Yifu Qiu, Varun R. Embar, Yizhe Zhang, Navdeep Jaitly, Shay B Cohen, and Benjamin Han. Elicit- ing in-context retrieval and reasoning for long-context large language models. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Findings of the Asso- ciation for Computational Linguistics: ACL 2025, p. 3176–3192, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-256-5. doi: 10.18653/v1/2025. findings-acl.165. URL https://aclanthology.org/2025.findings-acl.165/. Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. Null it out: Guarding protected attributes by iterative nullspace projection. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics, p. 7237–7256, Online, 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.647. URL https: //aclanthology.org/2020.acl-main.647. Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. FlashAttention-3: Fast and accurate attention with asynchrony and low-precision.In Ad- vances in Neural Information Processing Systems 38: Annual Conference on Neural Infor- mation Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 13 Published as a conference paper at ICLR 2026 15, 2024, 2024. URL http://papers.nips.c/paper_files/paper/2024/hash/ 7ede97c3e082c6df10a8d6103a2eebd2-Abstract-Conference.html. Adi Simhi, Jonathan Herzig, Itay Itzhak, Dana Arad, Zorik Gekhman, Roi Reichart, Fazl Barez, Gabriel Stanovsky, Idan Szpektor, and Yonatan Belinkov. HACK: Hallucinations along certainty and knowledge axes, 2025. URL https://arxiv.org/abs/2510.24222. Alessandro Stolfo, Vidhisha Balachandran, Safoora Yousefi, Eric Horvitz, and Besmira Nushi. Im- proving instruction-following in language models through activation steering. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview. net/forum?id=wozhdnRCtw. Nishant Subramani, Nivedita Suresh, and Matthew Peters.Extracting latent steering vectors from pretrained language models. In Findings of the Association for Computational Linguis- tics: ACL 2022, p. 566–581, Dublin, Ireland, 2022. Association for Computational Linguis- tics. doi: 10.18653/v1/2022.findings-acl.48. URL https://aclanthology.org/2022. findings-acl.48. Xinyu Tang, Xiaolei Wang, Zhihao Lv, Yingqian Min, Xin Zhao, Binbin Hu, Ziqi Liu, and Zhiqiang Zhang. Unlocking general long chain-of-thought reasoning capabilities of large language models via representation engineering. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 6832–6849, Vienna, Austria, 2025. Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025. acl-long.339. URL https://aclanthology.org/2025.acl-long.339/. Yuan Tian and Tianyi Zhang. Selective prompt anchoring for code generation. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/ forum?id=aEnkBIhYvO. Alexander Matt Turner, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and Monte Mac- Diarmid. Activation addition: Steering language models without optimization. ArXiv preprint, abs/2308.10248, 2023. URL https://arxiv.org/abs/2308.10248. Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J. Vazquez, Ulisse Mini, and Monte MacDiarmid. Steering language models with activation engineering, 2024. URL https://arxiv.org/abs/2308.10248. Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, p. 5797–5808, Florence, Italy, 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1580. URL https://aclanthology.org/P19-1580. Tianlong Wang, Xianfeng Jiao, Yinghao Zhu, Zhongzhi Chen, Yifan He, Xu Chu, Junyi Gao, Yasha Wang, and Liantao Ma. Adaptive activation steering: A tuning-free llm truthfulness improvement method for diverse hallucinations categories.In Proceedings of the ACM on Web Conference 2025, W ’25, p. 2562–2578, New York, NY, USA, 2025. Association for Computing Machinery. ISBN 9798400712746. doi: 10.1145/3696410.3714640. URL https://doi.org/10.1145/3696410.3714640. Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does LLM safety training fail? In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023. URL http://papers.nips.c/paper_files/paper/2023/hash/ fd6613131889a4b656206c50a8bd7790-Abstract-Conference.html. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou.Chain-of-thought prompting elicits reasoning in large lan- guage models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NeurIPS 2022, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 9781713871088. 14 Published as a conference paper at ICLR 2026 Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, and Yao Fu. Retrieval head mechanis- tically explains long-context factuality. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=EytBpUGB1Z. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL https://arxiv.org/abs/2505.09388. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, p. 2369–2380, Brussels, Belgium, 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1259. URL https://aclanthology.org/D18-1259. Qingru Zhang, Chandan Singh, Liyuan Liu, Xiaodong Liu, Bin Yu, Jianfeng Gao, and Tuo Zhao. Tell your model where to attend: Post-hoc attention steering for LLMs. In The Twelfth Interna- tional Conference on Learning Representations, 2024. URL https://openreview.net/ forum?id=xZDWO0oejD. Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico Kolter, and Dan Hendrycks. Representation engineering: A top-down ap- proach to ai transparency, 2023. URL https://arxiv.org/abs/2310.01405. ASYNTHETIC DATASET FOR TOKEN-LEVEL RELEVANCE SUPERVISION To supervise attention steering, we construct a synthetic dataset that enables precise control over token-level relevance. Each sample comprises two contexts (C 1 , C 2 ), each paired with a question and answer tuple (Q 1 , A 1 and Q 2 , A 2 ). This structure allows us to define relevance by contrasting identical token spans across different query contexts. Table 4: Constructed prompt triplets for both answer spans. Each group provides a neutral, positive, and negative variant based on question-context alignment. GroupPrompt NeutralContext: The portfolio manager allocatescapital across equities and bonds. PositiveQuestion: What does the portfolio manager allocate across equities and bonds? Context: The portfolio manager allocatescapital across equities and bonds. NegativeQuestion: What does the climate model simulate? Context: The portfolio manager allocates capital across equities and bonds. NeutralContext: The climate model simulatessea-level rise under different scenarios. PositiveQuestion: What does the climate model simulate? Context: The climate model simulates sea-level rise under different scenarios. NegativeQuestion: What does the portfolio manager allocate across equities and bonds? Context: The climate model simulates sea-level rise under different scenarios. With every pair of (C,Q,A) triplets, as shown in Table 4, we can derive two supervision samples: one for the answer span “capital” in C 1 , and another for the answer span “sea-level rise” in C 2 . 15 Published as a conference paper at ICLR 2026 Table 5: Synthetic data instance. Context 1 (C 1 )The portfolio manager allocates capital across equities and bonds. Context 2 (C 2 )The climate model simulates sea-level rise under different scenarios. Question 1 (Q 1 )What does the portfolio manager allocate across equities and bonds? Answer 1 (A 1 )capital Question 2 (Q 2 )What does the climate model simulate? Answer 2 (A 2 )sea-level rise For each answer, we construct three variants: (1) a positive (relevant) prompt where the question and context are aligned (e.g., Q 1 for C 1 , and Q 2 for C 2 ), (2) a negative (irrelevant) prompt where the question mismatches the context (e.g., Q 1 for C 2 , and Q 2 for C 1 ), and (3) a neutral prompt containing only the context. This allows us to collect three classes of key embeddings for the answer spans within the context: h + for positive, h − for negative, and h for neutral. In Figure 1, we empirically show that, for some key-value heads, different token spans exhibit a consistent shift in their key embeddings from negative to positive variants. This validates the construction and use of these relevance supervision signals. Practical construction details. The synthetic dataset is lightweight to produce. We use a fixed template as shown in Table 5 and automatically prompt an GPT-4o to produce contrastive samples, using the prompt provided in Figure 6. This process requires no manual annotation. After collecting the generated samples, we convert them into JSON format for subsequent use. BMULTI-EXPERT PROJECTION LEARNING SAMPLES FOR AdaSEKA After constructing the synthetic dataset, we prepared three additional task-specific datasets, making a total of four, for multi-expert projection learning (Section 3.3). As shown in Table 6, each sample consists of a neutral and a positive prompt pair. For the Counterfact (Meng et al., 2022) dataset and the BiasBios (De-Arteaga et al., 2019) datasets, these pairs are collected from their respective training sets, following the original prompt templates outlined in Table 1. For each sample, we extract the key embeddings for the answer spans directly from the context. A distinct procedure is adopted for HotpotQA (Yang et al., 2018) to account for its multi-hop nature. The context is formed by concatenating all candidate paragraphs, and the key embeddings from all supporting facts are subsequently extracted and concatenated. Each expert projection is learned from a set of 200 randomly sampled instances from the training set for each task, using a fixed random seed of 42 to ensure reproducibility. CGEOMETRIC INTUITION OF THE SEKA TRANSFORMATION To provide geometric insight into the effect of SEKA’s key editing, consider the case where the projection matrix P is given by U ⊤ , with U ∈ R d k ×r having orthonormal columns that span the relevance subspace (i.e., U = U + or U − as previously defined). For simplicity, assume g = 1 and focus solely on the positive (or negative) projection. The transformation then becomes: k ′ j = (I + U ⊤ )k j .(7) Any vector x∈ R d k can be decomposed as x = x ∥ + x ⊥ ,where x ∥ = U ⊤ x, x ⊥ = x− U ⊤ x.(8) This decomposition is orthogonal. Specifically, x ⊤ ∥ x ⊥ = (U ⊤ x) ⊤ (x− U ⊤ x) = x ⊤ U ⊤ x− x ⊤ U ⊤ U ⊤ x(9) = x ⊤ U ⊤ x− x ⊤ U ⊤ x = 0,(10) using the idempotency of the projection ((U ⊤ ) 2 = U ⊤ ). 16 Published as a conference paper at ICLR 2026 Synthetic Samples Generation Prompt We are collaboratively generating a total of 100 synthetic examples. You will generate examples in batches of exactly 20 per response. Across ALL batches in this conversation, every example must be globally unique. Before generating a new batch, you MUST: 1. Review ALL previous examples in the conversation. 2. Ensure no repeated entities, contexts, events, sentence structures, questions, or answers. 3. Ensure no near-duplicates, paraphrased duplicates, or re-themed duplicates. Each example must follow this structure (exact formatting): Example k: Context 1: <C1> Context 2: <C2> Question 1: <Q1> Answer 1: <A1> Question 2: <Q2> Answer 2: <A2> Generation requirements: - Each context must be one concise fictional sentence. - C1 and C2 must be semantically unrelated. - Q1 must ask about a span that appears verbatim in C1. - Q2 must ask about a span that appears verbatim in C2. - A1 and A2 must be exact substrings (contiguous spans) of C1 and C2. - No context, entity, theme, setting, or question type may repeat across any batch. - Avoid any resemblance to earlier examples in wording, structure, or domain. - No extra explanation or commentary. Your task now: Read all previous examples in the conversation so far. Then generate the next 20 completely new, globally unique examples. Stop after exactly 20 examples. Figure 6: Prompt template used to generate synthetic contrastive examples. Applying the transformation, we have (I + U ⊤ )x = x + U ⊤ x = x− U ⊤ x + U ⊤ x + U ⊤ x(11) = x− U ⊤ x | z x ⊥ +2 U ⊤ x | z x ∥ = x ⊥ + 2x ∥ .(12) which shows that the component along the subspace is amplified (doubled), while the orthogonal component remains unchanged. While the g = 1 case offers geometric clarity, the result generalises for any g ∈ R: (I + gUU ⊤ )x = x ⊥ + (1 + g)x ∥ .(13) Thus, the relevance-aligned component is scaled by (1 + g), while all orthogonal directions are preserved. This operation is neither a projection nor an orthogonal transformation, but a targeted linear modification that selectively amplifies directions aligned with the relevance subspace. SEKA leverages this property to boost relevant token features in a controlled and interpretable manner, enabling precise, token-wise attention steering without interfering with unrelated components. While the geometric interpretation above clarifies how SEKA amplifies components of key vectors aligned with a learned relevance subspace, it is important to clarify what this subspace represents. SEKA is not intended to encode or manipulate semantic meaning. Its effect is deliberately confined 17 Published as a conference paper at ICLR 2026 Table 6: Constructed prompt pairs for multi-expert projection learning. Each dataset provides neu- tral and positive variants based on question-context alignment. DatasetVariantPrompt CounterFactNeutralContext: Previously, The mother tongue of Danielle Darrieux is French. Currently, The mother tongue of Danielle Darrieux isEnglish. PositiveQuestion: Danielle Darrieux, a native ? Context: Previously, The mother tongue of Danielle Darrieux is French. Currently, The mother tongue of Danielle Darrieux is English. BiasBiosNeutralContext: Nora is an assistantprofessor of international relations at Bahcesehir Uni- versity in Istanbul. She is also a Ronald D. Asmus Policy Entrepreneur Fellow with the German Marshall Fund and is a Visiting Fellow at the Centre for International Studies (CIS) at the University of Oxford. This commentary first appeared at Sada, an online journal published by the Carnegie Endowment for International Peace. PositiveQuestion: Nora has the occupation of a/an ? Context: Nora is an assistant professor of international relations at Bahcesehir Uni- versity in Istanbul. She is also a Ronald D. Asmus Policy Entrepreneur Fellow with the German Marshall Fund and is a Visiting Fellow at the Centre for International Studies (CIS) at the University of Oxford. This commentary first appeared at Sada, an online journal published by the Carnegie Endowment for International Peace. HotpotQANeutralContext: Radio City (Indian radio station) Radio City is India’s first private FM radio station and was started on 3 July 2001. ... Arthur’s Magazine Arthur’sMag- azine(1844–1846)wasanAmericanliteraryperiodicalpublishedinPhiladelphia inthe19thcentury. Edited by T.S. Arthur, it featured work by Edgar A. Poe, J.H. Ingraham, Sarah Josepha Hale, Thomas G. Spear, and others. In May 1846 it was merged into “Godey’s Lady’s Book”. ... First for Women FirstforWomenisa woman ́ smagazinepublishedbyBauerMediaGroupintheUSA. The magazine was started in 1989. It is based in Englewood Cliffs, New Jersey. ... The company started first as a denim line, later evolving into a men’s and women’s clothing line. PositiveQuestion: Which magazine was started first Arthur’s Magazine or First for Women? Context: Radio City (Indian radio station) Radio City is India’s first private FM radio station and was started on 3 July 2001. ... Arthur’s MagazineArthur’sMag- azine(1844–1846)wasanAmericanliteraryperiodicalpublishedinPhiladelphia inthe19thcentury. Edited by T.S. Arthur, it featured work by Edgar A. Poe, J.H. Ingraham, Sarah Josepha Hale, Thomas G. Spear, and others. In May 1846 it was merged into “Godey’s Lady’s Book”. ... First for WomenFirstforWomenisa woman’smagazinepublishedbyBauerMediaGroupintheUSA. The magazine was started in 1989. It is based in Englewood Cliffs, New Jersey. ... The company started first as a denim line, later evolving into a men’s and women’s clothing line. to the attention to route subspace of the transformer, consistent with prior mechanistic analyses (Elhage et al., 2021; Olsson et al., 2022). Modern transformer-circuits work decomposes the action of an attention head as H (h) (R) = A (h) (R) ⊗ W (h) O W (h) V R,(14) where A (h) is the query-key similarity tensor governing which tokens attend to which, and W (h) O W (h) V writes attended features into the residual stream (Elhage et al., 2021). This formula- tion explicitly separates routing (Q/K) from semantic write operations (V/MLP). Further, studies of induction and retrieval heads (Olsson et al., 2022) show that Q/K vectors imple- ment token-matching and algorithmic routing behaviour, such as copying and continuation, while semantic information is primarily stored in value vectors and MLP activations. These findings align with our design that SEKA aims to modify only the routing (relevance) subspace, leaving the seman- tic subspace untouched. 18 Published as a conference paper at ICLR 2026 DSEKA AND AdaSEKA ALGORITHMS We provide detailed pseudocode for our proposed methods, SEKA and AdaSEKA. Algorithm 1 de- tails the standard SEKA method. It involves an offline phase to learn fixed positive and negative projection matrices from contrastive data using SVD. During inference, a hook then applies these static projections to the key embeddings of highlighted tokens. Algorithm 2 describes the more flex- ible AdaSEKA framework. In essence, standard SEKA can be viewed as a special case of AdaSEKA with a single expert and no dynamic coefficient calculation. AdaSEKA generalises this by loading a bank of multiple expert SVD components offline. For each new prompt, it then performs a dynamic, query-aware pre-computation: it calculates routing coefficients based on the query’s alignment with each expert and constructs a bespoke projection matrix on-the-fly. This tailored projection is subse- quently applied during generation via the key-editing hook. Algorithm 1 Spectral Editing Key Amplification (SEKA) Require: Tripletsh, h + , h − ℓ,h , variance threshold γ, gains g + ,g − Ensure: ProjectionsP + ℓ,h , P − ℓ,h and a key-editing hook 1: for all layer ℓ and head h do 2:Ω + ℓ,h ← 1 n h ⊤ h + , Ω − ℓ,h ← 1 n h ⊤ h − 3:(U + ℓ,h , S + ℓ,h , V + ℓ,h )← SVD(Ω + ℓ,h ) , (U − ℓ,h , S − ℓ,h , V − ℓ,h )← SVD(Ω − ℓ,h ) 4: k + ← mink : P k i=1 S + ℓ,h,i / P i S + ℓ,h,i ≥ γ , k − ← mink : P k i=1 S − ℓ,h,i / P i S − ℓ,h,i ≥ γ 5: P + ℓ,h ←U + ℓ,h,:k + U +⊤ ℓ,h,:k + , P − ℓ,h ←U − ℓ,h,k − : U −⊤ ℓ,h,k − : 6: end for 7: Hook applied to each selected (ℓ,h) (registered per layer ℓ; ℓ is fixed within the hook). 8:Input: K∈R B×T×H×d , mask m 9:Note: For brevity we omit the explicit layer index on K; projections remain P ± ℓ,h . 10:for b=1..B, t=1..T, h=1..H do 11:if m b,t =1 then 12:∆← g + P + ℓ,h + g − P − ℓ,h K[b,t,h, :]/2 13:K[b,t,h, :]←K[b,t,h, :] + ∆ 14:return K to the attention computation 15: Register the hook for selected (ℓ,h) before generation and remove it afterwards. EDETAILS OF STANDARD BENCHMARKS We evaluate our method on three established benchmarks adapted from the PASTA frame- work (Zhang et al., 2024). We introduce significant improvements to the evaluation protocols, such as case-insensitive scoring, to ensure a more robust assessment. The JSON Formatting task was omitted as modern models achieve near-perfect performance, rendering it less useful for discrimi- nating capabilities. E.1COUNTERFACT The COUNTERFACT benchmark (Meng et al., 2022) evaluates an LLM’s ability to prioritise new contextual information over its pre-trained knowledge. Here, each fact is represented as a sub- ject–relation–object triple (s,r,o), where s denotes the subject entity, r the relation, and o the ob- ject. Task Format. The model receives input structured as: “Previously, s r o old . Currently, s r o new . question.” The challenge arises because models often default to pre-trained asso- ciations rather than attending to the new, contradictory information provided in the context. 19 Published as a conference paper at ICLR 2026 Algorithm 2 Query-Driven Adaptive SEKA (AdaSEKA) Require: SVD componentsU + m,ℓ,h , S + m,ℓ,h for M experts, top components K, gain g Ensure: A key-editing hook using dynamically computed projections 1: Store expert SVD componentsU + m,ℓ,h , S + m,ℓ,h for all experts m, layers ℓ, and heads h. 2: For a given prompt with input IDs I : 3:Obtain last-token query vectors q ℓ,h for each selected layer ℓ and head h. 4: for all selected layer ℓ and head h do 5:for all expert m = 1..M do 6:Calculate coefficient α m,ℓ,h (q ℓ,h )∝ P K k=1 (q ⊤ ℓ,h u +(k) m,ℓ,h )· σ +(k) m,ℓ,h (as per Eq. 6) 7:end for 8:Construct P dynamic,ℓ,h ← P M m=1 α m,ℓ,h (q ℓ,h ) U + m,ℓ,h,:,:K (U + m,ℓ,h,:,:K ) ⊤ 9:Store P dynamic,ℓ,h for use in the hook. 10: end for 11: Hook applied to each selected (ℓ,h) (registered per layer ℓ; ℓ is fixed within the hook). 12:Input: K∈R B×T×H×d , mask m 13:Note: For brevity we omit the explicit layer index on K. 14:for b=1..B, t=1..T, h=1..H do 15:if m b,t =1 then 16:∆←g· P dynamic,ℓ,h K[b,t,h, :] 17:K[b,t,h, :]←K[b,t,h, :] + ∆ 18:return K to the attention computation 19: Register the hook for selected (ℓ,h) before generation and remove it afterwards. Prompt: “Previously, Kevin Garnett is a professional basketball player. Currently, **Kevin Garnett is a professional baseball player**. Kevin Garnett is a professional” Target: The model should generate “baseball player” rather than its pre-trained association of “basketball player”. Evaluation Metrics. Following (Zhang et al., 2024), to evaluate the model’s ability to recall the new fact, we measure its internal preferences at the point of generation, rather than relying on parsing free-form text. For a given prompt, we provide the model with the entire context and question, and then assess the log probabilities it assigns to the potential next tokens. • Efficacy Score (ES): This metric directly measures if the model prioritises the new, correct fact (o new ) over the old, incorrect fact (o old ). It is the percentage of times the model assigns a higher probability to the first token of the new fact than to the first token of the old fact. A high ES indicates that the model has successfully updated its belief based on the context. ES = 1 N N X i=1 I[P LLM (o (i) new ) > P LLM (o (i) old )] • Paraphrase Score (PS): This metric measures generalisation by calculating the average Efficacy Score across a collection of human-written paraphrases of the original question. E.2BIASBIOS The BIASBIOS dataset (De-Arteaga et al., 2019) consists of biographies and was originally designed to explore gender bias in occupation prediction. The first sentence of each biography explicitly states the person’s occupation, while subsequent sentences provide potentially distracting career details. Task Format. Each biography is appended with the prompt “person has the occupation of ”, and the model must predict the correct occupation from a list of 28 candidates. 20 Published as a conference paper at ICLR 2026 Prompt: “**Winnie is an American photographer living in New York.** Specialized in fash- ion photography and portrait, she applies her talent on both humans and animals... Winnie has the occupation of ” Target: “photographer” Evaluation Metrics. We measure standard top-1 Accuracy across the 28 candidate occupations, implementing case-insensitive matching to ensure semantic equivalence is correctly evaluated. E.3PRONOUNS CHANGING This task evaluates instruction-following through linguistic transformation. Models are instructed to “substitute ‘she’ and ‘he’ with ‘they’.” This requires simultaneously adhering to the transformation rule while preserving the original content. Prompt: “Mary is an Associate Professor in the Department of Curriculum Instruction at St. John University, she holds a doctorate in Reading/Writing/Literacy from the University of Pennsylvania... **substitute ‘she’ and ‘he’ with ‘they’ and generate the occupation of Mary after changing pronouns**.” Target: “Mary is an associate professor... they hold a doctorate... Mary has the occupation of Associate Professor.” Enhanced Evaluation Metric. As noted during the public peer review of Zhang et al. (2024) 2 , the original metric rewards empty strings for perfectly “converting” zero pronouns, resulting in misleadingly high scores. To address this, we introduce the Pronoun-weighted Lexical Overlap Score (P. Score), which unifies instruction-following and content preservation into a single metric. The P. Score modulates the credit for lexical overlap with the original text by the success rate of pronoun conversion. It is defined as: P. Score = w pron ×|T ori ∩ T gen | |T ori | ,(15) where w pron is the fraction of successfully converted pronouns, and T ori and T gen are the sets of non- pronoun content tokens from the original and generated texts, respectively. This ensures that empty generations receive a score of zero and that content preservation is only credited when instruction- following occurs. We evaluate two variants: one (P. Score) targeting core subject pronouns (“she”, “he”) and another (A. P. Score) targeting a complete set of gendered pronouns (“she”, “he”, “her”, “him”, “hers”, “his”, “herself”, “himself”). FTECHNICAL SETUP This appendix section details the hyperparameters used for the SEKA and AdaSEKA experiments. For the CounterFact and Bias in Bios benchmarks, we performed a grid search to tune the hyperpa- rameters on a validation set of 500 samples (indices 4500–4999), following the experimental setup of PASTA (Zhang et al., 2024). The final evaluation was then conducted on the test set (indices 5000–10000). For the Pronoun Changing task, hyperparameters were tuned on a separate small development set. All experiments across all models used greedy decoding. The standard SEKA method requires tuning four hyperparameters: the variance threshold for projec- tion construction (γ), the relevance-sensitivity threshold for KV-head selection (δ min ) and the posi- tive/negative steering gains (g + and g − ). The AdaSEKA framework simplifies this process, requiring only the tuning of the KV-head selection threshold (δ min ) and a single steering gain coefficient (g). The selected hyperparameters for each model and task are provided in Table 7. 2 https://openreview.net/forum?id=xZDWO0oejD¬eId=3kDI7QRqSI 21 Published as a conference paper at ICLR 2026 Table 7: Hyperparameters for SEKA and AdaSEKA methods. SEKA uses the variance threshold (γ), KV-head selection threshold (δ min ), positive gain (g + ), and negative gain (g − ). AdaSEKA uses the KV-head selection threshold (δ min ) and steering gain (g). SEKAAdaSEKA ModelTaskγ δ min g + g − δ min g Qwen3-4B-Base CounterFact0.9600.131.560.000.13.0 Bias in Bios0.9980.121.000.800.10.5 Pronoun Changing0.8800.220.160.000.50.6 Qwen3-8B-Base CounterFact0.8500.122.400.000.13.0 Bias in Bios0.9980.120.600.300.10.5 Pronoun Changing0.9000.200.190.000.50.6 Qwen3-14B-Base CounterFact0.8700.102.420.000.13.0 Bias in Bios0.9900.150.600.300.31.0 Pronoun Changing0.8800.230.160.000.60.6 Gemma-3-4B CounterFact0.9900.602.000.000.23.0 Bias in Bios0.8000.120.800.000.20.8 Pronoun Changing0.8000.200.400.000.41.0 Gemma-3-12B CounterFact0.9900.501.000.000.1-5.0 Bias in Bios0.9940.000.400.000.70.5 Pronoun Changing0.7000.40-0.500.000.5-0.4 Hyper-parameters Sensitivity. To explore SEKA’s sensitivity to its hyper-parameters, we con- duct an experimental analysis by varying each parameter independently while keeping all oth- ers fixed at their optimal configurations on the validation set (Table 7). We randomly select 500 test samples across the three benchmark tasks and adapt a one-at-a-time sweep over the following ranges: γ ∈ 0.75, 0.80, 0.85, 0.90, 0.95, δ min ∈ 0.10, 0.20, 0.30, 0.40, 0.50, 0.60, g + ∈0.1, 0.2, 0.4, 0.6, 0.8, 1.0, 1.5, 2.0, and g − ∈0.00, 0.20, 0.40, 0.60, 0.80. 0.750.80.850.90.95 0.5 0.6 0.7 0.8 0.9 1 0.20.40.600.511.5200.20.40.60.8 BiasBios (Acc.)CounterFact (ES)PronChange (P. Score) Qwen3-4B Main Metrics γδₘᵢₙg⁺g⁻ 0.750.80.850.90.95 0.4 0.5 0.6 0.7 0.8 0.9 1 0.20.40.600.511.5200.20.40.60.8 BiasBios (Acc.)CounterFact (ES)PronChange (P. Score) Qwen3-8B Main Metrics γδₘᵢₙg⁺g⁻ 0.750.80.850.90.95 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0.20.40.600.511.5200.20.40.60.8 BiasBios (Acc.)CounterFact (ES)PronChange (P. Score) Gemma-3-4B Main Metrics γδₘᵢₙg⁺g⁻ 0.750.80.850.90.95 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0.20.40.600.511.5200.20.40.60.8 BiasBios (Acc.)CounterFact (ES)PronChange (P. Score) Gemma-3-12B Main Metrics γδₘᵢₙg⁺g⁻ Figure 7: Sensitivity of SEKA to hyper-parameters across three benchmark tasks. Each curve varies a single hyper-parameter while keeping others fixed at their optimal settings on the validation set. Three findings are observed from the results in Figure 7: • δ min and g + are the most influential. These parameters determine which heads are steered and the strength of amplification. Performance drops when too few or too many heads 22 Published as a conference paper at ICLR 2026 (depending on the tasks) are selected or when the positive gain is either too small to steer effectively or too large, which leads to over-amplification and degradation. • Models from the same family show similar trends. Qwen3-4B and Qwen3-8B display nearly identical sensitivity patterns on CounterFact, both favouring low δ min and showing stability across γ. Gemma 3 models exhibit higher variance with respect to γ. • Task characteristics differ across models. Stability patterns are task-model dependent. For example, Gemma-3-4B shows pronounced variability on PronChange at higher g + val- ues, whereas CounterFact remains comparatively stable. In contrast, both Qwen3 models maintain strong robustness on BiasBios and PronChange but are noticeably more sensitive on CounterFact. These differences suggest that tasks requiring factual override (Coun- terFact) and tasks requiring instruction-following (PronChange) stress models in different ways, resulting in varying sensitivity. GMECHANISTIC INSIGHT VIA ATTENTION VISUALISATION To illustrate SEKA ’s effect on model behaviour, we visualise the mean attention across all heads in selected layers for a CounterFact data sample: “Previously Patrick Roy professionally plays the sport hockey. Currently Patrick Roy **professionally plays the sport basketball**. Patrick Roy is a professional ”. As shown in Figure 8, before SEKA is applied, the model’s attention to the ma- nipulated subspan (“was employed in Oslo”) is low, with little focus on the relevant passage. After SEKA steering, attention in the affected layers becomes more concentrated on the target subspan, clearly demonstrating SEKA ’s ability to selectively and effectively redirect model attention. This targeted effect aligns with the observed accuracy gains on benchmark tasks. Previously Patrick Roy professionally plays the sport hockey . Currently Patrick Roy professionally plays the sport basketball . Patrick Roy is a professional Previously Patrick Roy professionally plays the sport hockey . Currently Patrick Roy professionally plays the sport basketball . Patrick Roy is a professional Original - L17 | mean of all heads Previously Patrick Roy professionally plays the sport hockey . Currently Patrick Roy professionally plays the sport basketball . Patrick Roy is a professional Previously Patrick Roy professionally plays the sport hockey . Currently Patrick Roy professionally plays the sport basketball . Patrick Roy is a professional SEKA - L17 | mean of all heads 0.0 0.2 0.4 0.6 0.8 1.0 Previously Patrick Roy professionally plays the sport hockey . Currently Patrick Roy professionally plays the sport basketball . Patrick Roy is a professional Previously Patrick Roy professionally plays the sport hockey . Currently Patrick Roy professionally plays the sport basketball . Patrick Roy is a professional Original - L18 | mean of all heads Previously Patrick Roy professionally plays the sport hockey . Currently Patrick Roy professionally plays the sport basketball . Patrick Roy is a professional Previously Patrick Roy professionally plays the sport hockey . Currently Patrick Roy professionally plays the sport basketball . Patrick Roy is a professional SEKA - L18 | mean of all heads 0.0 0.2 0.4 0.6 0.8 1.0 Previously Patrick Roy professionally plays the sport hockey . Currently Patrick Roy professionally plays the sport basketball . Patrick Roy is a professional Previously Patrick Roy professionally plays the sport hockey . Currently Patrick Roy professionally plays the sport basketball . Patrick Roy is a professional Original - L21 | mean of all heads Previously Patrick Roy professionally plays the sport hockey . Currently Patrick Roy professionally plays the sport basketball . Patrick Roy is a professional Previously Patrick Roy professionally plays the sport hockey . Currently Patrick Roy professionally plays the sport basketball . Patrick Roy is a professional SEKA - L21 | mean of all heads 0.0 0.2 0.4 0.6 0.8 1.0 Previously Patrick Roy professionally plays the sport hockey . Currently Patrick Roy professionally plays the sport basketball . Patrick Roy is a professional Previously Patrick Roy professionally plays the sport hockey . Currently Patrick Roy professionally plays the sport basketball . Patrick Roy is a professional Original - L29 | mean of all heads Previously Patrick Roy professionally plays the sport hockey . Currently Patrick Roy professionally plays the sport basketball . Patrick Roy is a professional Previously Patrick Roy professionally plays the sport hockey . Currently Patrick Roy professionally plays the sport basketball . Patrick Roy is a professional SEKA - L29 | mean of all heads 0.0 0.2 0.4 0.6 0.8 1.0 Figure 8: Layer-wise mean attention (all heads) in Qwen3-4B-Base at selected layers for the Coun- terFact data sample, shown before and after SEKA is applied. HCOMPLETE RESULTS OF PASTA WITH DIFFERENT CONFIGURATIONS In the main results (Table 2), we reported the strongest performance for the PASTA baseline to ensure a fair comparison. For completeness, Table 8 provides a detailed breakdown of PASTA’s per- formance across three different head-selection configurations. The first configuration replicates the original head search method, which identifies the top-k performing heads by individually evaluating the steering effect of every attention head (Zhang et al., 2024). The other two configurations ex- plore a hybrid approach by combining SEKA-style head selection with PASTA’s attention steering. To address the misalignment between SEKA’s key-value head selection and PASTA’s attention head steering, we test two strategies. The first is applying the SEKA selection computation directly on the outputs of the attention heads. The second uses the results of the key-value head selection and 23 Published as a conference paper at ICLR 2026 applies them to attention heads via an interleaved repetition as the grouped-query attention mecha- nisms. For both hybrid methods, the selection criterion follows the SEKA methodology. Table 8: Complete PASTA results with different configurations: (1) using SEKA’s KV-head con- figuration (runtime 1–2 minutes), (2) using attention head configuration transformed from SEKA’s KV-heads (1–2 minutes), and (3) using PASTA’s original head-search routine (≈ 2 hours). ModelPASTA Configuration CounterFactBias in BiosPronoun Changing ESPSAcc.P. ScoreA. P. Score Qwen3-4B-Base SEKA KV-heads83.6280.4389.5855.4054.08 Transformed attention heads82.6083.0279.3495.8294.64 Original head-search97.1696.0387.8447.2047.20 Qwen3-8B-Base SEKA KV-heads82.0871.7286.3262.9161.44 Transformed attention heads78.2079.6977.3098.8698.72 Original head-search92.7091.6881.0482.2786.35 Qwen3-14B-Base SEKA KV-heads69.5263.3185.5285.5385.53 Transformed attention heads50.0661.0174.9878.7984.57 Original head-search76.8466.3388.4690.9890.94 Gemma-3-4B SEKA KV-heads55.6637.0571.4267.3966.43 Transformed attention heads78.3659.5382.5821.8921.65 Original head-search74.2852.9678.0216.0915.99 Gemma-3-12B SEKA KV-heads68.3065.7694.7265.4565.13 Transformed attention heads68.3065.7694.7265.6765.25 Original head-search62.6871.7276.2468.4768.01 IQUALITATIVE EXAMPLES I.1COUNTERFACT EXAMPLES Examples in Table 9 and 10 illustrate SEKA’s ability to steer the model towards newly provided factual information in the prompt, overriding its pre-trained knowledge. Table 9: CounterFact Example 1: Overriding a known location. Prompt Previously David Sainsbury, Baron Sainsbury of Turville used to work in London. Currently David Sainsbury, Baron Sainsbury of Turville used to work in Berlin. After 15 years of work, he published Pillboxes in 1985. David Sainsbury, Baron Sainsbury of Turville took up work in ModelGenerationCorrect? Gemma3-4BLondon✗ SEKA-Gemma3-4BBerlin✓ Table 10: CounterFact Example 2: Overriding a known job title. Prompt Previously Jean Baptiste Pompallier holds the position of bishop. Currently Jean Baptiste Pom- pallier holds the position of cardinal. Jean Baptiste Pompallier has the position of ModelGenerationCorrect? Gemma3-4Bbishop✗ SEKA-Gemma3-4Bcardinal✓ 24 Published as a conference paper at ICLR 2026 I.2BIAS IN BIOS EXAMPLES Examples shown in Table 11 and 12 demonstrate SEKA’s ability to focus the model’s attention on the correct, highlighted sentence in a biography, ignoring distracting information. Table 11: Bias in Bios Example 1: Correcting a prediction. Prompt Tania is an Illinois poet who has published one chapbook... and two full-length collections. Her most recent book... is divided into ten sections... Its predecessor Simple Weight is also concerned with the Biblical narrative... Barbara Crooker said of that collection, ”The poems have weight emotional, spiritual, political but are anything but simple.”.Tania has the occupation of a/an Qwen3-4B (Original)SEKA-Qwen3-4B Top Predictionjournalistpoet Target poetpoet Table 12: Bias in Bios Example 2: Improving confidence in the correct answer. Prompt Michelle, PhD is an Adjunct Assistant Professor in the Emory Department of Psychiatry and Behavioral Sciences. She received her Ph.D. in 2003 from Emory University. Her areas of expertise include intergenerational processes of risk and resilience... She supervises postdoctoral residents... Michelle has the occupation of a/an Qwen3-4B (Original)SEKA-Qwen3-4B Top Predictionpsychologistpsychologist Targetprofessorprofessor LogProb of Target (“professor”)-16.45-14.28 (Improved) I.3PRONOUNS CHANGING EXAMPLES Tables 13 and 14 demonstrate how SEKA addresses specific failures in following instructions. While the original models generally perform well in this task (over 0.9 P. Score), they occasionally make errors. SEKA can improve performance in certain instances. Table 13: Pronouns Changing Example 1. Prompt Jason is an assistant professor of English... For the aforementioned text, substitute ‘she’ and ‘he’ with ‘they’ and generate the occupation of Jason after changing pronouns. Qwen3-4B Generation (Incorrect) Jason is an assistant professor of English and as well as the coordinator of digital writing at Miami University of Ohio. He receivedhis PhD from Ohio State University in English with a focus on rhetoric, composition, and literacy. His most current publication is “Re-Inventing Invention: A Performance in Three Acts.”He is also a member of the Digital Writing Collaborative and a member of the Computers, Research, and Pedagogy committee. SEKA-Qwen3-4B Generation (Corrected) Jason is an assistant professor of English and as well as the coordinator of digital writing at Miami University of Ohio.They receivedtheir PhD from Ohio State University in English with a focus on rhetoric, composition, and literacy.Their most current publication is “Re-Inventing Invention: A Performance in Three Acts.”They are also a member of the Digital Writing Collaborative and a member of the Computers, Research, and Pedagogy committee. 25 Published as a conference paper at ICLR 2026 Table 14: Pronouns Changing Example 2. Prompt Anna, JD, MA, LPC, is a depth psychotherapist in Houston, Texas... She holds degrees in philoso- phy, law, and clinical psychology... For the aforementioned text, substitute ‘she’ and ‘he’ with ‘they’ and generate the occupation of Anna after changing pronouns. Qwen3-4B Generation (Incorrect) Anna, JD, MA, LPC, is a depth psychotherapist in Houston, Texas, with emphasis and training in Jungian psychology.She holds degrees in philosophy, law, and clinical psychology.She is a frequent lecturer/teacher at The Jung Center, recently teaching their two-year extensive introduction to Jung and analytical psychology. SEKA-Qwen3-4B Generation (Corrected) Anna, JD, MA, LPC, is a depth psychotherapist in Houston, Texas, with emphasis and training in Jungian psychology. They hold degrees in philosophy, law, and clinical psychology.They are a frequent lecturer/teacher at The Jung Center, recently teaching their two-year extensive introduction to Jung and analytical psychology. 020406080100 0.4 0.5 0.6 0.7 0.8 0.9 1 020406080100020406080100020406080100 BiasBios (Acc.)CounterFact (ES)PronChange (P. Score) Main Metrics Qwen3-4BQwen3-8BGemma-3-4BGemma-3-12B Projection samples Figure 9: The performance of SEKA with varying numbers of synthetic samples used for learning projections across different models and tasks. JPROJECTION SAMPLE EFFICIENCY ANALYSIS To explore how varying data quantity of synthetic samples affect the quality of learned subspace rep- resentations, we conduct an analysis on the end-to-end performance on the three tasks in the standard benchmark using projections extracted from different number of synthetic samples for SEKA. As shown in Figure 9, SEKA is generally data efficient across models and tasks. Performance typi- cally stabilises once roughly 50 synthetic samples are used, though the exact threshold depends on the task, architecture, and model size. More samples do not always yield higher peak performance, but they consistently produce more stable behaviour. With only a few samples, projections can overfit to the synthetic pairs and intro- duce unpredictable variance. Larger sample sizes mainly reduce this variance even when accuracy plateaus. Two additional observations emerge when breaking down the results. First, models within the same family display similar behaviour patterns. For Qwen3 models, CounterFact stabilises rela- tively early, while Gemma3 models, especially Gemma3-12B, require more samples for the same task. BiasBios and Pronouns Changing tend to stabilise faster across most settings. Second, though family-level similarities are observed, model size still introduces noticeable differences. Qwen3-8B is the clearest example: both Pronoun Changing and BiasBios fluctuate when fewer than 50 samples are used but become stable afterwards, but this fluctuation is not observed in Qwen3-4B. 26 Published as a conference paper at ICLR 2026 KCOMPLETE RESULTS FORδ min THRESHOLD ON LOST-IN-THE-MIDDLE As noted in Section 5.2, the optimal KV-head selection threshold (δ min ) can vary with model size. Figure 10 illustrates the effect of varying this threshold on the performance of the Qwen3-4B and Qwen3-14B models. 1st5th10th15th20th25th30th 0.35 0.4 0.45 0.5 0.55 1st5th10th15th20th25th30th 0.55 0.6 0.65 Original ModelSEKA (δmin=0.16)SEKA (δmin=0.165)SEKA (δmin=0.17) SEKA (δmin=0.175)SEKA (δmin=0.18) Gold Answer Position in ContextGold Answer Position in Context Exact Match Score Exact Match Score Qwen3-4B-BaseQwen3-14B-Base Figure 10: Exact match scores on the lost-in-the-middle task when applying SEKA to the middle region with different δ min thresholds for Qwen3-4B and Qwen3-14B. LTHE USE OF LARGE LANGUAGE MODELS (LLMS) We used LLMs as general-purpose tools to refine the writing and debug the code for this paper. The LLMs were not used for research ideation or to generate any significant portion of the text. The authors take full responsibility for the content of this paper. 27