Paper deep dive
Adversarial Lens: Exploiting Attention Layers to Generate Adversarial Examples for Evaluation
Kaustubh Dhole
Models: LLaMA-3.1-8B-Instruct
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/11/2026, 1:03:29 AM
Summary
The paper introduces 'Adversarial Lens', a method for generating adversarial examples for LLMs by exploiting token distributions within intermediate attention layers. By using lens-tuning to extract token predictions from these layers, the authors demonstrate that substituting tokens or using them to condition autoregressive generation can effectively degrade the performance of LLM-based evaluators on argument quality assessment tasks, while maintaining semantic similarity to original inputs.
Entities (4)
Relation Signals (3)
Adversarial Lens → evaluatedon → ArgQuality
confidence 95% · We conduct experiments on argument quality assessment using the ArgQuality dataset
LLaMA-3.1-Instruct-8B → usedwith → Lens-tuning
confidence 95% · We lens-tune the LLaMA-3.1-Instruct-8B model so that we can display the tokens at each layer
Adversarial Lens → degradesperformanceof → LLaMA-3.1-Instruct-8B
confidence 90% · Our results show that attention-based adversarial examples lead to measurable drops in evaluation performance
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Recent advances in mechanistic interpretability suggest that intermediate attention layers encode token-level hypotheses that are iteratively refined toward the final output. In this work, we exploit this property to generate adversarial examples directly from attention-layer token distributions. Unlike prompt-based or gradient-based attacks, our approach leverages model-internal token predictions, producing perturbations that are both plausible and internally consistent with the model's own generation process. We evaluate whether tokens extracted from intermediate layers can serve as effective adversarial perturbations for downstream evaluation tasks. We conduct experiments on argument quality assessment using the ArgQuality dataset, with LLaMA-3.1-Instruct-8B serving as both the generator and evaluator. Our results show that attention-based adversarial examples lead to measurable drops in evaluation performance while remaining semantically similar to the original inputs. However, we also observe that substitutions drawn from certain layers and token positions can introduce grammatical degradation, limiting their practical effectiveness. Overall, our findings highlight both the promise and current limitations of using intermediate-layer representations as a principled source of adversarial examples for stress-testing LLM-based evaluation pipelines.
Tags
Links
- Source: https://arxiv.org/abs/2512.23837
- Canonical: https://arxiv.org/abs/2512.23837
Trouble viewing inline? Open PDF directly →
Full Text
24,058 characters extracted from source content.
Expand or collapse full text
Adversarial Lens: Exploiting Attention Layers to Generate Adversarial Examples for Evaluation Kaustubh Dhole Department of Computer Science Emory University kdhole@emory.edu Abstract Recent advances in mechanistic interpretability suggest that intermediate attention layers en- code token-level hypotheses that are iteratively refined toward the final output. In this work, we exploit this property to generate adversarial examples directly from attention-layer token distributions. Unlike prompt-based or gradient- based attacks, our approach leverages model- internal token predictions, producing pertur- bations that are both plausible and internally consistent with the model’s own generation pro- cess. We evaluate whether tokens extracted from intermediate layers can serve as effective adversarial perturbations for downstream eval- uation tasks. We conduct experiments on ar- gument quality assessment using the ArgQual- ity dataset, with LLaMA-3.1-Instruct-8B serv- ing as both the generator and evaluator. Our results show that attention-based adversarial examples lead to measurable drops in evalua- tion performance while remaining semantically similar to the original inputs. However, we also observe that substitutions drawn from cer- tain layers and token positions can introduce grammatical degradation, limiting their prac- tical effectiveness. Overall, our findings high- light both the promise and current limitations of using intermediate-layer representations as a principled source of adversarial examples for stress-testing LLM-based evaluation pipelines. 1 Introduction Recent efforts in mechanistic interpretability have highlighted the wealth of information encoded within the layers of large language models (LLMs) (Meng et al., 2022; Sharkey et al., 2025). These layers, which are often overlooked in favor of the final outputs, have been shown to act as iter- ative predictors of the eventual response (Jastrzeb- ski et al., 2018; nostalgebraist, 2020), providing insights into the model’s generation process. While most of these techniques have heavily focused on interpretability, we argue that they could potentially be adapted to generate paraphrastic and adversarial examples for evaluation tasks – which generally operate over model generated data. Probing the attention layers has multiple advan- tages – First, since the layers act as both iterative indicators of the final output (Belrose et al., 2023), and store related entities (Meng et al., 2022; Her- nandez et al., 2024), they can provide natural lan- guage variations by potentially treating LLMs as knowledge bases. Second, these generations, can be obtained early on without necessitating running over all the layers (Din et al., 2024; Pal et al., 2023). Third, these generations might provide cues for model hallucinations (Yuksekgonul et al., 2024) as gradual deviations from the original token are ob- tained from the model itself. From an adversarial point of view, tokens from intermediate layers are valuable as they can act as perturbations to the orig- inal input. Specifically, the outputs are iteratively refined, since as the activations move towards the last layer they tend to move towards the direction of the negative gradient (Jastrzebski et al., 2018) or each successive layer achieving lower perplex- ity (Belrose et al., 2023). This is also precisely how adversarial examples are constructed – perturbing towards the direction of the positive gradient of the loss (Goodfellow et al., 2015). Hence, in this study, we explore whether such fine-grained information extracted from the atten- tion layers of large language models (LLMs) can be leveraged to generate adversarial examples on downstream natural language tasks, particularly critical tasks such as evaluation. Specifically, in this work, we introduce two attention-based adversarial generation methods: attention-based token substitution and attention- based conditional generation, both of which lever- age intermediate-layer token predictions to con- struct plausible yet adversarial inputs. Our paper is organized as follows: §2 first dis- cusses related work in interpretability and genera- arXiv:2512.23837v1 [cs.CL] 29 Dec 2025 tion evaluation. §3 and §4 defines and implements the two approaches for generating examples. §5 finally discusses the results and analysis. 2 Related Work We now discuss some of the related work in mech- anistic interpretability and LLM based evaluation to place our work in context. Recent interpretability studies have explored the information encoded within the internal layers of large language models (LLMs) to better understand how models generate subsequent tokens. For in- stance, LogitLens and TunedLens (nostalgebraist, 2020; Belrose et al., 2023) demonstrate that inter- mediate layers can be made to predict tokens simi- lar to those generated in the final layer, by attaching a trained or untrained unembedding matrix to them. Methods such as ROME (Meng et al., 2022) high- light the role of specific components, like MLP layers, in acting as key-value stores (Geva et al., 2021) that retain critical entities related to the in- put, such as associating “Seattle” with “Space Nee- dle.” Besides, LLMs assign greater attention to constraint tokens when their outputs are factual vis-à-vis when they are hallucinating (Yuksekgonul et al., 2024). These findings underscore the util- ity of the attention layers beyond interpretability but also as knowledge probes (Alain and Bengio, 2016), to extract inherent knowledge. Hence, in this work, we explore if the information from these layers can act as a resource for generating adversar- ial examples. Some techniques have been explored to probe intermediate layers to reveal token predictions (Bel- rose et al., 2023) for interpretability and early exit- ing to improve inference time (Geva et al., 2021). ResNets (Jastrzebski et al., 2018) have been shown to perform iterative feature refinement (where each block improves slightly but keeps the semantics of the representation of the previous layer)). On the other hand, evaluation of LLM-generated outputs has become increasingly important as mod- els are deployed in high-stakes settings. LLMs-as- judges have emerged as a common paradigm for evaluating generated text, either through prompting or via reward models trained on human preferences. In retrieval-augmented generation (RAG) systems, LLM-based evaluators are frequently used to as- sess dimensions such as answer quality, grounded- ness, and context relevance (Dhole, 2025, 2024). Colman Domingo was there to attend Domingo was there to attend Essence Figure 1: Tokens are extracted from intermediate layers to generate adversarial examples Recent work has also begun to examine the ro- bustness of such evaluators, demonstrating that groundedness and factuality judgments can be ma- nipulated through adversarial perturbations (Dhole et al., 2025). These findings motivate a closer ex- amination of whether adversarial examples derived from model internals pose additional risks to LLM- based evaluation frameworks. 3 Example Generation Prompt-based methods generate adversarial exam- ples by instructing an LLM to rewrite or manipulate a given input according to a natural language di- rective (Dhole et al., 2024; Dhole and Agichtein, 2024). Typically, the prompt includes a task de- scription, an example instance, and its associated label, followed by an instruction to generate an adversarial or misleading variant. While such methods are flexible and easy to ap- ply in black-box settings, the generated examples may not correspond to naturally occurring model mistakes. As a result, prompt-based adversaries may diverge substantially from the original input distribution and fail to reflect the kinds of errors that arise organically during model generation. For this reason, we focus our study on attention-based methods that exploit the model’s own intermediate token predictions. 3.1 Attention Based Methods In this section, we discuss our two methods of generating examples and how we use them for eval- uation. Both the methods start by substituting a token t x from a given text sequence. Tokens Predicted by Attention Layers In or- der to extract a novel tokent ′ x , we first gather the token distributions of all the layers at all the posi- tions. To do the same, we pass the input sequence through a LLaMA-3.1 instruct 8B (Dubey et al., Figure 2: This is the expanded figure with the complete evaluation prompt. Here, the token ‘disinformation’ is used to manipulate the model into changing the rating of the argument. 2024a) by first lens-tuning (Belrose et al., 2023) it over the final layer’s vocabulary. Lens-tuning, helps gauge the token distribution at each attention head and each layer as shown in Figure 1. Specifi- cally, the token distribution is obtained by passing each attention layer’s outputs through a linear layer and an unembedding matrix. Lens-tuning involves training this layer to minimize the KL-divergence between the vocabulary distribution of the final layer and that of an intermediate layer. 1 3.2 Attention Token Substitution This approach involves substituting a single token from an input sequence with a novel token esti- mated from the attention layers’ distribution. For- mally, the transformationTfor a givenn-token sequencet 1∶n perturbed at positionx ∈ [1, n]can be described as follows: T∶ (t 1 ,...,t x−1 ,t x ,t x+1 ,...)→ (t 1 ,...,t x−1 ,t ′ x ,t x+1 ,...) wheret ′ x = A(l, x) is selected from the attention to- ken distributionAat positionxof an attention layer l, representing a plausible yet distinct alternative to the original token t x . This method is lightweight and is likely to intro- duce minor semantic changes, allowing for efficient token-level editing with minimal disruption to the overall meaning. 3.3 Attention Token Conditioned Generation We propose a second method to address the poten- tial syntactic inconsistencies of the previous token substitution. In this approach, the substituted se- quence until tokenxserves as input for autoregres- sively generating the remaining sequence tokens, ensuring coherence and fluency. The transforma- tion is described as: T∶ (t 1 ,...,t x ,t x+1 ,...,t n )→ (t 1 ,...,t ′ x ,t ′ x+1 ,...,t ′ m ) 1 The tuned lens model is available on Hugging- Face (Wolf et al., 2020) athf.co/kdhole/Llama-3. 1-8B-Instruct-tuned-lens In this case, the model autoregressively gen- erates all tokens after positionxviz.,t ′ x+1 ,...,t ′ n by conditioning on the modified sequence t 1 ,...,t x−1 ,t ′ x , thereby aligning the entire sequence syntactically and semantically. The first token t ′ x = A(l, x)is obtained as earlier from the attention distribution. This method produces novel and diverse exam- ples while ensuring syntactic correctness and co- herence of the sequence. However, the newly gen- erated tokens may introduce shifts in meaning, di- verging from the original context. To minimize this effect, we choose token positions to substitute at the latter parts of the sequence so that there is a large overlap in the initial context used to dictate the rest of the tokens. The full procedure for ad- versarial example generation, including token and layer selection, is summarized in Algorithm 1. Algorithm 1: Adversarial Example Generation 1 Inputs: Input sequence t 1∶n ; model M; tuned-lens methodL; token/layer selection methodS; 2 Output: Modified sequence t ′ 1∶n ; 3 Train tuned-lens model (done once):; 4 M L ←L(M); 5 Compute internal token distributions:; 6 A = M L (t 1∶n ) where A ∈R L×n ; 7 Select token/layer position(s):; 8(x, l) =S(M L , A); 9 t ′ = A(x, l); 10 Substitute token:; 11 Replace t x with t ′ to obtain t ′ 1∶n ; 12 return t ′ 1∶n ; 4 Methods and Experiments For evaluation, we use the LLaMA-3.1-Instruct- 8B model (Dubey et al., 2024b). We focus ex- clusively on argument quality assessment using the ArgQuality corpus (Habernal and Gurevych, 2016), a task well-suited for token-level perturba- tions since small lexical changes can significantly alter perceived argument strength. L 16 32 48 64 80 128 28.423.403.389.437.479.420 24.408.403.417.423.471.389 20.451.403.366.451.429.403 16 .408.389.394.394.389.371 12.408.375.451.403.366.366 8.479.417.389.394.389.389 Table 1: Effect of Attention Token Substitution using adversarial tokens from different layers (L) and different token positions (T). L 16 32 48 64 80 128 28.280.400.480.393.333.316 24.330.400.370.346.400.500 20.300.380.430.328.263.500 16.320.410.360.439.471.591 12.280.460.380.482.526.450 8 .250.410.400.418.350.273 Table 2: Effect of Attention Token Conditioned Gener- ation using adversarial tokens from different layers (L) and different token positions (T). ArgQuality classifies arguments into three cat- egories: low, average, and high quality. We con- struct evaluation instances in the form of (topic, stance, chosen argument, rejected argument) tuples and measure how often the model correctly prefers the higher-quality argument. This setting allows us to directly assess whether attention-based adversar- ial examples can degrade evaluation performance without substantially altering semantic content. We first assess answer quality, by evaluating whether these examples are useful for testing the robustness capabilities of both trained and fine- tuned models. This was done by first evaluating the model’s performance on ArgQuality’s test set and the modified test set generated from the two meth- ods discussed in §3. Our evaluation set consists of 75 test examples from the ArgQuality corpus which we transform in the form of (topic, stance, chosen argument, rejected argument) tuples. Adverserial Example Generation: We lens- tune the LLaMA-3.1-Instruct-8B model so that we can display the tokens at each layer (Belrose et al., 2023) and use them for generating adversarial se- quences. We specifically generate the adversar- ial counterparts for the chosen and rejected argu- ments in the same. In our experiments, we choose x = 10andl = 18. For evaluation, we use a few-shot prompt displayed in Figure 3. Few-shotFine-tuned Original Test Set.42.60 Adversarial Lens Test Set.34.57 Table 3: Argument Quality Evaluator Performance 5 Results The results indicate that adversarial tokens ex- tracted from a wide range of layers and token po- sitions can negatively impact evaluation accuracy. However, we also observe that certain configura- tions—particularly substitutions at later token po- sitions—can paradoxically improve performance. This suggests that not all intermediate-layer tokens act as effective adversarial perturbations, and that both layer depth and token position play a critical role in determining adversarial effectiveness. Ta- bles 1 and 2 show that substitutions drawn from mid-to-late layers (e.g., layers 16–28) generally in- duce larger performance drops than those from ear- lier layers, particularly when applied at moderate token positions. However, very late token substi- tutions occasionally improve accuracy, likely by introducing clarifying or corrective lexical choices. Table 3 further confirms this trend at the aggre- gate level: evaluation accuracy drops from 0.42 to 0.34 in the few-shot setting and from 0.60 to 0.57 in the fine-tuned setting when adversarial ex- amples generated via our attention-based methods are introduced. Together, these results indicate that attention-layer-derived perturbations can reliably degrade evaluator performance, though their im- pact is highly sensitive to where in the sequence and from which layer the token is extracted. 6 Conclusion Evaluation tasks provide a natural test bed for attention-based adversarial example generation, as LLM-based judges routinely consume model- generated text that may already contain subtle in- consistencies or errors. In this work, we show that intermediate attention layers can be exploited to generate adversarial examples without requiring access to the model’s final layer or gradients. While our results demonstrate consistent suc- cessful attacks, we also find that many intermediate- layer substitutions lead to grammatical degradation, limiting their effectiveness as practical adversaries. In that regard, we introduce the attention token conditioned approach. Our preliminary study high- lights the need for more selective token and layer "Rate the quality of the given argument among ’low’, ’average’ and ’high’. Just mention either of the options and do not provide an explanation. The argument should be rated high if it convinces the reader towards the expected stance for a controversial topic. " " topic is ’Ban Plastic Water Bottles’. stance is ’No bad for the economy’. is the argument: U.S. alone grew by over 13%. According to research and consulting done by the Beverage Marketing Corporation, the global bottled water industry has exploded to over $35 billion. Americans alone paid $7.7 billion for bottled water in 2002. In 2001, for example, globally bottled water companies produced over 130,000 million liters of water. This produced roughly 35,000 million dollars in revenue for the world’s thousands of bottled water companies in 2001. : ’average’" " topic is ’Is porn wrong’. stance is ’Yes porn is wrong’. is the argument: Porn is definitely wrong. Porn is like an addiction to some people which is unhealthy and can lead to guilt and lust. An addiction to porn gives an unhealthy image of real sex. Porn promotes the fact that sex is totally based on pleasure, but it is actually based on love and affection also. Porn inspired numerous crimes that sometimes abuse the rights and virginity of many people. : ’high’" " topic is ’William Farquhar ought to be honoured as the rightful founder of Singapore’. stance is ’Yes of course’. is the argument: Farquhar contributed significantly, even forking out his own money to start up the colony carved out of the jungle, by first offering money as an incentive for people to hunt and to exterminate rats and centipedes. Raffles did nothing of that sort. : ’low’" f" topic is topic. stance is stance. is the argument: arg :" Figure 3: The few-shot prompt used for rating argument quality. selection mechanisms, and the potential to extract knowledge from intermediate layers gradually. Future work should explore principled criteria for identifying syntactically valid and semantically impactful substitutions, as well as extending these methods to domains where strict linguistic structure is less critical. For example, structured domains such as electronic health records, where substitut- ing diagnosis or procedure codes may have sig- nificant downstream effects, present a promising direction for applying attention-based adversarial methods. We focus on evaluation tasks rather than gen- eral classification, as evaluation models are espe- cially likely to encounter generations influenced by near-final-layer representations. Overall, this study demonstrates that intermediate-layer repre- sentations offer a promising—but currently imper- fect—source of adversarial examples for stress- testing LLM evaluation pipelines. Acknowledgments The author thanks Eugene Agichtein from Emory University for insightful discussions. References Guillaume Alain and Yoshua Bengio. 2016. Under- standing intermediate layers using linear classifier probes. Nora Belrose, Zach Furman, Logan Smith, Danny Ha- lawi, Igor Ostrovsky, Lev McKinney, Stella Bider- man, and Jacob Steinhardt. 2023. Eliciting latent predictions from transformers with the tuned lens. arXiv preprint arXiv:2303.08112. Kaustubh Dhole. 2024. PyTerrier-GenRank: The PyTer- rier Plugin for Reranking with Large Language Mod- els. Kaustubh Dhole and Eugene Agichtein. 2024. Llm judges for retrieval augmented argumentation. Kaustubh Dhole, Ramraj Chandradevan, and Eugene Agichtein. 2025. AdvERSEM: Adversarial robust- ness testing and training of LLM-based groundedness evaluators via semantic structure manipulation. In Proceedings of the 14th Joint Conference on Lexical and Computational Semantics (*SEM 2025), pages 395–408, Suzhou, China. Association for Computa- tional Linguistics. Kaustubh D Dhole. 2025.To retrieve or not to retrieve?uncertainty detection for dynamic re- trieval augmented generation.arXiv preprint arXiv:2501.09292. Kaustubh D. Dhole, Kai Shu, and Eugene Agichtein. 2024. Conqret: Benchmarking fine-grained evalua- tion of retrieval augmented argumentation with llm judges. Alexander Yom Din, Taelin Karidi, Leshem Choshen, and Mor Geva. 2024. Jump to conclusions: Short- cutting transformers with linear transformations. In Proceedings of the 2024 Joint International Con- ference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 9615–9625. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024a. The llama 3 herd of models. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024b. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. Transformer feed-forward layers are key- value memories. In Proceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Pro- cessing, pages 5484–5495, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and harnessing adversar- ial examples. In International Conference on Learn- ing Representations. Ivan Habernal and Iryna Gurevych. 2016. Which argu- ment is more convincing? analyzing and predicting convincingness of web arguments using bidirectional lstm. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 1589–1599. Evan Hernandez, Arnab Sen Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg, Jacob Andreas, Yonatan Belinkov, and David Bau. 2024. Linearity of rela- tion decoding in transformer language models. In The Twelfth International Conference on Learning Representations. Stanisław Jastrzebski, Devansh Arpit, Nicolas Ballas, Vikas Verma, Tong Che, and Yoshua Bengio. 2018. Residual connections encourage iterative inference. In International Conference on Learning Representa- tions. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associ- ations in gpt. Advances in Neural Information Pro- cessing Systems, 35:17359–17372. nostalgebraist.2020.Interpreting gpt:Thelogitlens.https://w. lesswrong.com/posts/AcKRB8wDpdaN6v6ru/ interpreting-gpt-the-logit-lens.Accessed: 2024-12-24. Koyena Pal, Jiuding Sun, Andrew Yuan, Byron C Wal- lace, and David Bau. 2023. Future lens: Anticipating subsequent tokens from a single hidden state. In Pro- ceedings of the 27th Conference on Computational Natural Language Learning (CoNLL), pages 548– 560. Lee Sharkey, Bilal Chughtai, Joshua Batson, Jack Lind- sey, Jeff Wu, Lucius Bushnaq, Nicholas Goldowsky- Dill, Stefan Heimersheim, Alejandro Ortega, Joseph Bloom, et al. 2025. Open problems in mechanistic interpretability. arXiv preprint arXiv:2501.16496. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, et al. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 con- ference on empirical methods in natural language processing: system demonstrations, pages 38–45. Mert Yuksekgonul, Varun Chandrasekaran, Erik Jones, Suriya Gunasekar, Ranjita Naik, Hamid Palangi, Ece Kamar, and Besmira Nushi. 2024. Attention satis- fies: A constraint-satisfaction lens on factual errors of language models. In The Twelfth International Conference on Learning Representations.