Paper deep dive
Pitfalls in Evaluating Interpretability Agents
Tal Haklay, Nikhil Prakash, Sana Pandey, Antonio Torralba, Aaron Mueller, Jacob Andreas, Tamar Rott Shaham, Yonatan Belinkov
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 3/23/2026, 12:11:43 PM
Summary
The paper investigates the challenges of evaluating autonomous interpretability agents designed for circuit analysis in deep neural networks. It introduces an agentic system that iteratively designs experiments and refines hypotheses, comparing its performance against human expert explanations across six tasks. The authors identify significant pitfalls in replication-based evaluation, such as subjectivity in human labels, potential memorization by LLMs, and the limitations of outcome-based comparisons. To mitigate these, they propose an unsupervised intrinsic evaluation metric based on functional interchangeability.
Entities (5)
Relation Signals (3)
TransformerLens â implements â Interpretability Tools
confidence 100% · Our tools are implemented with TransformerLens
Agentic System â performs â Circuit analysis
confidence 95% · we build an agentic system in which a research agent iteratively designs experiments and refines hypotheses
Claude Opus 4.1 â powers â Agentic System
confidence 95% · Claude Opus 4.1 (Anthropic, 2025), which underlies our agent
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Automated interpretability systems aim to reduce the need for human labor and scale analysis to increasingly large models and diverse tasks. Recent efforts toward this goal leverage large language models (LLMs) at increasing levels of autonomy, ranging from fixed one-shot workflows to fully autonomous interpretability agents. This shift creates a corresponding need to scale evaluation approaches to keep pace with both the volume and complexity of generated explanations. We investigate this challenge in the context of automated circuit analysis -- explaining the roles of model components when performing specific tasks. To this end, we build an agentic system in which a research agent iteratively designs experiments and refines hypotheses. When evaluated against human expert explanations across six circuit analysis tasks in the literature, the system appears competitive. However, closer examination reveals several pitfalls of replication-based evaluation: human expert explanations can be subjective or incomplete, outcome-based comparisons obscure the research process, and LLM-based systems may reproduce published findings via memorization or informed guessing. To address some of these pitfalls, we propose an unsupervised intrinsic evaluation based on the functional interchangeability of model components. Our work demonstrates fundamental challenges in evaluating complex automated interpretability systems and reveals key limitations of replication-based evaluation.
Tags
Links
- Source: https://arxiv.org/abs/2603.20101v1
- Canonical: https://arxiv.org/abs/2603.20101v1
Trouble viewing inline? Open PDF directly â
Full Text
125,280 characters extracted from source content.
Expand or collapse full text
Pitfalls in Evaluating Interpretability Agents Tal Haklay 1 Nikhil Prakash 2 Sana Pandey 3 Antonio Torralba 3 Aaron Mueller 4 Jacob Andreas 3 Tamar Rott Shaham 3 Yonatan Belinkov 1 5 Abstract Automated interpretability systems aim to reduce the need for human labor and scale analysis to in- creasingly large models and diverse tasks. Recent efforts toward this goal leverage large language models (LLMs) at increasing levels of autonomy, ranging from fixed one-shot workflows to fully autonomous interpretability agents. This shift creates a corresponding need to scale evaluation approaches to keep pace with both the volume and complexity of generated explanations. We inves- tigate this challenge in the context of automated circuit analysisâexplaining the roles of model components when performing specific tasks. To this end, we build an agentic system in which a research agent iteratively designs experiments and refines hypotheses. When evaluated against hu- man expert explanations across six circuit analysis tasks in the literature, the system appears competi- tive. However, closer examination reveals several pitfalls of replication-based evaluation: human expert explanations can be subjective or incom- plete, outcome-based comparisons obscure the research process, and LLM-based systems may reproduce published findings via memorization or informed guessing. To address some of these pitfalls, we propose an unsupervised intrinsic eval- uation based on the functional interchangeability of model components. Our work demonstrates fundamental challenges in evaluating complex au- tomated interpretability systems and reveals key limitations of replication-based evaluation. 1 1. Introduction Interpretability research aims to explain the internal com- putations of deep neural networks, such as how models im- 1 TechnionâIIT 2 Northeastern University 3 MIT 4 Boston Uni- versity 5 Kempner Institute at Harvard University. Correspondence to: Tal Haklay <tal.ha@campus.technion.ac.il>. Preprint. March 23, 2026. 1 Codeavailableathttps://github.com/ technion-cs-nlp/interp-agent. plement in-context learning, solve arithmetic tasks, or track grammatical relationships across sentences (Elhage et al., 2021; Wang et al., 2022; Hanna et al., 2023; Todd et al., 2024; Davies et al., 2023; Prakash et al., 2024; Nikankin et al., 2025; Todd et al., 2025; Feucht et al., 2025). This work typically requires researchers to formulate hypotheses, design targeted experiments, and iteratively refine their con- clusionsâa process in which human expertise becomes a bottleneck. A growing body of work seeks to address this bottleneck by automating interpretability workflows (Her- nandez et al., 2021; Bills et al., 2023; Rott Shaham et al.; Paulo et al., 2025; Gur-Arieh et al., 2025; Minder et al., 2025). Rather than relying on expert-crafted procedures, these systems execute analysis subroutines autonomously, often by leveraging AI models themselves. For example, LLMs are commonly used to interpret neurons or features by identifying patterns in the input tokens that activate them. In such approaches, human researchers define which ex- periments to run, while the LLM automates the analysis. Recently, Rott Shaham et al. demonstrated that autonomous interpretability agents can go furtherâdesigning experi- ments themselves and using the results to iteratively refine their hypotheses, much like human researchers. However, to trust such automated systems to produce accurate conclu- sions, they must be evaluated carefully. Evaluating the discoveries of autonomous interpretability systems is inherently challenging: their outputs are open- ended, can vary substantially across runs, and often lack clear ground-truth explanations. Moreover, equally valid ex- planations may differ in their level of abstraction or framing. Together, these factors make a scalable, automated evalu- ation particularly difficult. We investigate this evaluation challenge through the task of analyzing circuits in language modelsâsubnetworks of attention-heads and MLPs that implement specific computational pathways (Elhage et al., 2021; Meng et al., 2022). To understand how a model solves a task, researchers aim to identify these circuits, characterize the functional roles of individual components, and group components into clusters that jointly implement higher-level mechanisms (Wang et al., 2022; Hanna et al., 2023; Prakash et al., 2024; Merullo et al., 2024; Tigges et al., 2024; Garc Ì Ä±a- Carrasco et al., 2024). While substantial prior work has focused on automating the discovery of circuit components 1 arXiv:2603.20101v1 [cs.AI] 20 Mar 2026 Pitfalls in Evaluating Interpretability Agents CLUSTER A CLUSTER B CLUSTER C logit_lens() attention_pattern() activation_patching() INTERMEDIATE RESULTS EXPERIMENT CALLS logit_lens( prompt=âThe capital city of France isâ, token_position=-1, layer_head_pair=(15, 13)) attention_pattern( prompt=âThe capital city of France isâ, token_position=-1, layer_head_pair=(15, 13)) activation_patching( source_prompt=âThe capital city of France isâ, counterfactul_prompt=âThe capital city of Spain isâ, token_position_clean=-1, token_position_counterfatual=-1, layer_head_pair=(15, 13)) patching results: clean probabilities:[('France', 0.78), ('france', 0.12)] patched probabilities:[('France', 0.68),('france', 0.1)] probability differences:[('France', -0.1),('france', -0.02)] attention scores: [('The', 0.0), (' capital', 0.1),(' city', 0.1), (' of', 0.0),('France', 0.8),(' is', 0.0)] top tokens: [('France', 0.59), ('Fr', 0.12), ('croissant', 0.11)] Figure 1. Top: Overview of the system workflow. (A) A researcher specifies a task and a circuit to analyze. (B) A researcher agent iteratively analyzes each component independently, autonomously designing and running experiments as needed. (C) Claude clusters components based on shared functionality inferred from the generated hypotheses. Bottom: Example of tool calls and the corresponding results returned by the tools. (Conmy et al., 2023; Syed et al.; Haklay et al., 2025), we focus on a less explored stage of the pipeline: assigning semantic, functional interpretations to the components of an already identified circuit for the task. This stage produces the human-interpretable explanations that describe how indi- vidual components contribute to the modelâs behavior on the task. To this end, we study an agentic system, in which a re- search agent operates iterativelyâautonomously designing experiments and refining its analysis over multiple steps. To assess whether such systems can replicate expertsâ re- search outcomes, we collect six prior circuit-analysis tasks from the literature and compare system outputs to explana- tions produced by human researchers (Meng et al., 2022; Hanna et al., 2023; Tigges et al., 2024; Prakash et al., 2024; Merullo et al., 2024). Under this evaluation, our system appears competitive with prior analyses. However, closer examination reveals that apparent agreement with human- authored explanations masks deeper challenges in evaluat- ing automated interpretability systems. Human-provided explanations may be incomplete or subjective; indeed, we identify cases in which our systems produce more accurate interpretations than those reported in prior work. Further- more, Claude Opus 4.1 (Anthropic, 2025), which underlies our agent, exhibits evidence of memorization on at least one evaluated task and can infer others using superficial cues rather than experimental reasoning. These failure modes point to broader limitations of replication-based evaluation. Evaluations that rely on hu- man expert analyses are costly to obtain, can be subjec- tive, and may reflect memorization rather than genuine un- derstanding. To address these challenges, we propose an unsupervised intrinsic evaluation based on the functional interchangeability of model components, enabling assess- ment of component descriptions without reliance on expert judgment and at scale. While this metric does not address all the pitfalls discussed in this paper, it serves as an initial step toward scalable, unsupervised evaluation of automated circuit analysis systems. To summarize, our work makes three main contributions: (i) a systematic analysis of an agentic system for circuit anal- ysis; (i) an exposition of key pitfalls in replication-based evaluation; and (i) an unsupervised evaluation approach based on functional similarity. 2. Automatic Circuit Analysis Our goal is to automate the procedure of circuit analysis: given a completion task (e.g., Indirect Object Identification (IOI); Wang et al. 2022), a set of task-specific prompts, and a list of model components participating in the taskâs com- putation (i.e., the underlying circuit), we aim to produce a human-interpretable description of the functional roles of the circuitâs components. For example, such a descrip- tion may explain what types of task-relevant information an attention head retrieves from the input, or what types of computations an MLP performs to facilitate correct pre- 2 Pitfalls in Evaluating Interpretability Agents diction. Prior work has largely been performed by human researchers using interpretability techniques such as atten- tion pattern analysis and causal interventions (Vig et al., 2020; Wang et al., 2022). This process is inherently iterative, typically requiring researchers to identify patterns across large volumes of experimental results and to formulate and test hypotheses using targeted prompts. Consequently, prior work in circuit analysis has typically required substantial time and the involvement of multiple researchers. To automate this process, we develop an agentic system cen- tered on a research agent that autonomously and iteratively proposes hypotheses and designs experiments. Given a set of 20 task prompts P and a set of components C identified as belonging to the taskâs circuit (e.g., attention heads or MLPs), the system outputs a collection of component clus- ters and their corresponding natural-language descriptions, (K i ,d i ) m i=1 , where eachK i â Cgroups components by shared functionality. Figure 1 provides a high-level overview of the system, which follows a three-stage workflow: (i) a researcher specifies a task and a circuit to analyze; (i) each circuit component is analyzed independently by a research agent to produce a task-specific functional description; and (i) an LLM clusters the components based on shared functionality. We next describe stages (i) and (i). 2 2.1. Component-Level Analysis At this stage, the research agent produces a task-specific ex- planation for each of the circuit components independently. This analysis is grounded in experimental outputs from stan- dard tools that are widely used in mechanistic interpretabil- ity work: vocabulary projections, causal interventions, and attention patterns. See Section 2.2 for details. The research agent is built on Claude Opus 4.1, guided by a system prompt (see App. A.1) and equipped with the tools described below. Analogous to a human researcher, the agent operates iteratively. Each iteration of the agent follows a structured three-step process: (i) It begins by analyzing the experimental results obtained in the previous iteration. (i) Based on this evidence, it articulates a set of3â5plausible hypotheses that could explain the observed behavior. (i) It then proposes the next set of experiments, specifying both the function calls and the rationale behind each experiment. For an example of a single agent iteration, see App. C.1. An experimental run concludes once the agent returns a <final hypothesis>tag without making any further function calls. The final output of the agent is a concise de- 2 Given the progress in circuit discovery (Conmy et al., 2023; Hanna et al., 2024; Mueller et al., 2025; Ameisen et al., 2025), we assume stage (i) is given; that is, we assume access to the ground truth components. scription of the componentâs functionality, accompanied by a brief summary of key evidence supporting it (see App. C.2 for an example). 2.2. Interpretability Tools Our tools are implemented with TransformerLens (Nanda & Bloom, 2022), and described next. Logit lens (nostalgebraist, 2020): The logit lens projects in- termediate hidden states into the modelâs output space using the final LayerNorm and the unembedding matrix. For a hid- den stateh (l) t at layerl, the resulting distribution over the vo- cabulary items is given byp (l) t = softmax(W U LN(h (l) t )). This tool has been applied in numerous studies to analyze intermediate computations in language models (Geva et al., 2021; 2022; Katz et al., 2024, inter alia). Activation patching (Vig et al., 2020; Geiger et al., 2020): A causal intervention on a model componentcthat measures how the model output would have changed if the output of cwere replaced with its value obtained from processing a different input. Given a clean promptx clean and a counter- factual promptx cf , activation patching replaces the output ofcat a target position in the clean run with the corre- sponding activation from the counterfactual run. Such tools have been instrumental in numerous interpretability studies (Wang et al., 2022; Meng et al., 2022; Prakash et al., 2024). Attention patterns:Given a prompt, a token position, and an attention head, this function returns the corresponding attention pattern. Such attention measures are simple yet useful tools that help researchers form hypotheses about the roles of attention heads (Clark et al., 2019; Vig et al., 2020; Wang et al., 2022). To mitigate attention sink effects, we ignore the BOS token position and re-normalize the attention weights (Vig & Belinkov, 2019). Token Positions: Given a prompt, this function returns the tokenized prompt along with token indices. This tool allows the agent to identify which token positions to target when running experiments, and is particularly useful when the agent generates new prompts for analysis. The interaction with these tools is mediated through a struc- tured interface: at each iteration, the agent specifies which tools to invoke, and their corresponding arguments, includ- ing generating input prompts (e.g., designing prompt pairs for activation patching). Then, an external system executes the requested functions, and the outputs are returned to the agent for subsequent analysis. As an initialization, we pre- compute attention-pattern and logit-lens of 10 task prompts and include the outputs in the prompt to the agent. 3 Pitfalls in Evaluating Interpretability Agents Explanation _ Explanation _ Explanation _ Hypothesis Figure 2. The judge workflow. The system produces hypotheses explaining the functionality of individual components and clusters. A judge model is tasked with matching the hypothesis to one of the descriptions reported by the researchers in the original paper. 2.3. Clustering Components by Shared Functionality Prior work has shown that multiple components within a cir- cuit may implement similar functionality and can therefore be grouped into clusters, enabling higher-level summaries of model behavior (Wang et al., 2022; Hanna et al., 2023; Prakash et al., 2024). Moreover, such clusters typically consist of components that operate at the same token posi- tion. After the agent produces a final description for each component, we perform a second analysis in which these component-level descriptions are aggregated and clustered by functional similarity. This stage does not involve any ad- ditional experiments. Instead, we prompt Claude Opus 4.1 to cluster components based on their final functional role descriptions and the associated evidence (see App A.2). Fol- lowing prior work, clustering is performed separately for each token position. The outcome of this stage is a set of clusters, each accompanied by a description of the shared functionality of components in it. 3. Extrinsic Evaluation with Human Explanations Evaluating explanations is a core and long-standing chal- lenge in interpretability research (Doshi-Velez & Kim, 2017; Jacovi & Goldberg, 2020; Schwettmann et al., 2023; Huang et al., 2023). While human judgments are often treated as the gold standard, manual evaluation introduces a substantial scalability bottleneck, making it impractical for automated interpretability research. We therefore seek an evaluation protocol that is both scalable and automated, allowing ex- planations to be assessed without human supervision. A common approach for evaluating research agents is to test whether they replicate findings from published work. This paradigm underlies benchmarks across domains in- cluding machine learning engineering (Chan et al., 2025), astrophysics (Ye et al., 2025), and scientific discovery more broadly (Starace et al., 2025). In our setup, we test whether our system can reproduce explanations produced by hu- man researchers. We identify six studies that performed circuit analysis, including explicit descriptions of compo- nentsâ functionalities and corresponding clusters (Wang et al., 2022; Hanna et al., 2023; Prakash et al., 2024; Merullo et al., 2024; Tigges et al., 2024; Garc Ì Ä±a-Carrasco et al., 2024). Table 1 provides a summary of the task setups. Ad- ditional details for each task are provided in App. B. Table 1. Summary of tasks from prior circuit analyses. Each cluster represents a group of components with shared functionality. TaskModelCircuit Size#Clusters IOIGPT-2-Small18 Heads6 IOIPythia-160M14 Heads7 Greater-ThanGPT-2-Small8 Heads, 4 MLPs2 AcronymsGPT-2-Small8 Heads3 Colored ObjectsGPT-2-Medium27 Heads3 Entity TrackingLLaMA-7B64 Heads4 3.1. Metrics To measure how well agent-generated explanations match those reported in prior work, we use an LLM-as-a-judge approach. A judge model (GPT-5) receives an explanation from the agent and is tasked with matching it with one human explanation out of all possible human explanations for the circuit clusters (see Figure 2). The judge is allowed to assign multiple agent-generated explanations to the same human-generated explanation. We repeat this process for two types of the agentâs explanations; first, we evaluate explanations of each component (stage i), then we evaluate the agentâs explanations for the functionality of each cluster (stage i). We evaluate performance using three metrics: Component Functionality Accuracy. The proportion of in- dividual component-level explanations that the judge model correctly matches to their human-labeled clusters. Cluster Functionality Accuracy. The proportion of com- ponents in each agent-generated cluster that the judge as- signs to the correct human-labeled cluster. Formally, let K =K 1 ,...,K m denote the set of clusters produced by the agent. We compute the fraction of the correctly assigned components in each clusterK j and average across clusters: Accuracy = 1 m m X j=1 1 |K j | X câK j 1[Ëy(c) = y(c)] (1) Component Assignment Accuracy. This metric measures how well the predicted clusters align with expert-defined clusters from prior work and does not involve the judge. Since cluster labels are arbitrary (the system might call a cluster âAâ while the paper calls it âName Moversâ), we cannot directly compare assignments. Instead, we find the optimal one-to-one matching between predicted and ex- pert clusters with the Hungarian algorithm (Kuhn, 2010), which maximizes the number of components that end up in corresponding clusters. We then report the fraction of components correctly assigned under the optimal matching. 4 Pitfalls in Evaluating Interpretability Agents IOI GPT-2 Small IOI Pythia-160M Greater-Than Acronyms Colored Objects Entity Tracking 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy Component Functionality Accuracy IOI GPT-2 Small IOI Pythia-160M Greater-Than Acronyms Colored Objects Entity Tracking 0.0 0.2 0.4 0.6 0.8 1.0 Cluster Functionality Accuracy IOI GPT-2 Small IOI Pythia-160M Greater-Than Acronyms Colored Objects Entity Tracking 0.0 0.2 0.4 0.6 0.8 1.0 Component Assignment Accuracy AgenticOne-shot Figure 3. Performance comparison across six circuit analysis tasks. Left: Component Functionality Accuracy measures how well individual component explanations match human-labeled clusters. Middle: Cluster Functionality Accuracy measures how well cluster explanations match human-labeled clusters. Right: Component Assignment Accuracy assesses cluster alignment with expert-defined clusters using optimal matching. While the systems obtain relatively high results, they usually do not perfectly match expert explanations. We report both Component Assignment Accuracy and Clus- ter Functionality Accuracy, as they evaluate different aspects of agreement: the former assesses agreement on which com- ponents share a function, while the latter assesses agreement on what that function is. 3.2. Does Greater Autonomy Improve Performance? As a baseline to our agentic system, we also examined a common paradigm in automated interpretability. Rather than using a research agent to analyze each component, we provide an LLM with experimental outputs as static input and ask it to produce interpretations in a single pass. This approach follows prior work in which LLMs are used to in- terpret activation patterns of neurons or sparse autoencoder (SAE) features (Bills et al., 2023; Paulo et al., 2025). In this setup, which we refer to as the one-shot system, we select 10 of the 20 prompts provided in the system prompt and apply all available interpretability tools described in Sec. 2.2. For patching experiments, we adopt a simple strategy by randomly sampling 10 counterfactual prompts from the task dataset. All resulting experimental outputs are then provided to Claude Opus 4.1 via the system prompt, which analyzes them and produces a description of each componentâs functional role. 3.3. Results For each task, we run the component-level analysis three times and perform clustering five times per analysis, yield- ing fifteen clusterings; all reported results are averaged across runs. In total, the agent issued 6,382 tool calls, aver- aging 14.2 calls and 4.5 iterations per component analysis. Figure 3 presents the evaluation results. While neither sys- tem consistently matches human-level analyses, both the agentic and one-shot systems achieve competitive perfor- mance. Notably, increased autonomy does not necessarily lead to consistent performance gains: the agentic system performs comparably to the one-shot baseline. We further analyze the comparable performance in Section 4.2. 4. Pitfalls in Evaluating Automated Interpretability Systems While the evaluation framework in Section 3 is intended to assess whether the two systems can reproduce human cir- cuit analyses, closer examination reveals several important limitations. We examine these limitations in detail below. 4.1. Findings in Prior Work May Be Ambiguous Disagreements between the systemâs hypotheses and pub- lished descriptions do not always reflect model error. In several cases, they arise from subjectivity or incompleteness in prior human interpretations. For example, attention head 2.6 in the IOI task on Pythia-160M had been described as a previous-token head (Tigges et al., 2024), yet the agentâs experiments did not consistently reveal a previous-token attention pattern. To test this, we conducted a broader evalu- ation over 150 examples from the PILE dataset (Gao et al., 2020), The previous token received the highest attention score in only 42% of cases, raising the question of whether this head should really be labeled a âprevious-tokenâ head. Similarly, in the Entity Tracking task (Prakash et al., 2024), which uses prompts of the form âThe apple is in box C, the cat is in box D, . . . Box C contains theâ, a large group of components is labeled as value fetcher heads, responsible for retrieving the value associated with the correct object. In contrast, both systems identify several heads whose be- havior does not consistently match the description in the original paper. For further analysis, see App. D. These cases highlight a broader challenge in circuit analysis: observed behaviors are not always consistent across inputs, and when a component exhibits a pattern in only a subset of test cases, assigning it a single functional explanation can become ambiguous. 5 Pitfalls in Evaluating Interpretability Agents âSarah and Michael went to the park. Sarah talked toâ âMark wrote and James read the letter. Mark then handed it toâ âDavid and Lisa met David's friend. Lisa gave a gift toâ âThe company and the client discussed terms. The company sent a proposal toâ âTest if the head's behavior is specific to giving actions or works with other verb types by creating prompts with different action typesâ âTest with prompts where the same person is mentioned twice to see if position or identity matters moreâ âTest with prompts where both entities perform actions to see how the head handles ambiguityâ âTest if the head works with non-human entities and abstract concepts to confirm it's about syntactic roles rather than semantic understanding of givingâ IOI example: âWhen Mary and John went to the store, John gave a drink toâ Figure 4. Examples of the agentâs experimental designs. Blue: an IOI task example from the initial prompt set provided to the agent. Orange: the agentâs motivation for the experiment. Green: new example prompts proposed by the agent to test its hypotheses. 4.2. Outcome-Based Evaluation ignores Research Process Our evaluation compares final explanations against those re- ported in prior work. However, this outcome-based compar- ison discards information about the research process itself: two researchers may arrive at the same conclusion through fundamentally different investigative paths. Figure 4 shows that the agent engages in substantive hypothesis generation and experimental design, with particular emphasis on test- ing the generalization of its hypotheses and refining them by evaluating prompts that differ from those provided initially. In contrast, the one-shot baseline is unable to refine its hy- pothesis or explore alternative explanations. We view this as a key advantage of the agentic system over the one-shot baseline. Yet the current evaluation framework provides no direct mechanism to reward this behavior. 4.3. Are We Actually Measuring Reasoning? 4.3.1. SIGNS OF MEMORIZATION A key concern in using LLMs such as Claude Opus 4.1 as the backbone for both systems is the possibility that the model memorized the published findings during training 3 (Carlini et al., 2021; 2023). By directly prompting Claude to list all circuitâs components and their functionalities (without run- ning any experiments), we confirmed that it has memorized at least the IOI circuit, including the exact components (e.g., layer and head indices) and the functionality attributed to each head (See App. E.1). Additionally, GPT-5, the model used as our judge, also showed signs of direct recall of the IOI task. See Figure 5 and App. E.1 for concrete examples. Our experience suggests that, without explicit tests for mem- orization, such phenomena are easy to overlook: (i) During analysis, Claude did not state that it recognized the IOI 3 We use Claude through API without retrieval capabilities, so there is no risk of it recalling information from retrieved sources. Systems with search capability would need to account for that too. task; (i) Most generated explanations did not reuse the ter- minology employed in the original paper; and (i) most surprisingly, both systemsâ reproductions were not perfect, despite effectively knowing the researchersâ findings. Even in other tasks where Claude did not explicitly acknowl- edge familiarity with published findings, it was often able to infer plausible functionalities that closely resembled those reported in the papers. For instance, it was able to âguessâ a high-level algorithm similar to that identified in the Entity Tracking circuit (see App. E.2). We acknowledge that this issue may be amplified in our experimental setting, where only a small set of well-established behaviors are docu- mented in the literature. Consequently, the hypothesis space is relatively small, making it easier for the agent to arrive at accurate guesses without conducting any experiments. We argue that these observations have important implica- tions for future research on automated interpretability. Com- parisons between research agents and published work should be made with care, explicitly accounting for mem- orization and informed guessing. The limited number of well-studied circuit analysis tasks further amplifies these risks, making replication-based evaluations particularly vul- nerable to memorization and overfitting. 4.3.2. DISENTANGLING MEMORIZATION FROM REASONING We further investigate whether Claudeâs performance in the IOI task reflects genuine reasoning grounded in experimen- tal evidence, or whether it reflects memorization. To disentangle memorization from reasoning, we progres- sively add noise to the experimental results and examine the resulting impact on system performance. If Claude does not rely on the experimental evidence, we would expect only limited degradation as noise increases. For each tool (Section 2.2), we generate a ânoisedâ version of the original results by randomly permuting them. For example, in the 6 Pitfalls in Evaluating Interpretability Agents âThis attention head identifies and predicts indirect objects in a giving/transfer scenarios by attending to the complementary person when one entity in a pair performs an action, effectively implementing a "name mover" pattern that copies the non-acting entity as the recipient.â âThe circuit consists of several types of specialized attention heads: ### Name Mover Heads - **L9H9, L9H6, L10H0** (among others in layers 9-11) - Role: These heads directly copy the correct name from its earlier position to the final token position, enabling the model to output the correct indirect object ### S-Inhibition Heads (S-Inhibition = Subject Inhibition) - **L7H3, L7H9, L8H6, L8H10** - Role: These heads suppress attention to the subject (the repeated name that is NOT the answer), preventing the model from incorrectly predicting the subject ### Duplicate Token Heads - **L0H1, L0H10, L1H4, L2H2, L3H0** - Role: These early-layer heads identify which name appears twice in the sequence, marking it as the subject to be avoided ### Previous Token Heads - **L2H2, L4H11** (among others) - Role: These heads attend from each position to the previous token, helping to route information through the circuit ### Induction Heads - **L5H5, L5H8, L5H9, L6H9** - Role: These implement a copying pattern, attending from the second occurrence of a name back to tokens following its first occurrenceâ Describe the functional role of head 9.9 in the ioi circuit What do you know about the ioi circuit? Figure 5. Top: Example of indirect memorization in a final hy- pothesis produced by the one-shot system for attention head 9.9 in the IOI task. The system explicitly uses the term âname mover headâ, reproducing the terminology introduced in the original IOI paper. For more details on âname moverâ heads, see App. B.1. Bottom: Example of direct recall of the IOI circuit by Claude when it is explicitly asked to recall the circuit from memory. For more details on the prompt used and the full response, see App. E.1. logit-lens experiments, we randomize the probabilities as- signed to each token. See App. F for further details. We then construct a noisy result using a mixing parameterαâ [0, 1]: (1â α)· originalresult + α· permutedresult. (2) We run both systems on the IOI task forα â [0, 0.5, 0.6, 0.7, 0.8, 0.9, 1] , repeating each experiment three times with different seeds. Results are shown in Figure 6. Whenα > 0.7, both sys- tems exhibit substantial performance degradation, suggest- ing that their behavior relies more heavily on experimental evidence than on memorization. At lowerαvalues, we observe minimal performance degradation. This may either reflect memorization or suggest the system is able to re- cover the underlying structure of the original results despite the noise. Overall, these results suggest that memorization alone cannot fully explain system behavior across noise regimes. 0.00.20.40.60.81.0 0.2 0.4 0.6 0.8 Accuracy Agentic Non-Agentic Figure 6. Each point represents the mean component functionality accuracy, averaged across all components in the IOI task, at a given noise levelα. Both systems exhibit clear performance degradation as noise increases. Running the experiments with additional ran- dom seeds yields similar trends (see App. F). 5.Intrinsic Evaluation Without Ground Truth Explanations In this section, we present an alternative, fully automated evaluation approach that does not rely on expert explana- tions. The core idea is to assess how coherent (or tight) the discov- ered clusters are: if a system groups together components that truly share an underlying function, then components within the same cluster should exhibit similar behavior, whereas components in different clusters should behave differently. To evaluate this, we require a distance function that places functionally similar components close together and functionally distinct components far apart. Rather than relying on subjective human judgments, we define a model- grounded, behavior-based measure of component similarity via swap-invariance: if two components implement the same underlying function, then swapping one componentâs weights with those of the other should leave the modelâs behavior largely unchanged. Implementation details. Concretely, we focus on atten- tion heads. For a given pair of heads, we consider swaps of their KQ circuits and separately their OV circuits (El- hage et al., 2021). The OV circuit, defined by the value and output projection matrices(W V ,W O ), determines what information the head writes to the residual stream. The KQ circuit, defined by the key and query projection matrices (W K ,W Q ), determines which token positions the head at- tends to. For each pair of attention heads, we separately swap their KQ and OV circuits and measure the resulting change in the modelâs output behavior. Evaluation metrics. To quantify the effect of each swap, we measure the JensenâShannon distance (i.e., the square root of the JensenâShannon divergence) between 7 Pitfalls in Evaluating Interpretability Agents Table 2. Intrinsic cluster quality. Silhouette-based cluster quality scores for random, expert-defined, agentic, and one-shot clusterings across tasks. For agentic and one-shot systems, we report the best and mean scores across runs. The final column reports the Kendall rank correlation (Ï ) and associated p-value between intrinsic cluster quality and component assignment accuracy. RandomExpertAgenticOne-shotKendall (Ï,p) TaskMeanBestMeanBest IOI-0.28± 0.070.210.03± 0.090.12-0.08± 0.130.21(0.739, 0.00) IOI (Pythia-160M)-0.24± 0.080.100.07± 0.100.150.00± 0.080.09(0.034, 0.81) Acronyms-0.18± 0.130.16-0.02± 0.170.300.00± 0.100.30(0.506, 0.02) Colored-Objects-0.28± 0.070.020.01± 0.070.050.02± 0.050.06(0.318, 0.02) Entity-Tracking-0.24± 0.010.17-0.13± 0.02-0.09-0.09± 0.02-0.08(0.378, 0.00) the modelâs original next-token prediction distribution and the distribution obtained after the swap. We then define the distance between two attention headsh 1 andh 2 as the average of the distances induced by swapping their KQ and OV circuits: dist(h 1 ,h 2 ) = 1 2 q JSD KQ (h 1 ,h 2 ) + p JSD OV (h 1 ,h 2 ) (3) We measure the distance between every pair of heads within each task by running these swap experiments on 40 task examples. This distance yields a fully unsupervised, model- grounded measure of functional similarity between attention heads, independent of any expert explanations. Using this distance, we compute the average silhouette score under a given clustering assignment, which serves as an intrinsic measure of cluster quality. We evaluate cluster quality across four clustering types: (1) clusters produced by the agentic system, (2) clusters produced by the one-shot system, (3) clusters reported in the original papers (referred to as expert clusters), and (4) random clusterings. For the latter, we randomly assign circuit components to clusters while matching the cluster- size distributions of the system-generated clusterings. Results. Table 2 reports the results of this analysis. As expected, random clusterings yield low, negative silhou- ette scores, serving as a baseline for this analysis. Expert clusters achieve much higher scores. Both the agentic and one-shot systems achieve silhouette scores close to zero on average, but still above the random baseline. However, some individual clustering assignments attain relatively high scores, suggesting that while average performance under this evaluation is modest, the systems occasionally discover meaningful structure in the circuit representations. We additionally compute the Kendall rank correlation be- tween our intrinsic metric and Component Assignment Ac- curacy, which measures agreement with clusters reported in prior work. For all tasks except the IOI Pythia-160, we observe positive correlations, indicating that clusters that are more coherent under our intrinsic, unsupervised metric often align with expert-defined clusters. While the strength of this relationship varies across tasks, these results suggest that the proposed intrinsic measure captures meaningful structure consistent with prior analyses. Figure 7 illustrates this con- sistency. We plot the swap-based distances from attention head(10, 0), identified in prior work as a NAME MOVER head, to all other heads in the IOI circuit. Heads within the NAME MOVER cluster exhibit systematically smaller swap-induced distances than heads from other clusters. Notably, both the extrinsic and the intrinsic evaluations ex- hibit substantial run-to-run variance due to the stochasticity of the underlying model. The proposed intrinsic metric provides an unsupervised measure of cluster quality, allow- ing practitioners to compare multiple runs and select more coherent clustering without relying on ground-truth expla- nations. We emphasize that this approach does not address all of the pitfalls discussed in Section 4. Accordingly, it should be viewed as a proof of concept for unsupervised evaluation. Name Mover Induction Negative Name Mover S Inhibition Previous Token Duplicate Token 0.00025 0.00050 0.00075 0.00100 0.00125 0.00150 0.00175 0.00200 Distance Distances From head (10, 0) Figure 7. Swap-based distances from attention head(10, 0)(a NAME MOVER head) to all other heads in the IOI circuit, grouped by the reported functional clusters. Other NAME MOVER heads are closer to head (10, 0) than heads from different clusters. 8 Pitfalls in Evaluating Interpretability Agents 6. Conclusions As automated interpretability systems become more au- tonomous and widely deployed, evaluation methodologies must evolve to keep pace with their capabilities. In this work, we show that commonly used replication-based evalu- ationsâcomparing agent-generated explanations to prior hu- man analysesâcan be systematically misleading. Through empirical analysis, we identify several failure modes of this paradigm, including reliance on incomplete or subjective ground-truth explanations, insensitivity to the reasoning pro- cess by which explanations are produced, and vulnerability to memorization or informed guessing by LLMs. To begin addressing these challenges, we propose an unsu- pervised intrinsic evaluation framework based on functional interchangeability of model components. By grounding evaluation directly in model behavior, our approach enables scalable assessment of cluster coherence without relying on human evaluation or prior expert explanations. While this metric is limited in scope and does not resolve all iden- tified pitfalls, our results suggest that it captures meaning- ful functional structure and complements replication-based evaluations. Taken together, our findings highlight the central role of evaluation in the design of automated interpretability sys- tems. As such systems take on increasingly open-ended scientific roles, effective evaluation will require methodolo- gies that are robust to memorization, sensitive to reasoning processes, and grounded in the behavior of the underlying models. We view this work as a step toward such evaluation frameworks and as a call for more principled benchmarks for agentic interpretability. Impact Statement This work aims to advance the evaluation of automated interpretability systems, which are increasingly used to ana- lyze and explain the behavior of machine learning models. Reliable interpretability is important for transparency and oversight, particularly as models are deployed in high-stakes settings. Our findings highlight that commonly used replication- based evaluation frameworks can obscure important failure modes, including memorization of prior analyses, reliance on superficial cues, and sensitivity to subjective or incom- plete ground-truth explanations. These issues could lead to overconfident or misleading conclusions about the ca- pabilities and trustworthiness of automated interpretability systems. By identifying these pitfalls, our work aims to support the development of more robust and reliable evaluation method- ologies. We hope these insights will help future research anticipate evaluation failures, design stronger benchmarks, and apply automated interpretability tools more cautiously and responsibly. Acknowledgments This research was supported by the Israel Science Founda- tion (grant No. 2942/25), and the European Union (ERC, Control-LM, 101165402). Views and opinions expressed are those of the authors only and do not necessarily re- flect those of the European Union or the European Re- search Council Executive Agency; neither the European Union nor the granting authority can be held responsible for them. Additional support was provided by the ARL grant #W911NF-24-2-0069, the MIT-IBM Watson AI Lab grant #W1771646, Hyundai Motor Company, and the DARPA AIQ program through the DARPA CMO contract number HR00112520025. TH, NP, AM, and YB were supported by Coefficient Giving. TH is also supported by the Council for Higher Education (VATAT) Scholarship for Outstanding Women Doctoral Students in Hi-Tech. SP is supported by the National Science Foundation Graduate Research Fellow- ship Program under Grant No. (2141064). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. References Ameisen, E., Lindsey, J., Pearce, A., Gurnee, W., Turner, N. L., Carter, S., Hosmer, B., Marcus, J., Sklar, M., Templeton, A., Bricken, T., McDougall, C., Cunningham, H., Henighan, T., Jermyn, A., Jones, A., Persic, A., Qi, Z., Thompson, T. B., Zimmerman, S., Rivoire, K., Conerly, T., Olah, C., and Batson, J. Circuit tracing: Revealing computational graphs in language mod- els.https://transformer-circuits.pub/ 2025/attribution-graphs/methods.html, March 2025. Anthropic.Claude 4.1 model card.https://w. anthropic.com/news/claude-opus-4-1 , 2025. Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., OâBrien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., Skowron, A., Sutawika, L., and van der Wal, O. Pythia: A suite for analyz- ing large language models across training and scaling. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), International Confer- ence on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, p. 2397â2430. PMLR, 9 Pitfalls in Evaluating Interpretability Agents 2023. URLhttps://proceedings.mlr.press/ v202/biderman23a.html. Bills, S., Cammarata, N., Mossing, D., Tillman, H., Gao, L., Goh, G., Sutskever, I., Leike, J., Wu, J., and Saunders, W.Language models can explain neurons in language models.https: //openaipublic.blob.core.windows.net/ neuron-explainer/paper/index.html, 2023. Carlini, N., Tram ` er, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T. B., Song, D., Erlingsson, Ì U., Oprea, A., and Raffel, C. Extracting training data from large language models. In Bailey, M. D. and Greenstadt, R. (eds.), 30th USENIX Security Symposium, USENIX Security 2021, August 11-13, 2021, p. 2633â2650. USENIX As- sociation, 2021.URLhttps://w.usenix. org/conference/usenixsecurity21/ presentation/carlini-extracting. Carlini, N., Ippolito, D., Jagielski, M., Lee, K., Tram ` er, F., and Zhang, C. Quantifying memorization across neural language models.In The Eleventh Interna- tional Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URLhttps://openreview.net/forum? id=TatRHT_1cK. Chan, J. S., Chowdhury, N., Jaffe, O., Aung, J., Sher- burn, D., Mays, E., Starace, G., Liu, K., Maksin, L., Patwardhan, T., Madry, A., and Weng, L. Mle-bench: Evaluating machine learning agents on machine learn- ing engineering. In The Thirteenth International Confer- ence on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URLhttps: //openreview.net/forum?id=6s5uXNWGIh. Clark, K., Khandelwal, U., Levy, O., and Manning, C. D. What does BERT look at? an analysis of bertâs attention. In Linzen, T., Chrupala, G., Belinkov, Y., and Hupkes, D. (eds.), Proceedings of the 2019 ACL Workshop Black- boxNLP: Analyzing and Interpreting Neural Networks for NLP, BlackboxNLP@ACL 2019, Florence, Italy, August 1, 2019, p. 276â286. Association for Computational Linguistics, 2019. doi: 10.18653/V1/W19-4828. URL https://doi.org/10.18653/v1/W19-4828. Conmy, A., Mavor-Parker, A., Lynch, A., Heimersheim, S., and Garriga-Alonso, A. Towards automated circuit dis- covery for mechanistic interpretability. Advances in Neu- ral Information Processing Systems, 36:16318â16352, 2023. Davies, X., Nadeau, M., Prakash, N., Shaham, T. R., and Bau, D. Discovering variable binding circuitry with desiderata, 2023. URLhttps://arxiv.org/abs/ 2307.03637. Doshi-Velez, F. and Kim, B.Towards a rigorous sci- ence of interpretable machine learning. arXiv preprint arXiv:1702.08608, 2017. Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., DasSarma, N., Drain, D., Ganguli, D., Hatfield-Dodds, Z., Hernandez, D., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan, J., McCandlish, S., and Olah, C. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. URLhttps://transformer-circuits. pub/2021/framework/index.html. Feucht, S., Todd, E., Wallace, B., and Bau, D. The dual- route model of induction. In Second Conference on Lan- guage Modeling, 2025. URLhttps://arxiv.org/ abs/2504.03022. Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, I., et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. Garc Ì Ä±a-Carrasco, J., Mat Ì e, A., and Trujillo, J. C. How does GPT-2 predict acronyms? extracting and understanding a circuit via mechanistic interpretability. In Dasgupta, S., Mandt, S., and Li, Y. (eds.), International Conference on Artificial Intelligence and Statistics, 2-4 May 2024, Palau de Congressos, Valencia, Spain, volume 238 of Pro- ceedings of Machine Learning Research, p. 3322â3330. PMLR, 2024. URLhttps://proceedings.mlr. press/v238/garcia-carrasco24a.html. Geiger, A., Richardson, K., and Potts, C. Neural nat- ural language inference models partially embed theo- ries of lexical entailment and negation. In Alishahi, A., Belinkov, Y., Chrupala, G., Hupkes, D., Pinter, Y., and Sajjad, H. (eds.), Proceedings of the Third Black- boxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, BlackboxNLP@EMNLP 2020, On- line, November 2020, p. 163â173. Association for Com- putational Linguistics, 2020. doi: 10.18653/V1/2020. BLACKBOXNLP-1.16. URLhttps://doi.org/ 10.18653/v1/2020.blackboxnlp-1.16. Geva, M., Schuster, R., Berant, J., and Levy, O. Transformer feed-forward layers are key-value memories. In Proceed- ings of the 2021 Conference on Empirical Methods in Natural Language Processing, p. 5484â5495, 2021. Geva, M., Caciularu, A., Wang, K., and Goldberg, Y. Trans- former feed-forward layers build predictions by promot- ing concepts in the vocabulary space. In Proceedings 10 Pitfalls in Evaluating Interpretability Agents of the 2022 conference on empirical methods in natural language processing, p. 30â45, 2022. Gur-Arieh, Y., Mayan, R., Agassy, C., Geiger, A., and Geva, M. Enhancing automated interpretability with output- centric feature descriptions. In Che, W., Nabende, J., Shutova, E., and Pilehvar, M. T. (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July 27 - August 1, 2025, p. 5757â5778. Association for Computational Linguistics, 2025. doi: 10. 18653/V1/2025.ACL-LONG.288. URLhttps://doi. org/10.18653/v1/2025.acl-long.288. Haklay, T., Orgad, H., Bau, D., Mueller, A., and Belinkov, Y. Position-aware automatic circuit discovery. In Che, W., Nabende, J., Shutova, E., and Pilehvar, M. T. (eds.), Pro- ceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July 27 - August 1, 2025, p. 2792â2817. Association for Computational Linguistics, 2025. URLhttps://aclanthology.org/2025. acl-long.141/. Hanna, M., Liu, O., and Variengien, A. How does gpt-2 compute greater-than?: Interpreting mathematical abili- ties in a pre-trained language model. Advances in Neural Information Processing Systems, 36:76033â76060, 2023. Hanna, M., Pezzelle, S., and Belinkov, Y. Have faith in faithfulness: Going beyond circuit overlap when finding model mechanisms. In First Conference on Language Modeling, 2024. URLhttps://openreview.net/ forum?id=TZ0CCGDcuT. Hernandez, E., Schwettmann, S., Bau, D., Bagashvili, T., Torralba, A., and Andreas, J. Natural language descrip- tions of deep visual features. In International Conference on Learning Representations, 2021. Huang, J., Geiger, A., DâOosterlinck, K., Wu, Z., and Potts, C. Rigorously assessing natural language ex- planations of neurons.In Belinkov, Y., Hao, S., Jumelet, J., Kim, N., McCarthy, A., and Mohebbi, H. (eds.), Proceedings of the 6th BlackboxNLP Work- shop: Analyzing and Interpreting Neural Networks for NLP, BlackboxNLP@EMNLP 2023, Singapore, Decem- ber 7, 2023, p. 317â331. Association for Compu- tational Linguistics, 2023.doi: 10.18653/V1/2023. BLACKBOXNLP-1.24. URLhttps://doi.org/ 10.18653/v1/2023.blackboxnlp-1.24. Jacovi, A. and Goldberg, Y. Towards faithfully interpretable NLP systems: How should we define and evaluate faith- fulness?In Jurafsky, D., Chai, J., Schluter, N., and Tetreault, J. R. (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, p. 4198â4205. As- sociation for Computational Linguistics, 2020. doi: 10. 18653/V1/2020.ACL-MAIN.386. URLhttps://doi. org/10.18653/v1/2020.acl-main.386. Katz, S., Belinkov, Y., Geva, M., and Wolf, L. Back- ward lens: Projecting language model gradients into the vocabulary space. In Al-Onaizan, Y., Bansal, M., and Chen, Y. (eds.), Proceedings of the 2024 Confer- ence on Empirical Methods in Natural Language Pro- cessing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, p. 2390â2422. Association for Compu- tational Linguistics, 2024.doi: 10.18653/V1/2024. EMNLP-MAIN.142. URLhttps://doi.org/10. 18653/v1/2024.emnlp-main.142. Kuhn, H. W. The hungarian method for the assignment problem. In J Ì unger, M., Liebling, T. M., Naddef, D., Nemhauser, G. L., Pulleyblank, W. R., Reinelt, G., Ri- naldi, G., and Wolsey, L. A. (eds.), 50 Years of Inte- ger Programming 1958-2008 - From the Early Years to the State-of-the-Art, p. 29â47. Springer, 2010. doi: 10.1007/978-3-540-68279-0\2. URL https://doi. org/10.1007/978-3-540-68279-0_2. Meng, K., Bau, D., Andonian, A., and Belinkov, Y. Locating and editing factual associations in gpt. Advances in neural information processing systems, 35:17359â17372, 2022. Merullo, J., Eickhoff, C., and Pavlick, E. Circuit component reuse across tasks in transformer language models. In The Twelfth International Conference on Learning Repre- sentations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URLhttps://openreview. net/forum?id=fpoAYV6Wsk. Minder, J., Dumas, C., Slocum, S., Casademunt, H., Holmes, C., West, R., and Nanda, N. Narrow finetuning leaves clearly readable traces in activation differences, 2025. URL https://arxiv.org/abs/2510.13900. Mueller, A., Geiger, A., Wiegreffe, S., Arad, D., Arcuschin, I., Belfki, A., Chan, Y. S., Fiotto-Kaufman, J. F., Hak- lay, T., Hanna, M., Huang, J., Gupta, R., Nikankin, Y., Orgad, H., Prakash, N., Reusch, A., Sankaranarayanan, A., Shao, S., Stolfo, A., Tutek, M., Zur, A., Bau, D., and Belinkov, Y. MIB: A mechanistic interpretability benchmark. In Forty-second International Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 13-19, 2025. OpenReview.net, 2025. URLhttps: //openreview.net/forum?id=sSrOwve6vb. Nanda, N. and Bloom, J.Transformerlens.https: //github.com/TransformerLensOrg/ TransformerLens, 2022. 11 Pitfalls in Evaluating Interpretability Agents Nikankin, Y., Reusch, A., Mueller, A., and Belinkov, Y. Arithmetic without algorithms: Language models solve math with a bag of heuristics. In The Thirteenth Inter- national Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URLhttps://openreview.net/forum? id=O9YTt26r2P. nostalgebraist.Thelogitlens,2020. URLhttps://w.lesswrong. com/posts/AcKRB8wDpdaN6v6ru/ interpreting-gpt-the-logit-lens .Blog post. Paulo, G., Mallen, A., Juang, C., and Belrose, N. Automati- cally interpreting millions of features in large language models. In Forty-second International Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 13-19, 2025. OpenReview.net, 2025. URLhttps: //openreview.net/forum?id=EemtbhJOXc. Prakash, N., Shaham, T. R., Haklay, T., Belinkov, Y., and Bau, D. Fine-tuning enhances existing mechanisms: A case study on entity tracking. In The Twelfth Interna- tional Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URLhttps://openreview.net/forum? id=8sKcAWOf2D. Rott Shaham, T., Schwettmann, S., Wang, F., Rajaram, A., Hernandez, E., Andreas, J., and Torralba, A. A mul- timodal automated interpretability agent. In Forty-first International Conference on Machine Learning. Schwettmann, S., Rott Shaham, T., Materzynska, J., Chowd- hury, N., Li, S., Andreas, J., Bau, D., and Torralba, A. Find: A function description benchmark for evaluating interpretability methods. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. Starace, G., Jaffe, O., Sherburn, D., Aung, J., Chan, J. S., Maksin, L., Dias, R., Mays, E., Kinsella, B., Thomp- son, W., Heidecke, J., Glaese, A., and Patwardhan, T. Paperbench: Evaluating aiâs ability to replicate AI re- search. In Forty-second International Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 13-19, 2025. OpenReview.net, 2025. URLhttps: //openreview.net/forum?id=xF5PuTLPbn. Syed, A., Rager, C., and Conmy, A. Attribution patching outperforms automated circuit discovery, 2023. URL https://arxiv. org/abs/2310.10348. Tigges, C., Hanna, M., Yu, Q., and Biderman, S. LLM circuit analyses are consistent across training and scale. In Globersons, A., Mackey, L., Belgrave, D., Fan, A., Pa- quet, U., Tomczak, J. M., and Zhang, C. (eds.), Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, 2024. Todd, E., Li, M. L., Sharma, A. S., Mueller, A., Wal- lace, B. C., and Bau, D. Function vectors in large language models.In Proceedings of the 2024 In- ternational Conference on Learning Representations, 2024. URLhttps://openreview.net/forum? id=AwyxtyMwaG. arXiv:2310.15213. Todd, E., Brinkmann, J., Gandikota, R., and Bau, D. In- context algebra, 2025. URLhttps://arxiv.org/ abs/2512.16902. Vig, J. and Belinkov, Y. Analyzing the structure of atten- tion in a transformer language model. arXiv preprint arXiv:1906.04284, 2019. Vig, J., Gehrmann, S., Belinkov, Y., Qian, S., Nevo, D., Singer, Y., and Shieber, S. M. Investigating gender bias in language models using causal mediation analysis. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Process- ing Systems 33: Annual Conference on Neural Informa- tion Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. Wang, K., Variengien, A., Conmy, A., Shlegeris, B., and Steinhardt, J. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. arXiv preprint arXiv:2211.00593, 2022. Ye, C., Yuan, S., Cooray, S., Dillmann, S., Roque, I. L. V., Baron, D., Frank, P., Martin-Alvarez, S., Kob- lischke, N., Qu, F. J., Yang, D., Wechsler, R., and Ciuca, I. Replicationbench: Can AI agents replicate astrophysics research papers? CoRR, abs/2510.24591, 2025. doi: 10.48550/ARXIV.2510.24591. URLhttps: //doi.org/10.48550/arXiv.2510.24591. 12 Pitfalls in Evaluating Interpretability Agents Appendix A. Tools Instructions for invoking and interpreting the interpretability tools are provided to both systems through their respective system prompts. ModelInvest Class 1 class ModelInvest: 2 3 def __init__(self, model_name: str, hf_name: str = None, hf_tokenizer: str = None): 4 """ 5 Initialize the ModelInvest class with a given model. 6 7 Args: 8 model_name: Name of the model in TransformerLens 9 hf_name: Optional HuggingFace model name for loading via HF 10 11 Example: 12 >>> # Initialize with TransformerLens model only 13 >>> model_invest = ModelInvest("gpt2-small") 14 >>> 15 >>> # Initialize with both TransformerLens and HuggingFace model 16 >>> model_invest = ModelInvest("llama-7b", "meta-llama/Llama-2-7b") 17 """ 18 model, tokenizer = None, None 19 if hf_name and hf_tokenizer: 20 model = AutoModelForCausalLM.from_pretrained( 21 hf_name, 22 torch_dtype=torch.float32, 23 device_map="cpu" 24 ) 25 tokenizer = AutoTokenizer.from_pretrained(hf_tokenizer) 26 tokenizer.pad_token_id = tokenizer.eos_token_id 27 tokenizer.padding_side = "right" 28 tokenizer.add_bos_token = False 29 30 model = HookedTransformer.from_pretrained( 31 model_name=model_name, 32 hf_model=model, 33 tokenizer=tokenizer, 34 center_writing_weights=False, 35 center_unembed=False, 36 trust_remote_code=True, 37 fold_ln=False, 38 device_map="auto") 39 model.set_use_attn_result(True) 40 #model.set_use_attn_in(True) 41 #model.set_use_hook_mlp_in(True) 42 #model.set_use_split_qkv_input(True) 43 self.model = model 44 self.model_name = model_name 45 self.prepend_bos=True 46 47 48 def get_token_indices_in_prompt(self, prompts: List[str]) -> List[Tuple[str, int]]: 49 """ 50 Get tokens and their indices in a list of prompts. 51 13 Pitfalls in Evaluating Interpretability Agents 52 This function is very important. It can be used before calling other functions 53 (logit_lens, attention_map_generation, run_patching) so it will be clear what 54 is the position of each token. Understanding token positions is critical for 55 accurate interpretation of model internals and for targeted interventions. 56 57 Args: 58 prompt: Input text, list of prompts (List[str]) 59 60 Returns: 61 Formatted string with numbered prompts and their tokens 62 63 Example: 64 >>> # List of prompts 65 >>> prompts = ["Hello world", "Hi there"] 66 >>> model_invest.get_token_indices_in_prompt(prompts) 67 Prompt 1: "Hello world" 68 Tokens: [(âHelloâ, 0), (â worldâ, 1)] 69 70 Prompt 2: "Hi there" 71 Tokens: [(âHiâ, 0), (â thereâ, 1)] 72 73 >>> # For character-level tokenizers, each character might be a token 74 >>> prompt = "Hi!" 75 >>> model_invest.get_token_indices_in_prompt(prompt) 76 [(âHâ, 0), (âiâ, 1), (â!â, 2)] 77 """ 78 pass 79 80 81 def attention_map_generation(self, prompts: List[str], query_positions: List[ int], layer_head_pairs: List[Tuple[int, int]]) -> str: 82 """ 83 Generate attention maps for multiple prompts across multiple layer-head combinations. 84 85 Args: 86 prompts: List of input texts 87 query_positions: List of positions in prompts to examine attention from (one per prompt) 88 layer_head_pairs: List of tuples (layer, head) to examine 89 90 Returns: 91 String description of attention maps for all combinations 92 93 Example: 94 95 >>> # List of prompts and layer-head pairs 96 >>> prompts = ["Hello world", "The cat sat"] 97 >>> query_positions = [0, 1] # First token for prompt 1, second token for prompt 2 98 >>> layer_head_pairs = [(2, 1), (3, 5), (4, 0)] 99 >>> attention_maps = model_invest.attention_map_generation(prompts, query_positions, layer_head_pairs) 100 101 """ 102 pass 103 104 105 def run_patching(self, source_prompts: List[str], counterfactual_prompts: List[ str], 14 Pitfalls in Evaluating Interpretability Agents 106 Optional[int]]], token_positions_source: List[int], 107 token_positions_counterfactual: List[int], layer_head_pairs: List[ Tuple[int, tokens: List[str] = None, top_k: int = 10) -> str: 108 """ 109 110 Performs causal intervention by replacing activations from source prompts with 111 counterfactual activations at specified positions. Analyzes probability changes 112 to determine whether components store semantic content vs. structural patterns. 113 114 Args: 115 source_prompts: List of clean/original prompts 116 counterfactual_prompts: List of prompts with alternative entities/contexts 117 token_positions_source: Token positions in source prompts to patch FROM 118 token_positions_counterfactual: Token positions in counterfactual prompts to patch TO 119 layer_head_pairs: List of (layer, head) tuples. Use None for head to patch MLP 120 tokens: Specific tokens to analyze probability changes for (optional) 121 top_k: Number of top probability tokens to return if tokens not specified 122 123 Returns: 124 Formatted results showing probability changes for each layer-head combination 125 126 Example: 127 >>> # Test what "Italy" activation encodes 128 >>> sources = ["The capital city of France is Paris. The capital city of Italy is"] 129 >>> counterfactuals = ["The capital city of Austria is Vienna. The capital city of Germany is"] 130 >>> layer_head_pairs = [(8, 3), (10, 5), (12, None)] # Test attention heads + MLP 131 >>> results = model_invest.run_patching(sources, counterfactuals, 132 ... token_positions_source=[10], # "Italy " position 133 ... token_positions_counterfactual=[10], # "Germany" position 134 ... layer_head_pairs) 135 136 137 ** Experimental Guidelines: ** 138 139 1. ** Prompt Design ** : Source prompts must stop right before the key prediction token. 140 The probability shift of this key token reveals what the patched activation encoded. 141 Examples: "The capital city of France is", "The author of 1984 is", "I want to eat" 142 143 2. ** What Properties to Test ** : 144 145 ** Value Representation Test: ** 146 - Patch entity A -> entity B 147 - If probability increases for entity Bâs associated answer: activation stores semantic identity 148 - Example: Italy->Austria patch increases "Vienna" probability = country identity encoded 149 150 ** Position Representation Test: ** 151 - Same patch as above 152 - If probability increases for entity Aâs associated answer: activation stores structural position 15 Pitfalls in Evaluating Interpretability Agents 153 - Example: Italy->Austria patch increases "Paris" probability = positional pattern encoded 154 155 156 ** Interpretation Framework: ** 157 158 ** Direction of Shift Reveals Representational Content Type: ** 159 160 ### PATCHING INCREASES CORRECT TOKEN PROBABILITY (patched_probabilities - clean_probabilities > 0): 161 ** Interpretation ** : The original component was INTERFERING with correct prediction 162 ** Action ** : Removing/noising this component would IMPROVE performance 163 ** Example ** : Patch increases "Rome" probability -> head was suppressing the correct answer 164 165 ### PATCHING DECREASES CORRECT TOKEN PROBABILITY (patched_probabilities - clean_probabilities < 0): 166 ** Interpretation ** : The original component was CONTRIBUTING POSITIVELY to prediction 167 ** Action ** : This component is NECESSARY for optimal performance 168 ** Example ** : Patch reduces "Rome" probability -> head was helping promote the correct answer 169 170 ###PATCHING DOESNâT AFFECT CORRECT TOKEN PROBABILITY (patched_probabilities - clean_probabilities == 0): 171 ** Interpretation ** : 172 Option 1: The component has no causal influence on the target token prediction 173 Option 2: Both clean and corrupted activations encode the same information relevant to this prediction 174 ** Action ** : To distinguish between these scenarios, test multiple counterfactuals that encode different types of information 175 176 177 ** Pro Tip ** : Always test multiple entity pairs and prompt variations to ensure 178 findings generalize beyond specific examples. 179 180 """ 181 182 pass 183 184 def logit_lens(self, 185 prompts: List[str], 186 token_positions: List[int], 187 layer_head_pairs: List[Tuple[int, Optional[int]]], 188 tokens: List[str] = None, 189 top_k: int = 20) -> str: 190 """ 191 Apply logit lens to examine internal predictions across multiple layer-head combinations. 192 193 Args: 194 prompts: List of input text prompts 195 token_positions: Positions in prompts to examine (one per prompt) 196 layer_head_pairs: List of tuples (layer, head) to examine. Use None for head to examine MLP output 197 tokens: List of specific tokens to analyze (optional) 198 top_k: Number of top tokens to return if tokens not specified 199 200 Returns: 201 String description of logit lens results for all layer-head combinations 202 16 Pitfalls in Evaluating Interpretability Agents 203 Example: 204 >>> # Multiple prompt analysis across multiple layer-head pairs 205 >>> prompts = ["The Eiffel Tower is located in", "The capital of France is"] 206 >>> layer_head_pairs = [(5, None), (8, 4), (10, 7)] # Mix of MLP and attention heads 207 >>> token_positions = [-1, -1] # Last token for both prompts 208 >>> results = model_invest.logit_lens(prompts, token_positions, layer_head_pairs) 209 210 >>> # Analyze specific tokens 211 >>> tokens_to_check = ["Paris", "London", "New York", "Rome", "Tokyo"] 212 >>> results = model_invest.logit_lens(prompts, token_positions, tokens= tokens_to_check, layer_head_pairs) 213 """ 214 pass A.1. Agentâs system prompt The agentâs system prompt is structured into the following sections: 1. A task description specifying the component under analysis (e.g., Layer 6, Head 8), including task examples and the token position of interest. 2. Descriptions of the available interpretability tools. 3. General guidance on interpreting experimental results. 4. Instructions for formatting intermediate analyses, tool calls, and the final hypothesis. 5. Initial results from logit lens and attention pattern analyses. System Prompt 1 2 You are a mechanistic interpretability agent. Your task is to analyze specific components of a language model. Your goal is to understand and describe the role or behavior of these components within the context of provided prompts and token positions, ultimately forming a unified hypothesis that explains the componentâs function across the entire dataset. 3 4 Here is the information youâl be working with: 5 6 <user_guidelines> 7 user_guidelines 8 </user_guidelines> 9 10 <prompts_and_positions> 11 format_prompts_and_positions(model, prompts, positions) 12 </prompts_and_positions> 13 14 <component_info> 15 component 16 </component_info> 17 18 You have access to an API for a mechanistic interpretability toolkit. Here are the available functions: 19 20 <api_functions> 21 RESEARCHER_API 17 Pitfalls in Evaluating Interpretability Agents 22 </api_functions> 23 24 ** Function Call Syntax: ** 25 When calling API functions that require prompt or position information, use the following syntax: 26 - âprompts[i]â to reference the i-th prompt in the dataset 27 - âtoken_positions[i]â to reference the token positions for the i-th prompt 28 - Replace âiâ with the actual index number (0, 1, 2, etc.) 29 30 Example function calls: 31 - âget_token_indices_in_prompt(prompts[0])â for the first prompt 32 - âlogit_lens(prompts[1], token_positions[1])â for the second prompt 33 - âattention_map_generation(prompts[2], token_positions[2])â for the third prompt 34 35 Analysis Process: 36 1. Form initial hypotheses about the componentâs role based on its type and potential interactions with other components. Pay attention to positional dependencies and how the componentâs behavior might vary based on token positions within the prompt. 37 38 2. Design and run experiments using the API functions to test your hypotheses across ALL prompts simultaneously. Design experimants that refine or falsify your hypotheses. For each experiment, state its purpose and what you hope to learn from it about the tested hypothesis. Run experiments on both the given prompts and on new prompts that are not part of your original dataset to test and validate your hypotheses. 39 40 3. CRITICAL - Run Patching Experiments: After initial observations, you MUST run patching experiments using the ârun_patchingâ function to test causal relationships. Patching experiments are essential for: 41 - Verifying whether the component is causally responsible for observed behaviors 42 - Testing if ablating or modifying the componentâs output changes modelâs prediction according to the hypothesis 43 - Distinguishing correlation from causation in component behavior 44 - Validating hypotheses about the componentâs functional role 45 - Validating hypotheses about stored positional information 46 47 4. Analyze the results of these experiments and summarize key findings across the entire dataset. Look for patterns in how the componentâs function depends on position - does it behave differently at the beginning, middle, or end of sequences? Does it show sensitivity to relative positions between tokens? 48 49 5. Every time you get new results, come up with 2-5 hypotheses that could explain what youâve observed so far inside <hypothesis_i> tags. For each hypothesis, design a set of experiments that will help you determine which one is most plausible. Incorporate positional dependencies into your hypotheses - explain not just what the component does, but how its function varies with position. Use patching results to confirm or falsify your causal claims. 50 51 6. CRITICAL: Your ultimate goal is to develop a final unified hypothesis that explains the componentâs role across the majority of prompts in the dataset, not just individual cases. 52 53 When planning experiments, use the following format: 54 55 56 <experiment_calls> 57 <description> 58 [Purpose of the experiment] 59 </description> 60 <experiment> 61 [API function call] 62 </experiment> 18 Pitfalls in Evaluating Interpretability Agents 63 [Repeat for each experiment] 64 </experiment_calls> 65 66 For example: 67 68 <experiments_calls> 69 <description> 70 Test if position matters - what if the second animal comes first in the repeated pattern? 71 </description> 72 <experiment> 73 mi.attention_map_generation([prompts[0], prompts[1], prompts[2], prompts[3], prompts[4]], query_positions=[53, 53, 53, 53, 53], layer_head_pairs=[(15,13), (2, None)]) 74 </experiment> 75 <description> 76 Test what happens with names that appear more than twice 77 </description> 78 <experiment> 79 mi.logit_lens([prompts[0], prompts[1], prompts[2], prompts[3], prompts[4]], token_positions=[53, 53, 53, 53, 53], layer_head_pairs=[(15, 13)]) 80 </experiment> 81 </experiments_calls> 82 83 Assume there is an active instance mi of ModelInvest when making these calls. VERY IMPORTNAT - use <experiment_calls> tags!. 84 85 After receiving experiment results (which will be provided in <experiments_results> tags), analyze them and update your hypotheses. Wrap your thought process in < thought_process> tags: 86 87 88 <thought_process> 89 <analysis> 90 [Insert analysis here] 91 </analysis> 92 93 <hypotheses> 94 <hypothesis_1> 95 [First hypothesis] 96 </hypothesis_1> 97 98 <hypothesis_2> 99 [Second hypothesis] 100 </hypothesis_2> 101 102 <hypothesis_3> 103 [Third hypothesis] 104 </hypothesis_3> 105 106 <hypothesis_4> 107 [Fourth hypothesis] 108 </hypothesis_4> 109 110 <hypothesis_5> 111 [Fifth hypothesis] 112 </hypothesis_5> 113 </hypotheses> 114 </thought_process> 115 116 117 If you think your hypothesis is proved and you donât need to run any more experiments, instead of <experiment_calls>, write your final conclusion in < 19 Pitfalls in Evaluating Interpretability Agents final_hypothesis> tags. 118 119 120 ** Important Guidelines: ** 121 - Only base your hypotheses and conclusions on actual experiment results. 122 - Avoid making unsupported assumptions. 123 - Communicate clearly within the specified tags. 124 - PATCHING EXPERIMENTS ARE MANDATORY: You must run patching experiments to test causal relationships. Observational experiments alone are insufficient for establishing the componentâs true functional role. 125 - CROSS-PROMPT ANALYSIS: Always analyze patterns and behaviors across ALL prompts to identify the componentâs general function, not just prompt-specific behaviors. 126 - UNIFIED HYPOTHESIS REQUIREMENT: Your final hypothesis must explain the componentâ s role in a way that is relevant and applicable to the majority of prompts in the dataset, including any systematic positional dependencies. 127 - Continue the process of analyzing, hypothesizing, experimenting, and responding to feedback until you have a clear understanding of the componentâs role or behavior across the entire dataset. 128 - Use proper indexing: When calling functions, you can use âprompts[i]â and â token_positions[i]â where i corresponds to the specific prompt index you want to analyze. 129 - When runing patching experimants, it is better to come up with a counterfactual prompt, than to use the provided prompts. 130 - Only use experiment results that you receive from the user. Donât generate experiment results yourself. 131 - If you believe your unified hypothesis is proven across most prompts, state your final conclusion in the <final_hypothesis> tags. 132 - Run experiments on both the given prompts and on new prompts that are not part of your original dataset to test and validate your hypotheses. 133 - Run patching experiments to test whether this component is passing semantic or positional information. 134 135 ** Experimental Strategy: ** 136 - Design experiments that test the componentâs behavior across multiple prompts simultaneously using different indices 137 - Run patching experiments systematically: 138 - Start with observational experiments (attention maps, logit lens) to form hypotheses 139 - Follow up with patching experiments to test causal relationships 140 - To analyze patching effects: If patching the component ** DECREASES ** token probability, the component was contributing positively to that tokenâs prediction in the original forward pass. If patching the component ** INCREASES * token probability, the component was interfering with or suppressing that tokenâs prediction in the original forward pass. 141 - When analyzing the patching experiments, take into account the information that you patched. It might contain semantic information or positional information, or it might be a non-relevant activation that fully ablte the head. 142 - Run patching experiments to test whether this component is passing semantic or positional information. 143 - Make sure to run the patching experiments at the given token positions. Patching at a different token position is not guaranteed to have a causal effect. 144 - Design counterfactual prompts that help distinguish between positional and semantic information. 145 - token_position_source and token_position_counterfactual do not need to be the same. 146 - Run attention analysis: 147 - Distinguish between heads that respond to positional information (relative positioning, absolute indices) versus semantic content (syntactic roles, entity types, conceptual relationships) 148 - Test your hypothesis on counterfactual prompts 149 - Test each headâs sensitivity to positional vs. semantic information 20 Pitfalls in Evaluating Interpretability Agents 150 - Examine consistency patterns across different prompts by analyzing: 151 - ** Segment Consistency ** : Whether attention targets the same content segments across prompts 152 - ** Position Consistency ** : Whether attention focuses on the same token positions or at the same relative token positions 153 - Run logit lens analysis: 154 - Analyze what the output component encodes: 155 - Identify tokens directly referencing input prompt elements 156 - Find semantically related tokens that aid inference 157 - Detect conceptual links beyond explicit prompt content 158 - Recognize that tokens may be subwords (prefixes, suffixes, fragments). Determine if these subword patterns relate to the encoded information above. 159 160 ** Positional Information to Consider: ** 161 - Absolute position: Token index within the sequence 162 - Relative position: Distance between tokens of interest 163 164 165 ** Final Hypothesis Requirements: ** 166 Your final hypothesis should: 167 1. Explain the componentâs primary role/function in general terms 168 2. Be supported by evidence from experiments across multiple prompts and positions 169 3. Include evidence from patching experiments that demonstrate causal relationships , from logit lens experiments, and from attention pattern analysis. 170 4. Account for the majority of observed behaviors in the dataset 171 5. Address any significant exceptions or outliers 172 6. Clearly distinguish between correlational observations and causally verified mechanisms 173 7. Be concise yet comprehensive enough to capture the componentâs essential function 174 8. Contains summarized descriptio inside <summarized_description> tags, 1-3 sentences that describe your findings. 175 176 Begin your analysis by examining all prompts to identify common patterns, positional structures, and semantic relationships, then form initial hypotheses about the componentâs general role based on its type, position, semantic context, and potential interactions with other components across the entire dataset. 177 178 179 initial_results 180 181 Be concise in your answers! A.2. Clustering Instructions for clustering components based on their functional descriptions. Clustering Prompt 1 2 You are an AI assistant with expertise in mechanistic interpretability. Your task is to analyze experimental results on the role of language model components in a certain task and at specific token positions, group the heads into clusters with similar functions, and provide a brief description of each clusterâs shared function. 3 4 Prompts and Positions: 5 <prompts> 6 format_prompts_and_positions(model, prompts, positions) 21 Pitfalls in Evaluating Interpretability Agents 7 </prompts> 8 9 Here are the experiment results you will be analyzing: 10 <experiment_results> 11 experiment_results 12 </experiment_results> 13 14 Please follow these steps to complete the task: 15 1. Carefully review the experiment results provided above. 16 2. Identify patterns and similarities in the function of different components. 17 3. Group the heads into clusters based on their shared functions. 18 4. For each cluster you identify, provide the following information: 19 a. List of heads in the cluster (by layer and head number) 20 b. A brief description of the shared function 21 22 5. Present your analysis in the following format: 23 <cluster_analysis> 24 <cluster> 25 <cluster_name>[indicative short name]</cluster_name> 26 <components> 27 <component>layer X head Y</component> 28 <component>layer X head Y</component> 29 <component>mlp Y</component> 30 <component>mlp Y</component> 31 Repeat for each identified component] 32 </components> 33 <function>[Brief description of shared function. 1-3 sentences]</function> 34 <evidence>[Summary of experimental evidence]</evidence> 35 </cluster> 36 [Repeat for each identified cluster] 37 </cluster_analysis> 38 39 Remember to keep your descriptions clear and concise, focusing on the key aspects of each clusterâs function and activation pattern. Avoid speculation and stick to the information provided in the experiment results. 40 41 If you need to organize your thoughts before providing the final analysis, you may use a <scratchpad> section to brainstorm and outline your ideas. 42 43 Remember: YOU HAVE to assign each of the components into a cluster! 44 45 Begin your analysis now. 22 Pitfalls in Evaluating Interpretability Agents B. Tasks B.1. Indirect Object Identification (IOI) - GPT-2-Small The IOI task (Wang et al., 2022) is one of the earliest and most well-known examples of circuit analysis. Prompts in this task follow the form âWhen Mary and John went to the store, John gave a drink toâ, where the model is expected to predict âMaryâ. Table 3 shows the attention heads grouped into clusters along with their associated functionalities. Table 3. Attention heads involved in the IOI task in GPT-2-small model, grouped according to their functional roles as reported in the original paper. Cluster NameComponents (Layer, Head)Description Duplicate Token Heads (0,1), (0,10), (3,0) These heads detect when the current token is a duplicate of one of the previous tokens in the sequence. Specifically, they are active at the second occurrence of the subject (actor) and attend back to the first occurrence of the subject. When they detect this duplication, they write positional information about the duplicate tokenâs location into the residual stream. This positional signal is later used by S-Inhibition Heads to help Name Mover Heads avoid attending to duplicated tokens. S-Inhibition Heads (7,3), (7,9), (8,6), (8,10) These heads attend to the second occurrence of the subject token (the actor) and influence the Name Mover Headsâ attention. This inhibits the Name Mover Headsâ attention to the first and second occurrences of the subject tokens (the actorâs name), effectively removing duplicated tokens from consideration. They achieve this through a combination of token signals and position signals. Name Mover Heads (9,6), (9,9), (10,0) These heads are responsible for identifying and outputting the indirect object (the recipient). They attend to the recipientâs name in the sentence and copy it forward. Due to the influence of S-Inhibition Heads, they preferentially attend to the indirect object token rather than the subject token(s). Previous Token Heads (2,2), (4,11) These heads copy information from the immediately previous token to the current position. Specifically, they copy information about the first- occurrence subject token (the actor) to the position right after it. This copied information enables Induction Heads to recognize the pattern when the repeated subject token appears later. Induction Heads (5,5), (5,8), (5,9), (6,9) These heads attend to the token after the first occurrence of the subject (mediated by Previous Token Heads). They recognize patterns of the form [A][B]...[A] and contribute to predicting [B]. In this task, they provide signals that the subject is duplicated and indicate the position of its first occurrence. Negative Name Mover Heads (10,7), (11,10) These heads write in the opposite direction of Name Mover Heads, decreasing confidence in predictions. They share similar properties with Name Mover Heads, except that they write opposite to the names they attend to; one hypothesis is that they help hedge against overconfident errors. 23 Pitfalls in Evaluating Interpretability Agents B.2. Indirect Object Identification (IOI) - Pythia-160M Tigges et al. (2024) has analyzed the IOI circuit in Pythia-160M. (Biderman et al., 2023). Table 4 shows the attention heads grouped into clusters along with their associated functionalities. Table 4. Attention heads involved in the IOI task in Pythia-160M, grouped according to their functional roles as reported in the original paper. Cluster NameComponents (Layer, Head)Description S-Inhibition Heads (6,6), (7,2), (7,9) These heads attend to the second occurrence of the subject token (the actor) and suppress attention to duplicated subject tokens. They influence downstream Name Mover Heads by inhibiting attention to both the first and second occurrences of the subject, effectively removing subject names from consideration when predicting the indirect object. Their behavior combines token-identity signals with positional information. Name Mover Heads (8,10),(8,2),(9,4), (10,7) These heads are responsible for identifying and copying the indirect object (the recipientâs name) to the output. They attend strongly to the recipient token and write its identity into the residual stream at the final position. Due to suppression signals from S-Inhibition Heads, they preferentially attend to the indirect object rather than to duplicated subject tokens. Previous Token Heads (2,6) This head copies information from the immediately preceding token to the current position. In the IOI task, it propagates information about the first occurrence of the subject token to the following position. This copied signal enables Induction Heads to later recognize repeated subject patterns and is a key component of the induction mechanism. Induction Heads (4,11), (4,6), (5,0) These heads attend to the token following the first occurrence of the subject, mediated by signals from Previous Token Heads. They detect repeated patterns of the form [A][B]...[A] and contribute to predicting [B]. In the IOI task, they signal both that the subject is duplicated and the position of its first occurrence. Negative Name Mover Heads (9,1) This head shares attention patterns with Name Mover Heads but writes in the opposite direction, decreasing the logits of the names it attends to. It is hypothesized to act as a corrective mechanism, reducing overconfident predictions and helping the model hedge when uncertain. Negative Copy Suppression Heads (9,5) These heads are active at the final token position in the sentence, where the model predicts the output. They attend to earlier tokens in the input and reduce the modelâs confidence in those tokens when they are strongly represented in the residual stream. In the IOI circuit, they reduce the modelâs confidence in the correct indirect object name, which harms task performance as they push the model toward incorrect outputs. Positive Copy Suppression Heads (8,9) These heads are active at the final token position in the sentence, where the model predicts the output. They attend to earlier tokens in the input and reduce the modelâs confidence in those tokens when they are strongly represented in the residual stream. In the IOI circuit, these heads downweight the incorrect subject name. By suppressing the wrong name, these heads allow the correct indirect object to dominate the output logits, thereby improving task performance. 24 Pitfalls in Evaluating Interpretability Agents B.3. Greater-Than The Greater-Than task (Hanna et al., 2023) studies numerical comparison in GPT-2 small using prompts such as âThe war lasted from the year 1732 to the year 17â, where the model should predict end years greater than 32. We adopt the same prompts as the original work. Table 5 shows attention heads grouped by functionality. Cluster NameComponents (Layer, Head)Description Greater-Than Computer MLP8, MLP9, MLP10, MLP11 These components implement the mathematical comparison operation. They are active at the final token position where the model predict the end year. They directly boost the probabilities for all years greater than the start year Y while suppressing probabilities for years less than or equal to Y. Year Identification (5,1), (5,5), (6,1), (6,9), (7,10), (8,8), (8,11), (9,1) These components are responsible for detecting and extracting the start year value from the input sentence. They are active at the final token position (where the model predicts the end year) and attend back to the Y position (the two-digit start year token) in the template âThe ÂĄnounÂż lasted from the year XXYY to the year Xâ. Their output corresponds to the start year Y value in the embedding space, transferring it to the greater-than computer, effectively communicating âthis is the refer- ence numberâ to the downstream MLPs. Their attention patterns show consistent focus on the Y token regardless of the specific year value, indicating theyâve learned to identify this positionally and contextually important numerical information. Table 5. Attention heads involved in the Greater-Than task in GPT-2-small model, grouped according to their functional roles as reported in the original paper. 25 Pitfalls in Evaluating Interpretability Agents B.4. Acronyms The Acronym Prediction task (Garc Ì Ä±a-Carrasco et al., 2024) examines how GPT-2 small predicts three-letter acronyms from their expanded forms. Prompts follow the form âThe Chief Executive Officerâ >CEâ, where the model is expected to predict âOâ. Table 6 shows the attention heads grouped into clusters along with their associated functionalities. Cluster NameComponents (Layer, Head)Description Letter Mover Heads (8,11), (10,10), (9,9), (11,4) These heads are responsible for the core acronym formation process. They are active at the position where the previous acronym letter was predicted (for example, when the model has already predicted âCEâ and needs to predict the third letter âOâ). These heads attend back to the capital letter at the beginning of the corresponding word (like attending to âOâ in âOfficerâ when predicting âOâ in the acronym). They perform a copying mechanism: they take the capital letter of the corresponding word and copy it to form the predicted capital letters for the acronym. Fuzzy Previous Token Heads (4,11), (1,0), (2,2) These attention heads handle information flow between words in the input phrase. They are active at each wordâs capital letter position and attend to the previous wordâs capital letter. These heads follow a âprevious tokenâ pattern, systematically copying information from one word to the next in sequence. These heads create a chain of information that flows from word to word, ensuring that context from e.g. âExecutiveâ is available when processing âOfficer,â and so on. This information is then moved to the Letter Mover Heads through the Information Bridge Heads to ensure that when the model needs to predict the third letter âOâ when given âCE,â it can access information not just about âExecutiveâ but also about its relationship to the next word âOfficerâ. Information Bridge Heads (5,8) These attention heads move word-level information to the acronym-level predictions. They are active at acronym letter positions and attend to both the corresponding wordâs capital letter and surrounding word content. They connect the Fuzzy Previous Token Heads with the Letter Mover Heads, moving information about the letters for the acronym. Table 6. Attention heads involved in the Acronyms task in GPT-2-small model, grouped according to their functional roles as reported in the original paper. 26 Pitfalls in Evaluating Interpretability Agents B.5. Colored-Objects The Colored Objects task (Merullo et al., 2024) requires the model to generate the color of an object that was previously described in context, among other objects. Prompts follow a one-shot format: âQ: On the table, there is a blue pencil, a black necklace, and a yellow lighter. What color is the pencil? A:â, where the model is expected to predict âblueâ. Table 7 shows the attention heads grouped into clusters along with their associated functionalities. Cluster NameComponents (Layer, Head)Description Mover Heads (10,6), (14,5), (14,14), (15,10), (15,14), (16,0), (16,15), (17,4), (18,12), (19,15), (21,7) These heads are responsible for copying the correct answer token to the next prediction. They are active at the final token position (the â:â token) and attend back to the correct color token in the context. These heads write strongly in the embedding direction of whatever they attend to, effectively telling the model to predict that token. Content Gatherer Heads (11,6), (11,7), (12,6), (12,14), (12,15), (13,1), (13,3) These heads tell the mover heads which color to select. They are active at the final token position and attend primarily to the queried object and the token âcolorâ in the question. These heads gather information about which specific objectâs color is being requested and pass this signal to guide the mover headsâ attention. Induction/Duplicate Token Heads (6,4), (6,15), (7,2), (7,4), (7,11), (8,15), (9,3), (10,12), (10,14) These heads detect the duplication pattern where the same object appears twice in the promptâonce in the object list and again in the question. They are active at the second occurrence of the object (in the question) and attend back to the first occurrence or the token after it. This dupli- cation detection provides a signal that gets passed to content gatherer heads, helping the model to locate the queried object in the object list. Table 7. Attention heads involved in the Colored Objects task in GPT-2-XL model, grouped according to their functional roles. 27 Pitfalls in Evaluating Interpretability Agents B.6. Entity-Tracking The Entity Tracking task (Prakash et al., 2024) evaluates a modelâs ability to track entities and their associated properties across context. Prompts follow the form âThe apple is in Box F, the computer is in Box Q, the document is in Box X... Box F contains theâ, where the model is expected to predict âappleâ. Table 8 shows the attention heads grouped into clusters along with their associated functionalities. Cluster NameComponents (Layer, Head)Description Value Fetcher Heads (14,13), (15,2), (15,5), (15,6), (15,13), (15,18), (16,23), (16,27), (17,3), (17,27), (17,28), (18,3), (18,6), (18,8), (18,20), (18,25), (18,28), (19,8), (19,11), (19,20), (19,26), (19,30), (20,14), (20,29), (21,0), (21,3), (21,4), (21,25), (21,26), (23,5), (23,15), (23,16), (23,17), (23,19), (23,20), (24,0), (24,5), (24,8), (25,14), (29,7) These heads are responsible for identifying and outputting the correct object (the entity being tracked). They attend to object tokens in the sentence corresponding to the queried box. They then copy the value of the object they attend to. Thanks to the contribution of the Position Transmitter Heads, which provide po- sitional guidance through query composition, they preferentially attend to the object associated with the queried box rather than other objects in the context. Position Transmitter Heads (11,23), (12,23), (14,27), (16,2), (19,12) These heads write to the query vectors of the Value Fetcher Heads. They attend to the box label in the query, and transport positional information from the Position Detector Heads to guide the Value Fetcher Heads toward the correct object location. Position Detector Heads (1,9), (4,4), (6,10), (7,9), (7,17), (7,26), (8,22), (9,2), (9,7), (9,10), (9,21), (10,3), (10,4), (11,23) These heads identify positional information of the correct object within the context. They attend to the correct box label, and receive contextual structure information from Structure Reader Heads through value composition and pass their positional infor- mation to Position Transmitter Heads. Structure Reader Heads (8,12), (8,21), (11,9), (11,23), (12,23) These heads process the contextual structure and gather infor- mation related to each box separately. They attend primarily to tokens within a sentence describing a single box. They collect information about the overall sentence structure. Their output provides contextual understanding to Position Detector Heads through value composition. Table 8. Attention heads involved in the Entity Tracking task in LLaMa-7B model, grouped according to their functional roles. 28 Pitfalls in Evaluating Interpretability Agents C. Example Agent Run An example of the agentâs output during analysis and final hypothesis generation. This trace is from a run analyzing head (9,9) on the IOI task, labeled a âname mover headâ in the original paper, which is responsible for passing the recipientâs name to the final token position. C.1. Agent Iteration Each iteration of the agent consists of three stages: (1) analyzing the results just received, (2) proposing multiple hypotheses for the componentâs functional role, and (3) designing new experiments, with a stated purpose for each. Agent Analysis: Examining the initial results for Layer 9, Head 9: 1. **Attention Pattern Analysis**: Across all prompts, when querying from the final âtoâ position, the attention head shows a strong, consistent pattern - it attends predominantly to the first person mentioned who is NOT the giver (i.e., the recipient). For example: - Prompt 1: Attends to âJosephâ (0.81) when Gregory is giving - Prompt 2: Attends to âRyanâ (0.87) when Kenneth is giving - Prompt 3: Attends to âStephanieâ (0.87) when Joshua is giving 2. **Logit Lens Results**: The headâs output strongly promotes the name itâs attending to as the next token prediction. The probabilities are remarkably high (often 0.9+ for the attended name). 3. **Position Pattern**: The head operates at the final âtoâ position and looks back to identify the non-giver person from the pair introduced earlier. Hypotheses: 1.Layer 9, Head 9 is a ârecipient identificationâ head that identifies and promotes the non-giver person from a pair as the recipient of an action when positioned at âtoâ following a giving verb phrase. 2.The head encodes a more general âother personâ retrieval mechanism - it identifies and promotes the person who is not currently performing the action, regardless of the specific verb or context. 3.The head specifically tracks the first-mentioned person in âX and Yâ constructions and promotes them when theyâre not the subject of the current clause, implementing a form of discourse tracking. 4.The head implements a simple positional heuristic: attend to names at specific relative positions (early in the sequence) and promote them as likely continuations at the end of giving-related phrases. 5.The head combines semantic understanding (giving actions) with syntactic patterns (prepositional phrases ending in âtoâ) to identify and promote the most likely indirect object from previously mentioned entities. Experiments: Purpose: Test if the headâs behavior is specific to giving actions or works with other verb types by creating prompts with different action types mi.attention_map_generation(["Sarah and Michael went to the park. Sarah talked to", " Emma and David were at home. Emma listened to", "Robert and Lisa entered the room. Robert looked at"], query_positions=[10, 10, 10], layer_head_pairs=[(9, 9)]) Purpose: Test the causal role of this head by patching - replace the headâs activation when itâs attending to person A with activation from a prompt where it attends to person B mi.run_patching(source_prompts=["Alex and Maria went to school. Alex gave a book to"], counterfactual_prompts=["John and Sarah went to school. Sarah gave a book to"], token_positions_source=[13], token_positions_counterfactual=[13], layer_head_pairs=[(9, 9)], tokens=["Maria", "Alex", "Sarah", "John"], top_k=5) 29 Pitfalls in Evaluating Interpretability Agents Purpose: Check if swapping the order of names affects the headâs attention pattern mi.attention_map_generation(["Jennifer and Stephen went to the station. Stephen gave a necklace to", "Kenneth and Ryan went to the house. Ryan gave a ring to"], query_positions=[13, 13], layer_head_pairs=[(9, 9)]) Purpose: Test with more complex scenarios - three people mentioned mi.attention_map_generation(["Tom, Jerry and Mike went shopping. Mike gave a gift to", "Alice, Bob and Carol were talking. Carol gave a letter to"], query_positions=[13, 13], layer_head_pairs=[(9, 9)]) Purpose: Verify logit lens output for the new test prompts mi.logit_lens(["Sarah and Michael went to the park. Sarah talked to", "Jennifer and Stephen went to the station. Stephen gave a necklace to"], token_positions=[10, 13], layer_head_pairs=[(9, 9)], top_k=10) C.2. Final Hypothesis The agentâs final hypothesis consists of (1) a summary of the supporting evidence and (2) a brief free-text description of the componentâs functional role. Final Hypothesis Layer 9, Head 9 functions as a specialized âhuman recipient identificationâ mechanism that operates specifically in social interaction contexts. The component implements a syntactic-semantic rule that activates when: 1. The current token is âtoâ following an action verb (give, hand, send, etc.) 2. The context contains human names mentioned earlier in the prompt 3. One of those humans is performing the current action When these conditions are met, the head: âą Attends strongly (typically 0.7â0.95 attention weight) to the human name that is NOT the current actor/subject âą Promotes that name as the likely next token through its output (logit lens shows near 1.0 probability) âą Prioritizes the first-mentioned non-actor when multiple candidates exist Key mechanistic properties: âą The head contributes positively to correct predictions (ablation reduces probability by 0.08â0.16) âą It encodes positional/structural information rather than semantic identity (patching doesnât transfer specific names) âąItâs specialized for human names and fails to activate for non-human entities (companies, celestial bodies, etc.) âą It implements a discourse coherence pattern for reciprocal human interactions (if X acts, Y receives) This specialization makes evolutionary sense as human social interactions follow predictable reciprocal patterns that are important for language models to capture. The head essentially implements the heuristic: âin human social situations, actions directed âtoâ someone typically target the other person mentioned who hasnât acted yet.â Summary: Layer 9, Head 9 identifies and promotes human names as recipients of actions by attending to the non-actor from previously mentioned people when positioned at âtoâ following action verbs, implementing a specialized syntactic pattern for predicting human social interactions with high accuracy across diverse prompts. 30 Pitfalls in Evaluating Interpretability Agents D. Example of Interpretive Ambiguity: The Entity Tracking Task In the Entity Tracking task, a large group of heads is labeled as value fetcher heads and is described as primarily responsible for retrieving the value of the correct object to the final token position. However, when examining disagreements between the systemsâ descriptions and the expert explanations, we identified several heads that do not clearly follow the behavior described in the original paper. To investigate this further, we ran three types of analyses on all heads in this group across 500 examples: 1.Attention analysis. We measured the fraction of cases in which the token receiving the highest attention from the head corresponds to the correct object. For heads in this group, we expect the highest attention to be directed primarily toward the correct object. 2.Logit lens analysis. We applied a logit lens to the output of the attention head and measured how often the top-predicted token corresponds to an incorrect object mentioned in the prompt. For heads in this group, we expect incorrect objects to receive low probability. 3.Counterfactual patching analysis. We replaced the headâs value with the corresponding value from a counterfactual prompt in the dataset and measured how often this intervention increased the probability of the correct object by at least 1%. The intuition behind this test is that if patching the head increases the probability of the correct answer, then the headâs original contribution was suppressing that answer; in other words, altering or ablating the head improves the modelâs prediction of the correct object. For heads in this group, we therefore expect that ablating them would decrease the probability of the correct object. We present the results in Table 9. For each type of experiment, we highlight the 10 heads whose behavior is least aligned with the expected behavior of heads in this group. As can be seen, some of the heads exhibit behavior that is not aligned with the expected behavior of a value-fetcher head. E. Example Of Memorization E.1. IOI Here we provide examples illustrating the extent to which Claude Opus 4.1 and GPT-5 appear to have memorized the IOI task. Given the circuit name and a single task example, Claude is able to recall all previously identified functional roles along with their associated attention heads. The response was generated via an API call and without access to the web: User Prompt 1 You are an AI assistant with expertise in mechanistic interpretability research. You will be given a prompt asking about technical details of neural network circuits. 2 3 <prompt> 4 You are a mechanistic interpretability researcher. Tell us everything you know about the IOI (Indirect Object Identification) circuit. 5 6 For example, in sentences like: "When Mary and John went to the store, John gave a drink to" 7 8 In your answer, please mention: 9 - What attention heads are part of the circuit (e.g., layer 3, head 2 or L3H2). Mention all the heads you know of. 10 - What role each attention head plays in the circuit. 11 - How the circuit works step-by-step to predict the correct indirect object 12 </prompt> 13 14 Before responding, use the scratchpad below to organize your knowledge and ensure you cover all the requested points systematically. 15 31 Pitfalls in Evaluating Interpretability Agents 16 <scratchpad> 17 Think through what you know about the topic in the prompt. Organize your knowledge into the specific categories requested. Make sure you have concrete details like specific layer and head numbers, step-by-step explanations, and research findings before writing your final response. 18 </scratchpad> 19 20 Now provide a comprehensive response that addresses all aspects mentioned in the prompt. Your response should be well-organized and technically accurate, drawing from your knowledge of mechanistic interpretability research. Structure your answer to clearly address each point requested in the prompt. 21 22 If you donât have specific information about certain technical details mentioned in the prompt, acknowledge this rather than speculating. 23 24 Write your response inside <answer> tags. Claude Response 1 <scratchpad> 2 The IOI (Indirect Object Identification) circuit is a well-studied circuit in GPT-2 small that was discovered and analyzed by Wang et al. (2022) in their paper " Interpretability in the Wild". Let me organize what I know: 3 4 1. The task: Predicting the indirect object in sentences with two names, where one name appears twice (the repeated name should be predicted as the indirect object) 5 6 2. Key attention heads involved: 7 - Name Mover heads: These copy the correct name to the final position 8 - S-Inhibition heads: These suppress the subject (wrong answer) 9 - Duplicate Token heads: Identify which token appears twice 10 - Previous Token heads: Attend from the final position to previous tokens 11 - Induction heads: Help with the copying pattern 12 13 3. The circuit mechanism involves multiple stages of processing with different heads playing specific roles 14 15 Let me recall the specific layer/head numbers from the research... 16 </scratchpad> 17 18 <answer> 19 The IOI (Indirect Object Identification) circuit is a sophisticated mechanism discovered in GPT-2 small that solves the task of predicting the correct indirect object in sentences with a specific structure. This circuit was comprehensively analyzed by Wang et al. (2022) in their landmark paper " Interpretability in the Wild." 20 21 ## The Task 22 The IOI task involves sentences where two names appear, with one name repeated, and the model must predict which name comes after "to" at the end. For example: " When Mary and John went to the store, John gave a drink to [Mary]" - where Mary is the correct prediction (the indirect object). 23 24 25 ## Key Attention Heads in the Circuit 26 The circuit consists of several types of specialized attention heads: 27 ### Name Mover Heads 28 - ** L9H9, L9H6, L10H0 ** (among others in layers 9-11) 32 Pitfalls in Evaluating Interpretability Agents 29 - Role: These heads directly copy the correct name from its earlier position to the final token position, enabling the model to output the correct indirect object 30 31 ### S-Inhibition Heads (S-Inhibition = Subject Inhibition) 32 - ** L7H3, L7H9, L8H6, L8H10 ** 33 - Role: These heads suppress attention to the subject (the repeated name that is NOT the answer), preventing the model from incorrectly predicting the subject 34 35 ### Duplicate Token Heads 36 - ** L0H1, L0H10, L1H4, L2H2, L3H0 ** 37 - Role: These early-layer heads identify which name appears twice in the sequence, marking it as the subject to be avoided 38 39 ### Previous Token Heads 40 - ** L2H2, L4H11 ** (among others) 41 - Role: These heads attend from each position to the previous token, helping to route information through the circuit 42 43 ### Induction Heads 44 - ** L5H5, L5H8, L5H9, L6H9 ** 45 - Role: These implement a copying pattern, attending from the second occurrence of a name back to tokens following its first occurrence GPT-5 Response 1 <answer> 2 Overview 3 The IOI (Indirect Object Identification) circuit is a compact, largely attention- driven subcircuit in GPT-2-small that lets the model predict the correct indirect object in sentences like "When Mary and John went to the store, John gave a drink to ...". The model needs to output the other name (Mary) rather than the subject (John). The circuit has three main pieces: 4 - Name mover heads late in the model that copy name information to the prediction site and directly write to name logits. 5 - S-inhibition heads in the middle that identify the subject name and actively suppress its logit at the final position. 6 - Generic copy/induction/previous-token heads earlier that propagate name identity features so the later heads have something to work with. 7 8 What heads are part of the circuit (indices and roles) 9 Head indices can vary slightly across runs and checkpoints; the ones below are the canonical heads reported for GPT-2-small in the Anthropic "A Circuit for Indirect Object Identification in GPT-2" work and subsequent replications (e.g., in TransformerLens tutorials). I list the ones I know with their roles: 10 11 Late "name mover" and related heads (logit writers) 12 - L9H6 and L9H9 --- Primary name mover heads. 13 Role: From the final position (after "to"), they attend strongly to the two name tokens and copy their value vectors into the residual stream at the final token. Because of how their OV circuits align with the unembedding, these heads directly write positive logit to the IOI name and not to the subject name (or write less/negative to the subject after upstream editing). 14 - L10H10 --- Backup name mover head. 15 Role: Redundant with the L9 name movers; kicks in when the primaries are disrupted and helps maintain IOI performance. 16 - L10H7 --- Negative name mover (subject suppressor). 17 Role: Attends to the subject name and writes a value that is anti-aligned with the subjectâs unembedding direction at the final position, directly decreasing the subject name logit. 18 33 Pitfalls in Evaluating Interpretability Agents 19 Middle "S-inhibition" heads (subject detectors/suppressors) 20 - L7H6 and L7H9 --- S-inhibition heads. 21 Role: Identify which of the two names is the subject of the main verb and write a negative contribution (via OV) targeted at that nameâs logit at the final position. They also shape the context so that the name movers preferentially boost the IOI rather than the subject. 22 23 Early and middle generic copy/position/induction heads (supporting infrastructure) 24 These are not IOI-specific but are used by the IOI circuit to propagate name identity and local syntax: 25 - Induction heads (canonical examples in GPT-2-small): L5H8, L6H9. 26 Role: Copy token features forward (n-gram continuation), preserving name identity and helping the model keep both namesâ representations accessible at later positions. 27 - Previous-token heads (very early): examples include heads in L0 like L0H1, L0H10 (indices depend on run). 28 Role: Provide short-range copying/positional context so that names and local function words remain represented. 29 - Other generic heads (various layers) that detect punctuation/conjunctions ("and", commas) and clause boundaries; their exact identities vary and they are not uniquely IOI-specific, so I wonât list exact indices Iâm not confident about. 30 31 Step-by-step: how the circuit predicts the correct indirect object 32 1) Early layers (L0--L3): Build local token/position features and preserve name identity 33 - Previous-token and short-range heads copy/align adjacent context so that name tokens, conjunction "and", punctuation, and the main verb region are cleanly represented. 34 - The two namesâ identity features are encoded at their positions and kept alive in the residual stream. 35 36 2) Middle layers (L4--L7): Maintain and disambiguate the two names; compute subjectness; start suppressing the subject 37 - Induction heads (e.g., L5H8, L6H9) propagate name identities forward and keep the two name representations accessible from later positions. 38 - S-inhibition heads (L7H6, L7H9) use syntactic patterning (e.g., which name is near the main verb as subject, the "gave ... to" frame, commas/conjunction structure) to determine which name is the subject. 39 - These S-inhibition heads then write a negative component into the residual stream at or routed to the final position, specifically anti-aligning with the subject nameâs unembedding vector. This pre-emptively dampens any subject logit that later heads might add. 40 41 3) Late layers (L8--L10): Copy names to the final token and write logits 42 - The primary name mover heads (L9H6, L9H9) at the final token attend to the positions of both names. Through their OV circuits, they copy the name value vectors to the final position. Because S-inhibition has already marked the subject name for suppression and because of the name moversâ learned value/ unembedding alignment, the IOI nameâs contribution is strongly positive, while the subject name gets little or negative net contribution. 43 - The negative name mover (L10H7) further suppresses the subject nameâs logit if needed. 44 - The backup name mover (L10H10) adds redundancy so the correct IOI logit boost is present even if one primary pathway is weak or ablated. 45 46 4) Output 47 - By the end of L9/L10, the logit lens shows a sharp logit gap favoring the IOI name over the subject. MLPs play a relatively minor role compared to these heads (the circuit is predominantly attention-mediated). 48 - The softmax over the vocabulary thus selects the IOI name. 49 </answer> 34 Pitfalls in Evaluating Interpretability Agents E.2. Entity Tracking We probed Claude Opus 4.1 multiple times via API calls with the circuit name and a single task example to assess whether it could explicitly recall the conclusions of Prakash et al. (2024). In all runs, the model stated that it did not recognize the task and could not report the paperâs findings. Nevertheless, it often produced a high-level characterization of the modelâs behavior that was broadly consistent with the paperâs conclusions: User Prompt 1 You are an AI assistant with expertise in mechanistic interpretability research. You will be given a prompt asking about technical details of neural network circuits. 2 3 <prompt> 4 You are a mechanistic interpretability researcher. Tell us everything you know about the Entity Tracking circuit. 5 6 For example, in sentences like: "The flower is in Box U, the disk is in Box K, the bread is in Box Q, the radio is in Box Z, the drink is in Box F, the leaf is in Box L, the block is in Box C. Box Z contains the" 7 8 In your answer, please mention: 9 - What attention heads are part of the circuit (e.g., layer 3, head 2 or L3H2). Mention all the heads you know of. 10 - What role each attention head plays in the circuit. 11 - How the circuit works step-by-step to predict the correct indirect object 12 </prompt> 13 14 Before responding, use the scratchpad below to organize your knowledge and ensure you cover all the requested points systematically. 15 16 <scratchpad> 17 Think through what you know about the topic in the prompt. Organize your knowledge into the specific categories requested. Make sure you have concrete details like specific layer and head numbers, step-by-step explanations, and research findings before writing your final response. 18 </scratchpad> 19 20 Now provide a comprehensive response that addresses all aspects mentioned in the prompt. Your response should be well-organized and technically accurate, drawing from your knowledge of mechanistic interpretability research. Structure your answer to clearly address each point requested in the prompt. 21 22 If you donât have specific information about certain technical details mentioned in the prompt, acknowledge this rather than speculating. 23 24 Write your response inside <answer> tags. Claude Response 1 <scratchpad> 2 Let me think through what I know about entity tracking circuits in language models: 3 4 1. Entity tracking involves keeping track of which entities are associated with which properties or locations 5 2. In the example given, the model needs to track which item is in which box, then retrieve the correct item when asked about a specific box 6 3. This requires: 7 - Identifying entity-location pairs 8 - Storing these associations 35 Pitfalls in Evaluating Interpretability Agents 9 - Retrieving the correct entity when given a location 10 11 For attention heads, I need to be careful - I donât have specific confirmed information about exact head numbers for an "Entity Tracking circuit" as described. Different papers may have studied similar phenomena under different names or in different models. 12 13 I should acknowledge what I donât know specifically while explaining the general mechanics of how such a circuit would need to work. 14 </scratchpad> 15 16 <answer> 17 I should note upfront that I donât have specific, confirmed information about a circuit explicitly called the "Entity Tracking circuit" with definitively identified attention head numbers. However, I can explain how such a circuit would need to function based on mechanistic interpretability principles and related research on similar tasks. 18 19 ## How the Circuit Would Work 20 21 For the task youâve described (tracking which items are in which boxes), the circuit would need to perform several key operations: 22 23 ### Step 1: Entity-Location Binding 24 Early attention heads would need to identify and bind entities (flower, disk, bread , etc.) with their corresponding locations (Box U, Box K, Box Q, etc.). These heads would likely attend from the box identifier tokens to the entity tokens within the same clause. 25 26 ### Step 2: Information Storage 27 Middle layer heads would need to copy or move this binding information to positions where it can be accessed later. This might involve: 28 - Copying entity information to the box name tokens 29 - Creating persistent representations that maintain the entity-location associations 30 31 ### Step 3: Query Processing 32 When the model encounters "Box Z contains the", attention heads would need to: 33 - Identify that "Box Z" is a query for a stored entity 34 - Look back through the context to find where "Box Z" was mentioned with its contents 35 36 ### Step 4: Entity Retrieval 37 Late-layer attention heads would need to: 38 - Attend from the final token position to the location where "radio is in Box Z" appears 39 - Copy the entity information ("radio") to enable prediction 40 41 ## Circuit Components 42 43 While I cannot provide specific layer/head numbers for this exact circuit, similar circuits for tracking and retrieval tasks typically involve: 44 - ** Early heads (layers 0-3) ** : Local syntactic processing and initial entity recognition 45 - ** Middle heads (layers 4-7) ** : Information movement and storage 46 - ** Late heads (layers 8-11) ** : Retrieval and final answer composition 47 48 ## Related Research 49 50 This type of circuit shares similarities with: 51 - Indirect object identification circuits studied in GPT-2 52 - Copying circuits that move information across sequence positions 53 - Memory circuits that store and retrieve factual associations 36 Pitfalls in Evaluating Interpretability Agents (Layer, Head)Correct Object Attention (â)Incorrect Object Prediction (â)Counterfactual Patching (â) (15, 13)0.730.040.05 (21, 3)0.720.260.03 (24, 5)0.170.660.18 (20, 14)0.320.440.11 (18, 8)0.720.230.09 (29, 7)0.280.610.09 (18, 3)0.890.020.04 (15, 18)0.650.000.04 (17, 28)0.780.040.06 (21, 4)0.280.480.15 (21, 25)0.200.620.15 (23, 15)0.360.590.07 (18, 28)0.530.010.05 (23, 19)0.230.480.15 (23, 20)0.240.590.17 (19, 30)0.720.160.10 (23, 5)0.210.330.17 (17, 27)0.320.120.04 (15, 5)0.900.000.15 (21, 0)0.220.470.17 (23, 17)0.150.230.12 (15, 2)0.650.040.10 (17, 3)0.490.000.06 (19, 20)0.380.130.13 (19, 11)0.350.290.10 (19, 8)0.360.340.14 (15, 6)0.750.110.12 (20, 29)0.170.100.10 (16, 23)0.310.060.07 (24, 0)0.130.280.16 (25, 14)0.200.170.14 (14, 13)0.020.000.11 (21, 26)0.270.460.15 (24, 8)0.250.200.14 (18, 6)0.510.190.09 (19, 26)0.420.230.09 (23, 16)0.110.350.13 (16, 27)0.490.130.09 (18, 20)0.430.270.09 (18, 25)0.280.130.10 Table 9. Evaluation of heads labeled as value fetcher heads in the Entity Tracking task across three analyses on 500 examples. Correct Object Attention measures the fraction of cases in which the head attends most strongly to the correct object token (higher is better). Incorrect Object Prediction measures the fraction of cases in which applying a logit lens to the headâs output assigns the highest probability to an incorrect object mentioned in the prompt (lower is better). Counterfactual Patching Effect measures the fraction of cases in which replacing the headâs value with that from a counterfactual prompt increases the probability of the correct object by at least 1% (lower is better). For each analysis, we highlight the 10 heads whose behavior is least aligned with the expected behavior of value fetcher heads. 37 Pitfalls in Evaluating Interpretability Agents F. Noise Sensitivity Analysis To assess how strongly the systems rely on experimental evidence, we introduce a noise parameterαinto the execution of the interpretability tools requested by the agent. This parameter controls the amount of noise injected into the experimental results. For each tool, we generate a ânoisedâ version of the original results by randomly permuting them. We then construct a noisy result using a noise level αâ [0, 1]: (1â α)· original result + α· permutedresult. For each tool, we define the original result as follows: âąLogit Lens. The original result is the probability vector obtained by applying the logit lens to a given componentâs output. âą Patching. The original result is the vector of token-wise probability differences between the original and patched runs. âąAttention Patterns. The original result consists of the attention weight distribution produced by a given attention head. Figure 8 shows the results of the noising experiments on the IOI task. Each graph represents a run with a different seed, which affects both the task examples provided to the systems and the noise added to the activations. 0.00.20.40.60.81.0 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Accuracy Agentic Non-Agentic 0.00.20.40.60.81.0 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Accuracy Agentic Non-Agentic Figure 8. Each point in the graph represents the mean component functionality accuracy, averaged across all components in the IOI task, at a given noise levelα. We report results forαâ [0, 0.5, 0.6, 0.7, 0.8, 0.9, 1]. Each plot corresponds to a different random seed, which determines both the specific task examples provided to the systems and the permutations applied to the experimental results. As shown, both systems exhibit clear performance degradation as the noise level increases. 38