Paper deep dive
Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models
Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, Mor Geva
Models: LLaMA-2-13B, LLaMA-2-70B, LLaMA-2-7B, Vicuna-13B, Vicuna-7B
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/12/2026, 8:14:48 PM
Summary
Patchscopes is a unifying framework for inspecting hidden representations of large language models (LLMs) by patching internal representations into different inference passes. It generalizes prior interpretability methods like logit lens and causal tracing, while offering superior expressivity, robustness across layers, and training-data-free operation. The framework enables novel capabilities such as verbalizing entity resolution processes, cross-model inspection, and multi-hop reasoning error correction.
Entities (5)
Relation Signals (3)
Patchscopes → generalizes → Logit Lens
confidence 95% · We show that many prior interpretability methods based on projecting representations into the vocabulary space... can be viewed as instances of this framework.
Patchscopes → generalizes → Causal Tracing
confidence 95% · More broadly, Patchscopes also cover recent mechanistic interpretability methods that analyze internal processes in LLMs with inference computation interventions. Specifically, causal tracing (Meng et al., 2022a)...
Patchscopes → enables → Multi-hop reasoning error correction
confidence 90% · Lastly, we showcase the utility of Patchscopes for fixing latent multi-hop reasoning errors
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Understanding the internal representations of large language models (LLMs) can help explain models' behavior and verify their alignment with human values. Given the capabilities of LLMs in generating human-understandable text, we propose leveraging the model itself to explain its internal representations in natural language. We introduce a framework called Patchscopes and show how it can be used to answer a wide range of questions about an LLM's computation. We show that many prior interpretability methods based on projecting representations into the vocabulary space and intervening on the LLM computation can be viewed as instances of this framework. Moreover, several of their shortcomings such as failure in inspecting early layers or lack of expressivity can be mitigated by Patchscopes. Beyond unifying prior inspection techniques, Patchscopes also opens up new possibilities such as using a more capable model to explain the representations of a smaller model, and multihop reasoning error correction.
Tags
Links
- Source: https://arxiv.org/abs/2401.06102
- Canonical: https://arxiv.org/abs/2401.06102
Trouble viewing inline? Open PDF directly →
Full Text
109,195 characters extracted from source content.
Expand or collapse full text
Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models Asma Ghandeharioun * 1 Avi Caciularu * 1 Adam Pearce 1 Lucas Dixon 1 Mor Geva 1 2 Abstract Understanding the internal representations of large language models (LLMs) can help explain models’ behavior and verify their alignment with human values. Given the capabilities of LLMs in generating human-understandable text, we pro- pose leveraging the model itself to explain its in- ternal representations in natural language. We introduce a framework calledPatchscopes and show how it can be used to answer a wide range of questions about an LLM’s computation. We show that many prior interpretability meth- ods based on projecting representations into the vocabulary space and intervening on the LLM computation can be viewed as instances of this framework. Moreover, several of their short- comings such as failure in inspecting early lay- ers or lack of expressivity can be mitigated by Patchscopes. Beyond unifying prior inspec- tion techniques,Patchscopesalso opens up newpossibilities such as using a more capable model to explain the representations of a smaller model, and multihop reasoning error correction 1 . 1. Introduction The question of what information is captured in the hidden representations of large language models (LLMs) is of key importance in control and understanding of modern genera- tive AI, and has drawn substantial attention recently (Casper et al., 2022; Madsen et al., 2022; Patel & Pavlick, 2021; Nanda et al., 2023). To tackle this question, prior work has introduced a diverse array of interpretability methods, which largely rely on three prominent approaches: training linear * Equal contribution 1 Google Research 2 Tel Aviv Univer- sity. Correspondence to: Asma Ghandeharioun<aghandehari- oun@google.com>. Proceedings of the41 st International Conference on Machine Learning, Vienna, Austria. PMLR 235, 2024. Copyright 2024 by the author(s). 1 Code is publicly available athttps://pair-code. github.io/interpretability/patchscopes. Step 4: Running Execution on Patched Target Amazon`sformerCEO attended Oscars f( ) = cat->cat; 135->135; hello->hello; ? Step 1: Feeding Source Prompt to Source Model Step 2: Transforming Hidden State Step 3: Feeding Target Prompt to Target Model Jeff Bezos Figure 1.Illustration of our framework, showing aPatchscope for decoding the hidden representation of“CEO”in the source prompt (left). We use a target prompt (right) comprised of few- shot demonstrations of string repetitions to encourage the LLM to explain its internal representation.Step 1: Run the forward computation on the source prompt in the source model.Step 2: Optionally transform the hidden state at the source layer.Step 3: Run the forward computation on the target prompt up to the target layer in the target model.Step 4: Patch the target representation of the token“?”at the target layer with the transformed repre- sentation (from step 2), then continue the forward computation from that layer onward. The modularity ofPatchscopesallows designing a variety of methods by configuring the target prompt, model and transformation. classifiers, called probes, on top of hidden representations (Belinkov & Glass, 2019; Belinkov, 2022; Alain & Bengio, 2017), projecting representations to the model’s vocabulary space (nostalgebraist, 2020; Din et al., 2023; Belrose et al., 2023), and intervening on the computation to identify if a representation is critical for certain predictions (Meng et al., 2022a; Wallat et al., 2020; Wang et al., 2022; Conmy et al., 2023; Geva et al., 2023). Despite the wide success of these methods, they each exhibit practical shortcomings. First, probing relies on supervised training for pre-defined classes, which is hard to scale when there is a large number of classes or when all the categories are not known a priori. Second, the accuracy of vocabulary 1 arXiv:2401.06102v4 [cs.CL] 6 Jun 2024 Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models Table 1.Many prior inspection methods with various objectives can be viewed asPatchscopes. The rows highlighted in green showPatchscopeconfigurations that overcome several limita- tions of prior methods through more expressive inspection that is training-data free and is more robust across layers. Inspection Objective Expressive Training Data Free Robust Across Layers Inspecting Output Distribution Few-Shot Token IdentityPatchscope(§4.1) Ë Logit Lens (nostalgebraist, 2020), Embedding Space Analysis (Dar et al., 2023) Ëé Tuned Lens (Belrose et al., 2023) Ë For learning mappings Ë Future Lens (Pal et al., 2023) Ë For learning mappings Ë Feature Extraction Zero-Shot Feat. Ext.Patchscope(§4.2) Ë LRE Attribute Lens (Hernandez et al., 2023b) Ë For linear relation approx. Ë Probing (e.g., Belinkov & Glass, 2019; Belinkov, 2022; Alain & Bengio, 2017; Wang et al., 2023) é For training probe Ë Entity Resolution Entity DescriptionPatchscope(§4.3) Ë X-Model Entity Desc.Patchscope(§4.4) Ë For learning mappings Ë Causal Tracing (Meng et al., 2022a) éË Attention Knockout (Wang et al., 2022; Conmy et al., 2023; Geva et al., 2023) éË Inspection Application Early Exiting, e.g., Linear Shortcuts (Din et al., 2023) Ë For learning mappings Ë Caption Generation, e.g., Linear Mapping (Merullo et al., 2022) Ë For learning mappings Ë projections substantially decreases in early layers and the outputs are often hard to interpret. Last, all the above meth- ods are not as expressive as one might like: they provide class probabilities or most likely tokens, as opposed to a high-quality explanation in natural language. In this work, we argue that the advanced capabilities of LLMs in generating human-like text can be leveraged for “translating” the information in their representations for humans. We introduce a modular framework, called Patchscopes(§3), that can be configured to query various kinds of information from LLM representations. Patchscopesdecode specific information from a repre- sentation within an LLM by “patching” it into the inference pass on a different prompt that has been designed to en- courage the extraction of that information. 2 Such configura- tion (aPatchscope) can be viewed as an inspection tool geared towards a particular objective, as illustrated in Fig. 1. We show that many existing methods, including those that rely on vocabulary projections and computation interven- tions, can be cast asPatchscopes. Moreover, new con- figurations of our framework introduce more effective tools in addressing the same questions, while mitigating sev- eral limitations of prior approaches. Also,Patchscopes enables addressing underexplored questions, such as fine- grained analysis of the input contextualization process and the extent to which a more expressive model can be used to inspect hidden representations of a smaller model. 2 While patching (or “activation patching”) by itself is not a new technique, to the best of our knowledge, we are the first to propose using it for decoding information from hidden representations in a configurable and expressive manner. See more details in §3.2. We conduct a series of experiments to evaluate the benefits and opportunities introduced byPatchscopes, focusing on auto-regressive LLMs. First, we consider the problem of estimating the model’s next-token prediction from its intermediate representations (see§4.1). Across multiple LLMs, we show that using a few-shot token identity prompt, a prompt in the form of“tok 1 →tok 1 ; tok 2 → tok 2 ;...;tok k ”wheretok i refers to a random token, leads to substantial gains over vocabulary projection meth- ods. Next, we evaluate how wellPatchscopescan de- code specific attributes of an entity from its LLM represen- tations, when these are detached from the original context (see§4.2). We observe that, despite using no training data, Patchscopessignificantly outperforms probing in six out of twelve commonsense and factual reasoning tasks, and works comparably well in all but one of the remaining six. Beyond output estimation and attribute decoding, Patchscopescan address questions that are hard to answer with existing methods.In§4.3, we apply Patchscopesto study how LLMs contextualize input entity names in early layers, where vocabulary projections mostly fail and other methods, at best, provide only a binary signal of whether the entity has been resolved (Youssef et al., 2023; Tenney et al., 2019). With a newPatchscope, we are able toverbalizethe gradual entity resolution process. For example, we show that, as the model processes the final token of“Alexander the Great”throughout the layers, it reflects different entities starting from“Great Britain”, to“the Great Depression”, to finally resolving“Alexander the Great”. Then, in§4.4 we show how one can further improvePatchscopeexpressivity by using a stronger target model, e.g., Vicuna 13B instead of Vicuna 7B. Lastly, we showcase the utility ofPatchscopesfor fix- ing latent multi-hop reasoning errors, particularly when the model is capable of conducting each reasoning step correctly, but fails when they need to be composed in- context (§5). Building on top of the data provided by Her- nandez et al. (2023b), we introduce a more complex task that requires two steps of factual reasoning.Patchscope achieves 50% accuracy on this task, outperforming chain-of- thought (Wei et al., 2022) (35.71%) and vanilla generations (19.57%). To conclude, our work makes the following contributions: We proposePatchscopes, a general modular framework for decoding information from the hidden representations in LLMs. We show that prominent interpretability methods can be viewed as instances ofPatchscopes, and new configurations result in more expressive, robust across lay- ers, and training-data free alternatives that mitigate their shortcomings. In addition, novel configurations introduce unexplored possibilities of stronger inspection techniques, as well as practical benefits, such as correcting multi-hop 2 Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models reasoning errors. 2. Related Work Activation patching is a causal intervention, commonly used as a tool for studying if certain activations play a key role in a model’s computation (Geiger et al., 2021; Vig et al., 2020). Patching has been used largely for localizing specific infor- mation to specific layers and token positions (Goldowsky- Dill et al., 2023; Meng et al., 2022a;b; Stolfo et al., 2023; Merullo et al., 2023), and for finding information propaga- tion paths in the computation (Wang et al., 2022; Geva et al., 2023; Hendel et al., 2023; Hanna et al., 2023; Lieberum et al., 2023). Prior works have also used specific forms of cross-model patching called stitching, in non-transformer architectures, mostly to analyze representational similar- ity (e.g., Bansal et al., 2021; Csisz ́ arik et al., 2021; Lenc & Vedaldi, 2015). Despite certain limitations (Hase et al., 2023; Zhang & Nanda, 2023), patching remains a principal tool for mechanistic interpretability (Conmy et al., 2023). Given promising results from emerging interpretability ef- forts that employ LLMs to generate human-like text for inspection (e.g., Mousi et al., 2023; Slobodkin et al., 2023; Bills et al., 2023), we argue that using patching only for lo- calization purposes is myopic, and propose to use it for “translating” LLM representations into natural language. Very recently, patching has been used to study new prob- lem setups (e.g., Pal et al., 2023; Hernandez et al., 2023b), all of which can be seen as different configurations of our proposed framework (see §3.2). Among the growing research efforts in inspecting hidden representations of neural networks, probing classifiers are perhaps the most common (e.g., Alain & Bengio, 2017; Belinkov & Glass, 2019; Belinkov, 2022; Wang et al., 2023), and methods using projections into the vocabulary space or their extensions to other domains are another key category (e.g., Merullo et al., 2023; Geva et al., 2022b; nostalgebraist, 2020; Belrose et al., 2023; Dar et al., 2023; Din et al., 2023; Langedijk et al., 2023; Vilas et al., 2023). While various other latent inspection methods exist (e.g., Zhou et al., 2018; Strobelt et al., 2017; Ghandeharioun et al., 2021; Kim et al., 2018), the above are the most relevant to this work. 3.Patchscopes In this section, we introducePatchscopesand show how it extends prior interpretability methods with new capabili- ties. While not limited to particular LLM architectures, this work focuses on auto-regressive transformer-based LLMs. 3.1. Framework Description The key idea inPatchscopesis to leverage the advanced capabilities of LLMs to generate human-like text for “trans- lating” the information encoded in their own hidden repre- sentations. Concretely, given a hidden representation ob- tained from an LLM inference pass, we propose to decode specific information from it bypatchingit into a different inference pass (of the same or a different LLM) that encour- ages the translation of that specific information. Notably, the rest of the forward computation after patching can augment the representation with additional information, hence, this approach does not guarantee that the patched representation itself storesallthat information. However, dispatching the representation from its original context (the source prompt) stops contextualization and guarantees that no further information from the source prompt is incorpo- rated in the post-patching computation. Thus, our frame- work reveals if specific informationcan be decoded from the patched representation via the post-patching computation. Given an input sequence ofntokensS=⟨s 1 ,...,s n ⟩and a modelMwithLlayers,h ℓ i denotes the hidden repre- sentation obtained at layerℓ∈[1,...,L]and position i∈[1,...,n], when runningMonS. To inspecth ℓ i , we consider a separate inference pass of a modelM ∗ with L ∗ layers on a target sequenceT=⟨t 1 ,...,t m ⟩ofmto- kens. Specifically, we choose a hidden representation ̄ h ℓ ∗ i ∗ at layerℓ ∗ ∈[1,...,L ∗ ]and positioni ∗ ∈[1,...,m]in the execution ofM ∗ onT. Moreover, we define a map- ping functionf(h;θ) :R d 7→R d ∗ parameterized byθthat operates on hidden representations ofM, wheredandd ∗ denote the hidden dimension of representations inMand M ∗ , respectively. This function can be the identity function, a linear or affine function learned on task-specific pairs of representations, or even more complex functions that incor- porate other sources of data. Thepatchingoperation refers to dynamically replacing the representation ̄ h ℓ ∗ i ∗ during the inference ofM ∗ onTwithf(h ℓ i ). Namely, by applying ̄ h ℓ ∗ i ∗ ←f(h ℓ i ), we intervene on the generation process and modify the computation after layerℓ ∗ . Overall, aPatchscopeintervention applied to a represen- tation determined by(S,i,M,ℓ), is defined by a quintuplet (T,i ∗ ,f,M ∗ ,ℓ ∗ )of a target promptT, a target positioni ∗ in this prompt, a mapping functionf, a target modelM ∗ , and a target layerℓ ∗ of this model. It is possible thatMand M ∗ are the same model,SandTare the same prompt, and fis the identity functionI(i.e.,I(h) =h). Next, we show how this formulation covers prior interpretability methods and further extends them with new capabilities. 3 Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models 3.2.PatchscopesEncompasses Prior Methods Many recent methods inspect LLM representations by pro- jecting them to the output vocabulary space (nostalgebraist, 2020; Din et al., 2023; Belrose et al., 2023). Formally, an estimation of the output distribution is obtained from the representationh ℓ i at positioniand layerℓby: p ℓ i = softmax(W U f(h ℓ i ))∈R |V| , whereW U ∈R |V|×d is the model’s unembedding matrix andfis a simple mapping function, such as the identity function or an affine mapping. We note that the operation applied tof(h ℓ i )is the same computation applied by the model to the last-layer representation for obtaining the next- token prediction. Therefore, prior methods that inspect representations in the vocabulary space can be viewed as a class ofPatchscopesthat maps representations from any source layerℓto the last target layerL ∗ . Differences between these methods lie in the choice off; logit lens (nostalgebraist, 2020; Dar et al., 2023) applies the identity function, linear shortcuts (Din et al., 2023) uses a linear mapping function, and tuned lens (Belrose et al., 2023) trains an affine mapping. Recently, Hernandez et al. (2023b) introduced LRE Attribute Lens that buildsfbased on a relation linearity assumption, and showed its effectiveness in attribute extraction. This class of methods has proven to be effective for different applications, for example, in improving inference efficiency via early exiting (Din et al., 2023). While the majority of methods and applications in this category use a single model (M ∗ =M), Merullo et al. (2022) had demonstrated successful caption generation with a generative image model asMand a language model asM ∗ . Another category of inspection methods intervene on the LLM computation. Contemporary to our work, Pal et al. (2023) have investigated whether it is possible to anticipate multiple generated tokens ahead from a given hidden repre- sentation, rather than estimating just the next-token predic- tion. Their method (Future Lens) uses a target prompt differ- ent from the original prompt (i.e.,T̸=S) and is designed to decode subsequent tokens from information encoded in a hidden representationh ℓ i . Example target prompts are “The multi-tokens present here are ”and“Hello! Could you please tell me more about ”. Future Lens can be cast as anotherPatchscopewithM ∗ =Mandℓ ∗ =ℓ. More broadly,Patchscopesalso cover recent mechanis- tic interpretability methods that analyze internal processes in LLMs with inference computation interventions. Specif- ically, causal tracing (Meng et al., 2022a) uses a source prompt augmented with Gaussian noise as the target prompt. Other previous methods have intervened on one or more tar- get layers during inference by patching zero vectors to the computation (Wang et al., 2022; Conmy et al., 2023; Geva et al., 2023), namely, settingf(h) =0. For a configuration summary of how these interpretability methods can be cast asPatchscopeinstances, see §A, Tab. 4. 3.3.PatchscopesEnables Novel Inspection Methods Prior work has utilized specific patching configurations for interpretability, largely patching the same model while using the same prompt (i.e.,M ∗ =M,T=S). The framing ofPatchscopesintroduces a wide range of unexplored setups potentially unlocking new inspection capabilities. Specifically, we observe that modifying the target prompt enables an expressive decoding of a wide range of features, detached from the source prompt computation. For instance, we can use the prompt“The capital of X is”to check if the capital city of a given country is extractable from its (last token) hidden representation at a specific layer. Similarly, a prompt like“Tell me facts about X”can be leveraged to assess whether the model has resolved the entity name in a specific layer. Contrary to probing, this approach is not restricted by the number of classes of the chosen feature. Moreover, when the inspected model is not expressive enough to answer certain queries, patching representations into a more capable model could be useful (Hernandez et al., 2022; Singh et al., 2023; Schwettmann et al., 2023). 4. Experiments In this section, we evaluate our framework on decoding next-token predictions (§4.1), extracting attributes (§4.2), analyzing the contextualization of entity names (§4.3), and leveraging stronger models for inspection via cross-model patching (§4.4). See a summary in Tab. 1. 4.1. Decoding of Next-Token Predictions As introduced in§3.2, letp L be the output probability dis- tribution for some input, obtained by multiplying the final- layer last-position hidden representationh L by the unem- bedding matrixW U ∈R |V|×d . We wish to estimatep L from intermediate representationsh ℓ s.t.ℓ < L. Particu- larly, we ask how early in the computation the model has concluded its final prediction from the given context. In our experiments, we consider multiple LLMs – LLaMA2 (13B) (Touvron et al., 2023b), Vicuna (13B) (Chiang et al., 2023), GPT-J (6B) (Wang & Komatsuzaki, 2021), and Pythia (12B) (Biderman et al., 2023) (see more details in §B.1). MethodsWe compare vocabulary projection methods (§3.2) with a newPatchscope. Each method yields an estimated output probability ̃ p ℓ by patching an intermedi- ate representationh ℓ to the model’s final layer. Here, we focus on the common setting whereM=M ∗ , and discuss extensions toM̸=M ∗ in §4.4. 4 Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models 0.0 0.5 1.0 precision@1 Vicuna (13B)Llama2 (13B)Pythia (12B)GPT-J (6B) 1020 layer 0 10 surprisal 1020 layer 1020 layer 1020 layer Logit LensTuned LensToken Identity (Ours) Figure 2.Precision@1 (↑is better) and Surprisal (↓is better) of next-token prediction estimation in multiple models. From layer 10 and upwards, the token identity method (ours) consistently outperforms the rest of the baselines across all the models. • Logit Lens:Following prior work (nostalgebraist, 2020; Geva et al., 2022a), we definefas the identity function, meaning no change is applied to the patched representa- tion. That is,f(h) :=I(h). •Tuned Lens:Motivated by Belrose et al. (2023); Din et al. (2023), we employ an affine mapping function be- tween representations at layerℓand the final layerL. Specifically, we feed the model examples from a train- ing setTand for each examples∈ Tobtain a pair (h ℓ s ,h L s ) of hidden representations. Then, we fit a lin- ear regression model to find the matrixA ℓ ∈R d×d and the bias vectorb ℓ ∈R d that are numerical mini- mizers for P s∈T ||Ah ℓ s −h L s +b|| 2 . We definefas: f(h ℓ ) : =A ℓ h ℓ +b ℓ . •Token IdentityPatchscope: Unlike the previous methods, here we use a target prompt that is differ- ent from the source prompt (T̸=S) and is meant to encourage decoding the token identity of the hid- den representation.Also, while the above methods skip the computation between layerslandL, here we modify it such that all the information from the source prompt computation is discarded, except for the patched representation. We craft a prompt withkdemonstra- tions representing an identity-like function, formatted as “tok 1 →tok 1 ;tok 2 →tok 2 ;...;tok k ”. See §B.3 for further details and an experiment showing this method’s robustness to different demonstrations. Note that thisPatchscopedoes not require any training. EvaluationFollowing Din et al. (2023), we use Pile eval- uation set and the following metrics (see details in §B.2): •Precision@1(↑is better): The portion of examples for which the highest-probability tokentin the estimated probability distribution matches the highest-probability token in the original output distribution. That is, if arg max t ( ̃ p ℓ t ) = arg max t (p L t ). • Surprisal(↓is better): The minus log-probability of the highest-probability token in the predicted distribution ̃ p ℓ according top L , i.e.,−logp L ̃ t , where ̃ t= arg max t ( ̃ p ℓ t ). ResultsAcross all the models, from layer 10 and upwards, the token identityPatchscopeconsistently outperforms the other baselines, obtaining a gain of up to 98% in lay- ers 18-22 (see Fig. 2). This demonstrates the utility of leveraging the model’s decoding procedure for inspecting representations of different source prompts, and shows that in most caseshidden representations in early layers carry the prediction information regardless of their context. In the first 10 layers, performance of all methods is wors- ened, with the token identity prompt performing on-par with logit lens, and tuned lens performing slightly better, which could be due to the additional training of its mappings. Low performance in these layers is expected, as it is where the input contextualization happens. In§4.3, we introduce a Patchscopegeared towards unraveling this process. 4.2. Extraction of Specific Attributes Classification probes are arguably the most commonly used method for checking if certain attributes are encoded in hidden representations (Belinkov, 2022; Belinkov & Glass, 2019). However, they need to be trained, and the range of attribute classes needs to be known a priori. Here we show that repurposingPatchscopesfor attribute extraction overcomes these limitations. First, it does not require train- ing. Second, it is not limited by a predefined set of labels, but rather benefits from an open vocabulary. In addition, by taking advantage of the model’s nonlinearities, it can capture more complex relations compared to linear probes. ExperimentalSetupConsiderfactualandcom- monsense knowledge represented as triplets(σ,ρ,ω) of a subject (e.g.,“United States”), a relation (e.g.,“largest city of”), and an object (e.g., “New York City”). We investigate to what extent the objectωcan be extracted from the last token representation of the subjectσin an arbitrary input context. To this end, we conduct experiments on 8 commonsense and 25 factual knowledge tasks curated by Hernandez et al. (2023b). This dataset includes(σ,ρ,ω)triplets for different relations, along with prompt templates that verbalize them in natural language. We conduct experiments with GPT-J (6B) (Wang & Komatsuzaki, 2021), filtering the data to keep only the examples whereωappears in the the model’s continuation of the prompt up to 20 tokens. The choice of 20 balances computation cost with accommodating the open-ended nature of Patchscopes, as the ground truth token does not necessarily appear in the next immediate token in a fluent 5 Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models Table 2.Feature extraction accuracy (mean±std). Comparing zero- shot feature extractionPatchscopeto a logistic regression probe shows that despite usingno training data, it has a signifi- cantly higher accuracy than baseline in 6 out of 12 tasks. We use pairwise t-test with Bonferroni correction for comparing the two methods. ∗ and ∗ indicatep<1e−5andp<1e−4, respectively. TaskProbePatchscope Commonsense Fruit inside color37.4±6.638.0±18.7 Fruit outside color35.5±3.171.0±13.3 ∗ Object superclass65.6±10.5 ∗ 54.8±11.3 Substance phase73.8±3.791.9±1.7 ∗ Task done by tool10.1±3.248.1±13.2 ∗ Factual Company CEO5.0±2.647.8±13.9 ∗ Country currency17.7±2.251.0±8.9 ∗ Food from country5.1±3.763.8±11.3 ∗ Plays pos. in sport75.9±9.172.2±7.2 Plays pro sport53.8±10.346.3±14.2 Product by co.58.9±7.263.2±10.7 Star constellation17.5±5.318.4±5.1 response. For each example, we sample 5 utterances from the WikiText-103 dataset (Merity et al., 2016) that include σand use them asS. Lastly, we keep tasks with at least 15 samples, which results in 5 commonsense and 7 factual tasks with a total of 1,453 datapoints. See details in §C. MethodsWe compare our proposedPatchscope against linear probing (K ̈ ohn, 2015; Gupta et al., 2015). •Zero-shot Feature ExtractionPatchscope: We craft Tas a general verbalization ofρfollowed by a place- holder forσ, such thati ∗ =m. For example, we use T←“The largest city in x” with “x” as a placeholder for the subject. To extract the object from the entity representation inS, we patch the represen- tation of token “x” at layerℓ ∗ with the representation of “States” from layerℓ, and consider if the gener- ated text includesω. The remaining configurations of thisPatchscopearef←I,M ∗ ← M,i←the last token ofσinS. We consider all combinations of ℓ∈[1,...,L]×ℓ ∗ ∈[1,...,L ∗ ]. Later in this section, we discuss the role ofℓpertaining to attribute extraction. •Logistic Regression Probe: LetΩrepresent the range of possible objects for a given relation. We use the set of unique values ofωin the training set as a proxy for Ω. We train a logistic regression probe (K ̈ ohn, 2015; Gupta et al., 2015) for each layer that predictsω∈Ω from the last token representation ofσ. Given that 6 out of 12 tasks have fewer than 40 datapoints, we use three-fold cross-validation for training and evaluation of this baseline. Note that we have excluded tasks where the probe fails completely due to insufficient number of training examples (fewer than 15 datapoints). EvaluationWe measure the average attribute extraction accuracy. For a given sample, thePatchscopeis consid- ered correct if∃ℓ ∗ ∈[1,...,L ∗ ]where the generated text up to 20 tokens includesω. For the probe, a prediction is correct if the highest probability is assigned toω. ResultsTab. 2 summarizes the results, averaged over ℓ∈[1,...,L]. We conduct a T-test with Bonferroni cor- rection to compare the two methods. Despite using no training data and having no restrictions on the output, the Patchscopeachieves a significantly higher accuracy than the probe on six out of twelve tasks(p<1e−5), and works comparably well in all but one of the remaining six. These results suggest that, in the majority of cases, the source rep- resentation without its original context carries enough infor- mation about many attributes that a targetedPatchscope can extract. We also study how the accuracy changes across the source layers, and observe thatPatchscopeconsis- tently outperforms the baseline in early layers, outperforms or works on par with the baseline in mid layers, and almost all cases where it performs worse than the baseline occur in later layers. Our interpretation is that given the language modeling training objective, the representations shift toward next-token prediction in the later layers. Therefore, the at- tribute of interest would not be as readily accessible via the model’s computation in these layers. This interpretation is also aligned with recent findings that show no decline in using linear relational embedding in predictingωonly when the next token also happens to beω(Hernandez et al., 2023b). Note that this pattern explains the higher standard deviation ofPatchscopeaccuracy observed in Tab. 2. We discuss this phenomenon in more detail in §C (see Fig. 6). 4.3. Analyzing Entity Resolution in Early Layers The previous sections focused on analyzing the information encoded in a single hidden state. Here we turn to consider a more global question of how LLMs resolve entity mentions across multiple layers. Concretely, given a subject entity name, such as“the summer Olympics of 1996”, how does the model contextualize the input tokens of the entity and at which layer is it fully resolved? Answering these questions is hard with existing methods; vocabulary projections focus on the output prediction and fail to show clear patterns in early layers, and probing is restricted to outputs from a fixed number of classes, which may not be expressive enough to describe this process. Al- ternative approaches have studied this process indirectly via interventions (Meng et al., 2022a), showing that the model constructs a subject representation at the last token of the entity name. However, it is still unclear how this 6 Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models Table 3.Illustrating entity resolution via a qualitative example. The expressive generations show that as we go through the layers, more tokens from the context get integrated into the current representa- tion. The “Generation” column shows the automatically generated text. The “Explanation” column shows our own manually coded interpretation, aiming to specify what entity the generation refers to and how that relates to the tokens processed.M ∗ =M ← Vicuna (13B),ℓ ∗ =ℓ,S←"Diana,Princess of Wales". ℓGenerationExplanation 1-2: Country in the United KingdomWales 3: Country in EuropeWales 4 : Title held by female sovereigns in their own right or by queens consort Princess of Wales (unspecific) 5 : Title given to the wife of the Prince of Wales (and later King) Princess of Wales (unspecific) 6 : Diana, Princess of Wales (1961-1997), the first wife of Prince Charles, Prince of Wales, who was famous for her beauty and humanitarian work Diana, Princess of Wales contextualization is performed. We analyze how LLMs contextualize input entity names by leveragingPatchscopes. Particularly, we craft a target prompt for generating a description of a given subject, and apply it to the hidden representation at the last subject po- sition in the source prompt – where the model forms the subject representation (Geva et al., 2023; Hernandez et al., 2023a) – across the early layers. This will allow us to see how the model describes the subject in each layer. AnalysisSettingWeuseafew-shottarget prompt template for decoding an entity description: “subject 1 : description 1 , ..., subject k : description k , x”, while patching the last position corresponding tox. We take the 200 most popular and 200 least popular subject entities from the PopQA dataset (Mallen et al., 2023). The popular entities should appear frequently in LLMs’ pre-training data, and are thus likely to be captured by the model, while resolving the rare entities is expected to be more challenging (Kandpal et al., 2023; Mallen et al., 2023). Then, for the source prompt we use the entity name, and for the target prompt we samplek= 3random subject entities. We obtain a short (up to one sentence) description of every subject entity from Wikipedia. Our target prompt and more technical details are provided in§D.1. We patch the last position representations from the first 10 layers of Vicuna 13B to the target prompt and evaluate the generated subject name and description. Specifically, the generated descriptions are evaluated against the descriptions from Wikipedia using RougeL (Lin, 2004). Evaluation with Rouge1 (Lin, 2004) and Sentence-Bert (Reimers & Gurevych, 2019) shows 02468 layer 0.0 0.1 0.2 0.3 0.4 RougeL M 7B, M* 7B M 13B, M* 13B M 7B, M* 13B popular entities rare entities Figure 3.RougeL scores of the generated descriptions against de- scriptions from Wikipedia, using Vicuna models. similar trends (see §D.2). ResultsTab. 3 illustrates the generations by Vicuna 13B for a sample subject entity, when patching its representation at different layers to the target prompt (see more examples in§D.3). For most entities, the contextualization process is spread over the first layers, with the last subject token grad- ually encompassing more distant positions across layers. This trend can be quantitatively observed by the similarity between the generated descriptions and the descriptions from Wikipedia, as measured by RougeL. See Fig. 3 where M=M ∗ . For both models, similarity increases in the first 5 layers and then slowly decreases. This decrease could potentially be attributed to contamination caused by the representation of the placeholder token “x” remaining in the early layers, when patching is applied to a later layer. Note that this potential issue is only applicable to multi-token generation scenarios as future positions can still attend to the placeholder position in early layers, potentially interfering with the model’s ability to accurately generate descriptions for the patched token. See§D.3 for qualitative examples corroborating this interpretation. As expected, the scores for rare, long-tail entities are significantly lower than those of popular entities. See§D.2 for additional results with Pythia where the smaller model seems to outperform the larger model, possibly because the larger model is biased toward output generation at the expense of input contextualization. To summarize, this analysis showsPatchscopes’ utility for inspecting the contextualization process in early layers. 4.4. Expressiveness from Cross-Model Patching A possible avenue for improving inspection capabilities is to explain a given model with a model that is more expressive (Bills et al., 2023). InPatchscopes, this means to patch a representation ofMinto a more expressive modelM ∗ . However, it is not clear if such an intervention would yield plausible results, due to possible discrepancies between the two models resulting from different architectures, optimiza- tion processes, and so on. Here, we present experiments on next-token prediction and entity resolution that exemplify 7 Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models The current CEO of the company that created Visual Basic Script SourceTarget The current CEO of the company that created Visual Basic Script is: A) A person named "Joe" B) A person named "John" C) A person named "Mike" D) A person named "Jim" The answer is D) A person named "Jim". is Satya Nadella. He has been the CEO since 2014. Figure 4.AnillustrationofCoTPatchscopeonasingleexample.Inthisexample,π 1 ← “the company that created Visual Basic Script” ,π 2 ←“The current CEO of”,S=T←[π 2 ][π 1 ] = “The current CEO of the company that created Visual Basic Script”. Note thatM=M ∗ andf←I. not only the feasibility, but also the opportunities unlocked by patching across models of the same family. Next-Token PredictionWe repeat the experiment in§4.1, using the token identityPatchscope. To overcome dis- crepancies between the models, we learn affine mappings between their layers (similarly to Tuned Lens). Our results show that source and target layers on the diagonal exhibit the highest precision values, and that patching represen- tations to an early layer of the larger model are the most effective. Overall, suggesting that whenM ∗ andMare from the same model family, it is possible to leverageM ∗ for decoding information from the representations ofM. For detailed results on Precision@1 and Surprisal, see §E. Entity Resolution in Early LayersWe now show that us- ing a large model asM ∗ can enhance the output expressivity. To this end, we repeat our entity resolution experiment in §4.3 with Vicuna model family, settingM←7B,M ∗ ←13B. Fig. 3 shows the cross-model patching results (green lines) compared to the same-model patchingM←7B,M ∗ ←7B (blue line). The results show that cross-model patching from a smaller model to its larger version generally improves the ability to inspect the input contextualization, both for pop- ular and rare entities. For Pythia, since the smaller model outperforms the larger one, cross-model patching is not as effective (see §D.2). 5. Application: Correcting Multi-Hop Errors Multi-hop reasoning is a challenging problem (Zhong et al., 2023). While a language model may be capable of correctly answering each step independently, it could still fail at pro- cessing the connection between different steps, resulting in an incorrect prediction. Recent attempts to improve multi- hop reasoning rely on prompting the model to generate a step-by-step answer autoregressively (e.g., Wei et al., 2022; Yao et al., 2023; Besta et al., 2024), some with an itera- tive process of self-refinement (e.g., Madaan et al., 2023). However, achieving similar benefits might be possible via directly rewiring the model’s intermediate computation. Here, we show thatPatchscopescan improve multi-hop reasoning performancewithoutgenerating the reasoning steps, particularly in cases where the model fails at com- pleting a multi-hop query despite being successful in each reasoning step independently. ViaPatchscopes, one can surgically operate on the model representations, reroute its intermediate answer to one reasoning step, simplify the consequent step, and ultimately correct the final prediction. DataBuilding on Hernandez et al. (2023b), we systemati- cally generate all valid multi-hop factual and commonsense reasoning queries whereω 1 =σ 2 . We conduct experiments on Vicuna (13B), focusing on samples whereMaccurately represents bothτ 1 andτ 2 independently, that is,ωappears in the next 20 tokensMgenerates conditioned on the prompt πthat verbalizesσandρ. This process yields 1,104 multi- 8 Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models hop reasoning samples, out of which 46 satisfy the above criteria and are used for evaluation. See more details in §F. Experimental SetupFollowing the notation in§4.2, letτ 1 = (σ 1 ,ρ 1 ,ω 1 )represent the relationρ 1 be- tween a subject entityσ 1 and an object entityω 1 . Let τ 2 = (σ 2 ,ρ 2 ,ω 2 )represent another tuple such that σ 2 =ω 1 . A multi-hop reasoning query pertaining to τ 1 andτ 2 is a prompt composed of two parts:π 1 is a verbalization ofσ 1 andρ 1 from whichω 1 can be inferred; π 2 is a verbalization ofρ 2 , from whichω 2 can be inferred after its concatenation withπ 1 . For example, Letτ 1 ← (“Visual Basic”,“product of”,“Microsoft”) andτ 2 ←(“Microsoft”, “company CEO”, “Satya Nadella”). An example verbalization of these tuples isπ 1 ←“the company that created Visual Basic ”,π 2 ←“The current CEO of”, leading to the multi-hop query[π 2 ][π 1 ] =“The current CEO of the company that created Visual Basic”. MethodWe introduce a Chain-of-Thought (CoT) Patchscopeto fix multi-hop reasoning via intervening on the computation graph and rerouting representation likely to captureω 1 in place ofσ 2 . Concretely,Srefers to the formed query discussed above, and we use the following configura- tion:T←S,M ∗ ←M,i←n,i ∗ ←the token preceding π 1 . We evaluate the outputs in terms of accuracy, similarly to§4.2. For a sampleS, thePatchscopeis considered accurate if∃(ℓ,ℓ ∗ ) :ℓ∈[1,...,L],ℓ ∗ ∈[1,...,L ∗ ]where the autoregressive generation up to 20 tokens includesω 2 . Fig. 4 illustrates the CoTPatchscopewith an example. We use the following configuration for CoTPatchscope: S←π 1 ,T←π 2 ,i←n,i ∗ ←m, which is equivalent to S=T←[π 2 ][π 1 ]and adjusting the attention mask such that no token inShas visibility toπ 2 and no token inThas visibility toπ 1 . In addition, we consider two baselines. Vanilla BaselineFor this baseline, we setS←[π 1 ][π 2 ], we let the model autoregressively generate up to 20 tokens and check whetherω 2 appears in the generation. Chain-of-Thought BaselineHere, the setup and eval- uation is similar to the vanilla baseline, except that we prepend"Let’s think step by step."toS, fol- lowing (Wei et al., 2022). We then let the model gener- ate up to 20 tokens and check whetherω 2 appears in the generation. Note that this experiment uses Vicuna (13B). Vicuna is based on LLaMA, with supervised finetuning on additional instruction data, which makes it amenable to chain-of-thought prompting. ResultsWhile the vanilla baseline accuracy is only 19.57%, and CoT baseline accuracy is 35.71%, our pro- posedPatchscopeachieves 50% accuracy. For more details about the interaction betweenℓandℓ ∗ , and how it affects the success rate, see Fig. 12 in §F. We emphasize that our primary goal in this section is not to devise a new method to solve multi-hop queries that is necessarily a competitor to CoT, but rather to make a proof-of-concept while comparing with CoT as a common reference. We highlight that we have taken advantage of the extra structural information about the queries given the prior knowledge about how they were synthesized. One could po- tentially automate this by learning the right places to patch. Li et al. (2024) recently proposed an optimization-based approach for directly patching multi-head self-attention in mid-layers, which can be viewed as soft position-selection, and works effectively in multihop error correction, suggest- ing that inferring the right positions to patch could be effec- tive. However, even if optimal source and target position are automatically decided upfront, Patchscope and CoT may not be directly comparable. CoT generates multiple steps which fundamentally extend the computational power of the LLM (Merrill & Sabharwal, 2024), but a Patchscope with pre-identified(l,l ∗ )only makesO(1)inference passes. 6. Conclusion We presentPatchscopes, a simple and effective frame- work that leverages the ability of LLMs to generate human- like text for decoding information from intermediate LLM representations. We show that many existing interpretability methods can be cast as specificPatchscopeinstances, and even these only cover a small portion of the frame- work’s possible configurations. Moreover, new underex- ploredPatchscopessubstantially improve our ability to decode various types of information from the model’s inter- nal computation, such as the output prediction and knowl- edge attributes, typically outperforming prominent methods that rely on projection to the vocabulary and probing. In addition, our framework enables new capabilities, such as analyzing the contextualization process of input tokens in early LLM layers, and can correct multi-hop reasoning. There are multiple future research directions to consider. An important factor in the effectiveness of a chosen tar- get prompt is how the information from the patched po- sition propagates during inference to other positions and across layers. Understanding how to best use a given target prompt, perhaps automatically, is an important factor for usingPatchscopes. Another avenue for future work is investigating the effectiveness of few-shot target prompts compared to zero-shot/instruction-based prompts. More expressive instruction-based target prompts, for example, could enable extracting more complex information. Addi- tionally, while our cross-model patching focused on models from the same family, it will be valuable to explore which mapping functions would enable patching across models 9 Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models from different families and perhaps with different architec- tures. Other directions for future work include applications across different domains and modalities, investigating vari- ants with simultaneous multi-token patching or multi-layer patching to mitigate the risks of placeholder contamination, and presenting recipes for task-specific and task-agnostic Patchscopes. Acknowledgements We thank Amir Globerson for feedback on writing and pre- sentation of results. We thank Ardavan Saeedi, Martin Wat- tenberg, Ellie Pavlick, the AI Explorables team 3 at Google Research, and Jasmijn Bastings for their helpful comments. Impact Statement Societal ImpactThis paper presents a new framework for interpreting hidden representations of large language models. Interpretability methods in general can be used to investigate models’ reliability and safety prior to deploy- ment. We hope thatPatchscopesframework facilitates progress in this area with the introduction of inspection tools that are more expressive, robust across layers, and do not require training data. LimitationsWhile our proposed framework is not limited to any particular architecture or domain, the experimental evidence provided in this paper focuses on autoregressive Transformer-based language models, and future work is needed to verify its effectiveness in other setups. References Alain, G. and Bengio, Y. Understanding intermediate layers using linear classifier probes.5th International Con- ference on Learning Representations, Workshop Track Proceedings, 2017. Bansal, Y., Nakkiran, P., and Barak, B. Revisiting model stitching to compare neural representations.Advances in neural information processing systems, 34:225–236, 2021. Belinkov, Y. Probing classifiers: Promises, shortcomings, and advances.Computational Linguistics, 48(1):207–219, 2022. Belinkov, Y. and Glass, J. Analysis methods in neural language processing: A survey.Transactions of the Asso- ciation for Computational Linguistics, 7:49–72, 2019. Belrose, N., Furman, Z., Smith, L., Halawi, D., Ostrovsky, I., McKinney, L., Biderman, S., and Steinhardt, J. Eliciting 3 https://pair.withgoogle.com/explorables latent predictions from transformers with the tuned lens. arXiv preprint arXiv:2303.08112, 2023. Besta, M., Blach, N., Kubicek, A., Gerstenberger, R., Gi- aninazzi, L., Gajda, J., Lehmann, T., Podstawski, M., Niewiadomski, H., Nyczyk, P., et al. Graph of thoughts: Solving elaborate problems with large language models. Proceedings of the 38th AAAI Conference on Artificial Intelligence, 2024. Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O’Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al. Pythia: A suite for analyz- ing large language models across training and scaling. In International Conference on Machine Learning (ICML), 2023. Bills, S., Cammarata, N., Mossing, D., Tillman, H., Gao, L., Goh, G., Sutskever, I., Leike, J., Wu, J., and Saunders, W. Language models can explain neurons in language models.URL https://openaipublic. blob. core. windows. net/neuron-explainer/paper/index. html.(Date accessed: 14.05. 2023), 2023. Casper, S., Rauker, T., Ho, A., and Hadfield-Menell, D. Sok: Toward transparent ai: A survey on interpreting the inner structures of deep neural networks. InFirst IEEE Conference on Secure and Trustworthy Machine Learning, 2022. Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J. E., Stoica, I., and Xing, E. P. Vicuna: An open-source chatbot impressing GPT-4 with 90%* ChatGPT qual- ity, March 2023. URLhttps://lmsys.org/blog/ 2023-03-30-vicuna/. Conmy, A., Mavor-Parker, A. N., Lynch, A., Heimersheim, S., and Garriga-Alonso, A. Towards automated circuit dis- covery for mechanistic interpretability. InThirty-seventh Conference on Neural Information Processing Systems, 2023. Csisz ́ arik, A., K ̋ or ̈ osi-Szab ́ o, P., Matszangosz, ́ A. K., Papp, G., and Varga, D. Similarity and matching of neural network representations. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.),Advances in Neural Information Processing Systems, 2021. URLhttps: //openreview.net/forum?id=aedFIIRRfXr. Dar, G., Geva, M., Gupta, A., and Berant, J. Analyzing transformers in embedding space. InAnnual Meeting of the Association for Computational Linguistics, 2023. Din, A. Y., Karidi, T., Choshen, L., and Geva, M. Jump to conclusions: Short-cutting transformers with linear transformations.arXiv preprint arXiv:2303.09435, 2023. 10 Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. The Pile: An 800GB dataset of diverse text for language modeling.arXiv preprint arXiv:2101.00027, 2020. Geiger, A., Lu, H., Icard, T., and Potts, C. Causal abstrac- tions of neural networks.Advances in Neural Information Processing Systems, 34:9574–9586, 2021. Geva, M., Caciularu, A., Dar, G., Roit, P., Sadde, S., Shlain, M., Tamir, B., and Goldberg, Y. LM-debugger: An inter- active tool for inspection and intervention in transformer- based language models. In Che, W. and Shutova, E. (eds.),Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing: System Demonstrations, p. 12–21, Abu Dhabi, UAE, Decem- ber 2022a. Association for Computational Linguistics. doi: 10.18653/v1/2022.emnlp-demos.2. URLhttps: //aclanthology.org/2022.emnlp-demos.2. Geva, M., Caciularu, A., Wang, K., and Goldberg, Y. Trans- former feed-forward layers build predictions by promot- ing concepts in the vocabulary space. In Goldberg, Y., Kozareva, Z., and Zhang, Y. (eds.),Proceedings of the 2022 Conference on Empirical Methods in Natural Lan- guage Processing, p. 30–45, Abu Dhabi, United Arab Emirates, December 2022b. Association for Computa- tional Linguistics. doi: 10.18653/v1/2022.emnlp-main. 3.URLhttps://aclanthology.org/2022. emnlp-main.3. Geva, M., Bastings, J., Filippova, K., and Globerson, A. Dis- secting recall of factual associations in auto-regressive language models. In Bouamor, H., Pino, J., and Bali, K. (eds.),Proceedings of the 2023 Conference on Em- pirical Methods in Natural Language Processing, p. 12216–12235, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023. emnlp-main.751. URLhttps://aclanthology. org/2023.emnlp-main.751. Ghandeharioun, A., Kim, B., Li, C.-L., Jou, B., Eoff, B., and Picard, R. W. Dissect: Disentangled simultane- ous explanations via concept traversals.arXiv preprint arXiv:2105.15164, 2021. Goldowsky-Dill, N., MacLeod, C., Sato, L., and Arora, A. Localizing model behavior with path patching.arXiv preprint arXiv:2304.05969, 2023. Gupta, A., Boleda, G., Baroni, M., and Pad ́ o, S. Distribu- tional vectors encode referential attributes. InProceed- ings of the 2015 Conference on Empirical Methods in Natural Language Processing, p. 12–21, 2015. Hanna, M., Liu, O., and Variengien, A. How does GPT-2 compute greater-than?: Interpreting mathematical abili- ties in a pre-trained language model. InThirty-seventh Conference on Neural Information Processing Systems, 2023. URLhttps://openreview.net/forum? id=p4PckNQR8k. Hase, P., Bansal, M., Kim, B., and Ghandeharioun, A. Does localization inform editing? surprising differences in causality-based localization vs. knowledge editing in lan- guage models.arXiv preprint arXiv:2301.04213, 2023. Hendel, R., Geva, M., and Globerson, A. In-context learning creates task vectors. InThe 2023 Conference on Empiri- cal Methods in Natural Language Processing, 2023. Hernandez, E., Schwettmann, S., Bau, D., Bagashvili, T., Torralba, A., and Andreas, J.Natural language de- scriptions of deep features. InInternational Confer- ence on Learning Representations, 2022. URLhttps: //openreview.net/forum?id=NudBMY-tzDr. Hernandez, E., Li, B. Z., and Andreas, J. Measuring and ma- nipulating knowledge representations in language models. arXiv preprint arXiv:2304.00740, 2023a. Hernandez, E., Sharma, A. S., Haklay, T., Meng, K., Watten- berg, M., Andreas, J., Belinkov, Y., and Bau, D. Linear- ity of relation decoding in transformer language models. arXiv preprint arXiv:2308.09124, 2023b. Kandpal, N., Deng, H., Roberts, A., Wallace, E., and Raffel, C. Large language models struggle to learn long-tail knowledge. InInternational Conference on Machine Learning, p. 15696–15707. PMLR, 2023. Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F., et al. Interpretability beyond feature attribu- tion: Quantitative testing with concept activation vectors (tcav). InInternational conference on machine learning, p. 2668–2677. PMLR, 2018. K ̈ ohn, A. What’s in an embedding? analyzing word embed- dings through multilingual evaluation. InProceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, p. 2067–2073, 2015. Langedijk, A., Mohebbi, H., Sarti, G., Zuidema, W., and Jumelet, J.Decoderlens: Layerwise interpreta- tion of encoder-decoder transformers.arXiv preprint arXiv:2310.03686, 2023. Lenc, K. and Vedaldi, A. Understanding image representa- tions by measuring their equivariance and equivalence. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 991–999, 2015. 11 Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models Li, Z., Jiang, G., Xie, H., Song, L., Lian, D., and Wei, Y. Understanding and patching compositional reasoning in llms.arXiv preprint arXiv:2402.14328, 2024. Lieberum, T., Rahtz, M., Kram ́ ar, J., Irving, G., Shah, R., and Mikulik, V. Does circuit analysis interpretability scale? evidence from multiple choice capabilities in chin- chilla.arXiv preprint arXiv:2307.09458, 2023. Lin, C.-Y.ROUGE: A package for automatic evalua- tion of summaries. InText Summarization Branches Out, p. 74–81, Barcelona, Spain, July 2004. Asso- ciation for Computational Linguistics. URLhttps: //aclanthology.org/W04-1013. Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., Gupta, S., Majumder, B. P., Hermann, K., Welleck, S., Yazdanbakhsh, A., and Clark, P. Self-refine: Iter- ative refinement with self-feedback. InThirty-seventh Conference on Neural Information Processing Systems, 2023. URLhttps://openreview.net/forum? id=S37hOerQLB. Madsen, A., Reddy, S., and Chandar, S. Post-hoc inter- pretability for neural nlp: A survey.ACM Computing Surveys, 55(8):1–42, 2022. Mallen, A., Asai, A., Zhong, V., Das, R., Khashabi, D., and Hajishirzi, H. When not to trust language mod- els: Investigating effectiveness of parametric and non- parametric memories. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.),Proceedings of the 61st Annual Meet- ing of the Association for Computational Linguistics (Vol- ume 1: Long Papers), p. 9802–9822, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.546.URLhttps: //aclanthology.org/2023.acl-long.546. Meng, K., Bau, D., Andonian, A., and Belinkov, Y. Locating and editing factual associations in gpt.Advances in Neu- ral Information Processing Systems, 35:17359–17372, 2022a. Meng, K., Sharma, A. S., Andonian, A. J., Belinkov, Y., and Bau, D. Mass-editing memory in a transformer. InThe Eleventh International Conference on Learning Repre- sentations, 2022b. Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. InInternational Conference on Learning Representations, 2016. Merrill, W. and Sabharwal, A. The expressive power of transformers with chain of thought. InThe Twelfth Inter- national Conference on Learning Representations, 2024. Merullo, J., Castricato, L., Eickhoff, C., and Pavlick, E. Lin- early mapping from image to text space. InThe Eleventh International Conference on Learning Representations, 2022. Merullo, J., Eickhoff, C., and Pavlick, E. A mechanism for solving relational tasks in transformer language models. arXiv preprint arXiv:2305.16130, 2023. Mousi, B., Durrani, N., and Dalvi, F. Can llms facilitate interpretation of pre-trained language models? InPro- ceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Association for Com- putational Linguistics, December 2023. URLhttps: //browse.arxiv.org/pdf/2305.13386.pdf. Nanda, N., Lee, A., and Wattenberg, M. Emergent lin- ear representations in world models of self-supervised sequence models. In Belinkov, Y., Hao, S., Jumelet, J., Kim, N., McCarthy, A., and Mohebbi, H. (eds.), Proceedings of the 6th BlackboxNLP Workshop: An- alyzing and Interpreting Neural Networks for NLP, p. 16–30, Singapore, December 2023. Association for Computational Linguistics.doi: 10.18653/v1/2023. blackboxnlp-1.2. URLhttps://aclanthology. org/2023.blackboxnlp-1.2. nostalgebraist.interpreting gpt: the logit lens.Less- Wrong, 2020.URLhttps://w.lesswrong. com/posts/AcKRB8wDpdaN6v6ru/ interpreting-gpt-the-logit-lens. Pal, K., Sun, J., Yuan, A., Wallace, B. C., and Bau, D. Future lens: Anticipating subsequent tokens from a single hidden state. InProceedings of the 27th Conference on Computational Natural Language Learning (CoNLL), p. 548–560, 2023. Patel, R. and Pavlick, E. Mapping language models to grounded conceptual spaces. InInternational Conference on Learning Representations, 2021. Reimers, N. and Gurevych, I. Sentence-bert: Sentence em- beddings using siamese bert-networks. InProceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing. Association for Computational Linguistics, 11 2019. URLhttps://arxiv.org/ abs/1908.10084. Schwettmann, S., Chowdhury, N., Klein, S., Bau, D., and Torralba, A. Multimodal neurons in pretrained text-only transformers. InProceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, p. 2862–2867, 2023. Singh, C., Hsu, A., Antonello, R., Jain, S., Huth, A., Yu, B., and Gao, J. Explaining black box text modules in natural 12 Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models language with language models. InXAI in Action: Past, Present, and Future Applications, 2023. URLhttps: //openreview.net/forum?id=3BX9tM03GT. Slobodkin, A., Goldman, O., Caciularu, A., Dagan, I., and Ravfogel, S. The curious case of hallucinatory (un)answerability: Finding truths in the hidden states of over-confident large language models. In Bouamor, H., Pino, J., and Bali, K. (eds.),Proceedings of the 2023 Conference on Empirical Methods in Natural Lan- guage Processing, p. 3607–3625, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.220. URLhttps:// aclanthology.org/2023.emnlp-main.220. Stolfo, A., Belinkov, Y., and Sachan, M. A mechanis- tic interpretation of arithmetic reasoning in language models using causal mediation analysis. In Bouamor, H., Pino, J., and Bali, K. (eds.),Proceedings of the 2023 Conference on Empirical Methods in Natural Lan- guage Processing, p. 7035–7052, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.435. URLhttps:// aclanthology.org/2023.emnlp-main.435. Strobelt, H., Gehrmann, S., Pfister, H., and Rush, A. M. Lstmvis: A tool for visual analysis of hidden state dy- namics in recurrent neural networks.IEEE transactions on visualization and computer graphics, 24(1):667–676, 2017. Tenney, I., Das, D., and Pavlick, E. Bert rediscovers the classical nlp pipeline. InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, p. 4593–4601, 2019. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi ` ere, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models.arXiv preprint arXiv:2302.13971, 2023a. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models.arXiv preprint arXiv:2307.09288, 2023b. Vig, J., Gehrmann, S., Belinkov, Y., Qian, S., Nevo, D., Singer, Y., and Shieber, S. Investigating gender bias in language models using causal mediation analysis.Ad- vances in neural information processing systems, 33: 12388–12401, 2020. Vilas, M. G., Schauml ̈ offel, T., and Roig, G. Analyzing vision transformers for image classification in class em- bedding space.arXiv preprint arXiv:2310.18969, 2023. Wallat, J., Singh, J., and Anand, A.BERTnesia: In- vestigating the capture and forgetting of knowledge in BERT. In Alishahi, A., Belinkov, Y., Chrupała, G., Hupkes, D., Pinter, Y., and Sajjad, H. (eds.),Proceed- ings of the Third BlackboxNLP Workshop on Analyz- ing and Interpreting Neural Networks for NLP, p. 174– 183, Online, November 2020. Association for Computa- tional Linguistics. doi: 10.18653/v1/2020.blackboxnlp-1. 17.URLhttps://aclanthology.org/2020. blackboxnlp-1.17. Wang, B. and Komatsuzaki, A.GPT-J-6B: A 6BillionParameterAutoregressiveLanguage Model.https://github.com/kingoflolz/ mesh-transformer-jax, May 2021. Wang, K. R., Variengien, A., Conmy, A., Shlegeris, B., and Steinhardt, J. Interpretability in the wild: a circuit for indi- rect object identification in GPT-2 small. InThe Eleventh International Conference on Learning Representations, 2022. Wang, Z., Ku, A., Baldridge, J., Griffiths, T. L., and Kim, B. Gaussian Process Probes (GPP) for uncertainty-aware probing.arXiv preprint arXiv:2305.18213, 2023. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35: 24824–24837, 2022. Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., and Narasimhan, K. R. Tree of thoughts: Deliberate problem solving with large language models. InThirty- seventh Conference on Neural Information Processing Systems, 2023. Youssef, P., Koras ̧, O., Li, M., Schl ̈ otterer, J., and Seifert, C. Give me the facts! a survey on factual knowledge probing in pre-trained language models. InFindings of the Association for Computational Linguistics: EMNLP 2023, p. 15588–15605, 2023. Zhang, F. and Nanda, N. Towards best practices of activation patching in language models: Metrics and methods.arXiv preprint arXiv:2309.16042, 2023. Zhong, Z., Wu, Z., Manning, C. D., Potts, C., and Chen, D.Mquake: Assessing knowledge editing in lan- guage models via multi-hop questions.arXiv preprint arXiv:2305.14795, 2023. Zhou, B., Bau, D., Oliva, A., and Torralba, A. Interpreting deep visual representations via network dissection.IEEE transactions on pattern analysis and machine intelligence, 41(9):2131–2145, 2018. 13 Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models A. Detailed Configuration of Prior Methods asPatchscopeInstances In this section, we first reiteratePatchscopespremise and distinguish it from prior work. We then detailPatchscope configurations that replicate prior methods. We highlight that clearly formulatingPatchscopesin the form of source and target quintuplets, respectively(S,i,M,l)and(T,i ∗ ,M ∗ ,l ∗ ), is not in itself a contribution of this work. However, it is what facilitates the main contributions that are 3-fold: 1) The framework ties methods that otherwise might seem unrelated, e.g., logit lens and attention knockout, 2) makes it easy to come up with new configurations that significantly improve them, 3) makes it easy to see the space that prior methods do not cover, and therefore open up new applications altogether. We focus on a few inspection objectives. Note that some of the baselines considered have not been originally designed to address these objectives, but are the closest solutions in literature. Particularly, we ask, given a (set of) hidden representation(s) detached from their context, how well can we do next-token prediction, feature extraction, and analysis of model’s contextualization process. A.1. Next-token Prediction Goal:Estimating output probability distributionp L given a hidden representationh l , particularly to inspect when the model has enough information to make the final prediction. Prior Work:Many recent methods to estimate output probability distribution given an inner representation use projection onto the vocabulary space (e.g., nostalgebraist, 2020; Din et al., 2023; Belrose et al., 2023). Considering(S,i,M,l)and (T,i ∗ ,M ∗ ,l ∗ )formulation above, all these methods can be seen asPatchscopeswhereM=M ∗ ,l ∗ =L, whereLis the number of layers. DistinguishingPatchscope& its Improvements:We propose the token identityPatchscopethat unlike (Belrose et al., 2023; Din et al., 2023) does not require a learned mapping, yet is more successful. Note that in this task, setting l ∗ =Lwhich all the prior methods discussed above have in common, means that the choice ofTdoes not matter. However, in our proposedPatchscope, we show that settingl ∗ =lwherelis arbitrary layer, and using a few-shot token identity promptTthat encourages repetition results in significant improvements, and by using the target model’s computation from layerl ∗ onward, there is no need for training a separate mapping function. A.2. Feature Extraction Goal:Extracting specific attributes of the subject given its hidden representationh l (e.g., extracting “largest city of” from the representation of “States” in “United States”). Prior Work:Classification probes (Alain & Bengio, 2017) are the most commonly used methods for this purpose. They work by training a classifier on a dataset of prompts and their corresponding labels (e.g., countries and their corresponding largest cities). DistinguishingPatchscope& its Improvements:We show an intuitivePatchscope(i.e.,T=“The largest city in x”) significantly outperforms probing across various tasks and layers. Note that thisPatchscope1) does not require supervised training, 2) is not limited by a predefined set of labels, and 3) is more accurate than a linear probe. A.3. Analysis of Model’s Contextualization Process Goal:Identifying how entity tokens are processed, and at which layer the entity is fully resolved. Prior Work:It is hard to answer this question with existing methods. First, methods that use vocabulary projection are focused on output prediction (rather than input processing) and fail particularly in early layers, precisely where entity resolution happens. Second, probing requires a predefined set of classes, which is not expressive enough to show the gradual entity resolution process, and nontrivial. For example, consider “Alexander the grea” input. We found that “Great Britain” and “Great depression” arose in the gradual resolution process. However, these are non-trivial choices a priori. We can train a multi-class classifier to choose one out of all possible entities (or binary classifiers for every possible entity), which is unlikely to work given the huge space of possible entities. Lastly, most prior work using activation patching aims to answer a different question: whether a certain representation plays a key role in the model’s forward computation and the 14 Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models final prediction (e.g., Meng et al., 2022a; Geva et al., 2023). However, they can be viewed asPatchscopeinstances, and provide indirect signals about the entity resolution process, namely, showing subject information accrues in the last token after a few layers. DistinguishingPatchscope& its Improvements:To the best of our knowledge, there are not any methods that verbalize the gradual entity resolution process in coherent text, and the proposedPatchscopeis the first to do that. In summary, as discussed in 3.2, many prior methods can be seen asPatchscopes. Tab. 4 details the configurations of Patchscopesthat yield various methods introduced in prior work. Table 4.Patchscopesis a novel framework for inspection of hidden representations in language models. Many prior inspection methods with various objectives can be viewed asPatchscopes, as detailed in the “Configuration” column (see notation description in§3). The rows highlighted in green show our new configurations that overcome several limitations of prior methods through more expressive inspection that is training-data free and is more robust across layers. Additionally, the generality of this framework enables novel inspection possibilities that were unexplored before. When the target prompt (T) is not specified, it means that the output would be invariant to the choice ofT. When not specified,f←IandM ∗ =M. Inspection Objective Expressive Training Data Free Robust Across Layers Configuration Inspecting Output Distribution Few-shot token identityPatchscope(§4.1)Ë ℓ ∗ ←ℓ, T←“tok 1 →tok 1 ; tok 2 →tok 2 ;...;tok k ” Logit Lens (nostalgebraist, 2020), Embedding Space Analysis (Dar et al., 2023) Ëé ℓ ∗ ←L ∗ Tuned Lens (Belrose et al., 2023) Ë For learning mappings Ëℓ ∗ ←L ∗ ,f←Affine Future Lens (Pal et al., 2023)Ë For learning mappings Ëℓ ∗ ←ℓ,f←Linear,T←Fixed or learned soft prompt Feature Extraction Zero-shot feature extractionPatchscope(§4.2)Ë ℓ ∗ ←j ′ ∈[1,...,L ∗ ],i ∗ ←m, T←relation verbalization followed byx LRE Attribute Lens (Hernandez et al., 2023b) Ë For linear relation approx. Ë ℓ ∗ ←L ∗ ,f←Linear with additional variables,T←S Probing (e.g., Belinkov & Glass, 2019; Belinkov, 2022; Alain & Bengio, 2017; Wang et al., 2023) é For training probe ËN/A Entity Resolution Entity descriptionPatchscope(§4.3)Ë ℓ ∗ ←ℓ,i ∗ ←m, T←“subject 1 : description 1 , ..., subject k : description k , x” X-model entity descriptionPatchscope(§4.4)Ë For learning mappings Ë M ∗ ←a larger variant ofM,ℓ ∗ ←ℓ,i ∗ ←m, T←“subject 1 : description 1 , ... , subject k : description k , x” Causal Tracing (Meng et al., 2022a)éËℓ ∗ ←ℓ,T←S+ε,ε∼N(0,σ) Attention Knockout (Wang et al., 2022; Conmy et al., 2023; Geva et al., 2023) éËℓ ∗ ←Multiple,f←0,T←S Inspection Application Early Exiting, e.g., Linear Shortcuts (Din et al., 2023) Ë For learning mappings Ë ℓ ∗ ←L ∗ ,f←Affine Caption Generation, e.g., Linear Mapping (Merullo et al., 2022) Ë For learning mappings ËM ∗ ←A language model of choice,ℓ ∗ ←L ∗ ,f←Affine B. Next-Token Prediction Additional Details and Experimental Results In this section, we provide more information about the models, data, and additional target prompts for few-shot token identityPatchscopeconfigurations. It is worth highlighting that after patching, we continue computations from the target layer onward. Therefore, since in thisPatchscope, source layer and target layer are the same, there are going to be more computations compared to baseline methods like LogitLens (nostalgebraist, 2020; Geva et al., 2022a) and Tuned Lens (Belrose et al., 2023; Din et al., 2023) where the target layer is fixed at the final layer. B.1. Models We use LLaMA2 (13B) 4 (Touvron et al., 2023b), Vicuna (13B) 5 (Chiang et al., 2023), GPT-J (6B) (Wang & Komatsuzaki, 2021) 6 , and Pythia (12B) 7 (Biderman et al., 2023). LLaMA2 was pre-trained on 2T tokens from a mix of publicly available data. Vicuna is a LLaMA1 (Touvron et al., 2023a) model that was pre-trained on 1T tokens and fine-tuned on 70K 4 LLaMA 2 is licensed under the LLAMA 2 Community License, Copyright © Meta Platforms, Inc. All Rights Reserved. 5 Vicuna is subject to the LLAMA 2 Community License, terms of use of the data generated by OpenAI, and privacy practices of ShareGPT. The code is released under the Apache License 2.0. 6 GPT-J-6B weights are licensed under version 2.0 of the Apache License. 7 Pythia is released under Apache License 2.0, Copyright © 2023 EleutherAI. 15 Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models 010203040 layer 0.0 0.2 0.4 0.6 0.8 1.0 precision@1 450, 41, 8355 2, 7509, 564, 23, 10532, 443 2426, 11 98 65, 32354, 435, 3 Single token prompt (a) Precision@1 010203040 layer 0 5 10 15 surprisal 450, 41, 8355 2, 7509, 564, 23, 10532, 443 2426, 11 98 65, 32354, 435, 3 Single token prompt (b) Surprisal Figure 5.Next-token prediction results for LLaMA2, using various token identity demonstrations (the token IDs appear in the legend). We report precision@1 (↑is better), and surprisal (↓is better). user-shared conversations 8 . The primary architectural differences between LLaMA2 and Vicuna (LLaMA1) include a different context length and grouped-query attention. Pythia and GPT-J were pre-trained using a deduplicated version of the Pile corpus 9 (Gao et al., 2020), and for about 300B and 402B tokens, respectively. B.2. Training and Evaluation Data We use 12,000 random samples from the Pile, partitioned into 10,000 examples for training the affine mappings, and 2,000 examples for evaluation. In our pre-processing strategy, we introduce randomness in the patching positions by trimming the input sequence length of each example. B.3. Additional Few-Shot Token Identity Prompts In this section, we provide additional details about the selection of the demonstrations for the token identity baseline, and further evaluate the robustness of LLaMA2 (13B) (Touvron et al., 2023b) to various token identity prompts. Demonstrations ConstructionFor the demonstrations used in this experiment, we sample a random set ofktokens for all the models, wherekwas also randomly sampled from the interval[1,...,10]. Robustness to Additional token IDs’ DemonstrationsWe randomly generate five realizations of token IDs series of varying lengths, formatted as “tok 1 →tok 1 ; tok 2 →tok 2 ;...; tok k ”, similarly to the procedure from §4.1. We also include an ablation experiment, introducing theSingle Token Promptbaseline. In this approach, the representation is transferred from the original multi-token prompt to a single-token prompt, while preserving the same position. This is crucial for maintaining consistent positional biases during the computation in the forward pass. The baseline is implemented by creating a target prompt in the format: “[PAD] [PAD]...[PAD] tok k ”. Here, the number of [PAD]tokens is chosen such that “tok k ” retains its original position from the source prompt. The results are illustrated in Fig. 5, where a comprehensive overview of the evaluation metrics can be found in§4.1. The results indicate the stability of the token identity baseline across a range of token identity demonstrations, particularly notable in the upper layers of the model. TheSingle Token Promptbaseline overall seems to be less effective than using our token ID baseline. 16 Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models Table 5.Comparison between the zero-shot feature extractionPatchscopeand the logistic regression probe shows that despite using no training data, thePatchscopehas a significantly higher accuracy than the baseline in most tasks(p <1e−5). Pairwise t-statistics and the corresponding p-values are included in the table. The columns corresponding to each method show accuracy (mean±std). TaskProbePatchscopeT-statisticp-value Commonsense Fruit inside color37.41±6.5837.99±18.670.1260.901 Fruit outside color35.50±3.0971.00±13.26 ∗ 12.426<1e−5 Object superclass68.92±10.69 ∗ 55.71±10.81−5.25<1e−4 Substance phase73.77±3.7491.92±1.73 ∗ 25.647<1e−5 Task done by person0±062.96±16.513 ∗ 19.632<1e−5 Task done by tool10.14±3.2348.12±13.23 ∗ 18.231<1e−5 Work location0±013.58±9.37 ∗ 7.45990<1e−5 Factual Company CEO4.99±2.5647.82±13.89 ∗ 16.700<1e−5 Country capital city0±061.61±14.14 ∗ 22.426<1e−5 Country currency17.70±2.2050.95±8.85 ∗ 20.293<1e−5 Country largest city0±067.78±11.47 ∗ 30.427<1e−5 Food from country5.13±3.6663.80±11.34 ∗ 26.710<1e−5 Person father0±025.34±8.42 ∗ 15.482<1e−5 Person plays position in sport75.89±9.1472.20±7.21−2.0660.049 Person plays pro sport53.87±10.2846.28±14.19−2.0200.054 Product by company58.91±7.1563.24±10.741.7570.091 Star constellation17.54±5.3018.35±5.06−2.980.006 Superhero archnemesis0±041.73±18.72 ∗ 11.47044<1e−5 Superhero person0±028.32±14.05 ∗ 10.37461<1e−5 C. More Details on Attribute Extraction Experiments Dataset DetailsWe start from the factual and commonsense reasoning subsets introduced by Hernandez et al. (2023b) 10 . This dataset includes 8 commonsense and 25 factual relations. Recall that each datapoint is representing a triplet(σ,ρ,ω), whereσrefers to the subject,ρrepresents the relation, andωstands for the object (see 4.2 for notation details). For each data point representing the triplet(σ,ρ,ω), we sample 5 utterances fromWikitext-103dataset (Merity et al., 2016) including σ. We then truncate the sampled text to a window of random length up to 20 tokens that containsσ. This constitutes our source prompt,S. Note that for each model, we filter the data to samples for which the underlying model correctly encodes the tuple. For experiments with zero-shot target promptTthat includesρfollowed byσ, we autoregressively generate the next 20 tokens, and only keep examples whereωappears in the generation. For experiments with few-shot demonstrations inT, after generating the next 20 tokens, we do an additional post-processing step. If the demonstration template of an example is identified in the generation, all the following tokens would be dropped. The example is used for evaluation only ifωappears in the post-processed truncated generation. To have a reasonable amount of data for training the classification probe baseline, tasks with fewer than 15 datapoints are dropped from the analysis. For GPT-J, 5 commonsense and 7 factual reasoning tasks remain after applying the above postprocessing steps. Results on Additional TasksWe provide additional results on various factual and commonsense reasoning tasks. For a comparison of the zero-shot feature extractionPatchscopeand the logistic regression probe averaged across layers, see t-statistic details in Tab. 5. Performance Breakdown Across Source LayersFig. 6 depicts how accuracy changes across source layersℓ∈[1,...,L]. Considering the early layers, we observe that thePatchscopeconsistently outperforms the baseline. This further confirms our hypothesis that prior methods are particularly limited in surfacing information in the early layers, which often cannot be decoded via linear functions. However,Patchscopeis able to extract such attributes significantly earlier. Note that this observation does not mean the attribute isexplicitlyencoded in a representation, but that there is enough information encoded such that the attribute can be extracted from the representation alone, without its original context, using the model’s computation. In the middle layers,Patchscopeworks similarly or better than the baseline. Interestingly, we observe 8 Conversations were collected fromwww.sharegpt.com. 9 Pile dataset is available under MIT license. 10 The dataset released by Hernandez et al. (2023b) is available under MIT license. 17 Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models that almost all cases wherePatchscopeperforms worse than the baseline occur in later layers. Our interpretation is that given the language modeling training objective, the representations shift toward next-token prediction in the later layers. Therefore, the attribute of interest would not be as readily accessible via the model’s computation in these layers. This interpretation is also aligned with recent findings that show no decline in using linear relational embedding in predictingω only when the next token also happens to beω(Hernandez et al., 2023b). We postulate that when the immediate next token is not the attribute of interest, it does not necessarily mean the attribute information is lost, but rather it may not be as easily accessible on the surface. We hypothesize that using aPatchscopewith a more expressive mappingfcould improve attribute extraction accuracy in the later layers, which we leave for future work to verify. Source-Target Layer InterplayFig. 7 visualizes the interaction betweenℓandℓ ∗ . These heatmaps show attribute extraction success rate for the zero-shot feature extractionPatchscopefor a fixed(ℓ,ℓ ∗ )combination. The lower left quadrants show setups where bothℓandℓ ∗ represent early to middle layers, and the success rate is maximal. The right half of the heatmaps represents lateℓ, which achieves lower success rate due to token representations shifting toward next-token prediction as discussed earlier. In addition, we notice lower success rate in the top half of the heatmaps which represents late ℓ ∗ . It is worth noting that in this task, the accuracy is not only based on the immediate next-token prediction, but rather whetherωappears in the next 20 autoregressively generated tokens. The placeholder token “x” does still remain in the input, and its representation persists in the early layers in the target computation. This explains why a lower attribute extraction rate is observed in laterℓ ∗ values. We leave it to the future work to investigate adaptations toPatchscopesto control contamination from the placeholder tokens and make them more amenable to lateℓ ∗ choices. Statistical Test DetailsThe null hypothesis is: “There is no significant difference between probing accuracy and patchscope accuracy”. We calculate attribute extraction accuracy per layer, in a 40-layer model, across various tasks. Since we run a test for each task independently (see Tab. 2), we use the conservative Bonferroni correction to address the multiple comparison problem. That is, for a desired overallα= 0.05, we considerα/number of tasks= 0.05 12 = 0.004for each individual task. D. Additional Information and Results on the Entity Resolution Experiment D.1. Experimental Setup Recall that we use a few-shot target prompt template for decoding an entity description:“subject 1 : description 1 , subject 2 :description 2 , ..., subject k :description k , x”, while patching the last position which corresponds to “x”. Specifically, we use the following target prompt, obtained randomly: “Syria: Country in the Middle East, Leonardo DiCaprio: American actor, Samsung: South Korean multinational major appliance and consumer electronics corporation, x”and task the model to generate the completion after the patched representation in “x”. For the subject description, which is composed ofk= 3random subject entities, we used the wptools 11 python package for obtaining a description of every subject entity from Wikipedia. D.2. Additional Quantitative Results In this section, we present the Rouge1 (Lin, 2004) and SBERT score (Reimers & Gurevych, 2019) results, as well as the results for the Pythia models (Biderman et al., 2023). 12 In Fig. 8 and Fig. 9, we present the Rouge1 and the SBERT results, respectively, complementary to the RougeL results in Fig. 3 from§4.3. Note that for Pythia, the smaller model (6.9B) outperforms the larger one (12B), and hence, our cross-model patching method would not be able to improve the inspection of the smaller model, unlike the trends we observed for Vicuna. D.3. Additional Qualitative Results In this section, we provide more examples and discuss our observations about the gradual process of entity resolution. As shown in Tab. 6 and Tab. 7, it is interesting to observe that the resolution process for the same input can look different across models, suggesting they assign different likelihoods to different entities, and weigh context differently. For example, 11 https://github.com/siznax/wptools/ 12 We used the package fromhttps://w.sbert.net/, with the sentence-transformers/all-MiniLM-L6-v2 model. 18 Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models 0510152025 Source Layer ( ) 65 70 75 80 85 90 95 Attribute Extraction Acc. Attribute Extraction Acc. substance phase (commonsense) # Source prompts: 182 - # Classes: 3 Zero-Shot Feat. Ext. PatchscopeLogistic Reg. Probe (a) Substance Phase 0510152025 Source Layer ( ) 0 20 40 60 Attribute Extraction Acc. Attribute Extraction Acc. task done by tool (commonsense) # Source prompts: 54 - # Classes: 12 Zero-Shot Feat. Ext. PatchscopeLogistic Reg. Probe (b) Task Done By Tool 0510152025 Source Layer ( ) 10 20 30 40 50 60 Attribute Extraction Acc. Attribute Extraction Acc. country currency (factual) # Source prompts: 83 - # Classes: 14 Zero-Shot Feat. Ext. PatchscopeLogistic Reg. Probe (c) Country Currency 0510152025 Source Layer ( ) 0 20 40 60 80 Attribute Extraction Acc. Attribute Extraction Acc. food from country (factual) # Source prompts: 27 - # Classes: 10 Zero-Shot Feat. Ext. PatchscopeLogistic Reg. Probe (d) Food From Country 0510152025 Source Layer ( ) 0 10 20 30 40 50 60 70 Attribute Extraction Acc. Attribute Extraction Acc. company ceo (factual) # Source prompts: 50 - # Classes: 12 Zero-Shot Feat. Ext. PatchscopeLogistic Reg. Probe (e) Company CEO 0510152025 Source Layer ( ) 20 30 40 50 60 70 80 90 Attribute Extraction Acc. Attribute Extraction Acc. fruit outside color (commonsense) # Source prompts: 56 - # Classes: 6 Zero-Shot Feat. Ext. PatchscopeLogistic Reg. Probe (f) Fruit Outside Color 0510152025 Source Layer ( ) 30 40 50 60 70 Attribute Extraction Acc. Attribute Extraction Acc. product by company (factual) # Source prompts: 219 - # Classes: 11 Zero-Shot Feat. Ext. PatchscopeLogistic Reg. Probe (g) Product By Company 0510152025 Source Layer ( ) 10 20 30 40 50 60 70 Attribute Extraction Acc. Attribute Extraction Acc. person plays pro sport (factual) # Source prompts: 131 - # Classes: 5 Zero-Shot Feat. Ext. PatchscopeLogistic Reg. Probe (h) Person Plays Pro Sport 0510152025 Source Layer ( ) 50 60 70 80 Attribute Extraction Acc. Attribute Extraction Acc. person plays position in sport (factual) # Source prompts: 383 - # Classes: 7 Zero-Shot Feat. Ext. PatchscopeLogistic Reg. Probe (i) Person Plays Position in Sport 0510152025 Source Layer ( ) 30 40 50 60 70 80 Attribute Extraction Acc. Attribute Extraction Acc. object superclass (commonsense) # Source prompts: 173 - # Classes: 8 Zero-Shot Feat. Ext. PatchscopeLogistic Reg. Probe (j) Object Superclass 0510152025 Source Layer ( ) 0 10 20 30 40 50 60 70 Attribute Extraction Acc. Attribute Extraction Acc. fruit inside color (commonsense) # Source prompts: 39 - # Classes: 3 Zero-Shot Feat. Ext. PatchscopeLogistic Reg. Probe (k) Fruit Inside Color 0510152025 Source Layer ( ) 5 10 15 20 25 30 Attribute Extraction Acc. Attribute Extraction Acc. star constellation (factual) # Source prompts: 56 - # Classes: 10 Zero-Shot Feat. Ext. PatchscopeLogistic Reg. Probe (l) Star Constellation 0510152025 Source Layer ( ) 0 10 20 30 40 Attribute Extraction Acc. Attribute Extraction Acc. person father (factual) # Source prompts: 27 - # Classes: 7 Zero-Shot Feat. Ext. PatchscopeLogistic Reg. Probe (m) Person Father 0510152025 Source Layer ( ) 0 20 40 60 Attribute Extraction Acc. Attribute Extraction Acc. superhero archnemesis (factual) # Source prompts: 28 - # Classes: 7 Zero-Shot Feat. Ext. PatchscopeLogistic Reg. Probe (n) Superhero Archnemesis 0510152025 Source Layer ( ) 0 20 40 60 80 Attribute Extraction Acc. Attribute Extraction Acc. task done by person (commonsense) # Source prompts: 33 - # Classes: 10 Zero-Shot Feat. Ext. PatchscopeLogistic Reg. Probe (o) Task Done By Person 0510152025 Source Layer ( ) 0 10 20 30 40 50 Attribute Extraction Acc. Attribute Extraction Acc. work location (commonsense) # Source prompts: 19 - # Classes: 4 Zero-Shot Feat. Ext. PatchscopeLogistic Reg. Probe (p) Work Location 0510152025 Source Layer ( ) 0 10 20 30 40 50 Attribute Extraction Acc. Attribute Extraction Acc. superhero person (factual) # Source prompts: 45 - # Classes: 12 Zero-Shot Feat. Ext. PatchscopeLogistic Reg. Probe (q) Superhero Person 0510152025 Source Layer ( ) 0 20 40 60 80 Attribute Extraction Acc. Attribute Extraction Acc. country capital city (factual) # Source prompts: 94 - # Classes: 21 Zero-Shot Feat. Ext. PatchscopeLogistic Reg. Probe (r) Country Capital City 0510152025 Source Layer ( ) 0 20 40 60 80 Attribute Extraction Acc. Attribute Extraction Acc. country largest city (factual) # Source prompts: 91 - # Classes: 21 Zero-Shot Feat. Ext. PatchscopeLogistic Reg. Probe (s) Country Largest City Figure 6. Feature extraction accuracy with respect to source layer (ℓ) across various factual and commonsense reasoning tasks. The zero-shot feature extractionPatchscopeworks consistently better than the logistic regression probe in early layers, and mostly in mid layers. There is a decline inPatchscopeaccuracy in laterℓas the source representations shift toward next-token prediction. 19 Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models 02468101214161820222426 Source Layer ( ) 0 2 4 6 8 10 12 14 16 18 20 22 24 26 Target Layer ( * ) Attribute Extraction Acc. substance phase (commonsense) # Source prompts: 182 - # Classes: 3 0 10 20 30 40 50 60 70 (a) Substance Phase 02468101214161820222426 Source Layer ( ) 0 2 4 6 8 10 12 14 16 18 20 22 24 26 Target Layer ( * ) Attribute Extraction Acc. task done by tool (commonsense) # Source prompts: 54 - # Classes: 12 0 10 20 30 40 50 (b) Task Done By Tool 02468101214161820222426 Source Layer ( ) 0 2 4 6 8 10 12 14 16 18 20 22 24 26 Target Layer ( * ) Attribute Extraction Acc. country currency (factual) # Source prompts: 83 - # Classes: 14 0 5 10 15 20 25 30 35 40 (c) Country Currency 02468101214161820222426 Source Layer ( ) 0 2 4 6 8 10 12 14 16 18 20 22 24 26 Target Layer ( * ) Attribute Extraction Acc. food from country (factual) # Source prompts: 27 - # Classes: 10 0 10 20 30 40 50 60 70 (d) Food From Country 02468101214161820222426 Source Layer ( ) 0 2 4 6 8 10 12 14 16 18 20 22 24 26 Target Layer ( * ) Attribute Extraction Acc. company ceo (factual) # Source prompts: 50 - # Classes: 12 0 10 20 30 40 50 (e) Company CEO 02468101214161820222426 Source Layer ( ) 0 2 4 6 8 10 12 14 16 18 20 22 24 26 Target Layer ( * ) Attribute Extraction Acc. fruit outside color (commonsense) # Source prompts: 56 - # Classes: 6 0 10 20 30 40 50 (f) Fruit Outside Color 02468101214161820222426 Source Layer ( ) 0 2 4 6 8 10 12 14 16 18 20 22 24 26 Target Layer ( * ) Attribute Extraction Acc. product by company (factual) # Source prompts: 219 - # Classes: 11 0 5 10 15 20 25 30 35 40 (g) Product By Company 02468101214161820222426 Source Layer ( ) 0 2 4 6 8 10 12 14 16 18 20 22 24 26 Target Layer ( * ) Attribute Extraction Acc. person plays pro sport (factual) # Source prompts: 131 - # Classes: 5 0 5 10 15 20 25 30 (h) Person Plays Pro Sport 02468101214161820222426 Source Layer ( ) 0 2 4 6 8 10 12 14 16 18 20 22 24 26 Target Layer ( * ) Attribute Extraction Acc. person plays position in sport (factual) # Source prompts: 383 - # Classes: 7 0 10 20 30 40 50 (i) Person Plays Position in Sport 02468101214161820222426 Source Layer ( ) 0 2 4 6 8 10 12 14 16 18 20 22 24 26 Target Layer ( * ) Attribute Extraction Acc. object superclass (commonsense) # Source prompts: 173 - # Classes: 8 0 5 10 15 20 25 30 (j) Object Superclass 02468101214161820222426 Source Layer ( ) 0 2 4 6 8 10 12 14 16 18 20 22 24 26 Target Layer ( * ) Attribute Extraction Acc. fruit inside color (commonsense) # Source prompts: 39 - # Classes: 3 0 10 20 30 40 50 (k) Fruit Inside Color 02468101214161820222426 Source Layer ( ) 0 2 4 6 8 10 12 14 16 18 20 22 24 26 Target Layer ( * ) Attribute Extraction Acc. star constellation (factual) # Source prompts: 56 - # Classes: 10 0 2 4 6 8 10 12 14 16 (l) Star Constellation 02468101214161820222426 Source Layer ( ) 0 2 4 6 8 10 12 14 16 18 20 22 24 26 Target Layer ( * ) Attribute Extraction Acc. person father (factual) # Source prompts: 27 - # Classes: 7 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 20.0 (m) Person Father 02468101214161820222426 Source Layer ( ) 0 2 4 6 8 10 12 14 16 18 20 22 24 26 Target Layer ( * ) Attribute Extraction Acc. superhero archnemesis (factual) # Source prompts: 28 - # Classes: 7 0 10 20 30 40 50 (n) Superhero Archnemesis 02468101214161820222426 Source Layer ( ) 0 2 4 6 8 10 12 14 16 18 20 22 24 26 Target Layer ( * ) Attribute Extraction Acc. task done by person (commonsense) # Source prompts: 33 - # Classes: 10 0 10 20 30 40 50 60 70 (o) Task Done By Person 02468101214161820222426 Source Layer ( ) 0 2 4 6 8 10 12 14 16 18 20 22 24 26 Target Layer ( * ) Attribute Extraction Acc. work location (commonsense) # Source prompts: 19 - # Classes: 4 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 20.0 (p) Work Location 02468101214161820222426 Source Layer ( ) 0 2 4 6 8 10 12 14 16 18 20 22 24 26 Target Layer ( * ) Attribute Extraction Acc. superhero person (factual) # Source prompts: 45 - # Classes: 12 0 5 10 15 20 25 30 (q) Superhero Person 02468101214161820222426 Source Layer ( ) 0 2 4 6 8 10 12 14 16 18 20 22 24 26 Target Layer ( * ) Attribute Extraction Acc. country capital city (factual) # Source prompts: 94 - # Classes: 21 0 10 20 30 40 50 60 (r) Country Capital City 02468101214161820222426 Source Layer ( ) 0 2 4 6 8 10 12 14 16 18 20 22 24 26 Target Layer ( * ) Attribute Extraction Acc. country largest city (factual) # Source prompts: 91 - # Classes: 21 0 10 20 30 40 50 60 (s) Country Largest City Figure 7. The interaction between source and target layers in the zero-shot feature extractionPatchscopeacross various factual and commonsense reasoning tasks. Each cell(ℓ,ℓ ∗ )in the heatmap shows the attribute extraction success rate where source and target layers are fixed toℓandℓ ∗ , respectively. Particularly, there is a higher success rate in the lower left quadrants, representing early to mid source and target layer combinations. The right half of the heatmaps shows late source layers, where the source representation has shifted toward next-token prediction, leading to a lower success rate in attribute extraction. The top half of the heatmaps shows late target layers. When the accuracy is a function of more than a single next-token, the placeholder token representation still remains in the early layers, leading to a lower attribute extraction rate. 20 Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models 02468 layer 0.0 0.1 0.2 0.3 0.4 Rouge1 M 7B, M* 7B M 13B, M* 13B M 7B, M* 13B popular entities rare entities (a) Vicuna 02468 layer 0.0 0.1 0.2 0.3 Rouge1 M 6.9B, M* 6.9B M 12B, M* 12B M 6.9B, M* 12B popular entities rare entities (b) Pythia Figure 8.Rouge1 scores of the generated descriptions against descriptions from Wikipedia. 02468 layer 0.0 0.1 0.2 0.3 0.4 0.5 0.6 SBERT M 7B, M* 7B M 13B, M* 13B M 7B, M* 13B popular entities rare entities (a) Vicuna 02468 layer 0.0 0.1 0.2 0.3 0.4 0.5 0.6 SBERT M 6.9B, M* 6.9B M 12B, M* 12B M 6.9B, M* 12B popular entities rare entities (b) Pythia Figure 9.SBERT scores of the generated descriptions against descriptions from Wikipedia. 05101520253035 target layer 0 5 10 15 20 25 30 source layer 0.70.70.50.50.40.30.20.2 0.70.70.60.60.50.40.40.3 0.70.70.60.60.60.50.40.3 0.70.60.60.60.60.60.50.4 0.70.60.50.60.60.70.60.6 0.70.50.50.50.60.70.70.7 0.60.50.50.60.60.70.70.7 0.0 0.2 0.4 0.6 0.8 1.0 (a)Vicuna:M←7B,M ∗ ←13B 05101520253035 target layer 0 5 10 15 20 25 30 source layer 0.80.60.60.50.30.30.20.2 0.80.80.70.60.50.40.30.3 0.80.80.80.70.60.50.40.3 0.80.80.80.70.70.60.50.4 0.70.80.70.70.70.70.60.5 0.70.70.70.70.70.70.70.5 0.70.70.70.70.70.70.70.6 0.0 0.2 0.4 0.6 0.8 1.0 (b)Pythia:M←6.9B,M ∗ ←12B Figure 10.Precision@1 scores (↑is better) on next-token prediction estimation in Vicuna and Pythia with cross-modelPatchscopes as Vicuna 13B processes “Will Smith”, it goes from“Smithsonian Museum”to the“Smith rock band”to the actor and rapper,“Will Smith”. However, Pythia 12B starts with“Smith & Wesson weapon manufacturing company”before it resolves the entity as the American actor,“Will Smith”. We also observe another phenomenon which we refer to as placeholder contamination. It is the case where the remaining representation of the placeholder entity “x” in the early layers interferes with the model’s capability in generating descriptions for the patched token. For example, see Vicuna 13B response to “Paris Hilton” entity in Tab. 6. First, we see the gradual process of going from“Rigatoni”pasta, to“Hilton Hotels”, to the socialite“Paris Hilton”in layers 1-6. But in layers 7 and onward, the generation seems to describe the placeholder token “x”rather than the “Paris Hilton” entity: “Placeholder for a variable or concept”,“Variable representing any number of things or concepts”or“x is a placeholder”. For future, we would like to quantify these qualitative observations, study to what extend this contamination can be mitigated with a different placeholder choice, and why some models might be more susceptible to this contamination than others. 21 Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models 05101520253035 target layer 0 5 10 15 20 25 30 source layer 1.11.94.04.44.85.86.77.0 1.31.92.52.73.13.94.55.2 1.52.32.82.62.83.64.44.7 1.42.83.32.82.42.73.23.9 1.53.64.33.72.81.92.12.7 1.84.86.15.44.12.41.91.8 2.15.33.93.32.71.81.71.5 1 2 3 4 5 6 7 8 (a)Vicuna:M←7B,M ∗ ←13B 05101520253035 target layer 0 5 10 15 20 25 30 source layer 1.12.32.84.15.16.48.35.7 1.21.21.52.43.03.84.34.6 1.21.21.32.02.22.83.44.0 1.31.21.41.71.82.12.63.5 1.41.31.51.81.61.82.03.0 1.61.41.62.11.71.71.72.7 2.01.51.82.31.91.81.62.3 1 2 3 4 5 6 7 8 (b)Pythia:M←6.9B,M ∗ ←12B Figure 11.Surprisal scores (↑is better) on next-token prediction estimation in Vicuna and Pythia cross-modelPatchscopes. E. Additional Results on the Cross-Model Patching Experiment Fig. 10 depicts the Precision@1 scores for different combinations of source-target layers, showing that patching with a simple affinePatchscopeproves to be effective with precision of up to 0.7 and 0.8 for Vicuna and Pythia, respectively. Specifically, patching representations to an early layer of the larger model seems to be the most effective. Furthermore, it appears that there is a subtle matching among some layers of the models, with the diagonal consistently exhibiting higher values. As shown in Fig. 11, similar trends are observed in the surprisal results for both models. Overall, these results show that, whenM ∗ andMare from the same model family, it is possible to leverageM ∗ for decoding information from the representations ofM. Notably, our findings extend observations by Csisz ́ arik et al. (2021) from patching representations across deep convolutional neural networks with the same architecture but different initializations. F. More Details on the Multi-Hop Reasoning Experiment DataBuilding on Hernandez et al. (2023b), we systematically generate all valid multi-hop factual and commonsense reasoning queries where the objectω 1 in the first tripletτ 1 is equal to the subjectσ 2 in the second tripletτ 2 . This process yields 1,104 multi-hop reasoning samples. We then filter examples to the ones where the model accurately represents both triplets independently. Tab. 8 summarizes information about the samples where Vicuna (13B) correctly represents each reasoning step. Out of 1,104 samples that require two steps of commonsense or factual reasoning, 46 satisfy the above criteria, with 8 unique relation combinations. More Detailed ResultsFig. 12 shows the interaction betweenℓandℓ ∗ and how it affects the success rate. Patching representations from most source layersℓinto early-to-midℓ ∗ (6-16) is the most effective in making the right prediction. Our interpretation is that patching into lateℓ ∗ is not effective becauseρ 2 has already been processed and the result has been copied to the last position, therefore it cannot incorporate the proxy ofσ 2 . We also observe thatℓ ∗ ≤ℓis more successful on average. G. Compute Resources All experiments were conducted using A100 80GB GPUs, with the exception of experiments with GPT-J (Wang & Komatsuzaki, 2021) where we used A100 40GB GPUs. 22 Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models Table 6.Additional qualitative examples illustrating entity resolution inVicuna 13B. The expressive generations show that as we go through the layers, more tokens from the context get integrated into the current representation, referred to as “Tokens Covered”. Here, M←Vicuna 13B,M ∗ ←M,ℓ ∗ ←ℓ. STokens CoveredℓGeneration Notes/Entity Described by the Generation "Will Smith" "Smith"1sonian: National museum and research complex in Washington, D.C.Smithsonian Museum "Smith"2sonas: American rock band An amalgam ofSmithband and SONA band "Will Smith"3: American actor and rapperWill Smith "Queen Victoria" "Victoria"1s Secret: American multinational women’s lingerie, beauty and body-care products companyVictoria’s Secret "Victoria"2s Secret: American lingerie and beauty products retailerVictoria’s Secret "Queen Victoria"3: Queen of the United Kingdom of Great Britain and Ireland from 1837 to 1901Queen Victoria "Queen Victoria"4-8: Queen of the United Kingdom of Great Britain and Ireland from 1837 until her death in 1901Queen Victoria "Queen Victoria"9: Queen of England (reigned from 1837-1901)Queen Victoria "Queen Victoria"10: 19th century British queen who ruled from 1837-1901Queen Victoria "Paris Hilton" "ton"1y: a type of pastaRigatoni/Rigatony/Rigaton ́ e "Hilton"2Hotels: American multinational hospitality companyHiltonHotel "Hilton"3: Hotel chain founded by Conrad HiltonHiltonHotel "Hilton"4: Hotel and resorts companyHiltonHotel "Paris Hilton"5-6: Socialite and television personalityParis Hilton N/A7: Placeholder for a variable or conceptPlaceholder contamination N/A8-9: Variable representing any number of things or conceptsPlaceholder contamination N/A10: x is a placeholderPlaceholder contamination "James Bond" "Bond"1: Unit of measurement for the strength of a liquidBondUnit "James Bond"2 : James Bond, the fictional British secret agent created by novelist Ian Fleming and portrayed by actor Daniel Craig in the 2006 film ”Casino Royale,” James Bond "James Bond"3-5: Fictional British secret agentJames Bond "James Bond"6: Code name for fictional British secret agent James BondJames Bond "James Bond"7: Code name for a fictional British secret agent created by novelist Ian FlemingJames Bond "James Bond"8 : Subject of a fictional British secret agent, created by novelist Ian Fleming and portrayed in a series of films. James Bond James Bond9: Subject of the 007 novels and filmsJames Bond N/A10: Variable representing the number of the film in the James Bond seriesPlaceholder contamination "Dwayne Johnson" "Johnson"1-2 & Johnson: American multinational corporation in the fields of pharmaceuticals, medical devices, and consumer packaged goods Johnson& Johnson "Dwayne Johnson"3-4: American actor and professional wrestler, also known as ”The Rock”Dwayne Johnson "Dwayne Johnson"5: Also known as ”The Rock,” American actor and professional wrestlerDwayne Johnson "Dwayne Johnson"6: Former professional wrestler and actorDwayne Johnson N/A7: Abbreviation for the United StatesPlaceholder contamination N/A8: Variable representing any number of things or conceptsPlaceholder contamination N/A9: Variable representing any number of other options.Placeholder contamination N/A10: Variable representing any number of other options or possibilities.Placeholder contamination 23 Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models Table 7.Additional qualitative examples illustrating entity resolution inPythia 12B. The expressive generations show that as we go through the layers, more tokens from the context get integrated into the current representation, referred to as “Tokens Covered”. Here, M←Pythia 12B,M ∗ ←M,ℓ ∗ ←ℓ. STokens CoveredℓGenerationNotes/Entity Described by the Generation "Will Smith" "Smith"1& Wesson: American firearms manufacturerSmith& Wesson "Will Smith"2: American actorWill Smith S ← "Alexander the Great" "Great"1Britain: Country in the European UnionGreatBritain "the Great"2Wall Street Crash of 1929: Financial crisis in the United Statesthe GreatDepression "the Great"3Wall Street Bubble: The Great Depressionthe GreatDepression "the Great"4Wall Street: Wall Street in New York City Wall Street (related to the Great Depression) "Alexander the Great"5: Ancient Greek ruler, and the first to rule all of the then known worldAlexander the Great "George Washington" "Washington"1: Capital of the United StatesWashingtonD.C. "Washington"2: American capital cityWashingtonD.C. "George Washington"3: American presidentGeorge Washington "George Washington"4-6: American revolutionaryGeorge Washington "George Washington"7: American presidentGeorge Washington "George Washington"8: George WashingtonGeorge Washington "George Washington"9: George Washington, the first President of the United StatesGeorge Washington "George Washington"10 : George Washington, Thomas Jefferson, Abraham Lincoln, and Theodore Roosevelt: American presidents George Washington "Titanic" "c"1loud: Apple’s cloud-based storage servicecloud "Titanic"2: Sinking ocean linerTitanic "Titanic"3: The ship that sank in the Atlantic Ocean in 1912Titanic "Titanic"4-6: British passenger shipTitanic "Titanic"7-9: The ship that sankTitanic "Titanic"10: Titanic, Titanic: British luxury passenger linerTitanic "Saturday Night Live" "Live"1Nation: American concert promoterLiveNation "Live"2Nation: American television networkLiveNation "Saturday Night Live"3: American television programSaturday Night Live "Saturday Night Live"4-10: American television comedy showSaturday Night Live "Nineteen Eighty-Four" "Four"1Seasons: American hotel chainFourSeasons Hotel "Four"2Seasons: The four seasons of the yearFourSeasons (of the year) "ighty-Four"3Lions: Chinese professional football clubCangzhou MightyLions F.C. "Nineteen Eighty-Four"4: Number of the novel 1984 by George OrwellNineteen Eighty-Four(Novel) "Nineteen Eighty-Four"5: George Orwell’s novel, and the 1984 film.Nineteen Eighty-Four(Novel & Film) "Nineteen Eighty-Four"6: George Orwell’s novelNineteen Eighty-Four(Novel) "Nineteen Eighty-Four"7: George Orwell: British novelistNineteen Eighty-Four(Novel) "Nineteen Eighty-Four"8: 1984 novel by George OrwellNineteen Eighty-Four(Novel) "Nineteen Eighty-Four"9-10 : 1984 novel by George Orwell, and the 1984 film adaptation directed by Michael Radford. Nineteen Eighty-Four(Novel & Film) 24 Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models Table 8.Sample statistics for the multi-hop reasoning experiment whereMcorrectly represents bothτ 1 andτ 2 . ρ 1 ρ 2 # Samples Company CEOPerson Father4 Food from CountryCountry Capital City10 Food from CountryCountry Currency3 Food from CountryCountry Language9 Food from CountryCountry Largest City11 Person FatherPerson Father1 Person FatherPerson Mother1 Product by CompanyCompany CEO7 Total46 02468101214161820222426283032343638 Source Layer ( ) 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 Target Layer ( * ) Self-correction in Multi-hop Reasoning # samples: 73600 0.00 0.02 0.04 0.06 0.08 0.10 0.12 Figure 12.The interaction between source (ℓ) and target (ℓ ∗ ) layers in correcting multi-hop reasoning errors. The majority of success cases correspond to early-to-midℓ ∗ . We observe higher cumulative success rate in the lower right triangle which corresponds toℓ ∗ ≤ℓ. 25