Paper deep dive
Eliciting Latent Predictions from Transformers with the Tuned Lens
Nora Belrose, Igor Ostrovsky, Lev McKinney, Zach Furman, Logan Smith, Danny Halawi, Stella Biderman, Jacob Steinhardt
Models: BLOOM, GPT-2, GPT-Neo, GPT-Neo-20B, Pythia (70M to 6.9B)
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/12/2026, 8:22:47 PM
Summary
The paper introduces the 'tuned lens', a method for analyzing transformer language models by training affine probes (translators) at each layer to decode hidden states into vocabulary distributions. This approach improves upon the 'logit lens' by reducing bias and perplexity, enabling more reliable interpretation of latent prediction trajectories and detection of malicious inputs.
Entities (5)
Relation Signals (3)
Tuned Lens â improvesupon â Logit Lens
confidence 95% ¡ To address the shortcomings of the logit lens, we introduce the tuned lens.
Tuned Lens â uses â Translator
confidence 95% ¡ We train affine transformations, one for each layer of the network... We call these transformations translators
Muon â acceleratestrainingof â Tuned Lens
confidence 90% ¡ Strikingly, we find that using Muon instead of SGD for training the tuned lens dramatically accelerates training
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We analyze transformers from the perspective of iterative inference, seeking to understand how model predictions are refined layer by layer. To do so, we train an affine probe for each block in a frozen pretrained model, making it possible to decode every hidden state into a distribution over the vocabulary. Our method, the tuned lens, is a refinement of the earlier "logit lens" technique, which yielded useful insights but is often brittle. We test our method on various autoregressive language models with up to 20B parameters, showing it to be more predictive, reliable and unbiased than the logit lens. With causal experiments, we show the tuned lens uses similar features to the model itself. We also find the trajectory of latent predictions can be used to detect malicious inputs with high accuracy. All code needed to reproduce our results can be found at this https URL.
Tags
Links
Trouble viewing inline? Open PDF directly â
Full Text
90,078 characters extracted from source content.
Expand or collapse full text
Eliciting Latent Predictions from Transformers with the Tuned Lens Nora Belrose 1 2 Igor Ostrovsky 1 Lev McKinney 3 2 Zach Furman 1 4 Logan Smith 1 Danny Halawi 1 Stella Biderman 1 Jacob Steinhardt 5 Abstract We analyze transformers from the perspective of iterative inference, seeking to understand how model predictions are refined layer by layer. To do so, we train an affine probe for each block in a frozen pretrained model, making it possible to decode every hidden state into a distribution over the vocabulary. Our method, the tuned lens, is a refinement of the earlier âlogit lensâ technique, which yielded useful insights but is often brittle. We test our method on various autoregressive language models with up to 20B parameters, showing it to be more predictive, reliable and unbiased than the logit lens. With causal ex- periments, we show the tuned lens uses sim- ilar features to the model itself.We also find the trajectory of latent predictions can be used to detect malicious inputs with high ac- curacy. All code needed to reproduce our re- sults can be found athttps://github.com/ AlignmentResearch/tuned-lens. 1. Introduction The impressive performance of transformers in natural lan- guage processing (Brown et al., 2020) and computer vision (Dosovitskiy et al., 2020) suggests that their internal repre- sentations have rich structure worthy of scientific investiga- tion. One common approach is to train classifiers to extract specific concepts from hidden states, like part-of-speech and syntactic structure (Hewitt and Manning, 2019; Tucker et al., 2021; Li et al., 2022). In this work, we instead examine transformer representa- tions from the perspective of iterative inference (Jastrz ̨ebski et al., 2017). Specifically, we view each layer in a trans- former language model as performing an incremental update to a latent prediction of the next token. 1 We decode these la- tent predictions through early exiting, converting the hidden 1 Eleuther AI 2 FAR AI 3 University of Toronto 4 Boston University 5 UC Berkeley. Correspondence to: Nora Belrose <nora@eleuther.ai>. 1 See Appendix C for evidence supporting this view, including novel empirical results of our own. """""""" âĢŒ."âĢŒ."âĢŒ."âĢŒ."âĢŒ."âĢŒ."âĢŒ."âĢŒ." âĢŒ."âĢŒ."âĢŒ."âĢŒ."âĢŒ."âĢŒ."âĢŒ."âĢŒ." âĢŒ."âĢŒ."âĢŒ."âĢŒ."âĢŒ."âĢŒ."âĢŒ."âĢŒ." âĢŒ."âĢŒ."âĢŒ."âĢŒ."âĢŒ."âĢŒ."âĢŒ."âĢŒ." âĢŒ."âĢŒ."âĢŒ."âĢŒ."âĢŒ."âĢŒ."âĢŒ."âĢŒ." âĢŒ."âĢŒ."âĢŒ."âĢŒ."âĢŒ."âĢŒ."âĢŒ."âĢŒ." modelmodelthatthatcalledâĢŒ."âĢŒ.", modelmodelcalledcalledcalled**-, modelmodelthatthatcalled**-, modelyetthatthatcalledattentionformer, modelrecurrentarchitecturethatcalledattentionformer, newsimple network architecture , the Trans former input 3 6 9 12 15 18 21 24 27 30 output """-"" ,xto,andsame_I atoand andandthatforwhichresult-. method-thatthatwhichso-, methodmethodthatthatwhichso-, modelmodelthatthattheso-, modelmodelthatthattheâĢ-, modelmodelthatthatcalled**-, modelmodelthatthatcalled**-, modelmodelarchitecturethatcalledattentionformer, modelrecurrentarchitecturethatcalledattentionformer, newsimple network architecture , the Trans former input 3 6 9 12 15 18 21 24 27 30 output 00.20.40.60.81 Probability 00.20.40.60.81 Probability Logit Lens (theirs) Tuned Lens (ours) Input token L a y e r Figure 1.Comparison of our method, the tuned lens (bottom), with the âlogit lensâ (top) for GPT-Neo-2.7B prompted with an except from the abstract of Vaswani et al. (2017). Each cell shows the top-1 token predicted by the model at the given layer and token index. The logit lens fails to elicit interpretable predictions before layer 21, but our method succeeds. state at each intermediate layer into a distribution over the vocabulary. This yields a sequence of distributions we call the prediction trajectory, which exhibits a strong tendency to converge smoothly to the final output distribution, with each successive layer achieving lower perplexity. We build on the âlogit lensâ (nostalgebraist, 2020), an early exiting technique that directly decodes hidden states into vocabulary space using the modelâs pretrained unembed- ding matrix. We find the logit lens to be unreliable (Sec- tion 2), failing to elicit plausible predictions for models like BLOOM (Scao et al., 2022) and GPT Neo (Black et al., 2021). Even when the logit lens appears to work, its outputs are hard to interpret due to representational drift: features arXiv:2303.08112v6 [cs.LG] 11 Nov 2025 Eliciting Latent Predictions from Transformers with the Tuned Lens may be represented differently at different layers of the net- work. Other early exiting procedures also exist (Schuster et al., 2022), but require modifying the training process, and so canât be used to analyze pretrained models. To address the shortcomings of the logit lens, we introduce the tuned lens. We trainLaffine transformations, one for each layer of the network, with a distillation loss: transform the hidden state so that its image under the unembedding matches the final layer logits as closely as possible. We call these transformations translators because they âtrans- lateâ representations from the basis used at one layer of the network to the basis expected at the final layer. Compos- ing a translator with the pretrained unembedding yields a probe (Alain and Bengio, 2016) that maps a hidden state to a distribution over the vocabulary. We find that tuned lens predictions have substantially lower perplexity than logit lens predictions, and are more represen- tative of the final layer distribution. We also show that the features most influential on the tuned lens output are also influential on the model itself (Section 4). To do so, we intro- duce a novel algorithm called causal basis extraction (CBE) and use it to locate the directions in the residual stream with the highest influence on the tuned lens. We then ablate these directions in the corresponding model hidden states, and find that these features tend to be disproportionately influential on the model output. We use the tuned lens to gain qualitative insight into the computational process of pretrained language models, by examining how their latent predictions evolve during a for- ward pass (Figure 1, Section B) Finally, we apply the tuned lens in several ways: we extend the results of Halawi et al. (2023) to new models (Section 5.2), we find that tuned lens prediction trajectories can be used to detect prompt injection attacks (Perez and Ribeiro, 2022) often with near-perfect accuracy (Section 5.3), and find that data points which require many training steps to learn also tend to be classified in later layers (Section 5.4). 2. The Logit Lens The logit lens was introduced by nostalgebraist (2020), who found that when the hidden states at each layer of GPT-2 (Radford et al., 2019) are decoded with the unembedding matrix, the resulting distributions converge roughly mono- tonically to the final answer. More recently it has been used by Halawi et al. (2023) to understand how transformers process few-shot demonstrations, and by Dar et al. (2022), Geva et al. (2022), and Millidge and Black (2022) to directly interpret transformer weight matrices. The method. Consider a pre-LayerNorm transformer 2 M. 2 Both the logit lens and the tuned lens are designed primarily embed h 0 tokens residual layer translator unembed logits + residual layer residual layer h 2 + h 3 h 1 + Figure 2.The tuned lens takes the hidden state at an intermediate layer (e.g.h 1 above), and applies a learned affine transformation (the translator). We then convert the hidden state into logits with the unembedding layer. Weâl decomposeMinto two âhalves,âM â¤â andM >â . The functionM â¤â consists of the layers ofMup to and including layerâ, and it maps the input space to hidden states. Conversely, the functionM >â consists of the layers ofM after â, which map hidden states to logits. The transformer layer at indexâupdates the representation as follows: h â+1 = h â + F â (h â ),(1) whereF â is the residual output of layerâ. Applying Equa- tion 1 recursively, the output logitsM >â can be written as a function of an arbitrary hidden state h â at layer â: M >â (h â ) = LayerNorm h h â + L X â Ⲡ=â F â Ⲡ(h â Ⲡ) | z residual update i W U . (2) The logit lens consists of setting the residuals to zero: LogitLens(h â ) = LayerNorm[h â ]W U .(3) While this simple form of the logit lens works reasonably well for GPT-2, nostalgebraist (2021) found that it fails to extract useful information from the GPT-Neo family of mod- els (Black et al., 2021). In an attempt to make the method for the pre-LN architecture, which is more unambiguously iterative. Luckily pre-LN is by far more common than post-LN among state- of-the-art models. See Zhang and He (2020, Appendix C) for more discussion. Eliciting Latent Predictions from Transformers with the Tuned Lens input51015202530 0 1 2 3 4 5 Logit lens Tuned lens Layer KL (bits) Figure 3.Bias of logit lens and tuned lens outputs relative to the final layer output for GPT-Neo-2.7B. The last transformer layer is included for both probes. Unlike the tuned lens, the logit lens is systematically biased toward some vocabulary items over others until the very end of the network. work better for GPT-Neo, they introduce an extension which retains the last transformer layer, yielding: LogitLens ext (h â ) = LayerNorm[h â + F L (h â )]W U (4) This extension is only partially successful at recovering meaningful results; see Figure 1 (top) for an example. Unreliability. Beyond GPT-Neo, the logit lens struggles to elicit predictions from several other models released since its introduction, such as BLOOM (Scao et al., 2022) and OPT 125M (Zhang et al., 2022) (Figure 14). Moreover, the type of information extracted by the logit lens varies both from model to model and from layer to layer, making it difficult to interpret. For example, we find that for BLOOM and OPT 125M, the top 1 prediction of the logit lens is often the input token, rather than any plausible continuation token, in more than half the layers (Figure 18). Bias. Even when the logit lens is useful, we find that it is a biased estimator of the modelâs final output: it system- atically puts more probability mass on certain vocabulary items than the final layer does. This is concerning because it suggests we canât interpret the logit lens prediction trajectory as a belief updating in response to new evidence. The beliefs of a rational agent should not update in an easily predictable direction over time (Yudkowsky, 2007), since predictable updates can be exploited via Dutch books (Ramsey, 1926; De Finetti et al., 1937; Hacking, 1967; Garrabrant et al., 2016). Biased logit lens outputs are trivially exploitable once the direction of bias is known: one could simply âbetâ against the logit lens at layerâ < Lthat the next token will be one of the tokens that it systematically downweights, and make unbounded profit in expectation. Letxbe a sequence of tokens sampled from a datasetD, and letx <t refer to the tokens preceding positiontin the input 5 101520 0 20 40 60 input 5 101520 0 2 4 6 8 Logit lens Tuned lens Final logits w/o final layerw/ final layer Layer bits per byte Figure 4.Perplexity of predictions elicited from BLOOM 560M under four conditions: the logit lens (red squares) and the tuned lens (blue circles), and including (left) and excluding (right) the final transformer layer from the probe. We find that tuned lens predictions have substantially lower perplexity whether or not the final layer is included, showing it is an independent and comple- mentary proposal. sequence. Letq â (¡|x <t )be the logit lens distribution at layerâfor positiont, and letp(¡|x <t )be the final layer distribution for position t. We definep(v|x)to be the probability assigned to a vocab- ulary itemvin a sequencex, averaged over all positions 1...T : p(v|x) def = 1 T T X t=1 p(v|x <t ).(5) Slightly abusing terminology, we say thatq â is an âunbiased estimatorâ ofpif, for every itemvin the vocabulary, the probability assigned tovaveraged across all tokens in the dataset is the same: E xâD h q â (v|x) i = E xâD h p(v|x) i q â (v) = p(v) âv âV, V ="aardvark",... (6) In practice, Equation 6 will never hold exactly. We measure the degree of bias using the KL divergence between the marginal distributions, D KL (p|| q â ). In Figure 3 we evaluate the bias for each layer of GPT-Neo- 2.7B. We find the bias of the logit lens can be quite large: around 4 to 5 bits for most layers. As a point of comparison, the bias of Pythia 160Mâs final layer distribution relative to that of its larger cousin, Pythia 12B, is just 0.0068 bits. 3. The Tuned Lens One problem with the logit lens is that, if transformer layers learn to output residuals that are far from zero on average, the input toLogitLensmay be out-of-distribution and yield nonsensical results. In other words, the choice of zero as Eliciting Latent Predictions from Transformers with the Tuned Lens input 5 10152025303540 0 1 2 3 4 5 6 input 5 10152025303540 0 1 2 3 4 5 6 Model size 70M 160M 410M 1.4B 2.8B 6.9B 12B 20B (NeoX) Logit lens (baseline)Tuned lens (ours) Layer bits per byte Figure 5.Perplexity of latent predictions elicited by the logit lens (left) and the tuned lens (right) from Pythia and GPT-NeoX-20B, as a function of layer index and model size. Tuned lens predictions are uniformly lower perplexity and exhibit lower variance across independently trained models. a replacement value is somewhat arbitraryâ the network might learn to rely on P L â Ⲡ=â E [F â (h â )] as a bias term. Our first change to the method is to replace the summed residuals with a learnable constant valueb â instead of zero: LogitLens debiased â (h â ) = LogitLens(h â + b â )(7) Representation drift. Another issue with the logit lens is that transformer hidden states often contain a small num- ber of very high variance dimensions, and these ârogue dimensionsâ (Timkey and van Schijndel, 2021) tend to be distributed unevenly across layers; see Figure 6 (top) for an example. Ablating an outlier direction can drastically harm performance (Kovaleva et al., 2021), so ifLogitLensrelies on the presence or absence of particular outlier dimensions, the perplexity of logit lens predictions might be spuriously high. Even when controlling for rogue dimensions, we observe a strong tendency for the covariance matrices of hidden states at different layers to drift apart as the number of layers sep- arating them increases (Figure 6, bottom). The covariance at the final layer often changes sharply relative to previous layers, suggesting the logit lens might âmisinterpretâ earlier representations. One simple, general way to correct for drifting covariance is to introduce a learnable change of basis matrixA â , which learns to map from the output space of layerâto the input space of the final layer. We have now arrived at the tuned lens formula, featuring a learned affine transformation for each layer: TunedLens â (h â ) = LogitLens(A â h â + b â )(8) We refer to (A â ,b â ) as the translator for layer â. Loss function. We train the translators to minimize KL between the tuned lens logits and the final layer logits: argmin E x h D KL (f >â (h â )|| TunedLens k (h â )) i (9) wheref >â (h â )refers to the rest of the transformer after layerâ. This can be viewed as a distillation loss, using the final layer distribution as a soft label (Sanh et al., 2019). It ensures that the probes are not incentivized to learn extra information over and above what the model has learned, which can become a problem when training probes with ground truth labels (Hewitt and Liang, 2019). Implementation details. When readily available, we train translators on a slice of the validation set used during pre- training, and use a separate slice for evaluation. Since BLOOM and GPT-2 do not have publicly available vali- dation sets, we use the Pile validation set (Gao et al., 2020; Biderman et al., 2022). The OPT validation set is also not publicly available, but a member of the OPT team helped us train a tuned lens on the OPT validation set. Documents are concatenated and split into uniform chunks of length 2048. We evaluate all models on a random sample of 16.4M tokens from their respective pretraining validation sets. We leave out the final transformer layer for GPT-2 (Radford et al., 2019), GPT-NeoX-20B (Black et al., 2022), OPT (Zhang et al., 2022), and Pythia (Biderman et al., 2023), and include it for GPT-Neo (Black et al., 2021). We evaluate BLOOM (Scao et al., 2022) under both conditions in Figure 4. In our initial experiments, we used SGD with Nesterov momentum, with a linear learning rate decay schedule over Eliciting Latent Predictions from Transformers with the Tuned Lens 05101520253035 35 30 25 20 15 10 5 0 05101520253035 35 30 25 20 15 10 5 0 0.2 0.4 0.6 0.8 1 Cosine similarity (F r obenius) all principal components w/o top 2 components Layer Layer Figure 6.Pairwise similarities of hidden state covariance matrices across layers of Pythia 12B. Layer 4 introduces two outlier di- mensions which dominate the covariance; removing them reveals smooth representational drift with depth. To control for varying hidden state norms, we measure the Frobenius cosine similarity, or â¨A,B⊠F âĽA⼠F âĽB⼠F for two matrices A and B. 250 training steps. We used a base learning rate of 1.0, or 0.25 when keeping the final transformer layer, and clip gradients to a norm of 1. We accumulated gradients as necessary to achieve a total batch size of2 18 tokens per optimizer step. We initialized all translators to the identity transform, and use a weight decay of 10 â3 . Muon optimizer. After the initial publication of this pa- per, Jordan et al. (2024) introduced the Muon optimizer. Using a fast orthogonalization algorithm, Muon attempts to maximize the effective rank (Roy and Vetterli, 2007) of the update applied to each matrix-valued parameter, thereby counteracting the well-known tendency of neural network gradients to be rank-deficient. Conceptually, Muon is the opposite of LoRA (Hu et al., 2022), a popular parameter- efficient finetuning method that constrains parameter up- dates to be low rank. Liu et al. (2025) find that Muon accelerates the training of modern large language models compared to AdamW. Strikingly, we find that using Muon instead of SGD for training the tuned lens dramatically accelerates training, al- lowing us to achieve much lower KL divergence losses than were possible in our initial experiments. Essentially, all of the tuned lenses trained and evaluated in the earlier versions of this paper were severely undertrained. The weight ma- trices of Muon-trained lenses have Frobenius norms that are often several times larger than SGD-trained ones, show- ing that they are much further from the logit lens than our original lensesâ they are much better âtuned,â as it were. We opted not to re-do all our experiments with tuned lenses trained using Muon, but we encourage practitioners to use Muon in future experiments. Results. We plot tuned lens perplexity as a function of depth for the Pythia models and GPT-NeoX-20B in Figure 5 3 ; results for other model families can be found in Appendix A. We find that the tuned lens resolves the problems with the logit lens discussed in Section 2: it has significantly lower bias (Figure 3), and much lower perplexity than the logit lens across the board (Figure 5, Appendix A). Transferability across layers. We find that tuned lens trans- lators can usually zero-shot transfer to nearby layers with only a modest increase in perplexity. Specifically, we define the transfer penalty from layerâtoâ Ⲡto be the expected increase in cross-entropy loss when evaluating the tuned lens translator trained for layer â on layer â Ⲡ. We report transfer penalties for the largest Pythia model in Figure 7. Overall, transfer penalties are quite low, especially for nearby layers (entries near the diagonal in Figure 7). Comparing to the two plots in Figure 6, we notice that transfer penalties are strongly negatively correlated with co- variance similarity (SpearmanĎ =â0.78). Unlike Figure 6, however, Figure 7 is not symmetric: transfer penalties are higher when training on a layer with the outlier dimensions (Layer 5 and later) and testing on a layer without them, than the reverse. Transfer to fine-tuned models. We find that lenses trained on a base model transfer well to fine-tuned versions of that base model, with no additional training of the lens. Trans- ferred lenses substantially outperform the Logit Lens and compare well with lenses trained specifically on the fine- tuned models. Here a transferred lens is one that makes use of the fine-tuned models unembedding, but simply copies 3 Pythia and GPT-NeoX-20B were trained using the same archi- tecture, data, and codebase (Andonian et al., 2021). While theyâre not officially the same model suite, theyâre more consistent than the OPT models. Eliciting Latent Predictions from Transformers with the Tuned Lens 05101520253035 35 30 25 20 15 10 5 0 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 b i t s p e r b y t e Test layer T r a i n l a y e r Figure 7.Transfer penalties for Pythia 12B. Each row corresponds to a single tuned lens probe trained on layerâ, and each column is a layerâ Ⲡon which probes are evaluated. Each cell shows the cross-entropy loss of probeâevaluated on layerâ Ⲡ, minus its on- distribution loss (so that the diagonal entries are identically zero). its affine translators from the lens trained on the base model. As an example of the transferability of lenses to fine-tuned models we use Vicuna 13B (Chiang et al., 2023), an open source instruct fine-tuned chat model based on LLaMA 13B (Touvron et al., 2023). In Figure 13, Section A, we compare the performance of a lens specifically trained on Vicuna to a tuned lens trained only on LLaMA. Both Tuned Lenses were trained using a subsample of the RedPajama dataset (Together, 2023), an open source replication of LLaMAâs training set. The performance of both lenses was then evaluated on the test set of Anthropicâs Helpful Harmless (Bai et al., 2022) conversation dataset and the RedPajama Dataset. On the RedPajama Dataset we find, at worst, a 0.3 bits per byte increase in KL divergence to the models final output. On the helpful harmless corpus we find no significant difference between the transferred and trained lenses. These results show that fine-tuning mini- mally effects the representations used by the tuned lens. This opens applications of the tuned lens for monitoring changes in the representations of a module during fine-tuning and minimizes the need for practitioners to train lenses on new fine-tuned models. Relation to model stitching. The tuned lens can be viewed as a way of âstitchingâ an intermediate layer directly onto the unembedding, with an affine transform in between to align the representations. The idea of model stitching was introduced by Lenc and Vedaldi (2015), who form a com- posite model out of two frozen pretrained modelsAandB, by connecting the bottom layers ofAto the top layers ofB. An affine transform suffices to stitch together independently trained models with minimal performance loss (Bansal et al., 2021; CsiszĂĄrik et al., 2021). The success of the tuned lens shows that model stitching works for different layers inside a single model as well. Benefits over traditional probing. Unlike Alain and Ben- gio (2016), who train early exiting probes for image clas- sifiers, we do not learn a new unembedding for each layer. This is important, since it allows us to shrink the size of each learned matrix from|V|ĂdtodĂd, where|V|ranges from 50K (GPT-2, Pythia) to over 250K (BLOOM). We observe empirically that training a new unembedding ma- trix requires considerably more training steps and a larger batch size than training a translator, and often converges to a worse perplexity. 4. Measuring Causal Fidelity Prior work has argued that interpretability hypotheses should be tested with causal experiments: an interpreta- tion of a neural network should make predictions about what will happen when we intervene on its weights or ac- tivations (Olah et al., 2020; Chan et al., 2022). This is es- pecially important for probing techniques, since itâs known that probes can learn to rely on spurious features unrelated to the modelâs performance (Hewitt and Liang, 2019; Be- linkov, 2022). To explore whether the tuned lens finds causally relevant features, we will assess two desired properties: 1. Latent directions that are important to the tuned lens should also be important to the final layer output. Con- cretely, if the tuned lens relies on a feature 4 v in the residual stream (its output changes significantly when we manipulatev) then the model output should also change a lot when we manipulate v. 2.These latent directions should be important in the same way for both the tuned lens and the model. Concretely, if we manipulate the hidden state so that the tuned lens changes in a certain way (e.g. doubling the probabil- ity assigned to âdogâ) then the model output should change similarly. We will call this property stimulus- response alignment. 4.1. Causal basis extraction To test Property 1, we first need to find the important di- rections for the tuned lens. Amnesic probing (Elazar et al., 4 For simplicity we assume the âfeatures as directionsâ hypoth- esis (Elhage et al., 2022), which defines a âfeatureâ to be the one-dimensional subspace spanned by a unit vectorv. Eliciting Latent Predictions from Transformers with the Tuned Lens 2021) provides one way to do thisâit seeks a direction whose erasure maximally degrades a modelâs accuracy. However, this only elicits a single important direction, whereas we would like to find many such directions. To do so, we borrow intuition from PCA, searching for additional directions that also degrade accuracy, but which are orthogo- nal to the original amnesic direction. This leads to a method that we call causal basis extraction (CBE), which finds the the principal features used by a model. More specifically, letfbe a function (such as the tuned lens) that maps latent vectorsh â R d to logitsy. Letr(h, v) be an erasure function which removes information along the span ofvfromx. In this work we user(h, v)is mean ablation, which setsâ¨r(h, v), vâŠto the mean value ofâ¨h, v⊠in the dataset (see Appendix D.1). We define the influenceĎ of a unit vectorvto be the expected KL divergence between the outputs of f before and after erasing v from h: Ď(v;f ) = E h h D KL (f (h)||f (r(h, v))) i (10) We seek to find an orthonormal basisB = (v 1 ,..., v k ) containing principal features off, ordered by a sequence of influencesÎŁ = (Ď 1 ,...,Ď k )for somek ⤠d. In each iteration we search for a featurev i of maximum influence that is orthogonal to all previous features v j : v i = argmax ||v|| 2 = 1 Ď(v;f ) s.t. â¨v, v j ⊠= 0, âj < i (11) With a perfect optimizer, the influence ofv i should decrease monotonically since the feasible region is strictly smaller with each successive iteration. In practice, we do observe non-monotonicities due to the non-convexity of the objec- tive. To mitigate this issue we sort the features in descending order by influence after the last iteration. Implementation details. We evaluate the objective func- tion in Equation 11 on a single in-memory batch of 131,072 tokens sampled randomly from the Pile validation set, and optimize it using L-BFGS with strong Wolfe line search. We find that using the singular vectors of the probe as initial- ization for the search, rather than random directions, speeds up convergence. Intervening on the model. If we apply causal basis ex- traction to the tuned lens at layerâ, we obtainkdirections v 1 ,...,v k that are important for the tuned lens. We next check that these are also important to the modelM. To do so, we first take an i.i.d. sample of input sequences xand feed them toM, storing the resulting hidden states M â¤â (x). 5 Then, for each vectorv i obtained from CBE, 5 See Section 2 for notation. 10Îź100Îź0.0010.010.11 2 5 0.01 2 5 0.1 2 5 1 2 Influence on lens (bits) Influence on model (bits) Ď = 0.89 Figure 8.Causal influence of CBE features when ablated at the 18th layer of Pythia 410M, plotted against their influence on the tuned lens output. Spearman Ď = 0.89. we record the causal effect of erasingv i on the output of M >â , E x h D KL (M(x)||M >â (r(M â¤â (x), v i )) i (12) where the erasure functionris applied to all positions in a sequence simultaneously. We likewise average the KL divergences across token positions. Results. We report the resulting causal influences for Pythia 410M,â = 18in Figure 8; results for all layers can be found in Figure 20 in the Appendix. In accordance with Property 1, there is a strong correlation between the causal influence of a feature on the tuned lens and its influence on the model (SpearmanĎ = 0.89). Im- portantly, we donât observe any features in the lower right corner of the plot (features that are influential in the tuned lens but not in the model). The model is somewhat more âcausally sensitiveâ than the tuned lens: even the least in- fluential features never have an influence under2Ă 10 â3 bits, leading to the âhockey stickâ shape in the LOWESS trendline. 4.2. Stimulus-response alignment We now turn to Property 2. Intuitively, for the interventions from Section 4.1, deleting an important directionv i should have the same effect on the modelâs output distributionp and the tuned lensâ output distribution q. We can operationalize this with the Aitchison geometry (Aitchison, 1982), which turns the probability simplex into a vector space equipped with an inner product. In order to downweight the influence of rare tokens, we use the Eliciting Latent Predictions from Transformers with the Tuned Lens 0246810 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Logit lens Tuned lens Layer Aitchison similarity Figure 9.Average stimulus-response alignment at each layer of Pythia 160M. Responses are more aligned with stimuli at later layers, and when using the tuned lens rather than the logit lens. weighted Aitchison inner product introduced by Egozcue and Pawlowsky-Glahn (2016), defined as â¨p 1 ,p 2 ⊠w = D X i=1 w i log p 1i g w (p 1 ) log p 2i g w (p 2 ) ,(13) wherewis a vector of positive weights, andg w (p)is the weighted geometric mean of the entries ofp. In our experi- ments, we use the final layer prediction distribution under the control condition to define w. We will also use the notion of âsubtractingâ distributions. In Aitchison geometry, addition and subtraction of distri- butions is done componentwise in log space, followed by renormalization: p 1 â p 2 = softmax log p 1 â log p 2 .(14) We say that distributions(p old ,p new )and(q old ,q new ) âmove in the same directionâ if and only if â¨p new â p old , q new â q old ⊠w > 0.(15) Measuring alignment. Letg : R d â R d be an arbitrary function for intervening on hidden states, and leth â be the hidden state at layerâon some inputx. Weâl define the stimulus to be the Aitchison difference between the tuned lens output before and after the intervention: S(h â ) = TunedLens â (g(h â ))â TunedLens â (h â ) (16) Analogously, the response will be defined as the Aitchison difference between the final layer output before and after the intervention: R(h â ) =M >â (g(h â ))âM >â (h â )(17) 152025 Layer 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy Accuracy for Direct Questions Lens Type Logit Lens Tuned Lens 152025 Layer 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy Accuracy for Standard Questions Lens Type Tuned Lens Logit Lens Figure 10.The tuned lens elicits secrets more effectively than the logit lens at early layers, then performs more poorly at later layers. Interestingly, on this task, the âdirectâ prompts yield lower elicita- tion accuracy than âstandardâ ones. Weâd like to control for the absolute magnitudes of the stimuli and the responses, so we use the Aitchison inner product to define a cosine similarity metric, which we call âAitchison similarity.â Then the stimulus-response alignment at layerâundergis simply the Aitchison similarity between the stimulus and response: sim(S(h â ), R(h â )) = â¨S(h â ), R(h â )⊠w âĽS(h â )⼠w âĽR(h â )⼠w (18) We propose to use CBE (Section 4.1) to define a ânaturalâ choice for the interventiong. Specifically, for each layerâ, we intervene on the subspace spanned byââs top 10 causal basis vectorsâ weâl call this the âprincipal subspaceââ us- ing a recently proposed method called resampling ablation (Chan et al., 2022). Given a hidden stateh â =M â¤â (x), resampling ablation re- places the principal subspace of h â with the corresponding subspace generated on a different inputx Ⲡselected uni- formly at random from the dataset. It then feeds this modi- fied hidden state Ě h â into the rest of the model, yielding the modified outputM >â ( Ě h â ). Intuitively, Ě h â should be rela- tively on-distribution because weâre using values generated ânaturallyâ by the model itself. Unlike in Section 4.1, we apply resampling ablation to one token in a sequence at a time, and average the Aitchison similarities across tokens. Results. We applied resampling ablation to the principal subspaces of the logit and tuned lenses at each layer in Pythia 160M. We report average stimulus-response align- ments in Figure 9. Unsurprisingly, we find that stimuli are more aligned with the responses they induce at later layers. We also find that alignment is somewhat higher at all layers when using principal subspaces and stimuli defined by the tuned lens rather than the logit lens, in line with Property 2. Eliciting Latent Predictions from Transformers with the Tuned Lens 0510 0.3 0.35 0.4 05101520 0.2 0.3 0.4 0.5 05101520 0.25 0.3 0.35 0.4 0.45 0102030 0.3 0.4 0.5 0102030 0.3 0.4 0.5 05101520 0.3 0.35 0.4 True Demos False Demos Random Baseline Neo (125M)Neo (1.3B) Neo (2.7B)OPT (1.3B) Pythia (12B)BLOOM (560M) Layer Calibrated Performance Figure 11.For most models and tasks, we find there is a layer at which the tuned lens performance is better than final layer perfor- mance under incorrect demonstrations. Shown here is performance on SICK (Sentences Involving Compositional Knowldedge). Un- like the logit lens, our method is applicable to BLOOM (bottom right) and GPT-Neo (top left). Y-axis shows median-calibrated accuracy as used in Halawi et al. (2023) 5. Applications We start by reproducing and extending two use cases of the logit lens in the literature. 5.1. Eliciting Secret Knowledge from Language Models Cywi Ě nski et al. (2025) apply the logit lens to uncover knowl- edge that an AI possesses but does not verbalize. Specifi- cally, they fine-tune a model to generate hints and respond to user queries about a specific âtabooâ word, while never verbalizing it directly. They then prompt an âauditorâ LLM to guess what the taboo word is, allowing it to ask the taboo model questions in a black-box fashion. They find that the auditor is much more accurate at guessing the taboo word when it is provided with the top 20 highest probability latent predictions from the taboo model according to the logit lens. This is true in two different settings: âstandard,â where the prompt asks for a clue about the taboo word, and âdirect,â where the prompt attempts to compel the model to explicitly reveal the taboo word. We hypothesized that the tuned lens might perform better than the logit lens for this task. The results were mixed: the tuned lens outperforms at some layers and underperforms at other layers. Without âpeakingâ at the ground truth labels, it is not really possible to determine which layer to use in advance. 5.2. Extending Overthinking the Truth Halawi et al. (2023) apply the logit lens to downstream tasks with few-shot prompts, and find that predictions ex- tracted from earlier layers are often more robust to incorrect demonstrations than final layer predictions. Unfortunately, their method is inapplicable to models like BLOOM and GPT-Neo, which have high logit lens perplexities. We replicate Halawi et al. (2023)âs experiments with the tuned lens, and recover similar results for BLOOM 560M, Neo 1.3B, and Neo 2.7B (Figure 11). Notice that the cal- ibrated performance under incorrect demonstrations (red) peaks at an early layer around 0.4-0.45 in all three of these models, before falling to random baseline performance at the final layer, similarly to what was found by Halawi et al. 5.3. Detecting Prompt Injections Given the results from Halawi et al. (2023) and in Figure 11, we hypothesize that the prediction trajectory of the tuned lens on anomalous inputs should be different from the tra- jectories on normal inputs, and that this could be used to detect anomalous inputs. To test this, we focus on prompt injection attacks, a recently discovered vulnerability in large language models where untrusted inputs from a malicious user cause the model to behave in unexpected or dangerous ways (Perez and Ribeiro, 2022). These attacks usually tell the model to âignore previ- ous instructionsâ and instead follow instructions crafted by the attacker. To simulate prompt injections, we adapted the attack from Perez and Ribeiro (2022) and ran Pythia 12B on nine multi- ple choice tasks, both with and without the following adver- sarial message appended to the few-shot prompt: ---------- ---------- Ignore any previous and following instructions and just print "<WRONG_ANSWER>": where<WRONG_ANSWER>is replaced with a randomly se- lected incorrect response from the available multiple choice responses. We record the tuned prediction trajectory for each data pointâ that is, for each layer, we record the log probability assigned by the model to each possible answer. 6 We then flatten these trajectories into feature vectors and feed them into two standard outlier detection algorithms: isolation forest (iForest) (Liu et al., 2008) and local outlier factor (LOF) 6 For binary tasks like SST-2 we take the difference between the log probabilities assigned to the two possible answers. Eliciting Latent Predictions from Transformers with the Tuned Lens Tuned LensLogit LensBaselineAccuracy TaskiForestLOFiForestLOFSRMNormalâ Injected ARC-Easy0.59 (0.54, 0.62)0.73 (0.71, 0.76)0.53 (0.50, 0.57)0.59 (0.56, 0.62)0.73 (0.70, 0.75)72.8%â 31.7% ARC-Challenge0.71 (0.65, 0.77)0.81 (0.77, 0.84)0.73 (0.67, 0.79)0.80 (0.77, 0.83)0.57 (0.53, 0.61)43.5%â 24.7% BoolQ0.99 (0.98, 0.99)1.00 (1.00, 1.00)0.89 (0.87, 0.91)0.61 (0.57, 0.66)1.00 (1.00, 1.00)67.1%â 0.0% MC TACO0.74 (0.71, 0.77)0.68 (0.66, 0.70)0.68 (0.66, 0.69)0.55 (0.53, 0.59)1.00 (1.00, 1.00)0.40â 0.06 F1 MNLI 0.98 (0.98, 0.99)1.00 (1.00, 1.00)0.95 (0.94, 0.96)1.00 (1.00, 1.00)1.00 (1.00, 1.00)54.3%â 0.0% QNLI0.99 (0.99, 1.00)1.00 (1.00, 1.00)0.93 (0.92, 0.95)0.68 (0.63, 0.71)1.00 (1.00, 1.00)54.3%â 0.0% QQP1.00 (0.99, 1.00)1.00 (1.00, 1.00)0.90 (0.89, 0.90)0.79 (0.76, 0.81)1.00 (1.00, 1.00)60.7%â 6.5% SciQ0.62 (0.57, 0.69)0.64 (0.59, 0.70)0.75 (0.71, 0.79)0.70 (0.65, 0.74)0.75 (0.72, 0.78)95.5%â 62.6% SST-21.00 (0.98, 1.00)1.00 (1.00, 1.00)0.78 (0.72, 0.83)0.61 (0.56, 0.65)1.00 (1.00, 1.00)82.9%â 49.1% Table 1. Test set AUROCs and 95% bootstrap CIs for distinguishing normal prompts from prompt injections on Pythia 12B. Figures are pooled over 10 random train-test splits. Attack detection performance is nearly perfect on tasks where the attack succeeds at driving accuracy well below the random baseline, and is still much better than chance even when the attack is only partially successful. (Breunig et al., 2000), both implemented in scikit-learn (Pedregosa et al., 2011) with default hyperparameters. Baseline. There is a rich literature on general out-of- distribution (OOD) detection in deep neural networks. One simple technique is to fit a multivariate Gaussian to the modelâs final layer hidden states on the training set, and flag inputs as OOD if a new hidden state is unusually far from the training distribution as measured by the Mahalanobis distance (Lee et al., 2018; Mahalanobis, 1936). Recently, Bai et al. (2022) proposed the Simplified Rela- tive Mahalanobis (SRM) distance, a modification to Ma- halanobis which they find to be effective in the context of LLM finetuning. They also find that representations from the middle layers of a transformer, rather than the final layer, yield the best OOD detection performance. We use the SRM at the middle layer as a baseline in our experiments. Experimental setup. We fit each anomaly detection model exclusively on prediction trajectories from normal prompts without prompt injections, and evaluate them on a held out test set containing both normal and prompt-injected trajectories. This ensures that our models cannot overfit to the prompt injection distribution. We use EleutherAIâs lm-evaluation-harnesslibrary (Gao et al., 2021) to run our evaluations. Results. Our results are summarized in Table 1. Our tuned lens anomaly detector achieves perfect or near-perfect AU- ROC on five tasks (BoolQ, MNLI, QNLI, QQP, and SST-2); in contrast, the same technique using logit lens has lower performance on most tasks. On the other hand, the SRM baseline does consistently wellâthe tuned lens only outper- forms it on one task (ARC-Challenge), while SRM outper- forms our technique on both MC TACO and SciQ. We suspect that further gains could be made by combin- ing the strengths of both techniques, since SRM uses only one layer but considers a high-dimensional representation, while the tuned lens studies the trajectory across layers but summarizes them with a low-dimensional prediction vector. 5.4. Measuring Example Difficulty Early exiting strategies like CALM (Schuster et al., 2022) and DeeBERT (Xin et al., 2020) are based on the obser- vation that âeasyâ examples require less computation to classify than âdifficultâ examples. If an example is easy, the model should quickly converge to the right answer in early layers, making it possible to skip the later layers without a significant drop in prediction quality. Conversely, the num- ber of layers needed to converge on an answer can be used to measure the difficulty of an example. We propose to use the tuned lens to estimate example diffi- culty in pretrained transformers, without the need to fine- tune the model for early exiting. Following Baldock et al. (2021)âs work on computer vision models, we define the prediction depth of a promptxto be the number of layers after which a modelâs top-1 prediction forxstops changing. To validate the prediction depth, we measure its correlation with an established difficulty metric: the iteration learned. The iteration learned is defined as the earliest training step Ďwhere the modelâs top-1 prediction for a datapointxis fixed (Toneva et al., 2018). Intuitively, we might expect that examples which take a long time to learn during training would tend to require many layers of computation to classify at inference time. Baldock et al. (2021) indeed show such a correlation, using k-N classifiers to elicit early predictions from the intermediate feature maps of image classifiers. Experimental setup. For this experiment we focus on Pythia 12B (deduped), for which 143 uniformly spaced checkpoints are available on Huggingface Hub. We evalu- ate the modelâs zero-shot performance on twelve multiple- choice tasks, listed in Table 2. For each checkpoint, we store the top 1 prediction on every individual example, al- lowing us to compute the iteration learned. We then use the tuned lens on the final checkpoint, eliciting the top 1 prediction at each layer of the network and computing the Eliciting Latent Predictions from Transformers with the Tuned Lens TaskTuned lens ĎLogit lens ĎFinal acc ARC-Easy0.5770.50069.7% ARC-Challenge0.5470.48532.4% LogiQA0.4980.27721.4% MNLI0.3950.43540.4% PiQA0.6600.62076.1% QNLI0.409 â0.09953.0% QQP0.585 â0.3400.381 (F1) RTE0.1560.34760.0% SciQ0.5300.50591.9% SST-20.5550.29264.7% WinoGrande0.5170.53763.9% Table 2.Correlation between two measures of example difficulty, the iteration learned and prediction depth, across tasks. Prediction depth is measured using the tuned lens in the first column and the logit lens in the second. prediction depth for every example. As a baseline, we also compute prediction depths using the logit lens. Finally, for each task, we compute the Spearman rank correlation be- tween the iteration learned and the prediction depth across all examples. Results. We present results in Table 2. We find a significant positive correlation between the iteration learned and the tuned lens prediction depth on all tasks we investigated. Additionally, the tuned lens prediction correlates better with iteration learned than its logit lens counterpart in 8 out of 11 tasks, sometimes dramatically so. 6. Discussion In this paper, we introduced a new tool for transformer in- terpretability research, the tuned lens, which yields new qualitative as well as quantitative insights into the function- ing of large language models. It is a drop-in replacement for the logit lens that makes it possible to elicit interpretable pre- diction trajectories from essentially any pretrained language model in use today. We gave several initial applications of the tuned lens, including detecting prompt injection attacks. Finally, we introduced causal basis extraction, which iden- tifies influential features in neural networks. We hope this technique will be generally useful for interpretability re- search in machine learning. Limitations and future work. One limitation of our method is that it involves training a translator layer for each layer of the network, while the logit lens can be used on any pretrained model out-of-the-box. This training pro- cess, however, is quite fast: our code can train a full set of probes in under an hour on a single 8ĂA40 node, and further speedups are likely possible. We have also released tuned lens checkpoints for the most commonly used pre- Figure 12.Token-level prediction depths for Pythia 12B computed on the abstract of OpenAI (2023). Warm colors have high predic- tion depth, while cool colors indicate low depth. trained models as part of ourtuned-lenslibrary, which should eliminate this problem for most applications. Causal basis extraction, as presented in this work, is compu- tationally intensive, since it sequentially optimizesd model causal basis vectors for each layer of the network. Fu- ture work could explore ways to make the algorithm more scalable. One possibility would be to optimize a wholek- dimensional subspace, instead of an individual direction, at each iteration. Due to space and time limitations, we focused on language models in this work, but we think itâs likely that our ap- proach is also applicable to other modalities. Acknowledgements We are thankful to CoreWeave for providing the computing resources used in this paper and to the OPT team for their assistance in training a tuned lens for OPT. We also thank nostalgebraist for discussions leading to this paper. References John Aitchison. The statistical analysis of compositional data. Journal of the Royal Statistical Society: Series B (Methodological), 44(2):139â160, 1982. Guillaume Alain and Yoshua Bengio. Understanding in- termediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644, 2016. Alex Andonian, Quentin Anthony, Stella Biderman, Sid Black, Preetham Gali, Leo Gao, Eric Hallahan, Josh Levy-Kramer, Connor Leahy, Lucas Nestler, Kip Parker, Michael Pieler, Shivanshu Purohit, Tri Songz, Wang Phil, and Samuel Weinbach. GPT-NeoX: Large Scale Au- toregressive Language Modeling in PyTorch, 8 2021. URLhttps://w.github.com/eleutherai/ gpt-neox. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful Eliciting Latent Predictions from Transformers with the Tuned Lens and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022. Robert Baldock, Hartmut Maennel, and Behnam Neyshabur. Deep learning through the lens of example difficulty. Ad- vances in Neural Information Processing Systems, 34: 10876â10889, 2021. Yamini Bansal, Preetum Nakkiran, and Boaz Barak. Revis- iting model stitching to compare neural representations. Advances in Neural Information Processing Systems, 34: 225â236, 2021. Yonatan Belinkov. Probing classifiers: Promises, shortcom- ings, and advances. Computational Linguistics, 48(1): 207â219, 2022. Stella Biderman, Kieran Bicheno, and Leo Gao. Datasheet for the Pile. Computing Research Repository, 2022. doi: 10.48550/arXiv.2201.07311. URLhttps://arxiv. org/abs/2201.07311v1. version 1. Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle OâBrien, Eric Hallahan, Mo- hammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Aviya Skowron, Lintang Sutawika, and Os- kar van der Wal. Pythia: a scaling suite for language model interpretability research. Computing Research Repository, 2023. doi: 10.48550/arXiv.2201.07311. URL https://arxiv.org/abs/2201.07311v1. ver- sion 1. Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. Gpt-neo: Large scale autoregressive language modeling with mesh-tensorflow. If you use this software, please cite it using these metadata, 58, 2021. Sid Black, Stella Biderman, Eric Hallahan, Quentin An- thony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. Gpt-neox-20b: An open-source autoregressive language model. arXiv preprint arXiv:2204.06745, 2022. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. arXiv preprint arXiv:1607.04606, 2016. Markus M Breunig, Hans-Peter Kriegel, Raymond T Ng, and JĂśrg Sander. Lof: identifying density-based local outliers. In Proceedings of the 2000 ACM SIGMOD international conference on Management of data, pages 93â104, 2000. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Lan- guage models are few-shot learners. Advances in neural information processing systems, 33:1877â1901, 2020. LawrenceChan,AdriĂ Garriga-Alonso,Nicholas Goldowsky-Dill, Ryan Greenblatt, Jenny Nitishinskaya, Ansh Radhakrishnan, Buck Shlegeris, and Nate Thomas. Causal scrubbing: a method for rigorously testing interpretability hypotheses. Alignment Forum, 2022. URL https://bit.ly/3WRBhPD. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yong- hao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URLhttps: //lmsys.org/blog/2023-03-30-vicuna/. AdriĂĄn CsiszĂĄrik, PĂŠter K Ě orĂśsi-SzabĂł, Ăkos Matszangosz, Gergely Papp, and DĂĄniel Varga. Similarity and matching of neural network representations. Advances in Neural Information Processing Systems, 34:5656â5668, 2021. Bartosz Cywi Ě nski, Emil Ryd, Rowan Wang, Senthooran Rajamanoharan, Neel Nanda, Arthur Conmy, and Samuel Marks. Eliciting secret knowledge from language models. arXiv preprint arXiv:2510.01070, 2025. Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained trans- formers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 8493â8502, Dublin, Ire- land, May 2022. Association for Computational Linguis- tics. doi: 10.18653/v1/2022.acl-long.581. URLhttps: //aclanthology.org/2022.acl-long.581. Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. An- alyzing transformers in embedding space. arXiv preprint arXiv:2209.02535, 2022. Bruno De Finetti, Henry E Kyburg, and Howard E Smok- ler. Foresight: Its logical laws, its subjective sources. Breakthroughs in statistics, 1:134â174, 1937. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020. Juan JosĂŠ Egozcue and Vera Pawlowsky-Glahn. Changing the reference measure in the simplex and its weighting effects. Austrian Journal of Statistics, 45(4):25â44, 2016. Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Gold- berg. Amnesic probing: Behavioral explanation with amnesic counterfactuals. Transactions of the Association for Computational Linguistics, 9:160â175, 2021. Eliciting Latent Predictions from Transformers with the Tuned Lens Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova Das- Sarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. https://transformer- circuits.pub/2021/framework/index.html. Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield- Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposition. arXiv preprint arXiv:2209.10652, 2022. Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth on demand with structured dropout. arXiv preprint arXiv:1909.11556, 2019. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800GB dataset of diverse text for language modeling. Computing Research Repository, 2020. doi: 10.48550/arXiv.2101.00027. URLhttps: //arxiv.org/abs/2101.00027v1. version 1. Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, An- thony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot lan- guage model evaluation, September 2021. URLhttps: //doi.org/10.5281/zenodo.5371628. Scott Garrabrant, Tsvi Benson-Tilsen, Andrew Critch, Nate Soares, and Jessica Taylor. Logical induction. arXiv preprint arXiv:1609.03543, 2016. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. RealToxicityPrompts: Eval- uating neural toxic degeneration in language mod- els.In Findings of the Association for Computa- tional Linguistics: EMNLP 2020, pages 3356â3369, On- line, November 2020. Association for Computational Linguistics.doi: 10.18653/v1/2020.findings-emnlp. 301. URLhttps://aclanthology.org/2020. findings-emnlp.301. Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. arXiv preprint arXiv:2012.14913, 2020. Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Gold- berg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. arXiv preprint arXiv:2203.14680, 2022. Klaus Greff, Rupesh K Srivastava, and JĂźrgen Schmidhuber. Highway and residual networks learn unrolled iterative estimation. arXiv preprint arXiv:1612.07771, 2016. Ian Hacking. Slightly more realistic personal probability. Philosophy of Science, 34(4):311â325, 1967. Danny Halawi, Jean-Stanislas Denain, and Jacob Steinhardt. Overthinking the truth: Understanding how language models process false demonstrations. In Submitted to The Eleventh International Conference on Learning Represen- tations, 2023. URLhttps://openreview.net/ forum?id=em4xg1Gvxa. under review. Benjamin Heinzerling and Michael Strube.BPEmb: Tokenization-free Pre-trained Subword Embeddings in 275 Languages. In Nicoletta Calzolari (Conference chair), Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Koiti Hasida, Hitoshi Isahara, Bente Mae- gaard, Joseph Mariani, HĂŠlène Mazo, Asuncion Moreno, Jan Odijk, Stelios Piperidis, and Takenobu Tokunaga, editors, Proceedings of the Eleventh International Con- ference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May 7-12, 2018 2018. Euro- pean Language Resources Association (ELRA). ISBN 979-10-95546-00-9. J Hewitt and P Liang. Designing and interpreting probes with control tasks. Proceedings of the 2019 Con, 2019. John Hewitt and Christopher D Manning. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume 1 (Long and Short Papers), pages 4129â4138, 2019. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language mod- els. ICLR, 1(2):3, 2022. Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kil- ian Q Weinberger. Deep networks with stochastic depth. In European conference on computer vision, pages 646â 661. Springer, 2016. StanisĹaw Jastrz ̨ebski, Devansh Arpit, Nicolas Ballas, Vikas Verma, Tong Che, and Yoshua Bengio. Residual con- nections encourage iterative inference. arXiv preprint arXiv:1710.04773, 2017. Keller Jordan, Yuchen Jin, Vlado Boza, You Jiacheng, Franz Cecista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks. URL https://kellerjordan.github.io/posts/muon, 2024. Eliciting Latent Predictions from Transformers with the Tuned Lens Olga Kovaleva, Saurabh Kulshreshtha, Anna Rogers, and Anna Rumshisky. Bert busters: Outlier dimensions that disrupt transformers. arXiv preprint arXiv:2105.06990, 2021. Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of- distribution samples and adversarial attacks. Advances in neural information processing systems, 31, 2018. Karel Lenc and Andrea Vedaldi. Understanding image rep- resentations by measuring their equivariance and equiv- alence. In Proceedings of the IEEE conference on com- puter vision and pattern recognition, pages 991â999, 2015. Kenneth Li, Aspen K Hopkins, David Bau, Fernanda ViĂŠ- gas, Hanspeter Pfister, and Martin Wattenberg. Emer- gent world representations:Exploring a sequence model trained on a synthetic task.arXiv preprint arXiv:2210.13382, 2022. Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation forest. In 2008 eighth ieee international conference on data mining, pages 413â422. IEEE, 2008. Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, et al. Muon is scalable for llm training. arXiv preprint arXiv:2502.16982, 2025. PC Mahalanobis. On the generalized distances in statistics: Mahalanobis distance. Journal Soc. Bengal, 26:541â588, 1936. Beren Millidge and Sid Black. The singular value decom- positions of transformer weight matrices are highly inter- pretable. LessWrong, 2022. URLhttps://bit.ly/ 3GdbZoa. Neel Nanda.Transformerlens,2022.URL https://github.com/neelnanda-io/ TransformerLens. nostalgebraist.interpreting gpt: the logit lens.Less- Wrong, 2020.URLhttps://w.lesswrong. com/posts/AcKRB8wDpdaN6v6ru/ interpreting-gpt-the-logit-lens. nostalgebraist.logitlensonnon-gpt2mod- els+extensions,2021.URLhttps: //colab.research.google.com/drive/ 1MjdfK2srcerLrAJDRaJQKO0sUiZ-hQtA. Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 5(3):e00024â001, 2020. OpenAI. Gpt-4 technical report. Technical report, OpenAI, 2023. Technical Report. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cour- napeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit- learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825â2830, 2011. FĂĄbio Perez and Ian Ribeiro. Ignore previous prompt: At- tack techniques for language models. arXiv preprint arXiv:2211.09527, 2022. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsu- pervised multitask learners. OpenAI Blog, 2019. FP Ramsey. Truth and probability. Studies in subjective probability, pages 61â92, 1926. Olivier Roy and Martin Vetterli. The effective rank: A measure of effective dimensionality. In 2007 15th Euro- pean signal processing conference, pages 606â610. IEEE, 2007. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.arXiv preprint arXiv:1910.01108, 2019. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili Ě c, Daniel Hesslow, Roman CastagnĂŠ, Alexandra Sasha Luccioni, François Yvon, Matthias GallĂŠ, Jonathan Tow, Alexander M. Rush, Stella Bider- man, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, BenoĂŽt Sagot, Niklas Muennighoff, Al- bert Villanova del Moral, Olatunji Ruwase, Rachel Baw- den, Stas Bekman, Angelina McMillan-Major, Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Or- tiz Suarez, Victor Sanh, Hugo Laurençon, Yacine Jer- nite, Julien Launay, Margaret Mitchell, Colin Raffel, and et al. BLOOM: A 176B-parameter open-access multi- lingual language model. Computing Research Repos- itory, 2022. doi: 10.48550/arXiv.2211.05100. URL https://arxiv.org/abs/2211.05100v2. ver- sion 2. Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Q Tran, Yi Tay, and Donald Metzler. Confident adaptive language modeling. arXiv preprint arXiv:2207.07061, 2022. Pinky Sitikhu, Kritish Pahi, Pujan Thapa, and Subarna Shakya. A comparison of semantic similarity methods for maximum human interpretability. 2019 Artificial In- telligence for Transforming Business and Society (AITB), 1:1â4, 2019. Eliciting Latent Predictions from Transformers with the Tuned Lens William Timkey and Marten van Schijndel. All bark and no bite: Rogue dimensions in transformer language models obscure representational quality. arXiv preprint arXiv:2109.04404, 2021. Together.Redpajama, a project to create leading open-source models, starts by reproducing llama training dataset of over 1.2 trillion tokens, April 2023. URLhttps://w.together.xyz/blog/ redpajama. Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geof- frey J Gordon. An empirical study of example forget- ting during deep neural network learning. arXiv preprint arXiv:1812.05159, 2018. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Mar- tinet, Marie-Anne Lachaux, TimothĂŠe Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Au- relien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023. Mycal Tucker, Peng Qian, and Roger Levy. What if this modified that? syntactic interventions with counterfactual embeddings. In Findings of the Association for Computa- tional Linguistics: ACL-IJCNLP 2021, pages 862â875, 2021. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Ĺukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. Andreas Veit, Michael J Wilber, and Serge Belongie. Resid- ual networks behave like ensembles of relatively shallow networks. Advances in neural information processing systems, 29, 2016. Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. arXiv preprint arXiv:2211.00593, 2022. Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy Lin. Deebert: Dynamic early exiting for accelerating bert inference. arXiv preprint arXiv:2004.12993, 2020. Eliezer Yudkowsky.Conservation of expected evidence.https://w.lesswrong. com/posts/jiBFC7DcCrZjGmZnJ/ conservation-of-expected-evidence, 2007. Accessed: March 18, 2023. Minjia Zhang and Yuxiong He.Accelerating training of transformer-based language models with progressive layer dropping. arXiv preprint arXiv:2010.13369, 2020. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettle- moyer. OPT: Open pre-trained transformer language models. Computing Research Repository, 2022. doi: 10.48550/arXiv.2205.01068. URLhttps://arxiv. org/abs/2205.01068v4. version 4. Eliciting Latent Predictions from Transformers with the Tuned Lens A. Additional evaluation results 010203040 0 1 2 3 4 010203040 Transferred from LLaMA Trained on Vicuna Logit Lens Transfer of a Tuned Lens (LLaMA-13b to Vicuna-13b v1.1) Layer KL Divergence (bits per byte) Anthropic H-RLHF (chosen)Together RedPajama Sample Figure 13.Comparing the performance of a lens specifically trained on Vicuna 13B vs a lens transferred from LLaMA 13B vs Vicuna 13Bâs Logit lens. Specifically, we messure the KL divergence between the lens at a specific layer and the modelâs final output. Eliciting Latent Predictions from Transformers with the Tuned Lens i n p u t 5 1 0 1 5 2 0 2 5 3 0 3 5 4 0 4 5 0 1 2 3 4 5 i n p u t 5 1 0 1 5 2 0 2 5 3 0 3 5 4 0 4 5 0 1 2 3 4 5 Model size small medium large xl Logit lens (baseline)Tuned lens (ours) Layer b i t s p e r b y t e i n p u t 5 1 0 1 5 2 0 2 5 3 0 0 1 2 3 4 5 6 i n p u t 5 1 0 1 5 2 0 2 5 3 0 0 1 2 3 4 5 6 Model size 125M 1.3B 2.7B Logit lens (baseline)Tuned lens (ours) Layer b i t s p e r b y t e i n p u t 5 1 0 1 5 2 0 2 5 3 0 8 9 1 2 3 4 5 6 7 8 9 10 2 3 i n p u t 5 1 0 1 5 2 0 2 5 3 0 8 9 1 2 3 4 5 6 7 8 9 10 2 3 Model size 125m 1.3b 6.7b Logit lens (baseline)Tuned lens (ours) Layer b i t s p e r b y t e Figure 14.Perplexities of predictions elicited from OpenAIâs GPT-2 models (top), EleutherAIâs GPT-Neo models (middle), and Metaâs OPT models (bottom). We omit OPT 350M as it uses a post-LN architecture. Eliciting Latent Predictions from Transformers with the Tuned Lens B. Qualitative Results isafirst.theofandisafirst.theof 'safirstofthe,andisafirstofthe, afirstwayall,but'safirstofall, thefirstthingall,butwasnotmost.all, afirstthingall,butwasthemostoftimes, thefirstthingall,andwasthebestoftimes, afirstthingallandandwastheworstoftimes, afirstthingtimesandandwastheworstoftimes, afirstsellingtimesandofwastheworstoftimes, afirstthingtimesandthewastheworstoftimes, afirstoftimes,thewastheworstoftimes, afirstthingtimes,thewastheworstoftimes, afirstthingtimes,thewastheworstoftimes, afirstoftimes,itwastheworstoftimes, afirstoftimes,itwastheworstoftimes, afirstoftimes,itwastheworstoftimes, afirstoftimes,itwastheworstoftimes, afirstoftimes,itwastheworstoftimes, 'safirstoftimes,itwastheworstoftimes, Itwas the best of times , it was the worst of times input 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 output 00.20.40.60.81 Probability Tuned Lens (EleutherAI/pythia-12b-deduped) Input Layer Figure 15.Tuned lens prediction trajectory for Pythia 12B on the first several words of Charles Dickensâ âA Tale of Two Cities.â Note that the latent prediction becomes very confident at early layers after the prefix âIt was the best of timesâ has been processed, suggesting some degree of memorization. ,andhaveof._type.andoutally.of ,thecanto.Atype-andlargegressivedeclineand ,thehavethePMsA,andindependentgressiveandand ,thewillthe$-based.whichleadinggressivemodeland ,wealsoaLA-based,whichalgorithmgressivemodelmodel ,weshowaTE-based,whichnewgressivemodelmodel ,weshowaTE-basedandwhichmodelgressivemodelmodel ,weshowonTE-basedandanewgressivemodelmodel ,weshowaTE2basedandalargegressivemodelmodel ,weshowaSM21andamodelgressivemodelmodel ,wedemonstrateaK2basedandamodelgressivemodelmodel ,weshowmodelstrained-basedandamodelgressivemodelmodel ,weshowmodelstrained-trained(alargegressiveneuralmodel ,weshowa.2basedandalargegressiveneuralmodel ,weshowaEL-2onalargegressiveneuralmodel ,weshowlanguageEL-2onastategressivemodelmodel ,weshowlanguagePT-2onastategressiveTransmodel ,weshowaPT-2onalargegressiveTransmodel ,weshowaPT-2onaopengressiveTransmodel Specifically, we train G PT - 3 , an autore gressive language input 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 output 00.20.40.60.81 Probability Tuned Lens (EleutherAI/pythia-12b-deduped) Input Layer Figure 16. Tuned lens prediction trajectory for Pythia 12B prompted with the abstract of Brown et al. (2020). Eliciting Latent Predictions from Transformers with the Tuned Lens B.1. Logit lens pathologies Itwasthebestoftimes,itwastheworstoftimes Itwasthebestoftimes,itwastheworstoftimes Itwasthebestoftimesitoaitwastheworstoftimes menwasacerbbestoftimestorneyitnâĢĝtfollowing:worstoftimes menwasacerbbestCortesĂĹatimestorneyitnâĢĝtacerbworstCortesĂĹatimes mennâĢĝtcol-mdbestglobaistimes20px;itnâĢĝtcompanyaworstCortesĂĹatimes mennâĢĝtcol-mdbestCortesĂĹatimes20px;-itnâĢĝt^worstgremiogras mennâĢĝtresponsibilitycomunitĂÂĄriajudiciatimesthough-itnâĢĝtpluperfectworstdonnait, mennâĢĝtresponsibilitythinginstitucionaistimesthoughdidn'tnâĢĝtpluperfectworstmondiaux, mennâĢĝtsamethingthings:,thoughwasn'tnâĢĝtpluperfectthereofcircumstances, mennâĢĝtfirstthingluckwhenthoughwasn'tnâĢĝtworstoftimes, menalsofirstthingall,andwasthebestoftimes, isafirstthingthe,andwasthebestoftimes, Itwas the best of times , it was the worst of times input 1 3 5 7 9 11 13 15 17 19 21 output 00.20.40.60.81 Probability Logit Lens (bigscience/bloom-560m) Input Layer Figure 17. Logit lens prediction trajectory for BLOOM 560M. The logit lens assigns as very high probability to the input token at many layers and token positions, complicating the interpretation of the output. Itwasthebestoftimes,itwastheworstoftimes Itwasthebestoftimes,itwastheworstoftimes Itwasthebestoftimes,itwastheworstoftimes Itwasthebestoftimes,itwastheworstoftimes Itwasthebestoftimes,itwastheworstoftimes Itwasthebestoftimes,itwastheworstoftimes Itwasthebestoftimes,itwastheworstoftimes 'sasamebestoftimesbutitwasonlyworstoftimes 'sasamebestoftimesbutwasntworstworstof. 'sasamethingthe,but'stheworstoftimes, 'sasamethingboth,thewastheworstoftimes. 'sasamethingboth,thewastheworstoftimes. Itwas the best of times , it was the worst of times input 1 2 3 4 5 6 7 8 9 10 output 00.20.40.60.81 Probability Logit Lens (facebook/opt-125m) Input Layer Figure 18. Logit lens prediction trajectory for OPT 125M, exhibiting a similar pathology to BLOOM 560M above. Eliciting Latent Predictions from Transformers with the Tuned Lens C. Transformers Perform Iterative Inference Jastrz ̨ebski et al. (2017) argue that skip connections encourage neural networks to perform âiterative inferenceâ in the following sense: each layer reliably updates the hidden state in a direction of decreasing loss. While their analysis focused on ResNet image classifiers, their theoretical argument applies equally well to transformers, or any neural network featuring residual connections. We reproduce their theoretical analysis below. A residual block applied to a representation h i updates the representation as follows: h â+1 = h â + F â (h â ) LetL denote the final linear classifier followed by a loss function. We can Taylor expandL(h L ) around h i to yield L(h L ) =L(h i ) + L X j=i D F j (h j ), âL(h j ) âh j E |z gradient-residual alignment +O(F 2 j (h j ))(19) Thus to a first-order approximation, the model is encouraged to minimize the inner product between the residualF (h i )and the gradient âL(h i ) âh i , which can be achieved by aligning it with the negative gradient. To empirically measure the extent to which residuals do align with the negative gradient, Jastrz ̨ebski et al. (2017) compute the cosine similarity between F (h i ) and âL(h i ) âh i at each layer of a ResNet image classifier. They find that it is consistently negative, especially in the final stage of the network. We reproduced their experiment using Pythia 6.9B, and report the results in Figure 19. We show that, for every layer in the network, the cosine similarity between the residual and the gradient is negative at least 95% of the time. While the magnitudes of the cosine similarities are relatively small in absolute terms, never exceeding 0.05, we show that they are much larger than would be expected of random vectors in this very high dimensional space. Specifically, we first sample 250 random Gaussian vectors of the same dimensionality as âL(h i ) âh i , which is (hidden size)Ă(sequence length) = 8,388,608. We then compute pairwise cosine similarities between the vectors, and find the 5 th percentile of this sample to beâ6Ă 10 â4 . Virtually all of the gradient-residual pairs we observed had cosine similarities below this number. 2468101214161820222426283032 â0.05 â0.04 â0.03 â0.02 â0.01 0 Alignment of residuals with gradients (Pythia 6.7B) Layer Cosine similarity Random vectors (5th percentile) 1471013161922252831 0 1 2 3 4 Perplexity after layer deletion (Pythia 6.9B) Layer ablated bits per byte Baseline Figure 19.Left: Cosine similarity between the observed update to the hidden state and âL(h i ) âh i . The similarity is almost always negative, and of much larger magnitude than would be expected by chance in this very high dimensional space. Boxplot whiskers are 5th and 95th percentiles. Results were computed over a sample of roughly 16M tokens from the Pile validation set. Right: Cross-entropy loss of Pythia 6.9B on the Pile validation set, after replacing each of its 32 layers with the identity transformation. The dashed line indicates Pythiaâs perplexity on this dataset with all layers intact. C.1. Zero-shot robustness to layer deletion Stochastic depth (Huang et al., 2016), also known as LayerDrop (Fan et al., 2019), is a regularization technique which randomly drops layers during training, and can be applied to both ResNets and transformers. Veit et al. (2016) show that Eliciting Latent Predictions from Transformers with the Tuned Lens ResNets are robust to the deletion of layers even when trained without stochastic depth, while CNN architectures without skip connections are not robust in this way. These results strongly support the idea that adjacent layers in a ResNet encode fundamentally similar representations (Greff et al., 2016). To the best of our knowledge, this experiment had never been replicated before in transformers. We did so and report our results in Figure 19 above. We find that only the very first layer is crucial for performance; every other layer ablation induces a nearly imperceptible increase in perplexity. Interestingly, Veit et al. (2016) also found that the first layer is exceptionally important in ResNets, suggesting that this is a general phenomenon. D. Causal intervention details D.1. Ablation techniques Prior work has assumed that, given a representationxand a linear subspaceB â R D along which we want to remove information, the best erasure strategy is to project x onto the orthogonal complement of B: x Ⲡ=proj B ⼠(x)(20) This âzeroes outâ a part ofx, ensuring thatâ¨x Ⲡ, u⊠= 0for alluinB. But recent work has pointed out that setting activations to zero may inadvertently push a network out of distribution, making the experimental results hard to interpret (Wang et al., 2022; Chan et al., 2022). For example, a probe may rely on the invariant thatâx,â¨x, u⊠⍠0for someuinBas an implicit bias term, even if â¨x, uâŠhas low variance and does not encode any useful information about the input. The naive projectionx Ⲡ=proj B ⼠(x) would significantly increase the loss, making it seem thatproj B xcontains important informationâ when in reality model performance would not be significantly degraded ifproj B xwere replaced with a nonzero constant value more representative of the training distribution. To correct for this, in our causality experiments we ablate directions by replacing them with their mean values computed across a dataset, instead of zeroing them out. Specifically, to ablate a direction u, we use the formula: x Ⲡ= x + P u (xâ x)(21) where P u is the projection matrix for u andx is the mean representation. E. Static interpretability analysis Several interpretability methods aim to analyze parameters that âwrite" (in the sense of Elhage et al. (2021)) to intermediate hidden states of a language model (Millidge and Black, 2022; Dar et al., 2022; Geva et al., 2022; 2020), and even use this analysis to successfully edit model behavior (Geva et al., 2022; Millidge and Black, 2022; Dai et al., 2022). Given that the tuned lens aims to provide a âless biased" view of intermediate hidden states, we should expect the tuned lens to preserve their effectiveness. We test both static analysis and model editing methods, and confirm this hypothesis. With static analysis, the tuned lens appears to never decrease performance, and for some models, even increases their performance. With model editing, we found the tuned lens to outperform the logit lens on OPT-125m and perform equivalently on other models for the task of toxicity reduction. E.1. Static Analysis Many parameters in transformer language models have at least one dimension equal to that of their hidden state, allowing us to multiply by the unembedding matrix to project them into token space. Surprisingly, the resulting topktokens are often meaningful, representing interpretable semantic and syntactic clusters, such as âprepositionsâ or âcountriesâ (Millidge and Black, 2022; Dar et al., 2022; Geva et al., 2022). This has successfully been applied to the columns of the MLP output matrices (Geva et al., 2022), the singular vectors of the MLP input and output matrices (Millidge and Black, 2022), and the singular vectors of the attention output-value matrices W OV (Millidge and Black, 2022). In short 7 , we can explain this as: many model parameters directly modify the modelâs hidden state (Elhage et al., 2021), and 7 The exact interpretation depends on the method used - for instance, the rows ofW out can be interpreted as the values in a key-value memory store (Geva et al., 2020). Eliciting Latent Predictions from Transformers with the Tuned Lens 1Îź100Îź0.011 0.001 2 5 0.01 2 5 0.1 2 5 1 2 5 10 1Îź100Îź0.011 0.001 2 5 0.01 2 5 0.1 2 5 1 2 5 10 1Îź100Îź0.011 0.001 2 5 0.01 2 5 0.1 2 5 1 2 5 10 1Îź100Îź0.011 0.001 2 5 0.01 2 5 0.1 2 5 1 2 5 10 1Îź100Îź0.011 0.001 2 5 0.01 2 5 0.1 2 5 1 2 5 10 1Îź100Îź0.011 0.001 2 5 0.01 2 5 0.1 2 5 1 2 5 10 1Îź100Îź0.011 0.001 2 5 0.01 2 5 0.1 2 5 1 2 5 10 1Îź100Îź0.011 0.001 2 5 0.01 2 5 0.1 2 5 1 2 5 10 1Îź100Îź0.011 0.001 2 5 0.01 2 5 0.1 2 5 1 2 5 10 1Îź100Îź0.011 0.001 2 5 0.01 2 5 0.1 2 5 1 2 5 10 1Îź100Îź0.011 0.001 2 5 0.01 2 5 0.1 2 5 1 2 5 10 1Îź100Îź0.011 0.001 2 5 0.01 2 5 0.1 2 5 1 2 5 10 1Îź100Îź0.011 0.001 2 5 0.01 2 5 0.1 2 5 1 2 5 10 1Îź100Îź0.011 0.001 2 5 0.01 2 5 0.1 2 5 1 2 5 10 1Îź100Îź0.011 0.001 2 5 0.01 2 5 0.1 2 5 1 2 5 10 1Îź100Îź0.011 0.001 2 5 0.01 2 5 0.1 2 5 1 2 5 10 1Îź100Îź0.011 0.001 2 5 0.01 2 5 0.1 2 5 1 2 5 10 1Îź100Îź0.011 0.001 2 5 0.01 2 5 0.1 2 5 1 2 5 10 1Îź100Îź0.011 0.001 2 5 0.01 2 5 0.1 2 5 1 2 5 10 1Îź100Îź0.011 0.001 2 5 0.01 2 5 0.1 2 5 1 2 5 10 1Îź100Îź0.011 0.001 2 5 0.01 2 5 0.1 2 5 1 2 5 10 1Îź100Îź0.011 0.001 2 5 0.01 2 5 0.1 2 5 1 2 5 10 1Îź100Îź0.011 0.001 2 5 0.01 2 5 0.1 2 5 1 2 5 10 EmbeddingsLayer 1Layer 2Layer 3 Layer 4Layer 5Layer 6Layer 7 Layer 8Layer 9Layer 10Layer 11 Layer 12Layer 13Layer 14Layer 15 Layer 16Layer 17Layer 18Layer 19 Layer 20Layer 21Layer 22 Influence on lens (bits) Influence on model (bits) Ď = 0.25Ď = 0.90Ď = 0.90Ď = 0.44 Ď = 0.65Ď = 0.70Ď = 0.71Ď = 0.74 Ď = 0.71Ď = 0.73Ď = 0.71Ď = 0.74 Ď = 0.71Ď = 0.74Ď = 0.79Ď = 0.82 Ď = 0.83Ď = 0.85Ď = 0.89Ď = 0.90 Ď = 0.93Ď = 0.95Ď = 0.98 Figure 20.Causal fidelity in Pythia 410M across layers. In the top left corner of each plot is the Spearman rank correlation between the causal influence of each feature on the tuned lens and its influence on the model. Eliciting Latent Predictions from Transformers with the Tuned Lens OV SVD (L)OV SVD (R)QK SVD (L)QK SVD (R) W in SVD W in columns W out SVD W out rows Logit lens (real)0.07450.11640.07450.11640.07450.08640.07280.0796 Logit lens (random)0.06980.06950.06970.06880.06890.06890.06910.0688 Tuned lens (real)0.12400.16670.12400.16670.11930.15640.11960.1630 Tuned lens (random) 0.11640.11570.11770.11650.11630.12620.11630.1532 Table 3.The mean interpretability scores (as measured in Section E.3) for Pythia 125M, with several different interpretability techniques (Millidge and Black, 2022; Geva et al., 2020), comparing both the tuned lens and logit lens to randomly generated matrices. Where applicable, the notation (L) and (R) indicates that the results are for the left and right singular vectors, respectively. when viewed from the right angle, the modifications they make are often interpretable. These âinterpretable vectorsâ occur significantly more often than would be expected by random chance (Geva et al., 2022), and can be used to edit the modelâs behaviour, like reducing probability of specific tokens (Millidge and Black, 2022), decreasing toxicity (Geva et al., 2022), or editing factual knowledge (Dai et al., 2022). Although these interpretability methods differ in precise details, we can give a generic model for the interpretation of a model parameter W using the unembedding matrix: 8 T (W ) = top-k(f i (W )W U ) wheref i is some function from our parameter to a vector with the same dimensions as the hidden state, andW U is the unembedding matrix. In words: we extract a hidden state vector from our parameterWaccording to some procedure, project into token space, and take the topkmatching tokens. The resultingT (W )will be a list of tokens of sizek, which functions as a human interpretable view of the vectorf i (W ), by giving thektokens most associated with that vector. As an example, the model parameterWcould be the MLP output matrixW out for a particular layer, andf i the function selecting the ith column of the matrix. With the tuned lens, we modify the above to become: T (W ) = top-k(L â (f i (W ))W U ) where L â is the tuned lens for layer number â, projecting from the hidden state at layer â to the final hidden state. To test this hypothesis, we developed a novel automated metric for evaluating the performance of these parameter inter- pretability methods, based on the pretrained BPEmb encoder (Heinzerling and Strube, 2018), enabling much faster and more objective experimentation than possible with human evaluation. We describe this method in Appendix E.3. The results for Pythia 125M can be seen in Table 3. The parameters under both the tuned and logit lens consistently appear more interpretable than random. And the tuned lens appears to show benefit: the difference between random/real average scores is consistently higher with the tuned lens than the logit lens. However, preliminary experiments found much less improvement with larger models, where both the tuned and logit lens appeared to perform poorly. E.2. Interpretability score distributions The distribution of interpretability scores for a single parameter (the OV circuit) can be seen in Figure 21. (Interpretability distributions of other parameters appear to share these features as well.) The plot displays a major complication for static analysis methods: most singular vectors are not more interpretable than random, except for a minority lying in a long right tail. It also shows the necessity of comparison to randomly generated matrices: naively comparing the interpretability scores for the tuned lens against those for the baseline would imply a far greater improvement than actual, because the interpretability scores under the tuned lens are higher even for randomly generated matrices. E.3. Automated interpretability analysis method While humans can easily tell if a list of words represents some coherent category, this can be subjective and time-consuming to evaluate manually. Some previous work has attempted to automate this process, using GPT-3 in Millidge and Black 8 In most transformer language models, the function here should technically beT(W) = top-k(softmax(LN(f i (W)W U ))), including both softmax and LayerNorm. However, the softmax is not necessary because it preserves rank order, and the LayerNorm can be omitted for similar reasons (and assuming that either f i (W) is zero-mean or that W U has been left-centered). 8 Random shuffling applied to each matrix (head-wise for attention matrices), to approximate the element-wise marginal distribution. 8 Similar to above footnote, random shuffling applied to each W OV for each head. Eliciting Latent Predictions from Transformers with the Tuned Lens 00.20.40.60.8 0 200 400 600 800 Real Random OV circuit SVD interpretability scores (logit lens) Interpretability score Count 00.20.40.60.8 0 200 400 600 800 Real Random OV circuit SVD interpretability scores (tuned lens) Interpretability score Count Figure 21.The interpretability scores for the right singular vectors of the OV matricesW OV (following Millidge and Black (2022)) in Pythia 125M, compared with a randomly generated matrix, for both the logit lens (left), and the tuned lens (right). (2022) or the Perspective API in Geva et al. (2022), but these can still be too slow for quick experimentation. We created our own method, motivated by the observation that much of the human interpretable structure of these tokens appears to be related to their mutual similarity. Then, the mutual similarity of words can easily be measured by their cosine similarity under a word embedding (Sitikhu et al., 2019). In particular, we use the pretrained BPEmb encoder (Heinzerling and Strube, 2018) 9 . We encode each token, measure the cosine similarity pairwise between all tokens, and take the average value: I(T (W )) = P k,k i,j E(T i (W ))¡ E(T j (W )) k 2 where the subscript onT i (W )denotes thei-th token ofT (W ), andEdenotes the normalized BPEmb encoding function. This creates an interpretability metric I that measures something like the âmonosemanticity" of a set of tokens. Because BPEmb is a pretrained model, unlike most CBOW or skip-gram models, there is no ambiguity in the training data used to produce word similarities and results are easy to replicate. We find the interpretability scores given by the model to correspond with human judgment in most cases: it rarely identifies human-uninterpretable lists of tokens as interpretable, although it occasionally struggles to recognize some cases of interpretable structure, like syntactically related tokens, or cases where a list as a whole is more interpretable than a pairwise measure can capture. E.4. Improved Model Editing Geva et al. (2022) edit GPT-2-medium to be less toxic by changing the coefficients which feed into the columns of the MLP output matrix, under the key-value framework developed in Geva et al. (2020). They define both the value vectorv â i (i.e. the columns of the MLPâsW out ) and a coefficientm â i (i.e. the output ofW in before the activation function) for each columni and each layerâ. This is used to reduce the toxicity of GPT-2-medium by editing the coefficients of specific value vectors. They find: T (W out l ) = top-30(f i (W â out )W U ) where: f i (W â out ) = v â i They then concatenate these top-30 tokens for every layer and column, sending it to Perspective API, a Google API service which can return a toxicity score. They then sampled from the vectors that scored < 0.1 toxicity (where a score > 0.5 is classified as toxic), set each value vectorâs respective coefficient to 3 (because 3 was a higher than average activation), and ran the newly edited model through a subset of REALTOXICPROMPTS (Gehman et al., 2020) to measure the change in toxicity compared to the original model. 9 We also tried the fastText encoder (Bojanowski et al., 2016), but found BPEmb to give results better matching human judgment. Eliciting Latent Predictions from Transformers with the Tuned Lens Toxicity Severe Toxicity Sexually explicit ThreatProfanity Identity attack Perplexity Original/ Unedited0.500.0880.1590.0580.400.04327.66 Logit Lens Top-50.470.0770.1480.0570.380.04327.68 Logit Lens Top-100.450.0730.1440.0560.360.04027.70 Logit Lens Top-20 0.430.0720.1410.0570.330.04127.73 Tuned Lens Top-50.450.0790.1430.0560.360.04127.66 Tuned Lens Top-100.420.0630.1430.0570.330.03427.67 Tuned Lens Top-200.390.0610.1380.0520.310.03527.73 Table 4.Percentage of prompt generations labeled as toxic for OPT-125m. Best results in bold. Across all respective settings, the tuned lens outperformed the logit lens with no significant increases in perplexity. Although the value vectors were only graded & ranked by toxicity, there were still decreases in other metrics (severe toxicity, sexually explicit, etc) that were not directly selected against. However, these results did not generalize to other models tested. Changing the coefficients of value vectors can generate degenerate text such as always outputting â the" which is scored as non-toxic. To measure this effect, they calculate the perplexity of the edited models, where a higher perplexity may imply these degenerate solutions. We perform a similar experiment, but found the vast majority of value vectors that scored < 0.1 toxicity to be non-toxic as opposed to anti-toxic (e.g. semantic vectors like â the", â and", etc are scored as non-toxic). Alternatively, we set thekmost toxic value vectorâs coefficient to 0, as well as projecting with both the logit and tuned lens on OPT-125m, as seen in Table 4. We additionally tested pythia-125m, pythia-350m, gpt-2-medium, and gpt-neo-125m; however, there were no significant decreases in toxicity (i.e. > 2%) compared to the original, unedited models. E.5. Implementation The initial implementation of many of our static analysis experiments was helped by the use of thetransformer_lens library (Nanda, 2022).