Paper deep dive
Comparing Optimization Targets for Contrast-Consistent Search
Hugo Fry, Seamus Fallows, Ian Fan, Jamie Wright, Nandi Schoots
Models: DeBERTa, GPT-Neo, UnifiedQA T5-Large
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%
Last extracted: 3/12/2026, 6:33:46 PM
Summary
The paper investigates the optimization target of Contrast-Consistent Search (CCS), an unsupervised method for extracting truth representations from large language models. The authors introduce the Midpoint-Displacement (MD) loss function as a proxy for CCS, demonstrating that it achieves comparable or superior performance and provides insights into the trade-offs inherent in the CCS objective.
Entities (4)
Relation Signals (2)
Contrast-Consistent Search → extractsfrom → Large Language Models
confidence 95% · CCS is an unsupervised method for extracting knowledge from the hidden states of large language models.
Midpoint-Displacement loss function → isproxyfor → Contrast-Consistent Search
confidence 90% · We demonstrate that for a certain hyper-parameter value this MD loss function leads to a prober with very similar weights to CCS.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We investigate the optimization target of Contrast-Consistent Search (CCS), which aims to recover the internal representations of truth of a large language model. We present a new loss function that we call the Midpoint-Displacement (MD) loss function. We demonstrate that for a certain hyper-parameter value this MD loss function leads to a prober with very similar weights to CCS. We further show that this hyper-parameter is not optimal and that with a better hyper-parameter the MD loss function attains a higher test accuracy than CCS.
Tags
Links
Trouble viewing inline? Open PDF directly →
Full Text
31,901 characters extracted from source content.
Expand or collapse full text
Comparing Optimization Targets for Contrast-Consistent Search Hugo Fry Independent hugo.fry@fryfamily.co.uk Seamus Fallows Independent seamusfallows1@gmail.com Ian Fan Independent Jamie Wright Oxford University Nandi Schoots King’s College London Abstract We investigate the optimization target of Contrast-Consistent Search (CCS), which aims to recover the internal representations of truth of a large language model. We present a new loss function that we call the Midpoint-Displacement (MD) loss function. We demonstrate that for a certain hyper-parameter value this MD loss function leads to a prober with very similar weights to CCS. We further show that this hyper-parameter is not optimal and that with a better hyper-parameter the MD loss function attains a higher test accuracy than CCS. 1 Introduction The increased deployment of large language models to real-world applications has continued to highlight the discrepancy between the growing capabilities of language models and our limited understanding of their behavior [OpenAI, 2023]. It is widely believed that language models have internal representations encoding knowledge of the world [Li et al., 2023]. In order to deploy such systems safely, a better understanding of these internal representations is needed. In particular, language models will often confidently output false statements or ‘hallucinate’ in a number of settings [Ji et al., 2023]. It is therefore important to develop techniques aimed at discovering internal representations of truth. Mechanistic interpretability aims to reverse engineer the algorithms that the model weights im- plement at the level of individual neurons [Cammarata et al., 2020, Wang et al., 2022]. Other interpretability work more broadly aims to develop automatic techniques for probing and modifying human-interpretable concepts from a model’s activations [Ghorbani et al., 2019, Wang et al., 2022]. Recent work in the field of interpretability has presented contrast-consistent search (CCS) [Burns et al., 2022] as an unsupervised method for extracting knowledge from the hidden states of large language models. It is able to learn the truth value of statements using the negation consistency property of truth: a statement and its negation must have opposite truth values. By comparing the model representations corresponding to these two statements, one can find a direction in activation space that satisfies this consistency constraint. In this paper, we investigate the optimization target of CCS. Our contributions are as follows: • In Section 3.1, we present some conceptual clarifications on how CCS classifies truth and how it should be interpreted. •In Section 3.2, we present a heuristic explanation of CCS’s optimization target. Motivated by this explanation, we introduce the Midpoint-Displacement (MD) loss function as a proxy optimisation target for CCS. Socially Responsible Language Modelling Research (SoLaR) NeurIPS 2023 arXiv:2311.00488v1 [cs.LG] 1 Nov 2023 •In Section 4, we present an experimental comparison of many loss functions across a number of datasets and models. In particular, we demonstrate that, for a certain choice of the hyper- parameter, the truth direction found by the MD loss function is a good proxy for CCS, as measured by cosine similarity. •Further, we tentatively show that this hyper-parameter is not optimal and that with a better hyper-parameter the MD loss function outperforms the current state of the art (CCS), along with all other loss functions that we have tested. 2 Background Contrast-Consistent Search (CCS), developed by Burns et al. [2022], is an unsupervised method for extracting knowledge from the hidden states of large language models. Given only unlabelled model activations, CCS is able to accurately classify statements according to their truth value. It does this by utilising the negation consistency property of truth: a statement and its negation must have opposite truth values. Framed in terms of probabilities, given the probabilitypthat a proposition is true, the probability its negation is true is1−p. CCS works by finding a direction in activation space that satisfies this consistency constraint. Dataset.We take a dataset ofcontrast pairs(x + i ,x − i ) n i=1 consisting of natural language statements x + i and their logical oppositesx − i . The pairs are formed by taking a questionq i and appending with one of two mutually exclusive answers. Contrast pairs are fed to a pre-trained language model to obtain a set of representations(φ + i ,φ − i ) n i=1 , whereφ ± i :=φ(x ± i )∈R d is the activation vector of a particular layer for the inputx ± i . These representations are then used to train a linear classifier according to an objective designed to enforce negation consistency on contrast pairs. Below is an example of a contrast pair: q i : “Are cats mammals?” x − i : “Are cats mammals? No.” x + i : “Are cats mammals? Yes.” In order to avoid training the classifier to simply detect the presence of the mutually exclusive answers, the setsφ + i andφ − i should be independently normalized. The normalized representations are given by ̃ φ ± i := φ ± i −μ ± σ ± ,(1) where(μ ± ,σ ± )are the means and standard deviations of the respective sets. For convenience, we will omit the tilde and simply useφ ± i to represent the normalized representations. Loss Function.A linear classifierp θ,b :φ→σ(θ T φ+b)is trained on the normalized activations, whereσis the sigmoid function,θis a vector of weights andbis a bias. The loss function is given by L CCS (θ,b) := 1 n n X i=1 1−p θ,b (φ + i )−p θ,b (φ − i ) 2 + min p θ,b (φ + i ),p θ,b (φ − i ) 2 .(2) The first term encourages the classifier to find features that are negation-consistent and the second term is included to disincentivise the degenerate assignmentp θ,b (φ + i ) =p θ,b (φ − i ) = 0.5. Metrics.The accuracy of CCS is calculated using the ground-truth dataset labels. That is, CCS probers are trained in an unsupervised way, but evaluated using supervised labels. Inference.To make a prediction on an exampleq i , after training, the average ̄p i (q i ) := 1 2 (p θ,b (φ + i ) + (1−p θ,b (φ − i )))(3) is compared to0.5with ̄p i >0.5corresponding to either the answer "yes" or "no" based on whichever gives the maximum accuracy on a given test set. 1 Burns et al. [2022] show that a CCS prober trained with the above loss function outperforms zero-shot outputs of the model with a mean accuracy of 71.2%, against 67.2% for zero-shot. 1 Note that this supervised method is used for simplicity of evaluation; Burns et al. [2022] describe a completely unsupervised method based on conjunctions for determining this assignment. 2 3 Methods: Introducing New Loss Functions In this section we introduce an alternative loss function to CCS. First, we provide some conceptual clarification and address a possible misconception about how CCS works. The code used to run our experiments can be found athttps://github.com/ash-ai-safety-hub/g3-nandi. 3.1 Clarifying CCS A natural guess for how CCS is able to accurately classify truth is that the normalized model repre- sentations are approximately clustered according to truth and that CCS is able to find a hyperplane that separates these two clusters. However, this explanation turns out to be incorrect. In Appendix A.1 we provide a specific example showing that model activations do not in fact cluster in this way. Recall that after training, an exampleq i is classified according to ̄p(q i )>0.5. Using equation 3, this condition reduces to σ(θ T φ + i +b)> σ(θ T φ − i +b) ⇒θ T (φ + i −φ − i )>0. We see that CCS is classifying only according to the displacement vectors(φ + i −φ − i ). Since these are translation invariant, CCS does not require the contrast pairs to be separable by a hyperplane. Additionally, the original paper presents CCS as learning probabilities for the truth values of contrast pairs. We suggest abandoning this probabilities framing. We show in Appendix A.2 that CCS can still perform well even when the probabilities are strongly clustered around 0.5. This paints a different picture to that presented in [Burns et al., 2022]. 3.2 Midpoint-Displacement (MD) Loss Function In this section we present a heuristic explanation of CCS’s optimization target and use this explanation to introduce a new loss function. Note first that the CCS loss function incentivises increasing the separation of the prober outputs of contrast pairs|p(φ + i )−p(φ − i )|. Consider a CCS prober p(φ) =σ(θ T φ+b)in whichθis constrained to a fixed norm 2 |θ|=c. Using the normalized weight vector ˆ θ, we define the following quantities: u i :=φ + i −φ − i ,and σ 2 d := 1 n X i ( ˆ θ T u i ) 2 . (4) Hereu i is the displacement between the activations of a contrast pair andσ 2 d is the mean square separation of the activations of the contrast pairs along the directionθ. Furthermore, we analogously define v i :=φ + i +φ − i ,and σ 2 m := 1 n X i ( ˆ θ T v i ) 2 . (5) Here v i 2 is the midpoint of the activations of a contrast pair and σ 2 m 4 is the mean square value of the midpoint of the activations of the contrast pairs along the directionθ. In order for CCS to increase the difference between prober outputs, one might expect that CCS finds a direction that increases the difference of the prober inputs (since sigmoid is a monotonically increasing function). That is to say, one might expect CCS will find a direction that maximisesσ 2 d . However, ifσ 2 m is much larger thanσ 2 d , then the input to the sigmoid for each contrast pair (i.e. θ T φ + i +bandθ T φ − i +b) will be pushed into the same saturation regime of the sigmoid. This results in a lower difference in prober outputs of contrast pairs, which in turn results in a trade off between maximisingσ 2 d while minimisingσ 2 m . 2 We use|...|to denote the Euclidean norm throughout this paper. 3 It should be stressed that this trade-off betweenσ 2 d andσ 2 m is purely an artifact of the double saturation of sigmoid used in the CCS prober. Since this trade off should occur no matter what|θ|=cis constrained to, we propose that the unconstrained CCS prober is in general optimising for some balance betweenσ 2 d andσ 2 m . To test this hypothesis, we propose a new loss function and demonstrate that this new loss function is a good proxy optimisation target for CCS. The new loss function is given by L MD = (λ−1)σ 2 d +λ·σ 2 m ,(6) whereλ∈[0,1]is a hyper-parameter controlling the relative trade off betweenσ 2 d andσ 2 m , and the weight vectorθis constrained to satisfy|θ|= 1. In Appendix B we introduce two new loss functions: the Mean Absolute (MA) loss function and the Square Mean Root (SMR) loss function. In the following experiments we compare the MD loss function with a number of other loss functions, including the CCS, MA and SMR loss functions along with Principal Component Analysis (PCA). 4 Results: Comparison of MD with CCS In this section we investigate the empirical similarity between the MD and CCS loss functions. The MD-CCS and MD-Accuracy (MD-Acc) loss functions are both trained using the Midpoint- Displacement loss function but with different hyper-parameter searches. The implementation details of our experiments can be found in Appendix C. For comparison, we include probers trained using a variety of loss functions. The MA and SMR loss functions are both based on taking the mean displacements (further details can be found in Appendix B). The PCA loss function identifies the first principal component of the displacements. The random probers (Rand.) are found by randomly initializing 10 weight vectors and taking the average resulting accuracy. Lastly, the supervised probers (Superv.) are probers trained on labelled data with the same structure as the CCS probers,p(φ) =σ(θ T φ+b). In Table 1 we show the test accuracies of probers trained using the MD loss function on various datasets and models. We tentatively find that the accuracies of the new probers are similar to those achieved by CCS, and often out-perform CCS. Note that MD-Acc probers get a higher test accuracy than both the MD-CCS and CCS probers for three out of four models, for an average difference of around 4%. Model Loss Function CCSMD-CCSMD-AccMASMRPCARand.Superv. UQA (E)0.68630.69020.74140.73990.74190.73830.63630.8839 UQA (D)0.83050.82000.81800.75500.74600.75250.62860.9140 DeBERTa0.77400.78550.87350.86500.85850.86050.72880.9135 GPT-Neo0.55100.57550.58980.58200.55550.57370.56030.7580 Average0.71050.71780.75570.73550.72550.73130.63850.8674 Table 1: We compare test accuracies of different loss functions averaged over five datasets, using the activations of a number of models. For each row we have emboldened the loss function that obtained the highest average test accuracy, not including the supervised loss. The (E) and (D) labels refer to the encoder and decoder layers of the UQA model. In Table 2 we find that the average cosine similarity between the weight vector of the CCS prober and the weight vector of the prober trained using our new MD method is about 0.63. For reference, the probability of two uniformly sampled 1024-dimensional unit vectors having a cosine similarity of 0.63 or higher is approximately10 −237 . Note that the CCS probers had an average cosine similarity with themselves of only 0.78. This suggests that the MD-CCS loss function is a good proxy optimization target for CCS. The only difference between the MD-CCS and MD-Acc loss functions is the value of their hyper- parameterλthat controls the relative trade off betweenσ 2 d andσ 2 m . Since they give very different cosine similarities, we find the similarity of MD to CCS is dependent on this trade off. 4 Model Loss Function CCSMD-CCSMD-AccMASMRPCARand.Superv. UQA (E)0.83590.70340.29950.14480.19910.24060.02220.2583 UQA (D)0.87870.72690.53030.16870.24320.17920.02280.6014 DeBERTa0.86430.62090.27860.23090.20240.07410.02020.4617 GPT-Neo0.52770.48300.41640.02260.04850.19010.02450.1347 Average0.77670.63360.38120.14180.17330.17100.02240.3640 Table 2: We compute the average cosine similarities of the directions found using different loss functions to the directions of 20 CCS probers. We average over five datasets using the activations of four different models. For each row we have emboldened the loss function that obtained the highest average cosine similarity with CCS, not including the CCS loss function. Note that the (E) and (D) refer to the encoder and decoder layers of the UQA model. In Appendix D, we include a breakdown of the experimental results for each dataset and model along with the hyper-parameters that are used for each loss function. It should be noted that the hyper-parameter used for the MD-CCS loss function does not change substantially between datasets or models. This suggests that the form of the proxy optimization target we have identified for CCS is robust to changes in the dataset and model. 5 Discussion Our Midpoint-Displacement loss function sheds light on which aspects of the data CCS is picking up on. We find that the Midpoint-Displacement loss function produces probers that behave very similarly to those produced by CCS, as measured by cosine similarity. We verified that other reasonable loss functions, with comparable accuracy to CCS, do not get high cosine similarity to CCS. Our findings suggest that the specific loss function formulation is not essential for performance, in that there are multiple loss functions that achieve high accuracy. Instead, the unique training data that CCS uses, which allows the identification of displacement betweenφ + andφ − , is what drives the success of CCS. 5.1 Limitations While the new loss functions we have proposed are unsupervised, they contain a hyper-parameter. The hyper-parameter is chosen using a supervised grid search and therefore supervised labels are currently required to use our loss functions. Additionally, the results obtained in this paper could be strengthened by testing across more datasets and models. 5.2 Future Work Future work could also consider developing an unsupervised method of determining the hyper- parameter based on data statistics. This would eliminate the need for any supervised labels. Alterna- tively, we could establish a hyper-parameter with high transferability between datasets. Moreover, we propose that a small number of supervised examples may be sufficient to find a dataset-specific hyper-parameter. In this paper we have evaluated similarity between probers through the metric of cosine similarity. Future work could additionally consider comparing the behavioral similarity between probers, by measuring their empirical pairwise agreement for contrast pairs in the test dataset. 5.3 Social Impact Statement Our work is motivated by avoiding harmful behaviors of language models, such as deception. To detect deception in LLMs, we may compare model outputs to the model’s latent knowledge. CCS is a nascent method for extracting truthfulness from latent representations. We aim to clarify CCS and we hope our results will inform future evaluation techniques that extract latent knowledge. We do not foresee harmful applications of our work. 5 Acknowledgments This research was supported by the AI Safety Hub Labs programme. References Sid Black, Gao Leo, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, March 2021. URLhttps://doi. org/10.5281/zenodo.5297715. If you use this software, please cite it using these metadata. Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering Latent Knowledge in Language Models Without Supervision, December 2022. Nick Cammarata, Shan Carter, Gabriel Goh, Chris Olah, Michael Petrov, and Ludwig Schubert. Thread: Circuits, mar 2020. ISSN 2476-0757. URLhttps://distill.pub/2020/circuits. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions, May 2019. Amirata Ghorbani, James Wexler, James Zou, and Been Kim. arxiv.org/pdf/1902.03129.pdf, 2019. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention, 2021. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation.ACM Comput. Surv., 55(12):248:1–248:38, 2023. doi: 10.1145/3571730. URLhttps://doi.org/10.1145/ 3571730. Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. Unifiedqa: Crossing format boundaries with a single qa system, 2020. Kenneth Li, Aspen K. Hopkins, David Bau, Fernanda B. Viégas, Hanspeter Pfister, and Martin Wattenberg. Emergent world representations: Exploring a sequence model trained on a synthetic task. InThe Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URLhttps://openreview.net/pdf?id= DeG07_TcZvT. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. InProceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142–150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URLhttps: //aclanthology.org/P11-1015. Julian McAuley and Jure Leskovec. Hidden factors and hidden topics: Understanding rating dimensions with review text. InProceedings of the 7th ACM Conference on Recommender Systems, RecSys ’13, page 165–172, New York, NY, USA, 2013. Association for Computing Machinery. ISBN 9781450324090. doi: 10.1145/2507157.2507163. URLhttps://doi.org/ 10.1145/2507157.2507163. OpenAI. Gpt-4 technical report, 2023. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems, February 2020. Kevin Wang, Variengien Re, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the Wild: A Circuit for Indirect Object Identification in GPT-2 Small, 2022. URLhttps: //arxiv.org/abs/2211.00593. Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classifi- cation. InProceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1, NIPS’15, page 649–657, Cambridge, MA, USA, 2015. MIT Press. 6 A Clarifying CCS A.1 CCS Does Not Determine Truth Using a Hyperplane Letθbe the weight vector of the linear prober found by CCS and denote the unit-normalised weight vector by ˆ θ. We call this normalised weight vector ‘the direction found by CCS’. In Figure 1 we project the activations vectorsφ i corresponding to datapointsx i onto the first principal component and onto the direction found by CCS (when trained on the decoder). We consider the activations for which CCS outputs 0.5 and show that this ‘decision boundary’ does not cleanly separate the true and false datapoints. Figure 1: We consider activations of the T5-base model on the BoolQ train dataset. On thex-axis we plot the projections of the activationsφ i onto the first principle component and on they-axis we plot the datapointsφ i projected onto ˆ θ. We colour the datapointsφ i by the (ground-truth) truth-labels ‘true’ (red) and ‘false’ (blue) of the datapointsx i . The horizontal line indicates the inputs for which CCS outputs 0.5. A.2 CCS is Miscalibrated In Figure 2, we show histograms of CCS prober outputs evaluated on the last hidden state of both the encoder and decoder of UQA for the BoolQ dataset (compare to Figure 1 of Burns et al. [2022]). Interpreting the output of CCS as a probability would suggest much higher confidence for the encoder than the decoder. However, from Appendix D, the test accuracy on the encoder (0.523) is significantly worse than on the decoder (0.978). This suggests that the ouput of CCS should not be interpreted as probabilities. 7 (a)(b) Figure 2: (a) Histogram displaying the CCS prober outputs evaluated on the last hidden state of the encoder of UnifiedQA T5-Large for the BoolQ dataset. (b) Histogram displaying the CCS prober outputs evaluated on the last hidden state of the decoder of UnifiedQA T5-Large for the BoolQ dtatset. Despite the encoder having a higher confidence in the prober outputs, the encoder has a lower test accuracy (0.523) than the decoder (0.978). B Mean-Based Loss Functions We define the displacement vector between the activations of a contrast pair as follows. u i =φ + i −φ − i . We define the mean and standard deviation of the modulus of the displacements along a directionθ as follows: μ= 1 n X i |θ T u i |, σ= s 1 n X i (|θ T u i |−μ) 2 . We then define the Mean Absolute (MA) loss function as L MA = (1−λ)μ+λ·σ. The Square Mean Root (SMR) is given by μ SMR = s 1 n X i (θ T u i ) 2 . Analogously, we define the SMR loss function to be L SMR = (1−λ)μ SMR +λ·σ. In both loss functions, we constrain|θ|= 1. C Implementation Details For the experiments presented in this paper, all probers were trained for 1000 epochs with a learning rate of 0.01. All of the experiments were run for the following five datasets: • IMDB [Maas et al., 2011] • Amazon [McAuley and Leskovec, 2013] • BoolQ [Clark et al., 2019] • RTE [Wang et al., 2020] 8 • AG News [Zhang et al., 2015] and the following three models: • UnifiedQA T5-Large (encoder-decoder architecture) [Khashabi et al., 2020] • GPT-Neo (decoder-only architecture) [Black et al., 2021] • DeBERTa (encoder-only architecture) [He et al., 2021] We use the hidden states of the encoder and decoder where available. In our experiments, we used 1000 examples from each dataset consisting of a 600-400 train-test split. For each loss function on each dataset and model, we use a grid search to determine the hyper-parameter that maximised train accuracy as follows: 1.Create eleven hyper-parameter values by splitting the interval[0,0.99]into eleven evenly spaced points (separated by 0.099). 2.For each of these values, we train three probers with a random seed. We evaluate the average train accuracy over the three probers. 3.We identify the hyper-parameter with the highest average train accuracy. Let this hyper- parameter beλ ⋆ . 4. Consider the new interval given by[λ ⋆ −0.099,λ ⋆ + 0.099]∩[0,0.99]. 5. Repeat the process again on the new interval by splitting it evenly into eleven points. 6.Identify the hyper-parameter with the highest average train accuracy for the points on this new interval. This should calculate the hyper-parameter with a precision of roughly±0.02. An analogous grid search is performed to identify the hyper-parameter used for the MD-CCS loss function, by finding the hyper-parameter that maximises average cosine similarity with 20 CCS probers. In the case of the MD-CCS hyper-parameter, the initial interval we consider is[0.9,0.999] as opposed to[0,0.99](initial experiments always converged to a hyper-parameter in this interval - so we used this initial interval to reduce compute time). Having found the optimal hyper-parameterλ ⋆ , we then train ten probers withλ=λ ⋆ and random seeds. From these ten trained probers, we pick the prober that minimises the loss function (this is in line with the method used in Burns et al. [2022]). This is the prober that is used to compute the values found in the tables of this paper. 9 D Granular Data for Models and Specific Datasets D.1 Accuracies Dataset Loss Function CCSMD-CCSMD-AccMASMRPCARand.Superv. IMDB0.92000.92250.92000.92250.92000.92200.79080.9300 Amazon0.94380.94690.94690.94690.94690.94690.73440.9469 BoolQ0.52250.51670.78250.79000.79250.78000.60470.8875 AG News0.52750.53750.53000.52750.52750.51500.51520.9500 RTE0.51750.52750.52750.51250.52250.52750.53650.7050 Average0.68630.69020.74140.73990.74190.73830.63630.8839 Table 3: We compare test accuracies of different loss functions using the last hidden state of the encoder of UnifiedQA T5-Large for a number of datasets. Dataset Loss Function CCSMD-CCSMD-AccMASMRPCARand.Superv. IMDB0.92750.92500.92000.93250.93250.91500.71900.9200 Amazon0.94000.94250.94500.60500.59000.76750.63550.9425 BoolQ0.97750.96250.96250.98000.98750.87250.63030.9850 AG News0.59240.53000.53000.56750.53250.51750.57500.9550 RTE0.71500.74000.73250.69000.68750.69000.58330.7675 Average0.83050.82000.81800.75500.74600.75250.62860.9140 Table 4: We compare test accuracies of different loss functions using the last hidden state of the decoder of UnifiedQA T5-Large for a number of datasets. Dataset Loss Function CCSMD-CCSMD-AccMASMRPCARand.Superv. IMDB0.62250.70500.76500.74000.52250.74350.55250.8225 Amazon0.59500.62000.62000.61750.71000.57250.64220.8800 BoolQ0.50750.53500.533750.50750.50750.52250.53600.5800 AG News0.52500.50750.51500.53750.53250.51750.54780.9500 RTE0.50500.51000.51500.50750.50500.51250.52320.5575 Average0.55100.57550.58980.58200.55550.57370.56030.7580 Table 5: We compare test accuracies of different loss functions using the last hidden state of the decoder of GTP-Neo for a number of datasets. Dataset Loss Function CCSMD-CCSMD-AccMASMRPCARand.Superv. IMDB0.95500.95000.94750.95500.94750.95000.80670.9550 Amazon0.94750.94250.94750.94000.95000.94250.81450.9500 BoolQ0.66000.74750.81000.81250.81250.81000.71000.8275 AG News0.50000.50000.82500.77000.73000.82500.65450.9425 RTE0.80750.78750.83750.84750.85250.7750.65850.8925 Average0.77400.78550.87350.86500.85850.86050.72880.9135 Table 6: We compare test accuracies of different loss functions using the last hidden state of the encoder of DeBERTa for a number of datasets. 10 D.2 Cosine Similarities Dataset Loss Function CCSMD-CCSMD-AccMASMRPCARand.Superv. IMDB0.93050.70650.29290.20640.31670.29300.02140.3862 Amazon0.96350.81770.31490.27770.38670.31500.02310.7639 BoolQ0.80820.80000.01870.03320.04420.01940.01810.0860 AG News0.50810.48170.47580.19970.20090.18350.02670.0328 RTE0.96910.71110.39540.00690.04680.39190.02150.0224 Average0.83590.70340.29950.14480.19910.24060.02220.2583 Table 7: We compute the average cosine similarities of the directions found using different loss functions to the directions of 20 CCS probers. We use the last hidden state of the encoder of UnifiedQA T5-Large for a number of datasets. Dataset Loss Function CCSMD-CCSMD-AccMASMRPCARand.Superv. IMDB0.99050.71110.40480.38670.38150.23780.02120.6790 Amazon0.83570.60210.39690.03870.00760.07600.01650.7499 BoolQ0.98500.86940.53930.27680.28080.11580.01940.8640 AG News0.59240.53480.45470.01090.42740.35480.02540.0219 RTE0.98990.91690.85580.13030.11870.11150.03140.6924 Average0.87870.72690.53030.16870.24320.17920.02280.6014 Table 8: We compute the average cosine similarities of the directions found using different loss functions to the directions of 20 CCS probers. We use the last hidden state of the decoder of UnifiedQA T5-Large for a number of datasets. Dataset Loss Function CCSMD-CCSMD-AccMASMRPCARand.Superv. IMDB0.62960.24830.07850.05490.03670.06240.02120.3465 Amazon0.30050.30590.30660.00920.03590.13510.02660.2310 BoolQ0.64780.66830.66280.01750.01750.35810.02130.0136 AG News0.36440.46270.31230.01330.01230.17810.02980.0531 RTE0.69640.72960.72180.01830.14000.21660.02370.0295 Average0.52770.48300.41640.02260.04850.19010.02450.1347 Table 9: We compute the average cosine similarities of the directions found using different loss functions to the directions of 20 CCS probers. We use the last hidden state of the decoder of GPT-Neo for a number of datasets. Dataset Loss Function CCSMD-CCSMD-AccMASMRPCARand.Superv. IMDB0.93760.55040.54660.25990.16850.05540.01430.6811 Amazon0.95380.27600.24060.16570.24010.06480.01780.8104 BoolQ0.78230.73770.03570.04480.05430.03580.01750.2912 AG News0.70030.64010.01450.17500.01400.01460.02220.0435 RTE0.94760.90020.55570.50930.53490.19980.02940.4823 Average0.86430.62090.27860.23090.20240.07410.02020.4617 Table 10: We compute the average cosine similarities of the directions found using different loss functions to the directions of 20 CCS probers. We use the last hidden state of the encoder of DeBERTa for a number of datasets. 11 D.3 Results from Hyper-Parameter Grid Search We find that the MD-CCS hyper-parameter is always close to 1. This suggests that the trade-off most similar to CCS is one that prioritises minimisingσ 2 m (which has a coefficient of at least 0.9) over maximisingσ 2 d (which has a coefficient of at most 0.1). For the MD-Acc loss function, we find that the hyper-parameter is often either close to 0 or close to 1. When the hyper-parameter is close to 1, the MD-CCS and MD-Acc have landed on the same hyper-parameter. On the other hand, when MD-Acc has aλis close to 0, the loss function is very similar to PCA. Dataset Loss Function MD-CCSMD-AccMASMR IMDB0.98910.00000.75240.6930 Amazon0.98120.00000.69300.5940 BoolQ0.97520.00000.00000.0000 AG News0.98710.98010.00000.0000 RTE0.98910.09900.99000.1980 Table 11: Hyper-parameters values for the encoder of UnifiedQA T5-Large. Dataset Loss Function MD-CCSMD-AccMASMR IMDB0.98710.89100.67320.4950 Amazon0.96530.71280.69300.6930 BoolQ0.95350.79200.59400.5544 AG News0.98120.89100.90090.0000 RTE0.90000.81180.37620.1980 Table 12: Hyper-parameters values for the decoder of UnifiedQA T5-Large. Dataset Loss Function MD-CCSMD-AccMASMR IMDB0.98910.99000.59400.4950 Amazon0.98910.89100.49500.6534 BoolQ0.97520.00000.00000.0990 AG News0.98910.00000.00000.8910 RTE0.97920.33660.31680.2772 Table 13: Hyper-parameters values for the encoder of DeBERTa. Dataset Loss Function MD-CCSMD-AccMASMR IMDB0.97130.09900.57420.4950 Amazon0.99110.99000.59400.5940 BoolQ0.98710.99000.91080.4950 AG News0.98710.89100.97020.9900 RTE0.98710.99000.99000.5940 Table 14: Hyper-parameters values for the decoder of GPT-Neo. 12